2024-09-11T03:20:46.967 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T03:20:46.986 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T03:20:47.075 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900062 branch: wip-hemanth-testing-2024-09-10-0723 description: rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} email: hyelloji@redhat.com first_in_suite: false job_id: '7900062' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 - volumes: size: 50 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-hemanth-testing-2024-09-10-0723 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms inject socket failures: 5000 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 filestore queue throttle high multiple: 2 filestore queue throttle max multiple: 10 journal throttle high multiple: 2 journal throttle max multiple: 10 osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd heartbeat use min delay socket: true osd max backfills: 9 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: min_in: 4 workunit: branch: wip-hemanth-testing-2024-09-10-0723 sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 owner: scheduled_hyelloji@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 9167 sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-hemanth-testing-2024-09-10-0723 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 targets: smithi084.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLeljjbTw4J22x3OB4GmUqi+eqCwa86YQ448NnR6fstXlLzAQxujSnQ++CX7W2jcDipAy2qxo0LP2zDwEPT0BVc= smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOwUxU0gK4yy/bX+ucjWYyCTgQ1OvlrbWyM5gwhY9rID8Qn/BwP3Nz/rD/xwBd1YWLfgwWkeunV24xociQ4gYI0= smithi175.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNhYPNVIQviFK1I5w557C7EZOe1Z+yaw13p9+Wx76f6NtTgb7C08KzX04WpdcWfp6EiJsIlxu1S3iT5gxdxEQgM= tasks: - install: branch: reef exclude_packages: - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - ceph-base-debuginfo - ceph-common-debuginfo - ceph-immutable-object-cache-debuginfo - ceph-radosgw-debuginfo - ceph-test-debuginfo - ceph-base-debuginfo - ceph-mgr-debuginfo - ceph-mds-debuginfo - ceph-mon-debuginfo - ceph-osd-debuginfo - ceph-fuse-debuginfo - librados-devel-debuginfo - libcephfs2-debuginfo - librados2-debuginfo - librbd1-debuginfo - python3-cephfs-debuginfo - python3-rados-debuginfo - python3-rbd-debuginfo - python3-rgw-debuginfo - rbd-fuse-debuginfo - rbd-mirror-debuginfo - rbd-nbd-debuginfo - ceph-volume - cephadm: conf: mon: auth allow insecure global id reclaim: true - thrashosds: chance_pgnum_grow: 3 chance_pgpnum_fix: 1 timeout: 1200 - rados: clients: - client.2 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_17:16:14 tube: smithi user: hyelloji verbose: false verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T03:20:47.075 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa; will attempt to use it 2024-09-11T03:20:47.076 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks 2024-09-11T03:20:47.076 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T03:20:47.077 INFO:teuthology.task.internal:Saving configuration 2024-09-11T03:20:47.092 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T03:20:47.098 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T03:20:47.131 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900062', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 03:15:48.955365', 'locked_by': 'scheduled_hyelloji@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLeljjbTw4J22x3OB4GmUqi+eqCwa86YQ448NnR6fstXlLzAQxujSnQ++CX7W2jcDipAy2qxo0LP2zDwEPT0BVc='} 2024-09-11T03:20:47.161 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900062', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 03:15:48.954123', 'locked_by': 'scheduled_hyelloji@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOwUxU0gK4yy/bX+ucjWYyCTgQ1OvlrbWyM5gwhY9rID8Qn/BwP3Nz/rD/xwBd1YWLfgwWkeunV24xociQ4gYI0='} 2024-09-11T03:20:47.194 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi175.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900062', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 03:15:48.956269', 'locked_by': 'scheduled_hyelloji@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNhYPNVIQviFK1I5w557C7EZOe1Z+yaw13p9+Wx76f6NtTgb7C08KzX04WpdcWfp6EiJsIlxu1S3iT5gxdxEQgM='} 2024-09-11T03:20:47.194 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T03:20:47.200 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-09-11T03:20:47.201 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-09-11T03:20:47.201 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-09-11T03:20:47.201 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T03:20:47.301 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7bc56c7be0>, signals=[15]) 2024-09-11T03:20:47.301 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T03:20:47.309 INFO:teuthology.task.internal:Opening connections... 2024-09-11T03:20:47.309 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi084.front.sepia.ceph.com 2024-09-11T03:20:47.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T03:20:47.396 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-09-11T03:20:47.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T03:20:47.478 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi175.front.sepia.ceph.com 2024-09-11T03:20:47.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T03:20:47.565 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T03:20:47.572 DEBUG:teuthology.orchestra.run.smithi084:> uname -m 2024-09-11T03:20:47.591 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2024-09-11T03:20:47.592 DEBUG:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2024-09-11T03:20:47.649 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="CentOS Stream" 2024-09-11T03:20:47.650 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="9" 2024-09-11T03:20:47.650 INFO:teuthology.orchestra.run.smithi084.stdout:ID="centos" 2024-09-11T03:20:47.650 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="rhel fedora" 2024-09-11T03:20:47.650 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="9" 2024-09-11T03:20:47.650 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el9" 2024-09-11T03:20:47.650 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T03:20:47.650 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2024-09-11T03:20:47.650 INFO:teuthology.orchestra.run.smithi084.stdout:LOGO="fedora-logo-icon" 2024-09-11T03:20:47.650 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T03:20:47.650 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://centos.org/" 2024-09-11T03:20:47.650 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T03:20:47.650 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T03:20:47.651 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T03:20:47.651 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2024-09-11T03:20:47.684 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-09-11T03:20:47.703 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-09-11T03:20:47.703 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-09-11T03:20:47.761 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2024-09-11T03:20:47.761 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="9" 2024-09-11T03:20:47.761 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2024-09-11T03:20:47.761 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2024-09-11T03:20:47.761 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="9" 2024-09-11T03:20:47.761 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el9" 2024-09-11T03:20:47.762 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T03:20:47.762 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2024-09-11T03:20:47.762 INFO:teuthology.orchestra.run.smithi121.stdout:LOGO="fedora-logo-icon" 2024-09-11T03:20:47.762 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T03:20:47.762 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2024-09-11T03:20:47.762 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T03:20:47.762 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T03:20:47.762 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T03:20:47.762 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-09-11T03:20:47.791 DEBUG:teuthology.orchestra.run.smithi175:> uname -m 2024-09-11T03:20:47.810 INFO:teuthology.orchestra.run.smithi175.stdout:x86_64 2024-09-11T03:20:47.810 DEBUG:teuthology.orchestra.run.smithi175:> cat /etc/os-release 2024-09-11T03:20:47.868 INFO:teuthology.orchestra.run.smithi175.stdout:NAME="CentOS Stream" 2024-09-11T03:20:47.868 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION="9" 2024-09-11T03:20:47.868 INFO:teuthology.orchestra.run.smithi175.stdout:ID="centos" 2024-09-11T03:20:47.868 INFO:teuthology.orchestra.run.smithi175.stdout:ID_LIKE="rhel fedora" 2024-09-11T03:20:47.868 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION_ID="9" 2024-09-11T03:20:47.868 INFO:teuthology.orchestra.run.smithi175.stdout:PLATFORM_ID="platform:el9" 2024-09-11T03:20:47.868 INFO:teuthology.orchestra.run.smithi175.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T03:20:47.868 INFO:teuthology.orchestra.run.smithi175.stdout:ANSI_COLOR="0;31" 2024-09-11T03:20:47.869 INFO:teuthology.orchestra.run.smithi175.stdout:LOGO="fedora-logo-icon" 2024-09-11T03:20:47.869 INFO:teuthology.orchestra.run.smithi175.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T03:20:47.869 INFO:teuthology.orchestra.run.smithi175.stdout:HOME_URL="https://centos.org/" 2024-09-11T03:20:47.869 INFO:teuthology.orchestra.run.smithi175.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T03:20:47.869 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T03:20:47.869 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T03:20:47.869 INFO:teuthology.lock.ops:Updating smithi175.front.sepia.ceph.com on lock server 2024-09-11T03:20:47.900 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T03:20:47.910 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T03:20:47.921 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T03:20:47.921 DEBUG:teuthology.orchestra.run.smithi084:> test '!' -e /home/ubuntu/cephtest 2024-09-11T03:20:47.924 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-09-11T03:20:47.926 DEBUG:teuthology.orchestra.run.smithi175:> test '!' -e /home/ubuntu/cephtest 2024-09-11T03:20:47.943 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T03:20:47.950 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T03:20:47.950 DEBUG:teuthology.orchestra.run.smithi084:> test -z $(ls -A /var/lib/ceph) 2024-09-11T03:20:47.981 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-09-11T03:20:47.984 DEBUG:teuthology.orchestra.run.smithi175:> test -z $(ls -A /var/lib/ceph) 2024-09-11T03:20:48.074 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T03:20:48.186 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T03:20:48.202 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-11T03:20:48.202 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-11T03:20:48.203 DEBUG:teuthology.orchestra.run.smithi084:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T03:20:48.203 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T03:20:48.203 DEBUG:teuthology.orchestra.run.smithi175:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T03:20:48.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:20:48.221 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-09-11T03:20:48.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:20:48.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:20:48.222 DEBUG:teuthology.orchestra.run.smithi175:> uname -r 2024-09-11T03:20:48.223 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-09-11T03:20:48.280 INFO:teuthology.orchestra.run.smithi084.stdout:5.14.0-503.el9.x86_64 2024-09-11T03:20:48.280 INFO:teuthology.task.kernel:Running kernel on smithi084: 5.14.0-503.el9.x86_64 2024-09-11T03:20:48.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum install -y kernel 2024-09-11T03:20:48.282 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-503.el9.x86_64 2024-09-11T03:20:48.282 INFO:teuthology.orchestra.run.smithi175.stdout:5.14.0-503.el9.x86_64 2024-09-11T03:20:48.282 INFO:teuthology.task.kernel:Running kernel on smithi121: 5.14.0-503.el9.x86_64 2024-09-11T03:20:48.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2024-09-11T03:20:48.283 INFO:teuthology.task.kernel:Running kernel on smithi175: 5.14.0-503.el9.x86_64 2024-09-11T03:20:48.283 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum install -y kernel 2024-09-11T03:20:50.653 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-09-11T03:20:51.301 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - BaseOS 6.1 MB/s | 8.2 MB 00:01 2024-09-11T03:20:51.570 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.2 MB 00:01 2024-09-11T03:20:52.722 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-11T03:20:53.826 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - AppStream 15 MB/s | 20 MB 00:01 2024-09-11T03:20:55.342 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - AppStream 7.7 MB/s | 20 MB 00:02 2024-09-11T03:20:58.062 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.5 MB 00:01 2024-09-11T03:20:58.568 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 6.5 MB 00:00 2024-09-11T03:20:59.621 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 19 kB 00:00 2024-09-11T03:20:59.692 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-11T03:20:59.994 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 19 kB 00:00 2024-09-11T03:21:00.409 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-11T03:21:01.165 INFO:teuthology.orchestra.run.smithi175.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-11T03:21:01.351 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 19 kB 00:00 2024-09-11T03:21:02.480 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:01 2024-09-11T03:21:05.847 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-09-11T03:21:06.587 INFO:teuthology.orchestra.run.smithi175.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-09-11T03:21:07.837 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T03:21:07.837 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T03:21:07.910 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-09-11T03:21:07.921 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:21:07.931 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:21:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-09-11T03:21:07.931 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:21:07.931 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-11T03:21:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T03:21:07.931 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-11T03:21:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T03:21:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T03:21:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T03:21:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:21:07.932 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-11T03:21:07.932 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:21:07.932 INFO:teuthology.orchestra.run.smithi121.stdout:Install 4 Packages 2024-09-11T03:21:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:21:07.933 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 79 M 2024-09-11T03:21:07.933 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 126 M 2024-09-11T03:21:07.933 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-11T03:21:08.265 INFO:teuthology.orchestra.run.smithi121.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 79 kB/s | 23 kB 00:00 2024-09-11T03:21:08.593 INFO:teuthology.orchestra.run.smithi175.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T03:21:08.593 INFO:teuthology.orchestra.run.smithi175.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T03:21:08.694 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:21:08.702 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:21:08.702 INFO:teuthology.orchestra.run.smithi175.stdout: Package Architecture Version Repository Size 2024-09-11T03:21:08.702 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:21:08.702 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-11T03:21:08.702 INFO:teuthology.orchestra.run.smithi175.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T03:21:08.703 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-09-11T03:21:08.703 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T03:21:08.703 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T03:21:08.703 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T03:21:08.703 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:21:08.703 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-11T03:21:08.703 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:21:08.703 INFO:teuthology.orchestra.run.smithi175.stdout:Install 4 Packages 2024-09-11T03:21:08.703 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:21:08.704 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 79 M 2024-09-11T03:21:08.704 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 126 M 2024-09-11T03:21:08.704 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-11T03:21:08.830 INFO:teuthology.orchestra.run.smithi175.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 284 kB/s | 23 kB 00:00 2024-09-11T03:21:09.908 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T03:21:09.908 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T03:21:10.022 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:21:10.029 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:21:10.029 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-09-11T03:21:10.029 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:21:10.029 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-11T03:21:10.029 INFO:teuthology.orchestra.run.smithi084.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T03:21:10.029 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-09-11T03:21:10.029 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T03:21:10.029 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T03:21:10.030 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T03:21:10.030 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:21:10.030 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-11T03:21:10.030 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:21:10.030 INFO:teuthology.orchestra.run.smithi084.stdout:Install 4 Packages 2024-09-11T03:21:10.030 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:21:10.031 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 79 M 2024-09-11T03:21:10.031 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 126 M 2024-09-11T03:21:10.031 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-11T03:21:10.198 INFO:teuthology.orchestra.run.smithi121.stdout:(2/4): kernel-modules-core-5.14.0-505.el9.x86_6 15 MB/s | 29 MB 00:01 2024-09-11T03:21:10.436 INFO:teuthology.orchestra.run.smithi084.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 78 kB/s | 23 kB 00:00 2024-09-11T03:21:11.338 INFO:teuthology.orchestra.run.smithi175.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 6.0 MB/s | 16 MB 00:02 2024-09-11T03:21:15.029 INFO:teuthology.orchestra.run.smithi175.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 5.5 MB/s | 35 MB 00:06 2024-09-11T03:21:15.402 INFO:teuthology.orchestra.run.smithi084.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 3.0 MB/s | 16 MB 00:05 2024-09-11T03:21:16.264 INFO:teuthology.orchestra.run.smithi121.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 1.9 MB/s | 16 MB 00:08 2024-09-11T03:21:16.436 INFO:teuthology.orchestra.run.smithi084.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 4.8 MB/s | 29 MB 00:05 2024-09-11T03:21:16.973 INFO:teuthology.orchestra.run.smithi121.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 3.8 MB/s | 35 MB 00:08 2024-09-11T03:21:16.973 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:21:16.973 INFO:teuthology.orchestra.run.smithi121.stdout:Total 8.7 MB/s | 79 MB 00:09 2024-09-11T03:21:17.038 INFO:teuthology.orchestra.run.smithi175.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 3.5 MB/s | 29 MB 00:08 2024-09-11T03:21:17.038 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:21:17.038 INFO:teuthology.orchestra.run.smithi175.stdout:Total 9.4 MB/s | 79 MB 00:08 2024-09-11T03:21:17.553 INFO:teuthology.orchestra.run.smithi084.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 4.7 MB/s | 35 MB 00:07 2024-09-11T03:21:17.553 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:21:17.553 INFO:teuthology.orchestra.run.smithi084.stdout:Total 10 MB/s | 79 MB 00:07 2024-09-11T03:21:17.658 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-11T03:21:17.716 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-11T03:21:18.004 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-11T03:21:18.005 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-11T03:21:18.102 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-11T03:21:18.102 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-11T03:21:18.237 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-11T03:21:18.590 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-11T03:21:18.590 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-11T03:21:18.971 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-11T03:21:18.971 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-11T03:21:19.127 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-11T03:21:19.127 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-11T03:21:19.565 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-11T03:21:19.566 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-11T03:21:21.403 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-11T03:21:21.585 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-11T03:21:22.008 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-11T03:21:22.068 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T03:21:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T03:21:22.302 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T03:21:22.379 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T03:21:22.736 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T03:21:22.821 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T03:21:24.941 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T03:21:25.069 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T03:21:25.174 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T03:21:25.306 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T03:21:25.621 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T03:21:25.748 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T03:21:29.893 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T03:21:29.927 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:21:30.134 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T03:21:30.173 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:21:30.632 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T03:21:30.756 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:21:34.775 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:21:35.025 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:21:35.583 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:22:12.975 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:22:13.014 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:22:13.095 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:22:13.134 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:22:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:22:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T03:22:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T03:22:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T03:22:13.513 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:22:13.514 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T03:22:13.514 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T03:22:13.514 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T03:22:13.619 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:22:13.658 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:22:13.932 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:22:13.932 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T03:22:13.932 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T03:22:13.932 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T03:22:14.239 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:22:14.239 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:22:14.239 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-11T03:22:14.239 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T03:22:14.239 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T03:22:14.239 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T03:22:14.239 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T03:22:14.239 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:22:14.240 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:22:14.364 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:22:14.364 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:22:14.365 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-11T03:22:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T03:22:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T03:22:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T03:22:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T03:22:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:22:14.365 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:22:14.623 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T03:22:14.706 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T03:22:14.706 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T03:22:14.706 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T03:22:14.706 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T03:22:14.706 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T03:22:14.707 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-11T03:22:14.707 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T03:22:14.707 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T03:22:14.732 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T03:22:14.765 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T03:22:14.766 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T03:22:14.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -qi grub2-tools 2024-09-11T03:22:14.789 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T03:22:14.789 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:22:14.790 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-11T03:22:14.790 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T03:22:14.790 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T03:22:14.790 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T03:22:14.790 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T03:22:14.790 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:22:14.790 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:22:14.804 INFO:teuthology.orchestra.run.smithi121.stdout:Name : grub2-tools 2024-09-11T03:22:14.805 INFO:teuthology.orchestra.run.smithi121.stdout:Epoch : 1 2024-09-11T03:22:14.805 INFO:teuthology.orchestra.run.smithi121.stdout:Version : 2.06 2024-09-11T03:22:14.805 INFO:teuthology.orchestra.run.smithi121.stdout:Release : 82.el9 2024-09-11T03:22:14.805 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: x86_64 2024-09-11T03:22:14.805 INFO:teuthology.orchestra.run.smithi121.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T03:22:14.805 INFO:teuthology.orchestra.run.smithi121.stdout:Group : Unspecified 2024-09-11T03:22:14.806 INFO:teuthology.orchestra.run.smithi121.stdout:Size : 8274670 2024-09-11T03:22:14.806 INFO:teuthology.orchestra.run.smithi121.stdout:License : GPLv3+ 2024-09-11T03:22:14.806 INFO:teuthology.orchestra.run.smithi121.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T03:22:14.806 INFO:teuthology.orchestra.run.smithi121.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T03:22:14.806 INFO:teuthology.orchestra.run.smithi121.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T03:22:14.806 INFO:teuthology.orchestra.run.smithi121.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T03:22:14.806 INFO:teuthology.orchestra.run.smithi121.stdout:Packager : builder@centos.org 2024-09-11T03:22:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:Vendor : CentOS 2024-09-11T03:22:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T03:22:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:Summary : Support tools for GRUB. 2024-09-11T03:22:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:Description : 2024-09-11T03:22:14.807 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:22:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T03:22:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T03:22:14.808 INFO:teuthology.orchestra.run.smithi121.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T03:22:14.808 INFO:teuthology.orchestra.run.smithi121.stdout:hardware devices. 2024-09-11T03:22:14.808 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:22:14.808 INFO:teuthology.orchestra.run.smithi121.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T03:22:14.809 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T03:22:14.810 INFO:teuthology.task.kernel:Updating grub on smithi121 to boot 5.14.0-505.el9.x86_64 2024-09-11T03:22:14.810 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T03:22:14.823 INFO:teuthology.orchestra.run.smithi175.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T03:22:14.823 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T03:22:14.823 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T03:22:14.823 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T03:22:14.824 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T03:22:14.824 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-11T03:22:14.824 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi175.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T03:22:14.824 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T03:22:14.886 INFO:teuthology.orchestra.run.smithi175.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T03:22:14.887 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T03:22:14.887 DEBUG:teuthology.orchestra.run.smithi175:> sudo rpm -qi grub2-tools 2024-09-11T03:22:14.941 INFO:teuthology.orchestra.run.smithi175.stdout:Name : grub2-tools 2024-09-11T03:22:14.941 INFO:teuthology.orchestra.run.smithi175.stdout:Epoch : 1 2024-09-11T03:22:14.941 INFO:teuthology.orchestra.run.smithi175.stdout:Version : 2.06 2024-09-11T03:22:14.941 INFO:teuthology.orchestra.run.smithi175.stdout:Release : 82.el9 2024-09-11T03:22:14.941 INFO:teuthology.orchestra.run.smithi175.stdout:Architecture: x86_64 2024-09-11T03:22:14.941 INFO:teuthology.orchestra.run.smithi175.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T03:22:14.941 INFO:teuthology.orchestra.run.smithi175.stdout:Group : Unspecified 2024-09-11T03:22:14.941 INFO:teuthology.orchestra.run.smithi175.stdout:Size : 8274670 2024-09-11T03:22:14.941 INFO:teuthology.orchestra.run.smithi175.stdout:License : GPLv3+ 2024-09-11T03:22:14.942 INFO:teuthology.orchestra.run.smithi175.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T03:22:14.942 INFO:teuthology.orchestra.run.smithi175.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T03:22:14.942 INFO:teuthology.orchestra.run.smithi175.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T03:22:14.942 INFO:teuthology.orchestra.run.smithi175.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T03:22:14.942 INFO:teuthology.orchestra.run.smithi175.stdout:Packager : builder@centos.org 2024-09-11T03:22:14.942 INFO:teuthology.orchestra.run.smithi175.stdout:Vendor : CentOS 2024-09-11T03:22:14.942 INFO:teuthology.orchestra.run.smithi175.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T03:22:14.942 INFO:teuthology.orchestra.run.smithi175.stdout:Summary : Support tools for GRUB. 2024-09-11T03:22:14.942 INFO:teuthology.orchestra.run.smithi175.stdout:Description : 2024-09-11T03:22:14.942 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:22:14.942 INFO:teuthology.orchestra.run.smithi175.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T03:22:14.942 INFO:teuthology.orchestra.run.smithi175.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T03:22:14.942 INFO:teuthology.orchestra.run.smithi175.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T03:22:14.943 INFO:teuthology.orchestra.run.smithi175.stdout:hardware devices. 2024-09-11T03:22:14.943 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:22:14.943 INFO:teuthology.orchestra.run.smithi175.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T03:22:14.943 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T03:22:14.944 INFO:teuthology.task.kernel:Updating grub on smithi175 to boot 5.14.0-505.el9.x86_64 2024-09-11T03:22:14.944 DEBUG:teuthology.orchestra.run.smithi175:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T03:22:15.137 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T03:22:15.229 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T03:22:15.230 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T03:22:15.232 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T03:22:15.232 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T03:22:15.232 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T03:22:15.232 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-11T03:22:15.232 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi084.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T03:22:15.232 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T03:22:15.290 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T03:22:15.290 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T03:22:15.291 DEBUG:teuthology.orchestra.run.smithi084:> sudo rpm -qi grub2-tools 2024-09-11T03:22:15.332 INFO:teuthology.orchestra.run.smithi084.stdout:Name : grub2-tools 2024-09-11T03:22:15.332 INFO:teuthology.orchestra.run.smithi084.stdout:Epoch : 1 2024-09-11T03:22:15.332 INFO:teuthology.orchestra.run.smithi084.stdout:Version : 2.06 2024-09-11T03:22:15.332 INFO:teuthology.orchestra.run.smithi084.stdout:Release : 82.el9 2024-09-11T03:22:15.332 INFO:teuthology.orchestra.run.smithi084.stdout:Architecture: x86_64 2024-09-11T03:22:15.332 INFO:teuthology.orchestra.run.smithi084.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T03:22:15.332 INFO:teuthology.orchestra.run.smithi084.stdout:Group : Unspecified 2024-09-11T03:22:15.332 INFO:teuthology.orchestra.run.smithi084.stdout:Size : 8274670 2024-09-11T03:22:15.333 INFO:teuthology.orchestra.run.smithi084.stdout:License : GPLv3+ 2024-09-11T03:22:15.333 INFO:teuthology.orchestra.run.smithi084.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T03:22:15.333 INFO:teuthology.orchestra.run.smithi084.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T03:22:15.333 INFO:teuthology.orchestra.run.smithi084.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T03:22:15.333 INFO:teuthology.orchestra.run.smithi084.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T03:22:15.333 INFO:teuthology.orchestra.run.smithi084.stdout:Packager : builder@centos.org 2024-09-11T03:22:15.333 INFO:teuthology.orchestra.run.smithi084.stdout:Vendor : CentOS 2024-09-11T03:22:15.333 INFO:teuthology.orchestra.run.smithi084.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T03:22:15.333 INFO:teuthology.orchestra.run.smithi084.stdout:Summary : Support tools for GRUB. 2024-09-11T03:22:15.333 INFO:teuthology.orchestra.run.smithi084.stdout:Description : 2024-09-11T03:22:15.333 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:22:15.333 INFO:teuthology.orchestra.run.smithi084.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T03:22:15.333 INFO:teuthology.orchestra.run.smithi084.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T03:22:15.334 INFO:teuthology.orchestra.run.smithi084.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T03:22:15.334 INFO:teuthology.orchestra.run.smithi084.stdout:hardware devices. 2024-09-11T03:22:15.334 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:22:15.334 INFO:teuthology.orchestra.run.smithi084.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T03:22:15.335 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T03:22:15.335 INFO:teuthology.task.kernel:Updating grub on smithi084 to boot 5.14.0-505.el9.x86_64 2024-09-11T03:22:15.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T03:22:15.399 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2024-09-11T03:22:15.610 INFO:teuthology.orchestra.run.smithi175.stderr:Generating grub configuration file ... 2024-09-11T03:22:15.940 INFO:teuthology.orchestra.run.smithi084.stderr:Generating grub configuration file ... 2024-09-11T03:22:16.651 INFO:teuthology.orchestra.run.smithi121.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T03:22:16.674 INFO:teuthology.orchestra.run.smithi121.stderr:done 2024-09-11T03:22:16.676 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-09-11T03:22:16.711 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.fFGAmDrfN5 2024-09-11T03:22:16.712 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fFGAmDrfN5 2024-09-11T03:22:16.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.fFGAmDrfN5 2024-09-11T03:22:16.919 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.fFGAmDrfN5 is 6KB 2024-09-11T03:22:16.939 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.fFGAmDrfN5 2024-09-11T03:22:16.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T03:22:16.980 INFO:teuthology.orchestra.run.smithi175.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T03:22:17.016 INFO:teuthology.orchestra.run.smithi121.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T03:22:17.017 INFO:teuthology.orchestra.run.smithi121.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T03:22:17.017 INFO:teuthology.orchestra.run.smithi121.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T03:22:17.017 INFO:teuthology.orchestra.run.smithi121.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T03:22:17.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T03:22:17.020 INFO:teuthology.orchestra.run.smithi175.stderr:done 2024-09-11T03:22:17.022 DEBUG:teuthology.orchestra.run.smithi175:> mktemp 2024-09-11T03:22:17.039 INFO:teuthology.orchestra.run.smithi175.stdout:/tmp/tmp.xbIt1zkimw 2024-09-11T03:22:17.040 DEBUG:teuthology.orchestra.run.smithi175:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xbIt1zkimw 2024-09-11T03:22:17.106 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0666 /tmp/tmp.xbIt1zkimw 2024-09-11T03:22:17.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2024-09-11T03:22:17.240 DEBUG:teuthology.orchestra.remote:smithi175:/tmp/tmp.xbIt1zkimw is 6KB 2024-09-11T03:22:17.288 DEBUG:teuthology.orchestra.run.smithi175:> rm -fr /tmp/tmp.xbIt1zkimw 2024-09-11T03:22:17.298 INFO:teuthology.orchestra.run.smithi084.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T03:22:17.302 DEBUG:teuthology.orchestra.run.smithi175:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T03:22:17.336 INFO:teuthology.orchestra.run.smithi084.stderr:done 2024-09-11T03:22:17.338 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-09-11T03:22:17.353 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.wB10o1DRDu 2024-09-11T03:22:17.353 DEBUG:teuthology.orchestra.run.smithi084:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wB10o1DRDu 2024-09-11T03:22:17.365 INFO:teuthology.orchestra.run.smithi175.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T03:22:17.365 INFO:teuthology.orchestra.run.smithi175.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T03:22:17.365 INFO:teuthology.orchestra.run.smithi175.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T03:22:17.365 INFO:teuthology.orchestra.run.smithi175.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T03:22:17.366 DEBUG:teuthology.orchestra.run.smithi175:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T03:22:17.420 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.wB10o1DRDu 2024-09-11T03:22:17.532 DEBUG:teuthology.orchestra.run.smithi175:> sudo shutdown -r now 2024-09-11T03:22:17.556 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.wB10o1DRDu is 6KB 2024-09-11T03:22:17.604 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.wB10o1DRDu 2024-09-11T03:22:17.619 DEBUG:teuthology.orchestra.run.smithi084:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T03:22:17.687 INFO:teuthology.orchestra.run.smithi084.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T03:22:17.687 INFO:teuthology.orchestra.run.smithi084.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T03:22:17.687 INFO:teuthology.orchestra.run.smithi084.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T03:22:17.687 INFO:teuthology.orchestra.run.smithi084.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T03:22:17.688 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T03:22:17.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo shutdown -r now 2024-09-11T03:22:47.204 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-11T03:22:47.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-11T03:22:47.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T03:22:47.535 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-11T03:22:47.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-09-11T03:22:47.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T03:22:47.849 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-11T03:22:47.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-09-11T03:22:47.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T03:23:05.661 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-09-11T03:23:05.917 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.175 2024-09-11T03:23:06.173 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-09-11T03:23:14.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-11T03:23:14.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T03:23:14.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-09-11T03:23:14.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T03:23:15.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-09-11T03:23:15.175 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T03:23:17.761 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-09-11T03:23:17.981 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.175 2024-09-11T03:23:18.237 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-09-11T03:23:29.773 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-11T03:23:29.774 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T03:23:29.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-09-11T03:23:29.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T03:23:30.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-09-11T03:23:30.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T03:23:48.413 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.175 2024-09-11T03:24:01.748 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-09-11T03:24:02.069 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-09-11T03:24:02.317 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-11T03:24:02.317 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-11T03:24:02.317 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-09-11T03:24:02.360 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-505.el9.x86_64 2024-09-11T03:24:02.361 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T03:24:02.361 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T03:24:02.361 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T03:24:02.432 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-09-11T03:24:02.432 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-11T03:24:02.432 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-09-11T03:24:02.447 INFO:teuthology.orchestra.run.smithi084.stdout:5.14.0-505.el9.x86_64 2024-09-11T03:24:02.447 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T03:24:02.447 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T03:24:02.447 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T03:24:03.361 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-11T03:24:03.362 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T03:24:03.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-09-11T03:24:03.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T03:24:03.425 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2024-09-11T03:24:03.448 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-11T03:24:03.448 DEBUG:teuthology.orchestra.run.smithi084:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T03:24:03.534 INFO:teuthology.orchestra.run.smithi084.stdout:ttyS1 2024-09-11T03:24:03.561 DEBUG:teuthology.parallel:result is None 2024-09-11T03:24:03.622 DEBUG:teuthology.parallel:result is None 2024-09-11T03:24:03.835 DEBUG:teuthology.orchestra.run.smithi175:> true 2024-09-11T03:24:04.170 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-09-11T03:24:04.170 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-505.el9.x86_64"... 2024-09-11T03:24:04.171 DEBUG:teuthology.orchestra.run.smithi175:> uname -r 2024-09-11T03:24:04.186 INFO:teuthology.orchestra.run.smithi175.stdout:5.14.0-505.el9.x86_64 2024-09-11T03:24:04.186 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T03:24:04.186 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T03:24:04.186 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T03:24:05.186 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-11T03:24:05.187 DEBUG:teuthology.orchestra.run.smithi175:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T03:24:05.296 INFO:teuthology.orchestra.run.smithi175.stdout:ttyS1 2024-09-11T03:24:05.330 DEBUG:teuthology.parallel:result is None 2024-09-11T03:24:05.330 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T03:24:05.341 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T03:24:05.341 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T03:24:05.344 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T03:24:05.346 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T03:24:05.391 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T03:24:05.398 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T03:24:05.415 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T03:24:05.415 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T03:24:05.418 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T03:24:05.420 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T03:24:05.476 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T03:24:05.482 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T03:24:05.482 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T03:24:05.485 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T03:24:05.487 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T03:24:05.515 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T03:24:05.524 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T03:24:05.526 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T03:24:05.535 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T03:24:05.560 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T03:24:05.578 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T03:24:05.580 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T03:24:05.587 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T03:24:05.587 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T03:24:05.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T03:24:05.592 DEBUG:teuthology.orchestra.run.smithi175:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T03:24:05.643 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T03:24:05.652 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T03:24:05.652 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T03:24:05.654 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T03:24:05.659 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T03:24:05.694 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T03:24:05.804 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T03:24:05.845 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T03:24:05.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T03:24:05.967 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-11T03:24:05.967 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T03:24:06.033 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T03:24:06.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T03:24:06.132 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T03:24:06.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T03:24:06.261 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-11T03:24:06.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T03:24:06.329 DEBUG:teuthology.orchestra.run.smithi175:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T03:24:06.378 DEBUG:teuthology.orchestra.run.smithi175:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T03:24:06.426 DEBUG:teuthology.orchestra.run.smithi175:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T03:24:06.513 DEBUG:teuthology.orchestra.run.smithi175:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T03:24:06.547 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-11T03:24:06.547 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T03:24:06.613 DEBUG:teuthology.orchestra.run.smithi084:> sudo service rsyslog restart 2024-09-11T03:24:06.615 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-09-11T03:24:06.617 DEBUG:teuthology.orchestra.run.smithi175:> sudo service rsyslog restart 2024-09-11T03:24:06.657 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T03:24:06.661 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T03:24:06.696 INFO:teuthology.orchestra.run.smithi175.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T03:24:07.017 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T03:24:07.025 INFO:teuthology.task.internal:Starting timer... 2024-09-11T03:24:07.025 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T03:24:07.038 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T03:24:07.051 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-11T03:24:07.051 DEBUG:teuthology.orchestra.run.smithi084:> sudo service auditd rotate 2024-09-11T03:24:07.108 INFO:teuthology.orchestra.run.smithi084.stdout:Rotating logs: 2024-09-11T03:24:07.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2024-09-11T03:24:07.162 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2024-09-11T03:24:07.164 DEBUG:teuthology.orchestra.run.smithi175:> sudo service auditd rotate 2024-09-11T03:24:07.221 INFO:teuthology.orchestra.run.smithi175.stdout:Rotating logs: 2024-09-11T03:24:07.223 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T03:24:07.223 DEBUG:teuthology.orchestra.run.smithi084:> /usr/sbin/getenforce 2024-09-11T03:24:07.270 INFO:teuthology.orchestra.run.smithi084.stdout:Permissive 2024-09-11T03:24:07.270 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2024-09-11T03:24:07.293 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2024-09-11T03:24:07.293 DEBUG:teuthology.orchestra.run.smithi175:> /usr/sbin/getenforce 2024-09-11T03:24:07.340 INFO:teuthology.orchestra.run.smithi175.stdout:Permissive 2024-09-11T03:24:07.340 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi084.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi121.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi175.front.sepia.ceph.com': 'permissive'} 2024-09-11T03:24:07.341 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-11T03:24:07.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:24:07.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-11T03:24:07.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:24:07.395 DEBUG:teuthology.orchestra.run.smithi175:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-11T03:24:07.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:24:07.420 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T03:24:07.420 DEBUG:teuthology.orchestra.run.smithi084:> sudo /usr/sbin/setenforce permissive 2024-09-11T03:24:07.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2024-09-11T03:24:07.494 DEBUG:teuthology.orchestra.run.smithi175:> sudo /usr/sbin/setenforce permissive 2024-09-11T03:24:07.531 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T03:24:07.540 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-11T03:24:07.544 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-11T03:24:07.695 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T03:24:07.706 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T03:24:07.707 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi084.front.sepia.ceph.com,smithi121.front.sepia.ceph.com,smithi175.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T03:27:58.036 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi084.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com'), Remote(name='ubuntu@smithi175.front.sepia.ceph.com')] 2024-09-11T03:27:58.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-09-11T03:27:58.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T03:27:58.124 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-09-11T03:27:58.204 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-09-11T03:27:58.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-11T03:27:58.205 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T03:27:58.304 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-09-11T03:27:58.389 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-11T03:27:58.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-09-11T03:27:58.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T03:27:58.515 DEBUG:teuthology.orchestra.run.smithi175:> true 2024-09-11T03:27:58.593 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-09-11T03:27:58.594 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T03:27:58.608 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T03:27:58.608 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T03:27:58.609 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T03:27:58.611 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T03:27:58.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T03:27:58.614 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T03:27:58.614 DEBUG:teuthology.orchestra.run.smithi175:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T03:27:58.642 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T03:27:58.645 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T03:27:58.657 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T03:27:58.660 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T03:27:58.666 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T03:27:58.681 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T03:27:58.683 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: ntpd: command not found 2024-09-11T03:27:58.686 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2024-09-11T03:27:58.695 INFO:teuthology.orchestra.run.smithi084.stdout:506 Cannot talk to daemon 2024-09-11T03:27:58.698 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2024-09-11T03:27:58.707 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: ntpd: command not found 2024-09-11T03:27:58.710 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T03:27:58.712 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T03:27:58.718 INFO:teuthology.orchestra.run.smithi175.stdout:506 Cannot talk to daemon 2024-09-11T03:27:58.724 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T03:27:58.726 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T03:27:58.733 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T03:27:58.748 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T03:27:58.783 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-09-11T03:27:58.786 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T03:27:58.786 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-09-11T03:27:58.786 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T03:27:58.786 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T03:27:58.786 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T03:27:58.786 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T03:27:58.790 INFO:teuthology.orchestra.run.smithi084.stderr:bash: line 1: ntpq: command not found 2024-09-11T03:27:58.794 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T03:27:58.794 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-09-11T03:27:58.794 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T03:27:58.794 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T03:27:58.794 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T03:27:58.794 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T03:27:58.801 INFO:teuthology.orchestra.run.smithi175.stderr:bash: line 1: ntpq: command not found 2024-09-11T03:27:58.804 INFO:teuthology.orchestra.run.smithi175.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T03:27:58.804 INFO:teuthology.orchestra.run.smithi175.stdout:=============================================================================== 2024-09-11T03:27:58.804 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T03:27:58.804 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T03:27:58.805 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T03:27:58.805 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T03:27:58.805 INFO:teuthology.run_tasks:Running task install... 2024-09-11T03:27:58.813 DEBUG:teuthology.task.install:project ceph 2024-09-11T03:27:58.813 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}} 2024-09-11T03:27:58.813 DEBUG:teuthology.task.install:config {'branch': 'reef', 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'} 2024-09-11T03:27:58.814 INFO:teuthology.task.install:Using flavor: default 2024-09-11T03:27:58.819 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T03:27:58.819 INFO:teuthology.task.install:extra packages: [] 2024-09-11T03:27:58.819 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'tag': None, 'wait_for_package': False} 2024-09-11T03:27:58.820 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T03:27:58.820 INFO:teuthology.packaging:ref: None 2024-09-11T03:27:58.820 INFO:teuthology.packaging:tag: None 2024-09-11T03:27:58.820 INFO:teuthology.packaging:branch: reef 2024-09-11T03:27:58.820 INFO:teuthology.packaging:sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T03:27:58.820 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-11T03:27:58.822 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'tag': None, 'wait_for_package': False} 2024-09-11T03:27:58.823 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T03:27:58.823 INFO:teuthology.packaging:ref: None 2024-09-11T03:27:58.823 INFO:teuthology.packaging:tag: None 2024-09-11T03:27:58.823 INFO:teuthology.packaging:branch: reef 2024-09-11T03:27:58.823 INFO:teuthology.packaging:sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T03:27:58.823 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-11T03:27:58.824 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'tag': None, 'wait_for_package': False} 2024-09-11T03:27:58.824 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T03:27:58.824 INFO:teuthology.packaging:ref: None 2024-09-11T03:27:58.824 INFO:teuthology.packaging:tag: None 2024-09-11T03:27:58.824 INFO:teuthology.packaging:branch: reef 2024-09-11T03:27:58.824 INFO:teuthology.packaging:sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T03:27:58.824 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-11T03:27:59.072 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/reef/7a853b1706d076d58e0da7a5d3b9b9eb36276880/centos/9/flavors/default/ 2024-09-11T03:27:59.072 INFO:teuthology.task.install.rpm:Package version is 18.2.4-837.g7a853b17 2024-09-11T03:27:59.117 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/reef/7a853b1706d076d58e0da7a5d3b9b9eb36276880/centos/9/flavors/default/ 2024-09-11T03:27:59.117 INFO:teuthology.task.install.rpm:Package version is 18.2.4-837.g7a853b17 2024-09-11T03:27:59.119 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/reef/7a853b1706d076d58e0da7a5d3b9b9eb36276880/centos/9/flavors/default/ 2024-09-11T03:27:59.119 INFO:teuthology.task.install.rpm:Package version is 18.2.4-837.g7a853b17 2024-09-11T03:27:59.204 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/reef/7a853b1706d076d58e0da7a5d3b9b9eb36276880/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/7a853b1706d076d58e0da7a5d3b9b9eb36276880/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/7a853b1706d076d58e0da7a5d3b9b9eb36276880/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T03:27:59.205 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-11T03:27:59.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T03:27:59.232 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-11T03:27:59.232 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T03:27:59.233 INFO:teuthology.packaging:ref: None 2024-09-11T03:27:59.233 INFO:teuthology.packaging:tag: None 2024-09-11T03:27:59.233 INFO:teuthology.packaging:branch: reef 2024-09-11T03:27:59.233 INFO:teuthology.packaging:sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T03:27:59.233 DEBUG:teuthology.orchestra.run.smithi084:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T03:27:59.238 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/reef/7a853b1706d076d58e0da7a5d3b9b9eb36276880/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/7a853b1706d076d58e0da7a5d3b9b9eb36276880/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/7a853b1706d076d58e0da7a5d3b9b9eb36276880/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T03:27:59.238 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-11T03:27:59.238 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T03:27:59.246 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/reef/7a853b1706d076d58e0da7a5d3b9b9eb36276880/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/7a853b1706d076d58e0da7a5d3b9b9eb36276880/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/7a853b1706d076d58e0da7a5d3b9b9eb36276880/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T03:27:59.246 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-11T03:27:59.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T03:27:59.265 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-11T03:27:59.265 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T03:27:59.265 INFO:teuthology.packaging:ref: None 2024-09-11T03:27:59.265 INFO:teuthology.packaging:tag: None 2024-09-11T03:27:59.265 INFO:teuthology.packaging:branch: reef 2024-09-11T03:27:59.265 INFO:teuthology.packaging:sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T03:27:59.265 DEBUG:teuthology.orchestra.run.smithi175:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T03:27:59.273 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-11T03:27:59.274 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T03:27:59.274 INFO:teuthology.packaging:ref: None 2024-09-11T03:27:59.274 INFO:teuthology.packaging:tag: None 2024-09-11T03:27:59.274 INFO:teuthology.packaging:branch: reef 2024-09-11T03:27:59.274 INFO:teuthology.packaging:sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T03:27:59.274 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T03:27:59.300 DEBUG:teuthology.orchestra.run.smithi084:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T03:27:59.331 DEBUG:teuthology.orchestra.run.smithi175:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T03:27:59.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T03:27:59.375 DEBUG:teuthology.orchestra.run.smithi084:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T03:27:59.407 DEBUG:teuthology.orchestra.run.smithi175:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T03:27:59.415 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T03:27:59.438 INFO:teuthology.orchestra.run.smithi084.stdout:check_obsoletes = 1 2024-09-11T03:27:59.440 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum clean all 2024-09-11T03:27:59.471 INFO:teuthology.orchestra.run.smithi175.stdout:check_obsoletes = 1 2024-09-11T03:27:59.473 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum clean all 2024-09-11T03:27:59.479 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2024-09-11T03:27:59.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2024-09-11T03:27:59.673 INFO:teuthology.orchestra.run.smithi084.stdout:45 files removed 2024-09-11T03:27:59.700 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-radosgw 2024-09-11T03:27:59.703 INFO:teuthology.orchestra.run.smithi121.stdout:45 files removed 2024-09-11T03:27:59.713 INFO:teuthology.orchestra.run.smithi175.stdout:45 files removed 2024-09-11T03:27:59.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw 2024-09-11T03:27:59.739 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-radosgw 2024-09-11T03:28:00.270 INFO:teuthology.orchestra.run.smithi084.stdout:ceph packages for x86_64 273 kB/s | 77 kB 00:00 2024-09-11T03:28:00.276 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 293 kB/s | 77 kB 00:00 2024-09-11T03:28:00.296 INFO:teuthology.orchestra.run.smithi175.stdout:ceph packages for x86_64 285 kB/s | 77 kB 00:00 2024-09-11T03:28:00.569 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 46 kB/s | 11 kB 00:00 2024-09-11T03:28:00.570 INFO:teuthology.orchestra.run.smithi084.stdout:ceph noarch packages 47 kB/s | 11 kB 00:00 2024-09-11T03:28:00.571 INFO:teuthology.orchestra.run.smithi175.stdout:ceph noarch packages 49 kB/s | 11 kB 00:00 2024-09-11T03:28:00.830 INFO:teuthology.orchestra.run.smithi175.stdout:ceph source packages 8.4 kB/s | 1.9 kB 00:00 2024-09-11T03:28:00.844 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-11T03:28:00.846 INFO:teuthology.orchestra.run.smithi084.stdout:ceph source packages 7.8 kB/s | 1.9 kB 00:00 2024-09-11T03:28:01.414 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-09-11T03:28:01.471 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-11T03:28:02.486 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.2 MB 00:01 2024-09-11T03:28:03.115 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - AppStream 33 MB/s | 20 MB 00:00 2024-09-11T03:28:06.430 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - AppStream 5.3 MB/s | 20 MB 00:03 2024-09-11T03:28:07.537 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 5.1 MB/s | 20 MB 00:03 2024-09-11T03:28:08.674 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-11T03:28:10.150 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - Extras packages 118 kB/s | 19 kB 00:00 2024-09-11T03:28:10.980 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 6.5 MB 00:00 2024-09-11T03:28:11.488 INFO:teuthology.orchestra.run.smithi175.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-11T03:28:12.681 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 19 kB 00:00 2024-09-11T03:28:13.037 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.5 MB 00:01 2024-09-11T03:28:14.177 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-11T03:28:14.679 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 19 kB 00:00 2024-09-11T03:28:15.833 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-11T03:28:16.944 INFO:teuthology.orchestra.run.smithi175.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-09-11T03:28:19.318 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:28:19.337 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:28:19.337 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-09-11T03:28:19.337 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:28:19.337 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-11T03:28:19.337 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-radosgw x86_64 2:18.2.4-837.g7a853b17.el9 ceph 7.7 M 2024-09-11T03:28:19.337 INFO:teuthology.orchestra.run.smithi175.stdout:Upgrading: 2024-09-11T03:28:19.338 INFO:teuthology.orchestra.run.smithi175.stdout: librados2 x86_64 2:18.2.4-837.g7a853b17.el9 ceph 3.3 M 2024-09-11T03:28:19.338 INFO:teuthology.orchestra.run.smithi175.stdout: librbd1 x86_64 2:18.2.4-837.g7a853b17.el9 ceph 3.0 M 2024-09-11T03:28:19.338 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-09-11T03:28:19.338 INFO:teuthology.orchestra.run.smithi175.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-11T03:28:19.338 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-base x86_64 2:18.2.4-837.g7a853b17.el9 ceph 5.1 M 2024-09-11T03:28:19.338 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-common x86_64 2:18.2.4-837.g7a853b17.el9 ceph 18 M 2024-09-11T03:28:19.338 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-selinux x86_64 2:18.2.4-837.g7a853b17.el9 ceph 25 k 2024-09-11T03:28:19.338 INFO:teuthology.orchestra.run.smithi175.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-11T03:28:19.338 INFO:teuthology.orchestra.run.smithi175.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T03:28:19.338 INFO:teuthology.orchestra.run.smithi175.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-11T03:28:19.338 INFO:teuthology.orchestra.run.smithi175.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-11T03:28:19.338 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs2 x86_64 2:18.2.4-837.g7a853b17.el9 ceph 712 k 2024-09-11T03:28:19.339 INFO:teuthology.orchestra.run.smithi175.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T03:28:19.339 INFO:teuthology.orchestra.run.smithi175.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T03:28:19.339 INFO:teuthology.orchestra.run.smithi175.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T03:28:19.339 INFO:teuthology.orchestra.run.smithi175.stdout: libradosstriper1 x86_64 2:18.2.4-837.g7a853b17.el9 ceph 477 k 2024-09-11T03:28:19.339 INFO:teuthology.orchestra.run.smithi175.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T03:28:19.339 INFO:teuthology.orchestra.run.smithi175.stdout: librgw2 x86_64 2:18.2.4-837.g7a853b17.el9 ceph 4.5 M 2024-09-11T03:28:19.339 INFO:teuthology.orchestra.run.smithi175.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T03:28:19.339 INFO:teuthology.orchestra.run.smithi175.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T03:28:19.339 INFO:teuthology.orchestra.run.smithi175.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-11T03:28:19.339 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-argparse x86_64 2:18.2.4-837.g7a853b17.el9 ceph 45 k 2024-09-11T03:28:19.339 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-common x86_64 2:18.2.4-837.g7a853b17.el9 ceph 129 k 2024-09-11T03:28:19.339 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cephfs x86_64 2:18.2.4-837.g7a853b17.el9 ceph 161 k 2024-09-11T03:28:19.339 INFO:teuthology.orchestra.run.smithi175.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T03:28:19.340 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rados x86_64 2:18.2.4-837.g7a853b17.el9 ceph 319 k 2024-09-11T03:28:19.340 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rbd x86_64 2:18.2.4-837.g7a853b17.el9 ceph 300 k 2024-09-11T03:28:19.340 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rgw x86_64 2:18.2.4-837.g7a853b17.el9 ceph 99 k 2024-09-11T03:28:19.340 INFO:teuthology.orchestra.run.smithi175.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-11T03:28:19.340 INFO:teuthology.orchestra.run.smithi175.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T03:28:19.340 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:28:19.340 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-11T03:28:19.340 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:28:19.340 INFO:teuthology.orchestra.run.smithi175.stdout:Install 28 Packages 2024-09-11T03:28:19.340 INFO:teuthology.orchestra.run.smithi175.stdout:Upgrade 2 Packages 2024-09-11T03:28:19.340 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:28:19.341 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 53 M 2024-09-11T03:28:19.341 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-11T03:28:19.515 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-11T03:28:19.861 INFO:teuthology.orchestra.run.smithi175.stdout:(1/30): ceph-base-18.2.4-837.g7a853b17.el9.x86_ 12 MB/s | 5.1 MB 00:00 2024-09-11T03:28:19.912 INFO:teuthology.orchestra.run.smithi175.stdout:(2/30): ceph-selinux-18.2.4-837.g7a853b17.el9.x 504 kB/s | 25 kB 00:00 2024-09-11T03:28:20.012 INFO:teuthology.orchestra.run.smithi175.stdout:(3/30): libcephfs2-18.2.4-837.g7a853b17.el9.x86 7.0 MB/s | 712 kB 00:00 2024-09-11T03:28:20.104 INFO:teuthology.orchestra.run.smithi175.stdout:(4/30): ceph-radosgw-18.2.4-837.g7a853b17.el9.x 11 MB/s | 7.7 MB 00:00 2024-09-11T03:28:20.129 INFO:teuthology.orchestra.run.smithi175.stdout:(5/30): libradosstriper1-18.2.4-837.g7a853b17.e 4.0 MB/s | 477 kB 00:00 2024-09-11T03:28:20.188 INFO:teuthology.orchestra.run.smithi175.stdout:(6/30): python3-ceph-argparse-18.2.4-837.g7a853 770 kB/s | 45 kB 00:00 2024-09-11T03:28:20.279 INFO:teuthology.orchestra.run.smithi175.stdout:(7/30): python3-ceph-common-18.2.4-837.g7a853b1 1.4 MB/s | 129 kB 00:00 2024-09-11T03:28:20.346 INFO:teuthology.orchestra.run.smithi175.stdout:(8/30): librgw2-18.2.4-837.g7a853b17.el9.x86_64 18 MB/s | 4.5 MB 00:00 2024-09-11T03:28:20.372 INFO:teuthology.orchestra.run.smithi175.stdout:(9/30): python3-cephfs-18.2.4-837.g7a853b17.el9 1.7 MB/s | 161 kB 00:00 2024-09-11T03:28:20.397 INFO:teuthology.orchestra.run.smithi175.stdout:(10/30): python3-rados-18.2.4-837.g7a853b17.el9 6.2 MB/s | 319 kB 00:00 2024-09-11T03:28:20.423 INFO:teuthology.orchestra.run.smithi175.stdout:(11/30): python3-rbd-18.2.4-837.g7a853b17.el9.x 5.8 MB/s | 300 kB 00:00 2024-09-11T03:28:20.447 INFO:teuthology.orchestra.run.smithi175.stdout:(12/30): python3-rgw-18.2.4-837.g7a853b17.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-11T03:28:20.615 INFO:teuthology.orchestra.run.smithi175.stdout:(13/30): ceph-common-18.2.4-837.g7a853b17.el9.x 15 MB/s | 18 MB 00:01 2024-09-11T03:28:20.998 INFO:teuthology.orchestra.run.smithi175.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 186 kB/s | 107 kB 00:00 2024-09-11T03:28:21.023 INFO:teuthology.orchestra.run.smithi175.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 278 kB/s | 160 kB 00:00 2024-09-11T03:28:21.051 INFO:teuthology.orchestra.run.smithi175.stdout:(16/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 104 kB/s | 45 kB 00:00 2024-09-11T03:28:21.199 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-09-11T03:28:21.226 INFO:teuthology.orchestra.run.smithi175.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-09-11T03:28:21.243 INFO:teuthology.orchestra.run.smithi175.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 220 kB/s | 42 kB 00:00 2024-09-11T03:28:21.293 INFO:teuthology.orchestra.run.smithi175.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.2 MB/s | 662 kB 00:00 2024-09-11T03:28:21.319 INFO:teuthology.orchestra.run.smithi175.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 1.2 MB/s | 111 kB 00:00 2024-09-11T03:28:21.344 INFO:teuthology.orchestra.run.smithi175.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.4 MB/s | 26 kB 00:00 2024-09-11T03:28:21.369 INFO:teuthology.orchestra.run.smithi175.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.4 MB/s | 309 kB 00:00 2024-09-11T03:28:21.386 INFO:teuthology.orchestra.run.smithi175.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 1.2 MB/s | 49 kB 00:00 2024-09-11T03:28:21.411 INFO:teuthology.orchestra.run.smithi175.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-11T03:28:21.445 INFO:teuthology.orchestra.run.smithi175.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 5.7 MB/s | 191 kB 00:00 2024-09-11T03:28:21.487 INFO:teuthology.orchestra.run.smithi175.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-11T03:28:21.554 INFO:teuthology.orchestra.run.smithi175.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-11T03:28:21.687 INFO:teuthology.orchestra.run.smithi175.stdout:(28/30): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-11T03:28:21.755 INFO:teuthology.orchestra.run.smithi175.stdout:(29/30): librados2-18.2.4-837.g7a853b17.el9.x86 12 MB/s | 3.3 MB 00:00 2024-09-11T03:28:21.846 INFO:teuthology.orchestra.run.smithi175.stdout:(30/30): librbd1-18.2.4-837.g7a853b17.el9.x86_6 10 MB/s | 3.0 MB 00:00 2024-09-11T03:28:21.851 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:28:21.851 INFO:teuthology.orchestra.run.smithi175.stdout:Total 21 MB/s | 53 MB 00:02 2024-09-11T03:28:21.866 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:28:21.882 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:28:21.883 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-09-11T03:28:21.883 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:28:21.883 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-11T03:28:21.883 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw x86_64 2:18.2.4-837.g7a853b17.el9 ceph 7.7 M 2024-09-11T03:28:21.883 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrading: 2024-09-11T03:28:21.883 INFO:teuthology.orchestra.run.smithi084.stdout: librados2 x86_64 2:18.2.4-837.g7a853b17.el9 ceph 3.3 M 2024-09-11T03:28:21.883 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1 x86_64 2:18.2.4-837.g7a853b17.el9 ceph 3.0 M 2024-09-11T03:28:21.883 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-09-11T03:28:21.883 INFO:teuthology.orchestra.run.smithi084.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-11T03:28:21.883 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base x86_64 2:18.2.4-837.g7a853b17.el9 ceph 5.1 M 2024-09-11T03:28:21.883 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common x86_64 2:18.2.4-837.g7a853b17.el9 ceph 18 M 2024-09-11T03:28:21.883 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux x86_64 2:18.2.4-837.g7a853b17.el9 ceph 25 k 2024-09-11T03:28:21.884 INFO:teuthology.orchestra.run.smithi084.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-11T03:28:21.884 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T03:28:21.884 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-11T03:28:21.884 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-11T03:28:21.884 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2 x86_64 2:18.2.4-837.g7a853b17.el9 ceph 712 k 2024-09-11T03:28:21.884 INFO:teuthology.orchestra.run.smithi084.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T03:28:21.884 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T03:28:21.884 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T03:28:21.884 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1 x86_64 2:18.2.4-837.g7a853b17.el9 ceph 477 k 2024-09-11T03:28:21.884 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T03:28:21.884 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2 x86_64 2:18.2.4-837.g7a853b17.el9 ceph 4.5 M 2024-09-11T03:28:21.884 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T03:28:21.885 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T03:28:21.885 INFO:teuthology.orchestra.run.smithi084.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-11T03:28:21.885 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse x86_64 2:18.2.4-837.g7a853b17.el9 ceph 45 k 2024-09-11T03:28:21.885 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common x86_64 2:18.2.4-837.g7a853b17.el9 ceph 129 k 2024-09-11T03:28:21.885 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs x86_64 2:18.2.4-837.g7a853b17.el9 ceph 161 k 2024-09-11T03:28:21.885 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T03:28:21.885 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados x86_64 2:18.2.4-837.g7a853b17.el9 ceph 319 k 2024-09-11T03:28:21.885 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd x86_64 2:18.2.4-837.g7a853b17.el9 ceph 300 k 2024-09-11T03:28:21.885 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw x86_64 2:18.2.4-837.g7a853b17.el9 ceph 99 k 2024-09-11T03:28:21.885 INFO:teuthology.orchestra.run.smithi084.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-11T03:28:21.885 INFO:teuthology.orchestra.run.smithi084.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T03:28:21.885 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:28:21.886 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-11T03:28:21.886 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:28:21.886 INFO:teuthology.orchestra.run.smithi084.stdout:Install 28 Packages 2024-09-11T03:28:21.886 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrade 2 Packages 2024-09-11T03:28:21.886 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:28:21.886 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 53 M 2024-09-11T03:28:21.886 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-11T03:28:21.918 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-11T03:28:21.953 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-11T03:28:21.953 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-11T03:28:22.547 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-11T03:28:22.547 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-11T03:28:22.600 INFO:teuthology.orchestra.run.smithi084.stdout:(1/30): ceph-base-18.2.4-837.g7a853b17.el9.x86_ 12 MB/s | 5.1 MB 00:00 2024-09-11T03:28:22.658 INFO:teuthology.orchestra.run.smithi084.stdout:(2/30): ceph-selinux-18.2.4-837.g7a853b17.el9.x 432 kB/s | 25 kB 00:00 2024-09-11T03:28:22.733 INFO:teuthology.orchestra.run.smithi084.stdout:(3/30): libcephfs2-18.2.4-837.g7a853b17.el9.x86 9.3 MB/s | 712 kB 00:00 2024-09-11T03:28:22.851 INFO:teuthology.orchestra.run.smithi084.stdout:(4/30): ceph-radosgw-18.2.4-837.g7a853b17.el9.x 12 MB/s | 7.7 MB 00:00 2024-09-11T03:28:22.875 INFO:teuthology.orchestra.run.smithi084.stdout:(5/30): libradosstriper1-18.2.4-837.g7a853b17.e 3.3 MB/s | 477 kB 00:00 2024-09-11T03:28:22.942 INFO:teuthology.orchestra.run.smithi084.stdout:(6/30): python3-ceph-argparse-18.2.4-837.g7a853 675 kB/s | 45 kB 00:00 2024-09-11T03:28:23.018 INFO:teuthology.orchestra.run.smithi084.stdout:(7/30): python3-ceph-common-18.2.4-837.g7a853b1 1.7 MB/s | 129 kB 00:00 2024-09-11T03:28:23.084 INFO:teuthology.orchestra.run.smithi084.stdout:(8/30): librgw2-18.2.4-837.g7a853b17.el9.x86_64 19 MB/s | 4.5 MB 00:00 2024-09-11T03:28:23.110 INFO:teuthology.orchestra.run.smithi084.stdout:(9/30): python3-cephfs-18.2.4-837.g7a853b17.el9 1.7 MB/s | 161 kB 00:00 2024-09-11T03:28:23.143 INFO:teuthology.orchestra.run.smithi084.stdout:(10/30): python3-rados-18.2.4-837.g7a853b17.el9 5.3 MB/s | 319 kB 00:00 2024-09-11T03:28:23.168 INFO:teuthology.orchestra.run.smithi084.stdout:(11/30): python3-rbd-18.2.4-837.g7a853b17.el9.x 5.0 MB/s | 300 kB 00:00 2024-09-11T03:28:23.194 INFO:teuthology.orchestra.run.smithi084.stdout:(12/30): python3-rgw-18.2.4-837.g7a853b17.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-11T03:28:23.369 INFO:teuthology.orchestra.run.smithi084.stdout:(13/30): ceph-common-18.2.4-837.g7a853b17.el9.x 15 MB/s | 18 MB 00:01 2024-09-11T03:28:23.511 INFO:teuthology.orchestra.run.smithi084.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 320 kB/s | 45 kB 00:00 2024-09-11T03:28:23.524 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:28:23.536 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:28:23.536 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-11T03:28:23.536 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:28:23.536 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-11T03:28:23.536 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:18.2.4-837.g7a853b17.el9 ceph 7.7 M 2024-09-11T03:28:23.536 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2024-09-11T03:28:23.537 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:18.2.4-837.g7a853b17.el9 ceph 3.3 M 2024-09-11T03:28:23.537 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:18.2.4-837.g7a853b17.el9 ceph 3.0 M 2024-09-11T03:28:23.537 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-11T03:28:23.537 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-11T03:28:23.537 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:18.2.4-837.g7a853b17.el9 ceph 5.1 M 2024-09-11T03:28:23.537 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:18.2.4-837.g7a853b17.el9 ceph 18 M 2024-09-11T03:28:23.537 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:18.2.4-837.g7a853b17.el9 ceph 25 k 2024-09-11T03:28:23.537 INFO:teuthology.orchestra.run.smithi121.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-11T03:28:23.537 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T03:28:23.537 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-11T03:28:23.537 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-11T03:28:23.537 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:18.2.4-837.g7a853b17.el9 ceph 712 k 2024-09-11T03:28:23.538 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T03:28:23.538 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T03:28:23.538 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T03:28:23.538 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:18.2.4-837.g7a853b17.el9 ceph 477 k 2024-09-11T03:28:23.538 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T03:28:23.538 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:18.2.4-837.g7a853b17.el9 ceph 4.5 M 2024-09-11T03:28:23.538 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T03:28:23.538 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T03:28:23.538 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-11T03:28:23.538 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:18.2.4-837.g7a853b17.el9 ceph 45 k 2024-09-11T03:28:23.538 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:18.2.4-837.g7a853b17.el9 ceph 129 k 2024-09-11T03:28:23.538 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:18.2.4-837.g7a853b17.el9 ceph 161 k 2024-09-11T03:28:23.538 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T03:28:23.538 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:18.2.4-837.g7a853b17.el9 ceph 319 k 2024-09-11T03:28:23.539 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:18.2.4-837.g7a853b17.el9 ceph 300 k 2024-09-11T03:28:23.539 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:18.2.4-837.g7a853b17.el9 ceph 99 k 2024-09-11T03:28:23.539 INFO:teuthology.orchestra.run.smithi121.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-11T03:28:23.539 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T03:28:23.539 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:28:23.539 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-11T03:28:23.539 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:28:23.539 INFO:teuthology.orchestra.run.smithi121.stdout:Install 28 Packages 2024-09-11T03:28:23.539 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 2 Packages 2024-09-11T03:28:23.539 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:28:23.540 INFO:teuthology.orchestra.run.smithi084.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 291 kB/s | 107 kB 00:00 2024-09-11T03:28:23.540 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 53 M 2024-09-11T03:28:23.540 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-11T03:28:23.553 INFO:teuthology.orchestra.run.smithi084.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 446 kB/s | 160 kB 00:00 2024-09-11T03:28:23.604 INFO:teuthology.orchestra.run.smithi084.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 844 kB/s | 42 kB 00:00 2024-09-11T03:28:23.646 INFO:teuthology.orchestra.run.smithi084.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-11T03:28:23.679 INFO:teuthology.orchestra.run.smithi084.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 1.4 MB/s | 111 kB 00:00 2024-09-11T03:28:23.721 INFO:teuthology.orchestra.run.smithi084.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 4.0 MB/s | 309 kB 00:00 2024-09-11T03:28:23.738 INFO:teuthology.orchestra.run.smithi084.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.5 MB/s | 26 kB 00:00 2024-09-11T03:28:23.772 INFO:teuthology.orchestra.run.smithi084.stdout:(22/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.5 MB/s | 662 kB 00:00 2024-09-11T03:28:23.788 INFO:teuthology.orchestra.run.smithi084.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-09-11T03:28:23.838 INFO:teuthology.orchestra.run.smithi084.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-09-11T03:28:23.881 INFO:teuthology.orchestra.run.smithi084.stdout:(25/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.6 MB/s | 839 kB 00:00 2024-09-11T03:28:23.906 INFO:teuthology.orchestra.run.smithi084.stdout:(26/30): re2-20211101-20.el9.x86_64.rpm 2.8 MB/s | 191 kB 00:00 2024-09-11T03:28:23.941 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-11T03:28:23.973 INFO:teuthology.orchestra.run.smithi084.stdout:(27/30): libarrow-9.0.0-11.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-09-11T03:28:24.015 INFO:teuthology.orchestra.run.smithi084.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-09-11T03:28:24.040 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-11T03:28:24.068 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-11T03:28:24.126 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-11T03:28:24.173 INFO:teuthology.orchestra.run.smithi084.stdout:(29/30): librados2-18.2.4-837.g7a853b17.el9.x86 12 MB/s | 3.3 MB 00:00 2024-09-11T03:28:24.265 INFO:teuthology.orchestra.run.smithi084.stdout:(30/30): librbd1-18.2.4-837.g7a853b17.el9.x86_6 10 MB/s | 3.0 MB 00:00 2024-09-11T03:28:24.269 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:28:24.269 INFO:teuthology.orchestra.run.smithi084.stdout:Total 22 MB/s | 53 MB 00:02 2024-09-11T03:28:24.337 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-11T03:28:24.371 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-11T03:28:24.371 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-11T03:28:24.375 INFO:teuthology.orchestra.run.smithi121.stdout:(1/30): ceph-base-18.2.4-837.g7a853b17.el9.x86_ 9.1 MB/s | 5.1 MB 00:00 2024-09-11T03:28:24.425 INFO:teuthology.orchestra.run.smithi121.stdout:(2/30): ceph-selinux-18.2.4-837.g7a853b17.el9.x 504 kB/s | 25 kB 00:00 2024-09-11T03:28:24.442 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-11T03:28:24.492 INFO:teuthology.orchestra.run.smithi121.stdout:(3/30): libcephfs2-18.2.4-837.g7a853b17.el9.x86 10 MB/s | 712 kB 00:00 2024-09-11T03:28:24.558 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : librados2-2:18.2.4-837.g7a853b17.el9.x86_64 5/32 2024-09-11T03:28:24.609 INFO:teuthology.orchestra.run.smithi121.stdout:(4/30): ceph-radosgw-18.2.4-837.g7a853b17.el9.x 9.8 MB/s | 7.7 MB 00:00 2024-09-11T03:28:24.643 INFO:teuthology.orchestra.run.smithi121.stdout:(5/30): libradosstriper1-18.2.4-837.g7a853b17.e 3.1 MB/s | 477 kB 00:00 2024-09-11T03:28:24.651 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados2-2:18.2.4-837.g7a853b17.el9.x86_64 5/32 2024-09-11T03:28:24.701 INFO:teuthology.orchestra.run.smithi121.stdout:(6/30): python3-ceph-argparse-18.2.4-837.g7a853 771 kB/s | 45 kB 00:00 2024-09-11T03:28:24.743 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rados-2:18.2.4-837.g7a853b17.el9.x86_64 6/32 2024-09-11T03:28:24.768 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephfs2-2:18.2.4-837.g7a853b17.el9.x86_64 7/32 2024-09-11T03:28:24.777 INFO:teuthology.orchestra.run.smithi121.stdout:(7/30): python3-ceph-common-18.2.4-837.g7a853b1 1.7 MB/s | 129 kB 00:00 2024-09-11T03:28:24.831 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephfs2-2:18.2.4-837.g7a853b17.el9.x86_64 7/32 2024-09-11T03:28:24.852 INFO:teuthology.orchestra.run.smithi121.stdout:(8/30): librgw2-18.2.4-837.g7a853b17.el9.x86_64 18 MB/s | 4.5 MB 00:00 2024-09-11T03:28:24.859 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-11T03:28:24.877 INFO:teuthology.orchestra.run.smithi121.stdout:(9/30): python3-cephfs-18.2.4-837.g7a853b17.el9 1.6 MB/s | 161 kB 00:00 2024-09-11T03:28:24.902 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ceph-argparse-2:18.2.4-837.g7a853b17.el9.x 9/32 2024-09-11T03:28:24.903 INFO:teuthology.orchestra.run.smithi121.stdout:(10/30): python3-rados-18.2.4-837.g7a853b17.el9 6.2 MB/s | 319 kB 00:00 2024-09-11T03:28:24.928 INFO:teuthology.orchestra.run.smithi121.stdout:(11/30): python3-rbd-18.2.4-837.g7a853b17.el9.x 5.8 MB/s | 300 kB 00:00 2024-09-11T03:28:24.953 INFO:teuthology.orchestra.run.smithi121.stdout:(12/30): python3-rgw-18.2.4-837.g7a853b17.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-11T03:28:24.968 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cephfs-2:18.2.4-837.g7a853b17.el9.x86_64 10/32 2024-09-11T03:28:24.981 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-11T03:28:24.981 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-11T03:28:24.995 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libradosstriper1-2:18.2.4-837.g7a853b17.el9.x86_64 11/32 2024-09-11T03:28:25.053 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libradosstriper1-2:18.2.4-837.g7a853b17.el9.x86_64 11/32 2024-09-11T03:28:25.145 INFO:teuthology.orchestra.run.smithi121.stdout:(13/30): ceph-common-18.2.4-837.g7a853b17.el9.x 14 MB/s | 18 MB 00:01 2024-09-11T03:28:25.157 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-11T03:28:25.194 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-11T03:28:25.236 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-11T03:28:25.267 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-09-11T03:28:25.294 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-11T03:28:25.295 INFO:teuthology.orchestra.run.smithi121.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 291 kB/s | 107 kB 00:00 2024-09-11T03:28:25.320 INFO:teuthology.orchestra.run.smithi121.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 258 kB/s | 45 kB 00:00 2024-09-11T03:28:25.330 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-11T03:28:25.346 INFO:teuthology.orchestra.run.smithi121.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 408 kB/s | 160 kB 00:00 2024-09-11T03:28:25.363 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-11T03:28:25.396 INFO:teuthology.orchestra.run.smithi121.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 844 kB/s | 42 kB 00:00 2024-09-11T03:28:25.446 INFO:teuthology.orchestra.run.smithi121.stdout:(18/30): fmt-8.1.1-5.el9.x86_64.rpm 2.2 MB/s | 111 kB 00:00 2024-09-11T03:28:25.471 INFO:teuthology.orchestra.run.smithi121.stdout:(19/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-09-11T03:28:25.505 INFO:teuthology.orchestra.run.smithi121.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 5.2 MB/s | 309 kB 00:00 2024-09-11T03:28:25.539 INFO:teuthology.orchestra.run.smithi121.stdout:(21/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 662 kB 00:00 2024-09-11T03:28:25.555 INFO:teuthology.orchestra.run.smithi121.stdout:(22/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 536 kB/s | 26 kB 00:00 2024-09-11T03:28:25.589 INFO:teuthology.orchestra.run.smithi121.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 2.5 MB/s | 67 kB 00:00 2024-09-11T03:28:25.615 INFO:teuthology.orchestra.run.smithi121.stdout:(24/30): liboath-2.6.7-2.el9.x86_64.rpm 641 kB/s | 49 kB 00:00 2024-09-11T03:28:25.649 INFO:teuthology.orchestra.run.smithi121.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-11T03:28:25.665 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-11T03:28:25.673 INFO:teuthology.orchestra.run.smithi121.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 10 MB/s | 839 kB 00:00 2024-09-11T03:28:25.701 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : librbd1-2:18.2.4-837.g7a853b17.el9.x86_64 20/32 2024-09-11T03:28:25.766 INFO:teuthology.orchestra.run.smithi121.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-11T03:28:25.791 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librbd1-2:18.2.4-837.g7a853b17.el9.x86_64 20/32 2024-09-11T03:28:25.822 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rbd-2:18.2.4-837.g7a853b17.el9.x86_64 21/32 2024-09-11T03:28:25.824 INFO:teuthology.orchestra.run.smithi121.stdout:(28/30): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-11T03:28:25.867 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-11T03:28:25.925 INFO:teuthology.orchestra.run.smithi121.stdout:(29/30): librados2-18.2.4-837.g7a853b17.el9.x86 13 MB/s | 3.3 MB 00:00 2024-09-11T03:28:26.008 INFO:teuthology.orchestra.run.smithi121.stdout:(30/30): librbd1-18.2.4-837.g7a853b17.el9.x86_6 12 MB/s | 3.0 MB 00:00 2024-09-11T03:28:26.012 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:28:26.012 INFO:teuthology.orchestra.run.smithi121.stdout:Total 21 MB/s | 53 MB 00:02 2024-09-11T03:28:26.081 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-11T03:28:26.114 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-11T03:28:26.114 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-11T03:28:26.271 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-11T03:28:26.304 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librgw2-2:18.2.4-837.g7a853b17.el9.x86_64 24/32 2024-09-11T03:28:26.374 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librgw2-2:18.2.4-837.g7a853b17.el9.x86_64 24/32 2024-09-11T03:28:26.378 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-11T03:28:26.450 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rgw-2:18.2.4-837.g7a853b17.el9.x86_64 25/32 2024-09-11T03:28:26.457 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-11T03:28:26.481 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ceph-common-2:18.2.4-837.g7a853b17.el9.x86 26/32 2024-09-11T03:28:26.486 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-11T03:28:26.536 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-11T03:28:26.751 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-11T03:28:26.751 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-11T03:28:26.852 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-11T03:28:26.960 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librados2-2:18.2.4-837.g7a853b17.el9.x86_64 5/32 2024-09-11T03:28:27.053 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-2:18.2.4-837.g7a853b17.el9.x86_64 5/32 2024-09-11T03:28:27.137 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rados-2:18.2.4-837.g7a853b17.el9.x86_64 6/32 2024-09-11T03:28:27.178 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs2-2:18.2.4-837.g7a853b17.el9.x86_64 7/32 2024-09-11T03:28:27.241 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs2-2:18.2.4-837.g7a853b17.el9.x86_64 7/32 2024-09-11T03:28:27.278 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-11T03:28:27.322 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-argparse-2:18.2.4-837.g7a853b17.el9.x 9/32 2024-09-11T03:28:27.387 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cephfs-2:18.2.4-837.g7a853b17.el9.x86_64 10/32 2024-09-11T03:28:27.414 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libradosstriper1-2:18.2.4-837.g7a853b17.el9.x86_64 11/32 2024-09-11T03:28:27.472 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libradosstriper1-2:18.2.4-837.g7a853b17.el9.x86_64 11/32 2024-09-11T03:28:27.577 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-11T03:28:27.613 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-11T03:28:27.663 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-11T03:28:27.694 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-09-11T03:28:27.721 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-11T03:28:27.757 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-11T03:28:27.790 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-11T03:28:28.095 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-11T03:28:28.136 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librbd1-2:18.2.4-837.g7a853b17.el9.x86_64 20/32 2024-09-11T03:28:28.228 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-2:18.2.4-837.g7a853b17.el9.x86_64 20/32 2024-09-11T03:28:28.232 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-11T03:28:28.258 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rbd-2:18.2.4-837.g7a853b17.el9.x86_64 21/32 2024-09-11T03:28:28.293 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-11T03:28:28.317 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-11T03:28:28.346 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-11T03:28:28.405 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-11T03:28:28.543 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 27/32 2024-09-11T03:28:28.577 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 27/32 2024-09-11T03:28:28.697 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-11T03:28:28.720 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-11T03:28:28.722 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librgw2-2:18.2.4-837.g7a853b17.el9.x86_64 24/32 2024-09-11T03:28:28.776 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librgw2-2:18.2.4-837.g7a853b17.el9.x86_64 24/32 2024-09-11T03:28:28.837 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:18.2.4-837.g7a853b17.el9.x86_64 5/32 2024-09-11T03:28:28.853 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rgw-2:18.2.4-837.g7a853b17.el9.x86_64 25/32 2024-09-11T03:28:28.882 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-common-2:18.2.4-837.g7a853b17.el9.x86 26/32 2024-09-11T03:28:28.920 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:18.2.4-837.g7a853b17.el9.x86_64 5/32 2024-09-11T03:28:29.012 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:18.2.4-837.g7a853b17.el9.x86_64 6/32 2024-09-11T03:28:29.040 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:18.2.4-837.g7a853b17.el9.x86_64 7/32 2024-09-11T03:28:29.101 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:18.2.4-837.g7a853b17.el9.x86_64 7/32 2024-09-11T03:28:29.123 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 27/32 2024-09-11T03:28:29.130 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-11T03:28:29.172 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-base-2:18.2.4-837.g7a853b17.el9.x86_64 28/32 2024-09-11T03:28:29.173 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:18.2.4-837.g7a853b17.el9.x 9/32 2024-09-11T03:28:29.239 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:18.2.4-837.g7a853b17.el9.x86_64 10/32 2024-09-11T03:28:29.266 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:18.2.4-837.g7a853b17.el9.x86_64 11/32 2024-09-11T03:28:29.279 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-base-2:18.2.4-837.g7a853b17.el9.x86_64 28/32 2024-09-11T03:28:29.279 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T03:28:29.279 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T03:28:29.279 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:28:29.323 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:18.2.4-837.g7a853b17.el9.x86_64 11/32 2024-09-11T03:28:29.429 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-11T03:28:29.465 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-11T03:28:29.506 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-11T03:28:29.538 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-09-11T03:28:29.548 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-selinux-2:18.2.4-837.g7a853b17.el9.x86_64 29/32 2024-09-11T03:28:29.564 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-11T03:28:29.601 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-11T03:28:29.633 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-11T03:28:29.931 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-11T03:28:29.971 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:18.2.4-837.g7a853b17.el9.x86_64 20/32 2024-09-11T03:28:30.062 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:18.2.4-837.g7a853b17.el9.x86_64 20/32 2024-09-11T03:28:30.093 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:18.2.4-837.g7a853b17.el9.x86_64 21/32 2024-09-11T03:28:30.128 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-11T03:28:30.531 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-11T03:28:30.557 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:18.2.4-837.g7a853b17.el9.x86_64 24/32 2024-09-11T03:28:30.612 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:18.2.4-837.g7a853b17.el9.x86_64 24/32 2024-09-11T03:28:30.680 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:18.2.4-837.g7a853b17.el9.x86_64 25/32 2024-09-11T03:28:30.709 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:18.2.4-837.g7a853b17.el9.x86 26/32 2024-09-11T03:28:30.993 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 27/32 2024-09-11T03:28:31.037 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 27/32 2024-09-11T03:28:31.582 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 27/32 2024-09-11T03:28:31.623 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-base-2:18.2.4-837.g7a853b17.el9.x86_64 28/32 2024-09-11T03:28:31.728 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-base-2:18.2.4-837.g7a853b17.el9.x86_64 28/32 2024-09-11T03:28:31.728 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T03:28:31.729 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T03:28:31.729 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:28:31.766 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-selinux-2:18.2.4-837.g7a853b17.el9.x86_64 29/32 2024-09-11T03:28:32.803 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 27/32 2024-09-11T03:28:32.848 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 27/32 2024-09-11T03:28:33.390 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 27/32 2024-09-11T03:28:33.435 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:18.2.4-837.g7a853b17.el9.x86_64 28/32 2024-09-11T03:28:33.536 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:18.2.4-837.g7a853b17.el9.x86_64 28/32 2024-09-11T03:28:33.536 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T03:28:33.536 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T03:28:33.536 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:28:33.569 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:18.2.4-837.g7a853b17.el9.x86_64 29/32 2024-09-11T03:28:39.484 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-selinux-2:18.2.4-837.g7a853b17.el9.x86_64 29/32 2024-09-11T03:28:39.484 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /sys 2024-09-11T03:28:39.484 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /proc 2024-09-11T03:28:39.484 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /mnt 2024-09-11T03:28:39.484 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /var/tmp 2024-09-11T03:28:39.484 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /home 2024-09-11T03:28:39.484 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /root 2024-09-11T03:28:39.484 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /tmp 2024-09-11T03:28:39.485 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:28:40.261 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-radosgw-2:18.2.4-837.g7a853b17.el9.x86_64 30/32 2024-09-11T03:28:40.286 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-837.g7a853b17.el9.x86_64 30/32 2024-09-11T03:28:40.286 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:28:40.286 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T03:28:40.286 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T03:28:40.286 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T03:28:40.286 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:28:40.288 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-11T03:28:40.381 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-11T03:28:40.381 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-11T03:28:41.567 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-11T03:28:41.568 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-base-2:18.2.4-837.g7a853b17.el9.x86_64 1/32 2024-09-11T03:28:41.568 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 2/32 2024-09-11T03:28:41.568 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-radosgw-2:18.2.4-837.g7a853b17.el9.x86_64 3/32 2024-09-11T03:28:41.568 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-selinux-2:18.2.4-837.g7a853b17.el9.x86_64 4/32 2024-09-11T03:28:41.568 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephfs2-2:18.2.4-837.g7a853b17.el9.x86_64 5/32 2024-09-11T03:28:41.568 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libradosstriper1-2:18.2.4-837.g7a853b17.el9.x86_64 6/32 2024-09-11T03:28:41.568 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librgw2-2:18.2.4-837.g7a853b17.el9.x86_64 7/32 2024-09-11T03:28:41.568 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ceph-argparse-2:18.2.4-837.g7a853b17.el9.x 8/32 2024-09-11T03:28:41.568 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ceph-common-2:18.2.4-837.g7a853b17.el9.x86 9/32 2024-09-11T03:28:41.568 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cephfs-2:18.2.4-837.g7a853b17.el9.x86_64 10/32 2024-09-11T03:28:41.568 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rados-2:18.2.4-837.g7a853b17.el9.x86_64 11/32 2024-09-11T03:28:41.569 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rbd-2:18.2.4-837.g7a853b17.el9.x86_64 12/32 2024-09-11T03:28:41.569 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rgw-2:18.2.4-837.g7a853b17.el9.x86_64 13/32 2024-09-11T03:28:41.569 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-11T03:28:41.569 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-11T03:28:41.569 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-11T03:28:41.569 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-11T03:28:41.569 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-11T03:28:41.569 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-11T03:28:41.569 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-11T03:28:41.569 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-09-11T03:28:41.569 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-11T03:28:41.569 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-11T03:28:41.570 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-11T03:28:41.570 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-11T03:28:41.570 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-11T03:28:41.570 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-11T03:28:41.570 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-11T03:28:41.570 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados2-2:18.2.4-837.g7a853b17.el9.x86_64 29/32 2024-09-11T03:28:41.570 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-11T03:28:41.570 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librbd1-2:18.2.4-837.g7a853b17.el9.x86_64 31/32 2024-09-11T03:28:41.872 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-selinux-2:18.2.4-837.g7a853b17.el9.x86_64 29/32 2024-09-11T03:28:41.872 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /sys 2024-09-11T03:28:41.872 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /proc 2024-09-11T03:28:41.872 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /mnt 2024-09-11T03:28:41.872 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /var/tmp 2024-09-11T03:28:41.872 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /home 2024-09-11T03:28:41.872 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /root 2024-09-11T03:28:41.872 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /tmp 2024-09-11T03:28:41.873 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:28:42.638 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-11T03:28:42.638 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:28:42.638 INFO:teuthology.orchestra.run.smithi175.stdout:Upgraded: 2024-09-11T03:28:42.638 INFO:teuthology.orchestra.run.smithi175.stdout: librados2-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:42.638 INFO:teuthology.orchestra.run.smithi175.stdout: librbd1-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:42.638 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-11T03:28:42.638 INFO:teuthology.orchestra.run.smithi175.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-11T03:28:42.639 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-base-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:42.639 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:42.639 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-radosgw-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:42.639 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-selinux-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:42.639 INFO:teuthology.orchestra.run.smithi175.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-11T03:28:42.639 INFO:teuthology.orchestra.run.smithi175.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T03:28:42.639 INFO:teuthology.orchestra.run.smithi175.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-11T03:28:42.639 INFO:teuthology.orchestra.run.smithi175.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-11T03:28:42.639 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs2-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:42.639 INFO:teuthology.orchestra.run.smithi175.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T03:28:42.639 INFO:teuthology.orchestra.run.smithi175.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T03:28:42.639 INFO:teuthology.orchestra.run.smithi175.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T03:28:42.640 INFO:teuthology.orchestra.run.smithi175.stdout: libradosstriper1-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:42.640 INFO:teuthology.orchestra.run.smithi175.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T03:28:42.640 INFO:teuthology.orchestra.run.smithi175.stdout: librgw2-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:42.640 INFO:teuthology.orchestra.run.smithi175.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T03:28:42.640 INFO:teuthology.orchestra.run.smithi175.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T03:28:42.640 INFO:teuthology.orchestra.run.smithi175.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-11T03:28:42.640 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-argparse-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:42.640 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:42.640 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cephfs-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:42.640 INFO:teuthology.orchestra.run.smithi175.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T03:28:42.640 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rados-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:42.640 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rbd-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:42.640 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rgw-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:42.641 INFO:teuthology.orchestra.run.smithi175.stdout: re2-1:20211101-20.el9.x86_64 2024-09-11T03:28:42.641 INFO:teuthology.orchestra.run.smithi175.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T03:28:42.641 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:28:42.641 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:28:42.645 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-radosgw-2:18.2.4-837.g7a853b17.el9.x86_64 30/32 2024-09-11T03:28:42.676 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-837.g7a853b17.el9.x86_64 30/32 2024-09-11T03:28:42.676 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:28:42.676 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T03:28:42.676 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T03:28:42.676 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T03:28:42.677 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:28:42.678 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-11T03:28:42.764 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-11T03:28:42.765 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-11T03:28:43.019 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-test 2024-09-11T03:28:43.472 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:18.2.4-837.g7a853b17.el9.x86_64 29/32 2024-09-11T03:28:43.472 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2024-09-11T03:28:43.472 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2024-09-11T03:28:43.472 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2024-09-11T03:28:43.472 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2024-09-11T03:28:43.472 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2024-09-11T03:28:43.473 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /root 2024-09-11T03:28:43.473 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2024-09-11T03:28:43.473 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:28:43.592 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:27 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:28:43.724 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:28:43.725 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:28:43.725 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-09-11T03:28:43.725 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:28:43.725 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-11T03:28:43.725 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-test x86_64 2:18.2.4-837.g7a853b17.el9 ceph 40 M 2024-09-11T03:28:43.725 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-09-11T03:28:43.725 INFO:teuthology.orchestra.run.smithi175.stdout: libcephsqlite x86_64 2:18.2.4-837.g7a853b17.el9 ceph 166 k 2024-09-11T03:28:43.725 INFO:teuthology.orchestra.run.smithi175.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T03:28:43.725 INFO:teuthology.orchestra.run.smithi175.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T03:28:43.726 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:28:43.726 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-11T03:28:43.726 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:28:43.726 INFO:teuthology.orchestra.run.smithi175.stdout:Install 4 Packages 2024-09-11T03:28:43.726 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:28:43.726 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 40 M 2024-09-11T03:28:43.727 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 164 M 2024-09-11T03:28:43.727 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-11T03:28:43.858 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-11T03:28:43.858 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-base-2:18.2.4-837.g7a853b17.el9.x86_64 1/32 2024-09-11T03:28:43.858 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 2/32 2024-09-11T03:28:43.858 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-radosgw-2:18.2.4-837.g7a853b17.el9.x86_64 3/32 2024-09-11T03:28:43.858 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-selinux-2:18.2.4-837.g7a853b17.el9.x86_64 4/32 2024-09-11T03:28:43.858 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs2-2:18.2.4-837.g7a853b17.el9.x86_64 5/32 2024-09-11T03:28:43.858 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libradosstriper1-2:18.2.4-837.g7a853b17.el9.x86_64 6/32 2024-09-11T03:28:43.858 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librgw2-2:18.2.4-837.g7a853b17.el9.x86_64 7/32 2024-09-11T03:28:43.859 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-argparse-2:18.2.4-837.g7a853b17.el9.x 8/32 2024-09-11T03:28:43.859 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-common-2:18.2.4-837.g7a853b17.el9.x86 9/32 2024-09-11T03:28:43.859 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cephfs-2:18.2.4-837.g7a853b17.el9.x86_64 10/32 2024-09-11T03:28:43.859 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rados-2:18.2.4-837.g7a853b17.el9.x86_64 11/32 2024-09-11T03:28:43.859 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rbd-2:18.2.4-837.g7a853b17.el9.x86_64 12/32 2024-09-11T03:28:43.859 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rgw-2:18.2.4-837.g7a853b17.el9.x86_64 13/32 2024-09-11T03:28:43.859 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-11T03:28:43.859 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-11T03:28:43.859 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-11T03:28:43.859 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-11T03:28:43.859 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-11T03:28:43.859 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-11T03:28:43.859 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-11T03:28:43.860 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-09-11T03:28:43.860 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-11T03:28:43.860 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-11T03:28:43.860 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-11T03:28:43.860 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-11T03:28:43.860 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-11T03:28:43.860 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-11T03:28:43.860 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-11T03:28:43.860 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-2:18.2.4-837.g7a853b17.el9.x86_64 29/32 2024-09-11T03:28:43.860 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-11T03:28:43.860 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-2:18.2.4-837.g7a853b17.el9.x86_64 31/32 2024-09-11T03:28:44.070 INFO:teuthology.orchestra.run.smithi175.stdout:(1/4): libcephsqlite-18.2.4-837.g7a853b17.el9.x 756 kB/s | 166 kB 00:00 2024-09-11T03:28:44.218 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:18.2.4-837.g7a853b17.el9.x86_64 30/32 2024-09-11T03:28:44.244 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-837.g7a853b17.el9.x86_64 30/32 2024-09-11T03:28:44.244 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:28:44.244 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T03:28:44.245 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T03:28:44.245 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T03:28:44.245 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:28:44.246 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-11T03:28:44.324 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-11T03:28:44.324 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-11T03:28:44.445 INFO:teuthology.orchestra.run.smithi175.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 170 kB/s | 64 kB 00:00 2024-09-11T03:28:44.487 INFO:teuthology.orchestra.run.smithi175.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 478 kB/s | 304 kB 00:00 2024-09-11T03:28:44.937 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-11T03:28:44.937 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:28:44.938 INFO:teuthology.orchestra.run.smithi084.stdout:Upgraded: 2024-09-11T03:28:44.938 INFO:teuthology.orchestra.run.smithi084.stdout: librados2-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:44.938 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:44.938 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-11T03:28:44.938 INFO:teuthology.orchestra.run.smithi084.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-11T03:28:44.938 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:44.938 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:44.938 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:44.938 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:44.938 INFO:teuthology.orchestra.run.smithi084.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-11T03:28:44.938 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T03:28:44.939 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-11T03:28:44.939 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-11T03:28:44.939 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:44.939 INFO:teuthology.orchestra.run.smithi084.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T03:28:44.939 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T03:28:44.939 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T03:28:44.939 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:44.939 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T03:28:44.939 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:44.939 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T03:28:44.939 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T03:28:44.939 INFO:teuthology.orchestra.run.smithi084.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-11T03:28:44.940 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:44.940 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:44.940 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:44.940 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T03:28:44.940 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:44.940 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:44.940 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:44.940 INFO:teuthology.orchestra.run.smithi084.stdout: re2-1:20211101-20.el9.x86_64 2024-09-11T03:28:44.940 INFO:teuthology.orchestra.run.smithi084.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T03:28:44.940 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:28:44.940 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:28:45.196 INFO:teuthology.orchestra.run.smithi175.stdout:(4/4): ceph-test-18.2.4-837.g7a853b17.el9.x86_6 29 MB/s | 40 MB 00:01 2024-09-11T03:28:45.198 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:28:45.198 INFO:teuthology.orchestra.run.smithi175.stdout:Total 27 MB/s | 40 MB 00:01 2024-09-11T03:28:45.226 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-11T03:28:45.246 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-11T03:28:45.246 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-11T03:28:45.283 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-test 2024-09-11T03:28:45.490 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-11T03:28:45.490 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:18.2.4-837.g7a853b17.el9.x86_64 1/32 2024-09-11T03:28:45.491 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 2/32 2024-09-11T03:28:45.491 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:18.2.4-837.g7a853b17.el9.x86_64 3/32 2024-09-11T03:28:45.491 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:18.2.4-837.g7a853b17.el9.x86_64 4/32 2024-09-11T03:28:45.491 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:18.2.4-837.g7a853b17.el9.x86_64 5/32 2024-09-11T03:28:45.491 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:18.2.4-837.g7a853b17.el9.x86_64 6/32 2024-09-11T03:28:45.491 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:18.2.4-837.g7a853b17.el9.x86_64 7/32 2024-09-11T03:28:45.491 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:18.2.4-837.g7a853b17.el9.x 8/32 2024-09-11T03:28:45.491 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:18.2.4-837.g7a853b17.el9.x86 9/32 2024-09-11T03:28:45.491 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:18.2.4-837.g7a853b17.el9.x86_64 10/32 2024-09-11T03:28:45.491 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:18.2.4-837.g7a853b17.el9.x86_64 11/32 2024-09-11T03:28:45.491 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:18.2.4-837.g7a853b17.el9.x86_64 12/32 2024-09-11T03:28:45.491 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:18.2.4-837.g7a853b17.el9.x86_64 13/32 2024-09-11T03:28:45.491 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-11T03:28:45.492 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-11T03:28:45.492 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-11T03:28:45.492 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-11T03:28:45.492 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-11T03:28:45.492 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-11T03:28:45.492 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-11T03:28:45.492 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-09-11T03:28:45.492 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-11T03:28:45.492 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-11T03:28:45.492 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-11T03:28:45.492 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-11T03:28:45.492 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-11T03:28:45.492 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-11T03:28:45.492 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-11T03:28:45.493 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:18.2.4-837.g7a853b17.el9.x86_64 29/32 2024-09-11T03:28:45.493 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-11T03:28:45.493 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:18.2.4-837.g7a853b17.el9.x86_64 31/32 2024-09-11T03:28:45.564 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-11T03:28:45.565 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-11T03:28:45.845 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:26 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:28:45.976 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:28:45.977 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:28:45.977 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-09-11T03:28:45.977 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:28:45.977 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-11T03:28:45.977 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test x86_64 2:18.2.4-837.g7a853b17.el9 ceph 40 M 2024-09-11T03:28:45.977 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-09-11T03:28:45.977 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite x86_64 2:18.2.4-837.g7a853b17.el9 ceph 166 k 2024-09-11T03:28:45.977 INFO:teuthology.orchestra.run.smithi084.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T03:28:45.978 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T03:28:45.978 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:28:45.978 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-11T03:28:45.978 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:28:45.978 INFO:teuthology.orchestra.run.smithi084.stdout:Install 4 Packages 2024-09-11T03:28:45.978 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:28:45.978 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 40 M 2024-09-11T03:28:45.979 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 164 M 2024-09-11T03:28:45.979 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-11T03:28:46.054 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-11T03:28:46.139 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-11T03:28:46.187 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-11T03:28:46.211 INFO:teuthology.orchestra.run.smithi084.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.6 MB/s | 304 kB 00:00 2024-09-11T03:28:46.214 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephsqlite-2:18.2.4-837.g7a853b17.el9.x86_64 3/4 2024-09-11T03:28:46.236 INFO:teuthology.orchestra.run.smithi084.stdout:(2/4): libcephsqlite-18.2.4-837.g7a853b17.el9.x 799 kB/s | 166 kB 00:00 2024-09-11T03:28:46.262 INFO:teuthology.orchestra.run.smithi084.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-11T03:28:46.449 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-11T03:28:46.449 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:28:46.449 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2024-09-11T03:28:46.449 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:46.450 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:46.450 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-11T03:28:46.450 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-11T03:28:46.450 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:46.450 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:46.450 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:46.450 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:46.450 INFO:teuthology.orchestra.run.smithi121.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-11T03:28:46.450 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T03:28:46.450 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-11T03:28:46.450 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-11T03:28:46.450 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:46.450 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T03:28:46.451 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T03:28:46.451 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T03:28:46.451 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:46.451 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T03:28:46.451 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:46.451 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T03:28:46.451 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T03:28:46.451 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-11T03:28:46.451 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:46.451 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:46.451 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:46.451 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T03:28:46.451 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:46.451 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:46.452 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:46.452 INFO:teuthology.orchestra.run.smithi121.stdout: re2-1:20211101-20.el9.x86_64 2024-09-11T03:28:46.452 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T03:28:46.452 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:28:46.452 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:28:46.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-test 2024-09-11T03:28:47.287 INFO:teuthology.orchestra.run.smithi084.stdout:(4/4): ceph-test-18.2.4-837.g7a853b17.el9.x86_6 32 MB/s | 40 MB 00:01 2024-09-11T03:28:47.289 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:28:47.289 INFO:teuthology.orchestra.run.smithi084.stdout:Total 31 MB/s | 40 MB 00:01 2024-09-11T03:28:47.316 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-11T03:28:47.336 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-11T03:28:47.336 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-11T03:28:47.361 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:26 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:28:47.501 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:28:47.502 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:28:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-11T03:28:47.502 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:28:47.502 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-11T03:28:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:18.2.4-837.g7a853b17.el9 ceph 40 M 2024-09-11T03:28:47.502 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-11T03:28:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:18.2.4-837.g7a853b17.el9 ceph 166 k 2024-09-11T03:28:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T03:28:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T03:28:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:28:47.502 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-11T03:28:47.503 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:28:47.503 INFO:teuthology.orchestra.run.smithi121.stdout:Install 4 Packages 2024-09-11T03:28:47.503 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:28:47.503 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 40 M 2024-09-11T03:28:47.503 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 164 M 2024-09-11T03:28:47.503 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-11T03:28:47.652 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-11T03:28:47.652 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-11T03:28:47.821 INFO:teuthology.orchestra.run.smithi121.stdout:(1/4): libcephsqlite-18.2.4-837.g7a853b17.el9.x 758 kB/s | 166 kB 00:00 2024-09-11T03:28:48.121 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-11T03:28:48.211 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-11T03:28:48.269 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-11T03:28:48.296 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephsqlite-2:18.2.4-837.g7a853b17.el9.x86_64 3/4 2024-09-11T03:28:48.447 INFO:teuthology.orchestra.run.smithi121.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 102 kB/s | 64 kB 00:00 2024-09-11T03:28:48.489 INFO:teuthology.orchestra.run.smithi121.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 343 kB/s | 304 kB 00:00 2024-09-11T03:28:48.855 INFO:teuthology.orchestra.run.smithi121.stdout:(4/4): ceph-test-18.2.4-837.g7a853b17.el9.x86_6 32 MB/s | 40 MB 00:01 2024-09-11T03:28:48.858 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:28:48.858 INFO:teuthology.orchestra.run.smithi121.stdout:Total 30 MB/s | 40 MB 00:01 2024-09-11T03:28:48.886 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-11T03:28:48.906 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-11T03:28:48.906 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-11T03:28:49.224 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-11T03:28:49.224 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-11T03:28:49.693 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-11T03:28:49.765 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-11T03:28:49.813 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-11T03:28:49.841 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:18.2.4-837.g7a853b17.el9.x86_64 3/4 2024-09-11T03:28:49.890 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephsqlite-2:18.2.4-837.g7a853b17.el9.x86_64 3/4 2024-09-11T03:28:49.970 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-test-2:18.2.4-837.g7a853b17.el9.x86_64 4/4 2024-09-11T03:28:51.916 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-test-2:18.2.4-837.g7a853b17.el9.x86_64 4/4 2024-09-11T03:28:51.917 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-test-2:18.2.4-837.g7a853b17.el9.x86_64 1/4 2024-09-11T03:28:51.917 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephsqlite-2:18.2.4-837.g7a853b17.el9.x86_64 2/4 2024-09-11T03:28:51.917 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-11T03:28:52.004 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephsqlite-2:18.2.4-837.g7a853b17.el9.x86_64 3/4 2024-09-11T03:28:52.077 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-test-2:18.2.4-837.g7a853b17.el9.x86_64 4/4 2024-09-11T03:28:52.212 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-11T03:28:52.212 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:28:52.212 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-11T03:28:52.212 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-test-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:52.212 INFO:teuthology.orchestra.run.smithi175.stdout: libcephsqlite-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:52.213 INFO:teuthology.orchestra.run.smithi175.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T03:28:52.213 INFO:teuthology.orchestra.run.smithi175.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T03:28:52.213 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:28:52.213 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:28:52.569 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph 2024-09-11T03:28:53.136 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:37 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:28:53.280 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:28:53.282 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:28:53.282 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-09-11T03:28:53.282 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:28:53.282 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-11T03:28:53.282 INFO:teuthology.orchestra.run.smithi175.stdout: ceph x86_64 2:18.2.4-837.g7a853b17.el9 ceph 6.5 k 2024-09-11T03:28:53.282 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-09-11T03:28:53.283 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mds x86_64 2:18.2.4-837.g7a853b17.el9 ceph 2.1 M 2024-09-11T03:28:53.283 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr x86_64 2:18.2.4-837.g7a853b17.el9 ceph 1.5 M 2024-09-11T03:28:53.283 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-modules-core noarch 2:18.2.4-837.g7a853b17.el9 ceph-noarch 246 k 2024-09-11T03:28:53.283 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mon x86_64 2:18.2.4-837.g7a853b17.el9 ceph 4.7 M 2024-09-11T03:28:53.283 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-osd x86_64 2:18.2.4-837.g7a853b17.el9 ceph 17 M 2024-09-11T03:28:53.283 INFO:teuthology.orchestra.run.smithi175.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T03:28:53.283 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T03:28:53.283 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T03:28:53.283 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T03:28:53.283 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T03:28:53.283 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T03:28:53.283 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T03:28:53.284 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-11T03:28:53.284 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T03:28:53.284 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T03:28:53.284 INFO:teuthology.orchestra.run.smithi175.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T03:28:53.284 INFO:teuthology.orchestra.run.smithi175.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T03:28:53.284 INFO:teuthology.orchestra.run.smithi175.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T03:28:53.284 INFO:teuthology.orchestra.run.smithi175.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T03:28:53.284 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T03:28:53.284 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T03:28:53.284 INFO:teuthology.orchestra.run.smithi175.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T03:28:53.284 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T03:28:53.284 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T03:28:53.285 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T03:28:53.285 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T03:28:53.285 INFO:teuthology.orchestra.run.smithi175.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T03:28:53.285 INFO:teuthology.orchestra.run.smithi175.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T03:28:53.285 INFO:teuthology.orchestra.run.smithi175.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T03:28:53.285 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T03:28:53.285 INFO:teuthology.orchestra.run.smithi175.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T03:28:53.285 INFO:teuthology.orchestra.run.smithi175.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T03:28:53.285 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:28:53.285 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-11T03:28:53.285 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:28:53.285 INFO:teuthology.orchestra.run.smithi175.stdout:Install 33 Packages 2024-09-11T03:28:53.285 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:28:53.286 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 29 M 2024-09-11T03:28:53.286 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 104 M 2024-09-11T03:28:53.286 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-11T03:28:53.546 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:18.2.4-837.g7a853b17.el9.x86_64 3/4 2024-09-11T03:28:53.621 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:18.2.4-837.g7a853b17.el9.x86_64 4/4 2024-09-11T03:28:53.871 INFO:teuthology.orchestra.run.smithi175.stdout:(1/33): ceph-18.2.4-837.g7a853b17.el9.x86_64.rp 47 kB/s | 6.5 kB 00:00 2024-09-11T03:28:54.021 INFO:teuthology.orchestra.run.smithi175.stdout:(2/33): ceph-mgr-18.2.4-837.g7a853b17.el9.x86_6 5.1 MB/s | 1.5 MB 00:00 2024-09-11T03:28:54.071 INFO:teuthology.orchestra.run.smithi175.stdout:(3/33): ceph-mds-18.2.4-837.g7a853b17.el9.x86_6 6.2 MB/s | 2.1 MB 00:00 2024-09-11T03:28:54.073 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-test-2:18.2.4-837.g7a853b17.el9.x86_64 4/4 2024-09-11T03:28:54.073 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-test-2:18.2.4-837.g7a853b17.el9.x86_64 1/4 2024-09-11T03:28:54.073 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephsqlite-2:18.2.4-837.g7a853b17.el9.x86_64 2/4 2024-09-11T03:28:54.074 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-11T03:28:54.138 INFO:teuthology.orchestra.run.smithi175.stdout:(4/33): ceph-mgr-modules-core-18.2.4-837.g7a853 3.6 MB/s | 246 kB 00:00 2024-09-11T03:28:54.397 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-11T03:28:54.398 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:28:54.398 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-11T03:28:54.398 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:54.398 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:54.398 INFO:teuthology.orchestra.run.smithi084.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T03:28:54.399 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T03:28:54.399 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:28:54.399 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:28:54.571 INFO:teuthology.orchestra.run.smithi175.stdout:(5/33): ceph-osd-18.2.4-837.g7a853b17.el9.x86_6 30 MB/s | 17 MB 00:00 2024-09-11T03:28:54.597 INFO:teuthology.orchestra.run.smithi175.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 552 kB/s | 253 kB 00:00 2024-09-11T03:28:54.672 INFO:teuthology.orchestra.run.smithi175.stdout:(7/33): ceph-mon-18.2.4-837.g7a853b17.el9.x86_6 5.8 MB/s | 4.7 MB 00:00 2024-09-11T03:28:54.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph 2024-09-11T03:28:54.730 INFO:teuthology.orchestra.run.smithi175.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 797 kB/s | 106 kB 00:00 2024-09-11T03:28:54.906 INFO:teuthology.orchestra.run.smithi175.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 713 kB/s | 125 kB 00:00 2024-09-11T03:28:55.048 INFO:teuthology.orchestra.run.smithi175.stdout:(10/33): python3-urllib3-1.26.5-6.el9.noarch.rp 1.5 MB/s | 215 kB 00:00 2024-09-11T03:28:55.081 INFO:teuthology.orchestra.run.smithi175.stdout:(11/33): python3-pycparser-2.20-6.el9.noarch.rp 330 kB/s | 135 kB 00:00 2024-09-11T03:28:55.199 INFO:teuthology.orchestra.run.smithi175.stdout:(12/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-11T03:28:55.216 INFO:teuthology.orchestra.run.smithi175.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 260 kB/s | 35 kB 00:00 2024-09-11T03:28:55.244 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:36 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:28:55.249 INFO:teuthology.orchestra.run.smithi175.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-11T03:28:55.283 INFO:teuthology.orchestra.run.smithi175.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-09-11T03:28:55.299 INFO:teuthology.orchestra.run.smithi175.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 498 kB/s | 42 kB 00:00 2024-09-11T03:28:55.325 INFO:teuthology.orchestra.run.smithi175.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-09-11T03:28:55.358 INFO:teuthology.orchestra.run.smithi175.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 413 kB/s | 11 kB 00:00 2024-09-11T03:28:55.379 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:28:55.382 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:28:55.382 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-09-11T03:28:55.382 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:28:55.382 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-11T03:28:55.382 INFO:teuthology.orchestra.run.smithi084.stdout: ceph x86_64 2:18.2.4-837.g7a853b17.el9 ceph 6.5 k 2024-09-11T03:28:55.382 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-09-11T03:28:55.382 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds x86_64 2:18.2.4-837.g7a853b17.el9 ceph 2.1 M 2024-09-11T03:28:55.382 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr x86_64 2:18.2.4-837.g7a853b17.el9 ceph 1.5 M 2024-09-11T03:28:55.382 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core noarch 2:18.2.4-837.g7a853b17.el9 ceph-noarch 246 k 2024-09-11T03:28:55.382 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon x86_64 2:18.2.4-837.g7a853b17.el9 ceph 4.7 M 2024-09-11T03:28:55.383 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd x86_64 2:18.2.4-837.g7a853b17.el9 ceph 17 M 2024-09-11T03:28:55.383 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T03:28:55.383 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T03:28:55.383 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T03:28:55.383 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T03:28:55.383 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T03:28:55.383 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T03:28:55.383 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T03:28:55.383 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-11T03:28:55.383 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T03:28:55.383 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T03:28:55.384 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T03:28:55.384 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T03:28:55.384 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T03:28:55.384 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T03:28:55.384 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T03:28:55.384 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T03:28:55.384 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T03:28:55.384 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T03:28:55.384 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T03:28:55.384 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T03:28:55.384 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T03:28:55.384 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T03:28:55.385 INFO:teuthology.orchestra.run.smithi084.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T03:28:55.385 INFO:teuthology.orchestra.run.smithi084.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T03:28:55.385 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T03:28:55.385 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T03:28:55.385 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T03:28:55.385 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:28:55.385 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-11T03:28:55.385 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:28:55.385 INFO:teuthology.orchestra.run.smithi084.stdout:Install 33 Packages 2024-09-11T03:28:55.385 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:28:55.386 INFO:teuthology.orchestra.run.smithi175.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 761 kB/s | 18 kB 00:00 2024-09-11T03:28:55.386 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 29 M 2024-09-11T03:28:55.386 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 104 M 2024-09-11T03:28:55.386 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-11T03:28:55.409 INFO:teuthology.orchestra.run.smithi175.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 924 kB/s | 23 kB 00:00 2024-09-11T03:28:55.434 INFO:teuthology.orchestra.run.smithi175.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-09-11T03:28:55.459 INFO:teuthology.orchestra.run.smithi175.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-11T03:28:55.485 INFO:teuthology.orchestra.run.smithi175.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-11T03:28:55.510 INFO:teuthology.orchestra.run.smithi175.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-09-11T03:28:55.544 INFO:teuthology.orchestra.run.smithi175.stdout:(25/33): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2024-09-11T03:28:55.569 INFO:teuthology.orchestra.run.smithi175.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 940 kB/s | 79 kB 00:00 2024-09-11T03:28:55.603 INFO:teuthology.orchestra.run.smithi175.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-09-11T03:28:55.628 INFO:teuthology.orchestra.run.smithi175.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-11T03:28:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-test-2:18.2.4-837.g7a853b17.el9.x86_64 4/4 2024-09-11T03:28:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:18.2.4-837.g7a853b17.el9.x86_64 1/4 2024-09-11T03:28:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:18.2.4-837.g7a853b17.el9.x86_64 2/4 2024-09-11T03:28:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-11T03:28:55.644 INFO:teuthology.orchestra.run.smithi084.stdout:(1/33): ceph-18.2.4-837.g7a853b17.el9.x86_64.rp 51 kB/s | 6.5 kB 00:00 2024-09-11T03:28:55.653 INFO:teuthology.orchestra.run.smithi175.stdout:(29/33): python3-portend-3.1.0-2.el9.noarch.rpm 150 kB/s | 16 kB 00:00 2024-09-11T03:28:55.678 INFO:teuthology.orchestra.run.smithi175.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 486 kB/s | 36 kB 00:00 2024-09-11T03:28:55.704 INFO:teuthology.orchestra.run.smithi175.stdout:(31/33): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-11T03:28:55.729 INFO:teuthology.orchestra.run.smithi175.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-09-11T03:28:55.762 INFO:teuthology.orchestra.run.smithi175.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-11T03:28:55.767 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:28:55.767 INFO:teuthology.orchestra.run.smithi175.stdout:Total 12 MB/s | 29 MB 00:02 2024-09-11T03:28:55.819 INFO:teuthology.orchestra.run.smithi084.stdout:(2/33): ceph-mgr-18.2.4-837.g7a853b17.el9.x86_6 4.8 MB/s | 1.5 MB 00:00 2024-09-11T03:28:55.886 INFO:teuthology.orchestra.run.smithi084.stdout:(3/33): ceph-mds-18.2.4-837.g7a853b17.el9.x86_6 5.7 MB/s | 2.1 MB 00:00 2024-09-11T03:28:55.891 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-11T03:28:55.928 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-11T03:28:55.928 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-11T03:28:55.967 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-11T03:28:55.967 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:28:55.968 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-11T03:28:55.968 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:55.968 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:28:55.968 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T03:28:55.968 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T03:28:55.968 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:28:55.968 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:28:56.011 INFO:teuthology.orchestra.run.smithi084.stdout:(4/33): ceph-mon-18.2.4-837.g7a853b17.el9.x86_6 13 MB/s | 4.7 MB 00:00 2024-09-11T03:28:56.037 INFO:teuthology.orchestra.run.smithi084.stdout:(5/33): ceph-mgr-modules-core-18.2.4-837.g7a853 1.6 MB/s | 246 kB 00:00 2024-09-11T03:28:56.254 INFO:teuthology.orchestra.run.smithi084.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-09-11T03:28:56.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph 2024-09-11T03:28:56.322 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-11T03:28:56.322 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-11T03:28:56.437 INFO:teuthology.orchestra.run.smithi084.stdout:(7/33): ceph-osd-18.2.4-837.g7a853b17.el9.x86_6 27 MB/s | 17 MB 00:00 2024-09-11T03:28:56.463 INFO:teuthology.orchestra.run.smithi084.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 511 kB/s | 106 kB 00:00 2024-09-11T03:28:56.521 INFO:teuthology.orchestra.run.smithi084.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-09-11T03:28:56.571 INFO:teuthology.orchestra.run.smithi084.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 2.3 MB/s | 1.2 MB 00:00 2024-09-11T03:28:56.597 INFO:teuthology.orchestra.run.smithi084.stdout:(11/33): python3-pycparser-2.20-6.el9.noarch.rp 852 kB/s | 135 kB 00:00 2024-09-11T03:28:56.747 INFO:teuthology.orchestra.run.smithi084.stdout:(12/33): python3-urllib3-1.26.5-6.el9.noarch.rp 953 kB/s | 215 kB 00:00 2024-09-11T03:28:56.826 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:35 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:28:56.960 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:28:56.962 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:28:56.962 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-11T03:28:56.963 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:28:56.963 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-11T03:28:56.963 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:18.2.4-837.g7a853b17.el9 ceph 6.5 k 2024-09-11T03:28:56.963 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-11T03:28:56.963 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:18.2.4-837.g7a853b17.el9 ceph 2.1 M 2024-09-11T03:28:56.963 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:18.2.4-837.g7a853b17.el9 ceph 1.5 M 2024-09-11T03:28:56.963 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:18.2.4-837.g7a853b17.el9 ceph-noarch 246 k 2024-09-11T03:28:56.963 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:18.2.4-837.g7a853b17.el9 ceph 4.7 M 2024-09-11T03:28:56.963 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:18.2.4-837.g7a853b17.el9 ceph 17 M 2024-09-11T03:28:56.963 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T03:28:56.963 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T03:28:56.963 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T03:28:56.963 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T03:28:56.964 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T03:28:56.964 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T03:28:56.964 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T03:28:56.964 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-11T03:28:56.964 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T03:28:56.964 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T03:28:56.964 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T03:28:56.964 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T03:28:56.964 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T03:28:56.964 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T03:28:56.964 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T03:28:56.964 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T03:28:56.964 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T03:28:56.965 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T03:28:56.965 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T03:28:56.965 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T03:28:56.965 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T03:28:56.965 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T03:28:56.965 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T03:28:56.965 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T03:28:56.965 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T03:28:56.965 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T03:28:56.965 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T03:28:56.965 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:28:56.965 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-11T03:28:56.965 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:28:56.966 INFO:teuthology.orchestra.run.smithi121.stdout:Install 33 Packages 2024-09-11T03:28:56.966 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:28:56.966 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 29 M 2024-09-11T03:28:56.966 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 104 M 2024-09-11T03:28:56.966 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-11T03:28:57.222 INFO:teuthology.orchestra.run.smithi084.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 56 kB/s | 35 kB 00:00 2024-09-11T03:28:57.347 INFO:teuthology.orchestra.run.smithi084.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 85 kB/s | 51 kB 00:00 2024-09-11T03:28:57.372 INFO:teuthology.orchestra.run.smithi084.stdout:(15/33): python3-mako-1.1.4-6.el9.noarch.rpm 215 kB/s | 172 kB 00:00 2024-09-11T03:28:57.431 INFO:teuthology.orchestra.run.smithi084.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 523 kB/s | 43 kB 00:00 2024-09-11T03:28:57.456 INFO:teuthology.orchestra.run.smithi084.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-11T03:28:57.473 INFO:teuthology.orchestra.run.smithi121.stdout:(1/33): ceph-18.2.4-837.g7a853b17.el9.x86_64.rp 52 kB/s | 6.5 kB 00:00 2024-09-11T03:28:57.490 INFO:teuthology.orchestra.run.smithi084.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 409 kB/s | 11 kB 00:00 2024-09-11T03:28:57.515 INFO:teuthology.orchestra.run.smithi084.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2024-09-11T03:28:57.531 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-11T03:28:57.540 INFO:teuthology.orchestra.run.smithi084.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-09-11T03:28:57.565 INFO:teuthology.orchestra.run.smithi084.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-11T03:28:57.591 INFO:teuthology.orchestra.run.smithi084.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-11T03:28:57.598 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-11T03:28:57.616 INFO:teuthology.orchestra.run.smithi084.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 395 kB/s | 20 kB 00:00 2024-09-11T03:28:57.639 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-11T03:28:57.641 INFO:teuthology.orchestra.run.smithi084.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-09-11T03:28:57.665 INFO:teuthology.orchestra.run.smithi121.stdout:(2/33): ceph-mgr-18.2.4-837.g7a853b17.el9.x86_6 4.6 MB/s | 1.5 MB 00:00 2024-09-11T03:28:57.667 INFO:teuthology.orchestra.run.smithi084.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-11T03:28:57.681 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-11T03:28:57.692 INFO:teuthology.orchestra.run.smithi084.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 665 kB/s | 16 kB 00:00 2024-09-11T03:28:57.717 INFO:teuthology.orchestra.run.smithi084.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-09-11T03:28:57.732 INFO:teuthology.orchestra.run.smithi121.stdout:(3/33): ceph-mds-18.2.4-837.g7a853b17.el9.x86_6 5.5 MB/s | 2.1 MB 00:00 2024-09-11T03:28:57.742 INFO:teuthology.orchestra.run.smithi084.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-11T03:28:57.768 INFO:teuthology.orchestra.run.smithi084.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-09-11T03:28:57.774 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-11T03:28:57.782 INFO:teuthology.orchestra.run.smithi121.stdout:(4/33): ceph-mgr-modules-core-18.2.4-837.g7a853 4.8 MB/s | 246 kB 00:00 2024-09-11T03:28:57.793 INFO:teuthology.orchestra.run.smithi084.stdout:(30/33): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-11T03:28:57.810 INFO:teuthology.orchestra.run.smithi084.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-11T03:28:57.835 INFO:teuthology.orchestra.run.smithi084.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.2 MB/s | 427 kB 00:00 2024-09-11T03:28:57.852 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-11T03:28:57.885 INFO:teuthology.orchestra.run.smithi084.stdout:(33/33): python3-toml-0.10.2-6.el9.noarch.rpm 63 kB/s | 42 kB 00:00 2024-09-11T03:28:57.890 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:28:57.890 INFO:teuthology.orchestra.run.smithi084.stdout:Total 12 MB/s | 29 MB 00:02 2024-09-11T03:28:57.900 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-11T03:28:57.952 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-11T03:28:57.974 INFO:teuthology.orchestra.run.smithi121.stdout:(5/33): ceph-mon-18.2.4-837.g7a853b17.el9.x86_6 9.3 MB/s | 4.7 MB 00:00 2024-09-11T03:28:57.982 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-11T03:28:58.014 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-11T03:28:58.025 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-11T03:28:58.050 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-11T03:28:58.051 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-11T03:28:58.061 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-11T03:28:58.075 INFO:teuthology.orchestra.run.smithi121.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 867 kB/s | 253 kB 00:00 2024-09-11T03:28:58.110 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-11T03:28:58.125 INFO:teuthology.orchestra.run.smithi121.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-11T03:28:58.154 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-11T03:28:58.183 INFO:teuthology.orchestra.run.smithi121.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-09-11T03:28:58.188 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-11T03:28:58.237 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-11T03:28:58.242 INFO:teuthology.orchestra.run.smithi121.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-09-11T03:28:58.301 INFO:teuthology.orchestra.run.smithi121.stdout:(10/33): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-11T03:28:58.334 INFO:teuthology.orchestra.run.smithi121.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 3.4 MB/s | 1.2 MB 00:00 2024-09-11T03:28:58.354 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-11T03:28:58.450 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-11T03:28:58.451 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-11T03:28:58.543 INFO:teuthology.orchestra.run.smithi121.stdout:(12/33): ceph-osd-18.2.4-837.g7a853b17.el9.x86_ 19 MB/s | 17 MB 00:00 2024-09-11T03:28:58.568 INFO:teuthology.orchestra.run.smithi121.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 149 kB/s | 35 kB 00:00 2024-09-11T03:28:58.588 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-11T03:28:58.618 INFO:teuthology.orchestra.run.smithi121.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 543 kB/s | 172 kB 00:00 2024-09-11T03:28:58.644 INFO:teuthology.orchestra.run.smithi121.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 508 kB/s | 51 kB 00:00 2024-09-11T03:28:58.677 INFO:teuthology.orchestra.run.smithi121.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 383 kB/s | 42 kB 00:00 2024-09-11T03:28:58.710 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-11T03:28:58.711 INFO:teuthology.orchestra.run.smithi121.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 471 kB/s | 43 kB 00:00 2024-09-11T03:28:58.744 INFO:teuthology.orchestra.run.smithi121.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 437 kB/s | 11 kB 00:00 2024-09-11T03:28:58.770 INFO:teuthology.orchestra.run.smithi121.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-11T03:28:58.784 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-11T03:28:58.795 INFO:teuthology.orchestra.run.smithi121.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 361 kB/s | 18 kB 00:00 2024-09-11T03:28:58.821 INFO:teuthology.orchestra.run.smithi121.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-11T03:28:58.825 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-11T03:28:58.849 INFO:teuthology.orchestra.run.smithi121.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-11T03:28:58.871 INFO:teuthology.orchestra.run.smithi121.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-09-11T03:28:58.883 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-11T03:28:58.897 INFO:teuthology.orchestra.run.smithi121.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-09-11T03:28:58.913 INFO:teuthology.orchestra.run.smithi121.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-11T03:28:58.939 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-11T03:28:58.939 INFO:teuthology.orchestra.run.smithi121.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 660 kB/s | 16 kB 00:00 2024-09-11T03:28:58.964 INFO:teuthology.orchestra.run.smithi121.stdout:(27/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-11T03:28:58.989 INFO:teuthology.orchestra.run.smithi121.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-11T03:28:59.015 INFO:teuthology.orchestra.run.smithi121.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-09-11T03:28:59.040 INFO:teuthology.orchestra.run.smithi121.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-11T03:28:59.065 INFO:teuthology.orchestra.run.smithi121.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 797 kB/s | 20 kB 00:00 2024-09-11T03:28:59.090 INFO:teuthology.orchestra.run.smithi121.stdout:(32/33): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-11T03:28:59.115 INFO:teuthology.orchestra.run.smithi121.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-11T03:28:59.119 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:28:59.119 INFO:teuthology.orchestra.run.smithi121.stdout:Total 14 MB/s | 29 MB 00:02 2024-09-11T03:28:59.129 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-11T03:28:59.188 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-11T03:28:59.236 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-11T03:28:59.266 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-11T03:28:59.273 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-11T03:28:59.273 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-11T03:28:59.355 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-11T03:28:59.403 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-11T03:28:59.519 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-11T03:28:59.686 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-11T03:28:59.686 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-837.g7a853b17.el9.n 28/33 2024-09-11T03:28:59.691 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-11T03:28:59.692 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-11T03:28:59.722 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-2:18.2.4-837.g7a853b17.el9.x86_64 29/33 2024-09-11T03:28:59.769 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-2:18.2.4-837.g7a853b17.el9.x86_64 29/33 2024-09-11T03:28:59.770 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:28:59.770 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T03:28:59.770 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T03:28:59.770 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T03:28:59.770 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:28:59.771 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-11T03:28:59.812 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-11T03:28:59.845 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-11T03:28:59.937 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-11T03:29:00.031 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-11T03:29:00.072 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-11T03:29:00.116 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-11T03:29:00.145 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-11T03:29:00.180 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-11T03:29:00.213 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-11T03:29:00.254 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-11T03:29:00.297 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-11T03:29:00.332 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-11T03:29:00.375 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-11T03:29:00.451 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-11T03:29:00.588 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-11T03:29:00.677 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-11T03:29:00.730 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-11T03:29:00.772 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-11T03:29:00.821 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-11T03:29:00.876 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-11T03:29:01.021 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-11T03:29:01.061 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-11T03:29:01.085 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-11T03:29:01.109 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-11T03:29:01.117 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-11T03:29:01.142 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-11T03:29:01.195 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-11T03:29:01.234 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-11T03:29:01.264 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-osd-2:18.2.4-837.g7a853b17.el9.x86_64 30/33 2024-09-11T03:29:01.284 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-11T03:29:01.292 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-osd-2:18.2.4-837.g7a853b17.el9.x86_64 30/33 2024-09-11T03:29:01.292 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:29:01.292 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T03:29:01.292 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T03:29:01.292 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T03:29:01.292 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:01.313 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-11T03:29:01.317 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-11T03:29:01.353 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-11T03:29:01.397 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-11T03:29:01.426 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-11T03:29:01.434 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-11T03:29:01.452 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-11T03:29:01.486 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-11T03:29:01.526 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-11T03:29:01.570 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-11T03:29:01.597 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-837.g7a853b17.el9.n 28/33 2024-09-11T03:29:01.604 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-11T03:29:01.627 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-2:18.2.4-837.g7a853b17.el9.x86_64 29/33 2024-09-11T03:29:01.646 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-11T03:29:01.653 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-2:18.2.4-837.g7a853b17.el9.x86_64 29/33 2024-09-11T03:29:01.653 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:29:01.654 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T03:29:01.654 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T03:29:01.654 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T03:29:01.654 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:01.725 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-11T03:29:01.748 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mon-2:18.2.4-837.g7a853b17.el9.x86_64 31/33 2024-09-11T03:29:01.772 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mon-2:18.2.4-837.g7a853b17.el9.x86_64 31/33 2024-09-11T03:29:01.772 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:29:01.772 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T03:29:01.772 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T03:29:01.773 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T03:29:01.773 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:01.863 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-11T03:29:01.950 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-11T03:29:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mds-2:18.2.4-837.g7a853b17.el9.x86_64 32/33 2024-09-11T03:29:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mds-2:18.2.4-837.g7a853b17.el9.x86_64 32/33 2024-09-11T03:29:01.998 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:29:01.998 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T03:29:01.998 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T03:29:01.998 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T03:29:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:02.012 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-11T03:29:02.060 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-11T03:29:02.061 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-2:18.2.4-837.g7a853b17.el9.x86_64 33/33 2024-09-11T03:29:02.116 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-11T03:29:02.172 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-11T03:29:02.358 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-11T03:29:02.416 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-11T03:29:02.486 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-11T03:29:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-11T03:29:02.615 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-11T03:29:02.731 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-11T03:29:02.891 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-837.g7a853b17.el9.n 28/33 2024-09-11T03:29:02.918 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:18.2.4-837.g7a853b17.el9.x86_64 29/33 2024-09-11T03:29:02.943 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:18.2.4-837.g7a853b17.el9.x86_64 29/33 2024-09-11T03:29:02.943 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:29:02.943 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T03:29:02.943 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T03:29:02.943 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T03:29:02.943 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:03.087 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-osd-2:18.2.4-837.g7a853b17.el9.x86_64 30/33 2024-09-11T03:29:03.116 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-osd-2:18.2.4-837.g7a853b17.el9.x86_64 30/33 2024-09-11T03:29:03.116 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:29:03.116 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T03:29:03.116 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T03:29:03.116 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T03:29:03.116 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:03.579 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mon-2:18.2.4-837.g7a853b17.el9.x86_64 31/33 2024-09-11T03:29:03.604 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mon-2:18.2.4-837.g7a853b17.el9.x86_64 31/33 2024-09-11T03:29:03.604 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:29:03.604 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T03:29:03.604 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T03:29:03.604 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T03:29:03.604 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:03.706 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-2:18.2.4-837.g7a853b17.el9.x86_64 33/33 2024-09-11T03:29:03.706 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-2:18.2.4-837.g7a853b17.el9.x86_64 1/33 2024-09-11T03:29:03.706 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mds-2:18.2.4-837.g7a853b17.el9.x86_64 2/33 2024-09-11T03:29:03.707 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-2:18.2.4-837.g7a853b17.el9.x86_64 3/33 2024-09-11T03:29:03.707 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mon-2:18.2.4-837.g7a853b17.el9.x86_64 4/33 2024-09-11T03:29:03.707 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-osd-2:18.2.4-837.g7a853b17.el9.x86_64 5/33 2024-09-11T03:29:03.707 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-837.g7a853b17.el9.n 6/33 2024-09-11T03:29:03.707 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-11T03:29:03.707 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-11T03:29:03.707 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-11T03:29:03.707 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-11T03:29:03.707 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-11T03:29:03.707 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-11T03:29:03.707 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-11T03:29:03.707 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-11T03:29:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-11T03:29:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-11T03:29:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-11T03:29:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-11T03:29:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-11T03:29:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-11T03:29:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-11T03:29:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-11T03:29:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-11T03:29:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-11T03:29:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-11T03:29:03.709 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-11T03:29:03.710 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-11T03:29:03.710 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-11T03:29:03.710 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-11T03:29:03.710 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-11T03:29:03.710 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-11T03:29:03.710 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-11T03:29:03.805 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mds-2:18.2.4-837.g7a853b17.el9.x86_64 32/33 2024-09-11T03:29:03.829 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mds-2:18.2.4-837.g7a853b17.el9.x86_64 32/33 2024-09-11T03:29:03.829 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:29:03.829 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T03:29:03.829 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T03:29:03.829 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T03:29:03.829 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:03.892 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-2:18.2.4-837.g7a853b17.el9.x86_64 33/33 2024-09-11T03:29:04.394 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:18.2.4-837.g7a853b17.el9.x86_64 30/33 2024-09-11T03:29:04.423 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:18.2.4-837.g7a853b17.el9.x86_64 30/33 2024-09-11T03:29:04.424 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:29:04.424 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T03:29:04.424 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T03:29:04.424 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T03:29:04.424 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:04.633 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-11T03:29:04.633 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:04.633 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-11T03:29:04.633 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:04.633 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mds-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:04.634 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:04.634 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-modules-core-2:18.2.4-837.g7a853b17.el9.noarch 2024-09-11T03:29:04.634 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mon-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:04.634 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-osd-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:04.634 INFO:teuthology.orchestra.run.smithi175.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T03:29:04.634 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T03:29:04.634 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T03:29:04.634 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T03:29:04.634 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T03:29:04.634 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T03:29:04.634 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T03:29:04.634 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T03:29:04.634 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T03:29:04.635 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T03:29:04.635 INFO:teuthology.orchestra.run.smithi175.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T03:29:04.635 INFO:teuthology.orchestra.run.smithi175.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T03:29:04.635 INFO:teuthology.orchestra.run.smithi175.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T03:29:04.635 INFO:teuthology.orchestra.run.smithi175.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T03:29:04.635 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T03:29:04.635 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T03:29:04.635 INFO:teuthology.orchestra.run.smithi175.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T03:29:04.635 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T03:29:04.635 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T03:29:04.635 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T03:29:04.635 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T03:29:04.636 INFO:teuthology.orchestra.run.smithi175.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T03:29:04.636 INFO:teuthology.orchestra.run.smithi175.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T03:29:04.636 INFO:teuthology.orchestra.run.smithi175.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T03:29:04.636 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T03:29:04.636 INFO:teuthology.orchestra.run.smithi175.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T03:29:04.636 INFO:teuthology.orchestra.run.smithi175.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T03:29:04.636 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:04.636 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:04.871 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:18.2.4-837.g7a853b17.el9.x86_64 31/33 2024-09-11T03:29:04.895 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:18.2.4-837.g7a853b17.el9.x86_64 31/33 2024-09-11T03:29:04.895 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:29:04.895 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T03:29:04.895 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T03:29:04.896 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T03:29:04.896 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:05.036 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-base 2024-09-11T03:29:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:18.2.4-837.g7a853b17.el9.x86_64 32/33 2024-09-11T03:29:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:18.2.4-837.g7a853b17.el9.x86_64 32/33 2024-09-11T03:29:05.120 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:29:05.120 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T03:29:05.121 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T03:29:05.121 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T03:29:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:05.317 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:18.2.4-837.g7a853b17.el9.x86_64 33/33 2024-09-11T03:29:05.573 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-2:18.2.4-837.g7a853b17.el9.x86_64 33/33 2024-09-11T03:29:05.574 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-2:18.2.4-837.g7a853b17.el9.x86_64 1/33 2024-09-11T03:29:05.574 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mds-2:18.2.4-837.g7a853b17.el9.x86_64 2/33 2024-09-11T03:29:05.574 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-2:18.2.4-837.g7a853b17.el9.x86_64 3/33 2024-09-11T03:29:05.574 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mon-2:18.2.4-837.g7a853b17.el9.x86_64 4/33 2024-09-11T03:29:05.574 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-osd-2:18.2.4-837.g7a853b17.el9.x86_64 5/33 2024-09-11T03:29:05.574 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-837.g7a853b17.el9.n 6/33 2024-09-11T03:29:05.574 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-11T03:29:05.574 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-11T03:29:05.574 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-11T03:29:05.574 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-11T03:29:05.574 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-11T03:29:05.574 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-11T03:29:05.574 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-11T03:29:05.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-11T03:29:05.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-11T03:29:05.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-11T03:29:05.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-11T03:29:05.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-11T03:29:05.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-11T03:29:05.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-11T03:29:05.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-11T03:29:05.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-11T03:29:05.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-11T03:29:05.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-11T03:29:05.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-11T03:29:05.575 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-11T03:29:05.576 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-11T03:29:05.576 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-11T03:29:05.576 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-11T03:29:05.576 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-11T03:29:05.576 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-11T03:29:05.576 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-11T03:29:05.628 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:49 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:29:05.718 INFO:teuthology.orchestra.run.smithi175.stdout:Package ceph-base-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:05.759 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:29:05.760 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-11T03:29:05.761 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:05.828 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install cephadm 2024-09-11T03:29:06.385 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:50 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:29:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-11T03:29:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:06.487 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-11T03:29:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core-2:18.2.4-837.g7a853b17.el9.noarch 2024-09-11T03:29:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T03:29:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T03:29:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T03:29:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T03:29:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T03:29:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T03:29:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T03:29:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T03:29:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T03:29:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T03:29:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T03:29:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T03:29:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T03:29:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T03:29:06.489 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T03:29:06.489 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T03:29:06.489 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T03:29:06.489 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T03:29:06.489 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T03:29:06.489 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T03:29:06.489 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T03:29:06.489 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T03:29:06.489 INFO:teuthology.orchestra.run.smithi084.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T03:29:06.489 INFO:teuthology.orchestra.run.smithi084.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T03:29:06.489 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T03:29:06.489 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T03:29:06.490 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T03:29:06.490 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:06.490 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:06.517 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:29:06.518 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:06.518 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-09-11T03:29:06.518 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:06.518 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-11T03:29:06.518 INFO:teuthology.orchestra.run.smithi175.stdout: cephadm noarch 2:18.2.4-837.g7a853b17.el9 ceph-noarch 226 k 2024-09-11T03:29:06.519 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:06.519 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-11T03:29:06.519 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:06.519 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-09-11T03:29:06.519 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:06.519 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 226 k 2024-09-11T03:29:06.519 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 217 k 2024-09-11T03:29:06.519 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-11T03:29:06.739 INFO:teuthology.orchestra.run.smithi175.stdout:cephadm-18.2.4-837.g7a853b17.el9.noarch.rpm 1.0 MB/s | 226 kB 00:00 2024-09-11T03:29:06.740 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:06.740 INFO:teuthology.orchestra.run.smithi175.stdout:Total 1.0 MB/s | 226 kB 00:00 2024-09-11T03:29:06.740 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-11T03:29:06.746 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-11T03:29:06.746 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-11T03:29:06.748 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-11T03:29:06.748 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-11T03:29:06.823 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-11T03:29:06.874 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-2:18.2.4-837.g7a853b17.el9.x86_64 33/33 2024-09-11T03:29:06.874 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:18.2.4-837.g7a853b17.el9.x86_64 1/33 2024-09-11T03:29:06.874 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:18.2.4-837.g7a853b17.el9.x86_64 2/33 2024-09-11T03:29:06.874 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:18.2.4-837.g7a853b17.el9.x86_64 3/33 2024-09-11T03:29:06.874 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:18.2.4-837.g7a853b17.el9.x86_64 4/33 2024-09-11T03:29:06.874 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:18.2.4-837.g7a853b17.el9.x86_64 5/33 2024-09-11T03:29:06.874 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-837.g7a853b17.el9.n 6/33 2024-09-11T03:29:06.874 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-11T03:29:06.875 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-11T03:29:06.875 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-11T03:29:06.875 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-11T03:29:06.875 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-11T03:29:06.875 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-11T03:29:06.875 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-11T03:29:06.875 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-11T03:29:06.875 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-11T03:29:06.875 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-11T03:29:06.875 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-11T03:29:06.875 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-11T03:29:06.875 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-11T03:29:06.875 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-11T03:29:06.875 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-11T03:29:06.876 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-11T03:29:06.876 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-11T03:29:06.876 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-11T03:29:06.876 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-11T03:29:06.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-base 2024-09-11T03:29:06.878 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-11T03:29:06.878 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-11T03:29:06.878 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-11T03:29:06.878 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-11T03:29:06.878 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-11T03:29:06.878 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-11T03:29:06.878 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-11T03:29:07.293 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: cephadm-2:18.2.4-837.g7a853b17.el9.noarch 1/1 2024-09-11T03:29:07.386 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : cephadm-2:18.2.4-837.g7a853b17.el9.noarch 1/1 2024-09-11T03:29:07.441 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:48 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:29:07.531 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-base-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:07.573 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:29:07.574 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-11T03:29:07.574 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:07.640 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-11T03:29:07.640 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:07.640 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-11T03:29:07.640 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:07.640 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:07.640 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:07.640 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:18.2.4-837.g7a853b17.el9.noarch 2024-09-11T03:29:07.640 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:07.641 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:07.641 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T03:29:07.641 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T03:29:07.641 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T03:29:07.641 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T03:29:07.641 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T03:29:07.641 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T03:29:07.641 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T03:29:07.641 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T03:29:07.641 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T03:29:07.641 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T03:29:07.641 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T03:29:07.641 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T03:29:07.642 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T03:29:07.642 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T03:29:07.642 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T03:29:07.642 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T03:29:07.642 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T03:29:07.642 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T03:29:07.642 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T03:29:07.642 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T03:29:07.642 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T03:29:07.642 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T03:29:07.642 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T03:29:07.642 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T03:29:07.642 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T03:29:07.642 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T03:29:07.643 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T03:29:07.643 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:07.643 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:07.656 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install cephadm 2024-09-11T03:29:07.752 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: cephadm-2:18.2.4-837.g7a853b17.el9.noarch 1/1 2024-09-11T03:29:07.998 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : cephadm-2:18.2.4-837.g7a853b17.el9.noarch 1/1 2024-09-11T03:29:07.998 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:07.998 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-11T03:29:07.998 INFO:teuthology.orchestra.run.smithi175.stdout: cephadm-2:18.2.4-837.g7a853b17.el9.noarch 2024-09-11T03:29:07.998 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:07.998 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:08.027 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-base 2024-09-11T03:29:08.215 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:49 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:29:08.226 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T03:29:08.346 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:29:08.347 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:08.347 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-09-11T03:29:08.347 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:08.347 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-11T03:29:08.347 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm noarch 2:18.2.4-837.g7a853b17.el9 ceph-noarch 226 k 2024-09-11T03:29:08.347 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:08.347 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-11T03:29:08.347 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:08.348 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-09-11T03:29:08.348 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:08.348 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 226 k 2024-09-11T03:29:08.348 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 217 k 2024-09-11T03:29:08.348 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-11T03:29:08.594 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:47 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:29:08.595 INFO:teuthology.orchestra.run.smithi084.stdout:cephadm-18.2.4-837.g7a853b17.el9.noarch.rpm 915 kB/s | 226 kB 00:00 2024-09-11T03:29:08.596 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:08.597 INFO:teuthology.orchestra.run.smithi084.stdout:Total 909 kB/s | 226 kB 00:00 2024-09-11T03:29:08.597 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-11T03:29:08.601 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-11T03:29:08.601 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-11T03:29:08.604 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-11T03:29:08.604 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-11T03:29:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-11T03:29:08.684 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-base-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:08.726 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:29:08.726 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-11T03:29:08.727 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:08.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install cephadm 2024-09-11T03:29:08.809 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:52 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:29:08.943 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:29:08.944 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:08.944 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2024-09-11T03:29:08.944 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:08.944 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-11T03:29:08.944 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-837.g7a853b17.el9 ceph 143 k 2024-09-11T03:29:08.944 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:08.944 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-11T03:29:08.944 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:08.944 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-09-11T03:29:08.944 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:08.944 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 143 k 2024-09-11T03:29:08.944 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 395 k 2024-09-11T03:29:08.945 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-11T03:29:09.057 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephadm-2:18.2.4-837.g7a853b17.el9.noarch 1/1 2024-09-11T03:29:09.144 INFO:teuthology.orchestra.run.smithi175.stdout:ceph-immutable-object-cache-18.2.4-837.g7a853b1 716 kB/s | 143 kB 00:00 2024-09-11T03:29:09.145 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:09.145 INFO:teuthology.orchestra.run.smithi175.stdout:Total 710 kB/s | 143 kB 00:00 2024-09-11T03:29:09.145 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-11T03:29:09.147 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : cephadm-2:18.2.4-837.g7a853b17.el9.noarch 1/1 2024-09-11T03:29:09.153 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-11T03:29:09.153 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-11T03:29:09.211 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-11T03:29:09.212 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-11T03:29:09.356 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:48 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:29:09.360 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-11T03:29:09.420 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-837.g7a853b17.e 1/1 2024-09-11T03:29:09.445 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-837.g7a853b17.e 1/1 2024-09-11T03:29:09.445 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:29:09.445 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T03:29:09.446 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:09.486 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:29:09.487 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:09.487 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-11T03:29:09.487 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:09.487 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-11T03:29:09.487 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:18.2.4-837.g7a853b17.el9 ceph-noarch 226 k 2024-09-11T03:29:09.487 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:09.487 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-11T03:29:09.487 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:09.488 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-11T03:29:09.488 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:09.488 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 226 k 2024-09-11T03:29:09.488 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 217 k 2024-09-11T03:29:09.488 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-11T03:29:09.505 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephadm-2:18.2.4-837.g7a853b17.el9.noarch 1/1 2024-09-11T03:29:09.706 INFO:teuthology.orchestra.run.smithi121.stdout:cephadm-18.2.4-837.g7a853b17.el9.noarch.rpm 1.0 MB/s | 226 kB 00:00 2024-09-11T03:29:09.706 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:09.706 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.0 MB/s | 226 kB 00:00 2024-09-11T03:29:09.707 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-11T03:29:09.712 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-11T03:29:09.712 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-11T03:29:09.715 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-11T03:29:09.715 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-11T03:29:09.783 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : cephadm-2:18.2.4-837.g7a853b17.el9.noarch 1/1 2024-09-11T03:29:09.783 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:09.783 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-11T03:29:09.783 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm-2:18.2.4-837.g7a853b17.el9.noarch 2024-09-11T03:29:09.783 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:09.783 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:09.786 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-11T03:29:10.008 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T03:29:10.218 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:18.2.4-837.g7a853b17.el9.noarch 1/1 2024-09-11T03:29:10.308 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:18.2.4-837.g7a853b17.el9.noarch 1/1 2024-09-11T03:29:10.393 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-837.g7a853b17.e 1/1 2024-09-11T03:29:10.394 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:10.394 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-11T03:29:10.394 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-immutable-object-cache-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:10.394 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:10.394 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:10.576 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:51 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:29:10.639 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr 2024-09-11T03:29:10.659 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:18.2.4-837.g7a853b17.el9.noarch 1/1 2024-09-11T03:29:10.710 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:29:10.711 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:10.711 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-09-11T03:29:10.711 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:10.711 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-11T03:29:10.711 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-837.g7a853b17.el9 ceph 143 k 2024-09-11T03:29:10.711 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:10.711 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-11T03:29:10.711 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:10.711 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-09-11T03:29:10.712 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:10.712 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 143 k 2024-09-11T03:29:10.712 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 395 k 2024-09-11T03:29:10.712 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-11T03:29:10.907 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:18.2.4-837.g7a853b17.el9.noarch 1/1 2024-09-11T03:29:10.907 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:10.907 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-11T03:29:10.907 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:18.2.4-837.g7a853b17.el9.noarch 2024-09-11T03:29:10.907 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:10.907 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:10.925 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-immutable-object-cache-18.2.4-837.g7a853b1 670 kB/s | 143 kB 00:00 2024-09-11T03:29:10.925 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:10.926 INFO:teuthology.orchestra.run.smithi084.stdout:Total 666 kB/s | 143 kB 00:00 2024-09-11T03:29:10.926 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-11T03:29:10.933 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-11T03:29:10.933 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-11T03:29:10.990 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-11T03:29:10.990 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-11T03:29:11.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T03:29:11.146 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-11T03:29:11.213 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:55 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:29:11.226 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-837.g7a853b17.e 1/1 2024-09-11T03:29:11.251 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-837.g7a853b17.e 1/1 2024-09-11T03:29:11.251 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:29:11.251 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T03:29:11.251 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:11.307 INFO:teuthology.orchestra.run.smithi175.stdout:Package ceph-mgr-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:11.349 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:29:11.350 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-11T03:29:11.350 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:11.423 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-fuse 2024-09-11T03:29:11.651 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:50 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:29:11.784 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:29:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:11.785 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-09-11T03:29:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:11.785 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-11T03:29:11.786 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-837.g7a853b17.el9 ceph 143 k 2024-09-11T03:29:11.786 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-11T03:29:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-11T03:29:11.786 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 143 k 2024-09-11T03:29:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 395 k 2024-09-11T03:29:11.786 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-11T03:29:11.970 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-immutable-object-cache-18.2.4-837.g7a853b1 776 kB/s | 143 kB 00:00 2024-09-11T03:29:11.971 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:11.971 INFO:teuthology.orchestra.run.smithi121.stdout:Total 769 kB/s | 143 kB 00:00 2024-09-11T03:29:11.971 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-11T03:29:11.979 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-11T03:29:11.979 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-11T03:29:11.988 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:55 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:29:12.035 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-11T03:29:12.035 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-11T03:29:12.120 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:29:12.121 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:12.121 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-09-11T03:29:12.121 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:12.121 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-11T03:29:12.121 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-fuse x86_64 2:18.2.4-837.g7a853b17.el9 ceph 853 k 2024-09-11T03:29:12.121 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-09-11T03:29:12.121 INFO:teuthology.orchestra.run.smithi175.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T03:29:12.121 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:12.121 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-11T03:29:12.121 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:12.121 INFO:teuthology.orchestra.run.smithi175.stdout:Install 2 Packages 2024-09-11T03:29:12.121 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:12.122 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 933 k 2024-09-11T03:29:12.122 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 2.7 M 2024-09-11T03:29:12.122 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-11T03:29:12.173 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-11T03:29:12.200 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-837.g7a853b17.e 1/1 2024-09-11T03:29:12.200 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:12.200 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-11T03:29:12.201 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:12.201 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:12.201 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:12.263 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-837.g7a853b17.e 1/1 2024-09-11T03:29:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-837.g7a853b17.e 1/1 2024-09-11T03:29:12.289 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:29:12.289 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T03:29:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:12.347 INFO:teuthology.orchestra.run.smithi175.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 446 kB/s | 80 kB 00:00 2024-09-11T03:29:12.429 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr 2024-09-11T03:29:12.448 INFO:teuthology.orchestra.run.smithi175.stdout:(2/2): ceph-fuse-18.2.4-837.g7a853b17.el9.x86_6 3.0 MB/s | 853 kB 00:00 2024-09-11T03:29:12.448 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:12.448 INFO:teuthology.orchestra.run.smithi175.stdout:Total 2.8 MB/s | 933 kB 00:00 2024-09-11T03:29:12.460 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-11T03:29:12.482 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-11T03:29:12.482 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-11T03:29:12.577 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-11T03:29:12.578 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-11T03:29:12.761 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-11T03:29:12.896 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T03:29:12.961 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 2/2 2024-09-11T03:29:13.004 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:54 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:29:13.097 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-mgr-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:13.139 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:29:13.140 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-11T03:29:13.140 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:13.227 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-fuse 2024-09-11T03:29:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-837.g7a853b17.e 1/1 2024-09-11T03:29:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:13.286 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-11T03:29:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:13.286 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:13.515 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr 2024-09-11T03:29:13.713 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 2/2 2024-09-11T03:29:13.713 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 1/2 2024-09-11T03:29:13.789 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:54 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:29:13.919 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:29:13.920 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:13.921 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-09-11T03:29:13.921 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:13.921 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-11T03:29:13.921 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse x86_64 2:18.2.4-837.g7a853b17.el9 ceph 853 k 2024-09-11T03:29:13.921 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-09-11T03:29:13.921 INFO:teuthology.orchestra.run.smithi084.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T03:29:13.921 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:13.921 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-11T03:29:13.921 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:13.921 INFO:teuthology.orchestra.run.smithi084.stdout:Install 2 Packages 2024-09-11T03:29:13.921 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:13.922 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 933 k 2024-09-11T03:29:13.922 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 2.7 M 2024-09-11T03:29:13.922 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-11T03:29:13.979 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T03:29:13.979 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:13.979 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-11T03:29:13.979 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T03:29:13.979 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:13.979 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:14.079 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:53 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:29:14.172 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-mgr-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:14.213 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:29:14.214 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-11T03:29:14.214 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:14.226 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install librados-devel 2024-09-11T03:29:14.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-fuse 2024-09-11T03:29:14.328 INFO:teuthology.orchestra.run.smithi084.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 630 kB/s | 80 kB 00:00 2024-09-11T03:29:14.495 INFO:teuthology.orchestra.run.smithi084.stdout:(2/2): ceph-fuse-18.2.4-837.g7a853b17.el9.x86_6 2.8 MB/s | 853 kB 00:00 2024-09-11T03:29:14.495 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:14.495 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.6 MB/s | 933 kB 00:00 2024-09-11T03:29:14.507 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-11T03:29:14.527 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-11T03:29:14.527 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-11T03:29:14.608 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-11T03:29:14.608 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-11T03:29:14.789 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-11T03:29:14.797 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:58 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:29:14.837 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:53 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:29:14.926 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T03:29:14.934 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:29:14.934 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:14.934 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2024-09-11T03:29:14.934 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:14.935 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-11T03:29:14.935 INFO:teuthology.orchestra.run.smithi175.stdout: librados-devel x86_64 2:18.2.4-837.g7a853b17.el9 ceph 127 k 2024-09-11T03:29:14.935 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:14.935 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-11T03:29:14.935 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:14.935 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-09-11T03:29:14.935 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:14.936 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 127 k 2024-09-11T03:29:14.936 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 456 k 2024-09-11T03:29:14.936 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-11T03:29:14.970 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:29:14.970 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:14.970 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-11T03:29:14.970 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:14.970 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-11T03:29:14.970 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:18.2.4-837.g7a853b17.el9 ceph 853 k 2024-09-11T03:29:14.970 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-11T03:29:14.971 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T03:29:14.971 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:14.971 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-11T03:29:14.971 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:14.971 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-09-11T03:29:14.971 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:14.971 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 933 k 2024-09-11T03:29:14.972 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 2.7 M 2024-09-11T03:29:14.972 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-11T03:29:14.990 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 2/2 2024-09-11T03:29:15.120 INFO:teuthology.orchestra.run.smithi175.stdout:librados-devel-18.2.4-837.g7a853b17.el9.x86_64. 686 kB/s | 127 kB 00:00 2024-09-11T03:29:15.121 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:15.121 INFO:teuthology.orchestra.run.smithi175.stdout:Total 681 kB/s | 127 kB 00:00 2024-09-11T03:29:15.121 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-11T03:29:15.126 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 610 kB/s | 80 kB 00:00 2024-09-11T03:29:15.127 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-11T03:29:15.127 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-11T03:29:15.183 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-11T03:29:15.184 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-11T03:29:15.260 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): ceph-fuse-18.2.4-837.g7a853b17.el9.x86_6 3.2 MB/s | 853 kB 00:00 2024-09-11T03:29:15.260 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:15.261 INFO:teuthology.orchestra.run.smithi121.stdout:Total 3.1 MB/s | 933 kB 00:00 2024-09-11T03:29:15.272 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-11T03:29:15.294 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-11T03:29:15.294 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-11T03:29:15.326 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-11T03:29:15.374 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-11T03:29:15.374 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-11T03:29:15.425 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librados-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:15.560 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-11T03:29:15.631 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 2/2 2024-09-11T03:29:15.631 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 1/2 2024-09-11T03:29:15.708 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T03:29:15.773 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 2/2 2024-09-11T03:29:15.840 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:15.895 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T03:29:15.895 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:15.895 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-11T03:29:15.895 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T03:29:15.896 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:15.896 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:16.083 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:16.083 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:16.083 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-11T03:29:16.083 INFO:teuthology.orchestra.run.smithi175.stdout: librados-devel-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:16.083 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:16.083 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:16.176 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados-devel 2024-09-11T03:29:16.313 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install libcephfs2 2024-09-11T03:29:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 2/2 2024-09-11T03:29:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 1/2 2024-09-11T03:29:16.736 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T03:29:16.736 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:16.737 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-11T03:29:16.737 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T03:29:16.737 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:16.737 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:16.748 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:57 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:29:16.882 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:29:16.883 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:00 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:29:16.884 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-09-11T03:29:16.884 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:16.884 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-11T03:29:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel x86_64 2:18.2.4-837.g7a853b17.el9 ceph 127 k 2024-09-11T03:29:16.884 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:16.884 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-11T03:29:16.884 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:16.884 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-09-11T03:29:16.885 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:16.885 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 127 k 2024-09-11T03:29:16.885 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 456 k 2024-09-11T03:29:16.885 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-11T03:29:16.971 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados-devel 2024-09-11T03:29:16.974 INFO:teuthology.orchestra.run.smithi175.stdout:Package libcephfs2-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:17.017 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:29:17.018 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-11T03:29:17.018 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:17.090 INFO:teuthology.orchestra.run.smithi084.stdout:librados-devel-18.2.4-837.g7a853b17.el9.x86_64. 616 kB/s | 127 kB 00:00 2024-09-11T03:29:17.091 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:17.091 INFO:teuthology.orchestra.run.smithi084.stdout:Total 610 kB/s | 127 kB 00:00 2024-09-11T03:29:17.091 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-11T03:29:17.092 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install libcephfs-devel 2024-09-11T03:29:17.096 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-11T03:29:17.096 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-11T03:29:17.153 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-11T03:29:17.153 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-11T03:29:17.302 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-11T03:29:17.401 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librados-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:17.534 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:56 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:29:17.645 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:01 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:29:17.668 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:29:17.669 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:17.669 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-09-11T03:29:17.669 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:17.669 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-11T03:29:17.669 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:18.2.4-837.g7a853b17.el9 ceph 127 k 2024-09-11T03:29:17.669 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:17.669 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-11T03:29:17.669 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:17.670 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-11T03:29:17.670 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:17.670 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 127 k 2024-09-11T03:29:17.670 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 456 k 2024-09-11T03:29:17.670 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-11T03:29:17.776 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:29:17.776 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:17.776 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2024-09-11T03:29:17.776 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:17.777 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-11T03:29:17.777 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs-devel x86_64 2:18.2.4-837.g7a853b17.el9 ceph 31 k 2024-09-11T03:29:17.777 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:17.777 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-11T03:29:17.777 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:17.777 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-09-11T03:29:17.777 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:17.777 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 31 k 2024-09-11T03:29:17.777 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 139 k 2024-09-11T03:29:17.777 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-11T03:29:17.791 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:17.848 INFO:teuthology.orchestra.run.smithi121.stdout:librados-devel-18.2.4-837.g7a853b17.el9.x86_64. 714 kB/s | 127 kB 00:00 2024-09-11T03:29:17.848 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:17.849 INFO:teuthology.orchestra.run.smithi121.stdout:Total 708 kB/s | 127 kB 00:00 2024-09-11T03:29:17.849 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-11T03:29:17.854 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-11T03:29:17.854 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-11T03:29:17.911 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-11T03:29:17.911 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-11T03:29:17.932 INFO:teuthology.orchestra.run.smithi175.stdout:libcephfs-devel-18.2.4-837.g7a853b17.el9.x86_64 200 kB/s | 31 kB 00:00 2024-09-11T03:29:17.933 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:17.933 INFO:teuthology.orchestra.run.smithi175.stdout:Total 199 kB/s | 31 kB 00:00 2024-09-11T03:29:17.933 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-11T03:29:17.936 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-11T03:29:17.936 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-11T03:29:17.955 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-11T03:29:17.955 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-11T03:29:18.018 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:18.018 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:18.018 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-11T03:29:18.018 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:18.018 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:18.018 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:18.039 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-11T03:29:18.055 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-11T03:29:18.143 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:18.160 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephfs-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:18.257 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs2 2024-09-11T03:29:18.541 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:18.550 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:18.780 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephfs-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:18.780 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:18.780 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-11T03:29:18.781 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs-devel-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:18.781 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:18.781 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:18.789 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:18.790 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:18.790 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-11T03:29:18.790 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:18.790 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:18.790 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:18.830 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:59 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:29:18.921 INFO:teuthology.orchestra.run.smithi084.stdout:Package libcephfs2-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:18.964 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:29:18.965 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-11T03:29:18.965 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:18.970 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install librados2 2024-09-11T03:29:19.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs2 2024-09-11T03:29:19.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs-devel 2024-09-11T03:29:19.528 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:03 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:29:19.607 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:58 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:29:19.619 INFO:teuthology.orchestra.run.smithi175.stdout:Package librados2-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:19.621 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:00 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:29:19.660 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:29:19.661 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-11T03:29:19.661 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:19.699 INFO:teuthology.orchestra.run.smithi121.stdout:Package libcephfs2-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:19.732 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install librbd1 2024-09-11T03:29:19.741 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:29:19.742 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-11T03:29:19.742 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:19.753 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:29:19.754 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:19.754 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-09-11T03:29:19.754 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:19.754 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-11T03:29:19.754 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel x86_64 2:18.2.4-837.g7a853b17.el9 ceph 31 k 2024-09-11T03:29:19.754 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:19.754 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-11T03:29:19.754 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:19.754 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-09-11T03:29:19.754 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:19.755 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 31 k 2024-09-11T03:29:19.755 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 139 k 2024-09-11T03:29:19.755 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-11T03:29:19.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs-devel 2024-09-11T03:29:19.910 INFO:teuthology.orchestra.run.smithi084.stdout:libcephfs-devel-18.2.4-837.g7a853b17.el9.x86_64 201 kB/s | 31 kB 00:00 2024-09-11T03:29:19.910 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:19.911 INFO:teuthology.orchestra.run.smithi084.stdout:Total 199 kB/s | 31 kB 00:00 2024-09-11T03:29:19.911 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-11T03:29:19.914 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-11T03:29:19.914 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-11T03:29:19.930 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-11T03:29:19.930 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-11T03:29:20.016 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-11T03:29:20.120 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:20.285 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:04 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:29:20.373 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:59 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:29:20.375 INFO:teuthology.orchestra.run.smithi175.stdout:Package librbd1-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:20.416 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:29:20.417 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-11T03:29:20.418 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:20.440 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:20.500 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-rados 2024-09-11T03:29:20.504 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:29:20.505 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:20.505 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-09-11T03:29:20.505 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:20.506 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-11T03:29:20.506 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:18.2.4-837.g7a853b17.el9 ceph 31 k 2024-09-11T03:29:20.506 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:20.506 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-11T03:29:20.506 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:20.506 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-11T03:29:20.507 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:20.507 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 31 k 2024-09-11T03:29:20.507 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 139 k 2024-09-11T03:29:20.507 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-11T03:29:20.660 INFO:teuthology.orchestra.run.smithi121.stdout:libcephfs-devel-18.2.4-837.g7a853b17.el9.x86_64 202 kB/s | 31 kB 00:00 2024-09-11T03:29:20.660 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:20.660 INFO:teuthology.orchestra.run.smithi121.stdout:Total 200 kB/s | 31 kB 00:00 2024-09-11T03:29:20.661 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-11T03:29:20.664 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-11T03:29:20.664 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-11T03:29:20.683 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:20.683 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:20.683 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-11T03:29:20.683 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:20.683 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:20.683 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:20.686 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-11T03:29:20.686 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-11T03:29:20.769 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-11T03:29:20.864 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados2 2024-09-11T03:29:20.879 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:21.056 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:05 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:29:21.145 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-rados-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:21.187 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:29:21.188 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-11T03:29:21.188 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:21.251 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:21.263 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-rgw 2024-09-11T03:29:21.421 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:02 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:29:21.457 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:21.457 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:21.458 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-11T03:29:21.458 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:21.458 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:21.458 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:21.511 INFO:teuthology.orchestra.run.smithi084.stdout:Package librados2-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:21.552 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:29:21.553 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-11T03:29:21.553 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:21.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados2 2024-09-11T03:29:21.635 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librbd1 2024-09-11T03:29:21.819 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:05 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:29:21.909 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-rgw-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:21.950 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:29:21.951 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-11T03:29:21.951 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:22.022 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-cephfs 2024-09-11T03:29:22.188 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:01 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:29:22.189 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:03 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:29:22.277 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:22.279 INFO:teuthology.orchestra.run.smithi084.stdout:Package librbd1-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:22.318 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:29:22.319 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-11T03:29:22.320 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:22.321 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:29:22.322 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-11T03:29:22.322 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:22.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librbd1 2024-09-11T03:29:22.403 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rados 2024-09-11T03:29:22.575 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:06 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:29:22.664 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-cephfs-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:22.705 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:29:22.707 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-11T03:29:22.707 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:22.774 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-rbd 2024-09-11T03:29:22.931 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:01 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:29:22.954 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:03 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:29:23.022 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:23.045 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rados-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:23.063 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:29:23.064 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-11T03:29:23.065 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:23.088 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:29:23.089 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-11T03:29:23.089 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:23.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rados 2024-09-11T03:29:23.169 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rgw 2024-09-11T03:29:23.333 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:07 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:29:23.424 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-rbd-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:23.465 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:29:23.466 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-11T03:29:23.466 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:23.538 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install rbd-fuse 2024-09-11T03:29:23.685 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:02 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:29:23.727 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:04 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:29:23.774 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rados-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:23.815 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:29:23.815 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-11T03:29:23.815 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:23.816 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rgw-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:23.857 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:29:23.858 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-11T03:29:23.858 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:23.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rgw 2024-09-11T03:29:23.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-cephfs 2024-09-11T03:29:24.097 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:08 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:29:24.229 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:29:24.230 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:24.230 INFO:teuthology.orchestra.run.smithi175.stdout: Package Architecture Version Repository Size 2024-09-11T03:29:24.230 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:24.230 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-11T03:29:24.230 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-fuse x86_64 2:18.2.4-837.g7a853b17.el9 ceph 87 k 2024-09-11T03:29:24.230 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:24.230 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-11T03:29:24.230 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:24.230 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-09-11T03:29:24.230 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:24.231 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 87 k 2024-09-11T03:29:24.231 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 227 k 2024-09-11T03:29:24.231 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-11T03:29:24.426 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:03 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:29:24.426 INFO:teuthology.orchestra.run.smithi175.stdout:rbd-fuse-18.2.4-837.g7a853b17.el9.x86_64.rpm 444 kB/s | 87 kB 00:00 2024-09-11T03:29:24.427 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:24.427 INFO:teuthology.orchestra.run.smithi175.stdout:Total 441 kB/s | 87 kB 00:00 2024-09-11T03:29:24.427 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-11T03:29:24.434 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-11T03:29:24.434 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-11T03:29:24.488 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:05 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:29:24.490 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-11T03:29:24.490 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-11T03:29:24.514 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rgw-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:24.555 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:29:24.556 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-11T03:29:24.556 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:24.577 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-cephfs-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:24.618 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:29:24.619 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-11T03:29:24.619 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:24.621 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-cephfs 2024-09-11T03:29:24.635 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-11T03:29:24.702 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rbd 2024-09-11T03:29:24.729 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:25.104 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: rbd-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:25.173 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:04 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:29:25.261 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:06 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:29:25.262 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-cephfs-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:25.303 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-11T03:29:25.305 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:25.351 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rbd-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:25.359 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:25.359 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:25.359 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-11T03:29:25.359 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:25.359 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:25.360 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:25.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rbd 2024-09-11T03:29:25.392 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:29:25.393 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-11T03:29:25.393 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:25.469 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-fuse 2024-09-11T03:29:25.594 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install rbd-mirror 2024-09-11T03:29:25.925 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:04 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:29:26.014 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rbd-2:18.2.4-837.g7a853b17.el9.x86_64 is already installed. 2024-09-11T03:29:26.017 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:07 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:29:26.055 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:29:26.056 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-11T03:29:26.056 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:26.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-fuse 2024-09-11T03:29:26.147 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:29:26.148 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:26.148 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-09-11T03:29:26.148 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:26.148 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-11T03:29:26.148 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse x86_64 2:18.2.4-837.g7a853b17.el9 ceph 87 k 2024-09-11T03:29:26.148 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:26.148 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-11T03:29:26.149 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:26.149 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-09-11T03:29:26.149 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:26.149 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 87 k 2024-09-11T03:29:26.149 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 227 k 2024-09-11T03:29:26.149 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-11T03:29:26.163 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:10 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:29:26.298 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:29:26.298 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:26.299 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2024-09-11T03:29:26.299 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:26.299 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-11T03:29:26.299 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-mirror x86_64 2:18.2.4-837.g7a853b17.el9 ceph 3.0 M 2024-09-11T03:29:26.299 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:26.299 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-11T03:29:26.299 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:26.299 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-09-11T03:29:26.299 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:26.300 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 3.0 M 2024-09-11T03:29:26.300 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 12 M 2024-09-11T03:29:26.300 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-11T03:29:26.382 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-fuse-18.2.4-837.g7a853b17.el9.x86_64.rpm 373 kB/s | 87 kB 00:00 2024-09-11T03:29:26.382 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:26.382 INFO:teuthology.orchestra.run.smithi084.stdout:Total 371 kB/s | 87 kB 00:00 2024-09-11T03:29:26.382 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-11T03:29:26.389 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-11T03:29:26.389 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-11T03:29:26.445 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-11T03:29:26.445 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-11T03:29:26.583 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-11T03:29:26.649 INFO:teuthology.orchestra.run.smithi175.stdout:rbd-mirror-18.2.4-837.g7a853b17.el9.x86_64.rpm 8.6 MB/s | 3.0 MB 00:00 2024-09-11T03:29:26.649 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:26.649 INFO:teuthology.orchestra.run.smithi175.stdout:Total 8.6 MB/s | 3.0 MB 00:00 2024-09-11T03:29:26.649 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-11T03:29:26.656 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-11T03:29:26.656 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-11T03:29:26.673 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:26.673 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:05 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:29:26.731 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-11T03:29:26.731 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-11T03:29:26.803 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:29:26.804 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-09-11T03:29:26.804 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:26.804 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-11T03:29:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:18.2.4-837.g7a853b17.el9 ceph 87 k 2024-09-11T03:29:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:26.804 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-11T03:29:26.805 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:26.805 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-11T03:29:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:26.805 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 87 k 2024-09-11T03:29:26.805 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 227 k 2024-09-11T03:29:26.805 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-11T03:29:26.996 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-fuse-18.2.4-837.g7a853b17.el9.x86_64.rpm 455 kB/s | 87 kB 00:00 2024-09-11T03:29:26.996 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:26.997 INFO:teuthology.orchestra.run.smithi121.stdout:Total 452 kB/s | 87 kB 00:00 2024-09-11T03:29:26.997 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-11T03:29:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-11T03:29:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-11T03:29:27.046 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:27.060 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-11T03:29:27.061 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-11T03:29:27.148 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-11T03:29:27.205 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-11T03:29:27.205 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-mirror-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:27.230 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: rbd-mirror-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:27.230 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:29:27.230 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T03:29:27.230 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T03:29:27.231 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T03:29:27.231 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:27.250 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:27.250 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:27.250 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-11T03:29:27.250 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:27.250 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:27.250 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:27.469 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-mirror 2024-09-11T03:29:27.620 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:27.861 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-11T03:29:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:27.861 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:28.043 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:09 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:29:28.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-mirror 2024-09-11T03:29:28.177 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:29:28.178 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:28.178 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-09-11T03:29:28.178 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:28.179 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-11T03:29:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror x86_64 2:18.2.4-837.g7a853b17.el9 ceph 3.0 M 2024-09-11T03:29:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:28.179 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-11T03:29:28.179 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:28.179 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-09-11T03:29:28.179 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:28.179 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 3.0 M 2024-09-11T03:29:28.179 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 12 M 2024-09-11T03:29:28.179 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-11T03:29:28.390 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-mirror-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:28.391 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:28.391 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-11T03:29:28.391 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-mirror-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:28.391 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:28.391 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:28.514 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-mirror-18.2.4-837.g7a853b17.el9.x86_64.rpm 8.9 MB/s | 3.0 MB 00:00 2024-09-11T03:29:28.514 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:28.515 INFO:teuthology.orchestra.run.smithi084.stdout:Total 8.9 MB/s | 3.0 MB 00:00 2024-09-11T03:29:28.515 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-11T03:29:28.524 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-11T03:29:28.524 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-11T03:29:28.597 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-11T03:29:28.598 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-11T03:29:28.630 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:07 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:29:28.636 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install rbd-nbd 2024-09-11T03:29:28.765 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:29:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:28.766 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-09-11T03:29:28.766 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-11T03:29:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:18.2.4-837.g7a853b17.el9 ceph 3.0 M 2024-09-11T03:29:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-11T03:29:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-11T03:29:28.767 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 3.0 M 2024-09-11T03:29:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 12 M 2024-09-11T03:29:28.767 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-11T03:29:29.002 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-11T03:29:29.065 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-mirror-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:29.089 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-mirror-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:29.089 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:29:29.089 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T03:29:29.089 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T03:29:29.089 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T03:29:29.089 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:29.091 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-mirror-18.2.4-837.g7a853b17.el9.x86_64.rpm 9.2 MB/s | 3.0 MB 00:00 2024-09-11T03:29:29.092 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:29.092 INFO:teuthology.orchestra.run.smithi121.stdout:Total 9.2 MB/s | 3.0 MB 00:00 2024-09-11T03:29:29.092 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-11T03:29:29.101 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-11T03:29:29.101 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-11T03:29:29.175 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-11T03:29:29.175 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-11T03:29:29.217 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:13 ago on Wed 11 Sep 2024 03:28:16 AM UTC. 2024-09-11T03:29:29.357 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-11T03:29:29.358 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:29.358 INFO:teuthology.orchestra.run.smithi175.stdout: Package Architecture Version Repository Size 2024-09-11T03:29:29.358 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:29.358 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-11T03:29:29.358 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-nbd x86_64 2:18.2.4-837.g7a853b17.el9 ceph 172 k 2024-09-11T03:29:29.358 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:29.358 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-11T03:29:29.358 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-11T03:29:29.358 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-09-11T03:29:29.358 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:29.359 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 172 k 2024-09-11T03:29:29.359 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 494 k 2024-09-11T03:29:29.359 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-11T03:29:29.579 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-11T03:29:29.591 INFO:teuthology.orchestra.run.smithi175.stdout:rbd-nbd-18.2.4-837.g7a853b17.el9.x86_64.rpm 740 kB/s | 172 kB 00:00 2024-09-11T03:29:29.592 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:29.592 INFO:teuthology.orchestra.run.smithi175.stdout:Total 735 kB/s | 172 kB 00:00 2024-09-11T03:29:29.592 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-11T03:29:29.600 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-11T03:29:29.600 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-11T03:29:29.658 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:29.661 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-11T03:29:29.661 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-11T03:29:29.684 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T03:29:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T03:29:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T03:29:29.684 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T03:29:29.684 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:29.807 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-11T03:29:29.896 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-nbd-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:30.180 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-mirror-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-11T03:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:30.181 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:30.349 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: rbd-nbd-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:30.414 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-nbd 2024-09-11T03:29:30.605 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-nbd-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:30.606 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:30.606 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-11T03:29:30.606 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-nbd-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:30.606 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:29:30.606 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-11T03:29:30.741 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:30.741 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:30.742 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-11T03:29:30.742 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:30.742 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:30.742 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:30.853 DEBUG:teuthology.parallel:result is None 2024-09-11T03:29:30.960 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-nbd 2024-09-11T03:29:30.978 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:11 ago on Wed 11 Sep 2024 03:28:19 AM UTC. 2024-09-11T03:29:31.114 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-11T03:29:31.115 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:31.115 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-09-11T03:29:31.115 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:31.115 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-11T03:29:31.115 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd x86_64 2:18.2.4-837.g7a853b17.el9 ceph 172 k 2024-09-11T03:29:31.115 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:31.115 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-11T03:29:31.115 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-11T03:29:31.115 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-09-11T03:29:31.115 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:31.115 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 172 k 2024-09-11T03:29:31.116 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 494 k 2024-09-11T03:29:31.116 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-11T03:29:31.325 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-nbd-18.2.4-837.g7a853b17.el9.x86_64.rpm 821 kB/s | 172 kB 00:00 2024-09-11T03:29:31.325 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:31.326 INFO:teuthology.orchestra.run.smithi084.stdout:Total 816 kB/s | 172 kB 00:00 2024-09-11T03:29:31.326 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-11T03:29:31.334 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-11T03:29:31.334 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-11T03:29:31.391 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-11T03:29:31.391 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-11T03:29:31.532 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:10 ago on Wed 11 Sep 2024 03:28:21 AM UTC. 2024-09-11T03:29:31.537 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-11T03:29:31.662 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-nbd-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:31.663 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-11T03:29:31.664 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:31.664 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-09-11T03:29:31.664 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:31.664 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-11T03:29:31.664 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:18.2.4-837.g7a853b17.el9 ceph 172 k 2024-09-11T03:29:31.664 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:31.664 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-11T03:29:31.664 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-11T03:29:31.664 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-11T03:29:31.665 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:31.665 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 172 k 2024-09-11T03:29:31.665 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 494 k 2024-09-11T03:29:31.665 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-11T03:29:31.878 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-nbd-18.2.4-837.g7a853b17.el9.x86_64.rpm 808 kB/s | 172 kB 00:00 2024-09-11T03:29:31.878 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:29:31.878 INFO:teuthology.orchestra.run.smithi121.stdout:Total 803 kB/s | 172 kB 00:00 2024-09-11T03:29:31.878 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-11T03:29:31.886 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-11T03:29:31.886 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-11T03:29:31.942 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-11T03:29:31.943 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-11T03:29:32.025 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-nbd-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:32.090 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-11T03:29:32.181 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:32.266 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-nbd-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:32.266 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:32.267 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-11T03:29:32.267 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:32.267 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:29:32.267 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-11T03:29:32.524 DEBUG:teuthology.parallel:result is None 2024-09-11T03:29:32.569 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-nbd-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:32.821 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:18.2.4-837.g7a853b17.el9.x86_64 1/1 2024-09-11T03:29:32.821 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:32.822 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-11T03:29:32.822 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:18.2.4-837.g7a853b17.el9.x86_64 2024-09-11T03:29:32.822 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:29:32.822 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-11T03:29:33.047 DEBUG:teuthology.parallel:result is None 2024-09-11T03:29:33.048 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T03:29:33.048 INFO:teuthology.packaging:ref: None 2024-09-11T03:29:33.048 INFO:teuthology.packaging:tag: None 2024-09-11T03:29:33.048 INFO:teuthology.packaging:branch: reef 2024-09-11T03:29:33.048 INFO:teuthology.packaging:sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T03:29:33.048 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-11T03:29:33.339 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T03:29:33.361 INFO:teuthology.orchestra.run.smithi084.stdout:18.2.4-837.g7a853b17.el9 2024-09-11T03:29:33.362 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-837.g7a853b17.el9 2024-09-11T03:29:33.362 INFO:teuthology.task.install:The correct ceph version 18.2.4-837.g7a853b17 is installed. 2024-09-11T03:29:33.363 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T03:29:33.363 INFO:teuthology.packaging:ref: None 2024-09-11T03:29:33.363 INFO:teuthology.packaging:tag: None 2024-09-11T03:29:33.363 INFO:teuthology.packaging:branch: reef 2024-09-11T03:29:33.363 INFO:teuthology.packaging:sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T03:29:33.364 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-11T03:29:33.629 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T03:29:33.650 INFO:teuthology.orchestra.run.smithi121.stdout:18.2.4-837.g7a853b17.el9 2024-09-11T03:29:33.650 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-837.g7a853b17.el9 2024-09-11T03:29:33.650 INFO:teuthology.task.install:The correct ceph version 18.2.4-837.g7a853b17 is installed. 2024-09-11T03:29:33.652 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T03:29:33.652 INFO:teuthology.packaging:ref: None 2024-09-11T03:29:33.652 INFO:teuthology.packaging:tag: None 2024-09-11T03:29:33.652 INFO:teuthology.packaging:branch: reef 2024-09-11T03:29:33.652 INFO:teuthology.packaging:sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T03:29:33.652 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-11T03:29:33.919 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T03:29:33.944 INFO:teuthology.orchestra.run.smithi175.stdout:18.2.4-837.g7a853b17.el9 2024-09-11T03:29:33.945 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-837.g7a853b17.el9 2024-09-11T03:29:33.945 INFO:teuthology.task.install:The correct ceph version 18.2.4-837.g7a853b17 is installed. 2024-09-11T03:29:33.946 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T03:29:33.947 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-11T03:29:33.947 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T03:29:33.977 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-11T03:29:33.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T03:29:34.003 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-11T03:29:34.003 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T03:29:34.033 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T03:29:34.034 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-11T03:29:34.034 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T03:29:34.061 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T03:29:34.129 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-11T03:29:34.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T03:29:34.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T03:29:34.218 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-11T03:29:34.219 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T03:29:34.249 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T03:29:34.317 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T03:29:34.318 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-11T03:29:34.318 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T03:29:34.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T03:29:34.414 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-11T03:29:34.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T03:29:34.441 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T03:29:34.506 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-11T03:29:34.506 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T03:29:34.535 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T03:29:34.603 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T03:29:34.604 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-11T03:29:34.604 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T03:29:34.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T03:29:34.697 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-11T03:29:34.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T03:29:34.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T03:29:34.787 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-11T03:29:34.787 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T03:29:34.815 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T03:29:34.882 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-11T03:29:34.978 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'filestore queue throttle high multiple': 2, 'filestore queue throttle max multiple': 10, 'journal throttle high multiple': 2, 'journal throttle max multiple': 10, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd heartbeat use min delay socket': True, 'osd max backfills': 9, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'} 2024-09-11T03:29:34.978 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T03:29:34.978 INFO:tasks.cephadm:Cluster fsid is 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:29:34.979 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-11T03:29:34.979 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.84', 'mon.b': '172.21.15.121', 'mon.c': '172.21.15.175'} 2024-09-11T03:29:34.979 INFO:tasks.cephadm:First mon is mon.a on smithi084 2024-09-11T03:29:34.979 INFO:tasks.cephadm:First mgr is y 2024-09-11T03:29:34.979 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-11T03:29:34.979 DEBUG:teuthology.orchestra.run.smithi084:> sudo hostname $(hostname -s) 2024-09-11T03:29:35.007 DEBUG:teuthology.orchestra.run.smithi121:> sudo hostname $(hostname -s) 2024-09-11T03:29:35.032 DEBUG:teuthology.orchestra.run.smithi175:> sudo hostname $(hostname -s) 2024-09-11T03:29:35.061 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-11T03:29:35.061 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T03:29:35.215 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/82938/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-4854-g8f219c47', 'node_name': '172.21.2.8+braggi08', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-4854.g8f219c47'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-10 09:27:13.215997', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-hemanth-testing-2024-09-10-0723', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-09-11T03:29:35.367 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-hemanth-testing-2024-09-10-0723, sha1 8f219c476cb4f26fc5719c661c0323719a45cb30 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T03:29:35.369 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/x86_64/flavors/default/cephadm 2024-09-11T03:29:35.369 INFO:tasks.cephadm:Downloading cephadm from url: https://3.chacra.ceph.com/binaries/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/x86_64/flavors/default/cephadm 2024-09-11T03:29:35.369 DEBUG:teuthology.orchestra.run.smithi084:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-11T03:29:35.754 INFO:teuthology.orchestra.run.smithi084.stdout:-rw-r--r--. 1 ubuntu ubuntu 803561 Sep 11 03:29 /home/ubuntu/cephtest/cephadm 2024-09-11T03:29:35.755 DEBUG:teuthology.orchestra.run.smithi121:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-11T03:29:36.133 INFO:teuthology.orchestra.run.smithi121.stdout:-rw-r--r--. 1 ubuntu ubuntu 803561 Sep 11 03:29 /home/ubuntu/cephtest/cephadm 2024-09-11T03:29:36.133 DEBUG:teuthology.orchestra.run.smithi175:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-11T03:29:36.511 INFO:teuthology.orchestra.run.smithi175.stdout:-rw-r--r--. 1 ubuntu ubuntu 803561 Sep 11 03:29 /home/ubuntu/cephtest/cephadm 2024-09-11T03:29:36.511 DEBUG:teuthology.orchestra.run.smithi084:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-11T03:29:36.531 DEBUG:teuthology.orchestra.run.smithi121:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-11T03:29:36.550 DEBUG:teuthology.orchestra.run.smithi175:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-11T03:29:36.583 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 on all hosts... 2024-09-11T03:29:36.583 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 pull 2024-09-11T03:29:36.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 pull 2024-09-11T03:29:36.593 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 pull 2024-09-11T03:29:38.170 INFO:teuthology.orchestra.run.smithi175.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30... 2024-09-11T03:29:38.198 INFO:teuthology.orchestra.run.smithi084.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30... 2024-09-11T03:29:38.251 INFO:teuthology.orchestra.run.smithi121.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30... 2024-09-11T03:30:10.461 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:30:10.462 INFO:teuthology.orchestra.run.smithi084.stdout: "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", 2024-09-11T03:30:10.462 INFO:teuthology.orchestra.run.smithi084.stdout: "image_id": "589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480", 2024-09-11T03:30:10.462 INFO:teuthology.orchestra.run.smithi084.stdout: "repo_digests": [ 2024-09-11T03:30:10.462 INFO:teuthology.orchestra.run.smithi084.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e" 2024-09-11T03:30:10.462 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:30:10.462 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:30:10.560 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:30:10.560 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", 2024-09-11T03:30:10.561 INFO:teuthology.orchestra.run.smithi121.stdout: "image_id": "589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480", 2024-09-11T03:30:10.561 INFO:teuthology.orchestra.run.smithi121.stdout: "repo_digests": [ 2024-09-11T03:30:10.561 INFO:teuthology.orchestra.run.smithi121.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e" 2024-09-11T03:30:10.561 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:30:10.561 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:30:21.566 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:30:21.567 INFO:teuthology.orchestra.run.smithi175.stdout: "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", 2024-09-11T03:30:21.567 INFO:teuthology.orchestra.run.smithi175.stdout: "image_id": "589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480", 2024-09-11T03:30:21.567 INFO:teuthology.orchestra.run.smithi175.stdout: "repo_digests": [ 2024-09-11T03:30:21.567 INFO:teuthology.orchestra.run.smithi175.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e" 2024-09-11T03:30:21.567 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:30:21.567 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:30:21.593 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /etc/ceph 2024-09-11T03:30:21.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph 2024-09-11T03:30:21.661 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /etc/ceph 2024-09-11T03:30:21.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 777 /etc/ceph 2024-09-11T03:30:21.720 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /etc/ceph 2024-09-11T03:30:21.749 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 777 /etc/ceph 2024-09-11T03:30:21.776 INFO:tasks.cephadm:Writing seed config... 2024-09-11T03:30:21.777 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-09-11T03:30:21.777 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-11T03:30:21.777 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-11T03:30:21.777 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-11T03:30:21.777 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-09-11T03:30:21.777 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-09-11T03:30:21.778 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-09-11T03:30:21.778 INFO:tasks.cephadm: override: [global] ms inject socket failures = 5000 2024-09-11T03:30:21.778 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-09-11T03:30:21.778 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 3 2024-09-11T03:30:21.778 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-11T03:30:21.778 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-11T03:30:21.778 INFO:tasks.cephadm: override: [osd] bluestore zero block detection = True 2024-09-11T03:30:21.778 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-11T03:30:21.778 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-11T03:30:21.778 INFO:tasks.cephadm: override: [osd] filestore queue throttle high multiple = 2 2024-09-11T03:30:21.778 INFO:tasks.cephadm: override: [osd] filestore queue throttle max multiple = 10 2024-09-11T03:30:21.778 INFO:tasks.cephadm: override: [osd] journal throttle high multiple = 2 2024-09-11T03:30:21.778 INFO:tasks.cephadm: override: [osd] journal throttle max multiple = 10 2024-09-11T03:30:21.779 INFO:tasks.cephadm: override: [osd] osd backoff on peering = True 2024-09-11T03:30:21.779 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2024-09-11T03:30:21.779 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-09-11T03:30:21.779 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-09-11T03:30:21.779 INFO:tasks.cephadm: override: [osd] osd heartbeat use min delay socket = True 2024-09-11T03:30:21.779 INFO:tasks.cephadm: override: [osd] osd max backfills = 9 2024-09-11T03:30:21.779 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-09-11T03:30:21.779 INFO:tasks.cephadm: override: [osd] osd mclock override recovery settings = True 2024-09-11T03:30:21.779 INFO:tasks.cephadm: override: [osd] osd mclock profile = high_recovery_ops 2024-09-11T03:30:21.779 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-09-11T03:30:21.779 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-09-11T03:30:21.780 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-11T03:30:21.780 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-11T03:30:21.780 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-11T03:30:21.780 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-11T03:30:21.781 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-11T03:30:21.800 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 10fe7946-6fee-11ef-bcea-c7b262605968 mon client directed command retry = 5 mon election default strategy = 3 ms inject socket failures = 5000 osd_pool_default_min_size = 2 osd_pool_default_size = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore zero block detection = True debug ms = 1 debug osd = 20 filestore queue throttle high multiple = 2 filestore queue throttle max multiple = 10 journal throttle high multiple = 2 journal throttle max multiple = 10 osd backoff on peering = True osd blocked scrub grace period = 3600 osd heartbeat use min delay socket = True osd max backfills = 9 osd max markdown count = 1000 osd mclock override recovery settings = True osd scrub min interval = 60 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon scrub interval = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-11T03:30:21.801 DEBUG:teuthology.orchestra.run.smithi084:mon.a> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mon.a.service 2024-09-11T03:30:21.844 DEBUG:teuthology.orchestra.run.smithi084:mgr.y> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mgr.y.service 2024-09-11T03:30:21.887 INFO:tasks.cephadm:Bootstrapping... 2024-09-11T03:30:21.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 -v bootstrap --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.84 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-11T03:30:22.094 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-11T03:30:22.094 INFO:teuthology.orchestra.run.smithi084.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30', '-v', 'bootstrap', '--fsid', '10fe7946-6fee-11ef-bcea-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.84', '--skip-admin-label'] 2024-09-11T03:30:22.094 INFO:teuthology.orchestra.run.smithi084.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-11T03:30:22.095 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying podman|docker is present... 2024-09-11T03:30:22.118 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stdout 5.2.2 2024-09-11T03:30:22.118 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying lvm2 is present... 2024-09-11T03:30:22.118 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying time synchronization is in place... 2024-09-11T03:30:22.126 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-11T03:30:22.126 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-11T03:30:22.132 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-11T03:30:22.132 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-09-11T03:30:22.139 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout enabled 2024-09-11T03:30:22.145 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout active 2024-09-11T03:30:22.145 INFO:teuthology.orchestra.run.smithi084.stdout:Unit chronyd.service is enabled and running 2024-09-11T03:30:22.145 INFO:teuthology.orchestra.run.smithi084.stdout:Repeating the final host check... 2024-09-11T03:30:22.167 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stdout 5.2.2 2024-09-11T03:30:22.167 INFO:teuthology.orchestra.run.smithi084.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-11T03:30:22.167 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl is present 2024-09-11T03:30:22.167 INFO:teuthology.orchestra.run.smithi084.stdout:lvcreate is present 2024-09-11T03:30:22.173 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-11T03:30:22.173 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-11T03:30:22.179 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-11T03:30:22.180 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-09-11T03:30:22.186 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout enabled 2024-09-11T03:30:22.192 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout active 2024-09-11T03:30:22.193 INFO:teuthology.orchestra.run.smithi084.stdout:Unit chronyd.service is enabled and running 2024-09-11T03:30:22.193 INFO:teuthology.orchestra.run.smithi084.stdout:Host looks OK 2024-09-11T03:30:22.193 INFO:teuthology.orchestra.run.smithi084.stdout:Cluster fsid: 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:30:22.193 INFO:teuthology.orchestra.run.smithi084.stdout:Acquiring lock 140017050531536 on /run/cephadm/10fe7946-6fee-11ef-bcea-c7b262605968.lock 2024-09-11T03:30:22.193 INFO:teuthology.orchestra.run.smithi084.stdout:Lock 140017050531536 acquired on /run/cephadm/10fe7946-6fee-11ef-bcea-c7b262605968.lock 2024-09-11T03:30:22.193 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying IP 172.21.15.84 port 3300 ... 2024-09-11T03:30:22.194 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying IP 172.21.15.84 port 6789 ... 2024-09-11T03:30:22.194 INFO:teuthology.orchestra.run.smithi084.stdout:Base mon IP(s) is [172.21.15.84:3300, 172.21.15.84:6789], mon addrv is [v2:172.21.15.84:3300,v1:172.21.15.84:6789] 2024-09-11T03:30:22.197 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.84 metric 100 2024-09-11T03:30:22.197 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.84 metric 100 2024-09-11T03:30:22.199 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-11T03:30:22.199 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-11T03:30:22.199 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-11T03:30:22.202 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-11T03:30:22.203 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-11T03:30:22.203 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-11T03:30:22.203 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-11T03:30:22.204 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d751/64 scope link noprefixroute 2024-09-11T03:30:22.204 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-11T03:30:22.204 INFO:teuthology.orchestra.run.smithi084.stdout:Mon IP `172.21.15.84` is in CIDR network `172.21.0.0/20` 2024-09-11T03:30:22.204 INFO:teuthology.orchestra.run.smithi084.stdout:Mon IP `172.21.15.84` is in CIDR network `172.21.0.0/20` 2024-09-11T03:30:22.204 INFO:teuthology.orchestra.run.smithi084.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-11T03:30:22.205 INFO:teuthology.orchestra.run.smithi084.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-11T03:30:22.206 INFO:teuthology.orchestra.run.smithi084.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30... 2024-09-11T03:30:22.720 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stdout 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 2024-09-11T03:30:22.720 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30... 2024-09-11T03:30:22.720 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Getting image source signatures 2024-09-11T03:30:22.720 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Copying blob sha256:edd8cbe4deb828f809e8d76f308301dd93e971948e909d25cc03015fed1fd796 2024-09-11T03:30:22.721 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-11T03:30:22.721 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Copying config sha256:589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 2024-09-11T03:30:22.721 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-11T03:30:23.599 INFO:teuthology.orchestra.run.smithi084.stdout:ceph: stdout ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev) 2024-09-11T03:30:23.599 INFO:teuthology.orchestra.run.smithi084.stdout:Ceph version: ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev) 2024-09-11T03:30:23.599 INFO:teuthology.orchestra.run.smithi084.stdout:Extracting ceph user uid/gid from container image... 2024-09-11T03:30:24.612 INFO:teuthology.orchestra.run.smithi084.stdout:stat: stdout 167 167 2024-09-11T03:30:24.613 INFO:teuthology.orchestra.run.smithi084.stdout:Creating initial keys... 2024-09-11T03:30:25.494 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph-authtool: stdout AQDQDuFmARrrNRAAC13c+VRLOubQEsz7u5CjRg== 2024-09-11T03:30:26.424 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph-authtool: stdout AQDRDuFmE9TQMBAAgpJ5o44H3uCrXdC26/QX6Q== 2024-09-11T03:30:27.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph-authtool: stdout AQDSDuFmS6+tKxAANkZu4WGfgbsTuumiMHeGDg== 2024-09-11T03:30:27.306 INFO:teuthology.orchestra.run.smithi084.stdout:Creating initial monmap... 2024-09-11T03:30:28.194 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-11T03:30:28.194 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-11T03:30:28.194 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:30:28.194 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-11T03:30:28.194 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool for a [v2:172.21.15.84:3300,v1:172.21.15.84:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-11T03:30:28.194 INFO:teuthology.orchestra.run.smithi084.stdout:setting min_mon_release = quincy 2024-09-11T03:30:28.194 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: set fsid to 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:30:28.195 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-11T03:30:28.195 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:30:28.195 INFO:teuthology.orchestra.run.smithi084.stdout:Creating mon... 2024-09-11T03:30:29.115 INFO:teuthology.orchestra.run.smithi084.stdout:create mon.a on 2024-09-11T03:30:29.325 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-11T03:30:29.475 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-11T03:30:29.647 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-10fe7946-6fee-11ef-bcea-c7b262605968.target → /etc/systemd/system/ceph-10fe7946-6fee-11ef-bcea-c7b262605968.target. 2024-09-11T03:30:29.647 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-10fe7946-6fee-11ef-bcea-c7b262605968.target → /etc/systemd/system/ceph-10fe7946-6fee-11ef-bcea-c7b262605968.target. 2024-09-11T03:30:29.882 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mon.a 2024-09-11T03:30:29.882 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Failed to reset failed state of unit ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mon.a.service: Unit ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mon.a.service not loaded. 2024-09-11T03:30:30.042 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-10fe7946-6fee-11ef-bcea-c7b262605968.target.wants/ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mon.a.service → /etc/systemd/system/ceph-10fe7946-6fee-11ef-bcea-c7b262605968@.service. 2024-09-11T03:30:30.581 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-11T03:30:30.581 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout disabled 2024-09-11T03:30:30.587 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-11T03:30:30.588 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-09-11T03:30:30.588 INFO:teuthology.orchestra.run.smithi084.stdout:firewalld.service is not enabled 2024-09-11T03:30:30.588 INFO:teuthology.orchestra.run.smithi084.stdout:Not possible to enable service . firewalld.service is not available 2024-09-11T03:30:30.588 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mon to start... 2024-09-11T03:30:30.588 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mon... 2024-09-11T03:30:31.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:31 smithi084 ceph-mon[26885]: from='client.? 172.21.15.84:0/1228325039' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-11T03:30:31.592 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout cluster: 2024-09-11T03:30:31.594 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout id: 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:30:31.594 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-11T03:30:31.594 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-11T03:30:31.594 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout services: 2024-09-11T03:30:31.594 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.518104s) 2024-09-11T03:30:31.594 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-11T03:30:31.594 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-11T03:30:31.594 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-11T03:30:31.594 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout data: 2024-09-11T03:30:31.595 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-11T03:30:31.595 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-11T03:30:31.595 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-11T03:30:31.595 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout pgs: 2024-09-11T03:30:31.595 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-11T03:30:31.595 INFO:teuthology.orchestra.run.smithi084.stdout:mon is available 2024-09-11T03:30:31.595 INFO:teuthology.orchestra.run.smithi084.stdout:Assimilating anything we can from ceph.conf... 2024-09-11T03:30:32.630 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:32 smithi084 ceph-mon[26885]: from='client.? 172.21.15.84:0/2883085765' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-11T03:30:32.630 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:32 smithi084 ceph-mon[26885]: from='client.? 172.21.15.84:0/2883085765' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-11T03:30:32.634 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-11T03:30:32.634 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [global] 2024-09-11T03:30:32.634 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout fsid = 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:30:32.634 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-11T03:30:32.635 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.84:3300,v1:172.21.15.84:6789] 2024-09-11T03:30:32.635 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-11T03:30:32.635 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-11T03:30:32.635 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-11T03:30:32.635 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-11T03:30:32.635 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-11T03:30:32.635 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-11T03:30:32.635 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-11T03:30:32.635 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-11T03:30:32.635 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [osd] 2024-09-11T03:30:32.635 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout filestore_queue_throttle_high_multiple = 2 2024-09-11T03:30:32.635 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout filestore_queue_throttle_max_multiple = 10 2024-09-11T03:30:32.635 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-11T03:30:32.635 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-11T03:30:32.636 INFO:teuthology.orchestra.run.smithi084.stdout:Generating new minimal ceph.conf... 2024-09-11T03:30:33.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:33 smithi084 ceph-mon[26885]: from='client.? 172.21.15.84:0/3551783843' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:30:37.828 INFO:teuthology.orchestra.run.smithi084.stdout:Restarting the monitor... 2024-09-11T03:30:38.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:37 smithi084 systemd[1]: Stopping Ceph mon.a for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:30:39.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:39 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a[26881]: 2024-09-11T03:30:39.000+0000 7efc93a7f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:30:39.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:39 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a[26881]: 2024-09-11T03:30:39.000+0000 7efc93a7f640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-11T03:30:39.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:39 smithi084 podman[27130]: 2024-09-11 03:30:39.092908302 +0000 UTC m=+1.098979653 container died 7b66a7ca28b4ddc34e1c0ea4b43769fae163d79405700e14a4c3513840e6283f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T03:30:43.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:42 smithi084 podman[27130]: 2024-09-11 03:30:42.678101674 +0000 UTC m=+4.684173024 container cleanup 7b66a7ca28b4ddc34e1c0ea4b43769fae163d79405700e14a4c3513840e6283f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T03:30:43.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:42 smithi084 bash[27130]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a 2024-09-11T03:30:44.980 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:44 smithi084 podman[27142]: 2024-09-11 03:30:44.655928811 +0000 UTC m=+5.559357773 container remove 7b66a7ca28b4ddc34e1c0ea4b43769fae163d79405700e14a4c3513840e6283f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-11T03:30:44.980 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:44 smithi084 systemd[1]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-11T03:30:44.980 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:44 smithi084 systemd[1]: Stopped Ceph mon.a for 10fe7946-6fee-11ef-bcea-c7b262605968. 2024-09-11T03:30:44.980 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:44 smithi084 systemd[1]: Starting Ceph mon.a for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:30:45.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:44 smithi084 podman[27235]: 2024-09-11 03:30:44.883511258 +0000 UTC m=+0.015804390 image pull 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T03:30:46.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:45 smithi084 podman[27235]: 2024-09-11 03:30:45.884242202 +0000 UTC m=+1.016535329 container create 1dd196d5d5cb6424ef8d5b25da0d9defdcd8fbc53f8b2a3eb6b5476e5c9b5558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-11T03:30:46.752 INFO:teuthology.orchestra.run.smithi084.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-11T03:30:46.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 podman[27235]: 2024-09-11 03:30:46.690681905 +0000 UTC m=+1.822975032 container init 1dd196d5d5cb6424ef8d5b25da0d9defdcd8fbc53f8b2a3eb6b5476e5c9b5558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-09-11T03:30:46.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 podman[27235]: 2024-09-11 03:30:46.694286858 +0000 UTC m=+1.826579994 container start 1dd196d5d5cb6424ef8d5b25da0d9defdcd8fbc53f8b2a3eb6b5476e5c9b5558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T03:30:46.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: set uid:gid to 167:167 (ceph:ceph) 2024-09-11T03:30:46.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev), process ceph-mon, pid 2 2024-09-11T03:30:46.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: pidfile_write: ignore empty --pid-file 2024-09-11T03:30:46.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: load: jerasure load: lrc 2024-09-11T03:30:46.875 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: RocksDB version: 7.9.2 2024-09-11T03:30:46.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Git sha 0 2024-09-11T03:30:46.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Compile date 2024-09-10 08:02:51 2024-09-11T03:30:46.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: DB SUMMARY 2024-09-11T03:30:46.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: DB Session ID: TDA99EPNIPOPMVYXVITT 2024-09-11T03:30:46.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: CURRENT file: CURRENT 2024-09-11T03:30:46.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: IDENTITY file: IDENTITY 2024-09-11T03:30:46.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-11T03:30:46.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-11T03:30:46.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 96730 ; 2024-09-11T03:30:46.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.error_if_exists: 0 2024-09-11T03:30:46.876 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.create_if_missing: 0 2024-09-11T03:30:46.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.paranoid_checks: 1 2024-09-11T03:30:46.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-11T03:30:46.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-11T03:30:46.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-11T03:30:46.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.env: 0x56259010c180 2024-09-11T03:30:46.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.fs: PosixFileSystem 2024-09-11T03:30:46.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.info_log: 0x56259180b700 2024-09-11T03:30:46.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_file_opening_threads: 16 2024-09-11T03:30:46.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.statistics: (nil) 2024-09-11T03:30:46.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.use_fsync: 0 2024-09-11T03:30:46.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_log_file_size: 0 2024-09-11T03:30:46.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-11T03:30:46.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-11T03:30:46.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.keep_log_file_num: 1000 2024-09-11T03:30:46.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.recycle_log_file_num: 0 2024-09-11T03:30:46.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.allow_fallocate: 1 2024-09-11T03:30:46.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.allow_mmap_reads: 0 2024-09-11T03:30:46.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.allow_mmap_writes: 0 2024-09-11T03:30:46.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.use_direct_reads: 0 2024-09-11T03:30:46.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-11T03:30:46.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.create_missing_column_families: 0 2024-09-11T03:30:46.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.db_log_dir: 2024-09-11T03:30:46.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.wal_dir: 2024-09-11T03:30:46.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-11T03:30:46.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-11T03:30:46.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-11T03:30:46.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-11T03:30:46.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-11T03:30:46.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-11T03:30:46.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.advise_random_on_open: 1 2024-09-11T03:30:46.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.db_write_buffer_size: 0 2024-09-11T03:30:46.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.write_buffer_manager: 0x56259180fb80 2024-09-11T03:30:46.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-11T03:30:46.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-11T03:30:46.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-11T03:30:46.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.rate_limiter: (nil) 2024-09-11T03:30:46.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-11T03:30:46.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.wal_recovery_mode: 2 2024-09-11T03:30:46.880 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.enable_thread_tracking: 0 2024-09-11T03:30:46.880 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.enable_pipelined_write: 0 2024-09-11T03:30:46.880 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.unordered_write: 0 2024-09-11T03:30:46.880 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-11T03:30:46.880 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-11T03:30:46.880 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-11T03:30:46.880 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-11T03:30:46.880 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.row_cache: None 2024-09-11T03:30:46.880 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.wal_filter: None 2024-09-11T03:30:46.881 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-11T03:30:46.881 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.allow_ingest_behind: 0 2024-09-11T03:30:46.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.two_write_queues: 0 2024-09-11T03:30:46.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.manual_wal_flush: 0 2024-09-11T03:30:46.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.wal_compression: 0 2024-09-11T03:30:46.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.atomic_flush: 0 2024-09-11T03:30:46.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-11T03:30:46.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-11T03:30:46.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-11T03:30:46.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.log_readahead_size: 0 2024-09-11T03:30:46.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-11T03:30:46.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.best_efforts_recovery: 0 2024-09-11T03:30:46.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-11T03:30:46.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-11T03:30:46.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.allow_data_in_errors: 0 2024-09-11T03:30:46.883 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.db_host_id: __hostname__ 2024-09-11T03:30:46.883 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-11T03:30:46.883 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_background_jobs: 2 2024-09-11T03:30:46.883 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_background_compactions: -1 2024-09-11T03:30:46.883 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_subcompactions: 1 2024-09-11T03:30:46.883 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-11T03:30:46.883 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-11T03:30:46.883 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-11T03:30:46.883 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_total_wal_size: 0 2024-09-11T03:30:46.883 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-11T03:30:46.883 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-11T03:30:46.883 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-11T03:30:46.883 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-11T03:30:46.883 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_open_files: -1 2024-09-11T03:30:46.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.bytes_per_sync: 0 2024-09-11T03:30:46.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-11T03:30:46.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-11T03:30:46.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compaction_readahead_size: 0 2024-09-11T03:30:46.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_background_flushes: -1 2024-09-11T03:30:46.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Compression algorithms supported: 2024-09-11T03:30:46.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: kZSTD supported: 0 2024-09-11T03:30:46.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: kXpressCompression supported: 0 2024-09-11T03:30:46.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: kBZip2Compression supported: 0 2024-09-11T03:30:46.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-11T03:30:46.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: kLZ4Compression supported: 1 2024-09-11T03:30:46.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: kZlibCompression supported: 1 2024-09-11T03:30:46.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: kLZ4HCCompression supported: 1 2024-09-11T03:30:46.885 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: kSnappyCompression supported: 1 2024-09-11T03:30:46.885 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-11T03:30:46.885 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-11T03:30:46.885 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-11T03:30:46.885 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-11T03:30:46.885 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-11T03:30:46.885 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.merge_operator: 2024-09-11T03:30:46.885 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compaction_filter: None 2024-09-11T03:30:46.885 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compaction_filter_factory: None 2024-09-11T03:30:46.885 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.sst_partitioner_factory: None 2024-09-11T03:30:46.885 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-11T03:30:46.885 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-11T03:30:46.885 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56259180b320) 2024-09-11T03:30:46.886 INFO:journalctl@ceph.mon.a.smithi084.stdout: cache_index_and_filter_blocks: 1 2024-09-11T03:30:46.886 INFO:journalctl@ceph.mon.a.smithi084.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-11T03:30:46.886 INFO:journalctl@ceph.mon.a.smithi084.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-11T03:30:46.886 INFO:journalctl@ceph.mon.a.smithi084.stdout: pin_top_level_index_and_filter: 1 2024-09-11T03:30:46.886 INFO:journalctl@ceph.mon.a.smithi084.stdout: index_type: 0 2024-09-11T03:30:46.886 INFO:journalctl@ceph.mon.a.smithi084.stdout: data_block_index_type: 0 2024-09-11T03:30:46.886 INFO:journalctl@ceph.mon.a.smithi084.stdout: index_shortening: 1 2024-09-11T03:30:46.886 INFO:journalctl@ceph.mon.a.smithi084.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-11T03:30:46.886 INFO:journalctl@ceph.mon.a.smithi084.stdout: checksum: 4 2024-09-11T03:30:46.886 INFO:journalctl@ceph.mon.a.smithi084.stdout: no_block_cache: 0 2024-09-11T03:30:46.886 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_cache: 0x5625917ff610 2024-09-11T03:30:46.886 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_cache_name: BinnedLRUCache 2024-09-11T03:30:46.886 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_cache_options: 2024-09-11T03:30:46.887 INFO:journalctl@ceph.mon.a.smithi084.stdout: capacity : 536870912 2024-09-11T03:30:46.887 INFO:journalctl@ceph.mon.a.smithi084.stdout: num_shard_bits : 4 2024-09-11T03:30:46.887 INFO:journalctl@ceph.mon.a.smithi084.stdout: strict_capacity_limit : 0 2024-09-11T03:30:46.887 INFO:journalctl@ceph.mon.a.smithi084.stdout: high_pri_pool_ratio: 0.000 2024-09-11T03:30:46.887 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_cache_compressed: (nil) 2024-09-11T03:30:46.887 INFO:journalctl@ceph.mon.a.smithi084.stdout: persistent_cache: (nil) 2024-09-11T03:30:46.887 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_size: 4096 2024-09-11T03:30:46.887 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_size_deviation: 10 2024-09-11T03:30:46.887 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_restart_interval: 16 2024-09-11T03:30:46.887 INFO:journalctl@ceph.mon.a.smithi084.stdout: index_block_restart_interval: 1 2024-09-11T03:30:46.887 INFO:journalctl@ceph.mon.a.smithi084.stdout: metadata_block_size: 4096 2024-09-11T03:30:46.887 INFO:journalctl@ceph.mon.a.smithi084.stdout: partition_filters: 0 2024-09-11T03:30:46.887 INFO:journalctl@ceph.mon.a.smithi084.stdout: use_delta_encoding: 1 2024-09-11T03:30:46.887 INFO:journalctl@ceph.mon.a.smithi084.stdout: filter_policy: bloomfilter 2024-09-11T03:30:46.888 INFO:journalctl@ceph.mon.a.smithi084.stdout: whole_key_filtering: 1 2024-09-11T03:30:46.888 INFO:journalctl@ceph.mon.a.smithi084.stdout: verify_compression: 0 2024-09-11T03:30:46.888 INFO:journalctl@ceph.mon.a.smithi084.stdout: read_amp_bytes_per_bit: 0 2024-09-11T03:30:46.888 INFO:journalctl@ceph.mon.a.smithi084.stdout: format_version: 5 2024-09-11T03:30:46.888 INFO:journalctl@ceph.mon.a.smithi084.stdout: enable_index_compression: 1 2024-09-11T03:30:46.888 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_align: 0 2024-09-11T03:30:46.888 INFO:journalctl@ceph.mon.a.smithi084.stdout: max_auto_readahead_size: 262144 2024-09-11T03:30:46.888 INFO:journalctl@ceph.mon.a.smithi084.stdout: prepopulate_block_cache: 0 2024-09-11T03:30:46.888 INFO:journalctl@ceph.mon.a.smithi084.stdout: initial_auto_readahead_size: 8192 2024-09-11T03:30:46.888 INFO:journalctl@ceph.mon.a.smithi084.stdout: num_file_reads_for_auto_readahead: 2 2024-09-11T03:30:46.888 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.write_buffer_size: 33554432 2024-09-11T03:30:46.888 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_write_buffer_number: 2 2024-09-11T03:30:46.888 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compression: NoCompression 2024-09-11T03:30:46.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.bottommost_compression: Disabled 2024-09-11T03:30:46.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.prefix_extractor: nullptr 2024-09-11T03:30:46.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-11T03:30:46.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.num_levels: 7 2024-09-11T03:30:46.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-11T03:30:46.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-11T03:30:46.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-11T03:30:46.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-11T03:30:46.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-11T03:30:46.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-11T03:30:46.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-11T03:30:46.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-11T03:30:46.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-11T03:30:46.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-11T03:30:46.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-11T03:30:46.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-11T03:30:46.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-11T03:30:46.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compression_opts.level: 32767 2024-09-11T03:30:46.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compression_opts.strategy: 0 2024-09-11T03:30:46.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-11T03:30:46.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-11T03:30:46.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-11T03:30:46.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-11T03:30:46.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compression_opts.enabled: false 2024-09-11T03:30:46.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-11T03:30:46.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-11T03:30:46.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-11T03:30:46.891 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-11T03:30:46.891 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.target_file_size_base: 67108864 2024-09-11T03:30:46.891 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-11T03:30:46.891 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-11T03:30:46.891 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-11T03:30:46.891 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-11T03:30:46.891 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-11T03:30:46.891 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-11T03:30:46.891 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-11T03:30:46.891 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-11T03:30:46.891 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-11T03:30:46.891 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-11T03:30:46.891 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-11T03:30:46.891 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-11T03:30:46.892 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-11T03:30:46.892 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-11T03:30:46.892 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.arena_block_size: 1048576 2024-09-11T03:30:46.892 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-11T03:30:46.892 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-11T03:30:46.892 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.disable_auto_compactions: 0 2024-09-11T03:30:46.892 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-11T03:30:46.892 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-11T03:30:46.892 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-11T03:30:46.892 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-11T03:30:46.892 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-11T03:30:46.892 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-11T03:30:46.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-11T03:30:46.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-11T03:30:46.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-11T03:30:46.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-11T03:30:46.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-11T03:30:46.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.inplace_update_support: 0 2024-09-11T03:30:46.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-11T03:30:46.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-11T03:30:46.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-11T03:30:46.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-11T03:30:46.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.bloom_locality: 0 2024-09-11T03:30:46.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.max_successive_merges: 0 2024-09-11T03:30:46.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-11T03:30:46.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.paranoid_file_checks: 0 2024-09-11T03:30:46.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.force_consistency_checks: 1 2024-09-11T03:30:46.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.report_bg_io_stats: 0 2024-09-11T03:30:46.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.ttl: 2592000 2024-09-11T03:30:46.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-11T03:30:46.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-11T03:30:46.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-11T03:30:46.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.enable_blob_files: false 2024-09-11T03:30:46.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.min_blob_size: 0 2024-09-11T03:30:46.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.blob_file_size: 268435456 2024-09-11T03:30:46.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-11T03:30:46.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-11T03:30:46.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-11T03:30:46.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-11T03:30:46.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-11T03:30:46.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.blob_file_starting_level: 0 2024-09-11T03:30:46.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-11T03:30:46.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-11T03:30:46.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-11T03:30:46.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f018f878-359e-4bb4-8801-2a1e9de90a35 2024-09-11T03:30:46.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726025446721425, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-11T03:30:46.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-11T03:30:46.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726025446722351, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92719, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 289, "table_properties": {"data_size": 90766, "index_size": 273, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 12144, "raw_average_key_size": 48, "raw_value_size": 83913, "raw_average_value_size": 338, "num_data_blocks": 12, "num_entries": 248, "num_filter_entries": 248, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726025446, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f018f878-359e-4bb4-8801-2a1e9de90a35", "db_session_id": "TDA99EPNIPOPMVYXVITT", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-11T03:30:46.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726025446722450, "job": 1, "event": "recovery_finished"} 2024-09-11T03:30:46.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-11T03:30:46.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-11T03:30:46.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56259182ee00 2024-09-11T03:30:46.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: DB pointer 0x562591930000 2024-09-11T03:30:46.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-11T03:30:46.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-11T03:30:46.896 INFO:journalctl@ceph.mon.a.smithi084.stdout: ** DB Stats ** 2024-09-11T03:30:46.896 INFO:journalctl@ceph.mon.a.smithi084.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-11T03:30:46.896 INFO:journalctl@ceph.mon.a.smithi084.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-11T03:30:46.896 INFO:journalctl@ceph.mon.a.smithi084.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-11T03:30:46.896 INFO:journalctl@ceph.mon.a.smithi084.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-11T03:30:46.896 INFO:journalctl@ceph.mon.a.smithi084.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-11T03:30:46.896 INFO:journalctl@ceph.mon.a.smithi084.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-11T03:30:46.897 INFO:journalctl@ceph.mon.a.smithi084.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-11T03:30:46.897 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-09-11T03:30:46.897 INFO:journalctl@ceph.mon.a.smithi084.stdout: ** Compaction Stats [default] ** 2024-09-11T03:30:46.897 INFO:journalctl@ceph.mon.a.smithi084.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-11T03:30:46.897 INFO:journalctl@ceph.mon.a.smithi084.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-11T03:30:46.897 INFO:journalctl@ceph.mon.a.smithi084.stdout: L0 2/0 92.41 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-11T03:30:46.897 INFO:journalctl@ceph.mon.a.smithi084.stdout: Sum 2/0 92.41 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-11T03:30:46.897 INFO:journalctl@ceph.mon.a.smithi084.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-11T03:30:46.897 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-09-11T03:30:46.897 INFO:journalctl@ceph.mon.a.smithi084.stdout: ** Compaction Stats [default] ** 2024-09-11T03:30:46.897 INFO:journalctl@ceph.mon.a.smithi084.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-11T03:30:46.897 INFO:journalctl@ceph.mon.a.smithi084.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T03:30:46.897 INFO:journalctl@ceph.mon.a.smithi084.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 147.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-11T03:30:46.898 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-09-11T03:30:46.898 INFO:journalctl@ceph.mon.a.smithi084.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-11T03:30:46.898 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-09-11T03:30:46.898 INFO:journalctl@ceph.mon.a.smithi084.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-11T03:30:46.898 INFO:journalctl@ceph.mon.a.smithi084.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-11T03:30:46.898 INFO:journalctl@ceph.mon.a.smithi084.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-11T03:30:46.898 INFO:journalctl@ceph.mon.a.smithi084.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-11T03:30:46.898 INFO:journalctl@ceph.mon.a.smithi084.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-11T03:30:46.898 INFO:journalctl@ceph.mon.a.smithi084.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-11T03:30:46.898 INFO:journalctl@ceph.mon.a.smithi084.stdout: Cumulative compaction: 0.00 GB write, 28.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T03:30:46.898 INFO:journalctl@ceph.mon.a.smithi084.stdout: Interval compaction: 0.00 GB write, 28.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T03:30:46.898 INFO:journalctl@ceph.mon.a.smithi084.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-11T03:30:46.898 INFO:journalctl@ceph.mon.a.smithi084.stdout: Block cache BinnedLRUCache@0x5625917ff610#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-11T03:30:46.898 INFO:journalctl@ceph.mon.a.smithi084.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-11T03:30:46.899 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-09-11T03:30:46.899 INFO:journalctl@ceph.mon.a.smithi084.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-11T03:30:46.899 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: starting mon.a rank 0 at public addrs [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] at bind addrs [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:30:46.899 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: mon.a@-1(???) e1 preinit fsid 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:30:46.899 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: mon.a@-1(???).mds e1 new map 2024-09-11T03:30:46.899 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: mon.a@-1(???).mds e1 print_map 2024-09-11T03:30:46.899 INFO:journalctl@ceph.mon.a.smithi084.stdout: e1 2024-09-11T03:30:46.899 INFO:journalctl@ceph.mon.a.smithi084.stdout: btime 2024-09-11T03:30:30:532454+0000 2024-09-11T03:30:46.899 INFO:journalctl@ceph.mon.a.smithi084.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-11T03:30:46.899 INFO:journalctl@ceph.mon.a.smithi084.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-11T03:30:46.899 INFO:journalctl@ceph.mon.a.smithi084.stdout: legacy client fscid: -1 2024-09-11T03:30:46.899 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-09-11T03:30:46.899 INFO:journalctl@ceph.mon.a.smithi084.stdout: No filesystems configured 2024-09-11T03:30:46.899 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-11T03:30:46.900 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T03:30:46.900 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T03:30:46.900 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T03:30:46.900 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-11T03:30:46.900 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 bash[27235]: 1dd196d5d5cb6424ef8d5b25da0d9defdcd8fbc53f8b2a3eb6b5476e5c9b5558 2024-09-11T03:30:46.900 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 systemd[1]: Started Ceph mon.a for 10fe7946-6fee-11ef-bcea-c7b262605968. 2024-09-11T03:30:46.900 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-11T03:30:46.900 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: monmap epoch 1 2024-09-11T03:30:46.900 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: fsid 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:30:46.900 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: last_changed 2024-09-11T03:30:27.618796+0000 2024-09-11T03:30:46.900 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: created 2024-09-11T03:30:27.618796+0000 2024-09-11T03:30:46.900 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: min_mon_release 19 (squid) 2024-09-11T03:30:46.900 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: election_strategy: 1 2024-09-11T03:30:46.901 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-09-11T03:30:46.901 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: fsmap 2024-09-11T03:30:46.901 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: osdmap e1: 0 total, 0 up, 0 in 2024-09-11T03:30:46.901 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:46 smithi084 ceph-mon[27254]: mgrmap e1: no daemons active 2024-09-11T03:30:47.816 INFO:teuthology.orchestra.run.smithi084.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-11T03:30:47.838 INFO:teuthology.orchestra.run.smithi084.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-11T03:30:47.838 INFO:teuthology.orchestra.run.smithi084.stdout:Creating mgr... 2024-09-11T03:30:47.838 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-11T03:30:47.838 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-11T03:30:48.098 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mgr.y 2024-09-11T03:30:48.098 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Failed to reset failed state of unit ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mgr.y.service: Unit ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mgr.y.service not loaded. 2024-09-11T03:30:48.245 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-10fe7946-6fee-11ef-bcea-c7b262605968.target.wants/ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-10fe7946-6fee-11ef-bcea-c7b262605968@.service. 2024-09-11T03:30:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:48 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2981213745' entity='client.admin' 2024-09-11T03:30:48.768 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:30:48 smithi084 podman[27468]: 2024-09-11 03:30:48.68773289 +0000 UTC m=+0.330266406 container start f7bb1492bbd3ca344a42efbf8dbc2b49a8eca592b98356b22f6ac786d28dc20f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4) 2024-09-11T03:30:48.768 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:30:48 smithi084 bash[27468]: f7bb1492bbd3ca344a42efbf8dbc2b49a8eca592b98356b22f6ac786d28dc20f 2024-09-11T03:30:48.784 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-11T03:30:48.784 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout disabled 2024-09-11T03:30:48.792 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-11T03:30:48.793 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-09-11T03:30:48.793 INFO:teuthology.orchestra.run.smithi084.stdout:firewalld.service is not enabled 2024-09-11T03:30:48.793 INFO:teuthology.orchestra.run.smithi084.stdout:Not possible to enable service . firewalld.service is not available 2024-09-11T03:30:48.803 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-11T03:30:48.803 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout disabled 2024-09-11T03:30:48.811 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-11T03:30:48.812 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-09-11T03:30:48.812 INFO:teuthology.orchestra.run.smithi084.stdout:firewalld.service is not enabled 2024-09-11T03:30:48.812 INFO:teuthology.orchestra.run.smithi084.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-11T03:30:48.812 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mgr to start... 2024-09-11T03:30:48.812 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mgr... 2024-09-11T03:30:49.074 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:30:48 smithi084 systemd[1]: Started Ceph mgr.y for 10fe7946-6fee-11ef-bcea-c7b262605968. 2024-09-11T03:30:49.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3428834050' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-11T03:30:49.889 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-11T03:30:49.889 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-09-11T03:30:49.890 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsid": "10fe7946-6fee-11ef-bcea-c7b262605968", 2024-09-11T03:30:49.890 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "health": { 2024-09-11T03:30:49.890 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-11T03:30:49.890 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-11T03:30:49.890 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-11T03:30:49.890 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:49.890 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-11T03:30:49.890 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-11T03:30:49.890 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 0 2024-09-11T03:30:49.890 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-11T03:30:49.890 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-11T03:30:49.890 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "a" 2024-09-11T03:30:49.890 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-11T03:30:49.890 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-11T03:30:49.891 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-11T03:30:49.891 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T03:30:49.891 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-11T03:30:49.891 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-11T03:30:49.891 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:49.891 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-11T03:30:49.891 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T03:30:49.891 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-11T03:30:49.891 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-11T03:30:49.891 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-11T03:30:49.891 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-11T03:30:49.891 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-11T03:30:49.891 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-11T03:30:49.895 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:49.895 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-11T03:30:49.895 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-11T03:30:49.895 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-11T03:30:49.895 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-11T03:30:49.895 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-11T03:30:49.895 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-11T03:30:49.895 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-11T03:30:49.895 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-11T03:30:49.895 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-11T03:30:49.895 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:49.895 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-11T03:30:49.896 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T03:30:49.896 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "btime": "2024-09-11T03:30:30:532454+0000", 2024-09-11T03:30:49.896 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-11T03:30:49.896 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-11T03:30:49.896 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:49.896 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-11T03:30:49.896 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-11T03:30:49.896 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-11T03:30:49.896 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-11T03:30:49.896 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-11T03:30:49.896 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-11T03:30:49.896 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "restful" 2024-09-11T03:30:49.896 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-11T03:30:49.896 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T03:30:49.897 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:49.897 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-11T03:30:49.897 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T03:30:49.897 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modified": "2024-09-11T03:30:30.533123+0000", 2024-09-11T03:30:49.897 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T03:30:49.897 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:49.897 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-11T03:30:49.897 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-09-11T03:30:49.897 INFO:teuthology.orchestra.run.smithi084.stdout:mgr not available, waiting (1/15)... 2024-09-11T03:30:52.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:52 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/555644845' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-11T03:30:52.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-11T03:30:52.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-09-11T03:30:52.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsid": "10fe7946-6fee-11ef-bcea-c7b262605968", 2024-09-11T03:30:52.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "health": { 2024-09-11T03:30:52.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-11T03:30:52.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-11T03:30:52.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-11T03:30:52.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:52.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-11T03:30:52.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-11T03:30:52.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 0 2024-09-11T03:30:52.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-11T03:30:52.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-11T03:30:52.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "a" 2024-09-11T03:30:52.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-11T03:30:52.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-11T03:30:52.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-11T03:30:52.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T03:30:52.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-11T03:30:52.876 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-11T03:30:52.876 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:52.876 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-11T03:30:52.876 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T03:30:52.876 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-11T03:30:52.876 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-11T03:30:52.876 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-11T03:30:52.876 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-11T03:30:52.876 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-11T03:30:52.876 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-11T03:30:52.876 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:52.876 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-11T03:30:52.876 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-11T03:30:52.877 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-11T03:30:52.877 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-11T03:30:52.877 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-11T03:30:52.877 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-11T03:30:52.877 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-11T03:30:52.877 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-11T03:30:52.877 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-11T03:30:52.877 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:52.877 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-11T03:30:52.877 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T03:30:52.877 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "btime": "2024-09-11T03:30:30:532454+0000", 2024-09-11T03:30:52.877 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-11T03:30:52.877 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-11T03:30:52.877 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:52.878 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-11T03:30:52.878 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-11T03:30:52.878 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-11T03:30:52.878 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-11T03:30:52.878 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-11T03:30:52.878 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-11T03:30:52.878 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "restful" 2024-09-11T03:30:52.878 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-11T03:30:52.878 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T03:30:52.878 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:52.878 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-11T03:30:52.878 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T03:30:52.879 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modified": "2024-09-11T03:30:30.533123+0000", 2024-09-11T03:30:52.879 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T03:30:52.879 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:52.879 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-11T03:30:52.879 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-09-11T03:30:52.879 INFO:teuthology.orchestra.run.smithi084.stdout:mgr not available, waiting (2/15)... 2024-09-11T03:30:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:53 smithi084 ceph-mon[27254]: Activating manager daemon y 2024-09-11T03:30:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:53 smithi084 ceph-mon[27254]: mgrmap e2: y(active, starting, since 0.00313109s) 2024-09-11T03:30:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:53 smithi084 ceph-mon[27254]: from='mgr.14100 172.21.15.84:0/2958034913' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-11T03:30:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:53 smithi084 ceph-mon[27254]: from='mgr.14100 172.21.15.84:0/2958034913' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-11T03:30:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:53 smithi084 ceph-mon[27254]: from='mgr.14100 172.21.15.84:0/2958034913' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-11T03:30:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:53 smithi084 ceph-mon[27254]: from='mgr.14100 172.21.15.84:0/2958034913' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:30:54.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:53 smithi084 ceph-mon[27254]: from='mgr.14100 172.21.15.84:0/2958034913' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-11T03:30:54.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:53 smithi084 ceph-mon[27254]: Manager daemon y is now available 2024-09-11T03:30:54.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:53 smithi084 ceph-mon[27254]: from='mgr.14100 172.21.15.84:0/2958034913' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-11T03:30:54.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:53 smithi084 ceph-mon[27254]: from='mgr.14100 172.21.15.84:0/2958034913' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-11T03:30:54.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:53 smithi084 ceph-mon[27254]: from='mgr.14100 172.21.15.84:0/2958034913' entity='mgr.y' 2024-09-11T03:30:54.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:53 smithi084 ceph-mon[27254]: from='mgr.14100 172.21.15.84:0/2958034913' entity='mgr.y' 2024-09-11T03:30:54.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:53 smithi084 ceph-mon[27254]: from='mgr.14100 172.21.15.84:0/2958034913' entity='mgr.y' 2024-09-11T03:30:55.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:55 smithi084 ceph-mon[27254]: mgrmap e3: y(active, since 1.0069s) 2024-09-11T03:30:55.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:55 smithi084 ceph-mon[27254]: from='client.14112 172.21.15.84:0/286392650' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-11T03:30:56.133 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-11T03:30:56.134 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-09-11T03:30:56.134 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsid": "10fe7946-6fee-11ef-bcea-c7b262605968", 2024-09-11T03:30:56.134 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "health": { 2024-09-11T03:30:56.134 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-11T03:30:56.134 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-11T03:30:56.134 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-11T03:30:56.134 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:56.134 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-11T03:30:56.134 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-11T03:30:56.134 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 0 2024-09-11T03:30:56.134 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-11T03:30:56.134 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-11T03:30:56.134 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "a" 2024-09-11T03:30:56.134 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-11T03:30:56.135 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-11T03:30:56.135 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-11T03:30:56.135 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T03:30:56.135 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-11T03:30:56.135 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-11T03:30:56.135 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:56.135 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-11T03:30:56.135 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T03:30:56.135 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-11T03:30:56.135 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-11T03:30:56.135 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-11T03:30:56.135 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-11T03:30:56.135 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-11T03:30:56.135 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-11T03:30:56.139 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:56.139 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-11T03:30:56.139 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-11T03:30:56.139 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-11T03:30:56.139 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-11T03:30:56.139 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-11T03:30:56.139 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-11T03:30:56.139 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-11T03:30:56.139 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-11T03:30:56.139 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-11T03:30:56.139 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:56.140 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-11T03:30:56.140 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T03:30:56.140 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "btime": "2024-09-11T03:30:30:532454+0000", 2024-09-11T03:30:56.140 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-11T03:30:56.140 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-11T03:30:56.140 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:56.140 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-11T03:30:56.140 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-11T03:30:56.140 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-11T03:30:56.140 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-11T03:30:56.140 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-11T03:30:56.140 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-11T03:30:56.140 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "restful" 2024-09-11T03:30:56.140 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-11T03:30:56.141 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T03:30:56.141 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:56.141 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-11T03:30:56.141 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T03:30:56.141 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modified": "2024-09-11T03:30:30.533123+0000", 2024-09-11T03:30:56.141 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T03:30:56.141 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-11T03:30:56.141 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-11T03:30:56.141 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-09-11T03:30:56.141 INFO:teuthology.orchestra.run.smithi084.stdout:mgr is available 2024-09-11T03:30:57.000 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:56 smithi084 ceph-mon[27254]: mgrmap e4: y(active, since 2s) 2024-09-11T03:30:57.001 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1689504402' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-11T03:30:57.197 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-11T03:30:57.197 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [global] 2024-09-11T03:30:57.197 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout fsid = 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:30:57.197 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-11T03:30:57.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.84:3300,v1:172.21.15.84:6789] 2024-09-11T03:30:57.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-11T03:30:57.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-11T03:30:57.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-11T03:30:57.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-11T03:30:57.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-11T03:30:57.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-11T03:30:57.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-11T03:30:57.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-11T03:30:57.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [osd] 2024-09-11T03:30:57.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout filestore_queue_throttle_high_multiple = 2 2024-09-11T03:30:57.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout filestore_queue_throttle_max_multiple = 10 2024-09-11T03:30:57.198 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-11T03:30:57.199 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-11T03:30:57.199 INFO:teuthology.orchestra.run.smithi084.stdout:Enabling cephadm module... 2024-09-11T03:30:58.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:58 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3954356411' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-11T03:30:58.394 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:30:58 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: ignoring --setuser ceph since I am not root 2024-09-11T03:30:58.394 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:30:58 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: ignoring --setgroup ceph since I am not root 2024-09-11T03:30:59.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:59 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3954356411' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-11T03:30:59.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:30:59 smithi084 ceph-mon[27254]: mgrmap e5: y(active, since 4s) 2024-09-11T03:30:59.750 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-09-11T03:30:59.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-11T03:30:59.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-11T03:30:59.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-11T03:30:59.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-11T03:30:59.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-09-11T03:30:59.751 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for the mgr to restart... 2024-09-11T03:30:59.751 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mgr epoch 5... 2024-09-11T03:31:00.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:00 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3750512179' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-11T03:31:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:03 smithi084 ceph-mon[27254]: Active manager daemon y restarted 2024-09-11T03:31:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:03 smithi084 ceph-mon[27254]: Activating manager daemon y 2024-09-11T03:31:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:03 smithi084 ceph-mon[27254]: osdmap e2: 0 total, 0 up, 0 in 2024-09-11T03:31:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:03 smithi084 ceph-mon[27254]: mgrmap e6: y(active, starting, since 0.00358415s) 2024-09-11T03:31:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:03 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:31:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:03 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-11T03:31:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:03 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-11T03:31:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:03 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-11T03:31:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:03 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-11T03:31:03.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:03 smithi084 ceph-mon[27254]: Manager daemon y is now available 2024-09-11T03:31:04.864 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-09-11T03:31:04.864 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-11T03:31:04.864 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-11T03:31:04.864 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-09-11T03:31:04.864 INFO:teuthology.orchestra.run.smithi084.stdout:mgr epoch 5 is available 2024-09-11T03:31:04.864 INFO:teuthology.orchestra.run.smithi084.stdout:Setting orchestrator backend to cephadm... 2024-09-11T03:31:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-mon[27254]: mgrmap e7: y(active, since 1.00614s) 2024-09-11T03:31:05.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' 2024-09-11T03:31:05.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' 2024-09-11T03:31:05.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' 2024-09-11T03:31:05.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' 2024-09-11T03:31:05.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:31:05.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:31:05.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-11T03:31:05.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-11T03:31:05.495 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: 2024-09-11T03:31:05.338+0000 7f8b9a155640 -1 log_channel(cephadm) log [ERR] : [11/Sep/2024:03:31:05] ENGINE Error in HTTPServer.serve 2024-09-11T03:31:05.495 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: Traceback (most recent call last): 2024-09-11T03:31:05.495 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T03:31:05.495 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: self._connections.run(self.expiration_interval) 2024-09-11T03:31:05.495 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T03:31:05.496 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: self._run(expiration_interval) 2024-09-11T03:31:05.496 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T03:31:05.496 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T03:31:05.496 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T03:31:05.496 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T03:31:05.496 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T03:31:05.496 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: s = self.context.wrap_socket( 2024-09-11T03:31:05.496 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T03:31:05.497 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: return self.sslsocket_class._create( 2024-09-11T03:31:05.497 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T03:31:05.497 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: self.do_handshake() 2024-09-11T03:31:05.497 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T03:31:05.497 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: self._sslobj.do_handshake() 2024-09-11T03:31:05.497 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T03:31:05.497 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:05 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: 2024-09-11T03:31:06.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:06 smithi084 ceph-mon[27254]: Found migration_current of "None". Setting to last migration. 2024-09-11T03:31:06.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:06 smithi084 ceph-mon[27254]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-11T03:31:06.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:06 smithi084 ceph-mon[27254]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-11T03:31:06.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:06 smithi084 ceph-mon[27254]: [11/Sep/2024:03:31:05] ENGINE Bus STARTING 2024-09-11T03:31:06.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:06 smithi084 ceph-mon[27254]: mgrmap e8: y(active, since 2s) 2024-09-11T03:31:06.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:06 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:31:06.286 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:06 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' 2024-09-11T03:31:06.286 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:06 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:31:07.259 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-11T03:31:07.260 INFO:teuthology.orchestra.run.smithi084.stdout:Generating ssh key... 2024-09-11T03:31:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-mon[27254]: [11/Sep/2024:03:31:05] ENGINE Serving on http://172.21.15.84:8765 2024-09-11T03:31:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-mon[27254]: [11/Sep/2024:03:31:05] ENGINE Serving on https://172.21.15.84:7150 2024-09-11T03:31:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-mon[27254]: [11/Sep/2024:03:31:05] ENGINE Bus STARTED 2024-09-11T03:31:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-mon[27254]: [11/Sep/2024:03:31:05] ENGINE Error in HTTPServer.serve 2024-09-11T03:31:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout: Traceback (most recent call last): 2024-09-11T03:31:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T03:31:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout: self._connections.run(self.expiration_interval) 2024-09-11T03:31:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T03:31:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout: self._run(expiration_interval) 2024-09-11T03:31:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T03:31:07.367 INFO:journalctl@ceph.mon.a.smithi084.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T03:31:07.367 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T03:31:07.367 INFO:journalctl@ceph.mon.a.smithi084.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T03:31:07.367 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T03:31:07.367 INFO:journalctl@ceph.mon.a.smithi084.stdout: s = self.context.wrap_socket( 2024-09-11T03:31:07.367 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T03:31:07.367 INFO:journalctl@ceph.mon.a.smithi084.stdout: return self.sslsocket_class._create( 2024-09-11T03:31:07.367 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T03:31:07.367 INFO:journalctl@ceph.mon.a.smithi084.stdout: self.do_handshake() 2024-09-11T03:31:07.367 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T03:31:07.367 INFO:journalctl@ceph.mon.a.smithi084.stdout: self._sslobj.do_handshake() 2024-09-11T03:31:07.367 INFO:journalctl@ceph.mon.a.smithi084.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T03:31:07.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-mon[27254]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:31:08.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:08 smithi084 ceph-mon[27254]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:31:08.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:08 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' 2024-09-11T03:31:08.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:08 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' 2024-09-11T03:31:08.272 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: Generating public/private rsa key pair. 2024-09-11T03:31:08.272 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: Your identification has been saved in /tmp/tmpimzmhvsy/key 2024-09-11T03:31:08.272 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: Your public key has been saved in /tmp/tmpimzmhvsy/key.pub 2024-09-11T03:31:08.272 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: The key fingerprint is: 2024-09-11T03:31:08.273 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: SHA256:EY3oURK4ONfBDRBBEYI2LhOvEN0FWGdGHruOEYhJAb0 ceph-10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:31:08.273 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: The key's randomart image is: 2024-09-11T03:31:08.273 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: +---[RSA 3072]----+ 2024-09-11T03:31:08.273 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: |o*o=X%@*oo | 2024-09-11T03:31:08.273 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: |==+oo==+o.. | 2024-09-11T03:31:08.273 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: |=+oo.+oo. | 2024-09-11T03:31:08.273 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: |+.E o.o. . | 2024-09-11T03:31:08.273 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: |oo o. . S | 2024-09-11T03:31:08.273 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: |. + | 2024-09-11T03:31:08.273 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: | . . | 2024-09-11T03:31:08.273 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: | | 2024-09-11T03:31:08.273 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: | | 2024-09-11T03:31:08.273 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: +----[SHA256]-----+ 2024-09-11T03:31:09.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:09 smithi084 ceph-mon[27254]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:31:09.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:09 smithi084 ceph-mon[27254]: Generating ssh key... 2024-09-11T03:31:09.597 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5BOEFRRkhce6dQmawFsbFsovUaBB9Rf1s7Wz5RaemzIykhHledMrSFIEnRYvxMNoBjnME6GYpUXfWmDFGpoa5r+67HwAL2AkTb6Oxu+4ciJNC0DNM+pi1nXuq0cB4E5nEVg/6fCaNMNo6fpKuoNqkQupSDHiShJHAISI2o37VGODgSkBjK3+GhVYmsHuDoIS5BKjUQa9MZPaCk6i74w4ArjD8Mq9FFO29sV8A0vcYb4Av3AEKUC82iaU3pAcJzD9rNsLRxrU0RZhw9kxOwbkUWAgN0IYU09qRWgq8o6x+ZJovHHilKbInpO2/LkB7pNIw4JmiOSDQhbJu8oRu4kDU+CKnEuxkGlVXbOF4vn359jeka0nPP0Tbr88vaaLEKqPCPOgxhE6qFyi+oIVmfgYM+zxDHbPzbYA0ctrwpLQHn+9rwmW7UxFGP3t+m3RuzmaTGzrrV9Htb2D0Tk0FpLAlsQPFUoPdsB7fl0rN5+kvV3kEzq93hixaa0INQfQV05E= ceph-10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:31:09.597 INFO:teuthology.orchestra.run.smithi084.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-11T03:31:09.597 INFO:teuthology.orchestra.run.smithi084.stdout:Adding key to root@localhost authorized_keys... 2024-09-11T03:31:09.600 INFO:teuthology.orchestra.run.smithi084.stdout:Adding host smithi084... 2024-09-11T03:31:10.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:10 smithi084 ceph-mon[27254]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:31:11.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:11 smithi084 ceph-mon[27254]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi084", "addr": "172.21.15.84", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:31:11.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:11 smithi084 ceph-mon[27254]: Deploying cephadm binary to smithi084 2024-09-11T03:31:13.996 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout Added host 'smithi084' with addr '172.21.15.84' 2024-09-11T03:31:13.996 INFO:teuthology.orchestra.run.smithi084.stdout:Deploying unmanaged mon service... 2024-09-11T03:31:14.631 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:14 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' 2024-09-11T03:31:14.631 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:14 smithi084 ceph-mon[27254]: Added host smithi084 2024-09-11T03:31:14.632 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:14 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:31:15.415 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-11T03:31:15.415 INFO:teuthology.orchestra.run.smithi084.stdout:Deploying unmanaged mgr service... 2024-09-11T03:31:15.848 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:15 smithi084 ceph-mon[27254]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:31:15.848 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:15 smithi084 ceph-mon[27254]: Saving service mon spec with placement count:5 2024-09-11T03:31:15.848 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:15 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' 2024-09-11T03:31:17.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:17 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' 2024-09-11T03:31:17.381 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:17 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' 2024-09-11T03:31:17.381 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:17 smithi084 ceph-mon[27254]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:31:17.381 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:17 smithi084 ceph-mon[27254]: Saving service mgr spec with placement count:2 2024-09-11T03:31:17.381 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:17 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' 2024-09-11T03:31:17.382 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-11T03:31:19.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:18 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/280115912' entity='client.admin' 2024-09-11T03:31:19.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:18 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' 2024-09-11T03:31:20.240 INFO:teuthology.orchestra.run.smithi084.stdout:Enabling the dashboard module... 2024-09-11T03:31:20.641 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:20 smithi084 ceph-mon[27254]: from='mgr.14118 172.21.15.84:0/1838824909' entity='mgr.y' 2024-09-11T03:31:20.641 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:20 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2108035532' entity='client.admin' 2024-09-11T03:31:21.714 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/514505092' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-11T03:31:21.714 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:21 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: ignoring --setuser ceph since I am not root 2024-09-11T03:31:21.714 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:21 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: ignoring --setgroup ceph since I am not root 2024-09-11T03:31:22.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:22 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/514505092' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-11T03:31:22.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:22 smithi084 ceph-mon[27254]: mgrmap e9: y(active, since 18s) 2024-09-11T03:31:23.420 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-09-11T03:31:23.421 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-11T03:31:23.421 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-11T03:31:23.421 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-11T03:31:23.421 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-11T03:31:23.421 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-09-11T03:31:23.422 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for the mgr to restart... 2024-09-11T03:31:23.422 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mgr epoch 9... 2024-09-11T03:31:23.853 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:23 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/500145564' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-11T03:31:26.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:26 smithi084 ceph-mon[27254]: Active manager daemon y restarted 2024-09-11T03:31:26.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:26 smithi084 ceph-mon[27254]: Activating manager daemon y 2024-09-11T03:31:26.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:26 smithi084 ceph-mon[27254]: osdmap e3: 0 total, 0 up, 0 in 2024-09-11T03:31:26.977 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:26 smithi084 ceph-mon[27254]: mgrmap e10: y(active, starting, since 0.00374599s) 2024-09-11T03:31:26.977 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:31:26.977 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-11T03:31:26.977 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-11T03:31:26.977 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-11T03:31:26.977 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-11T03:31:26.977 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:26 smithi084 ceph-mon[27254]: Manager daemon y is now available 2024-09-11T03:31:27.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:31:27.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-11T03:31:27.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-11T03:31:27.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-mon[27254]: mgrmap e11: y(active, since 1.00605s) 2024-09-11T03:31:27.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-mon[27254]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-11T03:31:28.233 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: 2024-09-11T03:31:27.970+0000 7fab07aa9640 -1 log_channel(cephadm) log [ERR] : [11/Sep/2024:03:31:27] ENGINE Error in HTTPServer.serve 2024-09-11T03:31:28.233 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: Traceback (most recent call last): 2024-09-11T03:31:28.233 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T03:31:28.233 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: self._connections.run(self.expiration_interval) 2024-09-11T03:31:28.233 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T03:31:28.233 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: self._run(expiration_interval) 2024-09-11T03:31:28.233 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T03:31:28.234 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T03:31:28.234 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T03:31:28.234 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T03:31:28.234 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T03:31:28.234 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: s = self.context.wrap_socket( 2024-09-11T03:31:28.234 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T03:31:28.234 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: return self.sslsocket_class._create( 2024-09-11T03:31:28.234 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T03:31:28.234 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: self.do_handshake() 2024-09-11T03:31:28.234 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T03:31:28.234 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: self._sslobj.do_handshake() 2024-09-11T03:31:28.234 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T03:31:28.234 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:31:27 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: 2024-09-11T03:31:28.235 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-09-11T03:31:28.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-11T03:31:28.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-11T03:31:28.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-09-11T03:31:28.236 INFO:teuthology.orchestra.run.smithi084.stdout:mgr epoch 9 is available 2024-09-11T03:31:28.236 INFO:teuthology.orchestra.run.smithi084.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-11T03:31:28.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:28 smithi084 ceph-mon[27254]: [11/Sep/2024:03:31:27] ENGINE Bus STARTING 2024-09-11T03:31:28.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:28 smithi084 ceph-mon[27254]: [11/Sep/2024:03:31:27] ENGINE Serving on http://172.21.15.84:8765 2024-09-11T03:31:28.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:28 smithi084 ceph-mon[27254]: [11/Sep/2024:03:31:27] ENGINE Serving on https://172.21.15.84:7150 2024-09-11T03:31:28.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:28 smithi084 ceph-mon[27254]: [11/Sep/2024:03:31:27] ENGINE Bus STARTED 2024-09-11T03:31:28.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:28 smithi084 ceph-mon[27254]: [11/Sep/2024:03:31:27] ENGINE Error in HTTPServer.serve 2024-09-11T03:31:28.895 INFO:journalctl@ceph.mon.a.smithi084.stdout: Traceback (most recent call last): 2024-09-11T03:31:28.895 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T03:31:28.895 INFO:journalctl@ceph.mon.a.smithi084.stdout: self._connections.run(self.expiration_interval) 2024-09-11T03:31:28.895 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T03:31:28.895 INFO:journalctl@ceph.mon.a.smithi084.stdout: self._run(expiration_interval) 2024-09-11T03:31:28.895 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T03:31:28.895 INFO:journalctl@ceph.mon.a.smithi084.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T03:31:28.896 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T03:31:28.896 INFO:journalctl@ceph.mon.a.smithi084.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T03:31:28.896 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T03:31:28.896 INFO:journalctl@ceph.mon.a.smithi084.stdout: s = self.context.wrap_socket( 2024-09-11T03:31:28.896 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T03:31:28.896 INFO:journalctl@ceph.mon.a.smithi084.stdout: return self.sslsocket_class._create( 2024-09-11T03:31:28.896 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T03:31:28.896 INFO:journalctl@ceph.mon.a.smithi084.stdout: self.do_handshake() 2024-09-11T03:31:28.897 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T03:31:28.897 INFO:journalctl@ceph.mon.a.smithi084.stdout: self._sslobj.do_handshake() 2024-09-11T03:31:28.897 INFO:journalctl@ceph.mon.a.smithi084.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T03:31:29.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:29 smithi084 ceph-mon[27254]: mgrmap e12: y(active, since 2s) 2024-09-11T03:31:29.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:29 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:29.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:29 smithi084 ceph-mon[27254]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:31:29.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:29 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:30.176 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIFEDCCAvigAwIBAgIUN9FtewcaeVipPA9k94ucMp129tcwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkxMTAzMzEyOVoXDTM0\nMDkxMjAzMzEyOVowFzEVMBMGA1UEAwwMMTcyLjIxLjE1Ljg0MIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEA3msjgpqBBZbhhCC8Wkk3la+3X9gtDFiQvZyH\ncTkILgnD/63nMUTAoZo3dFDOtGjln2yXt9W8NBkBxUl1oq8H2aqzml6G96Zllkiw\nWMfZzESFEPW7/Pd0G8n1Ljf8ryQOaWfzdThNxnJXW57o69dWyRxzkvbeVhzm3PKp\niuOsiybE7JzqasEu9xs7IEvKzefcb+OjsfMYh9jA3svvXTHe0/UwZOGZwAoaIp0g\ntbuTMpviqRTeKH21XL5tTkVRw2K8hYhNRdfEgAfKegr5db7yKQj/lrx/lP/WM4Xz\n1N7sFoz92RbQSY4fQwam4HreNvN/dMSpC5lXK1l++Q5wHnqvL4GhUy31yLPRzuuh\nd1UbTU3bG71PWSrqD/usCweQ8JujH4ffcSiKqymcMOJGwIa6npwLiGSV7QAaY1fA\nVFBFOIExaN1iI28Wf+alhnTmbQdr6STKaEIUtT9PWBLs4mSHK+Vro5voH0dRclYh\nmUQb4mZILwWNuhpWYW5aJ28/SWt1o690LZMRf0dvOp5yQWuik6+mL7SOotitVoWp\nkB8LAjREanhJ9uqDdbgVT+E14wAo6e3rLm1ILWcZvGyCrqFWck8WsEfp/wvNEcHA\n9p4vEXR9uVXOj4Vydz/RXTJuieS/CgQ8npxaWPzgAbKun0rCDmIFCtNQkAzmABIH\nL2eZr5UCAwEAAaNUMFIwQgYDVR0RBDswOYIec21pdGhpMDg0LmZyb250LnNlcGlh\nLmNlcGguY29tghFkYXNoYm9hcmRfc2VydmVyc4cErBUPVDAMBgNVHRMBAf8EAjAA\nMA0GCSqGSIb3DQEBCwUAA4ICAQAZ2gXrpYRbfWrX2ZJa98qWI665YU1Nzm7XCiet\ncrb/ranHEnh14b66ETB11rT02qTnXQVY1nUPBc39gR/ZGE8LXiL0LoDtEthdnpdI\nm7p08+Fru3q7JkuilEKx8O7JQptOV/YNFQcAhpqt9+uJoBU4pzN+jqSkPu8vqfFP\n7jp6wCAPcU4sXevDn7B7ZEjyp0ZrwzaoGodPdfzhT/uwOKw5agCcDzioeVFO6X+4\nzevdjJWCSqY3NXAuqCrlVKfcHODGmwG9zaHVLMlYS4QhipVrosPj84mIi7CJ+SZD\n5JSaR0CEv72mdQpU68ytCkArjOnkXQ8MQaV9XBfkfZcsOfHyccH6k6tzPH4U8L3T\nrhgwobDSyBx3heZJ8lizIqM4k1cyDRr45ctBvIifJmO8huRZllxJ6Hvb6e0662TK\ntuSOql01Y0pFFnpWDo+o/fsadfCmEooRo+hFgWXxIB5y56Zvjbqj9ASCbJsn7IQO\nHRv8Jkf5nuZyi2mE2jF4uWfZmNk60oI2llapUDO1Y9GuAS565+D4JMrgqa2pTW4V\nK3QbRL0H4osTYOkGqb0ZUQeMLDuG0EZaeZrHrRD5qhwyDr2/a7mUc0brE/5htrNn\nPV4eVyGgFxp2RKhdeeYgaso59Mnay2MmYWeRnBfvnbrNFGVjVDrs8+DzVeaPfHZ5\n8OZ0+w==\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKQIBAAKCAgEA3msjgpqBBZbhhCC8Wkk3la+3X9gtDFiQvZyHcTkILgnD/63n\nMUTAoZo3dFDOtGjln2yXt9W8NBkBxUl1oq8H2aqzml6G96ZllkiwWMfZzESFEPW7\n/Pd0G8n1Ljf8ryQOaWfzdThNxnJXW57o69dWyRxzkvbeVhzm3PKpiuOsiybE7Jzq\nasEu9xs7IEvKzefcb+OjsfMYh9jA3svvXTHe0/UwZOGZwAoaIp0gtbuTMpviqRTe\nKH21XL5tTkVRw2K8hYhNRdfEgAfKegr5db7yKQj/lrx/lP/WM4Xz1N7sFoz92RbQ\nSY4fQwam4HreNvN/dMSpC5lXK1l++Q5wHnqvL4GhUy31yLPRzuuhd1UbTU3bG71P\nWSrqD/usCweQ8JujH4ffcSiKqymcMOJGwIa6npwLiGSV7QAaY1fAVFBFOIExaN1i\nI28Wf+alhnTmbQdr6STKaEIUtT9PWBLs4mSHK+Vro5voH0dRclYhmUQb4mZILwWN\nuhpWYW5aJ28/SWt1o690LZMRf0dvOp5yQWuik6+mL7SOotitVoWpkB8LAjREanhJ\n9uqDdbgVT+E14wAo6e3rLm1ILWcZvGyCrqFWck8WsEfp/wvNEcHA9p4vEXR9uVXO\nj4Vydz/RXTJuieS/CgQ8npxaWPzgAbKun0rCDmIFCtNQkAzmABIHL2eZr5UCAwEA\nAQKCAgAo0BSXR9Yiii3uEpq1X4eLWE/IDtEvWYYPwQwKbuptFj4oqcbJX+IQFU4V\nMinH6ZKvFxxc76OmFKuiQclL9Z21Z5Em468U/OvZsJNpaxywUqdvNgRNljOwULYE\nieH8WluppFlEenUtE7ird4SqmBxU8XtUraedOhbz1rRdJ8t2ykGq4F1Ugwfk8JKA\nKeA4OdJVlrooMmEfNlsn1tigq4DFuJFthYcyubfTf60ro9SOHhQwrgNY/z47GzdP\n/EDVX0YUI694QDBOxP82t1JQtJ3XgU7zIDoyA2MRF8SnHFPKjIsdHYZRZPqgEDWp\nbepeP361FMwnNHdOHVFDDs7sjNsdvzHJvz2qb9WAC6rKSSAATHEIC2TTGWPTuqrc\nt48BhxjjEPsP9aG8fV0e/UuRcloxRgSLnQMwbqu24cduhHtPfaC/4f5DXEoJVZlC\nrBNfj+lTs+AkmEcfc4Aua6aO0ltYdWrT/D5Q5byqZjWyI94aj2LTSoNNSbS3o0V8\nj00AfCw8ox9dqMMCEUYM6i51EuhToTk3GP3CNz6u4d3thYPWFc8+236OzuY3Myh4\nV8tAeEM+6QWPz62ZIXyFzaEVp8QEmHqpcvpTn9eVjRyWmgc9VHvOzBvFmgjEwUXr\nd06lsiWNjRPkjZEfg7+LQ+7lf7GI9+fvjxM8kCDg1E3e1dk22QKCAQEA8X90V6LX\nkAD35RujeZSj54AmZIzMsNtKZIAopdOegjQWKtKdZAh+4y/+46R7irmKoIFJ6psS\n13tjVyZi98GxgE9MlSByWcmWWx0nQpKEyw5a3nwVmD9C49C8/pw8HX9iR7HU9j6r\nPAqJWfPYnF1onxTMrAOFzsV0ESL8CtBP13Q+dE/kXer1vf/b2VAVqsuO47VzfQXh\nFORHOeHoC2TdBDgQAyexNldciRi6L95OHiN1+yxgHgQdQ/rkmWqsJM70RSaT/vYt\nIu5qD5gjGCJDQHvuQP9OJGdTmP+qledFnJyxm42oPxIlobIJ7D5N4kIkTC7dJ7Ch\nQBDA3ifXcMK0WQKCAQEA68ZgmyrelnTabY2dUgMnTo8wHOWDjCCOHXD8impdvy/B\na1pImyiLROSUgUvXfjfDa5jFJBVFdkqSchGJKnNx84aDyc4J/W8Q0WF79FI49pIn\ntjPUv6YkAsk8lOtmlM/bscMUfua+JouN95AGXveOBIA+q3aib/UOmCnn0xMjo4ps\nO3oZ5KG6KvaXNvNdVGyMOOjrz81yn16j2y12OOfJ94ySFnV2m5b/Xp/zlOZY0Gwm\nym60OiaDt3nnzZUqFwlkfmL1etxb2ZzaFGNtN8+FdrxMkLJqsnzNyXjCE9EQzU3u\n1Hw94u1wX+jUopIn+wLfYloRGiY2X0GMeRGrg3sdnQKCAQEA0EKOzUiURw5euUhV\nePLTSj/8W/gESXxDMlpHM4JEjppmh3Vr36ln63ongJ8HH0IUp4f3e4dR/1vb9nsX\nG/wAm9tG+s+pbX7n2BlTExkRxZ7ShE1TviDICXiok5WeTf3dfmCAe/7dYoDfpCAi\nq+3t3EpumuTKzt7+0gxueNDeGbqYZWvET63hUIHGUSi8FrdGuV5wgt/AFGksd2E2\n4CSSaSIDxQTE0U0TcvNNFRzMFzeTdhD1rszzg/1oiFkyrOeAReYX+ObJU/dwpZQY\np6Zcgs6dT5YcCE7HDAlnTbDNGlsS20eVURbQNxo3bnDob3eEn+JGPAUZegs4mByZ\nsRdnuQKCAQBijkslsZbX58LClrbTaHQO17Xx7lK9yYH+dpNqPKoxWAIPIO9GxHzq\n8G3PWmea4C51FKgbvDSMDQIIYqImt5lDhdAjFFiCsuGh1JEfARvEnjWxaRsefZGC\nBKKpUXlQ7U5VkUlI/qUvsZhHitHO8sZPxVftDqs6B4LxQlsglViJbTxu5ExckOok\nOW3eT36M4s7+nwarl6K5UbNG1Esd4G5BZ2ZNsw5AXa8j29LTRDKN9ReF+R6nTCA1\nGuU1v4uXk5qJZk5c7C4M2g8jBKjL8phEmNOcjKgjMh5HgI4IAJV4bIJQGhs4d65I\nuSfhSR0jRQEhuKp7dsu1p83Tcfpy/IzVAoIBAQCzXgemp817fbVMmANCnidc4vct\n/eTRXz7sfzkEJzWA9SpCxD7YOtKnGb55ekqyCsOowRY7naCWDf5e6oKVWX5jAbus\n8npc3uqa1+kUAZkmfq7DN0Ao/yQDyK2+vK/AvW5aqOFroJaTN1R5YFn+/vkm/S28\nIwUHYA2nRHY1/BudqvvJuQzsr3pXUK6FqUI3wX08IbiI4/mkN8SL0LF7yQOem5Ob\niFYs2vTOFzXvGE+gwzxMdaeQdDPvW8ThcTFk+Qwp4ES7Uue7bGK5hYZ3UhlA72YC\nSRJG1jwKzG5Zbx8jKxmAQ2O5HLngdpsijtakFUkpgA7a7tl3KjjR2W69y7fD\n-----END RSA PRIVATE KEY-----\n"} 2024-09-11T03:31:31.333 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-11T03:31:32.103 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:31 smithi084 ceph-mon[27254]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:31:32.103 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:31 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:33.037 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-11T03:31:33.037 INFO:teuthology.orchestra.run.smithi084.stdout:Creating initial admin user... 2024-09-11T03:31:33.436 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:33 smithi084 ceph-mon[27254]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:31:33.437 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:33 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:34.392 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$SFl0uckqPuxz6oPVjOVHNO597VL1ziiFzbFHbfX49SfDygXe2WF7u", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726025493, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-11T03:31:34.393 INFO:teuthology.orchestra.run.smithi084.stdout:Fetching dashboard port number... 2024-09-11T03:31:35.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:34 smithi084 ceph-mon[27254]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:31:35.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:34 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:35.574 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 8443 2024-09-11T03:31:35.583 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-11T03:31:35.584 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout disabled 2024-09-11T03:31:35.591 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-11T03:31:35.591 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-09-11T03:31:35.591 INFO:teuthology.orchestra.run.smithi084.stdout:firewalld.service is not enabled 2024-09-11T03:31:35.591 INFO:teuthology.orchestra.run.smithi084.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-11T03:31:35.593 INFO:teuthology.orchestra.run.smithi084.stdout:Ceph Dashboard is now available at: 2024-09-11T03:31:35.593 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:31:35.593 INFO:teuthology.orchestra.run.smithi084.stdout: URL: https://smithi084.front.sepia.ceph.com:8443/ 2024-09-11T03:31:35.593 INFO:teuthology.orchestra.run.smithi084.stdout: User: admin 2024-09-11T03:31:35.593 INFO:teuthology.orchestra.run.smithi084.stdout: Password: 4llhriimfa 2024-09-11T03:31:35.594 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:31:35.594 INFO:teuthology.orchestra.run.smithi084.stdout:Saving cluster configuration to /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config directory 2024-09-11T03:31:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:35 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2272872916' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-11T03:31:36.888 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-11T03:31:36.889 INFO:teuthology.orchestra.run.smithi084.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-11T03:31:36.889 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:31:36.889 INFO:teuthology.orchestra.run.smithi084.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-11T03:31:36.889 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:31:36.889 INFO:teuthology.orchestra.run.smithi084.stdout:Or, if you are only running a single cluster on this host: 2024-09-11T03:31:36.889 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:31:36.889 INFO:teuthology.orchestra.run.smithi084.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-11T03:31:36.889 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:31:36.889 INFO:teuthology.orchestra.run.smithi084.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-11T03:31:36.889 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:31:36.889 INFO:teuthology.orchestra.run.smithi084.stdout: ceph telemetry on 2024-09-11T03:31:36.890 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:31:36.890 INFO:teuthology.orchestra.run.smithi084.stdout:For more information see: 2024-09-11T03:31:36.890 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:31:36.890 INFO:teuthology.orchestra.run.smithi084.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-11T03:31:36.890 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:31:36.890 INFO:teuthology.orchestra.run.smithi084.stdout:Bootstrap complete. 2024-09-11T03:31:36.927 INFO:tasks.cephadm:Fetching config... 2024-09-11T03:31:36.928 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-11T03:31:36.928 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-11T03:31:36.943 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-11T03:31:36.943 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-11T03:31:36.944 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-11T03:31:37.000 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-11T03:31:37.000 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-11T03:31:37.000 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-11T03:31:37.066 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-11T03:31:37.066 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-11T03:31:37.066 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-11T03:31:37.121 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-11T03:31:37.121 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5BOEFRRkhce6dQmawFsbFsovUaBB9Rf1s7Wz5RaemzIykhHledMrSFIEnRYvxMNoBjnME6GYpUXfWmDFGpoa5r+67HwAL2AkTb6Oxu+4ciJNC0DNM+pi1nXuq0cB4E5nEVg/6fCaNMNo6fpKuoNqkQupSDHiShJHAISI2o37VGODgSkBjK3+GhVYmsHuDoIS5BKjUQa9MZPaCk6i74w4ArjD8Mq9FFO29sV8A0vcYb4Av3AEKUC82iaU3pAcJzD9rNsLRxrU0RZhw9kxOwbkUWAgN0IYU09qRWgq8o6x+ZJovHHilKbInpO2/LkB7pNIw4JmiOSDQhbJu8oRu4kDU+CKnEuxkGlVXbOF4vn359jeka0nPP0Tbr88vaaLEKqPCPOgxhE6qFyi+oIVmfgYM+zxDHbPzbYA0ctrwpLQHn+9rwmW7UxFGP3t+m3RuzmaTGzrrV9Htb2D0Tk0FpLAlsQPFUoPdsB7fl0rN5+kvV3kEzq93hixaa0INQfQV05E= ceph-10fe7946-6fee-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-11T03:31:37.195 INFO:teuthology.orchestra.run.smithi084.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5BOEFRRkhce6dQmawFsbFsovUaBB9Rf1s7Wz5RaemzIykhHledMrSFIEnRYvxMNoBjnME6GYpUXfWmDFGpoa5r+67HwAL2AkTb6Oxu+4ciJNC0DNM+pi1nXuq0cB4E5nEVg/6fCaNMNo6fpKuoNqkQupSDHiShJHAISI2o37VGODgSkBjK3+GhVYmsHuDoIS5BKjUQa9MZPaCk6i74w4ArjD8Mq9FFO29sV8A0vcYb4Av3AEKUC82iaU3pAcJzD9rNsLRxrU0RZhw9kxOwbkUWAgN0IYU09qRWgq8o6x+ZJovHHilKbInpO2/LkB7pNIw4JmiOSDQhbJu8oRu4kDU+CKnEuxkGlVXbOF4vn359jeka0nPP0Tbr88vaaLEKqPCPOgxhE6qFyi+oIVmfgYM+zxDHbPzbYA0ctrwpLQHn+9rwmW7UxFGP3t+m3RuzmaTGzrrV9Htb2D0Tk0FpLAlsQPFUoPdsB7fl0rN5+kvV3kEzq93hixaa0INQfQV05E= ceph-10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:31:37.207 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5BOEFRRkhce6dQmawFsbFsovUaBB9Rf1s7Wz5RaemzIykhHledMrSFIEnRYvxMNoBjnME6GYpUXfWmDFGpoa5r+67HwAL2AkTb6Oxu+4ciJNC0DNM+pi1nXuq0cB4E5nEVg/6fCaNMNo6fpKuoNqkQupSDHiShJHAISI2o37VGODgSkBjK3+GhVYmsHuDoIS5BKjUQa9MZPaCk6i74w4ArjD8Mq9FFO29sV8A0vcYb4Av3AEKUC82iaU3pAcJzD9rNsLRxrU0RZhw9kxOwbkUWAgN0IYU09qRWgq8o6x+ZJovHHilKbInpO2/LkB7pNIw4JmiOSDQhbJu8oRu4kDU+CKnEuxkGlVXbOF4vn359jeka0nPP0Tbr88vaaLEKqPCPOgxhE6qFyi+oIVmfgYM+zxDHbPzbYA0ctrwpLQHn+9rwmW7UxFGP3t+m3RuzmaTGzrrV9Htb2D0Tk0FpLAlsQPFUoPdsB7fl0rN5+kvV3kEzq93hixaa0INQfQV05E= ceph-10fe7946-6fee-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-11T03:31:37.260 INFO:teuthology.orchestra.run.smithi121.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5BOEFRRkhce6dQmawFsbFsovUaBB9Rf1s7Wz5RaemzIykhHledMrSFIEnRYvxMNoBjnME6GYpUXfWmDFGpoa5r+67HwAL2AkTb6Oxu+4ciJNC0DNM+pi1nXuq0cB4E5nEVg/6fCaNMNo6fpKuoNqkQupSDHiShJHAISI2o37VGODgSkBjK3+GhVYmsHuDoIS5BKjUQa9MZPaCk6i74w4ArjD8Mq9FFO29sV8A0vcYb4Av3AEKUC82iaU3pAcJzD9rNsLRxrU0RZhw9kxOwbkUWAgN0IYU09qRWgq8o6x+ZJovHHilKbInpO2/LkB7pNIw4JmiOSDQhbJu8oRu4kDU+CKnEuxkGlVXbOF4vn359jeka0nPP0Tbr88vaaLEKqPCPOgxhE6qFyi+oIVmfgYM+zxDHbPzbYA0ctrwpLQHn+9rwmW7UxFGP3t+m3RuzmaTGzrrV9Htb2D0Tk0FpLAlsQPFUoPdsB7fl0rN5+kvV3kEzq93hixaa0INQfQV05E= ceph-10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:31:37.273 DEBUG:teuthology.orchestra.run.smithi175:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5BOEFRRkhce6dQmawFsbFsovUaBB9Rf1s7Wz5RaemzIykhHledMrSFIEnRYvxMNoBjnME6GYpUXfWmDFGpoa5r+67HwAL2AkTb6Oxu+4ciJNC0DNM+pi1nXuq0cB4E5nEVg/6fCaNMNo6fpKuoNqkQupSDHiShJHAISI2o37VGODgSkBjK3+GhVYmsHuDoIS5BKjUQa9MZPaCk6i74w4ArjD8Mq9FFO29sV8A0vcYb4Av3AEKUC82iaU3pAcJzD9rNsLRxrU0RZhw9kxOwbkUWAgN0IYU09qRWgq8o6x+ZJovHHilKbInpO2/LkB7pNIw4JmiOSDQhbJu8oRu4kDU+CKnEuxkGlVXbOF4vn359jeka0nPP0Tbr88vaaLEKqPCPOgxhE6qFyi+oIVmfgYM+zxDHbPzbYA0ctrwpLQHn+9rwmW7UxFGP3t+m3RuzmaTGzrrV9Htb2D0Tk0FpLAlsQPFUoPdsB7fl0rN5+kvV3kEzq93hixaa0INQfQV05E= ceph-10fe7946-6fee-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-11T03:31:37.334 INFO:teuthology.orchestra.run.smithi175.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5BOEFRRkhce6dQmawFsbFsovUaBB9Rf1s7Wz5RaemzIykhHledMrSFIEnRYvxMNoBjnME6GYpUXfWmDFGpoa5r+67HwAL2AkTb6Oxu+4ciJNC0DNM+pi1nXuq0cB4E5nEVg/6fCaNMNo6fpKuoNqkQupSDHiShJHAISI2o37VGODgSkBjK3+GhVYmsHuDoIS5BKjUQa9MZPaCk6i74w4ArjD8Mq9FFO29sV8A0vcYb4Av3AEKUC82iaU3pAcJzD9rNsLRxrU0RZhw9kxOwbkUWAgN0IYU09qRWgq8o6x+ZJovHHilKbInpO2/LkB7pNIw4JmiOSDQhbJu8oRu4kDU+CKnEuxkGlVXbOF4vn359jeka0nPP0Tbr88vaaLEKqPCPOgxhE6qFyi+oIVmfgYM+zxDHbPzbYA0ctrwpLQHn+9rwmW7UxFGP3t+m3RuzmaTGzrrV9Htb2D0Tk0FpLAlsQPFUoPdsB7fl0rN5+kvV3kEzq93hixaa0INQfQV05E= ceph-10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:31:37.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-11T03:31:37.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1012849622' entity='client.admin' 2024-09-11T03:31:37.592 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:31:39.352 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-11T03:31:39.352 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-11T03:31:39.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:39 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/327472998' entity='client.admin' 2024-09-11T03:31:39.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:39 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:39.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:39 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:39.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:39 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi084", "name": "osd_memory_target"} : dispatch 2024-09-11T03:31:39.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:39 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:39.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:39 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:31:39.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:39 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:39.847 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:31:40.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:40 smithi084 ceph-mon[27254]: mgrmap e13: y(active, since 12s) 2024-09-11T03:31:41.256 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi121 2024-09-11T03:31:41.257 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-11T03:31:41.257 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/etc/ceph/ceph.conf 2024-09-11T03:31:41.277 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-11T03:31:41.277 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-11T03:31:41.335 INFO:tasks.cephadm:Adding host smithi121 to orchestrator... 2024-09-11T03:31:41.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch host add smithi121 2024-09-11T03:31:41.710 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:31:41.830 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:41 smithi084 ceph-mon[27254]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:31:41.830 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:41 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:41.830 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:41 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:31:41.830 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:41 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:31:41.831 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:41 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:31:41.831 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:41 smithi084 ceph-mon[27254]: Updating smithi084:/etc/ceph/ceph.conf 2024-09-11T03:31:41.831 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:41 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:41.831 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:41 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:41.831 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:41 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:42.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:42 smithi084 ceph-mon[27254]: Updating smithi084:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T03:31:42.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:42 smithi084 ceph-mon[27254]: Updating smithi084:/etc/ceph/ceph.client.admin.keyring 2024-09-11T03:31:42.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:42 smithi084 ceph-mon[27254]: Updating smithi084:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T03:31:43.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:43 smithi084 ceph-mon[27254]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi121", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:31:44.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:44 smithi084 ceph-mon[27254]: Deploying cephadm binary to smithi121 2024-09-11T03:31:45.431 INFO:teuthology.orchestra.run.smithi084.stdout:Added host 'smithi121' with addr '172.21.15.121' 2024-09-11T03:31:46.054 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch host ls --format=json 2024-09-11T03:31:46.293 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:31:46.700 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:46 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:46.700 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:46 smithi084 ceph-mon[27254]: Added host smithi121 2024-09-11T03:31:46.700 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:46 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:31:46.700 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:46 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:46.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:46 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:46.977 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:31:46.978 INFO:teuthology.orchestra.run.smithi084.stdout:[{"addr": "172.21.15.84", "hostname": "smithi084", "labels": [], "status": ""}, {"addr": "172.21.15.121", "hostname": "smithi121", "labels": [], "status": ""}] 2024-09-11T03:31:47.599 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi175 2024-09-11T03:31:47.599 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-11T03:31:47.599 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/etc/ceph/ceph.conf 2024-09-11T03:31:47.618 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-11T03:31:47.618 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-11T03:31:47.676 INFO:tasks.cephadm:Adding host smithi175 to orchestrator... 2024-09-11T03:31:47.677 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch host add smithi175 2024-09-11T03:31:47.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:47 smithi084 ceph-mon[27254]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:31:47.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:47 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:47.915 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:31:48.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:48 smithi084 ceph-mon[27254]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:31:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:49 smithi084 ceph-mon[27254]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi175", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:31:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:49 smithi084 ceph-mon[27254]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:31:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:49 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:49 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:49 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:49 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:49 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi121", "name": "osd_memory_target"} : dispatch 2024-09-11T03:31:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:49 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:31:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:49 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:31:49.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:49 smithi084 ceph-mon[27254]: Updating smithi121:/etc/ceph/ceph.conf 2024-09-11T03:31:49.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:49 smithi084 ceph-mon[27254]: Deploying cephadm binary to smithi175 2024-09-11T03:31:49.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:49 smithi084 ceph-mon[27254]: Updating smithi121:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T03:31:51.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:51 smithi084 ceph-mon[27254]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-09-11T03:31:51.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:51 smithi084 ceph-mon[27254]: Updating smithi121:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T03:31:51.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:51 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:51.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:51 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:51.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:51 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:51.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:51 smithi084 ceph-mon[27254]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:31:51.745 INFO:teuthology.orchestra.run.smithi084.stdout:Added host 'smithi175' with addr '172.21.15.175' 2024-09-11T03:31:52.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch host ls --format=json 2024-09-11T03:31:52.592 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:31:52.970 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:52 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:52.970 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:52 smithi084 ceph-mon[27254]: Added host smithi175 2024-09-11T03:31:52.970 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:52 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:31:52.971 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:52 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:52.971 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:52 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:52.971 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:52 smithi084 ceph-mon[27254]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:31:53.241 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:31:53.241 INFO:teuthology.orchestra.run.smithi084.stdout:[{"addr": "172.21.15.84", "hostname": "smithi084", "labels": [], "status": ""}, {"addr": "172.21.15.121", "hostname": "smithi121", "labels": [], "status": ""}, {"addr": "172.21.15.175", "hostname": "smithi175", "labels": [], "status": ""}] 2024-09-11T03:31:53.844 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-11T03:31:53.844 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd crush tunables default 2024-09-11T03:31:54.081 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:31:54.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:54 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:54.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:54 smithi084 ceph-mon[27254]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:31:55.105 INFO:teuthology.orchestra.run.smithi084.stderr:adjusted tunables profile to default 2024-09-11T03:31:55.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:55 smithi084 ceph-mon[27254]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:31:55.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:55 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3152965123' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-11T03:31:55.697 INFO:tasks.cephadm:Adding mon.a on smithi084 2024-09-11T03:31:55.698 INFO:tasks.cephadm:Adding mon.b on smithi121 2024-09-11T03:31:55.698 INFO:tasks.cephadm:Adding mon.c on smithi175 2024-09-11T03:31:55.698 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch apply mon '3;smithi084:172.21.15.84=a;smithi121:172.21.15.121=b;smithi175:172.21.15.175=c' 2024-09-11T03:31:56.034 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-11T03:31:56.085 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-11T03:31:56.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3152965123' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-11T03:31:56.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:56 smithi084 ceph-mon[27254]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T03:31:56.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:56.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:56.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:56.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:56.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi175", "name": "osd_memory_target"} : dispatch 2024-09-11T03:31:56.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi175", "name": "osd_memory_target"} : dispatch 2024-09-11T03:31:56.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:31:56.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:31:56.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:31:56.814 INFO:teuthology.orchestra.run.smithi175.stdout:Scheduled mon update... 2024-09-11T03:31:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:57 smithi084 ceph-mon[27254]: Updating smithi175:/etc/ceph/ceph.conf 2024-09-11T03:31:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:57 smithi084 ceph-mon[27254]: Updating smithi175:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T03:31:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:57 smithi084 ceph-mon[27254]: Updating smithi175:/etc/ceph/ceph.client.admin.keyring 2024-09-11T03:31:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:57 smithi084 ceph-mon[27254]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:31:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:57.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:57.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:31:57.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-11T03:31:57.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:31:57.407 DEBUG:teuthology.orchestra.run.smithi121:mon.b> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mon.b.service 2024-09-11T03:31:57.410 DEBUG:teuthology.orchestra.run.smithi175:mon.c> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mon.c.service 2024-09-11T03:31:57.412 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-11T03:31:57.412 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-11T03:31:58.370 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T03:31:58.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:58 smithi084 ceph-mon[27254]: Updating smithi175:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T03:31:58.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:58 smithi084 ceph-mon[27254]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi084:172.21.15.84=a;smithi121:172.21.15.121=b;smithi175:172.21.15.175=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:31:58.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:58 smithi084 ceph-mon[27254]: Saving service mon spec with placement smithi084:172.21.15.84=a;smithi121:172.21.15.121=b;smithi175:172.21.15.175=c;count:3 2024-09-11T03:31:58.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:58 smithi084 ceph-mon[27254]: Deploying daemon mon.c on smithi175 2024-09-11T03:31:59.353 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:31:59.354 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"10fe7946-6fee-11ef-bcea-c7b262605968","modified":"2024-09-11T03:30:27.618796Z","created":"2024-09-11T03:30:27.618796Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-11T03:31:59.354 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-11T03:31:59.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:31:59 smithi084 ceph-mon[27254]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:00.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:00 smithi084 ceph-mon[27254]: from='client.? 172.21.15.175:0/744670426' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-11T03:32:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:01 smithi084 ceph-mon[27254]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:01.713 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:01 smithi175 ceph-mon[30894]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-11T03:32:01.760 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-11T03:32:01.760 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-11T03:32:01.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:01 smithi175 bash[30876]: 96a16f0d967ce68b2b6845c16304edfd136f250360fc3a8d79e7f24896288225 2024-09-11T03:32:01.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:01 smithi175 systemd[1]: Started Ceph mon.c for 10fe7946-6fee-11ef-bcea-c7b262605968. 2024-09-11T03:32:02.009 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:32:07.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:32:07.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:07.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: mon.a calling monitor election 2024-09-11T03:32:07.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:07.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:07.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:07.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: mon.c calling monitor election 2024-09-11T03:32:07.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:07.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:07.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:07.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:07.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:07.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-11T03:32:07.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: monmap epoch 2 2024-09-11T03:32:07.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: fsid 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:32:07.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: last_changed 2024-09-11T03:32:01.686490+0000 2024-09-11T03:32:07.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: created 2024-09-11T03:30:27.618796+0000 2024-09-11T03:32:07.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: min_mon_release 19 (squid) 2024-09-11T03:32:07.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: election_strategy: 1 2024-09-11T03:32:07.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-09-11T03:32:07.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: 1: [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] mon.c 2024-09-11T03:32:07.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: fsmap 2024-09-11T03:32:07.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T03:32:07.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: mgrmap e13: y(active, since 40s) 2024-09-11T03:32:07.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: overall HEALTH_OK 2024-09-11T03:32:07.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:07.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:07.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:07.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-11T03:32:07.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:07.071 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:32:07.071 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":2,"fsid":"10fe7946-6fee-11ef-bcea-c7b262605968","modified":"2024-09-11T03:32:01.686490Z","created":"2024-09-11T03:30:27.618796Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:3300","nonce":0},{"type":"v1","addr":"172.21.15.175:6789","nonce":0}]},"addr":"172.21.15.175:6789/0","public_addr":"172.21.15.175:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-11T03:32:07.071 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 2 2024-09-11T03:32:07.071 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:32:07.072 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:07.072 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: mon.a calling monitor election 2024-09-11T03:32:07.072 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:07.072 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:07.072 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:07.072 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: mon.c calling monitor election 2024-09-11T03:32:07.072 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:07.072 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:07.072 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:07.072 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:07.073 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:07.073 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-11T03:32:07.073 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: monmap epoch 2 2024-09-11T03:32:07.073 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: fsid 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:32:07.073 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: last_changed 2024-09-11T03:32:01.686490+0000 2024-09-11T03:32:07.073 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: created 2024-09-11T03:30:27.618796+0000 2024-09-11T03:32:07.073 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: min_mon_release 19 (squid) 2024-09-11T03:32:07.073 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: election_strategy: 1 2024-09-11T03:32:07.073 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-09-11T03:32:07.073 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: 1: [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] mon.c 2024-09-11T03:32:07.073 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: fsmap 2024-09-11T03:32:07.073 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T03:32:07.073 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: mgrmap e13: y(active, since 40s) 2024-09-11T03:32:07.074 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: overall HEALTH_OK 2024-09-11T03:32:07.074 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:07.074 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:07.074 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:07.074 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-11T03:32:07.074 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:07.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:07 smithi175 ceph-mon[30894]: Deploying daemon mon.b on smithi121 2024-09-11T03:32:07.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.175:0/2367191011' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-11T03:32:07.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:08.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:07 smithi084 ceph-mon[27254]: Deploying daemon mon.b on smithi121 2024-09-11T03:32:08.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.175:0/2367191011' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-11T03:32:08.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:08.682 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-11T03:32:08.682 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-11T03:32:08.934 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:32:08.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:08 smithi175 ceph-mon[30894]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:09.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:08 smithi084 ceph-mon[27254]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:09.585 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:32:09.586 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":2,"fsid":"10fe7946-6fee-11ef-bcea-c7b262605968","modified":"2024-09-11T03:32:01.686490Z","created":"2024-09-11T03:30:27.618796Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:3300","nonce":0},{"type":"v1","addr":"172.21.15.175:6789","nonce":0}]},"addr":"172.21.15.175:6789/0","public_addr":"172.21.15.175:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-11T03:32:09.586 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 2 2024-09-11T03:32:09.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:09 smithi175 ceph-mon[30894]: from='client.? 172.21.15.175:0/1892807782' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-11T03:32:10.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:09 smithi084 ceph-mon[27254]: from='client.? 172.21.15.175:0/1892807782' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-11T03:32:10.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 podman[30709]: 2024-09-11 03:32:10.275031483 +0000 UTC m=+0.333974487 container start d891b9a0f3f53fe017a8185fd83c1cf2748533f0a7ac79779d10afcd8aceecde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-b, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T03:32:10.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: set uid:gid to 167:167 (ceph:ceph) 2024-09-11T03:32:10.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev), process ceph-mon, pid 2 2024-09-11T03:32:10.297 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: pidfile_write: ignore empty --pid-file 2024-09-11T03:32:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: load: jerasure load: lrc 2024-09-11T03:32:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: RocksDB version: 7.9.2 2024-09-11T03:32:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Git sha 0 2024-09-11T03:32:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Compile date 2024-09-10 08:02:51 2024-09-11T03:32:10.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: DB SUMMARY 2024-09-11T03:32:10.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: DB Session ID: TD1AVINWMOV6Q75Z9EIV 2024-09-11T03:32:10.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: CURRENT file: CURRENT 2024-09-11T03:32:10.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: IDENTITY file: IDENTITY 2024-09-11T03:32:10.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-11T03:32:10.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-11T03:32:10.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-09-11T03:32:10.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.error_if_exists: 0 2024-09-11T03:32:10.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.create_if_missing: 0 2024-09-11T03:32:10.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.paranoid_checks: 1 2024-09-11T03:32:10.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-11T03:32:10.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-11T03:32:10.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-11T03:32:10.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.env: 0x56226222d180 2024-09-11T03:32:10.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.fs: PosixFileSystem 2024-09-11T03:32:10.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.info_log: 0x562262b8d700 2024-09-11T03:32:10.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_file_opening_threads: 16 2024-09-11T03:32:10.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.statistics: (nil) 2024-09-11T03:32:10.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.use_fsync: 0 2024-09-11T03:32:10.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_log_file_size: 0 2024-09-11T03:32:10.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-11T03:32:10.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-11T03:32:10.675 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.keep_log_file_num: 1000 2024-09-11T03:32:10.675 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.recycle_log_file_num: 0 2024-09-11T03:32:10.675 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.allow_fallocate: 1 2024-09-11T03:32:10.675 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.allow_mmap_reads: 0 2024-09-11T03:32:10.675 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.allow_mmap_writes: 0 2024-09-11T03:32:10.675 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.use_direct_reads: 0 2024-09-11T03:32:10.675 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-11T03:32:10.675 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.create_missing_column_families: 0 2024-09-11T03:32:10.675 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.db_log_dir: 2024-09-11T03:32:10.675 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.wal_dir: 2024-09-11T03:32:10.676 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-11T03:32:10.676 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-11T03:32:10.676 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-11T03:32:10.676 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-11T03:32:10.676 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-11T03:32:10.676 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-11T03:32:10.676 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.advise_random_on_open: 1 2024-09-11T03:32:10.676 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.db_write_buffer_size: 0 2024-09-11T03:32:10.676 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.write_buffer_manager: 0x562262b91b80 2024-09-11T03:32:10.676 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-11T03:32:10.676 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-11T03:32:10.676 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-11T03:32:10.676 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.rate_limiter: (nil) 2024-09-11T03:32:10.676 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-11T03:32:10.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.wal_recovery_mode: 2 2024-09-11T03:32:10.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.enable_thread_tracking: 0 2024-09-11T03:32:10.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.enable_pipelined_write: 0 2024-09-11T03:32:10.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.unordered_write: 0 2024-09-11T03:32:10.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-11T03:32:10.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-11T03:32:10.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-11T03:32:10.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-11T03:32:10.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.row_cache: None 2024-09-11T03:32:10.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.wal_filter: None 2024-09-11T03:32:10.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-11T03:32:10.677 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.allow_ingest_behind: 0 2024-09-11T03:32:10.678 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.two_write_queues: 0 2024-09-11T03:32:10.678 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.manual_wal_flush: 0 2024-09-11T03:32:10.678 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.wal_compression: 0 2024-09-11T03:32:10.678 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.atomic_flush: 0 2024-09-11T03:32:10.678 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-11T03:32:10.678 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-11T03:32:10.678 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-11T03:32:10.678 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.log_readahead_size: 0 2024-09-11T03:32:10.678 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-11T03:32:10.678 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.best_efforts_recovery: 0 2024-09-11T03:32:10.678 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-11T03:32:10.678 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-11T03:32:10.678 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.allow_data_in_errors: 0 2024-09-11T03:32:10.678 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.db_host_id: __hostname__ 2024-09-11T03:32:10.679 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-11T03:32:10.679 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_background_jobs: 2 2024-09-11T03:32:10.679 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_background_compactions: -1 2024-09-11T03:32:10.679 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_subcompactions: 1 2024-09-11T03:32:10.680 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-11T03:32:10.680 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-11T03:32:10.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-11T03:32:10.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_total_wal_size: 0 2024-09-11T03:32:10.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-11T03:32:10.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-11T03:32:10.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-11T03:32:10.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-11T03:32:10.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_open_files: -1 2024-09-11T03:32:10.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.bytes_per_sync: 0 2024-09-11T03:32:10.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-11T03:32:10.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-11T03:32:10.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compaction_readahead_size: 0 2024-09-11T03:32:10.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_background_flushes: -1 2024-09-11T03:32:10.681 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Compression algorithms supported: 2024-09-11T03:32:10.682 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: kZSTD supported: 0 2024-09-11T03:32:10.682 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: kXpressCompression supported: 0 2024-09-11T03:32:10.682 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: kBZip2Compression supported: 0 2024-09-11T03:32:10.682 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-11T03:32:10.682 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: kLZ4Compression supported: 1 2024-09-11T03:32:10.682 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: kZlibCompression supported: 1 2024-09-11T03:32:10.682 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: kLZ4HCCompression supported: 1 2024-09-11T03:32:10.682 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: kSnappyCompression supported: 1 2024-09-11T03:32:10.682 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-11T03:32:10.682 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-11T03:32:10.682 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-11T03:32:10.682 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-11T03:32:10.683 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-11T03:32:10.683 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.merge_operator: 2024-09-11T03:32:10.683 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compaction_filter: None 2024-09-11T03:32:10.683 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compaction_filter_factory: None 2024-09-11T03:32:10.683 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.sst_partitioner_factory: None 2024-09-11T03:32:10.683 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-11T03:32:10.683 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-11T03:32:10.683 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562262b8d320) 2024-09-11T03:32:10.683 INFO:journalctl@ceph.mon.b.smithi121.stdout: cache_index_and_filter_blocks: 1 2024-09-11T03:32:10.683 INFO:journalctl@ceph.mon.b.smithi121.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-11T03:32:10.683 INFO:journalctl@ceph.mon.b.smithi121.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-11T03:32:10.683 INFO:journalctl@ceph.mon.b.smithi121.stdout: pin_top_level_index_and_filter: 1 2024-09-11T03:32:10.683 INFO:journalctl@ceph.mon.b.smithi121.stdout: index_type: 0 2024-09-11T03:32:10.684 INFO:journalctl@ceph.mon.b.smithi121.stdout: data_block_index_type: 0 2024-09-11T03:32:10.684 INFO:journalctl@ceph.mon.b.smithi121.stdout: index_shortening: 1 2024-09-11T03:32:10.684 INFO:journalctl@ceph.mon.b.smithi121.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-11T03:32:10.684 INFO:journalctl@ceph.mon.b.smithi121.stdout: checksum: 4 2024-09-11T03:32:10.684 INFO:journalctl@ceph.mon.b.smithi121.stdout: no_block_cache: 0 2024-09-11T03:32:10.684 INFO:journalctl@ceph.mon.b.smithi121.stdout: block_cache: 0x562262b81610 2024-09-11T03:32:10.684 INFO:journalctl@ceph.mon.b.smithi121.stdout: block_cache_name: BinnedLRUCache 2024-09-11T03:32:10.684 INFO:journalctl@ceph.mon.b.smithi121.stdout: block_cache_options: 2024-09-11T03:32:10.684 INFO:journalctl@ceph.mon.b.smithi121.stdout: capacity : 536870912 2024-09-11T03:32:10.684 INFO:journalctl@ceph.mon.b.smithi121.stdout: num_shard_bits : 4 2024-09-11T03:32:10.684 INFO:journalctl@ceph.mon.b.smithi121.stdout: strict_capacity_limit : 0 2024-09-11T03:32:10.684 INFO:journalctl@ceph.mon.b.smithi121.stdout: high_pri_pool_ratio: 0.000 2024-09-11T03:32:10.684 INFO:journalctl@ceph.mon.b.smithi121.stdout: block_cache_compressed: (nil) 2024-09-11T03:32:10.685 INFO:journalctl@ceph.mon.b.smithi121.stdout: persistent_cache: (nil) 2024-09-11T03:32:10.685 INFO:journalctl@ceph.mon.b.smithi121.stdout: block_size: 4096 2024-09-11T03:32:10.685 INFO:journalctl@ceph.mon.b.smithi121.stdout: block_size_deviation: 10 2024-09-11T03:32:10.685 INFO:journalctl@ceph.mon.b.smithi121.stdout: block_restart_interval: 16 2024-09-11T03:32:10.685 INFO:journalctl@ceph.mon.b.smithi121.stdout: index_block_restart_interval: 1 2024-09-11T03:32:10.685 INFO:journalctl@ceph.mon.b.smithi121.stdout: metadata_block_size: 4096 2024-09-11T03:32:10.685 INFO:journalctl@ceph.mon.b.smithi121.stdout: partition_filters: 0 2024-09-11T03:32:10.685 INFO:journalctl@ceph.mon.b.smithi121.stdout: use_delta_encoding: 1 2024-09-11T03:32:10.685 INFO:journalctl@ceph.mon.b.smithi121.stdout: filter_policy: bloomfilter 2024-09-11T03:32:10.685 INFO:journalctl@ceph.mon.b.smithi121.stdout: whole_key_filtering: 1 2024-09-11T03:32:10.685 INFO:journalctl@ceph.mon.b.smithi121.stdout: verify_compression: 0 2024-09-11T03:32:10.685 INFO:journalctl@ceph.mon.b.smithi121.stdout: read_amp_bytes_per_bit: 0 2024-09-11T03:32:10.685 INFO:journalctl@ceph.mon.b.smithi121.stdout: format_version: 5 2024-09-11T03:32:10.685 INFO:journalctl@ceph.mon.b.smithi121.stdout: enable_index_compression: 1 2024-09-11T03:32:10.686 INFO:journalctl@ceph.mon.b.smithi121.stdout: block_align: 0 2024-09-11T03:32:10.686 INFO:journalctl@ceph.mon.b.smithi121.stdout: max_auto_readahead_size: 262144 2024-09-11T03:32:10.686 INFO:journalctl@ceph.mon.b.smithi121.stdout: prepopulate_block_cache: 0 2024-09-11T03:32:10.686 INFO:journalctl@ceph.mon.b.smithi121.stdout: initial_auto_readahead_size: 8192 2024-09-11T03:32:10.686 INFO:journalctl@ceph.mon.b.smithi121.stdout: num_file_reads_for_auto_readahead: 2 2024-09-11T03:32:10.686 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.write_buffer_size: 33554432 2024-09-11T03:32:10.686 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_write_buffer_number: 2 2024-09-11T03:32:10.686 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compression: NoCompression 2024-09-11T03:32:10.686 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.bottommost_compression: Disabled 2024-09-11T03:32:10.686 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.prefix_extractor: nullptr 2024-09-11T03:32:10.686 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-11T03:32:10.686 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.num_levels: 7 2024-09-11T03:32:10.686 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-11T03:32:10.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-11T03:32:10.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-11T03:32:10.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-11T03:32:10.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-11T03:32:10.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-11T03:32:10.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-11T03:32:10.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-11T03:32:10.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-11T03:32:10.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-11T03:32:10.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-11T03:32:10.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-11T03:32:10.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-11T03:32:10.687 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compression_opts.level: 32767 2024-09-11T03:32:10.688 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compression_opts.strategy: 0 2024-09-11T03:32:10.688 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-11T03:32:10.688 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-11T03:32:10.688 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-11T03:32:10.688 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-11T03:32:10.688 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compression_opts.enabled: false 2024-09-11T03:32:10.688 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-11T03:32:10.688 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-11T03:32:10.688 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-11T03:32:10.688 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-11T03:32:10.688 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.target_file_size_base: 67108864 2024-09-11T03:32:10.688 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-11T03:32:10.688 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-11T03:32:10.688 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-11T03:32:10.689 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-11T03:32:10.689 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-11T03:32:10.689 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-11T03:32:10.689 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-11T03:32:10.689 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-11T03:32:10.689 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-11T03:32:10.689 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-11T03:32:10.689 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-11T03:32:10.689 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-11T03:32:10.689 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-11T03:32:10.689 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-11T03:32:10.689 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.arena_block_size: 1048576 2024-09-11T03:32:10.689 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-11T03:32:10.690 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-11T03:32:10.690 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.disable_auto_compactions: 0 2024-09-11T03:32:10.690 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-11T03:32:10.690 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-11T03:32:10.690 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-11T03:32:10.690 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-11T03:32:10.690 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-11T03:32:10.690 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-11T03:32:10.690 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-11T03:32:10.690 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-11T03:32:10.690 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-11T03:32:10.690 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-11T03:32:10.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-11T03:32:10.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.inplace_update_support: 0 2024-09-11T03:32:10.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-11T03:32:10.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-11T03:32:10.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-11T03:32:10.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-11T03:32:10.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.bloom_locality: 0 2024-09-11T03:32:10.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.max_successive_merges: 0 2024-09-11T03:32:10.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-11T03:32:10.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.paranoid_file_checks: 0 2024-09-11T03:32:10.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.force_consistency_checks: 1 2024-09-11T03:32:10.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.report_bg_io_stats: 0 2024-09-11T03:32:10.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.ttl: 2592000 2024-09-11T03:32:10.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-11T03:32:10.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-11T03:32:10.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-11T03:32:10.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.enable_blob_files: false 2024-09-11T03:32:10.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.min_blob_size: 0 2024-09-11T03:32:10.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.blob_file_size: 268435456 2024-09-11T03:32:10.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-11T03:32:10.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-11T03:32:10.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-11T03:32:10.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-11T03:32:10.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-11T03:32:10.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.blob_file_starting_level: 0 2024-09-11T03:32:10.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-11T03:32:10.693 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-11T03:32:10.693 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-11T03:32:10.693 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 21cbe84e-9a90-4779-a503-370b19359fbd 2024-09-11T03:32:10.693 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726025530301564, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-11T03:32:10.693 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-11T03:32:10.693 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726025530301960, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726025530, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "21cbe84e-9a90-4779-a503-370b19359fbd", "db_session_id": "TD1AVINWMOV6Q75Z9EIV", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-11T03:32:10.693 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726025530302051, "job": 1, "event": "recovery_finished"} 2024-09-11T03:32:10.693 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-11T03:32:10.693 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-11T03:32:10.693 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562262bb0e00 2024-09-11T03:32:10.693 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: DB pointer 0x562262cb2000 2024-09-11T03:32:10.693 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-11T03:32:10.693 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-11T03:32:10.693 INFO:journalctl@ceph.mon.b.smithi121.stdout: ** DB Stats ** 2024-09-11T03:32:10.694 INFO:journalctl@ceph.mon.b.smithi121.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-11T03:32:10.694 INFO:journalctl@ceph.mon.b.smithi121.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-11T03:32:10.694 INFO:journalctl@ceph.mon.b.smithi121.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-11T03:32:10.694 INFO:journalctl@ceph.mon.b.smithi121.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-11T03:32:10.694 INFO:journalctl@ceph.mon.b.smithi121.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-11T03:32:10.694 INFO:journalctl@ceph.mon.b.smithi121.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-11T03:32:10.694 INFO:journalctl@ceph.mon.b.smithi121.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-11T03:32:10.694 INFO:journalctl@ceph.mon.b.smithi121.stdout: 2024-09-11T03:32:10.694 INFO:journalctl@ceph.mon.b.smithi121.stdout: ** Compaction Stats [default] ** 2024-09-11T03:32:10.694 INFO:journalctl@ceph.mon.b.smithi121.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-11T03:32:10.694 INFO:journalctl@ceph.mon.b.smithi121.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-11T03:32:10.694 INFO:journalctl@ceph.mon.b.smithi121.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-11T03:32:10.694 INFO:journalctl@ceph.mon.b.smithi121.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-11T03:32:10.694 INFO:journalctl@ceph.mon.b.smithi121.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-11T03:32:10.695 INFO:journalctl@ceph.mon.b.smithi121.stdout: 2024-09-11T03:32:10.695 INFO:journalctl@ceph.mon.b.smithi121.stdout: ** Compaction Stats [default] ** 2024-09-11T03:32:10.695 INFO:journalctl@ceph.mon.b.smithi121.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-11T03:32:10.695 INFO:journalctl@ceph.mon.b.smithi121.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T03:32:10.695 INFO:journalctl@ceph.mon.b.smithi121.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-11T03:32:10.695 INFO:journalctl@ceph.mon.b.smithi121.stdout: 2024-09-11T03:32:10.695 INFO:journalctl@ceph.mon.b.smithi121.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-11T03:32:10.695 INFO:journalctl@ceph.mon.b.smithi121.stdout: 2024-09-11T03:32:10.695 INFO:journalctl@ceph.mon.b.smithi121.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-11T03:32:10.695 INFO:journalctl@ceph.mon.b.smithi121.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-11T03:32:10.695 INFO:journalctl@ceph.mon.b.smithi121.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-11T03:32:10.695 INFO:journalctl@ceph.mon.b.smithi121.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-11T03:32:10.695 INFO:journalctl@ceph.mon.b.smithi121.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-11T03:32:10.695 INFO:journalctl@ceph.mon.b.smithi121.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-11T03:32:10.696 INFO:journalctl@ceph.mon.b.smithi121.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T03:32:10.696 INFO:journalctl@ceph.mon.b.smithi121.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T03:32:10.696 INFO:journalctl@ceph.mon.b.smithi121.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-11T03:32:10.696 INFO:journalctl@ceph.mon.b.smithi121.stdout: Block cache BinnedLRUCache@0x562262b81610#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-11T03:32:10.696 INFO:journalctl@ceph.mon.b.smithi121.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-11T03:32:10.696 INFO:journalctl@ceph.mon.b.smithi121.stdout: 2024-09-11T03:32:10.696 INFO:journalctl@ceph.mon.b.smithi121.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-11T03:32:10.696 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-11T03:32:10.696 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: using public_addr v2:172.21.15.121:0/0 -> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] 2024-09-11T03:32:10.696 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: starting mon.b rank -1 at public addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] at bind addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:32:10.696 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.b@-1(???) e0 preinit fsid 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:32:10.696 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.b@-1(synchronizing).mds e1 new map 2024-09-11T03:32:10.696 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-11T03:32:10.697 INFO:journalctl@ceph.mon.b.smithi121.stdout: e1 2024-09-11T03:32:10.697 INFO:journalctl@ceph.mon.b.smithi121.stdout: btime 2024-09-11T03:30:30:532454+0000 2024-09-11T03:32:10.697 INFO:journalctl@ceph.mon.b.smithi121.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-11T03:32:10.697 INFO:journalctl@ceph.mon.b.smithi121.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-11T03:32:10.697 INFO:journalctl@ceph.mon.b.smithi121.stdout: legacy client fscid: -1 2024-09-11T03:32:10.697 INFO:journalctl@ceph.mon.b.smithi121.stdout: 2024-09-11T03:32:10.697 INFO:journalctl@ceph.mon.b.smithi121.stdout: No filesystems configured 2024-09-11T03:32:10.697 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-11T03:32:10.697 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-11T03:32:10.697 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-11T03:32:10.697 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-11T03:32:10.697 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-11T03:32:10.697 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-11T03:32:10.698 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-11T03:32:10.698 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T03:32:10.698 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T03:32:10.698 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T03:32:10.698 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:10.698 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: Added host smithi175 2024-09-11T03:32:10.698 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:10.698 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:10.698 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:10.698 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:10.698 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:10.698 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:32:10.698 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:10.698 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3152965123' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-11T03:32:10.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3152965123' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-11T03:32:10.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T03:32:10.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:10.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:10.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:10.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:10.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi175", "name": "osd_memory_target"} : dispatch 2024-09-11T03:32:10.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi175", "name": "osd_memory_target"} : dispatch 2024-09-11T03:32:10.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:32:10.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:10.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:10.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: Updating smithi175:/etc/ceph/ceph.conf 2024-09-11T03:32:10.699 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: Updating smithi175:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T03:32:10.700 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: Updating smithi175:/etc/ceph/ceph.client.admin.keyring 2024-09-11T03:32:10.700 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:10.700 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:10.700 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:10.700 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:10.700 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:10.700 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-11T03:32:10.700 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:10.700 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: Updating smithi175:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T03:32:10.700 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi084:172.21.15.84=a;smithi121:172.21.15.121=b;smithi175:172.21.15.175=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:32:10.700 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: Saving service mon spec with placement smithi084:172.21.15.84=a;smithi121:172.21.15.121=b;smithi175:172.21.15.175=c;count:3 2024-09-11T03:32:10.700 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: Deploying daemon mon.c on smithi175 2024-09-11T03:32:10.700 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:10.700 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.175:0/744670426' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-11T03:32:10.700 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:10.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:32:10.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:10.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.a calling monitor election 2024-09-11T03:32:10.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:10.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:10.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:10.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.c calling monitor election 2024-09-11T03:32:10.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:10.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:10.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:10.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:10.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:10.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-11T03:32:10.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: monmap epoch 2 2024-09-11T03:32:10.701 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: fsid 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:32:10.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: last_changed 2024-09-11T03:32:01.686490+0000 2024-09-11T03:32:10.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: created 2024-09-11T03:30:27.618796+0000 2024-09-11T03:32:10.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: min_mon_release 19 (squid) 2024-09-11T03:32:10.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: election_strategy: 1 2024-09-11T03:32:10.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-09-11T03:32:10.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: 1: [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] mon.c 2024-09-11T03:32:10.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: fsmap 2024-09-11T03:32:10.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T03:32:10.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mgrmap e13: y(active, since 40s) 2024-09-11T03:32:10.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: overall HEALTH_OK 2024-09-11T03:32:10.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:10.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:10.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:10.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-11T03:32:10.702 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:10.703 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: Deploying daemon mon.b on smithi121 2024-09-11T03:32:10.703 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.175:0/2367191011' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-11T03:32:10.703 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:10.703 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:10.703 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.175:0/1892807782' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-11T03:32:10.703 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 ceph-mon[30729]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-11T03:32:10.703 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 bash[30709]: d891b9a0f3f53fe017a8185fd83c1cf2748533f0a7ac79779d10afcd8aceecde 2024-09-11T03:32:10.703 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:10 smithi121 systemd[1]: Started Ceph mon.b for 10fe7946-6fee-11ef-bcea-c7b262605968. 2024-09-11T03:32:11.156 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-11T03:32:11.156 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-11T03:32:11.427 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:32:15.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:32:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: mon.a calling monitor election 2024-09-11T03:32:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: mon.c calling monitor election 2024-09-11T03:32:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: mon.b calling monitor election 2024-09-11T03:32:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-11T03:32:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: monmap epoch 3 2024-09-11T03:32:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: fsid 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:32:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: last_changed 2024-09-11T03:32:10.327171+0000 2024-09-11T03:32:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: created 2024-09-11T03:30:27.618796+0000 2024-09-11T03:32:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: min_mon_release 19 (squid) 2024-09-11T03:32:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: election_strategy: 1 2024-09-11T03:32:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-09-11T03:32:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: 1: [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] mon.c 2024-09-11T03:32:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: 2: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.b 2024-09-11T03:32:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: fsmap 2024-09-11T03:32:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T03:32:15.647 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: mgrmap e13: y(active, since 48s) 2024-09-11T03:32:15.647 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: overall HEALTH_OK 2024-09-11T03:32:15.647 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:15.647 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:15.647 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:15.647 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:32:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: mon.a calling monitor election 2024-09-11T03:32:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: mon.c calling monitor election 2024-09-11T03:32:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: mon.b calling monitor election 2024-09-11T03:32:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-11T03:32:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: monmap epoch 3 2024-09-11T03:32:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: fsid 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:32:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: last_changed 2024-09-11T03:32:10.327171+0000 2024-09-11T03:32:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: created 2024-09-11T03:30:27.618796+0000 2024-09-11T03:32:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: min_mon_release 19 (squid) 2024-09-11T03:32:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: election_strategy: 1 2024-09-11T03:32:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-09-11T03:32:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: 1: [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] mon.c 2024-09-11T03:32:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: 2: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.b 2024-09-11T03:32:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: fsmap 2024-09-11T03:32:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T03:32:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: mgrmap e13: y(active, since 48s) 2024-09-11T03:32:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: overall HEALTH_OK 2024-09-11T03:32:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:15.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:32:15.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: mon.a calling monitor election 2024-09-11T03:32:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: mon.c calling monitor election 2024-09-11T03:32:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:32:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: mon.b calling monitor election 2024-09-11T03:32:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-11T03:32:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: monmap epoch 3 2024-09-11T03:32:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: fsid 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:32:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: last_changed 2024-09-11T03:32:10.327171+0000 2024-09-11T03:32:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: created 2024-09-11T03:30:27.618796+0000 2024-09-11T03:32:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: min_mon_release 19 (squid) 2024-09-11T03:32:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: election_strategy: 1 2024-09-11T03:32:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-09-11T03:32:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: 1: [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] mon.c 2024-09-11T03:32:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: 2: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.b 2024-09-11T03:32:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: fsmap 2024-09-11T03:32:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T03:32:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: mgrmap e13: y(active, since 48s) 2024-09-11T03:32:15.730 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: overall HEALTH_OK 2024-09-11T03:32:15.730 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:15.730 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:15.730 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:15.730 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:15.730 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:32:15.730 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":3,"fsid":"10fe7946-6fee-11ef-bcea-c7b262605968","modified":"2024-09-11T03:32:10.327171Z","created":"2024-09-11T03:30:27.618796Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:3300","nonce":0},{"type":"v1","addr":"172.21.15.175:6789","nonce":0}]},"addr":"172.21.15.175:6789/0","public_addr":"172.21.15.175:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-11T03:32:15.731 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 3 2024-09-11T03:32:16.303 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-11T03:32:16.303 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph config generate-minimal-conf 2024-09-11T03:32:16.561 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:32:16.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:16.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:16 smithi084 ceph-mon[27254]: from='client.? 172.21.15.175:0/2027102275' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-11T03:32:16.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:16.667 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:16.667 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:16 smithi175 ceph-mon[30894]: from='client.? 172.21.15.175:0/2027102275' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-11T03:32:16.667 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:16.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:16.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:16 smithi121 ceph-mon[30729]: from='client.? 172.21.15.175:0/2027102275' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-11T03:32:16.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:32:17.202 INFO:teuthology.orchestra.run.smithi084.stdout:# minimal ceph.conf for 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:32:17.203 INFO:teuthology.orchestra.run.smithi084.stdout:[global] 2024-09-11T03:32:17.203 INFO:teuthology.orchestra.run.smithi084.stdout: fsid = 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:32:17.203 INFO:teuthology.orchestra.run.smithi084.stdout: mon_host = [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] 2024-09-11T03:32:17.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:17 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:17.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:17 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:17.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:17 smithi084 ceph-mon[27254]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:17.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:17 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:17.585 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:17 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:17.585 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:17 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:17.585 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:17 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:17.585 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:17 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:17.585 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3872475374' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:17.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:17 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:17.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:17 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:17.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:17 smithi175 ceph-mon[30894]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:17.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:17 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:17.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:17 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:17.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:17 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:17.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:17 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:17.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:17 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:17.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3872475374' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:17.892 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-11T03:32:17.892 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-11T03:32:17.892 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.conf 2024-09-11T03:32:17.914 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:17 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:17.914 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:17 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:17.914 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:17 smithi121 ceph-mon[30729]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:17.914 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:17 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:17.914 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:17 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:17.914 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:17 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:17.914 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:17 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:17.914 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:17 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:17.914 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3872475374' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:17.918 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-11T03:32:17.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-11T03:32:17.984 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-11T03:32:17.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.conf 2024-09-11T03:32:18.013 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-11T03:32:18.013 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-11T03:32:18.077 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-11T03:32:18.077 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/ceph/ceph.conf 2024-09-11T03:32:18.103 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-11T03:32:18.104 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-11T03:32:18.169 INFO:tasks.cephadm:Adding mgr.y on smithi084 2024-09-11T03:32:18.169 INFO:tasks.cephadm:Adding mgr.x on smithi121 2024-09-11T03:32:18.169 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch apply mgr '2;smithi084=y;smithi121=x' 2024-09-11T03:32:18.453 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:32:18.839 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: Updating smithi084:/etc/ceph/ceph.conf 2024-09-11T03:32:18.840 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: Updating smithi121:/etc/ceph/ceph.conf 2024-09-11T03:32:18.840 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: Updating smithi175:/etc/ceph/ceph.conf 2024-09-11T03:32:18.840 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: Updating smithi084:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T03:32:18.840 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: Updating smithi175:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T03:32:18.840 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: Updating smithi121:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T03:32:18.840 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.841 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.841 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.841 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.841 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.841 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.841 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.841 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.841 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.841 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.841 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.841 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-11T03:32:18.841 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-11T03:32:18.841 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:18.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: Updating smithi084:/etc/ceph/ceph.conf 2024-09-11T03:32:18.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: Updating smithi121:/etc/ceph/ceph.conf 2024-09-11T03:32:18.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: Updating smithi175:/etc/ceph/ceph.conf 2024-09-11T03:32:18.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: Updating smithi084:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T03:32:18.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: Updating smithi175:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T03:32:18.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: Updating smithi121:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T03:32:18.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-11T03:32:18.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-11T03:32:18.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: Updating smithi084:/etc/ceph/ceph.conf 2024-09-11T03:32:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: Updating smithi121:/etc/ceph/ceph.conf 2024-09-11T03:32:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: Updating smithi175:/etc/ceph/ceph.conf 2024-09-11T03:32:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: Updating smithi084:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T03:32:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: Updating smithi175:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T03:32:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: Updating smithi121:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T03:32:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:18.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-11T03:32:18.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-11T03:32:18.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:19.123 INFO:teuthology.orchestra.run.smithi175.stdout:Scheduled mgr update... 2024-09-11T03:32:19.707 DEBUG:teuthology.orchestra.run.smithi121:mgr.x> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mgr.x.service 2024-09-11T03:32:19.709 DEBUG:tasks.cephadm:set 0 configs 2024-09-11T03:32:19.710 INFO:tasks.cephadm:Deploying OSDs... 2024-09-11T03:32:19.710 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-11T03:32:19.710 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T03:32:19.728 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T03:32:19.728 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_1 2024-09-11T03:32:19.785 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T03:32:19.785 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T03:32:19.785 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-11T03:32:19.785 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T03:32:19.785 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T03:32:19.785 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-09-11 03:31:40.116898923 +0000 2024-09-11T03:32:19.785 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-09-11 03:27:29.264202481 +0000 2024-09-11T03:32:19.785 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-09-11 03:27:29.264202481 +0000 2024-09-11T03:32:19.786 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-09-11 03:27:29.264202481 +0000 2024-09-11T03:32:19.787 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T03:32:19.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:19 smithi084 ceph-mon[27254]: Reconfiguring mon.a (unknown last config time)... 2024-09-11T03:32:19.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:19 smithi084 ceph-mon[27254]: Reconfiguring daemon mon.a on smithi084 2024-09-11T03:32:19.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:19 smithi084 ceph-mon[27254]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:19.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:19.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:19.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-11T03:32:19.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-11T03:32:19.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:19.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:19.853 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-09-11T03:32:19.853 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-09-11T03:32:19.853 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000196935 s, 2.6 MB/s 2024-09-11T03:32:19.853 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T03:32:19.910 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_2 2024-09-11T03:32:19.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:19 smithi121 ceph-mon[30729]: Reconfiguring mon.a (unknown last config time)... 2024-09-11T03:32:19.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:19 smithi121 ceph-mon[30729]: Reconfiguring daemon mon.a on smithi084 2024-09-11T03:32:19.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:19 smithi121 ceph-mon[30729]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:19.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:19.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:19.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-11T03:32:19.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-11T03:32:19.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:19.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:19.967 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T03:32:19.967 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T03:32:19.967 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-11T03:32:19.967 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T03:32:19.967 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T03:32:19.967 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-09-11 03:31:40.146898380 +0000 2024-09-11T03:32:19.967 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-09-11 03:27:29.585196584 +0000 2024-09-11T03:32:19.967 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-09-11 03:27:29.585196584 +0000 2024-09-11T03:32:19.967 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-09-11 03:27:29.585196584 +0000 2024-09-11T03:32:19.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T03:32:19.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:19 smithi175 ceph-mon[30894]: Reconfiguring mon.a (unknown last config time)... 2024-09-11T03:32:19.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:19 smithi175 ceph-mon[30894]: Reconfiguring daemon mon.a on smithi084 2024-09-11T03:32:19.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:19 smithi175 ceph-mon[30894]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:19.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:19.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:19.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-11T03:32:19.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-11T03:32:19.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:19.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:20.032 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-09-11T03:32:20.032 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-09-11T03:32:20.032 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000193602 s, 2.6 MB/s 2024-09-11T03:32:20.033 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T03:32:20.090 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_3 2024-09-11T03:32:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T03:32:20.146 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T03:32:20.146 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-11T03:32:20.147 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T03:32:20.147 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T03:32:20.147 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-09-11 03:31:40.177897818 +0000 2024-09-11T03:32:20.147 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-09-11 03:27:29.933190191 +0000 2024-09-11T03:32:20.147 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-09-11 03:27:29.933190191 +0000 2024-09-11T03:32:20.147 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-09-11 03:27:29.933190191 +0000 2024-09-11T03:32:20.147 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T03:32:20.210 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-09-11T03:32:20.210 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-09-11T03:32:20.210 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000204245 s, 2.5 MB/s 2024-09-11T03:32:20.211 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T03:32:20.268 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_4 2024-09-11T03:32:20.324 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T03:32:20.324 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T03:32:20.324 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-11T03:32:20.325 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T03:32:20.325 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T03:32:20.325 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-09-11 03:31:40.206897292 +0000 2024-09-11T03:32:20.325 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-09-11 03:27:30.250184368 +0000 2024-09-11T03:32:20.325 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-09-11 03:27:30.250184368 +0000 2024-09-11T03:32:20.325 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-09-11 03:27:30.250184368 +0000 2024-09-11T03:32:20.325 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T03:32:20.387 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-09-11T03:32:20.387 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-09-11T03:32:20.387 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000198331 s, 2.6 MB/s 2024-09-11T03:32:20.389 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T03:32:20.445 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-11T03:32:20.445 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T03:32:20.462 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T03:32:20.462 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2024-09-11T03:32:20.518 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T03:32:20.518 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T03:32:20.518 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-11T03:32:20.518 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T03:32:20.518 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T03:32:20.519 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-09-11 03:32:09.760307409 +0000 2024-09-11T03:32:20.519 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-09-11 03:27:29.476431232 +0000 2024-09-11T03:32:20.519 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-09-11 03:27:29.476431232 +0000 2024-09-11T03:32:20.519 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-09-11 03:27:29.476431232 +0000 2024-09-11T03:32:20.519 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T03:32:20.582 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-09-11T03:32:20.582 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-09-11T03:32:20.582 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.00023294 s, 2.2 MB/s 2024-09-11T03:32:20.583 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T03:32:20.640 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2024-09-11T03:32:20.697 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T03:32:20.697 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T03:32:20.697 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-11T03:32:20.697 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T03:32:20.697 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T03:32:20.698 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-09-11 03:32:09.761307396 +0000 2024-09-11T03:32:20.698 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-09-11 03:27:29.800426964 +0000 2024-09-11T03:32:20.698 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-09-11 03:27:29.800426964 +0000 2024-09-11T03:32:20.698 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-09-11 03:27:29.800426964 +0000 2024-09-11T03:32:20.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T03:32:20.760 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-09-11T03:32:20.760 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-09-11T03:32:20.760 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000203561 s, 2.5 MB/s 2024-09-11T03:32:20.761 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T03:32:20.818 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2024-09-11T03:32:20.871 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:20 smithi175 ceph-mon[30894]: Reconfiguring mon.b (monmap changed)... 2024-09-11T03:32:20.871 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:20 smithi175 ceph-mon[30894]: Reconfiguring daemon mon.b on smithi121 2024-09-11T03:32:20.871 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:20 smithi175 ceph-mon[30894]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi084=y;smithi121=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:32:20.871 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:20 smithi175 ceph-mon[30894]: Saving service mgr spec with placement smithi084=y;smithi121=x;count:2 2024-09-11T03:32:20.871 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:20 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:20.871 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:20 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:20.871 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:20 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-11T03:32:20.871 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:20 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-11T03:32:20.871 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:20 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:20.876 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T03:32:20.877 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T03:32:20.877 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-11T03:32:20.877 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T03:32:20.877 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T03:32:20.877 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-09-11 03:32:09.761307396 +0000 2024-09-11T03:32:20.877 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-09-11 03:27:30.127422658 +0000 2024-09-11T03:32:20.877 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-09-11 03:27:30.127422658 +0000 2024-09-11T03:32:20.877 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-09-11 03:27:30.127422658 +0000 2024-09-11T03:32:20.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T03:32:20.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:20 smithi084 ceph-mon[27254]: Reconfiguring mon.b (monmap changed)... 2024-09-11T03:32:20.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:20 smithi084 ceph-mon[27254]: Reconfiguring daemon mon.b on smithi121 2024-09-11T03:32:20.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:20 smithi084 ceph-mon[27254]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi084=y;smithi121=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:32:20.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:20 smithi084 ceph-mon[27254]: Saving service mgr spec with placement smithi084=y;smithi121=x;count:2 2024-09-11T03:32:20.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:20 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:20.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:20 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:20.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:20 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-11T03:32:20.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:20 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-11T03:32:20.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:20 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:20.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:20 smithi121 ceph-mon[30729]: Reconfiguring mon.b (monmap changed)... 2024-09-11T03:32:20.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:20 smithi121 ceph-mon[30729]: Reconfiguring daemon mon.b on smithi121 2024-09-11T03:32:20.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:20 smithi121 ceph-mon[30729]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi084=y;smithi121=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:32:20.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:20 smithi121 ceph-mon[30729]: Saving service mgr spec with placement smithi084=y;smithi121=x;count:2 2024-09-11T03:32:20.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:20 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:20.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:20 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:20.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:20 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-11T03:32:20.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:20 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-11T03:32:20.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:20 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:20.940 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-09-11T03:32:20.940 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-09-11T03:32:20.940 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000201052 s, 2.5 MB/s 2024-09-11T03:32:20.941 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T03:32:20.997 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2024-09-11T03:32:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T03:32:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T03:32:21.053 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-11T03:32:21.053 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T03:32:21.054 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T03:32:21.054 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-09-11 03:32:09.761307396 +0000 2024-09-11T03:32:21.054 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-09-11 03:27:30.453418364 +0000 2024-09-11T03:32:21.054 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-09-11 03:27:30.453418364 +0000 2024-09-11T03:32:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-09-11 03:27:30.453418364 +0000 2024-09-11T03:32:21.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T03:32:21.117 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-09-11T03:32:21.117 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-09-11T03:32:21.117 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000200906 s, 2.5 MB/s 2024-09-11T03:32:21.118 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T03:32:21.174 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-11T03:32:21.174 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T03:32:21.192 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T03:32:21.193 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_1 2024-09-11T03:32:21.248 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T03:32:21.248 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T03:32:21.248 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-11T03:32:21.248 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T03:32:21.248 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T03:32:21.248 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2024-09-11 03:32:01.154860226 +0000 2024-09-11T03:32:21.248 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2024-09-11 03:27:37.520450677 +0000 2024-09-11T03:32:21.248 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2024-09-11 03:27:37.520450677 +0000 2024-09-11T03:32:21.249 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: 2024-09-11 03:27:37.520450677 +0000 2024-09-11T03:32:21.249 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T03:32:21.313 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2024-09-11T03:32:21.313 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2024-09-11T03:32:21.313 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000269207 s, 1.9 MB/s 2024-09-11T03:32:21.314 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T03:32:21.371 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_2 2024-09-11T03:32:21.428 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T03:32:21.428 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T03:32:21.429 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-11T03:32:21.429 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T03:32:21.429 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T03:32:21.429 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2024-09-11 03:32:01.154860226 +0000 2024-09-11T03:32:21.429 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2024-09-11 03:27:37.841445814 +0000 2024-09-11T03:32:21.429 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2024-09-11 03:27:37.841445814 +0000 2024-09-11T03:32:21.429 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: 2024-09-11 03:27:37.841445814 +0000 2024-09-11T03:32:21.429 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T03:32:21.492 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2024-09-11T03:32:21.492 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2024-09-11T03:32:21.492 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000279289 s, 1.8 MB/s 2024-09-11T03:32:21.494 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T03:32:21.550 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_3 2024-09-11T03:32:21.605 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T03:32:21.605 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T03:32:21.605 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-11T03:32:21.606 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T03:32:21.606 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T03:32:21.606 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2024-09-11 03:32:01.155860209 +0000 2024-09-11T03:32:21.606 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2024-09-11 03:27:38.164440921 +0000 2024-09-11T03:32:21.606 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2024-09-11 03:27:38.164440921 +0000 2024-09-11T03:32:21.606 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: 2024-09-11 03:27:38.164440921 +0000 2024-09-11T03:32:21.606 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T03:32:21.670 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2024-09-11T03:32:21.670 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2024-09-11T03:32:21.670 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000184192 s, 2.8 MB/s 2024-09-11T03:32:21.671 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T03:32:21.729 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_4 2024-09-11T03:32:21.784 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T03:32:21.785 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T03:32:21.785 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-11T03:32:21.785 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T03:32:21.785 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T03:32:21.785 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2024-09-11 03:32:01.155860209 +0000 2024-09-11T03:32:21.785 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2024-09-11 03:27:38.485436058 +0000 2024-09-11T03:32:21.785 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2024-09-11 03:27:38.485436058 +0000 2024-09-11T03:32:21.785 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: 2024-09-11 03:27:38.485436058 +0000 2024-09-11T03:32:21.785 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T03:32:21.848 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2024-09-11T03:32:21.848 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2024-09-11T03:32:21.849 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000205637 s, 2.5 MB/s 2024-09-11T03:32:21.850 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T03:32:21.879 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:21 smithi121 ceph-mon[30729]: Reconfiguring mon.c (monmap changed)... 2024-09-11T03:32:21.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:21 smithi121 ceph-mon[30729]: Reconfiguring daemon mon.c on smithi175 2024-09-11T03:32:21.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:21 smithi121 ceph-mon[30729]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:21.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:21.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:21.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:21.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:21.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:21.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:21.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-11T03:32:21.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-11T03:32:21.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-11T03:32:21.880 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:21.897 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:21 smithi084 ceph-mon[27254]: Reconfiguring mon.c (monmap changed)... 2024-09-11T03:32:21.898 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:21 smithi084 ceph-mon[27254]: Reconfiguring daemon mon.c on smithi175 2024-09-11T03:32:21.898 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:21 smithi084 ceph-mon[27254]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:21.898 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:21.898 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:21.898 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:21.898 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:21.898 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:21.898 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:21.898 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-11T03:32:21.899 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-11T03:32:21.899 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-11T03:32:21.899 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:21.906 INFO:tasks.cephadm:Deploying osd.0 on smithi084 with /dev/vg_nvme/lv_4... 2024-09-11T03:32:21.906 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-11T03:32:21.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:21 smithi175 ceph-mon[30894]: Reconfiguring mon.c (monmap changed)... 2024-09-11T03:32:21.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:21 smithi175 ceph-mon[30894]: Reconfiguring daemon mon.c on smithi175 2024-09-11T03:32:21.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:21 smithi175 ceph-mon[30894]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:21.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:21.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:21.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:21.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:21.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:21.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:21.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-11T03:32:21.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-11T03:32:21.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-11T03:32:21.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:22.159 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:32:22.891 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:22 smithi121 ceph-mon[30729]: Deploying daemon mgr.x on smithi121 2024-09-11T03:32:22.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:22 smithi084 ceph-mon[27254]: Deploying daemon mgr.x on smithi121 2024-09-11T03:32:22.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:22 smithi175 ceph-mon[30894]: Deploying daemon mgr.x on smithi121 2024-09-11T03:32:23.172 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Sep 11 03:32:23 smithi121 systemd[1]: Starting Ceph mgr.x for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:32:23.614 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Sep 11 03:32:23 smithi121 podman[32072]: 2024-09-11 03:32:23.313063243 +0000 UTC m=+0.111966101 container create 794a1afd76431951e5e2003dea929b03e4a98f5ba7f84bf2b58a3a86df60eee8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-11T03:32:23.615 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Sep 11 03:32:23 smithi121 podman[32072]: 2024-09-11 03:32:23.216752382 +0000 UTC m=+0.015655241 image pull 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e 2024-09-11T03:32:23.615 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Sep 11 03:32:23 smithi121 podman[32072]: 2024-09-11 03:32:23.473516535 +0000 UTC m=+0.272419395 container init 794a1afd76431951e5e2003dea929b03e4a98f5ba7f84bf2b58a3a86df60eee8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-x, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-11T03:32:23.615 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Sep 11 03:32:23 smithi121 podman[32072]: 2024-09-11 03:32:23.476484626 +0000 UTC m=+0.275387486 container start 794a1afd76431951e5e2003dea929b03e4a98f5ba7f84bf2b58a3a86df60eee8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-x, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4) 2024-09-11T03:32:23.615 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Sep 11 03:32:23 smithi121 ceph-mgr[32091]: -- 172.21.15.121:0/725100778 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556088ca10e0 con 0x556088c7f400 2024-09-11T03:32:23.615 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Sep 11 03:32:23 smithi121 bash[32072]: 794a1afd76431951e5e2003dea929b03e4a98f5ba7f84bf2b58a3a86df60eee8 2024-09-11T03:32:23.615 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Sep 11 03:32:23 smithi121 systemd[1]: Started Ceph mgr.x for 10fe7946-6fee-11ef-bcea-c7b262605968. 2024-09-11T03:32:23.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:23 smithi084 ceph-mon[27254]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:23.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:23 smithi121 ceph-mon[30729]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:23.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:23 smithi175 ceph-mon[30894]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:24.531 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:32:24.557 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi084:vg_nvme/lv_4 2024-09-11T03:32:24.801 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:32:24.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:24.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:24.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:24.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:24.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:24.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:24 smithi084 ceph-mon[27254]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:24.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:24.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:24.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:24.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:24.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:24.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:24 smithi121 ceph-mon[30729]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:24.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:24.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:24.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:24.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:24.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:24.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:24 smithi175 ceph-mon[30894]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:25.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:25.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:25.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:25.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:25.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:25.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:25 smithi084 ceph-mon[27254]: Reconfiguring mgr.y (unknown last config time)... 2024-09-11T03:32:25.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-11T03:32:25.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-11T03:32:25.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:25.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:25 smithi084 ceph-mon[27254]: Reconfiguring daemon mgr.y on smithi084 2024-09-11T03:32:25.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:25 smithi084 ceph-mon[27254]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:32:25.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:32:25.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:32:25.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:26.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:26.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:26.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:26.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:26.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:26.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:25 smithi121 ceph-mon[30729]: Reconfiguring mgr.y (unknown last config time)... 2024-09-11T03:32:26.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-11T03:32:26.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-11T03:32:26.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:26.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:25 smithi121 ceph-mon[30729]: Reconfiguring daemon mgr.y on smithi084 2024-09-11T03:32:26.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:25 smithi121 ceph-mon[30729]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:32:26.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:32:26.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:32:26.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:25 smithi175 ceph-mon[30894]: Reconfiguring mgr.y (unknown last config time)... 2024-09-11T03:32:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-11T03:32:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-11T03:32:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:25 smithi175 ceph-mon[30894]: Reconfiguring daemon mgr.y on smithi084 2024-09-11T03:32:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:25 smithi175 ceph-mon[30894]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:32:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:32:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:32:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:27.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:27.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:27.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:27.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:27.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:27.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:27.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:27 smithi084 ceph-mon[27254]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:27.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:27.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:27 smithi121 ceph-mon[30729]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:27.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:27.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:27.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:27.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:27.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:27.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:27.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:27 smithi175 ceph-mon[30894]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:27.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:28.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:28 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3573927103' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9a431e48-e46b-44f9-bfa7-cb56c1484e1c"} : dispatch 2024-09-11T03:32:28.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:28 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3573927103' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a431e48-e46b-44f9-bfa7-cb56c1484e1c"}]': finished 2024-09-11T03:32:28.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:28 smithi084 ceph-mon[27254]: osdmap e5: 1 total, 0 up, 1 in 2024-09-11T03:32:28.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:28 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:28.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:28 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3573927103' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9a431e48-e46b-44f9-bfa7-cb56c1484e1c"} : dispatch 2024-09-11T03:32:28.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:28 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3573927103' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a431e48-e46b-44f9-bfa7-cb56c1484e1c"}]': finished 2024-09-11T03:32:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:28 smithi121 ceph-mon[30729]: osdmap e5: 1 total, 0 up, 1 in 2024-09-11T03:32:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:28 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:28.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:28 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3573927103' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9a431e48-e46b-44f9-bfa7-cb56c1484e1c"} : dispatch 2024-09-11T03:32:28.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:28 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3573927103' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a431e48-e46b-44f9-bfa7-cb56c1484e1c"}]': finished 2024-09-11T03:32:28.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:28 smithi175 ceph-mon[30894]: osdmap e5: 1 total, 0 up, 1 in 2024-09-11T03:32:28.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:28 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:29.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:29 smithi084 ceph-mon[27254]: Standby manager daemon x started 2024-09-11T03:32:29.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:29 smithi084 ceph-mon[27254]: from='mgr.? 172.21.15.121:0/1150122680' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-11T03:32:29.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:29 smithi084 ceph-mon[27254]: from='mgr.? 172.21.15.121:0/1150122680' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-11T03:32:29.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:29 smithi084 ceph-mon[27254]: from='mgr.? 172.21.15.121:0/1150122680' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-11T03:32:29.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:29 smithi084 ceph-mon[27254]: from='mgr.? 172.21.15.121:0/1150122680' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-11T03:32:29.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:29 smithi084 ceph-mon[27254]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:29.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:29 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/19835345' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:32:29.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:29 smithi121 ceph-mon[30729]: Standby manager daemon x started 2024-09-11T03:32:29.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:29 smithi121 ceph-mon[30729]: from='mgr.? 172.21.15.121:0/1150122680' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-11T03:32:29.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:29 smithi121 ceph-mon[30729]: from='mgr.? 172.21.15.121:0/1150122680' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-11T03:32:29.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:29 smithi121 ceph-mon[30729]: from='mgr.? 172.21.15.121:0/1150122680' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-11T03:32:29.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:29 smithi121 ceph-mon[30729]: from='mgr.? 172.21.15.121:0/1150122680' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-11T03:32:29.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:29 smithi121 ceph-mon[30729]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:29.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:29 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/19835345' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:32:29.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:29 smithi175 ceph-mon[30894]: Standby manager daemon x started 2024-09-11T03:32:29.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:29 smithi175 ceph-mon[30894]: from='mgr.? 172.21.15.121:0/1150122680' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-11T03:32:29.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:29 smithi175 ceph-mon[30894]: from='mgr.? 172.21.15.121:0/1150122680' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-11T03:32:29.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:29 smithi175 ceph-mon[30894]: from='mgr.? 172.21.15.121:0/1150122680' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-11T03:32:29.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:29 smithi175 ceph-mon[30894]: from='mgr.? 172.21.15.121:0/1150122680' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-11T03:32:29.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:29 smithi175 ceph-mon[30894]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:29.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:29 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/19835345' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:32:30.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:30 smithi121 ceph-mon[30729]: mgrmap e14: y(active, since 62s), standbys: x 2024-09-11T03:32:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:30 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-11T03:32:30.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:30 smithi175 ceph-mon[30894]: mgrmap e14: y(active, since 62s), standbys: x 2024-09-11T03:32:30.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:30 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-11T03:32:30.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:30 smithi084 ceph-mon[27254]: mgrmap e14: y(active, since 62s), standbys: x 2024-09-11T03:32:30.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:30 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-11T03:32:31.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:31 smithi121 ceph-mon[30729]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:31.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:31 smithi175 ceph-mon[30894]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:31.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:31 smithi084 ceph-mon[27254]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:32.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:32 smithi084 ceph-mon[27254]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:32.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:32 smithi121 ceph-mon[30729]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:32.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:32 smithi175 ceph-mon[30894]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:34.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:34 smithi084 ceph-mon[27254]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:34.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:34 smithi121 ceph-mon[30729]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:34.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:34 smithi175 ceph-mon[30894]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:36.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:35 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-11T03:32:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:35 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:36.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:35 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-11T03:32:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:35 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:36.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:35 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-11T03:32:36.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:35 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:37.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:36 smithi121 ceph-mon[30729]: Deploying daemon osd.0 on smithi084 2024-09-11T03:32:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:36 smithi121 ceph-mon[30729]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:37.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:36 smithi084 ceph-mon[27254]: Deploying daemon osd.0 on smithi084 2024-09-11T03:32:37.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:36 smithi084 ceph-mon[27254]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:36 smithi175 ceph-mon[30894]: Deploying daemon osd.0 on smithi084 2024-09-11T03:32:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:36 smithi175 ceph-mon[30894]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:38.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:38 smithi121 ceph-mon[30729]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:38.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:38 smithi175 ceph-mon[30894]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:39.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:38 smithi084 ceph-mon[27254]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:40.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:40 smithi121 ceph-mon[30729]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:40.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:40 smithi175 ceph-mon[30894]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:40.991 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:40 smithi084 ceph-mon[27254]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:42 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:42 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:42 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:42.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:42 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:42.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:42 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:42.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:42 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:42.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:42 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:42.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:42 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:42.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:42 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:43.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:43 smithi084 ceph-mon[27254]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:43.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:43 smithi121 ceph-mon[30729]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:43.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:43 smithi175 ceph-mon[30894]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:44.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:44 smithi175 ceph-mon[30894]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:45.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:44 smithi084 ceph-mon[27254]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:45.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:44 smithi121 ceph-mon[30729]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:46.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:46.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:46.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:46.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:46.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:46.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:46.543 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 0 on host 'smithi084' 2024-09-11T03:32:47.181 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:46 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:47.181 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:46 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:47.181 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:46 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:47.181 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:46 smithi084 ceph-mon[27254]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:47.214 DEBUG:teuthology.orchestra.run.smithi084:osd.0> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.0.service 2024-09-11T03:32:47.216 INFO:tasks.cephadm:Deploying osd.1 on smithi084 with /dev/vg_nvme/lv_3... 2024-09-11T03:32:47.216 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-11T03:32:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:46 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:46 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:46 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:46 smithi175 ceph-mon[30894]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:46 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:32:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:46 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:46 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:46 smithi121 ceph-mon[30729]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:48.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:47 smithi175 ceph-mon[30894]: from='osd.0 [v2:172.21.15.84:6802/1376667411,v1:172.21.15.84:6803/1376667411]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-11T03:32:48.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:47 smithi175 ceph-mon[30894]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-11T03:32:48.295 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:47 smithi084 ceph-mon[27254]: from='osd.0 [v2:172.21.15.84:6802/1376667411,v1:172.21.15.84:6803/1376667411]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-11T03:32:48.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:47 smithi084 ceph-mon[27254]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-11T03:32:48.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:47 smithi121 ceph-mon[30729]: from='osd.0 [v2:172.21.15.84:6802/1376667411,v1:172.21.15.84:6803/1376667411]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-11T03:32:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:47 smithi121 ceph-mon[30729]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-11T03:32:48.853 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:32:49.111 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:48 smithi084 ceph-mon[27254]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-11T03:32:49.111 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:48 smithi084 ceph-mon[27254]: osdmap e6: 1 total, 0 up, 1 in 2024-09-11T03:32:49.111 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:48 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:49.111 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:48 smithi084 ceph-mon[27254]: from='osd.0 [v2:172.21.15.84:6802/1376667411,v1:172.21.15.84:6803/1376667411]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:32:49.111 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:48 smithi084 ceph-mon[27254]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:32:49.111 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:48 smithi084 ceph-mon[27254]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:49.111 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:32:48 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0[41076]: 2024-09-11T03:32:48.970+0000 7f3e4b001640 -1 osd.0 0 waiting for initial osdmap 2024-09-11T03:32:49.112 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:32:48 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0[41076]: 2024-09-11T03:32:48.972+0000 7f3e46e28640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T03:32:49.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:48 smithi121 ceph-mon[30729]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-11T03:32:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:48 smithi121 ceph-mon[30729]: osdmap e6: 1 total, 0 up, 1 in 2024-09-11T03:32:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:48 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:48 smithi121 ceph-mon[30729]: from='osd.0 [v2:172.21.15.84:6802/1376667411,v1:172.21.15.84:6803/1376667411]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:32:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:48 smithi121 ceph-mon[30729]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:32:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:48 smithi121 ceph-mon[30729]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:49.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:48 smithi175 ceph-mon[30894]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-11T03:32:49.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:48 smithi175 ceph-mon[30894]: osdmap e6: 1 total, 0 up, 1 in 2024-09-11T03:32:49.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:48 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:49.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:48 smithi175 ceph-mon[30894]: from='osd.0 [v2:172.21.15.84:6802/1376667411,v1:172.21.15.84:6803/1376667411]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:32:49.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:48 smithi175 ceph-mon[30894]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:32:49.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:48 smithi175 ceph-mon[30894]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:50.134 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:49 smithi084 ceph-mon[27254]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-11T03:32:50.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:49 smithi084 ceph-mon[27254]: osdmap e7: 1 total, 0 up, 1 in 2024-09-11T03:32:50.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:49 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:50.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:49 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:50.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:49 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:49 smithi175 ceph-mon[30894]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-11T03:32:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:49 smithi175 ceph-mon[30894]: osdmap e7: 1 total, 0 up, 1 in 2024-09-11T03:32:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:49 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:50.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:49 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:50.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:49 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:49 smithi121 ceph-mon[30729]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-11T03:32:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:49 smithi121 ceph-mon[30729]: osdmap e7: 1 total, 0 up, 1 in 2024-09-11T03:32:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:49 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:49 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:49 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:51.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:50 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:32:51.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:50 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:32:51.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:50 smithi084 ceph-mon[27254]: osd.0 [v2:172.21.15.84:6802/1376667411,v1:172.21.15.84:6803/1376667411] boot 2024-09-11T03:32:51.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:50 smithi084 ceph-mon[27254]: osdmap e8: 1 total, 1 up, 1 in 2024-09-11T03:32:51.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:50 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:51.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:50 smithi084 ceph-mon[27254]: Detected new or changed devices on smithi084 2024-09-11T03:32:51.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:50 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:51.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:50 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:51.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:50 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-11T03:32:51.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:50 smithi084 ceph-mon[27254]: Adjusting osd_memory_target on smithi084 to 17088M 2024-09-11T03:32:51.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:50 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:51.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:50 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:51.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:50 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:51.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:50 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:51.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:50 smithi084 ceph-mon[27254]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:50 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:32:51.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:50 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:32:51.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:50 smithi175 ceph-mon[30894]: osd.0 [v2:172.21.15.84:6802/1376667411,v1:172.21.15.84:6803/1376667411] boot 2024-09-11T03:32:51.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:50 smithi175 ceph-mon[30894]: osdmap e8: 1 total, 1 up, 1 in 2024-09-11T03:32:51.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:50 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:51.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:50 smithi175 ceph-mon[30894]: Detected new or changed devices on smithi084 2024-09-11T03:32:51.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:50 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:51.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:50 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:51.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:50 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-11T03:32:51.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:50 smithi175 ceph-mon[30894]: Adjusting osd_memory_target on smithi084 to 17088M 2024-09-11T03:32:51.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:50 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:51.230 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:50 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:51.230 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:50 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:51.230 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:50 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:51.230 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:50 smithi175 ceph-mon[30894]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:50 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:32:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:50 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:32:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:50 smithi121 ceph-mon[30729]: osd.0 [v2:172.21.15.84:6802/1376667411,v1:172.21.15.84:6803/1376667411] boot 2024-09-11T03:32:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:50 smithi121 ceph-mon[30729]: osdmap e8: 1 total, 1 up, 1 in 2024-09-11T03:32:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:50 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:32:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:50 smithi121 ceph-mon[30729]: Detected new or changed devices on smithi084 2024-09-11T03:32:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:50 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:50 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:51.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:50 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-11T03:32:51.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:50 smithi121 ceph-mon[30729]: Adjusting osd_memory_target on smithi084 to 17088M 2024-09-11T03:32:51.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:50 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:51.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:50 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:51.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:50 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:32:51.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:50 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:32:51.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:50 smithi121 ceph-mon[30729]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T03:32:51.980 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:32:52.009 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi084:vg_nvme/lv_3 2024-09-11T03:32:52.253 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:32:52.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:52 smithi084 ceph-mon[27254]: osdmap e9: 1 total, 1 up, 1 in 2024-09-11T03:32:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:52 smithi175 ceph-mon[30894]: osdmap e9: 1 total, 1 up, 1 in 2024-09-11T03:32:52.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:52 smithi121 ceph-mon[30729]: osdmap e9: 1 total, 1 up, 1 in 2024-09-11T03:32:53.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:53 smithi175 ceph-mon[30894]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:32:53.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:53 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:32:53.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:53 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:32:53.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:53 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:53.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:53 smithi084 ceph-mon[27254]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:32:53.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:53 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:32:53.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:53 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:32:53.516 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:53 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:53.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:53 smithi121 ceph-mon[30729]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:32:53.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:53 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:32:53.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:53 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:32:53.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:53 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:32:54.244 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:54 smithi084 ceph-mon[27254]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:32:54.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:54 smithi175 ceph-mon[30894]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:32:54.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:54 smithi121 ceph-mon[30729]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:32:55.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:55 smithi084 ceph-mon[27254]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:32:55.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:55 smithi175 ceph-mon[30894]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:32:55.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:55 smithi121 ceph-mon[30729]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:32:56.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:56 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/888806600' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "62fd6e25-3508-40b4-a0f4-f7de137e3bb5"} : dispatch 2024-09-11T03:32:56.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:56 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/888806600' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62fd6e25-3508-40b4-a0f4-f7de137e3bb5"}]': finished 2024-09-11T03:32:56.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:56 smithi175 ceph-mon[30894]: osdmap e10: 2 total, 1 up, 2 in 2024-09-11T03:32:56.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:56 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:32:56.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:56 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1720925346' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:32:56.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/888806600' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "62fd6e25-3508-40b4-a0f4-f7de137e3bb5"} : dispatch 2024-09-11T03:32:56.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/888806600' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62fd6e25-3508-40b4-a0f4-f7de137e3bb5"}]': finished 2024-09-11T03:32:56.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:56 smithi084 ceph-mon[27254]: osdmap e10: 2 total, 1 up, 2 in 2024-09-11T03:32:56.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:32:56.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1720925346' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:32:56.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:56 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/888806600' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "62fd6e25-3508-40b4-a0f4-f7de137e3bb5"} : dispatch 2024-09-11T03:32:56.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:56 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/888806600' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62fd6e25-3508-40b4-a0f4-f7de137e3bb5"}]': finished 2024-09-11T03:32:56.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:56 smithi121 ceph-mon[30729]: osdmap e10: 2 total, 1 up, 2 in 2024-09-11T03:32:56.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:56 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:32:56.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:56 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1720925346' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:32:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:57 smithi175 ceph-mon[30894]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:32:57.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:57 smithi084 ceph-mon[27254]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:32:57.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:57 smithi121 ceph-mon[30729]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:32:58.825 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:32:58 smithi084 ceph-mon[27254]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:32:58.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:32:58 smithi175 ceph-mon[30894]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:32:59.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:32:58 smithi121 ceph-mon[30729]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:00.959 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:00 smithi084 ceph-mon[27254]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:00.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:00 smithi175 ceph-mon[30894]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:01.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:00 smithi121 ceph-mon[30729]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:02.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:02 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-11T03:33:02.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:02 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:02 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-11T03:33:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:02 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:02 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-11T03:33:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:02 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:03 smithi084 ceph-mon[27254]: Deploying daemon osd.1 on smithi084 2024-09-11T03:33:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:03 smithi084 ceph-mon[27254]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:03.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:03 smithi121 ceph-mon[30729]: Deploying daemon osd.1 on smithi084 2024-09-11T03:33:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:03 smithi121 ceph-mon[30729]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:03.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:03 smithi175 ceph-mon[30894]: Deploying daemon osd.1 on smithi084 2024-09-11T03:33:03.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:03 smithi175 ceph-mon[30894]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:04.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:04 smithi175 ceph-mon[30894]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:05.038 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:04 smithi084 ceph-mon[27254]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:05.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:04 smithi121 ceph-mon[30729]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:06.946 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:06 smithi084 ceph-mon[27254]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:06.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:06 smithi175 ceph-mon[30894]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:07.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:06 smithi121 ceph-mon[30729]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:08 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:33:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:08 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:08 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:08.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:08 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:33:08.600 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:08 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:08.600 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:08 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:08.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:08 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:33:08.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:08 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:08.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:08 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:09 smithi175 ceph-mon[30894]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:09.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:09 smithi084 ceph-mon[27254]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:09.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:09 smithi121 ceph-mon[30729]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:11.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:11 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:11.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:11 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:11.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:11 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:11.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:11 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:33:11.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:11 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:11.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:11 smithi084 ceph-mon[27254]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:11.924 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:11 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:11.924 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:11 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:11.924 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:11 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:11.924 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:11 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:33:11.924 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:11 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:11.924 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:11 smithi121 ceph-mon[30729]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:11.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:11 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:11.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:11 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:11.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:11 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:11.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:11 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:33:11.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:11 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:11.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:11 smithi175 ceph-mon[30894]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:12.105 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 1 on host 'smithi084' 2024-09-11T03:33:12.790 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:12 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:33:12.791 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:12 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:12.791 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:12 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:12.818 DEBUG:teuthology.orchestra.run.smithi084:osd.1> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.1.service 2024-09-11T03:33:12.821 INFO:tasks.cephadm:Deploying osd.2 on smithi084 with /dev/vg_nvme/lv_2... 2024-09-11T03:33:12.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-11T03:33:12.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:12 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:33:12.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:12 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:12.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:12 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:12.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:12 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:33:12.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:12 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:12.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:12 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:13.385 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:33:13 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-1[46583]: 2024-09-11T03:33:13.126+0000 7f2c82c20640 -1 osd.1 0 log_to_monitors true 2024-09-11T03:33:13.584 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:33:13.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:13 smithi084 ceph-mon[27254]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:13.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:13 smithi084 ceph-mon[27254]: from='osd.1 [v2:172.21.15.84:6810/3138939249,v1:172.21.15.84:6811/3138939249]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-11T03:33:13.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:13 smithi121 ceph-mon[30729]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:13.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:13 smithi121 ceph-mon[30729]: from='osd.1 [v2:172.21.15.84:6810/3138939249,v1:172.21.15.84:6811/3138939249]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-11T03:33:13.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:13 smithi175 ceph-mon[30894]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:13.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:13 smithi175 ceph-mon[30894]: from='osd.1 [v2:172.21.15.84:6810/3138939249,v1:172.21.15.84:6811/3138939249]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-11T03:33:14.870 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:33:14 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-1[46583]: 2024-09-11T03:33:14.586+0000 7f2c7eba3640 -1 osd.1 0 waiting for initial osdmap 2024-09-11T03:33:14.870 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:33:14 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-1[46583]: 2024-09-11T03:33:14.588+0000 7f2c7a9ca640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T03:33:14.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:14 smithi084 ceph-mon[27254]: from='osd.1 [v2:172.21.15.84:6810/3138939249,v1:172.21.15.84:6811/3138939249]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-11T03:33:14.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:14 smithi084 ceph-mon[27254]: osdmap e11: 2 total, 1 up, 2 in 2024-09-11T03:33:14.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:14 smithi084 ceph-mon[27254]: from='osd.1 [v2:172.21.15.84:6810/3138939249,v1:172.21.15.84:6811/3138939249]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:33:14.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:14 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:33:14.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:14 smithi121 ceph-mon[30729]: from='osd.1 [v2:172.21.15.84:6810/3138939249,v1:172.21.15.84:6811/3138939249]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-11T03:33:14.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:14 smithi121 ceph-mon[30729]: osdmap e11: 2 total, 1 up, 2 in 2024-09-11T03:33:14.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:14 smithi121 ceph-mon[30729]: from='osd.1 [v2:172.21.15.84:6810/3138939249,v1:172.21.15.84:6811/3138939249]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:33:14.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:14 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:33:14.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:14 smithi175 ceph-mon[30894]: from='osd.1 [v2:172.21.15.84:6810/3138939249,v1:172.21.15.84:6811/3138939249]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-11T03:33:14.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:14 smithi175 ceph-mon[30894]: osdmap e11: 2 total, 1 up, 2 in 2024-09-11T03:33:14.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:14 smithi175 ceph-mon[30894]: from='osd.1 [v2:172.21.15.84:6810/3138939249,v1:172.21.15.84:6811/3138939249]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:33:14.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:14 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:33:15.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:15 smithi084 ceph-mon[27254]: from='osd.1 [v2:172.21.15.84:6810/3138939249,v1:172.21.15.84:6811/3138939249]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-11T03:33:15.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:15 smithi084 ceph-mon[27254]: osdmap e12: 2 total, 1 up, 2 in 2024-09-11T03:33:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:33:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:33:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:15 smithi084 ceph-mon[27254]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:15.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:15 smithi121 ceph-mon[30729]: from='osd.1 [v2:172.21.15.84:6810/3138939249,v1:172.21.15.84:6811/3138939249]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-11T03:33:15.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:15 smithi121 ceph-mon[30729]: osdmap e12: 2 total, 1 up, 2 in 2024-09-11T03:33:15.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:33:15.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:33:15.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:15 smithi121 ceph-mon[30729]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:15.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:15 smithi175 ceph-mon[30894]: from='osd.1 [v2:172.21.15.84:6810/3138939249,v1:172.21.15.84:6811/3138939249]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-11T03:33:15.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:15 smithi175 ceph-mon[30894]: osdmap e12: 2 total, 1 up, 2 in 2024-09-11T03:33:15.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:33:15.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:33:15.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:15 smithi175 ceph-mon[30894]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-11T03:33:16.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:16 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:33:16.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:16 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:33:16.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:33:16.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:16 smithi084 ceph-mon[27254]: osd.1 [v2:172.21.15.84:6810/3138939249,v1:172.21.15.84:6811/3138939249] boot 2024-09-11T03:33:16.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:16 smithi084 ceph-mon[27254]: osdmap e13: 2 total, 2 up, 2 in 2024-09-11T03:33:16.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:33:16.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:16.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:16.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-11T03:33:16.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-11T03:33:16.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:16.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:16.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:33:16.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:16 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:33:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:16 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:33:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:33:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:16 smithi121 ceph-mon[30729]: osd.1 [v2:172.21.15.84:6810/3138939249,v1:172.21.15.84:6811/3138939249] boot 2024-09-11T03:33:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:16 smithi121 ceph-mon[30729]: osdmap e13: 2 total, 2 up, 2 in 2024-09-11T03:33:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:33:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-11T03:33:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-11T03:33:16.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:16.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:16.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:33:16.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:16.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:16 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:33:16.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:16 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:33:16.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:33:16.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:16 smithi175 ceph-mon[30894]: osd.1 [v2:172.21.15.84:6810/3138939249,v1:172.21.15.84:6811/3138939249] boot 2024-09-11T03:33:16.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:16 smithi175 ceph-mon[30894]: osdmap e13: 2 total, 2 up, 2 in 2024-09-11T03:33:16.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-11T03:33:16.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:16.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:16.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-11T03:33:16.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-11T03:33:16.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:16.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:16.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:33:16.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:17.664 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:33:17.689 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi084:vg_nvme/lv_2 2024-09-11T03:33:17.712 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:17 smithi084 ceph-mon[27254]: Detected new or changed devices on smithi084 2024-09-11T03:33:17.712 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:17 smithi084 ceph-mon[27254]: Adjusting osd_memory_target on smithi084 to 8544M 2024-09-11T03:33:17.712 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:17 smithi084 ceph-mon[27254]: osdmap e14: 2 total, 2 up, 2 in 2024-09-11T03:33:17.712 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:17 smithi084 ceph-mon[27254]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:17.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:17 smithi121 ceph-mon[30729]: Detected new or changed devices on smithi084 2024-09-11T03:33:17.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:17 smithi121 ceph-mon[30729]: Adjusting osd_memory_target on smithi084 to 8544M 2024-09-11T03:33:17.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:17 smithi121 ceph-mon[30729]: osdmap e14: 2 total, 2 up, 2 in 2024-09-11T03:33:17.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:17 smithi121 ceph-mon[30729]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:17.931 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:33:17.979 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:17 smithi175 ceph-mon[30894]: Detected new or changed devices on smithi084 2024-09-11T03:33:17.979 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:17 smithi175 ceph-mon[30894]: Adjusting osd_memory_target on smithi084 to 8544M 2024-09-11T03:33:17.979 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:17 smithi175 ceph-mon[30894]: osdmap e14: 2 total, 2 up, 2 in 2024-09-11T03:33:17.979 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:17 smithi175 ceph-mon[30894]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:18.643 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:33:18.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:33:18.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:18.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:33:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:33:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:18.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:33:18.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:33:18.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:19.861 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:19 smithi084 ceph-mon[27254]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:33:19.861 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:19 smithi084 ceph-mon[27254]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:19.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:19 smithi121 ceph-mon[30729]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:33:19.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:19 smithi121 ceph-mon[30729]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:19.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:19 smithi175 ceph-mon[30894]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:33:19.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:19 smithi175 ceph-mon[30894]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:20.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:20 smithi084 ceph-mon[27254]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:20.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:20 smithi175 ceph-mon[30894]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:21.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:20 smithi121 ceph-mon[30729]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:21.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/565669799' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ff351425-4715-49dc-be49-f27bf205d57e"} : dispatch 2024-09-11T03:33:21.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/565669799' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff351425-4715-49dc-be49-f27bf205d57e"}]': finished 2024-09-11T03:33:21.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:21 smithi084 ceph-mon[27254]: osdmap e15: 3 total, 2 up, 3 in 2024-09-11T03:33:21.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:21.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2379315151' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:33:21.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:21 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/565669799' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ff351425-4715-49dc-be49-f27bf205d57e"} : dispatch 2024-09-11T03:33:21.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:21 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/565669799' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff351425-4715-49dc-be49-f27bf205d57e"}]': finished 2024-09-11T03:33:21.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:21 smithi175 ceph-mon[30894]: osdmap e15: 3 total, 2 up, 3 in 2024-09-11T03:33:21.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:21.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:21 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2379315151' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:33:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:21 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/565669799' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ff351425-4715-49dc-be49-f27bf205d57e"} : dispatch 2024-09-11T03:33:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:21 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/565669799' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff351425-4715-49dc-be49-f27bf205d57e"}]': finished 2024-09-11T03:33:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:21 smithi121 ceph-mon[30729]: osdmap e15: 3 total, 2 up, 3 in 2024-09-11T03:33:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:21 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2379315151' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:33:22.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:22 smithi175 ceph-mon[30894]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:23.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:22 smithi084 ceph-mon[27254]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:23.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:22 smithi121 ceph-mon[30729]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:24.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:24 smithi175 ceph-mon[30894]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:25.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:24 smithi084 ceph-mon[27254]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:25.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:24 smithi121 ceph-mon[30729]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:26.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:26 smithi084 ceph-mon[27254]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:26.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:26 smithi175 ceph-mon[30894]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:27.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:26 smithi121 ceph-mon[30729]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:28.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:28 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-11T03:33:28.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:28 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:28.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:28 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-11T03:33:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:28 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:28.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:28 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-11T03:33:28.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:28 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:29.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:29 smithi084 ceph-mon[27254]: Deploying daemon osd.2 on smithi084 2024-09-11T03:33:29.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:29 smithi084 ceph-mon[27254]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:29.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:29 smithi121 ceph-mon[30729]: Deploying daemon osd.2 on smithi084 2024-09-11T03:33:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:29 smithi121 ceph-mon[30729]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:29.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:29 smithi175 ceph-mon[30894]: Deploying daemon osd.2 on smithi084 2024-09-11T03:33:29.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:29 smithi175 ceph-mon[30894]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:30.799 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:30 smithi084 ceph-mon[27254]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:30.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:30 smithi175 ceph-mon[30894]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:31.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:30 smithi121 ceph-mon[30729]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:32.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:32 smithi175 ceph-mon[30894]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:33.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:32 smithi084 ceph-mon[27254]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:33.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:32 smithi121 ceph-mon[30729]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:34.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:34 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:33:34.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:34 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:34.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:34 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:34.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:34 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:33:34.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:34 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:34.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:34 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:34.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:34 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:33:34.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:34 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:34.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:34 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:35.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:35 smithi175 ceph-mon[30894]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:35.732 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:35 smithi084 ceph-mon[27254]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:35.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:35 smithi121 ceph-mon[30729]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:36.722 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:36 smithi084 ceph-mon[27254]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:36.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:36 smithi175 ceph-mon[30894]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:36 smithi121 ceph-mon[30729]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:37.938 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 2 on host 'smithi084' 2024-09-11T03:33:38.248 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:38 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:38.248 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:38 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:38.248 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:38 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:38.248 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:38 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:33:38.248 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:38 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:38.248 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:38 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:33:38.248 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:38 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:38.248 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:38 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:38 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:38 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:38 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:38 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:33:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:38 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:38 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:33:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:38 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:38 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:38 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:38 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:38 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:38 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:33:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:38 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:38.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:38 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:33:38.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:38 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:38.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:38 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:38.607 DEBUG:teuthology.orchestra.run.smithi084:osd.2> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.2.service 2024-09-11T03:33:38.650 INFO:tasks.cephadm:Deploying osd.3 on smithi084 with /dev/vg_nvme/lv_1... 2024-09-11T03:33:38.650 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-11T03:33:39.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:39 smithi084 ceph-mon[27254]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:39.394 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:33:39 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-2[52201]: 2024-09-11T03:33:39.139+0000 7fcf5e8d5640 -1 osd.2 0 log_to_monitors true 2024-09-11T03:33:39.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:39 smithi121 ceph-mon[30729]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:39.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:39 smithi175 ceph-mon[30894]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:39.899 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:33:40.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:40 smithi084 ceph-mon[27254]: from='osd.2 [v2:172.21.15.84:6818/3001811591,v1:172.21.15.84:6819/3001811591]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-11T03:33:40.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:40 smithi121 ceph-mon[30729]: from='osd.2 [v2:172.21.15.84:6818/3001811591,v1:172.21.15.84:6819/3001811591]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-11T03:33:40.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:40 smithi175 ceph-mon[30894]: from='osd.2 [v2:172.21.15.84:6818/3001811591,v1:172.21.15.84:6819/3001811591]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-11T03:33:41.394 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:33:41 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-2[52201]: 2024-09-11T03:33:41.020+0000 7fcf5a858640 -1 osd.2 0 waiting for initial osdmap 2024-09-11T03:33:41.394 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:33:41 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-2[52201]: 2024-09-11T03:33:41.024+0000 7fcf5667f640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T03:33:41.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:41 smithi084 ceph-mon[27254]: from='osd.2 [v2:172.21.15.84:6818/3001811591,v1:172.21.15.84:6819/3001811591]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-11T03:33:41.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:41 smithi084 ceph-mon[27254]: osdmap e16: 3 total, 2 up, 3 in 2024-09-11T03:33:41.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:41 smithi084 ceph-mon[27254]: from='osd.2 [v2:172.21.15.84:6818/3001811591,v1:172.21.15.84:6819/3001811591]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:33:41.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:41 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:41.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:41 smithi084 ceph-mon[27254]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:41 smithi121 ceph-mon[30729]: from='osd.2 [v2:172.21.15.84:6818/3001811591,v1:172.21.15.84:6819/3001811591]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-11T03:33:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:41 smithi121 ceph-mon[30729]: osdmap e16: 3 total, 2 up, 3 in 2024-09-11T03:33:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:41 smithi121 ceph-mon[30729]: from='osd.2 [v2:172.21.15.84:6818/3001811591,v1:172.21.15.84:6819/3001811591]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:33:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:41 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:41 smithi121 ceph-mon[30729]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:41.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:41 smithi175 ceph-mon[30894]: from='osd.2 [v2:172.21.15.84:6818/3001811591,v1:172.21.15.84:6819/3001811591]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-11T03:33:41.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:41 smithi175 ceph-mon[30894]: osdmap e16: 3 total, 2 up, 3 in 2024-09-11T03:33:41.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:41 smithi175 ceph-mon[30894]: from='osd.2 [v2:172.21.15.84:6818/3001811591,v1:172.21.15.84:6819/3001811591]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:33:41.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:41 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:41.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:41 smithi175 ceph-mon[30894]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:42.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:42 smithi084 ceph-mon[27254]: from='osd.2 [v2:172.21.15.84:6818/3001811591,v1:172.21.15.84:6819/3001811591]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-11T03:33:42.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:42 smithi084 ceph-mon[27254]: osdmap e17: 3 total, 2 up, 3 in 2024-09-11T03:33:42.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:42 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:42.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:42 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:42.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:42 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:42 smithi121 ceph-mon[30729]: from='osd.2 [v2:172.21.15.84:6818/3001811591,v1:172.21.15.84:6819/3001811591]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-11T03:33:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:42 smithi121 ceph-mon[30729]: osdmap e17: 3 total, 2 up, 3 in 2024-09-11T03:33:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:42 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:42 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:42 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:42.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:42 smithi175 ceph-mon[30894]: from='osd.2 [v2:172.21.15.84:6818/3001811591,v1:172.21.15.84:6819/3001811591]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-11T03:33:42.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:42 smithi175 ceph-mon[30894]: osdmap e17: 3 total, 2 up, 3 in 2024-09-11T03:33:42.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:42 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:42.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:42 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:42.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:42 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:43.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:33:43.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:33:43.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: Detected new or changed devices on smithi084 2024-09-11T03:33:43.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:43.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:43.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: osd.2 [v2:172.21.15.84:6818/3001811591,v1:172.21.15.84:6819/3001811591] boot 2024-09-11T03:33:43.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: osdmap e18: 3 total, 3 up, 3 in 2024-09-11T03:33:43.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:43.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-11T03:33:43.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-11T03:33:43.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-11T03:33:43.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: Adjusting osd_memory_target on smithi084 to 5696M 2024-09-11T03:33:43.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:43.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:43.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:33:43.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:43.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:43.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-11T03:33:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:33:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:33:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: Detected new or changed devices on smithi084 2024-09-11T03:33:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: osd.2 [v2:172.21.15.84:6818/3001811591,v1:172.21.15.84:6819/3001811591] boot 2024-09-11T03:33:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: osdmap e18: 3 total, 3 up, 3 in 2024-09-11T03:33:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-11T03:33:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-11T03:33:43.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-11T03:33:43.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: Adjusting osd_memory_target on smithi084 to 5696M 2024-09-11T03:33:43.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:43.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:43.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:33:43.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:43.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:43.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-11T03:33:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:33:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:33:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: Detected new or changed devices on smithi084 2024-09-11T03:33:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: osd.2 [v2:172.21.15.84:6818/3001811591,v1:172.21.15.84:6819/3001811591] boot 2024-09-11T03:33:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: osdmap e18: 3 total, 3 up, 3 in 2024-09-11T03:33:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-11T03:33:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-11T03:33:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-11T03:33:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-11T03:33:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: Adjusting osd_memory_target on smithi084 to 5696M 2024-09-11T03:33:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:33:43.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:43.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-11T03:33:43.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-11T03:33:43.610 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:33:43.635 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi084:vg_nvme/lv_1 2024-09-11T03:33:43.877 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:33:44.251 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:44 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-11T03:33:44.251 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:44 smithi084 ceph-mon[27254]: osdmap e19: 3 total, 3 up, 3 in 2024-09-11T03:33:44.251 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:44 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-11T03:33:44.251 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56296]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T03:33:44.252 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56296]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T03:33:44.252 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56296]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T03:33:44.252 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56296]: pam_unix(sudo:session): session closed for user root 2024-09-11T03:33:44.252 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56290]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T03:33:44.252 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56290]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T03:33:44.252 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56290]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T03:33:44.252 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56290]: pam_unix(sudo:session): session closed for user root 2024-09-11T03:33:44.252 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56293]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T03:33:44.252 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56293]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T03:33:44.252 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56293]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T03:33:44.253 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56293]: pam_unix(sudo:session): session closed for user root 2024-09-11T03:33:44.354 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:44 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-11T03:33:44.355 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:44 smithi175 ceph-mon[30894]: osdmap e19: 3 total, 3 up, 3 in 2024-09-11T03:33:44.355 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:44 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-11T03:33:44.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:44 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-11T03:33:44.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:44 smithi121 ceph-mon[30729]: osdmap e19: 3 total, 3 up, 3 in 2024-09-11T03:33:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:44 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-11T03:33:44.644 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56299]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T03:33:44.644 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56299]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T03:33:44.644 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56299]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T03:33:44.644 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56299]: pam_unix(sudo:session): session closed for user root 2024-09-11T03:33:44.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56310]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T03:33:44.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56310]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T03:33:44.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56310]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T03:33:44.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56310]: pam_unix(sudo:session): session closed for user root 2024-09-11T03:33:44.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56314]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T03:33:44.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56314]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T03:33:44.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56314]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T03:33:44.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56314]: pam_unix(sudo:session): session closed for user root 2024-09-11T03:33:44.645 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56304]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T03:33:44.645 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56304]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T03:33:44.646 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56304]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T03:33:44.646 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56304]: pam_unix(sudo:session): session closed for user root 2024-09-11T03:33:44.646 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56307]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T03:33:44.646 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56307]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T03:33:44.646 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56307]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T03:33:44.646 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:33:44 smithi084 sudo[56307]: pam_unix(sudo:session): session closed for user root 2024-09-11T03:33:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:44 smithi175 sudo[32819]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T03:33:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:44 smithi175 sudo[32819]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T03:33:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:44 smithi175 sudo[32819]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T03:33:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:44 smithi175 sudo[32819]: pam_unix(sudo:session): session closed for user root 2024-09-11T03:33:44.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:44 smithi175 sudo[32822]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T03:33:44.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:44 smithi175 sudo[32822]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T03:33:44.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:44 smithi175 sudo[32822]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T03:33:44.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:44 smithi175 sudo[32822]: pam_unix(sudo:session): session closed for user root 2024-09-11T03:33:44.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:44 smithi121 sudo[32728]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T03:33:44.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:44 smithi121 sudo[32728]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T03:33:44.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:44 smithi121 sudo[32728]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T03:33:44.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:44 smithi121 sudo[32728]: pam_unix(sudo:session): session closed for user root 2024-09-11T03:33:44.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:44 smithi121 sudo[32731]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T03:33:44.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:44 smithi121 sudo[32731]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T03:33:44.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:44 smithi121 sudo[32731]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T03:33:44.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:44 smithi121 sudo[32731]: pam_unix(sudo:session): session closed for user root 2024-09-11T03:33:45.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-11T03:33:45.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: osdmap e20: 3 total, 3 up, 3 in 2024-09-11T03:33:45.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:45.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T03:33:45.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T03:33:45.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:33:45.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:33:45.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:33:45.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T03:33:45.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:33:45.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:33:45.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:33:45.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T03:33:45.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:45.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T03:33:45.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:33:45.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:33:45.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:33:45.273 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T03:33:45.273 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:45.273 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:33:45.273 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:33:45.273 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:33:45.273 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:45.273 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:45 smithi084 ceph-mon[27254]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-11T03:33:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: osdmap e20: 3 total, 3 up, 3 in 2024-09-11T03:33:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T03:33:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T03:33:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:33:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:33:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:33:45.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T03:33:45.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:33:45.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:33:45.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:33:45.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T03:33:45.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:45.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T03:33:45.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:33:45.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:33:45.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:33:45.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T03:33:45.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:45.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:33:45.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:33:45.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:33:45.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:45.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:45 smithi121 ceph-mon[30729]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:45.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-11T03:33:45.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: osdmap e20: 3 total, 3 up, 3 in 2024-09-11T03:33:45.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:45.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T03:33:45.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T03:33:45.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:33:45.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:33:45.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:33:45.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T03:33:45.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:33:45.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:33:45.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:33:45.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T03:33:45.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:45.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T03:33:45.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:33:45.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:33:45.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:33:45.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T03:33:45.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:33:45.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:33:45.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:33:45.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:33:45.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:45.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:45 smithi175 ceph-mon[30894]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:46.190 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:46 smithi084 ceph-mon[27254]: osdmap e21: 3 total, 3 up, 3 in 2024-09-11T03:33:46.191 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:46 smithi084 ceph-mon[27254]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-11T03:33:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:46 smithi121 ceph-mon[30729]: osdmap e21: 3 total, 3 up, 3 in 2024-09-11T03:33:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:46 smithi121 ceph-mon[30729]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-11T03:33:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:46 smithi175 ceph-mon[30894]: osdmap e21: 3 total, 3 up, 3 in 2024-09-11T03:33:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:46 smithi175 ceph-mon[30894]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-11T03:33:47.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:47 smithi084 ceph-mon[27254]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:47.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:47 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4289299356' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2518a93e-c612-4731-91ce-5685c25ffa9f"} : dispatch 2024-09-11T03:33:47.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:47 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4289299356' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2518a93e-c612-4731-91ce-5685c25ffa9f"}]': finished 2024-09-11T03:33:47.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:47 smithi084 ceph-mon[27254]: osdmap e22: 4 total, 3 up, 4 in 2024-09-11T03:33:47.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:47 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:33:47.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:47 smithi121 ceph-mon[30729]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:47 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4289299356' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2518a93e-c612-4731-91ce-5685c25ffa9f"} : dispatch 2024-09-11T03:33:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:47 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4289299356' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2518a93e-c612-4731-91ce-5685c25ffa9f"}]': finished 2024-09-11T03:33:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:47 smithi121 ceph-mon[30729]: osdmap e22: 4 total, 3 up, 4 in 2024-09-11T03:33:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:47 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:33:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:47 smithi175 ceph-mon[30894]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:47 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4289299356' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2518a93e-c612-4731-91ce-5685c25ffa9f"} : dispatch 2024-09-11T03:33:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:47 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4289299356' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2518a93e-c612-4731-91ce-5685c25ffa9f"}]': finished 2024-09-11T03:33:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:47 smithi175 ceph-mon[30894]: osdmap e22: 4 total, 3 up, 4 in 2024-09-11T03:33:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:47 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:33:48.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:48 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/14911203' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:33:48.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:48 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/14911203' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:33:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:48 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/14911203' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:33:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:49 smithi084 ceph-mon[27254]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:49.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:49 smithi121 ceph-mon[30729]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:49 smithi175 ceph-mon[30894]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:50.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:50 smithi084 ceph-mon[27254]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:50.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:50 smithi175 ceph-mon[30894]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:51.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:50 smithi121 ceph-mon[30729]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:52.822 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:52 smithi084 ceph-mon[27254]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:52.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:52 smithi175 ceph-mon[30894]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:53.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:52 smithi121 ceph-mon[30729]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:54.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:54 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-11T03:33:54.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:54 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:54.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:54 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-11T03:33:54.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:54 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:54.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:54 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-11T03:33:54.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:54 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:33:55.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:55 smithi175 ceph-mon[30894]: Deploying daemon osd.3 on smithi084 2024-09-11T03:33:55.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:55 smithi175 ceph-mon[30894]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:55.552 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:55 smithi084 ceph-mon[27254]: Deploying daemon osd.3 on smithi084 2024-09-11T03:33:55.552 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:55 smithi084 ceph-mon[27254]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:55.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:55 smithi121 ceph-mon[30729]: Deploying daemon osd.3 on smithi084 2024-09-11T03:33:55.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:55 smithi121 ceph-mon[30729]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:56.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:56 smithi084 ceph-mon[27254]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:56.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:56 smithi175 ceph-mon[30894]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:57.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:56 smithi121 ceph-mon[30729]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:58.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:33:58 smithi084 ceph-mon[27254]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:58.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:33:58 smithi175 ceph-mon[30894]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:33:59.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:33:58 smithi121 ceph-mon[30729]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:34:00.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:00 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:34:00.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:00 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:34:00.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:00 smithi175 ceph-mon[30894]: osdmap e23: 4 total, 3 up, 4 in 2024-09-11T03:34:00.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:00.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:00.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:00.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:00.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:00 smithi175 ceph-mon[30894]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:34:01.017 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:00 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:34:01.017 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:00 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:34:01.017 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:00 smithi084 ceph-mon[27254]: osdmap e23: 4 total, 3 up, 4 in 2024-09-11T03:34:01.018 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:01.018 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:01.018 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:01.018 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:01.018 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:00 smithi084 ceph-mon[27254]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:34:01.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:00 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:34:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:00 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:34:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:00 smithi121 ceph-mon[30729]: osdmap e23: 4 total, 3 up, 4 in 2024-09-11T03:34:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:00 smithi121 ceph-mon[30729]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:34:03.350 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 3 on host 'smithi084' 2024-09-11T03:34:03.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:03.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:03.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:03 smithi084 ceph-mon[27254]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:34:03.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:03.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:03.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:03.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:03.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:03.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:03.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:03.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:03.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:03 smithi121 ceph-mon[30729]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:34:03.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:03.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:03.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:03.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:03.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:03.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:03.941 DEBUG:teuthology.orchestra.run.smithi084:osd.3> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.3.service 2024-09-11T03:34:03.944 INFO:tasks.cephadm:Deploying osd.4 on smithi121 with /dev/vg_nvme/lv_4... 2024-09-11T03:34:03.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-11T03:34:03.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:03.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:03.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:03 smithi175 ceph-mon[30894]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:34:03.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:03.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:03.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:03.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:03.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:03.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:04.196 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:34:04.752 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:04 smithi084 ceph-mon[27254]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:34:04.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:04 smithi121 ceph-mon[30729]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:34:04.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:04 smithi175 ceph-mon[30894]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:34:05.137 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 11 03:34:04 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-3[58214]: 2024-09-11T03:34:04.751+0000 7ff44ac67640 -1 osd.3 0 log_to_monitors true 2024-09-11T03:34:05.813 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:05 smithi121 ceph-mon[30729]: from='osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-11T03:34:05.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:05 smithi175 ceph-mon[30894]: from='osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-11T03:34:06.126 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:05 smithi084 ceph-mon[27254]: from='osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-11T03:34:06.486 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:34:06.511 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi121:vg_nvme/lv_4 2024-09-11T03:34:06.760 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:34:06.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:06 smithi121 ceph-mon[30729]: from='osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-11T03:34:06.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:06 smithi121 ceph-mon[30729]: osdmap e24: 4 total, 3 up, 4 in 2024-09-11T03:34:06.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:06 smithi121 ceph-mon[30729]: from='osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:34:06.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:06 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:06.969 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:06 smithi121 ceph-mon[30729]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:34:06.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:06 smithi175 ceph-mon[30894]: from='osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-11T03:34:06.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:06 smithi175 ceph-mon[30894]: osdmap e24: 4 total, 3 up, 4 in 2024-09-11T03:34:06.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:06 smithi175 ceph-mon[30894]: from='osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:34:06.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:06.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:06 smithi175 ceph-mon[30894]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:34:07.130 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:06 smithi084 ceph-mon[27254]: from='osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-11T03:34:07.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:06 smithi084 ceph-mon[27254]: osdmap e24: 4 total, 3 up, 4 in 2024-09-11T03:34:07.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:06 smithi084 ceph-mon[27254]: from='osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:34:07.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:07.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:06 smithi084 ceph-mon[27254]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:34:07.394 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-3[58214]: 2024-09-11T03:34:07.130+0000 7ff446bea640 -1 osd.3 0 waiting for initial osdmap 2024-09-11T03:34:07.394 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-3[58214]: 2024-09-11T03:34:07.135+0000 7ff43ea09640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T03:34:07.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-11T03:34:07.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: osdmap e25: 4 total, 3 up, 4 in 2024-09-11T03:34:07.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:07.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:07.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: Detected new or changed devices on smithi084 2024-09-11T03:34:07.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:07.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:07.973 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:07.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:07.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:07.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:07.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: Adjusting osd_memory_target on smithi084 to 4272M 2024-09-11T03:34:07.974 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:07.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:07.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:07.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:07.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129]' entity='osd.3' 2024-09-11T03:34:07.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:34:07.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:34:07.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:34:07.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:07.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:07.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-11T03:34:07.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: osdmap e25: 4 total, 3 up, 4 in 2024-09-11T03:34:07.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:07.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:07.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: Detected new or changed devices on smithi084 2024-09-11T03:34:07.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:07.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:07.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:07.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:07.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:07.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:07.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: Adjusting osd_memory_target on smithi084 to 4272M 2024-09-11T03:34:07.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:07.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:07.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:07.979 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:07.979 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129]' entity='osd.3' 2024-09-11T03:34:07.979 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:34:07.979 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:34:07.979 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:34:07.979 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:07.979 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:08.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-11T03:34:08.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: osdmap e25: 4 total, 3 up, 4 in 2024-09-11T03:34:08.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:08.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:08.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: Detected new or changed devices on smithi084 2024-09-11T03:34:08.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:08.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:08.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:08.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:08.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:08.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:08.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: Adjusting osd_memory_target on smithi084 to 4272M 2024-09-11T03:34:08.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:08.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:08.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:08.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:08.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129]' entity='osd.3' 2024-09-11T03:34:08.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:34:08.126 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:34:08.126 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:34:08.126 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:08.126 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:08.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:08 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:34:08.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:08 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:34:08.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:08 smithi084 ceph-mon[27254]: osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129] boot 2024-09-11T03:34:08.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:08 smithi084 ceph-mon[27254]: osdmap e26: 4 total, 4 up, 4 in 2024-09-11T03:34:08.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:08 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:08.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:08 smithi084 ceph-mon[27254]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:34:08.962 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:08 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:34:08.962 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:08 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:34:08.962 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:08 smithi121 ceph-mon[30729]: osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129] boot 2024-09-11T03:34:08.962 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:08 smithi121 ceph-mon[30729]: osdmap e26: 4 total, 4 up, 4 in 2024-09-11T03:34:08.962 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:08 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:08.962 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:08 smithi121 ceph-mon[30729]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:34:08.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:08 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:34:08.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:08 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:34:08.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:08 smithi175 ceph-mon[30894]: osd.3 [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129] boot 2024-09-11T03:34:08.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:08 smithi175 ceph-mon[30894]: osdmap e26: 4 total, 4 up, 4 in 2024-09-11T03:34:08.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:08 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-11T03:34:08.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:08 smithi175 ceph-mon[30894]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-11T03:34:10.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:10 smithi084 ceph-mon[27254]: from='client.? 172.21.15.121:0/2999554561' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d12f3c2d-95cc-4276-a7e1-bee982d58948"} : dispatch 2024-09-11T03:34:10.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:10 smithi084 ceph-mon[27254]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d12f3c2d-95cc-4276-a7e1-bee982d58948"} : dispatch 2024-09-11T03:34:10.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:10 smithi084 ceph-mon[27254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d12f3c2d-95cc-4276-a7e1-bee982d58948"}]': finished 2024-09-11T03:34:10.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:10 smithi084 ceph-mon[27254]: osdmap e27: 5 total, 4 up, 5 in 2024-09-11T03:34:10.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:10 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:10.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:10 smithi084 ceph-mon[27254]: from='client.? 172.21.15.121:0/1173098733' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:34:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.121:0/2999554561' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d12f3c2d-95cc-4276-a7e1-bee982d58948"} : dispatch 2024-09-11T03:34:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:10 smithi121 ceph-mon[30729]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d12f3c2d-95cc-4276-a7e1-bee982d58948"} : dispatch 2024-09-11T03:34:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:10 smithi121 ceph-mon[30729]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d12f3c2d-95cc-4276-a7e1-bee982d58948"}]': finished 2024-09-11T03:34:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:10 smithi121 ceph-mon[30729]: osdmap e27: 5 total, 4 up, 5 in 2024-09-11T03:34:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.121:0/1173098733' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:34:10.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:10 smithi175 ceph-mon[30894]: from='client.? 172.21.15.121:0/2999554561' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d12f3c2d-95cc-4276-a7e1-bee982d58948"} : dispatch 2024-09-11T03:34:10.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:10 smithi175 ceph-mon[30894]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d12f3c2d-95cc-4276-a7e1-bee982d58948"} : dispatch 2024-09-11T03:34:10.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:10 smithi175 ceph-mon[30894]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d12f3c2d-95cc-4276-a7e1-bee982d58948"}]': finished 2024-09-11T03:34:10.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:10 smithi175 ceph-mon[30894]: osdmap e27: 5 total, 4 up, 5 in 2024-09-11T03:34:10.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:10 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:10.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:10 smithi175 ceph-mon[30894]: from='client.? 172.21.15.121:0/1173098733' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:34:11.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:11 smithi084 ceph-mon[27254]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:11 smithi121 ceph-mon[30729]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:11 smithi175 ceph-mon[30894]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:12.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:12 smithi175 ceph-mon[30894]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:13.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:12 smithi084 ceph-mon[27254]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:13.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:12 smithi121 ceph-mon[30729]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:14.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:14 smithi121 ceph-mon[30729]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:14.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:14 smithi175 ceph-mon[30894]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:15.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:14 smithi084 ceph-mon[27254]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:16.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:16 smithi121 ceph-mon[30729]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:16.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:16 smithi175 ceph-mon[30894]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:17.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:16 smithi084 ceph-mon[27254]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:17.951 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:17 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-11T03:34:17.951 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:17 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:17.952 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:17 smithi121 ceph-mon[30729]: Deploying daemon osd.4 on smithi121 2024-09-11T03:34:17.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:17 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-11T03:34:17.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:17 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:17.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:17 smithi175 ceph-mon[30894]: Deploying daemon osd.4 on smithi121 2024-09-11T03:34:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:17 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-11T03:34:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:17 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:17 smithi084 ceph-mon[27254]: Deploying daemon osd.4 on smithi121 2024-09-11T03:34:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:18 smithi121 ceph-mon[30729]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:18.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:18 smithi175 ceph-mon[30894]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:19.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:18 smithi084 ceph-mon[27254]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:19.962 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:19 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:34:19.962 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:19 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:34:19.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:19 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:34:19.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:19 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:34:20.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:19 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:34:20.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:19 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:34:20.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:20 smithi175 ceph-mon[30894]: osdmap e28: 5 total, 4 up, 5 in 2024-09-11T03:34:20.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:20 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:20.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:20 smithi175 ceph-mon[30894]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:21.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:20 smithi084 ceph-mon[27254]: osdmap e28: 5 total, 4 up, 5 in 2024-09-11T03:34:21.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:20 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:21.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:20 smithi084 ceph-mon[27254]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:21.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:20 smithi121 ceph-mon[30729]: osdmap e28: 5 total, 4 up, 5 in 2024-09-11T03:34:21.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:20 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:21.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:20 smithi121 ceph-mon[30729]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:22.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:22 smithi175 ceph-mon[30894]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:23.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:22 smithi084 ceph-mon[27254]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:23.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:22 smithi121 ceph-mon[30729]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:23.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:23 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:23.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:23 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:23.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:23 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:23.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:23 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:23.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:23 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:23.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:23 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:23 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:23 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:23 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:24.762 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:24 smithi121 ceph-mon[30729]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:24.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:24 smithi175 ceph-mon[30894]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:25.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:24 smithi084 ceph-mon[27254]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:26.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:26.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:26.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:26.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:26.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:26.798 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 4 on host 'smithi121' 2024-09-11T03:34:27.353 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:27.354 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:27.354 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:27.382 DEBUG:teuthology.orchestra.run.smithi121:osd.4> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.4.service 2024-09-11T03:34:27.384 INFO:tasks.cephadm:Deploying osd.5 on smithi121 with /dev/vg_nvme/lv_3... 2024-09-11T03:34:27.384 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-11T03:34:27.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:27.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:27.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:27.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:27.751 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:27.751 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:28.171 INFO:journalctl@ceph.osd.4.smithi121.stdout:Sep 11 03:34:28 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-4[35097]: 2024-09-11T03:34:28.070+0000 7fa5fbca5640 -1 osd.4 0 log_to_monitors true 2024-09-11T03:34:28.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:28 smithi121 ceph-mon[30729]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:28.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:28 smithi121 ceph-mon[30729]: from='osd.4 [v2:172.21.15.121:6800/3873214513,v1:172.21.15.121:6801/3873214513]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-11T03:34:28.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:28 smithi121 ceph-mon[30729]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-11T03:34:28.566 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:34:28.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:28 smithi084 ceph-mon[27254]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:28.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:28 smithi084 ceph-mon[27254]: from='osd.4 [v2:172.21.15.121:6800/3873214513,v1:172.21.15.121:6801/3873214513]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-11T03:34:28.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:28 smithi084 ceph-mon[27254]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-11T03:34:28.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:28 smithi175 ceph-mon[30894]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:28.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:28 smithi175 ceph-mon[30894]: from='osd.4 [v2:172.21.15.121:6800/3873214513,v1:172.21.15.121:6801/3873214513]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-11T03:34:28.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:28 smithi175 ceph-mon[30894]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-11T03:34:29.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:29 smithi121 ceph-mon[30729]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-11T03:34:29.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:29 smithi121 ceph-mon[30729]: osdmap e29: 5 total, 4 up, 5 in 2024-09-11T03:34:29.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:29 smithi121 ceph-mon[30729]: from='osd.4 [v2:172.21.15.121:6800/3873214513,v1:172.21.15.121:6801/3873214513]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:34:29.607 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:29 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:29.608 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:29 smithi121 ceph-mon[30729]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:34:29.608 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:29 smithi121 ceph-mon[30729]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-09-11T03:34:29.608 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:29 smithi121 ceph-mon[30729]: osdmap e30: 5 total, 4 up, 5 in 2024-09-11T03:34:29.608 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:29 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:29.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:29 smithi084 ceph-mon[27254]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-11T03:34:29.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:29 smithi084 ceph-mon[27254]: osdmap e29: 5 total, 4 up, 5 in 2024-09-11T03:34:29.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:29 smithi084 ceph-mon[27254]: from='osd.4 [v2:172.21.15.121:6800/3873214513,v1:172.21.15.121:6801/3873214513]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:34:29.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:29 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:29.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:29 smithi084 ceph-mon[27254]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:34:29.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:29 smithi084 ceph-mon[27254]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-09-11T03:34:29.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:29 smithi084 ceph-mon[27254]: osdmap e30: 5 total, 4 up, 5 in 2024-09-11T03:34:29.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:29 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:29.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:29 smithi175 ceph-mon[30894]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-11T03:34:29.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:29 smithi175 ceph-mon[30894]: osdmap e29: 5 total, 4 up, 5 in 2024-09-11T03:34:29.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:29 smithi175 ceph-mon[30894]: from='osd.4 [v2:172.21.15.121:6800/3873214513,v1:172.21.15.121:6801/3873214513]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:34:29.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:29 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:29.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:29 smithi175 ceph-mon[30894]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:34:29.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:29 smithi175 ceph-mon[30894]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-09-11T03:34:29.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:29 smithi175 ceph-mon[30894]: osdmap e30: 5 total, 4 up, 5 in 2024-09-11T03:34:29.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:29 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:29.921 INFO:journalctl@ceph.osd.4.smithi121.stdout:Sep 11 03:34:29 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-4[35097]: 2024-09-11T03:34:29.702+0000 7fa5f7c28640 -1 osd.4 0 waiting for initial osdmap 2024-09-11T03:34:29.922 INFO:journalctl@ceph.osd.4.smithi121.stdout:Sep 11 03:34:29 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-4[35097]: 2024-09-11T03:34:29.706+0000 7fa5efa47640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T03:34:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:30 smithi084 ceph-mon[27254]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:30 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:30 smithi084 ceph-mon[27254]: from='osd.4 ' entity='osd.4' 2024-09-11T03:34:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:30 smithi084 ceph-mon[27254]: osd.4 [v2:172.21.15.121:6800/3873214513,v1:172.21.15.121:6801/3873214513] boot 2024-09-11T03:34:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:30 smithi084 ceph-mon[27254]: osdmap e31: 5 total, 5 up, 5 in 2024-09-11T03:34:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:30 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:30 smithi121 ceph-mon[30729]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:30 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:30 smithi121 ceph-mon[30729]: from='osd.4 ' entity='osd.4' 2024-09-11T03:34:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:30 smithi121 ceph-mon[30729]: osd.4 [v2:172.21.15.121:6800/3873214513,v1:172.21.15.121:6801/3873214513] boot 2024-09-11T03:34:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:30 smithi121 ceph-mon[30729]: osdmap e31: 5 total, 5 up, 5 in 2024-09-11T03:34:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:30 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:30.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:30 smithi175 ceph-mon[30894]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-11T03:34:30.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:30 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:30.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:30 smithi175 ceph-mon[30894]: from='osd.4 ' entity='osd.4' 2024-09-11T03:34:30.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:30 smithi175 ceph-mon[30894]: osd.4 [v2:172.21.15.121:6800/3873214513,v1:172.21.15.121:6801/3873214513] boot 2024-09-11T03:34:30.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:30 smithi175 ceph-mon[30894]: osdmap e31: 5 total, 5 up, 5 in 2024-09-11T03:34:30.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:30 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-11T03:34:31.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:31 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:34:31.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:31 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:34:31.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:31 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:31.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:31 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:31.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:31 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:31.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:31 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:31.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:31 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:31.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:31 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:31.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:31 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:31.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:31 smithi121 ceph-mon[30729]: osdmap e32: 5 total, 5 up, 5 in 2024-09-11T03:34:31.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:31 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:34:31.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:31 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:34:31.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:31 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:31.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:31 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:31.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:31 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:31.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:31 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:31.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:31 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:31.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:31 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:31.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:31 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:31.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:31 smithi084 ceph-mon[27254]: osdmap e32: 5 total, 5 up, 5 in 2024-09-11T03:34:31.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:31 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:34:31.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:31 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:34:31.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:31 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:31.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:31 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:31.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:31 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:31.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:31 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:31.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:31 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:31.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:31 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:31.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:31 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:31.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:31 smithi175 ceph-mon[30894]: osdmap e32: 5 total, 5 up, 5 in 2024-09-11T03:34:32.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:32 smithi121 ceph-mon[30729]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:32.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:32 smithi121 ceph-mon[30729]: Detected new or changed devices on smithi121 2024-09-11T03:34:32.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:32 smithi121 ceph-mon[30729]: Adjusting osd_memory_target on smithi121 to 17087M 2024-09-11T03:34:32.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:32 smithi121 ceph-mon[30729]: osdmap e33: 5 total, 5 up, 5 in 2024-09-11T03:34:32.466 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:34:32.492 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi121:vg_nvme/lv_3 2024-09-11T03:34:32.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:32 smithi084 ceph-mon[27254]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:32.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:32 smithi084 ceph-mon[27254]: Detected new or changed devices on smithi121 2024-09-11T03:34:32.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:32 smithi084 ceph-mon[27254]: Adjusting osd_memory_target on smithi121 to 17087M 2024-09-11T03:34:32.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:32 smithi084 ceph-mon[27254]: osdmap e33: 5 total, 5 up, 5 in 2024-09-11T03:34:32.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:32 smithi175 ceph-mon[30894]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:32.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:32 smithi175 ceph-mon[30894]: Detected new or changed devices on smithi121 2024-09-11T03:34:32.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:32 smithi175 ceph-mon[30894]: Adjusting osd_memory_target on smithi121 to 17087M 2024-09-11T03:34:32.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:32 smithi175 ceph-mon[30894]: osdmap e33: 5 total, 5 up, 5 in 2024-09-11T03:34:32.733 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:34:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:34 smithi121 ceph-mon[30729]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:34 smithi121 ceph-mon[30729]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:34:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:34 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:34:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:34 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:34:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:34 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:34.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:34 smithi084 ceph-mon[27254]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:34.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:34 smithi084 ceph-mon[27254]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:34:34.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:34 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:34:34.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:34 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:34:34.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:34 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:34.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:34 smithi175 ceph-mon[30894]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:34.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:34 smithi175 ceph-mon[30894]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:34:34.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:34 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:34:34.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:34 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:34:34.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:34 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:36.577 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:36 smithi121 ceph-mon[30729]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:36.578 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.121:0/2428273413' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "32164dd5-7e8f-4925-8526-bbb1e33e3364"} : dispatch 2024-09-11T03:34:36.578 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:36 smithi121 ceph-mon[30729]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "32164dd5-7e8f-4925-8526-bbb1e33e3364"} : dispatch 2024-09-11T03:34:36.578 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:36 smithi121 ceph-mon[30729]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32164dd5-7e8f-4925-8526-bbb1e33e3364"}]': finished 2024-09-11T03:34:36.578 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:36 smithi121 ceph-mon[30729]: osdmap e34: 6 total, 5 up, 6 in 2024-09-11T03:34:36.578 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:36 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:36.578 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.121:0/2131944517' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:34:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:36 smithi084 ceph-mon[27254]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.121:0/2428273413' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "32164dd5-7e8f-4925-8526-bbb1e33e3364"} : dispatch 2024-09-11T03:34:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:36 smithi084 ceph-mon[27254]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "32164dd5-7e8f-4925-8526-bbb1e33e3364"} : dispatch 2024-09-11T03:34:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:36 smithi084 ceph-mon[27254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32164dd5-7e8f-4925-8526-bbb1e33e3364"}]': finished 2024-09-11T03:34:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:36 smithi084 ceph-mon[27254]: osdmap e34: 6 total, 5 up, 6 in 2024-09-11T03:34:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:36 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.121:0/2131944517' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:34:36.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:36 smithi175 ceph-mon[30894]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:36.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.121:0/2428273413' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "32164dd5-7e8f-4925-8526-bbb1e33e3364"} : dispatch 2024-09-11T03:34:36.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:36 smithi175 ceph-mon[30894]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "32164dd5-7e8f-4925-8526-bbb1e33e3364"} : dispatch 2024-09-11T03:34:36.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:36 smithi175 ceph-mon[30894]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32164dd5-7e8f-4925-8526-bbb1e33e3364"}]': finished 2024-09-11T03:34:36.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:36 smithi175 ceph-mon[30894]: osdmap e34: 6 total, 5 up, 6 in 2024-09-11T03:34:36.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:36 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:36.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.121:0/2131944517' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:34:38.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:38 smithi084 ceph-mon[27254]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-11T03:34:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:38 smithi121 ceph-mon[30729]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-11T03:34:38.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:38 smithi175 ceph-mon[30894]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-11T03:34:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:40 smithi121 ceph-mon[30729]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-11T03:34:40.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:40 smithi084 ceph-mon[27254]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-11T03:34:40.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:40 smithi175 ceph-mon[30894]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-11T03:34:42.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:42 smithi084 ceph-mon[27254]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-11T03:34:42.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:42 smithi121 ceph-mon[30729]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-11T03:34:42.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:42 smithi175 ceph-mon[30894]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-11T03:34:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-11T03:34:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:43.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-11T03:34:43.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:43.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-11T03:34:43.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:44.479 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:44 smithi121 ceph-mon[30729]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-11T03:34:44.479 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:44 smithi121 ceph-mon[30729]: Deploying daemon osd.5 on smithi121 2024-09-11T03:34:44.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:44 smithi084 ceph-mon[27254]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-11T03:34:44.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:44 smithi084 ceph-mon[27254]: Deploying daemon osd.5 on smithi121 2024-09-11T03:34:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:44 smithi175 ceph-mon[30894]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-11T03:34:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:44 smithi175 ceph-mon[30894]: Deploying daemon osd.5 on smithi121 2024-09-11T03:34:45.627 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:45 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:34:45.628 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:45 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:34:45.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:45 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:34:45.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:45 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:34:45.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:45 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:34:45.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:45 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:34:46.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:46 smithi121 ceph-mon[30729]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-11T03:34:46.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:46 smithi121 ceph-mon[30729]: osdmap e35: 6 total, 5 up, 6 in 2024-09-11T03:34:46.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:46 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:46.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:46 smithi084 ceph-mon[27254]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-11T03:34:46.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:46 smithi084 ceph-mon[27254]: osdmap e35: 6 total, 5 up, 6 in 2024-09-11T03:34:46.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:46 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:46.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:46 smithi175 ceph-mon[30894]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-11T03:34:46.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:46 smithi175 ceph-mon[30894]: osdmap e35: 6 total, 5 up, 6 in 2024-09-11T03:34:46.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:46 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:48.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:48 smithi121 ceph-mon[30729]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:48.643 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:48 smithi084 ceph-mon[27254]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:48.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:48 smithi175 ceph-mon[30894]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:49.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:49 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:49.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:49 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:49.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:49 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:49.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:49 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:49.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:49 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:49.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:49 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:49.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:49 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:49.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:49 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:49.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:49 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:50.629 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:50 smithi121 ceph-mon[30729]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:50.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:50 smithi084 ceph-mon[27254]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:50.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:50 smithi175 ceph-mon[30894]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:52.822 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:52 smithi121 ceph-mon[30729]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:52.822 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:52 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:52.822 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:52 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:52.822 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:52 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:52.823 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:52 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:52.823 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:52 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:52.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:52 smithi084 ceph-mon[27254]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:52.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:52 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:52.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:52 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:52.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:52 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:52.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:52 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:52.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:52 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:52.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:52 smithi175 ceph-mon[30894]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:52.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:52 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:52.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:52 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:52.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:52 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:52.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:52 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:52.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:52 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:53.128 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 5 on host 'smithi121' 2024-09-11T03:34:53.775 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:53 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:53.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:53 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:53.776 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:53 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:53.802 DEBUG:teuthology.orchestra.run.smithi121:osd.5> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.5.service 2024-09-11T03:34:53.804 INFO:tasks.cephadm:Deploying osd.6 on smithi121 with /dev/vg_nvme/lv_2... 2024-09-11T03:34:53.804 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-11T03:34:53.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:53 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:53.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:53 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:53.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:53 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:53.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:53 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:34:53.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:53 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:53.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:53 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:54.072 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:34:53 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5[40512]: 2024-09-11T03:34:53.964+0000 7f3f13f5c640 -1 osd.5 0 log_to_monitors true 2024-09-11T03:34:54.544 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:34:54.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:54 smithi084 ceph-mon[27254]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:54.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:54 smithi084 ceph-mon[27254]: from='osd.5 [v2:172.21.15.121:6808/918612194,v1:172.21.15.121:6809/918612194]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-11T03:34:54.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:54 smithi084 ceph-mon[27254]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-11T03:34:54.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:54 smithi121 ceph-mon[30729]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:54.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:54 smithi121 ceph-mon[30729]: from='osd.5 [v2:172.21.15.121:6808/918612194,v1:172.21.15.121:6809/918612194]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-11T03:34:54.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:54 smithi121 ceph-mon[30729]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-11T03:34:54.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:54 smithi175 ceph-mon[30894]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:54.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:54 smithi175 ceph-mon[30894]: from='osd.5 [v2:172.21.15.121:6808/918612194,v1:172.21.15.121:6809/918612194]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-11T03:34:54.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:54 smithi175 ceph-mon[30894]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-11T03:34:55.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:55 smithi084 ceph-mon[27254]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-11T03:34:55.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:55 smithi084 ceph-mon[27254]: osdmap e36: 6 total, 5 up, 6 in 2024-09-11T03:34:55.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:55 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:55.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:55 smithi084 ceph-mon[27254]: from='osd.5 [v2:172.21.15.121:6808/918612194,v1:172.21.15.121:6809/918612194]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:34:55.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:55 smithi084 ceph-mon[27254]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:34:55.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:55 smithi121 ceph-mon[30729]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-11T03:34:55.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:55 smithi121 ceph-mon[30729]: osdmap e36: 6 total, 5 up, 6 in 2024-09-11T03:34:55.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:55 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:55.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:55 smithi121 ceph-mon[30729]: from='osd.5 [v2:172.21.15.121:6808/918612194,v1:172.21.15.121:6809/918612194]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:34:55.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:55 smithi121 ceph-mon[30729]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:34:55.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:55 smithi175 ceph-mon[30894]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-11T03:34:55.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:55 smithi175 ceph-mon[30894]: osdmap e36: 6 total, 5 up, 6 in 2024-09-11T03:34:55.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:55 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:55.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:55 smithi175 ceph-mon[30894]: from='osd.5 [v2:172.21.15.121:6808/918612194,v1:172.21.15.121:6809/918612194]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:34:55.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:55 smithi175 ceph-mon[30894]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:34:56.395 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:34:56 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5[40512]: 2024-09-11T03:34:56.033+0000 7f3f0fedf640 -1 osd.5 0 waiting for initial osdmap 2024-09-11T03:34:56.396 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:34:56 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5[40512]: 2024-09-11T03:34:56.036+0000 7f3f07cfe640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T03:34:56.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:56 smithi121 ceph-mon[30729]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:56.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:56 smithi121 ceph-mon[30729]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-09-11T03:34:56.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:56 smithi121 ceph-mon[30729]: osdmap e37: 6 total, 5 up, 6 in 2024-09-11T03:34:56.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:56 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:56.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:56 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:56.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:56 smithi121 ceph-mon[30729]: from='osd.5 ' entity='osd.5' 2024-09-11T03:34:56.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:56 smithi084 ceph-mon[27254]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:56.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:56 smithi084 ceph-mon[27254]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-09-11T03:34:56.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:56 smithi084 ceph-mon[27254]: osdmap e37: 6 total, 5 up, 6 in 2024-09-11T03:34:56.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:56.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:56.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:56 smithi084 ceph-mon[27254]: from='osd.5 ' entity='osd.5' 2024-09-11T03:34:56.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:56 smithi175 ceph-mon[30894]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-11T03:34:56.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:56 smithi175 ceph-mon[30894]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-09-11T03:34:56.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:56 smithi175 ceph-mon[30894]: osdmap e37: 6 total, 5 up, 6 in 2024-09-11T03:34:56.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:56 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:56.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:56 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:56.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:56 smithi175 ceph-mon[30894]: from='osd.5 ' entity='osd.5' 2024-09-11T03:34:57.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:57 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:34:57.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:57 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:34:57.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:57 smithi084 ceph-mon[27254]: osd.5 [v2:172.21.15.121:6808/918612194,v1:172.21.15.121:6809/918612194] boot 2024-09-11T03:34:57.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:57 smithi084 ceph-mon[27254]: osdmap e38: 6 total, 6 up, 6 in 2024-09-11T03:34:57.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:57.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:57 smithi084 ceph-mon[27254]: osdmap e39: 6 total, 6 up, 6 in 2024-09-11T03:34:57.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:57.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:57.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:57.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:57.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:57.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:57.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:57.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:57.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:57 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:34:57.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:57 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:34:57.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:57 smithi121 ceph-mon[30729]: osd.5 [v2:172.21.15.121:6808/918612194,v1:172.21.15.121:6809/918612194] boot 2024-09-11T03:34:57.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:57 smithi121 ceph-mon[30729]: osdmap e38: 6 total, 6 up, 6 in 2024-09-11T03:34:57.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:57 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:57.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:57 smithi121 ceph-mon[30729]: osdmap e39: 6 total, 6 up, 6 in 2024-09-11T03:34:57.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:57 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:57.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:57 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:57.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:57 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:57.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:57 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:57.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:57 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:57.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:57 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:57.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:57 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:57.923 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:57 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:57.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:57 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:34:57.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:57 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:34:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:57 smithi175 ceph-mon[30894]: osd.5 [v2:172.21.15.121:6808/918612194,v1:172.21.15.121:6809/918612194] boot 2024-09-11T03:34:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:57 smithi175 ceph-mon[30894]: osdmap e38: 6 total, 6 up, 6 in 2024-09-11T03:34:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:57 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-11T03:34:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:57 smithi175 ceph-mon[30894]: osdmap e39: 6 total, 6 up, 6 in 2024-09-11T03:34:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:57 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:57 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:57 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:57 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-11T03:34:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:57 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:57 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:57 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:34:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:57 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:34:58.662 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:34:58.689 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi121:vg_nvme/lv_2 2024-09-11T03:34:58.712 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:58 smithi121 ceph-mon[30729]: pgmap v133: 1 pgs: 1 unknown; 577 KiB data, 533 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:34:58.712 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:58 smithi121 ceph-mon[30729]: Detected new or changed devices on smithi121 2024-09-11T03:34:58.712 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:58 smithi121 ceph-mon[30729]: Adjusting osd_memory_target on smithi121 to 8543M 2024-09-11T03:34:58.713 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:58 smithi121 ceph-mon[30729]: osdmap e40: 6 total, 6 up, 6 in 2024-09-11T03:34:58.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:58 smithi084 ceph-mon[27254]: pgmap v133: 1 pgs: 1 unknown; 577 KiB data, 533 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:34:58.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:58 smithi084 ceph-mon[27254]: Detected new or changed devices on smithi121 2024-09-11T03:34:58.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:58 smithi084 ceph-mon[27254]: Adjusting osd_memory_target on smithi121 to 8543M 2024-09-11T03:34:58.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:58 smithi084 ceph-mon[27254]: osdmap e40: 6 total, 6 up, 6 in 2024-09-11T03:34:58.931 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:34:58.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:58 smithi175 ceph-mon[30894]: pgmap v133: 1 pgs: 1 unknown; 577 KiB data, 533 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:34:58.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:58 smithi175 ceph-mon[30894]: Detected new or changed devices on smithi121 2024-09-11T03:34:58.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:58 smithi175 ceph-mon[30894]: Adjusting osd_memory_target on smithi121 to 8543M 2024-09-11T03:34:58.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:58 smithi175 ceph-mon[30894]: osdmap e40: 6 total, 6 up, 6 in 2024-09-11T03:34:59.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:59 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:34:59.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:59 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:34:59.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:34:59 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:59.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:59 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:34:59.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:59 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:34:59.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:34:59 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:34:59.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:59 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:34:59.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:59 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:34:59.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:34:59 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:00.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:00 smithi084 ceph-mon[27254]: pgmap v136: 1 pgs: 1 unknown; 577 KiB data, 533 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:00.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:00 smithi084 ceph-mon[27254]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:35:00.919 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:00 smithi121 ceph-mon[30729]: pgmap v136: 1 pgs: 1 unknown; 577 KiB data, 533 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:00.919 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:00 smithi121 ceph-mon[30729]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:35:00.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:00 smithi175 ceph-mon[30894]: pgmap v136: 1 pgs: 1 unknown; 577 KiB data, 533 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:00.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:00 smithi175 ceph-mon[30894]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:35:02.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:02 smithi084 ceph-mon[27254]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 533 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:02.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:02 smithi084 ceph-mon[27254]: 1.0 scrub starts 2024-09-11T03:35:02.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:02 smithi084 ceph-mon[27254]: 1.0 scrub ok 2024-09-11T03:35:02.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:02 smithi084 ceph-mon[27254]: from='client.? 172.21.15.121:0/3493641271' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3f107a88-803f-4cd8-804a-241310c282ea"} : dispatch 2024-09-11T03:35:02.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:02 smithi084 ceph-mon[27254]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3f107a88-803f-4cd8-804a-241310c282ea"} : dispatch 2024-09-11T03:35:02.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:02 smithi084 ceph-mon[27254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f107a88-803f-4cd8-804a-241310c282ea"}]': finished 2024-09-11T03:35:02.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:02 smithi084 ceph-mon[27254]: osdmap e41: 7 total, 6 up, 7 in 2024-09-11T03:35:02.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:02 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:02.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:02 smithi084 ceph-mon[27254]: from='client.? 172.21.15.121:0/1165923341' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:35:02.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:02 smithi121 ceph-mon[30729]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 533 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:02.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:02 smithi121 ceph-mon[30729]: 1.0 scrub starts 2024-09-11T03:35:02.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:02 smithi121 ceph-mon[30729]: 1.0 scrub ok 2024-09-11T03:35:02.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:02 smithi121 ceph-mon[30729]: from='client.? 172.21.15.121:0/3493641271' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3f107a88-803f-4cd8-804a-241310c282ea"} : dispatch 2024-09-11T03:35:02.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:02 smithi121 ceph-mon[30729]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3f107a88-803f-4cd8-804a-241310c282ea"} : dispatch 2024-09-11T03:35:02.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:02 smithi121 ceph-mon[30729]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f107a88-803f-4cd8-804a-241310c282ea"}]': finished 2024-09-11T03:35:02.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:02 smithi121 ceph-mon[30729]: osdmap e41: 7 total, 6 up, 7 in 2024-09-11T03:35:02.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:02 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:02.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:02 smithi121 ceph-mon[30729]: from='client.? 172.21.15.121:0/1165923341' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:35:02.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:02 smithi175 ceph-mon[30894]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 533 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:02.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:02 smithi175 ceph-mon[30894]: 1.0 scrub starts 2024-09-11T03:35:02.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:02 smithi175 ceph-mon[30894]: 1.0 scrub ok 2024-09-11T03:35:02.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:02 smithi175 ceph-mon[30894]: from='client.? 172.21.15.121:0/3493641271' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3f107a88-803f-4cd8-804a-241310c282ea"} : dispatch 2024-09-11T03:35:02.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:02 smithi175 ceph-mon[30894]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3f107a88-803f-4cd8-804a-241310c282ea"} : dispatch 2024-09-11T03:35:02.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:02 smithi175 ceph-mon[30894]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f107a88-803f-4cd8-804a-241310c282ea"}]': finished 2024-09-11T03:35:02.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:02 smithi175 ceph-mon[30894]: osdmap e41: 7 total, 6 up, 7 in 2024-09-11T03:35:02.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:02 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:02.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:02 smithi175 ceph-mon[30894]: from='client.? 172.21.15.121:0/1165923341' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:35:04.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:04 smithi084 ceph-mon[27254]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-11T03:35:04.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:04 smithi121 ceph-mon[30729]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-11T03:35:04.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:04 smithi175 ceph-mon[30894]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-11T03:35:05.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:05 smithi121 ceph-mon[30729]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-11T03:35:05.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:05 smithi175 ceph-mon[30894]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-11T03:35:06.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:05 smithi084 ceph-mon[27254]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-11T03:35:07.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:07 smithi121 ceph-mon[30729]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T03:35:07.946 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:07 smithi084 ceph-mon[27254]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T03:35:07.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:07 smithi175 ceph-mon[30894]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T03:35:09.820 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:09 smithi121 ceph-mon[30729]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-11T03:35:09.820 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:09 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-11T03:35:09.820 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:09 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:09.820 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:09 smithi121 ceph-mon[30729]: Deploying daemon osd.6 on smithi121 2024-09-11T03:35:09.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:09 smithi175 ceph-mon[30894]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-11T03:35:09.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:09 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-11T03:35:09.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:09 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:09.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:09 smithi175 ceph-mon[30894]: Deploying daemon osd.6 on smithi121 2024-09-11T03:35:10.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:09 smithi084 ceph-mon[27254]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-11T03:35:10.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:09 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-11T03:35:10.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:09 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:10.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:09 smithi084 ceph-mon[27254]: Deploying daemon osd.6 on smithi121 2024-09-11T03:35:11.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:11 smithi121 ceph-mon[30729]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-11T03:35:11.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:11 smithi175 ceph-mon[30894]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-11T03:35:12.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:11 smithi084 ceph-mon[27254]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-11T03:35:13.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:13 smithi121 ceph-mon[30729]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:13.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:13 smithi175 ceph-mon[30894]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:14.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:13 smithi084 ceph-mon[27254]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:14.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:14 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:35:14.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:14 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:35:14.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:14 smithi121 ceph-mon[30729]: osdmap e42: 7 total, 6 up, 7 in 2024-09-11T03:35:14.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:14 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:14.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:14 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:35:14.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:14 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:35:14.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:14 smithi175 ceph-mon[30894]: osdmap e42: 7 total, 6 up, 7 in 2024-09-11T03:35:14.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:14 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:15.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:14 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:35:15.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:14 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:35:15.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:14 smithi084 ceph-mon[27254]: osdmap e42: 7 total, 6 up, 7 in 2024-09-11T03:35:15.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:14 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:15.874 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:15 smithi121 ceph-mon[30729]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:15.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:35:15.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:15.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:15.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:15 smithi175 ceph-mon[30894]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:15.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:35:15.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:15.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:16.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:15 smithi084 ceph-mon[27254]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:16.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:35:16.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:16.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:17.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:17 smithi121 ceph-mon[30729]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:17.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:17 smithi175 ceph-mon[30894]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:17 smithi084 ceph-mon[27254]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:18.252 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 6 on host 'smithi121' 2024-09-11T03:35:18.843 DEBUG:teuthology.orchestra.run.smithi121:osd.6> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.6.service 2024-09-11T03:35:18.885 INFO:tasks.cephadm:Deploying osd.7 on smithi121 with /dev/vg_nvme/lv_1... 2024-09-11T03:35:18.885 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-11T03:35:19.198 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:35:19.591 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:19.591 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:19.591 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:19.591 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:19.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:19.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:19.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:19.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:19.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:19.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:19.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:19.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:20.189 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:35:19 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:35:19.928+0000 7f048934d640 -1 osd.6 0 log_to_monitors true 2024-09-11T03:35:20.505 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:20 smithi121 ceph-mon[30729]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:20.506 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:20 smithi121 ceph-mon[30729]: from='osd.6 [v2:172.21.15.121:6816/3384813363,v1:172.21.15.121:6817/3384813363]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-11T03:35:20.506 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:20 smithi121 ceph-mon[30729]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-11T03:35:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:20 smithi084 ceph-mon[27254]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:20 smithi084 ceph-mon[27254]: from='osd.6 [v2:172.21.15.121:6816/3384813363,v1:172.21.15.121:6817/3384813363]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-11T03:35:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:20 smithi084 ceph-mon[27254]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-11T03:35:20.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:20 smithi175 ceph-mon[30894]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:20.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:20 smithi175 ceph-mon[30894]: from='osd.6 [v2:172.21.15.121:6816/3384813363,v1:172.21.15.121:6817/3384813363]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-11T03:35:20.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:20 smithi175 ceph-mon[30894]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-11T03:35:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:21 smithi121 ceph-mon[30729]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-11T03:35:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:21 smithi121 ceph-mon[30729]: osdmap e43: 7 total, 6 up, 7 in 2024-09-11T03:35:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:21 smithi121 ceph-mon[30729]: from='osd.6 [v2:172.21.15.121:6816/3384813363,v1:172.21.15.121:6817/3384813363]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:35:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:21 smithi121 ceph-mon[30729]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:35:21.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:21 smithi084 ceph-mon[27254]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-11T03:35:21.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:21 smithi084 ceph-mon[27254]: osdmap e43: 7 total, 6 up, 7 in 2024-09-11T03:35:21.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:21.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:21 smithi084 ceph-mon[27254]: from='osd.6 [v2:172.21.15.121:6816/3384813363,v1:172.21.15.121:6817/3384813363]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:35:21.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:21 smithi084 ceph-mon[27254]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:35:21.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:21 smithi175 ceph-mon[30894]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-11T03:35:21.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:21 smithi175 ceph-mon[30894]: osdmap e43: 7 total, 6 up, 7 in 2024-09-11T03:35:21.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:21.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:21 smithi175 ceph-mon[30894]: from='osd.6 [v2:172.21.15.121:6816/3384813363,v1:172.21.15.121:6817/3384813363]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:35:21.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:21 smithi175 ceph-mon[30894]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:35:21.921 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:35:21 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:35:21.704+0000 7f04852d0640 -1 osd.6 0 waiting for initial osdmap 2024-09-11T03:35:21.922 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:35:21 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:35:21.710+0000 7f047d0ef640 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T03:35:22.350 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:22 smithi121 ceph-mon[30729]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:22.350 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:22 smithi121 ceph-mon[30729]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-09-11T03:35:22.350 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:22 smithi121 ceph-mon[30729]: osdmap e44: 7 total, 6 up, 7 in 2024-09-11T03:35:22.350 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:22 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:22.350 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:22 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:22.351 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:22 smithi121 ceph-mon[30729]: from='osd.6 ' entity='osd.6' 2024-09-11T03:35:22.351 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:22 smithi121 ceph-mon[30729]: osd.6 [v2:172.21.15.121:6816/3384813363,v1:172.21.15.121:6817/3384813363] boot 2024-09-11T03:35:22.351 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:22 smithi121 ceph-mon[30729]: osdmap e45: 7 total, 7 up, 7 in 2024-09-11T03:35:22.351 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:22 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:22.353 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-11T03:35:22.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi121:vg_nvme/lv_1 2024-09-11T03:35:22.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:22 smithi084 ceph-mon[27254]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:22.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:22 smithi084 ceph-mon[27254]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-09-11T03:35:22.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:22 smithi084 ceph-mon[27254]: osdmap e44: 7 total, 6 up, 7 in 2024-09-11T03:35:22.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:22 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:22.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:22 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:22.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:22 smithi084 ceph-mon[27254]: from='osd.6 ' entity='osd.6' 2024-09-11T03:35:22.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:22 smithi084 ceph-mon[27254]: osd.6 [v2:172.21.15.121:6816/3384813363,v1:172.21.15.121:6817/3384813363] boot 2024-09-11T03:35:22.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:22 smithi084 ceph-mon[27254]: osdmap e45: 7 total, 7 up, 7 in 2024-09-11T03:35:22.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:22 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:22.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:22 smithi175 ceph-mon[30894]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-11T03:35:22.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:22 smithi175 ceph-mon[30894]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-09-11T03:35:22.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:22 smithi175 ceph-mon[30894]: osdmap e44: 7 total, 6 up, 7 in 2024-09-11T03:35:22.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:22 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:22.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:22 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:22.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:22 smithi175 ceph-mon[30894]: from='osd.6 ' entity='osd.6' 2024-09-11T03:35:22.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:22 smithi175 ceph-mon[30894]: osd.6 [v2:172.21.15.121:6816/3384813363,v1:172.21.15.121:6817/3384813363] boot 2024-09-11T03:35:22.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:22 smithi175 ceph-mon[30894]: osdmap e45: 7 total, 7 up, 7 in 2024-09-11T03:35:22.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:22 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-11T03:35:23.269 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:35:23.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:23 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:35:23.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:23 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:35:23.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:23 smithi121 ceph-mon[30729]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:23.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:23 smithi121 ceph-mon[30729]: osdmap e46: 7 total, 7 up, 7 in 2024-09-11T03:35:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:23 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:35:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:23 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:35:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:23 smithi084 ceph-mon[27254]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:23 smithi084 ceph-mon[27254]: osdmap e46: 7 total, 7 up, 7 in 2024-09-11T03:35:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:23 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:35:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:23 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:35:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:23 smithi175 ceph-mon[30894]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:23 smithi175 ceph-mon[30894]: osdmap e46: 7 total, 7 up, 7 in 2024-09-11T03:35:25.058 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:24 smithi121 ceph-mon[30729]: osdmap e47: 7 total, 7 up, 7 in 2024-09-11T03:35:25.058 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:24 smithi121 ceph-mon[30729]: from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:35:25.058 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:35:25.058 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:35:25.059 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:25.059 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:24 smithi121 ceph-mon[30729]: Detected new or changed devices on smithi121 2024-09-11T03:35:25.059 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:25.059 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:25.059 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:25.059 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:25.059 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:25.059 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:24 smithi121 ceph-mon[30729]: Adjusting osd_memory_target on smithi121 to 5695M 2024-09-11T03:35:25.059 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:25.059 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:25.059 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:35:25.059 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:25.060 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:35:25.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:24 smithi084 ceph-mon[27254]: osdmap e47: 7 total, 7 up, 7 in 2024-09-11T03:35:25.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:24 smithi084 ceph-mon[27254]: from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:35:25.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:35:25.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:35:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:24 smithi084 ceph-mon[27254]: Detected new or changed devices on smithi121 2024-09-11T03:35:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:24 smithi084 ceph-mon[27254]: Adjusting osd_memory_target on smithi121 to 5695M 2024-09-11T03:35:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:35:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:35:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:24 smithi175 ceph-mon[30894]: osdmap e47: 7 total, 7 up, 7 in 2024-09-11T03:35:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:24 smithi175 ceph-mon[30894]: from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:35:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:35:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:35:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:24 smithi175 ceph-mon[30894]: Detected new or changed devices on smithi121 2024-09-11T03:35:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:24 smithi175 ceph-mon[30894]: Adjusting osd_memory_target on smithi121 to 5695M 2024-09-11T03:35:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:35:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:35:25.979 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:25 smithi121 ceph-mon[30729]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:25.979 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:25.979 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:35:25.979 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:26.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:25 smithi084 ceph-mon[27254]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:26.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:26.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:35:26.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:25 smithi175 ceph-mon[30894]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:35:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.121:0/2507737129' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dfee7f95-1fb9-4493-9907-cb10d1837bdb"} : dispatch 2024-09-11T03:35:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:26 smithi084 ceph-mon[27254]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dfee7f95-1fb9-4493-9907-cb10d1837bdb"} : dispatch 2024-09-11T03:35:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:26 smithi084 ceph-mon[27254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dfee7f95-1fb9-4493-9907-cb10d1837bdb"}]': finished 2024-09-11T03:35:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:26 smithi084 ceph-mon[27254]: osdmap e48: 8 total, 7 up, 8 in 2024-09-11T03:35:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.121:0/2507737129' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dfee7f95-1fb9-4493-9907-cb10d1837bdb"} : dispatch 2024-09-11T03:35:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:26 smithi121 ceph-mon[30729]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dfee7f95-1fb9-4493-9907-cb10d1837bdb"} : dispatch 2024-09-11T03:35:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:26 smithi121 ceph-mon[30729]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dfee7f95-1fb9-4493-9907-cb10d1837bdb"}]': finished 2024-09-11T03:35:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:26 smithi121 ceph-mon[30729]: osdmap e48: 8 total, 7 up, 8 in 2024-09-11T03:35:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.121:0/2507737129' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dfee7f95-1fb9-4493-9907-cb10d1837bdb"} : dispatch 2024-09-11T03:35:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:26 smithi175 ceph-mon[30894]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dfee7f95-1fb9-4493-9907-cb10d1837bdb"} : dispatch 2024-09-11T03:35:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:26 smithi175 ceph-mon[30894]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dfee7f95-1fb9-4493-9907-cb10d1837bdb"}]': finished 2024-09-11T03:35:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:26 smithi175 ceph-mon[30894]: osdmap e48: 8 total, 7 up, 8 in 2024-09-11T03:35:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:27 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:35:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:27 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:35:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:27 smithi084 ceph-mon[27254]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:27 smithi084 ceph-mon[27254]: from='client.? 172.21.15.121:0/4250539910' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:35:28.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:27 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:35:28.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:27 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:35:28.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:27 smithi121 ceph-mon[30729]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:28.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:27 smithi121 ceph-mon[30729]: from='client.? 172.21.15.121:0/4250539910' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:35:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:27 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:35:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:27 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:35:28.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:27 smithi175 ceph-mon[30894]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:28.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:27 smithi175 ceph-mon[30894]: from='client.? 172.21.15.121:0/4250539910' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:35:30.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:29 smithi084 ceph-mon[27254]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:30.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:29 smithi121 ceph-mon[30729]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:29 smithi175 ceph-mon[30894]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:31.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:31 smithi121 ceph-mon[30729]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:31.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:31 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:35:31.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:31 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:35:32.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:31 smithi084 ceph-mon[27254]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:32.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:31 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:35:32.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:31 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:35:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:31 smithi175 ceph-mon[30894]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:31 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:35:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:31 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:35:33.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:32 smithi084 ceph-mon[27254]: osdmap e49: 8 total, 7 up, 8 in 2024-09-11T03:35:33.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:32 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:33.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:32 smithi121 ceph-mon[30729]: osdmap e49: 8 total, 7 up, 8 in 2024-09-11T03:35:33.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:32 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:32 smithi175 ceph-mon[30894]: osdmap e49: 8 total, 7 up, 8 in 2024-09-11T03:35:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:32 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:34.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:33 smithi084 ceph-mon[27254]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:34.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:33 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-11T03:35:34.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:33 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:34.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:33 smithi121 ceph-mon[30729]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:34.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:33 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-11T03:35:34.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:33 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:33 smithi175 ceph-mon[30894]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:33 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-11T03:35:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:33 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:35.104 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:34 smithi121 ceph-mon[30729]: Deploying daemon osd.7 on smithi121 2024-09-11T03:35:35.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:34 smithi084 ceph-mon[27254]: Deploying daemon osd.7 on smithi121 2024-09-11T03:35:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:34 smithi175 ceph-mon[30894]: Deploying daemon osd.7 on smithi121 2024-09-11T03:35:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:35 smithi084 ceph-mon[27254]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:35 smithi121 ceph-mon[30729]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:36.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:35 smithi175 ceph-mon[30894]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:38.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:37 smithi084 ceph-mon[27254]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:38.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:37 smithi121 ceph-mon[30729]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:38.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:37 smithi175 ceph-mon[30894]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:40.078 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:39 smithi121 ceph-mon[30729]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:40.078 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:39 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:35:40.078 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:39 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:40.078 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:39 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:40.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:39 smithi084 ceph-mon[27254]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:40.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:39 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:35:40.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:39 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:40.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:39 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:40.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:39 smithi175 ceph-mon[30894]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:40.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:39 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:35:40.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:39 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:40.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:39 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:41.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:41 smithi121 ceph-mon[30729]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:42.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:41 smithi084 ceph-mon[27254]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:41 smithi175 ceph-mon[30894]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:43.000 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 7 on host 'smithi121' 2024-09-11T03:35:43.618 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:43.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:43.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:43.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:35:43.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:43.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:35:43.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:43.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:43 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:43.643 DEBUG:teuthology.orchestra.run.smithi121:osd.7> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.7.service 2024-09-11T03:35:43.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:43.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:43.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:43.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:35:43.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:43.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:35:43.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:43.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:43 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:43.649 INFO:tasks.cephadm:Deploying osd.8 on smithi175 with /dev/vg_nvme/lv_4... 2024-09-11T03:35:43.649 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-11T03:35:43.676 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:43.676 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:43.676 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:43.676 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:35:43.676 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:43.676 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:35:43.677 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:43.677 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:43 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:43.896 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:35:44.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:44 smithi084 ceph-mon[27254]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:44.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:44 smithi084 ceph-mon[27254]: from='osd.7 [v2:172.21.15.121:6824/449222054,v1:172.21.15.121:6825/449222054]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-11T03:35:44.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:44 smithi084 ceph-mon[27254]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-11T03:35:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:44 smithi121 ceph-mon[30729]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:44 smithi121 ceph-mon[30729]: from='osd.7 [v2:172.21.15.121:6824/449222054,v1:172.21.15.121:6825/449222054]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-11T03:35:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:44 smithi121 ceph-mon[30729]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-11T03:35:44.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:35:44 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:35:44.237+0000 7f16723f5640 -1 osd.7 0 log_to_monitors true 2024-09-11T03:35:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:44 smithi175 ceph-mon[30894]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:44 smithi175 ceph-mon[30894]: from='osd.7 [v2:172.21.15.121:6824/449222054,v1:172.21.15.121:6825/449222054]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-11T03:35:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:44 smithi175 ceph-mon[30894]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-11T03:35:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:45 smithi121 ceph-mon[30729]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-11T03:35:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:45 smithi121 ceph-mon[30729]: osdmap e50: 8 total, 7 up, 8 in 2024-09-11T03:35:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:45 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:45 smithi121 ceph-mon[30729]: from='osd.7 [v2:172.21.15.121:6824/449222054,v1:172.21.15.121:6825/449222054]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:35:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:45 smithi121 ceph-mon[30729]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:35:45.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:45 smithi084 ceph-mon[27254]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-11T03:35:45.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:45 smithi084 ceph-mon[27254]: osdmap e50: 8 total, 7 up, 8 in 2024-09-11T03:35:45.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:45 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:45.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:45 smithi084 ceph-mon[27254]: from='osd.7 [v2:172.21.15.121:6824/449222054,v1:172.21.15.121:6825/449222054]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:35:45.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:45 smithi084 ceph-mon[27254]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:35:45.670 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:45 smithi175 ceph-mon[30894]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-11T03:35:45.670 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:45 smithi175 ceph-mon[30894]: osdmap e50: 8 total, 7 up, 8 in 2024-09-11T03:35:45.671 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:45 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:45.671 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:45 smithi175 ceph-mon[30894]: from='osd.7 [v2:172.21.15.121:6824/449222054,v1:172.21.15.121:6825/449222054]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:35:45.671 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:45 smithi175 ceph-mon[30894]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]} : dispatch 2024-09-11T03:35:46.172 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:35:45 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:35:45.795+0000 7f166e378640 -1 osd.7 0 waiting for initial osdmap 2024-09-11T03:35:46.172 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:35:45 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:35:45.803+0000 7f1666197640 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T03:35:46.334 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:35:46.358 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:46 smithi175 ceph-mon[30894]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:46.358 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:46 smithi175 ceph-mon[30894]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-09-11T03:35:46.358 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:46 smithi175 ceph-mon[30894]: osdmap e51: 8 total, 7 up, 8 in 2024-09-11T03:35:46.358 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:46 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:46.358 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:46 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:46.358 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:46 smithi175 ceph-mon[30894]: from='osd.7 ' entity='osd.7' 2024-09-11T03:35:46.360 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi175:vg_nvme/lv_4 2024-09-11T03:35:46.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:46 smithi121 ceph-mon[30729]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:46.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:46 smithi121 ceph-mon[30729]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-09-11T03:35:46.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:46 smithi121 ceph-mon[30729]: osdmap e51: 8 total, 7 up, 8 in 2024-09-11T03:35:46.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:46 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:46.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:46 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:46.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:46 smithi121 ceph-mon[30729]: from='osd.7 ' entity='osd.7' 2024-09-11T03:35:46.642 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:35:46.643 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:46 smithi084 ceph-mon[27254]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-11T03:35:46.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:46 smithi084 ceph-mon[27254]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-09-11T03:35:46.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:46 smithi084 ceph-mon[27254]: osdmap e51: 8 total, 7 up, 8 in 2024-09-11T03:35:46.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:46 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:46.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:46 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:46.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:46 smithi084 ceph-mon[27254]: from='osd.7 ' entity='osd.7' 2024-09-11T03:35:47.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:35:47.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:35:47.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: osd.7 [v2:172.21.15.121:6824/449222054,v1:172.21.15.121:6825/449222054] boot 2024-09-11T03:35:47.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: osdmap e52: 8 total, 8 up, 8 in 2024-09-11T03:35:47.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:47.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: Detected new or changed devices on smithi121 2024-09-11T03:35:47.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:47.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:47.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:47.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:47.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:47.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:47.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: Adjusting osd_memory_target on smithi121 to 4271M 2024-09-11T03:35:47.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:47.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:47.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:35:47.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:47.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: osdmap e53: 8 total, 8 up, 8 in 2024-09-11T03:35:47.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:35:47.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:35:47.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:35:47.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:35:47.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:47 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:47.653 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:35:47.653 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:35:47.653 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: osd.7 [v2:172.21.15.121:6824/449222054,v1:172.21.15.121:6825/449222054] boot 2024-09-11T03:35:47.653 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: osdmap e52: 8 total, 8 up, 8 in 2024-09-11T03:35:47.654 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:47.654 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: Detected new or changed devices on smithi121 2024-09-11T03:35:47.654 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:47.654 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:47.654 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:47.654 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:47.655 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:47.655 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:47.655 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: Adjusting osd_memory_target on smithi121 to 4271M 2024-09-11T03:35:47.655 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:47.655 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:47.655 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:35:47.655 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:47.655 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: osdmap e53: 8 total, 8 up, 8 in 2024-09-11T03:35:47.656 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:35:47.656 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:35:47.656 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:35:47.656 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:35:47.656 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:47 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:47.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:35:47.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:35:47.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: osd.7 [v2:172.21.15.121:6824/449222054,v1:172.21.15.121:6825/449222054] boot 2024-09-11T03:35:47.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: osdmap e52: 8 total, 8 up, 8 in 2024-09-11T03:35:47.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-11T03:35:47.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: Detected new or changed devices on smithi121 2024-09-11T03:35:47.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:47.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:47.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:47.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:47.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:47.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-11T03:35:47.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: Adjusting osd_memory_target on smithi121 to 4271M 2024-09-11T03:35:47.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:47.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:47.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:35:47.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:35:47.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: osdmap e53: 8 total, 8 up, 8 in 2024-09-11T03:35:47.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:35:47.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:35:47.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:35:47.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:35:47.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:47 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:48.853 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:48 smithi175 ceph-mon[30894]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:48.853 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:48 smithi175 ceph-mon[30894]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:35:48.853 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:48 smithi175 ceph-mon[30894]: osdmap e54: 8 total, 8 up, 8 in 2024-09-11T03:35:48.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:48 smithi084 ceph-mon[27254]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:48.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:48 smithi084 ceph-mon[27254]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:35:48.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:48 smithi084 ceph-mon[27254]: osdmap e54: 8 total, 8 up, 8 in 2024-09-11T03:35:48.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:48 smithi121 ceph-mon[30729]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:48.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:48 smithi121 ceph-mon[30729]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:35:48.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:48 smithi121 ceph-mon[30729]: osdmap e54: 8 total, 8 up, 8 in 2024-09-11T03:35:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.175:0/3067294509' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4"} : dispatch 2024-09-11T03:35:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:49 smithi084 ceph-mon[27254]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4"} : dispatch 2024-09-11T03:35:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:49 smithi084 ceph-mon[27254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4"}]': finished 2024-09-11T03:35:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:49 smithi084 ceph-mon[27254]: osdmap e55: 9 total, 8 up, 9 in 2024-09-11T03:35:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:49 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:35:49.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.175:0/3067294509' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4"} : dispatch 2024-09-11T03:35:49.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:49 smithi121 ceph-mon[30729]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4"} : dispatch 2024-09-11T03:35:49.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:49 smithi121 ceph-mon[30729]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4"}]': finished 2024-09-11T03:35:49.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:49 smithi121 ceph-mon[30729]: osdmap e55: 9 total, 8 up, 9 in 2024-09-11T03:35:49.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:49 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:35:49.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.175:0/3067294509' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4"} : dispatch 2024-09-11T03:35:49.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:49 smithi175 ceph-mon[30894]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4"} : dispatch 2024-09-11T03:35:49.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:49 smithi175 ceph-mon[30894]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4"}]': finished 2024-09-11T03:35:49.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:49 smithi175 ceph-mon[30894]: osdmap e55: 9 total, 8 up, 9 in 2024-09-11T03:35:49.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:49 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:35:50.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:50 smithi084 ceph-mon[27254]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:50.897 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:50 smithi084 ceph-mon[27254]: from='client.? 172.21.15.175:0/3954127991' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:35:50.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:50 smithi121 ceph-mon[30729]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:50.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:50 smithi121 ceph-mon[30729]: from='client.? 172.21.15.175:0/3954127991' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:35:50.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:50 smithi175 ceph-mon[30894]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:50.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:50 smithi175 ceph-mon[30894]: from='client.? 172.21.15.175:0/3954127991' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:35:52.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:52 smithi121 ceph-mon[30729]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:52.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:52 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:35:52.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:52 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:35:52.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:52 smithi121 ceph-mon[30729]: osdmap e56: 9 total, 8 up, 9 in 2024-09-11T03:35:52.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:52 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:35:52.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:52 smithi175 ceph-mon[30894]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:52.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:52 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:35:52.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:52 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:35:52.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:52 smithi175 ceph-mon[30894]: osdmap e56: 9 total, 8 up, 9 in 2024-09-11T03:35:52.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:52 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:35:53.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:52 smithi084 ceph-mon[27254]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:53.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:52 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:35:53.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:52 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:35:53.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:52 smithi084 ceph-mon[27254]: osdmap e56: 9 total, 8 up, 9 in 2024-09-11T03:35:53.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:52 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:35:53.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:53 smithi175 ceph-mon[30894]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:53 smithi084 ceph-mon[27254]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:54.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:53 smithi121 ceph-mon[30729]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:55.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:55 smithi175 ceph-mon[30894]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:56.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:55 smithi084 ceph-mon[27254]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:56.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:55 smithi121 ceph-mon[30729]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:57.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:57 smithi175 ceph-mon[30894]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:57.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:57 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-11T03:35:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:57 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:57 smithi175 ceph-mon[30894]: Deploying daemon osd.8 on smithi175 2024-09-11T03:35:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:57 smithi084 ceph-mon[27254]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-11T03:35:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:57 smithi084 ceph-mon[27254]: Deploying daemon osd.8 on smithi175 2024-09-11T03:35:58.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:57 smithi121 ceph-mon[30729]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:35:58.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:57 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-11T03:35:58.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:57 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:35:58.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:57 smithi121 ceph-mon[30729]: Deploying daemon osd.8 on smithi175 2024-09-11T03:35:59.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:35:59 smithi175 ceph-mon[30894]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:35:59 smithi084 ceph-mon[27254]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:00.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:35:59 smithi121 ceph-mon[30729]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:01.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:01 smithi175 ceph-mon[30894]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:02.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:01 smithi084 ceph-mon[27254]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:02.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:01 smithi121 ceph-mon[30729]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:03.940 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:03 smithi175 ceph-mon[30894]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:03.941 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:03.941 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:03.941 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:03 smithi084 ceph-mon[27254]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:03 smithi121 ceph-mon[30729]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:06.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:06 smithi084 ceph-mon[27254]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:06.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:06.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:06.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:06.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:06.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:06.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:06 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:36:06.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:06 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:36:06.646 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:06 smithi175 ceph-mon[30894]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:06.646 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:06.646 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:06.646 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:06.646 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:06.646 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:06.646 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:06 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:36:06.646 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:06 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:36:06.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:06 smithi121 ceph-mon[30729]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:06.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:06 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:06.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:06 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:06.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:06 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:06.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:06 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:06.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:06 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:06.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:06 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:36:06.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:06 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:36:07.254 INFO:teuthology.orchestra.run.smithi175.stdout:Created osd(s) 8 on host 'smithi175' 2024-09-11T03:36:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:07 smithi175 ceph-mon[30894]: osdmap e57: 9 total, 8 up, 9 in 2024-09-11T03:36:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:07.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:07 smithi084 ceph-mon[27254]: osdmap e57: 9 total, 8 up, 9 in 2024-09-11T03:36:07.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:07.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:07.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:07 smithi121 ceph-mon[30729]: osdmap e57: 9 total, 8 up, 9 in 2024-09-11T03:36:07.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:07.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:07.867 DEBUG:teuthology.orchestra.run.smithi175:osd.8> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.8.service 2024-09-11T03:36:07.870 INFO:tasks.cephadm:Deploying osd.9 on smithi175 with /dev/vg_nvme/lv_3... 2024-09-11T03:36:07.870 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-11T03:36:08.091 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:36:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:36:07.978+0000 7f7816c66640 -1 osd.8 0 log_to_monitors true 2024-09-11T03:36:08.369 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:08 smithi175 ceph-mon[30894]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:08.369 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:08 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:08.369 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:08 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:08.369 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:08 smithi175 ceph-mon[30894]: from='osd.8 [v2:172.21.15.175:6800/2660128384,v1:172.21.15.175:6801/2660128384]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-11T03:36:08.369 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:08 smithi175 ceph-mon[30894]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-11T03:36:08.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:08 smithi121 ceph-mon[30729]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:08.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:08 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:08.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:08 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:08.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:08 smithi121 ceph-mon[30729]: from='osd.8 [v2:172.21.15.175:6800/2660128384,v1:172.21.15.175:6801/2660128384]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-11T03:36:08.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:08 smithi121 ceph-mon[30729]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-11T03:36:08.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:08 smithi084 ceph-mon[27254]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:08.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:08 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:08.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:08 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:08.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:08 smithi084 ceph-mon[27254]: from='osd.8 [v2:172.21.15.175:6800/2660128384,v1:172.21.15.175:6801/2660128384]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-11T03:36:08.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:08 smithi084 ceph-mon[27254]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-11T03:36:09.106 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:36:09.602 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:09 smithi175 ceph-mon[30894]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-11T03:36:09.602 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:09 smithi175 ceph-mon[30894]: from='osd.8 [v2:172.21.15.175:6800/2660128384,v1:172.21.15.175:6801/2660128384]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:36:09.602 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:09 smithi175 ceph-mon[30894]: osdmap e58: 9 total, 8 up, 9 in 2024-09-11T03:36:09.602 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:09 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:09.602 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:09 smithi175 ceph-mon[30894]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:36:09.603 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:09 smithi175 ceph-mon[30894]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-11T03:36:09.603 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:09 smithi175 ceph-mon[30894]: osdmap e59: 9 total, 8 up, 9 in 2024-09-11T03:36:09.603 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:09 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:09.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:09 smithi084 ceph-mon[27254]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-11T03:36:09.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:09 smithi084 ceph-mon[27254]: from='osd.8 [v2:172.21.15.175:6800/2660128384,v1:172.21.15.175:6801/2660128384]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:36:09.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:09 smithi084 ceph-mon[27254]: osdmap e58: 9 total, 8 up, 9 in 2024-09-11T03:36:09.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:09 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:09.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:09 smithi084 ceph-mon[27254]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:36:09.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:09 smithi084 ceph-mon[27254]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-11T03:36:09.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:09 smithi084 ceph-mon[27254]: osdmap e59: 9 total, 8 up, 9 in 2024-09-11T03:36:09.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:09 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:09.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:09 smithi121 ceph-mon[30729]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-11T03:36:09.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:09 smithi121 ceph-mon[30729]: from='osd.8 [v2:172.21.15.175:6800/2660128384,v1:172.21.15.175:6801/2660128384]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:36:09.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:09 smithi121 ceph-mon[30729]: osdmap e58: 9 total, 8 up, 9 in 2024-09-11T03:36:09.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:09 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:09.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:09 smithi121 ceph-mon[30729]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:36:09.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:09 smithi121 ceph-mon[30729]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-11T03:36:09.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:09 smithi121 ceph-mon[30729]: osdmap e59: 9 total, 8 up, 9 in 2024-09-11T03:36:09.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:09 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:09.977 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:36:09 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:36:09.704+0000 7f7812be9640 -1 osd.8 0 waiting for initial osdmap 2024-09-11T03:36:09.977 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:36:09 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:36:09.711+0000 7f780aa08640 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T03:36:10.465 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:10 smithi175 ceph-mon[30894]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:10.465 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:10 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:10.465 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:10 smithi175 ceph-mon[30894]: from='osd.8 ' entity='osd.8' 2024-09-11T03:36:10.465 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:10 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:10.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:10 smithi084 ceph-mon[27254]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:10.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:10 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:10.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:10 smithi084 ceph-mon[27254]: from='osd.8 ' entity='osd.8' 2024-09-11T03:36:10.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:10 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:10 smithi121 ceph-mon[30729]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:10 smithi121 ceph-mon[30729]: from='osd.8 ' entity='osd.8' 2024-09-11T03:36:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:11.368 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:11 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:36:11.368 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:11 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:36:11.368 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:11 smithi175 ceph-mon[30894]: osd.8 [v2:172.21.15.175:6800/2660128384,v1:172.21.15.175:6801/2660128384] boot 2024-09-11T03:36:11.369 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:11 smithi175 ceph-mon[30894]: osdmap e60: 9 total, 9 up, 9 in 2024-09-11T03:36:11.369 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:11 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:11.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:11 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:36:11.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:11 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:36:11.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:11 smithi084 ceph-mon[27254]: osd.8 [v2:172.21.15.175:6800/2660128384,v1:172.21.15.175:6801/2660128384] boot 2024-09-11T03:36:11.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:11 smithi084 ceph-mon[27254]: osdmap e60: 9 total, 9 up, 9 in 2024-09-11T03:36:11.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:11 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:11.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:11 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:36:11.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:11 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:36:11.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:11 smithi121 ceph-mon[30729]: osd.8 [v2:172.21.15.175:6800/2660128384,v1:172.21.15.175:6801/2660128384] boot 2024-09-11T03:36:11.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:11 smithi121 ceph-mon[30729]: osdmap e60: 9 total, 9 up, 9 in 2024-09-11T03:36:11.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:11 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-11T03:36:12.461 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:12 smithi175 ceph-mon[30894]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:12.461 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:12 smithi175 ceph-mon[30894]: osdmap e61: 9 total, 9 up, 9 in 2024-09-11T03:36:12.461 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:12 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:12.461 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:12 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:12.461 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:12 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-11T03:36:12.461 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:12 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:12.461 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:12 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:12.461 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:12 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:12.461 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:12 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:12.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:12 smithi084 ceph-mon[27254]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:12.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:12 smithi084 ceph-mon[27254]: osdmap e61: 9 total, 9 up, 9 in 2024-09-11T03:36:12.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:12 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:12.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:12 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:12.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:12 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-11T03:36:12.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:12 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:12.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:12 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:12.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:12 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:12.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:12 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:12.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:12 smithi121 ceph-mon[30729]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-11T03:36:12.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:12 smithi121 ceph-mon[30729]: osdmap e61: 9 total, 9 up, 9 in 2024-09-11T03:36:12.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:12 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:12.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:12 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:12.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:12 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-11T03:36:12.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:12 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:12.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:12 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:12.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:12 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:12.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:12 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:13.096 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:36:13.124 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi175:vg_nvme/lv_3 2024-09-11T03:36:13.366 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:36:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:13 smithi175 ceph-mon[30894]: Detected new or changed devices on smithi175 2024-09-11T03:36:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:13 smithi175 ceph-mon[30894]: Adjusting osd_memory_target on smithi175 to 21184M 2024-09-11T03:36:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:13 smithi175 ceph-mon[30894]: osdmap e62: 9 total, 9 up, 9 in 2024-09-11T03:36:13.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:13 smithi084 ceph-mon[27254]: Detected new or changed devices on smithi175 2024-09-11T03:36:13.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:13 smithi084 ceph-mon[27254]: Adjusting osd_memory_target on smithi175 to 21184M 2024-09-11T03:36:13.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:13 smithi084 ceph-mon[27254]: osdmap e62: 9 total, 9 up, 9 in 2024-09-11T03:36:13.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:13 smithi121 ceph-mon[30729]: Detected new or changed devices on smithi175 2024-09-11T03:36:13.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:13 smithi121 ceph-mon[30729]: Adjusting osd_memory_target on smithi175 to 21184M 2024-09-11T03:36:13.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:13 smithi121 ceph-mon[30729]: osdmap e62: 9 total, 9 up, 9 in 2024-09-11T03:36:14.591 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:14 smithi175 ceph-mon[30894]: pgmap v195: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:14.591 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:14 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:36:14.591 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:14 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:36:14.591 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:14 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:14.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:14 smithi084 ceph-mon[27254]: pgmap v195: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:14.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:14 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:36:14.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:14 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:36:14.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:14 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:14.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:14 smithi121 ceph-mon[30729]: pgmap v195: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:14.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:14 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:36:14.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:14 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:36:14.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:14 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:15.621 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:15 smithi175 ceph-mon[30894]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:36:15.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:15 smithi084 ceph-mon[27254]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:36:15.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:15 smithi121 ceph-mon[30729]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:36:16.561 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:16 smithi175 ceph-mon[30894]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:16.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:16 smithi084 ceph-mon[27254]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:16.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:16 smithi121 ceph-mon[30729]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:17.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.175:0/67669156' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "50162e42-0443-41d1-8709-a0db61c087da"} : dispatch 2024-09-11T03:36:17.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:17 smithi084 ceph-mon[27254]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "50162e42-0443-41d1-8709-a0db61c087da"} : dispatch 2024-09-11T03:36:17.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:17 smithi084 ceph-mon[27254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50162e42-0443-41d1-8709-a0db61c087da"}]': finished 2024-09-11T03:36:17.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:17 smithi084 ceph-mon[27254]: osdmap e63: 10 total, 9 up, 10 in 2024-09-11T03:36:17.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:17 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:17.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.175:0/4036709457' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:36:17.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.175:0/67669156' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "50162e42-0443-41d1-8709-a0db61c087da"} : dispatch 2024-09-11T03:36:17.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:17 smithi121 ceph-mon[30729]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "50162e42-0443-41d1-8709-a0db61c087da"} : dispatch 2024-09-11T03:36:17.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:17 smithi121 ceph-mon[30729]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50162e42-0443-41d1-8709-a0db61c087da"}]': finished 2024-09-11T03:36:17.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:17 smithi121 ceph-mon[30729]: osdmap e63: 10 total, 9 up, 10 in 2024-09-11T03:36:17.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:17 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:17.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.175:0/4036709457' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:36:17.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.175:0/67669156' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "50162e42-0443-41d1-8709-a0db61c087da"} : dispatch 2024-09-11T03:36:17.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:17 smithi175 ceph-mon[30894]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "50162e42-0443-41d1-8709-a0db61c087da"} : dispatch 2024-09-11T03:36:17.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:17 smithi175 ceph-mon[30894]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50162e42-0443-41d1-8709-a0db61c087da"}]': finished 2024-09-11T03:36:17.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:17 smithi175 ceph-mon[30894]: osdmap e63: 10 total, 9 up, 10 in 2024-09-11T03:36:17.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:17 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:17.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.175:0/4036709457' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:36:18.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:18 smithi084 ceph-mon[27254]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-11T03:36:18.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:18 smithi121 ceph-mon[30729]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-11T03:36:18.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:18 smithi175 ceph-mon[30894]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-11T03:36:20.436 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:20 smithi175 ceph-mon[30894]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-11T03:36:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:20 smithi084 ceph-mon[27254]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-11T03:36:20.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:20 smithi121 ceph-mon[30729]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-11T03:36:22.626 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:22 smithi175 ceph-mon[30894]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-11T03:36:22.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:22 smithi084 ceph-mon[27254]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-11T03:36:22.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:22 smithi121 ceph-mon[30729]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-11T03:36:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-11T03:36:24.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:24.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:24.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:36:24.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:36:24.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:36:24.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:24 smithi175 ceph-mon[30894]: Deploying daemon osd.9 on smithi175 2024-09-11T03:36:24.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:24.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:24 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:36:24.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:24 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:36:24.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:24.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:24.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:24.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-11T03:36:24.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:24.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:24.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:36:24.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:36:24.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:36:24.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:24 smithi084 ceph-mon[27254]: Deploying daemon osd.9 on smithi175 2024-09-11T03:36:24.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:24.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:24 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:36:24.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:24 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:36:24.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:24.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:24.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:24.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-11T03:36:24.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:24.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:24.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:36:24.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:36:24.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:36:24.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:24 smithi121 ceph-mon[30729]: Deploying daemon osd.9 on smithi175 2024-09-11T03:36:24.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:24.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:24 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:36:24.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:24 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:36:24.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:24.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:24.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:25.391 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:25 smithi175 ceph-mon[30894]: osdmap e64: 10 total, 9 up, 10 in 2024-09-11T03:36:25.391 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:25.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:25 smithi084 ceph-mon[27254]: osdmap e64: 10 total, 9 up, 10 in 2024-09-11T03:36:25.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:25.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:25 smithi121 ceph-mon[30729]: osdmap e64: 10 total, 9 up, 10 in 2024-09-11T03:36:25.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:26 smithi084 ceph-mon[27254]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-11T03:36:26.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:26 smithi121 ceph-mon[30729]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-11T03:36:26.708 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:26 smithi175 ceph-mon[30894]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-11T03:36:27.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:27 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:36:27.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:27 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:36:27.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:27 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:36:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:27 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:36:27.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:27 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:36:27.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:27 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:36:28.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:28 smithi084 ceph-mon[27254]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:28.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:28 smithi084 ceph-mon[27254]: osdmap e65: 10 total, 9 up, 10 in 2024-09-11T03:36:28.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:28 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:28.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:28 smithi084 ceph-mon[27254]: 1.0 scrub starts 2024-09-11T03:36:28.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:28 smithi084 ceph-mon[27254]: 1.0 scrub ok 2024-09-11T03:36:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:28 smithi121 ceph-mon[30729]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:28 smithi121 ceph-mon[30729]: osdmap e65: 10 total, 9 up, 10 in 2024-09-11T03:36:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:28 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:28 smithi121 ceph-mon[30729]: 1.0 scrub starts 2024-09-11T03:36:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:28 smithi121 ceph-mon[30729]: 1.0 scrub ok 2024-09-11T03:36:28.703 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:28 smithi175 ceph-mon[30894]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:28.703 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:28 smithi175 ceph-mon[30894]: osdmap e65: 10 total, 9 up, 10 in 2024-09-11T03:36:28.703 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:28 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:28.704 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:28 smithi175 ceph-mon[30894]: 1.0 scrub starts 2024-09-11T03:36:28.704 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:28 smithi175 ceph-mon[30894]: 1.0 scrub ok 2024-09-11T03:36:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:30 smithi084 ceph-mon[27254]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:30 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:30 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:30 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:30 smithi121 ceph-mon[30729]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:30 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:30 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:30 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:30.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:30 smithi175 ceph-mon[30894]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:30.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:30 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:30.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:30 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:30.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:30 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:32.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:32 smithi084 ceph-mon[27254]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:32.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:32 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:32.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:32 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:32.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:32 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:36:32.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:32 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:36:32.659 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:32 smithi175 ceph-mon[30894]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:32.659 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:32 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:32.659 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:32 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:32.659 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:32 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:36:32.659 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:32 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:36:32.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:32 smithi121 ceph-mon[30729]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:32.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:32 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:32.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:32 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:32.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:32 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:36:32.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:32 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:36:32.932 INFO:teuthology.orchestra.run.smithi175.stdout:Created osd(s) 9 on host 'smithi175' 2024-09-11T03:36:33.521 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:33 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:33.521 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:33 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:33.521 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:33 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:33.521 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:33 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:33.521 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:33 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:33.522 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:33 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:33.522 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:33 smithi175 ceph-mon[30894]: osdmap e66: 10 total, 9 up, 10 in 2024-09-11T03:36:33.522 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:33 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:33.547 DEBUG:teuthology.orchestra.run.smithi175:osd.9> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.9.service 2024-09-11T03:36:33.550 INFO:tasks.cephadm:Deploying osd.10 on smithi175 with /dev/vg_nvme/lv_2... 2024-09-11T03:36:33.550 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-11T03:36:33.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:33 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:33.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:33 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:33.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:33 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:33.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:33 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:33.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:33 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:33.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:33 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:33.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:33 smithi084 ceph-mon[27254]: osdmap e66: 10 total, 9 up, 10 in 2024-09-11T03:36:33.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:33 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:33.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:33 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:33 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:33 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:33 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:33 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:33 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:33 smithi121 ceph-mon[30729]: osdmap e66: 10 total, 9 up, 10 in 2024-09-11T03:36:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:33 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:34.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:34 smithi084 ceph-mon[27254]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:34.668 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:34 smithi175 ceph-mon[30894]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:34.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:34 smithi121 ceph-mon[30729]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:34.977 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:36:34 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:36:34.765+0000 7fbd9657f640 -1 osd.9 0 log_to_monitors true 2024-09-11T03:36:35.282 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:36:35.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:35 smithi084 ceph-mon[27254]: from='osd.9 [v2:172.21.15.175:6808/1270991649,v1:172.21.15.175:6809/1270991649]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-11T03:36:35.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:35 smithi084 ceph-mon[27254]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-11T03:36:35.649 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:35 smithi175 ceph-mon[30894]: from='osd.9 [v2:172.21.15.175:6808/1270991649,v1:172.21.15.175:6809/1270991649]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-11T03:36:35.649 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:35 smithi175 ceph-mon[30894]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-11T03:36:35.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:35 smithi121 ceph-mon[30729]: from='osd.9 [v2:172.21.15.175:6808/1270991649,v1:172.21.15.175:6809/1270991649]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-11T03:36:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:35 smithi121 ceph-mon[30729]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-11T03:36:36.546 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:36:36 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:36:36.394+0000 7fbd92502640 -1 osd.9 0 waiting for initial osdmap 2024-09-11T03:36:36.546 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:36:36 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:36:36.402+0000 7fbd8e329640 -1 osd.9 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T03:36:36.546 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:36 smithi175 ceph-mon[30894]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:36.546 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:36 smithi175 ceph-mon[30894]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-11T03:36:36.546 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:36 smithi175 ceph-mon[30894]: osdmap e67: 10 total, 9 up, 10 in 2024-09-11T03:36:36.546 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:36 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:36.546 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:36 smithi175 ceph-mon[30894]: from='osd.9 [v2:172.21.15.175:6808/1270991649,v1:172.21.15.175:6809/1270991649]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:36:36.546 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:36 smithi175 ceph-mon[30894]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:36:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:36 smithi084 ceph-mon[27254]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:36 smithi084 ceph-mon[27254]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-11T03:36:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:36 smithi084 ceph-mon[27254]: osdmap e67: 10 total, 9 up, 10 in 2024-09-11T03:36:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:36 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:36 smithi084 ceph-mon[27254]: from='osd.9 [v2:172.21.15.175:6808/1270991649,v1:172.21.15.175:6809/1270991649]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:36:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:36 smithi084 ceph-mon[27254]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:36:36.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:36 smithi121 ceph-mon[30729]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:36 smithi121 ceph-mon[30729]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-11T03:36:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:36 smithi121 ceph-mon[30729]: osdmap e67: 10 total, 9 up, 10 in 2024-09-11T03:36:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:36 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:36 smithi121 ceph-mon[30729]: from='osd.9 [v2:172.21.15.175:6808/1270991649,v1:172.21.15.175:6809/1270991649]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:36:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:36 smithi121 ceph-mon[30729]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:36:37.626 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:37 smithi175 ceph-mon[30894]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-11T03:36:37.626 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:37 smithi175 ceph-mon[30894]: osdmap e68: 10 total, 9 up, 10 in 2024-09-11T03:36:37.627 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:37 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:37.627 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:37 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:37.627 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:37 smithi175 ceph-mon[30894]: Detected new or changed devices on smithi175 2024-09-11T03:36:37.627 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:37 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:37.627 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:37 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:37.627 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:37 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-11T03:36:37.627 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:37 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-11T03:36:37.627 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:37 smithi175 ceph-mon[30894]: Adjusting osd_memory_target on smithi175 to 10592M 2024-09-11T03:36:37.627 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:37 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:37.627 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:37 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:37.627 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:37 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:37.627 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:37 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:37.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:37 smithi121 ceph-mon[30729]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-11T03:36:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:37 smithi121 ceph-mon[30729]: osdmap e68: 10 total, 9 up, 10 in 2024-09-11T03:36:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:37 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:37 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:37 smithi121 ceph-mon[30729]: Detected new or changed devices on smithi175 2024-09-11T03:36:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:37 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:37 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:37 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-11T03:36:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:37 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-11T03:36:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:37 smithi121 ceph-mon[30729]: Adjusting osd_memory_target on smithi175 to 10592M 2024-09-11T03:36:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:37 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:37 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:37 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:37 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:37.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:37 smithi084 ceph-mon[27254]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-11T03:36:37.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:37 smithi084 ceph-mon[27254]: osdmap e68: 10 total, 9 up, 10 in 2024-09-11T03:36:37.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:37 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:37.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:37 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:37.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:37 smithi084 ceph-mon[27254]: Detected new or changed devices on smithi175 2024-09-11T03:36:37.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:37 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:37.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:37 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:37.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:37 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-11T03:36:37.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:37 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-11T03:36:37.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:37 smithi084 ceph-mon[27254]: Adjusting osd_memory_target on smithi175 to 10592M 2024-09-11T03:36:37.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:37 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:37.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:37 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:37.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:37 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:37.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:37 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:38.267 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:36:38.294 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi175:vg_nvme/lv_2 2024-09-11T03:36:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:38 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:36:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:38 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:36:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:38 smithi175 ceph-mon[30894]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:38 smithi175 ceph-mon[30894]: osd.9 [v2:172.21.15.175:6808/1270991649,v1:172.21.15.175:6809/1270991649] boot 2024-09-11T03:36:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:38 smithi175 ceph-mon[30894]: osdmap e69: 10 total, 10 up, 10 in 2024-09-11T03:36:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:38 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:38.538 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:36:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:38 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:36:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:38 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:36:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:38 smithi121 ceph-mon[30729]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:38 smithi121 ceph-mon[30729]: osd.9 [v2:172.21.15.175:6808/1270991649,v1:172.21.15.175:6809/1270991649] boot 2024-09-11T03:36:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:38 smithi121 ceph-mon[30729]: osdmap e69: 10 total, 10 up, 10 in 2024-09-11T03:36:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:38 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:38.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:38 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:36:38.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:38 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:36:38.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:38 smithi084 ceph-mon[27254]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-09-11T03:36:38.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:38 smithi084 ceph-mon[27254]: osd.9 [v2:172.21.15.175:6808/1270991649,v1:172.21.15.175:6809/1270991649] boot 2024-09-11T03:36:38.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:38 smithi084 ceph-mon[27254]: osdmap e69: 10 total, 10 up, 10 in 2024-09-11T03:36:38.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:38 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-11T03:36:39.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:39 smithi084 ceph-mon[27254]: osdmap e70: 10 total, 10 up, 10 in 2024-09-11T03:36:39.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:39 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:36:39.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:39 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:36:39.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:39 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:39.894 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:39 smithi175 ceph-mon[30894]: osdmap e70: 10 total, 10 up, 10 in 2024-09-11T03:36:39.895 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:39 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:36:39.895 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:39 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:36:39.895 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:39 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:39.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:39 smithi121 ceph-mon[30729]: osdmap e70: 10 total, 10 up, 10 in 2024-09-11T03:36:39.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:39 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:36:39.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:39 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:36:39.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:39 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:40.833 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:40 smithi175 ceph-mon[30894]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:40.872 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:40 smithi175 ceph-mon[30894]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:36:40.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:40 smithi084 ceph-mon[27254]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:40.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:40 smithi084 ceph-mon[27254]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:36:40.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:40 smithi121 ceph-mon[30729]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:40.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:40 smithi121 ceph-mon[30729]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:36:41.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:41 smithi084 ceph-mon[27254]: from='client.? 172.21.15.175:0/1982236137' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4b9426e1-bac8-47d3-ad58-54b89e8d1eb1"} : dispatch 2024-09-11T03:36:41.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:41 smithi084 ceph-mon[27254]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4b9426e1-bac8-47d3-ad58-54b89e8d1eb1"} : dispatch 2024-09-11T03:36:41.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:41 smithi084 ceph-mon[27254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b9426e1-bac8-47d3-ad58-54b89e8d1eb1"}]': finished 2024-09-11T03:36:41.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:41 smithi084 ceph-mon[27254]: osdmap e71: 11 total, 10 up, 11 in 2024-09-11T03:36:41.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:41 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:36:41.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:41 smithi121 ceph-mon[30729]: from='client.? 172.21.15.175:0/1982236137' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4b9426e1-bac8-47d3-ad58-54b89e8d1eb1"} : dispatch 2024-09-11T03:36:41.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:41 smithi121 ceph-mon[30729]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4b9426e1-bac8-47d3-ad58-54b89e8d1eb1"} : dispatch 2024-09-11T03:36:41.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:41 smithi121 ceph-mon[30729]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b9426e1-bac8-47d3-ad58-54b89e8d1eb1"}]': finished 2024-09-11T03:36:41.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:41 smithi121 ceph-mon[30729]: osdmap e71: 11 total, 10 up, 11 in 2024-09-11T03:36:41.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:41 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:36:41.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:41 smithi175 ceph-mon[30894]: from='client.? 172.21.15.175:0/1982236137' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4b9426e1-bac8-47d3-ad58-54b89e8d1eb1"} : dispatch 2024-09-11T03:36:41.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:41 smithi175 ceph-mon[30894]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4b9426e1-bac8-47d3-ad58-54b89e8d1eb1"} : dispatch 2024-09-11T03:36:41.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:41 smithi175 ceph-mon[30894]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b9426e1-bac8-47d3-ad58-54b89e8d1eb1"}]': finished 2024-09-11T03:36:41.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:41 smithi175 ceph-mon[30894]: osdmap e71: 11 total, 10 up, 11 in 2024-09-11T03:36:41.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:41 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:36:42.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:42 smithi084 ceph-mon[27254]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:42.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:42 smithi084 ceph-mon[27254]: from='client.? 172.21.15.175:0/3896029974' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:36:42.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:42 smithi121 ceph-mon[30729]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:42.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:42 smithi121 ceph-mon[30729]: from='client.? 172.21.15.175:0/3896029974' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:36:42.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:42 smithi175 ceph-mon[30894]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:42.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:42 smithi175 ceph-mon[30894]: from='client.? 172.21.15.175:0/3896029974' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:36:44.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:44 smithi084 ceph-mon[27254]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:44.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:44 smithi121 ceph-mon[30729]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:44.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:44 smithi175 ceph-mon[30894]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:46.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:46 smithi084 ceph-mon[27254]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:46.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:46 smithi121 ceph-mon[30729]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:46.942 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:46 smithi175 ceph-mon[30894]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:48.760 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:48 smithi175 ceph-mon[30894]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:48.760 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:48 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:36:48.760 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:48 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:36:48.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:48 smithi084 ceph-mon[27254]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:48.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:48 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:36:48.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:48 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:36:48.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:48 smithi121 ceph-mon[30729]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:48.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:48 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:36:48.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:48 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:36:49.782 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:49 smithi175 ceph-mon[30894]: osdmap e72: 11 total, 10 up, 11 in 2024-09-11T03:36:49.782 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:49 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:36:49.782 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:49 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-11T03:36:49.782 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:49 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:49 smithi084 ceph-mon[27254]: osdmap e72: 11 total, 10 up, 11 in 2024-09-11T03:36:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:49 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:36:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:49 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-11T03:36:49.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:49 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:49.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:49 smithi121 ceph-mon[30729]: osdmap e72: 11 total, 10 up, 11 in 2024-09-11T03:36:49.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:49 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:36:49.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:49 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-11T03:36:49.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:49 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:50.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:50 smithi084 ceph-mon[27254]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:50.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:50 smithi084 ceph-mon[27254]: Deploying daemon osd.10 on smithi175 2024-09-11T03:36:50.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:50 smithi121 ceph-mon[30729]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:50.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:50 smithi121 ceph-mon[30729]: Deploying daemon osd.10 on smithi175 2024-09-11T03:36:50.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:50 smithi175 ceph-mon[30894]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:50.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:50 smithi175 ceph-mon[30894]: Deploying daemon osd.10 on smithi175 2024-09-11T03:36:52.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:52 smithi121 ceph-mon[30729]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:52.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:52 smithi175 ceph-mon[30894]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:53.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:52 smithi084 ceph-mon[27254]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:54.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:54 smithi121 ceph-mon[30729]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:54.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:54 smithi175 ceph-mon[30894]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:55.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:54 smithi084 ceph-mon[27254]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:55.698 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:55 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:55.698 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:55 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:55.698 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:55 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:55.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:55 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:55.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:55 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:55.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:55 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:56.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:55 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:56.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:55 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:56.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:55 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:56.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:56 smithi121 ceph-mon[30729]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:56.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:56 smithi175 ceph-mon[30894]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:57.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:56 smithi084 ceph-mon[27254]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:58.306 INFO:teuthology.orchestra.run.smithi175.stdout:Created osd(s) 10 on host 'smithi175' 2024-09-11T03:36:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:58 smithi121 ceph-mon[30729]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:58 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:36:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:58 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:36:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:58 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:58 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:58 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:58 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:58 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:58 smithi121 ceph-mon[30729]: osdmap e73: 11 total, 10 up, 11 in 2024-09-11T03:36:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:58 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:36:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:58 smithi175 ceph-mon[30894]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:58 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:36:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:58 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:36:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:58 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:58 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:58 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:58 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:58.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:58 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:58.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:58 smithi175 ceph-mon[30894]: osdmap e73: 11 total, 10 up, 11 in 2024-09-11T03:36:58.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:58 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:36:58.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:58 smithi084 ceph-mon[27254]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:36:58.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:58 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:36:58.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:58 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:36:58.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:58 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:58.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:58 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:58.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:58 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:36:58.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:58 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:36:58.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:58 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:58.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:58 smithi084 ceph-mon[27254]: osdmap e73: 11 total, 10 up, 11 in 2024-09-11T03:36:58.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:58 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:36:58.925 DEBUG:teuthology.orchestra.run.smithi175:osd.10> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.10.service 2024-09-11T03:36:58.927 INFO:tasks.cephadm:Deploying osd.11 on smithi175 with /dev/vg_nvme/lv_1... 2024-09-11T03:36:58.927 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-11T03:36:59.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:59 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:59.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:59 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:59.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:59 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:59.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:59 smithi175 ceph-mon[30894]: from='osd.10 [v2:172.21.15.175:6816/3877631021,v1:172.21.15.175:6817/3877631021]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-11T03:36:59.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:36:59 smithi175 ceph-mon[30894]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-11T03:36:59.228 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:36:59 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:36:59.060+0000 7fc3bb02e640 -1 osd.10 0 log_to_monitors true 2024-09-11T03:36:59.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:59 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:59.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:59 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:59.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:59 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:59.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:59 smithi121 ceph-mon[30729]: from='osd.10 [v2:172.21.15.175:6816/3877631021,v1:172.21.15.175:6817/3877631021]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-11T03:36:59.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:36:59 smithi121 ceph-mon[30729]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-11T03:36:59.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:59 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:36:59.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:59 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:59.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:59 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:36:59.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:59 smithi084 ceph-mon[27254]: from='osd.10 [v2:172.21.15.175:6816/3877631021,v1:172.21.15.175:6817/3877631021]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-11T03:36:59.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:36:59 smithi084 ceph-mon[27254]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-11T03:36:59.961 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:37:00.566 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:37:00 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:37:00.315+0000 7fc3b6fb1640 -1 osd.10 0 waiting for initial osdmap 2024-09-11T03:37:00.566 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:37:00 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:37:00.323+0000 7fc3b2dd8640 -1 osd.10 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T03:37:00.566 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:00 smithi175 ceph-mon[30894]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:37:00.566 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:00 smithi175 ceph-mon[30894]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-11T03:37:00.566 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:00 smithi175 ceph-mon[30894]: osdmap e74: 11 total, 10 up, 11 in 2024-09-11T03:37:00.566 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:37:00.566 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:00 smithi175 ceph-mon[30894]: from='osd.10 [v2:172.21.15.175:6816/3877631021,v1:172.21.15.175:6817/3877631021]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:37:00.566 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:00 smithi175 ceph-mon[30894]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:37:00.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:00 smithi084 ceph-mon[27254]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:37:00.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:00 smithi084 ceph-mon[27254]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-11T03:37:00.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:00 smithi084 ceph-mon[27254]: osdmap e74: 11 total, 10 up, 11 in 2024-09-11T03:37:00.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:37:00.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:00 smithi084 ceph-mon[27254]: from='osd.10 [v2:172.21.15.175:6816/3877631021,v1:172.21.15.175:6817/3877631021]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:37:00.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:00 smithi084 ceph-mon[27254]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:37:00.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:00 smithi121 ceph-mon[30729]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:37:00.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:00 smithi121 ceph-mon[30729]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-11T03:37:00.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:00 smithi121 ceph-mon[30729]: osdmap e74: 11 total, 10 up, 11 in 2024-09-11T03:37:00.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:37:00.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:00 smithi121 ceph-mon[30729]: from='osd.10 [v2:172.21.15.175:6816/3877631021,v1:172.21.15.175:6817/3877631021]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:37:00.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:00 smithi121 ceph-mon[30729]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:37:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:01 smithi175 ceph-mon[30894]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-11T03:37:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:01 smithi175 ceph-mon[30894]: osdmap e75: 11 total, 10 up, 11 in 2024-09-11T03:37:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:01 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:37:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:01 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:37:01.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:01 smithi084 ceph-mon[27254]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-11T03:37:01.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:01 smithi084 ceph-mon[27254]: osdmap e75: 11 total, 10 up, 11 in 2024-09-11T03:37:01.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:01 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:37:01.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:01 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:37:01.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:01 smithi121 ceph-mon[30729]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-11T03:37:01.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:01 smithi121 ceph-mon[30729]: osdmap e75: 11 total, 10 up, 11 in 2024-09-11T03:37:01.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:01 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:37:01.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:01 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:37:02.603 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:02 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:37:02.603 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:02 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:37:02.603 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:02 smithi175 ceph-mon[30894]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:37:02.603 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:02 smithi175 ceph-mon[30894]: osd.10 [v2:172.21.15.175:6816/3877631021,v1:172.21.15.175:6817/3877631021] boot 2024-09-11T03:37:02.603 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:02 smithi175 ceph-mon[30894]: osdmap e76: 11 total, 11 up, 11 in 2024-09-11T03:37:02.603 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:02 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:37:02.603 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:02 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:02.603 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:02 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:02.603 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:02 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:02.603 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:02 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:02.603 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:02 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:02.604 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:02 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:02.604 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:02 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:37:02.604 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:02 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:37:02.604 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:02 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:02.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:02 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:37:02.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:02 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:37:02.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:02 smithi084 ceph-mon[27254]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:37:02.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:02 smithi084 ceph-mon[27254]: osd.10 [v2:172.21.15.175:6816/3877631021,v1:172.21.15.175:6817/3877631021] boot 2024-09-11T03:37:02.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:02 smithi084 ceph-mon[27254]: osdmap e76: 11 total, 11 up, 11 in 2024-09-11T03:37:02.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:02 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:37:02.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:02 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:02.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:02 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:02.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:02 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:02.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:02 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:02.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:02 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:02.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:02 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:02.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:02 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:37:02.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:02 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:37:02.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:02 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:02.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:02 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:37:02.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:02 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:37:02.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:02 smithi121 ceph-mon[30729]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 894 GiB / 894 GiB avail 2024-09-11T03:37:02.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:02 smithi121 ceph-mon[30729]: osd.10 [v2:172.21.15.175:6816/3877631021,v1:172.21.15.175:6817/3877631021] boot 2024-09-11T03:37:02.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:02 smithi121 ceph-mon[30729]: osdmap e76: 11 total, 11 up, 11 in 2024-09-11T03:37:02.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:02 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-11T03:37:02.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:02 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:02.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:02 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:02.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:02 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:02.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:02 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:02.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:02 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:02.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:02 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:02.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:02 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:37:02.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:02 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:37:02.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:02 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:03.590 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:03 smithi175 ceph-mon[30894]: Detected new or changed devices on smithi175 2024-09-11T03:37:03.590 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:03 smithi175 ceph-mon[30894]: Adjusting osd_memory_target on smithi175 to 7061M 2024-09-11T03:37:03.590 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:03 smithi175 ceph-mon[30894]: osdmap e77: 11 total, 11 up, 11 in 2024-09-11T03:37:03.593 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-11T03:37:03.620 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi175:vg_nvme/lv_1 2024-09-11T03:37:03.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:03 smithi084 ceph-mon[27254]: Detected new or changed devices on smithi175 2024-09-11T03:37:03.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:03 smithi084 ceph-mon[27254]: Adjusting osd_memory_target on smithi175 to 7061M 2024-09-11T03:37:03.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:03 smithi084 ceph-mon[27254]: osdmap e77: 11 total, 11 up, 11 in 2024-09-11T03:37:03.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:03 smithi121 ceph-mon[30729]: Detected new or changed devices on smithi175 2024-09-11T03:37:03.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:03 smithi121 ceph-mon[30729]: Adjusting osd_memory_target on smithi175 to 7061M 2024-09-11T03:37:03.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:03 smithi121 ceph-mon[30729]: osdmap e77: 11 total, 11 up, 11 in 2024-09-11T03:37:03.866 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:37:04.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:04 smithi084 ceph-mon[27254]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:04.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:04 smithi121 ceph-mon[30729]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:04.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:04 smithi175 ceph-mon[30894]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:05.582 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:05 smithi175 ceph-mon[30894]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:37:05.582 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:05 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:37:05.582 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:05 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:37:05.582 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:05 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:37:05.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:05 smithi084 ceph-mon[27254]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:37:05.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:05 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:37:05.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:05 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:37:05.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:05 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:37:05.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:05 smithi121 ceph-mon[30729]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:37:05.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:05 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-11T03:37:05.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:05 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-11T03:37:05.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:05 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:37:06.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:06 smithi175 ceph-mon[30894]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:06.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:06 smithi084 ceph-mon[27254]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:06.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:06 smithi121 ceph-mon[30729]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.175:0/227018430' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cd1a28a1-413c-4274-9876-e1bdb7703be6"} : dispatch 2024-09-11T03:37:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:07 smithi175 ceph-mon[30894]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cd1a28a1-413c-4274-9876-e1bdb7703be6"} : dispatch 2024-09-11T03:37:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:07 smithi175 ceph-mon[30894]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd1a28a1-413c-4274-9876-e1bdb7703be6"}]': finished 2024-09-11T03:37:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:07 smithi175 ceph-mon[30894]: osdmap e78: 12 total, 11 up, 12 in 2024-09-11T03:37:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:07.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.175:0/227018430' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cd1a28a1-413c-4274-9876-e1bdb7703be6"} : dispatch 2024-09-11T03:37:07.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:07 smithi084 ceph-mon[27254]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cd1a28a1-413c-4274-9876-e1bdb7703be6"} : dispatch 2024-09-11T03:37:07.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:07 smithi084 ceph-mon[27254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd1a28a1-413c-4274-9876-e1bdb7703be6"}]': finished 2024-09-11T03:37:07.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:07 smithi084 ceph-mon[27254]: osdmap e78: 12 total, 11 up, 12 in 2024-09-11T03:37:07.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:07.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.175:0/227018430' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cd1a28a1-413c-4274-9876-e1bdb7703be6"} : dispatch 2024-09-11T03:37:07.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:07 smithi121 ceph-mon[30729]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cd1a28a1-413c-4274-9876-e1bdb7703be6"} : dispatch 2024-09-11T03:37:07.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:07 smithi121 ceph-mon[30729]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd1a28a1-413c-4274-9876-e1bdb7703be6"}]': finished 2024-09-11T03:37:07.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:07 smithi121 ceph-mon[30729]: osdmap e78: 12 total, 11 up, 12 in 2024-09-11T03:37:07.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:08.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:08 smithi084 ceph-mon[27254]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:08.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:08 smithi084 ceph-mon[27254]: from='client.? 172.21.15.175:0/1126191929' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:37:08.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:08 smithi121 ceph-mon[30729]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:08.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:08 smithi121 ceph-mon[30729]: from='client.? 172.21.15.175:0/1126191929' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:37:08.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:08 smithi175 ceph-mon[30894]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:08.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:08 smithi175 ceph-mon[30894]: from='client.? 172.21.15.175:0/1126191929' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-11T03:37:10.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:10 smithi084 ceph-mon[27254]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:10.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:10 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:37:10.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:10 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:37:10.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:10 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:37:10.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:10 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:37:10.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:10 smithi121 ceph-mon[30729]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:10.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:10 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:37:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:10 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:37:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:10 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:37:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:10 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:37:10.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:10 smithi175 ceph-mon[30894]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:10.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:10 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:37:10.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:10 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:37:10.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:10 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:37:10.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:10 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:37:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:11 smithi175 ceph-mon[30894]: osdmap e79: 12 total, 11 up, 12 in 2024-09-11T03:37:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:11 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:11.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:11 smithi084 ceph-mon[27254]: osdmap e79: 12 total, 11 up, 12 in 2024-09-11T03:37:11.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:11 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:11.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:11 smithi121 ceph-mon[30729]: osdmap e79: 12 total, 11 up, 12 in 2024-09-11T03:37:11.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:11 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:12.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:12 smithi084 ceph-mon[27254]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:12.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:12 smithi121 ceph-mon[30729]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:12.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:12 smithi175 ceph-mon[30894]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:14.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:14 smithi084 ceph-mon[27254]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:14.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:14 smithi121 ceph-mon[30729]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:14.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:14 smithi175 ceph-mon[30894]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:15.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-11T03:37:15.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:37:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: Deploying daemon osd.11 on smithi175 2024-09-11T03:37:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=64} crc {auth=808111922,config=2788817630,health=861809967,kv=2038225040}) 2024-09-11T03:37:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2489569914}) 2024-09-11T03:37:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {kv=71,logm=29} crc {kv=3970974894,logm=878009365}) 2024-09-11T03:37:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2660648141}) 2024-09-11T03:37:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2834427297}) 2024-09-11T03:37:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1268162233}) 2024-09-11T03:37:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=591283183}) 2024-09-11T03:37:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=363236580}) 2024-09-11T03:37:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1124998426}) 2024-09-11T03:37:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3926884187}) 2024-09-11T03:37:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1229098322}) 2024-09-11T03:37:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2405096398}) 2024-09-11T03:37:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3439860768}) 2024-09-11T03:37:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1944387479}) 2024-09-11T03:37:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1336744254}) 2024-09-11T03:37:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642265124}) 2024-09-11T03:37:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2979683764}) 2024-09-11T03:37:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=82,mdsmap=3,mgr=15} crc {logm=1263319975,mdsmap=827622067,mgr=3243923575}) 2024-09-11T03:37:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=2,mgrstat=94} crc {mgr=4028747438,mgr_command_descs=899306875,mgr_metadata=2702562194,mgrstat=1602317102}) 2024-09-11T03:37:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=69,mon_config_key=31} crc {mgrstat=3288869952,mon_config_key=3593044671}) 2024-09-11T03:37:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1901096385}) 2024-09-11T03:37:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4146107309}) 2024-09-11T03:37:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=27,monmap=5,nvmeofgw=3,osd_metadata=11,osd_pg_creating=1,osdmap=53} crc {mon_config_key=995053694,monmap=4220057526,nvmeofgw=598337686,osd_metadata=2052844472,osd_pg_creating=1136303411,osdmap=2419183206}) 2024-09-11T03:37:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4251262434}) 2024-09-11T03:37:15.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=8} crc {osdmap=537043632}) 2024-09-11T03:37:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-11T03:37:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:37:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: Deploying daemon osd.11 on smithi175 2024-09-11T03:37:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=64} crc {auth=808111922,config=2788817630,health=861809967,kv=2038225040}) 2024-09-11T03:37:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2489569914}) 2024-09-11T03:37:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {kv=71,logm=29} crc {kv=3970974894,logm=878009365}) 2024-09-11T03:37:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2660648141}) 2024-09-11T03:37:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2834427297}) 2024-09-11T03:37:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1268162233}) 2024-09-11T03:37:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=591283183}) 2024-09-11T03:37:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=363236580}) 2024-09-11T03:37:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1124998426}) 2024-09-11T03:37:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3926884187}) 2024-09-11T03:37:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1229098322}) 2024-09-11T03:37:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2405096398}) 2024-09-11T03:37:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3439860768}) 2024-09-11T03:37:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1944387479}) 2024-09-11T03:37:15.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1336744254}) 2024-09-11T03:37:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642265124}) 2024-09-11T03:37:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2979683764}) 2024-09-11T03:37:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=82,mdsmap=3,mgr=15} crc {logm=1263319975,mdsmap=827622067,mgr=3243923575}) 2024-09-11T03:37:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=2,mgrstat=94} crc {mgr=4028747438,mgr_command_descs=899306875,mgr_metadata=2702562194,mgrstat=1602317102}) 2024-09-11T03:37:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=69,mon_config_key=31} crc {mgrstat=3288869952,mon_config_key=3593044671}) 2024-09-11T03:37:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1901096385}) 2024-09-11T03:37:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4146107309}) 2024-09-11T03:37:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=27,monmap=5,nvmeofgw=3,osd_metadata=11,osd_pg_creating=1,osdmap=53} crc {mon_config_key=995053694,monmap=4220057526,nvmeofgw=598337686,osd_metadata=2052844472,osd_pg_creating=1136303411,osdmap=2419183206}) 2024-09-11T03:37:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4251262434}) 2024-09-11T03:37:15.674 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=8} crc {osdmap=537043632}) 2024-09-11T03:37:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-11T03:37:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:37:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: Deploying daemon osd.11 on smithi175 2024-09-11T03:37:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=64} crc {auth=808111922,config=2788817630,health=861809967,kv=2038225040}) 2024-09-11T03:37:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2489569914}) 2024-09-11T03:37:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {kv=71,logm=29} crc {kv=3970974894,logm=878009365}) 2024-09-11T03:37:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2660648141}) 2024-09-11T03:37:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2834427297}) 2024-09-11T03:37:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1268162233}) 2024-09-11T03:37:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=591283183}) 2024-09-11T03:37:15.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=363236580}) 2024-09-11T03:37:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1124998426}) 2024-09-11T03:37:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3926884187}) 2024-09-11T03:37:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1229098322}) 2024-09-11T03:37:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2405096398}) 2024-09-11T03:37:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3439860768}) 2024-09-11T03:37:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1944387479}) 2024-09-11T03:37:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1336744254}) 2024-09-11T03:37:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642265124}) 2024-09-11T03:37:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2979683764}) 2024-09-11T03:37:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=82,mdsmap=3,mgr=15} crc {logm=1263319975,mdsmap=827622067,mgr=3243923575}) 2024-09-11T03:37:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=2,mgrstat=94} crc {mgr=4028747438,mgr_command_descs=899306875,mgr_metadata=2702562194,mgrstat=1602317102}) 2024-09-11T03:37:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=69,mon_config_key=31} crc {mgrstat=3288869952,mon_config_key=3593044671}) 2024-09-11T03:37:15.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1901096385}) 2024-09-11T03:37:15.730 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4146107309}) 2024-09-11T03:37:15.730 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=27,monmap=5,nvmeofgw=3,osd_metadata=11,osd_pg_creating=1,osdmap=53} crc {mon_config_key=995053694,monmap=4220057526,nvmeofgw=598337686,osd_metadata=2052844472,osd_pg_creating=1136303411,osdmap=2419183206}) 2024-09-11T03:37:15.730 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4251262434}) 2024-09-11T03:37:15.730 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=8} crc {osdmap=537043632}) 2024-09-11T03:37:16.403 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:16 smithi175 ceph-mon[30894]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:16.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:16 smithi084 ceph-mon[27254]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:16.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:16 smithi121 ceph-mon[30729]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:18.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:18 smithi084 ceph-mon[27254]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:18.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:18 smithi121 ceph-mon[30729]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:18.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:18 smithi175 ceph-mon[30894]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:20 smithi084 ceph-mon[27254]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:20 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:37:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:20 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:37:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:20 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:37:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:20 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:20 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:20.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:20 smithi121 ceph-mon[30729]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:20.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:20 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:37:20.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:20 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:37:20.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:20 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:37:20.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:20 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:20.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:20 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:20.695 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:20 smithi175 ceph-mon[30894]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:20.695 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:20 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:37:20.696 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:20 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:37:20.696 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:20 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:37:20.696 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:20 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:20.696 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:20 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:21.625 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:21 smithi175 ceph-mon[30894]: osdmap e80: 12 total, 11 up, 12 in 2024-09-11T03:37:21.626 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:21.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:21 smithi121 ceph-mon[30729]: osdmap e80: 12 total, 11 up, 12 in 2024-09-11T03:37:21.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:21.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:21 smithi084 ceph-mon[27254]: osdmap e80: 12 total, 11 up, 12 in 2024-09-11T03:37:21.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:22.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:22 smithi175 ceph-mon[30894]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:22.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:22 smithi121 ceph-mon[30729]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:22.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:22 smithi084 ceph-mon[27254]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:23.600 INFO:teuthology.orchestra.run.smithi175.stdout:Created osd(s) 11 on host 'smithi175' 2024-09-11T03:37:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:23 smithi175 ceph-mon[30894]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:23 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:23 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:23 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:37:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:23 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:37:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:23 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:24.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:23 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:37:24.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:23 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:24.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:23 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:24.282 DEBUG:teuthology.orchestra.run.smithi175:osd.11> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.11.service 2024-09-11T03:37:24.284 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-09-11T03:37:24.285 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd stat -f json 2024-09-11T03:37:24.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:23 smithi084 ceph-mon[27254]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:24.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:23 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:24.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:23 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:24.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:23 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:37:24.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:23 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:37:24.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:23 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:24.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:23 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:37:24.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:23 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:24.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:23 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:23 smithi121 ceph-mon[30729]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:23 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:23 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:23 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:37:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:23 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:37:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:23 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:23 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:37:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:23 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:23 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:24.541 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:25.173 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:24 smithi084 ceph-mon[27254]: from='osd.11 [v2:172.21.15.175:6824/1150051236,v1:172.21.15.175:6825/1150051236]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-11T03:37:25.174 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:24 smithi084 ceph-mon[27254]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-11T03:37:25.192 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:37:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:24 smithi175 ceph-mon[30894]: from='osd.11 [v2:172.21.15.175:6824/1150051236,v1:172.21.15.175:6825/1150051236]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-11T03:37:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:24 smithi175 ceph-mon[30894]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-11T03:37:25.227 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:37:24 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:37:24.922+0000 7f990fde9640 -1 osd.11 0 log_to_monitors true 2024-09-11T03:37:25.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:24 smithi121 ceph-mon[30729]: from='osd.11 [v2:172.21.15.175:6824/1150051236,v1:172.21.15.175:6825/1150051236]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-11T03:37:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:24 smithi121 ceph-mon[30729]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-11T03:37:25.904 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":81,"num_osds":12,"num_up_osds":11,"osd_up_since":1726025821,"num_in_osds":12,"osd_in_since":1726025826,"num_remapped_pgs":0} 2024-09-11T03:37:26.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:25 smithi084 ceph-mon[27254]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:26.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:25 smithi084 ceph-mon[27254]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-11T03:37:26.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:25 smithi084 ceph-mon[27254]: osdmap e81: 12 total, 11 up, 12 in 2024-09-11T03:37:26.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:26.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:25 smithi084 ceph-mon[27254]: from='osd.11 [v2:172.21.15.175:6824/1150051236,v1:172.21.15.175:6825/1150051236]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:37:26.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:25 smithi084 ceph-mon[27254]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:37:26.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:25 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4167868064' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-11T03:37:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:25 smithi175 ceph-mon[30894]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:25 smithi175 ceph-mon[30894]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-11T03:37:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:25 smithi175 ceph-mon[30894]: osdmap e81: 12 total, 11 up, 12 in 2024-09-11T03:37:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:25 smithi175 ceph-mon[30894]: from='osd.11 [v2:172.21.15.175:6824/1150051236,v1:172.21.15.175:6825/1150051236]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:37:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:25 smithi175 ceph-mon[30894]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:37:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:25 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4167868064' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-11T03:37:26.228 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:37:25 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:37:25.961+0000 7f990bd6c640 -1 osd.11 0 waiting for initial osdmap 2024-09-11T03:37:26.228 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:37:25 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:37:25.969+0000 7f9907b93640 -1 osd.11 82 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T03:37:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:25 smithi121 ceph-mon[30729]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:25 smithi121 ceph-mon[30729]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-11T03:37:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:25 smithi121 ceph-mon[30729]: osdmap e81: 12 total, 11 up, 12 in 2024-09-11T03:37:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:25 smithi121 ceph-mon[30729]: from='osd.11 [v2:172.21.15.175:6824/1150051236,v1:172.21.15.175:6825/1150051236]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:37:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:25 smithi121 ceph-mon[30729]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi175", "root=default"]} : dispatch 2024-09-11T03:37:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:25 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4167868064' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-11T03:37:26.905 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd stat -f json 2024-09-11T03:37:26.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:26 smithi175 ceph-mon[30894]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-11T03:37:26.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:26 smithi175 ceph-mon[30894]: osdmap e82: 12 total, 11 up, 12 in 2024-09-11T03:37:26.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:26.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:26.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:26 smithi175 ceph-mon[30894]: osd.11 [v2:172.21.15.175:6824/1150051236,v1:172.21.15.175:6825/1150051236] boot 2024-09-11T03:37:26.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:26 smithi175 ceph-mon[30894]: osdmap e83: 12 total, 12 up, 12 in 2024-09-11T03:37:26.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:27.153 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:27.274 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:26 smithi084 ceph-mon[27254]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-11T03:37:27.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:26 smithi084 ceph-mon[27254]: osdmap e82: 12 total, 11 up, 12 in 2024-09-11T03:37:27.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:27.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:27.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:26 smithi084 ceph-mon[27254]: osd.11 [v2:172.21.15.175:6824/1150051236,v1:172.21.15.175:6825/1150051236] boot 2024-09-11T03:37:27.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:26 smithi084 ceph-mon[27254]: osdmap e83: 12 total, 12 up, 12 in 2024-09-11T03:37:27.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:26 smithi121 ceph-mon[30729]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-11T03:37:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:26 smithi121 ceph-mon[30729]: osdmap e82: 12 total, 11 up, 12 in 2024-09-11T03:37:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:26 smithi121 ceph-mon[30729]: osd.11 [v2:172.21.15.175:6824/1150051236,v1:172.21.15.175:6825/1150051236] boot 2024-09-11T03:37:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:26 smithi121 ceph-mon[30729]: osdmap e83: 12 total, 12 up, 12 in 2024-09-11T03:37:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-11T03:37:27.805 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:37:28.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:28 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:37:28.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:28 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:37:28.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:28 smithi084 ceph-mon[27254]: pgmap v252: 1 pgs: 1 peering; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:28.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:28 smithi084 ceph-mon[27254]: Detected new or changed devices on smithi175 2024-09-11T03:37:28.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:28 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:28.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:28 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:28.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:28 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:28.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:28 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:28.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:28 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:28.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:28 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:28.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:28 smithi084 ceph-mon[27254]: Adjusting osd_memory_target on smithi175 to 5296M 2024-09-11T03:37:28.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:28 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:28.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:28 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:37:28.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:28 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:37:28.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:28 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:28.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:28 smithi084 ceph-mon[27254]: osdmap e84: 12 total, 12 up, 12 in 2024-09-11T03:37:28.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:28 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3341104808' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-11T03:37:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:28 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:37:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:28 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:37:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:28 smithi121 ceph-mon[30729]: pgmap v252: 1 pgs: 1 peering; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:28 smithi121 ceph-mon[30729]: Detected new or changed devices on smithi175 2024-09-11T03:37:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:28 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:28 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:28 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:28 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:28 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:28 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:28.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:28 smithi121 ceph-mon[30729]: Adjusting osd_memory_target on smithi175 to 5296M 2024-09-11T03:37:28.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:28 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:28.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:28 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:37:28.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:28 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:37:28.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:28 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:28.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:28 smithi121 ceph-mon[30729]: osdmap e84: 12 total, 12 up, 12 in 2024-09-11T03:37:28.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:28 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3341104808' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-11T03:37:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:28 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:37:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:28 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:37:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:28 smithi175 ceph-mon[30894]: pgmap v252: 1 pgs: 1 peering; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:37:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:28 smithi175 ceph-mon[30894]: Detected new or changed devices on smithi175 2024-09-11T03:37:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:28 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:28 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:28 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:28 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:28 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:28 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-11T03:37:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:28 smithi175 ceph-mon[30894]: Adjusting osd_memory_target on smithi175 to 5296M 2024-09-11T03:37:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:28 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:28 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:37:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:28 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:37:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:28 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:37:28.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:28 smithi175 ceph-mon[30894]: osdmap e84: 12 total, 12 up, 12 in 2024-09-11T03:37:28.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:28 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3341104808' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-11T03:37:28.564 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":84,"num_osds":12,"num_up_osds":12,"osd_up_since":1726025846,"num_in_osds":12,"osd_in_since":1726025826,"num_remapped_pgs":0} 2024-09-11T03:37:28.565 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-11T03:37:28.810 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:29.400 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:37:29.400 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":85,"fsid":"10fe7946-6fee-11ef-bcea-c7b262605968","created":"2024-09-11T03:30:30.532652+0000","modified":"2024-09-11T03:37:28.790029+0000","last_up_change":"2024-09-11T03:37:26.787638+0000","last_in_change":"2024-09-11T03:37:06.874435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T03:33:42.916933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9a431e48-e46b-44f9-bfa7-cb56c1484e1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":1376667411},{"type":"v1","addr":"172.21.15.84:6803","nonce":1376667411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":1376667411},{"type":"v1","addr":"172.21.15.84:6805","nonce":1376667411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":1376667411},{"type":"v1","addr":"172.21.15.84:6809","nonce":1376667411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":1376667411},{"type":"v1","addr":"172.21.15.84:6807","nonce":1376667411}]},"public_addr":"172.21.15.84:6803/1376667411","cluster_addr":"172.21.15.84:6805/1376667411","heartbeat_back_addr":"172.21.15.84:6809/1376667411","heartbeat_front_addr":"172.21.15.84:6807/1376667411","state":["exists","up"]},{"osd":1,"uuid":"62fd6e25-3508-40b4-a0f4-f7de137e3bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6811","nonce":3138939249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6813","nonce":3138939249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6817","nonce":3138939249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6815","nonce":3138939249}]},"public_addr":"172.21.15.84:6811/3138939249","cluster_addr":"172.21.15.84:6813/3138939249","heartbeat_back_addr":"172.21.15.84:6817/3138939249","heartbeat_front_addr":"172.21.15.84:6815/3138939249","state":["exists","up"]},{"osd":2,"uuid":"ff351425-4715-49dc-be49-f27bf205d57e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6819","nonce":3001811591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6821","nonce":3001811591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6825","nonce":3001811591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6823","nonce":3001811591}]},"public_addr":"172.21.15.84:6819/3001811591","cluster_addr":"172.21.15.84:6821/3001811591","heartbeat_back_addr":"172.21.15.84:6825/3001811591","heartbeat_front_addr":"172.21.15.84:6823/3001811591","state":["exists","up"]},{"osd":3,"uuid":"2518a93e-c612-4731-91ce-5685c25ffa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6827","nonce":302512129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6829","nonce":302512129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6833","nonce":302512129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6831","nonce":302512129}]},"public_addr":"172.21.15.84:6827/302512129","cluster_addr":"172.21.15.84:6829/302512129","heartbeat_back_addr":"172.21.15.84:6833/302512129","heartbeat_front_addr":"172.21.15.84:6831/302512129","state":["exists","up"]},{"osd":4,"uuid":"d12f3c2d-95cc-4276-a7e1-bee982d58948","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6801","nonce":3873214513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6803","nonce":3873214513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6807","nonce":3873214513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6805","nonce":3873214513}]},"public_addr":"172.21.15.121:6801/3873214513","cluster_addr":"172.21.15.121:6803/3873214513","heartbeat_back_addr":"172.21.15.121:6807/3873214513","heartbeat_front_addr":"172.21.15.121:6805/3873214513","state":["exists","up"]},{"osd":5,"uuid":"32164dd5-7e8f-4925-8526-bbb1e33e3364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6809","nonce":918612194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6811","nonce":918612194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6815","nonce":918612194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6813","nonce":918612194}]},"public_addr":"172.21.15.121:6809/918612194","cluster_addr":"172.21.15.121:6811/918612194","heartbeat_back_addr":"172.21.15.121:6815/918612194","heartbeat_front_addr":"172.21.15.121:6813/918612194","state":["exists","up"]},{"osd":6,"uuid":"3f107a88-803f-4cd8-804a-241310c282ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6817","nonce":3384813363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6819","nonce":3384813363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6823","nonce":3384813363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6821","nonce":3384813363}]},"public_addr":"172.21.15.121:6817/3384813363","cluster_addr":"172.21.15.121:6819/3384813363","heartbeat_back_addr":"172.21.15.121:6823/3384813363","heartbeat_front_addr":"172.21.15.121:6821/3384813363","state":["exists","up"]},{"osd":7,"uuid":"dfee7f95-1fb9-4493-9907-cb10d1837bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6825","nonce":449222054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6827","nonce":449222054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6831","nonce":449222054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6829","nonce":449222054}]},"public_addr":"172.21.15.121:6825/449222054","cluster_addr":"172.21.15.121:6827/449222054","heartbeat_back_addr":"172.21.15.121:6831/449222054","heartbeat_front_addr":"172.21.15.121:6829/449222054","state":["exists","up"]},{"osd":8,"uuid":"fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6801","nonce":2660128384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6803","nonce":2660128384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6807","nonce":2660128384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6805","nonce":2660128384}]},"public_addr":"172.21.15.175:6801/2660128384","cluster_addr":"172.21.15.175:6803/2660128384","heartbeat_back_addr":"172.21.15.175:6807/2660128384","heartbeat_front_addr":"172.21.15.175:6805/2660128384","state":["exists","up"]},{"osd":9,"uuid":"50162e42-0443-41d1-8709-a0db61c087da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6809","nonce":1270991649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6811","nonce":1270991649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6815","nonce":1270991649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6813","nonce":1270991649}]},"public_addr":"172.21.15.175:6809/1270991649","cluster_addr":"172.21.15.175:6811/1270991649","heartbeat_back_addr":"172.21.15.175:6815/1270991649","heartbeat_front_addr":"172.21.15.175:6813/1270991649","state":["exists","up"]},{"osd":10,"uuid":"4b9426e1-bac8-47d3-ad58-54b89e8d1eb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6817","nonce":3877631021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6819","nonce":3877631021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6823","nonce":3877631021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6821","nonce":3877631021}]},"public_addr":"172.21.15.175:6817/3877631021","cluster_addr":"172.21.15.175:6819/3877631021","heartbeat_back_addr":"172.21.15.175:6823/3877631021","heartbeat_front_addr":"172.21.15.175:6821/3877631021","state":["exists","up"]},{"osd":11,"uuid":"cd1a28a1-413c-4274-9876-e1bdb7703be6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6825","nonce":1150051236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6827","nonce":1150051236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6831","nonce":1150051236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6829","nonce":1150051236}]},"public_addr":"172.21.15.175:6825/1150051236","cluster_addr":"172.21.15.175:6827/1150051236","heartbeat_back_addr":"172.21.15.175:6831/1150051236","heartbeat_front_addr":"172.21.15.175:6829/1150051236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:36:32.154181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:36:48.010110+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:09.404566+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:36:23.629999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:09.429563+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:19.860207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:36:26.791738+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:36:57.010017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:36:09.012399+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:36:35.723049+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:00.040996+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:25.945561+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/108136515":"2024-09-12T03:31:26.598965+0000","172.21.15.84:0/3411515187":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6801/3365920551":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6800/3365920551":"2024-09-12T03:31:26.598965+0000","172.21.15.84:0/3885577441":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/22039254":"2024-09-12T03:31:03.077186+0000","172.21.15.84:6801/3496645948":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/4206337830":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6800/3496645948":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/1989775875":"2024-09-12T03:31:03.077186+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T03:37:30.032 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:29 smithi084 ceph-mon[27254]: pgmap v255: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:30.032 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:29 smithi084 ceph-mon[27254]: osdmap e85: 12 total, 12 up, 12 in 2024-09-11T03:37:30.032 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:29 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1835860854' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:37:30.062 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T03:33:42.916933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-11T03:37:30.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-11T03:37:30.146 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:29 smithi175 ceph-mon[30894]: pgmap v255: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:30.147 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:29 smithi175 ceph-mon[30894]: osdmap e85: 12 total, 12 up, 12 in 2024-09-11T03:37:30.147 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:29 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1835860854' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:37:30.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:29 smithi121 ceph-mon[30729]: pgmap v255: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:29 smithi121 ceph-mon[30729]: osdmap e85: 12 total, 12 up, 12 in 2024-09-11T03:37:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:29 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1835860854' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:37:30.306 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:30.925 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 1 2024-09-11T03:37:31.577 INFO:tasks.cephadm:Setting up client nodes... 2024-09-11T03:37:31.577 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-11T03:37:31.818 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:31.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:31 smithi084 ceph-mon[27254]: pgmap v257: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:31.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:31 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2762803166' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-11T03:37:32.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:31 smithi121 ceph-mon[30729]: pgmap v257: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:31 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2762803166' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-11T03:37:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:31 smithi175 ceph-mon[30894]: pgmap v257: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:31 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2762803166' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-11T03:37:32.531 INFO:teuthology.orchestra.run.smithi084.stdout:[client.0] 2024-09-11T03:37:32.531 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQB8EOFmzj2hHxAAN2mamOrS4J/hqMemrhp7Sg== 2024-09-11T03:37:33.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:32 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4174001937' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-11T03:37:33.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:32 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4174001937' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T03:37:33.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:32 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4174001937' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-11T03:37:33.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:32 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4174001937' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T03:37:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:32 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4174001937' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-11T03:37:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:32 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4174001937' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T03:37:33.310 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-11T03:37:33.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-11T03:37:33.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-11T03:37:33.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-11T03:37:33.601 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:37:34.630 INFO:teuthology.orchestra.run.smithi121.stdout:[client.1] 2024-09-11T03:37:34.630 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQB+EOFmpPJTJRAAKJWK3ta3wc2NQR+kQB1MnQ== 2024-09-11T03:37:34.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:34 smithi084 ceph-mon[27254]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-11T03:37:34.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:34 smithi121 ceph-mon[30729]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-11T03:37:34.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:34 smithi175 ceph-mon[30894]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-11T03:37:35.395 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-11T03:37:35.395 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-11T03:37:35.396 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-11T03:37:35.434 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-11T03:37:35.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:35 smithi121 ceph-mon[30729]: from='client.? 172.21.15.121:0/3211088572' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-11T03:37:35.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:35 smithi121 ceph-mon[30729]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-11T03:37:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:35 smithi121 ceph-mon[30729]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T03:37:35.692 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:37:35.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:35 smithi175 ceph-mon[30894]: from='client.? 172.21.15.121:0/3211088572' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-11T03:37:35.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:35 smithi175 ceph-mon[30894]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-11T03:37:35.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:35 smithi175 ceph-mon[30894]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T03:37:35.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:35 smithi084 ceph-mon[27254]: from='client.? 172.21.15.121:0/3211088572' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-11T03:37:35.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:35 smithi084 ceph-mon[27254]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-11T03:37:35.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:35 smithi084 ceph-mon[27254]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T03:37:36.414 INFO:teuthology.orchestra.run.smithi175.stdout:[client.2] 2024-09-11T03:37:36.414 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCAEOFmp153GBAAxln08YxSHTi07qmd9tiJoQ== 2024-09-11T03:37:36.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:36 smithi175 ceph-mon[30894]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-11T03:37:36.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.175:0/1177948872' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-11T03:37:36.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:36 smithi175 ceph-mon[30894]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-11T03:37:36.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:36 smithi175 ceph-mon[30894]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T03:37:36.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:36 smithi084 ceph-mon[27254]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-11T03:37:36.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.175:0/1177948872' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-11T03:37:36.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:36 smithi084 ceph-mon[27254]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-11T03:37:36.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:36 smithi084 ceph-mon[27254]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T03:37:36.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:36 smithi121 ceph-mon[30729]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-11T03:37:36.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.175:0/1177948872' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-11T03:37:36.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:36 smithi121 ceph-mon[30729]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-11T03:37:36.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:36 smithi121 ceph-mon[30729]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T03:37:37.060 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-11T03:37:37.060 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-09-11T03:37:37.060 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-09-11T03:37:37.098 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph config log 1 --format=json 2024-09-11T03:37:37.351 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:38.005 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:37:38.714 INFO:teuthology.orchestra.run.smithi084.stdout:[{"version":27,"timestamp":"2024-09-11T03:37:27.130919+0000","name":"","changes":[{"name":"osd/host:smithi175/osd_memory_target","previous_value":"7404363912","new_value":"5553272934"}]}] 2024-09-11T03:37:38.714 INFO:tasks.ceph_manager:config epoch is 27 2024-09-11T03:37:38.715 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-11T03:37:38.715 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-11T03:37:38.715 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph mgr dump --format=json 2024-09-11T03:37:38.741 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:38 smithi084 ceph-mon[27254]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T03:37:38.741 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:38 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:37:38.742 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:38 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:37:38.742 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:38 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2374709468' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-11T03:37:38.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:38 smithi121 ceph-mon[30729]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T03:37:38.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:38 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:37:38.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:38 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:37:38.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:38 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2374709468' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-11T03:37:38.962 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:38.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:38 smithi175 ceph-mon[30894]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T03:37:38.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:38 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:37:38.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:38 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:37:38.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:38 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2374709468' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-11T03:37:39.683 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:37:39.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:39 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:37:39.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:39 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:37:39.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:39 smithi121 ceph-mon[30729]: osdmap e86: 12 total, 12 up, 12 in 2024-09-11T03:37:39.975 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:39 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:37:39.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:39 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:37:39.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:39 smithi084 ceph-mon[27254]: osdmap e86: 12 total, 12 up, 12 in 2024-09-11T03:37:39.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:39 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:37:39.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:39 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:37:39.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:39 smithi175 ceph-mon[30894]: osdmap e86: 12 total, 12 up, 12 in 2024-09-11T03:37:40.312 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":15,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6800","nonce":3467187755},{"type":"v1","addr":"172.21.15.84:6801","nonce":3467187755}]},"active_addr":"172.21.15.84:6801/3467187755","active_change":"2024-09-11T03:31:26.599072+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24107,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.84:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":4191878656}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":3948643139}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":1837830100}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":45231202}]}]} 2024-09-11T03:37:40.318 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-11T03:37:40.318 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-11T03:37:40.318 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-11T03:37:40.595 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:40.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:40 smithi121 ceph-mon[30729]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-11T03:37:40.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:40 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/739652674' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-11T03:37:40.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:40 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:37:40.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:40 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:37:40.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:40 smithi175 ceph-mon[30894]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-11T03:37:40.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:40 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/739652674' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-11T03:37:40.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:40 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:37:40.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:40 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:37:40.980 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:40 smithi084 ceph-mon[27254]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-11T03:37:40.980 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:40 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/739652674' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-11T03:37:40.980 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:40 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:37:40.980 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:40 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:37:41.245 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:37:41.246 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":87,"fsid":"10fe7946-6fee-11ef-bcea-c7b262605968","created":"2024-09-11T03:30:30.532652+0000","modified":"2024-09-11T03:37:40.647926+0000","last_up_change":"2024-09-11T03:37:26.787638+0000","last_in_change":"2024-09-11T03:37:06.874435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T03:33:42.916933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9a431e48-e46b-44f9-bfa7-cb56c1484e1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":1376667411},{"type":"v1","addr":"172.21.15.84:6803","nonce":1376667411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":1376667411},{"type":"v1","addr":"172.21.15.84:6805","nonce":1376667411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":1376667411},{"type":"v1","addr":"172.21.15.84:6809","nonce":1376667411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":1376667411},{"type":"v1","addr":"172.21.15.84:6807","nonce":1376667411}]},"public_addr":"172.21.15.84:6803/1376667411","cluster_addr":"172.21.15.84:6805/1376667411","heartbeat_back_addr":"172.21.15.84:6809/1376667411","heartbeat_front_addr":"172.21.15.84:6807/1376667411","state":["exists","up"]},{"osd":1,"uuid":"62fd6e25-3508-40b4-a0f4-f7de137e3bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6811","nonce":3138939249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6813","nonce":3138939249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6817","nonce":3138939249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6815","nonce":3138939249}]},"public_addr":"172.21.15.84:6811/3138939249","cluster_addr":"172.21.15.84:6813/3138939249","heartbeat_back_addr":"172.21.15.84:6817/3138939249","heartbeat_front_addr":"172.21.15.84:6815/3138939249","state":["exists","up"]},{"osd":2,"uuid":"ff351425-4715-49dc-be49-f27bf205d57e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6819","nonce":3001811591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6821","nonce":3001811591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6825","nonce":3001811591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6823","nonce":3001811591}]},"public_addr":"172.21.15.84:6819/3001811591","cluster_addr":"172.21.15.84:6821/3001811591","heartbeat_back_addr":"172.21.15.84:6825/3001811591","heartbeat_front_addr":"172.21.15.84:6823/3001811591","state":["exists","up"]},{"osd":3,"uuid":"2518a93e-c612-4731-91ce-5685c25ffa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6827","nonce":302512129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6829","nonce":302512129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6833","nonce":302512129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6831","nonce":302512129}]},"public_addr":"172.21.15.84:6827/302512129","cluster_addr":"172.21.15.84:6829/302512129","heartbeat_back_addr":"172.21.15.84:6833/302512129","heartbeat_front_addr":"172.21.15.84:6831/302512129","state":["exists","up"]},{"osd":4,"uuid":"d12f3c2d-95cc-4276-a7e1-bee982d58948","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6801","nonce":3873214513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6803","nonce":3873214513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6807","nonce":3873214513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6805","nonce":3873214513}]},"public_addr":"172.21.15.121:6801/3873214513","cluster_addr":"172.21.15.121:6803/3873214513","heartbeat_back_addr":"172.21.15.121:6807/3873214513","heartbeat_front_addr":"172.21.15.121:6805/3873214513","state":["exists","up"]},{"osd":5,"uuid":"32164dd5-7e8f-4925-8526-bbb1e33e3364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6809","nonce":918612194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6811","nonce":918612194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6815","nonce":918612194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6813","nonce":918612194}]},"public_addr":"172.21.15.121:6809/918612194","cluster_addr":"172.21.15.121:6811/918612194","heartbeat_back_addr":"172.21.15.121:6815/918612194","heartbeat_front_addr":"172.21.15.121:6813/918612194","state":["exists","up"]},{"osd":6,"uuid":"3f107a88-803f-4cd8-804a-241310c282ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6817","nonce":3384813363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6819","nonce":3384813363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6823","nonce":3384813363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6821","nonce":3384813363}]},"public_addr":"172.21.15.121:6817/3384813363","cluster_addr":"172.21.15.121:6819/3384813363","heartbeat_back_addr":"172.21.15.121:6823/3384813363","heartbeat_front_addr":"172.21.15.121:6821/3384813363","state":["exists","up"]},{"osd":7,"uuid":"dfee7f95-1fb9-4493-9907-cb10d1837bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6825","nonce":449222054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6827","nonce":449222054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6831","nonce":449222054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6829","nonce":449222054}]},"public_addr":"172.21.15.121:6825/449222054","cluster_addr":"172.21.15.121:6827/449222054","heartbeat_back_addr":"172.21.15.121:6831/449222054","heartbeat_front_addr":"172.21.15.121:6829/449222054","state":["exists","up"]},{"osd":8,"uuid":"fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6801","nonce":2660128384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6803","nonce":2660128384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6807","nonce":2660128384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6805","nonce":2660128384}]},"public_addr":"172.21.15.175:6801/2660128384","cluster_addr":"172.21.15.175:6803/2660128384","heartbeat_back_addr":"172.21.15.175:6807/2660128384","heartbeat_front_addr":"172.21.15.175:6805/2660128384","state":["exists","up"]},{"osd":9,"uuid":"50162e42-0443-41d1-8709-a0db61c087da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6809","nonce":1270991649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6811","nonce":1270991649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6815","nonce":1270991649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6813","nonce":1270991649}]},"public_addr":"172.21.15.175:6809/1270991649","cluster_addr":"172.21.15.175:6811/1270991649","heartbeat_back_addr":"172.21.15.175:6815/1270991649","heartbeat_front_addr":"172.21.15.175:6813/1270991649","state":["exists","up"]},{"osd":10,"uuid":"4b9426e1-bac8-47d3-ad58-54b89e8d1eb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6817","nonce":3877631021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6819","nonce":3877631021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6823","nonce":3877631021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6821","nonce":3877631021}]},"public_addr":"172.21.15.175:6817/3877631021","cluster_addr":"172.21.15.175:6819/3877631021","heartbeat_back_addr":"172.21.15.175:6823/3877631021","heartbeat_front_addr":"172.21.15.175:6821/3877631021","state":["exists","up"]},{"osd":11,"uuid":"cd1a28a1-413c-4274-9876-e1bdb7703be6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6825","nonce":1150051236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6827","nonce":1150051236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6831","nonce":1150051236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6829","nonce":1150051236}]},"public_addr":"172.21.15.175:6825/1150051236","cluster_addr":"172.21.15.175:6827/1150051236","heartbeat_back_addr":"172.21.15.175:6831/1150051236","heartbeat_front_addr":"172.21.15.175:6829/1150051236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:36:32.154181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:36:48.010110+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:09.404566+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:36:23.629999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:09.429563+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:19.860207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:39.929028+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:36:57.010017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:38.007862+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:37.741144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:00.040996+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:25.945561+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/108136515":"2024-09-12T03:31:26.598965+0000","172.21.15.84:0/3411515187":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6801/3365920551":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6800/3365920551":"2024-09-12T03:31:26.598965+0000","172.21.15.84:0/3885577441":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/22039254":"2024-09-12T03:31:03.077186+0000","172.21.15.84:6801/3496645948":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/4206337830":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6800/3496645948":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/1989775875":"2024-09-12T03:31:03.077186+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T03:37:41.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:41 smithi084 ceph-mon[27254]: osdmap e87: 12 total, 12 up, 12 in 2024-09-11T03:37:41.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:41 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2363859033' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:37:41.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:41 smithi121 ceph-mon[30729]: osdmap e87: 12 total, 12 up, 12 in 2024-09-11T03:37:41.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:41 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2363859033' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:37:41.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:41 smithi175 ceph-mon[30894]: osdmap e87: 12 total, 12 up, 12 in 2024-09-11T03:37:41.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:41 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2363859033' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:37:41.999 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-11T03:37:42.000 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-11T03:37:42.242 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:42.873 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:37:42.873 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":87,"fsid":"10fe7946-6fee-11ef-bcea-c7b262605968","created":"2024-09-11T03:30:30.532652+0000","modified":"2024-09-11T03:37:40.647926+0000","last_up_change":"2024-09-11T03:37:26.787638+0000","last_in_change":"2024-09-11T03:37:06.874435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T03:33:42.916933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9a431e48-e46b-44f9-bfa7-cb56c1484e1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":1376667411},{"type":"v1","addr":"172.21.15.84:6803","nonce":1376667411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":1376667411},{"type":"v1","addr":"172.21.15.84:6805","nonce":1376667411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":1376667411},{"type":"v1","addr":"172.21.15.84:6809","nonce":1376667411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":1376667411},{"type":"v1","addr":"172.21.15.84:6807","nonce":1376667411}]},"public_addr":"172.21.15.84:6803/1376667411","cluster_addr":"172.21.15.84:6805/1376667411","heartbeat_back_addr":"172.21.15.84:6809/1376667411","heartbeat_front_addr":"172.21.15.84:6807/1376667411","state":["exists","up"]},{"osd":1,"uuid":"62fd6e25-3508-40b4-a0f4-f7de137e3bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6811","nonce":3138939249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6813","nonce":3138939249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6817","nonce":3138939249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6815","nonce":3138939249}]},"public_addr":"172.21.15.84:6811/3138939249","cluster_addr":"172.21.15.84:6813/3138939249","heartbeat_back_addr":"172.21.15.84:6817/3138939249","heartbeat_front_addr":"172.21.15.84:6815/3138939249","state":["exists","up"]},{"osd":2,"uuid":"ff351425-4715-49dc-be49-f27bf205d57e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6819","nonce":3001811591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6821","nonce":3001811591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6825","nonce":3001811591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6823","nonce":3001811591}]},"public_addr":"172.21.15.84:6819/3001811591","cluster_addr":"172.21.15.84:6821/3001811591","heartbeat_back_addr":"172.21.15.84:6825/3001811591","heartbeat_front_addr":"172.21.15.84:6823/3001811591","state":["exists","up"]},{"osd":3,"uuid":"2518a93e-c612-4731-91ce-5685c25ffa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6827","nonce":302512129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6829","nonce":302512129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6833","nonce":302512129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6831","nonce":302512129}]},"public_addr":"172.21.15.84:6827/302512129","cluster_addr":"172.21.15.84:6829/302512129","heartbeat_back_addr":"172.21.15.84:6833/302512129","heartbeat_front_addr":"172.21.15.84:6831/302512129","state":["exists","up"]},{"osd":4,"uuid":"d12f3c2d-95cc-4276-a7e1-bee982d58948","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6801","nonce":3873214513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6803","nonce":3873214513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6807","nonce":3873214513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6805","nonce":3873214513}]},"public_addr":"172.21.15.121:6801/3873214513","cluster_addr":"172.21.15.121:6803/3873214513","heartbeat_back_addr":"172.21.15.121:6807/3873214513","heartbeat_front_addr":"172.21.15.121:6805/3873214513","state":["exists","up"]},{"osd":5,"uuid":"32164dd5-7e8f-4925-8526-bbb1e33e3364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6809","nonce":918612194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6811","nonce":918612194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6815","nonce":918612194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6813","nonce":918612194}]},"public_addr":"172.21.15.121:6809/918612194","cluster_addr":"172.21.15.121:6811/918612194","heartbeat_back_addr":"172.21.15.121:6815/918612194","heartbeat_front_addr":"172.21.15.121:6813/918612194","state":["exists","up"]},{"osd":6,"uuid":"3f107a88-803f-4cd8-804a-241310c282ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6817","nonce":3384813363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6819","nonce":3384813363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6823","nonce":3384813363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6821","nonce":3384813363}]},"public_addr":"172.21.15.121:6817/3384813363","cluster_addr":"172.21.15.121:6819/3384813363","heartbeat_back_addr":"172.21.15.121:6823/3384813363","heartbeat_front_addr":"172.21.15.121:6821/3384813363","state":["exists","up"]},{"osd":7,"uuid":"dfee7f95-1fb9-4493-9907-cb10d1837bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6825","nonce":449222054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6827","nonce":449222054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6831","nonce":449222054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6829","nonce":449222054}]},"public_addr":"172.21.15.121:6825/449222054","cluster_addr":"172.21.15.121:6827/449222054","heartbeat_back_addr":"172.21.15.121:6831/449222054","heartbeat_front_addr":"172.21.15.121:6829/449222054","state":["exists","up"]},{"osd":8,"uuid":"fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6801","nonce":2660128384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6803","nonce":2660128384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6807","nonce":2660128384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6805","nonce":2660128384}]},"public_addr":"172.21.15.175:6801/2660128384","cluster_addr":"172.21.15.175:6803/2660128384","heartbeat_back_addr":"172.21.15.175:6807/2660128384","heartbeat_front_addr":"172.21.15.175:6805/2660128384","state":["exists","up"]},{"osd":9,"uuid":"50162e42-0443-41d1-8709-a0db61c087da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6809","nonce":1270991649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6811","nonce":1270991649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6815","nonce":1270991649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6813","nonce":1270991649}]},"public_addr":"172.21.15.175:6809/1270991649","cluster_addr":"172.21.15.175:6811/1270991649","heartbeat_back_addr":"172.21.15.175:6815/1270991649","heartbeat_front_addr":"172.21.15.175:6813/1270991649","state":["exists","up"]},{"osd":10,"uuid":"4b9426e1-bac8-47d3-ad58-54b89e8d1eb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6817","nonce":3877631021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6819","nonce":3877631021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6823","nonce":3877631021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6821","nonce":3877631021}]},"public_addr":"172.21.15.175:6817/3877631021","cluster_addr":"172.21.15.175:6819/3877631021","heartbeat_back_addr":"172.21.15.175:6823/3877631021","heartbeat_front_addr":"172.21.15.175:6821/3877631021","state":["exists","up"]},{"osd":11,"uuid":"cd1a28a1-413c-4274-9876-e1bdb7703be6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6825","nonce":1150051236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6827","nonce":1150051236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6831","nonce":1150051236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6829","nonce":1150051236}]},"public_addr":"172.21.15.175:6825/1150051236","cluster_addr":"172.21.15.175:6827/1150051236","heartbeat_back_addr":"172.21.15.175:6831/1150051236","heartbeat_front_addr":"172.21.15.175:6829/1150051236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:36:32.154181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:36:48.010110+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:09.404566+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:36:23.629999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:09.429563+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:19.860207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:39.929028+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:36:57.010017+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:38.007862+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:37.741144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:00.040996+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:37:25.945561+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/108136515":"2024-09-12T03:31:26.598965+0000","172.21.15.84:0/3411515187":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6801/3365920551":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6800/3365920551":"2024-09-12T03:31:26.598965+0000","172.21.15.84:0/3885577441":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/22039254":"2024-09-12T03:31:03.077186+0000","172.21.15.84:6801/3496645948":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/4206337830":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6800/3496645948":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/1989775875":"2024-09-12T03:31:03.077186+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T03:37:42.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:42 smithi121 ceph-mon[30729]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:42.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:42 smithi121 ceph-mon[30729]: 1.0 scrub starts 2024-09-11T03:37:42.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:42 smithi121 ceph-mon[30729]: 1.0 scrub ok 2024-09-11T03:37:42.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:42 smithi175 ceph-mon[30894]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:42.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:42 smithi175 ceph-mon[30894]: 1.0 scrub starts 2024-09-11T03:37:42.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:42 smithi175 ceph-mon[30894]: 1.0 scrub ok 2024-09-11T03:37:43.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:42 smithi084 ceph-mon[27254]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:43.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:42 smithi084 ceph-mon[27254]: 1.0 scrub starts 2024-09-11T03:37:43.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:42 smithi084 ceph-mon[27254]: 1.0 scrub ok 2024-09-11T03:37:43.629 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-11T03:37:43.630 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-11T03:37:43.630 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-11T03:37:43.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-11T03:37:43.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-11T03:37:43.632 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-11T03:37:43.632 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-11T03:37:43.633 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-11T03:37:43.633 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-09-11T03:37:43.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-09-11T03:37:43.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-09-11T03:37:43.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-09-11T03:37:43.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:43 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/612574345' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:37:43.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:43 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/612574345' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:37:43.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:43 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/612574345' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:37:44.299 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:44.395 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:44.396 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:44.396 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:44.397 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:44.400 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:44.400 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:44.401 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:44.631 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:44.710 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:44 smithi084 ceph-mon[27254]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:44.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:44 smithi175 ceph-mon[30894]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:44.998 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:44 smithi121 ceph-mon[30729]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:45.822 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:45 smithi084 ceph-mon[27254]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:45.823 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:45 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:37:45.823 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:45 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:37:45.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:45 smithi121 ceph-mon[30729]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:45.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:45 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:37:45.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:45 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:37:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:45 smithi175 ceph-mon[30894]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:45 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:37:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:45 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:37:47.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:46 smithi084 ceph-mon[27254]: osdmap e88: 12 total, 12 up, 12 in 2024-09-11T03:37:47.160 INFO:teuthology.orchestra.run.smithi084.stdout:193273528351 2024-09-11T03:37:47.160 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-11T03:37:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:46 smithi121 ceph-mon[30729]: osdmap e88: 12 total, 12 up, 12 in 2024-09-11T03:37:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:46 smithi175 ceph-mon[30894]: osdmap e88: 12 total, 12 up, 12 in 2024-09-11T03:37:47.862 INFO:teuthology.orchestra.run.smithi084.stdout:356482285574 2024-09-11T03:37:47.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-09-11T03:37:48.045 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:47 smithi084 ceph-mon[27254]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:48.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:47 smithi121 ceph-mon[30729]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:48.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:47 smithi175 ceph-mon[30894]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:50.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:49 smithi084 ceph-mon[27254]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:50.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:49 smithi121 ceph-mon[30729]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:49 smithi175 ceph-mon[30894]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:50.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:50 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:37:50.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:50 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:37:50.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:50 smithi084 ceph-mon[27254]: osdmap e89: 12 total, 12 up, 12 in 2024-09-11T03:37:51.061 INFO:teuthology.orchestra.run.smithi084.stdout:133143986218 2024-09-11T03:37:51.061 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-11T03:37:51.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:50 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:37:51.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:50 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:37:51.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:50 smithi121 ceph-mon[30729]: osdmap e89: 12 total, 12 up, 12 in 2024-09-11T03:37:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:50 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:37:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:50 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:37:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:50 smithi175 ceph-mon[30894]: osdmap e89: 12 total, 12 up, 12 in 2024-09-11T03:37:51.570 INFO:teuthology.orchestra.run.smithi084.stdout:77309411381 2024-09-11T03:37:51.570 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-11T03:37:51.802 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:51 smithi084 ceph-mon[27254]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:51.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:51 smithi175 ceph-mon[30894]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:52.123 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:51 smithi121 ceph-mon[30729]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:52.516 INFO:teuthology.orchestra.run.smithi084.stdout:257698037784 2024-09-11T03:37:52.516 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-09-11T03:37:52.957 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:52.959 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:53.482 INFO:teuthology.orchestra.run.smithi084.stdout:296352743441 2024-09-11T03:37:53.482 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-09-11T03:37:53.908 INFO:teuthology.orchestra.run.smithi084.stdout:34359738433 2024-09-11T03:37:53.908 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-11T03:37:54.010 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:53 smithi084 ceph-mon[27254]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:53 smithi121 ceph-mon[30729]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:53 smithi175 ceph-mon[30894]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:55.530 INFO:teuthology.orchestra.run.smithi084.stdout:326417514508 2024-09-11T03:37:55.530 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-09-11T03:37:56.045 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:55 smithi084 ceph-mon[27254]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:56.150 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:56.151 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:56.151 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:37:56.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:55 smithi121 ceph-mon[30729]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:55 smithi175 ceph-mon[30894]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:56.948 INFO:teuthology.orchestra.run.smithi084.stdout:163208757285 2024-09-11T03:37:56.948 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-11T03:37:57.348 INFO:teuthology.orchestra.run.smithi084.stdout:111669149745 2024-09-11T03:37:57.349 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-11T03:37:57.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:57 smithi084 ceph-mon[27254]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:58.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:57 smithi121 ceph-mon[30729]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:58.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:57 smithi175 ceph-mon[30894]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:58.421 INFO:teuthology.orchestra.run.smithi084.stdout:55834574906 2024-09-11T03:37:58.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-11T03:37:58.806 INFO:teuthology.orchestra.run.smithi084.stdout:193273528354 2024-09-11T03:37:59.042 INFO:teuthology.orchestra.run.smithi084.stdout:356482285577 2024-09-11T03:37:59.545 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528351 got 193273528354 for osd.6 2024-09-11T03:37:59.545 DEBUG:teuthology.parallel:result is None 2024-09-11T03:37:59.780 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:59 smithi084 ceph-mon[27254]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:37:59.780 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:59 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3601613832' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-11T03:37:59.780 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:37:59 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1187408481' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-11T03:38:00.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:59 smithi121 ceph-mon[30729]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:00.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:59 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3601613832' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-11T03:38:00.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:37:59 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1187408481' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-11T03:38:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:59 smithi175 ceph-mon[30894]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:59 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3601613832' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-11T03:38:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:37:59 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1187408481' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-11T03:38:00.516 INFO:tasks.cephadm.ceph_manager.ceph:need seq 356482285574 got 356482285577 for osd.11 2024-09-11T03:38:00.516 DEBUG:teuthology.parallel:result is None 2024-09-11T03:38:00.878 INFO:teuthology.orchestra.run.smithi084.stdout:77309411383 2024-09-11T03:38:01.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:38:01.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:38:01.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:38:01.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:38:01.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:38:01.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:38:01.069 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:38:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:38:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:38:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:38:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:38:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:38:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:38:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:38:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:38:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:38:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:38:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:38:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:38:01.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:38:01.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:38:01.405 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:01.405 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:01.406 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:01.456 INFO:teuthology.orchestra.run.smithi084.stdout:223338299420 2024-09-11T03:38:01.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-11T03:38:01.958 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411381 got 77309411383 for osd.2 2024-09-11T03:38:01.958 DEBUG:teuthology.parallel:result is None 2024-09-11T03:38:01.994 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:01.994 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:01.994 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:01.998 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:02.077 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:01 smithi084 ceph-mon[27254]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:02.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3576344810' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-11T03:38:02.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:01 smithi121 ceph-mon[30729]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:02.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3576344810' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-11T03:38:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:01 smithi175 ceph-mon[30894]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3576344810' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-11T03:38:02.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:02 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:38:02.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:02 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:38:03.052 INFO:teuthology.orchestra.run.smithi084.stdout:326417514510 2024-09-11T03:38:03.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:02 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:38:03.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:02 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:38:03.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:02 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:38:03.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:02 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:38:03.923 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:03 smithi084 ceph-mon[27254]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:03.923 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:03 smithi084 ceph-mon[27254]: osdmap e90: 12 total, 12 up, 12 in 2024-09-11T03:38:03.923 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:03 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1686849830' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-11T03:38:03.954 INFO:teuthology.orchestra.run.smithi084.stdout:55834574909 2024-09-11T03:38:04.037 INFO:teuthology.orchestra.run.smithi084.stdout:257698037786 2024-09-11T03:38:04.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:03 smithi121 ceph-mon[30729]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:03 smithi121 ceph-mon[30729]: osdmap e90: 12 total, 12 up, 12 in 2024-09-11T03:38:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:03 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1686849830' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-11T03:38:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:03 smithi175 ceph-mon[30894]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:03 smithi175 ceph-mon[30894]: osdmap e90: 12 total, 12 up, 12 in 2024-09-11T03:38:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:03 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1686849830' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-11T03:38:04.804 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037784 got 257698037786 for osd.8 2024-09-11T03:38:04.804 DEBUG:teuthology.parallel:result is None 2024-09-11T03:38:04.958 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:05.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:04 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1887102082' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-11T03:38:05.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:04 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3448501843' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-11T03:38:05.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:04 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:38:05.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:04 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:38:05.164 INFO:teuthology.orchestra.run.smithi084.stdout:296352743444 2024-09-11T03:38:05.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:04 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1887102082' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-11T03:38:05.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:04 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3448501843' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-11T03:38:05.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:04 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:38:05.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:04 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:38:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:04 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1887102082' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-11T03:38:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:04 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3448501843' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-11T03:38:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:04 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:38:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:04 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:38:05.525 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574906 got 55834574909 for osd.1 2024-09-11T03:38:05.526 DEBUG:teuthology.parallel:result is None 2024-09-11T03:38:05.891 INFO:teuthology.orchestra.run.smithi084.stdout:34359738435 2024-09-11T03:38:06.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:05 smithi084 ceph-mon[27254]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:06.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:05 smithi084 ceph-mon[27254]: osdmap e91: 12 total, 12 up, 12 in 2024-09-11T03:38:06.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:05 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3329490648' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-11T03:38:06.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:05 smithi121 ceph-mon[30729]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:06.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:05 smithi121 ceph-mon[30729]: osdmap e91: 12 total, 12 up, 12 in 2024-09-11T03:38:06.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:05 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3329490648' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-11T03:38:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:05 smithi175 ceph-mon[30894]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:05 smithi175 ceph-mon[30894]: osdmap e91: 12 total, 12 up, 12 in 2024-09-11T03:38:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:05 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3329490648' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-11T03:38:06.657 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738433 got 34359738435 for osd.0 2024-09-11T03:38:06.657 DEBUG:teuthology.parallel:result is None 2024-09-11T03:38:06.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:06 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4125145771' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-11T03:38:07.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:06 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4125145771' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-11T03:38:07.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:06 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4125145771' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-11T03:38:07.246 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417514508 got 326417514510 for osd.10 2024-09-11T03:38:07.246 DEBUG:teuthology.parallel:result is None 2024-09-11T03:38:08.001 INFO:teuthology.orchestra.run.smithi084.stdout:111669149747 2024-09-11T03:38:08.030 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:07 smithi084 ceph-mon[27254]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:08.073 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:07 smithi121 ceph-mon[30729]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:08.203 INFO:teuthology.orchestra.run.smithi084.stdout:133143986222 2024-09-11T03:38:08.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:07 smithi175 ceph-mon[30894]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:08.797 INFO:teuthology.orchestra.run.smithi084.stdout:163208757289 2024-09-11T03:38:09.038 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:08 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/495280936' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-11T03:38:09.038 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:08 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1791555408' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-11T03:38:09.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:08 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/495280936' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-11T03:38:09.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:08 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1791555408' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-11T03:38:09.183 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149745 got 111669149747 for osd.3 2024-09-11T03:38:09.184 DEBUG:teuthology.parallel:result is None 2024-09-11T03:38:09.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:08 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/495280936' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-11T03:38:09.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:08 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1791555408' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-11T03:38:09.759 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757285 got 163208757289 for osd.5 2024-09-11T03:38:09.760 DEBUG:teuthology.parallel:result is None 2024-09-11T03:38:10.128 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:09 smithi175 ceph-mon[30894]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:10.129 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:09 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2511819016' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-11T03:38:10.134 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:09 smithi084 ceph-mon[27254]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:10.134 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:09 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2511819016' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-11T03:38:10.161 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986218 got 133143986222 for osd.4 2024-09-11T03:38:10.161 DEBUG:teuthology.parallel:result is None 2024-09-11T03:38:10.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:09 smithi121 ceph-mon[30729]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:10.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:09 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2511819016' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-11T03:38:10.188 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:10.597 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743441 got 296352743444 for osd.9 2024-09-11T03:38:10.598 DEBUG:teuthology.parallel:result is None 2024-09-11T03:38:11.266 INFO:teuthology.orchestra.run.smithi084.stdout:223338299423 2024-09-11T03:38:12.015 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:11 smithi084 ceph-mon[27254]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:12.015 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:11 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2317944628' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-11T03:38:12.016 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299420 got 223338299423 for osd.7 2024-09-11T03:38:12.016 DEBUG:teuthology.parallel:result is None 2024-09-11T03:38:12.017 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-11T03:38:12.017 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:38:12.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:11 smithi121 ceph-mon[30729]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:12.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:11 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2317944628' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-11T03:38:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:11 smithi175 ceph-mon[30894]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:11 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2317944628' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-11T03:38:12.262 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:12.916 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:38:12.916 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:38:13.650 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2024-09-11T03:38:12.675905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":337372,"kb_used_data":13032,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124604452,"statfs":{"total":1151940427776,"available":1151594958848,"internally_reserved":0,"allocated":13344768,"data_stored":7196160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":331855163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.911146"},"pg_stats":[{"pgid":"1.0","version":"20'92","reported_seq":131,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-11T03:38:05.364116+0000","last_change":"2024-09-11T03:37:42.091504+0000","last_active":"2024-09-11T03:38:05.364116+0000","last_peered":"2024-09-11T03:38:05.364116+0000","last_clean":"2024-09-11T03:38:05.364116+0000","last_became_active":"2024-09-11T03:37:28.922429+0000","last_became_peered":"2024-09-11T03:37:28.922429+0000","last_unstale":"2024-09-11T03:38:05.364116+0000","last_undegraded":"2024-09-11T03:38:05.364116+0000","last_fullsized":"2024-09-11T03:38:05.364116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:38:58.309199+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285580,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28504,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716648,"statfs":{"total":95995035648,"available":95965847552,"internally_reserved":0,"allocated":1511424,"data_stored":996376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514512,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037788,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299424,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28504,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716648,"statfs":{"total":95995035648,"available":95965847552,"internally_reserved":0,"allocated":1511424,"data_stored":996376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528357,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28504,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716648,"statfs":{"total":95995035648,"available":95965847552,"internally_reserved":0,"allocated":1511424,"data_stored":996376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757290,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27988,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:38:13.650 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:38:13.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:13 smithi084 ceph-mon[27254]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:13.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:13 smithi084 ceph-mon[27254]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:38:13.896 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:14.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:13 smithi121 ceph-mon[30729]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:14.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:13 smithi121 ceph-mon[30729]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:38:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:13 smithi175 ceph-mon[30894]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:13 smithi175 ceph-mon[30894]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:38:14.515 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:38:14.515 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:38:15.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:14 smithi084 ceph-mon[27254]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:38:15.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:14 smithi121 ceph-mon[30729]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:38:15.203 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2024-09-11T03:38:12.675905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":337372,"kb_used_data":13032,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124604452,"statfs":{"total":1151940427776,"available":1151594958848,"internally_reserved":0,"allocated":13344768,"data_stored":7196160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":331855163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.911146"},"pg_stats":[{"pgid":"1.0","version":"20'92","reported_seq":131,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-11T03:38:05.364116+0000","last_change":"2024-09-11T03:37:42.091504+0000","last_active":"2024-09-11T03:38:05.364116+0000","last_peered":"2024-09-11T03:38:05.364116+0000","last_clean":"2024-09-11T03:38:05.364116+0000","last_became_active":"2024-09-11T03:37:28.922429+0000","last_became_peered":"2024-09-11T03:37:28.922429+0000","last_unstale":"2024-09-11T03:38:05.364116+0000","last_undegraded":"2024-09-11T03:38:05.364116+0000","last_fullsized":"2024-09-11T03:38:05.364116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:38:58.309199+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285580,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28504,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716648,"statfs":{"total":95995035648,"available":95965847552,"internally_reserved":0,"allocated":1511424,"data_stored":996376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514512,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037788,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299424,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28504,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716648,"statfs":{"total":95995035648,"available":95965847552,"internally_reserved":0,"allocated":1511424,"data_stored":996376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528357,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28504,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716648,"statfs":{"total":95995035648,"available":95965847552,"internally_reserved":0,"allocated":1511424,"data_stored":996376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757290,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27988,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:38:15.204 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-11T03:38:15.204 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-11T03:38:15.204 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-11T03:38:15.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph health --format=json 2024-09-11T03:38:15.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:14 smithi175 ceph-mon[30894]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:38:15.448 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:15.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:15 smithi084 ceph-mon[27254]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:16.101 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:38:16.101 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T03:38:16.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:15 smithi121 ceph-mon[30729]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:16.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:15 smithi175 ceph-mon[30894]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:16.849 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:16 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:38:16.849 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:16 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:38:16.849 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:16 smithi084 ceph-mon[27254]: osdmap e92: 12 total, 12 up, 12 in 2024-09-11T03:38:16.849 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:16 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2091492324' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-11T03:38:16.878 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-11T03:38:16.879 INFO:tasks.cephadm:Setup complete, yielding 2024-09-11T03:38:16.879 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-11T03:38:16.890 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 3, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-11T03:38:16.890 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-hemanth-testing-2024-09-10-0723'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'filestore queue throttle high multiple': 2, 'filestore queue throttle max multiple': 10, 'journal throttle high multiple': 2, 'journal throttle max multiple': 10, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd heartbeat use min delay socket': True, 'osd max backfills': 9, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-hemanth-testing-2024-09-10-0723', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}} 2024-09-11T03:38:16.890 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 3, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'min_in': 4} 2024-09-11T03:38:16.890 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-11T03:38:17.131 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:17.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:16 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:38:17.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:16 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:38:17.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:16 smithi121 ceph-mon[30729]: osdmap e92: 12 total, 12 up, 12 in 2024-09-11T03:38:17.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:16 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2091492324' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-11T03:38:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:16 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:38:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:16 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:38:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:16 smithi175 ceph-mon[30894]: osdmap e92: 12 total, 12 up, 12 in 2024-09-11T03:38:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:16 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2091492324' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-11T03:38:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:17 smithi084 ceph-mon[27254]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:18.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:17 smithi121 ceph-mon[30729]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:18.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:17 smithi175 ceph-mon[30894]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:18.607 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-11T03:38:18.607 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-11T03:38:18.607 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:38:18.850 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:19.106 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:18 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2280270132' entity='client.admin' 2024-09-11T03:38:19.106 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:38:19.106 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:38:19.106 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:38:19.106 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:38:19.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:18 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2280270132' entity='client.admin' 2024-09-11T03:38:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:38:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:38:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:38:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:38:19.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:18 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2280270132' entity='client.admin' 2024-09-11T03:38:19.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:38:19.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:38:19.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:38:19.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:38:19.484 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:38:19.485 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:38:20.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:19 smithi084 ceph-mon[27254]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:20.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:19 smithi084 ceph-mon[27254]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:38:20.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:19 smithi121 ceph-mon[30729]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:20.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:19 smithi121 ceph-mon[30729]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:38:20.185 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-09-11T03:38:18.677078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":337444,"kb_used_data":13104,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124604380,"statfs":{"total":1151940427776,"available":1151594885120,"internally_reserved":0,"allocated":13418496,"data_stored":7240422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":331855163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001953"},"pg_stats":[{"pgid":"1.0","version":"20'92","reported_seq":133,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-11T03:38:15.810851+0000","last_change":"2024-09-11T03:37:42.091504+0000","last_active":"2024-09-11T03:38:15.810851+0000","last_peered":"2024-09-11T03:38:15.810851+0000","last_clean":"2024-09-11T03:38:15.810851+0000","last_became_active":"2024-09-11T03:37:28.922429+0000","last_became_peered":"2024-09-11T03:37:28.922429+0000","last_unstale":"2024-09-11T03:38:15.810851+0000","last_undegraded":"2024-09-11T03:38:15.810851+0000","last_fullsized":"2024-09-11T03:38:15.810851+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:38:58.309199+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285581,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28516,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716636,"statfs":{"total":95995035648,"available":95965835264,"internally_reserved":0,"allocated":1523712,"data_stored":1003753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514513,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":474825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037789,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":474825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299425,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28504,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716648,"statfs":{"total":95995035648,"available":95965847552,"internally_reserved":0,"allocated":1511424,"data_stored":996376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528358,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28516,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716636,"statfs":{"total":95995035648,"available":95965835264,"internally_reserved":0,"allocated":1523712,"data_stored":1003753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986224,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27988,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":474825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":474825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:38:20.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:38:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:19 smithi175 ceph-mon[30894]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:19 smithi175 ceph-mon[30894]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:38:20.428 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:21.050 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:38:21.050 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:38:21.688 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-09-11T03:38:20.692826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":337464,"kb_used_data":13128,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124604360,"statfs":{"total":1151940427776,"available":1151594864640,"internally_reserved":0,"allocated":13443072,"data_stored":7255176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":331855163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.017357"},"pg_stats":[{"pgid":"1.0","version":"20'92","reported_seq":133,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-11T03:38:15.810851+0000","last_change":"2024-09-11T03:37:42.091504+0000","last_active":"2024-09-11T03:38:15.810851+0000","last_peered":"2024-09-11T03:38:15.810851+0000","last_clean":"2024-09-11T03:38:15.810851+0000","last_became_active":"2024-09-11T03:37:28.922429+0000","last_became_peered":"2024-09-11T03:37:28.922429+0000","last_unstale":"2024-09-11T03:38:15.810851+0000","last_undegraded":"2024-09-11T03:38:15.810851+0000","last_fullsized":"2024-09-11T03:38:15.810851+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:38:58.309199+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285581,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28516,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716636,"statfs":{"total":95995035648,"available":95965835264,"internally_reserved":0,"allocated":1523712,"data_stored":1003753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514513,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":474825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037789,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":474825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299425,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28504,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716648,"statfs":{"total":95995035648,"available":95965847552,"internally_reserved":0,"allocated":1511424,"data_stored":996376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528358,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28516,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716636,"statfs":{"total":95995035648,"available":95965835264,"internally_reserved":0,"allocated":1523712,"data_stored":1003753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986225,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":474825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":474825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":474825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":474825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27984,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":978944,"data_stored":467448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:38:21.689 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-11T03:38:21.689 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd dump 2024-09-11T03:38:21.931 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:22.052 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:21 smithi084 ceph-mon[27254]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:22.052 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:21 smithi084 ceph-mon[27254]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:38:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:21 smithi121 ceph-mon[30729]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:21 smithi121 ceph-mon[30729]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:38:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:21 smithi175 ceph-mon[30894]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:21 smithi175 ceph-mon[30894]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:38:22.570 INFO:teuthology.orchestra.run.smithi084.stdout:epoch 92 2024-09-11T03:38:22.570 INFO:teuthology.orchestra.run.smithi084.stdout:fsid 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:38:22.570 INFO:teuthology.orchestra.run.smithi084.stdout:created 2024-09-11T03:30:30.532652+0000 2024-09-11T03:38:22.571 INFO:teuthology.orchestra.run.smithi084.stdout:modified 2024-09-11T03:38:15.803357+0000 2024-09-11T03:38:22.571 INFO:teuthology.orchestra.run.smithi084.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-11T03:38:22.571 INFO:teuthology.orchestra.run.smithi084.stdout:crush_version 26 2024-09-11T03:38:22.571 INFO:teuthology.orchestra.run.smithi084.stdout:full_ratio 0.95 2024-09-11T03:38:22.571 INFO:teuthology.orchestra.run.smithi084.stdout:backfillfull_ratio 0.9 2024-09-11T03:38:22.571 INFO:teuthology.orchestra.run.smithi084.stdout:nearfull_ratio 0.85 2024-09-11T03:38:22.571 INFO:teuthology.orchestra.run.smithi084.stdout:require_min_compat_client luminous 2024-09-11T03:38:22.571 INFO:teuthology.orchestra.run.smithi084.stdout:min_compat_client jewel 2024-09-11T03:38:22.571 INFO:teuthology.orchestra.run.smithi084.stdout:require_osd_release squid 2024-09-11T03:38:22.571 INFO:teuthology.orchestra.run.smithi084.stdout:stretch_mode_enabled false 2024-09-11T03:38:22.571 INFO:teuthology.orchestra.run.smithi084.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-09-11T03:38:22.571 INFO:teuthology.orchestra.run.smithi084.stdout:max_osd 12 2024-09-11T03:38:22.571 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0 up in weight 1 up_from 8 up_thru 52 down_at 0 last_clean_interval [0,0) [v2:172.21.15.84:6802/1376667411,v1:172.21.15.84:6803/1376667411] [v2:172.21.15.84:6804/1376667411,v1:172.21.15.84:6805/1376667411] exists,up 9a431e48-e46b-44f9-bfa7-cb56c1484e1c 2024-09-11T03:38:22.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1 up in weight 1 up_from 13 up_thru 32 down_at 0 last_clean_interval [0,0) [v2:172.21.15.84:6810/3138939249,v1:172.21.15.84:6811/3138939249] [v2:172.21.15.84:6812/3138939249,v1:172.21.15.84:6813/3138939249] exists,up 62fd6e25-3508-40b4-a0f4-f7de137e3bb5 2024-09-11T03:38:22.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.84:6818/3001811591,v1:172.21.15.84:6819/3001811591] [v2:172.21.15.84:6820/3001811591,v1:172.21.15.84:6821/3001811591] exists,up ff351425-4715-49dc-be49-f27bf205d57e 2024-09-11T03:38:22.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129] [v2:172.21.15.84:6828/302512129,v1:172.21.15.84:6829/302512129] exists,up 2518a93e-c612-4731-91ce-5685c25ffa9f 2024-09-11T03:38:22.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6800/3873214513,v1:172.21.15.121:6801/3873214513] [v2:172.21.15.121:6802/3873214513,v1:172.21.15.121:6803/3873214513] exists,up d12f3c2d-95cc-4276-a7e1-bee982d58948 2024-09-11T03:38:22.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5 up in weight 1 up_from 38 up_thru 39 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6808/918612194,v1:172.21.15.121:6809/918612194] [v2:172.21.15.121:6810/918612194,v1:172.21.15.121:6811/918612194] exists,up 32164dd5-7e8f-4925-8526-bbb1e33e3364 2024-09-11T03:38:22.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6 up in weight 1 up_from 45 up_thru 46 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6816/3384813363,v1:172.21.15.121:6817/3384813363] [v2:172.21.15.121:6818/3384813363,v1:172.21.15.121:6819/3384813363] exists,up 3f107a88-803f-4cd8-804a-241310c282ea 2024-09-11T03:38:22.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7 up in weight 1 up_from 52 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6824/449222054,v1:172.21.15.121:6825/449222054] [v2:172.21.15.121:6826/449222054,v1:172.21.15.121:6827/449222054] exists,up dfee7f95-1fb9-4493-9907-cb10d1837bdb 2024-09-11T03:38:22.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8 up in weight 1 up_from 60 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.175:6800/2660128384,v1:172.21.15.175:6801/2660128384] [v2:172.21.15.175:6802/2660128384,v1:172.21.15.175:6803/2660128384] exists,up fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4 2024-09-11T03:38:22.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9 up in weight 1 up_from 69 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.175:6808/1270991649,v1:172.21.15.175:6809/1270991649] [v2:172.21.15.175:6810/1270991649,v1:172.21.15.175:6811/1270991649] exists,up 50162e42-0443-41d1-8709-a0db61c087da 2024-09-11T03:38:22.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10 up in weight 1 up_from 76 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.175:6816/3877631021,v1:172.21.15.175:6817/3877631021] [v2:172.21.15.175:6818/3877631021,v1:172.21.15.175:6819/3877631021] exists,up 4b9426e1-bac8-47d3-ad58-54b89e8d1eb1 2024-09-11T03:38:22.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11 up in weight 1 up_from 83 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.175:6824/1150051236,v1:172.21.15.175:6825/1150051236] [v2:172.21.15.175:6826/1150051236,v1:172.21.15.175:6827/1150051236] exists,up cd1a28a1-413c-4274-9876-e1bdb7703be6 2024-09-11T03:38:22.572 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/108136515 expires 2024-09-12T03:31:26.598965+0000 2024-09-11T03:38:22.573 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/3411515187 expires 2024-09-12T03:31:26.598965+0000 2024-09-11T03:38:22.573 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:6801/3365920551 expires 2024-09-12T03:31:26.598965+0000 2024-09-11T03:38:22.573 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:6800/3365920551 expires 2024-09-12T03:31:26.598965+0000 2024-09-11T03:38:22.573 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/3885577441 expires 2024-09-12T03:31:03.077186+0000 2024-09-11T03:38:22.573 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/22039254 expires 2024-09-12T03:31:03.077186+0000 2024-09-11T03:38:22.573 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:6801/3496645948 expires 2024-09-12T03:31:03.077186+0000 2024-09-11T03:38:22.573 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/4206337830 expires 2024-09-12T03:31:26.598965+0000 2024-09-11T03:38:22.573 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:6800/3496645948 expires 2024-09-12T03:31:03.077186+0000 2024-09-11T03:38:22.573 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/1989775875 expires 2024-09-12T03:31:03.077186+0000 2024-09-11T03:38:23.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:22 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:38:23.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:22 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:38:23.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:22 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/777276205' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-11T03:38:23.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:22 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:38:23.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:22 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:38:23.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:22 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/777276205' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-11T03:38:23.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:22 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:38:23.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:22 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:38:23.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:22 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/777276205' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-11T03:38:23.291 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 52 down_at 0 last_clean_interval [0,0) [v2:172.21.15.84:6802/1376667411,v1:172.21.15.84:6803/1376667411] [v2:172.21.15.84:6804/1376667411,v1:172.21.15.84:6805/1376667411] exists,up 9a431e48-e46b-44f9-bfa7-cb56c1484e1c', 'osd.1 up in weight 1 up_from 13 up_thru 32 down_at 0 last_clean_interval [0,0) [v2:172.21.15.84:6810/3138939249,v1:172.21.15.84:6811/3138939249] [v2:172.21.15.84:6812/3138939249,v1:172.21.15.84:6813/3138939249] exists,up 62fd6e25-3508-40b4-a0f4-f7de137e3bb5', 'osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.84:6818/3001811591,v1:172.21.15.84:6819/3001811591] [v2:172.21.15.84:6820/3001811591,v1:172.21.15.84:6821/3001811591] exists,up ff351425-4715-49dc-be49-f27bf205d57e', 'osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.84:6826/302512129,v1:172.21.15.84:6827/302512129] [v2:172.21.15.84:6828/302512129,v1:172.21.15.84:6829/302512129] exists,up 2518a93e-c612-4731-91ce-5685c25ffa9f', 'osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6800/3873214513,v1:172.21.15.121:6801/3873214513] [v2:172.21.15.121:6802/3873214513,v1:172.21.15.121:6803/3873214513] exists,up d12f3c2d-95cc-4276-a7e1-bee982d58948', 'osd.5 up in weight 1 up_from 38 up_thru 39 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6808/918612194,v1:172.21.15.121:6809/918612194] [v2:172.21.15.121:6810/918612194,v1:172.21.15.121:6811/918612194] exists,up 32164dd5-7e8f-4925-8526-bbb1e33e3364', 'osd.6 up in weight 1 up_from 45 up_thru 46 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6816/3384813363,v1:172.21.15.121:6817/3384813363] [v2:172.21.15.121:6818/3384813363,v1:172.21.15.121:6819/3384813363] exists,up 3f107a88-803f-4cd8-804a-241310c282ea', 'osd.7 up in weight 1 up_from 52 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6824/449222054,v1:172.21.15.121:6825/449222054] [v2:172.21.15.121:6826/449222054,v1:172.21.15.121:6827/449222054] exists,up dfee7f95-1fb9-4493-9907-cb10d1837bdb', 'osd.8 up in weight 1 up_from 60 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.175:6800/2660128384,v1:172.21.15.175:6801/2660128384] [v2:172.21.15.175:6802/2660128384,v1:172.21.15.175:6803/2660128384] exists,up fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4', 'osd.9 up in weight 1 up_from 69 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.175:6808/1270991649,v1:172.21.15.175:6809/1270991649] [v2:172.21.15.175:6810/1270991649,v1:172.21.15.175:6811/1270991649] exists,up 50162e42-0443-41d1-8709-a0db61c087da', 'osd.10 up in weight 1 up_from 76 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.175:6816/3877631021,v1:172.21.15.175:6817/3877631021] [v2:172.21.15.175:6818/3877631021,v1:172.21.15.175:6819/3877631021] exists,up 4b9426e1-bac8-47d3-ad58-54b89e8d1eb1', 'osd.11 up in weight 1 up_from 83 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.175:6824/1150051236,v1:172.21.15.175:6825/1150051236] [v2:172.21.15.175:6826/1150051236,v1:172.21.15.175:6827/1150051236] exists,up cd1a28a1-413c-4274-9876-e1bdb7703be6'] 2024-09-11T03:38:23.292 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon mon.a config show 2024-09-11T03:38:23.532 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:23.979 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:23 smithi084 ceph-mon[27254]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:23.979 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:23 smithi084 ceph-mon[27254]: osdmap e93: 12 total, 12 up, 12 in 2024-09-11T03:38:23.988 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:38:23.988 INFO:teuthology.orchestra.run.smithi084.stdout: "name": "mon.a", 2024-09-11T03:38:23.989 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster": "ceph", 2024-09-11T03:38:23.989 INFO:teuthology.orchestra.run.smithi084.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-11T03:38:23.989 INFO:teuthology.orchestra.run.smithi084.stdout: "admin_socket_mode": "", 2024-09-11T03:38:23.989 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-11T03:38:23.989 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_client_required": "cephx, none", 2024-09-11T03:38:23.989 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_cluster_required": "cephx", 2024-09-11T03:38:23.989 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_debug": "true", 2024-09-11T03:38:23.989 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-11T03:38:23.989 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-11T03:38:23.989 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_service_required": "cephx", 2024-09-11T03:38:23.989 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-11T03:38:23.989 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_supported": "", 2024-09-11T03:38:23.990 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_aio": "true", 2024-09-11T03:38:23.990 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-11T03:38:23.990 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_aio_poll_ms": "250", 2024-09-11T03:38:23.990 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_aio_reap_max": "16", 2024-09-11T03:38:23.990 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_async_discard_threads": "0", 2024-09-11T03:38:23.990 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_block_size": "4096", 2024-09-11T03:38:23.990 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_debug_aio": "false", 2024-09-11T03:38:23.990 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-11T03:38:23.990 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-11T03:38:23.990 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_debug_inflight_ios": "false", 2024-09-11T03:38:23.990 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_enable_discard": "false", 2024-09-11T03:38:23.990 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_flock_retry": "3", 2024-09-11T03:38:23.990 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-11T03:38:23.991 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_inject_crash": "0", 2024-09-11T03:38:23.991 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-11T03:38:23.991 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_ioring": "false", 2024-09-11T03:38:23.991 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_ioring_hipri": "false", 2024-09-11T03:38:23.991 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-11T03:38:23.991 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-11T03:38:23.991 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-11T03:38:23.991 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-11T03:38:23.991 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-11T03:38:23.991 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-11T03:38:23.991 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_type": "", 2024-09-11T03:38:23.991 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_alloc_size": "1048576", 2024-09-11T03:38:23.991 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_allocator": "hybrid", 2024-09-11T03:38:23.992 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_buffered_io": "true", 2024-09-11T03:38:23.992 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_check_for_zeros": "false", 2024-09-11T03:38:23.992 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-11T03:38:23.992 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-11T03:38:23.992 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_compact_log_sync": "false", 2024-09-11T03:38:23.992 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-11T03:38:23.992 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-11T03:38:23.992 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-11T03:38:23.992 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-11T03:38:23.992 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_max_log_runway": "4194304", 2024-09-11T03:38:23.992 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_max_prefetch": "1048576", 2024-09-11T03:38:23.993 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_min_flush_size": "524288", 2024-09-11T03:38:23.993 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_min_log_runway": "1048576", 2024-09-11T03:38:23.993 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_replay_recovery": "false", 2024-09-11T03:38:23.993 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-11T03:38:23.993 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-11T03:38:23.993 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_sync_write": "false", 2024-09-11T03:38:23.993 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-11T03:38:23.993 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-11T03:38:23.993 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-11T03:38:23.993 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_allocation_from_file": "true", 2024-09-11T03:38:23.993 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_allocator": "hybrid", 2024-09-11T03:38:23.993 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_async_db_compaction": "true", 2024-09-11T03:38:23.993 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-11T03:38:23.994 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-11T03:38:23.994 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-11T03:38:23.994 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-11T03:38:23.994 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bdev_label_multi": "true", 2024-09-11T03:38:23.994 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-11T03:38:23.994 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-11T03:38:23.994 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-11T03:38:23.994 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-11T03:38:23.994 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-11T03:38:23.994 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_create": "true", 2024-09-11T03:38:23.994 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_db_create": "false", 2024-09-11T03:38:23.994 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_db_path": "", 2024-09-11T03:38:23.994 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_db_size": "0", 2024-09-11T03:38:23.995 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_path": "", 2024-09-11T03:38:23.995 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_preallocate_file": "false", 2024-09-11T03:38:23.995 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_size": "107374182400", 2024-09-11T03:38:23.995 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_wal_create": "false", 2024-09-11T03:38:23.995 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_wal_path": "", 2024-09-11T03:38:23.995 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_wal_size": "100663296", 2024-09-11T03:38:23.995 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bluefs": "true", 2024-09-11T03:38:23.995 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-11T03:38:23.995 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-11T03:38:23.995 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-11T03:38:23.995 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-11T03:38:23.995 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-11T03:38:23.995 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-11T03:38:23.996 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-11T03:38:23.996 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-11T03:38:23.996 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-11T03:38:23.996 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_autotune": "true", 2024-09-11T03:38:23.996 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-11T03:38:23.996 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-11T03:38:23.996 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-11T03:38:23.996 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-11T03:38:23.996 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_size": "0", 2024-09-11T03:38:23.996 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-11T03:38:23.996 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-11T03:38:23.996 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-11T03:38:23.996 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-11T03:38:23.996 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_type": "2q", 2024-09-11T03:38:23.997 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-11T03:38:23.997 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_clone_cow": "true", 2024-09-11T03:38:23.997 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-11T03:38:23.997 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-11T03:38:23.997 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-11T03:38:23.997 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-11T03:38:23.997 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-11T03:38:23.997 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-11T03:38:23.997 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-11T03:38:23.997 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_mode": "none", 2024-09-11T03:38:23.998 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-11T03:38:23.998 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_csum_type": "crc32c", 2024-09-11T03:38:23.998 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-11T03:38:23.998 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-11T03:38:23.998 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_freelist": "false", 2024-09-11T03:38:23.998 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-11T03:38:23.998 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-11T03:38:23.998 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-11T03:38:23.998 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-11T03:38:23.998 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-11T03:38:23.998 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-11T03:38:23.998 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-11T03:38:23.998 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-11T03:38:23.999 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-11T03:38:23.999 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-11T03:38:23.999 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-11T03:38:23.999 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-11T03:38:23.999 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_small_allocations": "0", 2024-09-11T03:38:23.999 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-11T03:38:23.999 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_default_buffered_read": "true", 2024-09-11T03:38:23.999 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_default_buffered_write": "false", 2024-09-11T03:38:23.999 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-11T03:38:23.999 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-11T03:38:23.999 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-11T03:38:23.999 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-11T03:38:23.999 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-11T03:38:23.999 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-11T03:38:24.000 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-11T03:38:24.000 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-11T03:38:24.000 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-11T03:38:24.000 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fail_eio": "false", 2024-09-11T03:38:24.000 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-11T03:38:24.000 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-11T03:38:24.000 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-11T03:38:24.000 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-11T03:38:24.000 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-11T03:38:24.000 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-11T03:38:24.000 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_mount": "false", 2024-09-11T03:38:24.000 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-11T03:38:24.000 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_umount": "false", 2024-09-11T03:38:24.001 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-11T03:38:24.001 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-11T03:38:24.001 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-11T03:38:24.001 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-11T03:38:24.001 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-11T03:38:24.001 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-11T03:38:24.001 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-11T03:38:24.001 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-11T03:38:24.001 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_ignore_data_csum": "false", 2024-09-11T03:38:24.001 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-11T03:38:24.001 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-11T03:38:24.001 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-11T03:38:24.001 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-11T03:38:24.001 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_log_op_age": "5.000000", 2024-09-11T03:38:24.002 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_alloc_size": "0", 2024-09-11T03:38:24.002 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_blob_size": "0", 2024-09-11T03:38:24.002 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-11T03:38:24.002 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-11T03:38:24.002 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-11T03:38:24.002 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_deferred_txc": "32", 2024-09-11T03:38:24.002 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_min_alloc_size": "0", 2024-09-11T03:38:24.002 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-11T03:38:24.002 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-11T03:38:24.002 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_nid_prealloc": "1024", 2024-09-11T03:38:24.002 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-11T03:38:24.002 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-11T03:38:24.002 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-11T03:38:24.002 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-11T03:38:24.003 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_retry_disk_reads": "3", 2024-09-11T03:38:24.003 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_rocksdb_cf": "true", 2024-09-11T03:38:24.004 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-11T03:38:24.004 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-11T03:38:24.004 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-11T03:38:24.004 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-11T03:38:24.004 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-11T03:38:24.004 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-11T03:38:24.004 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-11T03:38:24.004 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-11T03:38:24.004 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_spdk_mem": "512", 2024-09-11T03:38:24.004 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-11T03:38:24.004 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-11T03:38:24.004 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-11T03:38:24.005 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-11T03:38:24.005 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-11T03:38:24.005 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-11T03:38:24.005 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-11T03:38:24.005 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_tracing": "false", 2024-09-11T03:38:24.005 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-11T03:38:24.005 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-11T03:38:24.005 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-11T03:38:24.005 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-11T03:38:24.005 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-11T03:38:24.005 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-11T03:38:24.005 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-11T03:38:24.005 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-11T03:38:24.005 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-11T03:38:24.006 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_write_v2": "false", 2024-09-11T03:38:24.006 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_write_v2_random": "false", 2024-09-11T03:38:24.006 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_zero_block_detection": "false", 2024-09-11T03:38:24.006 INFO:teuthology.orchestra.run.smithi084.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-11T03:38:24.006 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-11T03:38:24.006 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-11T03:38:24.006 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-11T03:38:24.006 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-11T03:38:24.006 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-11T03:38:24.006 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-11T03:38:24.006 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-11T03:38:24.006 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-11T03:38:24.007 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-11T03:38:24.007 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-11T03:38:24.007 INFO:teuthology.orchestra.run.smithi084.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-11T03:38:24.007 INFO:teuthology.orchestra.run.smithi084.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-11T03:38:24.007 INFO:teuthology.orchestra.run.smithi084.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-11T03:38:24.007 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_cluster_require_signatures": "false", 2024-09-11T03:38:24.007 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_cluster_require_version": "2", 2024-09-11T03:38:24.007 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_require_signatures": "false", 2024-09-11T03:38:24.007 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_require_version": "2", 2024-09-11T03:38:24.007 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_service_require_signatures": "false", 2024-09-11T03:38:24.007 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_service_require_version": "2", 2024-09-11T03:38:24.007 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_sign_messages": "true", 2024-09-11T03:38:24.007 INFO:teuthology.orchestra.run.smithi084.stdout: "chdir": "", 2024-09-11T03:38:24.007 INFO:teuthology.orchestra.run.smithi084.stdout: "client_acl_type": "", 2024-09-11T03:38:24.008 INFO:teuthology.orchestra.run.smithi084.stdout: "client_asio_thread_count": "2", 2024-09-11T03:38:24.008 INFO:teuthology.orchestra.run.smithi084.stdout: "client_cache_mid": "0.750000", 2024-09-11T03:38:24.008 INFO:teuthology.orchestra.run.smithi084.stdout: "client_cache_size": "16384", 2024-09-11T03:38:24.008 INFO:teuthology.orchestra.run.smithi084.stdout: "client_caps_release_delay": "5", 2024-09-11T03:38:24.008 INFO:teuthology.orchestra.run.smithi084.stdout: "client_check_pool_perm": "true", 2024-09-11T03:38:24.008 INFO:teuthology.orchestra.run.smithi084.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-11T03:38:24.008 INFO:teuthology.orchestra.run.smithi084.stdout: "client_debug_force_sync_read": "false", 2024-09-11T03:38:24.008 INFO:teuthology.orchestra.run.smithi084.stdout: "client_debug_getattr_caps": "false", 2024-09-11T03:38:24.008 INFO:teuthology.orchestra.run.smithi084.stdout: "client_debug_inject_features": "", 2024-09-11T03:38:24.008 INFO:teuthology.orchestra.run.smithi084.stdout: "client_debug_inject_tick_delay": "0", 2024-09-11T03:38:24.008 INFO:teuthology.orchestra.run.smithi084.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-11T03:38:24.008 INFO:teuthology.orchestra.run.smithi084.stdout: "client_die_on_failed_remount": "false", 2024-09-11T03:38:24.008 INFO:teuthology.orchestra.run.smithi084.stdout: "client_dirsize_rbytes": "true", 2024-09-11T03:38:24.009 INFO:teuthology.orchestra.run.smithi084.stdout: "client_force_lazyio": "false", 2024-09-11T03:38:24.009 INFO:teuthology.orchestra.run.smithi084.stdout: "client_fs": "", 2024-09-11T03:38:24.009 INFO:teuthology.orchestra.run.smithi084.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-11T03:38:24.009 INFO:teuthology.orchestra.run.smithi084.stdout: "client_inject_release_failure": "false", 2024-09-11T03:38:24.009 INFO:teuthology.orchestra.run.smithi084.stdout: "client_max_inline_size": "4096", 2024-09-11T03:38:24.009 INFO:teuthology.orchestra.run.smithi084.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-11T03:38:24.009 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mds_namespace": "", 2024-09-11T03:38:24.009 INFO:teuthology.orchestra.run.smithi084.stdout: "client_metadata": "", 2024-09-11T03:38:24.009 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mount_gid": "-1", 2024-09-11T03:38:24.009 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mount_timeout": "300", 2024-09-11T03:38:24.009 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mount_uid": "-1", 2024-09-11T03:38:24.009 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mountpoint": "/", 2024-09-11T03:38:24.009 INFO:teuthology.orchestra.run.smithi084.stdout: "client_notify_timeout": "10", 2024-09-11T03:38:24.010 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc": "true", 2024-09-11T03:38:24.010 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_max_dirty": "104857600", 2024-09-11T03:38:24.010 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-11T03:38:24.010 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_max_objects": "1000", 2024-09-11T03:38:24.010 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_size": "209715200", 2024-09-11T03:38:24.010 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_target_dirty": "8388608", 2024-09-11T03:38:24.010 INFO:teuthology.orchestra.run.smithi084.stdout: "client_permissions": "true", 2024-09-11T03:38:24.010 INFO:teuthology.orchestra.run.smithi084.stdout: "client_quota": "true", 2024-09-11T03:38:24.010 INFO:teuthology.orchestra.run.smithi084.stdout: "client_quota_df": "true", 2024-09-11T03:38:24.010 INFO:teuthology.orchestra.run.smithi084.stdout: "client_readahead_max_bytes": "0", 2024-09-11T03:38:24.010 INFO:teuthology.orchestra.run.smithi084.stdout: "client_readahead_max_periods": "4", 2024-09-11T03:38:24.010 INFO:teuthology.orchestra.run.smithi084.stdout: "client_readahead_min": "131072", 2024-09-11T03:38:24.010 INFO:teuthology.orchestra.run.smithi084.stdout: "client_reconnect_stale": "false", 2024-09-11T03:38:24.011 INFO:teuthology.orchestra.run.smithi084.stdout: "client_shutdown_timeout": "30", 2024-09-11T03:38:24.011 INFO:teuthology.orchestra.run.smithi084.stdout: "client_snapdir": ".snap", 2024-09-11T03:38:24.011 INFO:teuthology.orchestra.run.smithi084.stdout: "client_tick_interval": "1", 2024-09-11T03:38:24.011 INFO:teuthology.orchestra.run.smithi084.stdout: "client_trace": "", 2024-09-11T03:38:24.011 INFO:teuthology.orchestra.run.smithi084.stdout: "client_try_dentry_invalidate": "false", 2024-09-11T03:38:24.011 INFO:teuthology.orchestra.run.smithi084.stdout: "client_use_faked_inos": "false", 2024-09-11T03:38:24.011 INFO:teuthology.orchestra.run.smithi084.stdout: "client_use_random_mds": "false", 2024-09-11T03:38:24.011 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_graylog": "false", 2024-09-11T03:38:24.011 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-11T03:38:24.011 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_graylog_port": "12201", 2024-09-11T03:38:24.011 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_monitors": "default=true", 2024-09-11T03:38:24.011 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_syslog": "false", 2024-09-11T03:38:24.011 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-11T03:38:24.012 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_syslog_level": "info", 2024-09-11T03:38:24.012 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "-", 2024-09-11T03:38:24.012 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_network": "", 2024-09-11T03:38:24.012 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_network_interface": "", 2024-09-11T03:38:24.012 INFO:teuthology.orchestra.run.smithi084.stdout: "compressor_zlib_isal": "false", 2024-09-11T03:38:24.012 INFO:teuthology.orchestra.run.smithi084.stdout: "compressor_zlib_level": "5", 2024-09-11T03:38:24.012 INFO:teuthology.orchestra.run.smithi084.stdout: "compressor_zlib_winsize": "-15", 2024-09-11T03:38:24.012 INFO:teuthology.orchestra.run.smithi084.stdout: "compressor_zstd_level": "1", 2024-09-11T03:38:24.012 INFO:teuthology.orchestra.run.smithi084.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e", 2024-09-11T03:38:24.012 INFO:teuthology.orchestra.run.smithi084.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-11T03:38:24.012 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_alien_op_num_threads": "6", 2024-09-11T03:38:24.012 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-11T03:38:24.012 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-11T03:38:24.012 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-11T03:38:24.013 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_osd_stat_interval": "0", 2024-09-11T03:38:24.013 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_seastar_cpu_cores": "", 2024-09-11T03:38:24.013 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_seastar_num_threads": "0", 2024-09-11T03:38:24.013 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location": "", 2024-09-11T03:38:24.016 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location_hook": "", 2024-09-11T03:38:24.016 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location_hook_timeout": "10", 2024-09-11T03:38:24.016 INFO:teuthology.orchestra.run.smithi084.stdout: "daemonize": "false", 2024-09-11T03:38:24.016 INFO:teuthology.orchestra.run.smithi084.stdout: "daos_pool": "tank", 2024-09-11T03:38:24.016 INFO:teuthology.orchestra.run.smithi084.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-11T03:38:24.016 INFO:teuthology.orchestra.run.smithi084.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-11T03:38:24.016 INFO:teuthology.orchestra.run.smithi084.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-11T03:38:24.016 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_alienstore": "0/5", 2024-09-11T03:38:24.016 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_allow_any_pool_priority": "false", 2024-09-11T03:38:24.016 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_asok": "1/5", 2024-09-11T03:38:24.016 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_asok_assert_abort": "false", 2024-09-11T03:38:24.016 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_asserts_on_shutdown": "true", 2024-09-11T03:38:24.017 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_auth": "1/5", 2024-09-11T03:38:24.017 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_bdev": "1/3", 2024-09-11T03:38:24.017 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_bluefs": "1/5", 2024-09-11T03:38:24.017 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_bluestore": "1/5", 2024-09-11T03:38:24.017 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_buffer": "0/1", 2024-09-11T03:38:24.017 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_ceph_dedup": "0/5", 2024-09-11T03:38:24.017 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_ceph_exporter": "1/5", 2024-09-11T03:38:24.017 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_cephfs_mirror": "0/5", 2024-09-11T03:38:24.017 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_cephsqlite": "0/5", 2024-09-11T03:38:24.017 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_client": "0/5", 2024-09-11T03:38:24.017 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_compressor": "1/5", 2024-09-11T03:38:24.017 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_context": "0/1", 2024-09-11T03:38:24.017 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_crimson_interrupt": "0/5", 2024-09-11T03:38:24.017 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_crush": "1/1", 2024-09-11T03:38:24.018 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_crypto": "1/5", 2024-09-11T03:38:24.018 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_cyanstore": "0/5", 2024-09-11T03:38:24.018 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_deliberately_leak_memory": "false", 2024-09-11T03:38:24.018 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_disable_randomized_ping": "false", 2024-09-11T03:38:24.018 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_dpdk": "1/5", 2024-09-11T03:38:24.018 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_eventtrace": "1/5", 2024-09-11T03:38:24.018 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_filer": "0/1", 2024-09-11T03:38:24.018 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_filestore": "1/3", 2024-09-11T03:38:24.018 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_finisher": "1/1", 2024-09-11T03:38:24.018 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_fuse": "1/5", 2024-09-11T03:38:24.018 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_heartbeat_testing_span": "0", 2024-09-11T03:38:24.018 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_heartbeatmap": "1/5", 2024-09-11T03:38:24.018 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-11T03:38:24.018 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_javaclient": "1/5", 2024-09-11T03:38:24.019 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_journal": "1/3", 2024-09-11T03:38:24.019 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_journaler": "0/5", 2024-09-11T03:38:24.019 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_kstore": "1/5", 2024-09-11T03:38:24.019 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_lockdep": "0/1", 2024-09-11T03:38:24.019 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mclock": "1/5", 2024-09-11T03:38:24.019 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds": "1/5", 2024-09-11T03:38:24.019 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_balancer": "1/5", 2024-09-11T03:38:24.019 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_locker": "1/5", 2024-09-11T03:38:24.019 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_log": "1/5", 2024-09-11T03:38:24.019 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_log_expire": "1/5", 2024-09-11T03:38:24.019 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_migrator": "1/5", 2024-09-11T03:38:24.019 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_quiesce": "3/5", 2024-09-11T03:38:24.019 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_memstore": "1/5", 2024-09-11T03:38:24.019 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mgr": "2/5", 2024-09-11T03:38:24.020 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mgrc": "1/5", 2024-09-11T03:38:24.020 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mon": "20/20", 2024-09-11T03:38:24.020 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_monc": "0/10", 2024-09-11T03:38:24.020 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_ms": "1/1", 2024-09-11T03:38:24.020 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_none": "0/5", 2024-09-11T03:38:24.020 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_objclass": "0/5", 2024-09-11T03:38:24.020 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_objectcacher": "0/5", 2024-09-11T03:38:24.020 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_objecter": "0/1", 2024-09-11T03:38:24.020 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_optracker": "0/5", 2024-09-11T03:38:24.020 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_osd": "1/5", 2024-09-11T03:38:24.020 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_paxos": "20/20", 2024-09-11T03:38:24.020 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_perfcounter": "1/5", 2024-09-11T03:38:24.020 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_prioritycache": "1/5", 2024-09-11T03:38:24.020 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rados": "0/5", 2024-09-11T03:38:24.021 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rbd": "0/5", 2024-09-11T03:38:24.021 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rbd_mirror": "0/5", 2024-09-11T03:38:24.021 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rbd_pwl": "0/5", 2024-09-11T03:38:24.021 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rbd_replay": "0/5", 2024-09-11T03:38:24.021 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_refs": "0/0", 2024-09-11T03:38:24.021 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_reserver": "1/1", 2024-09-11T03:38:24.021 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw": "1/5", 2024-09-11T03:38:24.021 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_access": "1/5", 2024-09-11T03:38:24.021 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_datacache": "1/5", 2024-09-11T03:38:24.021 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_dbstore": "1/5", 2024-09-11T03:38:24.021 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_flight": "1/5", 2024-09-11T03:38:24.021 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-11T03:38:24.021 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_notification": "1/5", 2024-09-11T03:38:24.022 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_sync": "1/5", 2024-09-11T03:38:24.022 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rocksdb": "4/5", 2024-09-11T03:38:24.022 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore": "0/5", 2024-09-11T03:38:24.022 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_backref": "0/5", 2024-09-11T03:38:24.022 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_cache": "0/5", 2024-09-11T03:38:24.022 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_cleaner": "0/5", 2024-09-11T03:38:24.022 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_device": "0/5", 2024-09-11T03:38:24.022 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_epm": "0/5", 2024-09-11T03:38:24.022 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-11T03:38:24.022 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_journal": "0/5", 2024-09-11T03:38:24.022 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_lba": "0/5", 2024-09-11T03:38:24.022 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_odata": "0/5", 2024-09-11T03:38:24.022 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_omap": "0/5", 2024-09-11T03:38:24.022 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_onode": "0/5", 2024-09-11T03:38:24.023 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_t": "0/5", 2024-09-11T03:38:24.023 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_tm": "0/5", 2024-09-11T03:38:24.023 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_striper": "0/1", 2024-09-11T03:38:24.023 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_test": "0/5", 2024-09-11T03:38:24.023 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_throttle": "1/1", 2024-09-11T03:38:24.023 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_timer": "0/1", 2024-09-11T03:38:24.023 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_tp": "0/5", 2024-09-11T03:38:24.023 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_trace": "1/5", 2024-09-11T03:38:24.023 INFO:teuthology.orchestra.run.smithi084.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-11T03:38:24.023 INFO:teuthology.orchestra.run.smithi084.stdout: "device_failure_prediction_mode": "none", 2024-09-11T03:38:24.023 INFO:teuthology.orchestra.run.smithi084.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-11T03:38:24.023 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-11T03:38:24.023 INFO:teuthology.orchestra.run.smithi084.stdout: "err_to_graylog": "false", 2024-09-11T03:38:24.023 INFO:teuthology.orchestra.run.smithi084.stdout: "err_to_journald": "false", 2024-09-11T03:38:24.024 INFO:teuthology.orchestra.run.smithi084.stdout: "err_to_stderr": "true", 2024-09-11T03:38:24.024 INFO:teuthology.orchestra.run.smithi084.stdout: "err_to_syslog": "false", 2024-09-11T03:38:24.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event_tracing": "false", 2024-09-11T03:38:24.024 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_addr": "0.0.0.0", 2024-09-11T03:38:24.024 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_cert_file": "", 2024-09-11T03:38:24.024 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_http_port": "9926", 2024-09-11T03:38:24.024 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_key_file": "", 2024-09-11T03:38:24.024 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_prio_limit": "5", 2024-09-11T03:38:24.024 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-11T03:38:24.024 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_sort_metrics": "true", 2024-09-11T03:38:24.024 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_stats_period": "5", 2024-09-11T03:38:24.024 INFO:teuthology.orchestra.run.smithi084.stdout: "fake_statfs_for_testing": "0", 2024-09-11T03:38:24.024 INFO:teuthology.orchestra.run.smithi084.stdout: "fatal_signal_handlers": "true", 2024-09-11T03:38:24.025 INFO:teuthology.orchestra.run.smithi084.stdout: "filer_max_purge_ops": "10", 2024-09-11T03:38:24.025 INFO:teuthology.orchestra.run.smithi084.stdout: "filer_max_truncate_ops": "128", 2024-09-11T03:38:24.025 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_apply_finisher_threads": "1", 2024-09-11T03:38:24.025 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_blackhole": "false", 2024-09-11T03:38:24.025 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_btrfs_clone_range": "true", 2024-09-11T03:38:24.025 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_btrfs_snap": "true", 2024-09-11T03:38:24.025 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_caller_concurrency": "10", 2024-09-11T03:38:24.025 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_collect_device_partition_information": "true", 2024-09-11T03:38:24.025 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_commit_timeout": "600.000000", 2024-09-11T03:38:24.025 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_debug_inject_read_err": "false", 2024-09-11T03:38:24.025 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_debug_omap_check": "false", 2024-09-11T03:38:24.025 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_debug_verify_split": "false", 2024-09-11T03:38:24.025 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_dump_file": "", 2024-09-11T03:38:24.026 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-11T03:38:24.026 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-11T03:38:24.026 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fadvise": "true", 2024-09-11T03:38:24.026 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fail_eio": "true", 2024-09-11T03:38:24.026 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fd_cache_shards": "16", 2024-09-11T03:38:24.026 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fd_cache_size": "128", 2024-09-11T03:38:24.026 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fiemap": "false", 2024-09-11T03:38:24.026 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fiemap_threshold": "4096", 2024-09-11T03:38:24.026 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-11T03:38:24.026 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-11T03:38:24.026 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_inject_stall": "0", 2024-09-11T03:38:24.026 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_journal_parallel": "false", 2024-09-11T03:38:24.026 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_journal_trailing": "false", 2024-09-11T03:38:24.027 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_journal_writeahead": "false", 2024-09-11T03:38:24.027 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_kill_at": "0", 2024-09-11T03:38:24.027 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-11T03:38:24.027 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-11T03:38:24.027 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-11T03:38:24.027 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-11T03:38:24.027 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-11T03:38:24.027 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattrs": "0", 2024-09-11T03:38:24.027 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-11T03:38:24.027 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-11T03:38:24.027 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-11T03:38:24.027 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-11T03:38:24.027 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_xattr_value_size": "0", 2024-09-11T03:38:24.028 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-11T03:38:24.028 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-11T03:38:24.028 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-11T03:38:24.028 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_merge_threshold": "-10", 2024-09-11T03:38:24.028 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-11T03:38:24.028 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_odsync_write": "false", 2024-09-11T03:38:24.028 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_omap_backend": "rocksdb", 2024-09-11T03:38:24.028 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_omap_backend_path": "", 2024-09-11T03:38:24.028 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-11T03:38:24.028 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-11T03:38:24.028 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-11T03:38:24.028 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_op_thread_timeout": "60", 2024-09-11T03:38:24.028 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_op_threads": "2", 2024-09-11T03:38:24.029 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_punch_hole": "false", 2024-09-11T03:38:24.029 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-11T03:38:24.029 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-11T03:38:24.029 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-11T03:38:24.029 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-11T03:38:24.029 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-11T03:38:24.029 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-11T03:38:24.029 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-11T03:38:24.029 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-11T03:38:24.029 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-11T03:38:24.029 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_ops": "50", 2024-09-11T03:38:24.029 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-11T03:38:24.030 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_seek_data_hole": "false", 2024-09-11T03:38:24.030 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_sloppy_crc": "false", 2024-09-11T03:38:24.030 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-11T03:38:24.030 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_splice": "false", 2024-09-11T03:38:24.030 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_split_multiple": "2", 2024-09-11T03:38:24.030 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_split_rand_factor": "20", 2024-09-11T03:38:24.030 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_update_to": "1000", 2024-09-11T03:38:24.030 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-11T03:38:24.030 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-11T03:38:24.030 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-11T03:38:24.030 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-11T03:38:24.030 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-11T03:38:24.030 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-11T03:38:24.031 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_enable": "true", 2024-09-11T03:38:24.031 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-11T03:38:24.031 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-11T03:38:24.031 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-11T03:38:24.031 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-11T03:38:24.031 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-11T03:38:24.031 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-11T03:38:24.031 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_xfs_extsize": "false", 2024-09-11T03:38:24.031 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_zfs_snap": "false", 2024-09-11T03:38:24.031 INFO:teuthology.orchestra.run.smithi084.stdout: "fio_dir": "/tmp/fio", 2024-09-11T03:38:24.031 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "10fe7946-6fee-11ef-bcea-c7b262605968", 2024-09-11T03:38:24.031 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_allow_other": "true", 2024-09-11T03:38:24.031 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_atomic_o_trunc": "true", 2024-09-11T03:38:24.032 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_big_writes": "true", 2024-09-11T03:38:24.032 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_debug": "false", 2024-09-11T03:38:24.032 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_default_permissions": "false", 2024-09-11T03:38:24.032 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_disable_pagecache": "false", 2024-09-11T03:38:24.032 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_max_write": "0", 2024-09-11T03:38:24.032 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_multithreaded": "true", 2024-09-11T03:38:24.032 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_require_active_mds": "true", 2024-09-11T03:38:24.032 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_set_user_groups": "true", 2024-09-11T03:38:24.032 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_splice_move": "true", 2024-09-11T03:38:24.032 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_splice_read": "true", 2024-09-11T03:38:24.032 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_splice_write": "true", 2024-09-11T03:38:24.032 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-11T03:38:24.032 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_use_invalidate_cb": "true", 2024-09-11T03:38:24.033 INFO:teuthology.orchestra.run.smithi084.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-11T03:38:24.033 INFO:teuthology.orchestra.run.smithi084.stdout: "gss_target_name": "ceph", 2024-09-11T03:38:24.033 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_file": "", 2024-09-11T03:38:24.033 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_inject_failure": "0", 2024-09-11T03:38:24.033 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_interval": "5", 2024-09-11T03:38:24.033 INFO:teuthology.orchestra.run.smithi084.stdout: "host": "smithi084", 2024-09-11T03:38:24.033 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-11T03:38:24.033 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-11T03:38:24.033 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-11T03:38:24.033 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-11T03:38:24.033 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-11T03:38:24.033 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-11T03:38:24.033 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-11T03:38:24.034 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-11T03:38:24.034 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-11T03:38:24.034 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-11T03:38:24.034 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-11T03:38:24.034 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-11T03:38:24.034 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-11T03:38:24.034 INFO:teuthology.orchestra.run.smithi084.stdout: "inject_early_sigterm": "false", 2024-09-11T03:38:24.034 INFO:teuthology.orchestra.run.smithi084.stdout: "jaeger_agent_port": "6799", 2024-09-11T03:38:24.034 INFO:teuthology.orchestra.run.smithi084.stdout: "jaeger_tracing_enable": "false", 2024-09-11T03:38:24.034 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_aio": "true", 2024-09-11T03:38:24.034 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_align_min_size": "65536", 2024-09-11T03:38:24.034 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_block_align": "true", 2024-09-11T03:38:24.034 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_block_size": "4096", 2024-09-11T03:38:24.035 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_dio": "true", 2024-09-11T03:38:24.035 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_discard": "false", 2024-09-11T03:38:24.035 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_force_aio": "false", 2024-09-11T03:38:24.035 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_ignore_corruption": "false", 2024-09-11T03:38:24.035 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_max_write_bytes": "10485760", 2024-09-11T03:38:24.035 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_max_write_entries": "100", 2024-09-11T03:38:24.035 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_replay_from": "0", 2024-09-11T03:38:24.035 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-11T03:38:24.035 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-11T03:38:24.035 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-11T03:38:24.035 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-11T03:38:24.035 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_write_header_frequency": "0", 2024-09-11T03:38:24.035 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_zero_on_create": "false", 2024-09-11T03:38:24.036 INFO:teuthology.orchestra.run.smithi084.stdout: "journaler_prefetch_periods": "10", 2024-09-11T03:38:24.036 INFO:teuthology.orchestra.run.smithi084.stdout: "journaler_prezero_periods": "5", 2024-09-11T03:38:24.036 INFO:teuthology.orchestra.run.smithi084.stdout: "journaler_write_head_interval": "15", 2024-09-11T03:38:24.036 INFO:teuthology.orchestra.run.smithi084.stdout: "key": "", 2024-09-11T03:38:24.036 INFO:teuthology.orchestra.run.smithi084.stdout: "keyfile": "", 2024-09-11T03:38:24.036 INFO:teuthology.orchestra.run.smithi084.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-11T03:38:24.036 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_backend": "rocksdb", 2024-09-11T03:38:24.036 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_default_stripe_size": "65536", 2024-09-11T03:38:24.036 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_fsck_on_mount": "false", 2024-09-11T03:38:24.036 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-11T03:38:24.036 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_max_bytes": "67108864", 2024-09-11T03:38:24.036 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_max_ops": "512", 2024-09-11T03:38:24.036 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_nid_prealloc": "1024", 2024-09-11T03:38:24.037 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_onode_map_size": "1024", 2024-09-11T03:38:24.037 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-11T03:38:24.037 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_sync_submit_transaction": "false", 2024-09-11T03:38:24.037 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_sync_transaction": "false", 2024-09-11T03:38:24.037 INFO:teuthology.orchestra.run.smithi084.stdout: "librados_thread_count": "2", 2024-09-11T03:38:24.037 INFO:teuthology.orchestra.run.smithi084.stdout: "lockdep": "false", 2024-09-11T03:38:24.037 INFO:teuthology.orchestra.run.smithi084.stdout: "lockdep_force_backtrace": "false", 2024-09-11T03:38:24.037 INFO:teuthology.orchestra.run.smithi084.stdout: "log_coarse_timestamps": "true", 2024-09-11T03:38:24.037 INFO:teuthology.orchestra.run.smithi084.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-11T03:38:24.037 INFO:teuthology.orchestra.run.smithi084.stdout: "log_flush_on_exit": "false", 2024-09-11T03:38:24.037 INFO:teuthology.orchestra.run.smithi084.stdout: "log_graylog_host": "127.0.0.1", 2024-09-11T03:38:24.037 INFO:teuthology.orchestra.run.smithi084.stdout: "log_graylog_port": "12201", 2024-09-11T03:38:24.037 INFO:teuthology.orchestra.run.smithi084.stdout: "log_max_new": "1000", 2024-09-11T03:38:24.038 INFO:teuthology.orchestra.run.smithi084.stdout: "log_max_recent": "10000", 2024-09-11T03:38:24.038 INFO:teuthology.orchestra.run.smithi084.stdout: "log_stderr_prefix": "", 2024-09-11T03:38:24.038 INFO:teuthology.orchestra.run.smithi084.stdout: "log_stop_at_utilization": "0.970000", 2024-09-11T03:38:24.038 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_file": "true", 2024-09-11T03:38:24.038 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_graylog": "false", 2024-09-11T03:38:24.038 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_journald": "false", 2024-09-11T03:38:24.038 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_stderr": "false", 2024-09-11T03:38:24.038 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_syslog": "false", 2024-09-11T03:38:24.038 INFO:teuthology.orchestra.run.smithi084.stdout: "max_rotating_auth_attempts": "10", 2024-09-11T03:38:24.038 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-11T03:38:24.038 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_action_on_write_error": "1", 2024-09-11T03:38:24.038 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_alternate_name_max": "8192", 2024-09-11T03:38:24.038 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_asio_thread_count": "2", 2024-09-11T03:38:24.038 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_export_pin": "true", 2024-09-11T03:38:24.039 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_fragment_dirs": "true", 2024-09-11T03:38:24.039 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-11T03:38:24.039 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_fragment_interval": "5", 2024-09-11T03:38:24.039 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-11T03:38:24.039 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-11T03:38:24.039 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_interval": "10", 2024-09-11T03:38:24.039 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_max": "-1", 2024-09-11T03:38:24.039 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_max_until": "-1", 2024-09-11T03:38:24.039 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_merge_size": "50", 2024-09-11T03:38:24.039 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_midchunk": "0.300000", 2024-09-11T03:38:24.039 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-11T03:38:24.039 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_min_start": "0.200000", 2024-09-11T03:38:24.039 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_minchunk": "0.001000", 2024-09-11T03:38:24.040 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_mode": "0", 2024-09-11T03:38:24.040 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_need_max": "1.200000", 2024-09-11T03:38:24.040 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_need_min": "0.800000", 2024-09-11T03:38:24.040 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_overload_epochs": "2", 2024-09-11T03:38:24.040 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-11T03:38:24.040 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-11T03:38:24.040 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_split_bits": "3", 2024-09-11T03:38:24.040 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-11T03:38:24.040 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_split_size": "10000", 2024-09-11T03:38:24.040 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-11T03:38:24.040 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_target_decay": "10.000000", 2024-09-11T03:38:24.040 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-11T03:38:24.040 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_beacon_grace": "15.000000", 2024-09-11T03:38:24.041 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_beacon_interval": "4.000000", 2024-09-11T03:38:24.041 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-11T03:38:24.041 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-11T03:38:24.041 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_mid": "0.700000", 2024-09-11T03:38:24.041 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-11T03:38:24.041 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_quiesce_delay": "0", 2024-09-11T03:38:24.041 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-11T03:38:24.041 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-11T03:38:24.041 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-11T03:38:24.041 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_release_free_interval": "10", 2024-09-11T03:38:24.041 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_reservation": "0.050000", 2024-09-11T03:38:24.041 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-11T03:38:24.041 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_trim_interval": "1", 2024-09-11T03:38:24.042 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_trim_threshold": "262144", 2024-09-11T03:38:24.042 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-11T03:38:24.042 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-11T03:38:24.042 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-11T03:38:24.042 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_client_prealloc_inos": "1000", 2024-09-11T03:38:24.042 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-11T03:38:24.042 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_connect_bootstrapping": "false", 2024-09-11T03:38:24.042 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_damage_table_max_entries": "10000", 2024-09-11T03:38:24.042 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-11T03:38:24.042 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_debug_auth_pins": "false", 2024-09-11T03:38:24.042 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_debug_frag": "false", 2024-09-11T03:38:24.042 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_debug_scatterstat": "false", 2024-09-11T03:38:24.042 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_debug_subtrees": "false", 2024-09-11T03:38:24.043 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_decay_halflife": "5.000000", 2024-09-11T03:38:24.043 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_default_dir_hash": "2", 2024-09-11T03:38:24.043 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_defer_session_stale": "true", 2024-09-11T03:38:24.043 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_deny_all_reconnect": "false", 2024-09-11T03:38:24.043 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dir_keys_per_op": "16384", 2024-09-11T03:38:24.043 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dir_max_commit_size": "10", 2024-09-11T03:38:24.043 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dir_max_entries": "0", 2024-09-11T03:38:24.043 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dir_prefetch": "true", 2024-09-11T03:38:24.043 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-11T03:38:24.043 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-11T03:38:24.043 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dump_cache_on_map": "false", 2024-09-11T03:38:24.043 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-11T03:38:24.043 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-11T03:38:24.043 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_early_reply": "true", 2024-09-11T03:38:24.044 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_enable_op_tracker": "true", 2024-09-11T03:38:24.044 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_enforce_unique_name": "true", 2024-09-11T03:38:24.044 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-11T03:38:24.044 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-11T03:38:24.044 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_export_ephemeral_random": "true", 2024-09-11T03:38:24.044 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-11T03:38:24.044 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-11T03:38:24.044 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-11T03:38:24.044 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-11T03:38:24.044 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-11T03:38:24.044 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-11T03:38:24.044 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-11T03:38:24.044 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-11T03:38:24.045 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_health_summarize_threshold": "10", 2024-09-11T03:38:24.045 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-11T03:38:24.045 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-11T03:38:24.045 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_health_dummy": "false", 2024-09-11T03:38:24.045 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-11T03:38:24.045 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_migrator_session_race": "false", 2024-09-11T03:38:24.045 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-11T03:38:24.045 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-11T03:38:24.045 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-11T03:38:24.045 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_join_fs": "", 2024-09-11T03:38:24.045 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_journal_format": "1", 2024-09-11T03:38:24.045 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-11T03:38:24.045 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_create_at": "0", 2024-09-11T03:38:24.046 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_dirfrag_at": "0", 2024-09-11T03:38:24.046 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_export_at": "0", 2024-09-11T03:38:24.046 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_import_at": "0", 2024-09-11T03:38:24.046 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_journal_at": "0", 2024-09-11T03:38:24.046 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_journal_expire_at": "0", 2024-09-11T03:38:24.046 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_journal_replay_at": "0", 2024-09-11T03:38:24.046 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_link_at": "0", 2024-09-11T03:38:24.046 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_mdstable_at": "0", 2024-09-11T03:38:24.046 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_openc_at": "0", 2024-09-11T03:38:24.046 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_rename_at": "0", 2024-09-11T03:38:24.046 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_shutdown_at": "0", 2024-09-11T03:38:24.046 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_event_large_threshold": "524288", 2024-09-11T03:38:24.046 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_events_per_segment": "1024", 2024-09-11T03:38:24.046 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-11T03:38:24.047 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_max_events": "-1", 2024-09-11T03:38:24.047 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_max_segments": "128", 2024-09-11T03:38:24.047 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_pause": "false", 2024-09-11T03:38:24.047 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_segment_size": "0", 2024-09-11T03:38:24.047 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-11T03:38:24.047 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-11T03:38:24.047 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-11T03:38:24.047 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_trim_threshold": "128", 2024-09-11T03:38:24.047 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-11T03:38:24.047 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_warn_factor": "2.000000", 2024-09-11T03:38:24.047 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_caps_per_client": "1048576", 2024-09-11T03:38:24.047 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_completed_flushes": "100000", 2024-09-11T03:38:24.047 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_completed_requests": "100000", 2024-09-11T03:38:24.047 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_export_size": "20971520", 2024-09-11T03:38:24.048 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_file_recover": "32", 2024-09-11T03:38:24.048 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_purge_files": "64", 2024-09-11T03:38:24.048 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_purge_ops": "8192", 2024-09-11T03:38:24.048 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-11T03:38:24.048 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-11T03:38:24.048 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_snaps_per_dir": "100", 2024-09-11T03:38:24.048 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_metrics_update_interval": "2", 2024-09-11T03:38:24.048 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_min_caps_per_client": "100", 2024-09-11T03:38:24.048 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_min_caps_working_set": "10000", 2024-09-11T03:38:24.048 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-11T03:38:24.048 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_numa_node": "-1", 2024-09-11T03:38:24.048 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-11T03:38:24.048 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_complaint_time": "30.000000", 2024-09-11T03:38:24.048 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_history_duration": "600", 2024-09-11T03:38:24.049 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_history_size": "20", 2024-09-11T03:38:24.049 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_history_slow_op_size": "20", 2024-09-11T03:38:24.049 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-11T03:38:24.049 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_log_threshold": "5", 2024-09-11T03:38:24.049 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_ping_grace": "15", 2024-09-11T03:38:24.049 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_ping_interval": "5", 2024-09-11T03:38:24.049 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-11T03:38:24.049 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-11T03:38:24.049 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_max_caps": "30000", 2024-09-11T03:38:24.049 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-11T03:38:24.049 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-11T03:38:24.049 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-11T03:38:24.049 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_warning_threshold": "262144", 2024-09-11T03:38:24.049 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-11T03:38:24.050 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_replay_interval": "1.000000", 2024-09-11T03:38:24.050 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-11T03:38:24.050 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-11T03:38:24.050 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_root_ino_gid": "0", 2024-09-11T03:38:24.050 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_root_ino_uid": "0", 2024-09-11T03:38:24.050 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-11T03:38:24.050 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_scrub_stats_review_period": "1", 2024-09-11T03:38:24.050 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-11T03:38:24.050 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-11T03:38:24.050 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-11T03:38:24.050 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-11T03:38:24.050 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-11T03:38:24.050 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-11T03:38:24.050 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-11T03:38:24.051 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-11T03:38:24.051 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-11T03:38:24.051 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-11T03:38:24.051 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-11T03:38:24.051 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_shutdown_check": "0", 2024-09-11T03:38:24.051 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_skip_ino": "0", 2024-09-11T03:38:24.051 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-11T03:38:24.051 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_snap_max_uid": "4294967294", 2024-09-11T03:38:24.051 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_snap_min_uid": "0", 2024-09-11T03:38:24.051 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_snap_rstat": "false", 2024-09-11T03:38:24.051 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_standby_replay_damaged": "false", 2024-09-11T03:38:24.051 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_symlink_recovery": "true", 2024-09-11T03:38:24.051 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-11T03:38:24.051 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_thrash_exports": "0", 2024-09-11T03:38:24.052 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_thrash_fragments": "0", 2024-09-11T03:38:24.052 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_tick_interval": "5.000000", 2024-09-11T03:38:24.052 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_valgrind_exit": "false", 2024-09-11T03:38:24.052 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_verify_backtrace": "1", 2024-09-11T03:38:24.052 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_verify_scatter": "false", 2024-09-11T03:38:24.052 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-11T03:38:24.052 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_wipe_sessions": "false", 2024-09-11T03:38:24.052 INFO:teuthology.orchestra.run.smithi084.stdout: "mempool_debug": "false", 2024-09-11T03:38:24.052 INFO:teuthology.orchestra.run.smithi084.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-11T03:38:24.052 INFO:teuthology.orchestra.run.smithi084.stdout: "memstore_device_bytes": "1073741824", 2024-09-11T03:38:24.052 INFO:teuthology.orchestra.run.smithi084.stdout: "memstore_page_set": "false", 2024-09-11T03:38:24.052 INFO:teuthology.orchestra.run.smithi084.stdout: "memstore_page_size": "65536", 2024-09-11T03:38:24.052 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_client_bytes": "134217728", 2024-09-11T03:38:24.053 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_client_messages": "512", 2024-09-11T03:38:24.053 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-11T03:38:24.053 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-11T03:38:24.053 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-11T03:38:24.053 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-11T03:38:24.053 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_disabled_modules": "", 2024-09-11T03:38:24.053 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_enable_op_tracker": "true", 2024-09-11T03:38:24.053 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-11T03:38:24.053 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_max_pg_creating": "1024", 2024-09-11T03:38:24.053 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_max_pg_num_change": "128", 2024-09-11T03:38:24.053 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_mds_bytes": "134217728", 2024-09-11T03:38:24.053 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_mds_messages": "128", 2024-09-11T03:38:24.053 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-11T03:38:24.054 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_mon_bytes": "134217728", 2024-09-11T03:38:24.054 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_mon_messages": "128", 2024-09-11T03:38:24.054 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-11T03:38:24.054 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-11T03:38:24.054 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_history_duration": "600", 2024-09-11T03:38:24.054 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_history_size": "20", 2024-09-11T03:38:24.054 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-11T03:38:24.054 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-11T03:38:24.054 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_log_threshold": "5", 2024-09-11T03:38:24.054 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_osd_bytes": "536870912", 2024-09-11T03:38:24.054 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_osd_messages": "8192", 2024-09-11T03:38:24.054 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_pool": "true", 2024-09-11T03:38:24.054 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-11T03:38:24.055 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_standby_modules": "true", 2024-09-11T03:38:24.055 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_stats_period": "5", 2024-09-11T03:38:24.055 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_stats_threshold": "5", 2024-09-11T03:38:24.055 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_tick_period": "2", 2024-09-11T03:38:24.055 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-11T03:38:24.055 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-11T03:38:24.055 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_allow_pool_delete": "true", 2024-09-11T03:38:24.055 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_allow_pool_size_one": "false", 2024-09-11T03:38:24.055 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_auth_validate_all_caps": "true", 2024-09-11T03:38:24.055 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-11T03:38:24.055 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-11T03:38:24.055 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_bytes": "104857600", 2024-09-11T03:38:24.055 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_directed_command_retry": "5", 2024-09-11T03:38:24.055 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-11T03:38:24.056 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-11T03:38:24.056 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-11T03:38:24.056 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-11T03:38:24.056 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_parallel": "3", 2024-09-11T03:38:24.056 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_log_interval": "1.000000", 2024-09-11T03:38:24.056 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-11T03:38:24.056 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_ping_interval": "10.000000", 2024-09-11T03:38:24.056 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-11T03:38:24.056 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_target_rank": "-1", 2024-09-11T03:38:24.056 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-11T03:38:24.056 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-11T03:38:24.056 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-11T03:38:24.057 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_level": "debug", 2024-09-11T03:38:24.057 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_file": "true", 2024-09-11T03:38:24.057 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-11T03:38:24.057 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-11T03:38:24.057 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-11T03:38:24.057 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_journald": "true", 2024-09-11T03:38:24.057 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-11T03:38:24.057 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-11T03:38:24.057 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-11T03:38:24.057 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_compact_on_bootstrap": "false", 2024-09-11T03:38:24.057 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_compact_on_start": "false", 2024-09-11T03:38:24.057 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_compact_on_trim": "true", 2024-09-11T03:38:24.057 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-11T03:38:24.058 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-11T03:38:24.058 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-11T03:38:24.058 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cpu_threads": "4", 2024-09-11T03:38:24.058 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_crush_min_required_version": "hammer", 2024-09-11T03:38:24.058 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_daemon_bytes": "419430400", 2024-09-11T03:38:24.058 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-11T03:38:24.058 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_data_avail_crit": "5", 2024-09-11T03:38:24.058 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_data_avail_warn": "5", 2024-09-11T03:38:24.058 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_data_size_warn": "16106127360", 2024-09-11T03:38:24.058 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-11T03:38:24.058 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-11T03:38:24.058 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_dump_json": "false", 2024-09-11T03:38:24.058 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-11T03:38:24.058 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_dump_transactions": "false", 2024-09-11T03:38:24.059 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_extra_checks": "false", 2024-09-11T03:38:24.059 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-11T03:38:24.059 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-11T03:38:24.059 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_no_require_reef": "false", 2024-09-11T03:38:24.059 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_no_require_squid": "false", 2024-09-11T03:38:24.059 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-11T03:38:24.059 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-11T03:38:24.059 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-11T03:38:24.059 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_down_mkfs_grace": "60", 2024-09-11T03:38:24.059 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_down_uptime_grace": "60", 2024-09-11T03:38:24.059 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_election_default_strategy": "3", 2024-09-11T03:38:24.059 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_election_timeout": "5.000000", 2024-09-11T03:38:24.059 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-11T03:38:24.059 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_elector_ping_divisor": "2", 2024-09-11T03:38:24.060 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-11T03:38:24.060 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_enable_op_tracker": "true", 2024-09-11T03:38:24.060 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_fake_pool_delete": "false", 2024-09-11T03:38:24.060 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_force_quorum_join": "false", 2024-09-11T03:38:24.060 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-11T03:38:24.060 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_globalid_prealloc": "10000", 2024-09-11T03:38:24.060 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_detail_to_clog": "true", 2024-09-11T03:38:24.060 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_log_update_period": "5", 2024-09-11T03:38:24.060 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_max_detail": "50", 2024-09-11T03:38:24.060 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_to_clog": "true", 2024-09-11T03:38:24.060 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_to_clog_interval": "600", 2024-09-11T03:38:24.060 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-11T03:38:24.060 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_host": "[v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0]", 2024-09-11T03:38:24.060 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_host_override": "", 2024-09-11T03:38:24.061 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_initial_members": "", 2024-09-11T03:38:24.061 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-11T03:38:24.061 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-11T03:38:24.061 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-11T03:38:24.061 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-11T03:38:24.061 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-11T03:38:24.061 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_lease": "5.000000", 2024-09-11T03:38:24.061 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-11T03:38:24.061 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-11T03:38:24.061 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_log_full_interval": "50", 2024-09-11T03:38:24.061 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_log_max": "10000", 2024-09-11T03:38:24.061 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_log_max_summary": "50", 2024-09-11T03:38:24.061 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-11T03:38:24.062 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_log_epochs": "500", 2024-09-11T03:38:24.062 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-11T03:38:24.062 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-11T03:38:24.062 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-11T03:38:24.062 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_osd": "10000", 2024-09-11T03:38:24.062 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_pg_per_osd": "10000", 2024-09-11T03:38:24.062 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_pool_pg_num": "65536", 2024-09-11T03:38:24.062 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-11T03:38:24.062 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-11T03:38:24.062 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mds_force_trim_to": "0", 2024-09-11T03:38:24.062 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mds_skip_sanity": "false", 2024-09-11T03:38:24.062 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_memory_autotune": "true", 2024-09-11T03:38:24.062 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_memory_target": "2147483648", 2024-09-11T03:38:24.062 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_beacon_grace": "30", 2024-09-11T03:38:24.063 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-11T03:38:24.063 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_digest_period": "5", 2024-09-11T03:38:24.063 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_inactive_grace": "60", 2024-09-11T03:38:24.063 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-11T03:38:24.063 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-11T03:38:24.063 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_min_osdmap_epochs": "500", 2024-09-11T03:38:24.063 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-11T03:38:24.063 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-11T03:38:24.063 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_complaint_time": "30", 2024-09-11T03:38:24.063 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_history_duration": "600", 2024-09-11T03:38:24.063 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_history_size": "20", 2024-09-11T03:38:24.063 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_history_slow_op_size": "20", 2024-09-11T03:38:24.063 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-11T03:38:24.063 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_log_threshold": "5", 2024-09-11T03:38:24.064 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-11T03:38:24.064 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-11T03:38:24.064 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-11T03:38:24.064 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_auto_mark_in": "false", 2024-09-11T03:38:24.064 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-11T03:38:24.064 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-11T03:38:24.064 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-11T03:38:24.064 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_cache_size": "500", 2024-09-11T03:38:24.064 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-11T03:38:24.064 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-11T03:38:24.064 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-11T03:38:24.064 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_down_out_interval": "600", 2024-09-11T03:38:24.064 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-11T03:38:24.065 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-11T03:38:24.065 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_force_trim_to": "0", 2024-09-11T03:38:24.065 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-11T03:38:24.065 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-11T03:38:24.065 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-11T03:38:24.065 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-11T03:38:24.065 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-11T03:38:24.065 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-11T03:38:24.065 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-11T03:38:24.065 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_min_down_reporters": "2", 2024-09-11T03:38:24.065 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-11T03:38:24.065 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-11T03:38:24.065 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-11T03:38:24.065 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-11T03:38:24.066 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-11T03:38:24.066 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-11T03:38:24.066 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_report_timeout": "900", 2024-09-11T03:38:24.066 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-11T03:38:24.066 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-11T03:38:24.067 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-11T03:38:24.067 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-11T03:38:24.067 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-11T03:38:24.067 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-11T03:38:24.067 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-11T03:38:24.068 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-11T03:38:24.068 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-11T03:38:24.068 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_stuck_threshold": "60", 2024-09-11T03:38:24.068 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-11T03:38:24.068 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-11T03:38:24.068 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-11T03:38:24.068 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-11T03:38:24.068 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-11T03:38:24.068 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-11T03:38:24.068 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_probe_timeout": "2.000000", 2024-09-11T03:38:24.068 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_reweight_max_change": "0.050000", 2024-09-11T03:38:24.068 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_reweight_max_osds": "4", 2024-09-11T03:38:24.069 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-11T03:38:24.069 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-11T03:38:24.069 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-11T03:38:24.069 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-11T03:38:24.069 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-11T03:38:24.069 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_interval": "300", 2024-09-11T03:38:24.069 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_max_keys": "100", 2024-09-11T03:38:24.069 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_timeout": "300", 2024-09-11T03:38:24.069 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_session_timeout": "300", 2024-09-11T03:38:24.069 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_smart_report_timeout": "5", 2024-09-11T03:38:24.069 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stat_smooth_intervals": "6", 2024-09-11T03:38:24.069 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-11T03:38:24.069 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stretch_pool_min_size": "2", 2024-09-11T03:38:24.069 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stretch_pool_size": "4", 2024-09-11T03:38:24.070 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-11T03:38:24.070 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-11T03:38:24.070 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_debug": "false", 2024-09-11T03:38:24.070 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-11T03:38:24.070 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-11T03:38:24.070 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_provider_kill_at": "0", 2024-09-11T03:38:24.070 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_requester_kill_at": "0", 2024-09-11T03:38:24.070 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_timeout": "60.000000", 2024-09-11T03:38:24.070 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_target_pg_per_osd": "100", 2024-09-11T03:38:24.070 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_tick_interval": "5", 2024-09-11T03:38:24.070 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_timecheck_interval": "300.000000", 2024-09-11T03:38:24.070 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-11T03:38:24.070 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_use_min_delay_socket": "false", 2024-09-11T03:38:24.071 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_older_version_delay": "604800", 2024-09-11T03:38:24.071 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-11T03:38:24.071 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-11T03:38:24.071 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-11T03:38:24.071 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-11T03:38:24.071 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-11T03:38:24.071 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-11T03:38:24.071 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-11T03:38:24.071 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_misplaced": "false", 2024-09-11T03:38:24.071 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-11T03:38:24.071 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_older_version": "true", 2024-09-11T03:38:24.071 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-11T03:38:24.071 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-11T03:38:24.072 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-11T03:38:24.072 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-11T03:38:24.072 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-11T03:38:24.072 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-11T03:38:24.072 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-11T03:38:24.072 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-11T03:38:24.072 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-11T03:38:24.072 INFO:teuthology.orchestra.run.smithi084.stdout: "monmap": "", 2024-09-11T03:38:24.072 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-11T03:38:24.072 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-11T03:38:24.072 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-11T03:38:24.072 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-11T03:38:24.072 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-11T03:38:24.073 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-11T03:38:24.073 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_tracing_enabled": "false", 2024-09-11T03:38:24.073 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_op_threads": "3", 2024-09-11T03:38:24.073 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-11T03:38:24.073 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_cm": "false", 2024-09-11T03:38:24.073 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_device_name": "", 2024-09-11T03:38:24.073 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_dscp": "96", 2024-09-11T03:38:24.073 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-11T03:38:24.073 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-11T03:38:24.073 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_local_gid": "", 2024-09-11T03:38:24.073 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-11T03:38:24.073 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_port_num": "1", 2024-09-11T03:38:24.073 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-11T03:38:24.074 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-11T03:38:24.074 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-11T03:38:24.074 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-11T03:38:24.074 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_sl": "3", 2024-09-11T03:38:24.074 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_support_srq": "true", 2024-09-11T03:38:24.074 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_type": "ib", 2024-09-11T03:38:24.074 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_reap_threshold": "5", 2024-09-11T03:38:24.074 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_before_connect": "false", 2024-09-11T03:38:24.074 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_ipv4": "true", 2024-09-11T03:38:24.074 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_ipv6": "false", 2024-09-11T03:38:24.074 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_msgr1": "true", 2024-09-11T03:38:24.074 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_msgr2": "true", 2024-09-11T03:38:24.074 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_port_max": "7568", 2024-09-11T03:38:24.075 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_port_min": "6800", 2024-09-11T03:38:24.075 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-11T03:38:24.075 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_retry_count": "3", 2024-09-11T03:38:24.075 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_retry_delay": "5", 2024-09-11T03:38:24.075 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_client": "false", 2024-09-11T03:38:24.075 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_mds": "false", 2024-09-11T03:38:24.075 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_mgr": "false", 2024-09-11T03:38:24.075 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_mon": "false", 2024-09-11T03:38:24.075 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_osd": "false", 2024-09-11T03:38:24.075 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_client_mode": "crc secure", 2024-09-11T03:38:24.075 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-11T03:38:24.075 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_cluster_mode": "crc secure", 2024-09-11T03:38:24.075 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_cluster_type": "", 2024-09-11T03:38:24.075 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_compress_secure": "false", 2024-09-11T03:38:24.076 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_connection_idle_timeout": "900", 2024-09-11T03:38:24.076 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_connection_ready_timeout": "10", 2024-09-11T03:38:24.076 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_crc_data": "true", 2024-09-11T03:38:24.076 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_crc_header": "true", 2024-09-11T03:38:24.076 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_bad_msg": "false", 2024-09-11T03:38:24.076 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_bug": "true", 2024-09-11T03:38:24.076 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_old_message": "true", 2024-09-11T03:38:24.076 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_skipped_message": "false", 2024-09-11T03:38:24.076 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-11T03:38:24.076 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-11T03:38:24.076 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_coremask": "0xF", 2024-09-11T03:38:24.076 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-11T03:38:24.077 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-11T03:38:24.077 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_enable_tso": "true", 2024-09-11T03:38:24.077 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-11T03:38:24.077 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-11T03:38:24.077 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_hugepages": "", 2024-09-11T03:38:24.077 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-11T03:38:24.077 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-11T03:38:24.077 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_lro": "true", 2024-09-11T03:38:24.077 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_memory_channel": "4", 2024-09-11T03:38:24.077 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-11T03:38:24.077 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_pmd": "", 2024-09-11T03:38:24.077 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_port_id": "0", 2024-09-11T03:38:24.077 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-11T03:38:24.078 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-11T03:38:24.078 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dump_on_send": "false", 2024-09-11T03:38:24.078 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_initial_backoff": "0.200000", 2024-09-11T03:38:24.078 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_delay_max": "1.000000", 2024-09-11T03:38:24.078 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-11T03:38:24.078 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_delay_type": "", 2024-09-11T03:38:24.078 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-11T03:38:24.078 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_network_congestion": "0", 2024-09-11T03:38:24.078 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_socket_failures": "5000", 2024-09-11T03:38:24.078 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_learn_addr_from_peer": "true", 2024-09-11T03:38:24.078 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_max_accept_failures": "4", 2024-09-11T03:38:24.078 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_max_backoff": "15.000000", 2024-09-11T03:38:24.078 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_mon_client_mode": "secure crc", 2024-09-11T03:38:24.078 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-11T03:38:24.079 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_mon_service_mode": "secure crc", 2024-09-11T03:38:24.079 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_osd_compress_min_size": "1024", 2024-09-11T03:38:24.079 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_osd_compress_mode": "none", 2024-09-11T03:38:24.079 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-11T03:38:24.079 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-11T03:38:24.079 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_pq_min_cost": "65536", 2024-09-11T03:38:24.079 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_public_type": "", 2024-09-11T03:38:24.079 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_service_mode": "crc secure", 2024-09-11T03:38:24.079 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_tcp_listen_backlog": "512", 2024-09-11T03:38:24.079 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_tcp_nodelay": "true", 2024-09-11T03:38:24.079 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-11T03:38:24.079 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_tcp_rcvbuf": "0", 2024-09-11T03:38:24.079 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-11T03:38:24.080 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_type": "async+posix", 2024-09-11T03:38:24.080 INFO:teuthology.orchestra.run.smithi084.stdout: "no_config_file": "false", 2024-09-11T03:38:24.080 INFO:teuthology.orchestra.run.smithi084.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-11T03:38:24.080 INFO:teuthology.orchestra.run.smithi084.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-11T03:38:24.080 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_completion_locks_per_session": "32", 2024-09-11T03:38:24.080 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-11T03:38:24.080 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-11T03:38:24.080 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_inflight_ops": "1024", 2024-09-11T03:38:24.080 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-11T03:38:24.080 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-11T03:38:24.080 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_tick_interval": "5.000000", 2024-09-11T03:38:24.080 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_timeout": "10.000000", 2024-09-11T03:38:24.080 INFO:teuthology.orchestra.run.smithi084.stdout: "objectstore_blackhole": "false", 2024-09-11T03:38:24.080 INFO:teuthology.orchestra.run.smithi084.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-11T03:38:24.081 INFO:teuthology.orchestra.run.smithi084.stdout: "openssl_engine_opts": "", 2024-09-11T03:38:24.081 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_delay_time": "5.000000", 2024-09-11T03:38:24.081 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_hist_halflife": "1000", 2024-09-11T03:38:24.081 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_max_low_ops": "2", 2024-09-11T03:38:24.081 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_max_ops": "4", 2024-09-11T03:38:24.081 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-11T03:38:24.081 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-11T03:38:24.081 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_slop": "0.020000", 2024-09-11T03:38:24.081 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-11T03:38:24.081 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-11T03:38:24.081 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_asio_thread_count": "2", 2024-09-11T03:38:24.081 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_async_recovery_min_cost": "100", 2024-09-11T03:38:24.081 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-11T03:38:24.082 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-11T03:38:24.082 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backfill_scan_max": "512", 2024-09-11T03:38:24.082 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backfill_scan_min": "64", 2024-09-11T03:38:24.082 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backoff_on_degraded": "false", 2024-09-11T03:38:24.082 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backoff_on_peering": "false", 2024-09-11T03:38:24.082 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backoff_on_unfound": "true", 2024-09-11T03:38:24.082 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_beacon_report_interval": "300", 2024-09-11T03:38:24.082 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_bench_duration": "30", 2024-09-11T03:38:24.082 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-11T03:38:24.082 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_bench_max_block_size": "67108864", 2024-09-11T03:38:24.082 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-11T03:38:24.082 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_blkin_trace_all": "false", 2024-09-11T03:38:24.082 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-11T03:38:24.082 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-11T03:38:24.083 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-11T03:38:24.083 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-11T03:38:24.083 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_check_for_log_corruption": "false", 2024-09-11T03:38:24.083 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-11T03:38:24.083 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-11T03:38:24.083 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-11T03:38:24.083 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-11T03:38:24.083 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_class_update_on_start": "true", 2024-09-11T03:38:24.083 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_client_message_cap": "256", 2024-09-11T03:38:24.083 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_client_message_size_cap": "524288000", 2024-09-11T03:38:24.083 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_client_op_priority": "63", 2024-09-11T03:38:24.083 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_client_watch_timeout": "30", 2024-09-11T03:38:24.084 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_command_max_records": "256", 2024-09-11T03:38:24.084 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_compact_on_start": "false", 2024-09-11T03:38:24.084 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-11T03:38:24.084 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-11T03:38:24.084 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-11T03:38:24.084 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_crush_update_on_start": "true", 2024-09-11T03:38:24.084 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_crush_update_weight_set": "true", 2024-09-11T03:38:24.084 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-11T03:38:24.084 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-11T03:38:24.084 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-11T03:38:24.084 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-11T03:38:24.084 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-11T03:38:24.084 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_feed_pullee": "-1", 2024-09-11T03:38:24.084 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-11T03:38:24.085 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-11T03:38:24.085 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-11T03:38:24.085 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_misdirected_ops": "false", 2024-09-11T03:38:24.085 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_no_acting_change": "false", 2024-09-11T03:38:24.085 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_no_purge_strays": "false", 2024-09-11T03:38:24.085 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_op_order": "false", 2024-09-11T03:38:24.085 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-11T03:38:24.085 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-11T03:38:24.085 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-11T03:38:24.085 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-11T03:38:24.085 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_shutdown": "false", 2024-09-11T03:38:24.085 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-11T03:38:24.085 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-11T03:38:24.086 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_trim_objects": "false", 2024-09-11T03:38:24.086 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-11T03:38:24.086 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-11T03:38:24.086 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_verify_snaps": "false", 2024-09-11T03:38:24.086 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-11T03:38:24.086 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-11T03:38:24.086 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_keys": "1024", 2024-09-11T03:38:24.086 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-11T03:38:24.086 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-11T03:38:24.086 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-11T03:38:24.086 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_stride": "524288", 2024-09-11T03:38:24.086 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-11T03:38:24.086 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-11T03:38:24.087 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_default_notify_timeout": "30", 2024-09-11T03:38:24.087 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_delete_sleep": "0.000000", 2024-09-11T03:38:24.087 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-11T03:38:24.087 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-11T03:38:24.087 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-11T03:38:24.087 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_discard_disconnected_ops": "true", 2024-09-11T03:38:24.087 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_ec_partial_reads": "true", 2024-09-11T03:38:24.087 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_enable_op_tracker": "true", 2024-09-11T03:38:24.087 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-11T03:38:24.087 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-11T03:38:24.087 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-11T03:38:24.087 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-11T03:38:24.087 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_info": "true", 2024-09-11T03:38:24.087 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_shutdown": "true", 2024-09-11T03:38:24.088 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-11T03:38:24.088 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-11T03:38:24.088 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-11T03:38:24.088 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-11T03:38:24.088 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-11T03:38:24.088 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_function_tracing": "false", 2024-09-11T03:38:24.088 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_grace": "20", 2024-09-11T03:38:24.088 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_interval": "6", 2024-09-11T03:38:24.088 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-11T03:38:24.088 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_min_peers": "10", 2024-09-11T03:38:24.088 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_min_size": "2000", 2024-09-11T03:38:24.088 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_stale": "600", 2024-09-11T03:38:24.088 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-11T03:38:24.089 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_hit_set_max_size": "100000", 2024-09-11T03:38:24.089 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_hit_set_min_size": "1000", 2024-09-11T03:38:24.089 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-11T03:38:24.089 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-11T03:38:24.089 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-11T03:38:24.089 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-11T03:38:24.089 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-11T03:38:24.089 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-11T03:38:24.089 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_journal_size": "5120", 2024-09-11T03:38:24.089 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_kill_backfill_at": "0", 2024-09-11T03:38:24.089 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-11T03:38:24.089 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_cache_size": "50", 2024-09-11T03:38:24.089 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_dedup": "true", 2024-09-11T03:38:24.090 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_message_max": "40", 2024-09-11T03:38:24.090 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-11T03:38:24.090 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_share_max_epochs": "40", 2024-09-11T03:38:24.090 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_attr_name_len": "100", 2024-09-11T03:38:24.090 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_attr_size": "0", 2024-09-11T03:38:24.090 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_backfills": "1", 2024-09-11T03:38:24.090 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_markdown_count": "5", 2024-09-11T03:38:24.090 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_markdown_period": "600", 2024-09-11T03:38:24.090 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_object_name_len": "2048", 2024-09-11T03:38:24.090 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_object_namespace_len": "256", 2024-09-11T03:38:24.090 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_object_size": "134217728", 2024-09-11T03:38:24.090 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-11T03:38:24.090 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-11T03:38:24.091 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_pg_blocked_by": "16", 2024-09-11T03:38:24.091 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_pg_log_entries": "10000", 2024-09-11T03:38:24.091 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-11T03:38:24.091 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_pgls": "1024", 2024-09-11T03:38:24.091 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_push_cost": "8388608", 2024-09-11T03:38:24.091 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_push_objects": "10", 2024-09-11T03:38:24.091 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_scrubs": "3", 2024-09-11T03:38:24.091 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-11T03:38:24.091 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_trimming_pgs": "2", 2024-09-11T03:38:24.091 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_write_op_reply_len": "64", 2024-09-11T03:38:24.091 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_write_size": "90", 2024-09-11T03:38:24.091 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-11T03:38:24.091 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-11T03:38:24.091 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-11T03:38:24.092 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-11T03:38:24.092 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-11T03:38:24.092 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-11T03:38:24.092 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-11T03:38:24.092 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-11T03:38:24.092 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_profile": "balanced", 2024-09-11T03:38:24.092 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-11T03:38:24.092 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-11T03:38:24.092 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-11T03:38:24.092 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-11T03:38:24.092 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-11T03:38:24.092 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-11T03:38:24.092 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-11T03:38:24.093 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-11T03:38:24.093 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-11T03:38:24.093 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-11T03:38:24.093 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-11T03:38:24.093 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_base": "805306368", 2024-09-11T03:38:24.093 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_cache_min": "134217728", 2024-09-11T03:38:24.093 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-11T03:38:24.093 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-11T03:38:24.093 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_target": "4294967296", 2024-09-11T03:38:24.093 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_target_autotune": "false", 2024-09-11T03:38:24.093 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-11T03:38:24.093 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_min_pg_log_entries": "250", 2024-09-11T03:38:24.093 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_min_recovery_priority": "0", 2024-09-11T03:38:24.094 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-11T03:38:24.094 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-11T03:38:24.094 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_report_interval": "5", 2024-09-11T03:38:24.094 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-11T03:38:24.094 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-11T03:38:24.094 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_num_cache_shards": "32", 2024-09-11T03:38:24.094 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_num_op_tracker_shard": "32", 2024-09-11T03:38:24.094 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_numa_auto_affinity": "true", 2024-09-11T03:38:24.094 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_numa_node": "-1", 2024-09-11T03:38:24.094 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_numa_prefer_iface": "true", 2024-09-11T03:38:24.094 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-11T03:38:24.094 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_objecter_finishers": "1", 2024-09-11T03:38:24.094 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_objectstore": "bluestore", 2024-09-11T03:38:24.094 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_objectstore_fuse": "false", 2024-09-11T03:38:24.095 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-11T03:38:24.095 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_objectstore_tracing": "false", 2024-09-11T03:38:24.095 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_complaint_time": "30.000000", 2024-09-11T03:38:24.095 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_history_duration": "600", 2024-09-11T03:38:24.095 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_history_size": "20", 2024-09-11T03:38:24.095 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_history_slow_op_size": "20", 2024-09-11T03:38:24.095 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-11T03:38:24.095 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_log_threshold": "5", 2024-09-11T03:38:24.095 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_shards": "0", 2024-09-11T03:38:24.095 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_shards_hdd": "5", 2024-09-11T03:38:24.095 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_shards_ssd": "8", 2024-09-11T03:38:24.095 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-11T03:38:24.095 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-11T03:38:24.095 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-11T03:38:24.096 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-11T03:38:24.096 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_pq_min_cost": "65536", 2024-09-11T03:38:24.096 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-11T03:38:24.096 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_queue_cut_off": "high", 2024-09-11T03:38:24.096 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-11T03:38:24.096 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_thread_timeout": "15", 2024-09-11T03:38:24.096 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_open_classes_on_start": "true", 2024-09-11T03:38:24.096 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_os_flags": "0", 2024-09-11T03:38:24.096 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_peering_op_priority": "255", 2024-09-11T03:38:24.096 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_delete_cost": "1048576", 2024-09-11T03:38:24.096 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_delete_priority": "5", 2024-09-11T03:38:24.096 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-11T03:38:24.096 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-11T03:38:24.097 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-11T03:38:24.097 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_log_trim_max": "10000", 2024-09-11T03:38:24.097 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_log_trim_min": "100", 2024-09-11T03:38:24.097 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-11T03:38:24.097 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-11T03:38:24.097 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-11T03:38:24.097 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-11T03:38:24.097 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-11T03:38:24.097 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-11T03:38:24.097 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-11T03:38:24.097 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-11T03:38:24.097 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-11T03:38:24.098 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-11T03:38:24.098 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_crimson": "false", 2024-09-11T03:38:24.098 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-11T03:38:24.098 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-11T03:38:24.098 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-11T03:38:24.098 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-11T03:38:24.098 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-11T03:38:24.098 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-11T03:38:24.098 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-11T03:38:24.098 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-11T03:38:24.098 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flags": "0", 2024-09-11T03:38:24.098 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-11T03:38:24.098 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_min_size": "2", 2024-09-11T03:38:24.098 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-11T03:38:24.099 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_pg_num": "32", 2024-09-11T03:38:24.099 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_pgp_num": "0", 2024-09-11T03:38:24.099 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-11T03:38:24.099 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_read_ratio": "70", 2024-09-11T03:38:24.099 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_size": "3", 2024-09-11T03:38:24.099 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_type": "replicated", 2024-09-11T03:38:24.099 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-11T03:38:24.099 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-11T03:38:24.099 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_push_per_object_cost": "1000", 2024-09-11T03:38:24.099 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-11T03:38:24.099 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recover_clone_overlap": "true", 2024-09-11T03:38:24.099 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-11T03:38:24.099 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_cost": "20971520", 2024-09-11T03:38:24.099 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-11T03:38:24.100 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_active": "0", 2024-09-11T03:38:24.100 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-11T03:38:24.100 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-11T03:38:24.100 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-11T03:38:24.100 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-11T03:38:24.100 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_single_start": "1", 2024-09-11T03:38:24.100 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_op_priority": "3", 2024-09-11T03:38:24.100 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-11T03:38:24.100 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_priority": "5", 2024-09-11T03:38:24.100 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-11T03:38:24.100 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_sleep": "0.000000", 2024-09-11T03:38:24.100 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-11T03:38:24.100 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-11T03:38:24.101 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-11T03:38:24.101 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_repair_during_recovery": "false", 2024-09-11T03:38:24.101 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_requested_scrub_priority": "120", 2024-09-11T03:38:24.101 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-11T03:38:24.101 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-11T03:38:24.101 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_auto_repair": "false", 2024-09-11T03:38:24.101 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-11T03:38:24.101 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-11T03:38:24.101 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_begin_hour": "0", 2024-09-11T03:38:24.101 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_begin_week_day": "0", 2024-09-11T03:38:24.101 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_chunk_max": "25", 2024-09-11T03:38:24.101 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_chunk_min": "5", 2024-09-11T03:38:24.102 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_cost": "52428800", 2024-09-11T03:38:24.102 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-11T03:38:24.102 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_during_recovery": "false", 2024-09-11T03:38:24.102 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_end_hour": "0", 2024-09-11T03:38:24.102 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_end_week_day": "0", 2024-09-11T03:38:24.102 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_event_cost": "4096", 2024-09-11T03:38:24.102 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-11T03:38:24.102 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-11T03:38:24.102 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_invalid_stats": "true", 2024-09-11T03:38:24.102 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-11T03:38:24.102 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-11T03:38:24.102 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_max_preemptions": "5", 2024-09-11T03:38:24.102 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-11T03:38:24.102 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_priority": "5", 2024-09-11T03:38:24.103 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-11T03:38:24.103 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_retry_delay": "30", 2024-09-11T03:38:24.103 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-11T03:38:24.103 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-11T03:38:24.103 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_retry_trimming": "10", 2024-09-11T03:38:24.103 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_sleep": "0.000000", 2024-09-11T03:38:24.103 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-11T03:38:24.103 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-11T03:38:24.103 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-11T03:38:24.103 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-11T03:38:24.103 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_skip_data_digest": "false", 2024-09-11T03:38:24.103 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_smart_report_timeout": "5", 2024-09-11T03:38:24.103 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_cost": "1048576", 2024-09-11T03:38:24.103 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_priority": "5", 2024-09-11T03:38:24.104 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-11T03:38:24.104 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-11T03:38:24.104 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-11T03:38:24.104 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-11T03:38:24.104 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-11T03:38:24.104 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-11T03:38:24.104 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-11T03:38:24.104 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_target_transaction_size": "30", 2024-09-11T03:38:24.104 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-11T03:38:24.104 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-11T03:38:24.104 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-11T03:38:24.104 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-11T03:38:24.105 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-11T03:38:24.105 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-11T03:38:24.105 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-11T03:38:24.105 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-11T03:38:24.105 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-11T03:38:24.105 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-11T03:38:24.105 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tracing": "false", 2024-09-11T03:38:24.105 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_use_stale_snap": "false", 2024-09-11T03:38:24.105 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-11T03:38:24.105 INFO:teuthology.orchestra.run.smithi084.stdout: "osdc_blkin_trace_all": "false", 2024-09-11T03:38:24.105 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_kill_at": "0", 2024-09-11T03:38:24.105 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_max_join_drift": "10", 2024-09-11T03:38:24.105 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_min": "500", 2024-09-11T03:38:24.105 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_min_wait": "0.050000", 2024-09-11T03:38:24.106 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_propose_interval": "1.000000", 2024-09-11T03:38:24.106 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_service_trim_max": "500", 2024-09-11T03:38:24.106 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-11T03:38:24.106 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_service_trim_min": "250", 2024-09-11T03:38:24.106 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_stash_full_interval": "25", 2024-09-11T03:38:24.106 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_trim_max": "500", 2024-09-11T03:38:24.106 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_trim_min": "250", 2024-09-11T03:38:24.106 INFO:teuthology.orchestra.run.smithi084.stdout: "perf": "true", 2024-09-11T03:38:24.106 INFO:teuthology.orchestra.run.smithi084.stdout: "pid_file": "", 2024-09-11T03:38:24.106 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-11T03:38:24.106 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-11T03:38:24.106 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "-", 2024-09-11T03:38:24.106 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrv": "", 2024-09-11T03:38:24.107 INFO:teuthology.orchestra.run.smithi084.stdout: "public_bind_addr": "-", 2024-09-11T03:38:24.107 INFO:teuthology.orchestra.run.smithi084.stdout: "public_network": "", 2024-09-11T03:38:24.107 INFO:teuthology.orchestra.run.smithi084.stdout: "public_network_interface": "", 2024-09-11T03:38:24.107 INFO:teuthology.orchestra.run.smithi084.stdout: "qat_compressor_busy_polling": "false", 2024-09-11T03:38:24.107 INFO:teuthology.orchestra.run.smithi084.stdout: "qat_compressor_enabled": "false", 2024-09-11T03:38:24.107 INFO:teuthology.orchestra.run.smithi084.stdout: "qat_compressor_session_max_number": "256", 2024-09-11T03:38:24.107 INFO:teuthology.orchestra.run.smithi084.stdout: "rados_mon_op_timeout": "0", 2024-09-11T03:38:24.107 INFO:teuthology.orchestra.run.smithi084.stdout: "rados_osd_op_timeout": "0", 2024-09-11T03:38:24.107 INFO:teuthology.orchestra.run.smithi084.stdout: "rados_tracing": "false", 2024-09-11T03:38:24.107 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_atime_update_interval": "60", 2024-09-11T03:38:24.107 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-11T03:38:24.107 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_balance_parent_reads": "false", 2024-09-11T03:38:24.107 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_balance_snap_reads": "false", 2024-09-11T03:38:24.107 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_blkin_trace_all": "false", 2024-09-11T03:38:24.108 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-11T03:38:24.108 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-11T03:38:24.108 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache": "true", 2024-09-11T03:38:24.108 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-11T03:38:24.108 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-11T03:38:24.108 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-11T03:38:24.108 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-11T03:38:24.108 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_policy": "writearound", 2024-09-11T03:38:24.108 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_size": "33554432", 2024-09-11T03:38:24.108 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-11T03:38:24.108 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-11T03:38:24.108 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_clone_copy_on_read": "false", 2024-09-11T03:38:24.108 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_compression_hint": "none", 2024-09-11T03:38:24.109 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_concurrent_management_ops": "10", 2024-09-11T03:38:24.109 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-11T03:38:24.109 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_clone_format": "auto", 2024-09-11T03:38:24.109 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_data_pool": "", 2024-09-11T03:38:24.109 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_features": "61", 2024-09-11T03:38:24.109 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_format": "2", 2024-09-11T03:38:24.109 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_map_options": "", 2024-09-11T03:38:24.109 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_order": "22", 2024-09-11T03:38:24.109 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_pool": "rbd", 2024-09-11T03:38:24.109 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-11T03:38:24.109 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_stripe_count": "0", 2024-09-11T03:38:24.109 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_stripe_unit": "0", 2024-09-11T03:38:24.109 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-11T03:38:24.109 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-11T03:38:24.110 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-11T03:38:24.110 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_enable_alloc_hint": "true", 2024-09-11T03:38:24.110 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-11T03:38:24.110 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_io_scheduler": "simple", 2024-09-11T03:38:24.110 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-11T03:38:24.110 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-11T03:38:24.110 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-11T03:38:24.110 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-11T03:38:24.110 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-11T03:38:24.110 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-11T03:38:24.110 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-11T03:38:24.110 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-11T03:38:24.110 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-11T03:38:24.110 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_order": "24", 2024-09-11T03:38:24.111 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_pool": "", 2024-09-11T03:38:24.111 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_splay_width": "4", 2024-09-11T03:38:24.111 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_localize_parent_reads": "false", 2024-09-11T03:38:24.111 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_localize_snap_reads": "false", 2024-09-11T03:38:24.111 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-11T03:38:24.111 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-11T03:38:24.111 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-11T03:38:24.111 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-11T03:38:24.111 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-11T03:38:24.111 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-11T03:38:24.111 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-11T03:38:24.111 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-11T03:38:24.111 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-11T03:38:24.112 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-11T03:38:24.112 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-11T03:38:24.112 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-11T03:38:24.112 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-11T03:38:24.112 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-11T03:38:24.112 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-11T03:38:24.112 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-11T03:38:24.112 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-11T03:38:24.112 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-11T03:38:24.112 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-11T03:38:24.112 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-11T03:38:24.112 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-11T03:38:24.112 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-11T03:38:24.113 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-11T03:38:24.113 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-11T03:38:24.113 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-11T03:38:24.113 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-11T03:38:24.113 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-11T03:38:24.113 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-11T03:38:24.113 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-11T03:38:24.113 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-11T03:38:24.113 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-11T03:38:24.113 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mtime_update_interval": "60", 2024-09-11T03:38:24.113 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_non_blocking_aio": "true", 2024-09-11T03:38:24.113 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_op_thread_timeout": "60", 2024-09-11T03:38:24.113 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_op_threads": "1", 2024-09-11T03:38:24.114 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_parent_cache_enabled": "false", 2024-09-11T03:38:24.114 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-11T03:38:24.114 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-11T03:38:24.114 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-11T03:38:24.114 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_plugins": "", 2024-09-11T03:38:24.114 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_bps_burst": "0", 2024-09-11T03:38:24.114 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-11T03:38:24.114 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_bps_limit": "0", 2024-09-11T03:38:24.114 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_exclude_ops": "0", 2024-09-11T03:38:24.114 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_iops_burst": "0", 2024-09-11T03:38:24.114 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-11T03:38:24.114 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_iops_limit": "0", 2024-09-11T03:38:24.114 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-11T03:38:24.114 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-11T03:38:24.115 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-11T03:38:24.115 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-11T03:38:24.115 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-11T03:38:24.115 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-11T03:38:24.115 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-11T03:38:24.115 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-11T03:38:24.115 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-11T03:38:24.115 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-11T03:38:24.115 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-11T03:38:24.115 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-11T03:38:24.115 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-11T03:38:24.115 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-11T03:38:24.115 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_read_from_replica_policy": "default", 2024-09-11T03:38:24.116 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-11T03:38:24.116 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-11T03:38:24.116 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-11T03:38:24.116 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-11T03:38:24.116 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_skip_partial_discard": "true", 2024-09-11T03:38:24.116 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-11T03:38:24.116 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_tracing": "false", 2024-09-11T03:38:24.116 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_validate_names": "true", 2024-09-11T03:38:24.116 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_validate_pool": "true", 2024-09-11T03:38:24.116 INFO:teuthology.orchestra.run.smithi084.stdout: "restapi_base_url": "", 2024-09-11T03:38:24.116 INFO:teuthology.orchestra.run.smithi084.stdout: "restapi_log_level": "", 2024-09-11T03:38:24.116 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-11T03:38:24.116 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-11T03:38:24.117 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_acl_grants_max_num": "100", 2024-09-11T03:38:24.117 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_admin_entry": "admin", 2024-09-11T03:38:24.117 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-11T03:38:24.117 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_asio_assert_yielding": "false", 2024-09-11T03:38:24.117 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_backend_store": "rados", 2024-09-11T03:38:24.117 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_barbican_url": "", 2024-09-11T03:38:24.117 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_beast_enable_async": "true", 2024-09-11T03:38:24.117 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_counters_cache": "false", 2024-09-11T03:38:24.117 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-11T03:38:24.117 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-11T03:38:24.117 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-11T03:38:24.117 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-11T03:38:24.117 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-11T03:38:24.117 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-11T03:38:24.118 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-11T03:38:24.118 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-11T03:38:24.118 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cache_enabled": "true", 2024-09-11T03:38:24.118 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cache_expiry_interval": "900", 2024-09-11T03:38:24.118 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cache_lru_size": "25000", 2024-09-11T03:38:24.118 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_config_store": "rados", 2024-09-11T03:38:24.118 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_content_length_compat": "false", 2024-09-11T03:38:24.118 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_copy_obj_progress": "true", 2024-09-11T03:38:24.118 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-11T03:38:24.118 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cors_rules_max_num": "100", 2024-09-11T03:38:24.118 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cross_domain_policy": "", 2024-09-11T03:38:24.118 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-11T03:38:24.118 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_addr": "", 2024-09-11T03:38:24.118 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-11T03:38:24.118 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-11T03:38:24.119 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-11T03:38:24.119 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-11T03:38:24.119 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_password": "", 2024-09-11T03:38:24.119 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-11T03:38:24.119 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_username": "", 2024-09-11T03:38:24.119 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_require_ssl": "true", 2024-09-11T03:38:24.119 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-11T03:38:24.119 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-11T03:38:24.119 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-11T03:38:24.119 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-11T03:38:24.119 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-11T03:38:24.119 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-11T03:38:24.119 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-11T03:38:24.119 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-11T03:38:24.120 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-11T03:38:24.120 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-11T03:38:24.120 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-11T03:38:24.120 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-11T03:38:24.120 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-11T03:38:24.120 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-11T03:38:24.120 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-11T03:38:24.120 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_addr": "", 2024-09-11T03:38:24.120 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_auth": "token", 2024-09-11T03:38:24.120 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_namespace": "", 2024-09-11T03:38:24.120 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_prefix": "", 2024-09-11T03:38:24.120 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-11T03:38:24.120 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-11T03:38:24.121 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-11T03:38:24.121 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-11T03:38:24.121 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_token_file": "", 2024-09-11T03:38:24.121 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-11T03:38:24.121 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_buffersize": "524288", 2024-09-11T03:38:24.121 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-11T03:38:24.121 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_low_speed_time": "300", 2024-09-11T03:38:24.121 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-11T03:38:24.121 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-11T03:38:24.121 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-11T03:38:24.121 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-11T03:38:24.121 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-11T03:38:24.121 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-11T03:38:24.122 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-11T03:38:24.122 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-11T03:38:24.122 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-11T03:38:24.122 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-11T03:38:24.122 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-11T03:38:24.122 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-11T03:38:24.122 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-11T03:38:24.122 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-11T03:38:24.122 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-11T03:38:24.122 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-11T03:38:24.122 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-11T03:38:24.122 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-11T03:38:24.122 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-11T03:38:24.122 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_log_changes_size": "1000", 2024-09-11T03:38:24.123 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_log_num_shards": "128", 2024-09-11T03:38:24.123 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-11T03:38:24.123 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_log_window": "30", 2024-09-11T03:38:24.123 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-11T03:38:24.123 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-11T03:38:24.123 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-11T03:38:24.123 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-11T03:38:24.123 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-11T03:38:24.123 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-11T03:38:24.123 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-11T03:38:24.123 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-11T03:38:24.123 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-11T03:38:24.123 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-11T03:38:24.123 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-11T03:38:24.124 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-11T03:38:24.124 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-11T03:38:24.124 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_disable_s3select": "false", 2024-09-11T03:38:24.124 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-11T03:38:24.124 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-11T03:38:24.124 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-11T03:38:24.124 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-11T03:38:24.124 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-11T03:38:24.124 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-11T03:38:24.124 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-11T03:38:24.124 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-11T03:38:24.124 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-11T03:38:24.124 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-11T03:38:24.125 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-11T03:38:24.125 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-11T03:38:24.125 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dns_name": "", 2024-09-11T03:38:24.125 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dns_s3website_name": "", 2024-09-11T03:38:24.125 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dynamic_resharding": "true", 2024-09-11T03:38:24.125 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-11T03:38:24.125 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-11T03:38:24.125 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-11T03:38:24.125 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_gc_threads": "true", 2024-09-11T03:38:24.125 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_lc_threads": "true", 2024-09-11T03:38:24.125 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_mdsearch": "true", 2024-09-11T03:38:24.125 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_ops_log": "false", 2024-09-11T03:38:24.125 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_quota_threads": "true", 2024-09-11T03:38:24.126 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_static_website": "false", 2024-09-11T03:38:24.126 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_usage_log": "false", 2024-09-11T03:38:24.126 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enforce_swift_acls": "true", 2024-09-11T03:38:24.126 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_exit_timeout_secs": "120", 2024-09-11T03:38:24.126 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_expose_bucket": "false", 2024-09-11T03:38:24.126 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_extended_http_attrs": "", 2024-09-11T03:38:24.126 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_filter": "none", 2024-09-11T03:38:24.126 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-11T03:38:24.126 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_frontends": "beast port=7480", 2024-09-11T03:38:24.126 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-11T03:38:24.126 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_deferred": "50", 2024-09-11T03:38:24.126 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-11T03:38:24.126 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_objs": "32", 2024-09-11T03:38:24.126 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-11T03:38:24.127 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-11T03:38:24.127 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-11T03:38:24.127 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-11T03:38:24.127 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_processor_period": "3600", 2024-09-11T03:38:24.127 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-11T03:38:24.127 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-11T03:38:24.127 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-11T03:38:24.127 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-11T03:38:24.127 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_init_timeout": "300", 2024-09-11T03:38:24.127 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-11T03:38:24.127 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-11T03:38:24.127 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_kafka_connection_idle": "300", 2024-09-11T03:38:24.127 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-11T03:38:24.128 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-11T03:38:24.128 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-11T03:38:24.128 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-11T03:38:24.128 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-11T03:38:24.128 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_domain": "", 2024-09-11T03:38:24.128 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_password": "", 2024-09-11T03:38:24.128 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_password_path": "", 2024-09-11T03:38:24.128 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_project": "", 2024-09-11T03:38:24.128 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_tenant": "", 2024-09-11T03:38:24.128 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_token": "", 2024-09-11T03:38:24.128 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_token_path": "", 2024-09-11T03:38:24.128 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_user": "", 2024-09-11T03:38:24.128 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_domain": "", 2024-09-11T03:38:24.128 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_password": "", 2024-09-11T03:38:24.129 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_project": "", 2024-09-11T03:38:24.129 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-11T03:38:24.129 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_user": "", 2024-09-11T03:38:24.129 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-11T03:38:24.129 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-11T03:38:24.129 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-11T03:38:24.129 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-11T03:38:24.129 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-11T03:38:24.129 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_url": "", 2024-09-11T03:38:24.129 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-11T03:38:24.129 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_debug_interval": "-1", 2024-09-11T03:38:24.129 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_lock_max_time": "90", 2024-09-11T03:38:24.129 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_max_objs": "32", 2024-09-11T03:38:24.129 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_max_rules": "1000", 2024-09-11T03:38:24.130 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_max_worker": "3", 2024-09-11T03:38:24.130 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-11T03:38:24.130 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_thread_delay": "0", 2024-09-11T03:38:24.130 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-11T03:38:24.130 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_dnattr": "uid", 2024-09-11T03:38:24.130 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-11T03:38:24.130 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_searchfilter": "", 2024-09-11T03:38:24.130 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-11T03:38:24.130 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_uri": "", 2024-09-11T03:38:24.130 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-11T03:38:24.130 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-11T03:38:24.130 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-11T03:38:24.130 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-11T03:38:24.130 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_log_http_headers": "", 2024-09-11T03:38:24.131 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-11T03:38:24.131 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-11T03:38:24.131 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_log_object_name_utc": "false", 2024-09-11T03:38:24.131 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-11T03:38:24.131 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-11T03:38:24.131 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_attr_name_len": "0", 2024-09-11T03:38:24.131 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_attr_size": "0", 2024-09-11T03:38:24.131 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-11T03:38:24.131 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_chunk_size": "4194304", 2024-09-11T03:38:24.131 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-11T03:38:24.131 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-11T03:38:24.131 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-11T03:38:24.131 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_listing_results": "5000", 2024-09-11T03:38:24.131 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_notify_retries": "10", 2024-09-11T03:38:24.132 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-11T03:38:24.132 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_put_param_size": "1048576", 2024-09-11T03:38:24.132 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_put_size": "5368709120", 2024-09-11T03:38:24.132 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_slo_entries": "1000", 2024-09-11T03:38:24.132 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_md_log_max_shards": "64", 2024-09-11T03:38:24.132 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-11T03:38:24.132 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-11T03:38:24.132 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-11T03:38:24.132 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-11T03:38:24.132 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_mp_lock_max_time": "600", 2024-09-11T03:38:24.132 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-11T03:38:24.132 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-11T03:38:24.132 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-11T03:38:24.133 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-11T03:38:24.133 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-11T03:38:24.133 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-11T03:38:24.133 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-11T03:38:24.133 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-11T03:38:24.133 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_max_gc": "300", 2024-09-11T03:38:24.133 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-11T03:38:24.133 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-11T03:38:24.133 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-11T03:38:24.133 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-11T03:38:24.133 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-11T03:38:24.133 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-11T03:38:24.133 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-11T03:38:24.134 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_num_async_rados_threads": "32", 2024-09-11T03:38:24.134 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_num_control_oids": "8", 2024-09-11T03:38:24.134 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_numa_node": "-1", 2024-09-11T03:38:24.134 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-11T03:38:24.134 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-11T03:38:24.134 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_objexp_chunk_size": "100", 2024-09-11T03:38:24.134 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_objexp_gc_interval": "600", 2024-09-11T03:38:24.134 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-11T03:38:24.134 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-11T03:38:24.134 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-11T03:38:24.134 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_op_thread_timeout": "600", 2024-09-11T03:38:24.134 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_op_tracing": "false", 2024-09-11T03:38:24.134 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_opa_token": "", 2024-09-11T03:38:24.134 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_opa_url": "", 2024-09-11T03:38:24.135 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_opa_verify_ssl": "true", 2024-09-11T03:38:24.135 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-11T03:38:24.135 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-11T03:38:24.135 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ops_log_rados": "false", 2024-09-11T03:38:24.135 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ops_log_socket_path": "", 2024-09-11T03:38:24.135 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-11T03:38:24.135 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-11T03:38:24.135 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-11T03:38:24.135 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_period_push_interval": "2.000000", 2024-09-11T03:38:24.135 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-11T03:38:24.135 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-11T03:38:24.135 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-11T03:38:24.135 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-11T03:38:24.136 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_cache_lanes": "3", 2024-09-11T03:38:24.136 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-11T03:38:24.136 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-11T03:38:24.136 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_cache_partitions": "3", 2024-09-11T03:38:24.136 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-11T03:38:24.136 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_print_continue": "true", 2024-09-11T03:38:24.136 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-11T03:38:24.136 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-11T03:38:24.136 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-11T03:38:24.136 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-11T03:38:24.136 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-11T03:38:24.136 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_rados_tracing": "false", 2024-09-11T03:38:24.136 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_realm": "", 2024-09-11T03:38:24.137 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_realm_id": "", 2024-09-11T03:38:24.137 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-11T03:38:24.137 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_region": "", 2024-09-11T03:38:24.137 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-11T03:38:24.137 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-11T03:38:24.137 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-11T03:38:24.137 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-11T03:38:24.137 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_request_uri": "", 2024-09-11T03:38:24.137 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_batch_size": "64", 2024-09-11T03:38:24.137 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-11T03:38:24.137 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-11T03:38:24.137 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_max_aio": "128", 2024-09-11T03:38:24.137 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_num_logs": "16", 2024-09-11T03:38:24.137 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-11T03:38:24.138 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-11T03:38:24.138 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_thread_interval": "600", 2024-09-11T03:38:24.138 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-11T03:38:24.138 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_resolve_cname": "false", 2024-09-11T03:38:24.138 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-11T03:38:24.138 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_run_sync_thread": "true", 2024-09-11T03:38:24.138 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-11T03:38:24.138 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-11T03:38:24.138 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-11T03:38:24.138 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-11T03:38:24.138 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-11T03:38:24.138 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-11T03:38:24.138 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-11T03:38:24.138 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-11T03:38:24.139 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-11T03:38:24.139 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_scheduler_type": "throttler", 2024-09-11T03:38:24.139 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_script_uri": "", 2024-09-11T03:38:24.139 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_service_provider_name": "", 2024-09-11T03:38:24.139 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-11T03:38:24.139 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_client_id": "", 2024-09-11T03:38:24.139 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_client_secret": "", 2024-09-11T03:38:24.139 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_entry": "sts", 2024-09-11T03:38:24.139 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_key": "sts", 2024-09-11T03:38:24.139 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-11T03:38:24.139 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_min_session_duration": "900", 2024-09-11T03:38:24.139 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_token_introspection_url": "", 2024-09-11T03:38:24.139 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_account_in_url": "false", 2024-09-11T03:38:24.139 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_auth_entry": "auth", 2024-09-11T03:38:24.140 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_auth_url": "", 2024-09-11T03:38:24.140 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_custom_header": "", 2024-09-11T03:38:24.140 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-11T03:38:24.140 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_need_stats": "true", 2024-09-11T03:38:24.140 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_tenant_name": "", 2024-09-11T03:38:24.140 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_token_expiration": "86400", 2024-09-11T03:38:24.140 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_url": "", 2024-09-11T03:38:24.140 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_url_prefix": "swift", 2024-09-11T03:38:24.140 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-11T03:38:24.140 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-11T03:38:24.140 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-11T03:38:24.140 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_lease_period": "120", 2024-09-11T03:38:24.140 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-11T03:38:24.141 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-11T03:38:24.141 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-11T03:38:24.141 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-11T03:38:24.141 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-11T03:38:24.141 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-11T03:38:24.141 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-11T03:38:24.141 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-11T03:38:24.141 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-11T03:38:24.141 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_thread_pool_size": "512", 2024-09-11T03:38:24.141 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-11T03:38:24.141 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-11T03:38:24.141 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-11T03:38:24.141 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-11T03:38:24.142 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_comment": "", 2024-09-11T03:38:24.142 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_createby": "", 2024-09-11T03:38:24.142 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_encoding": "", 2024-09-11T03:38:24.142 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_flag": "false", 2024-09-11T03:38:24.142 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-11T03:38:24.142 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_origin": "", 2024-09-11T03:38:24.142 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-11T03:38:24.142 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_tracker": "", 2024-09-11T03:38:24.142 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_trust_forwarded_https": "false", 2024-09-11T03:38:24.142 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-11T03:38:24.142 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-11T03:38:24.142 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_usage_max_shards": "32", 2024-09-11T03:38:24.142 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_usage_max_user_shards": "1", 2024-09-11T03:38:24.142 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_use_opa_authz": "false", 2024-09-11T03:38:24.143 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_counters_cache": "false", 2024-09-11T03:38:24.143 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-11T03:38:24.143 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-11T03:38:24.143 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-11T03:38:24.143 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_max_buckets": "1000", 2024-09-11T03:38:24.143 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_policies_max_num": "100", 2024-09-11T03:38:24.143 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-11T03:38:24.143 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-11T03:38:24.143 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-11T03:38:24.143 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-11T03:38:24.143 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_unique_email": "true", 2024-09-11T03:38:24.143 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_verify_ssl": "true", 2024-09-11T03:38:24.143 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-11T03:38:24.144 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zone": "", 2024-09-11T03:38:24.144 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zone_id": "", 2024-09-11T03:38:24.144 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-11T03:38:24.144 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zonegroup": "", 2024-09-11T03:38:24.144 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zonegroup_id": "", 2024-09-11T03:38:24.144 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-11T03:38:24.144 INFO:teuthology.orchestra.run.smithi084.stdout: "rgwlc_auto_session_clear": "true", 2024-09-11T03:38:24.144 INFO:teuthology.orchestra.run.smithi084.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-11T03:38:24.144 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_block_size": "4096", 2024-09-11T03:38:24.144 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-11T03:38:24.144 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-11T03:38:24.144 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-11T03:38:24.144 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-11T03:38:24.144 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-11T03:38:24.145 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_size": "536870912", 2024-09-11T03:38:24.145 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-11T03:38:24.145 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-11T03:38:24.145 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-11T03:38:24.145 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-11T03:38:24.145 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-11T03:38:24.145 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-11T03:38:24.145 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-11T03:38:24.145 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-11T03:38:24.145 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_index_type": "binary_search", 2024-09-11T03:38:24.145 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-11T03:38:24.145 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-11T03:38:24.145 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_partition_filters": "false", 2024-09-11T03:38:24.145 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_perf": "false", 2024-09-11T03:38:24.146 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-11T03:38:24.146 INFO:teuthology.orchestra.run.smithi084.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-11T03:38:24.146 INFO:teuthology.orchestra.run.smithi084.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-11T03:38:24.146 INFO:teuthology.orchestra.run.smithi084.stdout: "run_dir": "/var/run/ceph", 2024-09-11T03:38:24.146 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_block_create": "true", 2024-09-11T03:38:24.146 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_cache_lru_size": "67108864", 2024-09-11T03:38:24.146 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-11T03:38:24.146 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-11T03:38:24.146 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_default_max_object_size": "16777216", 2024-09-11T03:38:24.146 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-11T03:38:24.146 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_device_size": "53687091200", 2024-09-11T03:38:24.146 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-11T03:38:24.146 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_full_integrity_check": "false", 2024-09-11T03:38:24.146 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_journal_batch_capacity": "16", 2024-09-11T03:38:24.147 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-11T03:38:24.147 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-11T03:38:24.147 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-11T03:38:24.147 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_main_device_type": "SSD", 2024-09-11T03:38:24.147 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-11T03:38:24.147 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-11T03:38:24.147 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-11T03:38:24.147 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-11T03:38:24.147 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-11T03:38:24.147 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-11T03:38:24.147 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_segment_size": "67108864", 2024-09-11T03:38:24.147 INFO:teuthology.orchestra.run.smithi084.stdout: "set_keepcaps": "false", 2024-09-11T03:38:24.147 INFO:teuthology.orchestra.run.smithi084.stdout: "setgroup": "ceph", 2024-09-11T03:38:24.147 INFO:teuthology.orchestra.run.smithi084.stdout: "setuser": "ceph", 2024-09-11T03:38:24.148 INFO:teuthology.orchestra.run.smithi084.stdout: "setuser_match_path": "", 2024-09-11T03:38:24.148 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-11T03:38:24.148 INFO:teuthology.orchestra.run.smithi084.stdout: "thp": "false", 2024-09-11T03:38:24.148 INFO:teuthology.orchestra.run.smithi084.stdout: "threadpool_default_timeout": "60", 2024-09-11T03:38:24.148 INFO:teuthology.orchestra.run.smithi084.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-11T03:38:24.148 INFO:teuthology.orchestra.run.smithi084.stdout: "throttler_perf_counter": "true", 2024-09-11T03:38:24.148 INFO:teuthology.orchestra.run.smithi084.stdout: "tmp_dir": "/tmp", 2024-09-11T03:38:24.148 INFO:teuthology.orchestra.run.smithi084.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-11T03:38:24.148 INFO:teuthology.orchestra.run.smithi084.stdout: "uadk_compressor_enabled": "false", 2024-09-11T03:38:24.148 INFO:teuthology.orchestra.run.smithi084.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-11T03:38:24.148 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:38:24.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:23 smithi121 ceph-mon[30729]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:24.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:23 smithi121 ceph-mon[30729]: osdmap e93: 12 total, 12 up, 12 in 2024-09-11T03:38:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:23 smithi175 ceph-mon[30894]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:23 smithi175 ceph-mon[30894]: osdmap e93: 12 total, 12 up, 12 in 2024-09-11T03:38:24.881 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-11T03:38:25.123 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:25.610 INFO:teuthology.orchestra.run.smithi084.stdout:mon.a: {} 2024-09-11T03:38:25.610 INFO:teuthology.orchestra.run.smithi084.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T03:38:25.620 INFO:teuthology.orchestra.run.smithi084.stdout:mon.c: {} 2024-09-11T03:38:25.620 INFO:teuthology.orchestra.run.smithi084.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T03:38:25.630 INFO:teuthology.orchestra.run.smithi084.stdout:mon.b: {} 2024-09-11T03:38:25.630 INFO:teuthology.orchestra.run.smithi084.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T03:38:26.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:25 smithi084 ceph-mon[27254]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:26.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:25 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1481484729' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-11T03:38:26.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:25 smithi121 ceph-mon[30729]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:26.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:25 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1481484729' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-11T03:38:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:25 smithi175 ceph-mon[30894]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:25 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1481484729' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-11T03:38:26.415 INFO:teuthology.run_tasks:Running task rados... 2024-09-11T03:38:26.428 INFO:tasks.rados:Beginning rados... 2024-09-11T03:38:26.428 INFO:tasks.rados:config is {'clients': ['client.2'], 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2024-09-11T03:38:26.428 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-hemanth-testing-2024-09-10-0723'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'filestore queue throttle high multiple': 2, 'filestore queue throttle max multiple': 10, 'journal throttle high multiple': 2, 'journal throttle max multiple': 10, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd heartbeat use min delay socket': True, 'osd max backfills': 9, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-hemanth-testing-2024-09-10-0723', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}} 2024-09-11T03:38:26.428 INFO:tasks.rados:config is {'clients': ['client.2'], 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2024-09-11T03:38:26.428 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-11T03:38:26.448 INFO:tasks.rados:joining rados 2024-09-11T03:38:26.448 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-11T03:38:26.448 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:38:26.448 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-11T03:38:26.449 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-11T03:38:26.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:26.450 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-11T03:38:26.450 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:38:26.450 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-11T03:38:26.450 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T03:38:26.451 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-11T03:38:26.451 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:38:26.451 INFO:tasks.rados:clients are ['client.0', 'client.1', 'client.2'] 2024-09-11T03:38:26.452 INFO:tasks.rados:starting run 0 out of 1 2024-09-11T03:38:26.452 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-11T03:38:26.452 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-11T03:38:26.479 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:26.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:26.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:26.581 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:38:26.614 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:26.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:26.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:38:26.716 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:26.738 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:26.743 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:26.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:26.745 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:26.845 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:38:26.871 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:26.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:26.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:38:26.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:26.999 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:26.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:27.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:27.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:38:27.123 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:27.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:38:27.225 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:27.248 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:27.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:27.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:27.281 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:27.282 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:27.351 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:27.376 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:27.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:27.377 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:27.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:27.504 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:27.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:27.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:27.606 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:27.628 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:27.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:27.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:27.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:27.755 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:27.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:27.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:27.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:27.874 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T03:38:27.880 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:27.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:27.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:27.982 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:28.008 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:28.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:28.009 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:28.111 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:38:28.133 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:28.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:28.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:38:28.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:27 smithi084 ceph-mon[27254]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:27 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1629846546' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:38:28.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:27 smithi121 ceph-mon[30729]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:28.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:27 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1629846546' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:38:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:27 smithi175 ceph-mon[30894]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:27 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1629846546' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:38:28.235 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:28.260 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:28.261 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:28.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:28.388 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:28.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:28.491 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:28.517 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:28.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:28.518 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:28.592 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-11T03:38:28.618 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:38:28.643 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:28.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:28.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:38:28.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:28.772 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:28.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:28.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:28.805 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:38:28.805 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:38:28.806 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:38:28.806 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:38:28.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:28.900 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:28.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:28.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:29.002 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:29.028 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:29.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:29.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:29.116 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_0' created 2024-09-11T03:38:29.129 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:38:29.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:28 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:38:29.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:28 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:38:29.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:28 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1629846546' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:38:29.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:28 smithi084 ceph-mon[27254]: osdmap e94: 12 total, 12 up, 12 in 2024-09-11T03:38:29.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:28 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1274735048' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-11T03:38:29.151 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:29.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:29.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:38:29.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:28 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:38:29.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:28 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:38:29.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:28 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1629846546' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:38:29.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:28 smithi121 ceph-mon[30729]: osdmap e94: 12 total, 12 up, 12 in 2024-09-11T03:38:29.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:28 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1274735048' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-11T03:38:29.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:28 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:38:29.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:28 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:38:29.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:28 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1629846546' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:38:29.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:28 smithi175 ceph-mon[30894]: osdmap e94: 12 total, 12 up, 12 in 2024-09-11T03:38:29.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:28 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1274735048' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-11T03:38:29.253 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:29.279 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:29.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:29.280 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:29.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:29.407 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:29.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:29.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:29.509 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:29.534 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:29.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:29.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:29.636 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:29.662 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:29.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:29.663 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:29.764 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:29.790 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:29.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:29.791 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:29.800 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-11T03:38:29.891 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:29.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:29.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:30.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:30.043 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:30.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:30.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:30.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:29 smithi084 ceph-mon[27254]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:30.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:29 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1274735048' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-11T03:38:30.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:29 smithi084 ceph-mon[27254]: osdmap e95: 12 total, 12 up, 12 in 2024-09-11T03:38:30.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:30.169 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:30.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:30.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:30.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:29 smithi121 ceph-mon[30729]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:30.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:29 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1274735048' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-11T03:38:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:29 smithi121 ceph-mon[30729]: osdmap e95: 12 total, 12 up, 12 in 2024-09-11T03:38:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:29 smithi175 ceph-mon[30894]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:29 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1274735048' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-11T03:38:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:29 smithi175 ceph-mon[30894]: osdmap e95: 12 total, 12 up, 12 in 2024-09-11T03:38:30.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:30.295 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:30.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:30.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:30.325 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:30.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T03:38:30.334 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:30.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:30.422 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:30.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:30.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:30.524 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:30.545 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:30.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:30.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:30.648 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:30.673 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:30.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:30.775 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:30.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:30.798 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:30.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:30.850 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:38:30.850 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:38:30.857 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:30.868 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:38:30.869 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:38:30.893 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:38:30.893 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:38:30.896 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:30.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:30.911 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:38:30.911 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:38:30.923 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:30.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:30.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:30.931 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:38:30.931 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:38:30.949 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:38:30.949 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:38:30.965 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:38:30.965 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:38:30.982 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:38:30.982 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:38:31.000 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:38:31.000 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:38:31.017 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:38:31.017 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:38:31.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:31.034 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:38:31.035 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:38:31.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:31.049 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:31.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:31.052 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:38:31.052 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:38:31.150 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:31.159 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:30 smithi084 ceph-mon[27254]: osdmap e96: 12 total, 12 up, 12 in 2024-09-11T03:38:31.159 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:30 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1292268875' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:38:31.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:30 smithi121 ceph-mon[30729]: osdmap e96: 12 total, 12 up, 12 in 2024-09-11T03:38:31.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:30 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1292268875' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:38:31.175 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:31.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:31.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:31.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:30 smithi175 ceph-mon[30894]: osdmap e96: 12 total, 12 up, 12 in 2024-09-11T03:38:31.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:30 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1292268875' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:38:31.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:31.302 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:31.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:31.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:31.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:38:31.427 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:31.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:31.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:38:31.529 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:31.554 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:31.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:31.555 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:31.656 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:31.681 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:31.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:31.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:31.784 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:31.809 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:31.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:31.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:31.912 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:31.938 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:31.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:31.939 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:32.040 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:32.066 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:32.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:32.067 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:32.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:31 smithi084 ceph-mon[27254]: pgmap v298: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:32.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:31 smithi084 ceph-mon[27254]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T03:38:32.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:31 smithi084 ceph-mon[27254]: osdmap e97: 12 total, 12 up, 12 in 2024-09-11T03:38:32.167 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:32.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:31 smithi121 ceph-mon[30729]: pgmap v298: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:32.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:31 smithi121 ceph-mon[30729]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T03:38:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:31 smithi121 ceph-mon[30729]: osdmap e97: 12 total, 12 up, 12 in 2024-09-11T03:38:32.193 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:32.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:32.194 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:31 smithi175 ceph-mon[30894]: pgmap v298: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:31 smithi175 ceph-mon[30894]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T03:38:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:31 smithi175 ceph-mon[30894]: osdmap e97: 12 total, 12 up, 12 in 2024-09-11T03:38:32.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:32.321 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:32.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:32.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:32.423 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:32.448 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:32.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:32.449 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:32.498 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:32.537 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:38:32.537 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:38:32.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:32.572 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:32.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:32.675 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:32.696 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:32.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:32.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:32.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:32.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:32.825 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:32.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:32.904 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:38:32.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:32.952 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:32.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:32.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:33.053 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:33.080 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:33.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:33.081 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:33.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:32 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4034796972' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:38:33.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:32 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4034796972' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:38:33.182 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:38:33.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:33.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:33.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:38:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:32 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4034796972' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:38:33.303 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:33.324 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:33.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:33.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:33.426 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:33.447 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:33.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:33.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:33.549 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:33.575 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:33.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:33.576 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:33.677 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:33.702 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:33.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:33.703 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:33.804 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:38:33.824 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:33.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:33.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:38:33.887 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:38:33.926 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:33.951 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:33.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:33.952 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:34.053 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:34.058 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T03:38:34.058 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:33 smithi084 ceph-mon[27254]: pgmap v300: 17 pgs: 14 active+clean, 3 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:34.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:33 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:38:34.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:33 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4034796972' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:38:34.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:33 smithi084 ceph-mon[27254]: osdmap e98: 12 total, 12 up, 12 in 2024-09-11T03:38:34.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:33 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2497653401' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:38:34.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:33 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3121154258' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-11T03:38:34.078 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:34.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:34.079 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:34.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:33 smithi121 ceph-mon[30729]: pgmap v300: 17 pgs: 14 active+clean, 3 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:34.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:33 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:38:34.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:33 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4034796972' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:38:34.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:33 smithi121 ceph-mon[30729]: osdmap e98: 12 total, 12 up, 12 in 2024-09-11T03:38:34.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:33 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2497653401' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:38:34.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:33 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3121154258' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-11T03:38:34.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:34.206 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:34.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:34.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:33 smithi175 ceph-mon[30894]: pgmap v300: 17 pgs: 14 active+clean, 3 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:33 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:38:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:33 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4034796972' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:38:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:33 smithi175 ceph-mon[30894]: osdmap e98: 12 total, 12 up, 12 in 2024-09-11T03:38:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:33 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2497653401' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:38:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:33 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3121154258' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-11T03:38:34.308 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:38:34.333 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:34.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:34.334 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:38:34.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:34.461 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:34.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:34.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:34.563 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:34.588 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:34.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:34.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:34.613 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-11T03:38:34.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:34.711 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:34.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:34.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:34.809 INFO:teuthology.orchestra.run.smithi084.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-11T03:38:34.813 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:34.839 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:34.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:34.840 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:34.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:34.967 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:34.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:34.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:35.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:35.095 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:35.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:35.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:34 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:38:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:34 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:38:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:34 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2497653401' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:38:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:34 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3121154258' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T03:38:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:34 smithi121 ceph-mon[30729]: osdmap e99: 12 total, 12 up, 12 in 2024-09-11T03:38:35.196 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:35.222 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:35.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:35.223 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:34 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:38:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:34 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:38:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:34 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2497653401' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:38:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:34 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3121154258' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T03:38:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:34 smithi175 ceph-mon[30894]: osdmap e99: 12 total, 12 up, 12 in 2024-09-11T03:38:35.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:35.350 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:35.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:35.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:34 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:38:35.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:34 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:38:35.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:34 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2497653401' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:38:35.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:34 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3121154258' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-11T03:38:35.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:34 smithi084 ceph-mon[27254]: osdmap e99: 12 total, 12 up, 12 in 2024-09-11T03:38:35.452 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:35.478 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:35.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:35.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:35.579 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:35.602 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:35.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:35.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:35.704 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:35.730 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:35.731 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:35.764 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:38:35.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:38:35.852 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:35.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:35.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:38:35.904 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T03:38:35.931 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:35.954 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:35.979 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:35.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:35.980 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:36.052 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:35 smithi084 ceph-mon[27254]: pgmap v303: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:36.052 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:35 smithi084 ceph-mon[27254]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T03:38:36.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:36.107 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:36.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:36.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:36.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:35 smithi121 ceph-mon[30729]: pgmap v303: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:36.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:35 smithi121 ceph-mon[30729]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T03:38:36.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:36.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:35 smithi175 ceph-mon[30894]: pgmap v303: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:36.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:35 smithi175 ceph-mon[30894]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T03:38:36.234 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:36.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:36.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:36.336 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:36.362 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:36.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:36.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:36.464 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:36.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:36.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:36.541 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:36.541 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:36.558 DEBUG:teuthology.orchestra.run.smithi175:> CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0 2024-09-11T03:38:36.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:36.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:36.610 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:36.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:36.658 INFO:tasks.rados.rados.2.smithi175.stdout:adding op weight read -> 100 2024-09-11T03:38:36.658 INFO:tasks.rados.rados.2.smithi175.stdout:adding op weight write -> 50 2024-09-11T03:38:36.659 INFO:tasks.rados.rados.2.smithi175.stdout:adding op weight delete -> 50 2024-09-11T03:38:36.659 INFO:tasks.rados.rados.2.smithi175.stdout:adding op weight snap_create -> 50 2024-09-11T03:38:36.659 INFO:tasks.rados.rados.2.smithi175.stdout:adding op weight snap_remove -> 50 2024-09-11T03:38:36.659 INFO:tasks.rados.rados.2.smithi175.stdout:adding op weight rollback -> 50 2024-09-11T03:38:36.659 INFO:tasks.rados.rados.2.smithi175.stdout:adding op weight copy_from -> 50 2024-09-11T03:38:36.659 INFO:tasks.rados.rados.2.smithi175.stdout:adding op weight write_excl -> 50 2024-09-11T03:38:36.659 INFO:tasks.rados.rados.2.smithi175.stdout:ceph version 18.2.4-837-g7a853b17 (7a853b1706d076d58e0da7a5d3b9b9eb36276880) reef (stable) 2024-09-11T03:38:36.659 INFO:tasks.rados.rados.2.smithi175.stdout:Configuration: 2024-09-11T03:38:36.659 INFO:tasks.rados.rados.2.smithi175.stdout: Number of operations: 4000 2024-09-11T03:38:36.659 INFO:tasks.rados.rados.2.smithi175.stdout: Number of objects: 50 2024-09-11T03:38:36.659 INFO:tasks.rados.rados.2.smithi175.stdout: Max in flight operations: 16 2024-09-11T03:38:36.659 INFO:tasks.rados.rados.2.smithi175.stdout: Object size (in bytes): 4000000 2024-09-11T03:38:36.659 INFO:tasks.rados.rados.2.smithi175.stdout: Write stride min: 400000 2024-09-11T03:38:36.660 INFO:tasks.rados.rados.2.smithi175.stdout: Write stride max: 800000 2024-09-11T03:38:36.711 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:36.732 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:36.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:36.761 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:36.834 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:36.860 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:36.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:36.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:36.915 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T03:38:36.915 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2806890797' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:38:36.915 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.175:0/3465170716' entity='client.2' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"} : dispatch 2024-09-11T03:38:36.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:36 smithi084 ceph-mon[27254]: from='client.? ' entity='client.2' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"} : dispatch 2024-09-11T03:38:36.917 INFO:tasks.rados.rados.2.smithi175.stdout:1: write initial oid 1 2024-09-11T03:38:36.917 INFO:tasks.rados.rados.2.smithi175.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-09-11T03:38:36.922 INFO:tasks.rados.rados.2.smithi175.stdout:1: writing smithi17557870-1 from 416807 to 892056 tid 1 2024-09-11T03:38:36.929 INFO:tasks.rados.rados.2.smithi175.stdout:1: writing smithi17557870-1 from 1542129 to 2016807 tid 2 2024-09-11T03:38:36.929 INFO:tasks.rados.rados.2.smithi175.stdout:2: write initial oid 2 2024-09-11T03:38:36.929 INFO:tasks.rados.rados.2.smithi175.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-09-11T03:38:36.934 INFO:tasks.rados.rados.2.smithi175.stdout:2: writing smithi17557870-2 from 416807 to 892056 tid 1 2024-09-11T03:38:36.940 INFO:tasks.rados.rados.2.smithi175.stdout:2: writing smithi17557870-2 from 1542129 to 2016807 tid 2 2024-09-11T03:38:36.940 INFO:tasks.rados.rados.2.smithi175.stdout:3: write initial oid 3 2024-09-11T03:38:36.940 INFO:tasks.rados.rados.2.smithi175.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-09-11T03:38:36.945 INFO:tasks.rados.rados.2.smithi175.stdout:3: writing smithi17557870-3 from 433614 to 984112 tid 1 2024-09-11T03:38:36.951 INFO:tasks.rados.rados.2.smithi175.stdout:3: writing smithi17557870-3 from 1600611 to 2033614 tid 2 2024-09-11T03:38:36.951 INFO:tasks.rados.rados.2.smithi175.stdout:4: write initial oid 4 2024-09-11T03:38:36.952 INFO:tasks.rados.rados.2.smithi175.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-09-11T03:38:36.957 INFO:tasks.rados.rados.2.smithi175.stdout:4: writing smithi17557870-4 from 450421 to 1076168 tid 1 2024-09-11T03:38:36.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:36.963 INFO:tasks.rados.rados.2.smithi175.stdout:4: writing smithi17557870-4 from 1659093 to 2050421 tid 2 2024-09-11T03:38:36.963 INFO:tasks.rados.rados.2.smithi175.stdout:5: write initial oid 5 2024-09-11T03:38:36.963 INFO:tasks.rados.rados.2.smithi175.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-09-11T03:38:36.969 INFO:tasks.rados.rados.2.smithi175.stdout:5: writing smithi17557870-5 from 467228 to 1168224 tid 1 2024-09-11T03:38:36.974 INFO:tasks.rados.rados.2.smithi175.stdout:5: writing smithi17557870-5 from 1717575 to 2067228 tid 2 2024-09-11T03:38:36.974 INFO:tasks.rados.rados.2.smithi175.stdout:6: write initial oid 6 2024-09-11T03:38:36.974 INFO:tasks.rados.rados.2.smithi175.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-09-11T03:38:36.981 INFO:tasks.rados.rados.2.smithi175.stdout:6: writing smithi17557870-6 from 484035 to 1260280 tid 1 2024-09-11T03:38:36.985 INFO:tasks.rados.rados.2.smithi175.stdout:6: writing smithi17557870-6 from 2059704 to 2084035 tid 2 2024-09-11T03:38:36.985 INFO:tasks.rados.rados.2.smithi175.stdout:7: write initial oid 7 2024-09-11T03:38:36.986 INFO:tasks.rados.rados.2.smithi175.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-09-11T03:38:36.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:36.987 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:36.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:36.990 INFO:tasks.rados.rados.2.smithi175.stdout:7: writing smithi17557870-7 from 500842 to 952336 tid 1 2024-09-11T03:38:36.997 INFO:tasks.rados.rados.2.smithi175.stdout:7: writing smithi17557870-7 from 1718186 to 2100842 tid 2 2024-09-11T03:38:36.997 INFO:tasks.rados.rados.2.smithi175.stdout:8: write initial oid 8 2024-09-11T03:38:36.997 INFO:tasks.rados.rados.2.smithi175.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-09-11T03:38:37.002 INFO:tasks.rados.rados.2.smithi175.stdout:8: writing smithi17557870-8 from 517649 to 1044392 tid 1 2024-09-11T03:38:37.008 INFO:tasks.rados.rados.2.smithi175.stdout:8: writing smithi17557870-8 from 1776668 to 2117649 tid 2 2024-09-11T03:38:37.008 INFO:tasks.rados.rados.2.smithi175.stdout:9: write initial oid 9 2024-09-11T03:38:37.009 INFO:tasks.rados.rados.2.smithi175.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-09-11T03:38:37.015 INFO:tasks.rados.rados.2.smithi175.stdout:9: writing smithi17557870-9 from 534456 to 1252801 tid 1 2024-09-11T03:38:37.020 INFO:tasks.rados.rados.2.smithi175.stdout:9: writing smithi17557870-9 from 1951503 to 2134456 tid 2 2024-09-11T03:38:37.020 INFO:tasks.rados.rados.2.smithi175.stdout:10: write initial oid 10 2024-09-11T03:38:37.021 INFO:tasks.rados.rados.2.smithi175.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-09-11T03:38:37.027 INFO:tasks.rados.rados.2.smithi175.stdout:10: writing smithi17557870-10 from 551263 to 1344857 tid 1 2024-09-11T03:38:37.032 INFO:tasks.rados.rados.2.smithi175.stdout:10: writing smithi17557870-10 from 1893632 to 2151263 tid 2 2024-09-11T03:38:37.032 INFO:tasks.rados.rados.2.smithi175.stdout:11: write initial oid 11 2024-09-11T03:38:37.032 INFO:tasks.rados.rados.2.smithi175.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-09-11T03:38:37.037 INFO:tasks.rados.rados.2.smithi175.stdout:11: writing smithi17557870-11 from 568070 to 1036913 tid 1 2024-09-11T03:38:37.044 INFO:tasks.rados.rados.2.smithi175.stdout:11: writing smithi17557870-11 from 1552114 to 2168070 tid 2 2024-09-11T03:38:37.044 INFO:tasks.rados.rados.2.smithi175.stdout:12: write initial oid 12 2024-09-11T03:38:37.044 INFO:tasks.rados.rados.2.smithi175.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-09-11T03:38:37.050 INFO:tasks.rados.rados.2.smithi175.stdout:12: writing smithi17557870-12 from 584877 to 1128969 tid 1 2024-09-11T03:38:37.056 INFO:tasks.rados.rados.2.smithi175.stdout:12: writing smithi17557870-12 from 1610596 to 2172599 tid 2 2024-09-11T03:38:37.056 INFO:tasks.rados.rados.2.smithi175.stdout:13: write initial oid 13 2024-09-11T03:38:37.056 INFO:tasks.rados.rados.2.smithi175.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-09-11T03:38:37.062 INFO:tasks.rados.rados.2.smithi175.stdout:13: writing smithi17557870-13 from 601684 to 1221025 tid 1 2024-09-11T03:38:37.068 INFO:tasks.rados.rados.2.smithi175.stdout:13: writing smithi17557870-13 from 1669078 to 2201684 tid 2 2024-09-11T03:38:37.068 INFO:tasks.rados.rados.2.smithi175.stdout:14: write initial oid 14 2024-09-11T03:38:37.068 INFO:tasks.rados.rados.2.smithi175.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-09-11T03:38:37.075 INFO:tasks.rados.rados.2.smithi175.stdout:14: writing smithi17557870-14 from 618491 to 1313081 tid 1 2024-09-11T03:38:37.080 INFO:tasks.rados.rados.2.smithi175.stdout:14: writing smithi17557870-14 from 2011207 to 2218491 tid 2 2024-09-11T03:38:37.081 INFO:tasks.rados.rados.2.smithi175.stdout:15: write initial oid 15 2024-09-11T03:38:37.081 INFO:tasks.rados.rados.2.smithi175.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-09-11T03:38:37.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:37.088 INFO:tasks.rados.rados.2.smithi175.stdout:15: writing smithi17557870-15 from 635298 to 1405137 tid 1 2024-09-11T03:38:37.092 INFO:tasks.rados.rados.2.smithi175.stdout:15: writing smithi17557870-15 from 2069689 to 2235298 tid 2 2024-09-11T03:38:37.092 INFO:tasks.rados.rados.2.smithi175.stdout:16: write initial oid 16 2024-09-11T03:38:37.092 INFO:tasks.rados.rados.2.smithi175.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-09-11T03:38:37.098 INFO:tasks.rados.rados.2.smithi175.stdout:16: writing smithi17557870-16 from 652105 to 1097193 tid 1 2024-09-11T03:38:37.105 INFO:tasks.rados.rados.2.smithi175.stdout:16: writing smithi17557870-16 from 1728171 to 2181159 tid 2 2024-09-11T03:38:37.105 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.105 INFO:tasks.rados.rados.2.smithi175.stdout:1: finishing write tid 1 to smithi17557870-1 2024-09-11T03:38:37.105 INFO:tasks.rados.rados.2.smithi175.stdout:1: finishing write tid 2 to smithi17557870-1 2024-09-11T03:38:37.105 INFO:tasks.rados.rados.2.smithi175.stdout:1: finishing write tid 3 to smithi17557870-1 2024-09-11T03:38:37.105 INFO:tasks.rados.rados.2.smithi175.stdout:1: finishing write tid 4 to smithi17557870-1 2024-09-11T03:38:37.105 INFO:tasks.rados.rados.2.smithi175.stdout:1: oid 1 updating version 0 to 1 2024-09-11T03:38:37.105 INFO:tasks.rados.rados.2.smithi175.stdout:1: oid 1 updating version 1 to 2 2024-09-11T03:38:37.105 INFO:tasks.rados.rados.2.smithi175.stdout:1: oid 1 updating version 2 to 3 2024-09-11T03:38:37.105 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-11T03:38:37.106 INFO:tasks.rados.rados.2.smithi175.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-11T03:38:37.106 INFO:tasks.rados.rados.2.smithi175.stdout:2: finishing write tid 1 to smithi17557870-2 2024-09-11T03:38:37.106 INFO:tasks.rados.rados.2.smithi175.stdout:1: done (15 left) 2024-09-11T03:38:37.106 INFO:tasks.rados.rados.2.smithi175.stdout:17: write initial oid 17 2024-09-11T03:38:37.106 INFO:tasks.rados.rados.2.smithi175.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-09-11T03:38:37.109 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:37.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:37.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:37.111 INFO:tasks.rados.rados.2.smithi175.stdout:17: writing smithi17557870-17 from 668912 to 1305602 tid 1 2024-09-11T03:38:37.117 INFO:tasks.rados.rados.2.smithi175.stdout:17: writing smithi17557870-17 from 1903006 to 2268912 tid 2 2024-09-11T03:38:37.117 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.117 INFO:tasks.rados.rados.2.smithi175.stdout:2: finishing write tid 2 to smithi17557870-2 2024-09-11T03:38:37.117 INFO:tasks.rados.rados.2.smithi175.stdout:2: finishing write tid 3 to smithi17557870-2 2024-09-11T03:38:37.117 INFO:tasks.rados.rados.2.smithi175.stdout:2: finishing write tid 4 to smithi17557870-2 2024-09-11T03:38:37.117 INFO:tasks.rados.rados.2.smithi175.stdout:2: oid 2 updating version 0 to 2 2024-09-11T03:38:37.118 INFO:tasks.rados.rados.2.smithi175.stdout:2: oid 2 updating version 2 to 3 2024-09-11T03:38:37.118 INFO:tasks.rados.rados.2.smithi175.stdout:2: oid 2 version 3 is already newer than 1 2024-09-11T03:38:37.118 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T03:38:37.118 INFO:tasks.rados.rados.2.smithi175.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-11T03:38:37.118 INFO:tasks.rados.rados.2.smithi175.stdout:3: finishing write tid 1 to smithi17557870-3 2024-09-11T03:38:37.118 INFO:tasks.rados.rados.2.smithi175.stdout:3: finishing write tid 2 to smithi17557870-3 2024-09-11T03:38:37.118 INFO:tasks.rados.rados.2.smithi175.stdout:2: done (15 left) 2024-09-11T03:38:37.118 INFO:tasks.rados.rados.2.smithi175.stdout:18: write initial oid 18 2024-09-11T03:38:37.118 INFO:tasks.rados.rados.2.smithi175.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-09-11T03:38:37.124 INFO:tasks.rados.rados.2.smithi175.stdout:18: writing smithi17557870-18 from 685719 to 1397658 tid 1 2024-09-11T03:38:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:18: writing smithi17557870-18 from 1845135 to 2285719 tid 2 2024-09-11T03:38:37.130 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:3: finishing write tid 3 to smithi17557870-3 2024-09-11T03:38:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:3: finishing write tid 4 to smithi17557870-3 2024-09-11T03:38:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:3: oid 3 updating version 0 to 1 2024-09-11T03:38:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:3: oid 3 updating version 1 to 2 2024-09-11T03:38:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:3: oid 3 updating version 2 to 3 2024-09-11T03:38:37.131 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-11T03:38:37.131 INFO:tasks.rados.rados.2.smithi175.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-11T03:38:37.131 INFO:tasks.rados.rados.2.smithi175.stdout:4: finishing write tid 1 to smithi17557870-4 2024-09-11T03:38:37.131 INFO:tasks.rados.rados.2.smithi175.stdout:4: finishing write tid 2 to smithi17557870-4 2024-09-11T03:38:37.131 INFO:tasks.rados.rados.2.smithi175.stdout:3: done (15 left) 2024-09-11T03:38:37.131 INFO:tasks.rados.rados.2.smithi175.stdout:19: write initial oid 19 2024-09-11T03:38:37.131 INFO:tasks.rados.rados.2.smithi175.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-09-11T03:38:37.137 INFO:tasks.rados.rados.2.smithi175.stdout:19: writing smithi17557870-19 from 702526 to 1489714 tid 1 2024-09-11T03:38:37.142 INFO:tasks.rados.rados.2.smithi175.stdout:19: writing smithi17557870-19 from 1903617 to 2302526 tid 2 2024-09-11T03:38:37.142 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.142 INFO:tasks.rados.rados.2.smithi175.stdout:4: finishing write tid 3 to smithi17557870-4 2024-09-11T03:38:37.142 INFO:tasks.rados.rados.2.smithi175.stdout:4: finishing write tid 4 to smithi17557870-4 2024-09-11T03:38:37.142 INFO:tasks.rados.rados.2.smithi175.stdout:4: oid 4 updating version 0 to 2 2024-09-11T03:38:37.143 INFO:tasks.rados.rados.2.smithi175.stdout:4: oid 4 version 2 is already newer than 1 2024-09-11T03:38:37.143 INFO:tasks.rados.rados.2.smithi175.stdout:4: oid 4 updating version 2 to 3 2024-09-11T03:38:37.143 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-11T03:38:37.143 INFO:tasks.rados.rados.2.smithi175.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-11T03:38:37.143 INFO:tasks.rados.rados.2.smithi175.stdout:5: finishing write tid 1 to smithi17557870-5 2024-09-11T03:38:37.143 INFO:tasks.rados.rados.2.smithi175.stdout:5: finishing write tid 2 to smithi17557870-5 2024-09-11T03:38:37.143 INFO:tasks.rados.rados.2.smithi175.stdout:4: done (15 left) 2024-09-11T03:38:37.143 INFO:tasks.rados.rados.2.smithi175.stdout:20: write initial oid 20 2024-09-11T03:38:37.143 INFO:tasks.rados.rados.2.smithi175.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-09-11T03:38:37.148 INFO:tasks.rados.rados.2.smithi175.stdout:20: writing smithi17557870-20 from 719333 to 1181770 tid 1 2024-09-11T03:38:37.155 INFO:tasks.rados.rados.2.smithi175.stdout:20: writing smithi17557870-20 from 1962099 to 2319333 tid 2 2024-09-11T03:38:37.155 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.155 INFO:tasks.rados.rados.2.smithi175.stdout:5: finishing write tid 3 to smithi17557870-5 2024-09-11T03:38:37.155 INFO:tasks.rados.rados.2.smithi175.stdout:5: finishing write tid 4 to smithi17557870-5 2024-09-11T03:38:37.155 INFO:tasks.rados.rados.2.smithi175.stdout:5: oid 5 updating version 0 to 2 2024-09-11T03:38:37.155 INFO:tasks.rados.rados.2.smithi175.stdout:5: oid 5 version 2 is already newer than 1 2024-09-11T03:38:37.155 INFO:tasks.rados.rados.2.smithi175.stdout:5: oid 5 updating version 2 to 3 2024-09-11T03:38:37.155 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-11T03:38:37.155 INFO:tasks.rados.rados.2.smithi175.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-11T03:38:37.155 INFO:tasks.rados.rados.2.smithi175.stdout:6: finishing write tid 1 to smithi17557870-6 2024-09-11T03:38:37.156 INFO:tasks.rados.rados.2.smithi175.stdout:6: finishing write tid 2 to smithi17557870-6 2024-09-11T03:38:37.156 INFO:tasks.rados.rados.2.smithi175.stdout:5: done (15 left) 2024-09-11T03:38:37.156 INFO:tasks.rados.rados.2.smithi175.stdout:21: write initial oid 21 2024-09-11T03:38:37.156 INFO:tasks.rados.rados.2.smithi175.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-09-11T03:38:37.161 INFO:tasks.rados.rados.2.smithi175.stdout:21: writing smithi17557870-21 from 736140 to 1273826 tid 1 2024-09-11T03:38:37.168 INFO:tasks.rados.rados.2.smithi175.stdout:21: writing smithi17557870-21 from 2020581 to 2336140 tid 2 2024-09-11T03:38:37.168 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.168 INFO:tasks.rados.rados.2.smithi175.stdout:6: finishing write tid 3 to smithi17557870-6 2024-09-11T03:38:37.168 INFO:tasks.rados.rados.2.smithi175.stdout:6: finishing write tid 4 to smithi17557870-6 2024-09-11T03:38:37.168 INFO:tasks.rados.rados.2.smithi175.stdout:6: oid 6 updating version 0 to 5 2024-09-11T03:38:37.169 INFO:tasks.rados.rados.2.smithi175.stdout:6: oid 6 version 5 is already newer than 4 2024-09-11T03:38:37.169 INFO:tasks.rados.rados.2.smithi175.stdout:6: oid 6 updating version 5 to 6 2024-09-11T03:38:37.169 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-11T03:38:37.169 INFO:tasks.rados.rados.2.smithi175.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-11T03:38:37.169 INFO:tasks.rados.rados.2.smithi175.stdout:7: finishing write tid 1 to smithi17557870-7 2024-09-11T03:38:37.170 INFO:tasks.rados.rados.2.smithi175.stdout:7: finishing write tid 2 to smithi17557870-7 2024-09-11T03:38:37.170 INFO:tasks.rados.rados.2.smithi175.stdout:7: finishing write tid 3 to smithi17557870-7 2024-09-11T03:38:37.170 INFO:tasks.rados.rados.2.smithi175.stdout:7: finishing write tid 4 to smithi17557870-7 2024-09-11T03:38:37.170 INFO:tasks.rados.rados.2.smithi175.stdout:7: oid 7 updating version 0 to 1 2024-09-11T03:38:37.170 INFO:tasks.rados.rados.2.smithi175.stdout:7: oid 7 updating version 1 to 2 2024-09-11T03:38:37.170 INFO:tasks.rados.rados.2.smithi175.stdout:7: oid 7 updating version 2 to 3 2024-09-11T03:38:37.171 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-11T03:38:37.171 INFO:tasks.rados.rados.2.smithi175.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-11T03:38:37.171 INFO:tasks.rados.rados.2.smithi175.stdout:8: finishing write tid 1 to smithi17557870-8 2024-09-11T03:38:37.171 INFO:tasks.rados.rados.2.smithi175.stdout:6: done (15 left) 2024-09-11T03:38:37.171 INFO:tasks.rados.rados.2.smithi175.stdout:7: done (14 left) 2024-09-11T03:38:37.171 INFO:tasks.rados.rados.2.smithi175.stdout:22: write initial oid 22 2024-09-11T03:38:37.172 INFO:tasks.rados.rados.2.smithi175.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-09-11T03:38:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2806890797' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:38:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.175:0/3465170716' entity='client.2' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"} : dispatch 2024-09-11T03:38:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:36 smithi121 ceph-mon[30729]: from='client.? ' entity='client.2' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"} : dispatch 2024-09-11T03:38:37.175 INFO:tasks.rados.rados.2.smithi175.stdout:22: writing smithi17557870-22 from 752947 to 1365882 tid 1 2024-09-11T03:38:37.180 INFO:tasks.rados.rados.2.smithi175.stdout:22: writing smithi17557870-22 from 1962710 to 2352947 tid 2 2024-09-11T03:38:37.180 INFO:tasks.rados.rados.2.smithi175.stdout:23: write initial oid 23 2024-09-11T03:38:37.181 INFO:tasks.rados.rados.2.smithi175.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-09-11T03:38:37.188 INFO:tasks.rados.rados.2.smithi175.stdout:23: writing smithi17557870-23 from 769754 to 1457938 tid 1 2024-09-11T03:38:37.193 INFO:tasks.rados.rados.2.smithi175.stdout:23: writing smithi17557870-23 from 2021192 to 2369754 tid 2 2024-09-11T03:38:37.193 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.194 INFO:tasks.rados.rados.2.smithi175.stdout:8: finishing write tid 2 to smithi17557870-8 2024-09-11T03:38:37.194 INFO:tasks.rados.rados.2.smithi175.stdout:8: finishing write tid 3 to smithi17557870-8 2024-09-11T03:38:37.194 INFO:tasks.rados.rados.2.smithi175.stdout:8: finishing write tid 4 to smithi17557870-8 2024-09-11T03:38:37.194 INFO:tasks.rados.rados.2.smithi175.stdout:8: oid 8 updating version 0 to 1 2024-09-11T03:38:37.194 INFO:tasks.rados.rados.2.smithi175.stdout:8: oid 8 updating version 1 to 3 2024-09-11T03:38:37.194 INFO:tasks.rados.rados.2.smithi175.stdout:8: oid 8 version 3 is already newer than 2 2024-09-11T03:38:37.194 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-11T03:38:37.195 INFO:tasks.rados.rados.2.smithi175.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-11T03:38:37.195 INFO:tasks.rados.rados.2.smithi175.stdout:9: finishing write tid 1 to smithi17557870-9 2024-09-11T03:38:37.195 INFO:tasks.rados.rados.2.smithi175.stdout:9: finishing write tid 2 to smithi17557870-9 2024-09-11T03:38:37.195 INFO:tasks.rados.rados.2.smithi175.stdout:8: done (15 left) 2024-09-11T03:38:37.195 INFO:tasks.rados.rados.2.smithi175.stdout:24: write initial oid 24 2024-09-11T03:38:37.195 INFO:tasks.rados.rados.2.smithi175.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-09-11T03:38:37.200 INFO:tasks.rados.rados.2.smithi175.stdout:24: writing smithi17557870-24 from 786561 to 1266347 tid 1 2024-09-11T03:38:37.206 INFO:tasks.rados.rados.2.smithi175.stdout:24: writing smithi17557870-24 from 1796027 to 2263691 tid 2 2024-09-11T03:38:37.207 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.207 INFO:tasks.rados.rados.2.smithi175.stdout:9: finishing write tid 3 to smithi17557870-9 2024-09-11T03:38:37.207 INFO:tasks.rados.rados.2.smithi175.stdout:9: finishing write tid 4 to smithi17557870-9 2024-09-11T03:38:37.207 INFO:tasks.rados.rados.2.smithi175.stdout:9: oid 9 updating version 0 to 5 2024-09-11T03:38:37.207 INFO:tasks.rados.rados.2.smithi175.stdout:9: oid 9 updating version 5 to 6 2024-09-11T03:38:37.207 INFO:tasks.rados.rados.2.smithi175.stdout:9: oid 9 version 6 is already newer than 4 2024-09-11T03:38:37.207 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 6 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-11T03:38:37.207 INFO:tasks.rados.rados.2.smithi175.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-11T03:38:37.207 INFO:tasks.rados.rados.2.smithi175.stdout:10: finishing write tid 1 to smithi17557870-10 2024-09-11T03:38:37.208 INFO:tasks.rados.rados.2.smithi175.stdout:10: finishing write tid 2 to smithi17557870-10 2024-09-11T03:38:37.208 INFO:tasks.rados.rados.2.smithi175.stdout:10: finishing write tid 3 to smithi17557870-10 2024-09-11T03:38:37.208 INFO:tasks.rados.rados.2.smithi175.stdout:10: finishing write tid 4 to smithi17557870-10 2024-09-11T03:38:37.208 INFO:tasks.rados.rados.2.smithi175.stdout:10: oid 10 updating version 0 to 1 2024-09-11T03:38:37.208 INFO:tasks.rados.rados.2.smithi175.stdout:10: oid 10 updating version 1 to 2 2024-09-11T03:38:37.208 INFO:tasks.rados.rados.2.smithi175.stdout:10: oid 10 updating version 2 to 3 2024-09-11T03:38:37.208 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-11T03:38:37.208 INFO:tasks.rados.rados.2.smithi175.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-11T03:38:37.208 INFO:tasks.rados.rados.2.smithi175.stdout:12: finishing write tid 1 to smithi17557870-12 2024-09-11T03:38:37.209 INFO:tasks.rados.rados.2.smithi175.stdout:9: done (15 left) 2024-09-11T03:38:37.209 INFO:tasks.rados.rados.2.smithi175.stdout:10: done (14 left) 2024-09-11T03:38:37.209 INFO:tasks.rados.rados.2.smithi175.stdout:25: write initial oid 25 2024-09-11T03:38:37.209 INFO:tasks.rados.rados.2.smithi175.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-09-11T03:38:37.210 INFO:tasks.rados.rados.2.smithi175.stdout:25: writing smithi17557870-25 from 403368 to 958403 tid 1 2024-09-11T03:38:37.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:37.217 INFO:tasks.rados.rados.2.smithi175.stdout:25: writing smithi17557870-25 from 1454509 to 2182184 tid 2 2024-09-11T03:38:37.217 INFO:tasks.rados.rados.2.smithi175.stdout:26: write initial oid 26 2024-09-11T03:38:37.217 INFO:tasks.rados.rados.2.smithi175.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-09-11T03:38:37.223 INFO:tasks.rados.rados.2.smithi175.stdout:26: writing smithi17557870-26 from 420175 to 1050459 tid 1 2024-09-11T03:38:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2806890797' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:38:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.175:0/3465170716' entity='client.2' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"} : dispatch 2024-09-11T03:38:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:36 smithi175 ceph-mon[30894]: from='client.? ' entity='client.2' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"} : dispatch 2024-09-11T03:38:37.230 INFO:tasks.rados.rados.2.smithi175.stdout:26: writing smithi17557870-26 from 1796638 to 2267971 tid 2 2024-09-11T03:38:37.231 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.231 INFO:tasks.rados.rados.2.smithi175.stdout:12: finishing write tid 2 to smithi17557870-12 2024-09-11T03:38:37.231 INFO:tasks.rados.rados.2.smithi175.stdout:12: finishing write tid 3 to smithi17557870-12 2024-09-11T03:38:37.231 INFO:tasks.rados.rados.2.smithi175.stdout:12: finishing write tid 4 to smithi17557870-12 2024-09-11T03:38:37.231 INFO:tasks.rados.rados.2.smithi175.stdout:12: oid 12 updating version 0 to 4 2024-09-11T03:38:37.231 INFO:tasks.rados.rados.2.smithi175.stdout:12: oid 12 updating version 4 to 6 2024-09-11T03:38:37.231 INFO:tasks.rados.rados.2.smithi175.stdout:12: oid 12 version 6 is already newer than 5 2024-09-11T03:38:37.231 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-11T03:38:37.232 INFO:tasks.rados.rados.2.smithi175.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-11T03:38:37.232 INFO:tasks.rados.rados.2.smithi175.stdout:13: finishing write tid 1 to smithi17557870-13 2024-09-11T03:38:37.232 INFO:tasks.rados.rados.2.smithi175.stdout:13: finishing write tid 2 to smithi17557870-13 2024-09-11T03:38:37.232 INFO:tasks.rados.rados.2.smithi175.stdout:12: done (15 left) 2024-09-11T03:38:37.232 INFO:tasks.rados.rados.2.smithi175.stdout:27: write initial oid 27 2024-09-11T03:38:37.232 INFO:tasks.rados.rados.2.smithi175.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-09-11T03:38:37.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:37.233 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:37.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:37.236 INFO:tasks.rados.rados.2.smithi175.stdout:27: writing smithi17557870-27 from 436982 to 1142515 tid 1 2024-09-11T03:38:37.243 INFO:tasks.rados.rados.2.smithi175.stdout:27: writing smithi17557870-27 from 1855120 to 2436982 tid 2 2024-09-11T03:38:37.243 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.244 INFO:tasks.rados.rados.2.smithi175.stdout:13: finishing write tid 3 to smithi17557870-13 2024-09-11T03:38:37.244 INFO:tasks.rados.rados.2.smithi175.stdout:13: finishing write tid 4 to smithi17557870-13 2024-09-11T03:38:37.244 INFO:tasks.rados.rados.2.smithi175.stdout:13: oid 13 updating version 0 to 9 2024-09-11T03:38:37.244 INFO:tasks.rados.rados.2.smithi175.stdout:13: oid 13 version 9 is already newer than 8 2024-09-11T03:38:37.244 INFO:tasks.rados.rados.2.smithi175.stdout:13: oid 13 version 9 is already newer than 7 2024-09-11T03:38:37.244 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 9 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-11T03:38:37.244 INFO:tasks.rados.rados.2.smithi175.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-11T03:38:37.244 INFO:tasks.rados.rados.2.smithi175.stdout:11: finishing write tid 1 to smithi17557870-11 2024-09-11T03:38:37.244 INFO:tasks.rados.rados.2.smithi175.stdout:11: finishing write tid 2 to smithi17557870-11 2024-09-11T03:38:37.244 INFO:tasks.rados.rados.2.smithi175.stdout:11: finishing write tid 3 to smithi17557870-11 2024-09-11T03:38:37.244 INFO:tasks.rados.rados.2.smithi175.stdout:11: finishing write tid 4 to smithi17557870-11 2024-09-11T03:38:37.244 INFO:tasks.rados.rados.2.smithi175.stdout:11: oid 11 updating version 0 to 2 2024-09-11T03:38:37.245 INFO:tasks.rados.rados.2.smithi175.stdout:11: oid 11 updating version 2 to 3 2024-09-11T03:38:37.245 INFO:tasks.rados.rados.2.smithi175.stdout:11: oid 11 version 3 is already newer than 1 2024-09-11T03:38:37.245 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-11T03:38:37.245 INFO:tasks.rados.rados.2.smithi175.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-11T03:38:37.245 INFO:tasks.rados.rados.2.smithi175.stdout:14: finishing write tid 1 to smithi17557870-14 2024-09-11T03:38:37.245 INFO:tasks.rados.rados.2.smithi175.stdout:11: done (15 left) 2024-09-11T03:38:37.245 INFO:tasks.rados.rados.2.smithi175.stdout:13: done (14 left) 2024-09-11T03:38:37.245 INFO:tasks.rados.rados.2.smithi175.stdout:28: write initial oid 28 2024-09-11T03:38:37.245 INFO:tasks.rados.rados.2.smithi175.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-09-11T03:38:37.249 INFO:tasks.rados.rados.2.smithi175.stdout:28: writing smithi17557870-28 from 453789 to 1234571 tid 1 2024-09-11T03:38:37.255 INFO:tasks.rados.rados.2.smithi175.stdout:28: writing smithi17557870-28 from 1913602 to 2388604 tid 2 2024-09-11T03:38:37.255 INFO:tasks.rados.rados.2.smithi175.stdout:29: write initial oid 29 2024-09-11T03:38:37.255 INFO:tasks.rados.rados.2.smithi175.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-09-11T03:38:37.260 INFO:tasks.rados.rados.2.smithi175.stdout:29: writing smithi17557870-29 from 470596 to 926627 tid 1 2024-09-11T03:38:37.267 INFO:tasks.rados.rados.2.smithi175.stdout:29: writing smithi17557870-29 from 1572084 to 2190744 tid 2 2024-09-11T03:38:37.268 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.268 INFO:tasks.rados.rados.2.smithi175.stdout:14: finishing write tid 2 to smithi17557870-14 2024-09-11T03:38:37.268 INFO:tasks.rados.rados.2.smithi175.stdout:14: finishing write tid 3 to smithi17557870-14 2024-09-11T03:38:37.268 INFO:tasks.rados.rados.2.smithi175.stdout:14: finishing write tid 4 to smithi17557870-14 2024-09-11T03:38:37.268 INFO:tasks.rados.rados.2.smithi175.stdout:14: oid 14 updating version 0 to 6 2024-09-11T03:38:37.268 INFO:tasks.rados.rados.2.smithi175.stdout:14: oid 14 version 6 is already newer than 4 2024-09-11T03:38:37.268 INFO:tasks.rados.rados.2.smithi175.stdout:14: oid 14 version 6 is already newer than 5 2024-09-11T03:38:37.268 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-11T03:38:37.268 INFO:tasks.rados.rados.2.smithi175.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-11T03:38:37.268 INFO:tasks.rados.rados.2.smithi175.stdout:15: finishing write tid 1 to smithi17557870-15 2024-09-11T03:38:37.268 INFO:tasks.rados.rados.2.smithi175.stdout:15: finishing write tid 2 to smithi17557870-15 2024-09-11T03:38:37.269 INFO:tasks.rados.rados.2.smithi175.stdout:14: done (15 left) 2024-09-11T03:38:37.269 INFO:tasks.rados.rados.2.smithi175.stdout:30: write initial oid 30 2024-09-11T03:38:37.269 INFO:tasks.rados.rados.2.smithi175.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-09-11T03:38:37.273 INFO:tasks.rados.rados.2.smithi175.stdout:30: writing smithi17557870-30 from 487403 to 1018683 tid 1 2024-09-11T03:38:37.278 INFO:tasks.rados.rados.2.smithi175.stdout:30: writing smithi17557870-30 from 1514213 to 1992884 tid 2 2024-09-11T03:38:37.278 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.279 INFO:tasks.rados.rados.2.smithi175.stdout:15: finishing write tid 3 to smithi17557870-15 2024-09-11T03:38:37.279 INFO:tasks.rados.rados.2.smithi175.stdout:15: finishing write tid 4 to smithi17557870-15 2024-09-11T03:38:37.279 INFO:tasks.rados.rados.2.smithi175.stdout:15: oid 15 updating version 0 to 1 2024-09-11T03:38:37.279 INFO:tasks.rados.rados.2.smithi175.stdout:15: oid 15 updating version 1 to 2 2024-09-11T03:38:37.279 INFO:tasks.rados.rados.2.smithi175.stdout:15: oid 15 updating version 2 to 3 2024-09-11T03:38:37.279 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-11T03:38:37.279 INFO:tasks.rados.rados.2.smithi175.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-11T03:38:37.279 INFO:tasks.rados.rados.2.smithi175.stdout:16: finishing write tid 1 to smithi17557870-16 2024-09-11T03:38:37.279 INFO:tasks.rados.rados.2.smithi175.stdout:15: done (15 left) 2024-09-11T03:38:37.279 INFO:tasks.rados.rados.2.smithi175.stdout:31: write initial oid 31 2024-09-11T03:38:37.279 INFO:tasks.rados.rados.2.smithi175.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-09-11T03:38:37.284 INFO:tasks.rados.rados.2.smithi175.stdout:31: writing smithi17557870-31 from 504210 to 1110739 tid 1 2024-09-11T03:38:37.291 INFO:tasks.rados.rados.2.smithi175.stdout:31: writing smithi17557870-31 from 1572695 to 2195024 tid 2 2024-09-11T03:38:37.291 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.291 INFO:tasks.rados.rados.2.smithi175.stdout:16: finishing write tid 2 to smithi17557870-16 2024-09-11T03:38:37.291 INFO:tasks.rados.rados.2.smithi175.stdout:16: finishing write tid 3 to smithi17557870-16 2024-09-11T03:38:37.291 INFO:tasks.rados.rados.2.smithi175.stdout:16: finishing write tid 4 to smithi17557870-16 2024-09-11T03:38:37.291 INFO:tasks.rados.rados.2.smithi175.stdout:16: oid 16 updating version 0 to 5 2024-09-11T03:38:37.291 INFO:tasks.rados.rados.2.smithi175.stdout:16: oid 16 updating version 5 to 6 2024-09-11T03:38:37.291 INFO:tasks.rados.rados.2.smithi175.stdout:16: oid 16 version 6 is already newer than 4 2024-09-11T03:38:37.291 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-11T03:38:37.292 INFO:tasks.rados.rados.2.smithi175.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-11T03:38:37.292 INFO:tasks.rados.rados.2.smithi175.stdout:17: finishing write tid 1 to smithi17557870-17 2024-09-11T03:38:37.292 INFO:tasks.rados.rados.2.smithi175.stdout:17: finishing write tid 2 to smithi17557870-17 2024-09-11T03:38:37.292 INFO:tasks.rados.rados.2.smithi175.stdout:17: finishing write tid 3 to smithi17557870-17 2024-09-11T03:38:37.292 INFO:tasks.rados.rados.2.smithi175.stdout:17: finishing write tid 4 to smithi17557870-17 2024-09-11T03:38:37.292 INFO:tasks.rados.rados.2.smithi175.stdout:17: oid 17 updating version 0 to 5 2024-09-11T03:38:37.292 INFO:tasks.rados.rados.2.smithi175.stdout:17: oid 17 version 5 is already newer than 4 2024-09-11T03:38:37.292 INFO:tasks.rados.rados.2.smithi175.stdout:17: oid 17 updating version 5 to 6 2024-09-11T03:38:37.292 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-11T03:38:37.292 INFO:tasks.rados.rados.2.smithi175.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-11T03:38:37.292 INFO:tasks.rados.rados.2.smithi175.stdout:18: finishing write tid 1 to smithi17557870-18 2024-09-11T03:38:37.292 INFO:tasks.rados.rados.2.smithi175.stdout:16: done (15 left) 2024-09-11T03:38:37.292 INFO:tasks.rados.rados.2.smithi175.stdout:17: done (14 left) 2024-09-11T03:38:37.293 INFO:tasks.rados.rados.2.smithi175.stdout:32: write initial oid 32 2024-09-11T03:38:37.293 INFO:tasks.rados.rados.2.smithi175.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-09-11T03:38:37.297 INFO:tasks.rados.rados.2.smithi175.stdout:32: writing smithi17557870-32 from 521017 to 1319148 tid 1 2024-09-11T03:38:37.302 INFO:tasks.rados.rados.2.smithi175.stdout:32: writing smithi17557870-32 from 1747530 to 2229870 tid 2 2024-09-11T03:38:37.302 INFO:tasks.rados.rados.2.smithi175.stdout:33: write initial oid 33 2024-09-11T03:38:37.302 INFO:tasks.rados.rados.2.smithi175.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-09-11T03:38:37.308 INFO:tasks.rados.rados.2.smithi175.stdout:33: writing smithi17557870-33 from 537824 to 1011204 tid 1 2024-09-11T03:38:37.316 INFO:tasks.rados.rados.2.smithi175.stdout:33: writing smithi17557870-33 from 1806012 to 2432010 tid 2 2024-09-11T03:38:37.316 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.316 INFO:tasks.rados.rados.2.smithi175.stdout:18: finishing write tid 2 to smithi17557870-18 2024-09-11T03:38:37.316 INFO:tasks.rados.rados.2.smithi175.stdout:18: finishing write tid 3 to smithi17557870-18 2024-09-11T03:38:37.316 INFO:tasks.rados.rados.2.smithi175.stdout:18: finishing write tid 4 to smithi17557870-18 2024-09-11T03:38:37.316 INFO:tasks.rados.rados.2.smithi175.stdout:18: oid 18 updating version 0 to 3 2024-09-11T03:38:37.316 INFO:tasks.rados.rados.2.smithi175.stdout:18: oid 18 version 3 is already newer than 1 2024-09-11T03:38:37.316 INFO:tasks.rados.rados.2.smithi175.stdout:18: oid 18 version 3 is already newer than 2 2024-09-11T03:38:37.316 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-11T03:38:37.316 INFO:tasks.rados.rados.2.smithi175.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-11T03:38:37.316 INFO:tasks.rados.rados.2.smithi175.stdout:19: finishing write tid 1 to smithi17557870-19 2024-09-11T03:38:37.317 INFO:tasks.rados.rados.2.smithi175.stdout:19: finishing write tid 2 to smithi17557870-19 2024-09-11T03:38:37.317 INFO:tasks.rados.rados.2.smithi175.stdout:19: finishing write tid 3 to smithi17557870-19 2024-09-11T03:38:37.317 INFO:tasks.rados.rados.2.smithi175.stdout:18: done (15 left) 2024-09-11T03:38:37.317 INFO:tasks.rados.rados.2.smithi175.stdout:34: write initial oid 34 2024-09-11T03:38:37.317 INFO:tasks.rados.rados.2.smithi175.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-09-11T03:38:37.321 INFO:tasks.rados.rados.2.smithi175.stdout:34: writing smithi17557870-34 from 554631 to 1103260 tid 1 2024-09-11T03:38:37.328 INFO:tasks.rados.rados.2.smithi175.stdout:34: writing smithi17557870-34 from 1748141 to 2234150 tid 2 2024-09-11T03:38:37.328 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.328 INFO:tasks.rados.rados.2.smithi175.stdout:19: finishing write tid 4 to smithi17557870-19 2024-09-11T03:38:37.328 INFO:tasks.rados.rados.2.smithi175.stdout:19: oid 19 updating version 0 to 6 2024-09-11T03:38:37.328 INFO:tasks.rados.rados.2.smithi175.stdout:19: oid 19 version 6 is already newer than 4 2024-09-11T03:38:37.328 INFO:tasks.rados.rados.2.smithi175.stdout:19: oid 19 version 6 is already newer than 5 2024-09-11T03:38:37.328 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-11T03:38:37.328 INFO:tasks.rados.rados.2.smithi175.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-11T03:38:37.329 INFO:tasks.rados.rados.2.smithi175.stdout:20: finishing write tid 1 to smithi17557870-20 2024-09-11T03:38:37.329 INFO:tasks.rados.rados.2.smithi175.stdout:19: done (15 left) 2024-09-11T03:38:37.329 INFO:tasks.rados.rados.2.smithi175.stdout:35: write initial oid 35 2024-09-11T03:38:37.329 INFO:tasks.rados.rados.2.smithi175.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-09-11T03:38:37.334 INFO:tasks.rados.rados.2.smithi175.stdout:35: writing smithi17557870-35 from 571438 to 1195316 tid 1 2024-09-11T03:38:37.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:37.341 INFO:tasks.rados.rados.2.smithi175.stdout:35: writing smithi17557870-35 from 1806623 to 2436290 tid 2 2024-09-11T03:38:37.341 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.342 INFO:tasks.rados.rados.2.smithi175.stdout:20: finishing write tid 2 to smithi17557870-20 2024-09-11T03:38:37.342 INFO:tasks.rados.rados.2.smithi175.stdout:20: finishing write tid 3 to smithi17557870-20 2024-09-11T03:38:37.342 INFO:tasks.rados.rados.2.smithi175.stdout:20: finishing write tid 4 to smithi17557870-20 2024-09-11T03:38:37.342 INFO:tasks.rados.rados.2.smithi175.stdout:20: oid 20 updating version 0 to 8 2024-09-11T03:38:37.342 INFO:tasks.rados.rados.2.smithi175.stdout:20: oid 20 updating version 8 to 9 2024-09-11T03:38:37.342 INFO:tasks.rados.rados.2.smithi175.stdout:20: oid 20 version 9 is already newer than 7 2024-09-11T03:38:37.342 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 9 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-11T03:38:37.342 INFO:tasks.rados.rados.2.smithi175.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-11T03:38:37.342 INFO:tasks.rados.rados.2.smithi175.stdout:21: finishing write tid 1 to smithi17557870-21 2024-09-11T03:38:37.342 INFO:tasks.rados.rados.2.smithi175.stdout:21: finishing write tid 2 to smithi17557870-21 2024-09-11T03:38:37.342 INFO:tasks.rados.rados.2.smithi175.stdout:21: finishing write tid 3 to smithi17557870-21 2024-09-11T03:38:37.342 INFO:tasks.rados.rados.2.smithi175.stdout:20: done (15 left) 2024-09-11T03:38:37.342 INFO:tasks.rados.rados.2.smithi175.stdout:36: write initial oid 36 2024-09-11T03:38:37.343 INFO:tasks.rados.rados.2.smithi175.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-09-11T03:38:37.348 INFO:tasks.rados.rados.2.smithi175.stdout:36: writing smithi17557870-36 from 588245 to 1287372 tid 1 2024-09-11T03:38:37.354 INFO:tasks.rados.rados.2.smithi175.stdout:36: writing smithi17557870-36 from 1865105 to 2354783 tid 2 2024-09-11T03:38:37.354 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.354 INFO:tasks.rados.rados.2.smithi175.stdout:21: finishing write tid 4 to smithi17557870-21 2024-09-11T03:38:37.355 INFO:tasks.rados.rados.2.smithi175.stdout:21: oid 21 updating version 0 to 7 2024-09-11T03:38:37.355 INFO:tasks.rados.rados.2.smithi175.stdout:21: oid 21 updating version 7 to 9 2024-09-11T03:38:37.355 INFO:tasks.rados.rados.2.smithi175.stdout:21: oid 21 version 9 is already newer than 8 2024-09-11T03:38:37.355 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 9 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-11T03:38:37.355 INFO:tasks.rados.rados.2.smithi175.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-11T03:38:37.355 INFO:tasks.rados.rados.2.smithi175.stdout:22: finishing write tid 1 to smithi17557870-22 2024-09-11T03:38:37.355 INFO:tasks.rados.rados.2.smithi175.stdout:22: finishing write tid 2 to smithi17557870-22 2024-09-11T03:38:37.355 INFO:tasks.rados.rados.2.smithi175.stdout:22: finishing write tid 3 to smithi17557870-22 2024-09-11T03:38:37.355 INFO:tasks.rados.rados.2.smithi175.stdout:22: finishing write tid 4 to smithi17557870-22 2024-09-11T03:38:37.355 INFO:tasks.rados.rados.2.smithi175.stdout:22: oid 22 updating version 0 to 9 2024-09-11T03:38:37.355 INFO:tasks.rados.rados.2.smithi175.stdout:22: oid 22 version 9 is already newer than 8 2024-09-11T03:38:37.355 INFO:tasks.rados.rados.2.smithi175.stdout:22: oid 22 version 9 is already newer than 7 2024-09-11T03:38:37.355 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 9 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-11T03:38:37.356 INFO:tasks.rados.rados.2.smithi175.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-11T03:38:37.356 INFO:tasks.rados.rados.2.smithi175.stdout:23: finishing write tid 1 to smithi17557870-23 2024-09-11T03:38:37.356 INFO:tasks.rados.rados.2.smithi175.stdout:23: finishing write tid 2 to smithi17557870-23 2024-09-11T03:38:37.356 INFO:tasks.rados.rados.2.smithi175.stdout:23: finishing write tid 3 to smithi17557870-23 2024-09-11T03:38:37.356 INFO:tasks.rados.rados.2.smithi175.stdout:21: done (15 left) 2024-09-11T03:38:37.356 INFO:tasks.rados.rados.2.smithi175.stdout:22: done (14 left) 2024-09-11T03:38:37.356 INFO:tasks.rados.rados.2.smithi175.stdout:37: write initial oid 37 2024-09-11T03:38:37.356 INFO:tasks.rados.rados.2.smithi175.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-09-11T03:38:37.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:37.357 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:37.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:37.361 INFO:tasks.rados.rados.2.smithi175.stdout:37: writing smithi17557870-37 from 605052 to 1379428 tid 1 2024-09-11T03:38:37.367 INFO:tasks.rados.rados.2.smithi175.stdout:37: writing smithi17557870-37 from 1923587 to 2556923 tid 2 2024-09-11T03:38:37.367 INFO:tasks.rados.rados.2.smithi175.stdout:38: write initial oid 38 2024-09-11T03:38:37.367 INFO:tasks.rados.rados.2.smithi175.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-09-11T03:38:37.373 INFO:tasks.rados.rados.2.smithi175.stdout:38: writing smithi17557870-38 from 621859 to 1071484 tid 1 2024-09-11T03:38:37.382 INFO:tasks.rados.rados.2.smithi175.stdout:38: writing smithi17557870-38 from 1865716 to 2621859 tid 2 2024-09-11T03:38:37.383 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.383 INFO:tasks.rados.rados.2.smithi175.stdout:23: finishing write tid 4 to smithi17557870-23 2024-09-11T03:38:37.383 INFO:tasks.rados.rados.2.smithi175.stdout:23: oid 23 updating version 0 to 5 2024-09-11T03:38:37.383 INFO:tasks.rados.rados.2.smithi175.stdout:23: oid 23 version 5 is already newer than 4 2024-09-11T03:38:37.383 INFO:tasks.rados.rados.2.smithi175.stdout:23: oid 23 updating version 5 to 6 2024-09-11T03:38:37.384 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-11T03:38:37.384 INFO:tasks.rados.rados.2.smithi175.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-11T03:38:37.384 INFO:tasks.rados.rados.2.smithi175.stdout:24: finishing write tid 1 to smithi17557870-24 2024-09-11T03:38:37.384 INFO:tasks.rados.rados.2.smithi175.stdout:24: finishing write tid 2 to smithi17557870-24 2024-09-11T03:38:37.384 INFO:tasks.rados.rados.2.smithi175.stdout:23: done (15 left) 2024-09-11T03:38:37.384 INFO:tasks.rados.rados.2.smithi175.stdout:39: write initial oid 39 2024-09-11T03:38:37.385 INFO:tasks.rados.rados.2.smithi175.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-09-11T03:38:37.388 INFO:tasks.rados.rados.2.smithi175.stdout:39: writing smithi17557870-39 from 638666 to 1163540 tid 1 2024-09-11T03:38:37.398 INFO:tasks.rados.rados.2.smithi175.stdout:39: writing smithi17557870-39 from 1924198 to 2561203 tid 2 2024-09-11T03:38:37.398 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.398 INFO:tasks.rados.rados.2.smithi175.stdout:24: finishing write tid 3 to smithi17557870-24 2024-09-11T03:38:37.398 INFO:tasks.rados.rados.2.smithi175.stdout:24: finishing write tid 4 to smithi17557870-24 2024-09-11T03:38:37.398 INFO:tasks.rados.rados.2.smithi175.stdout:24: oid 24 updating version 0 to 9 2024-09-11T03:38:37.399 INFO:tasks.rados.rados.2.smithi175.stdout:24: oid 24 version 9 is already newer than 7 2024-09-11T03:38:37.399 INFO:tasks.rados.rados.2.smithi175.stdout:24: oid 24 version 9 is already newer than 8 2024-09-11T03:38:37.399 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-11T03:38:37.399 INFO:tasks.rados.rados.2.smithi175.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-11T03:38:37.399 INFO:tasks.rados.rados.2.smithi175.stdout:25: finishing write tid 1 to smithi17557870-25 2024-09-11T03:38:37.399 INFO:tasks.rados.rados.2.smithi175.stdout:25: finishing write tid 2 to smithi17557870-25 2024-09-11T03:38:37.400 INFO:tasks.rados.rados.2.smithi175.stdout:25: finishing write tid 3 to smithi17557870-25 2024-09-11T03:38:37.400 INFO:tasks.rados.rados.2.smithi175.stdout:25: finishing write tid 4 to smithi17557870-25 2024-09-11T03:38:37.400 INFO:tasks.rados.rados.2.smithi175.stdout:25: oid 25 updating version 0 to 1 2024-09-11T03:38:37.400 INFO:tasks.rados.rados.2.smithi175.stdout:25: oid 25 updating version 1 to 2 2024-09-11T03:38:37.400 INFO:tasks.rados.rados.2.smithi175.stdout:25: oid 25 updating version 2 to 3 2024-09-11T03:38:37.400 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 3 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-11T03:38:37.400 INFO:tasks.rados.rados.2.smithi175.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-11T03:38:37.401 INFO:tasks.rados.rados.2.smithi175.stdout:26: finishing write tid 1 to smithi17557870-26 2024-09-11T03:38:37.401 INFO:tasks.rados.rados.2.smithi175.stdout:26: finishing write tid 2 to smithi17557870-26 2024-09-11T03:38:37.401 INFO:tasks.rados.rados.2.smithi175.stdout:26: finishing write tid 3 to smithi17557870-26 2024-09-11T03:38:37.401 INFO:tasks.rados.rados.2.smithi175.stdout:26: finishing write tid 4 to smithi17557870-26 2024-09-11T03:38:37.401 INFO:tasks.rados.rados.2.smithi175.stdout:26: oid 26 updating version 0 to 6 2024-09-11T03:38:37.401 INFO:tasks.rados.rados.2.smithi175.stdout:26: oid 26 version 6 is already newer than 4 2024-09-11T03:38:37.402 INFO:tasks.rados.rados.2.smithi175.stdout:26: oid 26 version 6 is already newer than 5 2024-09-11T03:38:37.402 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-11T03:38:37.402 INFO:tasks.rados.rados.2.smithi175.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-11T03:38:37.402 INFO:tasks.rados.rados.2.smithi175.stdout:27: finishing write tid 1 to smithi17557870-27 2024-09-11T03:38:37.402 INFO:tasks.rados.rados.2.smithi175.stdout:27: finishing write tid 2 to smithi17557870-27 2024-09-11T03:38:37.402 INFO:tasks.rados.rados.2.smithi175.stdout:24: done (15 left) 2024-09-11T03:38:37.402 INFO:tasks.rados.rados.2.smithi175.stdout:25: done (14 left) 2024-09-11T03:38:37.402 INFO:tasks.rados.rados.2.smithi175.stdout:26: done (13 left) 2024-09-11T03:38:37.402 INFO:tasks.rados.rados.2.smithi175.stdout:40: write initial oid 40 2024-09-11T03:38:37.402 INFO:tasks.rados.rados.2.smithi175.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-09-11T03:38:37.403 INFO:tasks.rados.rados.2.smithi175.stdout:40: writing smithi17557870-40 from 655473 to 1371949 tid 1 2024-09-11T03:38:37.410 INFO:tasks.rados.rados.2.smithi175.stdout:40: writing smithi17557870-40 from 2099033 to 2655473 tid 2 2024-09-11T03:38:37.410 INFO:tasks.rados.rados.2.smithi175.stdout:41: write initial oid 41 2024-09-11T03:38:37.410 INFO:tasks.rados.rados.2.smithi175.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-09-11T03:38:37.417 INFO:tasks.rados.rados.2.smithi175.stdout:41: writing smithi17557870-41 from 672280 to 1464005 tid 1 2024-09-11T03:38:37.424 INFO:tasks.rados.rados.2.smithi175.stdout:41: writing smithi17557870-41 from 2157515 to 2672280 tid 2 2024-09-11T03:38:37.424 INFO:tasks.rados.rados.2.smithi175.stdout:42: write initial oid 42 2024-09-11T03:38:37.424 INFO:tasks.rados.rados.2.smithi175.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-09-11T03:38:37.430 INFO:tasks.rados.rados.2.smithi175.stdout:42: writing smithi17557870-42 from 689087 to 1156061 tid 1 2024-09-11T03:38:37.438 INFO:tasks.rados.rados.2.smithi175.stdout:42: writing smithi17557870-42 from 1699644 to 2483976 tid 2 2024-09-11T03:38:37.438 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.438 INFO:tasks.rados.rados.2.smithi175.stdout:27: finishing write tid 3 to smithi17557870-27 2024-09-11T03:38:37.438 INFO:tasks.rados.rados.2.smithi175.stdout:27: finishing write tid 4 to smithi17557870-27 2024-09-11T03:38:37.439 INFO:tasks.rados.rados.2.smithi175.stdout:27: oid 27 updating version 0 to 10 2024-09-11T03:38:37.439 INFO:tasks.rados.rados.2.smithi175.stdout:27: oid 27 updating version 10 to 12 2024-09-11T03:38:37.439 INFO:tasks.rados.rados.2.smithi175.stdout:27: oid 27 version 12 is already newer than 11 2024-09-11T03:38:37.439 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-11T03:38:37.439 INFO:tasks.rados.rados.2.smithi175.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-11T03:38:37.439 INFO:tasks.rados.rados.2.smithi175.stdout:28: finishing write tid 1 to smithi17557870-28 2024-09-11T03:38:37.439 INFO:tasks.rados.rados.2.smithi175.stdout:28: finishing write tid 2 to smithi17557870-28 2024-09-11T03:38:37.439 INFO:tasks.rados.rados.2.smithi175.stdout:27: done (15 left) 2024-09-11T03:38:37.440 INFO:tasks.rados.rados.2.smithi175.stdout:43: write initial oid 43 2024-09-11T03:38:37.440 INFO:tasks.rados.rados.2.smithi175.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-09-11T03:38:37.444 INFO:tasks.rados.rados.2.smithi175.stdout:43: writing smithi17557870-43 from 705894 to 1248117 tid 1 2024-09-11T03:38:37.451 INFO:tasks.rados.rados.2.smithi175.stdout:43: writing smithi17557870-43 from 1758126 to 2402469 tid 2 2024-09-11T03:38:37.451 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.451 INFO:tasks.rados.rados.2.smithi175.stdout:28: finishing write tid 3 to smithi17557870-28 2024-09-11T03:38:37.451 INFO:tasks.rados.rados.2.smithi175.stdout:28: finishing write tid 4 to smithi17557870-28 2024-09-11T03:38:37.451 INFO:tasks.rados.rados.2.smithi175.stdout:28: oid 28 updating version 0 to 11 2024-09-11T03:38:37.451 INFO:tasks.rados.rados.2.smithi175.stdout:28: oid 28 updating version 11 to 12 2024-09-11T03:38:37.451 INFO:tasks.rados.rados.2.smithi175.stdout:28: oid 28 version 12 is already newer than 10 2024-09-11T03:38:37.451 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 12 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-11T03:38:37.451 INFO:tasks.rados.rados.2.smithi175.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-11T03:38:37.451 INFO:tasks.rados.rados.2.smithi175.stdout:29: finishing write tid 1 to smithi17557870-29 2024-09-11T03:38:37.452 INFO:tasks.rados.rados.2.smithi175.stdout:29: finishing write tid 2 to smithi17557870-29 2024-09-11T03:38:37.452 INFO:tasks.rados.rados.2.smithi175.stdout:28: done (15 left) 2024-09-11T03:38:37.452 INFO:tasks.rados.rados.2.smithi175.stdout:44: write initial oid 44 2024-09-11T03:38:37.452 INFO:tasks.rados.rados.2.smithi175.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-09-11T03:38:37.457 INFO:tasks.rados.rados.2.smithi175.stdout:44: writing smithi17557870-44 from 722701 to 1340173 tid 1 2024-09-11T03:38:37.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:37.465 INFO:tasks.rados.rados.2.smithi175.stdout:44: writing smithi17557870-44 from 1816608 to 2604609 tid 2 2024-09-11T03:38:37.465 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.465 INFO:tasks.rados.rados.2.smithi175.stdout:29: finishing write tid 3 to smithi17557870-29 2024-09-11T03:38:37.465 INFO:tasks.rados.rados.2.smithi175.stdout:29: finishing write tid 4 to smithi17557870-29 2024-09-11T03:38:37.465 INFO:tasks.rados.rados.2.smithi175.stdout:29: oid 29 updating version 0 to 11 2024-09-11T03:38:37.465 INFO:tasks.rados.rados.2.smithi175.stdout:29: oid 29 version 11 is already newer than 10 2024-09-11T03:38:37.465 INFO:tasks.rados.rados.2.smithi175.stdout:29: oid 29 updating version 11 to 12 2024-09-11T03:38:37.465 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-11T03:38:37.465 INFO:tasks.rados.rados.2.smithi175.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-11T03:38:37.465 INFO:tasks.rados.rados.2.smithi175.stdout:30: finishing write tid 1 to smithi17557870-30 2024-09-11T03:38:37.466 INFO:tasks.rados.rados.2.smithi175.stdout:29: done (15 left) 2024-09-11T03:38:37.466 INFO:tasks.rados.rados.2.smithi175.stdout:45: write initial oid 45 2024-09-11T03:38:37.466 INFO:tasks.rados.rados.2.smithi175.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-09-11T03:38:37.472 INFO:tasks.rados.rados.2.smithi175.stdout:45: writing smithi17557870-45 from 739508 to 1432229 tid 1 2024-09-11T03:38:37.478 INFO:tasks.rados.rados.2.smithi175.stdout:45: writing smithi17557870-45 from 1875090 to 2523102 tid 2 2024-09-11T03:38:37.478 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.479 INFO:tasks.rados.rados.2.smithi175.stdout:30: finishing write tid 2 to smithi17557870-30 2024-09-11T03:38:37.479 INFO:tasks.rados.rados.2.smithi175.stdout:30: finishing write tid 3 to smithi17557870-30 2024-09-11T03:38:37.479 INFO:tasks.rados.rados.2.smithi175.stdout:30: finishing write tid 4 to smithi17557870-30 2024-09-11T03:38:37.479 INFO:tasks.rados.rados.2.smithi175.stdout:30: oid 30 updating version 0 to 10 2024-09-11T03:38:37.479 INFO:tasks.rados.rados.2.smithi175.stdout:30: oid 30 updating version 10 to 12 2024-09-11T03:38:37.479 INFO:tasks.rados.rados.2.smithi175.stdout:30: oid 30 version 12 is already newer than 11 2024-09-11T03:38:37.479 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 12 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-11T03:38:37.479 INFO:tasks.rados.rados.2.smithi175.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-11T03:38:37.479 INFO:tasks.rados.rados.2.smithi175.stdout:31: finishing write tid 1 to smithi17557870-31 2024-09-11T03:38:37.479 INFO:tasks.rados.rados.2.smithi175.stdout:30: done (15 left) 2024-09-11T03:38:37.480 INFO:tasks.rados.rados.2.smithi175.stdout:46: write initial oid 46 2024-09-11T03:38:37.480 INFO:tasks.rados.rados.2.smithi175.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-09-11T03:38:37.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:37.481 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:37.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:37.486 INFO:tasks.rados.rados.2.smithi175.stdout:46: writing smithi17557870-46 from 756315 to 1524285 tid 1 2024-09-11T03:38:37.493 INFO:tasks.rados.rados.2.smithi175.stdout:46: writing smithi17557870-46 from 1933572 to 2725242 tid 2 2024-09-11T03:38:37.493 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.493 INFO:tasks.rados.rados.2.smithi175.stdout:31: finishing write tid 2 to smithi17557870-31 2024-09-11T03:38:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:31: finishing write tid 3 to smithi17557870-31 2024-09-11T03:38:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:31: finishing write tid 4 to smithi17557870-31 2024-09-11T03:38:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:31: oid 31 updating version 0 to 14 2024-09-11T03:38:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:31: oid 31 version 14 is already newer than 13 2024-09-11T03:38:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:31: oid 31 updating version 14 to 15 2024-09-11T03:38:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 15 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-11T03:38:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-11T03:38:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:32: finishing write tid 1 to smithi17557870-32 2024-09-11T03:38:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:32: finishing write tid 2 to smithi17557870-32 2024-09-11T03:38:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:31: done (15 left) 2024-09-11T03:38:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:47: write initial oid 47 2024-09-11T03:38:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-09-11T03:38:37.500 INFO:tasks.rados.rados.2.smithi175.stdout:47: writing smithi17557870-47 from 773122 to 1332694 tid 1 2024-09-11T03:38:37.508 INFO:tasks.rados.rados.2.smithi175.stdout:47: writing smithi17557870-47 from 1992054 to 2643735 tid 2 2024-09-11T03:38:37.508 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.509 INFO:tasks.rados.rados.2.smithi175.stdout:32: finishing write tid 3 to smithi17557870-32 2024-09-11T03:38:37.509 INFO:tasks.rados.rados.2.smithi175.stdout:32: finishing write tid 4 to smithi17557870-32 2024-09-11T03:38:37.509 INFO:tasks.rados.rados.2.smithi175.stdout:32: oid 32 updating version 0 to 1 2024-09-11T03:38:37.509 INFO:tasks.rados.rados.2.smithi175.stdout:32: oid 32 updating version 1 to 2 2024-09-11T03:38:37.509 INFO:tasks.rados.rados.2.smithi175.stdout:32: oid 32 updating version 2 to 3 2024-09-11T03:38:37.509 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 3 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-11T03:38:37.509 INFO:tasks.rados.rados.2.smithi175.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-11T03:38:37.509 INFO:tasks.rados.rados.2.smithi175.stdout:33: finishing write tid 1 to smithi17557870-33 2024-09-11T03:38:37.510 INFO:tasks.rados.rados.2.smithi175.stdout:33: finishing write tid 2 to smithi17557870-33 2024-09-11T03:38:37.510 INFO:tasks.rados.rados.2.smithi175.stdout:33: finishing write tid 3 to smithi17557870-33 2024-09-11T03:38:37.510 INFO:tasks.rados.rados.2.smithi175.stdout:33: finishing write tid 4 to smithi17557870-33 2024-09-11T03:38:37.510 INFO:tasks.rados.rados.2.smithi175.stdout:33: oid 33 updating version 0 to 14 2024-09-11T03:38:37.510 INFO:tasks.rados.rados.2.smithi175.stdout:33: oid 33 version 14 is already newer than 13 2024-09-11T03:38:37.510 INFO:tasks.rados.rados.2.smithi175.stdout:33: oid 33 updating version 14 to 15 2024-09-11T03:38:37.510 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 15 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-11T03:38:37.510 INFO:tasks.rados.rados.2.smithi175.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-11T03:38:37.511 INFO:tasks.rados.rados.2.smithi175.stdout:34: finishing write tid 1 to smithi17557870-34 2024-09-11T03:38:37.511 INFO:tasks.rados.rados.2.smithi175.stdout:32: done (15 left) 2024-09-11T03:38:37.511 INFO:tasks.rados.rados.2.smithi175.stdout:33: done (14 left) 2024-09-11T03:38:37.511 INFO:tasks.rados.rados.2.smithi175.stdout:48: write initial oid 48 2024-09-11T03:38:37.511 INFO:tasks.rados.rados.2.smithi175.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-09-11T03:38:37.515 INFO:tasks.rados.rados.2.smithi175.stdout:48: writing smithi17557870-48 from 789929 to 1424750 tid 1 2024-09-11T03:38:37.522 INFO:tasks.rados.rados.2.smithi175.stdout:48: writing smithi17557870-48 from 2050536 to 2789929 tid 2 2024-09-11T03:38:37.522 INFO:tasks.rados.rados.2.smithi175.stdout:49: write initial oid 49 2024-09-11T03:38:37.522 INFO:tasks.rados.rados.2.smithi175.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-09-11T03:38:37.528 INFO:tasks.rados.rados.2.smithi175.stdout:49: writing smithi17557870-49 from 406736 to 1116806 tid 1 2024-09-11T03:38:37.536 INFO:tasks.rados.rados.2.smithi175.stdout:49: writing smithi17557870-49 from 1709018 to 2364368 tid 2 2024-09-11T03:38:37.536 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.536 INFO:tasks.rados.rados.2.smithi175.stdout:34: finishing write tid 2 to smithi17557870-34 2024-09-11T03:38:37.536 INFO:tasks.rados.rados.2.smithi175.stdout:34: finishing write tid 3 to smithi17557870-34 2024-09-11T03:38:37.536 INFO:tasks.rados.rados.2.smithi175.stdout:34: finishing write tid 4 to smithi17557870-34 2024-09-11T03:38:37.536 INFO:tasks.rados.rados.2.smithi175.stdout:34: oid 34 updating version 0 to 8 2024-09-11T03:38:37.536 INFO:tasks.rados.rados.2.smithi175.stdout:34: oid 34 version 8 is already newer than 7 2024-09-11T03:38:37.537 INFO:tasks.rados.rados.2.smithi175.stdout:34: oid 34 updating version 8 to 9 2024-09-11T03:38:37.537 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 9 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-11T03:38:37.537 INFO:tasks.rados.rados.2.smithi175.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-11T03:38:37.537 INFO:tasks.rados.rados.2.smithi175.stdout:35: finishing write tid 1 to smithi17557870-35 2024-09-11T03:38:37.537 INFO:tasks.rados.rados.2.smithi175.stdout:35: finishing write tid 2 to smithi17557870-35 2024-09-11T03:38:37.537 INFO:tasks.rados.rados.2.smithi175.stdout:34: done (15 left) 2024-09-11T03:38:37.537 INFO:tasks.rados.rados.2.smithi175.stdout:50: write initial oid 50 2024-09-11T03:38:37.537 INFO:tasks.rados.rados.2.smithi175.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-09-11T03:38:37.543 INFO:tasks.rados.rados.2.smithi175.stdout:50: writing smithi17557870-50 from 423543 to 1208862 tid 1 2024-09-11T03:38:37.551 INFO:tasks.rados.rados.2.smithi175.stdout:50: writing smithi17557870-50 from 1767500 to 2566508 tid 2 2024-09-11T03:38:37.551 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 16 2024-09-11T03:38:37.551 INFO:tasks.rados.rados.2.smithi175.stdout:35: finishing write tid 3 to smithi17557870-35 2024-09-11T03:38:37.551 INFO:tasks.rados.rados.2.smithi175.stdout:35: finishing write tid 4 to smithi17557870-35 2024-09-11T03:38:37.551 INFO:tasks.rados.rados.2.smithi175.stdout:35: oid 35 updating version 0 to 3 2024-09-11T03:38:37.551 INFO:tasks.rados.rados.2.smithi175.stdout:35: oid 35 version 3 is already newer than 2 2024-09-11T03:38:37.552 INFO:tasks.rados.rados.2.smithi175.stdout:35: oid 35 version 3 is already newer than 1 2024-09-11T03:38:37.552 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 3 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-11T03:38:37.552 INFO:tasks.rados.rados.2.smithi175.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-11T03:38:37.552 INFO:tasks.rados.rados.2.smithi175.stdout:36: finishing write tid 1 to smithi17557870-36 2024-09-11T03:38:37.552 INFO:tasks.rados.rados.2.smithi175.stdout:36: finishing write tid 2 to smithi17557870-36 2024-09-11T03:38:37.552 INFO:tasks.rados.rados.2.smithi175.stdout:36: finishing write tid 3 to smithi17557870-36 2024-09-11T03:38:37.552 INFO:tasks.rados.rados.2.smithi175.stdout:36: finishing write tid 4 to smithi17557870-36 2024-09-11T03:38:37.552 INFO:tasks.rados.rados.2.smithi175.stdout:36: oid 36 updating version 0 to 8 2024-09-11T03:38:37.552 INFO:tasks.rados.rados.2.smithi175.stdout:36: oid 36 updating version 8 to 9 2024-09-11T03:38:37.553 INFO:tasks.rados.rados.2.smithi175.stdout:36: oid 36 version 9 is already newer than 7 2024-09-11T03:38:37.553 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-11T03:38:37.553 INFO:tasks.rados.rados.2.smithi175.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-11T03:38:37.553 INFO:tasks.rados.rados.2.smithi175.stdout:37: finishing write tid 1 to smithi17557870-37 2024-09-11T03:38:37.553 INFO:tasks.rados.rados.2.smithi175.stdout:37: finishing write tid 2 to smithi17557870-37 2024-09-11T03:38:37.553 INFO:tasks.rados.rados.2.smithi175.stdout:37: finishing write tid 3 to smithi17557870-37 2024-09-11T03:38:37.553 INFO:tasks.rados.rados.2.smithi175.stdout:35: done (15 left) 2024-09-11T03:38:37.553 INFO:tasks.rados.rados.2.smithi175.stdout:36: done (14 left) 2024-09-11T03:38:37.553 INFO:tasks.rados.rados.2.smithi175.stdout:51: copy_from oid 5 from oid 24 current snap is 0 2024-09-11T03:38:37.554 INFO:tasks.rados.rados.2.smithi175.stdout:37: finishing write tid 4 to smithi17557870-37 2024-09-11T03:38:37.554 INFO:tasks.rados.rados.2.smithi175.stdout:37: oid 37 updating version 0 to 1 2024-09-11T03:38:37.554 INFO:tasks.rados.rados.2.smithi175.stdout:37: oid 37 updating version 1 to 2 2024-09-11T03:38:37.554 INFO:tasks.rados.rados.2.smithi175.stdout:37: oid 37 updating version 2 to 3 2024-09-11T03:38:37.554 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 3 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-11T03:38:37.554 INFO:tasks.rados.rados.2.smithi175.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-11T03:38:37.554 INFO:tasks.rados.rados.2.smithi175.stdout:38: finishing write tid 1 to smithi17557870-38 2024-09-11T03:38:37.554 INFO:tasks.rados.rados.2.smithi175.stdout:38: finishing write tid 2 to smithi17557870-38 2024-09-11T03:38:37.554 INFO:tasks.rados.rados.2.smithi175.stdout:38: finishing write tid 3 to smithi17557870-38 2024-09-11T03:38:37.554 INFO:tasks.rados.rados.2.smithi175.stdout:38: finishing write tid 4 to smithi17557870-38 2024-09-11T03:38:37.554 INFO:tasks.rados.rados.2.smithi175.stdout:38: oid 38 updating version 0 to 14 2024-09-11T03:38:37.554 INFO:tasks.rados.rados.2.smithi175.stdout:38: oid 38 updating version 14 to 15 2024-09-11T03:38:37.554 INFO:tasks.rados.rados.2.smithi175.stdout:38: oid 38 version 15 is already newer than 13 2024-09-11T03:38:37.554 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 15 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-11T03:38:37.554 INFO:tasks.rados.rados.2.smithi175.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-11T03:38:37.555 INFO:tasks.rados.rados.2.smithi175.stdout:39: finishing write tid 1 to smithi17557870-39 2024-09-11T03:38:37.555 INFO:tasks.rados.rados.2.smithi175.stdout:39: finishing write tid 2 to smithi17557870-39 2024-09-11T03:38:37.555 INFO:tasks.rados.rados.2.smithi175.stdout:39: finishing write tid 3 to smithi17557870-39 2024-09-11T03:38:37.555 INFO:tasks.rados.rados.2.smithi175.stdout:39: finishing write tid 4 to smithi17557870-39 2024-09-11T03:38:37.555 INFO:tasks.rados.rados.2.smithi175.stdout:39: oid 39 updating version 0 to 6 2024-09-11T03:38:37.555 INFO:tasks.rados.rados.2.smithi175.stdout:39: oid 39 version 6 is already newer than 5 2024-09-11T03:38:37.555 INFO:tasks.rados.rados.2.smithi175.stdout:39: oid 39 version 6 is already newer than 4 2024-09-11T03:38:37.555 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-11T03:38:37.555 INFO:tasks.rados.rados.2.smithi175.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-11T03:38:37.555 INFO:tasks.rados.rados.2.smithi175.stdout:40: finishing write tid 1 to smithi17557870-40 2024-09-11T03:38:37.555 INFO:tasks.rados.rados.2.smithi175.stdout:37: done (14 left) 2024-09-11T03:38:37.555 INFO:tasks.rados.rados.2.smithi175.stdout:38: done (13 left) 2024-09-11T03:38:37.555 INFO:tasks.rados.rados.2.smithi175.stdout:39: done (12 left) 2024-09-11T03:38:37.555 INFO:tasks.rados.rados.2.smithi175.stdout:52: read oid 20 snap -1 2024-09-11T03:38:37.556 INFO:tasks.rados.rados.2.smithi175.stdout:52: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-11T03:38:37.556 INFO:tasks.rados.rados.2.smithi175.stdout:53: write oid 36 current snap is 0 2024-09-11T03:38:37.556 INFO:tasks.rados.rados.2.smithi175.stdout:40: finishing write tid 2 to smithi17557870-40 2024-09-11T03:38:37.556 INFO:tasks.rados.rados.2.smithi175.stdout:40: finishing write tid 3 to smithi17557870-40 2024-09-11T03:38:37.556 INFO:tasks.rados.rados.2.smithi175.stdout:53: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-09-11T03:38:37.556 INFO:tasks.rados.rados.2.smithi175.stdout:53: writing smithi17557870-36 from 440350 to 900918 tid 1 2024-09-11T03:38:37.560 INFO:tasks.rados.rados.2.smithi175.stdout:53: writing smithi17557870-36 from 1309629 to 1852295 tid 2 2024-09-11T03:38:37.567 INFO:tasks.rados.rados.2.smithi175.stdout:53: writing smithi17557870-36 from 2323973 to 2840350 tid 3 2024-09-11T03:38:37.567 INFO:tasks.rados.rados.2.smithi175.stdout:54: snap_create 2024-09-11T03:38:37.567 INFO:tasks.rados.rados.2.smithi175.stdout:40: finishing write tid 4 to smithi17557870-40 2024-09-11T03:38:37.567 INFO:tasks.rados.rados.2.smithi175.stdout:40: oid 40 updating version 0 to 5 2024-09-11T03:38:37.567 INFO:tasks.rados.rados.2.smithi175.stdout:40: oid 40 version 5 is already newer than 4 2024-09-11T03:38:37.568 INFO:tasks.rados.rados.2.smithi175.stdout:40: oid 40 updating version 5 to 6 2024-09-11T03:38:37.568 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-11T03:38:37.568 INFO:tasks.rados.rados.2.smithi175.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-11T03:38:37.568 INFO:tasks.rados.rados.2.smithi175.stdout:41: finishing write tid 1 to smithi17557870-41 2024-09-11T03:38:37.568 INFO:tasks.rados.rados.2.smithi175.stdout:41: finishing write tid 2 to smithi17557870-41 2024-09-11T03:38:37.568 INFO:tasks.rados.rados.2.smithi175.stdout:41: finishing write tid 3 to smithi17557870-41 2024-09-11T03:38:37.568 INFO:tasks.rados.rados.2.smithi175.stdout:41: finishing write tid 4 to smithi17557870-41 2024-09-11T03:38:37.568 INFO:tasks.rados.rados.2.smithi175.stdout:41: oid 41 updating version 0 to 9 2024-09-11T03:38:37.568 INFO:tasks.rados.rados.2.smithi175.stdout:41: oid 41 version 9 is already newer than 8 2024-09-11T03:38:37.568 INFO:tasks.rados.rados.2.smithi175.stdout:41: oid 41 version 9 is already newer than 7 2024-09-11T03:38:37.568 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 9 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-11T03:38:37.568 INFO:tasks.rados.rados.2.smithi175.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-11T03:38:37.568 INFO:tasks.rados.rados.2.smithi175.stdout:42: finishing write tid 1 to smithi17557870-42 2024-09-11T03:38:37.569 INFO:tasks.rados.rados.2.smithi175.stdout:42: finishing write tid 2 to smithi17557870-42 2024-09-11T03:38:37.569 INFO:tasks.rados.rados.2.smithi175.stdout:42: finishing write tid 3 to smithi17557870-42 2024-09-11T03:38:37.569 INFO:tasks.rados.rados.2.smithi175.stdout:42: finishing write tid 4 to smithi17557870-42 2024-09-11T03:38:37.569 INFO:tasks.rados.rados.2.smithi175.stdout:42: oid 42 updating version 0 to 5 2024-09-11T03:38:37.569 INFO:tasks.rados.rados.2.smithi175.stdout:42: oid 42 version 5 is already newer than 4 2024-09-11T03:38:37.569 INFO:tasks.rados.rados.2.smithi175.stdout:42: oid 42 updating version 5 to 6 2024-09-11T03:38:37.569 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-11T03:38:37.570 INFO:tasks.rados.rados.2.smithi175.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-11T03:38:37.570 INFO:tasks.rados.rados.2.smithi175.stdout:43: finishing write tid 1 to smithi17557870-43 2024-09-11T03:38:37.570 INFO:tasks.rados.rados.2.smithi175.stdout:43: finishing write tid 2 to smithi17557870-43 2024-09-11T03:38:37.570 INFO:tasks.rados.rados.2.smithi175.stdout:43: finishing write tid 3 to smithi17557870-43 2024-09-11T03:38:37.570 INFO:tasks.rados.rados.2.smithi175.stdout:43: finishing write tid 4 to smithi17557870-43 2024-09-11T03:38:37.570 INFO:tasks.rados.rados.2.smithi175.stdout:43: oid 43 updating version 0 to 8 2024-09-11T03:38:37.570 INFO:tasks.rados.rados.2.smithi175.stdout:43: oid 43 version 8 is already newer than 7 2024-09-11T03:38:37.570 INFO:tasks.rados.rados.2.smithi175.stdout:43: oid 43 updating version 8 to 9 2024-09-11T03:38:37.570 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 9 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-11T03:38:37.570 INFO:tasks.rados.rados.2.smithi175.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-11T03:38:37.570 INFO:tasks.rados.rados.2.smithi175.stdout:44: finishing write tid 1 to smithi17557870-44 2024-09-11T03:38:37.570 INFO:tasks.rados.rados.2.smithi175.stdout:44: finishing write tid 2 to smithi17557870-44 2024-09-11T03:38:37.570 INFO:tasks.rados.rados.2.smithi175.stdout:44: finishing write tid 3 to smithi17557870-44 2024-09-11T03:38:37.571 INFO:tasks.rados.rados.2.smithi175.stdout:44: finishing write tid 4 to smithi17557870-44 2024-09-11T03:38:37.571 INFO:tasks.rados.rados.2.smithi175.stdout:44: oid 44 updating version 0 to 5 2024-09-11T03:38:37.571 INFO:tasks.rados.rados.2.smithi175.stdout:44: oid 44 version 5 is already newer than 4 2024-09-11T03:38:37.571 INFO:tasks.rados.rados.2.smithi175.stdout:44: oid 44 updating version 5 to 6 2024-09-11T03:38:37.571 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-11T03:38:37.571 INFO:tasks.rados.rados.2.smithi175.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-11T03:38:37.571 INFO:tasks.rados.rados.2.smithi175.stdout:45: finishing write tid 1 to smithi17557870-45 2024-09-11T03:38:37.571 INFO:tasks.rados.rados.2.smithi175.stdout:45: finishing write tid 2 to smithi17557870-45 2024-09-11T03:38:37.571 INFO:tasks.rados.rados.2.smithi175.stdout:45: finishing write tid 3 to smithi17557870-45 2024-09-11T03:38:37.571 INFO:tasks.rados.rados.2.smithi175.stdout:45: finishing write tid 4 to smithi17557870-45 2024-09-11T03:38:37.571 INFO:tasks.rados.rados.2.smithi175.stdout:45: oid 45 updating version 0 to 8 2024-09-11T03:38:37.571 INFO:tasks.rados.rados.2.smithi175.stdout:45: oid 45 version 8 is already newer than 7 2024-09-11T03:38:37.571 INFO:tasks.rados.rados.2.smithi175.stdout:45: oid 45 updating version 8 to 9 2024-09-11T03:38:37.572 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-11T03:38:37.572 INFO:tasks.rados.rados.2.smithi175.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-11T03:38:37.572 INFO:tasks.rados.rados.2.smithi175.stdout:46: finishing write tid 1 to smithi17557870-46 2024-09-11T03:38:37.572 INFO:tasks.rados.rados.2.smithi175.stdout:46: finishing write tid 2 to smithi17557870-46 2024-09-11T03:38:37.572 INFO:tasks.rados.rados.2.smithi175.stdout:46: finishing write tid 3 to smithi17557870-46 2024-09-11T03:38:37.572 INFO:tasks.rados.rados.2.smithi175.stdout:46: finishing write tid 4 to smithi17557870-46 2024-09-11T03:38:37.572 INFO:tasks.rados.rados.2.smithi175.stdout:46: oid 46 updating version 0 to 3 2024-09-11T03:38:37.572 INFO:tasks.rados.rados.2.smithi175.stdout:46: oid 46 version 3 is already newer than 1 2024-09-11T03:38:37.572 INFO:tasks.rados.rados.2.smithi175.stdout:46: oid 46 version 3 is already newer than 2 2024-09-11T03:38:37.572 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 3 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-11T03:38:37.572 INFO:tasks.rados.rados.2.smithi175.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-11T03:38:37.572 INFO:tasks.rados.rados.2.smithi175.stdout:47: finishing write tid 1 to smithi17557870-47 2024-09-11T03:38:37.572 INFO:tasks.rados.rados.2.smithi175.stdout:47: finishing write tid 2 to smithi17557870-47 2024-09-11T03:38:37.573 INFO:tasks.rados.rados.2.smithi175.stdout:47: finishing write tid 3 to smithi17557870-47 2024-09-11T03:38:37.573 INFO:tasks.rados.rados.2.smithi175.stdout:47: finishing write tid 4 to smithi17557870-47 2024-09-11T03:38:37.573 INFO:tasks.rados.rados.2.smithi175.stdout:47: oid 47 updating version 0 to 11 2024-09-11T03:38:37.573 INFO:tasks.rados.rados.2.smithi175.stdout:47: oid 47 version 11 is already newer than 10 2024-09-11T03:38:37.573 INFO:tasks.rados.rados.2.smithi175.stdout:47: oid 47 updating version 11 to 12 2024-09-11T03:38:37.573 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-11T03:38:37.573 INFO:tasks.rados.rados.2.smithi175.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-11T03:38:37.573 INFO:tasks.rados.rados.2.smithi175.stdout:48: finishing write tid 1 to smithi17557870-48 2024-09-11T03:38:37.573 INFO:tasks.rados.rados.2.smithi175.stdout:48: finishing write tid 2 to smithi17557870-48 2024-09-11T03:38:37.573 INFO:tasks.rados.rados.2.smithi175.stdout:48: finishing write tid 3 to smithi17557870-48 2024-09-11T03:38:37.573 INFO:tasks.rados.rados.2.smithi175.stdout:48: finishing write tid 4 to smithi17557870-48 2024-09-11T03:38:37.573 INFO:tasks.rados.rados.2.smithi175.stdout:48: oid 48 updating version 0 to 8 2024-09-11T03:38:37.573 INFO:tasks.rados.rados.2.smithi175.stdout:48: oid 48 version 8 is already newer than 7 2024-09-11T03:38:37.574 INFO:tasks.rados.rados.2.smithi175.stdout:48: oid 48 updating version 8 to 9 2024-09-11T03:38:37.574 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 9 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-11T03:38:37.574 INFO:tasks.rados.rados.2.smithi175.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-11T03:38:37.574 INFO:tasks.rados.rados.2.smithi175.stdout:49: finishing write tid 1 to smithi17557870-49 2024-09-11T03:38:37.574 INFO:tasks.rados.rados.2.smithi175.stdout:49: finishing write tid 2 to smithi17557870-49 2024-09-11T03:38:37.574 INFO:tasks.rados.rados.2.smithi175.stdout:49: finishing write tid 3 to smithi17557870-49 2024-09-11T03:38:37.574 INFO:tasks.rados.rados.2.smithi175.stdout:49: finishing write tid 4 to smithi17557870-49 2024-09-11T03:38:37.574 INFO:tasks.rados.rados.2.smithi175.stdout:49: oid 49 updating version 0 to 10 2024-09-11T03:38:37.574 INFO:tasks.rados.rados.2.smithi175.stdout:49: oid 49 updating version 10 to 12 2024-09-11T03:38:37.574 INFO:tasks.rados.rados.2.smithi175.stdout:49: oid 49 version 12 is already newer than 11 2024-09-11T03:38:37.574 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-11T03:38:37.574 INFO:tasks.rados.rados.2.smithi175.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-11T03:38:37.574 INFO:tasks.rados.rados.2.smithi175.stdout:50: finishing write tid 1 to smithi17557870-50 2024-09-11T03:38:37.574 INFO:tasks.rados.rados.2.smithi175.stdout:50: finishing write tid 2 to smithi17557870-50 2024-09-11T03:38:37.574 INFO:tasks.rados.rados.2.smithi175.stdout:50: finishing write tid 3 to smithi17557870-50 2024-09-11T03:38:37.575 INFO:tasks.rados.rados.2.smithi175.stdout:50: finishing write tid 4 to smithi17557870-50 2024-09-11T03:38:37.575 INFO:tasks.rados.rados.2.smithi175.stdout:50: oid 50 updating version 0 to 16 2024-09-11T03:38:37.575 INFO:tasks.rados.rados.2.smithi175.stdout:50: oid 50 updating version 16 to 17 2024-09-11T03:38:37.575 INFO:tasks.rados.rados.2.smithi175.stdout:50: oid 50 updating version 17 to 18 2024-09-11T03:38:37.575 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 18 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-11T03:38:37.575 INFO:tasks.rados.rados.2.smithi175.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-11T03:38:37.575 INFO:tasks.rados.rados.2.smithi175.stdout:52: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-11T03:38:37.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:37.604 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:37.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:37.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:37.643 INFO:tasks.rados.rados.2.smithi175.stdout:51: finishing copy_from racing read to smithi17557870-5 2024-09-11T03:38:37.643 INFO:tasks.rados.rados.2.smithi175.stdout:51: finishing copy_from to smithi17557870-5 2024-09-11T03:38:37.644 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-11T03:38:37.644 INFO:tasks.rados.rados.2.smithi175.stdout:53: finishing write tid 1 to smithi17557870-36 2024-09-11T03:38:37.644 INFO:tasks.rados.rados.2.smithi175.stdout:53: finishing write tid 2 to smithi17557870-36 2024-09-11T03:38:37.644 INFO:tasks.rados.rados.2.smithi175.stdout:53: finishing write tid 3 to smithi17557870-36 2024-09-11T03:38:37.644 INFO:tasks.rados.rados.2.smithi175.stdout:53: finishing write tid 4 to smithi17557870-36 2024-09-11T03:38:37.644 INFO:tasks.rados.rados.2.smithi175.stdout:53: finishing write tid 5 to smithi17557870-36 2024-09-11T03:38:37.645 INFO:tasks.rados.rados.2.smithi175.stdout:53: oid 36 updating version 0 to 12 2024-09-11T03:38:37.645 INFO:tasks.rados.rados.2.smithi175.stdout:53: oid 36 version 12 is already newer than 10 2024-09-11T03:38:37.645 INFO:tasks.rados.rados.2.smithi175.stdout:53: oid 36 updating version 12 to 13 2024-09-11T03:38:37.645 INFO:tasks.rados.rados.2.smithi175.stdout:53: oid 36 version 13 is already newer than 11 2024-09-11T03:38:37.645 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 13 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-11T03:38:37.645 INFO:tasks.rados.rados.2.smithi175.stdout:53: left oid 36 (ObjNum 50 snap 0 seq_num 50) 2024-09-11T03:38:37.706 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:37.727 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:37.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:37.728 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:37.759 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-11T03:38:37.829 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:37.850 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:37.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:37.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:37.922 INFO:tasks.rados.rados.2.smithi175.stdout:40: done (14 left) 2024-09-11T03:38:37.922 INFO:tasks.rados.rados.2.smithi175.stdout:41: done (13 left) 2024-09-11T03:38:37.922 INFO:tasks.rados.rados.2.smithi175.stdout:42: done (12 left) 2024-09-11T03:38:37.922 INFO:tasks.rados.rados.2.smithi175.stdout:43: done (11 left) 2024-09-11T03:38:37.922 INFO:tasks.rados.rados.2.smithi175.stdout:44: done (10 left) 2024-09-11T03:38:37.922 INFO:tasks.rados.rados.2.smithi175.stdout:45: done (9 left) 2024-09-11T03:38:37.922 INFO:tasks.rados.rados.2.smithi175.stdout:46: done (8 left) 2024-09-11T03:38:37.922 INFO:tasks.rados.rados.2.smithi175.stdout:47: done (7 left) 2024-09-11T03:38:37.922 INFO:tasks.rados.rados.2.smithi175.stdout:48: done (6 left) 2024-09-11T03:38:37.922 INFO:tasks.rados.rados.2.smithi175.stdout:49: done (5 left) 2024-09-11T03:38:37.922 INFO:tasks.rados.rados.2.smithi175.stdout:50: done (4 left) 2024-09-11T03:38:37.923 INFO:tasks.rados.rados.2.smithi175.stdout:51: done (3 left) 2024-09-11T03:38:37.923 INFO:tasks.rados.rados.2.smithi175.stdout:52: done (2 left) 2024-09-11T03:38:37.923 INFO:tasks.rados.rados.2.smithi175.stdout:53: done (1 left) 2024-09-11T03:38:37.923 INFO:tasks.rados.rados.2.smithi175.stdout:54: done (0 left) 2024-09-11T03:38:37.923 INFO:tasks.rados.rados.2.smithi175.stdout:55: snap_remove snap 0 2024-09-11T03:38:37.953 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:37.974 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:37.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:37.976 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:37.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:37 smithi084 ceph-mon[27254]: pgmap v304: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:37.985 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:37 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:38:37.985 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:37 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:38:37.985 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2806890797' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:38:37.985 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:37 smithi084 ceph-mon[27254]: from='client.? ' entity='client.2' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-11T03:38:37.985 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:37 smithi084 ceph-mon[27254]: osdmap e100: 12 total, 12 up, 12 in 2024-09-11T03:38:38.072 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:38:38.072 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:38:38.072 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:38:38.072 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:38:38.072 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.072 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:20 1.0 e20/19 1:30306672:devicehealth::main.db.0000000000000000:head v 20'10, mlcod=20'8)", 2024-09-11T03:38:38.072 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.137485+0000", 2024-09-11T03:38:38.072 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.92757359500001, 2024-09-11T03:38:38.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000669062, 2024-09-11T03:38:38.073 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.073 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.073 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.073 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.073 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.073 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.073 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.137485+0000", 2024-09-11T03:38:38.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.073 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.073 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.074 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.137485+0000", 2024-09-11T03:38:38.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.074 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.137486+0000", 2024-09-11T03:38:38.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5300000000000003e-07 2024-09-11T03:38:38.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.074 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.137588+0000", 2024-09-11T03:38:38.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010207 2024-09-11T03:38:38.075 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.075 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.075 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.075 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.137599+0000", 2024-09-11T03:38:38.075 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0896e-05 2024-09-11T03:38:38.075 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.075 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.075 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.075 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.137606+0000", 2024-09-11T03:38:38.075 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5060000000000001e-06 2024-09-11T03:38:38.075 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.076 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.076 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.076 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.137665+0000", 2024-09-11T03:38:38.076 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9564000000000001e-05 2024-09-11T03:38:38.076 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.076 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.076 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.076 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.137677+0000", 2024-09-11T03:38:38.076 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2191999999999999e-05 2024-09-11T03:38:38.076 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.076 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.076 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.076 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138104+0000", 2024-09-11T03:38:38.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00042694299999999999 2024-09-11T03:38:38.077 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.077 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.077 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.077 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138154+0000", 2024-09-11T03:38:38.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0138e-05 2024-09-11T03:38:38.077 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.077 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.077 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.077 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.077 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.077 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:22 1.0 e20/19 1:30306672:devicehealth::main.db.0000000000000000:head v 20'12, mlcod=20'8)", 2024-09-11T03:38:38.078 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.138363+0000", 2024-09-11T03:38:38.078 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.926696064, 2024-09-11T03:38:38.078 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066382500000000005, 2024-09-11T03:38:38.078 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.078 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.078 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.078 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.078 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.078 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.078 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138363+0000", 2024-09-11T03:38:38.078 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.078 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.078 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138363+0000", 2024-09-11T03:38:38.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138363+0000", 2024-09-11T03:38:38.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.37e-07 2024-09-11T03:38:38.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138382+0000", 2024-09-11T03:38:38.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9031000000000001e-05 2024-09-11T03:38:38.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.080 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.080 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138393+0000", 2024-09-11T03:38:38.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.097e-05 2024-09-11T03:38:38.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.080 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.080 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138399+0000", 2024-09-11T03:38:38.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9869999999999996e-06 2024-09-11T03:38:38.081 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.081 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.081 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.081 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138443+0000", 2024-09-11T03:38:38.081 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3272999999999997e-05 2024-09-11T03:38:38.081 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.081 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.081 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.081 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138461+0000", 2024-09-11T03:38:38.081 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8340999999999999e-05 2024-09-11T03:38:38.081 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.081 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.081 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.082 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138981+0000", 2024-09-11T03:38:38.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052013600000000004 2024-09-11T03:38:38.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.082 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.082 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.082 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.139027+0000", 2024-09-11T03:38:38.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.545e-05 2024-09-11T03:38:38.082 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.082 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.082 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.082 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.082 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:43 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'25, mlcod=20'22)", 2024-09-11T03:38:38.082 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.202514+0000", 2024-09-11T03:38:38.083 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.86254499500001, 2024-09-11T03:38:38.083 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030994870000000002, 2024-09-11T03:38:38.083 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.083 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.083 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.083 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202514+0000", 2024-09-11T03:38:38.083 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.083 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202514+0000", 2024-09-11T03:38:38.084 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.084 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.084 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.084 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202514+0000", 2024-09-11T03:38:38.084 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7899999999999998e-07 2024-09-11T03:38:38.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.084 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.084 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.084 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202519+0000", 2024-09-11T03:38:38.084 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1949999999999996e-06 2024-09-11T03:38:38.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.085 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.085 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.085 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202531+0000", 2024-09-11T03:38:38.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1975e-05 2024-09-11T03:38:38.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.085 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.085 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.085 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202537+0000", 2024-09-11T03:38:38.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9649999999999996e-06 2024-09-11T03:38:38.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.085 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.086 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.086 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202581+0000", 2024-09-11T03:38:38.086 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4373999999999999e-05 2024-09-11T03:38:38.086 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.086 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.086 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.086 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202596+0000", 2024-09-11T03:38:38.086 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5367999999999999e-05 2024-09-11T03:38:38.086 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.086 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.086 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.086 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.205590+0000", 2024-09-11T03:38:38.087 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029938579999999998 2024-09-11T03:38:38.087 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.087 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.087 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.087 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.205613+0000", 2024-09-11T03:38:38.087 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3173e-05 2024-09-11T03:38:38.087 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.087 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.087 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.087 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.087 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.087 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:44 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'26, mlcod=20'22)", 2024-09-11T03:38:38.087 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.204969+0000", 2024-09-11T03:38:38.088 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.86008940800002, 2024-09-11T03:38:38.088 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010337199999999999, 2024-09-11T03:38:38.088 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.088 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.088 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.088 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.088 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.088 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.088 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.204969+0000", 2024-09-11T03:38:38.088 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.088 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.088 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.088 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.204969+0000", 2024-09-11T03:38:38.089 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.089 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.089 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.089 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.204970+0000", 2024-09-11T03:38:38.089 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4799999999999998e-07 2024-09-11T03:38:38.089 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.089 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.089 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.205016+0000", 2024-09-11T03:38:38.089 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5572999999999999e-05 2024-09-11T03:38:38.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.090 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.090 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.205030+0000", 2024-09-11T03:38:38.090 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3923e-05 2024-09-11T03:38:38.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.090 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.090 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.205037+0000", 2024-09-11T03:38:38.090 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6550000000000004e-06 2024-09-11T03:38:38.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.205429+0000", 2024-09-11T03:38:38.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00039158299999999998 2024-09-11T03:38:38.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.205447+0000", 2024-09-11T03:38:38.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8074999999999999e-05 2024-09-11T03:38:38.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.205954+0000", 2024-09-11T03:38:38.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00050735799999999996 2024-09-11T03:38:38.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.092 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.092 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.092 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206003+0000", 2024-09-11T03:38:38.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9005000000000003e-05 2024-09-11T03:38:38.092 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.092 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.092 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.092 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.092 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:46 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'28, mlcod=20'23)", 2024-09-11T03:38:38.092 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.206111+0000", 2024-09-11T03:38:38.093 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.85894775100002, 2024-09-11T03:38:38.093 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026967599999999999, 2024-09-11T03:38:38.093 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.093 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.093 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.093 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.093 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.093 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.093 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206111+0000", 2024-09-11T03:38:38.093 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.093 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.094 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.094 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.094 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206111+0000", 2024-09-11T03:38:38.094 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.094 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.094 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.094 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.094 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206112+0000", 2024-09-11T03:38:38.094 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0500000000000003e-07 2024-09-11T03:38:38.094 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.094 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.094 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.094 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206125+0000", 2024-09-11T03:38:38.095 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2908000000000001e-05 2024-09-11T03:38:38.095 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.095 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.095 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206137+0000", 2024-09-11T03:38:38.095 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.187e-05 2024-09-11T03:38:38.095 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.095 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.095 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206144+0000", 2024-09-11T03:38:38.095 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7919999999999993e-06 2024-09-11T03:38:38.095 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.096 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.096 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206185+0000", 2024-09-11T03:38:38.096 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0803999999999998e-05 2024-09-11T03:38:38.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.096 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.096 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206197+0000", 2024-09-11T03:38:38.096 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1962e-05 2024-09-11T03:38:38.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.096 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.208764+0000", 2024-09-11T03:38:38.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002566652 2024-09-11T03:38:38.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.208808+0000", 2024-09-11T03:38:38.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4166999999999999e-05 2024-09-11T03:38:38.097 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.097 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.097 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.098 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:48 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'30, mlcod=20'27)", 2024-09-11T03:38:38.098 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.209747+0000", 2024-09-11T03:38:38.098 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.85531180499999, 2024-09-11T03:38:38.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030822760000000001, 2024-09-11T03:38:38.098 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.098 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.098 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.098 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.098 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209747+0000", 2024-09-11T03:38:38.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.099 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.099 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209747+0000", 2024-09-11T03:38:38.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.099 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.099 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.099 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209748+0000", 2024-09-11T03:38:38.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4399999999999999e-07 2024-09-11T03:38:38.099 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.099 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.099 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209753+0000", 2024-09-11T03:38:38.100 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6130000000000003e-06 2024-09-11T03:38:38.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.100 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.100 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209772+0000", 2024-09-11T03:38:38.100 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8493000000000001e-05 2024-09-11T03:38:38.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.100 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.100 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209786+0000", 2024-09-11T03:38:38.100 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4399000000000001e-05 2024-09-11T03:38:38.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.101 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209832+0000", 2024-09-11T03:38:38.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5556999999999998e-05 2024-09-11T03:38:38.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.101 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209852+0000", 2024-09-11T03:38:38.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0262999999999999e-05 2024-09-11T03:38:38.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.101 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212794+0000", 2024-09-11T03:38:38.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029415259999999999 2024-09-11T03:38:38.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.102 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.102 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212829+0000", 2024-09-11T03:38:38.102 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5680999999999999e-05 2024-09-11T03:38:38.102 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.102 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.102 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.103 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:49 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'31, mlcod=20'27)", 2024-09-11T03:38:38.103 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.210165+0000", 2024-09-11T03:38:38.103 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.85489394799998, 2024-09-11T03:38:38.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028269739999999999, 2024-09-11T03:38:38.103 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.103 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.103 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.103 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.104 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.104 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210165+0000", 2024-09-11T03:38:38.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.104 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.104 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210165+0000", 2024-09-11T03:38:38.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210166+0000", 2024-09-11T03:38:38.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5300000000000004e-07 2024-09-11T03:38:38.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210171+0000", 2024-09-11T03:38:38.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1039999999999998e-06 2024-09-11T03:38:38.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.106 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.106 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210183+0000", 2024-09-11T03:38:38.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1848000000000001e-05 2024-09-11T03:38:38.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.106 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.106 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210189+0000", 2024-09-11T03:38:38.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4459999999999998e-06 2024-09-11T03:38:38.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.107 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.107 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210267+0000", 2024-09-11T03:38:38.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7540999999999995e-05 2024-09-11T03:38:38.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.107 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.107 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210288+0000", 2024-09-11T03:38:38.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0992000000000002e-05 2024-09-11T03:38:38.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.108 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.108 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.108 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212958+0000", 2024-09-11T03:38:38.108 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002670455 2024-09-11T03:38:38.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.108 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.108 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.108 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212992+0000", 2024-09-11T03:38:38.108 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3935e-05 2024-09-11T03:38:38.108 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.108 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.109 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.109 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:50 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'32, mlcod=20'28)", 2024-09-11T03:38:38.109 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.213250+0000", 2024-09-11T03:38:38.109 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.85180847499998, 2024-09-11T03:38:38.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00059172900000000004, 2024-09-11T03:38:38.109 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.109 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.109 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.109 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213250+0000", 2024-09-11T03:38:38.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213250+0000", 2024-09-11T03:38:38.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213251+0000", 2024-09-11T03:38:38.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2699999999999999e-07 2024-09-11T03:38:38.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213293+0000", 2024-09-11T03:38:38.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1560000000000002e-05 2024-09-11T03:38:38.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213302+0000", 2024-09-11T03:38:38.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5119999999999997e-06 2024-09-11T03:38:38.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213308+0000", 2024-09-11T03:38:38.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.207e-06 2024-09-11T03:38:38.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213362+0000", 2024-09-11T03:38:38.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3455000000000002e-05 2024-09-11T03:38:38.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213383+0000", 2024-09-11T03:38:38.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1348e-05 2024-09-11T03:38:38.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.113 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.113 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213798+0000", 2024-09-11T03:38:38.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00041505900000000001 2024-09-11T03:38:38.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213842+0000", 2024-09-11T03:38:38.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4060999999999997e-05 2024-09-11T03:38:38.114 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.114 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.114 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.114 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:53 1.0 e20/19 1:30306672:devicehealth::main.db.0000000000000000:head v 20'35, mlcod=20'33)", 2024-09-11T03:38:38.114 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.215534+0000", 2024-09-11T03:38:38.114 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.849524906, 2024-09-11T03:38:38.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071941300000000002, 2024-09-11T03:38:38.114 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.115 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.115 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.115 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.115 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.215534+0000", 2024-09-11T03:38:38.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.215534+0000", 2024-09-11T03:38:38.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.215535+0000", 2024-09-11T03:38:38.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0200000000000002e-07 2024-09-11T03:38:38.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.215572+0000", 2024-09-11T03:38:38.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7800999999999999e-05 2024-09-11T03:38:38.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.117 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.117 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.215582+0000", 2024-09-11T03:38:38.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8009999999999998e-06 2024-09-11T03:38:38.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.117 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.117 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.215587+0000", 2024-09-11T03:38:38.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8890000000000001e-06 2024-09-11T03:38:38.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.117 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.117 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.215657+0000", 2024-09-11T03:38:38.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9574999999999997e-05 2024-09-11T03:38:38.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.118 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.118 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.215676+0000", 2024-09-11T03:38:38.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9215000000000002e-05 2024-09-11T03:38:38.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.118 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.118 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.216212+0000", 2024-09-11T03:38:38.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00053568699999999997 2024-09-11T03:38:38.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.118 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.119 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.216253+0000", 2024-09-11T03:38:38.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1842999999999999e-05 2024-09-11T03:38:38.119 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.119 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.119 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.119 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:65 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'43, mlcod=20'40)", 2024-09-11T03:38:38.119 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.392629+0000", 2024-09-11T03:38:38.119 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.67242941299997, 2024-09-11T03:38:38.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000602393, 2024-09-11T03:38:38.119 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.120 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.120 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.120 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.392629+0000", 2024-09-11T03:38:38.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.392629+0000", 2024-09-11T03:38:38.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.392630+0000", 2024-09-11T03:38:38.121 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:38.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1699999999999999e-07 2024-09-11T03:38:38.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.392634+0000", 2024-09-11T03:38:38.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.755e-06 2024-09-11T03:38:38.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.392640+0000", 2024-09-11T03:38:38.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7739999999999999e-06 2024-09-11T03:38:38.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.392645+0000", 2024-09-11T03:38:38.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0470000000000001e-06 2024-09-11T03:38:38.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.124 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.124 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.392892+0000", 2024-09-11T03:38:38.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024706700000000001 2024-09-11T03:38:38.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.124 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.124 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.392910+0000", 2024-09-11T03:38:38.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7779e-05 2024-09-11T03:38:38.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.393206+0000", 2024-09-11T03:38:38.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029538699999999998 2024-09-11T03:38:38.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.393232+0000", 2024-09-11T03:38:38.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6267000000000001e-05 2024-09-11T03:38:38.125 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.125 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.125 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.126 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:66 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'44, mlcod=20'40)", 2024-09-11T03:38:38.126 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.393237+0000", 2024-09-11T03:38:38.126 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.67182154800003, 2024-09-11T03:38:38.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00060143700000000002, 2024-09-11T03:38:38.126 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.126 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.126 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.126 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.393237+0000", 2024-09-11T03:38:38.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.393237+0000", 2024-09-11T03:38:38.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.393238+0000", 2024-09-11T03:38:38.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3600000000000004e-07 2024-09-11T03:38:38.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.393258+0000", 2024-09-11T03:38:38.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0304000000000001e-05 2024-09-11T03:38:38.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.393266+0000", 2024-09-11T03:38:38.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.875e-06 2024-09-11T03:38:38.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.393271+0000", 2024-09-11T03:38:38.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5449999999999997e-06 2024-09-11T03:38:38.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.393309+0000", 2024-09-11T03:38:38.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7925e-05 2024-09-11T03:38:38.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.393325+0000", 2024-09-11T03:38:38.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6090999999999999e-05 2024-09-11T03:38:38.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.130 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.130 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.130 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.393789+0000", 2024-09-11T03:38:38.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000464695 2024-09-11T03:38:38.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.130 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.130 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.130 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.393839+0000", 2024-09-11T03:38:38.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9465999999999997e-05 2024-09-11T03:38:38.130 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.130 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.131 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.131 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:72 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'50, mlcod=20'46)", 2024-09-11T03:38:38.131 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.397352+0000", 2024-09-11T03:38:38.131 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.66770640499999, 2024-09-11T03:38:38.131 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00058999599999999997, 2024-09-11T03:38:38.131 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.131 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.131 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.131 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.397352+0000", 2024-09-11T03:38:38.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.397352+0000", 2024-09-11T03:38:38.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.397353+0000", 2024-09-11T03:38:38.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6100000000000001e-07 2024-09-11T03:38:38.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.397374+0000", 2024-09-11T03:38:38.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1345e-05 2024-09-11T03:38:38.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.397384+0000", 2024-09-11T03:38:38.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0105000000000001e-05 2024-09-11T03:38:38.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.397390+0000", 2024-09-11T03:38:38.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4559999999999999e-06 2024-09-11T03:38:38.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.397444+0000", 2024-09-11T03:38:38.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4088000000000001e-05 2024-09-11T03:38:38.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.397464+0000", 2024-09-11T03:38:38.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9673999999999998e-05 2024-09-11T03:38:38.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.397914+0000", 2024-09-11T03:38:38.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00045066000000000003 2024-09-11T03:38:38.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.397942+0000", 2024-09-11T03:38:38.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8107e-05 2024-09-11T03:38:38.135 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.136 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.136 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.136 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:75 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'53, mlcod=20'50)", 2024-09-11T03:38:38.136 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.399359+0000", 2024-09-11T03:38:38.136 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.66569955699998, 2024-09-11T03:38:38.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000583904, 2024-09-11T03:38:38.136 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.136 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.136 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.136 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.399359+0000", 2024-09-11T03:38:38.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.399359+0000", 2024-09-11T03:38:38.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.399360+0000", 2024-09-11T03:38:38.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6000000000000004e-07 2024-09-11T03:38:38.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.399378+0000", 2024-09-11T03:38:38.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8304999999999998e-05 2024-09-11T03:38:38.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.399389+0000", 2024-09-11T03:38:38.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0389e-05 2024-09-11T03:38:38.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.399395+0000", 2024-09-11T03:38:38.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1909999999999997e-06 2024-09-11T03:38:38.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.399449+0000", 2024-09-11T03:38:38.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3727999999999998e-05 2024-09-11T03:38:38.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.399466+0000", 2024-09-11T03:38:38.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7774999999999998e-05 2024-09-11T03:38:38.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.140 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.399915+0000", 2024-09-11T03:38:38.140 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00044891100000000003 2024-09-11T03:38:38.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.140 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.399943+0000", 2024-09-11T03:38:38.140 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8045e-05 2024-09-11T03:38:38.140 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.140 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.140 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.141 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:77 1.0 e20/19 1:30306672:devicehealth::main.db.0000000000000000:head v 20'55, mlcod=20'53)", 2024-09-11T03:38:38.141 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.401137+0000", 2024-09-11T03:38:38.141 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.66392193299998, 2024-09-11T03:38:38.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00082957199999999999, 2024-09-11T03:38:38.141 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.141 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.141 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.141 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401137+0000", 2024-09-11T03:38:38.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401137+0000", 2024-09-11T03:38:38.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401138+0000", 2024-09-11T03:38:38.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1999999999999999e-07 2024-09-11T03:38:38.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401163+0000", 2024-09-11T03:38:38.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5262999999999999e-05 2024-09-11T03:38:38.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401177+0000", 2024-09-11T03:38:38.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3937000000000001e-05 2024-09-11T03:38:38.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401183+0000", 2024-09-11T03:38:38.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2060000000000004e-06 2024-09-11T03:38:38.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401232+0000", 2024-09-11T03:38:38.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8943000000000002e-05 2024-09-11T03:38:38.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401254+0000", 2024-09-11T03:38:38.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1858999999999999e-05 2024-09-11T03:38:38.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401921+0000", 2024-09-11T03:38:38.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066758300000000002 2024-09-11T03:38:38.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401967+0000", 2024-09-11T03:38:38.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5061000000000001e-05 2024-09-11T03:38:38.146 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.146 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.146 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.146 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:78 1.0 e20/19 1:30306672:devicehealth::main.db.0000000000000000:head v 20'56, mlcod=20'53)", 2024-09-11T03:38:38.146 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.401530+0000", 2024-09-11T03:38:38.146 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.66352892200001, 2024-09-11T03:38:38.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00077311399999999998, 2024-09-11T03:38:38.146 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.147 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.147 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.147 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401530+0000", 2024-09-11T03:38:38.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401530+0000", 2024-09-11T03:38:38.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401531+0000", 2024-09-11T03:38:38.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4399999999999999e-07 2024-09-11T03:38:38.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401666+0000", 2024-09-11T03:38:38.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013499200000000001 2024-09-11T03:38:38.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401681+0000", 2024-09-11T03:38:38.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4859e-05 2024-09-11T03:38:38.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401687+0000", 2024-09-11T03:38:38.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7020000000000004e-06 2024-09-11T03:38:38.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401726+0000", 2024-09-11T03:38:38.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8868999999999997e-05 2024-09-11T03:38:38.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.401750+0000", 2024-09-11T03:38:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4233999999999999e-05 2024-09-11T03:38:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.402268+0000", 2024-09-11T03:38:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00051758200000000005 2024-09-11T03:38:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.402303+0000", 2024-09-11T03:38:38.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5132000000000002e-05 2024-09-11T03:38:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:99 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'73, mlcod=20'70)", 2024-09-11T03:38:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.465753+0000", 2024-09-11T03:38:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.59930594799999, 2024-09-11T03:38:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001236149, 2024-09-11T03:38:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.465753+0000", 2024-09-11T03:38:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.465753+0000", 2024-09-11T03:38:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.465754+0000", 2024-09-11T03:38:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7100000000000001e-07 2024-09-11T03:38:38.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.465784+0000", 2024-09-11T03:38:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0375e-05 2024-09-11T03:38:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.465793+0000", 2024-09-11T03:38:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2639999999999997e-06 2024-09-11T03:38:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.465799+0000", 2024-09-11T03:38:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9479999999999996e-06 2024-09-11T03:38:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.465841+0000", 2024-09-11T03:38:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1492000000000002e-05 2024-09-11T03:38:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.465859+0000", 2024-09-11T03:38:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8743e-05 2024-09-11T03:38:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.466945+0000", 2024-09-11T03:38:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001085239 2024-09-11T03:38:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.466989+0000", 2024-09-11T03:38:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4416999999999998e-05 2024-09-11T03:38:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:112 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'82, mlcod=20'80)", 2024-09-11T03:38:38.157 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.481285+0000", 2024-09-11T03:38:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.58377429799998, 2024-09-11T03:38:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00062958299999999997, 2024-09-11T03:38:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481285+0000", 2024-09-11T03:38:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481285+0000", 2024-09-11T03:38:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481285+0000", 2024-09-11T03:38:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9299999999999998e-07 2024-09-11T03:38:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481289+0000", 2024-09-11T03:38:38.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1819999999999999e-06 2024-09-11T03:38:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481300+0000", 2024-09-11T03:38:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.046e-05 2024-09-11T03:38:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481306+0000", 2024-09-11T03:38:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6250000000000001e-06 2024-09-11T03:38:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481365+0000", 2024-09-11T03:38:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8755000000000002e-05 2024-09-11T03:38:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481388+0000", 2024-09-11T03:38:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2705e-05 2024-09-11T03:38:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481876+0000", 2024-09-11T03:38:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00048821200000000002 2024-09-11T03:38:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481914+0000", 2024-09-11T03:38:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8050999999999999e-05 2024-09-11T03:38:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:113 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'83, mlcod=20'80)", 2024-09-11T03:38:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.481655+0000", 2024-09-11T03:38:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.58340426799998, 2024-09-11T03:38:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069826399999999998, 2024-09-11T03:38:38.162 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481655+0000", 2024-09-11T03:38:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481655+0000", 2024-09-11T03:38:38.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481655+0000", 2024-09-11T03:38:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6700000000000003e-07 2024-09-11T03:38:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481661+0000", 2024-09-11T03:38:38.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4070000000000001e-06 2024-09-11T03:38:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481674+0000", 2024-09-11T03:38:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3485999999999999e-05 2024-09-11T03:38:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481681+0000", 2024-09-11T03:38:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9070000000000001e-06 2024-09-11T03:38:38.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481732+0000", 2024-09-11T03:38:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0651e-05 2024-09-11T03:38:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.481753+0000", 2024-09-11T03:38:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1577999999999999e-05 2024-09-11T03:38:38.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.482314+0000", 2024-09-11T03:38:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00056093899999999999 2024-09-11T03:38:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.482353+0000", 2024-09-11T03:38:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8628999999999999e-05 2024-09-11T03:38:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.167 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:117 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'87, mlcod=20'84)", 2024-09-11T03:38:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.483823+0000", 2024-09-11T03:38:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.581235408, 2024-09-11T03:38:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074374300000000005, 2024-09-11T03:38:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:38.168 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.483823+0000", 2024-09-11T03:38:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.483823+0000", 2024-09-11T03:38:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.483824+0000", 2024-09-11T03:38:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3900000000000005e-07 2024-09-11T03:38:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.483859+0000", 2024-09-11T03:38:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4913000000000003e-05 2024-09-11T03:38:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.483869+0000", 2024-09-11T03:38:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0549e-05 2024-09-11T03:38:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.483876+0000", 2024-09-11T03:38:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2589999999999997e-06 2024-09-11T03:38:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.483954+0000", 2024-09-11T03:38:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8683000000000005e-05 2024-09-11T03:38:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.483978+0000", 2024-09-11T03:38:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3201999999999999e-05 2024-09-11T03:38:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.484534+0000", 2024-09-11T03:38:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00055668299999999998 2024-09-11T03:38:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.484567+0000", 2024-09-11T03:38:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2914999999999999e-05 2024-09-11T03:38:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.173 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "MOSDPGPull(1.0 e54/53 cost 1049576)", 2024-09-11T03:38:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:35:47.788200+0000", 2024-09-11T03:38:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 170.27685841900001, 2024-09-11T03:38:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010559440000000001, 2024-09-11T03:38:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-11T03:38:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:35:47.788200+0000", 2024-09-11T03:38:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:35:47.788200+0000", 2024-09-11T03:38:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:35:47.788201+0000", 2024-09-11T03:38:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.71e-07 2024-09-11T03:38:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:35:47.788212+0000", 2024-09-11T03:38:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0241000000000001e-05 2024-09-11T03:38:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:35:47.788223+0000", 2024-09-11T03:38:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1544e-05 2024-09-11T03:38:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:35:47.788230+0000", 2024-09-11T03:38:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1579999999999999e-06 2024-09-11T03:38:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:35:47.788315+0000", 2024-09-11T03:38:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5055000000000005e-05 2024-09-11T03:38:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:35:47.789256+0000", 2024-09-11T03:38:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00094107499999999998 2024-09-11T03:38:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.177 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:38.178 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:38.178 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:38:38.199 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:38.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:38.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:38.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:37 smithi175 ceph-mon[30894]: pgmap v304: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:38.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:37 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:38:38.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:37 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:38:38.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:37 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2806890797' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:38:38.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:37 smithi175 ceph-mon[30894]: from='client.? ' entity='client.2' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-11T03:38:38.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:37 smithi175 ceph-mon[30894]: osdmap e100: 12 total, 12 up, 12 in 2024-09-11T03:38:38.300 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:38.322 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:38.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:38.323 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:38.400 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:38:38.400 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:38:38.401 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:38:38.401 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:38:38.401 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:38:38.401 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:38:38.401 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:38:38.401 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:38:38.401 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:38:38.401 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:38:38.402 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:38:38.402 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:38:38.402 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:38:38.402 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:38:38.402 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:38:38.402 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:38:38.402 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:38:38.402 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:38:38.402 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:38:38.402 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:38:38.402 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:38:38.402 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:38:38.403 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:38:38.403 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:38:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:37 smithi121 ceph-mon[30729]: pgmap v304: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:38:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:37 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:38:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:37 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:38:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:37 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2806890797' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:38:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:37 smithi121 ceph-mon[30729]: from='client.? ' entity='client.2' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-11T03:38:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:37 smithi121 ceph-mon[30729]: osdmap e100: 12 total, 12 up, 12 in 2024-09-11T03:38:38.424 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:38.445 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:38.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:38.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:38.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:38.570 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:38.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:38.671 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:38:38.694 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:38.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:38.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:38:38.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:38.817 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:38.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:38.919 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:38.931 INFO:tasks.rados.rados.2.smithi175.stdout:55: done (0 left) 2024-09-11T03:38:38.932 INFO:tasks.rados.rados.2.smithi175.stdout:56: read oid 33 snap -1 2024-09-11T03:38:38.932 INFO:tasks.rados.rados.2.smithi175.stdout:56: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-11T03:38:38.932 INFO:tasks.rados.rados.2.smithi175.stdout:57: write (excl) oid 47 current snap is 1 2024-09-11T03:38:38.932 INFO:tasks.rados.rados.2.smithi175.stdout:57: seq_num 51 ranges {457157=535817,1768111=402677} 2024-09-11T03:38:38.937 INFO:tasks.rados.rados.2.smithi175.stdout:57: writing smithi17557870-47 from 457157 to 992974 tid 1 2024-09-11T03:38:38.941 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:38.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:38.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:38.943 INFO:tasks.rados.rados.2.smithi175.stdout:57: writing smithi17557870-47 from 1768111 to 2170788 tid 2 2024-09-11T03:38:38.943 INFO:tasks.rados.rados.2.smithi175.stdout:58: snap_create 2024-09-11T03:38:38.944 INFO:tasks.rados.rados.2.smithi175.stdout:56: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-11T03:38:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:57: finishing write tid 1 to smithi17557870-47 2024-09-11T03:38:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:57: finishing write tid 2 to smithi17557870-47 2024-09-11T03:38:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:57: finishing write tid 3 to smithi17557870-47 2024-09-11T03:38:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:57: finishing write tid 4 to smithi17557870-47 2024-09-11T03:38:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:57: oid 47 updating version 0 to 15 2024-09-11T03:38:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:57: oid 47 version 15 is already newer than 13 2024-09-11T03:38:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:57: oid 47 version 15 is already newer than 14 2024-09-11T03:38:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 15 (ObjNum 51 snap 1 seq_num 51) dirty exists 2024-09-11T03:38:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:57: left oid 47 (ObjNum 51 snap 1 seq_num 51) 2024-09-11T03:38:39.043 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:38:39.064 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:39.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:39.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:38:39.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:38 smithi084 ceph-mon[27254]: osdmap e101: 12 total, 12 up, 12 in 2024-09-11T03:38:39.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:38 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2660381237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:38:39.166 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:39.188 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:39.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:39.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:38 smithi175 ceph-mon[30894]: osdmap e101: 12 total, 12 up, 12 in 2024-09-11T03:38:39.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:38 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2660381237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:38:39.291 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:39.313 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:39.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:39.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:39.416 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:39.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:38 smithi121 ceph-mon[30729]: osdmap e101: 12 total, 12 up, 12 in 2024-09-11T03:38:39.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:38 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2660381237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:38:39.440 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:39.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:39.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:39.542 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:39.564 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:39.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:39.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:39.692 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:39.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:39.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:39.819 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:39.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:39.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:39.921 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:39.940 INFO:tasks.rados.rados.2.smithi175.stdout:56: done (2 left) 2024-09-11T03:38:39.940 INFO:tasks.rados.rados.2.smithi175.stdout:57: done (1 left) 2024-09-11T03:38:39.940 INFO:tasks.rados.rados.2.smithi175.stdout:58: done (0 left) 2024-09-11T03:38:39.940 INFO:tasks.rados.rados.2.smithi175.stdout:59: copy_from oid 44 from oid 48 current snap is 2 2024-09-11T03:38:39.940 INFO:tasks.rados.rados.2.smithi175.stdout:60: copy_from oid 11 from oid 3 current snap is 2 2024-09-11T03:38:39.940 INFO:tasks.rados.rados.2.smithi175.stdout:61: delete oid 37 current snap is 2 2024-09-11T03:38:39.943 INFO:tasks.rados.rados.2.smithi175.stdout:61: done (2 left) 2024-09-11T03:38:39.943 INFO:tasks.rados.rados.2.smithi175.stdout:62: snap_remove snap 1 2024-09-11T03:38:39.947 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:39.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:39.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:40.049 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:40.075 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:40.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:40.076 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:40.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:40.201 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:40.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:40.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:40.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:39 smithi175 ceph-mon[30894]: pgmap v307: 17 pgs: 17 active+clean; 30 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 15 op/s 2024-09-11T03:38:40.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:39 smithi175 ceph-mon[30894]: osdmap e102: 12 total, 12 up, 12 in 2024-09-11T03:38:40.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:39 smithi084 ceph-mon[27254]: pgmap v307: 17 pgs: 17 active+clean; 30 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 15 op/s 2024-09-11T03:38:40.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:39 smithi084 ceph-mon[27254]: osdmap e102: 12 total, 12 up, 12 in 2024-09-11T03:38:40.305 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:40.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:40.330 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T03:38:40.411 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:39 smithi121 ceph-mon[30729]: pgmap v307: 17 pgs: 17 active+clean; 30 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 15 op/s 2024-09-11T03:38:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:39 smithi121 ceph-mon[30729]: osdmap e102: 12 total, 12 up, 12 in 2024-09-11T03:38:40.431 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:40.452 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:40.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:40.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:40.554 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:40.580 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:40.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:40.581 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:40.682 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:40.706 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:40.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:40.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:40.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:38:40.830 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:40.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:40.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:38:40.925 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:40.932 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:40.942 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:38:40.948 INFO:tasks.rados.rados.2.smithi175.stdout:62: done (2 left) 2024-09-11T03:38:40.948 INFO:tasks.rados.rados.2.smithi175.stdout:63: snap_create 2024-09-11T03:38:40.948 INFO:tasks.rados.rados.2.smithi175.stdout:60: finishing copy_from racing read to smithi17557870-11 2024-09-11T03:38:40.948 INFO:tasks.rados.rados.2.smithi175.stdout:59: finishing copy_from racing read to smithi17557870-44 2024-09-11T03:38:40.948 INFO:tasks.rados.rados.2.smithi175.stdout:59: finishing copy_from to smithi17557870-44 2024-09-11T03:38:40.948 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 9 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-11T03:38:40.948 INFO:tasks.rados.rados.2.smithi175.stdout:60: finishing copy_from to smithi17557870-11 2024-09-11T03:38:40.948 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-11T03:38:40.958 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:40.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:40.959 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:41.060 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:41.082 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:41.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:41.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:41.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:40 smithi084 ceph-mon[27254]: osdmap e103: 12 total, 12 up, 12 in 2024-09-11T03:38:41.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:40 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/128494987' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:38:41.185 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:41.210 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:41.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:41.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:41.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:40 smithi175 ceph-mon[30894]: osdmap e103: 12 total, 12 up, 12 in 2024-09-11T03:38:41.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:40 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/128494987' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:38:41.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:41.319 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:38:41.339 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:41.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:41.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:40 smithi121 ceph-mon[30729]: osdmap e103: 12 total, 12 up, 12 in 2024-09-11T03:38:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:40 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/128494987' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:38:41.441 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:38:41.467 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:41.468 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:38:41.569 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:41.594 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:41.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:41.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:41.693 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:41.696 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:41.720 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:41.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:41.822 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:41.848 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:41.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:41.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:41.949 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T03:38:41.950 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:41.957 INFO:tasks.rados.rados.2.smithi175.stdout:59: done (2 left) 2024-09-11T03:38:41.957 INFO:tasks.rados.rados.2.smithi175.stdout:60: done (1 left) 2024-09-11T03:38:41.957 INFO:tasks.rados.rados.2.smithi175.stdout:63: done (0 left) 2024-09-11T03:38:41.957 INFO:tasks.rados.rados.2.smithi175.stdout:64: write (excl) oid 22 current snap is 3 2024-09-11T03:38:41.957 INFO:tasks.rados.rados.2.smithi175.stdout:64: seq_num 53 ranges {490771=686315,1885075=406346} 2024-09-11T03:38:41.963 INFO:tasks.rados.rados.2.smithi175.stdout:64: writing smithi17557870-22 from 490771 to 1177086 tid 1 2024-09-11T03:38:41.970 INFO:tasks.rados.rados.2.smithi175.stdout:64: writing smithi17557870-22 from 1885075 to 2291421 tid 2 2024-09-11T03:38:41.970 INFO:tasks.rados.rados.2.smithi175.stdout:65: snap_create 2024-09-11T03:38:41.975 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:41.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:41.976 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:41.983 INFO:tasks.rados.rados.2.smithi175.stdout:64: finishing write tid 1 to smithi17557870-22 2024-09-11T03:38:41.983 INFO:tasks.rados.rados.2.smithi175.stdout:64: finishing write tid 2 to smithi17557870-22 2024-09-11T03:38:41.983 INFO:tasks.rados.rados.2.smithi175.stdout:64: finishing write tid 3 to smithi17557870-22 2024-09-11T03:38:41.983 INFO:tasks.rados.rados.2.smithi175.stdout:64: finishing write tid 4 to smithi17557870-22 2024-09-11T03:38:41.983 INFO:tasks.rados.rados.2.smithi175.stdout:64: oid 22 updating version 0 to 20 2024-09-11T03:38:41.983 INFO:tasks.rados.rados.2.smithi175.stdout:64: oid 22 updating version 20 to 21 2024-09-11T03:38:41.984 INFO:tasks.rados.rados.2.smithi175.stdout:64: oid 22 updating version 21 to 22 2024-09-11T03:38:41.984 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 22 (ObjNum 53 snap 3 seq_num 53) dirty exists 2024-09-11T03:38:41.984 INFO:tasks.rados.rados.2.smithi175.stdout:64: left oid 22 (ObjNum 53 snap 3 seq_num 53) 2024-09-11T03:38:42.078 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:42.104 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:42.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:42.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:42.147 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:38:42.147 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:38:42.147 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:38:42.147 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:38:42.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:41 smithi175 ceph-mon[30894]: pgmap v310: 17 pgs: 17 active+clean; 30 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.8 MiB/s wr, 22 op/s 2024-09-11T03:38:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:41 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:38:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:41 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/128494987' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:38:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:41 smithi175 ceph-mon[30894]: osdmap e104: 12 total, 12 up, 12 in 2024-09-11T03:38:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:41 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2864713955' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:38:42.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:42.228 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:42.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:42.330 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:42.351 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:42.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:42.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:41 smithi084 ceph-mon[27254]: pgmap v310: 17 pgs: 17 active+clean; 30 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.8 MiB/s wr, 22 op/s 2024-09-11T03:38:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:41 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:38:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:41 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/128494987' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:38:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:41 smithi084 ceph-mon[27254]: osdmap e104: 12 total, 12 up, 12 in 2024-09-11T03:38:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:41 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2864713955' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:38:42.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:41 smithi121 ceph-mon[30729]: pgmap v310: 17 pgs: 17 active+clean; 30 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.8 MiB/s wr, 22 op/s 2024-09-11T03:38:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:41 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:38:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:41 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/128494987' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:38:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:41 smithi121 ceph-mon[30729]: osdmap e104: 12 total, 12 up, 12 in 2024-09-11T03:38:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:41 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2864713955' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:38:42.453 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:42.481 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:42.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:42.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:42.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:42.608 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:42.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:42.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:42.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:42.736 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:42.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:42.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:42.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:42.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:42.865 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:42.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:42.891 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T03:38:42.964 INFO:tasks.rados.rados.2.smithi175.stdout:64: done (1 left) 2024-09-11T03:38:42.964 INFO:tasks.rados.rados.2.smithi175.stdout:65: done (0 left) 2024-09-11T03:38:42.964 INFO:tasks.rados.rados.2.smithi175.stdout:66: rollback oid 24 current snap is 4 2024-09-11T03:38:42.965 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 2 2024-09-11T03:38:42.965 INFO:tasks.rados.rados.2.smithi175.stdout:67: read oid 42 snap -1 2024-09-11T03:38:42.965 INFO:tasks.rados.rados.2.smithi175.stdout:67: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-11T03:38:42.965 INFO:tasks.rados.rados.2.smithi175.stdout:68: snap_remove snap 3 2024-09-11T03:38:42.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:42.992 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:42.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:42.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:43.094 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:38:43.111 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:42 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2864713955' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:38:43.111 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:42 smithi084 ceph-mon[27254]: osdmap e105: 12 total, 12 up, 12 in 2024-09-11T03:38:43.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:43.118 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:43.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:38:43.219 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:38:43.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:42 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2864713955' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:38:43.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:42 smithi175 ceph-mon[30894]: osdmap e105: 12 total, 12 up, 12 in 2024-09-11T03:38:43.240 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:43.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:38:43.342 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:43.363 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:43.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:43.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:43.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:42 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2864713955' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:38:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:42 smithi121 ceph-mon[30729]: osdmap e105: 12 total, 12 up, 12 in 2024-09-11T03:38:43.464 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:43.490 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:43.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:43.592 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:38:43.618 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:43.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:43.619 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:38:43.661 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:38:43.720 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:38:43.728 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-11T03:38:43.789 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:43.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:43.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:38:43.891 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:38:43.916 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:43.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:43.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:38:43.963 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:43.965 INFO:tasks.rados.rados.2.smithi175.stdout:68: done (2 left) 2024-09-11T03:38:43.965 INFO:tasks.rados.rados.2.smithi175.stdout:69: read oid 30 snap -1 2024-09-11T03:38:43.965 INFO:tasks.rados.rados.2.smithi175.stdout:69: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-11T03:38:43.965 INFO:tasks.rados.rados.2.smithi175.stdout:70: write oid 10 current snap is 4 2024-09-11T03:38:43.965 INFO:tasks.rados.rados.2.smithi175.stdout:66: finishing rollback tid 0 to smithi17557870-24 2024-09-11T03:38:43.965 INFO:tasks.rados.rados.2.smithi175.stdout:70: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2024-09-11T03:38:43.970 INFO:tasks.rados.rados.2.smithi175.stdout:70: writing smithi17557870-10 from 507578 to 985495 tid 1 2024-09-11T03:38:43.974 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:38:43.975 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:38:43.976 INFO:tasks.rados.rados.2.smithi175.stdout:70: writing smithi17557870-10 from 1543557 to 2093561 tid 2 2024-09-11T03:38:43.981 INFO:tasks.rados.rados.2.smithi175.stdout:70: writing smithi17557870-10 from 2833665 to 2907578 tid 3 2024-09-11T03:38:43.981 INFO:tasks.rados.rados.2.smithi175.stdout:71: write (excl) oid 3 current snap is 4 2024-09-11T03:38:43.981 INFO:tasks.rados.rados.2.smithi175.stdout:66: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:38:43.981 INFO:tasks.rados.rados.2.smithi175.stdout:66: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:38:43.982 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 16 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-11T03:38:43.982 INFO:tasks.rados.rados.2.smithi175.stdout:67: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-11T03:38:43.996 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:38:43.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:38:43.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:43 smithi084 ceph-mon[27254]: pgmap v313: 17 pgs: 17 active+clean; 114 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 12 MiB/s wr, 60 op/s 2024-09-11T03:38:43.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:43 smithi084 ceph-mon[27254]: osdmap e106: 12 total, 12 up, 12 in 2024-09-11T03:38:43.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:43 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/563196413' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:38:44.014 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:38:44.014 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:38:44.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:44.030 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:38:44.030 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:38:44.044 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:44.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:44.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:44.046 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:38:44.046 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:38:44.063 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:38:44.063 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:38:44.066 INFO:tasks.rados.rados.2.smithi175.stdout:70: finishing write tid 1 to smithi17557870-10 2024-09-11T03:38:44.066 INFO:tasks.rados.rados.2.smithi175.stdout:69: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-11T03:38:44.080 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:38:44.080 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:38:44.097 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:38:44.097 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:38:44.114 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:38:44.114 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:38:44.130 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:38:44.130 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:38:44.146 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:38:44.147 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:38:44.148 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:38:44.149 INFO:tasks.rados.rados.2.smithi175.stdout:70: finishing write tid 2 to smithi17557870-10 2024-09-11T03:38:44.149 INFO:tasks.rados.rados.2.smithi175.stdout:70: finishing write tid 3 to smithi17557870-10 2024-09-11T03:38:44.149 INFO:tasks.rados.rados.2.smithi175.stdout:70: finishing write tid 4 to smithi17557870-10 2024-09-11T03:38:44.149 INFO:tasks.rados.rados.2.smithi175.stdout:70: finishing write tid 5 to smithi17557870-10 2024-09-11T03:38:44.149 INFO:tasks.rados.rados.2.smithi175.stdout:70: oid 10 updating version 0 to 18 2024-09-11T03:38:44.149 INFO:tasks.rados.rados.2.smithi175.stdout:70: oid 10 version 18 is already newer than 17 2024-09-11T03:38:44.149 INFO:tasks.rados.rados.2.smithi175.stdout:70: oid 10 updating version 18 to 19 2024-09-11T03:38:44.149 INFO:tasks.rados.rados.2.smithi175.stdout:70: oid 10 updating version 19 to 20 2024-09-11T03:38:44.149 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 20 (ObjNum 54 snap 4 seq_num 54) dirty exists 2024-09-11T03:38:44.149 INFO:tasks.rados.rados.2.smithi175.stdout:70: left oid 10 (ObjNum 54 snap 4 seq_num 54) 2024-09-11T03:38:44.150 INFO:tasks.rados.rados.2.smithi175.stdout:71: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-09-11T03:38:44.154 INFO:tasks.rados.rados.2.smithi175.stdout:71: writing smithi17557870-3 from 524385 to 1077551 tid 1 2024-09-11T03:38:44.159 INFO:tasks.rados.rados.2.smithi175.stdout:71: writing smithi17557870-3 from 1602039 to 2012054 tid 2 2024-09-11T03:38:44.165 INFO:tasks.rados.rados.2.smithi175.stdout:71: writing smithi17557870-3 from 2577441 to 2924385 tid 3 2024-09-11T03:38:44.165 INFO:tasks.rados.rados.2.smithi175.stdout:66: done (4 left) 2024-09-11T03:38:44.165 INFO:tasks.rados.rados.2.smithi175.stdout:67: done (3 left) 2024-09-11T03:38:44.165 INFO:tasks.rados.rados.2.smithi175.stdout:69: done (2 left) 2024-09-11T03:38:44.165 INFO:tasks.rados.rados.2.smithi175.stdout:70: done (1 left) 2024-09-11T03:38:44.165 INFO:tasks.rados.rados.2.smithi175.stdout:72: write (excl) oid 1 current snap is 4 2024-09-11T03:38:44.165 INFO:tasks.rados.rados.2.smithi175.stdout:71: finishing write tid 1 to smithi17557870-3 2024-09-11T03:38:44.165 INFO:tasks.rados.rados.2.smithi175.stdout:72: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2024-09-11T03:38:44.166 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:38:44.166 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:38:44.167 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:44.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:44.168 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:38:44.171 INFO:tasks.rados.rados.2.smithi175.stdout:72: writing smithi17557870-1 from 541192 to 1169607 tid 1 2024-09-11T03:38:44.176 INFO:tasks.rados.rados.2.smithi175.stdout:72: writing smithi17557870-1 from 1660521 to 2214194 tid 2 2024-09-11T03:38:44.182 INFO:tasks.rados.rados.2.smithi175.stdout:72: writing smithi17557870-1 from 2888511 to 2941192 tid 3 2024-09-11T03:38:44.182 INFO:tasks.rados.rados.2.smithi175.stdout:73: read oid 41 snap 2 2024-09-11T03:38:44.182 INFO:tasks.rados.rados.2.smithi175.stdout:73: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-11T03:38:44.182 INFO:tasks.rados.rados.2.smithi175.stdout:74: rollback oid 17 current snap is 4 2024-09-11T03:38:44.182 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 17 to 2 2024-09-11T03:38:44.182 INFO:tasks.rados.rados.2.smithi175.stdout:71: finishing write tid 2 to smithi17557870-3 2024-09-11T03:38:44.182 INFO:tasks.rados.rados.2.smithi175.stdout:71: finishing write tid 3 to smithi17557870-3 2024-09-11T03:38:44.182 INFO:tasks.rados.rados.2.smithi175.stdout:71: finishing write tid 4 to smithi17557870-3 2024-09-11T03:38:44.182 INFO:tasks.rados.rados.2.smithi175.stdout:71: finishing write tid 5 to smithi17557870-3 2024-09-11T03:38:44.182 INFO:tasks.rados.rados.2.smithi175.stdout:71: oid 3 updating version 0 to 25 2024-09-11T03:38:44.182 INFO:tasks.rados.rados.2.smithi175.stdout:71: oid 3 version 25 is already newer than 24 2024-09-11T03:38:44.182 INFO:tasks.rados.rados.2.smithi175.stdout:71: oid 3 updating version 25 to 27 2024-09-11T03:38:44.183 INFO:tasks.rados.rados.2.smithi175.stdout:71: oid 3 version 27 is already newer than 26 2024-09-11T03:38:44.183 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 27 (ObjNum 55 snap 4 seq_num 55) dirty exists 2024-09-11T03:38:44.183 INFO:tasks.rados.rados.2.smithi175.stdout:71: left oid 3 (ObjNum 55 snap 4 seq_num 55) 2024-09-11T03:38:44.183 INFO:tasks.rados.rados.2.smithi175.stdout:72: finishing write tid 1 to smithi17557870-1 2024-09-11T03:38:44.183 INFO:tasks.rados.rados.2.smithi175.stdout:72: finishing write tid 2 to smithi17557870-1 2024-09-11T03:38:44.183 INFO:tasks.rados.rados.2.smithi175.stdout:71: done (3 left) 2024-09-11T03:38:44.183 INFO:tasks.rados.rados.2.smithi175.stdout:75: write oid 22 current snap is 4 2024-09-11T03:38:44.183 INFO:tasks.rados.rados.2.smithi175.stdout:75: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2024-09-11T03:38:44.188 INFO:tasks.rados.rados.2.smithi175.stdout:75: writing smithi17557870-22 from 557999 to 1261663 tid 1 2024-09-11T03:38:44.192 INFO:tasks.rados.rados.2.smithi175.stdout:75: writing smithi17557870-22 from 1719003 to 2132687 tid 2 2024-09-11T03:38:44.197 INFO:tasks.rados.rados.2.smithi175.stdout:75: writing smithi17557870-22 from 2632287 to 2957999 tid 3 2024-09-11T03:38:44.197 INFO:tasks.rados.rados.2.smithi175.stdout:76: rollback oid 12 current snap is 4 2024-09-11T03:38:44.198 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 12 to 2 2024-09-11T03:38:44.198 INFO:tasks.rados.rados.2.smithi175.stdout:72: finishing write tid 3 to smithi17557870-1 2024-09-11T03:38:44.198 INFO:tasks.rados.rados.2.smithi175.stdout:77: delete oid 8 current snap is 4 2024-09-11T03:38:44.198 INFO:tasks.rados.rados.2.smithi175.stdout:72: finishing write tid 4 to smithi17557870-1 2024-09-11T03:38:44.198 INFO:tasks.rados.rados.2.smithi175.stdout:72: finishing write tid 5 to smithi17557870-1 2024-09-11T03:38:44.198 INFO:tasks.rados.rados.2.smithi175.stdout:72: oid 1 updating version 0 to 20 2024-09-11T03:38:44.198 INFO:tasks.rados.rados.2.smithi175.stdout:72: oid 1 version 20 is already newer than 17 2024-09-11T03:38:44.198 INFO:tasks.rados.rados.2.smithi175.stdout:72: oid 1 version 20 is already newer than 19 2024-09-11T03:38:44.198 INFO:tasks.rados.rados.2.smithi175.stdout:72: oid 1 version 20 is already newer than 18 2024-09-11T03:38:44.198 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 20 (ObjNum 56 snap 4 seq_num 56) dirty exists 2024-09-11T03:38:44.198 INFO:tasks.rados.rados.2.smithi175.stdout:72: left oid 1 (ObjNum 56 snap 4 seq_num 56) 2024-09-11T03:38:44.198 INFO:tasks.rados.rados.2.smithi175.stdout:74: finishing rollback tid 0 to smithi17557870-17 2024-09-11T03:38:44.198 INFO:tasks.rados.rados.2.smithi175.stdout:74: finishing rollback tid 1 to smithi17557870-17 2024-09-11T03:38:44.199 INFO:tasks.rados.rados.2.smithi175.stdout:74: finishing rollback tid 2 to smithi17557870-17 2024-09-11T03:38:44.199 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 24 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-11T03:38:44.199 INFO:tasks.rados.rados.2.smithi175.stdout:73: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-11T03:38:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:43 smithi175 ceph-mon[30894]: pgmap v313: 17 pgs: 17 active+clean; 114 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 12 MiB/s wr, 60 op/s 2024-09-11T03:38:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:43 smithi175 ceph-mon[30894]: osdmap e106: 12 total, 12 up, 12 in 2024-09-11T03:38:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:43 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/563196413' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:38:44.269 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:44.283 INFO:tasks.rados.rados.2.smithi175.stdout:75: finishing write tid 1 to smithi17557870-22 2024-09-11T03:38:44.284 INFO:tasks.rados.rados.2.smithi175.stdout:75: finishing write tid 2 to smithi17557870-22 2024-09-11T03:38:44.284 INFO:tasks.rados.rados.2.smithi175.stdout:72: done (5 left) 2024-09-11T03:38:44.284 INFO:tasks.rados.rados.2.smithi175.stdout:73: done (4 left) 2024-09-11T03:38:44.284 INFO:tasks.rados.rados.2.smithi175.stdout:74: done (3 left) 2024-09-11T03:38:44.284 INFO:tasks.rados.rados.2.smithi175.stdout:77: done (2 left) 2024-09-11T03:38:44.284 INFO:tasks.rados.rados.2.smithi175.stdout:78: read oid 30 snap 2 2024-09-11T03:38:44.284 INFO:tasks.rados.rados.2.smithi175.stdout:78: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-11T03:38:44.284 INFO:tasks.rados.rados.2.smithi175.stdout:79: write (excl) oid 37 current snap is 4 2024-09-11T03:38:44.284 INFO:tasks.rados.rados.2.smithi175.stdout:76: finishing rollback tid 0 to smithi17557870-12 2024-09-11T03:38:44.284 INFO:tasks.rados.rados.2.smithi175.stdout:79: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2024-09-11T03:38:44.289 INFO:tasks.rados.rados.2.smithi175.stdout:79: writing smithi17557870-37 from 591613 to 1045775 tid 1 2024-09-11T03:38:44.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:44.292 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:44.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:44.295 INFO:tasks.rados.rados.2.smithi175.stdout:79: writing smithi17557870-37 from 1719614 to 2136967 tid 2 2024-09-11T03:38:44.300 INFO:tasks.rados.rados.2.smithi175.stdout:79: writing smithi17557870-37 from 2570780 to 2991613 tid 3 2024-09-11T03:38:44.301 INFO:tasks.rados.rados.2.smithi175.stdout:80: write oid 35 current snap is 4 2024-09-11T03:38:44.301 INFO:tasks.rados.rados.2.smithi175.stdout:75: finishing write tid 3 to smithi17557870-22 2024-09-11T03:38:44.301 INFO:tasks.rados.rados.2.smithi175.stdout:76: finishing rollback tid 1 to smithi17557870-12 2024-09-11T03:38:44.301 INFO:tasks.rados.rados.2.smithi175.stdout:76: finishing rollback tid 2 to smithi17557870-12 2024-09-11T03:38:44.301 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 24 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-11T03:38:44.301 INFO:tasks.rados.rados.2.smithi175.stdout:75: finishing write tid 4 to smithi17557870-22 2024-09-11T03:38:44.301 INFO:tasks.rados.rados.2.smithi175.stdout:75: finishing write tid 5 to smithi17557870-22 2024-09-11T03:38:44.301 INFO:tasks.rados.rados.2.smithi175.stdout:75: oid 22 updating version 0 to 31 2024-09-11T03:38:44.301 INFO:tasks.rados.rados.2.smithi175.stdout:75: oid 22 version 31 is already newer than 29 2024-09-11T03:38:44.301 INFO:tasks.rados.rados.2.smithi175.stdout:75: oid 22 version 31 is already newer than 28 2024-09-11T03:38:44.301 INFO:tasks.rados.rados.2.smithi175.stdout:75: oid 22 version 31 is already newer than 30 2024-09-11T03:38:44.301 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 31 (ObjNum 57 snap 4 seq_num 57) dirty exists 2024-09-11T03:38:44.302 INFO:tasks.rados.rados.2.smithi175.stdout:75: left oid 22 (ObjNum 57 snap 4 seq_num 57) 2024-09-11T03:38:44.302 INFO:tasks.rados.rados.2.smithi175.stdout:80: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-09-11T03:38:44.306 INFO:tasks.rados.rados.2.smithi175.stdout:80: writing smithi17557870-35 from 608420 to 1137831 tid 1 2024-09-11T03:38:44.312 INFO:tasks.rados.rados.2.smithi175.stdout:80: writing smithi17557870-35 from 1778096 to 2339107 tid 2 2024-09-11T03:38:44.316 INFO:tasks.rados.rados.2.smithi175.stdout:80: writing smithi17557870-35 from 2881850 to 3008420 tid 3 2024-09-11T03:38:44.317 INFO:tasks.rados.rados.2.smithi175.stdout:75: done (4 left) 2024-09-11T03:38:44.317 INFO:tasks.rados.rados.2.smithi175.stdout:76: done (3 left) 2024-09-11T03:38:44.317 INFO:tasks.rados.rados.2.smithi175.stdout:81: read oid 13 snap -1 2024-09-11T03:38:44.317 INFO:tasks.rados.rados.2.smithi175.stdout:81: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-11T03:38:44.317 INFO:tasks.rados.rados.2.smithi175.stdout:82: snap_create 2024-09-11T03:38:44.317 INFO:tasks.rados.rados.2.smithi175.stdout:78: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-11T03:38:44.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:44.399 INFO:tasks.rados.rados.2.smithi175.stdout:79: finishing write tid 1 to smithi17557870-37 2024-09-11T03:38:44.399 INFO:tasks.rados.rados.2.smithi175.stdout:79: finishing write tid 2 to smithi17557870-37 2024-09-11T03:38:44.400 INFO:tasks.rados.rados.2.smithi175.stdout:79: finishing write tid 3 to smithi17557870-37 2024-09-11T03:38:44.400 INFO:tasks.rados.rados.2.smithi175.stdout:79: finishing write tid 4 to smithi17557870-37 2024-09-11T03:38:44.400 INFO:tasks.rados.rados.2.smithi175.stdout:79: finishing write tid 5 to smithi17557870-37 2024-09-11T03:38:44.400 INFO:tasks.rados.rados.2.smithi175.stdout:79: oid 37 updating version 0 to 14 2024-09-11T03:38:44.400 INFO:tasks.rados.rados.2.smithi175.stdout:79: oid 37 updating version 14 to 17 2024-09-11T03:38:44.400 INFO:tasks.rados.rados.2.smithi175.stdout:79: oid 37 version 17 is already newer than 15 2024-09-11T03:38:44.400 INFO:tasks.rados.rados.2.smithi175.stdout:79: oid 37 version 17 is already newer than 16 2024-09-11T03:38:44.400 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 17 (ObjNum 59 snap 4 seq_num 59) dirty exists 2024-09-11T03:38:44.400 INFO:tasks.rados.rados.2.smithi175.stdout:79: left oid 37 (ObjNum 59 snap 4 seq_num 59) 2024-09-11T03:38:44.400 INFO:tasks.rados.rados.2.smithi175.stdout:80: finishing write tid 1 to smithi17557870-35 2024-09-11T03:38:44.400 INFO:tasks.rados.rados.2.smithi175.stdout:80: finishing write tid 2 to smithi17557870-35 2024-09-11T03:38:44.400 INFO:tasks.rados.rados.2.smithi175.stdout:80: finishing write tid 3 to smithi17557870-35 2024-09-11T03:38:44.400 INFO:tasks.rados.rados.2.smithi175.stdout:80: finishing write tid 4 to smithi17557870-35 2024-09-11T03:38:44.401 INFO:tasks.rados.rados.2.smithi175.stdout:80: finishing write tid 5 to smithi17557870-35 2024-09-11T03:38:44.401 INFO:tasks.rados.rados.2.smithi175.stdout:80: oid 35 updating version 0 to 15 2024-09-11T03:38:44.401 INFO:tasks.rados.rados.2.smithi175.stdout:80: oid 35 updating version 15 to 17 2024-09-11T03:38:44.401 INFO:tasks.rados.rados.2.smithi175.stdout:80: oid 35 version 17 is already newer than 14 2024-09-11T03:38:44.401 INFO:tasks.rados.rados.2.smithi175.stdout:80: oid 35 version 17 is already newer than 16 2024-09-11T03:38:44.401 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 17 (ObjNum 60 snap 4 seq_num 60) dirty exists 2024-09-11T03:38:44.401 INFO:tasks.rados.rados.2.smithi175.stdout:80: left oid 35 (ObjNum 60 snap 4 seq_num 60) 2024-09-11T03:38:44.401 INFO:tasks.rados.rados.2.smithi175.stdout:81: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-11T03:38:44.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:44.421 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:44.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:43 smithi121 ceph-mon[30729]: pgmap v313: 17 pgs: 17 active+clean; 114 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 12 MiB/s wr, 60 op/s 2024-09-11T03:38:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:43 smithi121 ceph-mon[30729]: osdmap e106: 12 total, 12 up, 12 in 2024-09-11T03:38:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:43 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/563196413' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:38:44.522 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:44.544 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:44.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:44.545 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:44.646 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:44.668 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:44.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:44.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:44.770 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:44.791 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:44.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:44.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:44.893 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:44.914 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:44.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:44.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:44.972 INFO:tasks.rados.rados.2.smithi175.stdout:78: done (4 left) 2024-09-11T03:38:44.972 INFO:tasks.rados.rados.2.smithi175.stdout:79: done (3 left) 2024-09-11T03:38:44.972 INFO:tasks.rados.rados.2.smithi175.stdout:80: done (2 left) 2024-09-11T03:38:44.972 INFO:tasks.rados.rados.2.smithi175.stdout:81: done (1 left) 2024-09-11T03:38:44.972 INFO:tasks.rados.rados.2.smithi175.stdout:82: done (0 left) 2024-09-11T03:38:44.973 INFO:tasks.rados.rados.2.smithi175.stdout:83: delete oid 13 current snap is 5 2024-09-11T03:38:44.973 INFO:tasks.rados.rados.2.smithi175.stdout:83: done (0 left) 2024-09-11T03:38:44.974 INFO:tasks.rados.rados.2.smithi175.stdout:84: read oid 13 snap -1 2024-09-11T03:38:44.974 INFO:tasks.rados.rados.2.smithi175.stdout:84: expect deleted 2024-09-11T03:38:44.974 INFO:tasks.rados.rados.2.smithi175.stdout:85: snap_remove snap 2 2024-09-11T03:38:45.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:44 smithi084 ceph-mon[27254]: osdmap e107: 12 total, 12 up, 12 in 2024-09-11T03:38:45.016 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:45.034 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:38:45.034 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:38:45.038 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:45.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:45.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:45.044 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:45.140 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:45.163 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:45.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:45.164 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:45.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:44 smithi175 ceph-mon[30894]: osdmap e107: 12 total, 12 up, 12 in 2024-09-11T03:38:45.260 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:45.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:45.290 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:45.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:45.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:45.392 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:45.418 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:45.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:45.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:45.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:44 smithi121 ceph-mon[30729]: osdmap e107: 12 total, 12 up, 12 in 2024-09-11T03:38:45.520 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:45.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:45.543 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:45.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:45.645 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:45.667 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:45.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:45.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:45.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:45.795 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:45.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:45.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:45.897 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:45.920 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:45.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:45.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:45.957 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T03:38:45.975 INFO:tasks.rados.rados.2.smithi175.stdout:85: done (1 left) 2024-09-11T03:38:45.975 INFO:tasks.rados.rados.2.smithi175.stdout:86: read oid 6 snap -1 2024-09-11T03:38:45.975 INFO:tasks.rados.rados.2.smithi175.stdout:86: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-11T03:38:45.975 INFO:tasks.rados.rados.2.smithi175.stdout:87: read oid 24 snap 4 2024-09-11T03:38:45.975 INFO:tasks.rados.rados.2.smithi175.stdout:87: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-11T03:38:45.975 INFO:tasks.rados.rados.2.smithi175.stdout:88: rollback oid 17 current snap is 5 2024-09-11T03:38:45.975 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 17 to 4 2024-09-11T03:38:45.975 INFO:tasks.rados.rados.2.smithi175.stdout:84: done (3 left) 2024-09-11T03:38:45.975 INFO:tasks.rados.rados.2.smithi175.stdout:89: copy_from oid 30 from oid 39 current snap is 5 2024-09-11T03:38:45.976 INFO:tasks.rados.rados.2.smithi175.stdout:90: write oid 13 current snap is 5 2024-09-11T03:38:45.976 INFO:tasks.rados.rados.2.smithi175.stdout:90: seq_num 62 ranges {642034=796262,1895060=564680} 2024-09-11T03:38:45.982 INFO:tasks.rados.rados.2.smithi175.stdout:90: writing smithi17557870-13 from 642034 to 1438296 tid 1 2024-09-11T03:38:45.989 INFO:tasks.rados.rados.2.smithi175.stdout:90: writing smithi17557870-13 from 1895060 to 2459740 tid 2 2024-09-11T03:38:45.989 INFO:tasks.rados.rados.2.smithi175.stdout:91: copy_from oid 9 from oid 47 current snap is 5 2024-09-11T03:38:45.989 INFO:tasks.rados.rados.2.smithi175.stdout:88: finishing rollback tid 0 to smithi17557870-17 2024-09-11T03:38:45.989 INFO:tasks.rados.rados.2.smithi175.stdout:88: finishing rollback tid 1 to smithi17557870-17 2024-09-11T03:38:45.990 INFO:tasks.rados.rados.2.smithi175.stdout:88: finishing rollback tid 2 to smithi17557870-17 2024-09-11T03:38:45.990 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 32 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-11T03:38:45.990 INFO:tasks.rados.rados.2.smithi175.stdout:87: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-11T03:38:46.021 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:46.042 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:46.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:46.072 INFO:tasks.rados.rados.2.smithi175.stdout:90: finishing write tid 1 to smithi17557870-13 2024-09-11T03:38:46.072 INFO:tasks.rados.rados.2.smithi175.stdout:86: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-11T03:38:46.141 INFO:tasks.rados.rados.2.smithi175.stdout:90: finishing write tid 2 to smithi17557870-13 2024-09-11T03:38:46.141 INFO:tasks.rados.rados.2.smithi175.stdout:90: finishing write tid 3 to smithi17557870-13 2024-09-11T03:38:46.141 INFO:tasks.rados.rados.2.smithi175.stdout:90: finishing write tid 4 to smithi17557870-13 2024-09-11T03:38:46.142 INFO:tasks.rados.rados.2.smithi175.stdout:90: oid 13 updating version 0 to 35 2024-09-11T03:38:46.142 INFO:tasks.rados.rados.2.smithi175.stdout:90: oid 13 version 35 is already newer than 34 2024-09-11T03:38:46.142 INFO:tasks.rados.rados.2.smithi175.stdout:90: oid 13 version 35 is already newer than 33 2024-09-11T03:38:46.142 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 35 (ObjNum 62 snap 5 seq_num 62) dirty exists 2024-09-11T03:38:46.142 INFO:tasks.rados.rados.2.smithi175.stdout:90: left oid 13 (ObjNum 62 snap 5 seq_num 62) 2024-09-11T03:38:46.142 INFO:tasks.rados.rados.2.smithi175.stdout:89: finishing copy_from racing read to smithi17557870-30 2024-09-11T03:38:46.142 INFO:tasks.rados.rados.2.smithi175.stdout:89: finishing copy_from to smithi17557870-30 2024-09-11T03:38:46.142 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-11T03:38:46.142 INFO:tasks.rados.rados.2.smithi175.stdout:86: done (5 left) 2024-09-11T03:38:46.142 INFO:tasks.rados.rados.2.smithi175.stdout:87: done (4 left) 2024-09-11T03:38:46.142 INFO:tasks.rados.rados.2.smithi175.stdout:88: done (3 left) 2024-09-11T03:38:46.142 INFO:tasks.rados.rados.2.smithi175.stdout:89: done (2 left) 2024-09-11T03:38:46.142 INFO:tasks.rados.rados.2.smithi175.stdout:90: done (1 left) 2024-09-11T03:38:46.142 INFO:tasks.rados.rados.2.smithi175.stdout:92: snap_remove snap 4 2024-09-11T03:38:46.145 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:38:46.166 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:46.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:46.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:38:46.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:45 smithi175 ceph-mon[30894]: pgmap v316: 17 pgs: 17 active+clean; 114 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 12 MiB/s wr, 62 op/s 2024-09-11T03:38:46.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:45 smithi175 ceph-mon[30894]: osdmap e108: 12 total, 12 up, 12 in 2024-09-11T03:38:46.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:45 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:38:46.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:45 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:38:46.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:45 smithi084 ceph-mon[27254]: pgmap v316: 17 pgs: 17 active+clean; 114 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 12 MiB/s wr, 62 op/s 2024-09-11T03:38:46.232 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:45 smithi084 ceph-mon[27254]: osdmap e108: 12 total, 12 up, 12 in 2024-09-11T03:38:46.232 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:45 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:38:46.232 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:45 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:38:46.268 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:46.289 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:46.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:46.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:46.392 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:46.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:46.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:46.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:45 smithi121 ceph-mon[30729]: pgmap v316: 17 pgs: 17 active+clean; 114 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 12 MiB/s wr, 62 op/s 2024-09-11T03:38:46.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:45 smithi121 ceph-mon[30729]: osdmap e108: 12 total, 12 up, 12 in 2024-09-11T03:38:46.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:45 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:38:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:45 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:38:46.520 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:38:46.541 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:46.542 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:38:46.643 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:38:46.665 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:46.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:46.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:38:46.743 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:46.767 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:46.768 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:38:46.789 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:46.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:46.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:46.806 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:38:46.810 INFO:tasks.rados.rados.2.smithi175.stdout:92: done (1 left) 2024-09-11T03:38:46.810 INFO:tasks.rados.rados.2.smithi175.stdout:93: copy_from oid 37 from oid 19 current snap is 5 2024-09-11T03:38:46.810 INFO:tasks.rados.rados.2.smithi175.stdout:91: finishing copy_from racing read to smithi17557870-9 2024-09-11T03:38:46.810 INFO:tasks.rados.rados.2.smithi175.stdout:91: finishing copy_from to smithi17557870-9 2024-09-11T03:38:46.810 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 15 (ObjNum 51 snap 1 seq_num 51) dirty exists 2024-09-11T03:38:46.810 INFO:tasks.rados.rados.2.smithi175.stdout:91: done (1 left) 2024-09-11T03:38:46.810 INFO:tasks.rados.rados.2.smithi175.stdout:94: delete oid 38 current snap is 5 2024-09-11T03:38:46.812 INFO:tasks.rados.rados.2.smithi175.stdout:94: done (1 left) 2024-09-11T03:38:46.813 INFO:tasks.rados.rados.2.smithi175.stdout:95: write oid 23 current snap is 5 2024-09-11T03:38:46.813 INFO:tasks.rados.rados.2.smithi175.stdout:95: seq_num 64 ranges {675648=546760,2012024=568349} 2024-09-11T03:38:46.819 INFO:tasks.rados.rados.2.smithi175.stdout:95: writing smithi17557870-23 from 675648 to 1222408 tid 1 2024-09-11T03:38:46.827 INFO:tasks.rados.rados.2.smithi175.stdout:95: writing smithi17557870-23 from 2012024 to 2580373 tid 2 2024-09-11T03:38:46.827 INFO:tasks.rados.rados.2.smithi175.stdout:96: delete oid 48 current snap is 5 2024-09-11T03:38:46.827 INFO:tasks.rados.rados.2.smithi175.stdout:93: finishing copy_from racing read to smithi17557870-37 2024-09-11T03:38:46.827 INFO:tasks.rados.rados.2.smithi175.stdout:93: finishing copy_from to smithi17557870-37 2024-09-11T03:38:46.827 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-11T03:38:46.827 INFO:tasks.rados.rados.2.smithi175.stdout:95: finishing write tid 1 to smithi17557870-23 2024-09-11T03:38:46.828 INFO:tasks.rados.rados.2.smithi175.stdout:93: done (2 left) 2024-09-11T03:38:46.829 INFO:tasks.rados.rados.2.smithi175.stdout:96: done (1 left) 2024-09-11T03:38:46.829 INFO:tasks.rados.rados.2.smithi175.stdout:97: snap_create 2024-09-11T03:38:46.830 INFO:tasks.rados.rados.2.smithi175.stdout:95: finishing write tid 2 to smithi17557870-23 2024-09-11T03:38:46.830 INFO:tasks.rados.rados.2.smithi175.stdout:95: finishing write tid 3 to smithi17557870-23 2024-09-11T03:38:46.831 INFO:tasks.rados.rados.2.smithi175.stdout:95: finishing write tid 4 to smithi17557870-23 2024-09-11T03:38:46.831 INFO:tasks.rados.rados.2.smithi175.stdout:95: oid 23 updating version 0 to 15 2024-09-11T03:38:46.831 INFO:tasks.rados.rados.2.smithi175.stdout:95: oid 23 version 15 is already newer than 14 2024-09-11T03:38:46.831 INFO:tasks.rados.rados.2.smithi175.stdout:95: oid 23 updating version 15 to 16 2024-09-11T03:38:46.831 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 16 (ObjNum 64 snap 5 seq_num 64) dirty exists 2024-09-11T03:38:46.831 INFO:tasks.rados.rados.2.smithi175.stdout:95: left oid 23 (ObjNum 64 snap 5 seq_num 64) 2024-09-11T03:38:46.891 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:46.913 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:46.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:46.914 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:47.015 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:47.037 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:47.037 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:47.138 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:46 smithi084 ceph-mon[27254]: osdmap e109: 12 total, 12 up, 12 in 2024-09-11T03:38:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1399398799' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:38:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:46 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:38:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1399398799' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:38:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:46 smithi084 ceph-mon[27254]: osdmap e110: 12 total, 12 up, 12 in 2024-09-11T03:38:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2998878108' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:38:47.159 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:47.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:47.160 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:46 smithi175 ceph-mon[30894]: osdmap e109: 12 total, 12 up, 12 in 2024-09-11T03:38:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1399398799' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:38:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:46 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:38:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1399398799' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:38:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:46 smithi175 ceph-mon[30894]: osdmap e110: 12 total, 12 up, 12 in 2024-09-11T03:38:47.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2998878108' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:38:47.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:47.287 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:47.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:47.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:47.389 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:47.415 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:47.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:47.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:47.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:46 smithi121 ceph-mon[30729]: osdmap e109: 12 total, 12 up, 12 in 2024-09-11T03:38:47.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1399398799' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:38:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:46 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:38:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1399398799' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:38:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:46 smithi121 ceph-mon[30729]: osdmap e110: 12 total, 12 up, 12 in 2024-09-11T03:38:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2998878108' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:38:47.517 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:47.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:47.539 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:47.540 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:47.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:47.664 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:47.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:47.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:47.791 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:47.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:47.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:47.808 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 1 (10000) 2024-09-11T03:38:47.810 INFO:tasks.rados.rados.2.smithi175.stdout:95: done (1 left) 2024-09-11T03:38:47.810 INFO:tasks.rados.rados.2.smithi175.stdout:97: done (0 left) 2024-09-11T03:38:47.810 INFO:tasks.rados.rados.2.smithi175.stdout:98: snap_remove snap 5 2024-09-11T03:38:47.816 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:38:47.816 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:38:47.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:47.917 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:47.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:47.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:48.019 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:38:48.042 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:48.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:48.043 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:38:48.144 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:48.166 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:48.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:48.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:48.268 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:38:48.291 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:48.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:48.292 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:38:48.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:47 smithi084 ceph-mon[27254]: pgmap v319: 17 pgs: 17 active+clean; 122 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 2.1 MiB/s wr, 18 op/s 2024-09-11T03:38:48.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:47 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2998878108' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:38:48.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:47 smithi084 ceph-mon[27254]: osdmap e111: 12 total, 12 up, 12 in 2024-09-11T03:38:48.340 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T03:38:48.392 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:48.417 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:48.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:48.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:48.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:47 smithi121 ceph-mon[30729]: pgmap v319: 17 pgs: 17 active+clean; 122 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 2.1 MiB/s wr, 18 op/s 2024-09-11T03:38:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:47 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2998878108' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:38:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:47 smithi121 ceph-mon[30729]: osdmap e111: 12 total, 12 up, 12 in 2024-09-11T03:38:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:47 smithi175 ceph-mon[30894]: pgmap v319: 17 pgs: 17 active+clean; 122 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 2.1 MiB/s wr, 18 op/s 2024-09-11T03:38:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:47 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2998878108' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:38:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:47 smithi175 ceph-mon[30894]: osdmap e111: 12 total, 12 up, 12 in 2024-09-11T03:38:48.520 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:48.542 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:48.543 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:48.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:48.544 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:48.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:48.671 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:48.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:48.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:48.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:48.798 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:48.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:48.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:48.812 INFO:tasks.rados.rados.2.smithi175.stdout:98: done (0 left) 2024-09-11T03:38:48.812 INFO:tasks.rados.rados.2.smithi175.stdout:99: copy_from oid 14 from oid 48 current snap is 6 2024-09-11T03:38:48.812 INFO:tasks.rados.rados.2.smithi175.stdout:100: rollback oid 37 current snap is 6 2024-09-11T03:38:48.812 INFO:tasks.rados.rados.2.smithi175.stdout:100: done (1 left) 2024-09-11T03:38:48.812 INFO:tasks.rados.rados.2.smithi175.stdout:101: rollback oid 17 current snap is 6 2024-09-11T03:38:48.813 INFO:tasks.rados.rados.2.smithi175.stdout:101: done (1 left) 2024-09-11T03:38:48.813 INFO:tasks.rados.rados.2.smithi175.stdout:102: write oid 31 current snap is 6 2024-09-11T03:38:48.813 INFO:tasks.rados.rados.2.smithi175.stdout:102: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-09-11T03:38:48.819 INFO:tasks.rados.rados.2.smithi175.stdout:102: writing smithi17557870-31 from 709262 to 1406520 tid 1 2024-09-11T03:38:48.826 INFO:tasks.rados.rados.2.smithi175.stdout:102: writing smithi17557870-31 from 2012635 to 2584653 tid 2 2024-09-11T03:38:48.829 INFO:tasks.rados.rados.2.smithi175.stdout:102: writing smithi17557870-31 from 3046388 to 3109262 tid 3 2024-09-11T03:38:48.829 INFO:tasks.rados.rados.2.smithi175.stdout:103: copy_from oid 1 from oid 30 current snap is 6 2024-09-11T03:38:48.830 INFO:tasks.rados.rados.2.smithi175.stdout:99: finishing copy_from to smithi17557870-14 2024-09-11T03:38:48.830 INFO:tasks.rados.rados.2.smithi175.stdout:99: got expected ENOENT (src dne) 2024-09-11T03:38:48.830 INFO:tasks.rados.rados.2.smithi175.stdout:99: finishing copy_from racing read to smithi17557870-14 2024-09-11T03:38:48.830 INFO:tasks.rados.rados.2.smithi175.stdout:99: done (2 left) 2024-09-11T03:38:48.830 INFO:tasks.rados.rados.2.smithi175.stdout:104: delete oid 42 current snap is 6 2024-09-11T03:38:48.836 INFO:tasks.rados.rados.2.smithi175.stdout:104: done (2 left) 2024-09-11T03:38:48.837 INFO:tasks.rados.rados.2.smithi175.stdout:105: read oid 16 snap -1 2024-09-11T03:38:48.837 INFO:tasks.rados.rados.2.smithi175.stdout:105: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-11T03:38:48.837 INFO:tasks.rados.rados.2.smithi175.stdout:106: copy_from oid 39 from oid 40 current snap is 6 2024-09-11T03:38:48.837 INFO:tasks.rados.rados.2.smithi175.stdout:107: copy_from oid 35 from oid 20 current snap is 6 2024-09-11T03:38:48.837 INFO:tasks.rados.rados.2.smithi175.stdout:108: rollback oid 33 current snap is 6 2024-09-11T03:38:48.837 INFO:tasks.rados.rados.2.smithi175.stdout:108: done (5 left) 2024-09-11T03:38:48.837 INFO:tasks.rados.rados.2.smithi175.stdout:109: snap_create 2024-09-11T03:38:48.841 INFO:tasks.rados.rados.2.smithi175.stdout:105: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-11T03:38:48.900 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:48.918 INFO:tasks.rados.rados.2.smithi175.stdout:107: finishing copy_from racing read to smithi17557870-35 2024-09-11T03:38:48.919 INFO:tasks.rados.rados.2.smithi175.stdout:107: finishing copy_from to smithi17557870-35 2024-09-11T03:38:48.919 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 9 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-11T03:38:48.919 INFO:tasks.rados.rados.2.smithi175.stdout:106: finishing copy_from racing read to smithi17557870-39 2024-09-11T03:38:48.919 INFO:tasks.rados.rados.2.smithi175.stdout:106: finishing copy_from to smithi17557870-39 2024-09-11T03:38:48.919 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-11T03:38:48.919 INFO:tasks.rados.rados.2.smithi175.stdout:102: finishing write tid 1 to smithi17557870-31 2024-09-11T03:38:48.919 INFO:tasks.rados.rados.2.smithi175.stdout:102: finishing write tid 2 to smithi17557870-31 2024-09-11T03:38:48.919 INFO:tasks.rados.rados.2.smithi175.stdout:102: finishing write tid 3 to smithi17557870-31 2024-09-11T03:38:48.919 INFO:tasks.rados.rados.2.smithi175.stdout:102: finishing write tid 4 to smithi17557870-31 2024-09-11T03:38:48.919 INFO:tasks.rados.rados.2.smithi175.stdout:102: finishing write tid 5 to smithi17557870-31 2024-09-11T03:38:48.919 INFO:tasks.rados.rados.2.smithi175.stdout:102: oid 31 updating version 0 to 23 2024-09-11T03:38:48.919 INFO:tasks.rados.rados.2.smithi175.stdout:102: oid 31 version 23 is already newer than 22 2024-09-11T03:38:48.920 INFO:tasks.rados.rados.2.smithi175.stdout:102: oid 31 version 23 is already newer than 21 2024-09-11T03:38:48.920 INFO:tasks.rados.rados.2.smithi175.stdout:102: oid 31 version 23 is already newer than 20 2024-09-11T03:38:48.920 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 23 (ObjNum 66 snap 6 seq_num 66) dirty exists 2024-09-11T03:38:48.920 INFO:tasks.rados.rados.2.smithi175.stdout:102: left oid 31 (ObjNum 66 snap 6 seq_num 66) 2024-09-11T03:38:48.920 INFO:tasks.rados.rados.2.smithi175.stdout:103: finishing copy_from racing read to smithi17557870-1 2024-09-11T03:38:48.920 INFO:tasks.rados.rados.2.smithi175.stdout:103: finishing copy_from to smithi17557870-1 2024-09-11T03:38:48.920 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-11T03:38:48.921 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:48.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:48.922 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:49.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:49.047 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:49.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:49.149 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:49.173 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:49.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:49.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:49.276 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:49.298 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:49.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:49.299 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:49.320 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-11T03:38:49.400 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:38:49.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:49.421 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:49.422 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:38:49.523 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:49.544 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:49.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:49.578 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:38:49.579 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:38:49.582 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:49.598 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:38:49.598 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:38:49.616 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:38:49.617 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:38:49.633 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:38:49.633 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:38:49.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:49.650 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:38:49.650 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:38:49.668 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:38:49.668 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:38:49.673 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:49.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:38:49.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:49.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:49.685 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:38:49.686 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:38:49.703 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:38:49.703 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:38:49.719 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:38:49.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:38:49.736 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:38:49.737 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:38:49.775 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:49.796 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:49.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:49.797 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:49.819 INFO:tasks.rados.rados.2.smithi175.stdout:102: done (5 left) 2024-09-11T03:38:49.819 INFO:tasks.rados.rados.2.smithi175.stdout:103: done (4 left) 2024-09-11T03:38:49.819 INFO:tasks.rados.rados.2.smithi175.stdout:105: done (3 left) 2024-09-11T03:38:49.819 INFO:tasks.rados.rados.2.smithi175.stdout:106: done (2 left) 2024-09-11T03:38:49.819 INFO:tasks.rados.rados.2.smithi175.stdout:107: done (1 left) 2024-09-11T03:38:49.820 INFO:tasks.rados.rados.2.smithi175.stdout:109: done (0 left) 2024-09-11T03:38:49.820 INFO:tasks.rados.rados.2.smithi175.stdout:110: copy_from oid 3 from oid 37 current snap is 7 2024-09-11T03:38:49.820 INFO:tasks.rados.rados.2.smithi175.stdout:111: copy_from oid 43 from oid 30 current snap is 7 2024-09-11T03:38:49.820 INFO:tasks.rados.rados.2.smithi175.stdout:112: write oid 25 current snap is 7 2024-09-11T03:38:49.820 INFO:tasks.rados.rados.2.smithi175.stdout:112: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-09-11T03:38:49.825 INFO:tasks.rados.rados.2.smithi175.stdout:112: writing smithi17557870-25 from 742876 to 1190632 tid 1 2024-09-11T03:38:49.836 INFO:tasks.rados.rados.2.smithi175.stdout:112: writing smithi17557870-25 from 1729599 to 2305286 tid 2 2024-09-11T03:38:49.837 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:49 smithi084 ceph-mon[27254]: pgmap v322: 17 pgs: 17 active+clean; 122 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 3.7 MiB/s wr, 30 op/s 2024-09-11T03:38:49.837 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:49 smithi084 ceph-mon[27254]: osdmap e112: 12 total, 12 up, 12 in 2024-09-11T03:38:49.837 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/743461033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:38:49.838 INFO:tasks.rados.rados.2.smithi175.stdout:112: writing smithi17557870-25 from 3101234 to 3142876 tid 3 2024-09-11T03:38:49.838 INFO:tasks.rados.rados.2.smithi175.stdout:113: write (excl) oid 7 current snap is 7 2024-09-11T03:38:49.838 INFO:tasks.rados.rados.2.smithi175.stdout:113: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2024-09-11T03:38:49.844 INFO:tasks.rados.rados.2.smithi175.stdout:113: writing smithi17557870-7 from 759683 to 1399041 tid 1 2024-09-11T03:38:49.851 INFO:tasks.rados.rados.2.smithi175.stdout:113: writing smithi17557870-7 from 1904434 to 2623779 tid 2 2024-09-11T03:38:49.855 INFO:tasks.rados.rados.2.smithi175.stdout:113: writing smithi17557870-7 from 3128657 to 3159683 tid 3 2024-09-11T03:38:49.855 INFO:tasks.rados.rados.2.smithi175.stdout:114: delete oid 16 current snap is 7 2024-09-11T03:38:49.855 INFO:tasks.rados.rados.2.smithi175.stdout:111: finishing copy_from racing read to smithi17557870-43 2024-09-11T03:38:49.855 INFO:tasks.rados.rados.2.smithi175.stdout:112: finishing write tid 1 to smithi17557870-25 2024-09-11T03:38:49.855 INFO:tasks.rados.rados.2.smithi175.stdout:110: finishing copy_from racing read to smithi17557870-3 2024-09-11T03:38:49.855 INFO:tasks.rados.rados.2.smithi175.stdout:111: finishing copy_from to smithi17557870-43 2024-09-11T03:38:49.855 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-11T03:38:49.855 INFO:tasks.rados.rados.2.smithi175.stdout:110: finishing copy_from to smithi17557870-3 2024-09-11T03:38:49.856 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-11T03:38:49.856 INFO:tasks.rados.rados.2.smithi175.stdout:112: finishing write tid 2 to smithi17557870-25 2024-09-11T03:38:49.856 INFO:tasks.rados.rados.2.smithi175.stdout:112: finishing write tid 3 to smithi17557870-25 2024-09-11T03:38:49.856 INFO:tasks.rados.rados.2.smithi175.stdout:112: finishing write tid 4 to smithi17557870-25 2024-09-11T03:38:49.856 INFO:tasks.rados.rados.2.smithi175.stdout:112: finishing write tid 5 to smithi17557870-25 2024-09-11T03:38:49.856 INFO:tasks.rados.rados.2.smithi175.stdout:112: oid 25 updating version 0 to 7 2024-09-11T03:38:49.856 INFO:tasks.rados.rados.2.smithi175.stdout:112: oid 25 version 7 is already newer than 5 2024-09-11T03:38:49.856 INFO:tasks.rados.rados.2.smithi175.stdout:112: oid 25 version 7 is already newer than 6 2024-09-11T03:38:49.856 INFO:tasks.rados.rados.2.smithi175.stdout:112: oid 25 updating version 7 to 8 2024-09-11T03:38:49.856 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 8 (ObjNum 68 snap 7 seq_num 68) dirty exists 2024-09-11T03:38:49.856 INFO:tasks.rados.rados.2.smithi175.stdout:112: left oid 25 (ObjNum 68 snap 7 seq_num 68) 2024-09-11T03:38:49.856 INFO:tasks.rados.rados.2.smithi175.stdout:113: finishing write tid 1 to smithi17557870-7 2024-09-11T03:38:49.857 INFO:tasks.rados.rados.2.smithi175.stdout:110: done (4 left) 2024-09-11T03:38:49.857 INFO:tasks.rados.rados.2.smithi175.stdout:111: done (3 left) 2024-09-11T03:38:49.857 INFO:tasks.rados.rados.2.smithi175.stdout:112: done (2 left) 2024-09-11T03:38:49.857 INFO:tasks.rados.rados.2.smithi175.stdout:114: done (1 left) 2024-09-11T03:38:49.857 INFO:tasks.rados.rados.2.smithi175.stdout:115: snap_remove snap 6 2024-09-11T03:38:49.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:49.923 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:49.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:49.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:49.957 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:38:49.957 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:38:49.973 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:38:49.973 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:38:50.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:50.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:50.052 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:50.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:50.075 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:38:50.075 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:38:50.075 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:38:50.075 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:38:50.153 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:38:50.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:49 smithi121 ceph-mon[30729]: pgmap v322: 17 pgs: 17 active+clean; 122 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 3.7 MiB/s wr, 30 op/s 2024-09-11T03:38:50.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:49 smithi121 ceph-mon[30729]: osdmap e112: 12 total, 12 up, 12 in 2024-09-11T03:38:50.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/743461033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:38:50.175 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:50.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:50.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:38:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:49 smithi175 ceph-mon[30894]: pgmap v322: 17 pgs: 17 active+clean; 122 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 3.7 MiB/s wr, 30 op/s 2024-09-11T03:38:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:49 smithi175 ceph-mon[30894]: osdmap e112: 12 total, 12 up, 12 in 2024-09-11T03:38:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/743461033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:38:50.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:50.302 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:50.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:50.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:50.404 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:50.427 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:50.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:50.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:50.529 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:50.551 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:50.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:50.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:50.653 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:50.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:50.675 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:50.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:50.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:50.795 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T03:38:50.803 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:50.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:50.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:50.831 INFO:tasks.rados.rados.2.smithi175.stdout:115: done (1 left) 2024-09-11T03:38:50.831 INFO:tasks.rados.rados.2.smithi175.stdout:116: copy_from oid 8 from oid 30 current snap is 7 2024-09-11T03:38:50.831 INFO:tasks.rados.rados.2.smithi175.stdout:113: finishing write tid 2 to smithi17557870-7 2024-09-11T03:38:50.831 INFO:tasks.rados.rados.2.smithi175.stdout:113: finishing write tid 3 to smithi17557870-7 2024-09-11T03:38:50.831 INFO:tasks.rados.rados.2.smithi175.stdout:113: finishing write tid 4 to smithi17557870-7 2024-09-11T03:38:50.831 INFO:tasks.rados.rados.2.smithi175.stdout:113: finishing write tid 5 to smithi17557870-7 2024-09-11T03:38:50.831 INFO:tasks.rados.rados.2.smithi175.stdout:113: oid 7 updating version 0 to 11 2024-09-11T03:38:50.831 INFO:tasks.rados.rados.2.smithi175.stdout:113: oid 7 version 11 is already newer than 10 2024-09-11T03:38:50.831 INFO:tasks.rados.rados.2.smithi175.stdout:113: oid 7 version 11 is already newer than 8 2024-09-11T03:38:50.832 INFO:tasks.rados.rados.2.smithi175.stdout:113: oid 7 version 11 is already newer than 9 2024-09-11T03:38:50.832 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 11 (ObjNum 69 snap 7 seq_num 69) dirty exists 2024-09-11T03:38:50.832 INFO:tasks.rados.rados.2.smithi175.stdout:113: left oid 7 (ObjNum 69 snap 7 seq_num 69) 2024-09-11T03:38:50.832 INFO:tasks.rados.rados.2.smithi175.stdout:113: done (1 left) 2024-09-11T03:38:50.832 INFO:tasks.rados.rados.2.smithi175.stdout:117: write oid 13 current snap is 7 2024-09-11T03:38:50.832 INFO:tasks.rados.rados.2.smithi175.stdout:117: seq_num 71 ranges {793297=789856,2305045=723014} 2024-09-11T03:38:50.839 INFO:tasks.rados.rados.2.smithi175.stdout:117: writing smithi17557870-13 from 793297 to 1583153 tid 1 2024-09-11T03:38:50.848 INFO:tasks.rados.rados.2.smithi175.stdout:117: writing smithi17557870-13 from 2305045 to 3028059 tid 2 2024-09-11T03:38:50.848 INFO:tasks.rados.rados.2.smithi175.stdout:118: read oid 49 snap -1 2024-09-11T03:38:50.849 INFO:tasks.rados.rados.2.smithi175.stdout:118: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-11T03:38:50.849 INFO:tasks.rados.rados.2.smithi175.stdout:119: write (excl) oid 9 current snap is 7 2024-09-11T03:38:50.849 INFO:tasks.rados.rados.2.smithi175.stdout:116: finishing copy_from racing read to smithi17557870-8 2024-09-11T03:38:50.849 INFO:tasks.rados.rados.2.smithi175.stdout:119: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2024-09-11T03:38:50.853 INFO:tasks.rados.rados.2.smithi175.stdout:119: writing smithi17557870-9 from 410104 to 875209 tid 1 2024-09-11T03:38:50.860 INFO:tasks.rados.rados.2.smithi175.stdout:119: writing smithi17557870-9 from 1563527 to 2146552 tid 2 2024-09-11T03:38:50.866 INFO:tasks.rados.rados.2.smithi175.stdout:119: writing smithi17557870-9 from 2810926 to 3210104 tid 3 2024-09-11T03:38:50.866 INFO:tasks.rados.rados.2.smithi175.stdout:120: write (excl) oid 48 current snap is 7 2024-09-11T03:38:50.866 INFO:tasks.rados.rados.2.smithi175.stdout:116: finishing copy_from to smithi17557870-8 2024-09-11T03:38:50.866 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-11T03:38:50.867 INFO:tasks.rados.rados.2.smithi175.stdout:117: finishing write tid 1 to smithi17557870-13 2024-09-11T03:38:50.867 INFO:tasks.rados.rados.2.smithi175.stdout:120: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-09-11T03:38:50.871 INFO:tasks.rados.rados.2.smithi175.stdout:120: writing smithi17557870-48 from 426911 to 967265 tid 1 2024-09-11T03:38:50.878 INFO:tasks.rados.rados.2.smithi175.stdout:120: writing smithi17557870-48 from 1622009 to 2348692 tid 2 2024-09-11T03:38:50.884 INFO:tasks.rados.rados.2.smithi175.stdout:120: writing smithi17557870-48 from 3121996 to 3226911 tid 3 2024-09-11T03:38:50.885 INFO:tasks.rados.rados.2.smithi175.stdout:116: done (4 left) 2024-09-11T03:38:50.885 INFO:tasks.rados.rados.2.smithi175.stdout:121: read oid 12 snap -1 2024-09-11T03:38:50.885 INFO:tasks.rados.rados.2.smithi175.stdout:121: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-11T03:38:50.885 INFO:tasks.rados.rados.2.smithi175.stdout:122: rollback oid 1 current snap is 7 2024-09-11T03:38:50.885 INFO:tasks.rados.rados.2.smithi175.stdout:122: done (5 left) 2024-09-11T03:38:50.885 INFO:tasks.rados.rados.2.smithi175.stdout:123: delete oid 39 current snap is 7 2024-09-11T03:38:50.885 INFO:tasks.rados.rados.2.smithi175.stdout:117: finishing write tid 2 to smithi17557870-13 2024-09-11T03:38:50.886 INFO:tasks.rados.rados.2.smithi175.stdout:117: finishing write tid 3 to smithi17557870-13 2024-09-11T03:38:50.886 INFO:tasks.rados.rados.2.smithi175.stdout:117: finishing write tid 4 to smithi17557870-13 2024-09-11T03:38:50.886 INFO:tasks.rados.rados.2.smithi175.stdout:117: oid 13 updating version 0 to 42 2024-09-11T03:38:50.886 INFO:tasks.rados.rados.2.smithi175.stdout:117: oid 13 version 42 is already newer than 40 2024-09-11T03:38:50.886 INFO:tasks.rados.rados.2.smithi175.stdout:117: oid 13 version 42 is already newer than 41 2024-09-11T03:38:50.886 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 42 (ObjNum 71 snap 7 seq_num 71) dirty exists 2024-09-11T03:38:50.886 INFO:tasks.rados.rados.2.smithi175.stdout:117: left oid 13 (ObjNum 71 snap 7 seq_num 71) 2024-09-11T03:38:50.886 INFO:tasks.rados.rados.2.smithi175.stdout:118: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-11T03:38:50.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:50 smithi084 ceph-mon[27254]: osdmap e113: 12 total, 12 up, 12 in 2024-09-11T03:38:50.905 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:50.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:50.928 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:50.928 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:50.977 INFO:tasks.rados.rados.2.smithi175.stdout:119: finishing write tid 1 to smithi17557870-9 2024-09-11T03:38:50.977 INFO:tasks.rados.rados.2.smithi175.stdout:119: finishing write tid 2 to smithi17557870-9 2024-09-11T03:38:50.977 INFO:tasks.rados.rados.2.smithi175.stdout:119: finishing write tid 3 to smithi17557870-9 2024-09-11T03:38:50.978 INFO:tasks.rados.rados.2.smithi175.stdout:117: done (5 left) 2024-09-11T03:38:50.978 INFO:tasks.rados.rados.2.smithi175.stdout:118: done (4 left) 2024-09-11T03:38:50.978 INFO:tasks.rados.rados.2.smithi175.stdout:123: done (3 left) 2024-09-11T03:38:50.978 INFO:tasks.rados.rados.2.smithi175.stdout:124: read oid 45 snap -1 2024-09-11T03:38:50.978 INFO:tasks.rados.rados.2.smithi175.stdout:124: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-11T03:38:50.978 INFO:tasks.rados.rados.2.smithi175.stdout:125: read oid 30 snap -1 2024-09-11T03:38:50.978 INFO:tasks.rados.rados.2.smithi175.stdout:125: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-11T03:38:50.978 INFO:tasks.rados.rados.2.smithi175.stdout:126: write (excl) oid 25 current snap is 7 2024-09-11T03:38:50.978 INFO:tasks.rados.rados.2.smithi175.stdout:119: finishing write tid 4 to smithi17557870-9 2024-09-11T03:38:50.978 INFO:tasks.rados.rados.2.smithi175.stdout:126: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2024-09-11T03:38:50.985 INFO:tasks.rados.rados.2.smithi175.stdout:126: writing smithi17557870-25 from 460525 to 1151377 tid 1 2024-09-11T03:38:50.991 INFO:tasks.rados.rados.2.smithi175.stdout:126: writing smithi17557870-25 from 1622620 to 2352972 tid 2 2024-09-11T03:38:50.997 INFO:tasks.rados.rados.2.smithi175.stdout:126: writing smithi17557870-25 from 3060489 to 3260525 tid 3 2024-09-11T03:38:50.998 INFO:tasks.rados.rados.2.smithi175.stdout:127: read oid 6 snap -1 2024-09-11T03:38:50.998 INFO:tasks.rados.rados.2.smithi175.stdout:127: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-11T03:38:50.998 INFO:tasks.rados.rados.2.smithi175.stdout:119: finishing write tid 5 to smithi17557870-9 2024-09-11T03:38:50.998 INFO:tasks.rados.rados.2.smithi175.stdout:119: oid 9 updating version 0 to 24 2024-09-11T03:38:50.998 INFO:tasks.rados.rados.2.smithi175.stdout:119: oid 9 updating version 24 to 26 2024-09-11T03:38:50.998 INFO:tasks.rados.rados.2.smithi175.stdout:119: oid 9 version 26 is already newer than 25 2024-09-11T03:38:50.998 INFO:tasks.rados.rados.2.smithi175.stdout:119: oid 9 updating version 26 to 27 2024-09-11T03:38:50.998 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 27 (ObjNum 72 snap 7 seq_num 72) dirty exists 2024-09-11T03:38:50.998 INFO:tasks.rados.rados.2.smithi175.stdout:119: left oid 9 (ObjNum 72 snap 7 seq_num 72) 2024-09-11T03:38:50.998 INFO:tasks.rados.rados.2.smithi175.stdout:120: finishing write tid 1 to smithi17557870-48 2024-09-11T03:38:50.998 INFO:tasks.rados.rados.2.smithi175.stdout:120: finishing write tid 2 to smithi17557870-48 2024-09-11T03:38:50.998 INFO:tasks.rados.rados.2.smithi175.stdout:119: done (6 left) 2024-09-11T03:38:50.998 INFO:tasks.rados.rados.2.smithi175.stdout:128: rollback oid 8 current snap is 7 2024-09-11T03:38:50.999 INFO:tasks.rados.rados.2.smithi175.stdout:128: done (6 left) 2024-09-11T03:38:50.999 INFO:tasks.rados.rados.2.smithi175.stdout:129: write (excl) oid 13 current snap is 7 2024-09-11T03:38:50.999 INFO:tasks.rados.rados.2.smithi175.stdout:120: finishing write tid 3 to smithi17557870-48 2024-09-11T03:38:50.999 INFO:tasks.rados.rados.2.smithi175.stdout:121: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-11T03:38:51.008 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:51.029 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:51.053 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:51.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:51.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:51.066 INFO:tasks.rados.rados.2.smithi175.stdout:120: finishing write tid 4 to smithi17557870-48 2024-09-11T03:38:51.066 INFO:tasks.rados.rados.2.smithi175.stdout:120: finishing write tid 5 to smithi17557870-48 2024-09-11T03:38:51.066 INFO:tasks.rados.rados.2.smithi175.stdout:120: oid 48 updating version 0 to 15 2024-09-11T03:38:51.066 INFO:tasks.rados.rados.2.smithi175.stdout:120: oid 48 updating version 15 to 16 2024-09-11T03:38:51.066 INFO:tasks.rados.rados.2.smithi175.stdout:120: oid 48 updating version 16 to 17 2024-09-11T03:38:51.066 INFO:tasks.rados.rados.2.smithi175.stdout:120: oid 48 updating version 17 to 18 2024-09-11T03:38:51.066 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 18 (ObjNum 73 snap 7 seq_num 73) dirty exists 2024-09-11T03:38:51.066 INFO:tasks.rados.rados.2.smithi175.stdout:120: left oid 48 (ObjNum 73 snap 7 seq_num 73) 2024-09-11T03:38:51.066 INFO:tasks.rados.rados.2.smithi175.stdout:129: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2024-09-11T03:38:51.072 INFO:tasks.rados.rados.2.smithi175.stdout:129: writing smithi17557870-13 from 477332 to 1243433 tid 1 2024-09-11T03:38:51.077 INFO:tasks.rados.rados.2.smithi175.stdout:129: writing smithi17557870-13 from 1681102 to 2155112 tid 2 2024-09-11T03:38:51.084 INFO:tasks.rados.rados.2.smithi175.stdout:129: writing smithi17557870-13 from 2687912 to 3277332 tid 3 2024-09-11T03:38:51.085 INFO:tasks.rados.rados.2.smithi175.stdout:120: done (6 left) 2024-09-11T03:38:51.085 INFO:tasks.rados.rados.2.smithi175.stdout:121: done (5 left) 2024-09-11T03:38:51.085 INFO:tasks.rados.rados.2.smithi175.stdout:130: read oid 3 snap -1 2024-09-11T03:38:51.085 INFO:tasks.rados.rados.2.smithi175.stdout:130: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-11T03:38:51.085 INFO:tasks.rados.rados.2.smithi175.stdout:131: copy_from oid 21 from oid 44 current snap is 7 2024-09-11T03:38:51.085 INFO:tasks.rados.rados.2.smithi175.stdout:124: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-11T03:38:51.156 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:51.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:50 smithi121 ceph-mon[30729]: osdmap e113: 12 total, 12 up, 12 in 2024-09-11T03:38:51.172 INFO:tasks.rados.rados.2.smithi175.stdout:125: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-11T03:38:51.177 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:51.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:50 smithi175 ceph-mon[30894]: osdmap e113: 12 total, 12 up, 12 in 2024-09-11T03:38:51.258 INFO:tasks.rados.rados.2.smithi175.stdout:126: finishing write tid 1 to smithi17557870-25 2024-09-11T03:38:51.258 INFO:tasks.rados.rados.2.smithi175.stdout:126: finishing write tid 2 to smithi17557870-25 2024-09-11T03:38:51.258 INFO:tasks.rados.rados.2.smithi175.stdout:126: finishing write tid 3 to smithi17557870-25 2024-09-11T03:38:51.258 INFO:tasks.rados.rados.2.smithi175.stdout:126: finishing write tid 4 to smithi17557870-25 2024-09-11T03:38:51.258 INFO:tasks.rados.rados.2.smithi175.stdout:126: finishing write tid 5 to smithi17557870-25 2024-09-11T03:38:51.258 INFO:tasks.rados.rados.2.smithi175.stdout:126: oid 25 updating version 0 to 14 2024-09-11T03:38:51.258 INFO:tasks.rados.rados.2.smithi175.stdout:126: oid 25 version 14 is already newer than 12 2024-09-11T03:38:51.259 INFO:tasks.rados.rados.2.smithi175.stdout:126: oid 25 version 14 is already newer than 11 2024-09-11T03:38:51.259 INFO:tasks.rados.rados.2.smithi175.stdout:126: oid 25 version 14 is already newer than 13 2024-09-11T03:38:51.259 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 14 (ObjNum 75 snap 7 seq_num 75) dirty exists 2024-09-11T03:38:51.259 INFO:tasks.rados.rados.2.smithi175.stdout:126: left oid 25 (ObjNum 75 snap 7 seq_num 75) 2024-09-11T03:38:51.259 INFO:tasks.rados.rados.2.smithi175.stdout:127: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-11T03:38:51.278 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:51.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:51.300 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:51.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:51.327 INFO:tasks.rados.rados.2.smithi175.stdout:129: finishing write tid 1 to smithi17557870-13 2024-09-11T03:38:51.327 INFO:tasks.rados.rados.2.smithi175.stdout:129: finishing write tid 2 to smithi17557870-13 2024-09-11T03:38:51.327 INFO:tasks.rados.rados.2.smithi175.stdout:124: done (6 left) 2024-09-11T03:38:51.327 INFO:tasks.rados.rados.2.smithi175.stdout:125: done (5 left) 2024-09-11T03:38:51.327 INFO:tasks.rados.rados.2.smithi175.stdout:126: done (4 left) 2024-09-11T03:38:51.327 INFO:tasks.rados.rados.2.smithi175.stdout:127: done (3 left) 2024-09-11T03:38:51.327 INFO:tasks.rados.rados.2.smithi175.stdout:132: rollback oid 9 current snap is 7 2024-09-11T03:38:51.327 INFO:tasks.rados.rados.2.smithi175.stdout:132: done (3 left) 2024-09-11T03:38:51.327 INFO:tasks.rados.rados.2.smithi175.stdout:133: rollback oid 6 current snap is 7 2024-09-11T03:38:51.328 INFO:tasks.rados.rados.2.smithi175.stdout:133: done (3 left) 2024-09-11T03:38:51.328 INFO:tasks.rados.rados.2.smithi175.stdout:134: snap_create 2024-09-11T03:38:51.328 INFO:tasks.rados.rados.2.smithi175.stdout:129: finishing write tid 3 to smithi17557870-13 2024-09-11T03:38:51.328 INFO:tasks.rados.rados.2.smithi175.stdout:129: finishing write tid 4 to smithi17557870-13 2024-09-11T03:38:51.328 INFO:tasks.rados.rados.2.smithi175.stdout:129: finishing write tid 5 to smithi17557870-13 2024-09-11T03:38:51.328 INFO:tasks.rados.rados.2.smithi175.stdout:129: oid 13 updating version 0 to 43 2024-09-11T03:38:51.328 INFO:tasks.rados.rados.2.smithi175.stdout:129: oid 13 updating version 43 to 44 2024-09-11T03:38:51.328 INFO:tasks.rados.rados.2.smithi175.stdout:129: oid 13 updating version 44 to 46 2024-09-11T03:38:51.328 INFO:tasks.rados.rados.2.smithi175.stdout:129: oid 13 version 46 is already newer than 45 2024-09-11T03:38:51.328 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 46 (ObjNum 76 snap 7 seq_num 76) dirty exists 2024-09-11T03:38:51.328 INFO:tasks.rados.rados.2.smithi175.stdout:129: left oid 13 (ObjNum 76 snap 7 seq_num 76) 2024-09-11T03:38:51.328 INFO:tasks.rados.rados.2.smithi175.stdout:130: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-11T03:38:51.397 INFO:tasks.rados.rados.2.smithi175.stdout:131: finishing copy_from racing read to smithi17557870-21 2024-09-11T03:38:51.397 INFO:tasks.rados.rados.2.smithi175.stdout:131: finishing copy_from to smithi17557870-21 2024-09-11T03:38:51.397 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 9 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-11T03:38:51.401 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:51.422 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:51.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:51.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:51.468 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:51.524 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:38:51.546 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:51.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:51.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:38:51.648 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:51.670 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:51.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:51.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:51.772 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:51.793 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:51.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:51.794 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:51.812 INFO:tasks.rados.rados.2.smithi175.stdout:129: done (3 left) 2024-09-11T03:38:51.813 INFO:tasks.rados.rados.2.smithi175.stdout:130: done (2 left) 2024-09-11T03:38:51.813 INFO:tasks.rados.rados.2.smithi175.stdout:131: done (1 left) 2024-09-11T03:38:51.813 INFO:tasks.rados.rados.2.smithi175.stdout:134: done (0 left) 2024-09-11T03:38:51.813 INFO:tasks.rados.rados.2.smithi175.stdout:135: delete oid 27 current snap is 8 2024-09-11T03:38:51.815 INFO:tasks.rados.rados.2.smithi175.stdout:135: done (0 left) 2024-09-11T03:38:51.816 INFO:tasks.rados.rados.2.smithi175.stdout:136: snap_create 2024-09-11T03:38:51.895 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:38:51.904 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:51.917 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:51.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:51.918 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:38:52.008 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:51 smithi084 ceph-mon[27254]: pgmap v325: 17 pgs: 17 active+clean; 122 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2024-09-11T03:38:52.008 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:51 smithi084 ceph-mon[27254]: osdmap e114: 12 total, 12 up, 12 in 2024-09-11T03:38:52.008 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:51 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:38:52.009 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:51 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:38:52.009 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:51 smithi084 ceph-mon[27254]: osdmap e115: 12 total, 12 up, 12 in 2024-09-11T03:38:52.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:52.045 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:52.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:52.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:52.147 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:38:52.169 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:52.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:52.169 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:38:52.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:51 smithi121 ceph-mon[30729]: pgmap v325: 17 pgs: 17 active+clean; 122 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2024-09-11T03:38:52.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:51 smithi121 ceph-mon[30729]: osdmap e114: 12 total, 12 up, 12 in 2024-09-11T03:38:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:51 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:38:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:51 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:38:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:51 smithi121 ceph-mon[30729]: osdmap e115: 12 total, 12 up, 12 in 2024-09-11T03:38:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:51 smithi175 ceph-mon[30894]: pgmap v325: 17 pgs: 17 active+clean; 122 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2024-09-11T03:38:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:51 smithi175 ceph-mon[30894]: osdmap e114: 12 total, 12 up, 12 in 2024-09-11T03:38:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:51 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:38:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:51 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:38:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:51 smithi175 ceph-mon[30894]: osdmap e115: 12 total, 12 up, 12 in 2024-09-11T03:38:52.271 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:52.292 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:52.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:52.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:52.394 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:52.415 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:52.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:52.416 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:52.517 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:52.538 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:52.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:52.539 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:52.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:52.666 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:52.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:52.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:52.758 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:38:52.758 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:38:52.758 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:38:52.758 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:38:52.758 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:38:52.768 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:52.789 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:52.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:52.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:52.809 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T03:38:52.809 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:38:52.813 INFO:tasks.rados.rados.2.smithi175.stdout:136: done (0 left) 2024-09-11T03:38:52.813 INFO:tasks.rados.rados.2.smithi175.stdout:137: read oid 11 snap 8 2024-09-11T03:38:52.813 INFO:tasks.rados.rados.2.smithi175.stdout:137: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-11T03:38:52.813 INFO:tasks.rados.rados.2.smithi175.stdout:138: snap_remove snap 7 2024-09-11T03:38:52.891 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:38:52.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:52 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3021801818' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:38:52.897 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:52 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3620578407' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:38:52.897 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:52 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:38:52.897 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:52 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:38:52.897 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:52 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3021801818' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:38:52.897 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:52 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3620578407' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:38:52.897 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:52 smithi084 ceph-mon[27254]: osdmap e116: 12 total, 12 up, 12 in 2024-09-11T03:38:52.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:52.913 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:52.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:38:53.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:53.035 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:53.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:53.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:53.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:53.163 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:53.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:53.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:52 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3021801818' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:38:53.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:52 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3620578407' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:38:53.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:52 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:38:53.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:52 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:38:53.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:52 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3021801818' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:38:53.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:52 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3620578407' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:38:53.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:52 smithi121 ceph-mon[30729]: osdmap e116: 12 total, 12 up, 12 in 2024-09-11T03:38:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:52 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3021801818' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:38:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:52 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3620578407' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:38:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:52 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:38:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:52 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:38:53.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:52 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3021801818' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:38:53.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:52 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3620578407' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:38:53.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:52 smithi175 ceph-mon[30894]: osdmap e116: 12 total, 12 up, 12 in 2024-09-11T03:38:53.265 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:53.287 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:53.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:53.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:53.389 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:53.412 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:53.413 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:53.415 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:38:53.513 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:53.535 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:53.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:53.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:53.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:53.663 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:53.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:53.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:53.765 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:53.788 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:53.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:53.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:53.814 INFO:tasks.rados.rados.2.smithi175.stdout:138: done (1 left) 2024-09-11T03:38:53.814 INFO:tasks.rados.rados.2.smithi175.stdout:139: write (excl) oid 34 current snap is 9 2024-09-11T03:38:53.815 INFO:tasks.rados.rados.2.smithi175.stdout:139: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2024-09-11T03:38:53.820 INFO:tasks.rados.rados.2.smithi175.stdout:139: writing smithi17557870-34 from 510946 to 1143898 tid 1 2024-09-11T03:38:53.826 INFO:tasks.rados.rados.2.smithi175.stdout:139: writing smithi17557870-34 from 1798066 to 2275745 tid 2 2024-09-11T03:38:53.832 INFO:tasks.rados.rados.2.smithi175.stdout:139: writing smithi17557870-34 from 2742758 to 3199975 tid 3 2024-09-11T03:38:53.832 INFO:tasks.rados.rados.2.smithi175.stdout:140: read oid 33 snap -1 2024-09-11T03:38:53.832 INFO:tasks.rados.rados.2.smithi175.stdout:140: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-11T03:38:53.832 INFO:tasks.rados.rados.2.smithi175.stdout:141: delete oid 19 current snap is 9 2024-09-11T03:38:53.832 INFO:tasks.rados.rados.2.smithi175.stdout:137: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-11T03:38:53.890 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:53.898 INFO:tasks.rados.rados.2.smithi175.stdout:139: finishing write tid 1 to smithi17557870-34 2024-09-11T03:38:53.898 INFO:tasks.rados.rados.2.smithi175.stdout:139: finishing write tid 2 to smithi17557870-34 2024-09-11T03:38:53.898 INFO:tasks.rados.rados.2.smithi175.stdout:139: finishing write tid 3 to smithi17557870-34 2024-09-11T03:38:53.898 INFO:tasks.rados.rados.2.smithi175.stdout:139: finishing write tid 4 to smithi17557870-34 2024-09-11T03:38:53.898 INFO:tasks.rados.rados.2.smithi175.stdout:139: finishing write tid 5 to smithi17557870-34 2024-09-11T03:38:53.898 INFO:tasks.rados.rados.2.smithi175.stdout:139: oid 34 updating version 0 to 14 2024-09-11T03:38:53.898 INFO:tasks.rados.rados.2.smithi175.stdout:139: oid 34 version 14 is already newer than 13 2024-09-11T03:38:53.898 INFO:tasks.rados.rados.2.smithi175.stdout:139: oid 34 version 14 is already newer than 11 2024-09-11T03:38:53.898 INFO:tasks.rados.rados.2.smithi175.stdout:139: oid 34 version 14 is already newer than 12 2024-09-11T03:38:53.898 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 14 (ObjNum 78 snap 9 seq_num 78) dirty exists 2024-09-11T03:38:53.898 INFO:tasks.rados.rados.2.smithi175.stdout:139: left oid 34 (ObjNum 78 snap 9 seq_num 78) 2024-09-11T03:38:53.899 INFO:tasks.rados.rados.2.smithi175.stdout:140: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-11T03:38:53.911 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:53.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:53.912 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:53.981 INFO:tasks.rados.rados.2.smithi175.stdout:137: done (3 left) 2024-09-11T03:38:53.982 INFO:tasks.rados.rados.2.smithi175.stdout:139: done (2 left) 2024-09-11T03:38:53.982 INFO:tasks.rados.rados.2.smithi175.stdout:140: done (1 left) 2024-09-11T03:38:53.982 INFO:tasks.rados.rados.2.smithi175.stdout:141: done (0 left) 2024-09-11T03:38:53.982 INFO:tasks.rados.rados.2.smithi175.stdout:142: snap_remove snap 8 2024-09-11T03:38:54.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:54.033 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:54.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:54.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:54.135 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:53 smithi084 ceph-mon[27254]: pgmap v328: 17 pgs: 17 active+clean; 112 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.2 MiB/s wr, 32 op/s 2024-09-11T03:38:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:53 smithi084 ceph-mon[27254]: osdmap e117: 12 total, 12 up, 12 in 2024-09-11T03:38:54.157 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:54.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:54.158 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:54.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:53 smithi121 ceph-mon[30729]: pgmap v328: 17 pgs: 17 active+clean; 112 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.2 MiB/s wr, 32 op/s 2024-09-11T03:38:54.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:53 smithi121 ceph-mon[30729]: osdmap e117: 12 total, 12 up, 12 in 2024-09-11T03:38:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:53 smithi175 ceph-mon[30894]: pgmap v328: 17 pgs: 17 active+clean; 112 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.2 MiB/s wr, 32 op/s 2024-09-11T03:38:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:53 smithi175 ceph-mon[30894]: osdmap e117: 12 total, 12 up, 12 in 2024-09-11T03:38:54.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:54.284 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:54.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:54.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:54.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:54.410 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:54.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:54.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:54.458 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T03:38:54.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:54.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:54.537 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:54.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:54.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:54.665 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:54.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:54.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:54.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:54.793 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:54.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:54.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:54.843 INFO:tasks.rados.rados.2.smithi175.stdout:142: done (0 left) 2024-09-11T03:38:54.843 INFO:tasks.rados.rados.2.smithi175.stdout:143: read oid 28 snap -1 2024-09-11T03:38:54.843 INFO:tasks.rados.rados.2.smithi175.stdout:143: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-11T03:38:54.843 INFO:tasks.rados.rados.2.smithi175.stdout:144: snap_create 2024-09-11T03:38:54.854 INFO:tasks.rados.rados.2.smithi175.stdout:143: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-11T03:38:54.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-11T03:38:54.889 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:54.895 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:54.916 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:54.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:54.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:55.018 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:38:55.040 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:55.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:55.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:38:55.142 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:55.168 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:55.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:55.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:55.270 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:55.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:55.293 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:55.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:55.391 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:55.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:55.407 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:38:55.407 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:38:55.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:38:55.408 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:38:55.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.408 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:23 1.0 e20/19 1:30306672:devicehealth::main.db.0000000000000000:head v 20'13, mlcod=20'8)", 2024-09-11T03:38:55.408 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.138702+0000", 2024-09-11T03:38:55.408 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 311.26788605399997, 2024-09-11T03:38:55.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012651940000000001, 2024-09-11T03:38:55.408 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.409 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.409 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:55.409 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138702+0000", 2024-09-11T03:38:55.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138702+0000", 2024-09-11T03:38:55.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138702+0000", 2024-09-11T03:38:55.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8599999999999998e-07 2024-09-11T03:38:55.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138755+0000", 2024-09-11T03:38:55.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2907e-05 2024-09-11T03:38:55.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138775+0000", 2024-09-11T03:38:55.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9777000000000001e-05 2024-09-11T03:38:55.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138781+0000", 2024-09-11T03:38:55.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3380000000000001e-06 2024-09-11T03:38:55.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138816+0000", 2024-09-11T03:38:55.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4581e-05 2024-09-11T03:38:55.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.138833+0000", 2024-09-11T03:38:55.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7261e-05 2024-09-11T03:38:55.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:55.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.139931+0000", 2024-09-11T03:38:55.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001098056 2024-09-11T03:38:55.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.139967+0000", 2024-09-11T03:38:55.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5587999999999999e-05 2024-09-11T03:38:55.413 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.413 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.413 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.413 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:43 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'25, mlcod=20'22)", 2024-09-11T03:38:55.413 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.202535+0000", 2024-09-11T03:38:55.413 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 311.204052359, 2024-09-11T03:38:55.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034849519999999999, 2024-09-11T03:38:55.414 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.414 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.414 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:55.414 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202535+0000", 2024-09-11T03:38:55.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202535+0000", 2024-09-11T03:38:55.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202536+0000", 2024-09-11T03:38:55.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3099999999999997e-07 2024-09-11T03:38:55.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202558+0000", 2024-09-11T03:38:55.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2045999999999999e-05 2024-09-11T03:38:55.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202570+0000", 2024-09-11T03:38:55.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1909999999999999e-05 2024-09-11T03:38:55.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202576+0000", 2024-09-11T03:38:55.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5100000000000004e-06 2024-09-11T03:38:55.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202620+0000", 2024-09-11T03:38:55.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3381999999999999e-05 2024-09-11T03:38:55.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202632+0000", 2024-09-11T03:38:55.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2047e-05 2024-09-11T03:38:55.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:55.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.205973+0000", 2024-09-11T03:38:55.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033406640000000001 2024-09-11T03:38:55.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206020+0000", 2024-09-11T03:38:55.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7762000000000001e-05 2024-09-11T03:38:55.419 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.419 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.419 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.420 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:46 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'28, mlcod=20'23)", 2024-09-11T03:38:55.420 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.206155+0000", 2024-09-11T03:38:55.420 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 311.20043306000002, 2024-09-11T03:38:55.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026709669999999998, 2024-09-11T03:38:55.420 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.420 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.420 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:55.420 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206155+0000", 2024-09-11T03:38:55.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206155+0000", 2024-09-11T03:38:55.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206155+0000", 2024-09-11T03:38:55.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5899999999999996e-07 2024-09-11T03:38:55.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206160+0000", 2024-09-11T03:38:55.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8620000000000002e-06 2024-09-11T03:38:55.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206169+0000", 2024-09-11T03:38:55.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5380000000000002e-06 2024-09-11T03:38:55.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206183+0000", 2024-09-11T03:38:55.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4552999999999999e-05 2024-09-11T03:38:55.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206226+0000", 2024-09-11T03:38:55.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3260999999999998e-05 2024-09-11T03:38:55.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206246+0000", 2024-09-11T03:38:55.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9989000000000001e-05 2024-09-11T03:38:55.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:55.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.208789+0000", 2024-09-11T03:38:55.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025422180000000002 2024-09-11T03:38:55.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.208826+0000", 2024-09-11T03:38:55.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6986999999999997e-05 2024-09-11T03:38:55.424 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.424 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.424 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.425 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:49 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'31, mlcod=20'27)", 2024-09-11T03:38:55.425 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.210171+0000", 2024-09-11T03:38:55.425 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 311.196416441, 2024-09-11T03:38:55.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028043399999999998, 2024-09-11T03:38:55.426 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.427 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.427 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:55.427 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210171+0000", 2024-09-11T03:38:55.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210171+0000", 2024-09-11T03:38:55.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210172+0000", 2024-09-11T03:38:55.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4799999999999998e-07 2024-09-11T03:38:55.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210177+0000", 2024-09-11T03:38:55.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8520000000000003e-06 2024-09-11T03:38:55.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210187+0000", 2024-09-11T03:38:55.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0448999999999999e-05 2024-09-11T03:38:55.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210193+0000", 2024-09-11T03:38:55.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.409e-06 2024-09-11T03:38:55.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210245+0000", 2024-09-11T03:38:55.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1280999999999999e-05 2024-09-11T03:38:55.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210263+0000", 2024-09-11T03:38:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8173e-05 2024-09-11T03:38:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212939+0000", 2024-09-11T03:38:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026764559999999998 2024-09-11T03:38:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212976+0000", 2024-09-11T03:38:55.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6171999999999999e-05 2024-09-11T03:38:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:221 2.3 e103/95 2:c6c49f2c:::smithi17557870-44:head v 103'8, mlcod=100'5)", 2024-09-11T03:38:55.433 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:39.951509+0000", 2024-09-11T03:38:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 15.455078492, 2024-09-11T03:38:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046184859999999998, 2024-09-11T03:38:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.951509+0000", 2024-09-11T03:38:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.951509+0000", 2024-09-11T03:38:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.951509+0000", 2024-09-11T03:38:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5599999999999999e-07 2024-09-11T03:38:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.954344+0000", 2024-09-11T03:38:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028342129999999999 2024-09-11T03:38:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.954357+0000", 2024-09-11T03:38:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3072e-05 2024-09-11T03:38:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.954364+0000", 2024-09-11T03:38:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5000000000000002e-06 2024-09-11T03:38:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.954439+0000", 2024-09-11T03:38:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4949000000000004e-05 2024-09-11T03:38:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.954457+0000", 2024-09-11T03:38:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7445999999999999e-05 2024-09-11T03:38:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:55.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.956091+0000", 2024-09-11T03:38:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016345629999999999 2024-09-11T03:38:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.956128+0000", 2024-09-11T03:38:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6587000000000001e-05 2024-09-11T03:38:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:223 2.3 e103/95 2:c3768452:::smithi17557870-11:head v 103'10, mlcod=100'5)", 2024-09-11T03:38:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:39.955056+0000", 2024-09-11T03:38:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 15.451531852, 2024-09-11T03:38:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030991349999999998, 2024-09-11T03:38:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.955056+0000", 2024-09-11T03:38:55.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.955056+0000", 2024-09-11T03:38:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.955057+0000", 2024-09-11T03:38:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9599999999999998e-07 2024-09-11T03:38:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.957003+0000", 2024-09-11T03:38:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019464300000000001 2024-09-11T03:38:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.957012+0000", 2024-09-11T03:38:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0450000000000006e-06 2024-09-11T03:38:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.957019+0000", 2024-09-11T03:38:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7089999999999996e-06 2024-09-11T03:38:55.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.957089+0000", 2024-09-11T03:38:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0636000000000006e-05 2024-09-11T03:38:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.957104+0000", 2024-09-11T03:38:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4854e-05 2024-09-11T03:38:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.958127+0000", 2024-09-11T03:38:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010224159999999999 2024-09-11T03:38:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:39.958155+0000", 2024-09-11T03:38:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8249e-05 2024-09-11T03:38:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:233 2.e 2:7efbbea8:::smithi17557870-24:head [append 0~0] snapc 5=[5,4] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-11T03:38:55.443 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:42.965735+0000", 2024-09-11T03:38:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 12.440852314000001, 2024-09-11T03:38:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003713315, 2024-09-11T03:38:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:38:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:38:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:38:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 233 2024-09-11T03:38:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965735+0000", 2024-09-11T03:38:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965735+0000", 2024-09-11T03:38:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965737+0000", 2024-09-11T03:38:55.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8989999999999999e-06 2024-09-11T03:38:55.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965756+0000", 2024-09-11T03:38:55.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9089e-05 2024-09-11T03:38:55.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965788+0000", 2024-09-11T03:38:55.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1173000000000001e-05 2024-09-11T03:38:55.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965797+0000", 2024-09-11T03:38:55.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7540000000000001e-06 2024-09-11T03:38:55.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965869+0000", 2024-09-11T03:38:55.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2150000000000005e-05 2024-09-11T03:38:55.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:38:55.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965877+0000", 2024-09-11T03:38:55.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8220000000000007e-06 2024-09-11T03:38:55.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.967254+0000", 2024-09-11T03:38:55.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001376298 2024-09-11T03:38:55.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.967354+0000", 2024-09-11T03:38:55.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010094699999999999 2024-09-11T03:38:55.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:38:55.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.967570+0000", 2024-09-11T03:38:55.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021571200000000001 2024-09-11T03:38:55.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:38:55.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.969274+0000", 2024-09-11T03:38:55.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001703687 2024-09-11T03:38:55.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:38:55.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.969324+0000", 2024-09-11T03:38:55.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0259000000000001e-05 2024-09-11T03:38:55.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:38:55.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.969378+0000", 2024-09-11T03:38:55.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3733000000000002e-05 2024-09-11T03:38:55.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:55.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.969407+0000", 2024-09-11T03:38:55.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9133e-05 2024-09-11T03:38:55.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.969449+0000", 2024-09-11T03:38:55.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1659000000000002e-05 2024-09-11T03:38:55.452 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.452 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.452 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.452 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:234 2.e 2:7efbbea8:::smithi17557870-24:head [rollback 4] snapc 5=[5,4] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-11T03:38:55.452 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:42.965816+0000", 2024-09-11T03:38:55.452 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 12.440771429, 2024-09-11T03:38:55.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039127449999999996, 2024-09-11T03:38:55.453 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.453 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.453 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:55.453 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:38:55.453 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:38:55.453 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:38:55.453 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 234 2024-09-11T03:38:55.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.453 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965816+0000", 2024-09-11T03:38:55.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965816+0000", 2024-09-11T03:38:55.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965817+0000", 2024-09-11T03:38:55.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.01e-07 2024-09-11T03:38:55.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965819+0000", 2024-09-11T03:38:55.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.95e-06 2024-09-11T03:38:55.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965823+0000", 2024-09-11T03:38:55.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1280000000000001e-06 2024-09-11T03:38:55.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965826+0000", 2024-09-11T03:38:55.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2559999999999998e-06 2024-09-11T03:38:55.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965949+0000", 2024-09-11T03:38:55.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000123187 2024-09-11T03:38:55.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:38:55.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965955+0000", 2024-09-11T03:38:55.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6899999999999997e-06 2024-09-11T03:38:55.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.968350+0000", 2024-09-11T03:38:55.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0023946850000000001 2024-09-11T03:38:55.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.968437+0000", 2024-09-11T03:38:55.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6998999999999998e-05 2024-09-11T03:38:55.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:38:55.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.968615+0000", 2024-09-11T03:38:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017846800000000001 2024-09-11T03:38:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:38:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.969312+0000", 2024-09-11T03:38:55.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069691700000000002 2024-09-11T03:38:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:38:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.969329+0000", 2024-09-11T03:38:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6980999999999999e-05 2024-09-11T03:38:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:38:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.969634+0000", 2024-09-11T03:38:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000305004 2024-09-11T03:38:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.969691+0000", 2024-09-11T03:38:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6743000000000002e-05 2024-09-11T03:38:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.969729+0000", 2024-09-11T03:38:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8336e-05 2024-09-11T03:38:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.460 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:235 2.e 2:7efbbea8:::smithi17557870-24:head [append 0~0] snapc 5=[5,4] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-11T03:38:55.461 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:42.965850+0000", 2024-09-11T03:38:55.461 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 12.440737598, 2024-09-11T03:38:55.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039672969999999998, 2024-09-11T03:38:55.461 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.461 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.461 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:55.461 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:38:55.461 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:38:55.461 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:38:55.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:55.462 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:55.462 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:55.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 235 2024-09-11T03:38:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.463 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965850+0000", 2024-09-11T03:38:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965850+0000", 2024-09-11T03:38:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965850+0000", 2024-09-11T03:38:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3799999999999998e-07 2024-09-11T03:38:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965853+0000", 2024-09-11T03:38:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6809999999999998e-06 2024-09-11T03:38:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965857+0000", 2024-09-11T03:38:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1690000000000002e-06 2024-09-11T03:38:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.965860+0000", 2024-09-11T03:38:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8260000000000001e-06 2024-09-11T03:38:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.966000+0000", 2024-09-11T03:38:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000139603 2024-09-11T03:38:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:38:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.966006+0000", 2024-09-11T03:38:55.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3539999999999996e-06 2024-09-11T03:38:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.968941+0000", 2024-09-11T03:38:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029348270000000001 2024-09-11T03:38:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.968996+0000", 2024-09-11T03:38:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5457000000000001e-05 2024-09-11T03:38:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:38:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.969101+0000", 2024-09-11T03:38:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000104208 2024-09-11T03:38:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:38:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.969455+0000", 2024-09-11T03:38:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00035448499999999998 2024-09-11T03:38:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:38:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.969514+0000", 2024-09-11T03:38:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8659999999999997e-05 2024-09-11T03:38:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:38:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.969756+0000", 2024-09-11T03:38:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024276500000000001 2024-09-11T03:38:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:55.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.969794+0000", 2024-09-11T03:38:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7079999999999997e-05 2024-09-11T03:38:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:42.969817+0000", 2024-09-11T03:38:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3844000000000001e-05 2024-09-11T03:38:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.470 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:240 2.e 2:77dadb97:::smithi17557870-30:head [read 0~2487403,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107)", 2024-09-11T03:38:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:43.966521+0000", 2024-09-11T03:38:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 11.440066496, 2024-09-11T03:38:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055028159999999998, 2024-09-11T03:38:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:38:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:38:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:38:55.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:38:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 240 2024-09-11T03:38:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966521+0000", 2024-09-11T03:38:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966521+0000", 2024-09-11T03:38:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966523+0000", 2024-09-11T03:38:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.306e-06 2024-09-11T03:38:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966533+0000", 2024-09-11T03:38:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0699999999999996e-06 2024-09-11T03:38:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966557+0000", 2024-09-11T03:38:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4892000000000002e-05 2024-09-11T03:38:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966570+0000", 2024-09-11T03:38:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2699e-05 2024-09-11T03:38:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966653+0000", 2024-09-11T03:38:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2806999999999995e-05 2024-09-11T03:38:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:38:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966660+0000", 2024-09-11T03:38:55.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9979999999999997e-06 2024-09-11T03:38:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.968109+0000", 2024-09-11T03:38:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001448637 2024-09-11T03:38:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.968210+0000", 2024-09-11T03:38:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010137 2024-09-11T03:38:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.972024+0000", 2024-09-11T03:38:55.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0038140370000000002 2024-09-11T03:38:55.477 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.477 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.477 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.477 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:241 2.e 2:77dadb97:::smithi17557870-30:head [read 0~2487403,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107)", 2024-09-11T03:38:55.477 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:43.966593+0000", 2024-09-11T03:38:55.477 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 11.439995108, 2024-09-11T03:38:55.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059698279999999999, 2024-09-11T03:38:55.477 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.477 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:38:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:38:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:38:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:38:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 241 2024-09-11T03:38:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966593+0000", 2024-09-11T03:38:55.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966593+0000", 2024-09-11T03:38:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966593+0000", 2024-09-11T03:38:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6499999999999999e-07 2024-09-11T03:38:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966597+0000", 2024-09-11T03:38:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6219999999999998e-06 2024-09-11T03:38:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966694+0000", 2024-09-11T03:38:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6996000000000004e-05 2024-09-11T03:38:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966702+0000", 2024-09-11T03:38:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5019999999999992e-06 2024-09-11T03:38:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966744+0000", 2024-09-11T03:38:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1390999999999997e-05 2024-09-11T03:38:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:38:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966749+0000", 2024-09-11T03:38:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8720000000000001e-06 2024-09-11T03:38:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.970930+0000", 2024-09-11T03:38:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041811180000000002 2024-09-11T03:38:55.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.971036+0000", 2024-09-11T03:38:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105964 2024-09-11T03:38:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.972562+0000", 2024-09-11T03:38:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001526798 2024-09-11T03:38:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:242 2.e 2:77dadb97:::smithi17557870-30:head [sparse-read 0~1009951] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107)", 2024-09-11T03:38:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:43.966721+0000", 2024-09-11T03:38:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 11.439866942, 2024-09-11T03:38:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061840869999999996, 2024-09-11T03:38:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:38:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:38:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:38:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:38:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 242 2024-09-11T03:38:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.484 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966721+0000", 2024-09-11T03:38:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966721+0000", 2024-09-11T03:38:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966721+0000", 2024-09-11T03:38:55.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6300000000000004e-07 2024-09-11T03:38:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966726+0000", 2024-09-11T03:38:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0289999999999999e-06 2024-09-11T03:38:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966733+0000", 2024-09-11T03:38:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.559e-06 2024-09-11T03:38:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966739+0000", 2024-09-11T03:38:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5770000000000001e-06 2024-09-11T03:38:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966797+0000", 2024-09-11T03:38:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8431999999999999e-05 2024-09-11T03:38:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:38:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.966803+0000", 2024-09-11T03:38:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8459999999999996e-06 2024-09-11T03:38:55.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.972608+0000", 2024-09-11T03:38:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005805124 2024-09-11T03:38:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.972732+0000", 2024-09-11T03:38:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012437900000000001 2024-09-11T03:38:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.972905+0000", 2024-09-11T03:38:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000172378 2024-09-11T03:38:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:357 2.e 2:750de3b7:::smithi17557870-8:head [copy-from ver 6] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:38:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:50.832583+0000", 2024-09-11T03:38:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 4.5740048929999997, 2024-09-11T03:38:55.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0095011999999999996, 2024-09-11T03:38:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:38:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:38:55.492 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:38:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 357 2024-09-11T03:38:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832583+0000", 2024-09-11T03:38:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832583+0000", 2024-09-11T03:38:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832585+0000", 2024-09-11T03:38:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3810000000000002e-06 2024-09-11T03:38:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832594+0000", 2024-09-11T03:38:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4659999999999999e-06 2024-09-11T03:38:55.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832617+0000", 2024-09-11T03:38:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2881999999999999e-05 2024-09-11T03:38:55.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832626+0000", 2024-09-11T03:38:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6860000000000001e-06 2024-09-11T03:38:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832713+0000", 2024-09-11T03:38:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6335000000000006e-05 2024-09-11T03:38:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:38:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832721+0000", 2024-09-11T03:38:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2360000000000004e-06 2024-09-11T03:38:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.833953+0000", 2024-09-11T03:38:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012323670000000001 2024-09-11T03:38:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.834039+0000", 2024-09-11T03:38:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5753000000000004e-05 2024-09-11T03:38:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:38:55.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.836611+0000", 2024-09-11T03:38:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025717769999999999 2024-09-11T03:38:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:38:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.839098+0000", 2024-09-11T03:38:55.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024868720000000002 2024-09-11T03:38:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:38:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.840017+0000", 2024-09-11T03:38:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00091944100000000001 2024-09-11T03:38:55.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:38:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.842018+0000", 2024-09-11T03:38:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020006809999999998 2024-09-11T03:38:55.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.842054+0000", 2024-09-11T03:38:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6393999999999997e-05 2024-09-11T03:38:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.842084+0000", 2024-09-11T03:38:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9929000000000001e-05 2024-09-11T03:38:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.507 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:358 2.e 2:750de3b7:::smithi17557870-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:38:55.508 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:50.832651+0000", 2024-09-11T03:38:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 4.5739368330000003, 2024-09-11T03:38:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061318839999999998, 2024-09-11T03:38:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:38:55.509 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:38:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:38:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:38:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 358 2024-09-11T03:38:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832651+0000", 2024-09-11T03:38:55.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832651+0000", 2024-09-11T03:38:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832651+0000", 2024-09-11T03:38:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3e-07 2024-09-11T03:38:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832654+0000", 2024-09-11T03:38:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6809999999999998e-06 2024-09-11T03:38:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832673+0000", 2024-09-11T03:38:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8474e-05 2024-09-11T03:38:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832679+0000", 2024-09-11T03:38:55.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.55e-06 2024-09-11T03:38:55.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832787+0000", 2024-09-11T03:38:55.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010755399999999999 2024-09-11T03:38:55.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:38:55.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832793+0000", 2024-09-11T03:38:55.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6030000000000001e-06 2024-09-11T03:38:55.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.834241+0000", 2024-09-11T03:38:55.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00144798 2024-09-11T03:38:55.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:38:55.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.834268+0000", 2024-09-11T03:38:55.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6795e-05 2024-09-11T03:38:55.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.838571+0000", 2024-09-11T03:38:55.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043027009999999999 2024-09-11T03:38:55.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.838664+0000", 2024-09-11T03:38:55.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3589000000000001e-05 2024-09-11T03:38:55.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.838783+0000", 2024-09-11T03:38:55.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011832699999999999 2024-09-11T03:38:55.521 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.521 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.521 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:366 2.7 e114/95 2:e0eabcfb:::smithi17557870-9:head v 114'24, mlcod=112'22)", 2024-09-11T03:38:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:50.856091+0000", 2024-09-11T03:38:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 4.55049715, 2024-09-11T03:38:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001850373, 2024-09-11T03:38:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.856091+0000", 2024-09-11T03:38:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.856091+0000", 2024-09-11T03:38:55.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.856091+0000", 2024-09-11T03:38:55.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9700000000000002e-07 2024-09-11T03:38:55.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.856959+0000", 2024-09-11T03:38:55.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00086761800000000001 2024-09-11T03:38:55.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.856973+0000", 2024-09-11T03:38:55.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4479e-05 2024-09-11T03:38:55.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.856982+0000", 2024-09-11T03:38:55.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0080000000000008e-06 2024-09-11T03:38:55.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.857040+0000", 2024-09-11T03:38:55.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7778000000000001e-05 2024-09-11T03:38:55.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.857055+0000", 2024-09-11T03:38:55.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.524e-05 2024-09-11T03:38:55.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:55.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.857910+0000", 2024-09-11T03:38:55.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00085514200000000003 2024-09-11T03:38:55.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.857941+0000", 2024-09-11T03:38:55.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0710999999999998e-05 2024-09-11T03:38:55.526 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.526 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.526 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.527 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:367 2.7 e114/95 2:e0eabcfb:::smithi17557870-9:head v 114'25, mlcod=112'23)", 2024-09-11T03:38:55.527 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:50.863166+0000", 2024-09-11T03:38:55.527 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 4.5434212949999999, 2024-09-11T03:38:55.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001816647, 2024-09-11T03:38:55.527 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.527 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.527 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:55.527 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.863166+0000", 2024-09-11T03:38:55.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.863166+0000", 2024-09-11T03:38:55.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.863168+0000", 2024-09-11T03:38:55.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.254e-06 2024-09-11T03:38:55.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.864103+0000", 2024-09-11T03:38:55.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093530999999999996 2024-09-11T03:38:55.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.864119+0000", 2024-09-11T03:38:55.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6418000000000001e-05 2024-09-11T03:38:55.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.864128+0000", 2024-09-11T03:38:55.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0350000000000007e-06 2024-09-11T03:38:55.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.864182+0000", 2024-09-11T03:38:55.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3550000000000001e-05 2024-09-11T03:38:55.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.864197+0000", 2024-09-11T03:38:55.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5393999999999999e-05 2024-09-11T03:38:55.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:55.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.864955+0000", 2024-09-11T03:38:55.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075781299999999998 2024-09-11T03:38:55.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.864983+0000", 2024-09-11T03:38:55.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7872999999999999e-05 2024-09-11T03:38:55.531 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.531 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.532 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.532 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:383 2.e 2:77dadb97:::smithi17557870-30:head [read 0~2638666,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:38:55.532 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:50.978327+0000", 2024-09-11T03:38:55.532 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 4.4282608310000002, 2024-09-11T03:38:55.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019650319999999998, 2024-09-11T03:38:55.532 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.532 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.532 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:38:55.533 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:38:55.533 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:38:55.533 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:38:55.533 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 383 2024-09-11T03:38:55.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.533 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978327+0000", 2024-09-11T03:38:55.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978327+0000", 2024-09-11T03:38:55.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978329+0000", 2024-09-11T03:38:55.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9640000000000002e-06 2024-09-11T03:38:55.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978339+0000", 2024-09-11T03:38:55.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0219e-05 2024-09-11T03:38:55.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978362+0000", 2024-09-11T03:38:55.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2931000000000001e-05 2024-09-11T03:38:55.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978372+0000", 2024-09-11T03:38:55.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6609999999999995e-06 2024-09-11T03:38:55.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978466+0000", 2024-09-11T03:38:55.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4376999999999994e-05 2024-09-11T03:38:55.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978574+0000", 2024-09-11T03:38:55.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107627 2024-09-11T03:38:55.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.980292+0000", 2024-09-11T03:38:55.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017182530000000001 2024-09-11T03:38:55.537 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.537 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.537 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.537 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:384 2.e 2:77dadb97:::smithi17557870-30:head [sparse-read 0~1310720] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:38:55.538 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:50.978405+0000", 2024-09-11T03:38:55.538 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 4.4281825000000001, 2024-09-11T03:38:55.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021008559999999999, 2024-09-11T03:38:55.538 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.538 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.538 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:38:55.538 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:38:55.538 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:38:55.538 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:38:55.538 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 384 2024-09-11T03:38:55.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.538 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978405+0000", 2024-09-11T03:38:55.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978405+0000", 2024-09-11T03:38:55.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978406+0000", 2024-09-11T03:38:55.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.08e-07 2024-09-11T03:38:55.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978409+0000", 2024-09-11T03:38:55.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9390000000000002e-06 2024-09-11T03:38:55.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978414+0000", 2024-09-11T03:38:55.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0409999999999997e-06 2024-09-11T03:38:55.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978417+0000", 2024-09-11T03:38:55.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1089999999999998e-06 2024-09-11T03:38:55.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.980323+0000", 2024-09-11T03:38:55.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019066 2024-09-11T03:38:55.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.980397+0000", 2024-09-11T03:38:55.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3421000000000001e-05 2024-09-11T03:38:55.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.980506+0000", 2024-09-11T03:38:55.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000109338 2024-09-11T03:38:55.542 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.542 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.542 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.543 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:385 2.e 2:77dadb97:::smithi17557870-30:head [read 0~2638666,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:38:55.543 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:50.978458+0000", 2024-09-11T03:38:55.543 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 4.428129642, 2024-09-11T03:38:55.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026922880000000001, 2024-09-11T03:38:55.543 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.543 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.543 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:38:55.543 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:38:55.543 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:38:55.543 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:38:55.543 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 385 2024-09-11T03:38:55.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.544 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978458+0000", 2024-09-11T03:38:55.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978458+0000", 2024-09-11T03:38:55.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978458+0000", 2024-09-11T03:38:55.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1100000000000001e-07 2024-09-11T03:38:55.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978461+0000", 2024-09-11T03:38:55.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0259999999999999e-06 2024-09-11T03:38:55.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978466+0000", 2024-09-11T03:38:55.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8300000000000003e-06 2024-09-11T03:38:55.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.978470+0000", 2024-09-11T03:38:55.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9530000000000001e-06 2024-09-11T03:38:55.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.980534+0000", 2024-09-11T03:38:55.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020634849999999999 2024-09-11T03:38:55.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.980584+0000", 2024-09-11T03:38:55.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0107e-05 2024-09-11T03:38:55.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.981150+0000", 2024-09-11T03:38:55.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000566476 2024-09-11T03:38:55.549 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.549 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.549 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.549 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:402 2.7 e114/95 2:e23d1762:::smithi17557870-21:head v 114'28, mlcod=114'26)", 2024-09-11T03:38:55.549 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:51.091008+0000", 2024-09-11T03:38:55.549 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 4.3155801289999998, 2024-09-11T03:38:55.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066292620000000003, 2024-09-11T03:38:55.550 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:38:55.550 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:38:55.550 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:38:55.550 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:38:55.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:38:55.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.091008+0000", 2024-09-11T03:38:55.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:38:55.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.091008+0000", 2024-09-11T03:38:55.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:38:55.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:38:55.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.091009+0000", 2024-09-11T03:38:55.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1090000000000001e-06 2024-09-11T03:38:55.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:38:55.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.095931+0000", 2024-09-11T03:38:55.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049227560000000004 2024-09-11T03:38:55.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:38:55.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.095949+0000", 2024-09-11T03:38:55.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7271000000000001e-05 2024-09-11T03:38:55.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:38:55.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.095956+0000", 2024-09-11T03:38:55.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3470000000000001e-06 2024-09-11T03:38:55.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:38:55.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.096011+0000", 2024-09-11T03:38:55.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4970999999999999e-05 2024-09-11T03:38:55.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:38:55.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.096029+0000", 2024-09-11T03:38:55.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8068e-05 2024-09-11T03:38:55.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:38:55.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.097544+0000", 2024-09-11T03:38:55.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015149709999999999 2024-09-11T03:38:55.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:38:55.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:38:55.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:38:55.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.097637+0000", 2024-09-11T03:38:55.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2769000000000006e-05 2024-09-11T03:38:55.556 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.556 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.556 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.556 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:38:55.556 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:38:55.556 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:38:55.564 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:38:55.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:55.587 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:55.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:38:55.688 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:38:55.710 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:55.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:55.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:38:55.803 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:38:55.812 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:55.850 INFO:tasks.rados.rados.2.smithi175.stdout:143: done (1 left) 2024-09-11T03:38:55.851 INFO:tasks.rados.rados.2.smithi175.stdout:144: done (0 left) 2024-09-11T03:38:55.851 INFO:tasks.rados.rados.2.smithi175.stdout:145: read oid 6 snap 9 2024-09-11T03:38:55.851 INFO:tasks.rados.rados.2.smithi175.stdout:145: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-11T03:38:55.851 INFO:tasks.rados.rados.2.smithi175.stdout:146: copy_from oid 22 from oid 2 current snap is 10 2024-09-11T03:38:55.851 INFO:tasks.rados.rados.2.smithi175.stdout:147: read oid 14 snap -1 2024-09-11T03:38:55.851 INFO:tasks.rados.rados.2.smithi175.stdout:147: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-11T03:38:55.851 INFO:tasks.rados.rados.2.smithi175.stdout:148: snap_create 2024-09-11T03:38:55.862 INFO:tasks.rados.rados.2.smithi175.stdout:147: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-11T03:38:55.872 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:55.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:55.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:55.935 INFO:tasks.rados.rados.2.smithi175.stdout:145: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-11T03:38:55.974 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:55.995 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:55.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:55.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:56.005 INFO:tasks.rados.rados.2.smithi175.stdout:146: finishing copy_from racing read to smithi17557870-22 2024-09-11T03:38:56.005 INFO:tasks.rados.rados.2.smithi175.stdout:146: finishing copy_from to smithi17557870-22 2024-09-11T03:38:56.005 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T03:38:56.097 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:38:56.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:56.124 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:56.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:38:56.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:55 smithi084 ceph-mon[27254]: pgmap v331: 17 pgs: 17 active+clean; 115 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.9 MiB/s wr, 35 op/s 2024-09-11T03:38:56.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:55 smithi084 ceph-mon[27254]: osdmap e118: 12 total, 12 up, 12 in 2024-09-11T03:38:56.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:55 smithi121 ceph-mon[30729]: pgmap v331: 17 pgs: 17 active+clean; 115 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.9 MiB/s wr, 35 op/s 2024-09-11T03:38:56.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:55 smithi121 ceph-mon[30729]: osdmap e118: 12 total, 12 up, 12 in 2024-09-11T03:38:56.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:55 smithi175 ceph-mon[30894]: pgmap v331: 17 pgs: 17 active+clean; 115 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.9 MiB/s wr, 35 op/s 2024-09-11T03:38:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:55 smithi175 ceph-mon[30894]: osdmap e118: 12 total, 12 up, 12 in 2024-09-11T03:38:56.251 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:56.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:56.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:56.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:56.364 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:38:56.364 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:38:56.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:56.381 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:56.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:56.382 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:56.383 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:38:56.384 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:38:56.401 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:38:56.401 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:38:56.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:38:56.418 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:38:56.434 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:38:56.434 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:38:56.482 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:56.506 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:56.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:56.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:56.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:56.631 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:56.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:56.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:56.657 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:38:56.658 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:38:56.685 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:38:56.686 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:38:56.702 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:38:56.702 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:38:56.720 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:38:56.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:38:56.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:56.736 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:38:56.736 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:38:56.755 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:38:56.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:38:56.756 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:56.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:56.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:56.772 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:38:56.772 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:38:56.813 INFO:tasks.rados.rados.2.smithi175.stdout:145: done (3 left) 2024-09-11T03:38:56.813 INFO:tasks.rados.rados.2.smithi175.stdout:146: done (2 left) 2024-09-11T03:38:56.813 INFO:tasks.rados.rados.2.smithi175.stdout:147: done (1 left) 2024-09-11T03:38:56.814 INFO:tasks.rados.rados.2.smithi175.stdout:148: done (0 left) 2024-09-11T03:38:56.814 INFO:tasks.rados.rados.2.smithi175.stdout:149: snap_remove snap 9 2024-09-11T03:38:56.857 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:38:56.881 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:56.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:56.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:38:56.983 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:38:57.006 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:57.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:57.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:38:57.108 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:38:57.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:56 smithi084 ceph-mon[27254]: osdmap e119: 12 total, 12 up, 12 in 2024-09-11T03:38:57.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4271048371' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:38:57.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:56 smithi084 ceph-mon[27254]: osdmap e120: 12 total, 12 up, 12 in 2024-09-11T03:38:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:57.130 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:57.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:38:57.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:56 smithi121 ceph-mon[30729]: osdmap e119: 12 total, 12 up, 12 in 2024-09-11T03:38:57.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:56 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4271048371' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:38:57.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:56 smithi121 ceph-mon[30729]: osdmap e120: 12 total, 12 up, 12 in 2024-09-11T03:38:57.197 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T03:38:57.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:56 smithi175 ceph-mon[30894]: osdmap e119: 12 total, 12 up, 12 in 2024-09-11T03:38:57.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:56 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4271048371' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:38:57.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:56 smithi175 ceph-mon[30894]: osdmap e120: 12 total, 12 up, 12 in 2024-09-11T03:38:57.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:57.257 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:57.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:57.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:57.359 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:57.384 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:57.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:57.385 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:57.486 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:38:57.507 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:57.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:57.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:38:57.612 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:57.638 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:57.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:57.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:57.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:57.761 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:57.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:57.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:57.810 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.6 to 1 (10000) 2024-09-11T03:38:57.816 INFO:tasks.rados.rados.2.smithi175.stdout:149: done (0 left) 2024-09-11T03:38:57.816 INFO:tasks.rados.rados.2.smithi175.stdout:150: read oid 20 snap -1 2024-09-11T03:38:57.816 INFO:tasks.rados.rados.2.smithi175.stdout:150: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-11T03:38:57.816 INFO:tasks.rados.rados.2.smithi175.stdout:151: write (excl) oid 37 current snap is 11 2024-09-11T03:38:57.816 INFO:tasks.rados.rados.2.smithi175.stdout:151: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-09-11T03:38:57.823 INFO:tasks.rados.rados.2.smithi175.stdout:151: writing smithi17557870-37 from 544560 to 1328010 tid 1 2024-09-11T03:38:57.829 INFO:tasks.rados.rados.2.smithi175.stdout:151: writing smithi17557870-37 from 1915030 to 2396378 tid 2 2024-09-11T03:38:57.834 INFO:tasks.rados.rados.2.smithi175.stdout:151: writing smithi17557870-37 from 2797604 to 3277365 tid 3 2024-09-11T03:38:57.834 INFO:tasks.rados.rados.2.smithi175.stdout:152: write oid 34 current snap is 11 2024-09-11T03:38:57.834 INFO:tasks.rados.rados.2.smithi175.stdout:152: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-09-11T03:38:57.839 INFO:tasks.rados.rados.2.smithi175.stdout:152: writing smithi17557870-34 from 561367 to 1020066 tid 1 2024-09-11T03:38:57.846 INFO:tasks.rados.rados.2.smithi175.stdout:152: writing smithi17557870-34 from 1573512 to 2314871 tid 2 2024-09-11T03:38:57.853 INFO:tasks.rados.rados.2.smithi175.stdout:152: writing smithi17557870-34 from 2941380 to 3361367 tid 3 2024-09-11T03:38:57.853 INFO:tasks.rados.rados.2.smithi175.stdout:153: write (excl) oid 27 current snap is 11 2024-09-11T03:38:57.853 INFO:tasks.rados.rados.2.smithi175.stdout:153: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-09-11T03:38:57.858 INFO:tasks.rados.rados.2.smithi175.stdout:153: writing smithi17557870-27 from 578174 to 1112122 tid 1 2024-09-11T03:38:57.862 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:38:57.863 INFO:tasks.rados.rados.2.smithi175.stdout:153: writing smithi17557870-27 from 1515641 to 2000658 tid 2 2024-09-11T03:38:57.870 INFO:tasks.rados.rados.2.smithi175.stdout:153: writing smithi17557870-27 from 2736097 to 3238402 tid 3 2024-09-11T03:38:57.870 INFO:tasks.rados.rados.2.smithi175.stdout:154: read oid 45 snap -1 2024-09-11T03:38:57.870 INFO:tasks.rados.rados.2.smithi175.stdout:154: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-11T03:38:57.870 INFO:tasks.rados.rados.2.smithi175.stdout:155: copy_from oid 9 from oid 14 current snap is 11 2024-09-11T03:38:57.870 INFO:tasks.rados.rados.2.smithi175.stdout:150: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-11T03:38:57.886 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:57.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:57.887 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:38:57.952 INFO:tasks.rados.rados.2.smithi175.stdout:151: finishing write tid 1 to smithi17557870-37 2024-09-11T03:38:57.952 INFO:tasks.rados.rados.2.smithi175.stdout:151: finishing write tid 2 to smithi17557870-37 2024-09-11T03:38:57.952 INFO:tasks.rados.rados.2.smithi175.stdout:151: finishing write tid 3 to smithi17557870-37 2024-09-11T03:38:57.953 INFO:tasks.rados.rados.2.smithi175.stdout:151: finishing write tid 4 to smithi17557870-37 2024-09-11T03:38:57.953 INFO:tasks.rados.rados.2.smithi175.stdout:151: finishing write tid 5 to smithi17557870-37 2024-09-11T03:38:57.953 INFO:tasks.rados.rados.2.smithi175.stdout:151: oid 37 updating version 0 to 26 2024-09-11T03:38:57.953 INFO:tasks.rados.rados.2.smithi175.stdout:151: oid 37 updating version 26 to 27 2024-09-11T03:38:57.953 INFO:tasks.rados.rados.2.smithi175.stdout:151: oid 37 version 27 is already newer than 25 2024-09-11T03:38:57.953 INFO:tasks.rados.rados.2.smithi175.stdout:151: oid 37 version 27 is already newer than 24 2024-09-11T03:38:57.953 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 27 (ObjNum 80 snap 11 seq_num 80) dirty exists 2024-09-11T03:38:57.953 INFO:tasks.rados.rados.2.smithi175.stdout:151: left oid 37 (ObjNum 80 snap 11 seq_num 80) 2024-09-11T03:38:57.953 INFO:tasks.rados.rados.2.smithi175.stdout:152: finishing write tid 1 to smithi17557870-34 2024-09-11T03:38:57.953 INFO:tasks.rados.rados.2.smithi175.stdout:152: finishing write tid 2 to smithi17557870-34 2024-09-11T03:38:57.953 INFO:tasks.rados.rados.2.smithi175.stdout:152: finishing write tid 3 to smithi17557870-34 2024-09-11T03:38:57.953 INFO:tasks.rados.rados.2.smithi175.stdout:152: finishing write tid 4 to smithi17557870-34 2024-09-11T03:38:57.953 INFO:tasks.rados.rados.2.smithi175.stdout:150: done (5 left) 2024-09-11T03:38:57.954 INFO:tasks.rados.rados.2.smithi175.stdout:151: done (4 left) 2024-09-11T03:38:57.954 INFO:tasks.rados.rados.2.smithi175.stdout:156: snap_create 2024-09-11T03:38:57.954 INFO:tasks.rados.rados.2.smithi175.stdout:152: finishing write tid 5 to smithi17557870-34 2024-09-11T03:38:57.954 INFO:tasks.rados.rados.2.smithi175.stdout:152: oid 34 updating version 0 to 19 2024-09-11T03:38:57.954 INFO:tasks.rados.rados.2.smithi175.stdout:152: oid 34 updating version 19 to 20 2024-09-11T03:38:57.954 INFO:tasks.rados.rados.2.smithi175.stdout:152: oid 34 version 20 is already newer than 18 2024-09-11T03:38:57.954 INFO:tasks.rados.rados.2.smithi175.stdout:152: oid 34 updating version 20 to 21 2024-09-11T03:38:57.954 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 21 (ObjNum 81 snap 11 seq_num 81) dirty exists 2024-09-11T03:38:57.954 INFO:tasks.rados.rados.2.smithi175.stdout:152: left oid 34 (ObjNum 81 snap 11 seq_num 81) 2024-09-11T03:38:57.954 INFO:tasks.rados.rados.2.smithi175.stdout:153: finishing write tid 1 to smithi17557870-27 2024-09-11T03:38:57.954 INFO:tasks.rados.rados.2.smithi175.stdout:153: finishing write tid 2 to smithi17557870-27 2024-09-11T03:38:57.954 INFO:tasks.rados.rados.2.smithi175.stdout:153: finishing write tid 3 to smithi17557870-27 2024-09-11T03:38:57.955 INFO:tasks.rados.rados.2.smithi175.stdout:153: finishing write tid 4 to smithi17557870-27 2024-09-11T03:38:57.955 INFO:tasks.rados.rados.2.smithi175.stdout:153: finishing write tid 5 to smithi17557870-27 2024-09-11T03:38:57.955 INFO:tasks.rados.rados.2.smithi175.stdout:153: oid 27 updating version 0 to 36 2024-09-11T03:38:57.955 INFO:tasks.rados.rados.2.smithi175.stdout:153: oid 27 version 36 is already newer than 35 2024-09-11T03:38:57.955 INFO:tasks.rados.rados.2.smithi175.stdout:153: oid 27 version 36 is already newer than 34 2024-09-11T03:38:57.955 INFO:tasks.rados.rados.2.smithi175.stdout:153: oid 27 version 36 is already newer than 33 2024-09-11T03:38:57.955 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 36 (ObjNum 82 snap 11 seq_num 82) dirty exists 2024-09-11T03:38:57.955 INFO:tasks.rados.rados.2.smithi175.stdout:153: left oid 27 (ObjNum 82 snap 11 seq_num 82) 2024-09-11T03:38:57.955 INFO:tasks.rados.rados.2.smithi175.stdout:154: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-11T03:38:57.989 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:38:58.010 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:58.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:58.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:38:58.037 INFO:tasks.rados.rados.2.smithi175.stdout:155: finishing copy_from racing read to smithi17557870-9 2024-09-11T03:38:58.037 INFO:tasks.rados.rados.2.smithi175.stdout:155: finishing copy_from to smithi17557870-9 2024-09-11T03:38:58.037 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-11T03:38:58.112 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:58.138 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:58.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:58.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:57 smithi084 ceph-mon[27254]: pgmap v334: 17 pgs: 17 active+clean; 115 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.5 MiB/s wr, 14 op/s 2024-09-11T03:38:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:57 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1879128479' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-11T03:38:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:57 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1879128479' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T03:38:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:57 smithi084 ceph-mon[27254]: osdmap e121: 12 total, 12 up, 12 in 2024-09-11T03:38:58.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:57 smithi121 ceph-mon[30729]: pgmap v334: 17 pgs: 17 active+clean; 115 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.5 MiB/s wr, 14 op/s 2024-09-11T03:38:58.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:57 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1879128479' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-11T03:38:58.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:57 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1879128479' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T03:38:58.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:57 smithi121 ceph-mon[30729]: osdmap e121: 12 total, 12 up, 12 in 2024-09-11T03:38:58.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:57 smithi175 ceph-mon[30894]: pgmap v334: 17 pgs: 17 active+clean; 115 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.5 MiB/s wr, 14 op/s 2024-09-11T03:38:58.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:57 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1879128479' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-11T03:38:58.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:57 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1879128479' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T03:38:58.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:57 smithi175 ceph-mon[30894]: osdmap e121: 12 total, 12 up, 12 in 2024-09-11T03:38:58.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:58.265 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:58.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:58.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:58.307 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:38:58.367 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:38:58.388 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:58.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:58.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:38:58.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:58.515 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:58.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:58.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:58.618 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:38:58.639 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:58.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:58.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:38:58.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:38:58.767 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:58.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:58.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:38:58.863 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:38:58.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:38:58.873 INFO:tasks.rados.rados.2.smithi175.stdout:152: done (4 left) 2024-09-11T03:38:58.873 INFO:tasks.rados.rados.2.smithi175.stdout:153: done (3 left) 2024-09-11T03:38:58.873 INFO:tasks.rados.rados.2.smithi175.stdout:154: done (2 left) 2024-09-11T03:38:58.873 INFO:tasks.rados.rados.2.smithi175.stdout:155: done (1 left) 2024-09-11T03:38:58.873 INFO:tasks.rados.rados.2.smithi175.stdout:156: done (0 left) 2024-09-11T03:38:58.873 INFO:tasks.rados.rados.2.smithi175.stdout:157: read oid 33 snap -1 2024-09-11T03:38:58.873 INFO:tasks.rados.rados.2.smithi175.stdout:157: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-11T03:38:58.873 INFO:tasks.rados.rados.2.smithi175.stdout:158: copy_from oid 37 from oid 28 current snap is 12 2024-09-11T03:38:58.873 INFO:tasks.rados.rados.2.smithi175.stdout:159: read oid 24 snap -1 2024-09-11T03:38:58.873 INFO:tasks.rados.rados.2.smithi175.stdout:159: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-11T03:38:58.874 INFO:tasks.rados.rados.2.smithi175.stdout:160: write (excl) oid 22 current snap is 12 2024-09-11T03:38:58.874 INFO:tasks.rados.rados.2.smithi175.stdout:160: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-09-11T03:38:58.879 INFO:tasks.rados.rados.2.smithi175.stdout:160: writing smithi17557870-22 from 594981 to 1204178 tid 1 2024-09-11T03:38:58.887 INFO:tasks.rados.rados.2.smithi175.stdout:160: writing smithi17557870-22 from 1974123 to 2719151 tid 2 2024-09-11T03:38:58.891 INFO:tasks.rados.rados.2.smithi175.stdout:160: writing smithi17557870-22 from 3279873 to 3394981 tid 3 2024-09-11T03:38:58.891 INFO:tasks.rados.rados.2.smithi175.stdout:161: delete oid 40 current snap is 12 2024-09-11T03:38:58.891 INFO:tasks.rados.rados.2.smithi175.stdout:159: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-11T03:38:58.896 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:58.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:58.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:38:58.969 INFO:tasks.rados.rados.2.smithi175.stdout:157: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-11T03:38:58.998 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:59.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:59.020 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:59.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:59.051 INFO:tasks.rados.rados.2.smithi175.stdout:160: finishing write tid 1 to smithi17557870-22 2024-09-11T03:38:59.051 INFO:tasks.rados.rados.2.smithi175.stdout:158: finishing copy_from racing read to smithi17557870-37 2024-09-11T03:38:59.051 INFO:tasks.rados.rados.2.smithi175.stdout:157: done (4 left) 2024-09-11T03:38:59.051 INFO:tasks.rados.rados.2.smithi175.stdout:159: done (3 left) 2024-09-11T03:38:59.051 INFO:tasks.rados.rados.2.smithi175.stdout:161: done (2 left) 2024-09-11T03:38:59.051 INFO:tasks.rados.rados.2.smithi175.stdout:162: copy_from oid 50 from oid 5 current snap is 12 2024-09-11T03:38:59.051 INFO:tasks.rados.rados.2.smithi175.stdout:158: finishing copy_from to smithi17557870-37 2024-09-11T03:38:59.051 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 12 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-11T03:38:59.051 INFO:tasks.rados.rados.2.smithi175.stdout:160: finishing write tid 2 to smithi17557870-22 2024-09-11T03:38:59.051 INFO:tasks.rados.rados.2.smithi175.stdout:160: finishing write tid 3 to smithi17557870-22 2024-09-11T03:38:59.052 INFO:tasks.rados.rados.2.smithi175.stdout:160: finishing write tid 4 to smithi17557870-22 2024-09-11T03:38:59.052 INFO:tasks.rados.rados.2.smithi175.stdout:160: finishing write tid 5 to smithi17557870-22 2024-09-11T03:38:59.052 INFO:tasks.rados.rados.2.smithi175.stdout:160: oid 22 updating version 0 to 46 2024-09-11T03:38:59.052 INFO:tasks.rados.rados.2.smithi175.stdout:160: oid 22 updating version 46 to 47 2024-09-11T03:38:59.052 INFO:tasks.rados.rados.2.smithi175.stdout:160: oid 22 version 47 is already newer than 45 2024-09-11T03:38:59.052 INFO:tasks.rados.rados.2.smithi175.stdout:160: oid 22 updating version 47 to 48 2024-09-11T03:38:59.052 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 48 (ObjNum 83 snap 12 seq_num 83) dirty exists 2024-09-11T03:38:59.052 INFO:tasks.rados.rados.2.smithi175.stdout:160: left oid 22 (ObjNum 83 snap 12 seq_num 83) 2024-09-11T03:38:59.053 INFO:tasks.rados.rados.2.smithi175.stdout:158: done (2 left) 2024-09-11T03:38:59.053 INFO:tasks.rados.rados.2.smithi175.stdout:160: done (1 left) 2024-09-11T03:38:59.053 INFO:tasks.rados.rados.2.smithi175.stdout:163: read oid 6 snap -1 2024-09-11T03:38:59.053 INFO:tasks.rados.rados.2.smithi175.stdout:163: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-11T03:38:59.053 INFO:tasks.rados.rados.2.smithi175.stdout:164: copy_from oid 48 from oid 37 current snap is 12 2024-09-11T03:38:59.053 INFO:tasks.rados.rados.2.smithi175.stdout:165: write (excl) oid 26 current snap is 12 2024-09-11T03:38:59.053 INFO:tasks.rados.rados.2.smithi175.stdout:165: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-09-11T03:38:59.056 INFO:tasks.rados.rados.2.smithi175.stdout:165: writing smithi17557870-26 from 628595 to 1104643 tid 1 2024-09-11T03:38:59.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:58 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3203188928' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:38:59.064 INFO:tasks.rados.rados.2.smithi175.stdout:165: writing smithi17557870-26 from 1807440 to 2439784 tid 2 2024-09-11T03:38:59.069 INFO:tasks.rados.rados.2.smithi175.stdout:165: writing smithi17557870-26 from 2934719 to 3387193 tid 3 2024-09-11T03:38:59.069 INFO:tasks.rados.rados.2.smithi175.stdout:166: read oid 42 snap 10 2024-09-11T03:38:59.070 INFO:tasks.rados.rados.2.smithi175.stdout:166: expect deleted 2024-09-11T03:38:59.070 INFO:tasks.rados.rados.2.smithi175.stdout:167: write (excl) oid 1 current snap is 12 2024-09-11T03:38:59.070 INFO:tasks.rados.rados.2.smithi175.stdout:167: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-09-11T03:38:59.076 INFO:tasks.rados.rados.2.smithi175.stdout:167: writing smithi17557870-1 from 645402 to 1196699 tid 1 2024-09-11T03:38:59.082 INFO:tasks.rados.rados.2.smithi175.stdout:167: writing smithi17557870-1 from 1749569 to 2241924 tid 2 2024-09-11T03:38:59.089 INFO:tasks.rados.rados.2.smithi175.stdout:167: writing smithi17557870-1 from 2845789 to 3445402 tid 3 2024-09-11T03:38:59.089 INFO:tasks.rados.rados.2.smithi175.stdout:168: write oid 3 current snap is 12 2024-09-11T03:38:59.089 INFO:tasks.rados.rados.2.smithi175.stdout:168: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-09-11T03:38:59.096 INFO:tasks.rados.rados.2.smithi175.stdout:168: writing smithi17557870-3 from 662209 to 1288755 tid 1 2024-09-11T03:38:59.102 INFO:tasks.rados.rados.2.smithi175.stdout:168: writing smithi17557870-3 from 1808051 to 2444064 tid 2 2024-09-11T03:38:59.107 INFO:tasks.rados.rados.2.smithi175.stdout:168: writing smithi17557870-3 from 2873212 to 3462209 tid 3 2024-09-11T03:38:59.108 INFO:tasks.rados.rados.2.smithi175.stdout:169: write oid 17 current snap is 12 2024-09-11T03:38:59.108 INFO:tasks.rados.rados.2.smithi175.stdout:169: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-09-11T03:38:59.115 INFO:tasks.rados.rados.2.smithi175.stdout:169: writing smithi17557870-17 from 679016 to 1380811 tid 1 2024-09-11T03:38:59.120 INFO:tasks.rados.rados.2.smithi175.stdout:169: writing smithi17557870-17 from 1866533 to 2362557 tid 2 2024-09-11T03:38:59.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:38:59.125 INFO:tasks.rados.rados.2.smithi175.stdout:169: writing smithi17557870-17 from 2900635 to 3303278 tid 3 2024-09-11T03:38:59.126 INFO:tasks.rados.rados.2.smithi175.stdout:170: delete oid 4 current snap is 12 2024-09-11T03:38:59.126 INFO:tasks.rados.rados.2.smithi175.stdout:163: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-11T03:38:59.145 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:59.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:59.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:38:59.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:58 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3203188928' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:38:59.197 INFO:tasks.rados.rados.2.smithi175.stdout:162: finishing copy_from racing read to smithi17557870-50 2024-09-11T03:38:59.197 INFO:tasks.rados.rados.2.smithi175.stdout:162: finishing copy_from to smithi17557870-50 2024-09-11T03:38:59.197 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-11T03:38:59.197 INFO:tasks.rados.rados.2.smithi175.stdout:164: finishing copy_from racing read to smithi17557870-48 2024-09-11T03:38:59.197 INFO:tasks.rados.rados.2.smithi175.stdout:165: finishing write tid 1 to smithi17557870-26 2024-09-11T03:38:59.197 INFO:tasks.rados.rados.2.smithi175.stdout:164: finishing copy_from to smithi17557870-48 2024-09-11T03:38:59.197 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 12 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-11T03:38:59.198 INFO:tasks.rados.rados.2.smithi175.stdout:165: finishing write tid 2 to smithi17557870-26 2024-09-11T03:38:59.198 INFO:tasks.rados.rados.2.smithi175.stdout:162: done (8 left) 2024-09-11T03:38:59.198 INFO:tasks.rados.rados.2.smithi175.stdout:163: done (7 left) 2024-09-11T03:38:59.198 INFO:tasks.rados.rados.2.smithi175.stdout:164: done (6 left) 2024-09-11T03:38:59.198 INFO:tasks.rados.rados.2.smithi175.stdout:170: done (5 left) 2024-09-11T03:38:59.198 INFO:tasks.rados.rados.2.smithi175.stdout:171: copy_from oid 7 from oid 36 current snap is 12 2024-09-11T03:38:59.198 INFO:tasks.rados.rados.2.smithi175.stdout:165: finishing write tid 3 to smithi17557870-26 2024-09-11T03:38:59.198 INFO:tasks.rados.rados.2.smithi175.stdout:165: finishing write tid 4 to smithi17557870-26 2024-09-11T03:38:59.198 INFO:tasks.rados.rados.2.smithi175.stdout:165: finishing write tid 5 to smithi17557870-26 2024-09-11T03:38:59.198 INFO:tasks.rados.rados.2.smithi175.stdout:165: oid 26 updating version 0 to 23 2024-09-11T03:38:59.198 INFO:tasks.rados.rados.2.smithi175.stdout:165: oid 26 updating version 23 to 25 2024-09-11T03:38:59.198 INFO:tasks.rados.rados.2.smithi175.stdout:165: oid 26 version 25 is already newer than 24 2024-09-11T03:38:59.198 INFO:tasks.rados.rados.2.smithi175.stdout:165: oid 26 updating version 25 to 26 2024-09-11T03:38:59.199 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 26 (ObjNum 85 snap 12 seq_num 85) dirty exists 2024-09-11T03:38:59.199 INFO:tasks.rados.rados.2.smithi175.stdout:165: left oid 26 (ObjNum 85 snap 12 seq_num 85) 2024-09-11T03:38:59.199 INFO:tasks.rados.rados.2.smithi175.stdout:167: finishing write tid 1 to smithi17557870-1 2024-09-11T03:38:59.199 INFO:tasks.rados.rados.2.smithi175.stdout:167: finishing write tid 2 to smithi17557870-1 2024-09-11T03:38:59.199 INFO:tasks.rados.rados.2.smithi175.stdout:167: finishing write tid 3 to smithi17557870-1 2024-09-11T03:38:59.199 INFO:tasks.rados.rados.2.smithi175.stdout:167: finishing write tid 4 to smithi17557870-1 2024-09-11T03:38:59.199 INFO:tasks.rados.rados.2.smithi175.stdout:167: finishing write tid 5 to smithi17557870-1 2024-09-11T03:38:59.199 INFO:tasks.rados.rados.2.smithi175.stdout:167: oid 1 updating version 0 to 50 2024-09-11T03:38:59.199 INFO:tasks.rados.rados.2.smithi175.stdout:167: oid 1 version 50 is already newer than 49 2024-09-11T03:38:59.199 INFO:tasks.rados.rados.2.smithi175.stdout:167: oid 1 updating version 50 to 51 2024-09-11T03:38:59.199 INFO:tasks.rados.rados.2.smithi175.stdout:167: oid 1 version 51 is already newer than 48 2024-09-11T03:38:59.199 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 51 (ObjNum 86 snap 12 seq_num 86) dirty exists 2024-09-11T03:38:59.199 INFO:tasks.rados.rados.2.smithi175.stdout:167: left oid 1 (ObjNum 86 snap 12 seq_num 86) 2024-09-11T03:38:59.199 INFO:tasks.rados.rados.2.smithi175.stdout:168: finishing write tid 1 to smithi17557870-3 2024-09-11T03:38:59.200 INFO:tasks.rados.rados.2.smithi175.stdout:168: finishing write tid 2 to smithi17557870-3 2024-09-11T03:38:59.200 INFO:tasks.rados.rados.2.smithi175.stdout:168: finishing write tid 3 to smithi17557870-3 2024-09-11T03:38:59.200 INFO:tasks.rados.rados.2.smithi175.stdout:168: finishing write tid 4 to smithi17557870-3 2024-09-11T03:38:59.200 INFO:tasks.rados.rados.2.smithi175.stdout:168: finishing write tid 5 to smithi17557870-3 2024-09-11T03:38:59.200 INFO:tasks.rados.rados.2.smithi175.stdout:168: oid 3 updating version 0 to 55 2024-09-11T03:38:59.200 INFO:tasks.rados.rados.2.smithi175.stdout:168: oid 3 version 55 is already newer than 52 2024-09-11T03:38:59.200 INFO:tasks.rados.rados.2.smithi175.stdout:168: oid 3 version 55 is already newer than 54 2024-09-11T03:38:59.200 INFO:tasks.rados.rados.2.smithi175.stdout:168: oid 3 version 55 is already newer than 53 2024-09-11T03:38:59.200 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 55 (ObjNum 87 snap 12 seq_num 87) dirty exists 2024-09-11T03:38:59.200 INFO:tasks.rados.rados.2.smithi175.stdout:168: left oid 3 (ObjNum 87 snap 12 seq_num 87) 2024-09-11T03:38:59.200 INFO:tasks.rados.rados.2.smithi175.stdout:169: finishing write tid 1 to smithi17557870-17 2024-09-11T03:38:59.200 INFO:tasks.rados.rados.2.smithi175.stdout:169: finishing write tid 2 to smithi17557870-17 2024-09-11T03:38:59.200 INFO:tasks.rados.rados.2.smithi175.stdout:169: finishing write tid 3 to smithi17557870-17 2024-09-11T03:38:59.201 INFO:tasks.rados.rados.2.smithi175.stdout:169: finishing write tid 4 to smithi17557870-17 2024-09-11T03:38:59.201 INFO:tasks.rados.rados.2.smithi175.stdout:169: finishing write tid 5 to smithi17557870-17 2024-09-11T03:38:59.201 INFO:tasks.rados.rados.2.smithi175.stdout:169: oid 17 updating version 0 to 39 2024-09-11T03:38:59.201 INFO:tasks.rados.rados.2.smithi175.stdout:169: oid 17 version 39 is already newer than 36 2024-09-11T03:38:59.201 INFO:tasks.rados.rados.2.smithi175.stdout:169: oid 17 version 39 is already newer than 38 2024-09-11T03:38:59.201 INFO:tasks.rados.rados.2.smithi175.stdout:169: oid 17 version 39 is already newer than 37 2024-09-11T03:38:59.201 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 39 (ObjNum 88 snap 12 seq_num 88) dirty exists 2024-09-11T03:38:59.201 INFO:tasks.rados.rados.2.smithi175.stdout:169: left oid 17 (ObjNum 88 snap 12 seq_num 88) 2024-09-11T03:38:59.201 INFO:tasks.rados.rados.2.smithi175.stdout:165: done (5 left) 2024-09-11T03:38:59.201 INFO:tasks.rados.rados.2.smithi175.stdout:166: done (4 left) 2024-09-11T03:38:59.201 INFO:tasks.rados.rados.2.smithi175.stdout:167: done (3 left) 2024-09-11T03:38:59.201 INFO:tasks.rados.rados.2.smithi175.stdout:168: done (2 left) 2024-09-11T03:38:59.201 INFO:tasks.rados.rados.2.smithi175.stdout:169: done (1 left) 2024-09-11T03:38:59.201 INFO:tasks.rados.rados.2.smithi175.stdout:172: write oid 42 current snap is 12 2024-09-11T03:38:59.201 INFO:tasks.rados.rados.2.smithi175.stdout:172: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2024-09-11T03:38:59.202 INFO:tasks.rados.rados.2.smithi175.stdout:172: writing smithi17557870-42 from 712630 to 1164923 tid 1 2024-09-11T03:38:59.207 INFO:tasks.rados.rados.2.smithi175.stdout:172: writing smithi17557870-42 from 1583497 to 2083190 tid 2 2024-09-11T03:38:59.213 INFO:tasks.rados.rados.2.smithi175.stdout:172: writing smithi17557870-42 from 2555481 to 2980668 tid 3 2024-09-11T03:38:59.213 INFO:tasks.rados.rados.2.smithi175.stdout:173: read oid 19 snap 11 2024-09-11T03:38:59.213 INFO:tasks.rados.rados.2.smithi175.stdout:173: expect deleted 2024-09-11T03:38:59.213 INFO:tasks.rados.rados.2.smithi175.stdout:174: rollback oid 44 current snap is 12 2024-09-11T03:38:59.213 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 44 to 11 2024-09-11T03:38:59.213 INFO:tasks.rados.rados.2.smithi175.stdout:172: finishing write tid 1 to smithi17557870-42 2024-09-11T03:38:59.213 INFO:tasks.rados.rados.2.smithi175.stdout:171: finishing copy_from racing read to smithi17557870-7 2024-09-11T03:38:59.213 INFO:tasks.rados.rados.2.smithi175.stdout:171: finishing copy_from to smithi17557870-7 2024-09-11T03:38:59.214 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 13 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-11T03:38:59.214 INFO:tasks.rados.rados.2.smithi175.stdout:172: finishing write tid 2 to smithi17557870-42 2024-09-11T03:38:59.214 INFO:tasks.rados.rados.2.smithi175.stdout:171: done (3 left) 2024-09-11T03:38:59.214 INFO:tasks.rados.rados.2.smithi175.stdout:175: snap_remove snap 10 2024-09-11T03:38:59.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:58 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3203188928' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:38:59.248 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:38:59.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:59.271 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:59.271 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:38:59.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:38:59.396 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:38:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:59.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:38:59.485 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:38:59.485 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:38:59.497 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:38:59.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:59.520 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:59.520 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:38:59.621 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:59.643 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:59.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:59.644 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:59.746 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:38:59.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:59.769 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:38:59.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:38:59.870 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:38:59.878 INFO:tasks.rados.rados.2.smithi175.stdout:175: done (3 left) 2024-09-11T03:38:59.878 INFO:tasks.rados.rados.2.smithi175.stdout:176: copy_from oid 49 from oid 16 current snap is 12 2024-09-11T03:38:59.878 INFO:tasks.rados.rados.2.smithi175.stdout:174: finishing rollback tid 0 to smithi17557870-44 2024-09-11T03:38:59.878 INFO:tasks.rados.rados.2.smithi175.stdout:172: finishing write tid 3 to smithi17557870-42 2024-09-11T03:38:59.878 INFO:tasks.rados.rados.2.smithi175.stdout:172: finishing write tid 4 to smithi17557870-42 2024-09-11T03:38:59.878 INFO:tasks.rados.rados.2.smithi175.stdout:174: finishing rollback tid 1 to smithi17557870-44 2024-09-11T03:38:59.878 INFO:tasks.rados.rados.2.smithi175.stdout:174: finishing rollback tid 2 to smithi17557870-44 2024-09-11T03:38:59.879 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 18 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-11T03:38:59.879 INFO:tasks.rados.rados.2.smithi175.stdout:172: finishing write tid 5 to smithi17557870-42 2024-09-11T03:38:59.879 INFO:tasks.rados.rados.2.smithi175.stdout:172: oid 42 updating version 0 to 24 2024-09-11T03:38:59.879 INFO:tasks.rados.rados.2.smithi175.stdout:172: oid 42 version 24 is already newer than 22 2024-09-11T03:38:59.879 INFO:tasks.rados.rados.2.smithi175.stdout:172: oid 42 updating version 24 to 25 2024-09-11T03:38:59.879 INFO:tasks.rados.rados.2.smithi175.stdout:172: oid 42 version 25 is already newer than 23 2024-09-11T03:38:59.879 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 25 (ObjNum 90 snap 12 seq_num 90) dirty exists 2024-09-11T03:38:59.879 INFO:tasks.rados.rados.2.smithi175.stdout:172: left oid 42 (ObjNum 90 snap 12 seq_num 90) 2024-09-11T03:38:59.879 INFO:tasks.rados.rados.2.smithi175.stdout:172: done (3 left) 2024-09-11T03:38:59.879 INFO:tasks.rados.rados.2.smithi175.stdout:173: done (2 left) 2024-09-11T03:38:59.879 INFO:tasks.rados.rados.2.smithi175.stdout:174: done (1 left) 2024-09-11T03:38:59.879 INFO:tasks.rados.rados.2.smithi175.stdout:177: read oid 1 snap -1 2024-09-11T03:38:59.879 INFO:tasks.rados.rados.2.smithi175.stdout:177: expect (ObjNum 86 snap 12 seq_num 86) 2024-09-11T03:38:59.879 INFO:tasks.rados.rados.2.smithi175.stdout:178: read oid 2 snap 11 2024-09-11T03:38:59.880 INFO:tasks.rados.rados.2.smithi175.stdout:178: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-11T03:38:59.880 INFO:tasks.rados.rados.2.smithi175.stdout:179: rollback oid 42 current snap is 12 2024-09-11T03:38:59.880 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 42 to 11 2024-09-11T03:38:59.880 INFO:tasks.rados.rados.2.smithi175.stdout:180: write (excl) oid 47 current snap is 12 2024-09-11T03:38:59.880 INFO:tasks.rados.rados.2.smithi175.stdout:180: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2024-09-11T03:38:59.886 INFO:tasks.rados.rados.2.smithi175.stdout:180: writing smithi17557870-47 from 729437 to 1256979 tid 1 2024-09-11T03:38:59.894 INFO:tasks.rados.rados.2.smithi175.stdout:180: writing smithi17557870-47 from 1925626 to 2568977 tid 2 2024-09-11T03:38:59.898 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:38:59.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:38:59.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:38:59.900 INFO:tasks.rados.rados.2.smithi175.stdout:180: writing smithi17557870-47 from 3266551 to 3529437 tid 3 2024-09-11T03:38:59.900 INFO:tasks.rados.rados.2.smithi175.stdout:181: delete oid 26 current snap is 12 2024-09-11T03:38:59.900 INFO:tasks.rados.rados.2.smithi175.stdout:179: finishing rollback tid 0 to smithi17557870-42 2024-09-11T03:38:59.900 INFO:tasks.rados.rados.2.smithi175.stdout:179: finishing rollback tid 1 to smithi17557870-42 2024-09-11T03:38:59.900 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 27 (ObjNum 0 snap 0 seq_num 304) dirty dne 2024-09-11T03:38:59.900 INFO:tasks.rados.rados.2.smithi175.stdout:176: finishing copy_from to smithi17557870-49 2024-09-11T03:38:59.900 INFO:tasks.rados.rados.2.smithi175.stdout:176: got expected ENOENT (src dne) 2024-09-11T03:38:59.900 INFO:tasks.rados.rados.2.smithi175.stdout:176: finishing copy_from racing read to smithi17557870-49 2024-09-11T03:38:59.900 INFO:tasks.rados.rados.2.smithi175.stdout:178: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-11T03:38:59.965 INFO:tasks.rados.rados.2.smithi175.stdout:180: finishing write tid 1 to smithi17557870-47 2024-09-11T03:38:59.965 INFO:tasks.rados.rados.2.smithi175.stdout:177: expect (ObjNum 86 snap 12 seq_num 86) 2024-09-11T03:39:00.000 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:00.022 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:00.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:00.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:00.074 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-11T03:39:00.111 INFO:tasks.rados.rados.2.smithi175.stdout:180: finishing write tid 2 to smithi17557870-47 2024-09-11T03:39:00.111 INFO:tasks.rados.rados.2.smithi175.stdout:180: finishing write tid 3 to smithi17557870-47 2024-09-11T03:39:00.111 INFO:tasks.rados.rados.2.smithi175.stdout:180: finishing write tid 4 to smithi17557870-47 2024-09-11T03:39:00.111 INFO:tasks.rados.rados.2.smithi175.stdout:180: finishing write tid 5 to smithi17557870-47 2024-09-11T03:39:00.111 INFO:tasks.rados.rados.2.smithi175.stdout:180: oid 47 updating version 0 to 43 2024-09-11T03:39:00.111 INFO:tasks.rados.rados.2.smithi175.stdout:180: oid 47 updating version 43 to 45 2024-09-11T03:39:00.111 INFO:tasks.rados.rados.2.smithi175.stdout:180: oid 47 version 45 is already newer than 44 2024-09-11T03:39:00.111 INFO:tasks.rados.rados.2.smithi175.stdout:180: oid 47 updating version 45 to 46 2024-09-11T03:39:00.112 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 46 (ObjNum 91 snap 12 seq_num 91) dirty exists 2024-09-11T03:39:00.112 INFO:tasks.rados.rados.2.smithi175.stdout:180: left oid 47 (ObjNum 91 snap 12 seq_num 91) 2024-09-11T03:39:00.112 INFO:tasks.rados.rados.2.smithi175.stdout:176: done (5 left) 2024-09-11T03:39:00.112 INFO:tasks.rados.rados.2.smithi175.stdout:177: done (4 left) 2024-09-11T03:39:00.112 INFO:tasks.rados.rados.2.smithi175.stdout:178: done (3 left) 2024-09-11T03:39:00.112 INFO:tasks.rados.rados.2.smithi175.stdout:179: done (2 left) 2024-09-11T03:39:00.112 INFO:tasks.rados.rados.2.smithi175.stdout:180: done (1 left) 2024-09-11T03:39:00.112 INFO:tasks.rados.rados.2.smithi175.stdout:181: done (0 left) 2024-09-11T03:39:00.112 INFO:tasks.rados.rados.2.smithi175.stdout:182: write (excl) oid 18 current snap is 12 2024-09-11T03:39:00.112 INFO:tasks.rados.rados.2.smithi175.stdout:182: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2024-09-11T03:39:00.119 INFO:tasks.rados.rados.2.smithi175.stdout:182: writing smithi17557870-18 from 763051 to 1557444 tid 1 2024-09-11T03:39:00.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:00.126 INFO:tasks.rados.rados.2.smithi175.stdout:182: writing smithi17557870-18 from 2158943 to 2805963 tid 2 2024-09-11T03:39:00.130 INFO:tasks.rados.rados.2.smithi175.stdout:182: writing smithi17557870-18 from 3437750 to 3563051 tid 3 2024-09-11T03:39:00.130 INFO:tasks.rados.rados.2.smithi175.stdout:183: snap_remove snap 11 2024-09-11T03:39:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:59 smithi084 ceph-mon[27254]: pgmap v337: 17 pgs: 17 active+clean; 113 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 876 KiB/s wr, 15 op/s 2024-09-11T03:39:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:59 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:59 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3203188928' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:39:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:38:59 smithi084 ceph-mon[27254]: osdmap e122: 12 total, 12 up, 12 in 2024-09-11T03:39:00.146 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:00.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:00.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:00.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:59 smithi121 ceph-mon[30729]: pgmap v337: 17 pgs: 17 active+clean; 113 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 876 KiB/s wr, 15 op/s 2024-09-11T03:39:00.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:59 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:00.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:59 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3203188928' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:39:00.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:38:59 smithi121 ceph-mon[30729]: osdmap e122: 12 total, 12 up, 12 in 2024-09-11T03:39:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:59 smithi175 ceph-mon[30894]: pgmap v337: 17 pgs: 17 active+clean; 113 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 876 KiB/s wr, 15 op/s 2024-09-11T03:39:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:59 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:59 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3203188928' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:39:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:38:59 smithi175 ceph-mon[30894]: osdmap e122: 12 total, 12 up, 12 in 2024-09-11T03:39:00.248 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:00.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:00.273 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:00.273 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:00.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:00.401 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:00.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:00.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:00.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:00.528 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:00.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:00.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:00.629 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T03:39:00.629 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:00.651 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:00.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:00.652 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:00.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:00.773 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:00.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:00.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:00.817 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:39:00.875 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:00.884 INFO:tasks.rados.rados.2.smithi175.stdout:183: done (1 left) 2024-09-11T03:39:00.885 INFO:tasks.rados.rados.2.smithi175.stdout:184: delete oid 1 current snap is 12 2024-09-11T03:39:00.885 INFO:tasks.rados.rados.2.smithi175.stdout:182: finishing write tid 1 to smithi17557870-18 2024-09-11T03:39:00.885 INFO:tasks.rados.rados.2.smithi175.stdout:182: finishing write tid 2 to smithi17557870-18 2024-09-11T03:39:00.885 INFO:tasks.rados.rados.2.smithi175.stdout:182: finishing write tid 3 to smithi17557870-18 2024-09-11T03:39:00.885 INFO:tasks.rados.rados.2.smithi175.stdout:182: finishing write tid 4 to smithi17557870-18 2024-09-11T03:39:00.885 INFO:tasks.rados.rados.2.smithi175.stdout:182: finishing write tid 5 to smithi17557870-18 2024-09-11T03:39:00.885 INFO:tasks.rados.rados.2.smithi175.stdout:182: oid 18 updating version 0 to 19 2024-09-11T03:39:00.885 INFO:tasks.rados.rados.2.smithi175.stdout:182: oid 18 updating version 19 to 21 2024-09-11T03:39:00.885 INFO:tasks.rados.rados.2.smithi175.stdout:182: oid 18 version 21 is already newer than 18 2024-09-11T03:39:00.885 INFO:tasks.rados.rados.2.smithi175.stdout:182: oid 18 version 21 is already newer than 20 2024-09-11T03:39:00.885 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 21 (ObjNum 93 snap 12 seq_num 93) dirty exists 2024-09-11T03:39:00.885 INFO:tasks.rados.rados.2.smithi175.stdout:182: left oid 18 (ObjNum 93 snap 12 seq_num 93) 2024-09-11T03:39:00.886 INFO:tasks.rados.rados.2.smithi175.stdout:182: done (1 left) 2024-09-11T03:39:00.886 INFO:tasks.rados.rados.2.smithi175.stdout:184: done (0 left) 2024-09-11T03:39:00.886 INFO:tasks.rados.rados.2.smithi175.stdout:185: read oid 14 snap -1 2024-09-11T03:39:00.887 INFO:tasks.rados.rados.2.smithi175.stdout:185: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-11T03:39:00.887 INFO:tasks.rados.rados.2.smithi175.stdout:186: snap_create 2024-09-11T03:39:00.894 INFO:tasks.rados.rados.2.smithi175.stdout:185: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-11T03:39:00.898 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:00.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:00.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:01.001 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:01.022 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:01.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:01.023 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:01.094 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:01.114 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:01.124 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:01.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:00 smithi084 ceph-mon[27254]: osdmap e123: 12 total, 12 up, 12 in 2024-09-11T03:39:01.145 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:01.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:01.146 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:01.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:00 smithi121 ceph-mon[30729]: osdmap e123: 12 total, 12 up, 12 in 2024-09-11T03:39:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:00 smithi175 ceph-mon[30894]: osdmap e123: 12 total, 12 up, 12 in 2024-09-11T03:39:01.247 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:01.272 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:01.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:01.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:01.374 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:01.395 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:01.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:01.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:01.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:01.523 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:01.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:01.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:01.607 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:39:01.608 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:39:01.614 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:01.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:01.631 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:39:01.631 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:39:01.650 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:01.650 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:39:01.651 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:39:01.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:01.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:01.666 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:39:01.667 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:39:01.684 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:39:01.684 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:39:01.700 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:39:01.700 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:39:01.717 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:39:01.717 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:39:01.734 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:39:01.734 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:39:01.751 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:39:01.751 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:39:01.753 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:01.768 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:39:01.768 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:39:01.776 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:01.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:01.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:01.784 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:39:01.785 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:39:01.801 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:39:01.801 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:39:01.813 INFO:tasks.rados.rados.2.smithi175.stdout:185: done (1 left) 2024-09-11T03:39:01.813 INFO:tasks.rados.rados.2.smithi175.stdout:186: done (0 left) 2024-09-11T03:39:01.813 INFO:tasks.rados.rados.2.smithi175.stdout:187: write (excl) oid 38 current snap is 13 2024-09-11T03:39:01.813 INFO:tasks.rados.rados.2.smithi175.stdout:187: seq_num 95 ranges {796665=544891,1759554=650689,2976243=620422} 2024-09-11T03:39:01.820 INFO:tasks.rados.rados.2.smithi175.stdout:187: writing smithi17557870-38 from 796665 to 1341556 tid 1 2024-09-11T03:39:01.826 INFO:tasks.rados.rados.2.smithi175.stdout:187: writing smithi17557870-38 from 1759554 to 2410243 tid 2 2024-09-11T03:39:01.833 INFO:tasks.rados.rados.2.smithi175.stdout:187: writing smithi17557870-38 from 2976243 to 3596665 tid 3 2024-09-11T03:39:01.833 INFO:tasks.rados.rados.2.smithi175.stdout:188: delete oid 6 current snap is 13 2024-09-11T03:39:01.833 INFO:tasks.rados.rados.2.smithi175.stdout:187: finishing write tid 1 to smithi17557870-38 2024-09-11T03:39:01.833 INFO:tasks.rados.rados.2.smithi175.stdout:187: finishing write tid 2 to smithi17557870-38 2024-09-11T03:39:01.835 INFO:tasks.rados.rados.2.smithi175.stdout:188: done (1 left) 2024-09-11T03:39:01.835 INFO:tasks.rados.rados.2.smithi175.stdout:189: snap_create 2024-09-11T03:39:01.835 INFO:tasks.rados.rados.2.smithi175.stdout:187: finishing write tid 3 to smithi17557870-38 2024-09-11T03:39:01.835 INFO:tasks.rados.rados.2.smithi175.stdout:187: finishing write tid 4 to smithi17557870-38 2024-09-11T03:39:01.836 INFO:tasks.rados.rados.2.smithi175.stdout:187: finishing write tid 5 to smithi17557870-38 2024-09-11T03:39:01.836 INFO:tasks.rados.rados.2.smithi175.stdout:187: oid 38 updating version 0 to 58 2024-09-11T03:39:01.836 INFO:tasks.rados.rados.2.smithi175.stdout:187: oid 38 updating version 58 to 59 2024-09-11T03:39:01.836 INFO:tasks.rados.rados.2.smithi175.stdout:187: oid 38 updating version 59 to 60 2024-09-11T03:39:01.836 INFO:tasks.rados.rados.2.smithi175.stdout:187: oid 38 version 60 is already newer than 57 2024-09-11T03:39:01.836 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 60 (ObjNum 95 snap 13 seq_num 95) dirty exists 2024-09-11T03:39:01.836 INFO:tasks.rados.rados.2.smithi175.stdout:187: left oid 38 (ObjNum 95 snap 13 seq_num 95) 2024-09-11T03:39:01.877 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:01.899 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:01.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:01.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:02.001 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:02.023 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:02.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:02.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:02.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:02.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:01 smithi084 ceph-mon[27254]: pgmap v340: 17 pgs: 17 active+clean; 113 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3 op/s 2024-09-11T03:39:02.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:01 smithi084 ceph-mon[27254]: osdmap e124: 12 total, 12 up, 12 in 2024-09-11T03:39:02.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:01 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:39:02.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:01 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:39:02.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2279597715' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:02.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:01 smithi084 ceph-mon[27254]: osdmap e125: 12 total, 12 up, 12 in 2024-09-11T03:39:02.151 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:02.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:02.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:01 smithi121 ceph-mon[30729]: pgmap v340: 17 pgs: 17 active+clean; 113 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3 op/s 2024-09-11T03:39:02.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:01 smithi121 ceph-mon[30729]: osdmap e124: 12 total, 12 up, 12 in 2024-09-11T03:39:02.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:01 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:39:02.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:01 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:39:02.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2279597715' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:02.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:01 smithi121 ceph-mon[30729]: osdmap e125: 12 total, 12 up, 12 in 2024-09-11T03:39:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:01 smithi175 ceph-mon[30894]: pgmap v340: 17 pgs: 17 active+clean; 113 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3 op/s 2024-09-11T03:39:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:01 smithi175 ceph-mon[30894]: osdmap e124: 12 total, 12 up, 12 in 2024-09-11T03:39:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:01 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:39:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:01 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:39:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2279597715' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:01 smithi175 ceph-mon[30894]: osdmap e125: 12 total, 12 up, 12 in 2024-09-11T03:39:02.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:02.278 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:02.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:02.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:02.380 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:02.403 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:02.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:02.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:02.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:02.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:02.533 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:02.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:02.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:02.660 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:02.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:02.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:02.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:02.766 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:39:02.766 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:39:02.766 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:39:02.766 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:39:02.766 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:39:02.787 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:02.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:02.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:02.814 INFO:tasks.rados.rados.2.smithi175.stdout:187: done (1 left) 2024-09-11T03:39:02.815 INFO:tasks.rados.rados.2.smithi175.stdout:189: done (0 left) 2024-09-11T03:39:02.815 INFO:tasks.rados.rados.2.smithi175.stdout:190: snap_create 2024-09-11T03:39:02.888 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:02.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:03.015 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:03.041 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:03.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:03.042 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:03.054 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:39:03.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:03.163 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:03.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:03.265 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:03.287 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:03.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:03.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:03.388 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:03.409 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:03.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:03.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:03.489 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:03.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:03.537 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:03.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:03.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:03.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:03.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:03.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:03.766 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:03.787 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:03.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:03.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:03.812 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2024-09-11T03:39:03.816 INFO:tasks.rados.rados.2.smithi175.stdout:190: done (0 left) 2024-09-11T03:39:03.817 INFO:tasks.rados.rados.2.smithi175.stdout:191: delete oid 34 current snap is 15 2024-09-11T03:39:03.819 INFO:tasks.rados.rados.2.smithi175.stdout:191: done (0 left) 2024-09-11T03:39:03.819 INFO:tasks.rados.rados.2.smithi175.stdout:192: write (excl) oid 42 current snap is 15 2024-09-11T03:39:03.819 INFO:tasks.rados.rados.2.smithi175.stdout:192: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2024-09-11T03:39:03.825 INFO:tasks.rados.rados.2.smithi175.stdout:192: writing smithi17557870-42 from 447086 to 1217724 tid 1 2024-09-11T03:39:03.834 INFO:tasks.rados.rados.2.smithi175.stdout:192: writing smithi17557870-42 from 1935000 to 2733016 tid 2 2024-09-11T03:39:03.839 INFO:tasks.rados.rados.2.smithi175.stdout:192: writing smithi17557870-42 from 3458512 to 3647086 tid 3 2024-09-11T03:39:03.839 INFO:tasks.rados.rados.2.smithi175.stdout:193: delete oid 14 current snap is 15 2024-09-11T03:39:03.839 INFO:tasks.rados.rados.2.smithi175.stdout:192: finishing write tid 1 to smithi17557870-42 2024-09-11T03:39:03.839 INFO:tasks.rados.rados.2.smithi175.stdout:192: finishing write tid 2 to smithi17557870-42 2024-09-11T03:39:03.841 INFO:tasks.rados.rados.2.smithi175.stdout:192: finishing write tid 3 to smithi17557870-42 2024-09-11T03:39:03.841 INFO:tasks.rados.rados.2.smithi175.stdout:192: finishing write tid 4 to smithi17557870-42 2024-09-11T03:39:03.841 INFO:tasks.rados.rados.2.smithi175.stdout:192: finishing write tid 5 to smithi17557870-42 2024-09-11T03:39:03.841 INFO:tasks.rados.rados.2.smithi175.stdout:192: oid 42 updating version 0 to 29 2024-09-11T03:39:03.841 INFO:tasks.rados.rados.2.smithi175.stdout:192: oid 42 updating version 29 to 30 2024-09-11T03:39:03.841 INFO:tasks.rados.rados.2.smithi175.stdout:192: oid 42 updating version 30 to 31 2024-09-11T03:39:03.841 INFO:tasks.rados.rados.2.smithi175.stdout:192: oid 42 version 31 is already newer than 28 2024-09-11T03:39:03.842 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 31 (ObjNum 98 snap 15 seq_num 98) dirty exists 2024-09-11T03:39:03.842 INFO:tasks.rados.rados.2.smithi175.stdout:192: left oid 42 (ObjNum 98 snap 15 seq_num 98) 2024-09-11T03:39:03.842 INFO:tasks.rados.rados.2.smithi175.stdout:192: done (1 left) 2024-09-11T03:39:03.842 INFO:tasks.rados.rados.2.smithi175.stdout:193: done (0 left) 2024-09-11T03:39:03.842 INFO:tasks.rados.rados.2.smithi175.stdout:194: snap_create 2024-09-11T03:39:03.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:03.915 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:03.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:04.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:04.041 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:04.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:04.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:04.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:03 smithi084 ceph-mon[27254]: pgmap v343: 17 pgs: 17 active+clean; 105 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-11T03:39:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:03 smithi084 ceph-mon[27254]: osdmap e126: 12 total, 12 up, 12 in 2024-09-11T03:39:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:03 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/73662322' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:39:04.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:04.169 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:04.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:04.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:03 smithi121 ceph-mon[30729]: pgmap v343: 17 pgs: 17 active+clean; 105 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-11T03:39:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:03 smithi121 ceph-mon[30729]: osdmap e126: 12 total, 12 up, 12 in 2024-09-11T03:39:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:03 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/73662322' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:39:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:03 smithi175 ceph-mon[30894]: pgmap v343: 17 pgs: 17 active+clean; 105 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-11T03:39:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:03 smithi175 ceph-mon[30894]: osdmap e126: 12 total, 12 up, 12 in 2024-09-11T03:39:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:03 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/73662322' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:39:04.270 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:04.292 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:04.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:04.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:04.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:04.419 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:04.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:04.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:04.501 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:39:04.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:04.546 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:04.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:04.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:04.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 8 1 2024-09-11T03:39:04.647 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:04.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:04.675 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:04.675 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:04.776 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:04.802 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:04.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:04.803 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:04.833 INFO:tasks.rados.rados.2.smithi175.stdout:194: done (0 left) 2024-09-11T03:39:04.833 INFO:tasks.rados.rados.2.smithi175.stdout:195: rollback oid 47 current snap is 16 2024-09-11T03:39:04.834 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 47 to 14 2024-09-11T03:39:04.834 INFO:tasks.rados.rados.2.smithi175.stdout:196: write (excl) oid 30 current snap is 16 2024-09-11T03:39:04.834 INFO:tasks.rados.rados.2.smithi175.stdout:196: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2024-09-11T03:39:04.839 INFO:tasks.rados.rados.2.smithi175.stdout:196: writing smithi17557870-30 from 480700 to 1118189 tid 1 2024-09-11T03:39:04.845 INFO:tasks.rados.rados.2.smithi175.stdout:196: writing smithi17557870-30 from 1651964 to 2053649 tid 2 2024-09-11T03:39:04.853 INFO:tasks.rados.rados.2.smithi175.stdout:196: writing smithi17557870-30 from 2713358 to 3483971 tid 3 2024-09-11T03:39:04.853 INFO:tasks.rados.rados.2.smithi175.stdout:197: read oid 23 snap -1 2024-09-11T03:39:04.853 INFO:tasks.rados.rados.2.smithi175.stdout:197: expect (ObjNum 64 snap 5 seq_num 64) 2024-09-11T03:39:04.853 INFO:tasks.rados.rados.2.smithi175.stdout:198: rollback oid 5 current snap is 16 2024-09-11T03:39:04.853 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 5 to 14 2024-09-11T03:39:04.853 INFO:tasks.rados.rados.2.smithi175.stdout:195: finishing rollback tid 0 to smithi17557870-47 2024-09-11T03:39:04.854 INFO:tasks.rados.rados.2.smithi175.stdout:195: finishing rollback tid 1 to smithi17557870-47 2024-09-11T03:39:04.854 INFO:tasks.rados.rados.2.smithi175.stdout:195: finishing rollback tid 2 to smithi17557870-47 2024-09-11T03:39:04.854 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 54 (ObjNum 91 snap 12 seq_num 91) dirty exists 2024-09-11T03:39:04.854 INFO:tasks.rados.rados.2.smithi175.stdout:196: finishing write tid 1 to smithi17557870-30 2024-09-11T03:39:04.854 INFO:tasks.rados.rados.2.smithi175.stdout:196: finishing write tid 2 to smithi17557870-30 2024-09-11T03:39:04.854 INFO:tasks.rados.rados.2.smithi175.stdout:195: done (3 left) 2024-09-11T03:39:04.854 INFO:tasks.rados.rados.2.smithi175.stdout:199: read oid 13 snap -1 2024-09-11T03:39:04.854 INFO:tasks.rados.rados.2.smithi175.stdout:199: expect (ObjNum 76 snap 7 seq_num 76) 2024-09-11T03:39:04.854 INFO:tasks.rados.rados.2.smithi175.stdout:200: write oid 38 current snap is 16 2024-09-11T03:39:04.854 INFO:tasks.rados.rados.2.smithi175.stdout:200: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2024-09-11T03:39:04.859 INFO:tasks.rados.rados.2.smithi175.stdout:200: writing smithi17557870-38 from 497507 to 1210245 tid 1 2024-09-11T03:39:04.866 INFO:tasks.rados.rados.2.smithi175.stdout:200: writing smithi17557870-38 from 1710446 to 2372142 tid 2 2024-09-11T03:39:04.874 INFO:tasks.rados.rados.2.smithi175.stdout:200: writing smithi17557870-38 from 3140781 to 3697507 tid 3 2024-09-11T03:39:04.874 INFO:tasks.rados.rados.2.smithi175.stdout:201: delete oid 43 current snap is 16 2024-09-11T03:39:04.875 INFO:tasks.rados.rados.2.smithi175.stdout:198: finishing rollback tid 0 to smithi17557870-5 2024-09-11T03:39:04.875 INFO:tasks.rados.rados.2.smithi175.stdout:196: finishing write tid 3 to smithi17557870-30 2024-09-11T03:39:04.875 INFO:tasks.rados.rados.2.smithi175.stdout:198: finishing rollback tid 1 to smithi17557870-5 2024-09-11T03:39:04.875 INFO:tasks.rados.rados.2.smithi175.stdout:196: finishing write tid 4 to smithi17557870-30 2024-09-11T03:39:04.875 INFO:tasks.rados.rados.2.smithi175.stdout:198: finishing rollback tid 2 to smithi17557870-5 2024-09-11T03:39:04.875 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 12 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-11T03:39:04.875 INFO:tasks.rados.rados.2.smithi175.stdout:196: finishing write tid 5 to smithi17557870-30 2024-09-11T03:39:04.875 INFO:tasks.rados.rados.2.smithi175.stdout:196: oid 30 updating version 0 to 34 2024-09-11T03:39:04.875 INFO:tasks.rados.rados.2.smithi175.stdout:196: oid 30 version 34 is already newer than 33 2024-09-11T03:39:04.875 INFO:tasks.rados.rados.2.smithi175.stdout:196: oid 30 updating version 34 to 36 2024-09-11T03:39:04.875 INFO:tasks.rados.rados.2.smithi175.stdout:196: oid 30 version 36 is already newer than 35 2024-09-11T03:39:04.875 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 36 (ObjNum 100 snap 16 seq_num 100) dirty exists 2024-09-11T03:39:04.875 INFO:tasks.rados.rados.2.smithi175.stdout:196: left oid 30 (ObjNum 100 snap 16 seq_num 100) 2024-09-11T03:39:04.875 INFO:tasks.rados.rados.2.smithi175.stdout:199: expect (ObjNum 76 snap 7 seq_num 76) 2024-09-11T03:39:04.904 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:04.925 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:04.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:04.958 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T03:39:05.027 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:05.044 INFO:tasks.rados.rados.2.smithi175.stdout:200: finishing write tid 1 to smithi17557870-38 2024-09-11T03:39:05.044 INFO:tasks.rados.rados.2.smithi175.stdout:197: expect (ObjNum 64 snap 5 seq_num 64) 2024-09-11T03:39:05.048 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:05.048 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:39:05.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:05.049 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:04 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/73662322' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:39:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:04 smithi084 ceph-mon[27254]: osdmap e127: 12 total, 12 up, 12 in 2024-09-11T03:39:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:04 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1464341355' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:05.150 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:05.171 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:05.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:04 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/73662322' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:39:05.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:04 smithi121 ceph-mon[30729]: osdmap e127: 12 total, 12 up, 12 in 2024-09-11T03:39:05.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:04 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1464341355' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:05.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:05.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:05.184 INFO:tasks.rados.rados.2.smithi175.stdout:200: finishing write tid 2 to smithi17557870-38 2024-09-11T03:39:05.184 INFO:tasks.rados.rados.2.smithi175.stdout:200: finishing write tid 3 to smithi17557870-38 2024-09-11T03:39:05.184 INFO:tasks.rados.rados.2.smithi175.stdout:196: done (5 left) 2024-09-11T03:39:05.184 INFO:tasks.rados.rados.2.smithi175.stdout:197: done (4 left) 2024-09-11T03:39:05.184 INFO:tasks.rados.rados.2.smithi175.stdout:198: done (3 left) 2024-09-11T03:39:05.184 INFO:tasks.rados.rados.2.smithi175.stdout:199: done (2 left) 2024-09-11T03:39:05.184 INFO:tasks.rados.rados.2.smithi175.stdout:201: done (1 left) 2024-09-11T03:39:05.185 INFO:tasks.rados.rados.2.smithi175.stdout:202: snap_remove snap 14 2024-09-11T03:39:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:04 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/73662322' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:39:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:04 smithi175 ceph-mon[30894]: osdmap e127: 12 total, 12 up, 12 in 2024-09-11T03:39:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:04 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1464341355' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:05.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:05.300 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:05.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:05.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:05.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:05.428 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:05.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:05.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:05.556 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:05.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:05.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:05.658 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:05.681 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:05.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:05.739 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:05.783 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:05.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:05.809 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:05.809 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:05.837 INFO:tasks.rados.rados.2.smithi175.stdout:202: done (1 left) 2024-09-11T03:39:05.837 INFO:tasks.rados.rados.2.smithi175.stdout:203: read oid 12 snap 12 2024-09-11T03:39:05.837 INFO:tasks.rados.rados.2.smithi175.stdout:203: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-11T03:39:05.837 INFO:tasks.rados.rados.2.smithi175.stdout:204: copy_from oid 16 from oid 42 current snap is 16 2024-09-11T03:39:05.837 INFO:tasks.rados.rados.2.smithi175.stdout:200: finishing write tid 4 to smithi17557870-38 2024-09-11T03:39:05.837 INFO:tasks.rados.rados.2.smithi175.stdout:200: finishing write tid 5 to smithi17557870-38 2024-09-11T03:39:05.837 INFO:tasks.rados.rados.2.smithi175.stdout:200: oid 38 updating version 0 to 62 2024-09-11T03:39:05.837 INFO:tasks.rados.rados.2.smithi175.stdout:200: oid 38 updating version 62 to 63 2024-09-11T03:39:05.838 INFO:tasks.rados.rados.2.smithi175.stdout:200: oid 38 updating version 63 to 64 2024-09-11T03:39:05.838 INFO:tasks.rados.rados.2.smithi175.stdout:200: oid 38 updating version 64 to 65 2024-09-11T03:39:05.838 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 65 (ObjNum 101 snap 16 seq_num 101) dirty exists 2024-09-11T03:39:05.838 INFO:tasks.rados.rados.2.smithi175.stdout:200: left oid 38 (ObjNum 101 snap 16 seq_num 101) 2024-09-11T03:39:05.838 INFO:tasks.rados.rados.2.smithi175.stdout:200: done (2 left) 2024-09-11T03:39:05.838 INFO:tasks.rados.rados.2.smithi175.stdout:205: read oid 7 snap -1 2024-09-11T03:39:05.838 INFO:tasks.rados.rados.2.smithi175.stdout:205: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-11T03:39:05.838 INFO:tasks.rados.rados.2.smithi175.stdout:206: rollback oid 24 current snap is 16 2024-09-11T03:39:05.838 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 13 2024-09-11T03:39:05.838 INFO:tasks.rados.rados.2.smithi175.stdout:207: write oid 4 current snap is 16 2024-09-11T03:39:05.838 INFO:tasks.rados.rados.2.smithi175.stdout:207: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2024-09-11T03:39:05.843 INFO:tasks.rados.rados.2.smithi175.stdout:207: writing smithi17557870-4 from 531121 to 994357 tid 1 2024-09-11T03:39:05.852 INFO:tasks.rados.rados.2.smithi175.stdout:207: writing smithi17557870-4 from 1711057 to 2376422 tid 2 2024-09-11T03:39:05.859 INFO:tasks.rados.rados.2.smithi175.stdout:207: writing smithi17557870-4 from 3079274 to 3731121 tid 3 2024-09-11T03:39:05.860 INFO:tasks.rados.rados.2.smithi175.stdout:208: snap_remove snap 15 2024-09-11T03:39:05.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:05.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:05 smithi084 ceph-mon[27254]: pgmap v346: 17 pgs: 17 active+clean; 107 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 7.1 MiB/s wr, 39 op/s 2024-09-11T03:39:05.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:05 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:05.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:05 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1464341355' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:05.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:05 smithi084 ceph-mon[27254]: osdmap e128: 12 total, 12 up, 12 in 2024-09-11T03:39:05.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:05.936 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:05.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:06.016 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:06.038 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:06.059 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:06.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:06.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:06.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:06.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:05 smithi121 ceph-mon[30729]: pgmap v346: 17 pgs: 17 active+clean; 107 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 7.1 MiB/s wr, 39 op/s 2024-09-11T03:39:06.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:05 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:06.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:05 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1464341355' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:06.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:05 smithi121 ceph-mon[30729]: osdmap e128: 12 total, 12 up, 12 in 2024-09-11T03:39:06.182 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:06.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:06.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:05 smithi175 ceph-mon[30894]: pgmap v346: 17 pgs: 17 active+clean; 107 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 7.1 MiB/s wr, 39 op/s 2024-09-11T03:39:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:05 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:05 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1464341355' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:05 smithi175 ceph-mon[30894]: osdmap e128: 12 total, 12 up, 12 in 2024-09-11T03:39:06.284 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:06.310 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:06.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:06.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:06.412 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:06.437 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:06.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:06.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:06.505 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:06.539 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:06.564 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:06.566 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:06.667 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:06.692 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:06.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:06.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:06.701 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:39:06.702 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:39:06.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:39:06.702 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:39:06.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.702 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:216 2.d e102/95 2:b6a25a79:::smithi17557870-47:head v 102'13, mlcod=100'11)", 2024-09-11T03:39:06.702 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:38.941442+0000", 2024-09-11T03:39:06.702 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 27.759940877999998, 2024-09-11T03:39:06.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002147375, 2024-09-11T03:39:06.702 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.703 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.703 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.703 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.941442+0000", 2024-09-11T03:39:06.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.941442+0000", 2024-09-11T03:39:06.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.941442+0000", 2024-09-11T03:39:06.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4499999999999998e-07 2024-09-11T03:39:06.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.942764+0000", 2024-09-11T03:39:06.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0013217330000000001 2024-09-11T03:39:06.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.942782+0000", 2024-09-11T03:39:06.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.842e-05 2024-09-11T03:39:06.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.942789+0000", 2024-09-11T03:39:06.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7349999999999999e-06 2024-09-11T03:39:06.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.942840+0000", 2024-09-11T03:39:06.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1484999999999999e-05 2024-09-11T03:39:06.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.942858+0000", 2024-09-11T03:39:06.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7085e-05 2024-09-11T03:39:06.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.943558+0000", 2024-09-11T03:39:06.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070009 2024-09-11T03:39:06.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.943589+0000", 2024-09-11T03:39:06.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1482000000000001e-05 2024-09-11T03:39:06.707 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.707 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.708 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.708 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:217 2.d e102/95 2:b6a25a79:::smithi17557870-47:head v 102'14, mlcod=100'12)", 2024-09-11T03:39:06.708 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:38.945688+0000", 2024-09-11T03:39:06.708 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 27.755694849000001, 2024-09-11T03:39:06.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014121120000000001, 2024-09-11T03:39:06.708 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.708 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.708 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.709 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.945688+0000", 2024-09-11T03:39:06.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.945688+0000", 2024-09-11T03:39:06.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.945688+0000", 2024-09-11T03:39:06.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7700000000000004e-07 2024-09-11T03:39:06.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.946399+0000", 2024-09-11T03:39:06.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071032199999999999 2024-09-11T03:39:06.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.946408+0000", 2024-09-11T03:39:06.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6299999999999993e-06 2024-09-11T03:39:06.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.946414+0000", 2024-09-11T03:39:06.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.288e-06 2024-09-11T03:39:06.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.946457+0000", 2024-09-11T03:39:06.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2814e-05 2024-09-11T03:39:06.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.946471+0000", 2024-09-11T03:39:06.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4545000000000001e-05 2024-09-11T03:39:06.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.947071+0000", 2024-09-11T03:39:06.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00060026000000000003 2024-09-11T03:39:06.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:38.947100+0000", 2024-09-11T03:39:06.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8575999999999999e-05 2024-09-11T03:39:06.713 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.713 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.713 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.714 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:228 2.9 e105/95 2:95fa802a:::smithi17557870-22:head v 105'20, mlcod=100'17)", 2024-09-11T03:39:06.714 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:41.966714+0000", 2024-09-11T03:39:06.714 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 24.734668781, 2024-09-11T03:39:06.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001811712, 2024-09-11T03:39:06.714 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.714 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.714 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.714 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.966714+0000", 2024-09-11T03:39:06.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.966714+0000", 2024-09-11T03:39:06.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.966714+0000", 2024-09-11T03:39:06.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2399999999999999e-07 2024-09-11T03:39:06.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.967591+0000", 2024-09-11T03:39:06.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00087697199999999995 2024-09-11T03:39:06.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.967601+0000", 2024-09-11T03:39:06.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0152e-05 2024-09-11T03:39:06.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.967610+0000", 2024-09-11T03:39:06.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4780000000000008e-06 2024-09-11T03:39:06.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.967653+0000", 2024-09-11T03:39:06.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2753000000000002e-05 2024-09-11T03:39:06.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.967667+0000", 2024-09-11T03:39:06.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4876e-05 2024-09-11T03:39:06.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.968495+0000", 2024-09-11T03:39:06.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00082775800000000005 2024-09-11T03:39:06.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.968526+0000", 2024-09-11T03:39:06.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0399000000000002e-05 2024-09-11T03:39:06.718 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.718 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.719 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.719 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:229 2.9 e105/95 2:95fa802a:::smithi17557870-22:head v 105'21, mlcod=100'17)", 2024-09-11T03:39:06.719 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:41.972251+0000", 2024-09-11T03:39:06.719 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 24.729132085, 2024-09-11T03:39:06.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000975657, 2024-09-11T03:39:06.719 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.719 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.719 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.720 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.972251+0000", 2024-09-11T03:39:06.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.972251+0000", 2024-09-11T03:39:06.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.972252+0000", 2024-09-11T03:39:06.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1370000000000001e-06 2024-09-11T03:39:06.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.972388+0000", 2024-09-11T03:39:06.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013677599999999999 2024-09-11T03:39:06.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.972400+0000", 2024-09-11T03:39:06.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1087000000000001e-05 2024-09-11T03:39:06.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.972406+0000", 2024-09-11T03:39:06.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2280000000000003e-06 2024-09-11T03:39:06.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.972445+0000", 2024-09-11T03:39:06.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9499000000000002e-05 2024-09-11T03:39:06.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.972460+0000", 2024-09-11T03:39:06.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4379000000000001e-05 2024-09-11T03:39:06.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.973192+0000", 2024-09-11T03:39:06.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00073276799999999996 2024-09-11T03:39:06.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.973226+0000", 2024-09-11T03:39:06.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3782999999999998e-05 2024-09-11T03:39:06.724 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.724 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.724 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.724 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:243 2.d e107/95 2:b7bdfe52:::smithi17557870-10:head v 107'17, mlcod=102'14)", 2024-09-11T03:39:06.724 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:43.974741+0000", 2024-09-11T03:39:06.724 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 22.726641194999999, 2024-09-11T03:39:06.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031676069999999998, 2024-09-11T03:39:06.724 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.724 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.725 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.725 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.974741+0000", 2024-09-11T03:39:06.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.974741+0000", 2024-09-11T03:39:06.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.974742+0000", 2024-09-11T03:39:06.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9299999999999999e-07 2024-09-11T03:39:06.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.976618+0000", 2024-09-11T03:39:06.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018759779999999999 2024-09-11T03:39:06.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.976632+0000", 2024-09-11T03:39:06.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4092e-05 2024-09-11T03:39:06.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.976642+0000", 2024-09-11T03:39:06.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8030000000000008e-06 2024-09-11T03:39:06.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.976724+0000", 2024-09-11T03:39:06.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2281999999999996e-05 2024-09-11T03:39:06.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.976748+0000", 2024-09-11T03:39:06.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3737999999999999e-05 2024-09-11T03:39:06.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.977866+0000", 2024-09-11T03:39:06.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011179989999999999 2024-09-11T03:39:06.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.977909+0000", 2024-09-11T03:39:06.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3421999999999999e-05 2024-09-11T03:39:06.728 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.728 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.729 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.729 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:244 2.d e107/95 2:b7bdfe52:::smithi17557870-10:head v 107'18, mlcod=102'15)", 2024-09-11T03:39:06.729 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:43.978954+0000", 2024-09-11T03:39:06.729 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 22.722428141000002, 2024-09-11T03:39:06.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020137340000000001, 2024-09-11T03:39:06.729 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.729 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.729 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.729 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.978954+0000", 2024-09-11T03:39:06.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.978954+0000", 2024-09-11T03:39:06.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.978956+0000", 2024-09-11T03:39:06.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5540000000000001e-06 2024-09-11T03:39:06.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.979944+0000", 2024-09-11T03:39:06.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00098824799999999995 2024-09-11T03:39:06.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.979970+0000", 2024-09-11T03:39:06.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5704e-05 2024-09-11T03:39:06.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.979983+0000", 2024-09-11T03:39:06.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2683999999999999e-05 2024-09-11T03:39:06.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.980056+0000", 2024-09-11T03:39:06.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3418999999999996e-05 2024-09-11T03:39:06.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.980075+0000", 2024-09-11T03:39:06.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9279999999999998e-05 2024-09-11T03:39:06.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.980915+0000", 2024-09-11T03:39:06.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00083983699999999996 2024-09-11T03:39:06.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:43.980968+0000", 2024-09-11T03:39:06.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3007999999999998e-05 2024-09-11T03:39:06.733 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.733 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.733 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.733 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:341 2.9 e113/95 2:9718db5a:::smithi17557870-3:head v 113'37, mlcod=109'33)", 2024-09-11T03:39:06.733 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:49.833457+0000", 2024-09-11T03:39:06.733 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 16.867925934999999, 2024-09-11T03:39:06.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043446580000000004, 2024-09-11T03:39:06.734 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.734 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.734 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.734 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.833457+0000", 2024-09-11T03:39:06.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.833457+0000", 2024-09-11T03:39:06.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.833457+0000", 2024-09-11T03:39:06.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2699999999999999e-07 2024-09-11T03:39:06.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.835023+0000", 2024-09-11T03:39:06.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015654740000000001 2024-09-11T03:39:06.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.835055+0000", 2024-09-11T03:39:06.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1810999999999997e-05 2024-09-11T03:39:06.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.835073+0000", 2024-09-11T03:39:06.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8783e-05 2024-09-11T03:39:06.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.835176+0000", 2024-09-11T03:39:06.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000103163 2024-09-11T03:39:06.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.835208+0000", 2024-09-11T03:39:06.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1791000000000001e-05 2024-09-11T03:39:06.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.837739+0000", 2024-09-11T03:39:06.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025311309999999998 2024-09-11T03:39:06.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.837801+0000", 2024-09-11T03:39:06.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1977999999999995e-05 2024-09-11T03:39:06.738 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.738 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.738 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.739 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:350 2.6 e113/95 2:6dbd1e81:::smithi17557870-7:head v 113'8, mlcod=100'5)", 2024-09-11T03:39:06.739 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:49.847471+0000", 2024-09-11T03:39:06.739 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 16.853911243999999, 2024-09-11T03:39:06.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015724929999999999, 2024-09-11T03:39:06.739 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.739 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.739 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.740 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.847471+0000", 2024-09-11T03:39:06.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.847471+0000", 2024-09-11T03:39:06.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.847473+0000", 2024-09-11T03:39:06.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.719e-06 2024-09-11T03:39:06.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.847889+0000", 2024-09-11T03:39:06.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00041619299999999998 2024-09-11T03:39:06.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.847905+0000", 2024-09-11T03:39:06.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5624000000000002e-05 2024-09-11T03:39:06.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.847911+0000", 2024-09-11T03:39:06.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3899999999999998e-06 2024-09-11T03:39:06.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.847971+0000", 2024-09-11T03:39:06.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9271000000000002e-05 2024-09-11T03:39:06.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.847994+0000", 2024-09-11T03:39:06.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3594000000000001e-05 2024-09-11T03:39:06.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.848994+0000", 2024-09-11T03:39:06.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099986200000000002 2024-09-11T03:39:06.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.849044+0000", 2024-09-11T03:39:06.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9839999999999997e-05 2024-09-11T03:39:06.743 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.743 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.744 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.744 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:351 2.6 e113/95 2:6dbd1e81:::smithi17557870-7:head v 113'9, mlcod=100'6)", 2024-09-11T03:39:06.744 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:49.855158+0000", 2024-09-11T03:39:06.744 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 16.846224795000001, 2024-09-11T03:39:06.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001909549, 2024-09-11T03:39:06.744 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.744 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.744 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.744 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.855158+0000", 2024-09-11T03:39:06.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.855158+0000", 2024-09-11T03:39:06.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.855159+0000", 2024-09-11T03:39:06.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5990000000000001e-06 2024-09-11T03:39:06.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.855796+0000", 2024-09-11T03:39:06.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00063699899999999996 2024-09-11T03:39:06.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.855818+0000", 2024-09-11T03:39:06.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1413999999999999e-05 2024-09-11T03:39:06.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.855829+0000", 2024-09-11T03:39:06.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1072e-05 2024-09-11T03:39:06.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.855891+0000", 2024-09-11T03:39:06.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2526000000000004e-05 2024-09-11T03:39:06.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.855924+0000", 2024-09-11T03:39:06.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2061999999999999e-05 2024-09-11T03:39:06.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.857020+0000", 2024-09-11T03:39:06.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001096938 2024-09-11T03:39:06.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.857067+0000", 2024-09-11T03:39:06.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6938999999999999e-05 2024-09-11T03:39:06.749 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.749 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.749 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.749 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.1.0:143 2.9 e114/95 2:9718db5a:::smithi17557870-3:8 v 114'39, mlcod=109'35)", 2024-09-11T03:39:06.749 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:50.828000+0000", 2024-09-11T03:39:06.749 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 15.873382251000001, 2024-09-11T03:39:06.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040361319999999996, 2024-09-11T03:39:06.750 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.750 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.750 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.750 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828000+0000", 2024-09-11T03:39:06.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828000+0000", 2024-09-11T03:39:06.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828001+0000", 2024-09-11T03:39:06.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3799999999999998e-07 2024-09-11T03:39:06.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828007+0000", 2024-09-11T03:39:06.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1099999999999999e-06 2024-09-11T03:39:06.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828031+0000", 2024-09-11T03:39:06.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3938999999999999e-05 2024-09-11T03:39:06.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828041+0000", 2024-09-11T03:39:06.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0132999999999999e-05 2024-09-11T03:39:06.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828125+0000", 2024-09-11T03:39:06.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4591000000000004e-05 2024-09-11T03:39:06.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828148+0000", 2024-09-11T03:39:06.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2152999999999999e-05 2024-09-11T03:39:06.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.831988+0000", 2024-09-11T03:39:06.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0038402789999999998 2024-09-11T03:39:06.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832037+0000", 2024-09-11T03:39:06.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8588999999999998e-05 2024-09-11T03:39:06.754 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.755 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.755 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.755 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.1.0:144 2.6 e114/95 2:6dbd1e81:::smithi17557870-7:8 v 114'13, mlcod=113'10)", 2024-09-11T03:39:06.755 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:50.828166+0000", 2024-09-11T03:39:06.755 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 15.873216971, 2024-09-11T03:39:06.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039242779999999998, 2024-09-11T03:39:06.755 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.755 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.755 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.755 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828166+0000", 2024-09-11T03:39:06.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828166+0000", 2024-09-11T03:39:06.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828166+0000", 2024-09-11T03:39:06.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8899999999999999e-07 2024-09-11T03:39:06.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828171+0000", 2024-09-11T03:39:06.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7940000000000002e-06 2024-09-11T03:39:06.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828180+0000", 2024-09-11T03:39:06.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9570000000000008e-06 2024-09-11T03:39:06.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828187+0000", 2024-09-11T03:39:06.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0489999999999998e-06 2024-09-11T03:39:06.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828248+0000", 2024-09-11T03:39:06.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0785999999999997e-05 2024-09-11T03:39:06.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828270+0000", 2024-09-11T03:39:06.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2422999999999999e-05 2024-09-11T03:39:06.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832047+0000", 2024-09-11T03:39:06.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037765960000000001 2024-09-11T03:39:06.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832090+0000", 2024-09-11T03:39:06.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3084000000000003e-05 2024-09-11T03:39:06.759 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.759 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.759 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.759 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:357 2.e e114/95 2:750de3b7:::smithi17557870-8:head v 114'29, mlcod=109'26)", 2024-09-11T03:39:06.759 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:50.836742+0000", 2024-09-11T03:39:06.759 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 15.864640558, 2024-09-11T03:39:06.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031412340000000001, 2024-09-11T03:39:06.760 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.760 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.760 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.760 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.836742+0000", 2024-09-11T03:39:06.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.836742+0000", 2024-09-11T03:39:06.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.836742+0000", 2024-09-11T03:39:06.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.99e-07 2024-09-11T03:39:06.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.838471+0000", 2024-09-11T03:39:06.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001728446 2024-09-11T03:39:06.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.838490+0000", 2024-09-11T03:39:06.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8924999999999999e-05 2024-09-11T03:39:06.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.838499+0000", 2024-09-11T03:39:06.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5459999999999997e-06 2024-09-11T03:39:06.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.838571+0000", 2024-09-11T03:39:06.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1947999999999996e-05 2024-09-11T03:39:06.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.838595+0000", 2024-09-11T03:39:06.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3841000000000001e-05 2024-09-11T03:39:06.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.839851+0000", 2024-09-11T03:39:06.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001256028 2024-09-11T03:39:06.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.839883+0000", 2024-09-11T03:39:06.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2301000000000001e-05 2024-09-11T03:39:06.764 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.764 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.764 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.764 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:419 2.6 e117/95 2:6e06b7a8:::smithi17557870-19:head v 117'15, mlcod=113'11)", 2024-09-11T03:39:06.764 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:53.834509+0000", 2024-09-11T03:39:06.764 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 12.866873285, 2024-09-11T03:39:06.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001473264, 2024-09-11T03:39:06.764 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.764 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.764 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.764 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:53.834509+0000", 2024-09-11T03:39:06.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:53.834509+0000", 2024-09-11T03:39:06.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:53.834510+0000", 2024-09-11T03:39:06.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0800000000000004e-07 2024-09-11T03:39:06.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:53.834540+0000", 2024-09-11T03:39:06.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9830999999999999e-05 2024-09-11T03:39:06.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:53.834674+0000", 2024-09-11T03:39:06.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000134179 2024-09-11T03:39:06.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:53.834858+0000", 2024-09-11T03:39:06.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018399700000000001 2024-09-11T03:39:06.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:53.834919+0000", 2024-09-11T03:39:06.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1304999999999997e-05 2024-09-11T03:39:06.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:53.834939+0000", 2024-09-11T03:39:06.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0055e-05 2024-09-11T03:39:06.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:53.835925+0000", 2024-09-11T03:39:06.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00098535300000000001 2024-09-11T03:39:06.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:53.835983+0000", 2024-09-11T03:39:06.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7836e-05 2024-09-11T03:39:06.768 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.768 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.768 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.768 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:428 2.9 e119/95 2:95fa802a:::smithi17557870-22:head v 119'41, mlcod=113'37)", 2024-09-11T03:39:06.768 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:55.868208+0000", 2024-09-11T03:39:06.769 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 10.833174488999999, 2024-09-11T03:39:06.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0032435279999999999, 2024-09-11T03:39:06.769 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.769 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.769 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.769 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.868208+0000", 2024-09-11T03:39:06.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.868208+0000", 2024-09-11T03:39:06.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.868209+0000", 2024-09-11T03:39:06.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9100000000000003e-07 2024-09-11T03:39:06.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.869644+0000", 2024-09-11T03:39:06.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014347540000000001 2024-09-11T03:39:06.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.869666+0000", 2024-09-11T03:39:06.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2110999999999999e-05 2024-09-11T03:39:06.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.869677+0000", 2024-09-11T03:39:06.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0983e-05 2024-09-11T03:39:06.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.869790+0000", 2024-09-11T03:39:06.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011310400000000001 2024-09-11T03:39:06.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.869843+0000", 2024-09-11T03:39:06.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2951999999999997e-05 2024-09-11T03:39:06.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.871414+0000", 2024-09-11T03:39:06.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015717369999999999 2024-09-11T03:39:06.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.871452+0000", 2024-09-11T03:39:06.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7196000000000001e-05 2024-09-11T03:39:06.773 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.773 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.773 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.773 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:559 2.e e127/95 2:7b0c3397:::smithi17557870-14:head v 127'31, mlcod=110'28)", 2024-09-11T03:39:06.773 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:03.842322+0000", 2024-09-11T03:39:06.773 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.859061031, 2024-09-11T03:39:06.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011938529999999999, 2024-09-11T03:39:06.774 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.774 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.774 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.774 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.842322+0000", 2024-09-11T03:39:06.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.842322+0000", 2024-09-11T03:39:06.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.842322+0000", 2024-09-11T03:39:06.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4400000000000001e-07 2024-09-11T03:39:06.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.842333+0000", 2024-09-11T03:39:06.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1075e-05 2024-09-11T03:39:06.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.842357+0000", 2024-09-11T03:39:06.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4020999999999999e-05 2024-09-11T03:39:06.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.842366+0000", 2024-09-11T03:39:06.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5909999999999996e-06 2024-09-11T03:39:06.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.842473+0000", 2024-09-11T03:39:06.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010723 2024-09-11T03:39:06.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.842494+0000", 2024-09-11T03:39:06.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1307000000000001e-05 2024-09-11T03:39:06.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.843476+0000", 2024-09-11T03:39:06.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00098162599999999998 2024-09-11T03:39:06.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.843515+0000", 2024-09-11T03:39:06.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9659000000000001e-05 2024-09-11T03:39:06.779 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.779 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.779 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.779 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:564 2.e e128/95 2:77dadb97:::smithi17557870-30:head v 128'33, mlcod=114'29)", 2024-09-11T03:39:06.779 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:04.843631+0000", 2024-09-11T03:39:06.779 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.8577511330000001, 2024-09-11T03:39:06.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001534491, 2024-09-11T03:39:06.780 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.780 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.780 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.780 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.843631+0000", 2024-09-11T03:39:06.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.843631+0000", 2024-09-11T03:39:06.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.843632+0000", 2024-09-11T03:39:06.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2099999999999996e-07 2024-09-11T03:39:06.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.844098+0000", 2024-09-11T03:39:06.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00046579800000000001 2024-09-11T03:39:06.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.844110+0000", 2024-09-11T03:39:06.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2194e-05 2024-09-11T03:39:06.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.844118+0000", 2024-09-11T03:39:06.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5229999999999998e-06 2024-09-11T03:39:06.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.844182+0000", 2024-09-11T03:39:06.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4777000000000007e-05 2024-09-11T03:39:06.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.844203+0000", 2024-09-11T03:39:06.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0879000000000001e-05 2024-09-11T03:39:06.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.845127+0000", 2024-09-11T03:39:06.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00092401600000000005 2024-09-11T03:39:06.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.845166+0000", 2024-09-11T03:39:06.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8683000000000002e-05 2024-09-11T03:39:06.786 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.786 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.786 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.786 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:566 2.e e128/95 2:77dadb97:::smithi17557870-30:head v 128'35, mlcod=128'33)", 2024-09-11T03:39:06.786 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:04.855393+0000", 2024-09-11T03:39:06.786 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.845989828, 2024-09-11T03:39:06.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018870670000000001, 2024-09-11T03:39:06.786 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.786 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.787 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.787 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.855393+0000", 2024-09-11T03:39:06.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.855393+0000", 2024-09-11T03:39:06.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.855395+0000", 2024-09-11T03:39:06.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9599999999999999e-06 2024-09-11T03:39:06.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.855892+0000", 2024-09-11T03:39:06.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00049756900000000001 2024-09-11T03:39:06.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.855914+0000", 2024-09-11T03:39:06.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1403000000000002e-05 2024-09-11T03:39:06.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.855923+0000", 2024-09-11T03:39:06.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4040000000000001e-06 2024-09-11T03:39:06.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.855988+0000", 2024-09-11T03:39:06.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4832999999999994e-05 2024-09-11T03:39:06.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.856010+0000", 2024-09-11T03:39:06.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1719999999999999e-05 2024-09-11T03:39:06.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.857247+0000", 2024-09-11T03:39:06.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012369709999999999 2024-09-11T03:39:06.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.857280+0000", 2024-09-11T03:39:06.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3207000000000002e-05 2024-09-11T03:39:06.790 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.790 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.791 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.791 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:572 2.1 e128/95 2:8638b65e:::smithi17557870-5:head v 128'6, mlcod=100'3)", 2024-09-11T03:39:06.791 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:04.855949+0000", 2024-09-11T03:39:06.791 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.8454337409999999, 2024-09-11T03:39:06.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001259271, 2024-09-11T03:39:06.791 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.791 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.791 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.791 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.855949+0000", 2024-09-11T03:39:06.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.855949+0000", 2024-09-11T03:39:06.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.855949+0000", 2024-09-11T03:39:06.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.68e-07 2024-09-11T03:39:06.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.855956+0000", 2024-09-11T03:39:06.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.161e-06 2024-09-11T03:39:06.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.855963+0000", 2024-09-11T03:39:06.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0219999999999999e-06 2024-09-11T03:39:06.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.855968+0000", 2024-09-11T03:39:06.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2830000000000001e-06 2024-09-11T03:39:06.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.856025+0000", 2024-09-11T03:39:06.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6879999999999998e-05 2024-09-11T03:39:06.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.856043+0000", 2024-09-11T03:39:06.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7997999999999999e-05 2024-09-11T03:39:06.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.857155+0000", 2024-09-11T03:39:06.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001112086 2024-09-11T03:39:06.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.857208+0000", 2024-09-11T03:39:06.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3273000000000003e-05 2024-09-11T03:39:06.795 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.795 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.795 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.795 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:573 2.1 e128/95 2:8638b65e:::smithi17557870-5:head v 128'7, mlcod=100'3)", 2024-09-11T03:39:06.795 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:04.856269+0000", 2024-09-11T03:39:06.796 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.8451137580000001, 2024-09-11T03:39:06.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010272219999999999, 2024-09-11T03:39:06.796 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.796 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.796 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.796 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.856269+0000", 2024-09-11T03:39:06.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.856269+0000", 2024-09-11T03:39:06.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.856270+0000", 2024-09-11T03:39:06.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5099999999999999e-07 2024-09-11T03:39:06.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.856274+0000", 2024-09-11T03:39:06.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.95e-06 2024-09-11T03:39:06.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.856288+0000", 2024-09-11T03:39:06.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3352e-05 2024-09-11T03:39:06.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.856294+0000", 2024-09-11T03:39:06.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8119999999999996e-06 2024-09-11T03:39:06.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.856384+0000", 2024-09-11T03:39:06.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0026999999999997e-05 2024-09-11T03:39:06.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.856401+0000", 2024-09-11T03:39:06.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7169000000000001e-05 2024-09-11T03:39:06.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.857252+0000", 2024-09-11T03:39:06.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000851394 2024-09-11T03:39:06.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.857296+0000", 2024-09-11T03:39:06.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3866999999999999e-05 2024-09-11T03:39:06.799 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.800 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.800 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.800 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:593 2.e e129/95 2:7efbbea8:::smithi17557870-24:head v 129'38, mlcod=128'35)", 2024-09-11T03:39:06.800 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:05.840230+0000", 2024-09-11T03:39:06.800 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.86115216999999999, 2024-09-11T03:39:06.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001289697, 2024-09-11T03:39:06.800 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:06.800 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:06.800 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:06.800 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:06.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:06.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:05.840230+0000", 2024-09-11T03:39:06.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:06.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:05.840230+0000", 2024-09-11T03:39:06.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:06.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:06.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:05.840231+0000", 2024-09-11T03:39:06.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5200000000000002e-07 2024-09-11T03:39:06.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:06.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:05.840241+0000", 2024-09-11T03:39:06.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9399999999999997e-06 2024-09-11T03:39:06.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:06.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:05.840252+0000", 2024-09-11T03:39:06.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0777e-05 2024-09-11T03:39:06.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:06.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:05.840258+0000", 2024-09-11T03:39:06.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0650000000000004e-06 2024-09-11T03:39:06.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:06.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:05.840324+0000", 2024-09-11T03:39:06.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6809000000000004e-05 2024-09-11T03:39:06.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:06.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:05.840344+0000", 2024-09-11T03:39:06.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9284999999999999e-05 2024-09-11T03:39:06.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:06.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:05.841479+0000", 2024-09-11T03:39:06.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00113484 2024-09-11T03:39:06.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:06.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:06.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:06.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:05.841520+0000", 2024-09-11T03:39:06.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1529000000000002e-05 2024-09-11T03:39:06.804 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.804 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.804 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.804 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:06.804 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:06.804 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:39:06.805 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:06.811 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.8 to 1 (10000) 2024-09-11T03:39:06.815 INFO:tasks.rados.rados.2.smithi175.stdout:208: done (5 left) 2024-09-11T03:39:06.816 INFO:tasks.rados.rados.2.smithi175.stdout:209: write oid 26 current snap is 16 2024-09-11T03:39:06.816 INFO:tasks.rados.rados.2.smithi175.stdout:203: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-11T03:39:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:06.833 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:06.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:06.888 INFO:tasks.rados.rados.2.smithi175.stdout:206: finishing rollback tid 0 to smithi17557870-24 2024-09-11T03:39:06.888 INFO:tasks.rados.rados.2.smithi175.stdout:206: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:39:06.888 INFO:tasks.rados.rados.2.smithi175.stdout:206: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:39:06.888 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 40 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-11T03:39:06.888 INFO:tasks.rados.rados.2.smithi175.stdout:209: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2024-09-11T03:39:06.893 INFO:tasks.rados.rados.2.smithi175.stdout:209: writing smithi17557870-26 from 547928 to 1086413 tid 1 2024-09-11T03:39:06.900 INFO:tasks.rados.rados.2.smithi175.stdout:209: writing smithi17557870-26 from 1769539 to 2178562 tid 2 2024-09-11T03:39:06.906 INFO:tasks.rados.rados.2.smithi175.stdout:209: writing smithi17557870-26 from 2706697 to 3238751 tid 3 2024-09-11T03:39:06.906 INFO:tasks.rados.rados.2.smithi175.stdout:203: done (5 left) 2024-09-11T03:39:06.906 INFO:tasks.rados.rados.2.smithi175.stdout:206: done (4 left) 2024-09-11T03:39:06.906 INFO:tasks.rados.rados.2.smithi175.stdout:210: write (excl) oid 11 current snap is 16 2024-09-11T03:39:06.906 INFO:tasks.rados.rados.2.smithi175.stdout:210: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-09-11T03:39:06.912 INFO:tasks.rados.rados.2.smithi175.stdout:210: writing smithi17557870-11 from 564735 to 1178469 tid 1 2024-09-11T03:39:06.919 INFO:tasks.rados.rados.2.smithi175.stdout:210: writing smithi17557870-11 from 1828021 to 2497055 tid 2 2024-09-11T03:39:06.926 INFO:tasks.rados.rados.2.smithi175.stdout:210: writing smithi17557870-11 from 3134120 to 3764735 tid 3 2024-09-11T03:39:06.926 INFO:tasks.rados.rados.2.smithi175.stdout:211: delete oid 28 current snap is 16 2024-09-11T03:39:06.926 INFO:tasks.rados.rados.2.smithi175.stdout:204: finishing copy_from racing read to smithi17557870-16 2024-09-11T03:39:06.926 INFO:tasks.rados.rados.2.smithi175.stdout:207: finishing write tid 1 to smithi17557870-4 2024-09-11T03:39:06.926 INFO:tasks.rados.rados.2.smithi175.stdout:205: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-11T03:39:06.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:06.954 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:06.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:06.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:07.026 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:06 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:39:07.026 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:06 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:39:07.027 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:06 smithi084 ceph-mon[27254]: osdmap e129: 12 total, 12 up, 12 in 2024-09-11T03:39:07.027 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:06 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1971642112' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:39:07.027 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:06 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1971642112' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:39:07.027 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:06 smithi084 ceph-mon[27254]: osdmap e130: 12 total, 12 up, 12 in 2024-09-11T03:39:07.042 INFO:tasks.rados.rados.2.smithi175.stdout:204: finishing copy_from to smithi17557870-16 2024-09-11T03:39:07.042 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 31 (ObjNum 98 snap 15 seq_num 98) dirty exists 2024-09-11T03:39:07.042 INFO:tasks.rados.rados.2.smithi175.stdout:207: finishing write tid 2 to smithi17557870-4 2024-09-11T03:39:07.042 INFO:tasks.rados.rados.2.smithi175.stdout:207: finishing write tid 3 to smithi17557870-4 2024-09-11T03:39:07.042 INFO:tasks.rados.rados.2.smithi175.stdout:207: finishing write tid 4 to smithi17557870-4 2024-09-11T03:39:07.042 INFO:tasks.rados.rados.2.smithi175.stdout:207: finishing write tid 5 to smithi17557870-4 2024-09-11T03:39:07.043 INFO:tasks.rados.rados.2.smithi175.stdout:207: oid 4 updating version 0 to 31 2024-09-11T03:39:07.043 INFO:tasks.rados.rados.2.smithi175.stdout:207: oid 4 updating version 31 to 34 2024-09-11T03:39:07.043 INFO:tasks.rados.rados.2.smithi175.stdout:207: oid 4 version 34 is already newer than 33 2024-09-11T03:39:07.043 INFO:tasks.rados.rados.2.smithi175.stdout:207: oid 4 updating version 34 to 35 2024-09-11T03:39:07.043 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 35 (ObjNum 103 snap 16 seq_num 103) dirty exists 2024-09-11T03:39:07.043 INFO:tasks.rados.rados.2.smithi175.stdout:207: left oid 4 (ObjNum 103 snap 16 seq_num 103) 2024-09-11T03:39:07.043 INFO:tasks.rados.rados.2.smithi175.stdout:204: done (5 left) 2024-09-11T03:39:07.043 INFO:tasks.rados.rados.2.smithi175.stdout:205: done (4 left) 2024-09-11T03:39:07.044 INFO:tasks.rados.rados.2.smithi175.stdout:207: done (3 left) 2024-09-11T03:39:07.044 INFO:tasks.rados.rados.2.smithi175.stdout:211: done (2 left) 2024-09-11T03:39:07.044 INFO:tasks.rados.rados.2.smithi175.stdout:212: delete oid 21 current snap is 16 2024-09-11T03:39:07.044 INFO:tasks.rados.rados.2.smithi175.stdout:209: finishing write tid 1 to smithi17557870-26 2024-09-11T03:39:07.044 INFO:tasks.rados.rados.2.smithi175.stdout:209: finishing write tid 2 to smithi17557870-26 2024-09-11T03:39:07.044 INFO:tasks.rados.rados.2.smithi175.stdout:209: finishing write tid 3 to smithi17557870-26 2024-09-11T03:39:07.044 INFO:tasks.rados.rados.2.smithi175.stdout:209: finishing write tid 4 to smithi17557870-26 2024-09-11T03:39:07.044 INFO:tasks.rados.rados.2.smithi175.stdout:209: finishing write tid 5 to smithi17557870-26 2024-09-11T03:39:07.044 INFO:tasks.rados.rados.2.smithi175.stdout:209: oid 26 updating version 0 to 38 2024-09-11T03:39:07.044 INFO:tasks.rados.rados.2.smithi175.stdout:209: oid 26 updating version 38 to 41 2024-09-11T03:39:07.044 INFO:tasks.rados.rados.2.smithi175.stdout:209: oid 26 version 41 is already newer than 40 2024-09-11T03:39:07.044 INFO:tasks.rados.rados.2.smithi175.stdout:209: oid 26 version 41 is already newer than 39 2024-09-11T03:39:07.044 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 41 (ObjNum 104 snap 16 seq_num 104) dirty exists 2024-09-11T03:39:07.045 INFO:tasks.rados.rados.2.smithi175.stdout:209: left oid 26 (ObjNum 104 snap 16 seq_num 104) 2024-09-11T03:39:07.045 INFO:tasks.rados.rados.2.smithi175.stdout:210: finishing write tid 1 to smithi17557870-11 2024-09-11T03:39:07.045 INFO:tasks.rados.rados.2.smithi175.stdout:210: finishing write tid 2 to smithi17557870-11 2024-09-11T03:39:07.045 INFO:tasks.rados.rados.2.smithi175.stdout:210: finishing write tid 3 to smithi17557870-11 2024-09-11T03:39:07.045 INFO:tasks.rados.rados.2.smithi175.stdout:210: finishing write tid 4 to smithi17557870-11 2024-09-11T03:39:07.045 INFO:tasks.rados.rados.2.smithi175.stdout:210: finishing write tid 5 to smithi17557870-11 2024-09-11T03:39:07.045 INFO:tasks.rados.rados.2.smithi175.stdout:210: oid 11 updating version 0 to 26 2024-09-11T03:39:07.045 INFO:tasks.rados.rados.2.smithi175.stdout:210: oid 11 version 26 is already newer than 24 2024-09-11T03:39:07.045 INFO:tasks.rados.rados.2.smithi175.stdout:210: oid 11 updating version 26 to 27 2024-09-11T03:39:07.045 INFO:tasks.rados.rados.2.smithi175.stdout:210: oid 11 version 27 is already newer than 25 2024-09-11T03:39:07.045 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 27 (ObjNum 105 snap 16 seq_num 105) dirty exists 2024-09-11T03:39:07.045 INFO:tasks.rados.rados.2.smithi175.stdout:210: left oid 11 (ObjNum 105 snap 16 seq_num 105) 2024-09-11T03:39:07.046 INFO:tasks.rados.rados.2.smithi175.stdout:209: done (2 left) 2024-09-11T03:39:07.046 INFO:tasks.rados.rados.2.smithi175.stdout:210: done (1 left) 2024-09-11T03:39:07.046 INFO:tasks.rados.rados.2.smithi175.stdout:212: done (0 left) 2024-09-11T03:39:07.046 INFO:tasks.rados.rados.2.smithi175.stdout:213: write (excl) oid 48 current snap is 16 2024-09-11T03:39:07.047 INFO:tasks.rados.rados.2.smithi175.stdout:213: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2024-09-11T03:39:07.052 INFO:tasks.rados.rados.2.smithi175.stdout:213: writing smithi17557870-48 from 615156 to 1170990 tid 1 2024-09-11T03:39:07.056 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:07.057 INFO:tasks.rados.rados.2.smithi175.stdout:213: writing smithi17557870-48 from 1603467 to 2019828 tid 2 2024-09-11T03:39:07.065 INFO:tasks.rados.rados.2.smithi175.stdout:213: writing smithi17557870-48 from 2816389 to 3509884 tid 3 2024-09-11T03:39:07.065 INFO:tasks.rados.rados.2.smithi175.stdout:214: read oid 11 snap 12 2024-09-11T03:39:07.065 INFO:tasks.rados.rados.2.smithi175.stdout:214: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-11T03:39:07.065 INFO:tasks.rados.rados.2.smithi175.stdout:215: read oid 18 snap -1 2024-09-11T03:39:07.065 INFO:tasks.rados.rados.2.smithi175.stdout:215: expect (ObjNum 93 snap 12 seq_num 93) 2024-09-11T03:39:07.066 INFO:tasks.rados.rados.2.smithi175.stdout:216: snap_remove snap 13 2024-09-11T03:39:07.084 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:07.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:07.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:07.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:06 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:39:07.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:06 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:39:07.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:06 smithi121 ceph-mon[30729]: osdmap e129: 12 total, 12 up, 12 in 2024-09-11T03:39:07.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:06 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1971642112' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:39:07.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:06 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1971642112' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:39:07.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:06 smithi121 ceph-mon[30729]: osdmap e130: 12 total, 12 up, 12 in 2024-09-11T03:39:07.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:07.210 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:07.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:07.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:07.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:06 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:39:07.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:06 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:39:07.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:06 smithi175 ceph-mon[30894]: osdmap e129: 12 total, 12 up, 12 in 2024-09-11T03:39:07.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:06 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1971642112' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:39:07.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:06 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1971642112' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:39:07.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:06 smithi175 ceph-mon[30894]: osdmap e130: 12 total, 12 up, 12 in 2024-09-11T03:39:07.312 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:07.333 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:07.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:07.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:07.435 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:07.457 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:07.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:07.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:07.559 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:07.579 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:07.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:07.581 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:07.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:07.700 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:39:07.707 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:07.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:07.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:07.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:07.835 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:07.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:07.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:07.845 INFO:tasks.rados.rados.2.smithi175.stdout:216: done (3 left) 2024-09-11T03:39:07.846 INFO:tasks.rados.rados.2.smithi175.stdout:217: write (excl) oid 28 current snap is 16 2024-09-11T03:39:07.846 INFO:tasks.rados.rados.2.smithi175.stdout:213: finishing write tid 1 to smithi17557870-48 2024-09-11T03:39:07.846 INFO:tasks.rados.rados.2.smithi175.stdout:217: seq_num 109 ranges {631963=631083,2061949=560019,3243812=504767} 2024-09-11T03:39:07.852 INFO:tasks.rados.rados.2.smithi175.stdout:217: writing smithi17557870-28 from 631963 to 1263046 tid 1 2024-09-11T03:39:07.859 INFO:tasks.rados.rados.2.smithi175.stdout:217: writing smithi17557870-28 from 2061949 to 2621968 tid 2 2024-09-11T03:39:07.866 INFO:tasks.rados.rados.2.smithi175.stdout:217: writing smithi17557870-28 from 3243812 to 3748579 tid 3 2024-09-11T03:39:07.866 INFO:tasks.rados.rados.2.smithi175.stdout:218: rollback oid 27 current snap is 16 2024-09-11T03:39:07.866 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 27 to 12 2024-09-11T03:39:07.866 INFO:tasks.rados.rados.2.smithi175.stdout:213: finishing write tid 2 to smithi17557870-48 2024-09-11T03:39:07.866 INFO:tasks.rados.rados.2.smithi175.stdout:213: finishing write tid 3 to smithi17557870-48 2024-09-11T03:39:07.866 INFO:tasks.rados.rados.2.smithi175.stdout:213: finishing write tid 4 to smithi17557870-48 2024-09-11T03:39:07.867 INFO:tasks.rados.rados.2.smithi175.stdout:213: finishing write tid 5 to smithi17557870-48 2024-09-11T03:39:07.867 INFO:tasks.rados.rados.2.smithi175.stdout:213: oid 48 updating version 0 to 38 2024-09-11T03:39:07.867 INFO:tasks.rados.rados.2.smithi175.stdout:213: oid 48 updating version 38 to 39 2024-09-11T03:39:07.867 INFO:tasks.rados.rados.2.smithi175.stdout:213: oid 48 updating version 39 to 40 2024-09-11T03:39:07.867 INFO:tasks.rados.rados.2.smithi175.stdout:213: oid 48 version 40 is already newer than 37 2024-09-11T03:39:07.867 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 40 (ObjNum 108 snap 16 seq_num 108) dirty exists 2024-09-11T03:39:07.867 INFO:tasks.rados.rados.2.smithi175.stdout:213: left oid 48 (ObjNum 108 snap 16 seq_num 108) 2024-09-11T03:39:07.867 INFO:tasks.rados.rados.2.smithi175.stdout:214: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-11T03:39:07.931 INFO:tasks.rados.rados.2.smithi175.stdout:215: expect (ObjNum 93 snap 12 seq_num 93) 2024-09-11T03:39:07.936 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:07.962 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:07.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:07.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:08.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:07 smithi084 ceph-mon[27254]: pgmap v349: 17 pgs: 17 active+clean; 119 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 2.9 MiB/s wr, 22 op/s 2024-09-11T03:39:08.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:07 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:39:08.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:07 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:39:08.063 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:08.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 9 1 2024-09-11T03:39:08.086 INFO:tasks.rados.rados.2.smithi175.stdout:217: finishing write tid 1 to smithi17557870-28 2024-09-11T03:39:08.086 INFO:tasks.rados.rados.2.smithi175.stdout:217: finishing write tid 2 to smithi17557870-28 2024-09-11T03:39:08.086 INFO:tasks.rados.rados.2.smithi175.stdout:213: done (4 left) 2024-09-11T03:39:08.086 INFO:tasks.rados.rados.2.smithi175.stdout:214: done (3 left) 2024-09-11T03:39:08.087 INFO:tasks.rados.rados.2.smithi175.stdout:215: done (2 left) 2024-09-11T03:39:08.087 INFO:tasks.rados.rados.2.smithi175.stdout:219: write oid 22 current snap is 16 2024-09-11T03:39:08.087 INFO:tasks.rados.rados.2.smithi175.stdout:217: finishing write tid 3 to smithi17557870-28 2024-09-11T03:39:08.087 INFO:tasks.rados.rados.2.smithi175.stdout:219: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2024-09-11T03:39:08.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:08.088 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:08.088 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:08.093 INFO:tasks.rados.rados.2.smithi175.stdout:219: writing smithi17557870-22 from 648770 to 1355102 tid 1 2024-09-11T03:39:08.100 INFO:tasks.rados.rados.2.smithi175.stdout:219: writing smithi17557870-22 from 2120431 to 2540461 tid 2 2024-09-11T03:39:08.106 INFO:tasks.rados.rados.2.smithi175.stdout:219: writing smithi17557870-22 from 3271235 to 3703627 tid 3 2024-09-11T03:39:08.106 INFO:tasks.rados.rados.2.smithi175.stdout:220: delete oid 36 current snap is 16 2024-09-11T03:39:08.107 INFO:tasks.rados.rados.2.smithi175.stdout:217: finishing write tid 4 to smithi17557870-28 2024-09-11T03:39:08.107 INFO:tasks.rados.rados.2.smithi175.stdout:218: finishing rollback tid 0 to smithi17557870-27 2024-09-11T03:39:08.107 INFO:tasks.rados.rados.2.smithi175.stdout:217: finishing write tid 5 to smithi17557870-28 2024-09-11T03:39:08.107 INFO:tasks.rados.rados.2.smithi175.stdout:217: oid 28 updating version 0 to 77 2024-09-11T03:39:08.107 INFO:tasks.rados.rados.2.smithi175.stdout:217: oid 28 version 77 is already newer than 75 2024-09-11T03:39:08.107 INFO:tasks.rados.rados.2.smithi175.stdout:217: oid 28 version 77 is already newer than 76 2024-09-11T03:39:08.107 INFO:tasks.rados.rados.2.smithi175.stdout:217: oid 28 version 77 is already newer than 72 2024-09-11T03:39:08.107 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 77 (ObjNum 109 snap 16 seq_num 109) dirty exists 2024-09-11T03:39:08.107 INFO:tasks.rados.rados.2.smithi175.stdout:217: left oid 28 (ObjNum 109 snap 16 seq_num 109) 2024-09-11T03:39:08.107 INFO:tasks.rados.rados.2.smithi175.stdout:218: finishing rollback tid 1 to smithi17557870-27 2024-09-11T03:39:08.108 INFO:tasks.rados.rados.2.smithi175.stdout:218: finishing rollback tid 2 to smithi17557870-27 2024-09-11T03:39:08.108 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 48 (ObjNum 82 snap 11 seq_num 82) dirty exists 2024-09-11T03:39:08.108 INFO:tasks.rados.rados.2.smithi175.stdout:219: finishing write tid 1 to smithi17557870-22 2024-09-11T03:39:08.108 INFO:tasks.rados.rados.2.smithi175.stdout:219: finishing write tid 2 to smithi17557870-22 2024-09-11T03:39:08.109 INFO:tasks.rados.rados.2.smithi175.stdout:219: finishing write tid 3 to smithi17557870-22 2024-09-11T03:39:08.109 INFO:tasks.rados.rados.2.smithi175.stdout:219: finishing write tid 4 to smithi17557870-22 2024-09-11T03:39:08.109 INFO:tasks.rados.rados.2.smithi175.stdout:217: done (3 left) 2024-09-11T03:39:08.109 INFO:tasks.rados.rados.2.smithi175.stdout:218: done (2 left) 2024-09-11T03:39:08.109 INFO:tasks.rados.rados.2.smithi175.stdout:220: done (1 left) 2024-09-11T03:39:08.109 INFO:tasks.rados.rados.2.smithi175.stdout:221: copy_from oid 46 from oid 31 current snap is 16 2024-09-11T03:39:08.109 INFO:tasks.rados.rados.2.smithi175.stdout:222: rollback oid 8 current snap is 16 2024-09-11T03:39:08.110 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 8 to 12 2024-09-11T03:39:08.110 INFO:tasks.rados.rados.2.smithi175.stdout:223: delete oid 43 current snap is 16 2024-09-11T03:39:08.110 INFO:tasks.rados.rados.2.smithi175.stdout:219: finishing write tid 5 to smithi17557870-22 2024-09-11T03:39:08.110 INFO:tasks.rados.rados.2.smithi175.stdout:219: oid 22 updating version 0 to 80 2024-09-11T03:39:08.110 INFO:tasks.rados.rados.2.smithi175.stdout:219: oid 22 version 80 is already newer than 79 2024-09-11T03:39:08.110 INFO:tasks.rados.rados.2.smithi175.stdout:219: oid 22 updating version 80 to 81 2024-09-11T03:39:08.110 INFO:tasks.rados.rados.2.smithi175.stdout:219: oid 22 updating version 81 to 82 2024-09-11T03:39:08.110 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 82 (ObjNum 110 snap 16 seq_num 110) dirty exists 2024-09-11T03:39:08.110 INFO:tasks.rados.rados.2.smithi175.stdout:219: left oid 22 (ObjNum 110 snap 16 seq_num 110) 2024-09-11T03:39:08.111 INFO:tasks.rados.rados.2.smithi175.stdout:219: done (3 left) 2024-09-11T03:39:08.111 INFO:tasks.rados.rados.2.smithi175.stdout:223: done (2 left) 2024-09-11T03:39:08.112 INFO:tasks.rados.rados.2.smithi175.stdout:224: read oid 17 snap -1 2024-09-11T03:39:08.112 INFO:tasks.rados.rados.2.smithi175.stdout:224: expect (ObjNum 88 snap 12 seq_num 88) 2024-09-11T03:39:08.112 INFO:tasks.rados.rados.2.smithi175.stdout:225: copy_from oid 7 from oid 22 current snap is 16 2024-09-11T03:39:08.112 INFO:tasks.rados.rados.2.smithi175.stdout:226: snap_create 2024-09-11T03:39:08.112 INFO:tasks.rados.rados.2.smithi175.stdout:222: finishing rollback tid 0 to smithi17557870-8 2024-09-11T03:39:08.112 INFO:tasks.rados.rados.2.smithi175.stdout:222: finishing rollback tid 1 to smithi17557870-8 2024-09-11T03:39:08.112 INFO:tasks.rados.rados.2.smithi175.stdout:222: finishing rollback tid 2 to smithi17557870-8 2024-09-11T03:39:08.112 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 58 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-11T03:39:08.116 INFO:tasks.rados.rados.2.smithi175.stdout:224: expect (ObjNum 88 snap 12 seq_num 88) 2024-09-11T03:39:08.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:07 smithi121 ceph-mon[30729]: pgmap v349: 17 pgs: 17 active+clean; 119 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 2.9 MiB/s wr, 22 op/s 2024-09-11T03:39:08.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:07 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:39:08.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:07 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:39:08.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:08.215 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:08.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:08.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:07 smithi175 ceph-mon[30894]: pgmap v349: 17 pgs: 17 active+clean; 119 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 2.9 MiB/s wr, 22 op/s 2024-09-11T03:39:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:07 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:39:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:07 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:39:08.280 INFO:tasks.rados.rados.2.smithi175.stdout:221: finishing copy_from racing read to smithi17557870-46 2024-09-11T03:39:08.280 INFO:tasks.rados.rados.2.smithi175.stdout:225: finishing copy_from racing read to smithi17557870-7 2024-09-11T03:39:08.280 INFO:tasks.rados.rados.2.smithi175.stdout:221: finishing copy_from to smithi17557870-46 2024-09-11T03:39:08.280 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 23 (ObjNum 66 snap 6 seq_num 66) dirty exists 2024-09-11T03:39:08.280 INFO:tasks.rados.rados.2.smithi175.stdout:225: finishing copy_from to smithi17557870-7 2024-09-11T03:39:08.280 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 82 (ObjNum 110 snap 16 seq_num 110) dirty exists 2024-09-11T03:39:08.284 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:39:08.284 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:39:08.301 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:39:08.301 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:39:08.316 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:08.318 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:39:08.319 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:39:08.335 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:39:08.335 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:39:08.338 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:08.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:08.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:08.353 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:39:08.353 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:39:08.370 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:39:08.370 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:39:08.387 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:39:08.388 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:39:08.404 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:39:08.405 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:39:08.422 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:39:08.422 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:39:08.439 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:39:08.439 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:39:08.439 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:08.456 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:39:08.457 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:39:08.460 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:08.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:08.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:08.475 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:39:08.475 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:39:08.562 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:08.584 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:08.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:08.585 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:08.686 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:08.707 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:08.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:08.708 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:08.739 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T03:39:08.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:08.829 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:08.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:08.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:08.850 INFO:tasks.rados.rados.2.smithi175.stdout:221: done (4 left) 2024-09-11T03:39:08.850 INFO:tasks.rados.rados.2.smithi175.stdout:222: done (3 left) 2024-09-11T03:39:08.850 INFO:tasks.rados.rados.2.smithi175.stdout:224: done (2 left) 2024-09-11T03:39:08.850 INFO:tasks.rados.rados.2.smithi175.stdout:225: done (1 left) 2024-09-11T03:39:08.850 INFO:tasks.rados.rados.2.smithi175.stdout:226: done (0 left) 2024-09-11T03:39:08.850 INFO:tasks.rados.rados.2.smithi175.stdout:227: copy_from oid 25 from oid 6 current snap is 17 2024-09-11T03:39:08.851 INFO:tasks.rados.rados.2.smithi175.stdout:228: copy_from oid 32 from oid 10 current snap is 17 2024-09-11T03:39:08.851 INFO:tasks.rados.rados.2.smithi175.stdout:229: snap_create 2024-09-11T03:39:08.853 INFO:tasks.rados.rados.2.smithi175.stdout:227: finishing copy_from to smithi17557870-25 2024-09-11T03:39:08.853 INFO:tasks.rados.rados.2.smithi175.stdout:227: got expected ENOENT (src dne) 2024-09-11T03:39:08.854 INFO:tasks.rados.rados.2.smithi175.stdout:227: finishing copy_from racing read to smithi17557870-25 2024-09-11T03:39:08.859 INFO:tasks.rados.rados.2.smithi175.stdout:228: finishing copy_from racing read to smithi17557870-32 2024-09-11T03:39:08.861 INFO:tasks.rados.rados.2.smithi175.stdout:228: finishing copy_from to smithi17557870-32 2024-09-11T03:39:08.861 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 20 (ObjNum 54 snap 4 seq_num 54) dirty exists 2024-09-11T03:39:08.931 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:08.953 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:08.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:08.954 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:39:09.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:09.076 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:09.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:09.118 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:08 smithi121 ceph-mon[30729]: osdmap e131: 12 total, 12 up, 12 in 2024-09-11T03:39:09.118 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:08 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:39:09.118 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:08 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:39:09.118 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:08 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4050357766' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:09.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:08 smithi084 ceph-mon[27254]: osdmap e131: 12 total, 12 up, 12 in 2024-09-11T03:39:09.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:08 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:39:09.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:08 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:39:09.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:08 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4050357766' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:09.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:09.198 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:09.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:09.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:08 smithi175 ceph-mon[30894]: osdmap e131: 12 total, 12 up, 12 in 2024-09-11T03:39:09.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:08 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:39:09.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:08 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:39:09.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:08 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4050357766' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:09.300 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:09.321 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:09.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:09.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:09.393 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:09.393 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:09.423 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:09.447 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:09.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:09.449 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:09.549 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:09.560 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:39:09.560 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:39:09.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:09.571 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:09.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:09.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:09.694 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:09.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:09.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:09.796 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:09.822 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:09.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:09.823 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:09.852 INFO:tasks.rados.rados.2.smithi175.stdout:227: done (2 left) 2024-09-11T03:39:09.852 INFO:tasks.rados.rados.2.smithi175.stdout:228: done (1 left) 2024-09-11T03:39:09.852 INFO:tasks.rados.rados.2.smithi175.stdout:229: done (0 left) 2024-09-11T03:39:09.853 INFO:tasks.rados.rados.2.smithi175.stdout:230: copy_from oid 31 from oid 34 current snap is 18 2024-09-11T03:39:09.853 INFO:tasks.rados.rados.2.smithi175.stdout:231: rollback oid 47 current snap is 18 2024-09-11T03:39:09.853 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 47 to 17 2024-09-11T03:39:09.853 INFO:tasks.rados.rados.2.smithi175.stdout:232: read oid 50 snap -1 2024-09-11T03:39:09.853 INFO:tasks.rados.rados.2.smithi175.stdout:232: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-11T03:39:09.853 INFO:tasks.rados.rados.2.smithi175.stdout:233: rollback oid 22 current snap is 18 2024-09-11T03:39:09.853 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 22 to 17 2024-09-11T03:39:09.853 INFO:tasks.rados.rados.2.smithi175.stdout:234: rollback oid 29 current snap is 18 2024-09-11T03:39:09.853 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 29 to 16 2024-09-11T03:39:09.853 INFO:tasks.rados.rados.2.smithi175.stdout:235: snap_remove snap 12 2024-09-11T03:39:09.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:09 smithi084 ceph-mon[27254]: pgmap v352: 17 pgs: 17 active+clean; 125 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.0 MiB/s wr, 23 op/s 2024-09-11T03:39:09.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:09 smithi084 ceph-mon[27254]: osdmap e132: 12 total, 12 up, 12 in 2024-09-11T03:39:09.924 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:09.947 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:09.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:09.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:10.049 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:10.070 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:10.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:10.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:10.128 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:09 smithi175 ceph-mon[30894]: pgmap v352: 17 pgs: 17 active+clean; 125 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.0 MiB/s wr, 23 op/s 2024-09-11T03:39:10.128 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:09 smithi175 ceph-mon[30894]: osdmap e132: 12 total, 12 up, 12 in 2024-09-11T03:39:10.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:09 smithi121 ceph-mon[30729]: pgmap v352: 17 pgs: 17 active+clean; 125 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.0 MiB/s wr, 23 op/s 2024-09-11T03:39:10.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:09 smithi121 ceph-mon[30729]: osdmap e132: 12 total, 12 up, 12 in 2024-09-11T03:39:10.172 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:10.197 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:10.198 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:10.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T03:39:10.299 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:10.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:10.343 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:10.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:10.445 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:10.471 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:10.472 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:10.526 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:39:10.572 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:10.593 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:10.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:10.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:10.716 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:10.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:10.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:10.818 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:10.843 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:10.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:10.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:10.857 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.9 to 1 (10000) 2024-09-11T03:39:10.857 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:39:10.862 INFO:tasks.rados.rados.2.smithi175.stdout:235: done (5 left) 2024-09-11T03:39:10.862 INFO:tasks.rados.rados.2.smithi175.stdout:236: rollback oid 45 current snap is 18 2024-09-11T03:39:10.862 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 45 to 17 2024-09-11T03:39:10.862 INFO:tasks.rados.rados.2.smithi175.stdout:234: finishing rollback tid 0 to smithi17557870-29 2024-09-11T03:39:10.862 INFO:tasks.rados.rados.2.smithi175.stdout:234: finishing rollback tid 1 to smithi17557870-29 2024-09-11T03:39:10.862 INFO:tasks.rados.rados.2.smithi175.stdout:234: finishing rollback tid 2 to smithi17557870-29 2024-09-11T03:39:10.862 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 75 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-11T03:39:10.862 INFO:tasks.rados.rados.2.smithi175.stdout:230: finishing copy_from to smithi17557870-31 2024-09-11T03:39:10.863 INFO:tasks.rados.rados.2.smithi175.stdout:230: got expected ENOENT (src dne) 2024-09-11T03:39:10.863 INFO:tasks.rados.rados.2.smithi175.stdout:231: finishing rollback tid 0 to smithi17557870-47 2024-09-11T03:39:10.863 INFO:tasks.rados.rados.2.smithi175.stdout:230: finishing copy_from racing read to smithi17557870-31 2024-09-11T03:39:10.863 INFO:tasks.rados.rados.2.smithi175.stdout:231: finishing rollback tid 1 to smithi17557870-47 2024-09-11T03:39:10.863 INFO:tasks.rados.rados.2.smithi175.stdout:231: finishing rollback tid 2 to smithi17557870-47 2024-09-11T03:39:10.863 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 64 (ObjNum 91 snap 12 seq_num 91) dirty exists 2024-09-11T03:39:10.863 INFO:tasks.rados.rados.2.smithi175.stdout:232: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-11T03:39:10.924 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:10 smithi121 ceph-mon[30729]: osdmap e133: 12 total, 12 up, 12 in 2024-09-11T03:39:10.925 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3353182574' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:39:10.925 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3776639242' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:39:10.943 INFO:tasks.rados.rados.2.smithi175.stdout:233: finishing rollback tid 0 to smithi17557870-22 2024-09-11T03:39:10.943 INFO:tasks.rados.rados.2.smithi175.stdout:233: finishing rollback tid 1 to smithi17557870-22 2024-09-11T03:39:10.943 INFO:tasks.rados.rados.2.smithi175.stdout:233: finishing rollback tid 2 to smithi17557870-22 2024-09-11T03:39:10.943 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 86 (ObjNum 110 snap 16 seq_num 110) dirty exists 2024-09-11T03:39:10.943 INFO:tasks.rados.rados.2.smithi175.stdout:236: finishing rollback tid 0 to smithi17557870-45 2024-09-11T03:39:10.943 INFO:tasks.rados.rados.2.smithi175.stdout:236: finishing rollback tid 1 to smithi17557870-45 2024-09-11T03:39:10.943 INFO:tasks.rados.rados.2.smithi175.stdout:230: done (5 left) 2024-09-11T03:39:10.943 INFO:tasks.rados.rados.2.smithi175.stdout:231: done (4 left) 2024-09-11T03:39:10.943 INFO:tasks.rados.rados.2.smithi175.stdout:232: done (3 left) 2024-09-11T03:39:10.944 INFO:tasks.rados.rados.2.smithi175.stdout:233: done (2 left) 2024-09-11T03:39:10.944 INFO:tasks.rados.rados.2.smithi175.stdout:234: done (1 left) 2024-09-11T03:39:10.944 INFO:tasks.rados.rados.2.smithi175.stdout:237: delete oid 33 current snap is 18 2024-09-11T03:39:10.944 INFO:tasks.rados.rados.2.smithi175.stdout:236: finishing rollback tid 2 to smithi17557870-45 2024-09-11T03:39:10.944 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 35 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-11T03:39:10.945 INFO:tasks.rados.rados.2.smithi175.stdout:236: done (1 left) 2024-09-11T03:39:10.946 INFO:tasks.rados.rados.2.smithi175.stdout:237: done (0 left) 2024-09-11T03:39:10.946 INFO:tasks.rados.rados.2.smithi175.stdout:238: read oid 46 snap 16 2024-09-11T03:39:10.946 INFO:tasks.rados.rados.2.smithi175.stdout:238: expect (ObjNum 66 snap 6 seq_num 66) 2024-09-11T03:39:10.946 INFO:tasks.rados.rados.2.smithi175.stdout:239: read oid 45 snap -1 2024-09-11T03:39:10.946 INFO:tasks.rados.rados.2.smithi175.stdout:239: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-11T03:39:10.946 INFO:tasks.rados.rados.2.smithi175.stdout:240: read oid 20 snap 17 2024-09-11T03:39:10.946 INFO:tasks.rados.rados.2.smithi175.stdout:240: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-11T03:39:10.946 INFO:tasks.rados.rados.2.smithi175.stdout:241: copy_from oid 1 from oid 9 current snap is 18 2024-09-11T03:39:10.946 INFO:tasks.rados.rados.2.smithi175.stdout:242: copy_from oid 8 from oid 4 current snap is 18 2024-09-11T03:39:10.946 INFO:tasks.rados.rados.2.smithi175.stdout:243: copy_from oid 3 from oid 44 current snap is 18 2024-09-11T03:39:10.946 INFO:tasks.rados.rados.2.smithi175.stdout:244: read oid 21 snap -1 2024-09-11T03:39:10.946 INFO:tasks.rados.rados.2.smithi175.stdout:244: expect deleted 2024-09-11T03:39:10.946 INFO:tasks.rados.rados.2.smithi175.stdout:245: read oid 49 snap 16 2024-09-11T03:39:10.947 INFO:tasks.rados.rados.2.smithi175.stdout:245: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-11T03:39:10.947 INFO:tasks.rados.rados.2.smithi175.stdout:246: read oid 5 snap -1 2024-09-11T03:39:10.947 INFO:tasks.rados.rados.2.smithi175.stdout:246: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-11T03:39:10.947 INFO:tasks.rados.rados.2.smithi175.stdout:247: write (excl) oid 32 current snap is 18 2024-09-11T03:39:10.947 INFO:tasks.rados.rados.2.smithi175.stdout:247: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2024-09-11T03:39:10.947 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:10.954 INFO:tasks.rados.rados.2.smithi175.stdout:247: writing smithi17557870-32 from 715998 to 1323326 tid 1 2024-09-11T03:39:10.960 INFO:tasks.rados.rados.2.smithi175.stdout:247: writing smithi17557870-32 from 1838006 to 2265374 tid 2 2024-09-11T03:39:10.966 INFO:tasks.rados.rados.2.smithi175.stdout:247: writing smithi17557870-32 from 2864574 to 3342054 tid 3 2024-09-11T03:39:10.970 INFO:tasks.rados.rados.2.smithi175.stdout:247: writing smithi17557870-32 from 3863835 to 3915998 tid 4 2024-09-11T03:39:10.970 INFO:tasks.rados.rados.2.smithi175.stdout:248: write (excl) oid 18 current snap is 18 2024-09-11T03:39:10.970 INFO:tasks.rados.rados.2.smithi175.stdout:248: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2024-09-11T03:39:10.977 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:10.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:10.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:10.978 INFO:tasks.rados.rados.2.smithi175.stdout:248: writing smithi17557870-18 from 732805 to 1531735 tid 1 2024-09-11T03:39:10.986 INFO:tasks.rados.rados.2.smithi175.stdout:248: writing smithi17557870-18 from 2296488 to 2867514 tid 2 2024-09-11T03:39:10.990 INFO:tasks.rados.rados.2.smithi175.stdout:248: writing smithi17557870-18 from 3291997 to 3697102 tid 3 2024-09-11T03:39:10.990 INFO:tasks.rados.rados.2.smithi175.stdout:249: delete oid 31 current snap is 18 2024-09-11T03:39:10.991 INFO:tasks.rados.rados.2.smithi175.stdout:240: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-11T03:39:11.068 INFO:tasks.rados.rados.2.smithi175.stdout:239: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-11T03:39:11.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:11.094 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:39:11.094 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:39:11.101 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:11.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:11.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:11.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:10 smithi084 ceph-mon[27254]: osdmap e133: 12 total, 12 up, 12 in 2024-09-11T03:39:11.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:10 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3353182574' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:39:11.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:10 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3776639242' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:39:11.153 INFO:tasks.rados.rados.2.smithi175.stdout:243: finishing copy_from racing read to smithi17557870-3 2024-09-11T03:39:11.153 INFO:tasks.rados.rados.2.smithi175.stdout:246: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-11T03:39:11.203 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:11.225 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:11.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:11.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:11.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:10 smithi175 ceph-mon[30894]: osdmap e133: 12 total, 12 up, 12 in 2024-09-11T03:39:11.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:10 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3353182574' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:39:11.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:10 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3776639242' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:39:11.230 INFO:tasks.rados.rados.2.smithi175.stdout:241: finishing copy_from racing read to smithi17557870-1 2024-09-11T03:39:11.230 INFO:tasks.rados.rados.2.smithi175.stdout:243: finishing copy_from to smithi17557870-3 2024-09-11T03:39:11.230 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 18 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-11T03:39:11.230 INFO:tasks.rados.rados.2.smithi175.stdout:247: finishing write tid 1 to smithi17557870-32 2024-09-11T03:39:11.231 INFO:tasks.rados.rados.2.smithi175.stdout:238: expect (ObjNum 66 snap 6 seq_num 66) 2024-09-11T03:39:11.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:11.347 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:11.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:11.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:11.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:39:11.361 INFO:tasks.rados.rados.2.smithi175.stdout:245: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-11T03:39:11.449 INFO:tasks.rados.rados.2.smithi175.stdout:242: finishing copy_from racing read to smithi17557870-8 2024-09-11T03:39:11.449 INFO:tasks.rados.rados.2.smithi175.stdout:247: finishing write tid 2 to smithi17557870-32 2024-09-11T03:39:11.450 INFO:tasks.rados.rados.2.smithi175.stdout:241: finishing copy_from to smithi17557870-1 2024-09-11T03:39:11.450 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-11T03:39:11.450 INFO:tasks.rados.rados.2.smithi175.stdout:242: finishing copy_from to smithi17557870-8 2024-09-11T03:39:11.450 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 35 (ObjNum 103 snap 16 seq_num 103) dirty exists 2024-09-11T03:39:11.450 INFO:tasks.rados.rados.2.smithi175.stdout:247: finishing write tid 3 to smithi17557870-32 2024-09-11T03:39:11.450 INFO:tasks.rados.rados.2.smithi175.stdout:247: finishing write tid 4 to smithi17557870-32 2024-09-11T03:39:11.450 INFO:tasks.rados.rados.2.smithi175.stdout:247: finishing write tid 5 to smithi17557870-32 2024-09-11T03:39:11.450 INFO:tasks.rados.rados.2.smithi175.stdout:247: finishing write tid 6 to smithi17557870-32 2024-09-11T03:39:11.450 INFO:tasks.rados.rados.2.smithi175.stdout:247: oid 32 updating version 0 to 22 2024-09-11T03:39:11.450 INFO:tasks.rados.rados.2.smithi175.stdout:247: oid 32 updating version 22 to 24 2024-09-11T03:39:11.450 INFO:tasks.rados.rados.2.smithi175.stdout:247: oid 32 version 24 is already newer than 23 2024-09-11T03:39:11.450 INFO:tasks.rados.rados.2.smithi175.stdout:247: oid 32 version 24 is already newer than 21 2024-09-11T03:39:11.450 INFO:tasks.rados.rados.2.smithi175.stdout:247: oid 32 updating version 24 to 25 2024-09-11T03:39:11.450 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 25 (ObjNum 114 snap 18 seq_num 114) dirty exists 2024-09-11T03:39:11.451 INFO:tasks.rados.rados.2.smithi175.stdout:247: left oid 32 (ObjNum 114 snap 18 seq_num 114) 2024-09-11T03:39:11.451 INFO:tasks.rados.rados.2.smithi175.stdout:248: finishing write tid 1 to smithi17557870-18 2024-09-11T03:39:11.451 INFO:tasks.rados.rados.2.smithi175.stdout:248: finishing write tid 2 to smithi17557870-18 2024-09-11T03:39:11.451 INFO:tasks.rados.rados.2.smithi175.stdout:248: finishing write tid 3 to smithi17557870-18 2024-09-11T03:39:11.451 INFO:tasks.rados.rados.2.smithi175.stdout:248: finishing write tid 4 to smithi17557870-18 2024-09-11T03:39:11.451 INFO:tasks.rados.rados.2.smithi175.stdout:248: finishing write tid 5 to smithi17557870-18 2024-09-11T03:39:11.451 INFO:tasks.rados.rados.2.smithi175.stdout:248: oid 18 updating version 0 to 29 2024-09-11T03:39:11.451 INFO:tasks.rados.rados.2.smithi175.stdout:248: oid 18 updating version 29 to 30 2024-09-11T03:39:11.451 INFO:tasks.rados.rados.2.smithi175.stdout:248: oid 18 updating version 30 to 31 2024-09-11T03:39:11.451 INFO:tasks.rados.rados.2.smithi175.stdout:248: oid 18 updating version 31 to 32 2024-09-11T03:39:11.451 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 32 (ObjNum 115 snap 18 seq_num 115) dirty exists 2024-09-11T03:39:11.451 INFO:tasks.rados.rados.2.smithi175.stdout:248: left oid 18 (ObjNum 115 snap 18 seq_num 115) 2024-09-11T03:39:11.451 INFO:tasks.rados.rados.2.smithi175.stdout:238: done (11 left) 2024-09-11T03:39:11.451 INFO:tasks.rados.rados.2.smithi175.stdout:239: done (10 left) 2024-09-11T03:39:11.452 INFO:tasks.rados.rados.2.smithi175.stdout:240: done (9 left) 2024-09-11T03:39:11.452 INFO:tasks.rados.rados.2.smithi175.stdout:241: done (8 left) 2024-09-11T03:39:11.452 INFO:tasks.rados.rados.2.smithi175.stdout:242: done (7 left) 2024-09-11T03:39:11.452 INFO:tasks.rados.rados.2.smithi175.stdout:243: done (6 left) 2024-09-11T03:39:11.452 INFO:tasks.rados.rados.2.smithi175.stdout:244: done (5 left) 2024-09-11T03:39:11.452 INFO:tasks.rados.rados.2.smithi175.stdout:245: done (4 left) 2024-09-11T03:39:11.452 INFO:tasks.rados.rados.2.smithi175.stdout:246: done (3 left) 2024-09-11T03:39:11.452 INFO:tasks.rados.rados.2.smithi175.stdout:247: done (2 left) 2024-09-11T03:39:11.452 INFO:tasks.rados.rados.2.smithi175.stdout:248: done (1 left) 2024-09-11T03:39:11.452 INFO:tasks.rados.rados.2.smithi175.stdout:249: done (0 left) 2024-09-11T03:39:11.452 INFO:tasks.rados.rados.2.smithi175.stdout:250: write (excl) oid 10 current snap is 18 2024-09-11T03:39:11.452 INFO:tasks.rados.rados.2.smithi175.stdout:250: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2024-09-11T03:39:11.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:11.456 INFO:tasks.rados.rados.2.smithi175.stdout:250: writing smithi17557870-10 from 766419 to 1315847 tid 1 2024-09-11T03:39:11.462 INFO:tasks.rados.rados.2.smithi175.stdout:250: writing smithi17557870-10 from 2013452 to 2588147 tid 2 2024-09-11T03:39:11.469 INFO:tasks.rados.rados.2.smithi175.stdout:250: writing smithi17557870-10 from 3346843 to 3774492 tid 3 2024-09-11T03:39:11.469 INFO:tasks.rados.rados.2.smithi175.stdout:251: write (excl) oid 12 current snap is 18 2024-09-11T03:39:11.469 INFO:tasks.rados.rados.2.smithi175.stdout:251: seq_num 118 ranges {783226=624677,2071934=434706,2974266=638921} 2024-09-11T03:39:11.475 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:11.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:11.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:11.476 INFO:tasks.rados.rados.2.smithi175.stdout:251: writing smithi17557870-12 from 783226 to 1407903 tid 1 2024-09-11T03:39:11.482 INFO:tasks.rados.rados.2.smithi175.stdout:251: writing smithi17557870-12 from 2071934 to 2506640 tid 2 2024-09-11T03:39:11.489 INFO:tasks.rados.rados.2.smithi175.stdout:251: writing smithi17557870-12 from 2974266 to 3613187 tid 3 2024-09-11T03:39:11.489 INFO:tasks.rados.rados.2.smithi175.stdout:252: snap_remove snap 16 2024-09-11T03:39:11.576 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:11.597 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:11.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:11.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:11.699 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:11.720 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:11.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:11.774 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 10 1 2024-09-11T03:39:11.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T03:39:11.816 INFO:tasks.rados.rados.2.smithi175.stdout:252: done (2 left) 2024-09-11T03:39:11.817 INFO:tasks.rados.rados.2.smithi175.stdout:253: write oid 34 current snap is 18 2024-09-11T03:39:11.817 INFO:tasks.rados.rados.2.smithi175.stdout:250: finishing write tid 1 to smithi17557870-10 2024-09-11T03:39:11.817 INFO:tasks.rados.rados.2.smithi175.stdout:253: seq_num 119 ranges {400033=699926,1614063=385970} 2024-09-11T03:39:11.822 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:11.823 INFO:tasks.rados.rados.2.smithi175.stdout:253: writing smithi17557870-34 from 400033 to 1099959 tid 1 2024-09-11T03:39:11.829 INFO:tasks.rados.rados.2.smithi175.stdout:253: writing smithi17557870-34 from 1614063 to 2000033 tid 2 2024-09-11T03:39:11.829 INFO:tasks.rados.rados.2.smithi175.stdout:254: rollback oid 24 current snap is 18 2024-09-11T03:39:11.829 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 17 2024-09-11T03:39:11.829 INFO:tasks.rados.rados.2.smithi175.stdout:250: finishing write tid 2 to smithi17557870-10 2024-09-11T03:39:11.829 INFO:tasks.rados.rados.2.smithi175.stdout:250: finishing write tid 3 to smithi17557870-10 2024-09-11T03:39:11.829 INFO:tasks.rados.rados.2.smithi175.stdout:250: finishing write tid 4 to smithi17557870-10 2024-09-11T03:39:11.829 INFO:tasks.rados.rados.2.smithi175.stdout:250: finishing write tid 5 to smithi17557870-10 2024-09-11T03:39:11.829 INFO:tasks.rados.rados.2.smithi175.stdout:250: oid 10 updating version 0 to 69 2024-09-11T03:39:11.829 INFO:tasks.rados.rados.2.smithi175.stdout:250: oid 10 updating version 69 to 70 2024-09-11T03:39:11.829 INFO:tasks.rados.rados.2.smithi175.stdout:250: oid 10 version 70 is already newer than 68 2024-09-11T03:39:11.830 INFO:tasks.rados.rados.2.smithi175.stdout:250: oid 10 updating version 70 to 71 2024-09-11T03:39:11.830 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 71 (ObjNum 117 snap 18 seq_num 117) dirty exists 2024-09-11T03:39:11.830 INFO:tasks.rados.rados.2.smithi175.stdout:250: left oid 10 (ObjNum 117 snap 18 seq_num 117) 2024-09-11T03:39:11.830 INFO:tasks.rados.rados.2.smithi175.stdout:250: done (3 left) 2024-09-11T03:39:11.830 INFO:tasks.rados.rados.2.smithi175.stdout:255: copy_from oid 26 from oid 38 current snap is 18 2024-09-11T03:39:11.830 INFO:tasks.rados.rados.2.smithi175.stdout:251: finishing write tid 1 to smithi17557870-12 2024-09-11T03:39:11.830 INFO:tasks.rados.rados.2.smithi175.stdout:251: finishing write tid 2 to smithi17557870-12 2024-09-11T03:39:11.830 INFO:tasks.rados.rados.2.smithi175.stdout:251: finishing write tid 3 to smithi17557870-12 2024-09-11T03:39:11.830 INFO:tasks.rados.rados.2.smithi175.stdout:251: finishing write tid 4 to smithi17557870-12 2024-09-11T03:39:11.830 INFO:tasks.rados.rados.2.smithi175.stdout:251: finishing write tid 5 to smithi17557870-12 2024-09-11T03:39:11.830 INFO:tasks.rados.rados.2.smithi175.stdout:251: oid 12 updating version 0 to 83 2024-09-11T03:39:11.830 INFO:tasks.rados.rados.2.smithi175.stdout:251: oid 12 version 83 is already newer than 82 2024-09-11T03:39:11.830 INFO:tasks.rados.rados.2.smithi175.stdout:251: oid 12 version 83 is already newer than 80 2024-09-11T03:39:11.831 INFO:tasks.rados.rados.2.smithi175.stdout:251: oid 12 version 83 is already newer than 81 2024-09-11T03:39:11.831 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 83 (ObjNum 118 snap 18 seq_num 118) dirty exists 2024-09-11T03:39:11.831 INFO:tasks.rados.rados.2.smithi175.stdout:251: left oid 12 (ObjNum 118 snap 18 seq_num 118) 2024-09-11T03:39:11.831 INFO:tasks.rados.rados.2.smithi175.stdout:253: finishing write tid 1 to smithi17557870-34 2024-09-11T03:39:11.831 INFO:tasks.rados.rados.2.smithi175.stdout:251: done (3 left) 2024-09-11T03:39:11.831 INFO:tasks.rados.rados.2.smithi175.stdout:256: write (excl) oid 1 current snap is 18 2024-09-11T03:39:11.831 INFO:tasks.rados.rados.2.smithi175.stdout:256: seq_num 120 ranges {416840=775175,1672545=344295} 2024-09-11T03:39:11.835 INFO:tasks.rados.rados.2.smithi175.stdout:256: writing smithi17557870-1 from 416840 to 1192015 tid 1 2024-09-11T03:39:11.841 INFO:tasks.rados.rados.2.smithi175.stdout:256: writing smithi17557870-1 from 1672545 to 2016840 tid 2 2024-09-11T03:39:11.841 INFO:tasks.rados.rados.2.smithi175.stdout:257: delete oid 40 current snap is 18 2024-09-11T03:39:11.841 INFO:tasks.rados.rados.2.smithi175.stdout:253: finishing write tid 2 to smithi17557870-34 2024-09-11T03:39:11.841 INFO:tasks.rados.rados.2.smithi175.stdout:253: finishing write tid 3 to smithi17557870-34 2024-09-11T03:39:11.841 INFO:tasks.rados.rados.2.smithi175.stdout:253: finishing write tid 4 to smithi17557870-34 2024-09-11T03:39:11.841 INFO:tasks.rados.rados.2.smithi175.stdout:253: oid 34 updating version 0 to 46 2024-09-11T03:39:11.841 INFO:tasks.rados.rados.2.smithi175.stdout:253: oid 34 updating version 46 to 48 2024-09-11T03:39:11.841 INFO:tasks.rados.rados.2.smithi175.stdout:253: oid 34 version 48 is already newer than 47 2024-09-11T03:39:11.842 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 48 (ObjNum 119 snap 18 seq_num 119) dirty exists 2024-09-11T03:39:11.842 INFO:tasks.rados.rados.2.smithi175.stdout:253: left oid 34 (ObjNum 119 snap 18 seq_num 119) 2024-09-11T03:39:11.842 INFO:tasks.rados.rados.2.smithi175.stdout:254: finishing rollback tid 0 to smithi17557870-24 2024-09-11T03:39:11.842 INFO:tasks.rados.rados.2.smithi175.stdout:254: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:39:11.842 INFO:tasks.rados.rados.2.smithi175.stdout:254: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:39:11.842 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 72 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-11T03:39:11.842 INFO:tasks.rados.rados.2.smithi175.stdout:256: finishing write tid 1 to smithi17557870-1 2024-09-11T03:39:11.843 INFO:tasks.rados.rados.2.smithi175.stdout:256: finishing write tid 2 to smithi17557870-1 2024-09-11T03:39:11.843 INFO:tasks.rados.rados.2.smithi175.stdout:256: finishing write tid 3 to smithi17557870-1 2024-09-11T03:39:11.843 INFO:tasks.rados.rados.2.smithi175.stdout:253: done (4 left) 2024-09-11T03:39:11.843 INFO:tasks.rados.rados.2.smithi175.stdout:254: done (3 left) 2024-09-11T03:39:11.843 INFO:tasks.rados.rados.2.smithi175.stdout:257: done (2 left) 2024-09-11T03:39:11.843 INFO:tasks.rados.rados.2.smithi175.stdout:258: snap_remove snap 17 2024-09-11T03:39:11.846 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:11.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:11.847 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:11.948 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:11.970 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:11.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:11.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:12.043 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:39:12.072 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:12.093 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:12.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:12.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:12.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:11 smithi084 ceph-mon[27254]: pgmap v355: 17 pgs: 17 active+clean; 125 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 872 KiB/s wr, 7 op/s 2024-09-11T03:39:12.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:11 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:12.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:11 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3353182574' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:39:12.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:11 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3776639242' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:39:12.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:11 smithi084 ceph-mon[27254]: osdmap e134: 12 total, 12 up, 12 in 2024-09-11T03:39:12.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:11 smithi084 ceph-mon[27254]: osdmap e135: 12 total, 12 up, 12 in 2024-09-11T03:39:12.164 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:11 smithi121 ceph-mon[30729]: pgmap v355: 17 pgs: 17 active+clean; 125 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 872 KiB/s wr, 7 op/s 2024-09-11T03:39:12.164 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:11 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:12.164 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:11 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3353182574' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:39:12.164 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:11 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3776639242' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:39:12.164 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:11 smithi121 ceph-mon[30729]: osdmap e134: 12 total, 12 up, 12 in 2024-09-11T03:39:12.164 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:11 smithi121 ceph-mon[30729]: osdmap e135: 12 total, 12 up, 12 in 2024-09-11T03:39:12.194 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:12.217 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:12.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:12.218 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:11 smithi175 ceph-mon[30894]: pgmap v355: 17 pgs: 17 active+clean; 125 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 872 KiB/s wr, 7 op/s 2024-09-11T03:39:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:11 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:11 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3353182574' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:39:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:11 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3776639242' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:39:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:11 smithi175 ceph-mon[30894]: osdmap e134: 12 total, 12 up, 12 in 2024-09-11T03:39:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:11 smithi175 ceph-mon[30894]: osdmap e135: 12 total, 12 up, 12 in 2024-09-11T03:39:12.248 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:12.264 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:12.319 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:12.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:12.341 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:12.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:12.443 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:12.465 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:12.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:12.510 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:39:12.511 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:39:12.567 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:12.589 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:12.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:12.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:12.691 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:12.717 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:12.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:12.718 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:12.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:12.831 INFO:tasks.rados.rados.2.smithi175.stdout:258: done (2 left) 2024-09-11T03:39:12.831 INFO:tasks.rados.rados.2.smithi175.stdout:259: read oid 30 snap -1 2024-09-11T03:39:12.831 INFO:tasks.rados.rados.2.smithi175.stdout:259: expect (ObjNum 100 snap 16 seq_num 100) 2024-09-11T03:39:12.831 INFO:tasks.rados.rados.2.smithi175.stdout:260: copy_from oid 36 from oid 3 current snap is 18 2024-09-11T03:39:12.831 INFO:tasks.rados.rados.2.smithi175.stdout:261: rollback oid 27 current snap is 18 2024-09-11T03:39:12.832 INFO:tasks.rados.rados.2.smithi175.stdout:261: done (4 left) 2024-09-11T03:39:12.832 INFO:tasks.rados.rados.2.smithi175.stdout:262: snap_create 2024-09-11T03:39:12.832 INFO:tasks.rados.rados.2.smithi175.stdout:256: finishing write tid 4 to smithi17557870-1 2024-09-11T03:39:12.832 INFO:tasks.rados.rados.2.smithi175.stdout:256: oid 1 updating version 0 to 90 2024-09-11T03:39:12.832 INFO:tasks.rados.rados.2.smithi175.stdout:256: oid 1 version 90 is already newer than 88 2024-09-11T03:39:12.832 INFO:tasks.rados.rados.2.smithi175.stdout:256: oid 1 version 90 is already newer than 89 2024-09-11T03:39:12.832 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 90 (ObjNum 120 snap 18 seq_num 120) dirty exists 2024-09-11T03:39:12.832 INFO:tasks.rados.rados.2.smithi175.stdout:256: left oid 1 (ObjNum 120 snap 18 seq_num 120) 2024-09-11T03:39:12.832 INFO:tasks.rados.rados.2.smithi175.stdout:255: finishing copy_from racing read to smithi17557870-26 2024-09-11T03:39:12.832 INFO:tasks.rados.rados.2.smithi175.stdout:255: finishing copy_from to smithi17557870-26 2024-09-11T03:39:12.832 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 65 (ObjNum 101 snap 16 seq_num 101) dirty exists 2024-09-11T03:39:12.842 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:12.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:12.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:12.844 INFO:tasks.rados.rados.2.smithi175.stdout:260: finishing copy_from racing read to smithi17557870-36 2024-09-11T03:39:12.848 INFO:tasks.rados.rados.2.smithi175.stdout:260: finishing copy_from to smithi17557870-36 2024-09-11T03:39:12.849 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 18 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-11T03:39:12.850 INFO:tasks.rados.rados.2.smithi175.stdout:259: expect (ObjNum 100 snap 16 seq_num 100) 2024-09-11T03:39:12.944 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:12.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:12.969 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:12.969 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:13.029 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:39:13.029 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:39:13.036 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:39:13.036 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:39:13.053 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:39:13.053 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:39:13.069 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:39:13.069 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:39:13.071 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:13.085 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:39:13.085 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:39:13.092 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:13.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:13.093 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:13.103 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:39:13.103 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:39:13.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T03:39:13.119 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:39:13.120 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:39:13.137 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:39:13.137 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:39:13.154 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:39:13.154 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:39:13.172 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:39:13.173 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:39:13.188 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:39:13.188 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:39:13.194 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:13.205 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:39:13.206 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:39:13.214 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:13.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:13.316 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:13.337 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:13.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:13.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:13.357 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:39:13.439 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:13.460 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:13.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:13.461 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:13.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:13.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:13.585 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:13.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:13.686 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:13.707 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:13.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:13.708 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:13.809 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:13.826 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.10 to 1 (10000) 2024-09-11T03:39:13.829 INFO:tasks.rados.rados.2.smithi175.stdout:255: done (4 left) 2024-09-11T03:39:13.830 INFO:tasks.rados.rados.2.smithi175.stdout:256: done (3 left) 2024-09-11T03:39:13.830 INFO:tasks.rados.rados.2.smithi175.stdout:259: done (2 left) 2024-09-11T03:39:13.830 INFO:tasks.rados.rados.2.smithi175.stdout:260: done (1 left) 2024-09-11T03:39:13.830 INFO:tasks.rados.rados.2.smithi175.stdout:262: done (0 left) 2024-09-11T03:39:13.830 INFO:tasks.rados.rados.2.smithi175.stdout:263: read oid 25 snap -1 2024-09-11T03:39:13.830 INFO:tasks.rados.rados.2.smithi175.stdout:263: expect (ObjNum 75 snap 7 seq_num 75) 2024-09-11T03:39:13.830 INFO:tasks.rados.rados.2.smithi175.stdout:264: snap_remove snap 18 2024-09-11T03:39:13.831 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:13.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:13.842 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:39:13.843 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:39:13.843 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:39:13.843 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:39:13.932 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:13.955 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:13.956 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:14.057 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:14.079 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:14.080 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:14.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:13 smithi084 ceph-mon[27254]: pgmap v358: 17 pgs: 17 active+clean; 131 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.3 MiB/s wr, 30 op/s 2024-09-11T03:39:14.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:13 smithi084 ceph-mon[27254]: osdmap e136: 12 total, 12 up, 12 in 2024-09-11T03:39:14.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:13 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3755597971' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:14.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:13 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:39:14.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:13 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:39:14.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:13 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/491231820' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:39:14.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:13 smithi121 ceph-mon[30729]: pgmap v358: 17 pgs: 17 active+clean; 131 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.3 MiB/s wr, 30 op/s 2024-09-11T03:39:14.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:13 smithi121 ceph-mon[30729]: osdmap e136: 12 total, 12 up, 12 in 2024-09-11T03:39:14.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:13 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3755597971' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:14.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:13 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:39:14.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:13 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:39:14.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:13 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/491231820' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:39:14.181 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:14.203 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:14.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:14.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:14.221 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:39:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:13 smithi175 ceph-mon[30894]: pgmap v358: 17 pgs: 17 active+clean; 131 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.3 MiB/s wr, 30 op/s 2024-09-11T03:39:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:13 smithi175 ceph-mon[30894]: osdmap e136: 12 total, 12 up, 12 in 2024-09-11T03:39:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:13 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3755597971' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:13 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:39:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:13 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:39:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:13 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/491231820' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:39:14.305 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:14.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:14.328 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:14.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:14.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:14.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 11 1 2024-09-11T03:39:14.595 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:14.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T03:39:14.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:14.606 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:14.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:14.707 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:14.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:14.730 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:14.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:14.831 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:14.840 INFO:tasks.rados.rados.2.smithi175.stdout:264: done (1 left) 2024-09-11T03:39:14.840 INFO:tasks.rados.rados.2.smithi175.stdout:265: write (excl) oid 5 current snap is 19 2024-09-11T03:39:14.840 INFO:tasks.rados.rados.2.smithi175.stdout:265: seq_num 122 ranges {450454=642026,1505862=544592} 2024-09-11T03:39:14.845 INFO:tasks.rados.rados.2.smithi175.stdout:265: writing smithi17557870-5 from 450454 to 1092480 tid 1 2024-09-11T03:39:14.851 INFO:tasks.rados.rados.2.smithi175.stdout:265: writing smithi17557870-5 from 1505862 to 2050454 tid 2 2024-09-11T03:39:14.851 INFO:tasks.rados.rados.2.smithi175.stdout:266: copy_from oid 35 from oid 17 current snap is 19 2024-09-11T03:39:14.851 INFO:tasks.rados.rados.2.smithi175.stdout:263: expect (ObjNum 75 snap 7 seq_num 75) 2024-09-11T03:39:14.854 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:14.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:14.894 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:39:14.956 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:14.978 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:14.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:14.979 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:15.007 INFO:tasks.rados.rados.2.smithi175.stdout:265: finishing write tid 1 to smithi17557870-5 2024-09-11T03:39:15.008 INFO:tasks.rados.rados.2.smithi175.stdout:265: finishing write tid 2 to smithi17557870-5 2024-09-11T03:39:15.008 INFO:tasks.rados.rados.2.smithi175.stdout:265: finishing write tid 3 to smithi17557870-5 2024-09-11T03:39:15.008 INFO:tasks.rados.rados.2.smithi175.stdout:265: finishing write tid 4 to smithi17557870-5 2024-09-11T03:39:15.008 INFO:tasks.rados.rados.2.smithi175.stdout:265: oid 5 updating version 0 to 18 2024-09-11T03:39:15.008 INFO:tasks.rados.rados.2.smithi175.stdout:265: oid 5 updating version 18 to 19 2024-09-11T03:39:15.008 INFO:tasks.rados.rados.2.smithi175.stdout:265: oid 5 version 19 is already newer than 17 2024-09-11T03:39:15.008 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 19 (ObjNum 122 snap 19 seq_num 122) dirty exists 2024-09-11T03:39:15.008 INFO:tasks.rados.rados.2.smithi175.stdout:265: left oid 5 (ObjNum 122 snap 19 seq_num 122) 2024-09-11T03:39:15.008 INFO:tasks.rados.rados.2.smithi175.stdout:263: done (2 left) 2024-09-11T03:39:15.008 INFO:tasks.rados.rados.2.smithi175.stdout:265: done (1 left) 2024-09-11T03:39:15.008 INFO:tasks.rados.rados.2.smithi175.stdout:267: write oid 11 current snap is 19 2024-09-11T03:39:15.009 INFO:tasks.rados.rados.2.smithi175.stdout:267: seq_num 123 ranges {467261=717275,1847991=219270} 2024-09-11T03:39:15.014 INFO:tasks.rados.rados.2.smithi175.stdout:267: writing smithi17557870-11 from 467261 to 1184536 tid 1 2024-09-11T03:39:15.019 INFO:tasks.rados.rados.2.smithi175.stdout:267: writing smithi17557870-11 from 1847991 to 2067261 tid 2 2024-09-11T03:39:15.019 INFO:tasks.rados.rados.2.smithi175.stdout:268: snap_create 2024-09-11T03:39:15.019 INFO:tasks.rados.rados.2.smithi175.stdout:266: finishing copy_from racing read to smithi17557870-35 2024-09-11T03:39:15.025 INFO:tasks.rados.rados.2.smithi175.stdout:267: finishing write tid 1 to smithi17557870-11 2024-09-11T03:39:15.025 INFO:tasks.rados.rados.2.smithi175.stdout:266: finishing copy_from to smithi17557870-35 2024-09-11T03:39:15.025 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 39 (ObjNum 88 snap 12 seq_num 88) dirty exists 2024-09-11T03:39:15.025 INFO:tasks.rados.rados.2.smithi175.stdout:267: finishing write tid 2 to smithi17557870-11 2024-09-11T03:39:15.025 INFO:tasks.rados.rados.2.smithi175.stdout:267: finishing write tid 3 to smithi17557870-11 2024-09-11T03:39:15.025 INFO:tasks.rados.rados.2.smithi175.stdout:267: finishing write tid 4 to smithi17557870-11 2024-09-11T03:39:15.026 INFO:tasks.rados.rados.2.smithi175.stdout:267: oid 11 updating version 0 to 33 2024-09-11T03:39:15.026 INFO:tasks.rados.rados.2.smithi175.stdout:267: oid 11 updating version 33 to 34 2024-09-11T03:39:15.026 INFO:tasks.rados.rados.2.smithi175.stdout:267: oid 11 version 34 is already newer than 32 2024-09-11T03:39:15.026 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 34 (ObjNum 123 snap 19 seq_num 123) dirty exists 2024-09-11T03:39:15.026 INFO:tasks.rados.rados.2.smithi175.stdout:267: left oid 11 (ObjNum 123 snap 19 seq_num 123) 2024-09-11T03:39:15.074 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:15.080 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:15.102 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:15.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:15.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:15.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:14 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/491231820' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:39:15.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:14 smithi121 ceph-mon[30729]: osdmap e137: 12 total, 12 up, 12 in 2024-09-11T03:39:15.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:14 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/491231820' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:39:15.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:14 smithi084 ceph-mon[27254]: osdmap e137: 12 total, 12 up, 12 in 2024-09-11T03:39:15.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:15.225 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:15.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:15.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:15.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:14 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/491231820' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:39:15.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:14 smithi175 ceph-mon[30894]: osdmap e137: 12 total, 12 up, 12 in 2024-09-11T03:39:15.324 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:39:15.324 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:39:15.324 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:39:15.324 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:39:15.325 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:39:15.327 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:15.352 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:15.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:15.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:15.454 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:15.476 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:15.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:15.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:15.578 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:15.601 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:15.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:15.602 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:15.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:15.725 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:15.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:15.828 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:15.828 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:39:15.841 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.11 to 1 (10000) 2024-09-11T03:39:15.841 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:39:15.848 INFO:tasks.rados.rados.2.smithi175.stdout:266: done (2 left) 2024-09-11T03:39:15.848 INFO:tasks.rados.rados.2.smithi175.stdout:267: done (1 left) 2024-09-11T03:39:15.848 INFO:tasks.rados.rados.2.smithi175.stdout:268: done (0 left) 2024-09-11T03:39:15.848 INFO:tasks.rados.rados.2.smithi175.stdout:269: snap_remove snap 19 2024-09-11T03:39:15.853 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:15.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:15.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:15.955 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:15.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T03:39:15.980 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:15.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:15.981 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:16.082 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:16.108 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:16.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:16.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:15 smithi084 ceph-mon[27254]: pgmap v361: 17 pgs: 17 active+clean; 130 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 7.6 MiB/s wr, 41 op/s 2024-09-11T03:39:16.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:15 smithi084 ceph-mon[27254]: osdmap e138: 12 total, 12 up, 12 in 2024-09-11T03:39:16.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:15 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3813240821' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:16.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:15 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3105322119' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:39:16.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:15 smithi121 ceph-mon[30729]: pgmap v361: 17 pgs: 17 active+clean; 130 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 7.6 MiB/s wr, 41 op/s 2024-09-11T03:39:16.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:15 smithi121 ceph-mon[30729]: osdmap e138: 12 total, 12 up, 12 in 2024-09-11T03:39:16.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:15 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3813240821' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:16.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:15 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3105322119' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:39:16.201 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:39:16.209 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:16.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:15 smithi175 ceph-mon[30894]: pgmap v361: 17 pgs: 17 active+clean; 130 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 7.6 MiB/s wr, 41 op/s 2024-09-11T03:39:16.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:15 smithi175 ceph-mon[30894]: osdmap e138: 12 total, 12 up, 12 in 2024-09-11T03:39:16.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:15 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3813240821' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:16.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:15 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3105322119' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:39:16.230 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:16.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:16.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:16.332 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:16.355 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:16.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:16.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:16.363 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:16.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:16.478 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:16.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:16.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:16.602 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:16.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:16.704 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:16.715 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:39:16.716 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:39:16.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:39:16.716 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:39:16.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.716 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:221 2.3 2:c6c49f2c:::smithi17557870-44:head [copy-from ver 9] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:39:16.716 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941456+0000", 2024-09-11T03:39:16.716 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.773193194000001, 2024-09-11T03:39:16.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015312882, 2024-09-11T03:39:16.716 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.716 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.716 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:16.716 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.717 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.717 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.717 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 221 2024-09-11T03:39:16.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.717 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941456+0000", 2024-09-11T03:39:16.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941456+0000", 2024-09-11T03:39:16.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941459+0000", 2024-09-11T03:39:16.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9189999999999999e-06 2024-09-11T03:39:16.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941469+0000", 2024-09-11T03:39:16.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0069e-05 2024-09-11T03:39:16.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941490+0000", 2024-09-11T03:39:16.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0947000000000001e-05 2024-09-11T03:39:16.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941501+0000", 2024-09-11T03:39:16.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0169e-05 2024-09-11T03:39:16.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941585+0000", 2024-09-11T03:39:16.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4022000000000003e-05 2024-09-11T03:39:16.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:39:16.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941590+0000", 2024-09-11T03:39:16.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4929999999999998e-06 2024-09-11T03:39:16.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.942889+0000", 2024-09-11T03:39:16.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001298539 2024-09-11T03:39:16.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.942984+0000", 2024-09-11T03:39:16.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5513999999999994e-05 2024-09-11T03:39:16.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:39:16.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.950816+0000", 2024-09-11T03:39:16.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078315850000000003 2024-09-11T03:39:16.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956638+0000", 2024-09-11T03:39:16.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058225769999999998 2024-09-11T03:39:16.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:39:16.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956649+0000", 2024-09-11T03:39:16.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0492000000000001e-05 2024-09-11T03:39:16.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956690+0000", 2024-09-11T03:39:16.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.142e-05 2024-09-11T03:39:16.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:16.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956728+0000", 2024-09-11T03:39:16.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8195000000000003e-05 2024-09-11T03:39:16.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956769+0000", 2024-09-11T03:39:16.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0941e-05 2024-09-11T03:39:16.723 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.723 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.723 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.723 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:222 2.3 2:c6c49f2c:::smithi17557870-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:39:16.723 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:39:16.723 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.773115115000003, 2024-09-11T03:39:16.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015032419, 2024-09-11T03:39:16.724 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.724 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.724 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:16.724 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.724 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.724 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.724 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 222 2024-09-11T03:39:16.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.724 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:39:16.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:39:16.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:39:16.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.51e-07 2024-09-11T03:39:16.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941539+0000", 2024-09-11T03:39:16.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0060000000000003e-06 2024-09-11T03:39:16.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941547+0000", 2024-09-11T03:39:16.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9009999999999994e-06 2024-09-11T03:39:16.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941553+0000", 2024-09-11T03:39:16.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1519999999999997e-06 2024-09-11T03:39:16.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941691+0000", 2024-09-11T03:39:16.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013752600000000001 2024-09-11T03:39:16.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:16.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941697+0000", 2024-09-11T03:39:16.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3770000000000001e-06 2024-09-11T03:39:16.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943170+0000", 2024-09-11T03:39:16.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001473504 2024-09-11T03:39:16.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:39:16.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943225+0000", 2024-09-11T03:39:16.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4101000000000003e-05 2024-09-11T03:39:16.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956388+0000", 2024-09-11T03:39:16.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013163223999999999 2024-09-11T03:39:16.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956456+0000", 2024-09-11T03:39:16.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8563000000000001e-05 2024-09-11T03:39:16.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956567+0000", 2024-09-11T03:39:16.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011061399999999999 2024-09-11T03:39:16.729 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.729 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.729 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.729 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:223 2.3 2:c3768452:::smithi17557870-11:head [copy-from ver 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:39:16.729 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:39:16.729 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.773075069999997, 2024-09-11T03:39:16.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016727301999999999, 2024-09-11T03:39:16.730 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.730 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.730 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:16.730 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.730 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.731 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.731 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 223 2024-09-11T03:39:16.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.731 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:39:16.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:39:16.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:39:16.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0999999999999999e-07 2024-09-11T03:39:16.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941578+0000", 2024-09-11T03:39:16.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.021e-06 2024-09-11T03:39:16.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941583+0000", 2024-09-11T03:39:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0520000000000001e-06 2024-09-11T03:39:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941586+0000", 2024-09-11T03:39:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-09-11T03:39:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941750+0000", 2024-09-11T03:39:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016353200000000001 2024-09-11T03:39:16.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:16.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941756+0000", 2024-09-11T03:39:16.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.75e-06 2024-09-11T03:39:16.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943289+0000", 2024-09-11T03:39:16.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001533747 2024-09-11T03:39:16.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943383+0000", 2024-09-11T03:39:16.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3655999999999999e-05 2024-09-11T03:39:16.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:39:16.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.954899+0000", 2024-09-11T03:39:16.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011515463 2024-09-11T03:39:16.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:39:16.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.957952+0000", 2024-09-11T03:39:16.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030533560000000001 2024-09-11T03:39:16.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958200+0000", 2024-09-11T03:39:16.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024759100000000002 2024-09-11T03:39:16.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958229+0000", 2024-09-11T03:39:16.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9862e-05 2024-09-11T03:39:16.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:16.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958262+0000", 2024-09-11T03:39:16.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2267000000000001e-05 2024-09-11T03:39:16.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958302+0000", 2024-09-11T03:39:16.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0204000000000003e-05 2024-09-11T03:39:16.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.737 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.737 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:224 2.3 2:c3768452:::smithi17557870-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:39:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941599+0000", 2024-09-11T03:39:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.773050517999998, 2024-09-11T03:39:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014750406000000001, 2024-09-11T03:39:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 224 2024-09-11T03:39:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941599+0000", 2024-09-11T03:39:16.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941599+0000", 2024-09-11T03:39:16.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941599+0000", 2024-09-11T03:39:16.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2599999999999998e-07 2024-09-11T03:39:16.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941601+0000", 2024-09-11T03:39:16.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6249999999999999e-06 2024-09-11T03:39:16.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941605+0000", 2024-09-11T03:39:16.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3059999999999999e-06 2024-09-11T03:39:16.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941609+0000", 2024-09-11T03:39:16.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.506e-06 2024-09-11T03:39:16.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941789+0000", 2024-09-11T03:39:16.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000180261 2024-09-11T03:39:16.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:16.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941795+0000", 2024-09-11T03:39:16.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4970000000000001e-06 2024-09-11T03:39:16.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943572+0000", 2024-09-11T03:39:16.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001777725 2024-09-11T03:39:16.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:39:16.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943616+0000", 2024-09-11T03:39:16.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3550999999999997e-05 2024-09-11T03:39:16.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956192+0000", 2024-09-11T03:39:16.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012576517000000001 2024-09-11T03:39:16.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956267+0000", 2024-09-11T03:39:16.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4956000000000005e-05 2024-09-11T03:39:16.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956350+0000", 2024-09-11T03:39:16.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2136000000000001e-05 2024-09-11T03:39:16.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.743 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.744 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:225 2.2 2:43d79ac9:::smithi17557870-37:head [delete] snapc 3=[3] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:39:16.744 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941614+0000", 2024-09-11T03:39:16.744 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.773035972000002, 2024-09-11T03:39:16.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002934519, 2024-09-11T03:39:16.744 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.744 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.744 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:16.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.744 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 225 2024-09-11T03:39:16.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.744 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941614+0000", 2024-09-11T03:39:16.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941614+0000", 2024-09-11T03:39:16.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941614+0000", 2024-09-11T03:39:16.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.23e-07 2024-09-11T03:39:16.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941615+0000", 2024-09-11T03:39:16.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2240000000000001e-06 2024-09-11T03:39:16.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941619+0000", 2024-09-11T03:39:16.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6969999999999999e-06 2024-09-11T03:39:16.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941622+0000", 2024-09-11T03:39:16.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1480000000000002e-06 2024-09-11T03:39:16.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941699+0000", 2024-09-11T03:39:16.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6907000000000001e-05 2024-09-11T03:39:16.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:39:16.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941703+0000", 2024-09-11T03:39:16.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.454e-06 2024-09-11T03:39:16.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.942857+0000", 2024-09-11T03:39:16.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011535930000000001 2024-09-11T03:39:16.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.942929+0000", 2024-09-11T03:39:16.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1657999999999997e-05 2024-09-11T03:39:16.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,6", 2024-09-11T03:39:16.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943079+0000", 2024-09-11T03:39:16.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015060099999999999 2024-09-11T03:39:16.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:39:16.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943923+0000", 2024-09-11T03:39:16.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00084404200000000003 2024-09-11T03:39:16.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.944391+0000", 2024-09-11T03:39:16.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00046739500000000001 2024-09-11T03:39:16.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.944439+0000", 2024-09-11T03:39:16.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7973999999999998e-05 2024-09-11T03:39:16.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:16.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.944496+0000", 2024-09-11T03:39:16.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7781999999999997e-05 2024-09-11T03:39:16.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.944548+0000", 2024-09-11T03:39:16.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1820999999999997e-05 2024-09-11T03:39:16.751 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.751 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.751 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.752 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:343 2.2 2:494ba932:::smithi17557870-43:head [copy-from ver 6] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:39:16.752 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.820776+0000", 2024-09-11T03:39:16.752 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 26.893873928000001, 2024-09-11T03:39:16.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017461323000000001, 2024-09-11T03:39:16.752 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.752 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.752 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:16.752 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.752 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.752 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.752 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 343 2024-09-11T03:39:16.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.752 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820776+0000", 2024-09-11T03:39:16.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820776+0000", 2024-09-11T03:39:16.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820779+0000", 2024-09-11T03:39:16.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3000000000000002e-06 2024-09-11T03:39:16.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820797+0000", 2024-09-11T03:39:16.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7730000000000001e-05 2024-09-11T03:39:16.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820853+0000", 2024-09-11T03:39:16.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5909000000000003e-05 2024-09-11T03:39:16.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820872+0000", 2024-09-11T03:39:16.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9709999999999999e-05 2024-09-11T03:39:16.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820976+0000", 2024-09-11T03:39:16.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010374099999999999 2024-09-11T03:39:16.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:39:16.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820982+0000", 2024-09-11T03:39:16.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4389999999999999e-06 2024-09-11T03:39:16.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825547+0000", 2024-09-11T03:39:16.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045658260000000003 2024-09-11T03:39:16.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825626+0000", 2024-09-11T03:39:16.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8289999999999998e-05 2024-09-11T03:39:16.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,6", 2024-09-11T03:39:16.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830895+0000", 2024-09-11T03:39:16.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052695060000000002 2024-09-11T03:39:16.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:39:16.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.835040+0000", 2024-09-11T03:39:16.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041446549999999997 2024-09-11T03:39:16.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.835364+0000", 2024-09-11T03:39:16.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032384200000000002 2024-09-11T03:39:16.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838162+0000", 2024-09-11T03:39:16.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027987509999999999 2024-09-11T03:39:16.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:16.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838203+0000", 2024-09-11T03:39:16.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0209000000000001e-05 2024-09-11T03:39:16.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838237+0000", 2024-09-11T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4415000000000002e-05 2024-09-11T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:344 2.2 2:494ba932:::smithi17557870-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.820917+0000", 2024-09-11T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 26.893732302, 2024-09-11T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013455992, 2024-09-11T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.759 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:16.760 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.760 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.760 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.760 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 344 2024-09-11T03:39:16.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.760 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820917+0000", 2024-09-11T03:39:16.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:16.761 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:16.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:16.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820917+0000", 2024-09-11T03:39:16.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820918+0000", 2024-09-11T03:39:16.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.34e-07 2024-09-11T03:39:16.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820920+0000", 2024-09-11T03:39:16.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.48e-06 2024-09-11T03:39:16.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820927+0000", 2024-09-11T03:39:16.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.187e-06 2024-09-11T03:39:16.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820931+0000", 2024-09-11T03:39:16.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9349999999999996e-06 2024-09-11T03:39:16.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.821034+0000", 2024-09-11T03:39:16.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010277600000000001 2024-09-11T03:39:16.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:16.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.821040+0000", 2024-09-11T03:39:16.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6389999999999997e-06 2024-09-11T03:39:16.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825767+0000", 2024-09-11T03:39:16.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047268060000000001 2024-09-11T03:39:16.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:39:16.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825844+0000", 2024-09-11T03:39:16.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7066999999999999e-05 2024-09-11T03:39:16.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834208+0000", 2024-09-11T03:39:16.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083646199999999997 2024-09-11T03:39:16.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834284+0000", 2024-09-11T03:39:16.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5765000000000004e-05 2024-09-11T03:39:16.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834373+0000", 2024-09-11T03:39:16.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9283000000000006e-05 2024-09-11T03:39:16.767 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.767 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.768 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.768 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:1 2.2 2:43d79ac9:::smithi17557870-37:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:39:16.768 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.823045+0000", 2024-09-11T03:39:16.768 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 26.891604295, 2024-09-11T03:39:16.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050982270000000003, 2024-09-11T03:39:16.768 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.768 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.768 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:16.768 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.823045+0000", 2024-09-11T03:39:16.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.823045+0000", 2024-09-11T03:39:16.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.823046+0000", 2024-09-11T03:39:16.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.048e-06 2024-09-11T03:39:16.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.823052+0000", 2024-09-11T03:39:16.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1030000000000001e-06 2024-09-11T03:39:16.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.823061+0000", 2024-09-11T03:39:16.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0520000000000007e-06 2024-09-11T03:39:16.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.823069+0000", 2024-09-11T03:39:16.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0169999999999996e-06 2024-09-11T03:39:16.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.823136+0000", 2024-09-11T03:39:16.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7367999999999996e-05 2024-09-11T03:39:16.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:39:16.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.823143+0000", 2024-09-11T03:39:16.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5919999999999997e-06 2024-09-11T03:39:16.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825875+0000", 2024-09-11T03:39:16.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027328949999999999 2024-09-11T03:39:16.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825924+0000", 2024-09-11T03:39:16.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8912999999999997e-05 2024-09-11T03:39:16.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828144+0000", 2024-09-11T03:39:16.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002219239 2024-09-11T03:39:16.773 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.773 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.773 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.773 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:407 2.3 2:c3768452:::smithi17557870-11:a [read 0~2033614,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116)", 2024-09-11T03:39:16.773 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:52.814662+0000", 2024-09-11T03:39:16.773 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.899987877000001, 2024-09-11T03:39:16.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037255040000000001, 2024-09-11T03:39:16.773 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.773 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.773 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:16.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.774 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.774 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.774 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 407 2024-09-11T03:39:16.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.774 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814662+0000", 2024-09-11T03:39:16.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814662+0000", 2024-09-11T03:39:16.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814664+0000", 2024-09-11T03:39:16.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3910000000000001e-06 2024-09-11T03:39:16.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814675+0000", 2024-09-11T03:39:16.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0923e-05 2024-09-11T03:39:16.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814694+0000", 2024-09-11T03:39:16.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.929e-05 2024-09-11T03:39:16.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814723+0000", 2024-09-11T03:39:16.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8394e-05 2024-09-11T03:39:16.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814810+0000", 2024-09-11T03:39:16.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6970000000000002e-05 2024-09-11T03:39:16.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:39:16.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814819+0000", 2024-09-11T03:39:16.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2699999999999993e-06 2024-09-11T03:39:16.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.816030+0000", 2024-09-11T03:39:16.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012108150000000001 2024-09-11T03:39:16.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.816106+0000", 2024-09-11T03:39:16.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6631000000000005e-05 2024-09-11T03:39:16.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.818387+0000", 2024-09-11T03:39:16.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022808199999999998 2024-09-11T03:39:16.778 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.778 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.778 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.778 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:408 2.3 2:c3768452:::smithi17557870-11:a [sparse-read 0~1116110] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116)", 2024-09-11T03:39:16.778 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:52.814745+0000", 2024-09-11T03:39:16.778 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.899904768999999, 2024-09-11T03:39:16.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038057899999999999, 2024-09-11T03:39:16.779 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.779 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.779 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:16.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.779 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 408 2024-09-11T03:39:16.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.779 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814745+0000", 2024-09-11T03:39:16.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814745+0000", 2024-09-11T03:39:16.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814745+0000", 2024-09-11T03:39:16.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.34e-07 2024-09-11T03:39:16.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814748+0000", 2024-09-11T03:39:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.311e-06 2024-09-11T03:39:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814753+0000", 2024-09-11T03:39:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0699999999999997e-06 2024-09-11T03:39:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814756+0000", 2024-09-11T03:39:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.568e-06 2024-09-11T03:39:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814904+0000", 2024-09-11T03:39:16.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014731400000000001 2024-09-11T03:39:16.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:16.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814909+0000", 2024-09-11T03:39:16.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4700000000000001e-06 2024-09-11T03:39:16.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.818413+0000", 2024-09-11T03:39:16.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035035940000000001 2024-09-11T03:39:16.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.818465+0000", 2024-09-11T03:39:16.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2145000000000003e-05 2024-09-11T03:39:16.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.818551+0000", 2024-09-11T03:39:16.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5884000000000006e-05 2024-09-11T03:39:16.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.783 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:409 2.3 2:c3768452:::smithi17557870-11:a [read 0~2033614,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116)", 2024-09-11T03:39:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:52.814805+0000", 2024-09-11T03:39:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.899844585, 2024-09-11T03:39:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047010389999999997, 2024-09-11T03:39:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 409 2024-09-11T03:39:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814805+0000", 2024-09-11T03:39:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814805+0000", 2024-09-11T03:39:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814805+0000", 2024-09-11T03:39:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3099999999999999e-07 2024-09-11T03:39:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814808+0000", 2024-09-11T03:39:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5179999999999999e-06 2024-09-11T03:39:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814813+0000", 2024-09-11T03:39:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9949999999999996e-06 2024-09-11T03:39:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814817+0000", 2024-09-11T03:39:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6839999999999998e-06 2024-09-11T03:39:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814941+0000", 2024-09-11T03:39:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000124351 2024-09-11T03:39:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.814946+0000", 2024-09-11T03:39:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4890000000000003e-06 2024-09-11T03:39:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.818590+0000", 2024-09-11T03:39:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036439559999999998 2024-09-11T03:39:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.818655+0000", 2024-09-11T03:39:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4830000000000001e-05 2024-09-11T03:39:16.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:52.819506+0000", 2024-09-11T03:39:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00085088500000000001 2024-09-11T03:39:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:583 2.2 2:494ba932:::smithi17557870-43:head [stat,delete] snapc 11=[11,10,f,e] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-11T03:39:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:04.875310+0000", 2024-09-11T03:39:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.839339496000001, 2024-09-11T03:39:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030389340000000001, 2024-09-11T03:39:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 583 2024-09-11T03:39:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.875310+0000", 2024-09-11T03:39:16.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.875310+0000", 2024-09-11T03:39:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.875313+0000", 2024-09-11T03:39:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6029999999999999e-06 2024-09-11T03:39:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.875320+0000", 2024-09-11T03:39:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0380000000000002e-06 2024-09-11T03:39:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.875339+0000", 2024-09-11T03:39:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9613e-05 2024-09-11T03:39:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.875366+0000", 2024-09-11T03:39:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6423000000000001e-05 2024-09-11T03:39:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.875432+0000", 2024-09-11T03:39:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5718999999999999e-05 2024-09-11T03:39:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:39:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.875437+0000", 2024-09-11T03:39:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0810000000000001e-06 2024-09-11T03:39:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.876709+0000", 2024-09-11T03:39:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012726879999999999 2024-09-11T03:39:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.876781+0000", 2024-09-11T03:39:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2026000000000005e-05 2024-09-11T03:39:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,6", 2024-09-11T03:39:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.876948+0000", 2024-09-11T03:39:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016704799999999999 2024-09-11T03:39:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:39:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.877697+0000", 2024-09-11T03:39:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074867399999999995 2024-09-11T03:39:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.878091+0000", 2024-09-11T03:39:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039376400000000001 2024-09-11T03:39:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.878277+0000", 2024-09-11T03:39:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000186088 2024-09-11T03:39:16.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:16.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.878314+0000", 2024-09-11T03:39:16.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7401999999999999e-05 2024-09-11T03:39:16.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.878349+0000", 2024-09-11T03:39:16.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4767000000000001e-05 2024-09-11T03:39:16.796 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.796 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.796 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.797 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:607 2.3 2:c3768452:::smithi17557870-11:head [write 564735~613734,stat] snapc 11=[f,e] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-11T03:39:16.797 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:06.914163+0000", 2024-09-11T03:39:16.797 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.8004867769999997, 2024-09-11T03:39:16.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054773620000000004, 2024-09-11T03:39:16.797 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.797 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.797 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:16.797 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.797 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.797 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.797 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 607 2024-09-11T03:39:16.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.797 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.914163+0000", 2024-09-11T03:39:16.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.914163+0000", 2024-09-11T03:39:16.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.914166+0000", 2024-09-11T03:39:16.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2430000000000001e-06 2024-09-11T03:39:16.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.916572+0000", 2024-09-11T03:39:16.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024055209999999999 2024-09-11T03:39:16.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.916590+0000", 2024-09-11T03:39:16.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8487000000000001e-05 2024-09-11T03:39:16.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.916616+0000", 2024-09-11T03:39:16.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6152e-05 2024-09-11T03:39:16.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.916688+0000", 2024-09-11T03:39:16.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1240000000000002e-05 2024-09-11T03:39:16.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.916769+0000", 2024-09-11T03:39:16.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1767000000000005e-05 2024-09-11T03:39:16.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:39:16.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.916931+0000", 2024-09-11T03:39:16.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016213800000000001 2024-09-11T03:39:16.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:39:16.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.918750+0000", 2024-09-11T03:39:16.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001818297 2024-09-11T03:39:16.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.918987+0000", 2024-09-11T03:39:16.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000237407 2024-09-11T03:39:16.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.919548+0000", 2024-09-11T03:39:16.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056076800000000001 2024-09-11T03:39:16.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:16.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.919602+0000", 2024-09-11T03:39:16.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4449e-05 2024-09-11T03:39:16.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.919640+0000", 2024-09-11T03:39:16.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7892999999999998e-05 2024-09-11T03:39:16.803 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.803 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.803 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.803 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:608 2.3 2:c3768452:::smithi17557870-11:head [write 1828021~669034] snapc 11=[f,e] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-11T03:39:16.803 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:06.921074+0000", 2024-09-11T03:39:16.803 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.7935756689999991, 2024-09-11T03:39:16.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026918160000000001, 2024-09-11T03:39:16.803 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.803 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.804 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:16.804 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.804 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.804 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.804 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 608 2024-09-11T03:39:16.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.804 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.921074+0000", 2024-09-11T03:39:16.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.921074+0000", 2024-09-11T03:39:16.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.921076+0000", 2024-09-11T03:39:16.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5880000000000001e-06 2024-09-11T03:39:16.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.921755+0000", 2024-09-11T03:39:16.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00067913399999999997 2024-09-11T03:39:16.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.921770+0000", 2024-09-11T03:39:16.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5642e-05 2024-09-11T03:39:16.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.921777+0000", 2024-09-11T03:39:16.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7850000000000003e-06 2024-09-11T03:39:16.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.921846+0000", 2024-09-11T03:39:16.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9355000000000003e-05 2024-09-11T03:39:16.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.921927+0000", 2024-09-11T03:39:16.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0197000000000005e-05 2024-09-11T03:39:16.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:39:16.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.922038+0000", 2024-09-11T03:39:16.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011087 2024-09-11T03:39:16.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:39:16.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.923197+0000", 2024-09-11T03:39:16.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011590459999999999 2024-09-11T03:39:16.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.923621+0000", 2024-09-11T03:39:16.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00042454599999999998 2024-09-11T03:39:16.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.923689+0000", 2024-09-11T03:39:16.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7497999999999996e-05 2024-09-11T03:39:16.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:16.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.923730+0000", 2024-09-11T03:39:16.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1786999999999998e-05 2024-09-11T03:39:16.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.923766+0000", 2024-09-11T03:39:16.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5367999999999997e-05 2024-09-11T03:39:16.809 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.809 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.809 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.810 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:609 2.3 2:c3768452:::smithi17557870-11:head [write 3134120~630615] snapc 11=[f,e] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-11T03:39:16.810 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:06.927804+0000", 2024-09-11T03:39:16.810 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.7868456809999991, 2024-09-11T03:39:16.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002683936, 2024-09-11T03:39:16.810 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.810 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.810 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:16.810 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.810 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.810 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.810 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 609 2024-09-11T03:39:16.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.810 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.927804+0000", 2024-09-11T03:39:16.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.927804+0000", 2024-09-11T03:39:16.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.927805+0000", 2024-09-11T03:39:16.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.44e-06 2024-09-11T03:39:16.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.928397+0000", 2024-09-11T03:39:16.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059154300000000004 2024-09-11T03:39:16.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.928413+0000", 2024-09-11T03:39:16.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5696999999999999e-05 2024-09-11T03:39:16.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.928420+0000", 2024-09-11T03:39:16.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1060000000000002e-06 2024-09-11T03:39:16.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.928479+0000", 2024-09-11T03:39:16.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9027000000000003e-05 2024-09-11T03:39:16.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.928588+0000", 2024-09-11T03:39:16.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108756 2024-09-11T03:39:16.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:39:16.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.928768+0000", 2024-09-11T03:39:16.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018061600000000001 2024-09-11T03:39:16.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:39:16.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.930045+0000", 2024-09-11T03:39:16.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012771289999999999 2024-09-11T03:39:16.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.930334+0000", 2024-09-11T03:39:16.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028897500000000002 2024-09-11T03:39:16.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.930413+0000", 2024-09-11T03:39:16.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8807999999999995e-05 2024-09-11T03:39:16.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:16.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.930450+0000", 2024-09-11T03:39:16.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6733999999999998e-05 2024-09-11T03:39:16.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.930488+0000", 2024-09-11T03:39:16.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8105000000000002e-05 2024-09-11T03:39:16.816 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.816 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.816 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.816 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:621 2.3 2:c3768452:::smithi17557870-11:e [read 0~2033614,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e130)", 2024-09-11T03:39:16.816 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:07.066749+0000", 2024-09-11T03:39:16.816 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.6479010570000003, 2024-09-11T03:39:16.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002668419, 2024-09-11T03:39:16.816 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.816 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.816 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:16.817 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.817 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.817 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.817 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 621 2024-09-11T03:39:16.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.817 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066749+0000", 2024-09-11T03:39:16.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066749+0000", 2024-09-11T03:39:16.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066749+0000", 2024-09-11T03:39:16.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8100000000000003e-07 2024-09-11T03:39:16.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066755+0000", 2024-09-11T03:39:16.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.229e-06 2024-09-11T03:39:16.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066765+0000", 2024-09-11T03:39:16.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3519999999999996e-06 2024-09-11T03:39:16.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066773+0000", 2024-09-11T03:39:16.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9959999999999993e-06 2024-09-11T03:39:16.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.068329+0000", 2024-09-11T03:39:16.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015563630000000001 2024-09-11T03:39:16.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.068381+0000", 2024-09-11T03:39:16.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1842999999999998e-05 2024-09-11T03:39:16.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.069417+0000", 2024-09-11T03:39:16.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010360549999999999 2024-09-11T03:39:16.820 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.820 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.820 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.821 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:754 2.3 2:c3768452:::smithi17557870-11:head [write 467261~717275] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:39:16.821 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:15.016104+0000", 2024-09-11T03:39:16.821 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.6985454689999999, 2024-09-11T03:39:16.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010129861, 2024-09-11T03:39:16.821 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.821 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.821 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:16.821 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.821 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.821 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.821 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 754 2024-09-11T03:39:16.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.822 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.016104+0000", 2024-09-11T03:39:16.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.016104+0000", 2024-09-11T03:39:16.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.016108+0000", 2024-09-11T03:39:16.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8060000000000001e-06 2024-09-11T03:39:16.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.018233+0000", 2024-09-11T03:39:16.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021249900000000002 2024-09-11T03:39:16.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.018249+0000", 2024-09-11T03:39:16.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6052000000000001e-05 2024-09-11T03:39:16.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.018260+0000", 2024-09-11T03:39:16.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1100999999999999e-05 2024-09-11T03:39:16.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.018347+0000", 2024-09-11T03:39:16.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7194000000000006e-05 2024-09-11T03:39:16.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:39:16.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.018352+0000", 2024-09-11T03:39:16.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9559999999999996e-06 2024-09-11T03:39:16.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.019567+0000", 2024-09-11T03:39:16.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012144899999999999 2024-09-11T03:39:16.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.019630+0000", 2024-09-11T03:39:16.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2980999999999999e-05 2024-09-11T03:39:16.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:39:16.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.019722+0000", 2024-09-11T03:39:16.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1965000000000005e-05 2024-09-11T03:39:16.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:39:16.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022248+0000", 2024-09-11T03:39:16.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025267409999999999 2024-09-11T03:39:16.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022729+0000", 2024-09-11T03:39:16.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00048018799999999999 2024-09-11T03:39:16.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026172+0000", 2024-09-11T03:39:16.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034435500000000001 2024-09-11T03:39:16.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:16.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026205+0000", 2024-09-11T03:39:16.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2703000000000002e-05 2024-09-11T03:39:16.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026234+0000", 2024-09-11T03:39:16.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9144000000000001e-05 2024-09-11T03:39:16.827 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.827 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.828 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.828 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:755 2.3 2:c3768452:::smithi17557870-11:head [write 1847991~219270] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:39:16.828 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:15.020950+0000", 2024-09-11T03:39:16.828 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.693699702, 2024-09-11T03:39:16.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056636869999999997, 2024-09-11T03:39:16.828 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.828 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.828 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:16.828 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.828 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.828 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.829 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 755 2024-09-11T03:39:16.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.829 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.020950+0000", 2024-09-11T03:39:16.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.020950+0000", 2024-09-11T03:39:16.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.020951+0000", 2024-09-11T03:39:16.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5340000000000001e-06 2024-09-11T03:39:16.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021252+0000", 2024-09-11T03:39:16.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000300019 2024-09-11T03:39:16.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021271+0000", 2024-09-11T03:39:16.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9612000000000001e-05 2024-09-11T03:39:16.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021281+0000", 2024-09-11T03:39:16.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5210000000000008e-06 2024-09-11T03:39:16.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021405+0000", 2024-09-11T03:39:16.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012406599999999999 2024-09-11T03:39:16.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021483+0000", 2024-09-11T03:39:16.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8788000000000005e-05 2024-09-11T03:39:16.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:39:16.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021643+0000", 2024-09-11T03:39:16.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015989500000000001 2024-09-11T03:39:16.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:39:16.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022823+0000", 2024-09-11T03:39:16.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011795670000000001 2024-09-11T03:39:16.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.023025+0000", 2024-09-11T03:39:16.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000202072 2024-09-11T03:39:16.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026543+0000", 2024-09-11T03:39:16.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035177329999999999 2024-09-11T03:39:16.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:16.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026579+0000", 2024-09-11T03:39:16.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6513000000000001e-05 2024-09-11T03:39:16.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026614+0000", 2024-09-11T03:39:16.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4366999999999998e-05 2024-09-11T03:39:16.834 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.834 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.834 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.834 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:756 2.3 2:c3768452:::smithi17557870-11:head [setxattr _header (57) in=7b,truncate 2067261] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:39:16.834 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:15.021301+0000", 2024-09-11T03:39:16.834 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.6933481970000002, 2024-09-11T03:39:16.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054061910000000003, 2024-09-11T03:39:16.835 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.835 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.835 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:16.835 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.835 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.835 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.835 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 756 2024-09-11T03:39:16.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.835 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021301+0000", 2024-09-11T03:39:16.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021301+0000", 2024-09-11T03:39:16.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021302+0000", 2024-09-11T03:39:16.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5000000000000002e-07 2024-09-11T03:39:16.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021307+0000", 2024-09-11T03:39:16.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0980000000000001e-06 2024-09-11T03:39:16.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021315+0000", 2024-09-11T03:39:16.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0709999999999995e-06 2024-09-11T03:39:16.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021321+0000", 2024-09-11T03:39:16.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6779999999999997e-06 2024-09-11T03:39:16.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022288+0000", 2024-09-11T03:39:16.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00096644199999999997 2024-09-11T03:39:16.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022341+0000", 2024-09-11T03:39:16.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3677000000000001e-05 2024-09-11T03:39:16.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:39:16.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022432+0000", 2024-09-11T03:39:16.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0494999999999993e-05 2024-09-11T03:39:16.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:39:16.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022835+0000", 2024-09-11T03:39:16.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040375600000000002 2024-09-11T03:39:16.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.023136+0000", 2024-09-11T03:39:16.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030039599999999997 2024-09-11T03:39:16.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:16.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026642+0000", 2024-09-11T03:39:16.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003505991 2024-09-11T03:39:16.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:16.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026674+0000", 2024-09-11T03:39:16.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2289000000000002e-05 2024-09-11T03:39:16.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026708+0000", 2024-09-11T03:39:16.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3547999999999998e-05 2024-09-11T03:39:16.841 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.841 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.841 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.841 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:757 2.3 2:c3768452:::smithi17557870-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:39:16.841 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:15.021330+0000", 2024-09-11T03:39:16.841 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.6933194220000001, 2024-09-11T03:39:16.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055098129999999997, 2024-09-11T03:39:16.841 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:16.841 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:16.842 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:16.842 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:16.842 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:16.842 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:16.842 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 757 2024-09-11T03:39:16.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.842 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:16.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:16.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021330+0000", 2024-09-11T03:39:16.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:16.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021330+0000", 2024-09-11T03:39:16.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:16.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:16.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021331+0000", 2024-09-11T03:39:16.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7100000000000002e-07 2024-09-11T03:39:16.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:16.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021333+0000", 2024-09-11T03:39:16.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0940000000000002e-06 2024-09-11T03:39:16.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:16.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021347+0000", 2024-09-11T03:39:16.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3899e-05 2024-09-11T03:39:16.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:16.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021353+0000", 2024-09-11T03:39:16.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9059999999999999e-06 2024-09-11T03:39:16.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022759+0000", 2024-09-11T03:39:16.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0014062879999999999 2024-09-11T03:39:16.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:39:16.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022809+0000", 2024-09-11T03:39:16.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9629999999999997e-05 2024-09-11T03:39:16.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:16.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026734+0000", 2024-09-11T03:39:16.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039259209999999998 2024-09-11T03:39:16.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:16.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026779+0000", 2024-09-11T03:39:16.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4574999999999999e-05 2024-09-11T03:39:16.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:16.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:16.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:16.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026840+0000", 2024-09-11T03:39:16.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1029000000000001e-05 2024-09-11T03:39:16.846 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.846 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.846 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.846 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:16.847 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:16.847 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:39:16.848 INFO:tasks.rados.rados.2.smithi175.stdout:269: done (0 left) 2024-09-11T03:39:16.849 INFO:tasks.rados.rados.2.smithi175.stdout:270: write oid 16 current snap is 20 2024-09-11T03:39:16.849 INFO:tasks.rados.rados.2.smithi175.stdout:270: seq_num 124 ranges {484068=792524,1906473=177595} 2024-09-11T03:39:16.849 INFO:tasks.rados.rados.2.smithi175.stdout:270: writing smithi17557870-16 from 484068 to 1276592 tid 1 2024-09-11T03:39:16.849 INFO:tasks.rados.rados.2.smithi175.stdout:270: writing smithi17557870-16 from 1906473 to 2084068 tid 2 2024-09-11T03:39:16.849 INFO:tasks.rados.rados.2.smithi175.stdout:271: write (excl) oid 41 current snap is 20 2024-09-11T03:39:16.849 INFO:tasks.rados.rados.2.smithi175.stdout:271: seq_num 125 ranges {500875=467773,1564955=535920} 2024-09-11T03:39:16.849 INFO:tasks.rados.rados.2.smithi175.stdout:271: writing smithi17557870-41 from 500875 to 968648 tid 1 2024-09-11T03:39:16.849 INFO:tasks.rados.rados.2.smithi175.stdout:271: writing smithi17557870-41 from 1564955 to 2100875 tid 2 2024-09-11T03:39:16.849 INFO:tasks.rados.rados.2.smithi175.stdout:272: read oid 17 snap -1 2024-09-11T03:39:16.849 INFO:tasks.rados.rados.2.smithi175.stdout:272: expect (ObjNum 88 snap 12 seq_num 88) 2024-09-11T03:39:16.849 INFO:tasks.rados.rados.2.smithi175.stdout:273: snap_create 2024-09-11T03:39:16.849 INFO:tasks.rados.rados.2.smithi175.stdout:270: finishing write tid 1 to smithi17557870-16 2024-09-11T03:39:16.849 INFO:tasks.rados.rados.2.smithi175.stdout:270: finishing write tid 2 to smithi17557870-16 2024-09-11T03:39:16.850 INFO:tasks.rados.rados.2.smithi175.stdout:270: finishing write tid 3 to smithi17557870-16 2024-09-11T03:39:16.850 INFO:tasks.rados.rados.2.smithi175.stdout:270: finishing write tid 4 to smithi17557870-16 2024-09-11T03:39:16.850 INFO:tasks.rados.rados.2.smithi175.stdout:270: oid 16 updating version 0 to 47 2024-09-11T03:39:16.850 INFO:tasks.rados.rados.2.smithi175.stdout:270: oid 16 version 47 is already newer than 45 2024-09-11T03:39:16.850 INFO:tasks.rados.rados.2.smithi175.stdout:270: oid 16 version 47 is already newer than 46 2024-09-11T03:39:16.850 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 47 (ObjNum 124 snap 20 seq_num 124) dirty exists 2024-09-11T03:39:16.850 INFO:tasks.rados.rados.2.smithi175.stdout:270: left oid 16 (ObjNum 124 snap 20 seq_num 124) 2024-09-11T03:39:16.850 INFO:tasks.rados.rados.2.smithi175.stdout:271: finishing write tid 1 to smithi17557870-41 2024-09-11T03:39:16.850 INFO:tasks.rados.rados.2.smithi175.stdout:271: finishing write tid 2 to smithi17557870-41 2024-09-11T03:39:16.850 INFO:tasks.rados.rados.2.smithi175.stdout:271: finishing write tid 3 to smithi17557870-41 2024-09-11T03:39:16.850 INFO:tasks.rados.rados.2.smithi175.stdout:271: finishing write tid 4 to smithi17557870-41 2024-09-11T03:39:16.850 INFO:tasks.rados.rados.2.smithi175.stdout:271: oid 41 updating version 0 to 26 2024-09-11T03:39:16.850 INFO:tasks.rados.rados.2.smithi175.stdout:271: oid 41 updating version 26 to 27 2024-09-11T03:39:16.851 INFO:tasks.rados.rados.2.smithi175.stdout:271: oid 41 updating version 27 to 28 2024-09-11T03:39:16.851 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 28 (ObjNum 125 snap 20 seq_num 125) dirty exists 2024-09-11T03:39:16.851 INFO:tasks.rados.rados.2.smithi175.stdout:271: left oid 41 (ObjNum 125 snap 20 seq_num 125) 2024-09-11T03:39:16.851 INFO:tasks.rados.rados.2.smithi175.stdout:272: expect (ObjNum 88 snap 12 seq_num 88) 2024-09-11T03:39:16.863 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:16.864 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:39:16.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:39:16.880 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:39:16.881 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:39:16.887 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:16.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:16.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:16.896 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:39:16.897 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:39:16.914 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:39:16.914 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:39:16.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:16 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:39:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:16 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:39:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:16 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3813240821' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:16 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3105322119' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:39:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:16 smithi121 ceph-mon[30729]: osdmap e139: 12 total, 12 up, 12 in 2024-09-11T03:39:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:16 smithi121 ceph-mon[30729]: osdmap e140: 12 total, 12 up, 12 in 2024-09-11T03:39:16.931 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:39:16.931 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:39:16.947 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:39:16.947 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:39:16.964 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:39:16.964 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:39:16.980 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:39:16.980 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:39:16.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:16.996 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:39:16.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:39:17.010 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:17.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:17.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:17.013 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:39:17.013 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:39:17.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:16 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:39:17.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:16 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:39:17.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:16 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3813240821' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:17.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:16 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3105322119' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:39:17.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:16 smithi084 ceph-mon[27254]: osdmap e139: 12 total, 12 up, 12 in 2024-09-11T03:39:17.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:16 smithi084 ceph-mon[27254]: osdmap e140: 12 total, 12 up, 12 in 2024-09-11T03:39:17.030 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:39:17.030 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:39:17.046 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:39:17.046 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:39:17.112 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:17.133 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:17.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:17.134 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:16 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:39:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:16 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:39:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:16 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3813240821' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:16 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3105322119' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:39:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:16 smithi175 ceph-mon[30894]: osdmap e139: 12 total, 12 up, 12 in 2024-09-11T03:39:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:16 smithi175 ceph-mon[30894]: osdmap e140: 12 total, 12 up, 12 in 2024-09-11T03:39:17.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:17.257 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:17.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:17.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:17.359 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:17.380 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:17.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:17.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:17.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:17.503 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:17.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:17.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:17.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:17.627 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:17.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:17.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:17.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:17.750 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:17.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:17.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:17.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T03:39:17.851 INFO:tasks.rados.rados.2.smithi175.stdout:270: done (3 left) 2024-09-11T03:39:17.851 INFO:tasks.rados.rados.2.smithi175.stdout:271: done (2 left) 2024-09-11T03:39:17.851 INFO:tasks.rados.rados.2.smithi175.stdout:272: done (1 left) 2024-09-11T03:39:17.851 INFO:tasks.rados.rados.2.smithi175.stdout:273: done (0 left) 2024-09-11T03:39:17.852 INFO:tasks.rados.rados.2.smithi175.stdout:274: delete oid 22 current snap is 21 2024-09-11T03:39:17.852 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:17.855 INFO:tasks.rados.rados.2.smithi175.stdout:274: done (0 left) 2024-09-11T03:39:17.855 INFO:tasks.rados.rados.2.smithi175.stdout:275: delete oid 44 current snap is 21 2024-09-11T03:39:17.858 INFO:tasks.rados.rados.2.smithi175.stdout:275: done (0 left) 2024-09-11T03:39:17.858 INFO:tasks.rados.rados.2.smithi175.stdout:276: read oid 40 snap -1 2024-09-11T03:39:17.859 INFO:tasks.rados.rados.2.smithi175.stdout:276: expect deleted 2024-09-11T03:39:17.859 INFO:tasks.rados.rados.2.smithi175.stdout:277: write oid 39 current snap is 21 2024-09-11T03:39:17.859 INFO:tasks.rados.rados.2.smithi175.stdout:277: seq_num 128 ranges {551296=693520,2024048=127248} 2024-09-11T03:39:17.865 INFO:tasks.rados.rados.2.smithi175.stdout:277: writing smithi17557870-39 from 551296 to 1244816 tid 1 2024-09-11T03:39:17.870 INFO:tasks.rados.rados.2.smithi175.stdout:277: writing smithi17557870-39 from 2024048 to 2151296 tid 2 2024-09-11T03:39:17.870 INFO:tasks.rados.rados.2.smithi175.stdout:278: read oid 21 snap -1 2024-09-11T03:39:17.870 INFO:tasks.rados.rados.2.smithi175.stdout:278: expect deleted 2024-09-11T03:39:17.870 INFO:tasks.rados.rados.2.smithi175.stdout:279: write oid 33 current snap is 21 2024-09-11T03:39:17.871 INFO:tasks.rados.rados.2.smithi175.stdout:279: seq_num 129 ranges {568103=768769,2082530=85573} 2024-09-11T03:39:17.872 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:17.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:17.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:17.877 INFO:tasks.rados.rados.2.smithi175.stdout:279: writing smithi17557870-33 from 568103 to 1336872 tid 1 2024-09-11T03:39:17.882 INFO:tasks.rados.rados.2.smithi175.stdout:279: writing smithi17557870-33 from 2082530 to 2168103 tid 2 2024-09-11T03:39:17.882 INFO:tasks.rados.rados.2.smithi175.stdout:280: copy_from oid 27 from oid 35 current snap is 21 2024-09-11T03:39:17.882 INFO:tasks.rados.rados.2.smithi175.stdout:281: snap_remove snap 20 2024-09-11T03:39:17.974 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:17.999 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:18.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:18.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:18.096 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:39:18.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:18.123 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:18.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:18.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:17 smithi084 ceph-mon[27254]: pgmap v364: 17 pgs: 17 active+clean; 124 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 18 op/s 2024-09-11T03:39:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1340956367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:18.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:17 smithi121 ceph-mon[30729]: pgmap v364: 17 pgs: 17 active+clean; 124 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 18 op/s 2024-09-11T03:39:18.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1340956367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:18.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:18.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:17 smithi175 ceph-mon[30894]: pgmap v364: 17 pgs: 17 active+clean; 124 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 18 op/s 2024-09-11T03:39:18.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1340956367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:18.246 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:18.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:18.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:18.314 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T03:39:18.314 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:39:18.348 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:18.370 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:18.371 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:18.472 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:18.493 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:18.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:18.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:18.565 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:18.592 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:39:18.592 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:39:18.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:18.616 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:18.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:18.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:18.718 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:18.739 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:18.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:18.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:18.841 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:18.857 INFO:tasks.rados.rados.2.smithi175.stdout:281: done (5 left) 2024-09-11T03:39:18.857 INFO:tasks.rados.rados.2.smithi175.stdout:282: rollback oid 50 current snap is 21 2024-09-11T03:39:18.858 INFO:tasks.rados.rados.2.smithi175.stdout:282: done (5 left) 2024-09-11T03:39:18.858 INFO:tasks.rados.rados.2.smithi175.stdout:283: delete oid 47 current snap is 21 2024-09-11T03:39:18.858 INFO:tasks.rados.rados.2.smithi175.stdout:277: finishing write tid 1 to smithi17557870-39 2024-09-11T03:39:18.858 INFO:tasks.rados.rados.2.smithi175.stdout:277: finishing write tid 2 to smithi17557870-39 2024-09-11T03:39:18.858 INFO:tasks.rados.rados.2.smithi175.stdout:277: finishing write tid 3 to smithi17557870-39 2024-09-11T03:39:18.858 INFO:tasks.rados.rados.2.smithi175.stdout:277: finishing write tid 4 to smithi17557870-39 2024-09-11T03:39:18.858 INFO:tasks.rados.rados.2.smithi175.stdout:277: oid 39 updating version 0 to 30 2024-09-11T03:39:18.858 INFO:tasks.rados.rados.2.smithi175.stdout:277: oid 39 version 30 is already newer than 29 2024-09-11T03:39:18.858 INFO:tasks.rados.rados.2.smithi175.stdout:277: oid 39 updating version 30 to 31 2024-09-11T03:39:18.858 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 31 (ObjNum 128 snap 21 seq_num 128) dirty exists 2024-09-11T03:39:18.858 INFO:tasks.rados.rados.2.smithi175.stdout:277: left oid 39 (ObjNum 128 snap 21 seq_num 128) 2024-09-11T03:39:18.858 INFO:tasks.rados.rados.2.smithi175.stdout:279: finishing write tid 1 to smithi17557870-33 2024-09-11T03:39:18.858 INFO:tasks.rados.rados.2.smithi175.stdout:279: finishing write tid 2 to smithi17557870-33 2024-09-11T03:39:18.859 INFO:tasks.rados.rados.2.smithi175.stdout:279: finishing write tid 3 to smithi17557870-33 2024-09-11T03:39:18.859 INFO:tasks.rados.rados.2.smithi175.stdout:279: finishing write tid 4 to smithi17557870-33 2024-09-11T03:39:18.859 INFO:tasks.rados.rados.2.smithi175.stdout:279: oid 33 updating version 0 to 112 2024-09-11T03:39:18.859 INFO:tasks.rados.rados.2.smithi175.stdout:279: oid 33 version 112 is already newer than 111 2024-09-11T03:39:18.859 INFO:tasks.rados.rados.2.smithi175.stdout:279: oid 33 updating version 112 to 113 2024-09-11T03:39:18.859 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 113 (ObjNum 129 snap 21 seq_num 129) dirty exists 2024-09-11T03:39:18.859 INFO:tasks.rados.rados.2.smithi175.stdout:279: left oid 33 (ObjNum 129 snap 21 seq_num 129) 2024-09-11T03:39:18.859 INFO:tasks.rados.rados.2.smithi175.stdout:280: finishing copy_from racing read to smithi17557870-27 2024-09-11T03:39:18.859 INFO:tasks.rados.rados.2.smithi175.stdout:280: finishing copy_from to smithi17557870-27 2024-09-11T03:39:18.859 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 39 (ObjNum 88 snap 12 seq_num 88) dirty exists 2024-09-11T03:39:18.860 INFO:tasks.rados.rados.2.smithi175.stdout:276: done (5 left) 2024-09-11T03:39:18.860 INFO:tasks.rados.rados.2.smithi175.stdout:277: done (4 left) 2024-09-11T03:39:18.860 INFO:tasks.rados.rados.2.smithi175.stdout:278: done (3 left) 2024-09-11T03:39:18.860 INFO:tasks.rados.rados.2.smithi175.stdout:279: done (2 left) 2024-09-11T03:39:18.860 INFO:tasks.rados.rados.2.smithi175.stdout:280: done (1 left) 2024-09-11T03:39:18.860 INFO:tasks.rados.rados.2.smithi175.stdout:283: done (0 left) 2024-09-11T03:39:18.860 INFO:tasks.rados.rados.2.smithi175.stdout:284: copy_from oid 35 from oid 2 current snap is 21 2024-09-11T03:39:18.860 INFO:tasks.rados.rados.2.smithi175.stdout:285: snap_create 2024-09-11T03:39:18.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:18.865 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:18.865 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:18.866 INFO:tasks.rados.rados.2.smithi175.stdout:284: finishing copy_from racing read to smithi17557870-35 2024-09-11T03:39:18.870 INFO:tasks.rados.rados.2.smithi175.stdout:284: finishing copy_from to smithi17557870-35 2024-09-11T03:39:18.870 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T03:39:18.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:18.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:18.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:19.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:18 smithi084 ceph-mon[27254]: osdmap e141: 12 total, 12 up, 12 in 2024-09-11T03:39:19.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:39:19.015 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:39:19.015 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:39:19.015 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:39:19.089 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:19.112 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:19.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:19.113 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:18 smithi121 ceph-mon[30729]: osdmap e141: 12 total, 12 up, 12 in 2024-09-11T03:39:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:39:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:39:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:39:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:39:19.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:19.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:18 smithi175 ceph-mon[30894]: osdmap e141: 12 total, 12 up, 12 in 2024-09-11T03:39:19.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:39:19.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:39:19.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:39:19.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:39:19.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:19.236 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:19.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:19.279 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:39:19.280 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:39:19.282 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:39:19.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T03:39:19.337 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:19.381 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:19.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:19.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:19.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:19.505 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:19.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:19.561 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:39:19.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:19.628 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:19.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:19.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:19.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:19.752 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:19.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:19.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:19.852 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:39:19.854 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:19.862 INFO:tasks.rados.rados.2.smithi175.stdout:284: done (1 left) 2024-09-11T03:39:19.862 INFO:tasks.rados.rados.2.smithi175.stdout:285: done (0 left) 2024-09-11T03:39:19.862 INFO:tasks.rados.rados.2.smithi175.stdout:286: copy_from oid 19 from oid 38 current snap is 22 2024-09-11T03:39:19.862 INFO:tasks.rados.rados.2.smithi175.stdout:287: snap_remove snap 21 2024-09-11T03:39:19.879 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:19.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:19.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:19.966 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:19 smithi121 ceph-mon[30729]: pgmap v367: 17 pgs: 17 active+clean; 121 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.8 MiB/s wr, 12 op/s 2024-09-11T03:39:19.966 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:19 smithi121 ceph-mon[30729]: osdmap e142: 12 total, 12 up, 12 in 2024-09-11T03:39:19.966 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:19 smithi121 ceph-mon[30729]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:39:19.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:19.994 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:19 smithi084 ceph-mon[27254]: pgmap v367: 17 pgs: 17 active+clean; 121 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.8 MiB/s wr, 12 op/s 2024-09-11T03:39:19.994 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:19 smithi084 ceph-mon[27254]: osdmap e142: 12 total, 12 up, 12 in 2024-09-11T03:39:19.994 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:19 smithi084 ceph-mon[27254]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:39:20.002 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:20.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:20.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:20.025 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-09-11T03:39:18.855275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":127364348,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":686,"num_read_kb":441166,"num_write":644,"num_write_kb":205677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":859,"ondisk_log_size":859,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":576608,"kb_used_data":225888,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124365216,"statfs":{"total":1151940427776,"available":1151349981184,"internally_reserved":0,"allocated":231309312,"data_stored":221407267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":353154363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8886681,"num_objects":-9,"num_object_clones":-9,"num_object_copies":-27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9,"num_whiteouts":-2,"num_read":26,"num_read_kb":15481,"num_write":24,"num_write_kb":15625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.017891"},"pg_stats":[{"pgid":"2.f","version":"134'44","reported_seq":266,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.848652+0000","last_change":"2024-09-11T03:39:14.839783+0000","last_active":"2024-09-11T03:39:15.848652+0000","last_peered":"2024-09-11T03:39:15.848652+0000","last_clean":"2024-09-11T03:39:15.848652+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:39:15.848652+0000","last_undegraded":"2024-09-11T03:39:15.848652+0000","last_fullsized":"2024-09-11T03:39:15.848652+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:43.018982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059305899999999995,"stat_sum":{"num_bytes":11193363,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":6166,"num_write":32,"num_write_kb":13884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.e","version":"136'78","reported_seq":321,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-11T03:39:17.850570+0000","last_change":"2024-09-11T03:39:17.100909+0000","last_active":"2024-09-11T03:39:17.850570+0000","last_peered":"2024-09-11T03:39:17.850570+0000","last_clean":"2024-09-11T03:39:17.850570+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:39:17.850570+0000","last_undegraded":"2024-09-11T03:39:17.850570+0000","last_fullsized":"2024-09-11T03:39:17.850570+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:40.290332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046817400000000002,"stat_sum":{"num_bytes":9798382,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":83,"num_read_kb":75021,"num_write":34,"num_write_kb":14210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12","length":"4"}]},{"pgid":"2.d","version":"136'79","reported_seq":316,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-11T03:39:17.977771+0000","last_change":"2024-09-11T03:39:16.829987+0000","last_active":"2024-09-11T03:39:17.977771+0000","last_peered":"2024-09-11T03:39:17.977771+0000","last_clean":"2024-09-11T03:39:17.977771+0000","last_became_active":"2024-09-11T03:38:30.031497+0000","last_became_peered":"2024-09-11T03:38:30.031497+0000","last_unstale":"2024-09-11T03:39:17.977771+0000","last_undegraded":"2024-09-11T03:39:17.977771+0000","last_fullsized":"2024-09-11T03:39:17.977771+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:47.506929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047394800000000002,"stat_sum":{"num_bytes":13294205,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":59,"num_read_kb":49496,"num_write":48,"num_write_kb":10472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"12","length":"4"}]},{"pgid":"2.c","version":"138'40","reported_seq":281,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-11T03:39:16.839141+0000","last_change":"2024-09-11T03:39:16.839398+0000","last_active":"2024-09-11T03:39:16.839141+0000","last_peered":"2024-09-11T03:39:16.839141+0000","last_clean":"2024-09-11T03:39:16.839141+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:39:16.839141+0000","last_undegraded":"2024-09-11T03:39:16.839141+0000","last_fullsized":"2024-09-11T03:39:16.839141+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:55.072371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058270900000000005,"stat_sum":{"num_bytes":12672346,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":67,"num_read_kb":60071,"num_write":38,"num_write_kb":15009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"12","length":"4"}]},{"pgid":"1.0","version":"20'92","reported_seq":223,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.021000+0000","last_change":"2024-09-11T03:37:42.091504+0000","last_active":"2024-09-11T03:39:15.021000+0000","last_peered":"2024-09-11T03:39:15.021000+0000","last_clean":"2024-09-11T03:39:15.021000+0000","last_became_active":"2024-09-11T03:37:28.922429+0000","last_became_peered":"2024-09-11T03:37:28.922429+0000","last_unstale":"2024-09-11T03:39:15.021000+0000","last_undegraded":"2024-09-11T03:39:15.021000+0000","last_fullsized":"2024-09-11T03:39:15.021000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:04.500658+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"138'34","reported_seq":255,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.026675+0000","last_change":"2024-09-11T03:39:15.021313+0000","last_active":"2024-09-11T03:39:15.026675+0000","last_peered":"2024-09-11T03:39:15.026675+0000","last_clean":"2024-09-11T03:39:15.026675+0000","last_became_active":"2024-09-11T03:38:29.889091+0000","last_became_peered":"2024-09-11T03:38:29.889091+0000","last_unstale":"2024-09-11T03:39:15.026675+0000","last_undegraded":"2024-09-11T03:39:15.026675+0000","last_fullsized":"2024-09-11T03:39:15.026675+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:55.768769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019361059999999999,"stat_sum":{"num_bytes":4857190,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":15198,"num_write":21,"num_write_kb":9930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.0","version":"114'14","reported_seq":228,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.021130+0000","last_change":"2024-09-11T03:39:15.021368+0000","last_active":"2024-09-11T03:39:15.021130+0000","last_peered":"2024-09-11T03:39:15.021130+0000","last_clean":"2024-09-11T03:39:15.021130+0000","last_became_active":"2024-09-11T03:38:29.930120+0000","last_became_peered":"2024-09-11T03:38:29.930120+0000","last_unstale":"2024-09-11T03:39:15.021130+0000","last_undegraded":"2024-09-11T03:39:15.021130+0000","last_fullsized":"2024-09-11T03:39:15.021130+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:41.948777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054800200000000002,"stat_sum":{"num_bytes":3260525,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":5533,"num_write":14,"num_write_kb":3881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.1","version":"138'19","reported_seq":246,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-11T03:39:17.850591+0000","last_change":"2024-09-11T03:39:17.100943+0000","last_active":"2024-09-11T03:39:17.850591+0000","last_peered":"2024-09-11T03:39:17.850591+0000","last_clean":"2024-09-11T03:39:17.850591+0000","last_became_active":"2024-09-11T03:38:30.162952+0000","last_became_peered":"2024-09-11T03:38:30.162952+0000","last_unstale":"2024-09-11T03:39:17.850591+0000","last_undegraded":"2024-09-11T03:39:17.850591+0000","last_fullsized":"2024-09-11T03:39:17.850591+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:48.306649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047621099999999998,"stat_sum":{"num_bytes":2050454,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":10354,"num_write":11,"num_write_kb":4517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12","length":"4"}]},{"pgid":"2.2","version":"135'51","reported_seq":266,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.019575+0000","last_change":"2024-09-11T03:39:15.019723+0000","last_active":"2024-09-11T03:39:15.019575+0000","last_peered":"2024-09-11T03:39:15.019575+0000","last_clean":"2024-09-11T03:39:15.019575+0000","last_became_active":"2024-09-11T03:38:29.889523+0000","last_became_peered":"2024-09-11T03:38:29.889523+0000","last_unstale":"2024-09-11T03:39:15.019575+0000","last_undegraded":"2024-09-11T03:39:15.019575+0000","last_fullsized":"2024-09-11T03:39:15.019575+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:44.494077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035620600000000002,"stat_sum":{"num_bytes":2453789,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":5,"num_write":28,"num_write_kb":13972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.4","version":"136'52","reported_seq":268,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.846634+0000","last_change":"2024-09-11T03:39:15.017624+0000","last_active":"2024-09-11T03:39:15.846634+0000","last_peered":"2024-09-11T03:39:15.846634+0000","last_clean":"2024-09-11T03:39:15.846634+0000","last_became_active":"2024-09-11T03:38:30.163226+0000","last_became_peered":"2024-09-11T03:38:30.163226+0000","last_unstale":"2024-09-11T03:39:15.846634+0000","last_undegraded":"2024-09-11T03:39:15.846634+0000","last_fullsized":"2024-09-11T03:39:15.846634+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:42.429151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049723699999999998,"stat_sum":{"num_bytes":7932838,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":8,"num_write":39,"num_write_kb":13761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.5","version":"136'94","reported_seq":331,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.848688+0000","last_change":"2024-09-11T03:39:14.840149+0000","last_active":"2024-09-11T03:39:15.848688+0000","last_peered":"2024-09-11T03:39:15.848688+0000","last_clean":"2024-09-11T03:39:15.848688+0000","last_became_active":"2024-09-11T03:38:30.163080+0000","last_became_peered":"2024-09-11T03:38:30.163080+0000","last_unstale":"2024-09-11T03:39:15.848688+0000","last_undegraded":"2024-09-11T03:39:15.848688+0000","last_fullsized":"2024-09-11T03:39:15.848688+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:50.745787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094381,"stat_sum":{"num_bytes":15445501,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":51892,"num_write":71,"num_write_kb":23938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.6","version":"134'27","reported_seq":245,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.846294+0000","last_change":"2024-09-11T03:39:15.017828+0000","last_active":"2024-09-11T03:39:15.846294+0000","last_peered":"2024-09-11T03:39:15.846294+0000","last_clean":"2024-09-11T03:39:15.846294+0000","last_became_active":"2024-09-11T03:38:29.887945+0000","last_became_peered":"2024-09-11T03:38:29.887945+0000","last_unstale":"2024-09-11T03:39:15.846294+0000","last_undegraded":"2024-09-11T03:39:15.846294+0000","last_fullsized":"2024-09-11T03:39:15.846294+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:29.000919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070240700000000005,"stat_sum":{"num_bytes":3848770,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":16651,"num_write":16,"num_write_kb":9866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.7","version":"136'58","reported_seq":291,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-11T03:39:17.217643+0000","last_change":"2024-09-11T03:39:17.217769+0000","last_active":"2024-09-11T03:39:17.217643+0000","last_peered":"2024-09-11T03:39:17.217643+0000","last_clean":"2024-09-11T03:39:17.217643+0000","last_became_active":"2024-09-11T03:38:29.930154+0000","last_became_peered":"2024-09-11T03:38:29.930154+0000","last_unstale":"2024-09-11T03:39:17.217643+0000","last_undegraded":"2024-09-11T03:39:17.217643+0000","last_fullsized":"2024-09-11T03:39:17.217643+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:56.529038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000313863,"stat_sum":{"num_bytes":7613472,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":11832,"num_write":43,"num_write_kb":16991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12","length":"4"}]},{"pgid":"2.8","version":"136'37","reported_seq":257,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.021076+0000","last_change":"2024-09-11T03:39:15.021319+0000","last_active":"2024-09-11T03:39:15.021076+0000","last_peered":"2024-09-11T03:39:15.021076+0000","last_clean":"2024-09-11T03:39:15.021076+0000","last_became_active":"2024-09-11T03:38:30.162819+0000","last_became_peered":"2024-09-11T03:38:30.162819+0000","last_unstale":"2024-09-11T03:39:15.021076+0000","last_undegraded":"2024-09-11T03:39:15.021076+0000","last_fullsized":"2024-09-11T03:39:15.021076+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:56.698633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050697399999999999,"stat_sum":{"num_bytes":9798382,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":29182,"num_write":33,"num_write_kb":11869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.9","version":"136'108","reported_seq":353,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.846280+0000","last_change":"2024-09-11T03:39:15.017545+0000","last_active":"2024-09-11T03:39:15.846280+0000","last_peered":"2024-09-11T03:39:15.846280+0000","last_clean":"2024-09-11T03:39:15.846280+0000","last_became_active":"2024-09-11T03:38:29.888204+0000","last_became_peered":"2024-09-11T03:38:29.888204+0000","last_unstale":"2024-09-11T03:39:15.846280+0000","last_undegraded":"2024-09-11T03:39:15.846280+0000","last_fullsized":"2024-09-11T03:39:15.846280+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:40.557572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050069800000000005,"stat_sum":{"num_bytes":12857223,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":113,"num_read_kb":83664,"num_write":67,"num_write_kb":26037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.a","version":"134'7","reported_seq":225,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-11T03:39:16.838979+0000","last_change":"2024-09-11T03:39:16.839180+0000","last_active":"2024-09-11T03:39:16.838979+0000","last_peered":"2024-09-11T03:39:16.838979+0000","last_clean":"2024-09-11T03:39:16.838979+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:39:16.838979+0000","last_undegraded":"2024-09-11T03:39:16.838979+0000","last_fullsized":"2024-09-11T03:39:16.838979+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:38.805883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052170500000000002,"stat_sum":{"num_bytes":3109262,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":10361,"num_write":5,"num_write_kb":4561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"12","length":"4"}]},{"pgid":"2.b","version":"136'25","reported_seq":248,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.848626+0000","last_change":"2024-09-11T03:39:14.839769+0000","last_active":"2024-09-11T03:39:15.848626+0000","last_peered":"2024-09-11T03:39:15.848626+0000","last_clean":"2024-09-11T03:39:15.848626+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:39:15.848626+0000","last_undegraded":"2024-09-11T03:39:15.848626+0000","last_fullsized":"2024-09-11T03:39:15.848626+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:48.151468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058574600000000005,"stat_sum":{"num_bytes":6588278,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":15668,"num_write":21,"num_write_kb":10629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":126773980,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":610,"num_read_kb":441102,"num_write":521,"num_write_kb":203527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212983808,"data_stored":211654291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":767,"ondisk_log_size":767,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285593,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40896,"kb_used_data":13868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93704256,"statfs":{"total":95995035648,"available":95953158144,"internally_reserved":0,"allocated":14200832,"data_stored":13422708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514525,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":38176,"kb_used_data":11148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93706976,"statfs":{"total":95995035648,"available":95955943424,"internally_reserved":0,"allocated":11415552,"data_stored":10649358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743459,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56480,"kb_used_data":27648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93688672,"statfs":{"total":95995035648,"available":95937200128,"internally_reserved":0,"allocated":28311552,"data_stored":27392781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037801,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54432,"kb_used_data":23240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93690720,"statfs":{"total":95995035648,"available":95939297280,"internally_reserved":0,"allocated":23797760,"data_stored":22950125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299437,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54136,"kb_used_data":21696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93691016,"statfs":{"total":95995035648,"available":95939600384,"internally_reserved":0,"allocated":22216704,"data_stored":21345454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":38040,"kb_used_data":6844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93707112,"statfs":{"total":95995035648,"available":95956082688,"internally_reserved":0,"allocated":7008256,"data_stored":6268732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33792,"kb_used_data":5516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93711360,"statfs":{"total":95995035648,"available":95960432640,"internally_reserved":0,"allocated":5648384,"data_stored":4921070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986236,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54972,"kb_used_data":23780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93690180,"statfs":{"total":95995035648,"available":95938744320,"internally_reserved":0,"allocated":24350720,"data_stored":23458987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58448,"kb_used_data":27260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93686704,"statfs":{"total":95995035648,"available":95935184896,"internally_reserved":0,"allocated":27914240,"data_stored":27013312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":42528,"kb_used_data":14248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93702624,"statfs":{"total":95995035648,"available":95951486976,"internally_reserved":0,"allocated":14589952,"data_stored":13807848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49748,"kb_used_data":22712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93695404,"statfs":{"total":95995035648,"available":95944093696,"internally_reserved":0,"allocated":23257088,"data_stored":22453015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54960,"kb_used_data":27928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93690192,"statfs":{"total":95995035648,"available":95938756608,"internally_reserved":0,"allocated":28598272,"data_stored":27723877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":27041792,"data_stored":26878453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21688320,"data_stored":21599507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14868480,"data_stored":14797432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":26320896,"data_stored":26143540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":22806528,"data_stored":22621695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4091904,"data_stored":4075646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4907008,"data_stored":4886296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20127744,"data_stored":19971102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":22241280,"data_stored":22104701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":26730496,"data_stored":26531069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":9859072,"data_stored":9803934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12300288,"data_stored":12240916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:39:20.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:39:20.071 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:20.074 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:39:20.074 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:39:20.104 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:20.126 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:20.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:20.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:20.227 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:20.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:19 smithi175 ceph-mon[30894]: pgmap v367: 17 pgs: 17 active+clean; 121 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.8 MiB/s wr, 12 op/s 2024-09-11T03:39:20.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:19 smithi175 ceph-mon[30894]: osdmap e142: 12 total, 12 up, 12 in 2024-09-11T03:39:20.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:19 smithi175 ceph-mon[30894]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:39:20.253 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:20.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:20.255 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:20.356 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:20.381 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:20.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:20.382 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:20.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:20.506 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:20.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:20.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:20.538 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:20.545 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:20.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:20.629 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:20.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:20.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:20.731 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:20.752 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:20.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:20.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T03:39:20.854 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:20.869 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:39:20.874 INFO:tasks.rados.rados.2.smithi175.stdout:287: done (1 left) 2024-09-11T03:39:20.874 INFO:tasks.rados.rados.2.smithi175.stdout:288: delete oid 39 current snap is 22 2024-09-11T03:39:20.875 INFO:tasks.rados.rados.2.smithi175.stdout:286: finishing copy_from racing read to smithi17557870-19 2024-09-11T03:39:20.875 INFO:tasks.rados.rados.2.smithi175.stdout:286: finishing copy_from to smithi17557870-19 2024-09-11T03:39:20.875 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 65 (ObjNum 101 snap 16 seq_num 101) dirty exists 2024-09-11T03:39:20.876 INFO:tasks.rados.rados.2.smithi175.stdout:286: done (1 left) 2024-09-11T03:39:20.876 INFO:tasks.rados.rados.2.smithi175.stdout:288: done (0 left) 2024-09-11T03:39:20.876 INFO:tasks.rados.rados.2.smithi175.stdout:289: copy_from oid 17 from oid 22 current snap is 22 2024-09-11T03:39:20.876 INFO:tasks.rados.rados.2.smithi175.stdout:290: write oid 29 current snap is 22 2024-09-11T03:39:20.876 INFO:tasks.rados.rados.2.smithi175.stdout:290: seq_num 132 ranges {618524=710869,1857976=360548} 2024-09-11T03:39:20.878 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:20.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:20.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:20.883 INFO:tasks.rados.rados.2.smithi175.stdout:290: writing smithi17557870-29 from 618524 to 1329393 tid 1 2024-09-11T03:39:20.888 INFO:tasks.rados.rados.2.smithi175.stdout:290: writing smithi17557870-29 from 1857976 to 2218524 tid 2 2024-09-11T03:39:20.888 INFO:tasks.rados.rados.2.smithi175.stdout:291: rollback oid 27 current snap is 22 2024-09-11T03:39:20.888 INFO:tasks.rados.rados.2.smithi175.stdout:291: done (2 left) 2024-09-11T03:39:20.889 INFO:tasks.rados.rados.2.smithi175.stdout:292: write (excl) oid 41 current snap is 22 2024-09-11T03:39:20.889 INFO:tasks.rados.rados.2.smithi175.stdout:292: seq_num 133 ranges {635331=786118,1916458=318873} 2024-09-11T03:39:20.896 INFO:tasks.rados.rados.2.smithi175.stdout:292: writing smithi17557870-41 from 635331 to 1421449 tid 1 2024-09-11T03:39:20.901 INFO:tasks.rados.rados.2.smithi175.stdout:292: writing smithi17557870-41 from 1916458 to 2235331 tid 2 2024-09-11T03:39:20.901 INFO:tasks.rados.rados.2.smithi175.stdout:293: write oid 42 current snap is 22 2024-09-11T03:39:20.901 INFO:tasks.rados.rados.2.smithi175.stdout:289: finishing copy_from to smithi17557870-17 2024-09-11T03:39:20.901 INFO:tasks.rados.rados.2.smithi175.stdout:289: got expected ENOENT (src dne) 2024-09-11T03:39:20.901 INFO:tasks.rados.rados.2.smithi175.stdout:289: finishing copy_from racing read to smithi17557870-17 2024-09-11T03:39:20.901 INFO:tasks.rados.rados.2.smithi175.stdout:290: finishing write tid 1 to smithi17557870-29 2024-09-11T03:39:20.901 INFO:tasks.rados.rados.2.smithi175.stdout:290: finishing write tid 2 to smithi17557870-29 2024-09-11T03:39:20.902 INFO:tasks.rados.rados.2.smithi175.stdout:293: seq_num 134 ranges {652138=461367,1574940=677198} 2024-09-11T03:39:20.907 INFO:tasks.rados.rados.2.smithi175.stdout:293: writing smithi17557870-42 from 652138 to 1113505 tid 1 2024-09-11T03:39:20.915 INFO:tasks.rados.rados.2.smithi175.stdout:293: writing smithi17557870-42 from 1574940 to 2252138 tid 2 2024-09-11T03:39:20.915 INFO:tasks.rados.rados.2.smithi175.stdout:289: done (3 left) 2024-09-11T03:39:20.915 INFO:tasks.rados.rados.2.smithi175.stdout:294: copy_from oid 45 from oid 30 current snap is 22 2024-09-11T03:39:20.915 INFO:tasks.rados.rados.2.smithi175.stdout:290: finishing write tid 3 to smithi17557870-29 2024-09-11T03:39:20.915 INFO:tasks.rados.rados.2.smithi175.stdout:290: finishing write tid 4 to smithi17557870-29 2024-09-11T03:39:20.915 INFO:tasks.rados.rados.2.smithi175.stdout:290: oid 29 updating version 0 to 95 2024-09-11T03:39:20.915 INFO:tasks.rados.rados.2.smithi175.stdout:290: oid 29 updating version 95 to 97 2024-09-11T03:39:20.915 INFO:tasks.rados.rados.2.smithi175.stdout:290: oid 29 version 97 is already newer than 96 2024-09-11T03:39:20.915 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 97 (ObjNum 132 snap 22 seq_num 132) dirty exists 2024-09-11T03:39:20.915 INFO:tasks.rados.rados.2.smithi175.stdout:290: left oid 29 (ObjNum 132 snap 22 seq_num 132) 2024-09-11T03:39:20.915 INFO:tasks.rados.rados.2.smithi175.stdout:292: finishing write tid 1 to smithi17557870-41 2024-09-11T03:39:20.915 INFO:tasks.rados.rados.2.smithi175.stdout:290: done (3 left) 2024-09-11T03:39:20.916 INFO:tasks.rados.rados.2.smithi175.stdout:295: rollback oid 15 current snap is 22 2024-09-11T03:39:20.916 INFO:tasks.rados.rados.2.smithi175.stdout:295: done (3 left) 2024-09-11T03:39:20.916 INFO:tasks.rados.rados.2.smithi175.stdout:296: copy_from oid 46 from oid 32 current snap is 22 2024-09-11T03:39:20.916 INFO:tasks.rados.rados.2.smithi175.stdout:292: finishing write tid 2 to smithi17557870-41 2024-09-11T03:39:20.916 INFO:tasks.rados.rados.2.smithi175.stdout:292: finishing write tid 3 to smithi17557870-41 2024-09-11T03:39:20.916 INFO:tasks.rados.rados.2.smithi175.stdout:292: finishing write tid 4 to smithi17557870-41 2024-09-11T03:39:20.916 INFO:tasks.rados.rados.2.smithi175.stdout:292: oid 41 updating version 0 to 33 2024-09-11T03:39:20.916 INFO:tasks.rados.rados.2.smithi175.stdout:292: oid 41 updating version 33 to 35 2024-09-11T03:39:20.916 INFO:tasks.rados.rados.2.smithi175.stdout:292: oid 41 version 35 is already newer than 34 2024-09-11T03:39:20.916 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 35 (ObjNum 133 snap 22 seq_num 133) dirty exists 2024-09-11T03:39:20.916 INFO:tasks.rados.rados.2.smithi175.stdout:292: left oid 41 (ObjNum 133 snap 22 seq_num 133) 2024-09-11T03:39:20.916 INFO:tasks.rados.rados.2.smithi175.stdout:293: finishing write tid 1 to smithi17557870-42 2024-09-11T03:39:20.916 INFO:tasks.rados.rados.2.smithi175.stdout:292: done (3 left) 2024-09-11T03:39:20.916 INFO:tasks.rados.rados.2.smithi175.stdout:297: copy_from oid 39 from oid 2 current snap is 22 2024-09-11T03:39:20.917 INFO:tasks.rados.rados.2.smithi175.stdout:298: rollback oid 23 current snap is 22 2024-09-11T03:39:20.917 INFO:tasks.rados.rados.2.smithi175.stdout:298: done (4 left) 2024-09-11T03:39:20.917 INFO:tasks.rados.rados.2.smithi175.stdout:299: delete oid 37 current snap is 22 2024-09-11T03:39:20.918 INFO:tasks.rados.rados.2.smithi175.stdout:293: finishing write tid 2 to smithi17557870-42 2024-09-11T03:39:20.918 INFO:tasks.rados.rados.2.smithi175.stdout:299: done (4 left) 2024-09-11T03:39:20.918 INFO:tasks.rados.rados.2.smithi175.stdout:300: read oid 5 snap -1 2024-09-11T03:39:20.918 INFO:tasks.rados.rados.2.smithi175.stdout:300: expect (ObjNum 122 snap 19 seq_num 122) 2024-09-11T03:39:20.918 INFO:tasks.rados.rados.2.smithi175.stdout:301: read oid 40 snap -1 2024-09-11T03:39:20.918 INFO:tasks.rados.rados.2.smithi175.stdout:301: expect deleted 2024-09-11T03:39:20.918 INFO:tasks.rados.rados.2.smithi175.stdout:302: delete oid 50 current snap is 22 2024-09-11T03:39:20.918 INFO:tasks.rados.rados.2.smithi175.stdout:293: finishing write tid 3 to smithi17557870-42 2024-09-11T03:39:20.918 INFO:tasks.rados.rados.2.smithi175.stdout:293: finishing write tid 4 to smithi17557870-42 2024-09-11T03:39:20.918 INFO:tasks.rados.rados.2.smithi175.stdout:293: oid 42 updating version 0 to 43 2024-09-11T03:39:20.919 INFO:tasks.rados.rados.2.smithi175.stdout:293: oid 42 updating version 43 to 44 2024-09-11T03:39:20.919 INFO:tasks.rados.rados.2.smithi175.stdout:293: oid 42 version 44 is already newer than 42 2024-09-11T03:39:20.919 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 44 (ObjNum 134 snap 22 seq_num 134) dirty exists 2024-09-11T03:39:20.919 INFO:tasks.rados.rados.2.smithi175.stdout:293: left oid 42 (ObjNum 134 snap 22 seq_num 134) 2024-09-11T03:39:20.919 INFO:tasks.rados.rados.2.smithi175.stdout:293: done (6 left) 2024-09-11T03:39:20.919 INFO:tasks.rados.rados.2.smithi175.stdout:301: done (5 left) 2024-09-11T03:39:20.920 INFO:tasks.rados.rados.2.smithi175.stdout:302: done (4 left) 2024-09-11T03:39:20.920 INFO:tasks.rados.rados.2.smithi175.stdout:303: copy_from oid 40 from oid 6 current snap is 22 2024-09-11T03:39:20.920 INFO:tasks.rados.rados.2.smithi175.stdout:304: write (excl) oid 9 current snap is 22 2024-09-11T03:39:20.920 INFO:tasks.rados.rados.2.smithi175.stdout:304: seq_num 137 ranges {702559=403467,1750386=495032} 2024-09-11T03:39:20.924 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:20 smithi084 ceph-mon[27254]: osdmap e143: 12 total, 12 up, 12 in 2024-09-11T03:39:20.924 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:20 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3273212094' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:39:20.926 INFO:tasks.rados.rados.2.smithi175.stdout:304: writing smithi17557870-9 from 702559 to 1106026 tid 1 2024-09-11T03:39:20.934 INFO:tasks.rados.rados.2.smithi175.stdout:304: writing smithi17557870-9 from 1750386 to 2245418 tid 2 2024-09-11T03:39:20.934 INFO:tasks.rados.rados.2.smithi175.stdout:305: snap_create 2024-09-11T03:39:20.934 INFO:tasks.rados.rados.2.smithi175.stdout:303: finishing copy_from to smithi17557870-40 2024-09-11T03:39:20.934 INFO:tasks.rados.rados.2.smithi175.stdout:303: got expected ENOENT (src dne) 2024-09-11T03:39:20.934 INFO:tasks.rados.rados.2.smithi175.stdout:303: finishing copy_from racing read to smithi17557870-40 2024-09-11T03:39:20.934 INFO:tasks.rados.rados.2.smithi175.stdout:296: finishing copy_from racing read to smithi17557870-46 2024-09-11T03:39:20.934 INFO:tasks.rados.rados.2.smithi175.stdout:297: finishing copy_from racing read to smithi17557870-39 2024-09-11T03:39:20.935 INFO:tasks.rados.rados.2.smithi175.stdout:300: expect (ObjNum 122 snap 19 seq_num 122) 2024-09-11T03:39:20.979 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:21.001 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:21.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:21.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:21.011 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:39:21.015 INFO:tasks.rados.rados.2.smithi175.stdout:294: finishing copy_from racing read to smithi17557870-45 2024-09-11T03:39:21.015 INFO:tasks.rados.rados.2.smithi175.stdout:296: finishing copy_from to smithi17557870-46 2024-09-11T03:39:21.015 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 25 (ObjNum 114 snap 18 seq_num 114) dirty exists 2024-09-11T03:39:21.015 INFO:tasks.rados.rados.2.smithi175.stdout:297: finishing copy_from to smithi17557870-39 2024-09-11T03:39:21.015 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T03:39:21.015 INFO:tasks.rados.rados.2.smithi175.stdout:304: finishing write tid 1 to smithi17557870-9 2024-09-11T03:39:21.015 INFO:tasks.rados.rados.2.smithi175.stdout:304: finishing write tid 2 to smithi17557870-9 2024-09-11T03:39:21.015 INFO:tasks.rados.rados.2.smithi175.stdout:304: finishing write tid 3 to smithi17557870-9 2024-09-11T03:39:21.015 INFO:tasks.rados.rados.2.smithi175.stdout:304: finishing write tid 4 to smithi17557870-9 2024-09-11T03:39:21.016 INFO:tasks.rados.rados.2.smithi175.stdout:304: oid 9 updating version 0 to 63 2024-09-11T03:39:21.016 INFO:tasks.rados.rados.2.smithi175.stdout:304: oid 9 updating version 63 to 64 2024-09-11T03:39:21.016 INFO:tasks.rados.rados.2.smithi175.stdout:304: oid 9 updating version 64 to 65 2024-09-11T03:39:21.016 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 65 (ObjNum 137 snap 22 seq_num 137) dirty exists 2024-09-11T03:39:21.016 INFO:tasks.rados.rados.2.smithi175.stdout:304: left oid 9 (ObjNum 137 snap 22 seq_num 137) 2024-09-11T03:39:21.016 INFO:tasks.rados.rados.2.smithi175.stdout:294: finishing copy_from to smithi17557870-45 2024-09-11T03:39:21.016 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 36 (ObjNum 100 snap 16 seq_num 100) dirty exists 2024-09-11T03:39:21.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:21.125 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:21.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:21.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:21.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:20 smithi121 ceph-mon[30729]: osdmap e143: 12 total, 12 up, 12 in 2024-09-11T03:39:21.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:20 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3273212094' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:39:21.227 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:21.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:20 smithi175 ceph-mon[30894]: osdmap e143: 12 total, 12 up, 12 in 2024-09-11T03:39:21.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:20 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3273212094' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:39:21.248 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:21.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:21.249 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:21.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:21.371 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:21.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:21.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:21.443 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:39:21.444 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:39:21.473 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:21.486 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:39:21.486 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:39:21.495 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:21.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:21.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:21.504 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:39:21.504 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:39:21.513 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:39:21.513 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:39:21.520 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:39:21.520 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:39:21.535 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:39:21.535 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:39:21.552 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:39:21.552 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:39:21.568 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:39:21.569 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:39:21.585 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:39:21.585 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:39:21.597 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:21.601 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:39:21.602 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:39:21.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:21.619 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:21.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:21.619 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:39:21.620 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:39:21.635 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:39:21.636 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:39:21.652 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:39:21.652 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:39:21.669 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:39:21.669 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:39:21.719 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:21.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:21.742 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:21.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:21.818 INFO:tasks.rados.rados.2.smithi175.stdout:294: done (6 left) 2024-09-11T03:39:21.819 INFO:tasks.rados.rados.2.smithi175.stdout:296: done (5 left) 2024-09-11T03:39:21.819 INFO:tasks.rados.rados.2.smithi175.stdout:297: done (4 left) 2024-09-11T03:39:21.819 INFO:tasks.rados.rados.2.smithi175.stdout:300: done (3 left) 2024-09-11T03:39:21.819 INFO:tasks.rados.rados.2.smithi175.stdout:303: done (2 left) 2024-09-11T03:39:21.819 INFO:tasks.rados.rados.2.smithi175.stdout:304: done (1 left) 2024-09-11T03:39:21.819 INFO:tasks.rados.rados.2.smithi175.stdout:305: done (0 left) 2024-09-11T03:39:21.819 INFO:tasks.rados.rados.2.smithi175.stdout:306: write (excl) oid 5 current snap is 23 2024-09-11T03:39:21.819 INFO:tasks.rados.rados.2.smithi175.stdout:306: seq_num 138 ranges {719366=478716,1808868=510498} 2024-09-11T03:39:21.825 INFO:tasks.rados.rados.2.smithi175.stdout:306: writing smithi17557870-5 from 719366 to 1198082 tid 1 2024-09-11T03:39:21.832 INFO:tasks.rados.rados.2.smithi175.stdout:306: writing smithi17557870-5 from 1808868 to 2319366 tid 2 2024-09-11T03:39:21.832 INFO:tasks.rados.rados.2.smithi175.stdout:307: snap_create 2024-09-11T03:39:21.832 INFO:tasks.rados.rados.2.smithi175.stdout:306: finishing write tid 1 to smithi17557870-5 2024-09-11T03:39:21.834 INFO:tasks.rados.rados.2.smithi175.stdout:306: finishing write tid 2 to smithi17557870-5 2024-09-11T03:39:21.835 INFO:tasks.rados.rados.2.smithi175.stdout:306: finishing write tid 3 to smithi17557870-5 2024-09-11T03:39:21.835 INFO:tasks.rados.rados.2.smithi175.stdout:306: finishing write tid 4 to smithi17557870-5 2024-09-11T03:39:21.835 INFO:tasks.rados.rados.2.smithi175.stdout:306: oid 5 updating version 0 to 22 2024-09-11T03:39:21.835 INFO:tasks.rados.rados.2.smithi175.stdout:306: oid 5 updating version 22 to 23 2024-09-11T03:39:21.835 INFO:tasks.rados.rados.2.smithi175.stdout:306: oid 5 version 23 is already newer than 21 2024-09-11T03:39:21.835 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 23 (ObjNum 138 snap 23 seq_num 138) dirty exists 2024-09-11T03:39:21.835 INFO:tasks.rados.rados.2.smithi175.stdout:306: left oid 5 (ObjNum 138 snap 23 seq_num 138) 2024-09-11T03:39:21.842 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:21.863 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:21.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:21.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:21.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:21.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:21.988 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:21.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:22.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:21 smithi084 ceph-mon[27254]: pgmap v370: 17 pgs: 17 active+clean; 121 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.1 MiB/s wr, 3 op/s 2024-09-11T03:39:22.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:21 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:22.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3273212094' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:39:22.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:21 smithi084 ceph-mon[27254]: osdmap e144: 12 total, 12 up, 12 in 2024-09-11T03:39:22.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:21 smithi084 ceph-mon[27254]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:39:22.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1620305951' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:22.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:21 smithi084 ceph-mon[27254]: osdmap e145: 12 total, 12 up, 12 in 2024-09-11T03:39:22.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:22.101 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-09-11T03:39:20.871879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":127364348,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":686,"num_read_kb":441166,"num_write":644,"num_write_kb":205677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":859,"ondisk_log_size":859,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":573828,"kb_used_data":223108,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124367996,"statfs":{"total":1151940427776,"available":1151352827904,"internally_reserved":0,"allocated":228462592,"data_stored":218565626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":353154363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2285686,"num_objects":-3,"num_object_clones":-2,"num_object_copies":-9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":-1,"num_read":11,"num_read_kb":9947,"num_write":5,"num_write_kb":4557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.056031"},"pg_stats":[{"pgid":"2.f","version":"134'44","reported_seq":266,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.848652+0000","last_change":"2024-09-11T03:39:14.839783+0000","last_active":"2024-09-11T03:39:15.848652+0000","last_peered":"2024-09-11T03:39:15.848652+0000","last_clean":"2024-09-11T03:39:15.848652+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:39:15.848652+0000","last_undegraded":"2024-09-11T03:39:15.848652+0000","last_fullsized":"2024-09-11T03:39:15.848652+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:43.018982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059305899999999995,"stat_sum":{"num_bytes":11193363,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":6166,"num_write":32,"num_write_kb":13884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.e","version":"136'78","reported_seq":321,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-11T03:39:17.850570+0000","last_change":"2024-09-11T03:39:17.100909+0000","last_active":"2024-09-11T03:39:17.850570+0000","last_peered":"2024-09-11T03:39:17.850570+0000","last_clean":"2024-09-11T03:39:17.850570+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:39:17.850570+0000","last_undegraded":"2024-09-11T03:39:17.850570+0000","last_fullsized":"2024-09-11T03:39:17.850570+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:40.290332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046817400000000002,"stat_sum":{"num_bytes":9798382,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":83,"num_read_kb":75021,"num_write":34,"num_write_kb":14210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12","length":"4"}]},{"pgid":"2.d","version":"136'79","reported_seq":316,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-11T03:39:17.977771+0000","last_change":"2024-09-11T03:39:16.829987+0000","last_active":"2024-09-11T03:39:17.977771+0000","last_peered":"2024-09-11T03:39:17.977771+0000","last_clean":"2024-09-11T03:39:17.977771+0000","last_became_active":"2024-09-11T03:38:30.031497+0000","last_became_peered":"2024-09-11T03:38:30.031497+0000","last_unstale":"2024-09-11T03:39:17.977771+0000","last_undegraded":"2024-09-11T03:39:17.977771+0000","last_fullsized":"2024-09-11T03:39:17.977771+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:47.506929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047394800000000002,"stat_sum":{"num_bytes":13294205,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":59,"num_read_kb":49496,"num_write":48,"num_write_kb":10472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"12","length":"4"}]},{"pgid":"2.c","version":"138'40","reported_seq":281,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-11T03:39:16.839141+0000","last_change":"2024-09-11T03:39:16.839398+0000","last_active":"2024-09-11T03:39:16.839141+0000","last_peered":"2024-09-11T03:39:16.839141+0000","last_clean":"2024-09-11T03:39:16.839141+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:39:16.839141+0000","last_undegraded":"2024-09-11T03:39:16.839141+0000","last_fullsized":"2024-09-11T03:39:16.839141+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:55.072371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058270900000000005,"stat_sum":{"num_bytes":12672346,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":67,"num_read_kb":60071,"num_write":38,"num_write_kb":15009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"12","length":"4"}]},{"pgid":"1.0","version":"20'92","reported_seq":223,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.021000+0000","last_change":"2024-09-11T03:37:42.091504+0000","last_active":"2024-09-11T03:39:15.021000+0000","last_peered":"2024-09-11T03:39:15.021000+0000","last_clean":"2024-09-11T03:39:15.021000+0000","last_became_active":"2024-09-11T03:37:28.922429+0000","last_became_peered":"2024-09-11T03:37:28.922429+0000","last_unstale":"2024-09-11T03:39:15.021000+0000","last_undegraded":"2024-09-11T03:39:15.021000+0000","last_fullsized":"2024-09-11T03:39:15.021000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:04.500658+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"138'34","reported_seq":255,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.026675+0000","last_change":"2024-09-11T03:39:15.021313+0000","last_active":"2024-09-11T03:39:15.026675+0000","last_peered":"2024-09-11T03:39:15.026675+0000","last_clean":"2024-09-11T03:39:15.026675+0000","last_became_active":"2024-09-11T03:38:29.889091+0000","last_became_peered":"2024-09-11T03:38:29.889091+0000","last_unstale":"2024-09-11T03:39:15.026675+0000","last_undegraded":"2024-09-11T03:39:15.026675+0000","last_fullsized":"2024-09-11T03:39:15.026675+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:55.768769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019361059999999999,"stat_sum":{"num_bytes":4857190,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":15198,"num_write":21,"num_write_kb":9930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.0","version":"114'14","reported_seq":228,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.021130+0000","last_change":"2024-09-11T03:39:15.021368+0000","last_active":"2024-09-11T03:39:15.021130+0000","last_peered":"2024-09-11T03:39:15.021130+0000","last_clean":"2024-09-11T03:39:15.021130+0000","last_became_active":"2024-09-11T03:38:29.930120+0000","last_became_peered":"2024-09-11T03:38:29.930120+0000","last_unstale":"2024-09-11T03:39:15.021130+0000","last_undegraded":"2024-09-11T03:39:15.021130+0000","last_fullsized":"2024-09-11T03:39:15.021130+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:41.948777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054800200000000002,"stat_sum":{"num_bytes":3260525,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":5533,"num_write":14,"num_write_kb":3881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.1","version":"138'19","reported_seq":246,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-11T03:39:17.850591+0000","last_change":"2024-09-11T03:39:17.100943+0000","last_active":"2024-09-11T03:39:17.850591+0000","last_peered":"2024-09-11T03:39:17.850591+0000","last_clean":"2024-09-11T03:39:17.850591+0000","last_became_active":"2024-09-11T03:38:30.162952+0000","last_became_peered":"2024-09-11T03:38:30.162952+0000","last_unstale":"2024-09-11T03:39:17.850591+0000","last_undegraded":"2024-09-11T03:39:17.850591+0000","last_fullsized":"2024-09-11T03:39:17.850591+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:48.306649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047621099999999998,"stat_sum":{"num_bytes":2050454,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":10354,"num_write":11,"num_write_kb":4517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12","length":"4"}]},{"pgid":"2.2","version":"135'51","reported_seq":266,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.019575+0000","last_change":"2024-09-11T03:39:15.019723+0000","last_active":"2024-09-11T03:39:15.019575+0000","last_peered":"2024-09-11T03:39:15.019575+0000","last_clean":"2024-09-11T03:39:15.019575+0000","last_became_active":"2024-09-11T03:38:29.889523+0000","last_became_peered":"2024-09-11T03:38:29.889523+0000","last_unstale":"2024-09-11T03:39:15.019575+0000","last_undegraded":"2024-09-11T03:39:15.019575+0000","last_fullsized":"2024-09-11T03:39:15.019575+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:44.494077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035620600000000002,"stat_sum":{"num_bytes":2453789,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":5,"num_write":28,"num_write_kb":13972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.4","version":"136'52","reported_seq":268,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.846634+0000","last_change":"2024-09-11T03:39:15.017624+0000","last_active":"2024-09-11T03:39:15.846634+0000","last_peered":"2024-09-11T03:39:15.846634+0000","last_clean":"2024-09-11T03:39:15.846634+0000","last_became_active":"2024-09-11T03:38:30.163226+0000","last_became_peered":"2024-09-11T03:38:30.163226+0000","last_unstale":"2024-09-11T03:39:15.846634+0000","last_undegraded":"2024-09-11T03:39:15.846634+0000","last_fullsized":"2024-09-11T03:39:15.846634+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:42.429151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049723699999999998,"stat_sum":{"num_bytes":7932838,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":8,"num_write":39,"num_write_kb":13761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.5","version":"136'94","reported_seq":331,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.848688+0000","last_change":"2024-09-11T03:39:14.840149+0000","last_active":"2024-09-11T03:39:15.848688+0000","last_peered":"2024-09-11T03:39:15.848688+0000","last_clean":"2024-09-11T03:39:15.848688+0000","last_became_active":"2024-09-11T03:38:30.163080+0000","last_became_peered":"2024-09-11T03:38:30.163080+0000","last_unstale":"2024-09-11T03:39:15.848688+0000","last_undegraded":"2024-09-11T03:39:15.848688+0000","last_fullsized":"2024-09-11T03:39:15.848688+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:50.745787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094381,"stat_sum":{"num_bytes":15445501,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":51892,"num_write":71,"num_write_kb":23938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.6","version":"134'27","reported_seq":245,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.846294+0000","last_change":"2024-09-11T03:39:15.017828+0000","last_active":"2024-09-11T03:39:15.846294+0000","last_peered":"2024-09-11T03:39:15.846294+0000","last_clean":"2024-09-11T03:39:15.846294+0000","last_became_active":"2024-09-11T03:38:29.887945+0000","last_became_peered":"2024-09-11T03:38:29.887945+0000","last_unstale":"2024-09-11T03:39:15.846294+0000","last_undegraded":"2024-09-11T03:39:15.846294+0000","last_fullsized":"2024-09-11T03:39:15.846294+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:29.000919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070240700000000005,"stat_sum":{"num_bytes":3848770,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":16651,"num_write":16,"num_write_kb":9866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.7","version":"136'58","reported_seq":291,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-11T03:39:17.217643+0000","last_change":"2024-09-11T03:39:17.217769+0000","last_active":"2024-09-11T03:39:17.217643+0000","last_peered":"2024-09-11T03:39:17.217643+0000","last_clean":"2024-09-11T03:39:17.217643+0000","last_became_active":"2024-09-11T03:38:29.930154+0000","last_became_peered":"2024-09-11T03:38:29.930154+0000","last_unstale":"2024-09-11T03:39:17.217643+0000","last_undegraded":"2024-09-11T03:39:17.217643+0000","last_fullsized":"2024-09-11T03:39:17.217643+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:56.529038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000313863,"stat_sum":{"num_bytes":7613472,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":11832,"num_write":43,"num_write_kb":16991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12","length":"4"}]},{"pgid":"2.8","version":"136'37","reported_seq":257,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.021076+0000","last_change":"2024-09-11T03:39:15.021319+0000","last_active":"2024-09-11T03:39:15.021076+0000","last_peered":"2024-09-11T03:39:15.021076+0000","last_clean":"2024-09-11T03:39:15.021076+0000","last_became_active":"2024-09-11T03:38:30.162819+0000","last_became_peered":"2024-09-11T03:38:30.162819+0000","last_unstale":"2024-09-11T03:39:15.021076+0000","last_undegraded":"2024-09-11T03:39:15.021076+0000","last_fullsized":"2024-09-11T03:39:15.021076+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:56.698633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050697399999999999,"stat_sum":{"num_bytes":9798382,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":29182,"num_write":33,"num_write_kb":11869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.9","version":"136'108","reported_seq":353,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.846280+0000","last_change":"2024-09-11T03:39:15.017545+0000","last_active":"2024-09-11T03:39:15.846280+0000","last_peered":"2024-09-11T03:39:15.846280+0000","last_clean":"2024-09-11T03:39:15.846280+0000","last_became_active":"2024-09-11T03:38:29.888204+0000","last_became_peered":"2024-09-11T03:38:29.888204+0000","last_unstale":"2024-09-11T03:39:15.846280+0000","last_undegraded":"2024-09-11T03:39:15.846280+0000","last_fullsized":"2024-09-11T03:39:15.846280+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:40.557572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050069800000000005,"stat_sum":{"num_bytes":12857223,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":113,"num_read_kb":83664,"num_write":67,"num_write_kb":26037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]},{"pgid":"2.a","version":"134'7","reported_seq":225,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-11T03:39:16.838979+0000","last_change":"2024-09-11T03:39:16.839180+0000","last_active":"2024-09-11T03:39:16.838979+0000","last_peered":"2024-09-11T03:39:16.838979+0000","last_clean":"2024-09-11T03:39:16.838979+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:39:16.838979+0000","last_undegraded":"2024-09-11T03:39:16.838979+0000","last_fullsized":"2024-09-11T03:39:16.838979+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:38.805883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052170500000000002,"stat_sum":{"num_bytes":3109262,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":10361,"num_write":5,"num_write_kb":4561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"12","length":"4"}]},{"pgid":"2.b","version":"136'25","reported_seq":248,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-11T03:39:15.848626+0000","last_change":"2024-09-11T03:39:14.839769+0000","last_active":"2024-09-11T03:39:15.848626+0000","last_peered":"2024-09-11T03:39:15.848626+0000","last_clean":"2024-09-11T03:39:15.848626+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:39:15.848626+0000","last_undegraded":"2024-09-11T03:39:15.848626+0000","last_fullsized":"2024-09-11T03:39:15.848626+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:48.151468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058574600000000005,"stat_sum":{"num_bytes":6588278,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":15668,"num_write":21,"num_write_kb":10629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e","length":"2"},{"start":"11","length":"4"}]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":126773980,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":610,"num_read_kb":441102,"num_write":521,"num_write_kb":203527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210063360,"data_stored":208763994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":767,"ondisk_log_size":767,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285593,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40896,"kb_used_data":13868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93704256,"statfs":{"total":95995035648,"available":95953158144,"internally_reserved":0,"allocated":14200832,"data_stored":13422708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514525,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":38176,"kb_used_data":11148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93706976,"statfs":{"total":95995035648,"available":95955943424,"internally_reserved":0,"allocated":11415552,"data_stored":10649358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743459,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56480,"kb_used_data":27648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93688672,"statfs":{"total":95995035648,"available":95937200128,"internally_reserved":0,"allocated":28311552,"data_stored":27392781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037801,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54432,"kb_used_data":23240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93690720,"statfs":{"total":95995035648,"available":95939297280,"internally_reserved":0,"allocated":23797760,"data_stored":22950125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299437,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54136,"kb_used_data":21696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93691016,"statfs":{"total":95995035648,"available":95939600384,"internally_reserved":0,"allocated":22216704,"data_stored":21345454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":38040,"kb_used_data":6844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93707112,"statfs":{"total":95995035648,"available":95956082688,"internally_reserved":0,"allocated":7008256,"data_stored":6268732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33792,"kb_used_data":5516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93711360,"statfs":{"total":95995035648,"available":95960432640,"internally_reserved":0,"allocated":5648384,"data_stored":4921070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986237,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52192,"kb_used_data":21000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93692960,"statfs":{"total":95995035648,"available":95941591040,"internally_reserved":0,"allocated":21504000,"data_stored":20617346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58448,"kb_used_data":27260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93686704,"statfs":{"total":95995035648,"available":95935184896,"internally_reserved":0,"allocated":27914240,"data_stored":27013312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":42528,"kb_used_data":14248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93702624,"statfs":{"total":95995035648,"available":95951486976,"internally_reserved":0,"allocated":14589952,"data_stored":13807848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49748,"kb_used_data":22712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93695404,"statfs":{"total":95995035648,"available":95944093696,"internally_reserved":0,"allocated":23257088,"data_stored":22453015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54960,"kb_used_data":27928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93690192,"statfs":{"total":95995035648,"available":95938756608,"internally_reserved":0,"allocated":28598272,"data_stored":27723877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":27041792,"data_stored":26878453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21688320,"data_stored":21599507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14868480,"data_stored":14797432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":26320896,"data_stored":26143540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":19886080,"data_stored":19731398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4091904,"data_stored":4075646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4907008,"data_stored":4886296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20127744,"data_stored":19971102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":22241280,"data_stored":22104701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":26730496,"data_stored":26531069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":9859072,"data_stored":9803934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12300288,"data_stored":12240916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:39:22.102 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-11T03:39:22.102 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:39:22.103 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:39:22.103 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd out 6 2024-09-11T03:39:22.110 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:22.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:22.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:22.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:21 smithi121 ceph-mon[30729]: pgmap v370: 17 pgs: 17 active+clean; 121 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.1 MiB/s wr, 3 op/s 2024-09-11T03:39:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:21 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:21 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3273212094' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:39:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:21 smithi121 ceph-mon[30729]: osdmap e144: 12 total, 12 up, 12 in 2024-09-11T03:39:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:21 smithi121 ceph-mon[30729]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:39:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:21 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1620305951' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:21 smithi121 ceph-mon[30729]: osdmap e145: 12 total, 12 up, 12 in 2024-09-11T03:39:22.212 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:21 smithi175 ceph-mon[30894]: pgmap v370: 17 pgs: 17 active+clean; 121 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.1 MiB/s wr, 3 op/s 2024-09-11T03:39:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:21 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:21 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3273212094' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:39:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:21 smithi175 ceph-mon[30894]: osdmap e144: 12 total, 12 up, 12 in 2024-09-11T03:39:22.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:21 smithi175 ceph-mon[30894]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:39:22.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:21 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1620305951' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:22.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:21 smithi175 ceph-mon[30894]: osdmap e145: 12 total, 12 up, 12 in 2024-09-11T03:39:22.238 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:22.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:22.239 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:22.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T03:39:22.339 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:22.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:22.361 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:22.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:22.462 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:22.488 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:22.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:22.490 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:22.496 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:39:22.591 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:22.616 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:22.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:22.618 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:22.719 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:22.740 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:22.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:22.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:22.826 INFO:tasks.rados.rados.2.smithi175.stdout:306: done (1 left) 2024-09-11T03:39:22.826 INFO:tasks.rados.rados.2.smithi175.stdout:307: done (0 left) 2024-09-11T03:39:22.827 INFO:tasks.rados.rados.2.smithi175.stdout:308: read oid 46 snap 23 2024-09-11T03:39:22.827 INFO:tasks.rados.rados.2.smithi175.stdout:308: expect (ObjNum 114 snap 18 seq_num 114) 2024-09-11T03:39:22.827 INFO:tasks.rados.rados.2.smithi175.stdout:309: read oid 19 snap -1 2024-09-11T03:39:22.827 INFO:tasks.rados.rados.2.smithi175.stdout:309: expect (ObjNum 101 snap 16 seq_num 101) 2024-09-11T03:39:22.827 INFO:tasks.rados.rados.2.smithi175.stdout:310: snap_create 2024-09-11T03:39:22.835 INFO:tasks.rados.rados.2.smithi175.stdout:308: expect (ObjNum 114 snap 18 seq_num 114) 2024-09-11T03:39:22.841 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:22.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:22.864 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:22.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:22.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:22.989 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:22.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:22.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:23.022 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:39:23.023 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:39:23.023 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:39:23.023 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:39:23.044 INFO:tasks.rados.rados.2.smithi175.stdout:309: expect (ObjNum 101 snap 16 seq_num 101) 2024-09-11T03:39:23.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:23.112 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:23.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:23.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:23.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:23.235 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:23.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:23.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:23.337 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:23.358 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:23.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:23.359 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:23.413 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:23.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:23.481 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:23.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:23.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:23.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:23.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:23.606 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:23.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:23.708 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:23.731 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:23.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:23.733 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:23.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T03:39:23.827 INFO:tasks.rados.rados.2.smithi175.stdout:308: done (2 left) 2024-09-11T03:39:23.827 INFO:tasks.rados.rados.2.smithi175.stdout:309: done (1 left) 2024-09-11T03:39:23.827 INFO:tasks.rados.rados.2.smithi175.stdout:310: done (0 left) 2024-09-11T03:39:23.827 INFO:tasks.rados.rados.2.smithi175.stdout:311: snap_remove snap 22 2024-09-11T03:39:23.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:23.855 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:23.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:23.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:23.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:23 smithi121 ceph-mon[30729]: pgmap v373: 17 pgs: 17 active+clean; 115 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 6.4 MiB/s wr, 15 op/s 2024-09-11T03:39:23.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:23 smithi121 ceph-mon[30729]: osdmap e146: 12 total, 12 up, 12 in 2024-09-11T03:39:23.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:23.978 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:39:23.979 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:23.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:23.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:24.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:23 smithi084 ceph-mon[27254]: pgmap v373: 17 pgs: 17 active+clean; 115 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 6.4 MiB/s wr, 15 op/s 2024-09-11T03:39:24.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:23 smithi084 ceph-mon[27254]: osdmap e146: 12 total, 12 up, 12 in 2024-09-11T03:39:24.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:24.102 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:24.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:24.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:24.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:24.226 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:24.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:24.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:24.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:23 smithi175 ceph-mon[30894]: pgmap v373: 17 pgs: 17 active+clean; 115 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 6.4 MiB/s wr, 15 op/s 2024-09-11T03:39:24.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:23 smithi175 ceph-mon[30894]: osdmap e146: 12 total, 12 up, 12 in 2024-09-11T03:39:24.328 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:24.354 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:24.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:24.355 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:24.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:24 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a[27250]: 2024-09-11T03:39:24.050+0000 7fae0fb7d640 -1 mon.a@0(leader).osd e147 definitely_dead 0 2024-09-11T03:39:24.452 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:39:24.452 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:39:24.453 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:39:24.453 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:39:24.453 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:39:24.456 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:24.480 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:24.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:24.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:24.582 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:24.609 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:24.609 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:24.710 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:24.734 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:24.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:24.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:24.833 INFO:teuthology.orchestra.run.smithi084.stderr:marked out osd.6. 2024-09-11T03:39:24.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:24.838 INFO:tasks.rados.rados.2.smithi175.stdout:311: done (0 left) 2024-09-11T03:39:24.838 INFO:tasks.rados.rados.2.smithi175.stdout:312: copy_from oid 20 from oid 1 current snap is 25 2024-09-11T03:39:24.838 INFO:tasks.rados.rados.2.smithi175.stdout:313: snap_remove snap 23 2024-09-11T03:39:24.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:24.860 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:24.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:24.961 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:24.982 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:24.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:24.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:24.999 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:39:25.084 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:25.105 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:25.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:25.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:25.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:24 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:39:25.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:24 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:39:25.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:24 smithi084 ceph-mon[27254]: osdmap e147: 12 total, 12 up, 12 in 2024-09-11T03:39:25.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:24 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/546640128' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-11T03:39:25.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:24 smithi084 ceph-mon[27254]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-11T03:39:25.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:24 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:39:25.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:24 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:39:25.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:24 smithi121 ceph-mon[30729]: osdmap e147: 12 total, 12 up, 12 in 2024-09-11T03:39:25.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:24 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/546640128' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-11T03:39:25.125 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:24 smithi121 ceph-mon[30729]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-11T03:39:25.152 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T03:39:25.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:24 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:39:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:24 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:39:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:24 smithi175 ceph-mon[30894]: osdmap e147: 12 total, 12 up, 12 in 2024-09-11T03:39:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:24 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/546640128' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-11T03:39:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:24 smithi175 ceph-mon[30894]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-11T03:39:25.255 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:25.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:25.357 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:25.383 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:25.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:25.384 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:39:25.441 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:39:25.485 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:25.506 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:25.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:25.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:25.608 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:25.634 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:25.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:25.635 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:25.695 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:25.696 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:25.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:25.756 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:25.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:25.838 INFO:tasks.rados.rados.2.smithi175.stdout:313: done (1 left) 2024-09-11T03:39:25.838 INFO:tasks.rados.rados.2.smithi175.stdout:314: snap_remove snap 24 2024-09-11T03:39:25.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:25.878 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:25.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:25.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:25.880 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:39:25.880 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:39:25.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:39:25.881 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:39:25.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.881 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:221 2.3 e103/95 2:c6c49f2c:::smithi17557870-44:head v 103'8, mlcod=100'5)", 2024-09-11T03:39:25.881 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.951299+0000", 2024-09-11T03:39:25.881 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 45.920138051999999, 2024-09-11T03:39:25.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044372860000000004, 2024-09-11T03:39:25.881 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.881 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.881 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:25.881 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.951299+0000", 2024-09-11T03:39:25.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.951299+0000", 2024-09-11T03:39:25.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.951299+0000", 2024-09-11T03:39:25.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9600000000000001e-07 2024-09-11T03:39:25.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.952994+0000", 2024-09-11T03:39:25.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001694624 2024-09-11T03:39:25.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.953021+0000", 2024-09-11T03:39:25.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6962999999999999e-05 2024-09-11T03:39:25.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.953035+0000", 2024-09-11T03:39:25.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4078e-05 2024-09-11T03:39:25.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.953138+0000", 2024-09-11T03:39:25.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102546 2024-09-11T03:39:25.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.953161+0000", 2024-09-11T03:39:25.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3710000000000002e-05 2024-09-11T03:39:25.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:25.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.955661+0000", 2024-09-11T03:39:25.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002500015 2024-09-11T03:39:25.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.955736+0000", 2024-09-11T03:39:25.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5154000000000006e-05 2024-09-11T03:39:25.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.885 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.885 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:228 2.9 e105/95 2:95fa802a:::smithi17557870-22:head v 105'20, mlcod=100'17)", 2024-09-11T03:39:25.886 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:39:25.886 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 43.904494618000001, 2024-09-11T03:39:25.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01695953, 2024-09-11T03:39:25.886 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.886 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.886 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:25.886 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:39:25.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:39:25.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:39:25.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2600000000000002e-07 2024-09-11T03:39:25.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982417+0000", 2024-09-11T03:39:25.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01547431 2024-09-11T03:39:25.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982440+0000", 2024-09-11T03:39:25.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2506000000000001e-05 2024-09-11T03:39:25.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982448+0000", 2024-09-11T03:39:25.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2989999999999997e-06 2024-09-11T03:39:25.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982562+0000", 2024-09-11T03:39:25.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114084 2024-09-11T03:39:25.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982583+0000", 2024-09-11T03:39:25.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1160999999999999e-05 2024-09-11T03:39:25.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:25.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.983852+0000", 2024-09-11T03:39:25.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001268953 2024-09-11T03:39:25.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.983902+0000", 2024-09-11T03:39:25.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9691000000000002e-05 2024-09-11T03:39:25.890 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.890 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.890 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.890 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:345 2.0 2:0762776a:::smithi17557870-25:head [write 742876~447756] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:39:25.890 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.827240+0000", 2024-09-11T03:39:25.890 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.044197302999997, 2024-09-11T03:39:25.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076328890000000003, 2024-09-11T03:39:25.890 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.890 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.890 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:25.890 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:25.890 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:25.891 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:25.891 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 345 2024-09-11T03:39:25.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.891 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.827240+0000", 2024-09-11T03:39:25.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.827240+0000", 2024-09-11T03:39:25.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.827243+0000", 2024-09-11T03:39:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0110000000000001e-06 2024-09-11T03:39:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828270+0000", 2024-09-11T03:39:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001026838 2024-09-11T03:39:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828298+0000", 2024-09-11T03:39:25.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8561999999999999e-05 2024-09-11T03:39:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828312+0000", 2024-09-11T03:39:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3781000000000001e-05 2024-09-11T03:39:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828398+0000", 2024-09-11T03:39:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6336999999999997e-05 2024-09-11T03:39:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:39:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828405+0000", 2024-09-11T03:39:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9079999999999998e-06 2024-09-11T03:39:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830015+0000", 2024-09-11T03:39:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001609344 2024-09-11T03:39:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830095+0000", 2024-09-11T03:39:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0259000000000005e-05 2024-09-11T03:39:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10", 2024-09-11T03:39:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830248+0000", 2024-09-11T03:39:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015351500000000001 2024-09-11T03:39:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:39:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.831760+0000", 2024-09-11T03:39:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015116540000000001 2024-09-11T03:39:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.832800+0000", 2024-09-11T03:39:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010400800000000001 2024-09-11T03:39:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834760+0000", 2024-09-11T03:39:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019600809999999998 2024-09-11T03:39:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834804+0000", 2024-09-11T03:39:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3922999999999999e-05 2024-09-11T03:39:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834873+0000", 2024-09-11T03:39:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8596000000000005e-05 2024-09-11T03:39:25.896 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:346 2.0 2:0762776a:::smithi17557870-25:head [write 1729599~575687] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:39:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.833957+0000", 2024-09-11T03:39:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.037480578, 2024-09-11T03:39:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00864616, 2024-09-11T03:39:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:25.897 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 346 2024-09-11T03:39:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833957+0000", 2024-09-11T03:39:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833957+0000", 2024-09-11T03:39:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833958+0000", 2024-09-11T03:39:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.674e-06 2024-09-11T03:39:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838969+0000", 2024-09-11T03:39:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005010993 2024-09-11T03:39:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839016+0000", 2024-09-11T03:39:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6576000000000003e-05 2024-09-11T03:39:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839037+0000", 2024-09-11T03:39:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1356e-05 2024-09-11T03:39:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839197+0000", 2024-09-11T03:39:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000159338 2024-09-11T03:39:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839266+0000", 2024-09-11T03:39:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9758999999999994e-05 2024-09-11T03:39:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10", 2024-09-11T03:39:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839374+0000", 2024-09-11T03:39:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010791899999999999 2024-09-11T03:39:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:39:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.840763+0000", 2024-09-11T03:39:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013883999999999999 2024-09-11T03:39:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842254+0000", 2024-09-11T03:39:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001491409 2024-09-11T03:39:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842538+0000", 2024-09-11T03:39:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028420099999999998 2024-09-11T03:39:25.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842573+0000", 2024-09-11T03:39:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4381000000000002e-05 2024-09-11T03:39:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842603+0000", 2024-09-11T03:39:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0154e-05 2024-09-11T03:39:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.903 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:341 2.9 e113/95 2:9718db5a:::smithi17557870-3:head v 113'37, mlcod=109'33)", 2024-09-11T03:39:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.834470+0000", 2024-09-11T03:39:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.036966667999998, 2024-09-11T03:39:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048705010000000002, 2024-09-11T03:39:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834470+0000", 2024-09-11T03:39:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834470+0000", 2024-09-11T03:39:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834471+0000", 2024-09-11T03:39:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8900000000000001e-07 2024-09-11T03:39:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.837396+0000", 2024-09-11T03:39:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029247890000000001 2024-09-11T03:39:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.837411+0000", 2024-09-11T03:39:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5258e-05 2024-09-11T03:39:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.837422+0000", 2024-09-11T03:39:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0706999999999999e-05 2024-09-11T03:39:25.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.837480+0000", 2024-09-11T03:39:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8854999999999998e-05 2024-09-11T03:39:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.837499+0000", 2024-09-11T03:39:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8783999999999998e-05 2024-09-11T03:39:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839305+0000", 2024-09-11T03:39:25.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001805783 2024-09-11T03:39:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839341+0000", 2024-09-11T03:39:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6035999999999999e-05 2024-09-11T03:39:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:347 2.0 2:0762776a:::smithi17557870-25:head [write 3101234~41642] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:39:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.839105+0000", 2024-09-11T03:39:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.032332615999998, 2024-09-11T03:39:25.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003592423, 2024-09-11T03:39:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 347 2024-09-11T03:39:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839105+0000", 2024-09-11T03:39:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839105+0000", 2024-09-11T03:39:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839106+0000", 2024-09-11T03:39:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.483e-06 2024-09-11T03:39:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839292+0000", 2024-09-11T03:39:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000186381 2024-09-11T03:39:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839303+0000", 2024-09-11T03:39:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0560000000000001e-05 2024-09-11T03:39:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839311+0000", 2024-09-11T03:39:25.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1589999999999993e-06 2024-09-11T03:39:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.840075+0000", 2024-09-11T03:39:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00076384599999999999 2024-09-11T03:39:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.840138+0000", 2024-09-11T03:39:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3001000000000002e-05 2024-09-11T03:39:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10", 2024-09-11T03:39:25.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.840237+0000", 2024-09-11T03:39:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9061000000000006e-05 2024-09-11T03:39:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:39:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.840768+0000", 2024-09-11T03:39:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000530504 2024-09-11T03:39:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842378+0000", 2024-09-11T03:39:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016107459999999999 2024-09-11T03:39:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842644+0000", 2024-09-11T03:39:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0002653 2024-09-11T03:39:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842674+0000", 2024-09-11T03:39:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0626999999999997e-05 2024-09-11T03:39:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842697+0000", 2024-09-11T03:39:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2755000000000002e-05 2024-09-11T03:39:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:348 2.0 2:0762776a:::smithi17557870-25:head [setxattr _header (56) in=7b,truncate 3142876] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:39:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.839327+0000", 2024-09-11T03:39:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.032110562, 2024-09-11T03:39:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034573970000000001, 2024-09-11T03:39:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:25.915 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 348 2024-09-11T03:39:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839327+0000", 2024-09-11T03:39:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839327+0000", 2024-09-11T03:39:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839327+0000", 2024-09-11T03:39:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5700000000000002e-07 2024-09-11T03:39:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839331+0000", 2024-09-11T03:39:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0980000000000004e-06 2024-09-11T03:39:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839339+0000", 2024-09-11T03:39:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-09-11T03:39:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839345+0000", 2024-09-11T03:39:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7320000000000003e-06 2024-09-11T03:39:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.840492+0000", 2024-09-11T03:39:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011473290000000001 2024-09-11T03:39:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.840544+0000", 2024-09-11T03:39:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2092000000000002e-05 2024-09-11T03:39:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10", 2024-09-11T03:39:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.840625+0000", 2024-09-11T03:39:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0599000000000005e-05 2024-09-11T03:39:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:39:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.840892+0000", 2024-09-11T03:39:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00026706100000000002 2024-09-11T03:39:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842406+0000", 2024-09-11T03:39:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001513985 2024-09-11T03:39:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842725+0000", 2024-09-11T03:39:25.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031933500000000002 2024-09-11T03:39:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842755+0000", 2024-09-11T03:39:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9941e-05 2024-09-11T03:39:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842784+0000", 2024-09-11T03:39:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9118000000000001e-05 2024-09-11T03:39:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.921 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:349 2.0 2:0762776a:::smithi17557870-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:39:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.839353+0000", 2024-09-11T03:39:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.032083692999997, 2024-09-11T03:39:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035774140000000001, 2024-09-11T03:39:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.922 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 349 2024-09-11T03:39:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839353+0000", 2024-09-11T03:39:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839353+0000", 2024-09-11T03:39:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839354+0000", 2024-09-11T03:39:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9599999999999999e-07 2024-09-11T03:39:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839356+0000", 2024-09-11T03:39:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4980000000000001e-06 2024-09-11T03:39:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839363+0000", 2024-09-11T03:39:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7460000000000003e-06 2024-09-11T03:39:25.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839369+0000", 2024-09-11T03:39:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3730000000000001e-06 2024-09-11T03:39:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.840790+0000", 2024-09-11T03:39:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0014214659999999999 2024-09-11T03:39:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:39:25.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.840849+0000", 2024-09-11T03:39:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8977000000000001e-05 2024-09-11T03:39:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842816+0000", 2024-09-11T03:39:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001966537 2024-09-11T03:39:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842872+0000", 2024-09-11T03:39:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5934999999999999e-05 2024-09-11T03:39:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842931+0000", 2024-09-11T03:39:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9385999999999997e-05 2024-09-11T03:39:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:386 2.0 2:0762776a:::smithi17557870-25:head [write 460525~690852,stat] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:39:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:50.986853+0000", 2024-09-11T03:39:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.884584378, 2024-09-11T03:39:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041535269999999997, 2024-09-11T03:39:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.928 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 386 2024-09-11T03:39:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.986853+0000", 2024-09-11T03:39:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.986853+0000", 2024-09-11T03:39:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.986856+0000", 2024-09-11T03:39:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3390000000000001e-06 2024-09-11T03:39:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.987834+0000", 2024-09-11T03:39:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00097834700000000007 2024-09-11T03:39:25.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.987850+0000", 2024-09-11T03:39:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5048999999999999e-05 2024-09-11T03:39:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.987874+0000", 2024-09-11T03:39:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4068000000000001e-05 2024-09-11T03:39:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.987935+0000", 2024-09-11T03:39:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1079000000000003e-05 2024-09-11T03:39:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.988006+0000", 2024-09-11T03:39:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1669999999999997e-05 2024-09-11T03:39:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10", 2024-09-11T03:39:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.988116+0000", 2024-09-11T03:39:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000109762 2024-09-11T03:39:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:39:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.989650+0000", 2024-09-11T03:39:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015339119999999999 2024-09-11T03:39:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.990893+0000", 2024-09-11T03:39:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012433030000000001 2024-09-11T03:39:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.990934+0000", 2024-09-11T03:39:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0847999999999999e-05 2024-09-11T03:39:25.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.990973+0000", 2024-09-11T03:39:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9062e-05 2024-09-11T03:39:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.991006+0000", 2024-09-11T03:39:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3087999999999999e-05 2024-09-11T03:39:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.934 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:387 2.0 2:0762776a:::smithi17557870-25:head [write 1622620~730352] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:39:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:50.993410+0000", 2024-09-11T03:39:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.878026669999997, 2024-09-11T03:39:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042592350000000001, 2024-09-11T03:39:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 387 2024-09-11T03:39:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.993410+0000", 2024-09-11T03:39:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.993410+0000", 2024-09-11T03:39:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.993412+0000", 2024-09-11T03:39:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.33e-06 2024-09-11T03:39:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.994540+0000", 2024-09-11T03:39:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001127779 2024-09-11T03:39:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.994553+0000", 2024-09-11T03:39:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3458000000000001e-05 2024-09-11T03:39:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.994559+0000", 2024-09-11T03:39:25.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8000000000000004e-06 2024-09-11T03:39:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.994639+0000", 2024-09-11T03:39:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0593999999999994e-05 2024-09-11T03:39:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.994738+0000", 2024-09-11T03:39:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8789999999999994e-05 2024-09-11T03:39:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10", 2024-09-11T03:39:25.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.994894+0000", 2024-09-11T03:39:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015616100000000001 2024-09-11T03:39:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:39:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.996140+0000", 2024-09-11T03:39:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012450919999999999 2024-09-11T03:39:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.997551+0000", 2024-09-11T03:39:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001411914 2024-09-11T03:39:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:25.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.997600+0000", 2024-09-11T03:39:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8987999999999999e-05 2024-09-11T03:39:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.997636+0000", 2024-09-11T03:39:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5975e-05 2024-09-11T03:39:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.997670+0000", 2024-09-11T03:39:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3353999999999999e-05 2024-09-11T03:39:25.941 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:428 2.9 e119/95 2:95fa802a:::smithi17557870-22:head v 119'41, mlcod=113'37)", 2024-09-11T03:39:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:55.867789+0000", 2024-09-11T03:39:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.003648394999999, 2024-09-11T03:39:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053342210000000001, 2024-09-11T03:39:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.867789+0000", 2024-09-11T03:39:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.867789+0000", 2024-09-11T03:39:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.867789+0000", 2024-09-11T03:39:25.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8099999999999999e-07 2024-09-11T03:39:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.871598+0000", 2024-09-11T03:39:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038090229999999999 2024-09-11T03:39:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.871611+0000", 2024-09-11T03:39:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3474e-05 2024-09-11T03:39:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.871620+0000", 2024-09-11T03:39:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.191e-06 2024-09-11T03:39:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.871672+0000", 2024-09-11T03:39:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2074999999999999e-05 2024-09-11T03:39:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.871690+0000", 2024-09-11T03:39:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8601e-05 2024-09-11T03:39:25.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:39:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.873081+0000", 2024-09-11T03:39:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013902630000000001 2024-09-11T03:39:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.873123+0000", 2024-09-11T03:39:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2413000000000002e-05 2024-09-11T03:39:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:569 2.8 2:1984a3a9:::smithi17557870-23:head [read 0~3075648,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-09-11T03:39:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:04.854731+0000", 2024-09-11T03:39:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.016705727000002, 2024-09-11T03:39:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038708620000000001, 2024-09-11T03:39:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 569 2024-09-11T03:39:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854731+0000", 2024-09-11T03:39:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854731+0000", 2024-09-11T03:39:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854734+0000", 2024-09-11T03:39:25.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9730000000000002e-06 2024-09-11T03:39:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854748+0000", 2024-09-11T03:39:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3349999999999999e-05 2024-09-11T03:39:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854781+0000", 2024-09-11T03:39:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2842999999999997e-05 2024-09-11T03:39:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854796+0000", 2024-09-11T03:39:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5563e-05 2024-09-11T03:39:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854954+0000", 2024-09-11T03:39:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000157505 2024-09-11T03:39:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.855091+0000", 2024-09-11T03:39:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000136969 2024-09-11T03:39:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.858602+0000", 2024-09-11T03:39:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035116589999999999 2024-09-11T03:39:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:570 2.8 2:1984a3a9:::smithi17557870-23:head [read 0~3075648,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-09-11T03:39:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:04.854860+0000", 2024-09-11T03:39:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.016577275, 2024-09-11T03:39:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051107310000000003, 2024-09-11T03:39:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 570 2024-09-11T03:39:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854860+0000", 2024-09-11T03:39:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854860+0000", 2024-09-11T03:39:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854861+0000", 2024-09-11T03:39:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5300000000000002e-07 2024-09-11T03:39:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854869+0000", 2024-09-11T03:39:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7030000000000006e-06 2024-09-11T03:39:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854882+0000", 2024-09-11T03:39:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3247e-05 2024-09-11T03:39:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854891+0000", 2024-09-11T03:39:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0459999999999994e-06 2024-09-11T03:39:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.858628+0000", 2024-09-11T03:39:25.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037373969999999999 2024-09-11T03:39:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.858690+0000", 2024-09-11T03:39:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1452999999999996e-05 2024-09-11T03:39:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.859971+0000", 2024-09-11T03:39:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001280932 2024-09-11T03:39:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:571 2.8 2:1984a3a9:::smithi17557870-23:head [sparse-read 0~1350639] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-09-11T03:39:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:04.854908+0000", 2024-09-11T03:39:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.016529129999999, 2024-09-11T03:39:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052826729999999999, 2024-09-11T03:39:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 571 2024-09-11T03:39:25.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854908+0000", 2024-09-11T03:39:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854908+0000", 2024-09-11T03:39:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854909+0000", 2024-09-11T03:39:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7000000000000003e-07 2024-09-11T03:39:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854912+0000", 2024-09-11T03:39:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.444e-06 2024-09-11T03:39:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854922+0000", 2024-09-11T03:39:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5959999999999993e-06 2024-09-11T03:39:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.854929+0000", 2024-09-11T03:39:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4000000000000003e-06 2024-09-11T03:39:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.860011+0000", 2024-09-11T03:39:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050815469999999996 2024-09-11T03:39:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.860072+0000", 2024-09-11T03:39:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1597e-05 2024-09-11T03:39:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.860191+0000", 2024-09-11T03:39:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118219 2024-09-11T03:39:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:622 2.8 2:162357fa:::smithi17557870-18:head [read 0~3563051,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e130)", 2024-09-11T03:39:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:07.066754+0000", 2024-09-11T03:39:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.804682902, 2024-09-11T03:39:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044337379999999996, 2024-09-11T03:39:25.960 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 622 2024-09-11T03:39:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066754+0000", 2024-09-11T03:39:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066754+0000", 2024-09-11T03:39:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066757+0000", 2024-09-11T03:39:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5390000000000002e-06 2024-09-11T03:39:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066765+0000", 2024-09-11T03:39:25.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7510000000000008e-06 2024-09-11T03:39:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066783+0000", 2024-09-11T03:39:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8621e-05 2024-09-11T03:39:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066794+0000", 2024-09-11T03:39:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0710000000000001e-05 2024-09-11T03:39:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066868+0000", 2024-09-11T03:39:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4606999999999999e-05 2024-09-11T03:39:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066934+0000", 2024-09-11T03:39:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5092000000000006e-05 2024-09-11T03:39:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.071188+0000", 2024-09-11T03:39:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042544180000000003 2024-09-11T03:39:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.964 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:623 2.8 2:162357fa:::smithi17557870-18:head [read 0~3563051,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e130)", 2024-09-11T03:39:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:07.066819+0000", 2024-09-11T03:39:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.804618068, 2024-09-11T03:39:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058140739999999998, 2024-09-11T03:39:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 623 2024-09-11T03:39:25.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066819+0000", 2024-09-11T03:39:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066819+0000", 2024-09-11T03:39:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066820+0000", 2024-09-11T03:39:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3900000000000005e-07 2024-09-11T03:39:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066823+0000", 2024-09-11T03:39:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4750000000000002e-06 2024-09-11T03:39:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066840+0000", 2024-09-11T03:39:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6603999999999999e-05 2024-09-11T03:39:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066846+0000", 2024-09-11T03:39:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5409999999999997e-06 2024-09-11T03:39:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.071239+0000", 2024-09-11T03:39:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043923039999999997 2024-09-11T03:39:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.071311+0000", 2024-09-11T03:39:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2408000000000002e-05 2024-09-11T03:39:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.072633+0000", 2024-09-11T03:39:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013222030000000001 2024-09-11T03:39:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:624 2.8 2:162357fa:::smithi17557870-18:head [sparse-read 0~1957854] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e130)", 2024-09-11T03:39:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:07.066857+0000", 2024-09-11T03:39:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.804579818000001, 2024-09-11T03:39:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059990820000000002, 2024-09-11T03:39:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.969 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 624 2024-09-11T03:39:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066857+0000", 2024-09-11T03:39:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066857+0000", 2024-09-11T03:39:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066858+0000", 2024-09-11T03:39:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1699999999999998e-07 2024-09-11T03:39:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066861+0000", 2024-09-11T03:39:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2210000000000002e-06 2024-09-11T03:39:25.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066870+0000", 2024-09-11T03:39:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8149999999999995e-06 2024-09-11T03:39:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066883+0000", 2024-09-11T03:39:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3220999999999999e-05 2024-09-11T03:39:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.072671+0000", 2024-09-11T03:39:25.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0057879910000000001 2024-09-11T03:39:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.072735+0000", 2024-09-11T03:39:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3414e-05 2024-09-11T03:39:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.072856+0000", 2024-09-11T03:39:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000121803 2024-09-11T03:39:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:744 2.0 2:0762776a:::smithi17557870-25:head [sparse-read 0~1885763,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e137)", 2024-09-11T03:39:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:13.831210+0000", 2024-09-11T03:39:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 12.040227063, 2024-09-11T03:39:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053191080000000003, 2024-09-11T03:39:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 744 2024-09-11T03:39:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.974 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831210+0000", 2024-09-11T03:39:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831210+0000", 2024-09-11T03:39:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831212+0000", 2024-09-11T03:39:25.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8279999999999999e-06 2024-09-11T03:39:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831220+0000", 2024-09-11T03:39:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4770000000000003e-06 2024-09-11T03:39:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831237+0000", 2024-09-11T03:39:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6648000000000001e-05 2024-09-11T03:39:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831249+0000", 2024-09-11T03:39:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1472e-05 2024-09-11T03:39:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831304+0000", 2024-09-11T03:39:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5954e-05 2024-09-11T03:39:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:39:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831309+0000", 2024-09-11T03:39:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5040000000000004e-06 2024-09-11T03:39:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.832366+0000", 2024-09-11T03:39:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001056876 2024-09-11T03:39:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.832464+0000", 2024-09-11T03:39:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7978000000000003e-05 2024-09-11T03:39:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.836529+0000", 2024-09-11T03:39:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040653709999999999 2024-09-11T03:39:25.978 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:745 2.0 2:0762776a:::smithi17557870-25:head [sparse-read 0~1885763] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e137)", 2024-09-11T03:39:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:13.831271+0000", 2024-09-11T03:39:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 12.040165921, 2024-09-11T03:39:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054786310000000003, 2024-09-11T03:39:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:25.979 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 745 2024-09-11T03:39:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831271+0000", 2024-09-11T03:39:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831271+0000", 2024-09-11T03:39:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831272+0000", 2024-09-11T03:39:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0800000000000005e-07 2024-09-11T03:39:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831274+0000", 2024-09-11T03:39:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.593e-06 2024-09-11T03:39:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831280+0000", 2024-09-11T03:39:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.587e-06 2024-09-11T03:39:25.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831286+0000", 2024-09-11T03:39:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5820000000000001e-06 2024-09-11T03:39:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831368+0000", 2024-09-11T03:39:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2207999999999996e-05 2024-09-11T03:39:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831374+0000", 2024-09-11T03:39:25.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.443e-06 2024-09-11T03:39:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.836570+0000", 2024-09-11T03:39:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051954840000000002 2024-09-11T03:39:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.836643+0000", 2024-09-11T03:39:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3596000000000005e-05 2024-09-11T03:39:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.836750+0000", 2024-09-11T03:39:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010663 2024-09-11T03:39:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:746 2.0 2:0762776a:::smithi17557870-25:head [sparse-read 0~1885763] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e137)", 2024-09-11T03:39:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:13.831294+0000", 2024-09-11T03:39:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 12.040142823, 2024-09-11T03:39:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0057346979999999999, 2024-09-11T03:39:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:39:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:39:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:39:25.984 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:39:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:39:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 746 2024-09-11T03:39:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:39:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:39:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831294+0000", 2024-09-11T03:39:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:39:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831294+0000", 2024-09-11T03:39:25.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:39:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:39:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831295+0000", 2024-09-11T03:39:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7899999999999999e-07 2024-09-11T03:39:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:39:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831297+0000", 2024-09-11T03:39:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2019999999999998e-06 2024-09-11T03:39:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:39:25.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831303+0000", 2024-09-11T03:39:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.139e-06 2024-09-11T03:39:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:39:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831312+0000", 2024-09-11T03:39:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1379999999999996e-06 2024-09-11T03:39:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831406+0000", 2024-09-11T03:39:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4178999999999994e-05 2024-09-11T03:39:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:25.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831411+0000", 2024-09-11T03:39:25.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7890000000000002e-06 2024-09-11T03:39:25.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:39:25.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.836793+0000", 2024-09-11T03:39:25.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053814850000000001 2024-09-11T03:39:25.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:39:25.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.836892+0000", 2024-09-11T03:39:25.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8987999999999995e-05 2024-09-11T03:39:25.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:39:25.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:39:25.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:39:25.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.837029+0000", 2024-09-11T03:39:25.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013729899999999999 2024-09-11T03:39:25.989 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.989 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.989 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.989 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:39:25.989 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:39:25.989 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:39:25.989 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:26.015 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:26.016 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:26.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:26.140 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:26.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:26.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:26.142 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:25 smithi084 ceph-mon[27254]: pgmap v376: 17 pgs: 17 active+clean; 113 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.6 MiB/s wr, 19 op/s 2024-09-11T03:39:26.142 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:25 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/546640128' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-11T03:39:26.142 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:25 smithi084 ceph-mon[27254]: osdmap e148: 12 total, 12 up, 11 in 2024-09-11T03:39:26.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:25 smithi121 ceph-mon[30729]: pgmap v376: 17 pgs: 17 active+clean; 113 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.6 MiB/s wr, 19 op/s 2024-09-11T03:39:26.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:25 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/546640128' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-11T03:39:26.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:25 smithi121 ceph-mon[30729]: osdmap e148: 12 total, 12 up, 11 in 2024-09-11T03:39:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:25 smithi175 ceph-mon[30894]: pgmap v376: 17 pgs: 17 active+clean; 113 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.6 MiB/s wr, 19 op/s 2024-09-11T03:39:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:25 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/546640128' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-11T03:39:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:25 smithi175 ceph-mon[30894]: osdmap e148: 12 total, 12 up, 11 in 2024-09-11T03:39:26.242 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:26.267 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:26.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:26.268 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:26.369 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:26.394 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:26.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:26.395 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:26.496 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:26.518 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:26.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:26.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:26.620 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:26.641 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:26.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:26.731 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:39:26.731 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:39:26.743 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:26.748 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:39:26.748 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:39:26.766 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:39:26.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:39:26.768 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:26.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:26.769 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:26.783 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:39:26.784 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:39:26.800 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:39:26.800 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:39:26.817 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:39:26.817 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:39:26.817 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:39:26.823 INFO:tasks.rados.rados.2.smithi175.stdout:314: done (1 left) 2024-09-11T03:39:26.823 INFO:tasks.rados.rados.2.smithi175.stdout:315: write (excl) oid 27 current snap is 25 2024-09-11T03:39:26.823 INFO:tasks.rados.rados.2.smithi175.stdout:312: finishing copy_from racing read to smithi17557870-20 2024-09-11T03:39:26.823 INFO:tasks.rados.rados.2.smithi175.stdout:315: seq_num 139 ranges {736173=553965,1867350=468823} 2024-09-11T03:39:26.830 INFO:tasks.rados.rados.2.smithi175.stdout:315: writing smithi17557870-27 from 736173 to 1290138 tid 1 2024-09-11T03:39:26.836 INFO:tasks.rados.rados.2.smithi175.stdout:315: writing smithi17557870-27 from 1867350 to 2336173 tid 2 2024-09-11T03:39:26.836 INFO:tasks.rados.rados.2.smithi175.stdout:316: write (excl) oid 15 current snap is 25 2024-09-11T03:39:26.836 INFO:tasks.rados.rados.2.smithi175.stdout:312: finishing copy_from to smithi17557870-20 2024-09-11T03:39:26.836 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 90 (ObjNum 120 snap 18 seq_num 120) dirty exists 2024-09-11T03:39:26.836 INFO:tasks.rados.rados.2.smithi175.stdout:315: finishing write tid 1 to smithi17557870-27 2024-09-11T03:39:26.836 INFO:tasks.rados.rados.2.smithi175.stdout:316: seq_num 140 ranges {752980=629214,1809479=543501} 2024-09-11T03:39:26.837 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:39:26.837 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:39:26.843 INFO:tasks.rados.rados.2.smithi175.stdout:316: writing smithi17557870-15 from 752980 to 1382194 tid 1 2024-09-11T03:39:26.849 INFO:tasks.rados.rados.2.smithi175.stdout:316: writing smithi17557870-15 from 1809479 to 2352980 tid 2 2024-09-11T03:39:26.849 INFO:tasks.rados.rados.2.smithi175.stdout:312: done (2 left) 2024-09-11T03:39:26.849 INFO:tasks.rados.rados.2.smithi175.stdout:317: snap_create 2024-09-11T03:39:26.849 INFO:tasks.rados.rados.2.smithi175.stdout:315: finishing write tid 2 to smithi17557870-27 2024-09-11T03:39:26.849 INFO:tasks.rados.rados.2.smithi175.stdout:315: finishing write tid 3 to smithi17557870-27 2024-09-11T03:39:26.849 INFO:tasks.rados.rados.2.smithi175.stdout:315: finishing write tid 4 to smithi17557870-27 2024-09-11T03:39:26.850 INFO:tasks.rados.rados.2.smithi175.stdout:315: oid 27 updating version 0 to 66 2024-09-11T03:39:26.850 INFO:tasks.rados.rados.2.smithi175.stdout:315: oid 27 updating version 66 to 68 2024-09-11T03:39:26.850 INFO:tasks.rados.rados.2.smithi175.stdout:315: oid 27 version 68 is already newer than 67 2024-09-11T03:39:26.850 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 68 (ObjNum 139 snap 25 seq_num 139) dirty exists 2024-09-11T03:39:26.850 INFO:tasks.rados.rados.2.smithi175.stdout:315: left oid 27 (ObjNum 139 snap 25 seq_num 139) 2024-09-11T03:39:26.853 INFO:tasks.rados.rados.2.smithi175.stdout:316: finishing write tid 1 to smithi17557870-15 2024-09-11T03:39:26.854 INFO:tasks.rados.rados.2.smithi175.stdout:316: finishing write tid 2 to smithi17557870-15 2024-09-11T03:39:26.854 INFO:tasks.rados.rados.2.smithi175.stdout:316: finishing write tid 3 to smithi17557870-15 2024-09-11T03:39:26.854 INFO:tasks.rados.rados.2.smithi175.stdout:316: finishing write tid 4 to smithi17557870-15 2024-09-11T03:39:26.854 INFO:tasks.rados.rados.2.smithi175.stdout:316: oid 15 updating version 0 to 67 2024-09-11T03:39:26.854 INFO:tasks.rados.rados.2.smithi175.stdout:316: oid 15 updating version 67 to 68 2024-09-11T03:39:26.854 INFO:tasks.rados.rados.2.smithi175.stdout:316: oid 15 version 68 is already newer than 66 2024-09-11T03:39:26.855 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 68 (ObjNum 140 snap 25 seq_num 140) dirty exists 2024-09-11T03:39:26.855 INFO:tasks.rados.rados.2.smithi175.stdout:316: left oid 15 (ObjNum 140 snap 25 seq_num 140) 2024-09-11T03:39:26.859 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:39:26.859 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:39:26.870 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:26.876 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:39:26.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:39:26.878 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-11T03:39:26.893 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:39:26.893 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:39:26.895 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:26.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:26.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:26.910 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:39:26.910 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:39:26.927 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:39:26.927 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:39:26.997 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:27.018 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:27.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:27.019 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:27.040 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:26 smithi084 ceph-mon[27254]: osdmap e149: 12 total, 12 up, 11 in 2024-09-11T03:39:27.041 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3804357517' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:27.041 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1598421031' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:27.041 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:26 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:27.041 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3804357517' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:27.041 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:26 smithi084 ceph-mon[27254]: osdmap e150: 12 total, 12 up, 11 in 2024-09-11T03:39:27.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:27.129 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:39:27.141 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:27.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:27.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:27.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:26 smithi121 ceph-mon[30729]: osdmap e149: 12 total, 12 up, 11 in 2024-09-11T03:39:27.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3804357517' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:27.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1598421031' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:26 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3804357517' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:26 smithi121 ceph-mon[30729]: osdmap e150: 12 total, 12 up, 11 in 2024-09-11T03:39:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:26 smithi175 ceph-mon[30894]: osdmap e149: 12 total, 12 up, 11 in 2024-09-11T03:39:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3804357517' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1598421031' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:26 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:27.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3804357517' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:27.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:26 smithi175 ceph-mon[30894]: osdmap e150: 12 total, 12 up, 11 in 2024-09-11T03:39:27.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:27.265 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:27.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:27.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:27.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:27.388 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:27.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:27.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:27.490 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:27.514 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:27.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:27.516 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:27.617 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:27.620 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:39:27.621 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:39:27.640 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:27.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:27.641 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:27.742 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:27.766 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:27.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:27.852 INFO:tasks.rados.rados.2.smithi175.stdout:315: done (2 left) 2024-09-11T03:39:27.852 INFO:tasks.rados.rados.2.smithi175.stdout:316: done (1 left) 2024-09-11T03:39:27.852 INFO:tasks.rados.rados.2.smithi175.stdout:317: done (0 left) 2024-09-11T03:39:27.852 INFO:tasks.rados.rados.2.smithi175.stdout:318: read oid 26 snap -1 2024-09-11T03:39:27.852 INFO:tasks.rados.rados.2.smithi175.stdout:318: expect (ObjNum 101 snap 16 seq_num 101) 2024-09-11T03:39:27.852 INFO:tasks.rados.rados.2.smithi175.stdout:319: rollback oid 41 current snap is 26 2024-09-11T03:39:27.852 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 41 to 25 2024-09-11T03:39:27.852 INFO:tasks.rados.rados.2.smithi175.stdout:320: snap_create 2024-09-11T03:39:27.854 INFO:tasks.rados.rados.2.smithi175.stdout:319: finishing rollback tid 0 to smithi17557870-41 2024-09-11T03:39:27.854 INFO:tasks.rados.rados.2.smithi175.stdout:319: finishing rollback tid 1 to smithi17557870-41 2024-09-11T03:39:27.854 INFO:tasks.rados.rados.2.smithi175.stdout:319: finishing rollback tid 2 to smithi17557870-41 2024-09-11T03:39:27.854 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 40 (ObjNum 133 snap 22 seq_num 133) dirty exists 2024-09-11T03:39:27.861 INFO:tasks.rados.rados.2.smithi175.stdout:318: expect (ObjNum 101 snap 16 seq_num 101) 2024-09-11T03:39:27.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:27.888 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:27.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:27.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:27.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:28.012 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:28.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:28.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:28.114 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:28.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:27 smithi084 ceph-mon[27254]: pgmap v379: 17 pgs: 17 active+clean; 113 MiB data, 521 MiB used, 983 GiB / 983 GiB avail; 5.2 MiB/s rd, 242 KiB/s wr, 6 op/s; 129 KiB/s, 2 objects/s recovering 2024-09-11T03:39:28.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:27 smithi121 ceph-mon[30729]: pgmap v379: 17 pgs: 17 active+clean; 113 MiB data, 521 MiB used, 983 GiB / 983 GiB avail; 5.2 MiB/s rd, 242 KiB/s wr, 6 op/s; 129 KiB/s, 2 objects/s recovering 2024-09-11T03:39:28.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:28.177 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:28.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:27 smithi175 ceph-mon[30894]: pgmap v379: 17 pgs: 17 active+clean; 113 MiB data, 521 MiB used, 983 GiB / 983 GiB avail; 5.2 MiB/s rd, 242 KiB/s wr, 6 op/s; 129 KiB/s, 2 objects/s recovering 2024-09-11T03:39:28.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:28.300 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:28.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:28.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:28.386 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-11T03:39:28.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:28.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:28.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:28.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:28.549 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:28.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:28.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:28.638 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:39:28.651 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:28.674 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:28.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:28.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:28.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:28.797 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:28.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:28.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:28.866 INFO:tasks.rados.rados.2.smithi175.stdout:318: done (2 left) 2024-09-11T03:39:28.866 INFO:tasks.rados.rados.2.smithi175.stdout:319: done (1 left) 2024-09-11T03:39:28.866 INFO:tasks.rados.rados.2.smithi175.stdout:320: done (0 left) 2024-09-11T03:39:28.866 INFO:tasks.rados.rados.2.smithi175.stdout:321: snap_create 2024-09-11T03:39:28.899 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:28.924 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:28.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:28.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:29.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:29.050 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:29.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:29.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:29.078 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:28 smithi175 ceph-mon[30894]: osdmap e151: 12 total, 12 up, 11 in 2024-09-11T03:39:29.093 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:39:29.093 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:39:29.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:28 smithi084 ceph-mon[27254]: osdmap e151: 12 total, 12 up, 11 in 2024-09-11T03:39:29.152 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:29.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:28 smithi121 ceph-mon[30729]: osdmap e151: 12 total, 12 up, 11 in 2024-09-11T03:39:29.175 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:29.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:29.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:29.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:29.300 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:29.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:29.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:29.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:29.427 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:29.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:29.529 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:29.555 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:29.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:29.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:29.628 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:39:29.657 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:29.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:29.682 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:29.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:29.750 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-11T03:39:29.783 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:29.815 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:29.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:29.816 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:29.864 INFO:tasks.rados.rados.2.smithi175.stdout:321: done (0 left) 2024-09-11T03:39:29.864 INFO:tasks.rados.rados.2.smithi175.stdout:322: snap_create 2024-09-11T03:39:29.883 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:29.916 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:29.938 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:29.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:29.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:29.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:29 smithi175 ceph-mon[30894]: pgmap v382: 17 pgs: 17 active+clean; 111 MiB data, 517 MiB used, 983 GiB / 983 GiB avail; 5.9 MiB/s rd, 742 KiB/s wr, 5 op/s; 1.1 MiB/s, 5 objects/s recovering 2024-09-11T03:39:29.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:29 smithi175 ceph-mon[30894]: osdmap e152: 12 total, 12 up, 11 in 2024-09-11T03:39:29.996 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:39:30.003 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:29 smithi084 ceph-mon[27254]: pgmap v382: 17 pgs: 17 active+clean; 111 MiB data, 517 MiB used, 983 GiB / 983 GiB avail; 5.9 MiB/s rd, 742 KiB/s wr, 5 op/s; 1.1 MiB/s, 5 objects/s recovering 2024-09-11T03:39:30.003 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:29 smithi084 ceph-mon[27254]: osdmap e152: 12 total, 12 up, 11 in 2024-09-11T03:39:30.040 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:30.061 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:30.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:30.062 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:30.117 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:39:30.163 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:30.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:29 smithi121 ceph-mon[30729]: pgmap v382: 17 pgs: 17 active+clean; 111 MiB data, 517 MiB used, 983 GiB / 983 GiB avail; 5.9 MiB/s rd, 742 KiB/s wr, 5 op/s; 1.1 MiB/s, 5 objects/s recovering 2024-09-11T03:39:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:29 smithi121 ceph-mon[30729]: osdmap e152: 12 total, 12 up, 11 in 2024-09-11T03:39:30.184 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:30.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:30.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:30.286 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:30.312 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:30.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:30.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:30.375 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:39:30.376 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:39:30.394 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:39:30.394 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:39:30.413 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:30.415 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:39:30.415 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:39:30.432 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:39:30.432 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:39:30.438 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:30.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:30.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:30.451 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:30.455 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:39:30.456 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:39:30.474 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:39:30.475 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:39:30.491 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:39:30.492 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:39:30.499 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:39:30.500 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:39:30.508 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:39:30.508 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:39:30.525 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:39:30.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:39:30.540 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:30.543 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:39:30.543 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:39:30.559 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:39:30.559 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:39:30.561 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:30.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:30.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:30.576 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:39:30.576 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:39:30.610 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:39:30.610 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:39:30.610 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-11T03:39:30.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd test-reweight-by-utilization 110 0.05 2024-09-11T03:39:30.663 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:30.689 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:30.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:30.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:30.791 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:30.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:30.813 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:30.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:30.877 INFO:tasks.rados.rados.2.smithi175.stdout:322: done (0 left) 2024-09-11T03:39:30.877 INFO:tasks.rados.rados.2.smithi175.stdout:323: rollback oid 11 current snap is 29 2024-09-11T03:39:30.877 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 11 to 26 2024-09-11T03:39:30.877 INFO:tasks.rados.rados.2.smithi175.stdout:324: read oid 26 snap -1 2024-09-11T03:39:30.877 INFO:tasks.rados.rados.2.smithi175.stdout:324: expect (ObjNum 101 snap 16 seq_num 101) 2024-09-11T03:39:30.877 INFO:tasks.rados.rados.2.smithi175.stdout:325: snap_remove snap 25 2024-09-11T03:39:30.914 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:30.935 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:30.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:30.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:31.037 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:31.059 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:31.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:31.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:31.070 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:30 smithi175 ceph-mon[30894]: osdmap e153: 12 total, 12 up, 11 in 2024-09-11T03:39:31.070 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:30 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2804343485' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:31.097 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-11T03:39:31.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:30 smithi084 ceph-mon[27254]: osdmap e153: 12 total, 12 up, 11 in 2024-09-11T03:39:31.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:30 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2804343485' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:31.161 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:31.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:30 smithi121 ceph-mon[30729]: osdmap e153: 12 total, 12 up, 11 in 2024-09-11T03:39:31.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:30 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2804343485' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:31.187 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:31.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:31.289 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:31.314 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:31.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:31.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:31.336 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:39:31.416 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:31.441 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:31.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:31.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:31.543 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:31.565 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:31.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:31.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:31.628 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:31.667 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:31.692 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:31.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:31.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:31.773 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:39:31.773 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:39:31.773 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:39:31.773 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:39:31.794 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:31.816 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:31.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:31.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:31.819 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:39:31.824 INFO:tasks.rados.rados.2.smithi175.stdout:325: done (2 left) 2024-09-11T03:39:31.824 INFO:tasks.rados.rados.2.smithi175.stdout:326: rollback oid 47 current snap is 29 2024-09-11T03:39:31.824 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 47 to 26 2024-09-11T03:39:31.824 INFO:tasks.rados.rados.2.smithi175.stdout:323: finishing rollback tid 0 to smithi17557870-11 2024-09-11T03:39:31.824 INFO:tasks.rados.rados.2.smithi175.stdout:323: finishing rollback tid 1 to smithi17557870-11 2024-09-11T03:39:31.824 INFO:tasks.rados.rados.2.smithi175.stdout:323: finishing rollback tid 2 to smithi17557870-11 2024-09-11T03:39:31.824 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 42 (ObjNum 123 snap 19 seq_num 123) dirty exists 2024-09-11T03:39:31.824 INFO:tasks.rados.rados.2.smithi175.stdout:324: expect (ObjNum 101 snap 16 seq_num 101) 2024-09-11T03:39:31.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:31.960 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:31.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:31.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:31.987 INFO:tasks.rados.rados.2.smithi175.stdout:326: finishing rollback tid 1 to smithi17557870-47 2024-09-11T03:39:31.987 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 0 (ObjNum 225 snap 0 seq_num 1745123968) dirty dne 2024-09-11T03:39:31.987 INFO:tasks.rados.rados.2.smithi175.stdout:323: done (2 left) 2024-09-11T03:39:31.988 INFO:tasks.rados.rados.2.smithi175.stdout:324: done (1 left) 2024-09-11T03:39:31.988 INFO:tasks.rados.rados.2.smithi175.stdout:326: done (0 left) 2024-09-11T03:39:31.988 INFO:tasks.rados.rados.2.smithi175.stdout:327: copy_from oid 30 from oid 49 current snap is 29 2024-09-11T03:39:31.988 INFO:tasks.rados.rados.2.smithi175.stdout:328: read oid 1 snap 27 2024-09-11T03:39:31.988 INFO:tasks.rados.rados.2.smithi175.stdout:328: expect (ObjNum 120 snap 18 seq_num 120) 2024-09-11T03:39:31.988 INFO:tasks.rados.rados.2.smithi175.stdout:329: snap_create 2024-09-11T03:39:31.989 INFO:tasks.rados.rados.2.smithi175.stdout:328: expect (ObjNum 120 snap 18 seq_num 120) 2024-09-11T03:39:32.062 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:32.074 INFO:tasks.rados.rados.2.smithi175.stdout:327: finishing copy_from racing read to smithi17557870-30 2024-09-11T03:39:32.074 INFO:tasks.rados.rados.2.smithi175.stdout:327: finishing copy_from to smithi17557870-30 2024-09-11T03:39:32.074 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-11T03:39:32.083 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:32.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:32.084 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:32.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:31 smithi084 ceph-mon[27254]: pgmap v385: 17 pgs: 17 active+clean; 111 MiB data, 521 MiB used, 983 GiB / 983 GiB avail; 3.0 MiB/s rd, 742 KiB/s wr, 3 op/s; 993 KiB/s, 2 objects/s recovering 2024-09-11T03:39:32.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:31 smithi084 ceph-mon[27254]: osdmap e154: 12 total, 12 up, 11 in 2024-09-11T03:39:32.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:31 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/115233763' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:39:32.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:31 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:32.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:31 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/115233763' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:39:32.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:31 smithi084 ceph-mon[27254]: osdmap e155: 12 total, 12 up, 11 in 2024-09-11T03:39:32.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:31 smithi121 ceph-mon[30729]: pgmap v385: 17 pgs: 17 active+clean; 111 MiB data, 521 MiB used, 983 GiB / 983 GiB avail; 3.0 MiB/s rd, 742 KiB/s wr, 3 op/s; 993 KiB/s, 2 objects/s recovering 2024-09-11T03:39:32.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:31 smithi121 ceph-mon[30729]: osdmap e154: 12 total, 12 up, 11 in 2024-09-11T03:39:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:31 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/115233763' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:39:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:31 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:31 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/115233763' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:39:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:31 smithi121 ceph-mon[30729]: osdmap e155: 12 total, 12 up, 11 in 2024-09-11T03:39:32.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:32.211 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:32.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:31 smithi175 ceph-mon[30894]: pgmap v385: 17 pgs: 17 active+clean; 111 MiB data, 521 MiB used, 983 GiB / 983 GiB avail; 3.0 MiB/s rd, 742 KiB/s wr, 3 op/s; 993 KiB/s, 2 objects/s recovering 2024-09-11T03:39:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:31 smithi175 ceph-mon[30894]: osdmap e154: 12 total, 12 up, 11 in 2024-09-11T03:39:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:31 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/115233763' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:39:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:31 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:31 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/115233763' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:39:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:31 smithi175 ceph-mon[30894]: osdmap e155: 12 total, 12 up, 11 in 2024-09-11T03:39:32.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:32.338 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:32.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:32.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:32.362 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-11T03:39:32.440 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:32.466 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:32.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:32.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:32.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:32.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:32.594 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:32.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:32.603 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:39:32.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:32.720 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:32.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:32.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:32.822 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:32.847 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:32.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:32.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:32.881 INFO:tasks.rados.rados.2.smithi175.stdout:327: done (2 left) 2024-09-11T03:39:32.881 INFO:tasks.rados.rados.2.smithi175.stdout:328: done (1 left) 2024-09-11T03:39:32.881 INFO:tasks.rados.rados.2.smithi175.stdout:329: done (0 left) 2024-09-11T03:39:32.881 INFO:tasks.rados.rados.2.smithi175.stdout:330: rollback oid 32 current snap is 30 2024-09-11T03:39:32.881 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 32 to 26 2024-09-11T03:39:32.881 INFO:tasks.rados.rados.2.smithi175.stdout:331: write oid 4 current snap is 30 2024-09-11T03:39:32.881 INFO:tasks.rados.rados.2.smithi175.stdout:331: seq_num 141 ranges {769787=704463,2267961=101826} 2024-09-11T03:39:32.888 INFO:tasks.rados.rados.2.smithi175.stdout:331: writing smithi17557870-4 from 769787 to 1474250 tid 1 2024-09-11T03:39:32.894 INFO:tasks.rados.rados.2.smithi175.stdout:331: writing smithi17557870-4 from 2267961 to 2369787 tid 2 2024-09-11T03:39:32.894 INFO:tasks.rados.rados.2.smithi175.stdout:332: delete oid 33 current snap is 30 2024-09-11T03:39:32.894 INFO:tasks.rados.rados.2.smithi175.stdout:330: finishing rollback tid 0 to smithi17557870-32 2024-09-11T03:39:32.894 INFO:tasks.rados.rados.2.smithi175.stdout:330: finishing rollback tid 1 to smithi17557870-32 2024-09-11T03:39:32.894 INFO:tasks.rados.rados.2.smithi175.stdout:330: finishing rollback tid 2 to smithi17557870-32 2024-09-11T03:39:32.894 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 46 (ObjNum 114 snap 18 seq_num 114) dirty exists 2024-09-11T03:39:32.894 INFO:tasks.rados.rados.2.smithi175.stdout:331: finishing write tid 1 to smithi17557870-4 2024-09-11T03:39:32.895 INFO:tasks.rados.rados.2.smithi175.stdout:331: finishing write tid 2 to smithi17557870-4 2024-09-11T03:39:32.895 INFO:tasks.rados.rados.2.smithi175.stdout:331: finishing write tid 3 to smithi17557870-4 2024-09-11T03:39:32.895 INFO:tasks.rados.rados.2.smithi175.stdout:331: finishing write tid 4 to smithi17557870-4 2024-09-11T03:39:32.895 INFO:tasks.rados.rados.2.smithi175.stdout:331: oid 4 updating version 0 to 50 2024-09-11T03:39:32.896 INFO:tasks.rados.rados.2.smithi175.stdout:331: oid 4 version 50 is already newer than 49 2024-09-11T03:39:32.896 INFO:tasks.rados.rados.2.smithi175.stdout:331: oid 4 updating version 50 to 51 2024-09-11T03:39:32.896 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 51 (ObjNum 141 snap 30 seq_num 141) dirty exists 2024-09-11T03:39:32.896 INFO:tasks.rados.rados.2.smithi175.stdout:331: left oid 4 (ObjNum 141 snap 30 seq_num 141) 2024-09-11T03:39:32.897 INFO:tasks.rados.rados.2.smithi175.stdout:330: done (2 left) 2024-09-11T03:39:32.897 INFO:tasks.rados.rados.2.smithi175.stdout:331: done (1 left) 2024-09-11T03:39:32.897 INFO:tasks.rados.rados.2.smithi175.stdout:332: done (0 left) 2024-09-11T03:39:32.897 INFO:tasks.rados.rados.2.smithi175.stdout:333: read oid 40 snap 27 2024-09-11T03:39:32.897 INFO:tasks.rados.rados.2.smithi175.stdout:333: expect deleted 2024-09-11T03:39:32.897 INFO:tasks.rados.rados.2.smithi175.stdout:334: snap_create 2024-09-11T03:39:32.911 INFO:teuthology.orchestra.run.smithi084.stdout:moved 1 / 51 (1.96078%) 2024-09-11T03:39:32.911 INFO:teuthology.orchestra.run.smithi084.stdout:avg 4.63636 2024-09-11T03:39:32.911 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 1.72008 -> 1.66639 (expected baseline 2.05302) 2024-09-11T03:39:32.911 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.5 with 2 -> 2 pgs (0.431373 -> 0.431373 * mean) 2024-09-11T03:39:32.911 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.9 with 8 -> 7 pgs (1.72549 -> 1.5098 * mean) 2024-09-11T03:39:32.911 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:39:32.912 INFO:teuthology.orchestra.run.smithi084.stdout:oload 110 2024-09-11T03:39:32.912 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 0.05 2024-09-11T03:39:32.912 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2024-09-11T03:39:32.912 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0005 2024-09-11T03:39:32.912 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0006 2024-09-11T03:39:32.912 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8 weight 1.0000 -> 0.9500 2024-09-11T03:39:32.912 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9 weight 1.0000 -> 0.9500 2024-09-11T03:39:32.912 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3 weight 1.0000 -> 0.9500 2024-09-11T03:39:32.912 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0 weight 1.0000 -> 0.9500 2024-09-11T03:39:32.912 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2024-09-11T03:39:32.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:32.975 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:32.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:32.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:33.077 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:33.099 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:33.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:33.100 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:33.103 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:39:33.103 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:39:33.103 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:39:33.103 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:39:33.104 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:39:33.144 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:39:32 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: 2024-09-11T03:39:32.911+0000 7fab40ce9640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-09-11T03:39:33.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:33.227 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:33.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:33.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:33.329 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:33.351 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:33.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:33.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:33.453 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:33.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:33.476 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:33.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:33.536 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-11T03:39:33.536 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-09-11T03:39:33.577 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:39:33.577 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:33.603 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:33.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:33.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:33.705 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:33.726 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:33.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:33.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:33.792 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:33.828 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:33.835 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-11T03:39:33.853 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:33.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:33.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:33.890 INFO:tasks.rados.rados.2.smithi175.stdout:333: done (1 left) 2024-09-11T03:39:33.890 INFO:tasks.rados.rados.2.smithi175.stdout:334: done (0 left) 2024-09-11T03:39:33.890 INFO:tasks.rados.rados.2.smithi175.stdout:335: read oid 49 snap -1 2024-09-11T03:39:33.890 INFO:tasks.rados.rados.2.smithi175.stdout:335: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-11T03:39:33.890 INFO:tasks.rados.rados.2.smithi175.stdout:336: delete oid 21 current snap is 31 2024-09-11T03:39:33.892 INFO:tasks.rados.rados.2.smithi175.stdout:336: done (1 left) 2024-09-11T03:39:33.893 INFO:tasks.rados.rados.2.smithi175.stdout:337: rollback oid 2 current snap is 31 2024-09-11T03:39:33.893 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 2 to 30 2024-09-11T03:39:33.893 INFO:tasks.rados.rados.2.smithi175.stdout:338: write (excl) oid 36 current snap is 31 2024-09-11T03:39:33.893 INFO:tasks.rados.rados.2.smithi175.stdout:338: seq_num 144 ranges {420208=530210,1527054=766050} 2024-09-11T03:39:33.898 INFO:tasks.rados.rados.2.smithi175.stdout:338: writing smithi17557870-36 from 420208 to 950418 tid 1 2024-09-11T03:39:33.906 INFO:tasks.rados.rados.2.smithi175.stdout:338: writing smithi17557870-36 from 1527054 to 2293104 tid 2 2024-09-11T03:39:33.906 INFO:tasks.rados.rados.2.smithi175.stdout:339: snap_remove snap 27 2024-09-11T03:39:33.955 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:33.976 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:33.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:33.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:34.072 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:34.078 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:34.083 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:39:34.101 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:34.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:34.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:34.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:33 smithi084 ceph-mon[27254]: pgmap v388: 17 pgs: 17 active+clean; 111 MiB data, 526 MiB used, 983 GiB / 983 GiB avail; 4.6 MiB/s rd, 286 KiB/s wr, 6 op/s 2024-09-11T03:39:34.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:33 smithi084 ceph-mon[27254]: osdmap e156: 12 total, 12 up, 11 in 2024-09-11T03:39:34.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:33 smithi084 ceph-mon[27254]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:34.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:33 smithi084 ceph-mon[27254]: 2.3 scrub starts 2024-09-11T03:39:34.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:33 smithi084 ceph-mon[27254]: 2.3 scrub ok 2024-09-11T03:39:34.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:33 smithi121 ceph-mon[30729]: pgmap v388: 17 pgs: 17 active+clean; 111 MiB data, 526 MiB used, 983 GiB / 983 GiB avail; 4.6 MiB/s rd, 286 KiB/s wr, 6 op/s 2024-09-11T03:39:34.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:33 smithi121 ceph-mon[30729]: osdmap e156: 12 total, 12 up, 11 in 2024-09-11T03:39:34.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:33 smithi121 ceph-mon[30729]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:34.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:33 smithi121 ceph-mon[30729]: 2.3 scrub starts 2024-09-11T03:39:34.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:33 smithi121 ceph-mon[30729]: 2.3 scrub ok 2024-09-11T03:39:34.193 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:33 smithi175 ceph-mon[30894]: pgmap v388: 17 pgs: 17 active+clean; 111 MiB data, 526 MiB used, 983 GiB / 983 GiB avail; 4.6 MiB/s rd, 286 KiB/s wr, 6 op/s 2024-09-11T03:39:34.193 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:33 smithi175 ceph-mon[30894]: osdmap e156: 12 total, 12 up, 11 in 2024-09-11T03:39:34.193 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:33 smithi175 ceph-mon[30894]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:34.193 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:33 smithi175 ceph-mon[30894]: 2.3 scrub starts 2024-09-11T03:39:34.193 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:33 smithi175 ceph-mon[30894]: 2.3 scrub ok 2024-09-11T03:39:34.202 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:34.228 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:34.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:34.330 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:34.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:34.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:34.453 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:34.473 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:39:34.475 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:34.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:34.476 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:34.505 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:39:34.505 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:39:34.505 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:39:34.505 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:39:34.505 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.505 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:216 2.d 2:b6a25a79:::smithi17557870-47:head [write 457157~535817,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-11T03:39:34.505 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:38.938703+0000", 2024-09-11T03:39:34.506 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 55.565008712999997, 2024-09-11T03:39:34.506 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050651000000000003, 2024-09-11T03:39:34.506 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.506 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.506 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:34.506 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:34.506 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:34.506 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:34.506 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 216 2024-09-11T03:39:34.506 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.506 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.506 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.507 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.507 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:38.938703+0000", 2024-09-11T03:39:34.507 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.507 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.507 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.507 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.507 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:38.938703+0000", 2024-09-11T03:39:34.507 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.507 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.507 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:38.938707+0000", 2024-09-11T03:39:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.805e-06 2024-09-11T03:39:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:38.939123+0000", 2024-09-11T03:39:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00041593400000000002 2024-09-11T03:39:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:38.939153+0000", 2024-09-11T03:39:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9689999999999999e-05 2024-09-11T03:39:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:38.939166+0000", 2024-09-11T03:39:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3461e-05 2024-09-11T03:39:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:38.939246+0000", 2024-09-11T03:39:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0000000000000007e-05 2024-09-11T03:39:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:39:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:38.939255+0000", 2024-09-11T03:39:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1290000000000002e-06 2024-09-11T03:39:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:38.940920+0000", 2024-09-11T03:39:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001664207 2024-09-11T03:39:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:38.941035+0000", 2024-09-11T03:39:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000115059 2024-09-11T03:39:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:39:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:38.941215+0000", 2024-09-11T03:39:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00018079500000000001 2024-09-11T03:39:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:38.943082+0000", 2024-09-11T03:39:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00186696 2024-09-11T03:39:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:38.943382+0000", 2024-09-11T03:39:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00030010200000000001 2024-09-11T03:39:34.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.512 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.512 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:34.512 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:38.943696+0000", 2024-09-11T03:39:34.512 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000313405 2024-09-11T03:39:34.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.512 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.512 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:34.512 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:38.943738+0000", 2024-09-11T03:39:34.512 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.1739000000000001e-05 2024-09-11T03:39:34.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.513 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.513 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.513 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:38.943768+0000", 2024-09-11T03:39:34.513 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0814e-05 2024-09-11T03:39:34.513 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.513 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.513 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.513 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.513 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.513 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:243 2.d 2:b7bdfe52:::smithi17557870-10:head [write 507578~477917] snapc 5=[4] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T03:39:34.513 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:43.971811+0000", 2024-09-11T03:39:34.513 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 50.531900747999998, 2024-09-11T03:39:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.006812818, 2024-09-11T03:39:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 243 2024-09-11T03:39:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.971811+0000", 2024-09-11T03:39:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.515 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.515 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.515 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.515 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.971811+0000", 2024-09-11T03:39:34.515 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.515 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.515 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.515 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.515 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.971813+0000", 2024-09-11T03:39:34.515 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0829999999999998e-06 2024-09-11T03:39:34.515 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.515 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.515 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.972267+0000", 2024-09-11T03:39:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00045407599999999998 2024-09-11T03:39:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.972291+0000", 2024-09-11T03:39:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3218e-05 2024-09-11T03:39:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.972304+0000", 2024-09-11T03:39:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3397999999999999e-05 2024-09-11T03:39:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.972388+0000", 2024-09-11T03:39:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3891e-05 2024-09-11T03:39:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:39:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.972396+0000", 2024-09-11T03:39:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5700000000000004e-06 2024-09-11T03:39:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.518 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.518 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.974233+0000", 2024-09-11T03:39:34.518 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001837064 2024-09-11T03:39:34.518 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.518 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.518 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.518 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.974342+0000", 2024-09-11T03:39:34.518 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010918599999999999 2024-09-11T03:39:34.518 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.518 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.518 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:39:34.518 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.974547+0000", 2024-09-11T03:39:34.518 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00020537499999999999 2024-09-11T03:39:34.518 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.519 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.519 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:34.519 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.976167+0000", 2024-09-11T03:39:34.519 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001620039 2024-09-11T03:39:34.519 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.519 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.519 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:34.519 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.977310+0000", 2024-09-11T03:39:34.519 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001143301 2024-09-11T03:39:34.519 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.519 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.519 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:34.519 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.978507+0000", 2024-09-11T03:39:34.520 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0011963220000000001 2024-09-11T03:39:34.520 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.520 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.520 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:34.520 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.978570+0000", 2024-09-11T03:39:34.520 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3480999999999997e-05 2024-09-11T03:39:34.520 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.520 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.520 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.521 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.978624+0000", 2024-09-11T03:39:34.521 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3813999999999997e-05 2024-09-11T03:39:34.521 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.521 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.521 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.521 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.521 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.521 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:244 2.d 2:b7bdfe52:::smithi17557870-10:head [write 1543557~550004] snapc 5=[4] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T03:39:34.522 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:43.978181+0000", 2024-09-11T03:39:34.522 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 50.525531358000002, 2024-09-11T03:39:34.522 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030419039999999998, 2024-09-11T03:39:34.522 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.522 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.522 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:34.522 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:34.522 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:34.522 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:34.522 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 244 2024-09-11T03:39:34.522 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.522 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.522 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.522 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.523 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.978181+0000", 2024-09-11T03:39:34.523 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.523 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.523 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.523 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.523 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.978181+0000", 2024-09-11T03:39:34.523 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.523 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.523 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.523 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.523 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.978183+0000", 2024-09-11T03:39:34.523 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4490000000000002e-06 2024-09-11T03:39:34.523 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.523 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.524 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.524 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.978561+0000", 2024-09-11T03:39:34.524 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000378284 2024-09-11T03:39:34.524 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.524 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.524 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.524 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.978574+0000", 2024-09-11T03:39:34.524 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2965999999999999e-05 2024-09-11T03:39:34.524 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.524 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.524 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.524 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.978582+0000", 2024-09-11T03:39:34.524 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0020000000000006e-06 2024-09-11T03:39:34.525 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.525 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.525 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.525 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.978654+0000", 2024-09-11T03:39:34.525 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1798000000000006e-05 2024-09-11T03:39:34.525 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.525 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.525 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.525 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.978726+0000", 2024-09-11T03:39:34.525 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2041999999999999e-05 2024-09-11T03:39:34.525 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.525 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.525 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:39:34.526 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.978829+0000", 2024-09-11T03:39:34.526 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000102983 2024-09-11T03:39:34.526 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.526 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.526 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:34.526 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.979830+0000", 2024-09-11T03:39:34.526 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00100044 2024-09-11T03:39:34.526 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.526 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.526 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:34.526 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.980827+0000", 2024-09-11T03:39:34.526 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00099726299999999997 2024-09-11T03:39:34.526 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.526 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.526 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:34.527 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.981115+0000", 2024-09-11T03:39:34.527 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00028811599999999999 2024-09-11T03:39:34.527 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.527 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.527 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:34.527 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.981177+0000", 2024-09-11T03:39:34.527 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1662999999999996e-05 2024-09-11T03:39:34.527 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.527 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.527 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.527 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.981223+0000", 2024-09-11T03:39:34.528 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5898e-05 2024-09-11T03:39:34.528 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.528 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.528 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.528 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.528 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.528 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:345 2.0 e113/95 2:0762776a:::smithi17557870-25:head v 113'5, mlcod=100'2)", 2024-09-11T03:39:34.528 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:49.830511+0000", 2024-09-11T03:39:34.528 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 44.673201126000002, 2024-09-11T03:39:34.528 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035877550000000002, 2024-09-11T03:39:34.528 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.528 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.528 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:34.529 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.529 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.529 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.529 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.830511+0000", 2024-09-11T03:39:34.529 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.529 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.529 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.529 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.529 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.830511+0000", 2024-09-11T03:39:34.529 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.529 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.529 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.529 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.530 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.830511+0000", 2024-09-11T03:39:34.530 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6199999999999999e-07 2024-09-11T03:39:34.530 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.530 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.530 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.530 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.833180+0000", 2024-09-11T03:39:34.530 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0026691380000000002 2024-09-11T03:39:34.530 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.530 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.530 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.530 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.833191+0000", 2024-09-11T03:39:34.530 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0631999999999999e-05 2024-09-11T03:39:34.530 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.530 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.531 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.531 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.833197+0000", 2024-09-11T03:39:34.531 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8989999999999998e-06 2024-09-11T03:39:34.531 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.531 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.531 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.531 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.833242+0000", 2024-09-11T03:39:34.531 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5016999999999999e-05 2024-09-11T03:39:34.531 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.531 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.531 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.531 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.833256+0000", 2024-09-11T03:39:34.531 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.42e-05 2024-09-11T03:39:34.531 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.532 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.532 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:34.532 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.834069+0000", 2024-09-11T03:39:34.532 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00081280000000000002 2024-09-11T03:39:34.532 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.532 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.532 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.532 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.834099+0000", 2024-09-11T03:39:34.532 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9907e-05 2024-09-11T03:39:34.532 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.532 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.532 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.532 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.533 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.533 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:346 2.0 e113/95 2:0762776a:::smithi17557870-25:head v 113'6, mlcod=100'3)", 2024-09-11T03:39:34.533 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:49.839509+0000", 2024-09-11T03:39:34.533 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 44.664203260000001, 2024-09-11T03:39:34.533 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0026136000000000002, 2024-09-11T03:39:34.533 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.533 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.533 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:34.533 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.533 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.533 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.533 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.839509+0000", 2024-09-11T03:39:34.533 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.533 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.534 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.534 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.534 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.839509+0000", 2024-09-11T03:39:34.534 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.534 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.534 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.534 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.534 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.839510+0000", 2024-09-11T03:39:34.534 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9099999999999991e-07 2024-09-11T03:39:34.534 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.534 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.534 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.534 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.841196+0000", 2024-09-11T03:39:34.534 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001686016 2024-09-11T03:39:34.535 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.535 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.535 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.535 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.841206+0000", 2024-09-11T03:39:34.535 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0515e-05 2024-09-11T03:39:34.535 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.535 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.535 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.535 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.841212+0000", 2024-09-11T03:39:34.535 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9279999999999998e-06 2024-09-11T03:39:34.535 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.535 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.535 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.535 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.841261+0000", 2024-09-11T03:39:34.536 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8495999999999998e-05 2024-09-11T03:39:34.536 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.536 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.536 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.536 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.841281+0000", 2024-09-11T03:39:34.536 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0183999999999999e-05 2024-09-11T03:39:34.536 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.536 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.536 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:34.536 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.842080+0000", 2024-09-11T03:39:34.536 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00079953200000000004 2024-09-11T03:39:34.536 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.537 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.537 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.537 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:49.842122+0000", 2024-09-11T03:39:34.537 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.1937999999999997e-05 2024-09-11T03:39:34.537 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.537 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.537 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.537 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.537 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.537 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:357 2.e e114/95 2:750de3b7:::smithi17557870-8:head v 114'29, mlcod=109'26)", 2024-09-11T03:39:34.537 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.836763+0000", 2024-09-11T03:39:34.537 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 43.666949150000001, 2024-09-11T03:39:34.538 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0051249390000000002, 2024-09-11T03:39:34.538 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.538 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.538 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:34.538 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.538 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.538 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.538 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.836763+0000", 2024-09-11T03:39:34.538 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.538 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.538 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.538 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.539 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.836763+0000", 2024-09-11T03:39:34.539 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.539 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.539 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.539 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.539 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.836763+0000", 2024-09-11T03:39:34.539 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6300000000000001e-07 2024-09-11T03:39:34.539 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.539 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.539 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.539 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.839610+0000", 2024-09-11T03:39:34.539 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0028472900000000002 2024-09-11T03:39:34.539 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.540 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.540 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.540 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.839625+0000", 2024-09-11T03:39:34.540 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4793e-05 2024-09-11T03:39:34.540 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.540 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.540 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.540 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.839631+0000", 2024-09-11T03:39:34.540 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2040000000000002e-06 2024-09-11T03:39:34.541 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.541 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.541 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.541 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.839691+0000", 2024-09-11T03:39:34.541 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.995e-05 2024-09-11T03:39:34.541 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.541 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.541 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.541 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.839718+0000", 2024-09-11T03:39:34.541 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6491000000000001e-05 2024-09-11T03:39:34.541 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.542 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.542 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:34.542 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.841843+0000", 2024-09-11T03:39:34.542 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0021255050000000002 2024-09-11T03:39:34.542 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.542 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.542 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.542 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.841888+0000", 2024-09-11T03:39:34.542 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4443000000000001e-05 2024-09-11T03:39:34.542 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.542 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.542 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.542 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.542 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.543 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:561 2.d 2:b6a25a79:::smithi17557870-47:head [append 0~0] snapc 11=[11,10,f,e] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-11T03:39:34.543 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:04.835023+0000", 2024-09-11T03:39:34.543 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 29.6686893, 2024-09-11T03:39:34.543 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0033838039999999998, 2024-09-11T03:39:34.543 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.543 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.543 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:34.543 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:34.543 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:34.543 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:34.543 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 561 2024-09-11T03:39:34.543 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.543 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.544 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.544 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.544 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835023+0000", 2024-09-11T03:39:34.544 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.544 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.544 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.544 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.544 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835023+0000", 2024-09-11T03:39:34.544 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.544 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.544 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.545 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.545 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835025+0000", 2024-09-11T03:39:34.545 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2759999999999999e-06 2024-09-11T03:39:34.545 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.545 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.545 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.545 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835036+0000", 2024-09-11T03:39:34.545 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0852999999999999e-05 2024-09-11T03:39:34.545 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.545 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.545 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.545 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835057+0000", 2024-09-11T03:39:34.545 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1316999999999999e-05 2024-09-11T03:39:34.546 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.546 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.546 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.546 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835068+0000", 2024-09-11T03:39:34.546 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0352e-05 2024-09-11T03:39:34.546 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.546 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.546 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.546 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835136+0000", 2024-09-11T03:39:34.546 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8398000000000005e-05 2024-09-11T03:39:34.546 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.546 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.546 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:39:34.546 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835142+0000", 2024-09-11T03:39:34.546 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7209999999999999e-06 2024-09-11T03:39:34.547 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.547 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.547 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.547 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.836256+0000", 2024-09-11T03:39:34.547 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0011148029999999999 2024-09-11T03:39:34.547 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.547 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.547 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.547 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.836330+0000", 2024-09-11T03:39:34.547 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3497000000000005e-05 2024-09-11T03:39:34.547 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.547 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.547 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:39:34.548 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.836491+0000", 2024-09-11T03:39:34.548 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016083699999999999 2024-09-11T03:39:34.548 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.548 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.548 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:34.548 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.838228+0000", 2024-09-11T03:39:34.548 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001737204 2024-09-11T03:39:34.548 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.548 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.548 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:34.548 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.838299+0000", 2024-09-11T03:39:34.549 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0770000000000002e-05 2024-09-11T03:39:34.549 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.549 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.549 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:34.549 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.838342+0000", 2024-09-11T03:39:34.549 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3115999999999999e-05 2024-09-11T03:39:34.549 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.549 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.549 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:34.549 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.838376+0000", 2024-09-11T03:39:34.549 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4249999999999999e-05 2024-09-11T03:39:34.549 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.549 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.549 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.550 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.838407+0000", 2024-09-11T03:39:34.550 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0409999999999999e-05 2024-09-11T03:39:34.550 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.550 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.550 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.550 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.550 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.550 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:562 2.d 2:b6a25a79:::smithi17557870-47:head [rollback 10] snapc 11=[11,10,f,e] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-11T03:39:34.550 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:04.835088+0000", 2024-09-11T03:39:34.550 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 29.668623574000001, 2024-09-11T03:39:34.550 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035220730000000001, 2024-09-11T03:39:34.550 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.550 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.550 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:34.551 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:34.551 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:34.551 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:34.551 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 562 2024-09-11T03:39:34.551 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.551 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.551 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.551 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.551 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835088+0000", 2024-09-11T03:39:34.551 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.551 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.551 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.551 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.552 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835088+0000", 2024-09-11T03:39:34.552 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.552 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.552 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.552 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.552 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835089+0000", 2024-09-11T03:39:34.552 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1900000000000003e-07 2024-09-11T03:39:34.552 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.552 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.552 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.552 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835092+0000", 2024-09-11T03:39:34.552 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8940000000000002e-06 2024-09-11T03:39:34.552 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.552 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.553 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.553 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835101+0000", 2024-09-11T03:39:34.553 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4939999999999992e-06 2024-09-11T03:39:34.553 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.553 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.553 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.553 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835106+0000", 2024-09-11T03:39:34.553 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3159999999999998e-06 2024-09-11T03:39:34.553 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.553 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.553 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.553 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835211+0000", 2024-09-11T03:39:34.553 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010550100000000001 2024-09-11T03:39:34.554 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.554 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.554 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:34.554 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835219+0000", 2024-09-11T03:39:34.554 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2050000000000002e-06 2024-09-11T03:39:34.554 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.554 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.554 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.554 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.837069+0000", 2024-09-11T03:39:34.554 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0018495079999999999 2024-09-11T03:39:34.554 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.554 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.554 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.554 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.837158+0000", 2024-09-11T03:39:34.555 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.9247999999999997e-05 2024-09-11T03:39:34.555 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.555 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.555 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:39:34.555 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.837362+0000", 2024-09-11T03:39:34.555 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00020343100000000001 2024-09-11T03:39:34.555 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.555 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.555 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:34.555 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.838280+0000", 2024-09-11T03:39:34.555 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000918022 2024-09-11T03:39:34.555 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.555 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.556 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:34.556 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.838469+0000", 2024-09-11T03:39:34.556 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00018898999999999999 2024-09-11T03:39:34.556 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.556 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.556 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:34.556 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.838486+0000", 2024-09-11T03:39:34.556 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7397999999999998e-05 2024-09-11T03:39:34.556 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.556 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.556 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:34.556 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.838556+0000", 2024-09-11T03:39:34.557 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9844000000000004e-05 2024-09-11T03:39:34.557 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.557 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.557 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.557 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.838611+0000", 2024-09-11T03:39:34.557 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4703000000000001e-05 2024-09-11T03:39:34.557 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.557 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.557 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.557 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.557 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.557 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:563 2.d 2:b6a25a79:::smithi17557870-47:head [append 0~0] snapc 11=[11,10,f,e] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-11T03:39:34.557 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:04.835127+0000", 2024-09-11T03:39:34.557 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 29.668585115999999, 2024-09-11T03:39:34.558 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003697681, 2024-09-11T03:39:34.558 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.558 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.558 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:34.558 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:34.558 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:34.558 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:34.558 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 563 2024-09-11T03:39:34.558 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.558 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.558 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.559 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.559 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835127+0000", 2024-09-11T03:39:34.559 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.559 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.559 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.559 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.559 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835127+0000", 2024-09-11T03:39:34.559 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.559 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.559 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.559 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.559 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835127+0000", 2024-09-11T03:39:34.559 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5699999999999998e-07 2024-09-11T03:39:34.560 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.560 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.560 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.560 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835131+0000", 2024-09-11T03:39:34.560 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.76e-06 2024-09-11T03:39:34.560 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.560 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.560 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.560 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835138+0000", 2024-09-11T03:39:34.560 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.832e-06 2024-09-11T03:39:34.561 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.561 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.561 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.561 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835143+0000", 2024-09-11T03:39:34.561 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3290000000000001e-06 2024-09-11T03:39:34.561 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.561 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.561 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.561 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835251+0000", 2024-09-11T03:39:34.561 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000107383 2024-09-11T03:39:34.561 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.561 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.562 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:34.562 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.835255+0000", 2024-09-11T03:39:34.562 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9199999999999997e-06 2024-09-11T03:39:34.562 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.562 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.562 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.562 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.837808+0000", 2024-09-11T03:39:34.562 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025538599999999998 2024-09-11T03:39:34.562 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.562 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.562 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.562 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.837879+0000", 2024-09-11T03:39:34.562 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0246000000000005e-05 2024-09-11T03:39:34.562 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.563 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.563 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:39:34.563 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.838013+0000", 2024-09-11T03:39:34.563 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013447000000000001 2024-09-11T03:39:34.563 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.563 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.563 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:34.563 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.838619+0000", 2024-09-11T03:39:34.563 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00060556800000000001 2024-09-11T03:39:34.564 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.564 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.564 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:34.564 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.838675+0000", 2024-09-11T03:39:34.564 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6032999999999997e-05 2024-09-11T03:39:34.564 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.564 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.564 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:34.564 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.838722+0000", 2024-09-11T03:39:34.564 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7432000000000003e-05 2024-09-11T03:39:34.564 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.565 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.565 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:34.565 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.838781+0000", 2024-09-11T03:39:34.565 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8768999999999999e-05 2024-09-11T03:39:34.565 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.565 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.565 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.565 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:04.838825+0000", 2024-09-11T03:39:34.565 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3621999999999997e-05 2024-09-11T03:39:34.565 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.565 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.565 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.566 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.566 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.566 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:588 2.f e129/95 2:fda69313:::smithi17557870-16:head v 129'32, mlcod=124'28)", 2024-09-11T03:39:34.566 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:05.847833+0000", 2024-09-11T03:39:34.566 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 28.655878743999999, 2024-09-11T03:39:34.566 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0051080730000000003, 2024-09-11T03:39:34.566 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.566 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.566 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:34.566 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.566 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.566 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.566 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.847833+0000", 2024-09-11T03:39:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.847833+0000", 2024-09-11T03:39:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.847835+0000", 2024-09-11T03:39:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4309999999999999e-06 2024-09-11T03:39:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.850660+0000", 2024-09-11T03:39:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0028253430000000001 2024-09-11T03:39:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.850688+0000", 2024-09-11T03:39:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7455999999999999e-05 2024-09-11T03:39:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.850702+0000", 2024-09-11T03:39:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4747e-05 2024-09-11T03:39:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.569 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.569 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.569 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.850784+0000", 2024-09-11T03:39:34.569 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1337999999999999e-05 2024-09-11T03:39:34.569 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.569 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.569 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.569 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.850810+0000", 2024-09-11T03:39:34.569 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.633e-05 2024-09-11T03:39:34.569 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.569 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.569 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:34.570 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.852898+0000", 2024-09-11T03:39:34.570 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0020881599999999999 2024-09-11T03:39:34.570 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.570 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.570 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.570 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.852941+0000", 2024-09-11T03:39:34.570 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3268e-05 2024-09-11T03:39:34.570 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.570 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.570 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.570 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.570 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.570 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:646 2.d 2:b6874adf:::smithi17557870-17:head [read 0~3479016,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:39:34.571 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.111639+0000", 2024-09-11T03:39:34.571 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 26.392073052000001, 2024-09-11T03:39:34.571 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0033608539999999999, 2024-09-11T03:39:34.571 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.571 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.571 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:34.571 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:34.571 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:34.571 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:34.571 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 646 2024-09-11T03:39:34.571 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.571 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.571 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111639+0000", 2024-09-11T03:39:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111639+0000", 2024-09-11T03:39:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111641+0000", 2024-09-11T03:39:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.747e-06 2024-09-11T03:39:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.573 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.573 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.573 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111646+0000", 2024-09-11T03:39:34.573 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7760000000000003e-06 2024-09-11T03:39:34.573 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.573 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.573 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.573 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111660+0000", 2024-09-11T03:39:34.573 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3484e-05 2024-09-11T03:39:34.573 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.573 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.573 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.574 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111667+0000", 2024-09-11T03:39:34.574 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3329999999999999e-06 2024-09-11T03:39:34.574 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.574 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.574 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.574 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111731+0000", 2024-09-11T03:39:34.574 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4152999999999994e-05 2024-09-11T03:39:34.574 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.574 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.574 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.574 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111814+0000", 2024-09-11T03:39:34.574 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2119000000000004e-05 2024-09-11T03:39:34.574 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.115000+0000", 2024-09-11T03:39:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0031862420000000002 2024-09-11T03:39:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:647 2.d 2:b6874adf:::smithi17557870-17:head [sparse-read 0~1610566] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:39:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.111684+0000", 2024-09-11T03:39:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 26.392027556999999, 2024-09-11T03:39:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035183910000000001, 2024-09-11T03:39:34.576 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.576 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.576 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:34.576 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:34.576 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:34.576 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:34.576 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 647 2024-09-11T03:39:34.576 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.576 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.576 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.576 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.576 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111684+0000", 2024-09-11T03:39:34.576 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.577 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.577 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.577 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.577 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111684+0000", 2024-09-11T03:39:34.577 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.577 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.577 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.577 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.577 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111685+0000", 2024-09-11T03:39:34.577 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6499999999999999e-07 2024-09-11T03:39:34.577 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.577 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.578 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.578 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111688+0000", 2024-09-11T03:39:34.578 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0060000000000001e-06 2024-09-11T03:39:34.578 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.578 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.578 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.578 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111704+0000", 2024-09-11T03:39:34.578 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6435000000000001e-05 2024-09-11T03:39:34.578 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.578 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.578 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.578 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111709+0000", 2024-09-11T03:39:34.578 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5299999999999998e-06 2024-09-11T03:39:34.579 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.579 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.579 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.579 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.115059+0000", 2024-09-11T03:39:34.579 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0033495809999999999 2024-09-11T03:39:34.579 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.579 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.579 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.579 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.115117+0000", 2024-09-11T03:39:34.579 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8669999999999999e-05 2024-09-11T03:39:34.579 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.580 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.580 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.580 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.115203+0000", 2024-09-11T03:39:34.580 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5604000000000002e-05 2024-09-11T03:39:34.580 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.580 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.580 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.580 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.580 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.580 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:648 2.d 2:b6874adf:::smithi17557870-17:head [sparse-read 0~1610566] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:39:34.580 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.111715+0000", 2024-09-11T03:39:34.580 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 26.391997131, 2024-09-11T03:39:34.580 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0037000050000000001, 2024-09-11T03:39:34.580 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.581 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.581 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:34.581 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:34.581 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:34.581 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:34.581 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 648 2024-09-11T03:39:34.581 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.581 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.581 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.581 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.581 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:34.583 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111715+0000", 2024-09-11T03:39:34.583 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.583 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.583 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.583 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.583 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111715+0000", 2024-09-11T03:39:34.583 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.583 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.584 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.584 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.584 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111715+0000", 2024-09-11T03:39:34.584 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8000000000000002e-07 2024-09-11T03:39:34.584 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.584 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.584 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.584 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111718+0000", 2024-09-11T03:39:34.584 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4040000000000002e-06 2024-09-11T03:39:34.584 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.584 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.584 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.584 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111721+0000", 2024-09-11T03:39:34.585 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.585e-06 2024-09-11T03:39:34.585 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.585 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.585 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.585 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.111724+0000", 2024-09-11T03:39:34.585 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.88e-06 2024-09-11T03:39:34.585 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.585 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.585 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.585 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.115229+0000", 2024-09-11T03:39:34.585 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035047009999999998 2024-09-11T03:39:34.585 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.585 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.586 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.586 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.115305+0000", 2024-09-11T03:39:34.586 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6472999999999998e-05 2024-09-11T03:39:34.586 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.586 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.586 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.586 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.115415+0000", 2024-09-11T03:39:34.586 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000109682 2024-09-11T03:39:34.586 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.586 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.586 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.587 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.587 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.587 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:640 2.a e131/95 2:5014e305:::smithi17557870-46:head v 131'5, mlcod=100'2)", 2024-09-11T03:39:34.587 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:39:34.587 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 26.385293088000001, 2024-09-11T03:39:34.587 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068845190000000004, 2024-09-11T03:39:34.587 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.587 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.587 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:34.587 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.587 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.587 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.587 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:39:34.587 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.588 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.588 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.588 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.588 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:39:34.588 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.588 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.588 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.588 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.588 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:39:34.588 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7399999999999998e-07 2024-09-11T03:39:34.588 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.588 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.588 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.589 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.120287+0000", 2024-09-11T03:39:34.589 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0018676809999999999 2024-09-11T03:39:34.589 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.589 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.589 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.589 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.120324+0000", 2024-09-11T03:39:34.589 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7082000000000002e-05 2024-09-11T03:39:34.590 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.590 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.590 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.590 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.120345+0000", 2024-09-11T03:39:34.590 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0888000000000001e-05 2024-09-11T03:39:34.590 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.590 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.590 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.590 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.120456+0000", 2024-09-11T03:39:34.590 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000111328 2024-09-11T03:39:34.590 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.590 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.590 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.591 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.120478+0000", 2024-09-11T03:39:34.591 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1732000000000001e-05 2024-09-11T03:39:34.591 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.591 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.591 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:34.591 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125257+0000", 2024-09-11T03:39:34.591 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0047789479999999999 2024-09-11T03:39:34.591 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.591 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.591 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.591 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125303+0000", 2024-09-11T03:39:34.591 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6385999999999999e-05 2024-09-11T03:39:34.591 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.591 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.592 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.592 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.592 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.592 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:760 2.f e140/95 2:fda69313:::smithi17557870-16:head v 140'45, mlcod=131'42)", 2024-09-11T03:39:34.592 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:16.827820+0000", 2024-09-11T03:39:34.592 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 17.675892366999999, 2024-09-11T03:39:34.592 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030417489999999998, 2024-09-11T03:39:34.592 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.592 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.592 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:34.592 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.592 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.592 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.593 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.827820+0000", 2024-09-11T03:39:34.593 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.593 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.593 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.593 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.593 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.827820+0000", 2024-09-11T03:39:34.593 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.593 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.593 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.593 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.593 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.827820+0000", 2024-09-11T03:39:34.594 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1600000000000002e-07 2024-09-11T03:39:34.594 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.594 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.594 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.594 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.829149+0000", 2024-09-11T03:39:34.594 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013287139999999999 2024-09-11T03:39:34.594 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.594 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.594 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.594 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.829163+0000", 2024-09-11T03:39:34.594 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4661000000000001e-05 2024-09-11T03:39:34.594 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.594 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.595 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.595 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.829177+0000", 2024-09-11T03:39:34.595 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3385e-05 2024-09-11T03:39:34.595 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.595 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.595 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.595 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.829622+0000", 2024-09-11T03:39:34.595 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00044550200000000002 2024-09-11T03:39:34.595 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.595 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.595 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.596 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.829656+0000", 2024-09-11T03:39:34.596 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4008999999999999e-05 2024-09-11T03:39:34.596 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.596 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.596 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:34.596 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.830823+0000", 2024-09-11T03:39:34.596 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001167055 2024-09-11T03:39:34.596 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.596 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.596 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.596 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.830861+0000", 2024-09-11T03:39:34.597 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8107e-05 2024-09-11T03:39:34.597 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.597 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.597 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.597 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.597 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.597 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:770 2.d 2:b6874adf:::smithi17557870-17:head [read 0~3479016,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e140)", 2024-09-11T03:39:34.597 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:16.842781+0000", 2024-09-11T03:39:34.597 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 17.660930674999999, 2024-09-11T03:39:34.597 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003452344, 2024-09-11T03:39:34.597 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.597 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.597 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:34.598 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:34.598 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:34.598 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:34.598 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 770 2024-09-11T03:39:34.598 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.598 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.598 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.598 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.598 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.842781+0000", 2024-09-11T03:39:34.598 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.598 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.598 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.598 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.598 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.842781+0000", 2024-09-11T03:39:34.599 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.599 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.599 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.599 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.599 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.842782+0000", 2024-09-11T03:39:34.599 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8100000000000003e-07 2024-09-11T03:39:34.599 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.599 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.599 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.599 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.842785+0000", 2024-09-11T03:39:34.599 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5889999999999999e-06 2024-09-11T03:39:34.599 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.600 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.600 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.600 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.842792+0000", 2024-09-11T03:39:34.600 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4330000000000001e-06 2024-09-11T03:39:34.600 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.600 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.600 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.600 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.842796+0000", 2024-09-11T03:39:34.601 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6550000000000003e-06 2024-09-11T03:39:34.601 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.601 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.601 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.601 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.843328+0000", 2024-09-11T03:39:34.601 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00053115400000000004 2024-09-11T03:39:34.601 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.601 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.601 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.601 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.843380+0000", 2024-09-11T03:39:34.601 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2185999999999998e-05 2024-09-11T03:39:34.602 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.602 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.602 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.602 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:16.846234+0000", 2024-09-11T03:39:34.602 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0028538460000000002 2024-09-11T03:39:34.602 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.602 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.602 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.602 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.602 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.602 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:847 2.d 2:b1c0f730:::smithi17557870-20:head [copy-from ver 90] snapc 1a=[1a,19] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-11T03:39:34.602 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:39:34.602 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 9.6644169170000005, 2024-09-11T03:39:34.602 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013054913, 2024-09-11T03:39:34.603 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.603 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.603 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:34.603 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:34.603 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:34.603 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:34.603 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 847 2024-09-11T03:39:34.603 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.603 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.603 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.603 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.603 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:39:34.603 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.604 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.604 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.604 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.604 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:39:34.604 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.604 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.604 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.604 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.604 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839297+0000", 2024-09-11T03:39:34.604 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0049999999999999e-06 2024-09-11T03:39:34.604 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.604 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.605 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.605 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839309+0000", 2024-09-11T03:39:34.605 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1531e-05 2024-09-11T03:39:34.605 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.605 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.605 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.605 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839326+0000", 2024-09-11T03:39:34.605 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7788e-05 2024-09-11T03:39:34.605 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.605 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.605 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.606 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839336+0000", 2024-09-11T03:39:34.606 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5619999999999993e-06 2024-09-11T03:39:34.606 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.606 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.606 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.606 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839423+0000", 2024-09-11T03:39:34.606 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6790000000000001e-05 2024-09-11T03:39:34.606 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.606 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.606 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:39:34.606 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839434+0000", 2024-09-11T03:39:34.606 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0971e-05 2024-09-11T03:39:34.606 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.607 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.607 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.607 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841520+0000", 2024-09-11T03:39:34.607 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0020866299999999999 2024-09-11T03:39:34.607 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.607 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.607 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.607 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841590+0000", 2024-09-11T03:39:34.607 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9288000000000005e-05 2024-09-11T03:39:34.607 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.607 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.607 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:39:34.607 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.846430+0000", 2024-09-11T03:39:34.607 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0048408349999999999 2024-09-11T03:39:34.608 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.608 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.608 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:34.608 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849726+0000", 2024-09-11T03:39:34.608 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032955390000000001 2024-09-11T03:39:34.608 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.608 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.608 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:34.608 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852240+0000", 2024-09-11T03:39:34.608 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002513492 2024-09-11T03:39:34.608 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.608 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.608 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:34.609 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852278+0000", 2024-09-11T03:39:34.609 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7988999999999998e-05 2024-09-11T03:39:34.609 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.609 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.609 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:34.609 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852317+0000", 2024-09-11T03:39:34.609 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9310000000000001e-05 2024-09-11T03:39:34.609 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.609 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.609 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.609 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852350+0000", 2024-09-11T03:39:34.609 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3182999999999997e-05 2024-09-11T03:39:34.609 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.609 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.610 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.610 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.610 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.610 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:848 2.d 2:b1c0f730:::smithi17557870-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2024-09-11T03:39:34.610 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:24.839354+0000", 2024-09-11T03:39:34.610 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 9.6643575199999994, 2024-09-11T03:39:34.610 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010055476000000001, 2024-09-11T03:39:34.610 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.610 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.610 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:34.610 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:34.611 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:34.611 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:34.611 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 848 2024-09-11T03:39:34.611 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.611 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.611 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.611 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.611 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839354+0000", 2024-09-11T03:39:34.611 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.611 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.611 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.611 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.611 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839354+0000", 2024-09-11T03:39:34.611 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.611 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.612 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.612 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.612 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839355+0000", 2024-09-11T03:39:34.612 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6899999999999996e-07 2024-09-11T03:39:34.612 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.612 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.612 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.612 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839358+0000", 2024-09-11T03:39:34.612 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8729999999999999e-06 2024-09-11T03:39:34.612 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.612 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.612 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.613 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839367+0000", 2024-09-11T03:39:34.613 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3999999999999992e-06 2024-09-11T03:39:34.613 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.613 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.613 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.613 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839371+0000", 2024-09-11T03:39:34.613 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6059999999999996e-06 2024-09-11T03:39:34.613 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.613 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.613 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.613 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839509+0000", 2024-09-11T03:39:34.613 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013766700000000001 2024-09-11T03:39:34.613 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.613 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.614 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:34.614 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839515+0000", 2024-09-11T03:39:34.614 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4740000000000002e-06 2024-09-11T03:39:34.614 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.614 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.614 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.614 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841883+0000", 2024-09-11T03:39:34.614 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023678319999999998 2024-09-11T03:39:34.614 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.614 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.614 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:39:34.614 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841941+0000", 2024-09-11T03:39:34.614 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8241e-05 2024-09-11T03:39:34.615 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.615 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.615 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.615 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849208+0000", 2024-09-11T03:39:34.615 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0072661690000000003 2024-09-11T03:39:34.615 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.615 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.615 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.615 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849295+0000", 2024-09-11T03:39:34.615 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7582000000000003e-05 2024-09-11T03:39:34.615 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.615 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.615 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.615 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849410+0000", 2024-09-11T03:39:34.616 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114863 2024-09-11T03:39:34.616 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.616 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.616 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.616 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.616 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.616 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:876 2.d 2:b6a25a79:::smithi17557870-47:head [rollback 1c] snapc 1e=[1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-11T03:39:34.616 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:31.825650+0000", 2024-09-11T03:39:34.616 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 2.6780617530000002, 2024-09-11T03:39:34.616 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025414589999999998, 2024-09-11T03:39:34.616 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.616 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.616 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:34.617 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:34.617 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:34.617 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:34.617 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 876 2024-09-11T03:39:34.617 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.617 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.617 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.617 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.617 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.825650+0000", 2024-09-11T03:39:34.617 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.617 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.617 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.617 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.617 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.825650+0000", 2024-09-11T03:39:34.618 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.618 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.618 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.618 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.618 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.825652+0000", 2024-09-11T03:39:34.618 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5939999999999999e-06 2024-09-11T03:39:34.618 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.618 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.618 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.618 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.825656+0000", 2024-09-11T03:39:34.618 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9979999999999997e-06 2024-09-11T03:39:34.618 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.618 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.619 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.619 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.825667+0000", 2024-09-11T03:39:34.619 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0659000000000001e-05 2024-09-11T03:39:34.619 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.619 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.619 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.619 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.825675+0000", 2024-09-11T03:39:34.619 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.276e-06 2024-09-11T03:39:34.619 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.619 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.619 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.619 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.825746+0000", 2024-09-11T03:39:34.619 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1340999999999993e-05 2024-09-11T03:39:34.619 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.619 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.620 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:39:34.620 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.825753+0000", 2024-09-11T03:39:34.620 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0940000000000002e-06 2024-09-11T03:39:34.620 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.620 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.620 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.620 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.827212+0000", 2024-09-11T03:39:34.620 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014588159999999999 2024-09-11T03:39:34.620 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.620 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.621 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.621 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.827275+0000", 2024-09-11T03:39:34.621 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3164999999999996e-05 2024-09-11T03:39:34.621 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.621 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.621 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.621 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.828192+0000", 2024-09-11T03:39:34.621 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00091651600000000003 2024-09-11T03:39:34.621 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.621 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.621 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.621 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.622 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.622 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:877 2.e e155/95 2:77dadb97:::smithi17557870-30:head v 155'80, mlcod=136'76)", 2024-09-11T03:39:34.622 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:31.994724+0000", 2024-09-11T03:39:34.622 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 2.5089882660000002, 2024-09-11T03:39:34.622 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0046425659999999999, 2024-09-11T03:39:34.622 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:34.622 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:34.622 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:34.622 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:34.622 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.622 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:34.622 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.994724+0000", 2024-09-11T03:39:34.622 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.623 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.623 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.623 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:34.623 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.994724+0000", 2024-09-11T03:39:34.623 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:34.623 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.623 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.623 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:34.623 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.994724+0000", 2024-09-11T03:39:34.623 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2999999999999999e-07 2024-09-11T03:39:34.623 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.623 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.623 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:34.624 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.997621+0000", 2024-09-11T03:39:34.624 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0028969109999999998 2024-09-11T03:39:34.624 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.624 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.624 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:34.624 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.997638+0000", 2024-09-11T03:39:34.624 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7325000000000001e-05 2024-09-11T03:39:34.624 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.624 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.624 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:34.624 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.997648+0000", 2024-09-11T03:39:34.624 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5759999999999995e-06 2024-09-11T03:39:34.624 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.624 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.625 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:34.625 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.997708+0000", 2024-09-11T03:39:34.625 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0022000000000002e-05 2024-09-11T03:39:34.625 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.625 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.625 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:34.625 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.997726+0000", 2024-09-11T03:39:34.625 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8655e-05 2024-09-11T03:39:34.625 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.625 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.625 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:34.625 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.999316+0000", 2024-09-11T03:39:34.625 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00158965 2024-09-11T03:39:34.625 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:34.626 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:34.626 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:34.626 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:31.999366+0000", 2024-09-11T03:39:34.626 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0197000000000001e-05 2024-09-11T03:39:34.626 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.626 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.626 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.626 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:34.626 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:34.626 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:39:34.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:34.628 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:34.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:34.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:34.754 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:34.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:34.791 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:39:34.791 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:39:34.810 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:39:34.810 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:39:34.833 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:39:34.833 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:39:34.855 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:34.856 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:39:34.857 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:39:34.871 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:34.878 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:39:34.878 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:39:34.884 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:34.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:34.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:34.901 INFO:tasks.rados.rados.2.smithi175.stdout:339: done (3 left) 2024-09-11T03:39:34.901 INFO:tasks.rados.rados.2.smithi175.stdout:340: copy_from oid 47 from oid 16 current snap is 31 2024-09-11T03:39:34.901 INFO:tasks.rados.rados.2.smithi175.stdout:335: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-11T03:39:34.907 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:39:34.907 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:39:34.924 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:39:34.924 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:39:34.941 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:39:34.941 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:39:34.956 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:39:34.956 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:39:34.956 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:34 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:39:34.956 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:34 smithi084 ceph-mon[27254]: 2.a scrub ok 2024-09-11T03:39:34.956 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:34 smithi084 ceph-mon[27254]: 2.5 scrub starts 2024-09-11T03:39:34.956 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:34 smithi084 ceph-mon[27254]: 2.5 scrub ok 2024-09-11T03:39:34.957 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:34 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:39:34.957 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:34 smithi084 ceph-mon[27254]: 2.0 scrub ok 2024-09-11T03:39:34.957 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:34 smithi084 ceph-mon[27254]: osdmap e157: 12 total, 12 up, 11 in 2024-09-11T03:39:34.957 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:34 smithi084 ceph-mon[27254]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:34.957 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:34 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1538901511' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:34.974 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:39:34.974 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:39:34.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:34.990 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:39:34.990 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:39:34.995 INFO:tasks.rados.rados.2.smithi175.stdout:337: finishing rollback tid 0 to smithi17557870-2 2024-09-11T03:39:34.995 INFO:tasks.rados.rados.2.smithi175.stdout:337: finishing rollback tid 1 to smithi17557870-2 2024-09-11T03:39:34.995 INFO:tasks.rados.rados.2.smithi175.stdout:337: finishing rollback tid 2 to smithi17557870-2 2024-09-11T03:39:34.995 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 73 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T03:39:34.995 INFO:tasks.rados.rados.2.smithi175.stdout:338: finishing write tid 1 to smithi17557870-36 2024-09-11T03:39:34.995 INFO:tasks.rados.rados.2.smithi175.stdout:338: finishing write tid 2 to smithi17557870-36 2024-09-11T03:39:34.995 INFO:tasks.rados.rados.2.smithi175.stdout:338: finishing write tid 3 to smithi17557870-36 2024-09-11T03:39:34.996 INFO:tasks.rados.rados.2.smithi175.stdout:338: finishing write tid 4 to smithi17557870-36 2024-09-11T03:39:34.996 INFO:tasks.rados.rados.2.smithi175.stdout:338: oid 36 updating version 0 to 39 2024-09-11T03:39:34.996 INFO:tasks.rados.rados.2.smithi175.stdout:338: oid 36 updating version 39 to 40 2024-09-11T03:39:34.996 INFO:tasks.rados.rados.2.smithi175.stdout:338: oid 36 updating version 40 to 41 2024-09-11T03:39:34.996 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 41 (ObjNum 144 snap 31 seq_num 144) dirty exists 2024-09-11T03:39:34.996 INFO:tasks.rados.rados.2.smithi175.stdout:338: left oid 36 (ObjNum 144 snap 31 seq_num 144) 2024-09-11T03:39:34.996 INFO:tasks.rados.rados.2.smithi175.stdout:335: done (3 left) 2024-09-11T03:39:34.996 INFO:tasks.rados.rados.2.smithi175.stdout:337: done (2 left) 2024-09-11T03:39:34.997 INFO:tasks.rados.rados.2.smithi175.stdout:338: done (1 left) 2024-09-11T03:39:34.997 INFO:tasks.rados.rados.2.smithi175.stdout:341: copy_from oid 32 from oid 48 current snap is 31 2024-09-11T03:39:34.997 INFO:tasks.rados.rados.2.smithi175.stdout:340: finishing copy_from racing read to smithi17557870-47 2024-09-11T03:39:34.997 INFO:tasks.rados.rados.2.smithi175.stdout:340: finishing copy_from to smithi17557870-47 2024-09-11T03:39:34.997 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 47 (ObjNum 124 snap 20 seq_num 124) dirty exists 2024-09-11T03:39:34.997 INFO:tasks.rados.rados.2.smithi175.stdout:340: done (1 left) 2024-09-11T03:39:34.997 INFO:tasks.rados.rados.2.smithi175.stdout:342: write (excl) oid 5 current snap is 31 2024-09-11T03:39:34.997 INFO:tasks.rados.rados.2.smithi175.stdout:342: seq_num 145 ranges {437015=721812,1701889=509708} 2024-09-11T03:39:35.001 INFO:tasks.rados.rados.2.smithi175.stdout:342: writing smithi17557870-5 from 437015 to 1158827 tid 1 2024-09-11T03:39:35.006 INFO:tasks.rados.rados.2.smithi175.stdout:342: writing smithi17557870-5 from 1701889 to 2211597 tid 2 2024-09-11T03:39:35.007 INFO:tasks.rados.rados.2.smithi175.stdout:343: snap_create 2024-09-11T03:39:35.007 INFO:tasks.rados.rados.2.smithi175.stdout:342: finishing write tid 1 to smithi17557870-5 2024-09-11T03:39:35.007 INFO:tasks.rados.rados.2.smithi175.stdout:341: finishing copy_from racing read to smithi17557870-32 2024-09-11T03:39:35.009 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:39:35.009 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:39:35.010 INFO:tasks.rados.rados.2.smithi175.stdout:342: finishing write tid 2 to smithi17557870-5 2024-09-11T03:39:35.010 INFO:tasks.rados.rados.2.smithi175.stdout:341: finishing copy_from to smithi17557870-32 2024-09-11T03:39:35.010 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 40 (ObjNum 108 snap 16 seq_num 108) dirty exists 2024-09-11T03:39:35.010 INFO:tasks.rados.rados.2.smithi175.stdout:342: finishing write tid 3 to smithi17557870-5 2024-09-11T03:39:35.010 INFO:tasks.rados.rados.2.smithi175.stdout:342: finishing write tid 4 to smithi17557870-5 2024-09-11T03:39:35.010 INFO:tasks.rados.rados.2.smithi175.stdout:342: oid 5 updating version 0 to 28 2024-09-11T03:39:35.010 INFO:tasks.rados.rados.2.smithi175.stdout:342: oid 5 updating version 28 to 29 2024-09-11T03:39:35.010 INFO:tasks.rados.rados.2.smithi175.stdout:342: oid 5 version 29 is already newer than 27 2024-09-11T03:39:35.010 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 29 (ObjNum 145 snap 31 seq_num 145) dirty exists 2024-09-11T03:39:35.010 INFO:tasks.rados.rados.2.smithi175.stdout:342: left oid 5 (ObjNum 145 snap 31 seq_num 145) 2024-09-11T03:39:35.012 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:35.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:35.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:35.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:35.138 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:35.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:35.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:35.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:34 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:39:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:34 smithi121 ceph-mon[30729]: 2.a scrub ok 2024-09-11T03:39:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:34 smithi121 ceph-mon[30729]: 2.5 scrub starts 2024-09-11T03:39:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:34 smithi121 ceph-mon[30729]: 2.5 scrub ok 2024-09-11T03:39:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:34 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:39:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:34 smithi121 ceph-mon[30729]: 2.0 scrub ok 2024-09-11T03:39:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:34 smithi121 ceph-mon[30729]: osdmap e157: 12 total, 12 up, 11 in 2024-09-11T03:39:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:34 smithi121 ceph-mon[30729]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:34 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1538901511' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:34 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:39:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:34 smithi175 ceph-mon[30894]: 2.a scrub ok 2024-09-11T03:39:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:34 smithi175 ceph-mon[30894]: 2.5 scrub starts 2024-09-11T03:39:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:34 smithi175 ceph-mon[30894]: 2.5 scrub ok 2024-09-11T03:39:35.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:34 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:39:35.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:34 smithi175 ceph-mon[30894]: 2.0 scrub ok 2024-09-11T03:39:35.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:34 smithi175 ceph-mon[30894]: osdmap e157: 12 total, 12 up, 11 in 2024-09-11T03:39:35.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:34 smithi175 ceph-mon[30894]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:35.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:34 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1538901511' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:35.240 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:35.261 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:35.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:35.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:35.269 INFO:teuthology.orchestra.run.smithi084.stdout:moved 0 / 51 (0%) 2024-09-11T03:39:35.270 INFO:teuthology.orchestra.run.smithi084.stdout:avg 4.63636 2024-09-11T03:39:35.270 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 1.72008 -> 1.72008 (expected baseline 2.05302) 2024-09-11T03:39:35.270 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.5 with 2 -> 2 pgs (0.431373 -> 0.431373 * mean) 2024-09-11T03:39:35.270 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.9 with 8 -> 8 pgs (1.72549 -> 1.72549 * mean) 2024-09-11T03:39:35.270 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:39:35.270 INFO:teuthology.orchestra.run.smithi084.stdout:oload 1000 2024-09-11T03:39:35.270 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 0.05 2024-09-11T03:39:35.270 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2024-09-11T03:39:35.270 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0005 2024-09-11T03:39:35.270 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0052 2024-09-11T03:39:35.271 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2024-09-11T03:39:35.320 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-11T03:39:35.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:35.389 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:35.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:35.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:35.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:35.515 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:35.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:35.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:35.597 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:39:35.618 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:35.639 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:35.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:35.640 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:35.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:35.767 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:35.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:35.869 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:35.890 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:35.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:35.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:35.902 INFO:tasks.rados.rados.2.smithi175.stdout:341: done (2 left) 2024-09-11T03:39:35.902 INFO:tasks.rados.rados.2.smithi175.stdout:342: done (1 left) 2024-09-11T03:39:35.902 INFO:tasks.rados.rados.2.smithi175.stdout:343: done (0 left) 2024-09-11T03:39:35.902 INFO:tasks.rados.rados.2.smithi175.stdout:344: rollback oid 38 current snap is 32 2024-09-11T03:39:35.902 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 38 to 29 2024-09-11T03:39:35.903 INFO:tasks.rados.rados.2.smithi175.stdout:345: write (excl) oid 17 current snap is 32 2024-09-11T03:39:35.903 INFO:tasks.rados.rados.2.smithi175.stdout:345: seq_num 146 ranges {453822=797061,1760371=653366} 2024-09-11T03:39:35.909 INFO:tasks.rados.rados.2.smithi175.stdout:345: writing smithi17557870-17 from 453822 to 1250883 tid 1 2024-09-11T03:39:35.917 INFO:tasks.rados.rados.2.smithi175.stdout:345: writing smithi17557870-17 from 1760371 to 2413737 tid 2 2024-09-11T03:39:35.917 INFO:tasks.rados.rados.2.smithi175.stdout:346: write (excl) oid 10 current snap is 32 2024-09-11T03:39:35.917 INFO:tasks.rados.rados.2.smithi175.stdout:344: finishing rollback tid 0 to smithi17557870-38 2024-09-11T03:39:35.917 INFO:tasks.rados.rados.2.smithi175.stdout:344: finishing rollback tid 1 to smithi17557870-38 2024-09-11T03:39:35.917 INFO:tasks.rados.rados.2.smithi175.stdout:344: finishing rollback tid 2 to smithi17557870-38 2024-09-11T03:39:35.917 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 101 (ObjNum 101 snap 16 seq_num 101) dirty exists 2024-09-11T03:39:35.917 INFO:tasks.rados.rados.2.smithi175.stdout:345: finishing write tid 1 to smithi17557870-17 2024-09-11T03:39:35.917 INFO:tasks.rados.rados.2.smithi175.stdout:346: seq_num 147 ranges {470629=472310,1418853=513377} 2024-09-11T03:39:35.920 INFO:tasks.rados.rados.2.smithi175.stdout:346: writing smithi17557870-10 from 470629 to 942939 tid 1 2024-09-11T03:39:35.926 INFO:tasks.rados.rados.2.smithi175.stdout:346: writing smithi17557870-10 from 1418853 to 1932230 tid 2 2024-09-11T03:39:35.926 INFO:tasks.rados.rados.2.smithi175.stdout:344: done (2 left) 2024-09-11T03:39:35.926 INFO:tasks.rados.rados.2.smithi175.stdout:347: snap_create 2024-09-11T03:39:35.926 INFO:tasks.rados.rados.2.smithi175.stdout:345: finishing write tid 2 to smithi17557870-17 2024-09-11T03:39:35.926 INFO:tasks.rados.rados.2.smithi175.stdout:345: finishing write tid 3 to smithi17557870-17 2024-09-11T03:39:35.926 INFO:tasks.rados.rados.2.smithi175.stdout:345: finishing write tid 4 to smithi17557870-17 2024-09-11T03:39:35.926 INFO:tasks.rados.rados.2.smithi175.stdout:345: oid 17 updating version 0 to 93 2024-09-11T03:39:35.927 INFO:tasks.rados.rados.2.smithi175.stdout:345: oid 17 version 93 is already newer than 91 2024-09-11T03:39:35.927 INFO:tasks.rados.rados.2.smithi175.stdout:345: oid 17 version 93 is already newer than 92 2024-09-11T03:39:35.927 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 93 (ObjNum 146 snap 32 seq_num 146) dirty exists 2024-09-11T03:39:35.927 INFO:tasks.rados.rados.2.smithi175.stdout:345: left oid 17 (ObjNum 146 snap 32 seq_num 146) 2024-09-11T03:39:35.927 INFO:tasks.rados.rados.2.smithi175.stdout:346: finishing write tid 1 to smithi17557870-10 2024-09-11T03:39:35.928 INFO:tasks.rados.rados.2.smithi175.stdout:346: finishing write tid 2 to smithi17557870-10 2024-09-11T03:39:35.929 INFO:tasks.rados.rados.2.smithi175.stdout:346: finishing write tid 3 to smithi17557870-10 2024-09-11T03:39:35.929 INFO:tasks.rados.rados.2.smithi175.stdout:346: finishing write tid 4 to smithi17557870-10 2024-09-11T03:39:35.929 INFO:tasks.rados.rados.2.smithi175.stdout:346: oid 10 updating version 0 to 94 2024-09-11T03:39:35.929 INFO:tasks.rados.rados.2.smithi175.stdout:346: oid 10 updating version 94 to 95 2024-09-11T03:39:35.929 INFO:tasks.rados.rados.2.smithi175.stdout:346: oid 10 updating version 95 to 96 2024-09-11T03:39:35.929 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 96 (ObjNum 147 snap 32 seq_num 147) dirty exists 2024-09-11T03:39:35.929 INFO:tasks.rados.rados.2.smithi175.stdout:346: left oid 10 (ObjNum 147 snap 32 seq_num 147) 2024-09-11T03:39:35.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:35 smithi175 ceph-mon[30894]: pgmap v391: 17 pgs: 17 active+clean; 113 MiB data, 528 MiB used, 983 GiB / 983 GiB avail; 4.6 MiB/s rd, 972 KiB/s wr, 6 op/s 2024-09-11T03:39:35.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:35 smithi175 ceph-mon[30894]: osdmap e158: 12 total, 12 up, 11 in 2024-09-11T03:39:35.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:35 smithi175 ceph-mon[30894]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:35.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:36.018 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:36.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:36.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:36.104 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:39:36.105 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:39:36.120 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:36.141 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:36.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:35 smithi084 ceph-mon[27254]: pgmap v391: 17 pgs: 17 active+clean; 113 MiB data, 528 MiB used, 983 GiB / 983 GiB avail; 4.6 MiB/s rd, 972 KiB/s wr, 6 op/s 2024-09-11T03:39:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:35 smithi084 ceph-mon[27254]: osdmap e158: 12 total, 12 up, 11 in 2024-09-11T03:39:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:35 smithi084 ceph-mon[27254]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:36.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:35 smithi121 ceph-mon[30729]: pgmap v391: 17 pgs: 17 active+clean; 113 MiB data, 528 MiB used, 983 GiB / 983 GiB avail; 4.6 MiB/s rd, 972 KiB/s wr, 6 op/s 2024-09-11T03:39:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:35 smithi121 ceph-mon[30729]: osdmap e158: 12 total, 12 up, 11 in 2024-09-11T03:39:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:35 smithi121 ceph-mon[30729]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:36.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:36.269 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:36.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:36.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:36.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:36.393 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:36.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:36.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:36.495 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:36.516 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:36.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:36.517 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:36.618 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:36.638 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:36.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:36.639 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:36.662 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-11T03:39:36.663 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd test-reweight-by-utilization 110 1.0 2024-09-11T03:39:36.740 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:36.761 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-11T03:39:36.763 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:36.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:36.764 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:36.820 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:39:36.822 INFO:tasks.rados.rados.2.smithi175.stdout:345: done (2 left) 2024-09-11T03:39:36.823 INFO:tasks.rados.rados.2.smithi175.stdout:346: done (1 left) 2024-09-11T03:39:36.823 INFO:tasks.rados.rados.2.smithi175.stdout:347: done (0 left) 2024-09-11T03:39:36.823 INFO:tasks.rados.rados.2.smithi175.stdout:348: read oid 36 snap -1 2024-09-11T03:39:36.823 INFO:tasks.rados.rados.2.smithi175.stdout:348: expect (ObjNum 144 snap 31 seq_num 144) 2024-09-11T03:39:36.823 INFO:tasks.rados.rados.2.smithi175.stdout:349: write oid 43 current snap is 33 2024-09-11T03:39:36.823 INFO:tasks.rados.rados.2.smithi175.stdout:349: seq_num 148 ranges {487436=547559,1760982=657035} 2024-09-11T03:39:36.829 INFO:tasks.rados.rados.2.smithi175.stdout:349: writing smithi17557870-43 from 487436 to 1034995 tid 1 2024-09-11T03:39:36.838 INFO:tasks.rados.rados.2.smithi175.stdout:349: writing smithi17557870-43 from 1760982 to 2418017 tid 2 2024-09-11T03:39:36.838 INFO:tasks.rados.rados.2.smithi175.stdout:350: snap_create 2024-09-11T03:39:36.838 INFO:tasks.rados.rados.2.smithi175.stdout:349: finishing write tid 1 to smithi17557870-43 2024-09-11T03:39:36.838 INFO:tasks.rados.rados.2.smithi175.stdout:348: expect (ObjNum 144 snap 31 seq_num 144) 2024-09-11T03:39:36.865 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:36.887 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:36.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:36.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:36.946 INFO:tasks.rados.rados.2.smithi175.stdout:349: finishing write tid 2 to smithi17557870-43 2024-09-11T03:39:36.946 INFO:tasks.rados.rados.2.smithi175.stdout:349: finishing write tid 3 to smithi17557870-43 2024-09-11T03:39:36.947 INFO:tasks.rados.rados.2.smithi175.stdout:349: finishing write tid 4 to smithi17557870-43 2024-09-11T03:39:36.947 INFO:tasks.rados.rados.2.smithi175.stdout:349: oid 43 updating version 0 to 53 2024-09-11T03:39:36.947 INFO:tasks.rados.rados.2.smithi175.stdout:349: oid 43 updating version 53 to 55 2024-09-11T03:39:36.947 INFO:tasks.rados.rados.2.smithi175.stdout:349: oid 43 version 55 is already newer than 54 2024-09-11T03:39:36.947 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 55 (ObjNum 148 snap 33 seq_num 148) dirty exists 2024-09-11T03:39:36.947 INFO:tasks.rados.rados.2.smithi175.stdout:349: left oid 43 (ObjNum 148 snap 33 seq_num 148) 2024-09-11T03:39:36.989 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:37.012 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:37.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:37.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:37.015 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:39:37.114 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:37.135 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:37.136 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:36 smithi175 ceph-mon[30894]: osdmap e159: 12 total, 12 up, 11 in 2024-09-11T03:39:37.136 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4136589921' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:37.136 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:36 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:39:37.136 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:36 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:39:37.136 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4136589921' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:37.136 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:36 smithi175 ceph-mon[30894]: osdmap e160: 12 total, 12 up, 11 in 2024-09-11T03:39:37.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:37.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:37.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:36 smithi121 ceph-mon[30729]: osdmap e159: 12 total, 12 up, 11 in 2024-09-11T03:39:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4136589921' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:36 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:39:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:36 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:39:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4136589921' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:36 smithi121 ceph-mon[30729]: osdmap e160: 12 total, 12 up, 11 in 2024-09-11T03:39:37.238 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:37.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:37.259 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:37.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:37.360 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:37.381 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:36 smithi084 ceph-mon[27254]: osdmap e159: 12 total, 12 up, 11 in 2024-09-11T03:39:37.381 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4136589921' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:37.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:36 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:39:37.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:36 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:39:37.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4136589921' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:37.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:36 smithi084 ceph-mon[27254]: osdmap e160: 12 total, 12 up, 11 in 2024-09-11T03:39:37.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:37.383 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:37.383 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:37.429 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:37.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:37.484 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:39:37.484 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:39:37.505 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:37.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:37.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:37.607 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:37.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:37.634 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:37.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:37.702 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:39:37.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:37.771 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:37.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:37.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:37.825 INFO:tasks.rados.rados.2.smithi175.stdout:348: done (2 left) 2024-09-11T03:39:37.825 INFO:tasks.rados.rados.2.smithi175.stdout:349: done (1 left) 2024-09-11T03:39:37.825 INFO:tasks.rados.rados.2.smithi175.stdout:350: done (0 left) 2024-09-11T03:39:37.825 INFO:tasks.rados.rados.2.smithi175.stdout:351: copy_from oid 10 from oid 35 current snap is 34 2024-09-11T03:39:37.825 INFO:tasks.rados.rados.2.smithi175.stdout:352: snap_remove snap 31 2024-09-11T03:39:37.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:37.897 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:37.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:37.962 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:38.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:38.025 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:38.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:38.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:38.080 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:37 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:39:38.080 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:37 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:39:38.080 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:37 smithi175 ceph-mon[30894]: pgmap v394: 17 pgs: 17 active+clean; 119 MiB data, 539 MiB used, 983 GiB / 983 GiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 6 op/s 2024-09-11T03:39:38.080 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:37 smithi175 ceph-mon[30894]: osdmap e161: 12 total, 12 up, 11 in 2024-09-11T03:39:38.082 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:37 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:39:38.082 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:37 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:39:38.082 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:37 smithi084 ceph-mon[27254]: pgmap v394: 17 pgs: 17 active+clean; 119 MiB data, 539 MiB used, 983 GiB / 983 GiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 6 op/s 2024-09-11T03:39:38.082 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:37 smithi084 ceph-mon[27254]: osdmap e161: 12 total, 12 up, 11 in 2024-09-11T03:39:38.084 INFO:teuthology.orchestra.run.smithi084.stdout:moved 2 / 51 (3.92157%) 2024-09-11T03:39:38.084 INFO:teuthology.orchestra.run.smithi084.stdout:avg 4.63636 2024-09-11T03:39:38.084 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 1.72008 -> 1.61091 (expected baseline 2.05302) 2024-09-11T03:39:38.084 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.5 with 2 -> 3 pgs (0.431373 -> 0.647059 * mean) 2024-09-11T03:39:38.084 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.9 with 8 -> 7 pgs (1.72549 -> 1.5098 * mean) 2024-09-11T03:39:38.084 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:39:38.084 INFO:teuthology.orchestra.run.smithi084.stdout:oload 110 2024-09-11T03:39:38.084 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 1 2024-09-11T03:39:38.085 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2024-09-11T03:39:38.085 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0005 2024-09-11T03:39:38.085 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0006 2024-09-11T03:39:38.085 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8 weight 1.0000 -> 0.8249 2024-09-11T03:39:38.085 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9 weight 1.0000 -> 0.9081 2024-09-11T03:39:38.085 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3 weight 1.0000 -> 0.8234 2024-09-11T03:39:38.085 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0 weight 1.0000 -> 0.8569 2024-09-11T03:39:38.085 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2024-09-11T03:39:38.109 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-11T03:39:38.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:38.149 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:38.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:38.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:38.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:37 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:39:38.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:37 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:39:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:37 smithi121 ceph-mon[30729]: pgmap v394: 17 pgs: 17 active+clean; 119 MiB data, 539 MiB used, 983 GiB / 983 GiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 6 op/s 2024-09-11T03:39:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:37 smithi121 ceph-mon[30729]: osdmap e161: 12 total, 12 up, 11 in 2024-09-11T03:39:38.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:38.274 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:38.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:38.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:38.353 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:39:38 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: 2024-09-11T03:39:38.083+0000 7fab40ce9640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-09-11T03:39:38.354 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:39:38.375 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:38.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:38.398 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:38.398 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:38.500 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:38.521 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:38.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:38.522 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:38.569 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:39:38.569 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:39:38.588 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:39:38.588 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:39:38.618 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:39:38.618 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:39:38.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:38.643 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:39:38.643 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:39:38.646 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:38.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:38.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:38.662 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:39:38.662 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:39:38.679 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:39:38.679 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:39:38.695 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:39:38.696 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:39:38.712 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:39:38.712 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:39:38.729 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:39:38.729 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:39:38.746 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:39:38.746 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:39:38.747 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:38.763 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:39:38.763 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:39:38.772 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:38.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:38.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:38.785 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:39:38.785 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:39:38.827 INFO:tasks.rados.rados.2.smithi175.stdout:352: done (1 left) 2024-09-11T03:39:38.827 INFO:tasks.rados.rados.2.smithi175.stdout:353: snap_create 2024-09-11T03:39:38.827 INFO:tasks.rados.rados.2.smithi175.stdout:351: finishing copy_from racing read to smithi17557870-10 2024-09-11T03:39:38.827 INFO:tasks.rados.rados.2.smithi175.stdout:351: finishing copy_from to smithi17557870-10 2024-09-11T03:39:38.827 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T03:39:38.855 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:39:38.855 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:39:38.855 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:39:38.856 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:39:38.856 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.856 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:366 2.7 e114/95 2:e0eabcfb:::smithi17557870-9:head v 114'24, mlcod=112'22)", 2024-09-11T03:39:38.856 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.856154+0000", 2024-09-11T03:39:38.856 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 47.998255524000001, 2024-09-11T03:39:38.856 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025307979999999999, 2024-09-11T03:39:38.856 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.856 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.856 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.856 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.856 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.856 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.856 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.856154+0000", 2024-09-11T03:39:38.857 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.857 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.857 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.857 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.856154+0000", 2024-09-11T03:39:38.857 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.857 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.857 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.857 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.856154+0000", 2024-09-11T03:39:38.857 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1e-07 2024-09-11T03:39:38.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.858 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.858 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.858 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.857067+0000", 2024-09-11T03:39:38.858 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00091319100000000005 2024-09-11T03:39:38.858 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.858 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.858 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.858 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.857088+0000", 2024-09-11T03:39:38.858 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0264000000000001e-05 2024-09-11T03:39:38.858 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.858 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.858 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.858 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.857101+0000", 2024-09-11T03:39:38.859 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3645000000000001e-05 2024-09-11T03:39:38.859 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.859 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.859 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.859 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.857189+0000", 2024-09-11T03:39:38.859 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8090000000000005e-05 2024-09-11T03:39:38.859 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.859 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.859 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.859 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.857219+0000", 2024-09-11T03:39:38.859 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9893999999999999e-05 2024-09-11T03:39:38.859 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.859 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.860 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.860 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.858638+0000", 2024-09-11T03:39:38.860 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001419182 2024-09-11T03:39:38.860 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.860 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.860 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.860 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.858685+0000", 2024-09-11T03:39:38.860 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6322000000000001e-05 2024-09-11T03:39:38.860 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.860 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.860 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.860 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.860 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.861 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:367 2.7 e114/95 2:e0eabcfb:::smithi17557870-9:head v 114'25, mlcod=112'23)", 2024-09-11T03:39:38.861 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.863256+0000", 2024-09-11T03:39:38.861 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 47.991154004999999, 2024-09-11T03:39:38.861 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0020204070000000001, 2024-09-11T03:39:38.861 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.861 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.861 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.861 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.861 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.861 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.861 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.863256+0000", 2024-09-11T03:39:38.861 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.861 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.861 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.862 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.862 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.863256+0000", 2024-09-11T03:39:38.862 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.862 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.862 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.862 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.862 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.863257+0000", 2024-09-11T03:39:38.862 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.474e-06 2024-09-11T03:39:38.862 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.862 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.862 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.862 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.864197+0000", 2024-09-11T03:39:38.862 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00094002199999999999 2024-09-11T03:39:38.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.863 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.863 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.863 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.864216+0000", 2024-09-11T03:39:38.863 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9054000000000001e-05 2024-09-11T03:39:38.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.863 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.863 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.863 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.864230+0000", 2024-09-11T03:39:38.863 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3495e-05 2024-09-11T03:39:38.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.863 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.863 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.863 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.864329+0000", 2024-09-11T03:39:38.864 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9550999999999996e-05 2024-09-11T03:39:38.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.864 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.864 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.864 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.864349+0000", 2024-09-11T03:39:38.864 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9704999999999999e-05 2024-09-11T03:39:38.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.864 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.864 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.864 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.865236+0000", 2024-09-11T03:39:38.864 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00088712199999999995 2024-09-11T03:39:38.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.864 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.864 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.865 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.865276+0000", 2024-09-11T03:39:38.865 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9984000000000002e-05 2024-09-11T03:39:38.865 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.865 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.865 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.865 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.865 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.865 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:368 2.7 e114/95 2:e0eabcfb:::smithi17557870-9:head v 114'26, mlcod=114'24)", 2024-09-11T03:39:38.865 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.868582+0000", 2024-09-11T03:39:38.865 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 47.985827483999998, 2024-09-11T03:39:38.865 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001297798, 2024-09-11T03:39:38.865 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.866 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.866 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.866 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.866 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.866 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.866 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.868582+0000", 2024-09-11T03:39:38.866 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.866 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.866 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.866 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.866 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.868582+0000", 2024-09-11T03:39:38.866 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.866 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.866 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.867 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.867 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.868583+0000", 2024-09-11T03:39:38.867 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.017e-06 2024-09-11T03:39:38.867 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.867 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.867 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.867 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.869186+0000", 2024-09-11T03:39:38.867 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00060262499999999997 2024-09-11T03:39:38.867 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.867 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.867 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.867 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.869201+0000", 2024-09-11T03:39:38.868 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5234e-05 2024-09-11T03:39:38.868 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.868 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.868 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.868 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.869209+0000", 2024-09-11T03:39:38.868 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2700000000000004e-06 2024-09-11T03:39:38.868 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.868 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.868 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.868 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.869262+0000", 2024-09-11T03:39:38.868 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2371999999999999e-05 2024-09-11T03:39:38.868 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.868 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.869 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.869 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.869279+0000", 2024-09-11T03:39:38.869 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7883000000000001e-05 2024-09-11T03:39:38.869 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.869 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.869 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.869 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.869844+0000", 2024-09-11T03:39:38.869 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00056486000000000004 2024-09-11T03:39:38.869 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.869 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.869 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.869 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.869880+0000", 2024-09-11T03:39:38.869 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5537000000000002e-05 2024-09-11T03:39:38.869 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.870 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.870 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.870 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.870 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.870 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:386 2.0 e114/95 2:0762776a:::smithi17557870-25:head v 114'11, mlcod=113'8)", 2024-09-11T03:39:38.870 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.988307+0000", 2024-09-11T03:39:38.870 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 47.866102572999999, 2024-09-11T03:39:38.870 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025481689999999999, 2024-09-11T03:39:38.870 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.870 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.871 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.871 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.871 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.871 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.988307+0000", 2024-09-11T03:39:38.871 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.872 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.872 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.988307+0000", 2024-09-11T03:39:38.872 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.872 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.872 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.988309+0000", 2024-09-11T03:39:38.872 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9089999999999998e-06 2024-09-11T03:39:38.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.873 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.873 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.873 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.989577+0000", 2024-09-11T03:39:38.873 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012685940000000001 2024-09-11T03:39:38.873 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.873 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.873 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.873 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.989603+0000", 2024-09-11T03:39:38.873 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5398999999999999e-05 2024-09-11T03:39:38.873 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.873 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.873 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.873 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.989613+0000", 2024-09-11T03:39:38.874 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0519e-05 2024-09-11T03:39:38.874 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.874 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.874 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.874 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.989691+0000", 2024-09-11T03:39:38.874 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8112e-05 2024-09-11T03:39:38.874 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.874 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.874 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.874 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.989724+0000", 2024-09-11T03:39:38.874 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.2438999999999999e-05 2024-09-11T03:39:38.874 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.874 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.874 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.875 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.990816+0000", 2024-09-11T03:39:38.875 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001091854 2024-09-11T03:39:38.875 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.875 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.875 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.875 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.990855+0000", 2024-09-11T03:39:38.875 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9342999999999999e-05 2024-09-11T03:39:38.875 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.875 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.875 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.875 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.875 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.875 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:387 2.0 e114/95 2:0762776a:::smithi17557870-25:head v 114'12, mlcod=114'10)", 2024-09-11T03:39:38.875 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.995117+0000", 2024-09-11T03:39:38.876 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 47.859292732999997, 2024-09-11T03:39:38.876 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023580089999999999, 2024-09-11T03:39:38.876 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.876 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.876 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.876 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.876 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.876 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.876 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.995117+0000", 2024-09-11T03:39:38.876 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.876 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.876 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.876 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.877 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.995117+0000", 2024-09-11T03:39:38.877 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.877 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.877 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.877 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.995118+0000", 2024-09-11T03:39:38.877 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4900000000000005e-07 2024-09-11T03:39:38.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.877 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.877 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.877 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.996445+0000", 2024-09-11T03:39:38.877 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001327607 2024-09-11T03:39:38.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.878 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.878 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.996460+0000", 2024-09-11T03:39:38.878 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4572e-05 2024-09-11T03:39:38.878 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.878 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.878 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.996467+0000", 2024-09-11T03:39:38.878 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2760000000000003e-06 2024-09-11T03:39:38.878 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.878 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.878 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.996529+0000", 2024-09-11T03:39:38.878 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1883000000000004e-05 2024-09-11T03:39:38.879 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.879 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.879 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.879 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.996552+0000", 2024-09-11T03:39:38.879 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3258e-05 2024-09-11T03:39:38.879 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.879 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.879 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.879 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.997444+0000", 2024-09-11T03:39:38.879 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00089145800000000003 2024-09-11T03:39:38.879 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.879 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.879 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.880 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.997475+0000", 2024-09-11T03:39:38.880 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1205999999999999e-05 2024-09-11T03:39:38.880 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.880 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.880 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.880 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.880 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.880 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:402 2.7 e114/95 2:e23d1762:::smithi17557870-21:head v 114'28, mlcod=114'26)", 2024-09-11T03:39:38.880 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:51.091054+0000", 2024-09-11T03:39:38.880 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 47.763355021999999, 2024-09-11T03:39:38.880 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0062707429999999996, 2024-09-11T03:39:38.880 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.880 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.880 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.881 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.881 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.881 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.881 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.091054+0000", 2024-09-11T03:39:38.881 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.881 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.881 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.881 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.881 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.091054+0000", 2024-09-11T03:39:38.881 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.881 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.881 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.881 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.882 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.091056+0000", 2024-09-11T03:39:38.882 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.128e-06 2024-09-11T03:39:38.882 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.882 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.882 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.882 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.095985+0000", 2024-09-11T03:39:38.882 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0049292549999999996 2024-09-11T03:39:38.882 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.882 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.882 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.882 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.096004+0000", 2024-09-11T03:39:38.882 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8859e-05 2024-09-11T03:39:38.882 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.882 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.883 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.883 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.096013+0000", 2024-09-11T03:39:38.883 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6169999999999996e-06 2024-09-11T03:39:38.883 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.883 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.883 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.883 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.096101+0000", 2024-09-11T03:39:38.883 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7331999999999997e-05 2024-09-11T03:39:38.883 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.883 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.883 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.883 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.096120+0000", 2024-09-11T03:39:38.884 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9783999999999999e-05 2024-09-11T03:39:38.884 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.884 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.885 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.885 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.097285+0000", 2024-09-11T03:39:38.885 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001164237 2024-09-11T03:39:38.885 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.885 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.885 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.885 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.097325+0000", 2024-09-11T03:39:38.885 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0531000000000002e-05 2024-09-11T03:39:38.885 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.885 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.885 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.885 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.885 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.886 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:411 2.4 e117/95 2:220bb244:::smithi17557870-34:head v 117'11, mlcod=100'8)", 2024-09-11T03:39:38.886 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:53.824960+0000", 2024-09-11T03:39:38.886 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 45.029449247000002, 2024-09-11T03:39:38.886 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0019175419999999999, 2024-09-11T03:39:38.886 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.886 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.886 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.886 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.886 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.886 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.886 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.824960+0000", 2024-09-11T03:39:38.886 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.886 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.887 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.887 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.887 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.824960+0000", 2024-09-11T03:39:38.887 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.887 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.887 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.887 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.887 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.824960+0000", 2024-09-11T03:39:38.887 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8199999999999999e-07 2024-09-11T03:39:38.887 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.887 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.887 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.888 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.825676+0000", 2024-09-11T03:39:38.888 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00071519599999999997 2024-09-11T03:39:38.888 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.888 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.888 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.888 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.825709+0000", 2024-09-11T03:39:38.888 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3052999999999997e-05 2024-09-11T03:39:38.888 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.888 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.888 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.888 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.825725+0000", 2024-09-11T03:39:38.888 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6501999999999999e-05 2024-09-11T03:39:38.888 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.888 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.889 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.889 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.825805+0000", 2024-09-11T03:39:38.889 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9555000000000007e-05 2024-09-11T03:39:38.889 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.889 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.889 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.889 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.825821+0000", 2024-09-11T03:39:38.889 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6441000000000001e-05 2024-09-11T03:39:38.889 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.889 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.889 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.889 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.826837+0000", 2024-09-11T03:39:38.889 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0010155909999999999 2024-09-11T03:39:38.890 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.890 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.890 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.890 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.826878+0000", 2024-09-11T03:39:38.890 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.1022000000000001e-05 2024-09-11T03:39:38.890 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.890 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.890 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.890 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.890 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.890 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:413 2.4 e117/95 2:220bb244:::smithi17557870-34:head v 117'13, mlcod=117'11)", 2024-09-11T03:39:38.890 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:53.834055+0000", 2024-09-11T03:39:38.890 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 45.020354752999999, 2024-09-11T03:39:38.891 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0017130870000000001, 2024-09-11T03:39:38.891 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.891 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.891 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.891 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.891 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.891 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.834055+0000", 2024-09-11T03:39:38.891 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.891 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.891 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.834055+0000", 2024-09-11T03:39:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.834056+0000", 2024-09-11T03:39:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.389e-06 2024-09-11T03:39:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.834461+0000", 2024-09-11T03:39:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00040453700000000001 2024-09-11T03:39:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.893 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.834478+0000", 2024-09-11T03:39:38.893 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6925000000000001e-05 2024-09-11T03:39:38.893 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.893 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.893 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.834487+0000", 2024-09-11T03:39:38.893 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3440000000000007e-06 2024-09-11T03:39:38.893 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.893 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.893 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.834558+0000", 2024-09-11T03:39:38.893 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0673000000000006e-05 2024-09-11T03:39:38.894 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.894 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.894 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.894 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.834584+0000", 2024-09-11T03:39:38.894 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6642e-05 2024-09-11T03:39:38.894 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.894 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.894 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.894 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.835713+0000", 2024-09-11T03:39:38.894 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001128511 2024-09-11T03:39:38.894 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.894 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.894 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.895 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:53.835768+0000", 2024-09-11T03:39:38.895 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5065999999999998e-05 2024-09-11T03:39:38.895 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.895 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.895 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.895 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.895 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.895 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:602 2.4 e130/95 2:25f5f8b9:::smithi17557870-26:head v 130'38, mlcod=127'35)", 2024-09-11T03:39:38.895 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:06.897612+0000", 2024-09-11T03:39:38.895 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 31.956797065, 2024-09-11T03:39:38.895 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015331050000000001, 2024-09-11T03:39:38.895 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.895 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.896 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.896 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.896 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.896 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.896 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.897612+0000", 2024-09-11T03:39:38.896 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.896 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.896 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.896 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.896 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.897612+0000", 2024-09-11T03:39:38.896 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.896 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.897 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.897 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.897 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.897613+0000", 2024-09-11T03:39:38.897 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1899999999999999e-07 2024-09-11T03:39:38.897 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.897 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.897 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.897 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.898167+0000", 2024-09-11T03:39:38.897 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00055433399999999997 2024-09-11T03:39:38.897 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.897 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.897 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.897 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.898181+0000", 2024-09-11T03:39:38.898 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4358999999999999e-05 2024-09-11T03:39:38.898 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.898 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.898 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.898 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.898192+0000", 2024-09-11T03:39:38.898 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0482999999999999e-05 2024-09-11T03:39:38.898 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.898 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.898 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.898 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.898258+0000", 2024-09-11T03:39:38.898 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6037999999999994e-05 2024-09-11T03:39:38.898 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.898 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.899 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.899 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.898277+0000", 2024-09-11T03:39:38.899 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9528e-05 2024-09-11T03:39:38.899 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.899 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.899 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.899 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.899110+0000", 2024-09-11T03:39:38.899 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00083237000000000003 2024-09-11T03:39:38.899 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.899 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.899 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.899 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.899146+0000", 2024-09-11T03:39:38.899 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5774e-05 2024-09-11T03:39:38.899 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.900 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.900 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.900 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.900 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.900 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:604 2.4 e130/95 2:25f5f8b9:::smithi17557870-26:head v 130'40, mlcod=130'38)", 2024-09-11T03:39:38.900 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:06.908353+0000", 2024-09-11T03:39:38.900 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 31.946056354, 2024-09-11T03:39:38.900 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0020832049999999999, 2024-09-11T03:39:38.900 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.900 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.900 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.900 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.900 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.901 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.901 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.908353+0000", 2024-09-11T03:39:38.901 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.901 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.901 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.901 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.902 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.908353+0000", 2024-09-11T03:39:38.902 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.902 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.902 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.902 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.902 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.908354+0000", 2024-09-11T03:39:38.902 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0699999999999996e-07 2024-09-11T03:39:38.902 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.902 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.902 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.902 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.909652+0000", 2024-09-11T03:39:38.902 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001298177 2024-09-11T03:39:38.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.903 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.903 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.909663+0000", 2024-09-11T03:39:38.903 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0956e-05 2024-09-11T03:39:38.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.903 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.903 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.909668+0000", 2024-09-11T03:39:38.903 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9060000000000001e-06 2024-09-11T03:39:38.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.904 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.904 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.909717+0000", 2024-09-11T03:39:38.904 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9020999999999997e-05 2024-09-11T03:39:38.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.904 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.904 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.909739+0000", 2024-09-11T03:39:38.904 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.234e-05 2024-09-11T03:39:38.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.904 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.904 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.910403+0000", 2024-09-11T03:39:38.904 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000664008 2024-09-11T03:39:38.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.905 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.905 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:06.910436+0000", 2024-09-11T03:39:38.905 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3090000000000003e-05 2024-09-11T03:39:38.905 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.905 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.905 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.905 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:851 2.7 2:e6da0655:::smithi17557870-27:head [write 736173~553965,stat] snapc 1a=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-11T03:39:38.906 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:26.831961+0000", 2024-09-11T03:39:38.906 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 12.022448240999999, 2024-09-11T03:39:38.906 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0048828370000000001, 2024-09-11T03:39:38.906 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.906 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.906 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.906 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:38.906 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:38.906 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:38.906 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 851 2024-09-11T03:39:38.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.906 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.906 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.907 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.907 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.831961+0000", 2024-09-11T03:39:38.907 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.907 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.907 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.831961+0000", 2024-09-11T03:39:38.907 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.907 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.907 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.831963+0000", 2024-09-11T03:39:38.907 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.86e-06 2024-09-11T03:39:38.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.908 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.908 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.832372+0000", 2024-09-11T03:39:38.908 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00040875299999999998 2024-09-11T03:39:38.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.908 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.908 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.832385+0000", 2024-09-11T03:39:38.908 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3244999999999999e-05 2024-09-11T03:39:38.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.909 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.909 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.832397+0000", 2024-09-11T03:39:38.909 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2342e-05 2024-09-11T03:39:38.909 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.909 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.909 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.832498+0000", 2024-09-11T03:39:38.909 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000100732 2024-09-11T03:39:38.909 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.909 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:39:38.909 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.832507+0000", 2024-09-11T03:39:38.909 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.9309999999999997e-06 2024-09-11T03:39:38.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.910 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.910 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.910 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.833851+0000", 2024-09-11T03:39:38.910 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001343894 2024-09-11T03:39:38.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.910 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.910 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.910 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.833959+0000", 2024-09-11T03:39:38.910 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010828 2024-09-11T03:39:38.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.910 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.911 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:39:38.911 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.834088+0000", 2024-09-11T03:39:38.911 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000128639 2024-09-11T03:39:38.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.911 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.911 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:38.911 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.835317+0000", 2024-09-11T03:39:38.911 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012290000000000001 2024-09-11T03:39:38.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.911 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.911 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:38.911 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.836679+0000", 2024-09-11T03:39:38.911 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013623590000000001 2024-09-11T03:39:38.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.912 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:38.912 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.836779+0000", 2024-09-11T03:39:38.912 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9427999999999997e-05 2024-09-11T03:39:38.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.912 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.912 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.836815+0000", 2024-09-11T03:39:38.912 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6057999999999999e-05 2024-09-11T03:39:38.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.913 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.913 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.836844+0000", 2024-09-11T03:39:38.913 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9315999999999998e-05 2024-09-11T03:39:38.913 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.913 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.913 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.913 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:852 2.7 2:e6da0655:::smithi17557870-27:head [write 1867350~468823] snapc 1a=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-11T03:39:38.913 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:26.837175+0000", 2024-09-11T03:39:38.913 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 12.017234881, 2024-09-11T03:39:38.913 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003236287, 2024-09-11T03:39:38.913 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.914 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.914 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.914 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:38.914 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:38.914 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:38.914 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 852 2024-09-11T03:39:38.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.914 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.914 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.914 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.914 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837175+0000", 2024-09-11T03:39:38.914 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.915 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.915 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.915 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837175+0000", 2024-09-11T03:39:38.915 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.915 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.915 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.915 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837176+0000", 2024-09-11T03:39:38.915 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2669999999999999e-06 2024-09-11T03:39:38.915 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.915 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.916 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837474+0000", 2024-09-11T03:39:38.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00029848599999999999 2024-09-11T03:39:38.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.916 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.916 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837486+0000", 2024-09-11T03:39:38.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2009e-05 2024-09-11T03:39:38.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.916 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.916 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837492+0000", 2024-09-11T03:39:38.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8599999999999998e-06 2024-09-11T03:39:38.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837545+0000", 2024-09-11T03:39:38.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3139000000000001e-05 2024-09-11T03:39:38.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837643+0000", 2024-09-11T03:39:38.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.789e-05 2024-09-11T03:39:38.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:39:38.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837812+0000", 2024-09-11T03:39:38.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016848199999999999 2024-09-11T03:39:38.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:38.918 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.839259+0000", 2024-09-11T03:39:38.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001447453 2024-09-11T03:39:38.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:38.918 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840221+0000", 2024-09-11T03:39:38.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00096196400000000005 2024-09-11T03:39:38.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:38.919 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840328+0000", 2024-09-11T03:39:38.919 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000106511 2024-09-11T03:39:38.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.919 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.919 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840367+0000", 2024-09-11T03:39:38.919 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9501e-05 2024-09-11T03:39:38.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.919 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.919 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840411+0000", 2024-09-11T03:39:38.919 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3724999999999999e-05 2024-09-11T03:39:38.919 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.920 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.920 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.920 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.920 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.920 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:853 2.7 2:e6da0655:::smithi17557870-27:head [setxattr _header (57) in=7b,truncate 2336173] snapc 1a=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-11T03:39:38.920 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:26.837507+0000", 2024-09-11T03:39:38.920 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 12.016902384, 2024-09-11T03:39:38.920 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030766439999999999, 2024-09-11T03:39:38.920 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.920 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.920 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.920 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:38.921 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:38.921 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:38.921 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 853 2024-09-11T03:39:38.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.921 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.921 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837507+0000", 2024-09-11T03:39:38.921 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837507+0000", 2024-09-11T03:39:38.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837508+0000", 2024-09-11T03:39:38.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7299999999999995e-07 2024-09-11T03:39:38.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837514+0000", 2024-09-11T03:39:38.923 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7450000000000001e-06 2024-09-11T03:39:38.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.923 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.923 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.923 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837520+0000", 2024-09-11T03:39:38.923 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7490000000000001e-06 2024-09-11T03:39:38.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.923 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.923 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.923 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837524+0000", 2024-09-11T03:39:38.923 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0319999999999997e-06 2024-09-11T03:39:38.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.923 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.924 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.924 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.838759+0000", 2024-09-11T03:39:38.924 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001234726 2024-09-11T03:39:38.924 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.924 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.924 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.924 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.838825+0000", 2024-09-11T03:39:38.924 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6124000000000006e-05 2024-09-11T03:39:38.924 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.924 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.924 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:39:38.924 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.838909+0000", 2024-09-11T03:39:38.924 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4254999999999999e-05 2024-09-11T03:39:38.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.925 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.925 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:38.925 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.839304+0000", 2024-09-11T03:39:38.925 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00039483699999999999 2024-09-11T03:39:38.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.925 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.925 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:38.925 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840463+0000", 2024-09-11T03:39:38.925 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001158493 2024-09-11T03:39:38.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.925 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.926 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:38.926 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840500+0000", 2024-09-11T03:39:38.926 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7020999999999997e-05 2024-09-11T03:39:38.926 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.926 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.926 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840532+0000", 2024-09-11T03:39:38.926 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1810000000000002e-05 2024-09-11T03:39:38.926 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.926 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.926 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840584+0000", 2024-09-11T03:39:38.926 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2179000000000003e-05 2024-09-11T03:39:38.927 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.927 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.927 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.927 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:854 2.7 2:e6da0655:::smithi17557870-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e150)", 2024-09-11T03:39:38.927 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:26.837532+0000", 2024-09-11T03:39:38.927 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 12.016877324999999, 2024-09-11T03:39:38.927 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032582370000000002, 2024-09-11T03:39:38.927 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.927 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.927 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:38.927 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:38.928 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:38.928 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:38.928 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 854 2024-09-11T03:39:38.928 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.928 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.928 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.928 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.928 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837532+0000", 2024-09-11T03:39:38.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.928 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.928 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.928 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.928 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837532+0000", 2024-09-11T03:39:38.929 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.929 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.929 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.929 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837533+0000", 2024-09-11T03:39:38.929 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0800000000000005e-07 2024-09-11T03:39:38.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.929 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.929 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.929 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837535+0000", 2024-09-11T03:39:38.929 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.001e-06 2024-09-11T03:39:38.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.929 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.930 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.930 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837543+0000", 2024-09-11T03:39:38.930 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3080000000000008e-06 2024-09-11T03:39:38.930 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.930 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.930 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.930 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837548+0000", 2024-09-11T03:39:38.930 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6689999999999997e-06 2024-09-11T03:39:38.930 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.930 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.930 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.930 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.839166+0000", 2024-09-11T03:39:38.930 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001618565 2024-09-11T03:39:38.930 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.931 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:39:38.931 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.839246+0000", 2024-09-11T03:39:38.931 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9297999999999999e-05 2024-09-11T03:39:38.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.931 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.931 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840622+0000", 2024-09-11T03:39:38.931 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013767790000000001 2024-09-11T03:39:38.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.931 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.931 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840692+0000", 2024-09-11T03:39:38.931 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0020999999999999e-05 2024-09-11T03:39:38.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.932 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.932 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840790+0000", 2024-09-11T03:39:38.932 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8088e-05 2024-09-11T03:39:38.932 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.932 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.932 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.932 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:855 2.4 e150/95 2:2ef19c88:::smithi17557870-15:head v 150'53, mlcod=135'50)", 2024-09-11T03:39:38.932 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:26.850026+0000", 2024-09-11T03:39:38.932 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 12.004383424, 2024-09-11T03:39:38.933 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00202053, 2024-09-11T03:39:38.933 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.933 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.933 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.933 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.933 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.933 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:38.934 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.934 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.850026+0000", 2024-09-11T03:39:38.935 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.935 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.935 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.850026+0000", 2024-09-11T03:39:38.935 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.935 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.935 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.850026+0000", 2024-09-11T03:39:38.935 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0100000000000001e-07 2024-09-11T03:39:38.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.936 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.850845+0000", 2024-09-11T03:39:38.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000818782 2024-09-11T03:39:38.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.936 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.850857+0000", 2024-09-11T03:39:38.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1843e-05 2024-09-11T03:39:38.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.936 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.850865+0000", 2024-09-11T03:39:38.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2779999999999993e-06 2024-09-11T03:39:38.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.850926+0000", 2024-09-11T03:39:38.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0733000000000003e-05 2024-09-11T03:39:38.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.850947+0000", 2024-09-11T03:39:38.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0726000000000001e-05 2024-09-11T03:39:38.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.852006+0000", 2024-09-11T03:39:38.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0010591279999999999 2024-09-11T03:39:38.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.852047+0000", 2024-09-11T03:39:38.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0738999999999997e-05 2024-09-11T03:39:38.938 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.938 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.938 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.939 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:856 2.4 e150/95 2:2ef19c88:::smithi17557870-15:head v 150'54, mlcod=135'50)", 2024-09-11T03:39:38.939 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:26.853027+0000", 2024-09-11T03:39:38.939 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 12.001382814999999, 2024-09-11T03:39:38.939 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012686900000000001, 2024-09-11T03:39:38.939 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.939 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.939 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.939 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.939 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.939 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.853027+0000", 2024-09-11T03:39:38.939 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.940 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.940 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.853027+0000", 2024-09-11T03:39:38.940 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.940 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.940 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.853027+0000", 2024-09-11T03:39:38.940 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2600000000000002e-07 2024-09-11T03:39:38.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.940 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.853598+0000", 2024-09-11T03:39:38.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00057047499999999998 2024-09-11T03:39:38.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.853608+0000", 2024-09-11T03:39:38.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.059e-05 2024-09-11T03:39:38.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.853614+0000", 2024-09-11T03:39:38.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.614e-06 2024-09-11T03:39:38.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.853655+0000", 2024-09-11T03:39:38.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.1390000000000002e-05 2024-09-11T03:39:38.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.853672+0000", 2024-09-11T03:39:38.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6396999999999999e-05 2024-09-11T03:39:38.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.854263+0000", 2024-09-11T03:39:38.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00059161999999999995 2024-09-11T03:39:38.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.943 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.943 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.854295+0000", 2024-09-11T03:39:38.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1977999999999998e-05 2024-09-11T03:39:38.943 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.943 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.943 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.943 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:898 2.7 2:e23d1762:::smithi17557870-21:head [delete] snapc 20=[20,1f,1e,1d,1c] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e157)", 2024-09-11T03:39:38.943 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:33.891697+0000", 2024-09-11T03:39:38.944 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 4.9627129869999997, 2024-09-11T03:39:38.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022672870000000002, 2024-09-11T03:39:38.944 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.944 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.944 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:38.944 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:38.944 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:38.944 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:38.944 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 898 2024-09-11T03:39:38.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.944 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.945 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.891697+0000", 2024-09-11T03:39:38.945 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.945 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.891697+0000", 2024-09-11T03:39:38.945 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.945 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.891698+0000", 2024-09-11T03:39:38.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8920000000000001e-06 2024-09-11T03:39:38.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.946 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.946 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.891706+0000", 2024-09-11T03:39:38.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4490000000000002e-06 2024-09-11T03:39:38.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.946 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.946 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.891724+0000", 2024-09-11T03:39:38.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8219999999999998e-05 2024-09-11T03:39:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.891732+0000", 2024-09-11T03:39:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3410000000000003e-06 2024-09-11T03:39:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.891799+0000", 2024-09-11T03:39:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6711000000000006e-05 2024-09-11T03:39:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:39:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.891803+0000", 2024-09-11T03:39:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3549999999999998e-06 2024-09-11T03:39:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.892785+0000", 2024-09-11T03:39:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00098169699999999991 2024-09-11T03:39:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.892906+0000", 2024-09-11T03:39:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012052700000000001 2024-09-11T03:39:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.893964+0000", 2024-09-11T03:39:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001058095 2024-09-11T03:39:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:899 2.7 2:e0ec5763:::smithi17557870-2:head [append 0~0] snapc 20=[20,1f,1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-11T03:39:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:33.894175+0000", 2024-09-11T03:39:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 4.9602343730000005, 2024-09-11T03:39:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0021720340000000002, 2024-09-11T03:39:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.950 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.950 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.950 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:38.950 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:38.950 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:38.950 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 899 2024-09-11T03:39:38.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.950 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.950 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.950 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894175+0000", 2024-09-11T03:39:38.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894175+0000", 2024-09-11T03:39:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894177+0000", 2024-09-11T03:39:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6029999999999999e-06 2024-09-11T03:39:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894183+0000", 2024-09-11T03:39:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5889999999999999e-06 2024-09-11T03:39:38.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.952 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.952 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894196+0000", 2024-09-11T03:39:38.952 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3155e-05 2024-09-11T03:39:38.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.952 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.952 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894205+0000", 2024-09-11T03:39:38.952 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1489999999999994e-06 2024-09-11T03:39:38.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.952 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.953 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894264+0000", 2024-09-11T03:39:38.953 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9771000000000001e-05 2024-09-11T03:39:38.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.953 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.953 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.953 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894421+0000", 2024-09-11T03:39:38.953 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015673099999999999 2024-09-11T03:39:38.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.953 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.953 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:39:38.953 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894653+0000", 2024-09-11T03:39:38.953 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00023214700000000001 2024-09-11T03:39:38.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:38.954 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896217+0000", 2024-09-11T03:39:38.954 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015637019999999999 2024-09-11T03:39:38.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:38.954 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896256+0000", 2024-09-11T03:39:38.954 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9187999999999998e-05 2024-09-11T03:39:38.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:38.954 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896272+0000", 2024-09-11T03:39:38.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6013999999999999e-05 2024-09-11T03:39:38.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.955 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.955 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896313+0000", 2024-09-11T03:39:38.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0728e-05 2024-09-11T03:39:38.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.955 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.955 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896347+0000", 2024-09-11T03:39:38.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4257000000000001e-05 2024-09-11T03:39:38.955 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.955 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.956 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.956 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:900 2.7 2:e0ec5763:::smithi17557870-2:head [rollback 20] snapc 20=[20,1f,1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-11T03:39:38.956 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:33.894221+0000", 2024-09-11T03:39:38.956 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 4.9601884930000004, 2024-09-11T03:39:38.956 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022939969999999999, 2024-09-11T03:39:38.956 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.956 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.956 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.956 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:38.956 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:38.956 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:38.956 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 900 2024-09-11T03:39:38.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.957 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.957 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894221+0000", 2024-09-11T03:39:38.957 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.957 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894221+0000", 2024-09-11T03:39:38.957 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.958 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894221+0000", 2024-09-11T03:39:38.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.51e-07 2024-09-11T03:39:38.958 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.958 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.958 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894224+0000", 2024-09-11T03:39:38.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4490000000000002e-06 2024-09-11T03:39:38.958 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.958 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.958 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894230+0000", 2024-09-11T03:39:38.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5359999999999998e-06 2024-09-11T03:39:38.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.959 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.959 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894236+0000", 2024-09-11T03:39:38.959 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1889999999999998e-06 2024-09-11T03:39:38.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.959 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.959 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.895141+0000", 2024-09-11T03:39:38.959 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00090550899999999998 2024-09-11T03:39:38.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.959 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.960 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.895196+0000", 2024-09-11T03:39:38.960 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4657999999999997e-05 2024-09-11T03:39:38.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.960 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:39:38.960 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.895310+0000", 2024-09-11T03:39:38.960 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114274 2024-09-11T03:39:38.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.960 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:38.960 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896394+0000", 2024-09-11T03:39:38.960 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001083908 2024-09-11T03:39:38.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.961 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:38.961 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896445+0000", 2024-09-11T03:39:38.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0793e-05 2024-09-11T03:39:38.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.961 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:38.961 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896457+0000", 2024-09-11T03:39:38.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2061e-05 2024-09-11T03:39:38.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.961 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.961 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896491+0000", 2024-09-11T03:39:38.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4208000000000002e-05 2024-09-11T03:39:38.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.962 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.962 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896515+0000", 2024-09-11T03:39:38.962 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3961e-05 2024-09-11T03:39:38.962 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.962 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.962 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.962 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:901 2.7 2:e0ec5763:::smithi17557870-2:head [append 0~0] snapc 20=[20,1f,1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-11T03:39:38.962 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:33.894258+0000", 2024-09-11T03:39:38.963 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 4.9601514570000003, 2024-09-11T03:39:38.963 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002369674, 2024-09-11T03:39:38.963 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:38.963 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:38.963 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:38.963 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:38.963 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:38.963 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:38.963 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 901 2024-09-11T03:39:38.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.963 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:38.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.963 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:38.964 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894258+0000", 2024-09-11T03:39:38.964 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.964 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.964 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:38.964 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894258+0000", 2024-09-11T03:39:38.964 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:38.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.964 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.964 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:38.964 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894258+0000", 2024-09-11T03:39:38.964 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9000000000000002e-07 2024-09-11T03:39:38.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.965 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.965 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:38.965 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894262+0000", 2024-09-11T03:39:38.965 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0690000000000002e-06 2024-09-11T03:39:38.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.965 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.965 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:38.965 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894269+0000", 2024-09-11T03:39:38.965 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2910000000000002e-06 2024-09-11T03:39:38.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.965 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.965 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:38.966 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894283+0000", 2024-09-11T03:39:38.966 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4022999999999999e-05 2024-09-11T03:39:38.966 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.966 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.966 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:38.966 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.895708+0000", 2024-09-11T03:39:38.966 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014255209999999999 2024-09-11T03:39:38.966 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.966 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.966 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:38.966 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.895812+0000", 2024-09-11T03:39:38.966 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000103643 2024-09-11T03:39:38.967 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.967 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:39:38.967 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.895966+0000", 2024-09-11T03:39:38.967 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000154067 2024-09-11T03:39:38.967 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.967 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:38.967 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896523+0000", 2024-09-11T03:39:38.967 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00055707700000000001 2024-09-11T03:39:38.967 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.967 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:38.968 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896563+0000", 2024-09-11T03:39:38.968 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9473e-05 2024-09-11T03:39:38.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.968 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.968 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:38.968 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896586+0000", 2024-09-11T03:39:38.968 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2900000000000001e-05 2024-09-11T03:39:38.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.968 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.968 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:38.968 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896610+0000", 2024-09-11T03:39:38.968 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4411000000000001e-05 2024-09-11T03:39:38.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:38.968 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:38.969 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:38.969 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896628+0000", 2024-09-11T03:39:38.969 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7808999999999998e-05 2024-09-11T03:39:38.969 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.969 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.969 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.969 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:38.969 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:38.969 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:39:38.991 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:38.992 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:39.057 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-11T03:39:39.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd test-reweight-by-utilization 110 0.05 2024-09-11T03:39:39.093 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:39.119 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:39.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:39.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:39.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:38 smithi121 ceph-mon[30729]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:39.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:38 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1112997728' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:39.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:38 smithi121 ceph-mon[30729]: osdmap e162: 12 total, 12 up, 11 in 2024-09-11T03:39:39.219 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:38 smithi175 ceph-mon[30894]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:39.220 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:38 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1112997728' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:39.220 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:38 smithi175 ceph-mon[30894]: osdmap e162: 12 total, 12 up, 11 in 2024-09-11T03:39:39.220 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:39.241 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:39.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:39.343 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:39.364 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:39.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:39.365 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:39.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:38 smithi084 ceph-mon[27254]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:39.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:38 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1112997728' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:39.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:38 smithi084 ceph-mon[27254]: osdmap e162: 12 total, 12 up, 11 in 2024-09-11T03:39:39.415 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:39:39.466 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:39.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:39.488 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:39.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:39.589 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:39.611 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:39.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:39.667 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-11T03:39:39.713 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:39.734 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:39.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:39.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:39.828 INFO:tasks.rados.rados.2.smithi175.stdout:351: done (1 left) 2024-09-11T03:39:39.828 INFO:tasks.rados.rados.2.smithi175.stdout:353: done (0 left) 2024-09-11T03:39:39.828 INFO:tasks.rados.rados.2.smithi175.stdout:354: rollback oid 14 current snap is 35 2024-09-11T03:39:39.828 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 14 to 32 2024-09-11T03:39:39.828 INFO:tasks.rados.rados.2.smithi175.stdout:355: copy_from oid 7 from oid 42 current snap is 35 2024-09-11T03:39:39.828 INFO:tasks.rados.rados.2.smithi175.stdout:356: write (excl) oid 6 current snap is 35 2024-09-11T03:39:39.829 INFO:tasks.rados.rados.2.smithi175.stdout:356: seq_num 149 ranges {504243=622808,1819464=517046} 2024-09-11T03:39:39.834 INFO:tasks.rados.rados.2.smithi175.stdout:356: writing smithi17557870-6 from 504243 to 1127051 tid 1 2024-09-11T03:39:39.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:39.841 INFO:tasks.rados.rados.2.smithi175.stdout:356: writing smithi17557870-6 from 1819464 to 2336510 tid 2 2024-09-11T03:39:39.841 INFO:tasks.rados.rados.2.smithi175.stdout:357: write oid 36 current snap is 35 2024-09-11T03:39:39.842 INFO:tasks.rados.rados.2.smithi175.stdout:354: finishing rollback tid 1 to smithi17557870-14 2024-09-11T03:39:39.842 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 0 (ObjNum 129 snap 0 seq_num 1745094040) dirty dne 2024-09-11T03:39:39.842 INFO:tasks.rados.rados.2.smithi175.stdout:356: finishing write tid 1 to smithi17557870-6 2024-09-11T03:39:39.842 INFO:tasks.rados.rados.2.smithi175.stdout:355: finishing copy_from racing read to smithi17557870-7 2024-09-11T03:39:39.842 INFO:tasks.rados.rados.2.smithi175.stdout:357: seq_num 150 ranges {521050=698057,1877946=643104} 2024-09-11T03:39:39.842 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:39.849 INFO:tasks.rados.rados.2.smithi175.stdout:357: writing smithi17557870-36 from 521050 to 1219107 tid 1 2024-09-11T03:39:39.859 INFO:tasks.rados.rados.2.smithi175.stdout:357: writing smithi17557870-36 from 1877946 to 2521050 tid 2 2024-09-11T03:39:39.859 INFO:tasks.rados.rados.2.smithi175.stdout:354: done (3 left) 2024-09-11T03:39:39.859 INFO:tasks.rados.rados.2.smithi175.stdout:358: rollback oid 37 current snap is 35 2024-09-11T03:39:39.859 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 37 to 26 2024-09-11T03:39:39.859 INFO:tasks.rados.rados.2.smithi175.stdout:355: finishing copy_from to smithi17557870-7 2024-09-11T03:39:39.859 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 44 (ObjNum 134 snap 22 seq_num 134) dirty exists 2024-09-11T03:39:39.859 INFO:tasks.rados.rados.2.smithi175.stdout:356: finishing write tid 2 to smithi17557870-6 2024-09-11T03:39:39.859 INFO:tasks.rados.rados.2.smithi175.stdout:356: finishing write tid 3 to smithi17557870-6 2024-09-11T03:39:39.859 INFO:tasks.rados.rados.2.smithi175.stdout:356: finishing write tid 4 to smithi17557870-6 2024-09-11T03:39:39.860 INFO:tasks.rados.rados.2.smithi175.stdout:356: oid 6 updating version 0 to 122 2024-09-11T03:39:39.860 INFO:tasks.rados.rados.2.smithi175.stdout:356: oid 6 updating version 122 to 123 2024-09-11T03:39:39.860 INFO:tasks.rados.rados.2.smithi175.stdout:356: oid 6 version 123 is already newer than 121 2024-09-11T03:39:39.860 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 123 (ObjNum 149 snap 35 seq_num 149) dirty exists 2024-09-11T03:39:39.860 INFO:tasks.rados.rados.2.smithi175.stdout:356: left oid 6 (ObjNum 149 snap 35 seq_num 149) 2024-09-11T03:39:39.860 INFO:tasks.rados.rados.2.smithi175.stdout:357: finishing write tid 1 to smithi17557870-36 2024-09-11T03:39:39.860 INFO:tasks.rados.rados.2.smithi175.stdout:355: done (3 left) 2024-09-11T03:39:39.860 INFO:tasks.rados.rados.2.smithi175.stdout:356: done (2 left) 2024-09-11T03:39:39.860 INFO:tasks.rados.rados.2.smithi175.stdout:359: delete oid 20 current snap is 35 2024-09-11T03:39:39.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:39.861 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:39.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:39.862 INFO:tasks.rados.rados.2.smithi175.stdout:359: done (2 left) 2024-09-11T03:39:39.862 INFO:tasks.rados.rados.2.smithi175.stdout:360: read oid 40 snap -1 2024-09-11T03:39:39.862 INFO:tasks.rados.rados.2.smithi175.stdout:360: expect deleted 2024-09-11T03:39:39.862 INFO:tasks.rados.rados.2.smithi175.stdout:361: write (excl) oid 5 current snap is 35 2024-09-11T03:39:39.862 INFO:tasks.rados.rados.2.smithi175.stdout:361: seq_num 152 ranges {554664=448555,1478557=664373} 2024-09-11T03:39:39.866 INFO:tasks.rados.rados.2.smithi175.stdout:361: writing smithi17557870-5 from 554664 to 1003219 tid 1 2024-09-11T03:39:39.873 INFO:tasks.rados.rados.2.smithi175.stdout:361: writing smithi17557870-5 from 1478557 to 2142930 tid 2 2024-09-11T03:39:39.873 INFO:tasks.rados.rados.2.smithi175.stdout:362: write (excl) oid 16 current snap is 35 2024-09-11T03:39:39.873 INFO:tasks.rados.rados.2.smithi175.stdout:362: seq_num 153 ranges {571471=640157,1653392=524384} 2024-09-11T03:39:39.879 INFO:tasks.rados.rados.2.smithi175.stdout:362: writing smithi17557870-16 from 571471 to 1211628 tid 1 2024-09-11T03:39:39.885 INFO:tasks.rados.rados.2.smithi175.stdout:362: writing smithi17557870-16 from 1653392 to 2177776 tid 2 2024-09-11T03:39:39.885 INFO:tasks.rados.rados.2.smithi175.stdout:363: write (excl) oid 23 current snap is 35 2024-09-11T03:39:39.885 INFO:tasks.rados.rados.2.smithi175.stdout:363: seq_num 154 ranges {588278=715406,1711874=668042} 2024-09-11T03:39:39.892 INFO:tasks.rados.rados.2.smithi175.stdout:363: writing smithi17557870-23 from 588278 to 1303684 tid 1 2024-09-11T03:39:39.899 INFO:tasks.rados.rados.2.smithi175.stdout:363: writing smithi17557870-23 from 1711874 to 2379916 tid 2 2024-09-11T03:39:39.899 INFO:tasks.rados.rados.2.smithi175.stdout:364: write (excl) oid 46 current snap is 35 2024-09-11T03:39:39.899 INFO:tasks.rados.rados.2.smithi175.stdout:358: finishing rollback tid 1 to smithi17557870-37 2024-09-11T03:39:39.899 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 0 (ObjNum 209 snap 0 seq_num 197503216) dirty dne 2024-09-11T03:39:39.899 INFO:tasks.rados.rados.2.smithi175.stdout:357: finishing write tid 2 to smithi17557870-36 2024-09-11T03:39:39.899 INFO:tasks.rados.rados.2.smithi175.stdout:357: finishing write tid 3 to smithi17557870-36 2024-09-11T03:39:39.899 INFO:tasks.rados.rados.2.smithi175.stdout:364: seq_num 155 ranges {605085=790655,2170356=434729} 2024-09-11T03:39:39.906 INFO:tasks.rados.rados.2.smithi175.stdout:364: writing smithi17557870-46 from 605085 to 1395740 tid 1 2024-09-11T03:39:39.914 INFO:tasks.rados.rados.2.smithi175.stdout:364: writing smithi17557870-46 from 2170356 to 2605085 tid 2 2024-09-11T03:39:39.915 INFO:tasks.rados.rados.2.smithi175.stdout:358: done (6 left) 2024-09-11T03:39:39.915 INFO:tasks.rados.rados.2.smithi175.stdout:360: done (5 left) 2024-09-11T03:39:39.915 INFO:tasks.rados.rados.2.smithi175.stdout:365: read oid 31 snap -1 2024-09-11T03:39:39.915 INFO:tasks.rados.rados.2.smithi175.stdout:365: expect deleted 2024-09-11T03:39:39.915 INFO:tasks.rados.rados.2.smithi175.stdout:366: rollback oid 19 current snap is 35 2024-09-11T03:39:39.915 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 19 to 32 2024-09-11T03:39:39.915 INFO:tasks.rados.rados.2.smithi175.stdout:357: finishing write tid 4 to smithi17557870-36 2024-09-11T03:39:39.915 INFO:tasks.rados.rados.2.smithi175.stdout:357: oid 36 updating version 0 to 46 2024-09-11T03:39:39.915 INFO:tasks.rados.rados.2.smithi175.stdout:357: oid 36 updating version 46 to 47 2024-09-11T03:39:39.915 INFO:tasks.rados.rados.2.smithi175.stdout:357: oid 36 version 47 is already newer than 45 2024-09-11T03:39:39.915 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 47 (ObjNum 150 snap 35 seq_num 150) dirty exists 2024-09-11T03:39:39.915 INFO:tasks.rados.rados.2.smithi175.stdout:357: left oid 36 (ObjNum 150 snap 35 seq_num 150) 2024-09-11T03:39:39.915 INFO:tasks.rados.rados.2.smithi175.stdout:361: finishing write tid 1 to smithi17557870-5 2024-09-11T03:39:39.916 INFO:tasks.rados.rados.2.smithi175.stdout:357: done (6 left) 2024-09-11T03:39:39.916 INFO:tasks.rados.rados.2.smithi175.stdout:367: read oid 33 snap -1 2024-09-11T03:39:39.916 INFO:tasks.rados.rados.2.smithi175.stdout:367: expect deleted 2024-09-11T03:39:39.916 INFO:tasks.rados.rados.2.smithi175.stdout:368: write oid 44 current snap is 35 2024-09-11T03:39:39.916 INFO:tasks.rados.rados.2.smithi175.stdout:361: finishing write tid 2 to smithi17557870-5 2024-09-11T03:39:39.916 INFO:tasks.rados.rados.2.smithi175.stdout:361: finishing write tid 3 to smithi17557870-5 2024-09-11T03:39:39.916 INFO:tasks.rados.rados.2.smithi175.stdout:361: finishing write tid 4 to smithi17557870-5 2024-09-11T03:39:39.916 INFO:tasks.rados.rados.2.smithi175.stdout:361: oid 5 updating version 0 to 32 2024-09-11T03:39:39.916 INFO:tasks.rados.rados.2.smithi175.stdout:361: oid 5 version 32 is already newer than 31 2024-09-11T03:39:39.916 INFO:tasks.rados.rados.2.smithi175.stdout:361: oid 5 updating version 32 to 33 2024-09-11T03:39:39.916 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 33 (ObjNum 152 snap 35 seq_num 152) dirty exists 2024-09-11T03:39:39.916 INFO:tasks.rados.rados.2.smithi175.stdout:361: left oid 5 (ObjNum 152 snap 35 seq_num 152) 2024-09-11T03:39:39.916 INFO:tasks.rados.rados.2.smithi175.stdout:362: finishing write tid 1 to smithi17557870-16 2024-09-11T03:39:39.917 INFO:tasks.rados.rados.2.smithi175.stdout:362: finishing write tid 2 to smithi17557870-16 2024-09-11T03:39:39.917 INFO:tasks.rados.rados.2.smithi175.stdout:362: finishing write tid 3 to smithi17557870-16 2024-09-11T03:39:39.917 INFO:tasks.rados.rados.2.smithi175.stdout:362: finishing write tid 4 to smithi17557870-16 2024-09-11T03:39:39.917 INFO:tasks.rados.rados.2.smithi175.stdout:362: oid 16 updating version 0 to 56 2024-09-11T03:39:39.917 INFO:tasks.rados.rados.2.smithi175.stdout:362: oid 16 version 56 is already newer than 55 2024-09-11T03:39:39.917 INFO:tasks.rados.rados.2.smithi175.stdout:362: oid 16 updating version 56 to 57 2024-09-11T03:39:39.917 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 57 (ObjNum 153 snap 35 seq_num 153) dirty exists 2024-09-11T03:39:39.917 INFO:tasks.rados.rados.2.smithi175.stdout:362: left oid 16 (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:39:39.917 INFO:tasks.rados.rados.2.smithi175.stdout:363: finishing write tid 1 to smithi17557870-23 2024-09-11T03:39:39.917 INFO:tasks.rados.rados.2.smithi175.stdout:363: finishing write tid 2 to smithi17557870-23 2024-09-11T03:39:39.917 INFO:tasks.rados.rados.2.smithi175.stdout:363: finishing write tid 3 to smithi17557870-23 2024-09-11T03:39:39.917 INFO:tasks.rados.rados.2.smithi175.stdout:363: finishing write tid 4 to smithi17557870-23 2024-09-11T03:39:39.917 INFO:tasks.rados.rados.2.smithi175.stdout:363: oid 23 updating version 0 to 51 2024-09-11T03:39:39.918 INFO:tasks.rados.rados.2.smithi175.stdout:363: oid 23 version 51 is already newer than 50 2024-09-11T03:39:39.918 INFO:tasks.rados.rados.2.smithi175.stdout:363: oid 23 version 51 is already newer than 49 2024-09-11T03:39:39.918 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 51 (ObjNum 154 snap 35 seq_num 154) dirty exists 2024-09-11T03:39:39.918 INFO:tasks.rados.rados.2.smithi175.stdout:363: left oid 23 (ObjNum 154 snap 35 seq_num 154) 2024-09-11T03:39:39.918 INFO:tasks.rados.rados.2.smithi175.stdout:364: finishing write tid 1 to smithi17557870-46 2024-09-11T03:39:39.918 INFO:tasks.rados.rados.2.smithi175.stdout:368: seq_num 156 ranges {621892=465904,1712485=671711} 2024-09-11T03:39:39.918 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:39 smithi175 ceph-mon[30894]: pgmap v397: 17 pgs: 17 active+clean; 126 MiB data, 550 MiB used, 983 GiB / 983 GiB avail; 4.6 MiB/s rd, 3.3 MiB/s wr, 14 op/s 2024-09-11T03:39:39.918 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:39 smithi175 ceph-mon[30894]: osdmap e163: 12 total, 12 up, 11 in 2024-09-11T03:39:39.920 INFO:tasks.rados.rados.2.smithi175.stdout:368: writing smithi17557870-44 from 621892 to 1087796 tid 1 2024-09-11T03:39:39.927 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:39:39.929 INFO:tasks.rados.rados.2.smithi175.stdout:368: writing smithi17557870-44 from 1712485 to 2384196 tid 2 2024-09-11T03:39:39.929 INFO:tasks.rados.rados.2.smithi175.stdout:361: done (7 left) 2024-09-11T03:39:39.929 INFO:tasks.rados.rados.2.smithi175.stdout:362: done (6 left) 2024-09-11T03:39:39.929 INFO:tasks.rados.rados.2.smithi175.stdout:363: done (5 left) 2024-09-11T03:39:39.929 INFO:tasks.rados.rados.2.smithi175.stdout:369: copy_from oid 41 from oid 13 current snap is 35 2024-09-11T03:39:39.929 INFO:tasks.rados.rados.2.smithi175.stdout:364: finishing write tid 2 to smithi17557870-46 2024-09-11T03:39:39.929 INFO:tasks.rados.rados.2.smithi175.stdout:364: finishing write tid 3 to smithi17557870-46 2024-09-11T03:39:39.929 INFO:tasks.rados.rados.2.smithi175.stdout:366: finishing rollback tid 0 to smithi17557870-19 2024-09-11T03:39:39.929 INFO:tasks.rados.rados.2.smithi175.stdout:364: finishing write tid 4 to smithi17557870-46 2024-09-11T03:39:39.929 INFO:tasks.rados.rados.2.smithi175.stdout:364: oid 46 updating version 0 to 27 2024-09-11T03:39:39.929 INFO:tasks.rados.rados.2.smithi175.stdout:364: oid 46 version 27 is already newer than 26 2024-09-11T03:39:39.929 INFO:tasks.rados.rados.2.smithi175.stdout:364: oid 46 updating version 27 to 28 2024-09-11T03:39:39.929 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 28 (ObjNum 155 snap 35 seq_num 155) dirty exists 2024-09-11T03:39:39.929 INFO:tasks.rados.rados.2.smithi175.stdout:364: left oid 46 (ObjNum 155 snap 35 seq_num 155) 2024-09-11T03:39:39.930 INFO:tasks.rados.rados.2.smithi175.stdout:366: finishing rollback tid 1 to smithi17557870-19 2024-09-11T03:39:39.930 INFO:tasks.rados.rados.2.smithi175.stdout:366: finishing rollback tid 2 to smithi17557870-19 2024-09-11T03:39:39.930 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 85 (ObjNum 101 snap 16 seq_num 101) dirty exists 2024-09-11T03:39:39.930 INFO:tasks.rados.rados.2.smithi175.stdout:368: finishing write tid 1 to smithi17557870-44 2024-09-11T03:39:39.930 INFO:tasks.rados.rados.2.smithi175.stdout:364: done (5 left) 2024-09-11T03:39:39.930 INFO:tasks.rados.rados.2.smithi175.stdout:365: done (4 left) 2024-09-11T03:39:39.930 INFO:tasks.rados.rados.2.smithi175.stdout:366: done (3 left) 2024-09-11T03:39:39.930 INFO:tasks.rados.rados.2.smithi175.stdout:367: done (2 left) 2024-09-11T03:39:39.930 INFO:tasks.rados.rados.2.smithi175.stdout:370: read oid 40 snap -1 2024-09-11T03:39:39.930 INFO:tasks.rados.rados.2.smithi175.stdout:370: expect deleted 2024-09-11T03:39:39.930 INFO:tasks.rados.rados.2.smithi175.stdout:371: write oid 46 current snap is 35 2024-09-11T03:39:39.930 INFO:tasks.rados.rados.2.smithi175.stdout:371: seq_num 157 ranges {638699=541153,1770967=531722} 2024-09-11T03:39:39.935 INFO:tasks.rados.rados.2.smithi175.stdout:371: writing smithi17557870-46 from 638699 to 1179852 tid 1 2024-09-11T03:39:39.941 INFO:tasks.rados.rados.2.smithi175.stdout:371: writing smithi17557870-46 from 1770967 to 2302689 tid 2 2024-09-11T03:39:39.942 INFO:tasks.rados.rados.2.smithi175.stdout:372: delete oid 19 current snap is 35 2024-09-11T03:39:39.942 INFO:tasks.rados.rados.2.smithi175.stdout:368: finishing write tid 2 to smithi17557870-44 2024-09-11T03:39:39.942 INFO:tasks.rados.rados.2.smithi175.stdout:368: finishing write tid 3 to smithi17557870-44 2024-09-11T03:39:39.942 INFO:tasks.rados.rados.2.smithi175.stdout:368: finishing write tid 4 to smithi17557870-44 2024-09-11T03:39:39.942 INFO:tasks.rados.rados.2.smithi175.stdout:368: oid 44 updating version 0 to 49 2024-09-11T03:39:39.942 INFO:tasks.rados.rados.2.smithi175.stdout:368: oid 44 version 49 is already newer than 48 2024-09-11T03:39:39.942 INFO:tasks.rados.rados.2.smithi175.stdout:368: oid 44 version 49 is already newer than 47 2024-09-11T03:39:39.942 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 49 (ObjNum 156 snap 35 seq_num 156) dirty exists 2024-09-11T03:39:39.942 INFO:tasks.rados.rados.2.smithi175.stdout:368: left oid 44 (ObjNum 156 snap 35 seq_num 156) 2024-09-11T03:39:39.942 INFO:tasks.rados.rados.2.smithi175.stdout:369: finishing copy_from racing read to smithi17557870-41 2024-09-11T03:39:39.942 INFO:tasks.rados.rados.2.smithi175.stdout:369: finishing copy_from to smithi17557870-41 2024-09-11T03:39:39.942 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 46 (ObjNum 76 snap 7 seq_num 76) dirty exists 2024-09-11T03:39:39.944 INFO:tasks.rados.rados.2.smithi175.stdout:371: finishing write tid 1 to smithi17557870-46 2024-09-11T03:39:39.944 INFO:tasks.rados.rados.2.smithi175.stdout:368: done (4 left) 2024-09-11T03:39:39.944 INFO:tasks.rados.rados.2.smithi175.stdout:369: done (3 left) 2024-09-11T03:39:39.944 INFO:tasks.rados.rados.2.smithi175.stdout:370: done (2 left) 2024-09-11T03:39:39.944 INFO:tasks.rados.rados.2.smithi175.stdout:372: done (1 left) 2024-09-11T03:39:39.944 INFO:tasks.rados.rados.2.smithi175.stdout:373: delete oid 8 current snap is 35 2024-09-11T03:39:39.944 INFO:tasks.rados.rados.2.smithi175.stdout:371: finishing write tid 2 to smithi17557870-46 2024-09-11T03:39:39.945 INFO:tasks.rados.rados.2.smithi175.stdout:371: finishing write tid 3 to smithi17557870-46 2024-09-11T03:39:39.945 INFO:tasks.rados.rados.2.smithi175.stdout:371: finishing write tid 4 to smithi17557870-46 2024-09-11T03:39:39.945 INFO:tasks.rados.rados.2.smithi175.stdout:371: oid 46 updating version 0 to 29 2024-09-11T03:39:39.946 INFO:tasks.rados.rados.2.smithi175.stdout:371: oid 46 updating version 29 to 30 2024-09-11T03:39:39.946 INFO:tasks.rados.rados.2.smithi175.stdout:371: oid 46 updating version 30 to 31 2024-09-11T03:39:39.946 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 31 (ObjNum 157 snap 35 seq_num 157) dirty exists 2024-09-11T03:39:39.946 INFO:tasks.rados.rados.2.smithi175.stdout:371: left oid 46 (ObjNum 157 snap 35 seq_num 157) 2024-09-11T03:39:39.946 INFO:tasks.rados.rados.2.smithi175.stdout:371: done (1 left) 2024-09-11T03:39:39.946 INFO:tasks.rados.rados.2.smithi175.stdout:373: done (0 left) 2024-09-11T03:39:39.946 INFO:tasks.rados.rados.2.smithi175.stdout:374: rollback oid 12 current snap is 35 2024-09-11T03:39:39.946 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 12 to 29 2024-09-11T03:39:39.946 INFO:tasks.rados.rados.2.smithi175.stdout:375: snap_remove snap 33 2024-09-11T03:39:39.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:39.983 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:39.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:39.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:40.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:40.103 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:39 smithi084 ceph-mon[27254]: pgmap v397: 17 pgs: 17 active+clean; 126 MiB data, 550 MiB used, 983 GiB / 983 GiB avail; 4.6 MiB/s rd, 3.3 MiB/s wr, 14 op/s 2024-09-11T03:39:40.104 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:39 smithi084 ceph-mon[27254]: osdmap e163: 12 total, 12 up, 11 in 2024-09-11T03:39:40.106 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:40.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:40.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:40.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:39 smithi121 ceph-mon[30729]: pgmap v397: 17 pgs: 17 active+clean; 126 MiB data, 550 MiB used, 983 GiB / 983 GiB avail; 4.6 MiB/s rd, 3.3 MiB/s wr, 14 op/s 2024-09-11T03:39:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:39 smithi121 ceph-mon[30729]: osdmap e163: 12 total, 12 up, 11 in 2024-09-11T03:39:40.208 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:40.230 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:40.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:40.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:40.316 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:40.332 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:40.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:40.354 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:40.355 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:40.412 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:39:40.412 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:39:40.412 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:39:40.412 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:39:40.456 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:40.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:40.477 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:40.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:40.578 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:40.599 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:40.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:40.701 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:40.724 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:40.724 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:40.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:40.847 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:40.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:40.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:40.924 INFO:tasks.rados.rados.2.smithi175.stdout:375: done (1 left) 2024-09-11T03:39:40.924 INFO:tasks.rados.rados.2.smithi175.stdout:376: snap_remove snap 26 2024-09-11T03:39:40.924 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:39:40.945 INFO:teuthology.orchestra.run.smithi084.stdout:moved 1 / 51 (1.96078%) 2024-09-11T03:39:40.945 INFO:teuthology.orchestra.run.smithi084.stdout:avg 4.63636 2024-09-11T03:39:40.946 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 1.72008 -> 1.66639 (expected baseline 2.05302) 2024-09-11T03:39:40.946 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.5 with 2 -> 2 pgs (0.431373 -> 0.431373 * mean) 2024-09-11T03:39:40.946 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.9 with 8 -> 7 pgs (1.72549 -> 1.5098 * mean) 2024-09-11T03:39:40.946 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:39:40.946 INFO:teuthology.orchestra.run.smithi084.stdout:oload 110 2024-09-11T03:39:40.946 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 0.05 2024-09-11T03:39:40.946 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2024-09-11T03:39:40.946 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0006 2024-09-11T03:39:40.946 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0006 2024-09-11T03:39:40.946 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9 weight 1.0000 -> 0.9500 2024-09-11T03:39:40.946 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8 weight 1.0000 -> 0.9500 2024-09-11T03:39:40.946 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3 weight 1.0000 -> 0.9500 2024-09-11T03:39:40.946 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4 weight 1.0000 -> 0.9500 2024-09-11T03:39:40.947 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2024-09-11T03:39:40.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:40.976 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:40.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:40.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:40 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:39:40.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:40 smithi175 ceph-mon[30894]: 2.7 scrub ok 2024-09-11T03:39:40.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:40 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1648844375' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:39:40.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:40 smithi175 ceph-mon[30894]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:39:41.065 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-11T03:39:41.076 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:41.128 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:41.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:41.129 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:41.144 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:39:40 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: 2024-09-11T03:39:40.945+0000 7fab40ce9640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-09-11T03:39:41.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:40 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:39:41.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:40 smithi084 ceph-mon[27254]: 2.7 scrub ok 2024-09-11T03:39:41.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:40 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1648844375' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:39:41.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:40 smithi084 ceph-mon[27254]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:39:41.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:40 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:39:41.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:40 smithi121 ceph-mon[30729]: 2.7 scrub ok 2024-09-11T03:39:41.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:40 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1648844375' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:39:41.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:40 smithi121 ceph-mon[30729]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:39:41.230 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:41.252 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:41.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:41.253 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:41.317 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:39:41.355 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:41.377 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:41.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:41.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:41.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:41.501 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:41.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:41.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:41.603 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:41.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:41.661 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:41.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:41.762 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:41.776 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:39:41.783 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:41.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:41.784 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:41.790 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-11T03:39:41.790 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-09-11T03:39:41.807 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:39:41.807 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:39:41.807 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:39:41.807 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:39:41.807 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:39:41.824 INFO:tasks.rados.rados.2.smithi175.stdout:376: done (1 left) 2024-09-11T03:39:41.825 INFO:tasks.rados.rados.2.smithi175.stdout:377: snap_remove snap 32 2024-09-11T03:39:41.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:41.911 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:41.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:41.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:41.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:41 smithi175 ceph-mon[30894]: 2.4 scrub starts 2024-09-11T03:39:41.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:41 smithi175 ceph-mon[30894]: 2.4 scrub ok 2024-09-11T03:39:41.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:41 smithi175 ceph-mon[30894]: pgmap v400: 17 pgs: 17 active+clean; 126 MiB data, 560 MiB used, 983 GiB / 983 GiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 8 op/s 2024-09-11T03:39:41.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:41 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:41.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:41 smithi175 ceph-mon[30894]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:39:41.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:41 smithi175 ceph-mon[30894]: osdmap e164: 12 total, 12 up, 11 in 2024-09-11T03:39:41.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:41 smithi175 ceph-mon[30894]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:41.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:41 smithi175 ceph-mon[30894]: osdmap e165: 12 total, 12 up, 11 in 2024-09-11T03:39:42.012 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:42.037 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:42.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:42.139 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:42.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:41 smithi084 ceph-mon[27254]: 2.4 scrub starts 2024-09-11T03:39:42.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:41 smithi084 ceph-mon[27254]: 2.4 scrub ok 2024-09-11T03:39:42.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:41 smithi084 ceph-mon[27254]: pgmap v400: 17 pgs: 17 active+clean; 126 MiB data, 560 MiB used, 983 GiB / 983 GiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 8 op/s 2024-09-11T03:39:42.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:41 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:42.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:41 smithi084 ceph-mon[27254]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:39:42.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:41 smithi084 ceph-mon[27254]: osdmap e164: 12 total, 12 up, 11 in 2024-09-11T03:39:42.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:41 smithi084 ceph-mon[27254]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:42.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:41 smithi084 ceph-mon[27254]: osdmap e165: 12 total, 12 up, 11 in 2024-09-11T03:39:42.161 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:42.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:42.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:42.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:41 smithi121 ceph-mon[30729]: 2.4 scrub starts 2024-09-11T03:39:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:41 smithi121 ceph-mon[30729]: 2.4 scrub ok 2024-09-11T03:39:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:41 smithi121 ceph-mon[30729]: pgmap v400: 17 pgs: 17 active+clean; 126 MiB data, 560 MiB used, 983 GiB / 983 GiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 8 op/s 2024-09-11T03:39:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:41 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:41 smithi121 ceph-mon[30729]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:39:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:41 smithi121 ceph-mon[30729]: osdmap e164: 12 total, 12 up, 11 in 2024-09-11T03:39:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:41 smithi121 ceph-mon[30729]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:41 smithi121 ceph-mon[30729]: osdmap e165: 12 total, 12 up, 11 in 2024-09-11T03:39:42.263 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:42.288 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:42.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:42.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:42.311 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:42.390 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:42.404 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-11T03:39:42.415 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:42.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:42.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:42.517 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:42.538 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:42.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:42.539 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:42.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:42.666 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:42.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:42.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:42.687 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:39:42.768 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:42.794 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:42.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:42.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:42.827 INFO:tasks.rados.rados.2.smithi175.stdout:377: done (1 left) 2024-09-11T03:39:42.827 INFO:tasks.rados.rados.2.smithi175.stdout:378: snap_create 2024-09-11T03:39:42.827 INFO:tasks.rados.rados.2.smithi175.stdout:374: finishing rollback tid 0 to smithi17557870-12 2024-09-11T03:39:42.827 INFO:tasks.rados.rados.2.smithi175.stdout:374: finishing rollback tid 1 to smithi17557870-12 2024-09-11T03:39:42.827 INFO:tasks.rados.rados.2.smithi175.stdout:374: finishing rollback tid 2 to smithi17557870-12 2024-09-11T03:39:42.827 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 107 (ObjNum 118 snap 18 seq_num 118) dirty exists 2024-09-11T03:39:42.829 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:42.895 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:42.917 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:42.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:42.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:42.986 INFO:teuthology.orchestra.run.smithi084.stdout:moved 0 / 51 (0%) 2024-09-11T03:39:42.986 INFO:teuthology.orchestra.run.smithi084.stdout:avg 4.63636 2024-09-11T03:39:42.986 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 1.72008 -> 1.72008 (expected baseline 2.05302) 2024-09-11T03:39:42.986 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.5 with 2 -> 2 pgs (0.431373 -> 0.431373 * mean) 2024-09-11T03:39:42.986 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.9 with 8 -> 8 pgs (1.72549 -> 1.72549 * mean) 2024-09-11T03:39:42.986 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:39:42.986 INFO:teuthology.orchestra.run.smithi084.stdout:oload 1000 2024-09-11T03:39:42.987 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 1 2024-09-11T03:39:42.987 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2024-09-11T03:39:42.987 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0006 2024-09-11T03:39:42.987 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0058 2024-09-11T03:39:42.988 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2024-09-11T03:39:43.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:43.045 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:43.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:43.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:43.147 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:43.167 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:39:43.167 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:39:43.167 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:39:43.167 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:39:43.167 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.168 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:237 2.c 2:3603e322:::smithi17557870-42:head [read 0~2689087,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106)", 2024-09-11T03:39:43.168 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:42.965828+0000", 2024-09-11T03:39:43.168 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 60.200458629000003, 2024-09-11T03:39:43.168 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0047361859999999999, 2024-09-11T03:39:43.168 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.168 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.168 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:43.168 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:43.168 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:43.168 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:43.168 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 237 2024-09-11T03:39:43.169 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.169 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.169 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.169 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.169 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.965828+0000", 2024-09-11T03:39:43.169 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.169 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.169 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.169 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.169 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.965828+0000", 2024-09-11T03:39:43.169 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.170 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.170 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.170 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.965829+0000", 2024-09-11T03:39:43.170 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3499999999999995e-07 2024-09-11T03:39:43.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.170 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.170 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.170 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.965833+0000", 2024-09-11T03:39:43.170 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8439999999999997e-06 2024-09-11T03:39:43.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.170 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.170 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.171 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.965845+0000", 2024-09-11T03:39:43.171 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.152e-05 2024-09-11T03:39:43.171 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.171 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.171 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.171 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.965850+0000", 2024-09-11T03:39:43.171 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0520000000000001e-06 2024-09-11T03:39:43.171 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.171 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.171 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.171 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.965962+0000", 2024-09-11T03:39:43.172 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000111666 2024-09-11T03:39:43.172 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.172 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.172 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:43.172 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.965965+0000", 2024-09-11T03:39:43.172 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3299999999999999e-06 2024-09-11T03:39:43.172 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.172 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.172 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.172 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.968711+0000", 2024-09-11T03:39:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0027456220000000001 2024-09-11T03:39:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.968790+0000", 2024-09-11T03:39:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9091000000000006e-05 2024-09-11T03:39:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.970564+0000", 2024-09-11T03:39:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0017743259999999999 2024-09-11T03:39:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:238 2.c 2:3603e322:::smithi17557870-42:head [sparse-read 0~1251306] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106)", 2024-09-11T03:39:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:42.965874+0000", 2024-09-11T03:39:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 60.200412657999998, 2024-09-11T03:39:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050592939999999998, 2024-09-11T03:39:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 238 2024-09-11T03:39:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.965874+0000", 2024-09-11T03:39:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.965874+0000", 2024-09-11T03:39:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.965874+0000", 2024-09-11T03:39:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.39e-07 2024-09-11T03:39:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.965876+0000", 2024-09-11T03:39:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1569999999999998e-06 2024-09-11T03:39:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.965890+0000", 2024-09-11T03:39:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4107000000000001e-05 2024-09-11T03:39:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.965894+0000", 2024-09-11T03:39:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9060000000000004e-06 2024-09-11T03:39:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.965985+0000", 2024-09-11T03:39:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.0317999999999998e-05 2024-09-11T03:39:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.965988+0000", 2024-09-11T03:39:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8289999999999999e-06 2024-09-11T03:39:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.970630+0000", 2024-09-11T03:39:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0046426610000000002 2024-09-11T03:39:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.179 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.179 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.970752+0000", 2024-09-11T03:39:43.179 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012164599999999999 2024-09-11T03:39:43.179 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.179 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.179 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.179 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:42.970933+0000", 2024-09-11T03:39:43.179 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00018123099999999999 2024-09-11T03:39:43.179 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.179 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.179 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.179 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.179 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.180 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:363 2.c 2:327469ff:::smithi17557870-49:head [sparse-read 0~1365420,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:39:43.180 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.849855+0000", 2024-09-11T03:39:43.180 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 52.316431367, 2024-09-11T03:39:43.180 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0056744530000000003, 2024-09-11T03:39:43.180 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.180 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.180 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:43.180 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:43.180 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:43.180 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:43.180 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 363 2024-09-11T03:39:43.180 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.180 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.181 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.181 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.181 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849855+0000", 2024-09-11T03:39:43.181 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.181 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.181 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.181 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.181 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849855+0000", 2024-09-11T03:39:43.181 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.181 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.181 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.181 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.181 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849858+0000", 2024-09-11T03:39:43.181 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9040000000000001e-06 2024-09-11T03:39:43.182 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.182 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.182 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.182 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849870+0000", 2024-09-11T03:39:43.182 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2463000000000001e-05 2024-09-11T03:39:43.182 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.182 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.182 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.182 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849901+0000", 2024-09-11T03:39:43.182 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0747000000000002e-05 2024-09-11T03:39:43.182 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.182 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.182 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.182 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849936+0000", 2024-09-11T03:39:43.183 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5302000000000002e-05 2024-09-11T03:39:43.183 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.183 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.183 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.183 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850035+0000", 2024-09-11T03:39:43.183 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8622000000000005e-05 2024-09-11T03:39:43.183 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.183 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.183 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:39:43.183 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850042+0000", 2024-09-11T03:39:43.183 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2579999999999998e-06 2024-09-11T03:39:43.183 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.183 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.184 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.184 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.851748+0000", 2024-09-11T03:39:43.184 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0017055670000000001 2024-09-11T03:39:43.184 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.184 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.184 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.184 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.851847+0000", 2024-09-11T03:39:43.184 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8904e-05 2024-09-11T03:39:43.184 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.184 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.184 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.184 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.855530+0000", 2024-09-11T03:39:43.185 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0036826860000000001 2024-09-11T03:39:43.185 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.185 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.185 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.185 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.185 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.185 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:364 2.c 2:327469ff:::smithi17557870-49:head [sparse-read 0~1365420] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:39:43.185 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.849973+0000", 2024-09-11T03:39:43.185 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 52.316312957999997, 2024-09-11T03:39:43.185 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0059499059999999996, 2024-09-11T03:39:43.185 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.185 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.185 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:43.185 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:43.186 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:43.186 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:43.186 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 364 2024-09-11T03:39:43.186 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.186 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.186 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.186 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.186 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849973+0000", 2024-09-11T03:39:43.186 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.186 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.186 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.186 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.186 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849973+0000", 2024-09-11T03:39:43.187 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.187 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.187 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.187 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.187 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849975+0000", 2024-09-11T03:39:43.187 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.08e-06 2024-09-11T03:39:43.187 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.187 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.187 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.187 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849979+0000", 2024-09-11T03:39:43.187 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4390000000000002e-06 2024-09-11T03:39:43.187 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.188 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.188 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.188 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849993+0000", 2024-09-11T03:39:43.188 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4208e-05 2024-09-11T03:39:43.188 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.188 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.188 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.188 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850000+0000", 2024-09-11T03:39:43.188 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0199999999999997e-06 2024-09-11T03:39:43.188 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.188 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.188 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.188 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850114+0000", 2024-09-11T03:39:43.189 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011416700000000001 2024-09-11T03:39:43.189 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.189 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.189 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:43.189 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850118+0000", 2024-09-11T03:39:43.189 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3000000000000002e-06 2024-09-11T03:39:43.189 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.189 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.189 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.189 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.855637+0000", 2024-09-11T03:39:43.189 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0055188750000000003 2024-09-11T03:39:43.189 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.189 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.190 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.190 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.855739+0000", 2024-09-11T03:39:43.190 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010228 2024-09-11T03:39:43.190 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.190 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.190 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.190 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.855923+0000", 2024-09-11T03:39:43.190 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000184537 2024-09-11T03:39:43.190 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.190 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.190 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.190 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.190 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.191 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:365 2.c 2:327469ff:::smithi17557870-49:head [read 0~2806736,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:39:43.191 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.850028+0000", 2024-09-11T03:39:43.191 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 52.316258234000003, 2024-09-11T03:39:43.191 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0077480070000000003, 2024-09-11T03:39:43.191 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.191 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.191 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:43.191 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:43.191 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:43.191 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:43.191 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 365 2024-09-11T03:39:43.191 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.191 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.192 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.192 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.192 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850028+0000", 2024-09-11T03:39:43.192 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.192 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.192 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.192 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.192 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850028+0000", 2024-09-11T03:39:43.192 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.192 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.192 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.192 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.192 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850029+0000", 2024-09-11T03:39:43.192 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1200000000000003e-07 2024-09-11T03:39:43.193 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.193 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.193 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.193 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850033+0000", 2024-09-11T03:39:43.193 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4789999999999998e-06 2024-09-11T03:39:43.193 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.193 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.193 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.193 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850043+0000", 2024-09-11T03:39:43.193 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.38e-06 2024-09-11T03:39:43.193 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.193 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.193 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.194 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850056+0000", 2024-09-11T03:39:43.194 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3179999999999999e-05 2024-09-11T03:39:43.194 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.194 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.194 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.194 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850150+0000", 2024-09-11T03:39:43.194 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4549000000000006e-05 2024-09-11T03:39:43.194 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.194 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.194 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:43.194 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850155+0000", 2024-09-11T03:39:43.194 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0440000000000003e-06 2024-09-11T03:39:43.194 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.195 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.195 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.195 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.855967+0000", 2024-09-11T03:39:43.195 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0058113289999999996 2024-09-11T03:39:43.195 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.195 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.195 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.195 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.856051+0000", 2024-09-11T03:39:43.195 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3840999999999999e-05 2024-09-11T03:39:43.195 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.195 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.195 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.195 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.857776+0000", 2024-09-11T03:39:43.195 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0017255930000000001 2024-09-11T03:39:43.196 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.196 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.196 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.196 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.196 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.196 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:382 2.c 2:32ade24e:::smithi17557870-45:head [read 0~2739508,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:39:43.196 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.978369+0000", 2024-09-11T03:39:43.196 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 52.187917007000003, 2024-09-11T03:39:43.196 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0038033089999999999, 2024-09-11T03:39:43.196 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.196 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.196 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:43.196 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:43.197 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:43.197 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:43.197 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 382 2024-09-11T03:39:43.197 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.197 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.197 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.197 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.197 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.978369+0000", 2024-09-11T03:39:43.197 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.197 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.197 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.197 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.197 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.978369+0000", 2024-09-11T03:39:43.198 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.198 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.198 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.198 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.198 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.978370+0000", 2024-09-11T03:39:43.198 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1800000000000002e-07 2024-09-11T03:39:43.199 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.199 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.199 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.199 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.978372+0000", 2024-09-11T03:39:43.199 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9879999999999999e-06 2024-09-11T03:39:43.199 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.199 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.199 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.199 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.978376+0000", 2024-09-11T03:39:43.199 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6330000000000004e-06 2024-09-11T03:39:43.199 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.199 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.199 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.200 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.978383+0000", 2024-09-11T03:39:43.200 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1099999999999997e-06 2024-09-11T03:39:43.200 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.200 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.200 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.200 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.981218+0000", 2024-09-11T03:39:43.200 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0028346489999999998 2024-09-11T03:39:43.200 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.200 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.200 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.200 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.981290+0000", 2024-09-11T03:39:43.200 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1428999999999997e-05 2024-09-11T03:39:43.200 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.201 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.201 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.201 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.982173+0000", 2024-09-11T03:39:43.201 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00088318200000000004 2024-09-11T03:39:43.201 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.201 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.201 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.201 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.201 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.201 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:554 2.c 2:3603e322:::smithi17557870-42:head [write 447086~770638,stat] snapc 10=[10,f,e] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-11T03:39:43.201 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:03.827163+0000", 2024-09-11T03:39:43.201 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 39.339123147000002, 2024-09-11T03:39:43.201 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0056079759999999998, 2024-09-11T03:39:43.202 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.202 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.202 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:43.202 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:43.202 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:43.202 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:43.202 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 554 2024-09-11T03:39:43.202 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.202 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.202 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.202 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.202 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.827163+0000", 2024-09-11T03:39:43.202 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.203 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.203 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.203 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.203 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.827163+0000", 2024-09-11T03:39:43.203 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.203 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.203 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.203 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.203 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.827166+0000", 2024-09-11T03:39:43.203 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7219999999999999e-06 2024-09-11T03:39:43.203 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.203 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.204 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.204 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.827693+0000", 2024-09-11T03:39:43.204 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00052672999999999999 2024-09-11T03:39:43.204 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.204 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.204 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.204 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.827731+0000", 2024-09-11T03:39:43.204 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8373e-05 2024-09-11T03:39:43.204 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.204 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.204 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.204 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.827761+0000", 2024-09-11T03:39:43.204 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9927999999999999e-05 2024-09-11T03:39:43.205 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.205 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.205 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.205 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.827841+0000", 2024-09-11T03:39:43.205 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0247000000000006e-05 2024-09-11T03:39:43.205 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.205 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.205 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:39:43.205 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.827851+0000", 2024-09-11T03:39:43.205 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4409999999999999e-06 2024-09-11T03:39:43.206 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.206 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.206 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.206 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.829517+0000", 2024-09-11T03:39:43.206 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0016659349999999999 2024-09-11T03:39:43.206 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.206 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.206 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.206 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.829594+0000", 2024-09-11T03:39:43.206 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7294999999999998e-05 2024-09-11T03:39:43.206 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.206 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.206 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:39:43.207 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.829707+0000", 2024-09-11T03:39:43.207 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000113455 2024-09-11T03:39:43.207 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.207 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.207 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:43.207 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.831602+0000", 2024-09-11T03:39:43.207 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001894904 2024-09-11T03:39:43.207 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.207 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.207 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:43.207 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.831988+0000", 2024-09-11T03:39:43.207 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00038567000000000001 2024-09-11T03:39:43.207 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.208 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.208 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:43.208 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.832699+0000", 2024-09-11T03:39:43.208 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000711355 2024-09-11T03:39:43.208 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.208 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.208 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:43.208 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.832738+0000", 2024-09-11T03:39:43.208 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8170000000000002e-05 2024-09-11T03:39:43.208 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.208 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.208 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.208 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.832771+0000", 2024-09-11T03:39:43.209 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3751000000000003e-05 2024-09-11T03:39:43.209 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.209 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.209 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.209 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.209 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.209 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:555 2.c 2:3603e322:::smithi17557870-42:head [write 1935000~798016] snapc 10=[10,f,e] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-11T03:39:43.209 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:03.835648+0000", 2024-09-11T03:39:43.209 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 39.330638024999999, 2024-09-11T03:39:43.209 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0034660160000000001, 2024-09-11T03:39:43.209 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.209 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.209 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 555 2024-09-11T03:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.835648+0000", 2024-09-11T03:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.835648+0000", 2024-09-11T03:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.835651+0000", 2024-09-11T03:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.632e-06 2024-09-11T03:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.835982+0000", 2024-09-11T03:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00033134400000000002 2024-09-11T03:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.835997+0000", 2024-09-11T03:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4601e-05 2024-09-11T03:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.836003+0000", 2024-09-11T03:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4409999999999998e-06 2024-09-11T03:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.836052+0000", 2024-09-11T03:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9055999999999999e-05 2024-09-11T03:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.213 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.213 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.213 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.836119+0000", 2024-09-11T03:39:43.213 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6340999999999994e-05 2024-09-11T03:39:43.213 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.213 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:43.214 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:43.214 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:43.215 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:39:43.215 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.836219+0000", 2024-09-11T03:39:43.215 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010021 2024-09-11T03:39:43.215 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.215 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.216 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:43.216 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.837493+0000", 2024-09-11T03:39:43.216 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001274124 2024-09-11T03:39:43.216 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.216 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.216 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:43.216 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.838381+0000", 2024-09-11T03:39:43.216 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00088745299999999996 2024-09-11T03:39:43.216 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.216 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.217 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:43.217 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.839033+0000", 2024-09-11T03:39:43.217 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00065233999999999995 2024-09-11T03:39:43.217 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.217 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.217 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:43.217 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.839069+0000", 2024-09-11T03:39:43.217 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5667000000000003e-05 2024-09-11T03:39:43.217 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.217 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.217 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.217 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.839114+0000", 2024-09-11T03:39:43.218 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5806999999999997e-05 2024-09-11T03:39:43.218 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.218 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.218 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.218 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.218 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.218 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(osd.8.60:4 2.c 2:3603e322:::smithi17557870-42:head [assert-version v31,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e129)", 2024-09-11T03:39:43.218 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:05.838769+0000", 2024-09-11T03:39:43.218 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 37.327517882999999, 2024-09-11T03:39:43.218 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0048369429999999998, 2024-09-11T03:39:43.218 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.218 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.218 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:43.218 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.219 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.219 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.219 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838769+0000", 2024-09-11T03:39:43.219 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.219 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.219 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.219 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.219 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838769+0000", 2024-09-11T03:39:43.219 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.219 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.219 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.219 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.219 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838770+0000", 2024-09-11T03:39:43.219 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8470000000000001e-06 2024-09-11T03:39:43.220 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.220 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.220 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.220 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838776+0000", 2024-09-11T03:39:43.220 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1529999999999996e-06 2024-09-11T03:39:43.220 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.220 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.220 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.220 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838786+0000", 2024-09-11T03:39:43.220 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0597e-05 2024-09-11T03:39:43.220 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.220 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.221 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.221 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838796+0000", 2024-09-11T03:39:43.221 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0144e-05 2024-09-11T03:39:43.221 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.221 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.221 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.221 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838858+0000", 2024-09-11T03:39:43.221 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1895000000000003e-05 2024-09-11T03:39:43.221 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.221 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.221 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:39:43.221 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838866+0000", 2024-09-11T03:39:43.221 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0129999999999993e-06 2024-09-11T03:39:43.221 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.222 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.222 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.222 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.840166+0000", 2024-09-11T03:39:43.222 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001299297 2024-09-11T03:39:43.222 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.222 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.222 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.222 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.840237+0000", 2024-09-11T03:39:43.222 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1336999999999998e-05 2024-09-11T03:39:43.222 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.222 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.222 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.222 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.843606+0000", 2024-09-11T03:39:43.223 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0033686599999999999 2024-09-11T03:39:43.223 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.223 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.223 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.223 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.223 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.223 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:640 2.a 2:5014e305:::smithi17557870-46:head [copy-from ver 23] snapc 11=[e] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:39:43.223 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:39:43.223 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 35.055993246, 2024-09-11T03:39:43.223 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019221736, 2024-09-11T03:39:43.223 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.223 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.223 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:43.224 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:43.224 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:43.224 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:43.224 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 640 2024-09-11T03:39:43.224 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.224 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.224 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.224 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.224 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:39:43.224 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.224 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.224 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.224 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.224 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:39:43.225 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.225 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.225 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.225 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.225 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110295+0000", 2024-09-11T03:39:43.225 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0609999999999998e-06 2024-09-11T03:39:43.225 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.225 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.225 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.225 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110305+0000", 2024-09-11T03:39:43.225 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4169999999999998e-06 2024-09-11T03:39:43.225 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.225 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.225 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.226 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110321+0000", 2024-09-11T03:39:43.226 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6288000000000001e-05 2024-09-11T03:39:43.226 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.226 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.226 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.226 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110328+0000", 2024-09-11T03:39:43.226 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3950000000000003e-06 2024-09-11T03:39:43.226 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.226 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.226 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.226 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110396+0000", 2024-09-11T03:39:43.226 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7164999999999998e-05 2024-09-11T03:39:43.226 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.227 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.227 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.227 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110507+0000", 2024-09-11T03:39:43.227 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000111217 2024-09-11T03:39:43.227 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.227 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.227 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:39:43.227 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.118213+0000", 2024-09-11T03:39:43.227 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0077066249999999999 2024-09-11T03:39:43.227 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.227 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.227 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:43.227 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125103+0000", 2024-09-11T03:39:43.227 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068894359999999997 2024-09-11T03:39:43.228 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.228 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.228 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:43.228 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125459+0000", 2024-09-11T03:39:43.228 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000356146 2024-09-11T03:39:43.228 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.228 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.228 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:43.228 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129369+0000", 2024-09-11T03:39:43.228 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039099249999999999 2024-09-11T03:39:43.228 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.228 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.228 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:43.229 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129428+0000", 2024-09-11T03:39:43.229 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9113000000000001e-05 2024-09-11T03:39:43.229 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.229 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.229 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.229 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129515+0000", 2024-09-11T03:39:43.229 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6947999999999995e-05 2024-09-11T03:39:43.229 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.229 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.229 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.229 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.229 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.229 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:641 2.a 2:5014e305:::smithi17557870-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:39:43.230 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:39:43.230 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 35.055941855, 2024-09-11T03:39:43.230 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011882876000000001, 2024-09-11T03:39:43.230 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.230 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.230 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:43.230 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:43.230 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:43.230 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:43.230 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 641 2024-09-11T03:39:43.230 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.230 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.230 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.230 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.231 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:39:43.231 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.231 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.231 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.231 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.231 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:39:43.231 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.231 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.231 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.231 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.231 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:39:43.231 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7599999999999998e-07 2024-09-11T03:39:43.231 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.232 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.232 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.232 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110347+0000", 2024-09-11T03:39:43.232 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.685e-06 2024-09-11T03:39:43.232 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.232 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.232 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.232 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110351+0000", 2024-09-11T03:39:43.232 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4889999999999997e-06 2024-09-11T03:39:43.232 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.232 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.232 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.233 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110354+0000", 2024-09-11T03:39:43.233 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0510000000000001e-06 2024-09-11T03:39:43.233 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.233 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.233 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.233 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110644+0000", 2024-09-11T03:39:43.233 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00028940200000000002 2024-09-11T03:39:43.233 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.233 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.233 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:39:43.233 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110673+0000", 2024-09-11T03:39:43.233 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9833e-05 2024-09-11T03:39:43.233 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.234 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.234 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.234 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122038+0000", 2024-09-11T03:39:43.234 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011364915 2024-09-11T03:39:43.234 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.234 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.234 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.234 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122127+0000", 2024-09-11T03:39:43.234 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8987999999999996e-05 2024-09-11T03:39:43.234 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.234 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.234 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.234 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122227+0000", 2024-09-11T03:39:43.235 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010013699999999999 2024-09-11T03:39:43.235 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.235 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.235 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.235 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.235 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.235 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:752 2.c 2:3c007163:::smithi17557870-35:head [copy-from ver 39] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:39:43.235 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:39:43.235 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 28.157270573000002, 2024-09-11T03:39:43.235 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017596783000000001, 2024-09-11T03:39:43.235 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.235 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.235 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:43.235 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:43.236 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:43.236 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:43.236 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 752 2024-09-11T03:39:43.236 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.236 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.236 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.236 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.236 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:39:43.236 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.236 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.236 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.236 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.236 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:39:43.237 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.237 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.237 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.237 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.237 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009018+0000", 2024-09-11T03:39:43.237 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.48e-06 2024-09-11T03:39:43.237 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.237 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.237 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.237 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009029+0000", 2024-09-11T03:39:43.237 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0377000000000001e-05 2024-09-11T03:39:43.237 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009050+0000", 2024-09-11T03:39:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0849e-05 2024-09-11T03:39:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009058+0000", 2024-09-11T03:39:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8019999999999998e-06 2024-09-11T03:39:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009125+0000", 2024-09-11T03:39:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6388000000000002e-05 2024-09-11T03:39:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:39:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009130+0000", 2024-09-11T03:39:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0900000000000004e-06 2024-09-11T03:39:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010518+0000", 2024-09-11T03:39:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013880030000000001 2024-09-11T03:39:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010582+0000", 2024-09-11T03:39:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.368e-05 2024-09-11T03:39:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:39:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.015350+0000", 2024-09-11T03:39:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0047682870000000004 2024-09-11T03:39:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.019005+0000", 2024-09-11T03:39:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0036549040000000001 2024-09-11T03:39:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.024096+0000", 2024-09-11T03:39:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050917439999999996 2024-09-11T03:39:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026521+0000", 2024-09-11T03:39:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024247399999999999 2024-09-11T03:39:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026559+0000", 2024-09-11T03:39:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8232999999999998e-05 2024-09-11T03:39:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026613+0000", 2024-09-11T03:39:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3205999999999998e-05 2024-09-11T03:39:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:753 2.c 2:3c007163:::smithi17557870-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:39:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:39:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 28.157203486, 2024-09-11T03:39:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0094619260000000007, 2024-09-11T03:39:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 753 2024-09-11T03:39:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:39:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:39:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:39:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2100000000000002e-07 2024-09-11T03:39:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.245 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.245 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009085+0000", 2024-09-11T03:39:43.245 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5910000000000001e-06 2024-09-11T03:39:43.245 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.245 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.245 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.245 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009090+0000", 2024-09-11T03:39:43.245 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5180000000000004e-06 2024-09-11T03:39:43.245 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.245 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.245 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.246 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009094+0000", 2024-09-11T03:39:43.246 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0620000000000001e-06 2024-09-11T03:39:43.246 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.246 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.246 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.246 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009178+0000", 2024-09-11T03:39:43.246 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4525999999999997e-05 2024-09-11T03:39:43.246 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.246 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.246 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:43.246 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009182+0000", 2024-09-11T03:39:43.246 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4640000000000002e-06 2024-09-11T03:39:43.247 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.247 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.247 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.247 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010677+0000", 2024-09-11T03:39:43.247 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014957569999999999 2024-09-11T03:39:43.247 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.247 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.247 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:39:43.247 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010706+0000", 2024-09-11T03:39:43.247 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8873999999999998e-05 2024-09-11T03:39:43.247 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.247 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.248 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.248 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018385+0000", 2024-09-11T03:39:43.248 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0076790490000000003 2024-09-11T03:39:43.248 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.248 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.248 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.248 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018466+0000", 2024-09-11T03:39:43.248 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0483999999999997e-05 2024-09-11T03:39:43.248 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.248 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.248 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.249 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018545+0000", 2024-09-11T03:39:43.249 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9179999999999997e-05 2024-09-11T03:39:43.249 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.249 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.249 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.249 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.249 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.249 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:839 2.a 2:5014e305:::smithi17557870-46:19 [read 0~3915998,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e146)", 2024-09-11T03:39:43.249 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:22.827744+0000", 2024-09-11T03:39:43.249 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 20.338542345, 2024-09-11T03:39:43.249 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0051572420000000002, 2024-09-11T03:39:43.249 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.250 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.250 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:43.250 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:43.250 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:43.250 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:43.250 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 839 2024-09-11T03:39:43.250 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.250 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.250 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.250 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.250 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827744+0000", 2024-09-11T03:39:43.250 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.251 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.251 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.251 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.251 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827744+0000", 2024-09-11T03:39:43.251 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.251 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.251 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.251 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.251 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827746+0000", 2024-09-11T03:39:43.251 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.198e-06 2024-09-11T03:39:43.251 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.251 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.251 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.251 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827755+0000", 2024-09-11T03:39:43.252 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.789e-06 2024-09-11T03:39:43.252 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.252 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.252 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.252 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827774+0000", 2024-09-11T03:39:43.252 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.891e-05 2024-09-11T03:39:43.252 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.252 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.252 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.252 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827795+0000", 2024-09-11T03:39:43.252 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0978000000000001e-05 2024-09-11T03:39:43.252 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.252 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.253 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.253 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827865+0000", 2024-09-11T03:39:43.253 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0178999999999993e-05 2024-09-11T03:39:43.253 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.253 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.253 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:39:43.253 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827872+0000", 2024-09-11T03:39:43.253 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4250000000000003e-06 2024-09-11T03:39:43.253 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.253 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.253 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.253 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.829088+0000", 2024-09-11T03:39:43.253 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00121659 2024-09-11T03:39:43.254 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.254 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.254 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.254 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.829167+0000", 2024-09-11T03:39:43.254 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8774000000000002e-05 2024-09-11T03:39:43.254 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.254 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.254 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.254 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.832901+0000", 2024-09-11T03:39:43.254 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0037343989999999998 2024-09-11T03:39:43.254 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.254 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.254 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.254 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.255 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.255 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:840 2.a 2:5014e305:::smithi17557870-46:19 [sparse-read 0~2343134] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e146)", 2024-09-11T03:39:43.255 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:22.827818+0000", 2024-09-11T03:39:43.255 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 20.338468864999999, 2024-09-11T03:39:43.255 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0053240099999999997, 2024-09-11T03:39:43.255 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.255 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.255 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:43.255 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:43.255 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:43.255 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:43.255 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 840 2024-09-11T03:39:43.255 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.255 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.256 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.256 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.256 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827818+0000", 2024-09-11T03:39:43.256 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.256 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.256 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.256 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.256 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827818+0000", 2024-09-11T03:39:43.256 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.256 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.256 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.256 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.256 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827818+0000", 2024-09-11T03:39:43.257 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1600000000000001e-07 2024-09-11T03:39:43.257 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.257 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.257 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.257 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827822+0000", 2024-09-11T03:39:43.257 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3289999999999998e-06 2024-09-11T03:39:43.257 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.257 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.257 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.257 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827833+0000", 2024-09-11T03:39:43.257 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1558e-05 2024-09-11T03:39:43.257 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.257 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.258 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.258 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827838+0000", 2024-09-11T03:39:43.258 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2059999999999998e-06 2024-09-11T03:39:43.258 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.258 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.258 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.258 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827935+0000", 2024-09-11T03:39:43.258 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6779000000000003e-05 2024-09-11T03:39:43.258 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.258 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.258 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:43.258 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827939+0000", 2024-09-11T03:39:43.258 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3550000000000001e-06 2024-09-11T03:39:43.258 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.258 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.259 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.259 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.832953+0000", 2024-09-11T03:39:43.259 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050147880000000001 2024-09-11T03:39:43.259 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.259 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.259 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.259 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.833035+0000", 2024-09-11T03:39:43.259 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1193999999999995e-05 2024-09-11T03:39:43.259 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.259 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.259 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.259 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.833142+0000", 2024-09-11T03:39:43.259 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000107085 2024-09-11T03:39:43.260 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.260 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.260 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.260 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.260 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.260 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:841 2.a 2:5014e305:::smithi17557870-46:19 [sparse-read 0~2343134] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e146)", 2024-09-11T03:39:43.260 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:22.827859+0000", 2024-09-11T03:39:43.260 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 20.33842697, 2024-09-11T03:39:43.260 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0055418159999999998, 2024-09-11T03:39:43.260 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.260 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.260 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:43.260 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:43.261 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:43.261 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:43.261 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 841 2024-09-11T03:39:43.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.261 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.261 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.261 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827859+0000", 2024-09-11T03:39:43.261 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.261 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.261 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827859+0000", 2024-09-11T03:39:43.262 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.262 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.262 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827860+0000", 2024-09-11T03:39:43.262 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5199999999999997e-07 2024-09-11T03:39:43.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.262 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.262 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827863+0000", 2024-09-11T03:39:43.263 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9699999999999999e-06 2024-09-11T03:39:43.263 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.263 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.263 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.263 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827871+0000", 2024-09-11T03:39:43.263 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3089999999999996e-06 2024-09-11T03:39:43.263 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.263 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.263 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.263 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827877+0000", 2024-09-11T03:39:43.263 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7189999999999998e-06 2024-09-11T03:39:43.264 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.264 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.264 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.264 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827959+0000", 2024-09-11T03:39:43.264 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2321e-05 2024-09-11T03:39:43.264 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.264 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.264 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:43.264 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827962+0000", 2024-09-11T03:39:43.264 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5689999999999999e-06 2024-09-11T03:39:43.264 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.265 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.265 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.265 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.833169+0000", 2024-09-11T03:39:43.265 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0052065920000000003 2024-09-11T03:39:43.265 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.265 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.265 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.265 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.833246+0000", 2024-09-11T03:39:43.265 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7987999999999999e-05 2024-09-11T03:39:43.265 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.265 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.266 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.266 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.833401+0000", 2024-09-11T03:39:43.266 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015479600000000001 2024-09-11T03:39:43.266 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.266 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.266 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.266 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.266 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(osd.1.13:9 2.c 2:3603e322:::smithi17557870-42:head [assert-version v44,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163)", 2024-09-11T03:39:43.266 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:39.832424+0000", 2024-09-11T03:39:43.266 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 3.3338620400000001, 2024-09-11T03:39:43.267 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0034472740000000002, 2024-09-11T03:39:43.267 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.267 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.267 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:43.267 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.267 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.267 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.267 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.832424+0000", 2024-09-11T03:39:43.267 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.267 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.267 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.267 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.267 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.832424+0000", 2024-09-11T03:39:43.268 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.268 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.268 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.268 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.268 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.832427+0000", 2024-09-11T03:39:43.268 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1660000000000001e-06 2024-09-11T03:39:43.268 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.268 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.268 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.268 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.832433+0000", 2024-09-11T03:39:43.268 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6749999999999996e-06 2024-09-11T03:39:43.268 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.268 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.269 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.269 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.832458+0000", 2024-09-11T03:39:43.269 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4385000000000001e-05 2024-09-11T03:39:43.269 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.269 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.269 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.269 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.832470+0000", 2024-09-11T03:39:43.269 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2156e-05 2024-09-11T03:39:43.269 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.269 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.269 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.269 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.832522+0000", 2024-09-11T03:39:43.270 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1876000000000003e-05 2024-09-11T03:39:43.270 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.270 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.270 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:39:43.270 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.832526+0000", 2024-09-11T03:39:43.270 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.634e-06 2024-09-11T03:39:43.270 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.270 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.270 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.270 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.833435+0000", 2024-09-11T03:39:43.270 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00090834599999999998 2024-09-11T03:39:43.270 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.271 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.271 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.271 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.833537+0000", 2024-09-11T03:39:43.271 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010272699999999999 2024-09-11T03:39:43.271 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.271 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.271 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.271 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.835872+0000", 2024-09-11T03:39:43.271 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023343090000000001 2024-09-11T03:39:43.271 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.271 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.271 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.272 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.272 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.272 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:985 2.b e163/95 2:dde92fc6:::smithi17557870-41:head v 163'52, mlcod=158'48)", 2024-09-11T03:39:43.272 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:39:43.272 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 3.233335941, 2024-09-11T03:39:43.272 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0092288890000000005, 2024-09-11T03:39:43.272 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.272 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.272 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:43.272 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.272 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.272 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.272 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:39:43.273 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.273 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.273 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.273 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.273 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:39:43.273 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.273 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.273 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.273 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.273 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:39:43.273 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7799999999999997e-07 2024-09-11T03:39:43.273 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.273 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.274 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.274 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935287+0000", 2024-09-11T03:39:43.274 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023361599999999999 2024-09-11T03:39:43.274 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.274 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.274 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.274 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935321+0000", 2024-09-11T03:39:43.274 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3840000000000001e-05 2024-09-11T03:39:43.274 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.274 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.274 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.274 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935335+0000", 2024-09-11T03:39:43.275 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4184e-05 2024-09-11T03:39:43.275 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.275 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.275 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.275 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935428+0000", 2024-09-11T03:39:43.275 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3144000000000001e-05 2024-09-11T03:39:43.275 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.275 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.275 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.275 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935503+0000", 2024-09-11T03:39:43.275 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4697999999999995e-05 2024-09-11T03:39:43.275 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.275 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.275 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:43.276 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.942117+0000", 2024-09-11T03:39:43.276 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0066141350000000002 2024-09-11T03:39:43.276 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.276 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.276 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.276 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.942179+0000", 2024-09-11T03:39:43.276 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.245e-05 2024-09-11T03:39:43.276 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.276 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.276 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.276 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.276 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.276 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:990 2.a 2:5014e305:::smithi17557870-46:head [write 638699~541153] snapc 24=[24,23,22,20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-11T03:39:43.277 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:39:43.277 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 3.229354689, 2024-09-11T03:39:43.277 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0080995760000000007, 2024-09-11T03:39:43.277 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.277 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.277 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:43.277 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:39:43.277 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:39:43.277 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:43.277 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 990 2024-09-11T03:39:43.277 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.277 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.277 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.278 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.278 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:39:43.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.278 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.278 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.278 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:39:43.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.278 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.278 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.278 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936935+0000", 2024-09-11T03:39:43.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7800000000000001e-06 2024-09-11T03:39:43.279 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.279 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.279 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.279 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937090+0000", 2024-09-11T03:39:43.279 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000155857 2024-09-11T03:39:43.279 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.279 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.279 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.279 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937109+0000", 2024-09-11T03:39:43.279 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8216e-05 2024-09-11T03:39:43.279 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.279 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.279 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.279 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937119+0000", 2024-09-11T03:39:43.280 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0406e-05 2024-09-11T03:39:43.280 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.280 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.280 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.280 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937195+0000", 2024-09-11T03:39:43.280 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6372999999999995e-05 2024-09-11T03:39:43.280 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.280 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.280 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.280 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937287+0000", 2024-09-11T03:39:43.280 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1434000000000006e-05 2024-09-11T03:39:43.280 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.280 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.281 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:39:43.281 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937454+0000", 2024-09-11T03:39:43.281 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000167262 2024-09-11T03:39:43.281 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.281 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.281 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:43.281 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.943162+0000", 2024-09-11T03:39:43.281 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0057082230000000001 2024-09-11T03:39:43.281 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.281 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.281 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:43.281 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.943196+0000", 2024-09-11T03:39:43.281 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4025e-05 2024-09-11T03:39:43.281 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.282 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:39:43.282 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.944922+0000", 2024-09-11T03:39:43.282 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001726102 2024-09-11T03:39:43.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.282 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:39:43.282 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.944977+0000", 2024-09-11T03:39:43.282 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4849999999999998e-05 2024-09-11T03:39:43.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.283 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.283 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.945031+0000", 2024-09-11T03:39:43.283 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4048000000000002e-05 2024-09-11T03:39:43.283 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.283 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.283 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.283 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.283 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop_reply(client.24454.0:990 2.a e163/95 ondisk, result = 0)", 2024-09-11T03:39:43.283 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:39.939109+0000", 2024-09-11T03:39:43.283 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 3.2271773819999998, 2024-09-11T03:39:43.283 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.004059455, 2024-09-11T03:39:43.283 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:39:43.283 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:39:43.284 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:39:43.284 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:39:43.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.284 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:39:43.284 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.939109+0000", 2024-09-11T03:39:43.284 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.284 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:39:43.284 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.939109+0000", 2024-09-11T03:39:43.284 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:39:43.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.284 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:39:43.285 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.939110+0000", 2024-09-11T03:39:43.285 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4600000000000004e-07 2024-09-11T03:39:43.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.285 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:39:43.285 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.939113+0000", 2024-09-11T03:39:43.285 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5389999999999999e-06 2024-09-11T03:39:43.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.285 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:39:43.288 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.939123+0000", 2024-09-11T03:39:43.288 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8360000000000004e-06 2024-09-11T03:39:43.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.288 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.288 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:39:43.288 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.939130+0000", 2024-09-11T03:39:43.288 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2860000000000002e-06 2024-09-11T03:39:43.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.288 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.288 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:39:43.288 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.943152+0000", 2024-09-11T03:39:43.289 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0040215600000000004 2024-09-11T03:39:43.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.289 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:39:43.289 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.943158+0000", 2024-09-11T03:39:43.289 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0260000000000004e-06 2024-09-11T03:39:43.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:39:43.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:39:43.289 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:39:43.289 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.943169+0000", 2024-09-11T03:39:43.289 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0462000000000001e-05 2024-09-11T03:39:43.289 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.289 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.289 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.290 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:39:43.290 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:39:43.290 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:39:43.315 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:43.337 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:43.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:43.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:43.439 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:43.460 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:43.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:43.461 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:43.518 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:39:43.518 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:39:43.534 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:39:43.535 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:39:43.552 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:39:43.552 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:39:43.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:43.568 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:39:43.569 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:39:43.585 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:39:43.586 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:39:43.588 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:43.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:43.602 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:39:43.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:39:43.619 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:39:43.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:39:43.636 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:39:43.636 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:39:43.652 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:39:43.652 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:39:43.669 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:39:43.669 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:39:43.685 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:39:43.686 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:39:43.689 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:43.703 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:39:43.703 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:39:43.714 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:43.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:43.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:43.816 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:43.831 INFO:tasks.rados.rados.2.smithi175.stdout:374: done (1 left) 2024-09-11T03:39:43.831 INFO:tasks.rados.rados.2.smithi175.stdout:378: done (0 left) 2024-09-11T03:39:43.832 INFO:tasks.rados.rados.2.smithi175.stdout:379: read oid 19 snap -1 2024-09-11T03:39:43.832 INFO:tasks.rados.rados.2.smithi175.stdout:379: expect deleted 2024-09-11T03:39:43.832 INFO:tasks.rados.rados.2.smithi175.stdout:380: delete oid 31 current snap is 36 2024-09-11T03:39:43.834 INFO:tasks.rados.rados.2.smithi175.stdout:379: done (1 left) 2024-09-11T03:39:43.834 INFO:tasks.rados.rados.2.smithi175.stdout:380: done (0 left) 2024-09-11T03:39:43.834 INFO:tasks.rados.rados.2.smithi175.stdout:381: delete oid 9 current snap is 36 2024-09-11T03:39:43.835 INFO:tasks.rados.rados.2.smithi175.stdout:381: done (0 left) 2024-09-11T03:39:43.835 INFO:tasks.rados.rados.2.smithi175.stdout:382: delete oid 13 current snap is 36 2024-09-11T03:39:43.837 INFO:tasks.rados.rados.2.smithi175.stdout:382: done (0 left) 2024-09-11T03:39:43.837 INFO:tasks.rados.rados.2.smithi175.stdout:383: snap_remove snap 28 2024-09-11T03:39:43.838 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:43.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:43.839 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:43.923 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:43 smithi084 ceph-mon[27254]: pgmap v403: 17 pgs: 17 active+clean; 145 MiB data, 587 MiB used, 983 GiB / 983 GiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 15 op/s 2024-09-11T03:39:43.923 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:43 smithi084 ceph-mon[27254]: osdmap e166: 12 total, 12 up, 11 in 2024-09-11T03:39:43.923 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:43 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:39:43.923 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:43 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:39:43.924 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:43 smithi084 ceph-mon[27254]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:43.924 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:43 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3195575101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:43.940 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:43.961 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:43.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:43.962 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:43.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:43 smithi175 ceph-mon[30894]: pgmap v403: 17 pgs: 17 active+clean; 145 MiB data, 587 MiB used, 983 GiB / 983 GiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 15 op/s 2024-09-11T03:39:43.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:43 smithi175 ceph-mon[30894]: osdmap e166: 12 total, 12 up, 11 in 2024-09-11T03:39:43.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:43 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:39:43.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:43 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:39:43.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:43 smithi175 ceph-mon[30894]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:43.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:43 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3195575101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:44.063 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:44.089 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:44.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:44.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:44.171 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T03:39:44.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:43 smithi121 ceph-mon[30729]: pgmap v403: 17 pgs: 17 active+clean; 145 MiB data, 587 MiB used, 983 GiB / 983 GiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 15 op/s 2024-09-11T03:39:44.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:43 smithi121 ceph-mon[30729]: osdmap e166: 12 total, 12 up, 11 in 2024-09-11T03:39:44.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:43 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:39:44.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:43 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:39:44.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:43 smithi121 ceph-mon[30729]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:39:44.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:43 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3195575101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:44.191 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:44.216 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:44.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:44.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:44.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:39:44.318 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:44.342 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:44.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:44.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:44.445 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:44.467 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:44.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:44.468 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:44.569 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:44.594 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:44.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:44.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:44.595 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:44.609 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:44.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:44.722 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:44.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:44.824 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:44.838 INFO:tasks.rados.rados.2.smithi175.stdout:383: done (0 left) 2024-09-11T03:39:44.838 INFO:tasks.rados.rados.2.smithi175.stdout:384: write (excl) oid 13 current snap is 36 2024-09-11T03:39:44.838 INFO:tasks.rados.rados.2.smithi175.stdout:384: seq_num 163 ranges {739541=709000,2121859=426376} 2024-09-11T03:39:44.846 INFO:tasks.rados.rados.2.smithi175.stdout:384: writing smithi17557870-13 from 739541 to 1448541 tid 1 2024-09-11T03:39:44.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:44.849 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:44.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:44.852 INFO:tasks.rados.rados.2.smithi175.stdout:384: writing smithi17557870-13 from 2121859 to 2548235 tid 2 2024-09-11T03:39:44.852 INFO:tasks.rados.rados.2.smithi175.stdout:385: write (excl) oid 22 current snap is 36 2024-09-11T03:39:44.852 INFO:tasks.rados.rados.2.smithi175.stdout:384: finishing write tid 1 to smithi17557870-13 2024-09-11T03:39:44.853 INFO:tasks.rados.rados.2.smithi175.stdout:385: seq_num 164 ranges {756348=784249,2063988=686387} 2024-09-11T03:39:44.860 INFO:tasks.rados.rados.2.smithi175.stdout:385: writing smithi17557870-22 from 756348 to 1540597 tid 1 2024-09-11T03:39:44.867 INFO:tasks.rados.rados.2.smithi175.stdout:385: writing smithi17557870-22 from 2063988 to 2750375 tid 2 2024-09-11T03:39:44.867 INFO:tasks.rados.rados.2.smithi175.stdout:386: read oid 18 snap 34 2024-09-11T03:39:44.867 INFO:tasks.rados.rados.2.smithi175.stdout:386: expect (ObjNum 115 snap 18 seq_num 115) 2024-09-11T03:39:44.867 INFO:tasks.rados.rados.2.smithi175.stdout:387: snap_create 2024-09-11T03:39:44.867 INFO:tasks.rados.rados.2.smithi175.stdout:384: finishing write tid 2 to smithi17557870-13 2024-09-11T03:39:44.867 INFO:tasks.rados.rados.2.smithi175.stdout:384: finishing write tid 3 to smithi17557870-13 2024-09-11T03:39:44.867 INFO:tasks.rados.rados.2.smithi175.stdout:384: finishing write tid 4 to smithi17557870-13 2024-09-11T03:39:44.867 INFO:tasks.rados.rados.2.smithi175.stdout:384: oid 13 updating version 0 to 125 2024-09-11T03:39:44.867 INFO:tasks.rados.rados.2.smithi175.stdout:384: oid 13 version 125 is already newer than 124 2024-09-11T03:39:44.867 INFO:tasks.rados.rados.2.smithi175.stdout:384: oid 13 updating version 125 to 126 2024-09-11T03:39:44.867 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 126 (ObjNum 163 snap 36 seq_num 163) dirty exists 2024-09-11T03:39:44.867 INFO:tasks.rados.rados.2.smithi175.stdout:384: left oid 13 (ObjNum 163 snap 36 seq_num 163) 2024-09-11T03:39:44.868 INFO:tasks.rados.rados.2.smithi175.stdout:385: finishing write tid 1 to smithi17557870-22 2024-09-11T03:39:44.873 INFO:tasks.rados.rados.2.smithi175.stdout:385: finishing write tid 2 to smithi17557870-22 2024-09-11T03:39:44.873 INFO:tasks.rados.rados.2.smithi175.stdout:385: finishing write tid 3 to smithi17557870-22 2024-09-11T03:39:44.873 INFO:tasks.rados.rados.2.smithi175.stdout:385: finishing write tid 4 to smithi17557870-22 2024-09-11T03:39:44.873 INFO:tasks.rados.rados.2.smithi175.stdout:385: oid 22 updating version 0 to 130 2024-09-11T03:39:44.873 INFO:tasks.rados.rados.2.smithi175.stdout:385: oid 22 version 130 is already newer than 129 2024-09-11T03:39:44.873 INFO:tasks.rados.rados.2.smithi175.stdout:385: oid 22 version 130 is already newer than 128 2024-09-11T03:39:44.874 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 130 (ObjNum 164 snap 36 seq_num 164) dirty exists 2024-09-11T03:39:44.874 INFO:tasks.rados.rados.2.smithi175.stdout:385: left oid 22 (ObjNum 164 snap 36 seq_num 164) 2024-09-11T03:39:44.877 INFO:tasks.rados.rados.2.smithi175.stdout:386: expect (ObjNum 115 snap 18 seq_num 115) 2024-09-11T03:39:44.950 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:44.970 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:44.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:44.971 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:45.072 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:45.093 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:45.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:44 smithi084 ceph-mon[27254]: osdmap e167: 12 total, 12 up, 11 in 2024-09-11T03:39:45.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:45.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:45.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:44 smithi121 ceph-mon[30729]: osdmap e167: 12 total, 12 up, 11 in 2024-09-11T03:39:45.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:44 smithi175 ceph-mon[30894]: osdmap e167: 12 total, 12 up, 11 in 2024-09-11T03:39:45.243 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:45.264 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:45.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:45.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:45.366 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:45.387 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:45.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:45.388 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:45.489 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:45.510 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:45.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:45.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:45.511 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:39:45.511 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:39:45.611 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:45.632 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:45.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:45.735 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:45.761 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:45.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:45.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:45.842 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:39:45.844 INFO:tasks.rados.rados.2.smithi175.stdout:384: done (3 left) 2024-09-11T03:39:45.845 INFO:tasks.rados.rados.2.smithi175.stdout:385: done (2 left) 2024-09-11T03:39:45.845 INFO:tasks.rados.rados.2.smithi175.stdout:386: done (1 left) 2024-09-11T03:39:45.845 INFO:tasks.rados.rados.2.smithi175.stdout:387: done (0 left) 2024-09-11T03:39:45.845 INFO:tasks.rados.rados.2.smithi175.stdout:388: copy_from oid 32 from oid 47 current snap is 37 2024-09-11T03:39:45.845 INFO:tasks.rados.rados.2.smithi175.stdout:389: read oid 38 snap 34 2024-09-11T03:39:45.845 INFO:tasks.rados.rados.2.smithi175.stdout:389: expect (ObjNum 101 snap 16 seq_num 101) 2024-09-11T03:39:45.845 INFO:tasks.rados.rados.2.smithi175.stdout:390: write (excl) oid 23 current snap is 37 2024-09-11T03:39:45.845 INFO:tasks.rados.rados.2.smithi175.stdout:390: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2024-09-11T03:39:45.852 INFO:tasks.rados.rados.2.smithi175.stdout:390: writing smithi17557870-23 from 773155 to 1232653 tid 1 2024-09-11T03:39:45.857 INFO:tasks.rados.rados.2.smithi175.stdout:390: writing smithi17557870-23 from 1722470 to 2152515 tid 2 2024-09-11T03:39:45.860 INFO:tasks.rados.rados.2.smithi175.stdout:390: writing smithi17557870-23 from 2648676 to 2773155 tid 3 2024-09-11T03:39:45.860 INFO:tasks.rados.rados.2.smithi175.stdout:391: snap_remove snap 30 2024-09-11T03:39:45.863 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:45.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:45.886 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:45.886 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:45.987 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:46.009 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:46.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:46.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:46.108 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:45 smithi084 ceph-mon[27254]: pgmap v406: 17 pgs: 17 active+clean; 145 MiB data, 600 MiB used, 983 GiB / 983 GiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 17 op/s 2024-09-11T03:39:46.109 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:45 smithi084 ceph-mon[27254]: osdmap e168: 12 total, 12 up, 11 in 2024-09-11T03:39:46.109 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:45 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1170969973' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:46.110 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:46.132 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:46.132 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:46.134 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T03:39:46.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:45 smithi121 ceph-mon[30729]: pgmap v406: 17 pgs: 17 active+clean; 145 MiB data, 600 MiB used, 983 GiB / 983 GiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 17 op/s 2024-09-11T03:39:46.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:45 smithi121 ceph-mon[30729]: osdmap e168: 12 total, 12 up, 11 in 2024-09-11T03:39:46.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:45 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1170969973' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:45 smithi175 ceph-mon[30894]: pgmap v406: 17 pgs: 17 active+clean; 145 MiB data, 600 MiB used, 983 GiB / 983 GiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 17 op/s 2024-09-11T03:39:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:45 smithi175 ceph-mon[30894]: osdmap e168: 12 total, 12 up, 11 in 2024-09-11T03:39:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:45 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1170969973' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:46.233 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:46.253 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:46.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:46.254 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:46.355 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:46.381 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:46.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:46.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:46.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:46.505 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:46.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:46.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:46.557 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:39:46.607 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:46.633 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:46.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:46.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:46.735 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:46.757 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:46.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:46.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:46.827 INFO:tasks.rados.rados.2.smithi175.stdout:391: done (3 left) 2024-09-11T03:39:46.827 INFO:tasks.rados.rados.2.smithi175.stdout:392: delete oid 25 current snap is 37 2024-09-11T03:39:46.827 INFO:tasks.rados.rados.2.smithi175.stdout:389: expect (ObjNum 101 snap 16 seq_num 101) 2024-09-11T03:39:46.843 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:46.859 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:46.880 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:46.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:46.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:46.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:46.993 INFO:tasks.rados.rados.2.smithi175.stdout:388: finishing copy_from racing read to smithi17557870-32 2024-09-11T03:39:46.993 INFO:tasks.rados.rados.2.smithi175.stdout:388: finishing copy_from to smithi17557870-32 2024-09-11T03:39:46.993 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 47 (ObjNum 124 snap 20 seq_num 124) dirty exists 2024-09-11T03:39:46.994 INFO:tasks.rados.rados.2.smithi175.stdout:390: finishing write tid 1 to smithi17557870-23 2024-09-11T03:39:46.994 INFO:tasks.rados.rados.2.smithi175.stdout:390: finishing write tid 2 to smithi17557870-23 2024-09-11T03:39:46.994 INFO:tasks.rados.rados.2.smithi175.stdout:390: finishing write tid 3 to smithi17557870-23 2024-09-11T03:39:46.994 INFO:tasks.rados.rados.2.smithi175.stdout:388: done (3 left) 2024-09-11T03:39:46.994 INFO:tasks.rados.rados.2.smithi175.stdout:389: done (2 left) 2024-09-11T03:39:46.994 INFO:tasks.rados.rados.2.smithi175.stdout:392: done (1 left) 2024-09-11T03:39:46.995 INFO:tasks.rados.rados.2.smithi175.stdout:393: snap_remove snap 35 2024-09-11T03:39:47.008 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:47.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:47.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:47.097 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1170969973' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:47.097 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:46 smithi084 ceph-mon[27254]: osdmap e169: 12 total, 12 up, 11 in 2024-09-11T03:39:47.097 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:46 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:47.097 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:46 smithi084 ceph-mon[27254]: osdmap e170: 12 total, 12 up, 11 in 2024-09-11T03:39:47.110 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:47.131 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:47.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:47.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:47.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1170969973' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:46 smithi121 ceph-mon[30729]: osdmap e169: 12 total, 12 up, 11 in 2024-09-11T03:39:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:46 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:46 smithi121 ceph-mon[30729]: osdmap e170: 12 total, 12 up, 11 in 2024-09-11T03:39:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1170969973' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:46 smithi175 ceph-mon[30894]: osdmap e169: 12 total, 12 up, 11 in 2024-09-11T03:39:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:46 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:46 smithi175 ceph-mon[30894]: osdmap e170: 12 total, 12 up, 11 in 2024-09-11T03:39:47.233 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:47.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:47.256 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:47.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:47.358 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:47.375 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:39:47.375 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:39:47.380 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:47.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:47.381 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:47.481 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:47.503 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:47.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:47.504 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:47.565 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:47.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:47.631 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:47.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:47.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:47.733 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:47.755 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:47.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:47.756 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:47.854 INFO:tasks.rados.rados.2.smithi175.stdout:393: done (1 left) 2024-09-11T03:39:47.854 INFO:tasks.rados.rados.2.smithi175.stdout:394: snap_remove snap 29 2024-09-11T03:39:47.857 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:47.879 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:47.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:47.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:47.939 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:47 smithi084 ceph-mon[27254]: pgmap v409: 17 pgs: 17 active+clean; 153 MiB data, 606 MiB used, 983 GiB / 983 GiB avail; 5.8 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-11T03:39:47.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:48.002 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:48.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:48.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:48.038 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:39:48.039 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:39:48.056 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:39:48.056 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:39:48.072 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:39:48.072 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:39:48.088 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:39:48.088 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:39:48.104 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:39:48.105 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:39:48.106 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:48.121 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:39:48.122 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:39:48.127 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:48.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:48.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:48.139 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:39:48.139 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:39:48.156 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:39:48.157 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:39:48.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:47 smithi121 ceph-mon[30729]: pgmap v409: 17 pgs: 17 active+clean; 153 MiB data, 606 MiB used, 983 GiB / 983 GiB avail; 5.8 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-11T03:39:48.174 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:39:48.174 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:39:48.191 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:39:48.191 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:39:48.208 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:39:48.208 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:39:48.224 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:39:48.224 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:39:48.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:47 smithi175 ceph-mon[30894]: pgmap v409: 17 pgs: 17 active+clean; 153 MiB data, 606 MiB used, 983 GiB / 983 GiB avail; 5.8 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-11T03:39:48.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:48.255 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:48.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:48.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:48.357 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:48.378 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:48.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:48.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:48.444 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T03:39:48.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:48.506 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:48.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:48.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:48.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:48.633 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:48.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:48.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:48.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:48.756 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:48.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:48.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:48.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:39:48.861 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:48.866 INFO:tasks.rados.rados.2.smithi175.stdout:394: done (1 left) 2024-09-11T03:39:48.866 INFO:tasks.rados.rados.2.smithi175.stdout:395: delete oid 27 current snap is 37 2024-09-11T03:39:48.867 INFO:tasks.rados.rados.2.smithi175.stdout:390: finishing write tid 4 to smithi17557870-23 2024-09-11T03:39:48.867 INFO:tasks.rados.rados.2.smithi175.stdout:390: finishing write tid 5 to smithi17557870-23 2024-09-11T03:39:48.867 INFO:tasks.rados.rados.2.smithi175.stdout:390: oid 23 updating version 0 to 70 2024-09-11T03:39:48.867 INFO:tasks.rados.rados.2.smithi175.stdout:390: oid 23 updating version 70 to 72 2024-09-11T03:39:48.867 INFO:tasks.rados.rados.2.smithi175.stdout:390: oid 23 version 72 is already newer than 69 2024-09-11T03:39:48.867 INFO:tasks.rados.rados.2.smithi175.stdout:390: oid 23 version 72 is already newer than 71 2024-09-11T03:39:48.867 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 72 (ObjNum 165 snap 37 seq_num 165) dirty exists 2024-09-11T03:39:48.867 INFO:tasks.rados.rados.2.smithi175.stdout:390: left oid 23 (ObjNum 165 snap 37 seq_num 165) 2024-09-11T03:39:48.868 INFO:tasks.rados.rados.2.smithi175.stdout:390: done (1 left) 2024-09-11T03:39:48.868 INFO:tasks.rados.rados.2.smithi175.stdout:395: done (0 left) 2024-09-11T03:39:48.868 INFO:tasks.rados.rados.2.smithi175.stdout:396: delete oid 5 current snap is 37 2024-09-11T03:39:48.870 INFO:tasks.rados.rados.2.smithi175.stdout:396: done (0 left) 2024-09-11T03:39:48.870 INFO:tasks.rados.rados.2.smithi175.stdout:397: snap_remove snap 34 2024-09-11T03:39:48.888 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:48.889 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:48.951 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:39:48.951 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:39:48.952 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-11T03:39:48.952 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:39:48.990 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:49.015 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:49.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:49.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:49.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:48 smithi084 ceph-mon[27254]: osdmap e171: 12 total, 12 up, 11 in 2024-09-11T03:39:49.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:48 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4203926766' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:49.117 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:49.131 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:49.142 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:49.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:49.143 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:49.144 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:49.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:48 smithi121 ceph-mon[30729]: osdmap e171: 12 total, 12 up, 11 in 2024-09-11T03:39:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:48 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4203926766' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:49.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:48 smithi175 ceph-mon[30894]: osdmap e171: 12 total, 12 up, 11 in 2024-09-11T03:39:49.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:48 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4203926766' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:49.244 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:49.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:49.271 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:49.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:49.372 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:49.395 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:49.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:49.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:49.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:49.520 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:49.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:49.622 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:49.645 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:49.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:49.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:49.747 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:49.768 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:49.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:49.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:49.867 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:39:49.867 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:39:49.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:39:49.867 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:39:49.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.867 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:43 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'25, mlcod=20'22)", 2024-09-11T03:39:49.867 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.202514+0000", 2024-09-11T03:39:49.867 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 365.66390388100001, 2024-09-11T03:39:49.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030994870000000002, 2024-09-11T03:39:49.867 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.867 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.868 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.868 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202514+0000", 2024-09-11T03:39:49.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202514+0000", 2024-09-11T03:39:49.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202514+0000", 2024-09-11T03:39:49.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7899999999999998e-07 2024-09-11T03:39:49.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202519+0000", 2024-09-11T03:39:49.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1949999999999996e-06 2024-09-11T03:39:49.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202531+0000", 2024-09-11T03:39:49.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1975e-05 2024-09-11T03:39:49.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202537+0000", 2024-09-11T03:39:49.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9649999999999996e-06 2024-09-11T03:39:49.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202581+0000", 2024-09-11T03:39:49.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4373999999999999e-05 2024-09-11T03:39:49.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202596+0000", 2024-09-11T03:39:49.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5367999999999999e-05 2024-09-11T03:39:49.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.205590+0000", 2024-09-11T03:39:49.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029938579999999998 2024-09-11T03:39:49.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.205613+0000", 2024-09-11T03:39:49.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3173e-05 2024-09-11T03:39:49.872 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.872 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.872 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.872 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:46 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'28, mlcod=20'23)", 2024-09-11T03:39:49.872 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.206111+0000", 2024-09-11T03:39:49.872 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 365.66030663700002, 2024-09-11T03:39:49.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026967599999999999, 2024-09-11T03:39:49.872 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.872 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.872 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.872 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206111+0000", 2024-09-11T03:39:49.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206111+0000", 2024-09-11T03:39:49.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206112+0000", 2024-09-11T03:39:49.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0500000000000003e-07 2024-09-11T03:39:49.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206125+0000", 2024-09-11T03:39:49.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2908000000000001e-05 2024-09-11T03:39:49.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206137+0000", 2024-09-11T03:39:49.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.187e-05 2024-09-11T03:39:49.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206144+0000", 2024-09-11T03:39:49.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7919999999999993e-06 2024-09-11T03:39:49.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206185+0000", 2024-09-11T03:39:49.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0803999999999998e-05 2024-09-11T03:39:49.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206197+0000", 2024-09-11T03:39:49.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1962e-05 2024-09-11T03:39:49.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.208764+0000", 2024-09-11T03:39:49.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002566652 2024-09-11T03:39:49.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.208808+0000", 2024-09-11T03:39:49.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4166999999999999e-05 2024-09-11T03:39:49.876 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.876 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.877 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.877 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:48 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'30, mlcod=20'27)", 2024-09-11T03:39:49.877 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.209747+0000", 2024-09-11T03:39:49.877 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 365.65667069099999, 2024-09-11T03:39:49.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030822760000000001, 2024-09-11T03:39:49.877 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.877 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.877 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.877 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209747+0000", 2024-09-11T03:39:49.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209747+0000", 2024-09-11T03:39:49.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209748+0000", 2024-09-11T03:39:49.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4399999999999999e-07 2024-09-11T03:39:49.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209753+0000", 2024-09-11T03:39:49.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6130000000000003e-06 2024-09-11T03:39:49.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209772+0000", 2024-09-11T03:39:49.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8493000000000001e-05 2024-09-11T03:39:49.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209786+0000", 2024-09-11T03:39:49.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4399000000000001e-05 2024-09-11T03:39:49.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209832+0000", 2024-09-11T03:39:49.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5556999999999998e-05 2024-09-11T03:39:49.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209852+0000", 2024-09-11T03:39:49.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0262999999999999e-05 2024-09-11T03:39:49.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212794+0000", 2024-09-11T03:39:49.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029415259999999999 2024-09-11T03:39:49.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212829+0000", 2024-09-11T03:39:49.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5680999999999999e-05 2024-09-11T03:39:49.881 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.881 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.882 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.882 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:49 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'31, mlcod=20'27)", 2024-09-11T03:39:49.882 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.210165+0000", 2024-09-11T03:39:49.882 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 365.65625283399999, 2024-09-11T03:39:49.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:49.883 INFO:tasks.rados.rados.2.smithi175.stdout:397: done (0 left) 2024-09-11T03:39:49.883 INFO:tasks.rados.rados.2.smithi175.stdout:398: snap_create 2024-09-11T03:39:49.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028269739999999999, 2024-09-11T03:39:49.884 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.884 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.884 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.884 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210165+0000", 2024-09-11T03:39:49.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210165+0000", 2024-09-11T03:39:49.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210166+0000", 2024-09-11T03:39:49.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5300000000000004e-07 2024-09-11T03:39:49.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210171+0000", 2024-09-11T03:39:49.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1039999999999998e-06 2024-09-11T03:39:49.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210183+0000", 2024-09-11T03:39:49.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1848000000000001e-05 2024-09-11T03:39:49.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210189+0000", 2024-09-11T03:39:49.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4459999999999998e-06 2024-09-11T03:39:49.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210267+0000", 2024-09-11T03:39:49.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7540999999999995e-05 2024-09-11T03:39:49.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210288+0000", 2024-09-11T03:39:49.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0992000000000002e-05 2024-09-11T03:39:49.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212958+0000", 2024-09-11T03:39:49.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002670455 2024-09-11T03:39:49.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212992+0000", 2024-09-11T03:39:49.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3935e-05 2024-09-11T03:39:49.888 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.888 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.888 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.888 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:359 2.5 e114/95 2:a6d17fc0:::smithi17557870-13:head v 114'40, mlcod=110'38)", 2024-09-11T03:39:49.888 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:50.842632+0000", 2024-09-11T03:39:49.888 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 59.023785758000002, 2024-09-11T03:39:49.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002717297, 2024-09-11T03:39:49.889 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.889 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.889 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.889 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.842632+0000", 2024-09-11T03:39:49.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.842632+0000", 2024-09-11T03:39:49.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.842632+0000", 2024-09-11T03:39:49.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6100000000000002e-07 2024-09-11T03:39:49.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.844452+0000", 2024-09-11T03:39:49.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018200619999999999 2024-09-11T03:39:49.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.844463+0000", 2024-09-11T03:39:49.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0351e-05 2024-09-11T03:39:49.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.844470+0000", 2024-09-11T03:39:49.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6240000000000002e-06 2024-09-11T03:39:49.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.844510+0000", 2024-09-11T03:39:49.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9703999999999998e-05 2024-09-11T03:39:49.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.844526+0000", 2024-09-11T03:39:49.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6172e-05 2024-09-11T03:39:49.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.845322+0000", 2024-09-11T03:39:49.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00079569999999999999 2024-09-11T03:39:49.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.845349+0000", 2024-09-11T03:39:49.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7423000000000002e-05 2024-09-11T03:39:49.892 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.892 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.893 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.893 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:360 2.5 e114/95 2:a6d17fc0:::smithi17557870-13:head v 114'41, mlcod=112'39)", 2024-09-11T03:39:49.893 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:50.850576+0000", 2024-09-11T03:39:49.893 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 59.015841975999997, 2024-09-11T03:39:49.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020173880000000002, 2024-09-11T03:39:49.893 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.893 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.893 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.893 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.850576+0000", 2024-09-11T03:39:49.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.850576+0000", 2024-09-11T03:39:49.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.850576+0000", 2024-09-11T03:39:49.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9299999999999997e-07 2024-09-11T03:39:49.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.851595+0000", 2024-09-11T03:39:49.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001018892 2024-09-11T03:39:49.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.851612+0000", 2024-09-11T03:39:49.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6557000000000001e-05 2024-09-11T03:39:49.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.851618+0000", 2024-09-11T03:39:49.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1589999999999998e-06 2024-09-11T03:39:49.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.851681+0000", 2024-09-11T03:39:49.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2420000000000002e-05 2024-09-11T03:39:49.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.851705+0000", 2024-09-11T03:39:49.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4363e-05 2024-09-11T03:39:49.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.852568+0000", 2024-09-11T03:39:49.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000863272 2024-09-11T03:39:49.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.897 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.852593+0000", 2024-09-11T03:39:49.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5032e-05 2024-09-11T03:39:49.897 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.897 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.897 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.897 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:394 2.5 e114/95 2:a6d17fc0:::smithi17557870-13:head v 114'43, mlcod=114'41)", 2024-09-11T03:39:49.897 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:51.075167+0000", 2024-09-11T03:39:49.897 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 58.791250462000001, 2024-09-11T03:39:49.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020763550000000002, 2024-09-11T03:39:49.897 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.898 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.898 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.898 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.075167+0000", 2024-09-11T03:39:49.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.075167+0000", 2024-09-11T03:39:49.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.075169+0000", 2024-09-11T03:39:49.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5030000000000001e-06 2024-09-11T03:39:49.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.076292+0000", 2024-09-11T03:39:49.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00112346 2024-09-11T03:39:49.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.076310+0000", 2024-09-11T03:39:49.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7558e-05 2024-09-11T03:39:49.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.076319+0000", 2024-09-11T03:39:49.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6789999999999994e-06 2024-09-11T03:39:49.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.076401+0000", 2024-09-11T03:39:49.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2595000000000005e-05 2024-09-11T03:39:49.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.076426+0000", 2024-09-11T03:39:49.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5338e-05 2024-09-11T03:39:49.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.077210+0000", 2024-09-11T03:39:49.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000783265 2024-09-11T03:39:49.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:51.077244+0000", 2024-09-11T03:39:49.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3957e-05 2024-09-11T03:39:49.902 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.902 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.902 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.902 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:546 2.5 e125/95 2:a6e3bbd9:::smithi17557870-38:head v 125'58, mlcod=124'56)", 2024-09-11T03:39:49.902 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:01.828623+0000", 2024-09-11T03:39:49.902 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 48.037794759000001, 2024-09-11T03:39:49.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001735505, 2024-09-11T03:39:49.902 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.902 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.903 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.903 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.828623+0000", 2024-09-11T03:39:49.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.828623+0000", 2024-09-11T03:39:49.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.828624+0000", 2024-09-11T03:39:49.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.319e-06 2024-09-11T03:39:49.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.829511+0000", 2024-09-11T03:39:49.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00088681600000000001 2024-09-11T03:39:49.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.829531+0000", 2024-09-11T03:39:49.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9796000000000002e-05 2024-09-11T03:39:49.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.829540+0000", 2024-09-11T03:39:49.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3610000000000007e-06 2024-09-11T03:39:49.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.829600+0000", 2024-09-11T03:39:49.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9688999999999997e-05 2024-09-11T03:39:49.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.906 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.829620+0000", 2024-09-11T03:39:49.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9771000000000001e-05 2024-09-11T03:39:49.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.906 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.830330+0000", 2024-09-11T03:39:49.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070980900000000005 2024-09-11T03:39:49.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.830359+0000", 2024-09-11T03:39:49.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8943999999999999e-05 2024-09-11T03:39:49.907 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.907 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.907 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.907 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:547 2.5 e125/95 2:a6e3bbd9:::smithi17557870-38:head v 125'59, mlcod=125'57)", 2024-09-11T03:39:49.907 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:01.834898+0000", 2024-09-11T03:39:49.907 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 48.031520245000003, 2024-09-11T03:39:49.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018251249999999999, 2024-09-11T03:39:49.907 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.908 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.908 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.908 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.834898+0000", 2024-09-11T03:39:49.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.834898+0000", 2024-09-11T03:39:49.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.834898+0000", 2024-09-11T03:39:49.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9800000000000002e-07 2024-09-11T03:39:49.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.835827+0000", 2024-09-11T03:39:49.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00092875100000000001 2024-09-11T03:39:49.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.835843+0000", 2024-09-11T03:39:49.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5783000000000001e-05 2024-09-11T03:39:49.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.835851+0000", 2024-09-11T03:39:49.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0870000000000007e-06 2024-09-11T03:39:49.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.835911+0000", 2024-09-11T03:39:49.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9821999999999997e-05 2024-09-11T03:39:49.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.835937+0000", 2024-09-11T03:39:49.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6032000000000001e-05 2024-09-11T03:39:49.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.911 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.836691+0000", 2024-09-11T03:39:49.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00075364599999999996 2024-09-11T03:39:49.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.911 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:01.836723+0000", 2024-09-11T03:39:49.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2106000000000001e-05 2024-09-11T03:39:49.911 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.911 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.911 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.912 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:554 2.c e127/95 2:3603e322:::smithi17557870-42:head v 127'28, mlcod=123'26)", 2024-09-11T03:39:49.912 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:03.829907+0000", 2024-09-11T03:39:49.912 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 46.036510927000002, 2024-09-11T03:39:49.912 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019573770000000002, 2024-09-11T03:39:49.912 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.912 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.912 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.912 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.912 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.912 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.829907+0000", 2024-09-11T03:39:49.912 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.829907+0000", 2024-09-11T03:39:49.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.829907+0000", 2024-09-11T03:39:49.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6800000000000002e-07 2024-09-11T03:39:49.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.830867+0000", 2024-09-11T03:39:49.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00095966499999999995 2024-09-11T03:39:49.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.830879+0000", 2024-09-11T03:39:49.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2228999999999999e-05 2024-09-11T03:39:49.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.830886+0000", 2024-09-11T03:39:49.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3760000000000002e-06 2024-09-11T03:39:49.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.830953+0000", 2024-09-11T03:39:49.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7053999999999999e-05 2024-09-11T03:39:49.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.830975+0000", 2024-09-11T03:39:49.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1710000000000001e-05 2024-09-11T03:39:49.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.831832+0000", 2024-09-11T03:39:49.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00085665000000000003 2024-09-11T03:39:49.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.831864+0000", 2024-09-11T03:39:49.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2425000000000002e-05 2024-09-11T03:39:49.916 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.916 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.916 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.916 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:555 2.c e127/95 2:3603e322:::smithi17557870-42:head v 127'29, mlcod=123'27)", 2024-09-11T03:39:49.916 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:03.836337+0000", 2024-09-11T03:39:49.916 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 46.030080376000001, 2024-09-11T03:39:49.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001891177, 2024-09-11T03:39:49.916 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.916 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.917 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.917 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.917 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.836337+0000", 2024-09-11T03:39:49.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.917 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.836337+0000", 2024-09-11T03:39:49.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.836338+0000", 2024-09-11T03:39:49.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1399999999999996e-07 2024-09-11T03:39:49.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.837390+0000", 2024-09-11T03:39:49.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010520939999999999 2024-09-11T03:39:49.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.837401+0000", 2024-09-11T03:39:49.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0570999999999999e-05 2024-09-11T03:39:49.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.837406+0000", 2024-09-11T03:39:49.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4029999999999998e-06 2024-09-11T03:39:49.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.837447+0000", 2024-09-11T03:39:49.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0407999999999997e-05 2024-09-11T03:39:49.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.837461+0000", 2024-09-11T03:39:49.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4297000000000001e-05 2024-09-11T03:39:49.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.838194+0000", 2024-09-11T03:39:49.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00073251900000000005 2024-09-11T03:39:49.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:03.838229+0000", 2024-09-11T03:39:49.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5070999999999997e-05 2024-09-11T03:39:49.920 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.920 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.920 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.921 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:578 2.5 e128/95 2:a6e3bbd9:::smithi17557870-38:head v 128'62, mlcod=125'59)", 2024-09-11T03:39:49.921 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:04.863301+0000", 2024-09-11T03:39:49.921 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 45.003117125000003, 2024-09-11T03:39:49.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019588119999999999, 2024-09-11T03:39:49.921 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.921 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.921 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.921 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.863301+0000", 2024-09-11T03:39:49.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.863301+0000", 2024-09-11T03:39:49.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.863301+0000", 2024-09-11T03:39:49.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.92e-07 2024-09-11T03:39:49.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.864151+0000", 2024-09-11T03:39:49.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00085058200000000001 2024-09-11T03:39:49.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.864163+0000", 2024-09-11T03:39:49.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.187e-05 2024-09-11T03:39:49.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.864173+0000", 2024-09-11T03:39:49.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2790000000000004e-06 2024-09-11T03:39:49.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.864224+0000", 2024-09-11T03:39:49.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1771999999999998e-05 2024-09-11T03:39:49.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.864242+0000", 2024-09-11T03:39:49.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7946999999999999e-05 2024-09-11T03:39:49.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.865226+0000", 2024-09-11T03:39:49.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00098390200000000004 2024-09-11T03:39:49.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:04.865259+0000", 2024-09-11T03:39:49.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3268000000000001e-05 2024-09-11T03:39:49.925 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.925 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.925 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.925 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:752 2.c e138/95 2:3c007163:::smithi17557870-35:head v 138'40, mlcod=135'37)", 2024-09-11T03:39:49.925 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:39:49.925 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 34.850844015, 2024-09-11T03:39:49.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084184749999999999, 2024-09-11T03:39:49.925 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.925 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.925 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.925 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:39:49.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:39:49.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:39:49.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8400000000000001e-07 2024-09-11T03:39:49.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021686+0000", 2024-09-11T03:39:49.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061118680000000003 2024-09-11T03:39:49.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021706+0000", 2024-09-11T03:39:49.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9942999999999999e-05 2024-09-11T03:39:49.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021717+0000", 2024-09-11T03:39:49.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.098e-05 2024-09-11T03:39:49.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021806+0000", 2024-09-11T03:39:49.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9197999999999995e-05 2024-09-11T03:39:49.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021834+0000", 2024-09-11T03:39:49.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8251000000000001e-05 2024-09-11T03:39:49.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.023931+0000", 2024-09-11T03:39:49.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020966639999999998 2024-09-11T03:39:49.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.023992+0000", 2024-09-11T03:39:49.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1387e-05 2024-09-11T03:39:49.929 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.929 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.929 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.929 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:834 2.1 e145/95 2:8638b65e:::smithi17557870-5:head v 145'21, mlcod=138'18)", 2024-09-11T03:39:49.930 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:21.830361+0000", 2024-09-11T03:39:49.930 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 28.036057163999999, 2024-09-11T03:39:49.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024660149999999998, 2024-09-11T03:39:49.930 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.930 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.930 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.930 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:21.830361+0000", 2024-09-11T03:39:49.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:21.830361+0000", 2024-09-11T03:39:49.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:21.830361+0000", 2024-09-11T03:39:49.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8200000000000001e-07 2024-09-11T03:39:49.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:21.831742+0000", 2024-09-11T03:39:49.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0013808920000000001 2024-09-11T03:39:49.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:21.831761+0000", 2024-09-11T03:39:49.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9174e-05 2024-09-11T03:39:49.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:21.831776+0000", 2024-09-11T03:39:49.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5430999999999998e-05 2024-09-11T03:39:49.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:21.831855+0000", 2024-09-11T03:39:49.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8665999999999995e-05 2024-09-11T03:39:49.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:21.831876+0000", 2024-09-11T03:39:49.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0720000000000002e-05 2024-09-11T03:39:49.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:21.832794+0000", 2024-09-11T03:39:49.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00091815400000000002 2024-09-11T03:39:49.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:21.832827+0000", 2024-09-11T03:39:49.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2696e-05 2024-09-11T03:39:49.934 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.934 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.934 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.934 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:902 2.8 e157/95 2:1bdf1a36:::smithi17557870-36:head v 157'39, mlcod=136'36)", 2024-09-11T03:39:49.934 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:33.901852+0000", 2024-09-11T03:39:49.934 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 15.964566214, 2024-09-11T03:39:49.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017682189999999999, 2024-09-11T03:39:49.934 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.934 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.934 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.935 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.901852+0000", 2024-09-11T03:39:49.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.901852+0000", 2024-09-11T03:39:49.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.901852+0000", 2024-09-11T03:39:49.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.72e-07 2024-09-11T03:39:49.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.902810+0000", 2024-09-11T03:39:49.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00095792399999999997 2024-09-11T03:39:49.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.902818+0000", 2024-09-11T03:39:49.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7960000000000001e-06 2024-09-11T03:39:49.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.902823+0000", 2024-09-11T03:39:49.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8319999999999996e-06 2024-09-11T03:39:49.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.902856+0000", 2024-09-11T03:39:49.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2759000000000003e-05 2024-09-11T03:39:49.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.902868+0000", 2024-09-11T03:39:49.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.189e-05 2024-09-11T03:39:49.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.903588+0000", 2024-09-11T03:39:49.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00072041499999999997 2024-09-11T03:39:49.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.903620+0000", 2024-09-11T03:39:49.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1430999999999998e-05 2024-09-11T03:39:49.938 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.938 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.938 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.939 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:903 2.8 e157/95 2:1bdf1a36:::smithi17557870-36:head v 157'40, mlcod=136'37)", 2024-09-11T03:39:49.939 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:33.908357+0000", 2024-09-11T03:39:49.939 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 15.958060786000001, 2024-09-11T03:39:49.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001808851, 2024-09-11T03:39:49.939 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.939 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.939 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.939 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.908357+0000", 2024-09-11T03:39:49.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.908357+0000", 2024-09-11T03:39:49.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.908358+0000", 2024-09-11T03:39:49.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.68e-07 2024-09-11T03:39:49.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.909349+0000", 2024-09-11T03:39:49.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099184899999999994 2024-09-11T03:39:49.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.909360+0000", 2024-09-11T03:39:49.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.081e-05 2024-09-11T03:39:49.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.909365+0000", 2024-09-11T03:39:49.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0730000000000004e-06 2024-09-11T03:39:49.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.909413+0000", 2024-09-11T03:39:49.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7264e-05 2024-09-11T03:39:49.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.909428+0000", 2024-09-11T03:39:49.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5142e-05 2024-09-11T03:39:49.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.910141+0000", 2024-09-11T03:39:49.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071283499999999999 2024-09-11T03:39:49.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:33.910166+0000", 2024-09-11T03:39:49.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.531e-05 2024-09-11T03:39:49.944 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.944 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.944 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.944 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:947 2.8 e163/95 2:1bdf1a36:::smithi17557870-36:head v 163'45, mlcod=157'41)", 2024-09-11T03:39:49.944 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:39.852028+0000", 2024-09-11T03:39:49.944 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 10.014389810999999, 2024-09-11T03:39:49.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025214819999999998, 2024-09-11T03:39:49.944 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.945 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.945 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.945 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.852028+0000", 2024-09-11T03:39:49.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.852028+0000", 2024-09-11T03:39:49.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.852028+0000", 2024-09-11T03:39:49.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9000000000000001e-07 2024-09-11T03:39:49.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.853313+0000", 2024-09-11T03:39:49.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012846240000000001 2024-09-11T03:39:49.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.853327+0000", 2024-09-11T03:39:49.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3865e-05 2024-09-11T03:39:49.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.853336+0000", 2024-09-11T03:39:49.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3519999999999996e-06 2024-09-11T03:39:49.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.853397+0000", 2024-09-11T03:39:49.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1432000000000004e-05 2024-09-11T03:39:49.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.853416+0000", 2024-09-11T03:39:49.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.838e-05 2024-09-11T03:39:49.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.854515+0000", 2024-09-11T03:39:49.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001099395 2024-09-11T03:39:49.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.854549+0000", 2024-09-11T03:39:49.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4243999999999999e-05 2024-09-11T03:39:49.948 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.948 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.949 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.949 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:948 2.8 e163/95 2:1bdf1a36:::smithi17557870-36:head v 163'46, mlcod=158'43)", 2024-09-11T03:39:49.949 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:39.860671+0000", 2024-09-11T03:39:49.949 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 10.005746708, 2024-09-11T03:39:49.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020739399999999998, 2024-09-11T03:39:49.949 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.949 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.949 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.949 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.860671+0000", 2024-09-11T03:39:49.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.860671+0000", 2024-09-11T03:39:49.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.860673+0000", 2024-09-11T03:39:49.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.573e-06 2024-09-11T03:39:49.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.861695+0000", 2024-09-11T03:39:49.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010219840000000001 2024-09-11T03:39:49.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.861710+0000", 2024-09-11T03:39:49.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5074e-05 2024-09-11T03:39:49.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.861718+0000", 2024-09-11T03:39:49.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2670000000000006e-06 2024-09-11T03:39:49.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.861812+0000", 2024-09-11T03:39:49.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3738000000000003e-05 2024-09-11T03:39:49.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.861856+0000", 2024-09-11T03:39:49.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3996999999999999e-05 2024-09-11T03:39:49.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.953 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.862702+0000", 2024-09-11T03:39:49.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00084617899999999996 2024-09-11T03:39:49.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.953 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.862745+0000", 2024-09-11T03:39:49.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3127999999999998e-05 2024-09-11T03:39:49.953 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.953 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.953 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.954 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:964 2.8 e163/95 2:1984a3a9:::smithi17557870-23:head v 163'49, mlcod=163'46)", 2024-09-11T03:39:49.954 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:39.895494+0000", 2024-09-11T03:39:49.954 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.9709235019999998, 2024-09-11T03:39:49.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0023154959999999998, 2024-09-11T03:39:49.954 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.954 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.954 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.954 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.954 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.954 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.895494+0000", 2024-09-11T03:39:49.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.895494+0000", 2024-09-11T03:39:49.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.895498+0000", 2024-09-11T03:39:49.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2839999999999998e-06 2024-09-11T03:39:49.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.896562+0000", 2024-09-11T03:39:49.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010643600000000001 2024-09-11T03:39:49.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.896583+0000", 2024-09-11T03:39:49.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1121999999999999e-05 2024-09-11T03:39:49.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.896596+0000", 2024-09-11T03:39:49.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2916e-05 2024-09-11T03:39:49.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.896690+0000", 2024-09-11T03:39:49.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3944999999999996e-05 2024-09-11T03:39:49.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.896715+0000", 2024-09-11T03:39:49.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5051000000000001e-05 2024-09-11T03:39:49.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.897754+0000", 2024-09-11T03:39:49.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010387300000000001 2024-09-11T03:39:49.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.897810+0000", 2024-09-11T03:39:49.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6088000000000002e-05 2024-09-11T03:39:49.958 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.958 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.959 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.959 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.959 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:965 2.8 e163/95 2:1984a3a9:::smithi17557870-23:head v 163'50, mlcod=163'47)", 2024-09-11T03:39:49.959 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:39.901372+0000", 2024-09-11T03:39:49.959 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.9650455020000006, 2024-09-11T03:39:49.959 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019173269999999999, 2024-09-11T03:39:49.959 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:49.959 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:49.960 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:49.960 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:49.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.960 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:49.960 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.901372+0000", 2024-09-11T03:39:49.960 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.960 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.960 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:49.960 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.901372+0000", 2024-09-11T03:39:49.960 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:49.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:49.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.901374+0000", 2024-09-11T03:39:49.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6440000000000001e-06 2024-09-11T03:39:49.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:49.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.902447+0000", 2024-09-11T03:39:49.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001073411 2024-09-11T03:39:49.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:49.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.902473+0000", 2024-09-11T03:39:49.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5636e-05 2024-09-11T03:39:49.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:49.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.902488+0000", 2024-09-11T03:39:49.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5494000000000001e-05 2024-09-11T03:39:49.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:49.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.902583+0000", 2024-09-11T03:39:49.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4716999999999995e-05 2024-09-11T03:39:49.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:49.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.902607+0000", 2024-09-11T03:39:49.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3934000000000002e-05 2024-09-11T03:39:49.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:49.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.903250+0000", 2024-09-11T03:39:49.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00064266799999999999 2024-09-11T03:39:49.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:49.964 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:49.964 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:49.964 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.903290+0000", 2024-09-11T03:39:49.964 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9823000000000001e-05 2024-09-11T03:39:49.964 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.964 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.964 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.964 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:49.964 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:49.964 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:39:49.965 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:49 smithi084 ceph-mon[27254]: pgmap v412: 17 pgs: 17 active+clean; 150 MiB data, 603 MiB used, 983 GiB / 983 GiB avail; 5.8 MiB/s rd, 1.4 MiB/s wr, 9 op/s 2024-09-11T03:39:49.965 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:49 smithi084 ceph-mon[27254]: osdmap e172: 12 total, 12 up, 11 in 2024-09-11T03:39:49.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:49.966 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:49.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:50.067 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:50.087 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:50.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:50.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:49 smithi121 ceph-mon[30729]: pgmap v412: 17 pgs: 17 active+clean; 150 MiB data, 603 MiB used, 983 GiB / 983 GiB avail; 5.8 MiB/s rd, 1.4 MiB/s wr, 9 op/s 2024-09-11T03:39:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:49 smithi121 ceph-mon[30729]: osdmap e172: 12 total, 12 up, 11 in 2024-09-11T03:39:50.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:50.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:50.211 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:50.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:49 smithi175 ceph-mon[30894]: pgmap v412: 17 pgs: 17 active+clean; 150 MiB data, 603 MiB used, 983 GiB / 983 GiB avail; 5.8 MiB/s rd, 1.4 MiB/s wr, 9 op/s 2024-09-11T03:39:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:49 smithi175 ceph-mon[30894]: osdmap e172: 12 total, 12 up, 11 in 2024-09-11T03:39:50.311 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:50.337 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:50.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:50.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:50.439 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:50.465 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:50.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:50.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:50.566 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:50.593 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:50.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:50.594 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:50.695 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:50.716 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:50.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:50.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:50.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:50.839 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:50.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:50.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:50.868 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:39:50.870 INFO:tasks.rados.rados.2.smithi175.stdout:398: done (0 left) 2024-09-11T03:39:50.870 INFO:tasks.rados.rados.2.smithi175.stdout:399: delete oid 13 current snap is 38 2024-09-11T03:39:50.872 INFO:tasks.rados.rados.2.smithi175.stdout:399: done (0 left) 2024-09-11T03:39:50.872 INFO:tasks.rados.rados.2.smithi175.stdout:400: rollback oid 46 current snap is 38 2024-09-11T03:39:50.872 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 46 to 37 2024-09-11T03:39:50.873 INFO:tasks.rados.rados.2.smithi175.stdout:401: rollback oid 37 current snap is 38 2024-09-11T03:39:50.873 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 37 to 36 2024-09-11T03:39:50.873 INFO:tasks.rados.rados.2.smithi175.stdout:402: copy_from oid 19 from oid 2 current snap is 38 2024-09-11T03:39:50.873 INFO:tasks.rados.rados.2.smithi175.stdout:403: snap_create 2024-09-11T03:39:50.875 INFO:tasks.rados.rados.2.smithi175.stdout:401: finishing rollback tid 1 to smithi17557870-37 2024-09-11T03:39:50.875 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T03:39:50.876 INFO:tasks.rados.rados.2.smithi175.stdout:400: finishing rollback tid 0 to smithi17557870-46 2024-09-11T03:39:50.877 INFO:tasks.rados.rados.2.smithi175.stdout:400: finishing rollback tid 1 to smithi17557870-46 2024-09-11T03:39:50.877 INFO:tasks.rados.rados.2.smithi175.stdout:400: finishing rollback tid 2 to smithi17557870-46 2024-09-11T03:39:50.877 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 34 (ObjNum 157 snap 35 seq_num 157) dirty exists 2024-09-11T03:39:50.882 INFO:tasks.rados.rados.2.smithi175.stdout:402: finishing copy_from racing read to smithi17557870-19 2024-09-11T03:39:50.884 INFO:tasks.rados.rados.2.smithi175.stdout:402: finishing copy_from to smithi17557870-19 2024-09-11T03:39:50.884 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 73 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T03:39:50.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:50.943 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T03:39:50.967 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:50.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:50.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:51.069 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:51.093 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:51.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:51.094 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:51.110 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:39:51.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:50 smithi084 ceph-mon[27254]: osdmap e173: 12 total, 12 up, 11 in 2024-09-11T03:39:51.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:50 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2135474658' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:39:51.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:50 smithi121 ceph-mon[30729]: osdmap e173: 12 total, 12 up, 11 in 2024-09-11T03:39:51.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:50 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2135474658' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:39:51.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:51.221 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:51.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:51.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:50 smithi175 ceph-mon[30894]: osdmap e173: 12 total, 12 up, 11 in 2024-09-11T03:39:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:50 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2135474658' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:39:51.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:51.349 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:51.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:51.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:51.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:51.477 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:51.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:51.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:51.579 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:51.590 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:51.590 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:51.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:51.604 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:51.604 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:51.626 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:51.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:51.732 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:51.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:51.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:51.827 INFO:tasks.rados.rados.2.smithi175.stdout:400: done (3 left) 2024-09-11T03:39:51.827 INFO:tasks.rados.rados.2.smithi175.stdout:401: done (2 left) 2024-09-11T03:39:51.827 INFO:tasks.rados.rados.2.smithi175.stdout:402: done (1 left) 2024-09-11T03:39:51.828 INFO:tasks.rados.rados.2.smithi175.stdout:403: done (0 left) 2024-09-11T03:39:51.828 INFO:tasks.rados.rados.2.smithi175.stdout:404: copy_from oid 6 from oid 22 current snap is 39 2024-09-11T03:39:51.828 INFO:tasks.rados.rados.2.smithi175.stdout:405: rollback oid 9 current snap is 39 2024-09-11T03:39:51.828 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 9 to 36 2024-09-11T03:39:51.828 INFO:tasks.rados.rados.2.smithi175.stdout:406: write oid 33 current snap is 39 2024-09-11T03:39:51.828 INFO:tasks.rados.rados.2.smithi175.stdout:406: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-09-11T03:39:51.833 INFO:tasks.rados.rados.2.smithi175.stdout:406: writing smithi17557870-33 from 457190 to 1009286 tid 1 2024-09-11T03:39:51.834 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:51.839 INFO:tasks.rados.rados.2.smithi175.stdout:406: writing smithi17557870-33 from 1614880 to 2195921 tid 2 2024-09-11T03:39:51.843 INFO:tasks.rados.rados.2.smithi175.stdout:406: writing smithi17557870-33 from 2785791 to 2857190 tid 3 2024-09-11T03:39:51.843 INFO:tasks.rados.rados.2.smithi175.stdout:407: read oid 37 snap -1 2024-09-11T03:39:51.843 INFO:tasks.rados.rados.2.smithi175.stdout:407: expect deleted 2024-09-11T03:39:51.844 INFO:tasks.rados.rados.2.smithi175.stdout:408: write (excl) oid 48 current snap is 39 2024-09-11T03:39:51.844 INFO:tasks.rados.rados.2.smithi175.stdout:405: finishing rollback tid 1 to smithi17557870-9 2024-09-11T03:39:51.844 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 0 (ObjNum 129 snap 0 seq_num 1745188976) dirty dne 2024-09-11T03:39:51.844 INFO:tasks.rados.rados.2.smithi175.stdout:404: finishing copy_from racing read to smithi17557870-6 2024-09-11T03:39:51.844 INFO:tasks.rados.rados.2.smithi175.stdout:404: finishing copy_from to smithi17557870-6 2024-09-11T03:39:51.844 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 130 (ObjNum 164 snap 36 seq_num 164) dirty exists 2024-09-11T03:39:51.844 INFO:tasks.rados.rados.2.smithi175.stdout:406: finishing write tid 1 to smithi17557870-33 2024-09-11T03:39:51.844 INFO:tasks.rados.rados.2.smithi175.stdout:406: finishing write tid 2 to smithi17557870-33 2024-09-11T03:39:51.844 INFO:tasks.rados.rados.2.smithi175.stdout:408: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2024-09-11T03:39:51.849 INFO:tasks.rados.rados.2.smithi175.stdout:408: writing smithi17557870-48 from 473997 to 1101342 tid 1 2024-09-11T03:39:51.854 INFO:tasks.rados.rados.2.smithi175.stdout:408: writing smithi17557870-48 from 1673362 to 2114414 tid 2 2024-09-11T03:39:51.860 INFO:tasks.rados.rados.2.smithi175.stdout:408: writing smithi17557870-48 from 2529567 to 2873997 tid 3 2024-09-11T03:39:51.860 INFO:tasks.rados.rados.2.smithi175.stdout:404: done (4 left) 2024-09-11T03:39:51.860 INFO:tasks.rados.rados.2.smithi175.stdout:405: done (3 left) 2024-09-11T03:39:51.860 INFO:tasks.rados.rados.2.smithi175.stdout:409: read oid 27 snap 36 2024-09-11T03:39:51.860 INFO:tasks.rados.rados.2.smithi175.stdout:409: expect (ObjNum 139 snap 25 seq_num 139) 2024-09-11T03:39:51.861 INFO:tasks.rados.rados.2.smithi175.stdout:410: delete oid 49 current snap is 39 2024-09-11T03:39:51.861 INFO:tasks.rados.rados.2.smithi175.stdout:406: finishing write tid 3 to smithi17557870-33 2024-09-11T03:39:51.861 INFO:tasks.rados.rados.2.smithi175.stdout:406: finishing write tid 4 to smithi17557870-33 2024-09-11T03:39:51.861 INFO:tasks.rados.rados.2.smithi175.stdout:406: finishing write tid 5 to smithi17557870-33 2024-09-11T03:39:51.861 INFO:tasks.rados.rados.2.smithi175.stdout:406: oid 33 updating version 0 to 136 2024-09-11T03:39:51.861 INFO:tasks.rados.rados.2.smithi175.stdout:406: oid 33 updating version 136 to 138 2024-09-11T03:39:51.862 INFO:tasks.rados.rados.2.smithi175.stdout:406: oid 33 version 138 is already newer than 135 2024-09-11T03:39:51.862 INFO:tasks.rados.rados.2.smithi175.stdout:406: oid 33 version 138 is already newer than 137 2024-09-11T03:39:51.862 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 138 (ObjNum 170 snap 39 seq_num 170) dirty exists 2024-09-11T03:39:51.862 INFO:tasks.rados.rados.2.smithi175.stdout:406: left oid 33 (ObjNum 170 snap 39 seq_num 170) 2024-09-11T03:39:51.862 INFO:tasks.rados.rados.2.smithi175.stdout:408: finishing write tid 1 to smithi17557870-48 2024-09-11T03:39:51.862 INFO:tasks.rados.rados.2.smithi175.stdout:408: finishing write tid 2 to smithi17557870-48 2024-09-11T03:39:51.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:51.863 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:51.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:51.865 INFO:tasks.rados.rados.2.smithi175.stdout:408: finishing write tid 3 to smithi17557870-48 2024-09-11T03:39:51.865 INFO:tasks.rados.rados.2.smithi175.stdout:408: finishing write tid 4 to smithi17557870-48 2024-09-11T03:39:51.865 INFO:tasks.rados.rados.2.smithi175.stdout:408: finishing write tid 5 to smithi17557870-48 2024-09-11T03:39:51.865 INFO:tasks.rados.rados.2.smithi175.stdout:408: oid 48 updating version 0 to 81 2024-09-11T03:39:51.866 INFO:tasks.rados.rados.2.smithi175.stdout:408: oid 48 updating version 81 to 82 2024-09-11T03:39:51.866 INFO:tasks.rados.rados.2.smithi175.stdout:408: oid 48 version 82 is already newer than 79 2024-09-11T03:39:51.866 INFO:tasks.rados.rados.2.smithi175.stdout:408: oid 48 version 82 is already newer than 80 2024-09-11T03:39:51.866 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 82 (ObjNum 171 snap 39 seq_num 171) dirty exists 2024-09-11T03:39:51.866 INFO:tasks.rados.rados.2.smithi175.stdout:408: left oid 48 (ObjNum 171 snap 39 seq_num 171) 2024-09-11T03:39:51.866 INFO:tasks.rados.rados.2.smithi175.stdout:406: done (4 left) 2024-09-11T03:39:51.866 INFO:tasks.rados.rados.2.smithi175.stdout:407: done (3 left) 2024-09-11T03:39:51.867 INFO:tasks.rados.rados.2.smithi175.stdout:408: done (2 left) 2024-09-11T03:39:51.867 INFO:tasks.rados.rados.2.smithi175.stdout:410: done (1 left) 2024-09-11T03:39:51.867 INFO:tasks.rados.rados.2.smithi175.stdout:411: write (excl) oid 11 current snap is 39 2024-09-11T03:39:51.867 INFO:tasks.rados.rados.2.smithi175.stdout:411: seq_num 173 ranges {507611=777843,2073973=444721} 2024-09-11T03:39:51.869 INFO:tasks.rados.rados.2.smithi175.stdout:411: writing smithi17557870-11 from 507611 to 1285454 tid 1 2024-09-11T03:39:51.876 INFO:tasks.rados.rados.2.smithi175.stdout:411: writing smithi17557870-11 from 2073973 to 2518694 tid 2 2024-09-11T03:39:51.876 INFO:tasks.rados.rados.2.smithi175.stdout:412: snap_create 2024-09-11T03:39:51.876 INFO:tasks.rados.rados.2.smithi175.stdout:409: expect (ObjNum 139 snap 25 seq_num 139) 2024-09-11T03:39:51.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:51.989 INFO:tasks.rados.rados.2.smithi175.stdout:411: finishing write tid 1 to smithi17557870-11 2024-09-11T03:39:51.989 INFO:tasks.rados.rados.2.smithi175.stdout:411: finishing write tid 2 to smithi17557870-11 2024-09-11T03:39:51.989 INFO:tasks.rados.rados.2.smithi175.stdout:411: finishing write tid 3 to smithi17557870-11 2024-09-11T03:39:51.989 INFO:tasks.rados.rados.2.smithi175.stdout:411: finishing write tid 4 to smithi17557870-11 2024-09-11T03:39:51.989 INFO:tasks.rados.rados.2.smithi175.stdout:411: oid 11 updating version 0 to 55 2024-09-11T03:39:51.989 INFO:tasks.rados.rados.2.smithi175.stdout:411: oid 11 updating version 55 to 57 2024-09-11T03:39:51.989 INFO:tasks.rados.rados.2.smithi175.stdout:411: oid 11 version 57 is already newer than 56 2024-09-11T03:39:51.990 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 57 (ObjNum 173 snap 39 seq_num 173) dirty exists 2024-09-11T03:39:51.990 INFO:tasks.rados.rados.2.smithi175.stdout:411: left oid 11 (ObjNum 173 snap 39 seq_num 173) 2024-09-11T03:39:51.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:51.991 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:51.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:52.092 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:52.112 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:51 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:39:52.112 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:51 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:39:52.112 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:51 smithi084 ceph-mon[27254]: pgmap v415: 17 pgs: 17 active+clean; 150 MiB data, 593 MiB used, 983 GiB / 983 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-09-11T03:39:52.112 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2135474658' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:39:52.112 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:51 smithi084 ceph-mon[27254]: osdmap e174: 12 total, 12 up, 11 in 2024-09-11T03:39:52.112 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:51 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:39:52.112 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:51 smithi084 ceph-mon[27254]: 2.1 scrub ok 2024-09-11T03:39:52.112 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:51 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:52.112 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:51 smithi084 ceph-mon[27254]: osdmap e175: 12 total, 12 up, 11 in 2024-09-11T03:39:52.116 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:52.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:52.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:51 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:39:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:51 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:39:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:51 smithi121 ceph-mon[30729]: pgmap v415: 17 pgs: 17 active+clean; 150 MiB data, 593 MiB used, 983 GiB / 983 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-09-11T03:39:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2135474658' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:39:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:51 smithi121 ceph-mon[30729]: osdmap e174: 12 total, 12 up, 11 in 2024-09-11T03:39:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:51 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:39:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:51 smithi121 ceph-mon[30729]: 2.1 scrub ok 2024-09-11T03:39:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:51 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:51 smithi121 ceph-mon[30729]: osdmap e175: 12 total, 12 up, 11 in 2024-09-11T03:39:52.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:51 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:39:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:51 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:39:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:51 smithi175 ceph-mon[30894]: pgmap v415: 17 pgs: 17 active+clean; 150 MiB data, 593 MiB used, 983 GiB / 983 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-09-11T03:39:52.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2135474658' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:39:52.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:51 smithi175 ceph-mon[30894]: osdmap e174: 12 total, 12 up, 11 in 2024-09-11T03:39:52.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:51 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:39:52.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:51 smithi175 ceph-mon[30894]: 2.1 scrub ok 2024-09-11T03:39:52.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:51 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:39:52.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:51 smithi175 ceph-mon[30894]: osdmap e175: 12 total, 12 up, 11 in 2024-09-11T03:39:52.242 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:52.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:52.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:52.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:52.366 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:52.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:52.468 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:52.492 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:52.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:52.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:52.594 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:52.615 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:52.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:52.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:52.689 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:39:52.690 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:39:52.690 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:39:52.690 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:39:52.717 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:52.739 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:52.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:52.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:52.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:52.868 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:52.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:52.897 INFO:tasks.rados.rados.2.smithi175.stdout:409: done (2 left) 2024-09-11T03:39:52.897 INFO:tasks.rados.rados.2.smithi175.stdout:411: done (1 left) 2024-09-11T03:39:52.897 INFO:tasks.rados.rados.2.smithi175.stdout:412: done (0 left) 2024-09-11T03:39:52.897 INFO:tasks.rados.rados.2.smithi175.stdout:413: rollback oid 41 current snap is 40 2024-09-11T03:39:52.897 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 41 to 36 2024-09-11T03:39:52.897 INFO:tasks.rados.rados.2.smithi175.stdout:414: write (excl) oid 15 current snap is 40 2024-09-11T03:39:52.897 INFO:tasks.rados.rados.2.smithi175.stdout:414: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2024-09-11T03:39:52.902 INFO:tasks.rados.rados.2.smithi175.stdout:414: writing smithi17557870-15 from 524418 to 977510 tid 1 2024-09-11T03:39:52.909 INFO:tasks.rados.rados.2.smithi175.stdout:414: writing smithi17557870-15 from 1732455 to 2320834 tid 2 2024-09-11T03:39:52.912 INFO:tasks.rados.rados.2.smithi175.stdout:414: writing smithi17557870-15 from 2779130 to 2924418 tid 3 2024-09-11T03:39:52.913 INFO:tasks.rados.rados.2.smithi175.stdout:415: snap_remove snap 39 2024-09-11T03:39:52.970 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:52.992 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:52.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:52.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:53.094 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:53.116 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:53.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:53.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:53.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:52 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:39:53.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:52 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:39:53.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:52 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:39:53.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:52 smithi084 ceph-mon[27254]: 2.d scrub ok 2024-09-11T03:39:53.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:52 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:39:53.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:52 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:39:53.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:52 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:39:53.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:52 smithi121 ceph-mon[30729]: 2.d scrub ok 2024-09-11T03:39:53.218 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:52 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:39:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:52 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:39:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:52 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:39:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:52 smithi175 ceph-mon[30894]: 2.d scrub ok 2024-09-11T03:39:53.238 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:53.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:53.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:53.295 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T03:39:53.340 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:53.361 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:53.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:53.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:53.402 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:39:53.403 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:39:53.463 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:53.485 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:53.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:53.531 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:39:53.587 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:53.608 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:53.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:53.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:53.710 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:53.732 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:53.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:53.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:53.781 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:39:53.782 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:39:53.799 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:39:53.799 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:39:53.818 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:39:53.818 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:39:53.835 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:39:53.835 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:39:53.837 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:53.852 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:39:53.852 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:39:53.863 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:53.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:53.864 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:53.869 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:39:53.869 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:39:53.886 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:39:53.887 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:39:53.900 INFO:tasks.rados.rados.2.smithi175.stdout:415: done (2 left) 2024-09-11T03:39:53.900 INFO:tasks.rados.rados.2.smithi175.stdout:416: read oid 46 snap -1 2024-09-11T03:39:53.900 INFO:tasks.rados.rados.2.smithi175.stdout:416: expect (ObjNum 157 snap 35 seq_num 157) 2024-09-11T03:39:53.900 INFO:tasks.rados.rados.2.smithi175.stdout:417: delete oid 19 current snap is 40 2024-09-11T03:39:53.901 INFO:tasks.rados.rados.2.smithi175.stdout:413: finishing rollback tid 0 to smithi17557870-41 2024-09-11T03:39:53.901 INFO:tasks.rados.rados.2.smithi175.stdout:413: finishing rollback tid 1 to smithi17557870-41 2024-09-11T03:39:53.901 INFO:tasks.rados.rados.2.smithi175.stdout:413: finishing rollback tid 2 to smithi17557870-41 2024-09-11T03:39:53.901 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 84 (ObjNum 76 snap 7 seq_num 76) dirty exists 2024-09-11T03:39:53.901 INFO:tasks.rados.rados.2.smithi175.stdout:414: finishing write tid 1 to smithi17557870-15 2024-09-11T03:39:53.901 INFO:tasks.rados.rados.2.smithi175.stdout:414: finishing write tid 2 to smithi17557870-15 2024-09-11T03:39:53.901 INFO:tasks.rados.rados.2.smithi175.stdout:414: finishing write tid 3 to smithi17557870-15 2024-09-11T03:39:53.901 INFO:tasks.rados.rados.2.smithi175.stdout:414: finishing write tid 4 to smithi17557870-15 2024-09-11T03:39:53.901 INFO:tasks.rados.rados.2.smithi175.stdout:414: finishing write tid 5 to smithi17557870-15 2024-09-11T03:39:53.901 INFO:tasks.rados.rados.2.smithi175.stdout:414: oid 15 updating version 0 to 72 2024-09-11T03:39:53.901 INFO:tasks.rados.rados.2.smithi175.stdout:414: oid 15 version 72 is already newer than 71 2024-09-11T03:39:53.901 INFO:tasks.rados.rados.2.smithi175.stdout:414: oid 15 version 72 is already newer than 70 2024-09-11T03:39:53.901 INFO:tasks.rados.rados.2.smithi175.stdout:414: oid 15 version 72 is already newer than 69 2024-09-11T03:39:53.901 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 72 (ObjNum 174 snap 40 seq_num 174) dirty exists 2024-09-11T03:39:53.902 INFO:tasks.rados.rados.2.smithi175.stdout:414: left oid 15 (ObjNum 174 snap 40 seq_num 174) 2024-09-11T03:39:53.902 INFO:tasks.rados.rados.2.smithi175.stdout:413: done (3 left) 2024-09-11T03:39:53.902 INFO:tasks.rados.rados.2.smithi175.stdout:414: done (2 left) 2024-09-11T03:39:53.902 INFO:tasks.rados.rados.2.smithi175.stdout:417: done (1 left) 2024-09-11T03:39:53.902 INFO:tasks.rados.rados.2.smithi175.stdout:418: read oid 25 snap -1 2024-09-11T03:39:53.902 INFO:tasks.rados.rados.2.smithi175.stdout:418: expect deleted 2024-09-11T03:39:53.903 INFO:tasks.rados.rados.2.smithi175.stdout:419: delete oid 8 current snap is 40 2024-09-11T03:39:53.903 INFO:tasks.rados.rados.2.smithi175.stdout:418: done (2 left) 2024-09-11T03:39:53.903 INFO:tasks.rados.rados.2.smithi175.stdout:419: done (1 left) 2024-09-11T03:39:53.904 INFO:tasks.rados.rados.2.smithi175.stdout:420: snap_create 2024-09-11T03:39:53.905 INFO:tasks.rados.rados.2.smithi175.stdout:416: expect (ObjNum 157 snap 35 seq_num 157) 2024-09-11T03:39:53.907 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:39:53.907 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:39:53.924 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:39:53.924 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:39:53.940 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:39:53.941 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:39:53.956 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:39:53.956 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:39:53.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:53.972 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:39:53.973 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:39:53.992 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:53.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:53.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:54.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:54.119 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:54.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:54.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:53 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:39:54.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:53 smithi084 ceph-mon[27254]: 2.9 scrub ok 2024-09-11T03:39:54.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:53 smithi084 ceph-mon[27254]: pgmap v418: 17 pgs: 17 active+clean; 121 MiB data, 566 MiB used, 983 GiB / 983 GiB avail; 1.7 MiB/s rd, 492 KiB/s wr, 4 op/s 2024-09-11T03:39:54.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:53 smithi084 ceph-mon[27254]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:39:54.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:53 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:39:54.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:53 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:39:54.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:53 smithi084 ceph-mon[27254]: osdmap e176: 12 total, 12 up, 11 in 2024-09-11T03:39:54.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:53 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/673313665' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:54.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:53 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:39:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:53 smithi121 ceph-mon[30729]: 2.9 scrub ok 2024-09-11T03:39:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:53 smithi121 ceph-mon[30729]: pgmap v418: 17 pgs: 17 active+clean; 121 MiB data, 566 MiB used, 983 GiB / 983 GiB avail; 1.7 MiB/s rd, 492 KiB/s wr, 4 op/s 2024-09-11T03:39:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:53 smithi121 ceph-mon[30729]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:39:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:53 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:39:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:53 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:39:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:53 smithi121 ceph-mon[30729]: osdmap e176: 12 total, 12 up, 11 in 2024-09-11T03:39:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:53 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/673313665' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:54.220 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2024-09-11T03:39:52.710314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":126536565,"num_objects":57,"num_object_clones":13,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":6,"num_read":844,"num_read_kb":551766,"num_write":804,"num_write_kb":276412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":6182111,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1315,"ondisk_log_size":1315,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":579144,"kb_used_data":252704,"kb_used_omap":17,"kb_used_meta":326190,"kb_avail":1030617528,"statfs":{"total":1055945392128,"available":1055352348672,"internally_reserved":0,"allocated":258768896,"data_stored":260328857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":334019445},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-30380053,"num_objects":-14,"num_object_clones":-12,"num_object_copies":-42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-14,"num_whiteouts":1,"num_read":12,"num_read_kb":7084,"num_write":7,"num_write_kb":1970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000660"},"pg_stats":[{"pgid":"2.f","version":"173'77","reported_seq":472,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-11T03:39:50.869684+0000","last_change":"2024-09-11T03:39:49.868848+0000","last_active":"2024-09-11T03:39:50.869684+0000","last_peered":"2024-09-11T03:39:50.869684+0000","last_clean":"2024-09-11T03:39:50.869684+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:39:50.869684+0000","last_undegraded":"2024-09-11T03:39:50.869684+0000","last_fullsized":"2024-09-11T03:39:50.869684+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:54.927155+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.002790874,"stat_sum":{"num_bytes":8756414,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":6169,"num_write":44,"num_write_kb":16759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1e","length":"3"},{"start":"22","length":"1"},{"start":"24","length":"2"}]},{"pgid":"2.e","version":"170'98","reported_seq":482,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T03:39:47.858821+0000","last_change":"2024-09-11T03:39:47.859009+0000","last_active":"2024-09-11T03:39:47.858821+0000","last_peered":"2024-09-11T03:39:47.858821+0000","last_clean":"2024-09-11T03:39:47.858821+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:39:47.858821+0000","last_undegraded":"2024-09-11T03:39:47.858821+0000","last_fullsized":"2024-09-11T03:39:47.858821+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049520200000000003,"stat_sum":{"num_bytes":8924418,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":84,"num_read_kb":75021,"num_write":36,"num_write_kb":16951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1e","length":"1"},{"start":"20","length":"1"},{"start":"22","length":"1"},{"start":"25","length":"1"}]},{"pgid":"2.d","version":"170'130","reported_seq":505,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T03:39:47.857263+0000","last_change":"2024-09-11T03:39:47.857508+0000","last_active":"2024-09-11T03:39:47.857263+0000","last_peered":"2024-09-11T03:39:47.857263+0000","last_clean":"2024-09-11T03:39:47.857263+0000","last_became_active":"2024-09-11T03:38:30.031497+0000","last_became_peered":"2024-09-11T03:38:30.031497+0000","last_unstale":"2024-09-11T03:39:47.857263+0000","last_undegraded":"2024-09-11T03:39:47.857263+0000","last_fullsized":"2024-09-11T03:39:47.857263+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:49.649333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045566999999999998,"stat_sum":{"num_bytes":15013577,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":68,"num_read_kb":49498,"num_write":61,"num_write_kb":18830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1e","length":"1"},{"start":"20","length":"1"},{"start":"22","length":"1"},{"start":"25","length":"1"}]},{"pgid":"2.c","version":"175'47","reported_seq":462,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-11T03:39:51.862189+0000","last_change":"2024-09-11T03:39:49.868993+0000","last_active":"2024-09-11T03:39:51.862189+0000","last_peered":"2024-09-11T03:39:51.862189+0000","last_clean":"2024-09-11T03:39:51.862189+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:39:51.862189+0000","last_undegraded":"2024-09-11T03:39:51.862189+0000","last_fullsized":"2024-09-11T03:39:51.862189+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:52.157880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077442799999999997,"stat_sum":{"num_bytes":10756381,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":80,"num_read_kb":72374,"num_write":45,"num_write_kb":21687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"1f","length":"2"},{"start":"24","length":"2"}]},{"pgid":"1.0","version":"20'92","reported_seq":307,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-11T03:39:50.869165+0000","last_change":"2024-09-11T03:39:26.121012+0000","last_active":"2024-09-11T03:39:50.869165+0000","last_peered":"2024-09-11T03:39:50.869165+0000","last_clean":"2024-09-11T03:39:50.869165+0000","last_became_active":"2024-09-11T03:39:26.116345+0000","last_became_peered":"2024-09-11T03:39:26.116345+0000","last_unstale":"2024-09-11T03:39:50.869165+0000","last_undegraded":"2024-09-11T03:39:50.869165+0000","last_fullsized":"2024-09-11T03:39:50.869165+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:28.457905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"168'53","reported_seq":455,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-11T03:39:50.874891+0000","last_change":"2024-09-11T03:39:49.865044+0000","last_active":"2024-09-11T03:39:50.874891+0000","last_peered":"2024-09-11T03:39:50.874891+0000","last_clean":"2024-09-11T03:39:50.874891+0000","last_became_active":"2024-09-11T03:38:29.889091+0000","last_became_peered":"2024-09-11T03:38:29.889091+0000","last_unstale":"2024-09-11T03:39:50.874891+0000","last_undegraded":"2024-09-11T03:39:50.874891+0000","last_fullsized":"2024-09-11T03:39:50.874891+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'44","last_scrub_stamp":"2024-09-11T03:39:32.965569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:32.965569+0000","objects_scrubbed":2,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:37.779521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059293599999999996,"stat_sum":{"num_bytes":4689153,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":15199,"num_write":28,"num_write_kb":11041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1e","length":"3"},{"start":"22","length":"1"},{"start":"24","length":"2"}]},{"pgid":"2.0","version":"173'22","reported_seq":415,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-11T03:39:50.868955+0000","last_change":"2024-09-11T03:39:49.871426+0000","last_active":"2024-09-11T03:39:50.868955+0000","last_peered":"2024-09-11T03:39:50.868955+0000","last_clean":"2024-09-11T03:39:50.868955+0000","last_became_active":"2024-09-11T03:38:29.930120+0000","last_became_peered":"2024-09-11T03:38:29.930120+0000","last_unstale":"2024-09-11T03:39:50.868955+0000","last_undegraded":"2024-09-11T03:39:50.868955+0000","last_fullsized":"2024-09-11T03:39:50.868955+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'14","last_scrub_stamp":"2024-09-11T03:39:33.465057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:33.465057+0000","objects_scrubbed":1,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:43.079404+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003625671,"stat_sum":{"num_bytes":3260525,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":14,"num_read_kb":5533,"num_write":15,"num_write_kb":3881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1e","length":"3"},{"start":"22","length":"1"},{"start":"24","length":"2"}]},{"pgid":"2.1","version":"170'43","reported_seq":415,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T03:39:47.858698+0000","last_change":"2024-09-11T03:39:47.858909+0000","last_active":"2024-09-11T03:39:47.858698+0000","last_peered":"2024-09-11T03:39:47.858698+0000","last_clean":"2024-09-11T03:39:47.858698+0000","last_became_active":"2024-09-11T03:38:30.162952+0000","last_became_peered":"2024-09-11T03:38:30.162952+0000","last_unstale":"2024-09-11T03:39:47.858698+0000","last_undegraded":"2024-09-11T03:39:47.858698+0000","last_fullsized":"2024-09-11T03:39:47.858698+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057640299999999996,"stat_sum":{"num_bytes":4899112,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":19745,"num_write":23,"num_write_kb":7775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1e","length":"1"},{"start":"20","length":"1"},{"start":"22","length":"1"},{"start":"25","length":"1"}]},{"pgid":"2.2","version":"174'57","reported_seq":472,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-11T03:39:50.874954+0000","last_change":"2024-09-11T03:39:49.864900+0000","last_active":"2024-09-11T03:39:50.874954+0000","last_peered":"2024-09-11T03:39:50.874954+0000","last_clean":"2024-09-11T03:39:50.874954+0000","last_became_active":"2024-09-11T03:39:26.115670+0000","last_became_peered":"2024-09-11T03:39:26.115670+0000","last_unstale":"2024-09-11T03:39:50.874954+0000","last_undegraded":"2024-09-11T03:39:50.874954+0000","last_fullsized":"2024-09-11T03:39:50.874954+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:57.816606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048402900000000002,"stat_sum":{"num_bytes":2487436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":6,"num_write":33,"num_write_kb":15149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1e","length":"3"},{"start":"22","length":"1"},{"start":"24","length":"2"}]},{"pgid":"2.4","version":"150'55","reported_seq":457,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-11T03:39:50.868873+0000","last_change":"2024-09-11T03:39:49.864556+0000","last_active":"2024-09-11T03:39:50.868873+0000","last_peered":"2024-09-11T03:39:50.868873+0000","last_clean":"2024-09-11T03:39:50.868873+0000","last_became_active":"2024-09-11T03:38:30.163226+0000","last_became_peered":"2024-09-11T03:38:30.163226+0000","last_unstale":"2024-09-11T03:39:50.868873+0000","last_undegraded":"2024-09-11T03:39:50.868873+0000","last_fullsized":"2024-09-11T03:39:50.868873+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'55","last_scrub_stamp":"2024-09-11T03:39:40.378705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:40.378705+0000","objects_scrubbed":3,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:49.762383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052328400000000005,"stat_sum":{"num_bytes":8050520,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":18894,"num_write":43,"num_write_kb":14907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1e","length":"3"},{"start":"22","length":"1"},{"start":"24","length":"2"}]},{"pgid":"2.5","version":"174'146","reported_seq":566,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-11T03:39:50.869826+0000","last_change":"2024-09-11T03:39:49.872547+0000","last_active":"2024-09-11T03:39:50.869826+0000","last_peered":"2024-09-11T03:39:50.869826+0000","last_clean":"2024-09-11T03:39:50.869826+0000","last_became_active":"2024-09-11T03:38:30.163080+0000","last_became_peered":"2024-09-11T03:38:30.163080+0000","last_unstale":"2024-09-11T03:39:50.869826+0000","last_undegraded":"2024-09-11T03:39:50.869826+0000","last_fullsized":"2024-09-11T03:39:50.869826+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'97","last_scrub_stamp":"2024-09-11T03:39:33.455974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:33.455974+0000","objects_scrubbed":5,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:34.775078+0000","scrub_duration":2,"objects_trimmed":2,"snaptrim_duration":0.0064775659999999997,"stat_sum":{"num_bytes":14655638,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":89,"num_read_kb":70038,"num_write":85,"num_write_kb":26096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1e","length":"3"},{"start":"22","length":"1"},{"start":"24","length":"2"}]},{"pgid":"2.6","version":"174'64","reported_seq":450,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-11T03:39:50.883004+0000","last_change":"2024-09-11T03:39:49.866942+0000","last_active":"2024-09-11T03:39:50.883004+0000","last_peered":"2024-09-11T03:39:50.883004+0000","last_clean":"2024-09-11T03:39:50.883004+0000","last_became_active":"2024-09-11T03:38:29.887945+0000","last_became_peered":"2024-09-11T03:38:29.887945+0000","last_unstale":"2024-09-11T03:39:50.883004+0000","last_undegraded":"2024-09-11T03:39:50.883004+0000","last_fullsized":"2024-09-11T03:39:50.883004+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:39:55.705166+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.0029077640000000002,"stat_sum":{"num_bytes":4268945,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":28539,"num_write":22,"num_write_kb":17647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1e","length":"3"},{"start":"22","length":"1"},{"start":"24","length":"2"}]},{"pgid":"2.7","version":"173'100","reported_seq":527,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-11T03:39:51.864231+0000","last_change":"2024-09-11T03:39:49.872998+0000","last_active":"2024-09-11T03:39:51.864231+0000","last_peered":"2024-09-11T03:39:51.864231+0000","last_clean":"2024-09-11T03:39:51.864231+0000","last_became_active":"2024-09-11T03:39:26.115678+0000","last_became_peered":"2024-09-11T03:39:26.115678+0000","last_unstale":"2024-09-11T03:39:51.864231+0000","last_undegraded":"2024-09-11T03:39:51.864231+0000","last_fullsized":"2024-09-11T03:39:51.864231+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'75","last_scrub_stamp":"2024-09-11T03:39:39.753844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:39.753844+0000","objects_scrubbed":4,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:08.517822+0000","scrub_duration":2,"objects_trimmed":2,"snaptrim_duration":0.0031801619999999998,"stat_sum":{"num_bytes":4352980,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":44,"num_read_kb":18917,"num_write":56,"num_write_kb":22267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4066399,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,3],"acting":[10,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"1f","length":"2"},{"start":"24","length":"2"}]},{"pgid":"2.8","version":"173'86","reported_seq":477,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-11T03:39:50.868946+0000","last_change":"2024-09-11T03:39:49.872460+0000","last_active":"2024-09-11T03:39:50.868946+0000","last_peered":"2024-09-11T03:39:50.868946+0000","last_clean":"2024-09-11T03:39:50.868946+0000","last_became_active":"2024-09-11T03:38:30.162819+0000","last_became_peered":"2024-09-11T03:38:30.162819+0000","last_unstale":"2024-09-11T03:39:50.868946+0000","last_undegraded":"2024-09-11T03:39:50.868946+0000","last_fullsized":"2024-09-11T03:39:50.868946+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0046399010000000001,"stat_sum":{"num_bytes":10116553,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":55,"num_read_kb":49836,"num_write":50,"num_write_kb":16790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1e","length":"3"},{"start":"22","length":"1"},{"start":"24","length":"2"}]},{"pgid":"2.9","version":"172'132","reported_seq":549,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-11T03:39:50.868706+0000","last_change":"2024-09-11T03:39:49.864462+0000","last_active":"2024-09-11T03:39:50.868706+0000","last_peered":"2024-09-11T03:39:50.868706+0000","last_clean":"2024-09-11T03:39:50.868706+0000","last_became_active":"2024-09-11T03:38:29.888204+0000","last_became_peered":"2024-09-11T03:38:29.888204+0000","last_unstale":"2024-09-11T03:39:50.868706+0000","last_undegraded":"2024-09-11T03:39:50.868706+0000","last_fullsized":"2024-09-11T03:39:50.868706+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046221899999999999,"stat_sum":{"num_bytes":11882483,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":120,"num_read_kb":83667,"num_write":82,"num_write_kb":29423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1e","length":"3"},{"start":"22","length":"1"},{"start":"24","length":"2"}]},{"pgid":"2.a","version":"174'33","reported_seq":428,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-11T03:39:51.862320+0000","last_change":"2024-09-11T03:39:49.871835+0000","last_active":"2024-09-11T03:39:51.862320+0000","last_peered":"2024-09-11T03:39:51.862320+0000","last_clean":"2024-09-11T03:39:51.862320+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:39:51.862320+0000","last_undegraded":"2024-09-11T03:39:51.862320+0000","last_fullsized":"2024-09-11T03:39:51.862320+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'8","last_scrub_stamp":"2024-09-11T03:39:33.449882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:33.449882+0000","objects_scrubbed":1,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:41.169497+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0032151570000000002,"stat_sum":{"num_bytes":2638699,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":22595,"num_write":16,"num_write_kb":10633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"1f","length":"2"},{"start":"24","length":"2"}]},{"pgid":"2.b","version":"173'80","reported_seq":474,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-11T03:39:50.869696+0000","last_change":"2024-09-11T03:39:49.869494+0000","last_active":"2024-09-11T03:39:50.869696+0000","last_peered":"2024-09-11T03:39:50.869696+0000","last_clean":"2024-09-11T03:39:50.869696+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:39:50.869696+0000","last_undegraded":"2024-09-11T03:39:50.869696+0000","last_fullsized":"2024-09-11T03:39:50.869696+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0034815879999999999,"stat_sum":{"num_bytes":11193363,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":30,"num_read_kb":15671,"num_write":42,"num_write_kb":24426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1e","length":"3"},{"start":"22","length":"1"},{"start":"24","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":125946197,"num_objects":55,"num_object_clones":13,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":6,"num_read":768,"num_read_kb":551702,"num_write":681,"num_write_kb":274262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":4066399,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":239423488,"data_stored":250397101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1223,"ondisk_log_size":1223,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285600,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45048,"kb_used_data":18016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93700104,"statfs":{"total":95995035648,"available":95948906496,"internally_reserved":0,"allocated":18448384,"data_stored":19583041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514532,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37728,"kb_used_data":10700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93707424,"statfs":{"total":95995035648,"available":95956402176,"internally_reserved":0,"allocated":10956800,"data_stored":10047263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743465,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66516,"kb_used_data":39488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93678636,"statfs":{"total":95995035648,"available":95926923264,"internally_reserved":0,"allocated":40435712,"data_stored":43416116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037809,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56664,"kb_used_data":25472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93688488,"statfs":{"total":95995035648,"available":95937011712,"internally_reserved":0,"allocated":26083328,"data_stored":25090577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299444,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57776,"kb_used_data":26588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93687376,"statfs":{"total":95995035648,"available":95935873024,"internally_reserved":0,"allocated":27226112,"data_stored":27047179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":193273528377,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36712,"kb_used_data":5524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708440,"statfs":{"total":95995035648,"available":95957442560,"internally_reserved":0,"allocated":5656576,"data_stored":4765879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986243,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":50512,"kb_used_data":19324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93694640,"statfs":{"total":95995035648,"available":95943311360,"internally_reserved":0,"allocated":19787776,"data_stored":18788817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149768,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70500,"kb_used_data":39312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93674652,"statfs":{"total":95995035648,"available":95922843648,"internally_reserved":0,"allocated":40255488,"data_stored":41767723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":43052,"kb_used_data":16024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93702100,"statfs":{"total":95995035648,"available":95950950400,"internally_reserved":0,"allocated":16408576,"data_stored":17489454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53420,"kb_used_data":22228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93691732,"statfs":{"total":95995035648,"available":95940333568,"internally_reserved":0,"allocated":22761472,"data_stored":21816538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61216,"kb_used_data":30028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93683936,"statfs":{"total":95995035648,"available":95932350464,"internally_reserved":0,"allocated":30748672,"data_stored":30516270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28762112,"data_stored":29386475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20762624,"data_stored":20678585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14458880,"data_stored":16384167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38293504,"data_stored":40654256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":17801216,"data_stored":17659022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3657728,"data_stored":3627926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4079616,"data_stored":4066399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24707072,"data_stored":25388456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24084480,"data_stored":23952624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":37941248,"data_stored":41773721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8957952,"data_stored":8909310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":15917056,"data_stored":17916160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:39:54.222 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-11T03:39:54.222 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-11T03:39:54.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:53 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:39:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:53 smithi175 ceph-mon[30894]: 2.9 scrub ok 2024-09-11T03:39:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:53 smithi175 ceph-mon[30894]: pgmap v418: 17 pgs: 17 active+clean; 121 MiB data, 566 MiB used, 983 GiB / 983 GiB avail; 1.7 MiB/s rd, 492 KiB/s wr, 4 op/s 2024-09-11T03:39:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:53 smithi175 ceph-mon[30894]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:39:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:53 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:39:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:53 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:39:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:53 smithi175 ceph-mon[30894]: osdmap e176: 12 total, 12 up, 11 in 2024-09-11T03:39:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:53 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/673313665' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:39:54.248 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:54.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:54.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:54.376 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:54.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:54.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:54.478 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:54.502 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:54.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:54.503 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:54.604 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:54.626 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:54.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:54.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:54.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:54.754 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:54.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:54.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:54.858 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:39:54.881 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:54.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:54.882 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:39:54.905 INFO:tasks.rados.rados.2.smithi175.stdout:416: done (1 left) 2024-09-11T03:39:54.905 INFO:tasks.rados.rados.2.smithi175.stdout:420: done (0 left) 2024-09-11T03:39:54.905 INFO:tasks.rados.rados.2.smithi175.stdout:421: read oid 30 snap -1 2024-09-11T03:39:54.905 INFO:tasks.rados.rados.2.smithi175.stdout:421: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-11T03:39:54.905 INFO:tasks.rados.rados.2.smithi175.stdout:422: snap_create 2024-09-11T03:39:54.920 INFO:tasks.rados.rados.2.smithi175.stdout:421: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-11T03:39:54.942 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:54.942 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:54.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:55.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:55.009 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:55.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:55.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:55.136 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:55.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:55.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:54 smithi121 ceph-mon[30729]: 2.f scrub starts 2024-09-11T03:39:55.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:54 smithi121 ceph-mon[30729]: 2.f scrub ok 2024-09-11T03:39:55.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:54 smithi121 ceph-mon[30729]: osdmap e177: 12 total, 12 up, 11 in 2024-09-11T03:39:55.189 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:54 smithi084 ceph-mon[27254]: 2.f scrub starts 2024-09-11T03:39:55.189 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:54 smithi084 ceph-mon[27254]: 2.f scrub ok 2024-09-11T03:39:55.189 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:54 smithi084 ceph-mon[27254]: osdmap e177: 12 total, 12 up, 11 in 2024-09-11T03:39:55.210 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:55.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:54 smithi175 ceph-mon[30894]: 2.f scrub starts 2024-09-11T03:39:55.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:54 smithi175 ceph-mon[30894]: 2.f scrub ok 2024-09-11T03:39:55.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:54 smithi175 ceph-mon[30894]: osdmap e177: 12 total, 12 up, 11 in 2024-09-11T03:39:55.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:55.263 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:55.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:55.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:55.365 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:55.390 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:55.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:55.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:55.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:55.521 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:55.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:55.622 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:55.643 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:55.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:55.745 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:55.765 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:55.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:55.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:55.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:55.893 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:55.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:55.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:55.906 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:39:55.906 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 1 pg_num to 11 2024-09-11T03:39:55.910 INFO:tasks.rados.rados.2.smithi175.stdout:421: done (1 left) 2024-09-11T03:39:55.910 INFO:tasks.rados.rados.2.smithi175.stdout:422: done (0 left) 2024-09-11T03:39:55.910 INFO:tasks.rados.rados.2.smithi175.stdout:423: write (excl) oid 24 current snap is 42 2024-09-11T03:39:55.910 INFO:tasks.rados.rados.2.smithi175.stdout:423: seq_num 177 ranges {574839=795192,1907901=452059} 2024-09-11T03:39:55.917 INFO:tasks.rados.rados.2.smithi175.stdout:423: writing smithi17557870-24 from 574839 to 1370031 tid 1 2024-09-11T03:39:55.923 INFO:tasks.rados.rados.2.smithi175.stdout:423: writing smithi17557870-24 from 1907901 to 2359960 tid 2 2024-09-11T03:39:55.923 INFO:tasks.rados.rados.2.smithi175.stdout:424: write oid 45 current snap is 42 2024-09-11T03:39:55.923 INFO:tasks.rados.rados.2.smithi175.stdout:424: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2024-09-11T03:39:55.929 INFO:tasks.rados.rados.2.smithi175.stdout:424: writing smithi17557870-45 from 591646 to 1062087 tid 1 2024-09-11T03:39:55.935 INFO:tasks.rados.rados.2.smithi175.stdout:424: writing smithi17557870-45 from 1566383 to 2162100 tid 2 2024-09-11T03:39:55.940 INFO:tasks.rados.rados.2.smithi175.stdout:424: writing smithi17557870-45 from 2888822 to 2991646 tid 3 2024-09-11T03:39:55.940 INFO:tasks.rados.rados.2.smithi175.stdout:425: snap_remove snap 40 2024-09-11T03:39:55.994 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:56.015 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:56.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:56.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:56.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:56.143 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:56.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:56.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:56.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:55 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:39:56.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:55 smithi084 ceph-mon[27254]: 2.b scrub ok 2024-09-11T03:39:56.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:55 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:39:56.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:55 smithi084 ceph-mon[27254]: 2.8 scrub ok 2024-09-11T03:39:56.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:55 smithi084 ceph-mon[27254]: pgmap v421: 17 pgs: 17 active+clean; 107 MiB data, 535 MiB used, 983 GiB / 983 GiB avail; 1.7 MiB/s rd, 492 KiB/s wr, 4 op/s 2024-09-11T03:39:56.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:55 smithi084 ceph-mon[27254]: osdmap e178: 12 total, 12 up, 11 in 2024-09-11T03:39:56.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:55 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:39:56.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:55 smithi084 ceph-mon[27254]: 2.e scrub ok 2024-09-11T03:39:56.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:55 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4206199193' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:56.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:55 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2424325912' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-11T03:39:56.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:55 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:39:56.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:55 smithi121 ceph-mon[30729]: 2.b scrub ok 2024-09-11T03:39:56.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:55 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:39:56.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:55 smithi121 ceph-mon[30729]: 2.8 scrub ok 2024-09-11T03:39:56.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:55 smithi121 ceph-mon[30729]: pgmap v421: 17 pgs: 17 active+clean; 107 MiB data, 535 MiB used, 983 GiB / 983 GiB avail; 1.7 MiB/s rd, 492 KiB/s wr, 4 op/s 2024-09-11T03:39:56.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:55 smithi121 ceph-mon[30729]: osdmap e178: 12 total, 12 up, 11 in 2024-09-11T03:39:56.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:55 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:39:56.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:55 smithi121 ceph-mon[30729]: 2.e scrub ok 2024-09-11T03:39:56.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:55 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4206199193' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:56.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:55 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2424325912' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-11T03:39:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:55 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:39:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:55 smithi175 ceph-mon[30894]: 2.b scrub ok 2024-09-11T03:39:56.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:55 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:39:56.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:55 smithi175 ceph-mon[30894]: 2.8 scrub ok 2024-09-11T03:39:56.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:55 smithi175 ceph-mon[30894]: pgmap v421: 17 pgs: 17 active+clean; 107 MiB data, 535 MiB used, 983 GiB / 983 GiB avail; 1.7 MiB/s rd, 492 KiB/s wr, 4 op/s 2024-09-11T03:39:56.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:55 smithi175 ceph-mon[30894]: osdmap e178: 12 total, 12 up, 11 in 2024-09-11T03:39:56.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:55 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:39:56.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:55 smithi175 ceph-mon[30894]: 2.e scrub ok 2024-09-11T03:39:56.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:55 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4206199193' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:39:56.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:55 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2424325912' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-11T03:39:56.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:56.272 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:56.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:56.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:56.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:56.400 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:56.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:56.502 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:56.525 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:56.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:56.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:56.627 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:56.653 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:56.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:56.654 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:56.755 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:56.781 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:56.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:56.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:56.829 INFO:tasks.rados.rados.2.smithi175.stdout:425: done (2 left) 2024-09-11T03:39:56.829 INFO:tasks.rados.rados.2.smithi175.stdout:426: delete oid 4 current snap is 42 2024-09-11T03:39:56.829 INFO:tasks.rados.rados.2.smithi175.stdout:423: finishing write tid 1 to smithi17557870-24 2024-09-11T03:39:56.829 INFO:tasks.rados.rados.2.smithi175.stdout:423: finishing write tid 2 to smithi17557870-24 2024-09-11T03:39:56.829 INFO:tasks.rados.rados.2.smithi175.stdout:423: finishing write tid 3 to smithi17557870-24 2024-09-11T03:39:56.830 INFO:tasks.rados.rados.2.smithi175.stdout:423: finishing write tid 4 to smithi17557870-24 2024-09-11T03:39:56.830 INFO:tasks.rados.rados.2.smithi175.stdout:423: oid 24 updating version 0 to 106 2024-09-11T03:39:56.830 INFO:tasks.rados.rados.2.smithi175.stdout:423: oid 24 version 106 is already newer than 105 2024-09-11T03:39:56.830 INFO:tasks.rados.rados.2.smithi175.stdout:423: oid 24 updating version 106 to 107 2024-09-11T03:39:56.830 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 107 (ObjNum 177 snap 42 seq_num 177) dirty exists 2024-09-11T03:39:56.830 INFO:tasks.rados.rados.2.smithi175.stdout:423: left oid 24 (ObjNum 177 snap 42 seq_num 177) 2024-09-11T03:39:56.830 INFO:tasks.rados.rados.2.smithi175.stdout:424: finishing write tid 1 to smithi17557870-45 2024-09-11T03:39:56.830 INFO:tasks.rados.rados.2.smithi175.stdout:424: finishing write tid 2 to smithi17557870-45 2024-09-11T03:39:56.830 INFO:tasks.rados.rados.2.smithi175.stdout:424: finishing write tid 3 to smithi17557870-45 2024-09-11T03:39:56.830 INFO:tasks.rados.rados.2.smithi175.stdout:424: finishing write tid 4 to smithi17557870-45 2024-09-11T03:39:56.830 INFO:tasks.rados.rados.2.smithi175.stdout:424: finishing write tid 5 to smithi17557870-45 2024-09-11T03:39:56.830 INFO:tasks.rados.rados.2.smithi175.stdout:424: oid 45 updating version 0 to 52 2024-09-11T03:39:56.830 INFO:tasks.rados.rados.2.smithi175.stdout:424: oid 45 version 52 is already newer than 50 2024-09-11T03:39:56.831 INFO:tasks.rados.rados.2.smithi175.stdout:424: oid 45 version 52 is already newer than 51 2024-09-11T03:39:56.831 INFO:tasks.rados.rados.2.smithi175.stdout:424: oid 45 version 52 is already newer than 49 2024-09-11T03:39:56.831 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 52 (ObjNum 178 snap 42 seq_num 178) dirty exists 2024-09-11T03:39:56.831 INFO:tasks.rados.rados.2.smithi175.stdout:424: left oid 45 (ObjNum 178 snap 42 seq_num 178) 2024-09-11T03:39:56.832 INFO:tasks.rados.rados.2.smithi175.stdout:423: done (2 left) 2024-09-11T03:39:56.832 INFO:tasks.rados.rados.2.smithi175.stdout:424: done (1 left) 2024-09-11T03:39:56.833 INFO:tasks.rados.rados.2.smithi175.stdout:426: done (0 left) 2024-09-11T03:39:56.833 INFO:tasks.rados.rados.2.smithi175.stdout:427: rollback oid 42 current snap is 42 2024-09-11T03:39:56.833 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 42 to 36 2024-09-11T03:39:56.833 INFO:tasks.rados.rados.2.smithi175.stdout:428: snap_create 2024-09-11T03:39:56.836 INFO:tasks.rados.rados.2.smithi175.stdout:427: finishing rollback tid 0 to smithi17557870-42 2024-09-11T03:39:56.837 INFO:tasks.rados.rados.2.smithi175.stdout:427: finishing rollback tid 1 to smithi17557870-42 2024-09-11T03:39:56.837 INFO:tasks.rados.rados.2.smithi175.stdout:427: finishing rollback tid 2 to smithi17557870-42 2024-09-11T03:39:56.837 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 56 (ObjNum 134 snap 22 seq_num 134) dirty exists 2024-09-11T03:39:56.883 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:56.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:56.909 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:56.909 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:56.943 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:39:56.966 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:56 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:39:56.966 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:56 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:39:56.966 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4206199193' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:56.966 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2424325912' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-11T03:39:56.966 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:56 smithi084 ceph-mon[27254]: osdmap e179: 12 total, 12 up, 11 in 2024-09-11T03:39:56.966 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-11T03:39:56.966 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-11T03:39:56.966 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:56 smithi084 ceph-mon[27254]: osdmap e180: 12 total, 12 up, 11 in 2024-09-11T03:39:57.010 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:57.035 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:57.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:57.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:57.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:57.163 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:57.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:57.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:56 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:39:57.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:56 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:39:57.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:56 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4206199193' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:57.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:56 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2424325912' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-11T03:39:57.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:56 smithi121 ceph-mon[30729]: osdmap e179: 12 total, 12 up, 11 in 2024-09-11T03:39:57.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:56 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-11T03:39:57.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:56 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-11T03:39:57.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:56 smithi121 ceph-mon[30729]: osdmap e180: 12 total, 12 up, 11 in 2024-09-11T03:39:57.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:56 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:39:57.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:56 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:39:57.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:56 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4206199193' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:39:57.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:56 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2424325912' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-11T03:39:57.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:56 smithi175 ceph-mon[30894]: osdmap e179: 12 total, 12 up, 11 in 2024-09-11T03:39:57.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:56 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-11T03:39:57.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:56 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-11T03:39:57.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:56 smithi175 ceph-mon[30894]: osdmap e180: 12 total, 12 up, 11 in 2024-09-11T03:39:57.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:57.292 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:57.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:57.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:57.394 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:57.417 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:57.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:57.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:57.456 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:39:57.456 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:39:57.520 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:57.541 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:57.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:57.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:57.643 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:39:57.665 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:57.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:57.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:39:57.767 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:57.793 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:57.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:57.794 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:57.830 INFO:tasks.rados.rados.2.smithi175.stdout:427: done (1 left) 2024-09-11T03:39:57.830 INFO:tasks.rados.rados.2.smithi175.stdout:428: done (0 left) 2024-09-11T03:39:57.830 INFO:tasks.rados.rados.2.smithi175.stdout:429: read oid 8 snap -1 2024-09-11T03:39:57.830 INFO:tasks.rados.rados.2.smithi175.stdout:429: expect deleted 2024-09-11T03:39:57.830 INFO:tasks.rados.rados.2.smithi175.stdout:430: write oid 23 current snap is 43 2024-09-11T03:39:57.830 INFO:tasks.rados.rados.2.smithi175.stdout:430: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2024-09-11T03:39:57.837 INFO:tasks.rados.rados.2.smithi175.stdout:430: writing smithi17557870-23 from 625260 to 1246199 tid 1 2024-09-11T03:39:57.843 INFO:tasks.rados.rados.2.smithi175.stdout:430: writing smithi17557870-23 from 1683347 to 2282733 tid 2 2024-09-11T03:39:57.848 INFO:tasks.rados.rados.2.smithi175.stdout:430: writing smithi17557870-23 from 2943668 to 3025260 tid 3 2024-09-11T03:39:57.848 INFO:tasks.rados.rados.2.smithi175.stdout:431: write (excl) oid 47 current snap is 43 2024-09-11T03:39:57.848 INFO:tasks.rados.rados.2.smithi175.stdout:431: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2024-09-11T03:39:57.855 INFO:tasks.rados.rados.2.smithi175.stdout:431: writing smithi17557870-47 from 642067 to 1338255 tid 1 2024-09-11T03:39:57.861 INFO:tasks.rados.rados.2.smithi175.stdout:431: writing smithi17557870-47 from 2025476 to 2484873 tid 2 2024-09-11T03:39:57.865 INFO:tasks.rados.rados.2.smithi175.stdout:431: writing smithi17557870-47 from 2971091 to 3042067 tid 3 2024-09-11T03:39:57.866 INFO:tasks.rados.rados.2.smithi175.stdout:432: read oid 28 snap -1 2024-09-11T03:39:57.866 INFO:tasks.rados.rados.2.smithi175.stdout:432: expect (ObjNum 109 snap 16 seq_num 109) 2024-09-11T03:39:57.866 INFO:tasks.rados.rados.2.smithi175.stdout:433: copy_from oid 31 from oid 50 current snap is 43 2024-09-11T03:39:57.866 INFO:tasks.rados.rados.2.smithi175.stdout:430: finishing write tid 1 to smithi17557870-23 2024-09-11T03:39:57.866 INFO:tasks.rados.rados.2.smithi175.stdout:430: finishing write tid 2 to smithi17557870-23 2024-09-11T03:39:57.866 INFO:tasks.rados.rados.2.smithi175.stdout:429: done (4 left) 2024-09-11T03:39:57.866 INFO:tasks.rados.rados.2.smithi175.stdout:434: read oid 40 snap -1 2024-09-11T03:39:57.866 INFO:tasks.rados.rados.2.smithi175.stdout:434: expect deleted 2024-09-11T03:39:57.866 INFO:tasks.rados.rados.2.smithi175.stdout:435: rollback oid 32 current snap is 43 2024-09-11T03:39:57.866 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 32 to 36 2024-09-11T03:39:57.866 INFO:tasks.rados.rados.2.smithi175.stdout:430: finishing write tid 3 to smithi17557870-23 2024-09-11T03:39:57.866 INFO:tasks.rados.rados.2.smithi175.stdout:430: finishing write tid 4 to smithi17557870-23 2024-09-11T03:39:57.866 INFO:tasks.rados.rados.2.smithi175.stdout:430: finishing write tid 5 to smithi17557870-23 2024-09-11T03:39:57.866 INFO:tasks.rados.rados.2.smithi175.stdout:430: oid 23 updating version 0 to 88 2024-09-11T03:39:57.867 INFO:tasks.rados.rados.2.smithi175.stdout:430: oid 23 updating version 88 to 90 2024-09-11T03:39:57.867 INFO:tasks.rados.rados.2.smithi175.stdout:430: oid 23 version 90 is already newer than 89 2024-09-11T03:39:57.867 INFO:tasks.rados.rados.2.smithi175.stdout:430: oid 23 updating version 90 to 91 2024-09-11T03:39:57.867 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 91 (ObjNum 180 snap 43 seq_num 180) dirty exists 2024-09-11T03:39:57.867 INFO:tasks.rados.rados.2.smithi175.stdout:430: left oid 23 (ObjNum 180 snap 43 seq_num 180) 2024-09-11T03:39:57.867 INFO:tasks.rados.rados.2.smithi175.stdout:431: finishing write tid 1 to smithi17557870-47 2024-09-11T03:39:57.867 INFO:tasks.rados.rados.2.smithi175.stdout:431: finishing write tid 2 to smithi17557870-47 2024-09-11T03:39:57.867 INFO:tasks.rados.rados.2.smithi175.stdout:430: done (5 left) 2024-09-11T03:39:57.867 INFO:tasks.rados.rados.2.smithi175.stdout:436: read oid 49 snap -1 2024-09-11T03:39:57.867 INFO:tasks.rados.rados.2.smithi175.stdout:436: expect deleted 2024-09-11T03:39:57.867 INFO:tasks.rados.rados.2.smithi175.stdout:437: read oid 23 snap -1 2024-09-11T03:39:57.867 INFO:tasks.rados.rados.2.smithi175.stdout:437: expect (ObjNum 180 snap 43 seq_num 180) 2024-09-11T03:39:57.867 INFO:tasks.rados.rados.2.smithi175.stdout:438: snap_remove snap 37 2024-09-11T03:39:57.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:39:57.919 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:57.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:57.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:39:58.021 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:39:58.044 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:58.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:39:58.112 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T03:39:58.147 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:39:58.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:57 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:39:58.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:57 smithi121 ceph-mon[30729]: 2.2 scrub ok 2024-09-11T03:39:58.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:57 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:39:58.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:57 smithi121 ceph-mon[30729]: 2.c scrub ok 2024-09-11T03:39:58.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:57 smithi121 ceph-mon[30729]: pgmap v424: 17 pgs: 17 active+clean; 117 MiB data, 550 MiB used, 983 GiB / 983 GiB avail; 1023 B/s rd, 1.9 MiB/s wr, 7 op/s 2024-09-11T03:39:58.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:57 smithi121 ceph-mon[30729]: osdmap e181: 12 total, 12 up, 11 in 2024-09-11T03:39:58.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:58.177 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:58.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:39:58.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:57 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:39:58.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:57 smithi175 ceph-mon[30894]: 2.2 scrub ok 2024-09-11T03:39:58.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:57 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:39:58.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:57 smithi175 ceph-mon[30894]: 2.c scrub ok 2024-09-11T03:39:58.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:57 smithi175 ceph-mon[30894]: pgmap v424: 17 pgs: 17 active+clean; 117 MiB data, 550 MiB used, 983 GiB / 983 GiB avail; 1023 B/s rd, 1.9 MiB/s wr, 7 op/s 2024-09-11T03:39:58.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:57 smithi175 ceph-mon[30894]: osdmap e181: 12 total, 12 up, 11 in 2024-09-11T03:39:58.251 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:57 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:39:58.251 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:57 smithi084 ceph-mon[27254]: 2.2 scrub ok 2024-09-11T03:39:58.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:57 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:39:58.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:57 smithi084 ceph-mon[27254]: 2.c scrub ok 2024-09-11T03:39:58.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:57 smithi084 ceph-mon[27254]: pgmap v424: 17 pgs: 17 active+clean; 117 MiB data, 550 MiB used, 983 GiB / 983 GiB avail; 1023 B/s rd, 1.9 MiB/s wr, 7 op/s 2024-09-11T03:39:58.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:57 smithi084 ceph-mon[27254]: osdmap e181: 12 total, 12 up, 11 in 2024-09-11T03:39:58.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:39:58.303 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:58.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:58.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:39:58.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:39:58.431 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:58.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:39:58.532 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:58.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:58.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:58.657 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:58.660 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:58.684 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:58.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:58.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:58.698 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:39:58.786 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:58.811 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:58.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:58.834 INFO:tasks.rados.rados.2.smithi175.stdout:438: done (7 left) 2024-09-11T03:39:58.834 INFO:tasks.rados.rados.2.smithi175.stdout:439: snap_create 2024-09-11T03:39:58.835 INFO:tasks.rados.rados.2.smithi175.stdout:431: finishing write tid 3 to smithi17557870-47 2024-09-11T03:39:58.835 INFO:tasks.rados.rados.2.smithi175.stdout:431: finishing write tid 4 to smithi17557870-47 2024-09-11T03:39:58.835 INFO:tasks.rados.rados.2.smithi175.stdout:435: finishing rollback tid 0 to smithi17557870-32 2024-09-11T03:39:58.835 INFO:tasks.rados.rados.2.smithi175.stdout:431: finishing write tid 5 to smithi17557870-47 2024-09-11T03:39:58.835 INFO:tasks.rados.rados.2.smithi175.stdout:431: oid 47 updating version 0 to 143 2024-09-11T03:39:58.835 INFO:tasks.rados.rados.2.smithi175.stdout:431: oid 47 version 143 is already newer than 142 2024-09-11T03:39:58.835 INFO:tasks.rados.rados.2.smithi175.stdout:431: oid 47 version 143 is already newer than 140 2024-09-11T03:39:58.835 INFO:tasks.rados.rados.2.smithi175.stdout:431: oid 47 version 143 is already newer than 141 2024-09-11T03:39:58.835 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 143 (ObjNum 181 snap 43 seq_num 181) dirty exists 2024-09-11T03:39:58.835 INFO:tasks.rados.rados.2.smithi175.stdout:431: left oid 47 (ObjNum 181 snap 43 seq_num 181) 2024-09-11T03:39:58.835 INFO:tasks.rados.rados.2.smithi175.stdout:435: finishing rollback tid 1 to smithi17557870-32 2024-09-11T03:39:58.835 INFO:tasks.rados.rados.2.smithi175.stdout:435: finishing rollback tid 2 to smithi17557870-32 2024-09-11T03:39:58.835 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 90 (ObjNum 108 snap 16 seq_num 108) dirty exists 2024-09-11T03:39:58.836 INFO:tasks.rados.rados.2.smithi175.stdout:433: finishing copy_from to smithi17557870-31 2024-09-11T03:39:58.836 INFO:tasks.rados.rados.2.smithi175.stdout:433: got expected ENOENT (src dne) 2024-09-11T03:39:58.836 INFO:tasks.rados.rados.2.smithi175.stdout:433: finishing copy_from racing read to smithi17557870-31 2024-09-11T03:39:58.836 INFO:tasks.rados.rados.2.smithi175.stdout:437: expect (ObjNum 180 snap 43 seq_num 180) 2024-09-11T03:39:58.913 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:58.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:58.936 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:58.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:59.037 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:39:59.058 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:59.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:59.059 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:39:59.096 INFO:tasks.rados.rados.2.smithi175.stdout:432: expect (ObjNum 109 snap 16 seq_num 109) 2024-09-11T03:39:59.150 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:58 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-11T03:39:59.150 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:58 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-11T03:39:59.150 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:58 smithi084 ceph-mon[27254]: osdmap e182: 12 total, 12 up, 11 in 2024-09-11T03:39:59.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:59.186 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:59.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:59.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:59.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:58 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-11T03:39:59.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:58 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-11T03:39:59.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:58 smithi175 ceph-mon[30894]: osdmap e182: 12 total, 12 up, 11 in 2024-09-11T03:39:59.288 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:59.301 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:39:59.301 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:39:59.310 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:59.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:59.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:39:59.318 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:39:59.318 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:39:59.334 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:39:59.334 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:39:59.350 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:39:59.350 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:39:59.367 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:39:59.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:39:59.384 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:39:59.384 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:39:59.401 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:39:59.401 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:39:59.412 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:39:59.419 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:39:59.420 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:39:59.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:58 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-11T03:39:59.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:58 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-11T03:39:59.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:58 smithi121 ceph-mon[30729]: osdmap e182: 12 total, 12 up, 11 in 2024-09-11T03:39:59.433 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:39:59.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:59.435 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:39:59.436 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:39:59.437 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:39:59.453 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:39:59.454 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:39:59.471 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:39:59.471 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:39:59.488 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:39:59.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:39:59.511 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:39:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:39:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:39:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:39:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14156.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-11T03:39:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.055116+0000", 2024-09-11T03:39:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 375.45569559500001, 2024-09-11T03:39:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.070523625000000006, 2024-09-11T03:39:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.512 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14156", 2024-09-11T03:39:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3948643139", 2024-09-11T03:39:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2024-09-11T03:39:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.055116+0000", 2024-09-11T03:39:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.055116+0000", 2024-09-11T03:39:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.055117+0000", 2024-09-11T03:39:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5710000000000001e-06 2024-09-11T03:39:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.055124+0000", 2024-09-11T03:39:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2690000000000002e-06 2024-09-11T03:39:59.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.055133+0000", 2024-09-11T03:39:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0680000000000003e-06 2024-09-11T03:39:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.055142+0000", 2024-09-11T03:39:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3569999999999998e-06 2024-09-11T03:39:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.055196+0000", 2024-09-11T03:39:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4527000000000002e-05 2024-09-11T03:39:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:39:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.055204+0000", 2024-09-11T03:39:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0069999999999997e-06 2024-09-11T03:39:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.124564+0000", 2024-09-11T03:39:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.069359259000000006 2024-09-11T03:39:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.124698+0000", 2024-09-11T03:39:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013475100000000001 2024-09-11T03:39:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,2", 2024-09-11T03:39:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.124876+0000", 2024-09-11T03:39:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017765899999999999 2024-09-11T03:39:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:39:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.125265+0000", 2024-09-11T03:39:59.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00038935199999999998 2024-09-11T03:39:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.125507+0000", 2024-09-11T03:39:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000241635 2024-09-11T03:39:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.125584+0000", 2024-09-11T03:39:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6909000000000005e-05 2024-09-11T03:39:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.125614+0000", 2024-09-11T03:39:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9952000000000001e-05 2024-09-11T03:39:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.125639+0000", 2024-09-11T03:39:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5309000000000001e-05 2024-09-11T03:39:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14156.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-11T03:39:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.200524+0000", 2024-09-11T03:39:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 375.31028674999999, 2024-09-11T03:39:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012479581, 2024-09-11T03:39:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14156", 2024-09-11T03:39:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3948643139", 2024-09-11T03:39:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 47 2024-09-11T03:39:59.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200524+0000", 2024-09-11T03:39:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200524+0000", 2024-09-11T03:39:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200525+0000", 2024-09-11T03:39:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6899999999999997e-07 2024-09-11T03:39:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200587+0000", 2024-09-11T03:39:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1970000000000005e-05 2024-09-11T03:39:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200594+0000", 2024-09-11T03:39:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3679999999999996e-06 2024-09-11T03:39:59.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200600+0000", 2024-09-11T03:39:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5709999999999997e-06 2024-09-11T03:39:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209170+0000", 2024-09-11T03:39:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085703499999999991 2024-09-11T03:39:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209216+0000", 2024-09-11T03:39:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5969e-05 2024-09-11T03:39:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,2", 2024-09-11T03:39:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209289+0000", 2024-09-11T03:39:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3116999999999998e-05 2024-09-11T03:39:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212824+0000", 2024-09-11T03:39:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003534414 2024-09-11T03:39:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212872+0000", 2024-09-11T03:39:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8430999999999998e-05 2024-09-11T03:39:59.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:39:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212927+0000", 2024-09-11T03:39:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4595000000000002e-05 2024-09-11T03:39:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212968+0000", 2024-09-11T03:39:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1100000000000003e-05 2024-09-11T03:39:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213004+0000", 2024-09-11T03:39:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6127000000000002e-05 2024-09-11T03:39:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14156.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-11T03:39:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.200610+0000", 2024-09-11T03:39:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 375.310201054, 2024-09-11T03:39:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012956390999999999, 2024-09-11T03:39:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.527 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14156", 2024-09-11T03:39:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3948643139", 2024-09-11T03:39:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 48 2024-09-11T03:39:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200610+0000", 2024-09-11T03:39:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200610+0000", 2024-09-11T03:39:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200611+0000", 2024-09-11T03:39:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4899999999999995e-07 2024-09-11T03:39:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200620+0000", 2024-09-11T03:39:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.577e-06 2024-09-11T03:39:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200627+0000", 2024-09-11T03:39:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7490000000000001e-06 2024-09-11T03:39:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200631+0000", 2024-09-11T03:39:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6490000000000001e-06 2024-09-11T03:39:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209523+0000", 2024-09-11T03:39:59.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0088927220000000005 2024-09-11T03:39:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209582+0000", 2024-09-11T03:39:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8662000000000002e-05 2024-09-11T03:39:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,2", 2024-09-11T03:39:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209688+0000", 2024-09-11T03:39:59.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010570599999999999 2024-09-11T03:39:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212917+0000", 2024-09-11T03:39:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0032291500000000001 2024-09-11T03:39:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:39:59.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213018+0000", 2024-09-11T03:39:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010094600000000001 2024-09-11T03:39:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213479+0000", 2024-09-11T03:39:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00046086400000000001 2024-09-11T03:39:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213530+0000", 2024-09-11T03:39:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1286999999999999e-05 2024-09-11T03:39:59.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213566+0000", 2024-09-11T03:39:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6529999999999998e-05 2024-09-11T03:39:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14156.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-11T03:39:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.200638+0000", 2024-09-11T03:39:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 375.31017323200001, 2024-09-11T03:39:59.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013092797, 2024-09-11T03:39:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14156", 2024-09-11T03:39:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3948643139", 2024-09-11T03:39:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 49 2024-09-11T03:39:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200638+0000", 2024-09-11T03:39:59.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200638+0000", 2024-09-11T03:39:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200638+0000", 2024-09-11T03:39:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1899999999999998e-07 2024-09-11T03:39:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200642+0000", 2024-09-11T03:39:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3289999999999998e-06 2024-09-11T03:39:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200655+0000", 2024-09-11T03:39:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3722e-05 2024-09-11T03:39:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200660+0000", 2024-09-11T03:39:59.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3579999999999996e-06 2024-09-11T03:39:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209942+0000", 2024-09-11T03:39:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0092820890000000003 2024-09-11T03:39:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209990+0000", 2024-09-11T03:39:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8162999999999999e-05 2024-09-11T03:39:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,2", 2024-09-11T03:39:59.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210092+0000", 2024-09-11T03:39:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000101548 2024-09-11T03:39:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:39:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213580+0000", 2024-09-11T03:39:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034881489999999999 2024-09-11T03:39:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213640+0000", 2024-09-11T03:39:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0519000000000001e-05 2024-09-11T03:39:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213669+0000", 2024-09-11T03:39:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.83e-05 2024-09-11T03:39:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213694+0000", 2024-09-11T03:39:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5316e-05 2024-09-11T03:39:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213731+0000", 2024-09-11T03:39:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6884999999999997e-05 2024-09-11T03:39:59.540 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14156.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-11T03:39:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.200667+0000", 2024-09-11T03:39:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 375.31014419899998, 2024-09-11T03:39:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014059617, 2024-09-11T03:39:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:59.541 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14156", 2024-09-11T03:39:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3948643139", 2024-09-11T03:39:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 50 2024-09-11T03:39:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200667+0000", 2024-09-11T03:39:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200667+0000", 2024-09-11T03:39:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200667+0000", 2024-09-11T03:39:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0999999999999999e-07 2024-09-11T03:39:59.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200728+0000", 2024-09-11T03:39:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1017999999999998e-05 2024-09-11T03:39:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200739+0000", 2024-09-11T03:39:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0386e-05 2024-09-11T03:39:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200742+0000", 2024-09-11T03:39:59.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4539999999999999e-06 2024-09-11T03:39:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213058+0000", 2024-09-11T03:39:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012315557 2024-09-11T03:39:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213110+0000", 2024-09-11T03:39:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2104999999999997e-05 2024-09-11T03:39:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,2", 2024-09-11T03:39:59.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213193+0000", 2024-09-11T03:39:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3172999999999998e-05 2024-09-11T03:39:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:39:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213739+0000", 2024-09-11T03:39:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00054597000000000001 2024-09-11T03:39:59.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.214586+0000", 2024-09-11T03:39:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00084747399999999997 2024-09-11T03:39:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.214644+0000", 2024-09-11T03:39:59.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7027999999999997e-05 2024-09-11T03:39:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.214692+0000", 2024-09-11T03:39:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8562e-05 2024-09-11T03:39:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.214727+0000", 2024-09-11T03:39:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4480000000000002e-05 2024-09-11T03:39:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.548 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14156.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-11T03:39:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.200748+0000", 2024-09-11T03:39:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 375.31006274700002, 2024-09-11T03:39:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01421853, 2024-09-11T03:39:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.552 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14156", 2024-09-11T03:39:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3948643139", 2024-09-11T03:39:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 51 2024-09-11T03:39:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200748+0000", 2024-09-11T03:39:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200748+0000", 2024-09-11T03:39:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200749+0000", 2024-09-11T03:39:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.65e-07 2024-09-11T03:39:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200751+0000", 2024-09-11T03:39:59.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3180000000000001e-06 2024-09-11T03:39:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200754+0000", 2024-09-11T03:39:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2899999999999998e-06 2024-09-11T03:39:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200758+0000", 2024-09-11T03:39:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4970000000000001e-06 2024-09-11T03:39:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213794+0000", 2024-09-11T03:39:59.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013035639 2024-09-11T03:39:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213866+0000", 2024-09-11T03:39:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2150000000000005e-05 2024-09-11T03:39:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,2", 2024-09-11T03:39:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.213973+0000", 2024-09-11T03:39:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000106878 2024-09-11T03:39:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:39:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.214738+0000", 2024-09-11T03:39:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076590899999999995 2024-09-11T03:39:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.214835+0000", 2024-09-11T03:39:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6897000000000004e-05 2024-09-11T03:39:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.214884+0000", 2024-09-11T03:39:59.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8899000000000001e-05 2024-09-11T03:39:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.214933+0000", 2024-09-11T03:39:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9079999999999998e-05 2024-09-11T03:39:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.214967+0000", 2024-09-11T03:39:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3608000000000001e-05 2024-09-11T03:39:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14156.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-11T03:39:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.200764+0000", 2024-09-11T03:39:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 375.31004708299997, 2024-09-11T03:39:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014358964, 2024-09-11T03:39:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.560 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14156", 2024-09-11T03:39:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3948643139", 2024-09-11T03:39:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 52 2024-09-11T03:39:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200764+0000", 2024-09-11T03:39:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200764+0000", 2024-09-11T03:39:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200764+0000", 2024-09-11T03:39:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5699999999999998e-07 2024-09-11T03:39:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200773+0000", 2024-09-11T03:39:59.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1489999999999994e-06 2024-09-11T03:39:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200778+0000", 2024-09-11T03:39:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9009999999999999e-06 2024-09-11T03:39:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.200783+0000", 2024-09-11T03:39:59.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6909999999999996e-06 2024-09-11T03:39:59.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.214189+0000", 2024-09-11T03:39:59.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013405851 2024-09-11T03:39:59.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.214265+0000", 2024-09-11T03:39:59.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6064999999999998e-05 2024-09-11T03:39:59.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,2", 2024-09-11T03:39:59.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.214347+0000", 2024-09-11T03:39:59.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1623000000000001e-05 2024-09-11T03:39:59.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:39:59.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.214977+0000", 2024-09-11T03:39:59.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00063059900000000002 2024-09-11T03:39:59.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.215008+0000", 2024-09-11T03:39:59.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0852000000000002e-05 2024-09-11T03:39:59.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.215043+0000", 2024-09-11T03:39:59.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5188999999999998e-05 2024-09-11T03:39:59.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:59.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.215089+0000", 2024-09-11T03:39:59.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6143000000000002e-05 2024-09-11T03:39:59.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.215123+0000", 2024-09-11T03:39:59.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3544000000000003e-05 2024-09-11T03:39:59.567 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.567 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.567 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.568 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:228 2.9 2:95fa802a:::smithi17557870-22:head [write 490771~686315,stat] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-11T03:39:59.568 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:41.965436+0000", 2024-09-11T03:39:59.568 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 77.545374752000001, 2024-09-11T03:39:59.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018731718000000001, 2024-09-11T03:39:59.568 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.568 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.569 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:59.569 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.569 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:39:59.569 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:59.569 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 228 2024-09-11T03:39:59.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.569 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.965436+0000", 2024-09-11T03:39:59.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.965436+0000", 2024-09-11T03:39:59.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.965439+0000", 2024-09-11T03:39:59.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2879999999999999e-06 2024-09-11T03:39:59.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.966276+0000", 2024-09-11T03:39:59.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00083751099999999998 2024-09-11T03:39:59.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.966299+0000", 2024-09-11T03:39:59.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2532999999999999e-05 2024-09-11T03:39:59.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.966308+0000", 2024-09-11T03:39:59.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4839999999999993e-06 2024-09-11T03:39:59.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.966374+0000", 2024-09-11T03:39:59.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5534e-05 2024-09-11T03:39:59.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.966442+0000", 2024-09-11T03:39:59.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8368999999999995e-05 2024-09-11T03:39:59.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:39:59.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.966571+0000", 2024-09-11T03:39:59.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012857999999999999 2024-09-11T03:39:59.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:39:59.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.968401+0000", 2024-09-11T03:39:59.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001830304 2024-09-11T03:39:59.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.968632+0000", 2024-09-11T03:39:59.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000230752 2024-09-11T03:39:59.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984080+0000", 2024-09-11T03:39:59.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015448066999999999 2024-09-11T03:39:59.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:59.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984117+0000", 2024-09-11T03:39:59.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7150000000000002e-05 2024-09-11T03:39:59.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984168+0000", 2024-09-11T03:39:59.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1146000000000001e-05 2024-09-11T03:39:59.578 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.578 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.579 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.579 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:229 2.9 2:95fa802a:::smithi17557870-22:head [write 1885075~406346] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-11T03:39:59.579 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:41.971562+0000", 2024-09-11T03:39:59.579 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 77.539249061999996, 2024-09-11T03:39:59.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012698844000000001, 2024-09-11T03:39:59.580 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.580 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.580 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:59.580 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.580 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:39:59.580 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:59.581 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 229 2024-09-11T03:39:59.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.581 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971562+0000", 2024-09-11T03:39:59.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971562+0000", 2024-09-11T03:39:59.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971563+0000", 2024-09-11T03:39:59.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1200000000000001e-06 2024-09-11T03:39:59.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971924+0000", 2024-09-11T03:39:59.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00036046400000000001 2024-09-11T03:39:59.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971934+0000", 2024-09-11T03:39:59.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0744000000000001e-05 2024-09-11T03:39:59.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971942+0000", 2024-09-11T03:39:59.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1489999999999997e-06 2024-09-11T03:39:59.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971984+0000", 2024-09-11T03:39:59.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2178000000000002e-05 2024-09-11T03:39:59.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.972080+0000", 2024-09-11T03:39:59.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6362999999999999e-05 2024-09-11T03:39:59.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:39:59.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.972181+0000", 2024-09-11T03:39:59.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000100406 2024-09-11T03:39:59.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:39:59.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.973247+0000", 2024-09-11T03:39:59.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001066949 2024-09-11T03:39:59.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.973329+0000", 2024-09-11T03:39:59.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1897000000000005e-05 2024-09-11T03:39:59.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984202+0000", 2024-09-11T03:39:59.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010872728999999999 2024-09-11T03:39:59.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:59.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984235+0000", 2024-09-11T03:39:59.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3011e-05 2024-09-11T03:39:59.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984261+0000", 2024-09-11T03:39:59.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5834e-05 2024-09-11T03:39:59.589 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.589 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.589 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.589 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:230 2.9 2:95fa802a:::smithi17557870-22:head [setxattr _header (56) in=7b,truncate 2890771] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-11T03:39:59.589 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:41.971958+0000", 2024-09-11T03:39:59.589 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 77.538853066000001, 2024-09-11T03:39:59.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012401317, 2024-09-11T03:39:59.589 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.590 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.590 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:59.590 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.590 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:39:59.590 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:59.590 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 230 2024-09-11T03:39:59.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.590 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971958+0000", 2024-09-11T03:39:59.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971958+0000", 2024-09-11T03:39:59.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971959+0000", 2024-09-11T03:39:59.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4799999999999998e-07 2024-09-11T03:39:59.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971962+0000", 2024-09-11T03:39:59.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4350000000000001e-06 2024-09-11T03:39:59.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971969+0000", 2024-09-11T03:39:59.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2599999999999999e-06 2024-09-11T03:39:59.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971975+0000", 2024-09-11T03:39:59.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4559999999999999e-06 2024-09-11T03:39:59.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.972749+0000", 2024-09-11T03:39:59.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000774458 2024-09-11T03:39:59.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.972832+0000", 2024-09-11T03:39:59.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2508000000000004e-05 2024-09-11T03:39:59.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:39:59.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.972959+0000", 2024-09-11T03:39:59.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012697699999999999 2024-09-11T03:39:59.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:39:59.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.973268+0000", 2024-09-11T03:39:59.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000308809 2024-09-11T03:39:59.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.973443+0000", 2024-09-11T03:39:59.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017556700000000001 2024-09-11T03:39:59.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984289+0000", 2024-09-11T03:39:59.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010846035 2024-09-11T03:39:59.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:59.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984318+0000", 2024-09-11T03:39:59.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9249000000000001e-05 2024-09-11T03:39:59.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984359+0000", 2024-09-11T03:39:59.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1014999999999999e-05 2024-09-11T03:39:59.595 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.596 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.596 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.596 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:231 2.9 2:95fa802a:::smithi17557870-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e105)", 2024-09-11T03:39:59.596 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:41.971984+0000", 2024-09-11T03:39:59.596 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 77.538826989, 2024-09-11T03:39:59.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012499783, 2024-09-11T03:39:59.596 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.596 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.596 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:39:59.596 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.597 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:39:59.597 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:59.597 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 231 2024-09-11T03:39:59.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.597 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971984+0000", 2024-09-11T03:39:59.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971984+0000", 2024-09-11T03:39:59.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971985+0000", 2024-09-11T03:39:59.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2399999999999999e-07 2024-09-11T03:39:59.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971989+0000", 2024-09-11T03:39:59.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9659999999999998e-06 2024-09-11T03:39:59.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.972013+0000", 2024-09-11T03:39:59.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4581999999999999e-05 2024-09-11T03:39:59.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.972019+0000", 2024-09-11T03:39:59.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5609999999999998e-06 2024-09-11T03:39:59.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.973159+0000", 2024-09-11T03:39:59.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011401759999999999 2024-09-11T03:39:59.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-11T03:39:59.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.973228+0000", 2024-09-11T03:39:59.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8903000000000001e-05 2024-09-11T03:39:59.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984382+0000", 2024-09-11T03:39:59.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011154301 2024-09-11T03:39:59.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984428+0000", 2024-09-11T03:39:59.601 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:39:59.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6258999999999999e-05 2024-09-11T03:39:59.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984484+0000", 2024-09-11T03:39:59.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5611e-05 2024-09-11T03:39:59.603 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.603 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.604 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.604 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:341 2.9 2:9718db5a:::smithi17557870-3:head [copy-from ver 6] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:39:59.604 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:49.820733+0000", 2024-09-11T03:39:59.604 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 69.690077794999993, 2024-09-11T03:39:59.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018872793999999998, 2024-09-11T03:39:59.604 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.604 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.604 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:59.604 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.604 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:39:59.605 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:59.605 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 341 2024-09-11T03:39:59.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.605 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820733+0000", 2024-09-11T03:39:59.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820733+0000", 2024-09-11T03:39:59.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820736+0000", 2024-09-11T03:39:59.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9500000000000001e-06 2024-09-11T03:39:59.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820748+0000", 2024-09-11T03:39:59.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1516e-05 2024-09-11T03:39:59.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820778+0000", 2024-09-11T03:39:59.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9827000000000001e-05 2024-09-11T03:39:59.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820800+0000", 2024-09-11T03:39:59.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2265000000000001e-05 2024-09-11T03:39:59.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820873+0000", 2024-09-11T03:39:59.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3386999999999994e-05 2024-09-11T03:39:59.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:39:59.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820880+0000", 2024-09-11T03:39:59.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6800000000000004e-06 2024-09-11T03:39:59.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.821954+0000", 2024-09-11T03:39:59.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010737500000000001 2024-09-11T03:39:59.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.822016+0000", 2024-09-11T03:39:59.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1929999999999998e-05 2024-09-11T03:39:59.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:39:59.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.832720+0000", 2024-09-11T03:39:59.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010704709 2024-09-11T03:39:59.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:39:59.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.837688+0000", 2024-09-11T03:39:59.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049680089999999998 2024-09-11T03:39:59.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.837979+0000", 2024-09-11T03:39:59.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029093800000000003 2024-09-11T03:39:59.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.839534+0000", 2024-09-11T03:39:59.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015545170000000001 2024-09-11T03:39:59.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:59.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.839569+0000", 2024-09-11T03:39:59.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5386999999999998e-05 2024-09-11T03:39:59.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.839606+0000", 2024-09-11T03:39:59.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6928999999999998e-05 2024-09-11T03:39:59.613 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.613 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.613 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.613 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:342 2.9 2:9718db5a:::smithi17557870-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:39:59.613 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:49.820827+0000", 2024-09-11T03:39:59.613 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 69.689984406999997, 2024-09-11T03:39:59.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016114883999999999, 2024-09-11T03:39:59.613 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.613 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.613 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:39:59.613 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.614 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:39:59.614 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:59.614 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 342 2024-09-11T03:39:59.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.614 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820827+0000", 2024-09-11T03:39:59.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820827+0000", 2024-09-11T03:39:59.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820827+0000", 2024-09-11T03:39:59.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4600000000000005e-07 2024-09-11T03:39:59.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820831+0000", 2024-09-11T03:39:59.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.219e-06 2024-09-11T03:39:59.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820840+0000", 2024-09-11T03:39:59.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4399999999999994e-06 2024-09-11T03:39:59.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820847+0000", 2024-09-11T03:39:59.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4789999999999999e-06 2024-09-11T03:39:59.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820919+0000", 2024-09-11T03:39:59.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1223999999999994e-05 2024-09-11T03:39:59.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:59.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820922+0000", 2024-09-11T03:39:59.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9270000000000001e-06 2024-09-11T03:39:59.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.822138+0000", 2024-09-11T03:39:59.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001215982 2024-09-11T03:39:59.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:39:59.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.822186+0000", 2024-09-11T03:39:59.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8905e-05 2024-09-11T03:39:59.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.836672+0000", 2024-09-11T03:39:59.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014485434 2024-09-11T03:39:59.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.836817+0000", 2024-09-11T03:39:59.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014537900000000001 2024-09-11T03:39:59.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.836942+0000", 2024-09-11T03:39:59.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000124349 2024-09-11T03:39:59.619 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.619 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.619 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.619 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:428 2.9 2:95fa802a:::smithi17557870-22:head [copy-from ver 3] snapc b=[b] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-11T03:39:59.619 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:55.852514+0000", 2024-09-11T03:39:59.619 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 63.658297351000002, 2024-09-11T03:39:59.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020845635000000001, 2024-09-11T03:39:59.620 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.620 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.620 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:59.620 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.620 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:39:59.620 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:59.620 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 428 2024-09-11T03:39:59.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.620 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852514+0000", 2024-09-11T03:39:59.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852514+0000", 2024-09-11T03:39:59.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852514+0000", 2024-09-11T03:39:59.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0399999999999995e-07 2024-09-11T03:39:59.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852521+0000", 2024-09-11T03:39:59.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3879999999999997e-06 2024-09-11T03:39:59.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852533+0000", 2024-09-11T03:39:59.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2028000000000001e-05 2024-09-11T03:39:59.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852541+0000", 2024-09-11T03:39:59.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7000000000000008e-06 2024-09-11T03:39:59.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.856017+0000", 2024-09-11T03:39:59.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034762170000000002 2024-09-11T03:39:59.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.856106+0000", 2024-09-11T03:39:59.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9219000000000001e-05 2024-09-11T03:39:59.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:39:59.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.867527+0000", 2024-09-11T03:39:59.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011420625 2024-09-11T03:39:59.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:39:59.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.871293+0000", 2024-09-11T03:39:59.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037662189999999999 2024-09-11T03:39:59.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.871599+0000", 2024-09-11T03:39:59.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00030639900000000001 2024-09-11T03:39:59.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.873296+0000", 2024-09-11T03:39:59.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016962170000000001 2024-09-11T03:39:59.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:59.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.873330+0000", 2024-09-11T03:39:59.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4644000000000002e-05 2024-09-11T03:39:59.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.873359+0000", 2024-09-11T03:39:59.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9275e-05 2024-09-11T03:39:59.626 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.626 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.626 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.626 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:429 2.9 2:95fa802a:::smithi17557870-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e119)", 2024-09-11T03:39:59.626 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:55.852553+0000", 2024-09-11T03:39:59.626 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 63.658258025000002, 2024-09-11T03:39:59.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018332642, 2024-09-11T03:39:59.626 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.626 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.626 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:39:59.626 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.627 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:39:59.627 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:59.627 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 429 2024-09-11T03:39:59.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.627 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852553+0000", 2024-09-11T03:39:59.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852553+0000", 2024-09-11T03:39:59.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852554+0000", 2024-09-11T03:39:59.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2600000000000002e-07 2024-09-11T03:39:59.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852556+0000", 2024-09-11T03:39:59.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.373e-06 2024-09-11T03:39:59.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852564+0000", 2024-09-11T03:39:59.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4440000000000008e-06 2024-09-11T03:39:59.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852570+0000", 2024-09-11T03:39:59.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7459999999999997e-06 2024-09-11T03:39:59.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.856347+0000", 2024-09-11T03:39:59.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037767970000000001 2024-09-11T03:39:59.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:39:59.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.856411+0000", 2024-09-11T03:39:59.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3522000000000006e-05 2024-09-11T03:39:59.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.870676+0000", 2024-09-11T03:39:59.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014265582000000001 2024-09-11T03:39:59.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.870780+0000", 2024-09-11T03:39:59.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000103786 2024-09-11T03:39:59.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.870886+0000", 2024-09-11T03:39:59.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010586600000000001 2024-09-11T03:39:59.632 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.632 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.632 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.633 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:649 2.6 2:6dbd1e81:::smithi17557870-7:head [copy-from ver 82] snapc 11=[e] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:39:59.633 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:08.111685+0000", 2024-09-11T03:39:59.633 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 51.399126268000003, 2024-09-11T03:39:59.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017973807000000001, 2024-09-11T03:39:59.634 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.634 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.634 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:59.634 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.634 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:39:59.634 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:59.634 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 649 2024-09-11T03:39:59.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.634 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111685+0000", 2024-09-11T03:39:59.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111685+0000", 2024-09-11T03:39:59.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111686+0000", 2024-09-11T03:39:59.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.339e-06 2024-09-11T03:39:59.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111692+0000", 2024-09-11T03:39:59.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4759999999999997e-06 2024-09-11T03:39:59.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111706+0000", 2024-09-11T03:39:59.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4388e-05 2024-09-11T03:39:59.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111714+0000", 2024-09-11T03:39:59.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8730000000000007e-06 2024-09-11T03:39:59.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111782+0000", 2024-09-11T03:39:59.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7984000000000005e-05 2024-09-11T03:39:59.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111857+0000", 2024-09-11T03:39:59.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4741999999999997e-05 2024-09-11T03:39:59.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,8", 2024-09-11T03:39:59.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118631+0000", 2024-09-11T03:39:59.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067745890000000001 2024-09-11T03:39:59.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:39:59.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.123831+0000", 2024-09-11T03:39:59.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051996869999999997 2024-09-11T03:39:59.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.127180+0000", 2024-09-11T03:39:59.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003348668 2024-09-11T03:39:59.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129539+0000", 2024-09-11T03:39:59.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0023596239999999998 2024-09-11T03:39:59.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:59.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129607+0000", 2024-09-11T03:39:59.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7556000000000002e-05 2024-09-11T03:39:59.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129659+0000", 2024-09-11T03:39:59.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1881e-05 2024-09-11T03:39:59.640 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.641 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.641 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.641 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:650 2.6 2:6dbd1e81:::smithi17557870-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:39:59.641 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:08.111977+0000", 2024-09-11T03:39:59.641 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 51.398834430999997, 2024-09-11T03:39:59.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01107229, 2024-09-11T03:39:59.641 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.641 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.642 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:39:59.642 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.642 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:39:59.642 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:59.642 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 650 2024-09-11T03:39:59.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.642 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111977+0000", 2024-09-11T03:39:59.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111977+0000", 2024-09-11T03:39:59.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111977+0000", 2024-09-11T03:39:59.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5199999999999996e-07 2024-09-11T03:39:59.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111981+0000", 2024-09-11T03:39:59.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.507e-06 2024-09-11T03:39:59.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111993+0000", 2024-09-11T03:39:59.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1844999999999999e-05 2024-09-11T03:39:59.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.112000+0000", 2024-09-11T03:39:59.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.849e-06 2024-09-11T03:39:59.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.112157+0000", 2024-09-11T03:39:59.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015713700000000001 2024-09-11T03:39:59.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:39:59.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.112199+0000", 2024-09-11T03:39:59.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2101000000000002e-05 2024-09-11T03:39:59.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122900+0000", 2024-09-11T03:39:59.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010700734999999999 2024-09-11T03:39:59.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122961+0000", 2024-09-11T03:39:59.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1187999999999998e-05 2024-09-11T03:39:59.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.123049+0000", 2024-09-11T03:39:59.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8276000000000007e-05 2024-09-11T03:39:59.646 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.646 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.646 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.646 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:941 2.6 2:6dbd1e81:::smithi17557870-7:head [copy-from ver 44] snapc 24=[24,23,22,20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-11T03:39:59.647 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:39.829469+0000", 2024-09-11T03:39:59.647 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 19.681341965000001, 2024-09-11T03:39:59.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016117155000000001, 2024-09-11T03:39:59.647 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.647 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.647 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:59.647 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.647 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:39:59.647 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:59.647 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 941 2024-09-11T03:39:59.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.648 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829469+0000", 2024-09-11T03:39:59.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829469+0000", 2024-09-11T03:39:59.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829471+0000", 2024-09-11T03:39:59.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2060000000000001e-06 2024-09-11T03:39:59.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829480+0000", 2024-09-11T03:39:59.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3739999999999998e-06 2024-09-11T03:39:59.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829496+0000", 2024-09-11T03:39:59.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6728e-05 2024-09-11T03:39:59.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829504+0000", 2024-09-11T03:39:59.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5170000000000002e-06 2024-09-11T03:39:59.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829576+0000", 2024-09-11T03:39:59.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1734000000000001e-05 2024-09-11T03:39:59.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:39:59.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829582+0000", 2024-09-11T03:39:59.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7340000000000002e-06 2024-09-11T03:39:59.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.831253+0000", 2024-09-11T03:39:59.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016706379999999999 2024-09-11T03:39:59.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.831319+0000", 2024-09-11T03:39:59.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5893000000000001e-05 2024-09-11T03:39:59.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,8", 2024-09-11T03:39:59.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.839473+0000", 2024-09-11T03:39:59.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081536119999999993 2024-09-11T03:39:59.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:39:59.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.843019+0000", 2024-09-11T03:39:59.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0035461310000000001 2024-09-11T03:39:59.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.844908+0000", 2024-09-11T03:39:59.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018889460000000001 2024-09-11T03:39:59.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.845471+0000", 2024-09-11T03:39:59.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00056334899999999999 2024-09-11T03:39:59.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:59.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.845531+0000", 2024-09-11T03:39:59.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0361e-05 2024-09-11T03:39:59.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.845586+0000", 2024-09-11T03:39:59.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4932000000000002e-05 2024-09-11T03:39:59.654 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.654 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.654 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.655 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:942 2.6 2:6dbd1e81:::smithi17557870-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e163)", 2024-09-11T03:39:59.655 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:39.829523+0000", 2024-09-11T03:39:59.655 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 19.681288259999999, 2024-09-11T03:39:59.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013066657000000001, 2024-09-11T03:39:59.655 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.655 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.655 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:39:59.655 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.655 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:39:59.655 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:59.655 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 942 2024-09-11T03:39:59.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.656 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829523+0000", 2024-09-11T03:39:59.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829523+0000", 2024-09-11T03:39:59.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829523+0000", 2024-09-11T03:39:59.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1099999999999996e-07 2024-09-11T03:39:59.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829526+0000", 2024-09-11T03:39:59.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4609999999999998e-06 2024-09-11T03:39:59.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829615+0000", 2024-09-11T03:39:59.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9015000000000001e-05 2024-09-11T03:39:59.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829624+0000", 2024-09-11T03:39:59.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5470000000000002e-06 2024-09-11T03:39:59.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829692+0000", 2024-09-11T03:39:59.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7605000000000001e-05 2024-09-11T03:39:59.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:39:59.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829698+0000", 2024-09-11T03:39:59.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4990000000000002e-06 2024-09-11T03:39:59.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.831462+0000", 2024-09-11T03:39:59.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001764278 2024-09-11T03:39:59.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:39:59.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.831504+0000", 2024-09-11T03:39:59.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2524000000000001e-05 2024-09-11T03:39:59.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.842403+0000", 2024-09-11T03:39:59.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010899030000000001 2024-09-11T03:39:59.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.842492+0000", 2024-09-11T03:39:59.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8579999999999996e-05 2024-09-11T03:39:59.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.842590+0000", 2024-09-11T03:39:59.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7607000000000002e-05 2024-09-11T03:39:59.661 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.661 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.661 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.662 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1045 2.6 2:6e06b7a8:::smithi17557870-19:head [copy-from ver 73] snapc 27=[27,26] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-11T03:39:59.662 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:50.873832+0000", 2024-09-11T03:39:59.662 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 8.6369790119999994, 2024-09-11T03:39:59.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011841935, 2024-09-11T03:39:59.662 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:39:59.662 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:39:59.662 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:39:59.662 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:39:59.662 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:39:59.662 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:39:59.662 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1045 2024-09-11T03:39:59.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.663 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:39:59.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:39:59.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:50.873832+0000", 2024-09-11T03:39:59.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:39:59.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:50.873832+0000", 2024-09-11T03:39:59.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:39:59.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:39:59.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:50.873834+0000", 2024-09-11T03:39:59.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0190000000000001e-06 2024-09-11T03:39:59.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:39:59.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:50.873842+0000", 2024-09-11T03:39:59.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4610000000000002e-06 2024-09-11T03:39:59.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:39:59.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:50.873858+0000", 2024-09-11T03:39:59.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6518e-05 2024-09-11T03:39:59.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:39:59.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:50.873866+0000", 2024-09-11T03:39:59.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0450000000000001e-06 2024-09-11T03:39:59.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:39:59.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:50.873916+0000", 2024-09-11T03:39:59.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9534999999999999e-05 2024-09-11T03:39:59.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:39:59.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:50.873976+0000", 2024-09-11T03:39:59.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0739999999999998e-05 2024-09-11T03:39:59.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,8", 2024-09-11T03:39:59.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:50.881353+0000", 2024-09-11T03:39:59.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073766200000000004 2024-09-11T03:39:59.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:39:59.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:50.883307+0000", 2024-09-11T03:39:59.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019537780000000002 2024-09-11T03:39:59.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:50.884236+0000", 2024-09-11T03:39:59.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00092870099999999998 2024-09-11T03:39:59.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:39:59.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:50.885551+0000", 2024-09-11T03:39:59.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0013150289999999999 2024-09-11T03:39:59.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:39:59.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:50.885617+0000", 2024-09-11T03:39:59.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6200999999999999e-05 2024-09-11T03:39:59.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:39:59.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:39:59.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:39:59.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:50.885674+0000", 2024-09-11T03:39:59.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7287999999999998e-05 2024-09-11T03:39:59.668 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.669 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.669 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.669 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:39:59.669 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:39:59.669 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:39:59.690 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:59.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:59.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:39:59.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:39:59.818 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:39:59.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:59.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:39:59.833 INFO:tasks.rados.rados.2.smithi175.stdout:431: done (7 left) 2024-09-11T03:39:59.833 INFO:tasks.rados.rados.2.smithi175.stdout:432: done (6 left) 2024-09-11T03:39:59.833 INFO:tasks.rados.rados.2.smithi175.stdout:433: done (5 left) 2024-09-11T03:39:59.833 INFO:tasks.rados.rados.2.smithi175.stdout:434: done (4 left) 2024-09-11T03:39:59.833 INFO:tasks.rados.rados.2.smithi175.stdout:435: done (3 left) 2024-09-11T03:39:59.833 INFO:tasks.rados.rados.2.smithi175.stdout:436: done (2 left) 2024-09-11T03:39:59.833 INFO:tasks.rados.rados.2.smithi175.stdout:437: done (1 left) 2024-09-11T03:39:59.834 INFO:tasks.rados.rados.2.smithi175.stdout:439: done (0 left) 2024-09-11T03:39:59.834 INFO:tasks.rados.rados.2.smithi175.stdout:440: delete oid 11 current snap is 44 2024-09-11T03:39:59.837 INFO:tasks.rados.rados.2.smithi175.stdout:440: done (0 left) 2024-09-11T03:39:59.837 INFO:tasks.rados.rados.2.smithi175.stdout:441: write oid 33 current snap is 44 2024-09-11T03:39:59.837 INFO:tasks.rados.rados.2.smithi175.stdout:441: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2024-09-11T03:39:59.844 INFO:tasks.rados.rados.2.smithi175.stdout:441: writing smithi17557870-33 from 675681 to 1238720 tid 1 2024-09-11T03:39:59.851 INFO:tasks.rados.rados.2.smithi175.stdout:441: writing smithi17557870-33 from 1858793 to 2605506 tid 2 2024-09-11T03:39:59.854 INFO:tasks.rados.rados.2.smithi175.stdout:441: writing smithi17557870-33 from 3025937 to 3075681 tid 3 2024-09-11T03:39:59.854 INFO:tasks.rados.rados.2.smithi175.stdout:442: write oid 19 current snap is 44 2024-09-11T03:39:59.854 INFO:tasks.rados.rados.2.smithi175.stdout:442: seq_num 184 ranges {692488=638288,1917275=606724} 2024-09-11T03:39:59.861 INFO:tasks.rados.rados.2.smithi175.stdout:442: writing smithi17557870-19 from 692488 to 1330776 tid 1 2024-09-11T03:39:59.869 INFO:tasks.rados.rados.2.smithi175.stdout:442: writing smithi17557870-19 from 1917275 to 2523999 tid 2 2024-09-11T03:39:59.869 INFO:tasks.rados.rados.2.smithi175.stdout:443: copy_from oid 32 from oid 28 current snap is 44 2024-09-11T03:39:59.869 INFO:tasks.rados.rados.2.smithi175.stdout:441: finishing write tid 1 to smithi17557870-33 2024-09-11T03:39:59.869 INFO:tasks.rados.rados.2.smithi175.stdout:441: finishing write tid 2 to smithi17557870-33 2024-09-11T03:39:59.869 INFO:tasks.rados.rados.2.smithi175.stdout:441: finishing write tid 3 to smithi17557870-33 2024-09-11T03:39:59.869 INFO:tasks.rados.rados.2.smithi175.stdout:441: finishing write tid 4 to smithi17557870-33 2024-09-11T03:39:59.869 INFO:tasks.rados.rados.2.smithi175.stdout:441: finishing write tid 5 to smithi17557870-33 2024-09-11T03:39:59.869 INFO:tasks.rados.rados.2.smithi175.stdout:441: oid 33 updating version 0 to 144 2024-09-11T03:39:59.870 INFO:tasks.rados.rados.2.smithi175.stdout:441: oid 33 updating version 144 to 145 2024-09-11T03:39:59.870 INFO:tasks.rados.rados.2.smithi175.stdout:441: oid 33 version 145 is already newer than 143 2024-09-11T03:39:59.870 INFO:tasks.rados.rados.2.smithi175.stdout:441: oid 33 version 145 is already newer than 142 2024-09-11T03:39:59.870 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 145 (ObjNum 183 snap 44 seq_num 183) dirty exists 2024-09-11T03:39:59.870 INFO:tasks.rados.rados.2.smithi175.stdout:441: left oid 33 (ObjNum 183 snap 44 seq_num 183) 2024-09-11T03:39:59.870 INFO:tasks.rados.rados.2.smithi175.stdout:442: finishing write tid 1 to smithi17557870-19 2024-09-11T03:39:59.870 INFO:tasks.rados.rados.2.smithi175.stdout:441: done (2 left) 2024-09-11T03:39:59.870 INFO:tasks.rados.rados.2.smithi175.stdout:444: snap_remove snap 43 2024-09-11T03:39:59.920 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:39:59.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:39:59.942 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:39:59.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:00.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:00.068 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:00.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:59 smithi084 ceph-mon[27254]: 1.0 scrub starts 2024-09-11T03:40:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:59 smithi084 ceph-mon[27254]: 1.0 scrub ok 2024-09-11T03:40:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:59 smithi084 ceph-mon[27254]: pgmap v427: 27 pgs: 10 unknown, 17 active+clean; 121 MiB data, 561 MiB used, 983 GiB / 983 GiB avail; 5.1 MiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-11T03:40:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:59 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:40:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:59 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:40:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:59 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/207242469' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:39:59 smithi084 ceph-mon[27254]: osdmap e183: 12 total, 12 up, 11 in 2024-09-11T03:40:00.173 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:00.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:00.198 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:00.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:59 smithi175 ceph-mon[30894]: 1.0 scrub starts 2024-09-11T03:40:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:59 smithi175 ceph-mon[30894]: 1.0 scrub ok 2024-09-11T03:40:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:59 smithi175 ceph-mon[30894]: pgmap v427: 27 pgs: 10 unknown, 17 active+clean; 121 MiB data, 561 MiB used, 983 GiB / 983 GiB avail; 5.1 MiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-11T03:40:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:59 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:40:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:59 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:40:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:59 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/207242469' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:39:59 smithi175 ceph-mon[30894]: osdmap e183: 12 total, 12 up, 11 in 2024-09-11T03:40:00.299 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:00.323 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:00.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:00.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:00.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:59 smithi121 ceph-mon[30729]: 1.0 scrub starts 2024-09-11T03:40:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:59 smithi121 ceph-mon[30729]: 1.0 scrub ok 2024-09-11T03:40:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:59 smithi121 ceph-mon[30729]: pgmap v427: 27 pgs: 10 unknown, 17 active+clean; 121 MiB data, 561 MiB used, 983 GiB / 983 GiB avail; 5.1 MiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-11T03:40:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:59 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:40:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:59 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:40:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:59 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/207242469' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:39:59 smithi121 ceph-mon[30729]: osdmap e183: 12 total, 12 up, 11 in 2024-09-11T03:40:00.424 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:00.447 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:00.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:00.448 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:00.549 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:00.570 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:00.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:00.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:00.658 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:40:00.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:00.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:00.700 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:00.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:00.802 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:00.825 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:00.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:00.826 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:00.834 INFO:tasks.rados.rados.2.smithi175.stdout:444: done (2 left) 2024-09-11T03:40:00.834 INFO:tasks.rados.rados.2.smithi175.stdout:445: snap_create 2024-09-11T03:40:00.834 INFO:tasks.rados.rados.2.smithi175.stdout:442: finishing write tid 2 to smithi17557870-19 2024-09-11T03:40:00.834 INFO:tasks.rados.rados.2.smithi175.stdout:442: finishing write tid 3 to smithi17557870-19 2024-09-11T03:40:00.835 INFO:tasks.rados.rados.2.smithi175.stdout:442: finishing write tid 4 to smithi17557870-19 2024-09-11T03:40:00.835 INFO:tasks.rados.rados.2.smithi175.stdout:442: oid 19 updating version 0 to 89 2024-09-11T03:40:00.835 INFO:tasks.rados.rados.2.smithi175.stdout:442: oid 19 version 89 is already newer than 88 2024-09-11T03:40:00.835 INFO:tasks.rados.rados.2.smithi175.stdout:442: oid 19 updating version 89 to 90 2024-09-11T03:40:00.835 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 90 (ObjNum 184 snap 44 seq_num 184) dirty exists 2024-09-11T03:40:00.835 INFO:tasks.rados.rados.2.smithi175.stdout:442: left oid 19 (ObjNum 184 snap 44 seq_num 184) 2024-09-11T03:40:00.835 INFO:tasks.rados.rados.2.smithi175.stdout:443: finishing copy_from racing read to smithi17557870-32 2024-09-11T03:40:00.835 INFO:tasks.rados.rados.2.smithi175.stdout:443: finishing copy_from to smithi17557870-32 2024-09-11T03:40:00.835 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 77 (ObjNum 109 snap 16 seq_num 109) dirty exists 2024-09-11T03:40:00.927 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:00.949 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:00.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:00.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:00.975 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T03:40:01.013 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:01.051 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:01.077 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:01.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:01.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:01.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:01.205 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:01.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:00 smithi175 ceph-mon[30894]: overall HEALTH_OK 2024-09-11T03:40:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:00 smithi175 ceph-mon[30894]: 1.a scrub starts 2024-09-11T03:40:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:00 smithi175 ceph-mon[30894]: 1.a scrub ok 2024-09-11T03:40:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:00 smithi175 ceph-mon[30894]: osdmap e184: 12 total, 12 up, 11 in 2024-09-11T03:40:01.255 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:00 smithi084 ceph-mon[27254]: overall HEALTH_OK 2024-09-11T03:40:01.255 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:00 smithi084 ceph-mon[27254]: 1.a scrub starts 2024-09-11T03:40:01.255 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:00 smithi084 ceph-mon[27254]: 1.a scrub ok 2024-09-11T03:40:01.255 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:00 smithi084 ceph-mon[27254]: osdmap e184: 12 total, 12 up, 11 in 2024-09-11T03:40:01.307 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:01.328 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:01.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:01.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:00 smithi121 ceph-mon[30729]: overall HEALTH_OK 2024-09-11T03:40:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:00 smithi121 ceph-mon[30729]: 1.a scrub starts 2024-09-11T03:40:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:00 smithi121 ceph-mon[30729]: 1.a scrub ok 2024-09-11T03:40:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:00 smithi121 ceph-mon[30729]: osdmap e184: 12 total, 12 up, 11 in 2024-09-11T03:40:01.430 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:01.454 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:01.455 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:01.536 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:01.556 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:01.577 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:01.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:01.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:01.679 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:01.705 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:01.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:01.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:01.807 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:01.826 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:40:01.829 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:01.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:01.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:01.831 INFO:tasks.rados.rados.2.smithi175.stdout:442: done (2 left) 2024-09-11T03:40:01.831 INFO:tasks.rados.rados.2.smithi175.stdout:443: done (1 left) 2024-09-11T03:40:01.831 INFO:tasks.rados.rados.2.smithi175.stdout:445: done (0 left) 2024-09-11T03:40:01.831 INFO:tasks.rados.rados.2.smithi175.stdout:446: read oid 17 snap -1 2024-09-11T03:40:01.831 INFO:tasks.rados.rados.2.smithi175.stdout:446: expect (ObjNum 146 snap 32 seq_num 146) 2024-09-11T03:40:01.831 INFO:tasks.rados.rados.2.smithi175.stdout:447: copy_from oid 8 from oid 28 current snap is 45 2024-09-11T03:40:01.831 INFO:tasks.rados.rados.2.smithi175.stdout:448: delete oid 36 current snap is 45 2024-09-11T03:40:01.837 INFO:tasks.rados.rados.2.smithi175.stdout:448: done (2 left) 2024-09-11T03:40:01.837 INFO:tasks.rados.rados.2.smithi175.stdout:449: read oid 29 snap -1 2024-09-11T03:40:01.837 INFO:tasks.rados.rados.2.smithi175.stdout:449: expect (ObjNum 132 snap 22 seq_num 132) 2024-09-11T03:40:01.837 INFO:tasks.rados.rados.2.smithi175.stdout:450: write oid 27 current snap is 45 2024-09-11T03:40:01.837 INFO:tasks.rados.rados.2.smithi175.stdout:450: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2024-09-11T03:40:01.846 INFO:tasks.rados.rados.2.smithi175.stdout:450: writing smithi17557870-27 from 726102 to 1514888 tid 1 2024-09-11T03:40:01.851 INFO:tasks.rados.rados.2.smithi175.stdout:450: writing smithi17557870-27 from 1917886 to 2528279 tid 2 2024-09-11T03:40:01.855 INFO:tasks.rados.rados.2.smithi175.stdout:450: writing smithi17557870-27 from 3108206 to 3126102 tid 3 2024-09-11T03:40:01.855 INFO:tasks.rados.rados.2.smithi175.stdout:451: write oid 26 current snap is 45 2024-09-11T03:40:01.855 INFO:tasks.rados.rados.2.smithi175.stdout:451: seq_num 187 ranges {742909=464035,1976368=754051} 2024-09-11T03:40:01.861 INFO:tasks.rados.rados.2.smithi175.stdout:451: writing smithi17557870-26 from 742909 to 1206944 tid 1 2024-09-11T03:40:01.870 INFO:tasks.rados.rados.2.smithi175.stdout:451: writing smithi17557870-26 from 1976368 to 2730419 tid 2 2024-09-11T03:40:01.870 INFO:tasks.rados.rados.2.smithi175.stdout:452: delete oid 31 current snap is 45 2024-09-11T03:40:01.870 INFO:tasks.rados.rados.2.smithi175.stdout:446: expect (ObjNum 146 snap 32 seq_num 146) 2024-09-11T03:40:01.930 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:01.953 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:01.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:01.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:02.003 INFO:tasks.rados.rados.2.smithi175.stdout:449: expect (ObjNum 132 snap 22 seq_num 132) 2024-09-11T03:40:02.054 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:02.068 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:40:02.068 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:40:02.068 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:40:02.068 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:40:02.076 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:02.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:02.077 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:02.101 INFO:tasks.rados.rados.2.smithi175.stdout:447: finishing copy_from racing read to smithi17557870-8 2024-09-11T03:40:02.101 INFO:tasks.rados.rados.2.smithi175.stdout:447: finishing copy_from to smithi17557870-8 2024-09-11T03:40:02.101 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 77 (ObjNum 109 snap 16 seq_num 109) dirty exists 2024-09-11T03:40:02.101 INFO:tasks.rados.rados.2.smithi175.stdout:450: finishing write tid 1 to smithi17557870-27 2024-09-11T03:40:02.101 INFO:tasks.rados.rados.2.smithi175.stdout:450: finishing write tid 2 to smithi17557870-27 2024-09-11T03:40:02.101 INFO:tasks.rados.rados.2.smithi175.stdout:446: done (5 left) 2024-09-11T03:40:02.102 INFO:tasks.rados.rados.2.smithi175.stdout:447: done (4 left) 2024-09-11T03:40:02.102 INFO:tasks.rados.rados.2.smithi175.stdout:449: done (3 left) 2024-09-11T03:40:02.102 INFO:tasks.rados.rados.2.smithi175.stdout:452: done (2 left) 2024-09-11T03:40:02.102 INFO:tasks.rados.rados.2.smithi175.stdout:453: copy_from oid 45 from oid 6 current snap is 45 2024-09-11T03:40:02.102 INFO:tasks.rados.rados.2.smithi175.stdout:450: finishing write tid 3 to smithi17557870-27 2024-09-11T03:40:02.102 INFO:tasks.rados.rados.2.smithi175.stdout:450: finishing write tid 4 to smithi17557870-27 2024-09-11T03:40:02.102 INFO:tasks.rados.rados.2.smithi175.stdout:450: finishing write tid 5 to smithi17557870-27 2024-09-11T03:40:02.102 INFO:tasks.rados.rados.2.smithi175.stdout:450: oid 27 updating version 0 to 104 2024-09-11T03:40:02.102 INFO:tasks.rados.rados.2.smithi175.stdout:450: oid 27 version 104 is already newer than 103 2024-09-11T03:40:02.102 INFO:tasks.rados.rados.2.smithi175.stdout:450: oid 27 version 104 is already newer than 102 2024-09-11T03:40:02.102 INFO:tasks.rados.rados.2.smithi175.stdout:450: oid 27 version 104 is already newer than 101 2024-09-11T03:40:02.102 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 104 (ObjNum 186 snap 45 seq_num 186) dirty exists 2024-09-11T03:40:02.102 INFO:tasks.rados.rados.2.smithi175.stdout:450: left oid 27 (ObjNum 186 snap 45 seq_num 186) 2024-09-11T03:40:02.103 INFO:tasks.rados.rados.2.smithi175.stdout:451: finishing write tid 1 to smithi17557870-26 2024-09-11T03:40:02.103 INFO:tasks.rados.rados.2.smithi175.stdout:451: finishing write tid 2 to smithi17557870-26 2024-09-11T03:40:02.103 INFO:tasks.rados.rados.2.smithi175.stdout:451: finishing write tid 3 to smithi17557870-26 2024-09-11T03:40:02.103 INFO:tasks.rados.rados.2.smithi175.stdout:451: finishing write tid 4 to smithi17557870-26 2024-09-11T03:40:02.103 INFO:tasks.rados.rados.2.smithi175.stdout:451: oid 26 updating version 0 to 74 2024-09-11T03:40:02.103 INFO:tasks.rados.rados.2.smithi175.stdout:451: oid 26 version 74 is already newer than 73 2024-09-11T03:40:02.103 INFO:tasks.rados.rados.2.smithi175.stdout:451: oid 26 updating version 74 to 75 2024-09-11T03:40:02.103 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 75 (ObjNum 187 snap 45 seq_num 187) dirty exists 2024-09-11T03:40:02.103 INFO:tasks.rados.rados.2.smithi175.stdout:451: left oid 26 (ObjNum 187 snap 45 seq_num 187) 2024-09-11T03:40:02.103 INFO:tasks.rados.rados.2.smithi175.stdout:450: done (2 left) 2024-09-11T03:40:02.103 INFO:tasks.rados.rados.2.smithi175.stdout:451: done (1 left) 2024-09-11T03:40:02.103 INFO:tasks.rados.rados.2.smithi175.stdout:454: snap_remove snap 36 2024-09-11T03:40:02.163 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:40:02.163 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:40:02.163 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11] 2024-09-11T03:40:02.163 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd out 10 2024-09-11T03:40:02.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:02.203 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:02.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:02.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:01 smithi175 ceph-mon[30894]: 1.8 scrub starts 2024-09-11T03:40:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:01 smithi175 ceph-mon[30894]: 1.8 scrub ok 2024-09-11T03:40:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:01 smithi175 ceph-mon[30894]: 1.3 scrub starts 2024-09-11T03:40:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:01 smithi175 ceph-mon[30894]: 1.3 scrub ok 2024-09-11T03:40:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:01 smithi175 ceph-mon[30894]: 1.4 scrub starts 2024-09-11T03:40:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:01 smithi175 ceph-mon[30894]: 1.4 scrub ok 2024-09-11T03:40:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:01 smithi175 ceph-mon[30894]: 1.9 scrub starts 2024-09-11T03:40:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:01 smithi175 ceph-mon[30894]: 1.9 scrub ok 2024-09-11T03:40:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:01 smithi175 ceph-mon[30894]: pgmap v430: 27 pgs: 10 unknown, 17 active+clean; 121 MiB data, 565 MiB used, 983 GiB / 983 GiB avail; 5.1 MiB/s rd, 890 KiB/s wr, 9 op/s 2024-09-11T03:40:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:01 smithi175 ceph-mon[30894]: 1.6 scrub starts 2024-09-11T03:40:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:01 smithi175 ceph-mon[30894]: 1.6 scrub ok 2024-09-11T03:40:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/222306060' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:40:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:01 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/222306060' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:40:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:01 smithi175 ceph-mon[30894]: osdmap e185: 12 total, 12 up, 11 in 2024-09-11T03:40:02.277 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:01 smithi084 ceph-mon[27254]: 1.8 scrub starts 2024-09-11T03:40:02.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:01 smithi084 ceph-mon[27254]: 1.8 scrub ok 2024-09-11T03:40:02.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:01 smithi084 ceph-mon[27254]: 1.3 scrub starts 2024-09-11T03:40:02.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:01 smithi084 ceph-mon[27254]: 1.3 scrub ok 2024-09-11T03:40:02.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:01 smithi084 ceph-mon[27254]: 1.4 scrub starts 2024-09-11T03:40:02.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:01 smithi084 ceph-mon[27254]: 1.4 scrub ok 2024-09-11T03:40:02.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:01 smithi084 ceph-mon[27254]: 1.9 scrub starts 2024-09-11T03:40:02.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:01 smithi084 ceph-mon[27254]: 1.9 scrub ok 2024-09-11T03:40:02.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:01 smithi084 ceph-mon[27254]: pgmap v430: 27 pgs: 10 unknown, 17 active+clean; 121 MiB data, 565 MiB used, 983 GiB / 983 GiB avail; 5.1 MiB/s rd, 890 KiB/s wr, 9 op/s 2024-09-11T03:40:02.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:01 smithi084 ceph-mon[27254]: 1.6 scrub starts 2024-09-11T03:40:02.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:01 smithi084 ceph-mon[27254]: 1.6 scrub ok 2024-09-11T03:40:02.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/222306060' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:40:02.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:01 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:02.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/222306060' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:40:02.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:01 smithi084 ceph-mon[27254]: osdmap e185: 12 total, 12 up, 11 in 2024-09-11T03:40:02.305 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:02.314 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:40:02.331 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:02.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:02.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:01 smithi121 ceph-mon[30729]: 1.8 scrub starts 2024-09-11T03:40:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:01 smithi121 ceph-mon[30729]: 1.8 scrub ok 2024-09-11T03:40:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:01 smithi121 ceph-mon[30729]: 1.3 scrub starts 2024-09-11T03:40:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:01 smithi121 ceph-mon[30729]: 1.3 scrub ok 2024-09-11T03:40:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:01 smithi121 ceph-mon[30729]: 1.4 scrub starts 2024-09-11T03:40:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:01 smithi121 ceph-mon[30729]: 1.4 scrub ok 2024-09-11T03:40:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:01 smithi121 ceph-mon[30729]: 1.9 scrub starts 2024-09-11T03:40:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:01 smithi121 ceph-mon[30729]: 1.9 scrub ok 2024-09-11T03:40:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:01 smithi121 ceph-mon[30729]: pgmap v430: 27 pgs: 10 unknown, 17 active+clean; 121 MiB data, 565 MiB used, 983 GiB / 983 GiB avail; 5.1 MiB/s rd, 890 KiB/s wr, 9 op/s 2024-09-11T03:40:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:01 smithi121 ceph-mon[30729]: 1.6 scrub starts 2024-09-11T03:40:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:01 smithi121 ceph-mon[30729]: 1.6 scrub ok 2024-09-11T03:40:02.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/222306060' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:40:02.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:01 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:02.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/222306060' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:40:02.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:01 smithi121 ceph-mon[30729]: osdmap e185: 12 total, 12 up, 11 in 2024-09-11T03:40:02.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:02.457 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:02.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:02.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:02.559 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:02.581 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:02.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:02.582 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:02.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:02.704 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:02.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:02.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:02.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:02.832 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:02.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:02.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:02.952 INFO:tasks.rados.rados.2.smithi175.stdout:454: done (1 left) 2024-09-11T03:40:02.952 INFO:tasks.rados.rados.2.smithi175.stdout:455: rollback oid 24 current snap is 45 2024-09-11T03:40:02.953 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 42 2024-09-11T03:40:02.953 INFO:tasks.rados.rados.2.smithi175.stdout:453: finishing copy_from racing read to smithi17557870-45 2024-09-11T03:40:02.953 INFO:tasks.rados.rados.2.smithi175.stdout:453: finishing copy_from to smithi17557870-45 2024-09-11T03:40:02.953 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 130 (ObjNum 164 snap 36 seq_num 164) dirty exists 2024-09-11T03:40:02.953 INFO:tasks.rados.rados.2.smithi175.stdout:453: done (1 left) 2024-09-11T03:40:02.953 INFO:tasks.rados.rados.2.smithi175.stdout:456: write oid 13 current snap is 45 2024-09-11T03:40:02.953 INFO:tasks.rados.rados.2.smithi175.stdout:456: seq_num 189 ranges {776523=614533,1976979=757720} 2024-09-11T03:40:02.960 INFO:tasks.rados.rados.2.smithi175.stdout:456: writing smithi17557870-13 from 776523 to 1391056 tid 1 2024-09-11T03:40:02.961 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:02.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:02.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:02.968 INFO:tasks.rados.rados.2.smithi175.stdout:456: writing smithi17557870-13 from 1976979 to 2734699 tid 2 2024-09-11T03:40:02.968 INFO:tasks.rados.rados.2.smithi175.stdout:457: copy_from oid 9 from oid 50 current snap is 45 2024-09-11T03:40:02.968 INFO:tasks.rados.rados.2.smithi175.stdout:455: finishing rollback tid 0 to smithi17557870-24 2024-09-11T03:40:02.968 INFO:tasks.rados.rados.2.smithi175.stdout:455: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:40:02.968 INFO:tasks.rados.rados.2.smithi175.stdout:455: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:40:02.968 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 116 (ObjNum 177 snap 42 seq_num 177) dirty exists 2024-09-11T03:40:02.968 INFO:tasks.rados.rados.2.smithi175.stdout:456: finishing write tid 1 to smithi17557870-13 2024-09-11T03:40:02.968 INFO:tasks.rados.rados.2.smithi175.stdout:455: done (2 left) 2024-09-11T03:40:02.969 INFO:tasks.rados.rados.2.smithi175.stdout:458: snap_create 2024-09-11T03:40:02.969 INFO:tasks.rados.rados.2.smithi175.stdout:457: finishing copy_from to smithi17557870-9 2024-09-11T03:40:02.969 INFO:tasks.rados.rados.2.smithi175.stdout:457: got expected ENOENT (src dne) 2024-09-11T03:40:02.970 INFO:tasks.rados.rados.2.smithi175.stdout:457: finishing copy_from racing read to smithi17557870-9 2024-09-11T03:40:02.970 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T03:40:02.971 INFO:tasks.rados.rados.2.smithi175.stdout:456: finishing write tid 2 to smithi17557870-13 2024-09-11T03:40:02.971 INFO:tasks.rados.rados.2.smithi175.stdout:456: finishing write tid 3 to smithi17557870-13 2024-09-11T03:40:02.971 INFO:tasks.rados.rados.2.smithi175.stdout:456: finishing write tid 4 to smithi17557870-13 2024-09-11T03:40:02.972 INFO:tasks.rados.rados.2.smithi175.stdout:456: oid 13 updating version 0 to 153 2024-09-11T03:40:02.972 INFO:tasks.rados.rados.2.smithi175.stdout:456: oid 13 version 153 is already newer than 151 2024-09-11T03:40:02.972 INFO:tasks.rados.rados.2.smithi175.stdout:456: oid 13 version 153 is already newer than 152 2024-09-11T03:40:02.972 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 153 (ObjNum 189 snap 45 seq_num 189) dirty exists 2024-09-11T03:40:02.972 INFO:tasks.rados.rados.2.smithi175.stdout:456: left oid 13 (ObjNum 189 snap 45 seq_num 189) 2024-09-11T03:40:03.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:03.085 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:03.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:03.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:03.187 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:03.208 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:03.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:03.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:03.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:02 smithi175 ceph-mon[30894]: 1.2 scrub starts 2024-09-11T03:40:03.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:02 smithi175 ceph-mon[30894]: 1.2 scrub ok 2024-09-11T03:40:03.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:02 smithi175 ceph-mon[30894]: 1.5 scrub starts 2024-09-11T03:40:03.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:03.331 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:03.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:02 smithi084 ceph-mon[27254]: 1.2 scrub starts 2024-09-11T03:40:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:02 smithi084 ceph-mon[27254]: 1.2 scrub ok 2024-09-11T03:40:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:02 smithi084 ceph-mon[27254]: 1.5 scrub starts 2024-09-11T03:40:03.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:02 smithi121 ceph-mon[30729]: 1.2 scrub starts 2024-09-11T03:40:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:02 smithi121 ceph-mon[30729]: 1.2 scrub ok 2024-09-11T03:40:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:02 smithi121 ceph-mon[30729]: 1.5 scrub starts 2024-09-11T03:40:03.433 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:03.433 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:03.461 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:03.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:03.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:03.477 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:03.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:03.589 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:03.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:03.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:03.691 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:03.717 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:03.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:03.718 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:03.819 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:03.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:03.842 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:03.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:03.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:03.954 INFO:tasks.rados.rados.2.smithi175.stdout:456: done (2 left) 2024-09-11T03:40:03.954 INFO:tasks.rados.rados.2.smithi175.stdout:457: done (1 left) 2024-09-11T03:40:03.954 INFO:tasks.rados.rados.2.smithi175.stdout:458: done (0 left) 2024-09-11T03:40:03.954 INFO:tasks.rados.rados.2.smithi175.stdout:459: delete oid 43 current snap is 46 2024-09-11T03:40:03.958 INFO:tasks.rados.rados.2.smithi175.stdout:459: done (0 left) 2024-09-11T03:40:03.958 INFO:tasks.rados.rados.2.smithi175.stdout:460: delete oid 11 current snap is 46 2024-09-11T03:40:03.960 INFO:tasks.rados.rados.2.smithi175.stdout:460: done (0 left) 2024-09-11T03:40:03.960 INFO:tasks.rados.rados.2.smithi175.stdout:461: snap_remove snap 44 2024-09-11T03:40:03.969 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:03.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:03.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:04.032 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:03 smithi084 ceph-mon[27254]: pgmap v433: 27 pgs: 1 unknown, 26 active+clean; 134 MiB data, 582 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:40:04.033 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:03 smithi084 ceph-mon[27254]: 1.7 scrub starts 2024-09-11T03:40:04.033 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:03 smithi084 ceph-mon[27254]: osdmap e186: 12 total, 12 up, 11 in 2024-09-11T03:40:04.070 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:04.097 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:04.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:04.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:04.198 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:04.224 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:04.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:04.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:03 smithi175 ceph-mon[30894]: pgmap v433: 27 pgs: 1 unknown, 26 active+clean; 134 MiB data, 582 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:40:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:03 smithi175 ceph-mon[30894]: 1.7 scrub starts 2024-09-11T03:40:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:03 smithi175 ceph-mon[30894]: osdmap e186: 12 total, 12 up, 11 in 2024-09-11T03:40:04.246 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:04.326 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:04.352 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:04.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:04.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:04.358 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:40:04.358 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:40:04.358 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:40:04.358 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:40:04.358 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:40:04.358 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:40:04.358 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:40:04.358 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:40:04.358 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:40:04.358 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:40:04.359 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:40:04.359 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:40:04.359 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:40:04.359 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:40:04.361 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:40:04.361 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:40:04.379 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:40:04.379 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:40:04.396 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:40:04.396 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:40:04.413 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:40:04.413 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:40:04.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:03 smithi121 ceph-mon[30729]: pgmap v433: 27 pgs: 1 unknown, 26 active+clean; 134 MiB data, 582 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:40:04.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:03 smithi121 ceph-mon[30729]: 1.7 scrub starts 2024-09-11T03:40:04.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:03 smithi121 ceph-mon[30729]: osdmap e186: 12 total, 12 up, 11 in 2024-09-11T03:40:04.429 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:40:04.429 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:40:04.453 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:04.474 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:04.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:04.576 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:04.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:04.599 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:04.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:04.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:04 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a[27250]: 2024-09-11T03:40:04.322+0000 7fae0fb7d640 -1 mon.a@0(leader).osd e187 definitely_dead 0 2024-09-11T03:40:04.700 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:04.722 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:04.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:04.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:04.824 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:04.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:04.846 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:04.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:04.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:04.956 INFO:teuthology.orchestra.run.smithi084.stderr:marked out osd.10. 2024-09-11T03:40:04.962 INFO:tasks.rados.rados.2.smithi175.stdout:461: done (0 left) 2024-09-11T03:40:04.962 INFO:tasks.rados.rados.2.smithi175.stdout:462: write (excl) oid 7 current snap is 46 2024-09-11T03:40:04.962 INFO:tasks.rados.rados.2.smithi175.stdout:462: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2024-09-11T03:40:04.967 INFO:tasks.rados.rados.2.smithi175.stdout:462: writing smithi17557870-7 from 426944 to 983577 tid 1 2024-09-11T03:40:04.973 INFO:tasks.rados.rados.2.smithi175.stdout:462: writing smithi17557870-7 from 1468778 to 2090178 tid 2 2024-09-11T03:40:04.974 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:04.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:04.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:04.980 INFO:tasks.rados.rados.2.smithi175.stdout:462: writing smithi17557870-7 from 2872744 to 3226944 tid 3 2024-09-11T03:40:04.980 INFO:tasks.rados.rados.2.smithi175.stdout:463: snap_create 2024-09-11T03:40:04.980 INFO:tasks.rados.rados.2.smithi175.stdout:462: finishing write tid 1 to smithi17557870-7 2024-09-11T03:40:04.980 INFO:tasks.rados.rados.2.smithi175.stdout:462: finishing write tid 2 to smithi17557870-7 2024-09-11T03:40:04.983 INFO:tasks.rados.rados.2.smithi175.stdout:462: finishing write tid 3 to smithi17557870-7 2024-09-11T03:40:04.983 INFO:tasks.rados.rados.2.smithi175.stdout:462: finishing write tid 4 to smithi17557870-7 2024-09-11T03:40:04.983 INFO:tasks.rados.rados.2.smithi175.stdout:462: finishing write tid 5 to smithi17557870-7 2024-09-11T03:40:04.984 INFO:tasks.rados.rados.2.smithi175.stdout:462: oid 7 updating version 0 to 91 2024-09-11T03:40:04.984 INFO:tasks.rados.rados.2.smithi175.stdout:462: oid 7 updating version 91 to 92 2024-09-11T03:40:04.984 INFO:tasks.rados.rados.2.smithi175.stdout:462: oid 7 updating version 92 to 94 2024-09-11T03:40:04.984 INFO:tasks.rados.rados.2.smithi175.stdout:462: oid 7 version 94 is already newer than 93 2024-09-11T03:40:04.984 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 94 (ObjNum 192 snap 46 seq_num 192) dirty exists 2024-09-11T03:40:04.984 INFO:tasks.rados.rados.2.smithi175.stdout:462: left oid 7 (ObjNum 192 snap 46 seq_num 192) 2024-09-11T03:40:05.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:05.101 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:05.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:05.203 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:04 smithi175 ceph-mon[30894]: 1.1 scrub starts 2024-09-11T03:40:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:04 smithi175 ceph-mon[30894]: osdmap e187: 12 total, 12 up, 11 in 2024-09-11T03:40:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:04 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1075964588' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:04 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2197353092' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-09-11T03:40:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:04 smithi175 ceph-mon[30894]: Client client.admin marked osd.10 out, while it was still marked up 2024-09-11T03:40:05.228 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:05.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:05.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:05.257 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:04 smithi084 ceph-mon[27254]: 1.1 scrub starts 2024-09-11T03:40:05.257 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:04 smithi084 ceph-mon[27254]: osdmap e187: 12 total, 12 up, 11 in 2024-09-11T03:40:05.257 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:04 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1075964588' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:05.257 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:04 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2197353092' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-09-11T03:40:05.258 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:04 smithi084 ceph-mon[27254]: Client client.admin marked osd.10 out, while it was still marked up 2024-09-11T03:40:05.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:05.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:05.357 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:05.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:04 smithi121 ceph-mon[30729]: 1.1 scrub starts 2024-09-11T03:40:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:04 smithi121 ceph-mon[30729]: osdmap e187: 12 total, 12 up, 11 in 2024-09-11T03:40:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:04 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1075964588' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:04 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2197353092' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-09-11T03:40:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:04 smithi121 ceph-mon[30729]: Client client.admin marked osd.10 out, while it was still marked up 2024-09-11T03:40:05.459 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:05.470 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:40:05.485 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:05.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:05.486 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:05.587 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:05.610 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:05.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:05.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:05.708 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:05.709 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:05.712 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:05.738 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:05.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:05.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:05.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:05.865 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:05.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:05.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:05.963 INFO:tasks.rados.rados.2.smithi175.stdout:462: done (1 left) 2024-09-11T03:40:05.963 INFO:tasks.rados.rados.2.smithi175.stdout:463: done (0 left) 2024-09-11T03:40:05.963 INFO:tasks.rados.rados.2.smithi175.stdout:464: write (excl) oid 33 current snap is 47 2024-09-11T03:40:05.963 INFO:tasks.rados.rados.2.smithi175.stdout:464: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2024-09-11T03:40:05.966 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:05.970 INFO:tasks.rados.rados.2.smithi175.stdout:464: writing smithi17557870-33 from 443751 to 1075633 tid 1 2024-09-11T03:40:05.977 INFO:tasks.rados.rados.2.smithi175.stdout:464: writing smithi17557870-33 from 1810907 to 2575965 tid 2 2024-09-11T03:40:05.981 INFO:tasks.rados.rados.2.smithi175.stdout:464: writing smithi17557870-33 from 3067461 to 3243751 tid 3 2024-09-11T03:40:05.981 INFO:tasks.rados.rados.2.smithi175.stdout:465: write oid 22 current snap is 47 2024-09-11T03:40:05.981 INFO:tasks.rados.rados.2.smithi175.stdout:464: finishing write tid 1 to smithi17557870-33 2024-09-11T03:40:05.981 INFO:tasks.rados.rados.2.smithi175.stdout:465: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2024-09-11T03:40:05.987 INFO:tasks.rados.rados.2.smithi175.stdout:465: writing smithi17557870-22 from 460558 to 1167689 tid 1 2024-09-11T03:40:05.992 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:05.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:05.993 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:05.993 INFO:tasks.rados.rados.2.smithi175.stdout:465: writing smithi17557870-22 from 1869389 to 2378105 tid 2 2024-09-11T03:40:05.998 INFO:tasks.rados.rados.2.smithi175.stdout:465: writing smithi17557870-22 from 3094884 to 3260558 tid 3 2024-09-11T03:40:05.999 INFO:tasks.rados.rados.2.smithi175.stdout:466: snap_create 2024-09-11T03:40:05.999 INFO:tasks.rados.rados.2.smithi175.stdout:464: finishing write tid 2 to smithi17557870-33 2024-09-11T03:40:05.999 INFO:tasks.rados.rados.2.smithi175.stdout:464: finishing write tid 3 to smithi17557870-33 2024-09-11T03:40:05.999 INFO:tasks.rados.rados.2.smithi175.stdout:464: finishing write tid 4 to smithi17557870-33 2024-09-11T03:40:05.999 INFO:tasks.rados.rados.2.smithi175.stdout:464: finishing write tid 5 to smithi17557870-33 2024-09-11T03:40:05.999 INFO:tasks.rados.rados.2.smithi175.stdout:464: oid 33 updating version 0 to 151 2024-09-11T03:40:05.999 INFO:tasks.rados.rados.2.smithi175.stdout:464: oid 33 updating version 151 to 153 2024-09-11T03:40:05.999 INFO:tasks.rados.rados.2.smithi175.stdout:464: oid 33 updating version 153 to 154 2024-09-11T03:40:05.999 INFO:tasks.rados.rados.2.smithi175.stdout:464: oid 33 version 154 is already newer than 152 2024-09-11T03:40:05.999 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 154 (ObjNum 193 snap 47 seq_num 193) dirty exists 2024-09-11T03:40:05.999 INFO:tasks.rados.rados.2.smithi175.stdout:464: left oid 33 (ObjNum 193 snap 47 seq_num 193) 2024-09-11T03:40:05.999 INFO:tasks.rados.rados.2.smithi175.stdout:465: finishing write tid 1 to smithi17557870-22 2024-09-11T03:40:05.999 INFO:tasks.rados.rados.2.smithi175.stdout:465: finishing write tid 2 to smithi17557870-22 2024-09-11T03:40:06.000 INFO:tasks.rados.rados.2.smithi175.stdout:465: finishing write tid 3 to smithi17557870-22 2024-09-11T03:40:06.000 INFO:tasks.rados.rados.2.smithi175.stdout:465: finishing write tid 4 to smithi17557870-22 2024-09-11T03:40:06.000 INFO:tasks.rados.rados.2.smithi175.stdout:465: finishing write tid 5 to smithi17557870-22 2024-09-11T03:40:06.000 INFO:tasks.rados.rados.2.smithi175.stdout:465: oid 22 updating version 0 to 157 2024-09-11T03:40:06.001 INFO:tasks.rados.rados.2.smithi175.stdout:465: oid 22 updating version 157 to 159 2024-09-11T03:40:06.001 INFO:tasks.rados.rados.2.smithi175.stdout:465: oid 22 version 159 is already newer than 156 2024-09-11T03:40:06.001 INFO:tasks.rados.rados.2.smithi175.stdout:465: oid 22 version 159 is already newer than 158 2024-09-11T03:40:06.001 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 159 (ObjNum 194 snap 47 seq_num 194) dirty exists 2024-09-11T03:40:06.001 INFO:tasks.rados.rados.2.smithi175.stdout:465: left oid 22 (ObjNum 194 snap 47 seq_num 194) 2024-09-11T03:40:06.094 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:06.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:06.116 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:06.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:06.217 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:05 smithi175 ceph-mon[30894]: pgmap v436: 27 pgs: 27 active+clean; 135 MiB data, 587 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:40:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:05 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2197353092' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-09-11T03:40:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:05 smithi175 ceph-mon[30894]: osdmap e188: 12 total, 12 up, 10 in 2024-09-11T03:40:06.242 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:06.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:06.243 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:06.263 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:05 smithi084 ceph-mon[27254]: pgmap v436: 27 pgs: 27 active+clean; 135 MiB data, 587 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:40:06.263 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:05 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2197353092' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-09-11T03:40:06.263 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:05 smithi084 ceph-mon[27254]: osdmap e188: 12 total, 12 up, 10 in 2024-09-11T03:40:06.289 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T03:40:06.344 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:06.368 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:06.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:06.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:05 smithi121 ceph-mon[30729]: pgmap v436: 27 pgs: 27 active+clean; 135 MiB data, 587 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:40:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:05 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2197353092' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-09-11T03:40:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:05 smithi121 ceph-mon[30729]: osdmap e188: 12 total, 12 up, 10 in 2024-09-11T03:40:06.469 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:06.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:06.495 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:06.495 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:06.596 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:06.621 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:06.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:06.622 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:06.723 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:06.747 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:06.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:06.748 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:06.767 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:06.781 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:06.831 INFO:tasks.rados.rados.2.smithi175.stdout:464: done (2 left) 2024-09-11T03:40:06.831 INFO:tasks.rados.rados.2.smithi175.stdout:465: done (1 left) 2024-09-11T03:40:06.831 INFO:tasks.rados.rados.2.smithi175.stdout:466: done (0 left) 2024-09-11T03:40:06.831 INFO:tasks.rados.rados.2.smithi175.stdout:467: rollback oid 19 current snap is 48 2024-09-11T03:40:06.831 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 19 to 38 2024-09-11T03:40:06.832 INFO:tasks.rados.rados.2.smithi175.stdout:468: write (excl) oid 7 current snap is 48 2024-09-11T03:40:06.832 INFO:tasks.rados.rados.2.smithi175.stdout:468: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2024-09-11T03:40:06.838 INFO:tasks.rados.rados.2.smithi175.stdout:468: writing smithi17557870-7 from 477365 to 1259745 tid 1 2024-09-11T03:40:06.846 INFO:tasks.rados.rados.2.smithi175.stdout:468: writing smithi17557870-7 from 1927871 to 2696598 tid 2 2024-09-11T03:40:06.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:06.850 INFO:tasks.rados.rados.2.smithi175.stdout:468: writing smithi17557870-7 from 3122307 to 3277365 tid 3 2024-09-11T03:40:06.850 INFO:tasks.rados.rados.2.smithi175.stdout:469: delete oid 12 current snap is 48 2024-09-11T03:40:06.850 INFO:tasks.rados.rados.2.smithi175.stdout:467: finishing rollback tid 0 to smithi17557870-19 2024-09-11T03:40:06.850 INFO:tasks.rados.rados.2.smithi175.stdout:467: finishing rollback tid 1 to smithi17557870-19 2024-09-11T03:40:06.850 INFO:tasks.rados.rados.2.smithi175.stdout:467: finishing rollback tid 2 to smithi17557870-19 2024-09-11T03:40:06.850 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 97 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T03:40:06.850 INFO:tasks.rados.rados.2.smithi175.stdout:468: finishing write tid 1 to smithi17557870-7 2024-09-11T03:40:06.851 INFO:tasks.rados.rados.2.smithi175.stdout:468: finishing write tid 2 to smithi17557870-7 2024-09-11T03:40:06.852 INFO:tasks.rados.rados.2.smithi175.stdout:468: finishing write tid 3 to smithi17557870-7 2024-09-11T03:40:06.852 INFO:tasks.rados.rados.2.smithi175.stdout:468: finishing write tid 4 to smithi17557870-7 2024-09-11T03:40:06.852 INFO:tasks.rados.rados.2.smithi175.stdout:467: done (2 left) 2024-09-11T03:40:06.852 INFO:tasks.rados.rados.2.smithi175.stdout:469: done (1 left) 2024-09-11T03:40:06.852 INFO:tasks.rados.rados.2.smithi175.stdout:470: delete oid 33 current snap is 48 2024-09-11T03:40:06.852 INFO:tasks.rados.rados.2.smithi175.stdout:468: finishing write tid 5 to smithi17557870-7 2024-09-11T03:40:06.852 INFO:tasks.rados.rados.2.smithi175.stdout:468: oid 7 updating version 0 to 100 2024-09-11T03:40:06.852 INFO:tasks.rados.rados.2.smithi175.stdout:468: oid 7 version 100 is already newer than 98 2024-09-11T03:40:06.852 INFO:tasks.rados.rados.2.smithi175.stdout:468: oid 7 updating version 100 to 101 2024-09-11T03:40:06.852 INFO:tasks.rados.rados.2.smithi175.stdout:468: oid 7 version 101 is already newer than 99 2024-09-11T03:40:06.852 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 101 (ObjNum 195 snap 48 seq_num 195) dirty exists 2024-09-11T03:40:06.852 INFO:tasks.rados.rados.2.smithi175.stdout:468: left oid 7 (ObjNum 195 snap 48 seq_num 195) 2024-09-11T03:40:06.853 INFO:tasks.rados.rados.2.smithi175.stdout:468: done (1 left) 2024-09-11T03:40:06.853 INFO:tasks.rados.rados.2.smithi175.stdout:470: done (0 left) 2024-09-11T03:40:06.853 INFO:tasks.rados.rados.2.smithi175.stdout:471: delete oid 45 current snap is 48 2024-09-11T03:40:06.856 INFO:tasks.rados.rados.2.smithi175.stdout:471: done (0 left) 2024-09-11T03:40:06.856 INFO:tasks.rados.rados.2.smithi175.stdout:472: snap_remove snap 38 2024-09-11T03:40:06.873 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:06.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:06.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:06.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:07.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:07.001 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:07.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:07.102 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:07.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:07.125 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:07.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:07.226 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:07.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:06 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:40:07.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:06 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:40:07.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:06 smithi175 ceph-mon[30894]: osdmap e189: 12 total, 12 up, 10 in 2024-09-11T03:40:07.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:06 smithi175 ceph-mon[30894]: 1.6 scrub starts 2024-09-11T03:40:07.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:06 smithi175 ceph-mon[30894]: 1.7 scrub starts 2024-09-11T03:40:07.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:06 smithi175 ceph-mon[30894]: osdmap e190: 12 total, 12 up, 10 in 2024-09-11T03:40:07.253 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:07.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:07.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:07.300 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:06 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:40:07.300 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:06 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:40:07.300 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:06 smithi084 ceph-mon[27254]: osdmap e189: 12 total, 12 up, 10 in 2024-09-11T03:40:07.300 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:06 smithi084 ceph-mon[27254]: 1.6 scrub starts 2024-09-11T03:40:07.300 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:06 smithi084 ceph-mon[27254]: 1.7 scrub starts 2024-09-11T03:40:07.300 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:06 smithi084 ceph-mon[27254]: osdmap e190: 12 total, 12 up, 10 in 2024-09-11T03:40:07.355 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:07.376 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:07.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:07.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:07.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:06 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:40:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:06 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:40:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:06 smithi121 ceph-mon[30729]: osdmap e189: 12 total, 12 up, 10 in 2024-09-11T03:40:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:06 smithi121 ceph-mon[30729]: 1.6 scrub starts 2024-09-11T03:40:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:06 smithi121 ceph-mon[30729]: 1.7 scrub starts 2024-09-11T03:40:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:06 smithi121 ceph-mon[30729]: osdmap e190: 12 total, 12 up, 10 in 2024-09-11T03:40:07.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:07.503 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:07.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:07.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:07.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:07.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:40:07.630 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:07.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:07.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:07.732 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:07.757 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:07.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:07.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:07.829 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:40:07.833 INFO:tasks.rados.rados.2.smithi175.stdout:472: done (0 left) 2024-09-11T03:40:07.833 INFO:tasks.rados.rados.2.smithi175.stdout:473: read oid 42 snap -1 2024-09-11T03:40:07.833 INFO:tasks.rados.rados.2.smithi175.stdout:473: expect (ObjNum 134 snap 22 seq_num 134) 2024-09-11T03:40:07.833 INFO:tasks.rados.rados.2.smithi175.stdout:474: snap_remove snap 47 2024-09-11T03:40:07.859 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:07.882 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:07.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:07.883 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:07.894 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:07.894 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:07.984 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:08.006 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:08.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:08.007 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:08.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:08.134 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:08.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:08.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:08.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:07 smithi084 ceph-mon[27254]: 1.5 scrub starts 2024-09-11T03:40:08.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:07 smithi084 ceph-mon[27254]: pgmap v439: 27 pgs: 27 active+clean; 134 MiB data, 549 MiB used, 893 GiB / 894 GiB avail; 5.2 MiB/s rd, 2.6 MiB/s wr, 8 op/s; 1.7 MiB/s, 2 objects/s recovering 2024-09-11T03:40:08.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:07 smithi084 ceph-mon[27254]: 1.1 scrub starts 2024-09-11T03:40:08.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1103830829' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:40:08.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:07 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:08.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1103830829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:40:08.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:07 smithi084 ceph-mon[27254]: osdmap e191: 12 total, 12 up, 10 in 2024-09-11T03:40:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:07 smithi175 ceph-mon[30894]: 1.5 scrub starts 2024-09-11T03:40:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:07 smithi175 ceph-mon[30894]: pgmap v439: 27 pgs: 27 active+clean; 134 MiB data, 549 MiB used, 893 GiB / 894 GiB avail; 5.2 MiB/s rd, 2.6 MiB/s wr, 8 op/s; 1.7 MiB/s, 2 objects/s recovering 2024-09-11T03:40:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:07 smithi175 ceph-mon[30894]: 1.1 scrub starts 2024-09-11T03:40:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1103830829' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:40:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:07 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:08.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1103830829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:40:08.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:07 smithi175 ceph-mon[30894]: osdmap e191: 12 total, 12 up, 10 in 2024-09-11T03:40:08.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:08.260 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:08.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:08.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:07 smithi121 ceph-mon[30729]: 1.5 scrub starts 2024-09-11T03:40:08.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:07 smithi121 ceph-mon[30729]: pgmap v439: 27 pgs: 27 active+clean; 134 MiB data, 549 MiB used, 893 GiB / 894 GiB avail; 5.2 MiB/s rd, 2.6 MiB/s wr, 8 op/s; 1.7 MiB/s, 2 objects/s recovering 2024-09-11T03:40:08.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:07 smithi121 ceph-mon[30729]: 1.1 scrub starts 2024-09-11T03:40:08.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1103830829' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:40:08.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:07 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:08.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1103830829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:40:08.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:07 smithi121 ceph-mon[30729]: osdmap e191: 12 total, 12 up, 10 in 2024-09-11T03:40:08.362 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:08.384 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:08.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:08.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:08.486 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:08.508 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:08.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:08.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:08.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:08.631 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:08.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:08.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:08.733 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:08.758 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:08.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:08.759 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:08.837 INFO:tasks.rados.rados.2.smithi175.stdout:474: done (1 left) 2024-09-11T03:40:08.837 INFO:tasks.rados.rados.2.smithi175.stdout:475: delete oid 30 current snap is 48 2024-09-11T03:40:08.837 INFO:tasks.rados.rados.2.smithi175.stdout:473: expect (ObjNum 134 snap 22 seq_num 134) 2024-09-11T03:40:08.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:08.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:08.882 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:08.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:08.923 INFO:tasks.rados.rados.2.smithi175.stdout:473: done (1 left) 2024-09-11T03:40:08.923 INFO:tasks.rados.rados.2.smithi175.stdout:475: done (0 left) 2024-09-11T03:40:08.923 INFO:tasks.rados.rados.2.smithi175.stdout:476: copy_from oid 21 from oid 19 current snap is 48 2024-09-11T03:40:08.924 INFO:tasks.rados.rados.2.smithi175.stdout:477: read oid 38 snap 46 2024-09-11T03:40:08.924 INFO:tasks.rados.rados.2.smithi175.stdout:477: expect (ObjNum 101 snap 16 seq_num 101) 2024-09-11T03:40:08.924 INFO:tasks.rados.rados.2.smithi175.stdout:478: read oid 7 snap 46 2024-09-11T03:40:08.924 INFO:tasks.rados.rados.2.smithi175.stdout:478: expect (ObjNum 192 snap 46 seq_num 192) 2024-09-11T03:40:08.924 INFO:tasks.rados.rados.2.smithi175.stdout:479: copy_from oid 20 from oid 48 current snap is 48 2024-09-11T03:40:08.924 INFO:tasks.rados.rados.2.smithi175.stdout:480: write (excl) oid 36 current snap is 48 2024-09-11T03:40:08.924 INFO:tasks.rados.rados.2.smithi175.stdout:480: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-09-11T03:40:08.928 INFO:tasks.rados.rados.2.smithi175.stdout:480: writing smithi17557870-36 from 561400 to 1036378 tid 1 2024-09-11T03:40:08.936 INFO:tasks.rados.rados.2.smithi175.stdout:480: writing smithi17557870-36 from 1820281 to 2340004 tid 2 2024-09-11T03:40:08.941 INFO:tasks.rados.rados.2.smithi175.stdout:480: writing smithi17557870-36 from 2859422 to 3317001 tid 3 2024-09-11T03:40:08.941 INFO:tasks.rados.rados.2.smithi175.stdout:481: rollback oid 24 current snap is 48 2024-09-11T03:40:08.942 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 46 2024-09-11T03:40:08.942 INFO:tasks.rados.rados.2.smithi175.stdout:477: expect (ObjNum 101 snap 16 seq_num 101) 2024-09-11T03:40:08.983 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:09.004 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:09.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:09.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:09.102 INFO:tasks.rados.rados.2.smithi175.stdout:479: finishing copy_from racing read to smithi17557870-20 2024-09-11T03:40:09.102 INFO:tasks.rados.rados.2.smithi175.stdout:476: finishing copy_from racing read to smithi17557870-21 2024-09-11T03:40:09.102 INFO:tasks.rados.rados.2.smithi175.stdout:480: finishing write tid 1 to smithi17557870-36 2024-09-11T03:40:09.102 INFO:tasks.rados.rados.2.smithi175.stdout:479: finishing copy_from to smithi17557870-20 2024-09-11T03:40:09.103 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 82 (ObjNum 171 snap 39 seq_num 171) dirty exists 2024-09-11T03:40:09.103 INFO:tasks.rados.rados.2.smithi175.stdout:476: finishing copy_from to smithi17557870-21 2024-09-11T03:40:09.103 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 97 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T03:40:09.103 INFO:tasks.rados.rados.2.smithi175.stdout:478: expect (ObjNum 192 snap 46 seq_num 192) 2024-09-11T03:40:09.106 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:09.127 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:09.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:09.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:09.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T03:40:09.190 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:09.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:09.250 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:09.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:09.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:09.256 INFO:tasks.rados.rados.2.smithi175.stdout:480: finishing write tid 2 to smithi17557870-36 2024-09-11T03:40:09.256 INFO:tasks.rados.rados.2.smithi175.stdout:481: finishing rollback tid 0 to smithi17557870-24 2024-09-11T03:40:09.257 INFO:tasks.rados.rados.2.smithi175.stdout:481: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:40:09.257 INFO:tasks.rados.rados.2.smithi175.stdout:481: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:40:09.257 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 128 (ObjNum 177 snap 42 seq_num 177) dirty exists 2024-09-11T03:40:09.257 INFO:tasks.rados.rados.2.smithi175.stdout:480: finishing write tid 3 to smithi17557870-36 2024-09-11T03:40:09.257 INFO:tasks.rados.rados.2.smithi175.stdout:480: finishing write tid 4 to smithi17557870-36 2024-09-11T03:40:09.257 INFO:tasks.rados.rados.2.smithi175.stdout:476: done (5 left) 2024-09-11T03:40:09.257 INFO:tasks.rados.rados.2.smithi175.stdout:477: done (4 left) 2024-09-11T03:40:09.257 INFO:tasks.rados.rados.2.smithi175.stdout:478: done (3 left) 2024-09-11T03:40:09.257 INFO:tasks.rados.rados.2.smithi175.stdout:479: done (2 left) 2024-09-11T03:40:09.257 INFO:tasks.rados.rados.2.smithi175.stdout:481: done (1 left) 2024-09-11T03:40:09.257 INFO:tasks.rados.rados.2.smithi175.stdout:482: delete oid 17 current snap is 48 2024-09-11T03:40:09.257 INFO:tasks.rados.rados.2.smithi175.stdout:480: finishing write tid 5 to smithi17557870-36 2024-09-11T03:40:09.257 INFO:tasks.rados.rados.2.smithi175.stdout:480: oid 36 updating version 0 to 107 2024-09-11T03:40:09.258 INFO:tasks.rados.rados.2.smithi175.stdout:480: oid 36 updating version 107 to 109 2024-09-11T03:40:09.258 INFO:tasks.rados.rados.2.smithi175.stdout:480: oid 36 version 109 is already newer than 108 2024-09-11T03:40:09.258 INFO:tasks.rados.rados.2.smithi175.stdout:480: oid 36 version 109 is already newer than 106 2024-09-11T03:40:09.258 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 109 (ObjNum 200 snap 48 seq_num 200) dirty exists 2024-09-11T03:40:09.258 INFO:tasks.rados.rados.2.smithi175.stdout:480: left oid 36 (ObjNum 200 snap 48 seq_num 200) 2024-09-11T03:40:09.258 INFO:tasks.rados.rados.2.smithi175.stdout:480: done (1 left) 2024-09-11T03:40:09.258 INFO:tasks.rados.rados.2.smithi175.stdout:482: done (0 left) 2024-09-11T03:40:09.258 INFO:tasks.rados.rados.2.smithi175.stdout:483: copy_from oid 10 from oid 30 current snap is 48 2024-09-11T03:40:09.259 INFO:tasks.rados.rados.2.smithi175.stdout:484: snap_create 2024-09-11T03:40:09.259 INFO:tasks.rados.rados.2.smithi175.stdout:483: finishing copy_from to smithi17557870-10 2024-09-11T03:40:09.259 INFO:tasks.rados.rados.2.smithi175.stdout:483: got expected ENOENT (src dne) 2024-09-11T03:40:09.259 INFO:tasks.rados.rados.2.smithi175.stdout:483: finishing copy_from racing read to smithi17557870-10 2024-09-11T03:40:09.352 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:09.373 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:09.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:09.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:09.475 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:09.496 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:09.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:09.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:09.598 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:09.620 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:09.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:09.621 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:09.722 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:09.744 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:09.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:09.745 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:09.758 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:40:09.758 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:40:09.773 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:40:09.774 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:40:09.792 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:40:09.792 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:40:09.792 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:09.809 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:40:09.809 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:40:09.827 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:40:09.827 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:40:09.843 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:40:09.844 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:40:09.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:09.860 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:40:09.861 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:40:09.867 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:09.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:09.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:09.877 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:40:09.877 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:40:09.894 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:40:09.894 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:40:09.911 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:40:09.911 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:40:09.928 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:40:09.928 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:40:09.945 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:40:09.945 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:40:09.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:09.975 INFO:tasks.rados.rados.2.smithi175.stdout:483: done (1 left) 2024-09-11T03:40:09.975 INFO:tasks.rados.rados.2.smithi175.stdout:484: done (0 left) 2024-09-11T03:40:09.976 INFO:tasks.rados.rados.2.smithi175.stdout:485: copy_from oid 32 from oid 5 current snap is 49 2024-09-11T03:40:09.976 INFO:tasks.rados.rados.2.smithi175.stdout:486: delete oid 25 current snap is 49 2024-09-11T03:40:09.977 INFO:tasks.rados.rados.2.smithi175.stdout:485: finishing copy_from to smithi17557870-32 2024-09-11T03:40:09.977 INFO:tasks.rados.rados.2.smithi175.stdout:485: got expected ENOENT (src dne) 2024-09-11T03:40:09.978 INFO:tasks.rados.rados.2.smithi175.stdout:485: finishing copy_from racing read to smithi17557870-32 2024-09-11T03:40:09.978 INFO:tasks.rados.rados.2.smithi175.stdout:485: done (1 left) 2024-09-11T03:40:09.978 INFO:tasks.rados.rados.2.smithi175.stdout:486: done (0 left) 2024-09-11T03:40:09.978 INFO:tasks.rados.rados.2.smithi175.stdout:487: snap_remove snap 45 2024-09-11T03:40:09.989 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:09.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:09.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:10.033 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:09 smithi084 ceph-mon[27254]: pgmap v442: 27 pgs: 1 peering, 26 active+clean; 135 MiB data, 555 MiB used, 893 GiB / 894 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 10 op/s; 2.3 MiB/s, 6 objects/s recovering 2024-09-11T03:40:10.034 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:09 smithi084 ceph-mon[27254]: osdmap e192: 12 total, 12 up, 10 in 2024-09-11T03:40:10.034 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:09 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2905116873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:10.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:10.113 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:10.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:10.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:10.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:09 smithi121 ceph-mon[30729]: pgmap v442: 27 pgs: 1 peering, 26 active+clean; 135 MiB data, 555 MiB used, 893 GiB / 894 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 10 op/s; 2.3 MiB/s, 6 objects/s recovering 2024-09-11T03:40:10.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:09 smithi121 ceph-mon[30729]: osdmap e192: 12 total, 12 up, 10 in 2024-09-11T03:40:10.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:09 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2905116873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:10.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:10.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:09 smithi175 ceph-mon[30894]: pgmap v442: 27 pgs: 1 peering, 26 active+clean; 135 MiB data, 555 MiB used, 893 GiB / 894 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 10 op/s; 2.3 MiB/s, 6 objects/s recovering 2024-09-11T03:40:10.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:09 smithi175 ceph-mon[30894]: osdmap e192: 12 total, 12 up, 10 in 2024-09-11T03:40:10.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:09 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2905116873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:10.236 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:10.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:10.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:10.338 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:10.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:10.361 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:10.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:10.462 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:10.484 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:10.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:10.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:10.586 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:10.608 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:10.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:10.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:10.688 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:40:10.709 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:10.733 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:10.738 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:10.839 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:10.860 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:10.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:10.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:10.932 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:40:10.932 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:40:10.933 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:40:10.933 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:40:10.962 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:10.983 INFO:tasks.rados.rados.2.smithi175.stdout:487: done (0 left) 2024-09-11T03:40:10.983 INFO:tasks.rados.rados.2.smithi175.stdout:488: write (excl) oid 2 current snap is 49 2024-09-11T03:40:10.983 INFO:tasks.rados.rados.2.smithi175.stdout:488: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2024-09-11T03:40:10.985 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:10.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:10.986 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:10.991 INFO:tasks.rados.rados.2.smithi175.stdout:488: writing smithi17557870-2 from 611821 to 1312546 tid 1 2024-09-11T03:40:11.000 INFO:tasks.rados.rados.2.smithi175.stdout:488: writing smithi17557870-2 from 1879374 to 2662777 tid 2 2024-09-11T03:40:11.005 INFO:tasks.rados.rados.2.smithi175.stdout:488: writing smithi17557870-2 from 3341691 to 3411821 tid 3 2024-09-11T03:40:11.005 INFO:tasks.rados.rados.2.smithi175.stdout:489: read oid 40 snap -1 2024-09-11T03:40:11.005 INFO:tasks.rados.rados.2.smithi175.stdout:489: expect deleted 2024-09-11T03:40:11.005 INFO:tasks.rados.rados.2.smithi175.stdout:490: write oid 15 current snap is 49 2024-09-11T03:40:11.005 INFO:tasks.rados.rados.2.smithi175.stdout:488: finishing write tid 1 to smithi17557870-2 2024-09-11T03:40:11.005 INFO:tasks.rados.rados.2.smithi175.stdout:488: finishing write tid 2 to smithi17557870-2 2024-09-11T03:40:11.005 INFO:tasks.rados.rados.2.smithi175.stdout:490: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2024-09-11T03:40:11.014 INFO:tasks.rados.rados.2.smithi175.stdout:490: writing smithi17557870-15 from 628628 to 1404602 tid 1 2024-09-11T03:40:11.021 INFO:tasks.rados.rados.2.smithi175.stdout:490: writing smithi17557870-15 from 1937856 to 2464917 tid 2 2024-09-11T03:40:11.027 INFO:tasks.rados.rados.2.smithi175.stdout:490: writing smithi17557870-15 from 3252761 to 3428628 tid 3 2024-09-11T03:40:11.027 INFO:tasks.rados.rados.2.smithi175.stdout:491: read oid 5 snap 48 2024-09-11T03:40:11.027 INFO:tasks.rados.rados.2.smithi175.stdout:491: expect deleted 2024-09-11T03:40:11.027 INFO:tasks.rados.rados.2.smithi175.stdout:492: read oid 37 snap -1 2024-09-11T03:40:11.028 INFO:tasks.rados.rados.2.smithi175.stdout:492: expect deleted 2024-09-11T03:40:11.028 INFO:tasks.rados.rados.2.smithi175.stdout:488: finishing write tid 3 to smithi17557870-2 2024-09-11T03:40:11.028 INFO:tasks.rados.rados.2.smithi175.stdout:488: finishing write tid 4 to smithi17557870-2 2024-09-11T03:40:11.028 INFO:tasks.rados.rados.2.smithi175.stdout:488: finishing write tid 5 to smithi17557870-2 2024-09-11T03:40:11.028 INFO:tasks.rados.rados.2.smithi175.stdout:488: oid 2 updating version 0 to 112 2024-09-11T03:40:11.028 INFO:tasks.rados.rados.2.smithi175.stdout:488: oid 2 version 112 is already newer than 111 2024-09-11T03:40:11.028 INFO:tasks.rados.rados.2.smithi175.stdout:488: oid 2 updating version 112 to 113 2024-09-11T03:40:11.028 INFO:tasks.rados.rados.2.smithi175.stdout:488: oid 2 version 113 is already newer than 110 2024-09-11T03:40:11.028 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 113 (ObjNum 203 snap 49 seq_num 203) dirty exists 2024-09-11T03:40:11.028 INFO:tasks.rados.rados.2.smithi175.stdout:488: left oid 2 (ObjNum 203 snap 49 seq_num 203) 2024-09-11T03:40:11.028 INFO:tasks.rados.rados.2.smithi175.stdout:493: read oid 45 snap -1 2024-09-11T03:40:11.029 INFO:tasks.rados.rados.2.smithi175.stdout:493: expect deleted 2024-09-11T03:40:11.029 INFO:tasks.rados.rados.2.smithi175.stdout:488: done (5 left) 2024-09-11T03:40:11.029 INFO:tasks.rados.rados.2.smithi175.stdout:494: snap_create 2024-09-11T03:40:11.029 INFO:tasks.rados.rados.2.smithi175.stdout:490: finishing write tid 1 to smithi17557870-15 2024-09-11T03:40:11.029 INFO:tasks.rados.rados.2.smithi175.stdout:490: finishing write tid 2 to smithi17557870-15 2024-09-11T03:40:11.030 INFO:tasks.rados.rados.2.smithi175.stdout:490: finishing write tid 3 to smithi17557870-15 2024-09-11T03:40:11.030 INFO:tasks.rados.rados.2.smithi175.stdout:490: finishing write tid 4 to smithi17557870-15 2024-09-11T03:40:11.030 INFO:tasks.rados.rados.2.smithi175.stdout:490: finishing write tid 5 to smithi17557870-15 2024-09-11T03:40:11.030 INFO:tasks.rados.rados.2.smithi175.stdout:490: oid 15 updating version 0 to 80 2024-09-11T03:40:11.030 INFO:tasks.rados.rados.2.smithi175.stdout:490: oid 15 updating version 80 to 82 2024-09-11T03:40:11.030 INFO:tasks.rados.rados.2.smithi175.stdout:490: oid 15 version 82 is already newer than 81 2024-09-11T03:40:11.030 INFO:tasks.rados.rados.2.smithi175.stdout:490: oid 15 updating version 82 to 83 2024-09-11T03:40:11.030 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 83 (ObjNum 204 snap 49 seq_num 204) dirty exists 2024-09-11T03:40:11.031 INFO:tasks.rados.rados.2.smithi175.stdout:490: left oid 15 (ObjNum 204 snap 49 seq_num 204) 2024-09-11T03:40:11.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:11.113 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:11.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:11.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:11.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:10 smithi084 ceph-mon[27254]: osdmap e193: 12 total, 12 up, 10 in 2024-09-11T03:40:11.215 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:11.236 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:11.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:11.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:11.238 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:10 smithi175 ceph-mon[30894]: osdmap e193: 12 total, 12 up, 10 in 2024-09-11T03:40:11.338 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:11.363 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:11.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:11.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:10 smithi121 ceph-mon[30729]: osdmap e193: 12 total, 12 up, 10 in 2024-09-11T03:40:11.465 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:11.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:11.487 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:11.487 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:11.566 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T03:40:11.589 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:11.609 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:11.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:11.610 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:11.665 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:11.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:11.722 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [6, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:40:11.722 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:40:11.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:40:11.737 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:11.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:11.831 INFO:tasks.rados.rados.2.smithi175.stdout:489: done (5 left) 2024-09-11T03:40:11.832 INFO:tasks.rados.rados.2.smithi175.stdout:490: done (4 left) 2024-09-11T03:40:11.832 INFO:tasks.rados.rados.2.smithi175.stdout:491: done (3 left) 2024-09-11T03:40:11.832 INFO:tasks.rados.rados.2.smithi175.stdout:492: done (2 left) 2024-09-11T03:40:11.832 INFO:tasks.rados.rados.2.smithi175.stdout:493: done (1 left) 2024-09-11T03:40:11.832 INFO:tasks.rados.rados.2.smithi175.stdout:494: done (0 left) 2024-09-11T03:40:11.832 INFO:tasks.rados.rados.2.smithi175.stdout:495: rollback oid 4 current snap is 50 2024-09-11T03:40:11.832 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 4 to 42 2024-09-11T03:40:11.832 INFO:tasks.rados.rados.2.smithi175.stdout:496: delete oid 23 current snap is 50 2024-09-11T03:40:11.832 INFO:tasks.rados.rados.2.smithi175.stdout:495: finishing rollback tid 1 to smithi17557870-4 2024-09-11T03:40:11.833 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 84 (ObjNum 33 snap 0 seq_num 1713071920) dirty dne 2024-09-11T03:40:11.835 INFO:tasks.rados.rados.2.smithi175.stdout:495: done (1 left) 2024-09-11T03:40:11.835 INFO:tasks.rados.rados.2.smithi175.stdout:496: done (0 left) 2024-09-11T03:40:11.835 INFO:tasks.rados.rados.2.smithi175.stdout:497: snap_remove snap 49 2024-09-11T03:40:11.839 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:11.859 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:11.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:11.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:11.961 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:11.982 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:11.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:11.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:12.073 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:11 smithi084 ceph-mon[27254]: pgmap v445: 27 pgs: 1 peering, 26 active+clean; 135 MiB data, 563 MiB used, 893 GiB / 894 GiB avail; 1.5 KiB/s rd, 722 KiB/s wr, 5 op/s; 601 KiB/s, 3 objects/s recovering 2024-09-11T03:40:12.074 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:11 smithi084 ceph-mon[27254]: osdmap e194: 12 total, 12 up, 10 in 2024-09-11T03:40:12.074 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:11 smithi084 ceph-mon[27254]: osdmap e195: 12 total, 12 up, 10 in 2024-09-11T03:40:12.084 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:12.105 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:12.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:12.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:12.207 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:12.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:12.232 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:12.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:12.245 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:12.251 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:12.333 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:12.354 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:12.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:12.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:12.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:11 smithi121 ceph-mon[30729]: pgmap v445: 27 pgs: 1 peering, 26 active+clean; 135 MiB data, 563 MiB used, 893 GiB / 894 GiB avail; 1.5 KiB/s rd, 722 KiB/s wr, 5 op/s; 601 KiB/s, 3 objects/s recovering 2024-09-11T03:40:12.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:11 smithi121 ceph-mon[30729]: osdmap e194: 12 total, 12 up, 10 in 2024-09-11T03:40:12.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:11 smithi121 ceph-mon[30729]: osdmap e195: 12 total, 12 up, 10 in 2024-09-11T03:40:12.456 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:11 smithi175 ceph-mon[30894]: pgmap v445: 27 pgs: 1 peering, 26 active+clean; 135 MiB data, 563 MiB used, 893 GiB / 894 GiB avail; 1.5 KiB/s rd, 722 KiB/s wr, 5 op/s; 601 KiB/s, 3 objects/s recovering 2024-09-11T03:40:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:11 smithi175 ceph-mon[30894]: osdmap e194: 12 total, 12 up, 10 in 2024-09-11T03:40:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:11 smithi175 ceph-mon[30894]: osdmap e195: 12 total, 12 up, 10 in 2024-09-11T03:40:12.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:12.478 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:12.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:12.581 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:12.606 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:12.607 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:12.708 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:12.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:40:12.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:12.729 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:12.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:12.830 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:12.831 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:40:12.835 INFO:tasks.rados.rados.2.smithi175.stdout:497: done (0 left) 2024-09-11T03:40:12.835 INFO:tasks.rados.rados.2.smithi175.stdout:498: rollback oid 31 current snap is 50 2024-09-11T03:40:12.835 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 31 to 42 2024-09-11T03:40:12.835 INFO:tasks.rados.rados.2.smithi175.stdout:499: rollback oid 14 current snap is 50 2024-09-11T03:40:12.835 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 14 to 42 2024-09-11T03:40:12.835 INFO:tasks.rados.rados.2.smithi175.stdout:500: rollback oid 35 current snap is 50 2024-09-11T03:40:12.835 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 35 to 41 2024-09-11T03:40:12.835 INFO:tasks.rados.rados.2.smithi175.stdout:501: read oid 27 snap -1 2024-09-11T03:40:12.835 INFO:tasks.rados.rados.2.smithi175.stdout:501: expect (ObjNum 186 snap 45 seq_num 186) 2024-09-11T03:40:12.835 INFO:tasks.rados.rados.2.smithi175.stdout:502: copy_from oid 5 from oid 13 current snap is 50 2024-09-11T03:40:12.835 INFO:tasks.rados.rados.2.smithi175.stdout:503: rollback oid 16 current snap is 50 2024-09-11T03:40:12.836 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 16 to 48 2024-09-11T03:40:12.836 INFO:tasks.rados.rados.2.smithi175.stdout:504: copy_from oid 20 from oid 34 current snap is 50 2024-09-11T03:40:12.836 INFO:tasks.rados.rados.2.smithi175.stdout:505: snap_remove snap 46 2024-09-11T03:40:12.860 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:12.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:12.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:12.962 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:12.987 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:12.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:12.988 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:13.089 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:13.111 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:13.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:13.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:13.189 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:40:13.189 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:40:13.189 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:40:13.189 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:40:13.189 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:40:13.213 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:13.239 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:13.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:13.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:13.341 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:13.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:13.363 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:13.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:13.393 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:12 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3816905538' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:40:13.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:12 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:13.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:12 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3816905538' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:40:13.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:12 smithi084 ceph-mon[27254]: osdmap e196: 12 total, 12 up, 10 in 2024-09-11T03:40:13.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:12 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3816905538' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:40:13.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:12 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:13.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:12 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3816905538' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:40:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:12 smithi121 ceph-mon[30729]: osdmap e196: 12 total, 12 up, 10 in 2024-09-11T03:40:13.464 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:12 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3816905538' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:40:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:12 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:12 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3816905538' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:40:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:12 smithi175 ceph-mon[30894]: osdmap e196: 12 total, 12 up, 10 in 2024-09-11T03:40:13.489 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:13.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:13.490 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:13.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:13.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:13.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:13.719 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:13.744 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:13.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:13.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:13.769 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T03:40:13.836 INFO:tasks.rados.rados.2.smithi175.stdout:505: done (7 left) 2024-09-11T03:40:13.836 INFO:tasks.rados.rados.2.smithi175.stdout:506: delete oid 19 current snap is 50 2024-09-11T03:40:13.836 INFO:tasks.rados.rados.2.smithi175.stdout:499: finishing rollback tid 1 to smithi17557870-14 2024-09-11T03:40:13.836 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 0 (ObjNum 593 snap 0 seq_num 0) dirty dne 2024-09-11T03:40:13.837 INFO:tasks.rados.rados.2.smithi175.stdout:503: finishing rollback tid 0 to smithi17557870-16 2024-09-11T03:40:13.837 INFO:tasks.rados.rados.2.smithi175.stdout:503: finishing rollback tid 1 to smithi17557870-16 2024-09-11T03:40:13.837 INFO:tasks.rados.rados.2.smithi175.stdout:503: finishing rollback tid 2 to smithi17557870-16 2024-09-11T03:40:13.837 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 101 (ObjNum 153 snap 35 seq_num 153) dirty exists 2024-09-11T03:40:13.837 INFO:tasks.rados.rados.2.smithi175.stdout:500: finishing rollback tid 0 to smithi17557870-35 2024-09-11T03:40:13.837 INFO:tasks.rados.rados.2.smithi175.stdout:500: finishing rollback tid 1 to smithi17557870-35 2024-09-11T03:40:13.837 INFO:tasks.rados.rados.2.smithi175.stdout:500: finishing rollback tid 2 to smithi17557870-35 2024-09-11T03:40:13.837 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 134 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T03:40:13.837 INFO:tasks.rados.rados.2.smithi175.stdout:501: expect (ObjNum 186 snap 45 seq_num 186) 2024-09-11T03:40:13.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:13.873 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:13.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:13.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:13.937 INFO:tasks.rados.rados.2.smithi175.stdout:498: finishing rollback tid 1 to smithi17557870-31 2024-09-11T03:40:13.937 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-11T03:40:13.937 INFO:tasks.rados.rados.2.smithi175.stdout:502: finishing copy_from racing read to smithi17557870-5 2024-09-11T03:40:13.937 INFO:tasks.rados.rados.2.smithi175.stdout:504: finishing copy_from racing read to smithi17557870-20 2024-09-11T03:40:13.937 INFO:tasks.rados.rados.2.smithi175.stdout:502: finishing copy_from to smithi17557870-5 2024-09-11T03:40:13.937 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 153 (ObjNum 189 snap 45 seq_num 189) dirty exists 2024-09-11T03:40:13.937 INFO:tasks.rados.rados.2.smithi175.stdout:504: finishing copy_from to smithi17557870-20 2024-09-11T03:40:13.938 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 48 (ObjNum 119 snap 18 seq_num 119) dirty exists 2024-09-11T03:40:13.938 INFO:tasks.rados.rados.2.smithi175.stdout:498: done (7 left) 2024-09-11T03:40:13.938 INFO:tasks.rados.rados.2.smithi175.stdout:499: done (6 left) 2024-09-11T03:40:13.938 INFO:tasks.rados.rados.2.smithi175.stdout:500: done (5 left) 2024-09-11T03:40:13.938 INFO:tasks.rados.rados.2.smithi175.stdout:501: done (4 left) 2024-09-11T03:40:13.938 INFO:tasks.rados.rados.2.smithi175.stdout:502: done (3 left) 2024-09-11T03:40:13.938 INFO:tasks.rados.rados.2.smithi175.stdout:503: done (2 left) 2024-09-11T03:40:13.938 INFO:tasks.rados.rados.2.smithi175.stdout:504: done (1 left) 2024-09-11T03:40:13.938 INFO:tasks.rados.rados.2.smithi175.stdout:506: done (0 left) 2024-09-11T03:40:13.938 INFO:tasks.rados.rados.2.smithi175.stdout:507: copy_from oid 42 from oid 33 current snap is 50 2024-09-11T03:40:13.938 INFO:tasks.rados.rados.2.smithi175.stdout:508: snap_remove snap 42 2024-09-11T03:40:13.975 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:13.997 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:13.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:13.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:14.099 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:14.120 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:14.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:14.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:14.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:14.248 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:14.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:14.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:14.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:13 smithi084 ceph-mon[27254]: pgmap v448: 27 pgs: 1 peering, 26 active+clean; 131 MiB data, 560 MiB used, 893 GiB / 894 GiB avail; 8.2 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-09-11T03:40:14.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:13 smithi084 ceph-mon[27254]: osdmap e197: 12 total, 12 up, 10 in 2024-09-11T03:40:14.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:14.371 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:14.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:14.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:14.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:13 smithi121 ceph-mon[30729]: pgmap v448: 27 pgs: 1 peering, 26 active+clean; 131 MiB data, 560 MiB used, 893 GiB / 894 GiB avail; 8.2 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-09-11T03:40:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:13 smithi121 ceph-mon[30729]: osdmap e197: 12 total, 12 up, 10 in 2024-09-11T03:40:14.443 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:14.453 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:14.474 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:13 smithi175 ceph-mon[30894]: pgmap v448: 27 pgs: 1 peering, 26 active+clean; 131 MiB data, 560 MiB used, 893 GiB / 894 GiB avail; 8.2 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-09-11T03:40:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:13 smithi175 ceph-mon[30894]: osdmap e197: 12 total, 12 up, 10 in 2024-09-11T03:40:14.500 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:14.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:14.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:14.601 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:14.609 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:40:14.609 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:40:14.623 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:14.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:14.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:14.725 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:14.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:14.747 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:14.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:14.840 INFO:tasks.rados.rados.2.smithi175.stdout:508: done (1 left) 2024-09-11T03:40:14.840 INFO:tasks.rados.rados.2.smithi175.stdout:509: write oid 32 current snap is 50 2024-09-11T03:40:14.840 INFO:tasks.rados.rados.2.smithi175.stdout:509: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2024-09-11T03:40:14.847 INFO:tasks.rados.rados.2.smithi175.stdout:509: writing smithi17557870-32 from 679049 to 1397123 tid 1 2024-09-11T03:40:14.848 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:14.855 INFO:tasks.rados.rados.2.smithi175.stdout:509: writing smithi17557870-32 from 2113302 to 2787690 tid 2 2024-09-11T03:40:14.860 INFO:tasks.rados.rados.2.smithi175.stdout:509: writing smithi17557870-32 from 3335030 to 3479049 tid 3 2024-09-11T03:40:14.860 INFO:tasks.rados.rados.2.smithi175.stdout:510: read oid 18 snap -1 2024-09-11T03:40:14.860 INFO:tasks.rados.rados.2.smithi175.stdout:510: expect (ObjNum 115 snap 18 seq_num 115) 2024-09-11T03:40:14.860 INFO:tasks.rados.rados.2.smithi175.stdout:511: copy_from oid 5 from oid 25 current snap is 50 2024-09-11T03:40:14.860 INFO:tasks.rados.rados.2.smithi175.stdout:507: finishing copy_from to smithi17557870-42 2024-09-11T03:40:14.860 INFO:tasks.rados.rados.2.smithi175.stdout:507: got expected ENOENT (src dne) 2024-09-11T03:40:14.860 INFO:tasks.rados.rados.2.smithi175.stdout:507: finishing copy_from racing read to smithi17557870-42 2024-09-11T03:40:14.860 INFO:tasks.rados.rados.2.smithi175.stdout:509: finishing write tid 1 to smithi17557870-32 2024-09-11T03:40:14.861 INFO:tasks.rados.rados.2.smithi175.stdout:509: finishing write tid 2 to smithi17557870-32 2024-09-11T03:40:14.861 INFO:tasks.rados.rados.2.smithi175.stdout:507: done (3 left) 2024-09-11T03:40:14.861 INFO:tasks.rados.rados.2.smithi175.stdout:512: copy_from oid 42 from oid 11 current snap is 50 2024-09-11T03:40:14.861 INFO:tasks.rados.rados.2.smithi175.stdout:513: snap_create 2024-09-11T03:40:14.862 INFO:tasks.rados.rados.2.smithi175.stdout:509: finishing write tid 3 to smithi17557870-32 2024-09-11T03:40:14.862 INFO:tasks.rados.rados.2.smithi175.stdout:509: finishing write tid 4 to smithi17557870-32 2024-09-11T03:40:14.862 INFO:tasks.rados.rados.2.smithi175.stdout:509: finishing write tid 5 to smithi17557870-32 2024-09-11T03:40:14.862 INFO:tasks.rados.rados.2.smithi175.stdout:509: oid 32 updating version 0 to 113 2024-09-11T03:40:14.862 INFO:tasks.rados.rados.2.smithi175.stdout:509: oid 32 version 113 is already newer than 112 2024-09-11T03:40:14.862 INFO:tasks.rados.rados.2.smithi175.stdout:509: oid 32 version 113 is already newer than 110 2024-09-11T03:40:14.862 INFO:tasks.rados.rados.2.smithi175.stdout:509: oid 32 version 113 is already newer than 111 2024-09-11T03:40:14.862 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 113 (ObjNum 207 snap 50 seq_num 207) dirty exists 2024-09-11T03:40:14.862 INFO:tasks.rados.rados.2.smithi175.stdout:509: left oid 32 (ObjNum 207 snap 50 seq_num 207) 2024-09-11T03:40:14.863 INFO:tasks.rados.rados.2.smithi175.stdout:511: finishing copy_from to smithi17557870-5 2024-09-11T03:40:14.863 INFO:tasks.rados.rados.2.smithi175.stdout:511: got expected ENOENT (src dne) 2024-09-11T03:40:14.863 INFO:tasks.rados.rados.2.smithi175.stdout:511: finishing copy_from racing read to smithi17557870-5 2024-09-11T03:40:14.863 INFO:tasks.rados.rados.2.smithi175.stdout:512: finishing copy_from to smithi17557870-42 2024-09-11T03:40:14.863 INFO:tasks.rados.rados.2.smithi175.stdout:512: got expected ENOENT (src dne) 2024-09-11T03:40:14.864 INFO:tasks.rados.rados.2.smithi175.stdout:512: finishing copy_from racing read to smithi17557870-42 2024-09-11T03:40:14.872 INFO:tasks.rados.rados.2.smithi175.stdout:510: expect (ObjNum 115 snap 18 seq_num 115) 2024-09-11T03:40:14.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:14.875 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:14.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:14.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:14.999 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:14.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:15.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:15.101 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:15.122 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:15.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:15.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:15.247 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:15.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:15.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:15.286 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:15 smithi084 ceph-mon[27254]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:40:15.286 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:15 smithi084 ceph-mon[27254]: osdmap e198: 12 total, 12 up, 10 in 2024-09-11T03:40:15.314 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-09-11T03:40:13.833510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149194056,"num_objects":72,"num_object_clones":27,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":8,"num_read":930,"num_read_kb":675160,"num_write":852,"num_write_kb":324398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14297451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2569,"ondisk_log_size":2569,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":586424,"kb_used_data":278664,"kb_used_omap":15,"kb_used_meta":307504,"kb_avail":936865096,"statfs":{"total":959950356480,"available":959349858304,"internally_reserved":0,"allocated":285351936,"data_stored":306678910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":314884533},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8117833,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":51,"num_read_kb":47043,"num_write":32,"num_write_kb":15886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.859267"},"pg_stats":[{"pgid":"2.f","version":"191'96","reported_seq":592,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:11.012138+0000","last_change":"2024-09-11T03:40:11.012333+0000","last_active":"2024-09-11T03:40:11.012138+0000","last_peered":"2024-09-11T03:40:11.012138+0000","last_clean":"2024-09-11T03:40:11.012138+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:40:11.012138+0000","last_undegraded":"2024-09-11T03:40:11.012138+0000","last_fullsized":"2024-09-11T03:40:11.012138+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'82","last_scrub_stamp":"2024-09-11T03:39:53.261846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:53.261846+0000","objects_scrubbed":4,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:09.338245+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00069781900000000002,"stat_sum":{"num_bytes":7815255,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":30,"num_read_kb":6170,"num_write":50,"num_write_kb":18140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"26","length":"1"},{"start":"28","length":"1"},{"start":"2e","length":"2"},{"start":"31","length":"1"}]},{"pgid":"2.e","version":"196'133","reported_seq":648,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-11T03:40:12.841102+0000","last_change":"2024-09-11T03:40:12.841953+0000","last_active":"2024-09-11T03:40:12.841102+0000","last_peered":"2024-09-11T03:40:12.841102+0000","last_clean":"2024-09-11T03:40:12.841102+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:40:12.841102+0000","last_undegraded":"2024-09-11T03:40:12.841102+0000","last_fullsized":"2024-09-11T03:40:12.841102+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'103","last_scrub_stamp":"2024-09-11T03:39:55.201092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:55.201092+0000","objects_scrubbed":2,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:19.600192+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0044149599999999999,"stat_sum":{"num_bytes":10860789,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":96,"num_read_kb":83452,"num_write":46,"num_write_kb":21913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"28","length":"1"},{"start":"2e","length":"2"},{"start":"31","length":"1"},{"start":"33","length":"1"}]},{"pgid":"2.d","version":"196'156","reported_seq":665,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-11T03:40:12.853090+0000","last_change":"2024-09-11T03:40:12.840056+0000","last_active":"2024-09-11T03:40:12.853090+0000","last_peered":"2024-09-11T03:40:12.853090+0000","last_clean":"2024-09-11T03:40:12.853090+0000","last_became_active":"2024-09-11T03:38:30.031497+0000","last_became_peered":"2024-09-11T03:38:30.031497+0000","last_unstale":"2024-09-11T03:40:12.853090+0000","last_undegraded":"2024-09-11T03:40:12.853090+0000","last_fullsized":"2024-09-11T03:40:12.853090+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'138","last_scrub_stamp":"2024-09-11T03:39:52.253596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:52.253596+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:20.228873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0013513329999999999,"stat_sum":{"num_bytes":13141506,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":83,"num_read_kb":60635,"num_write":69,"num_write_kb":24790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"28","length":"1"},{"start":"2e","length":"2"},{"start":"31","length":"1"},{"start":"33","length":"1"}]},{"pgid":"2.c","version":"194'86","reported_seq":600,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:11.030933+0000","last_change":"2024-09-11T03:40:11.031083+0000","last_active":"2024-09-11T03:40:11.030933+0000","last_peered":"2024-09-11T03:40:11.030933+0000","last_clean":"2024-09-11T03:40:11.030933+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:40:11.030933+0000","last_undegraded":"2024-09-11T03:40:11.030933+0000","last_fullsized":"2024-09-11T03:40:11.030933+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'52","last_scrub_stamp":"2024-09-11T03:39:56.458534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:56.458534+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:08.179504+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001732953,"stat_sum":{"num_bytes":11874975,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":91,"num_read_kb":79607,"num_write":54,"num_write_kb":25522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"26","length":"1"},{"start":"28","length":"1"},{"start":"2e","length":"2"},{"start":"31","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":365,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:10.985863+0000","last_change":"2024-09-11T03:39:58.567923+0000","last_active":"2024-09-11T03:40:10.985863+0000","last_peered":"2024-09-11T03:40:10.985863+0000","last_clean":"2024-09-11T03:40:10.985863+0000","last_became_active":"2024-09-11T03:39:57.836928+0000","last_became_peered":"2024-09-11T03:39:57.836928+0000","last_unstale":"2024-09-11T03:40:10.985863+0000","last_undegraded":"2024-09-11T03:40:10.985863+0000","last_fullsized":"2024-09-11T03:40:10.985863+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:39:58.567842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:39:58.567842+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:05.438976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"191'68","reported_seq":565,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:11.007981+0000","last_change":"2024-09-11T03:40:11.008172+0000","last_active":"2024-09-11T03:40:11.007981+0000","last_peered":"2024-09-11T03:40:11.007981+0000","last_clean":"2024-09-11T03:40:11.007981+0000","last_became_active":"2024-09-11T03:38:29.889091+0000","last_became_peered":"2024-09-11T03:38:29.889091+0000","last_unstale":"2024-09-11T03:40:11.007981+0000","last_undegraded":"2024-09-11T03:40:11.007981+0000","last_fullsized":"2024-09-11T03:40:11.007981+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'44","last_scrub_stamp":"2024-09-11T03:39:32.965569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:32.965569+0000","objects_scrubbed":2,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:02.230308+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00045590000000000002,"stat_sum":{"num_bytes":5529503,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":30,"num_read_kb":15200,"num_write":33,"num_write_kb":12236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"26","length":"1"},{"start":"28","length":"1"},{"start":"2e","length":"2"},{"start":"31","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":42,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:11.009428+0000","last_change":"2024-09-11T03:40:00.262368+0000","last_active":"2024-09-11T03:40:11.009428+0000","last_peered":"2024-09-11T03:40:11.009428+0000","last_clean":"2024-09-11T03:40:11.009428+0000","last_became_active":"2024-09-11T03:39:59.833111+0000","last_became_peered":"2024-09-11T03:39:59.833111+0000","last_unstale":"2024-09-11T03:40:11.009428+0000","last_undegraded":"2024-09-11T03:40:11.009428+0000","last_fullsized":"2024-09-11T03:40:11.009428+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.262301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:28.856759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"193'25","reported_seq":527,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:10.986047+0000","last_change":"2024-09-11T03:40:10.986277+0000","last_active":"2024-09-11T03:40:10.986047+0000","last_peered":"2024-09-11T03:40:10.986047+0000","last_clean":"2024-09-11T03:40:10.986047+0000","last_became_active":"2024-09-11T03:40:05.966325+0000","last_became_peered":"2024-09-11T03:40:05.966325+0000","last_unstale":"2024-09-11T03:40:10.986047+0000","last_undegraded":"2024-09-11T03:40:10.986047+0000","last_fullsized":"2024-09-11T03:40:10.986047+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'14","last_scrub_stamp":"2024-09-11T03:39:33.465057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:33.465057+0000","objects_scrubbed":1,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:49.369950+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00060567599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":5533,"num_write":15,"num_write_kb":3881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"26","length":"1"},{"start":"28","length":"1"},{"start":"2e","length":"2"},{"start":"31","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":371,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:10.985699+0000","last_change":"2024-09-11T03:40:01.496304+0000","last_active":"2024-09-11T03:40:10.985699+0000","last_peered":"2024-09-11T03:40:10.985699+0000","last_clean":"2024-09-11T03:40:10.985699+0000","last_became_active":"2024-09-11T03:40:00.099236+0000","last_became_peered":"2024-09-11T03:40:00.099236+0000","last_unstale":"2024-09-11T03:40:10.985699+0000","last_undegraded":"2024-09-11T03:40:10.985699+0000","last_fullsized":"2024-09-11T03:40:10.985699+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:01.496214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:01.496214+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:29.501697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"196'52","reported_seq":555,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-11T03:40:12.851732+0000","last_change":"2024-09-11T03:40:12.837971+0000","last_active":"2024-09-11T03:40:12.851732+0000","last_peered":"2024-09-11T03:40:12.851732+0000","last_clean":"2024-09-11T03:40:12.851732+0000","last_became_active":"2024-09-11T03:38:30.162952+0000","last_became_peered":"2024-09-11T03:38:30.162952+0000","last_unstale":"2024-09-11T03:40:12.851732+0000","last_undegraded":"2024-09-11T03:40:12.851732+0000","last_fullsized":"2024-09-11T03:40:12.851732+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'49","last_scrub_stamp":"2024-09-11T03:39:51.194245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:51.194245+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:01.220306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00067987600000000005,"stat_sum":{"num_bytes":3176523,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":19745,"num_write":25,"num_write_kb":10878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"28","length":"1"},{"start":"2e","length":"2"},{"start":"31","length":"1"},{"start":"33","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":270,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-11T03:40:12.837566+0000","last_change":"2024-09-11T03:40:07.837246+0000","last_active":"2024-09-11T03:40:12.837566+0000","last_peered":"2024-09-11T03:40:12.837566+0000","last_clean":"2024-09-11T03:40:12.837566+0000","last_became_active":"2024-09-11T03:40:05.966251+0000","last_became_peered":"2024-09-11T03:40:05.966251+0000","last_unstale":"2024-09-11T03:40:12.837566+0000","last_undegraded":"2024-09-11T03:40:12.837566+0000","last_fullsized":"2024-09-11T03:40:12.837566+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:37.216754+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":12,"num_write_kb":196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":144254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"194'65","reported_seq":582,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:11.009612+0000","last_change":"2024-09-11T03:40:11.009787+0000","last_active":"2024-09-11T03:40:11.009612+0000","last_peered":"2024-09-11T03:40:11.009612+0000","last_clean":"2024-09-11T03:40:11.009612+0000","last_became_active":"2024-09-11T03:39:26.115670+0000","last_became_peered":"2024-09-11T03:39:26.115670+0000","last_unstale":"2024-09-11T03:40:11.009612+0000","last_undegraded":"2024-09-11T03:40:11.009612+0000","last_fullsized":"2024-09-11T03:40:11.009612+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'57","last_scrub_stamp":"2024-09-11T03:39:56.268853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:56.268853+0000","objects_scrubbed":1,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:07.500803+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020826109999999998,"stat_sum":{"num_bytes":2487436,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":15,"num_read_kb":6,"num_write":34,"num_write_kb":15149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"26","length":"1"},{"start":"28","length":"1"},{"start":"2e","length":"2"},{"start":"31","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":31,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-11T03:40:12.833108+0000","last_change":"2024-09-11T03:40:06.833351+0000","last_active":"2024-09-11T03:40:12.833108+0000","last_peered":"2024-09-11T03:40:12.833108+0000","last_clean":"2024-09-11T03:40:12.833108+0000","last_became_active":"2024-09-11T03:40:05.963517+0000","last_became_peered":"2024-09-11T03:40:05.963517+0000","last_unstale":"2024-09-11T03:40:12.833108+0000","last_undegraded":"2024-09-11T03:40:12.833108+0000","last_fullsized":"2024-09-11T03:40:12.833108+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:36.470223+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":192337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"194'83","reported_seq":606,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:11.030134+0000","last_change":"2024-09-11T03:40:11.010010+0000","last_active":"2024-09-11T03:40:11.030134+0000","last_peered":"2024-09-11T03:40:11.030134+0000","last_clean":"2024-09-11T03:40:11.030134+0000","last_became_active":"2024-09-11T03:40:05.966715+0000","last_became_peered":"2024-09-11T03:40:05.966715+0000","last_unstale":"2024-09-11T03:40:11.030134+0000","last_undegraded":"2024-09-11T03:40:11.030134+0000","last_fullsized":"2024-09-11T03:40:11.030134+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'55","last_scrub_stamp":"2024-09-11T03:39:40.378705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:40.378705+0000","objects_scrubbed":3,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:03.938648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046988899999999997,"stat_sum":{"num_bytes":11647289,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":18897,"num_write":57,"num_write_kb":18703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":7190868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"26","length":"1"},{"start":"28","length":"1"},{"start":"2e","length":"2"},{"start":"31","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":270,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-11T03:40:12.837480+0000","last_change":"2024-09-11T03:40:06.985002+0000","last_active":"2024-09-11T03:40:12.837480+0000","last_peered":"2024-09-11T03:40:12.837480+0000","last_clean":"2024-09-11T03:40:12.837480+0000","last_became_active":"2024-09-11T03:40:05.964868+0000","last_became_peered":"2024-09-11T03:40:05.964868+0000","last_unstale":"2024-09-11T03:40:12.837480+0000","last_undegraded":"2024-09-11T03:40:12.837480+0000","last_fullsized":"2024-09-11T03:40:12.837480+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.855022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.855022+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:36.260431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":144253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"194'161","reported_seq":683,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:11.012964+0000","last_change":"2024-09-11T03:40:11.013140+0000","last_active":"2024-09-11T03:40:11.012964+0000","last_peered":"2024-09-11T03:40:11.012964+0000","last_clean":"2024-09-11T03:40:11.012964+0000","last_became_active":"2024-09-11T03:38:30.163080+0000","last_became_peered":"2024-09-11T03:38:30.163080+0000","last_unstale":"2024-09-11T03:40:11.012964+0000","last_undegraded":"2024-09-11T03:40:11.012964+0000","last_fullsized":"2024-09-11T03:40:11.012964+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'97","last_scrub_stamp":"2024-09-11T03:39:33.455974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:33.455974+0000","objects_scrubbed":5,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:35.263579+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0015256829999999999,"stat_sum":{"num_bytes":15092620,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":108,"num_read_kb":91833,"num_write":90,"num_write_kb":27437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"26","length":"1"},{"start":"28","length":"1"},{"start":"2e","length":"2"},{"start":"31","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":31,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-11T03:40:12.837216+0000","last_change":"2024-09-11T03:40:06.831598+0000","last_active":"2024-09-11T03:40:12.837216+0000","last_peered":"2024-09-11T03:40:12.837216+0000","last_clean":"2024-09-11T03:40:12.837216+0000","last_became_active":"2024-09-11T03:40:05.965205+0000","last_became_peered":"2024-09-11T03:40:05.965205+0000","last_unstale":"2024-09-11T03:40:12.837216+0000","last_undegraded":"2024-09-11T03:40:12.837216+0000","last_fullsized":"2024-09-11T03:40:12.837216+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:36.232323+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":192338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"194'95","reported_seq":577,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:11.011746+0000","last_change":"2024-09-11T03:40:11.011977+0000","last_active":"2024-09-11T03:40:11.011746+0000","last_peered":"2024-09-11T03:40:11.011746+0000","last_clean":"2024-09-11T03:40:11.011746+0000","last_became_active":"2024-09-11T03:38:29.887945+0000","last_became_peered":"2024-09-11T03:38:29.887945+0000","last_unstale":"2024-09-11T03:40:11.011746+0000","last_undegraded":"2024-09-11T03:40:11.011746+0000","last_fullsized":"2024-09-11T03:40:11.011746+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024031920000000002,"stat_sum":{"num_bytes":11220437,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":43097,"num_write":39,"num_write_kb":22029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"26","length":"1"},{"start":"28","length":"1"},{"start":"2e","length":"2"},{"start":"31","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":141,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:11.029209+0000","last_change":"2024-09-11T03:40:00.357039+0000","last_active":"2024-09-11T03:40:11.029209+0000","last_peered":"2024-09-11T03:40:11.029209+0000","last_clean":"2024-09-11T03:40:11.029209+0000","last_became_active":"2024-09-11T03:39:59.839301+0000","last_became_peered":"2024-09-11T03:39:59.839301+0000","last_unstale":"2024-09-11T03:40:11.029209+0000","last_undegraded":"2024-09-11T03:40:11.029209+0000","last_fullsized":"2024-09-11T03:40:11.029209+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.356985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.356985+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:14.670619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"196'114","reported_seq":681,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-11T03:40:12.844482+0000","last_change":"2024-09-11T03:40:12.844482+0000","last_active":"2024-09-11T03:40:12.844482+0000","last_peered":"2024-09-11T03:40:12.844482+0000","last_clean":"2024-09-11T03:40:12.844482+0000","last_became_active":"2024-09-11T03:40:08.836627+0000","last_became_peered":"2024-09-11T03:40:08.836627+0000","last_unstale":"2024-09-11T03:40:12.844482+0000","last_undegraded":"2024-09-11T03:40:12.844482+0000","last_fullsized":"2024-09-11T03:40:12.844482+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'75","last_scrub_stamp":"2024-09-11T03:39:39.753844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:39.753844+0000","objects_scrubbed":4,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:44.504473+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0071285100000000002,"stat_sum":{"num_bytes":9392888,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":57,"num_read_kb":32519,"num_write":67,"num_write_kb":27143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":6433401,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"28","length":"1"},{"start":"2e","length":"2"},{"start":"31","length":"1"},{"start":"33","length":"1"}]},{"pgid":"2.8","version":"192'109","reported_seq":605,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:10.985891+0000","last_change":"2024-09-11T03:40:10.986126+0000","last_active":"2024-09-11T03:40:10.985891+0000","last_peered":"2024-09-11T03:40:10.985891+0000","last_clean":"2024-09-11T03:40:10.985891+0000","last_became_active":"2024-09-11T03:38:30.162819+0000","last_became_peered":"2024-09-11T03:38:30.162819+0000","last_unstale":"2024-09-11T03:40:10.985891+0000","last_undegraded":"2024-09-11T03:40:10.985891+0000","last_fullsized":"2024-09-11T03:40:10.985891+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'86","last_scrub_stamp":"2024-09-11T03:39:54.511997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:54.511997+0000","objects_scrubbed":4,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:22.965627+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00048227699999999998,"stat_sum":{"num_bytes":14060840,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":68,"num_read_kb":67572,"num_write":61,"num_write_kb":19482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"26","length":"1"},{"start":"28","length":"1"},{"start":"2e","length":"2"},{"start":"31","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":287,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-11T03:40:12.837718+0000","last_change":"2024-09-11T03:40:00.202716+0000","last_active":"2024-09-11T03:40:12.837718+0000","last_peered":"2024-09-11T03:40:12.837718+0000","last_clean":"2024-09-11T03:40:12.837718+0000","last_became_active":"2024-09-11T03:40:00.099329+0000","last_became_peered":"2024-09-11T03:40:00.099329+0000","last_unstale":"2024-09-11T03:40:12.837718+0000","last_undegraded":"2024-09-11T03:40:12.837718+0000","last_fullsized":"2024-09-11T03:40:12.837718+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.202664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.202664+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:20.173041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"194'171","reported_seq":697,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:11.011593+0000","last_change":"2024-09-11T03:40:11.011798+0000","last_active":"2024-09-11T03:40:11.011593+0000","last_peered":"2024-09-11T03:40:11.011593+0000","last_clean":"2024-09-11T03:40:11.011593+0000","last_became_active":"2024-09-11T03:38:29.888204+0000","last_became_peered":"2024-09-11T03:38:29.888204+0000","last_unstale":"2024-09-11T03:40:11.011593+0000","last_undegraded":"2024-09-11T03:40:11.011593+0000","last_fullsized":"2024-09-11T03:40:11.011593+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'138","last_scrub_stamp":"2024-09-11T03:39:52.405246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:52.405246+0000","objects_scrubbed":6,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:15.795295+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0022796169999999998,"stat_sum":{"num_bytes":18240078,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":136,"num_read_kb":100304,"num_write":104,"num_write_kb":37511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"26","length":"1"},{"start":"28","length":"1"},{"start":"2e","length":"2"},{"start":"31","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":371,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:10.985581+0000","last_change":"2024-09-11T03:40:00.516395+0000","last_active":"2024-09-11T03:40:10.985581+0000","last_peered":"2024-09-11T03:40:10.985581+0000","last_clean":"2024-09-11T03:40:10.985581+0000","last_became_active":"2024-09-11T03:39:59.838010+0000","last_became_peered":"2024-09-11T03:39:59.838010+0000","last_unstale":"2024-09-11T03:40:10.985581+0000","last_undegraded":"2024-09-11T03:40:10.985581+0000","last_fullsized":"2024-09-11T03:40:10.985581+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.516339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.516339+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:08.765216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"186'37","reported_seq":528,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:11.029508+0000","last_change":"2024-09-11T03:40:11.029735+0000","last_active":"2024-09-11T03:40:11.029508+0000","last_peered":"2024-09-11T03:40:11.029508+0000","last_clean":"2024-09-11T03:40:11.029508+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:40:11.029508+0000","last_undegraded":"2024-09-11T03:40:11.029508+0000","last_fullsized":"2024-09-11T03:40:11.029508+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'8","last_scrub_stamp":"2024-09-11T03:39:33.449882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:33.449882+0000","objects_scrubbed":1,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:50.337989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046408300000000001,"stat_sum":{"num_bytes":2638699,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":34895,"num_write":16,"num_write_kb":10633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"26","length":"1"},{"start":"28","length":"1"},{"start":"2e","length":"2"},{"start":"31","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":49,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:11.007611+0000","last_change":"2024-09-11T03:40:00.207222+0000","last_active":"2024-09-11T03:40:11.007611+0000","last_peered":"2024-09-11T03:40:11.007611+0000","last_clean":"2024-09-11T03:40:11.007611+0000","last_became_active":"2024-09-11T03:39:59.849648+0000","last_became_peered":"2024-09-11T03:39:59.849648+0000","last_unstale":"2024-09-11T03:40:11.007611+0000","last_undegraded":"2024-09-11T03:40:11.007611+0000","last_fullsized":"2024-09-11T03:40:11.007611+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.207167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.207167+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:25.470578+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"191'106","reported_seq":600,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-11T03:40:11.011876+0000","last_change":"2024-09-11T03:40:11.012063+0000","last_active":"2024-09-11T03:40:11.011876+0000","last_peered":"2024-09-11T03:40:11.011876+0000","last_clean":"2024-09-11T03:40:11.011876+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:40:11.011876+0000","last_undegraded":"2024-09-11T03:40:11.011876+0000","last_fullsized":"2024-09-11T03:40:11.011876+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'86","last_scrub_stamp":"2024-09-11T03:39:54.238844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:54.238844+0000","objects_scrubbed":5,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:56.087173+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00043843199999999999,"stat_sum":{"num_bytes":11210170,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":32,"num_read_kb":15671,"num_write":47,"num_write_kb":28169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"26","length":"1"},{"start":"28","length":"1"},{"start":"2e","length":"2"},{"start":"31","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":148389008,"num_objects":69,"num_object_clones":27,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":8,"num_read":902,"num_read_kb":675136,"num_write":807,"num_write_kb":323616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":13624269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":269271040,"data_stored":300001323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1557,"ondisk_log_size":1557,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":805048,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":45,"num_write_kb":782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":673182,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285604,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45920,"kb_used_data":18892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93699232,"statfs":{"total":95995035648,"available":95948013568,"internally_reserved":0,"allocated":19345408,"data_stored":20723046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":326417514536,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743470,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63964,"kb_used_data":32772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93681188,"statfs":{"total":95995035648,"available":95929536512,"internally_reserved":0,"allocated":33558528,"data_stored":37644950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037813,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61016,"kb_used_data":29828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93684136,"statfs":{"total":95995035648,"available":95932555264,"internally_reserved":0,"allocated":30543872,"data_stored":30679674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299448,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62152,"kb_used_data":30960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93683000,"statfs":{"total":95995035648,"available":95931392000,"internally_reserved":0,"allocated":31703040,"data_stored":32490318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":193273528381,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37696,"kb_used_data":6500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93707456,"statfs":{"total":95995035648,"available":95956434944,"internally_reserved":0,"allocated":6656000,"data_stored":5677149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986247,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57236,"kb_used_data":26044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93687916,"statfs":{"total":95995035648,"available":95936425984,"internally_reserved":0,"allocated":26669056,"data_stored":25921576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72380,"kb_used_data":41188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93672772,"statfs":{"total":95995035648,"available":95920918528,"internally_reserved":0,"allocated":42176512,"data_stored":47274707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54624,"kb_used_data":23432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93690528,"statfs":{"total":95995035648,"available":95939100672,"internally_reserved":0,"allocated":23994368,"data_stored":27657549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66712,"kb_used_data":35520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93678440,"statfs":{"total":95995035648,"available":95926722560,"internally_reserved":0,"allocated":36372480,"data_stored":41513182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64724,"kb_used_data":33528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93680428,"statfs":{"total":95995035648,"available":95928758272,"internally_reserved":0,"allocated":34332672,"data_stored":37096759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32100352,"data_stored":35802587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34127872,"data_stored":40210846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":23040000,"data_stored":27653145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":39702528,"data_stored":45754811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24449024,"data_stored":24635512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4206592,"data_stored":4173565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4079616,"data_stored":4066399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":29143040,"data_stored":30868466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":27971584,"data_stored":29049658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":31289344,"data_stored":36326302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2387968,"data_stored":2367002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16773120,"data_stored":19093030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:40:15.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:15.372 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:15.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:15.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:15 smithi121 ceph-mon[30729]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:40:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:15 smithi121 ceph-mon[30729]: osdmap e198: 12 total, 12 up, 10 in 2024-09-11T03:40:15.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:15 smithi175 ceph-mon[30894]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:40:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:15 smithi175 ceph-mon[30894]: osdmap e198: 12 total, 12 up, 10 in 2024-09-11T03:40:15.498 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:15.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:15.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:15.545 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:40:15.546 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:40:15.546 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:40:15.546 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:40:15.546 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:40:15.546 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:40:15.601 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:15.623 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:15.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:15.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:15.725 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:15.747 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:15.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:15.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:15.752 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:40:15.752 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:40:15.769 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:40:15.770 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:40:15.787 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:40:15.787 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:40:15.804 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:40:15.804 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:40:15.821 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:40:15.852 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:40:15.853 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:40:15.853 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:40:15.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:15.854 INFO:tasks.rados.rados.2.smithi175.stdout:509: done (4 left) 2024-09-11T03:40:15.854 INFO:tasks.rados.rados.2.smithi175.stdout:510: done (3 left) 2024-09-11T03:40:15.854 INFO:tasks.rados.rados.2.smithi175.stdout:511: done (2 left) 2024-09-11T03:40:15.854 INFO:tasks.rados.rados.2.smithi175.stdout:512: done (1 left) 2024-09-11T03:40:15.854 INFO:tasks.rados.rados.2.smithi175.stdout:513: done (0 left) 2024-09-11T03:40:15.854 INFO:tasks.rados.rados.2.smithi175.stdout:514: write (excl) oid 39 current snap is 51 2024-09-11T03:40:15.855 INFO:tasks.rados.rados.2.smithi175.stdout:514: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2024-09-11T03:40:15.855 INFO:tasks.rados.rados.2.smithi175.stdout:514: writing smithi17557870-39 from 695856 to 1489179 tid 1 2024-09-11T03:40:15.856 INFO:tasks.rados.rados.2.smithi175.stdout:514: writing smithi17557870-39 from 2171784 to 2706183 tid 2 2024-09-11T03:40:15.857 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:40:15.858 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:40:15.861 INFO:tasks.rados.rados.2.smithi175.stdout:514: writing smithi17557870-39 from 3362453 to 3495856 tid 3 2024-09-11T03:40:15.861 INFO:tasks.rados.rados.2.smithi175.stdout:515: write (excl) oid 13 current snap is 51 2024-09-11T03:40:15.862 INFO:tasks.rados.rados.2.smithi175.stdout:514: finishing write tid 1 to smithi17557870-39 2024-09-11T03:40:15.862 INFO:tasks.rados.rados.2.smithi175.stdout:514: finishing write tid 2 to smithi17557870-39 2024-09-11T03:40:15.862 INFO:tasks.rados.rados.2.smithi175.stdout:515: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2024-09-11T03:40:15.867 INFO:tasks.rados.rados.2.smithi175.stdout:515: writing smithi17557870-13 from 712663 to 1181235 tid 1 2024-09-11T03:40:15.874 INFO:tasks.rados.rados.2.smithi175.stdout:515: writing smithi17557870-13 from 1713913 to 2391970 tid 2 2024-09-11T03:40:15.876 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:40:15.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:40:15.880 INFO:tasks.rados.rados.2.smithi175.stdout:515: writing smithi17557870-13 from 2873523 to 3465256 tid 3 2024-09-11T03:40:15.881 INFO:tasks.rados.rados.2.smithi175.stdout:516: read oid 38 snap -1 2024-09-11T03:40:15.881 INFO:tasks.rados.rados.2.smithi175.stdout:516: expect (ObjNum 101 snap 16 seq_num 101) 2024-09-11T03:40:15.881 INFO:tasks.rados.rados.2.smithi175.stdout:517: write (excl) oid 2 current snap is 51 2024-09-11T03:40:15.881 INFO:tasks.rados.rados.2.smithi175.stdout:514: finishing write tid 3 to smithi17557870-39 2024-09-11T03:40:15.881 INFO:tasks.rados.rados.2.smithi175.stdout:514: finishing write tid 4 to smithi17557870-39 2024-09-11T03:40:15.881 INFO:tasks.rados.rados.2.smithi175.stdout:514: finishing write tid 5 to smithi17557870-39 2024-09-11T03:40:15.881 INFO:tasks.rados.rados.2.smithi175.stdout:514: oid 39 updating version 0 to 116 2024-09-11T03:40:15.881 INFO:tasks.rados.rados.2.smithi175.stdout:514: oid 39 version 116 is already newer than 115 2024-09-11T03:40:15.881 INFO:tasks.rados.rados.2.smithi175.stdout:514: oid 39 updating version 116 to 117 2024-09-11T03:40:15.881 INFO:tasks.rados.rados.2.smithi175.stdout:514: oid 39 updating version 117 to 118 2024-09-11T03:40:15.881 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 118 (ObjNum 208 snap 51 seq_num 208) dirty exists 2024-09-11T03:40:15.881 INFO:tasks.rados.rados.2.smithi175.stdout:514: left oid 39 (ObjNum 208 snap 51 seq_num 208) 2024-09-11T03:40:15.882 INFO:tasks.rados.rados.2.smithi175.stdout:515: finishing write tid 1 to smithi17557870-13 2024-09-11T03:40:15.882 INFO:tasks.rados.rados.2.smithi175.stdout:515: finishing write tid 2 to smithi17557870-13 2024-09-11T03:40:15.882 INFO:tasks.rados.rados.2.smithi175.stdout:517: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2024-09-11T03:40:15.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:15.883 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:15.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:15.887 INFO:tasks.rados.rados.2.smithi175.stdout:517: writing smithi17557870-2 from 729470 to 1273291 tid 1 2024-09-11T03:40:15.893 INFO:tasks.rados.rados.2.smithi175.stdout:517: writing smithi17557870-2 from 1772395 to 2310463 tid 2 2024-09-11T03:40:15.895 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:40:15.895 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:40:15.899 INFO:tasks.rados.rados.2.smithi175.stdout:517: writing smithi17557870-2 from 2900946 to 3303951 tid 3 2024-09-11T03:40:15.899 INFO:tasks.rados.rados.2.smithi175.stdout:514: done (3 left) 2024-09-11T03:40:15.899 INFO:tasks.rados.rados.2.smithi175.stdout:518: copy_from oid 40 from oid 46 current snap is 51 2024-09-11T03:40:15.899 INFO:tasks.rados.rados.2.smithi175.stdout:515: finishing write tid 3 to smithi17557870-13 2024-09-11T03:40:15.899 INFO:tasks.rados.rados.2.smithi175.stdout:515: finishing write tid 4 to smithi17557870-13 2024-09-11T03:40:15.899 INFO:tasks.rados.rados.2.smithi175.stdout:515: finishing write tid 5 to smithi17557870-13 2024-09-11T03:40:15.899 INFO:tasks.rados.rados.2.smithi175.stdout:515: oid 13 updating version 0 to 168 2024-09-11T03:40:15.899 INFO:tasks.rados.rados.2.smithi175.stdout:515: oid 13 version 168 is already newer than 167 2024-09-11T03:40:15.899 INFO:tasks.rados.rados.2.smithi175.stdout:515: oid 13 updating version 168 to 169 2024-09-11T03:40:15.899 INFO:tasks.rados.rados.2.smithi175.stdout:515: oid 13 updating version 169 to 170 2024-09-11T03:40:15.899 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 170 (ObjNum 209 snap 51 seq_num 209) dirty exists 2024-09-11T03:40:15.900 INFO:tasks.rados.rados.2.smithi175.stdout:515: left oid 13 (ObjNum 209 snap 51 seq_num 209) 2024-09-11T03:40:15.900 INFO:tasks.rados.rados.2.smithi175.stdout:516: expect (ObjNum 101 snap 16 seq_num 101) 2024-09-11T03:40:15.983 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:16.005 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:16.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:16.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:16.059 INFO:tasks.rados.rados.2.smithi175.stdout:517: finishing write tid 1 to smithi17557870-2 2024-09-11T03:40:16.060 INFO:tasks.rados.rados.2.smithi175.stdout:517: finishing write tid 2 to smithi17557870-2 2024-09-11T03:40:16.060 INFO:tasks.rados.rados.2.smithi175.stdout:515: done (3 left) 2024-09-11T03:40:16.060 INFO:tasks.rados.rados.2.smithi175.stdout:516: done (2 left) 2024-09-11T03:40:16.060 INFO:tasks.rados.rados.2.smithi175.stdout:519: snap_create 2024-09-11T03:40:16.060 INFO:tasks.rados.rados.2.smithi175.stdout:517: finishing write tid 3 to smithi17557870-2 2024-09-11T03:40:16.060 INFO:tasks.rados.rados.2.smithi175.stdout:517: finishing write tid 4 to smithi17557870-2 2024-09-11T03:40:16.060 INFO:tasks.rados.rados.2.smithi175.stdout:517: finishing write tid 5 to smithi17557870-2 2024-09-11T03:40:16.060 INFO:tasks.rados.rados.2.smithi175.stdout:517: oid 2 updating version 0 to 123 2024-09-11T03:40:16.060 INFO:tasks.rados.rados.2.smithi175.stdout:517: oid 2 version 123 is already newer than 120 2024-09-11T03:40:16.061 INFO:tasks.rados.rados.2.smithi175.stdout:517: oid 2 version 123 is already newer than 121 2024-09-11T03:40:16.061 INFO:tasks.rados.rados.2.smithi175.stdout:517: oid 2 version 123 is already newer than 122 2024-09-11T03:40:16.061 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 123 (ObjNum 210 snap 51 seq_num 210) dirty exists 2024-09-11T03:40:16.061 INFO:tasks.rados.rados.2.smithi175.stdout:517: left oid 2 (ObjNum 210 snap 51 seq_num 210) 2024-09-11T03:40:16.061 INFO:tasks.rados.rados.2.smithi175.stdout:518: finishing copy_from racing read to smithi17557870-40 2024-09-11T03:40:16.061 INFO:tasks.rados.rados.2.smithi175.stdout:518: finishing copy_from to smithi17557870-40 2024-09-11T03:40:16.061 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 34 (ObjNum 157 snap 35 seq_num 157) dirty exists 2024-09-11T03:40:16.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:16 smithi084 ceph-mon[27254]: pgmap v451: 27 pgs: 27 active+clean; 142 MiB data, 573 MiB used, 893 GiB / 894 GiB avail; 11 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-11T03:40:16.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:16 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3507325846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:16.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:16 smithi084 ceph-mon[27254]: osdmap e199: 12 total, 12 up, 10 in 2024-09-11T03:40:16.107 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:16.123 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:16.123 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:16.128 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:16.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:16.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:16.227 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:40:16.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:16.255 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:16.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:16.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:16.357 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:16.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:16.379 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:16.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:16.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:16 smithi121 ceph-mon[30729]: pgmap v451: 27 pgs: 27 active+clean; 142 MiB data, 573 MiB used, 893 GiB / 894 GiB avail; 11 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-11T03:40:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:16 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3507325846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:16 smithi121 ceph-mon[30729]: osdmap e199: 12 total, 12 up, 10 in 2024-09-11T03:40:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:16 smithi175 ceph-mon[30894]: pgmap v451: 27 pgs: 27 active+clean; 142 MiB data, 573 MiB used, 893 GiB / 894 GiB avail; 11 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-11T03:40:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:16 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3507325846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:16 smithi175 ceph-mon[30894]: osdmap e199: 12 total, 12 up, 10 in 2024-09-11T03:40:16.481 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:16.501 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:16.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:16.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:16.603 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:16.627 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:16.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:16.628 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:16.729 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:16.750 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:16.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:16.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:16.835 INFO:tasks.rados.rados.2.smithi175.stdout:517: done (2 left) 2024-09-11T03:40:16.835 INFO:tasks.rados.rados.2.smithi175.stdout:518: done (1 left) 2024-09-11T03:40:16.835 INFO:tasks.rados.rados.2.smithi175.stdout:519: done (0 left) 2024-09-11T03:40:16.835 INFO:tasks.rados.rados.2.smithi175.stdout:520: read oid 5 snap 48 2024-09-11T03:40:16.836 INFO:tasks.rados.rados.2.smithi175.stdout:520: expect deleted 2024-09-11T03:40:16.836 INFO:tasks.rados.rados.2.smithi175.stdout:521: read oid 29 snap -1 2024-09-11T03:40:16.836 INFO:tasks.rados.rados.2.smithi175.stdout:521: expect (ObjNum 132 snap 22 seq_num 132) 2024-09-11T03:40:16.836 INFO:tasks.rados.rados.2.smithi175.stdout:522: read oid 7 snap 48 2024-09-11T03:40:16.836 INFO:tasks.rados.rados.2.smithi175.stdout:522: expect (ObjNum 195 snap 48 seq_num 195) 2024-09-11T03:40:16.836 INFO:tasks.rados.rados.2.smithi175.stdout:523: snap_remove snap 50 2024-09-11T03:40:16.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:16.878 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:16.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:16.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:16.980 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:17.006 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:17.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:17.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:17.108 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:17.132 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:17.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:17.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:17.234 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:17.256 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:17.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:17.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:17.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T03:40:17.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:17.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:17.381 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:17.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:17.409 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:17.482 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:17.504 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:17.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:17.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:17.533 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:40:17.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:17.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:17.629 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:17.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:17.730 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:17.756 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:17.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:17.757 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:17.836 INFO:tasks.rados.rados.2.smithi175.stdout:523: done (3 left) 2024-09-11T03:40:17.836 INFO:tasks.rados.rados.2.smithi175.stdout:524: read oid 14 snap 51 2024-09-11T03:40:17.836 INFO:tasks.rados.rados.2.smithi175.stdout:524: expect deleted 2024-09-11T03:40:17.836 INFO:tasks.rados.rados.2.smithi175.stdout:525: delete oid 15 current snap is 52 2024-09-11T03:40:17.836 INFO:tasks.rados.rados.2.smithi175.stdout:521: expect (ObjNum 132 snap 22 seq_num 132) 2024-09-11T03:40:17.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:17.879 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:17.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:17.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:17.924 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:17 smithi121 ceph-mon[30729]: pgmap v454: 27 pgs: 27 active+clean; 145 MiB data, 577 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 4.5 MiB/s wr, 18 op/s 2024-09-11T03:40:17.924 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:17 smithi121 ceph-mon[30729]: osdmap e200: 12 total, 12 up, 10 in 2024-09-11T03:40:17.924 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:17 smithi121 ceph-mon[30729]: 2.6 scrub starts 2024-09-11T03:40:17.924 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:17 smithi121 ceph-mon[30729]: 2.6 scrub ok 2024-09-11T03:40:17.930 INFO:tasks.rados.rados.2.smithi175.stdout:522: expect (ObjNum 195 snap 48 seq_num 195) 2024-09-11T03:40:17.981 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:18.002 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:18.004 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:18.008 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:18.009 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:18.105 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:18.115 INFO:tasks.rados.rados.2.smithi175.stdout:520: done (4 left) 2024-09-11T03:40:18.115 INFO:tasks.rados.rados.2.smithi175.stdout:521: done (3 left) 2024-09-11T03:40:18.115 INFO:tasks.rados.rados.2.smithi175.stdout:522: done (2 left) 2024-09-11T03:40:18.115 INFO:tasks.rados.rados.2.smithi175.stdout:524: done (1 left) 2024-09-11T03:40:18.116 INFO:tasks.rados.rados.2.smithi175.stdout:525: done (0 left) 2024-09-11T03:40:18.116 INFO:tasks.rados.rados.2.smithi175.stdout:526: write (excl) oid 31 current snap is 52 2024-09-11T03:40:18.116 INFO:tasks.rados.rados.2.smithi175.stdout:526: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2024-09-11T03:40:18.123 INFO:tasks.rados.rados.2.smithi175.stdout:526: writing smithi17557870-31 from 763084 to 1457403 tid 1 2024-09-11T03:40:18.125 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:18.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:18.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:18.128 INFO:tasks.rados.rados.2.smithi175.stdout:526: writing smithi17557870-31 from 1889359 to 2431096 tid 2 2024-09-11T03:40:18.134 INFO:tasks.rados.rados.2.smithi175.stdout:526: writing smithi17557870-31 from 2955792 to 3497694 tid 3 2024-09-11T03:40:18.134 INFO:tasks.rados.rados.2.smithi175.stdout:527: delete oid 4 current snap is 52 2024-09-11T03:40:18.134 INFO:tasks.rados.rados.2.smithi175.stdout:526: finishing write tid 1 to smithi17557870-31 2024-09-11T03:40:18.135 INFO:tasks.rados.rados.2.smithi175.stdout:526: finishing write tid 2 to smithi17557870-31 2024-09-11T03:40:18.136 INFO:tasks.rados.rados.2.smithi175.stdout:527: done (1 left) 2024-09-11T03:40:18.136 INFO:tasks.rados.rados.2.smithi175.stdout:528: delete oid 8 current snap is 52 2024-09-11T03:40:18.137 INFO:tasks.rados.rados.2.smithi175.stdout:526: finishing write tid 3 to smithi17557870-31 2024-09-11T03:40:18.137 INFO:tasks.rados.rados.2.smithi175.stdout:526: finishing write tid 4 to smithi17557870-31 2024-09-11T03:40:18.138 INFO:tasks.rados.rados.2.smithi175.stdout:526: finishing write tid 5 to smithi17557870-31 2024-09-11T03:40:18.138 INFO:tasks.rados.rados.2.smithi175.stdout:526: oid 31 updating version 0 to 127 2024-09-11T03:40:18.138 INFO:tasks.rados.rados.2.smithi175.stdout:526: oid 31 updating version 127 to 128 2024-09-11T03:40:18.138 INFO:tasks.rados.rados.2.smithi175.stdout:526: oid 31 version 128 is already newer than 126 2024-09-11T03:40:18.138 INFO:tasks.rados.rados.2.smithi175.stdout:526: oid 31 updating version 128 to 129 2024-09-11T03:40:18.138 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 129 (ObjNum 212 snap 52 seq_num 212) dirty exists 2024-09-11T03:40:18.138 INFO:tasks.rados.rados.2.smithi175.stdout:526: left oid 31 (ObjNum 212 snap 52 seq_num 212) 2024-09-11T03:40:18.138 INFO:tasks.rados.rados.2.smithi175.stdout:526: done (1 left) 2024-09-11T03:40:18.138 INFO:tasks.rados.rados.2.smithi175.stdout:528: done (0 left) 2024-09-11T03:40:18.138 INFO:tasks.rados.rados.2.smithi175.stdout:529: write (excl) oid 35 current snap is 52 2024-09-11T03:40:18.138 INFO:tasks.rados.rados.2.smithi175.stdout:529: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2024-09-11T03:40:18.143 INFO:tasks.rados.rados.2.smithi175.stdout:529: writing smithi17557870-35 from 413505 to 1049924 tid 1 2024-09-11T03:40:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:17 smithi084 ceph-mon[27254]: pgmap v454: 27 pgs: 27 active+clean; 145 MiB data, 577 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 4.5 MiB/s wr, 18 op/s 2024-09-11T03:40:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:17 smithi084 ceph-mon[27254]: osdmap e200: 12 total, 12 up, 10 in 2024-09-11T03:40:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:17 smithi084 ceph-mon[27254]: 2.6 scrub starts 2024-09-11T03:40:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:17 smithi084 ceph-mon[27254]: 2.6 scrub ok 2024-09-11T03:40:18.150 INFO:tasks.rados.rados.2.smithi175.stdout:529: writing smithi17557870-35 from 1664805 to 2353869 tid 2 2024-09-11T03:40:18.157 INFO:tasks.rados.rados.2.smithi175.stdout:529: writing smithi17557870-35 from 3038061 to 3530132 tid 3 2024-09-11T03:40:18.157 INFO:tasks.rados.rados.2.smithi175.stdout:530: snap_create 2024-09-11T03:40:18.157 INFO:tasks.rados.rados.2.smithi175.stdout:529: finishing write tid 1 to smithi17557870-35 2024-09-11T03:40:18.157 INFO:tasks.rados.rados.2.smithi175.stdout:529: finishing write tid 2 to smithi17557870-35 2024-09-11T03:40:18.160 INFO:tasks.rados.rados.2.smithi175.stdout:529: finishing write tid 3 to smithi17557870-35 2024-09-11T03:40:18.161 INFO:tasks.rados.rados.2.smithi175.stdout:529: finishing write tid 4 to smithi17557870-35 2024-09-11T03:40:18.161 INFO:tasks.rados.rados.2.smithi175.stdout:529: finishing write tid 5 to smithi17557870-35 2024-09-11T03:40:18.161 INFO:tasks.rados.rados.2.smithi175.stdout:529: oid 35 updating version 0 to 135 2024-09-11T03:40:18.161 INFO:tasks.rados.rados.2.smithi175.stdout:529: oid 35 updating version 135 to 138 2024-09-11T03:40:18.161 INFO:tasks.rados.rados.2.smithi175.stdout:529: oid 35 version 138 is already newer than 137 2024-09-11T03:40:18.161 INFO:tasks.rados.rados.2.smithi175.stdout:529: oid 35 version 138 is already newer than 136 2024-09-11T03:40:18.161 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 138 (ObjNum 215 snap 52 seq_num 215) dirty exists 2024-09-11T03:40:18.161 INFO:tasks.rados.rados.2.smithi175.stdout:529: left oid 35 (ObjNum 215 snap 52 seq_num 215) 2024-09-11T03:40:18.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:17 smithi175 ceph-mon[30894]: pgmap v454: 27 pgs: 27 active+clean; 145 MiB data, 577 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 4.5 MiB/s wr, 18 op/s 2024-09-11T03:40:18.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:17 smithi175 ceph-mon[30894]: osdmap e200: 12 total, 12 up, 10 in 2024-09-11T03:40:18.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:17 smithi175 ceph-mon[30894]: 2.6 scrub starts 2024-09-11T03:40:18.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:17 smithi175 ceph-mon[30894]: 2.6 scrub ok 2024-09-11T03:40:18.228 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:18.250 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:18.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:18.352 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:18.373 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:18.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:18.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:18.475 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:18.496 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:18.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:18.598 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:18.621 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:18.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:18.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:18.723 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:18.744 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:18.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:18.745 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:18.749 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T03:40:18.841 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:40:18.844 INFO:tasks.rados.rados.2.smithi175.stdout:529: done (1 left) 2024-09-11T03:40:18.844 INFO:tasks.rados.rados.2.smithi175.stdout:530: done (0 left) 2024-09-11T03:40:18.844 INFO:tasks.rados.rados.2.smithi175.stdout:531: read oid 20 snap -1 2024-09-11T03:40:18.844 INFO:tasks.rados.rados.2.smithi175.stdout:531: expect (ObjNum 119 snap 18 seq_num 119) 2024-09-11T03:40:18.844 INFO:tasks.rados.rados.2.smithi175.stdout:532: read oid 4 snap -1 2024-09-11T03:40:18.844 INFO:tasks.rados.rados.2.smithi175.stdout:532: expect deleted 2024-09-11T03:40:18.845 INFO:tasks.rados.rados.2.smithi175.stdout:533: copy_from oid 40 from oid 32 current snap is 53 2024-09-11T03:40:18.845 INFO:tasks.rados.rados.2.smithi175.stdout:534: rollback oid 36 current snap is 53 2024-09-11T03:40:18.845 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 36 to 41 2024-09-11T03:40:18.845 INFO:tasks.rados.rados.2.smithi175.stdout:535: write (excl) oid 7 current snap is 53 2024-09-11T03:40:18.845 INFO:tasks.rados.rados.2.smithi175.stdout:535: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2024-09-11T03:40:18.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:18.851 INFO:tasks.rados.rados.2.smithi175.stdout:535: writing smithi17557870-7 from 430312 to 1141980 tid 1 2024-09-11T03:40:18.861 INFO:tasks.rados.rados.2.smithi175.stdout:535: writing smithi17557870-7 from 1723287 to 2272362 tid 2 2024-09-11T03:40:18.865 INFO:tasks.rados.rados.2.smithi175.stdout:535: writing smithi17557870-7 from 2781837 to 3485180 tid 3 2024-09-11T03:40:18.865 INFO:tasks.rados.rados.2.smithi175.stdout:536: write oid 6 current snap is 53 2024-09-11T03:40:18.865 INFO:tasks.rados.rados.2.smithi175.stdout:536: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2024-09-11T03:40:18.870 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:18.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:18.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:18.872 INFO:tasks.rados.rados.2.smithi175.stdout:536: writing smithi17557870-6 from 447119 to 1234036 tid 1 2024-09-11T03:40:18.878 INFO:tasks.rados.rados.2.smithi175.stdout:536: writing smithi17557870-6 from 1665416 to 2358149 tid 2 2024-09-11T03:40:18.884 INFO:tasks.rados.rados.2.smithi175.stdout:536: writing smithi17557870-6 from 2976554 to 3491169 tid 3 2024-09-11T03:40:18.884 INFO:tasks.rados.rados.2.smithi175.stdout:537: delete oid 9 current snap is 53 2024-09-11T03:40:18.884 INFO:tasks.rados.rados.2.smithi175.stdout:534: finishing rollback tid 0 to smithi17557870-36 2024-09-11T03:40:18.884 INFO:tasks.rados.rados.2.smithi175.stdout:534: finishing rollback tid 1 to smithi17557870-36 2024-09-11T03:40:18.885 INFO:tasks.rados.rados.2.smithi175.stdout:534: finishing rollback tid 2 to smithi17557870-36 2024-09-11T03:40:18.885 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 123 (ObjNum 150 snap 35 seq_num 150) dirty exists 2024-09-11T03:40:18.885 INFO:tasks.rados.rados.2.smithi175.stdout:531: expect (ObjNum 119 snap 18 seq_num 119) 2024-09-11T03:40:18.899 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:40:18.950 INFO:tasks.rados.rados.2.smithi175.stdout:535: finishing write tid 1 to smithi17557870-7 2024-09-11T03:40:18.951 INFO:tasks.rados.rados.2.smithi175.stdout:533: finishing copy_from racing read to smithi17557870-40 2024-09-11T03:40:18.952 INFO:tasks.rados.rados.2.smithi175.stdout:535: finishing write tid 2 to smithi17557870-7 2024-09-11T03:40:18.952 INFO:tasks.rados.rados.2.smithi175.stdout:533: finishing copy_from to smithi17557870-40 2024-09-11T03:40:18.952 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 113 (ObjNum 207 snap 50 seq_num 207) dirty exists 2024-09-11T03:40:18.952 INFO:tasks.rados.rados.2.smithi175.stdout:535: finishing write tid 3 to smithi17557870-7 2024-09-11T03:40:18.952 INFO:tasks.rados.rados.2.smithi175.stdout:531: done (6 left) 2024-09-11T03:40:18.952 INFO:tasks.rados.rados.2.smithi175.stdout:532: done (5 left) 2024-09-11T03:40:18.952 INFO:tasks.rados.rados.2.smithi175.stdout:533: done (4 left) 2024-09-11T03:40:18.952 INFO:tasks.rados.rados.2.smithi175.stdout:534: done (3 left) 2024-09-11T03:40:18.952 INFO:tasks.rados.rados.2.smithi175.stdout:537: done (2 left) 2024-09-11T03:40:18.953 INFO:tasks.rados.rados.2.smithi175.stdout:538: copy_from oid 23 from oid 48 current snap is 53 2024-09-11T03:40:18.953 INFO:tasks.rados.rados.2.smithi175.stdout:535: finishing write tid 4 to smithi17557870-7 2024-09-11T03:40:18.953 INFO:tasks.rados.rados.2.smithi175.stdout:535: finishing write tid 5 to smithi17557870-7 2024-09-11T03:40:18.953 INFO:tasks.rados.rados.2.smithi175.stdout:535: oid 7 updating version 0 to 106 2024-09-11T03:40:18.954 INFO:tasks.rados.rados.2.smithi175.stdout:535: oid 7 updating version 106 to 108 2024-09-11T03:40:18.954 INFO:tasks.rados.rados.2.smithi175.stdout:535: oid 7 version 108 is already newer than 107 2024-09-11T03:40:18.954 INFO:tasks.rados.rados.2.smithi175.stdout:535: oid 7 version 108 is already newer than 105 2024-09-11T03:40:18.954 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 108 (ObjNum 216 snap 53 seq_num 216) dirty exists 2024-09-11T03:40:18.954 INFO:tasks.rados.rados.2.smithi175.stdout:535: left oid 7 (ObjNum 216 snap 53 seq_num 216) 2024-09-11T03:40:18.954 INFO:tasks.rados.rados.2.smithi175.stdout:536: finishing write tid 1 to smithi17557870-6 2024-09-11T03:40:18.954 INFO:tasks.rados.rados.2.smithi175.stdout:536: finishing write tid 2 to smithi17557870-6 2024-09-11T03:40:18.954 INFO:tasks.rados.rados.2.smithi175.stdout:536: finishing write tid 3 to smithi17557870-6 2024-09-11T03:40:18.954 INFO:tasks.rados.rados.2.smithi175.stdout:536: finishing write tid 4 to smithi17557870-6 2024-09-11T03:40:18.954 INFO:tasks.rados.rados.2.smithi175.stdout:536: finishing write tid 5 to smithi17557870-6 2024-09-11T03:40:18.954 INFO:tasks.rados.rados.2.smithi175.stdout:536: oid 6 updating version 0 to 183 2024-09-11T03:40:18.954 INFO:tasks.rados.rados.2.smithi175.stdout:536: oid 6 version 183 is already newer than 182 2024-09-11T03:40:18.954 INFO:tasks.rados.rados.2.smithi175.stdout:536: oid 6 updating version 183 to 184 2024-09-11T03:40:18.954 INFO:tasks.rados.rados.2.smithi175.stdout:536: oid 6 version 184 is already newer than 181 2024-09-11T03:40:18.955 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 184 (ObjNum 217 snap 53 seq_num 217) dirty exists 2024-09-11T03:40:18.955 INFO:tasks.rados.rados.2.smithi175.stdout:536: left oid 6 (ObjNum 217 snap 53 seq_num 217) 2024-09-11T03:40:18.955 INFO:tasks.rados.rados.2.smithi175.stdout:535: done (2 left) 2024-09-11T03:40:18.955 INFO:tasks.rados.rados.2.smithi175.stdout:536: done (1 left) 2024-09-11T03:40:18.955 INFO:tasks.rados.rados.2.smithi175.stdout:539: delete oid 19 current snap is 53 2024-09-11T03:40:18.955 INFO:tasks.rados.rados.2.smithi175.stdout:539: done (1 left) 2024-09-11T03:40:18.955 INFO:tasks.rados.rados.2.smithi175.stdout:540: snap_remove snap 48 2024-09-11T03:40:18.972 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:18.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:18.994 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:18.994 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:19.017 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:40:19.043 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:18 smithi121 ceph-mon[30729]: osdmap e201: 12 total, 12 up, 10 in 2024-09-11T03:40:19.043 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:18 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1359790028' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:40:19.043 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:40:19.095 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:19.118 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:19.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:19.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:19.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:18 smithi084 ceph-mon[27254]: osdmap e201: 12 total, 12 up, 10 in 2024-09-11T03:40:19.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:18 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1359790028' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:40:19.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:40:19.220 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:19.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:18 smithi175 ceph-mon[30894]: osdmap e201: 12 total, 12 up, 10 in 2024-09-11T03:40:19.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:18 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1359790028' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:40:19.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:40:19.241 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:19.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:19.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:19.365 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:19.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:19.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:19.467 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:19.489 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:19.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:19.490 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:19.498 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:19.498 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:19.582 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:19.590 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:19.612 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:19.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:19.613 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:19.714 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:19.737 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:19.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:19.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:19.854 INFO:tasks.rados.rados.2.smithi175.stdout:540: done (1 left) 2024-09-11T03:40:19.854 INFO:tasks.rados.rados.2.smithi175.stdout:541: write oid 28 current snap is 53 2024-09-11T03:40:19.854 INFO:tasks.rados.rados.2.smithi175.stdout:541: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-09-11T03:40:19.860 INFO:tasks.rados.rados.2.smithi175.stdout:541: writing smithi17557870-28 from 497540 to 1110204 tid 1 2024-09-11T03:40:19.861 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:19.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:19.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:19.866 INFO:tasks.rados.rados.2.smithi175.stdout:541: writing smithi17557870-28 from 1840862 to 2280922 tid 2 2024-09-11T03:40:19.873 INFO:tasks.rados.rados.2.smithi175.stdout:541: writing smithi17557870-28 from 3058823 to 3523607 tid 3 2024-09-11T03:40:19.873 INFO:tasks.rados.rados.2.smithi175.stdout:542: write (excl) oid 41 current snap is 53 2024-09-11T03:40:19.873 INFO:tasks.rados.rados.2.smithi175.stdout:538: finishing copy_from racing read to smithi17557870-23 2024-09-11T03:40:19.874 INFO:tasks.rados.rados.2.smithi175.stdout:538: finishing copy_from to smithi17557870-23 2024-09-11T03:40:19.874 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 82 (ObjNum 171 snap 39 seq_num 171) dirty exists 2024-09-11T03:40:19.874 INFO:tasks.rados.rados.2.smithi175.stdout:541: finishing write tid 1 to smithi17557870-28 2024-09-11T03:40:19.874 INFO:tasks.rados.rados.2.smithi175.stdout:542: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2024-09-11T03:40:19.878 INFO:tasks.rados.rados.2.smithi175.stdout:542: writing smithi17557870-41 from 514347 to 918613 tid 1 2024-09-11T03:40:19.885 INFO:tasks.rados.rados.2.smithi175.stdout:542: writing smithi17557870-41 from 1499344 to 2199415 tid 2 2024-09-11T03:40:19.892 INFO:tasks.rados.rados.2.smithi175.stdout:542: writing smithi17557870-41 from 2686246 to 3362302 tid 3 2024-09-11T03:40:19.892 INFO:tasks.rados.rados.2.smithi175.stdout:538: done (2 left) 2024-09-11T03:40:19.892 INFO:tasks.rados.rados.2.smithi175.stdout:543: rollback oid 32 current snap is 53 2024-09-11T03:40:19.892 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 32 to 52 2024-09-11T03:40:19.893 INFO:tasks.rados.rados.2.smithi175.stdout:541: finishing write tid 2 to smithi17557870-28 2024-09-11T03:40:19.893 INFO:tasks.rados.rados.2.smithi175.stdout:541: finishing write tid 3 to smithi17557870-28 2024-09-11T03:40:19.893 INFO:tasks.rados.rados.2.smithi175.stdout:541: finishing write tid 4 to smithi17557870-28 2024-09-11T03:40:19.893 INFO:tasks.rados.rados.2.smithi175.stdout:541: finishing write tid 5 to smithi17557870-28 2024-09-11T03:40:19.893 INFO:tasks.rados.rados.2.smithi175.stdout:541: oid 28 updating version 0 to 188 2024-09-11T03:40:19.893 INFO:tasks.rados.rados.2.smithi175.stdout:541: oid 28 updating version 188 to 191 2024-09-11T03:40:19.893 INFO:tasks.rados.rados.2.smithi175.stdout:541: oid 28 version 191 is already newer than 190 2024-09-11T03:40:19.893 INFO:tasks.rados.rados.2.smithi175.stdout:541: oid 28 version 191 is already newer than 189 2024-09-11T03:40:19.893 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 191 (ObjNum 220 snap 53 seq_num 220) dirty exists 2024-09-11T03:40:19.893 INFO:tasks.rados.rados.2.smithi175.stdout:541: left oid 28 (ObjNum 220 snap 53 seq_num 220) 2024-09-11T03:40:19.893 INFO:tasks.rados.rados.2.smithi175.stdout:542: finishing write tid 1 to smithi17557870-41 2024-09-11T03:40:19.893 INFO:tasks.rados.rados.2.smithi175.stdout:542: finishing write tid 2 to smithi17557870-41 2024-09-11T03:40:19.893 INFO:tasks.rados.rados.2.smithi175.stdout:541: done (2 left) 2024-09-11T03:40:19.893 INFO:tasks.rados.rados.2.smithi175.stdout:544: write oid 42 current snap is 53 2024-09-11T03:40:19.894 INFO:tasks.rados.rados.2.smithi175.stdout:544: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2024-09-11T03:40:19.897 INFO:tasks.rados.rados.2.smithi175.stdout:544: writing smithi17557870-42 from 531154 to 1010669 tid 1 2024-09-11T03:40:19.903 INFO:tasks.rados.rados.2.smithi175.stdout:544: writing smithi17557870-42 from 1557826 to 2001555 tid 2 2024-09-11T03:40:19.910 INFO:tasks.rados.rados.2.smithi175.stdout:544: writing smithi17557870-42 from 2713669 to 3200997 tid 3 2024-09-11T03:40:19.913 INFO:tasks.rados.rados.2.smithi175.stdout:544: writing smithi17557870-42 from 3679295 to 3731154 tid 4 2024-09-11T03:40:19.913 INFO:tasks.rados.rados.2.smithi175.stdout:545: write oid 31 current snap is 53 2024-09-11T03:40:19.913 INFO:tasks.rados.rados.2.smithi175.stdout:542: finishing write tid 3 to smithi17557870-41 2024-09-11T03:40:19.913 INFO:tasks.rados.rados.2.smithi175.stdout:542: finishing write tid 4 to smithi17557870-41 2024-09-11T03:40:19.913 INFO:tasks.rados.rados.2.smithi175.stdout:542: finishing write tid 5 to smithi17557870-41 2024-09-11T03:40:19.914 INFO:tasks.rados.rados.2.smithi175.stdout:542: oid 41 updating version 0 to 129 2024-09-11T03:40:19.914 INFO:tasks.rados.rados.2.smithi175.stdout:542: oid 41 version 129 is already newer than 126 2024-09-11T03:40:19.914 INFO:tasks.rados.rados.2.smithi175.stdout:542: oid 41 version 129 is already newer than 127 2024-09-11T03:40:19.914 INFO:tasks.rados.rados.2.smithi175.stdout:542: oid 41 version 129 is already newer than 128 2024-09-11T03:40:19.914 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 129 (ObjNum 221 snap 53 seq_num 221) dirty exists 2024-09-11T03:40:19.914 INFO:tasks.rados.rados.2.smithi175.stdout:542: left oid 41 (ObjNum 221 snap 53 seq_num 221) 2024-09-11T03:40:19.914 INFO:tasks.rados.rados.2.smithi175.stdout:543: finishing rollback tid 0 to smithi17557870-32 2024-09-11T03:40:19.914 INFO:tasks.rados.rados.2.smithi175.stdout:543: finishing rollback tid 1 to smithi17557870-32 2024-09-11T03:40:19.914 INFO:tasks.rados.rados.2.smithi175.stdout:545: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2024-09-11T03:40:19.919 INFO:tasks.rados.rados.2.smithi175.stdout:545: writing smithi17557870-31 from 547961 to 1102725 tid 1 2024-09-11T03:40:19.925 INFO:tasks.rados.rados.2.smithi175.stdout:545: writing smithi17557870-31 from 1616308 to 2320048 tid 2 2024-09-11T03:40:19.932 INFO:tasks.rados.rados.2.smithi175.stdout:545: writing smithi17557870-31 from 2741092 to 3439692 tid 3 2024-09-11T03:40:19.932 INFO:tasks.rados.rados.2.smithi175.stdout:542: done (3 left) 2024-09-11T03:40:19.932 INFO:tasks.rados.rados.2.smithi175.stdout:546: rollback oid 46 current snap is 53 2024-09-11T03:40:19.933 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 46 to 41 2024-09-11T03:40:19.933 INFO:tasks.rados.rados.2.smithi175.stdout:543: finishing rollback tid 2 to smithi17557870-32 2024-09-11T03:40:19.933 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 133 (ObjNum 207 snap 50 seq_num 207) dirty exists 2024-09-11T03:40:19.933 INFO:tasks.rados.rados.2.smithi175.stdout:544: finishing write tid 1 to smithi17557870-42 2024-09-11T03:40:19.933 INFO:tasks.rados.rados.2.smithi175.stdout:545: finishing write tid 1 to smithi17557870-31 2024-09-11T03:40:19.933 INFO:tasks.rados.rados.2.smithi175.stdout:544: finishing write tid 2 to smithi17557870-42 2024-09-11T03:40:19.933 INFO:tasks.rados.rados.2.smithi175.stdout:544: finishing write tid 3 to smithi17557870-42 2024-09-11T03:40:19.933 INFO:tasks.rados.rados.2.smithi175.stdout:544: finishing write tid 4 to smithi17557870-42 2024-09-11T03:40:19.933 INFO:tasks.rados.rados.2.smithi175.stdout:544: finishing write tid 5 to smithi17557870-42 2024-09-11T03:40:19.933 INFO:tasks.rados.rados.2.smithi175.stdout:544: finishing write tid 6 to smithi17557870-42 2024-09-11T03:40:19.933 INFO:tasks.rados.rados.2.smithi175.stdout:544: oid 42 updating version 0 to 143 2024-09-11T03:40:19.933 INFO:tasks.rados.rados.2.smithi175.stdout:544: oid 42 version 143 is already newer than 141 2024-09-11T03:40:19.933 INFO:tasks.rados.rados.2.smithi175.stdout:544: oid 42 version 143 is already newer than 140 2024-09-11T03:40:19.933 INFO:tasks.rados.rados.2.smithi175.stdout:544: oid 42 version 143 is already newer than 142 2024-09-11T03:40:19.934 INFO:tasks.rados.rados.2.smithi175.stdout:544: oid 42 version 143 is already newer than 139 2024-09-11T03:40:19.934 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 143 (ObjNum 222 snap 53 seq_num 222) dirty exists 2024-09-11T03:40:19.934 INFO:tasks.rados.rados.2.smithi175.stdout:544: left oid 42 (ObjNum 222 snap 53 seq_num 222) 2024-09-11T03:40:19.934 INFO:tasks.rados.rados.2.smithi175.stdout:545: finishing write tid 2 to smithi17557870-31 2024-09-11T03:40:19.934 INFO:tasks.rados.rados.2.smithi175.stdout:543: done (3 left) 2024-09-11T03:40:19.934 INFO:tasks.rados.rados.2.smithi175.stdout:544: done (2 left) 2024-09-11T03:40:19.934 INFO:tasks.rados.rados.2.smithi175.stdout:547: write (excl) oid 7 current snap is 53 2024-09-11T03:40:19.934 INFO:tasks.rados.rados.2.smithi175.stdout:547: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2024-09-11T03:40:19.938 INFO:tasks.rados.rados.2.smithi175.stdout:547: writing smithi17557870-7 from 564768 to 1194781 tid 1 2024-09-11T03:40:19.943 INFO:tasks.rados.rados.2.smithi175.stdout:547: writing smithi17557870-7 from 1674790 to 2122188 tid 2 2024-09-11T03:40:19.951 INFO:tasks.rados.rados.2.smithi175.stdout:547: writing smithi17557870-7 from 2768515 to 3394740 tid 3 2024-09-11T03:40:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:548: read oid 20 snap -1 2024-09-11T03:40:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:548: expect (ObjNum 119 snap 18 seq_num 119) 2024-09-11T03:40:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:549: rollback oid 27 current snap is 53 2024-09-11T03:40:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 27 to 51 2024-09-11T03:40:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:546: finishing rollback tid 0 to smithi17557870-46 2024-09-11T03:40:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:546: finishing rollback tid 1 to smithi17557870-46 2024-09-11T03:40:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:546: finishing rollback tid 2 to smithi17557870-46 2024-09-11T03:40:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 41 (ObjNum 157 snap 35 seq_num 157) dirty exists 2024-09-11T03:40:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:545: finishing write tid 3 to smithi17557870-31 2024-09-11T03:40:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:545: finishing write tid 4 to smithi17557870-31 2024-09-11T03:40:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:545: finishing write tid 5 to smithi17557870-31 2024-09-11T03:40:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:545: oid 31 updating version 0 to 135 2024-09-11T03:40:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:545: oid 31 version 135 is already newer than 134 2024-09-11T03:40:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:545: oid 31 updating version 135 to 136 2024-09-11T03:40:19.953 INFO:tasks.rados.rados.2.smithi175.stdout:545: oid 31 updating version 136 to 137 2024-09-11T03:40:19.953 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 137 (ObjNum 223 snap 53 seq_num 223) dirty exists 2024-09-11T03:40:19.953 INFO:tasks.rados.rados.2.smithi175.stdout:545: left oid 31 (ObjNum 223 snap 53 seq_num 223) 2024-09-11T03:40:19.953 INFO:tasks.rados.rados.2.smithi175.stdout:547: finishing write tid 1 to smithi17557870-7 2024-09-11T03:40:19.953 INFO:tasks.rados.rados.2.smithi175.stdout:547: finishing write tid 2 to smithi17557870-7 2024-09-11T03:40:19.953 INFO:tasks.rados.rados.2.smithi175.stdout:545: done (4 left) 2024-09-11T03:40:19.953 INFO:tasks.rados.rados.2.smithi175.stdout:546: done (3 left) 2024-09-11T03:40:19.953 INFO:tasks.rados.rados.2.smithi175.stdout:550: write (excl) oid 25 current snap is 53 2024-09-11T03:40:19.953 INFO:tasks.rados.rados.2.smithi175.stdout:550: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2024-09-11T03:40:19.958 INFO:tasks.rados.rados.2.smithi175.stdout:550: writing smithi17557870-25 from 581575 to 1286837 tid 1 2024-09-11T03:40:19.962 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:19.964 INFO:tasks.rados.rados.2.smithi175.stdout:550: writing smithi17557870-25 from 1733272 to 2440681 tid 2 2024-09-11T03:40:19.972 INFO:tasks.rados.rados.2.smithi175.stdout:550: writing smithi17557870-25 from 3195938 to 3633435 tid 3 2024-09-11T03:40:19.972 INFO:tasks.rados.rados.2.smithi175.stdout:551: copy_from oid 14 from oid 22 current snap is 53 2024-09-11T03:40:19.972 INFO:tasks.rados.rados.2.smithi175.stdout:549: finishing rollback tid 0 to smithi17557870-27 2024-09-11T03:40:19.972 INFO:tasks.rados.rados.2.smithi175.stdout:549: finishing rollback tid 1 to smithi17557870-27 2024-09-11T03:40:19.973 INFO:tasks.rados.rados.2.smithi175.stdout:549: finishing rollback tid 2 to smithi17557870-27 2024-09-11T03:40:19.973 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 141 (ObjNum 186 snap 45 seq_num 186) dirty exists 2024-09-11T03:40:19.973 INFO:tasks.rados.rados.2.smithi175.stdout:548: expect (ObjNum 119 snap 18 seq_num 119) 2024-09-11T03:40:19.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:19.987 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:19.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:20.035 INFO:tasks.rados.rados.2.smithi175.stdout:547: finishing write tid 3 to smithi17557870-7 2024-09-11T03:40:20.035 INFO:tasks.rados.rados.2.smithi175.stdout:547: finishing write tid 4 to smithi17557870-7 2024-09-11T03:40:20.035 INFO:tasks.rados.rados.2.smithi175.stdout:547: finishing write tid 5 to smithi17557870-7 2024-09-11T03:40:20.035 INFO:tasks.rados.rados.2.smithi175.stdout:547: oid 7 updating version 0 to 116 2024-09-11T03:40:20.035 INFO:tasks.rados.rados.2.smithi175.stdout:547: oid 7 version 116 is already newer than 114 2024-09-11T03:40:20.035 INFO:tasks.rados.rados.2.smithi175.stdout:547: oid 7 version 116 is already newer than 115 2024-09-11T03:40:20.035 INFO:tasks.rados.rados.2.smithi175.stdout:547: oid 7 updating version 116 to 117 2024-09-11T03:40:20.036 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 117 (ObjNum 224 snap 53 seq_num 224) dirty exists 2024-09-11T03:40:20.036 INFO:tasks.rados.rados.2.smithi175.stdout:547: left oid 7 (ObjNum 224 snap 53 seq_num 224) 2024-09-11T03:40:20.036 INFO:tasks.rados.rados.2.smithi175.stdout:550: finishing write tid 1 to smithi17557870-25 2024-09-11T03:40:20.036 INFO:tasks.rados.rados.2.smithi175.stdout:547: done (4 left) 2024-09-11T03:40:20.036 INFO:tasks.rados.rados.2.smithi175.stdout:548: done (3 left) 2024-09-11T03:40:20.036 INFO:tasks.rados.rados.2.smithi175.stdout:549: done (2 left) 2024-09-11T03:40:20.036 INFO:tasks.rados.rados.2.smithi175.stdout:552: read oid 50 snap -1 2024-09-11T03:40:20.036 INFO:tasks.rados.rados.2.smithi175.stdout:552: expect deleted 2024-09-11T03:40:20.036 INFO:tasks.rados.rados.2.smithi175.stdout:553: delete oid 42 current snap is 53 2024-09-11T03:40:20.036 INFO:tasks.rados.rados.2.smithi175.stdout:550: finishing write tid 2 to smithi17557870-25 2024-09-11T03:40:20.036 INFO:tasks.rados.rados.2.smithi175.stdout:550: finishing write tid 3 to smithi17557870-25 2024-09-11T03:40:20.037 INFO:tasks.rados.rados.2.smithi175.stdout:550: finishing write tid 4 to smithi17557870-25 2024-09-11T03:40:20.037 INFO:tasks.rados.rados.2.smithi175.stdout:550: finishing write tid 5 to smithi17557870-25 2024-09-11T03:40:20.037 INFO:tasks.rados.rados.2.smithi175.stdout:550: oid 25 updating version 0 to 28 2024-09-11T03:40:20.037 INFO:tasks.rados.rados.2.smithi175.stdout:550: oid 25 updating version 28 to 29 2024-09-11T03:40:20.037 INFO:tasks.rados.rados.2.smithi175.stdout:550: oid 25 version 29 is already newer than 27 2024-09-11T03:40:20.037 INFO:tasks.rados.rados.2.smithi175.stdout:550: oid 25 version 29 is already newer than 26 2024-09-11T03:40:20.037 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 29 (ObjNum 225 snap 53 seq_num 225) dirty exists 2024-09-11T03:40:20.037 INFO:tasks.rados.rados.2.smithi175.stdout:550: left oid 25 (ObjNum 225 snap 53 seq_num 225) 2024-09-11T03:40:20.037 INFO:tasks.rados.rados.2.smithi175.stdout:551: finishing copy_from racing read to smithi17557870-14 2024-09-11T03:40:20.037 INFO:tasks.rados.rados.2.smithi175.stdout:551: finishing copy_from to smithi17557870-14 2024-09-11T03:40:20.037 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 159 (ObjNum 194 snap 47 seq_num 194) dirty exists 2024-09-11T03:40:20.038 INFO:tasks.rados.rados.2.smithi175.stdout:550: done (3 left) 2024-09-11T03:40:20.038 INFO:tasks.rados.rados.2.smithi175.stdout:551: done (2 left) 2024-09-11T03:40:20.038 INFO:tasks.rados.rados.2.smithi175.stdout:552: done (1 left) 2024-09-11T03:40:20.038 INFO:tasks.rados.rados.2.smithi175.stdout:553: done (0 left) 2024-09-11T03:40:20.038 INFO:tasks.rados.rados.2.smithi175.stdout:554: snap_remove snap 52 2024-09-11T03:40:20.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:20.099 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:40:20.100 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:40:20.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:19 smithi084 ceph-mon[27254]: pgmap v457: 27 pgs: 27 active+clean; 141 MiB data, 582 MiB used, 893 GiB / 894 GiB avail; 8.4 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-11T03:40:20.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:19 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:40:20.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:19 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:40:20.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:19 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:40:20.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:19 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:40:20.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:19 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1359790028' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:40:20.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:19 smithi084 ceph-mon[27254]: osdmap e202: 12 total, 12 up, 10 in 2024-09-11T03:40:20.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:40:20.101 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:40:20.101 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:40:20.109 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:20.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:20.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:20.116 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:40:20.116 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:40:20.132 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:40:20.132 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:40:20.148 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:40:20.148 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:40:20.165 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:40:20.165 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:40:20.166 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:19 smithi121 ceph-mon[30729]: pgmap v457: 27 pgs: 27 active+clean; 141 MiB data, 582 MiB used, 893 GiB / 894 GiB avail; 8.4 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-11T03:40:20.166 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:19 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:40:20.166 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:19 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:40:20.166 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:19 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:40:20.166 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:19 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:40:20.166 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:19 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1359790028' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:40:20.166 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:19 smithi121 ceph-mon[30729]: osdmap e202: 12 total, 12 up, 10 in 2024-09-11T03:40:20.167 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:40:20.167 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:40:20.167 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:40:20.182 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:40:20.183 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:40:20.198 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:40:20.198 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:40:20.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:20.215 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:40:20.215 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:40:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:19 smithi175 ceph-mon[30894]: pgmap v457: 27 pgs: 27 active+clean; 141 MiB data, 582 MiB used, 893 GiB / 894 GiB avail; 8.4 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-11T03:40:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:19 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:40:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:19 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:40:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:19 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:40:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:19 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:40:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:19 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1359790028' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:40:20.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:19 smithi175 ceph-mon[30894]: osdmap e202: 12 total, 12 up, 10 in 2024-09-11T03:40:20.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:40:20.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:40:20.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:40:20.232 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:40:20.232 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:40:20.233 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:20.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:20.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:20.248 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:40:20.248 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:40:20.265 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:40:20.265 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:40:20.281 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:40:20.281 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:40:20.305 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T03:40:20.316 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [6, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:40:20.316 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:40:20.317 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-11T03:40:20.317 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:40:20.335 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:20.369 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:20.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:20.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:20.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:20.492 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:20.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:20.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:20.549 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:40:20.594 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:20.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:20.617 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:20.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:20.718 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:20.742 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:20.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:20.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:20.844 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:20.863 INFO:tasks.rados.rados.2.smithi175.stdout:554: done (0 left) 2024-09-11T03:40:20.863 INFO:tasks.rados.rados.2.smithi175.stdout:555: copy_from oid 24 from oid 26 current snap is 53 2024-09-11T03:40:20.863 INFO:tasks.rados.rados.2.smithi175.stdout:556: snap_create 2024-09-11T03:40:20.869 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:20.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:20.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:20.872 INFO:tasks.rados.rados.2.smithi175.stdout:555: finishing copy_from racing read to smithi17557870-24 2024-09-11T03:40:20.875 INFO:tasks.rados.rados.2.smithi175.stdout:555: finishing copy_from to smithi17557870-24 2024-09-11T03:40:20.875 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 75 (ObjNum 187 snap 45 seq_num 187) dirty exists 2024-09-11T03:40:20.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:20 smithi121 ceph-mon[30729]: osdmap e203: 12 total, 12 up, 10 in 2024-09-11T03:40:20.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:20 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1689612442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:20.938 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:20 smithi084 ceph-mon[27254]: osdmap e203: 12 total, 12 up, 10 in 2024-09-11T03:40:20.939 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:20 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1689612442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:20.971 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:20.992 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:20.993 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:20.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:21.007 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:40:21.007 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:40:21.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:40:21.007 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:40:21.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.008 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPull(1.0 e40/39 cost 1049576)", 2024-09-11T03:40:21.008 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:34:57.777569+0000", 2024-09-11T03:40:21.008 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 323.22204506600002, 2024-09-11T03:40:21.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026267769999999998, 2024-09-11T03:40:21.008 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.008 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.008 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-11T03:40:21.008 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:34:57.777569+0000", 2024-09-11T03:40:21.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:34:57.777569+0000", 2024-09-11T03:40:21.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:34:57.777570+0000", 2024-09-11T03:40:21.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4199999999999995e-07 2024-09-11T03:40:21.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:34:57.777578+0000", 2024-09-11T03:40:21.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7000000000000008e-06 2024-09-11T03:40:21.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:34:57.777587+0000", 2024-09-11T03:40:21.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6909999999999992e-06 2024-09-11T03:40:21.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:34:57.777594+0000", 2024-09-11T03:40:21.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.511e-06 2024-09-11T03:40:21.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:34:57.777638+0000", 2024-09-11T03:40:21.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4165000000000002e-05 2024-09-11T03:40:21.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:34:57.780196+0000", 2024-09-11T03:40:21.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025580680000000001 2024-09-11T03:40:21.011 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.011 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.012 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.012 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:243 2.d e107/95 2:b7bdfe52:::smithi17557870-10:head v 107'17, mlcod=102'14)", 2024-09-11T03:40:21.012 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:43.974889+0000", 2024-09-11T03:40:21.012 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.024725744999998, 2024-09-11T03:40:21.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002116897, 2024-09-11T03:40:21.012 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.012 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.012 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.012 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:43.974889+0000", 2024-09-11T03:40:21.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:43.974889+0000", 2024-09-11T03:40:21.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:43.974889+0000", 2024-09-11T03:40:21.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8299999999999996e-07 2024-09-11T03:40:21.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:43.975881+0000", 2024-09-11T03:40:21.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00099154100000000008 2024-09-11T03:40:21.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:43.975894+0000", 2024-09-11T03:40:21.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3363000000000001e-05 2024-09-11T03:40:21.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:43.975905+0000", 2024-09-11T03:40:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0276e-05 2024-09-11T03:40:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:43.975959+0000", 2024-09-11T03:40:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4756000000000002e-05 2024-09-11T03:40:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:43.975977+0000", 2024-09-11T03:40:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7447000000000001e-05 2024-09-11T03:40:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:43.976955+0000", 2024-09-11T03:40:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00097843800000000009 2024-09-11T03:40:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:43.977006+0000", 2024-09-11T03:40:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0392999999999997e-05 2024-09-11T03:40:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:359 2.5 e114/95 2:a6d17fc0:::smithi17557870-13:head v 114'40, mlcod=110'38)", 2024-09-11T03:40:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:50.842723+0000", 2024-09-11T03:40:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 90.156891590000001, 2024-09-11T03:40:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028384259999999998, 2024-09-11T03:40:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.842723+0000", 2024-09-11T03:40:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.842723+0000", 2024-09-11T03:40:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.842725+0000", 2024-09-11T03:40:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0600000000000002e-06 2024-09-11T03:40:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.844365+0000", 2024-09-11T03:40:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016401619999999999 2024-09-11T03:40:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.844377+0000", 2024-09-11T03:40:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1908e-05 2024-09-11T03:40:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.844382+0000", 2024-09-11T03:40:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2429999999999996e-06 2024-09-11T03:40:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.844476+0000", 2024-09-11T03:40:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3325000000000005e-05 2024-09-11T03:40:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.844492+0000", 2024-09-11T03:40:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.609e-05 2024-09-11T03:40:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.845527+0000", 2024-09-11T03:40:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010352110000000001 2024-09-11T03:40:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.845561+0000", 2024-09-11T03:40:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4427000000000001e-05 2024-09-11T03:40:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:360 2.5 e114/95 2:a6d17fc0:::smithi17557870-13:head v 114'41, mlcod=112'39)", 2024-09-11T03:40:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:50.850669+0000", 2024-09-11T03:40:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 90.148945961999999, 2024-09-11T03:40:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021434679999999999, 2024-09-11T03:40:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.850669+0000", 2024-09-11T03:40:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.850669+0000", 2024-09-11T03:40:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.850671+0000", 2024-09-11T03:40:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0899999999999999e-06 2024-09-11T03:40:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.851812+0000", 2024-09-11T03:40:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011413040000000001 2024-09-11T03:40:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.851851+0000", 2024-09-11T03:40:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9310000000000001e-05 2024-09-11T03:40:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.851857+0000", 2024-09-11T03:40:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7579999999999998e-06 2024-09-11T03:40:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.851916+0000", 2024-09-11T03:40:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9410999999999998e-05 2024-09-11T03:40:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.851931+0000", 2024-09-11T03:40:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4968e-05 2024-09-11T03:40:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.852780+0000", 2024-09-11T03:40:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00084874400000000002 2024-09-11T03:40:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.852812+0000", 2024-09-11T03:40:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1883e-05 2024-09-11T03:40:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:372 2.f e114/95 2:fba9fb28:::smithi17557870-48:head v 114'16, mlcod=114'14)", 2024-09-11T03:40:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:50.881512+0000", 2024-09-11T03:40:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 90.118102081999993, 2024-09-11T03:40:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045150010000000003, 2024-09-11T03:40:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.881512+0000", 2024-09-11T03:40:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.881512+0000", 2024-09-11T03:40:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.881515+0000", 2024-09-11T03:40:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1940000000000001e-06 2024-09-11T03:40:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.885161+0000", 2024-09-11T03:40:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036461419999999998 2024-09-11T03:40:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.885172+0000", 2024-09-11T03:40:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0808000000000001e-05 2024-09-11T03:40:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.885198+0000", 2024-09-11T03:40:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6502000000000001e-05 2024-09-11T03:40:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.885249+0000", 2024-09-11T03:40:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0655999999999997e-05 2024-09-11T03:40:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.885263+0000", 2024-09-11T03:40:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3890000000000001e-05 2024-09-11T03:40:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.885983+0000", 2024-09-11T03:40:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072063800000000001 2024-09-11T03:40:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.886027+0000", 2024-09-11T03:40:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4171000000000001e-05 2024-09-11T03:40:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:394 2.5 e114/95 2:a6d17fc0:::smithi17557870-13:head v 114'43, mlcod=114'41)", 2024-09-11T03:40:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:51.075253+0000", 2024-09-11T03:40:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 89.924361558000001, 2024-09-11T03:40:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024089340000000002, 2024-09-11T03:40:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.075253+0000", 2024-09-11T03:40:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.075253+0000", 2024-09-11T03:40:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.075256+0000", 2024-09-11T03:40:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1269999999999999e-06 2024-09-11T03:40:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.076506+0000", 2024-09-11T03:40:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001250255 2024-09-11T03:40:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.076531+0000", 2024-09-11T03:40:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4389e-05 2024-09-11T03:40:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.076541+0000", 2024-09-11T03:40:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0693000000000001e-05 2024-09-11T03:40:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.076645+0000", 2024-09-11T03:40:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103959 2024-09-11T03:40:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.076667+0000", 2024-09-11T03:40:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1403000000000002e-05 2024-09-11T03:40:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.077610+0000", 2024-09-11T03:40:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00094286999999999995 2024-09-11T03:40:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.077662+0000", 2024-09-11T03:40:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2237999999999997e-05 2024-09-11T03:40:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:396 2.5 e114/95 2:a6d17fc0:::smithi17557870-13:head v 114'45, mlcod=114'43)", 2024-09-11T03:40:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:51.086142+0000", 2024-09-11T03:40:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 89.913472001000002, 2024-09-11T03:40:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025332610000000002, 2024-09-11T03:40:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.086142+0000", 2024-09-11T03:40:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.086142+0000", 2024-09-11T03:40:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.086144+0000", 2024-09-11T03:40:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0410000000000001e-06 2024-09-11T03:40:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.087832+0000", 2024-09-11T03:40:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016887289999999999 2024-09-11T03:40:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.087846+0000", 2024-09-11T03:40:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3827e-05 2024-09-11T03:40:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.087852+0000", 2024-09-11T03:40:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6210000000000001e-06 2024-09-11T03:40:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.087911+0000", 2024-09-11T03:40:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9551e-05 2024-09-11T03:40:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.087936+0000", 2024-09-11T03:40:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4607999999999999e-05 2024-09-11T03:40:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.088641+0000", 2024-09-11T03:40:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070496199999999995 2024-09-11T03:40:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.088676+0000", 2024-09-11T03:40:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4922000000000002e-05 2024-09-11T03:40:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:545 2.5 e125/95 2:a6e3bbd9:::smithi17557870-38:head v 125'57, mlcod=124'55)", 2024-09-11T03:40:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:01.822591+0000", 2024-09-11T03:40:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.177023507000001, 2024-09-11T03:40:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019860500000000001, 2024-09-11T03:40:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.822591+0000", 2024-09-11T03:40:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.822591+0000", 2024-09-11T03:40:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.822592+0000", 2024-09-11T03:40:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1699999999999997e-07 2024-09-11T03:40:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.823712+0000", 2024-09-11T03:40:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011198849999999999 2024-09-11T03:40:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.823721+0000", 2024-09-11T03:40:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0250000000000008e-06 2024-09-11T03:40:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.823728+0000", 2024-09-11T03:40:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2899999999999997e-06 2024-09-11T03:40:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.823835+0000", 2024-09-11T03:40:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106655 2024-09-11T03:40:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.823860+0000", 2024-09-11T03:40:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.516e-05 2024-09-11T03:40:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.824541+0000", 2024-09-11T03:40:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00068172899999999995 2024-09-11T03:40:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.824577+0000", 2024-09-11T03:40:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5589000000000001e-05 2024-09-11T03:40:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:547 2.5 e125/95 2:a6e3bbd9:::smithi17557870-38:head v 125'59, mlcod=125'57)", 2024-09-11T03:40:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:01.835034+0000", 2024-09-11T03:40:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.164580510999997, 2024-09-11T03:40:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001883774, 2024-09-11T03:40:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.835034+0000", 2024-09-11T03:40:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.835034+0000", 2024-09-11T03:40:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.835036+0000", 2024-09-11T03:40:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-09-11T03:40:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.835920+0000", 2024-09-11T03:40:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00088384499999999996 2024-09-11T03:40:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.835934+0000", 2024-09-11T03:40:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4599e-05 2024-09-11T03:40:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.835943+0000", 2024-09-11T03:40:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6349999999999995e-06 2024-09-11T03:40:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.836008+0000", 2024-09-11T03:40:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5198999999999997e-05 2024-09-11T03:40:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.836031+0000", 2024-09-11T03:40:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2399e-05 2024-09-11T03:40:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.836881+0000", 2024-09-11T03:40:21.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00085055999999999999 2024-09-11T03:40:21.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:01.836918+0000", 2024-09-11T03:40:21.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6653000000000003e-05 2024-09-11T03:40:21.049 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.049 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.049 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.050 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:578 2.5 e128/95 2:a6e3bbd9:::smithi17557870-38:head v 128'62, mlcod=125'59)", 2024-09-11T03:40:21.050 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:04.863266+0000", 2024-09-11T03:40:21.050 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.136348631000004, 2024-09-11T03:40:21.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033614750000000001, 2024-09-11T03:40:21.050 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.050 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.050 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.050 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.863266+0000", 2024-09-11T03:40:21.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.863266+0000", 2024-09-11T03:40:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.863266+0000", 2024-09-11T03:40:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8300000000000001e-07 2024-09-11T03:40:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.865143+0000", 2024-09-11T03:40:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018766639999999999 2024-09-11T03:40:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.865163+0000", 2024-09-11T03:40:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0182000000000001e-05 2024-09-11T03:40:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.865173+0000", 2024-09-11T03:40:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0049e-05 2024-09-11T03:40:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.865280+0000", 2024-09-11T03:40:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010726999999999999 2024-09-11T03:40:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.865304+0000", 2024-09-11T03:40:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3416000000000001e-05 2024-09-11T03:40:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.866589+0000", 2024-09-11T03:40:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001285212 2024-09-11T03:40:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.866627+0000", 2024-09-11T03:40:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8498999999999999e-05 2024-09-11T03:40:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:579 2.5 e128/95 2:a6e3bbd9:::smithi17557870-38:head v 128'63, mlcod=125'60)", 2024-09-11T03:40:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:04.868962+0000", 2024-09-11T03:40:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.130652338999994, 2024-09-11T03:40:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019824719999999999, 2024-09-11T03:40:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.868962+0000", 2024-09-11T03:40:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.868962+0000", 2024-09-11T03:40:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.868969+0000", 2024-09-11T03:40:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1470000000000004e-06 2024-09-11T03:40:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.870118+0000", 2024-09-11T03:40:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011488480000000001 2024-09-11T03:40:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.870129+0000", 2024-09-11T03:40:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1117e-05 2024-09-11T03:40:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.870135+0000", 2024-09-11T03:40:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2330000000000003e-06 2024-09-11T03:40:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.870205+0000", 2024-09-11T03:40:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9264999999999995e-05 2024-09-11T03:40:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.870224+0000", 2024-09-11T03:40:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.95e-05 2024-09-11T03:40:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.870914+0000", 2024-09-11T03:40:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00069016200000000002 2024-09-11T03:40:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.870945+0000", 2024-09-11T03:40:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0199999999999999e-05 2024-09-11T03:40:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:588 2.f e129/95 2:fda69313:::smithi17557870-16:head v 129'32, mlcod=124'28)", 2024-09-11T03:40:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:05.847680+0000", 2024-09-11T03:40:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 75.151934183999998, 2024-09-11T03:40:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075524320000000004, 2024-09-11T03:40:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.847680+0000", 2024-09-11T03:40:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.847680+0000", 2024-09-11T03:40:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.847681+0000", 2024-09-11T03:40:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.184e-06 2024-09-11T03:40:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.853139+0000", 2024-09-11T03:40:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054575960000000003 2024-09-11T03:40:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.853154+0000", 2024-09-11T03:40:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5285999999999999e-05 2024-09-11T03:40:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.853179+0000", 2024-09-11T03:40:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4380999999999999e-05 2024-09-11T03:40:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.853234+0000", 2024-09-11T03:40:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5164999999999998e-05 2024-09-11T03:40:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.853250+0000", 2024-09-11T03:40:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6541999999999998e-05 2024-09-11T03:40:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.855165+0000", 2024-09-11T03:40:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019147560000000001 2024-09-11T03:40:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.855233+0000", 2024-09-11T03:40:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7521999999999995e-05 2024-09-11T03:40:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:616 2.f e130/95 2:fba9fb28:::smithi17557870-48:head v 130'39, mlcod=130'37)", 2024-09-11T03:40:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:07.067189+0000", 2024-09-11T03:40:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.932425791, 2024-09-11T03:40:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002533141, 2024-09-11T03:40:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.067189+0000", 2024-09-11T03:40:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.067189+0000", 2024-09-11T03:40:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.067190+0000", 2024-09-11T03:40:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3570000000000001e-06 2024-09-11T03:40:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.068532+0000", 2024-09-11T03:40:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013415759999999999 2024-09-11T03:40:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.068552+0000", 2024-09-11T03:40:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0550000000000001e-05 2024-09-11T03:40:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.068564+0000", 2024-09-11T03:40:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1717e-05 2024-09-11T03:40:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.068661+0000", 2024-09-11T03:40:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7182999999999994e-05 2024-09-11T03:40:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.068686+0000", 2024-09-11T03:40:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4949000000000001e-05 2024-09-11T03:40:21.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.069657+0000", 2024-09-11T03:40:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00097101499999999999 2024-09-11T03:40:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.069722+0000", 2024-09-11T03:40:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4794000000000003e-05 2024-09-11T03:40:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:760 2.f e140/95 2:fda69313:::smithi17557870-16:head v 140'45, mlcod=131'42)", 2024-09-11T03:40:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:16.827550+0000", 2024-09-11T03:40:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.172063977999997, 2024-09-11T03:40:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002611403, 2024-09-11T03:40:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:16.827550+0000", 2024-09-11T03:40:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:16.827550+0000", 2024-09-11T03:40:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:16.827551+0000", 2024-09-11T03:40:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8400000000000001e-07 2024-09-11T03:40:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:16.829185+0000", 2024-09-11T03:40:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016341719999999999 2024-09-11T03:40:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:16.829194+0000", 2024-09-11T03:40:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9290000000000004e-06 2024-09-11T03:40:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:16.829203+0000", 2024-09-11T03:40:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8610000000000004e-06 2024-09-11T03:40:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:16.829249+0000", 2024-09-11T03:40:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5939000000000002e-05 2024-09-11T03:40:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:16.829264+0000", 2024-09-11T03:40:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5373999999999999e-05 2024-09-11T03:40:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:16.830129+0000", 2024-09-11T03:40:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00086500399999999995 2024-09-11T03:40:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:16.830162+0000", 2024-09-11T03:40:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.294e-05 2024-09-11T03:40:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:847 2.d e148/95 2:b1c0f730:::smithi17557870-20:head v 148'82, mlcod=136'79)", 2024-09-11T03:40:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:24.846640+0000", 2024-09-11T03:40:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 56.152974104999998, 2024-09-11T03:40:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053917130000000002, 2024-09-11T03:40:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.846640+0000", 2024-09-11T03:40:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.846640+0000", 2024-09-11T03:40:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.846641+0000", 2024-09-11T03:40:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3500000000000002e-07 2024-09-11T03:40:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.850322+0000", 2024-09-11T03:40:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036810390000000001 2024-09-11T03:40:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.850336+0000", 2024-09-11T03:40:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4688999999999999e-05 2024-09-11T03:40:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.850345+0000", 2024-09-11T03:40:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3010000000000007e-06 2024-09-11T03:40:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.850409+0000", 2024-09-11T03:40:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4367000000000002e-05 2024-09-11T03:40:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.850434+0000", 2024-09-11T03:40:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4589000000000001e-05 2024-09-11T03:40:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.851980+0000", 2024-09-11T03:40:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001546513 2024-09-11T03:40:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.852032+0000", 2024-09-11T03:40:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1879999999999998e-05 2024-09-11T03:40:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1059 2.f e175/95 2:fba9fb28:::smithi17557870-48:head v 175'79, mlcod=172'75)", 2024-09-11T03:40:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.851902+0000", 2024-09-11T03:40:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.14771206, 2024-09-11T03:40:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023845289999999998, 2024-09-11T03:40:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.851902+0000", 2024-09-11T03:40:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.851902+0000", 2024-09-11T03:40:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.851903+0000", 2024-09-11T03:40:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7100000000000001e-07 2024-09-11T03:40:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.853088+0000", 2024-09-11T03:40:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001185771 2024-09-11T03:40:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.853101+0000", 2024-09-11T03:40:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2446000000000001e-05 2024-09-11T03:40:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.853107+0000", 2024-09-11T03:40:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1770000000000003e-06 2024-09-11T03:40:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.853147+0000", 2024-09-11T03:40:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0160000000000002e-05 2024-09-11T03:40:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.853163+0000", 2024-09-11T03:40:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6067e-05 2024-09-11T03:40:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.854240+0000", 2024-09-11T03:40:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001077129 2024-09-11T03:40:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.854287+0000", 2024-09-11T03:40:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6607999999999998e-05 2024-09-11T03:40:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1181 2.5 e186/95 2:a6d17fc0:::smithi17557870-13:head v 186'151, mlcod=182'148)", 2024-09-11T03:40:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:02.962697+0000", 2024-09-11T03:40:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.036917316, 2024-09-11T03:40:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002036534, 2024-09-11T03:40:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.962697+0000", 2024-09-11T03:40:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.962697+0000", 2024-09-11T03:40:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.962698+0000", 2024-09-11T03:40:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1200000000000002e-07 2024-09-11T03:40:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.963933+0000", 2024-09-11T03:40:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001235112 2024-09-11T03:40:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.963945+0000", 2024-09-11T03:40:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1756999999999999e-05 2024-09-11T03:40:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.963950+0000", 2024-09-11T03:40:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6439999999999997e-06 2024-09-11T03:40:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.964022+0000", 2024-09-11T03:40:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1888e-05 2024-09-11T03:40:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.964038+0000", 2024-09-11T03:40:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5159e-05 2024-09-11T03:40:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.964697+0000", 2024-09-11T03:40:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00065979500000000002 2024-09-11T03:40:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.964734+0000", 2024-09-11T03:40:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6366999999999999e-05 2024-09-11T03:40:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1182 2.5 e186/95 2:a6d17fc0:::smithi17557870-13:head v 186'152, mlcod=186'150)", 2024-09-11T03:40:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:02.970285+0000", 2024-09-11T03:40:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.029329257000001, 2024-09-11T03:40:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022652280000000002, 2024-09-11T03:40:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.970285+0000", 2024-09-11T03:40:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.970285+0000", 2024-09-11T03:40:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.970286+0000", 2024-09-11T03:40:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.23e-07 2024-09-11T03:40:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.971601+0000", 2024-09-11T03:40:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013151599999999999 2024-09-11T03:40:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.971613+0000", 2024-09-11T03:40:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1326e-05 2024-09-11T03:40:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.971620+0000", 2024-09-11T03:40:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0539999999999997e-06 2024-09-11T03:40:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.971686+0000", 2024-09-11T03:40:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6602999999999999e-05 2024-09-11T03:40:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.971708+0000", 2024-09-11T03:40:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1569999999999998e-05 2024-09-11T03:40:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.972510+0000", 2024-09-11T03:40:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00080264099999999999 2024-09-11T03:40:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:02.972550+0000", 2024-09-11T03:40:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0051e-05 2024-09-11T03:40:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1251 2.7 e194/191 2:e0ec5763:::smithi17557870-2:head v 194'110, mlcod=186'107)", 2024-09-11T03:40:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:10.994443+0000", 2024-09-11T03:40:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.005170994, 2024-09-11T03:40:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002088557, 2024-09-11T03:40:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:10.994443+0000", 2024-09-11T03:40:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:10.994443+0000", 2024-09-11T03:40:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:10.994444+0000", 2024-09-11T03:40:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4000000000000002e-07 2024-09-11T03:40:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:10.995256+0000", 2024-09-11T03:40:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000811972 2024-09-11T03:40:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:10.995268+0000", 2024-09-11T03:40:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2187e-05 2024-09-11T03:40:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:10.995278+0000", 2024-09-11T03:40:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5820000000000008e-06 2024-09-11T03:40:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:10.995360+0000", 2024-09-11T03:40:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2550000000000001e-05 2024-09-11T03:40:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:10.995381+0000", 2024-09-11T03:40:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0520999999999999e-05 2024-09-11T03:40:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:10.996491+0000", 2024-09-11T03:40:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00111049 2024-09-11T03:40:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:10.996532+0000", 2024-09-11T03:40:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0714999999999999e-05 2024-09-11T03:40:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1290 2.d e196/95 2:b1c0f730:::smithi17557870-20:head v 196'156, mlcod=192'152)", 2024-09-11T03:40:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:12.850226+0000", 2024-09-11T03:40:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.1493880769999993, 2024-09-11T03:40:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054838100000000004, 2024-09-11T03:40:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.850226+0000", 2024-09-11T03:40:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.850226+0000", 2024-09-11T03:40:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.850227+0000", 2024-09-11T03:40:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7700000000000004e-07 2024-09-11T03:40:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.853952+0000", 2024-09-11T03:40:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003724757 2024-09-11T03:40:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.853966+0000", 2024-09-11T03:40:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4469e-05 2024-09-11T03:40:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.853973+0000", 2024-09-11T03:40:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6669999999999999e-06 2024-09-11T03:40:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.854041+0000", 2024-09-11T03:40:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7669999999999994e-05 2024-09-11T03:40:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.854067+0000", 2024-09-11T03:40:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6363000000000002e-05 2024-09-11T03:40:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.855661+0000", 2024-09-11T03:40:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001594088 2024-09-11T03:40:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.855710+0000", 2024-09-11T03:40:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9119000000000002e-05 2024-09-11T03:40:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:21.103 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:40:21.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:21.124 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:21.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:21.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:21.226 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:20 smithi175 ceph-mon[30894]: osdmap e203: 12 total, 12 up, 10 in 2024-09-11T03:40:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:20 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1689612442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:21.252 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:21.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:21.254 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:21.354 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:21.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:21.377 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:21.378 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:21.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:21.491 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:40:21.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:21.501 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:21.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:21.601 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:21.623 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:21.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:21.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:21.659 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:40:21.660 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:40:21.724 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:21.749 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:21.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:21.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:21.836 INFO:tasks.rados.rados.2.smithi175.stdout:555: done (1 left) 2024-09-11T03:40:21.836 INFO:tasks.rados.rados.2.smithi175.stdout:556: done (0 left) 2024-09-11T03:40:21.837 INFO:tasks.rados.rados.2.smithi175.stdout:557: snap_remove snap 41 2024-09-11T03:40:21.851 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:21.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T03:40:21.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:21.875 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:21.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:21.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:21.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:21.998 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:21.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:22.099 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:40:22.121 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:22.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:22.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:22.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:21 smithi084 ceph-mon[27254]: pgmap v460: 27 pgs: 27 active+clean; 141 MiB data, 584 MiB used, 893 GiB / 894 GiB avail; 255 B/s rd, 363 KiB/s wr, 2 op/s 2024-09-11T03:40:22.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:21 smithi084 ceph-mon[27254]: osdmap e204: 12 total, 12 up, 10 in 2024-09-11T03:40:22.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:21 smithi084 ceph-mon[27254]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:40:22.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:21 smithi084 ceph-mon[27254]: osdmap e205: 12 total, 12 up, 10 in 2024-09-11T03:40:22.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:21 smithi121 ceph-mon[30729]: pgmap v460: 27 pgs: 27 active+clean; 141 MiB data, 584 MiB used, 893 GiB / 894 GiB avail; 255 B/s rd, 363 KiB/s wr, 2 op/s 2024-09-11T03:40:22.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:21 smithi121 ceph-mon[30729]: osdmap e204: 12 total, 12 up, 10 in 2024-09-11T03:40:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:21 smithi121 ceph-mon[30729]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:40:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:21 smithi121 ceph-mon[30729]: osdmap e205: 12 total, 12 up, 10 in 2024-09-11T03:40:22.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:21 smithi175 ceph-mon[30894]: pgmap v460: 27 pgs: 27 active+clean; 141 MiB data, 584 MiB used, 893 GiB / 894 GiB avail; 255 B/s rd, 363 KiB/s wr, 2 op/s 2024-09-11T03:40:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:21 smithi175 ceph-mon[30894]: osdmap e204: 12 total, 12 up, 10 in 2024-09-11T03:40:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:21 smithi175 ceph-mon[30894]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:40:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:21 smithi175 ceph-mon[30894]: osdmap e205: 12 total, 12 up, 10 in 2024-09-11T03:40:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:22.245 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:22.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:22.344 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-09-11T03:40:20.853494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":147888328,"num_objects":73,"num_object_clones":27,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":10,"num_read":959,"num_read_kb":709439,"num_write":879,"num_write_kb":333056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14297451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2673,"ondisk_log_size":2673,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":598168,"kb_used_data":290416,"kb_used_omap":15,"kb_used_meta":307504,"kb_avail":936853352,"statfs":{"total":959950356480,"available":959337832448,"internally_reserved":0,"allocated":297385984,"data_stored":314939870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":314884533},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3700702,"num_objects":-3,"num_object_clones":-3,"num_object_copies":-9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":7,"num_write_kb":1452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019879"},"pg_stats":[{"pgid":"2.f","version":"198'105","reported_seq":627,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.854993+0000","last_change":"2024-09-11T03:40:14.842541+0000","last_active":"2024-09-11T03:40:15.854993+0000","last_peered":"2024-09-11T03:40:15.854993+0000","last_clean":"2024-09-11T03:40:15.854993+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:40:15.854993+0000","last_undegraded":"2024-09-11T03:40:15.854993+0000","last_fullsized":"2024-09-11T03:40:15.854993+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'82","last_scrub_stamp":"2024-09-11T03:39:53.261846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:53.261846+0000","objects_scrubbed":4,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:16.320084+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0057306739999999998,"stat_sum":{"num_bytes":7815255,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":30,"num_read_kb":6170,"num_write":52,"num_write_kb":18140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"28","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"4"},{"start":"33","length":"1"}]},{"pgid":"2.e","version":"198'141","reported_seq":681,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-11T03:40:17.833741+0000","last_change":"2024-09-11T03:40:17.833925+0000","last_active":"2024-09-11T03:40:17.833741+0000","last_peered":"2024-09-11T03:40:17.833741+0000","last_clean":"2024-09-11T03:40:17.833741+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:40:17.833741+0000","last_undegraded":"2024-09-11T03:40:17.833741+0000","last_fullsized":"2024-09-11T03:40:17.833741+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'103","last_scrub_stamp":"2024-09-11T03:39:55.201092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:55.201092+0000","objects_scrubbed":2,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:20.728228+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00045383799999999997,"stat_sum":{"num_bytes":10860789,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":96,"num_read_kb":83452,"num_write":46,"num_write_kb":21913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2c","length":"1"},{"start":"30","length":"1"},{"start":"33","length":"2"}]},{"pgid":"2.d","version":"198'162","reported_seq":697,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-11T03:40:18.137916+0000","last_change":"2024-09-11T03:40:18.138076+0000","last_active":"2024-09-11T03:40:18.137916+0000","last_peered":"2024-09-11T03:40:18.137916+0000","last_clean":"2024-09-11T03:40:18.137916+0000","last_became_active":"2024-09-11T03:38:30.031497+0000","last_became_peered":"2024-09-11T03:38:30.031497+0000","last_unstale":"2024-09-11T03:40:18.137916+0000","last_undegraded":"2024-09-11T03:40:18.137916+0000","last_fullsized":"2024-09-11T03:40:18.137916+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'138","last_scrub_stamp":"2024-09-11T03:39:52.253596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:52.253596+0000","objects_scrubbed":3,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:59.580347+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00036993799999999999,"stat_sum":{"num_bytes":13141506,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":83,"num_read_kb":60635,"num_write":69,"num_write_kb":24790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2c","length":"1"},{"start":"30","length":"1"},{"start":"33","length":"2"}]},{"pgid":"2.c","version":"198'98","reported_seq":638,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.839513+0000","last_change":"2024-09-11T03:40:14.842992+0000","last_active":"2024-09-11T03:40:15.839513+0000","last_peered":"2024-09-11T03:40:15.839513+0000","last_clean":"2024-09-11T03:40:15.839513+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:40:15.839513+0000","last_undegraded":"2024-09-11T03:40:15.839513+0000","last_fullsized":"2024-09-11T03:40:15.839513+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'52","last_scrub_stamp":"2024-09-11T03:39:56.458534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:56.458534+0000","objects_scrubbed":6,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:17.111357+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0044101920000000003,"stat_sum":{"num_bytes":7949645,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":93,"num_read_kb":79607,"num_write":56,"num_write_kb":25522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"28","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"4"},{"start":"33","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":375,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.856875+0000","last_change":"2024-09-11T03:39:58.567923+0000","last_active":"2024-09-11T03:40:15.856875+0000","last_peered":"2024-09-11T03:40:15.856875+0000","last_clean":"2024-09-11T03:40:15.856875+0000","last_became_active":"2024-09-11T03:39:57.836928+0000","last_became_peered":"2024-09-11T03:39:57.836928+0000","last_unstale":"2024-09-11T03:40:15.856875+0000","last_undegraded":"2024-09-11T03:40:15.856875+0000","last_fullsized":"2024-09-11T03:40:15.856875+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:39:58.567842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:39:58.567842+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:05.438976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"198'70","reported_seq":594,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.901099+0000","last_change":"2024-09-11T03:40:14.866158+0000","last_active":"2024-09-11T03:40:15.901099+0000","last_peered":"2024-09-11T03:40:15.901099+0000","last_clean":"2024-09-11T03:40:15.901099+0000","last_became_active":"2024-09-11T03:38:29.889091+0000","last_became_peered":"2024-09-11T03:38:29.889091+0000","last_unstale":"2024-09-11T03:40:15.901099+0000","last_undegraded":"2024-09-11T03:40:15.901099+0000","last_fullsized":"2024-09-11T03:40:15.901099+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'44","last_scrub_stamp":"2024-09-11T03:39:32.965569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:32.965569+0000","objects_scrubbed":2,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:44.689545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002443368,"stat_sum":{"num_bytes":5529503,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":30,"num_read_kb":15200,"num_write":33,"num_write_kb":12236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"28","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"4"},{"start":"33","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":52,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.907106+0000","last_change":"2024-09-11T03:40:00.262368+0000","last_active":"2024-09-11T03:40:15.907106+0000","last_peered":"2024-09-11T03:40:15.907106+0000","last_clean":"2024-09-11T03:40:15.907106+0000","last_became_active":"2024-09-11T03:39:59.833111+0000","last_became_peered":"2024-09-11T03:39:59.833111+0000","last_unstale":"2024-09-11T03:40:15.907106+0000","last_undegraded":"2024-09-11T03:40:15.907106+0000","last_fullsized":"2024-09-11T03:40:15.907106+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.262301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:28.856759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"193'25","reported_seq":554,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.856659+0000","last_change":"2024-09-11T03:40:14.839480+0000","last_active":"2024-09-11T03:40:15.856659+0000","last_peered":"2024-09-11T03:40:15.856659+0000","last_clean":"2024-09-11T03:40:15.856659+0000","last_became_active":"2024-09-11T03:40:05.966325+0000","last_became_peered":"2024-09-11T03:40:05.966325+0000","last_unstale":"2024-09-11T03:40:15.856659+0000","last_undegraded":"2024-09-11T03:40:15.856659+0000","last_fullsized":"2024-09-11T03:40:15.856659+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'14","last_scrub_stamp":"2024-09-11T03:39:33.465057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:33.465057+0000","objects_scrubbed":1,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:36.517628+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00050579800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":5533,"num_write":15,"num_write_kb":3881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"28","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"4"},{"start":"33","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":381,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.856686+0000","last_change":"2024-09-11T03:40:01.496304+0000","last_active":"2024-09-11T03:40:15.856686+0000","last_peered":"2024-09-11T03:40:15.856686+0000","last_clean":"2024-09-11T03:40:15.856686+0000","last_became_active":"2024-09-11T03:40:00.099236+0000","last_became_peered":"2024-09-11T03:40:00.099236+0000","last_unstale":"2024-09-11T03:40:15.856686+0000","last_undegraded":"2024-09-11T03:40:15.856686+0000","last_fullsized":"2024-09-11T03:40:15.856686+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:01.496214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:01.496214+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:29.501697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"196'52","reported_seq":583,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-11T03:40:17.833688+0000","last_change":"2024-09-11T03:40:17.833921+0000","last_active":"2024-09-11T03:40:17.833688+0000","last_peered":"2024-09-11T03:40:17.833688+0000","last_clean":"2024-09-11T03:40:17.833688+0000","last_became_active":"2024-09-11T03:38:30.162952+0000","last_became_peered":"2024-09-11T03:38:30.162952+0000","last_unstale":"2024-09-11T03:40:17.833688+0000","last_undegraded":"2024-09-11T03:40:17.833688+0000","last_fullsized":"2024-09-11T03:40:17.833688+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'49","last_scrub_stamp":"2024-09-11T03:39:51.194245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:51.194245+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:09.288675+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043314100000000001,"stat_sum":{"num_bytes":3176523,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":31,"num_read_kb":19745,"num_write":25,"num_write_kb":10878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2c","length":"1"},{"start":"30","length":"1"},{"start":"33","length":"2"}]},{"pgid":"1.1","version":"20'92","reported_seq":280,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-11T03:40:18.137609+0000","last_change":"2024-09-11T03:40:07.837246+0000","last_active":"2024-09-11T03:40:18.137609+0000","last_peered":"2024-09-11T03:40:18.137609+0000","last_clean":"2024-09-11T03:40:18.137609+0000","last_became_active":"2024-09-11T03:40:05.966251+0000","last_became_peered":"2024-09-11T03:40:05.966251+0000","last_unstale":"2024-09-11T03:40:18.137609+0000","last_undegraded":"2024-09-11T03:40:18.137609+0000","last_fullsized":"2024-09-11T03:40:18.137609+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:37.216754+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":12,"num_write_kb":196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":144254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"199'68","reported_seq":613,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.909554+0000","last_change":"2024-09-11T03:40:14.866046+0000","last_active":"2024-09-11T03:40:15.909554+0000","last_peered":"2024-09-11T03:40:15.909554+0000","last_clean":"2024-09-11T03:40:15.909554+0000","last_became_active":"2024-09-11T03:39:26.115670+0000","last_became_peered":"2024-09-11T03:39:26.115670+0000","last_unstale":"2024-09-11T03:40:15.909554+0000","last_undegraded":"2024-09-11T03:40:15.909554+0000","last_fullsized":"2024-09-11T03:40:15.909554+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'57","last_scrub_stamp":"2024-09-11T03:39:56.268853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:56.268853+0000","objects_scrubbed":1,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:10.551637+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0023757090000000002,"stat_sum":{"num_bytes":5126135,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":16,"num_read_kb":6,"num_write":35,"num_write_kb":17726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"28","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"4"},{"start":"33","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":41,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-11T03:40:17.835079+0000","last_change":"2024-09-11T03:40:06.833351+0000","last_active":"2024-09-11T03:40:17.835079+0000","last_peered":"2024-09-11T03:40:17.835079+0000","last_clean":"2024-09-11T03:40:17.835079+0000","last_became_active":"2024-09-11T03:40:05.963517+0000","last_became_peered":"2024-09-11T03:40:05.963517+0000","last_unstale":"2024-09-11T03:40:17.835079+0000","last_undegraded":"2024-09-11T03:40:17.835079+0000","last_fullsized":"2024-09-11T03:40:17.835079+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:36.470223+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":192337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"198'89","reported_seq":639,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.907251+0000","last_change":"2024-09-11T03:40:14.842151+0000","last_active":"2024-09-11T03:40:15.907251+0000","last_peered":"2024-09-11T03:40:15.907251+0000","last_clean":"2024-09-11T03:40:15.907251+0000","last_became_active":"2024-09-11T03:40:05.966715+0000","last_became_peered":"2024-09-11T03:40:05.966715+0000","last_unstale":"2024-09-11T03:40:15.907251+0000","last_undegraded":"2024-09-11T03:40:15.907251+0000","last_fullsized":"2024-09-11T03:40:15.907251+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'55","last_scrub_stamp":"2024-09-11T03:39:40.378705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:40.378705+0000","objects_scrubbed":3,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:58.662163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0042282470000000001,"stat_sum":{"num_bytes":11647289,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":18897,"num_write":57,"num_write_kb":18703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":7190868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"28","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"4"},{"start":"33","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":280,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-11T03:40:18.137460+0000","last_change":"2024-09-11T03:40:06.985002+0000","last_active":"2024-09-11T03:40:18.137460+0000","last_peered":"2024-09-11T03:40:18.137460+0000","last_clean":"2024-09-11T03:40:18.137460+0000","last_became_active":"2024-09-11T03:40:05.964868+0000","last_became_peered":"2024-09-11T03:40:05.964868+0000","last_unstale":"2024-09-11T03:40:18.137460+0000","last_undegraded":"2024-09-11T03:40:18.137460+0000","last_fullsized":"2024-09-11T03:40:18.137460+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.855022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.855022+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:36.260431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":144253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"199'170","reported_seq":723,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.886161+0000","last_change":"2024-09-11T03:40:14.839729+0000","last_active":"2024-09-11T03:40:15.886161+0000","last_peered":"2024-09-11T03:40:15.886161+0000","last_clean":"2024-09-11T03:40:15.886161+0000","last_became_active":"2024-09-11T03:38:30.163080+0000","last_became_peered":"2024-09-11T03:38:30.163080+0000","last_unstale":"2024-09-11T03:40:15.886161+0000","last_undegraded":"2024-09-11T03:40:15.886161+0000","last_fullsized":"2024-09-11T03:40:15.886161+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'97","last_scrub_stamp":"2024-09-11T03:39:33.455974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:33.455974+0000","objects_scrubbed":5,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:53.343451+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00295985,"stat_sum":{"num_bytes":16878389,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":119,"num_read_kb":108495,"num_write":95,"num_write_kb":29136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"28","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"4"},{"start":"33","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":41,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-11T03:40:17.833361+0000","last_change":"2024-09-11T03:40:06.831598+0000","last_active":"2024-09-11T03:40:17.833361+0000","last_peered":"2024-09-11T03:40:17.833361+0000","last_clean":"2024-09-11T03:40:17.833361+0000","last_became_active":"2024-09-11T03:40:05.965205+0000","last_became_peered":"2024-09-11T03:40:05.965205+0000","last_unstale":"2024-09-11T03:40:17.833361+0000","last_undegraded":"2024-09-11T03:40:17.833361+0000","last_fullsized":"2024-09-11T03:40:17.833361+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:36.232323+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":192338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"198'103","reported_seq":610,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.907236+0000","last_change":"2024-09-11T03:40:14.840845+0000","last_active":"2024-09-11T03:40:15.907236+0000","last_peered":"2024-09-11T03:40:15.907236+0000","last_clean":"2024-09-11T03:40:15.907236+0000","last_became_active":"2024-09-11T03:38:29.887945+0000","last_became_peered":"2024-09-11T03:38:29.887945+0000","last_unstale":"2024-09-11T03:40:15.907236+0000","last_undegraded":"2024-09-11T03:40:15.907236+0000","last_fullsized":"2024-09-11T03:40:15.907236+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:38:28.875337+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029974509999999999,"stat_sum":{"num_bytes":6910333,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":44,"num_read_kb":43097,"num_write":40,"num_write_kb":22029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"28","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"4"},{"start":"33","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":151,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.839356+0000","last_change":"2024-09-11T03:40:00.357039+0000","last_active":"2024-09-11T03:40:15.839356+0000","last_peered":"2024-09-11T03:40:15.839356+0000","last_clean":"2024-09-11T03:40:15.839356+0000","last_became_active":"2024-09-11T03:39:59.839301+0000","last_became_peered":"2024-09-11T03:39:59.839301+0000","last_unstale":"2024-09-11T03:40:15.839356+0000","last_undegraded":"2024-09-11T03:40:15.839356+0000","last_fullsized":"2024-09-11T03:40:15.839356+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.356985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.356985+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:14.670619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"201'125","reported_seq":719,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-11T03:40:17.840090+0000","last_change":"2024-09-11T03:40:17.840300+0000","last_active":"2024-09-11T03:40:17.840090+0000","last_peered":"2024-09-11T03:40:17.840090+0000","last_clean":"2024-09-11T03:40:17.840090+0000","last_became_active":"2024-09-11T03:40:08.836627+0000","last_became_peered":"2024-09-11T03:40:08.836627+0000","last_unstale":"2024-09-11T03:40:17.840090+0000","last_undegraded":"2024-09-11T03:40:17.840090+0000","last_fullsized":"2024-09-11T03:40:17.840090+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'75","last_scrub_stamp":"2024-09-11T03:39:39.753844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:39.753844+0000","objects_scrubbed":4,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:42.411003+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0067512140000000002,"stat_sum":{"num_bytes":9617516,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":59,"num_read_kb":32520,"num_write":72,"num_write_kb":28595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":6433401,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2c","length":"1"},{"start":"30","length":"1"},{"start":"33","length":"2"}]},{"pgid":"2.8","version":"198'119","reported_seq":643,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.856650+0000","last_change":"2024-09-11T03:40:14.842379+0000","last_active":"2024-09-11T03:40:15.856650+0000","last_peered":"2024-09-11T03:40:15.856650+0000","last_clean":"2024-09-11T03:40:15.856650+0000","last_became_active":"2024-09-11T03:38:30.162819+0000","last_became_peered":"2024-09-11T03:38:30.162819+0000","last_unstale":"2024-09-11T03:40:15.856650+0000","last_undegraded":"2024-09-11T03:40:15.856650+0000","last_fullsized":"2024-09-11T03:40:15.856650+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'86","last_scrub_stamp":"2024-09-11T03:39:54.511997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:54.511997+0000","objects_scrubbed":4,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:16.626627+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0038355279999999999,"stat_sum":{"num_bytes":14060840,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":77,"num_read_kb":85186,"num_write":62,"num_write_kb":19482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"28","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"4"},{"start":"33","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":297,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-11T03:40:18.137465+0000","last_change":"2024-09-11T03:40:00.202716+0000","last_active":"2024-09-11T03:40:18.137465+0000","last_peered":"2024-09-11T03:40:18.137465+0000","last_clean":"2024-09-11T03:40:18.137465+0000","last_became_active":"2024-09-11T03:40:00.099329+0000","last_became_peered":"2024-09-11T03:40:00.099329+0000","last_unstale":"2024-09-11T03:40:18.137465+0000","last_undegraded":"2024-09-11T03:40:18.137465+0000","last_fullsized":"2024-09-11T03:40:18.137465+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.202664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.202664+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:20.173041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"198'179","reported_seq":730,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.907182+0000","last_change":"2024-09-11T03:40:14.844269+0000","last_active":"2024-09-11T03:40:15.907182+0000","last_peered":"2024-09-11T03:40:15.907182+0000","last_clean":"2024-09-11T03:40:15.907182+0000","last_became_active":"2024-09-11T03:38:29.888204+0000","last_became_peered":"2024-09-11T03:38:29.888204+0000","last_unstale":"2024-09-11T03:40:15.907182+0000","last_undegraded":"2024-09-11T03:40:15.907182+0000","last_fullsized":"2024-09-11T03:40:15.907182+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'138","last_scrub_stamp":"2024-09-11T03:39:52.405246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:52.405246+0000","objects_scrubbed":6,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:18.964220+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0063352820000000002,"stat_sum":{"num_bytes":16711835,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":136,"num_read_kb":100304,"num_write":104,"num_write_kb":37511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"28","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"4"},{"start":"33","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":381,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.856605+0000","last_change":"2024-09-11T03:40:00.516395+0000","last_active":"2024-09-11T03:40:15.856605+0000","last_peered":"2024-09-11T03:40:15.856605+0000","last_clean":"2024-09-11T03:40:15.856605+0000","last_became_active":"2024-09-11T03:39:59.838010+0000","last_became_peered":"2024-09-11T03:39:59.838010+0000","last_unstale":"2024-09-11T03:40:15.856605+0000","last_undegraded":"2024-09-11T03:40:15.856605+0000","last_fullsized":"2024-09-11T03:40:15.856605+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.516339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.516339+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:08.765216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"186'37","reported_seq":556,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.839406+0000","last_change":"2024-09-11T03:40:14.839081+0000","last_active":"2024-09-11T03:40:15.839406+0000","last_peered":"2024-09-11T03:40:15.839406+0000","last_clean":"2024-09-11T03:40:15.839406+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:40:15.839406+0000","last_undegraded":"2024-09-11T03:40:15.839406+0000","last_fullsized":"2024-09-11T03:40:15.839406+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'8","last_scrub_stamp":"2024-09-11T03:39:33.449882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:33.449882+0000","objects_scrubbed":1,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:44.709503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052931500000000004,"stat_sum":{"num_bytes":2638699,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":34895,"num_write":16,"num_write_kb":10633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"28","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"4"},{"start":"33","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":59,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.900945+0000","last_change":"2024-09-11T03:40:00.207222+0000","last_active":"2024-09-11T03:40:15.900945+0000","last_peered":"2024-09-11T03:40:15.900945+0000","last_clean":"2024-09-11T03:40:15.900945+0000","last_became_active":"2024-09-11T03:39:59.849648+0000","last_became_peered":"2024-09-11T03:39:59.849648+0000","last_unstale":"2024-09-11T03:40:15.900945+0000","last_undegraded":"2024-09-11T03:40:15.900945+0000","last_fullsized":"2024-09-11T03:40:15.900945+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.207167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.207167+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:25.470578+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"199'118","reported_seq":639,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-11T03:40:15.864331+0000","last_change":"2024-09-11T03:40:14.839489+0000","last_active":"2024-09-11T03:40:15.864331+0000","last_peered":"2024-09-11T03:40:15.864331+0000","last_clean":"2024-09-11T03:40:15.864331+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:40:15.864331+0000","last_undegraded":"2024-09-11T03:40:15.864331+0000","last_fullsized":"2024-09-11T03:40:15.864331+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'86","last_scrub_stamp":"2024-09-11T03:39:54.238844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:54.238844+0000","objects_scrubbed":5,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:01.258185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0026906460000000001,"stat_sum":{"num_bytes":15019023,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":15673,"num_write":57,"num_write_kb":31099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"28","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"4"},{"start":"33","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":147083280,"num_objects":70,"num_object_clones":27,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":10,"num_read":931,"num_read_kb":709415,"num_write":834,"num_write_kb":332274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":13624269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":281366528,"data_stored":308776080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1661,"ondisk_log_size":1661,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":805048,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":45,"num_write_kb":782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":673182,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285605,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":46256,"kb_used_data":19220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93698896,"statfs":{"total":95995035648,"available":95947669504,"internally_reserved":0,"allocated":19681280,"data_stored":22248460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":326417514537,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743471,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66068,"kb_used_data":34880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93679084,"statfs":{"total":95995035648,"available":95927382016,"internally_reserved":0,"allocated":35717120,"data_stored":37064632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037814,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63776,"kb_used_data":32580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93681376,"statfs":{"total":95995035648,"available":95929729024,"internally_reserved":0,"allocated":33361920,"data_stored":35288940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299449,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62380,"kb_used_data":31192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93682772,"statfs":{"total":95995035648,"available":95931158528,"internally_reserved":0,"allocated":31940608,"data_stored":32468965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":193273528382,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757315,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":39800,"kb_used_data":8608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93705352,"statfs":{"total":95995035648,"available":95954280448,"internally_reserved":0,"allocated":8814592,"data_stored":7815033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986249,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63072,"kb_used_data":31884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93682080,"statfs":{"total":95995035648,"available":95930449920,"internally_reserved":0,"allocated":32649216,"data_stored":35563536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71372,"kb_used_data":40184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93673780,"statfs":{"total":95995035648,"available":95921950720,"internally_reserved":0,"allocated":41148416,"data_stored":41650361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57612,"kb_used_data":26424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93687540,"statfs":{"total":95995035648,"available":95936040960,"internally_reserved":0,"allocated":27058176,"data_stored":28976945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62816,"kb_used_data":31620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93682336,"statfs":{"total":95995035648,"available":95930712064,"internally_reserved":0,"allocated":32378880,"data_stored":35903507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65016,"kb_used_data":33824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93680136,"statfs":{"total":95995035648,"available":95928459264,"internally_reserved":0,"allocated":34635776,"data_stored":37959491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34095104,"data_stored":38915443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30072832,"data_stored":34560439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":23564288,"data_stored":25337900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38612992,"data_stored":40089685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30294016,"data_stored":34187818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6303744,"data_stored":6270717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4079616,"data_stored":4066399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":30793728,"data_stored":33079569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":30728192,"data_stored":33618192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":33386496,"data_stored":35705204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2387968,"data_stored":2367002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":17047552,"data_stored":20577712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:40:22.346 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-11T03:40:22.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-09-11T03:40:22.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:22.354 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:22.367 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:22.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:22.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:22.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:22.490 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:22.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:22.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:22.592 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:22.618 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:22.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:22.619 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:22.622 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:40:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:40:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:40:22.622 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:40:22.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:22.740 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:22.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:22.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:22.838 INFO:tasks.rados.rados.2.smithi175.stdout:557: done (0 left) 2024-09-11T03:40:22.838 INFO:tasks.rados.rados.2.smithi175.stdout:558: rollback oid 19 current snap is 54 2024-09-11T03:40:22.838 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 19 to 53 2024-09-11T03:40:22.838 INFO:tasks.rados.rados.2.smithi175.stdout:559: read oid 47 snap 51 2024-09-11T03:40:22.838 INFO:tasks.rados.rados.2.smithi175.stdout:559: expect (ObjNum 181 snap 43 seq_num 181) 2024-09-11T03:40:22.838 INFO:tasks.rados.rados.2.smithi175.stdout:560: snap_create 2024-09-11T03:40:22.839 INFO:tasks.rados.rados.2.smithi175.stdout:558: finishing rollback tid 1 to smithi17557870-19 2024-09-11T03:40:22.839 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 0 (ObjNum 289 snap 0 seq_num 197299620) dirty dne 2024-09-11T03:40:22.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:22.849 INFO:tasks.rados.rados.2.smithi175.stdout:559: expect (ObjNum 181 snap 43 seq_num 181) 2024-09-11T03:40:22.855 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:22.864 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:22.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:22.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:22.966 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:22.966 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:40:22.987 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:22.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:22.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:23.090 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:23.112 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:23.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:23.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:23.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:23.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:23.237 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:23.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:23.338 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:23.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:23.361 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:23.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:23.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T03:40:23.402 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:23.463 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:23.484 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:23.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:23.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:23.607 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:40:23.608 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:23.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:23.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:23.710 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:23.732 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:23.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:23.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:23.813 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:40:23.814 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:40:23.829 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:40:23.829 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:40:23.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:23.838 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:40:23.839 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 1 pg_num to 21 2024-09-11T03:40:23.842 INFO:tasks.rados.rados.2.smithi175.stdout:558: done (2 left) 2024-09-11T03:40:23.842 INFO:tasks.rados.rados.2.smithi175.stdout:559: done (1 left) 2024-09-11T03:40:23.842 INFO:tasks.rados.rados.2.smithi175.stdout:560: done (0 left) 2024-09-11T03:40:23.842 INFO:tasks.rados.rados.2.smithi175.stdout:561: snap_create 2024-09-11T03:40:23.852 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:40:23.853 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:40:23.857 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:23.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:23.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:23.876 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:40:23.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:40:23.896 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:40:23.896 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:40:23.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:40:23.913 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:40:23.929 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:40:23.929 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:40:23.945 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:40:23.946 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:40:23.959 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:23.963 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:40:23.963 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:40:23.981 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:40:23.981 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:40:23.981 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:23 smithi121 ceph-mon[30729]: pgmap v463: 27 pgs: 27 active+clean; 157 MiB data, 612 MiB used, 893 GiB / 894 GiB avail; 5.5 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-11T03:40:23.981 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:23 smithi121 ceph-mon[30729]: osdmap e206: 12 total, 12 up, 10 in 2024-09-11T03:40:23.981 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:23 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3362856449' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:40:23.981 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:23 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2607240597' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-11T03:40:23.982 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:23 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3549480127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:23.984 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:23.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:23.985 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:23.997 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:40:23.997 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:40:24.014 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:40:24.014 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:40:24.081 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:24.081 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:24.086 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:24.107 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:24.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:24.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:23 smithi084 ceph-mon[27254]: pgmap v463: 27 pgs: 27 active+clean; 157 MiB data, 612 MiB used, 893 GiB / 894 GiB avail; 5.5 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-11T03:40:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:23 smithi084 ceph-mon[27254]: osdmap e206: 12 total, 12 up, 10 in 2024-09-11T03:40:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:23 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3362856449' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:40:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:23 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2607240597' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-11T03:40:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:23 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3549480127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:24.209 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:23 smithi175 ceph-mon[30894]: pgmap v463: 27 pgs: 27 active+clean; 157 MiB data, 612 MiB used, 893 GiB / 894 GiB avail; 5.5 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-11T03:40:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:23 smithi175 ceph-mon[30894]: osdmap e206: 12 total, 12 up, 10 in 2024-09-11T03:40:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:23 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3362856449' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:40:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:23 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2607240597' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-11T03:40:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:23 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3549480127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:24.235 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:24.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:24.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:24.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:24.358 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:24.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:24.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:24.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:24.483 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:24.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:24.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:24.585 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:24.606 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:24.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:24.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:24.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:24.731 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:24.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:24.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T03:40:24.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:24.847 INFO:tasks.rados.rados.2.smithi175.stdout:561: done (0 left) 2024-09-11T03:40:24.847 INFO:tasks.rados.rados.2.smithi175.stdout:562: read oid 13 snap -1 2024-09-11T03:40:24.847 INFO:tasks.rados.rados.2.smithi175.stdout:562: expect (ObjNum 209 snap 51 seq_num 209) 2024-09-11T03:40:24.847 INFO:tasks.rados.rados.2.smithi175.stdout:563: rollback oid 17 current snap is 56 2024-09-11T03:40:24.848 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 17 to 53 2024-09-11T03:40:24.848 INFO:tasks.rados.rados.2.smithi175.stdout:564: read oid 30 snap -1 2024-09-11T03:40:24.848 INFO:tasks.rados.rados.2.smithi175.stdout:564: expect deleted 2024-09-11T03:40:24.848 INFO:tasks.rados.rados.2.smithi175.stdout:565: snap_remove snap 55 2024-09-11T03:40:24.855 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:24.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:24.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:24.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:24 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:24.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:24 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3362856449' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:40:24.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:24 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2607240597' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-11T03:40:24.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:24 smithi084 ceph-mon[27254]: osdmap e207: 12 total, 12 up, 10 in 2024-09-11T03:40:24.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:24 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:40:24.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:24 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:40:24.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:24 smithi084 ceph-mon[27254]: 1.6 scrub starts 2024-09-11T03:40:24.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:24 smithi084 ceph-mon[27254]: 1.7 scrub starts 2024-09-11T03:40:24.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-11T03:40:24.958 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:24.984 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:24.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:24.985 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:25.033 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:40:25.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:25.107 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:25.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:25.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:25.153 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:24 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:25.153 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:24 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3362856449' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:40:25.153 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:24 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2607240597' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-11T03:40:25.153 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:24 smithi121 ceph-mon[30729]: osdmap e207: 12 total, 12 up, 10 in 2024-09-11T03:40:25.153 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:24 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:40:25.153 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:24 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:40:25.154 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:24 smithi121 ceph-mon[30729]: 1.6 scrub starts 2024-09-11T03:40:25.154 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:24 smithi121 ceph-mon[30729]: 1.7 scrub starts 2024-09-11T03:40:25.154 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-11T03:40:25.210 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:24 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:24 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3362856449' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:40:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:24 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2607240597' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-11T03:40:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:24 smithi175 ceph-mon[30894]: osdmap e207: 12 total, 12 up, 10 in 2024-09-11T03:40:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:24 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:40:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:24 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:40:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:24 smithi175 ceph-mon[30894]: 1.6 scrub starts 2024-09-11T03:40:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:24 smithi175 ceph-mon[30894]: 1.7 scrub starts 2024-09-11T03:40:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-11T03:40:25.235 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:25.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:25.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:25.336 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:25.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:25.362 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:25.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:25.464 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:25.486 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:25.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:25.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:25.537 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:25.537 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:25.587 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:25.613 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:25.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:25.614 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:25.715 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:25.741 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:25.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:25.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:25.843 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:25.849 INFO:tasks.rados.rados.2.smithi175.stdout:565: done (3 left) 2024-09-11T03:40:25.849 INFO:tasks.rados.rados.2.smithi175.stdout:566: write oid 37 current snap is 56 2024-09-11T03:40:25.849 INFO:tasks.rados.rados.2.smithi175.stdout:566: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2024-09-11T03:40:25.852 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:25 smithi084 ceph-mon[27254]: 1.5 scrub starts 2024-09-11T03:40:25.852 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:25 smithi084 ceph-mon[27254]: pgmap v466: 27 pgs: 27 active+clean; 156 MiB data, 616 MiB used, 893 GiB / 894 GiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 32 op/s 2024-09-11T03:40:25.853 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-11T03:40:25.853 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:25 smithi084 ceph-mon[27254]: osdmap e208: 12 total, 12 up, 10 in 2024-09-11T03:40:25.855 INFO:tasks.rados.rados.2.smithi175.stdout:566: writing smithi17557870-37 from 615189 to 1070949 tid 1 2024-09-11T03:40:25.863 INFO:tasks.rados.rados.2.smithi175.stdout:566: writing smithi17557870-37 from 1733883 to 2444961 tid 2 2024-09-11T03:40:25.869 INFO:tasks.rados.rados.2.smithi175.stdout:566: writing smithi17557870-37 from 3134431 to 3594472 tid 3 2024-09-11T03:40:25.869 INFO:tasks.rados.rados.2.smithi175.stdout:567: write (excl) oid 28 current snap is 56 2024-09-11T03:40:25.869 INFO:tasks.rados.rados.2.smithi175.stdout:567: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2024-09-11T03:40:25.871 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:25.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:25.875 INFO:tasks.rados.rados.2.smithi175.stdout:567: writing smithi17557870-28 from 631996 to 1163005 tid 1 2024-09-11T03:40:25.881 INFO:tasks.rados.rados.2.smithi175.stdout:567: writing smithi17557870-28 from 1792365 to 2247101 tid 2 2024-09-11T03:40:25.887 INFO:tasks.rados.rados.2.smithi175.stdout:567: writing smithi17557870-28 from 2761854 to 3433167 tid 3 2024-09-11T03:40:25.887 INFO:tasks.rados.rados.2.smithi175.stdout:568: write oid 49 current snap is 56 2024-09-11T03:40:25.888 INFO:tasks.rados.rados.2.smithi175.stdout:568: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2024-09-11T03:40:25.895 INFO:tasks.rados.rados.2.smithi175.stdout:568: writing smithi17557870-49 from 648803 to 1371414 tid 1 2024-09-11T03:40:25.902 INFO:tasks.rados.rados.2.smithi175.stdout:568: writing smithi17557870-49 from 1967200 to 2681947 tid 2 2024-09-11T03:40:25.909 INFO:tasks.rados.rados.2.smithi175.stdout:568: writing smithi17557870-49 from 3421983 to 3848803 tid 3 2024-09-11T03:40:25.909 INFO:tasks.rados.rados.2.smithi175.stdout:569: copy_from oid 27 from oid 42 current snap is 56 2024-09-11T03:40:25.909 INFO:tasks.rados.rados.2.smithi175.stdout:563: finishing rollback tid 1 to smithi17557870-17 2024-09-11T03:40:25.909 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 0 (ObjNum 129 snap 0 seq_num 197266212) dirty dne 2024-09-11T03:40:25.909 INFO:tasks.rados.rados.2.smithi175.stdout:562: expect (ObjNum 209 snap 51 seq_num 209) 2024-09-11T03:40:25.973 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:25.995 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:25.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:26.067 INFO:tasks.rados.rados.2.smithi175.stdout:566: finishing write tid 1 to smithi17557870-37 2024-09-11T03:40:26.067 INFO:tasks.rados.rados.2.smithi175.stdout:566: finishing write tid 2 to smithi17557870-37 2024-09-11T03:40:26.068 INFO:tasks.rados.rados.2.smithi175.stdout:566: finishing write tid 3 to smithi17557870-37 2024-09-11T03:40:26.068 INFO:tasks.rados.rados.2.smithi175.stdout:566: finishing write tid 4 to smithi17557870-37 2024-09-11T03:40:26.068 INFO:tasks.rados.rados.2.smithi175.stdout:566: finishing write tid 5 to smithi17557870-37 2024-09-11T03:40:26.068 INFO:tasks.rados.rados.2.smithi175.stdout:566: oid 37 updating version 0 to 116 2024-09-11T03:40:26.068 INFO:tasks.rados.rados.2.smithi175.stdout:566: oid 37 updating version 116 to 117 2024-09-11T03:40:26.068 INFO:tasks.rados.rados.2.smithi175.stdout:566: oid 37 version 117 is already newer than 115 2024-09-11T03:40:26.068 INFO:tasks.rados.rados.2.smithi175.stdout:566: oid 37 version 117 is already newer than 114 2024-09-11T03:40:26.068 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 117 (ObjNum 227 snap 56 seq_num 227) dirty exists 2024-09-11T03:40:26.068 INFO:tasks.rados.rados.2.smithi175.stdout:566: left oid 37 (ObjNum 227 snap 56 seq_num 227) 2024-09-11T03:40:26.068 INFO:tasks.rados.rados.2.smithi175.stdout:567: finishing write tid 1 to smithi17557870-28 2024-09-11T03:40:26.068 INFO:tasks.rados.rados.2.smithi175.stdout:567: finishing write tid 2 to smithi17557870-28 2024-09-11T03:40:26.068 INFO:tasks.rados.rados.2.smithi175.stdout:562: done (6 left) 2024-09-11T03:40:26.069 INFO:tasks.rados.rados.2.smithi175.stdout:563: done (5 left) 2024-09-11T03:40:26.069 INFO:tasks.rados.rados.2.smithi175.stdout:564: done (4 left) 2024-09-11T03:40:26.069 INFO:tasks.rados.rados.2.smithi175.stdout:566: done (3 left) 2024-09-11T03:40:26.069 INFO:tasks.rados.rados.2.smithi175.stdout:570: write oid 21 current snap is 56 2024-09-11T03:40:26.069 INFO:tasks.rados.rados.2.smithi175.stdout:567: finishing write tid 3 to smithi17557870-28 2024-09-11T03:40:26.069 INFO:tasks.rados.rados.2.smithi175.stdout:567: finishing write tid 4 to smithi17557870-28 2024-09-11T03:40:26.069 INFO:tasks.rados.rados.2.smithi175.stdout:567: finishing write tid 5 to smithi17557870-28 2024-09-11T03:40:26.069 INFO:tasks.rados.rados.2.smithi175.stdout:567: oid 28 updating version 0 to 206 2024-09-11T03:40:26.069 INFO:tasks.rados.rados.2.smithi175.stdout:567: oid 28 updating version 206 to 207 2024-09-11T03:40:26.069 INFO:tasks.rados.rados.2.smithi175.stdout:567: oid 28 updating version 207 to 208 2024-09-11T03:40:26.069 INFO:tasks.rados.rados.2.smithi175.stdout:567: oid 28 version 208 is already newer than 205 2024-09-11T03:40:26.069 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 208 (ObjNum 228 snap 56 seq_num 228) dirty exists 2024-09-11T03:40:26.069 INFO:tasks.rados.rados.2.smithi175.stdout:567: left oid 28 (ObjNum 228 snap 56 seq_num 228) 2024-09-11T03:40:26.070 INFO:tasks.rados.rados.2.smithi175.stdout:568: finishing write tid 1 to smithi17557870-49 2024-09-11T03:40:26.070 INFO:tasks.rados.rados.2.smithi175.stdout:568: finishing write tid 2 to smithi17557870-49 2024-09-11T03:40:26.070 INFO:tasks.rados.rados.2.smithi175.stdout:569: finishing copy_from to smithi17557870-27 2024-09-11T03:40:26.070 INFO:tasks.rados.rados.2.smithi175.stdout:569: got expected ENOENT (src dne) 2024-09-11T03:40:26.070 INFO:tasks.rados.rados.2.smithi175.stdout:569: finishing copy_from racing read to smithi17557870-27 2024-09-11T03:40:26.070 INFO:tasks.rados.rados.2.smithi175.stdout:568: finishing write tid 3 to smithi17557870-49 2024-09-11T03:40:26.070 INFO:tasks.rados.rados.2.smithi175.stdout:568: finishing write tid 4 to smithi17557870-49 2024-09-11T03:40:26.070 INFO:tasks.rados.rados.2.smithi175.stdout:568: finishing write tid 5 to smithi17557870-49 2024-09-11T03:40:26.070 INFO:tasks.rados.rados.2.smithi175.stdout:568: oid 49 updating version 0 to 146 2024-09-11T03:40:26.070 INFO:tasks.rados.rados.2.smithi175.stdout:568: oid 49 version 146 is already newer than 145 2024-09-11T03:40:26.070 INFO:tasks.rados.rados.2.smithi175.stdout:568: oid 49 updating version 146 to 147 2024-09-11T03:40:26.070 INFO:tasks.rados.rados.2.smithi175.stdout:568: oid 49 updating version 147 to 148 2024-09-11T03:40:26.070 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 148 (ObjNum 229 snap 56 seq_num 229) dirty exists 2024-09-11T03:40:26.070 INFO:tasks.rados.rados.2.smithi175.stdout:568: left oid 49 (ObjNum 229 snap 56 seq_num 229) 2024-09-11T03:40:26.071 INFO:tasks.rados.rados.2.smithi175.stdout:570: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2024-09-11T03:40:26.074 INFO:tasks.rados.rados.2.smithi175.stdout:570: writing smithi17557870-21 from 665610 to 1463470 tid 1 2024-09-11T03:40:26.079 INFO:tasks.rados.rados.2.smithi175.stdout:570: writing smithi17557870-21 from 1909329 to 2367734 tid 2 2024-09-11T03:40:26.084 INFO:tasks.rados.rados.2.smithi175.stdout:570: writing smithi17557870-21 from 2816700 to 3226910 tid 3 2024-09-11T03:40:26.084 INFO:tasks.rados.rados.2.smithi175.stdout:567: done (3 left) 2024-09-11T03:40:26.084 INFO:tasks.rados.rados.2.smithi175.stdout:568: done (2 left) 2024-09-11T03:40:26.084 INFO:tasks.rados.rados.2.smithi175.stdout:569: done (1 left) 2024-09-11T03:40:26.084 INFO:tasks.rados.rados.2.smithi175.stdout:571: write oid 39 current snap is 56 2024-09-11T03:40:26.085 INFO:tasks.rados.rados.2.smithi175.stdout:571: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2024-09-11T03:40:26.090 INFO:tasks.rados.rados.2.smithi175.stdout:571: writing smithi17557870-39 from 682417 to 1155526 tid 1 2024-09-11T03:40:26.095 INFO:tasks.rados.rados.2.smithi175.stdout:571: writing smithi17557870-39 from 1567811 to 2169874 tid 2 2024-09-11T03:40:26.097 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:26.103 INFO:tasks.rados.rados.2.smithi175.stdout:571: writing smithi17557870-39 from 2844123 to 3465605 tid 3 2024-09-11T03:40:26.103 INFO:tasks.rados.rados.2.smithi175.stdout:572: read oid 16 snap -1 2024-09-11T03:40:26.103 INFO:tasks.rados.rados.2.smithi175.stdout:572: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:40:26.103 INFO:tasks.rados.rados.2.smithi175.stdout:573: write oid 6 current snap is 56 2024-09-11T03:40:26.103 INFO:tasks.rados.rados.2.smithi175.stdout:570: finishing write tid 1 to smithi17557870-21 2024-09-11T03:40:26.103 INFO:tasks.rados.rados.2.smithi175.stdout:573: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-09-11T03:40:26.109 INFO:tasks.rados.rados.2.smithi175.stdout:573: writing smithi17557870-6 from 699224 to 1247582 tid 1 2024-09-11T03:40:26.116 INFO:tasks.rados.rados.2.smithi175.stdout:573: writing smithi17557870-6 from 2026293 to 2488367 tid 2 2024-09-11T03:40:26.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:26.121 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:26.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:26.123 INFO:tasks.rados.rados.2.smithi175.stdout:573: writing smithi17557870-6 from 3271546 to 3704300 tid 3 2024-09-11T03:40:26.123 INFO:tasks.rados.rados.2.smithi175.stdout:574: snap_create 2024-09-11T03:40:26.123 INFO:tasks.rados.rados.2.smithi175.stdout:570: finishing write tid 2 to smithi17557870-21 2024-09-11T03:40:26.123 INFO:tasks.rados.rados.2.smithi175.stdout:570: finishing write tid 3 to smithi17557870-21 2024-09-11T03:40:26.124 INFO:tasks.rados.rados.2.smithi175.stdout:570: finishing write tid 4 to smithi17557870-21 2024-09-11T03:40:26.124 INFO:tasks.rados.rados.2.smithi175.stdout:570: finishing write tid 5 to smithi17557870-21 2024-09-11T03:40:26.124 INFO:tasks.rados.rados.2.smithi175.stdout:570: oid 21 updating version 0 to 152 2024-09-11T03:40:26.124 INFO:tasks.rados.rados.2.smithi175.stdout:570: oid 21 version 152 is already newer than 150 2024-09-11T03:40:26.124 INFO:tasks.rados.rados.2.smithi175.stdout:570: oid 21 version 152 is already newer than 151 2024-09-11T03:40:26.124 INFO:tasks.rados.rados.2.smithi175.stdout:570: oid 21 version 152 is already newer than 149 2024-09-11T03:40:26.124 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 152 (ObjNum 230 snap 56 seq_num 230) dirty exists 2024-09-11T03:40:26.124 INFO:tasks.rados.rados.2.smithi175.stdout:570: left oid 21 (ObjNum 230 snap 56 seq_num 230) 2024-09-11T03:40:26.124 INFO:tasks.rados.rados.2.smithi175.stdout:571: finishing write tid 1 to smithi17557870-39 2024-09-11T03:40:26.124 INFO:tasks.rados.rados.2.smithi175.stdout:571: finishing write tid 2 to smithi17557870-39 2024-09-11T03:40:26.124 INFO:tasks.rados.rados.2.smithi175.stdout:571: finishing write tid 3 to smithi17557870-39 2024-09-11T03:40:26.124 INFO:tasks.rados.rados.2.smithi175.stdout:571: finishing write tid 4 to smithi17557870-39 2024-09-11T03:40:26.124 INFO:tasks.rados.rados.2.smithi175.stdout:571: finishing write tid 5 to smithi17557870-39 2024-09-11T03:40:26.124 INFO:tasks.rados.rados.2.smithi175.stdout:571: oid 39 updating version 0 to 145 2024-09-11T03:40:26.125 INFO:tasks.rados.rados.2.smithi175.stdout:571: oid 39 updating version 145 to 148 2024-09-11T03:40:26.125 INFO:tasks.rados.rados.2.smithi175.stdout:571: oid 39 version 148 is already newer than 146 2024-09-11T03:40:26.125 INFO:tasks.rados.rados.2.smithi175.stdout:571: oid 39 version 148 is already newer than 147 2024-09-11T03:40:26.125 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 148 (ObjNum 231 snap 56 seq_num 231) dirty exists 2024-09-11T03:40:26.125 INFO:tasks.rados.rados.2.smithi175.stdout:571: left oid 39 (ObjNum 231 snap 56 seq_num 231) 2024-09-11T03:40:26.125 INFO:tasks.rados.rados.2.smithi175.stdout:572: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:40:26.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:25 smithi121 ceph-mon[30729]: 1.5 scrub starts 2024-09-11T03:40:26.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:25 smithi121 ceph-mon[30729]: pgmap v466: 27 pgs: 27 active+clean; 156 MiB data, 616 MiB used, 893 GiB / 894 GiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 32 op/s 2024-09-11T03:40:26.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-11T03:40:26.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:25 smithi121 ceph-mon[30729]: osdmap e208: 12 total, 12 up, 10 in 2024-09-11T03:40:26.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:25 smithi175 ceph-mon[30894]: 1.5 scrub starts 2024-09-11T03:40:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:25 smithi175 ceph-mon[30894]: pgmap v466: 27 pgs: 27 active+clean; 156 MiB data, 616 MiB used, 893 GiB / 894 GiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 32 op/s 2024-09-11T03:40:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-11T03:40:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:25 smithi175 ceph-mon[30894]: osdmap e208: 12 total, 12 up, 10 in 2024-09-11T03:40:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:26.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:26.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:26.246 INFO:tasks.rados.rados.2.smithi175.stdout:573: finishing write tid 1 to smithi17557870-6 2024-09-11T03:40:26.246 INFO:tasks.rados.rados.2.smithi175.stdout:573: finishing write tid 2 to smithi17557870-6 2024-09-11T03:40:26.246 INFO:tasks.rados.rados.2.smithi175.stdout:573: finishing write tid 3 to smithi17557870-6 2024-09-11T03:40:26.246 INFO:tasks.rados.rados.2.smithi175.stdout:573: finishing write tid 4 to smithi17557870-6 2024-09-11T03:40:26.246 INFO:tasks.rados.rados.2.smithi175.stdout:573: finishing write tid 5 to smithi17557870-6 2024-09-11T03:40:26.246 INFO:tasks.rados.rados.2.smithi175.stdout:573: oid 6 updating version 0 to 212 2024-09-11T03:40:26.246 INFO:tasks.rados.rados.2.smithi175.stdout:573: oid 6 updating version 212 to 213 2024-09-11T03:40:26.246 INFO:tasks.rados.rados.2.smithi175.stdout:573: oid 6 version 213 is already newer than 211 2024-09-11T03:40:26.246 INFO:tasks.rados.rados.2.smithi175.stdout:573: oid 6 version 213 is already newer than 210 2024-09-11T03:40:26.246 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 213 (ObjNum 232 snap 56 seq_num 232) dirty exists 2024-09-11T03:40:26.246 INFO:tasks.rados.rados.2.smithi175.stdout:573: left oid 6 (ObjNum 232 snap 56 seq_num 232) 2024-09-11T03:40:26.293 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T03:40:26.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:26.366 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:26.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:26.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:26.468 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:26.490 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:26.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:26.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:26.532 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:40:26.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:26.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:26.613 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:26.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:26.645 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:40:26.715 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:26.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:26.737 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:26.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:26.838 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:26.839 INFO:tasks.rados.rados.2.smithi175.stdout:570: done (4 left) 2024-09-11T03:40:26.840 INFO:tasks.rados.rados.2.smithi175.stdout:571: done (3 left) 2024-09-11T03:40:26.840 INFO:tasks.rados.rados.2.smithi175.stdout:572: done (2 left) 2024-09-11T03:40:26.840 INFO:tasks.rados.rados.2.smithi175.stdout:573: done (1 left) 2024-09-11T03:40:26.840 INFO:tasks.rados.rados.2.smithi175.stdout:574: done (0 left) 2024-09-11T03:40:26.840 INFO:tasks.rados.rados.2.smithi175.stdout:575: snap_create 2024-09-11T03:40:26.863 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:26.864 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:26.898 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:26.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:26 smithi121 ceph-mon[30729]: osdmap e209: 12 total, 12 up, 10 in 2024-09-11T03:40:26.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:26 smithi121 ceph-mon[30729]: 1.1 scrub starts 2024-09-11T03:40:26.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:26 smithi121 ceph-mon[30729]: 1.7 scrub starts 2024-09-11T03:40:26.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-11T03:40:26.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-11T03:40:26.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:26 smithi121 ceph-mon[30729]: osdmap e210: 12 total, 12 up, 10 in 2024-09-11T03:40:26.964 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:26.986 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:26.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:26.987 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:27.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:26 smithi084 ceph-mon[27254]: osdmap e209: 12 total, 12 up, 10 in 2024-09-11T03:40:27.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:26 smithi084 ceph-mon[27254]: 1.1 scrub starts 2024-09-11T03:40:27.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:26 smithi084 ceph-mon[27254]: 1.7 scrub starts 2024-09-11T03:40:27.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-11T03:40:27.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-11T03:40:27.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:26 smithi084 ceph-mon[27254]: osdmap e210: 12 total, 12 up, 10 in 2024-09-11T03:40:27.036 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:27.036 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:27.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:27.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:27.112 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:27.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:27.213 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:26 smithi175 ceph-mon[30894]: osdmap e209: 12 total, 12 up, 10 in 2024-09-11T03:40:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:26 smithi175 ceph-mon[30894]: 1.1 scrub starts 2024-09-11T03:40:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:26 smithi175 ceph-mon[30894]: 1.7 scrub starts 2024-09-11T03:40:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-11T03:40:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-11T03:40:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:26 smithi175 ceph-mon[30894]: osdmap e210: 12 total, 12 up, 10 in 2024-09-11T03:40:27.234 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:27.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:27.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:27.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:27.358 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:27.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:27.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:27.405 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:40:27.406 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:40:27.422 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:40:27.423 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:40:27.439 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:40:27.439 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:40:27.455 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:40:27.455 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:40:27.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:27.473 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:40:27.473 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:40:27.481 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:27.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:27.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:27.489 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:40:27.489 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:40:27.505 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:40:27.506 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:40:27.522 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:40:27.522 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:40:27.539 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:40:27.540 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:40:27.557 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:40:27.557 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:40:27.574 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:40:27.575 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:40:27.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:27.592 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:40:27.592 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:40:27.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:27.605 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:27.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:27.629 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:40:27.707 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:27.733 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:27.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:27.734 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:27.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T03:40:27.835 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:27.844 INFO:tasks.rados.rados.2.smithi175.stdout:575: done (0 left) 2024-09-11T03:40:27.845 INFO:tasks.rados.rados.2.smithi175.stdout:576: write oid 27 current snap is 58 2024-09-11T03:40:27.845 INFO:tasks.rados.rados.2.smithi175.stdout:576: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2024-09-11T03:40:27.852 INFO:tasks.rados.rados.2.smithi175.stdout:576: writing smithi17557870-27 from 716031 to 1339638 tid 1 2024-09-11T03:40:27.860 INFO:tasks.rados.rados.2.smithi175.stdout:576: writing smithi17557870-27 from 2084775 to 2690507 tid 2 2024-09-11T03:40:27.864 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:27.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:27.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:27.866 INFO:tasks.rados.rados.2.smithi175.stdout:576: writing smithi17557870-27 from 3298969 to 3916031 tid 3 2024-09-11T03:40:27.867 INFO:tasks.rados.rados.2.smithi175.stdout:577: snap_remove snap 57 2024-09-11T03:40:27.966 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:27.988 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:40:27.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:27.989 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:27.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:28.090 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:28.100 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:27 smithi121 ceph-mon[30729]: 1.3 scrub starts 2024-09-11T03:40:28.100 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:27 smithi121 ceph-mon[30729]: 1.14 scrub starts 2024-09-11T03:40:28.100 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:27 smithi121 ceph-mon[30729]: 1.5 scrub starts 2024-09-11T03:40:28.100 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:27 smithi121 ceph-mon[30729]: pgmap v469: 37 pgs: 1 active+clean+scrubbing, 6 unknown, 30 active+clean; 138 MiB data, 580 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 4.0 MiB/s wr, 22 op/s 2024-09-11T03:40:28.100 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:27 smithi121 ceph-mon[30729]: 1.2 scrub starts 2024-09-11T03:40:28.100 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:27 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/189600046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:28.100 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:27 smithi121 ceph-mon[30729]: 1.1 scrub starts 2024-09-11T03:40:28.100 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:27 smithi121 ceph-mon[30729]: 1.7 scrub starts 2024-09-11T03:40:28.100 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:27 smithi121 ceph-mon[30729]: osdmap e211: 12 total, 12 up, 10 in 2024-09-11T03:40:28.112 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:28.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:28.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:27 smithi084 ceph-mon[27254]: 1.3 scrub starts 2024-09-11T03:40:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:27 smithi084 ceph-mon[27254]: 1.14 scrub starts 2024-09-11T03:40:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:27 smithi084 ceph-mon[27254]: 1.5 scrub starts 2024-09-11T03:40:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:27 smithi084 ceph-mon[27254]: pgmap v469: 37 pgs: 1 active+clean+scrubbing, 6 unknown, 30 active+clean; 138 MiB data, 580 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 4.0 MiB/s wr, 22 op/s 2024-09-11T03:40:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:27 smithi084 ceph-mon[27254]: 1.2 scrub starts 2024-09-11T03:40:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:27 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/189600046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:27 smithi084 ceph-mon[27254]: 1.1 scrub starts 2024-09-11T03:40:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:27 smithi084 ceph-mon[27254]: 1.7 scrub starts 2024-09-11T03:40:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:27 smithi084 ceph-mon[27254]: osdmap e211: 12 total, 12 up, 10 in 2024-09-11T03:40:28.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:27 smithi175 ceph-mon[30894]: 1.3 scrub starts 2024-09-11T03:40:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:27 smithi175 ceph-mon[30894]: 1.14 scrub starts 2024-09-11T03:40:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:27 smithi175 ceph-mon[30894]: 1.5 scrub starts 2024-09-11T03:40:28.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:27 smithi175 ceph-mon[30894]: pgmap v469: 37 pgs: 1 active+clean+scrubbing, 6 unknown, 30 active+clean; 138 MiB data, 580 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 4.0 MiB/s wr, 22 op/s 2024-09-11T03:40:28.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:27 smithi175 ceph-mon[30894]: 1.2 scrub starts 2024-09-11T03:40:28.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:27 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/189600046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:28.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:27 smithi175 ceph-mon[30894]: 1.1 scrub starts 2024-09-11T03:40:28.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:27 smithi175 ceph-mon[30894]: 1.7 scrub starts 2024-09-11T03:40:28.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:27 smithi175 ceph-mon[30894]: osdmap e211: 12 total, 12 up, 10 in 2024-09-11T03:40:28.231 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:28.235 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:28.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:28.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:28.337 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:28.363 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:28.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:28.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:40:28.434 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:40:28.434 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:40:28.434 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:40:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:40:28.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:28.487 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:28.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:28.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:28.589 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:28.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:28.614 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:28.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:28.715 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:28.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:28.742 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:28.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:28.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:28.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:28.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:28.870 INFO:tasks.rados.rados.2.smithi175.stdout:577: done (1 left) 2024-09-11T03:40:28.871 INFO:tasks.rados.rados.2.smithi175.stdout:578: copy_from oid 20 from oid 11 current snap is 58 2024-09-11T03:40:28.871 INFO:tasks.rados.rados.2.smithi175.stdout:576: finishing write tid 1 to smithi17557870-27 2024-09-11T03:40:28.871 INFO:tasks.rados.rados.2.smithi175.stdout:576: finishing write tid 2 to smithi17557870-27 2024-09-11T03:40:28.871 INFO:tasks.rados.rados.2.smithi175.stdout:576: finishing write tid 3 to smithi17557870-27 2024-09-11T03:40:28.871 INFO:tasks.rados.rados.2.smithi175.stdout:576: finishing write tid 4 to smithi17557870-27 2024-09-11T03:40:28.871 INFO:tasks.rados.rados.2.smithi175.stdout:576: finishing write tid 5 to smithi17557870-27 2024-09-11T03:40:28.871 INFO:tasks.rados.rados.2.smithi175.stdout:576: oid 27 updating version 0 to 157 2024-09-11T03:40:28.871 INFO:tasks.rados.rados.2.smithi175.stdout:576: oid 27 version 157 is already newer than 156 2024-09-11T03:40:28.871 INFO:tasks.rados.rados.2.smithi175.stdout:576: oid 27 version 157 is already newer than 154 2024-09-11T03:40:28.871 INFO:tasks.rados.rados.2.smithi175.stdout:576: oid 27 version 157 is already newer than 155 2024-09-11T03:40:28.871 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 157 (ObjNum 233 snap 58 seq_num 233) dirty exists 2024-09-11T03:40:28.871 INFO:tasks.rados.rados.2.smithi175.stdout:576: left oid 27 (ObjNum 233 snap 58 seq_num 233) 2024-09-11T03:40:28.871 INFO:tasks.rados.rados.2.smithi175.stdout:576: done (1 left) 2024-09-11T03:40:28.871 INFO:tasks.rados.rados.2.smithi175.stdout:579: write (excl) oid 24 current snap is 58 2024-09-11T03:40:28.872 INFO:tasks.rados.rados.2.smithi175.stdout:579: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2024-09-11T03:40:28.878 INFO:tasks.rados.rados.2.smithi175.stdout:579: writing smithi17557870-24 from 732838 to 1431694 tid 1 2024-09-11T03:40:28.884 INFO:tasks.rados.rados.2.smithi175.stdout:579: writing smithi17557870-24 from 2026904 to 2492647 tid 2 2024-09-11T03:40:28.891 INFO:tasks.rados.rados.2.smithi175.stdout:579: writing smithi17557870-24 from 3210039 to 3781690 tid 3 2024-09-11T03:40:28.891 INFO:tasks.rados.rados.2.smithi175.stdout:580: rollback oid 14 current snap is 58 2024-09-11T03:40:28.891 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 14 to 53 2024-09-11T03:40:28.891 INFO:tasks.rados.rados.2.smithi175.stdout:578: finishing copy_from to smithi17557870-20 2024-09-11T03:40:28.891 INFO:tasks.rados.rados.2.smithi175.stdout:578: got expected ENOENT (src dne) 2024-09-11T03:40:28.891 INFO:tasks.rados.rados.2.smithi175.stdout:581: copy_from oid 31 from oid 12 current snap is 58 2024-09-11T03:40:28.891 INFO:tasks.rados.rados.2.smithi175.stdout:578: finishing copy_from racing read to smithi17557870-20 2024-09-11T03:40:28.891 INFO:tasks.rados.rados.2.smithi175.stdout:579: finishing write tid 1 to smithi17557870-24 2024-09-11T03:40:28.892 INFO:tasks.rados.rados.2.smithi175.stdout:579: finishing write tid 2 to smithi17557870-24 2024-09-11T03:40:28.892 INFO:tasks.rados.rados.2.smithi175.stdout:578: done (3 left) 2024-09-11T03:40:28.892 INFO:tasks.rados.rados.2.smithi175.stdout:582: write (excl) oid 45 current snap is 58 2024-09-11T03:40:28.892 INFO:tasks.rados.rados.2.smithi175.stdout:582: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2024-09-11T03:40:28.899 INFO:tasks.rados.rados.2.smithi175.stdout:582: writing smithi17557870-45 from 749645 to 1523750 tid 1 2024-09-11T03:40:28.906 INFO:tasks.rados.rados.2.smithi175.stdout:582: writing smithi17557870-45 from 2085386 to 2694787 tid 2 2024-09-11T03:40:28.913 INFO:tasks.rados.rados.2.smithi175.stdout:582: writing smithi17557870-45 from 3237462 to 3949645 tid 3 2024-09-11T03:40:28.913 INFO:tasks.rados.rados.2.smithi175.stdout:583: delete oid 18 current snap is 58 2024-09-11T03:40:28.913 INFO:tasks.rados.rados.2.smithi175.stdout:581: finishing copy_from to smithi17557870-31 2024-09-11T03:40:28.913 INFO:tasks.rados.rados.2.smithi175.stdout:581: got expected ENOENT (src dne) 2024-09-11T03:40:28.914 INFO:tasks.rados.rados.2.smithi175.stdout:581: finishing copy_from racing read to smithi17557870-31 2024-09-11T03:40:28.914 INFO:tasks.rados.rados.2.smithi175.stdout:579: finishing write tid 3 to smithi17557870-24 2024-09-11T03:40:28.914 INFO:tasks.rados.rados.2.smithi175.stdout:579: finishing write tid 4 to smithi17557870-24 2024-09-11T03:40:28.914 INFO:tasks.rados.rados.2.smithi175.stdout:579: finishing write tid 5 to smithi17557870-24 2024-09-11T03:40:28.914 INFO:tasks.rados.rados.2.smithi175.stdout:579: oid 24 updating version 0 to 161 2024-09-11T03:40:28.914 INFO:tasks.rados.rados.2.smithi175.stdout:579: oid 24 updating version 161 to 162 2024-09-11T03:40:28.914 INFO:tasks.rados.rados.2.smithi175.stdout:579: oid 24 version 162 is already newer than 160 2024-09-11T03:40:28.914 INFO:tasks.rados.rados.2.smithi175.stdout:579: oid 24 updating version 162 to 163 2024-09-11T03:40:28.914 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 163 (ObjNum 234 snap 58 seq_num 234) dirty exists 2024-09-11T03:40:28.914 INFO:tasks.rados.rados.2.smithi175.stdout:579: left oid 24 (ObjNum 234 snap 58 seq_num 234) 2024-09-11T03:40:28.914 INFO:tasks.rados.rados.2.smithi175.stdout:580: finishing rollback tid 0 to smithi17557870-14 2024-09-11T03:40:28.914 INFO:tasks.rados.rados.2.smithi175.stdout:580: finishing rollback tid 1 to smithi17557870-14 2024-09-11T03:40:28.914 INFO:tasks.rados.rados.2.smithi175.stdout:580: finishing rollback tid 2 to smithi17557870-14 2024-09-11T03:40:28.915 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 166 (ObjNum 194 snap 47 seq_num 194) dirty exists 2024-09-11T03:40:28.915 INFO:tasks.rados.rados.2.smithi175.stdout:582: finishing write tid 1 to smithi17557870-45 2024-09-11T03:40:28.915 INFO:tasks.rados.rados.2.smithi175.stdout:582: finishing write tid 2 to smithi17557870-45 2024-09-11T03:40:28.916 INFO:tasks.rados.rados.2.smithi175.stdout:582: finishing write tid 3 to smithi17557870-45 2024-09-11T03:40:28.916 INFO:tasks.rados.rados.2.smithi175.stdout:579: done (4 left) 2024-09-11T03:40:28.916 INFO:tasks.rados.rados.2.smithi175.stdout:580: done (3 left) 2024-09-11T03:40:28.916 INFO:tasks.rados.rados.2.smithi175.stdout:581: done (2 left) 2024-09-11T03:40:28.917 INFO:tasks.rados.rados.2.smithi175.stdout:583: done (1 left) 2024-09-11T03:40:28.917 INFO:tasks.rados.rados.2.smithi175.stdout:584: copy_from oid 38 from oid 28 current snap is 58 2024-09-11T03:40:28.917 INFO:tasks.rados.rados.2.smithi175.stdout:585: delete oid 49 current snap is 58 2024-09-11T03:40:28.917 INFO:tasks.rados.rados.2.smithi175.stdout:582: finishing write tid 4 to smithi17557870-45 2024-09-11T03:40:28.917 INFO:tasks.rados.rados.2.smithi175.stdout:582: finishing write tid 5 to smithi17557870-45 2024-09-11T03:40:28.917 INFO:tasks.rados.rados.2.smithi175.stdout:582: oid 45 updating version 0 to 152 2024-09-11T03:40:28.917 INFO:tasks.rados.rados.2.smithi175.stdout:582: oid 45 version 152 is already newer than 149 2024-09-11T03:40:28.917 INFO:tasks.rados.rados.2.smithi175.stdout:582: oid 45 version 152 is already newer than 151 2024-09-11T03:40:28.917 INFO:tasks.rados.rados.2.smithi175.stdout:582: oid 45 version 152 is already newer than 150 2024-09-11T03:40:28.917 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 152 (ObjNum 235 snap 58 seq_num 235) dirty exists 2024-09-11T03:40:28.917 INFO:tasks.rados.rados.2.smithi175.stdout:582: left oid 45 (ObjNum 235 snap 58 seq_num 235) 2024-09-11T03:40:28.918 INFO:tasks.rados.rados.2.smithi175.stdout:582: done (2 left) 2024-09-11T03:40:28.918 INFO:tasks.rados.rados.2.smithi175.stdout:585: done (1 left) 2024-09-11T03:40:28.918 INFO:tasks.rados.rados.2.smithi175.stdout:586: copy_from oid 47 from oid 29 current snap is 58 2024-09-11T03:40:28.918 INFO:tasks.rados.rados.2.smithi175.stdout:587: read oid 17 snap 53 2024-09-11T03:40:28.918 INFO:tasks.rados.rados.2.smithi175.stdout:587: expect deleted 2024-09-11T03:40:28.918 INFO:tasks.rados.rados.2.smithi175.stdout:588: rollback oid 32 current snap is 58 2024-09-11T03:40:28.919 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 32 to 56 2024-09-11T03:40:28.919 INFO:tasks.rados.rados.2.smithi175.stdout:589: copy_from oid 7 from oid 36 current snap is 58 2024-09-11T03:40:28.919 INFO:tasks.rados.rados.2.smithi175.stdout:590: write (excl) oid 5 current snap is 58 2024-09-11T03:40:28.919 INFO:tasks.rados.rados.2.smithi175.stdout:590: seq_num 238 ranges {400066=716205,1860832=139234} 2024-09-11T03:40:28.925 INFO:tasks.rados.rados.2.smithi175.stdout:590: writing smithi17557870-5 from 400066 to 1116271 tid 1 2024-09-11T03:40:28.932 INFO:tasks.rados.rados.2.smithi175.stdout:590: writing smithi17557870-5 from 1860832 to 2000066 tid 2 2024-09-11T03:40:28.932 INFO:tasks.rados.rados.2.smithi175.stdout:591: rollback oid 49 current snap is 58 2024-09-11T03:40:28.932 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 49 to 53 2024-09-11T03:40:28.932 INFO:tasks.rados.rados.2.smithi175.stdout:588: finishing rollback tid 0 to smithi17557870-32 2024-09-11T03:40:28.932 INFO:tasks.rados.rados.2.smithi175.stdout:588: finishing rollback tid 1 to smithi17557870-32 2024-09-11T03:40:28.932 INFO:tasks.rados.rados.2.smithi175.stdout:588: finishing rollback tid 2 to smithi17557870-32 2024-09-11T03:40:28.932 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 152 (ObjNum 207 snap 50 seq_num 207) dirty exists 2024-09-11T03:40:28.932 INFO:tasks.rados.rados.2.smithi175.stdout:586: finishing copy_from racing read to smithi17557870-47 2024-09-11T03:40:28.932 INFO:tasks.rados.rados.2.smithi175.stdout:586: finishing copy_from to smithi17557870-47 2024-09-11T03:40:28.932 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 97 (ObjNum 132 snap 22 seq_num 132) dirty exists 2024-09-11T03:40:28.932 INFO:tasks.rados.rados.2.smithi175.stdout:589: finishing copy_from racing read to smithi17557870-7 2024-09-11T03:40:28.933 INFO:tasks.rados.rados.2.smithi175.stdout:586: done (6 left) 2024-09-11T03:40:28.933 INFO:tasks.rados.rados.2.smithi175.stdout:587: done (5 left) 2024-09-11T03:40:28.933 INFO:tasks.rados.rados.2.smithi175.stdout:588: done (4 left) 2024-09-11T03:40:28.933 INFO:tasks.rados.rados.2.smithi175.stdout:592: write (excl) oid 46 current snap is 58 2024-09-11T03:40:28.933 INFO:tasks.rados.rados.2.smithi175.stdout:592: seq_num 239 ranges {416873=791454,1919314=97559} 2024-09-11T03:40:28.938 INFO:tasks.rados.rados.2.smithi175.stdout:592: writing smithi17557870-46 from 416873 to 1208327 tid 1 2024-09-11T03:40:28.943 INFO:tasks.rados.rados.2.smithi175.stdout:592: writing smithi17557870-46 from 1919314 to 2016873 tid 2 2024-09-11T03:40:28.943 INFO:tasks.rados.rados.2.smithi175.stdout:593: delete oid 34 current snap is 58 2024-09-11T03:40:28.943 INFO:tasks.rados.rados.2.smithi175.stdout:584: finishing copy_from racing read to smithi17557870-38 2024-09-11T03:40:28.943 INFO:tasks.rados.rados.2.smithi175.stdout:591: finishing rollback tid 1 to smithi17557870-49 2024-09-11T03:40:28.943 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 153 (ObjNum 33 snap 0 seq_num 1713071920) dirty dne 2024-09-11T03:40:28.943 INFO:tasks.rados.rados.2.smithi175.stdout:589: finishing copy_from to smithi17557870-7 2024-09-11T03:40:28.943 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 123 (ObjNum 150 snap 35 seq_num 150) dirty exists 2024-09-11T03:40:28.943 INFO:tasks.rados.rados.2.smithi175.stdout:590: finishing write tid 1 to smithi17557870-5 2024-09-11T03:40:28.943 INFO:tasks.rados.rados.2.smithi175.stdout:590: finishing write tid 2 to smithi17557870-5 2024-09-11T03:40:28.943 INFO:tasks.rados.rados.2.smithi175.stdout:590: finishing write tid 3 to smithi17557870-5 2024-09-11T03:40:28.943 INFO:tasks.rados.rados.2.smithi175.stdout:590: finishing write tid 4 to smithi17557870-5 2024-09-11T03:40:28.944 INFO:tasks.rados.rados.2.smithi175.stdout:590: oid 5 updating version 0 to 156 2024-09-11T03:40:28.944 INFO:tasks.rados.rados.2.smithi175.stdout:590: oid 5 version 156 is already newer than 154 2024-09-11T03:40:28.944 INFO:tasks.rados.rados.2.smithi175.stdout:590: oid 5 version 156 is already newer than 155 2024-09-11T03:40:28.944 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 156 (ObjNum 238 snap 58 seq_num 238) dirty exists 2024-09-11T03:40:28.944 INFO:tasks.rados.rados.2.smithi175.stdout:590: left oid 5 (ObjNum 238 snap 58 seq_num 238) 2024-09-11T03:40:28.944 INFO:tasks.rados.rados.2.smithi175.stdout:584: finishing copy_from to smithi17557870-38 2024-09-11T03:40:28.944 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 208 (ObjNum 228 snap 56 seq_num 228) dirty exists 2024-09-11T03:40:28.944 INFO:tasks.rados.rados.2.smithi175.stdout:592: finishing write tid 1 to smithi17557870-46 2024-09-11T03:40:28.945 INFO:tasks.rados.rados.2.smithi175.stdout:584: done (5 left) 2024-09-11T03:40:28.945 INFO:tasks.rados.rados.2.smithi175.stdout:589: done (4 left) 2024-09-11T03:40:28.945 INFO:tasks.rados.rados.2.smithi175.stdout:590: done (3 left) 2024-09-11T03:40:28.945 INFO:tasks.rados.rados.2.smithi175.stdout:591: done (2 left) 2024-09-11T03:40:28.945 INFO:tasks.rados.rados.2.smithi175.stdout:593: done (1 left) 2024-09-11T03:40:28.945 INFO:tasks.rados.rados.2.smithi175.stdout:594: write (excl) oid 27 current snap is 58 2024-09-11T03:40:28.945 INFO:tasks.rados.rados.2.smithi175.stdout:594: seq_num 241 ranges {450487=541952,1636278=414209} 2024-09-11T03:40:28.949 INFO:tasks.rados.rados.2.smithi175.stdout:594: writing smithi17557870-27 from 450487 to 992439 tid 1 2024-09-11T03:40:28.955 INFO:tasks.rados.rados.2.smithi175.stdout:594: writing smithi17557870-27 from 1636278 to 2050487 tid 2 2024-09-11T03:40:28.955 INFO:tasks.rados.rados.2.smithi175.stdout:595: snap_remove snap 51 2024-09-11T03:40:28.969 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:28.990 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:28.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:28.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:29.092 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:29.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:29.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:29.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:28 smithi121 ceph-mon[30729]: 1.3 scrub starts 2024-09-11T03:40:29.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:28 smithi121 ceph-mon[30729]: 1.4 scrub starts 2024-09-11T03:40:29.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:28 smithi121 ceph-mon[30729]: 1.5 scrub starts 2024-09-11T03:40:29.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:28 smithi121 ceph-mon[30729]: 1.0 scrub starts 2024-09-11T03:40:29.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:28 smithi121 ceph-mon[30729]: 1.6 scrub starts 2024-09-11T03:40:29.133 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:28 smithi121 ceph-mon[30729]: 1.c scrub starts 2024-09-11T03:40:29.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T03:40:29.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:28 smithi084 ceph-mon[27254]: 1.3 scrub starts 2024-09-11T03:40:29.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:28 smithi084 ceph-mon[27254]: 1.4 scrub starts 2024-09-11T03:40:29.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:28 smithi084 ceph-mon[27254]: 1.5 scrub starts 2024-09-11T03:40:29.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:28 smithi084 ceph-mon[27254]: 1.0 scrub starts 2024-09-11T03:40:29.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:28 smithi084 ceph-mon[27254]: 1.6 scrub starts 2024-09-11T03:40:29.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:28 smithi084 ceph-mon[27254]: 1.c scrub starts 2024-09-11T03:40:29.217 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:29.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:28 smithi175 ceph-mon[30894]: 1.3 scrub starts 2024-09-11T03:40:29.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:28 smithi175 ceph-mon[30894]: 1.4 scrub starts 2024-09-11T03:40:29.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:28 smithi175 ceph-mon[30894]: 1.5 scrub starts 2024-09-11T03:40:29.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:28 smithi175 ceph-mon[30894]: 1.0 scrub starts 2024-09-11T03:40:29.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:28 smithi175 ceph-mon[30894]: 1.6 scrub starts 2024-09-11T03:40:29.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:28 smithi175 ceph-mon[30894]: 1.c scrub starts 2024-09-11T03:40:29.238 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:29.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:29.340 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:29.367 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:29.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:29.412 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:40:29.468 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:29.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:29.494 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:29.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:29.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:29.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:29.620 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:29.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:29.721 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:29.744 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:29.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:29.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:29.841 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:40:29.841 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:40:29.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:40:29.841 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:40:29.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.842 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDScrubReserve(1.0,QREQUEST,e:65,from:7,reservation_nonce:2)", 2024-09-11T03:40:29.842 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:36:27.978100+0000", 2024-09-11T03:40:29.842 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 241.86265284800001, 2024-09-11T03:40:29.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012126909999999999, 2024-09-11T03:40:29.842 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.842 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.842 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:40:29.842 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.978100+0000", 2024-09-11T03:40:29.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.978100+0000", 2024-09-11T03:40:29.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.978101+0000", 2024-09-11T03:40:29.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.029e-06 2024-09-11T03:40:29.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.978106+0000", 2024-09-11T03:40:29.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5029999999999997e-06 2024-09-11T03:40:29.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.978121+0000", 2024-09-11T03:40:29.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4329e-05 2024-09-11T03:40:29.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.978130+0000", 2024-09-11T03:40:29.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9579999999999996e-06 2024-09-11T03:40:29.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.978198+0000", 2024-09-11T03:40:29.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8762000000000003e-05 2024-09-11T03:40:29.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:40:29.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.978204+0000", 2024-09-11T03:40:29.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2939999999999996e-06 2024-09-11T03:40:29.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.979261+0000", 2024-09-11T03:40:29.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010568929999999999 2024-09-11T03:40:29.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.979279+0000", 2024-09-11T03:40:29.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8453000000000001e-05 2024-09-11T03:40:29.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.979313+0000", 2024-09-11T03:40:29.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3470000000000003e-05 2024-09-11T03:40:29.846 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.846 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.846 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.847 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:225 2.2 e103/95 2:43d79ac9:::smithi17557870-37:head v 103'11, mlcod=100'8)", 2024-09-11T03:40:29.847 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.943277+0000", 2024-09-11T03:40:29.847 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 109.897475265, 2024-09-11T03:40:29.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000975161, 2024-09-11T03:40:29.847 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.847 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.847 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:29.847 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943277+0000", 2024-09-11T03:40:29.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943277+0000", 2024-09-11T03:40:29.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943278+0000", 2024-09-11T03:40:29.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5800000000000001e-07 2024-09-11T03:40:29.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943283+0000", 2024-09-11T03:40:29.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8640000000000004e-06 2024-09-11T03:40:29.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943298+0000", 2024-09-11T03:40:29.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.506e-05 2024-09-11T03:40:29.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943308+0000", 2024-09-11T03:40:29.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0821e-05 2024-09-11T03:40:29.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943381+0000", 2024-09-11T03:40:29.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3028000000000006e-05 2024-09-11T03:40:29.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943406+0000", 2024-09-11T03:40:29.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4119999999999999e-05 2024-09-11T03:40:29.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:29.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.944218+0000", 2024-09-11T03:40:29.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081275499999999997 2024-09-11T03:40:29.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.944253+0000", 2024-09-11T03:40:29.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4255000000000003e-05 2024-09-11T03:40:29.851 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.851 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.851 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.851 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(osd.5.0:18 2.2 e104/95 2:43d79ac9:::smithi17557870-37:3 v 104'13, mlcod=100'9)", 2024-09-11T03:40:29.851 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:41.272262+0000", 2024-09-11T03:40:29.851 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 108.568490365, 2024-09-11T03:40:29.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00099929800000000003, 2024-09-11T03:40:29.852 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.852 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.852 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:29.852 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.272262+0000", 2024-09-11T03:40:29.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.272262+0000", 2024-09-11T03:40:29.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.272263+0000", 2024-09-11T03:40:29.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0500000000000004e-07 2024-09-11T03:40:29.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.272270+0000", 2024-09-11T03:40:29.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8340000000000001e-06 2024-09-11T03:40:29.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.272283+0000", 2024-09-11T03:40:29.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2981e-05 2024-09-11T03:40:29.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.272295+0000", 2024-09-11T03:40:29.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2116999999999999e-05 2024-09-11T03:40:29.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.272383+0000", 2024-09-11T03:40:29.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7831999999999995e-05 2024-09-11T03:40:29.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.272406+0000", 2024-09-11T03:40:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3887e-05 2024-09-11T03:40:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.273199+0000", 2024-09-11T03:40:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079215399999999999 2024-09-11T03:40:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.273262+0000", 2024-09-11T03:40:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2988000000000001e-05 2024-09-11T03:40:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.856 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.856 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.856 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:343 2.2 e113/95 2:494ba932:::smithi17557870-43:head v 113'20, mlcod=107'17)", 2024-09-11T03:40:29.856 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.831626+0000", 2024-09-11T03:40:29.856 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:29.857 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 100.00912696899999, 2024-09-11T03:40:29.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035769930000000001, 2024-09-11T03:40:29.858 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.858 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.858 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:29.858 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.831626+0000", 2024-09-11T03:40:29.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.831626+0000", 2024-09-11T03:40:29.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.831626+0000", 2024-09-11T03:40:29.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2999999999999999e-07 2024-09-11T03:40:29.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833432+0000", 2024-09-11T03:40:29.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018063350000000001 2024-09-11T03:40:29.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833447+0000", 2024-09-11T03:40:29.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5065999999999999e-05 2024-09-11T03:40:29.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833455+0000", 2024-09-11T03:40:29.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2930000000000003e-06 2024-09-11T03:40:29.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833512+0000", 2024-09-11T03:40:29.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7648000000000001e-05 2024-09-11T03:40:29.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833538+0000", 2024-09-11T03:40:29.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5962e-05 2024-09-11T03:40:29.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:29.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.835162+0000", 2024-09-11T03:40:29.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016233580000000001 2024-09-11T03:40:29.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.835203+0000", 2024-09-11T03:40:29.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1100999999999998e-05 2024-09-11T03:40:29.863 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.863 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.863 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.863 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:366 2.7 2:e0eabcfb:::smithi17557870-9:head [write 410104~465105,stat] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:40:29.864 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:50.854890+0000", 2024-09-11T03:40:29.864 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.985862678999993, 2024-09-11T03:40:29.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040276269999999998, 2024-09-11T03:40:29.864 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.864 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.864 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:29.864 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:40:29.864 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:40:29.864 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:29.864 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 366 2024-09-11T03:40:29.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.864 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.854890+0000", 2024-09-11T03:40:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.854890+0000", 2024-09-11T03:40:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.854892+0000", 2024-09-11T03:40:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.629e-06 2024-09-11T03:40:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.855622+0000", 2024-09-11T03:40:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073058800000000001 2024-09-11T03:40:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.855638+0000", 2024-09-11T03:40:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5648e-05 2024-09-11T03:40:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.855662+0000", 2024-09-11T03:40:29.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4620000000000001e-05 2024-09-11T03:40:29.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.855717+0000", 2024-09-11T03:40:29.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4342000000000002e-05 2024-09-11T03:40:29.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.855789+0000", 2024-09-11T03:40:29.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2550000000000002e-05 2024-09-11T03:40:29.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:40:29.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.855901+0000", 2024-09-11T03:40:29.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111352 2024-09-11T03:40:29.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:40:29.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.857414+0000", 2024-09-11T03:40:29.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015131159999999999 2024-09-11T03:40:29.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.858043+0000", 2024-09-11T03:40:29.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00062914299999999998 2024-09-11T03:40:29.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.858836+0000", 2024-09-11T03:40:29.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079319899999999997 2024-09-11T03:40:29.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:29.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.858871+0000", 2024-09-11T03:40:29.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4523999999999997e-05 2024-09-11T03:40:29.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.858918+0000", 2024-09-11T03:40:29.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6916000000000003e-05 2024-09-11T03:40:29.870 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.870 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.870 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.870 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:367 2.7 2:e0eabcfb:::smithi17557870-9:head [write 1563527~583025] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:40:29.870 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:50.861918+0000", 2024-09-11T03:40:29.870 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.978835122000007, 2024-09-11T03:40:29.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035522819999999999, 2024-09-11T03:40:29.870 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.870 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.870 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:29.870 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:40:29.871 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:40:29.871 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:29.871 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 367 2024-09-11T03:40:29.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.871 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.861918+0000", 2024-09-11T03:40:29.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.861918+0000", 2024-09-11T03:40:29.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.861919+0000", 2024-09-11T03:40:29.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0300000000000001e-06 2024-09-11T03:40:29.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.862732+0000", 2024-09-11T03:40:29.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081320100000000005 2024-09-11T03:40:29.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.862745+0000", 2024-09-11T03:40:29.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2801e-05 2024-09-11T03:40:29.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.862750+0000", 2024-09-11T03:40:29.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5729999999999998e-06 2024-09-11T03:40:29.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.862843+0000", 2024-09-11T03:40:29.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3033000000000002e-05 2024-09-11T03:40:29.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.862912+0000", 2024-09-11T03:40:29.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8616999999999997e-05 2024-09-11T03:40:29.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:40:29.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.862997+0000", 2024-09-11T03:40:29.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4975000000000005e-05 2024-09-11T03:40:29.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:40:29.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.864288+0000", 2024-09-11T03:40:29.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001291237 2024-09-11T03:40:29.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.865104+0000", 2024-09-11T03:40:29.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081632099999999995 2024-09-11T03:40:29.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.865395+0000", 2024-09-11T03:40:29.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00029032800000000002 2024-09-11T03:40:29.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:29.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.865428+0000", 2024-09-11T03:40:29.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3327000000000001e-05 2024-09-11T03:40:29.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.865470+0000", 2024-09-11T03:40:29.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1838999999999997e-05 2024-09-11T03:40:29.876 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.876 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.876 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.876 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:368 2.7 2:e0eabcfb:::smithi17557870-9:head [write 2810926~399178] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:40:29.877 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:50.867602+0000", 2024-09-11T03:40:29.877 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.973150644, 2024-09-11T03:40:29.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024245920000000002, 2024-09-11T03:40:29.877 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.877 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.877 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:29.877 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:40:29.877 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:40:29.877 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:29.877 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 368 2024-09-11T03:40:29.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.877 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.867602+0000", 2024-09-11T03:40:29.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.867602+0000", 2024-09-11T03:40:29.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.867603+0000", 2024-09-11T03:40:29.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0410000000000001e-06 2024-09-11T03:40:29.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868107+0000", 2024-09-11T03:40:29.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050433300000000004 2024-09-11T03:40:29.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868118+0000", 2024-09-11T03:40:29.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0798999999999999e-05 2024-09-11T03:40:29.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868123+0000", 2024-09-11T03:40:29.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2429999999999996e-06 2024-09-11T03:40:29.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868196+0000", 2024-09-11T03:40:29.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2529999999999998e-05 2024-09-11T03:40:29.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868260+0000", 2024-09-11T03:40:29.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3569000000000001e-05 2024-09-11T03:40:29.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:40:29.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868344+0000", 2024-09-11T03:40:29.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4608e-05 2024-09-11T03:40:29.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:40:29.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869414+0000", 2024-09-11T03:40:29.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010696029999999999 2024-09-11T03:40:29.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869737+0000", 2024-09-11T03:40:29.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032352800000000001 2024-09-11T03:40:29.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869952+0000", 2024-09-11T03:40:29.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021448800000000001 2024-09-11T03:40:29.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:29.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869984+0000", 2024-09-11T03:40:29.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2406000000000001e-05 2024-09-11T03:40:29.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.870027+0000", 2024-09-11T03:40:29.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2444000000000002e-05 2024-09-11T03:40:29.883 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.883 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.883 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.883 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:369 2.7 2:e0eabcfb:::smithi17557870-9:head [setxattr _header (55) in=7b,truncate 3210104] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:40:29.883 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:50.868138+0000", 2024-09-11T03:40:29.883 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.972614985000007, 2024-09-11T03:40:29.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019888039999999998, 2024-09-11T03:40:29.883 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.883 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.883 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:29.883 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:40:29.883 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:40:29.884 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:29.884 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 369 2024-09-11T03:40:29.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.884 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868138+0000", 2024-09-11T03:40:29.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868138+0000", 2024-09-11T03:40:29.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868138+0000", 2024-09-11T03:40:29.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0200000000000001e-07 2024-09-11T03:40:29.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868159+0000", 2024-09-11T03:40:29.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0874e-05 2024-09-11T03:40:29.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868173+0000", 2024-09-11T03:40:29.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3801999999999999e-05 2024-09-11T03:40:29.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868180+0000", 2024-09-11T03:40:29.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4960000000000001e-06 2024-09-11T03:40:29.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868995+0000", 2024-09-11T03:40:29.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081590999999999999 2024-09-11T03:40:29.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869060+0000", 2024-09-11T03:40:29.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4568999999999998e-05 2024-09-11T03:40:29.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:40:29.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869147+0000", 2024-09-11T03:40:29.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6543000000000001e-05 2024-09-11T03:40:29.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:40:29.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869416+0000", 2024-09-11T03:40:29.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00026988700000000001 2024-09-11T03:40:29.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869773+0000", 2024-09-11T03:40:29.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035655000000000002 2024-09-11T03:40:29.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.870057+0000", 2024-09-11T03:40:29.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028380700000000002 2024-09-11T03:40:29.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:29.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.870090+0000", 2024-09-11T03:40:29.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3311e-05 2024-09-11T03:40:29.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.870127+0000", 2024-09-11T03:40:29.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6353000000000002e-05 2024-09-11T03:40:29.889 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.889 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.889 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.890 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:370 2.7 2:e0eabcfb:::smithi17557870-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:40:29.890 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:50.868190+0000", 2024-09-11T03:40:29.890 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.972562722999996, 2024-09-11T03:40:29.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020750069999999998, 2024-09-11T03:40:29.890 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.890 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.890 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:40:29.890 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:40:29.890 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:40:29.890 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:29.890 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 370 2024-09-11T03:40:29.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.890 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868190+0000", 2024-09-11T03:40:29.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868190+0000", 2024-09-11T03:40:29.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868190+0000", 2024-09-11T03:40:29.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3399999999999999e-07 2024-09-11T03:40:29.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868193+0000", 2024-09-11T03:40:29.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.227e-06 2024-09-11T03:40:29.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868199+0000", 2024-09-11T03:40:29.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7730000000000002e-06 2024-09-11T03:40:29.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868205+0000", 2024-09-11T03:40:29.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9390000000000002e-06 2024-09-11T03:40:29.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869359+0000", 2024-09-11T03:40:29.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011531169999999999 2024-09-11T03:40:29.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:40:29.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869405+0000", 2024-09-11T03:40:29.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6925999999999998e-05 2024-09-11T03:40:29.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.870161+0000", 2024-09-11T03:40:29.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075579900000000003 2024-09-11T03:40:29.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.870205+0000", 2024-09-11T03:40:29.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3662000000000003e-05 2024-09-11T03:40:29.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.870265+0000", 2024-09-11T03:40:29.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.003e-05 2024-09-11T03:40:29.895 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.895 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.895 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.895 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:402 2.7 2:e23d1762:::smithi17557870-21:head [copy-from ver 9] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:40:29.895 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:51.085659+0000", 2024-09-11T03:40:29.895 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.755093844000001, 2024-09-11T03:40:29.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012118869000000001, 2024-09-11T03:40:29.895 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.895 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.895 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:29.895 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:40:29.895 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:40:29.896 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:29.896 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 402 2024-09-11T03:40:29.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.896 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085659+0000", 2024-09-11T03:40:29.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085659+0000", 2024-09-11T03:40:29.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085661+0000", 2024-09-11T03:40:29.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1220000000000002e-06 2024-09-11T03:40:29.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085669+0000", 2024-09-11T03:40:29.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2779999999999993e-06 2024-09-11T03:40:29.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085689+0000", 2024-09-11T03:40:29.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9639999999999998e-05 2024-09-11T03:40:29.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085699+0000", 2024-09-11T03:40:29.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0524e-05 2024-09-11T03:40:29.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085789+0000", 2024-09-11T03:40:29.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9215000000000006e-05 2024-09-11T03:40:29.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085880+0000", 2024-09-11T03:40:29.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1187999999999995e-05 2024-09-11T03:40:29.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:40:29.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.090839+0000", 2024-09-11T03:40:29.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004958863 2024-09-11T03:40:29.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:40:29.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.093676+0000", 2024-09-11T03:40:29.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028368930000000001 2024-09-11T03:40:29.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.097457+0000", 2024-09-11T03:40:29.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003781197 2024-09-11T03:40:29.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.097707+0000", 2024-09-11T03:40:29.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024975900000000001 2024-09-11T03:40:29.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:29.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.097745+0000", 2024-09-11T03:40:29.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8596000000000001e-05 2024-09-11T03:40:29.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.097778+0000", 2024-09-11T03:40:29.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2594e-05 2024-09-11T03:40:29.902 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.902 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.902 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.902 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:403 2.7 2:e23d1762:::smithi17557870-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:40:29.902 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:51.085720+0000", 2024-09-11T03:40:29.902 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.755033023999999, 2024-09-11T03:40:29.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076054490000000002, 2024-09-11T03:40:29.902 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.902 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.902 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:40:29.902 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:40:29.902 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:40:29.903 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:29.903 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 403 2024-09-11T03:40:29.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.903 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085720+0000", 2024-09-11T03:40:29.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085720+0000", 2024-09-11T03:40:29.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085720+0000", 2024-09-11T03:40:29.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5599999999999995e-07 2024-09-11T03:40:29.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085742+0000", 2024-09-11T03:40:29.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1968e-05 2024-09-11T03:40:29.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085750+0000", 2024-09-11T03:40:29.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8699999999999992e-06 2024-09-11T03:40:29.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085756+0000", 2024-09-11T03:40:29.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-09-11T03:40:29.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.086016+0000", 2024-09-11T03:40:29.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00026035099999999997 2024-09-11T03:40:29.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:40:29.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.086071+0000", 2024-09-11T03:40:29.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4614999999999998e-05 2024-09-11T03:40:29.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.093154+0000", 2024-09-11T03:40:29.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070835050000000004 2024-09-11T03:40:29.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.093239+0000", 2024-09-11T03:40:29.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5309999999999995e-05 2024-09-11T03:40:29.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.093325+0000", 2024-09-11T03:40:29.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5654000000000004e-05 2024-09-11T03:40:29.907 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.907 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.907 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.907 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:405 2.7 2:e6da0655:::smithi17557870-27:head [delete] snapc 9=[9] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115)", 2024-09-11T03:40:29.907 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:51.814169+0000", 2024-09-11T03:40:29.907 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.026583608999999, 2024-09-11T03:40:29.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030515849999999999, 2024-09-11T03:40:29.907 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.908 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.908 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:29.908 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:40:29.908 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:40:29.908 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:29.908 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 405 2024-09-11T03:40:29.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.908 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.814169+0000", 2024-09-11T03:40:29.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.814169+0000", 2024-09-11T03:40:29.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.814171+0000", 2024-09-11T03:40:29.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.844e-06 2024-09-11T03:40:29.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.814178+0000", 2024-09-11T03:40:29.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-09-11T03:40:29.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.814194+0000", 2024-09-11T03:40:29.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5943e-05 2024-09-11T03:40:29.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.814221+0000", 2024-09-11T03:40:29.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6659999999999999e-05 2024-09-11T03:40:29.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.814291+0000", 2024-09-11T03:40:29.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0388000000000004e-05 2024-09-11T03:40:29.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:40:29.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.814298+0000", 2024-09-11T03:40:29.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4659999999999996e-06 2024-09-11T03:40:29.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.815820+0000", 2024-09-11T03:40:29.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015221639999999999 2024-09-11T03:40:29.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.815911+0000", 2024-09-11T03:40:29.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0989000000000006e-05 2024-09-11T03:40:29.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:40:29.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.816069+0000", 2024-09-11T03:40:29.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000158436 2024-09-11T03:40:29.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:40:29.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.816751+0000", 2024-09-11T03:40:29.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00068151400000000003 2024-09-11T03:40:29.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.816971+0000", 2024-09-11T03:40:29.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022063399999999999 2024-09-11T03:40:29.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.817149+0000", 2024-09-11T03:40:29.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017768100000000001 2024-09-11T03:40:29.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:29.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.817187+0000", 2024-09-11T03:40:29.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7889999999999998e-05 2024-09-11T03:40:29.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.817221+0000", 2024-09-11T03:40:29.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3640000000000003e-05 2024-09-11T03:40:29.914 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.914 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.914 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.914 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:2 2.7 2:e0ec5763:::smithi17557870-2:head [assert-version v3,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e119)", 2024-09-11T03:40:29.914 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:55.859247+0000", 2024-09-11T03:40:29.915 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 93.981506182000004, 2024-09-11T03:40:29.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045129180000000003, 2024-09-11T03:40:29.915 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.915 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.915 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:40:29.915 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.859247+0000", 2024-09-11T03:40:29.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.859247+0000", 2024-09-11T03:40:29.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.859248+0000", 2024-09-11T03:40:29.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6899999999999999e-06 2024-09-11T03:40:29.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.859254+0000", 2024-09-11T03:40:29.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7649999999999999e-06 2024-09-11T03:40:29.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.859264+0000", 2024-09-11T03:40:29.918 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:40:29.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0244000000000001e-05 2024-09-11T03:40:29.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.859274+0000", 2024-09-11T03:40:29.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7839999999999998e-06 2024-09-11T03:40:29.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.859345+0000", 2024-09-11T03:40:29.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0584e-05 2024-09-11T03:40:29.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:40:29.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.859351+0000", 2024-09-11T03:40:29.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2319999999999998e-06 2024-09-11T03:40:29.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.860628+0000", 2024-09-11T03:40:29.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001277542 2024-09-11T03:40:29.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.860700+0000", 2024-09-11T03:40:29.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1524000000000002e-05 2024-09-11T03:40:29.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.863759+0000", 2024-09-11T03:40:29.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003059553 2024-09-11T03:40:29.921 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.921 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.921 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.921 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:583 2.2 e128/95 2:494ba932:::smithi17557870-43:head v 128'41, mlcod=124'37)", 2024-09-11T03:40:29.921 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:04.877155+0000", 2024-09-11T03:40:29.921 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.963597282999999, 2024-09-11T03:40:29.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00099352800000000003, 2024-09-11T03:40:29.921 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.921 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.922 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:29.922 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.877155+0000", 2024-09-11T03:40:29.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.877155+0000", 2024-09-11T03:40:29.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.877156+0000", 2024-09-11T03:40:29.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2799999999999996e-07 2024-09-11T03:40:29.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.877171+0000", 2024-09-11T03:40:29.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.507e-05 2024-09-11T03:40:29.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.877199+0000", 2024-09-11T03:40:29.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8407000000000001e-05 2024-09-11T03:40:29.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.877209+0000", 2024-09-11T03:40:29.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-09-11T03:40:29.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.877287+0000", 2024-09-11T03:40:29.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8559999999999993e-05 2024-09-11T03:40:29.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.877310+0000", 2024-09-11T03:40:29.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2872000000000001e-05 2024-09-11T03:40:29.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:29.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.878111+0000", 2024-09-11T03:40:29.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00080112399999999996 2024-09-11T03:40:29.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.878149+0000", 2024-09-11T03:40:29.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7755e-05 2024-09-11T03:40:29.925 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.925 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.925 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.926 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(osd.5.0:45 2.2 e130/95 2:494ba932:::smithi17557870-43:11 v 130'45, mlcod=128'41)", 2024-09-11T03:40:29.926 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:06.816260+0000", 2024-09-11T03:40:29.926 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.024492257000006, 2024-09-11T03:40:29.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001130898, 2024-09-11T03:40:29.926 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.926 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.926 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:29.926 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.816260+0000", 2024-09-11T03:40:29.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.816260+0000", 2024-09-11T03:40:29.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.816261+0000", 2024-09-11T03:40:29.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1900000000000002e-07 2024-09-11T03:40:29.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.816268+0000", 2024-09-11T03:40:29.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0090000000000001e-06 2024-09-11T03:40:29.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.816280+0000", 2024-09-11T03:40:29.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1783999999999999e-05 2024-09-11T03:40:29.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.816288+0000", 2024-09-11T03:40:29.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1969999999999996e-06 2024-09-11T03:40:29.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.816357+0000", 2024-09-11T03:40:29.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8779e-05 2024-09-11T03:40:29.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.816381+0000", 2024-09-11T03:40:29.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3634000000000001e-05 2024-09-11T03:40:29.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:29.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.817334+0000", 2024-09-11T03:40:29.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00095372000000000002 2024-09-11T03:40:29.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.817391+0000", 2024-09-11T03:40:29.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7055999999999997e-05 2024-09-11T03:40:29.930 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.930 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.930 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.930 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:613 2.7 2:e23d1762:::smithi17557870-21:head [stat,delete] snapc 11=[f,e] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-11T03:40:29.930 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:07.043261+0000", 2024-09-11T03:40:29.930 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 82.797491773999994, 2024-09-11T03:40:29.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040861639999999998, 2024-09-11T03:40:29.931 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.931 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.931 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:29.931 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:40:29.931 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:40:29.931 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:29.931 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 613 2024-09-11T03:40:29.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.931 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043261+0000", 2024-09-11T03:40:29.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043261+0000", 2024-09-11T03:40:29.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043264+0000", 2024-09-11T03:40:29.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7769999999999998e-06 2024-09-11T03:40:29.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043275+0000", 2024-09-11T03:40:29.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1063000000000001e-05 2024-09-11T03:40:29.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043306+0000", 2024-09-11T03:40:29.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1327e-05 2024-09-11T03:40:29.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043322+0000", 2024-09-11T03:40:29.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5602000000000001e-05 2024-09-11T03:40:29.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043458+0000", 2024-09-11T03:40:29.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000136036 2024-09-11T03:40:29.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043552+0000", 2024-09-11T03:40:29.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4058e-05 2024-09-11T03:40:29.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:40:29.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043693+0000", 2024-09-11T03:40:29.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014124 2024-09-11T03:40:29.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:40:29.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.044599+0000", 2024-09-11T03:40:29.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090568099999999998 2024-09-11T03:40:29.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.045227+0000", 2024-09-11T03:40:29.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00062812899999999995 2024-09-11T03:40:29.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.047232+0000", 2024-09-11T03:40:29.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020049759999999999 2024-09-11T03:40:29.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:29.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.047299+0000", 2024-09-11T03:40:29.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6795e-05 2024-09-11T03:40:29.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.047347+0000", 2024-09-11T03:40:29.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8479999999999997e-05 2024-09-11T03:40:29.936 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.936 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.936 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.937 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:631 2.7 2:e6da0655:::smithi17557870-27:head [append 0~0] snapc 11=[e] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:40:29.937 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:07.867549+0000", 2024-09-11T03:40:29.937 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 81.973203566999999, 2024-09-11T03:40:29.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024185919999999998, 2024-09-11T03:40:29.937 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.937 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.937 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:29.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:40:29.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:40:29.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:29.937 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 631 2024-09-11T03:40:29.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.938 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867549+0000", 2024-09-11T03:40:29.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867549+0000", 2024-09-11T03:40:29.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867552+0000", 2024-09-11T03:40:29.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7e-06 2024-09-11T03:40:29.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867562+0000", 2024-09-11T03:40:29.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0226e-05 2024-09-11T03:40:29.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867593+0000", 2024-09-11T03:40:29.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1235999999999997e-05 2024-09-11T03:40:29.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867609+0000", 2024-09-11T03:40:29.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5324000000000001e-05 2024-09-11T03:40:29.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867761+0000", 2024-09-11T03:40:29.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015238799999999999 2024-09-11T03:40:29.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867971+0000", 2024-09-11T03:40:29.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00020959900000000001 2024-09-11T03:40:29.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:40:29.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.868164+0000", 2024-09-11T03:40:29.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019307800000000001 2024-09-11T03:40:29.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869787+0000", 2024-09-11T03:40:29.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016228600000000001 2024-09-11T03:40:29.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:40:29.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869801+0000", 2024-09-11T03:40:29.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4637999999999999e-05 2024-09-11T03:40:29.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869895+0000", 2024-09-11T03:40:29.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3615000000000004e-05 2024-09-11T03:40:29.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:29.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869933+0000", 2024-09-11T03:40:29.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8189000000000003e-05 2024-09-11T03:40:29.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869968+0000", 2024-09-11T03:40:29.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4739000000000001e-05 2024-09-11T03:40:29.943 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.943 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.943 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.943 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:632 2.7 2:e6da0655:::smithi17557870-27:head [rollback e] snapc 11=[e] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:40:29.943 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:07.867642+0000", 2024-09-11T03:40:29.943 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 81.973110618999996, 2024-09-11T03:40:29.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024234339999999999, 2024-09-11T03:40:29.943 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.943 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.943 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:29.944 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:40:29.944 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:40:29.944 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:29.944 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 632 2024-09-11T03:40:29.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.944 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867642+0000", 2024-09-11T03:40:29.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867642+0000", 2024-09-11T03:40:29.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867643+0000", 2024-09-11T03:40:29.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.28e-06 2024-09-11T03:40:29.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867649+0000", 2024-09-11T03:40:29.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5600000000000001e-06 2024-09-11T03:40:29.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867667+0000", 2024-09-11T03:40:29.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8406999999999998e-05 2024-09-11T03:40:29.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867681+0000", 2024-09-11T03:40:29.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3973e-05 2024-09-11T03:40:29.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.868700+0000", 2024-09-11T03:40:29.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010191779999999999 2024-09-11T03:40:29.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.868760+0000", 2024-09-11T03:40:29.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9339999999999998e-05 2024-09-11T03:40:29.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:40:29.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.868900+0000", 2024-09-11T03:40:29.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014041999999999999 2024-09-11T03:40:29.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869845+0000", 2024-09-11T03:40:29.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00094485599999999999 2024-09-11T03:40:29.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:40:29.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869977+0000", 2024-09-11T03:40:29.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013179500000000001 2024-09-11T03:40:29.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.870003+0000", 2024-09-11T03:40:29.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6064e-05 2024-09-11T03:40:29.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:29.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.870040+0000", 2024-09-11T03:40:29.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7076999999999998e-05 2024-09-11T03:40:29.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.870066+0000", 2024-09-11T03:40:29.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5483999999999999e-05 2024-09-11T03:40:29.949 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.949 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.949 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.949 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:633 2.7 2:e6da0655:::smithi17557870-27:head [append 0~0] snapc 11=[e] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:40:29.949 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:07.867728+0000", 2024-09-11T03:40:29.950 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 81.973024710000004, 2024-09-11T03:40:29.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024618119999999999, 2024-09-11T03:40:29.950 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.950 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.950 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:29.950 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:40:29.950 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:40:29.950 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:29.950 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 633 2024-09-11T03:40:29.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.950 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867728+0000", 2024-09-11T03:40:29.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867728+0000", 2024-09-11T03:40:29.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867729+0000", 2024-09-11T03:40:29.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3069999999999999e-06 2024-09-11T03:40:29.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867777+0000", 2024-09-11T03:40:29.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.757e-05 2024-09-11T03:40:29.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867809+0000", 2024-09-11T03:40:29.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2209000000000003e-05 2024-09-11T03:40:29.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867819+0000", 2024-09-11T03:40:29.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7860000000000008e-06 2024-09-11T03:40:29.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869311+0000", 2024-09-11T03:40:29.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001491722 2024-09-11T03:40:29.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869406+0000", 2024-09-11T03:40:29.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5612999999999994e-05 2024-09-11T03:40:29.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:40:29.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869547+0000", 2024-09-11T03:40:29.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000141175 2024-09-11T03:40:29.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:40:29.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.870077+0000", 2024-09-11T03:40:29.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052995199999999998 2024-09-11T03:40:29.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.870108+0000", 2024-09-11T03:40:29.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0657000000000001e-05 2024-09-11T03:40:29.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:29.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.870135+0000", 2024-09-11T03:40:29.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7506999999999999e-05 2024-09-11T03:40:29.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:40:29.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.870164+0000", 2024-09-11T03:40:29.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8504e-05 2024-09-11T03:40:29.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.870190+0000", 2024-09-11T03:40:29.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5809999999999999e-05 2024-09-11T03:40:29.955 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.955 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.956 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.956 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.11.83:3 2.7 2:e77ce595:::smithi17557870-31:head [assert-version v23,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:40:29.956 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:08.111371+0000", 2024-09-11T03:40:29.956 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 81.729381678999999, 2024-09-11T03:40:29.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023068860000000002, 2024-09-11T03:40:29.956 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:40:29.956 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:40:29.956 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:40:29.956 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:40:29.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:40:29.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.111371+0000", 2024-09-11T03:40:29.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:40:29.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.111371+0000", 2024-09-11T03:40:29.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:40:29.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:40:29.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.111373+0000", 2024-09-11T03:40:29.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.716e-06 2024-09-11T03:40:29.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:40:29.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.111378+0000", 2024-09-11T03:40:29.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2240000000000003e-06 2024-09-11T03:40:29.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:40:29.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.111388+0000", 2024-09-11T03:40:29.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9129999999999998e-06 2024-09-11T03:40:29.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:40:29.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.111394+0000", 2024-09-11T03:40:29.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1659999999999999e-06 2024-09-11T03:40:29.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:40:29.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.111455+0000", 2024-09-11T03:40:29.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0525000000000001e-05 2024-09-11T03:40:29.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:40:29.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.111522+0000", 2024-09-11T03:40:29.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7010999999999999e-05 2024-09-11T03:40:29.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:40:29.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:40:29.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:40:29.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.113678+0000", 2024-09-11T03:40:29.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002156331 2024-09-11T03:40:29.960 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.960 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.960 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.960 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:40:29.960 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:40:29.960 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:40:29.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:29.962 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:29.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:29.963 INFO:tasks.rados.rados.2.smithi175.stdout:595: done (2 left) 2024-09-11T03:40:29.964 INFO:tasks.rados.rados.2.smithi175.stdout:596: write (excl) oid 17 current snap is 58 2024-09-11T03:40:29.964 INFO:tasks.rados.rados.2.smithi175.stdout:596: seq_num 242 ranges {467294=617201,1578407=488887} 2024-09-11T03:40:29.964 INFO:tasks.rados.rados.2.smithi175.stdout:596: writing smithi17557870-17 from 467294 to 1084495 tid 1 2024-09-11T03:40:29.964 INFO:tasks.rados.rados.2.smithi175.stdout:596: writing smithi17557870-17 from 1578407 to 2067294 tid 2 2024-09-11T03:40:29.964 INFO:tasks.rados.rados.2.smithi175.stdout:597: copy_from oid 33 from oid 48 current snap is 58 2024-09-11T03:40:29.964 INFO:tasks.rados.rados.2.smithi175.stdout:592: finishing write tid 2 to smithi17557870-46 2024-09-11T03:40:29.964 INFO:tasks.rados.rados.2.smithi175.stdout:592: finishing write tid 3 to smithi17557870-46 2024-09-11T03:40:29.964 INFO:tasks.rados.rados.2.smithi175.stdout:592: finishing write tid 4 to smithi17557870-46 2024-09-11T03:40:29.964 INFO:tasks.rados.rados.2.smithi175.stdout:592: oid 46 updating version 0 to 49 2024-09-11T03:40:29.964 INFO:tasks.rados.rados.2.smithi175.stdout:592: oid 46 version 49 is already newer than 48 2024-09-11T03:40:29.964 INFO:tasks.rados.rados.2.smithi175.stdout:592: oid 46 version 49 is already newer than 47 2024-09-11T03:40:29.964 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 49 (ObjNum 239 snap 58 seq_num 239) dirty exists 2024-09-11T03:40:29.964 INFO:tasks.rados.rados.2.smithi175.stdout:592: left oid 46 (ObjNum 239 snap 58 seq_num 239) 2024-09-11T03:40:29.965 INFO:tasks.rados.rados.2.smithi175.stdout:592: done (3 left) 2024-09-11T03:40:29.965 INFO:tasks.rados.rados.2.smithi175.stdout:598: rollback oid 38 current snap is 58 2024-09-11T03:40:29.965 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 38 to 53 2024-09-11T03:40:29.965 INFO:tasks.rados.rados.2.smithi175.stdout:594: finishing write tid 1 to smithi17557870-27 2024-09-11T03:40:29.965 INFO:tasks.rados.rados.2.smithi175.stdout:594: finishing write tid 2 to smithi17557870-27 2024-09-11T03:40:29.965 INFO:tasks.rados.rados.2.smithi175.stdout:594: finishing write tid 3 to smithi17557870-27 2024-09-11T03:40:29.965 INFO:tasks.rados.rados.2.smithi175.stdout:594: finishing write tid 4 to smithi17557870-27 2024-09-11T03:40:29.965 INFO:tasks.rados.rados.2.smithi175.stdout:594: oid 27 updating version 0 to 162 2024-09-11T03:40:29.965 INFO:tasks.rados.rados.2.smithi175.stdout:594: oid 27 version 162 is already newer than 161 2024-09-11T03:40:29.965 INFO:tasks.rados.rados.2.smithi175.stdout:594: oid 27 version 162 is already newer than 160 2024-09-11T03:40:29.965 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 162 (ObjNum 241 snap 58 seq_num 241) dirty exists 2024-09-11T03:40:29.965 INFO:tasks.rados.rados.2.smithi175.stdout:594: left oid 27 (ObjNum 241 snap 58 seq_num 241) 2024-09-11T03:40:29.965 INFO:tasks.rados.rados.2.smithi175.stdout:596: finishing write tid 1 to smithi17557870-17 2024-09-11T03:40:29.965 INFO:tasks.rados.rados.2.smithi175.stdout:594: done (3 left) 2024-09-11T03:40:29.966 INFO:tasks.rados.rados.2.smithi175.stdout:599: rollback oid 50 current snap is 58 2024-09-11T03:40:29.966 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 50 to 56 2024-09-11T03:40:29.966 INFO:tasks.rados.rados.2.smithi175.stdout:600: snap_create 2024-09-11T03:40:29.966 INFO:tasks.rados.rados.2.smithi175.stdout:598: finishing rollback tid 0 to smithi17557870-38 2024-09-11T03:40:29.966 INFO:tasks.rados.rados.2.smithi175.stdout:599: finishing rollback tid 1 to smithi17557870-50 2024-09-11T03:40:29.966 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 0 (ObjNum 81 snap 0 seq_num 1745067440) dirty dne 2024-09-11T03:40:29.966 INFO:tasks.rados.rados.2.smithi175.stdout:598: finishing rollback tid 1 to smithi17557870-38 2024-09-11T03:40:29.966 INFO:tasks.rados.rados.2.smithi175.stdout:598: finishing rollback tid 2 to smithi17557870-38 2024-09-11T03:40:29.966 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 211 (ObjNum 101 snap 16 seq_num 101) dirty exists 2024-09-11T03:40:29.966 INFO:tasks.rados.rados.2.smithi175.stdout:596: finishing write tid 2 to smithi17557870-17 2024-09-11T03:40:29.966 INFO:tasks.rados.rados.2.smithi175.stdout:596: finishing write tid 3 to smithi17557870-17 2024-09-11T03:40:29.966 INFO:tasks.rados.rados.2.smithi175.stdout:596: finishing write tid 4 to smithi17557870-17 2024-09-11T03:40:29.966 INFO:tasks.rados.rados.2.smithi175.stdout:596: oid 17 updating version 0 to 173 2024-09-11T03:40:29.966 INFO:tasks.rados.rados.2.smithi175.stdout:596: oid 17 updating version 173 to 174 2024-09-11T03:40:29.967 INFO:tasks.rados.rados.2.smithi175.stdout:596: oid 17 updating version 174 to 175 2024-09-11T03:40:29.967 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 175 (ObjNum 242 snap 58 seq_num 242) dirty exists 2024-09-11T03:40:29.967 INFO:tasks.rados.rados.2.smithi175.stdout:596: left oid 17 (ObjNum 242 snap 58 seq_num 242) 2024-09-11T03:40:29.967 INFO:tasks.rados.rados.2.smithi175.stdout:597: finishing copy_from racing read to smithi17557870-33 2024-09-11T03:40:29.967 INFO:tasks.rados.rados.2.smithi175.stdout:597: finishing copy_from to smithi17557870-33 2024-09-11T03:40:29.967 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 82 (ObjNum 171 snap 39 seq_num 171) dirty exists 2024-09-11T03:40:30.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:30.087 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:30.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:30.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:30.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:29 smithi084 ceph-mon[27254]: 1.e scrub starts 2024-09-11T03:40:30.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:29 smithi084 ceph-mon[27254]: 1.10 scrub starts 2024-09-11T03:40:30.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:29 smithi084 ceph-mon[27254]: 1.b scrub starts 2024-09-11T03:40:30.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:29 smithi084 ceph-mon[27254]: 1.13 scrub starts 2024-09-11T03:40:30.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:29 smithi084 ceph-mon[27254]: 1.14 scrub starts 2024-09-11T03:40:30.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:29 smithi084 ceph-mon[27254]: 1.11 scrub starts 2024-09-11T03:40:30.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:29 smithi084 ceph-mon[27254]: pgmap v472: 37 pgs: 1 peering, 2 active+clean+scrubbing, 5 unknown, 29 active+clean; 142 MiB data, 575 MiB used, 893 GiB / 894 GiB avail; 4.4 MiB/s rd, 2.8 MiB/s wr, 8 op/s 2024-09-11T03:40:30.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:29 smithi084 ceph-mon[27254]: 1.2 scrub starts 2024-09-11T03:40:30.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:29 smithi084 ceph-mon[27254]: osdmap e212: 12 total, 12 up, 10 in 2024-09-11T03:40:30.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:29 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2537902605' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:40:30.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:29 smithi121 ceph-mon[30729]: 1.e scrub starts 2024-09-11T03:40:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:29 smithi121 ceph-mon[30729]: 1.10 scrub starts 2024-09-11T03:40:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:29 smithi121 ceph-mon[30729]: 1.b scrub starts 2024-09-11T03:40:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:29 smithi121 ceph-mon[30729]: 1.13 scrub starts 2024-09-11T03:40:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:29 smithi121 ceph-mon[30729]: 1.14 scrub starts 2024-09-11T03:40:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:29 smithi121 ceph-mon[30729]: 1.11 scrub starts 2024-09-11T03:40:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:29 smithi121 ceph-mon[30729]: pgmap v472: 37 pgs: 1 peering, 2 active+clean+scrubbing, 5 unknown, 29 active+clean; 142 MiB data, 575 MiB used, 893 GiB / 894 GiB avail; 4.4 MiB/s rd, 2.8 MiB/s wr, 8 op/s 2024-09-11T03:40:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:29 smithi121 ceph-mon[30729]: 1.2 scrub starts 2024-09-11T03:40:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:29 smithi121 ceph-mon[30729]: osdmap e212: 12 total, 12 up, 10 in 2024-09-11T03:40:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:29 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2537902605' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:40:30.178 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:40:30.189 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:30.215 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:30.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:30.216 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:29 smithi175 ceph-mon[30894]: 1.e scrub starts 2024-09-11T03:40:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:29 smithi175 ceph-mon[30894]: 1.10 scrub starts 2024-09-11T03:40:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:29 smithi175 ceph-mon[30894]: 1.b scrub starts 2024-09-11T03:40:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:29 smithi175 ceph-mon[30894]: 1.13 scrub starts 2024-09-11T03:40:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:29 smithi175 ceph-mon[30894]: 1.14 scrub starts 2024-09-11T03:40:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:29 smithi175 ceph-mon[30894]: 1.11 scrub starts 2024-09-11T03:40:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:29 smithi175 ceph-mon[30894]: pgmap v472: 37 pgs: 1 peering, 2 active+clean+scrubbing, 5 unknown, 29 active+clean; 142 MiB data, 575 MiB used, 893 GiB / 894 GiB avail; 4.4 MiB/s rd, 2.8 MiB/s wr, 8 op/s 2024-09-11T03:40:30.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:29 smithi175 ceph-mon[30894]: 1.2 scrub starts 2024-09-11T03:40:30.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:29 smithi175 ceph-mon[30894]: osdmap e212: 12 total, 12 up, 10 in 2024-09-11T03:40:30.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:29 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2537902605' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:40:30.277 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [6, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:40:30.277 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:40:30.278 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-11T03:40:30.278 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:40:30.317 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:30.343 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:30.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:30.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:30.445 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:30.466 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:30.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:30.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:30.569 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:30.570 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:30.571 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:30.592 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:30.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:30.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:30.694 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:30.715 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:30.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:30.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:30.817 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:30.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:30.842 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:30.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:30.860 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T03:40:30.879 INFO:tasks.rados.rados.2.smithi175.stdout:596: done (4 left) 2024-09-11T03:40:30.879 INFO:tasks.rados.rados.2.smithi175.stdout:597: done (3 left) 2024-09-11T03:40:30.879 INFO:tasks.rados.rados.2.smithi175.stdout:598: done (2 left) 2024-09-11T03:40:30.879 INFO:tasks.rados.rados.2.smithi175.stdout:599: done (1 left) 2024-09-11T03:40:30.879 INFO:tasks.rados.rados.2.smithi175.stdout:600: done (0 left) 2024-09-11T03:40:30.879 INFO:tasks.rados.rados.2.smithi175.stdout:601: read oid 10 snap 53 2024-09-11T03:40:30.879 INFO:tasks.rados.rados.2.smithi175.stdout:601: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-11T03:40:30.879 INFO:tasks.rados.rados.2.smithi175.stdout:602: copy_from oid 44 from oid 48 current snap is 59 2024-09-11T03:40:30.879 INFO:tasks.rados.rados.2.smithi175.stdout:603: rollback oid 38 current snap is 59 2024-09-11T03:40:30.880 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 38 to 53 2024-09-11T03:40:30.880 INFO:tasks.rados.rados.2.smithi175.stdout:604: snap_create 2024-09-11T03:40:30.882 INFO:tasks.rados.rados.2.smithi175.stdout:603: finishing rollback tid 0 to smithi17557870-38 2024-09-11T03:40:30.883 INFO:tasks.rados.rados.2.smithi175.stdout:603: finishing rollback tid 1 to smithi17557870-38 2024-09-11T03:40:30.883 INFO:tasks.rados.rados.2.smithi175.stdout:603: finishing rollback tid 2 to smithi17557870-38 2024-09-11T03:40:30.883 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 214 (ObjNum 101 snap 16 seq_num 101) dirty exists 2024-09-11T03:40:30.887 INFO:tasks.rados.rados.2.smithi175.stdout:601: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-11T03:40:30.943 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:30.952 INFO:tasks.rados.rados.2.smithi175.stdout:602: finishing copy_from racing read to smithi17557870-44 2024-09-11T03:40:30.952 INFO:tasks.rados.rados.2.smithi175.stdout:602: finishing copy_from to smithi17557870-44 2024-09-11T03:40:30.952 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 82 (ObjNum 171 snap 39 seq_num 171) dirty exists 2024-09-11T03:40:30.964 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:30.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:30.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:30.994 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:30 smithi084 ceph-mon[27254]: 1.f scrub starts 2024-09-11T03:40:30.994 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:30 smithi084 ceph-mon[27254]: 1.d scrub starts 2024-09-11T03:40:30.994 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:30 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:30.994 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:30 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2537902605' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:40:30.994 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:30 smithi084 ceph-mon[27254]: osdmap e213: 12 total, 12 up, 10 in 2024-09-11T03:40:31.065 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:31.089 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:31.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:31.090 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:31.094 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:40:31.094 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:40:31.102 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:40:31.108 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:40:31.109 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:40:31.125 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:40:31.125 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:40:31.140 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:40:31.140 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:40:31.157 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:40:31.157 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:40:31.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:30 smithi121 ceph-mon[30729]: 1.f scrub starts 2024-09-11T03:40:31.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:30 smithi121 ceph-mon[30729]: 1.d scrub starts 2024-09-11T03:40:31.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:30 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:31.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:30 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2537902605' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:40:31.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:30 smithi121 ceph-mon[30729]: osdmap e213: 12 total, 12 up, 10 in 2024-09-11T03:40:31.173 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:40:31.173 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:40:31.190 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:40:31.190 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:40:31.190 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:31.206 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:40:31.206 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:40:31.214 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:31.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:31.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:31.223 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:40:31.223 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:40:31.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:30 smithi175 ceph-mon[30894]: 1.f scrub starts 2024-09-11T03:40:31.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:30 smithi175 ceph-mon[30894]: 1.d scrub starts 2024-09-11T03:40:31.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:30 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:31.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:30 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2537902605' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:40:31.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:30 smithi175 ceph-mon[30894]: osdmap e213: 12 total, 12 up, 10 in 2024-09-11T03:40:31.241 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:40:31.242 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:40:31.258 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:40:31.259 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:40:31.275 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:40:31.276 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:40:31.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:31.338 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:31.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:31.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:31.440 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:31.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:31.466 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:31.466 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:31.559 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:31.560 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:31.567 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:31.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:31.592 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:31.593 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:31.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:31.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:31.716 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:31.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:31.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:31.839 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:31.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:31.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:31.843 INFO:tasks.rados.rados.2.smithi175.stdout:601: done (3 left) 2024-09-11T03:40:31.843 INFO:tasks.rados.rados.2.smithi175.stdout:602: done (2 left) 2024-09-11T03:40:31.843 INFO:tasks.rados.rados.2.smithi175.stdout:603: done (1 left) 2024-09-11T03:40:31.843 INFO:tasks.rados.rados.2.smithi175.stdout:604: done (0 left) 2024-09-11T03:40:31.843 INFO:tasks.rados.rados.2.smithi175.stdout:605: delete oid 32 current snap is 60 2024-09-11T03:40:31.845 INFO:tasks.rados.rados.2.smithi175.stdout:605: done (0 left) 2024-09-11T03:40:31.845 INFO:tasks.rados.rados.2.smithi175.stdout:606: delete oid 40 current snap is 60 2024-09-11T03:40:31.848 INFO:tasks.rados.rados.2.smithi175.stdout:606: done (0 left) 2024-09-11T03:40:31.848 INFO:tasks.rados.rados.2.smithi175.stdout:607: write oid 49 current snap is 60 2024-09-11T03:40:31.848 INFO:tasks.rados.rados.2.smithi175.stdout:607: seq_num 245 ranges {517715=559301,1870206=247509} 2024-09-11T03:40:31.854 INFO:tasks.rados.rados.2.smithi175.stdout:607: writing smithi17557870-49 from 517715 to 1077016 tid 1 2024-09-11T03:40:31.860 INFO:tasks.rados.rados.2.smithi175.stdout:607: writing smithi17557870-49 from 1870206 to 2117715 tid 2 2024-09-11T03:40:31.860 INFO:tasks.rados.rados.2.smithi175.stdout:608: rollback oid 29 current snap is 60 2024-09-11T03:40:31.861 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 29 to 59 2024-09-11T03:40:31.861 INFO:tasks.rados.rados.2.smithi175.stdout:607: finishing write tid 1 to smithi17557870-49 2024-09-11T03:40:31.861 INFO:tasks.rados.rados.2.smithi175.stdout:609: write (excl) oid 48 current snap is 60 2024-09-11T03:40:31.861 INFO:tasks.rados.rados.2.smithi175.stdout:609: seq_num 246 ranges {534522=634550,1812335=322187} 2024-09-11T03:40:31.867 INFO:tasks.rados.rados.2.smithi175.stdout:609: writing smithi17557870-48 from 534522 to 1169072 tid 1 2024-09-11T03:40:31.872 INFO:tasks.rados.rados.2.smithi175.stdout:609: writing smithi17557870-48 from 1812335 to 2134522 tid 2 2024-09-11T03:40:31.872 INFO:tasks.rados.rados.2.smithi175.stdout:610: snap_create 2024-09-11T03:40:31.873 INFO:tasks.rados.rados.2.smithi175.stdout:607: finishing write tid 2 to smithi17557870-49 2024-09-11T03:40:31.873 INFO:tasks.rados.rados.2.smithi175.stdout:607: finishing write tid 3 to smithi17557870-49 2024-09-11T03:40:31.873 INFO:tasks.rados.rados.2.smithi175.stdout:607: finishing write tid 4 to smithi17557870-49 2024-09-11T03:40:31.873 INFO:tasks.rados.rados.2.smithi175.stdout:607: oid 49 updating version 0 to 155 2024-09-11T03:40:31.873 INFO:tasks.rados.rados.2.smithi175.stdout:607: oid 49 version 155 is already newer than 154 2024-09-11T03:40:31.873 INFO:tasks.rados.rados.2.smithi175.stdout:607: oid 49 updating version 155 to 156 2024-09-11T03:40:31.873 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 156 (ObjNum 245 snap 60 seq_num 245) dirty exists 2024-09-11T03:40:31.873 INFO:tasks.rados.rados.2.smithi175.stdout:607: left oid 49 (ObjNum 245 snap 60 seq_num 245) 2024-09-11T03:40:31.873 INFO:tasks.rados.rados.2.smithi175.stdout:608: finishing rollback tid 0 to smithi17557870-29 2024-09-11T03:40:31.873 INFO:tasks.rados.rados.2.smithi175.stdout:608: finishing rollback tid 1 to smithi17557870-29 2024-09-11T03:40:31.873 INFO:tasks.rados.rados.2.smithi175.stdout:608: finishing rollback tid 2 to smithi17557870-29 2024-09-11T03:40:31.873 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 217 (ObjNum 132 snap 22 seq_num 132) dirty exists 2024-09-11T03:40:31.873 INFO:tasks.rados.rados.2.smithi175.stdout:609: finishing write tid 1 to smithi17557870-48 2024-09-11T03:40:31.875 INFO:tasks.rados.rados.2.smithi175.stdout:609: finishing write tid 2 to smithi17557870-48 2024-09-11T03:40:31.875 INFO:tasks.rados.rados.2.smithi175.stdout:609: finishing write tid 3 to smithi17557870-48 2024-09-11T03:40:31.875 INFO:tasks.rados.rados.2.smithi175.stdout:609: finishing write tid 4 to smithi17557870-48 2024-09-11T03:40:31.875 INFO:tasks.rados.rados.2.smithi175.stdout:609: oid 48 updating version 0 to 114 2024-09-11T03:40:31.875 INFO:tasks.rados.rados.2.smithi175.stdout:609: oid 48 updating version 114 to 116 2024-09-11T03:40:31.875 INFO:tasks.rados.rados.2.smithi175.stdout:609: oid 48 version 116 is already newer than 115 2024-09-11T03:40:31.875 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 116 (ObjNum 246 snap 60 seq_num 246) dirty exists 2024-09-11T03:40:31.875 INFO:tasks.rados.rados.2.smithi175.stdout:609: left oid 48 (ObjNum 246 snap 60 seq_num 246) 2024-09-11T03:40:31.941 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:31.962 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:31.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:31.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:32.065 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:32.091 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:32.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:32.092 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:32.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:31 smithi084 ceph-mon[27254]: 1.12 scrub starts 2024-09-11T03:40:32.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:31 smithi084 ceph-mon[27254]: pgmap v475: 37 pgs: 1 peering, 2 active+clean+scrubbing, 3 unknown, 31 active+clean; 142 MiB data, 589 MiB used, 893 GiB / 894 GiB avail; 1.2 MiB/s wr, 1 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-11T03:40:32.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:31 smithi084 ceph-mon[27254]: osdmap e214: 12 total, 12 up, 10 in 2024-09-11T03:40:32.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:31 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1903135245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:32.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:31 smithi084 ceph-mon[27254]: osdmap e215: 12 total, 12 up, 10 in 2024-09-11T03:40:32.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:31 smithi121 ceph-mon[30729]: 1.12 scrub starts 2024-09-11T03:40:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:31 smithi121 ceph-mon[30729]: pgmap v475: 37 pgs: 1 peering, 2 active+clean+scrubbing, 3 unknown, 31 active+clean; 142 MiB data, 589 MiB used, 893 GiB / 894 GiB avail; 1.2 MiB/s wr, 1 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-11T03:40:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:31 smithi121 ceph-mon[30729]: osdmap e214: 12 total, 12 up, 10 in 2024-09-11T03:40:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:31 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1903135245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:31 smithi121 ceph-mon[30729]: osdmap e215: 12 total, 12 up, 10 in 2024-09-11T03:40:32.192 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:32.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:32.214 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:32.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:31 smithi175 ceph-mon[30894]: 1.12 scrub starts 2024-09-11T03:40:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:31 smithi175 ceph-mon[30894]: pgmap v475: 37 pgs: 1 peering, 2 active+clean+scrubbing, 3 unknown, 31 active+clean; 142 MiB data, 589 MiB used, 893 GiB / 894 GiB avail; 1.2 MiB/s wr, 1 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-11T03:40:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:31 smithi175 ceph-mon[30894]: osdmap e214: 12 total, 12 up, 10 in 2024-09-11T03:40:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:31 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1903135245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:31 smithi175 ceph-mon[30894]: osdmap e215: 12 total, 12 up, 10 in 2024-09-11T03:40:32.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:32.337 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:32.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:32.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:32.367 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:40:32.367 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:40:32.439 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:32.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:32.465 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:32.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:32.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T03:40:32.521 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:40:32.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:32.587 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:32.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:32.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:32.689 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:32.711 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:32.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:32.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:32.744 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:40:32.813 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:32.839 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:32.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:32.841 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:32.841 INFO:tasks.rados.rados.2.smithi175.stdout:607: done (3 left) 2024-09-11T03:40:32.841 INFO:tasks.rados.rados.2.smithi175.stdout:608: done (2 left) 2024-09-11T03:40:32.842 INFO:tasks.rados.rados.2.smithi175.stdout:609: done (1 left) 2024-09-11T03:40:32.842 INFO:tasks.rados.rados.2.smithi175.stdout:610: done (0 left) 2024-09-11T03:40:32.842 INFO:tasks.rados.rados.2.smithi175.stdout:611: snap_create 2024-09-11T03:40:32.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:32.964 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:32.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:32.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:33.066 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:33.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:33.090 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:33.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:33.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:32 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:40:33.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:32 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:40:33.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:32 smithi084 ceph-mon[27254]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:40:33.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:32 smithi084 ceph-mon[27254]: osdmap e216: 12 total, 12 up, 10 in 2024-09-11T03:40:33.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:32 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:40:33.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:32 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:40:33.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:32 smithi121 ceph-mon[30729]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:40:33.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:32 smithi121 ceph-mon[30729]: osdmap e216: 12 total, 12 up, 10 in 2024-09-11T03:40:33.191 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:33.199 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:33.199 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:33.210 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-09-11T03:40:31.838289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":148715787,"num_objects":60,"num_object_clones":18,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":4,"num_read":1063,"num_read_kb":779877,"num_write":1006,"num_write_kb":386520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":14873568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3857,"ondisk_log_size":3857,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":599120,"kb_used_data":289884,"kb_used_omap":15,"kb_used_meta":307504,"kb_avail":936852400,"statfs":{"total":959950356480,"available":959336857600,"internally_reserved":0,"allocated":296841216,"data_stored":325462433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":314884533},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4373460,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":16,"num_read_kb":9,"num_write":36,"num_write_kb":15481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":576117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.999535"},"pg_stats":[{"pgid":"1.14","version":"20'92","reported_seq":12,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T03:40:27.848048+0000","last_change":"2024-09-11T03:40:27.847670+0000","last_active":"2024-09-11T03:40:27.848048+0000","last_peered":"2024-09-11T03:40:27.848048+0000","last_clean":"2024-09-11T03:40:27.848048+0000","last_became_active":"2024-09-11T03:40:27.847297+0000","last_became_peered":"2024-09-11T03:40:27.847297+0000","last_unstale":"2024-09-11T03:40:27.848048+0000","last_undegraded":"2024-09-11T03:40:27.848048+0000","last_fullsized":"2024-09-11T03:40:27.848048+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.356985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.356985+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":64113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"20'92","reported_seq":100,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:31.085060+0000","last_change":"2024-09-11T03:40:28.864342+0000","last_active":"2024-09-11T03:40:31.085060+0000","last_peered":"2024-09-11T03:40:31.085060+0000","last_clean":"2024-09-11T03:40:31.085060+0000","last_became_active":"2024-09-11T03:40:27.847557+0000","last_became_peered":"2024-09-11T03:40:27.847557+0000","last_unstale":"2024-09-11T03:40:31.085060+0000","last_undegraded":"2024-09-11T03:40:31.085060+0000","last_fullsized":"2024-09-11T03:40:31.085060+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.262301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:58.296602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"20'92","reported_seq":21,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:30.887494+0000","last_change":"2024-09-11T03:40:30.887449+0000","last_active":"2024-09-11T03:40:30.887494+0000","last_peered":"2024-09-11T03:40:30.887494+0000","last_clean":"2024-09-11T03:40:30.887494+0000","last_became_active":"2024-09-11T03:40:27.845249+0000","last_became_peered":"2024-09-11T03:40:27.845249+0000","last_unstale":"2024-09-11T03:40:30.887494+0000","last_undegraded":"2024-09-11T03:40:30.887494+0000","last_fullsized":"2024-09-11T03:40:30.887494+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:01.496214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:01.496214+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:41:00.143893+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'92","reported_seq":23,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:30.881709+0000","last_change":"2024-09-11T03:40:28.865187+0000","last_active":"2024-09-11T03:40:30.881709+0000","last_peered":"2024-09-11T03:40:30.881709+0000","last_clean":"2024-09-11T03:40:30.881709+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:40:30.881709+0000","last_undegraded":"2024-09-11T03:40:30.881709+0000","last_fullsized":"2024-09-11T03:40:30.881709+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:58.707730+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"20'92","reported_seq":21,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:30.887364+0000","last_change":"2024-09-11T03:40:28.873834+0000","last_active":"2024-09-11T03:40:30.887364+0000","last_peered":"2024-09-11T03:40:30.887364+0000","last_clean":"2024-09-11T03:40:30.887364+0000","last_became_active":"2024-09-11T03:40:27.844057+0000","last_became_peered":"2024-09-11T03:40:27.844057+0000","last_unstale":"2024-09-11T03:40:30.887364+0000","last_undegraded":"2024-09-11T03:40:30.887364+0000","last_fullsized":"2024-09-11T03:40:30.887364+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:39:58.567842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:39:58.567842+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:58.204412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"213'158","reported_seq":747,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:30.877565+0000","last_change":"2024-09-11T03:40:29.873367+0000","last_active":"2024-09-11T03:40:30.877565+0000","last_peered":"2024-09-11T03:40:30.877565+0000","last_clean":"2024-09-11T03:40:30.877565+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:40:30.877565+0000","last_undegraded":"2024-09-11T03:40:30.877565+0000","last_fullsized":"2024-09-11T03:40:30.877565+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'86","last_scrub_stamp":"2024-09-11T03:39:54.238844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:54.238844+0000","objects_scrubbed":5,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:00.434073+0000","scrub_duration":2,"objects_trimmed":2,"snaptrim_duration":0.0066712999999999998,"stat_sum":{"num_bytes":12772467,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":15675,"num_write":71,"num_write_kb":34497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2b","length":"1"},{"start":"35","length":"1"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":91,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:30.887324+0000","last_change":"2024-09-11T03:40:00.207222+0000","last_active":"2024-09-11T03:40:30.887324+0000","last_peered":"2024-09-11T03:40:30.887324+0000","last_clean":"2024-09-11T03:40:30.887324+0000","last_became_active":"2024-09-11T03:39:59.849648+0000","last_became_peered":"2024-09-11T03:39:59.849648+0000","last_unstale":"2024-09-11T03:40:30.887324+0000","last_undegraded":"2024-09-11T03:40:30.887324+0000","last_fullsized":"2024-09-11T03:40:30.887324+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.207167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.207167+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:14.014779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"206'45","reported_seq":616,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T03:40:26.840515+0000","last_change":"2024-09-11T03:40:25.848938+0000","last_active":"2024-09-11T03:40:26.840515+0000","last_peered":"2024-09-11T03:40:26.840515+0000","last_clean":"2024-09-11T03:40:26.840515+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:40:26.840515+0000","last_undegraded":"2024-09-11T03:40:26.840515+0000","last_fullsized":"2024-09-11T03:40:26.840515+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'8","last_scrub_stamp":"2024-09-11T03:39:33.449882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:33.449882+0000","objects_scrubbed":1,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:38.519904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00060955199999999997,"stat_sum":{"num_bytes":2638699,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":34895,"num_write":18,"num_write_kb":10633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2b","length":"1"},{"start":"39","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":414,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:30.894638+0000","last_change":"2024-09-11T03:40:00.516395+0000","last_active":"2024-09-11T03:40:30.894638+0000","last_peered":"2024-09-11T03:40:30.894638+0000","last_clean":"2024-09-11T03:40:30.894638+0000","last_became_active":"2024-09-11T03:39:59.838010+0000","last_became_peered":"2024-09-11T03:39:59.838010+0000","last_unstale":"2024-09-11T03:40:30.894638+0000","last_undegraded":"2024-09-11T03:40:30.894638+0000","last_fullsized":"2024-09-11T03:40:30.894638+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.516339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.516339+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:30.192680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"213'219","reported_seq":842,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:31.085063+0000","last_change":"2024-09-11T03:40:29.872066+0000","last_active":"2024-09-11T03:40:31.085063+0000","last_peered":"2024-09-11T03:40:31.085063+0000","last_clean":"2024-09-11T03:40:31.085063+0000","last_became_active":"2024-09-11T03:38:29.888204+0000","last_became_peered":"2024-09-11T03:38:29.888204+0000","last_unstale":"2024-09-11T03:40:31.085063+0000","last_undegraded":"2024-09-11T03:40:31.085063+0000","last_fullsized":"2024-09-11T03:40:31.085063+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'138","last_scrub_stamp":"2024-09-11T03:39:52.405246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:52.405246+0000","objects_scrubbed":6,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:09.343833+0000","scrub_duration":2,"objects_trimmed":2,"snaptrim_duration":0.0058122199999999999,"stat_sum":{"num_bytes":19698767,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":142,"num_read_kb":100308,"num_write":125,"num_write_kb":46781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b","length":"1"},{"start":"35","length":"1"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":320,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T03:40:27.845688+0000","last_change":"2024-09-11T03:40:00.202716+0000","last_active":"2024-09-11T03:40:27.845688+0000","last_peered":"2024-09-11T03:40:27.845688+0000","last_clean":"2024-09-11T03:40:27.845688+0000","last_became_active":"2024-09-11T03:40:00.099329+0000","last_became_peered":"2024-09-11T03:40:00.099329+0000","last_unstale":"2024-09-11T03:40:27.845688+0000","last_undegraded":"2024-09-11T03:40:27.845688+0000","last_fullsized":"2024-09-11T03:40:27.845688+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.202664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.202664+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:05.078311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"213'140","reported_seq":734,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:30.894581+0000","last_change":"2024-09-11T03:40:29.872856+0000","last_active":"2024-09-11T03:40:30.894581+0000","last_peered":"2024-09-11T03:40:30.894581+0000","last_clean":"2024-09-11T03:40:30.894581+0000","last_became_active":"2024-09-11T03:38:30.162819+0000","last_became_peered":"2024-09-11T03:38:30.162819+0000","last_unstale":"2024-09-11T03:40:30.894581+0000","last_undegraded":"2024-09-11T03:40:30.894581+0000","last_fullsized":"2024-09-11T03:40:30.894581+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'86","last_scrub_stamp":"2024-09-11T03:39:54.511997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:54.511997+0000","objects_scrubbed":4,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:08.702219+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0044774489999999997,"stat_sum":{"num_bytes":9327852,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":78,"num_read_kb":85186,"num_write":66,"num_write_kb":22289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"1"},{"start":"35","length":"1"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":2,"reported_epoch":210,"state":"peering","last_fresh":"2024-09-11T03:40:26.622477+0000","last_change":"2024-09-11T03:40:26.979341+0000","last_active":"2024-09-11T03:40:26.979341+0000","last_peered":"2024-09-11T03:40:26.622477+0000","last_clean":"2024-09-11T03:40:26.622477+0000","last_became_active":"2024-09-11T03:40:25.858015+0000","last_became_peered":"2024-09-11T03:40:25.858015+0000","last_unstale":"2024-09-11T03:40:26.622477+0000","last_undegraded":"2024-09-11T03:40:26.622477+0000","last_fullsized":"2024-09-11T03:40:26.622477+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.262301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"211'157","reported_seq":797,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T03:40:27.871117+0000","last_change":"2024-09-11T03:40:25.848281+0000","last_active":"2024-09-11T03:40:27.871117+0000","last_peered":"2024-09-11T03:40:27.871117+0000","last_clean":"2024-09-11T03:40:27.871117+0000","last_became_active":"2024-09-11T03:40:08.836627+0000","last_became_peered":"2024-09-11T03:40:08.836627+0000","last_unstale":"2024-09-11T03:40:27.871117+0000","last_undegraded":"2024-09-11T03:40:27.871117+0000","last_fullsized":"2024-09-11T03:40:27.871117+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'75","last_scrub_stamp":"2024-09-11T03:39:39.753844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:39.753844+0000","objects_scrubbed":4,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:03.133463+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00052813800000000004,"stat_sum":{"num_bytes":17193749,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":65,"num_read_kb":32524,"num_write":94,"num_write_kb":35680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":6433401,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b","length":"1"},{"start":"39","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":182,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T03:40:26.841710+0000","last_change":"2024-09-11T03:40:25.858792+0000","last_active":"2024-09-11T03:40:26.841710+0000","last_peered":"2024-09-11T03:40:26.841710+0000","last_clean":"2024-09-11T03:40:26.841710+0000","last_became_active":"2024-09-11T03:40:25.858281+0000","last_became_peered":"2024-09-11T03:40:25.858281+0000","last_unstale":"2024-09-11T03:40:26.841710+0000","last_undegraded":"2024-09-11T03:40:26.841710+0000","last_fullsized":"2024-09-11T03:40:26.841710+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.356985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.356985+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"213'125","reported_seq":717,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:31.085247+0000","last_change":"2024-09-11T03:40:29.876805+0000","last_active":"2024-09-11T03:40:31.085247+0000","last_peered":"2024-09-11T03:40:31.085247+0000","last_clean":"2024-09-11T03:40:31.085247+0000","last_became_active":"2024-09-11T03:38:29.887945+0000","last_became_peered":"2024-09-11T03:38:29.887945+0000","last_unstale":"2024-09-11T03:40:31.085247+0000","last_undegraded":"2024-09-11T03:40:31.085247+0000","last_fullsized":"2024-09-11T03:40:31.085247+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'103","last_scrub_stamp":"2024-09-11T03:40:17.171458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:17.171458+0000","objects_scrubbed":4,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:30.154832+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.010348226,"stat_sum":{"num_bytes":6285818,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":58,"num_read_kb":58348,"num_write":51,"num_write_kb":28075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b","length":"1"},{"start":"35","length":"1"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":79,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T03:40:27.840016+0000","last_change":"2024-09-11T03:40:27.840016+0000","last_active":"2024-09-11T03:40:27.840016+0000","last_peered":"2024-09-11T03:40:27.840016+0000","last_clean":"2024-09-11T03:40:27.840016+0000","last_became_active":"2024-09-11T03:40:25.852445+0000","last_became_peered":"2024-09-11T03:40:25.852445+0000","last_unstale":"2024-09-11T03:40:27.840016+0000","last_undegraded":"2024-09-11T03:40:27.840016+0000","last_fullsized":"2024-09-11T03:40:27.840016+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:57.341273+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"214'187","reported_seq":819,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:30.884334+0000","last_change":"2024-09-11T03:40:29.869178+0000","last_active":"2024-09-11T03:40:30.884334+0000","last_peered":"2024-09-11T03:40:30.884334+0000","last_clean":"2024-09-11T03:40:30.884334+0000","last_became_active":"2024-09-11T03:38:30.163080+0000","last_became_peered":"2024-09-11T03:38:30.163080+0000","last_unstale":"2024-09-11T03:40:30.884334+0000","last_undegraded":"2024-09-11T03:40:30.884334+0000","last_fullsized":"2024-09-11T03:40:30.884334+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'97","last_scrub_stamp":"2024-09-11T03:39:33.455974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:33.455974+0000","objects_scrubbed":5,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:42.018166+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0023647220000000001,"stat_sum":{"num_bytes":18840548,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":135,"num_read_kb":122491,"num_write":100,"num_write_kb":32879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2b","length":"1"},{"start":"35","length":"1"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":315,"reported_epoch":211,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T03:40:28.366702+0000","last_change":"2024-09-11T03:40:28.366569+0000","last_active":"2024-09-11T03:40:28.366702+0000","last_peered":"2024-09-11T03:40:28.366702+0000","last_clean":"2024-09-11T03:40:28.366702+0000","last_became_active":"2024-09-11T03:40:25.854852+0000","last_became_peered":"2024-09-11T03:40:25.854852+0000","last_unstale":"2024-09-11T03:40:28.366702+0000","last_undegraded":"2024-09-11T03:40:28.366702+0000","last_fullsized":"2024-09-11T03:40:28.366702+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.855022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.855022+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":72126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"213'103","reported_seq":723,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:31.085196+0000","last_change":"2024-09-11T03:40:29.872698+0000","last_active":"2024-09-11T03:40:31.085196+0000","last_peered":"2024-09-11T03:40:31.085196+0000","last_clean":"2024-09-11T03:40:31.085196+0000","last_became_active":"2024-09-11T03:40:05.966715+0000","last_became_peered":"2024-09-11T03:40:05.966715+0000","last_unstale":"2024-09-11T03:40:31.085196+0000","last_undegraded":"2024-09-11T03:40:31.085196+0000","last_fullsized":"2024-09-11T03:40:31.085196+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'55","last_scrub_stamp":"2024-09-11T03:39:40.378705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:40.378705+0000","objects_scrubbed":3,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:49.325179+0000","scrub_duration":1,"objects_trimmed":2,"snaptrim_duration":0.0062708520000000004,"stat_sum":{"num_bytes":5142942,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":36,"num_read_kb":18897,"num_write":59,"num_write_kb":18703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":7190868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b","length":"1"},{"start":"35","length":"1"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":79,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T03:40:27.843259+0000","last_change":"2024-09-11T03:40:27.843259+0000","last_active":"2024-09-11T03:40:27.843259+0000","last_peered":"2024-09-11T03:40:27.843259+0000","last_clean":"2024-09-11T03:40:27.843259+0000","last_became_active":"2024-09-11T03:40:25.854310+0000","last_became_peered":"2024-09-11T03:40:25.854310+0000","last_unstale":"2024-09-11T03:40:27.843259+0000","last_undegraded":"2024-09-11T03:40:27.843259+0000","last_fullsized":"2024-09-11T03:40:27.843259+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:57.465729+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":96168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"213'80","reported_seq":697,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:30.887438+0000","last_change":"2024-09-11T03:40:30.107699+0000","last_active":"2024-09-11T03:40:30.887438+0000","last_peered":"2024-09-11T03:40:30.887438+0000","last_clean":"2024-09-11T03:40:30.887438+0000","last_became_active":"2024-09-11T03:39:26.115670+0000","last_became_peered":"2024-09-11T03:39:26.115670+0000","last_unstale":"2024-09-11T03:40:30.887438+0000","last_undegraded":"2024-09-11T03:40:30.887438+0000","last_fullsized":"2024-09-11T03:40:30.887438+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'57","last_scrub_stamp":"2024-09-11T03:39:56.268853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:56.268853+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:03.051682+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0039247800000000001,"stat_sum":{"num_bytes":7294238,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":7,"num_write":41,"num_write_kb":22715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b","length":"1"},{"start":"35","length":"1"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":315,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T03:40:27.845373+0000","last_change":"2024-09-11T03:40:27.845373+0000","last_active":"2024-09-11T03:40:27.845373+0000","last_peered":"2024-09-11T03:40:27.845373+0000","last_clean":"2024-09-11T03:40:27.845373+0000","last_became_active":"2024-09-11T03:40:25.854269+0000","last_became_peered":"2024-09-11T03:40:25.854269+0000","last_unstale":"2024-09-11T03:40:27.845373+0000","last_undegraded":"2024-09-11T03:40:27.845373+0000","last_fullsized":"2024-09-11T03:40:27.845373+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:57.401983+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":72127,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"196'52","reported_seq":629,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T03:40:27.839868+0000","last_change":"2024-09-11T03:40:25.847454+0000","last_active":"2024-09-11T03:40:27.839868+0000","last_peered":"2024-09-11T03:40:27.839868+0000","last_clean":"2024-09-11T03:40:27.839868+0000","last_became_active":"2024-09-11T03:38:30.162952+0000","last_became_peered":"2024-09-11T03:38:30.162952+0000","last_unstale":"2024-09-11T03:40:27.839868+0000","last_undegraded":"2024-09-11T03:40:27.839868+0000","last_fullsized":"2024-09-11T03:40:27.839868+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'49","last_scrub_stamp":"2024-09-11T03:39:51.194245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:51.194245+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:18.268882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055623999999999995,"stat_sum":{"num_bytes":3176523,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":31,"num_read_kb":19745,"num_write":25,"num_write_kb":10878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b","length":"1"},{"start":"39","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":426,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:30.894933+0000","last_change":"2024-09-11T03:40:28.915804+0000","last_active":"2024-09-11T03:40:30.894933+0000","last_peered":"2024-09-11T03:40:30.894933+0000","last_clean":"2024-09-11T03:40:30.894933+0000","last_became_active":"2024-09-11T03:40:25.850944+0000","last_became_peered":"2024-09-11T03:40:25.850944+0000","last_unstale":"2024-09-11T03:40:30.894933+0000","last_undegraded":"2024-09-11T03:40:30.894933+0000","last_fullsized":"2024-09-11T03:40:30.894933+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:01.496214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:01.496214+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:58.802536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"203'29","reported_seq":633,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:30.894748+0000","last_change":"2024-09-11T03:40:29.868925+0000","last_active":"2024-09-11T03:40:30.894748+0000","last_peered":"2024-09-11T03:40:30.894748+0000","last_clean":"2024-09-11T03:40:30.894748+0000","last_became_active":"2024-09-11T03:40:05.966325+0000","last_became_peered":"2024-09-11T03:40:05.966325+0000","last_unstale":"2024-09-11T03:40:30.894748+0000","last_undegraded":"2024-09-11T03:40:30.894748+0000","last_fullsized":"2024-09-11T03:40:30.894748+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'14","last_scrub_stamp":"2024-09-11T03:39:33.465057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:33.465057+0000","objects_scrubbed":1,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:36.248970+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00039101099999999997,"stat_sum":{"num_bytes":3781575,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":5534,"num_write":20,"num_write_kb":5689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"1"},{"start":"35","length":"1"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":97,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:31.085067+0000","last_change":"2024-09-11T03:40:27.842400+0000","last_active":"2024-09-11T03:40:31.085067+0000","last_peered":"2024-09-11T03:40:31.085067+0000","last_clean":"2024-09-11T03:40:31.085067+0000","last_became_active":"2024-09-11T03:40:25.856407+0000","last_became_peered":"2024-09-11T03:40:25.856407+0000","last_unstale":"2024-09-11T03:40:31.085067+0000","last_undegraded":"2024-09-11T03:40:31.085067+0000","last_fullsized":"2024-09-11T03:40:31.085067+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.262301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.262301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:57.253368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"214'74","reported_seq":669,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:30.897429+0000","last_change":"2024-09-11T03:40:30.104108+0000","last_active":"2024-09-11T03:40:30.897429+0000","last_peered":"2024-09-11T03:40:30.897429+0000","last_clean":"2024-09-11T03:40:30.897429+0000","last_became_active":"2024-09-11T03:38:29.889091+0000","last_became_peered":"2024-09-11T03:38:29.889091+0000","last_unstale":"2024-09-11T03:40:30.897429+0000","last_undegraded":"2024-09-11T03:40:30.897429+0000","last_fullsized":"2024-09-11T03:40:30.897429+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'44","last_scrub_stamp":"2024-09-11T03:39:32.965569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:32.965569+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:00.809809+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000472459,"stat_sum":{"num_bytes":5495889,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":15200,"num_write":34,"num_write_kb":15043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b","length":"1"},{"start":"35","length":"1"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":417,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:30.894734+0000","last_change":"2024-09-11T03:40:27.843349+0000","last_active":"2024-09-11T03:40:30.894734+0000","last_peered":"2024-09-11T03:40:30.894734+0000","last_clean":"2024-09-11T03:40:30.894734+0000","last_became_active":"2024-09-11T03:40:25.854624+0000","last_became_peered":"2024-09-11T03:40:25.854624+0000","last_unstale":"2024-09-11T03:40:30.894734+0000","last_undegraded":"2024-09-11T03:40:30.894734+0000","last_fullsized":"2024-09-11T03:40:30.894734+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:39:58.567842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:39:58.567842+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:57.780325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"20'92","reported_seq":23,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:30.887411+0000","last_change":"2024-09-11T03:40:30.103678+0000","last_active":"2024-09-11T03:40:30.887411+0000","last_peered":"2024-09-11T03:40:30.887411+0000","last_clean":"2024-09-11T03:40:30.887411+0000","last_became_active":"2024-09-11T03:40:27.844262+0000","last_became_peered":"2024-09-11T03:40:27.844262+0000","last_unstale":"2024-09-11T03:40:30.887411+0000","last_undegraded":"2024-09-11T03:40:30.887411+0000","last_fullsized":"2024-09-11T03:40:30.887411+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:59.182236+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"209'124","reported_seq":717,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-11T03:40:26.842002+0000","last_change":"2024-09-11T03:40:25.850886+0000","last_active":"2024-09-11T03:40:26.842002+0000","last_peered":"2024-09-11T03:40:26.842002+0000","last_clean":"2024-09-11T03:40:26.842002+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:40:26.842002+0000","last_undegraded":"2024-09-11T03:40:26.842002+0000","last_fullsized":"2024-09-11T03:40:26.842002+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'52","last_scrub_stamp":"2024-09-11T03:39:56.458534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:56.458534+0000","objects_scrubbed":6,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:08.592385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001559085,"stat_sum":{"num_bytes":10702867,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":98,"num_read_kb":79610,"num_write":73,"num_write_kb":30549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2b","length":"1"},{"start":"39","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":262,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-11T03:40:29.866737+0000","last_change":"2024-09-11T03:40:28.885118+0000","last_active":"2024-09-11T03:40:29.866737+0000","last_peered":"2024-09-11T03:40:29.866737+0000","last_clean":"2024-09-11T03:40:29.866737+0000","last_became_active":"2024-09-11T03:40:27.846010+0000","last_became_peered":"2024-09-11T03:40:27.846010+0000","last_unstale":"2024-09-11T03:40:29.866737+0000","last_undegraded":"2024-09-11T03:40:29.866737+0000","last_fullsized":"2024-09-11T03:40:29.866737+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.855022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.855022+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:58.202679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":72127,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"206'168","reported_seq":758,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T03:40:27.845760+0000","last_change":"2024-09-11T03:40:25.850343+0000","last_active":"2024-09-11T03:40:27.845760+0000","last_peered":"2024-09-11T03:40:27.845760+0000","last_clean":"2024-09-11T03:40:27.845760+0000","last_became_active":"2024-09-11T03:38:30.031497+0000","last_became_peered":"2024-09-11T03:38:30.031497+0000","last_unstale":"2024-09-11T03:40:27.845760+0000","last_undegraded":"2024-09-11T03:40:27.845760+0000","last_fullsized":"2024-09-11T03:40:27.845760+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'138","last_scrub_stamp":"2024-09-11T03:39:52.253596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:52.253596+0000","objects_scrubbed":3,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:12.730567+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001263468,"stat_sum":{"num_bytes":7058907,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":110,"num_read_kb":90377,"num_write":69,"num_write_kb":24790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2b","length":"1"},{"start":"39","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":21,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:30.895296+0000","last_change":"2024-09-11T03:40:29.868042+0000","last_active":"2024-09-11T03:40:30.895296+0000","last_peered":"2024-09-11T03:40:30.895296+0000","last_clean":"2024-09-11T03:40:30.895296+0000","last_became_active":"2024-09-11T03:40:27.845535+0000","last_became_peered":"2024-09-11T03:40:27.845535+0000","last_unstale":"2024-09-11T03:40:30.895296+0000","last_undegraded":"2024-09-11T03:40:30.895296+0000","last_fullsized":"2024-09-11T03:40:30.895296+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:37:42.091422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:37:42.091422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:59.774446+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"206'152","reported_seq":737,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T03:40:27.840002+0000","last_change":"2024-09-11T03:40:25.848070+0000","last_active":"2024-09-11T03:40:27.840002+0000","last_peered":"2024-09-11T03:40:27.840002+0000","last_clean":"2024-09-11T03:40:27.840002+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:40:27.840002+0000","last_undegraded":"2024-09-11T03:40:27.840002+0000","last_fullsized":"2024-09-11T03:40:27.840002+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'103","last_scrub_stamp":"2024-09-11T03:39:55.201092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:55.201092+0000","objects_scrubbed":2,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:24.890034+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00054492899999999999,"stat_sum":{"num_bytes":13210269,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":99,"num_read_kb":83452,"num_write":49,"num_write_kb":28168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b","length":"1"},{"start":"39","length":"1"}]},{"pgid":"2.f","version":"206'112","reported_seq":712,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-11T03:40:30.877748+0000","last_change":"2024-09-11T03:40:29.869368+0000","last_active":"2024-09-11T03:40:30.877748+0000","last_peered":"2024-09-11T03:40:30.877748+0000","last_clean":"2024-09-11T03:40:30.877748+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:40:30.877748+0000","last_undegraded":"2024-09-11T03:40:30.877748+0000","last_fullsized":"2024-09-11T03:40:30.877748+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'82","last_scrub_stamp":"2024-09-11T03:39:53.261846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:53.261846+0000","objects_scrubbed":4,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:09.277654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0025971229999999998,"stat_sum":{"num_bytes":5445468,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":17597,"num_write":52,"num_write_kb":18140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2b","length":"1"},{"start":"35","length":"1"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":22,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-11T03:40:28.450776+0000","last_change":"2024-09-11T03:40:28.727627+0000","last_active":"2024-09-11T03:40:28.450776+0000","last_peered":"2024-09-11T03:40:28.450776+0000","last_clean":"2024-09-11T03:40:28.450776+0000","last_became_active":"2024-09-11T03:40:27.848775+0000","last_became_peered":"2024-09-11T03:40:27.848775+0000","last_unstale":"2024-09-11T03:40:28.450776+0000","last_undegraded":"2024-09-11T03:40:28.450776+0000","last_fullsized":"2024-09-11T03:40:28.450776+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.356985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.356985+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:40:58.450842+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":4,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":391793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":148066578,"num_objects":57,"num_object_clones":18,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":4,"num_read":1026,"num_read_kb":779846,"num_write":947,"num_write_kb":385509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":13624269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":280653824,"data_stored":322021071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1925,"ondisk_log_size":1925,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":649209,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":59,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1249299,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2457600,"data_stored":2443392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285607,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49808,"kb_used_data":22780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93695344,"statfs":{"total":95995035648,"available":95944032256,"internally_reserved":0,"allocated":23326720,"data_stored":28616270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":326417514539,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743473,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65600,"kb_used_data":34408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93679552,"statfs":{"total":95995035648,"available":95927861248,"internally_reserved":0,"allocated":35233792,"data_stored":36079149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037817,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57640,"kb_used_data":26444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93687512,"statfs":{"total":95995035648,"available":95936012288,"internally_reserved":0,"allocated":27078656,"data_stored":33444402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299452,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64840,"kb_used_data":33652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93680312,"statfs":{"total":95995035648,"available":95928639488,"internally_reserved":0,"allocated":34459648,"data_stored":38644510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":193273528384,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757318,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40444,"kb_used_data":9252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93704708,"statfs":{"total":95995035648,"available":95953620992,"internally_reserved":0,"allocated":9474048,"data_stored":8426118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986251,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62368,"kb_used_data":29700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93682784,"statfs":{"total":95995035648,"available":95931170816,"internally_reserved":0,"allocated":30412800,"data_stored":32271761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77316,"kb_used_data":46124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93667836,"statfs":{"total":95995035648,"available":95915864064,"internally_reserved":0,"allocated":47230976,"data_stored":54916017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57952,"kb_used_data":26760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93687200,"statfs":{"total":95995035648,"available":95935692800,"internally_reserved":0,"allocated":27402240,"data_stored":28314462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64316,"kb_used_data":33120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93680836,"statfs":{"total":95995035648,"available":95929176064,"internally_reserved":0,"allocated":33914880,"data_stored":35063965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58836,"kb_used_data":27644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93686316,"statfs":{"total":95995035648,"available":95934787584,"internally_reserved":0,"allocated":28307456,"data_stored":29685779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25501696,"data_stored":30164691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":31424512,"data_stored":33597339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":26619904,"data_stored":28885584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":44244992,"data_stored":52944883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":27934720,"data_stored":30813243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6778880,"data_stored":6758244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4079616,"data_stored":4066399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":34070528,"data_stored":39273557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24588288,"data_stored":31977776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":32452608,"data_stored":34309263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2387968,"data_stored":2367002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20570112,"data_stored":26863090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:40:33.212 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-11T03:40:33.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pg_num 31 2024-09-11T03:40:33.213 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:33.218 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:33.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:33.219 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:32 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:40:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:32 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:40:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:32 smithi175 ceph-mon[30894]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:40:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:32 smithi175 ceph-mon[30894]: osdmap e216: 12 total, 12 up, 10 in 2024-09-11T03:40:33.320 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:33.341 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:33.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:33.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:33.443 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:33.469 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:33.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:33.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:33.572 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:33.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:33.594 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:33.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:33.696 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:33.697 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:33.723 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:33.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:33.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:33.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:33.846 INFO:tasks.rados.rados.2.smithi175.stdout:611: done (0 left) 2024-09-11T03:40:33.846 INFO:tasks.rados.rados.2.smithi175.stdout:612: read oid 27 snap -1 2024-09-11T03:40:33.846 INFO:tasks.rados.rados.2.smithi175.stdout:612: expect (ObjNum 241 snap 58 seq_num 241) 2024-09-11T03:40:33.846 INFO:tasks.rados.rados.2.smithi175.stdout:613: read oid 22 snap -1 2024-09-11T03:40:33.846 INFO:tasks.rados.rados.2.smithi175.stdout:613: expect (ObjNum 194 snap 47 seq_num 194) 2024-09-11T03:40:33.847 INFO:tasks.rados.rados.2.smithi175.stdout:614: snap_remove snap 53 2024-09-11T03:40:33.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:33.848 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:33.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:33.915 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T03:40:33.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:33.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:40:33.980 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:33.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:33.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:34.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:33 smithi084 ceph-mon[27254]: pgmap v478: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 145 MiB data, 584 MiB used, 893 GiB / 894 GiB avail; 4.2 KiB/s rd, 3.9 MiB/s wr, 20 op/s; 184 KiB/s, 1 objects/s recovering 2024-09-11T03:40:34.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:33 smithi084 ceph-mon[27254]: osdmap e217: 12 total, 12 up, 10 in 2024-09-11T03:40:34.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:33 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2744001325' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:40:34.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:34.107 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:34.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:34.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:34.162 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:40:34.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:33 smithi121 ceph-mon[30729]: pgmap v478: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 145 MiB data, 584 MiB used, 893 GiB / 894 GiB avail; 4.2 KiB/s rd, 3.9 MiB/s wr, 20 op/s; 184 KiB/s, 1 objects/s recovering 2024-09-11T03:40:34.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:33 smithi121 ceph-mon[30729]: osdmap e217: 12 total, 12 up, 10 in 2024-09-11T03:40:34.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:33 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2744001325' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:40:34.209 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:34.215 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:33 smithi175 ceph-mon[30894]: pgmap v478: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 145 MiB data, 584 MiB used, 893 GiB / 894 GiB avail; 4.2 KiB/s rd, 3.9 MiB/s wr, 20 op/s; 184 KiB/s, 1 objects/s recovering 2024-09-11T03:40:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:33 smithi175 ceph-mon[30894]: osdmap e217: 12 total, 12 up, 10 in 2024-09-11T03:40:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:33 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2744001325' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:40:34.230 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:34.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:34.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:34.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:34.354 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:34.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:34.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:34.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:34.479 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:34.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:34.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:34.581 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:34.602 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:34.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:34.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:34.624 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:34.624 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:34.681 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:40:34.681 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:40:34.696 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:40:34.696 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:40:34.703 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:34.712 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:40:34.712 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:40:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:40:34.728 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:40:34.729 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:34.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:34.730 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:34.744 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:40:34.744 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:40:34.760 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:40:34.761 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:40:34.777 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:40:34.778 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:40:34.793 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:40:34.793 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:40:34.810 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:40:34.811 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:40:34.828 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:40:34.828 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:40:34.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:34.840 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:40:34.841 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 1 pg_num to 31 2024-09-11T03:40:34.844 INFO:tasks.rados.rados.2.smithi175.stdout:614: done (2 left) 2024-09-11T03:40:34.844 INFO:tasks.rados.rados.2.smithi175.stdout:615: copy_from oid 32 from oid 33 current snap is 62 2024-09-11T03:40:34.844 INFO:tasks.rados.rados.2.smithi175.stdout:612: expect (ObjNum 241 snap 58 seq_num 241) 2024-09-11T03:40:34.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:40:34.848 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:40:34.860 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:34.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:34.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:34.873 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:40:34.873 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:40:34.952 INFO:tasks.rados.rados.2.smithi175.stdout:613: expect (ObjNum 194 snap 47 seq_num 194) 2024-09-11T03:40:34.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:34.984 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:34.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:34.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:35.086 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:35.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:34 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/510551576' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-11T03:40:35.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:34 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/150997445' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:35.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:34 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2744001325' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:40:35.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:34 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/510551576' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-11T03:40:35.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:34 smithi084 ceph-mon[27254]: osdmap e218: 12 total, 12 up, 10 in 2024-09-11T03:40:35.091 INFO:tasks.rados.rados.2.smithi175.stdout:615: finishing copy_from racing read to smithi17557870-32 2024-09-11T03:40:35.091 INFO:tasks.rados.rados.2.smithi175.stdout:615: finishing copy_from to smithi17557870-32 2024-09-11T03:40:35.091 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 82 (ObjNum 171 snap 39 seq_num 171) dirty exists 2024-09-11T03:40:35.091 INFO:tasks.rados.rados.2.smithi175.stdout:612: done (2 left) 2024-09-11T03:40:35.091 INFO:tasks.rados.rados.2.smithi175.stdout:613: done (1 left) 2024-09-11T03:40:35.091 INFO:tasks.rados.rados.2.smithi175.stdout:615: done (0 left) 2024-09-11T03:40:35.091 INFO:tasks.rados.rados.2.smithi175.stdout:616: write (excl) oid 32 current snap is 62 2024-09-11T03:40:35.091 INFO:tasks.rados.rados.2.smithi175.stdout:616: seq_num 247 ranges {551329=709799,1870817=280512} 2024-09-11T03:40:35.097 INFO:tasks.rados.rados.2.smithi175.stdout:616: writing smithi17557870-32 from 551329 to 1261128 tid 1 2024-09-11T03:40:35.103 INFO:tasks.rados.rados.2.smithi175.stdout:616: writing smithi17557870-32 from 1870817 to 2151329 tid 2 2024-09-11T03:40:35.103 INFO:tasks.rados.rados.2.smithi175.stdout:617: copy_from oid 31 from oid 39 current snap is 62 2024-09-11T03:40:35.103 INFO:tasks.rados.rados.2.smithi175.stdout:616: finishing write tid 1 to smithi17557870-32 2024-09-11T03:40:35.103 INFO:tasks.rados.rados.2.smithi175.stdout:618: write (excl) oid 2 current snap is 62 2024-09-11T03:40:35.103 INFO:tasks.rados.rados.2.smithi175.stdout:618: seq_num 248 ranges {568136=785048,1929299=238837} 2024-09-11T03:40:35.107 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:35.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:35.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:35.110 INFO:tasks.rados.rados.2.smithi175.stdout:618: writing smithi17557870-2 from 568136 to 1353184 tid 1 2024-09-11T03:40:35.115 INFO:tasks.rados.rados.2.smithi175.stdout:618: writing smithi17557870-2 from 1929299 to 2168136 tid 2 2024-09-11T03:40:35.115 INFO:tasks.rados.rados.2.smithi175.stdout:619: copy_from oid 30 from oid 40 current snap is 62 2024-09-11T03:40:35.115 INFO:tasks.rados.rados.2.smithi175.stdout:616: finishing write tid 2 to smithi17557870-32 2024-09-11T03:40:35.115 INFO:tasks.rados.rados.2.smithi175.stdout:616: finishing write tid 3 to smithi17557870-32 2024-09-11T03:40:35.115 INFO:tasks.rados.rados.2.smithi175.stdout:616: finishing write tid 4 to smithi17557870-32 2024-09-11T03:40:35.116 INFO:tasks.rados.rados.2.smithi175.stdout:616: oid 32 updating version 0 to 168 2024-09-11T03:40:35.116 INFO:tasks.rados.rados.2.smithi175.stdout:616: oid 32 version 168 is already newer than 166 2024-09-11T03:40:35.116 INFO:tasks.rados.rados.2.smithi175.stdout:616: oid 32 version 168 is already newer than 167 2024-09-11T03:40:35.116 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 168 (ObjNum 247 snap 62 seq_num 247) dirty exists 2024-09-11T03:40:35.116 INFO:tasks.rados.rados.2.smithi175.stdout:616: left oid 32 (ObjNum 247 snap 62 seq_num 247) 2024-09-11T03:40:35.116 INFO:tasks.rados.rados.2.smithi175.stdout:616: done (3 left) 2024-09-11T03:40:35.116 INFO:tasks.rados.rados.2.smithi175.stdout:620: read oid 44 snap -1 2024-09-11T03:40:35.116 INFO:tasks.rados.rados.2.smithi175.stdout:620: expect (ObjNum 171 snap 39 seq_num 171) 2024-09-11T03:40:35.116 INFO:tasks.rados.rados.2.smithi175.stdout:621: delete oid 28 current snap is 62 2024-09-11T03:40:35.119 INFO:tasks.rados.rados.2.smithi175.stdout:621: done (4 left) 2024-09-11T03:40:35.119 INFO:tasks.rados.rados.2.smithi175.stdout:622: snap_create 2024-09-11T03:40:35.120 INFO:tasks.rados.rados.2.smithi175.stdout:619: finishing copy_from to smithi17557870-30 2024-09-11T03:40:35.120 INFO:tasks.rados.rados.2.smithi175.stdout:619: got expected ENOENT (src dne) 2024-09-11T03:40:35.120 INFO:tasks.rados.rados.2.smithi175.stdout:619: finishing copy_from racing read to smithi17557870-30 2024-09-11T03:40:35.120 INFO:tasks.rados.rados.2.smithi175.stdout:617: finishing copy_from racing read to smithi17557870-31 2024-09-11T03:40:35.120 INFO:tasks.rados.rados.2.smithi175.stdout:618: finishing write tid 1 to smithi17557870-2 2024-09-11T03:40:35.125 INFO:tasks.rados.rados.2.smithi175.stdout:617: finishing copy_from to smithi17557870-31 2024-09-11T03:40:35.125 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 148 (ObjNum 231 snap 56 seq_num 231) dirty exists 2024-09-11T03:40:35.125 INFO:tasks.rados.rados.2.smithi175.stdout:618: finishing write tid 2 to smithi17557870-2 2024-09-11T03:40:35.125 INFO:tasks.rados.rados.2.smithi175.stdout:618: finishing write tid 3 to smithi17557870-2 2024-09-11T03:40:35.125 INFO:tasks.rados.rados.2.smithi175.stdout:618: finishing write tid 4 to smithi17557870-2 2024-09-11T03:40:35.125 INFO:tasks.rados.rados.2.smithi175.stdout:618: oid 2 updating version 0 to 172 2024-09-11T03:40:35.125 INFO:tasks.rados.rados.2.smithi175.stdout:618: oid 2 updating version 172 to 176 2024-09-11T03:40:35.125 INFO:tasks.rados.rados.2.smithi175.stdout:618: oid 2 version 176 is already newer than 175 2024-09-11T03:40:35.126 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 176 (ObjNum 248 snap 62 seq_num 248) dirty exists 2024-09-11T03:40:35.126 INFO:tasks.rados.rados.2.smithi175.stdout:618: left oid 2 (ObjNum 248 snap 62 seq_num 248) 2024-09-11T03:40:35.126 INFO:tasks.rados.rados.2.smithi175.stdout:620: expect (ObjNum 171 snap 39 seq_num 171) 2024-09-11T03:40:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:34 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/510551576' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-11T03:40:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:34 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/150997445' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:34 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2744001325' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:40:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:34 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/510551576' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-11T03:40:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:34 smithi121 ceph-mon[30729]: osdmap e218: 12 total, 12 up, 10 in 2024-09-11T03:40:35.210 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:34 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/510551576' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-11T03:40:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:34 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/150997445' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:34 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2744001325' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:40:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:34 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/510551576' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-11T03:40:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:34 smithi175 ceph-mon[30894]: osdmap e218: 12 total, 12 up, 10 in 2024-09-11T03:40:35.231 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:35.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:35.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:35.333 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:35.355 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:35.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:35.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:35.415 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-11T03:40:35.457 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:35.480 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:35.480 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:35.581 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:35.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:35.605 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:35.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:35.658 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:40:35.705 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:35.726 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:35.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:35.828 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:35.850 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:35.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:35.851 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:35.895 INFO:tasks.rados.rados.2.smithi175.stdout:617: done (4 left) 2024-09-11T03:40:35.895 INFO:tasks.rados.rados.2.smithi175.stdout:618: done (3 left) 2024-09-11T03:40:35.895 INFO:tasks.rados.rados.2.smithi175.stdout:619: done (2 left) 2024-09-11T03:40:35.895 INFO:tasks.rados.rados.2.smithi175.stdout:620: done (1 left) 2024-09-11T03:40:35.895 INFO:tasks.rados.rados.2.smithi175.stdout:622: done (0 left) 2024-09-11T03:40:35.895 INFO:tasks.rados.rados.2.smithi175.stdout:623: delete oid 15 current snap is 63 2024-09-11T03:40:35.896 INFO:tasks.rados.rados.2.smithi175.stdout:623: done (0 left) 2024-09-11T03:40:35.896 INFO:tasks.rados.rados.2.smithi175.stdout:624: rollback oid 42 current snap is 63 2024-09-11T03:40:35.896 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 42 to 62 2024-09-11T03:40:35.896 INFO:tasks.rados.rados.2.smithi175.stdout:625: read oid 33 snap -1 2024-09-11T03:40:35.896 INFO:tasks.rados.rados.2.smithi175.stdout:625: expect (ObjNum 171 snap 39 seq_num 171) 2024-09-11T03:40:35.896 INFO:tasks.rados.rados.2.smithi175.stdout:626: copy_from oid 4 from oid 44 current snap is 63 2024-09-11T03:40:35.896 INFO:tasks.rados.rados.2.smithi175.stdout:627: write oid 36 current snap is 63 2024-09-11T03:40:35.896 INFO:tasks.rados.rados.2.smithi175.stdout:627: seq_num 251 ranges {618557=727148,2104745=113812} 2024-09-11T03:40:35.903 INFO:tasks.rados.rados.2.smithi175.stdout:627: writing smithi17557870-36 from 618557 to 1345705 tid 1 2024-09-11T03:40:35.909 INFO:tasks.rados.rados.2.smithi175.stdout:627: writing smithi17557870-36 from 2104745 to 2218557 tid 2 2024-09-11T03:40:35.909 INFO:tasks.rados.rados.2.smithi175.stdout:628: write (excl) oid 21 current snap is 63 2024-09-11T03:40:35.909 INFO:tasks.rados.rados.2.smithi175.stdout:624: finishing rollback tid 1 to smithi17557870-42 2024-09-11T03:40:35.909 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 0 (ObjNum 129 snap 0 seq_num 1745149432) dirty dne 2024-09-11T03:40:35.909 INFO:tasks.rados.rados.2.smithi175.stdout:625: expect (ObjNum 171 snap 39 seq_num 171) 2024-09-11T03:40:35.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:35.973 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:35.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:35.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:36.054 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:35 smithi175 ceph-mon[30894]: pgmap v481: 37 pgs: 37 active+clean; 145 MiB data, 591 MiB used, 893 GiB / 894 GiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 28 op/s; 94 KiB/s, 0 objects/s recovering 2024-09-11T03:40:36.054 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:35 smithi175 ceph-mon[30894]: 1.2 scrub starts 2024-09-11T03:40:36.054 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:35 smithi175 ceph-mon[30894]: 1.2 scrub ok 2024-09-11T03:40:36.054 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:35 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:40:36.054 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:35 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:40:36.055 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:35 smithi175 ceph-mon[30894]: 1.14 scrub starts 2024-09-11T03:40:36.055 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:35 smithi175 ceph-mon[30894]: 1.14 scrub ok 2024-09-11T03:40:36.055 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:35 smithi175 ceph-mon[30894]: 1.1 scrub starts 2024-09-11T03:40:36.055 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:35 smithi175 ceph-mon[30894]: 1.1 scrub ok 2024-09-11T03:40:36.055 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:35 smithi175 ceph-mon[30894]: 1.c scrub starts 2024-09-11T03:40:36.055 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:35 smithi175 ceph-mon[30894]: 1.c scrub ok 2024-09-11T03:40:36.065 INFO:tasks.rados.rados.2.smithi175.stdout:627: finishing write tid 1 to smithi17557870-36 2024-09-11T03:40:36.066 INFO:tasks.rados.rados.2.smithi175.stdout:626: finishing copy_from racing read to smithi17557870-4 2024-09-11T03:40:36.066 INFO:tasks.rados.rados.2.smithi175.stdout:627: finishing write tid 2 to smithi17557870-36 2024-09-11T03:40:36.066 INFO:tasks.rados.rados.2.smithi175.stdout:627: finishing write tid 3 to smithi17557870-36 2024-09-11T03:40:36.066 INFO:tasks.rados.rados.2.smithi175.stdout:627: finishing write tid 4 to smithi17557870-36 2024-09-11T03:40:36.066 INFO:tasks.rados.rados.2.smithi175.stdout:627: oid 36 updating version 0 to 144 2024-09-11T03:40:36.066 INFO:tasks.rados.rados.2.smithi175.stdout:627: oid 36 updating version 144 to 146 2024-09-11T03:40:36.066 INFO:tasks.rados.rados.2.smithi175.stdout:627: oid 36 version 146 is already newer than 145 2024-09-11T03:40:36.066 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 146 (ObjNum 251 snap 63 seq_num 251) dirty exists 2024-09-11T03:40:36.066 INFO:tasks.rados.rados.2.smithi175.stdout:627: left oid 36 (ObjNum 251 snap 63 seq_num 251) 2024-09-11T03:40:36.066 INFO:tasks.rados.rados.2.smithi175.stdout:626: finishing copy_from to smithi17557870-4 2024-09-11T03:40:36.066 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 82 (ObjNum 171 snap 39 seq_num 171) dirty exists 2024-09-11T03:40:36.066 INFO:tasks.rados.rados.2.smithi175.stdout:628: seq_num 252 ranges {635364=402397,1763227=472137} 2024-09-11T03:40:36.071 INFO:tasks.rados.rados.2.smithi175.stdout:628: writing smithi17557870-21 from 635364 to 1037761 tid 1 2024-09-11T03:40:36.074 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:36.078 INFO:tasks.rados.rados.2.smithi175.stdout:628: writing smithi17557870-21 from 1763227 to 2235364 tid 2 2024-09-11T03:40:36.078 INFO:tasks.rados.rados.2.smithi175.stdout:624: done (4 left) 2024-09-11T03:40:36.078 INFO:tasks.rados.rados.2.smithi175.stdout:625: done (3 left) 2024-09-11T03:40:36.078 INFO:tasks.rados.rados.2.smithi175.stdout:626: done (2 left) 2024-09-11T03:40:36.078 INFO:tasks.rados.rados.2.smithi175.stdout:627: done (1 left) 2024-09-11T03:40:36.078 INFO:tasks.rados.rados.2.smithi175.stdout:629: read oid 43 snap 56 2024-09-11T03:40:36.078 INFO:tasks.rados.rados.2.smithi175.stdout:629: expect deleted 2024-09-11T03:40:36.078 INFO:tasks.rados.rados.2.smithi175.stdout:630: read oid 3 snap -1 2024-09-11T03:40:36.079 INFO:tasks.rados.rados.2.smithi175.stdout:630: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-11T03:40:36.079 INFO:tasks.rados.rados.2.smithi175.stdout:631: delete oid 50 current snap is 63 2024-09-11T03:40:36.079 INFO:tasks.rados.rados.2.smithi175.stdout:628: finishing write tid 1 to smithi17557870-21 2024-09-11T03:40:36.080 INFO:tasks.rados.rados.2.smithi175.stdout:628: finishing write tid 2 to smithi17557870-21 2024-09-11T03:40:36.080 INFO:tasks.rados.rados.2.smithi175.stdout:628: finishing write tid 3 to smithi17557870-21 2024-09-11T03:40:36.080 INFO:tasks.rados.rados.2.smithi175.stdout:628: finishing write tid 4 to smithi17557870-21 2024-09-11T03:40:36.080 INFO:tasks.rados.rados.2.smithi175.stdout:628: oid 21 updating version 0 to 179 2024-09-11T03:40:36.081 INFO:tasks.rados.rados.2.smithi175.stdout:628: oid 21 version 179 is already newer than 178 2024-09-11T03:40:36.081 INFO:tasks.rados.rados.2.smithi175.stdout:628: oid 21 updating version 179 to 180 2024-09-11T03:40:36.081 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 180 (ObjNum 252 snap 63 seq_num 252) dirty exists 2024-09-11T03:40:36.081 INFO:tasks.rados.rados.2.smithi175.stdout:628: left oid 21 (ObjNum 252 snap 63 seq_num 252) 2024-09-11T03:40:36.087 INFO:tasks.rados.rados.2.smithi175.stdout:630: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-11T03:40:36.099 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:36.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:36.100 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:35 smithi084 ceph-mon[27254]: pgmap v481: 37 pgs: 37 active+clean; 145 MiB data, 591 MiB used, 893 GiB / 894 GiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 28 op/s; 94 KiB/s, 0 objects/s recovering 2024-09-11T03:40:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:35 smithi084 ceph-mon[27254]: 1.2 scrub starts 2024-09-11T03:40:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:35 smithi084 ceph-mon[27254]: 1.2 scrub ok 2024-09-11T03:40:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:35 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:40:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:35 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:40:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:35 smithi084 ceph-mon[27254]: 1.14 scrub starts 2024-09-11T03:40:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:35 smithi084 ceph-mon[27254]: 1.14 scrub ok 2024-09-11T03:40:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:35 smithi084 ceph-mon[27254]: 1.1 scrub starts 2024-09-11T03:40:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:35 smithi084 ceph-mon[27254]: 1.1 scrub ok 2024-09-11T03:40:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:35 smithi084 ceph-mon[27254]: 1.c scrub starts 2024-09-11T03:40:36.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:35 smithi084 ceph-mon[27254]: 1.c scrub ok 2024-09-11T03:40:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:35 smithi121 ceph-mon[30729]: pgmap v481: 37 pgs: 37 active+clean; 145 MiB data, 591 MiB used, 893 GiB / 894 GiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 28 op/s; 94 KiB/s, 0 objects/s recovering 2024-09-11T03:40:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:35 smithi121 ceph-mon[30729]: 1.2 scrub starts 2024-09-11T03:40:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:35 smithi121 ceph-mon[30729]: 1.2 scrub ok 2024-09-11T03:40:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:35 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:40:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:35 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:40:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:35 smithi121 ceph-mon[30729]: 1.14 scrub starts 2024-09-11T03:40:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:35 smithi121 ceph-mon[30729]: 1.14 scrub ok 2024-09-11T03:40:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:35 smithi121 ceph-mon[30729]: 1.1 scrub starts 2024-09-11T03:40:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:35 smithi121 ceph-mon[30729]: 1.1 scrub ok 2024-09-11T03:40:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:35 smithi121 ceph-mon[30729]: 1.c scrub starts 2024-09-11T03:40:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:35 smithi121 ceph-mon[30729]: 1.c scrub ok 2024-09-11T03:40:36.173 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:40:36.173 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:40:36.173 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:40:36.173 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:40:36.192 INFO:tasks.rados.rados.2.smithi175.stdout:628: done (3 left) 2024-09-11T03:40:36.192 INFO:tasks.rados.rados.2.smithi175.stdout:629: done (2 left) 2024-09-11T03:40:36.192 INFO:tasks.rados.rados.2.smithi175.stdout:630: done (1 left) 2024-09-11T03:40:36.193 INFO:tasks.rados.rados.2.smithi175.stdout:631: done (0 left) 2024-09-11T03:40:36.193 INFO:tasks.rados.rados.2.smithi175.stdout:632: rollback oid 10 current snap is 63 2024-09-11T03:40:36.193 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 10 to 56 2024-09-11T03:40:36.193 INFO:tasks.rados.rados.2.smithi175.stdout:633: copy_from oid 26 from oid 18 current snap is 63 2024-09-11T03:40:36.193 INFO:tasks.rados.rados.2.smithi175.stdout:634: write (excl) oid 7 current snap is 63 2024-09-11T03:40:36.193 INFO:tasks.rados.rados.2.smithi175.stdout:634: seq_num 254 ranges {668978=552895,1763838=505140} 2024-09-11T03:40:36.199 INFO:tasks.rados.rados.2.smithi175.stdout:634: writing smithi17557870-7 from 668978 to 1221873 tid 1 2024-09-11T03:40:36.201 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:36.205 INFO:tasks.rados.rados.2.smithi175.stdout:634: writing smithi17557870-7 from 1763838 to 2268978 tid 2 2024-09-11T03:40:36.206 INFO:tasks.rados.rados.2.smithi175.stdout:635: rollback oid 42 current snap is 63 2024-09-11T03:40:36.206 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 42 to 61 2024-09-11T03:40:36.206 INFO:tasks.rados.rados.2.smithi175.stdout:633: finishing copy_from to smithi17557870-26 2024-09-11T03:40:36.206 INFO:tasks.rados.rados.2.smithi175.stdout:633: got expected ENOENT (src dne) 2024-09-11T03:40:36.206 INFO:tasks.rados.rados.2.smithi175.stdout:636: copy_from oid 21 from oid 38 current snap is 63 2024-09-11T03:40:36.206 INFO:tasks.rados.rados.2.smithi175.stdout:633: finishing copy_from racing read to smithi17557870-26 2024-09-11T03:40:36.206 INFO:tasks.rados.rados.2.smithi175.stdout:632: finishing rollback tid 0 to smithi17557870-10 2024-09-11T03:40:36.206 INFO:tasks.rados.rados.2.smithi175.stdout:632: finishing rollback tid 1 to smithi17557870-10 2024-09-11T03:40:36.206 INFO:tasks.rados.rados.2.smithi175.stdout:632: finishing rollback tid 2 to smithi17557870-10 2024-09-11T03:40:36.206 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 181 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T03:40:36.206 INFO:tasks.rados.rados.2.smithi175.stdout:634: finishing write tid 1 to smithi17557870-7 2024-09-11T03:40:36.206 INFO:tasks.rados.rados.2.smithi175.stdout:632: done (4 left) 2024-09-11T03:40:36.206 INFO:tasks.rados.rados.2.smithi175.stdout:633: done (3 left) 2024-09-11T03:40:36.207 INFO:tasks.rados.rados.2.smithi175.stdout:637: write oid 24 current snap is 63 2024-09-11T03:40:36.207 INFO:tasks.rados.rados.2.smithi175.stdout:637: seq_num 255 ranges {685785=628144,1822320=463465} 2024-09-11T03:40:36.212 INFO:tasks.rados.rados.2.smithi175.stdout:637: writing smithi17557870-24 from 685785 to 1313929 tid 1 2024-09-11T03:40:36.218 INFO:tasks.rados.rados.2.smithi175.stdout:637: writing smithi17557870-24 from 1822320 to 2285785 tid 2 2024-09-11T03:40:36.219 INFO:tasks.rados.rados.2.smithi175.stdout:638: rollback oid 41 current snap is 63 2024-09-11T03:40:36.219 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 41 to 54 2024-09-11T03:40:36.219 INFO:tasks.rados.rados.2.smithi175.stdout:635: finishing rollback tid 1 to smithi17557870-42 2024-09-11T03:40:36.219 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 0 (ObjNum 129 snap 0 seq_num 1745149432) dirty dne 2024-09-11T03:40:36.219 INFO:tasks.rados.rados.2.smithi175.stdout:634: finishing write tid 2 to smithi17557870-7 2024-09-11T03:40:36.219 INFO:tasks.rados.rados.2.smithi175.stdout:634: finishing write tid 3 to smithi17557870-7 2024-09-11T03:40:36.219 INFO:tasks.rados.rados.2.smithi175.stdout:634: finishing write tid 4 to smithi17557870-7 2024-09-11T03:40:36.219 INFO:tasks.rados.rados.2.smithi175.stdout:634: oid 7 updating version 0 to 129 2024-09-11T03:40:36.219 INFO:tasks.rados.rados.2.smithi175.stdout:634: oid 7 updating version 129 to 130 2024-09-11T03:40:36.219 INFO:tasks.rados.rados.2.smithi175.stdout:634: oid 7 updating version 130 to 131 2024-09-11T03:40:36.219 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 131 (ObjNum 254 snap 63 seq_num 254) dirty exists 2024-09-11T03:40:36.219 INFO:tasks.rados.rados.2.smithi175.stdout:634: left oid 7 (ObjNum 254 snap 63 seq_num 254) 2024-09-11T03:40:36.219 INFO:tasks.rados.rados.2.smithi175.stdout:637: finishing write tid 1 to smithi17557870-24 2024-09-11T03:40:36.220 INFO:tasks.rados.rados.2.smithi175.stdout:634: done (4 left) 2024-09-11T03:40:36.220 INFO:tasks.rados.rados.2.smithi175.stdout:635: done (3 left) 2024-09-11T03:40:36.220 INFO:tasks.rados.rados.2.smithi175.stdout:639: delete oid 49 current snap is 63 2024-09-11T03:40:36.220 INFO:tasks.rados.rados.2.smithi175.stdout:636: finishing copy_from racing read to smithi17557870-21 2024-09-11T03:40:36.221 INFO:tasks.rados.rados.2.smithi175.stdout:638: finishing rollback tid 0 to smithi17557870-41 2024-09-11T03:40:36.221 INFO:tasks.rados.rados.2.smithi175.stdout:638: finishing rollback tid 1 to smithi17557870-41 2024-09-11T03:40:36.221 INFO:tasks.rados.rados.2.smithi175.stdout:638: finishing rollback tid 2 to smithi17557870-41 2024-09-11T03:40:36.221 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 172 (ObjNum 221 snap 53 seq_num 221) dirty exists 2024-09-11T03:40:36.223 INFO:tasks.rados.rados.2.smithi175.stdout:636: finishing copy_from to smithi17557870-21 2024-09-11T03:40:36.223 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 214 (ObjNum 101 snap 16 seq_num 101) dirty exists 2024-09-11T03:40:36.223 INFO:tasks.rados.rados.2.smithi175.stdout:636: done (3 left) 2024-09-11T03:40:36.223 INFO:tasks.rados.rados.2.smithi175.stdout:638: done (2 left) 2024-09-11T03:40:36.223 INFO:tasks.rados.rados.2.smithi175.stdout:639: done (1 left) 2024-09-11T03:40:36.224 INFO:tasks.rados.rados.2.smithi175.stdout:640: copy_from oid 3 from oid 2 current snap is 63 2024-09-11T03:40:36.224 INFO:tasks.rados.rados.2.smithi175.stdout:641: snap_create 2024-09-11T03:40:36.224 INFO:tasks.rados.rados.2.smithi175.stdout:637: finishing write tid 2 to smithi17557870-24 2024-09-11T03:40:36.224 INFO:tasks.rados.rados.2.smithi175.stdout:637: finishing write tid 3 to smithi17557870-24 2024-09-11T03:40:36.224 INFO:tasks.rados.rados.2.smithi175.stdout:637: finishing write tid 4 to smithi17557870-24 2024-09-11T03:40:36.224 INFO:tasks.rados.rados.2.smithi175.stdout:637: oid 24 updating version 0 to 173 2024-09-11T03:40:36.224 INFO:tasks.rados.rados.2.smithi175.stdout:637: oid 24 version 173 is already newer than 172 2024-09-11T03:40:36.224 INFO:tasks.rados.rados.2.smithi175.stdout:637: oid 24 version 173 is already newer than 171 2024-09-11T03:40:36.224 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 173 (ObjNum 255 snap 63 seq_num 255) dirty exists 2024-09-11T03:40:36.224 INFO:tasks.rados.rados.2.smithi175.stdout:637: left oid 24 (ObjNum 255 snap 63 seq_num 255) 2024-09-11T03:40:36.225 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:36.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:36.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:36.229 INFO:tasks.rados.rados.2.smithi175.stdout:640: finishing copy_from racing read to smithi17557870-3 2024-09-11T03:40:36.231 INFO:tasks.rados.rados.2.smithi175.stdout:640: finishing copy_from to smithi17557870-3 2024-09-11T03:40:36.231 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 176 (ObjNum 248 snap 62 seq_num 248) dirty exists 2024-09-11T03:40:36.328 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:36.349 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:36.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:36.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:36.453 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:36.475 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:36.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:36.577 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:36.598 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:36.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:36.599 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:36.700 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:36.722 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:36.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:36.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:36.824 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:36.845 INFO:tasks.rados.rados.2.smithi175.stdout:637: done (2 left) 2024-09-11T03:40:36.845 INFO:tasks.rados.rados.2.smithi175.stdout:640: done (1 left) 2024-09-11T03:40:36.845 INFO:tasks.rados.rados.2.smithi175.stdout:641: done (0 left) 2024-09-11T03:40:36.845 INFO:tasks.rados.rados.2.smithi175.stdout:642: write oid 13 current snap is 64 2024-09-11T03:40:36.845 INFO:tasks.rados.rados.2.smithi175.stdout:642: seq_num 257 ranges {719399=778642,1939284=380115} 2024-09-11T03:40:36.847 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:36.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:36.848 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:36.853 INFO:tasks.rados.rados.2.smithi175.stdout:642: writing smithi17557870-13 from 719399 to 1498041 tid 1 2024-09-11T03:40:36.858 INFO:tasks.rados.rados.2.smithi175.stdout:642: writing smithi17557870-13 from 1939284 to 2319399 tid 2 2024-09-11T03:40:36.859 INFO:tasks.rados.rados.2.smithi175.stdout:643: read oid 31 snap -1 2024-09-11T03:40:36.859 INFO:tasks.rados.rados.2.smithi175.stdout:643: expect (ObjNum 231 snap 56 seq_num 231) 2024-09-11T03:40:36.859 INFO:tasks.rados.rados.2.smithi175.stdout:644: rollback oid 10 current snap is 64 2024-09-11T03:40:36.859 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 10 to 59 2024-09-11T03:40:36.859 INFO:tasks.rados.rados.2.smithi175.stdout:642: finishing write tid 1 to smithi17557870-13 2024-09-11T03:40:36.859 INFO:tasks.rados.rados.2.smithi175.stdout:645: write (excl) oid 24 current snap is 64 2024-09-11T03:40:36.859 INFO:tasks.rados.rados.2.smithi175.stdout:645: seq_num 258 ranges {736206=453891,1881413=454793} 2024-09-11T03:40:36.865 INFO:tasks.rados.rados.2.smithi175.stdout:645: writing smithi17557870-24 from 736206 to 1190097 tid 1 2024-09-11T03:40:36.871 INFO:tasks.rados.rados.2.smithi175.stdout:645: writing smithi17557870-24 from 1881413 to 2336206 tid 2 2024-09-11T03:40:36.872 INFO:tasks.rados.rados.2.smithi175.stdout:646: snap_create 2024-09-11T03:40:36.872 INFO:tasks.rados.rados.2.smithi175.stdout:644: finishing rollback tid 0 to smithi17557870-10 2024-09-11T03:40:36.872 INFO:tasks.rados.rados.2.smithi175.stdout:644: finishing rollback tid 1 to smithi17557870-10 2024-09-11T03:40:36.872 INFO:tasks.rados.rados.2.smithi175.stdout:644: finishing rollback tid 2 to smithi17557870-10 2024-09-11T03:40:36.872 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 185 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-11T03:40:36.872 INFO:tasks.rados.rados.2.smithi175.stdout:642: finishing write tid 2 to smithi17557870-13 2024-09-11T03:40:36.872 INFO:tasks.rados.rados.2.smithi175.stdout:642: finishing write tid 3 to smithi17557870-13 2024-09-11T03:40:36.872 INFO:tasks.rados.rados.2.smithi175.stdout:642: finishing write tid 4 to smithi17557870-13 2024-09-11T03:40:36.872 INFO:tasks.rados.rados.2.smithi175.stdout:642: oid 13 updating version 0 to 219 2024-09-11T03:40:36.872 INFO:tasks.rados.rados.2.smithi175.stdout:642: oid 13 version 219 is already newer than 218 2024-09-11T03:40:36.872 INFO:tasks.rados.rados.2.smithi175.stdout:642: oid 13 updating version 219 to 220 2024-09-11T03:40:36.872 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 220 (ObjNum 257 snap 64 seq_num 257) dirty exists 2024-09-11T03:40:36.873 INFO:tasks.rados.rados.2.smithi175.stdout:642: left oid 13 (ObjNum 257 snap 64 seq_num 257) 2024-09-11T03:40:36.873 INFO:tasks.rados.rados.2.smithi175.stdout:643: expect (ObjNum 231 snap 56 seq_num 231) 2024-09-11T03:40:36.949 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:36.970 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:36.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:36.986 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-11T03:40:37.072 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:37.090 INFO:tasks.rados.rados.2.smithi175.stdout:645: finishing write tid 1 to smithi17557870-24 2024-09-11T03:40:37.090 INFO:tasks.rados.rados.2.smithi175.stdout:645: finishing write tid 2 to smithi17557870-24 2024-09-11T03:40:37.090 INFO:tasks.rados.rados.2.smithi175.stdout:645: finishing write tid 3 to smithi17557870-24 2024-09-11T03:40:37.090 INFO:tasks.rados.rados.2.smithi175.stdout:645: finishing write tid 4 to smithi17557870-24 2024-09-11T03:40:37.090 INFO:tasks.rados.rados.2.smithi175.stdout:645: oid 24 updating version 0 to 175 2024-09-11T03:40:37.090 INFO:tasks.rados.rados.2.smithi175.stdout:645: oid 24 updating version 175 to 176 2024-09-11T03:40:37.091 INFO:tasks.rados.rados.2.smithi175.stdout:645: oid 24 updating version 176 to 177 2024-09-11T03:40:37.091 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 177 (ObjNum 258 snap 64 seq_num 258) dirty exists 2024-09-11T03:40:37.091 INFO:tasks.rados.rados.2.smithi175.stdout:645: left oid 24 (ObjNum 258 snap 64 seq_num 258) 2024-09-11T03:40:37.092 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:37.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:37.093 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:37.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.e scrub starts 2024-09-11T03:40:37.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.e scrub ok 2024-09-11T03:40:37.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.4 scrub starts 2024-09-11T03:40:37.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.4 scrub ok 2024-09-11T03:40:37.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.13 scrub starts 2024-09-11T03:40:37.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.13 scrub ok 2024-09-11T03:40:37.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.f scrub starts 2024-09-11T03:40:37.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.f scrub ok 2024-09-11T03:40:37.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.11 scrub starts 2024-09-11T03:40:37.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.11 scrub ok 2024-09-11T03:40:37.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.d scrub starts 2024-09-11T03:40:37.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.d scrub ok 2024-09-11T03:40:37.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: osdmap e219: 12 total, 12 up, 10 in 2024-09-11T03:40:37.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.5 scrub starts 2024-09-11T03:40:37.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.5 scrub ok 2024-09-11T03:40:37.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.7 scrub starts 2024-09-11T03:40:37.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.7 scrub ok 2024-09-11T03:40:37.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.6 scrub starts 2024-09-11T03:40:37.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: 1.6 scrub ok 2024-09-11T03:40:37.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-11T03:40:37.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:37.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-11T03:40:37.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:36 smithi084 ceph-mon[27254]: osdmap e220: 12 total, 12 up, 10 in 2024-09-11T03:40:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.e scrub starts 2024-09-11T03:40:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.e scrub ok 2024-09-11T03:40:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.4 scrub starts 2024-09-11T03:40:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.4 scrub ok 2024-09-11T03:40:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.13 scrub starts 2024-09-11T03:40:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.13 scrub ok 2024-09-11T03:40:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.f scrub starts 2024-09-11T03:40:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.f scrub ok 2024-09-11T03:40:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.11 scrub starts 2024-09-11T03:40:37.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.11 scrub ok 2024-09-11T03:40:37.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.d scrub starts 2024-09-11T03:40:37.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.d scrub ok 2024-09-11T03:40:37.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: osdmap e219: 12 total, 12 up, 10 in 2024-09-11T03:40:37.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.5 scrub starts 2024-09-11T03:40:37.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.5 scrub ok 2024-09-11T03:40:37.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.7 scrub starts 2024-09-11T03:40:37.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.7 scrub ok 2024-09-11T03:40:37.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.6 scrub starts 2024-09-11T03:40:37.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: 1.6 scrub ok 2024-09-11T03:40:37.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-11T03:40:37.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:37.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-11T03:40:37.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:36 smithi121 ceph-mon[30729]: osdmap e220: 12 total, 12 up, 10 in 2024-09-11T03:40:37.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:37.219 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:37.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:37.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.e scrub starts 2024-09-11T03:40:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.e scrub ok 2024-09-11T03:40:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.4 scrub starts 2024-09-11T03:40:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.4 scrub ok 2024-09-11T03:40:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.13 scrub starts 2024-09-11T03:40:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.13 scrub ok 2024-09-11T03:40:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.f scrub starts 2024-09-11T03:40:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.f scrub ok 2024-09-11T03:40:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.11 scrub starts 2024-09-11T03:40:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.11 scrub ok 2024-09-11T03:40:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.d scrub starts 2024-09-11T03:40:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.d scrub ok 2024-09-11T03:40:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: osdmap e219: 12 total, 12 up, 10 in 2024-09-11T03:40:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.5 scrub starts 2024-09-11T03:40:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.5 scrub ok 2024-09-11T03:40:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.7 scrub starts 2024-09-11T03:40:37.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.7 scrub ok 2024-09-11T03:40:37.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.6 scrub starts 2024-09-11T03:40:37.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: 1.6 scrub ok 2024-09-11T03:40:37.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-11T03:40:37.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:37.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-11T03:40:37.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:36 smithi175 ceph-mon[30894]: osdmap e220: 12 total, 12 up, 10 in 2024-09-11T03:40:37.278 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:40:37.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:37.347 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:37.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:37.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:37.449 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:37.472 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:37.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:37.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:37.574 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:37.598 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:37.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:37.599 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:37.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:37.725 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:37.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:37.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:37.769 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:40:37.769 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:40:37.769 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:40:37.770 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:40:37.770 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:40:37.827 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:37.845 INFO:tasks.rados.rados.2.smithi175.stdout:642: done (4 left) 2024-09-11T03:40:37.845 INFO:tasks.rados.rados.2.smithi175.stdout:643: done (3 left) 2024-09-11T03:40:37.845 INFO:tasks.rados.rados.2.smithi175.stdout:644: done (2 left) 2024-09-11T03:40:37.845 INFO:tasks.rados.rados.2.smithi175.stdout:645: done (1 left) 2024-09-11T03:40:37.845 INFO:tasks.rados.rados.2.smithi175.stdout:646: done (0 left) 2024-09-11T03:40:37.845 INFO:tasks.rados.rados.2.smithi175.stdout:647: snap_remove snap 54 2024-09-11T03:40:37.853 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:40:37.857 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:37.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:37.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:37.959 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:37.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:37 smithi175 ceph-mon[30894]: 1.b scrub starts 2024-09-11T03:40:37.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:37 smithi175 ceph-mon[30894]: 1.b scrub ok 2024-09-11T03:40:37.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:37 smithi175 ceph-mon[30894]: 1.3 scrub starts 2024-09-11T03:40:37.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:37 smithi175 ceph-mon[30894]: 1.3 scrub ok 2024-09-11T03:40:37.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:37 smithi175 ceph-mon[30894]: 1.10 scrub starts 2024-09-11T03:40:37.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:37 smithi175 ceph-mon[30894]: 1.10 scrub ok 2024-09-11T03:40:37.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:37 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:40:37.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:37 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:40:37.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:37 smithi175 ceph-mon[30894]: pgmap v484: 37 pgs: 37 active+clean; 154 MiB data, 604 MiB used, 893 GiB / 894 GiB avail; 14 MiB/s rd, 4.3 MiB/s wr, 18 op/s 2024-09-11T03:40:37.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:37 smithi175 ceph-mon[30894]: 1.0 scrub starts 2024-09-11T03:40:37.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:37 smithi175 ceph-mon[30894]: 1.0 scrub ok 2024-09-11T03:40:37.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:37 smithi175 ceph-mon[30894]: osdmap e221: 12 total, 12 up, 10 in 2024-09-11T03:40:37.981 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:37.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:37.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:38.085 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:38.109 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:38.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:38.113 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:38.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:37 smithi084 ceph-mon[27254]: 1.b scrub starts 2024-09-11T03:40:38.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:37 smithi084 ceph-mon[27254]: 1.b scrub ok 2024-09-11T03:40:38.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:37 smithi084 ceph-mon[27254]: 1.3 scrub starts 2024-09-11T03:40:38.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:37 smithi084 ceph-mon[27254]: 1.3 scrub ok 2024-09-11T03:40:38.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:37 smithi084 ceph-mon[27254]: 1.10 scrub starts 2024-09-11T03:40:38.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:37 smithi084 ceph-mon[27254]: 1.10 scrub ok 2024-09-11T03:40:38.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:37 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:40:38.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:37 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:40:38.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:37 smithi084 ceph-mon[27254]: pgmap v484: 37 pgs: 37 active+clean; 154 MiB data, 604 MiB used, 893 GiB / 894 GiB avail; 14 MiB/s rd, 4.3 MiB/s wr, 18 op/s 2024-09-11T03:40:38.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:37 smithi084 ceph-mon[27254]: 1.0 scrub starts 2024-09-11T03:40:38.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:37 smithi084 ceph-mon[27254]: 1.0 scrub ok 2024-09-11T03:40:38.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:37 smithi084 ceph-mon[27254]: osdmap e221: 12 total, 12 up, 10 in 2024-09-11T03:40:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:37 smithi121 ceph-mon[30729]: 1.b scrub starts 2024-09-11T03:40:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:37 smithi121 ceph-mon[30729]: 1.b scrub ok 2024-09-11T03:40:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:37 smithi121 ceph-mon[30729]: 1.3 scrub starts 2024-09-11T03:40:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:37 smithi121 ceph-mon[30729]: 1.3 scrub ok 2024-09-11T03:40:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:37 smithi121 ceph-mon[30729]: 1.10 scrub starts 2024-09-11T03:40:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:37 smithi121 ceph-mon[30729]: 1.10 scrub ok 2024-09-11T03:40:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:37 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:40:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:37 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:40:38.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:37 smithi121 ceph-mon[30729]: pgmap v484: 37 pgs: 37 active+clean; 154 MiB data, 604 MiB used, 893 GiB / 894 GiB avail; 14 MiB/s rd, 4.3 MiB/s wr, 18 op/s 2024-09-11T03:40:38.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:37 smithi121 ceph-mon[30729]: 1.0 scrub starts 2024-09-11T03:40:38.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:37 smithi121 ceph-mon[30729]: 1.0 scrub ok 2024-09-11T03:40:38.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:37 smithi121 ceph-mon[30729]: osdmap e221: 12 total, 12 up, 10 in 2024-09-11T03:40:38.210 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:38.232 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:38.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:38.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:38.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:40:38.334 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:38.387 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:38.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:38.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:38.426 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-11T03:40:38.488 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:38.510 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:38.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:38.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:38.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:38.633 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:38.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:38.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:38.657 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:40:38.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:40:38.670 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:40:38.673 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:40:38.673 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:40:38.689 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:40:38.689 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:40:38.706 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:40:38.706 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:40:38.726 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:40:38.726 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:40:38.735 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:38.748 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:40:38.749 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:40:38.754 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:38.757 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:38.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:38.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:38.766 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:40:38.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:40:38.783 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:40:38.783 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:40:38.800 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:40:38.800 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:40:38.817 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:40:38.818 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:40:38.834 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:40:38.834 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:40:38.848 INFO:tasks.rados.rados.2.smithi175.stdout:647: done (0 left) 2024-09-11T03:40:38.848 INFO:tasks.rados.rados.2.smithi175.stdout:648: snap_create 2024-09-11T03:40:38.857 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:40:38.857 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:40:38.858 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:38.879 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:38.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:38.981 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:39.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:39.004 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:39.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:39.068 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:38 smithi175 ceph-mon[30894]: 1.12 scrub starts 2024-09-11T03:40:39.068 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:38 smithi175 ceph-mon[30894]: 1.12 scrub ok 2024-09-11T03:40:39.068 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:38 smithi175 ceph-mon[30894]: 1.9 scrub starts 2024-09-11T03:40:39.068 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:38 smithi175 ceph-mon[30894]: 1.9 scrub ok 2024-09-11T03:40:39.068 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:38 smithi175 ceph-mon[30894]: 1.c scrub starts 2024-09-11T03:40:39.068 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:38 smithi175 ceph-mon[30894]: 1.c scrub ok 2024-09-11T03:40:39.068 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:38 smithi175 ceph-mon[30894]: 1.6 scrub starts 2024-09-11T03:40:39.068 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:38 smithi175 ceph-mon[30894]: 1.6 scrub ok 2024-09-11T03:40:39.068 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:38 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2105635343' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:39.069 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:38 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-11T03:40:39.069 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:38 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-11T03:40:39.069 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:38 smithi175 ceph-mon[30894]: osdmap e222: 12 total, 12 up, 10 in 2024-09-11T03:40:39.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:39.126 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:39.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:39.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:38 smithi084 ceph-mon[27254]: 1.12 scrub starts 2024-09-11T03:40:39.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:38 smithi084 ceph-mon[27254]: 1.12 scrub ok 2024-09-11T03:40:39.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:38 smithi084 ceph-mon[27254]: 1.9 scrub starts 2024-09-11T03:40:39.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:38 smithi084 ceph-mon[27254]: 1.9 scrub ok 2024-09-11T03:40:39.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:38 smithi084 ceph-mon[27254]: 1.c scrub starts 2024-09-11T03:40:39.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:38 smithi084 ceph-mon[27254]: 1.c scrub ok 2024-09-11T03:40:39.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:38 smithi084 ceph-mon[27254]: 1.6 scrub starts 2024-09-11T03:40:39.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:38 smithi084 ceph-mon[27254]: 1.6 scrub ok 2024-09-11T03:40:39.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:38 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2105635343' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:39.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:38 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-11T03:40:39.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:38 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-11T03:40:39.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:38 smithi084 ceph-mon[27254]: osdmap e222: 12 total, 12 up, 10 in 2024-09-11T03:40:39.167 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:39.167 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:39.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:38 smithi121 ceph-mon[30729]: 1.12 scrub starts 2024-09-11T03:40:39.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:38 smithi121 ceph-mon[30729]: 1.12 scrub ok 2024-09-11T03:40:39.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:38 smithi121 ceph-mon[30729]: 1.9 scrub starts 2024-09-11T03:40:39.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:38 smithi121 ceph-mon[30729]: 1.9 scrub ok 2024-09-11T03:40:39.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:38 smithi121 ceph-mon[30729]: 1.c scrub starts 2024-09-11T03:40:39.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:38 smithi121 ceph-mon[30729]: 1.c scrub ok 2024-09-11T03:40:39.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:38 smithi121 ceph-mon[30729]: 1.6 scrub starts 2024-09-11T03:40:39.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:38 smithi121 ceph-mon[30729]: 1.6 scrub ok 2024-09-11T03:40:39.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:38 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2105635343' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:39.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:38 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-11T03:40:39.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:38 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-11T03:40:39.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:38 smithi121 ceph-mon[30729]: osdmap e222: 12 total, 12 up, 10 in 2024-09-11T03:40:39.228 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:39.249 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:39.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:39.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:39.351 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:39.373 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:39.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:39.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:39.475 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:39.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:39.497 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:39.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:39.598 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:39.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:39.621 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:39.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:39.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:39.744 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:39.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:39.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:39.842 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-11T03:40:39.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:39.846 INFO:tasks.rados.rados.2.smithi175.stdout:648: done (0 left) 2024-09-11T03:40:39.846 INFO:tasks.rados.rados.2.smithi175.stdout:649: write oid 14 current snap is 66 2024-09-11T03:40:39.847 INFO:tasks.rados.rados.2.smithi175.stdout:649: seq_num 259 ranges {753013=645493,2056248=296765} 2024-09-11T03:40:39.854 INFO:tasks.rados.rados.2.smithi175.stdout:649: writing smithi17557870-14 from 753013 to 1398506 tid 1 2024-09-11T03:40:39.860 INFO:tasks.rados.rados.2.smithi175.stdout:649: writing smithi17557870-14 from 2056248 to 2353013 tid 2 2024-09-11T03:40:39.860 INFO:tasks.rados.rados.2.smithi175.stdout:650: rollback oid 19 current snap is 66 2024-09-11T03:40:39.861 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 19 to 61 2024-09-11T03:40:39.861 INFO:tasks.rados.rados.2.smithi175.stdout:649: finishing write tid 1 to smithi17557870-14 2024-09-11T03:40:39.861 INFO:tasks.rados.rados.2.smithi175.stdout:651: write oid 26 current snap is 66 2024-09-11T03:40:39.861 INFO:tasks.rados.rados.2.smithi175.stdout:651: seq_num 260 ranges {769820=720742,2114730=255090} 2024-09-11T03:40:39.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:39.869 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:39.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:39.869 INFO:tasks.rados.rados.2.smithi175.stdout:651: writing smithi17557870-26 from 769820 to 1490562 tid 1 2024-09-11T03:40:39.874 INFO:tasks.rados.rados.2.smithi175.stdout:651: writing smithi17557870-26 from 2114730 to 2369820 tid 2 2024-09-11T03:40:39.874 INFO:tasks.rados.rados.2.smithi175.stdout:652: copy_from oid 37 from oid 16 current snap is 66 2024-09-11T03:40:39.874 INFO:tasks.rados.rados.2.smithi175.stdout:650: finishing rollback tid 1 to smithi17557870-19 2024-09-11T03:40:39.874 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-11T03:40:39.874 INFO:tasks.rados.rados.2.smithi175.stdout:650: done (3 left) 2024-09-11T03:40:39.874 INFO:tasks.rados.rados.2.smithi175.stdout:653: write (excl) oid 18 current snap is 66 2024-09-11T03:40:39.874 INFO:tasks.rados.rados.2.smithi175.stdout:649: finishing write tid 2 to smithi17557870-14 2024-09-11T03:40:39.874 INFO:tasks.rados.rados.2.smithi175.stdout:649: finishing write tid 3 to smithi17557870-14 2024-09-11T03:40:39.874 INFO:tasks.rados.rados.2.smithi175.stdout:649: finishing write tid 4 to smithi17557870-14 2024-09-11T03:40:39.874 INFO:tasks.rados.rados.2.smithi175.stdout:649: oid 14 updating version 0 to 184 2024-09-11T03:40:39.874 INFO:tasks.rados.rados.2.smithi175.stdout:649: oid 14 updating version 184 to 185 2024-09-11T03:40:39.875 INFO:tasks.rados.rados.2.smithi175.stdout:649: oid 14 version 185 is already newer than 183 2024-09-11T03:40:39.875 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 185 (ObjNum 259 snap 66 seq_num 259) dirty exists 2024-09-11T03:40:39.875 INFO:tasks.rados.rados.2.smithi175.stdout:649: left oid 14 (ObjNum 259 snap 66 seq_num 259) 2024-09-11T03:40:39.875 INFO:tasks.rados.rados.2.smithi175.stdout:651: finishing write tid 1 to smithi17557870-26 2024-09-11T03:40:39.875 INFO:tasks.rados.rados.2.smithi175.stdout:653: seq_num 261 ranges {786627=795991,2173212=213415} 2024-09-11T03:40:39.882 INFO:tasks.rados.rados.2.smithi175.stdout:653: writing smithi17557870-18 from 786627 to 1582618 tid 1 2024-09-11T03:40:39.887 INFO:tasks.rados.rados.2.smithi175.stdout:653: writing smithi17557870-18 from 2173212 to 2386627 tid 2 2024-09-11T03:40:39.887 INFO:tasks.rados.rados.2.smithi175.stdout:649: done (3 left) 2024-09-11T03:40:39.887 INFO:tasks.rados.rados.2.smithi175.stdout:654: copy_from oid 5 from oid 7 current snap is 66 2024-09-11T03:40:39.887 INFO:tasks.rados.rados.2.smithi175.stdout:651: finishing write tid 2 to smithi17557870-26 2024-09-11T03:40:39.887 INFO:tasks.rados.rados.2.smithi175.stdout:651: finishing write tid 3 to smithi17557870-26 2024-09-11T03:40:39.887 INFO:tasks.rados.rados.2.smithi175.stdout:651: finishing write tid 4 to smithi17557870-26 2024-09-11T03:40:39.887 INFO:tasks.rados.rados.2.smithi175.stdout:651: oid 26 updating version 0 to 111 2024-09-11T03:40:39.887 INFO:tasks.rados.rados.2.smithi175.stdout:651: oid 26 updating version 111 to 112 2024-09-11T03:40:39.888 INFO:tasks.rados.rados.2.smithi175.stdout:651: oid 26 version 112 is already newer than 110 2024-09-11T03:40:39.888 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 112 (ObjNum 260 snap 66 seq_num 260) dirty exists 2024-09-11T03:40:39.888 INFO:tasks.rados.rados.2.smithi175.stdout:651: left oid 26 (ObjNum 260 snap 66 seq_num 260) 2024-09-11T03:40:39.888 INFO:tasks.rados.rados.2.smithi175.stdout:652: finishing copy_from racing read to smithi17557870-37 2024-09-11T03:40:39.888 INFO:tasks.rados.rados.2.smithi175.stdout:651: done (3 left) 2024-09-11T03:40:39.888 INFO:tasks.rados.rados.2.smithi175.stdout:655: read oid 40 snap -1 2024-09-11T03:40:39.888 INFO:tasks.rados.rados.2.smithi175.stdout:655: expect deleted 2024-09-11T03:40:39.888 INFO:tasks.rados.rados.2.smithi175.stdout:656: snap_remove snap 64 2024-09-11T03:40:39.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:39.992 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:39.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:39.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:40.090 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:40:40.093 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:40.116 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:40.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:40.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:40.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:39 smithi121 ceph-mon[30729]: 1.e scrub starts 2024-09-11T03:40:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:39 smithi121 ceph-mon[30729]: 1.e scrub ok 2024-09-11T03:40:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:39 smithi121 ceph-mon[30729]: 1.5 scrub starts 2024-09-11T03:40:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:39 smithi121 ceph-mon[30729]: 1.5 scrub ok 2024-09-11T03:40:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:39 smithi121 ceph-mon[30729]: 1.1b scrub starts 2024-09-11T03:40:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:39 smithi121 ceph-mon[30729]: 1.1b scrub ok 2024-09-11T03:40:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:39 smithi121 ceph-mon[30729]: 1.8 scrub starts 2024-09-11T03:40:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:39 smithi121 ceph-mon[30729]: 1.8 scrub ok 2024-09-11T03:40:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:39 smithi121 ceph-mon[30729]: pgmap v487: 47 pgs: 1 peering, 7 unknown, 39 active+clean; 164 MiB data, 621 MiB used, 893 GiB / 894 GiB avail; 16 MiB/s rd, 5.4 MiB/s wr, 12 op/s 2024-09-11T03:40:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:39 smithi121 ceph-mon[30729]: 1.d scrub starts 2024-09-11T03:40:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:39 smithi121 ceph-mon[30729]: 1.d scrub ok 2024-09-11T03:40:40.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:39 smithi121 ceph-mon[30729]: 1.7 scrub starts 2024-09-11T03:40:40.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:39 smithi121 ceph-mon[30729]: 1.7 scrub ok 2024-09-11T03:40:40.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:39 smithi121 ceph-mon[30729]: 1.a scrub starts 2024-09-11T03:40:40.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:39 smithi121 ceph-mon[30729]: 1.a scrub ok 2024-09-11T03:40:40.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:39 smithi121 ceph-mon[30729]: osdmap e223: 12 total, 12 up, 10 in 2024-09-11T03:40:40.210 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:39 smithi175 ceph-mon[30894]: 1.e scrub starts 2024-09-11T03:40:40.210 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:39 smithi175 ceph-mon[30894]: 1.e scrub ok 2024-09-11T03:40:40.211 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:39 smithi175 ceph-mon[30894]: 1.5 scrub starts 2024-09-11T03:40:40.211 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:39 smithi175 ceph-mon[30894]: 1.5 scrub ok 2024-09-11T03:40:40.211 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:39 smithi175 ceph-mon[30894]: 1.1b scrub starts 2024-09-11T03:40:40.211 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:39 smithi175 ceph-mon[30894]: 1.1b scrub ok 2024-09-11T03:40:40.211 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:39 smithi175 ceph-mon[30894]: 1.8 scrub starts 2024-09-11T03:40:40.211 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:39 smithi175 ceph-mon[30894]: 1.8 scrub ok 2024-09-11T03:40:40.211 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:39 smithi175 ceph-mon[30894]: pgmap v487: 47 pgs: 1 peering, 7 unknown, 39 active+clean; 164 MiB data, 621 MiB used, 893 GiB / 894 GiB avail; 16 MiB/s rd, 5.4 MiB/s wr, 12 op/s 2024-09-11T03:40:40.211 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:39 smithi175 ceph-mon[30894]: 1.d scrub starts 2024-09-11T03:40:40.211 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:39 smithi175 ceph-mon[30894]: 1.d scrub ok 2024-09-11T03:40:40.211 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:39 smithi175 ceph-mon[30894]: 1.7 scrub starts 2024-09-11T03:40:40.211 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:39 smithi175 ceph-mon[30894]: 1.7 scrub ok 2024-09-11T03:40:40.211 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:39 smithi175 ceph-mon[30894]: 1.a scrub starts 2024-09-11T03:40:40.211 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:39 smithi175 ceph-mon[30894]: 1.a scrub ok 2024-09-11T03:40:40.211 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:39 smithi175 ceph-mon[30894]: osdmap e223: 12 total, 12 up, 10 in 2024-09-11T03:40:40.218 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:40.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:40.240 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:40.240 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:40.341 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:40.362 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:40.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:40.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:39 smithi084 ceph-mon[27254]: 1.e scrub starts 2024-09-11T03:40:40.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:39 smithi084 ceph-mon[27254]: 1.e scrub ok 2024-09-11T03:40:40.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:39 smithi084 ceph-mon[27254]: 1.5 scrub starts 2024-09-11T03:40:40.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:39 smithi084 ceph-mon[27254]: 1.5 scrub ok 2024-09-11T03:40:40.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:39 smithi084 ceph-mon[27254]: 1.1b scrub starts 2024-09-11T03:40:40.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:39 smithi084 ceph-mon[27254]: 1.1b scrub ok 2024-09-11T03:40:40.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:39 smithi084 ceph-mon[27254]: 1.8 scrub starts 2024-09-11T03:40:40.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:39 smithi084 ceph-mon[27254]: 1.8 scrub ok 2024-09-11T03:40:40.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:39 smithi084 ceph-mon[27254]: pgmap v487: 47 pgs: 1 peering, 7 unknown, 39 active+clean; 164 MiB data, 621 MiB used, 893 GiB / 894 GiB avail; 16 MiB/s rd, 5.4 MiB/s wr, 12 op/s 2024-09-11T03:40:40.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:39 smithi084 ceph-mon[27254]: 1.d scrub starts 2024-09-11T03:40:40.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:39 smithi084 ceph-mon[27254]: 1.d scrub ok 2024-09-11T03:40:40.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:39 smithi084 ceph-mon[27254]: 1.7 scrub starts 2024-09-11T03:40:40.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:39 smithi084 ceph-mon[27254]: 1.7 scrub ok 2024-09-11T03:40:40.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:39 smithi084 ceph-mon[27254]: 1.a scrub starts 2024-09-11T03:40:40.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:39 smithi084 ceph-mon[27254]: 1.a scrub ok 2024-09-11T03:40:40.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:39 smithi084 ceph-mon[27254]: osdmap e223: 12 total, 12 up, 10 in 2024-09-11T03:40:40.464 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:40.485 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:40.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:40.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:40.566 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:40.566 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:40.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:40.611 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:40.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:40.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:40.713 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:40.735 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:40.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:40.736 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:40.837 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:40.844 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:40:40.850 INFO:tasks.rados.rados.2.smithi175.stdout:656: done (4 left) 2024-09-11T03:40:40.851 INFO:tasks.rados.rados.2.smithi175.stdout:657: snap_remove snap 62 2024-09-11T03:40:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:40.863 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:40.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:40.964 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:40.986 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:40.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:40.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:41.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:41.114 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:41.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:41.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:41.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:40 smithi084 ceph-mon[27254]: 1.b scrub starts 2024-09-11T03:40:41.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:40 smithi084 ceph-mon[27254]: 1.b scrub ok 2024-09-11T03:40:41.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:40 smithi084 ceph-mon[27254]: 1.15 scrub starts 2024-09-11T03:40:41.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:40 smithi084 ceph-mon[27254]: 1.15 scrub ok 2024-09-11T03:40:41.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:40 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1429298874' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:40:41.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:40 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:40:41.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:40 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:40:41.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:40 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1429298874' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:40:41.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:40 smithi084 ceph-mon[27254]: osdmap e224: 12 total, 12 up, 10 in 2024-09-11T03:40:41.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:40 smithi121 ceph-mon[30729]: 1.b scrub starts 2024-09-11T03:40:41.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:40 smithi121 ceph-mon[30729]: 1.b scrub ok 2024-09-11T03:40:41.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:40 smithi121 ceph-mon[30729]: 1.15 scrub starts 2024-09-11T03:40:41.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:40 smithi121 ceph-mon[30729]: 1.15 scrub ok 2024-09-11T03:40:41.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:40 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1429298874' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:40:41.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:40 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:40:41.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:40 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:40:41.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:40 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1429298874' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:40:41.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:40 smithi121 ceph-mon[30729]: osdmap e224: 12 total, 12 up, 10 in 2024-09-11T03:40:41.172 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:40 smithi175 ceph-mon[30894]: 1.b scrub starts 2024-09-11T03:40:41.173 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:40 smithi175 ceph-mon[30894]: 1.b scrub ok 2024-09-11T03:40:41.173 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:40 smithi175 ceph-mon[30894]: 1.15 scrub starts 2024-09-11T03:40:41.173 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:40 smithi175 ceph-mon[30894]: 1.15 scrub ok 2024-09-11T03:40:41.173 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:40 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1429298874' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:40:41.173 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:40 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:40:41.173 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:40 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:40:41.173 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:40 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1429298874' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:40:41.173 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:40 smithi175 ceph-mon[30894]: osdmap e224: 12 total, 12 up, 10 in 2024-09-11T03:40:41.174 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-11T03:40:41.216 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:41.238 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:41.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:41.239 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:41.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:41.365 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:41.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:41.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:41.419 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:40:41.466 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:41.488 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:41.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:41.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:41.563 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:40:41.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:41.616 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:41.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:41.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:41.718 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:41.744 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:41.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:41.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:41.800 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [6, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:40:41.800 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:40:41.800 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:40:41.806 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:41.845 INFO:tasks.rados.rados.2.smithi175.stdout:657: done (4 left) 2024-09-11T03:40:41.845 INFO:tasks.rados.rados.2.smithi175.stdout:658: copy_from oid 11 from oid 43 current snap is 66 2024-09-11T03:40:41.845 INFO:tasks.rados.rados.2.smithi175.stdout:659: write oid 20 current snap is 66 2024-09-11T03:40:41.846 INFO:tasks.rados.rados.2.smithi175.stdout:659: seq_num 262 ranges {403434=471240,1315341=400756} 2024-09-11T03:40:41.846 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:41.850 INFO:tasks.rados.rados.2.smithi175.stdout:659: writing smithi17557870-20 from 403434 to 874674 tid 1 2024-09-11T03:40:41.856 INFO:tasks.rados.rados.2.smithi175.stdout:659: writing smithi17557870-20 from 1315341 to 1716097 tid 2 2024-09-11T03:40:41.856 INFO:tasks.rados.rados.2.smithi175.stdout:660: copy_from oid 49 from oid 44 current snap is 66 2024-09-11T03:40:41.856 INFO:tasks.rados.rados.2.smithi175.stdout:652: finishing copy_from to smithi17557870-37 2024-09-11T03:40:41.856 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 101 (ObjNum 153 snap 35 seq_num 153) dirty exists 2024-09-11T03:40:41.856 INFO:tasks.rados.rados.2.smithi175.stdout:653: finishing write tid 1 to smithi17557870-18 2024-09-11T03:40:41.856 INFO:tasks.rados.rados.2.smithi175.stdout:653: finishing write tid 2 to smithi17557870-18 2024-09-11T03:40:41.856 INFO:tasks.rados.rados.2.smithi175.stdout:653: finishing write tid 3 to smithi17557870-18 2024-09-11T03:40:41.857 INFO:tasks.rados.rados.2.smithi175.stdout:653: finishing write tid 4 to smithi17557870-18 2024-09-11T03:40:41.857 INFO:tasks.rados.rados.2.smithi175.stdout:653: oid 18 updating version 0 to 153 2024-09-11T03:40:41.857 INFO:tasks.rados.rados.2.smithi175.stdout:653: oid 18 version 153 is already newer than 151 2024-09-11T03:40:41.857 INFO:tasks.rados.rados.2.smithi175.stdout:653: oid 18 version 153 is already newer than 152 2024-09-11T03:40:41.857 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 153 (ObjNum 261 snap 66 seq_num 261) dirty exists 2024-09-11T03:40:41.857 INFO:tasks.rados.rados.2.smithi175.stdout:653: left oid 18 (ObjNum 261 snap 66 seq_num 261) 2024-09-11T03:40:41.857 INFO:tasks.rados.rados.2.smithi175.stdout:654: finishing copy_from racing read to smithi17557870-5 2024-09-11T03:40:41.857 INFO:tasks.rados.rados.2.smithi175.stdout:654: finishing copy_from to smithi17557870-5 2024-09-11T03:40:41.857 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 131 (ObjNum 254 snap 63 seq_num 254) dirty exists 2024-09-11T03:40:41.857 INFO:tasks.rados.rados.2.smithi175.stdout:658: finishing copy_from to smithi17557870-11 2024-09-11T03:40:41.857 INFO:tasks.rados.rados.2.smithi175.stdout:658: got expected ENOENT (src dne) 2024-09-11T03:40:41.857 INFO:tasks.rados.rados.2.smithi175.stdout:658: finishing copy_from racing read to smithi17557870-11 2024-09-11T03:40:41.858 INFO:tasks.rados.rados.2.smithi175.stdout:659: finishing write tid 1 to smithi17557870-20 2024-09-11T03:40:41.858 INFO:tasks.rados.rados.2.smithi175.stdout:652: done (6 left) 2024-09-11T03:40:41.858 INFO:tasks.rados.rados.2.smithi175.stdout:653: done (5 left) 2024-09-11T03:40:41.858 INFO:tasks.rados.rados.2.smithi175.stdout:654: done (4 left) 2024-09-11T03:40:41.858 INFO:tasks.rados.rados.2.smithi175.stdout:655: done (3 left) 2024-09-11T03:40:41.858 INFO:tasks.rados.rados.2.smithi175.stdout:658: done (2 left) 2024-09-11T03:40:41.858 INFO:tasks.rados.rados.2.smithi175.stdout:661: read oid 46 snap -1 2024-09-11T03:40:41.858 INFO:tasks.rados.rados.2.smithi175.stdout:661: expect (ObjNum 239 snap 58 seq_num 239) 2024-09-11T03:40:41.858 INFO:tasks.rados.rados.2.smithi175.stdout:662: read oid 48 snap 63 2024-09-11T03:40:41.858 INFO:tasks.rados.rados.2.smithi175.stdout:662: expect (ObjNum 246 snap 60 seq_num 246) 2024-09-11T03:40:41.858 INFO:tasks.rados.rados.2.smithi175.stdout:663: delete oid 8 current snap is 66 2024-09-11T03:40:41.859 INFO:tasks.rados.rados.2.smithi175.stdout:663: done (4 left) 2024-09-11T03:40:41.860 INFO:tasks.rados.rados.2.smithi175.stdout:664: delete oid 2 current snap is 66 2024-09-11T03:40:41.860 INFO:tasks.rados.rados.2.smithi175.stdout:659: finishing write tid 2 to smithi17557870-20 2024-09-11T03:40:41.860 INFO:tasks.rados.rados.2.smithi175.stdout:659: finishing write tid 3 to smithi17557870-20 2024-09-11T03:40:41.860 INFO:tasks.rados.rados.2.smithi175.stdout:659: finishing write tid 4 to smithi17557870-20 2024-09-11T03:40:41.860 INFO:tasks.rados.rados.2.smithi175.stdout:659: oid 20 updating version 0 to 194 2024-09-11T03:40:41.860 INFO:tasks.rados.rados.2.smithi175.stdout:659: oid 20 updating version 194 to 195 2024-09-11T03:40:41.860 INFO:tasks.rados.rados.2.smithi175.stdout:659: oid 20 version 195 is already newer than 193 2024-09-11T03:40:41.860 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 195 (ObjNum 262 snap 66 seq_num 262) dirty exists 2024-09-11T03:40:41.860 INFO:tasks.rados.rados.2.smithi175.stdout:659: left oid 20 (ObjNum 262 snap 66 seq_num 262) 2024-09-11T03:40:41.860 INFO:tasks.rados.rados.2.smithi175.stdout:659: done (4 left) 2024-09-11T03:40:41.860 INFO:tasks.rados.rados.2.smithi175.stdout:664: done (3 left) 2024-09-11T03:40:41.860 INFO:tasks.rados.rados.2.smithi175.stdout:665: copy_from oid 5 from oid 21 current snap is 66 2024-09-11T03:40:41.861 INFO:tasks.rados.rados.2.smithi175.stdout:666: write oid 8 current snap is 66 2024-09-11T03:40:41.861 INFO:tasks.rados.rados.2.smithi175.stdout:666: seq_num 265 ranges {453855=696987,1890787=548083} 2024-09-11T03:40:41.868 INFO:tasks.rados.rados.2.smithi175.stdout:666: writing smithi17557870-8 from 453855 to 1150842 tid 1 2024-09-11T03:40:41.873 INFO:tasks.rados.rados.2.smithi175.stdout:666: writing smithi17557870-8 from 1890787 to 2438870 tid 2 2024-09-11T03:40:41.874 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:41.874 INFO:tasks.rados.rados.2.smithi175.stdout:667: read oid 30 snap 59 2024-09-11T03:40:41.874 INFO:tasks.rados.rados.2.smithi175.stdout:667: expect deleted 2024-09-11T03:40:41.874 INFO:tasks.rados.rados.2.smithi175.stdout:668: snap_remove snap 61 2024-09-11T03:40:41.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:41.875 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:41.929 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:41.929 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:41.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:42.001 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:42.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:42.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:42.077 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:41 smithi084 ceph-mon[27254]: 1.1c scrub starts 2024-09-11T03:40:42.077 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:41 smithi084 ceph-mon[27254]: 1.1c scrub ok 2024-09-11T03:40:42.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:41 smithi084 ceph-mon[27254]: 1.1e scrub starts 2024-09-11T03:40:42.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:41 smithi084 ceph-mon[27254]: 1.1e scrub ok 2024-09-11T03:40:42.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:41 smithi084 ceph-mon[27254]: 1.1a scrub starts 2024-09-11T03:40:42.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:41 smithi084 ceph-mon[27254]: 1.1a scrub ok 2024-09-11T03:40:42.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:41 smithi084 ceph-mon[27254]: 1.16 scrub starts 2024-09-11T03:40:42.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:41 smithi084 ceph-mon[27254]: 1.16 scrub ok 2024-09-11T03:40:42.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:41 smithi084 ceph-mon[27254]: 1.18 scrub starts 2024-09-11T03:40:42.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:41 smithi084 ceph-mon[27254]: 1.18 scrub ok 2024-09-11T03:40:42.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:41 smithi084 ceph-mon[27254]: 1.19 scrub starts 2024-09-11T03:40:42.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:41 smithi084 ceph-mon[27254]: 1.19 scrub ok 2024-09-11T03:40:42.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:41 smithi084 ceph-mon[27254]: pgmap v490: 47 pgs: 1 peering, 5 unknown, 41 active+clean; 165 MiB data, 640 MiB used, 893 GiB / 894 GiB avail; 5.0 MiB/s rd, 2 op/s 2024-09-11T03:40:42.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:41 smithi084 ceph-mon[27254]: osdmap e225: 12 total, 12 up, 10 in 2024-09-11T03:40:42.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:42.126 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:42.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:42.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:41 smithi121 ceph-mon[30729]: 1.1c scrub starts 2024-09-11T03:40:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:41 smithi121 ceph-mon[30729]: 1.1c scrub ok 2024-09-11T03:40:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:41 smithi121 ceph-mon[30729]: 1.1e scrub starts 2024-09-11T03:40:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:41 smithi121 ceph-mon[30729]: 1.1e scrub ok 2024-09-11T03:40:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:41 smithi121 ceph-mon[30729]: 1.1a scrub starts 2024-09-11T03:40:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:41 smithi121 ceph-mon[30729]: 1.1a scrub ok 2024-09-11T03:40:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:41 smithi121 ceph-mon[30729]: 1.16 scrub starts 2024-09-11T03:40:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:41 smithi121 ceph-mon[30729]: 1.16 scrub ok 2024-09-11T03:40:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:41 smithi121 ceph-mon[30729]: 1.18 scrub starts 2024-09-11T03:40:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:41 smithi121 ceph-mon[30729]: 1.18 scrub ok 2024-09-11T03:40:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:41 smithi121 ceph-mon[30729]: 1.19 scrub starts 2024-09-11T03:40:42.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:41 smithi121 ceph-mon[30729]: 1.19 scrub ok 2024-09-11T03:40:42.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:41 smithi121 ceph-mon[30729]: pgmap v490: 47 pgs: 1 peering, 5 unknown, 41 active+clean; 165 MiB data, 640 MiB used, 893 GiB / 894 GiB avail; 5.0 MiB/s rd, 2 op/s 2024-09-11T03:40:42.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:41 smithi121 ceph-mon[30729]: osdmap e225: 12 total, 12 up, 10 in 2024-09-11T03:40:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:41 smithi175 ceph-mon[30894]: 1.1c scrub starts 2024-09-11T03:40:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:41 smithi175 ceph-mon[30894]: 1.1c scrub ok 2024-09-11T03:40:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:41 smithi175 ceph-mon[30894]: 1.1e scrub starts 2024-09-11T03:40:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:41 smithi175 ceph-mon[30894]: 1.1e scrub ok 2024-09-11T03:40:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:41 smithi175 ceph-mon[30894]: 1.1a scrub starts 2024-09-11T03:40:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:41 smithi175 ceph-mon[30894]: 1.1a scrub ok 2024-09-11T03:40:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:41 smithi175 ceph-mon[30894]: 1.16 scrub starts 2024-09-11T03:40:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:41 smithi175 ceph-mon[30894]: 1.16 scrub ok 2024-09-11T03:40:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:41 smithi175 ceph-mon[30894]: 1.18 scrub starts 2024-09-11T03:40:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:41 smithi175 ceph-mon[30894]: 1.18 scrub ok 2024-09-11T03:40:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:41 smithi175 ceph-mon[30894]: 1.19 scrub starts 2024-09-11T03:40:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:41 smithi175 ceph-mon[30894]: 1.19 scrub ok 2024-09-11T03:40:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:41 smithi175 ceph-mon[30894]: pgmap v490: 47 pgs: 1 peering, 5 unknown, 41 active+clean; 165 MiB data, 640 MiB used, 893 GiB / 894 GiB avail; 5.0 MiB/s rd, 2 op/s 2024-09-11T03:40:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:41 smithi175 ceph-mon[30894]: osdmap e225: 12 total, 12 up, 10 in 2024-09-11T03:40:42.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:42.251 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:42.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:42.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:42.328 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:40:42.328 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:40:42.346 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:42.347 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:40:42.347 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:40:42.352 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:42.363 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:40:42.364 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:40:42.375 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:42.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:42.381 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:40:42.381 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:40:42.398 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:40:42.398 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:40:42.414 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:40:42.415 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:40:42.432 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:40:42.432 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:40:42.449 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:40:42.449 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:40:42.466 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:40:42.466 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:40:42.477 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:42.484 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:40:42.484 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:40:42.498 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:42.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:42.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:42.501 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:40:42.501 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:40:42.517 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:40:42.518 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:40:42.555 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-11T03:40:42.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:42.624 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:42.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:42.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:42.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:42.751 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:42.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:42.796 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:40:42.846 INFO:tasks.rados.rados.2.smithi175.stdout:668: done (6 left) 2024-09-11T03:40:42.846 INFO:tasks.rados.rados.2.smithi175.stdout:669: snap_create 2024-09-11T03:40:42.846 INFO:tasks.rados.rados.2.smithi175.stdout:662: expect (ObjNum 246 snap 60 seq_num 246) 2024-09-11T03:40:42.852 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:42.874 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:42.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:42.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:42.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:42.987 INFO:tasks.rados.rados.2.smithi175.stdout:661: expect (ObjNum 239 snap 58 seq_num 239) 2024-09-11T03:40:43.002 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:43.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:43.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:43.104 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:43.133 INFO:tasks.rados.rados.2.smithi175.stdout:660: finishing copy_from racing read to smithi17557870-49 2024-09-11T03:40:43.133 INFO:tasks.rados.rados.2.smithi175.stdout:665: finishing copy_from racing read to smithi17557870-5 2024-09-11T03:40:43.133 INFO:tasks.rados.rados.2.smithi175.stdout:666: finishing write tid 1 to smithi17557870-8 2024-09-11T03:40:43.133 INFO:tasks.rados.rados.2.smithi175.stdout:665: finishing copy_from to smithi17557870-5 2024-09-11T03:40:43.133 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 214 (ObjNum 101 snap 16 seq_num 101) dirty exists 2024-09-11T03:40:43.133 INFO:tasks.rados.rados.2.smithi175.stdout:666: finishing write tid 2 to smithi17557870-8 2024-09-11T03:40:43.133 INFO:tasks.rados.rados.2.smithi175.stdout:666: finishing write tid 3 to smithi17557870-8 2024-09-11T03:40:43.134 INFO:tasks.rados.rados.2.smithi175.stdout:660: finishing copy_from to smithi17557870-49 2024-09-11T03:40:43.134 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 82 (ObjNum 171 snap 39 seq_num 171) dirty exists 2024-09-11T03:40:43.134 INFO:tasks.rados.rados.2.smithi175.stdout:666: finishing write tid 4 to smithi17557870-8 2024-09-11T03:40:43.134 INFO:tasks.rados.rados.2.smithi175.stdout:666: oid 8 updating version 0 to 194 2024-09-11T03:40:43.134 INFO:tasks.rados.rados.2.smithi175.stdout:666: oid 8 updating version 194 to 195 2024-09-11T03:40:43.134 INFO:tasks.rados.rados.2.smithi175.stdout:666: oid 8 version 195 is already newer than 193 2024-09-11T03:40:43.134 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 195 (ObjNum 265 snap 66 seq_num 265) dirty exists 2024-09-11T03:40:43.134 INFO:tasks.rados.rados.2.smithi175.stdout:666: left oid 8 (ObjNum 265 snap 66 seq_num 265) 2024-09-11T03:40:43.136 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:43.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:43.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:43.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:42 smithi084 ceph-mon[27254]: 1.17 scrub starts 2024-09-11T03:40:43.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:42 smithi084 ceph-mon[27254]: 1.17 scrub ok 2024-09-11T03:40:43.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:42 smithi084 ceph-mon[27254]: 1.1d scrub starts 2024-09-11T03:40:43.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:42 smithi084 ceph-mon[27254]: 1.1d scrub ok 2024-09-11T03:40:43.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:42 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1202845557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:43.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:42 smithi084 ceph-mon[27254]: osdmap e226: 12 total, 12 up, 10 in 2024-09-11T03:40:43.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:42 smithi121 ceph-mon[30729]: 1.17 scrub starts 2024-09-11T03:40:43.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:42 smithi121 ceph-mon[30729]: 1.17 scrub ok 2024-09-11T03:40:43.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:42 smithi121 ceph-mon[30729]: 1.1d scrub starts 2024-09-11T03:40:43.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:42 smithi121 ceph-mon[30729]: 1.1d scrub ok 2024-09-11T03:40:43.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:42 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1202845557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:43.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:42 smithi121 ceph-mon[30729]: osdmap e226: 12 total, 12 up, 10 in 2024-09-11T03:40:43.205 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:42 smithi175 ceph-mon[30894]: 1.17 scrub starts 2024-09-11T03:40:43.205 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:42 smithi175 ceph-mon[30894]: 1.17 scrub ok 2024-09-11T03:40:43.205 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:42 smithi175 ceph-mon[30894]: 1.1d scrub starts 2024-09-11T03:40:43.205 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:42 smithi175 ceph-mon[30894]: 1.1d scrub ok 2024-09-11T03:40:43.205 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:42 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1202845557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:43.205 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:42 smithi175 ceph-mon[30894]: osdmap e226: 12 total, 12 up, 10 in 2024-09-11T03:40:43.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:43.264 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:43.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:43.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:43.306 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:40:43.306 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:40:43.306 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:40:43.306 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:40:43.306 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.306 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:243 2.d 2:b7bdfe52:::smithi17557870-10:head [write 507578~477917] snapc 5=[4] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T03:40:43.306 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:43.971811+0000", 2024-09-11T03:40:43.307 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 119.333653967, 2024-09-11T03:40:43.307 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.006812818, 2024-09-11T03:40:43.307 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.307 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.307 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:43.307 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:43.307 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:43.307 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:43.307 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 243 2024-09-11T03:40:43.307 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.307 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.307 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.307 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.308 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.971811+0000", 2024-09-11T03:40:43.308 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.308 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.308 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.308 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.308 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.971811+0000", 2024-09-11T03:40:43.308 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.308 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.308 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.308 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.308 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.971813+0000", 2024-09-11T03:40:43.308 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0829999999999998e-06 2024-09-11T03:40:43.308 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.309 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.309 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.309 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.972267+0000", 2024-09-11T03:40:43.309 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00045407599999999998 2024-09-11T03:40:43.309 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.309 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.309 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.309 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.972291+0000", 2024-09-11T03:40:43.309 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3218e-05 2024-09-11T03:40:43.309 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.310 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.310 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.972304+0000", 2024-09-11T03:40:43.310 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3397999999999999e-05 2024-09-11T03:40:43.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.310 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.310 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.972388+0000", 2024-09-11T03:40:43.310 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3891e-05 2024-09-11T03:40:43.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.310 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:40:43.310 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.972396+0000", 2024-09-11T03:40:43.311 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5700000000000004e-06 2024-09-11T03:40:43.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.311 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.311 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.974233+0000", 2024-09-11T03:40:43.311 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001837064 2024-09-11T03:40:43.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.311 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.311 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.974342+0000", 2024-09-11T03:40:43.311 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010918599999999999 2024-09-11T03:40:43.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.312 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:40:43.312 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.974547+0000", 2024-09-11T03:40:43.312 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00020537499999999999 2024-09-11T03:40:43.312 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.312 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.312 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:40:43.312 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.976167+0000", 2024-09-11T03:40:43.312 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001620039 2024-09-11T03:40:43.312 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.313 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.313 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:43.313 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.977310+0000", 2024-09-11T03:40:43.313 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001143301 2024-09-11T03:40:43.313 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.313 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.313 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:43.313 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.978507+0000", 2024-09-11T03:40:43.314 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0011963220000000001 2024-09-11T03:40:43.314 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.314 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.314 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:43.314 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.978570+0000", 2024-09-11T03:40:43.314 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3480999999999997e-05 2024-09-11T03:40:43.314 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.314 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.314 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.314 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:43.978624+0000", 2024-09-11T03:40:43.314 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3813999999999997e-05 2024-09-11T03:40:43.314 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.314 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.314 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.315 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.315 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.315 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:640 2.a e131/95 2:5014e305:::smithi17557870-46:head v 131'5, mlcod=100'2)", 2024-09-11T03:40:43.315 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:40:43.315 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 95.187046307000003, 2024-09-11T03:40:43.315 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068845190000000004, 2024-09-11T03:40:43.315 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.315 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.315 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:43.315 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.315 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.315 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.315 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:40:43.316 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.316 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.316 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:40:43.316 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.316 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.316 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:40:43.316 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7399999999999998e-07 2024-09-11T03:40:43.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.317 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.317 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.120287+0000", 2024-09-11T03:40:43.317 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0018676809999999999 2024-09-11T03:40:43.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.317 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.317 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.317 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.120324+0000", 2024-09-11T03:40:43.317 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7082000000000002e-05 2024-09-11T03:40:43.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.317 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.317 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.317 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.120345+0000", 2024-09-11T03:40:43.317 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0888000000000001e-05 2024-09-11T03:40:43.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.318 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.318 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.318 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.120456+0000", 2024-09-11T03:40:43.318 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000111328 2024-09-11T03:40:43.318 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.318 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.318 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.318 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.120478+0000", 2024-09-11T03:40:43.318 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1732000000000001e-05 2024-09-11T03:40:43.318 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.318 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.318 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:43.318 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125257+0000", 2024-09-11T03:40:43.319 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0047789479999999999 2024-09-11T03:40:43.319 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.319 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.319 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.319 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125303+0000", 2024-09-11T03:40:43.319 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6385999999999999e-05 2024-09-11T03:40:43.319 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.319 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.320 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.320 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.320 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.320 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:847 2.d 2:b1c0f730:::smithi17557870-20:head [copy-from ver 90] snapc 1a=[1a,19] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-11T03:40:43.320 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:40:43.320 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 78.466170136000002, 2024-09-11T03:40:43.320 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013054913, 2024-09-11T03:40:43.320 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.320 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.320 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:43.320 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:43.320 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:43.320 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:43.320 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 847 2024-09-11T03:40:43.321 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.321 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.321 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.321 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.321 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:40:43.321 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.321 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.321 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.321 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.321 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:40:43.321 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.321 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.321 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.321 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.322 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839297+0000", 2024-09-11T03:40:43.322 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0049999999999999e-06 2024-09-11T03:40:43.322 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.322 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.322 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.323 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839309+0000", 2024-09-11T03:40:43.323 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1531e-05 2024-09-11T03:40:43.323 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.323 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.323 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.323 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839326+0000", 2024-09-11T03:40:43.323 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7788e-05 2024-09-11T03:40:43.323 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.323 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.323 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.324 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839336+0000", 2024-09-11T03:40:43.324 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5619999999999993e-06 2024-09-11T03:40:43.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.324 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.324 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.324 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839423+0000", 2024-09-11T03:40:43.324 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6790000000000001e-05 2024-09-11T03:40:43.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.324 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.324 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:40:43.324 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839434+0000", 2024-09-11T03:40:43.324 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0971e-05 2024-09-11T03:40:43.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.324 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.325 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.325 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841520+0000", 2024-09-11T03:40:43.325 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0020866299999999999 2024-09-11T03:40:43.325 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.325 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.325 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.325 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841590+0000", 2024-09-11T03:40:43.325 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9288000000000005e-05 2024-09-11T03:40:43.325 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.325 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.325 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:40:43.325 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.846430+0000", 2024-09-11T03:40:43.325 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0048408349999999999 2024-09-11T03:40:43.325 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.326 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.326 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:40:43.326 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849726+0000", 2024-09-11T03:40:43.326 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032955390000000001 2024-09-11T03:40:43.326 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.326 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.326 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:43.326 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852240+0000", 2024-09-11T03:40:43.326 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002513492 2024-09-11T03:40:43.326 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.326 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.327 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:43.327 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852278+0000", 2024-09-11T03:40:43.327 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7988999999999998e-05 2024-09-11T03:40:43.327 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.327 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.327 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:43.327 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852317+0000", 2024-09-11T03:40:43.327 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9310000000000001e-05 2024-09-11T03:40:43.327 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.327 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.327 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.327 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852350+0000", 2024-09-11T03:40:43.327 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3182999999999997e-05 2024-09-11T03:40:43.327 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.328 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.328 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.328 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.328 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.328 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:848 2.d 2:b1c0f730:::smithi17557870-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2024-09-11T03:40:43.328 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:24.839354+0000", 2024-09-11T03:40:43.328 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 78.466110739000001, 2024-09-11T03:40:43.328 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010055476000000001, 2024-09-11T03:40:43.328 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.328 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.328 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:43.328 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:43.328 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:43.329 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:43.329 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 848 2024-09-11T03:40:43.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.329 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.329 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.329 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.329 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839354+0000", 2024-09-11T03:40:43.329 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.329 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.329 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.329 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839354+0000", 2024-09-11T03:40:43.329 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.330 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.330 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.330 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839355+0000", 2024-09-11T03:40:43.330 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6899999999999996e-07 2024-09-11T03:40:43.330 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.330 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.330 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.330 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839358+0000", 2024-09-11T03:40:43.330 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8729999999999999e-06 2024-09-11T03:40:43.330 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.330 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.330 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.330 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839367+0000", 2024-09-11T03:40:43.330 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3999999999999992e-06 2024-09-11T03:40:43.331 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.331 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.331 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.331 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839371+0000", 2024-09-11T03:40:43.331 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6059999999999996e-06 2024-09-11T03:40:43.331 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.331 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.331 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.331 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839509+0000", 2024-09-11T03:40:43.331 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013766700000000001 2024-09-11T03:40:43.331 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.331 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.331 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:40:43.331 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839515+0000", 2024-09-11T03:40:43.332 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4740000000000002e-06 2024-09-11T03:40:43.332 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.332 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.332 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.332 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841883+0000", 2024-09-11T03:40:43.332 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023678319999999998 2024-09-11T03:40:43.332 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.332 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.332 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:40:43.332 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841941+0000", 2024-09-11T03:40:43.332 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8241e-05 2024-09-11T03:40:43.332 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.332 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.332 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.333 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849208+0000", 2024-09-11T03:40:43.333 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0072661690000000003 2024-09-11T03:40:43.333 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.333 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.333 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.333 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849295+0000", 2024-09-11T03:40:43.333 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7582000000000003e-05 2024-09-11T03:40:43.333 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.333 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.333 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.333 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849410+0000", 2024-09-11T03:40:43.334 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114863 2024-09-11T03:40:43.334 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.334 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.334 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.334 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.334 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.334 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:907 2.d 2:b6a25a79:::smithi17557870-47:head [copy-from ver 47] snapc 20=[20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-11T03:40:43.334 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:34.902174+0000", 2024-09-11T03:40:43.334 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 68.403291680999999, 2024-09-11T03:40:43.334 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011234144, 2024-09-11T03:40:43.334 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.334 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.334 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:43.334 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:43.335 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:43.335 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:43.335 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 907 2024-09-11T03:40:43.335 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.335 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.335 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.335 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.335 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902174+0000", 2024-09-11T03:40:43.335 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.335 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.335 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.335 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.335 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902174+0000", 2024-09-11T03:40:43.336 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.336 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.336 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.336 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902175+0000", 2024-09-11T03:40:43.336 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0189999999999999e-06 2024-09-11T03:40:43.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.336 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.336 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.336 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902178+0000", 2024-09-11T03:40:43.336 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4410000000000002e-06 2024-09-11T03:40:43.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.337 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.337 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.337 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902188+0000", 2024-09-11T03:40:43.337 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0023999999999999e-05 2024-09-11T03:40:43.337 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.337 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.337 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.337 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902193+0000", 2024-09-11T03:40:43.337 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.31e-06 2024-09-11T03:40:43.337 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.338 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.338 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.338 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902248+0000", 2024-09-11T03:40:43.338 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4562999999999999e-05 2024-09-11T03:40:43.338 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.338 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.338 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.338 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902344+0000", 2024-09-11T03:40:43.338 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5841999999999995e-05 2024-09-11T03:40:43.338 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.339 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.339 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:40:43.339 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.907645+0000", 2024-09-11T03:40:43.339 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0053015579999999996 2024-09-11T03:40:43.339 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.339 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.339 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:40:43.339 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.909768+0000", 2024-09-11T03:40:43.339 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0021225020000000001 2024-09-11T03:40:43.339 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.340 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.340 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:43.340 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.912827+0000", 2024-09-11T03:40:43.340 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003059375 2024-09-11T03:40:43.340 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.340 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.340 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:43.340 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.913334+0000", 2024-09-11T03:40:43.340 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00050710499999999999 2024-09-11T03:40:43.340 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.340 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.341 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:43.341 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.913372+0000", 2024-09-11T03:40:43.341 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7486e-05 2024-09-11T03:40:43.341 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.341 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.341 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.341 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.913408+0000", 2024-09-11T03:40:43.341 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5918999999999999e-05 2024-09-11T03:40:43.341 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.341 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.341 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.341 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.341 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.341 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:908 2.d 2:b6a25a79:::smithi17557870-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e158)", 2024-09-11T03:40:43.342 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:34.902203+0000", 2024-09-11T03:40:43.342 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 68.403261846999996, 2024-09-11T03:40:43.342 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0073357650000000002, 2024-09-11T03:40:43.342 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.342 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.342 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:43.342 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:43.342 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:43.342 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:43.342 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 908 2024-09-11T03:40:43.342 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.342 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.342 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.342 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.343 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902203+0000", 2024-09-11T03:40:43.343 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.343 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.343 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.343 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902203+0000", 2024-09-11T03:40:43.343 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.343 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.343 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.343 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902204+0000", 2024-09-11T03:40:43.343 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4400000000000001e-07 2024-09-11T03:40:43.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.344 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.344 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.344 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902205+0000", 2024-09-11T03:40:43.344 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.266e-06 2024-09-11T03:40:43.344 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.344 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.344 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.344 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902210+0000", 2024-09-11T03:40:43.344 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9169999999999997e-06 2024-09-11T03:40:43.345 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.345 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.345 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.345 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902213+0000", 2024-09-11T03:40:43.345 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1099999999999999e-06 2024-09-11T03:40:43.345 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.345 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.345 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.345 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902498+0000", 2024-09-11T03:40:43.345 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00028460399999999999 2024-09-11T03:40:43.345 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.345 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.345 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:40:43.346 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902548+0000", 2024-09-11T03:40:43.346 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0503000000000001e-05 2024-09-11T03:40:43.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.346 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.346 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.346 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.909356+0000", 2024-09-11T03:40:43.346 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068078239999999996 2024-09-11T03:40:43.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.346 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.346 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.346 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.909444+0000", 2024-09-11T03:40:43.346 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8362999999999994e-05 2024-09-11T03:40:43.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.347 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.347 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.347 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.909539+0000", 2024-09-11T03:40:43.347 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4833999999999994e-05 2024-09-11T03:40:43.347 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.347 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.347 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.347 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.347 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.347 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1157 2.d 2:b6874adf:::smithi17557870-17:head [read 0~2453822,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:40:43.347 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:01.831703+0000", 2024-09-11T03:40:43.347 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 41.47376242, 2024-09-11T03:40:43.347 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0063272390000000001, 2024-09-11T03:40:43.347 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.348 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.348 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:43.348 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:43.348 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:43.348 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:43.348 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1157 2024-09-11T03:40:43.348 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.348 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.348 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.348 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.348 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.831703+0000", 2024-09-11T03:40:43.348 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.348 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.349 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.349 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.349 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.831703+0000", 2024-09-11T03:40:43.349 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.349 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.349 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.349 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.349 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.831703+0000", 2024-09-11T03:40:43.349 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0699999999999998e-07 2024-09-11T03:40:43.350 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.350 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.350 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.350 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.831706+0000", 2024-09-11T03:40:43.350 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.044e-06 2024-09-11T03:40:43.350 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.350 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.350 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.350 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.831712+0000", 2024-09-11T03:40:43.350 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1319999999999999e-06 2024-09-11T03:40:43.350 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.351 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.351 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.351 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.831716+0000", 2024-09-11T03:40:43.351 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.219e-06 2024-09-11T03:40:43.351 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.351 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.351 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.351 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.831829+0000", 2024-09-11T03:40:43.351 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000113459 2024-09-11T03:40:43.351 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.351 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.351 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:40:43.351 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.831837+0000", 2024-09-11T03:40:43.351 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8900000000000007e-06 2024-09-11T03:40:43.352 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.352 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.352 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.352 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.836112+0000", 2024-09-11T03:40:43.352 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0042748600000000001 2024-09-11T03:40:43.352 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.352 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.352 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.352 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.836239+0000", 2024-09-11T03:40:43.352 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012686699999999999 2024-09-11T03:40:43.352 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.352 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.352 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.352 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.838030+0000", 2024-09-11T03:40:43.353 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001791361 2024-09-11T03:40:43.353 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.353 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.353 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.353 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.353 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.353 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1158 2.d 2:b6874adf:::smithi17557870-17:head [sparse-read 0~1580355] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:40:43.353 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:01.831743+0000", 2024-09-11T03:40:43.353 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 41.473722441, 2024-09-11T03:40:43.353 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.006506909, 2024-09-11T03:40:43.353 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.353 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.354 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:43.354 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:43.354 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:43.354 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:43.354 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1158 2024-09-11T03:40:43.354 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.354 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.354 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.355 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.355 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.831743+0000", 2024-09-11T03:40:43.355 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.355 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.355 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.355 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.355 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.831743+0000", 2024-09-11T03:40:43.355 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.355 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.355 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.355 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.355 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.831743+0000", 2024-09-11T03:40:43.355 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9700000000000002e-07 2024-09-11T03:40:43.355 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.356 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.356 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.356 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.831748+0000", 2024-09-11T03:40:43.356 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9949999999999996e-06 2024-09-11T03:40:43.356 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.356 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.356 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.356 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.831754+0000", 2024-09-11T03:40:43.356 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8629999999999996e-06 2024-09-11T03:40:43.356 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.357 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.357 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.357 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.831758+0000", 2024-09-11T03:40:43.357 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.681e-06 2024-09-11T03:40:43.357 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.357 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.357 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.357 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.831860+0000", 2024-09-11T03:40:43.357 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000102525 2024-09-11T03:40:43.357 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.357 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.357 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:40:43.358 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.831864+0000", 2024-09-11T03:40:43.358 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.253e-06 2024-09-11T03:40:43.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.358 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.358 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.838067+0000", 2024-09-11T03:40:43.358 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0062032520000000002 2024-09-11T03:40:43.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.358 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.358 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.838151+0000", 2024-09-11T03:40:43.358 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4541000000000003e-05 2024-09-11T03:40:43.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.359 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.359 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.838250+0000", 2024-09-11T03:40:43.359 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8401999999999997e-05 2024-09-11T03:40:43.359 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.359 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.359 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.359 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.359 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.359 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:1159 2.e e185/95 2:750de3b7:::smithi17557870-8:head v 185'112, mlcod=180'109)", 2024-09-11T03:40:43.359 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:01.841135+0000", 2024-09-11T03:40:43.359 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 41.464330599999997, 2024-09-11T03:40:43.359 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0066451460000000002, 2024-09-11T03:40:43.359 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.359 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.360 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:43.360 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.360 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.360 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.360 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.841135+0000", 2024-09-11T03:40:43.360 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.360 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.360 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.360 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.360 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.841135+0000", 2024-09-11T03:40:43.361 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.361 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.361 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.361 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.361 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.841135+0000", 2024-09-11T03:40:43.361 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0199999999999998e-07 2024-09-11T03:40:43.361 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.361 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.361 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.361 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.845671+0000", 2024-09-11T03:40:43.361 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0045362650000000003 2024-09-11T03:40:43.361 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.362 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.362 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.362 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.845696+0000", 2024-09-11T03:40:43.362 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5054000000000001e-05 2024-09-11T03:40:43.362 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.362 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.362 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.362 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.845706+0000", 2024-09-11T03:40:43.362 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8509999999999994e-06 2024-09-11T03:40:43.362 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.362 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.362 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.363 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.845814+0000", 2024-09-11T03:40:43.363 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010741199999999999 2024-09-11T03:40:43.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.363 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.363 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.363 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.845835+0000", 2024-09-11T03:40:43.363 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1401000000000001e-05 2024-09-11T03:40:43.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.363 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.363 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:43.363 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.847730+0000", 2024-09-11T03:40:43.363 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0018953590000000001 2024-09-11T03:40:43.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.364 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.364 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.364 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.847780+0000", 2024-09-11T03:40:43.364 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9502000000000001e-05 2024-09-11T03:40:43.364 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.364 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.364 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.364 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.364 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.364 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1290 2.d 2:b1c0f730:::smithi17557870-20:head [copy-from ver 48] snapc 33=[32,30,2c,2b] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:40:43.365 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:12.836022+0000", 2024-09-11T03:40:43.365 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 30.469442855, 2024-09-11T03:40:43.365 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022892863999999999, 2024-09-11T03:40:43.365 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.365 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.365 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:43.365 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:43.365 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:43.365 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:43.365 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1290 2024-09-11T03:40:43.365 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.365 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.365 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.366 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.366 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836022+0000", 2024-09-11T03:40:43.366 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.366 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.366 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.366 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.366 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836022+0000", 2024-09-11T03:40:43.366 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.366 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.366 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.366 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.366 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836025+0000", 2024-09-11T03:40:43.366 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6690000000000002e-06 2024-09-11T03:40:43.366 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.367 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.367 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.367 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836033+0000", 2024-09-11T03:40:43.367 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0809999999999994e-06 2024-09-11T03:40:43.367 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.367 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.367 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.367 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836053+0000", 2024-09-11T03:40:43.367 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9429999999999999e-05 2024-09-11T03:40:43.368 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.368 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.368 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.368 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836063+0000", 2024-09-11T03:40:43.368 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0438000000000001e-05 2024-09-11T03:40:43.368 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.368 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.368 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.368 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836165+0000", 2024-09-11T03:40:43.368 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000101523 2024-09-11T03:40:43.368 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.368 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.369 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:40:43.369 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836172+0000", 2024-09-11T03:40:43.369 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6499999999999996e-06 2024-09-11T03:40:43.369 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.369 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.369 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.369 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839394+0000", 2024-09-11T03:40:43.369 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032215590000000001 2024-09-11T03:40:43.369 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.369 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.369 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.369 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839507+0000", 2024-09-11T03:40:43.369 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000113557 2024-09-11T03:40:43.369 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.369 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.370 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:40:43.370 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.849905+0000", 2024-09-11T03:40:43.370 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010398174 2024-09-11T03:40:43.370 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.370 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.370 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:40:43.370 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.853428+0000", 2024-09-11T03:40:43.370 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035222869999999998 2024-09-11T03:40:43.370 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.370 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.370 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:43.370 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.855923+0000", 2024-09-11T03:40:43.370 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024951840000000001 2024-09-11T03:40:43.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.371 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.371 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:43.371 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.858781+0000", 2024-09-11T03:40:43.371 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002858368 2024-09-11T03:40:43.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.371 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.371 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:43.371 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.858881+0000", 2024-09-11T03:40:43.371 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9500999999999995e-05 2024-09-11T03:40:43.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.371 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.371 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.372 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.858915+0000", 2024-09-11T03:40:43.372 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4443000000000002e-05 2024-09-11T03:40:43.372 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.372 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.372 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.372 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.372 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.372 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1291 2.d 2:b1c0f730:::smithi17557870-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:40:43.373 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:12.836223+0000", 2024-09-11T03:40:43.373 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 30.469242109, 2024-09-11T03:40:43.373 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016933050000000002, 2024-09-11T03:40:43.373 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.373 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.373 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:43.374 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:43.374 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:43.374 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:43.374 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1291 2024-09-11T03:40:43.374 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.374 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.374 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.375 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.375 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836223+0000", 2024-09-11T03:40:43.375 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.375 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.375 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.375 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.376 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836223+0000", 2024-09-11T03:40:43.376 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.376 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.376 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.376 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.376 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836224+0000", 2024-09-11T03:40:43.377 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0020000000000001e-06 2024-09-11T03:40:43.377 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.377 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.377 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.377 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836229+0000", 2024-09-11T03:40:43.377 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7180000000000004e-06 2024-09-11T03:40:43.378 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.378 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.378 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.378 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836245+0000", 2024-09-11T03:40:43.378 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5981000000000002e-05 2024-09-11T03:40:43.378 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.379 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.379 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.379 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836253+0000", 2024-09-11T03:40:43.379 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1720000000000007e-06 2024-09-11T03:40:43.379 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.379 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.380 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.380 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836317+0000", 2024-09-11T03:40:43.380 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4189999999999994e-05 2024-09-11T03:40:43.380 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.380 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.380 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:40:43.381 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836324+0000", 2024-09-11T03:40:43.381 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9249999999999998e-06 2024-09-11T03:40:43.381 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.381 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.381 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.381 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839738+0000", 2024-09-11T03:40:43.382 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0034135580000000001 2024-09-11T03:40:43.382 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.382 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.382 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:40:43.382 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839804+0000", 2024-09-11T03:40:43.382 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5974000000000002e-05 2024-09-11T03:40:43.383 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.383 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.383 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.383 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.852933+0000", 2024-09-11T03:40:43.383 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013129429999999999 2024-09-11T03:40:43.383 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.384 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.384 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.384 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.853032+0000", 2024-09-11T03:40:43.384 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8592999999999996e-05 2024-09-11T03:40:43.384 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.385 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.385 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.385 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.853156+0000", 2024-09-11T03:40:43.385 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012450699999999999 2024-09-11T03:40:43.385 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.385 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.386 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.386 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.386 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.386 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:1434 2.e e204/95 2:7efbbea8:::smithi17557870-24:head v 204'148, mlcod=203'145)", 2024-09-11T03:40:43.386 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:20.869726+0000", 2024-09-11T03:40:43.386 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 22.435739103, 2024-09-11T03:40:43.387 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0058169429999999998, 2024-09-11T03:40:43.387 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.387 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.387 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:43.387 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.387 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.387 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.388 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:20.869726+0000", 2024-09-11T03:40:43.388 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.388 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.388 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.388 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.388 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:20.869726+0000", 2024-09-11T03:40:43.388 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.389 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.389 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.389 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.389 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:20.869726+0000", 2024-09-11T03:40:43.389 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8900000000000001e-07 2024-09-11T03:40:43.390 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.390 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.390 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.390 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:20.873063+0000", 2024-09-11T03:40:43.390 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0033368600000000001 2024-09-11T03:40:43.390 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.390 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.390 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.390 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:20.873076+0000", 2024-09-11T03:40:43.390 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2367e-05 2024-09-11T03:40:43.390 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.391 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.391 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.391 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:20.873084+0000", 2024-09-11T03:40:43.391 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3149999999999992e-06 2024-09-11T03:40:43.391 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.391 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.391 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.391 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:20.873166+0000", 2024-09-11T03:40:43.391 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2539000000000004e-05 2024-09-11T03:40:43.391 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.391 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.391 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.391 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:20.873183+0000", 2024-09-11T03:40:43.392 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6886000000000001e-05 2024-09-11T03:40:43.392 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.392 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.392 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:43.392 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:20.875507+0000", 2024-09-11T03:40:43.392 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023239369999999999 2024-09-11T03:40:43.392 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.392 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.392 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.392 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:20.875543+0000", 2024-09-11T03:40:43.392 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5849999999999997e-05 2024-09-11T03:40:43.393 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.393 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.393 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.393 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.393 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.393 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1439 2.d 2:b6a25a79:::smithi17557870-47:35 [sparse-read 0~1600344,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e206)", 2024-09-11T03:40:43.393 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:22.839739+0000", 2024-09-11T03:40:43.393 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 20.465726473, 2024-09-11T03:40:43.393 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0071734040000000004, 2024-09-11T03:40:43.393 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.393 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.393 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:43.394 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:43.394 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:43.394 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:43.394 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1439 2024-09-11T03:40:43.394 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.394 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.394 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.394 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.394 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:43.396 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839739+0000", 2024-09-11T03:40:43.396 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.396 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.396 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.396 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.396 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839739+0000", 2024-09-11T03:40:43.396 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.396 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.397 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.397 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.397 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839741+0000", 2024-09-11T03:40:43.397 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9269999999999999e-06 2024-09-11T03:40:43.397 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.397 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.397 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.397 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839748+0000", 2024-09-11T03:40:43.397 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1330000000000001e-06 2024-09-11T03:40:43.397 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.397 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.397 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.397 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839811+0000", 2024-09-11T03:40:43.398 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3208999999999997e-05 2024-09-11T03:40:43.398 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.398 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.398 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.398 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839818+0000", 2024-09-11T03:40:43.398 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2729999999999997e-06 2024-09-11T03:40:43.398 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.398 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.398 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.398 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.840620+0000", 2024-09-11T03:40:43.398 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00080214800000000003 2024-09-11T03:40:43.398 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.398 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.398 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:40:43.399 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.840719+0000", 2024-09-11T03:40:43.399 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8240999999999997e-05 2024-09-11T03:40:43.399 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.399 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.399 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.399 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.845605+0000", 2024-09-11T03:40:43.399 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0048859810000000002 2024-09-11T03:40:43.399 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.399 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.399 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.399 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.845669+0000", 2024-09-11T03:40:43.399 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4109000000000006e-05 2024-09-11T03:40:43.399 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.400 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.400 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.400 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.846912+0000", 2024-09-11T03:40:43.400 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001243383 2024-09-11T03:40:43.400 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.400 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.400 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.400 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.400 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.400 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1440 2.d 2:b6a25a79:::smithi17557870-47:35 [read 0~3042067,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e206)", 2024-09-11T03:40:43.400 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:22.839838+0000", 2024-09-11T03:40:43.400 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 20.465626914000001, 2024-09-11T03:40:43.400 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0089542019999999996, 2024-09-11T03:40:43.401 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.401 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.401 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:43.401 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:43.401 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:43.401 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:43.401 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1440 2024-09-11T03:40:43.401 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.401 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.401 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.402 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.402 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839838+0000", 2024-09-11T03:40:43.402 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.402 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.402 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.402 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.402 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839838+0000", 2024-09-11T03:40:43.402 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.402 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.402 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.402 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.402 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839839+0000", 2024-09-11T03:40:43.403 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9500000000000002e-07 2024-09-11T03:40:43.403 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.403 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.403 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.403 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839843+0000", 2024-09-11T03:40:43.403 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3200000000000001e-06 2024-09-11T03:40:43.403 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.403 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.403 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.403 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839851+0000", 2024-09-11T03:40:43.403 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5170000000000002e-06 2024-09-11T03:40:43.403 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.403 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.404 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.404 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839856+0000", 2024-09-11T03:40:43.404 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5799999999999999e-06 2024-09-11T03:40:43.404 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.404 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.404 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.404 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.840764+0000", 2024-09-11T03:40:43.404 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00090780899999999998 2024-09-11T03:40:43.404 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.404 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.404 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:40:43.404 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.840857+0000", 2024-09-11T03:40:43.404 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2707999999999994e-05 2024-09-11T03:40:43.404 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.405 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.405 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.405 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.847171+0000", 2024-09-11T03:40:43.405 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0063140920000000003 2024-09-11T03:40:43.405 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.405 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.405 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.405 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.847241+0000", 2024-09-11T03:40:43.405 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0392e-05 2024-09-11T03:40:43.405 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.405 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.405 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.405 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.848793+0000", 2024-09-11T03:40:43.406 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015510890000000001 2024-09-11T03:40:43.406 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.406 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.406 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.406 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.406 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.406 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1441 2.d 2:b6a25a79:::smithi17557870-47:35 [sparse-read 0~1600344] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e206)", 2024-09-11T03:40:43.406 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:22.839865+0000", 2024-09-11T03:40:43.406 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 20.46559989, 2024-09-11T03:40:43.406 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0091540579999999996, 2024-09-11T03:40:43.407 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.407 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.407 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:43.407 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:43.407 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:43.407 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:43.407 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1441 2024-09-11T03:40:43.407 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.407 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.407 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.407 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.407 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839865+0000", 2024-09-11T03:40:43.407 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.408 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.408 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.408 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.408 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839865+0000", 2024-09-11T03:40:43.408 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.408 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.408 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.408 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.408 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839866+0000", 2024-09-11T03:40:43.408 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4499999999999997e-07 2024-09-11T03:40:43.408 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.408 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.408 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.409 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839869+0000", 2024-09-11T03:40:43.409 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.055e-06 2024-09-11T03:40:43.409 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.409 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.409 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.409 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839875+0000", 2024-09-11T03:40:43.409 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3319999999999997e-06 2024-09-11T03:40:43.409 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.409 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.410 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.410 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839880+0000", 2024-09-11T03:40:43.410 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0340000000000004e-06 2024-09-11T03:40:43.410 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.410 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.410 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.410 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.841228+0000", 2024-09-11T03:40:43.410 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001347521 2024-09-11T03:40:43.410 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.410 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.410 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:40:43.410 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.841306+0000", 2024-09-11T03:40:43.411 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8182000000000005e-05 2024-09-11T03:40:43.411 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.411 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.411 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.411 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.848858+0000", 2024-09-11T03:40:43.411 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0075521039999999996 2024-09-11T03:40:43.411 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.411 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.411 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.411 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.848919+0000", 2024-09-11T03:40:43.411 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.05e-05 2024-09-11T03:40:43.411 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.412 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.412 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.412 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.849019+0000", 2024-09-11T03:40:43.412 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000100785 2024-09-11T03:40:43.412 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.412 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.412 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.412 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.412 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.412 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1516 2.d 2:b6a25a79:::smithi17557870-47:head [copy-from ver 97] snapc 3b=[3a,38,37,35] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-09-11T03:40:43.412 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:28.919589+0000", 2024-09-11T03:40:43.412 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 14.385875979, 2024-09-11T03:40:43.413 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011956165, 2024-09-11T03:40:43.413 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.413 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.413 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:43.413 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:43.413 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:43.413 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:43.413 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1516 2024-09-11T03:40:43.413 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.414 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.414 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.414 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.414 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919589+0000", 2024-09-11T03:40:43.414 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.414 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.414 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.414 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.414 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919589+0000", 2024-09-11T03:40:43.414 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.414 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.414 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.415 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.415 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919591+0000", 2024-09-11T03:40:43.415 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5409999999999999e-06 2024-09-11T03:40:43.415 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.415 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.415 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.415 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919596+0000", 2024-09-11T03:40:43.415 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9280000000000001e-06 2024-09-11T03:40:43.415 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.415 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.415 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.415 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919610+0000", 2024-09-11T03:40:43.415 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4285e-05 2024-09-11T03:40:43.415 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.416 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.416 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.416 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919617+0000", 2024-09-11T03:40:43.416 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.435e-06 2024-09-11T03:40:43.416 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.416 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.416 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.416 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919688+0000", 2024-09-11T03:40:43.416 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0263000000000001e-05 2024-09-11T03:40:43.416 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.416 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.416 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.416 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919794+0000", 2024-09-11T03:40:43.417 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000106055 2024-09-11T03:40:43.417 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.417 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.417 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:40:43.417 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.922849+0000", 2024-09-11T03:40:43.417 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030557330000000001 2024-09-11T03:40:43.417 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.417 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.417 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:40:43.418 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.926556+0000", 2024-09-11T03:40:43.418 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0037066669999999999 2024-09-11T03:40:43.418 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.418 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.418 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:43.418 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.929197+0000", 2024-09-11T03:40:43.418 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002640529 2024-09-11T03:40:43.418 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.418 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.418 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:43.418 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.931432+0000", 2024-09-11T03:40:43.418 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022350339999999999 2024-09-11T03:40:43.419 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.419 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.419 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:43.419 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.931501+0000", 2024-09-11T03:40:43.419 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9584000000000003e-05 2024-09-11T03:40:43.419 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.419 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.419 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.419 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.931545+0000", 2024-09-11T03:40:43.419 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4110999999999998e-05 2024-09-11T03:40:43.419 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.419 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.420 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.420 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.420 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.420 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1517 2.d 2:b6a25a79:::smithi17557870-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2024-09-11T03:40:43.420 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:28.919645+0000", 2024-09-11T03:40:43.420 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 14.385820658, 2024-09-11T03:40:43.420 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0065414009999999996, 2024-09-11T03:40:43.420 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.420 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.420 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:43.421 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:43.421 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:43.421 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:43.421 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1517 2024-09-11T03:40:43.421 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.421 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.421 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.421 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.421 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919645+0000", 2024-09-11T03:40:43.421 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.421 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.421 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.421 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.422 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919645+0000", 2024-09-11T03:40:43.422 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.422 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.422 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.422 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.422 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919645+0000", 2024-09-11T03:40:43.422 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9299999999999998e-07 2024-09-11T03:40:43.422 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.422 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.422 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.422 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919648+0000", 2024-09-11T03:40:43.422 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8660000000000002e-06 2024-09-11T03:40:43.422 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.422 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.423 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.423 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919656+0000", 2024-09-11T03:40:43.423 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2900000000000002e-06 2024-09-11T03:40:43.423 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.423 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.423 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.423 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919662+0000", 2024-09-11T03:40:43.423 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7370000000000003e-06 2024-09-11T03:40:43.423 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.423 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.423 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.423 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919963+0000", 2024-09-11T03:40:43.423 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00030116499999999999 2024-09-11T03:40:43.423 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.424 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.424 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:40:43.424 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.920015+0000", 2024-09-11T03:40:43.424 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1940000000000001e-05 2024-09-11T03:40:43.424 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.424 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.424 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.424 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.925968+0000", 2024-09-11T03:40:43.424 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0059528740000000004 2024-09-11T03:40:43.424 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.424 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.424 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.424 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.926072+0000", 2024-09-11T03:40:43.425 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000103682 2024-09-11T03:40:43.425 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.425 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.425 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.425 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.926186+0000", 2024-09-11T03:40:43.425 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011425399999999999 2024-09-11T03:40:43.425 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.425 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.425 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.425 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.425 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.425 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1553 2.d 2:b7bdfe52:::smithi17557870-10:37 [read 0~2016807,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e214)", 2024-09-11T03:40:43.425 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:30.880323+0000", 2024-09-11T03:40:43.426 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 12.425142154, 2024-09-11T03:40:43.426 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0067779789999999999, 2024-09-11T03:40:43.426 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.426 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.426 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:43.426 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:43.426 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:43.426 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:43.426 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1553 2024-09-11T03:40:43.426 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.426 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.426 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.427 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.427 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880323+0000", 2024-09-11T03:40:43.427 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.427 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.427 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.427 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.427 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880323+0000", 2024-09-11T03:40:43.427 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.427 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.427 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.427 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.427 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880324+0000", 2024-09-11T03:40:43.428 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7400000000000003e-07 2024-09-11T03:40:43.428 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.428 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.428 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.428 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880327+0000", 2024-09-11T03:40:43.428 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5389999999999999e-06 2024-09-11T03:40:43.428 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.428 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.428 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.428 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880334+0000", 2024-09-11T03:40:43.429 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.029e-06 2024-09-11T03:40:43.429 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.429 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.429 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.429 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880341+0000", 2024-09-11T03:40:43.429 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3980000000000004e-06 2024-09-11T03:40:43.429 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.429 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.429 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.429 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880488+0000", 2024-09-11T03:40:43.429 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00014766700000000001 2024-09-11T03:40:43.429 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.429 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.430 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:40:43.430 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880495+0000", 2024-09-11T03:40:43.430 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7950000000000002e-06 2024-09-11T03:40:43.430 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.430 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.430 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.430 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.885670+0000", 2024-09-11T03:40:43.430 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0051744470000000004 2024-09-11T03:40:43.430 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.430 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.430 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.430 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.885724+0000", 2024-09-11T03:40:43.430 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4481999999999998e-05 2024-09-11T03:40:43.431 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.431 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.431 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.431 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.887101+0000", 2024-09-11T03:40:43.431 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001376948 2024-09-11T03:40:43.431 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.431 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.431 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.431 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.431 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.431 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1554 2.d 2:b7bdfe52:::smithi17557870-10:37 [read 0~2016807,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e214)", 2024-09-11T03:40:43.432 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:30.880351+0000", 2024-09-11T03:40:43.432 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 12.425114575, 2024-09-11T03:40:43.432 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0077014559999999998, 2024-09-11T03:40:43.432 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.432 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.432 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:43.432 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:43.432 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:43.432 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:43.432 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1554 2024-09-11T03:40:43.432 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.432 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.432 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.433 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.433 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880351+0000", 2024-09-11T03:40:43.433 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.433 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.433 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.433 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.433 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880351+0000", 2024-09-11T03:40:43.433 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.433 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.433 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.433 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.433 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880351+0000", 2024-09-11T03:40:43.434 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5299999999999999e-07 2024-09-11T03:40:43.434 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.434 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.434 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.434 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880357+0000", 2024-09-11T03:40:43.434 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4110000000000004e-06 2024-09-11T03:40:43.434 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.434 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.434 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.434 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880364+0000", 2024-09-11T03:40:43.434 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3549999999999999e-06 2024-09-11T03:40:43.434 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.434 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.434 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.435 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880370+0000", 2024-09-11T03:40:43.435 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6500000000000001e-06 2024-09-11T03:40:43.435 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.435 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.435 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.435 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880542+0000", 2024-09-11T03:40:43.435 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017261800000000001 2024-09-11T03:40:43.435 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.435 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.435 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:40:43.435 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880552+0000", 2024-09-11T03:40:43.435 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6229999999999992e-06 2024-09-11T03:40:43.436 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.436 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.436 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.436 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.887165+0000", 2024-09-11T03:40:43.436 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0066137189999999997 2024-09-11T03:40:43.436 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.436 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.436 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.436 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.887221+0000", 2024-09-11T03:40:43.436 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5826999999999999e-05 2024-09-11T03:40:43.436 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.437 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.437 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.437 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.888052+0000", 2024-09-11T03:40:43.437 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00083080000000000003 2024-09-11T03:40:43.437 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.437 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.437 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.437 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.437 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.437 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:1607 2.f e219/95 2:fa239c32:::smithi17557870-4:head v 219'119, mlcod=215'116)", 2024-09-11T03:40:43.438 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:35.909331+0000", 2024-09-11T03:40:43.438 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 7.3961342090000004, 2024-09-11T03:40:43.438 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0060340230000000003, 2024-09-11T03:40:43.438 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:43.438 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:43.438 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:43.438 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:43.438 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.438 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:43.438 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.909331+0000", 2024-09-11T03:40:43.438 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.438 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.439 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.439 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:43.439 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.909331+0000", 2024-09-11T03:40:43.439 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:43.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.439 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.439 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:43.439 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.909332+0000", 2024-09-11T03:40:43.439 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3900000000000004e-07 2024-09-11T03:40:43.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.439 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.439 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:43.439 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.911499+0000", 2024-09-11T03:40:43.440 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0021667700000000002 2024-09-11T03:40:43.440 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.440 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.440 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:43.440 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.911515+0000", 2024-09-11T03:40:43.440 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6640999999999999e-05 2024-09-11T03:40:43.440 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.441 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.441 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:43.441 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.911525+0000", 2024-09-11T03:40:43.441 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9979999999999998e-06 2024-09-11T03:40:43.441 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.441 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.441 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:43.441 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.911587+0000", 2024-09-11T03:40:43.441 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1599000000000004e-05 2024-09-11T03:40:43.441 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.442 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.442 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:43.442 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.911614+0000", 2024-09-11T03:40:43.442 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6891999999999999e-05 2024-09-11T03:40:43.442 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.442 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.442 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:43.442 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.915317+0000", 2024-09-11T03:40:43.442 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0037032160000000001 2024-09-11T03:40:43.442 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:43.442 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:43.442 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:43.442 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.915365+0000", 2024-09-11T03:40:43.443 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8068000000000001e-05 2024-09-11T03:40:43.443 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.443 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.443 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.443 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:43.443 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:43.443 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:40:43.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:43.465 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:43.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:43.472 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:40:43.554 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:40:43.555 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:40:43.566 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:43.588 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:43.589 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:43.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:43.716 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:43.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:43.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:43.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:43.839 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:43.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:43.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:43.848 INFO:tasks.rados.rados.2.smithi175.stdout:660: done (6 left) 2024-09-11T03:40:43.848 INFO:tasks.rados.rados.2.smithi175.stdout:661: done (5 left) 2024-09-11T03:40:43.849 INFO:tasks.rados.rados.2.smithi175.stdout:662: done (4 left) 2024-09-11T03:40:43.849 INFO:tasks.rados.rados.2.smithi175.stdout:665: done (3 left) 2024-09-11T03:40:43.849 INFO:tasks.rados.rados.2.smithi175.stdout:666: done (2 left) 2024-09-11T03:40:43.849 INFO:tasks.rados.rados.2.smithi175.stdout:667: done (1 left) 2024-09-11T03:40:43.849 INFO:tasks.rados.rados.2.smithi175.stdout:669: done (0 left) 2024-09-11T03:40:43.849 INFO:tasks.rados.rados.2.smithi175.stdout:670: write (excl) oid 21 current snap is 67 2024-09-11T03:40:43.849 INFO:tasks.rados.rados.2.smithi175.stdout:670: seq_num 266 ranges {470662=772236,1832916=408094} 2024-09-11T03:40:43.855 INFO:tasks.rados.rados.2.smithi175.stdout:670: writing smithi17557870-21 from 470662 to 1242898 tid 1 2024-09-11T03:40:43.860 INFO:tasks.rados.rados.2.smithi175.stdout:670: writing smithi17557870-21 from 1832916 to 2241010 tid 2 2024-09-11T03:40:43.860 INFO:tasks.rados.rados.2.smithi175.stdout:671: snap_create 2024-09-11T03:40:43.860 INFO:tasks.rados.rados.2.smithi175.stdout:670: finishing write tid 1 to smithi17557870-21 2024-09-11T03:40:43.863 INFO:tasks.rados.rados.2.smithi175.stdout:670: finishing write tid 2 to smithi17557870-21 2024-09-11T03:40:43.863 INFO:tasks.rados.rados.2.smithi175.stdout:670: finishing write tid 3 to smithi17557870-21 2024-09-11T03:40:43.863 INFO:tasks.rados.rados.2.smithi175.stdout:670: finishing write tid 4 to smithi17557870-21 2024-09-11T03:40:43.863 INFO:tasks.rados.rados.2.smithi175.stdout:670: oid 21 updating version 0 to 216 2024-09-11T03:40:43.863 INFO:tasks.rados.rados.2.smithi175.stdout:670: oid 21 updating version 216 to 217 2024-09-11T03:40:43.863 INFO:tasks.rados.rados.2.smithi175.stdout:670: oid 21 updating version 217 to 218 2024-09-11T03:40:43.863 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 218 (ObjNum 266 snap 67 seq_num 266) dirty exists 2024-09-11T03:40:43.863 INFO:tasks.rados.rados.2.smithi175.stdout:670: left oid 21 (ObjNum 266 snap 67 seq_num 266) 2024-09-11T03:40:43.941 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:43.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:43.964 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:43.964 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:44.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:44.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:44.088 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:44.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:44.101 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:43 smithi175 ceph-mon[30894]: 1.1b scrub starts 2024-09-11T03:40:44.101 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:43 smithi175 ceph-mon[30894]: 1.1b scrub ok 2024-09-11T03:40:44.101 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:43 smithi175 ceph-mon[30894]: pgmap v493: 47 pgs: 47 active+clean; 170 MiB data, 653 MiB used, 893 GiB / 894 GiB avail; 2.9 MiB/s rd, 1 op/s 2024-09-11T03:40:44.101 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:43 smithi175 ceph-mon[30894]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:40:44.101 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:43 smithi175 ceph-mon[30894]: osdmap e227: 12 total, 12 up, 10 in 2024-09-11T03:40:44.128 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-11T03:40:44.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:43 smithi121 ceph-mon[30729]: 1.1b scrub starts 2024-09-11T03:40:44.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:43 smithi121 ceph-mon[30729]: 1.1b scrub ok 2024-09-11T03:40:44.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:43 smithi121 ceph-mon[30729]: pgmap v493: 47 pgs: 47 active+clean; 170 MiB data, 653 MiB used, 893 GiB / 894 GiB avail; 2.9 MiB/s rd, 1 op/s 2024-09-11T03:40:44.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:43 smithi121 ceph-mon[30729]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:40:44.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:43 smithi121 ceph-mon[30729]: osdmap e227: 12 total, 12 up, 10 in 2024-09-11T03:40:44.189 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:44.210 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:44.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:44.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:44.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:43 smithi084 ceph-mon[27254]: 1.1b scrub starts 2024-09-11T03:40:44.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:43 smithi084 ceph-mon[27254]: 1.1b scrub ok 2024-09-11T03:40:44.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:43 smithi084 ceph-mon[27254]: pgmap v493: 47 pgs: 47 active+clean; 170 MiB data, 653 MiB used, 893 GiB / 894 GiB avail; 2.9 MiB/s rd, 1 op/s 2024-09-11T03:40:44.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:43 smithi084 ceph-mon[27254]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:40:44.217 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:43 smithi084 ceph-mon[27254]: osdmap e227: 12 total, 12 up, 10 in 2024-09-11T03:40:44.243 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-09-11T03:40:42.843448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":177826098,"num_objects":77,"num_object_clones":35,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":3,"num_read":1141,"num_read_kb":867582,"num_write":1044,"num_write_kb":427031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":13624269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4990,"ondisk_log_size":4990,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":669080,"kb_used_data":348820,"kb_used_omap":15,"kb_used_meta":319984,"kb_avail":936782440,"statfs":{"total":959950356480,"available":959265218560,"internally_reserved":0,"allocated":357191680,"data_stored":413132715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":327664053},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5036292,"num_objects":-2,"num_object_clones":0,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":-2,"num_read":12,"num_read_kb":11824,"num_write":8,"num_write_kb":8295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.998307"},"pg_stats":[{"pgid":"1.1e","version":"20'92","reported_seq":24,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-11T03:40:41.845052+0000","last_change":"2024-09-11T03:40:40.236609+0000","last_active":"2024-09-11T03:40:41.845052+0000","last_peered":"2024-09-11T03:40:41.845052+0000","last_clean":"2024-09-11T03:40:41.845052+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:40:41.845052+0000","last_undegraded":"2024-09-11T03:40:41.845052+0000","last_fullsized":"2024-09-11T03:40:41.845052+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.236546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.236546+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:03.367532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'92","reported_seq":24,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:41.348954+0000","last_change":"2024-09-11T03:40:41.348889+0000","last_active":"2024-09-11T03:40:41.348954+0000","last_peered":"2024-09-11T03:40:41.348954+0000","last_clean":"2024-09-11T03:40:41.348954+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:40:41.348954+0000","last_undegraded":"2024-09-11T03:40:41.348954+0000","last_fullsized":"2024-09-11T03:40:41.348954+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:41.348828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:41.348828+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:09.856306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'92","reported_seq":22,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.846507+0000","last_change":"2024-09-11T03:40:40.188174+0000","last_active":"2024-09-11T03:40:40.846507+0000","last_peered":"2024-09-11T03:40:40.846507+0000","last_clean":"2024-09-11T03:40:40.846507+0000","last_became_active":"2024-09-11T03:40:39.848230+0000","last_became_peered":"2024-09-11T03:40:39.848230+0000","last_unstale":"2024-09-11T03:40:40.846507+0000","last_undegraded":"2024-09-11T03:40:40.846507+0000","last_fullsized":"2024-09-11T03:40:40.846507+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.188124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.188124+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:05.549148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"20'92","reported_seq":14,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.846480+0000","last_change":"2024-09-11T03:40:39.860534+0000","last_active":"2024-09-11T03:40:40.846480+0000","last_peered":"2024-09-11T03:40:40.846480+0000","last_clean":"2024-09-11T03:40:40.846480+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:40:40.846480+0000","last_undegraded":"2024-09-11T03:40:40.846480+0000","last_fullsized":"2024-09-11T03:40:40.846480+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.258430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.258430+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"20'92","reported_seq":263,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.278801+0000","last_change":"2024-09-11T03:40:40.278752+0000","last_active":"2024-09-11T03:40:40.278801+0000","last_peered":"2024-09-11T03:40:40.278801+0000","last_clean":"2024-09-11T03:40:40.278801+0000","last_became_active":"2024-09-11T03:40:39.848781+0000","last_became_peered":"2024-09-11T03:40:39.848781+0000","last_unstale":"2024-09-11T03:40:40.278801+0000","last_undegraded":"2024-09-11T03:40:40.278801+0000","last_fullsized":"2024-09-11T03:40:40.278801+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.278703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.278703+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:06.105420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"20'92","reported_seq":22,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.851174+0000","last_change":"2024-09-11T03:40:40.718553+0000","last_active":"2024-09-11T03:40:40.851174+0000","last_peered":"2024-09-11T03:40:40.851174+0000","last_clean":"2024-09-11T03:40:40.851174+0000","last_became_active":"2024-09-11T03:40:39.851698+0000","last_became_peered":"2024-09-11T03:40:39.851698+0000","last_unstale":"2024-09-11T03:40:40.851174+0000","last_undegraded":"2024-09-11T03:40:40.851174+0000","last_fullsized":"2024-09-11T03:40:40.851174+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.718503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.718503+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:06.580046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.18","version":"20'92","reported_seq":139,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:41.025758+0000","last_change":"2024-09-11T03:40:40.365656+0000","last_active":"2024-09-11T03:40:41.025758+0000","last_peered":"2024-09-11T03:40:41.025758+0000","last_clean":"2024-09-11T03:40:41.025758+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:40:41.025758+0000","last_undegraded":"2024-09-11T03:40:41.025758+0000","last_fullsized":"2024-09-11T03:40:41.025758+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.365604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.365604+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:00.211289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"20'92","reported_seq":22,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:41.207387+0000","last_change":"2024-09-11T03:40:41.207341+0000","last_active":"2024-09-11T03:40:41.207387+0000","last_peered":"2024-09-11T03:40:41.207387+0000","last_clean":"2024-09-11T03:40:41.207387+0000","last_became_active":"2024-09-11T03:40:39.850243+0000","last_became_peered":"2024-09-11T03:40:39.850243+0000","last_unstale":"2024-09-11T03:40:41.207387+0000","last_undegraded":"2024-09-11T03:40:41.207387+0000","last_fullsized":"2024-09-11T03:40:41.207387+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:41.207291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:41.207291+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:56.540051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"20'92","reported_seq":22,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.848777+0000","last_change":"2024-09-11T03:40:40.309475+0000","last_active":"2024-09-11T03:40:40.848777+0000","last_peered":"2024-09-11T03:40:40.848777+0000","last_clean":"2024-09-11T03:40:40.848777+0000","last_became_active":"2024-09-11T03:40:39.849950+0000","last_became_peered":"2024-09-11T03:40:39.849950+0000","last_unstale":"2024-09-11T03:40:40.848777+0000","last_undegraded":"2024-09-11T03:40:40.848777+0000","last_fullsized":"2024-09-11T03:40:40.848777+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.309432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.309432+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:04.484327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"20'92","reported_seq":22,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.849757+0000","last_change":"2024-09-11T03:40:39.928143+0000","last_active":"2024-09-11T03:40:40.849757+0000","last_peered":"2024-09-11T03:40:40.849757+0000","last_clean":"2024-09-11T03:40:40.849757+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:40:40.849757+0000","last_undegraded":"2024-09-11T03:40:40.849757+0000","last_fullsized":"2024-09-11T03:40:40.849757+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.928088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.928088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:06.193518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"20'92","reported_seq":45,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T03:40:37.845516+0000","last_change":"2024-09-11T03:40:35.195938+0000","last_active":"2024-09-11T03:40:37.845516+0000","last_peered":"2024-09-11T03:40:37.845516+0000","last_clean":"2024-09-11T03:40:37.845516+0000","last_became_active":"2024-09-11T03:40:27.847297+0000","last_became_peered":"2024-09-11T03:40:27.847297+0000","last_unstale":"2024-09-11T03:40:37.845516+0000","last_undegraded":"2024-09-11T03:40:37.845516+0000","last_fullsized":"2024-09-11T03:40:37.845516+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.195893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.195893+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:54.375033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"20'92","reported_seq":131,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.848872+0000","last_change":"2024-09-11T03:40:35.277715+0000","last_active":"2024-09-11T03:40:40.848872+0000","last_peered":"2024-09-11T03:40:40.848872+0000","last_clean":"2024-09-11T03:40:40.848872+0000","last_became_active":"2024-09-11T03:40:27.847557+0000","last_became_peered":"2024-09-11T03:40:27.847557+0000","last_unstale":"2024-09-11T03:40:40.848872+0000","last_undegraded":"2024-09-11T03:40:40.848872+0000","last_fullsized":"2024-09-11T03:40:40.848872+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.277668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.277668+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:39.014977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"20'92","reported_seq":52,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:41.025675+0000","last_change":"2024-09-11T03:40:37.284871+0000","last_active":"2024-09-11T03:40:41.025675+0000","last_peered":"2024-09-11T03:40:41.025675+0000","last_clean":"2024-09-11T03:40:41.025675+0000","last_became_active":"2024-09-11T03:40:27.845249+0000","last_became_peered":"2024-09-11T03:40:27.845249+0000","last_unstale":"2024-09-11T03:40:41.025675+0000","last_undegraded":"2024-09-11T03:40:41.025675+0000","last_fullsized":"2024-09-11T03:40:41.025675+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:37.284800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:37.284800+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:53.535974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'92","reported_seq":54,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.851274+0000","last_change":"2024-09-11T03:40:35.703292+0000","last_active":"2024-09-11T03:40:40.851274+0000","last_peered":"2024-09-11T03:40:40.851274+0000","last_clean":"2024-09-11T03:40:40.851274+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:40:40.851274+0000","last_undegraded":"2024-09-11T03:40:40.851274+0000","last_fullsized":"2024-09-11T03:40:40.851274+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.703203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.703203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:49.443821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"20'92","reported_seq":52,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:41.025597+0000","last_change":"2024-09-11T03:40:36.320692+0000","last_active":"2024-09-11T03:40:41.025597+0000","last_peered":"2024-09-11T03:40:41.025597+0000","last_clean":"2024-09-11T03:40:41.025597+0000","last_became_active":"2024-09-11T03:40:27.844057+0000","last_became_peered":"2024-09-11T03:40:27.844057+0000","last_unstale":"2024-09-11T03:40:41.025597+0000","last_undegraded":"2024-09-11T03:40:41.025597+0000","last_fullsized":"2024-09-11T03:40:41.025597+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:36.320646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:36.320646+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:02.386305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"222'178","reported_seq":808,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.846779+0000","last_change":"2024-09-11T03:40:40.846921+0000","last_active":"2024-09-11T03:40:40.846779+0000","last_peered":"2024-09-11T03:40:40.846779+0000","last_clean":"2024-09-11T03:40:40.846779+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:40:40.846779+0000","last_undegraded":"2024-09-11T03:40:40.846779+0000","last_fullsized":"2024-09-11T03:40:40.846779+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'86","last_scrub_stamp":"2024-09-11T03:39:54.238844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:54.238844+0000","objects_scrubbed":5,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:17.423422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00031813799999999998,"stat_sum":{"num_bytes":13227142,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":15676,"num_write":79,"num_write_kb":38272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"37","length":"2"},{"start":"42","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":128,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:41.025855+0000","last_change":"2024-09-11T03:40:38.319593+0000","last_active":"2024-09-11T03:40:41.025855+0000","last_peered":"2024-09-11T03:40:41.025855+0000","last_clean":"2024-09-11T03:40:41.025855+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:40:41.025855+0000","last_undegraded":"2024-09-11T03:40:41.025855+0000","last_fullsized":"2024-09-11T03:40:41.025855+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.319543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.319543+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:03.765057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"222'55","reported_seq":699,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-11T03:40:41.862352+0000","last_change":"2024-09-11T03:40:41.846249+0000","last_active":"2024-09-11T03:40:41.862352+0000","last_peered":"2024-09-11T03:40:41.862352+0000","last_clean":"2024-09-11T03:40:41.862352+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:40:41.862352+0000","last_undegraded":"2024-09-11T03:40:41.862352+0000","last_fullsized":"2024-09-11T03:40:41.862352+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'8","last_scrub_stamp":"2024-09-11T03:39:33.449882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:33.449882+0000","objects_scrubbed":1,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:49.368764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001168587,"stat_sum":{"num_bytes":3527712,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":46720,"num_write":22,"num_write_kb":11502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"38","length":"1"},{"start":"40","length":"1"},{"start":"42","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":451,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.849487+0000","last_change":"2024-09-11T03:40:37.885659+0000","last_active":"2024-09-11T03:40:40.849487+0000","last_peered":"2024-09-11T03:40:40.849487+0000","last_clean":"2024-09-11T03:40:40.849487+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:40:40.849487+0000","last_undegraded":"2024-09-11T03:40:40.849487+0000","last_fullsized":"2024-09-11T03:40:40.849487+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:37.885602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:37.885602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:59.955114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"222'234","reported_seq":907,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.849520+0000","last_change":"2024-09-11T03:40:40.849520+0000","last_active":"2024-09-11T03:40:40.849520+0000","last_peered":"2024-09-11T03:40:40.849520+0000","last_clean":"2024-09-11T03:40:40.849520+0000","last_became_active":"2024-09-11T03:38:29.888204+0000","last_became_peered":"2024-09-11T03:38:29.888204+0000","last_unstale":"2024-09-11T03:40:40.849520+0000","last_undegraded":"2024-09-11T03:40:40.849520+0000","last_fullsized":"2024-09-11T03:40:40.849520+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'138","last_scrub_stamp":"2024-09-11T03:39:52.405246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:52.405246+0000","objects_scrubbed":6,"log_size":234,"log_dups_size":0,"ondisk_log_size":234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:53.907592+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00078056600000000001,"stat_sum":{"num_bytes":18823840,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":170,"num_read_kb":137110,"num_write":127,"num_write_kb":48899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"37","length":"2"},{"start":"42","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":348,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T03:40:37.850421+0000","last_change":"2024-09-11T03:40:37.850324+0000","last_active":"2024-09-11T03:40:37.850421+0000","last_peered":"2024-09-11T03:40:37.850421+0000","last_clean":"2024-09-11T03:40:37.850421+0000","last_became_active":"2024-09-11T03:40:37.849988+0000","last_became_peered":"2024-09-11T03:40:37.849988+0000","last_unstale":"2024-09-11T03:40:37.850421+0000","last_undegraded":"2024-09-11T03:40:37.850421+0000","last_fullsized":"2024-09-11T03:40:37.850421+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:00.202664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:00.202664+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"223'153","reported_seq":790,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.850163+0000","last_change":"2024-09-11T03:40:40.850163+0000","last_active":"2024-09-11T03:40:40.850163+0000","last_peered":"2024-09-11T03:40:40.850163+0000","last_clean":"2024-09-11T03:40:40.850163+0000","last_became_active":"2024-09-11T03:38:30.162819+0000","last_became_peered":"2024-09-11T03:38:30.162819+0000","last_unstale":"2024-09-11T03:40:40.850163+0000","last_undegraded":"2024-09-11T03:40:40.850163+0000","last_fullsized":"2024-09-11T03:40:40.850163+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'86","last_scrub_stamp":"2024-09-11T03:39:54.511997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:54.511997+0000","objects_scrubbed":4,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:02.384410+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00045542500000000002,"stat_sum":{"num_bytes":12555439,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":81,"num_read_kb":85188,"num_write":74,"num_write_kb":24099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"37","length":"2"},{"start":"42","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":68,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-11T03:40:41.844878+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:40:41.844878+0000","last_peered":"2024-09-11T03:40:41.844878+0000","last_clean":"2024-09-11T03:40:41.844878+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:40:41.844878+0000","last_undegraded":"2024-09-11T03:40:41.844878+0000","last_fullsized":"2024-09-11T03:40:41.844878+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:52.822589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"219'181","reported_seq":871,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T03:40:37.847122+0000","last_change":"2024-09-11T03:40:34.861176+0000","last_active":"2024-09-11T03:40:37.847122+0000","last_peered":"2024-09-11T03:40:37.847122+0000","last_clean":"2024-09-11T03:40:37.847122+0000","last_became_active":"2024-09-11T03:40:08.836627+0000","last_became_peered":"2024-09-11T03:40:08.836627+0000","last_unstale":"2024-09-11T03:40:37.847122+0000","last_undegraded":"2024-09-11T03:40:37.847122+0000","last_fullsized":"2024-09-11T03:40:37.847122+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'75","last_scrub_stamp":"2024-09-11T03:39:39.753844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:39.753844+0000","objects_scrubbed":4,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:08.442464+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0081724780000000004,"stat_sum":{"num_bytes":25081650,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":90,"num_read_kb":53029,"num_write":108,"num_write_kb":45874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":6433401,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"37","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":226,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-11T03:40:41.845084+0000","last_change":"2024-09-11T03:40:35.272628+0000","last_active":"2024-09-11T03:40:41.845084+0000","last_peered":"2024-09-11T03:40:41.845084+0000","last_clean":"2024-09-11T03:40:41.845084+0000","last_became_active":"2024-09-11T03:40:25.858281+0000","last_became_peered":"2024-09-11T03:40:25.858281+0000","last_unstale":"2024-09-11T03:40:41.845084+0000","last_undegraded":"2024-09-11T03:40:41.845084+0000","last_fullsized":"2024-09-11T03:40:41.845084+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.272563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.272563+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:53.042850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"223'134","reported_seq":770,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.849433+0000","last_change":"2024-09-11T03:40:40.849433+0000","last_active":"2024-09-11T03:40:40.849433+0000","last_peered":"2024-09-11T03:40:40.849433+0000","last_clean":"2024-09-11T03:40:40.849433+0000","last_became_active":"2024-09-11T03:38:29.887945+0000","last_became_peered":"2024-09-11T03:38:29.887945+0000","last_unstale":"2024-09-11T03:40:40.849433+0000","last_undegraded":"2024-09-11T03:40:40.849433+0000","last_fullsized":"2024-09-11T03:40:40.849433+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'103","last_scrub_stamp":"2024-09-11T03:40:17.171458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:17.171458+0000","objects_scrubbed":4,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:29.733596+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00051484100000000004,"stat_sum":{"num_bytes":7343853,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":60,"num_read_kb":58349,"num_write":55,"num_write_kb":29109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"37","length":"2"},{"start":"42","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":115,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T03:40:37.852102+0000","last_change":"2024-09-11T03:40:37.852004+0000","last_active":"2024-09-11T03:40:37.852102+0000","last_peered":"2024-09-11T03:40:37.852102+0000","last_clean":"2024-09-11T03:40:37.852102+0000","last_became_active":"2024-09-11T03:40:37.851080+0000","last_became_peered":"2024-09-11T03:40:37.851080+0000","last_unstale":"2024-09-11T03:40:37.852102+0000","last_undegraded":"2024-09-11T03:40:37.852102+0000","last_fullsized":"2024-09-11T03:40:37.852102+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:36.181642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:36.181642+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"222'205","reported_seq":878,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.846857+0000","last_change":"2024-09-11T03:40:40.847011+0000","last_active":"2024-09-11T03:40:40.846857+0000","last_peered":"2024-09-11T03:40:40.846857+0000","last_clean":"2024-09-11T03:40:40.846857+0000","last_became_active":"2024-09-11T03:38:30.163080+0000","last_became_peered":"2024-09-11T03:38:30.163080+0000","last_unstale":"2024-09-11T03:40:40.846857+0000","last_undegraded":"2024-09-11T03:40:40.846857+0000","last_fullsized":"2024-09-11T03:40:40.846857+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'97","last_scrub_stamp":"2024-09-11T03:39:33.455974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:33.455974+0000","objects_scrubbed":5,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:46.506100+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00035354800000000002,"stat_sum":{"num_bytes":19999305,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":136,"num_read_kb":122492,"num_write":106,"num_write_kb":34012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"37","length":"2"},{"start":"42","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":359,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T03:40:38.367213+0000","last_change":"2024-09-11T03:40:38.367213+0000","last_active":"2024-09-11T03:40:38.367213+0000","last_peered":"2024-09-11T03:40:38.367213+0000","last_clean":"2024-09-11T03:40:38.367213+0000","last_became_active":"2024-09-11T03:40:37.848794+0000","last_became_peered":"2024-09-11T03:40:37.848794+0000","last_unstale":"2024-09-11T03:40:38.367213+0000","last_undegraded":"2024-09-11T03:40:38.367213+0000","last_fullsized":"2024-09-11T03:40:38.367213+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.367170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.367170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:42.727574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"224'114","reported_seq":778,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.852250+0000","last_change":"2024-09-11T03:40:40.852483+0000","last_active":"2024-09-11T03:40:40.852250+0000","last_peered":"2024-09-11T03:40:40.852250+0000","last_clean":"2024-09-11T03:40:40.852250+0000","last_became_active":"2024-09-11T03:40:05.966715+0000","last_became_peered":"2024-09-11T03:40:05.966715+0000","last_unstale":"2024-09-11T03:40:40.852250+0000","last_undegraded":"2024-09-11T03:40:40.852250+0000","last_fullsized":"2024-09-11T03:40:40.852250+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'55","last_scrub_stamp":"2024-09-11T03:39:40.378705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:40.378705+0000","objects_scrubbed":3,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:57.751677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0032534740000000001,"stat_sum":{"num_bytes":6118774,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":38,"num_read_kb":18898,"num_write":63,"num_write_kb":19657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":7190868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"37","length":"2"},{"start":"42","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":115,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T03:40:37.850089+0000","last_change":"2024-09-11T03:40:37.849984+0000","last_active":"2024-09-11T03:40:37.850089+0000","last_peered":"2024-09-11T03:40:37.850089+0000","last_clean":"2024-09-11T03:40:37.850089+0000","last_became_active":"2024-09-11T03:40:37.849603+0000","last_became_peered":"2024-09-11T03:40:37.849603+0000","last_unstale":"2024-09-11T03:40:37.850089+0000","last_undegraded":"2024-09-11T03:40:37.850089+0000","last_fullsized":"2024-09-11T03:40:37.850089+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:36.364833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:36.364833+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"224'90","reported_seq":749,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:41.027379+0000","last_change":"2024-09-11T03:40:41.027540+0000","last_active":"2024-09-11T03:40:41.027379+0000","last_peered":"2024-09-11T03:40:41.027379+0000","last_clean":"2024-09-11T03:40:41.027379+0000","last_became_active":"2024-09-11T03:39:26.115670+0000","last_became_peered":"2024-09-11T03:39:26.115670+0000","last_unstale":"2024-09-11T03:40:41.027379+0000","last_undegraded":"2024-09-11T03:40:41.027379+0000","last_fullsized":"2024-09-11T03:40:41.027379+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'57","last_scrub_stamp":"2024-09-11T03:39:56.268853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:56.268853+0000","objects_scrubbed":1,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:05.232467+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017558249999999999,"stat_sum":{"num_bytes":9865709,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":20,"num_read_kb":7,"num_write":43,"num_write_kb":25227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"37","length":"2"},{"start":"42","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":345,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T03:40:37.844555+0000","last_change":"2024-09-11T03:40:35.325550+0000","last_active":"2024-09-11T03:40:37.844555+0000","last_peered":"2024-09-11T03:40:37.844555+0000","last_clean":"2024-09-11T03:40:37.844555+0000","last_became_active":"2024-09-11T03:40:25.854269+0000","last_became_peered":"2024-09-11T03:40:25.854269+0000","last_unstale":"2024-09-11T03:40:37.844555+0000","last_undegraded":"2024-09-11T03:40:37.844555+0000","last_fullsized":"2024-09-11T03:40:37.844555+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.325493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.325493+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:04.715032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"218'60","reported_seq":680,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T03:40:37.847425+0000","last_change":"2024-09-11T03:40:34.857778+0000","last_active":"2024-09-11T03:40:37.847425+0000","last_peered":"2024-09-11T03:40:37.847425+0000","last_clean":"2024-09-11T03:40:37.847425+0000","last_became_active":"2024-09-11T03:38:30.162952+0000","last_became_peered":"2024-09-11T03:38:30.162952+0000","last_unstale":"2024-09-11T03:40:37.847425+0000","last_undegraded":"2024-09-11T03:40:37.847425+0000","last_fullsized":"2024-09-11T03:40:37.847425+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'49","last_scrub_stamp":"2024-09-11T03:39:51.194245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:51.194245+0000","objects_scrubbed":2,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:11.256012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0051926760000000002,"stat_sum":{"num_bytes":4031962,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":19746,"num_write":29,"num_write_kb":11714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"37","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":457,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.849636+0000","last_change":"2024-09-11T03:40:34.916937+0000","last_active":"2024-09-11T03:40:40.849636+0000","last_peered":"2024-09-11T03:40:40.849636+0000","last_clean":"2024-09-11T03:40:40.849636+0000","last_became_active":"2024-09-11T03:40:25.850944+0000","last_became_peered":"2024-09-11T03:40:25.850944+0000","last_unstale":"2024-09-11T03:40:40.849636+0000","last_undegraded":"2024-09-11T03:40:40.849636+0000","last_fullsized":"2024-09-11T03:40:40.849636+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:34.916889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:34.916889+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:45.003655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"203'29","reported_seq":675,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.849882+0000","last_change":"2024-09-11T03:40:40.850193+0000","last_active":"2024-09-11T03:40:40.849882+0000","last_peered":"2024-09-11T03:40:40.849882+0000","last_clean":"2024-09-11T03:40:40.849882+0000","last_became_active":"2024-09-11T03:40:05.966325+0000","last_became_peered":"2024-09-11T03:40:05.966325+0000","last_unstale":"2024-09-11T03:40:40.849882+0000","last_undegraded":"2024-09-11T03:40:40.849882+0000","last_fullsized":"2024-09-11T03:40:40.849882+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'14","last_scrub_stamp":"2024-09-11T03:39:33.465057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:33.465057+0000","objects_scrubbed":1,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:55.833047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00059583200000000002,"stat_sum":{"num_bytes":3781575,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":5534,"num_write":20,"num_write_kb":5689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"37","length":"2"},{"start":"42","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":128,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.848764+0000","last_change":"2024-09-11T03:40:36.259209+0000","last_active":"2024-09-11T03:40:40.848764+0000","last_peered":"2024-09-11T03:40:40.848764+0000","last_clean":"2024-09-11T03:40:40.848764+0000","last_became_active":"2024-09-11T03:40:25.856407+0000","last_became_peered":"2024-09-11T03:40:25.856407+0000","last_unstale":"2024-09-11T03:40:40.848764+0000","last_undegraded":"2024-09-11T03:40:40.848764+0000","last_fullsized":"2024-09-11T03:40:40.848764+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:36.259163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:36.259163+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:55.651280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"222'78","reported_seq":720,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:41.026115+0000","last_change":"2024-09-11T03:40:41.026290+0000","last_active":"2024-09-11T03:40:41.026115+0000","last_peered":"2024-09-11T03:40:41.026115+0000","last_clean":"2024-09-11T03:40:41.026115+0000","last_became_active":"2024-09-11T03:38:29.889091+0000","last_became_peered":"2024-09-11T03:38:29.889091+0000","last_unstale":"2024-09-11T03:40:41.026115+0000","last_undegraded":"2024-09-11T03:40:41.026115+0000","last_fullsized":"2024-09-11T03:40:41.026115+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'44","last_scrub_stamp":"2024-09-11T03:39:32.965569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:32.965569+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:55.085172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00047188200000000002,"stat_sum":{"num_bytes":5495889,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":24896,"num_write":34,"num_write_kb":15043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"37","length":"2"},{"start":"42","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":448,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.849954+0000","last_change":"2024-09-11T03:40:36.901245+0000","last_active":"2024-09-11T03:40:40.849954+0000","last_peered":"2024-09-11T03:40:40.849954+0000","last_clean":"2024-09-11T03:40:40.849954+0000","last_became_active":"2024-09-11T03:40:25.854624+0000","last_became_peered":"2024-09-11T03:40:25.854624+0000","last_unstale":"2024-09-11T03:40:40.849954+0000","last_undegraded":"2024-09-11T03:40:40.849954+0000","last_fullsized":"2024-09-11T03:40:40.849954+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:36.901166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:36.901166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:43.207763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"20'92","reported_seq":54,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:41.025710+0000","last_change":"2024-09-11T03:40:35.282811+0000","last_active":"2024-09-11T03:40:41.025710+0000","last_peered":"2024-09-11T03:40:41.025710+0000","last_clean":"2024-09-11T03:40:41.025710+0000","last_became_active":"2024-09-11T03:40:27.844262+0000","last_became_peered":"2024-09-11T03:40:27.844262+0000","last_unstale":"2024-09-11T03:40:41.025710+0000","last_undegraded":"2024-09-11T03:40:41.025710+0000","last_fullsized":"2024-09-11T03:40:41.025710+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.282762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.282762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:51.256113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"225'144","reported_seq":807,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-11T03:40:41.867217+0000","last_change":"2024-09-11T03:40:41.847685+0000","last_active":"2024-09-11T03:40:41.867217+0000","last_peered":"2024-09-11T03:40:41.867217+0000","last_clean":"2024-09-11T03:40:41.867217+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:40:41.867217+0000","last_undegraded":"2024-09-11T03:40:41.867217+0000","last_fullsized":"2024-09-11T03:40:41.867217+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'52","last_scrub_stamp":"2024-09-11T03:39:56.458534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:56.458534+0000","objects_scrubbed":6,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:23.772883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024259920000000001,"stat_sum":{"num_bytes":16403665,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":104,"num_read_kb":79612,"num_write":85,"num_write_kb":36193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"38","length":"1"},{"start":"40","length":"1"},{"start":"42","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":307,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-11T03:40:39.845498+0000","last_change":"2024-09-11T03:40:38.199696+0000","last_active":"2024-09-11T03:40:39.845498+0000","last_peered":"2024-09-11T03:40:39.845498+0000","last_clean":"2024-09-11T03:40:39.845498+0000","last_became_active":"2024-09-11T03:40:37.850623+0000","last_became_peered":"2024-09-11T03:40:37.850623+0000","last_unstale":"2024-09-11T03:40:39.845498+0000","last_undegraded":"2024-09-11T03:40:39.845498+0000","last_fullsized":"2024-09-11T03:40:39.845498+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.199636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.199636+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:45.493648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"220'185","reported_seq":820,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T03:40:37.844753+0000","last_change":"2024-09-11T03:40:34.853724+0000","last_active":"2024-09-11T03:40:37.844753+0000","last_peered":"2024-09-11T03:40:37.844753+0000","last_clean":"2024-09-11T03:40:37.844753+0000","last_became_active":"2024-09-11T03:38:30.031497+0000","last_became_peered":"2024-09-11T03:38:30.031497+0000","last_unstale":"2024-09-11T03:40:37.844753+0000","last_undegraded":"2024-09-11T03:40:37.844753+0000","last_fullsized":"2024-09-11T03:40:37.844753+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'138","last_scrub_stamp":"2024-09-11T03:39:52.253596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:52.253596+0000","objects_scrubbed":3,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:01.098853+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020936689999999998,"stat_sum":{"num_bytes":11344725,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":123,"num_read_kb":99272,"num_write":78,"num_write_kb":28038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"37","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":66,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.849751+0000","last_change":"2024-09-11T03:40:38.901246+0000","last_active":"2024-09-11T03:40:40.849751+0000","last_peered":"2024-09-11T03:40:40.849751+0000","last_clean":"2024-09-11T03:40:40.849751+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:40:40.849751+0000","last_undegraded":"2024-09-11T03:40:40.849751+0000","last_fullsized":"2024-09-11T03:40:40.849751+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.901175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.901175+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:48.712523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"220'177","reported_seq":801,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T03:40:37.846823+0000","last_change":"2024-09-11T03:40:34.857979+0000","last_active":"2024-09-11T03:40:37.846823+0000","last_peered":"2024-09-11T03:40:37.846823+0000","last_clean":"2024-09-11T03:40:37.846823+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:40:37.846823+0000","last_undegraded":"2024-09-11T03:40:37.846823+0000","last_fullsized":"2024-09-11T03:40:37.846823+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'103","last_scrub_stamp":"2024-09-11T03:39:55.201092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:55.201092+0000","objects_scrubbed":2,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:05.704255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0053466,"stat_sum":{"num_bytes":10358288,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":104,"num_read_kb":83455,"num_write":64,"num_write_kb":31821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"37","length":"1"}]},{"pgid":"2.f","version":"222'121","reported_seq":765,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-11T03:40:40.846970+0000","last_change":"2024-09-11T03:40:40.847172+0000","last_active":"2024-09-11T03:40:40.846970+0000","last_peered":"2024-09-11T03:40:40.846970+0000","last_clean":"2024-09-11T03:40:40.846970+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:40:40.846970+0000","last_undegraded":"2024-09-11T03:40:40.846970+0000","last_fullsized":"2024-09-11T03:40:40.846970+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'82","last_scrub_stamp":"2024-09-11T03:39:53.261846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:39:53.261846+0000","objects_scrubbed":4,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:40:57.763901+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00050756900000000003,"stat_sum":{"num_bytes":9276202,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":17598,"num_write":57,"num_write_kb":21882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"37","length":"2"},{"start":"42","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":67,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-11T03:40:38.353992+0000","last_change":"2024-09-11T03:40:38.353992+0000","last_active":"2024-09-11T03:40:38.353992+0000","last_peered":"2024-09-11T03:40:38.353992+0000","last_clean":"2024-09-11T03:40:38.353992+0000","last_became_active":"2024-09-11T03:40:37.847449+0000","last_became_peered":"2024-09-11T03:40:37.847449+0000","last_unstale":"2024-09-11T03:40:38.353992+0000","last_undegraded":"2024-09-11T03:40:38.353992+0000","last_fullsized":"2024-09-11T03:40:38.353992+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.353895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.353895+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:00.706440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":177235730,"num_objects":75,"num_object_clones":35,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":3,"num_read":1141,"num_read_kb":867582,"num_write":1044,"num_write_kb":427031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":13624269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":337973248,"data_stored":404481583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2138,"ondisk_log_size":2138,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2334720,"data_stored":2316960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285610,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54060,"kb_used_data":22864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93691092,"statfs":{"total":95995035648,"available":95939678208,"internally_reserved":0,"allocated":23412736,"data_stored":27536702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":326417514542,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743475,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70040,"kb_used_data":38844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93675112,"statfs":{"total":95995035648,"available":95923314688,"internally_reserved":0,"allocated":39776256,"data_stored":46885257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037819,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63324,"kb_used_data":32136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93681828,"statfs":{"total":95995035648,"available":95930191872,"internally_reserved":0,"allocated":32907264,"data_stored":43780452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299454,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72732,"kb_used_data":37376,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93672420,"statfs":{"total":95995035648,"available":95920558080,"internally_reserved":0,"allocated":38273024,"data_stored":42554122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":193273528387,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":42036,"kb_used_data":10844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93703116,"statfs":{"total":95995035648,"available":95951990784,"internally_reserved":0,"allocated":11104256,"data_stored":10015898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986253,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71140,"kb_used_data":39944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93674012,"statfs":{"total":95995035648,"available":95922188288,"internally_reserved":0,"allocated":40902656,"data_stored":49049118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95384,"kb_used_data":60028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93649768,"statfs":{"total":95995035648,"available":95897362432,"internally_reserved":0,"allocated":61468672,"data_stored":71989538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67100,"kb_used_data":35904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93678052,"statfs":{"total":95995035648,"available":95926325248,"internally_reserved":0,"allocated":36765696,"data_stored":41655449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67820,"kb_used_data":36632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93677332,"statfs":{"total":95995035648,"available":95925587968,"internally_reserved":0,"allocated":37511168,"data_stored":37699538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65444,"kb_used_data":34248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93679708,"statfs":{"total":95995035648,"available":95928020992,"internally_reserved":0,"allocated":35069952,"data_stored":41966641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32141312,"data_stored":40098015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34897920,"data_stored":36150460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":33996800,"data_stored":39937973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":58359808,"data_stored":69936266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":38301696,"data_stored":47508172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8286208,"data_stored":8265572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4079616,"data_stored":4066399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":35454976,"data_stored":40803796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":30294016,"data_stored":42231374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36872192,"data_stored":45033233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2387968,"data_stored":2367002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":22900736,"data_stored":28083321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:40:44.262 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:44.313 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:44.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:44.335 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:44.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:44.371 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:40:44.436 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:44.463 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:44.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:44.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:44.591 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:44.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:44.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:44.693 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:44.714 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:44.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:44.715 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:44.816 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:44.838 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:44.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:44.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:44.849 INFO:tasks.rados.rados.2.smithi175.stdout:670: done (1 left) 2024-09-11T03:40:44.849 INFO:tasks.rados.rados.2.smithi175.stdout:671: done (0 left) 2024-09-11T03:40:44.849 INFO:tasks.rados.rados.2.smithi175.stdout:672: snap_remove snap 63 2024-09-11T03:40:44.876 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:40:44.877 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:40:44.877 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:40:44.877 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:40:44.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:44.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:44.968 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:44.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:45.069 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:45.091 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:45.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:45.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:45.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:45.197 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:40:45.219 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:45.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:45.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:45.321 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:45.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:45.343 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:45.343 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:45.438 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:45.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:45.469 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:45.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:45.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:45.571 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:45.574 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-11T03:40:45.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:45.593 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:45.593 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:45.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:45.720 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:45.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:45.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:45.822 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:45.830 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:40:45.846 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:40:45.848 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:45.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:45.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:45.852 INFO:tasks.rados.rados.2.smithi175.stdout:672: done (0 left) 2024-09-11T03:40:45.852 INFO:tasks.rados.rados.2.smithi175.stdout:673: write (excl) oid 49 current snap is 68 2024-09-11T03:40:45.852 INFO:tasks.rados.rados.2.smithi175.stdout:673: seq_num 267 ranges {487469=563838,1607751=551752} 2024-09-11T03:40:45.858 INFO:tasks.rados.rados.2.smithi175.stdout:673: writing smithi17557870-49 from 487469 to 1051307 tid 1 2024-09-11T03:40:45.864 INFO:tasks.rados.rados.2.smithi175.stdout:673: writing smithi17557870-49 from 1607751 to 2159503 tid 2 2024-09-11T03:40:45.865 INFO:tasks.rados.rados.2.smithi175.stdout:674: delete oid 22 current snap is 68 2024-09-11T03:40:45.866 INFO:tasks.rados.rados.2.smithi175.stdout:673: finishing write tid 1 to smithi17557870-49 2024-09-11T03:40:45.868 INFO:tasks.rados.rados.2.smithi175.stdout:673: finishing write tid 2 to smithi17557870-49 2024-09-11T03:40:45.868 INFO:tasks.rados.rados.2.smithi175.stdout:673: finishing write tid 3 to smithi17557870-49 2024-09-11T03:40:45.869 INFO:tasks.rados.rados.2.smithi175.stdout:673: finishing write tid 4 to smithi17557870-49 2024-09-11T03:40:45.869 INFO:tasks.rados.rados.2.smithi175.stdout:673: oid 49 updating version 0 to 160 2024-09-11T03:40:45.869 INFO:tasks.rados.rados.2.smithi175.stdout:673: oid 49 version 160 is already newer than 159 2024-09-11T03:40:45.869 INFO:tasks.rados.rados.2.smithi175.stdout:673: oid 49 version 160 is already newer than 158 2024-09-11T03:40:45.869 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 160 (ObjNum 267 snap 68 seq_num 267) dirty exists 2024-09-11T03:40:45.869 INFO:tasks.rados.rados.2.smithi175.stdout:673: left oid 49 (ObjNum 267 snap 68 seq_num 267) 2024-09-11T03:40:45.869 INFO:tasks.rados.rados.2.smithi175.stdout:673: done (1 left) 2024-09-11T03:40:45.869 INFO:tasks.rados.rados.2.smithi175.stdout:674: done (0 left) 2024-09-11T03:40:45.869 INFO:tasks.rados.rados.2.smithi175.stdout:675: write (excl) oid 39 current snap is 68 2024-09-11T03:40:45.869 INFO:tasks.rados.rados.2.smithi175.stdout:675: seq_num 269 ranges {521083=714336,1724715=555421} 2024-09-11T03:40:45.875 INFO:tasks.rados.rados.2.smithi175.stdout:675: writing smithi17557870-39 from 521083 to 1235419 tid 1 2024-09-11T03:40:45.881 INFO:tasks.rados.rados.2.smithi175.stdout:675: writing smithi17557870-39 from 1724715 to 2280136 tid 2 2024-09-11T03:40:45.881 INFO:tasks.rados.rados.2.smithi175.stdout:676: snap_create 2024-09-11T03:40:45.881 INFO:tasks.rados.rados.2.smithi175.stdout:675: finishing write tid 1 to smithi17557870-39 2024-09-11T03:40:45.883 INFO:tasks.rados.rados.2.smithi175.stdout:675: finishing write tid 2 to smithi17557870-39 2024-09-11T03:40:45.883 INFO:tasks.rados.rados.2.smithi175.stdout:675: finishing write tid 3 to smithi17557870-39 2024-09-11T03:40:45.883 INFO:tasks.rados.rados.2.smithi175.stdout:675: finishing write tid 4 to smithi17557870-39 2024-09-11T03:40:45.884 INFO:tasks.rados.rados.2.smithi175.stdout:675: oid 39 updating version 0 to 185 2024-09-11T03:40:45.884 INFO:tasks.rados.rados.2.smithi175.stdout:675: oid 39 updating version 185 to 186 2024-09-11T03:40:45.884 INFO:tasks.rados.rados.2.smithi175.stdout:675: oid 39 version 186 is already newer than 184 2024-09-11T03:40:45.884 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 186 (ObjNum 269 snap 68 seq_num 269) dirty exists 2024-09-11T03:40:45.884 INFO:tasks.rados.rados.2.smithi175.stdout:675: left oid 39 (ObjNum 269 snap 68 seq_num 269) 2024-09-11T03:40:45.912 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:40:45.912 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:40:45.929 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:40:45.929 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:40:45.945 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:40:45.945 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:40:45.950 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:45.961 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:40:45.961 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:40:45.971 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:45.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:45.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:45.978 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:40:45.978 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:40:45.995 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:40:45.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:40:46.012 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:40:46.013 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:40:46.029 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:40:46.029 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:40:46.046 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:40:46.046 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:40:46.062 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:40:46.063 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:40:46.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:45 smithi084 ceph-mon[27254]: pgmap v496: 47 pgs: 47 active+clean; 177 MiB data, 666 MiB used, 893 GiB / 894 GiB avail; 2.9 MiB/s rd, 4.2 MiB/s wr, 9 op/s 2024-09-11T03:40:46.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:45 smithi084 ceph-mon[27254]: osdmap e228: 12 total, 12 up, 10 in 2024-09-11T03:40:46.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:45 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1783051704' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:40:46.072 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:46.079 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:40:46.080 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:40:46.096 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:40:46.096 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:40:46.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:46.097 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:46.097 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:46.111 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:45 smithi175 ceph-mon[30894]: pgmap v496: 47 pgs: 47 active+clean; 177 MiB data, 666 MiB used, 893 GiB / 894 GiB avail; 2.9 MiB/s rd, 4.2 MiB/s wr, 9 op/s 2024-09-11T03:40:46.111 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:45 smithi175 ceph-mon[30894]: osdmap e228: 12 total, 12 up, 10 in 2024-09-11T03:40:46.111 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:45 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1783051704' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:40:46.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:45 smithi121 ceph-mon[30729]: pgmap v496: 47 pgs: 47 active+clean; 177 MiB data, 666 MiB used, 893 GiB / 894 GiB avail; 2.9 MiB/s rd, 4.2 MiB/s wr, 9 op/s 2024-09-11T03:40:46.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:45 smithi121 ceph-mon[30729]: osdmap e228: 12 total, 12 up, 10 in 2024-09-11T03:40:46.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:45 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1783051704' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:40:46.198 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:46.220 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:46.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:46.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:46.309 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:46.309 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:46.321 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:46.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:46.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:46.444 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:46.466 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:46.467 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:46.568 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:46.604 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:46.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:46.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:46.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:46.732 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:46.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:46.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:46.834 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:46.845 INFO:tasks.rados.rados.2.smithi175.stdout:675: done (1 left) 2024-09-11T03:40:46.845 INFO:tasks.rados.rados.2.smithi175.stdout:676: done (0 left) 2024-09-11T03:40:46.845 INFO:tasks.rados.rados.2.smithi175.stdout:677: rollback oid 5 current snap is 69 2024-09-11T03:40:46.845 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 5 to 60 2024-09-11T03:40:46.845 INFO:tasks.rados.rados.2.smithi175.stdout:678: write oid 11 current snap is 69 2024-09-11T03:40:46.845 INFO:tasks.rados.rados.2.smithi175.stdout:678: seq_num 270 ranges {537890=789585,1783197=699079} 2024-09-11T03:40:46.852 INFO:tasks.rados.rados.2.smithi175.stdout:678: writing smithi17557870-11 from 537890 to 1327475 tid 1 2024-09-11T03:40:46.857 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:46.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:46.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:46.859 INFO:tasks.rados.rados.2.smithi175.stdout:678: writing smithi17557870-11 from 1783197 to 2482276 tid 2 2024-09-11T03:40:46.859 INFO:tasks.rados.rados.2.smithi175.stdout:679: write oid 30 current snap is 69 2024-09-11T03:40:46.859 INFO:tasks.rados.rados.2.smithi175.stdout:679: seq_num 271 ranges {554697=464834,1725326=559090} 2024-09-11T03:40:46.864 INFO:tasks.rados.rados.2.smithi175.stdout:679: writing smithi17557870-30 from 554697 to 1019531 tid 1 2024-09-11T03:40:46.871 INFO:tasks.rados.rados.2.smithi175.stdout:679: writing smithi17557870-30 from 1725326 to 2284416 tid 2 2024-09-11T03:40:46.871 INFO:tasks.rados.rados.2.smithi175.stdout:680: copy_from oid 47 from oid 23 current snap is 69 2024-09-11T03:40:46.871 INFO:tasks.rados.rados.2.smithi175.stdout:677: finishing rollback tid 0 to smithi17557870-5 2024-09-11T03:40:46.871 INFO:tasks.rados.rados.2.smithi175.stdout:677: finishing rollback tid 1 to smithi17557870-5 2024-09-11T03:40:46.871 INFO:tasks.rados.rados.2.smithi175.stdout:677: finishing rollback tid 2 to smithi17557870-5 2024-09-11T03:40:46.871 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 217 (ObjNum 238 snap 58 seq_num 238) dirty exists 2024-09-11T03:40:46.871 INFO:tasks.rados.rados.2.smithi175.stdout:677: done (3 left) 2024-09-11T03:40:46.872 INFO:tasks.rados.rados.2.smithi175.stdout:681: write (excl) oid 28 current snap is 69 2024-09-11T03:40:46.872 INFO:tasks.rados.rados.2.smithi175.stdout:678: finishing write tid 1 to smithi17557870-11 2024-09-11T03:40:46.872 INFO:tasks.rados.rados.2.smithi175.stdout:681: seq_num 272 ranges {571504=540083,1783808=702748} 2024-09-11T03:40:46.877 INFO:tasks.rados.rados.2.smithi175.stdout:681: writing smithi17557870-28 from 571504 to 1111587 tid 1 2024-09-11T03:40:46.886 INFO:tasks.rados.rados.2.smithi175.stdout:681: writing smithi17557870-28 from 1783808 to 2486556 tid 2 2024-09-11T03:40:46.886 INFO:tasks.rados.rados.2.smithi175.stdout:682: snap_create 2024-09-11T03:40:46.886 INFO:tasks.rados.rados.2.smithi175.stdout:678: finishing write tid 2 to smithi17557870-11 2024-09-11T03:40:46.886 INFO:tasks.rados.rados.2.smithi175.stdout:678: finishing write tid 3 to smithi17557870-11 2024-09-11T03:40:46.886 INFO:tasks.rados.rados.2.smithi175.stdout:678: finishing write tid 4 to smithi17557870-11 2024-09-11T03:40:46.886 INFO:tasks.rados.rados.2.smithi175.stdout:678: oid 11 updating version 0 to 85 2024-09-11T03:40:46.886 INFO:tasks.rados.rados.2.smithi175.stdout:678: oid 11 version 85 is already newer than 83 2024-09-11T03:40:46.886 INFO:tasks.rados.rados.2.smithi175.stdout:678: oid 11 version 85 is already newer than 84 2024-09-11T03:40:46.887 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 85 (ObjNum 270 snap 69 seq_num 270) dirty exists 2024-09-11T03:40:46.887 INFO:tasks.rados.rados.2.smithi175.stdout:678: left oid 11 (ObjNum 270 snap 69 seq_num 270) 2024-09-11T03:40:46.887 INFO:tasks.rados.rados.2.smithi175.stdout:679: finishing write tid 1 to smithi17557870-30 2024-09-11T03:40:46.887 INFO:tasks.rados.rados.2.smithi175.stdout:679: finishing write tid 2 to smithi17557870-30 2024-09-11T03:40:46.887 INFO:tasks.rados.rados.2.smithi175.stdout:679: finishing write tid 3 to smithi17557870-30 2024-09-11T03:40:46.887 INFO:tasks.rados.rados.2.smithi175.stdout:679: finishing write tid 4 to smithi17557870-30 2024-09-11T03:40:46.887 INFO:tasks.rados.rados.2.smithi175.stdout:679: oid 30 updating version 0 to 206 2024-09-11T03:40:46.887 INFO:tasks.rados.rados.2.smithi175.stdout:679: oid 30 version 206 is already newer than 205 2024-09-11T03:40:46.887 INFO:tasks.rados.rados.2.smithi175.stdout:679: oid 30 version 206 is already newer than 204 2024-09-11T03:40:46.887 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 206 (ObjNum 271 snap 69 seq_num 271) dirty exists 2024-09-11T03:40:46.887 INFO:tasks.rados.rados.2.smithi175.stdout:679: left oid 30 (ObjNum 271 snap 69 seq_num 271) 2024-09-11T03:40:46.887 INFO:tasks.rados.rados.2.smithi175.stdout:681: finishing write tid 1 to smithi17557870-28 2024-09-11T03:40:46.888 INFO:tasks.rados.rados.2.smithi175.stdout:680: finishing copy_from racing read to smithi17557870-47 2024-09-11T03:40:46.888 INFO:tasks.rados.rados.2.smithi175.stdout:680: finishing copy_from to smithi17557870-47 2024-09-11T03:40:46.888 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 82 (ObjNum 171 snap 39 seq_num 171) dirty exists 2024-09-11T03:40:46.889 INFO:tasks.rados.rados.2.smithi175.stdout:681: finishing write tid 2 to smithi17557870-28 2024-09-11T03:40:46.889 INFO:tasks.rados.rados.2.smithi175.stdout:681: finishing write tid 3 to smithi17557870-28 2024-09-11T03:40:46.889 INFO:tasks.rados.rados.2.smithi175.stdout:681: finishing write tid 4 to smithi17557870-28 2024-09-11T03:40:46.889 INFO:tasks.rados.rados.2.smithi175.stdout:681: oid 28 updating version 0 to 244 2024-09-11T03:40:46.889 INFO:tasks.rados.rados.2.smithi175.stdout:681: oid 28 version 244 is already newer than 243 2024-09-11T03:40:46.889 INFO:tasks.rados.rados.2.smithi175.stdout:681: oid 28 updating version 244 to 245 2024-09-11T03:40:46.889 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 245 (ObjNum 272 snap 69 seq_num 272) dirty exists 2024-09-11T03:40:46.889 INFO:tasks.rados.rados.2.smithi175.stdout:681: left oid 28 (ObjNum 272 snap 69 seq_num 272) 2024-09-11T03:40:46.938 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:46 smithi175 ceph-mon[30894]: 2.5 scrub starts 2024-09-11T03:40:46.938 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:46 smithi175 ceph-mon[30894]: 2.5 scrub ok 2024-09-11T03:40:46.938 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:46 smithi175 ceph-mon[30894]: 2.3 scrub starts 2024-09-11T03:40:46.938 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:46 smithi175 ceph-mon[30894]: 2.3 scrub ok 2024-09-11T03:40:46.938 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:46 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:46.938 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1783051704' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:40:46.938 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:46 smithi175 ceph-mon[30894]: osdmap e229: 12 total, 12 up, 10 in 2024-09-11T03:40:46.938 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2778963774' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:46.938 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:46 smithi175 ceph-mon[30894]: osdmap e230: 12 total, 12 up, 10 in 2024-09-11T03:40:46.959 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:46.965 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-11T03:40:46.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:46.981 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:46.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:47.081 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:47.103 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:47.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:47.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:46 smithi084 ceph-mon[27254]: 2.5 scrub starts 2024-09-11T03:40:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:46 smithi084 ceph-mon[27254]: 2.5 scrub ok 2024-09-11T03:40:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:46 smithi084 ceph-mon[27254]: 2.3 scrub starts 2024-09-11T03:40:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:46 smithi084 ceph-mon[27254]: 2.3 scrub ok 2024-09-11T03:40:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:46 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1783051704' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:40:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:46 smithi084 ceph-mon[27254]: osdmap e229: 12 total, 12 up, 10 in 2024-09-11T03:40:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2778963774' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:46 smithi084 ceph-mon[27254]: osdmap e230: 12 total, 12 up, 10 in 2024-09-11T03:40:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:46 smithi121 ceph-mon[30729]: 2.5 scrub starts 2024-09-11T03:40:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:46 smithi121 ceph-mon[30729]: 2.5 scrub ok 2024-09-11T03:40:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:46 smithi121 ceph-mon[30729]: 2.3 scrub starts 2024-09-11T03:40:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:46 smithi121 ceph-mon[30729]: 2.3 scrub ok 2024-09-11T03:40:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:46 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1783051704' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:40:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:46 smithi121 ceph-mon[30729]: osdmap e229: 12 total, 12 up, 10 in 2024-09-11T03:40:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2778963774' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:46 smithi121 ceph-mon[30729]: osdmap e230: 12 total, 12 up, 10 in 2024-09-11T03:40:47.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:47.227 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:47.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:47.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:47.247 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:40:47.329 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:47.350 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:47.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:47.351 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:47.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:47.478 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:47.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:47.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:47.605 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:47.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:47.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:47.707 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:47.729 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:47.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:47.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:47.764 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:47.764 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:47.831 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:47.852 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:47.854 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:47.869 INFO:tasks.rados.rados.2.smithi175.stdout:678: done (4 left) 2024-09-11T03:40:47.869 INFO:tasks.rados.rados.2.smithi175.stdout:679: done (3 left) 2024-09-11T03:40:47.869 INFO:tasks.rados.rados.2.smithi175.stdout:680: done (2 left) 2024-09-11T03:40:47.870 INFO:tasks.rados.rados.2.smithi175.stdout:681: done (1 left) 2024-09-11T03:40:47.870 INFO:tasks.rados.rados.2.smithi175.stdout:682: done (0 left) 2024-09-11T03:40:47.870 INFO:tasks.rados.rados.2.smithi175.stdout:683: read oid 27 snap -1 2024-09-11T03:40:47.870 INFO:tasks.rados.rados.2.smithi175.stdout:683: expect (ObjNum 241 snap 58 seq_num 241) 2024-09-11T03:40:47.870 INFO:tasks.rados.rados.2.smithi175.stdout:684: write (excl) oid 45 current snap is 70 2024-09-11T03:40:47.870 INFO:tasks.rados.rados.2.smithi175.stdout:684: seq_num 273 ranges {588311=615332,1842290=562759} 2024-09-11T03:40:47.876 INFO:tasks.rados.rados.2.smithi175.stdout:684: writing smithi17557870-45 from 588311 to 1203643 tid 1 2024-09-11T03:40:47.883 INFO:tasks.rados.rados.2.smithi175.stdout:684: writing smithi17557870-45 from 1842290 to 2405049 tid 2 2024-09-11T03:40:47.883 INFO:tasks.rados.rados.2.smithi175.stdout:685: write oid 31 current snap is 70 2024-09-11T03:40:47.883 INFO:tasks.rados.rados.2.smithi175.stdout:685: seq_num 274 ranges {605118=406934,1617125=706417} 2024-09-11T03:40:47.888 INFO:tasks.rados.rados.2.smithi175.stdout:685: writing smithi17557870-31 from 605118 to 1012052 tid 1 2024-09-11T03:40:47.895 INFO:tasks.rados.rados.2.smithi175.stdout:685: writing smithi17557870-31 from 1617125 to 2323542 tid 2 2024-09-11T03:40:47.895 INFO:tasks.rados.rados.2.smithi175.stdout:686: rollback oid 7 current snap is 70 2024-09-11T03:40:47.895 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 7 to 59 2024-09-11T03:40:47.895 INFO:tasks.rados.rados.2.smithi175.stdout:683: expect (ObjNum 241 snap 58 seq_num 241) 2024-09-11T03:40:47.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:47.976 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:47.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:47.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:47.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:47 smithi175 ceph-mon[30894]: pgmap v499: 47 pgs: 47 active+clean; 178 MiB data, 675 MiB used, 893 GiB / 894 GiB avail; 3.1 MiB/s rd, 2.5 MiB/s wr, 9 op/s 2024-09-11T03:40:48.000 INFO:tasks.rados.rados.2.smithi175.stdout:684: finishing write tid 1 to smithi17557870-45 2024-09-11T03:40:48.000 INFO:tasks.rados.rados.2.smithi175.stdout:684: finishing write tid 2 to smithi17557870-45 2024-09-11T03:40:48.000 INFO:tasks.rados.rados.2.smithi175.stdout:684: finishing write tid 3 to smithi17557870-45 2024-09-11T03:40:48.000 INFO:tasks.rados.rados.2.smithi175.stdout:684: finishing write tid 4 to smithi17557870-45 2024-09-11T03:40:48.000 INFO:tasks.rados.rados.2.smithi175.stdout:684: oid 45 updating version 0 to 163 2024-09-11T03:40:48.000 INFO:tasks.rados.rados.2.smithi175.stdout:684: oid 45 version 163 is already newer than 162 2024-09-11T03:40:48.000 INFO:tasks.rados.rados.2.smithi175.stdout:684: oid 45 version 163 is already newer than 161 2024-09-11T03:40:48.000 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 163 (ObjNum 273 snap 70 seq_num 273) dirty exists 2024-09-11T03:40:48.000 INFO:tasks.rados.rados.2.smithi175.stdout:684: left oid 45 (ObjNum 273 snap 70 seq_num 273) 2024-09-11T03:40:48.001 INFO:tasks.rados.rados.2.smithi175.stdout:683: done (3 left) 2024-09-11T03:40:48.001 INFO:tasks.rados.rados.2.smithi175.stdout:684: done (2 left) 2024-09-11T03:40:48.001 INFO:tasks.rados.rados.2.smithi175.stdout:687: snap_create 2024-09-11T03:40:48.001 INFO:tasks.rados.rados.2.smithi175.stdout:685: finishing write tid 1 to smithi17557870-31 2024-09-11T03:40:48.001 INFO:tasks.rados.rados.2.smithi175.stdout:686: finishing rollback tid 0 to smithi17557870-7 2024-09-11T03:40:48.001 INFO:tasks.rados.rados.2.smithi175.stdout:686: finishing rollback tid 1 to smithi17557870-7 2024-09-11T03:40:48.001 INFO:tasks.rados.rados.2.smithi175.stdout:685: finishing write tid 2 to smithi17557870-31 2024-09-11T03:40:48.001 INFO:tasks.rados.rados.2.smithi175.stdout:686: finishing rollback tid 2 to smithi17557870-7 2024-09-11T03:40:48.001 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 142 (ObjNum 150 snap 35 seq_num 150) dirty exists 2024-09-11T03:40:48.001 INFO:tasks.rados.rados.2.smithi175.stdout:685: finishing write tid 3 to smithi17557870-31 2024-09-11T03:40:48.001 INFO:tasks.rados.rados.2.smithi175.stdout:685: finishing write tid 4 to smithi17557870-31 2024-09-11T03:40:48.001 INFO:tasks.rados.rados.2.smithi175.stdout:685: oid 31 updating version 0 to 220 2024-09-11T03:40:48.001 INFO:tasks.rados.rados.2.smithi175.stdout:685: oid 31 updating version 220 to 221 2024-09-11T03:40:48.002 INFO:tasks.rados.rados.2.smithi175.stdout:685: oid 31 version 221 is already newer than 219 2024-09-11T03:40:48.002 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 221 (ObjNum 274 snap 70 seq_num 274) dirty exists 2024-09-11T03:40:48.002 INFO:tasks.rados.rados.2.smithi175.stdout:685: left oid 31 (ObjNum 274 snap 70 seq_num 274) 2024-09-11T03:40:48.077 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:48.098 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:48.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:48.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:48.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:47 smithi084 ceph-mon[27254]: pgmap v499: 47 pgs: 47 active+clean; 178 MiB data, 675 MiB used, 893 GiB / 894 GiB avail; 3.1 MiB/s rd, 2.5 MiB/s wr, 9 op/s 2024-09-11T03:40:48.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:47 smithi121 ceph-mon[30729]: pgmap v499: 47 pgs: 47 active+clean; 178 MiB data, 675 MiB used, 893 GiB / 894 GiB avail; 3.1 MiB/s rd, 2.5 MiB/s wr, 9 op/s 2024-09-11T03:40:48.200 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:48.222 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:48.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:48.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:48.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:48.350 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:48.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:48.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:48.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:48.477 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:48.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:48.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:48.492 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-11T03:40:48.579 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:48.602 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:48.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:48.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:48.704 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:48.729 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:48.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:48.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:48.736 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:40:48.747 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:40:48.831 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:48.852 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:48.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:48.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:48.871 INFO:tasks.rados.rados.2.smithi175.stdout:685: done (2 left) 2024-09-11T03:40:48.871 INFO:tasks.rados.rados.2.smithi175.stdout:686: done (1 left) 2024-09-11T03:40:48.871 INFO:tasks.rados.rados.2.smithi175.stdout:687: done (0 left) 2024-09-11T03:40:48.871 INFO:tasks.rados.rados.2.smithi175.stdout:688: read oid 25 snap -1 2024-09-11T03:40:48.871 INFO:tasks.rados.rados.2.smithi175.stdout:688: expect (ObjNum 225 snap 53 seq_num 225) 2024-09-11T03:40:48.871 INFO:tasks.rados.rados.2.smithi175.stdout:689: read oid 48 snap -1 2024-09-11T03:40:48.871 INFO:tasks.rados.rados.2.smithi175.stdout:689: expect (ObjNum 246 snap 60 seq_num 246) 2024-09-11T03:40:48.871 INFO:tasks.rados.rados.2.smithi175.stdout:690: rollback oid 9 current snap is 71 2024-09-11T03:40:48.871 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 9 to 59 2024-09-11T03:40:48.871 INFO:tasks.rados.rados.2.smithi175.stdout:691: snap_remove snap 58 2024-09-11T03:40:48.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:48.976 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:48.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:49.002 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:49.078 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:49.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:49.100 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:49.101 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:49.107 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:48 smithi175 ceph-mon[30894]: osdmap e231: 12 total, 12 up, 10 in 2024-09-11T03:40:49.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:48 smithi084 ceph-mon[27254]: osdmap e231: 12 total, 12 up, 10 in 2024-09-11T03:40:49.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:48 smithi121 ceph-mon[30729]: osdmap e231: 12 total, 12 up, 10 in 2024-09-11T03:40:49.202 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:49.214 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:49.214 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:49.224 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:49.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:49.225 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:49.246 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [6, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:40:49.246 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:40:49.246 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-11T03:40:49.247 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight-by-utilization 110 0.05 2024-09-11T03:40:49.301 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:40:49.326 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:49.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:49.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:49.466 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:49.489 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:49.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:49.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:49.518 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:40:49.518 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:40:49.549 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:40:49.549 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:40:49.573 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:40:49.573 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:40:49.586 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:49.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:49.595 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:49.597 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:40:49.597 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:40:49.615 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:49.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:49.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:49.617 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:40:49.617 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:40:49.638 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:40:49.638 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:40:49.655 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:40:49.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:40:49.672 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:40:49.672 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:40:49.689 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:40:49.689 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:40:49.705 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:40:49.706 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:40:49.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:49.723 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:40:49.723 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:40:49.741 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:40:49.743 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:49.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:49.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:49.817 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-11T03:40:49.845 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:49.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:49.868 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:49.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:49.873 INFO:tasks.rados.rados.2.smithi175.stdout:691: done (3 left) 2024-09-11T03:40:49.873 INFO:tasks.rados.rados.2.smithi175.stdout:692: snap_create 2024-09-11T03:40:49.873 INFO:tasks.rados.rados.2.smithi175.stdout:689: expect (ObjNum 246 snap 60 seq_num 246) 2024-09-11T03:40:49.969 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:49.991 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:49.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:50.019 INFO:tasks.rados.rados.2.smithi175.stdout:690: finishing rollback tid 1 to smithi17557870-9 2024-09-11T03:40:50.019 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 0 (ObjNum 33 snap 0 seq_num 197401748) dirty dne 2024-09-11T03:40:50.019 INFO:tasks.rados.rados.2.smithi175.stdout:688: expect (ObjNum 225 snap 53 seq_num 225) 2024-09-11T03:40:50.074 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:40:50.092 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:50.113 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:50.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:50.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:50.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:49 smithi084 ceph-mon[27254]: pgmap v502: 47 pgs: 47 active+clean; 187 MiB data, 685 MiB used, 893 GiB / 894 GiB avail; 5.4 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2024-09-11T03:40:50.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:49 smithi084 ceph-mon[27254]: osdmap e232: 12 total, 12 up, 10 in 2024-09-11T03:40:50.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:49 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:40:50.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:49 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:40:50.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/534073707' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:49 smithi121 ceph-mon[30729]: pgmap v502: 47 pgs: 47 active+clean; 187 MiB data, 685 MiB used, 893 GiB / 894 GiB avail; 5.4 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2024-09-11T03:40:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:49 smithi121 ceph-mon[30729]: osdmap e232: 12 total, 12 up, 10 in 2024-09-11T03:40:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:49 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:40:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:49 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:40:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/534073707' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:50.194 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:49 smithi175 ceph-mon[30894]: pgmap v502: 47 pgs: 47 active+clean; 187 MiB data, 685 MiB used, 893 GiB / 894 GiB avail; 5.4 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2024-09-11T03:40:50.195 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:49 smithi175 ceph-mon[30894]: osdmap e232: 12 total, 12 up, 10 in 2024-09-11T03:40:50.195 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:49 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:40:50.195 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:49 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:40:50.195 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/534073707' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:50.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:50.241 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:50.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:50.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:50.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:50.369 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:50.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:50.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:50.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:50.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:50.498 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:50.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:50.536 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:40:50.536 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:40:50.536 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:40:50.536 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:40:50.536 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:40:50.599 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:50.621 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:50.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:50.622 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:50.723 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:50.743 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:50.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:50.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:50.845 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:50.866 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:50.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:50.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:50.889 INFO:tasks.rados.rados.2.smithi175.stdout:688: done (3 left) 2024-09-11T03:40:50.889 INFO:tasks.rados.rados.2.smithi175.stdout:689: done (2 left) 2024-09-11T03:40:50.889 INFO:tasks.rados.rados.2.smithi175.stdout:690: done (1 left) 2024-09-11T03:40:50.889 INFO:tasks.rados.rados.2.smithi175.stdout:692: done (0 left) 2024-09-11T03:40:50.889 INFO:tasks.rados.rados.2.smithi175.stdout:693: write oid 24 current snap is 72 2024-09-11T03:40:50.889 INFO:tasks.rados.rados.2.smithi175.stdout:693: seq_num 275 ranges {621925=482183,1559254=566428} 2024-09-11T03:40:50.894 INFO:tasks.rados.rados.2.smithi175.stdout:693: writing smithi17557870-24 from 621925 to 1104108 tid 1 2024-09-11T03:40:50.901 INFO:tasks.rados.rados.2.smithi175.stdout:693: writing smithi17557870-24 from 1559254 to 2125682 tid 2 2024-09-11T03:40:50.901 INFO:tasks.rados.rados.2.smithi175.stdout:694: snap_remove snap 68 2024-09-11T03:40:50.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:50.989 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:50.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:50.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:51.021 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:50 smithi084 ceph-mon[27254]: osdmap e233: 12 total, 12 up, 10 in 2024-09-11T03:40:51.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:51.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:51.113 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:51.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:51.132 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:50 smithi175 ceph-mon[30894]: osdmap e233: 12 total, 12 up, 10 in 2024-09-11T03:40:51.159 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-11T03:40:51.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:50 smithi121 ceph-mon[30729]: osdmap e233: 12 total, 12 up, 10 in 2024-09-11T03:40:51.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:51.241 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:51.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:51.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:51.343 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:51.365 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:51.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:51.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:51.410 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:40:51.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:51.494 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:51.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:51.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:51.596 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:51.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:51.622 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:51.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:51.723 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:51.746 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:51.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:51.843 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:40:51.844 INFO:teuthology.orchestra.run.smithi084.stdout:moved 0 / 141 (0%) 2024-09-11T03:40:51.844 INFO:teuthology.orchestra.run.smithi084.stdout:avg 14.1 2024-09-11T03:40:51.844 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 2.34308 -> 2.34308 (expected baseline 3.5623) 2024-09-11T03:40:51.844 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.5 with 11 -> 11 pgs (0.780142 -> 0.780142 * mean) 2024-09-11T03:40:51.844 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.7 with 19 -> 19 pgs (1.34752 -> 1.34752 * mean) 2024-09-11T03:40:51.844 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:40:51.845 INFO:teuthology.orchestra.run.smithi084.stdout:oload 110 2024-09-11T03:40:51.845 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 0.05 2024-09-11T03:40:51.845 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2024-09-11T03:40:51.845 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0008 2024-09-11T03:40:51.845 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0008 2024-09-11T03:40:51.845 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3 weight 1.0000 -> 0.9500 2024-09-11T03:40:51.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:51.847 INFO:tasks.rados.rados.2.smithi175.stdout:694: done (1 left) 2024-09-11T03:40:51.847 INFO:tasks.rados.rados.2.smithi175.stdout:695: write oid 32 current snap is 72 2024-09-11T03:40:51.848 INFO:tasks.rados.rados.2.smithi175.stdout:693: finishing write tid 1 to smithi17557870-24 2024-09-11T03:40:51.848 INFO:tasks.rados.rados.2.smithi175.stdout:695: seq_num 276 ranges {638732=557432,1617736=710086} 2024-09-11T03:40:51.854 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:40:51.854 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:40:51.854 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:40:51.854 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:40:51.854 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.854 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:363 2.c 2:327469ff:::smithi17557870-49:head [sparse-read 0~1365420,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:40:51.854 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.849855+0000", 2024-09-11T03:40:51.854 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 121.002591176, 2024-09-11T03:40:51.854 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0056744530000000003, 2024-09-11T03:40:51.854 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.855 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.855 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:51.855 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.855 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.855 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.855 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 363 2024-09-11T03:40:51.855 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.855 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.855 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.855 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.855 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849855+0000", 2024-09-11T03:40:51.855 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.855 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.855 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.856 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.856 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849855+0000", 2024-09-11T03:40:51.856 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.856 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.856 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.856 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.856 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849858+0000", 2024-09-11T03:40:51.856 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9040000000000001e-06 2024-09-11T03:40:51.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.857 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.857 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.857 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849870+0000", 2024-09-11T03:40:51.857 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2463000000000001e-05 2024-09-11T03:40:51.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.857 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.857 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.857 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849901+0000", 2024-09-11T03:40:51.857 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0747000000000002e-05 2024-09-11T03:40:51.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.857 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.857 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.858 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849936+0000", 2024-09-11T03:40:51.858 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5302000000000002e-05 2024-09-11T03:40:51.858 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.858 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.858 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.858 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850035+0000", 2024-09-11T03:40:51.858 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8622000000000005e-05 2024-09-11T03:40:51.858 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.858 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.858 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:40:51.858 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850042+0000", 2024-09-11T03:40:51.858 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2579999999999998e-06 2024-09-11T03:40:51.858 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.858 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.859 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.859 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.851748+0000", 2024-09-11T03:40:51.859 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0017055670000000001 2024-09-11T03:40:51.859 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.859 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.859 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.859 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.851847+0000", 2024-09-11T03:40:51.859 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8904e-05 2024-09-11T03:40:51.859 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.860 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.860 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.860 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.855530+0000", 2024-09-11T03:40:51.860 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0036826860000000001 2024-09-11T03:40:51.860 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.860 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.860 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.860 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.860 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.860 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:364 2.c 2:327469ff:::smithi17557870-49:head [sparse-read 0~1365420] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:40:51.860 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.849973+0000", 2024-09-11T03:40:51.861 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 121.002472767, 2024-09-11T03:40:51.861 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0059499059999999996, 2024-09-11T03:40:51.861 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.861 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.861 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:51.861 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.861 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.861 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.861 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 364 2024-09-11T03:40:51.862 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.862 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.862 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.862 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.862 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849973+0000", 2024-09-11T03:40:51.862 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.862 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.862 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.862 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.862 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849973+0000", 2024-09-11T03:40:51.862 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.862 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.862 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.863 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.863 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849975+0000", 2024-09-11T03:40:51.863 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.08e-06 2024-09-11T03:40:51.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.863 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.863 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.863 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849979+0000", 2024-09-11T03:40:51.863 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4390000000000002e-06 2024-09-11T03:40:51.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.863 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.863 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.864 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.849993+0000", 2024-09-11T03:40:51.864 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4208e-05 2024-09-11T03:40:51.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.864 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.864 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.864 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850000+0000", 2024-09-11T03:40:51.864 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0199999999999997e-06 2024-09-11T03:40:51.865 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.865 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.865 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.865 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850114+0000", 2024-09-11T03:40:51.865 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011416700000000001 2024-09-11T03:40:51.865 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.865 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.865 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:40:51.865 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850118+0000", 2024-09-11T03:40:51.865 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3000000000000002e-06 2024-09-11T03:40:51.865 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.865 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.866 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.866 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.855637+0000", 2024-09-11T03:40:51.866 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0055188750000000003 2024-09-11T03:40:51.866 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.867 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.867 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.867 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.855739+0000", 2024-09-11T03:40:51.867 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010228 2024-09-11T03:40:51.867 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.867 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.867 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.867 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.855923+0000", 2024-09-11T03:40:51.867 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000184537 2024-09-11T03:40:51.867 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.868 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.868 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.868 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.868 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.868 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:365 2.c 2:327469ff:::smithi17557870-49:head [read 0~2806736,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:40:51.868 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.850028+0000", 2024-09-11T03:40:51.868 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 121.00241804300001, 2024-09-11T03:40:51.868 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0077480070000000003, 2024-09-11T03:40:51.868 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.868 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.868 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:51.868 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.868 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.869 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.869 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 365 2024-09-11T03:40:51.869 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.869 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.869 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.869 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.869 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850028+0000", 2024-09-11T03:40:51.870 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.870 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.870 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.870 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.870 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850028+0000", 2024-09-11T03:40:51.870 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.870 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.870 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.870 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.870 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850029+0000", 2024-09-11T03:40:51.870 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1200000000000003e-07 2024-09-11T03:40:51.871 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.871 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.871 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850033+0000", 2024-09-11T03:40:51.871 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4789999999999998e-06 2024-09-11T03:40:51.871 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.871 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.871 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850043+0000", 2024-09-11T03:40:51.871 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.38e-06 2024-09-11T03:40:51.871 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.871 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.871 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850056+0000", 2024-09-11T03:40:51.872 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3179999999999999e-05 2024-09-11T03:40:51.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.872 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.872 INFO:tasks.rados.rados.2.smithi175.stdout:695: writing smithi17557870-32 from 638732 to 1196164 tid 1 2024-09-11T03:40:51.873 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850150+0000", 2024-09-11T03:40:51.873 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4549000000000006e-05 2024-09-11T03:40:51.873 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.873 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.873 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:40:51.873 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850155+0000", 2024-09-11T03:40:51.873 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0440000000000003e-06 2024-09-11T03:40:51.873 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.873 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.873 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.874 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.855967+0000", 2024-09-11T03:40:51.874 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0058113289999999996 2024-09-11T03:40:51.874 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.874 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.874 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.874 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.856051+0000", 2024-09-11T03:40:51.874 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3840999999999999e-05 2024-09-11T03:40:51.874 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.874 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.874 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.874 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.857776+0000", 2024-09-11T03:40:51.874 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0017255930000000001 2024-09-11T03:40:51.874 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.875 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.875 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.875 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.875 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.875 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:554 2.c 2:3603e322:::smithi17557870-42:head [write 447086~770638,stat] snapc 10=[10,f,e] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-11T03:40:51.875 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:03.827163+0000", 2024-09-11T03:40:51.875 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 108.025282956, 2024-09-11T03:40:51.875 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0056079759999999998, 2024-09-11T03:40:51.875 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.875 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.875 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:51.875 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.875 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.876 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.876 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 554 2024-09-11T03:40:51.876 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.876 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.876 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.876 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.876 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.827163+0000", 2024-09-11T03:40:51.876 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.876 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.876 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.876 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.876 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.827163+0000", 2024-09-11T03:40:51.876 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.877 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.877 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.877 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.827166+0000", 2024-09-11T03:40:51.877 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7219999999999999e-06 2024-09-11T03:40:51.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.877 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.877 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.877 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.827693+0000", 2024-09-11T03:40:51.877 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00052672999999999999 2024-09-11T03:40:51.878 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.878 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.878 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.827731+0000", 2024-09-11T03:40:51.878 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8373e-05 2024-09-11T03:40:51.878 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.878 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.878 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.827761+0000", 2024-09-11T03:40:51.878 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9927999999999999e-05 2024-09-11T03:40:51.878 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.879 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.879 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.879 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.827841+0000", 2024-09-11T03:40:51.879 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0247000000000006e-05 2024-09-11T03:40:51.879 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.879 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.879 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:40:51.879 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.827851+0000", 2024-09-11T03:40:51.879 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4409999999999999e-06 2024-09-11T03:40:51.879 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.879 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.879 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.880 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.829517+0000", 2024-09-11T03:40:51.880 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0016659349999999999 2024-09-11T03:40:51.880 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.880 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.880 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.880 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.829594+0000", 2024-09-11T03:40:51.880 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7294999999999998e-05 2024-09-11T03:40:51.880 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.880 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.880 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:40:51.880 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.829707+0000", 2024-09-11T03:40:51.880 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000113455 2024-09-11T03:40:51.881 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.881 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.881 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:40:51.881 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.831602+0000", 2024-09-11T03:40:51.881 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001894904 2024-09-11T03:40:51.881 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.881 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.881 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:51.882 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.831988+0000", 2024-09-11T03:40:51.882 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00038567000000000001 2024-09-11T03:40:51.882 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.882 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.882 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:51.882 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.832699+0000", 2024-09-11T03:40:51.882 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000711355 2024-09-11T03:40:51.882 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.883 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.883 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:51.883 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.832738+0000", 2024-09-11T03:40:51.883 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8170000000000002e-05 2024-09-11T03:40:51.883 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.883 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.883 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.883 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:03.832771+0000", 2024-09-11T03:40:51.884 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3751000000000003e-05 2024-09-11T03:40:51.884 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.884 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.884 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.884 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.884 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.884 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:640 2.a 2:5014e305:::smithi17557870-46:head [copy-from ver 23] snapc 11=[e] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:40:51.885 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:40:51.885 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 103.742153055, 2024-09-11T03:40:51.885 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019221736, 2024-09-11T03:40:51.885 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.885 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.885 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:51.885 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.885 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 640 2024-09-11T03:40:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:40:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:40:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110295+0000", 2024-09-11T03:40:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0609999999999998e-06 2024-09-11T03:40:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110305+0000", 2024-09-11T03:40:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4169999999999998e-06 2024-09-11T03:40:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110321+0000", 2024-09-11T03:40:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6288000000000001e-05 2024-09-11T03:40:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110328+0000", 2024-09-11T03:40:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3950000000000003e-06 2024-09-11T03:40:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110396+0000", 2024-09-11T03:40:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7164999999999998e-05 2024-09-11T03:40:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110507+0000", 2024-09-11T03:40:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000111217 2024-09-11T03:40:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:40:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.118213+0000", 2024-09-11T03:40:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0077066249999999999 2024-09-11T03:40:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:40:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125103+0000", 2024-09-11T03:40:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068894359999999997 2024-09-11T03:40:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125459+0000", 2024-09-11T03:40:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000356146 2024-09-11T03:40:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129369+0000", 2024-09-11T03:40:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039099249999999999 2024-09-11T03:40:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129428+0000", 2024-09-11T03:40:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9113000000000001e-05 2024-09-11T03:40:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129515+0000", 2024-09-11T03:40:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6947999999999995e-05 2024-09-11T03:40:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:641 2.a 2:5014e305:::smithi17557870-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:40:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:40:51.893 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 103.742101664, 2024-09-11T03:40:51.893 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011882876000000001, 2024-09-11T03:40:51.893 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.893 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.893 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:51.893 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.893 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.893 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.893 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 641 2024-09-11T03:40:51.893 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.893 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.894 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.894 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:40:51.894 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.894 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.894 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.894 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.894 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:40:51.894 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.894 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.894 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.894 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.895 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:40:51.895 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7599999999999998e-07 2024-09-11T03:40:51.895 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.895 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.895 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.895 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110347+0000", 2024-09-11T03:40:51.895 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.685e-06 2024-09-11T03:40:51.895 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.895 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.895 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.896 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110351+0000", 2024-09-11T03:40:51.896 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4889999999999997e-06 2024-09-11T03:40:51.896 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.896 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.896 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.896 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110354+0000", 2024-09-11T03:40:51.896 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0510000000000001e-06 2024-09-11T03:40:51.896 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.896 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.896 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.896 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110644+0000", 2024-09-11T03:40:51.896 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00028940200000000002 2024-09-11T03:40:51.897 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.897 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.897 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:40:51.897 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110673+0000", 2024-09-11T03:40:51.897 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9833e-05 2024-09-11T03:40:51.897 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.897 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.897 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.897 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122038+0000", 2024-09-11T03:40:51.897 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011364915 2024-09-11T03:40:51.897 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.898 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.898 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.898 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122127+0000", 2024-09-11T03:40:51.898 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8987999999999996e-05 2024-09-11T03:40:51.898 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.898 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.898 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.898 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122227+0000", 2024-09-11T03:40:51.898 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010013699999999999 2024-09-11T03:40:51.898 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.898 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.899 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.899 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.899 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.899 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:752 2.c 2:3c007163:::smithi17557870-35:head [copy-from ver 39] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:40:51.899 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:40:51.899 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 96.843430381999994, 2024-09-11T03:40:51.899 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017596783000000001, 2024-09-11T03:40:51.899 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.899 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.899 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:51.900 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.900 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.900 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.900 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 752 2024-09-11T03:40:51.900 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.900 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.900 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.900 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.900 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:40:51.900 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:40:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009018+0000", 2024-09-11T03:40:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.48e-06 2024-09-11T03:40:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009029+0000", 2024-09-11T03:40:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0377000000000001e-05 2024-09-11T03:40:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009050+0000", 2024-09-11T03:40:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0849e-05 2024-09-11T03:40:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009058+0000", 2024-09-11T03:40:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8019999999999998e-06 2024-09-11T03:40:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009125+0000", 2024-09-11T03:40:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6388000000000002e-05 2024-09-11T03:40:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:40:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009130+0000", 2024-09-11T03:40:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0900000000000004e-06 2024-09-11T03:40:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010518+0000", 2024-09-11T03:40:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013880030000000001 2024-09-11T03:40:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010582+0000", 2024-09-11T03:40:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.368e-05 2024-09-11T03:40:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:40:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.015350+0000", 2024-09-11T03:40:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0047682870000000004 2024-09-11T03:40:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:40:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.019005+0000", 2024-09-11T03:40:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0036549040000000001 2024-09-11T03:40:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.024096+0000", 2024-09-11T03:40:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050917439999999996 2024-09-11T03:40:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026521+0000", 2024-09-11T03:40:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024247399999999999 2024-09-11T03:40:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026559+0000", 2024-09-11T03:40:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8232999999999998e-05 2024-09-11T03:40:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026613+0000", 2024-09-11T03:40:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3205999999999998e-05 2024-09-11T03:40:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.909 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:753 2.c 2:3c007163:::smithi17557870-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:40:51.910 INFO:tasks.rados.rados.2.smithi175.stdout:695: writing smithi17557870-32 from 1617736 to 2327822 tid 2 2024-09-11T03:40:51.910 INFO:tasks.rados.rados.2.smithi175.stdout:696: snap_create 2024-09-11T03:40:51.910 INFO:tasks.rados.rados.2.smithi175.stdout:693: finishing write tid 2 to smithi17557870-24 2024-09-11T03:40:51.910 INFO:tasks.rados.rados.2.smithi175.stdout:693: finishing write tid 3 to smithi17557870-24 2024-09-11T03:40:51.911 INFO:tasks.rados.rados.2.smithi175.stdout:693: finishing write tid 4 to smithi17557870-24 2024-09-11T03:40:51.911 INFO:tasks.rados.rados.2.smithi175.stdout:693: oid 24 updating version 0 to 213 2024-09-11T03:40:51.911 INFO:tasks.rados.rados.2.smithi175.stdout:693: oid 24 updating version 213 to 214 2024-09-11T03:40:51.911 INFO:tasks.rados.rados.2.smithi175.stdout:693: oid 24 version 214 is already newer than 212 2024-09-11T03:40:51.911 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 214 (ObjNum 275 snap 72 seq_num 275) dirty exists 2024-09-11T03:40:51.911 INFO:tasks.rados.rados.2.smithi175.stdout:693: left oid 24 (ObjNum 275 snap 72 seq_num 275) 2024-09-11T03:40:51.911 INFO:tasks.rados.rados.2.smithi175.stdout:695: finishing write tid 1 to smithi17557870-32 2024-09-11T03:40:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:40:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 96.843363295000003, 2024-09-11T03:40:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0094619260000000007, 2024-09-11T03:40:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 753 2024-09-11T03:40:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:40:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:40:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.914 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.914 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:40:51.914 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2100000000000002e-07 2024-09-11T03:40:51.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.914 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.914 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.914 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009085+0000", 2024-09-11T03:40:51.914 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5910000000000001e-06 2024-09-11T03:40:51.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.914 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.914 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.914 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009090+0000", 2024-09-11T03:40:51.914 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5180000000000004e-06 2024-09-11T03:40:51.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.915 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.915 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009094+0000", 2024-09-11T03:40:51.915 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0620000000000001e-06 2024-09-11T03:40:51.915 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.915 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.915 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009178+0000", 2024-09-11T03:40:51.915 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4525999999999997e-05 2024-09-11T03:40:51.915 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.915 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:40:51.915 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009182+0000", 2024-09-11T03:40:51.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4640000000000002e-06 2024-09-11T03:40:51.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.916 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.916 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010677+0000", 2024-09-11T03:40:51.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014957569999999999 2024-09-11T03:40:51.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.916 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:40:51.916 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010706+0000", 2024-09-11T03:40:51.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8873999999999998e-05 2024-09-11T03:40:51.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018385+0000", 2024-09-11T03:40:51.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0076790490000000003 2024-09-11T03:40:51.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018466+0000", 2024-09-11T03:40:51.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0483999999999997e-05 2024-09-11T03:40:51.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018545+0000", 2024-09-11T03:40:51.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9179999999999997e-05 2024-09-11T03:40:51.918 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.918 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.918 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.918 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:839 2.a 2:5014e305:::smithi17557870-46:19 [read 0~3915998,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e146)", 2024-09-11T03:40:51.918 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:22.827744+0000", 2024-09-11T03:40:51.918 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 89.024702153999996, 2024-09-11T03:40:51.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0051572420000000002, 2024-09-11T03:40:51.918 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.918 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.918 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:51.918 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.918 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.919 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.919 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 839 2024-09-11T03:40:51.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.919 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.919 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.919 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827744+0000", 2024-09-11T03:40:51.919 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.919 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.919 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827744+0000", 2024-09-11T03:40:51.919 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.920 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.920 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.920 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827746+0000", 2024-09-11T03:40:51.920 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.198e-06 2024-09-11T03:40:51.920 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.920 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.920 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.920 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827755+0000", 2024-09-11T03:40:51.920 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.789e-06 2024-09-11T03:40:51.920 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.920 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.920 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.920 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827774+0000", 2024-09-11T03:40:51.921 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.891e-05 2024-09-11T03:40:51.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.921 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827795+0000", 2024-09-11T03:40:51.921 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0978000000000001e-05 2024-09-11T03:40:51.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.921 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827865+0000", 2024-09-11T03:40:51.921 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0178999999999993e-05 2024-09-11T03:40:51.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:40:51.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827872+0000", 2024-09-11T03:40:51.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4250000000000003e-06 2024-09-11T03:40:51.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.829088+0000", 2024-09-11T03:40:51.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00121659 2024-09-11T03:40:51.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.829167+0000", 2024-09-11T03:40:51.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8774000000000002e-05 2024-09-11T03:40:51.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.923 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.923 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.832901+0000", 2024-09-11T03:40:51.923 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0037343989999999998 2024-09-11T03:40:51.923 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.924 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.924 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.924 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.924 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.924 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:840 2.a 2:5014e305:::smithi17557870-46:19 [sparse-read 0~2343134] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e146)", 2024-09-11T03:40:51.924 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:22.827818+0000", 2024-09-11T03:40:51.924 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 89.024628673999999, 2024-09-11T03:40:51.924 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0053240099999999997, 2024-09-11T03:40:51.924 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.924 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.925 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:51.925 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.925 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.925 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.925 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 840 2024-09-11T03:40:51.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.925 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.925 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.925 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.925 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827818+0000", 2024-09-11T03:40:51.925 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.926 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.926 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827818+0000", 2024-09-11T03:40:51.926 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.926 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.927 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.927 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827818+0000", 2024-09-11T03:40:51.927 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1600000000000001e-07 2024-09-11T03:40:51.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.927 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.927 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827822+0000", 2024-09-11T03:40:51.927 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3289999999999998e-06 2024-09-11T03:40:51.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.928 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.928 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827833+0000", 2024-09-11T03:40:51.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1558e-05 2024-09-11T03:40:51.928 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.928 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.928 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.929 INFO:tasks.rados.rados.2.smithi175.stdout:695: finishing write tid 2 to smithi17557870-32 2024-09-11T03:40:51.929 INFO:tasks.rados.rados.2.smithi175.stdout:695: finishing write tid 3 to smithi17557870-32 2024-09-11T03:40:51.929 INFO:tasks.rados.rados.2.smithi175.stdout:695: finishing write tid 4 to smithi17557870-32 2024-09-11T03:40:51.929 INFO:tasks.rados.rados.2.smithi175.stdout:695: oid 32 updating version 0 to 195 2024-09-11T03:40:51.929 INFO:tasks.rados.rados.2.smithi175.stdout:695: oid 32 version 195 is already newer than 194 2024-09-11T03:40:51.929 INFO:tasks.rados.rados.2.smithi175.stdout:695: oid 32 updating version 195 to 196 2024-09-11T03:40:51.929 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 196 (ObjNum 276 snap 72 seq_num 276) dirty exists 2024-09-11T03:40:51.930 INFO:tasks.rados.rados.2.smithi175.stdout:695: left oid 32 (ObjNum 276 snap 72 seq_num 276) 2024-09-11T03:40:51.930 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827838+0000", 2024-09-11T03:40:51.930 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2059999999999998e-06 2024-09-11T03:40:51.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.931 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.931 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827935+0000", 2024-09-11T03:40:51.931 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6779000000000003e-05 2024-09-11T03:40:51.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.931 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:40:51.931 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827939+0000", 2024-09-11T03:40:51.932 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3550000000000001e-06 2024-09-11T03:40:51.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.932 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.932 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.832953+0000", 2024-09-11T03:40:51.932 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050147880000000001 2024-09-11T03:40:51.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.932 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.932 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.833035+0000", 2024-09-11T03:40:51.932 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1193999999999995e-05 2024-09-11T03:40:51.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.833142+0000", 2024-09-11T03:40:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000107085 2024-09-11T03:40:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:841 2.a 2:5014e305:::smithi17557870-46:19 [sparse-read 0~2343134] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e146)", 2024-09-11T03:40:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:22.827859+0000", 2024-09-11T03:40:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 89.024586779000003, 2024-09-11T03:40:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0055418159999999998, 2024-09-11T03:40:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 841 2024-09-11T03:40:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827859+0000", 2024-09-11T03:40:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827859+0000", 2024-09-11T03:40:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827860+0000", 2024-09-11T03:40:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5199999999999997e-07 2024-09-11T03:40:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827863+0000", 2024-09-11T03:40:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9699999999999999e-06 2024-09-11T03:40:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827871+0000", 2024-09-11T03:40:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3089999999999996e-06 2024-09-11T03:40:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827877+0000", 2024-09-11T03:40:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7189999999999998e-06 2024-09-11T03:40:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827959+0000", 2024-09-11T03:40:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2321e-05 2024-09-11T03:40:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:40:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.827962+0000", 2024-09-11T03:40:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5689999999999999e-06 2024-09-11T03:40:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.833169+0000", 2024-09-11T03:40:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0052065920000000003 2024-09-11T03:40:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.833246+0000", 2024-09-11T03:40:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7987999999999999e-05 2024-09-11T03:40:51.939 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.939 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.939 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:22.833401+0000", 2024-09-11T03:40:51.939 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015479600000000001 2024-09-11T03:40:51.939 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.939 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.939 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.939 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.939 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:985 2.b e163/95 2:dde92fc6:::smithi17557870-41:head v 163'52, mlcod=158'48)", 2024-09-11T03:40:51.939 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:40:51.940 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 71.919495749999996, 2024-09-11T03:40:51.940 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0092288890000000005, 2024-09-11T03:40:51.940 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.940 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.940 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:51.940 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.940 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.940 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:40:51.940 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:40:51.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:40:51.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7799999999999997e-07 2024-09-11T03:40:51.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935287+0000", 2024-09-11T03:40:51.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023361599999999999 2024-09-11T03:40:51.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935321+0000", 2024-09-11T03:40:51.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3840000000000001e-05 2024-09-11T03:40:51.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935335+0000", 2024-09-11T03:40:51.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4184e-05 2024-09-11T03:40:51.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.943 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.943 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935428+0000", 2024-09-11T03:40:51.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3144000000000001e-05 2024-09-11T03:40:51.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.943 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.943 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935503+0000", 2024-09-11T03:40:51.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4697999999999995e-05 2024-09-11T03:40:51.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:51.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.942117+0000", 2024-09-11T03:40:51.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0066141350000000002 2024-09-11T03:40:51.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.942179+0000", 2024-09-11T03:40:51.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.245e-05 2024-09-11T03:40:51.944 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.944 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.944 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.945 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:990 2.a 2:5014e305:::smithi17557870-46:head [write 638699~541153] snapc 24=[24,23,22,20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-11T03:40:51.945 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:40:51.945 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 71.915514497999993, 2024-09-11T03:40:51.945 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0080995760000000007, 2024-09-11T03:40:51.945 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.945 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.945 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:51.945 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.945 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.946 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.946 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 990 2024-09-11T03:40:51.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.946 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.946 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.946 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:40:51.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.946 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.946 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:40:51.947 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.947 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936935+0000", 2024-09-11T03:40:51.947 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7800000000000001e-06 2024-09-11T03:40:51.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937090+0000", 2024-09-11T03:40:51.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000155857 2024-09-11T03:40:51.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937109+0000", 2024-09-11T03:40:51.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8216e-05 2024-09-11T03:40:51.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937119+0000", 2024-09-11T03:40:51.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0406e-05 2024-09-11T03:40:51.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937195+0000", 2024-09-11T03:40:51.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6372999999999995e-05 2024-09-11T03:40:51.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937287+0000", 2024-09-11T03:40:51.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1434000000000006e-05 2024-09-11T03:40:51.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.950 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:40:51.950 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937454+0000", 2024-09-11T03:40:51.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000167262 2024-09-11T03:40:51.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.950 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:51.950 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.943162+0000", 2024-09-11T03:40:51.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0057082230000000001 2024-09-11T03:40:51.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:51.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.943196+0000", 2024-09-11T03:40:51.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4025e-05 2024-09-11T03:40:51.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:40:51.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.944922+0000", 2024-09-11T03:40:51.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001726102 2024-09-11T03:40:51.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:51.952 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.944977+0000", 2024-09-11T03:40:51.952 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4849999999999998e-05 2024-09-11T03:40:51.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.952 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.952 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.945031+0000", 2024-09-11T03:40:51.952 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4048000000000002e-05 2024-09-11T03:40:51.952 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.952 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.952 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.953 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1175 2.c 2:32ade24e:::smithi17557870-45:head [copy-from ver 130] snapc 2e=[2e,2c,2b,28,26] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:40:51.953 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:40:51.953 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 49.750306606000002, 2024-09-11T03:40:51.953 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015807878000000001, 2024-09-11T03:40:51.953 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.953 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.953 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:51.953 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.953 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.953 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.953 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1175 2024-09-11T03:40:51.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.954 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.954 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:40:51.954 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.954 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:40:51.954 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.955 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.955 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102142+0000", 2024-09-11T03:40:51.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3369999999999998e-06 2024-09-11T03:40:51.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.955 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.955 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102152+0000", 2024-09-11T03:40:51.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8400000000000007e-06 2024-09-11T03:40:51.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.956 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.956 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102172+0000", 2024-09-11T03:40:51.956 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0537e-05 2024-09-11T03:40:51.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.956 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.956 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102183+0000", 2024-09-11T03:40:51.956 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0423e-05 2024-09-11T03:40:51.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.957 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102253+0000", 2024-09-11T03:40:51.957 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0528e-05 2024-09-11T03:40:51.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:40:51.957 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102259+0000", 2024-09-11T03:40:51.957 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1589999999999998e-06 2024-09-11T03:40:51.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103477+0000", 2024-09-11T03:40:51.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012174550000000001 2024-09-11T03:40:51.958 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.958 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.958 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103545+0000", 2024-09-11T03:40:51.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7595999999999995e-05 2024-09-11T03:40:51.958 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.958 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.958 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:40:51.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.110323+0000", 2024-09-11T03:40:51.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0067785730000000004 2024-09-11T03:40:51.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.959 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:40:51.959 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.113411+0000", 2024-09-11T03:40:51.959 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030882769999999999 2024-09-11T03:40:51.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.959 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:51.959 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117742+0000", 2024-09-11T03:40:51.959 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0043301920000000001 2024-09-11T03:40:51.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.960 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:51.960 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117835+0000", 2024-09-11T03:40:51.960 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3848e-05 2024-09-11T03:40:51.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.960 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:51.960 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117916+0000", 2024-09-11T03:40:51.960 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0505000000000003e-05 2024-09-11T03:40:51.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.960 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.961 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117948+0000", 2024-09-11T03:40:51.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1608e-05 2024-09-11T03:40:51.961 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.961 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.961 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.961 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1176 2.c 2:32ade24e:::smithi17557870-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:40:51.961 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:40:51.961 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 49.750242342, 2024-09-11T03:40:51.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010847406, 2024-09-11T03:40:51.961 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.962 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.962 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:40:51.962 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.962 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.962 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.962 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1176 2024-09-11T03:40:51.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.962 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.962 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.962 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:40:51.962 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.963 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.963 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:40:51.963 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.963 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.963 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:40:51.963 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4999999999999998e-07 2024-09-11T03:40:51.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.964 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.964 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.964 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102206+0000", 2024-09-11T03:40:51.964 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9429999999999999e-06 2024-09-11T03:40:51.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.964 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.964 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.964 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102214+0000", 2024-09-11T03:40:51.964 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0399999999999993e-06 2024-09-11T03:40:51.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.964 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.965 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.965 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102220+0000", 2024-09-11T03:40:51.965 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2989999999999996e-06 2024-09-11T03:40:51.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.965 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.965 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.965 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102307+0000", 2024-09-11T03:40:51.965 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7635999999999999e-05 2024-09-11T03:40:51.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.965 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.965 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:40:51.966 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102310+0000", 2024-09-11T03:40:51.966 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.2160000000000002e-06 2024-09-11T03:40:51.966 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.966 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.966 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.966 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103661+0000", 2024-09-11T03:40:51.966 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013507860000000001 2024-09-11T03:40:51.966 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.966 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.966 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:40:51.966 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103686+0000", 2024-09-11T03:40:51.967 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.44e-05 2024-09-11T03:40:51.967 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.967 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.967 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.112884+0000", 2024-09-11T03:40:51.967 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0091986110000000006 2024-09-11T03:40:51.967 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.967 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.967 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.112967+0000", 2024-09-11T03:40:51.967 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2478000000000006e-05 2024-09-11T03:40:51.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.968 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.968 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.968 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.113051+0000", 2024-09-11T03:40:51.968 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4647000000000005e-05 2024-09-11T03:40:51.968 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.968 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.968 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.968 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.969 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1279 2.c 2:3c007163:::smithi17557870-35:head [append 0~0] snapc 33=[32,30,2c,2b] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:40:51.969 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:12.835889+0000", 2024-09-11T03:40:51.969 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 39.016556842, 2024-09-11T03:40:51.969 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0057757520000000003, 2024-09-11T03:40:51.969 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.969 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.969 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:51.969 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.969 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.969 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.969 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1279 2024-09-11T03:40:51.969 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.970 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.970 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.970 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.970 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835889+0000", 2024-09-11T03:40:51.970 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.970 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.970 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.970 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.970 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835889+0000", 2024-09-11T03:40:51.970 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.971 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.971 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.971 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.971 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835891+0000", 2024-09-11T03:40:51.971 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0229999999999999e-06 2024-09-11T03:40:51.971 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.971 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.971 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.971 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835901+0000", 2024-09-11T03:40:51.971 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.781e-06 2024-09-11T03:40:51.971 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.972 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.972 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.972 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835921+0000", 2024-09-11T03:40:51.972 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9408000000000001e-05 2024-09-11T03:40:51.972 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.972 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.972 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.972 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835931+0000", 2024-09-11T03:40:51.972 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0061000000000001e-05 2024-09-11T03:40:51.972 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.972 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.973 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.973 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836022+0000", 2024-09-11T03:40:51.973 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1710000000000001e-05 2024-09-11T03:40:51.973 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.973 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.973 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:40:51.973 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836053+0000", 2024-09-11T03:40:51.973 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0994000000000002e-05 2024-09-11T03:40:51.973 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.973 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.973 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.973 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.838279+0000", 2024-09-11T03:40:51.974 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022257380000000001 2024-09-11T03:40:51.974 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.974 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.974 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.974 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.838374+0000", 2024-09-11T03:40:51.974 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4786999999999999e-05 2024-09-11T03:40:51.974 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.974 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.974 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:40:51.974 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.838603+0000", 2024-09-11T03:40:51.974 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00022876 2024-09-11T03:40:51.975 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.975 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.975 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:40:51.975 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.840801+0000", 2024-09-11T03:40:51.975 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0021986060000000001 2024-09-11T03:40:51.975 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.975 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.975 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:51.975 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.841408+0000", 2024-09-11T03:40:51.975 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00060647800000000003 2024-09-11T03:40:51.975 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.975 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.976 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:51.976 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.841588+0000", 2024-09-11T03:40:51.976 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00018001000000000001 2024-09-11T03:40:51.976 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.976 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.976 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:51.976 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.841631+0000", 2024-09-11T03:40:51.976 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3129e-05 2024-09-11T03:40:51.976 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.976 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.976 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.977 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.841665+0000", 2024-09-11T03:40:51.977 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4267000000000002e-05 2024-09-11T03:40:51.977 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.977 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.977 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.977 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.977 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.977 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1280 2.c 2:3c007163:::smithi17557870-35:head [rollback 2b] snapc 33=[32,30,2c,2b] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:40:51.977 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:12.835951+0000", 2024-09-11T03:40:51.977 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 39.016495130999999, 2024-09-11T03:40:51.977 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0065671339999999996, 2024-09-11T03:40:51.977 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.978 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.978 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:51.978 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.978 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.978 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.978 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1280 2024-09-11T03:40:51.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.978 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.978 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.978 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835951+0000", 2024-09-11T03:40:51.979 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.979 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.979 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.979 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.979 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835951+0000", 2024-09-11T03:40:51.979 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.979 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.979 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.979 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.979 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835952+0000", 2024-09-11T03:40:51.979 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4300000000000003e-07 2024-09-11T03:40:51.980 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.980 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.980 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.980 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835955+0000", 2024-09-11T03:40:51.980 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5099999999999999e-06 2024-09-11T03:40:51.980 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.980 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.980 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.980 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835969+0000", 2024-09-11T03:40:51.980 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.36e-05 2024-09-11T03:40:51.980 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.981 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.981 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.981 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835975+0000", 2024-09-11T03:40:51.981 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3230000000000003e-06 2024-09-11T03:40:51.981 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.981 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.981 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.981 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836209+0000", 2024-09-11T03:40:51.981 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00023352 2024-09-11T03:40:51.981 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.981 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.982 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:40:51.982 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836215+0000", 2024-09-11T03:40:51.982 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5799999999999997e-06 2024-09-11T03:40:51.982 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.982 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.982 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.982 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839733+0000", 2024-09-11T03:40:51.982 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035178369999999998 2024-09-11T03:40:51.982 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.982 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.983 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.983 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839818+0000", 2024-09-11T03:40:51.983 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4962000000000004e-05 2024-09-11T03:40:51.983 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.983 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.983 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:40:51.983 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.840012+0000", 2024-09-11T03:40:51.983 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000193962 2024-09-11T03:40:51.983 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.983 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.984 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:40:51.984 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.840806+0000", 2024-09-11T03:40:51.984 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00079423099999999995 2024-09-11T03:40:51.984 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.984 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.984 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:51.984 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.841732+0000", 2024-09-11T03:40:51.984 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00092522800000000005 2024-09-11T03:40:51.984 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.984 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.984 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:51.985 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.842435+0000", 2024-09-11T03:40:51.985 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00070363399999999997 2024-09-11T03:40:51.985 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.985 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.985 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:51.985 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.842485+0000", 2024-09-11T03:40:51.985 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9580999999999998e-05 2024-09-11T03:40:51.985 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.985 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.985 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.985 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.842518+0000", 2024-09-11T03:40:51.986 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3522999999999997e-05 2024-09-11T03:40:51.986 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.986 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.986 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.986 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.986 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.986 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1281 2.c 2:3c007163:::smithi17557870-35:head [append 0~0] snapc 33=[32,30,2c,2b] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:40:51.987 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:12.836002+0000", 2024-09-11T03:40:51.987 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 39.016444045999997, 2024-09-11T03:40:51.987 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0067871590000000001, 2024-09-11T03:40:51.987 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.987 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.987 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:51.988 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.988 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.988 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.988 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1281 2024-09-11T03:40:51.988 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.988 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.988 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.988 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.988 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836002+0000", 2024-09-11T03:40:51.988 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.988 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.989 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.989 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.989 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836002+0000", 2024-09-11T03:40:51.989 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.989 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.989 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.989 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836003+0000", 2024-09-11T03:40:51.989 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3e-07 2024-09-11T03:40:51.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.989 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.990 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.990 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836021+0000", 2024-09-11T03:40:51.990 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8142999999999999e-05 2024-09-11T03:40:51.990 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.990 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.990 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.990 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836075+0000", 2024-09-11T03:40:51.990 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4398999999999999e-05 2024-09-11T03:40:51.990 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.990 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.990 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.990 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836087+0000", 2024-09-11T03:40:51.991 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.166e-05 2024-09-11T03:40:51.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.991 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.991 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836268+0000", 2024-09-11T03:40:51.991 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00018073399999999999 2024-09-11T03:40:51.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.991 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:40:51.991 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836275+0000", 2024-09-11T03:40:51.991 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9639999999999997e-06 2024-09-11T03:40:51.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.992 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.992 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:51.992 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.840402+0000", 2024-09-11T03:40:51.992 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0041267789999999997 2024-09-11T03:40:51.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.992 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.992 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:51.992 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.840480+0000", 2024-09-11T03:40:51.993 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8801000000000007e-05 2024-09-11T03:40:51.993 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.993 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.993 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:40:51.993 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.840597+0000", 2024-09-11T03:40:51.993 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000116137 2024-09-11T03:40:51.993 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.993 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.993 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:40:51.993 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.840948+0000", 2024-09-11T03:40:51.993 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00035135599999999999 2024-09-11T03:40:51.993 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.994 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.994 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:51.994 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.841775+0000", 2024-09-11T03:40:51.994 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00082670300000000003 2024-09-11T03:40:51.994 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.994 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.994 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:51.994 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.842655+0000", 2024-09-11T03:40:51.994 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00088039100000000001 2024-09-11T03:40:51.994 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.994 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.994 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:51.995 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.842741+0000", 2024-09-11T03:40:51.995 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5958000000000006e-05 2024-09-11T03:40:51.995 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.995 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.995 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:51.995 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.842789+0000", 2024-09-11T03:40:51.995 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8403999999999999e-05 2024-09-11T03:40:51.995 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.995 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:51.995 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:51.995 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.996 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.996 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1507 2.c 2:32ade24e:::smithi17557870-45:head [write 749645~774105,stat] snapc 3b=[3a,38,37,35] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-09-11T03:40:51.996 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:28.901035+0000", 2024-09-11T03:40:51.996 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 22.951411105999998, 2024-09-11T03:40:51.996 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0053564779999999996, 2024-09-11T03:40:51.996 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:51.996 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:51.996 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:51.996 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:51.996 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:51.996 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:51.996 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1507 2024-09-11T03:40:51.997 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.997 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:51.997 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.997 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:51.997 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.901035+0000", 2024-09-11T03:40:51.997 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.997 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.997 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.997 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:51.997 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.901035+0000", 2024-09-11T03:40:51.997 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:51.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.998 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:51.998 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.901037+0000", 2024-09-11T03:40:51.998 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6789999999999999e-06 2024-09-11T03:40:51.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.998 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:51.998 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.901505+0000", 2024-09-11T03:40:51.998 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00046822500000000001 2024-09-11T03:40:51.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.999 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:51.999 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.901520+0000", 2024-09-11T03:40:51.999 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4858000000000001e-05 2024-09-11T03:40:51.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.999 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.999 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:51.999 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.901527+0000", 2024-09-11T03:40:51.999 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7490000000000001e-06 2024-09-11T03:40:51.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:51.999 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:51.999 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:52.000 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.901584+0000", 2024-09-11T03:40:52.000 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7185999999999998e-05 2024-09-11T03:40:52.000 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.000 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.000 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:40:52.000 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.901589+0000", 2024-09-11T03:40:52.000 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9509999999999997e-06 2024-09-11T03:40:52.000 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.000 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.000 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:52.000 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.903015+0000", 2024-09-11T03:40:52.001 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001425983 2024-09-11T03:40:52.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.001 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.001 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:52.001 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.903134+0000", 2024-09-11T03:40:52.001 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000119454 2024-09-11T03:40:52.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.001 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.001 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:40:52.001 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.903320+0000", 2024-09-11T03:40:52.001 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000185887 2024-09-11T03:40:52.002 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.002 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.002 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:40:52.002 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.904818+0000", 2024-09-11T03:40:52.002 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014982089999999999 2024-09-11T03:40:52.002 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.002 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.002 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:52.002 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.906174+0000", 2024-09-11T03:40:52.002 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013556429999999999 2024-09-11T03:40:52.002 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.003 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.003 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:52.003 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.906272+0000", 2024-09-11T03:40:52.003 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7968999999999997e-05 2024-09-11T03:40:52.003 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.003 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.003 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:52.003 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.906341+0000", 2024-09-11T03:40:52.003 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9172999999999997e-05 2024-09-11T03:40:52.003 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.003 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.004 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:52.004 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.906392+0000", 2024-09-11T03:40:52.004 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0512e-05 2024-09-11T03:40:52.004 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:52.004 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:52.004 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:52.004 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.004 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.004 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1719 2.c 2:327469ff:::smithi17557870-49:head [write 487469~563838,stat] snapc 45=[45,44,43,3e,3d,3c,3a] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-11T03:40:52.004 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:45.859743+0000", 2024-09-11T03:40:52.005 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 5.9927035640000001, 2024-09-11T03:40:52.005 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0078899869999999993, 2024-09-11T03:40:52.005 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:40:52.005 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:40:52.005 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:40:52.005 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:40:52.005 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:40:52.005 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:40:52.005 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1719 2024-09-11T03:40:52.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.005 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:40:52.006 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.006 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:40:52.006 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.859743+0000", 2024-09-11T03:40:52.006 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:52.006 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.006 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.006 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:40:52.006 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.859743+0000", 2024-09-11T03:40:52.006 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:40:52.006 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.006 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.007 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:40:52.007 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.859745+0000", 2024-09-11T03:40:52.007 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.712e-06 2024-09-11T03:40:52.007 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.007 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.007 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:40:52.007 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860066+0000", 2024-09-11T03:40:52.007 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00032013499999999998 2024-09-11T03:40:52.007 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.007 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.007 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:40:52.008 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860090+0000", 2024-09-11T03:40:52.008 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4590999999999999e-05 2024-09-11T03:40:52.008 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.008 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.008 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:40:52.008 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860102+0000", 2024-09-11T03:40:52.008 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2269999999999999e-05 2024-09-11T03:40:52.008 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.008 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.008 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:52.008 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860167+0000", 2024-09-11T03:40:52.009 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4806000000000003e-05 2024-09-11T03:40:52.009 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.009 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.009 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:40:52.009 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860172+0000", 2024-09-11T03:40:52.009 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7219999999999999e-06 2024-09-11T03:40:52.009 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.009 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.009 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:40:52.009 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.862974+0000", 2024-09-11T03:40:52.009 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0028017480000000002 2024-09-11T03:40:52.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.010 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.010 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:40:52.010 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.863071+0000", 2024-09-11T03:40:52.010 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7738000000000005e-05 2024-09-11T03:40:52.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.010 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.010 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:40:52.010 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.863253+0000", 2024-09-11T03:40:52.010 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00018170999999999999 2024-09-11T03:40:52.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.010 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.011 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:40:52.011 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.864451+0000", 2024-09-11T03:40:52.011 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00119782 2024-09-11T03:40:52.011 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.011 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.011 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:52.011 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.865698+0000", 2024-09-11T03:40:52.011 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001246973 2024-09-11T03:40:52.011 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.011 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.011 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:40:52.011 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.867563+0000", 2024-09-11T03:40:52.012 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001865189 2024-09-11T03:40:52.012 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.012 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.012 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:40:52.012 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.867601+0000", 2024-09-11T03:40:52.012 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8333e-05 2024-09-11T03:40:52.012 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:40:52.012 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:40:52.012 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:40:52.012 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.867633+0000", 2024-09-11T03:40:52.012 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1239999999999999e-05 2024-09-11T03:40:52.013 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:52.013 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:52.013 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:52.013 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:40:52.013 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:40:52.013 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:40:52.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:52.014 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:52.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:52.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:52.142 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:52.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:52.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:51 smithi084 ceph-mon[27254]: pgmap v505: 47 pgs: 47 active+clean; 187 MiB data, 693 MiB used, 893 GiB / 894 GiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 8 op/s 2024-09-11T03:40:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:51 smithi084 ceph-mon[27254]: osdmap e234: 12 total, 12 up, 10 in 2024-09-11T03:40:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:51 smithi084 ceph-mon[27254]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:40:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:51 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'62260'}"} : dispatch 2024-09-11T03:40:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3273644265' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:40:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:51 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:51 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'62260'}"}]': finished 2024-09-11T03:40:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3273644265' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:40:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:51 smithi084 ceph-mon[27254]: osdmap e235: 12 total, 12 up, 10 in 2024-09-11T03:40:52.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:51 smithi121 ceph-mon[30729]: pgmap v505: 47 pgs: 47 active+clean; 187 MiB data, 693 MiB used, 893 GiB / 894 GiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 8 op/s 2024-09-11T03:40:52.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:51 smithi121 ceph-mon[30729]: osdmap e234: 12 total, 12 up, 10 in 2024-09-11T03:40:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:51 smithi121 ceph-mon[30729]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:40:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:51 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'62260'}"} : dispatch 2024-09-11T03:40:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3273644265' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:40:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:51 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:51 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'62260'}"}]': finished 2024-09-11T03:40:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3273644265' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:40:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:51 smithi121 ceph-mon[30729]: osdmap e235: 12 total, 12 up, 10 in 2024-09-11T03:40:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:51 smithi175 ceph-mon[30894]: pgmap v505: 47 pgs: 47 active+clean; 187 MiB data, 693 MiB used, 893 GiB / 894 GiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 8 op/s 2024-09-11T03:40:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:51 smithi175 ceph-mon[30894]: osdmap e234: 12 total, 12 up, 10 in 2024-09-11T03:40:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:51 smithi175 ceph-mon[30894]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:40:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:51 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'62260'}"} : dispatch 2024-09-11T03:40:52.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3273644265' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:40:52.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:51 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:52.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:51 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'62260'}"}]': finished 2024-09-11T03:40:52.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3273644265' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:40:52.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:51 smithi175 ceph-mon[30894]: osdmap e235: 12 total, 12 up, 10 in 2024-09-11T03:40:52.244 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:52.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:52.266 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:52.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:52.368 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:52.391 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:52.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:52.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:52.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:40:52.494 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:52.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:52.545 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:52.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:52.646 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:52.668 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:52.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:52.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:52.696 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T03:40:52.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:52.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:52.798 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:52.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:52.799 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-11T03:40:52.799 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-11T03:40:52.849 INFO:tasks.rados.rados.2.smithi175.stdout:693: done (2 left) 2024-09-11T03:40:52.849 INFO:tasks.rados.rados.2.smithi175.stdout:695: done (1 left) 2024-09-11T03:40:52.849 INFO:tasks.rados.rados.2.smithi175.stdout:696: done (0 left) 2024-09-11T03:40:52.850 INFO:tasks.rados.rados.2.smithi175.stdout:697: snap_remove snap 65 2024-09-11T03:40:52.899 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:52.921 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:52.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:53.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:53.048 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:53.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:53.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:53.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:53.172 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:53.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:53.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:53.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:53.286 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:53.300 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:53.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:53.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:53.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:53.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:53.424 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:53.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:53.525 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:53.548 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:53.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:53.552 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:53.653 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:53.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:53.680 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:53.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:53.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:53.806 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:53.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:53.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:53.815 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:40:53.816 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:40:53.829 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:40:53.830 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:40:53.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:40:53.848 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:40:53.850 INFO:tasks.rados.rados.2.smithi175.stdout:697: done (0 left) 2024-09-11T03:40:53.850 INFO:tasks.rados.rados.2.smithi175.stdout:698: snap_remove snap 72 2024-09-11T03:40:53.870 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:40:53.870 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:40:53.876 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:53.876 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:53.894 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:40:53.894 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:40:53.908 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:53.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:40:53.913 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:40:53.931 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:40:53.931 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:40:53.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:53.935 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:53.935 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:53.946 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:40:53.946 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:40:53.963 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:40:53.963 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:40:53.980 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:40:53.981 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:40:53.997 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:40:53.998 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:40:54.015 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:40:54.015 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:40:54.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:54.061 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:54.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:54.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:54.103 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:53 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:40:54.104 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:53 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:40:54.104 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:53 smithi084 ceph-mon[27254]: pgmap v508: 47 pgs: 47 active+clean; 191 MiB data, 705 MiB used, 893 GiB / 894 GiB avail; 5.7 MiB/s rd, 955 KiB/s wr, 8 op/s 2024-09-11T03:40:54.104 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:53 smithi084 ceph-mon[27254]: osdmap e236: 12 total, 12 up, 10 in 2024-09-11T03:40:54.104 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:53 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2073573906' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:54.163 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:54.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:53 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:40:54.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:53 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:40:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:53 smithi121 ceph-mon[30729]: pgmap v508: 47 pgs: 47 active+clean; 191 MiB data, 705 MiB used, 893 GiB / 894 GiB avail; 5.7 MiB/s rd, 955 KiB/s wr, 8 op/s 2024-09-11T03:40:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:53 smithi121 ceph-mon[30729]: osdmap e236: 12 total, 12 up, 10 in 2024-09-11T03:40:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:53 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2073573906' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:54.187 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:54.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:54.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:53 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:40:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:53 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:40:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:53 smithi175 ceph-mon[30894]: pgmap v508: 47 pgs: 47 active+clean; 191 MiB data, 705 MiB used, 893 GiB / 894 GiB avail; 5.7 MiB/s rd, 955 KiB/s wr, 8 op/s 2024-09-11T03:40:54.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:53 smithi175 ceph-mon[30894]: osdmap e236: 12 total, 12 up, 10 in 2024-09-11T03:40:54.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:53 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2073573906' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:54.289 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:54.313 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:54.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:54.415 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:54.442 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:54.442 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:54.543 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:54.568 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:54.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:54.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:54.669 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:54.695 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:54.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:54.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:54.797 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:54.819 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:54.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:54.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:54.861 INFO:tasks.rados.rados.2.smithi175.stdout:698: done (0 left) 2024-09-11T03:40:54.861 INFO:tasks.rados.rados.2.smithi175.stdout:699: read oid 17 snap -1 2024-09-11T03:40:54.861 INFO:tasks.rados.rados.2.smithi175.stdout:699: expect (ObjNum 242 snap 58 seq_num 242) 2024-09-11T03:40:54.861 INFO:tasks.rados.rados.2.smithi175.stdout:700: copy_from oid 6 from oid 25 current snap is 73 2024-09-11T03:40:54.861 INFO:tasks.rados.rados.2.smithi175.stdout:701: copy_from oid 22 from oid 50 current snap is 73 2024-09-11T03:40:54.862 INFO:tasks.rados.rados.2.smithi175.stdout:702: rollback oid 41 current snap is 73 2024-09-11T03:40:54.862 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 41 to 69 2024-09-11T03:40:54.862 INFO:tasks.rados.rados.2.smithi175.stdout:703: write (excl) oid 12 current snap is 73 2024-09-11T03:40:54.862 INFO:tasks.rados.rados.2.smithi175.stdout:703: seq_num 277 ranges {655539=632681,2076218=570097} 2024-09-11T03:40:54.868 INFO:tasks.rados.rados.2.smithi175.stdout:703: writing smithi17557870-12 from 655539 to 1288220 tid 1 2024-09-11T03:40:54.876 INFO:tasks.rados.rados.2.smithi175.stdout:703: writing smithi17557870-12 from 2076218 to 2646315 tid 2 2024-09-11T03:40:54.876 INFO:tasks.rados.rados.2.smithi175.stdout:704: write (excl) oid 43 current snap is 73 2024-09-11T03:40:54.876 INFO:tasks.rados.rados.2.smithi175.stdout:701: finishing copy_from to smithi17557870-22 2024-09-11T03:40:54.876 INFO:tasks.rados.rados.2.smithi175.stdout:701: got expected ENOENT (src dne) 2024-09-11T03:40:54.876 INFO:tasks.rados.rados.2.smithi175.stdout:704: seq_num 278 ranges {672346=707930,2134700=537646} 2024-09-11T03:40:54.883 INFO:tasks.rados.rados.2.smithi175.stdout:704: writing smithi17557870-43 from 672346 to 1380276 tid 1 2024-09-11T03:40:54.891 INFO:tasks.rados.rados.2.smithi175.stdout:704: writing smithi17557870-43 from 2134700 to 2672346 tid 2 2024-09-11T03:40:54.891 INFO:tasks.rados.rados.2.smithi175.stdout:705: snap_remove snap 60 2024-09-11T03:40:54.921 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:54.943 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:54.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:54.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:54.990 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:54 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:40:54.991 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:54 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:40:54.991 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:54 smithi084 ceph-mon[27254]: osdmap e237: 12 total, 12 up, 10 in 2024-09-11T03:40:55.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:55.069 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:55.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:55.171 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:40:55.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:54 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:40:55.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:54 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:40:55.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:54 smithi121 ceph-mon[30729]: osdmap e237: 12 total, 12 up, 10 in 2024-09-11T03:40:55.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:55.194 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:55.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:40:55.207 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:55.207 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:55.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:54 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:40:55.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:54 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:40:55.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:54 smithi175 ceph-mon[30894]: osdmap e237: 12 total, 12 up, 10 in 2024-09-11T03:40:55.286 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:40:55.295 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:55.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:55.352 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:55.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:55.453 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:55.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:55.481 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:55.481 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:55.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:55.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:55.610 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:55.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:55.712 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:55.737 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:55.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:55.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:55.824 INFO:teuthology.orchestra.run.smithi084.stdout:moved 0 / 141 (0%) 2024-09-11T03:40:55.824 INFO:teuthology.orchestra.run.smithi084.stdout:avg 14.1 2024-09-11T03:40:55.824 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 2.34308 -> 2.34308 (expected baseline 3.5623) 2024-09-11T03:40:55.824 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.5 with 11 -> 11 pgs (0.780142 -> 0.780142 * mean) 2024-09-11T03:40:55.824 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.7 with 19 -> 19 pgs (1.34752 -> 1.34752 * mean) 2024-09-11T03:40:55.824 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:40:55.824 INFO:teuthology.orchestra.run.smithi084.stdout:oload 1000 2024-09-11T03:40:55.824 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 1 2024-09-11T03:40:55.825 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2024-09-11T03:40:55.825 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0008 2024-09-11T03:40:55.825 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0077 2024-09-11T03:40:55.825 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2024-09-11T03:40:55.838 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:55.861 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:55.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:55.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:55.865 INFO:tasks.rados.rados.2.smithi175.stdout:705: done (6 left) 2024-09-11T03:40:55.865 INFO:tasks.rados.rados.2.smithi175.stdout:701: finishing copy_from racing read to smithi17557870-22 2024-09-11T03:40:55.865 INFO:tasks.rados.rados.2.smithi175.stdout:702: finishing rollback tid 0 to smithi17557870-41 2024-09-11T03:40:55.865 INFO:tasks.rados.rados.2.smithi175.stdout:702: finishing rollback tid 1 to smithi17557870-41 2024-09-11T03:40:55.865 INFO:tasks.rados.rados.2.smithi175.stdout:702: finishing rollback tid 2 to smithi17557870-41 2024-09-11T03:40:55.865 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 204 (ObjNum 221 snap 53 seq_num 221) dirty exists 2024-09-11T03:40:55.865 INFO:tasks.rados.rados.2.smithi175.stdout:699: expect (ObjNum 242 snap 58 seq_num 242) 2024-09-11T03:40:55.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:55 smithi084 ceph-mon[27254]: pgmap v511: 47 pgs: 47 active+clean; 192 MiB data, 709 MiB used, 893 GiB / 894 GiB avail; 5.7 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-11T03:40:55.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:55 smithi084 ceph-mon[27254]: osdmap e238: 12 total, 12 up, 10 in 2024-09-11T03:40:55.931 INFO:tasks.rados.rados.2.smithi175.stdout:703: finishing write tid 1 to smithi17557870-12 2024-09-11T03:40:55.931 INFO:tasks.rados.rados.2.smithi175.stdout:700: finishing copy_from racing read to smithi17557870-6 2024-09-11T03:40:55.931 INFO:tasks.rados.rados.2.smithi175.stdout:700: finishing copy_from to smithi17557870-6 2024-09-11T03:40:55.931 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 29 (ObjNum 225 snap 53 seq_num 225) dirty exists 2024-09-11T03:40:55.931 INFO:tasks.rados.rados.2.smithi175.stdout:703: finishing write tid 2 to smithi17557870-12 2024-09-11T03:40:55.931 INFO:tasks.rados.rados.2.smithi175.stdout:703: finishing write tid 3 to smithi17557870-12 2024-09-11T03:40:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:703: finishing write tid 4 to smithi17557870-12 2024-09-11T03:40:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:703: oid 12 updating version 0 to 222 2024-09-11T03:40:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:703: oid 12 version 222 is already newer than 221 2024-09-11T03:40:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:703: oid 12 updating version 222 to 223 2024-09-11T03:40:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 223 (ObjNum 277 snap 73 seq_num 277) dirty exists 2024-09-11T03:40:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:703: left oid 12 (ObjNum 277 snap 73 seq_num 277) 2024-09-11T03:40:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:704: finishing write tid 1 to smithi17557870-43 2024-09-11T03:40:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:704: finishing write tid 2 to smithi17557870-43 2024-09-11T03:40:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:706: read oid 33 snap 70 2024-09-11T03:40:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:706: expect (ObjNum 171 snap 39 seq_num 171) 2024-09-11T03:40:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:699: done (6 left) 2024-09-11T03:40:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:700: done (5 left) 2024-09-11T03:40:55.933 INFO:tasks.rados.rados.2.smithi175.stdout:701: done (4 left) 2024-09-11T03:40:55.933 INFO:tasks.rados.rados.2.smithi175.stdout:702: done (3 left) 2024-09-11T03:40:55.933 INFO:tasks.rados.rados.2.smithi175.stdout:703: done (2 left) 2024-09-11T03:40:55.933 INFO:tasks.rados.rados.2.smithi175.stdout:707: snap_remove snap 69 2024-09-11T03:40:55.963 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:55.989 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:55.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:55.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:56.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:56.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:56.113 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:56.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:56.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:55 smithi121 ceph-mon[30729]: pgmap v511: 47 pgs: 47 active+clean; 192 MiB data, 709 MiB used, 893 GiB / 894 GiB avail; 5.7 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-11T03:40:56.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:55 smithi121 ceph-mon[30729]: osdmap e238: 12 total, 12 up, 10 in 2024-09-11T03:40:56.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:55 smithi175 ceph-mon[30894]: pgmap v511: 47 pgs: 47 active+clean; 192 MiB data, 709 MiB used, 893 GiB / 894 GiB avail; 5.7 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-11T03:40:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:55 smithi175 ceph-mon[30894]: osdmap e238: 12 total, 12 up, 10 in 2024-09-11T03:40:56.240 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:56.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:56.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:56.266 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T03:40:56.342 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:56.368 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:56.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:56.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:56.470 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:40:56.495 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:56.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:56.496 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:40:56.597 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:56.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:56.623 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:56.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:56.669 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-11T03:40:56.669 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight-by-utilization 110 0.05 2024-09-11T03:40:56.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:56.742 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:40:56.750 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:56.751 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:56.751 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:56.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:56.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:56.849 INFO:tasks.rados.rados.2.smithi175.stdout:704: finishing write tid 3 to smithi17557870-43 2024-09-11T03:40:56.849 INFO:tasks.rados.rados.2.smithi175.stdout:704: finishing write tid 4 to smithi17557870-43 2024-09-11T03:40:56.849 INFO:tasks.rados.rados.2.smithi175.stdout:704: oid 43 updating version 0 to 121 2024-09-11T03:40:56.849 INFO:tasks.rados.rados.2.smithi175.stdout:704: oid 43 version 121 is already newer than 120 2024-09-11T03:40:56.849 INFO:tasks.rados.rados.2.smithi175.stdout:704: oid 43 version 121 is already newer than 119 2024-09-11T03:40:56.849 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 121 (ObjNum 278 snap 73 seq_num 278) dirty exists 2024-09-11T03:40:56.849 INFO:tasks.rados.rados.2.smithi175.stdout:704: left oid 43 (ObjNum 278 snap 73 seq_num 278) 2024-09-11T03:40:56.849 INFO:tasks.rados.rados.2.smithi175.stdout:706: expect (ObjNum 171 snap 39 seq_num 171) 2024-09-11T03:40:56.853 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:56.877 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:56.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:56.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:56.979 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:57.003 INFO:tasks.rados.rados.2.smithi175.stdout:704: done (2 left) 2024-09-11T03:40:57.003 INFO:tasks.rados.rados.2.smithi175.stdout:706: done (1 left) 2024-09-11T03:40:57.003 INFO:tasks.rados.rados.2.smithi175.stdout:707: done (0 left) 2024-09-11T03:40:57.003 INFO:tasks.rados.rados.2.smithi175.stdout:708: snap_remove snap 56 2024-09-11T03:40:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:57.006 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:57.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:57.107 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:57.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:56 smithi084 ceph-mon[27254]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:40:57.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:56 smithi084 ceph-mon[27254]: osdmap e239: 12 total, 12 up, 10 in 2024-09-11T03:40:57.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:56 smithi084 ceph-mon[27254]: osdmap e240: 12 total, 12 up, 10 in 2024-09-11T03:40:57.129 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:57.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:57.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:57.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:56 smithi121 ceph-mon[30729]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:40:57.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:56 smithi121 ceph-mon[30729]: osdmap e239: 12 total, 12 up, 10 in 2024-09-11T03:40:57.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:56 smithi121 ceph-mon[30729]: osdmap e240: 12 total, 12 up, 10 in 2024-09-11T03:40:57.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:56 smithi175 ceph-mon[30894]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:40:57.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:56 smithi175 ceph-mon[30894]: osdmap e239: 12 total, 12 up, 10 in 2024-09-11T03:40:57.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:56 smithi175 ceph-mon[30894]: osdmap e240: 12 total, 12 up, 10 in 2024-09-11T03:40:57.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:57.254 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:57.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:57.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:57.355 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:57.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:57.377 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:57.377 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:57.478 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:57.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:57.500 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:57.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:57.602 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:40:57.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:57.624 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:57.625 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:40:57.725 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:57.747 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:57.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:57.748 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:57.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:57.867 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:40:57.867 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:40:57.879 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:40:57.884 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:57.887 INFO:tasks.rados.rados.2.smithi175.stdout:708: done (0 left) 2024-09-11T03:40:57.887 INFO:tasks.rados.rados.2.smithi175.stdout:709: write (excl) oid 28 current snap is 73 2024-09-11T03:40:57.888 INFO:tasks.rados.rados.2.smithi175.stdout:709: seq_num 279 ranges {689153=783179,2076829=457413} 2024-09-11T03:40:57.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:57.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:57.893 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:57 smithi084 ceph-mon[27254]: pgmap v514: 47 pgs: 47 active+clean; 203 MiB data, 720 MiB used, 893 GiB / 894 GiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 8 op/s 2024-09-11T03:40:57.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:57 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2246559281' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:40:57.895 INFO:tasks.rados.rados.2.smithi175.stdout:709: writing smithi17557870-28 from 689153 to 1472332 tid 1 2024-09-11T03:40:57.902 INFO:tasks.rados.rados.2.smithi175.stdout:709: writing smithi17557870-28 from 2076829 to 2534242 tid 2 2024-09-11T03:40:57.902 INFO:tasks.rados.rados.2.smithi175.stdout:710: rollback oid 25 current snap is 73 2024-09-11T03:40:57.902 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 25 to 59 2024-09-11T03:40:57.902 INFO:tasks.rados.rados.2.smithi175.stdout:709: finishing write tid 1 to smithi17557870-28 2024-09-11T03:40:57.902 INFO:tasks.rados.rados.2.smithi175.stdout:711: snap_remove snap 70 2024-09-11T03:40:57.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:58.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:58.018 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:58.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:58.118 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:58.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:58.141 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:58.141 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:58.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:57 smithi121 ceph-mon[30729]: pgmap v514: 47 pgs: 47 active+clean; 203 MiB data, 720 MiB used, 893 GiB / 894 GiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 8 op/s 2024-09-11T03:40:58.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:57 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2246559281' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:40:58.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:57 smithi175 ceph-mon[30894]: pgmap v514: 47 pgs: 47 active+clean; 203 MiB data, 720 MiB used, 893 GiB / 894 GiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 8 op/s 2024-09-11T03:40:58.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:57 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2246559281' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:40:58.242 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:40:58.263 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:58.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:58.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:40:58.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:58.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:58.392 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:58.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:58.492 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:58.517 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:58.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:58.620 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:40:58.642 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:58.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:58.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:40:58.666 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T03:40:58.743 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:58.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:58.766 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:58.766 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:58.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:40:58.885 INFO:tasks.rados.rados.2.smithi175.stdout:711: done (2 left) 2024-09-11T03:40:58.886 INFO:tasks.rados.rados.2.smithi175.stdout:712: snap_remove snap 67 2024-09-11T03:40:58.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:58 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:58.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:58 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2246559281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:40:58.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:58 smithi084 ceph-mon[27254]: osdmap e241: 12 total, 12 up, 10 in 2024-09-11T03:40:58.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:58 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:40:58.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:58 smithi084 ceph-mon[27254]: 2.0 scrub ok 2024-09-11T03:40:58.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:58.895 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:58.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:40:58.996 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:40:59.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:59.020 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:40:59.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:40:59.120 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:59.120 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:59.122 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:40:59.148 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:59.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:59.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:40:59.155 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:40:59.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:58 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:59.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:58 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2246559281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:40:59.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:58 smithi121 ceph-mon[30729]: osdmap e241: 12 total, 12 up, 10 in 2024-09-11T03:40:59.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:58 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:40:59.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:58 smithi121 ceph-mon[30729]: 2.0 scrub ok 2024-09-11T03:40:59.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:58 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:40:59.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:58 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2246559281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:40:59.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:58 smithi175 ceph-mon[30894]: osdmap e241: 12 total, 12 up, 10 in 2024-09-11T03:40:59.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:58 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:40:59.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:58 smithi175 ceph-mon[30894]: 2.0 scrub ok 2024-09-11T03:40:59.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:59.275 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:59.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:59.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:59.377 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:59.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:59.399 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:59.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:59.500 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:40:59.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:59.528 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:59.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:40:59.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:40:59.654 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:59.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:40:59.757 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:40:59.779 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:40:59.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:59.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:40:59.871 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:40:59.871 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:40:59.871 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:40:59.871 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:40:59.871 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:40:59.871 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:40:59.879 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:40:59.880 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:40:59.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:40:59.902 INFO:tasks.rados.rados.2.smithi175.stdout:712: done (2 left) 2024-09-11T03:40:59.902 INFO:tasks.rados.rados.2.smithi175.stdout:713: copy_from oid 10 from oid 6 current snap is 73 2024-09-11T03:40:59.902 INFO:tasks.rados.rados.2.smithi175.stdout:710: finishing rollback tid 0 to smithi17557870-25 2024-09-11T03:40:59.902 INFO:tasks.rados.rados.2.smithi175.stdout:714: read oid 13 snap 71 2024-09-11T03:40:59.902 INFO:tasks.rados.rados.2.smithi175.stdout:714: expect (ObjNum 257 snap 64 seq_num 257) 2024-09-11T03:40:59.902 INFO:tasks.rados.rados.2.smithi175.stdout:715: read oid 22 snap 66 2024-09-11T03:40:59.902 INFO:tasks.rados.rados.2.smithi175.stdout:715: expect (ObjNum 194 snap 47 seq_num 194) 2024-09-11T03:40:59.903 INFO:tasks.rados.rados.2.smithi175.stdout:716: copy_from oid 32 from oid 9 current snap is 73 2024-09-11T03:40:59.903 INFO:tasks.rados.rados.2.smithi175.stdout:709: finishing write tid 2 to smithi17557870-28 2024-09-11T03:40:59.903 INFO:tasks.rados.rados.2.smithi175.stdout:710: finishing rollback tid 1 to smithi17557870-25 2024-09-11T03:40:59.903 INFO:tasks.rados.rados.2.smithi175.stdout:709: finishing write tid 3 to smithi17557870-28 2024-09-11T03:40:59.903 INFO:tasks.rados.rados.2.smithi175.stdout:710: finishing rollback tid 2 to smithi17557870-25 2024-09-11T03:40:59.903 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 33 (ObjNum 225 snap 53 seq_num 225) dirty exists 2024-09-11T03:40:59.903 INFO:tasks.rados.rados.2.smithi175.stdout:709: finishing write tid 4 to smithi17557870-28 2024-09-11T03:40:59.903 INFO:tasks.rados.rados.2.smithi175.stdout:709: oid 28 updating version 0 to 276 2024-09-11T03:40:59.903 INFO:tasks.rados.rados.2.smithi175.stdout:709: oid 28 updating version 276 to 277 2024-09-11T03:40:59.903 INFO:tasks.rados.rados.2.smithi175.stdout:709: oid 28 version 277 is already newer than 275 2024-09-11T03:40:59.903 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 277 (ObjNum 279 snap 73 seq_num 279) dirty exists 2024-09-11T03:40:59.903 INFO:tasks.rados.rados.2.smithi175.stdout:709: left oid 28 (ObjNum 279 snap 73 seq_num 279) 2024-09-11T03:40:59.903 INFO:tasks.rados.rados.2.smithi175.stdout:709: done (5 left) 2024-09-11T03:40:59.904 INFO:tasks.rados.rados.2.smithi175.stdout:710: done (4 left) 2024-09-11T03:40:59.904 INFO:tasks.rados.rados.2.smithi175.stdout:717: read oid 27 snap -1 2024-09-11T03:40:59.904 INFO:tasks.rados.rados.2.smithi175.stdout:717: expect (ObjNum 241 snap 58 seq_num 241) 2024-09-11T03:40:59.904 INFO:tasks.rados.rados.2.smithi175.stdout:718: read oid 38 snap 59 2024-09-11T03:40:59.904 INFO:tasks.rados.rados.2.smithi175.stdout:718: expect (ObjNum 101 snap 16 seq_num 101) 2024-09-11T03:40:59.904 INFO:tasks.rados.rados.2.smithi175.stdout:719: rollback oid 8 current snap is 73 2024-09-11T03:40:59.904 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 8 to 59 2024-09-11T03:40:59.904 INFO:tasks.rados.rados.2.smithi175.stdout:720: rollback oid 47 current snap is 73 2024-09-11T03:40:59.904 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 47 to 66 2024-09-11T03:40:59.904 INFO:tasks.rados.rados.2.smithi175.stdout:721: rollback oid 24 current snap is 73 2024-09-11T03:40:59.904 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 66 2024-09-11T03:40:59.904 INFO:tasks.rados.rados.2.smithi175.stdout:722: copy_from oid 39 from oid 20 current snap is 73 2024-09-11T03:40:59.904 INFO:tasks.rados.rados.2.smithi175.stdout:723: write (excl) oid 29 current snap is 73 2024-09-11T03:40:59.905 INFO:tasks.rados.rados.2.smithi175.stdout:723: seq_num 280 ranges {705960=458428,1735311=717424} 2024-09-11T03:40:59.905 INFO:tasks.rados.rados.2.smithi175.stdout:723: writing smithi17557870-29 from 705960 to 1164388 tid 1 2024-09-11T03:40:59.905 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:40:59.906 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:40:59.909 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:40:59.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:40:59.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:40:59.912 INFO:tasks.rados.rados.2.smithi175.stdout:723: writing smithi17557870-29 from 1735311 to 2452735 tid 2 2024-09-11T03:40:59.912 INFO:tasks.rados.rados.2.smithi175.stdout:724: delete oid 23 current snap is 73 2024-09-11T03:40:59.912 INFO:tasks.rados.rados.2.smithi175.stdout:716: finishing copy_from to smithi17557870-32 2024-09-11T03:40:59.913 INFO:tasks.rados.rados.2.smithi175.stdout:716: got expected ENOENT (src dne) 2024-09-11T03:40:59.913 INFO:tasks.rados.rados.2.smithi175.stdout:716: finishing copy_from racing read to smithi17557870-32 2024-09-11T03:40:59.913 INFO:tasks.rados.rados.2.smithi175.stdout:717: expect (ObjNum 241 snap 58 seq_num 241) 2024-09-11T03:40:59.924 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:40:59.924 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:40:59.942 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:40:59.942 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:40:59.959 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:40:59.959 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:40:59.960 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:59 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:40:59.960 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:59 smithi084 ceph-mon[27254]: 2.1 scrub ok 2024-09-11T03:40:59.960 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:59 smithi084 ceph-mon[27254]: pgmap v517: 47 pgs: 47 active+clean; 191 MiB data, 714 MiB used, 893 GiB / 894 GiB avail; 4.8 MiB/s rd, 1.8 MiB/s wr, 9 op/s 2024-09-11T03:40:59.960 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:59 smithi084 ceph-mon[27254]: osdmap e242: 12 total, 12 up, 10 in 2024-09-11T03:40:59.960 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:40:59 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2603799521' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:40:59.976 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:40:59.976 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:40:59.991 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:40:59.992 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:41:00.009 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:41:00.009 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:41:00.011 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:00.020 INFO:tasks.rados.rados.2.smithi175.stdout:720: finishing rollback tid 0 to smithi17557870-47 2024-09-11T03:41:00.020 INFO:tasks.rados.rados.2.smithi175.stdout:719: finishing rollback tid 0 to smithi17557870-8 2024-09-11T03:41:00.020 INFO:tasks.rados.rados.2.smithi175.stdout:719: finishing rollback tid 1 to smithi17557870-8 2024-09-11T03:41:00.020 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 235 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T03:41:00.020 INFO:tasks.rados.rados.2.smithi175.stdout:721: finishing rollback tid 0 to smithi17557870-24 2024-09-11T03:41:00.020 INFO:tasks.rados.rados.2.smithi175.stdout:721: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:41:00.020 INFO:tasks.rados.rados.2.smithi175.stdout:716: done (11 left) 2024-09-11T03:41:00.020 INFO:tasks.rados.rados.2.smithi175.stdout:717: done (10 left) 2024-09-11T03:41:00.020 INFO:tasks.rados.rados.2.smithi175.stdout:719: done (9 left) 2024-09-11T03:41:00.020 INFO:tasks.rados.rados.2.smithi175.stdout:724: done (8 left) 2024-09-11T03:41:00.021 INFO:tasks.rados.rados.2.smithi175.stdout:725: write (excl) oid 36 current snap is 73 2024-09-11T03:41:00.021 INFO:tasks.rados.rados.2.smithi175.stdout:721: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:41:00.021 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 238 (ObjNum 258 snap 64 seq_num 258) dirty exists 2024-09-11T03:41:00.021 INFO:tasks.rados.rados.2.smithi175.stdout:720: finishing rollback tid 1 to smithi17557870-47 2024-09-11T03:41:00.021 INFO:tasks.rados.rados.2.smithi175.stdout:720: finishing rollback tid 2 to smithi17557870-47 2024-09-11T03:41:00.021 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 233 (ObjNum 132 snap 22 seq_num 132) dirty exists 2024-09-11T03:41:00.021 INFO:tasks.rados.rados.2.smithi175.stdout:714: expect (ObjNum 257 snap 64 seq_num 257) 2024-09-11T03:41:00.026 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:41:00.026 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:41:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:00.033 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:00.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:00.120 INFO:tasks.rados.rados.2.smithi175.stdout:718: expect (ObjNum 101 snap 16 seq_num 101) 2024-09-11T03:41:00.134 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:00.155 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:00.156 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:00.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:59 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:41:00.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:59 smithi121 ceph-mon[30729]: 2.1 scrub ok 2024-09-11T03:41:00.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:59 smithi121 ceph-mon[30729]: pgmap v517: 47 pgs: 47 active+clean; 191 MiB data, 714 MiB used, 893 GiB / 894 GiB avail; 4.8 MiB/s rd, 1.8 MiB/s wr, 9 op/s 2024-09-11T03:41:00.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:59 smithi121 ceph-mon[30729]: osdmap e242: 12 total, 12 up, 10 in 2024-09-11T03:41:00.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:40:59 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2603799521' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:59 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:41:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:59 smithi175 ceph-mon[30894]: 2.1 scrub ok 2024-09-11T03:41:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:59 smithi175 ceph-mon[30894]: pgmap v517: 47 pgs: 47 active+clean; 191 MiB data, 714 MiB used, 893 GiB / 894 GiB avail; 4.8 MiB/s rd, 1.8 MiB/s wr, 9 op/s 2024-09-11T03:41:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:59 smithi175 ceph-mon[30894]: osdmap e242: 12 total, 12 up, 10 in 2024-09-11T03:41:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:40:59 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2603799521' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:00.257 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:00.278 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:00.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:00.279 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:00.284 INFO:tasks.rados.rados.2.smithi175.stdout:723: finishing write tid 1 to smithi17557870-29 2024-09-11T03:41:00.284 INFO:tasks.rados.rados.2.smithi175.stdout:722: finishing copy_from racing read to smithi17557870-39 2024-09-11T03:41:00.285 INFO:tasks.rados.rados.2.smithi175.stdout:723: finishing write tid 2 to smithi17557870-29 2024-09-11T03:41:00.285 INFO:tasks.rados.rados.2.smithi175.stdout:723: finishing write tid 3 to smithi17557870-29 2024-09-11T03:41:00.285 INFO:tasks.rados.rados.2.smithi175.stdout:725: seq_num 282 ranges {739574=725279,1968628=721093} 2024-09-11T03:41:00.292 INFO:tasks.rados.rados.2.smithi175.stdout:725: writing smithi17557870-36 from 739574 to 1464853 tid 1 2024-09-11T03:41:00.299 INFO:tasks.rados.rados.2.smithi175.stdout:725: writing smithi17557870-36 from 1968628 to 2689721 tid 2 2024-09-11T03:41:00.299 INFO:tasks.rados.rados.2.smithi175.stdout:714: done (8 left) 2024-09-11T03:41:00.299 INFO:tasks.rados.rados.2.smithi175.stdout:718: done (7 left) 2024-09-11T03:41:00.300 INFO:tasks.rados.rados.2.smithi175.stdout:720: done (6 left) 2024-09-11T03:41:00.300 INFO:tasks.rados.rados.2.smithi175.stdout:721: done (5 left) 2024-09-11T03:41:00.300 INFO:tasks.rados.rados.2.smithi175.stdout:726: read oid 9 snap 71 2024-09-11T03:41:00.300 INFO:tasks.rados.rados.2.smithi175.stdout:726: expect deleted 2024-09-11T03:41:00.300 INFO:tasks.rados.rados.2.smithi175.stdout:727: snap_create 2024-09-11T03:41:00.300 INFO:tasks.rados.rados.2.smithi175.stdout:723: finishing write tid 4 to smithi17557870-29 2024-09-11T03:41:00.300 INFO:tasks.rados.rados.2.smithi175.stdout:723: oid 29 updating version 0 to 230 2024-09-11T03:41:00.300 INFO:tasks.rados.rados.2.smithi175.stdout:723: oid 29 updating version 230 to 232 2024-09-11T03:41:00.300 INFO:tasks.rados.rados.2.smithi175.stdout:723: oid 29 version 232 is already newer than 231 2024-09-11T03:41:00.300 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 232 (ObjNum 280 snap 73 seq_num 280) dirty exists 2024-09-11T03:41:00.300 INFO:tasks.rados.rados.2.smithi175.stdout:723: left oid 29 (ObjNum 280 snap 73 seq_num 280) 2024-09-11T03:41:00.300 INFO:tasks.rados.rados.2.smithi175.stdout:722: finishing copy_from to smithi17557870-39 2024-09-11T03:41:00.300 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 195 (ObjNum 262 snap 66 seq_num 262) dirty exists 2024-09-11T03:41:00.301 INFO:tasks.rados.rados.2.smithi175.stdout:715: expect (ObjNum 194 snap 47 seq_num 194) 2024-09-11T03:41:00.380 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:00.401 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:00.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:00.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:00.438 INFO:tasks.rados.rados.2.smithi175.stdout:713: finishing copy_from racing read to smithi17557870-10 2024-09-11T03:41:00.438 INFO:tasks.rados.rados.2.smithi175.stdout:713: finishing copy_from to smithi17557870-10 2024-09-11T03:41:00.438 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 29 (ObjNum 225 snap 53 seq_num 225) dirty exists 2024-09-11T03:41:00.438 INFO:tasks.rados.rados.2.smithi175.stdout:725: finishing write tid 1 to smithi17557870-36 2024-09-11T03:41:00.438 INFO:tasks.rados.rados.2.smithi175.stdout:725: finishing write tid 2 to smithi17557870-36 2024-09-11T03:41:00.438 INFO:tasks.rados.rados.2.smithi175.stdout:725: finishing write tid 3 to smithi17557870-36 2024-09-11T03:41:00.438 INFO:tasks.rados.rados.2.smithi175.stdout:725: finishing write tid 4 to smithi17557870-36 2024-09-11T03:41:00.438 INFO:tasks.rados.rados.2.smithi175.stdout:725: oid 36 updating version 0 to 170 2024-09-11T03:41:00.438 INFO:tasks.rados.rados.2.smithi175.stdout:725: oid 36 version 170 is already newer than 169 2024-09-11T03:41:00.438 INFO:tasks.rados.rados.2.smithi175.stdout:725: oid 36 updating version 170 to 171 2024-09-11T03:41:00.438 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 171 (ObjNum 282 snap 73 seq_num 282) dirty exists 2024-09-11T03:41:00.439 INFO:tasks.rados.rados.2.smithi175.stdout:725: left oid 36 (ObjNum 282 snap 73 seq_num 282) 2024-09-11T03:41:00.503 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:00.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:00.525 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:00.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:00.627 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:00.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:00.649 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:00.649 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:00.750 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:00.771 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:00.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:00.772 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:00.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:00.899 INFO:teuthology.orchestra.run.smithi084.stdout:moved 2 / 141 (1.41844%) 2024-09-11T03:41:00.899 INFO:teuthology.orchestra.run.smithi084.stdout:avg 14.1 2024-09-11T03:41:00.899 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 2.34308 -> 2.21133 (expected baseline 3.5623) 2024-09-11T03:41:00.899 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.5 with 11 -> 11 pgs (0.780142 -> 0.780142 * mean) 2024-09-11T03:41:00.899 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.7 with 19 -> 18 pgs (1.34752 -> 1.2766 * mean) 2024-09-11T03:41:00.899 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:41:00.899 INFO:teuthology.orchestra.run.smithi084.stdout:oload 110 2024-09-11T03:41:00.899 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 0.05 2024-09-11T03:41:00.899 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2024-09-11T03:41:00.899 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0008 2024-09-11T03:41:00.899 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0009 2024-09-11T03:41:00.900 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7 weight 1.0000 -> 0.9500 2024-09-11T03:41:00.900 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3 weight 0.9500 -> 0.9000 2024-09-11T03:41:00.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:00.901 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:00.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:00.902 INFO:tasks.rados.rados.2.smithi175.stdout:713: done (6 left) 2024-09-11T03:41:00.902 INFO:tasks.rados.rados.2.smithi175.stdout:715: done (5 left) 2024-09-11T03:41:00.903 INFO:tasks.rados.rados.2.smithi175.stdout:722: done (4 left) 2024-09-11T03:41:00.903 INFO:tasks.rados.rados.2.smithi175.stdout:723: done (3 left) 2024-09-11T03:41:00.903 INFO:tasks.rados.rados.2.smithi175.stdout:725: done (2 left) 2024-09-11T03:41:00.903 INFO:tasks.rados.rados.2.smithi175.stdout:726: done (1 left) 2024-09-11T03:41:00.903 INFO:tasks.rados.rados.2.smithi175.stdout:727: done (0 left) 2024-09-11T03:41:00.903 INFO:tasks.rados.rados.2.smithi175.stdout:728: read oid 49 snap -1 2024-09-11T03:41:00.903 INFO:tasks.rados.rados.2.smithi175.stdout:728: expect (ObjNum 267 snap 68 seq_num 267) 2024-09-11T03:41:00.903 INFO:tasks.rados.rados.2.smithi175.stdout:729: read oid 48 snap 73 2024-09-11T03:41:00.903 INFO:tasks.rados.rados.2.smithi175.stdout:729: expect (ObjNum 246 snap 60 seq_num 246) 2024-09-11T03:41:00.903 INFO:tasks.rados.rados.2.smithi175.stdout:730: delete oid 4 current snap is 74 2024-09-11T03:41:00.907 INFO:tasks.rados.rados.2.smithi175.stdout:728: expect (ObjNum 267 snap 68 seq_num 267) 2024-09-11T03:41:01.001 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:01.027 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:01.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:01.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:01.062 INFO:tasks.rados.rados.2.smithi175.stdout:729: expect (ObjNum 246 snap 60 seq_num 246) 2024-09-11T03:41:01.083 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:41:01.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:00 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:41:01.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:00 smithi084 ceph-mon[27254]: 2.a scrub ok 2024-09-11T03:41:01.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:00 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:41:01.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:00 smithi084 ceph-mon[27254]: 2.7 scrub ok 2024-09-11T03:41:01.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:00 smithi084 ceph-mon[27254]: osdmap e243: 12 total, 12 up, 10 in 2024-09-11T03:41:01.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:00 smithi084 ceph-mon[27254]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:41:01.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'58984','7':'62260'}"} : dispatch 2024-09-11T03:41:01.129 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:01.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:01.152 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:01.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:00 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:41:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:00 smithi121 ceph-mon[30729]: 2.a scrub ok 2024-09-11T03:41:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:00 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:41:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:00 smithi121 ceph-mon[30729]: 2.7 scrub ok 2024-09-11T03:41:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:00 smithi121 ceph-mon[30729]: osdmap e243: 12 total, 12 up, 10 in 2024-09-11T03:41:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:00 smithi121 ceph-mon[30729]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:41:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'58984','7':'62260'}"} : dispatch 2024-09-11T03:41:01.201 INFO:tasks.rados.rados.2.smithi175.stdout:728: done (2 left) 2024-09-11T03:41:01.201 INFO:tasks.rados.rados.2.smithi175.stdout:729: done (1 left) 2024-09-11T03:41:01.201 INFO:tasks.rados.rados.2.smithi175.stdout:730: done (0 left) 2024-09-11T03:41:01.201 INFO:tasks.rados.rados.2.smithi175.stdout:731: snap_create 2024-09-11T03:41:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:00 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:41:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:00 smithi175 ceph-mon[30894]: 2.a scrub ok 2024-09-11T03:41:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:00 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:41:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:00 smithi175 ceph-mon[30894]: 2.7 scrub ok 2024-09-11T03:41:01.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:00 smithi175 ceph-mon[30894]: osdmap e243: 12 total, 12 up, 10 in 2024-09-11T03:41:01.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:00 smithi175 ceph-mon[30894]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:41:01.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'3':'58984','7':'62260'}"} : dispatch 2024-09-11T03:41:01.253 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:01.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:01.275 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:01.276 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:01.377 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:01.397 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:01.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:01.398 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:01.499 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:01.507 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:41:01.507 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:41:01.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:41:01.508 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:41:01.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.508 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:43 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'25, mlcod=20'22)", 2024-09-11T03:41:01.508 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.202514+0000", 2024-09-11T03:41:01.508 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 437.30489596699999, 2024-09-11T03:41:01.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030994870000000002, 2024-09-11T03:41:01.508 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.508 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.508 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.508 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202514+0000", 2024-09-11T03:41:01.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202514+0000", 2024-09-11T03:41:01.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202514+0000", 2024-09-11T03:41:01.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7899999999999998e-07 2024-09-11T03:41:01.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202519+0000", 2024-09-11T03:41:01.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1949999999999996e-06 2024-09-11T03:41:01.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202531+0000", 2024-09-11T03:41:01.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1975e-05 2024-09-11T03:41:01.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202537+0000", 2024-09-11T03:41:01.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9649999999999996e-06 2024-09-11T03:41:01.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202581+0000", 2024-09-11T03:41:01.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4373999999999999e-05 2024-09-11T03:41:01.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.202596+0000", 2024-09-11T03:41:01.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5367999999999999e-05 2024-09-11T03:41:01.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.205590+0000", 2024-09-11T03:41:01.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029938579999999998 2024-09-11T03:41:01.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.205613+0000", 2024-09-11T03:41:01.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3173e-05 2024-09-11T03:41:01.512 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.512 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.512 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.513 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:46 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'28, mlcod=20'23)", 2024-09-11T03:41:01.513 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.206111+0000", 2024-09-11T03:41:01.513 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 437.301298723, 2024-09-11T03:41:01.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026967599999999999, 2024-09-11T03:41:01.513 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.513 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.513 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.513 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206111+0000", 2024-09-11T03:41:01.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206111+0000", 2024-09-11T03:41:01.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206112+0000", 2024-09-11T03:41:01.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0500000000000003e-07 2024-09-11T03:41:01.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206125+0000", 2024-09-11T03:41:01.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2908000000000001e-05 2024-09-11T03:41:01.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206137+0000", 2024-09-11T03:41:01.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.187e-05 2024-09-11T03:41:01.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206144+0000", 2024-09-11T03:41:01.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7919999999999993e-06 2024-09-11T03:41:01.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206185+0000", 2024-09-11T03:41:01.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0803999999999998e-05 2024-09-11T03:41:01.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.206197+0000", 2024-09-11T03:41:01.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1962e-05 2024-09-11T03:41:01.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.208764+0000", 2024-09-11T03:41:01.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002566652 2024-09-11T03:41:01.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.208808+0000", 2024-09-11T03:41:01.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4166999999999999e-05 2024-09-11T03:41:01.517 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.517 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.517 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.517 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:48 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'30, mlcod=20'27)", 2024-09-11T03:41:01.517 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.209747+0000", 2024-09-11T03:41:01.517 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 437.29766277700003, 2024-09-11T03:41:01.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030822760000000001, 2024-09-11T03:41:01.517 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.518 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.518 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.518 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209747+0000", 2024-09-11T03:41:01.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209747+0000", 2024-09-11T03:41:01.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209748+0000", 2024-09-11T03:41:01.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4399999999999999e-07 2024-09-11T03:41:01.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209753+0000", 2024-09-11T03:41:01.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6130000000000003e-06 2024-09-11T03:41:01.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209772+0000", 2024-09-11T03:41:01.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8493000000000001e-05 2024-09-11T03:41:01.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209786+0000", 2024-09-11T03:41:01.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4399000000000001e-05 2024-09-11T03:41:01.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209832+0000", 2024-09-11T03:41:01.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5556999999999998e-05 2024-09-11T03:41:01.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.209852+0000", 2024-09-11T03:41:01.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0262999999999999e-05 2024-09-11T03:41:01.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212794+0000", 2024-09-11T03:41:01.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029415259999999999 2024-09-11T03:41:01.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212829+0000", 2024-09-11T03:41:01.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5680999999999999e-05 2024-09-11T03:41:01.521 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.521 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.521 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.522 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.14156.0:49 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'31, mlcod=20'27)", 2024-09-11T03:41:01.522 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:33:44.210165+0000", 2024-09-11T03:41:01.522 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 437.29724492000003, 2024-09-11T03:41:01.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028269739999999999, 2024-09-11T03:41:01.523 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.523 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.523 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.523 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210165+0000", 2024-09-11T03:41:01.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210165+0000", 2024-09-11T03:41:01.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210166+0000", 2024-09-11T03:41:01.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5300000000000004e-07 2024-09-11T03:41:01.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210171+0000", 2024-09-11T03:41:01.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1039999999999998e-06 2024-09-11T03:41:01.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210183+0000", 2024-09-11T03:41:01.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1848000000000001e-05 2024-09-11T03:41:01.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210189+0000", 2024-09-11T03:41:01.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4459999999999998e-06 2024-09-11T03:41:01.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210267+0000", 2024-09-11T03:41:01.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7540999999999995e-05 2024-09-11T03:41:01.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.210288+0000", 2024-09-11T03:41:01.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0992000000000002e-05 2024-09-11T03:41:01.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212958+0000", 2024-09-11T03:41:01.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002670455 2024-09-11T03:41:01.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:33:44.212992+0000", 2024-09-11T03:41:01.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3935e-05 2024-09-11T03:41:01.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.527 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.527 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:359 2.5 e114/95 2:a6d17fc0:::smithi17557870-13:head v 114'40, mlcod=110'38)", 2024-09-11T03:41:01.527 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:50.842632+0000", 2024-09-11T03:41:01.527 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 130.66477784400001, 2024-09-11T03:41:01.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002717297, 2024-09-11T03:41:01.527 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.527 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.527 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.528 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.842632+0000", 2024-09-11T03:41:01.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.842632+0000", 2024-09-11T03:41:01.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.842632+0000", 2024-09-11T03:41:01.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6100000000000002e-07 2024-09-11T03:41:01.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.844452+0000", 2024-09-11T03:41:01.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018200619999999999 2024-09-11T03:41:01.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.844463+0000", 2024-09-11T03:41:01.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0351e-05 2024-09-11T03:41:01.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.844470+0000", 2024-09-11T03:41:01.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6240000000000002e-06 2024-09-11T03:41:01.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.844510+0000", 2024-09-11T03:41:01.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9703999999999998e-05 2024-09-11T03:41:01.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.844526+0000", 2024-09-11T03:41:01.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6172e-05 2024-09-11T03:41:01.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.845322+0000", 2024-09-11T03:41:01.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00079569999999999999 2024-09-11T03:41:01.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.845349+0000", 2024-09-11T03:41:01.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7423000000000002e-05 2024-09-11T03:41:01.532 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.532 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.532 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.532 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:752 2.c e138/95 2:3c007163:::smithi17557870-35:head v 138'40, mlcod=135'37)", 2024-09-11T03:41:01.532 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:41:01.532 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 106.491836101, 2024-09-11T03:41:01.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084184749999999999, 2024-09-11T03:41:01.532 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.532 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.532 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.532 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:41:01.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:41:01.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:41:01.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8400000000000001e-07 2024-09-11T03:41:01.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021686+0000", 2024-09-11T03:41:01.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061118680000000003 2024-09-11T03:41:01.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021706+0000", 2024-09-11T03:41:01.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9942999999999999e-05 2024-09-11T03:41:01.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021717+0000", 2024-09-11T03:41:01.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.098e-05 2024-09-11T03:41:01.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021806+0000", 2024-09-11T03:41:01.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9197999999999995e-05 2024-09-11T03:41:01.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021834+0000", 2024-09-11T03:41:01.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8251000000000001e-05 2024-09-11T03:41:01.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.023931+0000", 2024-09-11T03:41:01.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020966639999999998 2024-09-11T03:41:01.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.023992+0000", 2024-09-11T03:41:01.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1387e-05 2024-09-11T03:41:01.536 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.536 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.536 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.536 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:947 2.8 e163/95 2:1bdf1a36:::smithi17557870-36:head v 163'45, mlcod=157'41)", 2024-09-11T03:41:01.537 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:39.852028+0000", 2024-09-11T03:41:01.537 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 81.655381896999998, 2024-09-11T03:41:01.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025214819999999998, 2024-09-11T03:41:01.537 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.537 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.537 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.537 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.852028+0000", 2024-09-11T03:41:01.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.852028+0000", 2024-09-11T03:41:01.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.852028+0000", 2024-09-11T03:41:01.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9000000000000001e-07 2024-09-11T03:41:01.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.853313+0000", 2024-09-11T03:41:01.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012846240000000001 2024-09-11T03:41:01.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.853327+0000", 2024-09-11T03:41:01.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3865e-05 2024-09-11T03:41:01.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.853336+0000", 2024-09-11T03:41:01.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3519999999999996e-06 2024-09-11T03:41:01.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.853397+0000", 2024-09-11T03:41:01.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1432000000000004e-05 2024-09-11T03:41:01.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.853416+0000", 2024-09-11T03:41:01.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.838e-05 2024-09-11T03:41:01.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.854515+0000", 2024-09-11T03:41:01.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001099395 2024-09-11T03:41:01.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.854549+0000", 2024-09-11T03:41:01.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4243999999999999e-05 2024-09-11T03:41:01.541 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.541 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.541 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.541 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1175 2.c e185/95 2:32ade24e:::smithi17557870-45:head v 185'66, mlcod=182'62)", 2024-09-11T03:41:01.541 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:02.110649+0000", 2024-09-11T03:41:01.541 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 59.396761116999997, 2024-09-11T03:41:01.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00675758, 2024-09-11T03:41:01.541 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.541 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.541 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.542 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.110649+0000", 2024-09-11T03:41:01.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.110649+0000", 2024-09-11T03:41:01.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.110649+0000", 2024-09-11T03:41:01.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4699999999999998e-07 2024-09-11T03:41:01.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.115096+0000", 2024-09-11T03:41:01.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044473220000000001 2024-09-11T03:41:01.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.115114+0000", 2024-09-11T03:41:01.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7825e-05 2024-09-11T03:41:01.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.115124+0000", 2024-09-11T03:41:01.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5200000000000003e-06 2024-09-11T03:41:01.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.115202+0000", 2024-09-11T03:41:01.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8102000000000005e-05 2024-09-11T03:41:01.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.115231+0000", 2024-09-11T03:41:01.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8765e-05 2024-09-11T03:41:01.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.117364+0000", 2024-09-11T03:41:01.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021330630000000001 2024-09-11T03:41:01.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.117406+0000", 2024-09-11T03:41:01.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2735999999999999e-05 2024-09-11T03:41:01.545 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.545 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.545 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.546 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1285 2.1 e196/95 2:8638b65e:::smithi17557870-5:head v 196'52, mlcod=173'49)", 2024-09-11T03:41:01.546 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:12.850227+0000", 2024-09-11T03:41:01.546 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 48.657182483, 2024-09-11T03:41:01.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059143149999999999, 2024-09-11T03:41:01.546 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.546 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.546 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.547 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.850227+0000", 2024-09-11T03:41:01.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.850227+0000", 2024-09-11T03:41:01.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.850228+0000", 2024-09-11T03:41:01.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6199999999999997e-07 2024-09-11T03:41:01.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853054+0000", 2024-09-11T03:41:01.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028262000000000001 2024-09-11T03:41:01.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853087+0000", 2024-09-11T03:41:01.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2724e-05 2024-09-11T03:41:01.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853100+0000", 2024-09-11T03:41:01.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3456e-05 2024-09-11T03:41:01.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853187+0000", 2024-09-11T03:41:01.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6211999999999994e-05 2024-09-11T03:41:01.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853216+0000", 2024-09-11T03:41:01.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9825999999999999e-05 2024-09-11T03:41:01.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856078+0000", 2024-09-11T03:41:01.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028619689999999998 2024-09-11T03:41:01.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856142+0000", 2024-09-11T03:41:01.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3266e-05 2024-09-11T03:41:01.550 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.551 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.551 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.551 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.1.0:366 2.4 e206/188 2:2ef19c88:::smithi17557870-15:32 v 206'95, mlcod=201'91)", 2024-09-11T03:41:01.551 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:22.836592+0000", 2024-09-11T03:41:01.551 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 38.670817458000002, 2024-09-11T03:41:01.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0032115030000000001, 2024-09-11T03:41:01.551 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.551 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.551 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.551 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:22.836592+0000", 2024-09-11T03:41:01.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:22.836592+0000", 2024-09-11T03:41:01.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:22.836593+0000", 2024-09-11T03:41:01.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6e-07 2024-09-11T03:41:01.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:22.836599+0000", 2024-09-11T03:41:01.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3389999999999998e-06 2024-09-11T03:41:01.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:22.836625+0000", 2024-09-11T03:41:01.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.597e-05 2024-09-11T03:41:01.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:22.836639+0000", 2024-09-11T03:41:01.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4221e-05 2024-09-11T03:41:01.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:22.836703+0000", 2024-09-11T03:41:01.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3897000000000002e-05 2024-09-11T03:41:01.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:22.836725+0000", 2024-09-11T03:41:01.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2340999999999998e-05 2024-09-11T03:41:01.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:22.839752+0000", 2024-09-11T03:41:01.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030267610000000002 2024-09-11T03:41:01.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:22.839804+0000", 2024-09-11T03:41:01.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1714e-05 2024-09-11T03:41:01.555 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.555 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.555 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.555 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1526 2.1 e212/95 2:8638b65e:::smithi17557870-5:head v 212'54, mlcod=186'51)", 2024-09-11T03:41:01.555 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:28.928493+0000", 2024-09-11T03:41:01.555 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 32.578917212, 2024-09-11T03:41:01.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025179960000000002, 2024-09-11T03:41:01.556 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.556 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.556 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.556 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.928493+0000", 2024-09-11T03:41:01.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.928493+0000", 2024-09-11T03:41:01.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.928493+0000", 2024-09-11T03:41:01.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2e-07 2024-09-11T03:41:01.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929175+0000", 2024-09-11T03:41:01.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00068189800000000001 2024-09-11T03:41:01.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929189+0000", 2024-09-11T03:41:01.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3662999999999999e-05 2024-09-11T03:41:01.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929197+0000", 2024-09-11T03:41:01.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4069999999999993e-06 2024-09-11T03:41:01.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929287+0000", 2024-09-11T03:41:01.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0000000000000006e-05 2024-09-11T03:41:01.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929324+0000", 2024-09-11T03:41:01.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6804999999999997e-05 2024-09-11T03:41:01.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.930952+0000", 2024-09-11T03:41:01.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016284660000000001 2024-09-11T03:41:01.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.931011+0000", 2024-09-11T03:41:01.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8337000000000001e-05 2024-09-11T03:41:01.560 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.560 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.560 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.560 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1513 2.5 e212/95 2:a6e3bbd9:::smithi17557870-38:head v 212'178, mlcod=203'174)", 2024-09-11T03:41:01.560 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:28.928696+0000", 2024-09-11T03:41:01.560 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 32.578714194, 2024-09-11T03:41:01.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0099738110000000008, 2024-09-11T03:41:01.560 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.560 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.560 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.560 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.928696+0000", 2024-09-11T03:41:01.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.928696+0000", 2024-09-11T03:41:01.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.928696+0000", 2024-09-11T03:41:01.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9500000000000001e-07 2024-09-11T03:41:01.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935839+0000", 2024-09-11T03:41:01.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.007143473 2024-09-11T03:41:01.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935850+0000", 2024-09-11T03:41:01.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.026e-05 2024-09-11T03:41:01.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935854+0000", 2024-09-11T03:41:01.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8080000000000004e-06 2024-09-11T03:41:01.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935903+0000", 2024-09-11T03:41:01.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8380000000000001e-05 2024-09-11T03:41:01.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935917+0000", 2024-09-11T03:41:01.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4192999999999999e-05 2024-09-11T03:41:01.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.938638+0000", 2024-09-11T03:41:01.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002721298 2024-09-11T03:41:01.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.938669+0000", 2024-09-11T03:41:01.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1204000000000001e-05 2024-09-11T03:41:01.564 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.564 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.564 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.564 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "pg_update_log_missing(2.c epoch 212/95 rep_tid 123 entries 212'131 (0'0) error 2:327469ff:::smithi17557870-49:head by client.24454.0:1530 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 212'128)", 2024-09-11T03:41:01.565 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:28.933564+0000", 2024-09-11T03:41:01.565 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 32.573845478000003, 2024-09-11T03:41:01.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031675290000000001, 2024-09-11T03:41:01.565 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.565 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.565 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-11T03:41:01.565 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.933564+0000", 2024-09-11T03:41:01.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.933564+0000", 2024-09-11T03:41:01.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.933566+0000", 2024-09-11T03:41:01.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1379999999999999e-06 2024-09-11T03:41:01.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.933576+0000", 2024-09-11T03:41:01.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0762e-05 2024-09-11T03:41:01.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.933600+0000", 2024-09-11T03:41:01.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4108e-05 2024-09-11T03:41:01.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.933616+0000", 2024-09-11T03:41:01.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5401999999999999e-05 2024-09-11T03:41:01.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.933727+0000", 2024-09-11T03:41:01.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000111005 2024-09-11T03:41:01.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.936732+0000", 2024-09-11T03:41:01.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030051140000000001 2024-09-11T03:41:01.568 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.568 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.568 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.568 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1527 2.1 e212/95 2:8638b65e:::smithi17557870-5:head v 212'55, mlcod=186'51)", 2024-09-11T03:41:01.568 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:28.933895+0000", 2024-09-11T03:41:01.568 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 32.573515297, 2024-09-11T03:41:01.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003130903, 2024-09-11T03:41:01.569 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.569 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.569 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.569 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.933895+0000", 2024-09-11T03:41:01.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.933895+0000", 2024-09-11T03:41:01.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.933895+0000", 2024-09-11T03:41:01.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4300000000000002e-07 2024-09-11T03:41:01.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.933991+0000", 2024-09-11T03:41:01.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5346999999999994e-05 2024-09-11T03:41:01.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.934004+0000", 2024-09-11T03:41:01.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.287e-05 2024-09-11T03:41:01.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.934011+0000", 2024-09-11T03:41:01.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4150000000000002e-06 2024-09-11T03:41:01.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.934064+0000", 2024-09-11T03:41:01.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2655000000000003e-05 2024-09-11T03:41:01.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.934087+0000", 2024-09-11T03:41:01.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3023e-05 2024-09-11T03:41:01.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.936976+0000", 2024-09-11T03:41:01.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028891609999999999 2024-09-11T03:41:01.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.937025+0000", 2024-09-11T03:41:01.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9589000000000002e-05 2024-09-11T03:41:01.573 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.573 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.573 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.573 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.1.0:384 2.4 e213/188 2:220bb244:::smithi17557870-34:3b v 213'101, mlcod=206'97)", 2024-09-11T03:41:01.573 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:29.867045+0000", 2024-09-11T03:41:01.573 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 31.640364655999999, 2024-09-11T03:41:01.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025818439999999998, 2024-09-11T03:41:01.573 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.573 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.573 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.573 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.867045+0000", 2024-09-11T03:41:01.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.867045+0000", 2024-09-11T03:41:01.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.867046+0000", 2024-09-11T03:41:01.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8899999999999999e-07 2024-09-11T03:41:01.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.867052+0000", 2024-09-11T03:41:01.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6109999999999999e-06 2024-09-11T03:41:01.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.867072+0000", 2024-09-11T03:41:01.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9201000000000001e-05 2024-09-11T03:41:01.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.867082+0000", 2024-09-11T03:41:01.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0757000000000001e-05 2024-09-11T03:41:01.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.867177+0000", 2024-09-11T03:41:01.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4160000000000006e-05 2024-09-11T03:41:01.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.867203+0000", 2024-09-11T03:41:01.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6531000000000001e-05 2024-09-11T03:41:01.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.869087+0000", 2024-09-11T03:41:01.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018843589999999999 2024-09-11T03:41:01.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.869627+0000", 2024-09-11T03:41:01.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00053963600000000002 2024-09-11T03:41:01.577 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.577 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.577 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.577 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.1.0:386 2.4 e213/188 2:2ef19c88:::smithi17557870-15:35 v 213'103, mlcod=206'97)", 2024-09-11T03:41:01.578 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:29.868216+0000", 2024-09-11T03:41:01.578 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 31.639194070999999, 2024-09-11T03:41:01.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025292050000000001, 2024-09-11T03:41:01.578 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.578 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.578 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.578 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.868216+0000", 2024-09-11T03:41:01.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.868216+0000", 2024-09-11T03:41:01.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.868217+0000", 2024-09-11T03:41:01.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4000000000000001e-07 2024-09-11T03:41:01.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.868223+0000", 2024-09-11T03:41:01.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1469999999999998e-06 2024-09-11T03:41:01.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.868233+0000", 2024-09-11T03:41:01.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8700000000000004e-06 2024-09-11T03:41:01.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.868239+0000", 2024-09-11T03:41:01.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.827e-06 2024-09-11T03:41:01.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.869857+0000", 2024-09-11T03:41:01.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016175479999999999 2024-09-11T03:41:01.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.869877+0000", 2024-09-11T03:41:01.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0205000000000001e-05 2024-09-11T03:41:01.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.870705+0000", 2024-09-11T03:41:01.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00082836699999999999 2024-09-11T03:41:01.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.870745+0000", 2024-09-11T03:41:01.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9501e-05 2024-09-11T03:41:01.583 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.583 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.583 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.583 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1609 2.8 e219/95 2:1bdf1a36:::smithi17557870-36:head v 219'144, mlcod=213'140)", 2024-09-11T03:41:01.583 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:35.907539+0000", 2024-09-11T03:41:01.583 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 25.599871137000001, 2024-09-11T03:41:01.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025005769999999999, 2024-09-11T03:41:01.583 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.583 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.584 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.584 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.907539+0000", 2024-09-11T03:41:01.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.907539+0000", 2024-09-11T03:41:01.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.907539+0000", 2024-09-11T03:41:01.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.48e-07 2024-09-11T03:41:01.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.908793+0000", 2024-09-11T03:41:01.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012540660000000001 2024-09-11T03:41:01.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.908804+0000", 2024-09-11T03:41:01.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.083e-05 2024-09-11T03:41:01.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.908810+0000", 2024-09-11T03:41:01.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4860000000000002e-06 2024-09-11T03:41:01.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.908869+0000", 2024-09-11T03:41:01.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8579999999999998e-05 2024-09-11T03:41:01.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.908894+0000", 2024-09-11T03:41:01.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5233e-05 2024-09-11T03:41:01.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.909994+0000", 2024-09-11T03:41:01.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001100202 2024-09-11T03:41:01.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.910039+0000", 2024-09-11T03:41:01.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5031999999999998e-05 2024-09-11T03:41:01.587 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.587 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.587 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.588 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1720 2.c e229/95 2:327469ff:::smithi17557870-49:head v 229'149, mlcod=225'144)", 2024-09-11T03:41:01.588 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:45.866778+0000", 2024-09-11T03:41:01.588 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 15.640632255, 2024-09-11T03:41:01.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002486472, 2024-09-11T03:41:01.588 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.588 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.588 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.588 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:45.866778+0000", 2024-09-11T03:41:01.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:45.866778+0000", 2024-09-11T03:41:01.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:45.866778+0000", 2024-09-11T03:41:01.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.54e-07 2024-09-11T03:41:01.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:45.867852+0000", 2024-09-11T03:41:01.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001074017 2024-09-11T03:41:01.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:45.867872+0000", 2024-09-11T03:41:01.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9153000000000001e-05 2024-09-11T03:41:01.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:45.867880+0000", 2024-09-11T03:41:01.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0900000000000005e-06 2024-09-11T03:41:01.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:45.867954+0000", 2024-09-11T03:41:01.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4705999999999999e-05 2024-09-11T03:41:01.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:45.867979+0000", 2024-09-11T03:41:01.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4328000000000002e-05 2024-09-11T03:41:01.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:45.869222+0000", 2024-09-11T03:41:01.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012431009999999999 2024-09-11T03:41:01.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:45.869264+0000", 2024-09-11T03:41:01.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2323000000000001e-05 2024-09-11T03:41:01.592 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.592 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.592 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.592 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1843 2.5 e243/95 2:a286c222:::smithi17557870-29:head v 243'231, mlcod=241'228)", 2024-09-11T03:41:01.592 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:59.915728+0000", 2024-09-11T03:41:01.592 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.591682042, 2024-09-11T03:41:01.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025568240000000001, 2024-09-11T03:41:01.593 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.593 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.593 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.593 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.915728+0000", 2024-09-11T03:41:01.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.915728+0000", 2024-09-11T03:41:01.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.915729+0000", 2024-09-11T03:41:01.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8100000000000001e-07 2024-09-11T03:41:01.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.917366+0000", 2024-09-11T03:41:01.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001637152 2024-09-11T03:41:01.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.917377+0000", 2024-09-11T03:41:01.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0987e-05 2024-09-11T03:41:01.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.917385+0000", 2024-09-11T03:41:01.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0590000000000003e-06 2024-09-11T03:41:01.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.917435+0000", 2024-09-11T03:41:01.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9521999999999998e-05 2024-09-11T03:41:01.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.917449+0000", 2024-09-11T03:41:01.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4912000000000001e-05 2024-09-11T03:41:01.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.918235+0000", 2024-09-11T03:41:01.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00078576000000000004 2024-09-11T03:41:01.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.918285+0000", 2024-09-11T03:41:01.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9450999999999998e-05 2024-09-11T03:41:01.596 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.597 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.597 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.597 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1847 2.8 e243/95 2:1bdf1a36:::smithi17557870-36:head v 243'169, mlcod=241'165)", 2024-09-11T03:41:01.597 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:00.295521+0000", 2024-09-11T03:41:01.597 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.211889188, 2024-09-11T03:41:01.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003306153, 2024-09-11T03:41:01.597 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:01.597 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:01.597 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:01.597 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:01.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:01.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:00.295521+0000", 2024-09-11T03:41:01.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:01.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:00.295521+0000", 2024-09-11T03:41:01.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:01.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:01.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:00.295522+0000", 2024-09-11T03:41:01.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0240000000000001e-06 2024-09-11T03:41:01.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:01.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:00.297283+0000", 2024-09-11T03:41:01.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017614460000000001 2024-09-11T03:41:01.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:01.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:00.297299+0000", 2024-09-11T03:41:01.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6087e-05 2024-09-11T03:41:01.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:01.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:00.297308+0000", 2024-09-11T03:41:01.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3080000000000008e-06 2024-09-11T03:41:01.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:01.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:00.297377+0000", 2024-09-11T03:41:01.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9657000000000001e-05 2024-09-11T03:41:01.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:01.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:00.297407+0000", 2024-09-11T03:41:01.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9513e-05 2024-09-11T03:41:01.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:01.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:00.298788+0000", 2024-09-11T03:41:01.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0013814960000000001 2024-09-11T03:41:01.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:01.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:01.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:01.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:00.298827+0000", 2024-09-11T03:41:01.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8621999999999997e-05 2024-09-11T03:41:01.601 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.601 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.601 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.601 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:01.601 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:01.601 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:41:01.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:01.602 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:01.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:01.703 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:01.724 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:01.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:01.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:01.826 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:01.848 INFO:tasks.rados.rados.2.smithi175.stdout:731: done (0 left) 2024-09-11T03:41:01.849 INFO:tasks.rados.rados.2.smithi175.stdout:732: read oid 35 snap -1 2024-09-11T03:41:01.849 INFO:tasks.rados.rados.2.smithi175.stdout:732: expect (ObjNum 215 snap 52 seq_num 215) 2024-09-11T03:41:01.849 INFO:tasks.rados.rados.2.smithi175.stdout:733: copy_from oid 14 from oid 24 current snap is 75 2024-09-11T03:41:01.849 INFO:tasks.rados.rados.2.smithi175.stdout:734: copy_from oid 25 from oid 21 current snap is 75 2024-09-11T03:41:01.849 INFO:tasks.rados.rados.2.smithi175.stdout:735: copy_from oid 3 from oid 19 current snap is 75 2024-09-11T03:41:01.849 INFO:tasks.rados.rados.2.smithi175.stdout:736: delete oid 17 current snap is 75 2024-09-11T03:41:01.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:01.852 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:01.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:01.853 INFO:tasks.rados.rados.2.smithi175.stdout:735: finishing copy_from to smithi17557870-3 2024-09-11T03:41:01.853 INFO:tasks.rados.rados.2.smithi175.stdout:735: got expected ENOENT (src dne) 2024-09-11T03:41:01.853 INFO:tasks.rados.rados.2.smithi175.stdout:735: finishing copy_from racing read to smithi17557870-3 2024-09-11T03:41:01.853 INFO:tasks.rados.rados.2.smithi175.stdout:735: done (4 left) 2024-09-11T03:41:01.853 INFO:tasks.rados.rados.2.smithi175.stdout:736: done (3 left) 2024-09-11T03:41:01.853 INFO:tasks.rados.rados.2.smithi175.stdout:737: delete oid 47 current snap is 75 2024-09-11T03:41:01.854 INFO:tasks.rados.rados.2.smithi175.stdout:737: done (3 left) 2024-09-11T03:41:01.854 INFO:tasks.rados.rados.2.smithi175.stdout:738: snap_remove snap 74 2024-09-11T03:41:01.953 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:01.975 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:01.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:01.976 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:02.077 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:02.098 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:02.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:02.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:02.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:01 smithi121 ceph-mon[30729]: pgmap v520: 47 pgs: 47 active+clean; 191 MiB data, 708 MiB used, 893 GiB / 894 GiB avail; 1.5 MiB/s rd, 1 op/s 2024-09-11T03:41:02.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:01 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'58984','7':'62260'}"}]': finished 2024-09-11T03:41:02.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:01 smithi121 ceph-mon[30729]: osdmap e244: 12 total, 12 up, 10 in 2024-09-11T03:41:02.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:01 smithi121 ceph-mon[30729]: osdmap e245: 12 total, 12 up, 10 in 2024-09-11T03:41:02.199 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:02.221 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:02.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:02.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:02.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:01 smithi084 ceph-mon[27254]: pgmap v520: 47 pgs: 47 active+clean; 191 MiB data, 708 MiB used, 893 GiB / 894 GiB avail; 1.5 MiB/s rd, 1 op/s 2024-09-11T03:41:02.223 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:01 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'58984','7':'62260'}"}]': finished 2024-09-11T03:41:02.223 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:01 smithi084 ceph-mon[27254]: osdmap e244: 12 total, 12 up, 10 in 2024-09-11T03:41:02.223 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:01 smithi084 ceph-mon[27254]: osdmap e245: 12 total, 12 up, 10 in 2024-09-11T03:41:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:01 smithi175 ceph-mon[30894]: pgmap v520: 47 pgs: 47 active+clean; 191 MiB data, 708 MiB used, 893 GiB / 894 GiB avail; 1.5 MiB/s rd, 1 op/s 2024-09-11T03:41:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:01 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'58984','7':'62260'}"}]': finished 2024-09-11T03:41:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:01 smithi175 ceph-mon[30894]: osdmap e244: 12 total, 12 up, 10 in 2024-09-11T03:41:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:01 smithi175 ceph-mon[30894]: osdmap e245: 12 total, 12 up, 10 in 2024-09-11T03:41:02.249 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T03:41:02.324 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:02.345 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:02.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:02.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:02.447 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:02.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:02.469 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:02.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:02.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:02.593 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:02.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:02.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:02.695 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:02.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:02.717 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:02.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:02.803 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-11T03:41:02.803 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-11T03:41:02.804 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:02.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:02.851 INFO:tasks.rados.rados.2.smithi175.stdout:738: done (3 left) 2024-09-11T03:41:02.852 INFO:tasks.rados.rados.2.smithi175.stdout:739: delete oid 46 current snap is 75 2024-09-11T03:41:02.852 INFO:tasks.rados.rados.2.smithi175.stdout:732: expect (ObjNum 215 snap 52 seq_num 215) 2024-09-11T03:41:02.853 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:41:02.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:02.871 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:02.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:02.971 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:02.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:02.993 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:02.994 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:03.023 INFO:tasks.rados.rados.2.smithi175.stdout:733: finishing copy_from racing read to smithi17557870-14 2024-09-11T03:41:03.023 INFO:tasks.rados.rados.2.smithi175.stdout:733: finishing copy_from to smithi17557870-14 2024-09-11T03:41:03.024 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 238 (ObjNum 258 snap 64 seq_num 258) dirty exists 2024-09-11T03:41:03.024 INFO:tasks.rados.rados.2.smithi175.stdout:734: finishing copy_from racing read to smithi17557870-25 2024-09-11T03:41:03.024 INFO:tasks.rados.rados.2.smithi175.stdout:734: finishing copy_from to smithi17557870-25 2024-09-11T03:41:03.024 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 218 (ObjNum 266 snap 67 seq_num 266) dirty exists 2024-09-11T03:41:03.024 INFO:tasks.rados.rados.2.smithi175.stdout:732: done (3 left) 2024-09-11T03:41:03.024 INFO:tasks.rados.rados.2.smithi175.stdout:733: done (2 left) 2024-09-11T03:41:03.024 INFO:tasks.rados.rados.2.smithi175.stdout:734: done (1 left) 2024-09-11T03:41:03.024 INFO:tasks.rados.rados.2.smithi175.stdout:739: done (0 left) 2024-09-11T03:41:03.024 INFO:tasks.rados.rados.2.smithi175.stdout:740: snap_create 2024-09-11T03:41:03.094 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:03.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:03.116 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:03.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:03.217 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:03.239 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:03.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:03.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:03.298 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:03.338 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:03.338 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:03.341 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:03.362 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:03.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:03.363 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:03.464 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:03.490 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:03.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:03.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:03.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:03.617 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:03.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:03.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:03.719 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:03.741 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:03.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:03.742 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:03.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:03.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:03.870 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:03.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:03.903 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:41:03.906 INFO:tasks.rados.rados.2.smithi175.stdout:740: done (0 left) 2024-09-11T03:41:03.906 INFO:tasks.rados.rados.2.smithi175.stdout:741: copy_from oid 32 from oid 12 current snap is 76 2024-09-11T03:41:03.906 INFO:tasks.rados.rados.2.smithi175.stdout:742: read oid 2 snap 59 2024-09-11T03:41:03.906 INFO:tasks.rados.rados.2.smithi175.stdout:742: expect (ObjNum 210 snap 51 seq_num 210) 2024-09-11T03:41:03.906 INFO:tasks.rados.rados.2.smithi175.stdout:743: read oid 47 snap -1 2024-09-11T03:41:03.906 INFO:tasks.rados.rados.2.smithi175.stdout:743: expect deleted 2024-09-11T03:41:03.906 INFO:tasks.rados.rados.2.smithi175.stdout:744: read oid 36 snap -1 2024-09-11T03:41:03.906 INFO:tasks.rados.rados.2.smithi175.stdout:744: expect (ObjNum 282 snap 73 seq_num 282) 2024-09-11T03:41:03.906 INFO:tasks.rados.rados.2.smithi175.stdout:745: read oid 29 snap -1 2024-09-11T03:41:03.907 INFO:tasks.rados.rados.2.smithi175.stdout:745: expect (ObjNum 280 snap 73 seq_num 280) 2024-09-11T03:41:03.907 INFO:tasks.rados.rados.2.smithi175.stdout:746: snap_remove snap 73 2024-09-11T03:41:03.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:03.994 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:03.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:03.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:04.009 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:03 smithi084 ceph-mon[27254]: 1.7 scrub starts 2024-09-11T03:41:04.010 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:03 smithi084 ceph-mon[27254]: 1.7 scrub ok 2024-09-11T03:41:04.010 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:03 smithi084 ceph-mon[27254]: pgmap v523: 47 pgs: 2 peering, 45 active+clean; 177 MiB data, 697 MiB used, 893 GiB / 894 GiB avail; 20 MiB/s rd, 1.5 MiB/s wr, 26 op/s; 35 KiB/s, 0 objects/s recovering 2024-09-11T03:41:04.010 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:03 smithi084 ceph-mon[27254]: 1.2 scrub starts 2024-09-11T03:41:04.010 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:03 smithi084 ceph-mon[27254]: 1.2 scrub ok 2024-09-11T03:41:04.010 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:03 smithi084 ceph-mon[27254]: osdmap e246: 12 total, 12 up, 10 in 2024-09-11T03:41:04.010 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:03 smithi084 ceph-mon[27254]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-11T03:41:04.010 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:03 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4144276756' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:04.025 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:41:04.025 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:41:04.025 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:41:04.025 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:41:04.096 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:04.119 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:04.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:04.120 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:03 smithi121 ceph-mon[30729]: 1.7 scrub starts 2024-09-11T03:41:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:03 smithi121 ceph-mon[30729]: 1.7 scrub ok 2024-09-11T03:41:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:03 smithi121 ceph-mon[30729]: pgmap v523: 47 pgs: 2 peering, 45 active+clean; 177 MiB data, 697 MiB used, 893 GiB / 894 GiB avail; 20 MiB/s rd, 1.5 MiB/s wr, 26 op/s; 35 KiB/s, 0 objects/s recovering 2024-09-11T03:41:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:03 smithi121 ceph-mon[30729]: 1.2 scrub starts 2024-09-11T03:41:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:03 smithi121 ceph-mon[30729]: 1.2 scrub ok 2024-09-11T03:41:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:03 smithi121 ceph-mon[30729]: osdmap e246: 12 total, 12 up, 10 in 2024-09-11T03:41:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:03 smithi121 ceph-mon[30729]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-11T03:41:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:03 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4144276756' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:04.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:03 smithi175 ceph-mon[30894]: 1.7 scrub starts 2024-09-11T03:41:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:03 smithi175 ceph-mon[30894]: 1.7 scrub ok 2024-09-11T03:41:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:03 smithi175 ceph-mon[30894]: pgmap v523: 47 pgs: 2 peering, 45 active+clean; 177 MiB data, 697 MiB used, 893 GiB / 894 GiB avail; 20 MiB/s rd, 1.5 MiB/s wr, 26 op/s; 35 KiB/s, 0 objects/s recovering 2024-09-11T03:41:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:03 smithi175 ceph-mon[30894]: 1.2 scrub starts 2024-09-11T03:41:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:03 smithi175 ceph-mon[30894]: 1.2 scrub ok 2024-09-11T03:41:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:03 smithi175 ceph-mon[30894]: osdmap e246: 12 total, 12 up, 10 in 2024-09-11T03:41:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:03 smithi175 ceph-mon[30894]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-11T03:41:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:03 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4144276756' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:04.247 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:04.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:04.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:04.349 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:04.373 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:04.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:04.474 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:04.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:04.502 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:04.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:04.603 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:04.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:04.626 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:04.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:04.728 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:04.752 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:04.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:04.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:04.879 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:04.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:04.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:04.913 INFO:tasks.rados.rados.2.smithi175.stdout:746: done (5 left) 2024-09-11T03:41:04.913 INFO:tasks.rados.rados.2.smithi175.stdout:747: snap_create 2024-09-11T03:41:04.913 INFO:tasks.rados.rados.2.smithi175.stdout:745: expect (ObjNum 280 snap 73 seq_num 280) 2024-09-11T03:41:04.981 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:05.002 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:05.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:05.003 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:05.009 INFO:teuthology.orchestra.run.smithi084.stdout:moved 0 / 141 (0%) 2024-09-11T03:41:05.010 INFO:teuthology.orchestra.run.smithi084.stdout:avg 14.1 2024-09-11T03:41:05.010 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 2.21133 -> 2.21133 (expected baseline 3.5623) 2024-09-11T03:41:05.010 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.5 with 11 -> 11 pgs (0.780142 -> 0.780142 * mean) 2024-09-11T03:41:05.010 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.7 with 18 -> 18 pgs (1.2766 -> 1.2766 * mean) 2024-09-11T03:41:05.010 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:41:05.010 INFO:teuthology.orchestra.run.smithi084.stdout:oload 1000 2024-09-11T03:41:05.010 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 1 2024-09-11T03:41:05.010 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2024-09-11T03:41:05.010 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0008 2024-09-11T03:41:05.010 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0077 2024-09-11T03:41:05.011 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2024-09-11T03:41:05.061 INFO:tasks.rados.rados.2.smithi175.stdout:741: finishing copy_from racing read to smithi17557870-32 2024-09-11T03:41:05.061 INFO:tasks.rados.rados.2.smithi175.stdout:744: expect (ObjNum 282 snap 73 seq_num 282) 2024-09-11T03:41:05.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:05.125 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:05.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:05.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:04 smithi084 ceph-mon[27254]: 2.4 scrub starts 2024-09-11T03:41:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:04 smithi084 ceph-mon[27254]: 2.4 scrub ok 2024-09-11T03:41:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:04 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:41:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:04 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4144276756' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:04 smithi084 ceph-mon[27254]: osdmap e247: 12 total, 12 up, 10 in 2024-09-11T03:41:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:04 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:41:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:04 smithi084 ceph-mon[27254]: 2.b scrub ok 2024-09-11T03:41:05.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:04 smithi121 ceph-mon[30729]: 2.4 scrub starts 2024-09-11T03:41:05.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:04 smithi121 ceph-mon[30729]: 2.4 scrub ok 2024-09-11T03:41:05.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:04 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:41:05.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:04 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4144276756' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:05.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:04 smithi121 ceph-mon[30729]: osdmap e247: 12 total, 12 up, 10 in 2024-09-11T03:41:05.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:04 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:41:05.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:04 smithi121 ceph-mon[30729]: 2.b scrub ok 2024-09-11T03:41:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:04 smithi175 ceph-mon[30894]: 2.4 scrub starts 2024-09-11T03:41:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:04 smithi175 ceph-mon[30894]: 2.4 scrub ok 2024-09-11T03:41:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:04 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:41:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:04 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4144276756' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:04 smithi175 ceph-mon[30894]: osdmap e247: 12 total, 12 up, 10 in 2024-09-11T03:41:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:04 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:41:05.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:04 smithi175 ceph-mon[30894]: 2.b scrub ok 2024-09-11T03:41:05.228 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:05.238 INFO:tasks.rados.rados.2.smithi175.stdout:742: expect (ObjNum 210 snap 51 seq_num 210) 2024-09-11T03:41:05.248 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:05.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:05.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:05.351 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:05.376 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:05.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:05.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:05.445 INFO:tasks.rados.rados.2.smithi175.stdout:741: finishing copy_from to smithi17557870-32 2024-09-11T03:41:05.445 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 223 (ObjNum 277 snap 73 seq_num 277) dirty exists 2024-09-11T03:41:05.462 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T03:41:05.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:05.504 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:05.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:05.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:05.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:05.627 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:05.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:05.728 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:05.750 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:05.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:05.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:05.878 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:05.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:05.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:05.921 INFO:tasks.rados.rados.2.smithi175.stdout:741: done (5 left) 2024-09-11T03:41:05.921 INFO:tasks.rados.rados.2.smithi175.stdout:742: done (4 left) 2024-09-11T03:41:05.921 INFO:tasks.rados.rados.2.smithi175.stdout:743: done (3 left) 2024-09-11T03:41:05.921 INFO:tasks.rados.rados.2.smithi175.stdout:744: done (2 left) 2024-09-11T03:41:05.921 INFO:tasks.rados.rados.2.smithi175.stdout:745: done (1 left) 2024-09-11T03:41:05.921 INFO:tasks.rados.rados.2.smithi175.stdout:747: done (0 left) 2024-09-11T03:41:05.921 INFO:tasks.rados.rados.2.smithi175.stdout:748: delete oid 13 current snap is 77 2024-09-11T03:41:05.922 INFO:tasks.rados.rados.2.smithi175.stdout:748: done (0 left) 2024-09-11T03:41:05.923 INFO:tasks.rados.rados.2.smithi175.stdout:749: rollback oid 40 current snap is 77 2024-09-11T03:41:05.923 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 40 to 66 2024-09-11T03:41:05.923 INFO:tasks.rados.rados.2.smithi175.stdout:750: copy_from oid 24 from oid 49 current snap is 77 2024-09-11T03:41:05.923 INFO:tasks.rados.rados.2.smithi175.stdout:751: delete oid 36 current snap is 77 2024-09-11T03:41:05.925 INFO:tasks.rados.rados.2.smithi175.stdout:749: finishing rollback tid 1 to smithi17557870-40 2024-09-11T03:41:05.925 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 118 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T03:41:05.926 INFO:tasks.rados.rados.2.smithi175.stdout:749: done (2 left) 2024-09-11T03:41:05.926 INFO:tasks.rados.rados.2.smithi175.stdout:751: done (1 left) 2024-09-11T03:41:05.926 INFO:tasks.rados.rados.2.smithi175.stdout:752: copy_from oid 47 from oid 8 current snap is 77 2024-09-11T03:41:05.927 INFO:tasks.rados.rados.2.smithi175.stdout:753: read oid 25 snap 75 2024-09-11T03:41:05.927 INFO:tasks.rados.rados.2.smithi175.stdout:753: expect (ObjNum 266 snap 67 seq_num 266) 2024-09-11T03:41:05.927 INFO:tasks.rados.rados.2.smithi175.stdout:754: delete oid 26 current snap is 77 2024-09-11T03:41:05.927 INFO:tasks.rados.rados.2.smithi175.stdout:752: finishing copy_from to smithi17557870-47 2024-09-11T03:41:05.927 INFO:tasks.rados.rados.2.smithi175.stdout:752: got expected ENOENT (src dne) 2024-09-11T03:41:05.927 INFO:tasks.rados.rados.2.smithi175.stdout:752: finishing copy_from racing read to smithi17557870-47 2024-09-11T03:41:05.928 INFO:tasks.rados.rados.2.smithi175.stdout:752: done (3 left) 2024-09-11T03:41:05.928 INFO:tasks.rados.rados.2.smithi175.stdout:754: done (2 left) 2024-09-11T03:41:05.928 INFO:tasks.rados.rados.2.smithi175.stdout:755: copy_from oid 28 from oid 17 current snap is 77 2024-09-11T03:41:05.929 INFO:tasks.rados.rados.2.smithi175.stdout:756: copy_from oid 26 from oid 46 current snap is 77 2024-09-11T03:41:05.929 INFO:tasks.rados.rados.2.smithi175.stdout:757: read oid 22 snap -1 2024-09-11T03:41:05.929 INFO:tasks.rados.rados.2.smithi175.stdout:757: expect deleted 2024-09-11T03:41:05.929 INFO:tasks.rados.rados.2.smithi175.stdout:758: rollback oid 30 current snap is 77 2024-09-11T03:41:05.929 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 30 to 66 2024-09-11T03:41:05.929 INFO:tasks.rados.rados.2.smithi175.stdout:759: snap_create 2024-09-11T03:41:05.929 INFO:tasks.rados.rados.2.smithi175.stdout:756: finishing copy_from to smithi17557870-26 2024-09-11T03:41:05.929 INFO:tasks.rados.rados.2.smithi175.stdout:756: got expected ENOENT (src dne) 2024-09-11T03:41:05.930 INFO:tasks.rados.rados.2.smithi175.stdout:756: finishing copy_from racing read to smithi17557870-26 2024-09-11T03:41:05.930 INFO:tasks.rados.rados.2.smithi175.stdout:755: finishing copy_from to smithi17557870-28 2024-09-11T03:41:05.930 INFO:tasks.rados.rados.2.smithi175.stdout:755: got expected ENOENT (src dne) 2024-09-11T03:41:05.930 INFO:tasks.rados.rados.2.smithi175.stdout:755: finishing copy_from racing read to smithi17557870-28 2024-09-11T03:41:05.934 INFO:tasks.rados.rados.2.smithi175.stdout:750: finishing copy_from racing read to smithi17557870-24 2024-09-11T03:41:05.934 INFO:tasks.rados.rados.2.smithi175.stdout:758: finishing rollback tid 0 to smithi17557870-30 2024-09-11T03:41:05.934 INFO:tasks.rados.rados.2.smithi175.stdout:758: finishing rollback tid 1 to smithi17557870-30 2024-09-11T03:41:05.934 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 249 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T03:41:05.938 INFO:tasks.rados.rados.2.smithi175.stdout:753: expect (ObjNum 266 snap 67 seq_num 266) 2024-09-11T03:41:05.946 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:41:05.947 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:41:05.953 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:41:05.953 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:41:05.969 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:41:05.969 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:41:05.969 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:05 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:41:05.969 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:05 smithi084 ceph-mon[27254]: 2.c scrub ok 2024-09-11T03:41:05.969 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:05 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:41:05.969 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:05 smithi084 ceph-mon[27254]: 2.2 scrub ok 2024-09-11T03:41:05.970 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:05 smithi084 ceph-mon[27254]: pgmap v526: 47 pgs: 2 peering, 45 active+clean; 185 MiB data, 700 MiB used, 893 GiB / 894 GiB avail; 21 MiB/s rd, 3.0 MiB/s wr, 31 op/s; 35 KiB/s, 0 objects/s recovering 2024-09-11T03:41:05.970 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:05 smithi084 ceph-mon[27254]: osdmap e248: 12 total, 12 up, 10 in 2024-09-11T03:41:05.970 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:05 smithi084 ceph-mon[27254]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:41:05.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:05.986 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:41:05.986 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:41:06.002 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:06.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:06.003 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:41:06.003 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:41:06.021 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:41:06.021 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:41:06.038 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:41:06.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:41:06.054 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:41:06.054 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:41:06.070 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:41:06.070 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:41:06.086 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:41:06.086 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:41:06.087 INFO:tasks.rados.rados.2.smithi175.stdout:750: finishing copy_from to smithi17557870-24 2024-09-11T03:41:06.087 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 160 (ObjNum 267 snap 68 seq_num 267) dirty exists 2024-09-11T03:41:06.103 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:41:06.103 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:41:06.103 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:06.119 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:41:06.120 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:41:06.124 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:06.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:06.125 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:06.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:05 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:41:06.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:05 smithi121 ceph-mon[30729]: 2.c scrub ok 2024-09-11T03:41:06.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:05 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:41:06.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:05 smithi121 ceph-mon[30729]: 2.2 scrub ok 2024-09-11T03:41:06.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:05 smithi121 ceph-mon[30729]: pgmap v526: 47 pgs: 2 peering, 45 active+clean; 185 MiB data, 700 MiB used, 893 GiB / 894 GiB avail; 21 MiB/s rd, 3.0 MiB/s wr, 31 op/s; 35 KiB/s, 0 objects/s recovering 2024-09-11T03:41:06.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:05 smithi121 ceph-mon[30729]: osdmap e248: 12 total, 12 up, 10 in 2024-09-11T03:41:06.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:05 smithi121 ceph-mon[30729]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:41:06.226 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:05 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:41:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:05 smithi175 ceph-mon[30894]: 2.c scrub ok 2024-09-11T03:41:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:05 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:41:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:05 smithi175 ceph-mon[30894]: 2.2 scrub ok 2024-09-11T03:41:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:05 smithi175 ceph-mon[30894]: pgmap v526: 47 pgs: 2 peering, 45 active+clean; 185 MiB data, 700 MiB used, 893 GiB / 894 GiB avail; 21 MiB/s rd, 3.0 MiB/s wr, 31 op/s; 35 KiB/s, 0 objects/s recovering 2024-09-11T03:41:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:05 smithi175 ceph-mon[30894]: osdmap e248: 12 total, 12 up, 10 in 2024-09-11T03:41:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:05 smithi175 ceph-mon[30894]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:41:06.248 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:06.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:06.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:06.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:06.379 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:06.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:06.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:06.481 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:06.502 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:06.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:06.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:06.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:06.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:06.627 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:06.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:06.642 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:41:06.728 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:06.749 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:06.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:06.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:06.850 INFO:tasks.rados.rados.2.smithi175.stdout:750: done (6 left) 2024-09-11T03:41:06.850 INFO:tasks.rados.rados.2.smithi175.stdout:753: done (5 left) 2024-09-11T03:41:06.850 INFO:tasks.rados.rados.2.smithi175.stdout:755: done (4 left) 2024-09-11T03:41:06.851 INFO:tasks.rados.rados.2.smithi175.stdout:756: done (3 left) 2024-09-11T03:41:06.851 INFO:tasks.rados.rados.2.smithi175.stdout:757: done (2 left) 2024-09-11T03:41:06.851 INFO:tasks.rados.rados.2.smithi175.stdout:758: done (1 left) 2024-09-11T03:41:06.851 INFO:tasks.rados.rados.2.smithi175.stdout:759: done (0 left) 2024-09-11T03:41:06.851 INFO:tasks.rados.rados.2.smithi175.stdout:760: snap_remove snap 77 2024-09-11T03:41:06.851 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:06.877 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:06.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:06.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:06.907 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-11T03:41:06.907 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd test-reweight-by-utilization 110 3.0 2024-09-11T03:41:06.979 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:07.004 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:07.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:07.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:07.106 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:07.129 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:07.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:07.131 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:07.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:06 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1846571000' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:07.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:06 smithi084 ceph-mon[27254]: osdmap e249: 12 total, 12 up, 10 in 2024-09-11T03:41:07.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:06 smithi084 ceph-mon[27254]: osdmap e250: 12 total, 12 up, 10 in 2024-09-11T03:41:07.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:06 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1846571000' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:07.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:06 smithi121 ceph-mon[30729]: osdmap e249: 12 total, 12 up, 10 in 2024-09-11T03:41:07.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:06 smithi121 ceph-mon[30729]: osdmap e250: 12 total, 12 up, 10 in 2024-09-11T03:41:07.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:06 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1846571000' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:07.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:06 smithi175 ceph-mon[30894]: osdmap e249: 12 total, 12 up, 10 in 2024-09-11T03:41:07.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:06 smithi175 ceph-mon[30894]: osdmap e250: 12 total, 12 up, 10 in 2024-09-11T03:41:07.232 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:07.255 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:07.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:07.256 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:07.357 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:07.369 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:07.379 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:07.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:07.380 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:07.400 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:07.481 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:07.502 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:07.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:07.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:07.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:07.630 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:07.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:07.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:07.732 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:07.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:07.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:07.852 INFO:tasks.rados.rados.2.smithi175.stdout:760: done (0 left) 2024-09-11T03:41:07.852 INFO:tasks.rados.rados.2.smithi175.stdout:761: snap_remove snap 71 2024-09-11T03:41:07.856 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:07.882 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:07.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:07.883 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:07.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:08.010 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:08.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:08.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:08.036 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:07 smithi084 ceph-mon[27254]: pgmap v529: 47 pgs: 47 active+clean; 190 MiB data, 705 MiB used, 893 GiB / 894 GiB avail; 9.1 MiB/s rd, 2.6 MiB/s wr, 11 op/s 2024-09-11T03:41:08.036 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:07 smithi084 ceph-mon[27254]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-11T03:41:08.036 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:07 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:41:08.036 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:07 smithi084 ceph-mon[27254]: osdmap e251: 12 total, 12 up, 10 in 2024-09-11T03:41:08.059 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:41:08.059 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:41:08.112 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:08.138 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:08.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:08.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:07 smithi175 ceph-mon[30894]: pgmap v529: 47 pgs: 47 active+clean; 190 MiB data, 705 MiB used, 893 GiB / 894 GiB avail; 9.1 MiB/s rd, 2.6 MiB/s wr, 11 op/s 2024-09-11T03:41:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:07 smithi175 ceph-mon[30894]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-11T03:41:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:07 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:41:08.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:07 smithi175 ceph-mon[30894]: osdmap e251: 12 total, 12 up, 10 in 2024-09-11T03:41:08.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:08.261 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:08.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:08.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:08.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:08.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:07 smithi121 ceph-mon[30729]: pgmap v529: 47 pgs: 47 active+clean; 190 MiB data, 705 MiB used, 893 GiB / 894 GiB avail; 9.1 MiB/s rd, 2.6 MiB/s wr, 11 op/s 2024-09-11T03:41:08.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:07 smithi121 ceph-mon[30729]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-11T03:41:08.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:07 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:41:08.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:07 smithi121 ceph-mon[30729]: osdmap e251: 12 total, 12 up, 10 in 2024-09-11T03:41:08.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:08.387 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:08.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:08.488 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:08.514 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:08.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:08.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:08.616 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:08.639 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:08.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:08.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:08.654 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T03:41:08.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:08.764 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:08.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:08.857 INFO:tasks.rados.rados.2.smithi175.stdout:761: done (0 left) 2024-09-11T03:41:08.858 INFO:tasks.rados.rados.2.smithi175.stdout:762: write oid 3 current snap is 78 2024-09-11T03:41:08.858 INFO:tasks.rados.rados.2.smithi175.stdout:762: seq_num 290 ranges {474030=643624,1520131=735769} 2024-09-11T03:41:08.862 INFO:tasks.rados.rados.2.smithi175.stdout:762: writing smithi17557870-3 from 474030 to 1117654 tid 1 2024-09-11T03:41:08.865 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:08.869 INFO:tasks.rados.rados.2.smithi175.stdout:762: writing smithi17557870-3 from 1520131 to 2255900 tid 2 2024-09-11T03:41:08.869 INFO:tasks.rados.rados.2.smithi175.stdout:763: read oid 29 snap -1 2024-09-11T03:41:08.869 INFO:tasks.rados.rados.2.smithi175.stdout:763: expect (ObjNum 280 snap 73 seq_num 280) 2024-09-11T03:41:08.869 INFO:tasks.rados.rados.2.smithi175.stdout:764: delete oid 43 current snap is 78 2024-09-11T03:41:08.869 INFO:tasks.rados.rados.2.smithi175.stdout:762: finishing write tid 1 to smithi17557870-3 2024-09-11T03:41:08.872 INFO:tasks.rados.rados.2.smithi175.stdout:762: finishing write tid 2 to smithi17557870-3 2024-09-11T03:41:08.872 INFO:tasks.rados.rados.2.smithi175.stdout:762: finishing write tid 3 to smithi17557870-3 2024-09-11T03:41:08.872 INFO:tasks.rados.rados.2.smithi175.stdout:762: finishing write tid 4 to smithi17557870-3 2024-09-11T03:41:08.873 INFO:tasks.rados.rados.2.smithi175.stdout:762: oid 3 updating version 0 to 292 2024-09-11T03:41:08.873 INFO:tasks.rados.rados.2.smithi175.stdout:762: oid 3 version 292 is already newer than 291 2024-09-11T03:41:08.873 INFO:tasks.rados.rados.2.smithi175.stdout:762: oid 3 updating version 292 to 293 2024-09-11T03:41:08.873 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 293 (ObjNum 290 snap 78 seq_num 290) dirty exists 2024-09-11T03:41:08.873 INFO:tasks.rados.rados.2.smithi175.stdout:762: left oid 3 (ObjNum 290 snap 78 seq_num 290) 2024-09-11T03:41:08.873 INFO:tasks.rados.rados.2.smithi175.stdout:762: done (2 left) 2024-09-11T03:41:08.873 INFO:tasks.rados.rados.2.smithi175.stdout:764: done (1 left) 2024-09-11T03:41:08.873 INFO:tasks.rados.rados.2.smithi175.stdout:765: write oid 45 current snap is 78 2024-09-11T03:41:08.873 INFO:tasks.rados.rados.2.smithi175.stdout:765: seq_num 292 ranges {507644=794122,1920742=623085} 2024-09-11T03:41:08.879 INFO:tasks.rados.rados.2.smithi175.stdout:765: writing smithi17557870-45 from 507644 to 1301766 tid 1 2024-09-11T03:41:08.887 INFO:tasks.rados.rados.2.smithi175.stdout:765: writing smithi17557870-45 from 1920742 to 2543827 tid 2 2024-09-11T03:41:08.887 INFO:tasks.rados.rados.2.smithi175.stdout:766: read oid 44 snap -1 2024-09-11T03:41:08.887 INFO:tasks.rados.rados.2.smithi175.stdout:766: expect (ObjNum 171 snap 39 seq_num 171) 2024-09-11T03:41:08.887 INFO:tasks.rados.rados.2.smithi175.stdout:767: snap_create 2024-09-11T03:41:08.887 INFO:tasks.rados.rados.2.smithi175.stdout:763: expect (ObjNum 280 snap 73 seq_num 280) 2024-09-11T03:41:08.894 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:08.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:08.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:08.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:09.011 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:09.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:09.018 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:09.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:09.035 INFO:tasks.rados.rados.2.smithi175.stdout:765: finishing write tid 1 to smithi17557870-45 2024-09-11T03:41:09.035 INFO:tasks.rados.rados.2.smithi175.stdout:765: finishing write tid 2 to smithi17557870-45 2024-09-11T03:41:09.035 INFO:tasks.rados.rados.2.smithi175.stdout:765: finishing write tid 3 to smithi17557870-45 2024-09-11T03:41:09.035 INFO:tasks.rados.rados.2.smithi175.stdout:765: finishing write tid 4 to smithi17557870-45 2024-09-11T03:41:09.035 INFO:tasks.rados.rados.2.smithi175.stdout:765: oid 45 updating version 0 to 175 2024-09-11T03:41:09.035 INFO:tasks.rados.rados.2.smithi175.stdout:765: oid 45 updating version 175 to 176 2024-09-11T03:41:09.035 INFO:tasks.rados.rados.2.smithi175.stdout:765: oid 45 version 176 is already newer than 174 2024-09-11T03:41:09.035 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 176 (ObjNum 292 snap 78 seq_num 292) dirty exists 2024-09-11T03:41:09.035 INFO:tasks.rados.rados.2.smithi175.stdout:765: left oid 45 (ObjNum 292 snap 78 seq_num 292) 2024-09-11T03:41:09.035 INFO:tasks.rados.rados.2.smithi175.stdout:766: expect (ObjNum 171 snap 39 seq_num 171) 2024-09-11T03:41:09.118 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:09.139 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:09.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:09.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:09.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:08 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:41:09.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:08 smithi121 ceph-mon[30729]: 2.d scrub ok 2024-09-11T03:41:09.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:08 smithi121 ceph-mon[30729]: osdmap e252: 12 total, 12 up, 10 in 2024-09-11T03:41:09.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:08 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:41:09.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:08 smithi175 ceph-mon[30894]: 2.d scrub ok 2024-09-11T03:41:09.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:08 smithi175 ceph-mon[30894]: osdmap e252: 12 total, 12 up, 10 in 2024-09-11T03:41:09.242 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:09.260 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:08 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:41:09.261 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:08 smithi084 ceph-mon[27254]: 2.d scrub ok 2024-09-11T03:41:09.261 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:08 smithi084 ceph-mon[27254]: osdmap e252: 12 total, 12 up, 10 in 2024-09-11T03:41:09.262 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:09.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:09.263 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:09.364 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:09.385 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:09.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:09.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:09.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:41:09.488 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:09.501 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:09.508 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:09.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:09.509 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:09.514 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:41:09.514 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:41:09.610 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:09.632 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:09.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:09.633 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:09.734 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:09.755 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:09.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:09.756 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:09.851 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:41:09.854 INFO:tasks.rados.rados.2.smithi175.stdout:763: done (3 left) 2024-09-11T03:41:09.854 INFO:tasks.rados.rados.2.smithi175.stdout:765: done (2 left) 2024-09-11T03:41:09.854 INFO:tasks.rados.rados.2.smithi175.stdout:766: done (1 left) 2024-09-11T03:41:09.854 INFO:tasks.rados.rados.2.smithi175.stdout:767: done (0 left) 2024-09-11T03:41:09.854 INFO:tasks.rados.rados.2.smithi175.stdout:768: snap_remove snap 66 2024-09-11T03:41:09.857 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:09.879 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:09.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:09.980 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:10.004 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:10.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:10.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:10.106 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:10.128 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:10.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:10.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:10.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:09 smithi084 ceph-mon[27254]: pgmap v532: 47 pgs: 47 active+clean; 190 MiB data, 711 MiB used, 893 GiB / 894 GiB avail; 9.8 MiB/s rd, 1.8 MiB/s wr, 9 op/s 2024-09-11T03:41:10.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:09 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/410020005' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:41:10.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:09 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:10.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:09 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/410020005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:41:10.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:09 smithi084 ceph-mon[27254]: osdmap e253: 12 total, 12 up, 10 in 2024-09-11T03:41:10.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T03:41:10.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:09 smithi121 ceph-mon[30729]: pgmap v532: 47 pgs: 47 active+clean; 190 MiB data, 711 MiB used, 893 GiB / 894 GiB avail; 9.8 MiB/s rd, 1.8 MiB/s wr, 9 op/s 2024-09-11T03:41:10.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:09 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/410020005' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:41:10.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:09 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:10.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:09 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/410020005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:41:10.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:09 smithi121 ceph-mon[30729]: osdmap e253: 12 total, 12 up, 10 in 2024-09-11T03:41:10.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:09 smithi175 ceph-mon[30894]: pgmap v532: 47 pgs: 47 active+clean; 190 MiB data, 711 MiB used, 893 GiB / 894 GiB avail; 9.8 MiB/s rd, 1.8 MiB/s wr, 9 op/s 2024-09-11T03:41:10.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:09 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/410020005' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:41:10.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:09 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:10.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:09 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/410020005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:41:10.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:09 smithi175 ceph-mon[30894]: osdmap e253: 12 total, 12 up, 10 in 2024-09-11T03:41:10.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:10.255 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:10.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:10.356 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:10.382 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:10.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:10.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:10.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:10.509 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:10.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:10.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:10.636 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:10.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:10.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:10.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:10.760 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:10.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:10.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:10.773 INFO:teuthology.orchestra.run.smithi084.stdout:moved 7 / 141 (4.96454%) 2024-09-11T03:41:10.773 INFO:teuthology.orchestra.run.smithi084.stdout:avg 14.1 2024-09-11T03:41:10.773 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 2.21133 -> 2.50799 (expected baseline 3.5623) 2024-09-11T03:41:10.773 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.5 with 11 -> 13 pgs (0.780142 -> 0.921986 * mean) 2024-09-11T03:41:10.773 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.7 with 18 -> 16 pgs (1.2766 -> 1.13475 * mean) 2024-09-11T03:41:10.773 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:41:10.773 INFO:teuthology.orchestra.run.smithi084.stdout:oload 110 2024-09-11T03:41:10.773 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 3 2024-09-11T03:41:10.774 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2024-09-11T03:41:10.774 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0008 2024-09-11T03:41:10.774 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0009 2024-09-11T03:41:10.774 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3 weight 0.9000 -> 0.6417 2024-09-11T03:41:10.774 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7 weight 0.9500 -> 0.8256 2024-09-11T03:41:10.774 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9 weight 1.0000 -> 0.8957 2024-09-11T03:41:10.774 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2024-09-11T03:41:10.855 INFO:tasks.rados.rados.2.smithi175.stdout:768: done (0 left) 2024-09-11T03:41:10.855 INFO:tasks.rados.rados.2.smithi175.stdout:769: read oid 16 snap -1 2024-09-11T03:41:10.856 INFO:tasks.rados.rados.2.smithi175.stdout:769: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:41:10.856 INFO:tasks.rados.rados.2.smithi175.stdout:770: snap_remove snap 78 2024-09-11T03:41:10.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:10.887 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:10.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:10.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:11.013 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:11.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:11.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:11.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:10 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:41:11.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:10 smithi084 ceph-mon[27254]: 2.e scrub ok 2024-09-11T03:41:11.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:10 smithi084 ceph-mon[27254]: osdmap e254: 12 total, 12 up, 10 in 2024-09-11T03:41:11.100 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:41:10 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: 2024-09-11T03:41:10.773+0000 7fab40ce9640 -1 mgr.server reply reply (3) No such process no change 2024-09-11T03:41:11.115 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:11.136 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:11.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:11.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:11.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:10 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:41:11.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:10 smithi175 ceph-mon[30894]: 2.e scrub ok 2024-09-11T03:41:11.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:10 smithi175 ceph-mon[30894]: osdmap e254: 12 total, 12 up, 10 in 2024-09-11T03:41:11.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:11.263 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:11.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:11.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:11.365 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:11.388 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:11.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:11.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:11.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:10 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:41:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:10 smithi121 ceph-mon[30729]: 2.e scrub ok 2024-09-11T03:41:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:10 smithi121 ceph-mon[30729]: osdmap e254: 12 total, 12 up, 10 in 2024-09-11T03:41:11.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:11.516 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:11.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:11.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:11.618 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:11.640 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:11.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:11.742 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:11.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:11.765 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:11.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:11.800 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:11.822 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:11.853 INFO:tasks.rados.rados.2.smithi175.stdout:770: done (1 left) 2024-09-11T03:41:11.853 INFO:tasks.rados.rados.2.smithi175.stdout:771: write (excl) oid 34 current snap is 79 2024-09-11T03:41:11.853 INFO:tasks.rados.rados.2.smithi175.stdout:771: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2024-09-11T03:41:11.858 INFO:tasks.rados.rados.2.smithi175.stdout:771: writing smithi17557870-34 from 524451 to 993822 tid 1 2024-09-11T03:41:11.864 INFO:tasks.rados.rados.2.smithi175.stdout:771: writing smithi17557870-34 from 1579224 to 2062320 tid 2 2024-09-11T03:41:11.866 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:11.869 INFO:tasks.rados.rados.2.smithi175.stdout:771: writing smithi17557870-34 from 2529878 to 2924451 tid 3 2024-09-11T03:41:11.869 INFO:tasks.rados.rados.2.smithi175.stdout:772: rollback oid 13 current snap is 79 2024-09-11T03:41:11.869 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 13 to 76 2024-09-11T03:41:11.870 INFO:tasks.rados.rados.2.smithi175.stdout:769: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:41:11.887 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:11.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:11.888 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:11.989 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:11.994 INFO:tasks.rados.rados.2.smithi175.stdout:771: finishing write tid 1 to smithi17557870-34 2024-09-11T03:41:11.994 INFO:tasks.rados.rados.2.smithi175.stdout:771: finishing write tid 2 to smithi17557870-34 2024-09-11T03:41:11.994 INFO:tasks.rados.rados.2.smithi175.stdout:769: done (2 left) 2024-09-11T03:41:11.994 INFO:tasks.rados.rados.2.smithi175.stdout:773: read oid 46 snap -1 2024-09-11T03:41:11.994 INFO:tasks.rados.rados.2.smithi175.stdout:773: expect deleted 2024-09-11T03:41:11.995 INFO:tasks.rados.rados.2.smithi175.stdout:774: delete oid 4 current snap is 79 2024-09-11T03:41:11.995 INFO:tasks.rados.rados.2.smithi175.stdout:771: finishing write tid 3 to smithi17557870-34 2024-09-11T03:41:11.995 INFO:tasks.rados.rados.2.smithi175.stdout:771: finishing write tid 4 to smithi17557870-34 2024-09-11T03:41:11.995 INFO:tasks.rados.rados.2.smithi175.stdout:772: finishing rollback tid 1 to smithi17557870-13 2024-09-11T03:41:11.995 INFO:tasks.rados.rados.2.smithi175.stdout:771: finishing write tid 5 to smithi17557870-34 2024-09-11T03:41:11.995 INFO:tasks.rados.rados.2.smithi175.stdout:771: oid 34 updating version 0 to 140 2024-09-11T03:41:11.995 INFO:tasks.rados.rados.2.smithi175.stdout:771: oid 34 version 140 is already newer than 138 2024-09-11T03:41:11.995 INFO:tasks.rados.rados.2.smithi175.stdout:771: oid 34 version 140 is already newer than 137 2024-09-11T03:41:11.995 INFO:tasks.rados.rados.2.smithi175.stdout:771: oid 34 version 140 is already newer than 139 2024-09-11T03:41:11.995 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 140 (ObjNum 293 snap 79 seq_num 293) dirty exists 2024-09-11T03:41:11.995 INFO:tasks.rados.rados.2.smithi175.stdout:771: left oid 34 (ObjNum 293 snap 79 seq_num 293) 2024-09-11T03:41:11.995 INFO:tasks.rados.rados.2.smithi175.stdout:772: finishing rollback tid 2 to smithi17557870-13 2024-09-11T03:41:11.995 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 244 (ObjNum 257 snap 64 seq_num 257) dirty exists 2024-09-11T03:41:11.996 INFO:tasks.rados.rados.2.smithi175.stdout:771: done (3 left) 2024-09-11T03:41:11.996 INFO:tasks.rados.rados.2.smithi175.stdout:772: done (2 left) 2024-09-11T03:41:11.996 INFO:tasks.rados.rados.2.smithi175.stdout:774: done (1 left) 2024-09-11T03:41:11.997 INFO:tasks.rados.rados.2.smithi175.stdout:775: copy_from oid 49 from oid 5 current snap is 79 2024-09-11T03:41:11.997 INFO:tasks.rados.rados.2.smithi175.stdout:776: copy_from oid 12 from oid 24 current snap is 79 2024-09-11T03:41:11.997 INFO:tasks.rados.rados.2.smithi175.stdout:777: read oid 29 snap -1 2024-09-11T03:41:11.997 INFO:tasks.rados.rados.2.smithi175.stdout:777: expect (ObjNum 280 snap 73 seq_num 280) 2024-09-11T03:41:11.997 INFO:tasks.rados.rados.2.smithi175.stdout:778: write (excl) oid 7 current snap is 79 2024-09-11T03:41:11.997 INFO:tasks.rados.rados.2.smithi175.stdout:778: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2024-09-11T03:41:12.001 INFO:tasks.rados.rados.2.smithi175.stdout:778: writing smithi17557870-7 from 558065 to 1177934 tid 1 2024-09-11T03:41:12.007 INFO:tasks.rados.rados.2.smithi175.stdout:778: writing smithi17557870-7 from 1579835 to 2066600 tid 2 2024-09-11T03:41:12.012 INFO:tasks.rados.rados.2.smithi175.stdout:778: writing smithi17557870-7 from 2468371 to 2958065 tid 3 2024-09-11T03:41:12.012 INFO:tasks.rados.rados.2.smithi175.stdout:779: read oid 23 snap 76 2024-09-11T03:41:12.012 INFO:tasks.rados.rados.2.smithi175.stdout:779: expect deleted 2024-09-11T03:41:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:780: read oid 2 snap -1 2024-09-11T03:41:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:780: expect deleted 2024-09-11T03:41:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:773: done (6 left) 2024-09-11T03:41:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:781: write (excl) oid 46 current snap is 79 2024-09-11T03:41:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:777: expect (ObjNum 280 snap 73 seq_num 280) 2024-09-11T03:41:12.017 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:12.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:12.018 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:12.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:11 smithi084 ceph-mon[27254]: pgmap v535: 47 pgs: 47 active+clean; 190 MiB data, 711 MiB used, 893 GiB / 894 GiB avail; 1.8 MiB/s rd, 607 KiB/s wr, 3 op/s 2024-09-11T03:41:12.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:11 smithi084 ceph-mon[27254]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:41:12.084 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:11 smithi084 ceph-mon[27254]: osdmap e255: 12 total, 12 up, 10 in 2024-09-11T03:41:12.118 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:12.140 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:12.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:12.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:12.160 INFO:tasks.rados.rados.2.smithi175.stdout:775: finishing copy_from racing read to smithi17557870-49 2024-09-11T03:41:12.161 INFO:tasks.rados.rados.2.smithi175.stdout:776: finishing copy_from racing read to smithi17557870-12 2024-09-11T03:41:12.161 INFO:tasks.rados.rados.2.smithi175.stdout:778: finishing write tid 1 to smithi17557870-7 2024-09-11T03:41:12.161 INFO:tasks.rados.rados.2.smithi175.stdout:778: finishing write tid 2 to smithi17557870-7 2024-09-11T03:41:12.161 INFO:tasks.rados.rados.2.smithi175.stdout:776: finishing copy_from to smithi17557870-12 2024-09-11T03:41:12.162 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 160 (ObjNum 267 snap 68 seq_num 267) dirty exists 2024-09-11T03:41:12.162 INFO:tasks.rados.rados.2.smithi175.stdout:778: finishing write tid 3 to smithi17557870-7 2024-09-11T03:41:12.162 INFO:tasks.rados.rados.2.smithi175.stdout:775: finishing copy_from to smithi17557870-49 2024-09-11T03:41:12.162 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 217 (ObjNum 238 snap 58 seq_num 238) dirty exists 2024-09-11T03:41:12.162 INFO:tasks.rados.rados.2.smithi175.stdout:778: finishing write tid 4 to smithi17557870-7 2024-09-11T03:41:12.162 INFO:tasks.rados.rados.2.smithi175.stdout:778: finishing write tid 5 to smithi17557870-7 2024-09-11T03:41:12.163 INFO:tasks.rados.rados.2.smithi175.stdout:778: oid 7 updating version 0 to 163 2024-09-11T03:41:12.163 INFO:tasks.rados.rados.2.smithi175.stdout:778: oid 7 version 163 is already newer than 162 2024-09-11T03:41:12.163 INFO:tasks.rados.rados.2.smithi175.stdout:778: oid 7 version 163 is already newer than 160 2024-09-11T03:41:12.163 INFO:tasks.rados.rados.2.smithi175.stdout:778: oid 7 version 163 is already newer than 161 2024-09-11T03:41:12.163 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 163 (ObjNum 295 snap 79 seq_num 295) dirty exists 2024-09-11T03:41:12.163 INFO:tasks.rados.rados.2.smithi175.stdout:778: left oid 7 (ObjNum 295 snap 79 seq_num 295) 2024-09-11T03:41:12.164 INFO:tasks.rados.rados.2.smithi175.stdout:781: seq_num 296 ranges {574872=695118,2038317=630423} 2024-09-11T03:41:12.166 INFO:tasks.rados.rados.2.smithi175.stdout:781: writing smithi17557870-46 from 574872 to 1269990 tid 1 2024-09-11T03:41:12.174 INFO:tasks.rados.rados.2.smithi175.stdout:781: writing smithi17557870-46 from 2038317 to 2668740 tid 2 2024-09-11T03:41:12.175 INFO:tasks.rados.rados.2.smithi175.stdout:775: done (6 left) 2024-09-11T03:41:12.175 INFO:tasks.rados.rados.2.smithi175.stdout:776: done (5 left) 2024-09-11T03:41:12.175 INFO:tasks.rados.rados.2.smithi175.stdout:777: done (4 left) 2024-09-11T03:41:12.175 INFO:tasks.rados.rados.2.smithi175.stdout:778: done (3 left) 2024-09-11T03:41:12.175 INFO:tasks.rados.rados.2.smithi175.stdout:779: done (2 left) 2024-09-11T03:41:12.175 INFO:tasks.rados.rados.2.smithi175.stdout:780: done (1 left) 2024-09-11T03:41:12.175 INFO:tasks.rados.rados.2.smithi175.stdout:782: read oid 6 snap -1 2024-09-11T03:41:12.175 INFO:tasks.rados.rados.2.smithi175.stdout:782: expect (ObjNum 225 snap 53 seq_num 225) 2024-09-11T03:41:12.175 INFO:tasks.rados.rados.2.smithi175.stdout:783: read oid 17 snap -1 2024-09-11T03:41:12.176 INFO:tasks.rados.rados.2.smithi175.stdout:783: expect deleted 2024-09-11T03:41:12.176 INFO:tasks.rados.rados.2.smithi175.stdout:784: snap_remove snap 75 2024-09-11T03:41:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:11 smithi175 ceph-mon[30894]: pgmap v535: 47 pgs: 47 active+clean; 190 MiB data, 711 MiB used, 893 GiB / 894 GiB avail; 1.8 MiB/s rd, 607 KiB/s wr, 3 op/s 2024-09-11T03:41:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:11 smithi175 ceph-mon[30894]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:41:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:11 smithi175 ceph-mon[30894]: osdmap e255: 12 total, 12 up, 10 in 2024-09-11T03:41:12.242 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:12.263 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:12.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:12.365 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:12.387 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:12.388 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:11 smithi121 ceph-mon[30729]: pgmap v535: 47 pgs: 47 active+clean; 190 MiB data, 711 MiB used, 893 GiB / 894 GiB avail; 1.8 MiB/s rd, 607 KiB/s wr, 3 op/s 2024-09-11T03:41:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:11 smithi121 ceph-mon[30729]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:41:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:11 smithi121 ceph-mon[30729]: osdmap e255: 12 total, 12 up, 10 in 2024-09-11T03:41:12.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:12.510 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:12.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:12.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:12.612 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:12.638 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:12.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:12.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:12.658 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:41:12.658 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:41:12.740 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:12.762 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:12.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:12.763 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:12.857 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:41:12.857 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:41:12.864 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:12.873 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:41:12.874 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:41:12.890 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:41:12.890 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:41:12.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:12.891 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:12.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:12.905 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:41:12.906 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:41:12.922 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:41:12.922 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:41:12.930 INFO:tasks.rados.rados.2.smithi175.stdout:784: done (3 left) 2024-09-11T03:41:12.930 INFO:tasks.rados.rados.2.smithi175.stdout:785: snap_create 2024-09-11T03:41:12.930 INFO:tasks.rados.rados.2.smithi175.stdout:781: finishing write tid 1 to smithi17557870-46 2024-09-11T03:41:12.930 INFO:tasks.rados.rados.2.smithi175.stdout:781: finishing write tid 2 to smithi17557870-46 2024-09-11T03:41:12.931 INFO:tasks.rados.rados.2.smithi175.stdout:781: finishing write tid 3 to smithi17557870-46 2024-09-11T03:41:12.931 INFO:tasks.rados.rados.2.smithi175.stdout:781: finishing write tid 4 to smithi17557870-46 2024-09-11T03:41:12.931 INFO:tasks.rados.rados.2.smithi175.stdout:781: oid 46 updating version 0 to 66 2024-09-11T03:41:12.931 INFO:tasks.rados.rados.2.smithi175.stdout:781: oid 46 updating version 66 to 68 2024-09-11T03:41:12.931 INFO:tasks.rados.rados.2.smithi175.stdout:781: oid 46 version 68 is already newer than 67 2024-09-11T03:41:12.931 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 68 (ObjNum 296 snap 79 seq_num 296) dirty exists 2024-09-11T03:41:12.931 INFO:tasks.rados.rados.2.smithi175.stdout:781: left oid 46 (ObjNum 296 snap 79 seq_num 296) 2024-09-11T03:41:12.931 INFO:tasks.rados.rados.2.smithi175.stdout:782: expect (ObjNum 225 snap 53 seq_num 225) 2024-09-11T03:41:12.945 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:41:12.945 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:41:12.962 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:41:12.963 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:41:12.979 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:41:12.979 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:41:12.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:12.995 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:41:12.995 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:41:13.013 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:41:13.013 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:41:13.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:13.014 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:13.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:13.029 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:41:13.029 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:41:13.045 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:41:13.045 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:41:13.117 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:13.138 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:13.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:13.139 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:13.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:12 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3440268659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:13.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:12 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3440268659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:13.240 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:13.262 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:13.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:13.364 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:13.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:13.386 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:13.386 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:13.389 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:41:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:12 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3440268659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:13.487 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:13.508 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:13.509 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:13.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:13.637 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:13.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:13.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:13.691 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:13.739 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:13.765 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:13.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:13.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:13.867 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:13.889 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:13.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:13.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:13.932 INFO:tasks.rados.rados.2.smithi175.stdout:781: done (3 left) 2024-09-11T03:41:13.932 INFO:tasks.rados.rados.2.smithi175.stdout:782: done (2 left) 2024-09-11T03:41:13.932 INFO:tasks.rados.rados.2.smithi175.stdout:783: done (1 left) 2024-09-11T03:41:13.932 INFO:tasks.rados.rados.2.smithi175.stdout:785: done (0 left) 2024-09-11T03:41:13.932 INFO:tasks.rados.rados.2.smithi175.stdout:786: rollback oid 44 current snap is 80 2024-09-11T03:41:13.932 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 44 to 59 2024-09-11T03:41:13.932 INFO:tasks.rados.rados.2.smithi175.stdout:787: write oid 20 current snap is 80 2024-09-11T03:41:13.932 INFO:tasks.rados.rados.2.smithi175.stdout:787: seq_num 297 ranges {591679=486720,1813152=490434} 2024-09-11T03:41:13.937 INFO:tasks.rados.rados.2.smithi175.stdout:787: writing smithi17557870-20 from 591679 to 1078399 tid 1 2024-09-11T03:41:13.945 INFO:tasks.rados.rados.2.smithi175.stdout:787: writing smithi17557870-20 from 1813152 to 2303586 tid 2 2024-09-11T03:41:13.945 INFO:tasks.rados.rados.2.smithi175.stdout:788: snap_create 2024-09-11T03:41:13.945 INFO:tasks.rados.rados.2.smithi175.stdout:786: finishing rollback tid 0 to smithi17557870-44 2024-09-11T03:41:13.945 INFO:tasks.rados.rados.2.smithi175.stdout:786: finishing rollback tid 1 to smithi17557870-44 2024-09-11T03:41:13.945 INFO:tasks.rados.rados.2.smithi175.stdout:786: finishing rollback tid 2 to smithi17557870-44 2024-09-11T03:41:13.945 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 89 (ObjNum 171 snap 39 seq_num 171) dirty exists 2024-09-11T03:41:13.945 INFO:tasks.rados.rados.2.smithi175.stdout:787: finishing write tid 1 to smithi17557870-20 2024-09-11T03:41:13.947 INFO:tasks.rados.rados.2.smithi175.stdout:787: finishing write tid 2 to smithi17557870-20 2024-09-11T03:41:13.947 INFO:tasks.rados.rados.2.smithi175.stdout:787: finishing write tid 3 to smithi17557870-20 2024-09-11T03:41:13.947 INFO:tasks.rados.rados.2.smithi175.stdout:787: finishing write tid 4 to smithi17557870-20 2024-09-11T03:41:13.948 INFO:tasks.rados.rados.2.smithi175.stdout:787: oid 20 updating version 0 to 265 2024-09-11T03:41:13.948 INFO:tasks.rados.rados.2.smithi175.stdout:787: oid 20 version 265 is already newer than 263 2024-09-11T03:41:13.948 INFO:tasks.rados.rados.2.smithi175.stdout:787: oid 20 version 265 is already newer than 264 2024-09-11T03:41:13.948 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 265 (ObjNum 297 snap 80 seq_num 297) dirty exists 2024-09-11T03:41:13.948 INFO:tasks.rados.rados.2.smithi175.stdout:787: left oid 20 (ObjNum 297 snap 80 seq_num 297) 2024-09-11T03:41:13.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:14.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:14.019 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:14.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:14.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:14.145 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:14.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:14.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:13 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:41:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:13 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:41:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:13 smithi175 ceph-mon[30894]: pgmap v538: 47 pgs: 47 active+clean; 184 MiB data, 702 MiB used, 893 GiB / 894 GiB avail; 9.0 MiB/s rd, 683 KiB/s wr, 9 op/s 2024-09-11T03:41:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:13 smithi175 ceph-mon[30894]: osdmap e256: 12 total, 12 up, 10 in 2024-09-11T03:41:14.247 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:14.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:14.270 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:14.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:14.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:13 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:41:14.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:13 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:41:14.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:13 smithi084 ceph-mon[27254]: pgmap v538: 47 pgs: 47 active+clean; 184 MiB data, 702 MiB used, 893 GiB / 894 GiB avail; 9.0 MiB/s rd, 683 KiB/s wr, 9 op/s 2024-09-11T03:41:14.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:13 smithi084 ceph-mon[27254]: osdmap e256: 12 total, 12 up, 10 in 2024-09-11T03:41:14.366 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T03:41:14.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:14.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:13 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:41:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:13 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:41:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:13 smithi121 ceph-mon[30729]: pgmap v538: 47 pgs: 47 active+clean; 184 MiB data, 702 MiB used, 893 GiB / 894 GiB avail; 9.0 MiB/s rd, 683 KiB/s wr, 9 op/s 2024-09-11T03:41:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:13 smithi121 ceph-mon[30729]: osdmap e256: 12 total, 12 up, 10 in 2024-09-11T03:41:14.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:14.429 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:14.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:14.531 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:14.557 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:14.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:14.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:14.659 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:14.665 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:14.682 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:14.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:14.684 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:14.784 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:14.810 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:14.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:14.912 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:14.938 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:14.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:14.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:14.940 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:41:14.947 INFO:tasks.rados.rados.2.smithi175.stdout:786: done (2 left) 2024-09-11T03:41:14.947 INFO:tasks.rados.rados.2.smithi175.stdout:787: done (1 left) 2024-09-11T03:41:14.948 INFO:tasks.rados.rados.2.smithi175.stdout:788: done (0 left) 2024-09-11T03:41:14.948 INFO:tasks.rados.rados.2.smithi175.stdout:789: rollback oid 31 current snap is 81 2024-09-11T03:41:14.948 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 31 to 59 2024-09-11T03:41:14.948 INFO:tasks.rados.rados.2.smithi175.stdout:790: copy_from oid 25 from oid 27 current snap is 81 2024-09-11T03:41:14.948 INFO:tasks.rados.rados.2.smithi175.stdout:791: delete oid 42 current snap is 81 2024-09-11T03:41:14.950 INFO:tasks.rados.rados.2.smithi175.stdout:791: done (2 left) 2024-09-11T03:41:14.950 INFO:tasks.rados.rados.2.smithi175.stdout:792: snap_create 2024-09-11T03:41:14.952 INFO:tasks.rados.rados.2.smithi175.stdout:789: finishing rollback tid 0 to smithi17557870-31 2024-09-11T03:41:14.952 INFO:tasks.rados.rados.2.smithi175.stdout:789: finishing rollback tid 1 to smithi17557870-31 2024-09-11T03:41:14.953 INFO:tasks.rados.rados.2.smithi175.stdout:789: finishing rollback tid 2 to smithi17557870-31 2024-09-11T03:41:14.953 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 252 (ObjNum 223 snap 53 seq_num 223) dirty exists 2024-09-11T03:41:14.959 INFO:tasks.rados.rados.2.smithi175.stdout:790: finishing copy_from racing read to smithi17557870-25 2024-09-11T03:41:14.962 INFO:tasks.rados.rados.2.smithi175.stdout:790: finishing copy_from to smithi17557870-25 2024-09-11T03:41:14.962 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 162 (ObjNum 241 snap 58 seq_num 241) dirty exists 2024-09-11T03:41:15.040 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:15.065 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:15.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:15.066 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:15.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:15.193 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:15.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:15.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:15.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:14 smithi175 ceph-mon[30894]: osdmap e257: 12 total, 12 up, 10 in 2024-09-11T03:41:15.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:14 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/110686343' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:15.242 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:14 smithi084 ceph-mon[27254]: osdmap e257: 12 total, 12 up, 10 in 2024-09-11T03:41:15.242 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:14 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/110686343' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:15.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:15.320 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:15.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:15.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:15.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:14 smithi121 ceph-mon[30729]: osdmap e257: 12 total, 12 up, 10 in 2024-09-11T03:41:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:14 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/110686343' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:15.422 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:15.447 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:15.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:15.448 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:15.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:15.572 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:15.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:15.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:15.661 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:41:15.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:15.699 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:15.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:15.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:15.769 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:41:15.769 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:41:15.769 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:41:15.769 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:41:15.769 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:41:15.801 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:15.826 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:15.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:15.828 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:15.929 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:15.951 INFO:tasks.rados.rados.2.smithi175.stdout:789: done (2 left) 2024-09-11T03:41:15.951 INFO:tasks.rados.rados.2.smithi175.stdout:790: done (1 left) 2024-09-11T03:41:15.951 INFO:tasks.rados.rados.2.smithi175.stdout:792: done (0 left) 2024-09-11T03:41:15.951 INFO:tasks.rados.rados.2.smithi175.stdout:793: delete oid 35 current snap is 82 2024-09-11T03:41:15.955 INFO:tasks.rados.rados.2.smithi175.stdout:793: done (0 left) 2024-09-11T03:41:15.955 INFO:tasks.rados.rados.2.smithi175.stdout:794: delete oid 19 current snap is 82 2024-09-11T03:41:15.956 INFO:tasks.rados.rados.2.smithi175.stdout:794: done (0 left) 2024-09-11T03:41:15.956 INFO:tasks.rados.rados.2.smithi175.stdout:795: snap_remove snap 59 2024-09-11T03:41:15.957 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:15.958 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:16.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:16.085 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:16.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:16.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:16.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:15 smithi084 ceph-mon[27254]: pgmap v541: 47 pgs: 47 active+clean; 170 MiB data, 682 MiB used, 893 GiB / 894 GiB avail; 9.0 MiB/s rd, 683 KiB/s wr, 9 op/s 2024-09-11T03:41:16.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:15 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:16.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:15 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/110686343' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:16.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:15 smithi084 ceph-mon[27254]: osdmap e258: 12 total, 12 up, 10 in 2024-09-11T03:41:16.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:16.208 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:16.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:16.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:16.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:15 smithi175 ceph-mon[30894]: pgmap v541: 47 pgs: 47 active+clean; 170 MiB data, 682 MiB used, 893 GiB / 894 GiB avail; 9.0 MiB/s rd, 683 KiB/s wr, 9 op/s 2024-09-11T03:41:16.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:15 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:16.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:15 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/110686343' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:16.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:15 smithi175 ceph-mon[30894]: osdmap e258: 12 total, 12 up, 10 in 2024-09-11T03:41:16.309 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:16.335 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:16.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:16.336 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:16.342 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T03:41:16.371 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:15 smithi121 ceph-mon[30729]: pgmap v541: 47 pgs: 47 active+clean; 170 MiB data, 682 MiB used, 893 GiB / 894 GiB avail; 9.0 MiB/s rd, 683 KiB/s wr, 9 op/s 2024-09-11T03:41:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:15 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:15 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/110686343' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:15 smithi121 ceph-mon[30729]: osdmap e258: 12 total, 12 up, 10 in 2024-09-11T03:41:16.436 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:16.457 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:16.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:16.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:16.559 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:16.584 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:16.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:16.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:16.686 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:16.708 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:16.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:16.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:16.797 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [6, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:41:16.797 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:41:16.797 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T03:41:16.797 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 1.000000 2024-09-11T03:41:16.797 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-11T03:41:16.809 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:16.835 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:16.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:16.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:16.855 INFO:tasks.rados.rados.2.smithi175.stdout:795: done (0 left) 2024-09-11T03:41:16.855 INFO:tasks.rados.rados.2.smithi175.stdout:796: write (excl) oid 30 current snap is 82 2024-09-11T03:41:16.855 INFO:tasks.rados.rados.2.smithi175.stdout:796: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2024-09-11T03:41:16.864 INFO:tasks.rados.rados.2.smithi175.stdout:796: writing smithi17557870-30 from 658907 to 1446623 tid 1 2024-09-11T03:41:16.870 INFO:tasks.rados.rados.2.smithi175.stdout:796: writing smithi17557870-30 from 1930727 to 2428499 tid 2 2024-09-11T03:41:16.874 INFO:tasks.rados.rados.2.smithi175.stdout:796: writing smithi17557870-30 from 3032909 to 3058907 tid 3 2024-09-11T03:41:16.874 INFO:tasks.rados.rados.2.smithi175.stdout:797: rollback oid 34 current snap is 82 2024-09-11T03:41:16.874 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 34 to 80 2024-09-11T03:41:16.874 INFO:tasks.rados.rados.2.smithi175.stdout:796: finishing write tid 1 to smithi17557870-30 2024-09-11T03:41:16.874 INFO:tasks.rados.rados.2.smithi175.stdout:796: finishing write tid 2 to smithi17557870-30 2024-09-11T03:41:16.874 INFO:tasks.rados.rados.2.smithi175.stdout:798: read oid 28 snap -1 2024-09-11T03:41:16.874 INFO:tasks.rados.rados.2.smithi175.stdout:798: expect (ObjNum 279 snap 73 seq_num 279) 2024-09-11T03:41:16.875 INFO:tasks.rados.rados.2.smithi175.stdout:799: read oid 47 snap 79 2024-09-11T03:41:16.875 INFO:tasks.rados.rados.2.smithi175.stdout:799: expect deleted 2024-09-11T03:41:16.875 INFO:tasks.rados.rados.2.smithi175.stdout:800: write (excl) oid 27 current snap is 82 2024-09-11T03:41:16.875 INFO:tasks.rados.rados.2.smithi175.stdout:800: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-09-11T03:41:16.880 INFO:tasks.rados.rados.2.smithi175.stdout:800: writing smithi17557870-27 from 675714 to 1138679 tid 1 2024-09-11T03:41:16.886 INFO:tasks.rados.rados.2.smithi175.stdout:800: writing smithi17557870-27 from 1589209 to 2230639 tid 2 2024-09-11T03:41:16.892 INFO:tasks.rados.rados.2.smithi175.stdout:800: writing smithi17557870-27 from 2943979 to 3075714 tid 3 2024-09-11T03:41:16.892 INFO:tasks.rados.rados.2.smithi175.stdout:801: rollback oid 11 current snap is 82 2024-09-11T03:41:16.892 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 11 to 80 2024-09-11T03:41:16.892 INFO:tasks.rados.rados.2.smithi175.stdout:799: done (5 left) 2024-09-11T03:41:16.892 INFO:tasks.rados.rados.2.smithi175.stdout:802: rollback oid 4 current snap is 82 2024-09-11T03:41:16.892 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 4 to 80 2024-09-11T03:41:16.892 INFO:tasks.rados.rados.2.smithi175.stdout:797: finishing rollback tid 0 to smithi17557870-34 2024-09-11T03:41:16.892 INFO:tasks.rados.rados.2.smithi175.stdout:796: finishing write tid 3 to smithi17557870-30 2024-09-11T03:41:16.892 INFO:tasks.rados.rados.2.smithi175.stdout:796: finishing write tid 4 to smithi17557870-30 2024-09-11T03:41:16.892 INFO:tasks.rados.rados.2.smithi175.stdout:803: write oid 45 current snap is 82 2024-09-11T03:41:16.892 INFO:tasks.rados.rados.2.smithi175.stdout:796: finishing write tid 5 to smithi17557870-30 2024-09-11T03:41:16.892 INFO:tasks.rados.rados.2.smithi175.stdout:796: oid 30 updating version 0 to 276 2024-09-11T03:41:16.892 INFO:tasks.rados.rados.2.smithi175.stdout:796: oid 30 version 276 is already newer than 274 2024-09-11T03:41:16.892 INFO:tasks.rados.rados.2.smithi175.stdout:796: oid 30 updating version 276 to 277 2024-09-11T03:41:16.893 INFO:tasks.rados.rados.2.smithi175.stdout:796: oid 30 version 277 is already newer than 275 2024-09-11T03:41:16.893 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 277 (ObjNum 301 snap 82 seq_num 301) dirty exists 2024-09-11T03:41:16.893 INFO:tasks.rados.rados.2.smithi175.stdout:796: left oid 30 (ObjNum 301 snap 82 seq_num 301) 2024-09-11T03:41:16.893 INFO:tasks.rados.rados.2.smithi175.stdout:797: finishing rollback tid 1 to smithi17557870-34 2024-09-11T03:41:16.893 INFO:tasks.rados.rados.2.smithi175.stdout:797: finishing rollback tid 2 to smithi17557870-34 2024-09-11T03:41:16.893 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 148 (ObjNum 293 snap 79 seq_num 293) dirty exists 2024-09-11T03:41:16.893 INFO:tasks.rados.rados.2.smithi175.stdout:798: expect (ObjNum 279 snap 73 seq_num 279) 2024-09-11T03:41:16.909 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:41:16.910 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:41:16.934 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:16.934 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:41:16.935 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:41:16.937 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:16.960 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:41:16.960 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:41:16.962 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:16.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:16.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:16.984 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:41:16.985 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:41:17.001 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:41:17.002 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:41:17.013 INFO:tasks.rados.rados.2.smithi175.stdout:800: finishing write tid 1 to smithi17557870-27 2024-09-11T03:41:17.013 INFO:tasks.rados.rados.2.smithi175.stdout:800: finishing write tid 2 to smithi17557870-27 2024-09-11T03:41:17.013 INFO:tasks.rados.rados.2.smithi175.stdout:802: finishing rollback tid 1 to smithi17557870-4 2024-09-11T03:41:17.013 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 0 (ObjNum 1265 snap 0 seq_num 1745349648) dirty dne 2024-09-11T03:41:17.013 INFO:tasks.rados.rados.2.smithi175.stdout:803: seq_num 303 ranges {692521=538214,1931338=785088} 2024-09-11T03:41:17.019 INFO:tasks.rados.rados.2.smithi175.stdout:803: writing smithi17557870-45 from 692521 to 1230735 tid 1 2024-09-11T03:41:17.020 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:41:17.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:41:17.028 INFO:tasks.rados.rados.2.smithi175.stdout:803: writing smithi17557870-45 from 1931338 to 2716426 tid 2 2024-09-11T03:41:17.028 INFO:tasks.rados.rados.2.smithi175.stdout:796: done (6 left) 2024-09-11T03:41:17.028 INFO:tasks.rados.rados.2.smithi175.stdout:797: done (5 left) 2024-09-11T03:41:17.028 INFO:tasks.rados.rados.2.smithi175.stdout:798: done (4 left) 2024-09-11T03:41:17.028 INFO:tasks.rados.rados.2.smithi175.stdout:802: done (3 left) 2024-09-11T03:41:17.028 INFO:tasks.rados.rados.2.smithi175.stdout:804: write oid 36 current snap is 82 2024-09-11T03:41:17.028 INFO:tasks.rados.rados.2.smithi175.stdout:804: seq_num 304 ranges {709328=613463,1989820=645099} 2024-09-11T03:41:17.035 INFO:tasks.rados.rados.2.smithi175.stdout:804: writing smithi17557870-36 from 709328 to 1322791 tid 1 2024-09-11T03:41:17.038 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:41:17.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:41:17.043 INFO:tasks.rados.rados.2.smithi175.stdout:804: writing smithi17557870-36 from 1989820 to 2634919 tid 2 2024-09-11T03:41:17.043 INFO:tasks.rados.rados.2.smithi175.stdout:805: read oid 13 snap -1 2024-09-11T03:41:17.043 INFO:tasks.rados.rados.2.smithi175.stdout:805: expect (ObjNum 257 snap 64 seq_num 257) 2024-09-11T03:41:17.043 INFO:tasks.rados.rados.2.smithi175.stdout:806: snap_create 2024-09-11T03:41:17.043 INFO:tasks.rados.rados.2.smithi175.stdout:800: finishing write tid 3 to smithi17557870-27 2024-09-11T03:41:17.043 INFO:tasks.rados.rados.2.smithi175.stdout:800: finishing write tid 4 to smithi17557870-27 2024-09-11T03:41:17.043 INFO:tasks.rados.rados.2.smithi175.stdout:800: finishing write tid 5 to smithi17557870-27 2024-09-11T03:41:17.043 INFO:tasks.rados.rados.2.smithi175.stdout:800: oid 27 updating version 0 to 263 2024-09-11T03:41:17.043 INFO:tasks.rados.rados.2.smithi175.stdout:800: oid 27 version 263 is already newer than 261 2024-09-11T03:41:17.043 INFO:tasks.rados.rados.2.smithi175.stdout:800: oid 27 version 263 is already newer than 262 2024-09-11T03:41:17.044 INFO:tasks.rados.rados.2.smithi175.stdout:800: oid 27 version 263 is already newer than 260 2024-09-11T03:41:17.044 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 263 (ObjNum 302 snap 82 seq_num 302) dirty exists 2024-09-11T03:41:17.044 INFO:tasks.rados.rados.2.smithi175.stdout:800: left oid 27 (ObjNum 302 snap 82 seq_num 302) 2024-09-11T03:41:17.044 INFO:tasks.rados.rados.2.smithi175.stdout:801: finishing rollback tid 0 to smithi17557870-11 2024-09-11T03:41:17.044 INFO:tasks.rados.rados.2.smithi175.stdout:801: finishing rollback tid 1 to smithi17557870-11 2024-09-11T03:41:17.044 INFO:tasks.rados.rados.2.smithi175.stdout:801: finishing rollback tid 2 to smithi17557870-11 2024-09-11T03:41:17.044 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 93 (ObjNum 270 snap 69 seq_num 270) dirty exists 2024-09-11T03:41:17.044 INFO:tasks.rados.rados.2.smithi175.stdout:803: finishing write tid 1 to smithi17557870-45 2024-09-11T03:41:17.044 INFO:tasks.rados.rados.2.smithi175.stdout:803: finishing write tid 2 to smithi17557870-45 2024-09-11T03:41:17.044 INFO:tasks.rados.rados.2.smithi175.stdout:803: finishing write tid 3 to smithi17557870-45 2024-09-11T03:41:17.044 INFO:tasks.rados.rados.2.smithi175.stdout:803: finishing write tid 4 to smithi17557870-45 2024-09-11T03:41:17.044 INFO:tasks.rados.rados.2.smithi175.stdout:803: oid 45 updating version 0 to 220 2024-09-11T03:41:17.044 INFO:tasks.rados.rados.2.smithi175.stdout:803: oid 45 version 220 is already newer than 219 2024-09-11T03:41:17.044 INFO:tasks.rados.rados.2.smithi175.stdout:803: oid 45 updating version 220 to 221 2024-09-11T03:41:17.045 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 221 (ObjNum 303 snap 82 seq_num 303) dirty exists 2024-09-11T03:41:17.045 INFO:tasks.rados.rados.2.smithi175.stdout:803: left oid 45 (ObjNum 303 snap 82 seq_num 303) 2024-09-11T03:41:17.045 INFO:tasks.rados.rados.2.smithi175.stdout:804: finishing write tid 1 to smithi17557870-36 2024-09-11T03:41:17.045 INFO:tasks.rados.rados.2.smithi175.stdout:804: finishing write tid 2 to smithi17557870-36 2024-09-11T03:41:17.046 INFO:tasks.rados.rados.2.smithi175.stdout:805: expect (ObjNum 257 snap 64 seq_num 257) 2024-09-11T03:41:17.055 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:41:17.055 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:41:17.064 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:17.072 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:41:17.072 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:41:17.085 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:17.086 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:17.088 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:41:17.088 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:41:17.106 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:41:17.106 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:41:17.123 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:41:17.123 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:41:17.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:16 smithi084 ceph-mon[27254]: osdmap e259: 12 total, 12 up, 10 in 2024-09-11T03:41:17.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:16 smithi084 ceph-mon[27254]: osdmap e260: 12 total, 12 up, 10 in 2024-09-11T03:41:17.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:16 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1422401594' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:17.151 INFO:tasks.rados.rados.2.smithi175.stdout:804: finishing write tid 3 to smithi17557870-36 2024-09-11T03:41:17.151 INFO:tasks.rados.rados.2.smithi175.stdout:804: finishing write tid 4 to smithi17557870-36 2024-09-11T03:41:17.151 INFO:tasks.rados.rados.2.smithi175.stdout:804: oid 36 updating version 0 to 189 2024-09-11T03:41:17.151 INFO:tasks.rados.rados.2.smithi175.stdout:804: oid 36 updating version 189 to 190 2024-09-11T03:41:17.151 INFO:tasks.rados.rados.2.smithi175.stdout:804: oid 36 version 190 is already newer than 188 2024-09-11T03:41:17.151 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 190 (ObjNum 304 snap 82 seq_num 304) dirty exists 2024-09-11T03:41:17.152 INFO:tasks.rados.rados.2.smithi175.stdout:804: left oid 36 (ObjNum 304 snap 82 seq_num 304) 2024-09-11T03:41:17.187 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:17.209 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:17.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:17.210 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:16 smithi175 ceph-mon[30894]: osdmap e259: 12 total, 12 up, 10 in 2024-09-11T03:41:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:16 smithi175 ceph-mon[30894]: osdmap e260: 12 total, 12 up, 10 in 2024-09-11T03:41:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:16 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1422401594' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:17.311 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:17.333 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:17.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:17.334 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:16 smithi121 ceph-mon[30729]: osdmap e259: 12 total, 12 up, 10 in 2024-09-11T03:41:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:16 smithi121 ceph-mon[30729]: osdmap e260: 12 total, 12 up, 10 in 2024-09-11T03:41:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:16 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1422401594' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:17.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:17.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:17.462 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:17.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:17.537 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:41:17.563 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:17.600 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:17.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:17.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:17.640 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:41:17.640 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:41:17.702 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:17.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:17.724 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:17.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:17.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:17.851 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:17.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:17.868 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:17.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:17.961 INFO:tasks.rados.rados.2.smithi175.stdout:800: done (5 left) 2024-09-11T03:41:17.961 INFO:tasks.rados.rados.2.smithi175.stdout:801: done (4 left) 2024-09-11T03:41:17.961 INFO:tasks.rados.rados.2.smithi175.stdout:803: done (3 left) 2024-09-11T03:41:17.961 INFO:tasks.rados.rados.2.smithi175.stdout:804: done (2 left) 2024-09-11T03:41:17.961 INFO:tasks.rados.rados.2.smithi175.stdout:805: done (1 left) 2024-09-11T03:41:17.961 INFO:tasks.rados.rados.2.smithi175.stdout:806: done (0 left) 2024-09-11T03:41:17.961 INFO:tasks.rados.rados.2.smithi175.stdout:807: snap_create 2024-09-11T03:41:17.980 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:17.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:18.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:18.107 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:18.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:18.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:18.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:18.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:17 smithi175 ceph-mon[30894]: pgmap v544: 47 pgs: 47 active+clean; 178 MiB data, 685 MiB used, 893 GiB / 894 GiB avail; 7.2 MiB/s rd, 1.8 MiB/s wr, 9 op/s 2024-09-11T03:41:18.234 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:18.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:18.269 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:17 smithi084 ceph-mon[27254]: pgmap v544: 47 pgs: 47 active+clean; 178 MiB data, 685 MiB used, 893 GiB / 894 GiB avail; 7.2 MiB/s rd, 1.8 MiB/s wr, 9 op/s 2024-09-11T03:41:18.305 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T03:41:18.306 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:18.337 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:18.361 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:18.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:18.362 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:18.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:17 smithi121 ceph-mon[30729]: pgmap v544: 47 pgs: 47 active+clean; 178 MiB data, 685 MiB used, 893 GiB / 894 GiB avail; 7.2 MiB/s rd, 1.8 MiB/s wr, 9 op/s 2024-09-11T03:41:18.463 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:18.484 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:18.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:18.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:18.587 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:18.608 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:18.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:18.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:18.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:18.737 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:18.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:18.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:18.841 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:18.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:18.864 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:18.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:18.963 INFO:tasks.rados.rados.2.smithi175.stdout:807: done (0 left) 2024-09-11T03:41:18.963 INFO:tasks.rados.rados.2.smithi175.stdout:808: delete oid 6 current snap is 84 2024-09-11T03:41:18.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:18.969 INFO:tasks.rados.rados.2.smithi175.stdout:808: done (0 left) 2024-09-11T03:41:18.969 INFO:tasks.rados.rados.2.smithi175.stdout:809: rollback oid 19 current snap is 84 2024-09-11T03:41:18.969 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 19 to 76 2024-09-11T03:41:18.969 INFO:tasks.rados.rados.2.smithi175.stdout:810: read oid 33 snap -1 2024-09-11T03:41:18.969 INFO:tasks.rados.rados.2.smithi175.stdout:810: expect (ObjNum 171 snap 39 seq_num 171) 2024-09-11T03:41:18.969 INFO:tasks.rados.rados.2.smithi175.stdout:811: read oid 21 snap -1 2024-09-11T03:41:18.969 INFO:tasks.rados.rados.2.smithi175.stdout:811: expect (ObjNum 266 snap 67 seq_num 266) 2024-09-11T03:41:18.970 INFO:tasks.rados.rados.2.smithi175.stdout:812: write oid 17 current snap is 84 2024-09-11T03:41:18.970 INFO:tasks.rados.rados.2.smithi175.stdout:812: seq_num 306 ranges {742942=480314,1823137=648768} 2024-09-11T03:41:18.975 INFO:tasks.rados.rados.2.smithi175.stdout:812: writing smithi17557870-17 from 742942 to 1223256 tid 1 2024-09-11T03:41:18.983 INFO:tasks.rados.rados.2.smithi175.stdout:812: writing smithi17557870-17 from 1823137 to 2471905 tid 2 2024-09-11T03:41:18.983 INFO:tasks.rados.rados.2.smithi175.stdout:813: write (excl) oid 8 current snap is 84 2024-09-11T03:41:18.983 INFO:tasks.rados.rados.2.smithi175.stdout:813: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2024-09-11T03:41:18.989 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:18.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:18.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:18.990 INFO:tasks.rados.rados.2.smithi175.stdout:813: writing smithi17557870-8 from 759749 to 1315312 tid 1 2024-09-11T03:41:18.996 INFO:tasks.rados.rados.2.smithi175.stdout:813: writing smithi17557870-8 from 1765266 to 2557692 tid 2 2024-09-11T03:41:19.000 INFO:tasks.rados.rados.2.smithi175.stdout:813: writing smithi17557870-8 from 2964741 to 3159749 tid 3 2024-09-11T03:41:19.000 INFO:tasks.rados.rados.2.smithi175.stdout:814: copy_from oid 5 from oid 27 current snap is 84 2024-09-11T03:41:19.000 INFO:tasks.rados.rados.2.smithi175.stdout:810: expect (ObjNum 171 snap 39 seq_num 171) 2024-09-11T03:41:19.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:19.116 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:19.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:19.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:19.149 INFO:tasks.rados.rados.2.smithi175.stdout:809: finishing rollback tid 1 to smithi17557870-19 2024-09-11T03:41:19.149 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 0 (ObjNum 33 snap 0 seq_num 1713071920) dirty dne 2024-09-11T03:41:19.149 INFO:tasks.rados.rados.2.smithi175.stdout:812: finishing write tid 1 to smithi17557870-17 2024-09-11T03:41:19.149 INFO:tasks.rados.rados.2.smithi175.stdout:809: done (5 left) 2024-09-11T03:41:19.149 INFO:tasks.rados.rados.2.smithi175.stdout:810: done (4 left) 2024-09-11T03:41:19.149 INFO:tasks.rados.rados.2.smithi175.stdout:815: snap_create 2024-09-11T03:41:19.149 INFO:tasks.rados.rados.2.smithi175.stdout:811: expect (ObjNum 266 snap 67 seq_num 266) 2024-09-11T03:41:19.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:18 smithi121 ceph-mon[30729]: osdmap e261: 12 total, 12 up, 10 in 2024-09-11T03:41:19.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:19.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:18 smithi175 ceph-mon[30894]: osdmap e261: 12 total, 12 up, 10 in 2024-09-11T03:41:19.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:19.241 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:19.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:19.304 INFO:tasks.rados.rados.2.smithi175.stdout:812: finishing write tid 2 to smithi17557870-17 2024-09-11T03:41:19.304 INFO:tasks.rados.rados.2.smithi175.stdout:812: finishing write tid 3 to smithi17557870-17 2024-09-11T03:41:19.304 INFO:tasks.rados.rados.2.smithi175.stdout:812: finishing write tid 4 to smithi17557870-17 2024-09-11T03:41:19.304 INFO:tasks.rados.rados.2.smithi175.stdout:812: oid 17 updating version 0 to 274 2024-09-11T03:41:19.304 INFO:tasks.rados.rados.2.smithi175.stdout:812: oid 17 updating version 274 to 276 2024-09-11T03:41:19.304 INFO:tasks.rados.rados.2.smithi175.stdout:812: oid 17 version 276 is already newer than 275 2024-09-11T03:41:19.304 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 276 (ObjNum 306 snap 84 seq_num 306) dirty exists 2024-09-11T03:41:19.304 INFO:tasks.rados.rados.2.smithi175.stdout:812: left oid 17 (ObjNum 306 snap 84 seq_num 306) 2024-09-11T03:41:19.304 INFO:tasks.rados.rados.2.smithi175.stdout:813: finishing write tid 1 to smithi17557870-8 2024-09-11T03:41:19.304 INFO:tasks.rados.rados.2.smithi175.stdout:813: finishing write tid 2 to smithi17557870-8 2024-09-11T03:41:19.305 INFO:tasks.rados.rados.2.smithi175.stdout:813: finishing write tid 3 to smithi17557870-8 2024-09-11T03:41:19.305 INFO:tasks.rados.rados.2.smithi175.stdout:813: finishing write tid 4 to smithi17557870-8 2024-09-11T03:41:19.305 INFO:tasks.rados.rados.2.smithi175.stdout:813: finishing write tid 5 to smithi17557870-8 2024-09-11T03:41:19.305 INFO:tasks.rados.rados.2.smithi175.stdout:813: oid 8 updating version 0 to 280 2024-09-11T03:41:19.305 INFO:tasks.rados.rados.2.smithi175.stdout:813: oid 8 version 280 is already newer than 279 2024-09-11T03:41:19.305 INFO:tasks.rados.rados.2.smithi175.stdout:813: oid 8 updating version 280 to 281 2024-09-11T03:41:19.305 INFO:tasks.rados.rados.2.smithi175.stdout:813: oid 8 version 281 is already newer than 278 2024-09-11T03:41:19.305 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 281 (ObjNum 307 snap 84 seq_num 307) dirty exists 2024-09-11T03:41:19.305 INFO:tasks.rados.rados.2.smithi175.stdout:813: left oid 8 (ObjNum 307 snap 84 seq_num 307) 2024-09-11T03:41:19.305 INFO:tasks.rados.rados.2.smithi175.stdout:814: finishing copy_from racing read to smithi17557870-5 2024-09-11T03:41:19.305 INFO:tasks.rados.rados.2.smithi175.stdout:814: finishing copy_from to smithi17557870-5 2024-09-11T03:41:19.305 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 263 (ObjNum 302 snap 82 seq_num 302) dirty exists 2024-09-11T03:41:19.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:18 smithi084 ceph-mon[27254]: osdmap e261: 12 total, 12 up, 10 in 2024-09-11T03:41:19.343 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:19.364 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:19.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:19.466 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:19.488 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:19.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:19.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:19.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:19.612 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:19.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:19.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:19.714 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:19.735 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:19.736 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:19.784 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:19.838 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:19.860 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:19.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:19.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:19.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:19.971 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:41:19.976 INFO:tasks.rados.rados.2.smithi175.stdout:811: done (4 left) 2024-09-11T03:41:19.976 INFO:tasks.rados.rados.2.smithi175.stdout:812: done (3 left) 2024-09-11T03:41:19.977 INFO:tasks.rados.rados.2.smithi175.stdout:813: done (2 left) 2024-09-11T03:41:19.977 INFO:tasks.rados.rados.2.smithi175.stdout:814: done (1 left) 2024-09-11T03:41:19.977 INFO:tasks.rados.rados.2.smithi175.stdout:815: done (0 left) 2024-09-11T03:41:19.977 INFO:tasks.rados.rados.2.smithi175.stdout:816: delete oid 9 current snap is 85 2024-09-11T03:41:19.980 INFO:tasks.rados.rados.2.smithi175.stdout:816: done (0 left) 2024-09-11T03:41:19.980 INFO:tasks.rados.rados.2.smithi175.stdout:817: write oid 25 current snap is 85 2024-09-11T03:41:19.980 INFO:tasks.rados.rados.2.smithi175.stdout:817: seq_num 309 ranges {793363=706061,2282230=796095} 2024-09-11T03:41:19.985 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:19.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:19.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:19.987 INFO:tasks.rados.rados.2.smithi175.stdout:817: writing smithi17557870-25 from 793363 to 1499424 tid 1 2024-09-11T03:41:19.996 INFO:tasks.rados.rados.2.smithi175.stdout:817: writing smithi17557870-25 from 2282230 to 3078325 tid 2 2024-09-11T03:41:19.996 INFO:tasks.rados.rados.2.smithi175.stdout:818: write (excl) oid 10 current snap is 85 2024-09-11T03:41:19.996 INFO:tasks.rados.rados.2.smithi175.stdout:817: finishing write tid 1 to smithi17557870-25 2024-09-11T03:41:19.997 INFO:tasks.rados.rados.2.smithi175.stdout:818: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2024-09-11T03:41:20.002 INFO:tasks.rados.rados.2.smithi175.stdout:818: writing smithi17557870-10 from 410170 to 1191480 tid 1 2024-09-11T03:41:20.010 INFO:tasks.rados.rados.2.smithi175.stdout:818: writing smithi17557870-10 from 1940712 to 2596818 tid 2 2024-09-11T03:41:20.014 INFO:tasks.rados.rados.2.smithi175.stdout:818: writing smithi17557870-10 from 3163363 to 3210170 tid 3 2024-09-11T03:41:20.014 INFO:tasks.rados.rados.2.smithi175.stdout:819: read oid 34 snap -1 2024-09-11T03:41:20.014 INFO:tasks.rados.rados.2.smithi175.stdout:819: expect (ObjNum 293 snap 79 seq_num 293) 2024-09-11T03:41:20.014 INFO:tasks.rados.rados.2.smithi175.stdout:820: write oid 2 current snap is 85 2024-09-11T03:41:20.014 INFO:tasks.rados.rados.2.smithi175.stdout:817: finishing write tid 2 to smithi17557870-25 2024-09-11T03:41:20.014 INFO:tasks.rados.rados.2.smithi175.stdout:817: finishing write tid 3 to smithi17557870-25 2024-09-11T03:41:20.014 INFO:tasks.rados.rados.2.smithi175.stdout:817: finishing write tid 4 to smithi17557870-25 2024-09-11T03:41:20.015 INFO:tasks.rados.rados.2.smithi175.stdout:817: oid 25 updating version 0 to 220 2024-09-11T03:41:20.015 INFO:tasks.rados.rados.2.smithi175.stdout:817: oid 25 updating version 220 to 221 2024-09-11T03:41:20.015 INFO:tasks.rados.rados.2.smithi175.stdout:817: oid 25 version 221 is already newer than 219 2024-09-11T03:41:20.015 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 221 (ObjNum 309 snap 85 seq_num 309) dirty exists 2024-09-11T03:41:20.015 INFO:tasks.rados.rados.2.smithi175.stdout:817: left oid 25 (ObjNum 309 snap 85 seq_num 309) 2024-09-11T03:41:20.015 INFO:tasks.rados.rados.2.smithi175.stdout:820: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2024-09-11T03:41:20.018 INFO:tasks.rados.rados.2.smithi175.stdout:820: writing smithi17557870-2 from 426977 to 883536 tid 1 2024-09-11T03:41:20.026 INFO:tasks.rados.rados.2.smithi175.stdout:820: writing smithi17557870-2 from 1482841 to 2282605 tid 2 2024-09-11T03:41:20.032 INFO:tasks.rados.rados.2.smithi175.stdout:820: writing smithi17557870-2 from 2958080 to 3226977 tid 3 2024-09-11T03:41:20.032 INFO:tasks.rados.rados.2.smithi175.stdout:817: done (3 left) 2024-09-11T03:41:20.032 INFO:tasks.rados.rados.2.smithi175.stdout:821: read oid 16 snap -1 2024-09-11T03:41:20.032 INFO:tasks.rados.rados.2.smithi175.stdout:821: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:41:20.032 INFO:tasks.rados.rados.2.smithi175.stdout:822: snap_remove snap 80 2024-09-11T03:41:20.087 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:20.108 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:20.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:20.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:20.210 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:19 smithi175 ceph-mon[30894]: pgmap v547: 47 pgs: 47 active+clean; 163 MiB data, 664 MiB used, 893 GiB / 894 GiB avail; 7.2 MiB/s rd, 3.4 MiB/s wr, 17 op/s 2024-09-11T03:41:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:19 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:41:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:19 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:41:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:19 smithi175 ceph-mon[30894]: osdmap e262: 12 total, 12 up, 10 in 2024-09-11T03:41:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:41:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:41:20.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:41:20.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:19 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3148631016' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:41:20.232 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:20.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:20.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:20.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:19 smithi084 ceph-mon[27254]: pgmap v547: 47 pgs: 47 active+clean; 163 MiB data, 664 MiB used, 893 GiB / 894 GiB avail; 7.2 MiB/s rd, 3.4 MiB/s wr, 17 op/s 2024-09-11T03:41:20.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:19 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:41:20.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:19 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:41:20.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:19 smithi084 ceph-mon[27254]: osdmap e262: 12 total, 12 up, 10 in 2024-09-11T03:41:20.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:41:20.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:41:20.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:41:20.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:19 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3148631016' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:41:20.333 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:20.354 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:20.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:20.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:19 smithi121 ceph-mon[30729]: pgmap v547: 47 pgs: 47 active+clean; 163 MiB data, 664 MiB used, 893 GiB / 894 GiB avail; 7.2 MiB/s rd, 3.4 MiB/s wr, 17 op/s 2024-09-11T03:41:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:19 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:41:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:19 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:41:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:19 smithi121 ceph-mon[30729]: osdmap e262: 12 total, 12 up, 10 in 2024-09-11T03:41:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:41:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:41:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:41:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:19 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3148631016' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:41:20.456 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:20.477 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:20.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:20.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:20.579 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:20.601 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:20.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:20.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:20.703 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:20.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:41:20.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:20.726 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:20.726 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:20.826 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:20.848 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:20.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:20.909 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:41:20.909 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:41:20.950 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:20.971 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:20.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:20.973 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:20.973 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-11T03:41:20.979 INFO:tasks.rados.rados.2.smithi175.stdout:822: done (4 left) 2024-09-11T03:41:20.979 INFO:tasks.rados.rados.2.smithi175.stdout:823: read oid 48 snap -1 2024-09-11T03:41:20.979 INFO:tasks.rados.rados.2.smithi175.stdout:823: expect (ObjNum 246 snap 60 seq_num 246) 2024-09-11T03:41:20.979 INFO:tasks.rados.rados.2.smithi175.stdout:824: read oid 43 snap -1 2024-09-11T03:41:20.979 INFO:tasks.rados.rados.2.smithi175.stdout:824: expect deleted 2024-09-11T03:41:20.979 INFO:tasks.rados.rados.2.smithi175.stdout:825: rollback oid 17 current snap is 85 2024-09-11T03:41:20.979 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 17 to 84 2024-09-11T03:41:20.979 INFO:tasks.rados.rados.2.smithi175.stdout:818: finishing write tid 1 to smithi17557870-10 2024-09-11T03:41:20.979 INFO:tasks.rados.rados.2.smithi175.stdout:818: finishing write tid 2 to smithi17557870-10 2024-09-11T03:41:20.979 INFO:tasks.rados.rados.2.smithi175.stdout:818: finishing write tid 3 to smithi17557870-10 2024-09-11T03:41:20.979 INFO:tasks.rados.rados.2.smithi175.stdout:818: finishing write tid 4 to smithi17557870-10 2024-09-11T03:41:20.980 INFO:tasks.rados.rados.2.smithi175.stdout:818: finishing write tid 5 to smithi17557870-10 2024-09-11T03:41:20.980 INFO:tasks.rados.rados.2.smithi175.stdout:818: oid 10 updating version 0 to 281 2024-09-11T03:41:20.980 INFO:tasks.rados.rados.2.smithi175.stdout:818: oid 10 version 281 is already newer than 278 2024-09-11T03:41:20.980 INFO:tasks.rados.rados.2.smithi175.stdout:818: oid 10 version 281 is already newer than 279 2024-09-11T03:41:20.980 INFO:tasks.rados.rados.2.smithi175.stdout:818: oid 10 version 281 is already newer than 280 2024-09-11T03:41:20.980 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 281 (ObjNum 310 snap 85 seq_num 310) dirty exists 2024-09-11T03:41:20.980 INFO:tasks.rados.rados.2.smithi175.stdout:818: left oid 10 (ObjNum 310 snap 85 seq_num 310) 2024-09-11T03:41:20.980 INFO:tasks.rados.rados.2.smithi175.stdout:820: finishing write tid 1 to smithi17557870-2 2024-09-11T03:41:20.980 INFO:tasks.rados.rados.2.smithi175.stdout:819: expect (ObjNum 293 snap 79 seq_num 293) 2024-09-11T03:41:21.073 INFO:tasks.rados.rados.2.smithi175.stdout:820: finishing write tid 2 to smithi17557870-2 2024-09-11T03:41:21.073 INFO:tasks.rados.rados.2.smithi175.stdout:820: finishing write tid 3 to smithi17557870-2 2024-09-11T03:41:21.073 INFO:tasks.rados.rados.2.smithi175.stdout:820: finishing write tid 4 to smithi17557870-2 2024-09-11T03:41:21.073 INFO:tasks.rados.rados.2.smithi175.stdout:820: finishing write tid 5 to smithi17557870-2 2024-09-11T03:41:21.073 INFO:tasks.rados.rados.2.smithi175.stdout:820: oid 2 updating version 0 to 268 2024-09-11T03:41:21.073 INFO:tasks.rados.rados.2.smithi175.stdout:820: oid 2 version 268 is already newer than 265 2024-09-11T03:41:21.073 INFO:tasks.rados.rados.2.smithi175.stdout:820: oid 2 version 268 is already newer than 267 2024-09-11T03:41:21.073 INFO:tasks.rados.rados.2.smithi175.stdout:820: oid 2 version 268 is already newer than 266 2024-09-11T03:41:21.073 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 268 (ObjNum 311 snap 85 seq_num 311) dirty exists 2024-09-11T03:41:21.073 INFO:tasks.rados.rados.2.smithi175.stdout:820: left oid 2 (ObjNum 311 snap 85 seq_num 311) 2024-09-11T03:41:21.074 INFO:tasks.rados.rados.2.smithi175.stdout:821: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:41:21.074 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:21.096 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:21.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:21.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:21.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:20 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:21.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:20 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3148631016' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:41:21.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:20 smithi084 ceph-mon[27254]: osdmap e263: 12 total, 12 up, 10 in 2024-09-11T03:41:21.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:20 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1115172776' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:41:21.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:20 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:41:21.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:20 smithi084 ceph-mon[27254]: 2.9 scrub ok 2024-09-11T03:41:21.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:20 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:41:21.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:20 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:41:21.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:20 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:41:21.193 INFO:tasks.rados.rados.2.smithi175.stdout:823: expect (ObjNum 246 snap 60 seq_num 246) 2024-09-11T03:41:21.198 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:21.220 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:21.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:21.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:20 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:20 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3148631016' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:41:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:20 smithi175 ceph-mon[30894]: osdmap e263: 12 total, 12 up, 10 in 2024-09-11T03:41:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:20 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1115172776' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:41:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:20 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:41:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:20 smithi175 ceph-mon[30894]: 2.9 scrub ok 2024-09-11T03:41:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:20 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:41:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:20 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:41:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:20 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:41:21.322 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:21.333 INFO:tasks.rados.rados.2.smithi175.stdout:825: finishing rollback tid 0 to smithi17557870-17 2024-09-11T03:41:21.333 INFO:tasks.rados.rados.2.smithi175.stdout:825: finishing rollback tid 1 to smithi17557870-17 2024-09-11T03:41:21.333 INFO:tasks.rados.rados.2.smithi175.stdout:825: finishing rollback tid 2 to smithi17557870-17 2024-09-11T03:41:21.333 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 285 (ObjNum 306 snap 84 seq_num 306) dirty exists 2024-09-11T03:41:21.333 INFO:tasks.rados.rados.2.smithi175.stdout:818: done (6 left) 2024-09-11T03:41:21.333 INFO:tasks.rados.rados.2.smithi175.stdout:819: done (5 left) 2024-09-11T03:41:21.333 INFO:tasks.rados.rados.2.smithi175.stdout:820: done (4 left) 2024-09-11T03:41:21.333 INFO:tasks.rados.rados.2.smithi175.stdout:821: done (3 left) 2024-09-11T03:41:21.334 INFO:tasks.rados.rados.2.smithi175.stdout:823: done (2 left) 2024-09-11T03:41:21.334 INFO:tasks.rados.rados.2.smithi175.stdout:824: done (1 left) 2024-09-11T03:41:21.334 INFO:tasks.rados.rados.2.smithi175.stdout:825: done (0 left) 2024-09-11T03:41:21.334 INFO:tasks.rados.rados.2.smithi175.stdout:826: copy_from oid 13 from oid 23 current snap is 85 2024-09-11T03:41:21.334 INFO:tasks.rados.rados.2.smithi175.stdout:827: copy_from oid 36 from oid 50 current snap is 85 2024-09-11T03:41:21.334 INFO:tasks.rados.rados.2.smithi175.stdout:828: read oid 22 snap -1 2024-09-11T03:41:21.334 INFO:tasks.rados.rados.2.smithi175.stdout:828: expect deleted 2024-09-11T03:41:21.334 INFO:tasks.rados.rados.2.smithi175.stdout:829: snap_remove snap 81 2024-09-11T03:41:21.344 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:21.345 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:20 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:20 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3148631016' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:41:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:20 smithi121 ceph-mon[30729]: osdmap e263: 12 total, 12 up, 10 in 2024-09-11T03:41:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:20 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1115172776' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:41:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:20 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:41:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:20 smithi121 ceph-mon[30729]: 2.9 scrub ok 2024-09-11T03:41:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:20 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:41:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:20 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:41:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:20 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:41:21.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:21.467 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:21.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:21.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:21.569 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:21.590 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:21.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:21.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:21.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:21.718 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:21.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:21.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:21.820 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:21.846 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:21.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:21.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:21.853 INFO:tasks.rados.rados.2.smithi175.stdout:829: done (3 left) 2024-09-11T03:41:21.854 INFO:tasks.rados.rados.2.smithi175.stdout:830: copy_from oid 8 from oid 18 current snap is 85 2024-09-11T03:41:21.854 INFO:tasks.rados.rados.2.smithi175.stdout:827: finishing copy_from to smithi17557870-36 2024-09-11T03:41:21.854 INFO:tasks.rados.rados.2.smithi175.stdout:827: got expected ENOENT (src dne) 2024-09-11T03:41:21.854 INFO:tasks.rados.rados.2.smithi175.stdout:827: finishing copy_from racing read to smithi17557870-36 2024-09-11T03:41:21.854 INFO:tasks.rados.rados.2.smithi175.stdout:826: finishing copy_from to smithi17557870-13 2024-09-11T03:41:21.854 INFO:tasks.rados.rados.2.smithi175.stdout:826: got expected ENOENT (src dne) 2024-09-11T03:41:21.854 INFO:tasks.rados.rados.2.smithi175.stdout:826: finishing copy_from racing read to smithi17557870-13 2024-09-11T03:41:21.854 INFO:tasks.rados.rados.2.smithi175.stdout:826: done (3 left) 2024-09-11T03:41:21.854 INFO:tasks.rados.rados.2.smithi175.stdout:827: done (2 left) 2024-09-11T03:41:21.854 INFO:tasks.rados.rados.2.smithi175.stdout:828: done (1 left) 2024-09-11T03:41:21.854 INFO:tasks.rados.rados.2.smithi175.stdout:831: snap_remove snap 83 2024-09-11T03:41:21.948 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:21.969 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:21.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:21.970 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:22.071 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:22.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:22.094 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:22.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:22.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:21 smithi084 ceph-mon[27254]: 2.f scrub starts 2024-09-11T03:41:22.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:21 smithi084 ceph-mon[27254]: 2.f scrub ok 2024-09-11T03:41:22.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:21 smithi084 ceph-mon[27254]: pgmap v550: 47 pgs: 47 active+clean; 163 MiB data, 636 MiB used, 893 GiB / 894 GiB avail; 1023 B/s rd, 1.6 MiB/s wr, 7 op/s 2024-09-11T03:41:22.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1115172776' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:41:22.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:21 smithi084 ceph-mon[27254]: osdmap e264: 12 total, 12 up, 10 in 2024-09-11T03:41:22.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:21 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:41:22.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:21 smithi084 ceph-mon[27254]: 2.8 scrub ok 2024-09-11T03:41:22.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:21 smithi084 ceph-mon[27254]: osdmap e265: 12 total, 12 up, 10 in 2024-09-11T03:41:22.194 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:22.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T03:41:22.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:22.217 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:22.217 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:21 smithi175 ceph-mon[30894]: 2.f scrub starts 2024-09-11T03:41:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:21 smithi175 ceph-mon[30894]: 2.f scrub ok 2024-09-11T03:41:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:21 smithi175 ceph-mon[30894]: pgmap v550: 47 pgs: 47 active+clean; 163 MiB data, 636 MiB used, 893 GiB / 894 GiB avail; 1023 B/s rd, 1.6 MiB/s wr, 7 op/s 2024-09-11T03:41:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:21 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1115172776' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:41:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:21 smithi175 ceph-mon[30894]: osdmap e264: 12 total, 12 up, 10 in 2024-09-11T03:41:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:21 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:41:22.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:21 smithi175 ceph-mon[30894]: 2.8 scrub ok 2024-09-11T03:41:22.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:21 smithi175 ceph-mon[30894]: osdmap e265: 12 total, 12 up, 10 in 2024-09-11T03:41:22.311 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:22.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:22.343 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:22.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:22.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:22.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:21 smithi121 ceph-mon[30729]: 2.f scrub starts 2024-09-11T03:41:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:21 smithi121 ceph-mon[30729]: 2.f scrub ok 2024-09-11T03:41:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:21 smithi121 ceph-mon[30729]: pgmap v550: 47 pgs: 47 active+clean; 163 MiB data, 636 MiB used, 893 GiB / 894 GiB avail; 1023 B/s rd, 1.6 MiB/s wr, 7 op/s 2024-09-11T03:41:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:21 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1115172776' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:41:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:21 smithi121 ceph-mon[30729]: osdmap e264: 12 total, 12 up, 10 in 2024-09-11T03:41:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:21 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:41:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:21 smithi121 ceph-mon[30729]: 2.8 scrub ok 2024-09-11T03:41:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:21 smithi121 ceph-mon[30729]: osdmap e265: 12 total, 12 up, 10 in 2024-09-11T03:41:22.445 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:22.467 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:22.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:22.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:22.568 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:22.590 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:22.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:22.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:22.646 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:41:22.691 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:22.714 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:22.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:22.715 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:22.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:22.842 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:22.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:22.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:22.855 INFO:tasks.rados.rados.2.smithi175.stdout:831: done (1 left) 2024-09-11T03:41:22.855 INFO:tasks.rados.rados.2.smithi175.stdout:832: snap_create 2024-09-11T03:41:22.855 INFO:tasks.rados.rados.2.smithi175.stdout:830: finishing copy_from racing read to smithi17557870-8 2024-09-11T03:41:22.855 INFO:tasks.rados.rados.2.smithi175.stdout:830: finishing copy_from to smithi17557870-8 2024-09-11T03:41:22.855 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 153 (ObjNum 261 snap 66 seq_num 261) dirty exists 2024-09-11T03:41:22.916 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:41:22.916 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:41:22.925 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:41:22.925 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:41:22.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:41:22.943 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:41:22.945 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:22.950 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:22.969 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:22.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:22.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:22.973 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:41:22.973 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:41:22.994 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:41:22.994 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:41:23.011 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:41:23.012 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:41:23.028 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:41:23.028 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:41:23.045 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:41:23.045 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:41:23.062 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:41:23.062 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:41:23.062 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:22 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:41:23.062 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:22 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:41:23.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:22 smithi084 ceph-mon[27254]: osdmap e266: 12 total, 12 up, 10 in 2024-09-11T03:41:23.063 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:22 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2284115200' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:23.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:23.078 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:41:23.079 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:41:23.094 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:23.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:23.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:23.095 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:41:23.096 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:41:23.113 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:41:23.114 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:41:23.195 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:23.219 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:23.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:23.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:23.321 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:23.342 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:23.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:23.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:23.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:22 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:41:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:22 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:41:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:22 smithi121 ceph-mon[30729]: osdmap e266: 12 total, 12 up, 10 in 2024-09-11T03:41:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:22 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2284115200' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:23.443 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:23.464 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:23.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:23.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:23.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:22 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:41:23.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:22 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:41:23.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:22 smithi175 ceph-mon[30894]: osdmap e266: 12 total, 12 up, 10 in 2024-09-11T03:41:23.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:22 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2284115200' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:23.566 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:23.587 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:23.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:23.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:23.689 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:23.710 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:23.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:23.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:23.812 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:23.834 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:23.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:23.857 INFO:tasks.rados.rados.2.smithi175.stdout:830: done (1 left) 2024-09-11T03:41:23.857 INFO:tasks.rados.rados.2.smithi175.stdout:832: done (0 left) 2024-09-11T03:41:23.857 INFO:tasks.rados.rados.2.smithi175.stdout:833: rollback oid 47 current snap is 86 2024-09-11T03:41:23.857 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 47 to 79 2024-09-11T03:41:23.857 INFO:tasks.rados.rados.2.smithi175.stdout:834: delete oid 45 current snap is 86 2024-09-11T03:41:23.860 INFO:tasks.rados.rados.2.smithi175.stdout:833: finishing rollback tid 1 to smithi17557870-47 2024-09-11T03:41:23.860 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-11T03:41:23.860 INFO:tasks.rados.rados.2.smithi175.stdout:833: done (1 left) 2024-09-11T03:41:23.860 INFO:tasks.rados.rados.2.smithi175.stdout:834: done (0 left) 2024-09-11T03:41:23.860 INFO:tasks.rados.rados.2.smithi175.stdout:835: delete oid 43 current snap is 86 2024-09-11T03:41:23.863 INFO:tasks.rados.rados.2.smithi175.stdout:835: done (0 left) 2024-09-11T03:41:23.863 INFO:tasks.rados.rados.2.smithi175.stdout:836: write oid 2 current snap is 86 2024-09-11T03:41:23.863 INFO:tasks.rados.rados.2.smithi175.stdout:836: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2024-09-11T03:41:23.870 INFO:tasks.rados.rados.2.smithi175.stdout:836: writing smithi17557870-2 from 477398 to 1276057 tid 1 2024-09-11T03:41:23.876 INFO:tasks.rados.rados.2.smithi175.stdout:836: writing smithi17557870-2 from 1774640 to 2438084 tid 2 2024-09-11T03:41:23.880 INFO:tasks.rados.rados.2.smithi175.stdout:836: writing smithi17557870-2 from 2873055 to 3277398 tid 3 2024-09-11T03:41:23.881 INFO:tasks.rados.rados.2.smithi175.stdout:837: write (excl) oid 21 current snap is 86 2024-09-11T03:41:23.881 INFO:tasks.rados.rados.2.smithi175.stdout:836: finishing write tid 1 to smithi17557870-2 2024-09-11T03:41:23.881 INFO:tasks.rados.rados.2.smithi175.stdout:836: finishing write tid 2 to smithi17557870-2 2024-09-11T03:41:23.881 INFO:tasks.rados.rados.2.smithi175.stdout:837: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2024-09-11T03:41:23.885 INFO:tasks.rados.rados.2.smithi175.stdout:837: writing smithi17557870-21 from 494205 to 968113 tid 1 2024-09-11T03:41:23.890 INFO:tasks.rados.rados.2.smithi175.stdout:837: writing smithi17557870-21 from 1433122 to 1840224 tid 2 2024-09-11T03:41:23.895 INFO:tasks.rados.rados.2.smithi175.stdout:837: writing smithi17557870-21 from 2384125 to 2846809 tid 3 2024-09-11T03:41:23.895 INFO:tasks.rados.rados.2.smithi175.stdout:838: snap_remove snap 82 2024-09-11T03:41:23.936 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:23.960 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:23.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:24.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:24.064 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:24.087 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:24.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:24.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:24.102 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:41:24.102 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:41:24.103 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:41:24.103 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:41:24.103 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:41:24.103 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:23 smithi084 ceph-mon[27254]: pgmap v553: 47 pgs: 47 active+clean; 127 MiB data, 584 MiB used, 893 GiB / 894 GiB avail; 16 MiB/s rd, 998 KiB/s wr, 18 op/s 2024-09-11T03:41:24.103 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:23 smithi084 ceph-mon[27254]: osdmap e267: 12 total, 12 up, 10 in 2024-09-11T03:41:24.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:24.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:24.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:24.317 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:24.343 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:24.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:24.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:24.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:23 smithi121 ceph-mon[30729]: pgmap v553: 47 pgs: 47 active+clean; 127 MiB data, 584 MiB used, 893 GiB / 894 GiB avail; 16 MiB/s rd, 998 KiB/s wr, 18 op/s 2024-09-11T03:41:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:23 smithi121 ceph-mon[30729]: osdmap e267: 12 total, 12 up, 10 in 2024-09-11T03:41:24.445 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:24.467 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:24.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:24.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:23 smithi175 ceph-mon[30894]: pgmap v553: 47 pgs: 47 active+clean; 127 MiB data, 584 MiB used, 893 GiB / 894 GiB avail; 16 MiB/s rd, 998 KiB/s wr, 18 op/s 2024-09-11T03:41:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:23 smithi175 ceph-mon[30894]: osdmap e267: 12 total, 12 up, 10 in 2024-09-11T03:41:24.569 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:24.594 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:24.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:24.595 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:24.696 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:24.721 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:24.723 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:24.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:24.849 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:24.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:24.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T03:41:24.862 INFO:tasks.rados.rados.2.smithi175.stdout:838: done (2 left) 2024-09-11T03:41:24.862 INFO:tasks.rados.rados.2.smithi175.stdout:839: read oid 24 snap -1 2024-09-11T03:41:24.862 INFO:tasks.rados.rados.2.smithi175.stdout:839: expect (ObjNum 267 snap 68 seq_num 267) 2024-09-11T03:41:24.862 INFO:tasks.rados.rados.2.smithi175.stdout:840: copy_from oid 15 from oid 37 current snap is 86 2024-09-11T03:41:24.862 INFO:tasks.rados.rados.2.smithi175.stdout:836: finishing write tid 3 to smithi17557870-2 2024-09-11T03:41:24.862 INFO:tasks.rados.rados.2.smithi175.stdout:836: finishing write tid 4 to smithi17557870-2 2024-09-11T03:41:24.862 INFO:tasks.rados.rados.2.smithi175.stdout:836: finishing write tid 5 to smithi17557870-2 2024-09-11T03:41:24.862 INFO:tasks.rados.rados.2.smithi175.stdout:836: oid 2 updating version 0 to 276 2024-09-11T03:41:24.862 INFO:tasks.rados.rados.2.smithi175.stdout:836: oid 2 updating version 276 to 278 2024-09-11T03:41:24.863 INFO:tasks.rados.rados.2.smithi175.stdout:836: oid 2 updating version 278 to 279 2024-09-11T03:41:24.863 INFO:tasks.rados.rados.2.smithi175.stdout:836: oid 2 version 279 is already newer than 277 2024-09-11T03:41:24.863 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 279 (ObjNum 314 snap 86 seq_num 314) dirty exists 2024-09-11T03:41:24.863 INFO:tasks.rados.rados.2.smithi175.stdout:836: left oid 2 (ObjNum 314 snap 86 seq_num 314) 2024-09-11T03:41:24.863 INFO:tasks.rados.rados.2.smithi175.stdout:836: done (3 left) 2024-09-11T03:41:24.863 INFO:tasks.rados.rados.2.smithi175.stdout:841: rollback oid 47 current snap is 86 2024-09-11T03:41:24.863 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 47 to 85 2024-09-11T03:41:24.863 INFO:tasks.rados.rados.2.smithi175.stdout:837: finishing write tid 1 to smithi17557870-21 2024-09-11T03:41:24.863 INFO:tasks.rados.rados.2.smithi175.stdout:837: finishing write tid 2 to smithi17557870-21 2024-09-11T03:41:24.863 INFO:tasks.rados.rados.2.smithi175.stdout:837: finishing write tid 3 to smithi17557870-21 2024-09-11T03:41:24.863 INFO:tasks.rados.rados.2.smithi175.stdout:837: finishing write tid 4 to smithi17557870-21 2024-09-11T03:41:24.863 INFO:tasks.rados.rados.2.smithi175.stdout:837: finishing write tid 5 to smithi17557870-21 2024-09-11T03:41:24.863 INFO:tasks.rados.rados.2.smithi175.stdout:837: oid 21 updating version 0 to 283 2024-09-11T03:41:24.864 INFO:tasks.rados.rados.2.smithi175.stdout:837: oid 21 updating version 283 to 284 2024-09-11T03:41:24.864 INFO:tasks.rados.rados.2.smithi175.stdout:837: oid 21 version 284 is already newer than 281 2024-09-11T03:41:24.864 INFO:tasks.rados.rados.2.smithi175.stdout:837: oid 21 version 284 is already newer than 282 2024-09-11T03:41:24.864 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 284 (ObjNum 315 snap 86 seq_num 315) dirty exists 2024-09-11T03:41:24.864 INFO:tasks.rados.rados.2.smithi175.stdout:837: left oid 21 (ObjNum 315 snap 86 seq_num 315) 2024-09-11T03:41:24.864 INFO:tasks.rados.rados.2.smithi175.stdout:837: done (3 left) 2024-09-11T03:41:24.864 INFO:tasks.rados.rados.2.smithi175.stdout:842: copy_from oid 38 from oid 25 current snap is 86 2024-09-11T03:41:24.864 INFO:tasks.rados.rados.2.smithi175.stdout:843: delete oid 8 current snap is 86 2024-09-11T03:41:24.864 INFO:tasks.rados.rados.2.smithi175.stdout:841: finishing rollback tid 1 to smithi17557870-47 2024-09-11T03:41:24.865 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-11T03:41:24.869 INFO:tasks.rados.rados.2.smithi175.stdout:839: expect (ObjNum 267 snap 68 seq_num 267) 2024-09-11T03:41:24.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:24.975 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:24.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:24.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:25.026 INFO:tasks.rados.rados.2.smithi175.stdout:840: finishing copy_from racing read to smithi17557870-15 2024-09-11T03:41:25.027 INFO:tasks.rados.rados.2.smithi175.stdout:840: finishing copy_from to smithi17557870-15 2024-09-11T03:41:25.027 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 101 (ObjNum 153 snap 35 seq_num 153) dirty exists 2024-09-11T03:41:25.027 INFO:tasks.rados.rados.2.smithi175.stdout:842: finishing copy_from racing read to smithi17557870-38 2024-09-11T03:41:25.027 INFO:tasks.rados.rados.2.smithi175.stdout:842: finishing copy_from to smithi17557870-38 2024-09-11T03:41:25.027 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 221 (ObjNum 309 snap 85 seq_num 309) dirty exists 2024-09-11T03:41:25.027 INFO:tasks.rados.rados.2.smithi175.stdout:839: done (4 left) 2024-09-11T03:41:25.027 INFO:tasks.rados.rados.2.smithi175.stdout:840: done (3 left) 2024-09-11T03:41:25.027 INFO:tasks.rados.rados.2.smithi175.stdout:841: done (2 left) 2024-09-11T03:41:25.027 INFO:tasks.rados.rados.2.smithi175.stdout:842: done (1 left) 2024-09-11T03:41:25.027 INFO:tasks.rados.rados.2.smithi175.stdout:843: done (0 left) 2024-09-11T03:41:25.027 INFO:tasks.rados.rados.2.smithi175.stdout:844: rollback oid 27 current snap is 86 2024-09-11T03:41:25.027 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 27 to 79 2024-09-11T03:41:25.027 INFO:tasks.rados.rados.2.smithi175.stdout:845: rollback oid 46 current snap is 86 2024-09-11T03:41:25.028 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 46 to 85 2024-09-11T03:41:25.028 INFO:tasks.rados.rados.2.smithi175.stdout:846: rollback oid 10 current snap is 86 2024-09-11T03:41:25.028 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 10 to 85 2024-09-11T03:41:25.028 INFO:tasks.rados.rados.2.smithi175.stdout:847: write (excl) oid 15 current snap is 86 2024-09-11T03:41:25.028 INFO:tasks.rados.rados.2.smithi175.stdout:847: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2024-09-11T03:41:25.032 INFO:tasks.rados.rados.2.smithi175.stdout:847: writing smithi17557870-15 from 527819 to 1152225 tid 1 2024-09-11T03:41:25.038 INFO:tasks.rados.rados.2.smithi175.stdout:847: writing smithi17557870-15 from 1833733 to 2244504 tid 2 2024-09-11T03:41:25.044 INFO:tasks.rados.rados.2.smithi175.stdout:847: writing smithi17557870-15 from 2722618 to 3324199 tid 3 2024-09-11T03:41:25.044 INFO:tasks.rados.rados.2.smithi175.stdout:848: rollback oid 21 current snap is 86 2024-09-11T03:41:25.045 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 21 to 85 2024-09-11T03:41:25.045 INFO:tasks.rados.rados.2.smithi175.stdout:845: finishing rollback tid 0 to smithi17557870-46 2024-09-11T03:41:25.045 INFO:tasks.rados.rados.2.smithi175.stdout:846: finishing rollback tid 0 to smithi17557870-10 2024-09-11T03:41:25.045 INFO:tasks.rados.rados.2.smithi175.stdout:845: finishing rollback tid 1 to smithi17557870-46 2024-09-11T03:41:25.045 INFO:tasks.rados.rados.2.smithi175.stdout:845: finishing rollback tid 2 to smithi17557870-46 2024-09-11T03:41:25.045 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 74 (ObjNum 296 snap 79 seq_num 296) dirty exists 2024-09-11T03:41:25.045 INFO:tasks.rados.rados.2.smithi175.stdout:846: finishing rollback tid 1 to smithi17557870-10 2024-09-11T03:41:25.045 INFO:tasks.rados.rados.2.smithi175.stdout:846: finishing rollback tid 2 to smithi17557870-10 2024-09-11T03:41:25.045 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 298 (ObjNum 310 snap 85 seq_num 310) dirty exists 2024-09-11T03:41:25.045 INFO:tasks.rados.rados.2.smithi175.stdout:844: finishing rollback tid 0 to smithi17557870-27 2024-09-11T03:41:25.045 INFO:tasks.rados.rados.2.smithi175.stdout:844: finishing rollback tid 1 to smithi17557870-27 2024-09-11T03:41:25.045 INFO:tasks.rados.rados.2.smithi175.stdout:844: finishing rollback tid 2 to smithi17557870-27 2024-09-11T03:41:25.046 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 290 (ObjNum 241 snap 58 seq_num 241) dirty exists 2024-09-11T03:41:25.046 INFO:tasks.rados.rados.2.smithi175.stdout:844: done (4 left) 2024-09-11T03:41:25.046 INFO:tasks.rados.rados.2.smithi175.stdout:845: done (3 left) 2024-09-11T03:41:25.046 INFO:tasks.rados.rados.2.smithi175.stdout:846: done (2 left) 2024-09-11T03:41:25.046 INFO:tasks.rados.rados.2.smithi175.stdout:849: write oid 38 current snap is 86 2024-09-11T03:41:25.046 INFO:tasks.rados.rados.2.smithi175.stdout:847: finishing write tid 1 to smithi17557870-15 2024-09-11T03:41:25.047 INFO:tasks.rados.rados.2.smithi175.stdout:847: finishing write tid 2 to smithi17557870-15 2024-09-11T03:41:25.047 INFO:tasks.rados.rados.2.smithi175.stdout:849: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2024-09-11T03:41:25.050 INFO:tasks.rados.rados.2.smithi175.stdout:849: writing smithi17557870-38 from 544626 to 1244281 tid 1 2024-09-11T03:41:25.057 INFO:tasks.rados.rados.2.smithi175.stdout:849: writing smithi17557870-38 from 1892215 to 2446644 tid 2 2024-09-11T03:41:25.063 INFO:tasks.rados.rados.2.smithi175.stdout:849: writing smithi17557870-38 from 3150041 to 3344626 tid 3 2024-09-11T03:41:25.063 INFO:tasks.rados.rados.2.smithi175.stdout:850: copy_from oid 35 from oid 16 current snap is 86 2024-09-11T03:41:25.063 INFO:tasks.rados.rados.2.smithi175.stdout:847: finishing write tid 3 to smithi17557870-15 2024-09-11T03:41:25.064 INFO:tasks.rados.rados.2.smithi175.stdout:848: finishing rollback tid 0 to smithi17557870-21 2024-09-11T03:41:25.064 INFO:tasks.rados.rados.2.smithi175.stdout:848: finishing rollback tid 1 to smithi17557870-21 2024-09-11T03:41:25.064 INFO:tasks.rados.rados.2.smithi175.stdout:847: finishing write tid 4 to smithi17557870-15 2024-09-11T03:41:25.064 INFO:tasks.rados.rados.2.smithi175.stdout:847: finishing write tid 5 to smithi17557870-15 2024-09-11T03:41:25.064 INFO:tasks.rados.rados.2.smithi175.stdout:847: oid 15 updating version 0 to 155 2024-09-11T03:41:25.065 INFO:tasks.rados.rados.2.smithi175.stdout:847: oid 15 version 155 is already newer than 154 2024-09-11T03:41:25.065 INFO:tasks.rados.rados.2.smithi175.stdout:847: oid 15 updating version 155 to 156 2024-09-11T03:41:25.065 INFO:tasks.rados.rados.2.smithi175.stdout:847: oid 15 updating version 156 to 157 2024-09-11T03:41:25.065 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 157 (ObjNum 317 snap 86 seq_num 317) dirty exists 2024-09-11T03:41:25.065 INFO:tasks.rados.rados.2.smithi175.stdout:847: left oid 15 (ObjNum 317 snap 86 seq_num 317) 2024-09-11T03:41:25.065 INFO:tasks.rados.rados.2.smithi175.stdout:848: finishing rollback tid 2 to smithi17557870-21 2024-09-11T03:41:25.065 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 293 (ObjNum 266 snap 67 seq_num 266) dirty exists 2024-09-11T03:41:25.065 INFO:tasks.rados.rados.2.smithi175.stdout:849: finishing write tid 1 to smithi17557870-38 2024-09-11T03:41:25.065 INFO:tasks.rados.rados.2.smithi175.stdout:849: finishing write tid 2 to smithi17557870-38 2024-09-11T03:41:25.065 INFO:tasks.rados.rados.2.smithi175.stdout:847: done (3 left) 2024-09-11T03:41:25.065 INFO:tasks.rados.rados.2.smithi175.stdout:848: done (2 left) 2024-09-11T03:41:25.066 INFO:tasks.rados.rados.2.smithi175.stdout:851: write oid 8 current snap is 86 2024-09-11T03:41:25.066 INFO:tasks.rados.rados.2.smithi175.stdout:851: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2024-09-11T03:41:25.070 INFO:tasks.rados.rados.2.smithi175.stdout:851: writing smithi17557870-8 from 561433 to 1336337 tid 1 2024-09-11T03:41:25.076 INFO:tasks.rados.rados.2.smithi175.stdout:851: writing smithi17557870-8 from 1950697 to 2365137 tid 2 2024-09-11T03:41:25.077 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:25.081 INFO:tasks.rados.rados.2.smithi175.stdout:851: writing smithi17557870-8 from 2777464 to 3361433 tid 3 2024-09-11T03:41:25.081 INFO:tasks.rados.rados.2.smithi175.stdout:852: snap_remove snap 76 2024-09-11T03:41:25.101 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:25.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:25.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:25.148 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:25.202 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:25.224 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:25.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:25.225 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:25.326 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:25.348 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:25.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:25.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:25.450 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:25.471 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:25.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:25.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:25.573 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:25.594 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:25.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:25.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:25.667 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:41:25.667 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:41:25.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:41:25.668 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:41:25.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.668 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:341 2.9 e113/95 2:9718db5a:::smithi17557870-3:head v 113'37, mlcod=109'33)", 2024-09-11T03:41:25.668 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:49.833457+0000", 2024-09-11T03:41:25.668 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 155.83346448899999, 2024-09-11T03:41:25.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043446580000000004, 2024-09-11T03:41:25.668 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.668 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.668 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.668 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.833457+0000", 2024-09-11T03:41:25.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.833457+0000", 2024-09-11T03:41:25.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.833457+0000", 2024-09-11T03:41:25.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2699999999999999e-07 2024-09-11T03:41:25.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.835023+0000", 2024-09-11T03:41:25.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015654740000000001 2024-09-11T03:41:25.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.835055+0000", 2024-09-11T03:41:25.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1810999999999997e-05 2024-09-11T03:41:25.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.835073+0000", 2024-09-11T03:41:25.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8783e-05 2024-09-11T03:41:25.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.835176+0000", 2024-09-11T03:41:25.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000103163 2024-09-11T03:41:25.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.835208+0000", 2024-09-11T03:41:25.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1791000000000001e-05 2024-09-11T03:41:25.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.837739+0000", 2024-09-11T03:41:25.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025311309999999998 2024-09-11T03:41:25.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.837801+0000", 2024-09-11T03:41:25.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1977999999999995e-05 2024-09-11T03:41:25.673 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.673 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.673 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.673 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.1.0:143 2.9 e114/95 2:9718db5a:::smithi17557870-3:8 v 114'39, mlcod=109'35)", 2024-09-11T03:41:25.673 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:50.828000+0000", 2024-09-11T03:41:25.674 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 154.83892080499999, 2024-09-11T03:41:25.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040361319999999996, 2024-09-11T03:41:25.674 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.674 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.674 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.674 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828000+0000", 2024-09-11T03:41:25.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828000+0000", 2024-09-11T03:41:25.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828001+0000", 2024-09-11T03:41:25.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3799999999999998e-07 2024-09-11T03:41:25.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828007+0000", 2024-09-11T03:41:25.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1099999999999999e-06 2024-09-11T03:41:25.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828031+0000", 2024-09-11T03:41:25.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3938999999999999e-05 2024-09-11T03:41:25.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828041+0000", 2024-09-11T03:41:25.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0132999999999999e-05 2024-09-11T03:41:25.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828125+0000", 2024-09-11T03:41:25.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4591000000000004e-05 2024-09-11T03:41:25.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.828148+0000", 2024-09-11T03:41:25.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2152999999999999e-05 2024-09-11T03:41:25.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.831988+0000", 2024-09-11T03:41:25.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0038402789999999998 2024-09-11T03:41:25.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:50.832037+0000", 2024-09-11T03:41:25.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8588999999999998e-05 2024-09-11T03:41:25.679 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.679 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.679 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.679 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:640 2.a e131/95 2:5014e305:::smithi17557870-46:head v 131'5, mlcod=100'2)", 2024-09-11T03:41:25.679 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:41:25.679 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 137.54850247499999, 2024-09-11T03:41:25.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010809757999999999, 2024-09-11T03:41:25.679 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.679 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.679 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.679 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:41:25.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:41:25.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:41:25.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6100000000000002e-07 2024-09-11T03:41:25.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122019+0000", 2024-09-11T03:41:25.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0036004240000000001 2024-09-11T03:41:25.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122067+0000", 2024-09-11T03:41:25.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7589000000000001e-05 2024-09-11T03:41:25.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122077+0000", 2024-09-11T03:41:25.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0036e-05 2024-09-11T03:41:25.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122133+0000", 2024-09-11T03:41:25.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6350000000000001e-05 2024-09-11T03:41:25.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122152+0000", 2024-09-11T03:41:25.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8595999999999999e-05 2024-09-11T03:41:25.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129179+0000", 2024-09-11T03:41:25.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0070268079999999998 2024-09-11T03:41:25.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129228+0000", 2024-09-11T03:41:25.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9594e-05 2024-09-11T03:41:25.684 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.684 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.685 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.685 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:649 2.6 e131/95 2:6dbd1e81:::smithi17557870-7:head v 131'25, mlcod=123'21)", 2024-09-11T03:41:25.685 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:08.118731+0000", 2024-09-11T03:41:25.685 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 137.54819067299999, 2024-09-11T03:41:25.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01067091, 2024-09-11T03:41:25.685 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.685 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.687 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.687 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118731+0000", 2024-09-11T03:41:25.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118731+0000", 2024-09-11T03:41:25.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118731+0000", 2024-09-11T03:41:25.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4399999999999998e-07 2024-09-11T03:41:25.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.121866+0000", 2024-09-11T03:41:25.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031341720000000002 2024-09-11T03:41:25.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.121900+0000", 2024-09-11T03:41:25.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4266e-05 2024-09-11T03:41:25.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.121914+0000", 2024-09-11T03:41:25.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3668e-05 2024-09-11T03:41:25.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122002+0000", 2024-09-11T03:41:25.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7995e-05 2024-09-11T03:41:25.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122029+0000", 2024-09-11T03:41:25.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7199000000000002e-05 2024-09-11T03:41:25.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129354+0000", 2024-09-11T03:41:25.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073255899999999999 2024-09-11T03:41:25.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129401+0000", 2024-09-11T03:41:25.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7076000000000001e-05 2024-09-11T03:41:25.690 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.690 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.691 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.691 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:847 2.d e148/95 2:b1c0f730:::smithi17557870-20:head v 148'82, mlcod=136'79)", 2024-09-11T03:41:25.691 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:24.846626+0000", 2024-09-11T03:41:25.691 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 120.820295558, 2024-09-11T03:41:25.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054620850000000002, 2024-09-11T03:41:25.691 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.691 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.691 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.691 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:24.846626+0000", 2024-09-11T03:41:25.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:24.846626+0000", 2024-09-11T03:41:25.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:24.846626+0000", 2024-09-11T03:41:25.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7800000000000001e-07 2024-09-11T03:41:25.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:24.850138+0000", 2024-09-11T03:41:25.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0035119779999999998 2024-09-11T03:41:25.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:24.850160+0000", 2024-09-11T03:41:25.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1824999999999999e-05 2024-09-11T03:41:25.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:24.850170+0000", 2024-09-11T03:41:25.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0244999999999999e-05 2024-09-11T03:41:25.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:24.850248+0000", 2024-09-11T03:41:25.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7976e-05 2024-09-11T03:41:25.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:24.850277+0000", 2024-09-11T03:41:25.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9067000000000001e-05 2024-09-11T03:41:25.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:24.852034+0000", 2024-09-11T03:41:25.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017573479999999999 2024-09-11T03:41:25.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:24.852088+0000", 2024-09-11T03:41:25.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3467999999999997e-05 2024-09-11T03:41:25.695 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.695 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.695 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.695 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:941 2.6 e163/95 2:6dbd1e81:::smithi17557870-7:head v 163'30, mlcod=134'27)", 2024-09-11T03:41:25.695 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:39.839973+0000", 2024-09-11T03:41:25.696 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 105.826948675, 2024-09-11T03:41:25.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045523389999999999, 2024-09-11T03:41:25.696 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.696 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.696 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.696 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.839973+0000", 2024-09-11T03:41:25.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.839973+0000", 2024-09-11T03:41:25.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.839974+0000", 2024-09-11T03:41:25.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.891e-06 2024-09-11T03:41:25.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.842135+0000", 2024-09-11T03:41:25.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021609149999999998 2024-09-11T03:41:25.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.842161+0000", 2024-09-11T03:41:25.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5721999999999999e-05 2024-09-11T03:41:25.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.842173+0000", 2024-09-11T03:41:25.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1931e-05 2024-09-11T03:41:25.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.842261+0000", 2024-09-11T03:41:25.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7911999999999994e-05 2024-09-11T03:41:25.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.842288+0000", 2024-09-11T03:41:25.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6673e-05 2024-09-11T03:41:25.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.844485+0000", 2024-09-11T03:41:25.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002197605 2024-09-11T03:41:25.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.844525+0000", 2024-09-11T03:41:25.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9690000000000001e-05 2024-09-11T03:41:25.699 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.700 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.700 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.700 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1048 2.9 e175/95 2:98bb65f5:::smithi17557870-6:head v 175'134, mlcod=168'130)", 2024-09-11T03:41:25.700 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:51.832890+0000", 2024-09-11T03:41:25.700 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 93.834031413000005, 2024-09-11T03:41:25.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062655929999999999, 2024-09-11T03:41:25.700 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.700 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.700 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.700 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.832890+0000", 2024-09-11T03:41:25.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.832890+0000", 2024-09-11T03:41:25.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.832890+0000", 2024-09-11T03:41:25.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8200000000000001e-07 2024-09-11T03:41:25.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.835037+0000", 2024-09-11T03:41:25.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021466749999999998 2024-09-11T03:41:25.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.835060+0000", 2024-09-11T03:41:25.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3458000000000001e-05 2024-09-11T03:41:25.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.835071+0000", 2024-09-11T03:41:25.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0721e-05 2024-09-11T03:41:25.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.835126+0000", 2024-09-11T03:41:25.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5513999999999997e-05 2024-09-11T03:41:25.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.835158+0000", 2024-09-11T03:41:25.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1841000000000002e-05 2024-09-11T03:41:25.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.839122+0000", 2024-09-11T03:41:25.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039640470000000001 2024-09-11T03:41:25.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.839155+0000", 2024-09-11T03:41:25.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3055000000000001e-05 2024-09-11T03:41:25.704 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.704 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.704 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.704 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1159 2.e e185/95 2:750de3b7:::smithi17557870-8:head v 185'112, mlcod=180'109)", 2024-09-11T03:41:25.705 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:01.840564+0000", 2024-09-11T03:41:25.705 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 83.826357039000001, 2024-09-11T03:41:25.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054067209999999997, 2024-09-11T03:41:25.705 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.705 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.705 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.705 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.840564+0000", 2024-09-11T03:41:25.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.840564+0000", 2024-09-11T03:41:25.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.840565+0000", 2024-09-11T03:41:25.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5999999999999999e-07 2024-09-11T03:41:25.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.843315+0000", 2024-09-11T03:41:25.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027503409999999999 2024-09-11T03:41:25.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.843345+0000", 2024-09-11T03:41:25.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9845999999999999e-05 2024-09-11T03:41:25.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.843357+0000", 2024-09-11T03:41:25.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2653999999999999e-05 2024-09-11T03:41:25.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.843447+0000", 2024-09-11T03:41:25.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0124000000000007e-05 2024-09-11T03:41:25.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.843475+0000", 2024-09-11T03:41:25.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8019999999999999e-05 2024-09-11T03:41:25.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.845926+0000", 2024-09-11T03:41:25.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024504029999999999 2024-09-11T03:41:25.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.845971+0000", 2024-09-11T03:41:25.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4972999999999998e-05 2024-09-11T03:41:25.710 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.710 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.710 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.710 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1290 2.d e196/95 2:b1c0f730:::smithi17557870-20:head v 196'156, mlcod=192'152)", 2024-09-11T03:41:25.710 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:12.850156+0000", 2024-09-11T03:41:25.710 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 72.816765107999998, 2024-09-11T03:41:25.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0080446430000000006, 2024-09-11T03:41:25.711 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.711 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.711 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.711 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.850156+0000", 2024-09-11T03:41:25.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.850156+0000", 2024-09-11T03:41:25.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.850157+0000", 2024-09-11T03:41:25.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7199999999999998e-07 2024-09-11T03:41:25.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853697+0000", 2024-09-11T03:41:25.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003539717 2024-09-11T03:41:25.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853706+0000", 2024-09-11T03:41:25.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4749999999999999e-06 2024-09-11T03:41:25.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853713+0000", 2024-09-11T03:41:25.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5089999999999999e-06 2024-09-11T03:41:25.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853795+0000", 2024-09-11T03:41:25.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2208999999999998e-05 2024-09-11T03:41:25.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853817+0000", 2024-09-11T03:41:25.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2129000000000001e-05 2024-09-11T03:41:25.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.858133+0000", 2024-09-11T03:41:25.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043163480000000002 2024-09-11T03:41:25.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.858201+0000", 2024-09-11T03:41:25.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7484000000000006e-05 2024-09-11T03:41:25.715 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.715 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.715 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.715 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1285 2.1 e196/95 2:8638b65e:::smithi17557870-5:head v 196'52, mlcod=173'49)", 2024-09-11T03:41:25.715 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:12.851406+0000", 2024-09-11T03:41:25.715 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 72.815515035000004, 2024-09-11T03:41:25.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046481020000000003, 2024-09-11T03:41:25.716 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.716 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.716 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.716 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.851406+0000", 2024-09-11T03:41:25.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.851406+0000", 2024-09-11T03:41:25.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.851408+0000", 2024-09-11T03:41:25.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6050000000000001e-06 2024-09-11T03:41:25.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853520+0000", 2024-09-11T03:41:25.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002112325 2024-09-11T03:41:25.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853542+0000", 2024-09-11T03:41:25.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1659e-05 2024-09-11T03:41:25.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853554+0000", 2024-09-11T03:41:25.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2091e-05 2024-09-11T03:41:25.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853661+0000", 2024-09-11T03:41:25.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107092 2024-09-11T03:41:25.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853691+0000", 2024-09-11T03:41:25.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9873e-05 2024-09-11T03:41:25.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.855975+0000", 2024-09-11T03:41:25.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002283916 2024-09-11T03:41:25.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856054+0000", 2024-09-11T03:41:25.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9541000000000003e-05 2024-09-11T03:41:25.720 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.720 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.720 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.720 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1434 2.e e204/95 2:7efbbea8:::smithi17557870-24:head v 204'148, mlcod=203'145)", 2024-09-11T03:41:25.720 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:20.870644+0000", 2024-09-11T03:41:25.721 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 64.796276798999997, 2024-09-11T03:41:25.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059669240000000002, 2024-09-11T03:41:25.721 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.721 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.721 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.721 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.870644+0000", 2024-09-11T03:41:25.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.870644+0000", 2024-09-11T03:41:25.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.870645+0000", 2024-09-11T03:41:25.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7000000000000004e-07 2024-09-11T03:41:25.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.872367+0000", 2024-09-11T03:41:25.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001722122 2024-09-11T03:41:25.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.872388+0000", 2024-09-11T03:41:25.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1064999999999999e-05 2024-09-11T03:41:25.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.872401+0000", 2024-09-11T03:41:25.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2179e-05 2024-09-11T03:41:25.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.872486+0000", 2024-09-11T03:41:25.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5871999999999994e-05 2024-09-11T03:41:25.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.872515+0000", 2024-09-11T03:41:25.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8435000000000001e-05 2024-09-11T03:41:25.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.876574+0000", 2024-09-11T03:41:25.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040593340000000004 2024-09-11T03:41:25.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.876611+0000", 2024-09-11T03:41:25.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7147000000000002e-05 2024-09-11T03:41:25.725 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.725 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.725 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.725 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1516 2.d e212/95 2:b6a25a79:::smithi17557870-47:head v 212'170, mlcod=206'166)", 2024-09-11T03:41:25.725 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:28.923076+0000", 2024-09-11T03:41:25.725 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 56.743845546999999, 2024-09-11T03:41:25.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081530539999999999, 2024-09-11T03:41:25.726 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.726 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.726 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.726 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.923076+0000", 2024-09-11T03:41:25.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.923076+0000", 2024-09-11T03:41:25.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.923076+0000", 2024-09-11T03:41:25.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1799999999999999e-07 2024-09-11T03:41:25.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927708+0000", 2024-09-11T03:41:25.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046320040000000003 2024-09-11T03:41:25.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927726+0000", 2024-09-11T03:41:25.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7730000000000001e-05 2024-09-11T03:41:25.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927734+0000", 2024-09-11T03:41:25.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2989999999999997e-06 2024-09-11T03:41:25.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927816+0000", 2024-09-11T03:41:25.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2256999999999995e-05 2024-09-11T03:41:25.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927842+0000", 2024-09-11T03:41:25.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5369999999999999e-05 2024-09-11T03:41:25.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.931178+0000", 2024-09-11T03:41:25.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033369039999999999 2024-09-11T03:41:25.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.931229+0000", 2024-09-11T03:41:25.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0272000000000003e-05 2024-09-11T03:41:25.730 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.730 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.730 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.730 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1524 2.6 e212/95 2:6dbd1e81:::smithi17557870-7:head v 212'123, mlcod=204'119)", 2024-09-11T03:41:25.730 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:28.927804+0000", 2024-09-11T03:41:25.730 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 56.739117094999997, 2024-09-11T03:41:25.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087752009999999998, 2024-09-11T03:41:25.730 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.730 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.730 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.730 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927804+0000", 2024-09-11T03:41:25.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927804+0000", 2024-09-11T03:41:25.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927804+0000", 2024-09-11T03:41:25.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8799999999999998e-07 2024-09-11T03:41:25.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929502+0000", 2024-09-11T03:41:25.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016974690000000001 2024-09-11T03:41:25.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929523+0000", 2024-09-11T03:41:25.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1373999999999999e-05 2024-09-11T03:41:25.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929543+0000", 2024-09-11T03:41:25.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9950999999999999e-05 2024-09-11T03:41:25.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929633+0000", 2024-09-11T03:41:25.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0158e-05 2024-09-11T03:41:25.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929657+0000", 2024-09-11T03:41:25.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3737999999999999e-05 2024-09-11T03:41:25.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.936522+0000", 2024-09-11T03:41:25.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068650589999999997 2024-09-11T03:41:25.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.936579+0000", 2024-09-11T03:41:25.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7163999999999997e-05 2024-09-11T03:41:25.734 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.735 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.735 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.735 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1526 2.1 e212/95 2:8638b65e:::smithi17557870-5:head v 212'54, mlcod=186'51)", 2024-09-11T03:41:25.735 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:28.929630+0000", 2024-09-11T03:41:25.735 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 56.737291321000001, 2024-09-11T03:41:25.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071503080000000002, 2024-09-11T03:41:25.736 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.736 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.736 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.736 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929630+0000", 2024-09-11T03:41:25.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929630+0000", 2024-09-11T03:41:25.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929631+0000", 2024-09-11T03:41:25.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.282e-06 2024-09-11T03:41:25.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.930342+0000", 2024-09-11T03:41:25.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071113199999999995 2024-09-11T03:41:25.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.930359+0000", 2024-09-11T03:41:25.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6512999999999999e-05 2024-09-11T03:41:25.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.930370+0000", 2024-09-11T03:41:25.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1362999999999999e-05 2024-09-11T03:41:25.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.930438+0000", 2024-09-11T03:41:25.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8319999999999996e-05 2024-09-11T03:41:25.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.930456+0000", 2024-09-11T03:41:25.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7901e-05 2024-09-11T03:41:25.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.936716+0000", 2024-09-11T03:41:25.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062595539999999996 2024-09-11T03:41:25.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.936780+0000", 2024-09-11T03:41:25.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4243000000000001e-05 2024-09-11T03:41:25.744 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.745 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.745 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.745 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1545 2.9 e213/95 2:92af84a9:::smithi17557870-33:head v 213'219, mlcod=213'217)", 2024-09-11T03:41:25.745 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:29.887838+0000", 2024-09-11T03:41:25.746 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 55.779083327999999, 2024-09-11T03:41:25.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.006347826, 2024-09-11T03:41:25.746 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.746 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.746 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.746 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.887838+0000", 2024-09-11T03:41:25.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.887838+0000", 2024-09-11T03:41:25.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.887840+0000", 2024-09-11T03:41:25.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7320000000000001e-06 2024-09-11T03:41:25.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.889338+0000", 2024-09-11T03:41:25.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001498468 2024-09-11T03:41:25.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.889364+0000", 2024-09-11T03:41:25.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5528999999999999e-05 2024-09-11T03:41:25.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.889374+0000", 2024-09-11T03:41:25.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0515e-05 2024-09-11T03:41:25.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.889443+0000", 2024-09-11T03:41:25.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8479999999999995e-05 2024-09-11T03:41:25.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.889466+0000", 2024-09-11T03:41:25.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3153999999999999e-05 2024-09-11T03:41:25.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.894151+0000", 2024-09-11T03:41:25.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046857019999999999 2024-09-11T03:41:25.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:29.894186+0000", 2024-09-11T03:41:25.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4245999999999997e-05 2024-09-11T03:41:25.754 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.754 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.754 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.754 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1589 2.7 e218/191 2:e77ce595:::smithi17557870-31:head v 218'174, mlcod=218'168)", 2024-09-11T03:41:25.754 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:35.116398+0000", 2024-09-11T03:41:25.754 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 50.550522764, 2024-09-11T03:41:25.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0089646050000000005, 2024-09-11T03:41:25.754 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.754 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.755 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.755 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116398+0000", 2024-09-11T03:41:25.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116398+0000", 2024-09-11T03:41:25.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116400+0000", 2024-09-11T03:41:25.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6109999999999999e-06 2024-09-11T03:41:25.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119261+0000", 2024-09-11T03:41:25.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028605309999999999 2024-09-11T03:41:25.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119281+0000", 2024-09-11T03:41:25.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0491000000000001e-05 2024-09-11T03:41:25.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119292+0000", 2024-09-11T03:41:25.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0759e-05 2024-09-11T03:41:25.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119350+0000", 2024-09-11T03:41:25.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8588999999999998e-05 2024-09-11T03:41:25.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119369+0000", 2024-09-11T03:41:25.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8227e-05 2024-09-11T03:41:25.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125320+0000", 2024-09-11T03:41:25.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059510409999999998 2024-09-11T03:41:25.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125363+0000", 2024-09-11T03:41:25.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3356000000000003e-05 2024-09-11T03:41:25.759 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.759 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.759 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.759 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1634 2.7 e219/191 2:e23d1762:::smithi17557870-21:head v 219'181, mlcod=219'179)", 2024-09-11T03:41:25.759 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:36.216756+0000", 2024-09-11T03:41:25.759 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 49.450165519999999, 2024-09-11T03:41:25.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074123089999999997, 2024-09-11T03:41:25.759 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.759 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.759 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.759 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.216756+0000", 2024-09-11T03:41:25.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.216756+0000", 2024-09-11T03:41:25.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.216756+0000", 2024-09-11T03:41:25.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1699999999999997e-07 2024-09-11T03:41:25.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.218798+0000", 2024-09-11T03:41:25.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020419280000000001 2024-09-11T03:41:25.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.218810+0000", 2024-09-11T03:41:25.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2069000000000001e-05 2024-09-11T03:41:25.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.218817+0000", 2024-09-11T03:41:25.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2999999999999998e-06 2024-09-11T03:41:25.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.218869+0000", 2024-09-11T03:41:25.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2380000000000003e-05 2024-09-11T03:41:25.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.218888+0000", 2024-09-11T03:41:25.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8524000000000001e-05 2024-09-11T03:41:25.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224125+0000", 2024-09-11T03:41:25.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0052376059999999997 2024-09-11T03:41:25.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224168+0000", 2024-09-11T03:41:25.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2685000000000002e-05 2024-09-11T03:41:25.766 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.766 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.766 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.767 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1816 2.d e243/95 2:b7bdfe52:::smithi17557870-10:head v 243'237, mlcod=243'233)", 2024-09-11T03:41:25.767 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:59.922567+0000", 2024-09-11T03:41:25.767 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 25.744354413, 2024-09-11T03:41:25.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094601100000000007, 2024-09-11T03:41:25.768 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.768 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.768 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.768 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.922567+0000", 2024-09-11T03:41:25.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.922567+0000", 2024-09-11T03:41:25.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.922568+0000", 2024-09-11T03:41:25.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.345e-06 2024-09-11T03:41:25.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929634+0000", 2024-09-11T03:41:25.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0070657130000000004 2024-09-11T03:41:25.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929654+0000", 2024-09-11T03:41:25.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9972000000000001e-05 2024-09-11T03:41:25.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929663+0000", 2024-09-11T03:41:25.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3770000000000002e-06 2024-09-11T03:41:25.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929731+0000", 2024-09-11T03:41:25.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8095000000000004e-05 2024-09-11T03:41:25.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929759+0000", 2024-09-11T03:41:25.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7248999999999999e-05 2024-09-11T03:41:25.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.931991+0000", 2024-09-11T03:41:25.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022320679999999998 2024-09-11T03:41:25.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.932027+0000", 2024-09-11T03:41:25.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6291000000000002e-05 2024-09-11T03:41:25.777 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.777 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.777 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.778 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1866 2.e e245/95 2:7b0c3397:::smithi17557870-14:head v 245'242, mlcod=243'238)", 2024-09-11T03:41:25.778 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:01.857519+0000", 2024-09-11T03:41:25.778 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 23.809402250000002, 2024-09-11T03:41:25.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042409709999999996, 2024-09-11T03:41:25.778 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.779 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.779 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.779 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.857519+0000", 2024-09-11T03:41:25.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.857519+0000", 2024-09-11T03:41:25.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.857519+0000", 2024-09-11T03:41:25.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1300000000000001e-07 2024-09-11T03:41:25.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.859673+0000", 2024-09-11T03:41:25.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021534639999999999 2024-09-11T03:41:25.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.859700+0000", 2024-09-11T03:41:25.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7535e-05 2024-09-11T03:41:25.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.859713+0000", 2024-09-11T03:41:25.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2958e-05 2024-09-11T03:41:25.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.859798+0000", 2024-09-11T03:41:25.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4825000000000002e-05 2024-09-11T03:41:25.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.859818+0000", 2024-09-11T03:41:25.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9661e-05 2024-09-11T03:41:25.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.861715+0000", 2024-09-11T03:41:25.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001897298 2024-09-11T03:41:25.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.861760+0000", 2024-09-11T03:41:25.788 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:25.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4916999999999997e-05 2024-09-11T03:41:25.789 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.789 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.789 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.790 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1895 2.e e249/95 2:7efbbea8:::smithi17557870-24:head v 249'251, mlcod=246'244)", 2024-09-11T03:41:25.790 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:05.931958+0000", 2024-09-11T03:41:25.790 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 19.734962881000001, 2024-09-11T03:41:25.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054396949999999996, 2024-09-11T03:41:25.790 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:41:25.790 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:41:25.790 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:25.790 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:41:25.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:41:25.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.931958+0000", 2024-09-11T03:41:25.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:41:25.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.931958+0000", 2024-09-11T03:41:25.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:41:25.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:41:25.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.931959+0000", 2024-09-11T03:41:25.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1699999999999997e-07 2024-09-11T03:41:25.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:41:25.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.933281+0000", 2024-09-11T03:41:25.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0013219779999999999 2024-09-11T03:41:25.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:41:25.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.933303+0000", 2024-09-11T03:41:25.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2076e-05 2024-09-11T03:41:25.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:41:25.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.933313+0000", 2024-09-11T03:41:25.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6059999999999992e-06 2024-09-11T03:41:25.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:41:25.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.933380+0000", 2024-09-11T03:41:25.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7279000000000004e-05 2024-09-11T03:41:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:41:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.933400+0000", 2024-09-11T03:41:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0292000000000002e-05 2024-09-11T03:41:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:41:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.937348+0000", 2024-09-11T03:41:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039472919999999998 2024-09-11T03:41:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:41:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:41:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:41:25.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.937398+0000", 2024-09-11T03:41:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0454999999999997e-05 2024-09-11T03:41:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:41:25.794 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:41:25.794 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:41:25.814 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:25.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:25.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:25.861 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:41:25.916 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:25.939 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:25.940 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:25.983 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:41:25.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:25 smithi084 ceph-mon[27254]: pgmap v556: 47 pgs: 47 active+clean; 142 MiB data, 605 MiB used, 893 GiB / 894 GiB avail; 19 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-11T03:41:25.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:25 smithi084 ceph-mon[27254]: osdmap e268: 12 total, 12 up, 10 in 2024-09-11T03:41:25.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:25 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1423550331' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:25.989 INFO:tasks.rados.rados.2.smithi175.stdout:852: done (3 left) 2024-09-11T03:41:25.989 INFO:tasks.rados.rados.2.smithi175.stdout:853: delete oid 45 current snap is 86 2024-09-11T03:41:25.989 INFO:tasks.rados.rados.2.smithi175.stdout:849: finishing write tid 3 to smithi17557870-38 2024-09-11T03:41:25.989 INFO:tasks.rados.rados.2.smithi175.stdout:849: finishing write tid 4 to smithi17557870-38 2024-09-11T03:41:25.989 INFO:tasks.rados.rados.2.smithi175.stdout:849: finishing write tid 5 to smithi17557870-38 2024-09-11T03:41:25.989 INFO:tasks.rados.rados.2.smithi175.stdout:849: oid 38 updating version 0 to 260 2024-09-11T03:41:25.989 INFO:tasks.rados.rados.2.smithi175.stdout:849: oid 38 version 260 is already newer than 257 2024-09-11T03:41:25.989 INFO:tasks.rados.rados.2.smithi175.stdout:849: oid 38 version 260 is already newer than 258 2024-09-11T03:41:25.989 INFO:tasks.rados.rados.2.smithi175.stdout:849: oid 38 version 260 is already newer than 259 2024-09-11T03:41:25.989 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 260 (ObjNum 318 snap 86 seq_num 318) dirty exists 2024-09-11T03:41:25.989 INFO:tasks.rados.rados.2.smithi175.stdout:849: left oid 38 (ObjNum 318 snap 86 seq_num 318) 2024-09-11T03:41:25.989 INFO:tasks.rados.rados.2.smithi175.stdout:850: finishing copy_from racing read to smithi17557870-35 2024-09-11T03:41:25.990 INFO:tasks.rados.rados.2.smithi175.stdout:851: finishing write tid 1 to smithi17557870-8 2024-09-11T03:41:25.990 INFO:tasks.rados.rados.2.smithi175.stdout:850: finishing copy_from to smithi17557870-35 2024-09-11T03:41:25.990 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 101 (ObjNum 153 snap 35 seq_num 153) dirty exists 2024-09-11T03:41:25.990 INFO:tasks.rados.rados.2.smithi175.stdout:851: finishing write tid 2 to smithi17557870-8 2024-09-11T03:41:25.990 INFO:tasks.rados.rados.2.smithi175.stdout:851: finishing write tid 3 to smithi17557870-8 2024-09-11T03:41:25.990 INFO:tasks.rados.rados.2.smithi175.stdout:851: finishing write tid 4 to smithi17557870-8 2024-09-11T03:41:25.990 INFO:tasks.rados.rados.2.smithi175.stdout:851: finishing write tid 5 to smithi17557870-8 2024-09-11T03:41:25.990 INFO:tasks.rados.rados.2.smithi175.stdout:851: oid 8 updating version 0 to 289 2024-09-11T03:41:25.990 INFO:tasks.rados.rados.2.smithi175.stdout:851: oid 8 version 289 is already newer than 288 2024-09-11T03:41:25.990 INFO:tasks.rados.rados.2.smithi175.stdout:851: oid 8 version 289 is already newer than 287 2024-09-11T03:41:25.990 INFO:tasks.rados.rados.2.smithi175.stdout:851: oid 8 version 289 is already newer than 286 2024-09-11T03:41:25.990 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 289 (ObjNum 319 snap 86 seq_num 319) dirty exists 2024-09-11T03:41:25.990 INFO:tasks.rados.rados.2.smithi175.stdout:851: left oid 8 (ObjNum 319 snap 86 seq_num 319) 2024-09-11T03:41:25.994 INFO:tasks.rados.rados.2.smithi175.stdout:849: done (3 left) 2024-09-11T03:41:25.994 INFO:tasks.rados.rados.2.smithi175.stdout:850: done (2 left) 2024-09-11T03:41:25.994 INFO:tasks.rados.rados.2.smithi175.stdout:851: done (1 left) 2024-09-11T03:41:25.994 INFO:tasks.rados.rados.2.smithi175.stdout:853: done (0 left) 2024-09-11T03:41:25.994 INFO:tasks.rados.rados.2.smithi175.stdout:854: read oid 8 snap -1 2024-09-11T03:41:25.994 INFO:tasks.rados.rados.2.smithi175.stdout:854: expect (ObjNum 319 snap 86 seq_num 319) 2024-09-11T03:41:25.995 INFO:tasks.rados.rados.2.smithi175.stdout:855: rollback oid 27 current snap is 86 2024-09-11T03:41:25.995 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 27 to 84 2024-09-11T03:41:25.995 INFO:tasks.rados.rados.2.smithi175.stdout:856: snap_create 2024-09-11T03:41:26.000 INFO:tasks.rados.rados.2.smithi175.stdout:855: finishing rollback tid 0 to smithi17557870-27 2024-09-11T03:41:26.000 INFO:tasks.rados.rados.2.smithi175.stdout:855: finishing rollback tid 1 to smithi17557870-27 2024-09-11T03:41:26.000 INFO:tasks.rados.rados.2.smithi175.stdout:855: finishing rollback tid 2 to smithi17557870-27 2024-09-11T03:41:26.000 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 300 (ObjNum 302 snap 82 seq_num 302) dirty exists 2024-09-11T03:41:26.011 INFO:tasks.rados.rados.2.smithi175.stdout:854: expect (ObjNum 319 snap 86 seq_num 319) 2024-09-11T03:41:26.041 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:26.062 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:26.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:26.063 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:26.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:26.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:25 smithi121 ceph-mon[30729]: pgmap v556: 47 pgs: 47 active+clean; 142 MiB data, 605 MiB used, 893 GiB / 894 GiB avail; 19 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-11T03:41:26.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:25 smithi121 ceph-mon[30729]: osdmap e268: 12 total, 12 up, 10 in 2024-09-11T03:41:26.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:25 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1423550331' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:26.190 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:26.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:26.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:25 smithi175 ceph-mon[30894]: pgmap v556: 47 pgs: 47 active+clean; 142 MiB data, 605 MiB used, 893 GiB / 894 GiB avail; 19 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-11T03:41:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:25 smithi175 ceph-mon[30894]: osdmap e268: 12 total, 12 up, 10 in 2024-09-11T03:41:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:25 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1423550331' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:26.292 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:26.317 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:26.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:26.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:26.419 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:26.441 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:26.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:26.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:26.569 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:26.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:26.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:26.671 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:26.694 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:26.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:26.695 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:26.753 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [6, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:41:26.753 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:41:26.754 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-11T03:41:26.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:41:26.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T03:41:26.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:26.829 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:26.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:26.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:26.856 INFO:tasks.rados.rados.2.smithi175.stdout:854: done (2 left) 2024-09-11T03:41:26.856 INFO:tasks.rados.rados.2.smithi175.stdout:855: done (1 left) 2024-09-11T03:41:26.856 INFO:tasks.rados.rados.2.smithi175.stdout:856: done (0 left) 2024-09-11T03:41:26.856 INFO:tasks.rados.rados.2.smithi175.stdout:857: rollback oid 14 current snap is 87 2024-09-11T03:41:26.857 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 14 to 85 2024-09-11T03:41:26.857 INFO:tasks.rados.rados.2.smithi175.stdout:858: read oid 28 snap -1 2024-09-11T03:41:26.857 INFO:tasks.rados.rados.2.smithi175.stdout:858: expect (ObjNum 279 snap 73 seq_num 279) 2024-09-11T03:41:26.857 INFO:tasks.rados.rados.2.smithi175.stdout:859: read oid 11 snap -1 2024-09-11T03:41:26.857 INFO:tasks.rados.rados.2.smithi175.stdout:859: expect (ObjNum 270 snap 69 seq_num 270) 2024-09-11T03:41:26.857 INFO:tasks.rados.rados.2.smithi175.stdout:860: copy_from oid 41 from oid 37 current snap is 87 2024-09-11T03:41:26.857 INFO:tasks.rados.rados.2.smithi175.stdout:861: snap_create 2024-09-11T03:41:26.861 INFO:tasks.rados.rados.2.smithi175.stdout:857: finishing rollback tid 0 to smithi17557870-14 2024-09-11T03:41:26.861 INFO:tasks.rados.rados.2.smithi175.stdout:857: finishing rollback tid 1 to smithi17557870-14 2024-09-11T03:41:26.861 INFO:tasks.rados.rados.2.smithi175.stdout:857: finishing rollback tid 2 to smithi17557870-14 2024-09-11T03:41:26.861 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 297 (ObjNum 258 snap 64 seq_num 258) dirty exists 2024-09-11T03:41:26.863 INFO:tasks.rados.rados.2.smithi175.stdout:858: expect (ObjNum 279 snap 73 seq_num 279) 2024-09-11T03:41:26.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:26.952 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:26.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:26.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:26.985 INFO:tasks.rados.rados.2.smithi175.stdout:859: expect (ObjNum 270 snap 69 seq_num 270) 2024-09-11T03:41:27.026 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:41:27.054 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:27.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:26 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:41:27.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:26 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:41:27.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1423550331' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:27.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:26 smithi084 ceph-mon[27254]: osdmap e269: 12 total, 12 up, 10 in 2024-09-11T03:41:27.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:26 smithi084 ceph-mon[27254]: osdmap e270: 12 total, 12 up, 10 in 2024-09-11T03:41:27.062 INFO:tasks.rados.rados.2.smithi175.stdout:860: finishing copy_from racing read to smithi17557870-41 2024-09-11T03:41:27.062 INFO:tasks.rados.rados.2.smithi175.stdout:860: finishing copy_from to smithi17557870-41 2024-09-11T03:41:27.062 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 101 (ObjNum 153 snap 35 seq_num 153) dirty exists 2024-09-11T03:41:27.075 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:27.075 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:27.137 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:27.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:26 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:41:27.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:26 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:41:27.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1423550331' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:27.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:26 smithi121 ceph-mon[30729]: osdmap e269: 12 total, 12 up, 10 in 2024-09-11T03:41:27.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:26 smithi121 ceph-mon[30729]: osdmap e270: 12 total, 12 up, 10 in 2024-09-11T03:41:27.153 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:27.176 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:27.199 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:27.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:26 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:41:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:26 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:41:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1423550331' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:26 smithi175 ceph-mon[30894]: osdmap e269: 12 total, 12 up, 10 in 2024-09-11T03:41:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:26 smithi175 ceph-mon[30894]: osdmap e270: 12 total, 12 up, 10 in 2024-09-11T03:41:27.300 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:27.321 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:27.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:27.322 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:27.423 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:27.444 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:27.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:27.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:27.521 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:41:27.521 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:41:27.521 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:41:27.521 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:41:27.546 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:27.566 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:27.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:27.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:27.668 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:27.691 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:27.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:27.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:27.793 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:27.815 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:27.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:27.816 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:27.866 INFO:tasks.rados.rados.2.smithi175.stdout:857: done (4 left) 2024-09-11T03:41:27.866 INFO:tasks.rados.rados.2.smithi175.stdout:858: done (3 left) 2024-09-11T03:41:27.866 INFO:tasks.rados.rados.2.smithi175.stdout:859: done (2 left) 2024-09-11T03:41:27.866 INFO:tasks.rados.rados.2.smithi175.stdout:860: done (1 left) 2024-09-11T03:41:27.866 INFO:tasks.rados.rados.2.smithi175.stdout:861: done (0 left) 2024-09-11T03:41:27.866 INFO:tasks.rados.rados.2.smithi175.stdout:862: read oid 37 snap -1 2024-09-11T03:41:27.866 INFO:tasks.rados.rados.2.smithi175.stdout:862: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:41:27.866 INFO:tasks.rados.rados.2.smithi175.stdout:863: read oid 46 snap -1 2024-09-11T03:41:27.866 INFO:tasks.rados.rados.2.smithi175.stdout:863: expect (ObjNum 296 snap 79 seq_num 296) 2024-09-11T03:41:27.867 INFO:tasks.rados.rados.2.smithi175.stdout:864: write oid 6 current snap is 88 2024-09-11T03:41:27.867 INFO:tasks.rados.rados.2.smithi175.stdout:864: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2024-09-11T03:41:27.872 INFO:tasks.rados.rados.2.smithi175.stdout:864: writing smithi17557870-6 from 595047 to 1236802 tid 1 2024-09-11T03:41:27.877 INFO:tasks.rados.rados.2.smithi175.stdout:864: writing smithi17557870-6 from 1667661 to 2085770 tid 2 2024-09-11T03:41:27.884 INFO:tasks.rados.rados.2.smithi175.stdout:864: writing smithi17557870-6 from 2548663 to 3311685 tid 3 2024-09-11T03:41:27.884 INFO:tasks.rados.rados.2.smithi175.stdout:865: delete oid 48 current snap is 88 2024-09-11T03:41:27.884 INFO:tasks.rados.rados.2.smithi175.stdout:863: expect (ObjNum 296 snap 79 seq_num 296) 2024-09-11T03:41:27.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:27.939 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:27.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:27.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:27.957 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:41:27.958 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:41:27.979 INFO:tasks.rados.rados.2.smithi175.stdout:862: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:41:27.986 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:41:27.986 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:41:28.004 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:41:28.005 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:41:28.020 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:41:28.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:41:28.036 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:41:28.036 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:41:28.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:28.054 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:41:28.054 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:41:28.062 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:28.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:28.072 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:41:28.072 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:41:28.088 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:41:28.089 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:41:28.100 INFO:tasks.rados.rados.2.smithi175.stdout:864: finishing write tid 1 to smithi17557870-6 2024-09-11T03:41:28.100 INFO:tasks.rados.rados.2.smithi175.stdout:864: finishing write tid 2 to smithi17557870-6 2024-09-11T03:41:28.100 INFO:tasks.rados.rados.2.smithi175.stdout:864: finishing write tid 3 to smithi17557870-6 2024-09-11T03:41:28.101 INFO:tasks.rados.rados.2.smithi175.stdout:864: finishing write tid 4 to smithi17557870-6 2024-09-11T03:41:28.101 INFO:tasks.rados.rados.2.smithi175.stdout:864: finishing write tid 5 to smithi17557870-6 2024-09-11T03:41:28.101 INFO:tasks.rados.rados.2.smithi175.stdout:864: oid 6 updating version 0 to 324 2024-09-11T03:41:28.101 INFO:tasks.rados.rados.2.smithi175.stdout:864: oid 6 updating version 324 to 326 2024-09-11T03:41:28.101 INFO:tasks.rados.rados.2.smithi175.stdout:864: oid 6 updating version 326 to 327 2024-09-11T03:41:28.101 INFO:tasks.rados.rados.2.smithi175.stdout:864: oid 6 version 327 is already newer than 325 2024-09-11T03:41:28.101 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 327 (ObjNum 321 snap 88 seq_num 321) dirty exists 2024-09-11T03:41:28.101 INFO:tasks.rados.rados.2.smithi175.stdout:864: left oid 6 (ObjNum 321 snap 88 seq_num 321) 2024-09-11T03:41:28.101 INFO:tasks.rados.rados.2.smithi175.stdout:862: done (3 left) 2024-09-11T03:41:28.101 INFO:tasks.rados.rados.2.smithi175.stdout:863: done (2 left) 2024-09-11T03:41:28.101 INFO:tasks.rados.rados.2.smithi175.stdout:864: done (1 left) 2024-09-11T03:41:28.101 INFO:tasks.rados.rados.2.smithi175.stdout:865: done (0 left) 2024-09-11T03:41:28.101 INFO:tasks.rados.rados.2.smithi175.stdout:866: read oid 3 snap 84 2024-09-11T03:41:28.102 INFO:tasks.rados.rados.2.smithi175.stdout:866: expect (ObjNum 290 snap 78 seq_num 290) 2024-09-11T03:41:28.102 INFO:tasks.rados.rados.2.smithi175.stdout:867: snap_create 2024-09-11T03:41:28.106 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:41:28.106 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:41:28.110 INFO:tasks.rados.rados.2.smithi175.stdout:866: expect (ObjNum 290 snap 78 seq_num 290) 2024-09-11T03:41:28.123 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:41:28.123 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:41:28.140 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:41:28.140 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:41:28.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:27 smithi084 ceph-mon[27254]: pgmap v559: 47 pgs: 47 active+clean; 134 MiB data, 614 MiB used, 893 GiB / 894 GiB avail; 3.5 MiB/s rd, 4.8 MiB/s wr, 14 op/s 2024-09-11T03:41:28.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:27 smithi084 ceph-mon[27254]: osdmap e271: 12 total, 12 up, 10 in 2024-09-11T03:41:28.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:27 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2907212505' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:28.158 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:41:28.158 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:41:28.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:28.174 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:41:28.174 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:41:28.186 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:28.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:28.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:28.288 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:28.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:27 smithi121 ceph-mon[30729]: pgmap v559: 47 pgs: 47 active+clean; 134 MiB data, 614 MiB used, 893 GiB / 894 GiB avail; 3.5 MiB/s rd, 4.8 MiB/s wr, 14 op/s 2024-09-11T03:41:28.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:27 smithi121 ceph-mon[30729]: osdmap e271: 12 total, 12 up, 10 in 2024-09-11T03:41:28.291 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:27 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2907212505' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:28.310 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:28.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:28.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:28.318 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T03:41:28.411 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:28.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:28.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:27 smithi175 ceph-mon[30894]: pgmap v559: 47 pgs: 47 active+clean; 134 MiB data, 614 MiB used, 893 GiB / 894 GiB avail; 3.5 MiB/s rd, 4.8 MiB/s wr, 14 op/s 2024-09-11T03:41:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:27 smithi175 ceph-mon[30894]: osdmap e271: 12 total, 12 up, 10 in 2024-09-11T03:41:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:27 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2907212505' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:28.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:28.556 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:28.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:28.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:28.564 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:41:28.658 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:28.682 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:28.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:28.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:28.784 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:28.809 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:28.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:28.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:28.910 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":561,"stamp":"2024-09-11T03:41:27.864582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":139319597,"num_objects":61,"num_object_clones":20,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":2,"num_read":1494,"num_read_kb":1206525,"num_write":1254,"num_write_kb":518714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":13624269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5882,"ondisk_log_size":5882,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":630760,"kb_used_data":304256,"kb_used_omap":15,"kb_used_meta":324144,"kb_avail":936820760,"statfs":{"total":959950356480,"available":959304458240,"internally_reserved":0,"allocated":311558144,"data_stored":335939292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":331923893},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9868380,"num_objects":-7,"num_object_clones":-6,"num_object_copies":-21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7,"num_whiteouts":-2,"num_read":7,"num_read_kb":2,"num_write":16,"num_write_kb":11160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.010231"},"pg_stats":[{"pgid":"1.1e","version":"20'92","reported_seq":114,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-11T03:41:26.873462+0000","last_change":"2024-09-11T03:40:40.236609+0000","last_active":"2024-09-11T03:41:26.873462+0000","last_peered":"2024-09-11T03:41:26.873462+0000","last_clean":"2024-09-11T03:41:26.873462+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:41:26.873462+0000","last_undegraded":"2024-09-11T03:41:26.873462+0000","last_fullsized":"2024-09-11T03:41:26.873462+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.236546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.236546+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:03.367532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'92","reported_seq":114,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.989181+0000","last_change":"2024-09-11T03:40:41.348889+0000","last_active":"2024-09-11T03:41:25.989181+0000","last_peered":"2024-09-11T03:41:25.989181+0000","last_clean":"2024-09-11T03:41:25.989181+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:41:25.989181+0000","last_undegraded":"2024-09-11T03:41:25.989181+0000","last_fullsized":"2024-09-11T03:41:25.989181+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:41.348828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:41.348828+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:09.856306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'92","reported_seq":112,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.987986+0000","last_change":"2024-09-11T03:40:40.188174+0000","last_active":"2024-09-11T03:41:25.987986+0000","last_peered":"2024-09-11T03:41:25.987986+0000","last_clean":"2024-09-11T03:41:25.987986+0000","last_became_active":"2024-09-11T03:40:39.848230+0000","last_became_peered":"2024-09-11T03:40:39.848230+0000","last_unstale":"2024-09-11T03:41:25.987986+0000","last_undegraded":"2024-09-11T03:41:25.987986+0000","last_fullsized":"2024-09-11T03:41:25.987986+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.188124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.188124+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:05.549148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"20'92","reported_seq":112,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.987890+0000","last_change":"2024-09-11T03:40:42.166354+0000","last_active":"2024-09-11T03:41:25.987890+0000","last_peered":"2024-09-11T03:41:25.987890+0000","last_clean":"2024-09-11T03:41:25.987890+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:41:25.987890+0000","last_undegraded":"2024-09-11T03:41:25.987890+0000","last_fullsized":"2024-09-11T03:41:25.987890+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:42.166295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:42.166295+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:10.918752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"20'92","reported_seq":353,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-11T03:41:24.862085+0000","last_change":"2024-09-11T03:40:40.278752+0000","last_active":"2024-09-11T03:41:24.862085+0000","last_peered":"2024-09-11T03:41:24.862085+0000","last_clean":"2024-09-11T03:41:24.862085+0000","last_became_active":"2024-09-11T03:40:39.848781+0000","last_became_peered":"2024-09-11T03:40:39.848781+0000","last_unstale":"2024-09-11T03:41:24.862085+0000","last_undegraded":"2024-09-11T03:41:24.862085+0000","last_fullsized":"2024-09-11T03:41:24.862085+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.278703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.278703+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:06.105420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"20'92","reported_seq":112,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.989947+0000","last_change":"2024-09-11T03:40:40.718553+0000","last_active":"2024-09-11T03:41:25.989947+0000","last_peered":"2024-09-11T03:41:25.989947+0000","last_clean":"2024-09-11T03:41:25.989947+0000","last_became_active":"2024-09-11T03:40:39.851698+0000","last_became_peered":"2024-09-11T03:40:39.851698+0000","last_unstale":"2024-09-11T03:41:25.989947+0000","last_undegraded":"2024-09-11T03:41:25.989947+0000","last_fullsized":"2024-09-11T03:41:25.989947+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.718503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.718503+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:06.580046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.18","version":"20'92","reported_seq":227,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.986612+0000","last_change":"2024-09-11T03:40:40.365656+0000","last_active":"2024-09-11T03:41:25.986612+0000","last_peered":"2024-09-11T03:41:25.986612+0000","last_clean":"2024-09-11T03:41:25.986612+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:41:25.986612+0000","last_undegraded":"2024-09-11T03:41:25.986612+0000","last_fullsized":"2024-09-11T03:41:25.986612+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.365604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.365604+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:00.211289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"20'92","reported_seq":112,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.987975+0000","last_change":"2024-09-11T03:40:41.207341+0000","last_active":"2024-09-11T03:41:25.987975+0000","last_peered":"2024-09-11T03:41:25.987975+0000","last_clean":"2024-09-11T03:41:25.987975+0000","last_became_active":"2024-09-11T03:40:39.850243+0000","last_became_peered":"2024-09-11T03:40:39.850243+0000","last_unstale":"2024-09-11T03:41:25.987975+0000","last_undegraded":"2024-09-11T03:41:25.987975+0000","last_fullsized":"2024-09-11T03:41:25.987975+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:41.207291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:41.207291+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:56.540051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"20'92","reported_seq":112,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.989477+0000","last_change":"2024-09-11T03:40:40.309475+0000","last_active":"2024-09-11T03:41:25.989477+0000","last_peered":"2024-09-11T03:41:25.989477+0000","last_clean":"2024-09-11T03:41:25.989477+0000","last_became_active":"2024-09-11T03:40:39.849950+0000","last_became_peered":"2024-09-11T03:40:39.849950+0000","last_unstale":"2024-09-11T03:41:25.989477+0000","last_undegraded":"2024-09-11T03:41:25.989477+0000","last_fullsized":"2024-09-11T03:41:25.989477+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.309432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.309432+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:04.484327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"20'92","reported_seq":112,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.989270+0000","last_change":"2024-09-11T03:40:39.928143+0000","last_active":"2024-09-11T03:41:25.989270+0000","last_peered":"2024-09-11T03:41:25.989270+0000","last_clean":"2024-09-11T03:41:25.989270+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:41:25.989270+0000","last_undegraded":"2024-09-11T03:41:25.989270+0000","last_fullsized":"2024-09-11T03:41:25.989270+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.928088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.928088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:06.193518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"20'92","reported_seq":134,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-11T03:41:21.856042+0000","last_change":"2024-09-11T03:40:35.195938+0000","last_active":"2024-09-11T03:41:21.856042+0000","last_peered":"2024-09-11T03:41:21.856042+0000","last_clean":"2024-09-11T03:41:21.856042+0000","last_became_active":"2024-09-11T03:40:27.847297+0000","last_became_peered":"2024-09-11T03:40:27.847297+0000","last_unstale":"2024-09-11T03:41:21.856042+0000","last_undegraded":"2024-09-11T03:41:21.856042+0000","last_fullsized":"2024-09-11T03:41:21.856042+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.195893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.195893+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:43.112855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"20'92","reported_seq":221,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.988940+0000","last_change":"2024-09-11T03:40:35.277715+0000","last_active":"2024-09-11T03:41:25.988940+0000","last_peered":"2024-09-11T03:41:25.988940+0000","last_clean":"2024-09-11T03:41:25.988940+0000","last_became_active":"2024-09-11T03:40:27.847557+0000","last_became_peered":"2024-09-11T03:40:27.847557+0000","last_unstale":"2024-09-11T03:41:25.988940+0000","last_undegraded":"2024-09-11T03:41:25.988940+0000","last_fullsized":"2024-09-11T03:41:25.988940+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.277668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.277668+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:39.014977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"20'92","reported_seq":140,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.986620+0000","last_change":"2024-09-11T03:40:37.284871+0000","last_active":"2024-09-11T03:41:25.986620+0000","last_peered":"2024-09-11T03:41:25.986620+0000","last_clean":"2024-09-11T03:41:25.986620+0000","last_became_active":"2024-09-11T03:40:27.845249+0000","last_became_peered":"2024-09-11T03:40:27.845249+0000","last_unstale":"2024-09-11T03:41:25.986620+0000","last_undegraded":"2024-09-11T03:41:25.986620+0000","last_fullsized":"2024-09-11T03:41:25.986620+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:37.284800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:37.284800+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:53.535974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'92","reported_seq":144,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.990075+0000","last_change":"2024-09-11T03:40:35.703292+0000","last_active":"2024-09-11T03:41:25.990075+0000","last_peered":"2024-09-11T03:41:25.990075+0000","last_clean":"2024-09-11T03:41:25.990075+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:41:25.990075+0000","last_undegraded":"2024-09-11T03:41:25.990075+0000","last_fullsized":"2024-09-11T03:41:25.990075+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.703203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.703203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:49.443821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"20'92","reported_seq":140,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.986624+0000","last_change":"2024-09-11T03:40:36.320692+0000","last_active":"2024-09-11T03:41:25.986624+0000","last_peered":"2024-09-11T03:41:25.986624+0000","last_clean":"2024-09-11T03:41:25.986624+0000","last_became_active":"2024-09-11T03:40:27.844057+0000","last_became_peered":"2024-09-11T03:40:27.844057+0000","last_unstale":"2024-09-11T03:41:25.986624+0000","last_undegraded":"2024-09-11T03:41:25.986624+0000","last_fullsized":"2024-09-11T03:41:25.986624+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:36.320646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:36.320646+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:02.386305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"260'254","reported_seq":1118,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.988311+0000","last_change":"2024-09-11T03:41:25.988542+0000","last_active":"2024-09-11T03:41:25.988311+0000","last_peered":"2024-09-11T03:41:25.988311+0000","last_clean":"2024-09-11T03:41:25.988311+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:41:25.988311+0000","last_undegraded":"2024-09-11T03:41:25.988311+0000","last_fullsized":"2024-09-11T03:41:25.988311+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'232","last_scrub_stamp":"2024-09-11T03:41:04.001653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:04.001653+0000","objects_scrubbed":10,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:22.372517+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00045154,"stat_sum":{"num_bytes":8773320,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":47,"num_read_kb":15678,"num_write":91,"num_write_kb":45694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3d","length":"1"},{"start":"4e","length":"1"},{"start":"52","length":"4"}]},{"pgid":"1.8","version":"20'92","reported_seq":216,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.986802+0000","last_change":"2024-09-11T03:40:38.319593+0000","last_active":"2024-09-11T03:41:25.986802+0000","last_peered":"2024-09-11T03:41:25.986802+0000","last_clean":"2024-09-11T03:41:25.986802+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:41:25.986802+0000","last_undegraded":"2024-09-11T03:41:25.986802+0000","last_fullsized":"2024-09-11T03:41:25.986802+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.319543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.319543+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:03.765057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"268'74","reported_seq":956,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-11T03:41:26.873459+0000","last_change":"2024-09-11T03:41:25.991878+0000","last_active":"2024-09-11T03:41:26.873459+0000","last_peered":"2024-09-11T03:41:26.873459+0000","last_clean":"2024-09-11T03:41:26.873459+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:41:26.873459+0000","last_undegraded":"2024-09-11T03:41:26.873459+0000","last_fullsized":"2024-09-11T03:41:26.873459+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'57","last_scrub_stamp":"2024-09-11T03:40:59.151899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:59.151899+0000","objects_scrubbed":1,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:27.513115+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00046993300000000002,"stat_sum":{"num_bytes":2974872,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":47,"num_read_kb":46721,"num_write":29,"num_write_kb":12797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"4e","length":"1"},{"start":"53","length":"3"}]},{"pgid":"1.9","version":"20'92","reported_seq":541,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.988885+0000","last_change":"2024-09-11T03:40:37.885659+0000","last_active":"2024-09-11T03:41:25.988885+0000","last_peered":"2024-09-11T03:41:25.988885+0000","last_clean":"2024-09-11T03:41:25.988885+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:41:25.988885+0000","last_undegraded":"2024-09-11T03:41:25.988885+0000","last_fullsized":"2024-09-11T03:41:25.988885+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:37.885602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:37.885602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:59.955114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"269'323","reported_seq":1248,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.996419+0000","last_change":"2024-09-11T03:41:25.996633+0000","last_active":"2024-09-11T03:41:25.996419+0000","last_peered":"2024-09-11T03:41:25.996419+0000","last_clean":"2024-09-11T03:41:25.996419+0000","last_became_active":"2024-09-11T03:38:29.888204+0000","last_became_peered":"2024-09-11T03:38:29.888204+0000","last_unstale":"2024-09-11T03:41:25.996419+0000","last_undegraded":"2024-09-11T03:41:25.996419+0000","last_fullsized":"2024-09-11T03:41:25.996419+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'311","last_scrub_stamp":"2024-09-11T03:41:20.086889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:20.086889+0000","objects_scrubbed":6,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:24.959661+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.006636996,"stat_sum":{"num_bytes":12218755,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":232,"num_read_kb":215551,"num_write":142,"num_write_kb":56367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3d","length":"1"},{"start":"4e","length":"1"},{"start":"52","length":"4"}]},{"pgid":"1.a","version":"20'92","reported_seq":447,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-11T03:41:22.857723+0000","last_change":"2024-09-11T03:40:39.331499+0000","last_active":"2024-09-11T03:41:22.857723+0000","last_peered":"2024-09-11T03:41:22.857723+0000","last_clean":"2024-09-11T03:41:22.857723+0000","last_became_active":"2024-09-11T03:40:37.849988+0000","last_became_peered":"2024-09-11T03:40:37.849988+0000","last_unstale":"2024-09-11T03:41:22.857723+0000","last_undegraded":"2024-09-11T03:41:22.857723+0000","last_fullsized":"2024-09-11T03:41:22.857723+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.331453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.331453+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:40.696793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"269'192","reported_seq":1073,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.993478+0000","last_change":"2024-09-11T03:41:25.993688+0000","last_active":"2024-09-11T03:41:25.993478+0000","last_peered":"2024-09-11T03:41:25.993478+0000","last_clean":"2024-09-11T03:41:25.993478+0000","last_became_active":"2024-09-11T03:38:30.162819+0000","last_became_peered":"2024-09-11T03:38:30.162819+0000","last_unstale":"2024-09-11T03:41:25.993478+0000","last_undegraded":"2024-09-11T03:41:25.993478+0000","last_fullsized":"2024-09-11T03:41:25.993478+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'190","last_scrub_stamp":"2024-09-11T03:41:21.007493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:21.007493+0000","objects_scrubbed":3,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:22.175098+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0046941559999999997,"stat_sum":{"num_bytes":5495955,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":94858,"num_write":84,"num_write_kb":26743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3d","length":"1"},{"start":"4e","length":"1"},{"start":"52","length":"4"}]},{"pgid":"1.b","version":"20'92","reported_seq":158,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-11T03:41:26.873407+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:41:26.873407+0000","last_peered":"2024-09-11T03:41:26.873407+0000","last_clean":"2024-09-11T03:41:26.873407+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:41:26.873407+0000","last_undegraded":"2024-09-11T03:41:26.873407+0000","last_fullsized":"2024-09-11T03:41:26.873407+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:52.822589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"265'274","reported_seq":1203,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-11T03:41:21.858619+0000","last_change":"2024-09-11T03:41:21.858808+0000","last_active":"2024-09-11T03:41:21.858619+0000","last_peered":"2024-09-11T03:41:21.858619+0000","last_clean":"2024-09-11T03:41:21.858619+0000","last_became_active":"2024-09-11T03:40:08.836627+0000","last_became_peered":"2024-09-11T03:40:08.836627+0000","last_unstale":"2024-09-11T03:41:21.858619+0000","last_undegraded":"2024-09-11T03:41:21.858619+0000","last_fullsized":"2024-09-11T03:41:21.858619+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'242","last_scrub_stamp":"2024-09-11T03:40:59.291412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:59.291412+0000","objects_scrubbed":9,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:05.048172+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0028502319999999999,"stat_sum":{"num_bytes":16050675,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":129,"num_read_kb":88611,"num_write":129,"num_write_kb":50816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":6433401,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3d","length":"1"},{"start":"52","length":"2"}]},{"pgid":"1.4","version":"20'92","reported_seq":316,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-11T03:41:26.873673+0000","last_change":"2024-09-11T03:40:35.272628+0000","last_active":"2024-09-11T03:41:26.873673+0000","last_peered":"2024-09-11T03:41:26.873673+0000","last_clean":"2024-09-11T03:41:26.873673+0000","last_became_active":"2024-09-11T03:40:25.858281+0000","last_became_peered":"2024-09-11T03:40:25.858281+0000","last_unstale":"2024-09-11T03:41:26.873673+0000","last_undegraded":"2024-09-11T03:41:26.873673+0000","last_fullsized":"2024-09-11T03:41:26.873673+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.272563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.272563+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:53.042850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":1041,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.996040+0000","last_change":"2024-09-11T03:41:25.996368+0000","last_active":"2024-09-11T03:41:25.996040+0000","last_peered":"2024-09-11T03:41:25.996040+0000","last_clean":"2024-09-11T03:41:25.996040+0000","last_became_active":"2024-09-11T03:38:29.887945+0000","last_became_peered":"2024-09-11T03:38:29.887945+0000","last_unstale":"2024-09-11T03:41:25.996040+0000","last_undegraded":"2024-09-11T03:41:25.996040+0000","last_fullsized":"2024-09-11T03:41:25.996040+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'103","last_scrub_stamp":"2024-09-11T03:40:17.171458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:17.171458+0000","objects_scrubbed":4,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:31.865890+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0066722869999999998,"stat_sum":{"num_bytes":2958065,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":62,"num_read_kb":58350,"num_write":62,"num_write_kb":30670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3d","length":"1"},{"start":"4e","length":"1"},{"start":"52","length":"4"}]},{"pgid":"1.5","version":"20'92","reported_seq":212,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-11T03:41:21.855759+0000","last_change":"2024-09-11T03:40:38.224278+0000","last_active":"2024-09-11T03:41:21.855759+0000","last_peered":"2024-09-11T03:41:21.855759+0000","last_clean":"2024-09-11T03:41:21.855759+0000","last_became_active":"2024-09-11T03:40:37.851080+0000","last_became_peered":"2024-09-11T03:40:37.851080+0000","last_unstale":"2024-09-11T03:41:21.855759+0000","last_undegraded":"2024-09-11T03:41:21.855759+0000","last_fullsized":"2024-09-11T03:41:21.855759+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.224231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.224231+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:07.626847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"269'266","reported_seq":1200,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.995240+0000","last_change":"2024-09-11T03:41:25.995432+0000","last_active":"2024-09-11T03:41:25.995240+0000","last_peered":"2024-09-11T03:41:25.995240+0000","last_clean":"2024-09-11T03:41:25.995240+0000","last_became_active":"2024-09-11T03:38:30.163080+0000","last_became_peered":"2024-09-11T03:38:30.163080+0000","last_unstale":"2024-09-11T03:41:25.995240+0000","last_undegraded":"2024-09-11T03:41:25.995240+0000","last_fullsized":"2024-09-11T03:41:25.995240+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'209","last_scrub_stamp":"2024-09-11T03:40:45.175807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:45.175807+0000","objects_scrubbed":8,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:08.264386+0000","scrub_duration":6,"objects_trimmed":2,"snaptrim_duration":0.0072955499999999996,"stat_sum":{"num_bytes":16571801,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":204,"num_read_kb":194405,"num_write":123,"num_write_kb":43302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3d","length":"1"},{"start":"4e","length":"1"},{"start":"52","length":"4"}]},{"pgid":"1.6","version":"20'92","reported_seq":450,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-11T03:41:22.857929+0000","last_change":"2024-09-11T03:40:38.367213+0000","last_active":"2024-09-11T03:41:22.857929+0000","last_peered":"2024-09-11T03:41:22.857929+0000","last_clean":"2024-09-11T03:41:22.857929+0000","last_became_active":"2024-09-11T03:40:37.848794+0000","last_became_peered":"2024-09-11T03:40:37.848794+0000","last_unstale":"2024-09-11T03:41:22.857929+0000","last_undegraded":"2024-09-11T03:41:22.857929+0000","last_fullsized":"2024-09-11T03:41:22.857929+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.367170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.367170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:05.132471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"269'159","reported_seq":1069,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:26.001816+0000","last_change":"2024-09-11T03:41:26.002070+0000","last_active":"2024-09-11T03:41:26.001816+0000","last_peered":"2024-09-11T03:41:26.001816+0000","last_clean":"2024-09-11T03:41:26.001816+0000","last_became_active":"2024-09-11T03:40:05.966715+0000","last_became_peered":"2024-09-11T03:40:05.966715+0000","last_unstale":"2024-09-11T03:41:26.001816+0000","last_undegraded":"2024-09-11T03:41:26.001816+0000","last_fullsized":"2024-09-11T03:41:26.001816+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'130","last_scrub_stamp":"2024-09-11T03:41:03.349362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:03.349362+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:20.299923+0000","scrub_duration":2,"objects_trimmed":2,"snaptrim_duration":0.011651138,"stat_sum":{"num_bytes":6252270,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":53,"num_read_kb":31644,"num_write":77,"num_write_kb":25086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":7190868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3d","length":"1"},{"start":"4e","length":"1"},{"start":"52","length":"4"}]},{"pgid":"1.7","version":"20'92","reported_seq":69,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.986876+0000","last_change":"2024-09-11T03:41:02.232514+0000","last_active":"2024-09-11T03:41:25.986876+0000","last_peered":"2024-09-11T03:41:25.986876+0000","last_clean":"2024-09-11T03:41:25.986876+0000","last_became_active":"2024-09-11T03:41:01.853968+0000","last_became_peered":"2024-09-11T03:41:01.853968+0000","last_unstale":"2024-09-11T03:41:25.986876+0000","last_undegraded":"2024-09-11T03:41:25.986876+0000","last_fullsized":"2024-09-11T03:41:25.986876+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:02.232437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:02.232437+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:13.312679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"269'123","reported_seq":1019,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.989909+0000","last_change":"2024-09-11T03:41:25.990320+0000","last_active":"2024-09-11T03:41:25.989909+0000","last_peered":"2024-09-11T03:41:25.989909+0000","last_clean":"2024-09-11T03:41:25.989909+0000","last_became_active":"2024-09-11T03:39:26.115670+0000","last_became_peered":"2024-09-11T03:39:26.115670+0000","last_unstale":"2024-09-11T03:41:25.989909+0000","last_undegraded":"2024-09-11T03:41:25.989909+0000","last_fullsized":"2024-09-11T03:41:25.989909+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'111","last_scrub_stamp":"2024-09-11T03:41:04.281773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:04.281773+0000","objects_scrubbed":5,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:06.737370+0000","scrub_duration":2,"objects_trimmed":2,"snaptrim_duration":0.0035671040000000002,"stat_sum":{"num_bytes":2571471,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":8,"num_write":48,"num_write_kb":26445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3d","length":"1"},{"start":"4e","length":"1"},{"start":"52","length":"4"}]},{"pgid":"1.1","version":"20'92","reported_seq":436,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-11T03:41:22.857704+0000","last_change":"2024-09-11T03:40:35.325550+0000","last_active":"2024-09-11T03:41:22.857704+0000","last_peered":"2024-09-11T03:41:22.857704+0000","last_clean":"2024-09-11T03:41:22.857704+0000","last_became_active":"2024-09-11T03:40:25.854269+0000","last_became_peered":"2024-09-11T03:40:25.854269+0000","last_unstale":"2024-09-11T03:41:22.857704+0000","last_undegraded":"2024-09-11T03:41:22.857704+0000","last_fullsized":"2024-09-11T03:41:22.857704+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.325493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.325493+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:00.762795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"265'99","reported_seq":956,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-11T03:41:21.858153+0000","last_change":"2024-09-11T03:41:21.858494+0000","last_active":"2024-09-11T03:41:21.858153+0000","last_peered":"2024-09-11T03:41:21.858153+0000","last_clean":"2024-09-11T03:41:21.858153+0000","last_became_active":"2024-09-11T03:38:30.162952+0000","last_became_peered":"2024-09-11T03:38:30.162952+0000","last_unstale":"2024-09-11T03:41:21.858153+0000","last_undegraded":"2024-09-11T03:41:21.858153+0000","last_fullsized":"2024-09-11T03:41:21.858153+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'87","last_scrub_stamp":"2024-09-11T03:40:58.304220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:58.304220+0000","objects_scrubbed":3,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:58.843029+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002415206,"stat_sum":{"num_bytes":5075780,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":19746,"num_write":34,"num_write_kb":20545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3d","length":"1"},{"start":"52","length":"2"}]},{"pgid":"1.2","version":"20'92","reported_seq":399,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.990297+0000","last_change":"2024-09-11T03:41:02.760300+0000","last_active":"2024-09-11T03:41:25.990297+0000","last_peered":"2024-09-11T03:41:25.990297+0000","last_clean":"2024-09-11T03:41:25.990297+0000","last_became_active":"2024-09-11T03:41:01.851367+0000","last_became_peered":"2024-09-11T03:41:01.851367+0000","last_unstale":"2024-09-11T03:41:25.990297+0000","last_undegraded":"2024-09-11T03:41:25.990297+0000","last_fullsized":"2024-09-11T03:41:25.990297+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:02.760253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:02.760253+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:31.006129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"269'65","reported_seq":961,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.991997+0000","last_change":"2024-09-11T03:41:25.992159+0000","last_active":"2024-09-11T03:41:25.991997+0000","last_peered":"2024-09-11T03:41:25.991997+0000","last_clean":"2024-09-11T03:41:25.991997+0000","last_became_active":"2024-09-11T03:40:05.966325+0000","last_became_peered":"2024-09-11T03:40:05.966325+0000","last_unstale":"2024-09-11T03:41:25.991997+0000","last_undegraded":"2024-09-11T03:41:25.991997+0000","last_fullsized":"2024-09-11T03:41:25.991997+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'33","last_scrub_stamp":"2024-09-11T03:40:58.014124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:58.014124+0000","objects_scrubbed":2,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:03.679414+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0029345819999999998,"stat_sum":{"num_bytes":6370086,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":36600,"num_write":28,"num_write_kb":11573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3d","length":"1"},{"start":"4e","length":"1"},{"start":"52","length":"4"}]},{"pgid":"1.3","version":"20'92","reported_seq":218,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.989050+0000","last_change":"2024-09-11T03:40:36.259209+0000","last_active":"2024-09-11T03:41:25.989050+0000","last_peered":"2024-09-11T03:41:25.989050+0000","last_clean":"2024-09-11T03:41:25.989050+0000","last_became_active":"2024-09-11T03:40:25.856407+0000","last_became_peered":"2024-09-11T03:40:25.856407+0000","last_unstale":"2024-09-11T03:41:25.989050+0000","last_undegraded":"2024-09-11T03:41:25.989050+0000","last_fullsized":"2024-09-11T03:41:25.989050+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:36.259163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:36.259163+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:55.651280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"269'103","reported_seq":986,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.990647+0000","last_change":"2024-09-11T03:41:25.990899+0000","last_active":"2024-09-11T03:41:25.990647+0000","last_peered":"2024-09-11T03:41:25.990647+0000","last_clean":"2024-09-11T03:41:25.990647+0000","last_became_active":"2024-09-11T03:38:29.889091+0000","last_became_peered":"2024-09-11T03:38:29.889091+0000","last_unstale":"2024-09-11T03:41:25.990647+0000","last_undegraded":"2024-09-11T03:41:25.990647+0000","last_fullsized":"2024-09-11T03:41:25.990647+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'78","last_scrub_stamp":"2024-09-11T03:40:45.292704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:45.292704+0000","objects_scrubbed":2,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:51.132172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0040186789999999998,"stat_sum":{"num_bytes":5411887,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":48,"num_read_kb":34593,"num_write":42,"num_write_kb":16498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3d","length":"1"},{"start":"4e","length":"1"},{"start":"52","length":"4"}]},{"pgid":"1.0","version":"20'92","reported_seq":538,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.989250+0000","last_change":"2024-09-11T03:40:36.901245+0000","last_active":"2024-09-11T03:41:25.989250+0000","last_peered":"2024-09-11T03:41:25.989250+0000","last_clean":"2024-09-11T03:41:25.989250+0000","last_became_active":"2024-09-11T03:40:25.854624+0000","last_became_peered":"2024-09-11T03:40:25.854624+0000","last_unstale":"2024-09-11T03:41:25.989250+0000","last_undegraded":"2024-09-11T03:41:25.989250+0000","last_fullsized":"2024-09-11T03:41:25.989250+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:36.901166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:36.901166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:43.207763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"20'92","reported_seq":142,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.986865+0000","last_change":"2024-09-11T03:40:35.282811+0000","last_active":"2024-09-11T03:41:25.986865+0000","last_peered":"2024-09-11T03:41:25.986865+0000","last_clean":"2024-09-11T03:41:25.986865+0000","last_became_active":"2024-09-11T03:40:27.844262+0000","last_became_peered":"2024-09-11T03:40:27.844262+0000","last_unstale":"2024-09-11T03:41:25.986865+0000","last_undegraded":"2024-09-11T03:41:25.986865+0000","last_fullsized":"2024-09-11T03:41:25.986865+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.282762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.282762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:51.256113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"269'217","reported_seq":1118,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-11T03:41:26.873798+0000","last_change":"2024-09-11T03:41:25.999529+0000","last_active":"2024-09-11T03:41:26.873798+0000","last_peered":"2024-09-11T03:41:26.873798+0000","last_clean":"2024-09-11T03:41:26.873798+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:41:26.873798+0000","last_undegraded":"2024-09-11T03:41:26.873798+0000","last_fullsized":"2024-09-11T03:41:26.873798+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'172","last_scrub_stamp":"2024-09-11T03:41:04.106166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:04.106166+0000","objects_scrubbed":5,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:30.498922+0000","scrub_duration":3,"objects_trimmed":2,"snaptrim_duration":0.0077663810000000002,"stat_sum":{"num_bytes":12600865,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":128,"num_read_kb":98768,"num_write":105,"num_write_kb":45578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"4e","length":"1"},{"start":"53","length":"3"}]},{"pgid":"1.e","version":"20'92","reported_seq":397,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-11T03:41:24.862229+0000","last_change":"2024-09-11T03:40:38.199696+0000","last_active":"2024-09-11T03:41:24.862229+0000","last_peered":"2024-09-11T03:41:24.862229+0000","last_clean":"2024-09-11T03:41:24.862229+0000","last_became_active":"2024-09-11T03:40:37.850623+0000","last_became_peered":"2024-09-11T03:40:37.850623+0000","last_unstale":"2024-09-11T03:41:24.862229+0000","last_undegraded":"2024-09-11T03:41:24.862229+0000","last_fullsized":"2024-09-11T03:41:24.862229+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.199636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.199636+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:45.493648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"266'291","reported_seq":1157,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-11T03:41:22.859394+0000","last_change":"2024-09-11T03:41:22.859540+0000","last_active":"2024-09-11T03:41:22.859394+0000","last_peered":"2024-09-11T03:41:22.859394+0000","last_clean":"2024-09-11T03:41:22.859394+0000","last_became_active":"2024-09-11T03:38:30.031497+0000","last_became_peered":"2024-09-11T03:38:30.031497+0000","last_unstale":"2024-09-11T03:41:22.859394+0000","last_undegraded":"2024-09-11T03:41:22.859394+0000","last_fullsized":"2024-09-11T03:41:22.859394+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'249","last_scrub_stamp":"2024-09-11T03:41:08.270845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:08.270845+0000","objects_scrubbed":10,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:12.948109+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001439517,"stat_sum":{"num_bytes":12377573,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":140,"num_read_kb":105480,"num_write":103,"num_write_kb":38900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3d","length":"1"},{"start":"52","length":"2"},{"start":"55","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":156,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.989195+0000","last_change":"2024-09-11T03:40:38.901246+0000","last_active":"2024-09-11T03:41:25.989195+0000","last_peered":"2024-09-11T03:41:25.989195+0000","last_clean":"2024-09-11T03:41:25.989195+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:41:25.989195+0000","last_undegraded":"2024-09-11T03:41:25.989195+0000","last_fullsized":"2024-09-11T03:41:25.989195+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.901175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.901175+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:48.712523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"265'283","reported_seq":1135,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-11T03:41:21.865986+0000","last_change":"2024-09-11T03:41:21.858919+0000","last_active":"2024-09-11T03:41:21.865986+0000","last_peered":"2024-09-11T03:41:21.865986+0000","last_clean":"2024-09-11T03:41:21.865986+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:41:21.865986+0000","last_undegraded":"2024-09-11T03:41:21.865986+0000","last_fullsized":"2024-09-11T03:41:21.865986+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'259","last_scrub_stamp":"2024-09-11T03:41:09.347904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:09.347904+0000","objects_scrubbed":11,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:38.266938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024509390000000001,"stat_sum":{"num_bytes":18319861,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":115,"num_read_kb":83461,"num_write":99,"num_write_kb":45818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3d","length":"1"},{"start":"52","length":"2"}]},{"pgid":"2.f","version":"260'136","reported_seq":1043,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-11T03:41:25.988639+0000","last_change":"2024-09-11T03:41:25.988867+0000","last_active":"2024-09-11T03:41:25.988639+0000","last_peered":"2024-09-11T03:41:25.988639+0000","last_clean":"2024-09-11T03:41:25.988639+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:41:25.988639+0000","last_undegraded":"2024-09-11T03:41:25.988639+0000","last_fullsized":"2024-09-11T03:41:25.988639+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'136","last_scrub_stamp":"2024-09-11T03:41:20.167983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:20.167983+0000","objects_scrubbed":2,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:26.853686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00069800099999999996,"stat_sum":{"num_bytes":4705993,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":82051,"num_write":58,"num_write_kb":21882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3d","length":"1"},{"start":"4e","length":"1"},{"start":"52","length":"4"}]},{"pgid":"1.c","version":"20'92","reported_seq":158,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-11T03:41:22.855481+0000","last_change":"2024-09-11T03:40:38.353992+0000","last_active":"2024-09-11T03:41:22.855481+0000","last_peered":"2024-09-11T03:41:22.855481+0000","last_clean":"2024-09-11T03:41:22.855481+0000","last_became_active":"2024-09-11T03:40:37.847449+0000","last_became_peered":"2024-09-11T03:40:37.847449+0000","last_unstale":"2024-09-11T03:41:22.855481+0000","last_undegraded":"2024-09-11T03:41:22.855481+0000","last_fullsized":"2024-09-11T03:41:22.855481+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.353895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.353895+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:44.612649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":138729229,"num_objects":59,"num_object_clones":20,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":2,"num_read":1494,"num_read_kb":1206525,"num_write":1254,"num_write_kb":518714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":13624269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274108416,"data_stored":309601155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3030,"ondisk_log_size":3030,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285619,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56448,"kb_used_data":25252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93688704,"statfs":{"total":95995035648,"available":95937232896,"internally_reserved":0,"allocated":25858048,"data_stored":27685448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":326417514551,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743484,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67120,"kb_used_data":35924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93678032,"statfs":{"total":95995035648,"available":95926304768,"internally_reserved":0,"allocated":36786176,"data_stored":38519584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037828,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60280,"kb_used_data":24932,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93684872,"statfs":{"total":95995035648,"available":95933308928,"internally_reserved":0,"allocated":25530368,"data_stored":24114204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299463,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62008,"kb_used_data":26652,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93683144,"statfs":{"total":95995035648,"available":95931539456,"internally_reserved":0,"allocated":27291648,"data_stored":30477780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":193273528396,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757330,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":39460,"kb_used_data":8268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93705692,"statfs":{"total":95995035648,"available":95954628608,"internally_reserved":0,"allocated":8466432,"data_stored":10777218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986262,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":69544,"kb_used_data":38348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93675608,"statfs":{"total":95995035648,"available":95923822592,"internally_reserved":0,"allocated":39268352,"data_stored":46628176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149787,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82596,"kb_used_data":47240,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93662556,"statfs":{"total":95995035648,"available":95910457344,"internally_reserved":0,"allocated":48373760,"data_stored":50970794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411424,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64276,"kb_used_data":33080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93680876,"statfs":{"total":95995035648,"available":95929217024,"internally_reserved":0,"allocated":33873920,"data_stored":37953888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56708,"kb_used_data":25520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93688444,"statfs":{"total":95995035648,"available":95936966656,"internally_reserved":0,"allocated":26132480,"data_stored":26678336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738476,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72320,"kb_used_data":39040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93672832,"statfs":{"total":95995035648,"available":95920979968,"internally_reserved":0,"allocated":39976960,"data_stored":42133864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28266496,"data_stored":30014152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22966272,"data_stored":24762152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30552064,"data_stored":35869168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":44916736,"data_stored":48751446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":35082240,"data_stored":44353054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5095424,"data_stored":8659786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4079616,"data_stored":4066399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":23920640,"data_stored":28360348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":22364160,"data_stored":22198020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":33329152,"data_stored":36300236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2387968,"data_stored":2367002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":21147648,"data_stored":23899392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:41:28.913 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-11T03:41:28.913 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 26 2024-09-11T03:41:28.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:28.935 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:28.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:28.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:28.997 INFO:tasks.rados.rados.2.smithi175.stdout:866: done (1 left) 2024-09-11T03:41:28.998 INFO:tasks.rados.rados.2.smithi175.stdout:867: done (0 left) 2024-09-11T03:41:28.998 INFO:tasks.rados.rados.2.smithi175.stdout:868: write oid 22 current snap is 89 2024-09-11T03:41:28.998 INFO:tasks.rados.rados.2.smithi175.stdout:868: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2024-09-11T03:41:29.005 INFO:tasks.rados.rados.2.smithi175.stdout:868: writing smithi17557870-22 from 628661 to 1420914 tid 1 2024-09-11T03:41:29.011 INFO:tasks.rados.rados.2.smithi175.stdout:868: writing smithi17557870-22 from 2184625 to 2606403 tid 2 2024-09-11T03:41:29.016 INFO:tasks.rados.rados.2.smithi175.stdout:868: writing smithi17557870-22 from 3403509 to 3428661 tid 3 2024-09-11T03:41:29.016 INFO:tasks.rados.rados.2.smithi175.stdout:869: read oid 5 snap -1 2024-09-11T03:41:29.016 INFO:tasks.rados.rados.2.smithi175.stdout:869: expect (ObjNum 302 snap 82 seq_num 302) 2024-09-11T03:41:29.016 INFO:tasks.rados.rados.2.smithi175.stdout:870: copy_from oid 45 from oid 7 current snap is 89 2024-09-11T03:41:29.016 INFO:tasks.rados.rados.2.smithi175.stdout:868: finishing write tid 1 to smithi17557870-22 2024-09-11T03:41:29.016 INFO:tasks.rados.rados.2.smithi175.stdout:868: finishing write tid 2 to smithi17557870-22 2024-09-11T03:41:29.016 INFO:tasks.rados.rados.2.smithi175.stdout:871: snap_create 2024-09-11T03:41:29.017 INFO:tasks.rados.rados.2.smithi175.stdout:868: finishing write tid 3 to smithi17557870-22 2024-09-11T03:41:29.018 INFO:tasks.rados.rados.2.smithi175.stdout:868: finishing write tid 4 to smithi17557870-22 2024-09-11T03:41:29.019 INFO:tasks.rados.rados.2.smithi175.stdout:868: finishing write tid 5 to smithi17557870-22 2024-09-11T03:41:29.019 INFO:tasks.rados.rados.2.smithi175.stdout:868: oid 22 updating version 0 to 329 2024-09-11T03:41:29.019 INFO:tasks.rados.rados.2.smithi175.stdout:868: oid 22 version 329 is already newer than 328 2024-09-11T03:41:29.019 INFO:tasks.rados.rados.2.smithi175.stdout:868: oid 22 updating version 329 to 330 2024-09-11T03:41:29.019 INFO:tasks.rados.rados.2.smithi175.stdout:868: oid 22 updating version 330 to 331 2024-09-11T03:41:29.019 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 331 (ObjNum 323 snap 89 seq_num 323) dirty exists 2024-09-11T03:41:29.019 INFO:tasks.rados.rados.2.smithi175.stdout:868: left oid 22 (ObjNum 323 snap 89 seq_num 323) 2024-09-11T03:41:29.029 INFO:tasks.rados.rados.2.smithi175.stdout:869: expect (ObjNum 302 snap 82 seq_num 302) 2024-09-11T03:41:29.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:29.049 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:41:29.049 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:41:29.058 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:29.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:29.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:29.090 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:41:29.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:28 smithi084 ceph-mon[27254]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:41:29.160 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:29.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:29.182 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:29.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:29.202 INFO:tasks.rados.rados.2.smithi175.stdout:870: finishing copy_from racing read to smithi17557870-45 2024-09-11T03:41:29.203 INFO:tasks.rados.rados.2.smithi175.stdout:870: finishing copy_from to smithi17557870-45 2024-09-11T03:41:29.203 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 163 (ObjNum 295 snap 79 seq_num 295) dirty exists 2024-09-11T03:41:29.282 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:29.304 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:29.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:29.347 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:28 smithi121 ceph-mon[30729]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:41:29.405 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:29.427 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:29.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:29.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:28 smithi175 ceph-mon[30894]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:41:29.516 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:29.529 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:29.550 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:29.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:29.551 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:29.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T03:41:29.651 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:29.673 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:29.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:29.674 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:29.775 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:29.797 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:29.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:29.798 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:29.876 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:41:29.899 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:29.921 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:29.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:29.922 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:30.007 INFO:tasks.rados.rados.2.smithi175.stdout:868: done (3 left) 2024-09-11T03:41:30.007 INFO:tasks.rados.rados.2.smithi175.stdout:869: done (2 left) 2024-09-11T03:41:30.007 INFO:tasks.rados.rados.2.smithi175.stdout:870: done (1 left) 2024-09-11T03:41:30.007 INFO:tasks.rados.rados.2.smithi175.stdout:871: done (0 left) 2024-09-11T03:41:30.007 INFO:tasks.rados.rados.2.smithi175.stdout:872: read oid 47 snap -1 2024-09-11T03:41:30.007 INFO:tasks.rados.rados.2.smithi175.stdout:872: expect deleted 2024-09-11T03:41:30.007 INFO:tasks.rados.rados.2.smithi175.stdout:873: write oid 9 current snap is 90 2024-09-11T03:41:30.008 INFO:tasks.rados.rados.2.smithi175.stdout:873: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2024-09-11T03:41:30.013 INFO:tasks.rados.rados.2.smithi175.stdout:873: writing smithi17557870-9 from 645468 to 1112970 tid 1 2024-09-11T03:41:30.019 INFO:tasks.rados.rados.2.smithi175.stdout:873: writing smithi17557870-9 from 1726754 to 2292190 tid 2 2024-09-11T03:41:30.023 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:30.026 INFO:tasks.rados.rados.2.smithi175.stdout:873: writing smithi17557870-9 from 2798226 to 3395064 tid 3 2024-09-11T03:41:30.026 INFO:tasks.rados.rados.2.smithi175.stdout:874: rollback oid 26 current snap is 90 2024-09-11T03:41:30.026 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 26 to 84 2024-09-11T03:41:30.026 INFO:tasks.rados.rados.2.smithi175.stdout:873: finishing write tid 1 to smithi17557870-9 2024-09-11T03:41:30.026 INFO:tasks.rados.rados.2.smithi175.stdout:873: finishing write tid 2 to smithi17557870-9 2024-09-11T03:41:30.026 INFO:tasks.rados.rados.2.smithi175.stdout:872: done (2 left) 2024-09-11T03:41:30.026 INFO:tasks.rados.rados.2.smithi175.stdout:875: snap_remove snap 79 2024-09-11T03:41:30.043 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:30.044 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:30.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:30.046 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:30.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:30.163 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:29 smithi084 ceph-mon[27254]: pgmap v562: 47 pgs: 47 active+clean; 136 MiB data, 613 MiB used, 893 GiB / 894 GiB avail; 6.8 MiB/s rd, 3.9 MiB/s wr, 17 op/s 2024-09-11T03:41:30.164 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:29 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:41:30.164 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:29 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:41:30.164 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:29 smithi084 ceph-mon[27254]: osdmap e272: 12 total, 12 up, 10 in 2024-09-11T03:41:30.169 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:30.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:30.271 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:30.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:29 smithi121 ceph-mon[30729]: pgmap v562: 47 pgs: 47 active+clean; 136 MiB data, 613 MiB used, 893 GiB / 894 GiB avail; 6.8 MiB/s rd, 3.9 MiB/s wr, 17 op/s 2024-09-11T03:41:30.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:29 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:41:30.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:29 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:41:30.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:29 smithi121 ceph-mon[30729]: osdmap e272: 12 total, 12 up, 10 in 2024-09-11T03:41:30.294 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:30.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:30.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:30.396 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:30.409 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:41:30.409 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:41:30.434 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:30.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:30.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:29 smithi175 ceph-mon[30894]: pgmap v562: 47 pgs: 47 active+clean; 136 MiB data, 613 MiB used, 893 GiB / 894 GiB avail; 6.8 MiB/s rd, 3.9 MiB/s wr, 17 op/s 2024-09-11T03:41:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:29 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:41:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:29 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:41:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:29 smithi175 ceph-mon[30894]: osdmap e272: 12 total, 12 up, 10 in 2024-09-11T03:41:30.536 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:30.562 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:30.563 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:30.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:30.686 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:30.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:30.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:30.788 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:30.814 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:30.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:30.815 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:30.916 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:30.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:30.939 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:30.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:31.007 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 2 pg_num to 26 2024-09-11T03:41:31.008 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:41:31.012 INFO:tasks.rados.rados.2.smithi175.stdout:875: done (2 left) 2024-09-11T03:41:31.012 INFO:tasks.rados.rados.2.smithi175.stdout:876: write (excl) oid 29 current snap is 90 2024-09-11T03:41:31.012 INFO:tasks.rados.rados.2.smithi175.stdout:876: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2024-09-11T03:41:31.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T03:41:31.019 INFO:tasks.rados.rados.2.smithi175.stdout:876: writing smithi17557870-29 from 662275 to 1205026 tid 1 2024-09-11T03:41:31.024 INFO:tasks.rados.rados.2.smithi175.stdout:876: writing smithi17557870-29 from 1785236 to 2210683 tid 2 2024-09-11T03:41:31.033 INFO:tasks.rados.rados.2.smithi175.stdout:876: writing smithi17557870-29 from 2942002 to 3462275 tid 3 2024-09-11T03:41:31.033 INFO:tasks.rados.rados.2.smithi175.stdout:877: read oid 35 snap -1 2024-09-11T03:41:31.033 INFO:tasks.rados.rados.2.smithi175.stdout:877: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:41:31.033 INFO:tasks.rados.rados.2.smithi175.stdout:878: copy_from oid 40 from oid 48 current snap is 90 2024-09-11T03:41:31.033 INFO:tasks.rados.rados.2.smithi175.stdout:874: finishing rollback tid 1 to smithi17557870-26 2024-09-11T03:41:31.033 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 0 (ObjNum 33 snap 0 seq_num 1713067256) dirty dne 2024-09-11T03:41:31.033 INFO:tasks.rados.rados.2.smithi175.stdout:873: finishing write tid 3 to smithi17557870-9 2024-09-11T03:41:31.033 INFO:tasks.rados.rados.2.smithi175.stdout:873: finishing write tid 4 to smithi17557870-9 2024-09-11T03:41:31.033 INFO:tasks.rados.rados.2.smithi175.stdout:873: finishing write tid 5 to smithi17557870-9 2024-09-11T03:41:31.034 INFO:tasks.rados.rados.2.smithi175.stdout:873: oid 9 updating version 0 to 303 2024-09-11T03:41:31.034 INFO:tasks.rados.rados.2.smithi175.stdout:873: oid 9 updating version 303 to 304 2024-09-11T03:41:31.034 INFO:tasks.rados.rados.2.smithi175.stdout:873: oid 9 updating version 304 to 306 2024-09-11T03:41:31.034 INFO:tasks.rados.rados.2.smithi175.stdout:873: oid 9 version 306 is already newer than 305 2024-09-11T03:41:31.034 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 306 (ObjNum 324 snap 90 seq_num 324) dirty exists 2024-09-11T03:41:31.034 INFO:tasks.rados.rados.2.smithi175.stdout:873: left oid 9 (ObjNum 324 snap 90 seq_num 324) 2024-09-11T03:41:31.034 INFO:tasks.rados.rados.2.smithi175.stdout:876: finishing write tid 1 to smithi17557870-29 2024-09-11T03:41:31.034 INFO:tasks.rados.rados.2.smithi175.stdout:876: finishing write tid 2 to smithi17557870-29 2024-09-11T03:41:31.034 INFO:tasks.rados.rados.2.smithi175.stdout:873: done (4 left) 2024-09-11T03:41:31.034 INFO:tasks.rados.rados.2.smithi175.stdout:874: done (3 left) 2024-09-11T03:41:31.034 INFO:tasks.rados.rados.2.smithi175.stdout:879: read oid 46 snap -1 2024-09-11T03:41:31.034 INFO:tasks.rados.rados.2.smithi175.stdout:879: expect (ObjNum 296 snap 79 seq_num 296) 2024-09-11T03:41:31.034 INFO:tasks.rados.rados.2.smithi175.stdout:880: rollback oid 11 current snap is 90 2024-09-11T03:41:31.034 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 11 to 88 2024-09-11T03:41:31.035 INFO:tasks.rados.rados.2.smithi175.stdout:881: copy_from oid 26 from oid 15 current snap is 90 2024-09-11T03:41:31.035 INFO:tasks.rados.rados.2.smithi175.stdout:882: snap_create 2024-09-11T03:41:31.035 INFO:tasks.rados.rados.2.smithi175.stdout:878: finishing copy_from to smithi17557870-40 2024-09-11T03:41:31.036 INFO:tasks.rados.rados.2.smithi175.stdout:878: got expected ENOENT (src dne) 2024-09-11T03:41:31.036 INFO:tasks.rados.rados.2.smithi175.stdout:878: finishing copy_from racing read to smithi17557870-40 2024-09-11T03:41:31.036 INFO:tasks.rados.rados.2.smithi175.stdout:876: finishing write tid 3 to smithi17557870-29 2024-09-11T03:41:31.036 INFO:tasks.rados.rados.2.smithi175.stdout:876: finishing write tid 4 to smithi17557870-29 2024-09-11T03:41:31.036 INFO:tasks.rados.rados.2.smithi175.stdout:876: finishing write tid 5 to smithi17557870-29 2024-09-11T03:41:31.036 INFO:tasks.rados.rados.2.smithi175.stdout:876: oid 29 updating version 0 to 273 2024-09-11T03:41:31.036 INFO:tasks.rados.rados.2.smithi175.stdout:876: oid 29 version 273 is already newer than 270 2024-09-11T03:41:31.036 INFO:tasks.rados.rados.2.smithi175.stdout:876: oid 29 version 273 is already newer than 272 2024-09-11T03:41:31.036 INFO:tasks.rados.rados.2.smithi175.stdout:876: oid 29 version 273 is already newer than 271 2024-09-11T03:41:31.036 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 273 (ObjNum 325 snap 90 seq_num 325) dirty exists 2024-09-11T03:41:31.036 INFO:tasks.rados.rados.2.smithi175.stdout:876: left oid 29 (ObjNum 325 snap 90 seq_num 325) 2024-09-11T03:41:31.036 INFO:tasks.rados.rados.2.smithi175.stdout:880: finishing rollback tid 0 to smithi17557870-11 2024-09-11T03:41:31.037 INFO:tasks.rados.rados.2.smithi175.stdout:880: finishing rollback tid 1 to smithi17557870-11 2024-09-11T03:41:31.037 INFO:tasks.rados.rados.2.smithi175.stdout:880: finishing rollback tid 2 to smithi17557870-11 2024-09-11T03:41:31.037 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 107 (ObjNum 270 snap 69 seq_num 270) dirty exists 2024-09-11T03:41:31.037 INFO:tasks.rados.rados.2.smithi175.stdout:879: expect (ObjNum 296 snap 79 seq_num 296) 2024-09-11T03:41:31.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:31.083 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:31.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:31.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:31.131 INFO:tasks.rados.rados.2.smithi175.stdout:877: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:41:31.185 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:31.207 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:31.207 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:31.255 INFO:tasks.rados.rados.2.smithi175.stdout:881: finishing copy_from racing read to smithi17557870-26 2024-09-11T03:41:31.255 INFO:tasks.rados.rados.2.smithi175.stdout:881: finishing copy_from to smithi17557870-26 2024-09-11T03:41:31.255 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 157 (ObjNum 317 snap 86 seq_num 317) dirty exists 2024-09-11T03:41:31.265 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:41:31.308 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:31.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:31 smithi084 ceph-mon[27254]: osdmap e273: 12 total, 12 up, 10 in 2024-09-11T03:41:31.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:31 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2590520758' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:41:31.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:31 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/18190206' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-11T03:41:31.330 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:31.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:31.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:31.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:31 smithi121 ceph-mon[30729]: osdmap e273: 12 total, 12 up, 10 in 2024-09-11T03:41:31.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:31 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2590520758' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:41:31.385 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:31 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/18190206' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-11T03:41:31.432 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:31.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:31.454 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:31.454 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:31.472 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:41:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:31 smithi175 ceph-mon[30894]: osdmap e273: 12 total, 12 up, 10 in 2024-09-11T03:41:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:31 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2590520758' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:41:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:31 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/18190206' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-11T03:41:31.555 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:31.576 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:31.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:31.577 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:31.678 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:31.700 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:31.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:31.701 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:31.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:31.823 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:31.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:31.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:31.856 INFO:tasks.rados.rados.2.smithi175.stdout:876: done (6 left) 2024-09-11T03:41:31.856 INFO:tasks.rados.rados.2.smithi175.stdout:877: done (5 left) 2024-09-11T03:41:31.856 INFO:tasks.rados.rados.2.smithi175.stdout:878: done (4 left) 2024-09-11T03:41:31.857 INFO:tasks.rados.rados.2.smithi175.stdout:879: done (3 left) 2024-09-11T03:41:31.857 INFO:tasks.rados.rados.2.smithi175.stdout:880: done (2 left) 2024-09-11T03:41:31.857 INFO:tasks.rados.rados.2.smithi175.stdout:881: done (1 left) 2024-09-11T03:41:31.857 INFO:tasks.rados.rados.2.smithi175.stdout:882: done (0 left) 2024-09-11T03:41:31.857 INFO:tasks.rados.rados.2.smithi175.stdout:883: snap_remove snap 89 2024-09-11T03:41:31.895 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:41:31.895 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:41:31.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:31.947 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:31.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:31.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:32.049 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:32.072 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:32.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:32.073 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:32 smithi121 ceph-mon[30729]: pgmap v565: 47 pgs: 47 active+clean; 136 MiB data, 617 MiB used, 893 GiB / 894 GiB avail; 6.8 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-11T03:41:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:32 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:32 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2590520758' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:41:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:32 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/18190206' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-11T03:41:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:32 smithi121 ceph-mon[30729]: osdmap e274: 12 total, 12 up, 10 in 2024-09-11T03:41:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:32 smithi121 ceph-mon[30729]: osdmap e275: 12 total, 12 up, 10 in 2024-09-11T03:41:32.173 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:32.195 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:32.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:32.196 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:32.297 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:32.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:32 smithi084 ceph-mon[27254]: pgmap v565: 47 pgs: 47 active+clean; 136 MiB data, 617 MiB used, 893 GiB / 894 GiB avail; 6.8 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-11T03:41:32.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:32 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:32.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:32 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2590520758' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:41:32.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:32 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/18190206' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-11T03:41:32.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:32 smithi084 ceph-mon[27254]: osdmap e274: 12 total, 12 up, 10 in 2024-09-11T03:41:32.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:32 smithi084 ceph-mon[27254]: osdmap e275: 12 total, 12 up, 10 in 2024-09-11T03:41:32.320 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:32.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:32.422 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:32.444 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:32.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:32.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:32.453 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:32 smithi175 ceph-mon[30894]: pgmap v565: 47 pgs: 47 active+clean; 136 MiB data, 617 MiB used, 893 GiB / 894 GiB avail; 6.8 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-11T03:41:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:32 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:32 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2590520758' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:41:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:32 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/18190206' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-11T03:41:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:32 smithi175 ceph-mon[30894]: osdmap e274: 12 total, 12 up, 10 in 2024-09-11T03:41:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:32 smithi175 ceph-mon[30894]: osdmap e275: 12 total, 12 up, 10 in 2024-09-11T03:41:32.488 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T03:41:32.546 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:32.567 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:32.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:32.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:32.669 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:32.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:32.699 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:32.699 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:32.727 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:41:32.800 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:32.820 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:32.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:32.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:32.861 INFO:tasks.rados.rados.2.smithi175.stdout:883: done (0 left) 2024-09-11T03:41:32.861 INFO:tasks.rados.rados.2.smithi175.stdout:884: write (excl) oid 41 current snap is 91 2024-09-11T03:41:32.861 INFO:tasks.rados.rados.2.smithi175.stdout:884: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2024-09-11T03:41:32.868 INFO:tasks.rados.rados.2.smithi175.stdout:884: writing smithi17557870-41 from 679082 to 1297082 tid 1 2024-09-11T03:41:32.875 INFO:tasks.rados.rados.2.smithi175.stdout:884: writing smithi17557870-41 from 1843718 to 2412823 tid 2 2024-09-11T03:41:32.881 INFO:tasks.rados.rados.2.smithi175.stdout:884: writing smithi17557870-41 from 2853072 to 3479082 tid 3 2024-09-11T03:41:32.881 INFO:tasks.rados.rados.2.smithi175.stdout:885: copy_from oid 46 from oid 24 current snap is 91 2024-09-11T03:41:32.881 INFO:tasks.rados.rados.2.smithi175.stdout:884: finishing write tid 1 to smithi17557870-41 2024-09-11T03:41:32.881 INFO:tasks.rados.rados.2.smithi175.stdout:884: finishing write tid 2 to smithi17557870-41 2024-09-11T03:41:32.881 INFO:tasks.rados.rados.2.smithi175.stdout:886: rollback oid 45 current snap is 91 2024-09-11T03:41:32.881 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 45 to 86 2024-09-11T03:41:32.882 INFO:tasks.rados.rados.2.smithi175.stdout:887: write (excl) oid 30 current snap is 91 2024-09-11T03:41:32.882 INFO:tasks.rados.rados.2.smithi175.stdout:887: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2024-09-11T03:41:32.887 INFO:tasks.rados.rados.2.smithi175.stdout:887: writing smithi17557870-30 from 695889 to 1105491 tid 1 2024-09-11T03:41:32.894 INFO:tasks.rados.rados.2.smithi175.stdout:887: writing smithi17557870-30 from 1618553 to 2331316 tid 2 2024-09-11T03:41:32.901 INFO:tasks.rados.rados.2.smithi175.stdout:887: writing smithi17557870-30 from 2996848 to 3495889 tid 3 2024-09-11T03:41:32.901 INFO:tasks.rados.rados.2.smithi175.stdout:888: snap_create 2024-09-11T03:41:32.901 INFO:tasks.rados.rados.2.smithi175.stdout:886: finishing rollback tid 0 to smithi17557870-45 2024-09-11T03:41:32.901 INFO:tasks.rados.rados.2.smithi175.stdout:884: finishing write tid 3 to smithi17557870-41 2024-09-11T03:41:32.901 INFO:tasks.rados.rados.2.smithi175.stdout:884: finishing write tid 4 to smithi17557870-41 2024-09-11T03:41:32.901 INFO:tasks.rados.rados.2.smithi175.stdout:886: finishing rollback tid 1 to smithi17557870-45 2024-09-11T03:41:32.901 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 225 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T03:41:32.901 INFO:tasks.rados.rados.2.smithi175.stdout:884: finishing write tid 5 to smithi17557870-41 2024-09-11T03:41:32.901 INFO:tasks.rados.rados.2.smithi175.stdout:884: oid 41 updating version 0 to 263 2024-09-11T03:41:32.901 INFO:tasks.rados.rados.2.smithi175.stdout:884: oid 41 version 263 is already newer than 261 2024-09-11T03:41:32.901 INFO:tasks.rados.rados.2.smithi175.stdout:884: oid 41 version 263 is already newer than 262 2024-09-11T03:41:32.901 INFO:tasks.rados.rados.2.smithi175.stdout:884: oid 41 version 263 is already newer than 260 2024-09-11T03:41:32.901 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 263 (ObjNum 326 snap 91 seq_num 326) dirty exists 2024-09-11T03:41:32.902 INFO:tasks.rados.rados.2.smithi175.stdout:884: left oid 41 (ObjNum 326 snap 91 seq_num 326) 2024-09-11T03:41:32.902 INFO:tasks.rados.rados.2.smithi175.stdout:885: finishing copy_from racing read to smithi17557870-46 2024-09-11T03:41:32.902 INFO:tasks.rados.rados.2.smithi175.stdout:885: finishing copy_from to smithi17557870-46 2024-09-11T03:41:32.902 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 160 (ObjNum 267 snap 68 seq_num 267) dirty exists 2024-09-11T03:41:32.902 INFO:tasks.rados.rados.2.smithi175.stdout:887: finishing write tid 1 to smithi17557870-30 2024-09-11T03:41:32.902 INFO:tasks.rados.rados.2.smithi175.stdout:887: finishing write tid 2 to smithi17557870-30 2024-09-11T03:41:32.902 INFO:tasks.rados.rados.2.smithi175.stdout:887: finishing write tid 3 to smithi17557870-30 2024-09-11T03:41:32.903 INFO:tasks.rados.rados.2.smithi175.stdout:887: finishing write tid 4 to smithi17557870-30 2024-09-11T03:41:32.903 INFO:tasks.rados.rados.2.smithi175.stdout:887: finishing write tid 5 to smithi17557870-30 2024-09-11T03:41:32.903 INFO:tasks.rados.rados.2.smithi175.stdout:887: oid 30 updating version 0 to 301 2024-09-11T03:41:32.903 INFO:tasks.rados.rados.2.smithi175.stdout:887: oid 30 updating version 301 to 302 2024-09-11T03:41:32.903 INFO:tasks.rados.rados.2.smithi175.stdout:887: oid 30 updating version 302 to 304 2024-09-11T03:41:32.903 INFO:tasks.rados.rados.2.smithi175.stdout:887: oid 30 version 304 is already newer than 303 2024-09-11T03:41:32.903 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 304 (ObjNum 327 snap 91 seq_num 327) dirty exists 2024-09-11T03:41:32.904 INFO:tasks.rados.rados.2.smithi175.stdout:887: left oid 30 (ObjNum 327 snap 91 seq_num 327) 2024-09-11T03:41:32.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:32.943 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:32.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:32.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:33.019 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:41:33.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:41:33.034 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:41:33.035 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:41:33.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:33.050 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:41:33.050 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:41:33.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:33.066 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:33.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:33.066 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:41:33.067 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:41:33.084 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:41:33.084 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:41:33.099 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:41:33.099 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:41:33.116 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:41:33.116 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:41:33.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:33 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:41:33.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:33 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:41:33.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:33 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-11T03:41:33.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:33 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-11T03:41:33.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:33 smithi121 ceph-mon[30729]: osdmap e276: 12 total, 12 up, 10 in 2024-09-11T03:41:33.124 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:33 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/345445220' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:33.133 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:41:33.133 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:41:33.149 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:41:33.150 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:41:33.166 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:41:33.166 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:41:33.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:33.183 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:41:33.183 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:41:33.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:33 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:41:33.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:33 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:41:33.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:33 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-11T03:41:33.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:33 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-11T03:41:33.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:33 smithi084 ceph-mon[27254]: osdmap e276: 12 total, 12 up, 10 in 2024-09-11T03:41:33.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:33 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/345445220' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:33.187 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:33.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:33.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:33.200 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:41:33.201 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:41:33.289 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:33.315 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:41:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:33.317 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:33.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:33.418 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:33.439 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:33.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:33.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:33.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:33 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:41:33.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:33 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:41:33.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:33 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-11T03:41:33.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:33 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-11T03:41:33.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:33 smithi175 ceph-mon[30894]: osdmap e276: 12 total, 12 up, 10 in 2024-09-11T03:41:33.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:33 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/345445220' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:33.540 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:33.566 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:33.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:33.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:33.668 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:33.690 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:33.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:33.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:33.792 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:33.818 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:33.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:33.819 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:33.865 INFO:tasks.rados.rados.2.smithi175.stdout:884: done (4 left) 2024-09-11T03:41:33.865 INFO:tasks.rados.rados.2.smithi175.stdout:885: done (3 left) 2024-09-11T03:41:33.865 INFO:tasks.rados.rados.2.smithi175.stdout:886: done (2 left) 2024-09-11T03:41:33.865 INFO:tasks.rados.rados.2.smithi175.stdout:887: done (1 left) 2024-09-11T03:41:33.865 INFO:tasks.rados.rados.2.smithi175.stdout:888: done (0 left) 2024-09-11T03:41:33.865 INFO:tasks.rados.rados.2.smithi175.stdout:889: snap_remove snap 85 2024-09-11T03:41:33.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:33.942 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:33.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:33.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:33.971 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T03:41:34.045 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:34.066 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:34.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:34.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:34.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:34 smithi084 ceph-mon[27254]: pgmap v568: 47 pgs: 47 active+clean; 142 MiB data, 616 MiB used, 893 GiB / 894 GiB avail; 17 MiB/s rd, 3.2 MiB/s wr, 21 op/s 2024-09-11T03:41:34.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:34 smithi084 ceph-mon[27254]: osdmap e277: 12 total, 12 up, 10 in 2024-09-11T03:41:34.167 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:34.193 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:34.194 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:34.251 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:41:34.295 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:34.321 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:34.322 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:34.339 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:41:34.371 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:34 smithi121 ceph-mon[30729]: pgmap v568: 47 pgs: 47 active+clean; 142 MiB data, 616 MiB used, 893 GiB / 894 GiB avail; 17 MiB/s rd, 3.2 MiB/s wr, 21 op/s 2024-09-11T03:41:34.372 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:34 smithi121 ceph-mon[30729]: osdmap e277: 12 total, 12 up, 10 in 2024-09-11T03:41:34.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:34.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:34.444 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:34.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:34 smithi175 ceph-mon[30894]: pgmap v568: 47 pgs: 47 active+clean; 142 MiB data, 616 MiB used, 893 GiB / 894 GiB avail; 17 MiB/s rd, 3.2 MiB/s wr, 21 op/s 2024-09-11T03:41:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:34 smithi175 ceph-mon[30894]: osdmap e277: 12 total, 12 up, 10 in 2024-09-11T03:41:34.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:34.566 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:34.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:34.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:34.582 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:34.667 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:34.688 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:34.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:34.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:34.782 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:41:34.782 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:41:34.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:41:34.782 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:41:34.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.782 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:221 2.3 2:c6c49f2c:::smithi17557870-44:head [copy-from ver 9] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:41:34.782 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941456+0000", 2024-09-11T03:41:34.783 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 174.77265944300001, 2024-09-11T03:41:34.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015312882, 2024-09-11T03:41:34.783 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.783 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.783 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:34.783 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.783 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.783 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.783 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 221 2024-09-11T03:41:34.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.783 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941456+0000", 2024-09-11T03:41:34.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941456+0000", 2024-09-11T03:41:34.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941459+0000", 2024-09-11T03:41:34.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9189999999999999e-06 2024-09-11T03:41:34.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941469+0000", 2024-09-11T03:41:34.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0069e-05 2024-09-11T03:41:34.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941490+0000", 2024-09-11T03:41:34.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0947000000000001e-05 2024-09-11T03:41:34.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941501+0000", 2024-09-11T03:41:34.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0169e-05 2024-09-11T03:41:34.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941585+0000", 2024-09-11T03:41:34.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4022000000000003e-05 2024-09-11T03:41:34.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:41:34.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941590+0000", 2024-09-11T03:41:34.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4929999999999998e-06 2024-09-11T03:41:34.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.942889+0000", 2024-09-11T03:41:34.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001298539 2024-09-11T03:41:34.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.942984+0000", 2024-09-11T03:41:34.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5513999999999994e-05 2024-09-11T03:41:34.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:41:34.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.950816+0000", 2024-09-11T03:41:34.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078315850000000003 2024-09-11T03:41:34.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956638+0000", 2024-09-11T03:41:34.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058225769999999998 2024-09-11T03:41:34.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:41:34.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956649+0000", 2024-09-11T03:41:34.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0492000000000001e-05 2024-09-11T03:41:34.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956690+0000", 2024-09-11T03:41:34.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.142e-05 2024-09-11T03:41:34.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:34.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956728+0000", 2024-09-11T03:41:34.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8195000000000003e-05 2024-09-11T03:41:34.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956769+0000", 2024-09-11T03:41:34.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0941e-05 2024-09-11T03:41:34.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.789 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.790 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:222 2.3 2:c6c49f2c:::smithi17557870-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:41:34.790 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:41:34.790 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 174.77258136399999, 2024-09-11T03:41:34.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015032419, 2024-09-11T03:41:34.790 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.790 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.790 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:41:34.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.790 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 222 2024-09-11T03:41:34.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.791 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:41:34.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:41:34.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:41:34.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.51e-07 2024-09-11T03:41:34.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941539+0000", 2024-09-11T03:41:34.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0060000000000003e-06 2024-09-11T03:41:34.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941547+0000", 2024-09-11T03:41:34.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9009999999999994e-06 2024-09-11T03:41:34.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941553+0000", 2024-09-11T03:41:34.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1519999999999997e-06 2024-09-11T03:41:34.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941691+0000", 2024-09-11T03:41:34.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013752600000000001 2024-09-11T03:41:34.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:41:34.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941697+0000", 2024-09-11T03:41:34.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3770000000000001e-06 2024-09-11T03:41:34.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943170+0000", 2024-09-11T03:41:34.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001473504 2024-09-11T03:41:34.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:41:34.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943225+0000", 2024-09-11T03:41:34.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4101000000000003e-05 2024-09-11T03:41:34.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956388+0000", 2024-09-11T03:41:34.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013163223999999999 2024-09-11T03:41:34.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956456+0000", 2024-09-11T03:41:34.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8563000000000001e-05 2024-09-11T03:41:34.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956567+0000", 2024-09-11T03:41:34.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011061399999999999 2024-09-11T03:41:34.795 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.795 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.795 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.796 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:223 2.3 2:c3768452:::smithi17557870-11:head [copy-from ver 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:41:34.796 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:41:34.796 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 174.772541319, 2024-09-11T03:41:34.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016727301999999999, 2024-09-11T03:41:34.796 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.796 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.796 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:34.796 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.796 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.796 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.797 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 223 2024-09-11T03:41:34.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.797 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:41:34.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:41:34.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:41:34.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0999999999999999e-07 2024-09-11T03:41:34.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941578+0000", 2024-09-11T03:41:34.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.021e-06 2024-09-11T03:41:34.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941583+0000", 2024-09-11T03:41:34.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0520000000000001e-06 2024-09-11T03:41:34.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941586+0000", 2024-09-11T03:41:34.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-09-11T03:41:34.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941750+0000", 2024-09-11T03:41:34.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016353200000000001 2024-09-11T03:41:34.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:41:34.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941756+0000", 2024-09-11T03:41:34.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.75e-06 2024-09-11T03:41:34.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943289+0000", 2024-09-11T03:41:34.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001533747 2024-09-11T03:41:34.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943383+0000", 2024-09-11T03:41:34.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3655999999999999e-05 2024-09-11T03:41:34.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:41:34.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.954899+0000", 2024-09-11T03:41:34.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011515463 2024-09-11T03:41:34.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:41:34.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.957952+0000", 2024-09-11T03:41:34.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030533560000000001 2024-09-11T03:41:34.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958200+0000", 2024-09-11T03:41:34.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024759100000000002 2024-09-11T03:41:34.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958229+0000", 2024-09-11T03:41:34.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9862e-05 2024-09-11T03:41:34.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:34.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958262+0000", 2024-09-11T03:41:34.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2267000000000001e-05 2024-09-11T03:41:34.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958302+0000", 2024-09-11T03:41:34.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0204000000000003e-05 2024-09-11T03:41:34.803 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.803 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.803 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.803 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:224 2.3 2:c3768452:::smithi17557870-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:41:34.803 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941599+0000", 2024-09-11T03:41:34.803 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 174.77251676700001, 2024-09-11T03:41:34.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014750406000000001, 2024-09-11T03:41:34.803 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.803 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.803 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:41:34.804 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.804 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.804 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.804 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 224 2024-09-11T03:41:34.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.804 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941599+0000", 2024-09-11T03:41:34.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941599+0000", 2024-09-11T03:41:34.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941599+0000", 2024-09-11T03:41:34.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2599999999999998e-07 2024-09-11T03:41:34.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941601+0000", 2024-09-11T03:41:34.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6249999999999999e-06 2024-09-11T03:41:34.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941605+0000", 2024-09-11T03:41:34.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3059999999999999e-06 2024-09-11T03:41:34.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941609+0000", 2024-09-11T03:41:34.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.506e-06 2024-09-11T03:41:34.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941789+0000", 2024-09-11T03:41:34.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000180261 2024-09-11T03:41:34.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:41:34.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941795+0000", 2024-09-11T03:41:34.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4970000000000001e-06 2024-09-11T03:41:34.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943572+0000", 2024-09-11T03:41:34.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001777725 2024-09-11T03:41:34.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:41:34.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943616+0000", 2024-09-11T03:41:34.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3550999999999997e-05 2024-09-11T03:41:34.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956192+0000", 2024-09-11T03:41:34.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012576517000000001 2024-09-11T03:41:34.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956267+0000", 2024-09-11T03:41:34.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4956000000000005e-05 2024-09-11T03:41:34.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956350+0000", 2024-09-11T03:41:34.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2136000000000001e-05 2024-09-11T03:41:34.809 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.809 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.810 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.810 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:343 2.2 2:494ba932:::smithi17557870-43:head [copy-from ver 6] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:41:34.810 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.820776+0000", 2024-09-11T03:41:34.810 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 164.893340177, 2024-09-11T03:41:34.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017461323000000001, 2024-09-11T03:41:34.810 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.810 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.810 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:34.810 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.810 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.810 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.810 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 343 2024-09-11T03:41:34.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.811 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820776+0000", 2024-09-11T03:41:34.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820776+0000", 2024-09-11T03:41:34.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820779+0000", 2024-09-11T03:41:34.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3000000000000002e-06 2024-09-11T03:41:34.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820797+0000", 2024-09-11T03:41:34.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7730000000000001e-05 2024-09-11T03:41:34.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820853+0000", 2024-09-11T03:41:34.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5909000000000003e-05 2024-09-11T03:41:34.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820872+0000", 2024-09-11T03:41:34.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9709999999999999e-05 2024-09-11T03:41:34.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820976+0000", 2024-09-11T03:41:34.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010374099999999999 2024-09-11T03:41:34.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:41:34.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820982+0000", 2024-09-11T03:41:34.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4389999999999999e-06 2024-09-11T03:41:34.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825547+0000", 2024-09-11T03:41:34.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045658260000000003 2024-09-11T03:41:34.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825626+0000", 2024-09-11T03:41:34.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8289999999999998e-05 2024-09-11T03:41:34.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,6", 2024-09-11T03:41:34.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830895+0000", 2024-09-11T03:41:34.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052695060000000002 2024-09-11T03:41:34.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:41:34.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.835040+0000", 2024-09-11T03:41:34.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041446549999999997 2024-09-11T03:41:34.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.835364+0000", 2024-09-11T03:41:34.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032384200000000002 2024-09-11T03:41:34.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838162+0000", 2024-09-11T03:41:34.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027987509999999999 2024-09-11T03:41:34.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:34.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838203+0000", 2024-09-11T03:41:34.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0209000000000001e-05 2024-09-11T03:41:34.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838237+0000", 2024-09-11T03:41:34.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4415000000000002e-05 2024-09-11T03:41:34.817 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.817 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.817 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.817 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:344 2.2 2:494ba932:::smithi17557870-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:41:34.817 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.820917+0000", 2024-09-11T03:41:34.817 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 164.89319855100001, 2024-09-11T03:41:34.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013455992, 2024-09-11T03:41:34.817 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.817 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.817 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:41:34.818 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.818 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.818 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.818 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 344 2024-09-11T03:41:34.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.818 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820917+0000", 2024-09-11T03:41:34.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820917+0000", 2024-09-11T03:41:34.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820918+0000", 2024-09-11T03:41:34.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.34e-07 2024-09-11T03:41:34.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820920+0000", 2024-09-11T03:41:34.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.48e-06 2024-09-11T03:41:34.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820927+0000", 2024-09-11T03:41:34.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.187e-06 2024-09-11T03:41:34.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820931+0000", 2024-09-11T03:41:34.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9349999999999996e-06 2024-09-11T03:41:34.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.821034+0000", 2024-09-11T03:41:34.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010277600000000001 2024-09-11T03:41:34.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:41:34.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.821040+0000", 2024-09-11T03:41:34.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6389999999999997e-06 2024-09-11T03:41:34.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825767+0000", 2024-09-11T03:41:34.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047268060000000001 2024-09-11T03:41:34.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:41:34.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825844+0000", 2024-09-11T03:41:34.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7066999999999999e-05 2024-09-11T03:41:34.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834208+0000", 2024-09-11T03:41:34.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083646199999999997 2024-09-11T03:41:34.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834284+0000", 2024-09-11T03:41:34.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5765000000000004e-05 2024-09-11T03:41:34.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834373+0000", 2024-09-11T03:41:34.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9283000000000006e-05 2024-09-11T03:41:34.822 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.822 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.822 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.823 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:607 2.3 2:c3768452:::smithi17557870-11:head [write 564735~613734,stat] snapc 11=[f,e] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-11T03:41:34.823 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:06.914163+0000", 2024-09-11T03:41:34.823 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 147.799953026, 2024-09-11T03:41:34.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054773620000000004, 2024-09-11T03:41:34.823 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.823 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.823 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:34.823 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.823 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.823 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.823 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 607 2024-09-11T03:41:34.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.823 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.914163+0000", 2024-09-11T03:41:34.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.914163+0000", 2024-09-11T03:41:34.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.914166+0000", 2024-09-11T03:41:34.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2430000000000001e-06 2024-09-11T03:41:34.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.916572+0000", 2024-09-11T03:41:34.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024055209999999999 2024-09-11T03:41:34.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.916590+0000", 2024-09-11T03:41:34.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8487000000000001e-05 2024-09-11T03:41:34.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.916616+0000", 2024-09-11T03:41:34.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6152e-05 2024-09-11T03:41:34.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.916688+0000", 2024-09-11T03:41:34.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1240000000000002e-05 2024-09-11T03:41:34.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.916769+0000", 2024-09-11T03:41:34.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1767000000000005e-05 2024-09-11T03:41:34.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:41:34.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.916931+0000", 2024-09-11T03:41:34.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016213800000000001 2024-09-11T03:41:34.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:41:34.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.918750+0000", 2024-09-11T03:41:34.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001818297 2024-09-11T03:41:34.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.918987+0000", 2024-09-11T03:41:34.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000237407 2024-09-11T03:41:34.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.919548+0000", 2024-09-11T03:41:34.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056076800000000001 2024-09-11T03:41:34.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:34.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.919602+0000", 2024-09-11T03:41:34.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4449e-05 2024-09-11T03:41:34.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.919640+0000", 2024-09-11T03:41:34.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7892999999999998e-05 2024-09-11T03:41:34.828 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.828 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.828 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.829 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:754 2.3 2:c3768452:::smithi17557870-11:head [write 467261~717275] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:41:34.829 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:15.016104+0000", 2024-09-11T03:41:34.829 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 139.698011718, 2024-09-11T03:41:34.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010129861, 2024-09-11T03:41:34.829 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.829 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.829 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:34.829 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.829 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.829 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.830 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 754 2024-09-11T03:41:34.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.830 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.016104+0000", 2024-09-11T03:41:34.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.016104+0000", 2024-09-11T03:41:34.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.016108+0000", 2024-09-11T03:41:34.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8060000000000001e-06 2024-09-11T03:41:34.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.018233+0000", 2024-09-11T03:41:34.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021249900000000002 2024-09-11T03:41:34.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.018249+0000", 2024-09-11T03:41:34.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6052000000000001e-05 2024-09-11T03:41:34.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.018260+0000", 2024-09-11T03:41:34.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1100999999999999e-05 2024-09-11T03:41:34.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.018347+0000", 2024-09-11T03:41:34.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7194000000000006e-05 2024-09-11T03:41:34.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:41:34.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.018352+0000", 2024-09-11T03:41:34.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9559999999999996e-06 2024-09-11T03:41:34.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.019567+0000", 2024-09-11T03:41:34.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012144899999999999 2024-09-11T03:41:34.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.019630+0000", 2024-09-11T03:41:34.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2980999999999999e-05 2024-09-11T03:41:34.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:41:34.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.019722+0000", 2024-09-11T03:41:34.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1965000000000005e-05 2024-09-11T03:41:34.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:41:34.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022248+0000", 2024-09-11T03:41:34.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025267409999999999 2024-09-11T03:41:34.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022729+0000", 2024-09-11T03:41:34.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00048018799999999999 2024-09-11T03:41:34.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026172+0000", 2024-09-11T03:41:34.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034435500000000001 2024-09-11T03:41:34.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:34.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026205+0000", 2024-09-11T03:41:34.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2703000000000002e-05 2024-09-11T03:41:34.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026234+0000", 2024-09-11T03:41:34.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9144000000000001e-05 2024-09-11T03:41:34.835 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.835 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.835 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.836 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:755 2.3 2:c3768452:::smithi17557870-11:head [write 1847991~219270] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:41:34.836 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:15.020950+0000", 2024-09-11T03:41:34.836 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 139.693165951, 2024-09-11T03:41:34.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056636869999999997, 2024-09-11T03:41:34.836 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.836 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.836 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:34.836 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.836 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.836 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.836 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 755 2024-09-11T03:41:34.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.837 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.020950+0000", 2024-09-11T03:41:34.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.020950+0000", 2024-09-11T03:41:34.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.020951+0000", 2024-09-11T03:41:34.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5340000000000001e-06 2024-09-11T03:41:34.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021252+0000", 2024-09-11T03:41:34.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000300019 2024-09-11T03:41:34.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021271+0000", 2024-09-11T03:41:34.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9612000000000001e-05 2024-09-11T03:41:34.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021281+0000", 2024-09-11T03:41:34.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5210000000000008e-06 2024-09-11T03:41:34.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021405+0000", 2024-09-11T03:41:34.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012406599999999999 2024-09-11T03:41:34.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021483+0000", 2024-09-11T03:41:34.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8788000000000005e-05 2024-09-11T03:41:34.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:41:34.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021643+0000", 2024-09-11T03:41:34.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015989500000000001 2024-09-11T03:41:34.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:41:34.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022823+0000", 2024-09-11T03:41:34.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011795670000000001 2024-09-11T03:41:34.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.023025+0000", 2024-09-11T03:41:34.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000202072 2024-09-11T03:41:34.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026543+0000", 2024-09-11T03:41:34.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035177329999999999 2024-09-11T03:41:34.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:34.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026579+0000", 2024-09-11T03:41:34.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6513000000000001e-05 2024-09-11T03:41:34.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026614+0000", 2024-09-11T03:41:34.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4366999999999998e-05 2024-09-11T03:41:34.841 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.841 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.842 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.842 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:756 2.3 2:c3768452:::smithi17557870-11:head [setxattr _header (57) in=7b,truncate 2067261] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:41:34.842 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:15.021301+0000", 2024-09-11T03:41:34.842 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 139.692814446, 2024-09-11T03:41:34.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054061910000000003, 2024-09-11T03:41:34.842 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.842 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.842 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:34.842 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.842 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.842 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.842 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 756 2024-09-11T03:41:34.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.843 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021301+0000", 2024-09-11T03:41:34.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021301+0000", 2024-09-11T03:41:34.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021302+0000", 2024-09-11T03:41:34.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5000000000000002e-07 2024-09-11T03:41:34.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021307+0000", 2024-09-11T03:41:34.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0980000000000001e-06 2024-09-11T03:41:34.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021315+0000", 2024-09-11T03:41:34.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0709999999999995e-06 2024-09-11T03:41:34.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021321+0000", 2024-09-11T03:41:34.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6779999999999997e-06 2024-09-11T03:41:34.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022288+0000", 2024-09-11T03:41:34.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00096644199999999997 2024-09-11T03:41:34.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022341+0000", 2024-09-11T03:41:34.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3677000000000001e-05 2024-09-11T03:41:34.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:41:34.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022432+0000", 2024-09-11T03:41:34.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0494999999999993e-05 2024-09-11T03:41:34.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:41:34.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022835+0000", 2024-09-11T03:41:34.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040375600000000002 2024-09-11T03:41:34.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.023136+0000", 2024-09-11T03:41:34.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030039599999999997 2024-09-11T03:41:34.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026642+0000", 2024-09-11T03:41:34.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003505991 2024-09-11T03:41:34.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:34.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026674+0000", 2024-09-11T03:41:34.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2289000000000002e-05 2024-09-11T03:41:34.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026708+0000", 2024-09-11T03:41:34.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3547999999999998e-05 2024-09-11T03:41:34.848 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.848 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.848 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.848 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:757 2.3 2:c3768452:::smithi17557870-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:41:34.848 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:15.021330+0000", 2024-09-11T03:41:34.848 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 139.692785671, 2024-09-11T03:41:34.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055098129999999997, 2024-09-11T03:41:34.848 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.848 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.848 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:41:34.848 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.849 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.849 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.849 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 757 2024-09-11T03:41:34.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.849 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021330+0000", 2024-09-11T03:41:34.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021330+0000", 2024-09-11T03:41:34.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021331+0000", 2024-09-11T03:41:34.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7100000000000002e-07 2024-09-11T03:41:34.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021333+0000", 2024-09-11T03:41:34.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0940000000000002e-06 2024-09-11T03:41:34.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021347+0000", 2024-09-11T03:41:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3899e-05 2024-09-11T03:41:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.021353+0000", 2024-09-11T03:41:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9059999999999999e-06 2024-09-11T03:41:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022759+0000", 2024-09-11T03:41:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0014062879999999999 2024-09-11T03:41:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:41:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.022809+0000", 2024-09-11T03:41:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9629999999999997e-05 2024-09-11T03:41:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026734+0000", 2024-09-11T03:41:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039259209999999998 2024-09-11T03:41:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026779+0000", 2024-09-11T03:41:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4574999999999999e-05 2024-09-11T03:41:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:15.026840+0000", 2024-09-11T03:41:34.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1029000000000001e-05 2024-09-11T03:41:34.853 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.853 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.853 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.853 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1068 2.3 2:c3768452:::smithi17557870-11:head [write 507611~777843,stat] snapc 28=[28,27,26] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:41:34.853 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.871196+0000", 2024-09-11T03:41:34.853 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 102.84292012900001, 2024-09-11T03:41:34.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.036362196999999999, 2024-09-11T03:41:34.853 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.853 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.853 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:34.853 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.853 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.854 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.854 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1068 2024-09-11T03:41:34.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.854 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.871196+0000", 2024-09-11T03:41:34.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.871196+0000", 2024-09-11T03:41:34.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.871198+0000", 2024-09-11T03:41:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5000000000000002e-06 2024-09-11T03:41:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873403+0000", 2024-09-11T03:41:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022048739999999999 2024-09-11T03:41:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873438+0000", 2024-09-11T03:41:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4518999999999999e-05 2024-09-11T03:41:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873453+0000", 2024-09-11T03:41:34.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5733999999999999e-05 2024-09-11T03:41:34.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873597+0000", 2024-09-11T03:41:34.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143702 2024-09-11T03:41:34.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873668+0000", 2024-09-11T03:41:34.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0996999999999998e-05 2024-09-11T03:41:34.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:41:34.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873819+0000", 2024-09-11T03:41:34.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015105300000000001 2024-09-11T03:41:34.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:41:34.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.875607+0000", 2024-09-11T03:41:34.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017877419999999999 2024-09-11T03:41:34.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.875920+0000", 2024-09-11T03:41:34.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031312799999999997 2024-09-11T03:41:34.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907471+0000", 2024-09-11T03:41:34.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.031551348 2024-09-11T03:41:34.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:34.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907514+0000", 2024-09-11T03:41:34.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2873000000000001e-05 2024-09-11T03:41:34.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907558+0000", 2024-09-11T03:41:34.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3727000000000003e-05 2024-09-11T03:41:34.859 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.859 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.859 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.859 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1069 2.3 2:c3768452:::smithi17557870-11:head [write 2073973~444721] snapc 28=[28,27,26] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:41:34.859 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.877745+0000", 2024-09-11T03:41:34.859 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 102.836370652, 2024-09-11T03:41:34.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029936307999999998, 2024-09-11T03:41:34.859 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.859 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.859 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:34.859 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.859 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.860 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.860 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1069 2024-09-11T03:41:34.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.860 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.877745+0000", 2024-09-11T03:41:34.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.877745+0000", 2024-09-11T03:41:34.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.877747+0000", 2024-09-11T03:41:34.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.61e-06 2024-09-11T03:41:34.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878169+0000", 2024-09-11T03:41:34.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00042167499999999999 2024-09-11T03:41:34.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878181+0000", 2024-09-11T03:41:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2714999999999999e-05 2024-09-11T03:41:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878187+0000", 2024-09-11T03:41:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2389999999999999e-06 2024-09-11T03:41:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878284+0000", 2024-09-11T03:41:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6632000000000006e-05 2024-09-11T03:41:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878363+0000", 2024-09-11T03:41:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9369000000000005e-05 2024-09-11T03:41:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:41:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878472+0000", 2024-09-11T03:41:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108995 2024-09-11T03:41:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:41:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879469+0000", 2024-09-11T03:41:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00099629100000000006 2024-09-11T03:41:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879835+0000", 2024-09-11T03:41:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036609700000000002 2024-09-11T03:41:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907609+0000", 2024-09-11T03:41:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027773923999999998 2024-09-11T03:41:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907647+0000", 2024-09-11T03:41:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8124000000000003e-05 2024-09-11T03:41:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907682+0000", 2024-09-11T03:41:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4637000000000001e-05 2024-09-11T03:41:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.865 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.865 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.865 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1070 2.3 2:c3768452:::smithi17557870-11:head [setxattr _header (57) in=7b,truncate 2907611] snapc 28=[28,27,26] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:41:34.865 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.878201+0000", 2024-09-11T03:41:34.865 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 102.835914856, 2024-09-11T03:41:34.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029581774000000002, 2024-09-11T03:41:34.865 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.865 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.865 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:34.865 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.865 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1070 2024-09-11T03:41:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878201+0000", 2024-09-11T03:41:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878201+0000", 2024-09-11T03:41:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878201+0000", 2024-09-11T03:41:34.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4299999999999998e-07 2024-09-11T03:41:34.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878222+0000", 2024-09-11T03:41:34.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0591999999999999e-05 2024-09-11T03:41:34.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878228+0000", 2024-09-11T03:41:34.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.857e-06 2024-09-11T03:41:34.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878232+0000", 2024-09-11T03:41:34.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6459999999999999e-06 2024-09-11T03:41:34.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879123+0000", 2024-09-11T03:41:34.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089174700000000003 2024-09-11T03:41:34.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879183+0000", 2024-09-11T03:41:34.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9502000000000001e-05 2024-09-11T03:41:34.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:41:34.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879287+0000", 2024-09-11T03:41:34.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103952 2024-09-11T03:41:34.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:41:34.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879606+0000", 2024-09-11T03:41:34.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031872600000000002 2024-09-11T03:41:34.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879933+0000", 2024-09-11T03:41:34.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032754700000000003 2024-09-11T03:41:34.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907711+0000", 2024-09-11T03:41:34.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027778265999999999 2024-09-11T03:41:34.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:34.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907746+0000", 2024-09-11T03:41:34.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4403999999999998e-05 2024-09-11T03:41:34.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907783+0000", 2024-09-11T03:41:34.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7092000000000003e-05 2024-09-11T03:41:34.871 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.871 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.871 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.871 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1071 2.3 2:c3768452:::smithi17557870-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:41:34.871 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.878273+0000", 2024-09-11T03:41:34.871 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 102.835843201, 2024-09-11T03:41:34.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.02966705, 2024-09-11T03:41:34.871 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.871 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.871 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:41:34.871 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.871 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.871 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.872 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1071 2024-09-11T03:41:34.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.872 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878273+0000", 2024-09-11T03:41:34.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878273+0000", 2024-09-11T03:41:34.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878273+0000", 2024-09-11T03:41:34.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1600000000000002e-07 2024-09-11T03:41:34.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878275+0000", 2024-09-11T03:41:34.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0949999999999998e-06 2024-09-11T03:41:34.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878281+0000", 2024-09-11T03:41:34.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6550000000000001e-06 2024-09-11T03:41:34.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878284+0000", 2024-09-11T03:41:34.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5370000000000002e-06 2024-09-11T03:41:34.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879510+0000", 2024-09-11T03:41:34.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001225638 2024-09-11T03:41:34.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:41:34.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879586+0000", 2024-09-11T03:41:34.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6058999999999998e-05 2024-09-11T03:41:34.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907811+0000", 2024-09-11T03:41:34.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028224775000000001 2024-09-11T03:41:34.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907873+0000", 2024-09-11T03:41:34.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2310000000000005e-05 2024-09-11T03:41:34.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907940+0000", 2024-09-11T03:41:34.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6564999999999997e-05 2024-09-11T03:41:34.875 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.875 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.875 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.876 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1555 2.3 2:c6c49f2c:::smithi17557870-44:head [copy-from ver 82] snapc 3c=[3c,3a,38,37] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-11T03:41:34.876 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:30.880293+0000", 2024-09-11T03:41:34.876 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 63.833823178000003, 2024-09-11T03:41:34.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020483438999999999, 2024-09-11T03:41:34.876 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.876 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.876 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:34.876 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.876 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.876 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.876 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1555 2024-09-11T03:41:34.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.876 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880293+0000", 2024-09-11T03:41:34.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880293+0000", 2024-09-11T03:41:34.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880295+0000", 2024-09-11T03:41:34.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2409999999999998e-06 2024-09-11T03:41:34.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880305+0000", 2024-09-11T03:41:34.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5999999999999996e-06 2024-09-11T03:41:34.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880326+0000", 2024-09-11T03:41:34.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1248999999999999e-05 2024-09-11T03:41:34.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880335+0000", 2024-09-11T03:41:34.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4369999999999996e-06 2024-09-11T03:41:34.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880414+0000", 2024-09-11T03:41:34.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8560999999999995e-05 2024-09-11T03:41:34.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:41:34.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880422+0000", 2024-09-11T03:41:34.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.712e-06 2024-09-11T03:41:34.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887559+0000", 2024-09-11T03:41:34.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071376920000000002 2024-09-11T03:41:34.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887625+0000", 2024-09-11T03:41:34.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6256999999999999e-05 2024-09-11T03:41:34.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:41:34.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.893371+0000", 2024-09-11T03:41:34.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0057455579999999996 2024-09-11T03:41:34.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.899577+0000", 2024-09-11T03:41:34.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062060600000000002 2024-09-11T03:41:34.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:41:34.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900444+0000", 2024-09-11T03:41:34.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000867307 2024-09-11T03:41:34.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:34.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900703+0000", 2024-09-11T03:41:34.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00025901299999999999 2024-09-11T03:41:34.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:34.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900743+0000", 2024-09-11T03:41:34.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9471000000000002e-05 2024-09-11T03:41:34.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900776+0000", 2024-09-11T03:41:34.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3281000000000002e-05 2024-09-11T03:41:34.882 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.882 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.882 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.882 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1556 2.3 2:c6c49f2c:::smithi17557870-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e214)", 2024-09-11T03:41:34.883 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:30.880359+0000", 2024-09-11T03:41:34.883 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 63.833756481000002, 2024-09-11T03:41:34.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017105065999999999, 2024-09-11T03:41:34.883 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.883 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.883 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:41:34.883 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.883 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.883 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.883 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1556 2024-09-11T03:41:34.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.883 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880359+0000", 2024-09-11T03:41:34.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880359+0000", 2024-09-11T03:41:34.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880360+0000", 2024-09-11T03:41:34.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.75e-07 2024-09-11T03:41:34.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880362+0000", 2024-09-11T03:41:34.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4959999999999999e-06 2024-09-11T03:41:34.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880370+0000", 2024-09-11T03:41:34.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.153e-06 2024-09-11T03:41:34.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880376+0000", 2024-09-11T03:41:34.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0900000000000001e-06 2024-09-11T03:41:34.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880478+0000", 2024-09-11T03:41:34.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102482 2024-09-11T03:41:34.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:41:34.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880487+0000", 2024-09-11T03:41:34.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8109999999999992e-06 2024-09-11T03:41:34.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887725+0000", 2024-09-11T03:41:34.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072383930000000001 2024-09-11T03:41:34.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:41:34.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887757+0000", 2024-09-11T03:41:34.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1692000000000001e-05 2024-09-11T03:41:34.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.897328+0000", 2024-09-11T03:41:34.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095712149999999992 2024-09-11T03:41:34.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.897393+0000", 2024-09-11T03:41:34.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5005000000000005e-05 2024-09-11T03:41:34.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.897464+0000", 2024-09-11T03:41:34.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1254000000000006e-05 2024-09-11T03:41:34.888 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.888 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.889 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.889 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:2162 2.3 2:c3768452:::smithi17557870-11:head [sparse-read 0~1488664,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e270)", 2024-09-11T03:41:34.889 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:26.857665+0000", 2024-09-11T03:41:34.889 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 7.8564506229999997, 2024-09-11T03:41:34.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005897312, 2024-09-11T03:41:34.889 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.889 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.889 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:41:34.889 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.889 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.889 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.890 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2162 2024-09-11T03:41:34.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.890 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857665+0000", 2024-09-11T03:41:34.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857665+0000", 2024-09-11T03:41:34.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857668+0000", 2024-09-11T03:41:34.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3860000000000001e-06 2024-09-11T03:41:34.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857679+0000", 2024-09-11T03:41:34.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1596e-05 2024-09-11T03:41:34.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857704+0000", 2024-09-11T03:41:34.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4751000000000001e-05 2024-09-11T03:41:34.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857717+0000", 2024-09-11T03:41:34.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2629999999999999e-05 2024-09-11T03:41:34.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857801+0000", 2024-09-11T03:41:34.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4406000000000005e-05 2024-09-11T03:41:34.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:41:34.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857806+0000", 2024-09-11T03:41:34.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3800000000000002e-06 2024-09-11T03:41:34.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.861669+0000", 2024-09-11T03:41:34.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038621620000000001 2024-09-11T03:41:34.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.861750+0000", 2024-09-11T03:41:34.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1401000000000002e-05 2024-09-11T03:41:34.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.863563+0000", 2024-09-11T03:41:34.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018125999999999999 2024-09-11T03:41:34.894 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.894 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.894 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.894 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:2163 2.3 2:c3768452:::smithi17557870-11:head [sparse-read 0~1488664] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e270)", 2024-09-11T03:41:34.894 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:26.857746+0000", 2024-09-11T03:41:34.894 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 7.8563699089999997, 2024-09-11T03:41:34.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059895959999999998, 2024-09-11T03:41:34.894 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.894 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.894 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:41:34.894 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.894 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.895 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.895 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2163 2024-09-11T03:41:34.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.895 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857746+0000", 2024-09-11T03:41:34.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857746+0000", 2024-09-11T03:41:34.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857747+0000", 2024-09-11T03:41:34.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7700000000000004e-07 2024-09-11T03:41:34.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857750+0000", 2024-09-11T03:41:34.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1219999999999999e-06 2024-09-11T03:41:34.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857758+0000", 2024-09-11T03:41:34.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9219999999999997e-06 2024-09-11T03:41:34.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857764+0000", 2024-09-11T03:41:34.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2550000000000003e-06 2024-09-11T03:41:34.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857884+0000", 2024-09-11T03:41:34.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000119951 2024-09-11T03:41:34.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:41:34.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857888+0000", 2024-09-11T03:41:34.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9210000000000002e-06 2024-09-11T03:41:34.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.863595+0000", 2024-09-11T03:41:34.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0057070749999999998 2024-09-11T03:41:34.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.863656+0000", 2024-09-11T03:41:34.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1450000000000003e-05 2024-09-11T03:41:34.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.863736+0000", 2024-09-11T03:41:34.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9322999999999999e-05 2024-09-11T03:41:34.899 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.899 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.899 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.899 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:2164 2.3 2:c3768452:::smithi17557870-11:head [read 0~2537890,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e270)", 2024-09-11T03:41:34.899 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:26.857792+0000", 2024-09-11T03:41:34.899 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 7.8563242610000001, 2024-09-11T03:41:34.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071414449999999997, 2024-09-11T03:41:34.900 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:34.900 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:34.900 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:41:34.900 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:34.900 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:34.900 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:34.900 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2164 2024-09-11T03:41:34.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.900 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:34.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:34.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857792+0000", 2024-09-11T03:41:34.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:34.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857792+0000", 2024-09-11T03:41:34.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:34.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:34.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857792+0000", 2024-09-11T03:41:34.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5000000000000003e-07 2024-09-11T03:41:34.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:34.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857796+0000", 2024-09-11T03:41:34.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8190000000000002e-06 2024-09-11T03:41:34.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:34.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857804+0000", 2024-09-11T03:41:34.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0120000000000005e-06 2024-09-11T03:41:34.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:34.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857812+0000", 2024-09-11T03:41:34.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4869999999999996e-06 2024-09-11T03:41:34.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857919+0000", 2024-09-11T03:41:34.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107699 2024-09-11T03:41:34.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:41:34.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.857924+0000", 2024-09-11T03:41:34.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2440000000000001e-06 2024-09-11T03:41:34.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:34.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.863756+0000", 2024-09-11T03:41:34.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058315149999999998 2024-09-11T03:41:34.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:34.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.863806+0000", 2024-09-11T03:41:34.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0367000000000001e-05 2024-09-11T03:41:34.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:34.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:34.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:34.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:26.864933+0000", 2024-09-11T03:41:34.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001126752 2024-09-11T03:41:34.904 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.904 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.904 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.904 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:34.904 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:34.904 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:41:34.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:34.906 INFO:tasks.rados.rados.2.smithi175.stdout:889: done (0 left) 2024-09-11T03:41:34.906 INFO:tasks.rados.rados.2.smithi175.stdout:890: read oid 15 snap 86 2024-09-11T03:41:34.906 INFO:tasks.rados.rados.2.smithi175.stdout:890: expect (ObjNum 317 snap 86 seq_num 317) 2024-09-11T03:41:34.906 INFO:tasks.rados.rados.2.smithi175.stdout:891: delete oid 38 current snap is 92 2024-09-11T03:41:34.906 INFO:tasks.rados.rados.2.smithi175.stdout:891: done (1 left) 2024-09-11T03:41:34.906 INFO:tasks.rados.rados.2.smithi175.stdout:892: write oid 9 current snap is 92 2024-09-11T03:41:34.907 INFO:tasks.rados.rados.2.smithi175.stdout:892: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2024-09-11T03:41:34.907 INFO:tasks.rados.rados.2.smithi175.stdout:892: writing smithi17557870-9 from 729503 to 1289603 tid 1 2024-09-11T03:41:34.907 INFO:tasks.rados.rados.2.smithi175.stdout:892: writing smithi17557870-9 from 2019164 to 2735596 tid 2 2024-09-11T03:41:34.907 INFO:tasks.rados.rados.2.smithi175.stdout:892: writing smithi17557870-9 from 3335341 to 3529503 tid 3 2024-09-11T03:41:34.907 INFO:tasks.rados.rados.2.smithi175.stdout:893: snap_remove snap 90 2024-09-11T03:41:34.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:34.928 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:34.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:35.029 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:35.050 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:35.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:35.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:35.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:35.173 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:35.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:35.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:35.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:35.296 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:35.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:35.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:35 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:41:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:35 smithi084 ceph-mon[27254]: 2.6 scrub starts 2024-09-11T03:41:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:35 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-11T03:41:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:35 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-11T03:41:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:35 smithi084 ceph-mon[27254]: osdmap e278: 12 total, 12 up, 10 in 2024-09-11T03:41:35.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:35.419 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:35.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:35.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:35.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:35 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:41:35.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:35 smithi121 ceph-mon[30729]: 2.6 scrub starts 2024-09-11T03:41:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:35 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-11T03:41:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:35 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-11T03:41:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:35 smithi121 ceph-mon[30729]: osdmap e278: 12 total, 12 up, 10 in 2024-09-11T03:41:35.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:35 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:41:35.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:35 smithi175 ceph-mon[30894]: 2.6 scrub starts 2024-09-11T03:41:35.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:35 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-11T03:41:35.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:35 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-11T03:41:35.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:35 smithi175 ceph-mon[30894]: osdmap e278: 12 total, 12 up, 10 in 2024-09-11T03:41:35.521 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:35.543 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:35.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:35.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:35.630 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T03:41:35.645 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:35.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:35.667 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:35.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:35.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:35.790 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:35.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:35.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:35.843 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:41:35.860 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:41:35.867 INFO:tasks.rados.rados.2.smithi175.stdout:892: finishing write tid 1 to smithi17557870-9 2024-09-11T03:41:35.867 INFO:tasks.rados.rados.2.smithi175.stdout:892: finishing write tid 2 to smithi17557870-9 2024-09-11T03:41:35.867 INFO:tasks.rados.rados.2.smithi175.stdout:892: finishing write tid 3 to smithi17557870-9 2024-09-11T03:41:35.867 INFO:tasks.rados.rados.2.smithi175.stdout:892: finishing write tid 4 to smithi17557870-9 2024-09-11T03:41:35.867 INFO:tasks.rados.rados.2.smithi175.stdout:892: finishing write tid 5 to smithi17557870-9 2024-09-11T03:41:35.868 INFO:tasks.rados.rados.2.smithi175.stdout:892: oid 9 updating version 0 to 320 2024-09-11T03:41:35.868 INFO:tasks.rados.rados.2.smithi175.stdout:892: oid 9 updating version 320 to 322 2024-09-11T03:41:35.868 INFO:tasks.rados.rados.2.smithi175.stdout:892: oid 9 updating version 322 to 323 2024-09-11T03:41:35.868 INFO:tasks.rados.rados.2.smithi175.stdout:892: oid 9 version 323 is already newer than 321 2024-09-11T03:41:35.868 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 323 (ObjNum 329 snap 92 seq_num 329) dirty exists 2024-09-11T03:41:35.868 INFO:tasks.rados.rados.2.smithi175.stdout:892: left oid 9 (ObjNum 329 snap 92 seq_num 329) 2024-09-11T03:41:35.868 INFO:tasks.rados.rados.2.smithi175.stdout:892: done (2 left) 2024-09-11T03:41:35.868 INFO:tasks.rados.rados.2.smithi175.stdout:893: done (1 left) 2024-09-11T03:41:35.868 INFO:tasks.rados.rados.2.smithi175.stdout:894: rollback oid 21 current snap is 92 2024-09-11T03:41:35.868 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 21 to 84 2024-09-11T03:41:35.868 INFO:tasks.rados.rados.2.smithi175.stdout:895: write (excl) oid 23 current snap is 92 2024-09-11T03:41:35.869 INFO:tasks.rados.rados.2.smithi175.stdout:895: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2024-09-11T03:41:35.875 INFO:tasks.rados.rados.2.smithi175.stdout:895: writing smithi17557870-23 from 746310 to 1381659 tid 1 2024-09-11T03:41:35.882 INFO:tasks.rados.rados.2.smithi175.stdout:895: writing smithi17557870-23 from 2077646 to 2654089 tid 2 2024-09-11T03:41:35.887 INFO:tasks.rados.rados.2.smithi175.stdout:895: writing smithi17557870-23 from 3362764 to 3546310 tid 3 2024-09-11T03:41:35.887 INFO:tasks.rados.rados.2.smithi175.stdout:896: write oid 42 current snap is 92 2024-09-11T03:41:35.887 INFO:tasks.rados.rados.2.smithi175.stdout:896: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2024-09-11T03:41:35.892 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:35.895 INFO:tasks.rados.rados.2.smithi175.stdout:896: writing smithi17557870-42 from 763117 to 1473715 tid 1 2024-09-11T03:41:35.902 INFO:tasks.rados.rados.2.smithi175.stdout:896: writing smithi17557870-42 from 2136128 to 2856229 tid 2 2024-09-11T03:41:35.907 INFO:tasks.rados.rados.2.smithi175.stdout:896: writing smithi17557870-42 from 3390187 to 3563117 tid 3 2024-09-11T03:41:35.907 INFO:tasks.rados.rados.2.smithi175.stdout:897: snap_remove snap 88 2024-09-11T03:41:35.915 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:41:35.916 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:35.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:35.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:36.018 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:36.041 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:36.042 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:36.143 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:36 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:41:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:36 smithi084 ceph-mon[27254]: 2.5 scrub starts 2024-09-11T03:41:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:36 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:41:36.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:36 smithi084 ceph-mon[27254]: pgmap v571: 57 pgs: 2 peering, 10 unknown, 45 active+clean; 141 MiB data, 613 MiB used, 893 GiB / 894 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 26 op/s 2024-09-11T03:41:36.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:36 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:41:36.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:36 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:41:36.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:36 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:41:36.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:36 smithi084 ceph-mon[27254]: 2.5 scrub starts 2024-09-11T03:41:36.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/716890206' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:36.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/716890206' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:36.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:36 smithi084 ceph-mon[27254]: osdmap e279: 12 total, 12 up, 10 in 2024-09-11T03:41:36.165 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:36.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:36.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:36.267 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:36.289 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:36.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:36.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:36 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:41:36.300 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:36 smithi121 ceph-mon[30729]: 2.5 scrub starts 2024-09-11T03:41:36.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:36 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:41:36.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:36 smithi121 ceph-mon[30729]: pgmap v571: 57 pgs: 2 peering, 10 unknown, 45 active+clean; 141 MiB data, 613 MiB used, 893 GiB / 894 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 26 op/s 2024-09-11T03:41:36.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:36 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:41:36.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:36 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:41:36.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:36 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:41:36.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:36 smithi121 ceph-mon[30729]: 2.5 scrub starts 2024-09-11T03:41:36.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/716890206' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:36.301 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/716890206' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:36.302 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:36 smithi121 ceph-mon[30729]: osdmap e279: 12 total, 12 up, 10 in 2024-09-11T03:41:36.390 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:36.400 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:41:36.401 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:41:36.401 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:41:36.401 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:41:36.413 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:36.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:36.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:36 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:41:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:36 smithi175 ceph-mon[30894]: 2.5 scrub starts 2024-09-11T03:41:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:36 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:41:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:36 smithi175 ceph-mon[30894]: pgmap v571: 57 pgs: 2 peering, 10 unknown, 45 active+clean; 141 MiB data, 613 MiB used, 893 GiB / 894 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 26 op/s 2024-09-11T03:41:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:36 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:41:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:36 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:41:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:36 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:41:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:36 smithi175 ceph-mon[30894]: 2.5 scrub starts 2024-09-11T03:41:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/716890206' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/716890206' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:36 smithi175 ceph-mon[30894]: osdmap e279: 12 total, 12 up, 10 in 2024-09-11T03:41:36.514 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:36.540 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:36.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:36.541 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:36.564 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:36.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:36.663 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:36.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:36.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:36.676 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [6, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:41:36.677 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:41:36.677 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-11T03:41:36.677 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:41:36.765 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:36.786 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:36.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:36.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:36.884 INFO:tasks.rados.rados.2.smithi175.stdout:897: done (4 left) 2024-09-11T03:41:36.884 INFO:tasks.rados.rados.2.smithi175.stdout:898: read oid 37 snap -1 2024-09-11T03:41:36.884 INFO:tasks.rados.rados.2.smithi175.stdout:898: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:41:36.885 INFO:tasks.rados.rados.2.smithi175.stdout:899: write oid 20 current snap is 92 2024-09-11T03:41:36.885 INFO:tasks.rados.rados.2.smithi175.stdout:899: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2024-09-11T03:41:36.887 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:36.893 INFO:tasks.rados.rados.2.smithi175.stdout:899: writing smithi17557870-20 from 779924 to 1565771 tid 1 2024-09-11T03:41:36.900 INFO:tasks.rados.rados.2.smithi175.stdout:899: writing smithi17557870-20 from 2078257 to 2658369 tid 2 2024-09-11T03:41:36.906 INFO:tasks.rados.rados.2.smithi175.stdout:899: writing smithi17557870-20 from 3301257 to 3579924 tid 3 2024-09-11T03:41:36.906 INFO:tasks.rados.rados.2.smithi175.stdout:900: write (excl) oid 30 current snap is 92 2024-09-11T03:41:36.906 INFO:tasks.rados.rados.2.smithi175.stdout:900: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2024-09-11T03:41:36.914 INFO:tasks.rados.rados.2.smithi175.stdout:900: writing smithi17557870-30 from 796731 to 1257827 tid 1 2024-09-11T03:41:36.922 INFO:tasks.rados.rados.2.smithi175.stdout:900: writing smithi17557870-30 from 1736739 to 2460509 tid 2 2024-09-11T03:41:36.926 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:36.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:36.928 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:36.929 INFO:tasks.rados.rados.2.smithi175.stdout:900: writing smithi17557870-30 from 2928680 to 3596731 tid 3 2024-09-11T03:41:36.929 INFO:tasks.rados.rados.2.smithi175.stdout:901: snap_create 2024-09-11T03:41:36.929 INFO:tasks.rados.rados.2.smithi175.stdout:894: finishing rollback tid 0 to smithi17557870-21 2024-09-11T03:41:36.929 INFO:tasks.rados.rados.2.smithi175.stdout:894: finishing rollback tid 1 to smithi17557870-21 2024-09-11T03:41:36.929 INFO:tasks.rados.rados.2.smithi175.stdout:894: finishing rollback tid 2 to smithi17557870-21 2024-09-11T03:41:36.929 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 327 (ObjNum 266 snap 67 seq_num 266) dirty exists 2024-09-11T03:41:36.929 INFO:tasks.rados.rados.2.smithi175.stdout:896: finishing write tid 1 to smithi17557870-42 2024-09-11T03:41:36.930 INFO:tasks.rados.rados.2.smithi175.stdout:896: finishing write tid 2 to smithi17557870-42 2024-09-11T03:41:36.930 INFO:tasks.rados.rados.2.smithi175.stdout:896: finishing write tid 3 to smithi17557870-42 2024-09-11T03:41:36.930 INFO:tasks.rados.rados.2.smithi175.stdout:896: finishing write tid 4 to smithi17557870-42 2024-09-11T03:41:36.930 INFO:tasks.rados.rados.2.smithi175.stdout:896: finishing write tid 5 to smithi17557870-42 2024-09-11T03:41:36.930 INFO:tasks.rados.rados.2.smithi175.stdout:896: oid 42 updating version 0 to 230 2024-09-11T03:41:36.930 INFO:tasks.rados.rados.2.smithi175.stdout:896: oid 42 updating version 230 to 233 2024-09-11T03:41:36.930 INFO:tasks.rados.rados.2.smithi175.stdout:896: oid 42 version 233 is already newer than 232 2024-09-11T03:41:36.930 INFO:tasks.rados.rados.2.smithi175.stdout:896: oid 42 version 233 is already newer than 231 2024-09-11T03:41:36.930 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 233 (ObjNum 331 snap 92 seq_num 331) dirty exists 2024-09-11T03:41:36.930 INFO:tasks.rados.rados.2.smithi175.stdout:896: left oid 42 (ObjNum 331 snap 92 seq_num 331) 2024-09-11T03:41:36.930 INFO:tasks.rados.rados.2.smithi175.stdout:898: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:41:37.028 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:37.050 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:37.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:37.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:37.053 INFO:tasks.rados.rados.2.smithi175.stdout:899: finishing write tid 1 to smithi17557870-20 2024-09-11T03:41:37.053 INFO:tasks.rados.rados.2.smithi175.stdout:895: finishing write tid 1 to smithi17557870-23 2024-09-11T03:41:37.053 INFO:tasks.rados.rados.2.smithi175.stdout:895: finishing write tid 2 to smithi17557870-23 2024-09-11T03:41:37.053 INFO:tasks.rados.rados.2.smithi175.stdout:899: finishing write tid 2 to smithi17557870-20 2024-09-11T03:41:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:895: finishing write tid 3 to smithi17557870-23 2024-09-11T03:41:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:895: finishing write tid 4 to smithi17557870-23 2024-09-11T03:41:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:895: finishing write tid 5 to smithi17557870-23 2024-09-11T03:41:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:895: oid 23 updating version 0 to 195 2024-09-11T03:41:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:895: oid 23 version 195 is already newer than 194 2024-09-11T03:41:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:895: oid 23 updating version 195 to 196 2024-09-11T03:41:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:895: oid 23 version 196 is already newer than 193 2024-09-11T03:41:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 196 (ObjNum 330 snap 92 seq_num 330) dirty exists 2024-09-11T03:41:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:895: left oid 23 (ObjNum 330 snap 92 seq_num 330) 2024-09-11T03:41:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:899: finishing write tid 3 to smithi17557870-20 2024-09-11T03:41:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:899: finishing write tid 4 to smithi17557870-20 2024-09-11T03:41:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:899: finishing write tid 5 to smithi17557870-20 2024-09-11T03:41:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:899: oid 20 updating version 0 to 311 2024-09-11T03:41:37.055 INFO:tasks.rados.rados.2.smithi175.stdout:899: oid 20 updating version 311 to 313 2024-09-11T03:41:37.055 INFO:tasks.rados.rados.2.smithi175.stdout:899: oid 20 version 313 is already newer than 310 2024-09-11T03:41:37.055 INFO:tasks.rados.rados.2.smithi175.stdout:899: oid 20 version 313 is already newer than 312 2024-09-11T03:41:37.055 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 313 (ObjNum 332 snap 92 seq_num 332) dirty exists 2024-09-11T03:41:37.055 INFO:tasks.rados.rados.2.smithi175.stdout:899: left oid 20 (ObjNum 332 snap 92 seq_num 332) 2024-09-11T03:41:37.055 INFO:tasks.rados.rados.2.smithi175.stdout:890: expect (ObjNum 317 snap 86 seq_num 317) 2024-09-11T03:41:37.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T03:41:37.085 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:41:37.085 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:41:37.093 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:41:37.093 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:41:37.108 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:41:37.108 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:41:37.124 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:41:37.124 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:41:37.141 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:41:37.142 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:41:37.152 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:37.160 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:41:37.160 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:41:37.176 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:37.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:37.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:37.179 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:41:37.179 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:41:37.196 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:37.197 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:41:37.198 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:41:37.215 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:41:37.215 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:41:37.232 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:41:37.233 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:41:37.233 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:37 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:41:37.233 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:37 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:41:37.233 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:37 smithi084 ceph-mon[27254]: 2.3 scrub starts 2024-09-11T03:41:37.233 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:37 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:41:37.233 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:37 smithi084 ceph-mon[27254]: 2.5 scrub starts 2024-09-11T03:41:37.233 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:37 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:41:37.233 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:37 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:41:37.234 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:37 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:37.234 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:37 smithi084 ceph-mon[27254]: osdmap e280: 12 total, 12 up, 10 in 2024-09-11T03:41:37.249 INFO:tasks.rados.rados.2.smithi175.stdout:900: finishing write tid 1 to smithi17557870-30 2024-09-11T03:41:37.249 INFO:tasks.rados.rados.2.smithi175.stdout:900: finishing write tid 2 to smithi17557870-30 2024-09-11T03:41:37.249 INFO:tasks.rados.rados.2.smithi175.stdout:900: finishing write tid 3 to smithi17557870-30 2024-09-11T03:41:37.249 INFO:tasks.rados.rados.2.smithi175.stdout:900: finishing write tid 4 to smithi17557870-30 2024-09-11T03:41:37.249 INFO:tasks.rados.rados.2.smithi175.stdout:900: finishing write tid 5 to smithi17557870-30 2024-09-11T03:41:37.249 INFO:tasks.rados.rados.2.smithi175.stdout:900: oid 30 updating version 0 to 318 2024-09-11T03:41:37.250 INFO:tasks.rados.rados.2.smithi175.stdout:900: oid 30 version 318 is already newer than 316 2024-09-11T03:41:37.250 INFO:tasks.rados.rados.2.smithi175.stdout:900: oid 30 updating version 318 to 319 2024-09-11T03:41:37.250 INFO:tasks.rados.rados.2.smithi175.stdout:900: oid 30 version 319 is already newer than 317 2024-09-11T03:41:37.250 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 319 (ObjNum 333 snap 92 seq_num 333) dirty exists 2024-09-11T03:41:37.250 INFO:tasks.rados.rados.2.smithi175.stdout:900: left oid 30 (ObjNum 333 snap 92 seq_num 333) 2024-09-11T03:41:37.254 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:41:37.254 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:41:37.271 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:41:37.272 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:41:37.278 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:37.300 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:37.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:37.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:37.302 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:41:37.403 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:37.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:37 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:41:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:37 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:41:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:37 smithi121 ceph-mon[30729]: 2.3 scrub starts 2024-09-11T03:41:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:37 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:41:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:37 smithi121 ceph-mon[30729]: 2.5 scrub starts 2024-09-11T03:41:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:37 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:41:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:37 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:41:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:37 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:37 smithi121 ceph-mon[30729]: osdmap e280: 12 total, 12 up, 10 in 2024-09-11T03:41:37.424 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:37.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:37.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:37 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:41:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:37 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:41:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:37 smithi175 ceph-mon[30894]: 2.3 scrub starts 2024-09-11T03:41:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:37 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:41:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:37 smithi175 ceph-mon[30894]: 2.5 scrub starts 2024-09-11T03:41:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:37 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:41:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:37 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:41:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:37 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:37 smithi175 ceph-mon[30894]: osdmap e280: 12 total, 12 up, 10 in 2024-09-11T03:41:37.525 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:37.548 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:37.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:37.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:37.650 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:37.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:37.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:37.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:37.781 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:41:37.781 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:41:37.795 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:37.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:37.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:37.882 INFO:tasks.rados.rados.2.smithi175.stdout:890: done (7 left) 2024-09-11T03:41:37.882 INFO:tasks.rados.rados.2.smithi175.stdout:894: done (6 left) 2024-09-11T03:41:37.883 INFO:tasks.rados.rados.2.smithi175.stdout:895: done (5 left) 2024-09-11T03:41:37.883 INFO:tasks.rados.rados.2.smithi175.stdout:896: done (4 left) 2024-09-11T03:41:37.883 INFO:tasks.rados.rados.2.smithi175.stdout:898: done (3 left) 2024-09-11T03:41:37.883 INFO:tasks.rados.rados.2.smithi175.stdout:899: done (2 left) 2024-09-11T03:41:37.883 INFO:tasks.rados.rados.2.smithi175.stdout:900: done (1 left) 2024-09-11T03:41:37.883 INFO:tasks.rados.rados.2.smithi175.stdout:901: done (0 left) 2024-09-11T03:41:37.883 INFO:tasks.rados.rados.2.smithi175.stdout:902: snap_create 2024-09-11T03:41:37.896 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:37.919 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:37.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:37.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:38.020 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:38.046 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:38.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:38.047 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:38.085 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:41:38.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:38 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:41:38.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:38 smithi084 ceph-mon[27254]: pgmap v574: 57 pgs: 4 active+clean+scrubbing, 2 peering, 5 remapped+peering, 4 unknown, 42 active+clean; 130 MiB data, 610 MiB used, 893 GiB / 894 GiB avail 2024-09-11T03:41:38.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:38 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:41:38.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:38 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3606311439' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:38.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:38 smithi084 ceph-mon[27254]: 2.4 scrub starts 2024-09-11T03:41:38.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:38 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:41:38.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:38 smithi084 ceph-mon[27254]: 2.10 scrub starts 2024-09-11T03:41:38.086 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:38 smithi084 ceph-mon[27254]: osdmap e281: 12 total, 12 up, 10 in 2024-09-11T03:41:38.087 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:41:38.148 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:38.169 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:38.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:38.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:38.271 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:38.293 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:38.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:38.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:38.395 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:38.418 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:38.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:38.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:38.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:38 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:41:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:38 smithi121 ceph-mon[30729]: pgmap v574: 57 pgs: 4 active+clean+scrubbing, 2 peering, 5 remapped+peering, 4 unknown, 42 active+clean; 130 MiB data, 610 MiB used, 893 GiB / 894 GiB avail 2024-09-11T03:41:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:38 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:41:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:38 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3606311439' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:38 smithi121 ceph-mon[30729]: 2.4 scrub starts 2024-09-11T03:41:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:38 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:41:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:38 smithi121 ceph-mon[30729]: 2.10 scrub starts 2024-09-11T03:41:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:38 smithi121 ceph-mon[30729]: osdmap e281: 12 total, 12 up, 10 in 2024-09-11T03:41:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:38 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:41:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:38 smithi175 ceph-mon[30894]: pgmap v574: 57 pgs: 4 active+clean+scrubbing, 2 peering, 5 remapped+peering, 4 unknown, 42 active+clean; 130 MiB data, 610 MiB used, 893 GiB / 894 GiB avail 2024-09-11T03:41:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:38 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:41:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:38 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3606311439' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:38.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:38 smithi175 ceph-mon[30894]: 2.4 scrub starts 2024-09-11T03:41:38.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:38 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:41:38.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:38 smithi175 ceph-mon[30894]: 2.10 scrub starts 2024-09-11T03:41:38.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:38 smithi175 ceph-mon[30894]: osdmap e281: 12 total, 12 up, 10 in 2024-09-11T03:41:38.491 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:41:38.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T03:41:38.520 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:38.557 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:38.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:38.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:38.659 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:38.685 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:38.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:38.686 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:38.788 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:38.794 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:41:38.809 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:38.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:38.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:38.884 INFO:tasks.rados.rados.2.smithi175.stdout:902: done (0 left) 2024-09-11T03:41:38.885 INFO:tasks.rados.rados.2.smithi175.stdout:903: snap_create 2024-09-11T03:41:38.910 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:38.931 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:38.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:38.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:39.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:39.056 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:39.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:39.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:39.101 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":576,"stamp":"2024-09-11T03:41:37.879049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":139453286,"num_objects":60,"num_object_clones":20,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":3,"num_read":1394,"num_read_kb":1119456,"num_write":1185,"num_write_kb":494330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":13624269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7268,"ondisk_log_size":7268,"up":162,"acting":157,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":626896,"kb_used_data":298336,"kb_used_omap":15,"kb_used_meta":328304,"kb_avail":936824624,"statfs":{"total":959950356480,"available":959308414976,"internally_reserved":0,"allocated":305496064,"data_stored":331736321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":336183733},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7897129,"num_objects":-2,"num_object_clones":2,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":2,"num_read":-200,"num_read_kb":-191042,"num_write":-126,"num_write_kb":-45532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016560"},"pg_stats":[{"pgid":"1.1e","version":"20'92","reported_seq":134,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-11T03:41:36.888074+0000","last_change":"2024-09-11T03:40:40.236609+0000","last_active":"2024-09-11T03:41:36.888074+0000","last_peered":"2024-09-11T03:41:36.888074+0000","last_clean":"2024-09-11T03:41:36.888074+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:41:36.888074+0000","last_undegraded":"2024-09-11T03:41:36.888074+0000","last_fullsized":"2024-09-11T03:41:36.888074+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.236546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.236546+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:03.367532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'92","reported_seq":134,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.870882+0000","last_change":"2024-09-11T03:40:41.348889+0000","last_active":"2024-09-11T03:41:35.870882+0000","last_peered":"2024-09-11T03:41:35.870882+0000","last_clean":"2024-09-11T03:41:35.870882+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:41:35.870882+0000","last_undegraded":"2024-09-11T03:41:35.870882+0000","last_fullsized":"2024-09-11T03:41:35.870882+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:41.348828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:41.348828+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:09.856306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'92","reported_seq":132,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.863691+0000","last_change":"2024-09-11T03:40:40.188174+0000","last_active":"2024-09-11T03:41:35.863691+0000","last_peered":"2024-09-11T03:41:35.863691+0000","last_clean":"2024-09-11T03:41:35.863691+0000","last_became_active":"2024-09-11T03:40:39.848230+0000","last_became_peered":"2024-09-11T03:40:39.848230+0000","last_unstale":"2024-09-11T03:41:35.863691+0000","last_undegraded":"2024-09-11T03:41:35.863691+0000","last_fullsized":"2024-09-11T03:41:35.863691+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.188124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.188124+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:05.549148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T03:41:32.853860+0000","last_change":"2024-09-11T03:41:32.853860+0000","last_active":"2024-09-11T03:41:32.853860+0000","last_peered":"2024-09-11T03:41:32.853860+0000","last_clean":"2024-09-11T03:41:32.853860+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T03:41:32.853860+0000","last_undegraded":"2024-09-11T03:41:32.853860+0000","last_fullsized":"2024-09-11T03:41:32.853860+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:41:32.853860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:41:32.853860+0000","last_clean_scrub_stamp":"2024-09-11T03:41:32.853860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1b","version":"20'92","reported_seq":132,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.864046+0000","last_change":"2024-09-11T03:40:42.166354+0000","last_active":"2024-09-11T03:41:35.864046+0000","last_peered":"2024-09-11T03:41:35.864046+0000","last_clean":"2024-09-11T03:41:35.864046+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:41:35.864046+0000","last_undegraded":"2024-09-11T03:41:35.864046+0000","last_fullsized":"2024-09-11T03:41:35.864046+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:42.166295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:42.166295+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:10.918752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T03:41:32.853860+0000","last_change":"2024-09-11T03:41:32.853860+0000","last_active":"2024-09-11T03:41:32.853860+0000","last_peered":"2024-09-11T03:41:32.853860+0000","last_clean":"2024-09-11T03:41:32.853860+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T03:41:32.853860+0000","last_undegraded":"2024-09-11T03:41:32.853860+0000","last_fullsized":"2024-09-11T03:41:32.853860+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:41:32.853860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:41:32.853860+0000","last_clean_scrub_stamp":"2024-09-11T03:41:32.853860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1a","version":"20'92","reported_seq":373,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-11T03:41:34.863310+0000","last_change":"2024-09-11T03:40:40.278752+0000","last_active":"2024-09-11T03:41:34.863310+0000","last_peered":"2024-09-11T03:41:34.863310+0000","last_clean":"2024-09-11T03:41:34.863310+0000","last_became_active":"2024-09-11T03:40:39.848781+0000","last_became_peered":"2024-09-11T03:40:39.848781+0000","last_unstale":"2024-09-11T03:41:34.863310+0000","last_undegraded":"2024-09-11T03:41:34.863310+0000","last_fullsized":"2024-09-11T03:41:34.863310+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.278703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.278703+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:06.105420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"20'92","reported_seq":132,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.871085+0000","last_change":"2024-09-11T03:40:40.718553+0000","last_active":"2024-09-11T03:41:35.871085+0000","last_peered":"2024-09-11T03:41:35.871085+0000","last_clean":"2024-09-11T03:41:35.871085+0000","last_became_active":"2024-09-11T03:40:39.851698+0000","last_became_peered":"2024-09-11T03:40:39.851698+0000","last_unstale":"2024-09-11T03:41:35.871085+0000","last_undegraded":"2024-09-11T03:41:35.871085+0000","last_fullsized":"2024-09-11T03:41:35.871085+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.718503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.718503+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:06.580046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.18","version":"20'92","reported_seq":247,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.867505+0000","last_change":"2024-09-11T03:40:40.365656+0000","last_active":"2024-09-11T03:41:35.867505+0000","last_peered":"2024-09-11T03:41:35.867505+0000","last_clean":"2024-09-11T03:41:35.867505+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:41:35.867505+0000","last_undegraded":"2024-09-11T03:41:35.867505+0000","last_fullsized":"2024-09-11T03:41:35.867505+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.365604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.365604+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:00.211289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"274'162","reported_seq":1094,"reported_epoch":279,"state":"remapped+peering","last_fresh":"2024-09-11T03:41:35.873961+0000","last_change":"2024-09-11T03:41:35.873961+0000","last_active":"2024-09-11T03:41:35.873961+0000","last_peered":"2024-09-11T03:41:31.036278+0000","last_clean":"2024-09-11T03:41:31.036278+0000","last_became_active":"2024-09-11T03:40:05.966715+0000","last_became_peered":"2024-09-11T03:40:05.966715+0000","last_unstale":"2024-09-11T03:41:35.873961+0000","last_undegraded":"2024-09-11T03:41:35.873961+0000","last_fullsized":"2024-09-11T03:41:35.873961+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":5,"last_scrub":"241'130","last_scrub_stamp":"2024-09-11T03:41:03.349362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:03.349362+0000","objects_scrubbed":2,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4790045,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":15822,"num_write":39,"num_write_kb":14168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3595434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":0,"purged_snaps":[{"start":"51","length":"1"},{"start":"5b","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":132,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.863779+0000","last_change":"2024-09-11T03:40:41.207341+0000","last_active":"2024-09-11T03:41:35.863779+0000","last_peered":"2024-09-11T03:41:35.863779+0000","last_clean":"2024-09-11T03:41:35.863779+0000","last_became_active":"2024-09-11T03:40:39.850243+0000","last_became_peered":"2024-09-11T03:40:39.850243+0000","last_unstale":"2024-09-11T03:41:35.863779+0000","last_undegraded":"2024-09-11T03:41:35.863779+0000","last_fullsized":"2024-09-11T03:41:35.863779+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:41.207291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:41.207291+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:56.540051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":1225,"reported_epoch":279,"state":"remapped+peering","last_fresh":"2024-09-11T03:41:35.872331+0000","last_change":"2024-09-11T03:41:35.872331+0000","last_active":"2024-09-11T03:41:35.872331+0000","last_peered":"2024-09-11T03:41:31.029234+0000","last_clean":"2024-09-11T03:41:31.029234+0000","last_became_active":"2024-09-11T03:38:30.163080+0000","last_became_peered":"2024-09-11T03:38:30.163080+0000","last_unstale":"2024-09-11T03:41:35.872331+0000","last_undegraded":"2024-09-11T03:41:35.872331+0000","last_fullsized":"2024-09-11T03:41:35.872331+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":5,"last_scrub":"226'209","last_scrub_stamp":"2024-09-11T03:40:45.175807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:45.175807+0000","objects_scrubbed":8,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9148157,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":103,"num_read_kb":97203,"num_write":64,"num_write_kb":22379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":0,"purged_snaps":[{"start":"51","length":"1"},{"start":"5b","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":132,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.872824+0000","last_change":"2024-09-11T03:40:40.309475+0000","last_active":"2024-09-11T03:41:35.872824+0000","last_peered":"2024-09-11T03:41:35.872824+0000","last_clean":"2024-09-11T03:41:35.872824+0000","last_became_active":"2024-09-11T03:40:39.849950+0000","last_became_peered":"2024-09-11T03:40:39.849950+0000","last_unstale":"2024-09-11T03:41:35.872824+0000","last_undegraded":"2024-09-11T03:41:35.872824+0000","last_fullsized":"2024-09-11T03:41:35.872824+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.309432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.309432+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:04.484327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"269'171","reported_seq":1081,"reported_epoch":279,"state":"remapped+peering","last_fresh":"2024-09-11T03:41:35.870255+0000","last_change":"2024-09-11T03:41:35.870255+0000","last_active":"2024-09-11T03:41:35.870255+0000","last_peered":"2024-09-11T03:41:33.873665+0000","last_clean":"2024-09-11T03:41:33.873665+0000","last_became_active":"2024-09-11T03:41:33.871608+0000","last_became_peered":"2024-09-11T03:41:33.871608+0000","last_unstale":"2024-09-11T03:41:35.870255+0000","last_undegraded":"2024-09-11T03:41:35.870255+0000","last_fullsized":"2024-09-11T03:41:35.870255+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":5,"last_scrub":"198'103","last_scrub_stamp":"2024-09-11T03:40:17.171458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:17.171458+0000","objects_scrubbed":4,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001361847,"stat_sum":{"num_bytes":1479033,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":31,"num_read_kb":29175,"num_write":31,"num_write_kb":15335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,9],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[{"start":"51","length":"1"},{"start":"5b","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":132,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.866088+0000","last_change":"2024-09-11T03:40:39.928143+0000","last_active":"2024-09-11T03:41:35.866088+0000","last_peered":"2024-09-11T03:41:35.866088+0000","last_clean":"2024-09-11T03:41:35.866088+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:41:35.866088+0000","last_undegraded":"2024-09-11T03:41:35.866088+0000","last_fullsized":"2024-09-11T03:41:35.866088+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.928088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.928088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:06.193518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":1286,"reported_epoch":275,"state":"unknown","last_fresh":"2024-09-11T03:41:32.030552+0000","last_change":"2024-09-11T03:41:33.062686+0000","last_active":"2024-09-11T03:41:33.062686+0000","last_peered":"2024-09-11T03:41:32.030552+0000","last_clean":"2024-09-11T03:41:32.030552+0000","last_became_active":"2024-09-11T03:40:08.836627+0000","last_became_peered":"2024-09-11T03:40:08.836627+0000","last_unstale":"2024-09-11T03:41:32.030552+0000","last_undegraded":"2024-09-11T03:41:32.030552+0000","last_fullsized":"2024-09-11T03:41:32.030552+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":5,"last_scrub":"241'242","last_scrub_stamp":"2024-09-11T03:40:59.291412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:59.291412+0000","objects_scrubbed":9,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10375123,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":67,"num_read_kb":44307,"num_write":75,"num_write_kb":27773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3216701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"51","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":156,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-11T03:41:32.859206+0000","last_change":"2024-09-11T03:40:35.195938+0000","last_active":"2024-09-11T03:41:32.859206+0000","last_peered":"2024-09-11T03:41:32.859206+0000","last_clean":"2024-09-11T03:41:32.859206+0000","last_became_active":"2024-09-11T03:40:27.847297+0000","last_became_peered":"2024-09-11T03:40:27.847297+0000","last_unstale":"2024-09-11T03:41:32.859206+0000","last_undegraded":"2024-09-11T03:41:32.859206+0000","last_fullsized":"2024-09-11T03:41:32.859206+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.195893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.195893+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:43.112855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":20,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.875941+0000","last_change":"2024-09-11T03:41:35.875804+0000","last_active":"2024-09-11T03:41:35.875941+0000","last_peered":"2024-09-11T03:41:35.875941+0000","last_clean":"2024-09-11T03:41:35.875941+0000","last_became_active":"2024-09-11T03:41:35.874295+0000","last_became_peered":"2024-09-11T03:41:35.874295+0000","last_unstale":"2024-09-11T03:41:35.875941+0000","last_undegraded":"2024-09-11T03:41:35.875941+0000","last_fullsized":"2024-09-11T03:41:35.875941+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"241'33","last_scrub_stamp":"2024-09-11T03:40:58.014124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:58.014124+0000","objects_scrubbed":2,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010383040000000001,"stat_sum":{"num_bytes":1949712,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":18300,"num_write":14,"num_write_kb":5787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"51","length":"1"},{"start":"57","length":"1"},{"start":"5b","length":"2"}]},{"pgid":"1.13","version":"20'92","reported_seq":241,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.870516+0000","last_change":"2024-09-11T03:40:35.277715+0000","last_active":"2024-09-11T03:41:35.870516+0000","last_peered":"2024-09-11T03:41:35.870516+0000","last_clean":"2024-09-11T03:41:35.870516+0000","last_became_active":"2024-09-11T03:40:27.847557+0000","last_became_peered":"2024-09-11T03:40:27.847557+0000","last_unstale":"2024-09-11T03:41:35.870516+0000","last_undegraded":"2024-09-11T03:41:35.870516+0000","last_fullsized":"2024-09-11T03:41:35.870516+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.277668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.277668+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:39.014977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":1009,"reported_epoch":279,"state":"remapped+peering","last_fresh":"2024-09-11T03:41:35.868397+0000","last_change":"2024-09-11T03:41:35.868397+0000","last_active":"2024-09-11T03:41:35.868397+0000","last_peered":"2024-09-11T03:41:31.013591+0000","last_clean":"2024-09-11T03:41:31.013591+0000","last_became_active":"2024-09-11T03:38:30.162952+0000","last_became_peered":"2024-09-11T03:38:30.162952+0000","last_unstale":"2024-09-11T03:41:35.868397+0000","last_undegraded":"2024-09-11T03:41:35.868397+0000","last_fullsized":"2024-09-11T03:41:35.868397+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":5,"last_scrub":"241'87","last_scrub_stamp":"2024-09-11T03:40:58.304220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:58.304220+0000","objects_scrubbed":3,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1537857,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":12941,"num_write":17,"num_write_kb":10273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,9],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[{"start":"51","length":"1"},{"start":"5b","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":160,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.868168+0000","last_change":"2024-09-11T03:40:37.284871+0000","last_active":"2024-09-11T03:41:35.868168+0000","last_peered":"2024-09-11T03:41:35.868168+0000","last_clean":"2024-09-11T03:41:35.868168+0000","last_became_active":"2024-09-11T03:40:27.845249+0000","last_became_peered":"2024-09-11T03:40:27.845249+0000","last_unstale":"2024-09-11T03:41:35.868168+0000","last_undegraded":"2024-09-11T03:41:35.868168+0000","last_fullsized":"2024-09-11T03:41:35.868168+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:37.284800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:37.284800+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:53.535974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"269'123","reported_seq":1018,"reported_epoch":279,"state":"remapped+peering","last_fresh":"2024-09-11T03:41:35.870097+0000","last_change":"2024-09-11T03:41:35.870097+0000","last_active":"2024-09-11T03:41:35.870097+0000","last_peered":"2024-09-11T03:41:25.987035+0000","last_clean":"2024-09-11T03:41:25.987035+0000","last_became_active":"2024-09-11T03:39:26.115670+0000","last_became_peered":"2024-09-11T03:39:26.115670+0000","last_unstale":"2024-09-11T03:41:35.870097+0000","last_undegraded":"2024-09-11T03:41:35.870097+0000","last_fullsized":"2024-09-11T03:41:35.870097+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":5,"last_scrub":"241'111","last_scrub_stamp":"2024-09-11T03:41:04.281773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:04.281773+0000","objects_scrubbed":5,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1285736,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":4,"num_write":24,"num_write_kb":13223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,11],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[{"start":"51","length":"1"},{"start":"5b","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":164,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.871261+0000","last_change":"2024-09-11T03:40:35.703292+0000","last_active":"2024-09-11T03:41:35.871261+0000","last_peered":"2024-09-11T03:41:35.871261+0000","last_clean":"2024-09-11T03:41:35.871261+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:41:35.871261+0000","last_undegraded":"2024-09-11T03:41:35.871261+0000","last_fullsized":"2024-09-11T03:41:35.871261+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.703203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.703203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:49.443821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T03:41:32.853860+0000","last_change":"2024-09-11T03:41:32.853860+0000","last_active":"2024-09-11T03:41:32.853860+0000","last_peered":"2024-09-11T03:41:32.853860+0000","last_clean":"2024-09-11T03:41:32.853860+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T03:41:32.853860+0000","last_undegraded":"2024-09-11T03:41:32.853860+0000","last_fullsized":"2024-09-11T03:41:32.853860+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:41:32.853860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:41:32.853860+0000","last_clean_scrub_stamp":"2024-09-11T03:41:32.853860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.10","version":"20'92","reported_seq":160,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.867527+0000","last_change":"2024-09-11T03:40:36.320692+0000","last_active":"2024-09-11T03:41:35.867527+0000","last_peered":"2024-09-11T03:41:35.867527+0000","last_clean":"2024-09-11T03:41:35.867527+0000","last_became_active":"2024-09-11T03:40:27.844057+0000","last_became_peered":"2024-09-11T03:40:27.844057+0000","last_unstale":"2024-09-11T03:41:35.867527+0000","last_undegraded":"2024-09-11T03:41:35.867527+0000","last_fullsized":"2024-09-11T03:41:35.867527+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:36.320646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:36.320646+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:02.386305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"279'267","reported_seq":1177,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.868997+0000","last_change":"2024-09-11T03:41:35.869300+0000","last_active":"2024-09-11T03:41:35.868997+0000","last_peered":"2024-09-11T03:41:35.868997+0000","last_clean":"2024-09-11T03:41:35.868997+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:41:35.868997+0000","last_undegraded":"2024-09-11T03:41:35.868997+0000","last_fullsized":"2024-09-11T03:41:35.868997+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'232","last_scrub_stamp":"2024-09-11T03:41:04.001653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:04.001653+0000","objects_scrubbed":10,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:16.706316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.004746658,"stat_sum":{"num_bytes":13439507,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":50,"num_read_kb":15679,"num_write":97,"num_write_kb":49978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"51","length":"1"},{"start":"57","length":"1"},{"start":"5b","length":"2"}]},{"pgid":"1.8","version":"20'92","reported_seq":236,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.867622+0000","last_change":"2024-09-11T03:40:38.319593+0000","last_active":"2024-09-11T03:41:35.867622+0000","last_peered":"2024-09-11T03:41:35.867622+0000","last_clean":"2024-09-11T03:41:35.867622+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:41:35.867622+0000","last_undegraded":"2024-09-11T03:41:35.867622+0000","last_fullsized":"2024-09-11T03:41:35.867622+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.319543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.319543+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:03.765057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"280'84","reported_seq":1022,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-11T03:41:36.893492+0000","last_change":"2024-09-11T03:41:36.893892+0000","last_active":"2024-09-11T03:41:36.893492+0000","last_peered":"2024-09-11T03:41:36.893492+0000","last_clean":"2024-09-11T03:41:36.893492+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:41:36.893492+0000","last_undegraded":"2024-09-11T03:41:36.893492+0000","last_fullsized":"2024-09-11T03:41:36.893492+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'57","last_scrub_stamp":"2024-09-11T03:40:59.151899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:59.151899+0000","objects_scrubbed":1,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:00.636905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0062467959999999998,"stat_sum":{"num_bytes":5462341,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":65,"num_read_kb":68050,"num_write":30,"num_write_kb":15227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"51","length":"1"},{"start":"57","length":"1"},{"start":"5a","length":"3"}]},{"pgid":"1.9","version":"20'92","reported_seq":561,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.866338+0000","last_change":"2024-09-11T03:40:37.885659+0000","last_active":"2024-09-11T03:41:35.866338+0000","last_peered":"2024-09-11T03:41:35.866338+0000","last_clean":"2024-09-11T03:41:35.866338+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:41:35.866338+0000","last_undegraded":"2024-09-11T03:41:35.866338+0000","last_fullsized":"2024-09-11T03:41:35.866338+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:37.885602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:37.885602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:59.955114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"274'333","reported_seq":1326,"reported_epoch":279,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T03:41:36.124016+0000","last_change":"2024-09-11T03:41:36.123863+0000","last_active":"2024-09-11T03:41:36.124016+0000","last_peered":"2024-09-11T03:41:36.124016+0000","last_clean":"2024-09-11T03:41:36.124016+0000","last_became_active":"2024-09-11T03:41:33.865638+0000","last_became_peered":"2024-09-11T03:41:33.865638+0000","last_unstale":"2024-09-11T03:41:36.124016+0000","last_undegraded":"2024-09-11T03:41:36.124016+0000","last_fullsized":"2024-09-11T03:41:36.124016+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'311","last_scrub_stamp":"2024-09-11T03:41:20.086889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:20.086889+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0031433540000000001,"stat_sum":{"num_bytes":7630444,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":120172,"num_write":76,"num_write_kb":29680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"51","length":"1"},{"start":"57","length":"1"},{"start":"5b","length":"2"}]},{"pgid":"1.a","version":"20'92","reported_seq":467,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-11T03:41:32.864286+0000","last_change":"2024-09-11T03:40:39.331499+0000","last_active":"2024-09-11T03:41:32.864286+0000","last_peered":"2024-09-11T03:41:32.864286+0000","last_clean":"2024-09-11T03:41:32.864286+0000","last_became_active":"2024-09-11T03:40:37.849988+0000","last_became_peered":"2024-09-11T03:40:37.849988+0000","last_unstale":"2024-09-11T03:41:32.864286+0000","last_undegraded":"2024-09-11T03:41:32.864286+0000","last_fullsized":"2024-09-11T03:41:32.864286+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.331453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.331453+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:40.696793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"269'192","reported_seq":1134,"reported_epoch":279,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T03:41:35.937972+0000","last_change":"2024-09-11T03:41:35.937860+0000","last_active":"2024-09-11T03:41:35.937972+0000","last_peered":"2024-09-11T03:41:35.937972+0000","last_clean":"2024-09-11T03:41:35.937972+0000","last_became_active":"2024-09-11T03:41:33.863875+0000","last_became_peered":"2024-09-11T03:41:33.863875+0000","last_unstale":"2024-09-11T03:41:35.937972+0000","last_undegraded":"2024-09-11T03:41:35.937972+0000","last_fullsized":"2024-09-11T03:41:35.937972+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'190","last_scrub_stamp":"2024-09-11T03:41:21.007493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:21.007493+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00046692000000000001,"stat_sum":{"num_bytes":2747977,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":47,"num_read_kb":47429,"num_write":42,"num_write_kb":13371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"51","length":"1"},{"start":"57","length":"1"},{"start":"5b","length":"2"}]},{"pgid":"1.b","version":"20'92","reported_seq":178,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-11T03:41:36.888326+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:41:36.888326+0000","last_peered":"2024-09-11T03:41:36.888326+0000","last_clean":"2024-09-11T03:41:36.888326+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:41:36.888326+0000","last_undegraded":"2024-09-11T03:41:36.888326+0000","last_fullsized":"2024-09-11T03:41:36.888326+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:52.822589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"274'312","reported_seq":1287,"reported_epoch":276,"state":"peering","last_fresh":"2024-09-11T03:41:32.861989+0000","last_change":"2024-09-11T03:41:32.861989+0000","last_active":"2024-09-11T03:41:32.861989+0000","last_peered":"2024-09-11T03:41:32.030552+0000","last_clean":"2024-09-11T03:41:32.030552+0000","last_became_active":"2024-09-11T03:40:08.836627+0000","last_became_peered":"2024-09-11T03:40:08.836627+0000","last_unstale":"2024-09-11T03:41:32.861989+0000","last_undegraded":"2024-09-11T03:41:32.861989+0000","last_fullsized":"2024-09-11T03:41:32.861989+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'242","last_scrub_stamp":"2024-09-11T03:40:59.291412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:59.291412+0000","objects_scrubbed":9,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":2,"snaptrim_duration":0.00634178,"stat_sum":{"num_bytes":10375122,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":44307,"num_write":75,"num_write_kb":27772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":3216700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"51","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":336,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-11T03:41:36.889377+0000","last_change":"2024-09-11T03:40:35.272628+0000","last_active":"2024-09-11T03:41:36.889377+0000","last_peered":"2024-09-11T03:41:36.889377+0000","last_clean":"2024-09-11T03:41:36.889377+0000","last_became_active":"2024-09-11T03:40:25.858281+0000","last_became_peered":"2024-09-11T03:40:25.858281+0000","last_unstale":"2024-09-11T03:41:36.889377+0000","last_undegraded":"2024-09-11T03:41:36.889377+0000","last_fullsized":"2024-09-11T03:41:36.889377+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.272563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.272563+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:53.042850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":1096,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.874655+0000","last_change":"2024-09-11T03:41:35.875077+0000","last_active":"2024-09-11T03:41:35.874655+0000","last_peered":"2024-09-11T03:41:35.874655+0000","last_clean":"2024-09-11T03:41:35.874655+0000","last_became_active":"2024-09-11T03:41:33.865499+0000","last_became_peered":"2024-09-11T03:41:33.865499+0000","last_unstale":"2024-09-11T03:41:35.874655+0000","last_undegraded":"2024-09-11T03:41:35.874655+0000","last_fullsized":"2024-09-11T03:41:35.874655+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'103","last_scrub_stamp":"2024-09-11T03:40:17.171458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:17.171458+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0020692319999999998,"stat_sum":{"num_bytes":1479032,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":29175,"num_write":31,"num_write_kb":15335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"51","length":"1"},{"start":"57","length":"1"},{"start":"5b","length":"2"}]},{"pgid":"1.5","version":"20'92","reported_seq":234,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-11T03:41:32.859287+0000","last_change":"2024-09-11T03:40:38.224278+0000","last_active":"2024-09-11T03:41:32.859287+0000","last_peered":"2024-09-11T03:41:32.859287+0000","last_clean":"2024-09-11T03:41:32.859287+0000","last_became_active":"2024-09-11T03:40:37.851080+0000","last_became_peered":"2024-09-11T03:40:37.851080+0000","last_unstale":"2024-09-11T03:41:32.859287+0000","last_undegraded":"2024-09-11T03:41:32.859287+0000","last_fullsized":"2024-09-11T03:41:32.859287+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.224231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.224231+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:07.626847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"279'283","reported_seq":1276,"reported_epoch":279,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T03:41:36.148137+0000","last_change":"2024-09-11T03:41:36.147854+0000","last_active":"2024-09-11T03:41:36.148137+0000","last_peered":"2024-09-11T03:41:36.148137+0000","last_clean":"2024-09-11T03:41:36.148137+0000","last_became_active":"2024-09-11T03:41:33.866316+0000","last_became_peered":"2024-09-11T03:41:33.866316+0000","last_unstale":"2024-09-11T03:41:36.148137+0000","last_undegraded":"2024-09-11T03:41:36.148137+0000","last_fullsized":"2024-09-11T03:41:36.148137+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'209","last_scrub_stamp":"2024-09-11T03:40:45.175807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:45.175807+0000","objects_scrubbed":0,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.014519116,"stat_sum":{"num_bytes":9148157,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":104,"num_read_kb":97203,"num_write":65,"num_write_kb":22379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"51","length":"1"},{"start":"57","length":"1"},{"start":"5b","length":"2"}]},{"pgid":"1.6","version":"20'92","reported_seq":470,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-11T03:41:32.864825+0000","last_change":"2024-09-11T03:40:38.367213+0000","last_active":"2024-09-11T03:41:32.864825+0000","last_peered":"2024-09-11T03:41:32.864825+0000","last_clean":"2024-09-11T03:41:32.864825+0000","last_became_active":"2024-09-11T03:40:37.848794+0000","last_became_peered":"2024-09-11T03:40:37.848794+0000","last_unstale":"2024-09-11T03:41:32.864825+0000","last_undegraded":"2024-09-11T03:41:32.864825+0000","last_fullsized":"2024-09-11T03:41:32.864825+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.367170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.367170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:05.132471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"274'162","reported_seq":1127,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.872420+0000","last_change":"2024-09-11T03:41:35.875017+0000","last_active":"2024-09-11T03:41:35.872420+0000","last_peered":"2024-09-11T03:41:35.872420+0000","last_clean":"2024-09-11T03:41:35.872420+0000","last_became_active":"2024-09-11T03:41:33.869119+0000","last_became_peered":"2024-09-11T03:41:33.869119+0000","last_unstale":"2024-09-11T03:41:35.872420+0000","last_undegraded":"2024-09-11T03:41:35.872420+0000","last_fullsized":"2024-09-11T03:41:35.872420+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'130","last_scrub_stamp":"2024-09-11T03:41:03.349362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:03.349362+0000","objects_scrubbed":2,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0030238550000000002,"stat_sum":{"num_bytes":4790044,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":15822,"num_write":39,"num_write_kb":14168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3595434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"51","length":"1"},{"start":"57","length":"1"},{"start":"5b","length":"2"}]},{"pgid":"1.7","version":"20'92","reported_seq":89,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.867809+0000","last_change":"2024-09-11T03:41:02.232514+0000","last_active":"2024-09-11T03:41:35.867809+0000","last_peered":"2024-09-11T03:41:35.867809+0000","last_clean":"2024-09-11T03:41:35.867809+0000","last_became_active":"2024-09-11T03:41:01.853968+0000","last_became_peered":"2024-09-11T03:41:01.853968+0000","last_unstale":"2024-09-11T03:41:35.867809+0000","last_undegraded":"2024-09-11T03:41:35.867809+0000","last_fullsized":"2024-09-11T03:41:35.867809+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:02.232437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:02.232437+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:13.312679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"269'123","reported_seq":1081,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.868582+0000","last_change":"2024-09-11T03:41:35.868774+0000","last_active":"2024-09-11T03:41:35.868582+0000","last_peered":"2024-09-11T03:41:35.868582+0000","last_clean":"2024-09-11T03:41:35.868582+0000","last_became_active":"2024-09-11T03:41:33.861182+0000","last_became_peered":"2024-09-11T03:41:33.861182+0000","last_unstale":"2024-09-11T03:41:35.868582+0000","last_undegraded":"2024-09-11T03:41:35.868582+0000","last_fullsized":"2024-09-11T03:41:35.868582+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'111","last_scrub_stamp":"2024-09-11T03:41:04.281773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:04.281773+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00042134200000000001,"stat_sum":{"num_bytes":1285735,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":2214,"num_write":24,"num_write_kb":13222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"51","length":"1"},{"start":"57","length":"1"},{"start":"5b","length":"2"}]},{"pgid":"1.1","version":"20'92","reported_seq":456,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-11T03:41:32.864632+0000","last_change":"2024-09-11T03:40:35.325550+0000","last_active":"2024-09-11T03:41:32.864632+0000","last_peered":"2024-09-11T03:41:32.864632+0000","last_clean":"2024-09-11T03:41:32.864632+0000","last_became_active":"2024-09-11T03:40:25.854269+0000","last_became_peered":"2024-09-11T03:40:25.854269+0000","last_unstale":"2024-09-11T03:41:32.864632+0000","last_undegraded":"2024-09-11T03:41:32.864632+0000","last_fullsized":"2024-09-11T03:41:32.864632+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.325493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.325493+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:00.762795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"274'107","reported_seq":1015,"reported_epoch":276,"state":"peering","last_fresh":"2024-09-11T03:41:32.860724+0000","last_change":"2024-09-11T03:41:32.860724+0000","last_active":"2024-09-11T03:41:32.860724+0000","last_peered":"2024-09-11T03:41:32.030507+0000","last_clean":"2024-09-11T03:41:32.030507+0000","last_became_active":"2024-09-11T03:38:30.162952+0000","last_became_peered":"2024-09-11T03:38:30.162952+0000","last_unstale":"2024-09-11T03:41:32.860724+0000","last_undegraded":"2024-09-11T03:41:32.860724+0000","last_fullsized":"2024-09-11T03:41:32.860724+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'87","last_scrub_stamp":"2024-09-11T03:40:58.304220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:58.304220+0000","objects_scrubbed":3,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0042022129999999998,"stat_sum":{"num_bytes":1537857,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":12941,"num_write":17,"num_write_kb":10272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"51","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":419,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.866099+0000","last_change":"2024-09-11T03:41:02.760300+0000","last_active":"2024-09-11T03:41:35.866099+0000","last_peered":"2024-09-11T03:41:35.866099+0000","last_clean":"2024-09-11T03:41:35.866099+0000","last_became_active":"2024-09-11T03:41:01.851367+0000","last_became_peered":"2024-09-11T03:41:01.851367+0000","last_unstale":"2024-09-11T03:41:35.866099+0000","last_undegraded":"2024-09-11T03:41:35.866099+0000","last_fullsized":"2024-09-11T03:41:35.866099+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:02.760253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:02.760253+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:31.006129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"274'67","reported_seq":1015,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.866907+0000","last_change":"2024-09-11T03:41:35.867063+0000","last_active":"2024-09-11T03:41:35.866907+0000","last_peered":"2024-09-11T03:41:35.866907+0000","last_clean":"2024-09-11T03:41:35.866907+0000","last_became_active":"2024-09-11T03:41:33.864202+0000","last_became_peered":"2024-09-11T03:41:33.864202+0000","last_unstale":"2024-09-11T03:41:35.866907+0000","last_undegraded":"2024-09-11T03:41:35.866907+0000","last_fullsized":"2024-09-11T03:41:35.866907+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'33","last_scrub_stamp":"2024-09-11T03:40:58.014124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:58.014124+0000","objects_scrubbed":2,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00055133099999999996,"stat_sum":{"num_bytes":1949712,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":18300,"num_write":14,"num_write_kb":5786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"51","length":"1"},{"start":"57","length":"1"},{"start":"5b","length":"2"}]},{"pgid":"1.3","version":"20'92","reported_seq":238,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.870670+0000","last_change":"2024-09-11T03:40:36.259209+0000","last_active":"2024-09-11T03:41:35.870670+0000","last_peered":"2024-09-11T03:41:35.870670+0000","last_clean":"2024-09-11T03:41:35.870670+0000","last_became_active":"2024-09-11T03:40:25.856407+0000","last_became_peered":"2024-09-11T03:40:25.856407+0000","last_unstale":"2024-09-11T03:41:35.870670+0000","last_undegraded":"2024-09-11T03:41:35.870670+0000","last_fullsized":"2024-09-11T03:41:35.870670+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:36.259163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:36.259163+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:55.651280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"278'115","reported_seq":1054,"reported_epoch":279,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T03:41:36.085733+0000","last_change":"2024-09-11T03:41:36.085444+0000","last_active":"2024-09-11T03:41:36.085733+0000","last_peered":"2024-09-11T03:41:36.085733+0000","last_clean":"2024-09-11T03:41:36.085733+0000","last_became_active":"2024-09-11T03:41:33.862271+0000","last_became_peered":"2024-09-11T03:41:33.862271+0000","last_unstale":"2024-09-11T03:41:36.085733+0000","last_undegraded":"2024-09-11T03:41:36.085733+0000","last_fullsized":"2024-09-11T03:41:36.085733+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'78","last_scrub_stamp":"2024-09-11T03:40:45.292704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:45.292704+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046896500000000001,"stat_sum":{"num_bytes":2705943,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":21231,"num_write":22,"num_write_kb":8249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"51","length":"1"},{"start":"57","length":"1"},{"start":"5b","length":"2"}]},{"pgid":"1.0","version":"20'92","reported_seq":558,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.866751+0000","last_change":"2024-09-11T03:40:36.901245+0000","last_active":"2024-09-11T03:41:35.866751+0000","last_peered":"2024-09-11T03:41:35.866751+0000","last_clean":"2024-09-11T03:41:35.866751+0000","last_became_active":"2024-09-11T03:40:25.854624+0000","last_became_peered":"2024-09-11T03:40:25.854624+0000","last_unstale":"2024-09-11T03:41:35.866751+0000","last_undegraded":"2024-09-11T03:41:35.866751+0000","last_fullsized":"2024-09-11T03:41:35.866751+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:36.901166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:36.901166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:43.207763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"20'92","reported_seq":162,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.867706+0000","last_change":"2024-09-11T03:40:35.282811+0000","last_active":"2024-09-11T03:41:35.867706+0000","last_peered":"2024-09-11T03:41:35.867706+0000","last_clean":"2024-09-11T03:41:35.867706+0000","last_became_active":"2024-09-11T03:40:27.844262+0000","last_became_peered":"2024-09-11T03:40:27.844262+0000","last_unstale":"2024-09-11T03:41:35.867706+0000","last_undegraded":"2024-09-11T03:41:35.867706+0000","last_fullsized":"2024-09-11T03:41:35.867706+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.282762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.282762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:51.256113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"279'233","reported_seq":1187,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-11T03:41:36.889725+0000","last_change":"2024-09-11T03:41:36.890019+0000","last_active":"2024-09-11T03:41:36.889725+0000","last_peered":"2024-09-11T03:41:36.889725+0000","last_clean":"2024-09-11T03:41:36.889725+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:41:36.889725+0000","last_undegraded":"2024-09-11T03:41:36.889725+0000","last_fullsized":"2024-09-11T03:41:36.889725+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'172","last_scrub_stamp":"2024-09-11T03:41:04.106166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:04.106166+0000","objects_scrubbed":5,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:13.826364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00053441299999999997,"stat_sum":{"num_bytes":11227175,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":139,"num_read_kb":110293,"num_write":113,"num_write_kb":50034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"51","length":"1"},{"start":"57","length":"1"},{"start":"5a","length":"3"}]},{"pgid":"1.e","version":"20'92","reported_seq":417,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-11T03:41:34.863544+0000","last_change":"2024-09-11T03:40:38.199696+0000","last_active":"2024-09-11T03:41:34.863544+0000","last_peered":"2024-09-11T03:41:34.863544+0000","last_clean":"2024-09-11T03:41:34.863544+0000","last_became_active":"2024-09-11T03:40:37.850623+0000","last_became_peered":"2024-09-11T03:40:37.850623+0000","last_unstale":"2024-09-11T03:41:34.863544+0000","last_undegraded":"2024-09-11T03:41:34.863544+0000","last_fullsized":"2024-09-11T03:41:34.863544+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.199636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.199636+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:45.493648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"274'306","reported_seq":1216,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-11T03:41:32.865156+0000","last_change":"2024-09-11T03:41:32.865466+0000","last_active":"2024-09-11T03:41:32.865156+0000","last_peered":"2024-09-11T03:41:32.865156+0000","last_clean":"2024-09-11T03:41:32.865156+0000","last_became_active":"2024-09-11T03:38:30.031497+0000","last_became_peered":"2024-09-11T03:38:30.031497+0000","last_unstale":"2024-09-11T03:41:32.865156+0000","last_undegraded":"2024-09-11T03:41:32.865156+0000","last_fullsized":"2024-09-11T03:41:32.865156+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'249","last_scrub_stamp":"2024-09-11T03:41:08.270845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:08.270845+0000","objects_scrubbed":10,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:26.879661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000601842,"stat_sum":{"num_bytes":11400419,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":140,"num_read_kb":105480,"num_write":105,"num_write_kb":38900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"51","length":"1"},{"start":"5b","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":176,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.866149+0000","last_change":"2024-09-11T03:40:38.901246+0000","last_active":"2024-09-11T03:41:35.866149+0000","last_peered":"2024-09-11T03:41:35.866149+0000","last_clean":"2024-09-11T03:41:35.866149+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:41:35.866149+0000","last_undegraded":"2024-09-11T03:41:35.866149+0000","last_fullsized":"2024-09-11T03:41:35.866149+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.901175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.901175+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:48.712523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"276'304","reported_seq":1214,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-11T03:41:32.904319+0000","last_change":"2024-09-11T03:41:32.859690+0000","last_active":"2024-09-11T03:41:32.904319+0000","last_peered":"2024-09-11T03:41:32.904319+0000","last_clean":"2024-09-11T03:41:32.904319+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:41:32.904319+0000","last_undegraded":"2024-09-11T03:41:32.904319+0000","last_fullsized":"2024-09-11T03:41:32.904319+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'259","last_scrub_stamp":"2024-09-11T03:41:09.347904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:09.347904+0000","objects_scrubbed":11,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:10.274678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00038322299999999997,"stat_sum":{"num_bytes":18411797,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":137,"num_read_kb":111357,"num_write":112,"num_write_kb":49137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"51","length":"1"},{"start":"5b","length":"1"}]},{"pgid":"2.f","version":"278'142","reported_seq":1094,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-11T03:41:35.864469+0000","last_change":"2024-09-11T03:41:35.864755+0000","last_active":"2024-09-11T03:41:35.864469+0000","last_peered":"2024-09-11T03:41:35.864469+0000","last_clean":"2024-09-11T03:41:35.864469+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:41:35.864469+0000","last_undegraded":"2024-09-11T03:41:35.864469+0000","last_fullsized":"2024-09-11T03:41:35.864469+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'136","last_scrub_stamp":"2024-09-11T03:41:20.167983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:20.167983+0000","objects_scrubbed":2,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:43.959191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00067246600000000004,"stat_sum":{"num_bytes":4705993,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":98,"num_read_kb":82051,"num_write":59,"num_write_kb":21882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"51","length":"1"},{"start":"57","length":"1"},{"start":"5b","length":"2"}]},{"pgid":"1.c","version":"20'92","reported_seq":178,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-11T03:41:32.862584+0000","last_change":"2024-09-11T03:40:38.353992+0000","last_active":"2024-09-11T03:41:32.862584+0000","last_peered":"2024-09-11T03:41:32.862584+0000","last_clean":"2024-09-11T03:41:32.862584+0000","last_became_active":"2024-09-11T03:40:37.847449+0000","last_became_peered":"2024-09-11T03:40:37.847449+0000","last_unstale":"2024-09-11T03:41:32.862584+0000","last_undegraded":"2024-09-11T03:41:32.862584+0000","last_fullsized":"2024-09-11T03:41:32.862584+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.353895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.353895+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:41:44.612649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":138862918,"num_objects":58,"num_object_clones":20,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":3,"num_read":1394,"num_read_kb":1119456,"num_write":1185,"num_write_kb":494330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":13624269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286892032,"data_stored":326861355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4416,"ondisk_log_size":4416,"up":69,"acting":64,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285621,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":55352,"kb_used_data":24160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93689800,"statfs":{"total":95995035648,"available":95938355200,"internally_reserved":0,"allocated":24739840,"data_stored":26127621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":326417514553,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743486,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60704,"kb_used_data":29508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93684448,"statfs":{"total":95995035648,"available":95932874752,"internally_reserved":0,"allocated":30216192,"data_stored":36039202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037830,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65128,"kb_used_data":29776,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93680024,"statfs":{"total":95995035648,"available":95928344576,"internally_reserved":0,"allocated":30490624,"data_stored":30281580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299465,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64740,"kb_used_data":29392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93680412,"statfs":{"total":95995035648,"available":95928741888,"internally_reserved":0,"allocated":30097408,"data_stored":31624387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":193273528398,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757332,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":39584,"kb_used_data":8388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93705568,"statfs":{"total":95995035648,"available":95954501632,"internally_reserved":0,"allocated":8589312,"data_stored":8772043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986265,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71284,"kb_used_data":35936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93673868,"statfs":{"total":95995035648,"available":95922040832,"internally_reserved":0,"allocated":36798464,"data_stored":41902733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82280,"kb_used_data":46920,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93662872,"statfs":{"total":95995035648,"available":95910780928,"internally_reserved":0,"allocated":48046080,"data_stored":57092234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411426,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65328,"kb_used_data":34140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93679824,"statfs":{"total":95995035648,"available":95928139776,"internally_reserved":0,"allocated":34959360,"data_stored":40442829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58404,"kb_used_data":27212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93686748,"statfs":{"total":95995035648,"available":95935229952,"internally_reserved":0,"allocated":27865088,"data_stored":26436625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738478,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64092,"kb_used_data":32904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93681060,"statfs":{"total":95995035648,"available":95929405440,"internally_reserved":0,"allocated":33693696,"data_stored":33017067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29200384,"data_stored":29803895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576000,"data_stored":24438051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":33148928,"data_stored":40373024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":44466176,"data_stored":54791106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33521664,"data_stored":39912925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5095424,"data_stored":6572221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4079616,"data_stored":4066399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":26603520,"data_stored":29424565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":27201536,"data_stored":28283006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":29986816,"data_stored":37546901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2387968,"data_stored":2367002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":26624000,"data_stored":29282260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:41:39.104 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-11T03:41:39.105 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pg_num 41 2024-09-11T03:41:39.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:39.169 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:39 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:41:39.169 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:39 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:41:39.169 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:39 smithi121 ceph-mon[30729]: 2.3 scrub starts 2024-09-11T03:41:39.170 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:39 smithi121 ceph-mon[30729]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:41:39.170 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:39 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:41:39.170 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:39 smithi121 ceph-mon[30729]: 2.10 scrub starts 2024-09-11T03:41:39.170 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:39 smithi121 ceph-mon[30729]: osdmap e282: 12 total, 12 up, 10 in 2024-09-11T03:41:39.174 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:39.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:39.181 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:39.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:39.281 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:41:39.281 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:41:39.282 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:39.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:39 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:41:39.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:39 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:41:39.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:39 smithi084 ceph-mon[27254]: 2.3 scrub starts 2024-09-11T03:41:39.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:39 smithi084 ceph-mon[27254]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:41:39.297 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:39 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:41:39.297 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:39 smithi084 ceph-mon[27254]: 2.10 scrub starts 2024-09-11T03:41:39.297 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:39 smithi084 ceph-mon[27254]: osdmap e282: 12 total, 12 up, 10 in 2024-09-11T03:41:39.304 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:39.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:39.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:39.406 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:39.428 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:39.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:39.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:39.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:39 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:41:39.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:39 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:41:39.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:39 smithi175 ceph-mon[30894]: 2.3 scrub starts 2024-09-11T03:41:39.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:39 smithi175 ceph-mon[30894]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:41:39.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:39 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:41:39.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:39 smithi175 ceph-mon[30894]: 2.10 scrub starts 2024-09-11T03:41:39.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:39 smithi175 ceph-mon[30894]: osdmap e282: 12 total, 12 up, 10 in 2024-09-11T03:41:39.531 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:39.553 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:39.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:39.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:39.655 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:39.681 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:39.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:39.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:39.709 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:39.783 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:39.804 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:39.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:39.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:39.880 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:41:39.884 INFO:tasks.rados.rados.2.smithi175.stdout:903: done (0 left) 2024-09-11T03:41:39.884 INFO:tasks.rados.rados.2.smithi175.stdout:904: write oid 17 current snap is 95 2024-09-11T03:41:39.885 INFO:tasks.rados.rados.2.smithi175.stdout:904: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2024-09-11T03:41:39.888 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T03:41:39.889 INFO:tasks.rados.rados.2.smithi175.stdout:904: writing smithi17557870-17 from 413538 to 949883 tid 1 2024-09-11T03:41:39.895 INFO:tasks.rados.rados.2.smithi175.stdout:904: writing smithi17557870-17 from 1395221 to 1979002 tid 2 2024-09-11T03:41:39.903 INFO:tasks.rados.rados.2.smithi175.stdout:904: writing smithi17557870-17 from 2556103 to 3214720 tid 3 2024-09-11T03:41:39.903 INFO:tasks.rados.rados.2.smithi175.stdout:905: read oid 50 snap -1 2024-09-11T03:41:39.903 INFO:tasks.rados.rados.2.smithi175.stdout:905: expect deleted 2024-09-11T03:41:39.903 INFO:tasks.rados.rados.2.smithi175.stdout:904: finishing write tid 1 to smithi17557870-17 2024-09-11T03:41:39.903 INFO:tasks.rados.rados.2.smithi175.stdout:904: finishing write tid 2 to smithi17557870-17 2024-09-11T03:41:39.903 INFO:tasks.rados.rados.2.smithi175.stdout:906: read oid 25 snap 84 2024-09-11T03:41:39.903 INFO:tasks.rados.rados.2.smithi175.stdout:906: expect (ObjNum 241 snap 58 seq_num 241) 2024-09-11T03:41:39.903 INFO:tasks.rados.rados.2.smithi175.stdout:907: write (excl) oid 12 current snap is 95 2024-09-11T03:41:39.903 INFO:tasks.rados.rados.2.smithi175.stdout:907: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2024-09-11T03:41:39.906 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:39.909 INFO:tasks.rados.rados.2.smithi175.stdout:907: writing smithi17557870-12 from 430345 to 1158292 tid 1 2024-09-11T03:41:39.917 INFO:tasks.rados.rados.2.smithi175.stdout:907: writing smithi17557870-12 from 1570056 to 2297495 tid 2 2024-09-11T03:41:39.922 INFO:tasks.rados.rados.2.smithi175.stdout:907: writing smithi17557870-12 from 2699879 to 3169768 tid 3 2024-09-11T03:41:39.922 INFO:tasks.rados.rados.2.smithi175.stdout:908: copy_from oid 14 from oid 19 current snap is 95 2024-09-11T03:41:39.922 INFO:tasks.rados.rados.2.smithi175.stdout:904: finishing write tid 3 to smithi17557870-17 2024-09-11T03:41:39.922 INFO:tasks.rados.rados.2.smithi175.stdout:904: finishing write tid 4 to smithi17557870-17 2024-09-11T03:41:39.922 INFO:tasks.rados.rados.2.smithi175.stdout:904: finishing write tid 5 to smithi17557870-17 2024-09-11T03:41:39.922 INFO:tasks.rados.rados.2.smithi175.stdout:904: oid 17 updating version 0 to 318 2024-09-11T03:41:39.922 INFO:tasks.rados.rados.2.smithi175.stdout:904: oid 17 version 318 is already newer than 317 2024-09-11T03:41:39.922 INFO:tasks.rados.rados.2.smithi175.stdout:904: oid 17 version 318 is already newer than 315 2024-09-11T03:41:39.923 INFO:tasks.rados.rados.2.smithi175.stdout:904: oid 17 version 318 is already newer than 316 2024-09-11T03:41:39.923 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 318 (ObjNum 334 snap 95 seq_num 334) dirty exists 2024-09-11T03:41:39.923 INFO:tasks.rados.rados.2.smithi175.stdout:904: left oid 17 (ObjNum 334 snap 95 seq_num 334) 2024-09-11T03:41:39.923 INFO:tasks.rados.rados.2.smithi175.stdout:906: expect (ObjNum 241 snap 58 seq_num 241) 2024-09-11T03:41:39.928 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:39.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:39.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:40.027 INFO:tasks.rados.rados.2.smithi175.stdout:907: finishing write tid 1 to smithi17557870-12 2024-09-11T03:41:40.027 INFO:tasks.rados.rados.2.smithi175.stdout:907: finishing write tid 2 to smithi17557870-12 2024-09-11T03:41:40.027 INFO:tasks.rados.rados.2.smithi175.stdout:907: finishing write tid 3 to smithi17557870-12 2024-09-11T03:41:40.027 INFO:tasks.rados.rados.2.smithi175.stdout:907: finishing write tid 4 to smithi17557870-12 2024-09-11T03:41:40.027 INFO:tasks.rados.rados.2.smithi175.stdout:907: finishing write tid 5 to smithi17557870-12 2024-09-11T03:41:40.028 INFO:tasks.rados.rados.2.smithi175.stdout:907: oid 12 updating version 0 to 292 2024-09-11T03:41:40.028 INFO:tasks.rados.rados.2.smithi175.stdout:907: oid 12 version 292 is already newer than 291 2024-09-11T03:41:40.028 INFO:tasks.rados.rados.2.smithi175.stdout:907: oid 12 version 292 is already newer than 289 2024-09-11T03:41:40.028 INFO:tasks.rados.rados.2.smithi175.stdout:907: oid 12 version 292 is already newer than 290 2024-09-11T03:41:40.028 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 292 (ObjNum 335 snap 95 seq_num 335) dirty exists 2024-09-11T03:41:40.028 INFO:tasks.rados.rados.2.smithi175.stdout:907: left oid 12 (ObjNum 335 snap 95 seq_num 335) 2024-09-11T03:41:40.028 INFO:tasks.rados.rados.2.smithi175.stdout:908: finishing copy_from to smithi17557870-14 2024-09-11T03:41:40.028 INFO:tasks.rados.rados.2.smithi175.stdout:908: got expected ENOENT (src dne) 2024-09-11T03:41:40.028 INFO:tasks.rados.rados.2.smithi175.stdout:908: finishing copy_from racing read to smithi17557870-14 2024-09-11T03:41:40.028 INFO:tasks.rados.rados.2.smithi175.stdout:904: done (4 left) 2024-09-11T03:41:40.028 INFO:tasks.rados.rados.2.smithi175.stdout:905: done (3 left) 2024-09-11T03:41:40.028 INFO:tasks.rados.rados.2.smithi175.stdout:906: done (2 left) 2024-09-11T03:41:40.028 INFO:tasks.rados.rados.2.smithi175.stdout:907: done (1 left) 2024-09-11T03:41:40.029 INFO:tasks.rados.rados.2.smithi175.stdout:908: done (0 left) 2024-09-11T03:41:40.029 INFO:tasks.rados.rados.2.smithi175.stdout:909: write (excl) oid 50 current snap is 95 2024-09-11T03:41:40.029 INFO:tasks.rados.rados.2.smithi175.stdout:909: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2024-09-11T03:41:40.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:40.031 INFO:tasks.rados.rados.2.smithi175.stdout:909: writing smithi17557870-50 from 447152 to 850348 tid 1 2024-09-11T03:41:40.038 INFO:tasks.rados.rados.2.smithi175.stdout:909: writing smithi17557870-50 from 1512185 to 2099635 tid 2 2024-09-11T03:41:40.046 INFO:tasks.rados.rados.2.smithi175.stdout:909: writing smithi17557870-50 from 2727302 to 3408463 tid 3 2024-09-11T03:41:40.046 INFO:tasks.rados.rados.2.smithi175.stdout:910: rollback oid 43 current snap is 95 2024-09-11T03:41:40.046 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 43 to 91 2024-09-11T03:41:40.046 INFO:tasks.rados.rados.2.smithi175.stdout:909: finishing write tid 1 to smithi17557870-50 2024-09-11T03:41:40.046 INFO:tasks.rados.rados.2.smithi175.stdout:911: snap_create 2024-09-11T03:41:40.046 INFO:tasks.rados.rados.2.smithi175.stdout:909: finishing write tid 2 to smithi17557870-50 2024-09-11T03:41:40.048 INFO:tasks.rados.rados.2.smithi175.stdout:910: finishing rollback tid 1 to smithi17557870-43 2024-09-11T03:41:40.049 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 0 (ObjNum 1745071872 snap 22076 seq_num 304) dirty dne 2024-09-11T03:41:40.049 INFO:tasks.rados.rados.2.smithi175.stdout:909: finishing write tid 3 to smithi17557870-50 2024-09-11T03:41:40.049 INFO:tasks.rados.rados.2.smithi175.stdout:909: finishing write tid 4 to smithi17557870-50 2024-09-11T03:41:40.049 INFO:tasks.rados.rados.2.smithi175.stdout:909: finishing write tid 5 to smithi17557870-50 2024-09-11T03:41:40.049 INFO:tasks.rados.rados.2.smithi175.stdout:909: oid 50 updating version 0 to 335 2024-09-11T03:41:40.049 INFO:tasks.rados.rados.2.smithi175.stdout:909: oid 50 updating version 335 to 336 2024-09-11T03:41:40.049 INFO:tasks.rados.rados.2.smithi175.stdout:909: oid 50 updating version 336 to 337 2024-09-11T03:41:40.049 INFO:tasks.rados.rados.2.smithi175.stdout:909: oid 50 version 337 is already newer than 334 2024-09-11T03:41:40.049 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 337 (ObjNum 336 snap 95 seq_num 336) dirty exists 2024-09-11T03:41:40.049 INFO:tasks.rados.rados.2.smithi175.stdout:909: left oid 50 (ObjNum 336 snap 95 seq_num 336) 2024-09-11T03:41:40.052 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:40.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:40.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:40.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:40 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:41:40.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:40 smithi084 ceph-mon[27254]: pgmap v577: 57 pgs: 5 peering, 4 active+clean+scrubbing, 3 remapped+peering, 1 unknown, 44 active+clean; 144 MiB data, 630 MiB used, 893 GiB / 894 GiB avail; 826 KiB/s, 4 objects/s recovering 2024-09-11T03:41:40.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:40 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:41:40.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:40 smithi084 ceph-mon[27254]: 2.11 scrub starts 2024-09-11T03:41:40.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:40 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:41:40.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:40 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:41:40.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:40 smithi084 ceph-mon[27254]: 2.10 scrub starts 2024-09-11T03:41:40.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:40 smithi084 ceph-mon[27254]: 2.13 scrub starts 2024-09-11T03:41:40.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:40 smithi084 ceph-mon[27254]: 2.19 scrub starts 2024-09-11T03:41:40.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:40 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2469235885' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:41:40.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:40 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2469235885' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:41:40.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:40 smithi084 ceph-mon[27254]: osdmap e283: 12 total, 12 up, 10 in 2024-09-11T03:41:40.148 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:41:40.155 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:40 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:41:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:40 smithi121 ceph-mon[30729]: pgmap v577: 57 pgs: 5 peering, 4 active+clean+scrubbing, 3 remapped+peering, 1 unknown, 44 active+clean; 144 MiB data, 630 MiB used, 893 GiB / 894 GiB avail; 826 KiB/s, 4 objects/s recovering 2024-09-11T03:41:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:40 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:41:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:40 smithi121 ceph-mon[30729]: 2.11 scrub starts 2024-09-11T03:41:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:40 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:41:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:40 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:41:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:40 smithi121 ceph-mon[30729]: 2.10 scrub starts 2024-09-11T03:41:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:40 smithi121 ceph-mon[30729]: 2.13 scrub starts 2024-09-11T03:41:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:40 smithi121 ceph-mon[30729]: 2.19 scrub starts 2024-09-11T03:41:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:40 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2469235885' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:41:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:40 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2469235885' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:41:40.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:40 smithi121 ceph-mon[30729]: osdmap e283: 12 total, 12 up, 10 in 2024-09-11T03:41:40.177 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:40.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:40.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:40.279 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:40.301 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:40.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:40.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:40.378 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:40 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:41:40.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:40 smithi175 ceph-mon[30894]: pgmap v577: 57 pgs: 5 peering, 4 active+clean+scrubbing, 3 remapped+peering, 1 unknown, 44 active+clean; 144 MiB data, 630 MiB used, 893 GiB / 894 GiB avail; 826 KiB/s, 4 objects/s recovering 2024-09-11T03:41:40.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:40 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:41:40.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:40 smithi175 ceph-mon[30894]: 2.11 scrub starts 2024-09-11T03:41:40.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:40 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:41:40.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:40 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:41:40.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:40 smithi175 ceph-mon[30894]: 2.10 scrub starts 2024-09-11T03:41:40.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:40 smithi175 ceph-mon[30894]: 2.13 scrub starts 2024-09-11T03:41:40.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:40 smithi175 ceph-mon[30894]: 2.19 scrub starts 2024-09-11T03:41:40.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:40 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2469235885' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:41:40.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:40 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2469235885' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:41:40.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:40 smithi175 ceph-mon[30894]: osdmap e283: 12 total, 12 up, 10 in 2024-09-11T03:41:40.403 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:40.424 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:40.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:40.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:40.526 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:40.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:40.549 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:40.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:40.593 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: specified pg_num 41 < pg_num_max 32 2024-09-11T03:41:40.650 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:40.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:40.673 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:40.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:40.679 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:41:40.679 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:41:40.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:41:40.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:40.796 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:40.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:40.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:40.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:40.919 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:40.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:41.021 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:41.044 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:41.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:41.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:41.046 INFO:tasks.rados.rados.2.smithi175.stdout:909: done (2 left) 2024-09-11T03:41:41.046 INFO:tasks.rados.rados.2.smithi175.stdout:910: done (1 left) 2024-09-11T03:41:41.046 INFO:tasks.rados.rados.2.smithi175.stdout:911: done (0 left) 2024-09-11T03:41:41.046 INFO:tasks.rados.rados.2.smithi175.stdout:912: snap_remove snap 92 2024-09-11T03:41:41.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:41 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:41:41.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:41 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:41:41.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:41 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:41:41.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:41 smithi084 ceph-mon[27254]: 2.2 scrub ok 2024-09-11T03:41:41.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:41 smithi084 ceph-mon[27254]: 2.11 scrub starts 2024-09-11T03:41:41.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:41 smithi084 ceph-mon[27254]: 2.11 scrub ok 2024-09-11T03:41:41.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:41 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:41:41.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:41 smithi084 ceph-mon[27254]: 2.9 scrub ok 2024-09-11T03:41:41.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:41 smithi084 ceph-mon[27254]: 2.10 scrub starts 2024-09-11T03:41:41.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:41 smithi084 ceph-mon[27254]: 2.10 scrub ok 2024-09-11T03:41:41.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:41 smithi084 ceph-mon[27254]: 2.18 scrub starts 2024-09-11T03:41:41.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:41 smithi084 ceph-mon[27254]: 2.18 scrub ok 2024-09-11T03:41:41.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:41 smithi084 ceph-mon[27254]: 2.19 scrub starts 2024-09-11T03:41:41.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:41 smithi084 ceph-mon[27254]: 2.19 scrub ok 2024-09-11T03:41:41.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:41 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/148287238' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-11T03:41:41.147 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:41.173 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:41.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:41.174 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:41.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:41.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:41.297 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:41.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:41.309 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:41 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:41:41.309 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:41 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:41:41.309 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:41 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:41:41.309 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:41 smithi121 ceph-mon[30729]: 2.2 scrub ok 2024-09-11T03:41:41.309 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:41 smithi121 ceph-mon[30729]: 2.11 scrub starts 2024-09-11T03:41:41.309 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:41 smithi121 ceph-mon[30729]: 2.11 scrub ok 2024-09-11T03:41:41.310 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:41 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:41:41.310 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:41 smithi121 ceph-mon[30729]: 2.9 scrub ok 2024-09-11T03:41:41.310 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:41 smithi121 ceph-mon[30729]: 2.10 scrub starts 2024-09-11T03:41:41.310 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:41 smithi121 ceph-mon[30729]: 2.10 scrub ok 2024-09-11T03:41:41.310 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:41 smithi121 ceph-mon[30729]: 2.18 scrub starts 2024-09-11T03:41:41.310 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:41 smithi121 ceph-mon[30729]: 2.18 scrub ok 2024-09-11T03:41:41.311 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:41 smithi121 ceph-mon[30729]: 2.19 scrub starts 2024-09-11T03:41:41.311 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:41 smithi121 ceph-mon[30729]: 2.19 scrub ok 2024-09-11T03:41:41.311 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:41 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/148287238' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-11T03:41:41.335 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T03:41:41.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:41.406 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:41 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:41:41.406 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:41 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:41:41.406 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:41 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:41:41.406 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:41 smithi175 ceph-mon[30894]: 2.2 scrub ok 2024-09-11T03:41:41.406 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:41 smithi175 ceph-mon[30894]: 2.11 scrub starts 2024-09-11T03:41:41.406 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:41 smithi175 ceph-mon[30894]: 2.11 scrub ok 2024-09-11T03:41:41.406 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:41 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:41:41.406 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:41 smithi175 ceph-mon[30894]: 2.9 scrub ok 2024-09-11T03:41:41.407 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:41 smithi175 ceph-mon[30894]: 2.10 scrub starts 2024-09-11T03:41:41.407 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:41 smithi175 ceph-mon[30894]: 2.10 scrub ok 2024-09-11T03:41:41.407 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:41 smithi175 ceph-mon[30894]: 2.18 scrub starts 2024-09-11T03:41:41.407 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:41 smithi175 ceph-mon[30894]: 2.18 scrub ok 2024-09-11T03:41:41.407 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:41 smithi175 ceph-mon[30894]: 2.19 scrub starts 2024-09-11T03:41:41.407 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:41 smithi175 ceph-mon[30894]: 2.19 scrub ok 2024-09-11T03:41:41.407 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:41 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/148287238' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-11T03:41:41.439 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:41.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:41.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:41.449 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:41:41.540 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:41.544 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:41.563 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:41.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:41.564 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:41.618 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:41:41.664 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:41.686 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:41.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:41.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:41.788 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:41.809 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:41.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:41.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:41.859 INFO:tasks.rados.rados.2.smithi175.stdout:912: done (0 left) 2024-09-11T03:41:41.859 INFO:tasks.rados.rados.2.smithi175.stdout:913: snap_remove snap 87 2024-09-11T03:41:41.910 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:41.931 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:41.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:41.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:42.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:42.038 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:41:42.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:41:42.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:42.056 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:42.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:42.057 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:41:42.057 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:41:42.073 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:41:42.073 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:41:42.089 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:41:42.089 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:41:42.106 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:41:42.107 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:41:42.125 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:41:42.125 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:41:42.141 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:41:42.142 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:41:42.157 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:42.158 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:41:42.158 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:41:42.164 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:41:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:41:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:41:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:41:42.164 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:41:42.176 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:41:42.176 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:41:42.178 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:42.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:42.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:42.192 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:41:42.192 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:41:42.209 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:41:42.209 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:41:42.225 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:41:42.225 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:41:42.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:42.301 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:42.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:42.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:42.363 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:41:42.363 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: 2.1 scrub ok 2024-09-11T03:41:42.363 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: pgmap v580: 57 pgs: 7 peering, 4 active+clean+scrubbing, 1 remapped+peering, 45 active+clean; 149 MiB data, 641 MiB used, 893 GiB / 894 GiB avail; 47 MiB/s rd, 13 MiB/s wr, 87 op/s; 1.2 MiB/s, 6 objects/s recovering 2024-09-11T03:41:42.364 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:41:42.364 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: 2.8 scrub ok 2024-09-11T03:41:42.364 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: osdmap e284: 12 total, 12 up, 10 in 2024-09-11T03:41:42.364 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: 2.3 scrub starts 2024-09-11T03:41:42.364 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: 2.3 scrub ok 2024-09-11T03:41:42.365 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: 2.4 scrub starts 2024-09-11T03:41:42.365 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: 2.4 scrub ok 2024-09-11T03:41:42.365 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: 2.5 scrub starts 2024-09-11T03:41:42.365 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: 2.5 scrub ok 2024-09-11T03:41:42.365 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: 2.13 scrub starts 2024-09-11T03:41:42.365 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: 2.13 scrub ok 2024-09-11T03:41:42.365 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: 2.12 scrub starts 2024-09-11T03:41:42.365 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: 2.12 scrub ok 2024-09-11T03:41:42.365 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:42.366 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: osdmap e285: 12 total, 12 up, 10 in 2024-09-11T03:41:42.366 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:42 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3065331716' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:41:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: 2.1 scrub ok 2024-09-11T03:41:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: pgmap v580: 57 pgs: 7 peering, 4 active+clean+scrubbing, 1 remapped+peering, 45 active+clean; 149 MiB data, 641 MiB used, 893 GiB / 894 GiB avail; 47 MiB/s rd, 13 MiB/s wr, 87 op/s; 1.2 MiB/s, 6 objects/s recovering 2024-09-11T03:41:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:41:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: 2.8 scrub ok 2024-09-11T03:41:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: osdmap e284: 12 total, 12 up, 10 in 2024-09-11T03:41:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: 2.3 scrub starts 2024-09-11T03:41:42.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: 2.3 scrub ok 2024-09-11T03:41:42.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: 2.4 scrub starts 2024-09-11T03:41:42.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: 2.4 scrub ok 2024-09-11T03:41:42.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: 2.5 scrub starts 2024-09-11T03:41:42.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: 2.5 scrub ok 2024-09-11T03:41:42.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: 2.13 scrub starts 2024-09-11T03:41:42.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: 2.13 scrub ok 2024-09-11T03:41:42.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: 2.12 scrub starts 2024-09-11T03:41:42.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: 2.12 scrub ok 2024-09-11T03:41:42.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:42.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: osdmap e285: 12 total, 12 up, 10 in 2024-09-11T03:41:42.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:42 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3065331716' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:42.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:41:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: 2.1 scrub ok 2024-09-11T03:41:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: pgmap v580: 57 pgs: 7 peering, 4 active+clean+scrubbing, 1 remapped+peering, 45 active+clean; 149 MiB data, 641 MiB used, 893 GiB / 894 GiB avail; 47 MiB/s rd, 13 MiB/s wr, 87 op/s; 1.2 MiB/s, 6 objects/s recovering 2024-09-11T03:41:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:41:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: 2.8 scrub ok 2024-09-11T03:41:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: osdmap e284: 12 total, 12 up, 10 in 2024-09-11T03:41:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: 2.3 scrub starts 2024-09-11T03:41:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: 2.3 scrub ok 2024-09-11T03:41:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: 2.4 scrub starts 2024-09-11T03:41:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: 2.4 scrub ok 2024-09-11T03:41:42.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: 2.5 scrub starts 2024-09-11T03:41:42.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: 2.5 scrub ok 2024-09-11T03:41:42.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: 2.13 scrub starts 2024-09-11T03:41:42.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: 2.13 scrub ok 2024-09-11T03:41:42.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: 2.12 scrub starts 2024-09-11T03:41:42.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: 2.12 scrub ok 2024-09-11T03:41:42.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:42.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: osdmap e285: 12 total, 12 up, 10 in 2024-09-11T03:41:42.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:42 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3065331716' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:42.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:42.424 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:42.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:42.525 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:42.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:42.548 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:42.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:42.649 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:42.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:42.672 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:42.673 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:42.773 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:42.795 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:42.796 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:42.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:41:42.860 INFO:tasks.rados.rados.2.smithi175.stdout:913: done (0 left) 2024-09-11T03:41:42.860 INFO:tasks.rados.rados.2.smithi175.stdout:914: rollback oid 37 current snap is 96 2024-09-11T03:41:42.860 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 37 to 91 2024-09-11T03:41:42.860 INFO:tasks.rados.rados.2.smithi175.stdout:915: snap_remove snap 94 2024-09-11T03:41:42.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T03:41:42.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:42.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:42.946 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:42.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:43.048 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:43.070 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:43.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:43.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:43.101 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:43.124 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:41:43.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:43 smithi121 ceph-mon[30729]: 2.17 scrub starts 2024-09-11T03:41:43.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:43 smithi121 ceph-mon[30729]: 2.17 scrub ok 2024-09-11T03:41:43.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:43 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:41:43.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:43 smithi121 ceph-mon[30729]: 2.0 scrub ok 2024-09-11T03:41:43.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:43 smithi121 ceph-mon[30729]: 2.16 scrub starts 2024-09-11T03:41:43.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:43 smithi121 ceph-mon[30729]: 2.16 scrub ok 2024-09-11T03:41:43.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:43 smithi121 ceph-mon[30729]: 2.6 scrub starts 2024-09-11T03:41:43.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:43 smithi121 ceph-mon[30729]: 2.6 scrub ok 2024-09-11T03:41:43.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:43 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:41:43.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:43 smithi121 ceph-mon[30729]: 2.14 scrub ok 2024-09-11T03:41:43.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:43 smithi121 ceph-mon[30729]: osdmap e286: 12 total, 12 up, 10 in 2024-09-11T03:41:43.173 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:43.194 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:43.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:43.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:43.296 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:43.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:43.321 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:43.321 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:43.348 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:43 smithi084 ceph-mon[27254]: 2.17 scrub starts 2024-09-11T03:41:43.348 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:43 smithi084 ceph-mon[27254]: 2.17 scrub ok 2024-09-11T03:41:43.348 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:43 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:41:43.348 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:43 smithi084 ceph-mon[27254]: 2.0 scrub ok 2024-09-11T03:41:43.348 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:43 smithi084 ceph-mon[27254]: 2.16 scrub starts 2024-09-11T03:41:43.348 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:43 smithi084 ceph-mon[27254]: 2.16 scrub ok 2024-09-11T03:41:43.348 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:43 smithi084 ceph-mon[27254]: 2.6 scrub starts 2024-09-11T03:41:43.348 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:43 smithi084 ceph-mon[27254]: 2.6 scrub ok 2024-09-11T03:41:43.349 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:43 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:41:43.349 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:43 smithi084 ceph-mon[27254]: 2.14 scrub ok 2024-09-11T03:41:43.349 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:43 smithi084 ceph-mon[27254]: osdmap e286: 12 total, 12 up, 10 in 2024-09-11T03:41:43.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:43.443 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:43.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:43.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:43 smithi175 ceph-mon[30894]: 2.17 scrub starts 2024-09-11T03:41:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:43 smithi175 ceph-mon[30894]: 2.17 scrub ok 2024-09-11T03:41:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:43 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:41:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:43 smithi175 ceph-mon[30894]: 2.0 scrub ok 2024-09-11T03:41:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:43 smithi175 ceph-mon[30894]: 2.16 scrub starts 2024-09-11T03:41:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:43 smithi175 ceph-mon[30894]: 2.16 scrub ok 2024-09-11T03:41:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:43 smithi175 ceph-mon[30894]: 2.6 scrub starts 2024-09-11T03:41:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:43 smithi175 ceph-mon[30894]: 2.6 scrub ok 2024-09-11T03:41:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:43 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:41:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:43 smithi175 ceph-mon[30894]: 2.14 scrub ok 2024-09-11T03:41:43.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:43 smithi175 ceph-mon[30894]: osdmap e286: 12 total, 12 up, 10 in 2024-09-11T03:41:43.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:43.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:43.567 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:43.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:43.577 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:41:43.577 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:41:43.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:41:43.577 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:41:43.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.578 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:228 2.9 e105/95 2:95fa802a:::smithi17557870-22:head v 105'20, mlcod=100'17)", 2024-09-11T03:41:43.578 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:41:43.578 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 181.60991744099999, 2024-09-11T03:41:43.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01695953, 2024-09-11T03:41:43.578 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.578 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.578 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:43.578 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:41:43.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:41:43.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:41:43.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2600000000000002e-07 2024-09-11T03:41:43.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982417+0000", 2024-09-11T03:41:43.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01547431 2024-09-11T03:41:43.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982440+0000", 2024-09-11T03:41:43.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2506000000000001e-05 2024-09-11T03:41:43.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982448+0000", 2024-09-11T03:41:43.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2989999999999997e-06 2024-09-11T03:41:43.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982562+0000", 2024-09-11T03:41:43.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114084 2024-09-11T03:41:43.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982583+0000", 2024-09-11T03:41:43.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1160999999999999e-05 2024-09-11T03:41:43.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:43.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.983852+0000", 2024-09-11T03:41:43.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001268953 2024-09-11T03:41:43.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.983902+0000", 2024-09-11T03:41:43.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9691000000000002e-05 2024-09-11T03:41:43.582 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.582 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.582 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.583 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:345 2.0 2:0762776a:::smithi17557870-25:head [write 742876~447756] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:41:43.583 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.827240+0000", 2024-09-11T03:41:43.583 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 173.749620126, 2024-09-11T03:41:43.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076328890000000003, 2024-09-11T03:41:43.583 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.583 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.583 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:43.583 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:43.583 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:43.583 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:43.583 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 345 2024-09-11T03:41:43.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.584 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.827240+0000", 2024-09-11T03:41:43.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.827240+0000", 2024-09-11T03:41:43.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.827243+0000", 2024-09-11T03:41:43.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0110000000000001e-06 2024-09-11T03:41:43.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828270+0000", 2024-09-11T03:41:43.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001026838 2024-09-11T03:41:43.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828298+0000", 2024-09-11T03:41:43.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8561999999999999e-05 2024-09-11T03:41:43.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828312+0000", 2024-09-11T03:41:43.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3781000000000001e-05 2024-09-11T03:41:43.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828398+0000", 2024-09-11T03:41:43.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6336999999999997e-05 2024-09-11T03:41:43.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:41:43.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828405+0000", 2024-09-11T03:41:43.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9079999999999998e-06 2024-09-11T03:41:43.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830015+0000", 2024-09-11T03:41:43.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001609344 2024-09-11T03:41:43.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830095+0000", 2024-09-11T03:41:43.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0259000000000005e-05 2024-09-11T03:41:43.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10", 2024-09-11T03:41:43.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830248+0000", 2024-09-11T03:41:43.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015351500000000001 2024-09-11T03:41:43.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:41:43.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.831760+0000", 2024-09-11T03:41:43.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015116540000000001 2024-09-11T03:41:43.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:43.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.832800+0000", 2024-09-11T03:41:43.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010400800000000001 2024-09-11T03:41:43.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:43.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834760+0000", 2024-09-11T03:41:43.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019600809999999998 2024-09-11T03:41:43.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:43.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834804+0000", 2024-09-11T03:41:43.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3922999999999999e-05 2024-09-11T03:41:43.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834873+0000", 2024-09-11T03:41:43.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8596000000000005e-05 2024-09-11T03:41:43.589 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.590 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.590 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.590 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:346 2.0 2:0762776a:::smithi17557870-25:head [write 1729599~575687] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:41:43.590 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.833957+0000", 2024-09-11T03:41:43.590 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 173.74290340100001, 2024-09-11T03:41:43.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00864616, 2024-09-11T03:41:43.590 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.590 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.590 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:43.590 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:43.590 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:43.590 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:43.591 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 346 2024-09-11T03:41:43.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.591 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833957+0000", 2024-09-11T03:41:43.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833957+0000", 2024-09-11T03:41:43.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833958+0000", 2024-09-11T03:41:43.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.674e-06 2024-09-11T03:41:43.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838969+0000", 2024-09-11T03:41:43.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005010993 2024-09-11T03:41:43.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839016+0000", 2024-09-11T03:41:43.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6576000000000003e-05 2024-09-11T03:41:43.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839037+0000", 2024-09-11T03:41:43.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1356e-05 2024-09-11T03:41:43.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839197+0000", 2024-09-11T03:41:43.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000159338 2024-09-11T03:41:43.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839266+0000", 2024-09-11T03:41:43.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9758999999999994e-05 2024-09-11T03:41:43.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10", 2024-09-11T03:41:43.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839374+0000", 2024-09-11T03:41:43.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010791899999999999 2024-09-11T03:41:43.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:41:43.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.840763+0000", 2024-09-11T03:41:43.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013883999999999999 2024-09-11T03:41:43.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:43.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842254+0000", 2024-09-11T03:41:43.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001491409 2024-09-11T03:41:43.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:43.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842538+0000", 2024-09-11T03:41:43.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028420099999999998 2024-09-11T03:41:43.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:43.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842573+0000", 2024-09-11T03:41:43.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4381000000000002e-05 2024-09-11T03:41:43.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842603+0000", 2024-09-11T03:41:43.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0154e-05 2024-09-11T03:41:43.598 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.598 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.598 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.599 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:428 2.9 e119/95 2:95fa802a:::smithi17557870-22:head v 119'41, mlcod=113'37)", 2024-09-11T03:41:43.599 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:55.867789+0000", 2024-09-11T03:41:43.599 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 167.70907121799999, 2024-09-11T03:41:43.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053342210000000001, 2024-09-11T03:41:43.599 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.599 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.599 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:43.599 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.867789+0000", 2024-09-11T03:41:43.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.867789+0000", 2024-09-11T03:41:43.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.867789+0000", 2024-09-11T03:41:43.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8099999999999999e-07 2024-09-11T03:41:43.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.871598+0000", 2024-09-11T03:41:43.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038090229999999999 2024-09-11T03:41:43.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.871611+0000", 2024-09-11T03:41:43.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3474e-05 2024-09-11T03:41:43.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.871620+0000", 2024-09-11T03:41:43.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.191e-06 2024-09-11T03:41:43.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.871672+0000", 2024-09-11T03:41:43.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2074999999999999e-05 2024-09-11T03:41:43.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.871690+0000", 2024-09-11T03:41:43.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8601e-05 2024-09-11T03:41:43.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:43.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.873081+0000", 2024-09-11T03:41:43.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013902630000000001 2024-09-11T03:41:43.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.873123+0000", 2024-09-11T03:41:43.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2413000000000002e-05 2024-09-11T03:41:43.604 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.604 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.604 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.604 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:623 2.8 2:162357fa:::smithi17557870-18:head [read 0~3563051,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e130)", 2024-09-11T03:41:43.604 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:07.066819+0000", 2024-09-11T03:41:43.604 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 156.51004089099999, 2024-09-11T03:41:43.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058140739999999998, 2024-09-11T03:41:43.604 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.604 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.604 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:41:43.604 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:43.605 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:43.605 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:43.605 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 623 2024-09-11T03:41:43.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.605 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066819+0000", 2024-09-11T03:41:43.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066819+0000", 2024-09-11T03:41:43.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066820+0000", 2024-09-11T03:41:43.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3900000000000005e-07 2024-09-11T03:41:43.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066823+0000", 2024-09-11T03:41:43.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4750000000000002e-06 2024-09-11T03:41:43.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066840+0000", 2024-09-11T03:41:43.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6603999999999999e-05 2024-09-11T03:41:43.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066846+0000", 2024-09-11T03:41:43.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5409999999999997e-06 2024-09-11T03:41:43.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.071239+0000", 2024-09-11T03:41:43.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043923039999999997 2024-09-11T03:41:43.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.071311+0000", 2024-09-11T03:41:43.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2408000000000002e-05 2024-09-11T03:41:43.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.072633+0000", 2024-09-11T03:41:43.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013222030000000001 2024-09-11T03:41:43.608 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.608 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.608 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.608 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:624 2.8 2:162357fa:::smithi17557870-18:head [sparse-read 0~1957854] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e130)", 2024-09-11T03:41:43.608 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:07.066857+0000", 2024-09-11T03:41:43.609 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 156.510002641, 2024-09-11T03:41:43.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059990820000000002, 2024-09-11T03:41:43.609 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.609 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.609 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:41:43.609 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:43.609 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:43.609 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:43.609 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 624 2024-09-11T03:41:43.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.609 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066857+0000", 2024-09-11T03:41:43.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066857+0000", 2024-09-11T03:41:43.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066858+0000", 2024-09-11T03:41:43.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1699999999999998e-07 2024-09-11T03:41:43.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066861+0000", 2024-09-11T03:41:43.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2210000000000002e-06 2024-09-11T03:41:43.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066870+0000", 2024-09-11T03:41:43.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8149999999999995e-06 2024-09-11T03:41:43.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066883+0000", 2024-09-11T03:41:43.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3220999999999999e-05 2024-09-11T03:41:43.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.072671+0000", 2024-09-11T03:41:43.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0057879910000000001 2024-09-11T03:41:43.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.072735+0000", 2024-09-11T03:41:43.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3414e-05 2024-09-11T03:41:43.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.072856+0000", 2024-09-11T03:41:43.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000121803 2024-09-11T03:41:43.613 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.613 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.613 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.613 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:744 2.0 2:0762776a:::smithi17557870-25:head [sparse-read 0~1885763,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e137)", 2024-09-11T03:41:43.613 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:13.831210+0000", 2024-09-11T03:41:43.613 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 149.745649886, 2024-09-11T03:41:43.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053191080000000003, 2024-09-11T03:41:43.613 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.613 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.614 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:41:43.614 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:43.614 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:43.614 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:43.614 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 744 2024-09-11T03:41:43.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.614 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831210+0000", 2024-09-11T03:41:43.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831210+0000", 2024-09-11T03:41:43.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831212+0000", 2024-09-11T03:41:43.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8279999999999999e-06 2024-09-11T03:41:43.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831220+0000", 2024-09-11T03:41:43.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4770000000000003e-06 2024-09-11T03:41:43.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831237+0000", 2024-09-11T03:41:43.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6648000000000001e-05 2024-09-11T03:41:43.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831249+0000", 2024-09-11T03:41:43.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1472e-05 2024-09-11T03:41:43.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831304+0000", 2024-09-11T03:41:43.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5954e-05 2024-09-11T03:41:43.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:41:43.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831309+0000", 2024-09-11T03:41:43.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5040000000000004e-06 2024-09-11T03:41:43.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.832366+0000", 2024-09-11T03:41:43.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001056876 2024-09-11T03:41:43.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.832464+0000", 2024-09-11T03:41:43.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7978000000000003e-05 2024-09-11T03:41:43.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.836529+0000", 2024-09-11T03:41:43.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040653709999999999 2024-09-11T03:41:43.618 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.619 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.619 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:745 2.0 2:0762776a:::smithi17557870-25:head [sparse-read 0~1885763] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e137)", 2024-09-11T03:41:43.619 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:13.831271+0000", 2024-09-11T03:41:43.619 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 149.745588744, 2024-09-11T03:41:43.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054786310000000003, 2024-09-11T03:41:43.619 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.619 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.619 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:41:43.619 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:43.619 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:43.620 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:43.620 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 745 2024-09-11T03:41:43.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.620 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831271+0000", 2024-09-11T03:41:43.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831271+0000", 2024-09-11T03:41:43.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831272+0000", 2024-09-11T03:41:43.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0800000000000005e-07 2024-09-11T03:41:43.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831274+0000", 2024-09-11T03:41:43.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.593e-06 2024-09-11T03:41:43.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831280+0000", 2024-09-11T03:41:43.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.587e-06 2024-09-11T03:41:43.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831286+0000", 2024-09-11T03:41:43.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5820000000000001e-06 2024-09-11T03:41:43.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831368+0000", 2024-09-11T03:41:43.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2207999999999996e-05 2024-09-11T03:41:43.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:41:43.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831374+0000", 2024-09-11T03:41:43.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.443e-06 2024-09-11T03:41:43.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.836570+0000", 2024-09-11T03:41:43.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051954840000000002 2024-09-11T03:41:43.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.836643+0000", 2024-09-11T03:41:43.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3596000000000005e-05 2024-09-11T03:41:43.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.836750+0000", 2024-09-11T03:41:43.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010663 2024-09-11T03:41:43.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.624 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.624 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:746 2.0 2:0762776a:::smithi17557870-25:head [sparse-read 0~1885763] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e137)", 2024-09-11T03:41:43.625 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:13.831294+0000", 2024-09-11T03:41:43.625 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 149.74556564599999, 2024-09-11T03:41:43.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0057346979999999999, 2024-09-11T03:41:43.625 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.625 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.625 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:41:43.625 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:43.625 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:43.625 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:43.625 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 746 2024-09-11T03:41:43.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.625 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831294+0000", 2024-09-11T03:41:43.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831294+0000", 2024-09-11T03:41:43.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831295+0000", 2024-09-11T03:41:43.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7899999999999999e-07 2024-09-11T03:41:43.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831297+0000", 2024-09-11T03:41:43.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2019999999999998e-06 2024-09-11T03:41:43.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831303+0000", 2024-09-11T03:41:43.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.139e-06 2024-09-11T03:41:43.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831312+0000", 2024-09-11T03:41:43.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1379999999999996e-06 2024-09-11T03:41:43.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831406+0000", 2024-09-11T03:41:43.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4178999999999994e-05 2024-09-11T03:41:43.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:41:43.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831411+0000", 2024-09-11T03:41:43.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7890000000000002e-06 2024-09-11T03:41:43.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.836793+0000", 2024-09-11T03:41:43.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053814850000000001 2024-09-11T03:41:43.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.836892+0000", 2024-09-11T03:41:43.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8987999999999995e-05 2024-09-11T03:41:43.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.837029+0000", 2024-09-11T03:41:43.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013729899999999999 2024-09-11T03:41:43.630 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.630 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.630 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.630 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:985 2.b e163/95 2:dde92fc6:::smithi17557870-41:head v 163'52, mlcod=158'48)", 2024-09-11T03:41:43.630 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:39.932630+0000", 2024-09-11T03:41:43.630 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 123.64423043399999, 2024-09-11T03:41:43.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056057540000000001, 2024-09-11T03:41:43.631 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.631 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.631 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:43.631 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.932630+0000", 2024-09-11T03:41:43.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.932630+0000", 2024-09-11T03:41:43.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.932630+0000", 2024-09-11T03:41:43.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7000000000000001e-07 2024-09-11T03:41:43.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.935963+0000", 2024-09-11T03:41:43.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033333339999999999 2024-09-11T03:41:43.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.935973+0000", 2024-09-11T03:41:43.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9690000000000006e-06 2024-09-11T03:41:43.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.935980+0000", 2024-09-11T03:41:43.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.596e-06 2024-09-11T03:41:43.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.936026+0000", 2024-09-11T03:41:43.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5850000000000003e-05 2024-09-11T03:41:43.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.936038+0000", 2024-09-11T03:41:43.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2408999999999999e-05 2024-09-11T03:41:43.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:43.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.938200+0000", 2024-09-11T03:41:43.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021619260000000002 2024-09-11T03:41:43.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.938235+0000", 2024-09-11T03:41:43.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.54e-05 2024-09-11T03:41:43.634 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.635 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.635 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.635 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1048 2.9 e175/95 2:98bb65f5:::smithi17557870-6:head v 175'134, mlcod=168'130)", 2024-09-11T03:41:43.635 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.832944+0000", 2024-09-11T03:41:43.635 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.743916447, 2024-09-11T03:41:43.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058696390000000003, 2024-09-11T03:41:43.635 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.635 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.635 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:43.635 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.832944+0000", 2024-09-11T03:41:43.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.832944+0000", 2024-09-11T03:41:43.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.832944+0000", 2024-09-11T03:41:43.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.72e-07 2024-09-11T03:41:43.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.836610+0000", 2024-09-11T03:41:43.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036663709999999999 2024-09-11T03:41:43.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.836624+0000", 2024-09-11T03:41:43.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3419e-05 2024-09-11T03:41:43.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.836634+0000", 2024-09-11T03:41:43.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0193e-05 2024-09-11T03:41:43.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.836684+0000", 2024-09-11T03:41:43.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0188000000000001e-05 2024-09-11T03:41:43.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.836702+0000", 2024-09-11T03:41:43.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8567999999999999e-05 2024-09-11T03:41:43.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:43.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.838767+0000", 2024-09-11T03:41:43.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002064986 2024-09-11T03:41:43.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.838813+0000", 2024-09-11T03:41:43.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5741999999999997e-05 2024-09-11T03:41:43.639 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.639 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.639 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.639 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1152 2.b e183/95 2:de6b1316:::smithi17557870-32:head v 183'96, mlcod=182'92)", 2024-09-11T03:41:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:59.878348+0000", 2024-09-11T03:41:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 103.698512114, 2024-09-11T03:41:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064294529999999999, 2024-09-11T03:41:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.878348+0000", 2024-09-11T03:41:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.878348+0000", 2024-09-11T03:41:43.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.878348+0000", 2024-09-11T03:41:43.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8900000000000001e-07 2024-09-11T03:41:43.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.882607+0000", 2024-09-11T03:41:43.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042589139999999999 2024-09-11T03:41:43.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.882621+0000", 2024-09-11T03:41:43.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3925e-05 2024-09-11T03:41:43.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.882628+0000", 2024-09-11T03:41:43.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6000000000000003e-06 2024-09-11T03:41:43.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.882686+0000", 2024-09-11T03:41:43.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8579999999999998e-05 2024-09-11T03:41:43.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.882703+0000", 2024-09-11T03:41:43.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7167999999999999e-05 2024-09-11T03:41:43.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:43.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.884729+0000", 2024-09-11T03:41:43.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020253720000000001 2024-09-11T03:41:43.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.884777+0000", 2024-09-11T03:41:43.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8705000000000002e-05 2024-09-11T03:41:43.644 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.644 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.644 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.644 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1161 2.8 2:1bdf1a36:::smithi17557870-36:head [delete] snapc 2e=[2e,2c,2b,28,26] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:41:43.644 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:01.831716+0000", 2024-09-11T03:41:43.644 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 101.74514372500001, 2024-09-11T03:41:43.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065461870000000002, 2024-09-11T03:41:43.644 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.644 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.644 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:43.645 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:43.645 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:43.645 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:43.645 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1161 2024-09-11T03:41:43.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.645 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831716+0000", 2024-09-11T03:41:43.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831716+0000", 2024-09-11T03:41:43.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831718+0000", 2024-09-11T03:41:43.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1890000000000001e-06 2024-09-11T03:41:43.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831726+0000", 2024-09-11T03:41:43.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5190000000000003e-06 2024-09-11T03:41:43.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831741+0000", 2024-09-11T03:41:43.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5407e-05 2024-09-11T03:41:43.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831749+0000", 2024-09-11T03:41:43.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0129999999999993e-06 2024-09-11T03:41:43.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831837+0000", 2024-09-11T03:41:43.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7558000000000004e-05 2024-09-11T03:41:43.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:41:43.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831845+0000", 2024-09-11T03:41:43.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3799999999999994e-06 2024-09-11T03:41:43.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.834626+0000", 2024-09-11T03:41:43.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027804140000000001 2024-09-11T03:41:43.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.834743+0000", 2024-09-11T03:41:43.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117569 2024-09-11T03:41:43.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,9", 2024-09-11T03:41:43.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.834902+0000", 2024-09-11T03:41:43.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015838700000000001 2024-09-11T03:41:43.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:41:43.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.835777+0000", 2024-09-11T03:41:43.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087511700000000002 2024-09-11T03:41:43.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:43.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.836462+0000", 2024-09-11T03:41:43.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000684686 2024-09-11T03:41:43.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:43.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.838188+0000", 2024-09-11T03:41:43.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001726193 2024-09-11T03:41:43.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:43.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.838229+0000", 2024-09-11T03:41:43.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0991000000000001e-05 2024-09-11T03:41:43.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.838262+0000", 2024-09-11T03:41:43.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3763999999999997e-05 2024-09-11T03:41:43.651 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.651 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.651 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.651 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1555 2.3 e214/95 2:c6c49f2c:::smithi17557870-44:head v 214'74, mlcod=198'70)", 2024-09-11T03:41:43.651 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:30.893545+0000", 2024-09-11T03:41:43.651 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.683314523000007, 2024-09-11T03:41:43.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070433960000000004, 2024-09-11T03:41:43.651 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.651 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.651 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:43.652 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.893545+0000", 2024-09-11T03:41:43.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.893545+0000", 2024-09-11T03:41:43.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.893546+0000", 2024-09-11T03:41:43.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7500000000000001e-07 2024-09-11T03:41:43.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895656+0000", 2024-09-11T03:41:43.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002110476 2024-09-11T03:41:43.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895679+0000", 2024-09-11T03:41:43.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2795000000000001e-05 2024-09-11T03:41:43.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895692+0000", 2024-09-11T03:41:43.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3141e-05 2024-09-11T03:41:43.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895758+0000", 2024-09-11T03:41:43.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5871999999999995e-05 2024-09-11T03:41:43.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895786+0000", 2024-09-11T03:41:43.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8391e-05 2024-09-11T03:41:43.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:43.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900550+0000", 2024-09-11T03:41:43.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047632869999999997 2024-09-11T03:41:43.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900589+0000", 2024-09-11T03:41:43.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9159000000000002e-05 2024-09-11T03:41:43.656 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.656 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.656 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.656 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1609 2.8 2:1bdf1a36:::smithi17557870-36:head [write 618557~727148] snapc 40=[40,3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:41:43.656 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:35.905306+0000", 2024-09-11T03:41:43.657 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 67.671554454000002, 2024-09-11T03:41:43.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069719600000000001, 2024-09-11T03:41:43.657 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.657 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.657 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:43.657 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:43.657 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:43.657 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:43.657 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1609 2024-09-11T03:41:43.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.657 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.905306+0000", 2024-09-11T03:41:43.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.905306+0000", 2024-09-11T03:41:43.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.905307+0000", 2024-09-11T03:41:43.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.482e-06 2024-09-11T03:41:43.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907145+0000", 2024-09-11T03:41:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001838412 2024-09-11T03:41:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907157+0000", 2024-09-11T03:41:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-09-11T03:41:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907165+0000", 2024-09-11T03:41:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6340000000000001e-06 2024-09-11T03:41:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907215+0000", 2024-09-11T03:41:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0760999999999997e-05 2024-09-11T03:41:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907282+0000", 2024-09-11T03:41:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6934000000000007e-05 2024-09-11T03:41:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,9", 2024-09-11T03:41:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907400+0000", 2024-09-11T03:41:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117625 2024-09-11T03:41:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:41:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.908820+0000", 2024-09-11T03:41:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0014204140000000001 2024-09-11T03:41:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.910145+0000", 2024-09-11T03:41:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013246250000000001 2024-09-11T03:41:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.912194+0000", 2024-09-11T03:41:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020485640000000001 2024-09-11T03:41:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.912229+0000", 2024-09-11T03:41:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5128e-05 2024-09-11T03:41:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.912277+0000", 2024-09-11T03:41:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8746999999999999e-05 2024-09-11T03:41:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1868 2.0 2:0762776a:::smithi17557870-25:head [copy-from ver 218] snapc 4c=[4c,4b,49,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:41:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:01.850162+0000", 2024-09-11T03:41:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.726698124999999, 2024-09-11T03:41:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020677853999999999, 2024-09-11T03:41:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1868 2024-09-11T03:41:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850162+0000", 2024-09-11T03:41:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850162+0000", 2024-09-11T03:41:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850164+0000", 2024-09-11T03:41:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.588e-06 2024-09-11T03:41:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850174+0000", 2024-09-11T03:41:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8300000000000008e-06 2024-09-11T03:41:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850193+0000", 2024-09-11T03:41:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8346e-05 2024-09-11T03:41:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850204+0000", 2024-09-11T03:41:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1144e-05 2024-09-11T03:41:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850285+0000", 2024-09-11T03:41:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1525000000000003e-05 2024-09-11T03:41:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:41:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850293+0000", 2024-09-11T03:41:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.072e-06 2024-09-11T03:41:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.852390+0000", 2024-09-11T03:41:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002096601 2024-09-11T03:41:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.852527+0000", 2024-09-11T03:41:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013703899999999999 2024-09-11T03:41:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,11", 2024-09-11T03:41:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.862911+0000", 2024-09-11T03:41:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01038392 2024-09-11T03:41:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:41:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.867006+0000", 2024-09-11T03:41:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004095291 2024-09-11T03:41:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.868758+0000", 2024-09-11T03:41:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017514869999999999 2024-09-11T03:41:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.870714+0000", 2024-09-11T03:41:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019563229999999998 2024-09-11T03:41:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.870780+0000", 2024-09-11T03:41:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6459999999999997e-05 2024-09-11T03:41:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.870840+0000", 2024-09-11T03:41:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9228000000000003e-05 2024-09-11T03:41:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1869 2.0 2:0762776a:::smithi17557870-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:41:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:01.850232+0000", 2024-09-11T03:41:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.726627540999999, 2024-09-11T03:41:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016419715000000001, 2024-09-11T03:41:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:41:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1869 2024-09-11T03:41:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850232+0000", 2024-09-11T03:41:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850232+0000", 2024-09-11T03:41:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850233+0000", 2024-09-11T03:41:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5300000000000004e-07 2024-09-11T03:41:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850237+0000", 2024-09-11T03:41:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3660000000000002e-06 2024-09-11T03:41:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850247+0000", 2024-09-11T03:41:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.38e-06 2024-09-11T03:41:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850254+0000", 2024-09-11T03:41:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2099999999999996e-06 2024-09-11T03:41:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850365+0000", 2024-09-11T03:41:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110499 2024-09-11T03:41:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:41:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850369+0000", 2024-09-11T03:41:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.454e-06 2024-09-11T03:41:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.852747+0000", 2024-09-11T03:41:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023778480000000001 2024-09-11T03:41:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:41:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.852806+0000", 2024-09-11T03:41:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8931999999999998e-05 2024-09-11T03:41:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.866380+0000", 2024-09-11T03:41:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013574693000000001 2024-09-11T03:41:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.866512+0000", 2024-09-11T03:41:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013140999999999999 2024-09-11T03:41:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.866652+0000", 2024-09-11T03:41:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014027000000000001 2024-09-11T03:41:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1877 2.b e247/95 2:de6b1316:::smithi17557870-32:head v 247'232, mlcod=243'228)", 2024-09-11T03:41:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:03.916185+0000", 2024-09-11T03:41:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 39.660675011000002, 2024-09-11T03:41:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010307996, 2024-09-11T03:41:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.916185+0000", 2024-09-11T03:41:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.916185+0000", 2024-09-11T03:41:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.916185+0000", 2024-09-11T03:41:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1699999999999998e-07 2024-09-11T03:41:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924225+0000", 2024-09-11T03:41:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080395750000000002 2024-09-11T03:41:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924243+0000", 2024-09-11T03:41:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7955e-05 2024-09-11T03:41:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924271+0000", 2024-09-11T03:41:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7797000000000002e-05 2024-09-11T03:41:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924349+0000", 2024-09-11T03:41:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8137000000000001e-05 2024-09-11T03:41:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924370+0000", 2024-09-11T03:41:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1226e-05 2024-09-11T03:41:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.926454+0000", 2024-09-11T03:41:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002083438 2024-09-11T03:41:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.926493+0000", 2024-09-11T03:41:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9351000000000003e-05 2024-09-11T03:41:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1989 2.0 2:0762776a:::smithi17557870-25:head [copy-from ver 162] snapc 52=[52,51,4e,3d] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-11T03:41:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:14.949188+0000", 2024-09-11T03:41:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.627671944999999, 2024-09-11T03:41:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014886257999999999, 2024-09-11T03:41:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1989 2024-09-11T03:41:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949188+0000", 2024-09-11T03:41:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949188+0000", 2024-09-11T03:41:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949191+0000", 2024-09-11T03:41:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.706e-06 2024-09-11T03:41:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949204+0000", 2024-09-11T03:41:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3142999999999999e-05 2024-09-11T03:41:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949227+0000", 2024-09-11T03:41:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3320999999999999e-05 2024-09-11T03:41:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949238+0000", 2024-09-11T03:41:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0352e-05 2024-09-11T03:41:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949313+0000", 2024-09-11T03:41:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5450000000000004e-05 2024-09-11T03:41:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:41:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949319+0000", 2024-09-11T03:41:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9279999999999998e-06 2024-09-11T03:41:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.950847+0000", 2024-09-11T03:41:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015282690000000001 2024-09-11T03:41:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.950916+0000", 2024-09-11T03:41:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8559000000000005e-05 2024-09-11T03:41:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,11", 2024-09-11T03:41:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.956873+0000", 2024-09-11T03:41:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059576359999999997 2024-09-11T03:41:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:41:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.960470+0000", 2024-09-11T03:41:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035970749999999999 2024-09-11T03:41:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.963875+0000", 2024-09-11T03:41:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034049050000000002 2024-09-11T03:41:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.963958+0000", 2024-09-11T03:41:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2769999999999995e-05 2024-09-11T03:41:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:41:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.964021+0000", 2024-09-11T03:41:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2442999999999998e-05 2024-09-11T03:41:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.964074+0000", 2024-09-11T03:41:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3701e-05 2024-09-11T03:41:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1990 2.0 2:0762776a:::smithi17557870-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e258)", 2024-09-11T03:41:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:14.949265+0000", 2024-09-11T03:41:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.627594903999999, 2024-09-11T03:41:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010821580000000001, 2024-09-11T03:41:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:41:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:41:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:41:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:41:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:41:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1990 2024-09-11T03:41:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:41:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:41:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949265+0000", 2024-09-11T03:41:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:41:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949265+0000", 2024-09-11T03:41:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:41:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:41:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949266+0000", 2024-09-11T03:41:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3000000000000001e-07 2024-09-11T03:41:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:41:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949268+0000", 2024-09-11T03:41:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2249999999999999e-06 2024-09-11T03:41:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:41:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949273+0000", 2024-09-11T03:41:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6389999999999997e-06 2024-09-11T03:41:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:41:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949277+0000", 2024-09-11T03:41:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4630000000000001e-06 2024-09-11T03:41:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949380+0000", 2024-09-11T03:41:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103099 2024-09-11T03:41:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:41:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949387+0000", 2024-09-11T03:41:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9290000000000001e-06 2024-09-11T03:41:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.951035+0000", 2024-09-11T03:41:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001647995 2024-09-11T03:41:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:41:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.951082+0000", 2024-09-11T03:41:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7231999999999998e-05 2024-09-11T03:41:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:41:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.959865+0000", 2024-09-11T03:41:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087832770000000008 2024-09-11T03:41:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:41:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.959967+0000", 2024-09-11T03:41:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010153 2024-09-11T03:41:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:41:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:41:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:41:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.960087+0000", 2024-09-11T03:41:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011976099999999999 2024-09-11T03:41:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:41:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:41:43.695 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:41:43.695 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:43.720 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:43.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:43.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:43.822 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:43.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:43.845 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:43.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:43.858 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:41:43.862 INFO:tasks.rados.rados.2.smithi175.stdout:915: done (1 left) 2024-09-11T03:41:43.862 INFO:tasks.rados.rados.2.smithi175.stdout:916: rollback oid 38 current snap is 96 2024-09-11T03:41:43.862 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 38 to 91 2024-09-11T03:41:43.862 INFO:tasks.rados.rados.2.smithi175.stdout:914: finishing rollback tid 0 to smithi17557870-37 2024-09-11T03:41:43.862 INFO:tasks.rados.rados.2.smithi175.stdout:914: finishing rollback tid 1 to smithi17557870-37 2024-09-11T03:41:43.862 INFO:tasks.rados.rados.2.smithi175.stdout:914: finishing rollback tid 2 to smithi17557870-37 2024-09-11T03:41:43.862 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 127 (ObjNum 153 snap 35 seq_num 153) dirty exists 2024-09-11T03:41:43.862 INFO:tasks.rados.rados.2.smithi175.stdout:914: done (1 left) 2024-09-11T03:41:43.862 INFO:tasks.rados.rados.2.smithi175.stdout:917: snap_create 2024-09-11T03:41:43.863 INFO:tasks.rados.rados.2.smithi175.stdout:916: finishing rollback tid 1 to smithi17557870-38 2024-09-11T03:41:43.863 INFO:tasks.rados.rados.2.smithi175.stdout:916: finishing rollback tid 2 to smithi17557870-38 2024-09-11T03:41:43.863 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 304 (ObjNum 318 snap 86 seq_num 318) dirty exists 2024-09-11T03:41:43.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:43.966 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:43.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:43.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:44.068 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:44.089 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:44.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:44.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:44.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:44 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:41:44.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:44 smithi084 ceph-mon[27254]: 2.7 scrub ok 2024-09-11T03:41:44.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:44 smithi084 ceph-mon[27254]: pgmap v583: 57 pgs: 7 peering, 50 active+clean; 171 MiB data, 647 MiB used, 893 GiB / 894 GiB avail 2024-09-11T03:41:44.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:44 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:41:44.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:44 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:41:44.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:44 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:41:44.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:44 smithi084 ceph-mon[27254]: 2.15 scrub ok 2024-09-11T03:41:44.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:44 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/31818316' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:44.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:44 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:41:44.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:44 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:41:44.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:44 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/31818316' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:44.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:44 smithi084 ceph-mon[27254]: osdmap e287: 12 total, 12 up, 10 in 2024-09-11T03:41:44.191 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:44.219 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:44.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:44.221 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:44.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:44.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:44.344 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:44.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:44 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:41:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:44 smithi121 ceph-mon[30729]: 2.7 scrub ok 2024-09-11T03:41:44.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:44 smithi121 ceph-mon[30729]: pgmap v583: 57 pgs: 7 peering, 50 active+clean; 171 MiB data, 647 MiB used, 893 GiB / 894 GiB avail 2024-09-11T03:41:44.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:44 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:41:44.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:44 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:41:44.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:44 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:41:44.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:44 smithi121 ceph-mon[30729]: 2.15 scrub ok 2024-09-11T03:41:44.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:44 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/31818316' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:44.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:44 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:41:44.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:44 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:41:44.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:44 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/31818316' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:44.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:44 smithi121 ceph-mon[30729]: osdmap e287: 12 total, 12 up, 10 in 2024-09-11T03:41:44.445 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:44.468 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:44.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:44.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:44 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:41:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:44 smithi175 ceph-mon[30894]: 2.7 scrub ok 2024-09-11T03:41:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:44 smithi175 ceph-mon[30894]: pgmap v583: 57 pgs: 7 peering, 50 active+clean; 171 MiB data, 647 MiB used, 893 GiB / 894 GiB avail 2024-09-11T03:41:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:44 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:41:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:44 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:41:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:44 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:41:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:44 smithi175 ceph-mon[30894]: 2.15 scrub ok 2024-09-11T03:41:44.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:44 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/31818316' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:44.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:44 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:41:44.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:44 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:41:44.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:44 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/31818316' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:44.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:44 smithi175 ceph-mon[30894]: osdmap e287: 12 total, 12 up, 10 in 2024-09-11T03:41:44.479 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-11T03:41:44.570 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:44.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:44.594 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:44.595 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:44.696 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:44.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:44.718 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:44.718 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:44.735 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:41:44.819 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:44.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:44.842 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:44.843 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:44.862 INFO:tasks.rados.rados.2.smithi175.stdout:916: done (1 left) 2024-09-11T03:41:44.863 INFO:tasks.rados.rados.2.smithi175.stdout:917: done (0 left) 2024-09-11T03:41:44.863 INFO:tasks.rados.rados.2.smithi175.stdout:918: snap_create 2024-09-11T03:41:44.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:44.967 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:44.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:44.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:45.039 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:41:45.069 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:45.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:45.104 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:45.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:45.183 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:41:45.183 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:41:45.183 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:41:45.183 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:41:45.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:45.228 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:45.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:45.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:45.284 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:45.330 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:45.351 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:45.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:45.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:45.477 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:45.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:45.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:45.579 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:45.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:45.601 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:45.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:45.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:45.728 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:45.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:45.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:45.775 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:41:45.775 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:41:45.780 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:41:45.780 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:41:45.796 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:41:45.796 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:41:45.812 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:41:45.812 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:41:45.829 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:41:45.829 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:41:45.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:45.845 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:41:45.846 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:41:45.855 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:45.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:45.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:45.863 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:41:45.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:41:45.872 INFO:tasks.rados.rados.2.smithi175.stdout:918: done (0 left) 2024-09-11T03:41:45.872 INFO:tasks.rados.rados.2.smithi175.stdout:919: snap_remove snap 84 2024-09-11T03:41:45.880 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:41:45.880 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:41:45.897 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:41:45.897 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:41:45.915 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:41:45.915 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:41:45.932 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:41:45.932 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:41:45.932 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:45 smithi084 ceph-mon[27254]: pgmap v586: 57 pgs: 57 active+clean; 157 MiB data, 648 MiB used, 893 GiB / 894 GiB avail 2024-09-11T03:41:45.933 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:45 smithi084 ceph-mon[27254]: osdmap e288: 12 total, 12 up, 10 in 2024-09-11T03:41:45.933 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:45 smithi084 ceph-mon[27254]: 1.13 scrub starts 2024-09-11T03:41:45.933 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:45 smithi084 ceph-mon[27254]: 1.13 scrub ok 2024-09-11T03:41:45.933 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:45 smithi084 ceph-mon[27254]: 1.a scrub starts 2024-09-11T03:41:45.933 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:45 smithi084 ceph-mon[27254]: 1.a scrub ok 2024-09-11T03:41:45.933 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:45 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/573660179' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:45.941 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-11T03:41:45.948 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:41:45.948 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:41:45.957 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:46.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:46.005 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:46.005 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:46.106 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:46.128 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:46.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:46.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:46.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:45 smithi121 ceph-mon[30729]: pgmap v586: 57 pgs: 57 active+clean; 157 MiB data, 648 MiB used, 893 GiB / 894 GiB avail 2024-09-11T03:41:46.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:45 smithi121 ceph-mon[30729]: osdmap e288: 12 total, 12 up, 10 in 2024-09-11T03:41:46.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:45 smithi121 ceph-mon[30729]: 1.13 scrub starts 2024-09-11T03:41:46.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:45 smithi121 ceph-mon[30729]: 1.13 scrub ok 2024-09-11T03:41:46.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:45 smithi121 ceph-mon[30729]: 1.a scrub starts 2024-09-11T03:41:46.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:45 smithi121 ceph-mon[30729]: 1.a scrub ok 2024-09-11T03:41:46.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:45 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/573660179' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:46.226 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:41:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:45 smithi175 ceph-mon[30894]: pgmap v586: 57 pgs: 57 active+clean; 157 MiB data, 648 MiB used, 893 GiB / 894 GiB avail 2024-09-11T03:41:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:45 smithi175 ceph-mon[30894]: osdmap e288: 12 total, 12 up, 10 in 2024-09-11T03:41:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:45 smithi175 ceph-mon[30894]: 1.13 scrub starts 2024-09-11T03:41:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:45 smithi175 ceph-mon[30894]: 1.13 scrub ok 2024-09-11T03:41:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:45 smithi175 ceph-mon[30894]: 1.a scrub starts 2024-09-11T03:41:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:45 smithi175 ceph-mon[30894]: 1.a scrub ok 2024-09-11T03:41:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:45 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/573660179' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:46.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:46.256 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:46.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:46.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:46.358 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:46.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:46.381 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:46.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:46.450 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [6, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:41:46.451 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-11T03:41:46.478 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:41:46.482 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:46.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:46.518 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:46.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:46.620 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:46.643 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:46.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:46.673 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:41:46.673 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:41:46.730 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:46.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:46.755 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:46.769 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:46.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:46.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:46.863 INFO:tasks.rados.rados.2.smithi175.stdout:919: done (0 left) 2024-09-11T03:41:46.863 INFO:tasks.rados.rados.2.smithi175.stdout:920: write (excl) oid 10 current snap is 98 2024-09-11T03:41:46.863 INFO:tasks.rados.rados.2.smithi175.stdout:920: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2024-09-11T03:41:46.868 INFO:tasks.rados.rados.2.smithi175.stdout:920: writing smithi17557870-10 from 463959 to 942404 tid 1 2024-09-11T03:41:46.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:46.876 INFO:tasks.rados.rados.2.smithi175.stdout:920: writing smithi17557870-10 from 1570667 to 2301775 tid 2 2024-09-11T03:41:46.883 INFO:tasks.rados.rados.2.smithi175.stdout:920: writing smithi17557870-10 from 3038372 to 3530805 tid 3 2024-09-11T03:41:46.883 INFO:tasks.rados.rados.2.smithi175.stdout:921: rollback oid 5 current snap is 98 2024-09-11T03:41:46.884 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 5 to 95 2024-09-11T03:41:46.884 INFO:tasks.rados.rados.2.smithi175.stdout:920: finishing write tid 1 to smithi17557870-10 2024-09-11T03:41:46.884 INFO:tasks.rados.rados.2.smithi175.stdout:920: finishing write tid 2 to smithi17557870-10 2024-09-11T03:41:46.884 INFO:tasks.rados.rados.2.smithi175.stdout:922: rollback oid 8 current snap is 98 2024-09-11T03:41:46.884 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 8 to 97 2024-09-11T03:41:46.884 INFO:tasks.rados.rados.2.smithi175.stdout:923: write oid 49 current snap is 98 2024-09-11T03:41:46.884 INFO:tasks.rados.rados.2.smithi175.stdout:923: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2024-09-11T03:41:46.889 INFO:tasks.rados.rados.2.smithi175.stdout:923: writing smithi17557870-49 from 480766 to 1034460 tid 1 2024-09-11T03:41:46.895 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:46.896 INFO:tasks.rados.rados.2.smithi175.stdout:923: writing smithi17557870-49 from 1629149 to 2220268 tid 2 2024-09-11T03:41:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:46.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:46.902 INFO:tasks.rados.rados.2.smithi175.stdout:923: writing smithi17557870-49 from 2782148 to 3202206 tid 3 2024-09-11T03:41:46.902 INFO:tasks.rados.rados.2.smithi175.stdout:924: read oid 33 snap -1 2024-09-11T03:41:46.902 INFO:tasks.rados.rados.2.smithi175.stdout:924: expect (ObjNum 171 snap 39 seq_num 171) 2024-09-11T03:41:46.903 INFO:tasks.rados.rados.2.smithi175.stdout:925: read oid 11 snap -1 2024-09-11T03:41:46.903 INFO:tasks.rados.rados.2.smithi175.stdout:925: expect (ObjNum 270 snap 69 seq_num 270) 2024-09-11T03:41:46.903 INFO:tasks.rados.rados.2.smithi175.stdout:926: copy_from oid 39 from oid 7 current snap is 98 2024-09-11T03:41:46.903 INFO:tasks.rados.rados.2.smithi175.stdout:920: finishing write tid 3 to smithi17557870-10 2024-09-11T03:41:46.903 INFO:tasks.rados.rados.2.smithi175.stdout:920: finishing write tid 4 to smithi17557870-10 2024-09-11T03:41:46.903 INFO:tasks.rados.rados.2.smithi175.stdout:920: finishing write tid 5 to smithi17557870-10 2024-09-11T03:41:46.903 INFO:tasks.rados.rados.2.smithi175.stdout:920: oid 10 updating version 0 to 336 2024-09-11T03:41:46.903 INFO:tasks.rados.rados.2.smithi175.stdout:920: oid 10 version 336 is already newer than 335 2024-09-11T03:41:46.903 INFO:tasks.rados.rados.2.smithi175.stdout:920: oid 10 updating version 336 to 337 2024-09-11T03:41:46.903 INFO:tasks.rados.rados.2.smithi175.stdout:920: oid 10 version 337 is already newer than 334 2024-09-11T03:41:46.903 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 337 (ObjNum 337 snap 98 seq_num 337) dirty exists 2024-09-11T03:41:46.904 INFO:tasks.rados.rados.2.smithi175.stdout:920: left oid 10 (ObjNum 337 snap 98 seq_num 337) 2024-09-11T03:41:46.904 INFO:tasks.rados.rados.2.smithi175.stdout:921: finishing rollback tid 0 to smithi17557870-5 2024-09-11T03:41:46.904 INFO:tasks.rados.rados.2.smithi175.stdout:920: done (6 left) 2024-09-11T03:41:46.904 INFO:tasks.rados.rados.2.smithi175.stdout:927: copy_from oid 29 from oid 27 current snap is 98 2024-09-11T03:41:46.904 INFO:tasks.rados.rados.2.smithi175.stdout:921: finishing rollback tid 1 to smithi17557870-5 2024-09-11T03:41:46.904 INFO:tasks.rados.rados.2.smithi175.stdout:921: finishing rollback tid 2 to smithi17557870-5 2024-09-11T03:41:46.904 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 266 (ObjNum 302 snap 82 seq_num 302) dirty exists 2024-09-11T03:41:46.904 INFO:tasks.rados.rados.2.smithi175.stdout:922: finishing rollback tid 0 to smithi17557870-8 2024-09-11T03:41:46.904 INFO:tasks.rados.rados.2.smithi175.stdout:922: finishing rollback tid 1 to smithi17557870-8 2024-09-11T03:41:46.904 INFO:tasks.rados.rados.2.smithi175.stdout:922: finishing rollback tid 2 to smithi17557870-8 2024-09-11T03:41:46.904 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 329 (ObjNum 319 snap 86 seq_num 319) dirty exists 2024-09-11T03:41:46.904 INFO:tasks.rados.rados.2.smithi175.stdout:923: finishing write tid 1 to smithi17557870-49 2024-09-11T03:41:46.904 INFO:tasks.rados.rados.2.smithi175.stdout:923: finishing write tid 2 to smithi17557870-49 2024-09-11T03:41:46.905 INFO:tasks.rados.rados.2.smithi175.stdout:921: done (6 left) 2024-09-11T03:41:46.905 INFO:tasks.rados.rados.2.smithi175.stdout:922: done (5 left) 2024-09-11T03:41:46.905 INFO:tasks.rados.rados.2.smithi175.stdout:928: read oid 47 snap -1 2024-09-11T03:41:46.905 INFO:tasks.rados.rados.2.smithi175.stdout:928: expect deleted 2024-09-11T03:41:46.905 INFO:tasks.rados.rados.2.smithi175.stdout:929: rollback oid 48 current snap is 98 2024-09-11T03:41:46.905 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 48 to 95 2024-09-11T03:41:46.905 INFO:tasks.rados.rados.2.smithi175.stdout:930: rollback oid 4 current snap is 98 2024-09-11T03:41:46.905 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 4 to 93 2024-09-11T03:41:46.905 INFO:tasks.rados.rados.2.smithi175.stdout:931: rollback oid 24 current snap is 98 2024-09-11T03:41:46.905 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 96 2024-09-11T03:41:46.905 INFO:tasks.rados.rados.2.smithi175.stdout:932: write (excl) oid 16 current snap is 98 2024-09-11T03:41:46.905 INFO:tasks.rados.rados.2.smithi175.stdout:932: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2024-09-11T03:41:46.907 INFO:tasks.rados.rados.2.smithi175.stdout:932: writing smithi17557870-16 from 497573 to 1126516 tid 1 2024-09-11T03:41:46.915 INFO:tasks.rados.rados.2.smithi175.stdout:932: writing smithi17557870-16 from 1687631 to 2422408 tid 2 2024-09-11T03:41:46.924 INFO:tasks.rados.rados.2.smithi175.stdout:932: writing smithi17557870-16 from 3093218 to 3697573 tid 3 2024-09-11T03:41:46.924 INFO:tasks.rados.rados.2.smithi175.stdout:933: read oid 22 snap -1 2024-09-11T03:41:46.924 INFO:tasks.rados.rados.2.smithi175.stdout:933: expect (ObjNum 323 snap 89 seq_num 323) 2024-09-11T03:41:46.924 INFO:tasks.rados.rados.2.smithi175.stdout:934: snap_create 2024-09-11T03:41:46.924 INFO:tasks.rados.rados.2.smithi175.stdout:929: finishing rollback tid 1 to smithi17557870-48 2024-09-11T03:41:46.924 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T03:41:46.924 INFO:tasks.rados.rados.2.smithi175.stdout:931: finishing rollback tid 0 to smithi17557870-24 2024-09-11T03:41:46.924 INFO:tasks.rados.rados.2.smithi175.stdout:930: finishing rollback tid 1 to smithi17557870-4 2024-09-11T03:41:46.924 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 0 (ObjNum 33 snap 0 seq_num 1713067256) dirty dne 2024-09-11T03:41:46.925 INFO:tasks.rados.rados.2.smithi175.stdout:931: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:41:46.925 INFO:tasks.rados.rados.2.smithi175.stdout:923: finishing write tid 3 to smithi17557870-49 2024-09-11T03:41:46.925 INFO:tasks.rados.rados.2.smithi175.stdout:931: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:41:46.925 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 335 (ObjNum 267 snap 68 seq_num 267) dirty exists 2024-09-11T03:41:46.925 INFO:tasks.rados.rados.2.smithi175.stdout:923: finishing write tid 4 to smithi17557870-49 2024-09-11T03:41:46.925 INFO:tasks.rados.rados.2.smithi175.stdout:923: finishing write tid 5 to smithi17557870-49 2024-09-11T03:41:46.925 INFO:tasks.rados.rados.2.smithi175.stdout:923: oid 49 updating version 0 to 238 2024-09-11T03:41:46.925 INFO:tasks.rados.rados.2.smithi175.stdout:923: oid 49 updating version 238 to 240 2024-09-11T03:41:46.925 INFO:tasks.rados.rados.2.smithi175.stdout:923: oid 49 version 240 is already newer than 237 2024-09-11T03:41:46.925 INFO:tasks.rados.rados.2.smithi175.stdout:923: oid 49 version 240 is already newer than 239 2024-09-11T03:41:46.925 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 240 (ObjNum 338 snap 98 seq_num 338) dirty exists 2024-09-11T03:41:46.925 INFO:tasks.rados.rados.2.smithi175.stdout:923: left oid 49 (ObjNum 338 snap 98 seq_num 338) 2024-09-11T03:41:46.925 INFO:tasks.rados.rados.2.smithi175.stdout:932: finishing write tid 1 to smithi17557870-16 2024-09-11T03:41:46.926 INFO:tasks.rados.rados.2.smithi175.stdout:924: expect (ObjNum 171 snap 39 seq_num 171) 2024-09-11T03:41:46.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:46 smithi175 ceph-mon[30894]: osdmap e289: 12 total, 12 up, 10 in 2024-09-11T03:41:46.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:46 smithi175 ceph-mon[30894]: 1.0 scrub starts 2024-09-11T03:41:46.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:46 smithi175 ceph-mon[30894]: 1.0 scrub ok 2024-09-11T03:41:46.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:46 smithi175 ceph-mon[30894]: osdmap e290: 12 total, 12 up, 10 in 2024-09-11T03:41:46.998 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:47.019 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:47.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:47.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:47.077 INFO:tasks.rados.rados.2.smithi175.stdout:927: finishing copy_from racing read to smithi17557870-29 2024-09-11T03:41:47.077 INFO:tasks.rados.rados.2.smithi175.stdout:925: expect (ObjNum 270 snap 69 seq_num 270) 2024-09-11T03:41:47.121 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:47.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:47.144 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:47.144 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:47.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:46 smithi084 ceph-mon[27254]: osdmap e289: 12 total, 12 up, 10 in 2024-09-11T03:41:47.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:46 smithi084 ceph-mon[27254]: 1.0 scrub starts 2024-09-11T03:41:47.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:46 smithi084 ceph-mon[27254]: 1.0 scrub ok 2024-09-11T03:41:47.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:46 smithi084 ceph-mon[27254]: osdmap e290: 12 total, 12 up, 10 in 2024-09-11T03:41:47.157 INFO:tasks.rados.rados.2.smithi175.stdout:926: finishing copy_from racing read to smithi17557870-39 2024-09-11T03:41:47.157 INFO:tasks.rados.rados.2.smithi175.stdout:932: finishing write tid 2 to smithi17557870-16 2024-09-11T03:41:47.157 INFO:tasks.rados.rados.2.smithi175.stdout:927: finishing copy_from to smithi17557870-29 2024-09-11T03:41:47.157 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 300 (ObjNum 302 snap 82 seq_num 302) dirty exists 2024-09-11T03:41:47.158 INFO:tasks.rados.rados.2.smithi175.stdout:932: finishing write tid 3 to smithi17557870-16 2024-09-11T03:41:47.158 INFO:tasks.rados.rados.2.smithi175.stdout:932: finishing write tid 4 to smithi17557870-16 2024-09-11T03:41:47.158 INFO:tasks.rados.rados.2.smithi175.stdout:926: finishing copy_from to smithi17557870-39 2024-09-11T03:41:47.158 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 163 (ObjNum 295 snap 79 seq_num 295) dirty exists 2024-09-11T03:41:47.158 INFO:tasks.rados.rados.2.smithi175.stdout:932: finishing write tid 5 to smithi17557870-16 2024-09-11T03:41:47.158 INFO:tasks.rados.rados.2.smithi175.stdout:932: oid 16 updating version 0 to 151 2024-09-11T03:41:47.158 INFO:tasks.rados.rados.2.smithi175.stdout:932: oid 16 version 151 is already newer than 150 2024-09-11T03:41:47.158 INFO:tasks.rados.rados.2.smithi175.stdout:932: oid 16 updating version 151 to 153 2024-09-11T03:41:47.158 INFO:tasks.rados.rados.2.smithi175.stdout:932: oid 16 version 153 is already newer than 152 2024-09-11T03:41:47.158 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 153 (ObjNum 339 snap 98 seq_num 339) dirty exists 2024-09-11T03:41:47.158 INFO:tasks.rados.rados.2.smithi175.stdout:932: left oid 16 (ObjNum 339 snap 98 seq_num 339) 2024-09-11T03:41:47.158 INFO:tasks.rados.rados.2.smithi175.stdout:933: expect (ObjNum 323 snap 89 seq_num 323) 2024-09-11T03:41:47.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:46 smithi121 ceph-mon[30729]: osdmap e289: 12 total, 12 up, 10 in 2024-09-11T03:41:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:46 smithi121 ceph-mon[30729]: 1.0 scrub starts 2024-09-11T03:41:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:46 smithi121 ceph-mon[30729]: 1.0 scrub ok 2024-09-11T03:41:47.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:46 smithi121 ceph-mon[30729]: osdmap e290: 12 total, 12 up, 10 in 2024-09-11T03:41:47.245 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:47.271 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:47.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:47.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:47.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:47.398 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:47.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:47.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:47.439 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-11T03:41:47.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:47.521 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:47.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:47.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:47.624 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:47.645 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:47.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:47.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:47.722 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:41:47.750 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:47.776 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:47.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:47.872 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:41:47.876 INFO:tasks.rados.rados.2.smithi175.stdout:923: done (11 left) 2024-09-11T03:41:47.876 INFO:tasks.rados.rados.2.smithi175.stdout:924: done (10 left) 2024-09-11T03:41:47.876 INFO:tasks.rados.rados.2.smithi175.stdout:925: done (9 left) 2024-09-11T03:41:47.876 INFO:tasks.rados.rados.2.smithi175.stdout:926: done (8 left) 2024-09-11T03:41:47.876 INFO:tasks.rados.rados.2.smithi175.stdout:927: done (7 left) 2024-09-11T03:41:47.877 INFO:tasks.rados.rados.2.smithi175.stdout:928: done (6 left) 2024-09-11T03:41:47.877 INFO:tasks.rados.rados.2.smithi175.stdout:929: done (5 left) 2024-09-11T03:41:47.877 INFO:tasks.rados.rados.2.smithi175.stdout:930: done (4 left) 2024-09-11T03:41:47.877 INFO:tasks.rados.rados.2.smithi175.stdout:931: done (3 left) 2024-09-11T03:41:47.877 INFO:tasks.rados.rados.2.smithi175.stdout:932: done (2 left) 2024-09-11T03:41:47.877 INFO:tasks.rados.rados.2.smithi175.stdout:933: done (1 left) 2024-09-11T03:41:47.877 INFO:tasks.rados.rados.2.smithi175.stdout:934: done (0 left) 2024-09-11T03:41:47.877 INFO:tasks.rados.rados.2.smithi175.stdout:935: read oid 24 snap -1 2024-09-11T03:41:47.877 INFO:tasks.rados.rados.2.smithi175.stdout:935: expect (ObjNum 267 snap 68 seq_num 267) 2024-09-11T03:41:47.877 INFO:tasks.rados.rados.2.smithi175.stdout:936: write oid 43 current snap is 99 2024-09-11T03:41:47.877 INFO:tasks.rados.rados.2.smithi175.stdout:936: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2024-09-11T03:41:47.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:47.882 INFO:tasks.rados.rados.2.smithi175.stdout:936: writing smithi17557870-43 from 514380 to 1218572 tid 1 2024-09-11T03:41:47.888 INFO:tasks.rados.rados.2.smithi175.stdout:936: writing smithi17557870-43 from 1629760 to 2108195 tid 2 2024-09-11T03:41:47.893 INFO:tasks.rados.rados.2.smithi175.stdout:936: writing smithi17557870-43 from 2604288 to 3046890 tid 3 2024-09-11T03:41:47.893 INFO:tasks.rados.rados.2.smithi175.stdout:937: rollback oid 39 current snap is 99 2024-09-11T03:41:47.893 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 39 to 86 2024-09-11T03:41:47.893 INFO:tasks.rados.rados.2.smithi175.stdout:935: expect (ObjNum 267 snap 68 seq_num 267) 2024-09-11T03:41:47.903 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:47.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:48.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:48.036 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:48.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:48.054 INFO:tasks.rados.rados.2.smithi175.stdout:936: finishing write tid 1 to smithi17557870-43 2024-09-11T03:41:48.054 INFO:tasks.rados.rados.2.smithi175.stdout:936: finishing write tid 2 to smithi17557870-43 2024-09-11T03:41:48.054 INFO:tasks.rados.rados.2.smithi175.stdout:935: done (2 left) 2024-09-11T03:41:48.054 INFO:tasks.rados.rados.2.smithi175.stdout:938: write (excl) oid 5 current snap is 99 2024-09-11T03:41:48.054 INFO:tasks.rados.rados.2.smithi175.stdout:937: finishing rollback tid 0 to smithi17557870-39 2024-09-11T03:41:48.054 INFO:tasks.rados.rados.2.smithi175.stdout:937: finishing rollback tid 1 to smithi17557870-39 2024-09-11T03:41:48.054 INFO:tasks.rados.rados.2.smithi175.stdout:937: finishing rollback tid 2 to smithi17557870-39 2024-09-11T03:41:48.055 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 279 (ObjNum 262 snap 66 seq_num 262) dirty exists 2024-09-11T03:41:48.055 INFO:tasks.rados.rados.2.smithi175.stdout:936: finishing write tid 3 to smithi17557870-43 2024-09-11T03:41:48.055 INFO:tasks.rados.rados.2.smithi175.stdout:936: finishing write tid 4 to smithi17557870-43 2024-09-11T03:41:48.055 INFO:tasks.rados.rados.2.smithi175.stdout:936: finishing write tid 5 to smithi17557870-43 2024-09-11T03:41:48.055 INFO:tasks.rados.rados.2.smithi175.stdout:936: oid 43 updating version 0 to 126 2024-09-11T03:41:48.055 INFO:tasks.rados.rados.2.smithi175.stdout:936: oid 43 updating version 126 to 128 2024-09-11T03:41:48.055 INFO:tasks.rados.rados.2.smithi175.stdout:936: oid 43 version 128 is already newer than 125 2024-09-11T03:41:48.055 INFO:tasks.rados.rados.2.smithi175.stdout:936: oid 43 version 128 is already newer than 127 2024-09-11T03:41:48.055 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 128 (ObjNum 340 snap 99 seq_num 340) dirty exists 2024-09-11T03:41:48.055 INFO:tasks.rados.rados.2.smithi175.stdout:936: left oid 43 (ObjNum 340 snap 99 seq_num 340) 2024-09-11T03:41:48.055 INFO:tasks.rados.rados.2.smithi175.stdout:938: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-09-11T03:41:48.061 INFO:tasks.rados.rados.2.smithi175.stdout:938: writing smithi17557870-5 from 531187 to 1310628 tid 1 2024-09-11T03:41:48.069 INFO:tasks.rados.rados.2.smithi175.stdout:938: writing smithi17557870-5 from 2088242 to 2826688 tid 2 2024-09-11T03:41:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:938: writing smithi17557870-5 from 3431711 to 3731187 tid 3 2024-09-11T03:41:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:936: done (2 left) 2024-09-11T03:41:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:937: done (1 left) 2024-09-11T03:41:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:939: read oid 40 snap -1 2024-09-11T03:41:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:939: expect deleted 2024-09-11T03:41:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:940: rollback oid 47 current snap is 99 2024-09-11T03:41:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 47 to 93 2024-09-11T03:41:48.076 INFO:tasks.rados.rados.2.smithi175.stdout:938: finishing write tid 1 to smithi17557870-5 2024-09-11T03:41:48.076 INFO:tasks.rados.rados.2.smithi175.stdout:938: finishing write tid 2 to smithi17557870-5 2024-09-11T03:41:48.076 INFO:tasks.rados.rados.2.smithi175.stdout:941: write oid 22 current snap is 99 2024-09-11T03:41:48.076 INFO:tasks.rados.rados.2.smithi175.stdout:941: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2024-09-11T03:41:48.079 INFO:tasks.rados.rados.2.smithi175.stdout:941: writing smithi17557870-22 from 547994 to 1002684 tid 1 2024-09-11T03:41:48.085 INFO:tasks.rados.rados.2.smithi175.stdout:941: writing smithi17557870-22 from 1746724 to 2228828 tid 2 2024-09-11T03:41:48.090 INFO:tasks.rados.rados.2.smithi175.stdout:941: writing smithi17557870-22 from 2659134 to 3124280 tid 3 2024-09-11T03:41:48.094 INFO:tasks.rados.rados.2.smithi175.stdout:941: writing smithi17557870-22 from 3605976 to 3747994 tid 4 2024-09-11T03:41:48.094 INFO:tasks.rados.rados.2.smithi175.stdout:942: read oid 45 snap -1 2024-09-11T03:41:48.094 INFO:tasks.rados.rados.2.smithi175.stdout:942: expect deleted 2024-09-11T03:41:48.095 INFO:tasks.rados.rados.2.smithi175.stdout:943: rollback oid 10 current snap is 99 2024-09-11T03:41:48.095 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 10 to 91 2024-09-11T03:41:48.095 INFO:tasks.rados.rados.2.smithi175.stdout:940: finishing rollback tid 1 to smithi17557870-47 2024-09-11T03:41:48.095 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 0 (ObjNum 33 snap 0 seq_num 1713067256) dirty dne 2024-09-11T03:41:48.095 INFO:tasks.rados.rados.2.smithi175.stdout:938: finishing write tid 3 to smithi17557870-5 2024-09-11T03:41:48.095 INFO:tasks.rados.rados.2.smithi175.stdout:938: finishing write tid 4 to smithi17557870-5 2024-09-11T03:41:48.095 INFO:tasks.rados.rados.2.smithi175.stdout:938: finishing write tid 5 to smithi17557870-5 2024-09-11T03:41:48.095 INFO:tasks.rados.rados.2.smithi175.stdout:938: oid 5 updating version 0 to 270 2024-09-11T03:41:48.095 INFO:tasks.rados.rados.2.smithi175.stdout:938: oid 5 version 270 is already newer than 268 2024-09-11T03:41:48.095 INFO:tasks.rados.rados.2.smithi175.stdout:938: oid 5 version 270 is already newer than 267 2024-09-11T03:41:48.095 INFO:tasks.rados.rados.2.smithi175.stdout:938: oid 5 version 270 is already newer than 269 2024-09-11T03:41:48.095 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 270 (ObjNum 341 snap 99 seq_num 341) dirty exists 2024-09-11T03:41:48.096 INFO:tasks.rados.rados.2.smithi175.stdout:938: left oid 5 (ObjNum 341 snap 99 seq_num 341) 2024-09-11T03:41:48.096 INFO:tasks.rados.rados.2.smithi175.stdout:941: finishing write tid 1 to smithi17557870-22 2024-09-11T03:41:48.096 INFO:tasks.rados.rados.2.smithi175.stdout:941: finishing write tid 2 to smithi17557870-22 2024-09-11T03:41:48.096 INFO:tasks.rados.rados.2.smithi175.stdout:941: finishing write tid 3 to smithi17557870-22 2024-09-11T03:41:48.096 INFO:tasks.rados.rados.2.smithi175.stdout:938: done (5 left) 2024-09-11T03:41:48.096 INFO:tasks.rados.rados.2.smithi175.stdout:939: done (4 left) 2024-09-11T03:41:48.096 INFO:tasks.rados.rados.2.smithi175.stdout:940: done (3 left) 2024-09-11T03:41:48.096 INFO:tasks.rados.rados.2.smithi175.stdout:944: delete oid 30 current snap is 99 2024-09-11T03:41:48.097 INFO:tasks.rados.rados.2.smithi175.stdout:942: done (3 left) 2024-09-11T03:41:48.097 INFO:tasks.rados.rados.2.smithi175.stdout:944: done (2 left) 2024-09-11T03:41:48.097 INFO:tasks.rados.rados.2.smithi175.stdout:945: write oid 25 current snap is 99 2024-09-11T03:41:48.097 INFO:tasks.rados.rados.2.smithi175.stdout:945: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2024-09-11T03:41:48.103 INFO:tasks.rados.rados.2.smithi175.stdout:945: writing smithi17557870-25 from 581608 to 1303149 tid 1 2024-09-11T03:41:48.106 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:47 smithi175 ceph-mon[30894]: pgmap v589: 57 pgs: 57 active+clean; 161 MiB data, 650 MiB used, 893 GiB / 894 GiB avail 2024-09-11T03:41:48.106 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:47 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2819366735' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:41:48.109 INFO:tasks.rados.rados.2.smithi175.stdout:945: writing smithi17557870-25 from 1863688 to 2349461 tid 2 2024-09-11T03:41:48.119 INFO:tasks.rados.rados.2.smithi175.stdout:945: writing smithi17557870-25 from 3113980 to 3718023 tid 3 2024-09-11T03:41:48.119 INFO:tasks.rados.rados.2.smithi175.stdout:946: read oid 23 snap -1 2024-09-11T03:41:48.119 INFO:tasks.rados.rados.2.smithi175.stdout:946: expect (ObjNum 330 snap 92 seq_num 330) 2024-09-11T03:41:48.120 INFO:tasks.rados.rados.2.smithi175.stdout:947: delete oid 29 current snap is 99 2024-09-11T03:41:48.120 INFO:tasks.rados.rados.2.smithi175.stdout:943: finishing rollback tid 0 to smithi17557870-10 2024-09-11T03:41:48.120 INFO:tasks.rados.rados.2.smithi175.stdout:941: finishing write tid 4 to smithi17557870-22 2024-09-11T03:41:48.120 INFO:tasks.rados.rados.2.smithi175.stdout:943: finishing rollback tid 1 to smithi17557870-10 2024-09-11T03:41:48.120 INFO:tasks.rados.rados.2.smithi175.stdout:941: finishing write tid 5 to smithi17557870-22 2024-09-11T03:41:48.120 INFO:tasks.rados.rados.2.smithi175.stdout:943: finishing rollback tid 2 to smithi17557870-10 2024-09-11T03:41:48.121 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 342 (ObjNum 310 snap 85 seq_num 310) dirty exists 2024-09-11T03:41:48.121 INFO:tasks.rados.rados.2.smithi175.stdout:941: finishing write tid 6 to smithi17557870-22 2024-09-11T03:41:48.121 INFO:tasks.rados.rados.2.smithi175.stdout:941: oid 22 updating version 0 to 342 2024-09-11T03:41:48.121 INFO:tasks.rados.rados.2.smithi175.stdout:941: oid 22 version 342 is already newer than 340 2024-09-11T03:41:48.121 INFO:tasks.rados.rados.2.smithi175.stdout:941: oid 22 version 342 is already newer than 341 2024-09-11T03:41:48.122 INFO:tasks.rados.rados.2.smithi175.stdout:941: oid 22 version 342 is already newer than 339 2024-09-11T03:41:48.122 INFO:tasks.rados.rados.2.smithi175.stdout:941: oid 22 updating version 342 to 343 2024-09-11T03:41:48.122 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 343 (ObjNum 342 snap 99 seq_num 342) dirty exists 2024-09-11T03:41:48.122 INFO:tasks.rados.rados.2.smithi175.stdout:941: left oid 22 (ObjNum 342 snap 99 seq_num 342) 2024-09-11T03:41:48.122 INFO:tasks.rados.rados.2.smithi175.stdout:945: finishing write tid 1 to smithi17557870-25 2024-09-11T03:41:48.122 INFO:tasks.rados.rados.2.smithi175.stdout:945: finishing write tid 2 to smithi17557870-25 2024-09-11T03:41:48.123 INFO:tasks.rados.rados.2.smithi175.stdout:941: done (4 left) 2024-09-11T03:41:48.123 INFO:tasks.rados.rados.2.smithi175.stdout:943: done (3 left) 2024-09-11T03:41:48.123 INFO:tasks.rados.rados.2.smithi175.stdout:947: done (2 left) 2024-09-11T03:41:48.123 INFO:tasks.rados.rados.2.smithi175.stdout:948: snap_remove snap 93 2024-09-11T03:41:48.138 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:48.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:47 smithi084 ceph-mon[27254]: pgmap v589: 57 pgs: 57 active+clean; 161 MiB data, 650 MiB used, 893 GiB / 894 GiB avail 2024-09-11T03:41:48.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:47 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2819366735' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:41:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:48.160 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:48.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:48.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:47 smithi121 ceph-mon[30729]: pgmap v589: 57 pgs: 57 active+clean; 161 MiB data, 650 MiB used, 893 GiB / 894 GiB avail 2024-09-11T03:41:48.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:47 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2819366735' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:41:48.223 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:41:48.223 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:41:48.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:48.288 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:48.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:48.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:48.390 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:48.412 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:48.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:48.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:48.514 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:48.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:48.536 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:48.536 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:48.587 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:41:48.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:48.662 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:48.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:48.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:48.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:48.791 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:48.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:48.829 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:48.879 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T03:41:48.884 INFO:tasks.rados.rados.2.smithi175.stdout:948: done (2 left) 2024-09-11T03:41:48.884 INFO:tasks.rados.rados.2.smithi175.stdout:949: rollback oid 48 current snap is 99 2024-09-11T03:41:48.884 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 48 to 98 2024-09-11T03:41:48.884 INFO:tasks.rados.rados.2.smithi175.stdout:945: finishing write tid 3 to smithi17557870-25 2024-09-11T03:41:48.885 INFO:tasks.rados.rados.2.smithi175.stdout:945: finishing write tid 4 to smithi17557870-25 2024-09-11T03:41:48.885 INFO:tasks.rados.rados.2.smithi175.stdout:945: finishing write tid 5 to smithi17557870-25 2024-09-11T03:41:48.885 INFO:tasks.rados.rados.2.smithi175.stdout:945: oid 25 updating version 0 to 223 2024-09-11T03:41:48.885 INFO:tasks.rados.rados.2.smithi175.stdout:945: oid 25 updating version 223 to 225 2024-09-11T03:41:48.885 INFO:tasks.rados.rados.2.smithi175.stdout:945: oid 25 version 225 is already newer than 224 2024-09-11T03:41:48.885 INFO:tasks.rados.rados.2.smithi175.stdout:945: oid 25 version 225 is already newer than 222 2024-09-11T03:41:48.885 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 225 (ObjNum 344 snap 99 seq_num 344) dirty exists 2024-09-11T03:41:48.885 INFO:tasks.rados.rados.2.smithi175.stdout:945: left oid 25 (ObjNum 344 snap 99 seq_num 344) 2024-09-11T03:41:48.885 INFO:tasks.rados.rados.2.smithi175.stdout:946: expect (ObjNum 330 snap 92 seq_num 330) 2024-09-11T03:41:48.891 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:48.914 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:48.915 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:48.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:48 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:41:48.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:48 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:41:48.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:48 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:48.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:48 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2819366735' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:41:48.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:48 smithi175 ceph-mon[30894]: osdmap e291: 12 total, 12 up, 10 in 2024-09-11T03:41:48.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:48 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2244620924' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:41:49.007 INFO:tasks.rados.rados.2.smithi175.stdout:949: finishing rollback tid 1 to smithi17557870-48 2024-09-11T03:41:49.007 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 138 (ObjNum 65 snap 0 seq_num 197143679) dirty dne 2024-09-11T03:41:49.007 INFO:tasks.rados.rados.2.smithi175.stdout:945: done (2 left) 2024-09-11T03:41:49.007 INFO:tasks.rados.rados.2.smithi175.stdout:946: done (1 left) 2024-09-11T03:41:49.007 INFO:tasks.rados.rados.2.smithi175.stdout:949: done (0 left) 2024-09-11T03:41:49.007 INFO:tasks.rados.rados.2.smithi175.stdout:950: read oid 34 snap -1 2024-09-11T03:41:49.007 INFO:tasks.rados.rados.2.smithi175.stdout:950: expect (ObjNum 293 snap 79 seq_num 293) 2024-09-11T03:41:49.007 INFO:tasks.rados.rados.2.smithi175.stdout:951: snap_remove snap 96 2024-09-11T03:41:49.016 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:49.037 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:49.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:49.052 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-11T03:41:49.138 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:49.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:48 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:41:49.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:48 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:41:49.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:48 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:49.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:48 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2819366735' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:41:49.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:48 smithi084 ceph-mon[27254]: osdmap e291: 12 total, 12 up, 10 in 2024-09-11T03:41:49.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:48 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2244620924' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:41:49.164 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:49.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:49.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:49.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:48 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:41:49.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:48 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:41:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:48 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:48 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2819366735' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:41:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:48 smithi121 ceph-mon[30729]: osdmap e291: 12 total, 12 up, 10 in 2024-09-11T03:41:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:48 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2244620924' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:41:49.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:49.292 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:41:49.293 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:49.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:49.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:49.395 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:49.421 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:49.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:49.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:49.503 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:41:49.503 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:41:49.517 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:41:49.517 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:41:49.523 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:49.533 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:41:49.534 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:41:49.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:49.544 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:49.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:49.550 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:41:49.550 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:41:49.568 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:41:49.568 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:41:49.584 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:41:49.585 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:41:49.601 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:41:49.602 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:41:49.619 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:41:49.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:41:49.636 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:41:49.636 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:41:49.645 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:49.653 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:41:49.653 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:41:49.667 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:49.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:49.668 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:49.669 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:41:49.669 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:41:49.686 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:41:49.686 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:41:49.764 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:41:49.765 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:41:49.765 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:41:49.765 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:41:49.769 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:49.791 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:49.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:49.792 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:49.893 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:49.897 INFO:tasks.rados.rados.2.smithi175.stdout:951: done (1 left) 2024-09-11T03:41:49.897 INFO:tasks.rados.rados.2.smithi175.stdout:952: snap_create 2024-09-11T03:41:49.897 INFO:tasks.rados.rados.2.smithi175.stdout:950: expect (ObjNum 293 snap 79 seq_num 293) 2024-09-11T03:41:49.916 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:49.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:49.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:49.939 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:49 smithi084 ceph-mon[27254]: pgmap v592: 57 pgs: 57 active+clean; 156 MiB data, 650 MiB used, 893 GiB / 894 GiB avail 2024-09-11T03:41:49.939 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2244620924' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:41:49.939 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:49 smithi084 ceph-mon[27254]: osdmap e292: 12 total, 12 up, 10 in 2024-09-11T03:41:49.939 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1474915473' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:49.967 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-11T03:41:49.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:49 smithi175 ceph-mon[30894]: pgmap v592: 57 pgs: 57 active+clean; 156 MiB data, 650 MiB used, 893 GiB / 894 GiB avail 2024-09-11T03:41:49.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2244620924' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:41:49.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:49 smithi175 ceph-mon[30894]: osdmap e292: 12 total, 12 up, 10 in 2024-09-11T03:41:49.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1474915473' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:50.019 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:50.071 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:50.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:49 smithi121 ceph-mon[30729]: pgmap v592: 57 pgs: 57 active+clean; 156 MiB data, 650 MiB used, 893 GiB / 894 GiB avail 2024-09-11T03:41:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2244620924' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:41:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:49 smithi121 ceph-mon[30729]: osdmap e292: 12 total, 12 up, 10 in 2024-09-11T03:41:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1474915473' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:50.173 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:50.195 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:50.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:50.195 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:50.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:50.323 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:50.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:50.425 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:50.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:50.448 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:50.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:50.531 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-11T03:41:50.549 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:41:50.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:50.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:50.573 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:50.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:50.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:50.683 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:50.695 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:50.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:50.774 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:41:50.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:50.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:50.821 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:50.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:50.910 INFO:tasks.rados.rados.2.smithi175.stdout:950: done (1 left) 2024-09-11T03:41:50.910 INFO:tasks.rados.rados.2.smithi175.stdout:952: done (0 left) 2024-09-11T03:41:50.911 INFO:tasks.rados.rados.2.smithi175.stdout:953: write (excl) oid 46 current snap is 100 2024-09-11T03:41:50.911 INFO:tasks.rados.rados.2.smithi175.stdout:953: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2024-09-11T03:41:50.916 INFO:tasks.rados.rados.2.smithi175.stdout:953: writing smithi17557870-46 from 615222 to 1087261 tid 1 2024-09-11T03:41:50.921 INFO:tasks.rados.rados.2.smithi175.stdout:953: writing smithi17557870-46 from 1580652 to 2070094 tid 2 2024-09-11T03:41:50.922 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:50.929 INFO:tasks.rados.rados.2.smithi175.stdout:953: writing smithi17557870-46 from 2768826 to 3395413 tid 3 2024-09-11T03:41:50.929 INFO:tasks.rados.rados.2.smithi175.stdout:954: write (excl) oid 7 current snap is 100 2024-09-11T03:41:50.929 INFO:tasks.rados.rados.2.smithi175.stdout:953: finishing write tid 1 to smithi17557870-46 2024-09-11T03:41:50.930 INFO:tasks.rados.rados.2.smithi175.stdout:953: finishing write tid 2 to smithi17557870-46 2024-09-11T03:41:50.930 INFO:tasks.rados.rados.2.smithi175.stdout:954: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2024-09-11T03:41:50.935 INFO:tasks.rados.rados.2.smithi175.stdout:954: writing smithi17557870-7 from 632029 to 1179317 tid 1 2024-09-11T03:41:50.943 INFO:tasks.rados.rados.2.smithi175.stdout:954: writing smithi17557870-7 from 1639134 to 2388587 tid 2 2024-09-11T03:41:50.944 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:50.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:50.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:50.948 INFO:tasks.rados.rados.2.smithi175.stdout:954: writing smithi17557870-7 from 2796249 to 3234108 tid 3 2024-09-11T03:41:50.952 INFO:tasks.rados.rados.2.smithi175.stdout:954: writing smithi17557870-7 from 3653524 to 3832029 tid 4 2024-09-11T03:41:50.952 INFO:tasks.rados.rados.2.smithi175.stdout:955: delete oid 38 current snap is 100 2024-09-11T03:41:50.953 INFO:tasks.rados.rados.2.smithi175.stdout:953: finishing write tid 3 to smithi17557870-46 2024-09-11T03:41:50.953 INFO:tasks.rados.rados.2.smithi175.stdout:953: finishing write tid 4 to smithi17557870-46 2024-09-11T03:41:50.953 INFO:tasks.rados.rados.2.smithi175.stdout:953: finishing write tid 5 to smithi17557870-46 2024-09-11T03:41:50.953 INFO:tasks.rados.rados.2.smithi175.stdout:953: oid 46 updating version 0 to 164 2024-09-11T03:41:50.953 INFO:tasks.rados.rados.2.smithi175.stdout:953: oid 46 version 164 is already newer than 163 2024-09-11T03:41:50.954 INFO:tasks.rados.rados.2.smithi175.stdout:953: oid 46 version 164 is already newer than 161 2024-09-11T03:41:50.954 INFO:tasks.rados.rados.2.smithi175.stdout:953: oid 46 version 164 is already newer than 162 2024-09-11T03:41:50.954 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 164 (ObjNum 346 snap 100 seq_num 346) dirty exists 2024-09-11T03:41:50.954 INFO:tasks.rados.rados.2.smithi175.stdout:953: left oid 46 (ObjNum 346 snap 100 seq_num 346) 2024-09-11T03:41:50.954 INFO:tasks.rados.rados.2.smithi175.stdout:954: finishing write tid 1 to smithi17557870-7 2024-09-11T03:41:50.954 INFO:tasks.rados.rados.2.smithi175.stdout:954: finishing write tid 2 to smithi17557870-7 2024-09-11T03:41:50.955 INFO:tasks.rados.rados.2.smithi175.stdout:954: finishing write tid 3 to smithi17557870-7 2024-09-11T03:41:50.957 INFO:tasks.rados.rados.2.smithi175.stdout:954: finishing write tid 4 to smithi17557870-7 2024-09-11T03:41:50.957 INFO:tasks.rados.rados.2.smithi175.stdout:954: finishing write tid 5 to smithi17557870-7 2024-09-11T03:41:50.957 INFO:tasks.rados.rados.2.smithi175.stdout:953: done (2 left) 2024-09-11T03:41:50.957 INFO:tasks.rados.rados.2.smithi175.stdout:955: done (1 left) 2024-09-11T03:41:50.957 INFO:tasks.rados.rados.2.smithi175.stdout:956: copy_from oid 36 from oid 26 current snap is 100 2024-09-11T03:41:50.958 INFO:tasks.rados.rados.2.smithi175.stdout:954: finishing write tid 6 to smithi17557870-7 2024-09-11T03:41:50.958 INFO:tasks.rados.rados.2.smithi175.stdout:954: oid 7 updating version 0 to 173 2024-09-11T03:41:50.958 INFO:tasks.rados.rados.2.smithi175.stdout:954: oid 7 updating version 173 to 176 2024-09-11T03:41:50.958 INFO:tasks.rados.rados.2.smithi175.stdout:954: oid 7 version 176 is already newer than 174 2024-09-11T03:41:50.958 INFO:tasks.rados.rados.2.smithi175.stdout:954: oid 7 updating version 176 to 177 2024-09-11T03:41:50.959 INFO:tasks.rados.rados.2.smithi175.stdout:954: oid 7 version 177 is already newer than 175 2024-09-11T03:41:50.959 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 177 (ObjNum 347 snap 100 seq_num 347) dirty exists 2024-09-11T03:41:50.959 INFO:tasks.rados.rados.2.smithi175.stdout:954: left oid 7 (ObjNum 347 snap 100 seq_num 347) 2024-09-11T03:41:50.959 INFO:tasks.rados.rados.2.smithi175.stdout:954: done (1 left) 2024-09-11T03:41:50.959 INFO:tasks.rados.rados.2.smithi175.stdout:957: snap_create 2024-09-11T03:41:50.966 INFO:tasks.rados.rados.2.smithi175.stdout:956: finishing copy_from racing read to smithi17557870-36 2024-09-11T03:41:50.976 INFO:tasks.rados.rados.2.smithi175.stdout:956: finishing copy_from to smithi17557870-36 2024-09-11T03:41:50.976 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 157 (ObjNum 317 snap 86 seq_num 317) dirty exists 2024-09-11T03:41:51.045 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:51.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:50 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:41:51.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:50 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:41:51.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:50 smithi084 ceph-mon[27254]: osdmap e293: 12 total, 12 up, 10 in 2024-09-11T03:41:51.068 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:51.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:51.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:51.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:51.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:50 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:41:51.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:50 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:41:51.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:50 smithi121 ceph-mon[30729]: osdmap e293: 12 total, 12 up, 10 in 2024-09-11T03:41:51.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:51.194 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:51.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:51.212 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:50 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:41:51.213 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:50 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:41:51.213 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:50 smithi175 ceph-mon[30894]: osdmap e293: 12 total, 12 up, 10 in 2024-09-11T03:41:51.236 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:51.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:51.310 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:41:51.310 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:41:51.310 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:41:51.311 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:41:51.311 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:41:51.318 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:51.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:51.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:51.421 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:51.442 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:51.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:51.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:51.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:51.569 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:51.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:51.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:51.671 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:51.693 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:51.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:51.694 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:51.795 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:51.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:51.818 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:51.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:51.858 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T03:41:51.863 INFO:tasks.rados.rados.2.smithi175.stdout:956: done (1 left) 2024-09-11T03:41:51.863 INFO:tasks.rados.rados.2.smithi175.stdout:957: done (0 left) 2024-09-11T03:41:51.863 INFO:tasks.rados.rados.2.smithi175.stdout:958: rollback oid 11 current snap is 101 2024-09-11T03:41:51.863 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 11 to 95 2024-09-11T03:41:51.864 INFO:tasks.rados.rados.2.smithi175.stdout:959: read oid 22 snap -1 2024-09-11T03:41:51.864 INFO:tasks.rados.rados.2.smithi175.stdout:959: expect (ObjNum 342 snap 99 seq_num 342) 2024-09-11T03:41:51.864 INFO:tasks.rados.rados.2.smithi175.stdout:960: delete oid 3 current snap is 101 2024-09-11T03:41:51.867 INFO:tasks.rados.rados.2.smithi175.stdout:958: finishing rollback tid 0 to smithi17557870-11 2024-09-11T03:41:51.869 INFO:tasks.rados.rados.2.smithi175.stdout:958: finishing rollback tid 1 to smithi17557870-11 2024-09-11T03:41:51.869 INFO:tasks.rados.rados.2.smithi175.stdout:958: finishing rollback tid 2 to smithi17557870-11 2024-09-11T03:41:51.869 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 125 (ObjNum 270 snap 69 seq_num 270) dirty exists 2024-09-11T03:41:51.875 INFO:tasks.rados.rados.2.smithi175.stdout:958: done (2 left) 2024-09-11T03:41:51.875 INFO:tasks.rados.rados.2.smithi175.stdout:960: done (1 left) 2024-09-11T03:41:51.876 INFO:tasks.rados.rados.2.smithi175.stdout:961: copy_from oid 40 from oid 39 current snap is 101 2024-09-11T03:41:51.876 INFO:tasks.rados.rados.2.smithi175.stdout:959: expect (ObjNum 342 snap 99 seq_num 342) 2024-09-11T03:41:51.919 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-11T03:41:51.919 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:51.940 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:51.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:51.941 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:52.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:52.049 INFO:tasks.rados.rados.2.smithi175.stdout:961: finishing copy_from racing read to smithi17557870-40 2024-09-11T03:41:52.049 INFO:tasks.rados.rados.2.smithi175.stdout:961: finishing copy_from to smithi17557870-40 2024-09-11T03:41:52.049 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 279 (ObjNum 262 snap 66 seq_num 262) dirty exists 2024-09-11T03:41:52.049 INFO:tasks.rados.rados.2.smithi175.stdout:959: done (1 left) 2024-09-11T03:41:52.049 INFO:tasks.rados.rados.2.smithi175.stdout:961: done (0 left) 2024-09-11T03:41:52.049 INFO:tasks.rados.rados.2.smithi175.stdout:962: delete oid 33 current snap is 101 2024-09-11T03:41:52.051 INFO:tasks.rados.rados.2.smithi175.stdout:962: done (0 left) 2024-09-11T03:41:52.051 INFO:tasks.rados.rados.2.smithi175.stdout:963: copy_from oid 28 from oid 22 current snap is 101 2024-09-11T03:41:52.051 INFO:tasks.rados.rados.2.smithi175.stdout:964: delete oid 1 current snap is 101 2024-09-11T03:41:52.053 INFO:tasks.rados.rados.2.smithi175.stdout:964: done (1 left) 2024-09-11T03:41:52.053 INFO:tasks.rados.rados.2.smithi175.stdout:965: snap_remove snap 100 2024-09-11T03:41:52.069 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:52.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:52.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:51 smithi084 ceph-mon[27254]: pgmap v595: 57 pgs: 57 active+clean; 159 MiB data, 655 MiB used, 893 GiB / 894 GiB avail; 4.4 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-11T03:41:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:51 smithi084 ceph-mon[27254]: osdmap e294: 12 total, 12 up, 10 in 2024-09-11T03:41:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1785358088' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:41:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1785358088' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:41:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:51 smithi084 ceph-mon[27254]: osdmap e295: 12 total, 12 up, 10 in 2024-09-11T03:41:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1224807698' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:52.162 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:41:52.171 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:51 smithi121 ceph-mon[30729]: pgmap v595: 57 pgs: 57 active+clean; 159 MiB data, 655 MiB used, 893 GiB / 894 GiB avail; 4.4 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-11T03:41:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:51 smithi121 ceph-mon[30729]: osdmap e294: 12 total, 12 up, 10 in 2024-09-11T03:41:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1785358088' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:41:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1785358088' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:41:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:51 smithi121 ceph-mon[30729]: osdmap e295: 12 total, 12 up, 10 in 2024-09-11T03:41:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1224807698' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:52.195 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:52.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:52.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:51 smithi175 ceph-mon[30894]: pgmap v595: 57 pgs: 57 active+clean; 159 MiB data, 655 MiB used, 893 GiB / 894 GiB avail; 4.4 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-11T03:41:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:51 smithi175 ceph-mon[30894]: osdmap e294: 12 total, 12 up, 10 in 2024-09-11T03:41:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1785358088' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:41:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1785358088' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:41:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:51 smithi175 ceph-mon[30894]: osdmap e295: 12 total, 12 up, 10 in 2024-09-11T03:41:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1224807698' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:41:52.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:52.321 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:52.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:52.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:52.423 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:52.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:52.448 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:52.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:52.458 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-11T03:41:52.550 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:52.573 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:52.573 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:52.599 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:41:52.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:52.677 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:41:52.677 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:41:52.678 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:41:52.678 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:41:52.678 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.678 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:640 2.a e131/95 2:5014e305:::smithi17557870-46:head v 131'5, mlcod=100'2)", 2024-09-11T03:41:52.678 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:41:52.678 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 164.55786284999999, 2024-09-11T03:41:52.678 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068845190000000004, 2024-09-11T03:41:52.678 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.678 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.678 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:52.678 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.678 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.678 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.679 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:41:52.679 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.679 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.679 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.679 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.679 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:41:52.679 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.679 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.680 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.680 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.680 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:41:52.680 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7399999999999998e-07 2024-09-11T03:41:52.680 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.680 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.680 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.680 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.120287+0000", 2024-09-11T03:41:52.680 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0018676809999999999 2024-09-11T03:41:52.680 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.680 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.680 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.680 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.120324+0000", 2024-09-11T03:41:52.681 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7082000000000002e-05 2024-09-11T03:41:52.681 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.681 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.681 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.681 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.120345+0000", 2024-09-11T03:41:52.681 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0888000000000001e-05 2024-09-11T03:41:52.681 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.681 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.681 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.681 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.120456+0000", 2024-09-11T03:41:52.682 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000111328 2024-09-11T03:41:52.682 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.682 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.682 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.682 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.120478+0000", 2024-09-11T03:41:52.682 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1732000000000001e-05 2024-09-11T03:41:52.682 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.682 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.683 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:41:52.683 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125257+0000", 2024-09-11T03:41:52.683 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0047789479999999999 2024-09-11T03:41:52.683 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.683 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.683 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.683 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125303+0000", 2024-09-11T03:41:52.683 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6385999999999999e-05 2024-09-11T03:41:52.683 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.683 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.683 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.683 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.683 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.684 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:847 2.d 2:b1c0f730:::smithi17557870-20:head [copy-from ver 90] snapc 1a=[1a,19] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-11T03:41:52.684 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:41:52.684 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 147.83698667900001, 2024-09-11T03:41:52.684 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013054913, 2024-09-11T03:41:52.684 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.684 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.684 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:52.684 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.684 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.684 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.684 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 847 2024-09-11T03:41:52.684 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.684 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.684 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.685 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.685 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:41:52.685 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.685 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.685 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.685 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.685 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:41:52.685 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.685 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.685 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.685 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.685 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839297+0000", 2024-09-11T03:41:52.685 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0049999999999999e-06 2024-09-11T03:41:52.686 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.686 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.686 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.686 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839309+0000", 2024-09-11T03:41:52.686 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1531e-05 2024-09-11T03:41:52.686 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.686 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.686 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.686 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839326+0000", 2024-09-11T03:41:52.686 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7788e-05 2024-09-11T03:41:52.686 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.686 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.687 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.687 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839336+0000", 2024-09-11T03:41:52.687 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5619999999999993e-06 2024-09-11T03:41:52.687 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.687 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.687 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.687 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839423+0000", 2024-09-11T03:41:52.687 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6790000000000001e-05 2024-09-11T03:41:52.687 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.687 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.687 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:41:52.687 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839434+0000", 2024-09-11T03:41:52.688 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0971e-05 2024-09-11T03:41:52.688 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.688 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.688 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.688 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841520+0000", 2024-09-11T03:41:52.688 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0020866299999999999 2024-09-11T03:41:52.688 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.688 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.688 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.688 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841590+0000", 2024-09-11T03:41:52.688 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9288000000000005e-05 2024-09-11T03:41:52.688 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.688 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.689 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:41:52.689 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.846430+0000", 2024-09-11T03:41:52.689 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0048408349999999999 2024-09-11T03:41:52.689 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.689 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.689 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:41:52.689 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849726+0000", 2024-09-11T03:41:52.689 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032955390000000001 2024-09-11T03:41:52.689 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.689 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.689 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.690 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852240+0000", 2024-09-11T03:41:52.690 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002513492 2024-09-11T03:41:52.690 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.690 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.690 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.690 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852278+0000", 2024-09-11T03:41:52.690 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7988999999999998e-05 2024-09-11T03:41:52.690 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.690 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.690 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:41:52.691 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852317+0000", 2024-09-11T03:41:52.691 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9310000000000001e-05 2024-09-11T03:41:52.691 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.691 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.691 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.691 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852350+0000", 2024-09-11T03:41:52.691 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3182999999999997e-05 2024-09-11T03:41:52.691 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.691 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.691 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.691 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.691 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.691 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:848 2.d 2:b1c0f730:::smithi17557870-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2024-09-11T03:41:52.691 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:24.839354+0000", 2024-09-11T03:41:52.692 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 147.836927282, 2024-09-11T03:41:52.692 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010055476000000001, 2024-09-11T03:41:52.692 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.692 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.692 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:41:52.692 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.692 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.692 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.692 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 848 2024-09-11T03:41:52.692 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.692 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.693 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.693 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.693 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839354+0000", 2024-09-11T03:41:52.693 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.693 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.693 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.693 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.693 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839354+0000", 2024-09-11T03:41:52.693 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.693 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.693 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.693 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.693 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839355+0000", 2024-09-11T03:41:52.693 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6899999999999996e-07 2024-09-11T03:41:52.694 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.694 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.694 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.694 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839358+0000", 2024-09-11T03:41:52.694 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8729999999999999e-06 2024-09-11T03:41:52.694 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.694 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.694 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.694 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839367+0000", 2024-09-11T03:41:52.694 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3999999999999992e-06 2024-09-11T03:41:52.694 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.694 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.694 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.694 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839371+0000", 2024-09-11T03:41:52.695 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6059999999999996e-06 2024-09-11T03:41:52.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.695 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.695 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.695 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839509+0000", 2024-09-11T03:41:52.695 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013766700000000001 2024-09-11T03:41:52.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.695 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.695 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:41:52.695 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839515+0000", 2024-09-11T03:41:52.695 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4740000000000002e-06 2024-09-11T03:41:52.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.695 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.696 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.696 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841883+0000", 2024-09-11T03:41:52.696 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023678319999999998 2024-09-11T03:41:52.696 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.696 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.696 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:41:52.696 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841941+0000", 2024-09-11T03:41:52.696 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8241e-05 2024-09-11T03:41:52.696 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.696 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.696 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.697 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849208+0000", 2024-09-11T03:41:52.697 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0072661690000000003 2024-09-11T03:41:52.697 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.697 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.697 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.697 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849295+0000", 2024-09-11T03:41:52.697 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7582000000000003e-05 2024-09-11T03:41:52.697 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.697 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.697 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.697 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849410+0000", 2024-09-11T03:41:52.697 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114863 2024-09-11T03:41:52.697 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.697 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.698 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.698 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.698 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.698 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:907 2.d 2:b6a25a79:::smithi17557870-47:head [copy-from ver 47] snapc 20=[20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-11T03:41:52.698 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:34.902174+0000", 2024-09-11T03:41:52.698 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 137.774108224, 2024-09-11T03:41:52.698 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011234144, 2024-09-11T03:41:52.698 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.698 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.698 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:52.698 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.698 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.699 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.699 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 907 2024-09-11T03:41:52.699 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.699 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.699 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.699 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.699 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902174+0000", 2024-09-11T03:41:52.699 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.699 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.699 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.699 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.700 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902174+0000", 2024-09-11T03:41:52.700 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.700 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.700 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.700 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.700 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902175+0000", 2024-09-11T03:41:52.700 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0189999999999999e-06 2024-09-11T03:41:52.700 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.700 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.700 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.700 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902178+0000", 2024-09-11T03:41:52.700 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4410000000000002e-06 2024-09-11T03:41:52.700 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.701 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.701 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.701 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902188+0000", 2024-09-11T03:41:52.701 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0023999999999999e-05 2024-09-11T03:41:52.701 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.701 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.701 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.701 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902193+0000", 2024-09-11T03:41:52.701 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.31e-06 2024-09-11T03:41:52.701 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.701 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.701 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.701 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902248+0000", 2024-09-11T03:41:52.702 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4562999999999999e-05 2024-09-11T03:41:52.702 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.702 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.702 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.702 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902344+0000", 2024-09-11T03:41:52.702 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5841999999999995e-05 2024-09-11T03:41:52.702 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.702 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.702 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:41:52.702 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.907645+0000", 2024-09-11T03:41:52.702 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0053015579999999996 2024-09-11T03:41:52.702 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.702 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.702 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:41:52.703 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.909768+0000", 2024-09-11T03:41:52.703 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0021225020000000001 2024-09-11T03:41:52.703 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.703 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.703 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.703 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.912827+0000", 2024-09-11T03:41:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003059375 2024-09-11T03:41:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.913334+0000", 2024-09-11T03:41:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00050710499999999999 2024-09-11T03:41:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:41:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.913372+0000", 2024-09-11T03:41:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7486e-05 2024-09-11T03:41:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.705 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.705 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.913408+0000", 2024-09-11T03:41:52.705 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5918999999999999e-05 2024-09-11T03:41:52.705 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.705 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.705 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.705 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.705 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.705 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:908 2.d 2:b6a25a79:::smithi17557870-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e158)", 2024-09-11T03:41:52.705 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:34.902203+0000", 2024-09-11T03:41:52.705 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 137.77407839, 2024-09-11T03:41:52.706 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0073357650000000002, 2024-09-11T03:41:52.706 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.706 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.706 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:41:52.706 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.706 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.706 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.706 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 908 2024-09-11T03:41:52.706 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.706 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.706 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.706 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.706 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902203+0000", 2024-09-11T03:41:52.706 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.707 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.707 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.707 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.707 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902203+0000", 2024-09-11T03:41:52.707 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.707 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.707 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.707 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.707 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902204+0000", 2024-09-11T03:41:52.707 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4400000000000001e-07 2024-09-11T03:41:52.707 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.707 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.707 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.707 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902205+0000", 2024-09-11T03:41:52.708 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.266e-06 2024-09-11T03:41:52.708 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.708 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.708 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.708 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902210+0000", 2024-09-11T03:41:52.708 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9169999999999997e-06 2024-09-11T03:41:52.708 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.708 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.708 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.708 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902213+0000", 2024-09-11T03:41:52.708 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1099999999999999e-06 2024-09-11T03:41:52.708 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.708 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.708 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.709 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902498+0000", 2024-09-11T03:41:52.709 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00028460399999999999 2024-09-11T03:41:52.709 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.709 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.709 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:41:52.709 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902548+0000", 2024-09-11T03:41:52.709 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0503000000000001e-05 2024-09-11T03:41:52.709 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:52.712 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:52.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:52.713 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.713 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.713 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.909356+0000", 2024-09-11T03:41:52.713 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068078239999999996 2024-09-11T03:41:52.713 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.713 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.713 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.713 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.909444+0000", 2024-09-11T03:41:52.713 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8362999999999994e-05 2024-09-11T03:41:52.713 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.713 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.713 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.713 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.909539+0000", 2024-09-11T03:41:52.714 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4833999999999994e-05 2024-09-11T03:41:52.714 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.714 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.714 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.714 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.714 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.714 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1290 2.d 2:b1c0f730:::smithi17557870-20:head [copy-from ver 48] snapc 33=[32,30,2c,2b] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:41:52.714 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:12.836022+0000", 2024-09-11T03:41:52.714 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 99.840259398000001, 2024-09-11T03:41:52.714 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022892863999999999, 2024-09-11T03:41:52.714 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.714 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.714 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:52.715 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.715 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.715 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.715 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1290 2024-09-11T03:41:52.715 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.715 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.715 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.715 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.715 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836022+0000", 2024-09-11T03:41:52.715 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.715 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.715 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.715 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.716 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836022+0000", 2024-09-11T03:41:52.716 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.716 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.716 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.716 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.716 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836025+0000", 2024-09-11T03:41:52.716 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6690000000000002e-06 2024-09-11T03:41:52.716 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.716 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.716 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.716 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836033+0000", 2024-09-11T03:41:52.716 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0809999999999994e-06 2024-09-11T03:41:52.716 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.717 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.717 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.717 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836053+0000", 2024-09-11T03:41:52.717 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9429999999999999e-05 2024-09-11T03:41:52.717 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.717 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.717 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.717 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836063+0000", 2024-09-11T03:41:52.717 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0438000000000001e-05 2024-09-11T03:41:52.717 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.717 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.717 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.717 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836165+0000", 2024-09-11T03:41:52.718 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000101523 2024-09-11T03:41:52.718 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.718 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.718 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:41:52.718 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836172+0000", 2024-09-11T03:41:52.718 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6499999999999996e-06 2024-09-11T03:41:52.718 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.718 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.718 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.718 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839394+0000", 2024-09-11T03:41:52.718 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032215590000000001 2024-09-11T03:41:52.718 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.718 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839507+0000", 2024-09-11T03:41:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000113557 2024-09-11T03:41:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:41:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.849905+0000", 2024-09-11T03:41:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010398174 2024-09-11T03:41:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:41:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.853428+0000", 2024-09-11T03:41:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035222869999999998 2024-09-11T03:41:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.855923+0000", 2024-09-11T03:41:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024951840000000001 2024-09-11T03:41:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.858781+0000", 2024-09-11T03:41:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002858368 2024-09-11T03:41:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:41:52.721 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.858881+0000", 2024-09-11T03:41:52.721 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9500999999999995e-05 2024-09-11T03:41:52.721 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.721 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.721 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.721 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.858915+0000", 2024-09-11T03:41:52.721 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4443000000000002e-05 2024-09-11T03:41:52.721 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.721 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.721 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.721 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.721 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.721 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1291 2.d 2:b1c0f730:::smithi17557870-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:41:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:12.836223+0000", 2024-09-11T03:41:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 99.840058651999996, 2024-09-11T03:41:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016933050000000002, 2024-09-11T03:41:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:41:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1291 2024-09-11T03:41:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836223+0000", 2024-09-11T03:41:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836223+0000", 2024-09-11T03:41:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836224+0000", 2024-09-11T03:41:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0020000000000001e-06 2024-09-11T03:41:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836229+0000", 2024-09-11T03:41:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7180000000000004e-06 2024-09-11T03:41:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836245+0000", 2024-09-11T03:41:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5981000000000002e-05 2024-09-11T03:41:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836253+0000", 2024-09-11T03:41:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1720000000000007e-06 2024-09-11T03:41:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836317+0000", 2024-09-11T03:41:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4189999999999994e-05 2024-09-11T03:41:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:41:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836324+0000", 2024-09-11T03:41:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9249999999999998e-06 2024-09-11T03:41:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839738+0000", 2024-09-11T03:41:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0034135580000000001 2024-09-11T03:41:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:41:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839804+0000", 2024-09-11T03:41:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5974000000000002e-05 2024-09-11T03:41:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.852933+0000", 2024-09-11T03:41:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013129429999999999 2024-09-11T03:41:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.853032+0000", 2024-09-11T03:41:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8592999999999996e-05 2024-09-11T03:41:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.853156+0000", 2024-09-11T03:41:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012450699999999999 2024-09-11T03:41:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1439 2.d 2:b6a25a79:::smithi17557870-47:35 [sparse-read 0~1600344,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e206)", 2024-09-11T03:41:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:22.839739+0000", 2024-09-11T03:41:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 89.836543015999993, 2024-09-11T03:41:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0071734040000000004, 2024-09-11T03:41:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:41:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1439 2024-09-11T03:41:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839739+0000", 2024-09-11T03:41:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839739+0000", 2024-09-11T03:41:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839741+0000", 2024-09-11T03:41:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9269999999999999e-06 2024-09-11T03:41:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839748+0000", 2024-09-11T03:41:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1330000000000001e-06 2024-09-11T03:41:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839811+0000", 2024-09-11T03:41:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3208999999999997e-05 2024-09-11T03:41:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839818+0000", 2024-09-11T03:41:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2729999999999997e-06 2024-09-11T03:41:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.840620+0000", 2024-09-11T03:41:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00080214800000000003 2024-09-11T03:41:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:41:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.840719+0000", 2024-09-11T03:41:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8240999999999997e-05 2024-09-11T03:41:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.732 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.732 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.732 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.845605+0000", 2024-09-11T03:41:52.732 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0048859810000000002 2024-09-11T03:41:52.732 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.732 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.732 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.732 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.845669+0000", 2024-09-11T03:41:52.732 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4109000000000006e-05 2024-09-11T03:41:52.732 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.732 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.732 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.732 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.846912+0000", 2024-09-11T03:41:52.733 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001243383 2024-09-11T03:41:52.733 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.733 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.733 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.733 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.733 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.733 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1440 2.d 2:b6a25a79:::smithi17557870-47:35 [read 0~3042067,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e206)", 2024-09-11T03:41:52.733 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:22.839838+0000", 2024-09-11T03:41:52.733 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 89.836443457000001, 2024-09-11T03:41:52.733 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0089542019999999996, 2024-09-11T03:41:52.733 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.733 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.733 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:41:52.734 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.734 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.734 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.734 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1440 2024-09-11T03:41:52.734 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.734 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.734 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.734 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.734 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839838+0000", 2024-09-11T03:41:52.734 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.734 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.734 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.734 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.735 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839838+0000", 2024-09-11T03:41:52.735 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.735 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.735 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.735 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.735 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839839+0000", 2024-09-11T03:41:52.735 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9500000000000002e-07 2024-09-11T03:41:52.735 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.735 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.735 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.735 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839843+0000", 2024-09-11T03:41:52.735 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3200000000000001e-06 2024-09-11T03:41:52.735 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.736 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.736 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.736 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839851+0000", 2024-09-11T03:41:52.736 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5170000000000002e-06 2024-09-11T03:41:52.736 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.736 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.736 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.736 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839856+0000", 2024-09-11T03:41:52.736 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5799999999999999e-06 2024-09-11T03:41:52.736 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.736 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.840764+0000", 2024-09-11T03:41:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00090780899999999998 2024-09-11T03:41:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:41:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.840857+0000", 2024-09-11T03:41:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2707999999999994e-05 2024-09-11T03:41:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.847171+0000", 2024-09-11T03:41:52.738 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0063140920000000003 2024-09-11T03:41:52.738 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.738 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.738 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.738 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.847241+0000", 2024-09-11T03:41:52.738 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0392e-05 2024-09-11T03:41:52.738 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.738 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.738 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.738 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.848793+0000", 2024-09-11T03:41:52.738 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015510890000000001 2024-09-11T03:41:52.738 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.738 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.738 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.739 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.739 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.739 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1441 2.d 2:b6a25a79:::smithi17557870-47:35 [sparse-read 0~1600344] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e206)", 2024-09-11T03:41:52.739 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:22.839865+0000", 2024-09-11T03:41:52.739 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 89.836416432999997, 2024-09-11T03:41:52.739 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0091540579999999996, 2024-09-11T03:41:52.739 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.739 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.739 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:41:52.739 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.739 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.739 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.739 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1441 2024-09-11T03:41:52.740 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.740 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.740 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.740 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.740 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839865+0000", 2024-09-11T03:41:52.740 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.740 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.740 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.740 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.740 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839865+0000", 2024-09-11T03:41:52.740 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.740 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.740 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.740 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.741 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839866+0000", 2024-09-11T03:41:52.741 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4499999999999997e-07 2024-09-11T03:41:52.741 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.741 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.741 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.741 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839869+0000", 2024-09-11T03:41:52.741 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.055e-06 2024-09-11T03:41:52.741 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.741 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.741 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.741 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839875+0000", 2024-09-11T03:41:52.741 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3319999999999997e-06 2024-09-11T03:41:52.742 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.742 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.742 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.742 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839880+0000", 2024-09-11T03:41:52.742 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0340000000000004e-06 2024-09-11T03:41:52.742 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.742 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.742 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.742 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.841228+0000", 2024-09-11T03:41:52.742 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001347521 2024-09-11T03:41:52.742 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.742 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.742 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:41:52.743 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.841306+0000", 2024-09-11T03:41:52.743 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8182000000000005e-05 2024-09-11T03:41:52.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.743 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.743 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.743 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.848858+0000", 2024-09-11T03:41:52.743 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0075521039999999996 2024-09-11T03:41:52.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.743 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.743 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.743 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.848919+0000", 2024-09-11T03:41:52.743 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.05e-05 2024-09-11T03:41:52.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.743 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.744 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.744 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.849019+0000", 2024-09-11T03:41:52.744 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000100785 2024-09-11T03:41:52.744 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.744 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.744 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.744 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.744 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.744 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1516 2.d 2:b6a25a79:::smithi17557870-47:head [copy-from ver 97] snapc 3b=[3a,38,37,35] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-09-11T03:41:52.744 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:28.919589+0000", 2024-09-11T03:41:52.744 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 83.756692521999994, 2024-09-11T03:41:52.744 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011956165, 2024-09-11T03:41:52.744 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.745 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.745 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:52.745 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.745 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.745 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.745 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1516 2024-09-11T03:41:52.745 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.745 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.745 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.745 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.745 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919589+0000", 2024-09-11T03:41:52.745 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.745 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.746 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.746 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.746 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919589+0000", 2024-09-11T03:41:52.746 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.746 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.746 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.746 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.746 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919591+0000", 2024-09-11T03:41:52.746 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5409999999999999e-06 2024-09-11T03:41:52.746 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.746 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.746 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.746 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919596+0000", 2024-09-11T03:41:52.746 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9280000000000001e-06 2024-09-11T03:41:52.747 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.747 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.747 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.747 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919610+0000", 2024-09-11T03:41:52.747 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4285e-05 2024-09-11T03:41:52.747 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.747 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.747 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.747 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919617+0000", 2024-09-11T03:41:52.747 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.435e-06 2024-09-11T03:41:52.747 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.747 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.747 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.748 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919688+0000", 2024-09-11T03:41:52.748 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0263000000000001e-05 2024-09-11T03:41:52.748 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.748 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.748 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.748 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919794+0000", 2024-09-11T03:41:52.748 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000106055 2024-09-11T03:41:52.748 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.748 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.748 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:41:52.748 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.922849+0000", 2024-09-11T03:41:52.748 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030557330000000001 2024-09-11T03:41:52.748 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.749 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.749 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:41:52.749 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.926556+0000", 2024-09-11T03:41:52.749 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0037066669999999999 2024-09-11T03:41:52.749 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.749 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.749 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.749 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.929197+0000", 2024-09-11T03:41:52.749 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002640529 2024-09-11T03:41:52.749 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.749 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.749 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.749 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.931432+0000", 2024-09-11T03:41:52.750 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022350339999999999 2024-09-11T03:41:52.750 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.750 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.750 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:41:52.750 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.931501+0000", 2024-09-11T03:41:52.750 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9584000000000003e-05 2024-09-11T03:41:52.750 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.750 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.750 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.750 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.931545+0000", 2024-09-11T03:41:52.750 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4110999999999998e-05 2024-09-11T03:41:52.750 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.750 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.750 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.751 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.751 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.751 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1554 2.d 2:b7bdfe52:::smithi17557870-10:37 [read 0~2016807,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e214)", 2024-09-11T03:41:52.751 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:30.880351+0000", 2024-09-11T03:41:52.751 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 81.795931117999999, 2024-09-11T03:41:52.751 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0077014559999999998, 2024-09-11T03:41:52.751 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.751 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.751 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:41:52.751 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.751 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.751 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.751 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1554 2024-09-11T03:41:52.752 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.752 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.752 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.752 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.752 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880351+0000", 2024-09-11T03:41:52.752 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.752 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.752 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.752 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.752 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880351+0000", 2024-09-11T03:41:52.752 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.752 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.752 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.753 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.753 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880351+0000", 2024-09-11T03:41:52.753 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5299999999999999e-07 2024-09-11T03:41:52.753 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.753 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.753 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.753 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880357+0000", 2024-09-11T03:41:52.753 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4110000000000004e-06 2024-09-11T03:41:52.753 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.754 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.754 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.754 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880364+0000", 2024-09-11T03:41:52.754 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3549999999999999e-06 2024-09-11T03:41:52.754 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.754 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.754 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.754 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880370+0000", 2024-09-11T03:41:52.754 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6500000000000001e-06 2024-09-11T03:41:52.754 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.754 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.755 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.755 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880542+0000", 2024-09-11T03:41:52.755 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017261800000000001 2024-09-11T03:41:52.755 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.755 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.755 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:41:52.755 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.880552+0000", 2024-09-11T03:41:52.755 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6229999999999992e-06 2024-09-11T03:41:52.755 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.755 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.755 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.755 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.887165+0000", 2024-09-11T03:41:52.755 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0066137189999999997 2024-09-11T03:41:52.756 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.756 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.756 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.756 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.887221+0000", 2024-09-11T03:41:52.756 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5826999999999999e-05 2024-09-11T03:41:52.756 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.756 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.756 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.756 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:30.888052+0000", 2024-09-11T03:41:52.757 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00083080000000000003 2024-09-11T03:41:52.757 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.757 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.757 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.757 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.757 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.757 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:1895 2.e e249/95 2:7efbbea8:::smithi17557870-24:head v 249'251, mlcod=246'244)", 2024-09-11T03:41:52.757 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:05.931998+0000", 2024-09-11T03:41:52.757 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 46.744284104999998, 2024-09-11T03:41:52.757 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0081793049999999996, 2024-09-11T03:41:52.757 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.757 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.758 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:52.758 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.758 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.758 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.758 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:05.931998+0000", 2024-09-11T03:41:52.758 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.758 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.758 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.758 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.758 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:05.931998+0000", 2024-09-11T03:41:52.758 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.758 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.758 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.759 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.759 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:05.932006+0000", 2024-09-11T03:41:52.759 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9889999999999992e-06 2024-09-11T03:41:52.759 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.759 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.759 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.759 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:05.938135+0000", 2024-09-11T03:41:52.759 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0061292350000000002 2024-09-11T03:41:52.759 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.759 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.759 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.759 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:05.938156+0000", 2024-09-11T03:41:52.759 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1095999999999999e-05 2024-09-11T03:41:52.759 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.760 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.760 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.760 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:05.938165+0000", 2024-09-11T03:41:52.760 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3060000000000003e-06 2024-09-11T03:41:52.760 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.760 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.760 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.760 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:05.938229+0000", 2024-09-11T03:41:52.760 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3224000000000003e-05 2024-09-11T03:41:52.760 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.761 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.761 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.761 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:05.938248+0000", 2024-09-11T03:41:52.761 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9519999999999999e-05 2024-09-11T03:41:52.761 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.761 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.761 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:41:52.761 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:05.940139+0000", 2024-09-11T03:41:52.761 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001890624 2024-09-11T03:41:52.761 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.761 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.761 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.762 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:05.940177+0000", 2024-09-11T03:41:52.762 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8311e-05 2024-09-11T03:41:52.762 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.762 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.762 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.762 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.762 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.762 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2254 2.18 2:1984a3a9:::smithi17557870-23:head [write 746310~635349,stat] snapc 5d=[5d,5a,59,58,56] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:41:52.762 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.883765+0000", 2024-09-11T03:41:52.762 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 15.792516408999999, 2024-09-11T03:41:52.762 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022154396, 2024-09-11T03:41:52.762 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.762 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.763 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:52.763 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.763 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.763 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.763 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2254 2024-09-11T03:41:52.763 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.763 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.763 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.763 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.763 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.883765+0000", 2024-09-11T03:41:52.763 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.763 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.763 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.764 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.764 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.883765+0000", 2024-09-11T03:41:52.764 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.764 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.764 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.764 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.764 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.883768+0000", 2024-09-11T03:41:52.764 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4080000000000001e-06 2024-09-11T03:41:52.764 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.764 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.764 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.764 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884378+0000", 2024-09-11T03:41:52.764 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00061034200000000005 2024-09-11T03:41:52.765 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.765 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.765 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.765 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884399+0000", 2024-09-11T03:41:52.765 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1175e-05 2024-09-11T03:41:52.765 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.765 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.765 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.765 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884412+0000", 2024-09-11T03:41:52.765 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3015e-05 2024-09-11T03:41:52.765 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.765 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.766 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.766 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884520+0000", 2024-09-11T03:41:52.766 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000107706 2024-09-11T03:41:52.766 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.766 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.766 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:41:52.766 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884534+0000", 2024-09-11T03:41:52.766 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3923e-05 2024-09-11T03:41:52.766 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.766 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.766 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.766 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.895937+0000", 2024-09-11T03:41:52.766 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011403207 2024-09-11T03:41:52.767 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.767 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.767 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:41:52.767 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896424+0000", 2024-09-11T03:41:52.767 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00048641199999999998 2024-09-11T03:41:52.767 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.767 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.767 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.767 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.898138+0000", 2024-09-11T03:41:52.767 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001714272 2024-09-11T03:41:52.767 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.767 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.767 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.767 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.898309+0000", 2024-09-11T03:41:52.768 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017075200000000001 2024-09-11T03:41:52.768 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.768 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.768 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:41:52.768 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.898561+0000", 2024-09-11T03:41:52.768 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00025223500000000001 2024-09-11T03:41:52.768 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.768 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.768 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.769 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903048+0000", 2024-09-11T03:41:52.769 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0044876320000000001 2024-09-11T03:41:52.769 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.769 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.769 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.769 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903118+0000", 2024-09-11T03:41:52.769 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9608000000000002e-05 2024-09-11T03:41:52.769 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.769 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.769 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:41:52.770 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905694+0000", 2024-09-11T03:41:52.770 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025761650000000001 2024-09-11T03:41:52.770 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.770 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.770 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:41:52.770 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905749+0000", 2024-09-11T03:41:52.770 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5037999999999997e-05 2024-09-11T03:41:52.770 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.770 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.770 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.770 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905920+0000", 2024-09-11T03:41:52.770 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017050600000000001 2024-09-11T03:41:52.771 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.771 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.771 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.771 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.771 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.771 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2255 2.18 2:1984a3a9:::smithi17557870-23:head [write 2077646~576443] snapc 5d=[5d,5a,59,58,56] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:41:52.771 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.884463+0000", 2024-09-11T03:41:52.771 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 15.791818678, 2024-09-11T03:41:52.771 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022426167, 2024-09-11T03:41:52.771 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.771 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.771 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:52.771 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.772 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.772 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.772 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2255 2024-09-11T03:41:52.772 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.772 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.772 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.772 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.772 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884463+0000", 2024-09-11T03:41:52.772 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.772 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.772 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.773 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.773 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884463+0000", 2024-09-11T03:41:52.773 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.773 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.773 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.773 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.773 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884464+0000", 2024-09-11T03:41:52.773 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.64e-07 2024-09-11T03:41:52.774 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.774 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.774 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.774 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885204+0000", 2024-09-11T03:41:52.774 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00074007200000000004 2024-09-11T03:41:52.774 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.774 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.774 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.774 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885217+0000", 2024-09-11T03:41:52.774 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2547e-05 2024-09-11T03:41:52.774 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.775 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.775 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.775 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885226+0000", 2024-09-11T03:41:52.775 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1849999999999993e-06 2024-09-11T03:41:52.775 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.775 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.775 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.775 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885295+0000", 2024-09-11T03:41:52.775 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8657000000000004e-05 2024-09-11T03:41:52.775 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.775 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.776 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:41:52.776 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885307+0000", 2024-09-11T03:41:52.776 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2529e-05 2024-09-11T03:41:52.776 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.776 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.776 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.776 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896570+0000", 2024-09-11T03:41:52.776 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011262529 2024-09-11T03:41:52.776 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.776 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.776 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:41:52.777 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896624+0000", 2024-09-11T03:41:52.777 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4614000000000003e-05 2024-09-11T03:41:52.777 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.777 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.777 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.777 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.900547+0000", 2024-09-11T03:41:52.777 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039228850000000001 2024-09-11T03:41:52.777 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.777 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.777 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.778 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.900706+0000", 2024-09-11T03:41:52.778 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015907000000000001 2024-09-11T03:41:52.778 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.778 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.778 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:41:52.778 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.900901+0000", 2024-09-11T03:41:52.778 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00019514399999999999 2024-09-11T03:41:52.778 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.778 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.778 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.778 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903192+0000", 2024-09-11T03:41:52.779 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022906519999999998 2024-09-11T03:41:52.779 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.779 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.779 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.779 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.904724+0000", 2024-09-11T03:41:52.779 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015319520000000001 2024-09-11T03:41:52.779 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.779 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.779 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:41:52.779 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906254+0000", 2024-09-11T03:41:52.779 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015302759999999999 2024-09-11T03:41:52.780 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.780 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.780 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:41:52.780 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906358+0000", 2024-09-11T03:41:52.780 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010404300000000001 2024-09-11T03:41:52.780 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.780 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.780 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.780 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906889+0000", 2024-09-11T03:41:52.780 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000531048 2024-09-11T03:41:52.780 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.781 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.781 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.781 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.781 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.781 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2256 2.18 2:1984a3a9:::smithi17557870-23:head [write 3362764~183546] snapc 5d=[5d,5a,59,58,56] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:41:52.781 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.885408+0000", 2024-09-11T03:41:52.781 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 15.790873342999999, 2024-09-11T03:41:52.781 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021926139000000001, 2024-09-11T03:41:52.781 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.781 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.781 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:52.782 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.782 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.782 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.782 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2256 2024-09-11T03:41:52.782 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.782 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.782 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.782 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.782 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885408+0000", 2024-09-11T03:41:52.782 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.783 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.783 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.783 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.783 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885408+0000", 2024-09-11T03:41:52.783 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.783 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.783 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.783 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.783 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885410+0000", 2024-09-11T03:41:52.783 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4270000000000001e-06 2024-09-11T03:41:52.783 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.783 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.784 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.784 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885561+0000", 2024-09-11T03:41:52.784 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000151254 2024-09-11T03:41:52.784 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.784 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.784 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.784 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885573+0000", 2024-09-11T03:41:52.785 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1897e-05 2024-09-11T03:41:52.785 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.785 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.785 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.785 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885581+0000", 2024-09-11T03:41:52.785 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6290000000000001e-06 2024-09-11T03:41:52.785 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.785 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.785 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.785 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885643+0000", 2024-09-11T03:41:52.785 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2215e-05 2024-09-11T03:41:52.786 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.786 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.786 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:41:52.786 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885652+0000", 2024-09-11T03:41:52.786 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5580000000000006e-06 2024-09-11T03:41:52.786 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.786 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.786 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.786 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896678+0000", 2024-09-11T03:41:52.786 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011025956 2024-09-11T03:41:52.786 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.787 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.787 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:41:52.787 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896729+0000", 2024-09-11T03:41:52.787 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1019000000000001e-05 2024-09-11T03:41:52.787 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.787 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.787 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.787 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903250+0000", 2024-09-11T03:41:52.787 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0065205109999999997 2024-09-11T03:41:52.787 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.787 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.788 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.788 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903415+0000", 2024-09-11T03:41:52.788 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016545399999999999 2024-09-11T03:41:52.788 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.788 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.788 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:41:52.788 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903701+0000", 2024-09-11T03:41:52.788 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00028586399999999999 2024-09-11T03:41:52.788 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.788 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.788 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.789 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905681+0000", 2024-09-11T03:41:52.789 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0019798950000000002 2024-09-11T03:41:52.789 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.789 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.789 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.789 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905999+0000", 2024-09-11T03:41:52.789 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00031789999999999998 2024-09-11T03:41:52.789 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.789 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.789 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:41:52.789 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906904+0000", 2024-09-11T03:41:52.789 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00090535599999999996 2024-09-11T03:41:52.790 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.790 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.790 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:41:52.790 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907002+0000", 2024-09-11T03:41:52.790 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7653999999999997e-05 2024-09-11T03:41:52.790 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.790 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.790 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.790 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907335+0000", 2024-09-11T03:41:52.790 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00033254999999999998 2024-09-11T03:41:52.791 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.791 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.791 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.791 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.791 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.791 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2257 2.18 2:1984a3a9:::smithi17557870-23:head [setxattr _header (57) in=7b,truncate 3546310] snapc 5d=[5d,5a,59,58,56] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:41:52.791 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.885667+0000", 2024-09-11T03:41:52.791 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 15.790614322, 2024-09-11T03:41:52.791 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022144878999999999, 2024-09-11T03:41:52.791 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.791 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.791 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:52.792 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.792 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.792 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.792 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2257 2024-09-11T03:41:52.792 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.792 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.792 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.792 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.792 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885667+0000", 2024-09-11T03:41:52.792 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.792 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.792 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.793 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.793 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885667+0000", 2024-09-11T03:41:52.793 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.793 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.793 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.793 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.793 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885668+0000", 2024-09-11T03:41:52.793 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5600000000000005e-07 2024-09-11T03:41:52.793 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.793 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.793 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.793 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885673+0000", 2024-09-11T03:41:52.794 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4970000000000003e-06 2024-09-11T03:41:52.794 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.794 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.794 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.794 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885680+0000", 2024-09-11T03:41:52.794 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3780000000000003e-06 2024-09-11T03:41:52.794 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.794 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.794 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.794 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885686+0000", 2024-09-11T03:41:52.794 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0789999999999997e-06 2024-09-11T03:41:52.794 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.795 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.795 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.795 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886080+0000", 2024-09-11T03:41:52.795 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00039392399999999998 2024-09-11T03:41:52.795 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.795 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.795 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:41:52.795 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886091+0000", 2024-09-11T03:41:52.795 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0472000000000001e-05 2024-09-11T03:41:52.795 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.795 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.795 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.796 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896776+0000", 2024-09-11T03:41:52.796 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010684953000000001 2024-09-11T03:41:52.796 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.796 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.796 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:41:52.796 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896839+0000", 2024-09-11T03:41:52.796 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3293999999999994e-05 2024-09-11T03:41:52.796 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.796 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.796 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.796 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.904784+0000", 2024-09-11T03:41:52.797 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.007944843 2024-09-11T03:41:52.797 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.797 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.797 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.797 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.904909+0000", 2024-09-11T03:41:52.797 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012491600000000001 2024-09-11T03:41:52.797 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.797 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.797 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:41:52.797 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905105+0000", 2024-09-11T03:41:52.797 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000196336 2024-09-11T03:41:52.798 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.798 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.798 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.798 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906057+0000", 2024-09-11T03:41:52.798 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00095239800000000002 2024-09-11T03:41:52.798 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.798 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.798 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:41:52.798 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907348+0000", 2024-09-11T03:41:52.798 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012904489999999999 2024-09-11T03:41:52.798 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.798 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.799 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.799 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907457+0000", 2024-09-11T03:41:52.799 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108869 2024-09-11T03:41:52.799 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.799 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.799 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:41:52.799 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907543+0000", 2024-09-11T03:41:52.799 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6212999999999996e-05 2024-09-11T03:41:52.799 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.799 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.799 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.799 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907812+0000", 2024-09-11T03:41:52.799 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00026950200000000002 2024-09-11T03:41:52.800 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.800 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.800 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.800 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.800 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.800 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2258 2.18 2:1984a3a9:::smithi17557870-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:41:52.800 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.886095+0000", 2024-09-11T03:41:52.800 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 15.790186557, 2024-09-11T03:41:52.800 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022147336, 2024-09-11T03:41:52.800 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.800 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.801 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:41:52.801 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.801 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.801 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.801 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2258 2024-09-11T03:41:52.801 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.801 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.801 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.801 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.801 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886095+0000", 2024-09-11T03:41:52.801 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.801 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.802 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.802 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.802 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886095+0000", 2024-09-11T03:41:52.802 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.802 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.802 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.802 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.802 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886096+0000", 2024-09-11T03:41:52.802 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0200000000000001e-07 2024-09-11T03:41:52.802 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.802 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.802 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.802 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886100+0000", 2024-09-11T03:41:52.803 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9859999999999996e-06 2024-09-11T03:41:52.803 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.803 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.803 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.803 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886109+0000", 2024-09-11T03:41:52.803 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6349999999999995e-06 2024-09-11T03:41:52.803 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.803 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.803 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.803 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886116+0000", 2024-09-11T03:41:52.803 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0890000000000002e-06 2024-09-11T03:41:52.803 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.804 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.804 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.804 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886389+0000", 2024-09-11T03:41:52.804 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00027359200000000002 2024-09-11T03:41:52.804 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.804 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.804 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:41:52.804 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886399+0000", 2024-09-11T03:41:52.804 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9599999999999995e-06 2024-09-11T03:41:52.804 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.805 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.805 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.805 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896922+0000", 2024-09-11T03:41:52.805 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010522567 2024-09-11T03:41:52.805 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.805 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.805 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:41:52.805 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896977+0000", 2024-09-11T03:41:52.805 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5003999999999997e-05 2024-09-11T03:41:52.805 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.805 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.805 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.805 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905540+0000", 2024-09-11T03:41:52.806 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0085636329999999993 2024-09-11T03:41:52.806 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.806 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.806 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:41:52.806 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905625+0000", 2024-09-11T03:41:52.806 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4468999999999994e-05 2024-09-11T03:41:52.806 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.806 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.806 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.806 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907869+0000", 2024-09-11T03:41:52.806 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022438290000000001 2024-09-11T03:41:52.806 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.807 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.807 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.807 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.908014+0000", 2024-09-11T03:41:52.807 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00014511799999999999 2024-09-11T03:41:52.807 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.807 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.807 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.807 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.908243+0000", 2024-09-11T03:41:52.807 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00022865200000000001 2024-09-11T03:41:52.807 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.808 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.808 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.808 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.808 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.808 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2421 2.18 2:1bdf1a36:::smithi17557870-36:head [copy-from ver 157] snapc 65=[65,64,63,61,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-11T03:41:52.808 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:50.954849+0000", 2024-09-11T03:41:52.808 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 1.7214325270000002, 2024-09-11T03:41:52.808 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022711387999999999, 2024-09-11T03:41:52.808 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.808 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.808 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:41:52.808 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.809 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.809 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.809 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2421 2024-09-11T03:41:52.809 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.809 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.809 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.809 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.809 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954849+0000", 2024-09-11T03:41:52.810 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.810 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.810 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.810 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.810 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954849+0000", 2024-09-11T03:41:52.810 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.810 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.810 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.810 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.810 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954851+0000", 2024-09-11T03:41:52.810 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3090000000000001e-06 2024-09-11T03:41:52.810 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.811 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.811 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.811 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954856+0000", 2024-09-11T03:41:52.811 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9059999999999999e-06 2024-09-11T03:41:52.811 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.812 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.812 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.812 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954867+0000", 2024-09-11T03:41:52.812 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0879999999999999e-05 2024-09-11T03:41:52.812 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.812 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.812 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.812 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954873+0000", 2024-09-11T03:41:52.812 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.203e-06 2024-09-11T03:41:52.812 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.812 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.813 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.813 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954921+0000", 2024-09-11T03:41:52.813 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8180000000000003e-05 2024-09-11T03:41:52.813 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.813 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.813 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.813 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.955004+0000", 2024-09-11T03:41:52.813 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2824000000000005e-05 2024-09-11T03:41:52.813 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.813 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.814 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:41:52.814 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964099+0000", 2024-09-11T03:41:52.814 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0090952499999999992 2024-09-11T03:41:52.814 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.814 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.814 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.814 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.970182+0000", 2024-09-11T03:41:52.814 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0060831510000000002 2024-09-11T03:41:52.814 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.814 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.814 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:41:52.814 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977039+0000", 2024-09-11T03:41:52.815 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068571980000000001 2024-09-11T03:41:52.815 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.815 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.815 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:41:52.815 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977432+0000", 2024-09-11T03:41:52.815 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00039304900000000002 2024-09-11T03:41:52.815 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.815 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.815 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:41:52.816 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977504+0000", 2024-09-11T03:41:52.816 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1863e-05 2024-09-11T03:41:52.816 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.816 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.816 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.816 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977561+0000", 2024-09-11T03:41:52.816 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6575e-05 2024-09-11T03:41:52.816 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.816 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.816 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.817 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.817 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.817 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2422 2.18 2:1bdf1a36:::smithi17557870-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e294)", 2024-09-11T03:41:52.817 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:50.954885+0000", 2024-09-11T03:41:52.817 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 1.7213964900000001, 2024-09-11T03:41:52.817 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012999881, 2024-09-11T03:41:52.817 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:41:52.817 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:41:52.817 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:41:52.817 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:41:52.818 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:41:52.818 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:41:52.818 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2422 2024-09-11T03:41:52.818 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.818 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:41:52.818 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.818 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:41:52.818 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954885+0000", 2024-09-11T03:41:52.818 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.818 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.819 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.819 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:41:52.819 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954885+0000", 2024-09-11T03:41:52.819 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:41:52.819 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.819 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.819 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:41:52.819 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954886+0000", 2024-09-11T03:41:52.819 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.82e-07 2024-09-11T03:41:52.819 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.819 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.820 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:41:52.820 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954888+0000", 2024-09-11T03:41:52.820 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1909999999999999e-06 2024-09-11T03:41:52.820 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.820 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.820 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:41:52.820 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954899+0000", 2024-09-11T03:41:52.820 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0963e-05 2024-09-11T03:41:52.820 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.820 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.820 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:41:52.820 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954904+0000", 2024-09-11T03:41:52.821 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3539999999999999e-06 2024-09-11T03:41:52.821 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.821 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.821 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.821 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.955109+0000", 2024-09-11T03:41:52.821 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000204494 2024-09-11T03:41:52.821 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.821 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.821 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:41:52.821 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.955158+0000", 2024-09-11T03:41:52.821 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9502000000000001e-05 2024-09-11T03:41:52.822 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.822 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.822 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:41:52.822 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.967733+0000", 2024-09-11T03:41:52.822 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012574571999999999 2024-09-11T03:41:52.822 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.822 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.822 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:41:52.822 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.967803+0000", 2024-09-11T03:41:52.822 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0160999999999995e-05 2024-09-11T03:41:52.822 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:41:52.822 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:41:52.823 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:41:52.823 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.967885+0000", 2024-09-11T03:41:52.823 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2162000000000004e-05 2024-09-11T03:41:52.823 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.823 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.823 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.823 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:41:52.823 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:41:52.823 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:41:52.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:52.825 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:52.849 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:52.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:52.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:52.859 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:41:52.866 INFO:tasks.rados.rados.2.smithi175.stdout:965: done (1 left) 2024-09-11T03:41:52.866 INFO:tasks.rados.rados.2.smithi175.stdout:966: write oid 31 current snap is 101 2024-09-11T03:41:52.866 INFO:tasks.rados.rados.2.smithi175.stdout:963: finishing copy_from racing read to smithi17557870-28 2024-09-11T03:41:52.866 INFO:tasks.rados.rados.2.smithi175.stdout:966: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-09-11T03:41:52.872 INFO:tasks.rados.rados.2.smithi175.stdout:966: writing smithi17557870-31 from 716064 to 1355950 tid 1 2024-09-11T03:41:52.877 INFO:tasks.rados.rados.2.smithi175.stdout:966: writing smithi17557870-31 from 1815191 to 2315640 tid 2 2024-09-11T03:41:52.883 INFO:tasks.rados.rados.2.smithi175.stdout:966: writing smithi17557870-31 from 2817011 to 3343936 tid 3 2024-09-11T03:41:52.883 INFO:tasks.rados.rados.2.smithi175.stdout:967: copy_from oid 43 from oid 10 current snap is 101 2024-09-11T03:41:52.883 INFO:tasks.rados.rados.2.smithi175.stdout:963: finishing copy_from to smithi17557870-28 2024-09-11T03:41:52.883 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 343 (ObjNum 342 snap 99 seq_num 342) dirty exists 2024-09-11T03:41:52.883 INFO:tasks.rados.rados.2.smithi175.stdout:966: finishing write tid 1 to smithi17557870-31 2024-09-11T03:41:52.883 INFO:tasks.rados.rados.2.smithi175.stdout:966: finishing write tid 2 to smithi17557870-31 2024-09-11T03:41:52.883 INFO:tasks.rados.rados.2.smithi175.stdout:963: done (2 left) 2024-09-11T03:41:52.884 INFO:tasks.rados.rados.2.smithi175.stdout:968: snap_remove snap 95 2024-09-11T03:41:52.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:52.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:52.977 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:52.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:53.078 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:53.102 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:53.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:53.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:53.204 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:53.226 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:53.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:53.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:53.328 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:53.350 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:53.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:53.351 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:53.452 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:53.474 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:53.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:53.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:53.576 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:53.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:53.600 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:53.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:53.664 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-11T03:41:53.701 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:53.722 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:53.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:53.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:53.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:53.850 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:53.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:53.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:53.865 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T03:41:53.869 INFO:tasks.rados.rados.2.smithi175.stdout:968: done (2 left) 2024-09-11T03:41:53.869 INFO:tasks.rados.rados.2.smithi175.stdout:969: snap_create 2024-09-11T03:41:53.870 INFO:tasks.rados.rados.2.smithi175.stdout:966: finishing write tid 3 to smithi17557870-31 2024-09-11T03:41:53.870 INFO:tasks.rados.rados.2.smithi175.stdout:966: finishing write tid 4 to smithi17557870-31 2024-09-11T03:41:53.870 INFO:tasks.rados.rados.2.smithi175.stdout:966: finishing write tid 5 to smithi17557870-31 2024-09-11T03:41:53.870 INFO:tasks.rados.rados.2.smithi175.stdout:966: oid 31 updating version 0 to 342 2024-09-11T03:41:53.870 INFO:tasks.rados.rados.2.smithi175.stdout:966: oid 31 version 342 is already newer than 340 2024-09-11T03:41:53.870 INFO:tasks.rados.rados.2.smithi175.stdout:966: oid 31 version 342 is already newer than 339 2024-09-11T03:41:53.871 INFO:tasks.rados.rados.2.smithi175.stdout:966: oid 31 version 342 is already newer than 341 2024-09-11T03:41:53.871 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 342 (ObjNum 352 snap 101 seq_num 352) dirty exists 2024-09-11T03:41:53.871 INFO:tasks.rados.rados.2.smithi175.stdout:966: left oid 31 (ObjNum 352 snap 101 seq_num 352) 2024-09-11T03:41:53.871 INFO:tasks.rados.rados.2.smithi175.stdout:967: finishing copy_from racing read to smithi17557870-43 2024-09-11T03:41:53.871 INFO:tasks.rados.rados.2.smithi175.stdout:967: finishing copy_from to smithi17557870-43 2024-09-11T03:41:53.871 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 342 (ObjNum 310 snap 85 seq_num 310) dirty exists 2024-09-11T03:41:53.906 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:41:53.952 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:53.974 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:53.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:53.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:53 smithi175 ceph-mon[30894]: pgmap v598: 57 pgs: 57 active+clean; 173 MiB data, 684 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-11T03:41:53.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:53 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1224807698' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:53.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:53 smithi175 ceph-mon[30894]: osdmap e296: 12 total, 12 up, 10 in 2024-09-11T03:41:53.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:53 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3604261293' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:41:54.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:54.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:54.101 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:54.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:53 smithi084 ceph-mon[27254]: pgmap v598: 57 pgs: 57 active+clean; 173 MiB data, 684 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-11T03:41:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:53 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1224807698' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:53 smithi084 ceph-mon[27254]: osdmap e296: 12 total, 12 up, 10 in 2024-09-11T03:41:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:53 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3604261293' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:41:54.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:53 smithi121 ceph-mon[30729]: pgmap v598: 57 pgs: 57 active+clean; 173 MiB data, 684 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-11T03:41:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:53 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1224807698' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:41:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:53 smithi121 ceph-mon[30729]: osdmap e296: 12 total, 12 up, 10 in 2024-09-11T03:41:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:53 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3604261293' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:41:54.203 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:54.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:54.226 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:54.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:54.326 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:54.347 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:54.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:54.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:54.389 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:41:54.389 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:41:54.389 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:41:54.389 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:41:54.449 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:54.461 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:54.471 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:54.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:54.472 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:54.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-11T03:41:54.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:54.598 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:54.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:54.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:54.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:54.725 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:54.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:54.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:54.827 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:54.850 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:54.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:54.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:54.874 INFO:tasks.rados.rados.2.smithi175.stdout:966: done (2 left) 2024-09-11T03:41:54.874 INFO:tasks.rados.rados.2.smithi175.stdout:967: done (1 left) 2024-09-11T03:41:54.874 INFO:tasks.rados.rados.2.smithi175.stdout:969: done (0 left) 2024-09-11T03:41:54.874 INFO:tasks.rados.rados.2.smithi175.stdout:970: read oid 31 snap -1 2024-09-11T03:41:54.874 INFO:tasks.rados.rados.2.smithi175.stdout:970: expect (ObjNum 352 snap 101 seq_num 352) 2024-09-11T03:41:54.874 INFO:tasks.rados.rados.2.smithi175.stdout:971: read oid 20 snap -1 2024-09-11T03:41:54.874 INFO:tasks.rados.rados.2.smithi175.stdout:971: expect (ObjNum 332 snap 92 seq_num 332) 2024-09-11T03:41:54.874 INFO:tasks.rados.rados.2.smithi175.stdout:972: snap_create 2024-09-11T03:41:54.882 INFO:tasks.rados.rados.2.smithi175.stdout:971: expect (ObjNum 332 snap 92 seq_num 332) 2024-09-11T03:41:54.938 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:41:54.938 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:41:54.941 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:54.952 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:54.955 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:41:54.956 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:41:54.972 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:54.973 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:41:54.974 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:41:54.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:54.974 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:54 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3604261293' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:41:54.974 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:54 smithi084 ceph-mon[27254]: osdmap e297: 12 total, 12 up, 10 in 2024-09-11T03:41:54.975 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:54.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:54 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3604261293' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:41:54.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:54 smithi175 ceph-mon[30894]: osdmap e297: 12 total, 12 up, 10 in 2024-09-11T03:41:54.989 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:41:54.989 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:41:55.006 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:41:55.006 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:41:55.023 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:41:55.023 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:41:55.040 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:41:55.041 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:41:55.055 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-11T03:41:55.057 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:41:55.057 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:41:55.073 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:41:55.074 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:41:55.076 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:41:55.090 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:41:55.090 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:41:55.107 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:41:55.107 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:41:55.111 INFO:tasks.rados.rados.2.smithi175.stdout:970: expect (ObjNum 352 snap 101 seq_num 352) 2024-09-11T03:41:55.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:55.117 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:55.117 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:41:55.124 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:41:55.124 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:41:55.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:54 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3604261293' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:41:55.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:54 smithi121 ceph-mon[30729]: osdmap e297: 12 total, 12 up, 10 in 2024-09-11T03:41:55.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:55.243 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:55.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:55.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:55.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:55.352 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:41:55.367 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:55.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:55.469 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:41:55.490 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:55.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:41:55.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:55.618 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:55.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:55.720 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:55.742 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:55.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:55.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:55.796 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:41:55.796 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:41:55.844 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:55.865 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:55.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:55.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:55.879 INFO:tasks.rados.rados.2.smithi175.stdout:970: done (2 left) 2024-09-11T03:41:55.879 INFO:tasks.rados.rados.2.smithi175.stdout:971: done (1 left) 2024-09-11T03:41:55.880 INFO:tasks.rados.rados.2.smithi175.stdout:972: done (0 left) 2024-09-11T03:41:55.880 INFO:tasks.rados.rados.2.smithi175.stdout:973: read oid 21 snap -1 2024-09-11T03:41:55.880 INFO:tasks.rados.rados.2.smithi175.stdout:973: expect (ObjNum 266 snap 67 seq_num 266) 2024-09-11T03:41:55.880 INFO:tasks.rados.rados.2.smithi175.stdout:974: snap_create 2024-09-11T03:41:55.888 INFO:tasks.rados.rados.2.smithi175.stdout:973: expect (ObjNum 266 snap 67 seq_num 266) 2024-09-11T03:41:55.906 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:41:55.966 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:55.987 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:55.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:55.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:56.089 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:56.111 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:56.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:56.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:56.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:55 smithi084 ceph-mon[27254]: pgmap v601: 57 pgs: 57 active+clean; 186 MiB data, 700 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 24 op/s 2024-09-11T03:41:56.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:55 smithi084 ceph-mon[27254]: osdmap e298: 12 total, 12 up, 10 in 2024-09-11T03:41:56.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:55 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3271667550' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:56.165 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:56.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:55 smithi121 ceph-mon[30729]: pgmap v601: 57 pgs: 57 active+clean; 186 MiB data, 700 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 24 op/s 2024-09-11T03:41:56.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:55 smithi121 ceph-mon[30729]: osdmap e298: 12 total, 12 up, 10 in 2024-09-11T03:41:56.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:55 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3271667550' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:56.213 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:55 smithi175 ceph-mon[30894]: pgmap v601: 57 pgs: 57 active+clean; 186 MiB data, 700 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 24 op/s 2024-09-11T03:41:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:55 smithi175 ceph-mon[30894]: osdmap e298: 12 total, 12 up, 10 in 2024-09-11T03:41:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:55 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3271667550' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:56.239 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:56.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:56.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:56.341 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:56.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:56.364 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:56.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:56.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:56.490 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:56.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:56.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:56.529 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-11T03:41:56.592 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:56.613 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:56.614 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:56.715 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:41:56.741 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:56.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:56.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:41:56.773 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:41:56.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:56.859 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:41:56.859 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T03:41:56.864 INFO:tasks.rados.rados.2.smithi175.stdout:973: done (1 left) 2024-09-11T03:41:56.864 INFO:tasks.rados.rados.2.smithi175.stdout:974: done (0 left) 2024-09-11T03:41:56.864 INFO:tasks.rados.rados.2.smithi175.stdout:975: write (excl) oid 17 current snap is 104 2024-09-11T03:41:56.864 INFO:tasks.rados.rados.2.smithi175.stdout:975: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2024-09-11T03:41:56.869 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:56.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:56.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:56.872 INFO:tasks.rados.rados.2.smithi175.stdout:975: writing smithi17557870-17 from 732871 to 1448006 tid 1 2024-09-11T03:41:56.878 INFO:tasks.rados.rados.2.smithi175.stdout:975: writing smithi17557870-17 from 1873673 to 2517780 tid 2 2024-09-11T03:41:56.886 INFO:tasks.rados.rados.2.smithi175.stdout:975: writing smithi17557870-17 from 3244434 to 3932871 tid 3 2024-09-11T03:41:56.886 INFO:tasks.rados.rados.2.smithi175.stdout:976: write (excl) oid 16 current snap is 104 2024-09-11T03:41:56.886 INFO:tasks.rados.rados.2.smithi175.stdout:975: finishing write tid 1 to smithi17557870-17 2024-09-11T03:41:56.887 INFO:tasks.rados.rados.2.smithi175.stdout:975: finishing write tid 2 to smithi17557870-17 2024-09-11T03:41:56.887 INFO:tasks.rados.rados.2.smithi175.stdout:976: seq_num 354 ranges {749678=790384,2332155=504118,3271857=549469} 2024-09-11T03:41:56.894 INFO:tasks.rados.rados.2.smithi175.stdout:976: writing smithi17557870-16 from 749678 to 1540062 tid 1 2024-09-11T03:41:56.901 INFO:tasks.rados.rados.2.smithi175.stdout:976: writing smithi17557870-16 from 2332155 to 2836273 tid 2 2024-09-11T03:41:56.907 INFO:tasks.rados.rados.2.smithi175.stdout:976: writing smithi17557870-16 from 3271857 to 3821326 tid 3 2024-09-11T03:41:56.907 INFO:tasks.rados.rados.2.smithi175.stdout:977: delete oid 3 current snap is 104 2024-09-11T03:41:56.907 INFO:tasks.rados.rados.2.smithi175.stdout:975: finishing write tid 3 to smithi17557870-17 2024-09-11T03:41:56.907 INFO:tasks.rados.rados.2.smithi175.stdout:975: finishing write tid 4 to smithi17557870-17 2024-09-11T03:41:56.907 INFO:tasks.rados.rados.2.smithi175.stdout:975: finishing write tid 5 to smithi17557870-17 2024-09-11T03:41:56.907 INFO:tasks.rados.rados.2.smithi175.stdout:975: oid 17 updating version 0 to 355 2024-09-11T03:41:56.907 INFO:tasks.rados.rados.2.smithi175.stdout:975: oid 17 version 355 is already newer than 354 2024-09-11T03:41:56.907 INFO:tasks.rados.rados.2.smithi175.stdout:975: oid 17 version 355 is already newer than 352 2024-09-11T03:41:56.908 INFO:tasks.rados.rados.2.smithi175.stdout:975: oid 17 version 355 is already newer than 353 2024-09-11T03:41:56.908 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 355 (ObjNum 353 snap 104 seq_num 353) dirty exists 2024-09-11T03:41:56.908 INFO:tasks.rados.rados.2.smithi175.stdout:975: left oid 17 (ObjNum 353 snap 104 seq_num 353) 2024-09-11T03:41:56.908 INFO:tasks.rados.rados.2.smithi175.stdout:976: finishing write tid 1 to smithi17557870-16 2024-09-11T03:41:56.908 INFO:tasks.rados.rados.2.smithi175.stdout:976: finishing write tid 2 to smithi17557870-16 2024-09-11T03:41:56.908 INFO:tasks.rados.rados.2.smithi175.stdout:975: done (2 left) 2024-09-11T03:41:56.909 INFO:tasks.rados.rados.2.smithi175.stdout:977: done (1 left) 2024-09-11T03:41:56.909 INFO:tasks.rados.rados.2.smithi175.stdout:978: write (excl) oid 35 current snap is 104 2024-09-11T03:41:56.909 INFO:tasks.rados.rados.2.smithi175.stdout:978: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2024-09-11T03:41:56.915 INFO:tasks.rados.rados.2.smithi175.stdout:978: writing smithi17557870-35 from 783292 to 1324174 tid 1 2024-09-11T03:41:56.922 INFO:tasks.rados.rados.2.smithi175.stdout:978: writing smithi17557870-35 from 1932766 to 2440553 tid 2 2024-09-11T03:41:56.930 INFO:tasks.rados.rados.2.smithi175.stdout:978: writing smithi17557870-35 from 3210350 to 3782363 tid 3 2024-09-11T03:41:56.930 INFO:tasks.rados.rados.2.smithi175.stdout:979: read oid 40 snap -1 2024-09-11T03:41:56.930 INFO:tasks.rados.rados.2.smithi175.stdout:979: expect (ObjNum 262 snap 66 seq_num 262) 2024-09-11T03:41:56.930 INFO:tasks.rados.rados.2.smithi175.stdout:980: rollback oid 26 current snap is 104 2024-09-11T03:41:56.930 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 26 to 86 2024-09-11T03:41:56.930 INFO:tasks.rados.rados.2.smithi175.stdout:976: finishing write tid 3 to smithi17557870-16 2024-09-11T03:41:56.930 INFO:tasks.rados.rados.2.smithi175.stdout:976: finishing write tid 4 to smithi17557870-16 2024-09-11T03:41:56.930 INFO:tasks.rados.rados.2.smithi175.stdout:976: finishing write tid 5 to smithi17557870-16 2024-09-11T03:41:56.930 INFO:tasks.rados.rados.2.smithi175.stdout:976: oid 16 updating version 0 to 164 2024-09-11T03:41:56.930 INFO:tasks.rados.rados.2.smithi175.stdout:976: oid 16 updating version 164 to 165 2024-09-11T03:41:56.931 INFO:tasks.rados.rados.2.smithi175.stdout:976: oid 16 version 165 is already newer than 163 2024-09-11T03:41:56.931 INFO:tasks.rados.rados.2.smithi175.stdout:976: oid 16 version 165 is already newer than 162 2024-09-11T03:41:56.931 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 165 (ObjNum 354 snap 104 seq_num 354) dirty exists 2024-09-11T03:41:56.931 INFO:tasks.rados.rados.2.smithi175.stdout:976: left oid 16 (ObjNum 354 snap 104 seq_num 354) 2024-09-11T03:41:56.931 INFO:tasks.rados.rados.2.smithi175.stdout:978: finishing write tid 1 to smithi17557870-35 2024-09-11T03:41:56.931 INFO:tasks.rados.rados.2.smithi175.stdout:978: finishing write tid 2 to smithi17557870-35 2024-09-11T03:41:56.931 INFO:tasks.rados.rados.2.smithi175.stdout:976: done (3 left) 2024-09-11T03:41:56.931 INFO:tasks.rados.rados.2.smithi175.stdout:981: snap_create 2024-09-11T03:41:56.933 INFO:tasks.rados.rados.2.smithi175.stdout:980: finishing rollback tid 0 to smithi17557870-26 2024-09-11T03:41:56.933 INFO:tasks.rados.rados.2.smithi175.stdout:980: finishing rollback tid 1 to smithi17557870-26 2024-09-11T03:41:56.933 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 165 (ObjNum 129 snap 0 seq_num 1745317232) dirty dne 2024-09-11T03:41:56.933 INFO:tasks.rados.rados.2.smithi175.stdout:978: finishing write tid 3 to smithi17557870-35 2024-09-11T03:41:56.933 INFO:tasks.rados.rados.2.smithi175.stdout:978: finishing write tid 4 to smithi17557870-35 2024-09-11T03:41:56.933 INFO:tasks.rados.rados.2.smithi175.stdout:978: finishing write tid 5 to smithi17557870-35 2024-09-11T03:41:56.933 INFO:tasks.rados.rados.2.smithi175.stdout:978: oid 35 updating version 0 to 251 2024-09-11T03:41:56.934 INFO:tasks.rados.rados.2.smithi175.stdout:978: oid 35 version 251 is already newer than 250 2024-09-11T03:41:56.934 INFO:tasks.rados.rados.2.smithi175.stdout:978: oid 35 version 251 is already newer than 249 2024-09-11T03:41:56.934 INFO:tasks.rados.rados.2.smithi175.stdout:978: oid 35 version 251 is already newer than 248 2024-09-11T03:41:56.934 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 251 (ObjNum 356 snap 104 seq_num 356) dirty exists 2024-09-11T03:41:56.934 INFO:tasks.rados.rados.2.smithi175.stdout:978: left oid 35 (ObjNum 356 snap 104 seq_num 356) 2024-09-11T03:41:56.941 INFO:tasks.rados.rados.2.smithi175.stdout:979: expect (ObjNum 262 snap 66 seq_num 262) 2024-09-11T03:41:56.972 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:41:56.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:56.998 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:56.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:41:57.100 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:57.121 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:57.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:57.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:57.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:56 smithi084 ceph-mon[27254]: osdmap e299: 12 total, 12 up, 10 in 2024-09-11T03:41:57.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/996277240' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:41:57.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2817083682' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:41:57.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:56 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:57.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/996277240' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:41:57.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2817083682' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:41:57.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:56 smithi084 ceph-mon[27254]: osdmap e300: 12 total, 12 up, 10 in 2024-09-11T03:41:57.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:56 smithi121 ceph-mon[30729]: osdmap e299: 12 total, 12 up, 10 in 2024-09-11T03:41:57.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:56 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/996277240' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:41:57.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:56 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2817083682' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:41:57.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:56 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:57.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:56 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/996277240' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:41:57.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:56 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2817083682' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:41:57.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:56 smithi121 ceph-mon[30729]: osdmap e300: 12 total, 12 up, 10 in 2024-09-11T03:41:57.187 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:56 smithi175 ceph-mon[30894]: osdmap e299: 12 total, 12 up, 10 in 2024-09-11T03:41:57.187 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:56 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/996277240' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:41:57.187 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:56 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2817083682' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:41:57.188 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:56 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:41:57.188 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:56 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/996277240' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:41:57.188 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:56 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2817083682' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:41:57.188 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:56 smithi175 ceph-mon[30894]: osdmap e300: 12 total, 12 up, 10 in 2024-09-11T03:41:57.223 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:57.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:57.247 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:57.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:57.285 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:41:57.285 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:41:57.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:57.373 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:57.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:57.475 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:57.495 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-11T03:41:57.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:57.498 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:57.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:57.599 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:57.621 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:57.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:57.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:57.722 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:57.744 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:57.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:57.785 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:41:57.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:57.864 INFO:tasks.rados.rados.2.smithi175.stdout:978: done (3 left) 2024-09-11T03:41:57.864 INFO:tasks.rados.rados.2.smithi175.stdout:979: done (2 left) 2024-09-11T03:41:57.864 INFO:tasks.rados.rados.2.smithi175.stdout:980: done (1 left) 2024-09-11T03:41:57.865 INFO:tasks.rados.rados.2.smithi175.stdout:981: done (0 left) 2024-09-11T03:41:57.865 INFO:tasks.rados.rados.2.smithi175.stdout:982: read oid 36 snap -1 2024-09-11T03:41:57.865 INFO:tasks.rados.rados.2.smithi175.stdout:982: expect (ObjNum 317 snap 86 seq_num 317) 2024-09-11T03:41:57.865 INFO:tasks.rados.rados.2.smithi175.stdout:983: write oid 40 current snap is 105 2024-09-11T03:41:57.865 INFO:tasks.rados.rados.2.smithi175.stdout:983: seq_num 357 ranges {400099=616131,1591248=408851} 2024-09-11T03:41:57.870 INFO:tasks.rados.rados.2.smithi175.stdout:983: writing smithi17557870-40 from 400099 to 1016230 tid 1 2024-09-11T03:41:57.872 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:57.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:57.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:57.875 INFO:tasks.rados.rados.2.smithi175.stdout:983: writing smithi17557870-40 from 1591248 to 2000099 tid 2 2024-09-11T03:41:57.875 INFO:tasks.rados.rados.2.smithi175.stdout:984: write (excl) oid 30 current snap is 105 2024-09-11T03:41:57.875 INFO:tasks.rados.rados.2.smithi175.stdout:984: seq_num 358 ranges {416906=407733,1366083=511456} 2024-09-11T03:41:57.880 INFO:tasks.rados.rados.2.smithi175.stdout:984: writing smithi17557870-30 from 416906 to 824639 tid 1 2024-09-11T03:41:57.886 INFO:tasks.rados.rados.2.smithi175.stdout:984: writing smithi17557870-30 from 1366083 to 1877539 tid 2 2024-09-11T03:41:57.886 INFO:tasks.rados.rados.2.smithi175.stdout:985: snap_create 2024-09-11T03:41:57.886 INFO:tasks.rados.rados.2.smithi175.stdout:982: expect (ObjNum 317 snap 86 seq_num 317) 2024-09-11T03:41:57.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:57.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:57 smithi175 ceph-mon[30894]: pgmap v604: 57 pgs: 57 active+clean; 189 MiB data, 713 MiB used, 893 GiB / 894 GiB avail; 2.9 MiB/s rd, 3.9 MiB/s wr, 8 op/s 2024-09-11T03:41:57.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:57 smithi175 ceph-mon[30894]: 1.3 scrub starts 2024-09-11T03:41:57.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:57 smithi175 ceph-mon[30894]: 1.3 scrub ok 2024-09-11T03:41:57.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:57 smithi175 ceph-mon[30894]: osdmap e301: 12 total, 12 up, 10 in 2024-09-11T03:41:58.000 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:58.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:58.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:58.081 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-11T03:41:58.090 INFO:tasks.rados.rados.2.smithi175.stdout:983: finishing write tid 1 to smithi17557870-40 2024-09-11T03:41:58.091 INFO:tasks.rados.rados.2.smithi175.stdout:983: finishing write tid 2 to smithi17557870-40 2024-09-11T03:41:58.091 INFO:tasks.rados.rados.2.smithi175.stdout:983: finishing write tid 3 to smithi17557870-40 2024-09-11T03:41:58.091 INFO:tasks.rados.rados.2.smithi175.stdout:983: finishing write tid 4 to smithi17557870-40 2024-09-11T03:41:58.091 INFO:tasks.rados.rados.2.smithi175.stdout:983: oid 40 updating version 0 to 343 2024-09-11T03:41:58.091 INFO:tasks.rados.rados.2.smithi175.stdout:983: oid 40 updating version 343 to 344 2024-09-11T03:41:58.091 INFO:tasks.rados.rados.2.smithi175.stdout:983: oid 40 updating version 344 to 345 2024-09-11T03:41:58.091 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 345 (ObjNum 357 snap 105 seq_num 357) dirty exists 2024-09-11T03:41:58.091 INFO:tasks.rados.rados.2.smithi175.stdout:983: left oid 40 (ObjNum 357 snap 105 seq_num 357) 2024-09-11T03:41:58.091 INFO:tasks.rados.rados.2.smithi175.stdout:984: finishing write tid 1 to smithi17557870-30 2024-09-11T03:41:58.091 INFO:tasks.rados.rados.2.smithi175.stdout:984: finishing write tid 2 to smithi17557870-30 2024-09-11T03:41:58.091 INFO:tasks.rados.rados.2.smithi175.stdout:984: finishing write tid 3 to smithi17557870-30 2024-09-11T03:41:58.091 INFO:tasks.rados.rados.2.smithi175.stdout:984: finishing write tid 4 to smithi17557870-30 2024-09-11T03:41:58.091 INFO:tasks.rados.rados.2.smithi175.stdout:984: oid 30 updating version 0 to 356 2024-09-11T03:41:58.092 INFO:tasks.rados.rados.2.smithi175.stdout:984: oid 30 updating version 356 to 357 2024-09-11T03:41:58.092 INFO:tasks.rados.rados.2.smithi175.stdout:984: oid 30 updating version 357 to 358 2024-09-11T03:41:58.092 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 358 (ObjNum 358 snap 105 seq_num 358) dirty exists 2024-09-11T03:41:58.092 INFO:tasks.rados.rados.2.smithi175.stdout:984: left oid 30 (ObjNum 358 snap 105 seq_num 358) 2024-09-11T03:41:58.101 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:58.123 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:58.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:58.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:57 smithi084 ceph-mon[27254]: pgmap v604: 57 pgs: 57 active+clean; 189 MiB data, 713 MiB used, 893 GiB / 894 GiB avail; 2.9 MiB/s rd, 3.9 MiB/s wr, 8 op/s 2024-09-11T03:41:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:57 smithi084 ceph-mon[27254]: 1.3 scrub starts 2024-09-11T03:41:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:57 smithi084 ceph-mon[27254]: 1.3 scrub ok 2024-09-11T03:41:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:57 smithi084 ceph-mon[27254]: osdmap e301: 12 total, 12 up, 10 in 2024-09-11T03:41:58.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:57 smithi121 ceph-mon[30729]: pgmap v604: 57 pgs: 57 active+clean; 189 MiB data, 713 MiB used, 893 GiB / 894 GiB avail; 2.9 MiB/s rd, 3.9 MiB/s wr, 8 op/s 2024-09-11T03:41:58.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:57 smithi121 ceph-mon[30729]: 1.3 scrub starts 2024-09-11T03:41:58.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:57 smithi121 ceph-mon[30729]: 1.3 scrub ok 2024-09-11T03:41:58.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:57 smithi121 ceph-mon[30729]: osdmap e301: 12 total, 12 up, 10 in 2024-09-11T03:41:58.226 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:41:58.247 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:58.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:58.248 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:41:58.262 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:58.323 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:41:58.349 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:41:58.370 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:58.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:41:58.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:58.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:58.501 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:58.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:58.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:58.627 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:58.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:58.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:41:58.744 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:41:58.746 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:41:58.746 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:41:58.755 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:58.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:58.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:41:58.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:41:58.764 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:41:58.781 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:41:58.782 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:41:58.798 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:41:58.798 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:41:58.816 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:41:58.816 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:41:58.821 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:41:58.821 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:41:58.833 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:41:58.833 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:41:58.851 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:41:58.851 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:41:58.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:58.868 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:41:58.868 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:41:58.877 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:58.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:58.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:58.886 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:41:58.886 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:41:58.891 INFO:tasks.rados.rados.2.smithi175.stdout:982: done (3 left) 2024-09-11T03:41:58.891 INFO:tasks.rados.rados.2.smithi175.stdout:983: done (2 left) 2024-09-11T03:41:58.892 INFO:tasks.rados.rados.2.smithi175.stdout:984: done (1 left) 2024-09-11T03:41:58.892 INFO:tasks.rados.rados.2.smithi175.stdout:985: done (0 left) 2024-09-11T03:41:58.892 INFO:tasks.rados.rados.2.smithi175.stdout:986: copy_from oid 1 from oid 2 current snap is 106 2024-09-11T03:41:58.892 INFO:tasks.rados.rados.2.smithi175.stdout:987: rollback oid 18 current snap is 106 2024-09-11T03:41:58.892 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 18 to 101 2024-09-11T03:41:58.892 INFO:tasks.rados.rados.2.smithi175.stdout:988: rollback oid 10 current snap is 106 2024-09-11T03:41:58.892 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 10 to 101 2024-09-11T03:41:58.892 INFO:tasks.rados.rados.2.smithi175.stdout:989: snap_create 2024-09-11T03:41:58.895 INFO:tasks.rados.rados.2.smithi175.stdout:988: finishing rollback tid 0 to smithi17557870-10 2024-09-11T03:41:58.895 INFO:tasks.rados.rados.2.smithi175.stdout:988: finishing rollback tid 1 to smithi17557870-10 2024-09-11T03:41:58.895 INFO:tasks.rados.rados.2.smithi175.stdout:987: finishing rollback tid 0 to smithi17557870-18 2024-09-11T03:41:58.895 INFO:tasks.rados.rados.2.smithi175.stdout:988: finishing rollback tid 2 to smithi17557870-10 2024-09-11T03:41:58.896 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 359 (ObjNum 310 snap 85 seq_num 310) dirty exists 2024-09-11T03:41:58.896 INFO:tasks.rados.rados.2.smithi175.stdout:987: finishing rollback tid 1 to smithi17557870-18 2024-09-11T03:41:58.896 INFO:tasks.rados.rados.2.smithi175.stdout:987: finishing rollback tid 2 to smithi17557870-18 2024-09-11T03:41:58.896 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 196 (ObjNum 261 snap 66 seq_num 261) dirty exists 2024-09-11T03:41:58.904 INFO:tasks.rados.rados.2.smithi175.stdout:986: finishing copy_from racing read to smithi17557870-1 2024-09-11T03:41:58.910 INFO:tasks.rados.rados.2.smithi175.stdout:986: finishing copy_from to smithi17557870-1 2024-09-11T03:41:58.910 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 279 (ObjNum 314 snap 86 seq_num 314) dirty exists 2024-09-11T03:41:58.920 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:41:58.920 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:41:58.941 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:41:58.941 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:41:58.958 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:41:58.958 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:41:58.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:58 smithi175 ceph-mon[30894]: 1.17 scrub starts 2024-09-11T03:41:58.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:58 smithi175 ceph-mon[30894]: 1.17 scrub ok 2024-09-11T03:41:58.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:58 smithi175 ceph-mon[30894]: 1.7 scrub starts 2024-09-11T03:41:58.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:58 smithi175 ceph-mon[30894]: 1.7 scrub ok 2024-09-11T03:41:58.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:58 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2637813108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:58.980 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:58.991 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:58 smithi084 ceph-mon[27254]: 1.17 scrub starts 2024-09-11T03:41:58.991 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:58 smithi084 ceph-mon[27254]: 1.17 scrub ok 2024-09-11T03:41:58.991 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:58 smithi084 ceph-mon[27254]: 1.7 scrub starts 2024-09-11T03:41:58.991 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:58 smithi084 ceph-mon[27254]: 1.7 scrub ok 2024-09-11T03:41:58.991 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:58 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2637813108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:59.005 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:59.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:59.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:59.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:41:59.133 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:41:59.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:59.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:41:59.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:58 smithi121 ceph-mon[30729]: 1.17 scrub starts 2024-09-11T03:41:59.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:58 smithi121 ceph-mon[30729]: 1.17 scrub ok 2024-09-11T03:41:59.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:58 smithi121 ceph-mon[30729]: 1.7 scrub starts 2024-09-11T03:41:59.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:58 smithi121 ceph-mon[30729]: 1.7 scrub ok 2024-09-11T03:41:59.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:58 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2637813108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:41:59.235 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:59.258 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:59.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:59.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:59.360 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:59.383 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:59.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:59.384 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:59.485 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:41:59.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:59.509 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:59.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:41:59.512 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-11T03:41:59.610 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:41:59.632 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:59.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:59.633 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:41:59.734 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:41:59.755 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:41:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:59.756 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:41:59.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:41:59.858 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:41:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:41:59.880 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:41:59.880 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:41:59.889 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 1 (10000) 2024-09-11T03:41:59.893 INFO:tasks.rados.rados.2.smithi175.stdout:986: done (3 left) 2024-09-11T03:41:59.893 INFO:tasks.rados.rados.2.smithi175.stdout:987: done (2 left) 2024-09-11T03:41:59.893 INFO:tasks.rados.rados.2.smithi175.stdout:988: done (1 left) 2024-09-11T03:41:59.893 INFO:tasks.rados.rados.2.smithi175.stdout:989: done (0 left) 2024-09-11T03:41:59.893 INFO:tasks.rados.rados.2.smithi175.stdout:990: delete oid 8 current snap is 107 2024-09-11T03:41:59.902 INFO:tasks.rados.rados.2.smithi175.stdout:990: done (0 left) 2024-09-11T03:41:59.902 INFO:tasks.rados.rados.2.smithi175.stdout:991: rollback oid 39 current snap is 107 2024-09-11T03:41:59.903 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 39 to 106 2024-09-11T03:41:59.903 INFO:tasks.rados.rados.2.smithi175.stdout:992: write (excl) oid 13 current snap is 107 2024-09-11T03:41:59.903 INFO:tasks.rados.rados.2.smithi175.stdout:992: seq_num 360 ranges {450520=558231,1483047=515125} 2024-09-11T03:41:59.908 INFO:tasks.rados.rados.2.smithi175.stdout:992: writing smithi17557870-13 from 450520 to 1008751 tid 1 2024-09-11T03:41:59.913 INFO:tasks.rados.rados.2.smithi175.stdout:992: writing smithi17557870-13 from 1483047 to 1998172 tid 2 2024-09-11T03:41:59.913 INFO:tasks.rados.rados.2.smithi175.stdout:993: snap_remove snap 101 2024-09-11T03:41:59.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:00.003 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:00.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:00.105 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:00.127 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:00.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:00.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:00.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:59 smithi084 ceph-mon[27254]: pgmap v607: 57 pgs: 57 active+clean; 195 MiB data, 722 MiB used, 893 GiB / 894 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 16 op/s 2024-09-11T03:42:00.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:59 smithi084 ceph-mon[27254]: osdmap e302: 12 total, 12 up, 10 in 2024-09-11T03:42:00.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:41:59 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2059693981' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:42:00.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:59 smithi121 ceph-mon[30729]: pgmap v607: 57 pgs: 57 active+clean; 195 MiB data, 722 MiB used, 893 GiB / 894 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 16 op/s 2024-09-11T03:42:00.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:59 smithi121 ceph-mon[30729]: osdmap e302: 12 total, 12 up, 10 in 2024-09-11T03:42:00.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:41:59 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2059693981' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:42:00.194 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:42:00.195 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:59 smithi175 ceph-mon[30894]: pgmap v607: 57 pgs: 57 active+clean; 195 MiB data, 722 MiB used, 893 GiB / 894 GiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 16 op/s 2024-09-11T03:42:00.195 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:59 smithi175 ceph-mon[30894]: osdmap e302: 12 total, 12 up, 10 in 2024-09-11T03:42:00.195 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:41:59 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2059693981' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:42:00.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:00.250 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:00.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:00.303 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:42:00.303 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:42:00.304 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:42:00.304 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:42:00.304 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:42:00.352 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:00.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:00.375 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:00.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:00.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:00.500 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:00.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:00.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:00.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:00.625 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:00.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:00.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:00.727 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:00.750 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:00.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:00.751 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:00.764 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-11T03:42:00.852 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:00.875 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:00.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:00.882 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:00.911 INFO:tasks.rados.rados.2.smithi175.stdout:993: done (2 left) 2024-09-11T03:42:00.911 INFO:tasks.rados.rados.2.smithi175.stdout:994: snap_remove snap 98 2024-09-11T03:42:00.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:01.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:01.000 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:01.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:01.003 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:00 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2059693981' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:42:01.003 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:00 smithi084 ceph-mon[27254]: osdmap e303: 12 total, 12 up, 10 in 2024-09-11T03:42:01.003 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:42:01.003 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:42:01.003 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:42:01.003 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:42:01.004 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:01.004 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:00 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:01.101 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:01.123 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-11T03:42:01.124 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:01.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:01.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:01.148 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:00 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2059693981' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:42:01.148 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:00 smithi175 ceph-mon[30894]: osdmap e303: 12 total, 12 up, 10 in 2024-09-11T03:42:01.148 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:42:01.148 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:42:01.148 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:42:01.148 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:42:01.148 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:01.148 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:00 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:00 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2059693981' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:42:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:00 smithi121 ceph-mon[30729]: osdmap e303: 12 total, 12 up, 10 in 2024-09-11T03:42:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:42:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:42:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:42:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:42:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:00 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:01.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:01.250 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:01.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:01.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:01.352 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:01.373 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:42:01.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:01.374 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:01.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:01.475 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:01.486 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:01.496 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:01.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:01.497 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:01.598 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:01.621 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:01.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:01.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:01.723 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:01.744 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:01.745 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:01.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:01.861 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:42:01.866 INFO:tasks.rados.rados.2.smithi175.stdout:994: done (2 left) 2024-09-11T03:42:01.866 INFO:tasks.rados.rados.2.smithi175.stdout:995: copy_from oid 47 from oid 21 current snap is 107 2024-09-11T03:42:01.866 INFO:tasks.rados.rados.2.smithi175.stdout:991: finishing rollback tid 0 to smithi17557870-39 2024-09-11T03:42:01.866 INFO:tasks.rados.rados.2.smithi175.stdout:991: finishing rollback tid 1 to smithi17557870-39 2024-09-11T03:42:01.866 INFO:tasks.rados.rados.2.smithi175.stdout:991: finishing rollback tid 2 to smithi17557870-39 2024-09-11T03:42:01.866 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 289 (ObjNum 262 snap 66 seq_num 262) dirty exists 2024-09-11T03:42:01.866 INFO:tasks.rados.rados.2.smithi175.stdout:992: finishing write tid 1 to smithi17557870-13 2024-09-11T03:42:01.866 INFO:tasks.rados.rados.2.smithi175.stdout:992: finishing write tid 2 to smithi17557870-13 2024-09-11T03:42:01.866 INFO:tasks.rados.rados.2.smithi175.stdout:992: finishing write tid 3 to smithi17557870-13 2024-09-11T03:42:01.866 INFO:tasks.rados.rados.2.smithi175.stdout:992: finishing write tid 4 to smithi17557870-13 2024-09-11T03:42:01.866 INFO:tasks.rados.rados.2.smithi175.stdout:992: oid 13 updating version 0 to 334 2024-09-11T03:42:01.866 INFO:tasks.rados.rados.2.smithi175.stdout:992: oid 13 version 334 is already newer than 333 2024-09-11T03:42:01.867 INFO:tasks.rados.rados.2.smithi175.stdout:992: oid 13 updating version 334 to 335 2024-09-11T03:42:01.867 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 335 (ObjNum 360 snap 107 seq_num 360) dirty exists 2024-09-11T03:42:01.867 INFO:tasks.rados.rados.2.smithi175.stdout:992: left oid 13 (ObjNum 360 snap 107 seq_num 360) 2024-09-11T03:42:01.867 INFO:tasks.rados.rados.2.smithi175.stdout:991: done (2 left) 2024-09-11T03:42:01.867 INFO:tasks.rados.rados.2.smithi175.stdout:992: done (1 left) 2024-09-11T03:42:01.867 INFO:tasks.rados.rados.2.smithi175.stdout:996: write oid 6 current snap is 107 2024-09-11T03:42:01.867 INFO:tasks.rados.rados.2.smithi175.stdout:996: seq_num 361 ranges {467327=633480,1825176=242151} 2024-09-11T03:42:01.873 INFO:tasks.rados.rados.2.smithi175.stdout:996: writing smithi17557870-6 from 467327 to 1100807 tid 1 2024-09-11T03:42:01.874 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:01.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:01.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:01.878 INFO:tasks.rados.rados.2.smithi175.stdout:996: writing smithi17557870-6 from 1825176 to 2067327 tid 2 2024-09-11T03:42:01.879 INFO:tasks.rados.rados.2.smithi175.stdout:997: write (excl) oid 31 current snap is 107 2024-09-11T03:42:01.879 INFO:tasks.rados.rados.2.smithi175.stdout:997: seq_num 362 ranges {484134=708729,1883658=200476} 2024-09-11T03:42:01.885 INFO:tasks.rados.rados.2.smithi175.stdout:997: writing smithi17557870-31 from 484134 to 1192863 tid 1 2024-09-11T03:42:01.890 INFO:tasks.rados.rados.2.smithi175.stdout:997: writing smithi17557870-31 from 1883658 to 2084134 tid 2 2024-09-11T03:42:01.890 INFO:tasks.rados.rados.2.smithi175.stdout:998: snap_remove snap 103 2024-09-11T03:42:01.891 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:42:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:42:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:42:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:42:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:365 2.c 2:327469ff:::smithi17557870-49:head [read 0~2806736,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:42:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.850028+0000", 2024-09-11T03:42:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 190.97055069499999, 2024-09-11T03:42:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0077480070000000003, 2024-09-11T03:42:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:42:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:42:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:42:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:42:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 365 2024-09-11T03:42:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850028+0000", 2024-09-11T03:42:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850028+0000", 2024-09-11T03:42:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850029+0000", 2024-09-11T03:42:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1200000000000003e-07 2024-09-11T03:42:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850033+0000", 2024-09-11T03:42:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4789999999999998e-06 2024-09-11T03:42:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850043+0000", 2024-09-11T03:42:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.38e-06 2024-09-11T03:42:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850056+0000", 2024-09-11T03:42:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3179999999999999e-05 2024-09-11T03:42:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850150+0000", 2024-09-11T03:42:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4549000000000006e-05 2024-09-11T03:42:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:42:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850155+0000", 2024-09-11T03:42:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0440000000000003e-06 2024-09-11T03:42:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.855967+0000", 2024-09-11T03:42:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0058113289999999996 2024-09-11T03:42:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.856051+0000", 2024-09-11T03:42:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3840999999999999e-05 2024-09-11T03:42:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.857776+0000", 2024-09-11T03:42:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0017255930000000001 2024-09-11T03:42:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:640 2.a 2:5014e305:::smithi17557870-46:head [copy-from ver 23] snapc 11=[e] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:42:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:42:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 173.710285707, 2024-09-11T03:42:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019221736, 2024-09-11T03:42:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:42:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:42:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:42:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 640 2024-09-11T03:42:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:42:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:42:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110295+0000", 2024-09-11T03:42:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0609999999999998e-06 2024-09-11T03:42:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110305+0000", 2024-09-11T03:42:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4169999999999998e-06 2024-09-11T03:42:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110321+0000", 2024-09-11T03:42:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6288000000000001e-05 2024-09-11T03:42:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110328+0000", 2024-09-11T03:42:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3950000000000003e-06 2024-09-11T03:42:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110396+0000", 2024-09-11T03:42:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7164999999999998e-05 2024-09-11T03:42:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110507+0000", 2024-09-11T03:42:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000111217 2024-09-11T03:42:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:42:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.118213+0000", 2024-09-11T03:42:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0077066249999999999 2024-09-11T03:42:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:42:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125103+0000", 2024-09-11T03:42:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068894359999999997 2024-09-11T03:42:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:42:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125459+0000", 2024-09-11T03:42:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000356146 2024-09-11T03:42:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:42:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129369+0000", 2024-09-11T03:42:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039099249999999999 2024-09-11T03:42:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:42:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129428+0000", 2024-09-11T03:42:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9113000000000001e-05 2024-09-11T03:42:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129515+0000", 2024-09-11T03:42:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6947999999999995e-05 2024-09-11T03:42:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:641 2.a 2:5014e305:::smithi17557870-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:42:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:42:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 173.710234316, 2024-09-11T03:42:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011882876000000001, 2024-09-11T03:42:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:42:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:42:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:42:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:42:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 641 2024-09-11T03:42:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:42:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:42:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:42:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7599999999999998e-07 2024-09-11T03:42:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110347+0000", 2024-09-11T03:42:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.685e-06 2024-09-11T03:42:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110351+0000", 2024-09-11T03:42:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4889999999999997e-06 2024-09-11T03:42:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110354+0000", 2024-09-11T03:42:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0510000000000001e-06 2024-09-11T03:42:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110644+0000", 2024-09-11T03:42:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00028940200000000002 2024-09-11T03:42:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:42:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110673+0000", 2024-09-11T03:42:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9833e-05 2024-09-11T03:42:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122038+0000", 2024-09-11T03:42:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011364915 2024-09-11T03:42:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122127+0000", 2024-09-11T03:42:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8987999999999996e-05 2024-09-11T03:42:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122227+0000", 2024-09-11T03:42:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010013699999999999 2024-09-11T03:42:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:752 2.c 2:3c007163:::smithi17557870-35:head [copy-from ver 39] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:42:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:42:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 166.81156303399999, 2024-09-11T03:42:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017596783000000001, 2024-09-11T03:42:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:42:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:42:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:42:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 752 2024-09-11T03:42:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:42:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:42:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009018+0000", 2024-09-11T03:42:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.48e-06 2024-09-11T03:42:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009029+0000", 2024-09-11T03:42:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0377000000000001e-05 2024-09-11T03:42:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009050+0000", 2024-09-11T03:42:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0849e-05 2024-09-11T03:42:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009058+0000", 2024-09-11T03:42:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8019999999999998e-06 2024-09-11T03:42:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009125+0000", 2024-09-11T03:42:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6388000000000002e-05 2024-09-11T03:42:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:42:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009130+0000", 2024-09-11T03:42:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0900000000000004e-06 2024-09-11T03:42:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010518+0000", 2024-09-11T03:42:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013880030000000001 2024-09-11T03:42:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010582+0000", 2024-09-11T03:42:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.368e-05 2024-09-11T03:42:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:42:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.015350+0000", 2024-09-11T03:42:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0047682870000000004 2024-09-11T03:42:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:42:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.019005+0000", 2024-09-11T03:42:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0036549040000000001 2024-09-11T03:42:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:42:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.024096+0000", 2024-09-11T03:42:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050917439999999996 2024-09-11T03:42:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:42:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026521+0000", 2024-09-11T03:42:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024247399999999999 2024-09-11T03:42:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:42:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026559+0000", 2024-09-11T03:42:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8232999999999998e-05 2024-09-11T03:42:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026613+0000", 2024-09-11T03:42:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3205999999999998e-05 2024-09-11T03:42:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:753 2.c 2:3c007163:::smithi17557870-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:42:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:42:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 166.811495947, 2024-09-11T03:42:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0094619260000000007, 2024-09-11T03:42:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:42:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:42:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:42:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:42:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 753 2024-09-11T03:42:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:42:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:42:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:42:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2100000000000002e-07 2024-09-11T03:42:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009085+0000", 2024-09-11T03:42:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5910000000000001e-06 2024-09-11T03:42:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009090+0000", 2024-09-11T03:42:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5180000000000004e-06 2024-09-11T03:42:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009094+0000", 2024-09-11T03:42:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0620000000000001e-06 2024-09-11T03:42:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009178+0000", 2024-09-11T03:42:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4525999999999997e-05 2024-09-11T03:42:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:42:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009182+0000", 2024-09-11T03:42:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4640000000000002e-06 2024-09-11T03:42:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010677+0000", 2024-09-11T03:42:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014957569999999999 2024-09-11T03:42:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:42:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010706+0000", 2024-09-11T03:42:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8873999999999998e-05 2024-09-11T03:42:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018385+0000", 2024-09-11T03:42:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0076790490000000003 2024-09-11T03:42:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018466+0000", 2024-09-11T03:42:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0483999999999997e-05 2024-09-11T03:42:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018545+0000", 2024-09-11T03:42:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9179999999999997e-05 2024-09-11T03:42:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:985 2.b e163/95 2:dde92fc6:::smithi17557870-41:head v 163'52, mlcod=158'48)", 2024-09-11T03:42:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:42:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 141.88762840199999, 2024-09-11T03:42:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0092288890000000005, 2024-09-11T03:42:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:42:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:42:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:42:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7799999999999997e-07 2024-09-11T03:42:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935287+0000", 2024-09-11T03:42:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023361599999999999 2024-09-11T03:42:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935321+0000", 2024-09-11T03:42:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3840000000000001e-05 2024-09-11T03:42:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935335+0000", 2024-09-11T03:42:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4184e-05 2024-09-11T03:42:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935428+0000", 2024-09-11T03:42:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3144000000000001e-05 2024-09-11T03:42:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935503+0000", 2024-09-11T03:42:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4697999999999995e-05 2024-09-11T03:42:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:42:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.942117+0000", 2024-09-11T03:42:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0066141350000000002 2024-09-11T03:42:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.942179+0000", 2024-09-11T03:42:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.245e-05 2024-09-11T03:42:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:990 2.a 2:5014e305:::smithi17557870-46:head [write 638699~541153] snapc 24=[24,23,22,20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-11T03:42:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:42:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 141.88364715, 2024-09-11T03:42:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0080995760000000007, 2024-09-11T03:42:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:42:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:42:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:42:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 990 2024-09-11T03:42:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:42:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:42:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.931 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:01.931 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936935+0000", 2024-09-11T03:42:01.931 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7800000000000001e-06 2024-09-11T03:42:01.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.931 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:01.931 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937090+0000", 2024-09-11T03:42:01.931 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000155857 2024-09-11T03:42:01.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.931 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937109+0000", 2024-09-11T03:42:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8216e-05 2024-09-11T03:42:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937119+0000", 2024-09-11T03:42:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0406e-05 2024-09-11T03:42:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937195+0000", 2024-09-11T03:42:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6372999999999995e-05 2024-09-11T03:42:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937287+0000", 2024-09-11T03:42:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1434000000000006e-05 2024-09-11T03:42:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:42:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937454+0000", 2024-09-11T03:42:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000167262 2024-09-11T03:42:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:42:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.943162+0000", 2024-09-11T03:42:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0057082230000000001 2024-09-11T03:42:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:42:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.943196+0000", 2024-09-11T03:42:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4025e-05 2024-09-11T03:42:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:42:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.944922+0000", 2024-09-11T03:42:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001726102 2024-09-11T03:42:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:42:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.944977+0000", 2024-09-11T03:42:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4849999999999998e-05 2024-09-11T03:42:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.945031+0000", 2024-09-11T03:42:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4048000000000002e-05 2024-09-11T03:42:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1175 2.c 2:32ade24e:::smithi17557870-45:head [copy-from ver 130] snapc 2e=[2e,2c,2b,28,26] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:42:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:42:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 119.718439258, 2024-09-11T03:42:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015807878000000001, 2024-09-11T03:42:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:42:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:42:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:42:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1175 2024-09-11T03:42:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:42:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:42:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102142+0000", 2024-09-11T03:42:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3369999999999998e-06 2024-09-11T03:42:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102152+0000", 2024-09-11T03:42:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8400000000000007e-06 2024-09-11T03:42:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102172+0000", 2024-09-11T03:42:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0537e-05 2024-09-11T03:42:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:01.939 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102183+0000", 2024-09-11T03:42:01.939 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0423e-05 2024-09-11T03:42:01.939 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.939 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.939 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102253+0000", 2024-09-11T03:42:01.939 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0528e-05 2024-09-11T03:42:01.939 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.939 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:42:01.939 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102259+0000", 2024-09-11T03:42:01.939 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1589999999999998e-06 2024-09-11T03:42:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103477+0000", 2024-09-11T03:42:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012174550000000001 2024-09-11T03:42:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103545+0000", 2024-09-11T03:42:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7595999999999995e-05 2024-09-11T03:42:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:42:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.110323+0000", 2024-09-11T03:42:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0067785730000000004 2024-09-11T03:42:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:42:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.113411+0000", 2024-09-11T03:42:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030882769999999999 2024-09-11T03:42:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:42:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117742+0000", 2024-09-11T03:42:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0043301920000000001 2024-09-11T03:42:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:42:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117835+0000", 2024-09-11T03:42:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3848e-05 2024-09-11T03:42:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:42:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117916+0000", 2024-09-11T03:42:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0505000000000003e-05 2024-09-11T03:42:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117948+0000", 2024-09-11T03:42:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1608e-05 2024-09-11T03:42:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1176 2.c 2:32ade24e:::smithi17557870-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:42:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:42:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 119.718374994, 2024-09-11T03:42:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010847406, 2024-09-11T03:42:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:42:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:42:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:42:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:42:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1176 2024-09-11T03:42:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:42:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:42:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:42:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4999999999999998e-07 2024-09-11T03:42:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102206+0000", 2024-09-11T03:42:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9429999999999999e-06 2024-09-11T03:42:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102214+0000", 2024-09-11T03:42:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0399999999999993e-06 2024-09-11T03:42:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102220+0000", 2024-09-11T03:42:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2989999999999996e-06 2024-09-11T03:42:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102307+0000", 2024-09-11T03:42:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7635999999999999e-05 2024-09-11T03:42:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:42:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102310+0000", 2024-09-11T03:42:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.2160000000000002e-06 2024-09-11T03:42:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103661+0000", 2024-09-11T03:42:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013507860000000001 2024-09-11T03:42:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:42:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103686+0000", 2024-09-11T03:42:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.44e-05 2024-09-11T03:42:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.112884+0000", 2024-09-11T03:42:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0091986110000000006 2024-09-11T03:42:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.112967+0000", 2024-09-11T03:42:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2478000000000006e-05 2024-09-11T03:42:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.113051+0000", 2024-09-11T03:42:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4647000000000005e-05 2024-09-11T03:42:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1280 2.c 2:3c007163:::smithi17557870-35:head [rollback 2b] snapc 33=[32,30,2c,2b] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:42:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:12.835951+0000", 2024-09-11T03:42:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 108.98462778299999, 2024-09-11T03:42:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0065671339999999996, 2024-09-11T03:42:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:42:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:42:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:42:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1280 2024-09-11T03:42:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835951+0000", 2024-09-11T03:42:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835951+0000", 2024-09-11T03:42:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835952+0000", 2024-09-11T03:42:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4300000000000003e-07 2024-09-11T03:42:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835955+0000", 2024-09-11T03:42:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5099999999999999e-06 2024-09-11T03:42:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835969+0000", 2024-09-11T03:42:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.36e-05 2024-09-11T03:42:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.835975+0000", 2024-09-11T03:42:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3230000000000003e-06 2024-09-11T03:42:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836209+0000", 2024-09-11T03:42:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00023352 2024-09-11T03:42:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:42:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836215+0000", 2024-09-11T03:42:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5799999999999997e-06 2024-09-11T03:42:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839733+0000", 2024-09-11T03:42:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035178369999999998 2024-09-11T03:42:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839818+0000", 2024-09-11T03:42:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4962000000000004e-05 2024-09-11T03:42:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:42:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.840012+0000", 2024-09-11T03:42:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000193962 2024-09-11T03:42:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:42:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.840806+0000", 2024-09-11T03:42:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00079423099999999995 2024-09-11T03:42:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:42:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.841732+0000", 2024-09-11T03:42:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00092522800000000005 2024-09-11T03:42:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:42:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.842435+0000", 2024-09-11T03:42:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00070363399999999997 2024-09-11T03:42:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:42:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.842485+0000", 2024-09-11T03:42:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9580999999999998e-05 2024-09-11T03:42:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.842518+0000", 2024-09-11T03:42:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3522999999999997e-05 2024-09-11T03:42:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1281 2.c 2:3c007163:::smithi17557870-35:head [append 0~0] snapc 33=[32,30,2c,2b] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:42:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:12.836002+0000", 2024-09-11T03:42:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 108.984576698, 2024-09-11T03:42:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0067871590000000001, 2024-09-11T03:42:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:42:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:42:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:42:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1281 2024-09-11T03:42:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836002+0000", 2024-09-11T03:42:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836002+0000", 2024-09-11T03:42:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836003+0000", 2024-09-11T03:42:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3e-07 2024-09-11T03:42:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836021+0000", 2024-09-11T03:42:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8142999999999999e-05 2024-09-11T03:42:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836075+0000", 2024-09-11T03:42:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4398999999999999e-05 2024-09-11T03:42:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836087+0000", 2024-09-11T03:42:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.166e-05 2024-09-11T03:42:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836268+0000", 2024-09-11T03:42:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00018073399999999999 2024-09-11T03:42:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:42:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836275+0000", 2024-09-11T03:42:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9639999999999997e-06 2024-09-11T03:42:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.840402+0000", 2024-09-11T03:42:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0041267789999999997 2024-09-11T03:42:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.840480+0000", 2024-09-11T03:42:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8801000000000007e-05 2024-09-11T03:42:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:42:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.840597+0000", 2024-09-11T03:42:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000116137 2024-09-11T03:42:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:42:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.840948+0000", 2024-09-11T03:42:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00035135599999999999 2024-09-11T03:42:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:42:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.841775+0000", 2024-09-11T03:42:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00082670300000000003 2024-09-11T03:42:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:42:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.842655+0000", 2024-09-11T03:42:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00088039100000000001 2024-09-11T03:42:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:42:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.842741+0000", 2024-09-11T03:42:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5958000000000006e-05 2024-09-11T03:42:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.842789+0000", 2024-09-11T03:42:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8403999999999999e-05 2024-09-11T03:42:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1719 2.c 2:327469ff:::smithi17557870-49:head [write 487469~563838,stat] snapc 45=[45,44,43,3e,3d,3c,3a] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-11T03:42:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:45.859743+0000", 2024-09-11T03:42:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 75.960836216000004, 2024-09-11T03:42:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0078899869999999993, 2024-09-11T03:42:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:42:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:42:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:42:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1719 2024-09-11T03:42:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.859743+0000", 2024-09-11T03:42:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.859743+0000", 2024-09-11T03:42:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.859745+0000", 2024-09-11T03:42:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.712e-06 2024-09-11T03:42:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860066+0000", 2024-09-11T03:42:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00032013499999999998 2024-09-11T03:42:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860090+0000", 2024-09-11T03:42:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4590999999999999e-05 2024-09-11T03:42:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860102+0000", 2024-09-11T03:42:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2269999999999999e-05 2024-09-11T03:42:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860167+0000", 2024-09-11T03:42:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4806000000000003e-05 2024-09-11T03:42:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:42:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860172+0000", 2024-09-11T03:42:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7219999999999999e-06 2024-09-11T03:42:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.862974+0000", 2024-09-11T03:42:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0028017480000000002 2024-09-11T03:42:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.863071+0000", 2024-09-11T03:42:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7738000000000005e-05 2024-09-11T03:42:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:42:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.863253+0000", 2024-09-11T03:42:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00018170999999999999 2024-09-11T03:42:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:42:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.864451+0000", 2024-09-11T03:42:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00119782 2024-09-11T03:42:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:42:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.865698+0000", 2024-09-11T03:42:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001246973 2024-09-11T03:42:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:42:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.867563+0000", 2024-09-11T03:42:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001865189 2024-09-11T03:42:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:42:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.867601+0000", 2024-09-11T03:42:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8333e-05 2024-09-11T03:42:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.867633+0000", 2024-09-11T03:42:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1239999999999999e-05 2024-09-11T03:42:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1863 2.c 2:3c007163:::smithi17557870-35:head [read 0~3613505,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:42:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:01.850063+0000", 2024-09-11T03:42:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 59.970516136000001, 2024-09-11T03:42:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0078079589999999997, 2024-09-11T03:42:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:42:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:42:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:42:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:42:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1863 2024-09-11T03:42:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850063+0000", 2024-09-11T03:42:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850063+0000", 2024-09-11T03:42:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850066+0000", 2024-09-11T03:42:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8700000000000001e-06 2024-09-11T03:42:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850074+0000", 2024-09-11T03:42:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3380000000000005e-06 2024-09-11T03:42:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850096+0000", 2024-09-11T03:42:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1753e-05 2024-09-11T03:42:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850108+0000", 2024-09-11T03:42:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2300000000000001e-05 2024-09-11T03:42:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850235+0000", 2024-09-11T03:42:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000126604 2024-09-11T03:42:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:42:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850244+0000", 2024-09-11T03:42:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.0790000000000007e-06 2024-09-11T03:42:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.852514+0000", 2024-09-11T03:42:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022705199999999998 2024-09-11T03:42:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.852621+0000", 2024-09-11T03:42:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000106649 2024-09-11T03:42:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.857871+0000", 2024-09-11T03:42:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0052498459999999999 2024-09-11T03:42:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1864 2.c 2:3c007163:::smithi17557870-35:head [sparse-read 0~1995320] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:42:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:42:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 59.970445660000003, 2024-09-11T03:42:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0079536510000000008, 2024-09-11T03:42:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:42:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:42:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:42:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:42:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1864 2024-09-11T03:42:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:42:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:42:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850134+0000", 2024-09-11T03:42:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2399999999999998e-07 2024-09-11T03:42:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850137+0000", 2024-09-11T03:42:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4010000000000001e-06 2024-09-11T03:42:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850164+0000", 2024-09-11T03:42:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6616999999999999e-05 2024-09-11T03:42:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850171+0000", 2024-09-11T03:42:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7209999999999997e-06 2024-09-11T03:42:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850311+0000", 2024-09-11T03:42:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00014029900000000001 2024-09-11T03:42:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:42:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850317+0000", 2024-09-11T03:42:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7799999999999997e-06 2024-09-11T03:42:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.857902+0000", 2024-09-11T03:42:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0075851759999999999 2024-09-11T03:42:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.857978+0000", 2024-09-11T03:42:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6272000000000004e-05 2024-09-11T03:42:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.858087+0000", 2024-09-11T03:42:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108761 2024-09-11T03:42:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1865 2.c 2:3c007163:::smithi17557870-35:head [sparse-read 0~1995320] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:42:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:01.850224+0000", 2024-09-11T03:42:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 59.970355302999998, 2024-09-11T03:42:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.008147797, 2024-09-11T03:42:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:42:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:42:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:42:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:42:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1865 2024-09-11T03:42:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850224+0000", 2024-09-11T03:42:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850224+0000", 2024-09-11T03:42:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850224+0000", 2024-09-11T03:42:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6099999999999999e-07 2024-09-11T03:42:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850229+0000", 2024-09-11T03:42:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5449999999999997e-06 2024-09-11T03:42:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850244+0000", 2024-09-11T03:42:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5048999999999999e-05 2024-09-11T03:42:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850258+0000", 2024-09-11T03:42:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3505e-05 2024-09-11T03:42:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850344+0000", 2024-09-11T03:42:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6329999999999995e-05 2024-09-11T03:42:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:42:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850349+0000", 2024-09-11T03:42:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0610000000000003e-06 2024-09-11T03:42:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.858124+0000", 2024-09-11T03:42:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.007774792 2024-09-11T03:42:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.858236+0000", 2024-09-11T03:42:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011264000000000001 2024-09-11T03:42:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.858371+0000", 2024-09-11T03:42:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013501400000000001 2024-09-11T03:42:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:1868 2.0 e245/188 2:0762776a:::smithi17557870-25:head v 245'39, mlcod=242'35)", 2024-09-11T03:42:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:01.863124+0000", 2024-09-11T03:42:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 59.957455357000001, 2024-09-11T03:42:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0073720210000000003, 2024-09-11T03:42:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.863124+0000", 2024-09-11T03:42:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.863124+0000", 2024-09-11T03:42:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.863124+0000", 2024-09-11T03:42:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.91e-07 2024-09-11T03:42:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.868459+0000", 2024-09-11T03:42:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0053354500000000003 2024-09-11T03:42:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.868473+0000", 2024-09-11T03:42:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3930999999999999e-05 2024-09-11T03:42:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.868481+0000", 2024-09-11T03:42:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9030000000000004e-06 2024-09-11T03:42:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.868545+0000", 2024-09-11T03:42:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3975999999999999e-05 2024-09-11T03:42:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.868565+0000", 2024-09-11T03:42:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9551e-05 2024-09-11T03:42:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:42:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.870455+0000", 2024-09-11T03:42:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001890624 2024-09-11T03:42:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.870496+0000", 2024-09-11T03:42:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0295e-05 2024-09-11T03:42:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:1989 2.0 e258/188 2:0762776a:::smithi17557870-25:head v 258'49, mlcod=252'45)", 2024-09-11T03:42:01.993 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:14.957402+0000", 2024-09-11T03:42:01.993 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 46.863176639999999, 2024-09-11T03:42:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0064273830000000001, 2024-09-11T03:42:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:14.957402+0000", 2024-09-11T03:42:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:14.957402+0000", 2024-09-11T03:42:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:14.957402+0000", 2024-09-11T03:42:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.92e-07 2024-09-11T03:42:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:14.961652+0000", 2024-09-11T03:42:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0042496540000000003 2024-09-11T03:42:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:14.961670+0000", 2024-09-11T03:42:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7873999999999999e-05 2024-09-11T03:42:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:14.961678+0000", 2024-09-11T03:42:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1510000000000004e-06 2024-09-11T03:42:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:14.961746+0000", 2024-09-11T03:42:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7726999999999997e-05 2024-09-11T03:42:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:14.961765+0000", 2024-09-11T03:42:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9380000000000001e-05 2024-09-11T03:42:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:42:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:14.963772+0000", 2024-09-11T03:42:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0020067119999999999 2024-09-11T03:42:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:14.963830+0000", 2024-09-11T03:42:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7692999999999998e-05 2024-09-11T03:42:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2140 2.c 2:3c007163:::smithi17557870-35:head [copy-from ver 101] snapc 57=[57,56,51,4e] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-11T03:42:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:25.064188+0000", 2024-09-11T03:42:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 36.756390836999998, 2024-09-11T03:42:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013777315, 2024-09-11T03:42:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:42:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:42:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:42:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2140 2024-09-11T03:42:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064188+0000", 2024-09-11T03:42:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064188+0000", 2024-09-11T03:42:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064190+0000", 2024-09-11T03:42:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9700000000000002e-06 2024-09-11T03:42:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064198+0000", 2024-09-11T03:42:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3610000000000001e-06 2024-09-11T03:42:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064216+0000", 2024-09-11T03:42:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7768999999999999e-05 2024-09-11T03:42:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064225+0000", 2024-09-11T03:42:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7099999999999996e-06 2024-09-11T03:42:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064291+0000", 2024-09-11T03:42:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5634000000000002e-05 2024-09-11T03:42:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064367+0000", 2024-09-11T03:42:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6687999999999995e-05 2024-09-11T03:42:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:42:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.066720+0000", 2024-09-11T03:42:02.005 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023526010000000002 2024-09-11T03:42:02.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.005 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.005 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:42:02.005 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.069243+0000", 2024-09-11T03:42:02.005 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025227320000000002 2024-09-11T03:42:02.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.005 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.005 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:42:02.005 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.073940+0000", 2024-09-11T03:42:02.005 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0046973930000000002 2024-09-11T03:42:02.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.005 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.006 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:42:02.006 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.077850+0000", 2024-09-11T03:42:02.006 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039098279999999997 2024-09-11T03:42:02.006 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.006 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.006 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:42:02.006 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.077915+0000", 2024-09-11T03:42:02.006 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5160000000000006e-05 2024-09-11T03:42:02.006 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.006 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.006 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:02.006 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.077965+0000", 2024-09-11T03:42:02.006 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0469000000000001e-05 2024-09-11T03:42:02.007 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:02.007 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:02.007 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:02.007 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.007 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.007 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:2421 2.18 e294/281 2:1bdf1a36:::smithi17557870-36:head v 294'198, mlcod=0'0)", 2024-09-11T03:42:02.007 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:50.964269+0000", 2024-09-11T03:42:02.007 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 10.856309938000001, 2024-09-11T03:42:02.007 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013043020000000001, 2024-09-11T03:42:02.007 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:02.007 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:02.007 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:02.007 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:02.008 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.008 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:02.008 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964269+0000", 2024-09-11T03:42:02.008 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:02.008 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.008 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.008 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:02.008 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964269+0000", 2024-09-11T03:42:02.008 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:02.008 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.008 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.008 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:02.008 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964270+0000", 2024-09-11T03:42:02.009 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3510000000000001e-06 2024-09-11T03:42:02.009 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.009 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.009 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:02.009 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966680+0000", 2024-09-11T03:42:02.009 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002409698 2024-09-11T03:42:02.009 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.009 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.009 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:02.009 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966720+0000", 2024-09-11T03:42:02.009 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0265999999999997e-05 2024-09-11T03:42:02.009 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.009 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.010 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:02.010 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966738+0000", 2024-09-11T03:42:02.010 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7540000000000001e-05 2024-09-11T03:42:02.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.010 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.010 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:02.010 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966835+0000", 2024-09-11T03:42:02.010 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7059999999999996e-05 2024-09-11T03:42:02.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.010 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.010 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:02.010 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966862+0000", 2024-09-11T03:42:02.010 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7107e-05 2024-09-11T03:42:02.011 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.011 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.011 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:42:02.011 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977265+0000", 2024-09-11T03:42:02.011 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.01040258 2024-09-11T03:42:02.011 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.011 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.011 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:02.011 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977312+0000", 2024-09-11T03:42:02.011 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7417999999999999e-05 2024-09-11T03:42:02.011 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:02.011 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:02.011 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:02.012 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.012 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.012 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:2443 2.12 e296/281 2:494ba932:::smithi17557870-43:head v 296'131, mlcod=291'128)", 2024-09-11T03:42:02.012 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:52.895921+0000", 2024-09-11T03:42:02.012 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 8.9246575159999999, 2024-09-11T03:42:02.012 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0084055459999999999, 2024-09-11T03:42:02.012 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:42:02.012 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:42:02.012 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:02.012 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:42:02.012 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.012 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:42:02.012 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.895921+0000", 2024-09-11T03:42:02.012 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:02.013 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.013 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.013 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:42:02.013 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.895921+0000", 2024-09-11T03:42:02.013 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:42:02.013 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.013 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.013 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:42:02.013 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.895922+0000", 2024-09-11T03:42:02.013 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1699999999999998e-07 2024-09-11T03:42:02.013 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.013 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.013 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:42:02.014 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902380+0000", 2024-09-11T03:42:02.014 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0064576629999999998 2024-09-11T03:42:02.014 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.014 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.014 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:42:02.014 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902391+0000", 2024-09-11T03:42:02.014 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1246999999999999e-05 2024-09-11T03:42:02.014 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.014 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.014 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:42:02.014 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902398+0000", 2024-09-11T03:42:02.014 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7560000000000002e-06 2024-09-11T03:42:02.014 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.014 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.015 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:42:02.015 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902482+0000", 2024-09-11T03:42:02.015 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4881000000000003e-05 2024-09-11T03:42:02.015 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.015 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.015 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:42:02.015 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902500+0000", 2024-09-11T03:42:02.015 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7464000000000001e-05 2024-09-11T03:42:02.015 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.015 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.015 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:42:02.015 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.904289+0000", 2024-09-11T03:42:02.015 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0017895599999999999 2024-09-11T03:42:02.016 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:42:02.016 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:42:02.016 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:42:02.016 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.904327+0000", 2024-09-11T03:42:02.016 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7457999999999999e-05 2024-09-11T03:42:02.016 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:02.016 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:02.016 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:02.016 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:42:02.016 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:42:02.016 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:42:02.017 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:02.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:02.040 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:02.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:02.141 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:02.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: 1.4 scrub starts 2024-09-11T03:42:02.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: 1.4 scrub ok 2024-09-11T03:42:02.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: pgmap v610: 57 pgs: 57 active+clean; 196 MiB data, 736 MiB used, 893 GiB / 894 GiB avail; 14 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-11T03:42:02.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: 1.9 scrub starts 2024-09-11T03:42:02.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: 1.9 scrub ok 2024-09-11T03:42:02.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: osdmap e304: 12 total, 12 up, 10 in 2024-09-11T03:42:02.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: 1.1 scrub starts 2024-09-11T03:42:02.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: 1.1 scrub ok 2024-09-11T03:42:02.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:02.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:02.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:42:02.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:42:02.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:02.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2685642136' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:02.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:02.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:42:02.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:42:02.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2685642136' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:02.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:01 smithi084 ceph-mon[27254]: osdmap e305: 12 total, 12 up, 10 in 2024-09-11T03:42:02.157 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:42:02.163 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:02.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:02.164 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:02.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: 1.4 scrub starts 2024-09-11T03:42:02.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: 1.4 scrub ok 2024-09-11T03:42:02.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: pgmap v610: 57 pgs: 57 active+clean; 196 MiB data, 736 MiB used, 893 GiB / 894 GiB avail; 14 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-11T03:42:02.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: 1.9 scrub starts 2024-09-11T03:42:02.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: 1.9 scrub ok 2024-09-11T03:42:02.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: osdmap e304: 12 total, 12 up, 10 in 2024-09-11T03:42:02.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: 1.1 scrub starts 2024-09-11T03:42:02.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: 1.1 scrub ok 2024-09-11T03:42:02.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:02.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:02.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:42:02.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:42:02.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:02.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2685642136' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:02.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:02.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:42:02.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:42:02.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2685642136' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:02.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:01 smithi121 ceph-mon[30729]: osdmap e305: 12 total, 12 up, 10 in 2024-09-11T03:42:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: 1.4 scrub starts 2024-09-11T03:42:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: 1.4 scrub ok 2024-09-11T03:42:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: pgmap v610: 57 pgs: 57 active+clean; 196 MiB data, 736 MiB used, 893 GiB / 894 GiB avail; 14 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-09-11T03:42:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: 1.9 scrub starts 2024-09-11T03:42:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: 1.9 scrub ok 2024-09-11T03:42:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: osdmap e304: 12 total, 12 up, 10 in 2024-09-11T03:42:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: 1.1 scrub starts 2024-09-11T03:42:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: 1.1 scrub ok 2024-09-11T03:42:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:42:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:42:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2685642136' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:42:02.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:42:02.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2685642136' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:02.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:01 smithi175 ceph-mon[30894]: osdmap e305: 12 total, 12 up, 10 in 2024-09-11T03:42:02.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:02.291 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:02.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:02.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:02.393 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:02.418 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:02.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:02.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:02.520 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:02.542 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:02.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:02.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:02.630 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:02.645 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:02.666 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:02.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:02.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:02.768 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:02.789 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:02.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:02.863 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T03:42:02.868 INFO:tasks.rados.rados.2.smithi175.stdout:998: done (3 left) 2024-09-11T03:42:02.868 INFO:tasks.rados.rados.2.smithi175.stdout:999: read oid 42 snap -1 2024-09-11T03:42:02.868 INFO:tasks.rados.rados.2.smithi175.stdout:999: expect (ObjNum 331 snap 92 seq_num 331) 2024-09-11T03:42:02.868 INFO:tasks.rados.rados.2.smithi175.stdout:995: finishing copy_from racing read to smithi17557870-47 2024-09-11T03:42:02.868 INFO:tasks.rados.rados.2.smithi175.stdout:996: finishing write tid 1 to smithi17557870-6 2024-09-11T03:42:02.868 INFO:tasks.rados.rados.2.smithi175.stdout:996: finishing write tid 2 to smithi17557870-6 2024-09-11T03:42:02.868 INFO:tasks.rados.rados.2.smithi175.stdout:996: finishing write tid 3 to smithi17557870-6 2024-09-11T03:42:02.868 INFO:tasks.rados.rados.2.smithi175.stdout:996: finishing write tid 4 to smithi17557870-6 2024-09-11T03:42:02.868 INFO:tasks.rados.rados.2.smithi175.stdout:996: oid 6 updating version 0 to 335 2024-09-11T03:42:02.868 INFO:tasks.rados.rados.2.smithi175.stdout:996: oid 6 updating version 335 to 336 2024-09-11T03:42:02.868 INFO:tasks.rados.rados.2.smithi175.stdout:996: oid 6 updating version 336 to 337 2024-09-11T03:42:02.869 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 337 (ObjNum 361 snap 107 seq_num 361) dirty exists 2024-09-11T03:42:02.869 INFO:tasks.rados.rados.2.smithi175.stdout:996: left oid 6 (ObjNum 361 snap 107 seq_num 361) 2024-09-11T03:42:02.869 INFO:tasks.rados.rados.2.smithi175.stdout:995: finishing copy_from to smithi17557870-47 2024-09-11T03:42:02.869 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 327 (ObjNum 266 snap 67 seq_num 266) dirty exists 2024-09-11T03:42:02.869 INFO:tasks.rados.rados.2.smithi175.stdout:997: finishing write tid 1 to smithi17557870-31 2024-09-11T03:42:02.869 INFO:tasks.rados.rados.2.smithi175.stdout:997: finishing write tid 2 to smithi17557870-31 2024-09-11T03:42:02.869 INFO:tasks.rados.rados.2.smithi175.stdout:997: finishing write tid 3 to smithi17557870-31 2024-09-11T03:42:02.869 INFO:tasks.rados.rados.2.smithi175.stdout:997: finishing write tid 4 to smithi17557870-31 2024-09-11T03:42:02.869 INFO:tasks.rados.rados.2.smithi175.stdout:997: oid 31 updating version 0 to 349 2024-09-11T03:42:02.869 INFO:tasks.rados.rados.2.smithi175.stdout:997: oid 31 updating version 349 to 350 2024-09-11T03:42:02.869 INFO:tasks.rados.rados.2.smithi175.stdout:997: oid 31 version 350 is already newer than 348 2024-09-11T03:42:02.869 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 350 (ObjNum 362 snap 107 seq_num 362) dirty exists 2024-09-11T03:42:02.869 INFO:tasks.rados.rados.2.smithi175.stdout:997: left oid 31 (ObjNum 362 snap 107 seq_num 362) 2024-09-11T03:42:02.869 INFO:tasks.rados.rados.2.smithi175.stdout:995: done (3 left) 2024-09-11T03:42:02.870 INFO:tasks.rados.rados.2.smithi175.stdout:996: done (2 left) 2024-09-11T03:42:02.870 INFO:tasks.rados.rados.2.smithi175.stdout:997: done (1 left) 2024-09-11T03:42:02.870 INFO:tasks.rados.rados.2.smithi175.stdout:1000: read oid 1 snap 91 2024-09-11T03:42:02.870 INFO:tasks.rados.rados.2.smithi175.stdout:1000: expect (ObjNum 120 snap 18 seq_num 120) 2024-09-11T03:42:02.870 INFO:tasks.rados.rados.2.smithi175.stdout:1001: write (excl) oid 20 current snap is 107 2024-09-11T03:42:02.870 INFO:tasks.rados.rados.2.smithi175.stdout:1001: seq_num 363 ranges {500941=783978,1942140=158801} 2024-09-11T03:42:02.875 INFO:tasks.rados.rados.2.smithi175.stdout:1001: writing smithi17557870-20 from 500941 to 1284919 tid 1 2024-09-11T03:42:02.880 INFO:tasks.rados.rados.2.smithi175.stdout:1001: writing smithi17557870-20 from 1942140 to 2100941 tid 2 2024-09-11T03:42:02.880 INFO:tasks.rados.rados.2.smithi175.stdout:1002: rollback oid 30 current snap is 107 2024-09-11T03:42:02.880 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 30 to 105 2024-09-11T03:42:02.880 INFO:tasks.rados.rados.2.smithi175.stdout:1000: expect (ObjNum 120 snap 18 seq_num 120) 2024-09-11T03:42:02.891 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:02.915 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:02.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:02.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:02.966 INFO:tasks.rados.rados.2.smithi175.stdout:999: expect (ObjNum 331 snap 92 seq_num 331) 2024-09-11T03:42:03.017 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:03.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:03.039 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:03.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:03.054 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T03:42:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1001: finishing write tid 1 to smithi17557870-20 2024-09-11T03:42:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1001: finishing write tid 2 to smithi17557870-20 2024-09-11T03:42:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1001: finishing write tid 3 to smithi17557870-20 2024-09-11T03:42:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1002: finishing rollback tid 0 to smithi17557870-30 2024-09-11T03:42:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1001: finishing write tid 4 to smithi17557870-20 2024-09-11T03:42:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1001: oid 20 updating version 0 to 375 2024-09-11T03:42:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1001: oid 20 version 375 is already newer than 374 2024-09-11T03:42:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1001: oid 20 updating version 375 to 376 2024-09-11T03:42:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 376 (ObjNum 363 snap 107 seq_num 363) dirty exists 2024-09-11T03:42:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1001: left oid 20 (ObjNum 363 snap 107 seq_num 363) 2024-09-11T03:42:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1002: finishing rollback tid 1 to smithi17557870-30 2024-09-11T03:42:03.095 INFO:tasks.rados.rados.2.smithi175.stdout:1002: finishing rollback tid 2 to smithi17557870-30 2024-09-11T03:42:03.095 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 372 (ObjNum 358 snap 105 seq_num 358) dirty exists 2024-09-11T03:42:03.095 INFO:tasks.rados.rados.2.smithi175.stdout:999: done (3 left) 2024-09-11T03:42:03.095 INFO:tasks.rados.rados.2.smithi175.stdout:1000: done (2 left) 2024-09-11T03:42:03.095 INFO:tasks.rados.rados.2.smithi175.stdout:1001: done (1 left) 2024-09-11T03:42:03.095 INFO:tasks.rados.rados.2.smithi175.stdout:1002: done (0 left) 2024-09-11T03:42:03.095 INFO:tasks.rados.rados.2.smithi175.stdout:1003: snap_create 2024-09-11T03:42:03.140 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:03.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:02 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4154451665' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:42:03.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:02 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4154451665' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:42:03.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:02 smithi084 ceph-mon[27254]: osdmap e306: 12 total, 12 up, 10 in 2024-09-11T03:42:03.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:03.162 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:03.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:03.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:02 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4154451665' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:42:03.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:02 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4154451665' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:42:03.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:02 smithi121 ceph-mon[30729]: osdmap e306: 12 total, 12 up, 10 in 2024-09-11T03:42:03.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:02 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4154451665' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:42:03.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:02 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4154451665' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:42:03.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:02 smithi175 ceph-mon[30894]: osdmap e306: 12 total, 12 up, 10 in 2024-09-11T03:42:03.263 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:03.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:03.285 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:03.285 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:03.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:03.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:03.413 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:03.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:03.514 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:03.536 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:03.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:03.569 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-11T03:42:03.638 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:03.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:03.765 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:03.768 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:42:03.768 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:42:03.783 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:42:03.784 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:42:03.788 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:03.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:03.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:03.803 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:42:03.803 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:42:03.825 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:42:03.826 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:42:03.843 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:42:03.844 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:42:03.864 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:03.865 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:42:03.865 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:42:03.869 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:03.870 INFO:tasks.rados.rados.2.smithi175.stdout:1003: done (0 left) 2024-09-11T03:42:03.870 INFO:tasks.rados.rados.2.smithi175.stdout:1004: snap_remove snap 107 2024-09-11T03:42:03.884 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:42:03.885 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:42:03.890 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:03.912 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:42:03.912 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:42:03.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:03.914 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:03.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:03.928 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:42:03.928 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:42:03.945 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:42:03.945 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:42:03.961 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:42:03.961 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:42:03.978 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:42:03.978 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:42:04.016 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:04.038 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:04.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:04.039 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:04.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:03 smithi084 ceph-mon[27254]: 1.18 scrub starts 2024-09-11T03:42:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:03 smithi084 ceph-mon[27254]: 1.18 scrub ok 2024-09-11T03:42:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:03 smithi084 ceph-mon[27254]: pgmap v613: 57 pgs: 57 active+clean; 202 MiB data, 752 MiB used, 893 GiB / 894 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 8 op/s 2024-09-11T03:42:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:03 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:42:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:03 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:42:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:03 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4036723779' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:03 smithi084 ceph-mon[27254]: osdmap e307: 12 total, 12 up, 10 in 2024-09-11T03:42:04.166 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:04.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:04.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:04.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:03 smithi121 ceph-mon[30729]: 1.18 scrub starts 2024-09-11T03:42:04.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:03 smithi121 ceph-mon[30729]: 1.18 scrub ok 2024-09-11T03:42:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:03 smithi121 ceph-mon[30729]: pgmap v613: 57 pgs: 57 active+clean; 202 MiB data, 752 MiB used, 893 GiB / 894 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 8 op/s 2024-09-11T03:42:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:03 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:42:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:03 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:42:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:03 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4036723779' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:03 smithi121 ceph-mon[30729]: osdmap e307: 12 total, 12 up, 10 in 2024-09-11T03:42:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:03 smithi175 ceph-mon[30894]: 1.18 scrub starts 2024-09-11T03:42:04.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:03 smithi175 ceph-mon[30894]: 1.18 scrub ok 2024-09-11T03:42:04.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:03 smithi175 ceph-mon[30894]: pgmap v613: 57 pgs: 57 active+clean; 202 MiB data, 752 MiB used, 893 GiB / 894 GiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 8 op/s 2024-09-11T03:42:04.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:03 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:42:04.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:03 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:42:04.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:03 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4036723779' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:04.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:03 smithi175 ceph-mon[30894]: osdmap e307: 12 total, 12 up, 10 in 2024-09-11T03:42:04.268 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:04.290 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:04.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:04.292 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:04.393 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:04.419 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:04.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:04.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:04.521 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:04.542 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:04.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:04.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:04.557 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:42:04.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:04.670 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:04.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:04.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:04.798 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:04.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:04.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:04.872 INFO:tasks.rados.rados.2.smithi175.stdout:1004: done (0 left) 2024-09-11T03:42:04.872 INFO:tasks.rados.rados.2.smithi175.stdout:1005: snap_remove snap 105 2024-09-11T03:42:04.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:04.925 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:04.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:04.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:05.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:05.047 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:05.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:05.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:05.149 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:05.174 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:05.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:05.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:04 smithi175 ceph-mon[30894]: 1.10 scrub starts 2024-09-11T03:42:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:04 smithi175 ceph-mon[30894]: 1.10 scrub ok 2024-09-11T03:42:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:04 smithi175 ceph-mon[30894]: osdmap e308: 12 total, 12 up, 10 in 2024-09-11T03:42:05.245 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:04 smithi084 ceph-mon[27254]: 1.10 scrub starts 2024-09-11T03:42:05.246 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:04 smithi084 ceph-mon[27254]: 1.10 scrub ok 2024-09-11T03:42:05.246 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:04 smithi084 ceph-mon[27254]: osdmap e308: 12 total, 12 up, 10 in 2024-09-11T03:42:05.276 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:05.302 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:05.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:05.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:05.345 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:42:05.346 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:42:05.359 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:05.403 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:05.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:04 smithi121 ceph-mon[30729]: 1.10 scrub starts 2024-09-11T03:42:05.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:04 smithi121 ceph-mon[30729]: 1.10 scrub ok 2024-09-11T03:42:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:04 smithi121 ceph-mon[30729]: osdmap e308: 12 total, 12 up, 10 in 2024-09-11T03:42:05.429 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:05.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:05.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:05.531 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:05.557 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:05.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:05.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:05.659 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:05.685 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:05.686 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:05.787 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:05.813 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:05.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:05.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:05.867 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2024-09-11T03:42:05.874 INFO:tasks.rados.rados.2.smithi175.stdout:1005: done (0 left) 2024-09-11T03:42:05.874 INFO:tasks.rados.rados.2.smithi175.stdout:1006: read oid 14 snap 97 2024-09-11T03:42:05.874 INFO:tasks.rados.rados.2.smithi175.stdout:1006: expect (ObjNum 258 snap 64 seq_num 258) 2024-09-11T03:42:05.874 INFO:tasks.rados.rados.2.smithi175.stdout:1007: read oid 28 snap -1 2024-09-11T03:42:05.874 INFO:tasks.rados.rados.2.smithi175.stdout:1007: expect (ObjNum 342 snap 99 seq_num 342) 2024-09-11T03:42:05.874 INFO:tasks.rados.rados.2.smithi175.stdout:1008: snap_remove snap 86 2024-09-11T03:42:05.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:05.939 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:05.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:05.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:05.994 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T03:42:06.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:06.062 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:06.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:06.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:06.164 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:06.190 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:06.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:06.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:05 smithi175 ceph-mon[30894]: pgmap v616: 57 pgs: 57 active+clean; 206 MiB data, 761 MiB used, 893 GiB / 894 GiB avail; 1.2 KiB/s rd, 2.7 MiB/s wr, 11 op/s 2024-09-11T03:42:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:05 smithi175 ceph-mon[30894]: 1.16 scrub starts 2024-09-11T03:42:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:05 smithi175 ceph-mon[30894]: 1.16 scrub ok 2024-09-11T03:42:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:05 smithi175 ceph-mon[30894]: 1.6 scrub starts 2024-09-11T03:42:06.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:05 smithi175 ceph-mon[30894]: 1.6 scrub ok 2024-09-11T03:42:06.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:05 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/638588773' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:42:06.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:05 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/638588773' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:42:06.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:05 smithi175 ceph-mon[30894]: osdmap e309: 12 total, 12 up, 10 in 2024-09-11T03:42:06.244 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:05 smithi084 ceph-mon[27254]: pgmap v616: 57 pgs: 57 active+clean; 206 MiB data, 761 MiB used, 893 GiB / 894 GiB avail; 1.2 KiB/s rd, 2.7 MiB/s wr, 11 op/s 2024-09-11T03:42:06.245 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:05 smithi084 ceph-mon[27254]: 1.16 scrub starts 2024-09-11T03:42:06.245 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:05 smithi084 ceph-mon[27254]: 1.16 scrub ok 2024-09-11T03:42:06.245 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:05 smithi084 ceph-mon[27254]: 1.6 scrub starts 2024-09-11T03:42:06.245 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:05 smithi084 ceph-mon[27254]: 1.6 scrub ok 2024-09-11T03:42:06.245 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:05 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/638588773' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:42:06.245 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:05 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/638588773' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:42:06.245 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:05 smithi084 ceph-mon[27254]: osdmap e309: 12 total, 12 up, 10 in 2024-09-11T03:42:06.292 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:06.313 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:06.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:06.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:06.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:06.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:05 smithi121 ceph-mon[30729]: pgmap v616: 57 pgs: 57 active+clean; 206 MiB data, 761 MiB used, 893 GiB / 894 GiB avail; 1.2 KiB/s rd, 2.7 MiB/s wr, 11 op/s 2024-09-11T03:42:06.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:05 smithi121 ceph-mon[30729]: 1.16 scrub starts 2024-09-11T03:42:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:05 smithi121 ceph-mon[30729]: 1.16 scrub ok 2024-09-11T03:42:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:05 smithi121 ceph-mon[30729]: 1.6 scrub starts 2024-09-11T03:42:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:05 smithi121 ceph-mon[30729]: 1.6 scrub ok 2024-09-11T03:42:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:05 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/638588773' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:42:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:05 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/638588773' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:42:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:05 smithi121 ceph-mon[30729]: osdmap e309: 12 total, 12 up, 10 in 2024-09-11T03:42:06.440 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:06.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:06.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:06.539 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 8 1 2024-09-11T03:42:06.542 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:06.562 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:06.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:06.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:06.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:06.690 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:06.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:06.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:06.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:06.818 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:06.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:06.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:06.852 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:42:06.868 INFO:tasks.rados.rados.2.smithi175.stdout:1008: done (2 left) 2024-09-11T03:42:06.868 INFO:tasks.rados.rados.2.smithi175.stdout:1009: rollback oid 7 current snap is 108 2024-09-11T03:42:06.868 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 7 to 91 2024-09-11T03:42:06.868 INFO:tasks.rados.rados.2.smithi175.stdout:1010: read oid 34 snap -1 2024-09-11T03:42:06.868 INFO:tasks.rados.rados.2.smithi175.stdout:1010: expect (ObjNum 293 snap 79 seq_num 293) 2024-09-11T03:42:06.868 INFO:tasks.rados.rados.2.smithi175.stdout:1011: read oid 2 snap -1 2024-09-11T03:42:06.868 INFO:tasks.rados.rados.2.smithi175.stdout:1011: expect (ObjNum 314 snap 86 seq_num 314) 2024-09-11T03:42:06.868 INFO:tasks.rados.rados.2.smithi175.stdout:1012: snap_create 2024-09-11T03:42:06.869 INFO:tasks.rados.rados.2.smithi175.stdout:1006: expect (ObjNum 258 snap 64 seq_num 258) 2024-09-11T03:42:06.920 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:06.943 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:06.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:06.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:07.024 INFO:tasks.rados.rados.2.smithi175.stdout:1007: expect (ObjNum 342 snap 99 seq_num 342) 2024-09-11T03:42:07.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:07.069 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:07.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:07.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:07.082 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:06 smithi084 ceph-mon[27254]: 1.8 scrub starts 2024-09-11T03:42:07.083 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:06 smithi084 ceph-mon[27254]: 1.8 scrub ok 2024-09-11T03:42:07.083 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:06 smithi084 ceph-mon[27254]: osdmap e310: 12 total, 12 up, 10 in 2024-09-11T03:42:07.128 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:07.162 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:07.171 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:07.187 INFO:tasks.rados.rados.2.smithi175.stdout:1011: expect (ObjNum 314 snap 86 seq_num 314) 2024-09-11T03:42:07.193 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:07.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:07.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:07.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:06 smithi175 ceph-mon[30894]: 1.8 scrub starts 2024-09-11T03:42:07.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:06 smithi175 ceph-mon[30894]: 1.8 scrub ok 2024-09-11T03:42:07.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:06 smithi175 ceph-mon[30894]: osdmap e310: 12 total, 12 up, 10 in 2024-09-11T03:42:07.295 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:07.310 INFO:tasks.rados.rados.2.smithi175.stdout:1009: finishing rollback tid 0 to smithi17557870-7 2024-09-11T03:42:07.311 INFO:tasks.rados.rados.2.smithi175.stdout:1009: finishing rollback tid 1 to smithi17557870-7 2024-09-11T03:42:07.311 INFO:tasks.rados.rados.2.smithi175.stdout:1009: finishing rollback tid 2 to smithi17557870-7 2024-09-11T03:42:07.311 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 185 (ObjNum 295 snap 79 seq_num 295) dirty exists 2024-09-11T03:42:07.311 INFO:tasks.rados.rados.2.smithi175.stdout:1010: expect (ObjNum 293 snap 79 seq_num 293) 2024-09-11T03:42:07.316 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:07.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:07.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:07.418 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:06 smithi121 ceph-mon[30729]: 1.8 scrub starts 2024-09-11T03:42:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:06 smithi121 ceph-mon[30729]: 1.8 scrub ok 2024-09-11T03:42:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:06 smithi121 ceph-mon[30729]: osdmap e310: 12 total, 12 up, 10 in 2024-09-11T03:42:07.444 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:07.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:07.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:07.546 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:07.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:07.569 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:07.570 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:07.671 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:07.693 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:07.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:07.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:07.795 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:07.816 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:07.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:07.865 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.8 to 1 (10000) 2024-09-11T03:42:07.865 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:42:07.870 INFO:tasks.rados.rados.2.smithi175.stdout:1006: done (5 left) 2024-09-11T03:42:07.870 INFO:tasks.rados.rados.2.smithi175.stdout:1007: done (4 left) 2024-09-11T03:42:07.870 INFO:tasks.rados.rados.2.smithi175.stdout:1009: done (3 left) 2024-09-11T03:42:07.871 INFO:tasks.rados.rados.2.smithi175.stdout:1010: done (2 left) 2024-09-11T03:42:07.871 INFO:tasks.rados.rados.2.smithi175.stdout:1011: done (1 left) 2024-09-11T03:42:07.871 INFO:tasks.rados.rados.2.smithi175.stdout:1012: done (0 left) 2024-09-11T03:42:07.871 INFO:tasks.rados.rados.2.smithi175.stdout:1013: snap_create 2024-09-11T03:42:07.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:07.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:07.947 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:07.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:08.049 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:08.071 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:08.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:08.072 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:08.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:07 smithi084 ceph-mon[27254]: pgmap v619: 57 pgs: 1 peering, 56 active+clean; 200 MiB data, 755 MiB used, 893 GiB / 894 GiB avail; 4.8 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-11T03:42:08.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2018779369' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:42:08.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1967919998' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:42:08.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:07 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:08.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2018779369' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:42:08.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1967919998' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:42:08.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:07 smithi084 ceph-mon[27254]: osdmap e311: 12 total, 12 up, 10 in 2024-09-11T03:42:08.173 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:08.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:08.195 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:08.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:07 smithi175 ceph-mon[30894]: pgmap v619: 57 pgs: 1 peering, 56 active+clean; 200 MiB data, 755 MiB used, 893 GiB / 894 GiB avail; 4.8 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-11T03:42:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2018779369' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:42:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1967919998' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:42:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:07 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2018779369' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:42:08.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1967919998' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:42:08.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:07 smithi175 ceph-mon[30894]: osdmap e311: 12 total, 12 up, 10 in 2024-09-11T03:42:08.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:08.322 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:08.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:08.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:08.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:07 smithi121 ceph-mon[30729]: pgmap v619: 57 pgs: 1 peering, 56 active+clean; 200 MiB data, 755 MiB used, 893 GiB / 894 GiB avail; 4.8 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-11T03:42:08.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2018779369' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:42:08.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1967919998' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:42:08.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:07 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:08.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2018779369' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:42:08.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1967919998' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:42:08.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:07 smithi121 ceph-mon[30729]: osdmap e311: 12 total, 12 up, 10 in 2024-09-11T03:42:08.424 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:08.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:08.447 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:08.447 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:08.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-09-11T03:42:08.476+0000 7ff582ffd640 0 --2- 172.21.15.84:0/566580282 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff584104600 0x7ff58410caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-11T03:42:08.498 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:42:08.498 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:42:08.515 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:42:08.515 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:42:08.531 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:42:08.531 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:42:08.542 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:08.548 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:08.550 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:42:08.551 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:42:08.570 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:42:08.570 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:42:08.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:08.571 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:08.572 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:08.586 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:42:08.587 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:42:08.603 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:42:08.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:42:08.620 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:42:08.620 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:42:08.636 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:42:08.637 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:42:08.653 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:42:08.653 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:42:08.670 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:42:08.671 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:42:08.672 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:08.688 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:42:08.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:42:08.695 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:08.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:08.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:08.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:08.818 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:08.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:08.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:08.869 INFO:tasks.rados.rados.2.smithi175.stdout:1013: done (0 left) 2024-09-11T03:42:08.869 INFO:tasks.rados.rados.2.smithi175.stdout:1014: snap_create 2024-09-11T03:42:08.920 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:08.943 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:08.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:08.944 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:08.995 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:08 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3665955586' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:08.995 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:08 smithi084 ceph-mon[27254]: osdmap e312: 12 total, 12 up, 10 in 2024-09-11T03:42:09.021 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 9 1 2024-09-11T03:42:09.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:09.085 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:09.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:09.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:09.187 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:09.209 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:09.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:09.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:08 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3665955586' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:09.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:08 smithi175 ceph-mon[30894]: osdmap e312: 12 total, 12 up, 10 in 2024-09-11T03:42:09.311 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:09.336 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:09.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:09.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:09.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:08 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3665955586' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:08 smithi121 ceph-mon[30729]: osdmap e312: 12 total, 12 up, 10 in 2024-09-11T03:42:09.438 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:09.460 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:09.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:09.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:09.562 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:09.588 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:09.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:09.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:09.690 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:09.715 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:09.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:09.716 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:09.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:09.841 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:09.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:09.872 INFO:tasks.rados.rados.2.smithi175.stdout:1014: done (0 left) 2024-09-11T03:42:09.872 INFO:tasks.rados.rados.2.smithi175.stdout:1015: delete oid 8 current snap is 111 2024-09-11T03:42:09.873 INFO:tasks.rados.rados.2.smithi175.stdout:1015: done (0 left) 2024-09-11T03:42:09.873 INFO:tasks.rados.rados.2.smithi175.stdout:1016: rollback oid 18 current snap is 111 2024-09-11T03:42:09.873 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 18 to 91 2024-09-11T03:42:09.874 INFO:tasks.rados.rados.2.smithi175.stdout:1017: delete oid 30 current snap is 111 2024-09-11T03:42:09.875 INFO:tasks.rados.rados.2.smithi175.stdout:1016: finishing rollback tid 0 to smithi17557870-18 2024-09-11T03:42:09.875 INFO:tasks.rados.rados.2.smithi175.stdout:1017: done (1 left) 2024-09-11T03:42:09.875 INFO:tasks.rados.rados.2.smithi175.stdout:1018: read oid 39 snap -1 2024-09-11T03:42:09.875 INFO:tasks.rados.rados.2.smithi175.stdout:1018: expect (ObjNum 262 snap 66 seq_num 262) 2024-09-11T03:42:09.875 INFO:tasks.rados.rados.2.smithi175.stdout:1019: copy_from oid 9 from oid 32 current snap is 111 2024-09-11T03:42:09.876 INFO:tasks.rados.rados.2.smithi175.stdout:1020: snap_remove snap 104 2024-09-11T03:42:09.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:09.963 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:09.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:09.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:10.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:10.087 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:10.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:10.190 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:10.193 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:09 smithi084 ceph-mon[27254]: pgmap v622: 57 pgs: 2 active+undersized+remapped, 1 peering, 54 active+clean; 196 MiB data, 747 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 10 op/s; 2.6 MiB/s, 6 objects/s recovering 2024-09-11T03:42:10.193 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:09 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:42:10.193 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:09 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:42:10.193 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:09 smithi084 ceph-mon[27254]: osdmap e313: 12 total, 12 up, 10 in 2024-09-11T03:42:10.206 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:09 smithi175 ceph-mon[30894]: pgmap v622: 57 pgs: 2 active+undersized+remapped, 1 peering, 54 active+clean; 196 MiB data, 747 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 10 op/s; 2.6 MiB/s, 6 objects/s recovering 2024-09-11T03:42:10.206 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:09 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:42:10.207 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:09 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:42:10.207 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:09 smithi175 ceph-mon[30894]: osdmap e313: 12 total, 12 up, 10 in 2024-09-11T03:42:10.217 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:10.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:10.218 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:10.319 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:10.331 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:10.341 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:10.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:10.342 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:09 smithi121 ceph-mon[30729]: pgmap v622: 57 pgs: 2 active+undersized+remapped, 1 peering, 54 active+clean; 196 MiB data, 747 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 10 op/s; 2.6 MiB/s, 6 objects/s recovering 2024-09-11T03:42:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:09 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:42:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:09 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:42:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:09 smithi121 ceph-mon[30729]: osdmap e313: 12 total, 12 up, 10 in 2024-09-11T03:42:10.442 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:10.465 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:10.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:10.546 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:42:10.546 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:42:10.546 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:42:10.546 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:42:10.546 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:42:10.567 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:10.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:10.593 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:10.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:10.694 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:10.716 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:10.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:10.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:10.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:10.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:10.841 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:10.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:10.875 INFO:tasks.rados.rados.2.smithi175.stdout:1020: done (3 left) 2024-09-11T03:42:10.875 INFO:tasks.rados.rados.2.smithi175.stdout:1021: copy_from oid 47 from oid 15 current snap is 111 2024-09-11T03:42:10.875 INFO:tasks.rados.rados.2.smithi175.stdout:1016: finishing rollback tid 1 to smithi17557870-18 2024-09-11T03:42:10.876 INFO:tasks.rados.rados.2.smithi175.stdout:1016: finishing rollback tid 2 to smithi17557870-18 2024-09-11T03:42:10.876 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 210 (ObjNum 261 snap 66 seq_num 261) dirty exists 2024-09-11T03:42:10.876 INFO:tasks.rados.rados.2.smithi175.stdout:1018: expect (ObjNum 262 snap 66 seq_num 262) 2024-09-11T03:42:10.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:10.963 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:10.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:10.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:10.987 INFO:tasks.rados.rados.2.smithi175.stdout:1019: finishing copy_from racing read to smithi17557870-9 2024-09-11T03:42:10.987 INFO:tasks.rados.rados.2.smithi175.stdout:1019: finishing copy_from to smithi17557870-9 2024-09-11T03:42:10.987 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 223 (ObjNum 277 snap 73 seq_num 277) dirty exists 2024-09-11T03:42:10.987 INFO:tasks.rados.rados.2.smithi175.stdout:1016: done (3 left) 2024-09-11T03:42:10.987 INFO:tasks.rados.rados.2.smithi175.stdout:1018: done (2 left) 2024-09-11T03:42:10.987 INFO:tasks.rados.rados.2.smithi175.stdout:1019: done (1 left) 2024-09-11T03:42:10.987 INFO:tasks.rados.rados.2.smithi175.stdout:1022: rollback oid 16 current snap is 111 2024-09-11T03:42:10.987 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 16 to 110 2024-09-11T03:42:10.987 INFO:tasks.rados.rados.2.smithi175.stdout:1021: finishing copy_from racing read to smithi17557870-47 2024-09-11T03:42:10.988 INFO:tasks.rados.rados.2.smithi175.stdout:1021: finishing copy_from to smithi17557870-47 2024-09-11T03:42:10.988 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 157 (ObjNum 317 snap 86 seq_num 317) dirty exists 2024-09-11T03:42:10.988 INFO:tasks.rados.rados.2.smithi175.stdout:1021: done (1 left) 2024-09-11T03:42:10.988 INFO:tasks.rados.rados.2.smithi175.stdout:1023: write (excl) oid 36 current snap is 111 2024-09-11T03:42:10.988 INFO:tasks.rados.rados.2.smithi175.stdout:1023: seq_num 366 ranges {551362=726078,1717586=409779} 2024-09-11T03:42:10.993 INFO:tasks.rados.rados.2.smithi175.stdout:1023: writing smithi17557870-36 from 551362 to 1277440 tid 1 2024-09-11T03:42:10.998 INFO:tasks.rados.rados.2.smithi175.stdout:1023: writing smithi17557870-36 from 1717586 to 2127365 tid 2 2024-09-11T03:42:10.998 INFO:tasks.rados.rados.2.smithi175.stdout:1024: snap_remove snap 108 2024-09-11T03:42:11.065 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:11.086 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:11.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:11.087 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:11.188 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:11.209 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:11.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:11.210 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:11.311 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:11.338 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:11.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:11.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:11.440 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:11.465 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:11.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:11.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:11.567 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:11.588 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:11.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:11.589 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:11.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:42:11.665 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T03:42:11.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:11.728 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:11.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:11.830 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:11.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:11.852 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:11.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:11.864 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.9 to 1 (10000) 2024-09-11T03:42:11.869 INFO:tasks.rados.rados.2.smithi175.stdout:1024: done (2 left) 2024-09-11T03:42:11.869 INFO:tasks.rados.rados.2.smithi175.stdout:1025: write oid 28 current snap is 111 2024-09-11T03:42:11.869 INFO:tasks.rados.rados.2.smithi175.stdout:1022: finishing rollback tid 0 to smithi17557870-16 2024-09-11T03:42:11.869 INFO:tasks.rados.rados.2.smithi175.stdout:1025: seq_num 367 ranges {568169=401327,1376068=669790} 2024-09-11T03:42:11.874 INFO:tasks.rados.rados.2.smithi175.stdout:1025: writing smithi17557870-28 from 568169 to 969496 tid 1 2024-09-11T03:42:11.881 INFO:tasks.rados.rados.2.smithi175.stdout:1025: writing smithi17557870-28 from 1376068 to 2045858 tid 2 2024-09-11T03:42:11.881 INFO:tasks.rados.rados.2.smithi175.stdout:1026: read oid 30 snap -1 2024-09-11T03:42:11.881 INFO:tasks.rados.rados.2.smithi175.stdout:1026: expect deleted 2024-09-11T03:42:11.881 INFO:tasks.rados.rados.2.smithi175.stdout:1027: write (excl) oid 20 current snap is 111 2024-09-11T03:42:11.881 INFO:tasks.rados.rados.2.smithi175.stdout:1022: finishing rollback tid 1 to smithi17557870-16 2024-09-11T03:42:11.881 INFO:tasks.rados.rados.2.smithi175.stdout:1022: finishing rollback tid 2 to smithi17557870-16 2024-09-11T03:42:11.881 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 179 (ObjNum 354 snap 104 seq_num 354) dirty exists 2024-09-11T03:42:11.881 INFO:tasks.rados.rados.2.smithi175.stdout:1023: finishing write tid 1 to smithi17557870-36 2024-09-11T03:42:11.881 INFO:tasks.rados.rados.2.smithi175.stdout:1023: finishing write tid 2 to smithi17557870-36 2024-09-11T03:42:11.881 INFO:tasks.rados.rados.2.smithi175.stdout:1027: seq_num 368 ranges {584976=476576,1834550=350426} 2024-09-11T03:42:11.886 INFO:tasks.rados.rados.2.smithi175.stdout:1027: writing smithi17557870-20 from 584976 to 1061552 tid 1 2024-09-11T03:42:11.892 INFO:tasks.rados.rados.2.smithi175.stdout:1027: writing smithi17557870-20 from 1834550 to 2184976 tid 2 2024-09-11T03:42:11.893 INFO:tasks.rados.rados.2.smithi175.stdout:1022: done (4 left) 2024-09-11T03:42:11.893 INFO:tasks.rados.rados.2.smithi175.stdout:1028: snap_create 2024-09-11T03:42:11.893 INFO:tasks.rados.rados.2.smithi175.stdout:1023: finishing write tid 3 to smithi17557870-36 2024-09-11T03:42:11.893 INFO:tasks.rados.rados.2.smithi175.stdout:1023: finishing write tid 4 to smithi17557870-36 2024-09-11T03:42:11.893 INFO:tasks.rados.rados.2.smithi175.stdout:1023: oid 36 updating version 0 to 208 2024-09-11T03:42:11.893 INFO:tasks.rados.rados.2.smithi175.stdout:1023: oid 36 version 208 is already newer than 206 2024-09-11T03:42:11.893 INFO:tasks.rados.rados.2.smithi175.stdout:1023: oid 36 version 208 is already newer than 207 2024-09-11T03:42:11.893 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 208 (ObjNum 366 snap 111 seq_num 366) dirty exists 2024-09-11T03:42:11.893 INFO:tasks.rados.rados.2.smithi175.stdout:1023: left oid 36 (ObjNum 366 snap 111 seq_num 366) 2024-09-11T03:42:11.893 INFO:tasks.rados.rados.2.smithi175.stdout:1025: finishing write tid 1 to smithi17557870-28 2024-09-11T03:42:11.893 INFO:tasks.rados.rados.2.smithi175.stdout:1025: finishing write tid 2 to smithi17557870-28 2024-09-11T03:42:11.893 INFO:tasks.rados.rados.2.smithi175.stdout:1025: finishing write tid 3 to smithi17557870-28 2024-09-11T03:42:11.894 INFO:tasks.rados.rados.2.smithi175.stdout:1025: finishing write tid 4 to smithi17557870-28 2024-09-11T03:42:11.894 INFO:tasks.rados.rados.2.smithi175.stdout:1025: oid 28 updating version 0 to 385 2024-09-11T03:42:11.894 INFO:tasks.rados.rados.2.smithi175.stdout:1025: oid 28 version 385 is already newer than 384 2024-09-11T03:42:11.894 INFO:tasks.rados.rados.2.smithi175.stdout:1025: oid 28 updating version 385 to 386 2024-09-11T03:42:11.894 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 386 (ObjNum 367 snap 111 seq_num 367) dirty exists 2024-09-11T03:42:11.894 INFO:tasks.rados.rados.2.smithi175.stdout:1025: left oid 28 (ObjNum 367 snap 111 seq_num 367) 2024-09-11T03:42:11.894 INFO:tasks.rados.rados.2.smithi175.stdout:1027: finishing write tid 1 to smithi17557870-20 2024-09-11T03:42:11.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:11 smithi084 ceph-mon[27254]: pgmap v625: 57 pgs: 2 active+undersized+remapped, 1 peering, 54 active+clean; 196 MiB data, 746 MiB used, 893 GiB / 894 GiB avail; 7.0 MiB/s rd, 5 op/s; 2.6 MiB/s, 6 objects/s recovering 2024-09-11T03:42:11.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:11 smithi084 ceph-mon[27254]: osdmap e314: 12 total, 12 up, 10 in 2024-09-11T03:42:11.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:11 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/304060714' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:42:11.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:11 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/304060714' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:42:11.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:11 smithi084 ceph-mon[27254]: osdmap e315: 12 total, 12 up, 10 in 2024-09-11T03:42:11.895 INFO:tasks.rados.rados.2.smithi175.stdout:1027: finishing write tid 2 to smithi17557870-20 2024-09-11T03:42:11.896 INFO:tasks.rados.rados.2.smithi175.stdout:1027: finishing write tid 3 to smithi17557870-20 2024-09-11T03:42:11.896 INFO:tasks.rados.rados.2.smithi175.stdout:1027: finishing write tid 4 to smithi17557870-20 2024-09-11T03:42:11.896 INFO:tasks.rados.rados.2.smithi175.stdout:1027: oid 20 updating version 0 to 397 2024-09-11T03:42:11.896 INFO:tasks.rados.rados.2.smithi175.stdout:1027: oid 20 updating version 397 to 398 2024-09-11T03:42:11.896 INFO:tasks.rados.rados.2.smithi175.stdout:1027: oid 20 version 398 is already newer than 396 2024-09-11T03:42:11.896 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 398 (ObjNum 368 snap 111 seq_num 368) dirty exists 2024-09-11T03:42:11.896 INFO:tasks.rados.rados.2.smithi175.stdout:1027: left oid 20 (ObjNum 368 snap 111 seq_num 368) 2024-09-11T03:42:11.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:11.975 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:11.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:11.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:12.077 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:12.099 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:12.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:42:12.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:12.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:12.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:11 smithi121 ceph-mon[30729]: pgmap v625: 57 pgs: 2 active+undersized+remapped, 1 peering, 54 active+clean; 196 MiB data, 746 MiB used, 893 GiB / 894 GiB avail; 7.0 MiB/s rd, 5 op/s; 2.6 MiB/s, 6 objects/s recovering 2024-09-11T03:42:12.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:11 smithi121 ceph-mon[30729]: osdmap e314: 12 total, 12 up, 10 in 2024-09-11T03:42:12.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:11 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/304060714' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:42:12.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:11 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/304060714' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:42:12.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:11 smithi121 ceph-mon[30729]: osdmap e315: 12 total, 12 up, 10 in 2024-09-11T03:42:12.203 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:12.224 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:12.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:12.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:11 smithi175 ceph-mon[30894]: pgmap v625: 57 pgs: 2 active+undersized+remapped, 1 peering, 54 active+clean; 196 MiB data, 746 MiB used, 893 GiB / 894 GiB avail; 7.0 MiB/s rd, 5 op/s; 2.6 MiB/s, 6 objects/s recovering 2024-09-11T03:42:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:11 smithi175 ceph-mon[30894]: osdmap e314: 12 total, 12 up, 10 in 2024-09-11T03:42:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:11 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/304060714' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:42:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:11 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/304060714' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:42:12.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:11 smithi175 ceph-mon[30894]: osdmap e315: 12 total, 12 up, 10 in 2024-09-11T03:42:12.325 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:12.346 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:12.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:12.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:12.448 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:12.474 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:12.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:12.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:12.498 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 11 1 2024-09-11T03:42:12.526 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:12.569 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:12.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:12.602 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:12.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:12.704 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:12.730 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:12.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:12.731 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:12.832 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:12.857 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:12.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:12.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:12.877 INFO:tasks.rados.rados.2.smithi175.stdout:1023: done (4 left) 2024-09-11T03:42:12.878 INFO:tasks.rados.rados.2.smithi175.stdout:1025: done (3 left) 2024-09-11T03:42:12.878 INFO:tasks.rados.rados.2.smithi175.stdout:1026: done (2 left) 2024-09-11T03:42:12.878 INFO:tasks.rados.rados.2.smithi175.stdout:1027: done (1 left) 2024-09-11T03:42:12.878 INFO:tasks.rados.rados.2.smithi175.stdout:1028: done (0 left) 2024-09-11T03:42:12.878 INFO:tasks.rados.rados.2.smithi175.stdout:1029: read oid 22 snap -1 2024-09-11T03:42:12.878 INFO:tasks.rados.rados.2.smithi175.stdout:1029: expect (ObjNum 342 snap 99 seq_num 342) 2024-09-11T03:42:12.878 INFO:tasks.rados.rados.2.smithi175.stdout:1030: read oid 11 snap 99 2024-09-11T03:42:12.878 INFO:tasks.rados.rados.2.smithi175.stdout:1030: expect (ObjNum 270 snap 69 seq_num 270) 2024-09-11T03:42:12.878 INFO:tasks.rados.rados.2.smithi175.stdout:1031: snap_create 2024-09-11T03:42:12.889 INFO:tasks.rados.rados.2.smithi175.stdout:1030: expect (ObjNum 270 snap 69 seq_num 270) 2024-09-11T03:42:12.959 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:12.967 INFO:tasks.rados.rados.2.smithi175.stdout:1029: expect (ObjNum 342 snap 99 seq_num 342) 2024-09-11T03:42:12.979 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:12.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:12.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:13.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:13.103 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:13.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:13.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:13.204 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:13.225 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:13.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:13.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:13.328 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:13.348 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:13.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:13.349 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:13.450 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:13.471 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:13.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:13.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:13.546 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:42:13.547 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:42:13.554 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:13.566 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:42:13.567 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:42:13.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:13.584 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:13.585 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:42:13.586 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:42:13.599 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:13.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:13.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:13.603 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:42:13.604 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:42:13.621 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:42:13.621 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:42:13.638 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:42:13.638 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:42:13.654 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:42:13.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:42:13.671 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:42:13.671 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:42:13.688 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:42:13.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:42:13.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:13.705 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:42:13.705 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:42:13.722 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:42:13.722 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:42:13.726 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:13.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:13.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:13.740 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:42:13.740 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:42:13.829 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:13.851 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:13.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:13.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:13.882 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:42:13.885 INFO:tasks.rados.rados.2.smithi175.stdout:1029: done (2 left) 2024-09-11T03:42:13.885 INFO:tasks.rados.rados.2.smithi175.stdout:1030: done (1 left) 2024-09-11T03:42:13.885 INFO:tasks.rados.rados.2.smithi175.stdout:1031: done (0 left) 2024-09-11T03:42:13.885 INFO:tasks.rados.rados.2.smithi175.stdout:1032: snap_remove snap 97 2024-09-11T03:42:13.953 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:13.974 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:13.975 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:14.076 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:14.098 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:14.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:14.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:13 smithi084 ceph-mon[27254]: osdmap e316: 12 total, 12 up, 10 in 2024-09-11T03:42:14.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:13 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:42:14.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:13 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:42:14.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:13 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:42:14.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:13 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:42:14.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:13 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:42:14.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:13 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:42:14.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:13 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:42:14.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:13 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:42:14.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:13 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:42:14.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:13 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:14.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:13 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:42:14.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:13 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:42:14.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:13 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:42:14.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:13 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:14.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:13 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2152413710' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:14.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:13 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/142930099' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:14.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:13 smithi121 ceph-mon[30729]: osdmap e316: 12 total, 12 up, 10 in 2024-09-11T03:42:14.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:13 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:42:14.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:13 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:42:14.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:13 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:42:14.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:13 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:42:14.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:13 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:42:14.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:13 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:42:14.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:13 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:42:14.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:13 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:42:14.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:13 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:42:14.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:13 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:14.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:13 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:42:14.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:13 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:42:14.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:13 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:42:14.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:13 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:14.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:13 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2152413710' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:14.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:13 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/142930099' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:14.200 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:14.221 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:14.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:14.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:13 smithi175 ceph-mon[30894]: osdmap e316: 12 total, 12 up, 10 in 2024-09-11T03:42:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:13 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:42:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:13 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:42:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:13 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:42:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:13 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:42:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:13 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:42:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:13 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:42:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:13 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:42:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:13 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:42:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:13 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:42:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:13 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:13 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:42:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:13 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:42:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:13 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:42:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:13 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:42:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:13 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2152413710' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:14.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:13 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/142930099' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:14.322 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:14.344 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:14.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:14.345 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:14.446 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:14.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:14.468 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:14.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:14.569 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:14.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:14.591 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:14.591 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:14.692 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:14.704 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:42:14.704 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:42:14.713 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:14.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:14.714 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:14.815 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:14.837 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:14.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:14.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:14.895 INFO:tasks.rados.rados.2.smithi175.stdout:1032: done (0 left) 2024-09-11T03:42:14.895 INFO:tasks.rados.rados.2.smithi175.stdout:1033: write (excl) oid 47 current snap is 113 2024-09-11T03:42:14.895 INFO:tasks.rados.rados.2.smithi175.stdout:1033: seq_num 369 ranges {601783=551825,1776679=425104} 2024-09-11T03:42:14.902 INFO:tasks.rados.rados.2.smithi175.stdout:1033: writing smithi17557870-47 from 601783 to 1153608 tid 1 2024-09-11T03:42:14.908 INFO:tasks.rados.rados.2.smithi175.stdout:1033: writing smithi17557870-47 from 1776679 to 2201783 tid 2 2024-09-11T03:42:14.908 INFO:tasks.rados.rados.2.smithi175.stdout:1034: write (excl) oid 33 current snap is 113 2024-09-11T03:42:14.908 INFO:tasks.rados.rados.2.smithi175.stdout:1033: finishing write tid 1 to smithi17557870-47 2024-09-11T03:42:14.908 INFO:tasks.rados.rados.2.smithi175.stdout:1034: seq_num 370 ranges {618590=627074,1835161=383429} 2024-09-11T03:42:14.914 INFO:tasks.rados.rados.2.smithi175.stdout:1034: writing smithi17557870-33 from 618590 to 1245664 tid 1 2024-09-11T03:42:14.920 INFO:tasks.rados.rados.2.smithi175.stdout:1034: writing smithi17557870-33 from 1835161 to 2218590 tid 2 2024-09-11T03:42:14.920 INFO:tasks.rados.rados.2.smithi175.stdout:1035: write oid 11 current snap is 113 2024-09-11T03:42:14.920 INFO:tasks.rados.rados.2.smithi175.stdout:1035: seq_num 371 ranges {635397=702323,1893643=341754} 2024-09-11T03:42:14.927 INFO:tasks.rados.rados.2.smithi175.stdout:1035: writing smithi17557870-11 from 635397 to 1337720 tid 1 2024-09-11T03:42:14.932 INFO:tasks.rados.rados.2.smithi175.stdout:1035: writing smithi17557870-11 from 1893643 to 2235397 tid 2 2024-09-11T03:42:14.932 INFO:tasks.rados.rados.2.smithi175.stdout:1036: read oid 28 snap -1 2024-09-11T03:42:14.932 INFO:tasks.rados.rados.2.smithi175.stdout:1036: expect (ObjNum 367 snap 111 seq_num 367) 2024-09-11T03:42:14.932 INFO:tasks.rados.rados.2.smithi175.stdout:1037: snap_remove snap 110 2024-09-11T03:42:14.939 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:14.960 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:14.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:14.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:15.038 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:14 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:42:15.038 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:14 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:42:15.038 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:14 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:15.038 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:14 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2152413710' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:15.038 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:14 smithi084 ceph-mon[27254]: osdmap e317: 12 total, 12 up, 10 in 2024-09-11T03:42:15.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:15.087 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:15.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:15.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:14 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:42:15.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:14 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:42:15.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:14 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:15.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:14 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2152413710' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:15.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:14 smithi121 ceph-mon[30729]: osdmap e317: 12 total, 12 up, 10 in 2024-09-11T03:42:15.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:15.210 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:15.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:15.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:15.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:14 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:42:15.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:14 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:42:15.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:14 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:15.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:14 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2152413710' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:15.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:14 smithi175 ceph-mon[30894]: osdmap e317: 12 total, 12 up, 10 in 2024-09-11T03:42:15.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:15.338 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:15.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:15.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:15.440 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:15.461 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:15.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:15.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:15.564 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:15.588 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:15.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:15.690 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:15.711 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:15.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:15.712 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:15.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:15.838 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:15.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:15.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:15.898 INFO:tasks.rados.rados.2.smithi175.stdout:1037: done (4 left) 2024-09-11T03:42:15.898 INFO:tasks.rados.rados.2.smithi175.stdout:1038: rollback oid 8 current snap is 113 2024-09-11T03:42:15.898 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 8 to 106 2024-09-11T03:42:15.898 INFO:tasks.rados.rados.2.smithi175.stdout:1039: rollback oid 14 current snap is 113 2024-09-11T03:42:15.898 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 14 to 112 2024-09-11T03:42:15.898 INFO:tasks.rados.rados.2.smithi175.stdout:1040: write oid 40 current snap is 113 2024-09-11T03:42:15.898 INFO:tasks.rados.rados.2.smithi175.stdout:1040: seq_num 372 ranges {652204=777572,1952125=300079} 2024-09-11T03:42:15.905 INFO:tasks.rados.rados.2.smithi175.stdout:1040: writing smithi17557870-40 from 652204 to 1429776 tid 1 2024-09-11T03:42:15.910 INFO:tasks.rados.rados.2.smithi175.stdout:1040: writing smithi17557870-40 from 1952125 to 2252204 tid 2 2024-09-11T03:42:15.910 INFO:tasks.rados.rados.2.smithi175.stdout:1041: snap_create 2024-09-11T03:42:15.910 INFO:tasks.rados.rados.2.smithi175.stdout:1033: finishing write tid 2 to smithi17557870-47 2024-09-11T03:42:15.910 INFO:tasks.rados.rados.2.smithi175.stdout:1033: finishing write tid 3 to smithi17557870-47 2024-09-11T03:42:15.911 INFO:tasks.rados.rados.2.smithi175.stdout:1033: finishing write tid 4 to smithi17557870-47 2024-09-11T03:42:15.911 INFO:tasks.rados.rados.2.smithi175.stdout:1033: oid 47 updating version 0 to 408 2024-09-11T03:42:15.911 INFO:tasks.rados.rados.2.smithi175.stdout:1033: oid 47 version 408 is already newer than 407 2024-09-11T03:42:15.911 INFO:tasks.rados.rados.2.smithi175.stdout:1033: oid 47 version 408 is already newer than 406 2024-09-11T03:42:15.911 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 408 (ObjNum 369 snap 113 seq_num 369) dirty exists 2024-09-11T03:42:15.911 INFO:tasks.rados.rados.2.smithi175.stdout:1033: left oid 47 (ObjNum 369 snap 113 seq_num 369) 2024-09-11T03:42:15.911 INFO:tasks.rados.rados.2.smithi175.stdout:1034: finishing write tid 1 to smithi17557870-33 2024-09-11T03:42:15.911 INFO:tasks.rados.rados.2.smithi175.stdout:1034: finishing write tid 2 to smithi17557870-33 2024-09-11T03:42:15.911 INFO:tasks.rados.rados.2.smithi175.stdout:1034: finishing write tid 3 to smithi17557870-33 2024-09-11T03:42:15.911 INFO:tasks.rados.rados.2.smithi175.stdout:1034: finishing write tid 4 to smithi17557870-33 2024-09-11T03:42:15.911 INFO:tasks.rados.rados.2.smithi175.stdout:1034: oid 33 updating version 0 to 395 2024-09-11T03:42:15.911 INFO:tasks.rados.rados.2.smithi175.stdout:1034: oid 33 updating version 395 to 397 2024-09-11T03:42:15.911 INFO:tasks.rados.rados.2.smithi175.stdout:1034: oid 33 version 397 is already newer than 396 2024-09-11T03:42:15.911 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 397 (ObjNum 370 snap 113 seq_num 370) dirty exists 2024-09-11T03:42:15.911 INFO:tasks.rados.rados.2.smithi175.stdout:1034: left oid 33 (ObjNum 370 snap 113 seq_num 370) 2024-09-11T03:42:15.912 INFO:tasks.rados.rados.2.smithi175.stdout:1035: finishing write tid 1 to smithi17557870-11 2024-09-11T03:42:15.912 INFO:tasks.rados.rados.2.smithi175.stdout:1035: finishing write tid 2 to smithi17557870-11 2024-09-11T03:42:15.912 INFO:tasks.rados.rados.2.smithi175.stdout:1035: finishing write tid 3 to smithi17557870-11 2024-09-11T03:42:15.912 INFO:tasks.rados.rados.2.smithi175.stdout:1035: finishing write tid 4 to smithi17557870-11 2024-09-11T03:42:15.912 INFO:tasks.rados.rados.2.smithi175.stdout:1035: oid 11 updating version 0 to 139 2024-09-11T03:42:15.912 INFO:tasks.rados.rados.2.smithi175.stdout:1035: oid 11 version 139 is already newer than 135 2024-09-11T03:42:15.912 INFO:tasks.rados.rados.2.smithi175.stdout:1035: oid 11 version 139 is already newer than 138 2024-09-11T03:42:15.912 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 139 (ObjNum 371 snap 113 seq_num 371) dirty exists 2024-09-11T03:42:15.912 INFO:tasks.rados.rados.2.smithi175.stdout:1035: left oid 11 (ObjNum 371 snap 113 seq_num 371) 2024-09-11T03:42:15.912 INFO:tasks.rados.rados.2.smithi175.stdout:1036: expect (ObjNum 367 snap 111 seq_num 367) 2024-09-11T03:42:15.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:15.961 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:15.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:15.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:16.026 INFO:tasks.rados.rados.2.smithi175.stdout:1038: finishing rollback tid 1 to smithi17557870-8 2024-09-11T03:42:16.026 INFO:tasks.rados.rados.2.smithi175.stdout:1038: finishing rollback tid 2 to smithi17557870-8 2024-09-11T03:42:16.027 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 401 (ObjNum 319 snap 86 seq_num 319) dirty exists 2024-09-11T03:42:16.027 INFO:tasks.rados.rados.2.smithi175.stdout:1039: finishing rollback tid 0 to smithi17557870-14 2024-09-11T03:42:16.027 INFO:tasks.rados.rados.2.smithi175.stdout:1039: finishing rollback tid 1 to smithi17557870-14 2024-09-11T03:42:16.027 INFO:tasks.rados.rados.2.smithi175.stdout:1039: finishing rollback tid 2 to smithi17557870-14 2024-09-11T03:42:16.027 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 405 (ObjNum 258 snap 64 seq_num 258) dirty exists 2024-09-11T03:42:16.027 INFO:tasks.rados.rados.2.smithi175.stdout:1040: finishing write tid 1 to smithi17557870-40 2024-09-11T03:42:16.027 INFO:tasks.rados.rados.2.smithi175.stdout:1040: finishing write tid 2 to smithi17557870-40 2024-09-11T03:42:16.027 INFO:tasks.rados.rados.2.smithi175.stdout:1040: finishing write tid 3 to smithi17557870-40 2024-09-11T03:42:16.027 INFO:tasks.rados.rados.2.smithi175.stdout:1040: finishing write tid 4 to smithi17557870-40 2024-09-11T03:42:16.027 INFO:tasks.rados.rados.2.smithi175.stdout:1040: oid 40 updating version 0 to 348 2024-09-11T03:42:16.027 INFO:tasks.rados.rados.2.smithi175.stdout:1040: oid 40 version 348 is already newer than 346 2024-09-11T03:42:16.027 INFO:tasks.rados.rados.2.smithi175.stdout:1040: oid 40 version 348 is already newer than 347 2024-09-11T03:42:16.027 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 348 (ObjNum 372 snap 113 seq_num 372) dirty exists 2024-09-11T03:42:16.028 INFO:tasks.rados.rados.2.smithi175.stdout:1040: left oid 40 (ObjNum 372 snap 113 seq_num 372) 2024-09-11T03:42:16.063 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:16.087 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:16.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:16.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:16.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: pgmap v631: 57 pgs: 57 active+clean; 198 MiB data, 744 MiB used, 893 GiB / 894 GiB avail; 4.7 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2024-09-11T03:42:16.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: osdmap e318: 12 total, 12 up, 10 in 2024-09-11T03:42:16.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=3905478815,config=358384381,health=603722711,kv=1680469788}) 2024-09-11T03:42:16.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2983801092}) 2024-09-11T03:42:16.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=313122311}) 2024-09-11T03:42:16.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=3007818657,logm=545751517}) 2024-09-11T03:42:16.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3047322608}) 2024-09-11T03:42:16.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2243356999}) 2024-09-11T03:42:16.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2700514625}) 2024-09-11T03:42:16.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=158021889}) 2024-09-11T03:42:16.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1270182814}) 2024-09-11T03:42:16.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3973302446}) 2024-09-11T03:42:16.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1113950368}) 2024-09-11T03:42:16.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3261017605}) 2024-09-11T03:42:16.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3796288221}) 2024-09-11T03:42:16.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=105959214}) 2024-09-11T03:42:16.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2873019910}) 2024-09-11T03:42:16.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1328069412}) 2024-09-11T03:42:16.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1365018942}) 2024-09-11T03:42:16.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2595077145}) 2024-09-11T03:42:16.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1889773732}) 2024-09-11T03:42:16.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1746989816}) 2024-09-11T03:42:16.146 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1729221711}) 2024-09-11T03:42:16.147 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1884477471}) 2024-09-11T03:42:16.147 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=235564985}) 2024-09-11T03:42:16.147 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4003481475}) 2024-09-11T03:42:16.147 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3502470732}) 2024-09-11T03:42:16.147 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1768483374}) 2024-09-11T03:42:16.147 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=349679974}) 2024-09-11T03:42:16.147 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=958933000}) 2024-09-11T03:42:16.147 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1566422532}) 2024-09-11T03:42:16.147 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3130093375}) 2024-09-11T03:42:16.147 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1345571516}) 2024-09-11T03:42:16.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1362760159}) 2024-09-11T03:42:16.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2071618218}) 2024-09-11T03:42:16.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=10,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=63,mon_config_key=3} crc {logm=2238299690,mdsmap=827622067,mgr=2786812108,mgr_command_descs=899306875,mgr_metadata=2702562194,mgrstat=1541513686,mon_config_key=3950424102}) 2024-09-11T03:42:16.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3193399600}) 2024-09-11T03:42:16.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1635935487}) 2024-09-11T03:42:16.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=61,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osd_snap=18} crc {mon_config_key=3260283145,monmap=4220057526,nvmeofgw=598337686,osd_metadata=132375096,osd_pg_creating=1486795139,osd_snap=3704630831}) 2024-09-11T03:42:16.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=33,osdmap=67} crc {osd_snap=1762597422,osdmap=1761934212}) 2024-09-11T03:42:16.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3298071354}) 2024-09-11T03:42:16.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2163620905}) 2024-09-11T03:42:16.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3790492101}) 2024-09-11T03:42:16.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1348468520}) 2024-09-11T03:42:16.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2493839050}) 2024-09-11T03:42:16.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:15 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=72} crc {osdmap=1323846514}) 2024-09-11T03:42:16.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: pgmap v631: 57 pgs: 57 active+clean; 198 MiB data, 744 MiB used, 893 GiB / 894 GiB avail; 4.7 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2024-09-11T03:42:16.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: osdmap e318: 12 total, 12 up, 10 in 2024-09-11T03:42:16.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=3905478815,config=358384381,health=603722711,kv=1680469788}) 2024-09-11T03:42:16.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2983801092}) 2024-09-11T03:42:16.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=313122311}) 2024-09-11T03:42:16.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=3007818657,logm=545751517}) 2024-09-11T03:42:16.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3047322608}) 2024-09-11T03:42:16.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2243356999}) 2024-09-11T03:42:16.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2700514625}) 2024-09-11T03:42:16.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=158021889}) 2024-09-11T03:42:16.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1270182814}) 2024-09-11T03:42:16.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3973302446}) 2024-09-11T03:42:16.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1113950368}) 2024-09-11T03:42:16.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3261017605}) 2024-09-11T03:42:16.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3796288221}) 2024-09-11T03:42:16.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=105959214}) 2024-09-11T03:42:16.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2873019910}) 2024-09-11T03:42:16.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1328069412}) 2024-09-11T03:42:16.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1365018942}) 2024-09-11T03:42:16.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2595077145}) 2024-09-11T03:42:16.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1889773732}) 2024-09-11T03:42:16.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1746989816}) 2024-09-11T03:42:16.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1729221711}) 2024-09-11T03:42:16.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1884477471}) 2024-09-11T03:42:16.174 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=235564985}) 2024-09-11T03:42:16.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4003481475}) 2024-09-11T03:42:16.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3502470732}) 2024-09-11T03:42:16.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1768483374}) 2024-09-11T03:42:16.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=349679974}) 2024-09-11T03:42:16.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=958933000}) 2024-09-11T03:42:16.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1566422532}) 2024-09-11T03:42:16.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3130093375}) 2024-09-11T03:42:16.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1345571516}) 2024-09-11T03:42:16.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1362760159}) 2024-09-11T03:42:16.175 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2071618218}) 2024-09-11T03:42:16.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=10,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=63,mon_config_key=3} crc {logm=2238299690,mdsmap=827622067,mgr=2786812108,mgr_command_descs=899306875,mgr_metadata=2702562194,mgrstat=1541513686,mon_config_key=3950424102}) 2024-09-11T03:42:16.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3193399600}) 2024-09-11T03:42:16.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1635935487}) 2024-09-11T03:42:16.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=61,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osd_snap=18} crc {mon_config_key=3260283145,monmap=4220057526,nvmeofgw=598337686,osd_metadata=132375096,osd_pg_creating=1486795139,osd_snap=3704630831}) 2024-09-11T03:42:16.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=33,osdmap=67} crc {osd_snap=1762597422,osdmap=1761934212}) 2024-09-11T03:42:16.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3298071354}) 2024-09-11T03:42:16.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2163620905}) 2024-09-11T03:42:16.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3790492101}) 2024-09-11T03:42:16.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1348468520}) 2024-09-11T03:42:16.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2493839050}) 2024-09-11T03:42:16.176 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:15 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=72} crc {osdmap=1323846514}) 2024-09-11T03:42:16.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:16.210 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:16.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:16.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:16.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: pgmap v631: 57 pgs: 57 active+clean; 198 MiB data, 744 MiB used, 893 GiB / 894 GiB avail; 4.7 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2024-09-11T03:42:16.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: osdmap e318: 12 total, 12 up, 10 in 2024-09-11T03:42:16.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=3905478815,config=358384381,health=603722711,kv=1680469788}) 2024-09-11T03:42:16.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2983801092}) 2024-09-11T03:42:16.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=313122311}) 2024-09-11T03:42:16.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=3007818657,logm=545751517}) 2024-09-11T03:42:16.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3047322608}) 2024-09-11T03:42:16.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2243356999}) 2024-09-11T03:42:16.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2700514625}) 2024-09-11T03:42:16.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=158021889}) 2024-09-11T03:42:16.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1270182814}) 2024-09-11T03:42:16.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3973302446}) 2024-09-11T03:42:16.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1113950368}) 2024-09-11T03:42:16.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3261017605}) 2024-09-11T03:42:16.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3796288221}) 2024-09-11T03:42:16.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=105959214}) 2024-09-11T03:42:16.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2873019910}) 2024-09-11T03:42:16.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1328069412}) 2024-09-11T03:42:16.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1365018942}) 2024-09-11T03:42:16.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2595077145}) 2024-09-11T03:42:16.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1889773732}) 2024-09-11T03:42:16.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1746989816}) 2024-09-11T03:42:16.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1729221711}) 2024-09-11T03:42:16.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1884477471}) 2024-09-11T03:42:16.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=235564985}) 2024-09-11T03:42:16.230 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4003481475}) 2024-09-11T03:42:16.230 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3502470732}) 2024-09-11T03:42:16.230 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1768483374}) 2024-09-11T03:42:16.230 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=349679974}) 2024-09-11T03:42:16.230 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=958933000}) 2024-09-11T03:42:16.230 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1566422532}) 2024-09-11T03:42:16.230 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3130093375}) 2024-09-11T03:42:16.230 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1345571516}) 2024-09-11T03:42:16.230 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1362760159}) 2024-09-11T03:42:16.230 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2071618218}) 2024-09-11T03:42:16.230 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=10,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=63,mon_config_key=3} crc {logm=2238299690,mdsmap=827622067,mgr=2786812108,mgr_command_descs=899306875,mgr_metadata=2702562194,mgrstat=1541513686,mon_config_key=3950424102}) 2024-09-11T03:42:16.230 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3193399600}) 2024-09-11T03:42:16.230 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1635935487}) 2024-09-11T03:42:16.231 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=61,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osd_snap=18} crc {mon_config_key=3260283145,monmap=4220057526,nvmeofgw=598337686,osd_metadata=132375096,osd_pg_creating=1486795139,osd_snap=3704630831}) 2024-09-11T03:42:16.231 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=33,osdmap=67} crc {osd_snap=1762597422,osdmap=1761934212}) 2024-09-11T03:42:16.231 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3298071354}) 2024-09-11T03:42:16.231 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2163620905}) 2024-09-11T03:42:16.231 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3790492101}) 2024-09-11T03:42:16.231 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1348468520}) 2024-09-11T03:42:16.231 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2493839050}) 2024-09-11T03:42:16.231 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:15 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=72} crc {osdmap=1323846514}) 2024-09-11T03:42:16.311 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:16.332 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:16.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:16.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:16.434 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:16.457 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:16.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:16.458 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:16.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:42:16.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T03:42:16.558 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:16.580 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:16.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:16.581 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:16.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:16.708 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:16.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:16.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:16.721 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:16.810 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:16.832 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:16.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:16.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:16.865 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.11 to 1 (10000) 2024-09-11T03:42:16.871 INFO:tasks.rados.rados.2.smithi175.stdout:1033: done (7 left) 2024-09-11T03:42:16.871 INFO:tasks.rados.rados.2.smithi175.stdout:1034: done (6 left) 2024-09-11T03:42:16.872 INFO:tasks.rados.rados.2.smithi175.stdout:1035: done (5 left) 2024-09-11T03:42:16.872 INFO:tasks.rados.rados.2.smithi175.stdout:1036: done (4 left) 2024-09-11T03:42:16.872 INFO:tasks.rados.rados.2.smithi175.stdout:1038: done (3 left) 2024-09-11T03:42:16.872 INFO:tasks.rados.rados.2.smithi175.stdout:1039: done (2 left) 2024-09-11T03:42:16.872 INFO:tasks.rados.rados.2.smithi175.stdout:1040: done (1 left) 2024-09-11T03:42:16.872 INFO:tasks.rados.rados.2.smithi175.stdout:1041: done (0 left) 2024-09-11T03:42:16.872 INFO:tasks.rados.rados.2.smithi175.stdout:1042: snap_create 2024-09-11T03:42:16.934 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:16.957 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:16.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:17.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:17.081 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:17.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:17.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:16 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:42:17.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:16 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:42:17.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:16 smithi084 ceph-mon[27254]: osdmap e319: 12 total, 12 up, 10 in 2024-09-11T03:42:17.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:16 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4222215748' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:42:17.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:16 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4222215748' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:42:17.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:16 smithi084 ceph-mon[27254]: osdmap e320: 12 total, 12 up, 10 in 2024-09-11T03:42:17.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:16 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:42:17.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:16 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:42:17.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:16 smithi121 ceph-mon[30729]: osdmap e319: 12 total, 12 up, 10 in 2024-09-11T03:42:17.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:16 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4222215748' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:42:17.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:16 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4222215748' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:42:17.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:16 smithi121 ceph-mon[30729]: osdmap e320: 12 total, 12 up, 10 in 2024-09-11T03:42:17.182 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:17.204 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:17.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:17.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:16 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:42:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:16 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:42:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:16 smithi175 ceph-mon[30894]: osdmap e319: 12 total, 12 up, 10 in 2024-09-11T03:42:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:16 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4222215748' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:42:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:16 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4222215748' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:42:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:16 smithi175 ceph-mon[30894]: osdmap e320: 12 total, 12 up, 10 in 2024-09-11T03:42:17.306 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:17.330 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:17.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:17.331 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:17.432 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:17.455 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:17.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:17.456 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:17.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:17.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:42:17.583 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:17.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:17.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:17.609 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T03:42:17.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:42:17.684 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:17.710 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:17.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:17.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:17.812 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:17.837 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:42:17.838 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:42:17.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:17.839 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:17.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:17.859 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:42:17.859 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:42:17.859 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:17.873 INFO:tasks.rados.rados.2.smithi175.stdout:1042: done (0 left) 2024-09-11T03:42:17.873 INFO:tasks.rados.rados.2.smithi175.stdout:1043: snap_remove snap 112 2024-09-11T03:42:17.879 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:42:17.880 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:42:17.898 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:42:17.898 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:42:17.918 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:42:17.919 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:42:17.935 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:42:17.936 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:42:17.940 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:17.952 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:42:17.953 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:42:17.965 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:17.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:17.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:17.969 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:42:17.969 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:42:17.986 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:42:17.987 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:42:18.004 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:42:18.004 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:42:18.021 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:42:18.021 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:42:18.038 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:42:18.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:42:18.067 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:18.090 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:18.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:18.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:18.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:17 smithi084 ceph-mon[27254]: pgmap v634: 57 pgs: 57 active+clean; 203 MiB data, 756 MiB used, 893 GiB / 894 GiB avail; 6.4 MiB/s rd, 2.6 MiB/s wr, 14 op/s 2024-09-11T03:42:18.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/588804481' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:18.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:17 smithi084 ceph-mon[27254]: osdmap e321: 12 total, 12 up, 10 in 2024-09-11T03:42:18.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:17 smithi121 ceph-mon[30729]: pgmap v634: 57 pgs: 57 active+clean; 203 MiB data, 756 MiB used, 893 GiB / 894 GiB avail; 6.4 MiB/s rd, 2.6 MiB/s wr, 14 op/s 2024-09-11T03:42:18.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/588804481' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:18.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:17 smithi121 ceph-mon[30729]: osdmap e321: 12 total, 12 up, 10 in 2024-09-11T03:42:18.192 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:18.213 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:18.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:18.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:18.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:17 smithi175 ceph-mon[30894]: pgmap v634: 57 pgs: 57 active+clean; 203 MiB data, 756 MiB used, 893 GiB / 894 GiB avail; 6.4 MiB/s rd, 2.6 MiB/s wr, 14 op/s 2024-09-11T03:42:18.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/588804481' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:18.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:17 smithi175 ceph-mon[30894]: osdmap e321: 12 total, 12 up, 10 in 2024-09-11T03:42:18.315 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:18.339 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:18.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:18.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:18.441 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:18.465 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:18.466 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:18.477 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:42:18.477 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:42:18.477 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:42:18.477 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:42:18.567 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:18.593 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:18.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:18.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:18.695 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:18.764 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:18.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:18.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:18.866 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:18.875 INFO:tasks.rados.rados.2.smithi175.stdout:1043: done (0 left) 2024-09-11T03:42:18.875 INFO:tasks.rados.rados.2.smithi175.stdout:1044: rollback oid 26 current snap is 115 2024-09-11T03:42:18.875 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 26 to 113 2024-09-11T03:42:18.875 INFO:tasks.rados.rados.2.smithi175.stdout:1045: read oid 1 snap -1 2024-09-11T03:42:18.875 INFO:tasks.rados.rados.2.smithi175.stdout:1045: expect (ObjNum 314 snap 86 seq_num 314) 2024-09-11T03:42:18.875 INFO:tasks.rados.rados.2.smithi175.stdout:1046: snap_create 2024-09-11T03:42:18.876 INFO:tasks.rados.rados.2.smithi175.stdout:1044: finishing rollback tid 1 to smithi17557870-26 2024-09-11T03:42:18.876 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 165 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T03:42:18.885 INFO:tasks.rados.rados.2.smithi175.stdout:1045: expect (ObjNum 314 snap 86 seq_num 314) 2024-09-11T03:42:18.894 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:18.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:18.996 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:19.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:19.018 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:19.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:19.119 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:19.120 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:19.120 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:19.144 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:19.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:19.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:19.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:19.272 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:19.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:19.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:19.374 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:19.396 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:19.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:19.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:19.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:19.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T03:42:19.524 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:19.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:19.626 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:19.648 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:19.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:19.649 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:19.750 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:19.772 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:19.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:19.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:19.873 INFO:tasks.rados.rados.2.smithi175.stdout:1044: done (2 left) 2024-09-11T03:42:19.873 INFO:tasks.rados.rados.2.smithi175.stdout:1045: done (1 left) 2024-09-11T03:42:19.873 INFO:tasks.rados.rados.2.smithi175.stdout:1046: done (0 left) 2024-09-11T03:42:19.873 INFO:tasks.rados.rados.2.smithi175.stdout:1047: read oid 12 snap -1 2024-09-11T03:42:19.873 INFO:tasks.rados.rados.2.smithi175.stdout:1047: expect (ObjNum 335 snap 95 seq_num 335) 2024-09-11T03:42:19.874 INFO:tasks.rados.rados.2.smithi175.stdout:1048: write oid 42 current snap is 116 2024-09-11T03:42:19.874 INFO:tasks.rados.rados.2.smithi175.stdout:1048: seq_num 373 ranges {669011=569174,2010607=258404} 2024-09-11T03:42:19.874 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:19.880 INFO:tasks.rados.rados.2.smithi175.stdout:1048: writing smithi17557870-42 from 669011 to 1238185 tid 1 2024-09-11T03:42:19.887 INFO:tasks.rados.rados.2.smithi175.stdout:1048: writing smithi17557870-42 from 2010607 to 2269011 tid 2 2024-09-11T03:42:19.887 INFO:tasks.rados.rados.2.smithi175.stdout:1049: read oid 29 snap -1 2024-09-11T03:42:19.887 INFO:tasks.rados.rados.2.smithi175.stdout:1049: expect deleted 2024-09-11T03:42:19.887 INFO:tasks.rados.rados.2.smithi175.stdout:1050: copy_from oid 6 from oid 31 current snap is 116 2024-09-11T03:42:19.887 INFO:tasks.rados.rados.2.smithi175.stdout:1047: expect (ObjNum 335 snap 95 seq_num 335) 2024-09-11T03:42:19.898 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:19.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:19.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:20.000 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:20.021 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:20.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:20.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:20.097 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:19 smithi084 ceph-mon[27254]: pgmap v637: 57 pgs: 57 active+clean; 206 MiB data, 756 MiB used, 893 GiB / 894 GiB avail; 6.4 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-11T03:42:20.098 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:19 smithi084 ceph-mon[27254]: osdmap e322: 12 total, 12 up, 10 in 2024-09-11T03:42:20.105 INFO:tasks.rados.rados.2.smithi175.stdout:1048: finishing write tid 1 to smithi17557870-42 2024-09-11T03:42:20.105 INFO:tasks.rados.rados.2.smithi175.stdout:1047: done (3 left) 2024-09-11T03:42:20.105 INFO:tasks.rados.rados.2.smithi175.stdout:1051: read oid 1 snap -1 2024-09-11T03:42:20.106 INFO:tasks.rados.rados.2.smithi175.stdout:1051: expect (ObjNum 314 snap 86 seq_num 314) 2024-09-11T03:42:20.106 INFO:tasks.rados.rados.2.smithi175.stdout:1052: read oid 50 snap -1 2024-09-11T03:42:20.106 INFO:tasks.rados.rados.2.smithi175.stdout:1052: expect (ObjNum 336 snap 95 seq_num 336) 2024-09-11T03:42:20.106 INFO:tasks.rados.rados.2.smithi175.stdout:1053: read oid 48 snap -1 2024-09-11T03:42:20.106 INFO:tasks.rados.rados.2.smithi175.stdout:1053: expect deleted 2024-09-11T03:42:20.106 INFO:tasks.rados.rados.2.smithi175.stdout:1054: read oid 4 snap 91 2024-09-11T03:42:20.106 INFO:tasks.rados.rados.2.smithi175.stdout:1054: expect deleted 2024-09-11T03:42:20.106 INFO:tasks.rados.rados.2.smithi175.stdout:1055: read oid 10 snap -1 2024-09-11T03:42:20.106 INFO:tasks.rados.rados.2.smithi175.stdout:1055: expect (ObjNum 310 snap 85 seq_num 310) 2024-09-11T03:42:20.106 INFO:tasks.rados.rados.2.smithi175.stdout:1056: snap_create 2024-09-11T03:42:20.106 INFO:tasks.rados.rados.2.smithi175.stdout:1048: finishing write tid 2 to smithi17557870-42 2024-09-11T03:42:20.106 INFO:tasks.rados.rados.2.smithi175.stdout:1048: finishing write tid 3 to smithi17557870-42 2024-09-11T03:42:20.106 INFO:tasks.rados.rados.2.smithi175.stdout:1048: finishing write tid 4 to smithi17557870-42 2024-09-11T03:42:20.106 INFO:tasks.rados.rados.2.smithi175.stdout:1048: oid 42 updating version 0 to 268 2024-09-11T03:42:20.106 INFO:tasks.rados.rados.2.smithi175.stdout:1048: oid 42 version 268 is already newer than 267 2024-09-11T03:42:20.107 INFO:tasks.rados.rados.2.smithi175.stdout:1048: oid 42 updating version 268 to 269 2024-09-11T03:42:20.107 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 269 (ObjNum 373 snap 116 seq_num 373) dirty exists 2024-09-11T03:42:20.107 INFO:tasks.rados.rados.2.smithi175.stdout:1048: left oid 42 (ObjNum 373 snap 116 seq_num 373) 2024-09-11T03:42:20.107 INFO:tasks.rados.rados.2.smithi175.stdout:1050: finishing copy_from racing read to smithi17557870-6 2024-09-11T03:42:20.107 INFO:tasks.rados.rados.2.smithi175.stdout:1050: finishing copy_from to smithi17557870-6 2024-09-11T03:42:20.107 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 350 (ObjNum 362 snap 107 seq_num 362) dirty exists 2024-09-11T03:42:20.108 INFO:tasks.rados.rados.2.smithi175.stdout:1051: expect (ObjNum 314 snap 86 seq_num 314) 2024-09-11T03:42:20.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:20.146 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:20.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:20.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:20.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:19 smithi121 ceph-mon[30729]: pgmap v637: 57 pgs: 57 active+clean; 206 MiB data, 756 MiB used, 893 GiB / 894 GiB avail; 6.4 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-11T03:42:20.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:19 smithi121 ceph-mon[30729]: osdmap e322: 12 total, 12 up, 10 in 2024-09-11T03:42:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:19 smithi175 ceph-mon[30894]: pgmap v637: 57 pgs: 57 active+clean; 206 MiB data, 756 MiB used, 893 GiB / 894 GiB avail; 6.4 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-11T03:42:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:19 smithi175 ceph-mon[30894]: osdmap e322: 12 total, 12 up, 10 in 2024-09-11T03:42:20.231 INFO:tasks.rados.rados.2.smithi175.stdout:1052: expect (ObjNum 336 snap 95 seq_num 336) 2024-09-11T03:42:20.248 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:20.270 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:20.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:20.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:20.349 INFO:tasks.rados.rados.2.smithi175.stdout:1055: expect (ObjNum 310 snap 85 seq_num 310) 2024-09-11T03:42:20.363 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:42:20.364 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:42:20.371 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:20.392 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:20.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:20.393 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:20.494 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:20.515 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:20.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:20.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:20.617 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:20.638 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:20.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:20.640 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:20.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:20.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:20.765 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:20.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:20.866 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:20.869 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:42:20.880 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:42:20.887 INFO:tasks.rados.rados.2.smithi175.stdout:1048: done (8 left) 2024-09-11T03:42:20.887 INFO:tasks.rados.rados.2.smithi175.stdout:1049: done (7 left) 2024-09-11T03:42:20.887 INFO:tasks.rados.rados.2.smithi175.stdout:1050: done (6 left) 2024-09-11T03:42:20.887 INFO:tasks.rados.rados.2.smithi175.stdout:1051: done (5 left) 2024-09-11T03:42:20.888 INFO:tasks.rados.rados.2.smithi175.stdout:1052: done (4 left) 2024-09-11T03:42:20.888 INFO:tasks.rados.rados.2.smithi175.stdout:1053: done (3 left) 2024-09-11T03:42:20.888 INFO:tasks.rados.rados.2.smithi175.stdout:1054: done (2 left) 2024-09-11T03:42:20.888 INFO:tasks.rados.rados.2.smithi175.stdout:1055: done (1 left) 2024-09-11T03:42:20.888 INFO:tasks.rados.rados.2.smithi175.stdout:1056: done (0 left) 2024-09-11T03:42:20.888 INFO:tasks.rados.rados.2.smithi175.stdout:1057: read oid 22 snap 114 2024-09-11T03:42:20.888 INFO:tasks.rados.rados.2.smithi175.stdout:1057: expect (ObjNum 342 snap 99 seq_num 342) 2024-09-11T03:42:20.888 INFO:tasks.rados.rados.2.smithi175.stdout:1058: snap_remove snap 109 2024-09-11T03:42:20.891 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:20.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:20.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:20.993 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:21.002 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:20 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:42:21.002 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:20 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:42:21.002 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:20 smithi084 ceph-mon[27254]: osdmap e323: 12 total, 12 up, 10 in 2024-09-11T03:42:21.002 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:20 smithi084 ceph-mon[27254]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:42:21.002 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:20 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/222550303' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:42:21.015 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:21.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:21.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:21.034 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":639,"stamp":"2024-09-11T03:42:19.870841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":215985795,"num_objects":104,"num_object_clones":56,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":5,"num_read":872,"num_read_kb":723502,"num_write":702,"num_write_kb":294864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":10857193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8646,"ondisk_log_size":8646,"up":171,"acting":171,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":171,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":774072,"kb_used_data":424704,"kb_used_omap":15,"kb_used_meta":349104,"kb_avail":936677448,"statfs":{"total":959950356480,"available":959157706752,"internally_reserved":0,"allocated":434896896,"data_stored":561312077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":357482933},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7896880,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":-2,"num_read":31,"num_read_kb":26399,"num_write":23,"num_write_kb":5064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.974616"},"pg_stats":[{"pgid":"1.1e","version":"20'92","reported_seq":212,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.901613+0000","last_change":"2024-09-11T03:40:40.236609+0000","last_active":"2024-09-11T03:42:16.901613+0000","last_peered":"2024-09-11T03:42:16.901613+0000","last_clean":"2024-09-11T03:42:16.901613+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:42:16.901613+0000","last_undegraded":"2024-09-11T03:42:16.901613+0000","last_fullsized":"2024-09-11T03:42:16.901613+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.236546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.236546+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:09.146128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'92","reported_seq":212,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.124380+0000","last_change":"2024-09-11T03:40:41.348889+0000","last_active":"2024-09-11T03:42:16.124380+0000","last_peered":"2024-09-11T03:42:16.124380+0000","last_clean":"2024-09-11T03:42:16.124380+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:42:16.124380+0000","last_undegraded":"2024-09-11T03:42:16.124380+0000","last_fullsized":"2024-09-11T03:42:16.124380+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:41.348828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:41.348828+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:11.089042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'92","reported_seq":212,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:15.897380+0000","last_change":"2024-09-11T03:40:40.188174+0000","last_active":"2024-09-11T03:42:15.897380+0000","last_peered":"2024-09-11T03:42:15.897380+0000","last_clean":"2024-09-11T03:42:15.897380+0000","last_became_active":"2024-09-11T03:40:39.848230+0000","last_became_peered":"2024-09-11T03:40:39.848230+0000","last_unstale":"2024-09-11T03:42:15.897380+0000","last_undegraded":"2024-09-11T03:42:15.897380+0000","last_fullsized":"2024-09-11T03:42:15.897380+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.188124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.188124+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:08.246084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"319'214","reported_seq":1415,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:18.135392+0000","last_change":"2024-09-11T03:42:15.903530+0000","last_active":"2024-09-11T03:42:18.135392+0000","last_peered":"2024-09-11T03:42:18.135392+0000","last_clean":"2024-09-11T03:42:18.135392+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:42:18.135392+0000","last_undegraded":"2024-09-11T03:42:18.135392+0000","last_fullsized":"2024-09-11T03:42:18.135392+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"280'196","last_scrub_stamp":"2024-09-11T03:41:40.336904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.336904+0000","objects_scrubbed":2,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:49.342935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042926650000000002,"stat_sum":{"num_bytes":11119314,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":20483,"num_write":5,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4865740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.1b","version":"20'92","reported_seq":212,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:15.897262+0000","last_change":"2024-09-11T03:40:42.166354+0000","last_active":"2024-09-11T03:42:15.897262+0000","last_peered":"2024-09-11T03:42:15.897262+0000","last_clean":"2024-09-11T03:42:15.897262+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:42:15.897262+0000","last_undegraded":"2024-09-11T03:42:15.897262+0000","last_fullsized":"2024-09-11T03:42:15.897262+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:42.166295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:42.166295+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:13.230054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.19","version":"318'345","reported_seq":1552,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:17.871745+0000","last_change":"2024-09-11T03:42:15.899756+0000","last_active":"2024-09-11T03:42:17.871745+0000","last_peered":"2024-09-11T03:42:17.871745+0000","last_clean":"2024-09-11T03:42:17.871745+0000","last_became_active":"2024-09-11T03:41:38.886787+0000","last_became_peered":"2024-09-11T03:41:38.886787+0000","last_unstale":"2024-09-11T03:42:17.871745+0000","last_undegraded":"2024-09-11T03:42:17.871745+0000","last_fullsized":"2024-09-11T03:42:17.871745+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'333","last_scrub_stamp":"2024-09-11T03:41:40.379141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.379141+0000","objects_scrubbed":1,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:52.170720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00063786500000000005,"stat_sum":{"num_bytes":4270678,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.1a","version":"20'92","reported_seq":452,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-11T03:42:14.894994+0000","last_change":"2024-09-11T03:40:40.278752+0000","last_active":"2024-09-11T03:42:14.894994+0000","last_peered":"2024-09-11T03:42:14.894994+0000","last_clean":"2024-09-11T03:42:14.894994+0000","last_became_active":"2024-09-11T03:40:39.848781+0000","last_became_peered":"2024-09-11T03:40:39.848781+0000","last_unstale":"2024-09-11T03:42:14.894994+0000","last_undegraded":"2024-09-11T03:42:14.894994+0000","last_fullsized":"2024-09-11T03:42:14.894994+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.278703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.278703+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:09.913902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"20'92","reported_seq":212,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:15.904247+0000","last_change":"2024-09-11T03:40:40.718553+0000","last_active":"2024-09-11T03:42:15.904247+0000","last_peered":"2024-09-11T03:42:15.904247+0000","last_clean":"2024-09-11T03:42:15.904247+0000","last_became_active":"2024-09-11T03:40:39.851698+0000","last_became_peered":"2024-09-11T03:40:39.851698+0000","last_unstale":"2024-09-11T03:42:15.904247+0000","last_undegraded":"2024-09-11T03:42:15.904247+0000","last_fullsized":"2024-09-11T03:42:15.904247+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.718503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.718503+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:13.733914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.18","version":"20'92","reported_seq":334,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.160144+0000","last_change":"2024-09-11T03:42:02.103343+0000","last_active":"2024-09-11T03:42:16.160144+0000","last_peered":"2024-09-11T03:42:16.160144+0000","last_clean":"2024-09-11T03:42:16.160144+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:42:16.160144+0000","last_undegraded":"2024-09-11T03:42:16.160144+0000","last_fullsized":"2024-09-11T03:42:16.160144+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:02.103292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:02.103292+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:29.586019+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"274'162","reported_seq":1318,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:18.135923+0000","last_change":"2024-09-11T03:42:15.899999+0000","last_active":"2024-09-11T03:42:18.135923+0000","last_peered":"2024-09-11T03:42:18.135923+0000","last_clean":"2024-09-11T03:42:18.135923+0000","last_became_active":"2024-09-11T03:41:38.890690+0000","last_became_peered":"2024-09-11T03:41:38.890690+0000","last_unstale":"2024-09-11T03:42:18.135923+0000","last_undegraded":"2024-09-11T03:42:18.135923+0000","last_fullsized":"2024-09-11T03:42:18.135923+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'162","last_scrub_stamp":"2024-09-11T03:41:42.270480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:42.270480+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:54.988485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010241740000000001,"stat_sum":{"num_bytes":3327819,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[9,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":220,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:15.897603+0000","last_change":"2024-09-11T03:41:57.260720+0000","last_active":"2024-09-11T03:42:15.897603+0000","last_peered":"2024-09-11T03:42:15.897603+0000","last_clean":"2024-09-11T03:42:15.897603+0000","last_became_active":"2024-09-11T03:40:39.850243+0000","last_became_peered":"2024-09-11T03:40:39.850243+0000","last_unstale":"2024-09-11T03:42:15.897603+0000","last_undegraded":"2024-09-11T03:42:15.897603+0000","last_fullsized":"2024-09-11T03:42:15.897603+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:57.260664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:57.260664+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:11.818969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":1448,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:18.136252+0000","last_change":"2024-09-11T03:42:15.900459+0000","last_active":"2024-09-11T03:42:18.136252+0000","last_peered":"2024-09-11T03:42:18.136252+0000","last_clean":"2024-09-11T03:42:18.136252+0000","last_became_active":"2024-09-11T03:41:38.888228+0000","last_became_peered":"2024-09-11T03:41:38.888228+0000","last_unstale":"2024-09-11T03:42:18.136252+0000","last_undegraded":"2024-09-11T03:42:18.136252+0000","last_fullsized":"2024-09-11T03:42:18.136252+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:41:43.239726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:43.239726+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:56.211264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001295441,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":218,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.124426+0000","last_change":"2024-09-11T03:42:05.102506+0000","last_active":"2024-09-11T03:42:16.124426+0000","last_peered":"2024-09-11T03:42:16.124426+0000","last_clean":"2024-09-11T03:42:16.124426+0000","last_became_active":"2024-09-11T03:40:39.849950+0000","last_became_peered":"2024-09-11T03:40:39.849950+0000","last_unstale":"2024-09-11T03:42:16.124426+0000","last_undegraded":"2024-09-11T03:42:16.124426+0000","last_fullsized":"2024-09-11T03:42:16.124426+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:05.102457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:05.102457+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:18.344056+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"318'191","reported_seq":1314,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.161075+0000","last_change":"2024-09-11T03:42:16.161558+0000","last_active":"2024-09-11T03:42:16.161075+0000","last_peered":"2024-09-11T03:42:16.161075+0000","last_clean":"2024-09-11T03:42:16.161075+0000","last_became_active":"2024-09-11T03:41:38.887856+0000","last_became_peered":"2024-09-11T03:41:38.887856+0000","last_unstale":"2024-09-11T03:42:16.161075+0000","last_undegraded":"2024-09-11T03:42:16.161075+0000","last_fullsized":"2024-09-11T03:42:16.161075+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:41:42.059696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:42.059696+0000","objects_scrubbed":1,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:07.910515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010662169999999999,"stat_sum":{"num_bytes":6749143,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":1870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,9],"acting":[5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"58","length":"1"},{"start":"63","length":"1"},{"start":"6a","length":"2"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":211,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:15.903013+0000","last_change":"2024-09-11T03:40:39.928143+0000","last_active":"2024-09-11T03:42:15.903013+0000","last_peered":"2024-09-11T03:42:15.903013+0000","last_clean":"2024-09-11T03:42:15.903013+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:42:15.903013+0000","last_undegraded":"2024-09-11T03:42:15.903013+0000","last_fullsized":"2024-09-11T03:42:15.903013+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.928088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.928088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:08.756769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":1526,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.902571+0000","last_change":"2024-09-11T03:42:15.898195+0000","last_active":"2024-09-11T03:42:16.902571+0000","last_peered":"2024-09-11T03:42:16.902571+0000","last_clean":"2024-09-11T03:42:16.902571+0000","last_became_active":"2024-09-11T03:41:38.885486+0000","last_became_peered":"2024-09-11T03:41:38.885486+0000","last_unstale":"2024-09-11T03:42:16.902571+0000","last_undegraded":"2024-09-11T03:42:16.902571+0000","last_fullsized":"2024-09-11T03:42:16.902571+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:41:41.456545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.456545+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:00.230354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092932900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":244,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.902242+0000","last_change":"2024-09-11T03:40:35.195938+0000","last_active":"2024-09-11T03:42:16.902242+0000","last_peered":"2024-09-11T03:42:16.902242+0000","last_clean":"2024-09-11T03:42:16.902242+0000","last_became_active":"2024-09-11T03:40:27.847297+0000","last_became_peered":"2024-09-11T03:40:27.847297+0000","last_unstale":"2024-09-11T03:42:16.902242+0000","last_undegraded":"2024-09-11T03:42:16.902242+0000","last_fullsized":"2024-09-11T03:42:16.902242+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.195893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.195893+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:50.659810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":233,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:15.897588+0000","last_change":"2024-09-11T03:42:15.897959+0000","last_active":"2024-09-11T03:42:15.897588+0000","last_peered":"2024-09-11T03:42:15.897588+0000","last_clean":"2024-09-11T03:42:15.897588+0000","last_became_active":"2024-09-11T03:41:35.874295+0000","last_became_peered":"2024-09-11T03:41:35.874295+0000","last_unstale":"2024-09-11T03:42:15.897588+0000","last_undegraded":"2024-09-11T03:42:15.897588+0000","last_fullsized":"2024-09-11T03:42:15.897588+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:41:40.182948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.182948+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:02.849008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000621639,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"58","length":"1"},{"start":"63","length":"1"},{"start":"6a","length":"2"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.13","version":"20'92","reported_seq":327,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.124211+0000","last_change":"2024-09-11T03:41:45.086774+0000","last_active":"2024-09-11T03:42:16.124211+0000","last_peered":"2024-09-11T03:42:16.124211+0000","last_clean":"2024-09-11T03:42:16.124211+0000","last_became_active":"2024-09-11T03:40:27.847557+0000","last_became_peered":"2024-09-11T03:40:27.847557+0000","last_unstale":"2024-09-11T03:42:16.124211+0000","last_undegraded":"2024-09-11T03:42:16.124211+0000","last_fullsized":"2024-09-11T03:42:16.124211+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:45.086692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:45.086692+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:48.005883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":1219,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-11T03:42:14.895325+0000","last_change":"2024-09-11T03:42:14.895598+0000","last_active":"2024-09-11T03:42:14.895325+0000","last_peered":"2024-09-11T03:42:14.895325+0000","last_clean":"2024-09-11T03:42:14.895325+0000","last_became_active":"2024-09-11T03:41:38.886813+0000","last_became_peered":"2024-09-11T03:41:38.886813+0000","last_unstale":"2024-09-11T03:42:14.895325+0000","last_undegraded":"2024-09-11T03:42:14.895325+0000","last_fullsized":"2024-09-11T03:42:14.895325+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:41:40.077940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.077940+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:03.771186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052595599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,9],"acting":[4,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"58","length":"1"},{"start":"63","length":"1"},{"start":"6a","length":"2"},{"start":"6e","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":239,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.160214+0000","last_change":"2024-09-11T03:40:37.284871+0000","last_active":"2024-09-11T03:42:16.160214+0000","last_peered":"2024-09-11T03:42:16.160214+0000","last_clean":"2024-09-11T03:42:16.160214+0000","last_became_active":"2024-09-11T03:40:27.845249+0000","last_became_peered":"2024-09-11T03:40:27.845249+0000","last_unstale":"2024-09-11T03:42:16.160214+0000","last_undegraded":"2024-09-11T03:42:16.160214+0000","last_fullsized":"2024-09-11T03:42:16.160214+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:37.284800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:37.284800+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:55.054741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"319'145","reported_seq":1297,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:17.871874+0000","last_change":"2024-09-11T03:42:15.899914+0000","last_active":"2024-09-11T03:42:17.871874+0000","last_peered":"2024-09-11T03:42:17.871874+0000","last_clean":"2024-09-11T03:42:17.871874+0000","last_became_active":"2024-09-11T03:42:09.873469+0000","last_became_peered":"2024-09-11T03:42:09.873469+0000","last_unstale":"2024-09-11T03:42:17.871874+0000","last_undegraded":"2024-09-11T03:42:17.871874+0000","last_fullsized":"2024-09-11T03:42:17.871874+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"283'124","last_scrub_stamp":"2024-09-11T03:41:41.419995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.419995+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:43.642854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081549300000000003,"stat_sum":{"num_bytes":11430617,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":10807,"num_write":15,"num_write_kb":9128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":5991453,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":244,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:15.904318+0000","last_change":"2024-09-11T03:40:35.703292+0000","last_active":"2024-09-11T03:42:15.904318+0000","last_peered":"2024-09-11T03:42:15.904318+0000","last_clean":"2024-09-11T03:42:15.904318+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:42:15.904318+0000","last_undegraded":"2024-09-11T03:42:15.904318+0000","last_fullsized":"2024-09-11T03:42:15.904318+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.703203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.703203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:49.677961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":1235,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:18.135791+0000","last_change":"2024-09-11T03:42:15.899984+0000","last_active":"2024-09-11T03:42:18.135791+0000","last_peered":"2024-09-11T03:42:18.135791+0000","last_clean":"2024-09-11T03:42:18.135791+0000","last_became_active":"2024-09-11T03:41:38.886834+0000","last_became_peered":"2024-09-11T03:41:38.886834+0000","last_unstale":"2024-09-11T03:42:18.135791+0000","last_undegraded":"2024-09-11T03:42:18.135791+0000","last_fullsized":"2024-09-11T03:42:18.135791+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:41:41.300112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.300112+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:01.963682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011230770000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":247,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.160176+0000","last_change":"2024-09-11T03:42:03.151379+0000","last_active":"2024-09-11T03:42:16.160176+0000","last_peered":"2024-09-11T03:42:16.160176+0000","last_clean":"2024-09-11T03:42:16.160176+0000","last_became_active":"2024-09-11T03:40:27.844057+0000","last_became_peered":"2024-09-11T03:40:27.844057+0000","last_unstale":"2024-09-11T03:42:16.160176+0000","last_undegraded":"2024-09-11T03:42:16.160176+0000","last_fullsized":"2024-09-11T03:42:16.160176+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:03.151323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:03.151323+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:05.151390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"318'305","reported_seq":1413,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:15.897712+0000","last_change":"2024-09-11T03:42:15.897998+0000","last_active":"2024-09-11T03:42:15.897712+0000","last_peered":"2024-09-11T03:42:15.897712+0000","last_clean":"2024-09-11T03:42:15.897712+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:42:15.897712+0000","last_undegraded":"2024-09-11T03:42:15.897712+0000","last_fullsized":"2024-09-11T03:42:15.897712+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'232","last_scrub_stamp":"2024-09-11T03:41:04.001653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:04.001653+0000","objects_scrubbed":10,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:27.360099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00041804400000000001,"stat_sum":{"num_bytes":10941489,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":60,"num_read_kb":26483,"num_write":102,"num_write_kb":52867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"58","length":"1"},{"start":"63","length":"1"},{"start":"6a","length":"2"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":324,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.160289+0000","last_change":"2024-09-11T03:42:05.135033+0000","last_active":"2024-09-11T03:42:16.160289+0000","last_peered":"2024-09-11T03:42:16.160289+0000","last_clean":"2024-09-11T03:42:16.160289+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:42:16.160289+0000","last_undegraded":"2024-09-11T03:42:16.160289+0000","last_fullsized":"2024-09-11T03:42:16.160289+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:05.134977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:05.134977+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:05.203169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"318'101","reported_seq":1227,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.901276+0000","last_change":"2024-09-11T03:42:16.069744+0000","last_active":"2024-09-11T03:42:16.901276+0000","last_peered":"2024-09-11T03:42:16.901276+0000","last_clean":"2024-09-11T03:42:16.901276+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:42:16.901276+0000","last_undegraded":"2024-09-11T03:42:16.901276+0000","last_fullsized":"2024-09-11T03:42:16.901276+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'57","last_scrub_stamp":"2024-09-11T03:40:59.151899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:59.151899+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00037710100000000002,"stat_sum":{"num_bytes":4776703,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":68051,"num_write":35,"num_write_kb":16778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":649,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:15.902523+0000","last_change":"2024-09-11T03:42:00.816905+0000","last_active":"2024-09-11T03:42:15.902523+0000","last_peered":"2024-09-11T03:42:15.902523+0000","last_clean":"2024-09-11T03:42:15.902523+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:42:15.902523+0000","last_undegraded":"2024-09-11T03:42:15.902523+0000","last_fullsized":"2024-09-11T03:42:15.902523+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:00.816850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:00.816850+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:09.071150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"319'399","reported_seq":1609,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.126817+0000","last_change":"2024-09-11T03:42:16.126970+0000","last_active":"2024-09-11T03:42:16.126817+0000","last_peered":"2024-09-11T03:42:16.126817+0000","last_clean":"2024-09-11T03:42:16.126817+0000","last_became_active":"2024-09-11T03:41:33.865638+0000","last_became_peered":"2024-09-11T03:41:33.865638+0000","last_unstale":"2024-09-11T03:42:16.126817+0000","last_undegraded":"2024-09-11T03:42:16.126817+0000","last_fullsized":"2024-09-11T03:42:16.126817+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'337","last_scrub_stamp":"2024-09-11T03:41:40.102984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.102984+0000","objects_scrubbed":5,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:52.264457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023639450000000001,"stat_sum":{"num_bytes":24271967,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":56,"num_read_kb":69583,"num_write":17,"num_write_kb":7206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"58","length":"1"},{"start":"63","length":"1"},{"start":"6a","length":"2"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":566,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:18.135655+0000","last_change":"2024-09-11T03:41:45.227630+0000","last_active":"2024-09-11T03:42:18.135655+0000","last_peered":"2024-09-11T03:42:18.135655+0000","last_clean":"2024-09-11T03:42:18.135655+0000","last_became_active":"2024-09-11T03:40:37.849988+0000","last_became_peered":"2024-09-11T03:40:37.849988+0000","last_unstale":"2024-09-11T03:42:18.135655+0000","last_undegraded":"2024-09-11T03:42:18.135655+0000","last_fullsized":"2024-09-11T03:42:18.135655+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:45.227581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:45.227581+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:03.066353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"319'218","reported_seq":1366,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:15.904161+0000","last_change":"2024-09-11T03:42:15.904353+0000","last_active":"2024-09-11T03:42:15.904161+0000","last_peered":"2024-09-11T03:42:15.904161+0000","last_clean":"2024-09-11T03:42:15.904161+0000","last_became_active":"2024-09-11T03:41:33.863875+0000","last_became_peered":"2024-09-11T03:41:33.863875+0000","last_unstale":"2024-09-11T03:42:15.904161+0000","last_undegraded":"2024-09-11T03:42:15.904161+0000","last_fullsized":"2024-09-11T03:42:15.904161+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'192","last_scrub_stamp":"2024-09-11T03:41:40.877628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.877628+0000","objects_scrubbed":1,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:01.752629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001779194,"stat_sum":{"num_bytes":2386627,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"58","length":"1"},{"start":"63","length":"1"},{"start":"6a","length":"2"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":256,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.901228+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:42:16.901228+0000","last_peered":"2024-09-11T03:42:16.901228+0000","last_clean":"2024-09-11T03:42:16.901228+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:42:16.901228+0000","last_undegraded":"2024-09-11T03:42:16.901228+0000","last_fullsized":"2024-09-11T03:42:16.901228+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:54.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"319'372","reported_seq":1592,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.903017+0000","last_change":"2024-09-11T03:42:15.902632+0000","last_active":"2024-09-11T03:42:16.903017+0000","last_peered":"2024-09-11T03:42:16.903017+0000","last_clean":"2024-09-11T03:42:16.903017+0000","last_became_active":"2024-09-11T03:41:33.867674+0000","last_became_peered":"2024-09-11T03:41:33.867674+0000","last_unstale":"2024-09-11T03:42:16.903017+0000","last_undegraded":"2024-09-11T03:42:16.903017+0000","last_fullsized":"2024-09-11T03:42:16.903017+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'331","last_scrub_stamp":"2024-09-11T03:41:42.500501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:42.500501+0000","objects_scrubbed":9,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:45.486882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00538954,"stat_sum":{"num_bytes":23855551,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":28,"num_read_kb":31252,"num_write":10,"num_write_kb":5112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":423,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.901271+0000","last_change":"2024-09-11T03:42:00.181197+0000","last_active":"2024-09-11T03:42:16.901271+0000","last_peered":"2024-09-11T03:42:16.901271+0000","last_clean":"2024-09-11T03:42:16.901271+0000","last_became_active":"2024-09-11T03:40:25.858281+0000","last_became_peered":"2024-09-11T03:40:25.858281+0000","last_unstale":"2024-09-11T03:42:16.901271+0000","last_undegraded":"2024-09-11T03:42:16.901271+0000","last_fullsized":"2024-09-11T03:42:16.901271+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:00.181139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:00.181139+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:00.294383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":1296,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.124762+0000","last_change":"2024-09-11T03:42:16.125101+0000","last_active":"2024-09-11T03:42:16.124762+0000","last_peered":"2024-09-11T03:42:16.124762+0000","last_clean":"2024-09-11T03:42:16.124762+0000","last_became_active":"2024-09-11T03:41:33.865499+0000","last_became_peered":"2024-09-11T03:41:33.865499+0000","last_unstale":"2024-09-11T03:42:16.124762+0000","last_undegraded":"2024-09-11T03:42:16.124762+0000","last_fullsized":"2024-09-11T03:42:16.124762+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:41:42.104989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:42.104989+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:51.322994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053919199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"58","length":"1"},{"start":"63","length":"1"},{"start":"6a","length":"2"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":323,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.902099+0000","last_change":"2024-09-11T03:40:38.224278+0000","last_active":"2024-09-11T03:42:16.902099+0000","last_peered":"2024-09-11T03:42:16.902099+0000","last_clean":"2024-09-11T03:42:16.902099+0000","last_became_active":"2024-09-11T03:40:37.851080+0000","last_became_peered":"2024-09-11T03:40:37.851080+0000","last_unstale":"2024-09-11T03:42:16.902099+0000","last_undegraded":"2024-09-11T03:42:16.902099+0000","last_fullsized":"2024-09-11T03:42:16.902099+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.224231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.224231+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:09.894123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"318'369","reported_seq":1555,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:15.897987+0000","last_change":"2024-09-11T03:42:15.898788+0000","last_active":"2024-09-11T03:42:15.897987+0000","last_peered":"2024-09-11T03:42:15.897987+0000","last_clean":"2024-09-11T03:42:15.897987+0000","last_became_active":"2024-09-11T03:41:33.866316+0000","last_became_peered":"2024-09-11T03:41:33.866316+0000","last_unstale":"2024-09-11T03:42:15.897987+0000","last_undegraded":"2024-09-11T03:42:15.897987+0000","last_fullsized":"2024-09-11T03:42:15.897987+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'292","last_scrub_stamp":"2024-09-11T03:41:41.222192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.222192+0000","objects_scrubbed":9,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:06.346156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012738350000000001,"stat_sum":{"num_bytes":23924251,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":14,"num_read_kb":11825,"num_write":10,"num_write_kb":7255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"58","length":"1"},{"start":"63","length":"1"},{"start":"6a","length":"2"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":569,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:18.136012+0000","last_change":"2024-09-11T03:42:05.222525+0000","last_active":"2024-09-11T03:42:18.136012+0000","last_peered":"2024-09-11T03:42:18.136012+0000","last_clean":"2024-09-11T03:42:18.136012+0000","last_became_active":"2024-09-11T03:40:37.848794+0000","last_became_peered":"2024-09-11T03:40:37.848794+0000","last_unstale":"2024-09-11T03:42:18.136012+0000","last_undegraded":"2024-09-11T03:42:18.136012+0000","last_fullsized":"2024-09-11T03:42:18.136012+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:05.222462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:05.222462+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:33.609855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"318'173","reported_seq":1347,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.124644+0000","last_change":"2024-09-11T03:42:16.125014+0000","last_active":"2024-09-11T03:42:16.124644+0000","last_peered":"2024-09-11T03:42:16.124644+0000","last_clean":"2024-09-11T03:42:16.124644+0000","last_became_active":"2024-09-11T03:41:33.869119+0000","last_became_peered":"2024-09-11T03:41:33.869119+0000","last_unstale":"2024-09-11T03:42:16.124644+0000","last_undegraded":"2024-09-11T03:42:16.124644+0000","last_fullsized":"2024-09-11T03:42:16.124644+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'162","last_scrub_stamp":"2024-09-11T03:41:41.141743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.141743+0000","objects_scrubbed":2,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:02.786854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055043099999999999,"stat_sum":{"num_bytes":6252270,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":17,"num_read_kb":21092,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"58","length":"1"},{"start":"63","length":"1"},{"start":"6a","length":"2"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":60,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:17.871672+0000","last_change":"2024-09-11T03:41:58.387920+0000","last_active":"2024-09-11T03:42:17.871672+0000","last_peered":"2024-09-11T03:42:17.871672+0000","last_clean":"2024-09-11T03:42:17.871672+0000","last_became_active":"2024-09-11T03:41:57.865927+0000","last_became_peered":"2024-09-11T03:41:57.865927+0000","last_unstale":"2024-09-11T03:42:17.871672+0000","last_undegraded":"2024-09-11T03:42:17.871672+0000","last_fullsized":"2024-09-11T03:42:17.871672+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:58.387873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:58.387873+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:03.728844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"310'137","reported_seq":1305,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.160711+0000","last_change":"2024-09-11T03:42:16.161011+0000","last_active":"2024-09-11T03:42:16.160711+0000","last_peered":"2024-09-11T03:42:16.160711+0000","last_clean":"2024-09-11T03:42:16.160711+0000","last_became_active":"2024-09-11T03:41:33.861182+0000","last_became_peered":"2024-09-11T03:41:33.861182+0000","last_unstale":"2024-09-11T03:42:16.160711+0000","last_undegraded":"2024-09-11T03:42:16.160711+0000","last_fullsized":"2024-09-11T03:42:16.160711+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'123","last_scrub_stamp":"2024-09-11T03:41:40.064448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.064448+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:54.051502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00057606499999999995,"stat_sum":{"num_bytes":2571471,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"58","length":"1"},{"start":"63","length":"1"},{"start":"6a","length":"2"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":555,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:18.135632+0000","last_change":"2024-09-11T03:42:01.225777+0000","last_active":"2024-09-11T03:42:18.135632+0000","last_peered":"2024-09-11T03:42:18.135632+0000","last_clean":"2024-09-11T03:42:18.135632+0000","last_became_active":"2024-09-11T03:40:25.854269+0000","last_became_peered":"2024-09-11T03:40:25.854269+0000","last_unstale":"2024-09-11T03:42:18.135632+0000","last_undegraded":"2024-09-11T03:42:18.135632+0000","last_fullsized":"2024-09-11T03:42:18.135632+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:01.225716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:01.225716+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:14.417097+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"318'128","reported_seq":1285,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.901700+0000","last_change":"2024-09-11T03:42:15.897582+0000","last_active":"2024-09-11T03:42:16.901700+0000","last_peered":"2024-09-11T03:42:16.901700+0000","last_clean":"2024-09-11T03:42:16.901700+0000","last_became_active":"2024-09-11T03:41:33.866217+0000","last_became_peered":"2024-09-11T03:41:33.866217+0000","last_unstale":"2024-09-11T03:42:16.901700+0000","last_undegraded":"2024-09-11T03:42:16.901700+0000","last_fullsized":"2024-09-11T03:42:16.901700+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:41:40.497820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.497820+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:43.370615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056594100000000004,"stat_sum":{"num_bytes":5249074,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":7,"num_write_kb":1777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":30,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:15.902641+0000","last_change":"2024-09-11T03:42:06.870952+0000","last_active":"2024-09-11T03:42:15.902641+0000","last_peered":"2024-09-11T03:42:15.902641+0000","last_clean":"2024-09-11T03:42:15.902641+0000","last_became_active":"2024-09-11T03:42:06.870500+0000","last_became_peered":"2024-09-11T03:42:06.870500+0000","last_unstale":"2024-09-11T03:42:15.902641+0000","last_undegraded":"2024-09-11T03:42:15.902641+0000","last_fullsized":"2024-09-11T03:42:15.902641+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:02.760253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:02.760253+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"318'86","reported_seq":1245,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:15.902950+0000","last_change":"2024-09-11T03:42:15.903343+0000","last_active":"2024-09-11T03:42:15.902950+0000","last_peered":"2024-09-11T03:42:15.902950+0000","last_clean":"2024-09-11T03:42:15.902950+0000","last_became_active":"2024-09-11T03:41:33.864202+0000","last_became_peered":"2024-09-11T03:41:33.864202+0000","last_unstale":"2024-09-11T03:42:15.902950+0000","last_undegraded":"2024-09-11T03:42:15.902950+0000","last_fullsized":"2024-09-11T03:42:15.902950+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:41:41.843233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.843233+0000","objects_scrubbed":2,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:46.608447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072097899999999996,"stat_sum":{"num_bytes":5068305,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"58","length":"1"},{"start":"63","length":"1"},{"start":"6a","length":"2"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":325,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.124227+0000","last_change":"2024-09-11T03:41:57.122830+0000","last_active":"2024-09-11T03:42:16.124227+0000","last_peered":"2024-09-11T03:42:16.124227+0000","last_clean":"2024-09-11T03:42:16.124227+0000","last_became_active":"2024-09-11T03:40:25.856407+0000","last_became_peered":"2024-09-11T03:40:25.856407+0000","last_unstale":"2024-09-11T03:42:16.124227+0000","last_undegraded":"2024-09-11T03:42:16.124227+0000","last_fullsized":"2024-09-11T03:42:16.124227+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:57.122773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:57.122773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:57.896149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"319'141","reported_seq":1290,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.161957+0000","last_change":"2024-09-11T03:42:16.162163+0000","last_active":"2024-09-11T03:42:16.161957+0000","last_peered":"2024-09-11T03:42:16.161957+0000","last_clean":"2024-09-11T03:42:16.161957+0000","last_became_active":"2024-09-11T03:41:33.862271+0000","last_became_peered":"2024-09-11T03:41:33.862271+0000","last_unstale":"2024-09-11T03:42:16.161957+0000","last_undegraded":"2024-09-11T03:42:16.161957+0000","last_fullsized":"2024-09-11T03:42:16.161957+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'117","last_scrub_stamp":"2024-09-11T03:41:41.089786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.089786+0000","objects_scrubbed":3,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:06.733052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001737363,"stat_sum":{"num_bytes":6455964,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":12237,"num_write":6,"num_write_kb":1020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"58","length":"1"},{"start":"63","length":"1"},{"start":"6a","length":"2"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":646,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:15.902857+0000","last_change":"2024-09-11T03:41:45.957854+0000","last_active":"2024-09-11T03:42:15.902857+0000","last_peered":"2024-09-11T03:42:15.902857+0000","last_clean":"2024-09-11T03:42:15.902857+0000","last_became_active":"2024-09-11T03:40:25.854624+0000","last_became_peered":"2024-09-11T03:40:25.854624+0000","last_unstale":"2024-09-11T03:42:15.902857+0000","last_undegraded":"2024-09-11T03:42:15.902857+0000","last_fullsized":"2024-09-11T03:42:15.902857+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:45.957781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:45.957781+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:01.135709+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"20'92","reported_seq":241,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.160372+0000","last_change":"2024-09-11T03:40:35.282811+0000","last_active":"2024-09-11T03:42:16.160372+0000","last_peered":"2024-09-11T03:42:16.160372+0000","last_clean":"2024-09-11T03:42:16.160372+0000","last_became_active":"2024-09-11T03:40:27.844262+0000","last_became_peered":"2024-09-11T03:40:27.844262+0000","last_unstale":"2024-09-11T03:42:16.160372+0000","last_undegraded":"2024-09-11T03:42:16.160372+0000","last_fullsized":"2024-09-11T03:42:16.160372+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.282762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.282762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:54.069194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"318'265","reported_seq":1408,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.901504+0000","last_change":"2024-09-11T03:42:16.069785+0000","last_active":"2024-09-11T03:42:16.901504+0000","last_peered":"2024-09-11T03:42:16.901504+0000","last_clean":"2024-09-11T03:42:16.901504+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:42:16.901504+0000","last_undegraded":"2024-09-11T03:42:16.901504+0000","last_fullsized":"2024-09-11T03:42:16.901504+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'172","last_scrub_stamp":"2024-09-11T03:41:04.106166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:04.106166+0000","objects_scrubbed":5,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:21.104526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00027725500000000001,"stat_sum":{"num_bytes":13200455,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":150,"num_read_kb":120393,"num_write":123,"num_write_kb":53148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":496,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-11T03:42:14.895182+0000","last_change":"2024-09-11T03:40:38.199696+0000","last_active":"2024-09-11T03:42:14.895182+0000","last_peered":"2024-09-11T03:42:14.895182+0000","last_clean":"2024-09-11T03:42:14.895182+0000","last_became_active":"2024-09-11T03:40:37.850623+0000","last_became_peered":"2024-09-11T03:40:37.850623+0000","last_unstale":"2024-09-11T03:42:14.895182+0000","last_undegraded":"2024-09-11T03:42:14.895182+0000","last_fullsized":"2024-09-11T03:42:14.895182+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.199636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.199636+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:48.923701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"319'412","reported_seq":1533,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:18.136108+0000","last_change":"2024-09-11T03:42:15.903363+0000","last_active":"2024-09-11T03:42:18.136108+0000","last_peered":"2024-09-11T03:42:18.136108+0000","last_clean":"2024-09-11T03:42:18.136108+0000","last_became_active":"2024-09-11T03:38:30.031497+0000","last_became_peered":"2024-09-11T03:38:30.031497+0000","last_unstale":"2024-09-11T03:42:18.136108+0000","last_undegraded":"2024-09-11T03:42:18.136108+0000","last_fullsized":"2024-09-11T03:42:18.136108+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'249","last_scrub_stamp":"2024-09-11T03:41:08.270845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:08.270845+0000","objects_scrubbed":10,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0039688919999999999,"stat_sum":{"num_bytes":25450777,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":163,"num_read_kb":121964,"num_write":143,"num_write_kb":54261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":255,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:15.903158+0000","last_change":"2024-09-11T03:40:38.901246+0000","last_active":"2024-09-11T03:42:15.903158+0000","last_peered":"2024-09-11T03:42:15.903158+0000","last_clean":"2024-09-11T03:42:15.903158+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:42:15.903158+0000","last_undegraded":"2024-09-11T03:42:15.903158+0000","last_fullsized":"2024-09-11T03:42:15.903158+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.901175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.901175+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:50.952770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"319'405","reported_seq":1522,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.902138+0000","last_change":"2024-09-11T03:42:15.899210+0000","last_active":"2024-09-11T03:42:16.902138+0000","last_peered":"2024-09-11T03:42:16.902138+0000","last_clean":"2024-09-11T03:42:16.902138+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:42:16.902138+0000","last_undegraded":"2024-09-11T03:42:16.902138+0000","last_fullsized":"2024-09-11T03:42:16.902138+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'259","last_scrub_stamp":"2024-09-11T03:41:09.347904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:09.347904+0000","objects_scrubbed":11,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:29.356663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002066539,"stat_sum":{"num_bytes":17059336,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":160,"num_read_kb":127275,"num_write":133,"num_write_kb":51847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"2.f","version":"319'185","reported_seq":1331,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-11T03:42:15.902309+0000","last_change":"2024-09-11T03:42:15.902487+0000","last_active":"2024-09-11T03:42:15.902309+0000","last_peered":"2024-09-11T03:42:15.902309+0000","last_clean":"2024-09-11T03:42:15.902309+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:42:15.902309+0000","last_undegraded":"2024-09-11T03:42:15.902309+0000","last_fullsized":"2024-09-11T03:42:15.902309+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'136","last_scrub_stamp":"2024-09-11T03:41:20.167983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:20.167983+0000","objects_scrubbed":2,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:46.194312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0047355649999999997,"stat_sum":{"num_bytes":7033616,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":102,"num_read_kb":82053,"num_write":71,"num_write_kb":25608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"58","length":"1"},{"start":"63","length":"1"},{"start":"6a","length":"2"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":268,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:17.871539+0000","last_change":"2024-09-11T03:40:38.353992+0000","last_active":"2024-09-11T03:42:17.871539+0000","last_peered":"2024-09-11T03:42:17.871539+0000","last_clean":"2024-09-11T03:42:17.871539+0000","last_became_active":"2024-09-11T03:40:37.847449+0000","last_became_peered":"2024-09-11T03:40:37.847449+0000","last_unstale":"2024-09-11T03:42:17.871539+0000","last_undegraded":"2024-09-11T03:42:17.871539+0000","last_fullsized":"2024-09-11T03:42:17.871539+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.353895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.353895+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:48.326329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":215395427,"num_objects":102,"num_object_clones":56,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":5,"num_read":872,"num_read_kb":723502,"num_write":702,"num_write_kb":294864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":10857193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":404144128,"data_stored":546606990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5794,"ondisk_log_size":5794,"up":78,"acting":78,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285629,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58928,"kb_used_data":27736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93686224,"statfs":{"total":95995035648,"available":95934693376,"internally_reserved":0,"allocated":28401664,"data_stored":36659747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":326417514561,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743495,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91128,"kb_used_data":55780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93654024,"statfs":{"total":95995035648,"available":95901720576,"internally_reserved":0,"allocated":57118720,"data_stored":86321930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037838,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66456,"kb_used_data":31100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93678696,"statfs":{"total":95995035648,"available":95926984704,"internally_reserved":0,"allocated":31846400,"data_stored":41644714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299473,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82108,"kb_used_data":46756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93663044,"statfs":{"total":95995035648,"available":95910957056,"internally_reserved":0,"allocated":47878144,"data_stored":56362473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":193273528406,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757340,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47488,"kb_used_data":16288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93697664,"statfs":{"total":95995035648,"available":95946407936,"internally_reserved":0,"allocated":16678912,"data_stored":21930053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986273,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96104,"kb_used_data":60752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93649048,"statfs":{"total":95995035648,"available":95896625152,"internally_reserved":0,"allocated":62210048,"data_stored":81819715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":113916,"kb_used_data":74400,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93631236,"statfs":{"total":95995035648,"available":95878385664,"internally_reserved":0,"allocated":76185600,"data_stored":99680375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411435,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73948,"kb_used_data":38600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93671204,"statfs":{"total":95995035648,"available":95919312896,"internally_reserved":0,"allocated":39526400,"data_stored":52342168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65748,"kb_used_data":30396,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93679404,"statfs":{"total":95995035648,"available":95927709696,"internally_reserved":0,"allocated":31125504,"data_stored":33617164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738486,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78248,"kb_used_data":42896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93666904,"statfs":{"total":95995035648,"available":95914909696,"internally_reserved":0,"allocated":43925504,"data_stored":50933738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40914944,"data_stored":50145496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":27357184,"data_stored":31297614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35528704,"data_stored":49804840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":72052736,"data_stored":97008459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":58441728,"data_stored":79500165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12705792,"data_stored":19409255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4079616,"data_stored":4066399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":44990464,"data_stored":55700663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28065792,"data_stored":39316886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":52985856,"data_stored":83650014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2387968,"data_stored":2367002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24633344,"data_stored":34340197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:42:21.037 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:42:21.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:21.142 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:21.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:21.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:21.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:20 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:42:21.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:20 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:42:21.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:20 smithi121 ceph-mon[30729]: osdmap e323: 12 total, 12 up, 10 in 2024-09-11T03:42:21.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:20 smithi121 ceph-mon[30729]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:42:21.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:20 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/222550303' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:42:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:20 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:42:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:20 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:42:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:20 smithi175 ceph-mon[30894]: osdmap e323: 12 total, 12 up, 10 in 2024-09-11T03:42:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:20 smithi175 ceph-mon[30894]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:42:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:20 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/222550303' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:42:21.244 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:21.265 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:21.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:21.266 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:21.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:21.392 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:21.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:21.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:21.494 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:21.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:21.517 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:21.517 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:21.618 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:21.640 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:21.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:21.641 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:21.658 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:21.662 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:21.681 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:21.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:21.767 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:21.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:21.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:21.869 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:21.872 INFO:tasks.rados.rados.2.smithi175.stdout:1058: done (1 left) 2024-09-11T03:42:21.872 INFO:tasks.rados.rados.2.smithi175.stdout:1059: write (excl) oid 26 current snap is 117 2024-09-11T03:42:21.872 INFO:tasks.rados.rados.2.smithi175.stdout:1059: seq_num 374 ranges {685818=644423,2069089=216729} 2024-09-11T03:42:21.879 INFO:tasks.rados.rados.2.smithi175.stdout:1059: writing smithi17557870-26 from 685818 to 1330241 tid 1 2024-09-11T03:42:21.881 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:21 smithi084 ceph-mon[27254]: pgmap v640: 57 pgs: 57 active+clean; 206 MiB data, 756 MiB used, 893 GiB / 894 GiB avail; 511 B/s rd, 502 KiB/s wr, 3 op/s 2024-09-11T03:42:21.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:21 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:21.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/222550303' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:42:21.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:21 smithi084 ceph-mon[27254]: osdmap e324: 12 total, 12 up, 10 in 2024-09-11T03:42:21.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:21 smithi084 ceph-mon[27254]: osdmap e325: 12 total, 12 up, 10 in 2024-09-11T03:42:21.884 INFO:tasks.rados.rados.2.smithi175.stdout:1059: writing smithi17557870-26 from 2069089 to 2285818 tid 2 2024-09-11T03:42:21.885 INFO:tasks.rados.rados.2.smithi175.stdout:1060: copy_from oid 32 from oid 4 current snap is 117 2024-09-11T03:42:21.885 INFO:tasks.rados.rados.2.smithi175.stdout:1057: expect (ObjNum 342 snap 99 seq_num 342) 2024-09-11T03:42:21.895 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:21.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:21.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:22.018 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:22.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:22.049 INFO:tasks.rados.rados.2.smithi175.stdout:1059: finishing write tid 1 to smithi17557870-26 2024-09-11T03:42:22.050 INFO:tasks.rados.rados.2.smithi175.stdout:1060: finishing copy_from to smithi17557870-32 2024-09-11T03:42:22.050 INFO:tasks.rados.rados.2.smithi175.stdout:1060: got expected ENOENT (src dne) 2024-09-11T03:42:22.050 INFO:tasks.rados.rados.2.smithi175.stdout:1060: finishing copy_from racing read to smithi17557870-32 2024-09-11T03:42:22.050 INFO:tasks.rados.rados.2.smithi175.stdout:1059: finishing write tid 2 to smithi17557870-26 2024-09-11T03:42:22.050 INFO:tasks.rados.rados.2.smithi175.stdout:1059: finishing write tid 3 to smithi17557870-26 2024-09-11T03:42:22.050 INFO:tasks.rados.rados.2.smithi175.stdout:1059: finishing write tid 4 to smithi17557870-26 2024-09-11T03:42:22.050 INFO:tasks.rados.rados.2.smithi175.stdout:1059: oid 26 updating version 0 to 175 2024-09-11T03:42:22.050 INFO:tasks.rados.rados.2.smithi175.stdout:1059: oid 26 updating version 175 to 177 2024-09-11T03:42:22.050 INFO:tasks.rados.rados.2.smithi175.stdout:1059: oid 26 version 177 is already newer than 176 2024-09-11T03:42:22.050 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 177 (ObjNum 374 snap 117 seq_num 374) dirty exists 2024-09-11T03:42:22.050 INFO:tasks.rados.rados.2.smithi175.stdout:1059: left oid 26 (ObjNum 374 snap 117 seq_num 374) 2024-09-11T03:42:22.050 INFO:tasks.rados.rados.2.smithi175.stdout:1057: done (2 left) 2024-09-11T03:42:22.050 INFO:tasks.rados.rados.2.smithi175.stdout:1059: done (1 left) 2024-09-11T03:42:22.051 INFO:tasks.rados.rados.2.smithi175.stdout:1060: done (0 left) 2024-09-11T03:42:22.051 INFO:tasks.rados.rados.2.smithi175.stdout:1061: snap_remove snap 102 2024-09-11T03:42:22.120 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:22.142 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:22.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:22.143 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:22.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:21 smithi121 ceph-mon[30729]: pgmap v640: 57 pgs: 57 active+clean; 206 MiB data, 756 MiB used, 893 GiB / 894 GiB avail; 511 B/s rd, 502 KiB/s wr, 3 op/s 2024-09-11T03:42:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:21 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:21 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/222550303' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:42:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:21 smithi121 ceph-mon[30729]: osdmap e324: 12 total, 12 up, 10 in 2024-09-11T03:42:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:21 smithi121 ceph-mon[30729]: osdmap e325: 12 total, 12 up, 10 in 2024-09-11T03:42:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:21 smithi175 ceph-mon[30894]: pgmap v640: 57 pgs: 57 active+clean; 206 MiB data, 756 MiB used, 893 GiB / 894 GiB avail; 511 B/s rd, 502 KiB/s wr, 3 op/s 2024-09-11T03:42:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:21 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:21 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/222550303' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:42:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:21 smithi175 ceph-mon[30894]: osdmap e324: 12 total, 12 up, 10 in 2024-09-11T03:42:22.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:21 smithi175 ceph-mon[30894]: osdmap e325: 12 total, 12 up, 10 in 2024-09-11T03:42:22.244 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:22.265 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:22.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:22.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:22.367 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:22.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:22.389 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:22.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:22.491 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:22.513 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:22.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:22.530 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:42:22.530 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:42:22.614 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:22.620 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:42:22.621 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:42:22.629 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:42:22.630 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:42:22.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:22.638 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:22.638 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:22.647 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:42:22.647 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:42:22.663 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:42:22.663 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:42:22.680 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:42:22.680 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:42:22.697 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:42:22.697 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:42:22.714 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:42:22.714 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:42:22.731 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:42:22.732 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:42:22.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:22.748 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:42:22.749 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:42:22.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:22.761 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:22.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:22.765 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:42:22.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:42:22.782 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:42:22.783 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:42:22.799 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:42:22.799 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:42:22.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:22.889 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:22.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:22.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:22.894 INFO:tasks.rados.rados.2.smithi175.stdout:1061: done (0 left) 2024-09-11T03:42:22.895 INFO:tasks.rados.rados.2.smithi175.stdout:1062: write (excl) oid 45 current snap is 117 2024-09-11T03:42:22.895 INFO:tasks.rados.rados.2.smithi175.stdout:1062: seq_num 375 ranges {702625=719672,2127571=175054} 2024-09-11T03:42:22.902 INFO:tasks.rados.rados.2.smithi175.stdout:1062: writing smithi17557870-45 from 702625 to 1422297 tid 1 2024-09-11T03:42:22.908 INFO:tasks.rados.rados.2.smithi175.stdout:1062: writing smithi17557870-45 from 2127571 to 2302625 tid 2 2024-09-11T03:42:22.908 INFO:tasks.rados.rados.2.smithi175.stdout:1063: rollback oid 14 current snap is 117 2024-09-11T03:42:22.908 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 14 to 113 2024-09-11T03:42:22.908 INFO:tasks.rados.rados.2.smithi175.stdout:1062: finishing write tid 1 to smithi17557870-45 2024-09-11T03:42:22.908 INFO:tasks.rados.rados.2.smithi175.stdout:1064: rollback oid 15 current snap is 117 2024-09-11T03:42:22.908 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 15 to 91 2024-09-11T03:42:22.908 INFO:tasks.rados.rados.2.smithi175.stdout:1065: snap_remove snap 116 2024-09-11T03:42:22.991 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:23.012 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:23.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:23.115 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:23.141 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:23.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:23.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:23.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:22 smithi084 ceph-mon[27254]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:42:23.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:22 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4044505900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:23.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:22 smithi121 ceph-mon[30729]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:42:23.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:22 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4044505900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:23.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:22 smithi175 ceph-mon[30894]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:42:23.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:22 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4044505900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:23.242 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:23.265 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:23.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:23.266 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:23.270 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":642,"stamp":"2024-09-11T03:42:21.870067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":215985795,"num_objects":104,"num_object_clones":56,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":5,"num_read":913,"num_read_kb":782986,"num_write":702,"num_write_kb":294864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":10857193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8649,"ondisk_log_size":8649,"up":171,"acting":171,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":171,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":781228,"kb_used_data":431876,"kb_used_omap":15,"kb_used_meta":349104,"kb_avail":936670292,"statfs":{"total":959950356480,"available":959150379008,"internally_reserved":0,"allocated":442241024,"data_stored":570530843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":357482933},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3563096,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":-1,"num_read":44,"num_read_kb":59486,"num_write":11,"num_write_kb":2009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.996042"},"pg_stats":[{"pgid":"1.1e","version":"20'92","reported_seq":212,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.901613+0000","last_change":"2024-09-11T03:40:40.236609+0000","last_active":"2024-09-11T03:42:16.901613+0000","last_peered":"2024-09-11T03:42:16.901613+0000","last_clean":"2024-09-11T03:42:16.901613+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:42:16.901613+0000","last_undegraded":"2024-09-11T03:42:16.901613+0000","last_fullsized":"2024-09-11T03:42:16.901613+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.236546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.236546+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:09.146128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'92","reported_seq":222,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.890680+0000","last_change":"2024-09-11T03:40:41.348889+0000","last_active":"2024-09-11T03:42:20.890680+0000","last_peered":"2024-09-11T03:42:20.890680+0000","last_clean":"2024-09-11T03:42:20.890680+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:42:20.890680+0000","last_undegraded":"2024-09-11T03:42:20.890680+0000","last_fullsized":"2024-09-11T03:42:20.890680+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:41.348828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:41.348828+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:11.089042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'92","reported_seq":220,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:19.872183+0000","last_change":"2024-09-11T03:40:40.188174+0000","last_active":"2024-09-11T03:42:19.872183+0000","last_peered":"2024-09-11T03:42:19.872183+0000","last_clean":"2024-09-11T03:42:19.872183+0000","last_became_active":"2024-09-11T03:40:39.848230+0000","last_became_peered":"2024-09-11T03:40:39.848230+0000","last_unstale":"2024-09-11T03:42:19.872183+0000","last_undegraded":"2024-09-11T03:42:19.872183+0000","last_fullsized":"2024-09-11T03:42:19.872183+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.188124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.188124+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:08.246084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"319'214","reported_seq":1415,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:18.135392+0000","last_change":"2024-09-11T03:42:15.903530+0000","last_active":"2024-09-11T03:42:18.135392+0000","last_peered":"2024-09-11T03:42:18.135392+0000","last_clean":"2024-09-11T03:42:18.135392+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:42:18.135392+0000","last_undegraded":"2024-09-11T03:42:18.135392+0000","last_fullsized":"2024-09-11T03:42:18.135392+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"280'196","last_scrub_stamp":"2024-09-11T03:41:40.336904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.336904+0000","objects_scrubbed":2,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:49.342935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042926650000000002,"stat_sum":{"num_bytes":11119314,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":20483,"num_write":5,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4865740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.1b","version":"20'92","reported_seq":220,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:19.872480+0000","last_change":"2024-09-11T03:40:42.166354+0000","last_active":"2024-09-11T03:42:19.872480+0000","last_peered":"2024-09-11T03:42:19.872480+0000","last_clean":"2024-09-11T03:42:19.872480+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:42:19.872480+0000","last_undegraded":"2024-09-11T03:42:19.872480+0000","last_fullsized":"2024-09-11T03:42:19.872480+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:42.166295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:42.166295+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:13.230054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.19","version":"318'345","reported_seq":1552,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:17.871745+0000","last_change":"2024-09-11T03:42:15.899756+0000","last_active":"2024-09-11T03:42:17.871745+0000","last_peered":"2024-09-11T03:42:17.871745+0000","last_clean":"2024-09-11T03:42:17.871745+0000","last_became_active":"2024-09-11T03:41:38.886787+0000","last_became_peered":"2024-09-11T03:41:38.886787+0000","last_unstale":"2024-09-11T03:42:17.871745+0000","last_undegraded":"2024-09-11T03:42:17.871745+0000","last_fullsized":"2024-09-11T03:42:17.871745+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'333","last_scrub_stamp":"2024-09-11T03:41:40.379141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.379141+0000","objects_scrubbed":1,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:52.170720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00063786500000000005,"stat_sum":{"num_bytes":4270678,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.1a","version":"20'92","reported_seq":462,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:19.871971+0000","last_change":"2024-09-11T03:40:40.278752+0000","last_active":"2024-09-11T03:42:19.871971+0000","last_peered":"2024-09-11T03:42:19.871971+0000","last_clean":"2024-09-11T03:42:19.871971+0000","last_became_active":"2024-09-11T03:40:39.848781+0000","last_became_peered":"2024-09-11T03:40:39.848781+0000","last_unstale":"2024-09-11T03:42:19.871971+0000","last_undegraded":"2024-09-11T03:42:19.871971+0000","last_fullsized":"2024-09-11T03:42:19.871971+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.278703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.278703+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:09.913902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"20'92","reported_seq":220,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:19.883955+0000","last_change":"2024-09-11T03:40:40.718553+0000","last_active":"2024-09-11T03:42:19.883955+0000","last_peered":"2024-09-11T03:42:19.883955+0000","last_clean":"2024-09-11T03:42:19.883955+0000","last_became_active":"2024-09-11T03:40:39.851698+0000","last_became_peered":"2024-09-11T03:40:39.851698+0000","last_unstale":"2024-09-11T03:42:19.883955+0000","last_undegraded":"2024-09-11T03:42:19.883955+0000","last_fullsized":"2024-09-11T03:42:19.883955+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.718503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.718503+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:13.733914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.18","version":"20'92","reported_seq":344,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.882751+0000","last_change":"2024-09-11T03:42:02.103343+0000","last_active":"2024-09-11T03:42:20.882751+0000","last_peered":"2024-09-11T03:42:20.882751+0000","last_clean":"2024-09-11T03:42:20.882751+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:42:20.882751+0000","last_undegraded":"2024-09-11T03:42:20.882751+0000","last_fullsized":"2024-09-11T03:42:20.882751+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:02.103292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:02.103292+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:29.586019+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"274'162","reported_seq":1318,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:18.135923+0000","last_change":"2024-09-11T03:42:15.899999+0000","last_active":"2024-09-11T03:42:18.135923+0000","last_peered":"2024-09-11T03:42:18.135923+0000","last_clean":"2024-09-11T03:42:18.135923+0000","last_became_active":"2024-09-11T03:41:38.890690+0000","last_became_peered":"2024-09-11T03:41:38.890690+0000","last_unstale":"2024-09-11T03:42:18.135923+0000","last_undegraded":"2024-09-11T03:42:18.135923+0000","last_fullsized":"2024-09-11T03:42:18.135923+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'162","last_scrub_stamp":"2024-09-11T03:41:42.270480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:42.270480+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:54.988485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010241740000000001,"stat_sum":{"num_bytes":3327819,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[9,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":228,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:19.872341+0000","last_change":"2024-09-11T03:41:57.260720+0000","last_active":"2024-09-11T03:42:19.872341+0000","last_peered":"2024-09-11T03:42:19.872341+0000","last_clean":"2024-09-11T03:42:19.872341+0000","last_became_active":"2024-09-11T03:40:39.850243+0000","last_became_peered":"2024-09-11T03:40:39.850243+0000","last_unstale":"2024-09-11T03:42:19.872341+0000","last_undegraded":"2024-09-11T03:42:19.872341+0000","last_fullsized":"2024-09-11T03:42:19.872341+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:57.260664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:57.260664+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:11.818969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":1448,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:18.136252+0000","last_change":"2024-09-11T03:42:15.900459+0000","last_active":"2024-09-11T03:42:18.136252+0000","last_peered":"2024-09-11T03:42:18.136252+0000","last_clean":"2024-09-11T03:42:18.136252+0000","last_became_active":"2024-09-11T03:41:38.888228+0000","last_became_peered":"2024-09-11T03:41:38.888228+0000","last_unstale":"2024-09-11T03:42:18.136252+0000","last_undegraded":"2024-09-11T03:42:18.136252+0000","last_fullsized":"2024-09-11T03:42:18.136252+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:41:43.239726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:43.239726+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:56.211264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001295441,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":228,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.890224+0000","last_change":"2024-09-11T03:42:05.102506+0000","last_active":"2024-09-11T03:42:20.890224+0000","last_peered":"2024-09-11T03:42:20.890224+0000","last_clean":"2024-09-11T03:42:20.890224+0000","last_became_active":"2024-09-11T03:40:39.849950+0000","last_became_peered":"2024-09-11T03:40:39.849950+0000","last_unstale":"2024-09-11T03:42:20.890224+0000","last_undegraded":"2024-09-11T03:42:20.890224+0000","last_fullsized":"2024-09-11T03:42:20.890224+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:05.102457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:05.102457+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:18.344056+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"318'191","reported_seq":1333,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.883079+0000","last_change":"2024-09-11T03:42:18.872884+0000","last_active":"2024-09-11T03:42:20.883079+0000","last_peered":"2024-09-11T03:42:20.883079+0000","last_clean":"2024-09-11T03:42:20.883079+0000","last_became_active":"2024-09-11T03:41:38.887856+0000","last_became_peered":"2024-09-11T03:41:38.887856+0000","last_unstale":"2024-09-11T03:42:20.883079+0000","last_undegraded":"2024-09-11T03:42:20.883079+0000","last_fullsized":"2024-09-11T03:42:20.883079+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:41:42.059696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:42.059696+0000","objects_scrubbed":1,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:43.554033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087215100000000005,"stat_sum":{"num_bytes":6749143,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":1870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,9],"acting":[5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"},{"start":"72","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":219,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.107632+0000","last_change":"2024-09-11T03:40:39.928143+0000","last_active":"2024-09-11T03:42:20.107632+0000","last_peered":"2024-09-11T03:42:20.107632+0000","last_clean":"2024-09-11T03:42:20.107632+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:42:20.107632+0000","last_undegraded":"2024-09-11T03:42:20.107632+0000","last_fullsized":"2024-09-11T03:42:20.107632+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.928088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.928088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:08.756769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":1526,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.902571+0000","last_change":"2024-09-11T03:42:15.898195+0000","last_active":"2024-09-11T03:42:16.902571+0000","last_peered":"2024-09-11T03:42:16.902571+0000","last_clean":"2024-09-11T03:42:16.902571+0000","last_became_active":"2024-09-11T03:41:38.885486+0000","last_became_peered":"2024-09-11T03:41:38.885486+0000","last_unstale":"2024-09-11T03:42:16.902571+0000","last_undegraded":"2024-09-11T03:42:16.902571+0000","last_fullsized":"2024-09-11T03:42:16.902571+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:41:41.456545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.456545+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:00.230354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092932900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":244,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.902242+0000","last_change":"2024-09-11T03:40:35.195938+0000","last_active":"2024-09-11T03:42:16.902242+0000","last_peered":"2024-09-11T03:42:16.902242+0000","last_clean":"2024-09-11T03:42:16.902242+0000","last_became_active":"2024-09-11T03:40:27.847297+0000","last_became_peered":"2024-09-11T03:40:27.847297+0000","last_unstale":"2024-09-11T03:42:16.902242+0000","last_undegraded":"2024-09-11T03:42:16.902242+0000","last_fullsized":"2024-09-11T03:42:16.902242+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.195893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.195893+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:50.659810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":249,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:19.872141+0000","last_change":"2024-09-11T03:42:18.872146+0000","last_active":"2024-09-11T03:42:19.872141+0000","last_peered":"2024-09-11T03:42:19.872141+0000","last_clean":"2024-09-11T03:42:19.872141+0000","last_became_active":"2024-09-11T03:41:35.874295+0000","last_became_peered":"2024-09-11T03:41:35.874295+0000","last_unstale":"2024-09-11T03:42:19.872141+0000","last_undegraded":"2024-09-11T03:42:19.872141+0000","last_fullsized":"2024-09-11T03:42:19.872141+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:41:40.182948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.182948+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:54.921521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060089099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"},{"start":"72","length":"1"}]},{"pgid":"1.13","version":"20'92","reported_seq":337,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.890484+0000","last_change":"2024-09-11T03:41:45.086774+0000","last_active":"2024-09-11T03:42:20.890484+0000","last_peered":"2024-09-11T03:42:20.890484+0000","last_clean":"2024-09-11T03:42:20.890484+0000","last_became_active":"2024-09-11T03:40:27.847557+0000","last_became_peered":"2024-09-11T03:40:27.847557+0000","last_unstale":"2024-09-11T03:42:20.890484+0000","last_undegraded":"2024-09-11T03:42:20.890484+0000","last_fullsized":"2024-09-11T03:42:20.890484+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:45.086692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:45.086692+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:48.005883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":1242,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:19.872031+0000","last_change":"2024-09-11T03:42:18.880349+0000","last_active":"2024-09-11T03:42:19.872031+0000","last_peered":"2024-09-11T03:42:19.872031+0000","last_clean":"2024-09-11T03:42:19.872031+0000","last_became_active":"2024-09-11T03:41:38.886813+0000","last_became_peered":"2024-09-11T03:41:38.886813+0000","last_unstale":"2024-09-11T03:42:19.872031+0000","last_undegraded":"2024-09-11T03:42:19.872031+0000","last_fullsized":"2024-09-11T03:42:19.872031+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:41:40.077940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.077940+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:07.843971+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053909900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,9],"acting":[4,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"},{"start":"72","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":249,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.882772+0000","last_change":"2024-09-11T03:40:37.284871+0000","last_active":"2024-09-11T03:42:20.882772+0000","last_peered":"2024-09-11T03:42:20.882772+0000","last_clean":"2024-09-11T03:42:20.882772+0000","last_became_active":"2024-09-11T03:40:27.845249+0000","last_became_peered":"2024-09-11T03:40:27.845249+0000","last_unstale":"2024-09-11T03:42:20.882772+0000","last_undegraded":"2024-09-11T03:42:20.882772+0000","last_fullsized":"2024-09-11T03:42:20.882772+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:37.284800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:37.284800+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:55.054741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"319'145","reported_seq":1297,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:17.871874+0000","last_change":"2024-09-11T03:42:15.899914+0000","last_active":"2024-09-11T03:42:17.871874+0000","last_peered":"2024-09-11T03:42:17.871874+0000","last_clean":"2024-09-11T03:42:17.871874+0000","last_became_active":"2024-09-11T03:42:09.873469+0000","last_became_peered":"2024-09-11T03:42:09.873469+0000","last_unstale":"2024-09-11T03:42:17.871874+0000","last_undegraded":"2024-09-11T03:42:17.871874+0000","last_fullsized":"2024-09-11T03:42:17.871874+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"283'124","last_scrub_stamp":"2024-09-11T03:41:41.419995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.419995+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:43.642854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081549300000000003,"stat_sum":{"num_bytes":11430617,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":10807,"num_write":15,"num_write_kb":9128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":5991453,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":252,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:19.884108+0000","last_change":"2024-09-11T03:40:35.703292+0000","last_active":"2024-09-11T03:42:19.884108+0000","last_peered":"2024-09-11T03:42:19.884108+0000","last_clean":"2024-09-11T03:42:19.884108+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:42:19.884108+0000","last_undegraded":"2024-09-11T03:42:19.884108+0000","last_fullsized":"2024-09-11T03:42:19.884108+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.703203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.703203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:49.677961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":1235,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:18.135791+0000","last_change":"2024-09-11T03:42:15.899984+0000","last_active":"2024-09-11T03:42:18.135791+0000","last_peered":"2024-09-11T03:42:18.135791+0000","last_clean":"2024-09-11T03:42:18.135791+0000","last_became_active":"2024-09-11T03:41:38.886834+0000","last_became_peered":"2024-09-11T03:41:38.886834+0000","last_unstale":"2024-09-11T03:42:18.135791+0000","last_undegraded":"2024-09-11T03:42:18.135791+0000","last_fullsized":"2024-09-11T03:42:18.135791+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:41:41.300112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.300112+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:01.963682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011230770000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":257,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.882772+0000","last_change":"2024-09-11T03:42:03.151379+0000","last_active":"2024-09-11T03:42:20.882772+0000","last_peered":"2024-09-11T03:42:20.882772+0000","last_clean":"2024-09-11T03:42:20.882772+0000","last_became_active":"2024-09-11T03:40:27.844057+0000","last_became_peered":"2024-09-11T03:40:27.844057+0000","last_unstale":"2024-09-11T03:42:20.882772+0000","last_undegraded":"2024-09-11T03:42:20.882772+0000","last_fullsized":"2024-09-11T03:42:20.882772+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:03.151323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:03.151323+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:05.151390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"318'305","reported_seq":1429,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:19.872646+0000","last_change":"2024-09-11T03:42:18.872514+0000","last_active":"2024-09-11T03:42:19.872646+0000","last_peered":"2024-09-11T03:42:19.872646+0000","last_clean":"2024-09-11T03:42:19.872646+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:42:19.872646+0000","last_undegraded":"2024-09-11T03:42:19.872646+0000","last_fullsized":"2024-09-11T03:42:19.872646+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'232","last_scrub_stamp":"2024-09-11T03:41:04.001653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:04.001653+0000","objects_scrubbed":10,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:24.646908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00056746300000000004,"stat_sum":{"num_bytes":10941489,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":60,"num_read_kb":26483,"num_write":102,"num_write_kb":52867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"},{"start":"72","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":334,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.882893+0000","last_change":"2024-09-11T03:42:05.135033+0000","last_active":"2024-09-11T03:42:20.882893+0000","last_peered":"2024-09-11T03:42:20.882893+0000","last_clean":"2024-09-11T03:42:20.882893+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:42:20.882893+0000","last_undegraded":"2024-09-11T03:42:20.882893+0000","last_fullsized":"2024-09-11T03:42:20.882893+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:05.134977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:05.134977+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:05.203169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"318'101","reported_seq":1227,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.901276+0000","last_change":"2024-09-11T03:42:16.069744+0000","last_active":"2024-09-11T03:42:16.901276+0000","last_peered":"2024-09-11T03:42:16.901276+0000","last_clean":"2024-09-11T03:42:16.901276+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:42:16.901276+0000","last_undegraded":"2024-09-11T03:42:16.901276+0000","last_fullsized":"2024-09-11T03:42:16.901276+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'57","last_scrub_stamp":"2024-09-11T03:40:59.151899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:40:59.151899+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00037710100000000002,"stat_sum":{"num_bytes":4776703,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":68051,"num_write":35,"num_write_kb":16778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":657,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.107405+0000","last_change":"2024-09-11T03:42:00.816905+0000","last_active":"2024-09-11T03:42:20.107405+0000","last_peered":"2024-09-11T03:42:20.107405+0000","last_clean":"2024-09-11T03:42:20.107405+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:42:20.107405+0000","last_undegraded":"2024-09-11T03:42:20.107405+0000","last_fullsized":"2024-09-11T03:42:20.107405+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:00.816850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:00.816850+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:09.071150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"319'399","reported_seq":1634,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.892190+0000","last_change":"2024-09-11T03:42:18.874406+0000","last_active":"2024-09-11T03:42:20.892190+0000","last_peered":"2024-09-11T03:42:20.892190+0000","last_clean":"2024-09-11T03:42:20.892190+0000","last_became_active":"2024-09-11T03:41:33.865638+0000","last_became_peered":"2024-09-11T03:41:33.865638+0000","last_unstale":"2024-09-11T03:42:20.892190+0000","last_undegraded":"2024-09-11T03:42:20.892190+0000","last_fullsized":"2024-09-11T03:42:20.892190+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'337","last_scrub_stamp":"2024-09-11T03:41:40.102984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.102984+0000","objects_scrubbed":5,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:04.804583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000829359,"stat_sum":{"num_bytes":24271967,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":71,"num_read_kb":86432,"num_write":17,"num_write_kb":7206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"},{"start":"72","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":566,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:18.135655+0000","last_change":"2024-09-11T03:41:45.227630+0000","last_active":"2024-09-11T03:42:18.135655+0000","last_peered":"2024-09-11T03:42:18.135655+0000","last_clean":"2024-09-11T03:42:18.135655+0000","last_became_active":"2024-09-11T03:40:37.849988+0000","last_became_peered":"2024-09-11T03:40:37.849988+0000","last_unstale":"2024-09-11T03:42:18.135655+0000","last_undegraded":"2024-09-11T03:42:18.135655+0000","last_fullsized":"2024-09-11T03:42:18.135655+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:45.227581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:45.227581+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:03.066353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"319'218","reported_seq":1382,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.107738+0000","last_change":"2024-09-11T03:42:18.874890+0000","last_active":"2024-09-11T03:42:20.107738+0000","last_peered":"2024-09-11T03:42:20.107738+0000","last_clean":"2024-09-11T03:42:20.107738+0000","last_became_active":"2024-09-11T03:41:33.863875+0000","last_became_peered":"2024-09-11T03:41:33.863875+0000","last_unstale":"2024-09-11T03:42:20.107738+0000","last_undegraded":"2024-09-11T03:42:20.107738+0000","last_fullsized":"2024-09-11T03:42:20.107738+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'192","last_scrub_stamp":"2024-09-11T03:41:40.877628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.877628+0000","objects_scrubbed":1,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:44.943462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064304800000000001,"stat_sum":{"num_bytes":2386627,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"},{"start":"72","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":256,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.901228+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:42:16.901228+0000","last_peered":"2024-09-11T03:42:16.901228+0000","last_clean":"2024-09-11T03:42:16.901228+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:42:16.901228+0000","last_undegraded":"2024-09-11T03:42:16.901228+0000","last_fullsized":"2024-09-11T03:42:16.901228+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:54.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"319'372","reported_seq":1592,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.903017+0000","last_change":"2024-09-11T03:42:15.902632+0000","last_active":"2024-09-11T03:42:16.903017+0000","last_peered":"2024-09-11T03:42:16.903017+0000","last_clean":"2024-09-11T03:42:16.903017+0000","last_became_active":"2024-09-11T03:41:33.867674+0000","last_became_peered":"2024-09-11T03:41:33.867674+0000","last_unstale":"2024-09-11T03:42:16.903017+0000","last_undegraded":"2024-09-11T03:42:16.903017+0000","last_fullsized":"2024-09-11T03:42:16.903017+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'331","last_scrub_stamp":"2024-09-11T03:41:42.500501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:42.500501+0000","objects_scrubbed":9,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:45.486882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00538954,"stat_sum":{"num_bytes":23855551,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":28,"num_read_kb":31252,"num_write":10,"num_write_kb":5112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":423,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.901271+0000","last_change":"2024-09-11T03:42:00.181197+0000","last_active":"2024-09-11T03:42:16.901271+0000","last_peered":"2024-09-11T03:42:16.901271+0000","last_clean":"2024-09-11T03:42:16.901271+0000","last_became_active":"2024-09-11T03:40:25.858281+0000","last_became_peered":"2024-09-11T03:40:25.858281+0000","last_unstale":"2024-09-11T03:42:16.901271+0000","last_undegraded":"2024-09-11T03:42:16.901271+0000","last_fullsized":"2024-09-11T03:42:16.901271+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:00.181139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:00.181139+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:00.294383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":1315,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.890370+0000","last_change":"2024-09-11T03:42:18.874394+0000","last_active":"2024-09-11T03:42:20.890370+0000","last_peered":"2024-09-11T03:42:20.890370+0000","last_clean":"2024-09-11T03:42:20.890370+0000","last_became_active":"2024-09-11T03:41:33.865499+0000","last_became_peered":"2024-09-11T03:41:33.865499+0000","last_unstale":"2024-09-11T03:42:20.890370+0000","last_undegraded":"2024-09-11T03:42:20.890370+0000","last_fullsized":"2024-09-11T03:42:20.890370+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:41:42.104989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:42.104989+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:58.327927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080128600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"},{"start":"72","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":323,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.902099+0000","last_change":"2024-09-11T03:40:38.224278+0000","last_active":"2024-09-11T03:42:16.902099+0000","last_peered":"2024-09-11T03:42:16.902099+0000","last_clean":"2024-09-11T03:42:16.902099+0000","last_became_active":"2024-09-11T03:40:37.851080+0000","last_became_peered":"2024-09-11T03:40:37.851080+0000","last_unstale":"2024-09-11T03:42:16.902099+0000","last_undegraded":"2024-09-11T03:42:16.902099+0000","last_fullsized":"2024-09-11T03:42:16.902099+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.224231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.224231+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:09.894123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"318'369","reported_seq":1580,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.107377+0000","last_change":"2024-09-11T03:42:18.872518+0000","last_active":"2024-09-11T03:42:20.107377+0000","last_peered":"2024-09-11T03:42:20.107377+0000","last_clean":"2024-09-11T03:42:20.107377+0000","last_became_active":"2024-09-11T03:41:33.866316+0000","last_became_peered":"2024-09-11T03:41:33.866316+0000","last_unstale":"2024-09-11T03:42:20.107377+0000","last_undegraded":"2024-09-11T03:42:20.107377+0000","last_fullsized":"2024-09-11T03:42:20.107377+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'292","last_scrub_stamp":"2024-09-11T03:41:41.222192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.222192+0000","objects_scrubbed":9,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:49.796134+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00071649400000000003,"stat_sum":{"num_bytes":23924251,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":40,"num_read_kb":54460,"num_write":10,"num_write_kb":7255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"},{"start":"72","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":569,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:18.136012+0000","last_change":"2024-09-11T03:42:05.222525+0000","last_active":"2024-09-11T03:42:18.136012+0000","last_peered":"2024-09-11T03:42:18.136012+0000","last_clean":"2024-09-11T03:42:18.136012+0000","last_became_active":"2024-09-11T03:40:37.848794+0000","last_became_peered":"2024-09-11T03:40:37.848794+0000","last_unstale":"2024-09-11T03:42:18.136012+0000","last_undegraded":"2024-09-11T03:42:18.136012+0000","last_fullsized":"2024-09-11T03:42:18.136012+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:05.222462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:05.222462+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:33.609855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"322'174","reported_seq":1367,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.890860+0000","last_change":"2024-09-11T03:42:18.874030+0000","last_active":"2024-09-11T03:42:20.890860+0000","last_peered":"2024-09-11T03:42:20.890860+0000","last_clean":"2024-09-11T03:42:20.890860+0000","last_became_active":"2024-09-11T03:41:33.869119+0000","last_became_peered":"2024-09-11T03:41:33.869119+0000","last_unstale":"2024-09-11T03:42:20.890860+0000","last_undegraded":"2024-09-11T03:42:20.890860+0000","last_fullsized":"2024-09-11T03:42:20.890860+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'162","last_scrub_stamp":"2024-09-11T03:41:41.141743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.141743+0000","objects_scrubbed":2,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:44.195227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045267099999999998,"stat_sum":{"num_bytes":6252270,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":17,"num_read_kb":21092,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"},{"start":"72","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":60,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:17.871672+0000","last_change":"2024-09-11T03:41:58.387920+0000","last_active":"2024-09-11T03:42:17.871672+0000","last_peered":"2024-09-11T03:42:17.871672+0000","last_clean":"2024-09-11T03:42:17.871672+0000","last_became_active":"2024-09-11T03:41:57.865927+0000","last_became_peered":"2024-09-11T03:41:57.865927+0000","last_unstale":"2024-09-11T03:42:17.871672+0000","last_undegraded":"2024-09-11T03:42:17.871672+0000","last_fullsized":"2024-09-11T03:42:17.871672+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:58.387873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:58.387873+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:03.728844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"310'137","reported_seq":1324,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.882928+0000","last_change":"2024-09-11T03:42:18.872544+0000","last_active":"2024-09-11T03:42:20.882928+0000","last_peered":"2024-09-11T03:42:20.882928+0000","last_clean":"2024-09-11T03:42:20.882928+0000","last_became_active":"2024-09-11T03:41:33.861182+0000","last_became_peered":"2024-09-11T03:41:33.861182+0000","last_unstale":"2024-09-11T03:42:20.882928+0000","last_undegraded":"2024-09-11T03:42:20.882928+0000","last_fullsized":"2024-09-11T03:42:20.882928+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'123","last_scrub_stamp":"2024-09-11T03:41:40.064448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.064448+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:01.015487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00067326099999999998,"stat_sum":{"num_bytes":2571471,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"},{"start":"72","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":555,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:18.135632+0000","last_change":"2024-09-11T03:42:01.225777+0000","last_active":"2024-09-11T03:42:18.135632+0000","last_peered":"2024-09-11T03:42:18.135632+0000","last_clean":"2024-09-11T03:42:18.135632+0000","last_became_active":"2024-09-11T03:40:25.854269+0000","last_became_peered":"2024-09-11T03:40:25.854269+0000","last_unstale":"2024-09-11T03:42:18.135632+0000","last_undegraded":"2024-09-11T03:42:18.135632+0000","last_fullsized":"2024-09-11T03:42:18.135632+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:01.225716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:01.225716+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:14.417097+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"318'128","reported_seq":1285,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.901700+0000","last_change":"2024-09-11T03:42:15.897582+0000","last_active":"2024-09-11T03:42:16.901700+0000","last_peered":"2024-09-11T03:42:16.901700+0000","last_clean":"2024-09-11T03:42:16.901700+0000","last_became_active":"2024-09-11T03:41:33.866217+0000","last_became_peered":"2024-09-11T03:41:33.866217+0000","last_unstale":"2024-09-11T03:42:16.901700+0000","last_undegraded":"2024-09-11T03:42:16.901700+0000","last_fullsized":"2024-09-11T03:42:16.901700+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:41:40.497820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.497820+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:43.370615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056594100000000004,"stat_sum":{"num_bytes":5249074,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":7,"num_write_kb":1777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":38,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.107474+0000","last_change":"2024-09-11T03:42:06.870952+0000","last_active":"2024-09-11T03:42:20.107474+0000","last_peered":"2024-09-11T03:42:20.107474+0000","last_clean":"2024-09-11T03:42:20.107474+0000","last_became_active":"2024-09-11T03:42:06.870500+0000","last_became_peered":"2024-09-11T03:42:06.870500+0000","last_unstale":"2024-09-11T03:42:20.107474+0000","last_undegraded":"2024-09-11T03:42:20.107474+0000","last_fullsized":"2024-09-11T03:42:20.107474+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:02.760253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:02.760253+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:17.778058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"318'86","reported_seq":1261,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.107713+0000","last_change":"2024-09-11T03:42:18.875126+0000","last_active":"2024-09-11T03:42:20.107713+0000","last_peered":"2024-09-11T03:42:20.107713+0000","last_clean":"2024-09-11T03:42:20.107713+0000","last_became_active":"2024-09-11T03:41:33.864202+0000","last_became_peered":"2024-09-11T03:41:33.864202+0000","last_unstale":"2024-09-11T03:42:20.107713+0000","last_undegraded":"2024-09-11T03:42:20.107713+0000","last_fullsized":"2024-09-11T03:42:20.107713+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:41:41.843233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.843233+0000","objects_scrubbed":2,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:46.404067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00065587900000000005,"stat_sum":{"num_bytes":5068305,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"},{"start":"72","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":335,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.890498+0000","last_change":"2024-09-11T03:41:57.122830+0000","last_active":"2024-09-11T03:42:20.890498+0000","last_peered":"2024-09-11T03:42:20.890498+0000","last_clean":"2024-09-11T03:42:20.890498+0000","last_became_active":"2024-09-11T03:40:25.856407+0000","last_became_peered":"2024-09-11T03:40:25.856407+0000","last_unstale":"2024-09-11T03:42:20.890498+0000","last_undegraded":"2024-09-11T03:42:20.890498+0000","last_fullsized":"2024-09-11T03:42:20.890498+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:57.122773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:57.122773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:57.896149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"322'143","reported_seq":1311,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.882985+0000","last_change":"2024-09-11T03:42:18.875421+0000","last_active":"2024-09-11T03:42:20.882985+0000","last_peered":"2024-09-11T03:42:20.882985+0000","last_clean":"2024-09-11T03:42:20.882985+0000","last_became_active":"2024-09-11T03:41:33.862271+0000","last_became_peered":"2024-09-11T03:41:33.862271+0000","last_unstale":"2024-09-11T03:42:20.882985+0000","last_undegraded":"2024-09-11T03:42:20.882985+0000","last_fullsized":"2024-09-11T03:42:20.882985+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'117","last_scrub_stamp":"2024-09-11T03:41:41.089786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.089786+0000","objects_scrubbed":3,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:01.125665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0034361499999999998,"stat_sum":{"num_bytes":6455964,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":12237,"num_write":6,"num_write_kb":1020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"},{"start":"72","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":654,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.107851+0000","last_change":"2024-09-11T03:41:45.957854+0000","last_active":"2024-09-11T03:42:20.107851+0000","last_peered":"2024-09-11T03:42:20.107851+0000","last_clean":"2024-09-11T03:42:20.107851+0000","last_became_active":"2024-09-11T03:40:25.854624+0000","last_became_peered":"2024-09-11T03:40:25.854624+0000","last_unstale":"2024-09-11T03:42:20.107851+0000","last_undegraded":"2024-09-11T03:42:20.107851+0000","last_fullsized":"2024-09-11T03:42:20.107851+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:45.957781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:45.957781+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:01.135709+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"20'92","reported_seq":251,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.883102+0000","last_change":"2024-09-11T03:40:35.282811+0000","last_active":"2024-09-11T03:42:20.883102+0000","last_peered":"2024-09-11T03:42:20.883102+0000","last_clean":"2024-09-11T03:42:20.883102+0000","last_became_active":"2024-09-11T03:40:27.844262+0000","last_became_peered":"2024-09-11T03:40:27.844262+0000","last_unstale":"2024-09-11T03:42:20.883102+0000","last_undegraded":"2024-09-11T03:42:20.883102+0000","last_fullsized":"2024-09-11T03:42:20.883102+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.282762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.282762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:54.069194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"318'265","reported_seq":1408,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.901504+0000","last_change":"2024-09-11T03:42:16.069785+0000","last_active":"2024-09-11T03:42:16.901504+0000","last_peered":"2024-09-11T03:42:16.901504+0000","last_clean":"2024-09-11T03:42:16.901504+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:42:16.901504+0000","last_undegraded":"2024-09-11T03:42:16.901504+0000","last_fullsized":"2024-09-11T03:42:16.901504+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'172","last_scrub_stamp":"2024-09-11T03:41:04.106166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:04.106166+0000","objects_scrubbed":5,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:21.104526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00027725500000000001,"stat_sum":{"num_bytes":13200455,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":150,"num_read_kb":120393,"num_write":123,"num_write_kb":53148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":506,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:19.872085+0000","last_change":"2024-09-11T03:40:38.199696+0000","last_active":"2024-09-11T03:42:19.872085+0000","last_peered":"2024-09-11T03:42:19.872085+0000","last_clean":"2024-09-11T03:42:19.872085+0000","last_became_active":"2024-09-11T03:40:37.850623+0000","last_became_peered":"2024-09-11T03:40:37.850623+0000","last_unstale":"2024-09-11T03:42:19.872085+0000","last_undegraded":"2024-09-11T03:42:19.872085+0000","last_fullsized":"2024-09-11T03:42:19.872085+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.199636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.199636+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:48.923701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"319'412","reported_seq":1533,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:18.136108+0000","last_change":"2024-09-11T03:42:15.903363+0000","last_active":"2024-09-11T03:42:18.136108+0000","last_peered":"2024-09-11T03:42:18.136108+0000","last_clean":"2024-09-11T03:42:18.136108+0000","last_became_active":"2024-09-11T03:38:30.031497+0000","last_became_peered":"2024-09-11T03:38:30.031497+0000","last_unstale":"2024-09-11T03:42:18.136108+0000","last_undegraded":"2024-09-11T03:42:18.136108+0000","last_fullsized":"2024-09-11T03:42:18.136108+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'249","last_scrub_stamp":"2024-09-11T03:41:08.270845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:08.270845+0000","objects_scrubbed":10,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0039688919999999999,"stat_sum":{"num_bytes":25450777,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":163,"num_read_kb":121964,"num_write":143,"num_write_kb":54261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":263,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:20.107690+0000","last_change":"2024-09-11T03:40:38.901246+0000","last_active":"2024-09-11T03:42:20.107690+0000","last_peered":"2024-09-11T03:42:20.107690+0000","last_clean":"2024-09-11T03:42:20.107690+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:42:20.107690+0000","last_undegraded":"2024-09-11T03:42:20.107690+0000","last_fullsized":"2024-09-11T03:42:20.107690+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.901175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.901175+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:50.952770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"319'405","reported_seq":1522,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-09-11T03:42:16.902138+0000","last_change":"2024-09-11T03:42:15.899210+0000","last_active":"2024-09-11T03:42:16.902138+0000","last_peered":"2024-09-11T03:42:16.902138+0000","last_clean":"2024-09-11T03:42:16.902138+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:42:16.902138+0000","last_undegraded":"2024-09-11T03:42:16.902138+0000","last_fullsized":"2024-09-11T03:42:16.902138+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'259","last_scrub_stamp":"2024-09-11T03:41:09.347904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:09.347904+0000","objects_scrubbed":11,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:29.356663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002066539,"stat_sum":{"num_bytes":17059336,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":160,"num_read_kb":127275,"num_write":133,"num_write_kb":51847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"}]},{"pgid":"2.f","version":"319'185","reported_seq":1347,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-09-11T03:42:19.872520+0000","last_change":"2024-09-11T03:42:18.872171+0000","last_active":"2024-09-11T03:42:19.872520+0000","last_peered":"2024-09-11T03:42:19.872520+0000","last_clean":"2024-09-11T03:42:19.872520+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:42:19.872520+0000","last_undegraded":"2024-09-11T03:42:19.872520+0000","last_fullsized":"2024-09-11T03:42:19.872520+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'136","last_scrub_stamp":"2024-09-11T03:41:20.167983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:20.167983+0000","objects_scrubbed":2,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:36.860472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00040388499999999999,"stat_sum":{"num_bytes":7033616,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":102,"num_read_kb":82053,"num_write":71,"num_write_kb":25608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"63","length":"1"},{"start":"6a","length":"1"},{"start":"6e","length":"1"},{"start":"70","length":"1"},{"start":"72","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":268,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-09-11T03:42:17.871539+0000","last_change":"2024-09-11T03:40:38.353992+0000","last_active":"2024-09-11T03:42:17.871539+0000","last_peered":"2024-09-11T03:42:17.871539+0000","last_clean":"2024-09-11T03:42:17.871539+0000","last_became_active":"2024-09-11T03:40:37.847449+0000","last_became_peered":"2024-09-11T03:40:37.847449+0000","last_unstale":"2024-09-11T03:42:17.871539+0000","last_undegraded":"2024-09-11T03:42:17.871539+0000","last_fullsized":"2024-09-11T03:42:17.871539+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.353895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.353895+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:48.326329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":215395427,"num_objects":102,"num_object_clones":56,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":5,"num_read":913,"num_read_kb":782986,"num_write":702,"num_write_kb":294864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":10857193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":408936448,"data_stored":551852898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5797,"ondisk_log_size":5797,"up":78,"acting":78,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285629,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58928,"kb_used_data":27736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93686224,"statfs":{"total":95995035648,"available":95934693376,"internally_reserved":0,"allocated":28401664,"data_stored":36659747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":326417514561,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743495,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91128,"kb_used_data":55780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93654024,"statfs":{"total":95995035648,"available":95901720576,"internally_reserved":0,"allocated":57118720,"data_stored":86321930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037839,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68020,"kb_used_data":32672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93677132,"statfs":{"total":95995035648,"available":95925383168,"internally_reserved":0,"allocated":33456128,"data_stored":43237418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299474,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83232,"kb_used_data":47876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93661920,"statfs":{"total":95995035648,"available":95909806080,"internally_reserved":0,"allocated":49025024,"data_stored":58262597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":193273528406,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757341,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47556,"kb_used_data":16360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93697596,"statfs":{"total":95995035648,"available":95946338304,"internally_reserved":0,"allocated":16752640,"data_stored":21979361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986274,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96160,"kb_used_data":60812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93648992,"statfs":{"total":95995035648,"available":95896567808,"internally_reserved":0,"allocated":62271488,"data_stored":81860851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":113916,"kb_used_data":74400,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93631236,"statfs":{"total":95995035648,"available":95878385664,"internally_reserved":0,"allocated":76185600,"data_stored":99680375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411435,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73948,"kb_used_data":38600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93671204,"statfs":{"total":95995035648,"available":95919312896,"internally_reserved":0,"allocated":39526400,"data_stored":52342168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574961,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66636,"kb_used_data":31284,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93678516,"statfs":{"total":95995035648,"available":95926800384,"internally_reserved":0,"allocated":32034816,"data_stored":34729580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738487,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81704,"kb_used_data":46356,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93663448,"statfs":{"total":95995035648,"available":95911370752,"internally_reserved":0,"allocated":47468544,"data_stored":55456816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43311104,"data_stored":52768450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28192768,"data_stored":32360722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35528704,"data_stored":49804840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":72052736,"data_stored":97008459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":58441728,"data_stored":79500165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12705792,"data_stored":19409255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4079616,"data_stored":4066399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":44990464,"data_stored":55700663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":29626368,"data_stored":40876732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":52985856,"data_stored":83650014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2387968,"data_stored":2367002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24633344,"data_stored":34340197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:42:23.273 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-11T03:42:23.273 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:42:23.273 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-11T03:42:23.273 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-11T03:42:23.367 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:23.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:23.389 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:23.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:23.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:23.516 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:23.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:23.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:23.604 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:42:23.619 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:23.644 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:23.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:23.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:23.719 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:42:23.720 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:42:23.746 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:23.770 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:23.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:23.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:23.894 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:23.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:23.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:23.900 INFO:tasks.rados.rados.2.smithi175.stdout:1065: done (3 left) 2024-09-11T03:42:23.901 INFO:tasks.rados.rados.2.smithi175.stdout:1066: write (excl) oid 42 current snap is 117 2024-09-11T03:42:23.901 INFO:tasks.rados.rados.2.smithi175.stdout:1062: finishing write tid 2 to smithi17557870-45 2024-09-11T03:42:23.901 INFO:tasks.rados.rados.2.smithi175.stdout:1062: finishing write tid 3 to smithi17557870-45 2024-09-11T03:42:23.901 INFO:tasks.rados.rados.2.smithi175.stdout:1062: finishing write tid 4 to smithi17557870-45 2024-09-11T03:42:23.901 INFO:tasks.rados.rados.2.smithi175.stdout:1062: oid 45 updating version 0 to 277 2024-09-11T03:42:23.901 INFO:tasks.rados.rados.2.smithi175.stdout:1062: oid 45 version 277 is already newer than 276 2024-09-11T03:42:23.901 INFO:tasks.rados.rados.2.smithi175.stdout:1062: oid 45 updating version 277 to 278 2024-09-11T03:42:23.901 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 278 (ObjNum 375 snap 117 seq_num 375) dirty exists 2024-09-11T03:42:23.901 INFO:tasks.rados.rados.2.smithi175.stdout:1062: left oid 45 (ObjNum 375 snap 117 seq_num 375) 2024-09-11T03:42:23.902 INFO:tasks.rados.rados.2.smithi175.stdout:1064: finishing rollback tid 0 to smithi17557870-15 2024-09-11T03:42:23.902 INFO:tasks.rados.rados.2.smithi175.stdout:1066: seq_num 376 ranges {719432=794921,2186053=133379} 2024-09-11T03:42:23.908 INFO:tasks.rados.rados.2.smithi175.stdout:1066: writing smithi17557870-42 from 719432 to 1514353 tid 1 2024-09-11T03:42:23.913 INFO:tasks.rados.rados.2.smithi175.stdout:1066: writing smithi17557870-42 from 2186053 to 2319432 tid 2 2024-09-11T03:42:23.913 INFO:tasks.rados.rados.2.smithi175.stdout:1062: done (3 left) 2024-09-11T03:42:23.914 INFO:tasks.rados.rados.2.smithi175.stdout:1067: read oid 38 snap -1 2024-09-11T03:42:23.914 INFO:tasks.rados.rados.2.smithi175.stdout:1067: expect deleted 2024-09-11T03:42:23.914 INFO:tasks.rados.rados.2.smithi175.stdout:1068: read oid 39 snap -1 2024-09-11T03:42:23.914 INFO:tasks.rados.rados.2.smithi175.stdout:1068: expect (ObjNum 262 snap 66 seq_num 262) 2024-09-11T03:42:23.914 INFO:tasks.rados.rados.2.smithi175.stdout:1069: snap_remove snap 111 2024-09-11T03:42:23.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:24.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:24.024 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:24.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:24.126 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:24.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:23 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:42:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:23 smithi084 ceph-mon[27254]: 2.a scrub ok 2024-09-11T03:42:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:23 smithi084 ceph-mon[27254]: pgmap v643: 57 pgs: 57 active+clean; 207 MiB data, 764 MiB used, 893 GiB / 894 GiB avail; 15 MiB/s rd, 202 KiB/s wr, 11 op/s 2024-09-11T03:42:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:23 smithi084 ceph-mon[27254]: osdmap e326: 12 total, 12 up, 10 in 2024-09-11T03:42:24.147 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:24.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:24.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:24.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:23 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:42:24.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:23 smithi121 ceph-mon[30729]: 2.a scrub ok 2024-09-11T03:42:24.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:23 smithi121 ceph-mon[30729]: pgmap v643: 57 pgs: 57 active+clean; 207 MiB data, 764 MiB used, 893 GiB / 894 GiB avail; 15 MiB/s rd, 202 KiB/s wr, 11 op/s 2024-09-11T03:42:24.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:23 smithi121 ceph-mon[30729]: osdmap e326: 12 total, 12 up, 10 in 2024-09-11T03:42:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:23 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:42:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:23 smithi175 ceph-mon[30894]: 2.a scrub ok 2024-09-11T03:42:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:23 smithi175 ceph-mon[30894]: pgmap v643: 57 pgs: 57 active+clean; 207 MiB data, 764 MiB used, 893 GiB / 894 GiB avail; 15 MiB/s rd, 202 KiB/s wr, 11 op/s 2024-09-11T03:42:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:23 smithi175 ceph-mon[30894]: osdmap e326: 12 total, 12 up, 10 in 2024-09-11T03:42:24.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:24.274 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:24.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:24.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:24.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T03:42:24.376 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:24.401 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:24.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:24.402 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:24.503 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:24.525 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:24.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:24.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:24.627 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:24.649 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:24.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:24.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:24.751 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:24.776 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:24.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:24.777 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:24.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:24.905 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:24.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:24.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:24.907 INFO:tasks.rados.rados.2.smithi175.stdout:1069: done (5 left) 2024-09-11T03:42:24.907 INFO:tasks.rados.rados.2.smithi175.stdout:1070: rollback oid 46 current snap is 117 2024-09-11T03:42:24.907 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 46 to 99 2024-09-11T03:42:24.907 INFO:tasks.rados.rados.2.smithi175.stdout:1063: finishing rollback tid 0 to smithi17557870-14 2024-09-11T03:42:24.907 INFO:tasks.rados.rados.2.smithi175.stdout:1064: finishing rollback tid 1 to smithi17557870-15 2024-09-11T03:42:24.907 INFO:tasks.rados.rados.2.smithi175.stdout:1063: finishing rollback tid 1 to smithi17557870-14 2024-09-11T03:42:24.907 INFO:tasks.rados.rados.2.smithi175.stdout:1064: finishing rollback tid 2 to smithi17557870-15 2024-09-11T03:42:24.907 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 166 (ObjNum 317 snap 86 seq_num 317) dirty exists 2024-09-11T03:42:24.907 INFO:tasks.rados.rados.2.smithi175.stdout:1063: finishing rollback tid 2 to smithi17557870-14 2024-09-11T03:42:24.907 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 419 (ObjNum 258 snap 64 seq_num 258) dirty exists 2024-09-11T03:42:24.907 INFO:tasks.rados.rados.2.smithi175.stdout:1068: expect (ObjNum 262 snap 66 seq_num 262) 2024-09-11T03:42:25.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:25.012 INFO:tasks.rados.rados.2.smithi175.stdout:1066: finishing write tid 1 to smithi17557870-42 2024-09-11T03:42:25.013 INFO:tasks.rados.rados.2.smithi175.stdout:1066: finishing write tid 2 to smithi17557870-42 2024-09-11T03:42:25.013 INFO:tasks.rados.rados.2.smithi175.stdout:1066: finishing write tid 3 to smithi17557870-42 2024-09-11T03:42:25.013 INFO:tasks.rados.rados.2.smithi175.stdout:1066: finishing write tid 4 to smithi17557870-42 2024-09-11T03:42:25.013 INFO:tasks.rados.rados.2.smithi175.stdout:1066: oid 42 updating version 0 to 280 2024-09-11T03:42:25.013 INFO:tasks.rados.rados.2.smithi175.stdout:1066: oid 42 version 280 is already newer than 279 2024-09-11T03:42:25.013 INFO:tasks.rados.rados.2.smithi175.stdout:1066: oid 42 updating version 280 to 281 2024-09-11T03:42:25.013 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 281 (ObjNum 376 snap 117 seq_num 376) dirty exists 2024-09-11T03:42:25.013 INFO:tasks.rados.rados.2.smithi175.stdout:1066: left oid 42 (ObjNum 376 snap 117 seq_num 376) 2024-09-11T03:42:25.013 INFO:tasks.rados.rados.2.smithi175.stdout:1063: done (5 left) 2024-09-11T03:42:25.013 INFO:tasks.rados.rados.2.smithi175.stdout:1064: done (4 left) 2024-09-11T03:42:25.013 INFO:tasks.rados.rados.2.smithi175.stdout:1066: done (3 left) 2024-09-11T03:42:25.013 INFO:tasks.rados.rados.2.smithi175.stdout:1067: done (2 left) 2024-09-11T03:42:25.013 INFO:tasks.rados.rados.2.smithi175.stdout:1068: done (1 left) 2024-09-11T03:42:25.014 INFO:tasks.rados.rados.2.smithi175.stdout:1071: write oid 28 current snap is 117 2024-09-11T03:42:25.014 INFO:tasks.rados.rados.2.smithi175.stdout:1070: finishing rollback tid 0 to smithi17557870-46 2024-09-11T03:42:25.014 INFO:tasks.rados.rados.2.smithi175.stdout:1070: finishing rollback tid 1 to smithi17557870-46 2024-09-11T03:42:25.014 INFO:tasks.rados.rados.2.smithi175.stdout:1070: finishing rollback tid 2 to smithi17557870-46 2024-09-11T03:42:25.014 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 167 (ObjNum 267 snap 68 seq_num 267) dirty exists 2024-09-11T03:42:25.014 INFO:tasks.rados.rados.2.smithi175.stdout:1071: seq_num 377 ranges {736239=470170,1728182=571782} 2024-09-11T03:42:25.018 INFO:tasks.rados.rados.2.smithi175.stdout:1071: writing smithi17557870-28 from 736239 to 1206409 tid 1 2024-09-11T03:42:25.025 INFO:tasks.rados.rados.2.smithi175.stdout:1071: writing smithi17557870-28 from 1728182 to 2299964 tid 2 2024-09-11T03:42:25.025 INFO:tasks.rados.rados.2.smithi175.stdout:1070: done (1 left) 2024-09-11T03:42:25.025 INFO:tasks.rados.rados.2.smithi175.stdout:1072: copy_from oid 16 from oid 8 current snap is 117 2024-09-11T03:42:25.025 INFO:tasks.rados.rados.2.smithi175.stdout:1071: finishing write tid 1 to smithi17557870-28 2024-09-11T03:42:25.025 INFO:tasks.rados.rados.2.smithi175.stdout:1073: copy_from oid 45 from oid 15 current snap is 117 2024-09-11T03:42:25.025 INFO:tasks.rados.rados.2.smithi175.stdout:1074: copy_from oid 48 from oid 14 current snap is 117 2024-09-11T03:42:25.025 INFO:tasks.rados.rados.2.smithi175.stdout:1075: write oid 19 current snap is 117 2024-09-11T03:42:25.025 INFO:tasks.rados.rados.2.smithi175.stdout:1075: seq_num 378 ranges {753046=545419,1786664=431793} 2024-09-11T03:42:25.032 INFO:tasks.rados.rados.2.smithi175.stdout:1075: writing smithi17557870-19 from 753046 to 1298465 tid 1 2024-09-11T03:42:25.032 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:25.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:25.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:25.039 INFO:tasks.rados.rados.2.smithi175.stdout:1075: writing smithi17557870-19 from 1786664 to 2218457 tid 2 2024-09-11T03:42:25.039 INFO:tasks.rados.rados.2.smithi175.stdout:1076: read oid 46 snap -1 2024-09-11T03:42:25.039 INFO:tasks.rados.rados.2.smithi175.stdout:1076: expect (ObjNum 267 snap 68 seq_num 267) 2024-09-11T03:42:25.039 INFO:tasks.rados.rados.2.smithi175.stdout:1077: read oid 34 snap 113 2024-09-11T03:42:25.039 INFO:tasks.rados.rados.2.smithi175.stdout:1077: expect (ObjNum 293 snap 79 seq_num 293) 2024-09-11T03:42:25.039 INFO:tasks.rados.rados.2.smithi175.stdout:1078: rollback oid 35 current snap is 117 2024-09-11T03:42:25.039 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 35 to 99 2024-09-11T03:42:25.039 INFO:tasks.rados.rados.2.smithi175.stdout:1071: finishing write tid 2 to smithi17557870-28 2024-09-11T03:42:25.039 INFO:tasks.rados.rados.2.smithi175.stdout:1071: finishing write tid 3 to smithi17557870-28 2024-09-11T03:42:25.039 INFO:tasks.rados.rados.2.smithi175.stdout:1071: finishing write tid 4 to smithi17557870-28 2024-09-11T03:42:25.040 INFO:tasks.rados.rados.2.smithi175.stdout:1071: oid 28 updating version 0 to 405 2024-09-11T03:42:25.040 INFO:tasks.rados.rados.2.smithi175.stdout:1071: oid 28 updating version 405 to 407 2024-09-11T03:42:25.040 INFO:tasks.rados.rados.2.smithi175.stdout:1071: oid 28 version 407 is already newer than 406 2024-09-11T03:42:25.040 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 407 (ObjNum 377 snap 117 seq_num 377) dirty exists 2024-09-11T03:42:25.040 INFO:tasks.rados.rados.2.smithi175.stdout:1071: left oid 28 (ObjNum 377 snap 117 seq_num 377) 2024-09-11T03:42:25.040 INFO:tasks.rados.rados.2.smithi175.stdout:1073: finishing copy_from racing read to smithi17557870-45 2024-09-11T03:42:25.040 INFO:tasks.rados.rados.2.smithi175.stdout:1071: done (7 left) 2024-09-11T03:42:25.040 INFO:tasks.rados.rados.2.smithi175.stdout:1079: write (excl) oid 3 current snap is 117 2024-09-11T03:42:25.040 INFO:tasks.rados.rados.2.smithi175.stdout:1079: seq_num 379 ranges {769853=620668,1845146=524707} 2024-09-11T03:42:25.046 INFO:tasks.rados.rados.2.smithi175.stdout:1079: writing smithi17557870-3 from 769853 to 1390521 tid 1 2024-09-11T03:42:25.051 INFO:tasks.rados.rados.2.smithi175.stdout:1079: writing smithi17557870-3 from 1845146 to 2369853 tid 2 2024-09-11T03:42:25.052 INFO:tasks.rados.rados.2.smithi175.stdout:1080: write oid 26 current snap is 117 2024-09-11T03:42:25.052 INFO:tasks.rados.rados.2.smithi175.stdout:1080: seq_num 380 ranges {786660=695917,1903628=435462} 2024-09-11T03:42:25.059 INFO:tasks.rados.rados.2.smithi175.stdout:1080: writing smithi17557870-26 from 786660 to 1482577 tid 1 2024-09-11T03:42:25.064 INFO:tasks.rados.rados.2.smithi175.stdout:1080: writing smithi17557870-26 from 1903628 to 2339090 tid 2 2024-09-11T03:42:25.064 INFO:tasks.rados.rados.2.smithi175.stdout:1081: read oid 12 snap -1 2024-09-11T03:42:25.064 INFO:tasks.rados.rados.2.smithi175.stdout:1081: expect (ObjNum 335 snap 95 seq_num 335) 2024-09-11T03:42:25.064 INFO:tasks.rados.rados.2.smithi175.stdout:1082: rollback oid 40 current snap is 117 2024-09-11T03:42:25.064 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 40 to 91 2024-09-11T03:42:25.064 INFO:tasks.rados.rados.2.smithi175.stdout:1075: finishing write tid 1 to smithi17557870-19 2024-09-11T03:42:25.064 INFO:tasks.rados.rados.2.smithi175.stdout:1074: finishing copy_from racing read to smithi17557870-48 2024-09-11T03:42:25.065 INFO:tasks.rados.rados.2.smithi175.stdout:1072: finishing copy_from racing read to smithi17557870-16 2024-09-11T03:42:25.065 INFO:tasks.rados.rados.2.smithi175.stdout:1083: delete oid 33 current snap is 117 2024-09-11T03:42:25.065 INFO:tasks.rados.rados.2.smithi175.stdout:1076: expect (ObjNum 267 snap 68 seq_num 267) 2024-09-11T03:42:25.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:24 smithi084 ceph-mon[27254]: osdmap e327: 12 total, 12 up, 10 in 2024-09-11T03:42:25.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:25.137 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:25.141 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:25.157 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:25.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:25.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:25.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:24 smithi121 ceph-mon[30729]: osdmap e327: 12 total, 12 up, 10 in 2024-09-11T03:42:25.172 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:25.218 INFO:tasks.rados.rados.2.smithi175.stdout:1073: finishing copy_from to smithi17557870-45 2024-09-11T03:42:25.219 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 166 (ObjNum 317 snap 86 seq_num 317) dirty exists 2024-09-11T03:42:25.219 INFO:tasks.rados.rados.2.smithi175.stdout:1078: finishing rollback tid 0 to smithi17557870-35 2024-09-11T03:42:25.219 INFO:tasks.rados.rados.2.smithi175.stdout:1075: finishing write tid 2 to smithi17557870-19 2024-09-11T03:42:25.219 INFO:tasks.rados.rados.2.smithi175.stdout:1078: finishing rollback tid 1 to smithi17557870-35 2024-09-11T03:42:25.219 INFO:tasks.rados.rados.2.smithi175.stdout:1075: finishing write tid 3 to smithi17557870-19 2024-09-11T03:42:25.219 INFO:tasks.rados.rados.2.smithi175.stdout:1078: finishing rollback tid 2 to smithi17557870-35 2024-09-11T03:42:25.219 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 288 (ObjNum 153 snap 35 seq_num 153) dirty exists 2024-09-11T03:42:25.219 INFO:tasks.rados.rados.2.smithi175.stdout:1075: finishing write tid 4 to smithi17557870-19 2024-09-11T03:42:25.219 INFO:tasks.rados.rados.2.smithi175.stdout:1075: oid 19 updating version 0 to 194 2024-09-11T03:42:25.219 INFO:tasks.rados.rados.2.smithi175.stdout:1075: oid 19 updating version 194 to 195 2024-09-11T03:42:25.219 INFO:tasks.rados.rados.2.smithi175.stdout:1075: oid 19 updating version 195 to 196 2024-09-11T03:42:25.219 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 196 (ObjNum 378 snap 117 seq_num 378) dirty exists 2024-09-11T03:42:25.219 INFO:tasks.rados.rados.2.smithi175.stdout:1075: left oid 19 (ObjNum 378 snap 117 seq_num 378) 2024-09-11T03:42:25.220 INFO:tasks.rados.rados.2.smithi175.stdout:1072: finishing copy_from to smithi17557870-16 2024-09-11T03:42:25.220 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 401 (ObjNum 319 snap 86 seq_num 319) dirty exists 2024-09-11T03:42:25.220 INFO:tasks.rados.rados.2.smithi175.stdout:1074: finishing copy_from to smithi17557870-48 2024-09-11T03:42:25.220 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 419 (ObjNum 258 snap 64 seq_num 258) dirty exists 2024-09-11T03:42:25.220 INFO:tasks.rados.rados.2.smithi175.stdout:1077: expect (ObjNum 293 snap 79 seq_num 293) 2024-09-11T03:42:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:24 smithi175 ceph-mon[30894]: osdmap e327: 12 total, 12 up, 10 in 2024-09-11T03:42:25.259 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:25.280 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:25.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:25.281 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:25.315 INFO:tasks.rados.rados.2.smithi175.stdout:1079: finishing write tid 1 to smithi17557870-3 2024-09-11T03:42:25.315 INFO:tasks.rados.rados.2.smithi175.stdout:1079: finishing write tid 2 to smithi17557870-3 2024-09-11T03:42:25.315 INFO:tasks.rados.rados.2.smithi175.stdout:1079: finishing write tid 3 to smithi17557870-3 2024-09-11T03:42:25.315 INFO:tasks.rados.rados.2.smithi175.stdout:1079: finishing write tid 4 to smithi17557870-3 2024-09-11T03:42:25.315 INFO:tasks.rados.rados.2.smithi175.stdout:1079: oid 3 updating version 0 to 410 2024-09-11T03:42:25.315 INFO:tasks.rados.rados.2.smithi175.stdout:1079: oid 3 version 410 is already newer than 408 2024-09-11T03:42:25.315 INFO:tasks.rados.rados.2.smithi175.stdout:1079: oid 3 version 410 is already newer than 409 2024-09-11T03:42:25.315 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 410 (ObjNum 379 snap 117 seq_num 379) dirty exists 2024-09-11T03:42:25.315 INFO:tasks.rados.rados.2.smithi175.stdout:1079: left oid 3 (ObjNum 379 snap 117 seq_num 379) 2024-09-11T03:42:25.315 INFO:tasks.rados.rados.2.smithi175.stdout:1080: finishing write tid 1 to smithi17557870-26 2024-09-11T03:42:25.316 INFO:tasks.rados.rados.2.smithi175.stdout:1072: done (11 left) 2024-09-11T03:42:25.316 INFO:tasks.rados.rados.2.smithi175.stdout:1073: done (10 left) 2024-09-11T03:42:25.316 INFO:tasks.rados.rados.2.smithi175.stdout:1074: done (9 left) 2024-09-11T03:42:25.316 INFO:tasks.rados.rados.2.smithi175.stdout:1075: done (8 left) 2024-09-11T03:42:25.316 INFO:tasks.rados.rados.2.smithi175.stdout:1076: done (7 left) 2024-09-11T03:42:25.316 INFO:tasks.rados.rados.2.smithi175.stdout:1077: done (6 left) 2024-09-11T03:42:25.316 INFO:tasks.rados.rados.2.smithi175.stdout:1078: done (5 left) 2024-09-11T03:42:25.316 INFO:tasks.rados.rados.2.smithi175.stdout:1079: done (4 left) 2024-09-11T03:42:25.316 INFO:tasks.rados.rados.2.smithi175.stdout:1083: done (3 left) 2024-09-11T03:42:25.316 INFO:tasks.rados.rados.2.smithi175.stdout:1084: snap_remove snap 113 2024-09-11T03:42:25.382 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:25.403 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:25.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:25.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:25.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:25.530 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:25.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:25.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:25.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:25.657 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:25.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:25.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:25.759 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:25.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:25.785 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:25.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:25.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:25.908 INFO:tasks.rados.rados.2.smithi175.stdout:1084: done (3 left) 2024-09-11T03:42:25.908 INFO:tasks.rados.rados.2.smithi175.stdout:1085: snap_remove snap 114 2024-09-11T03:42:25.913 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:25.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:25.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:26.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:26.027 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:25 smithi084 ceph-mon[27254]: pgmap v646: 57 pgs: 57 active+clean; 204 MiB data, 761 MiB used, 893 GiB / 894 GiB avail; 18 MiB/s rd, 711 KiB/s wr, 15 op/s 2024-09-11T03:42:26.027 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:25 smithi084 ceph-mon[27254]: osdmap e328: 12 total, 12 up, 10 in 2024-09-11T03:42:26.027 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:25 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:42:26.027 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:25 smithi084 ceph-mon[27254]: 2.d scrub ok 2024-09-11T03:42:26.036 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:26.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:26.118 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:42:26.118 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:42:26.118 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 329, 2024-09-11T03:42:26.118 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "10fe7946-6fee-11ef-bcea-c7b262605968", 2024-09-11T03:42:26.118 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-09-11T03:30:30.532652+0000", 2024-09-11T03:42:26.118 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-09-11T03:42:25.901047+0000", 2024-09-11T03:42:26.118 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-09-11T03:37:26.787638+0000", 2024-09-11T03:42:26.118 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-09-11T03:40:04.952764+0000", 2024-09-11T03:42:26.118 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T03:42:26.118 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5804032, 2024-09-11T03:42:26.119 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-09-11T03:42:26.119 INFO:teuthology.orchestra.run.smithi084.stdout: "nodeep-scrub", 2024-09-11T03:42:26.119 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-09-11T03:42:26.119 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-09-11T03:42:26.119 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-09-11T03:42:26.119 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-09-11T03:42:26.119 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:42:26.119 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 26, 2024-09-11T03:42:26.119 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T03:42:26.119 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T03:42:26.119 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T03:42:26.119 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-09-11T03:42:26.119 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 2, 2024-09-11T03:42:26.119 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 12, 2024-09-11T03:42:26.120 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-09-11T03:42:26.120 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "jewel", 2024-09-11T03:42:26.120 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-09-11T03:42:26.120 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-09-11T03:42:26.120 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-09-11T03:42:26.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.120 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-09-11T03:42:26.120 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-09-11T03:42:26.120 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-11T03:33:42.916933+0000", 2024-09-11T03:42:26.120 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-11T03:42:26.120 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-11T03:42:26.120 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-11T03:42:26.120 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-11T03:42:26.121 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-11T03:42:26.121 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-11T03:42:26.121 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-11T03:42:26.121 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-11T03:42:26.121 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T03:42:26.121 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T03:42:26.121 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-11T03:42:26.121 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-11T03:42:26.121 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-11T03:42:26.121 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 31, 2024-09-11T03:42:26.121 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 31, 2024-09-11T03:42:26.121 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 31, 2024-09-11T03:42:26.121 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 31, 2024-09-11T03:42:26.122 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 31, 2024-09-11T03:42:26.122 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-11T03:42:26.122 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-11T03:42:26.122 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-11T03:42:26.122 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-11T03:42:26.122 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-11T03:42:26.122 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-11T03:42:26.122 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-11T03:42:26.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.122 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "222", 2024-09-11T03:42:26.122 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-11T03:42:26.122 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T03:42:26.122 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "220", 2024-09-11T03:42:26.122 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-11T03:42:26.123 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-11T03:42:26.123 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-11T03:42:26.123 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-11T03:42:26.123 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-11T03:42:26.123 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-11T03:42:26.123 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-11T03:42:26.123 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-11T03:42:26.123 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-11T03:42:26.123 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-11T03:42:26.123 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-11T03:42:26.123 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-11T03:42:26.123 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-11T03:42:26.123 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-11T03:42:26.123 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-11T03:42:26.124 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T03:42:26.124 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T03:42:26.124 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T03:42:26.124 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-11T03:42:26.124 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-11T03:42:26.124 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-11T03:42:26.124 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-11T03:42:26.124 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-11T03:42:26.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.124 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-11T03:42:26.124 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-11T03:42:26.124 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-11T03:42:26.125 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-11T03:42:26.125 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-11T03:42:26.125 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T03:42:26.125 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-11T03:42:26.125 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-11T03:42:26.125 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-11T03:42:26.125 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-11T03:42:26.125 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-11T03:42:26.125 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-11T03:42:26.125 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-09-11T03:42:26.125 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-09-11T03:42:26.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.125 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-11T03:42:26.126 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-09-11T03:42:26.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.126 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-11T03:42:26.126 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-11T03:42:26.126 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.940000057220459, 2024-09-11T03:42:26.126 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.940000057220459, 2024-09-11T03:42:26.126 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.82999998331069946, 2024-09-11T03:42:26.126 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.6100000143051147, 2024-09-11T03:42:26.126 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.6100000143051147, 2024-09-11T03:42:26.126 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-11T03:42:26.126 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-11T03:42:26.126 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-11T03:42:26.126 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.127 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-11T03:42:26.127 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_0", 2024-09-11T03:42:26.127 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-11T03:38:28.163534+0000", 2024-09-11T03:42:26.127 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-11T03:42:26.127 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T03:42:26.127 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-11T03:42:26.127 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-11T03:42:26.127 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-11T03:42:26.127 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-11T03:42:26.127 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-11T03:42:26.127 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-11T03:42:26.127 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T03:42:26.127 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T03:42:26.128 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-11T03:42:26.128 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-11T03:42:26.128 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-11T03:42:26.128 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 26, 2024-09-11T03:42:26.128 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 26, 2024-09-11T03:42:26.128 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 26, 2024-09-11T03:42:26.128 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 26, 2024-09-11T03:42:26.128 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 26, 2024-09-11T03:42:26.128 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-11T03:42:26.128 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-11T03:42:26.128 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-11T03:42:26.128 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-11T03:42:26.128 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-11T03:42:26.128 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-11T03:42:26.129 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-11T03:42:26.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.129 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "329", 2024-09-11T03:42:26.129 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-11T03:42:26.129 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T03:42:26.129 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "276", 2024-09-11T03:42:26.129 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-11T03:42:26.129 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-11T03:42:26.129 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 118, 2024-09-11T03:42:26.129 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 329, 2024-09-11T03:42:26.129 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-11T03:42:26.129 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-11T03:42:26.129 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-11T03:42:26.130 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-11T03:42:26.130 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-11T03:42:26.130 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-11T03:42:26.130 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-11T03:42:26.130 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-11T03:42:26.130 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-11T03:42:26.130 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-11T03:42:26.130 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-11T03:42:26.130 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T03:42:26.130 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T03:42:26.130 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T03:42:26.130 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-11T03:42:26.130 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-11T03:42:26.130 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-11T03:42:26.131 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-11T03:42:26.131 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-11T03:42:26.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.131 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-11T03:42:26.131 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-11T03:42:26.131 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-11T03:42:26.131 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-11T03:42:26.131 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-11T03:42:26.131 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T03:42:26.131 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-11T03:42:26.131 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-11T03:42:26.131 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-11T03:42:26.131 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-11T03:42:26.132 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-11T03:42:26.132 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-11T03:42:26.132 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-11T03:42:26.132 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-09-11T03:42:26.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.132 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-11T03:42:26.132 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-11T03:42:26.132 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.309999942779541, 2024-09-11T03:42:26.132 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.309999942779541, 2024-09-11T03:42:26.132 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.82999998331069946, 2024-09-11T03:42:26.132 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.9199999570846558, 2024-09-11T03:42:26.132 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.9199999570846558, 2024-09-11T03:42:26.132 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-11T03:42:26.132 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-11T03:42:26.133 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-11T03:42:26.133 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.133 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.133 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:42:26.133 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-11T03:42:26.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.133 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-11T03:42:26.133 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "9a431e48-e46b-44f9-bfa7-cb56c1484e1c", 2024-09-11T03:42:26.133 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:42:26.133 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:42:26.133 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:42:26.133 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:42:26.133 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:42:26.134 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:42:26.134 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 8, 2024-09-11T03:42:26.134 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 280, 2024-09-11T03:42:26.136 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:42:26.136 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:42:26.136 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:42:26.136 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.136 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.136 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-09-11T03:42:26.136 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:42:26.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.137 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.137 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-09-11T03:42:26.137 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:42:26.137 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.137 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.138 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:42:26.138 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.138 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.139 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-09-11T03:42:26.139 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:42:26.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.139 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.139 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-09-11T03:42:26.139 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:42:26.139 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.139 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.139 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:42:26.140 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.140 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.140 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-09-11T03:42:26.140 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:42:26.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.140 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.140 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-09-11T03:42:26.140 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:42:26.140 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.140 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.141 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:42:26.141 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.141 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.141 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-09-11T03:42:26.141 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:42:26.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.141 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.141 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-09-11T03:42:26.141 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:42:26.141 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.142 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.142 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6803/1376667411", 2024-09-11T03:42:26.142 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6805/1376667411", 2024-09-11T03:42:26.142 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6809/1376667411", 2024-09-11T03:42:26.142 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/1376667411", 2024-09-11T03:42:26.142 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:42:26.142 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:42:26.142 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:42:26.142 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.142 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-11T03:42:26.143 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "62fd6e25-3508-40b4-a0f4-f7de137e3bb5", 2024-09-11T03:42:26.143 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:42:26.143 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:42:26.143 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:42:26.143 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:42:26.143 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:42:26.143 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:42:26.143 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-09-11T03:42:26.143 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 280, 2024-09-11T03:42:26.143 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:42:26.143 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:42:26.143 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:42:26.143 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.144 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.144 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-09-11T03:42:26.144 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:42:26.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.144 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.144 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-09-11T03:42:26.144 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:42:26.144 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.144 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.144 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:42:26.144 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.145 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.145 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-09-11T03:42:26.145 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:42:26.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.145 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.145 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-09-11T03:42:26.145 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:42:26.145 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.145 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.145 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:42:26.146 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.146 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.146 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-09-11T03:42:26.146 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:42:26.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.146 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.146 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6817", 2024-09-11T03:42:26.146 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:42:26.146 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.146 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.147 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:42:26.147 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.147 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.147 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-09-11T03:42:26.147 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:42:26.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.147 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.147 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-09-11T03:42:26.147 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:42:26.147 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.147 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.148 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6811/3138939249", 2024-09-11T03:42:26.148 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6813/3138939249", 2024-09-11T03:42:26.148 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6817/3138939249", 2024-09-11T03:42:26.148 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/3138939249", 2024-09-11T03:42:26.148 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:42:26.148 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:42:26.148 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:42:26.148 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.148 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-11T03:42:26.148 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ff351425-4715-49dc-be49-f27bf205d57e", 2024-09-11T03:42:26.149 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:42:26.149 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:42:26.149 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:42:26.149 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:42:26.149 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:42:26.149 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:42:26.149 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 18, 2024-09-11T03:42:26.149 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 281, 2024-09-11T03:42:26.149 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:42:26.149 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:42:26.149 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:42:26.149 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.150 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.150 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6818", 2024-09-11T03:42:26.150 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:42:26.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.150 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.150 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6819", 2024-09-11T03:42:26.150 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:42:26.150 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.150 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.151 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:42:26.151 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.151 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.151 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6820", 2024-09-11T03:42:26.151 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:42:26.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.151 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.151 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6821", 2024-09-11T03:42:26.152 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:42:26.152 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.152 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.152 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:42:26.152 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.152 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.152 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6824", 2024-09-11T03:42:26.152 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:42:26.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.153 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.153 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6825", 2024-09-11T03:42:26.153 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:42:26.153 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.153 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.153 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:42:26.153 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.153 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.154 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6822", 2024-09-11T03:42:26.154 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:42:26.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.154 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.154 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6823", 2024-09-11T03:42:26.154 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:42:26.154 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.154 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.155 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6819/3001811591", 2024-09-11T03:42:26.155 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6821/3001811591", 2024-09-11T03:42:26.155 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6825/3001811591", 2024-09-11T03:42:26.155 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6823/3001811591", 2024-09-11T03:42:26.155 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:42:26.155 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:42:26.155 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:42:26.155 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.155 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-11T03:42:26.155 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "2518a93e-c612-4731-91ce-5685c25ffa9f", 2024-09-11T03:42:26.156 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:42:26.156 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:42:26.156 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:42:26.156 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:42:26.156 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:42:26.156 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:42:26.156 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 26, 2024-09-11T03:42:26.156 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 312, 2024-09-11T03:42:26.156 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:42:26.156 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:42:26.156 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:42:26.156 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.157 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.157 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6826", 2024-09-11T03:42:26.157 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:42:26.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.157 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.157 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6827", 2024-09-11T03:42:26.157 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:42:26.157 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.157 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.157 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:42:26.157 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.158 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.158 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6828", 2024-09-11T03:42:26.158 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:42:26.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.158 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.158 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6829", 2024-09-11T03:42:26.158 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:42:26.158 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.158 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.158 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:42:26.159 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.159 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.159 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6832", 2024-09-11T03:42:26.159 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:42:26.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.159 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.159 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6833", 2024-09-11T03:42:26.159 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:42:26.159 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.159 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.160 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:42:26.160 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.160 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.160 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6830", 2024-09-11T03:42:26.160 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:42:26.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.160 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.160 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6831", 2024-09-11T03:42:26.160 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:42:26.160 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.160 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.161 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6827/302512129", 2024-09-11T03:42:26.161 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6829/302512129", 2024-09-11T03:42:26.161 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6833/302512129", 2024-09-11T03:42:26.161 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6831/302512129", 2024-09-11T03:42:26.161 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:42:26.161 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:42:26.161 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:42:26.161 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.161 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-11T03:42:26.161 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d12f3c2d-95cc-4276-a7e1-bee982d58948", 2024-09-11T03:42:26.162 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:42:26.162 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:42:26.162 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:42:26.162 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:42:26.162 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:42:26.162 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:42:26.162 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-09-11T03:42:26.162 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 281, 2024-09-11T03:42:26.162 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:42:26.162 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:42:26.162 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:42:26.162 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.163 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.163 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6800", 2024-09-11T03:42:26.163 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:42:26.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.163 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.163 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6801", 2024-09-11T03:42:26.163 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:42:26.163 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.163 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.163 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:42:26.163 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.164 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.164 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6802", 2024-09-11T03:42:26.164 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:42:26.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.164 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.164 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6803", 2024-09-11T03:42:26.164 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:42:26.164 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.164 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.164 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:42:26.165 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.165 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.165 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6806", 2024-09-11T03:42:26.165 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:42:26.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.165 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.165 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6807", 2024-09-11T03:42:26.165 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:42:26.165 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.165 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.166 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:42:26.166 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.166 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.166 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6804", 2024-09-11T03:42:26.166 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:42:26.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.166 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.166 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6805", 2024-09-11T03:42:26.166 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:42:26.166 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.166 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.167 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.121:6801/3873214513", 2024-09-11T03:42:26.167 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.121:6803/3873214513", 2024-09-11T03:42:26.167 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.121:6807/3873214513", 2024-09-11T03:42:26.167 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.121:6805/3873214513", 2024-09-11T03:42:26.167 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:42:26.167 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:42:26.167 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:42:26.167 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.167 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-11T03:42:26.167 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "32164dd5-7e8f-4925-8526-bbb1e33e3364", 2024-09-11T03:42:26.168 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:42:26.168 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:42:26.168 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:42:26.168 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:42:26.168 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:42:26.168 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:42:26.168 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 38, 2024-09-11T03:42:26.168 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 281, 2024-09-11T03:42:26.168 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:42:26.168 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:42:26.168 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:42:26.168 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.169 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.169 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6808", 2024-09-11T03:42:26.169 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:42:26.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.169 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.169 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6809", 2024-09-11T03:42:26.169 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:42:26.169 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.169 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.169 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:42:26.169 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.170 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.170 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6810", 2024-09-11T03:42:26.170 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:42:26.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.170 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.170 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6811", 2024-09-11T03:42:26.170 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:42:26.170 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.170 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.170 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:42:26.170 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.171 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.171 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6814", 2024-09-11T03:42:26.171 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:42:26.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.171 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.171 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6815", 2024-09-11T03:42:26.171 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:42:26.171 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.171 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.171 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:42:26.172 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.172 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.172 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6812", 2024-09-11T03:42:26.172 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:42:26.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.172 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.172 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6813", 2024-09-11T03:42:26.172 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:42:26.172 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.172 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.173 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.121:6809/918612194", 2024-09-11T03:42:26.173 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.121:6811/918612194", 2024-09-11T03:42:26.173 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.121:6815/918612194", 2024-09-11T03:42:26.173 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.121:6813/918612194", 2024-09-11T03:42:26.173 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:42:26.173 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:42:26.173 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:42:26.173 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.173 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-11T03:42:26.173 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "3f107a88-803f-4cd8-804a-241310c282ea", 2024-09-11T03:42:26.173 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:42:26.174 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-09-11T03:42:26.174 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-09-11T03:42:26.174 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:42:26.174 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:42:26.174 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:42:26.174 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 45, 2024-09-11T03:42:26.174 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 95, 2024-09-11T03:42:26.174 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:42:26.174 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:42:26.174 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:42:26.174 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.174 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.174 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6816", 2024-09-11T03:42:26.175 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:42:26.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.175 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.175 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6817", 2024-09-11T03:42:26.175 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:42:26.175 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.175 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.175 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:42:26.175 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.175 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.176 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6818", 2024-09-11T03:42:26.176 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:42:26.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.176 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.176 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6819", 2024-09-11T03:42:26.176 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:42:26.176 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.176 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.176 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:42:26.176 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.177 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.177 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6822", 2024-09-11T03:42:26.177 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:42:26.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.177 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.177 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6823", 2024-09-11T03:42:26.177 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:42:26.177 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.177 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.177 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:42:26.177 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.178 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.178 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6820", 2024-09-11T03:42:26.178 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:42:26.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.178 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.178 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6821", 2024-09-11T03:42:26.178 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:42:26.178 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.179 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.179 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.121:6817/3384813363", 2024-09-11T03:42:26.179 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.121:6819/3384813363", 2024-09-11T03:42:26.179 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.121:6823/3384813363", 2024-09-11T03:42:26.179 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.121:6821/3384813363", 2024-09-11T03:42:26.179 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:42:26.179 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:42:26.179 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:42:26.179 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.179 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-11T03:42:26.180 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "dfee7f95-1fb9-4493-9907-cb10d1837bdb", 2024-09-11T03:42:26.180 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:42:26.180 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:42:26.180 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:42:26.180 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:42:26.180 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:42:26.180 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:42:26.180 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 52, 2024-09-11T03:42:26.180 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 309, 2024-09-11T03:42:26.180 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:42:26.180 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:42:26.180 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:42:26.181 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.181 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.181 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6824", 2024-09-11T03:42:26.181 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:42:26.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.181 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.181 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6825", 2024-09-11T03:42:26.181 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:42:26.181 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.181 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.182 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:42:26.182 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.182 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.182 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6826", 2024-09-11T03:42:26.182 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:42:26.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.182 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.182 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6827", 2024-09-11T03:42:26.182 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:42:26.182 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.182 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.183 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:42:26.183 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.183 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.183 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6830", 2024-09-11T03:42:26.183 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:42:26.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.183 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.183 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6831", 2024-09-11T03:42:26.183 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:42:26.183 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.183 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.184 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:42:26.184 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.184 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.184 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6828", 2024-09-11T03:42:26.184 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:42:26.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.184 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.184 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6829", 2024-09-11T03:42:26.184 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:42:26.184 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.185 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.185 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.121:6825/449222054", 2024-09-11T03:42:26.185 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.121:6827/449222054", 2024-09-11T03:42:26.185 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.121:6831/449222054", 2024-09-11T03:42:26.185 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.121:6829/449222054", 2024-09-11T03:42:26.185 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:42:26.185 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:42:26.185 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:42:26.185 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.185 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-09-11T03:42:26.186 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4", 2024-09-11T03:42:26.186 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:42:26.186 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:42:26.186 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:42:26.186 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:42:26.186 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:42:26.186 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:42:26.186 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 60, 2024-09-11T03:42:26.186 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 278, 2024-09-11T03:42:26.186 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:42:26.186 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:42:26.186 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:42:26.186 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.187 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.187 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6800", 2024-09-11T03:42:26.187 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:42:26.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.187 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.187 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6801", 2024-09-11T03:42:26.187 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:42:26.187 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.187 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.187 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:42:26.187 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.188 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.188 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6802", 2024-09-11T03:42:26.188 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:42:26.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.188 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.188 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6803", 2024-09-11T03:42:26.188 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:42:26.188 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.188 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.188 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:42:26.188 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.189 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.189 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6806", 2024-09-11T03:42:26.189 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:42:26.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.189 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.189 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6807", 2024-09-11T03:42:26.189 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:42:26.189 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.189 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.189 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:42:26.190 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.190 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.190 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6804", 2024-09-11T03:42:26.190 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:42:26.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.190 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.190 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6805", 2024-09-11T03:42:26.190 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:42:26.190 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.190 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.190 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.175:6801/2660128384", 2024-09-11T03:42:26.191 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.175:6803/2660128384", 2024-09-11T03:42:26.191 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.175:6807/2660128384", 2024-09-11T03:42:26.191 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.175:6805/2660128384", 2024-09-11T03:42:26.191 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:42:26.191 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:42:26.191 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:42:26.191 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.191 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-09-11T03:42:26.191 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "50162e42-0443-41d1-8709-a0db61c087da", 2024-09-11T03:42:26.191 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:42:26.191 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:42:26.191 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:42:26.192 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:42:26.192 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:42:26.192 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:42:26.192 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 69, 2024-09-11T03:42:26.192 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 312, 2024-09-11T03:42:26.192 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:42:26.192 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:42:26.192 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:42:26.192 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.192 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.192 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6808", 2024-09-11T03:42:26.192 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:42:26.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.193 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.193 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6809", 2024-09-11T03:42:26.193 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:42:26.193 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.193 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.193 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:42:26.193 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.193 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.193 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6810", 2024-09-11T03:42:26.193 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:42:26.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.194 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.194 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6811", 2024-09-11T03:42:26.194 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:42:26.194 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.194 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.194 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:42:26.194 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.194 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.194 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6814", 2024-09-11T03:42:26.194 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:42:26.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.195 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.195 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6815", 2024-09-11T03:42:26.195 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:42:26.195 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.195 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.195 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:42:26.195 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.195 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.195 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6812", 2024-09-11T03:42:26.196 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:42:26.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.196 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.196 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6813", 2024-09-11T03:42:26.196 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:42:26.196 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.196 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.196 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.175:6809/1270991649", 2024-09-11T03:42:26.196 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.175:6811/1270991649", 2024-09-11T03:42:26.196 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.175:6815/1270991649", 2024-09-11T03:42:26.196 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.175:6813/1270991649", 2024-09-11T03:42:26.196 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:42:26.197 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:42:26.197 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:42:26.197 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:26.198 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:25 smithi121 ceph-mon[30729]: pgmap v646: 57 pgs: 57 active+clean; 204 MiB data, 761 MiB used, 893 GiB / 894 GiB avail; 18 MiB/s rd, 711 KiB/s wr, 15 op/s 2024-09-11T03:42:26.198 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:25 smithi121 ceph-mon[30729]: osdmap e328: 12 total, 12 up, 10 in 2024-09-11T03:42:26.198 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:25 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:42:26.198 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:25 smithi121 ceph-mon[30729]: 2.d scrub ok 2024-09-11T03:42:26.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.199 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-09-11T03:42:26.199 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4b9426e1-bac8-47d3-ad58-54b89e8d1eb1", 2024-09-11T03:42:26.199 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:42:26.199 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-09-11T03:42:26.199 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-09-11T03:42:26.199 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:42:26.199 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:42:26.199 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:42:26.199 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 76, 2024-09-11T03:42:26.200 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 182, 2024-09-11T03:42:26.200 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:42:26.200 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:42:26.200 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:42:26.200 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.200 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.200 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6816", 2024-09-11T03:42:26.200 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:42:26.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.200 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.200 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6817", 2024-09-11T03:42:26.201 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:42:26.201 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.201 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.201 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:42:26.201 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.201 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.201 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6818", 2024-09-11T03:42:26.201 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:42:26.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.201 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.202 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6819", 2024-09-11T03:42:26.202 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:42:26.202 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.202 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.202 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:42:26.202 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.202 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.202 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6822", 2024-09-11T03:42:26.202 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:42:26.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.202 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.203 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6823", 2024-09-11T03:42:26.203 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:42:26.203 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.203 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.203 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:42:26.203 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.203 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.203 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6820", 2024-09-11T03:42:26.203 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:42:26.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.204 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.204 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6821", 2024-09-11T03:42:26.204 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:42:26.204 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.204 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.204 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.175:6817/3877631021", 2024-09-11T03:42:26.204 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.175:6819/3877631021", 2024-09-11T03:42:26.204 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.175:6823/3877631021", 2024-09-11T03:42:26.204 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.175:6821/3877631021", 2024-09-11T03:42:26.204 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:42:26.204 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:42:26.204 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:42:26.205 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.205 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-09-11T03:42:26.205 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "cd1a28a1-413c-4274-9876-e1bdb7703be6", 2024-09-11T03:42:26.205 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:42:26.205 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:42:26.205 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:42:26.205 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:42:26.205 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:42:26.205 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:42:26.205 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 83, 2024-09-11T03:42:26.205 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 222, 2024-09-11T03:42:26.205 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:42:26.206 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:42:26.206 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:42:26.206 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.206 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.206 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6824", 2024-09-11T03:42:26.206 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:42:26.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.206 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.206 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6825", 2024-09-11T03:42:26.206 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:42:26.206 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.207 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.207 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:42:26.207 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.207 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.207 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6826", 2024-09-11T03:42:26.207 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:42:26.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.207 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.207 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6827", 2024-09-11T03:42:26.207 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:42:26.208 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.208 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.208 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:42:26.208 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.208 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.208 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6830", 2024-09-11T03:42:26.208 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:42:26.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.208 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.208 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6831", 2024-09-11T03:42:26.209 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:42:26.209 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.209 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.209 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:42:26.209 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:42:26.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.209 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:42:26.209 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6828", 2024-09-11T03:42:26.209 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:42:26.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.209 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:42:26.210 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6829", 2024-09-11T03:42:26.210 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:42:26.210 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.210 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.210 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.175:6825/1150051236", 2024-09-11T03:42:26.210 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.175:6827/1150051236", 2024-09-11T03:42:26.210 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.175:6831/1150051236", 2024-09-11T03:42:26.210 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.175:6829/1150051236", 2024-09-11T03:42:26.210 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:42:26.210 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:42:26.210 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:42:26.210 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.211 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.211 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:42:26.211 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-09-11T03:42:26.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.211 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-11T03:42:26.211 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:42:26.211 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:42:26.211 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:42:26.211 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:42:26.211 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:42:26.211 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:42:19.799941+0000", 2024-09-11T03:42:26.211 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:42:26.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.212 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-11T03:42:26.212 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:42:26.212 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:42:26.212 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:42:26.212 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:42:26.212 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:42:26.212 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:41:47.349033+0000", 2024-09-11T03:42:26.212 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:42:26.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.212 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-11T03:42:26.212 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:42:26.213 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:42:26.213 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:42:26.213 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:42:26.213 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:42:26.213 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:42:02.898695+0000", 2024-09-11T03:42:26.213 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:42:26.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.213 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-11T03:42:26.213 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:42:26.213 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:42:26.213 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:42:26.213 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:42:26.213 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:42:26.214 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:41:28.941316+0000", 2024-09-11T03:42:26.214 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:42:26.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.214 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-11T03:42:26.214 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:42:26.214 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:42:26.214 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:42:26.214 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:42:26.214 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:42:26.214 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:42:08.783166+0000", 2024-09-11T03:42:26.214 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:42:26.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.215 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-11T03:42:26.215 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:42:26.215 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:42:26.215 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:42:26.215 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:42:26.215 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:42:26.215 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:42:25.432246+0000", 2024-09-11T03:42:26.215 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:42:26.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.216 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-11T03:42:26.216 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:42:26.216 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:42:26.216 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:42:26.216 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:42:26.216 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-09-11T03:42:26.216 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:42:13.715916+0000", 2024-09-11T03:42:26.216 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:42:26.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.216 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-11T03:42:26.216 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:42:26.217 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:42:26.217 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:42:26.217 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:42:26.217 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:42:26.217 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:41:42.942142+0000", 2024-09-11T03:42:26.217 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:42:26.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.217 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-09-11T03:42:26.217 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:42:26.217 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:42:26.217 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:42:26.217 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:42:26.217 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:42:26.218 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:41:31.651575+0000", 2024-09-11T03:42:26.218 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:42:26.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.218 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-09-11T03:42:26.218 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:42:26.218 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:42:26.218 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:42:26.218 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:42:26.218 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:42:26.218 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:41:36.385506+0000", 2024-09-11T03:42:26.218 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:42:26.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.219 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-09-11T03:42:26.219 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:42:26.219 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:42:26.219 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:42:26.219 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:42:26.219 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-09-11T03:42:26.219 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:41:49.805801+0000", 2024-09-11T03:42:26.219 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:42:26.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.219 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-09-11T03:42:26.219 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:42:26.219 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:42:26.220 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:42:26.220 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:42:26.220 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:42:26.220 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:42:15.880811+0000", 2024-09-11T03:42:26.220 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:42:26.220 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.220 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:42:26.220 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [], 2024-09-11T03:42:26.220 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [], 2024-09-11T03:42:26.220 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-09-11T03:42:26.220 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-09-11T03:42:26.220 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-09-11T03:42:26.220 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-09-11T03:42:26.221 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/108136515": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:42:26.221 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3411515187": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:42:26.221 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6801/3365920551": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:42:26.221 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/3365920551": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:42:26.221 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3885577441": "2024-09-12T03:31:03.077186+0000", 2024-09-11T03:42:26.221 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/22039254": "2024-09-12T03:31:03.077186+0000", 2024-09-11T03:42:26.221 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6801/3496645948": "2024-09-12T03:31:03.077186+0000", 2024-09-11T03:42:26.221 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4206337830": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:42:26.221 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/3496645948": "2024-09-12T03:31:03.077186+0000", 2024-09-11T03:42:26.221 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1989775875": "2024-09-12T03:31:03.077186+0000" 2024-09-11T03:42:26.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.221 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-09-11T03:42:26.221 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-09-11T03:42:26.221 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-09-11T03:42:26.222 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-09-11T03:42:26.222 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-09-11T03:42:26.222 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-09-11T03:42:26.222 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-09-11T03:42:26.222 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-09-11T03:42:26.222 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.222 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [ 2024-09-11T03:42:26.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.222 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-11T03:42:26.222 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-11T03:42:26.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.222 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 104, 2024-09-11T03:42:26.222 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-11T03:42:26.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.223 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 111, 2024-09-11T03:42:26.223 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-11T03:42:26.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.223 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 113, 2024-09-11T03:42:26.223 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 3 2024-09-11T03:42:26.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:42:26.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.223 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 118, 2024-09-11T03:42:26.223 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-11T03:42:26.223 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.223 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.224 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.224 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:42:26.224 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [ 2024-09-11T03:42:26.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.224 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-11T03:42:26.224 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-11T03:42:26.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:42:26.224 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 115, 2024-09-11T03:42:26.224 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-11T03:42:26.224 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.224 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:42:26.224 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.224 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:42:26.225 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-09-11T03:42:26.225 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-09-11T03:42:26.225 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-09-11T03:42:26.225 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-09-11T03:42:26.225 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-09-11T03:42:26.225 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-09-11T03:42:26.225 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-09-11T03:42:26.225 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-09-11T03:42:26.225 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-09-11T03:42:26.225 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:42:26.225 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:42:26.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:26.227 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:26.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:25 smithi175 ceph-mon[30894]: pgmap v646: 57 pgs: 57 active+clean; 204 MiB data, 761 MiB used, 893 GiB / 894 GiB avail; 18 MiB/s rd, 711 KiB/s wr, 15 op/s 2024-09-11T03:42:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:25 smithi175 ceph-mon[30894]: osdmap e328: 12 total, 12 up, 10 in 2024-09-11T03:42:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:25 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:42:26.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:25 smithi175 ceph-mon[30894]: 2.d scrub ok 2024-09-11T03:42:26.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:26.351 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:26.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:26.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:26.453 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:26.473 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:26.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:26.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:26.575 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:26.601 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:26.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:26.602 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:26.703 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:26.724 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:26.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:26.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:26.826 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:26.852 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:26.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:26.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:26.869 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:42:26.876 INFO:tasks.rados.rados.2.smithi175.stdout:1085: done (3 left) 2024-09-11T03:42:26.876 INFO:tasks.rados.rados.2.smithi175.stdout:1086: read oid 3 snap 115 2024-09-11T03:42:26.876 INFO:tasks.rados.rados.2.smithi175.stdout:1086: expect deleted 2024-09-11T03:42:26.876 INFO:tasks.rados.rados.2.smithi175.stdout:1087: rollback oid 38 current snap is 117 2024-09-11T03:42:26.877 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 38 to 99 2024-09-11T03:42:26.877 INFO:tasks.rados.rados.2.smithi175.stdout:1082: finishing rollback tid 0 to smithi17557870-40 2024-09-11T03:42:26.877 INFO:tasks.rados.rados.2.smithi175.stdout:1081: expect (ObjNum 335 snap 95 seq_num 335) 2024-09-11T03:42:26.955 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:26.979 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:26.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:26.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:27.070 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:42:27.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:27.094 INFO:tasks.rados.rados.2.smithi175.stdout:1082: finishing rollback tid 1 to smithi17557870-40 2024-09-11T03:42:27.094 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 350 (ObjNum 33 snap 0 seq_num 199111915) dirty dne 2024-09-11T03:42:27.094 INFO:tasks.rados.rados.2.smithi175.stdout:1080: finishing write tid 2 to smithi17557870-26 2024-09-11T03:42:27.094 INFO:tasks.rados.rados.2.smithi175.stdout:1080: finishing write tid 3 to smithi17557870-26 2024-09-11T03:42:27.095 INFO:tasks.rados.rados.2.smithi175.stdout:1080: finishing write tid 4 to smithi17557870-26 2024-09-11T03:42:27.095 INFO:tasks.rados.rados.2.smithi175.stdout:1080: oid 26 updating version 0 to 182 2024-09-11T03:42:27.095 INFO:tasks.rados.rados.2.smithi175.stdout:1080: oid 26 version 182 is already newer than 181 2024-09-11T03:42:27.095 INFO:tasks.rados.rados.2.smithi175.stdout:1080: oid 26 version 182 is already newer than 180 2024-09-11T03:42:27.095 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 182 (ObjNum 380 snap 117 seq_num 380) dirty exists 2024-09-11T03:42:27.095 INFO:tasks.rados.rados.2.smithi175.stdout:1080: left oid 26 (ObjNum 380 snap 117 seq_num 380) 2024-09-11T03:42:27.095 INFO:tasks.rados.rados.2.smithi175.stdout:1080: done (4 left) 2024-09-11T03:42:27.095 INFO:tasks.rados.rados.2.smithi175.stdout:1081: done (3 left) 2024-09-11T03:42:27.095 INFO:tasks.rados.rados.2.smithi175.stdout:1082: done (2 left) 2024-09-11T03:42:27.095 INFO:tasks.rados.rados.2.smithi175.stdout:1088: write (excl) oid 44 current snap is 117 2024-09-11T03:42:27.095 INFO:tasks.rados.rados.2.smithi175.stdout:1087: finishing rollback tid 1 to smithi17557870-38 2024-09-11T03:42:27.096 INFO:tasks.rados.rados.2.smithi175.stdout:1087: finishing rollback tid 2 to smithi17557870-38 2024-09-11T03:42:27.096 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 373 (ObjNum 318 snap 86 seq_num 318) dirty exists 2024-09-11T03:42:27.096 INFO:tasks.rados.rados.2.smithi175.stdout:1088: seq_num 382 ranges {420274=562768,1620592=439131} 2024-09-11T03:42:27.099 INFO:tasks.rados.rados.2.smithi175.stdout:1088: writing smithi17557870-44 from 420274 to 983042 tid 1 2024-09-11T03:42:27.105 INFO:tasks.rados.rados.2.smithi175.stdout:1088: writing smithi17557870-44 from 1620592 to 2059723 tid 2 2024-09-11T03:42:27.106 INFO:tasks.rados.rados.2.smithi175.stdout:1086: done (2 left) 2024-09-11T03:42:27.106 INFO:tasks.rados.rados.2.smithi175.stdout:1087: done (1 left) 2024-09-11T03:42:27.106 INFO:tasks.rados.rados.2.smithi175.stdout:1089: read oid 32 snap -1 2024-09-11T03:42:27.106 INFO:tasks.rados.rados.2.smithi175.stdout:1089: expect (ObjNum 277 snap 73 seq_num 277) 2024-09-11T03:42:27.106 INFO:tasks.rados.rados.2.smithi175.stdout:1090: rollback oid 43 current snap is 117 2024-09-11T03:42:27.106 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 43 to 115 2024-09-11T03:42:27.106 INFO:tasks.rados.rados.2.smithi175.stdout:1088: finishing write tid 1 to smithi17557870-44 2024-09-11T03:42:27.106 INFO:tasks.rados.rados.2.smithi175.stdout:1091: delete oid 5 current snap is 117 2024-09-11T03:42:27.107 INFO:tasks.rados.rados.2.smithi175.stdout:1089: expect (ObjNum 277 snap 73 seq_num 277) 2024-09-11T03:42:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:27.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:27.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:26 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:42:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:26 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:42:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:26 smithi084 ceph-mon[27254]: osdmap e329: 12 total, 12 up, 10 in 2024-09-11T03:42:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3096378394' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2253818410' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-11T03:42:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [9, 11]} : dispatch 2024-09-11T03:42:27.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.d", "id": [9, 5]} : dispatch 2024-09-11T03:42:27.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:26 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:42:27.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:26 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:42:27.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3096378394' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:27.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [9, 11]}]': finished 2024-09-11T03:42:27.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.d", "id": [9, 5]}]': finished 2024-09-11T03:42:27.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:26 smithi084 ceph-mon[27254]: osdmap e330: 12 total, 12 up, 10 in 2024-09-11T03:42:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:26 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:42:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:26 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:42:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:26 smithi121 ceph-mon[30729]: osdmap e329: 12 total, 12 up, 10 in 2024-09-11T03:42:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3096378394' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2253818410' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-11T03:42:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [9, 11]} : dispatch 2024-09-11T03:42:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.d", "id": [9, 5]} : dispatch 2024-09-11T03:42:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:26 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:42:27.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:26 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:42:27.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3096378394' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:27.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [9, 11]}]': finished 2024-09-11T03:42:27.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.d", "id": [9, 5]}]': finished 2024-09-11T03:42:27.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:26 smithi121 ceph-mon[30729]: osdmap e330: 12 total, 12 up, 10 in 2024-09-11T03:42:27.193 INFO:tasks.rados.rados.2.smithi175.stdout:1090: finishing rollback tid 0 to smithi17557870-43 2024-09-11T03:42:27.193 INFO:tasks.rados.rados.2.smithi175.stdout:1088: finishing write tid 2 to smithi17557870-44 2024-09-11T03:42:27.193 INFO:tasks.rados.rados.2.smithi175.stdout:1088: finishing write tid 3 to smithi17557870-44 2024-09-11T03:42:27.193 INFO:tasks.rados.rados.2.smithi175.stdout:1090: finishing rollback tid 1 to smithi17557870-43 2024-09-11T03:42:27.193 INFO:tasks.rados.rados.2.smithi175.stdout:1090: finishing rollback tid 2 to smithi17557870-43 2024-09-11T03:42:27.193 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 353 (ObjNum 310 snap 85 seq_num 310) dirty exists 2024-09-11T03:42:27.193 INFO:tasks.rados.rados.2.smithi175.stdout:1088: finishing write tid 4 to smithi17557870-44 2024-09-11T03:42:27.194 INFO:tasks.rados.rados.2.smithi175.stdout:1088: oid 44 updating version 0 to 151 2024-09-11T03:42:27.194 INFO:tasks.rados.rados.2.smithi175.stdout:1088: oid 44 updating version 151 to 153 2024-09-11T03:42:27.194 INFO:tasks.rados.rados.2.smithi175.stdout:1088: oid 44 version 153 is already newer than 152 2024-09-11T03:42:27.194 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 153 (ObjNum 382 snap 117 seq_num 382) dirty exists 2024-09-11T03:42:27.194 INFO:tasks.rados.rados.2.smithi175.stdout:1088: left oid 44 (ObjNum 382 snap 117 seq_num 382) 2024-09-11T03:42:27.194 INFO:tasks.rados.rados.2.smithi175.stdout:1088: done (3 left) 2024-09-11T03:42:27.194 INFO:tasks.rados.rados.2.smithi175.stdout:1089: done (2 left) 2024-09-11T03:42:27.194 INFO:tasks.rados.rados.2.smithi175.stdout:1090: done (1 left) 2024-09-11T03:42:27.194 INFO:tasks.rados.rados.2.smithi175.stdout:1091: done (0 left) 2024-09-11T03:42:27.194 INFO:tasks.rados.rados.2.smithi175.stdout:1092: read oid 23 snap -1 2024-09-11T03:42:27.194 INFO:tasks.rados.rados.2.smithi175.stdout:1092: expect (ObjNum 330 snap 92 seq_num 330) 2024-09-11T03:42:27.194 INFO:tasks.rados.rados.2.smithi175.stdout:1093: write (excl) oid 3 current snap is 117 2024-09-11T03:42:27.194 INFO:tasks.rados.rados.2.smithi175.stdout:1093: seq_num 384 ranges {453888=713266,1737556=442800} 2024-09-11T03:42:27.199 INFO:tasks.rados.rados.2.smithi175.stdout:1093: writing smithi17557870-3 from 453888 to 1167154 tid 1 2024-09-11T03:42:27.205 INFO:tasks.rados.rados.2.smithi175.stdout:1093: writing smithi17557870-3 from 1737556 to 2180356 tid 2 2024-09-11T03:42:27.205 INFO:tasks.rados.rados.2.smithi175.stdout:1094: copy_from oid 17 from oid 49 current snap is 117 2024-09-11T03:42:27.205 INFO:tasks.rados.rados.2.smithi175.stdout:1092: expect (ObjNum 330 snap 92 seq_num 330) 2024-09-11T03:42:27.210 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:26 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:42:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:26 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:42:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:26 smithi175 ceph-mon[30894]: osdmap e329: 12 total, 12 up, 10 in 2024-09-11T03:42:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3096378394' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2253818410' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-11T03:42:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [9, 11]} : dispatch 2024-09-11T03:42:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.d", "id": [9, 5]} : dispatch 2024-09-11T03:42:27.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:26 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:42:27.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:26 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:42:27.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3096378394' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:27.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [9, 11]}]': finished 2024-09-11T03:42:27.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.d", "id": [9, 5]}]': finished 2024-09-11T03:42:27.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:26 smithi175 ceph-mon[30894]: osdmap e330: 12 total, 12 up, 10 in 2024-09-11T03:42:27.235 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:27.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:27.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:27.321 INFO:tasks.rados.rados.2.smithi175.stdout:1093: finishing write tid 1 to smithi17557870-3 2024-09-11T03:42:27.321 INFO:tasks.rados.rados.2.smithi175.stdout:1093: finishing write tid 2 to smithi17557870-3 2024-09-11T03:42:27.321 INFO:tasks.rados.rados.2.smithi175.stdout:1093: finishing write tid 3 to smithi17557870-3 2024-09-11T03:42:27.321 INFO:tasks.rados.rados.2.smithi175.stdout:1093: finishing write tid 4 to smithi17557870-3 2024-09-11T03:42:27.321 INFO:tasks.rados.rados.2.smithi175.stdout:1093: oid 3 updating version 0 to 423 2024-09-11T03:42:27.321 INFO:tasks.rados.rados.2.smithi175.stdout:1093: oid 3 version 423 is already newer than 421 2024-09-11T03:42:27.322 INFO:tasks.rados.rados.2.smithi175.stdout:1093: oid 3 version 423 is already newer than 422 2024-09-11T03:42:27.322 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 423 (ObjNum 384 snap 117 seq_num 384) dirty exists 2024-09-11T03:42:27.322 INFO:tasks.rados.rados.2.smithi175.stdout:1093: left oid 3 (ObjNum 384 snap 117 seq_num 384) 2024-09-11T03:42:27.322 INFO:tasks.rados.rados.2.smithi175.stdout:1092: done (2 left) 2024-09-11T03:42:27.322 INFO:tasks.rados.rados.2.smithi175.stdout:1093: done (1 left) 2024-09-11T03:42:27.322 INFO:tasks.rados.rados.2.smithi175.stdout:1095: copy_from oid 40 from oid 29 current snap is 117 2024-09-11T03:42:27.322 INFO:tasks.rados.rados.2.smithi175.stdout:1096: rollback oid 16 current snap is 117 2024-09-11T03:42:27.322 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 16 to 91 2024-09-11T03:42:27.322 INFO:tasks.rados.rados.2.smithi175.stdout:1097: write oid 20 current snap is 117 2024-09-11T03:42:27.322 INFO:tasks.rados.rados.2.smithi175.stdout:1097: seq_num 385 ranges {470695=788515,1679685=586458} 2024-09-11T03:42:27.327 INFO:tasks.rados.rados.2.smithi175.stdout:1097: writing smithi17557870-20 from 470695 to 1259210 tid 1 2024-09-11T03:42:27.333 INFO:tasks.rados.rados.2.smithi175.stdout:1097: writing smithi17557870-20 from 1679685 to 2266143 tid 2 2024-09-11T03:42:27.333 INFO:tasks.rados.rados.2.smithi175.stdout:1098: delete oid 19 current snap is 117 2024-09-11T03:42:27.333 INFO:tasks.rados.rados.2.smithi175.stdout:1095: finishing copy_from to smithi17557870-40 2024-09-11T03:42:27.333 INFO:tasks.rados.rados.2.smithi175.stdout:1095: got expected ENOENT (src dne) 2024-09-11T03:42:27.333 INFO:tasks.rados.rados.2.smithi175.stdout:1095: finishing copy_from racing read to smithi17557870-40 2024-09-11T03:42:27.333 INFO:tasks.rados.rados.2.smithi175.stdout:1096: finishing rollback tid 0 to smithi17557870-16 2024-09-11T03:42:27.333 INFO:tasks.rados.rados.2.smithi175.stdout:1096: finishing rollback tid 1 to smithi17557870-16 2024-09-11T03:42:27.333 INFO:tasks.rados.rados.2.smithi175.stdout:1096: finishing rollback tid 2 to smithi17557870-16 2024-09-11T03:42:27.333 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 422 (ObjNum 153 snap 35 seq_num 153) dirty exists 2024-09-11T03:42:27.335 INFO:tasks.rados.rados.2.smithi175.stdout:1095: done (4 left) 2024-09-11T03:42:27.335 INFO:tasks.rados.rados.2.smithi175.stdout:1096: done (3 left) 2024-09-11T03:42:27.335 INFO:tasks.rados.rados.2.smithi175.stdout:1098: done (2 left) 2024-09-11T03:42:27.335 INFO:tasks.rados.rados.2.smithi175.stdout:1099: rollback oid 7 current snap is 117 2024-09-11T03:42:27.336 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 7 to 99 2024-09-11T03:42:27.336 INFO:tasks.rados.rados.2.smithi175.stdout:1100: write oid 18 current snap is 117 2024-09-11T03:42:27.336 INFO:tasks.rados.rados.2.smithi175.stdout:1100: seq_num 387 ranges {504309=539013,1796649=590127} 2024-09-11T03:42:27.337 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:27.341 INFO:tasks.rados.rados.2.smithi175.stdout:1100: writing smithi17557870-18 from 504309 to 1043322 tid 1 2024-09-11T03:42:27.348 INFO:tasks.rados.rados.2.smithi175.stdout:1100: writing smithi17557870-18 from 1796649 to 2386776 tid 2 2024-09-11T03:42:27.348 INFO:tasks.rados.rados.2.smithi175.stdout:1101: write (excl) oid 39 current snap is 117 2024-09-11T03:42:27.348 INFO:tasks.rados.rados.2.smithi175.stdout:1101: seq_num 388 ranges {521116=730615,1971484=549632} 2024-09-11T03:42:27.355 INFO:tasks.rados.rados.2.smithi175.stdout:1101: writing smithi17557870-39 from 521116 to 1251731 tid 1 2024-09-11T03:42:27.362 INFO:tasks.rados.rados.2.smithi175.stdout:1101: writing smithi17557870-39 from 1971484 to 2521116 tid 2 2024-09-11T03:42:27.362 INFO:tasks.rados.rados.2.smithi175.stdout:1102: write (excl) oid 1 current snap is 117 2024-09-11T03:42:27.362 INFO:tasks.rados.rados.2.smithi175.stdout:1099: finishing rollback tid 0 to smithi17557870-7 2024-09-11T03:42:27.362 INFO:tasks.rados.rados.2.smithi175.stdout:1099: finishing rollback tid 1 to smithi17557870-7 2024-09-11T03:42:27.362 INFO:tasks.rados.rados.2.smithi175.stdout:1099: finishing rollback tid 2 to smithi17557870-7 2024-09-11T03:42:27.362 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 201 (ObjNum 295 snap 79 seq_num 295) dirty exists 2024-09-11T03:42:27.362 INFO:tasks.rados.rados.2.smithi175.stdout:1100: finishing write tid 1 to smithi17557870-18 2024-09-11T03:42:27.363 INFO:tasks.rados.rados.2.smithi175.stdout:1100: finishing write tid 2 to smithi17557870-18 2024-09-11T03:42:27.363 INFO:tasks.rados.rados.2.smithi175.stdout:1102: seq_num 389 ranges {537923=405864,1513613=593796} 2024-09-11T03:42:27.363 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:27.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:27.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:27.368 INFO:tasks.rados.rados.2.smithi175.stdout:1102: writing smithi17557870-1 from 537923 to 943787 tid 1 2024-09-11T03:42:27.374 INFO:tasks.rados.rados.2.smithi175.stdout:1102: writing smithi17557870-1 from 1513613 to 2107409 tid 2 2024-09-11T03:42:27.374 INFO:tasks.rados.rados.2.smithi175.stdout:1099: done (5 left) 2024-09-11T03:42:27.374 INFO:tasks.rados.rados.2.smithi175.stdout:1103: snap_create 2024-09-11T03:42:27.374 INFO:tasks.rados.rados.2.smithi175.stdout:1100: finishing write tid 3 to smithi17557870-18 2024-09-11T03:42:27.374 INFO:tasks.rados.rados.2.smithi175.stdout:1100: finishing write tid 4 to smithi17557870-18 2024-09-11T03:42:27.374 INFO:tasks.rados.rados.2.smithi175.stdout:1100: oid 18 updating version 0 to 224 2024-09-11T03:42:27.375 INFO:tasks.rados.rados.2.smithi175.stdout:1100: oid 18 updating version 224 to 225 2024-09-11T03:42:27.375 INFO:tasks.rados.rados.2.smithi175.stdout:1100: oid 18 updating version 225 to 226 2024-09-11T03:42:27.375 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 226 (ObjNum 387 snap 117 seq_num 387) dirty exists 2024-09-11T03:42:27.375 INFO:tasks.rados.rados.2.smithi175.stdout:1100: left oid 18 (ObjNum 387 snap 117 seq_num 387) 2024-09-11T03:42:27.375 INFO:tasks.rados.rados.2.smithi175.stdout:1101: finishing write tid 1 to smithi17557870-39 2024-09-11T03:42:27.375 INFO:tasks.rados.rados.2.smithi175.stdout:1101: finishing write tid 2 to smithi17557870-39 2024-09-11T03:42:27.375 INFO:tasks.rados.rados.2.smithi175.stdout:1101: finishing write tid 3 to smithi17557870-39 2024-09-11T03:42:27.375 INFO:tasks.rados.rados.2.smithi175.stdout:1101: finishing write tid 4 to smithi17557870-39 2024-09-11T03:42:27.375 INFO:tasks.rados.rados.2.smithi175.stdout:1101: oid 39 updating version 0 to 310 2024-09-11T03:42:27.375 INFO:tasks.rados.rados.2.smithi175.stdout:1101: oid 39 version 310 is already newer than 309 2024-09-11T03:42:27.375 INFO:tasks.rados.rados.2.smithi175.stdout:1101: oid 39 updating version 310 to 311 2024-09-11T03:42:27.375 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 311 (ObjNum 388 snap 117 seq_num 388) dirty exists 2024-09-11T03:42:27.375 INFO:tasks.rados.rados.2.smithi175.stdout:1101: left oid 39 (ObjNum 388 snap 117 seq_num 388) 2024-09-11T03:42:27.376 INFO:tasks.rados.rados.2.smithi175.stdout:1102: finishing write tid 1 to smithi17557870-1 2024-09-11T03:42:27.376 INFO:tasks.rados.rados.2.smithi175.stdout:1102: finishing write tid 2 to smithi17557870-1 2024-09-11T03:42:27.376 INFO:tasks.rados.rados.2.smithi175.stdout:1102: finishing write tid 3 to smithi17557870-1 2024-09-11T03:42:27.376 INFO:tasks.rados.rados.2.smithi175.stdout:1102: finishing write tid 4 to smithi17557870-1 2024-09-11T03:42:27.376 INFO:tasks.rados.rados.2.smithi175.stdout:1102: oid 1 updating version 0 to 375 2024-09-11T03:42:27.377 INFO:tasks.rados.rados.2.smithi175.stdout:1102: oid 1 updating version 375 to 377 2024-09-11T03:42:27.377 INFO:tasks.rados.rados.2.smithi175.stdout:1102: oid 1 version 377 is already newer than 376 2024-09-11T03:42:27.377 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 377 (ObjNum 389 snap 117 seq_num 389) dirty exists 2024-09-11T03:42:27.377 INFO:tasks.rados.rados.2.smithi175.stdout:1102: left oid 1 (ObjNum 389 snap 117 seq_num 389) 2024-09-11T03:42:27.450 INFO:tasks.thrashosds.thrasher:j is {'epoch': 329, 'fsid': '10fe7946-6fee-11ef-bcea-c7b262605968', 'created': '2024-09-11T03:30:30.532652+0000', 'modified': '2024-09-11T03:42:25.901047+0000', 'last_up_change': '2024-09-11T03:37:26.787638+0000', 'last_in_change': '2024-09-11T03:40:04.952764+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T03:33:42.916933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '222', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '220', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.940000057220459, 'score_stable': 1.940000057220459, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.6100000143051147, 'raw_score_stable': 1.6100000143051147, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-11T03:38:28.163534+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '329', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '276', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 118, 'snap_epoch': 329, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.309999942779541, 'score_stable': 2.309999942779541, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.9199999570846558, 'raw_score_stable': 1.9199999570846558, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '9a431e48-e46b-44f9-bfa7-cb56c1484e1c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 280, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 1376667411}, {'type': 'v1', 'addr': '172.21.15.84:6803', 'nonce': 1376667411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 1376667411}, {'type': 'v1', 'addr': '172.21.15.84:6805', 'nonce': 1376667411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 1376667411}, {'type': 'v1', 'addr': '172.21.15.84:6809', 'nonce': 1376667411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 1376667411}, {'type': 'v1', 'addr': '172.21.15.84:6807', 'nonce': 1376667411}]}, 'public_addr': '172.21.15.84:6803/1376667411', 'cluster_addr': '172.21.15.84:6805/1376667411', 'heartbeat_back_addr': '172.21.15.84:6809/1376667411', 'heartbeat_front_addr': '172.21.15.84:6807/1376667411', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '62fd6e25-3508-40b4-a0f4-f7de137e3bb5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 280, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3138939249}, {'type': 'v1', 'addr': '172.21.15.84:6811', 'nonce': 3138939249}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 3138939249}, {'type': 'v1', 'addr': '172.21.15.84:6813', 'nonce': 3138939249}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6816', 'nonce': 3138939249}, {'type': 'v1', 'addr': '172.21.15.84:6817', 'nonce': 3138939249}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 3138939249}, {'type': 'v1', 'addr': '172.21.15.84:6815', 'nonce': 3138939249}]}, 'public_addr': '172.21.15.84:6811/3138939249', 'cluster_addr': '172.21.15.84:6813/3138939249', 'heartbeat_back_addr': '172.21.15.84:6817/3138939249', 'heartbeat_front_addr': '172.21.15.84:6815/3138939249', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ff351425-4715-49dc-be49-f27bf205d57e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 281, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6818', 'nonce': 3001811591}, {'type': 'v1', 'addr': '172.21.15.84:6819', 'nonce': 3001811591}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6820', 'nonce': 3001811591}, {'type': 'v1', 'addr': '172.21.15.84:6821', 'nonce': 3001811591}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6824', 'nonce': 3001811591}, {'type': 'v1', 'addr': '172.21.15.84:6825', 'nonce': 3001811591}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6822', 'nonce': 3001811591}, {'type': 'v1', 'addr': '172.21.15.84:6823', 'nonce': 3001811591}]}, 'public_addr': '172.21.15.84:6819/3001811591', 'cluster_addr': '172.21.15.84:6821/3001811591', 'heartbeat_back_addr': '172.21.15.84:6825/3001811591', 'heartbeat_front_addr': '172.21.15.84:6823/3001811591', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2518a93e-c612-4731-91ce-5685c25ffa9f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 312, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6826', 'nonce': 302512129}, {'type': 'v1', 'addr': '172.21.15.84:6827', 'nonce': 302512129}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6828', 'nonce': 302512129}, {'type': 'v1', 'addr': '172.21.15.84:6829', 'nonce': 302512129}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6832', 'nonce': 302512129}, {'type': 'v1', 'addr': '172.21.15.84:6833', 'nonce': 302512129}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6830', 'nonce': 302512129}, {'type': 'v1', 'addr': '172.21.15.84:6831', 'nonce': 302512129}]}, 'public_addr': '172.21.15.84:6827/302512129', 'cluster_addr': '172.21.15.84:6829/302512129', 'heartbeat_back_addr': '172.21.15.84:6833/302512129', 'heartbeat_front_addr': '172.21.15.84:6831/302512129', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd12f3c2d-95cc-4276-a7e1-bee982d58948', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 281, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6800', 'nonce': 3873214513}, {'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3873214513}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6802', 'nonce': 3873214513}, {'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3873214513}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6806', 'nonce': 3873214513}, {'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 3873214513}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6804', 'nonce': 3873214513}, {'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 3873214513}]}, 'public_addr': '172.21.15.121:6801/3873214513', 'cluster_addr': '172.21.15.121:6803/3873214513', 'heartbeat_back_addr': '172.21.15.121:6807/3873214513', 'heartbeat_front_addr': '172.21.15.121:6805/3873214513', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '32164dd5-7e8f-4925-8526-bbb1e33e3364', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 281, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6808', 'nonce': 918612194}, {'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 918612194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6810', 'nonce': 918612194}, {'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 918612194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6814', 'nonce': 918612194}, {'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 918612194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6812', 'nonce': 918612194}, {'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 918612194}]}, 'public_addr': '172.21.15.121:6809/918612194', 'cluster_addr': '172.21.15.121:6811/918612194', 'heartbeat_back_addr': '172.21.15.121:6815/918612194', 'heartbeat_front_addr': '172.21.15.121:6813/918612194', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '3f107a88-803f-4cd8-804a-241310c282ea', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6816', 'nonce': 3384813363}, {'type': 'v1', 'addr': '172.21.15.121:6817', 'nonce': 3384813363}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6818', 'nonce': 3384813363}, {'type': 'v1', 'addr': '172.21.15.121:6819', 'nonce': 3384813363}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6822', 'nonce': 3384813363}, {'type': 'v1', 'addr': '172.21.15.121:6823', 'nonce': 3384813363}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6820', 'nonce': 3384813363}, {'type': 'v1', 'addr': '172.21.15.121:6821', 'nonce': 3384813363}]}, 'public_addr': '172.21.15.121:6817/3384813363', 'cluster_addr': '172.21.15.121:6819/3384813363', 'heartbeat_back_addr': '172.21.15.121:6823/3384813363', 'heartbeat_front_addr': '172.21.15.121:6821/3384813363', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dfee7f95-1fb9-4493-9907-cb10d1837bdb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 309, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6824', 'nonce': 449222054}, {'type': 'v1', 'addr': '172.21.15.121:6825', 'nonce': 449222054}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6826', 'nonce': 449222054}, {'type': 'v1', 'addr': '172.21.15.121:6827', 'nonce': 449222054}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6830', 'nonce': 449222054}, {'type': 'v1', 'addr': '172.21.15.121:6831', 'nonce': 449222054}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6828', 'nonce': 449222054}, {'type': 'v1', 'addr': '172.21.15.121:6829', 'nonce': 449222054}]}, 'public_addr': '172.21.15.121:6825/449222054', 'cluster_addr': '172.21.15.121:6827/449222054', 'heartbeat_back_addr': '172.21.15.121:6831/449222054', 'heartbeat_front_addr': '172.21.15.121:6829/449222054', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 278, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6800', 'nonce': 2660128384}, {'type': 'v1', 'addr': '172.21.15.175:6801', 'nonce': 2660128384}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6802', 'nonce': 2660128384}, {'type': 'v1', 'addr': '172.21.15.175:6803', 'nonce': 2660128384}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6806', 'nonce': 2660128384}, {'type': 'v1', 'addr': '172.21.15.175:6807', 'nonce': 2660128384}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6804', 'nonce': 2660128384}, {'type': 'v1', 'addr': '172.21.15.175:6805', 'nonce': 2660128384}]}, 'public_addr': '172.21.15.175:6801/2660128384', 'cluster_addr': '172.21.15.175:6803/2660128384', 'heartbeat_back_addr': '172.21.15.175:6807/2660128384', 'heartbeat_front_addr': '172.21.15.175:6805/2660128384', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '50162e42-0443-41d1-8709-a0db61c087da', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 312, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6808', 'nonce': 1270991649}, {'type': 'v1', 'addr': '172.21.15.175:6809', 'nonce': 1270991649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6810', 'nonce': 1270991649}, {'type': 'v1', 'addr': '172.21.15.175:6811', 'nonce': 1270991649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6814', 'nonce': 1270991649}, {'type': 'v1', 'addr': '172.21.15.175:6815', 'nonce': 1270991649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6812', 'nonce': 1270991649}, {'type': 'v1', 'addr': '172.21.15.175:6813', 'nonce': 1270991649}]}, 'public_addr': '172.21.15.175:6809/1270991649', 'cluster_addr': '172.21.15.175:6811/1270991649', 'heartbeat_back_addr': '172.21.15.175:6815/1270991649', 'heartbeat_front_addr': '172.21.15.175:6813/1270991649', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '4b9426e1-bac8-47d3-ad58-54b89e8d1eb1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 182, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6816', 'nonce': 3877631021}, {'type': 'v1', 'addr': '172.21.15.175:6817', 'nonce': 3877631021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6818', 'nonce': 3877631021}, {'type': 'v1', 'addr': '172.21.15.175:6819', 'nonce': 3877631021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6822', 'nonce': 3877631021}, {'type': 'v1', 'addr': '172.21.15.175:6823', 'nonce': 3877631021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6820', 'nonce': 3877631021}, {'type': 'v1', 'addr': '172.21.15.175:6821', 'nonce': 3877631021}]}, 'public_addr': '172.21.15.175:6817/3877631021', 'cluster_addr': '172.21.15.175:6819/3877631021', 'heartbeat_back_addr': '172.21.15.175:6823/3877631021', 'heartbeat_front_addr': '172.21.15.175:6821/3877631021', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'cd1a28a1-413c-4274-9876-e1bdb7703be6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6824', 'nonce': 1150051236}, {'type': 'v1', 'addr': '172.21.15.175:6825', 'nonce': 1150051236}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6826', 'nonce': 1150051236}, {'type': 'v1', 'addr': '172.21.15.175:6827', 'nonce': 1150051236}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6830', 'nonce': 1150051236}, {'type': 'v1', 'addr': '172.21.15.175:6831', 'nonce': 1150051236}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6828', 'nonce': 1150051236}, {'type': 'v1', 'addr': '172.21.15.175:6829', 'nonce': 1150051236}]}, 'public_addr': '172.21.15.175:6825/1150051236', 'cluster_addr': '172.21.15.175:6827/1150051236', 'heartbeat_back_addr': '172.21.15.175:6831/1150051236', 'heartbeat_front_addr': '172.21.15.175:6829/1150051236', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:42:19.799941+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:41:47.349033+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:42:02.898695+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:41:28.941316+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:42:08.783166+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:42:25.432246+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-11T03:42:13.715916+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:41:42.942142+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:41:31.651575+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:41:36.385506+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-11T03:41:49.805801+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:42:15.880811+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/108136515': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:0/3411515187': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:6801/3365920551': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:6800/3365920551': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:0/3885577441': '2024-09-12T03:31:03.077186+0000', '172.21.15.84:0/22039254': '2024-09-12T03:31:03.077186+0000', '172.21.15.84:6801/3496645948': '2024-09-12T03:31:03.077186+0000', '172.21.15.84:0/4206337830': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:6800/3496645948': '2024-09-12T03:31:03.077186+0000', '172.21.15.84:0/1989775875': '2024-09-12T03:31:03.077186+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 104, 'length': 1}, {'begin': 111, 'length': 1}, {'begin': 113, 'length': 3}, {'begin': 118, 'length': 1}]}], 'new_removed_snaps': [{'pool': 2, 'snaps': [{'begin': 115, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T03:42:27.450 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:42:27.465 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:27.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:27.487 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:27.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:27.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:27.609 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:27.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:27.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:27.711 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:27.733 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:27.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:27.734 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:27.813 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:42:27.814 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:42:27.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:27.861 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:27.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:27.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:27.872 INFO:tasks.rados.rados.2.smithi175.stdout:1100: done (5 left) 2024-09-11T03:42:27.872 INFO:tasks.rados.rados.2.smithi175.stdout:1101: done (4 left) 2024-09-11T03:42:27.873 INFO:tasks.rados.rados.2.smithi175.stdout:1102: done (3 left) 2024-09-11T03:42:27.873 INFO:tasks.rados.rados.2.smithi175.stdout:1103: done (2 left) 2024-09-11T03:42:27.873 INFO:tasks.rados.rados.2.smithi175.stdout:1104: rollback oid 35 current snap is 118 2024-09-11T03:42:27.873 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 35 to 115 2024-09-11T03:42:27.873 INFO:tasks.rados.rados.2.smithi175.stdout:1105: rollback oid 43 current snap is 118 2024-09-11T03:42:27.873 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 43 to 99 2024-09-11T03:42:27.873 INFO:tasks.rados.rados.2.smithi175.stdout:1106: snap_remove snap 91 2024-09-11T03:42:27.963 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:27.985 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:27.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:27.987 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:28.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:28.110 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:28.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:28.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:28.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:27 smithi084 ceph-mon[27254]: pgmap v649: 57 pgs: 57 active+clean; 218 MiB data, 776 MiB used, 893 GiB / 894 GiB avail; 10 MiB/s rd, 3.1 MiB/s wr, 18 op/s 2024-09-11T03:42:28.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:27 smithi084 ceph-mon[27254]: osdmap e331: 12 total, 12 up, 10 in 2024-09-11T03:42:28.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:27 smithi121 ceph-mon[30729]: pgmap v649: 57 pgs: 57 active+clean; 218 MiB data, 776 MiB used, 893 GiB / 894 GiB avail; 10 MiB/s rd, 3.1 MiB/s wr, 18 op/s 2024-09-11T03:42:28.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:27 smithi121 ceph-mon[30729]: osdmap e331: 12 total, 12 up, 10 in 2024-09-11T03:42:28.211 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:27 smithi175 ceph-mon[30894]: pgmap v649: 57 pgs: 57 active+clean; 218 MiB data, 776 MiB used, 893 GiB / 894 GiB avail; 10 MiB/s rd, 3.1 MiB/s wr, 18 op/s 2024-09-11T03:42:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:27 smithi175 ceph-mon[30894]: osdmap e331: 12 total, 12 up, 10 in 2024-09-11T03:42:28.233 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:28.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:28.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:28.335 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:28.362 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:28.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:28.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:28.400 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T03:42:28.463 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:28.489 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:28.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:28.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:28.591 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:28.616 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:28.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:28.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:28.718 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:28.743 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:28.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:28.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:28.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:28.871 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:28.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:28.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:28.878 INFO:tasks.rados.rados.2.smithi175.stdout:1106: done (4 left) 2024-09-11T03:42:28.878 INFO:tasks.rados.rados.2.smithi175.stdout:1107: snap_create 2024-09-11T03:42:28.878 INFO:tasks.rados.rados.2.smithi175.stdout:1104: finishing rollback tid 0 to smithi17557870-35 2024-09-11T03:42:28.878 INFO:tasks.rados.rados.2.smithi175.stdout:1104: finishing rollback tid 1 to smithi17557870-35 2024-09-11T03:42:28.878 INFO:tasks.rados.rados.2.smithi175.stdout:1104: finishing rollback tid 2 to smithi17557870-35 2024-09-11T03:42:28.878 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 300 (ObjNum 356 snap 104 seq_num 356) dirty exists 2024-09-11T03:42:28.878 INFO:tasks.rados.rados.2.smithi175.stdout:1105: finishing rollback tid 0 to smithi17557870-43 2024-09-11T03:42:28.878 INFO:tasks.rados.rados.2.smithi175.stdout:1105: finishing rollback tid 1 to smithi17557870-43 2024-09-11T03:42:28.878 INFO:tasks.rados.rados.2.smithi175.stdout:1105: finishing rollback tid 2 to smithi17557870-43 2024-09-11T03:42:28.879 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 356 (ObjNum 340 snap 99 seq_num 340) dirty exists 2024-09-11T03:42:28.890 INFO:tasks.rados.rados.2.smithi175.stdout:1097: finishing write tid 1 to smithi17557870-20 2024-09-11T03:42:28.891 INFO:tasks.rados.rados.2.smithi175.stdout:1097: finishing write tid 2 to smithi17557870-20 2024-09-11T03:42:28.891 INFO:tasks.rados.rados.2.smithi175.stdout:1097: finishing write tid 3 to smithi17557870-20 2024-09-11T03:42:28.892 INFO:tasks.rados.rados.2.smithi175.stdout:1097: finishing write tid 4 to smithi17557870-20 2024-09-11T03:42:28.892 INFO:tasks.rados.rados.2.smithi175.stdout:1097: oid 20 updating version 0 to 430 2024-09-11T03:42:28.892 INFO:tasks.rados.rados.2.smithi175.stdout:1097: oid 20 version 430 is already newer than 428 2024-09-11T03:42:28.892 INFO:tasks.rados.rados.2.smithi175.stdout:1097: oid 20 version 430 is already newer than 429 2024-09-11T03:42:28.892 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 430 (ObjNum 385 snap 117 seq_num 385) dirty exists 2024-09-11T03:42:28.892 INFO:tasks.rados.rados.2.smithi175.stdout:1097: left oid 20 (ObjNum 385 snap 117 seq_num 385) 2024-09-11T03:42:28.900 INFO:tasks.rados.rados.2.smithi175.stdout:1094: finishing copy_from racing read to smithi17557870-17 2024-09-11T03:42:28.903 INFO:tasks.rados.rados.2.smithi175.stdout:1094: finishing copy_from to smithi17557870-17 2024-09-11T03:42:28.903 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 240 (ObjNum 338 snap 98 seq_num 338) dirty exists 2024-09-11T03:42:28.972 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:28.994 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:28.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:29.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:29.103 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:29.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:29.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:29.136 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:29.138 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:29.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:29.240 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:29.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:29.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:29.342 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:29.369 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:29.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:29.470 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:29.493 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:29.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:29.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:29.575 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:42:29.575 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:42:29.590 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:42:29.590 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:42:29.595 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:29.609 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:42:29.609 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:42:29.616 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:29.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:29.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:29.626 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:42:29.626 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:42:29.643 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:42:29.643 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:42:29.660 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:42:29.660 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:42:29.676 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:42:29.677 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:42:29.693 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:42:29.693 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:42:29.710 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:42:29.710 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:42:29.721 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:29.728 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:42:29.728 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:42:29.745 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:42:29.745 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:42:29.746 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:29.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:29.748 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:29.761 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:42:29.761 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:42:29.848 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:29.869 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:29.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:29.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:29.895 INFO:tasks.rados.rados.2.smithi175.stdout:1094: done (4 left) 2024-09-11T03:42:29.895 INFO:tasks.rados.rados.2.smithi175.stdout:1097: done (3 left) 2024-09-11T03:42:29.895 INFO:tasks.rados.rados.2.smithi175.stdout:1104: done (2 left) 2024-09-11T03:42:29.895 INFO:tasks.rados.rados.2.smithi175.stdout:1105: done (1 left) 2024-09-11T03:42:29.895 INFO:tasks.rados.rados.2.smithi175.stdout:1107: done (0 left) 2024-09-11T03:42:29.895 INFO:tasks.rados.rados.2.smithi175.stdout:1108: snap_create 2024-09-11T03:42:29.971 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:29.997 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:29.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:29.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:30.099 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:30.121 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:30.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:30.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:30.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:29 smithi084 ceph-mon[27254]: pgmap v652: 57 pgs: 1 remapped+peering, 56 active+clean; 225 MiB data, 783 MiB used, 893 GiB / 894 GiB avail; 15 MiB/s rd, 3.8 MiB/s wr, 25 op/s 2024-09-11T03:42:30.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:29 smithi084 ceph-mon[27254]: osdmap e332: 12 total, 12 up, 10 in 2024-09-11T03:42:30.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:29 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:42:30.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:29 smithi084 ceph-mon[27254]: 2.e scrub ok 2024-09-11T03:42:30.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:29 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3611144122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:29 smithi121 ceph-mon[30729]: pgmap v652: 57 pgs: 1 remapped+peering, 56 active+clean; 225 MiB data, 783 MiB used, 893 GiB / 894 GiB avail; 15 MiB/s rd, 3.8 MiB/s wr, 25 op/s 2024-09-11T03:42:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:29 smithi121 ceph-mon[30729]: osdmap e332: 12 total, 12 up, 10 in 2024-09-11T03:42:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:29 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:42:30.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:29 smithi121 ceph-mon[30729]: 2.e scrub ok 2024-09-11T03:42:30.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:29 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3611144122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:30.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:29 smithi175 ceph-mon[30894]: pgmap v652: 57 pgs: 1 remapped+peering, 56 active+clean; 225 MiB data, 783 MiB used, 893 GiB / 894 GiB avail; 15 MiB/s rd, 3.8 MiB/s wr, 25 op/s 2024-09-11T03:42:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:29 smithi175 ceph-mon[30894]: osdmap e332: 12 total, 12 up, 10 in 2024-09-11T03:42:30.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:29 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:42:30.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:29 smithi175 ceph-mon[30894]: 2.e scrub ok 2024-09-11T03:42:30.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:29 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3611144122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:30.248 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:30.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:30.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:30.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:30.372 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:30.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:30.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:30.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:30.500 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:30.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:30.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:30.602 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:30.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:30.629 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:30.629 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:30.730 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:30.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:30.757 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:30.757 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:30.765 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:42:30.765 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:42:30.765 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:42:30.765 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:42:30.858 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:30.883 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:30.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:30.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:30.899 INFO:tasks.rados.rados.2.smithi175.stdout:1108: done (0 left) 2024-09-11T03:42:30.899 INFO:tasks.rados.rados.2.smithi175.stdout:1109: snap_create 2024-09-11T03:42:30.984 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:31.009 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:31.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:31.010 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:31.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:42:31.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:31.137 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:31.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:31.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:31.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:30 smithi121 ceph-mon[30729]: osdmap e333: 12 total, 12 up, 10 in 2024-09-11T03:42:31.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:30 smithi175 ceph-mon[30894]: osdmap e333: 12 total, 12 up, 10 in 2024-09-11T03:42:31.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:31.264 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:31.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:31.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:31.268 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:30 smithi084 ceph-mon[27254]: osdmap e333: 12 total, 12 up, 10 in 2024-09-11T03:42:31.299 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T03:42:31.340 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:31.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:31.391 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:31.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:31.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:31.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:31.514 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:31.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:31.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:31.616 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:31.642 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:31.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:31.643 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:31.744 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:31.770 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:31.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:31.771 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:31.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:31.888 INFO:tasks.rados.rados.2.smithi175.stdout:1109: done (0 left) 2024-09-11T03:42:31.888 INFO:tasks.rados.rados.2.smithi175.stdout:1110: read oid 37 snap -1 2024-09-11T03:42:31.888 INFO:tasks.rados.rados.2.smithi175.stdout:1110: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:42:31.888 INFO:tasks.rados.rados.2.smithi175.stdout:1111: read oid 50 snap 118 2024-09-11T03:42:31.888 INFO:tasks.rados.rados.2.smithi175.stdout:1111: expect (ObjNum 336 snap 95 seq_num 336) 2024-09-11T03:42:31.888 INFO:tasks.rados.rados.2.smithi175.stdout:1112: write oid 15 current snap is 121 2024-09-11T03:42:31.888 INFO:tasks.rados.rados.2.smithi175.stdout:1112: seq_num 390 ranges {554730=481113,1572095=737454} 2024-09-11T03:42:31.893 INFO:tasks.rados.rados.2.smithi175.stdout:1112: writing smithi17557870-15 from 554730 to 1035843 tid 1 2024-09-11T03:42:31.898 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:31.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:31.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:31.900 INFO:tasks.rados.rados.2.smithi175.stdout:1112: writing smithi17557870-15 from 1572095 to 2309549 tid 2 2024-09-11T03:42:31.900 INFO:tasks.rados.rados.2.smithi175.stdout:1113: write oid 24 current snap is 121 2024-09-11T03:42:31.900 INFO:tasks.rados.rados.2.smithi175.stdout:1113: seq_num 391 ranges {571537=556362,1630577=597465} 2024-09-11T03:42:31.907 INFO:tasks.rados.rados.2.smithi175.stdout:1113: writing smithi17557870-24 from 571537 to 1127899 tid 1 2024-09-11T03:42:31.914 INFO:tasks.rados.rados.2.smithi175.stdout:1113: writing smithi17557870-24 from 1630577 to 2228042 tid 2 2024-09-11T03:42:31.914 INFO:tasks.rados.rados.2.smithi175.stdout:1114: read oid 40 snap -1 2024-09-11T03:42:31.914 INFO:tasks.rados.rados.2.smithi175.stdout:1114: expect deleted 2024-09-11T03:42:31.914 INFO:tasks.rados.rados.2.smithi175.stdout:1115: rollback oid 38 current snap is 121 2024-09-11T03:42:31.915 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 38 to 99 2024-09-11T03:42:31.915 INFO:tasks.rados.rados.2.smithi175.stdout:1110: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:42:32.001 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:32.022 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:32.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:32.023 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:32.038 INFO:tasks.rados.rados.2.smithi175.stdout:1112: finishing write tid 1 to smithi17557870-15 2024-09-11T03:42:32.039 INFO:tasks.rados.rados.2.smithi175.stdout:1112: finishing write tid 2 to smithi17557870-15 2024-09-11T03:42:32.039 INFO:tasks.rados.rados.2.smithi175.stdout:1112: finishing write tid 3 to smithi17557870-15 2024-09-11T03:42:32.039 INFO:tasks.rados.rados.2.smithi175.stdout:1110: done (5 left) 2024-09-11T03:42:32.039 INFO:tasks.rados.rados.2.smithi175.stdout:1116: snap_create 2024-09-11T03:42:32.039 INFO:tasks.rados.rados.2.smithi175.stdout:1112: finishing write tid 4 to smithi17557870-15 2024-09-11T03:42:32.039 INFO:tasks.rados.rados.2.smithi175.stdout:1112: oid 15 updating version 0 to 180 2024-09-11T03:42:32.039 INFO:tasks.rados.rados.2.smithi175.stdout:1112: oid 15 version 180 is already newer than 178 2024-09-11T03:42:32.039 INFO:tasks.rados.rados.2.smithi175.stdout:1112: oid 15 version 180 is already newer than 179 2024-09-11T03:42:32.039 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 180 (ObjNum 390 snap 121 seq_num 390) dirty exists 2024-09-11T03:42:32.039 INFO:tasks.rados.rados.2.smithi175.stdout:1112: left oid 15 (ObjNum 390 snap 121 seq_num 390) 2024-09-11T03:42:32.039 INFO:tasks.rados.rados.2.smithi175.stdout:1111: expect (ObjNum 336 snap 95 seq_num 336) 2024-09-11T03:42:32.123 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:32.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:31 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:42:32.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:31 smithi084 ceph-mon[27254]: 2.b scrub ok 2024-09-11T03:42:32.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:31 smithi084 ceph-mon[27254]: pgmap v655: 57 pgs: 1 remapped+peering, 56 active+clean; 225 MiB data, 785 MiB used, 893 GiB / 894 GiB avail; 7.9 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-11T03:42:32.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:31 smithi084 ceph-mon[27254]: osdmap e334: 12 total, 12 up, 10 in 2024-09-11T03:42:32.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:31 smithi084 ceph-mon[27254]: osdmap e335: 12 total, 12 up, 10 in 2024-09-11T03:42:32.149 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:32.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:32.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:32.155 INFO:tasks.rados.rados.2.smithi175.stdout:1113: finishing write tid 1 to smithi17557870-24 2024-09-11T03:42:32.155 INFO:tasks.rados.rados.2.smithi175.stdout:1115: finishing rollback tid 0 to smithi17557870-38 2024-09-11T03:42:32.155 INFO:tasks.rados.rados.2.smithi175.stdout:1113: finishing write tid 2 to smithi17557870-24 2024-09-11T03:42:32.156 INFO:tasks.rados.rados.2.smithi175.stdout:1113: finishing write tid 3 to smithi17557870-24 2024-09-11T03:42:32.156 INFO:tasks.rados.rados.2.smithi175.stdout:1115: finishing rollback tid 1 to smithi17557870-38 2024-09-11T03:42:32.156 INFO:tasks.rados.rados.2.smithi175.stdout:1115: finishing rollback tid 2 to smithi17557870-38 2024-09-11T03:42:32.156 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 391 (ObjNum 318 snap 86 seq_num 318) dirty exists 2024-09-11T03:42:32.156 INFO:tasks.rados.rados.2.smithi175.stdout:1113: finishing write tid 4 to smithi17557870-24 2024-09-11T03:42:32.156 INFO:tasks.rados.rados.2.smithi175.stdout:1113: oid 24 updating version 0 to 438 2024-09-11T03:42:32.156 INFO:tasks.rados.rados.2.smithi175.stdout:1113: oid 24 version 438 is already newer than 437 2024-09-11T03:42:32.157 INFO:tasks.rados.rados.2.smithi175.stdout:1113: oid 24 updating version 438 to 439 2024-09-11T03:42:32.157 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 439 (ObjNum 391 snap 121 seq_num 391) dirty exists 2024-09-11T03:42:32.157 INFO:tasks.rados.rados.2.smithi175.stdout:1113: left oid 24 (ObjNum 391 snap 121 seq_num 391) 2024-09-11T03:42:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:31 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:42:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:31 smithi121 ceph-mon[30729]: 2.b scrub ok 2024-09-11T03:42:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:31 smithi121 ceph-mon[30729]: pgmap v655: 57 pgs: 1 remapped+peering, 56 active+clean; 225 MiB data, 785 MiB used, 893 GiB / 894 GiB avail; 7.9 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-11T03:42:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:31 smithi121 ceph-mon[30729]: osdmap e334: 12 total, 12 up, 10 in 2024-09-11T03:42:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:31 smithi121 ceph-mon[30729]: osdmap e335: 12 total, 12 up, 10 in 2024-09-11T03:42:32.193 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:42:32.193 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:42:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:31 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:42:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:31 smithi175 ceph-mon[30894]: 2.b scrub ok 2024-09-11T03:42:32.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:31 smithi175 ceph-mon[30894]: pgmap v655: 57 pgs: 1 remapped+peering, 56 active+clean; 225 MiB data, 785 MiB used, 893 GiB / 894 GiB avail; 7.9 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-11T03:42:32.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:31 smithi175 ceph-mon[30894]: osdmap e334: 12 total, 12 up, 10 in 2024-09-11T03:42:32.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:31 smithi175 ceph-mon[30894]: osdmap e335: 12 total, 12 up, 10 in 2024-09-11T03:42:32.252 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:32.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:32.279 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:32.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:32.380 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:32.405 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:32.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:32.507 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:32.533 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:32.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:32.585 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:42:32.635 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:32.657 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:32.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:32.658 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:32.759 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:32.785 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:32.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:32.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:32.887 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:32.904 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:42:32.909 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:32.910 INFO:tasks.rados.rados.2.smithi175.stdout:1111: done (5 left) 2024-09-11T03:42:32.910 INFO:tasks.rados.rados.2.smithi175.stdout:1112: done (4 left) 2024-09-11T03:42:32.910 INFO:tasks.rados.rados.2.smithi175.stdout:1113: done (3 left) 2024-09-11T03:42:32.910 INFO:tasks.rados.rados.2.smithi175.stdout:1114: done (2 left) 2024-09-11T03:42:32.910 INFO:tasks.rados.rados.2.smithi175.stdout:1115: done (1 left) 2024-09-11T03:42:32.910 INFO:tasks.rados.rados.2.smithi175.stdout:1116: done (0 left) 2024-09-11T03:42:32.910 INFO:tasks.rados.rados.2.smithi175.stdout:1117: delete oid 6 current snap is 122 2024-09-11T03:42:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:32.911 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:32.914 INFO:tasks.rados.rados.2.smithi175.stdout:1117: done (0 left) 2024-09-11T03:42:32.914 INFO:tasks.rados.rados.2.smithi175.stdout:1118: read oid 46 snap -1 2024-09-11T03:42:32.914 INFO:tasks.rados.rados.2.smithi175.stdout:1118: expect (ObjNum 267 snap 68 seq_num 267) 2024-09-11T03:42:32.914 INFO:tasks.rados.rados.2.smithi175.stdout:1119: read oid 19 snap 120 2024-09-11T03:42:32.914 INFO:tasks.rados.rados.2.smithi175.stdout:1119: expect deleted 2024-09-11T03:42:32.914 INFO:tasks.rados.rados.2.smithi175.stdout:1120: write oid 1 current snap is 122 2024-09-11T03:42:32.914 INFO:tasks.rados.rados.2.smithi175.stdout:1120: seq_num 393 ranges {605151=706860,2031188=573963} 2024-09-11T03:42:32.920 INFO:tasks.rados.rados.2.smithi175.stdout:1120: writing smithi17557870-1 from 605151 to 1312011 tid 1 2024-09-11T03:42:32.928 INFO:tasks.rados.rados.2.smithi175.stdout:1120: writing smithi17557870-1 from 2031188 to 2605151 tid 2 2024-09-11T03:42:32.928 INFO:tasks.rados.rados.2.smithi175.stdout:1121: write (excl) oid 40 current snap is 122 2024-09-11T03:42:32.928 INFO:tasks.rados.rados.2.smithi175.stdout:1121: seq_num 394 ranges {621958=782109,2089670=532288} 2024-09-11T03:42:32.935 INFO:tasks.rados.rados.2.smithi175.stdout:1121: writing smithi17557870-40 from 621958 to 1404067 tid 1 2024-09-11T03:42:32.942 INFO:tasks.rados.rados.2.smithi175.stdout:1121: writing smithi17557870-40 from 2089670 to 2621958 tid 2 2024-09-11T03:42:32.942 INFO:tasks.rados.rados.2.smithi175.stdout:1122: snap_create 2024-09-11T03:42:32.942 INFO:tasks.rados.rados.2.smithi175.stdout:1118: expect (ObjNum 267 snap 68 seq_num 267) 2024-09-11T03:42:32.962 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":657,"stamp":"2024-09-11T03:42:31.885177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":227885176,"num_objects":109,"num_object_clones":60,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":4,"num_read":1003,"num_read_kb":865543,"num_write":785,"num_write_kb":323728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":29825344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8919,"ondisk_log_size":8919,"up":171,"acting":171,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":809556,"kb_used_data":459496,"kb_used_omap":15,"kb_used_meta":349104,"kb_avail":936641964,"statfs":{"total":959950356480,"available":959121371136,"internally_reserved":0,"allocated":470523904,"data_stored":632370670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":357482933},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6027958,"num_objects":-3,"num_object_clones":-1,"num_object_copies":-9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":29,"num_read_kb":28482,"num_write":38,"num_write_kb":10380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":18968151,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014471"},"pg_stats":[{"pgid":"1.1e","version":"20'92","reported_seq":232,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-09-11T03:42:26.878075+0000","last_change":"2024-09-11T03:40:40.236609+0000","last_active":"2024-09-11T03:42:26.878075+0000","last_peered":"2024-09-11T03:42:26.878075+0000","last_clean":"2024-09-11T03:42:26.878075+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:42:26.878075+0000","last_undegraded":"2024-09-11T03:42:26.878075+0000","last_fullsized":"2024-09-11T03:42:26.878075+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.236546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.236546+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:09.146128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'92","reported_seq":242,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.899055+0000","last_change":"2024-09-11T03:40:41.348889+0000","last_active":"2024-09-11T03:42:30.899055+0000","last_peered":"2024-09-11T03:42:30.899055+0000","last_clean":"2024-09-11T03:42:30.899055+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:42:30.899055+0000","last_undegraded":"2024-09-11T03:42:30.899055+0000","last_fullsized":"2024-09-11T03:42:30.899055+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:41.348828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:41.348828+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:11.089042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'92","reported_seq":242,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.899266+0000","last_change":"2024-09-11T03:40:40.188174+0000","last_active":"2024-09-11T03:42:30.899266+0000","last_peered":"2024-09-11T03:42:30.899266+0000","last_clean":"2024-09-11T03:42:30.899266+0000","last_became_active":"2024-09-11T03:40:39.848230+0000","last_became_peered":"2024-09-11T03:40:39.848230+0000","last_unstale":"2024-09-11T03:42:30.899266+0000","last_undegraded":"2024-09-11T03:42:30.899266+0000","last_fullsized":"2024-09-11T03:42:30.899266+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.188124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.188124+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:08.246084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"326'218","reported_seq":1480,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-09-11T03:42:27.876652+0000","last_change":"2024-09-11T03:42:26.875394+0000","last_active":"2024-09-11T03:42:27.876652+0000","last_peered":"2024-09-11T03:42:27.876652+0000","last_clean":"2024-09-11T03:42:27.876652+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:42:27.876652+0000","last_undegraded":"2024-09-11T03:42:27.876652+0000","last_fullsized":"2024-09-11T03:42:27.876652+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"280'196","last_scrub_stamp":"2024-09-11T03:41:40.336904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.336904+0000","objects_scrubbed":2,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:01.606236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010300280000000001,"stat_sum":{"num_bytes":11119314,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":41271,"num_write":5,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4865740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.1b","version":"20'92","reported_seq":242,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.899323+0000","last_change":"2024-09-11T03:40:42.166354+0000","last_active":"2024-09-11T03:42:30.899323+0000","last_peered":"2024-09-11T03:42:30.899323+0000","last_clean":"2024-09-11T03:42:30.899323+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:42:30.899323+0000","last_undegraded":"2024-09-11T03:42:30.899323+0000","last_fullsized":"2024-09-11T03:42:30.899323+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:42.166295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:42.166295+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:13.230054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.19","version":"330'357","reported_seq":1622,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-09-11T03:42:27.875089+0000","last_change":"2024-09-11T03:42:26.875933+0000","last_active":"2024-09-11T03:42:27.875089+0000","last_peered":"2024-09-11T03:42:27.875089+0000","last_clean":"2024-09-11T03:42:27.875089+0000","last_became_active":"2024-09-11T03:41:38.886787+0000","last_became_peered":"2024-09-11T03:41:38.886787+0000","last_unstale":"2024-09-11T03:42:27.875089+0000","last_undegraded":"2024-09-11T03:42:27.875089+0000","last_fullsized":"2024-09-11T03:42:27.875089+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'333","last_scrub_stamp":"2024-09-11T03:41:40.379141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.379141+0000","objects_scrubbed":1,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:06.534788+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0028508159999999999,"stat_sum":{"num_bytes":6354812,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":2892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.1a","version":"20'92","reported_seq":482,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-09-11T03:42:29.894039+0000","last_change":"2024-09-11T03:40:40.278752+0000","last_active":"2024-09-11T03:42:29.894039+0000","last_peered":"2024-09-11T03:42:29.894039+0000","last_clean":"2024-09-11T03:42:29.894039+0000","last_became_active":"2024-09-11T03:40:39.848781+0000","last_became_peered":"2024-09-11T03:40:39.848781+0000","last_unstale":"2024-09-11T03:42:29.894039+0000","last_undegraded":"2024-09-11T03:42:29.894039+0000","last_fullsized":"2024-09-11T03:42:29.894039+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.278703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.278703+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:09.913902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"20'92","reported_seq":240,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.742625+0000","last_change":"2024-09-11T03:40:40.718553+0000","last_active":"2024-09-11T03:42:30.742625+0000","last_peered":"2024-09-11T03:42:30.742625+0000","last_clean":"2024-09-11T03:42:30.742625+0000","last_became_active":"2024-09-11T03:40:39.851698+0000","last_became_peered":"2024-09-11T03:40:39.851698+0000","last_unstale":"2024-09-11T03:42:30.742625+0000","last_undegraded":"2024-09-11T03:42:30.742625+0000","last_fullsized":"2024-09-11T03:42:30.742625+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.718503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.718503+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:13.733914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.18","version":"20'92","reported_seq":364,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.900293+0000","last_change":"2024-09-11T03:42:02.103343+0000","last_active":"2024-09-11T03:42:30.900293+0000","last_peered":"2024-09-11T03:42:30.900293+0000","last_clean":"2024-09-11T03:42:30.900293+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:42:30.900293+0000","last_undegraded":"2024-09-11T03:42:30.900293+0000","last_fullsized":"2024-09-11T03:42:30.900293+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:02.103292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:02.103292+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:29.586019+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"330'174","reported_seq":1388,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-09-11T03:42:27.877170+0000","last_change":"2024-09-11T03:42:26.878850+0000","last_active":"2024-09-11T03:42:27.877170+0000","last_peered":"2024-09-11T03:42:27.877170+0000","last_clean":"2024-09-11T03:42:27.877170+0000","last_became_active":"2024-09-11T03:41:38.890690+0000","last_became_peered":"2024-09-11T03:41:38.890690+0000","last_unstale":"2024-09-11T03:42:27.877170+0000","last_undegraded":"2024-09-11T03:42:27.877170+0000","last_fullsized":"2024-09-11T03:42:27.877170+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'162","last_scrub_stamp":"2024-09-11T03:41:42.270480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:42.270480+0000","objects_scrubbed":1,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:10.903590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042773250000000002,"stat_sum":{"num_bytes":3327819,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[9,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":250,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.899506+0000","last_change":"2024-09-11T03:41:57.260720+0000","last_active":"2024-09-11T03:42:30.899506+0000","last_peered":"2024-09-11T03:42:30.899506+0000","last_clean":"2024-09-11T03:42:30.899506+0000","last_became_active":"2024-09-11T03:40:39.850243+0000","last_became_peered":"2024-09-11T03:40:39.850243+0000","last_unstale":"2024-09-11T03:42:30.899506+0000","last_undegraded":"2024-09-11T03:42:30.899506+0000","last_fullsized":"2024-09-11T03:42:30.899506+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:57.260664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:57.260664+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:11.818969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":1506,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-09-11T03:42:27.877278+0000","last_change":"2024-09-11T03:42:26.875880+0000","last_active":"2024-09-11T03:42:27.877278+0000","last_peered":"2024-09-11T03:42:27.877278+0000","last_clean":"2024-09-11T03:42:27.877278+0000","last_became_active":"2024-09-11T03:41:38.888228+0000","last_became_peered":"2024-09-11T03:41:38.888228+0000","last_unstale":"2024-09-11T03:42:27.877278+0000","last_undegraded":"2024-09-11T03:42:27.877278+0000","last_fullsized":"2024-09-11T03:42:27.877278+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:41:43.239726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:43.239726+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:49.181753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001013485,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":248,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.899074+0000","last_change":"2024-09-11T03:42:05.102506+0000","last_active":"2024-09-11T03:42:30.899074+0000","last_peered":"2024-09-11T03:42:30.899074+0000","last_clean":"2024-09-11T03:42:30.899074+0000","last_became_active":"2024-09-11T03:40:39.849950+0000","last_became_peered":"2024-09-11T03:40:39.849950+0000","last_unstale":"2024-09-11T03:42:30.899074+0000","last_undegraded":"2024-09-11T03:42:30.899074+0000","last_fullsized":"2024-09-11T03:42:30.899074+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:05.102457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:05.102457+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:18.344056+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"332'203","reported_seq":1403,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.900640+0000","last_change":"2024-09-11T03:42:28.884168+0000","last_active":"2024-09-11T03:42:30.900640+0000","last_peered":"2024-09-11T03:42:30.900640+0000","last_clean":"2024-09-11T03:42:30.900640+0000","last_became_active":"2024-09-11T03:41:38.887856+0000","last_became_peered":"2024-09-11T03:41:38.887856+0000","last_unstale":"2024-09-11T03:42:30.900640+0000","last_undegraded":"2024-09-11T03:42:30.900640+0000","last_fullsized":"2024-09-11T03:42:30.900640+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:41:42.059696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:42.059696+0000","objects_scrubbed":1,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:45.979843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073102419999999998,"stat_sum":{"num_bytes":8207700,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":15,"num_write_kb":2825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,9],"acting":[5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5d","length":"1"},{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":241,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.899896+0000","last_change":"2024-09-11T03:40:39.928143+0000","last_active":"2024-09-11T03:42:30.899896+0000","last_peered":"2024-09-11T03:42:30.899896+0000","last_clean":"2024-09-11T03:42:30.899896+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:42:30.899896+0000","last_undegraded":"2024-09-11T03:42:30.899896+0000","last_fullsized":"2024-09-11T03:42:30.899896+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.928088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.928088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:08.756769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":1587,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-09-11T03:42:27.871439+0000","last_change":"2024-09-11T03:42:26.975783+0000","last_active":"2024-09-11T03:42:27.871439+0000","last_peered":"2024-09-11T03:42:27.871439+0000","last_clean":"2024-09-11T03:42:27.871439+0000","last_became_active":"2024-09-11T03:41:38.885486+0000","last_became_peered":"2024-09-11T03:41:38.885486+0000","last_unstale":"2024-09-11T03:42:27.871439+0000","last_undegraded":"2024-09-11T03:42:27.871439+0000","last_fullsized":"2024-09-11T03:42:27.871439+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:41:41.456545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.456545+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:03.220571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011554340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":266,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-09-11T03:42:27.871395+0000","last_change":"2024-09-11T03:40:35.195938+0000","last_active":"2024-09-11T03:42:27.871395+0000","last_peered":"2024-09-11T03:42:27.871395+0000","last_clean":"2024-09-11T03:42:27.871395+0000","last_became_active":"2024-09-11T03:40:27.847297+0000","last_became_peered":"2024-09-11T03:40:27.847297+0000","last_unstale":"2024-09-11T03:42:27.871395+0000","last_undegraded":"2024-09-11T03:42:27.871395+0000","last_fullsized":"2024-09-11T03:42:27.871395+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.195893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.195893+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:50.659810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":310,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.899470+0000","last_change":"2024-09-11T03:42:28.884298+0000","last_active":"2024-09-11T03:42:30.899470+0000","last_peered":"2024-09-11T03:42:30.899470+0000","last_clean":"2024-09-11T03:42:30.899470+0000","last_became_active":"2024-09-11T03:41:35.874295+0000","last_became_peered":"2024-09-11T03:41:35.874295+0000","last_unstale":"2024-09-11T03:42:30.899470+0000","last_undegraded":"2024-09-11T03:42:30.899470+0000","last_fullsized":"2024-09-11T03:42:30.899470+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:41:40.182948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.182948+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:08.939672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037490470000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"5d","length":"1"},{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.13","version":"20'92","reported_seq":357,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.898864+0000","last_change":"2024-09-11T03:41:45.086774+0000","last_active":"2024-09-11T03:42:30.898864+0000","last_peered":"2024-09-11T03:42:30.898864+0000","last_clean":"2024-09-11T03:42:30.898864+0000","last_became_active":"2024-09-11T03:40:27.847557+0000","last_became_peered":"2024-09-11T03:40:27.847557+0000","last_unstale":"2024-09-11T03:42:30.898864+0000","last_undegraded":"2024-09-11T03:42:30.898864+0000","last_fullsized":"2024-09-11T03:42:30.898864+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:45.086692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:45.086692+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:48.005883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":1300,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-09-11T03:42:29.894099+0000","last_change":"2024-09-11T03:42:28.879897+0000","last_active":"2024-09-11T03:42:29.894099+0000","last_peered":"2024-09-11T03:42:29.894099+0000","last_clean":"2024-09-11T03:42:29.894099+0000","last_became_active":"2024-09-11T03:41:38.886813+0000","last_became_peered":"2024-09-11T03:41:38.886813+0000","last_unstale":"2024-09-11T03:42:29.894099+0000","last_undegraded":"2024-09-11T03:42:29.894099+0000","last_fullsized":"2024-09-11T03:42:29.894099+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:41:40.077940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.077940+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:43.953418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073715599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,9],"acting":[4,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5d","length":"1"},{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":269,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.900651+0000","last_change":"2024-09-11T03:40:37.284871+0000","last_active":"2024-09-11T03:42:30.900651+0000","last_peered":"2024-09-11T03:42:30.900651+0000","last_clean":"2024-09-11T03:42:30.900651+0000","last_became_active":"2024-09-11T03:40:27.845249+0000","last_became_peered":"2024-09-11T03:40:27.845249+0000","last_unstale":"2024-09-11T03:42:30.900651+0000","last_undegraded":"2024-09-11T03:42:30.900651+0000","last_fullsized":"2024-09-11T03:42:30.900651+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:37.284800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:37.284800+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:55.054741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"331'168","reported_seq":1375,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-09-11T03:42:27.879501+0000","last_change":"2024-09-11T03:42:26.877723+0000","last_active":"2024-09-11T03:42:27.879501+0000","last_peered":"2024-09-11T03:42:27.879501+0000","last_clean":"2024-09-11T03:42:27.879501+0000","last_became_active":"2024-09-11T03:42:09.873469+0000","last_became_peered":"2024-09-11T03:42:09.873469+0000","last_unstale":"2024-09-11T03:42:27.879501+0000","last_undegraded":"2024-09-11T03:42:27.879501+0000","last_fullsized":"2024-09-11T03:42:27.879501+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"283'124","last_scrub_stamp":"2024-09-11T03:41:41.419995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.419995+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:04.364451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046155609999999998,"stat_sum":{"num_bytes":13716680,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":14,"num_read_kb":10807,"num_write":21,"num_write_kb":9128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":5991453,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":272,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.742822+0000","last_change":"2024-09-11T03:40:35.703292+0000","last_active":"2024-09-11T03:42:30.742822+0000","last_peered":"2024-09-11T03:42:30.742822+0000","last_clean":"2024-09-11T03:42:30.742822+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:42:30.742822+0000","last_undegraded":"2024-09-11T03:42:30.742822+0000","last_fullsized":"2024-09-11T03:42:30.742822+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.703203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.703203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:49.677961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":1293,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-09-11T03:42:27.876874+0000","last_change":"2024-09-11T03:42:26.875350+0000","last_active":"2024-09-11T03:42:27.876874+0000","last_peered":"2024-09-11T03:42:27.876874+0000","last_clean":"2024-09-11T03:42:27.876874+0000","last_became_active":"2024-09-11T03:41:38.886834+0000","last_became_peered":"2024-09-11T03:41:38.886834+0000","last_unstale":"2024-09-11T03:42:27.876874+0000","last_undegraded":"2024-09-11T03:42:27.876874+0000","last_fullsized":"2024-09-11T03:42:27.876874+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:41:41.300112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.300112+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:05.629870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00094476100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":277,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.900412+0000","last_change":"2024-09-11T03:42:03.151379+0000","last_active":"2024-09-11T03:42:30.900412+0000","last_peered":"2024-09-11T03:42:30.900412+0000","last_clean":"2024-09-11T03:42:30.900412+0000","last_became_active":"2024-09-11T03:40:27.844057+0000","last_became_peered":"2024-09-11T03:40:27.844057+0000","last_unstale":"2024-09-11T03:42:30.900412+0000","last_undegraded":"2024-09-11T03:42:30.900412+0000","last_fullsized":"2024-09-11T03:42:30.900412+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:03.151323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:03.151323+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:05.151390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"332'313","reported_seq":1514,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.899503+0000","last_change":"2024-09-11T03:42:30.424690+0000","last_active":"2024-09-11T03:42:30.899503+0000","last_peered":"2024-09-11T03:42:30.899503+0000","last_clean":"2024-09-11T03:42:30.899503+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:42:30.899503+0000","last_undegraded":"2024-09-11T03:42:30.899503+0000","last_fullsized":"2024-09-11T03:42:30.899503+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'313","last_scrub_stamp":"2024-09-11T03:42:30.424582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:42:30.424582+0000","objects_scrubbed":5,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:45.604247+0000","scrub_duration":5,"objects_trimmed":1,"snaptrim_duration":0.003779372,"stat_sum":{"num_bytes":9818302,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":79,"num_read_kb":38400,"num_write":106,"num_write_kb":54118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"5d","length":"1"},{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":354,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.900485+0000","last_change":"2024-09-11T03:42:05.135033+0000","last_active":"2024-09-11T03:42:30.900485+0000","last_peered":"2024-09-11T03:42:30.900485+0000","last_clean":"2024-09-11T03:42:30.900485+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:42:30.900485+0000","last_undegraded":"2024-09-11T03:42:30.900485+0000","last_fullsized":"2024-09-11T03:42:30.900485+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:05.134977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:05.134977+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:05.203169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"330'109","reported_seq":1302,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-09-11T03:42:26.880749+0000","last_change":"2024-09-11T03:42:26.880958+0000","last_active":"2024-09-11T03:42:26.880749+0000","last_peered":"2024-09-11T03:42:26.880749+0000","last_clean":"2024-09-11T03:42:26.880749+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:42:26.880749+0000","last_undegraded":"2024-09-11T03:42:26.880749+0000","last_fullsized":"2024-09-11T03:42:26.880749+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'101","last_scrub_stamp":"2024-09-11T03:42:22.098703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:42:22.098703+0000","objects_scrubbed":2,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:33.068026+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0025588070000000002,"stat_sum":{"num_bytes":5738184,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":79439,"num_write":37,"num_write_kb":16778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":679,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.899532+0000","last_change":"2024-09-11T03:42:00.816905+0000","last_active":"2024-09-11T03:42:30.899532+0000","last_peered":"2024-09-11T03:42:30.899532+0000","last_clean":"2024-09-11T03:42:30.899532+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:42:30.899532+0000","last_undegraded":"2024-09-11T03:42:30.899532+0000","last_fullsized":"2024-09-11T03:42:30.899532+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:00.816850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:00.816850+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:09.071150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"332'431","reported_seq":1721,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.898808+0000","last_change":"2024-09-11T03:42:28.886264+0000","last_active":"2024-09-11T03:42:30.898808+0000","last_peered":"2024-09-11T03:42:30.898808+0000","last_clean":"2024-09-11T03:42:30.898808+0000","last_became_active":"2024-09-11T03:41:33.865638+0000","last_became_peered":"2024-09-11T03:41:33.865638+0000","last_unstale":"2024-09-11T03:42:30.898808+0000","last_undegraded":"2024-09-11T03:42:30.898808+0000","last_fullsized":"2024-09-11T03:42:30.898808+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'337","last_scrub_stamp":"2024-09-11T03:41:40.102984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.102984+0000","objects_scrubbed":5,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:45.704257+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0072959380000000001,"stat_sum":{"num_bytes":26402142,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":77,"num_read_kb":86435,"num_write":30,"num_write_kb":10475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5d","length":"1"},{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":586,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-09-11T03:42:27.876633+0000","last_change":"2024-09-11T03:41:45.227630+0000","last_active":"2024-09-11T03:42:27.876633+0000","last_peered":"2024-09-11T03:42:27.876633+0000","last_clean":"2024-09-11T03:42:27.876633+0000","last_became_active":"2024-09-11T03:40:37.849988+0000","last_became_peered":"2024-09-11T03:40:37.849988+0000","last_unstale":"2024-09-11T03:42:27.876633+0000","last_undegraded":"2024-09-11T03:42:27.876633+0000","last_fullsized":"2024-09-11T03:42:27.876633+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:45.227581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:45.227581+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:03.066353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"332'228","reported_seq":1453,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.900014+0000","last_change":"2024-09-11T03:42:28.884403+0000","last_active":"2024-09-11T03:42:30.900014+0000","last_peered":"2024-09-11T03:42:30.900014+0000","last_clean":"2024-09-11T03:42:30.900014+0000","last_became_active":"2024-09-11T03:41:33.863875+0000","last_became_peered":"2024-09-11T03:41:33.863875+0000","last_unstale":"2024-09-11T03:42:30.900014+0000","last_undegraded":"2024-09-11T03:42:30.900014+0000","last_fullsized":"2024-09-11T03:42:30.900014+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'192","last_scrub_stamp":"2024-09-11T03:41:40.877628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.877628+0000","objects_scrubbed":1,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:41.291928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036445789999999998,"stat_sum":{"num_bytes":3633300,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8,"num_write_kb":1104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"5d","length":"1"},{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":276,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-09-11T03:42:26.878050+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:42:26.878050+0000","last_peered":"2024-09-11T03:42:26.878050+0000","last_clean":"2024-09-11T03:42:26.878050+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:42:26.878050+0000","last_undegraded":"2024-09-11T03:42:26.878050+0000","last_fullsized":"2024-09-11T03:42:26.878050+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:54.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"326'378","reported_seq":1659,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-09-11T03:42:27.871617+0000","last_change":"2024-09-11T03:42:26.976222+0000","last_active":"2024-09-11T03:42:27.871617+0000","last_peered":"2024-09-11T03:42:27.871617+0000","last_clean":"2024-09-11T03:42:27.871617+0000","last_became_active":"2024-09-11T03:41:33.867674+0000","last_became_peered":"2024-09-11T03:41:33.867674+0000","last_unstale":"2024-09-11T03:42:27.871617+0000","last_undegraded":"2024-09-11T03:42:27.871617+0000","last_fullsized":"2024-09-11T03:42:27.871617+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'331","last_scrub_stamp":"2024-09-11T03:41:42.500501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:42.500501+0000","objects_scrubbed":9,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:59.845274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00064354600000000005,"stat_sum":{"num_bytes":23855551,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":28,"num_read_kb":31252,"num_write":10,"num_write_kb":5112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":443,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-09-11T03:42:26.878558+0000","last_change":"2024-09-11T03:42:00.181197+0000","last_active":"2024-09-11T03:42:26.878558+0000","last_peered":"2024-09-11T03:42:26.878558+0000","last_clean":"2024-09-11T03:42:26.878558+0000","last_became_active":"2024-09-11T03:40:25.858281+0000","last_became_peered":"2024-09-11T03:40:25.858281+0000","last_unstale":"2024-09-11T03:42:26.878558+0000","last_undegraded":"2024-09-11T03:42:26.878558+0000","last_fullsized":"2024-09-11T03:42:26.878558+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:00.181139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:00.181139+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:00.294383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":1373,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.899206+0000","last_change":"2024-09-11T03:42:28.883492+0000","last_active":"2024-09-11T03:42:30.899206+0000","last_peered":"2024-09-11T03:42:30.899206+0000","last_clean":"2024-09-11T03:42:30.899206+0000","last_became_active":"2024-09-11T03:41:33.865499+0000","last_became_peered":"2024-09-11T03:41:33.865499+0000","last_unstale":"2024-09-11T03:42:30.899206+0000","last_undegraded":"2024-09-11T03:42:30.899206+0000","last_fullsized":"2024-09-11T03:42:30.899206+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:41:42.104989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:42.104989+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:55.363605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042172219999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5d","length":"1"},{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":345,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-09-11T03:42:27.871452+0000","last_change":"2024-09-11T03:40:38.224278+0000","last_active":"2024-09-11T03:42:27.871452+0000","last_peered":"2024-09-11T03:42:27.871452+0000","last_clean":"2024-09-11T03:42:27.871452+0000","last_became_active":"2024-09-11T03:40:37.851080+0000","last_became_peered":"2024-09-11T03:40:37.851080+0000","last_unstale":"2024-09-11T03:42:27.871452+0000","last_undegraded":"2024-09-11T03:42:27.871452+0000","last_fullsized":"2024-09-11T03:42:27.871452+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.224231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.224231+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:09.894123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"332'387","reported_seq":1660,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.899524+0000","last_change":"2024-09-11T03:42:28.889450+0000","last_active":"2024-09-11T03:42:30.899524+0000","last_peered":"2024-09-11T03:42:30.899524+0000","last_clean":"2024-09-11T03:42:30.899524+0000","last_became_active":"2024-09-11T03:41:33.866316+0000","last_became_peered":"2024-09-11T03:41:33.866316+0000","last_unstale":"2024-09-11T03:42:30.899524+0000","last_undegraded":"2024-09-11T03:42:30.899524+0000","last_fullsized":"2024-09-11T03:42:30.899524+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'292","last_scrub_stamp":"2024-09-11T03:41:41.222192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.222192+0000","objects_scrubbed":9,"log_size":387,"log_dups_size":0,"ondisk_log_size":387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:03.704163+0000","scrub_duration":2,"objects_trimmed":4,"snaptrim_duration":0.0084823729999999996,"stat_sum":{"num_bytes":20006250,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":65809,"num_write":15,"num_write_kb":8232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"5d","length":"1"},{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":589,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-09-11T03:42:27.877008+0000","last_change":"2024-09-11T03:42:05.222525+0000","last_active":"2024-09-11T03:42:27.877008+0000","last_peered":"2024-09-11T03:42:27.877008+0000","last_clean":"2024-09-11T03:42:27.877008+0000","last_became_active":"2024-09-11T03:40:37.848794+0000","last_became_peered":"2024-09-11T03:40:37.848794+0000","last_unstale":"2024-09-11T03:42:27.877008+0000","last_undegraded":"2024-09-11T03:42:27.877008+0000","last_fullsized":"2024-09-11T03:42:27.877008+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:05.222462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:05.222462+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:33.609855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"332'184","reported_seq":1440,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.899215+0000","last_change":"2024-09-11T03:42:28.883486+0000","last_active":"2024-09-11T03:42:30.899215+0000","last_peered":"2024-09-11T03:42:30.899215+0000","last_clean":"2024-09-11T03:42:30.899215+0000","last_became_active":"2024-09-11T03:41:33.869119+0000","last_became_peered":"2024-09-11T03:41:33.869119+0000","last_unstale":"2024-09-11T03:42:30.899215+0000","last_undegraded":"2024-09-11T03:42:30.899215+0000","last_fullsized":"2024-09-11T03:42:30.899215+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'162","last_scrub_stamp":"2024-09-11T03:41:41.141743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.141743+0000","objects_scrubbed":2,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:48.867425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037727110000000002,"stat_sum":{"num_bytes":8638930,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":33838,"num_write":10,"num_write_kb":1948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5d","length":"1"},{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":80,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-09-11T03:42:27.875306+0000","last_change":"2024-09-11T03:41:58.387920+0000","last_active":"2024-09-11T03:42:27.875306+0000","last_peered":"2024-09-11T03:42:27.875306+0000","last_clean":"2024-09-11T03:42:27.875306+0000","last_became_active":"2024-09-11T03:41:57.865927+0000","last_became_peered":"2024-09-11T03:41:57.865927+0000","last_unstale":"2024-09-11T03:42:27.875306+0000","last_undegraded":"2024-09-11T03:42:27.875306+0000","last_fullsized":"2024-09-11T03:42:27.875306+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:58.387873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:58.387873+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:03.728844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"334'139","reported_seq":1403,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.905741+0000","last_change":"2024-09-11T03:42:30.906270+0000","last_active":"2024-09-11T03:42:30.905741+0000","last_peered":"2024-09-11T03:42:30.905741+0000","last_clean":"2024-09-11T03:42:30.905741+0000","last_became_active":"2024-09-11T03:42:30.902838+0000","last_became_peered":"2024-09-11T03:42:30.902838+0000","last_unstale":"2024-09-11T03:42:30.905741+0000","last_undegraded":"2024-09-11T03:42:30.905741+0000","last_fullsized":"2024-09-11T03:42:30.905741+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'123","last_scrub_stamp":"2024-09-11T03:41:40.064448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.064448+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:02.317511+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0028616510000000002,"stat_sum":{"num_bytes":2571471,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,11],"acting":[5,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5d","length":"1"},{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":575,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-09-11T03:42:27.876778+0000","last_change":"2024-09-11T03:42:01.225777+0000","last_active":"2024-09-11T03:42:27.876778+0000","last_peered":"2024-09-11T03:42:27.876778+0000","last_clean":"2024-09-11T03:42:27.876778+0000","last_became_active":"2024-09-11T03:40:25.854269+0000","last_became_peered":"2024-09-11T03:40:25.854269+0000","last_unstale":"2024-09-11T03:42:27.876778+0000","last_undegraded":"2024-09-11T03:42:27.876778+0000","last_fullsized":"2024-09-11T03:42:27.876778+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:01.225716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:01.225716+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:14.417097+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"330'130","reported_seq":1347,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-09-11T03:42:27.871462+0000","last_change":"2024-09-11T03:42:26.975489+0000","last_active":"2024-09-11T03:42:27.871462+0000","last_peered":"2024-09-11T03:42:27.871462+0000","last_clean":"2024-09-11T03:42:27.871462+0000","last_became_active":"2024-09-11T03:41:33.866217+0000","last_became_peered":"2024-09-11T03:41:33.866217+0000","last_unstale":"2024-09-11T03:42:27.871462+0000","last_undegraded":"2024-09-11T03:42:27.871462+0000","last_fullsized":"2024-09-11T03:42:27.871462+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:41:40.497820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.497820+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:47.804551+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00068644300000000001,"stat_sum":{"num_bytes":5249074,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":1777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":60,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.899658+0000","last_change":"2024-09-11T03:42:06.870952+0000","last_active":"2024-09-11T03:42:30.899658+0000","last_peered":"2024-09-11T03:42:30.899658+0000","last_clean":"2024-09-11T03:42:30.899658+0000","last_became_active":"2024-09-11T03:42:06.870500+0000","last_became_peered":"2024-09-11T03:42:06.870500+0000","last_unstale":"2024-09-11T03:42:30.899658+0000","last_undegraded":"2024-09-11T03:42:30.899658+0000","last_fullsized":"2024-09-11T03:42:30.899658+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:02.760253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:02.760253+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:17.778058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"332'88","reported_seq":1325,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.899875+0000","last_change":"2024-09-11T03:42:28.885883+0000","last_active":"2024-09-11T03:42:30.899875+0000","last_peered":"2024-09-11T03:42:30.899875+0000","last_clean":"2024-09-11T03:42:30.899875+0000","last_became_active":"2024-09-11T03:41:33.864202+0000","last_became_peered":"2024-09-11T03:41:33.864202+0000","last_unstale":"2024-09-11T03:42:30.899875+0000","last_undegraded":"2024-09-11T03:42:30.899875+0000","last_fullsized":"2024-09-11T03:42:30.899875+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:41:41.843233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.843233+0000","objects_scrubbed":2,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:03.904276+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0046981139999999998,"stat_sum":{"num_bytes":3781608,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"5d","length":"1"},{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":355,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.898872+0000","last_change":"2024-09-11T03:41:57.122830+0000","last_active":"2024-09-11T03:42:30.898872+0000","last_peered":"2024-09-11T03:42:30.898872+0000","last_clean":"2024-09-11T03:42:30.898872+0000","last_became_active":"2024-09-11T03:40:25.856407+0000","last_became_peered":"2024-09-11T03:40:25.856407+0000","last_unstale":"2024-09-11T03:42:30.898872+0000","last_undegraded":"2024-09-11T03:42:30.898872+0000","last_fullsized":"2024-09-11T03:42:30.898872+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:57.122773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:57.122773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:57.896149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"332'157","reported_seq":1382,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.900383+0000","last_change":"2024-09-11T03:42:28.882535+0000","last_active":"2024-09-11T03:42:30.900383+0000","last_peered":"2024-09-11T03:42:30.900383+0000","last_clean":"2024-09-11T03:42:30.900383+0000","last_became_active":"2024-09-11T03:41:33.862271+0000","last_became_peered":"2024-09-11T03:41:33.862271+0000","last_unstale":"2024-09-11T03:42:30.900383+0000","last_undegraded":"2024-09-11T03:42:30.900383+0000","last_fullsized":"2024-09-11T03:42:30.900383+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'117","last_scrub_stamp":"2024-09-11T03:41:41.089786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.089786+0000","objects_scrubbed":3,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:44.575974+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0057835930000000001,"stat_sum":{"num_bytes":7457863,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":18,"num_read_kb":12238,"num_write":10,"num_write_kb":1999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5d","length":"1"},{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":676,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.899955+0000","last_change":"2024-09-11T03:41:45.957854+0000","last_active":"2024-09-11T03:42:30.899955+0000","last_peered":"2024-09-11T03:42:30.899955+0000","last_clean":"2024-09-11T03:42:30.899955+0000","last_became_active":"2024-09-11T03:40:25.854624+0000","last_became_peered":"2024-09-11T03:40:25.854624+0000","last_unstale":"2024-09-11T03:42:30.899955+0000","last_undegraded":"2024-09-11T03:42:30.899955+0000","last_fullsized":"2024-09-11T03:42:30.899955+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:45.957781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:45.957781+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:01.135709+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"20'92","reported_seq":271,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.900596+0000","last_change":"2024-09-11T03:40:35.282811+0000","last_active":"2024-09-11T03:42:30.900596+0000","last_peered":"2024-09-11T03:42:30.900596+0000","last_clean":"2024-09-11T03:42:30.900596+0000","last_became_active":"2024-09-11T03:40:27.844262+0000","last_became_peered":"2024-09-11T03:40:27.844262+0000","last_unstale":"2024-09-11T03:42:30.900596+0000","last_undegraded":"2024-09-11T03:42:30.900596+0000","last_fullsized":"2024-09-11T03:42:30.900596+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.282762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.282762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:54.069194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"330'296","reported_seq":1496,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-09-11T03:42:26.882137+0000","last_change":"2024-09-11T03:42:26.882339+0000","last_active":"2024-09-11T03:42:26.882137+0000","last_peered":"2024-09-11T03:42:26.882137+0000","last_clean":"2024-09-11T03:42:26.882137+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:42:26.882137+0000","last_undegraded":"2024-09-11T03:42:26.882137+0000","last_fullsized":"2024-09-11T03:42:26.882137+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'172","last_scrub_stamp":"2024-09-11T03:41:04.106166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:04.106166+0000","objects_scrubbed":5,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0034255650000000002,"stat_sum":{"num_bytes":18731110,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":156,"num_read_kb":120396,"num_write":138,"num_write_kb":58989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":526,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-09-11T03:42:29.894250+0000","last_change":"2024-09-11T03:40:38.199696+0000","last_active":"2024-09-11T03:42:29.894250+0000","last_peered":"2024-09-11T03:42:29.894250+0000","last_clean":"2024-09-11T03:42:29.894250+0000","last_became_active":"2024-09-11T03:40:37.850623+0000","last_became_peered":"2024-09-11T03:40:37.850623+0000","last_unstale":"2024-09-11T03:42:29.894250+0000","last_undegraded":"2024-09-11T03:42:29.894250+0000","last_fullsized":"2024-09-11T03:42:29.894250+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.199636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.199636+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:48.923701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"334'438","reported_seq":1633,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.908358+0000","last_change":"2024-09-11T03:42:30.908358+0000","last_active":"2024-09-11T03:42:30.908358+0000","last_peered":"2024-09-11T03:42:30.908358+0000","last_clean":"2024-09-11T03:42:30.908358+0000","last_became_active":"2024-09-11T03:42:30.902959+0000","last_became_peered":"2024-09-11T03:42:30.902959+0000","last_unstale":"2024-09-11T03:42:30.908358+0000","last_undegraded":"2024-09-11T03:42:30.908358+0000","last_fullsized":"2024-09-11T03:42:30.908358+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'426","last_scrub_stamp":"2024-09-11T03:42:25.281790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:42:25.281790+0000","objects_scrubbed":10,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:25.907515+0000","scrub_duration":3,"objects_trimmed":3,"snaptrim_duration":0.0043624359999999999,"stat_sum":{"num_bytes":21047789,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":174,"num_read_kb":136324,"num_write":148,"num_write_kb":59200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17460823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5d","length":"1"},{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":285,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.899996+0000","last_change":"2024-09-11T03:40:38.901246+0000","last_active":"2024-09-11T03:42:30.899996+0000","last_peered":"2024-09-11T03:42:30.899996+0000","last_clean":"2024-09-11T03:42:30.899996+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:42:30.899996+0000","last_undegraded":"2024-09-11T03:42:30.899996+0000","last_fullsized":"2024-09-11T03:42:30.899996+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.901175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.901175+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:50.952770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"330'427","reported_seq":1604,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-09-11T03:42:27.871504+0000","last_change":"2024-09-11T03:42:26.977286+0000","last_active":"2024-09-11T03:42:27.871504+0000","last_peered":"2024-09-11T03:42:27.871504+0000","last_clean":"2024-09-11T03:42:27.871504+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:42:27.871504+0000","last_undegraded":"2024-09-11T03:42:27.871504+0000","last_fullsized":"2024-09-11T03:42:27.871504+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'259","last_scrub_stamp":"2024-09-11T03:41:09.347904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:09.347904+0000","objects_scrubbed":11,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:34.576923+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00227181,"stat_sum":{"num_bytes":17059336,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":160,"num_read_kb":127275,"num_write":135,"num_write_kb":51847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"2.f","version":"332'201","reported_seq":1425,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-11T03:42:30.899593+0000","last_change":"2024-09-11T03:42:28.883903+0000","last_active":"2024-09-11T03:42:30.899593+0000","last_peered":"2024-09-11T03:42:30.899593+0000","last_clean":"2024-09-11T03:42:30.899593+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:42:30.899593+0000","last_undegraded":"2024-09-11T03:42:30.899593+0000","last_fullsized":"2024-09-11T03:42:30.899593+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'136","last_scrub_stamp":"2024-09-11T03:41:20.167983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:20.167983+0000","objects_scrubbed":2,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:41.429673+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0034066489999999999,"stat_sum":{"num_bytes":10577573,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":104,"num_read_kb":82053,"num_write":75,"num_write_kb":31173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"5d","length":"1"},{"start":"68","length":"1"},{"start":"6f","length":"1"},{"start":"71","length":"1"},{"start":"73","length":"2"},{"start":"76","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":288,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-09-11T03:42:27.875120+0000","last_change":"2024-09-11T03:40:38.353992+0000","last_active":"2024-09-11T03:42:27.875120+0000","last_peered":"2024-09-11T03:42:27.875120+0000","last_clean":"2024-09-11T03:42:27.875120+0000","last_became_active":"2024-09-11T03:40:37.847449+0000","last_became_peered":"2024-09-11T03:40:37.847449+0000","last_unstale":"2024-09-11T03:42:27.875120+0000","last_undegraded":"2024-09-11T03:42:27.875120+0000","last_fullsized":"2024-09-11T03:42:27.875120+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.353895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.353895+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:48.326329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":227294808,"num_objects":107,"num_object_clones":60,"num_object_copies":321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107,"num_whiteouts":4,"num_read":1003,"num_read_kb":865543,"num_write":785,"num_write_kb":323728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":29825344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":435978240,"data_stored":612857497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6067,"ondisk_log_size":6067,"up":78,"acting":78,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285631,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63316,"kb_used_data":32128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93681836,"statfs":{"total":95995035648,"available":95930200064,"internally_reserved":0,"allocated":32899072,"data_stored":44304752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":326417514563,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743497,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90056,"kb_used_data":54704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93655096,"statfs":{"total":95995035648,"available":95902818304,"internally_reserved":0,"allocated":56016896,"data_stored":92562620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037841,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68308,"kb_used_data":32960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93676844,"statfs":{"total":95995035648,"available":95925088256,"internally_reserved":0,"allocated":33751040,"data_stored":43565090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299476,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86876,"kb_used_data":51528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93658276,"statfs":{"total":95995035648,"available":95906074624,"internally_reserved":0,"allocated":52764672,"data_stored":66200483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":193273528408,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757343,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63536,"kb_used_data":32348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93681616,"statfs":{"total":95995035648,"available":95929974784,"internally_reserved":0,"allocated":33124352,"data_stored":46442960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986276,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94064,"kb_used_data":58716,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93651088,"statfs":{"total":95995035648,"available":95898714112,"internally_reserved":0,"allocated":60125184,"data_stored":76762574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":113924,"kb_used_data":73700,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93631228,"statfs":{"total":95995035648,"available":95878377472,"internally_reserved":0,"allocated":75468800,"data_stored":103592396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411437,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76288,"kb_used_data":40932,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93668864,"statfs":{"total":95995035648,"available":95916916736,"internally_reserved":0,"allocated":41914368,"data_stored":56060634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574963,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71912,"kb_used_data":36556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93673240,"statfs":{"total":95995035648,"available":95921397760,"internally_reserved":0,"allocated":37433344,"data_stored":41544515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738489,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81276,"kb_used_data":45924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93663876,"statfs":{"total":95995035648,"available":95911809024,"internally_reserved":0,"allocated":47026176,"data_stored":61334646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42745856,"data_stored":58563460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":33468416,"data_stored":39092755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37781504,"data_stored":53432560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":71213056,"data_stored":100837890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56184832,"data_stored":74327372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28954624,"data_stored":43789952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4079616,"data_stored":4066399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":48607232,"data_stored":63555729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":29798400,"data_stored":41121584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":51761152,"data_stored":89808114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2387968,"data_stored":2367002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28995584,"data_stored":41894680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:42:32.966 INFO:tasks.thrashosds.thrasher:Setting 1.a to [4, 9, 2, 5, 11, 8] 2024-09-11T03:42:32.966 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.a', '4', '9', '2', '5', '11', '8'] 2024-09-11T03:42:32.966 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pg-upmap-items 1.a 4 9 2 5 11 8 2024-09-11T03:42:33.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:33.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:33.034 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:33.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:33.096 INFO:tasks.rados.rados.2.smithi175.stdout:1120: finishing write tid 1 to smithi17557870-1 2024-09-11T03:42:33.096 INFO:tasks.rados.rados.2.smithi175.stdout:1120: finishing write tid 2 to smithi17557870-1 2024-09-11T03:42:33.096 INFO:tasks.rados.rados.2.smithi175.stdout:1120: finishing write tid 3 to smithi17557870-1 2024-09-11T03:42:33.096 INFO:tasks.rados.rados.2.smithi175.stdout:1120: finishing write tid 4 to smithi17557870-1 2024-09-11T03:42:33.096 INFO:tasks.rados.rados.2.smithi175.stdout:1120: oid 1 updating version 0 to 393 2024-09-11T03:42:33.096 INFO:tasks.rados.rados.2.smithi175.stdout:1120: oid 1 updating version 393 to 394 2024-09-11T03:42:33.096 INFO:tasks.rados.rados.2.smithi175.stdout:1120: oid 1 updating version 394 to 395 2024-09-11T03:42:33.096 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 395 (ObjNum 393 snap 122 seq_num 393) dirty exists 2024-09-11T03:42:33.096 INFO:tasks.rados.rados.2.smithi175.stdout:1120: left oid 1 (ObjNum 393 snap 122 seq_num 393) 2024-09-11T03:42:33.096 INFO:tasks.rados.rados.2.smithi175.stdout:1121: finishing write tid 1 to smithi17557870-40 2024-09-11T03:42:33.097 INFO:tasks.rados.rados.2.smithi175.stdout:1121: finishing write tid 2 to smithi17557870-40 2024-09-11T03:42:33.097 INFO:tasks.rados.rados.2.smithi175.stdout:1121: finishing write tid 3 to smithi17557870-40 2024-09-11T03:42:33.097 INFO:tasks.rados.rados.2.smithi175.stdout:1121: finishing write tid 4 to smithi17557870-40 2024-09-11T03:42:33.097 INFO:tasks.rados.rados.2.smithi175.stdout:1121: oid 40 updating version 0 to 357 2024-09-11T03:42:33.097 INFO:tasks.rados.rados.2.smithi175.stdout:1121: oid 40 updating version 357 to 359 2024-09-11T03:42:33.097 INFO:tasks.rados.rados.2.smithi175.stdout:1121: oid 40 version 359 is already newer than 358 2024-09-11T03:42:33.097 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 359 (ObjNum 394 snap 122 seq_num 394) dirty exists 2024-09-11T03:42:33.097 INFO:tasks.rados.rados.2.smithi175.stdout:1121: left oid 40 (ObjNum 394 snap 122 seq_num 394) 2024-09-11T03:42:33.135 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:33.156 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:33.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:33.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:33.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:32 smithi121 ceph-mon[30729]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:42:33.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:32 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3382167767' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:42:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:32 smithi175 ceph-mon[30894]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:42:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:32 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3382167767' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:42:33.258 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:33.279 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:33.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:33.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:33.381 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:33.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:32 smithi084 ceph-mon[27254]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:42:33.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:32 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3382167767' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:42:33.402 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:33.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:33.403 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:33.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:33.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:33.527 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:33.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:33.628 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:33.637 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:33.648 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:33.650 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:33.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:33.651 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:33.669 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:33.751 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:33.777 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:33.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:33.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:33.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:33.904 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:33.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:33.912 INFO:tasks.rados.rados.2.smithi175.stdout:1118: done (4 left) 2024-09-11T03:42:33.912 INFO:tasks.rados.rados.2.smithi175.stdout:1119: done (3 left) 2024-09-11T03:42:33.912 INFO:tasks.rados.rados.2.smithi175.stdout:1120: done (2 left) 2024-09-11T03:42:33.913 INFO:tasks.rados.rados.2.smithi175.stdout:1121: done (1 left) 2024-09-11T03:42:33.913 INFO:tasks.rados.rados.2.smithi175.stdout:1122: done (0 left) 2024-09-11T03:42:33.913 INFO:tasks.rados.rados.2.smithi175.stdout:1123: delete oid 5 current snap is 123 2024-09-11T03:42:33.915 INFO:tasks.rados.rados.2.smithi175.stdout:1123: done (0 left) 2024-09-11T03:42:33.915 INFO:tasks.rados.rados.2.smithi175.stdout:1124: copy_from oid 44 from oid 3 current snap is 123 2024-09-11T03:42:33.915 INFO:tasks.rados.rados.2.smithi175.stdout:1125: rollback oid 41 current snap is 123 2024-09-11T03:42:33.915 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 41 to 120 2024-09-11T03:42:33.916 INFO:tasks.rados.rados.2.smithi175.stdout:1126: write (excl) oid 30 current snap is 123 2024-09-11T03:42:33.916 INFO:tasks.rados.rados.2.smithi175.stdout:1126: seq_num 396 ranges {655572=648960,1922987=732585} 2024-09-11T03:42:33.922 INFO:tasks.rados.rados.2.smithi175.stdout:1126: writing smithi17557870-30 from 655572 to 1304532 tid 1 2024-09-11T03:42:33.931 INFO:tasks.rados.rados.2.smithi175.stdout:1126: writing smithi17557870-30 from 1922987 to 2655572 tid 2 2024-09-11T03:42:33.931 INFO:tasks.rados.rados.2.smithi175.stdout:1127: read oid 9 snap -1 2024-09-11T03:42:33.931 INFO:tasks.rados.rados.2.smithi175.stdout:1127: expect (ObjNum 277 snap 73 seq_num 277) 2024-09-11T03:42:33.931 INFO:tasks.rados.rados.2.smithi175.stdout:1128: snap_create 2024-09-11T03:42:33.931 INFO:tasks.rados.rados.2.smithi175.stdout:1125: finishing rollback tid 0 to smithi17557870-41 2024-09-11T03:42:33.931 INFO:tasks.rados.rados.2.smithi175.stdout:1125: finishing rollback tid 1 to smithi17557870-41 2024-09-11T03:42:33.932 INFO:tasks.rados.rados.2.smithi175.stdout:1125: finishing rollback tid 2 to smithi17557870-41 2024-09-11T03:42:33.932 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 317 (ObjNum 326 snap 91 seq_num 326) dirty exists 2024-09-11T03:42:33.932 INFO:tasks.rados.rados.2.smithi175.stdout:1124: finishing copy_from racing read to smithi17557870-44 2024-09-11T03:42:33.932 INFO:tasks.rados.rados.2.smithi175.stdout:1126: finishing write tid 1 to smithi17557870-30 2024-09-11T03:42:33.932 INFO:tasks.rados.rados.2.smithi175.stdout:1124: finishing copy_from to smithi17557870-44 2024-09-11T03:42:33.932 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 423 (ObjNum 384 snap 117 seq_num 384) dirty exists 2024-09-11T03:42:33.936 INFO:tasks.rados.rados.2.smithi175.stdout:1126: finishing write tid 2 to smithi17557870-30 2024-09-11T03:42:33.936 INFO:tasks.rados.rados.2.smithi175.stdout:1126: finishing write tid 3 to smithi17557870-30 2024-09-11T03:42:33.938 INFO:tasks.rados.rados.2.smithi175.stdout:1126: finishing write tid 4 to smithi17557870-30 2024-09-11T03:42:33.938 INFO:tasks.rados.rados.2.smithi175.stdout:1126: oid 30 updating version 0 to 441 2024-09-11T03:42:33.938 INFO:tasks.rados.rados.2.smithi175.stdout:1126: oid 30 updating version 441 to 442 2024-09-11T03:42:33.938 INFO:tasks.rados.rados.2.smithi175.stdout:1126: oid 30 version 442 is already newer than 440 2024-09-11T03:42:33.939 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 442 (ObjNum 396 snap 123 seq_num 396) dirty exists 2024-09-11T03:42:33.939 INFO:tasks.rados.rados.2.smithi175.stdout:1126: left oid 30 (ObjNum 396 snap 123 seq_num 396) 2024-09-11T03:42:33.940 INFO:tasks.rados.rados.2.smithi175.stdout:1127: expect (ObjNum 277 snap 73 seq_num 277) 2024-09-11T03:42:34.006 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:34.018 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:33 smithi084 ceph-mon[27254]: pgmap v658: 57 pgs: 57 active+clean; 219 MiB data, 791 MiB used, 893 GiB / 894 GiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 12 op/s; 4.5 MiB/s, 12 objects/s recovering 2024-09-11T03:42:34.019 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:33 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:34.019 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:33 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3382167767' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:42:34.019 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:33 smithi084 ceph-mon[27254]: osdmap e336: 12 total, 12 up, 10 in 2024-09-11T03:42:34.028 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:34.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:34.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:34.130 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:34.152 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:34.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:34.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:34.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:33 smithi121 ceph-mon[30729]: pgmap v658: 57 pgs: 57 active+clean; 219 MiB data, 791 MiB used, 893 GiB / 894 GiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 12 op/s; 4.5 MiB/s, 12 objects/s recovering 2024-09-11T03:42:34.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:33 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:34.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:33 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3382167767' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:42:34.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:33 smithi121 ceph-mon[30729]: osdmap e336: 12 total, 12 up, 10 in 2024-09-11T03:42:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:33 smithi175 ceph-mon[30894]: pgmap v658: 57 pgs: 57 active+clean; 219 MiB data, 791 MiB used, 893 GiB / 894 GiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 12 op/s; 4.5 MiB/s, 12 objects/s recovering 2024-09-11T03:42:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:33 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:33 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3382167767' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:42:34.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:33 smithi175 ceph-mon[30894]: osdmap e336: 12 total, 12 up, 10 in 2024-09-11T03:42:34.254 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:34.275 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:34.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:34.276 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:34.377 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:34.398 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:34.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:34.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:34.500 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:34.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:34.521 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:34.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:34.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:34.649 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:34.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:34.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:34.751 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:34.772 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:34.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:34.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:34.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:34.900 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:34.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:34.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:34.914 INFO:teuthology.orchestra.run.smithi084.stderr:set 1.a pg_upmap_items mapping to [4->9,2->5,11->8] 2024-09-11T03:42:34.921 INFO:tasks.rados.rados.2.smithi175.stdout:1124: done (4 left) 2024-09-11T03:42:34.922 INFO:tasks.rados.rados.2.smithi175.stdout:1125: done (3 left) 2024-09-11T03:42:34.922 INFO:tasks.rados.rados.2.smithi175.stdout:1126: done (2 left) 2024-09-11T03:42:34.922 INFO:tasks.rados.rados.2.smithi175.stdout:1127: done (1 left) 2024-09-11T03:42:34.922 INFO:tasks.rados.rados.2.smithi175.stdout:1128: done (0 left) 2024-09-11T03:42:34.922 INFO:tasks.rados.rados.2.smithi175.stdout:1129: copy_from oid 27 from oid 26 current snap is 124 2024-09-11T03:42:34.922 INFO:tasks.rados.rados.2.smithi175.stdout:1130: delete oid 5 current snap is 124 2024-09-11T03:42:34.923 INFO:tasks.rados.rados.2.smithi175.stdout:1130: done (1 left) 2024-09-11T03:42:34.923 INFO:tasks.rados.rados.2.smithi175.stdout:1131: snap_create 2024-09-11T03:42:34.931 INFO:tasks.rados.rados.2.smithi175.stdout:1129: finishing copy_from racing read to smithi17557870-27 2024-09-11T03:42:34.933 INFO:tasks.rados.rados.2.smithi175.stdout:1129: finishing copy_from to smithi17557870-27 2024-09-11T03:42:34.933 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 182 (ObjNum 380 snap 117 seq_num 380) dirty exists 2024-09-11T03:42:35.002 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:35.024 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:35.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:35.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:35.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:35.152 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:35.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:35.165 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:34 smithi084 ceph-mon[27254]: osdmap e337: 12 total, 12 up, 10 in 2024-09-11T03:42:35.165 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:34 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1123561882' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.a", "id": [4, 9, 2, 5, 11, 8]} : dispatch 2024-09-11T03:42:35.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:34 smithi121 ceph-mon[30729]: osdmap e337: 12 total, 12 up, 10 in 2024-09-11T03:42:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:34 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1123561882' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.a", "id": [4, 9, 2, 5, 11, 8]} : dispatch 2024-09-11T03:42:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:34 smithi175 ceph-mon[30894]: osdmap e337: 12 total, 12 up, 10 in 2024-09-11T03:42:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:34 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1123561882' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.a", "id": [4, 9, 2, 5, 11, 8]} : dispatch 2024-09-11T03:42:35.243 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:42:35.243 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:42:35.243 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:42:35.243 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:42:35.244 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:42:35.257 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:35.283 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:35.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:35.385 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:35.409 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:35.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:35.510 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:35.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:35.538 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:35.538 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:35.636 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:42:35.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:35.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:35.702 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:35.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:35.804 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:35.826 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:35.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:35.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:35.927 INFO:tasks.rados.rados.2.smithi175.stdout:1129: done (1 left) 2024-09-11T03:42:35.927 INFO:tasks.rados.rados.2.smithi175.stdout:1131: done (0 left) 2024-09-11T03:42:35.927 INFO:tasks.rados.rados.2.smithi175.stdout:1132: rollback oid 4 current snap is 125 2024-09-11T03:42:35.927 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 4 to 115 2024-09-11T03:42:35.927 INFO:tasks.rados.rados.2.smithi175.stdout:1133: snap_remove snap 118 2024-09-11T03:42:35.928 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:35.949 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:35.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:35.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:35.972 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T03:42:36.052 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:36.074 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:42:36.074 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:42:36.074 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:42:36.074 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:42:36.074 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:42:36.074 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:42:36.074 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:42:36.074 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:42:36.075 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:42:36.075 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:42:36.075 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:42:36.075 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:42:36.075 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:42:36.075 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:42:36.075 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:42:36.075 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:42:36.075 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:42:36.075 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:42:36.076 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:42:36.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:42:36.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:42:36.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:42:36.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:42:36.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:42:36.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:36.077 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:36.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:36.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:35 smithi121 ceph-mon[30729]: pgmap v661: 57 pgs: 57 active+clean; 217 MiB data, 774 MiB used, 893 GiB / 894 GiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 17 op/s; 4.5 MiB/s, 12 objects/s recovering 2024-09-11T03:42:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:35 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1123561882' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.a", "id": [4, 9, 2, 5, 11, 8]}]': finished 2024-09-11T03:42:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:35 smithi121 ceph-mon[30729]: osdmap e338: 12 total, 12 up, 10 in 2024-09-11T03:42:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:35 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1985226825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:36.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:36.205 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:36.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:36.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:35 smithi175 ceph-mon[30894]: pgmap v661: 57 pgs: 57 active+clean; 217 MiB data, 774 MiB used, 893 GiB / 894 GiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 17 op/s; 4.5 MiB/s, 12 objects/s recovering 2024-09-11T03:42:36.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:35 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1123561882' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.a", "id": [4, 9, 2, 5, 11, 8]}]': finished 2024-09-11T03:42:36.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:35 smithi175 ceph-mon[30894]: osdmap e338: 12 total, 12 up, 10 in 2024-09-11T03:42:36.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:35 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1985226825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:36.307 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:36.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:36.329 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:36.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:36.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:35 smithi084 ceph-mon[27254]: pgmap v661: 57 pgs: 57 active+clean; 217 MiB data, 774 MiB used, 893 GiB / 894 GiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 17 op/s; 4.5 MiB/s, 12 objects/s recovering 2024-09-11T03:42:36.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:35 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1123561882' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.a", "id": [4, 9, 2, 5, 11, 8]}]': finished 2024-09-11T03:42:36.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:35 smithi084 ceph-mon[27254]: osdmap e338: 12 total, 12 up, 10 in 2024-09-11T03:42:36.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:35 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1985226825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:36.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:36.455 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:36.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:36.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:36.557 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:36.579 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:36.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:36.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:36.656 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:36.681 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:36.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:36.703 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:36.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:36.804 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:36.830 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:36.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:36.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:36.890 INFO:tasks.rados.rados.2.smithi175.stdout:1133: done (1 left) 2024-09-11T03:42:36.890 INFO:tasks.rados.rados.2.smithi175.stdout:1134: snap_create 2024-09-11T03:42:36.890 INFO:tasks.rados.rados.2.smithi175.stdout:1132: finishing rollback tid 1 to smithi17557870-4 2024-09-11T03:42:36.890 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T03:42:36.932 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:36.956 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:36.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:36.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:37.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:37.080 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:37.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:37.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:37.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:37.208 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:37.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:37.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:37.224 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:36 smithi084 ceph-mon[27254]: osdmap e339: 12 total, 12 up, 10 in 2024-09-11T03:42:37.224 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:36 smithi084 ceph-mon[27254]: osdmap e340: 12 total, 12 up, 10 in 2024-09-11T03:42:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:36 smithi175 ceph-mon[30894]: osdmap e339: 12 total, 12 up, 10 in 2024-09-11T03:42:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:36 smithi175 ceph-mon[30894]: osdmap e340: 12 total, 12 up, 10 in 2024-09-11T03:42:37.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:37.339 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:37.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:37.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:36 smithi121 ceph-mon[30729]: osdmap e339: 12 total, 12 up, 10 in 2024-09-11T03:42:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:36 smithi121 ceph-mon[30729]: osdmap e340: 12 total, 12 up, 10 in 2024-09-11T03:42:37.441 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:37.464 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:37.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:37.565 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:37.569 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:37.586 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:37.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:37.688 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:37.709 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:37.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:37.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:37.811 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:37.837 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:37.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:37.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:37.887 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:42:37.893 INFO:tasks.rados.rados.2.smithi175.stdout:1132: done (1 left) 2024-09-11T03:42:37.893 INFO:tasks.rados.rados.2.smithi175.stdout:1134: done (0 left) 2024-09-11T03:42:37.893 INFO:tasks.rados.rados.2.smithi175.stdout:1135: copy_from oid 30 from oid 19 current snap is 126 2024-09-11T03:42:37.893 INFO:tasks.rados.rados.2.smithi175.stdout:1136: write oid 35 current snap is 126 2024-09-11T03:42:37.893 INFO:tasks.rados.rados.2.smithi175.stdout:1136: seq_num 398 ranges {689186=799458,1923598=752130} 2024-09-11T03:42:37.900 INFO:tasks.rados.rados.2.smithi175.stdout:1136: writing smithi17557870-35 from 689186 to 1488644 tid 1 2024-09-11T03:42:37.907 INFO:tasks.rados.rados.2.smithi175.stdout:1136: writing smithi17557870-35 from 1923598 to 2675728 tid 2 2024-09-11T03:42:37.907 INFO:tasks.rados.rados.2.smithi175.stdout:1137: read oid 50 snap -1 2024-09-11T03:42:37.907 INFO:tasks.rados.rados.2.smithi175.stdout:1137: expect (ObjNum 336 snap 95 seq_num 336) 2024-09-11T03:42:37.907 INFO:tasks.rados.rados.2.smithi175.stdout:1138: snap_remove snap 125 2024-09-11T03:42:37.939 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:37.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:37.962 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:37.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:38.063 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:38.088 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:38.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:38.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:37 smithi084 ceph-mon[27254]: pgmap v664: 57 pgs: 57 active+clean; 223 MiB data, 785 MiB used, 893 GiB / 894 GiB avail; 5.8 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2024-09-11T03:42:38.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1740090330' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:38.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1740090330' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:38.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:37 smithi084 ceph-mon[27254]: osdmap e341: 12 total, 12 up, 10 in 2024-09-11T03:42:38.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:38.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:38.212 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:38.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:38.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:37 smithi175 ceph-mon[30894]: pgmap v664: 57 pgs: 57 active+clean; 223 MiB data, 785 MiB used, 893 GiB / 894 GiB avail; 5.8 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2024-09-11T03:42:38.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:37 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1740090330' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:38.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:37 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1740090330' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:38.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:37 smithi175 ceph-mon[30894]: osdmap e341: 12 total, 12 up, 10 in 2024-09-11T03:42:38.287 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:42:38.287 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:42:38.314 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:38.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:37 smithi121 ceph-mon[30729]: pgmap v664: 57 pgs: 57 active+clean; 223 MiB data, 785 MiB used, 893 GiB / 894 GiB avail; 5.8 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2024-09-11T03:42:38.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:37 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1740090330' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:38.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:37 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1740090330' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:38.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:37 smithi121 ceph-mon[30729]: osdmap e341: 12 total, 12 up, 10 in 2024-09-11T03:42:38.339 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:38.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:38.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:38.441 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:38.467 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:38.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:38.569 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:38.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:38.595 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:38.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:38.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:42:38.696 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:38.722 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:38.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:38.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:38.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:38.850 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:38.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:38.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:38.893 INFO:tasks.rados.rados.2.smithi175.stdout:1138: done (3 left) 2024-09-11T03:42:38.893 INFO:tasks.rados.rados.2.smithi175.stdout:1139: write (excl) oid 43 current snap is 126 2024-09-11T03:42:38.893 INFO:tasks.rados.rados.2.smithi175.stdout:1135: finishing copy_from to smithi17557870-30 2024-09-11T03:42:38.893 INFO:tasks.rados.rados.2.smithi175.stdout:1135: got expected ENOENT (src dne) 2024-09-11T03:42:38.893 INFO:tasks.rados.rados.2.smithi175.stdout:1139: seq_num 399 ranges {705993=474707,1582080=612141} 2024-09-11T03:42:38.899 INFO:tasks.rados.rados.2.smithi175.stdout:1139: writing smithi17557870-43 from 705993 to 1180700 tid 1 2024-09-11T03:42:38.905 INFO:tasks.rados.rados.2.smithi175.stdout:1139: writing smithi17557870-43 from 1582080 to 2194221 tid 2 2024-09-11T03:42:38.905 INFO:tasks.rados.rados.2.smithi175.stdout:1135: finishing copy_from racing read to smithi17557870-30 2024-09-11T03:42:38.905 INFO:tasks.rados.rados.2.smithi175.stdout:1136: finishing write tid 1 to smithi17557870-35 2024-09-11T03:42:38.905 INFO:tasks.rados.rados.2.smithi175.stdout:1136: finishing write tid 2 to smithi17557870-35 2024-09-11T03:42:38.905 INFO:tasks.rados.rados.2.smithi175.stdout:1136: finishing write tid 3 to smithi17557870-35 2024-09-11T03:42:38.905 INFO:tasks.rados.rados.2.smithi175.stdout:1136: finishing write tid 4 to smithi17557870-35 2024-09-11T03:42:38.905 INFO:tasks.rados.rados.2.smithi175.stdout:1136: oid 35 updating version 0 to 308 2024-09-11T03:42:38.905 INFO:tasks.rados.rados.2.smithi175.stdout:1136: oid 35 version 308 is already newer than 307 2024-09-11T03:42:38.906 INFO:tasks.rados.rados.2.smithi175.stdout:1136: oid 35 version 308 is already newer than 306 2024-09-11T03:42:38.906 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 308 (ObjNum 398 snap 126 seq_num 398) dirty exists 2024-09-11T03:42:38.906 INFO:tasks.rados.rados.2.smithi175.stdout:1136: left oid 35 (ObjNum 398 snap 126 seq_num 398) 2024-09-11T03:42:38.906 INFO:tasks.rados.rados.2.smithi175.stdout:1137: expect (ObjNum 336 snap 95 seq_num 336) 2024-09-11T03:42:38.951 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:38.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:38.975 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:38.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:39.024 INFO:tasks.rados.rados.2.smithi175.stdout:1139: finishing write tid 1 to smithi17557870-43 2024-09-11T03:42:39.024 INFO:tasks.rados.rados.2.smithi175.stdout:1139: finishing write tid 2 to smithi17557870-43 2024-09-11T03:42:39.024 INFO:tasks.rados.rados.2.smithi175.stdout:1139: finishing write tid 3 to smithi17557870-43 2024-09-11T03:42:39.024 INFO:tasks.rados.rados.2.smithi175.stdout:1139: finishing write tid 4 to smithi17557870-43 2024-09-11T03:42:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:1139: oid 43 updating version 0 to 360 2024-09-11T03:42:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:1139: oid 43 updating version 360 to 362 2024-09-11T03:42:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:1139: oid 43 version 362 is already newer than 361 2024-09-11T03:42:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 362 (ObjNum 399 snap 126 seq_num 399) dirty exists 2024-09-11T03:42:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:1139: left oid 43 (ObjNum 399 snap 126 seq_num 399) 2024-09-11T03:42:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:1135: done (3 left) 2024-09-11T03:42:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:1136: done (2 left) 2024-09-11T03:42:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:1137: done (1 left) 2024-09-11T03:42:39.026 INFO:tasks.rados.rados.2.smithi175.stdout:1139: done (0 left) 2024-09-11T03:42:39.026 INFO:tasks.rados.rados.2.smithi175.stdout:1140: write (excl) oid 40 current snap is 126 2024-09-11T03:42:39.026 INFO:tasks.rados.rados.2.smithi175.stdout:1140: seq_num 400 ranges {722800=549956,2040562=682238} 2024-09-11T03:42:39.030 INFO:tasks.rados.rados.2.smithi175.stdout:1140: writing smithi17557870-40 from 722800 to 1272756 tid 1 2024-09-11T03:42:39.038 INFO:tasks.rados.rados.2.smithi175.stdout:1140: writing smithi17557870-40 from 2040562 to 2722800 tid 2 2024-09-11T03:42:39.038 INFO:tasks.rados.rados.2.smithi175.stdout:1141: snap_create 2024-09-11T03:42:39.038 INFO:tasks.rados.rados.2.smithi175.stdout:1140: finishing write tid 1 to smithi17557870-40 2024-09-11T03:42:39.041 INFO:tasks.rados.rados.2.smithi175.stdout:1140: finishing write tid 2 to smithi17557870-40 2024-09-11T03:42:39.041 INFO:tasks.rados.rados.2.smithi175.stdout:1140: finishing write tid 3 to smithi17557870-40 2024-09-11T03:42:39.041 INFO:tasks.rados.rados.2.smithi175.stdout:1140: finishing write tid 4 to smithi17557870-40 2024-09-11T03:42:39.041 INFO:tasks.rados.rados.2.smithi175.stdout:1140: oid 40 updating version 0 to 363 2024-09-11T03:42:39.041 INFO:tasks.rados.rados.2.smithi175.stdout:1140: oid 40 updating version 363 to 365 2024-09-11T03:42:39.041 INFO:tasks.rados.rados.2.smithi175.stdout:1140: oid 40 version 365 is already newer than 364 2024-09-11T03:42:39.042 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 365 (ObjNum 400 snap 126 seq_num 400) dirty exists 2024-09-11T03:42:39.042 INFO:tasks.rados.rados.2.smithi175.stdout:1140: left oid 40 (ObjNum 400 snap 126 seq_num 400) 2024-09-11T03:42:39.075 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:39.096 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:39.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:39.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:39.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:39.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:39.228 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:39.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:39.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:39.355 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:39.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:39.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:39.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:39.481 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:39.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:39.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:39.587 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T03:42:39.609 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:39.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:39.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:39.670 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:39.711 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:39.738 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:39.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:39.839 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:39.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:39.869 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:39.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:39.938 INFO:tasks.rados.rados.2.smithi175.stdout:1140: done (1 left) 2024-09-11T03:42:39.938 INFO:tasks.rados.rados.2.smithi175.stdout:1141: done (0 left) 2024-09-11T03:42:39.938 INFO:tasks.rados.rados.2.smithi175.stdout:1142: write (excl) oid 48 current snap is 127 2024-09-11T03:42:39.938 INFO:tasks.rados.rados.2.smithi175.stdout:1142: seq_num 401 ranges {739607=625205,1982691=499457} 2024-09-11T03:42:39.945 INFO:tasks.rados.rados.2.smithi175.stdout:1142: writing smithi17557870-48 from 739607 to 1364812 tid 1 2024-09-11T03:42:39.951 INFO:tasks.rados.rados.2.smithi175.stdout:1142: writing smithi17557870-48 from 1982691 to 2482148 tid 2 2024-09-11T03:42:39.952 INFO:tasks.rados.rados.2.smithi175.stdout:1142: finishing write tid 1 to smithi17557870-48 2024-09-11T03:42:39.952 INFO:tasks.rados.rados.2.smithi175.stdout:1143: snap_remove snap 124 2024-09-11T03:42:39.970 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:39.995 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:39.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:40.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:40.123 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:40.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:40.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:39 smithi084 ceph-mon[27254]: pgmap v667: 57 pgs: 57 active+clean; 228 MiB data, 793 MiB used, 893 GiB / 894 GiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-11T03:42:40.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:39 smithi084 ceph-mon[27254]: osdmap e342: 12 total, 12 up, 10 in 2024-09-11T03:42:40.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:39 smithi121 ceph-mon[30729]: pgmap v667: 57 pgs: 57 active+clean; 228 MiB data, 793 MiB used, 893 GiB / 894 GiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-11T03:42:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:39 smithi121 ceph-mon[30729]: osdmap e342: 12 total, 12 up, 10 in 2024-09-11T03:42:40.225 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:40.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:39 smithi175 ceph-mon[30894]: pgmap v667: 57 pgs: 57 active+clean; 228 MiB data, 793 MiB used, 893 GiB / 894 GiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-11T03:42:40.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:39 smithi175 ceph-mon[30894]: osdmap e342: 12 total, 12 up, 10 in 2024-09-11T03:42:40.251 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:40.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:40.287 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:40.289 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:42:40.289 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:42:40.307 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:42:40.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:42:40.324 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:42:40.325 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:42:40.340 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:42:40.340 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:42:40.352 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:40.357 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:42:40.357 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:42:40.374 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:42:40.374 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:42:40.377 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:40.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:40.378 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:40.391 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:42:40.392 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:42:40.408 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:42:40.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:42:40.425 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:42:40.425 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:42:40.442 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:42:40.442 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:42:40.458 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:42:40.459 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:42:40.475 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:42:40.476 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:42:40.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:40.506 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:40.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:40.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:40.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:40.633 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:40.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:40.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:40.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:40.757 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:40.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:40.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:40.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:40.885 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:40.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:40.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:40.940 INFO:tasks.rados.rados.2.smithi175.stdout:1143: done (1 left) 2024-09-11T03:42:40.940 INFO:tasks.rados.rados.2.smithi175.stdout:1144: write (excl) oid 45 current snap is 127 2024-09-11T03:42:40.940 INFO:tasks.rados.rados.2.smithi175.stdout:1142: finishing write tid 2 to smithi17557870-48 2024-09-11T03:42:40.940 INFO:tasks.rados.rados.2.smithi175.stdout:1142: finishing write tid 3 to smithi17557870-48 2024-09-11T03:42:40.940 INFO:tasks.rados.rados.2.smithi175.stdout:1142: finishing write tid 4 to smithi17557870-48 2024-09-11T03:42:40.940 INFO:tasks.rados.rados.2.smithi175.stdout:1142: oid 48 updating version 0 to 425 2024-09-11T03:42:40.940 INFO:tasks.rados.rados.2.smithi175.stdout:1142: oid 48 version 425 is already newer than 423 2024-09-11T03:42:40.940 INFO:tasks.rados.rados.2.smithi175.stdout:1142: oid 48 version 425 is already newer than 424 2024-09-11T03:42:40.940 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 425 (ObjNum 401 snap 127 seq_num 401) dirty exists 2024-09-11T03:42:40.940 INFO:tasks.rados.rados.2.smithi175.stdout:1142: left oid 48 (ObjNum 401 snap 127 seq_num 401) 2024-09-11T03:42:40.941 INFO:tasks.rados.rados.2.smithi175.stdout:1144: seq_num 402 ranges {756414=700454,2041173=715241} 2024-09-11T03:42:40.948 INFO:tasks.rados.rados.2.smithi175.stdout:1144: writing smithi17557870-45 from 756414 to 1456868 tid 1 2024-09-11T03:42:40.956 INFO:tasks.rados.rados.2.smithi175.stdout:1144: writing smithi17557870-45 from 2041173 to 2756414 tid 2 2024-09-11T03:42:40.956 INFO:tasks.rados.rados.2.smithi175.stdout:1142: done (1 left) 2024-09-11T03:42:40.957 INFO:tasks.rados.rados.2.smithi175.stdout:1145: delete oid 16 current snap is 127 2024-09-11T03:42:40.957 INFO:tasks.rados.rados.2.smithi175.stdout:1144: finishing write tid 1 to smithi17557870-45 2024-09-11T03:42:40.958 INFO:tasks.rados.rados.2.smithi175.stdout:1145: done (1 left) 2024-09-11T03:42:40.958 INFO:tasks.rados.rados.2.smithi175.stdout:1146: write (excl) oid 41 current snap is 127 2024-09-11T03:42:40.958 INFO:tasks.rados.rados.2.smithi175.stdout:1146: seq_num 404 ranges {790028=567305,1874490=763137} 2024-09-11T03:42:40.965 INFO:tasks.rados.rados.2.smithi175.stdout:1146: writing smithi17557870-41 from 790028 to 1357333 tid 1 2024-09-11T03:42:40.973 INFO:tasks.rados.rados.2.smithi175.stdout:1146: writing smithi17557870-41 from 1874490 to 2637627 tid 2 2024-09-11T03:42:40.973 INFO:tasks.rados.rados.2.smithi175.stdout:1147: write oid 46 current snap is 127 2024-09-11T03:42:40.974 INFO:tasks.rados.rados.2.smithi175.stdout:1144: finishing write tid 2 to smithi17557870-45 2024-09-11T03:42:40.974 INFO:tasks.rados.rados.2.smithi175.stdout:1144: finishing write tid 3 to smithi17557870-45 2024-09-11T03:42:40.974 INFO:tasks.rados.rados.2.smithi175.stdout:1144: finishing write tid 4 to smithi17557870-45 2024-09-11T03:42:40.974 INFO:tasks.rados.rados.2.smithi175.stdout:1144: oid 45 updating version 0 to 316 2024-09-11T03:42:40.974 INFO:tasks.rados.rados.2.smithi175.stdout:1144: oid 45 version 316 is already newer than 315 2024-09-11T03:42:40.974 INFO:tasks.rados.rados.2.smithi175.stdout:1144: oid 45 version 316 is already newer than 312 2024-09-11T03:42:40.975 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 316 (ObjNum 402 snap 127 seq_num 402) dirty exists 2024-09-11T03:42:40.975 INFO:tasks.rados.rados.2.smithi175.stdout:1144: left oid 45 (ObjNum 402 snap 127 seq_num 402) 2024-09-11T03:42:40.975 INFO:tasks.rados.rados.2.smithi175.stdout:1146: finishing write tid 1 to smithi17557870-41 2024-09-11T03:42:40.975 INFO:tasks.rados.rados.2.smithi175.stdout:1147: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2024-09-11T03:42:40.978 INFO:tasks.rados.rados.2.smithi175.stdout:1147: writing smithi17557870-46 from 406835 to 1049389 tid 1 2024-09-11T03:42:40.984 INFO:tasks.rados.rados.2.smithi175.stdout:1147: writing smithi17557870-46 from 1532972 to 2039767 tid 2 2024-09-11T03:42:40.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:40.988 INFO:tasks.rados.rados.2.smithi175.stdout:1147: writing smithi17557870-46 from 2772312 to 2806835 tid 3 2024-09-11T03:42:40.989 INFO:tasks.rados.rados.2.smithi175.stdout:1144: done (2 left) 2024-09-11T03:42:40.989 INFO:tasks.rados.rados.2.smithi175.stdout:1148: write (excl) oid 7 current snap is 127 2024-09-11T03:42:40.989 INFO:tasks.rados.rados.2.smithi175.stdout:1146: finishing write tid 2 to smithi17557870-41 2024-09-11T03:42:40.989 INFO:tasks.rados.rados.2.smithi175.stdout:1146: finishing write tid 3 to smithi17557870-41 2024-09-11T03:42:40.989 INFO:tasks.rados.rados.2.smithi175.stdout:1146: finishing write tid 4 to smithi17557870-41 2024-09-11T03:42:40.989 INFO:tasks.rados.rados.2.smithi175.stdout:1146: oid 41 updating version 0 to 321 2024-09-11T03:42:40.989 INFO:tasks.rados.rados.2.smithi175.stdout:1146: oid 41 updating version 321 to 322 2024-09-11T03:42:40.989 INFO:tasks.rados.rados.2.smithi175.stdout:1146: oid 41 updating version 322 to 323 2024-09-11T03:42:40.989 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 323 (ObjNum 404 snap 127 seq_num 404) dirty exists 2024-09-11T03:42:40.989 INFO:tasks.rados.rados.2.smithi175.stdout:1146: left oid 41 (ObjNum 404 snap 127 seq_num 404) 2024-09-11T03:42:40.989 INFO:tasks.rados.rados.2.smithi175.stdout:1147: finishing write tid 1 to smithi17557870-46 2024-09-11T03:42:40.989 INFO:tasks.rados.rados.2.smithi175.stdout:1147: finishing write tid 2 to smithi17557870-46 2024-09-11T03:42:40.989 INFO:tasks.rados.rados.2.smithi175.stdout:1148: seq_num 406 ranges {423642=717803,1875101=766806} 2024-09-11T03:42:40.994 INFO:tasks.rados.rados.2.smithi175.stdout:1148: writing smithi17557870-7 from 423642 to 1141445 tid 1 2024-09-11T03:42:41.003 INFO:tasks.rados.rados.2.smithi175.stdout:1148: writing smithi17557870-7 from 1875101 to 2641907 tid 2 2024-09-11T03:42:41.004 INFO:tasks.rados.rados.2.smithi175.stdout:1146: done (2 left) 2024-09-11T03:42:41.004 INFO:tasks.rados.rados.2.smithi175.stdout:1149: read oid 5 snap -1 2024-09-11T03:42:41.004 INFO:tasks.rados.rados.2.smithi175.stdout:1149: expect deleted 2024-09-11T03:42:41.004 INFO:tasks.rados.rados.2.smithi175.stdout:1150: read oid 43 snap -1 2024-09-11T03:42:41.004 INFO:tasks.rados.rados.2.smithi175.stdout:1150: expect (ObjNum 399 snap 126 seq_num 399) 2024-09-11T03:42:41.004 INFO:tasks.rados.rados.2.smithi175.stdout:1151: write (excl) oid 2 current snap is 127 2024-09-11T03:42:41.004 INFO:tasks.rados.rados.2.smithi175.stdout:1147: finishing write tid 3 to smithi17557870-46 2024-09-11T03:42:41.004 INFO:tasks.rados.rados.2.smithi175.stdout:1151: seq_num 407 ranges {440449=793052,1933583=510464} 2024-09-11T03:42:41.009 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:41.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:41.010 INFO:tasks.rados.rados.2.smithi175.stdout:1151: writing smithi17557870-2 from 440449 to 1233501 tid 1 2024-09-11T03:42:41.017 INFO:tasks.rados.rados.2.smithi175.stdout:1151: writing smithi17557870-2 from 1933583 to 2444047 tid 2 2024-09-11T03:42:41.017 INFO:tasks.rados.rados.2.smithi175.stdout:1152: snap_remove snap 122 2024-09-11T03:42:41.060 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:42:41.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:41.136 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:41.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:41.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:41.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:40 smithi084 ceph-mon[27254]: osdmap e343: 12 total, 12 up, 10 in 2024-09-11T03:42:41.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:40 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3123211256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:41.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:40 smithi175 ceph-mon[30894]: osdmap e343: 12 total, 12 up, 10 in 2024-09-11T03:42:41.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:40 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3123211256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:41.238 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:41.262 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:41.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:41.263 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:41.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:41.390 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:41.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:41.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:42:41.396 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:42:41.396 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:42:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:42:41.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:40 smithi121 ceph-mon[30729]: osdmap e343: 12 total, 12 up, 10 in 2024-09-11T03:42:41.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:40 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3123211256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:41.492 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:41.513 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:41.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:41.615 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:41.641 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:41.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:41.642 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:41.743 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:41.769 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:41.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:41.770 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:41.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:41.897 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:41.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:41.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:41.945 INFO:tasks.rados.rados.2.smithi175.stdout:1152: done (5 left) 2024-09-11T03:42:41.945 INFO:tasks.rados.rados.2.smithi175.stdout:1153: write (excl) oid 50 current snap is 127 2024-09-11T03:42:41.945 INFO:tasks.rados.rados.2.smithi175.stdout:1147: finishing write tid 4 to smithi17557870-46 2024-09-11T03:42:41.945 INFO:tasks.rados.rados.2.smithi175.stdout:1153: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2024-09-11T03:42:41.949 INFO:tasks.rados.rados.2.smithi175.stdout:1153: writing smithi17557870-50 from 457256 to 925557 tid 1 2024-09-11T03:42:41.957 INFO:tasks.rados.rados.2.smithi175.stdout:1153: writing smithi17557870-50 from 1592065 to 2362540 tid 2 2024-09-11T03:42:41.959 INFO:tasks.rados.rados.2.smithi175.stdout:1153: writing smithi17557870-50 from 2854581 to 2857256 tid 3 2024-09-11T03:42:41.960 INFO:tasks.rados.rados.2.smithi175.stdout:1154: snap_remove snap 123 2024-09-11T03:42:41.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:42.001 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:41 smithi084 ceph-mon[27254]: pgmap v670: 57 pgs: 57 active+clean; 228 MiB data, 795 MiB used, 893 GiB / 894 GiB avail; 5.0 MiB/s rd, 920 KiB/s wr, 6 op/s 2024-09-11T03:42:42.001 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:41 smithi084 ceph-mon[27254]: osdmap e344: 12 total, 12 up, 10 in 2024-09-11T03:42:42.001 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:41 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:42:42.001 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:41 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:42:42.001 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:41 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:42.023 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:42.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:42.027 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T03:42:42.125 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:42.140 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:42.151 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:42.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:42.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:41 smithi175 ceph-mon[30894]: pgmap v670: 57 pgs: 57 active+clean; 228 MiB data, 795 MiB used, 893 GiB / 894 GiB avail; 5.0 MiB/s rd, 920 KiB/s wr, 6 op/s 2024-09-11T03:42:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:41 smithi175 ceph-mon[30894]: osdmap e344: 12 total, 12 up, 10 in 2024-09-11T03:42:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:41 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:42:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:41 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:42:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:41 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:42.253 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:42.275 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:42.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:42.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:42.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:42.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:42.403 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:42.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:42.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:41 smithi121 ceph-mon[30729]: pgmap v670: 57 pgs: 57 active+clean; 228 MiB data, 795 MiB used, 893 GiB / 894 GiB avail; 5.0 MiB/s rd, 920 KiB/s wr, 6 op/s 2024-09-11T03:42:42.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:41 smithi121 ceph-mon[30729]: osdmap e344: 12 total, 12 up, 10 in 2024-09-11T03:42:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:41 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:42:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:41 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:42:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:41 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:42.504 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:42.518 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [6, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:42:42.518 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:42:42.519 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-11T03:42:42.519 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:42:42.530 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:42.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:42.531 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:42.632 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:42.647 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:42.654 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:42.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:42.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:42.756 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:42.782 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:42.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:42.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:42.883 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:42.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:42.910 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:42.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:42.948 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:42:42.955 INFO:tasks.rados.rados.2.smithi175.stdout:1154: done (6 left) 2024-09-11T03:42:42.955 INFO:tasks.rados.rados.2.smithi175.stdout:1155: rollback oid 23 current snap is 127 2024-09-11T03:42:42.955 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 23 to 126 2024-09-11T03:42:42.955 INFO:tasks.rados.rados.2.smithi175.stdout:1147: finishing write tid 5 to smithi17557870-46 2024-09-11T03:42:42.955 INFO:tasks.rados.rados.2.smithi175.stdout:1147: oid 46 updating version 0 to 171 2024-09-11T03:42:42.955 INFO:tasks.rados.rados.2.smithi175.stdout:1147: oid 46 version 171 is already newer than 169 2024-09-11T03:42:42.955 INFO:tasks.rados.rados.2.smithi175.stdout:1147: oid 46 version 171 is already newer than 168 2024-09-11T03:42:42.955 INFO:tasks.rados.rados.2.smithi175.stdout:1147: oid 46 version 171 is already newer than 170 2024-09-11T03:42:42.955 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 171 (ObjNum 405 snap 127 seq_num 405) dirty exists 2024-09-11T03:42:42.955 INFO:tasks.rados.rados.2.smithi175.stdout:1147: left oid 46 (ObjNum 405 snap 127 seq_num 405) 2024-09-11T03:42:42.956 INFO:tasks.rados.rados.2.smithi175.stdout:1148: finishing write tid 1 to smithi17557870-7 2024-09-11T03:42:42.956 INFO:tasks.rados.rados.2.smithi175.stdout:1148: finishing write tid 2 to smithi17557870-7 2024-09-11T03:42:42.956 INFO:tasks.rados.rados.2.smithi175.stdout:1148: finishing write tid 3 to smithi17557870-7 2024-09-11T03:42:42.956 INFO:tasks.rados.rados.2.smithi175.stdout:1148: finishing write tid 4 to smithi17557870-7 2024-09-11T03:42:42.956 INFO:tasks.rados.rados.2.smithi175.stdout:1148: oid 7 updating version 0 to 207 2024-09-11T03:42:42.956 INFO:tasks.rados.rados.2.smithi175.stdout:1148: oid 7 version 207 is already newer than 206 2024-09-11T03:42:42.956 INFO:tasks.rados.rados.2.smithi175.stdout:1148: oid 7 version 207 is already newer than 205 2024-09-11T03:42:42.956 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 207 (ObjNum 406 snap 127 seq_num 406) dirty exists 2024-09-11T03:42:42.957 INFO:tasks.rados.rados.2.smithi175.stdout:1148: left oid 7 (ObjNum 406 snap 127 seq_num 406) 2024-09-11T03:42:42.957 INFO:tasks.rados.rados.2.smithi175.stdout:1150: expect (ObjNum 399 snap 126 seq_num 399) 2024-09-11T03:42:43.011 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:43.034 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:43.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:43.035 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:43.087 INFO:tasks.rados.rados.2.smithi175.stdout:1151: finishing write tid 1 to smithi17557870-2 2024-09-11T03:42:43.087 INFO:tasks.rados.rados.2.smithi175.stdout:1151: finishing write tid 2 to smithi17557870-2 2024-09-11T03:42:43.087 INFO:tasks.rados.rados.2.smithi175.stdout:1151: finishing write tid 3 to smithi17557870-2 2024-09-11T03:42:43.087 INFO:tasks.rados.rados.2.smithi175.stdout:1151: finishing write tid 4 to smithi17557870-2 2024-09-11T03:42:43.087 INFO:tasks.rados.rados.2.smithi175.stdout:1151: oid 2 updating version 0 to 391 2024-09-11T03:42:43.087 INFO:tasks.rados.rados.2.smithi175.stdout:1151: oid 2 updating version 391 to 392 2024-09-11T03:42:43.087 INFO:tasks.rados.rados.2.smithi175.stdout:1151: oid 2 version 392 is already newer than 390 2024-09-11T03:42:43.087 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 392 (ObjNum 407 snap 127 seq_num 407) dirty exists 2024-09-11T03:42:43.087 INFO:tasks.rados.rados.2.smithi175.stdout:1151: left oid 2 (ObjNum 407 snap 127 seq_num 407) 2024-09-11T03:42:43.087 INFO:tasks.rados.rados.2.smithi175.stdout:1153: finishing write tid 1 to smithi17557870-50 2024-09-11T03:42:43.088 INFO:tasks.rados.rados.2.smithi175.stdout:1147: done (6 left) 2024-09-11T03:42:43.088 INFO:tasks.rados.rados.2.smithi175.stdout:1148: done (5 left) 2024-09-11T03:42:43.088 INFO:tasks.rados.rados.2.smithi175.stdout:1149: done (4 left) 2024-09-11T03:42:43.088 INFO:tasks.rados.rados.2.smithi175.stdout:1150: done (3 left) 2024-09-11T03:42:43.088 INFO:tasks.rados.rados.2.smithi175.stdout:1151: done (2 left) 2024-09-11T03:42:43.088 INFO:tasks.rados.rados.2.smithi175.stdout:1156: read oid 27 snap -1 2024-09-11T03:42:43.088 INFO:tasks.rados.rados.2.smithi175.stdout:1156: expect (ObjNum 380 snap 117 seq_num 380) 2024-09-11T03:42:43.088 INFO:tasks.rados.rados.2.smithi175.stdout:1157: write oid 42 current snap is 127 2024-09-11T03:42:43.088 INFO:tasks.rados.rados.2.smithi175.stdout:1153: finishing write tid 2 to smithi17557870-50 2024-09-11T03:42:43.088 INFO:tasks.rados.rados.2.smithi175.stdout:1153: finishing write tid 3 to smithi17557870-50 2024-09-11T03:42:43.088 INFO:tasks.rados.rados.2.smithi175.stdout:1157: seq_num 409 ranges {474063=543550,1650547=514133,2765651=108412} 2024-09-11T03:42:43.091 INFO:tasks.rados.rados.2.smithi175.stdout:1157: writing smithi17557870-42 from 474063 to 1017613 tid 1 2024-09-11T03:42:43.098 INFO:tasks.rados.rados.2.smithi175.stdout:1157: writing smithi17557870-42 from 1650547 to 2164680 tid 2 2024-09-11T03:42:43.103 INFO:tasks.rados.rados.2.smithi175.stdout:1157: writing smithi17557870-42 from 2765651 to 2874063 tid 3 2024-09-11T03:42:43.103 INFO:tasks.rados.rados.2.smithi175.stdout:1158: write (excl) oid 34 current snap is 127 2024-09-11T03:42:43.103 INFO:tasks.rados.rados.2.smithi175.stdout:1153: finishing write tid 4 to smithi17557870-50 2024-09-11T03:42:43.103 INFO:tasks.rados.rados.2.smithi175.stdout:1153: finishing write tid 5 to smithi17557870-50 2024-09-11T03:42:43.103 INFO:tasks.rados.rados.2.smithi175.stdout:1153: oid 50 updating version 0 to 435 2024-09-11T03:42:43.103 INFO:tasks.rados.rados.2.smithi175.stdout:1153: oid 50 version 435 is already newer than 433 2024-09-11T03:42:43.103 INFO:tasks.rados.rados.2.smithi175.stdout:1153: oid 50 version 435 is already newer than 434 2024-09-11T03:42:43.103 INFO:tasks.rados.rados.2.smithi175.stdout:1153: oid 50 updating version 435 to 436 2024-09-11T03:42:43.103 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 436 (ObjNum 408 snap 127 seq_num 408) dirty exists 2024-09-11T03:42:43.103 INFO:tasks.rados.rados.2.smithi175.stdout:1153: left oid 50 (ObjNum 408 snap 127 seq_num 408) 2024-09-11T03:42:43.103 INFO:tasks.rados.rados.2.smithi175.stdout:1155: finishing rollback tid 0 to smithi17557870-23 2024-09-11T03:42:43.103 INFO:tasks.rados.rados.2.smithi175.stdout:1158: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2024-09-11T03:42:43.108 INFO:tasks.rados.rados.2.smithi175.stdout:1158: writing smithi17557870-34 from 490870 to 1109669 tid 1 2024-09-11T03:42:43.115 INFO:tasks.rados.rados.2.smithi175.stdout:1158: writing smithi17557870-34 from 1592676 to 2366820 tid 2 2024-09-11T03:42:43.118 INFO:tasks.rados.rados.2.smithi175.stdout:1158: writing smithi17557870-34 from 2793074 to 2890870 tid 3 2024-09-11T03:42:43.118 INFO:tasks.rados.rados.2.smithi175.stdout:1153: done (4 left) 2024-09-11T03:42:43.118 INFO:tasks.rados.rados.2.smithi175.stdout:1159: read oid 36 snap -1 2024-09-11T03:42:43.118 INFO:tasks.rados.rados.2.smithi175.stdout:1159: expect (ObjNum 366 snap 111 seq_num 366) 2024-09-11T03:42:43.118 INFO:tasks.rados.rados.2.smithi175.stdout:1160: copy_from oid 13 from oid 14 current snap is 127 2024-09-11T03:42:43.118 INFO:tasks.rados.rados.2.smithi175.stdout:1155: finishing rollback tid 1 to smithi17557870-23 2024-09-11T03:42:43.119 INFO:tasks.rados.rados.2.smithi175.stdout:1155: finishing rollback tid 2 to smithi17557870-23 2024-09-11T03:42:43.119 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 224 (ObjNum 330 snap 92 seq_num 330) dirty exists 2024-09-11T03:42:43.119 INFO:tasks.rados.rados.2.smithi175.stdout:1156: expect (ObjNum 380 snap 117 seq_num 380) 2024-09-11T03:42:43.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:43.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:42 smithi084 ceph-mon[27254]: osdmap e345: 12 total, 12 up, 10 in 2024-09-11T03:42:43.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:42 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:42:43.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:42 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:42:43.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:42 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/977711366' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:42:43.154 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:42:43.154 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:42:43.154 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:42:43.154 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:42:43.154 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:42:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:43.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:43.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:43.191 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:42:43.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:42 smithi175 ceph-mon[30894]: osdmap e345: 12 total, 12 up, 10 in 2024-09-11T03:42:43.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:42 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:42:43.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:42 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:42:43.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:42 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/977711366' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:42:43.232 INFO:tasks.rados.rados.2.smithi175.stdout:1157: finishing write tid 1 to smithi17557870-42 2024-09-11T03:42:43.232 INFO:tasks.rados.rados.2.smithi175.stdout:1157: finishing write tid 2 to smithi17557870-42 2024-09-11T03:42:43.232 INFO:tasks.rados.rados.2.smithi175.stdout:1157: finishing write tid 3 to smithi17557870-42 2024-09-11T03:42:43.232 INFO:tasks.rados.rados.2.smithi175.stdout:1157: finishing write tid 4 to smithi17557870-42 2024-09-11T03:42:43.232 INFO:tasks.rados.rados.2.smithi175.stdout:1157: finishing write tid 5 to smithi17557870-42 2024-09-11T03:42:43.232 INFO:tasks.rados.rados.2.smithi175.stdout:1157: oid 42 updating version 0 to 329 2024-09-11T03:42:43.232 INFO:tasks.rados.rados.2.smithi175.stdout:1157: oid 42 version 329 is already newer than 322 2024-09-11T03:42:43.232 INFO:tasks.rados.rados.2.smithi175.stdout:1157: oid 42 version 329 is already newer than 328 2024-09-11T03:42:43.232 INFO:tasks.rados.rados.2.smithi175.stdout:1157: oid 42 version 329 is already newer than 327 2024-09-11T03:42:43.233 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 329 (ObjNum 409 snap 127 seq_num 409) dirty exists 2024-09-11T03:42:43.233 INFO:tasks.rados.rados.2.smithi175.stdout:1157: left oid 42 (ObjNum 409 snap 127 seq_num 409) 2024-09-11T03:42:43.233 INFO:tasks.rados.rados.2.smithi175.stdout:1158: finishing write tid 1 to smithi17557870-34 2024-09-11T03:42:43.233 INFO:tasks.rados.rados.2.smithi175.stdout:1158: finishing write tid 2 to smithi17557870-34 2024-09-11T03:42:43.233 INFO:tasks.rados.rados.2.smithi175.stdout:1158: finishing write tid 3 to smithi17557870-34 2024-09-11T03:42:43.233 INFO:tasks.rados.rados.2.smithi175.stdout:1155: done (5 left) 2024-09-11T03:42:43.233 INFO:tasks.rados.rados.2.smithi175.stdout:1156: done (4 left) 2024-09-11T03:42:43.233 INFO:tasks.rados.rados.2.smithi175.stdout:1157: done (3 left) 2024-09-11T03:42:43.233 INFO:tasks.rados.rados.2.smithi175.stdout:1161: copy_from oid 17 from oid 44 current snap is 127 2024-09-11T03:42:43.233 INFO:tasks.rados.rados.2.smithi175.stdout:1158: finishing write tid 4 to smithi17557870-34 2024-09-11T03:42:43.234 INFO:tasks.rados.rados.2.smithi175.stdout:1158: finishing write tid 5 to smithi17557870-34 2024-09-11T03:42:43.234 INFO:tasks.rados.rados.2.smithi175.stdout:1158: oid 34 updating version 0 to 186 2024-09-11T03:42:43.234 INFO:tasks.rados.rados.2.smithi175.stdout:1158: oid 34 updating version 186 to 187 2024-09-11T03:42:43.234 INFO:tasks.rados.rados.2.smithi175.stdout:1158: oid 34 updating version 187 to 188 2024-09-11T03:42:43.234 INFO:tasks.rados.rados.2.smithi175.stdout:1158: oid 34 updating version 188 to 189 2024-09-11T03:42:43.234 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 189 (ObjNum 410 snap 127 seq_num 410) dirty exists 2024-09-11T03:42:43.234 INFO:tasks.rados.rados.2.smithi175.stdout:1158: left oid 34 (ObjNum 410 snap 127 seq_num 410) 2024-09-11T03:42:43.234 INFO:tasks.rados.rados.2.smithi175.stdout:1159: expect (ObjNum 366 snap 111 seq_num 366) 2024-09-11T03:42:43.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:43.283 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:43.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:43.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:43.373 INFO:tasks.rados.rados.2.smithi175.stdout:1160: finishing copy_from racing read to smithi17557870-13 2024-09-11T03:42:43.373 INFO:tasks.rados.rados.2.smithi175.stdout:1160: finishing copy_from to smithi17557870-13 2024-09-11T03:42:43.374 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 419 (ObjNum 258 snap 64 seq_num 258) dirty exists 2024-09-11T03:42:43.374 INFO:tasks.rados.rados.2.smithi175.stdout:1161: finishing copy_from racing read to smithi17557870-17 2024-09-11T03:42:43.374 INFO:tasks.rados.rados.2.smithi175.stdout:1161: finishing copy_from to smithi17557870-17 2024-09-11T03:42:43.374 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 423 (ObjNum 384 snap 117 seq_num 384) dirty exists 2024-09-11T03:42:43.374 INFO:tasks.rados.rados.2.smithi175.stdout:1158: done (3 left) 2024-09-11T03:42:43.374 INFO:tasks.rados.rados.2.smithi175.stdout:1159: done (2 left) 2024-09-11T03:42:43.374 INFO:tasks.rados.rados.2.smithi175.stdout:1160: done (1 left) 2024-09-11T03:42:43.374 INFO:tasks.rados.rados.2.smithi175.stdout:1161: done (0 left) 2024-09-11T03:42:43.374 INFO:tasks.rados.rados.2.smithi175.stdout:1162: write oid 50 current snap is 127 2024-09-11T03:42:43.374 INFO:tasks.rados.rados.2.smithi175.stdout:1162: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2024-09-11T03:42:43.377 INFO:tasks.rados.rados.2.smithi175.stdout:1162: writing smithi17557870-50 from 507677 to 918078 tid 1 2024-09-11T03:42:43.383 INFO:tasks.rados.rados.2.smithi175.stdout:1162: writing smithi17557870-50 from 1367511 to 1885313 tid 2 2024-09-11T03:42:43.385 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:43.388 INFO:tasks.rados.rados.2.smithi175.stdout:1162: writing smithi17557870-50 from 2420497 to 2907677 tid 3 2024-09-11T03:42:43.389 INFO:tasks.rados.rados.2.smithi175.stdout:1163: read oid 13 snap -1 2024-09-11T03:42:43.389 INFO:tasks.rados.rados.2.smithi175.stdout:1163: expect (ObjNum 258 snap 64 seq_num 258) 2024-09-11T03:42:43.389 INFO:tasks.rados.rados.2.smithi175.stdout:1164: snap_remove snap 99 2024-09-11T03:42:43.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:43.408 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:43.408 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:43.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:42 smithi121 ceph-mon[30729]: osdmap e345: 12 total, 12 up, 10 in 2024-09-11T03:42:43.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:42 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:42:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:42 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:42:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:42 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/977711366' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:42:43.509 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:43.529 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:43.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:43.530 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:43.582 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:43.631 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:43.652 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:43.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:43.653 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:43.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:43.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:43.777 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:43.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:43.852 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:43.879 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:43.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:43.906 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:43.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:43.961 INFO:tasks.rados.rados.2.smithi175.stdout:1164: done (2 left) 2024-09-11T03:42:43.962 INFO:tasks.rados.rados.2.smithi175.stdout:1165: copy_from oid 43 from oid 39 current snap is 127 2024-09-11T03:42:43.962 INFO:tasks.rados.rados.2.smithi175.stdout:1162: finishing write tid 1 to smithi17557870-50 2024-09-11T03:42:43.962 INFO:tasks.rados.rados.2.smithi175.stdout:1162: finishing write tid 2 to smithi17557870-50 2024-09-11T03:42:43.962 INFO:tasks.rados.rados.2.smithi175.stdout:1162: finishing write tid 3 to smithi17557870-50 2024-09-11T03:42:43.962 INFO:tasks.rados.rados.2.smithi175.stdout:1162: finishing write tid 4 to smithi17557870-50 2024-09-11T03:42:43.962 INFO:tasks.rados.rados.2.smithi175.stdout:1162: finishing write tid 5 to smithi17557870-50 2024-09-11T03:42:43.962 INFO:tasks.rados.rados.2.smithi175.stdout:1162: oid 50 updating version 0 to 441 2024-09-11T03:42:43.962 INFO:tasks.rados.rados.2.smithi175.stdout:1162: oid 50 updating version 441 to 442 2024-09-11T03:42:43.962 INFO:tasks.rados.rados.2.smithi175.stdout:1162: oid 50 version 442 is already newer than 440 2024-09-11T03:42:43.962 INFO:tasks.rados.rados.2.smithi175.stdout:1162: oid 50 version 442 is already newer than 439 2024-09-11T03:42:43.962 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 442 (ObjNum 411 snap 127 seq_num 411) dirty exists 2024-09-11T03:42:43.962 INFO:tasks.rados.rados.2.smithi175.stdout:1162: left oid 50 (ObjNum 411 snap 127 seq_num 411) 2024-09-11T03:42:43.962 INFO:tasks.rados.rados.2.smithi175.stdout:1163: expect (ObjNum 258 snap 64 seq_num 258) 2024-09-11T03:42:44.008 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:44.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:44.033 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:44.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:44.110 INFO:tasks.rados.rados.2.smithi175.stdout:1165: finishing copy_from racing read to smithi17557870-43 2024-09-11T03:42:44.111 INFO:tasks.rados.rados.2.smithi175.stdout:1165: finishing copy_from to smithi17557870-43 2024-09-11T03:42:44.111 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 311 (ObjNum 388 snap 117 seq_num 388) dirty exists 2024-09-11T03:42:44.111 INFO:tasks.rados.rados.2.smithi175.stdout:1162: done (2 left) 2024-09-11T03:42:44.111 INFO:tasks.rados.rados.2.smithi175.stdout:1163: done (1 left) 2024-09-11T03:42:44.111 INFO:tasks.rados.rados.2.smithi175.stdout:1165: done (0 left) 2024-09-11T03:42:44.111 INFO:tasks.rados.rados.2.smithi175.stdout:1166: read oid 13 snap -1 2024-09-11T03:42:44.111 INFO:tasks.rados.rados.2.smithi175.stdout:1166: expect (ObjNum 258 snap 64 seq_num 258) 2024-09-11T03:42:44.111 INFO:tasks.rados.rados.2.smithi175.stdout:1167: read oid 23 snap -1 2024-09-11T03:42:44.111 INFO:tasks.rados.rados.2.smithi175.stdout:1167: expect (ObjNum 330 snap 92 seq_num 330) 2024-09-11T03:42:44.111 INFO:tasks.rados.rados.2.smithi175.stdout:1168: write oid 6 current snap is 127 2024-09-11T03:42:44.111 INFO:tasks.rados.rados.2.smithi175.stdout:1168: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2024-09-11T03:42:44.116 INFO:tasks.rados.rados.2.smithi175.stdout:1168: writing smithi17557870-6 from 524484 to 1010134 tid 1 2024-09-11T03:42:44.122 INFO:tasks.rados.rados.2.smithi175.stdout:1168: writing smithi17557870-6 from 1425993 to 2087453 tid 2 2024-09-11T03:42:44.127 INFO:tasks.rados.rados.2.smithi175.stdout:1168: writing smithi17557870-6 from 2847920 to 2924484 tid 3 2024-09-11T03:42:44.127 INFO:tasks.rados.rados.2.smithi175.stdout:1169: copy_from oid 8 from oid 39 current snap is 127 2024-09-11T03:42:44.127 INFO:tasks.rados.rados.2.smithi175.stdout:1166: expect (ObjNum 258 snap 64 seq_num 258) 2024-09-11T03:42:44.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:44.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:43 smithi084 ceph-mon[27254]: pgmap v673: 57 pgs: 57 active+clean; 236 MiB data, 822 MiB used, 893 GiB / 894 GiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2024-09-11T03:42:44.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:43 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/977711366' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:42:44.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:43 smithi084 ceph-mon[27254]: osdmap e346: 12 total, 12 up, 10 in 2024-09-11T03:42:44.157 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:44.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:44.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:43 smithi175 ceph-mon[30894]: pgmap v673: 57 pgs: 57 active+clean; 236 MiB data, 822 MiB used, 893 GiB / 894 GiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2024-09-11T03:42:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:43 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/977711366' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:42:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:43 smithi175 ceph-mon[30894]: osdmap e346: 12 total, 12 up, 10 in 2024-09-11T03:42:44.259 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:44.278 INFO:tasks.rados.rados.2.smithi175.stdout:1167: expect (ObjNum 330 snap 92 seq_num 330) 2024-09-11T03:42:44.280 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:44.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:44.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:44.385 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:44.396 INFO:tasks.rados.rados.2.smithi175.stdout:1168: finishing write tid 1 to smithi17557870-6 2024-09-11T03:42:44.396 INFO:tasks.rados.rados.2.smithi175.stdout:1168: finishing write tid 2 to smithi17557870-6 2024-09-11T03:42:44.396 INFO:tasks.rados.rados.2.smithi175.stdout:1166: done (3 left) 2024-09-11T03:42:44.396 INFO:tasks.rados.rados.2.smithi175.stdout:1167: done (2 left) 2024-09-11T03:42:44.396 INFO:tasks.rados.rados.2.smithi175.stdout:1170: snap_remove snap 106 2024-09-11T03:42:44.408 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:44.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:44.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:44.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:43 smithi121 ceph-mon[30729]: pgmap v673: 57 pgs: 57 active+clean; 236 MiB data, 822 MiB used, 893 GiB / 894 GiB avail; 2.5 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2024-09-11T03:42:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:43 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/977711366' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:42:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:43 smithi121 ceph-mon[30729]: osdmap e346: 12 total, 12 up, 10 in 2024-09-11T03:42:44.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:44.534 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:44.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:44.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:44.637 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:44.658 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:44.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:44.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:44.760 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:44.781 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:44.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:44.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:44.883 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:44.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:44.905 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:44.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:44.968 INFO:tasks.rados.rados.2.smithi175.stdout:1170: done (2 left) 2024-09-11T03:42:44.968 INFO:tasks.rados.rados.2.smithi175.stdout:1171: read oid 13 snap 120 2024-09-11T03:42:44.969 INFO:tasks.rados.rados.2.smithi175.stdout:1171: expect (ObjNum 360 snap 107 seq_num 360) 2024-09-11T03:42:44.969 INFO:tasks.rados.rados.2.smithi175.stdout:1172: write (excl) oid 41 current snap is 127 2024-09-11T03:42:44.969 INFO:tasks.rados.rados.2.smithi175.stdout:1172: seq_num 413 ranges {541291=560899,1884475=521471} 2024-09-11T03:42:44.974 INFO:tasks.rados.rados.2.smithi175.stdout:1172: writing smithi17557870-41 from 541291 to 1102190 tid 1 2024-09-11T03:42:44.981 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:42:44.982 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:42:44.984 INFO:tasks.rados.rados.2.smithi175.stdout:1172: writing smithi17557870-41 from 1884475 to 2405946 tid 2 2024-09-11T03:42:44.984 INFO:tasks.rados.rados.2.smithi175.stdout:1173: rollback oid 26 current snap is 127 2024-09-11T03:42:44.984 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 26 to 126 2024-09-11T03:42:44.984 INFO:tasks.rados.rados.2.smithi175.stdout:1168: finishing write tid 3 to smithi17557870-6 2024-09-11T03:42:44.984 INFO:tasks.rados.rados.2.smithi175.stdout:1168: finishing write tid 4 to smithi17557870-6 2024-09-11T03:42:44.985 INFO:tasks.rados.rados.2.smithi175.stdout:1168: finishing write tid 5 to smithi17557870-6 2024-09-11T03:42:44.985 INFO:tasks.rados.rados.2.smithi175.stdout:1168: oid 6 updating version 0 to 369 2024-09-11T03:42:44.985 INFO:tasks.rados.rados.2.smithi175.stdout:1168: oid 6 version 369 is already newer than 368 2024-09-11T03:42:44.985 INFO:tasks.rados.rados.2.smithi175.stdout:1168: oid 6 version 369 is already newer than 366 2024-09-11T03:42:44.985 INFO:tasks.rados.rados.2.smithi175.stdout:1168: oid 6 version 369 is already newer than 367 2024-09-11T03:42:44.985 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 369 (ObjNum 412 snap 127 seq_num 412) dirty exists 2024-09-11T03:42:44.985 INFO:tasks.rados.rados.2.smithi175.stdout:1168: left oid 6 (ObjNum 412 snap 127 seq_num 412) 2024-09-11T03:42:44.985 INFO:tasks.rados.rados.2.smithi175.stdout:1169: finishing copy_from racing read to smithi17557870-8 2024-09-11T03:42:44.985 INFO:tasks.rados.rados.2.smithi175.stdout:1169: finishing copy_from to smithi17557870-8 2024-09-11T03:42:44.985 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 311 (ObjNum 388 snap 117 seq_num 388) dirty exists 2024-09-11T03:42:44.985 INFO:tasks.rados.rados.2.smithi175.stdout:1168: done (4 left) 2024-09-11T03:42:44.985 INFO:tasks.rados.rados.2.smithi175.stdout:1169: done (3 left) 2024-09-11T03:42:44.985 INFO:tasks.rados.rados.2.smithi175.stdout:1174: copy_from oid 20 from oid 12 current snap is 127 2024-09-11T03:42:44.986 INFO:tasks.rados.rados.2.smithi175.stdout:1171: expect (ObjNum 360 snap 107 seq_num 360) 2024-09-11T03:42:45.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:45.011 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:42:45.011 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:42:45.028 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:45.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:45.030 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:42:45.030 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:42:45.030 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:44 smithi084 ceph-mon[27254]: osdmap e347: 12 total, 12 up, 10 in 2024-09-11T03:42:45.045 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:42:45.046 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:42:45.061 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:42:45.062 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:42:45.072 INFO:tasks.rados.rados.2.smithi175.stdout:1172: finishing write tid 1 to smithi17557870-41 2024-09-11T03:42:45.073 INFO:tasks.rados.rados.2.smithi175.stdout:1172: finishing write tid 2 to smithi17557870-41 2024-09-11T03:42:45.073 INFO:tasks.rados.rados.2.smithi175.stdout:1172: finishing write tid 3 to smithi17557870-41 2024-09-11T03:42:45.073 INFO:tasks.rados.rados.2.smithi175.stdout:1172: finishing write tid 4 to smithi17557870-41 2024-09-11T03:42:45.073 INFO:tasks.rados.rados.2.smithi175.stdout:1172: oid 41 updating version 0 to 336 2024-09-11T03:42:45.073 INFO:tasks.rados.rados.2.smithi175.stdout:1172: oid 41 updating version 336 to 338 2024-09-11T03:42:45.073 INFO:tasks.rados.rados.2.smithi175.stdout:1172: oid 41 version 338 is already newer than 337 2024-09-11T03:42:45.073 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 338 (ObjNum 413 snap 127 seq_num 413) dirty exists 2024-09-11T03:42:45.073 INFO:tasks.rados.rados.2.smithi175.stdout:1172: left oid 41 (ObjNum 413 snap 127 seq_num 413) 2024-09-11T03:42:45.073 INFO:tasks.rados.rados.2.smithi175.stdout:1173: finishing rollback tid 0 to smithi17557870-26 2024-09-11T03:42:45.073 INFO:tasks.rados.rados.2.smithi175.stdout:1173: finishing rollback tid 1 to smithi17557870-26 2024-09-11T03:42:45.073 INFO:tasks.rados.rados.2.smithi175.stdout:1173: finishing rollback tid 2 to smithi17557870-26 2024-09-11T03:42:45.074 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 199 (ObjNum 380 snap 117 seq_num 380) dirty exists 2024-09-11T03:42:45.074 INFO:tasks.rados.rados.2.smithi175.stdout:1174: finishing copy_from racing read to smithi17557870-20 2024-09-11T03:42:45.074 INFO:tasks.rados.rados.2.smithi175.stdout:1174: finishing copy_from to smithi17557870-20 2024-09-11T03:42:45.074 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 292 (ObjNum 335 snap 95 seq_num 335) dirty exists 2024-09-11T03:42:45.074 INFO:tasks.rados.rados.2.smithi175.stdout:1171: done (3 left) 2024-09-11T03:42:45.074 INFO:tasks.rados.rados.2.smithi175.stdout:1172: done (2 left) 2024-09-11T03:42:45.074 INFO:tasks.rados.rados.2.smithi175.stdout:1173: done (1 left) 2024-09-11T03:42:45.074 INFO:tasks.rados.rados.2.smithi175.stdout:1174: done (0 left) 2024-09-11T03:42:45.074 INFO:tasks.rados.rados.2.smithi175.stdout:1175: copy_from oid 45 from oid 46 current snap is 127 2024-09-11T03:42:45.074 INFO:tasks.rados.rados.2.smithi175.stdout:1176: snap_remove snap 117 2024-09-11T03:42:45.078 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:42:45.078 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:42:45.097 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:42:45.098 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:42:45.114 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:42:45.114 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:42:45.130 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:45.131 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:42:45.132 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:42:45.147 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:42:45.147 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:42:45.152 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:45.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:45.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:45.163 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:42:45.163 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:42:45.180 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:42:45.181 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:42:45.197 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:42:45.197 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:42:45.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:44 smithi175 ceph-mon[30894]: osdmap e347: 12 total, 12 up, 10 in 2024-09-11T03:42:45.254 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:45.275 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:45.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:45.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:45.377 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:45.398 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:45.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:45.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:44 smithi121 ceph-mon[30729]: osdmap e347: 12 total, 12 up, 10 in 2024-09-11T03:42:45.500 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:45.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:45.523 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:45.524 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:45.625 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:45.647 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:45.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:45.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:45.749 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:45.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:45.771 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:45.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:45.872 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:45.898 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:45.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:45.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:45.976 INFO:tasks.rados.rados.2.smithi175.stdout:1176: done (1 left) 2024-09-11T03:42:45.976 INFO:tasks.rados.rados.2.smithi175.stdout:1177: copy_from oid 35 from oid 50 current snap is 127 2024-09-11T03:42:45.976 INFO:tasks.rados.rados.2.smithi175.stdout:1175: finishing copy_from racing read to smithi17557870-45 2024-09-11T03:42:45.976 INFO:tasks.rados.rados.2.smithi175.stdout:1175: finishing copy_from to smithi17557870-45 2024-09-11T03:42:45.977 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 171 (ObjNum 405 snap 127 seq_num 405) dirty exists 2024-09-11T03:42:45.977 INFO:tasks.rados.rados.2.smithi175.stdout:1175: done (1 left) 2024-09-11T03:42:45.977 INFO:tasks.rados.rados.2.smithi175.stdout:1178: read oid 23 snap -1 2024-09-11T03:42:45.977 INFO:tasks.rados.rados.2.smithi175.stdout:1178: expect (ObjNum 330 snap 92 seq_num 330) 2024-09-11T03:42:45.977 INFO:tasks.rados.rados.2.smithi175.stdout:1179: write (excl) oid 46 current snap is 127 2024-09-11T03:42:45.977 INFO:tasks.rados.rados.2.smithi175.stdout:1179: seq_num 414 ranges {558098=636148,1826604=665129} 2024-09-11T03:42:45.984 INFO:tasks.rados.rados.2.smithi175.stdout:1179: writing smithi17557870-46 from 558098 to 1194246 tid 1 2024-09-11T03:42:45.991 INFO:tasks.rados.rados.2.smithi175.stdout:1179: writing smithi17557870-46 from 1826604 to 2491733 tid 2 2024-09-11T03:42:45.991 INFO:tasks.rados.rados.2.smithi175.stdout:1180: write (excl) oid 8 current snap is 127 2024-09-11T03:42:45.992 INFO:tasks.rados.rados.2.smithi175.stdout:1180: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2024-09-11T03:42:45.998 INFO:tasks.rados.rados.2.smithi175.stdout:1180: writing smithi17557870-8 from 574905 to 1286302 tid 1 2024-09-11T03:42:46.000 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:46.004 INFO:tasks.rados.rados.2.smithi175.stdout:1180: writing smithi17557870-8 from 1885086 to 2410226 tid 2 2024-09-11T03:42:46.008 INFO:tasks.rados.rados.2.smithi175.stdout:1180: writing smithi17557870-8 from 2930189 to 2974905 tid 3 2024-09-11T03:42:46.008 INFO:tasks.rados.rados.2.smithi175.stdout:1181: copy_from oid 7 from oid 9 current snap is 127 2024-09-11T03:42:46.008 INFO:tasks.rados.rados.2.smithi175.stdout:1178: expect (ObjNum 330 snap 92 seq_num 330) 2024-09-11T03:42:46.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:46.025 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:46.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:46.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:45 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:42:46.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:45 smithi084 ceph-mon[27254]: 2.c scrub ok 2024-09-11T03:42:46.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:45 smithi084 ceph-mon[27254]: pgmap v676: 57 pgs: 57 active+clean; 239 MiB data, 835 MiB used, 893 GiB / 894 GiB avail; 7.9 MiB/s rd, 2.8 MiB/s wr, 20 op/s 2024-09-11T03:42:46.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:45 smithi084 ceph-mon[27254]: osdmap e348: 12 total, 12 up, 10 in 2024-09-11T03:42:46.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:45 smithi084 ceph-mon[27254]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:42:46.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:45 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1610252817' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:46.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:45 smithi084 ceph-mon[27254]: 2.18 scrub starts 2024-09-11T03:42:46.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:45 smithi084 ceph-mon[27254]: 2.18 scrub ok 2024-09-11T03:42:46.125 INFO:tasks.rados.rados.2.smithi175.stdout:1179: finishing write tid 1 to smithi17557870-46 2024-09-11T03:42:46.125 INFO:tasks.rados.rados.2.smithi175.stdout:1177: finishing copy_from racing read to smithi17557870-35 2024-09-11T03:42:46.125 INFO:tasks.rados.rados.2.smithi175.stdout:1179: finishing write tid 2 to smithi17557870-46 2024-09-11T03:42:46.125 INFO:tasks.rados.rados.2.smithi175.stdout:1179: finishing write tid 3 to smithi17557870-46 2024-09-11T03:42:46.125 INFO:tasks.rados.rados.2.smithi175.stdout:1179: finishing write tid 4 to smithi17557870-46 2024-09-11T03:42:46.125 INFO:tasks.rados.rados.2.smithi175.stdout:1179: oid 46 updating version 0 to 172 2024-09-11T03:42:46.125 INFO:tasks.rados.rados.2.smithi175.stdout:1179: oid 46 updating version 172 to 173 2024-09-11T03:42:46.125 INFO:tasks.rados.rados.2.smithi175.stdout:1179: oid 46 updating version 173 to 174 2024-09-11T03:42:46.125 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 174 (ObjNum 414 snap 127 seq_num 414) dirty exists 2024-09-11T03:42:46.125 INFO:tasks.rados.rados.2.smithi175.stdout:1179: left oid 46 (ObjNum 414 snap 127 seq_num 414) 2024-09-11T03:42:46.125 INFO:tasks.rados.rados.2.smithi175.stdout:1177: finishing copy_from to smithi17557870-35 2024-09-11T03:42:46.125 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 442 (ObjNum 411 snap 127 seq_num 411) dirty exists 2024-09-11T03:42:46.126 INFO:tasks.rados.rados.2.smithi175.stdout:1180: finishing write tid 1 to smithi17557870-8 2024-09-11T03:42:46.126 INFO:tasks.rados.rados.2.smithi175.stdout:1180: finishing write tid 2 to smithi17557870-8 2024-09-11T03:42:46.126 INFO:tasks.rados.rados.2.smithi175.stdout:1180: finishing write tid 3 to smithi17557870-8 2024-09-11T03:42:46.126 INFO:tasks.rados.rados.2.smithi175.stdout:1177: done (4 left) 2024-09-11T03:42:46.126 INFO:tasks.rados.rados.2.smithi175.stdout:1178: done (3 left) 2024-09-11T03:42:46.126 INFO:tasks.rados.rados.2.smithi175.stdout:1179: done (2 left) 2024-09-11T03:42:46.126 INFO:tasks.rados.rados.2.smithi175.stdout:1182: read oid 30 snap 119 2024-09-11T03:42:46.126 INFO:tasks.rados.rados.2.smithi175.stdout:1182: expect deleted 2024-09-11T03:42:46.126 INFO:tasks.rados.rados.2.smithi175.stdout:1183: write (excl) oid 45 current snap is 127 2024-09-11T03:42:46.126 INFO:tasks.rados.rados.2.smithi175.stdout:1180: finishing write tid 4 to smithi17557870-8 2024-09-11T03:42:46.126 INFO:tasks.rados.rados.2.smithi175.stdout:1183: seq_num 416 ranges {591712=786646,1943568=668798} 2024-09-11T03:42:46.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:46.132 INFO:tasks.rados.rados.2.smithi175.stdout:1183: writing smithi17557870-45 from 591712 to 1378358 tid 1 2024-09-11T03:42:46.139 INFO:tasks.rados.rados.2.smithi175.stdout:1183: writing smithi17557870-45 from 1943568 to 2612366 tid 2 2024-09-11T03:42:46.139 INFO:tasks.rados.rados.2.smithi175.stdout:1184: write (excl) oid 50 current snap is 127 2024-09-11T03:42:46.139 INFO:tasks.rados.rados.2.smithi175.stdout:1180: finishing write tid 5 to smithi17557870-8 2024-09-11T03:42:46.139 INFO:tasks.rados.rados.2.smithi175.stdout:1180: oid 8 updating version 0 to 465 2024-09-11T03:42:46.139 INFO:tasks.rados.rados.2.smithi175.stdout:1180: oid 8 updating version 465 to 467 2024-09-11T03:42:46.139 INFO:tasks.rados.rados.2.smithi175.stdout:1180: oid 8 updating version 467 to 468 2024-09-11T03:42:46.140 INFO:tasks.rados.rados.2.smithi175.stdout:1180: oid 8 version 468 is already newer than 466 2024-09-11T03:42:46.140 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 468 (ObjNum 415 snap 127 seq_num 415) dirty exists 2024-09-11T03:42:46.140 INFO:tasks.rados.rados.2.smithi175.stdout:1180: left oid 8 (ObjNum 415 snap 127 seq_num 415) 2024-09-11T03:42:46.140 INFO:tasks.rados.rados.2.smithi175.stdout:1181: finishing copy_from racing read to smithi17557870-7 2024-09-11T03:42:46.140 INFO:tasks.rados.rados.2.smithi175.stdout:1181: finishing copy_from to smithi17557870-7 2024-09-11T03:42:46.140 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 223 (ObjNum 277 snap 73 seq_num 277) dirty exists 2024-09-11T03:42:46.140 INFO:tasks.rados.rados.2.smithi175.stdout:1184: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2024-09-11T03:42:46.144 INFO:tasks.rados.rados.2.smithi175.stdout:1184: writing smithi17557870-50 from 608519 to 1070414 tid 1 2024-09-11T03:42:46.149 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T03:42:46.150 INFO:tasks.rados.rados.2.smithi175.stdout:1184: writing smithi17557870-50 from 1602050 to 2130859 tid 2 2024-09-11T03:42:46.151 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:46.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:46.156 INFO:tasks.rados.rados.2.smithi175.stdout:1184: writing smithi17557870-50 from 2585035 to 3008519 tid 3 2024-09-11T03:42:46.156 INFO:tasks.rados.rados.2.smithi175.stdout:1180: done (4 left) 2024-09-11T03:42:46.156 INFO:tasks.rados.rados.2.smithi175.stdout:1181: done (3 left) 2024-09-11T03:42:46.156 INFO:tasks.rados.rados.2.smithi175.stdout:1185: read oid 29 snap -1 2024-09-11T03:42:46.156 INFO:tasks.rados.rados.2.smithi175.stdout:1185: expect deleted 2024-09-11T03:42:46.156 INFO:tasks.rados.rados.2.smithi175.stdout:1186: read oid 11 snap -1 2024-09-11T03:42:46.156 INFO:tasks.rados.rados.2.smithi175.stdout:1186: expect (ObjNum 371 snap 113 seq_num 371) 2024-09-11T03:42:46.156 INFO:tasks.rados.rados.2.smithi175.stdout:1187: copy_from oid 43 from oid 6 current snap is 127 2024-09-11T03:42:46.156 INFO:tasks.rados.rados.2.smithi175.stdout:1183: finishing write tid 1 to smithi17557870-45 2024-09-11T03:42:46.156 INFO:tasks.rados.rados.2.smithi175.stdout:1182: done (5 left) 2024-09-11T03:42:46.156 INFO:tasks.rados.rados.2.smithi175.stdout:1188: copy_from oid 36 from oid 9 current snap is 127 2024-09-11T03:42:46.157 INFO:tasks.rados.rados.2.smithi175.stdout:1183: finishing write tid 2 to smithi17557870-45 2024-09-11T03:42:46.157 INFO:tasks.rados.rados.2.smithi175.stdout:1183: finishing write tid 3 to smithi17557870-45 2024-09-11T03:42:46.157 INFO:tasks.rados.rados.2.smithi175.stdout:1183: finishing write tid 4 to smithi17557870-45 2024-09-11T03:42:46.157 INFO:tasks.rados.rados.2.smithi175.stdout:1183: oid 45 updating version 0 to 444 2024-09-11T03:42:46.157 INFO:tasks.rados.rados.2.smithi175.stdout:1183: oid 45 version 444 is already newer than 443 2024-09-11T03:42:46.157 INFO:tasks.rados.rados.2.smithi175.stdout:1183: oid 45 updating version 444 to 445 2024-09-11T03:42:46.157 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 445 (ObjNum 416 snap 127 seq_num 416) dirty exists 2024-09-11T03:42:46.157 INFO:tasks.rados.rados.2.smithi175.stdout:1183: left oid 45 (ObjNum 416 snap 127 seq_num 416) 2024-09-11T03:42:46.157 INFO:tasks.rados.rados.2.smithi175.stdout:1184: finishing write tid 1 to smithi17557870-50 2024-09-11T03:42:46.157 INFO:tasks.rados.rados.2.smithi175.stdout:1184: finishing write tid 2 to smithi17557870-50 2024-09-11T03:42:46.157 INFO:tasks.rados.rados.2.smithi175.stdout:1183: done (5 left) 2024-09-11T03:42:46.157 INFO:tasks.rados.rados.2.smithi175.stdout:1189: snap_remove snap 126 2024-09-11T03:42:46.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:45 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:42:46.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:45 smithi175 ceph-mon[30894]: 2.c scrub ok 2024-09-11T03:42:46.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:45 smithi175 ceph-mon[30894]: pgmap v676: 57 pgs: 57 active+clean; 239 MiB data, 835 MiB used, 893 GiB / 894 GiB avail; 7.9 MiB/s rd, 2.8 MiB/s wr, 20 op/s 2024-09-11T03:42:46.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:45 smithi175 ceph-mon[30894]: osdmap e348: 12 total, 12 up, 10 in 2024-09-11T03:42:46.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:45 smithi175 ceph-mon[30894]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:42:46.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:45 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1610252817' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:46.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:45 smithi175 ceph-mon[30894]: 2.18 scrub starts 2024-09-11T03:42:46.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:45 smithi175 ceph-mon[30894]: 2.18 scrub ok 2024-09-11T03:42:46.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:46.276 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:46.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:46.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:46.368 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:42:46.378 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:46.400 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:46.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:46.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:46.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:45 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:42:46.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:45 smithi121 ceph-mon[30729]: 2.c scrub ok 2024-09-11T03:42:46.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:45 smithi121 ceph-mon[30729]: pgmap v676: 57 pgs: 57 active+clean; 239 MiB data, 835 MiB used, 893 GiB / 894 GiB avail; 7.9 MiB/s rd, 2.8 MiB/s wr, 20 op/s 2024-09-11T03:42:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:45 smithi121 ceph-mon[30729]: osdmap e348: 12 total, 12 up, 10 in 2024-09-11T03:42:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:45 smithi121 ceph-mon[30729]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:42:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:45 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1610252817' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:45 smithi121 ceph-mon[30729]: 2.18 scrub starts 2024-09-11T03:42:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:45 smithi121 ceph-mon[30729]: 2.18 scrub ok 2024-09-11T03:42:46.502 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:46.522 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:46.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:46.624 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:46.648 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:46.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:46.666 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":677,"stamp":"2024-09-11T03:42:44.965476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":250377438,"num_objects":121,"num_object_clones":72,"num_object_copies":363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121,"num_whiteouts":4,"num_read":1091,"num_read_kb":942138,"num_write":854,"num_write_kb":346109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":29825344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9067,"ondisk_log_size":9067,"up":171,"acting":171,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":171,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":855408,"kb_used_data":497040,"kb_used_omap":15,"kb_used_meta":357424,"kb_avail":936596112,"statfs":{"total":959950356480,"available":959074418688,"internally_reserved":0,"allocated":508968960,"data_stored":670485955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":366002613},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11124376,"num_objects":10,"num_object_clones":10,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":43,"num_read_kb":32169,"num_write":40,"num_write_kb":11448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.028051"},"pg_stats":[{"pgid":"1.1e","version":"20'92","reported_seq":261,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.951627+0000","last_change":"2024-09-11T03:40:40.236609+0000","last_active":"2024-09-11T03:42:40.951627+0000","last_peered":"2024-09-11T03:42:40.951627+0000","last_clean":"2024-09-11T03:42:40.951627+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:42:40.951627+0000","last_undegraded":"2024-09-11T03:42:40.951627+0000","last_fullsized":"2024-09-11T03:42:40.951627+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.236546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.236546+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:09.146128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'92","reported_seq":262,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.938295+0000","last_change":"2024-09-11T03:40:41.348889+0000","last_active":"2024-09-11T03:42:40.938295+0000","last_peered":"2024-09-11T03:42:40.938295+0000","last_clean":"2024-09-11T03:42:40.938295+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:42:40.938295+0000","last_undegraded":"2024-09-11T03:42:40.938295+0000","last_fullsized":"2024-09-11T03:42:40.938295+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:41.348828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:41.348828+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:11.089042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'92","reported_seq":262,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.938816+0000","last_change":"2024-09-11T03:40:40.188174+0000","last_active":"2024-09-11T03:42:40.938816+0000","last_peered":"2024-09-11T03:42:40.938816+0000","last_clean":"2024-09-11T03:42:40.938816+0000","last_became_active":"2024-09-11T03:40:39.848230+0000","last_became_peered":"2024-09-11T03:40:39.848230+0000","last_unstale":"2024-09-11T03:42:40.938816+0000","last_undegraded":"2024-09-11T03:42:40.938816+0000","last_fullsized":"2024-09-11T03:42:40.938816+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.188124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.188124+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:08.246084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"346'224","reported_seq":1557,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-11T03:42:43.122014+0000","last_change":"2024-09-11T03:42:42.961621+0000","last_active":"2024-09-11T03:42:43.122014+0000","last_peered":"2024-09-11T03:42:43.122014+0000","last_clean":"2024-09-11T03:42:43.122014+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:42:43.122014+0000","last_undegraded":"2024-09-11T03:42:43.122014+0000","last_fullsized":"2024-09-11T03:42:43.122014+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"280'196","last_scrub_stamp":"2024-09-11T03:41:40.336904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.336904+0000","objects_scrubbed":2,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:59.281324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039932049999999997,"stat_sum":{"num_bytes":11119314,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":51076,"num_write":7,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4865740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"78","length":"1"},{"start":"7c","length":"4"}]},{"pgid":"1.1b","version":"20'92","reported_seq":262,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.938607+0000","last_change":"2024-09-11T03:40:42.166354+0000","last_active":"2024-09-11T03:42:40.938607+0000","last_peered":"2024-09-11T03:42:40.938607+0000","last_clean":"2024-09-11T03:42:40.938607+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:42:40.938607+0000","last_undegraded":"2024-09-11T03:42:40.938607+0000","last_fullsized":"2024-09-11T03:42:40.938607+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:42.166295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:42.166295+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:13.230054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.19","version":"340'363","reported_seq":1696,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-11T03:42:42.952812+0000","last_change":"2024-09-11T03:42:42.953266+0000","last_active":"2024-09-11T03:42:42.952812+0000","last_peered":"2024-09-11T03:42:42.952812+0000","last_clean":"2024-09-11T03:42:42.952812+0000","last_became_active":"2024-09-11T03:41:38.886787+0000","last_became_peered":"2024-09-11T03:41:38.886787+0000","last_unstale":"2024-09-11T03:42:42.952812+0000","last_undegraded":"2024-09-11T03:42:42.952812+0000","last_fullsized":"2024-09-11T03:42:42.952812+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'333","last_scrub_stamp":"2024-09-11T03:41:40.379141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.379141+0000","objects_scrubbed":1,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:46.724049+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00070879799999999996,"stat_sum":{"num_bytes":6354812,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":2892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"78","length":"1"},{"start":"7c","length":"4"}]},{"pgid":"1.1a","version":"20'92","reported_seq":502,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-11T03:42:39.950532+0000","last_change":"2024-09-11T03:40:40.278752+0000","last_active":"2024-09-11T03:42:39.950532+0000","last_peered":"2024-09-11T03:42:39.950532+0000","last_clean":"2024-09-11T03:42:39.950532+0000","last_became_active":"2024-09-11T03:40:39.848781+0000","last_became_peered":"2024-09-11T03:40:39.848781+0000","last_unstale":"2024-09-11T03:42:39.950532+0000","last_undegraded":"2024-09-11T03:42:39.950532+0000","last_fullsized":"2024-09-11T03:42:39.950532+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.278703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.278703+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:09.913902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"20'92","reported_seq":262,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.953781+0000","last_change":"2024-09-11T03:40:40.718553+0000","last_active":"2024-09-11T03:42:40.953781+0000","last_peered":"2024-09-11T03:42:40.953781+0000","last_clean":"2024-09-11T03:42:40.953781+0000","last_became_active":"2024-09-11T03:40:39.851698+0000","last_became_peered":"2024-09-11T03:40:39.851698+0000","last_unstale":"2024-09-11T03:42:40.953781+0000","last_undegraded":"2024-09-11T03:42:40.953781+0000","last_fullsized":"2024-09-11T03:42:40.953781+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.718503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.718503+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:13.733914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.18","version":"20'92","reported_seq":384,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.998357+0000","last_change":"2024-09-11T03:42:02.103343+0000","last_active":"2024-09-11T03:42:40.998357+0000","last_peered":"2024-09-11T03:42:40.998357+0000","last_clean":"2024-09-11T03:42:40.998357+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:42:40.998357+0000","last_undegraded":"2024-09-11T03:42:40.998357+0000","last_fullsized":"2024-09-11T03:42:40.998357+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:02.103292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:02.103292+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:29.586019+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"340'182","reported_seq":1465,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-11T03:42:42.960095+0000","last_change":"2024-09-11T03:42:42.960473+0000","last_active":"2024-09-11T03:42:42.960095+0000","last_peered":"2024-09-11T03:42:42.960095+0000","last_clean":"2024-09-11T03:42:42.960095+0000","last_became_active":"2024-09-11T03:41:38.890690+0000","last_became_peered":"2024-09-11T03:41:38.890690+0000","last_unstale":"2024-09-11T03:42:42.960095+0000","last_undegraded":"2024-09-11T03:42:42.960095+0000","last_fullsized":"2024-09-11T03:42:42.960095+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'162","last_scrub_stamp":"2024-09-11T03:41:42.270480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:42.270480+0000","objects_scrubbed":1,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:50.821976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027433819999999999,"stat_sum":{"num_bytes":4546386,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":1191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[9,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"78","length":"1"},{"start":"7c","length":"4"}]},{"pgid":"1.17","version":"20'92","reported_seq":270,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.938880+0000","last_change":"2024-09-11T03:41:57.260720+0000","last_active":"2024-09-11T03:42:40.938880+0000","last_peered":"2024-09-11T03:42:40.938880+0000","last_clean":"2024-09-11T03:42:40.938880+0000","last_became_active":"2024-09-11T03:40:39.850243+0000","last_became_peered":"2024-09-11T03:40:39.850243+0000","last_unstale":"2024-09-11T03:42:40.938880+0000","last_undegraded":"2024-09-11T03:42:40.938880+0000","last_fullsized":"2024-09-11T03:42:40.938880+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:57.260664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:57.260664+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:11.818969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":1575,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-11T03:42:42.961038+0000","last_change":"2024-09-11T03:42:42.961426+0000","last_active":"2024-09-11T03:42:42.961038+0000","last_peered":"2024-09-11T03:42:42.961038+0000","last_clean":"2024-09-11T03:42:42.961038+0000","last_became_active":"2024-09-11T03:41:38.888228+0000","last_became_peered":"2024-09-11T03:41:38.888228+0000","last_unstale":"2024-09-11T03:42:42.961038+0000","last_undegraded":"2024-09-11T03:42:42.961038+0000","last_fullsized":"2024-09-11T03:42:42.961038+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:41:43.239726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:43.239726+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:07.542841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035842550000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"78","length":"1"},{"start":"7c","length":"4"}]},{"pgid":"1.16","version":"20'92","reported_seq":268,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.938406+0000","last_change":"2024-09-11T03:42:05.102506+0000","last_active":"2024-09-11T03:42:40.938406+0000","last_peered":"2024-09-11T03:42:40.938406+0000","last_clean":"2024-09-11T03:42:40.938406+0000","last_became_active":"2024-09-11T03:40:39.849950+0000","last_became_peered":"2024-09-11T03:40:39.849950+0000","last_unstale":"2024-09-11T03:42:40.938406+0000","last_undegraded":"2024-09-11T03:42:40.938406+0000","last_fullsized":"2024-09-11T03:42:40.938406+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:05.102457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:05.102457+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:18.344056+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"344'207","reported_seq":1449,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:41.008946+0000","last_change":"2024-09-11T03:42:41.001454+0000","last_active":"2024-09-11T03:42:41.008946+0000","last_peered":"2024-09-11T03:42:41.008946+0000","last_clean":"2024-09-11T03:42:41.008946+0000","last_became_active":"2024-09-11T03:41:38.887856+0000","last_became_peered":"2024-09-11T03:41:38.887856+0000","last_unstale":"2024-09-11T03:42:41.008946+0000","last_undegraded":"2024-09-11T03:42:41.008946+0000","last_fullsized":"2024-09-11T03:42:41.008946+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:41:42.059696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:42.059696+0000","objects_scrubbed":1,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:03.078560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002782545,"stat_sum":{"num_bytes":9692309,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":19,"num_write_kb":4275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,9],"acting":[5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"78","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.15","version":"20'92","reported_seq":261,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.969331+0000","last_change":"2024-09-11T03:40:39.928143+0000","last_active":"2024-09-11T03:42:40.969331+0000","last_peered":"2024-09-11T03:42:40.969331+0000","last_clean":"2024-09-11T03:42:40.969331+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:42:40.969331+0000","last_undegraded":"2024-09-11T03:42:40.969331+0000","last_fullsized":"2024-09-11T03:42:40.969331+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.928088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.928088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:08.756769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":1649,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-11T03:42:41.944415+0000","last_change":"2024-09-11T03:42:41.944674+0000","last_active":"2024-09-11T03:42:41.944415+0000","last_peered":"2024-09-11T03:42:41.944415+0000","last_clean":"2024-09-11T03:42:41.944415+0000","last_became_active":"2024-09-11T03:41:38.885486+0000","last_became_peered":"2024-09-11T03:41:38.885486+0000","last_unstale":"2024-09-11T03:42:41.944415+0000","last_undegraded":"2024-09-11T03:42:41.944415+0000","last_fullsized":"2024-09-11T03:42:41.944415+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:41:41.456545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.456545+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:44.173915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046750300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"78","length":"1"},{"start":"7c","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.14","version":"20'92","reported_seq":294,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-11T03:42:41.944035+0000","last_change":"2024-09-11T03:40:35.195938+0000","last_active":"2024-09-11T03:42:41.944035+0000","last_peered":"2024-09-11T03:42:41.944035+0000","last_clean":"2024-09-11T03:42:41.944035+0000","last_became_active":"2024-09-11T03:40:27.847297+0000","last_became_peered":"2024-09-11T03:40:27.847297+0000","last_unstale":"2024-09-11T03:42:41.944035+0000","last_undegraded":"2024-09-11T03:42:41.944035+0000","last_fullsized":"2024-09-11T03:42:41.944035+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.195893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.195893+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:50.659810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":352,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.939129+0000","last_change":"2024-09-11T03:42:40.939580+0000","last_active":"2024-09-11T03:42:40.939129+0000","last_peered":"2024-09-11T03:42:40.939129+0000","last_clean":"2024-09-11T03:42:40.939129+0000","last_became_active":"2024-09-11T03:41:35.874295+0000","last_became_peered":"2024-09-11T03:41:35.874295+0000","last_unstale":"2024-09-11T03:42:40.939129+0000","last_undegraded":"2024-09-11T03:42:40.939129+0000","last_fullsized":"2024-09-11T03:42:40.939129+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:41:40.182948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.182948+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:03.027066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088612200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"78","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.13","version":"20'92","reported_seq":377,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.938238+0000","last_change":"2024-09-11T03:41:45.086774+0000","last_active":"2024-09-11T03:42:40.938238+0000","last_peered":"2024-09-11T03:42:40.938238+0000","last_clean":"2024-09-11T03:42:40.938238+0000","last_became_active":"2024-09-11T03:40:27.847557+0000","last_became_peered":"2024-09-11T03:40:27.847557+0000","last_unstale":"2024-09-11T03:42:40.938238+0000","last_undegraded":"2024-09-11T03:42:40.938238+0000","last_fullsized":"2024-09-11T03:42:40.938238+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:45.086692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:45.086692+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:48.005883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":1338,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-11T03:42:39.950669+0000","last_change":"2024-09-11T03:42:38.905930+0000","last_active":"2024-09-11T03:42:39.950669+0000","last_peered":"2024-09-11T03:42:39.950669+0000","last_clean":"2024-09-11T03:42:39.950669+0000","last_became_active":"2024-09-11T03:41:38.886813+0000","last_became_peered":"2024-09-11T03:41:38.886813+0000","last_unstale":"2024-09-11T03:42:39.950669+0000","last_undegraded":"2024-09-11T03:42:39.950669+0000","last_fullsized":"2024-09-11T03:42:39.950669+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:41:40.077940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.077940+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:03.922401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00060032599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,9],"acting":[4,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"78","length":"1"},{"start":"7f","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":289,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.998425+0000","last_change":"2024-09-11T03:40:37.284871+0000","last_active":"2024-09-11T03:42:40.998425+0000","last_peered":"2024-09-11T03:42:40.998425+0000","last_clean":"2024-09-11T03:42:40.998425+0000","last_became_active":"2024-09-11T03:40:27.845249+0000","last_became_peered":"2024-09-11T03:40:27.845249+0000","last_unstale":"2024-09-11T03:42:40.998425+0000","last_undegraded":"2024-09-11T03:42:40.998425+0000","last_fullsized":"2024-09-11T03:42:40.998425+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:37.284800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:37.284800+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:55.054741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"346'191","reported_seq":1468,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-11T03:42:42.955442+0000","last_change":"2024-09-11T03:42:42.955858+0000","last_active":"2024-09-11T03:42:42.955442+0000","last_peered":"2024-09-11T03:42:42.955442+0000","last_clean":"2024-09-11T03:42:42.955442+0000","last_became_active":"2024-09-11T03:42:09.873469+0000","last_became_peered":"2024-09-11T03:42:09.873469+0000","last_unstale":"2024-09-11T03:42:42.955442+0000","last_undegraded":"2024-09-11T03:42:42.955442+0000","last_fullsized":"2024-09-11T03:42:42.955442+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"283'124","last_scrub_stamp":"2024-09-11T03:41:41.419995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.419995+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:59.199171+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0031727560000000001,"stat_sum":{"num_bytes":17526328,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":22771,"num_write":33,"num_write_kb":12679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":5991453,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"78","length":"1"},{"start":"7c","length":"4"}]},{"pgid":"1.11","version":"20'92","reported_seq":294,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.953892+0000","last_change":"2024-09-11T03:40:35.703292+0000","last_active":"2024-09-11T03:42:40.953892+0000","last_peered":"2024-09-11T03:42:40.953892+0000","last_clean":"2024-09-11T03:42:40.953892+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:42:40.953892+0000","last_undegraded":"2024-09-11T03:42:40.953892+0000","last_fullsized":"2024-09-11T03:42:40.953892+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.703203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.703203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:49.677961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":1362,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-11T03:42:42.960560+0000","last_change":"2024-09-11T03:42:42.960966+0000","last_active":"2024-09-11T03:42:42.960560+0000","last_peered":"2024-09-11T03:42:42.960560+0000","last_clean":"2024-09-11T03:42:42.960560+0000","last_became_active":"2024-09-11T03:41:38.886834+0000","last_became_peered":"2024-09-11T03:41:38.886834+0000","last_unstale":"2024-09-11T03:42:42.960560+0000","last_undegraded":"2024-09-11T03:42:42.960560+0000","last_fullsized":"2024-09-11T03:42:42.960560+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:41:41.300112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.300112+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:56.241429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00317454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"78","length":"1"},{"start":"7c","length":"4"}]},{"pgid":"1.10","version":"20'92","reported_seq":297,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.998373+0000","last_change":"2024-09-11T03:42:03.151379+0000","last_active":"2024-09-11T03:42:40.998373+0000","last_peered":"2024-09-11T03:42:40.998373+0000","last_clean":"2024-09-11T03:42:40.998373+0000","last_became_active":"2024-09-11T03:40:27.844057+0000","last_became_peered":"2024-09-11T03:40:27.844057+0000","last_unstale":"2024-09-11T03:42:40.998373+0000","last_undegraded":"2024-09-11T03:42:40.998373+0000","last_fullsized":"2024-09-11T03:42:40.998373+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:03.151323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:03.151323+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:05.151390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"344'323","reported_seq":1565,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.977343+0000","last_change":"2024-09-11T03:42:40.940056+0000","last_active":"2024-09-11T03:42:40.977343+0000","last_peered":"2024-09-11T03:42:40.977343+0000","last_clean":"2024-09-11T03:42:40.977343+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:42:40.977343+0000","last_undegraded":"2024-09-11T03:42:40.977343+0000","last_fullsized":"2024-09-11T03:42:40.977343+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'313","last_scrub_stamp":"2024-09-11T03:42:30.424582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:42:30.424582+0000","objects_scrubbed":5,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:35.964949+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00087020200000000004,"stat_sum":{"num_bytes":11148744,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":81,"num_read_kb":38401,"num_write":112,"num_write_kb":55419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"78","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.8","version":"20'92","reported_seq":374,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.998484+0000","last_change":"2024-09-11T03:42:05.135033+0000","last_active":"2024-09-11T03:42:40.998484+0000","last_peered":"2024-09-11T03:42:40.998484+0000","last_clean":"2024-09-11T03:42:40.998484+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:42:40.998484+0000","last_undegraded":"2024-09-11T03:42:40.998484+0000","last_fullsized":"2024-09-11T03:42:40.998484+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:05.134977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:05.134977+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:05.203169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"344'116","reported_seq":1370,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.991677+0000","last_change":"2024-09-11T03:42:40.951998+0000","last_active":"2024-09-11T03:42:40.991677+0000","last_peered":"2024-09-11T03:42:40.991677+0000","last_clean":"2024-09-11T03:42:40.991677+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:42:40.991677+0000","last_undegraded":"2024-09-11T03:42:40.991677+0000","last_fullsized":"2024-09-11T03:42:40.991677+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'101","last_scrub_stamp":"2024-09-11T03:42:22.098703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:42:22.098703+0000","objects_scrubbed":2,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:39.567906+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00041293599999999998,"stat_sum":{"num_bytes":7206899,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":84,"num_read_kb":84436,"num_write":42,"num_write_kb":17935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"78","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.9","version":"20'92","reported_seq":699,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.968982+0000","last_change":"2024-09-11T03:42:00.816905+0000","last_active":"2024-09-11T03:42:40.968982+0000","last_peered":"2024-09-11T03:42:40.968982+0000","last_clean":"2024-09-11T03:42:40.968982+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:42:40.968982+0000","last_undegraded":"2024-09-11T03:42:40.968982+0000","last_fullsized":"2024-09-11T03:42:40.968982+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:00.816850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:00.816850+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:09.071150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"332'431","reported_seq":1770,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.938496+0000","last_change":"2024-09-11T03:42:40.938882+0000","last_active":"2024-09-11T03:42:40.938496+0000","last_peered":"2024-09-11T03:42:40.938496+0000","last_clean":"2024-09-11T03:42:40.938496+0000","last_became_active":"2024-09-11T03:41:33.865638+0000","last_became_peered":"2024-09-11T03:41:33.865638+0000","last_unstale":"2024-09-11T03:42:40.938496+0000","last_undegraded":"2024-09-11T03:42:40.938496+0000","last_fullsized":"2024-09-11T03:42:40.938496+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'337","last_scrub_stamp":"2024-09-11T03:41:40.102984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.102984+0000","objects_scrubbed":5,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:43.175002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00063801000000000005,"stat_sum":{"num_bytes":26402142,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":94,"num_read_kb":112714,"num_write":30,"num_write_kb":10475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"78","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.a","version":"20'92","reported_seq":616,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-11T03:42:42.957447+0000","last_change":"2024-09-11T03:41:45.227630+0000","last_active":"2024-09-11T03:42:42.957447+0000","last_peered":"2024-09-11T03:42:42.957447+0000","last_clean":"2024-09-11T03:42:42.957447+0000","last_became_active":"2024-09-11T03:40:37.849988+0000","last_became_peered":"2024-09-11T03:40:37.849988+0000","last_unstale":"2024-09-11T03:42:42.957447+0000","last_undegraded":"2024-09-11T03:42:42.957447+0000","last_fullsized":"2024-09-11T03:42:42.957447+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:45.227581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:45.227581+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:03.066353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"332'228","reported_seq":1495,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.969589+0000","last_change":"2024-09-11T03:42:40.970023+0000","last_active":"2024-09-11T03:42:40.969589+0000","last_peered":"2024-09-11T03:42:40.969589+0000","last_clean":"2024-09-11T03:42:40.969589+0000","last_became_active":"2024-09-11T03:41:33.863875+0000","last_became_peered":"2024-09-11T03:41:33.863875+0000","last_unstale":"2024-09-11T03:42:40.969589+0000","last_undegraded":"2024-09-11T03:42:40.969589+0000","last_fullsized":"2024-09-11T03:42:40.969589+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'192","last_scrub_stamp":"2024-09-11T03:41:40.877628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.877628+0000","objects_scrubbed":1,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:01.490061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076181399999999998,"stat_sum":{"num_bytes":3633300,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8,"num_write_kb":1104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"78","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.b","version":"20'92","reported_seq":305,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.951610+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:42:40.951610+0000","last_peered":"2024-09-11T03:42:40.951610+0000","last_clean":"2024-09-11T03:42:40.951610+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:42:40.951610+0000","last_undegraded":"2024-09-11T03:42:40.951610+0000","last_fullsized":"2024-09-11T03:42:40.951610+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:54.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"345'396","reported_seq":1740,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-11T03:42:41.946940+0000","last_change":"2024-09-11T03:42:41.947307+0000","last_active":"2024-09-11T03:42:41.946940+0000","last_peered":"2024-09-11T03:42:41.946940+0000","last_clean":"2024-09-11T03:42:41.946940+0000","last_became_active":"2024-09-11T03:41:33.867674+0000","last_became_peered":"2024-09-11T03:41:33.867674+0000","last_unstale":"2024-09-11T03:42:41.946940+0000","last_undegraded":"2024-09-11T03:42:41.946940+0000","last_fullsized":"2024-09-11T03:42:41.946940+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'331","last_scrub_stamp":"2024-09-11T03:41:42.500501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:42.500501+0000","objects_scrubbed":9,"log_size":396,"log_dups_size":0,"ondisk_log_size":396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:54.043814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0030256369999999999,"stat_sum":{"num_bytes":25191521,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":46821,"num_write":15,"num_write_kb":8717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"78","length":"1"},{"start":"7c","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.4","version":"20'92","reported_seq":471,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.951697+0000","last_change":"2024-09-11T03:42:00.181197+0000","last_active":"2024-09-11T03:42:40.951697+0000","last_peered":"2024-09-11T03:42:40.951697+0000","last_clean":"2024-09-11T03:42:40.951697+0000","last_became_active":"2024-09-11T03:40:25.858281+0000","last_became_peered":"2024-09-11T03:40:25.858281+0000","last_unstale":"2024-09-11T03:42:40.951697+0000","last_undegraded":"2024-09-11T03:42:40.951697+0000","last_fullsized":"2024-09-11T03:42:40.951697+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:00.181139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:00.181139+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:00.294383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":1415,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.938647+0000","last_change":"2024-09-11T03:42:40.938968+0000","last_active":"2024-09-11T03:42:40.938647+0000","last_peered":"2024-09-11T03:42:40.938647+0000","last_clean":"2024-09-11T03:42:40.938647+0000","last_became_active":"2024-09-11T03:41:33.865499+0000","last_became_peered":"2024-09-11T03:41:33.865499+0000","last_unstale":"2024-09-11T03:42:40.938647+0000","last_undegraded":"2024-09-11T03:42:40.938647+0000","last_fullsized":"2024-09-11T03:42:40.938647+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:41:42.104989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:42.104989+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:51.209971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047210200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"78","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.5","version":"20'92","reported_seq":373,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-11T03:42:41.944037+0000","last_change":"2024-09-11T03:40:38.224278+0000","last_active":"2024-09-11T03:42:41.944037+0000","last_peered":"2024-09-11T03:42:41.944037+0000","last_clean":"2024-09-11T03:42:41.944037+0000","last_became_active":"2024-09-11T03:40:37.851080+0000","last_became_peered":"2024-09-11T03:40:37.851080+0000","last_unstale":"2024-09-11T03:42:41.944037+0000","last_undegraded":"2024-09-11T03:42:41.944037+0000","last_fullsized":"2024-09-11T03:42:41.944037+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.224231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.224231+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:09.894123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"340'399","reported_seq":1713,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.939833+0000","last_change":"2024-09-11T03:42:40.939833+0000","last_active":"2024-09-11T03:42:40.939833+0000","last_peered":"2024-09-11T03:42:40.939833+0000","last_clean":"2024-09-11T03:42:40.939833+0000","last_became_active":"2024-09-11T03:41:33.866316+0000","last_became_peered":"2024-09-11T03:41:33.866316+0000","last_unstale":"2024-09-11T03:42:40.939833+0000","last_undegraded":"2024-09-11T03:42:40.939833+0000","last_fullsized":"2024-09-11T03:42:40.939833+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'292","last_scrub_stamp":"2024-09-11T03:41:41.222192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.222192+0000","objects_scrubbed":9,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:58.468929+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00077383599999999997,"stat_sum":{"num_bytes":24631699,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":65810,"num_write":21,"num_write_kb":9484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"78","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.6","version":"20'92","reported_seq":619,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-11T03:42:42.958056+0000","last_change":"2024-09-11T03:42:05.222525+0000","last_active":"2024-09-11T03:42:42.958056+0000","last_peered":"2024-09-11T03:42:42.958056+0000","last_clean":"2024-09-11T03:42:42.958056+0000","last_became_active":"2024-09-11T03:40:37.848794+0000","last_became_peered":"2024-09-11T03:40:37.848794+0000","last_unstale":"2024-09-11T03:42:42.958056+0000","last_undegraded":"2024-09-11T03:42:42.958056+0000","last_fullsized":"2024-09-11T03:42:42.958056+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:05.222462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:05.222462+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:33.609855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"332'184","reported_seq":1484,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.939290+0000","last_change":"2024-09-11T03:42:40.939290+0000","last_active":"2024-09-11T03:42:40.939290+0000","last_peered":"2024-09-11T03:42:40.939290+0000","last_clean":"2024-09-11T03:42:40.939290+0000","last_became_active":"2024-09-11T03:41:33.869119+0000","last_became_peered":"2024-09-11T03:41:33.869119+0000","last_unstale":"2024-09-11T03:42:40.939290+0000","last_undegraded":"2024-09-11T03:42:40.939290+0000","last_fullsized":"2024-09-11T03:42:40.939290+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'162","last_scrub_stamp":"2024-09-11T03:41:41.141743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.141743+0000","objects_scrubbed":2,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:53.708937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075984499999999999,"stat_sum":{"num_bytes":8638930,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":33838,"num_write":10,"num_write_kb":1948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"78","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.7","version":"20'92","reported_seq":110,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-11T03:42:42.952650+0000","last_change":"2024-09-11T03:41:58.387920+0000","last_active":"2024-09-11T03:42:42.952650+0000","last_peered":"2024-09-11T03:42:42.952650+0000","last_clean":"2024-09-11T03:42:42.952650+0000","last_became_active":"2024-09-11T03:41:57.865927+0000","last_became_peered":"2024-09-11T03:41:57.865927+0000","last_unstale":"2024-09-11T03:42:42.952650+0000","last_undegraded":"2024-09-11T03:42:42.952650+0000","last_fullsized":"2024-09-11T03:42:42.952650+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:58.387873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:58.387873+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:03.728844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"334'139","reported_seq":1449,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.999025+0000","last_change":"2024-09-11T03:42:40.999025+0000","last_active":"2024-09-11T03:42:40.999025+0000","last_peered":"2024-09-11T03:42:40.999025+0000","last_clean":"2024-09-11T03:42:40.999025+0000","last_became_active":"2024-09-11T03:42:30.902838+0000","last_became_peered":"2024-09-11T03:42:30.902838+0000","last_unstale":"2024-09-11T03:42:40.999025+0000","last_undegraded":"2024-09-11T03:42:40.999025+0000","last_fullsized":"2024-09-11T03:42:40.999025+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'123","last_scrub_stamp":"2024-09-11T03:41:40.064448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.064448+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:50.557326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00048300300000000002,"stat_sum":{"num_bytes":2571471,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":7972,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,11],"acting":[5,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"78","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.1","version":"20'92","reported_seq":605,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-11T03:42:42.957604+0000","last_change":"2024-09-11T03:42:01.225777+0000","last_active":"2024-09-11T03:42:42.957604+0000","last_peered":"2024-09-11T03:42:42.957604+0000","last_clean":"2024-09-11T03:42:42.957604+0000","last_became_active":"2024-09-11T03:40:25.854269+0000","last_became_peered":"2024-09-11T03:40:25.854269+0000","last_unstale":"2024-09-11T03:42:42.957604+0000","last_undegraded":"2024-09-11T03:42:42.957604+0000","last_fullsized":"2024-09-11T03:42:42.957604+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:01.225716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:01.225716+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:14.417097+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"338'134","reported_seq":1414,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-11T03:42:41.944663+0000","last_change":"2024-09-11T03:42:41.944663+0000","last_active":"2024-09-11T03:42:41.944663+0000","last_peered":"2024-09-11T03:42:41.944663+0000","last_clean":"2024-09-11T03:42:41.944663+0000","last_became_active":"2024-09-11T03:41:33.866217+0000","last_became_peered":"2024-09-11T03:41:33.866217+0000","last_unstale":"2024-09-11T03:42:41.944663+0000","last_undegraded":"2024-09-11T03:42:41.944663+0000","last_fullsized":"2024-09-11T03:42:41.944663+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:41:40.497820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:40.497820+0000","objects_scrubbed":1,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:03.080795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000536371,"stat_sum":{"num_bytes":3731187,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":1777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"78","length":"1"},{"start":"7c","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.2","version":"20'92","reported_seq":80,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.969027+0000","last_change":"2024-09-11T03:42:06.870952+0000","last_active":"2024-09-11T03:42:40.969027+0000","last_peered":"2024-09-11T03:42:40.969027+0000","last_clean":"2024-09-11T03:42:40.969027+0000","last_became_active":"2024-09-11T03:42:06.870500+0000","last_became_peered":"2024-09-11T03:42:06.870500+0000","last_unstale":"2024-09-11T03:42:40.969027+0000","last_undegraded":"2024-09-11T03:42:40.969027+0000","last_fullsized":"2024-09-11T03:42:40.969027+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:02.760253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:02.760253+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:17.778058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"332'88","reported_seq":1367,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.969655+0000","last_change":"2024-09-11T03:42:40.970136+0000","last_active":"2024-09-11T03:42:40.969655+0000","last_peered":"2024-09-11T03:42:40.969655+0000","last_clean":"2024-09-11T03:42:40.969655+0000","last_became_active":"2024-09-11T03:41:33.864202+0000","last_became_peered":"2024-09-11T03:41:33.864202+0000","last_unstale":"2024-09-11T03:42:40.969655+0000","last_undegraded":"2024-09-11T03:42:40.969655+0000","last_fullsized":"2024-09-11T03:42:40.969655+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:41:41.843233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.843233+0000","objects_scrubbed":2,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:50.214775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00070189299999999998,"stat_sum":{"num_bytes":3781608,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"78","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.3","version":"20'92","reported_seq":375,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.938261+0000","last_change":"2024-09-11T03:41:57.122830+0000","last_active":"2024-09-11T03:42:40.938261+0000","last_peered":"2024-09-11T03:42:40.938261+0000","last_clean":"2024-09-11T03:42:40.938261+0000","last_became_active":"2024-09-11T03:40:25.856407+0000","last_became_peered":"2024-09-11T03:40:25.856407+0000","last_unstale":"2024-09-11T03:42:40.938261+0000","last_undegraded":"2024-09-11T03:42:40.938261+0000","last_fullsized":"2024-09-11T03:42:40.938261+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:57.122773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:57.122773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:57.896149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"340'161","reported_seq":1429,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.999273+0000","last_change":"2024-09-11T03:42:40.999273+0000","last_active":"2024-09-11T03:42:40.999273+0000","last_peered":"2024-09-11T03:42:40.999273+0000","last_clean":"2024-09-11T03:42:40.999273+0000","last_became_active":"2024-09-11T03:41:33.862271+0000","last_became_peered":"2024-09-11T03:41:33.862271+0000","last_unstale":"2024-09-11T03:42:40.999273+0000","last_undegraded":"2024-09-11T03:42:40.999273+0000","last_fullsized":"2024-09-11T03:42:40.999273+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'117","last_scrub_stamp":"2024-09-11T03:41:41.089786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:41.089786+0000","objects_scrubbed":3,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:46.481653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00060206599999999995,"stat_sum":{"num_bytes":9911751,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":19,"num_read_kb":12238,"num_write":11,"num_write_kb":4396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"78","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.0","version":"20'92","reported_seq":696,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.969577+0000","last_change":"2024-09-11T03:41:45.957854+0000","last_active":"2024-09-11T03:42:40.969577+0000","last_peered":"2024-09-11T03:42:40.969577+0000","last_clean":"2024-09-11T03:42:40.969577+0000","last_became_active":"2024-09-11T03:40:25.854624+0000","last_became_peered":"2024-09-11T03:40:25.854624+0000","last_unstale":"2024-09-11T03:42:40.969577+0000","last_undegraded":"2024-09-11T03:42:40.969577+0000","last_fullsized":"2024-09-11T03:42:40.969577+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:45.957781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:45.957781+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:01.135709+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"20'92","reported_seq":291,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.998673+0000","last_change":"2024-09-11T03:40:35.282811+0000","last_active":"2024-09-11T03:42:40.998673+0000","last_peered":"2024-09-11T03:42:40.998673+0000","last_clean":"2024-09-11T03:42:40.998673+0000","last_became_active":"2024-09-11T03:40:27.844262+0000","last_became_peered":"2024-09-11T03:40:27.844262+0000","last_unstale":"2024-09-11T03:42:40.998673+0000","last_undegraded":"2024-09-11T03:42:40.998673+0000","last_fullsized":"2024-09-11T03:42:40.998673+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.282762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.282762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:54.069194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"344'316","reported_seq":1573,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.960980+0000","last_change":"2024-09-11T03:42:40.956242+0000","last_active":"2024-09-11T03:42:40.960980+0000","last_peered":"2024-09-11T03:42:40.960980+0000","last_clean":"2024-09-11T03:42:40.960980+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:42:40.960980+0000","last_undegraded":"2024-09-11T03:42:40.960980+0000","last_fullsized":"2024-09-11T03:42:40.960980+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'172","last_scrub_stamp":"2024-09-11T03:41:04.106166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:04.106166+0000","objects_scrubbed":5,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0045009560000000004,"stat_sum":{"num_bytes":23234272,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":159,"num_read_kb":120398,"num_write":148,"num_write_kb":61889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"78","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.e","version":"20'92","reported_seq":546,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-09-11T03:42:39.950634+0000","last_change":"2024-09-11T03:40:38.199696+0000","last_active":"2024-09-11T03:42:39.950634+0000","last_peered":"2024-09-11T03:42:39.950634+0000","last_clean":"2024-09-11T03:42:39.950634+0000","last_became_active":"2024-09-11T03:40:37.850623+0000","last_became_peered":"2024-09-11T03:40:37.850623+0000","last_unstale":"2024-09-11T03:42:39.950634+0000","last_undegraded":"2024-09-11T03:42:39.950634+0000","last_fullsized":"2024-09-11T03:42:39.950634+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.199636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.199636+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:48.923701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"334'438","reported_seq":1675,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.998886+0000","last_change":"2024-09-11T03:42:40.999079+0000","last_active":"2024-09-11T03:42:40.998886+0000","last_peered":"2024-09-11T03:42:40.998886+0000","last_clean":"2024-09-11T03:42:40.998886+0000","last_became_active":"2024-09-11T03:42:30.902959+0000","last_became_peered":"2024-09-11T03:42:30.902959+0000","last_unstale":"2024-09-11T03:42:40.998886+0000","last_undegraded":"2024-09-11T03:42:40.998886+0000","last_fullsized":"2024-09-11T03:42:40.998886+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'426","last_scrub_stamp":"2024-09-11T03:42:25.281790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:42:25.281790+0000","objects_scrubbed":10,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:32.162253+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00042552200000000001,"stat_sum":{"num_bytes":21047789,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":174,"num_read_kb":136324,"num_write":148,"num_write_kb":59200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17460823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"78","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.d","version":"20'92","reported_seq":305,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.969306+0000","last_change":"2024-09-11T03:40:38.901246+0000","last_active":"2024-09-11T03:42:40.969306+0000","last_peered":"2024-09-11T03:42:40.969306+0000","last_clean":"2024-09-11T03:42:40.969306+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:42:40.969306+0000","last_undegraded":"2024-09-11T03:42:40.969306+0000","last_fullsized":"2024-09-11T03:42:40.969306+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.901175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.901175+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:50.952770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"340'444","reported_seq":1691,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-11T03:42:41.944790+0000","last_change":"2024-09-11T03:42:41.945000+0000","last_active":"2024-09-11T03:42:41.944790+0000","last_peered":"2024-09-11T03:42:41.944790+0000","last_clean":"2024-09-11T03:42:41.944790+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:42:41.944790+0000","last_undegraded":"2024-09-11T03:42:41.944790+0000","last_fullsized":"2024-09-11T03:42:41.944790+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'435","last_scrub_stamp":"2024-09-11T03:42:28.893309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:42:28.893309+0000","objects_scrubbed":8,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:45.883926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00061368900000000005,"stat_sum":{"num_bytes":17456914,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":164,"num_read_kb":127277,"num_write":143,"num_write_kb":54325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"78","length":"1"},{"start":"7c","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"2.f","version":"344'210","reported_seq":1475,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-09-11T03:42:40.941324+0000","last_change":"2024-09-11T03:42:40.941324+0000","last_active":"2024-09-11T03:42:40.941324+0000","last_peered":"2024-09-11T03:42:40.941324+0000","last_clean":"2024-09-11T03:42:40.941324+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:42:40.941324+0000","last_undegraded":"2024-09-11T03:42:40.941324+0000","last_fullsized":"2024-09-11T03:42:40.941324+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'136","last_scrub_stamp":"2024-09-11T03:41:20.167983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:41:20.167983+0000","objects_scrubbed":2,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:50.086404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002227475,"stat_sum":{"num_bytes":11959694,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":107,"num_read_kb":82054,"num_write":80,"num_write_kb":32272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"78","length":"1"},{"start":"7e","length":"2"}]},{"pgid":"1.c","version":"20'92","reported_seq":319,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-11T03:42:42.952568+0000","last_change":"2024-09-11T03:40:38.353992+0000","last_active":"2024-09-11T03:42:42.952568+0000","last_peered":"2024-09-11T03:42:42.952568+0000","last_clean":"2024-09-11T03:42:42.952568+0000","last_became_active":"2024-09-11T03:40:37.847449+0000","last_became_peered":"2024-09-11T03:40:37.847449+0000","last_unstale":"2024-09-11T03:42:42.952568+0000","last_undegraded":"2024-09-11T03:42:42.952568+0000","last_fullsized":"2024-09-11T03:42:42.952568+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.353895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.353895+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:42:48.326329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":249787070,"num_objects":119,"num_object_clones":72,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":4,"num_read":1091,"num_read_kb":942138,"num_write":854,"num_write_kb":346109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":29825344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":476839936,"data_stored":655888013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6215,"ondisk_log_size":6215,"up":78,"acting":78,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285634,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70452,"kb_used_data":39264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93674700,"statfs":{"total":95995035648,"available":95922892800,"internally_reserved":0,"allocated":40206336,"data_stored":59462516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":326417514566,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743500,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80556,"kb_used_data":45204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93664596,"statfs":{"total":95995035648,"available":95912546304,"internally_reserved":0,"allocated":46288896,"data_stored":72052573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037843,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72116,"kb_used_data":36768,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93673036,"statfs":{"total":95995035648,"available":95921188864,"internally_reserved":0,"allocated":37650432,"data_stored":54662052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299478,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97656,"kb_used_data":58140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93647496,"statfs":{"total":95995035648,"available":95895035904,"internally_reserved":0,"allocated":59535360,"data_stored":76222079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":193273528411,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757345,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72200,"kb_used_data":36848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93672952,"statfs":{"total":95995035648,"available":95921102848,"internally_reserved":0,"allocated":37732352,"data_stored":52314740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986278,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95588,"kb_used_data":60236,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93649564,"statfs":{"total":95995035648,"available":95897153536,"internally_reserved":0,"allocated":61681664,"data_stored":78423452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":124644,"kb_used_data":84440,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93620508,"statfs":{"total":95995035648,"available":95867400192,"internally_reserved":0,"allocated":86466560,"data_stored":105479282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411440,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79504,"kb_used_data":44152,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93665648,"statfs":{"total":95995035648,"available":95913623552,"internally_reserved":0,"allocated":45211648,"data_stored":53217518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74936,"kb_used_data":39588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93670216,"statfs":{"total":95995035648,"available":95918301184,"internally_reserved":0,"allocated":40538112,"data_stored":45941397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738491,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87756,"kb_used_data":52400,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93657396,"statfs":{"total":95995035648,"available":95905173504,"internally_reserved":0,"allocated":53657600,"data_stored":72710346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50675712,"data_stored":72038379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36450304,"data_stored":43407177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40906752,"data_stored":50473717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82026496,"data_stored":102600732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":58748928,"data_stored":77666962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33439744,"data_stored":49579272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4079616,"data_stored":4066399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":56594432,"data_stored":75458222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":33562624,"data_stored":52127832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":41848832,"data_stored":69174023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2387968,"data_stored":2367002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":36118528,"data_stored":56928296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:42:46.669 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-11T03:42:46.670 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pg_num 51 2024-09-11T03:42:46.695 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:46.730 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:46.751 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:46.775 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:46.776 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:46.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:46.893 INFO:tasks.rados.rados.2.smithi175.stdout:1189: done (5 left) 2024-09-11T03:42:46.893 INFO:tasks.rados.rados.2.smithi175.stdout:1190: read oid 10 snap -1 2024-09-11T03:42:46.893 INFO:tasks.rados.rados.2.smithi175.stdout:1190: expect (ObjNum 310 snap 85 seq_num 310) 2024-09-11T03:42:46.893 INFO:tasks.rados.rados.2.smithi175.stdout:1191: delete oid 19 current snap is 127 2024-09-11T03:42:46.893 INFO:tasks.rados.rados.2.smithi175.stdout:1184: finishing write tid 3 to smithi17557870-50 2024-09-11T03:42:46.893 INFO:tasks.rados.rados.2.smithi175.stdout:1184: finishing write tid 4 to smithi17557870-50 2024-09-11T03:42:46.893 INFO:tasks.rados.rados.2.smithi175.stdout:1184: finishing write tid 5 to smithi17557870-50 2024-09-11T03:42:46.893 INFO:tasks.rados.rados.2.smithi175.stdout:1184: oid 50 updating version 0 to 457 2024-09-11T03:42:46.894 INFO:tasks.rados.rados.2.smithi175.stdout:1184: oid 50 updating version 457 to 460 2024-09-11T03:42:46.894 INFO:tasks.rados.rados.2.smithi175.stdout:1184: oid 50 version 460 is already newer than 458 2024-09-11T03:42:46.894 INFO:tasks.rados.rados.2.smithi175.stdout:1184: oid 50 version 460 is already newer than 459 2024-09-11T03:42:46.894 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 460 (ObjNum 417 snap 127 seq_num 417) dirty exists 2024-09-11T03:42:46.894 INFO:tasks.rados.rados.2.smithi175.stdout:1184: left oid 50 (ObjNum 417 snap 127 seq_num 417) 2024-09-11T03:42:46.894 INFO:tasks.rados.rados.2.smithi175.stdout:1188: finishing copy_from racing read to smithi17557870-36 2024-09-11T03:42:46.894 INFO:tasks.rados.rados.2.smithi175.stdout:1187: finishing copy_from racing read to smithi17557870-43 2024-09-11T03:42:46.894 INFO:tasks.rados.rados.2.smithi175.stdout:1186: expect (ObjNum 371 snap 113 seq_num 371) 2024-09-11T03:42:46.903 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:46.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:46.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:46.994 INFO:tasks.rados.rados.2.smithi175.stdout:1188: finishing copy_from to smithi17557870-36 2024-09-11T03:42:46.994 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 223 (ObjNum 277 snap 73 seq_num 277) dirty exists 2024-09-11T03:42:46.994 INFO:tasks.rados.rados.2.smithi175.stdout:1187: finishing copy_from to smithi17557870-43 2024-09-11T03:42:46.994 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 369 (ObjNum 412 snap 127 seq_num 412) dirty exists 2024-09-11T03:42:46.994 INFO:tasks.rados.rados.2.smithi175.stdout:1184: done (6 left) 2024-09-11T03:42:46.995 INFO:tasks.rados.rados.2.smithi175.stdout:1185: done (5 left) 2024-09-11T03:42:46.995 INFO:tasks.rados.rados.2.smithi175.stdout:1186: done (4 left) 2024-09-11T03:42:46.995 INFO:tasks.rados.rados.2.smithi175.stdout:1187: done (3 left) 2024-09-11T03:42:46.995 INFO:tasks.rados.rados.2.smithi175.stdout:1188: done (2 left) 2024-09-11T03:42:46.995 INFO:tasks.rados.rados.2.smithi175.stdout:1191: done (1 left) 2024-09-11T03:42:46.995 INFO:tasks.rados.rados.2.smithi175.stdout:1192: read oid 37 snap -1 2024-09-11T03:42:46.995 INFO:tasks.rados.rados.2.smithi175.stdout:1192: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:42:46.995 INFO:tasks.rados.rados.2.smithi175.stdout:1193: snap_create 2024-09-11T03:42:46.996 INFO:tasks.rados.rados.2.smithi175.stdout:1190: expect (ObjNum 310 snap 85 seq_num 310) 2024-09-11T03:42:47.004 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:47.026 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:47.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:47.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:47.128 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:47.128 INFO:tasks.rados.rados.2.smithi175.stdout:1192: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:42:47.149 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:47.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:47.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:47.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:46 smithi084 ceph-mon[27254]: osdmap e349: 12 total, 12 up, 10 in 2024-09-11T03:42:47.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:46 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:47.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:46 smithi084 ceph-mon[27254]: osdmap e350: 12 total, 12 up, 10 in 2024-09-11T03:42:47.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:47.276 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:47.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:47.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:47.378 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:47.400 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:47.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:47.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:47.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:46 smithi121 ceph-mon[30729]: osdmap e349: 12 total, 12 up, 10 in 2024-09-11T03:42:47.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:46 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:47.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:46 smithi121 ceph-mon[30729]: osdmap e350: 12 total, 12 up, 10 in 2024-09-11T03:42:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:46 smithi175 ceph-mon[30894]: osdmap e349: 12 total, 12 up, 10 in 2024-09-11T03:42:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:46 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:46 smithi175 ceph-mon[30894]: osdmap e350: 12 total, 12 up, 10 in 2024-09-11T03:42:47.502 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:47.523 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:47.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:47.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:47.625 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:47.646 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:47.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:47.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:47.746 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:42:47.746 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:42:47.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:47.770 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:42:47.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:47.774 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:47.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:47.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:47.900 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:47.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:47.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:47.932 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:47.973 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:42:47.977 INFO:tasks.rados.rados.2.smithi175.stdout:1190: done (2 left) 2024-09-11T03:42:47.977 INFO:tasks.rados.rados.2.smithi175.stdout:1192: done (1 left) 2024-09-11T03:42:47.977 INFO:tasks.rados.rados.2.smithi175.stdout:1193: done (0 left) 2024-09-11T03:42:47.977 INFO:tasks.rados.rados.2.smithi175.stdout:1194: snap_remove snap 121 2024-09-11T03:42:48.002 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:48.025 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:48.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:48.026 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:48.126 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:48.148 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:48.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:48.150 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:48.250 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:48.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:48.274 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:48.274 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:48.376 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:48.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:47 smithi084 ceph-mon[27254]: pgmap v679: 57 pgs: 57 active+clean; 221 MiB data, 825 MiB used, 893 GiB / 894 GiB avail; 15 MiB/s rd, 5.1 MiB/s wr, 29 op/s 2024-09-11T03:42:48.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:47 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:42:48.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:47 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:42:48.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:47 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4164528790' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:48.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:47 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:42:48.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:47 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:42:48.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:47 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4164528790' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:48.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:47 smithi084 ceph-mon[27254]: osdmap e351: 12 total, 12 up, 10 in 2024-09-11T03:42:48.397 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:48.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:48.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:48.418 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T03:42:48.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:47 smithi121 ceph-mon[30729]: pgmap v679: 57 pgs: 57 active+clean; 221 MiB data, 825 MiB used, 893 GiB / 894 GiB avail; 15 MiB/s rd, 5.1 MiB/s wr, 29 op/s 2024-09-11T03:42:48.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:47 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:42:48.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:47 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:42:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:47 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4164528790' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:47 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:42:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:47 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:42:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:47 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4164528790' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:47 smithi121 ceph-mon[30729]: osdmap e351: 12 total, 12 up, 10 in 2024-09-11T03:42:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:47 smithi175 ceph-mon[30894]: pgmap v679: 57 pgs: 57 active+clean; 221 MiB data, 825 MiB used, 893 GiB / 894 GiB avail; 15 MiB/s rd, 5.1 MiB/s wr, 29 op/s 2024-09-11T03:42:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:47 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:42:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:47 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:42:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:47 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4164528790' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:47 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:42:48.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:47 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:42:48.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:47 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4164528790' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:48.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:47 smithi175 ceph-mon[30894]: osdmap e351: 12 total, 12 up, 10 in 2024-09-11T03:42:48.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:48.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:48.523 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:48.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:48.624 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:48.647 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:48.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:48.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:48.673 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:42:48.749 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:48.770 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:48.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:48.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:48.872 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:48.894 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:48.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:48.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:48.983 INFO:tasks.rados.rados.2.smithi175.stdout:1194: done (0 left) 2024-09-11T03:42:48.983 INFO:tasks.rados.rados.2.smithi175.stdout:1195: rollback oid 18 current snap is 128 2024-09-11T03:42:48.983 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 18 to 115 2024-09-11T03:42:48.983 INFO:tasks.rados.rados.2.smithi175.stdout:1196: write (excl) oid 44 current snap is 128 2024-09-11T03:42:48.983 INFO:tasks.rados.rados.2.smithi175.stdout:1196: seq_num 419 ranges {642133=728746,2119014=532478} 2024-09-11T03:42:48.990 INFO:tasks.rados.rados.2.smithi175.stdout:1196: writing smithi17557870-44 from 642133 to 1370879 tid 1 2024-09-11T03:42:48.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:48.998 INFO:tasks.rados.rados.2.smithi175.stdout:1196: writing smithi17557870-44 from 2119014 to 2651492 tid 2 2024-09-11T03:42:48.998 INFO:tasks.rados.rados.2.smithi175.stdout:1197: delete oid 40 current snap is 128 2024-09-11T03:42:48.998 INFO:tasks.rados.rados.2.smithi175.stdout:1195: finishing rollback tid 0 to smithi17557870-18 2024-09-11T03:42:48.998 INFO:tasks.rados.rados.2.smithi175.stdout:1195: finishing rollback tid 1 to smithi17557870-18 2024-09-11T03:42:48.998 INFO:tasks.rados.rados.2.smithi175.stdout:1195: finishing rollback tid 2 to smithi17557870-18 2024-09-11T03:42:48.998 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 236 (ObjNum 261 snap 66 seq_num 261) dirty exists 2024-09-11T03:42:48.998 INFO:tasks.rados.rados.2.smithi175.stdout:1196: finishing write tid 1 to smithi17557870-44 2024-09-11T03:42:48.999 INFO:tasks.rados.rados.2.smithi175.stdout:1195: done (2 left) 2024-09-11T03:42:49.000 INFO:tasks.rados.rados.2.smithi175.stdout:1197: done (1 left) 2024-09-11T03:42:49.000 INFO:tasks.rados.rados.2.smithi175.stdout:1198: snap_remove snap 119 2024-09-11T03:42:49.019 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:49.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:49.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:49.081 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:49.114 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:48 smithi121 ceph-mon[30729]: 2.3 scrub starts 2024-09-11T03:42:49.114 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:48 smithi121 ceph-mon[30729]: 2.3 scrub ok 2024-09-11T03:42:49.114 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:48 smithi121 ceph-mon[30729]: 2.6 scrub starts 2024-09-11T03:42:49.114 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:48 smithi121 ceph-mon[30729]: 2.6 scrub ok 2024-09-11T03:42:49.115 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:48 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:42:49.115 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:48 smithi121 ceph-mon[30729]: 2.15 scrub ok 2024-09-11T03:42:49.115 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:48 smithi121 ceph-mon[30729]: osdmap e352: 12 total, 12 up, 10 in 2024-09-11T03:42:49.116 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:42:49.116 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:42:49.120 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:49.141 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:49.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:49.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:49.243 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:49.265 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:49.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:49.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:49.297 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:48 smithi084 ceph-mon[27254]: 2.3 scrub starts 2024-09-11T03:42:49.297 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:48 smithi084 ceph-mon[27254]: 2.3 scrub ok 2024-09-11T03:42:49.297 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:48 smithi084 ceph-mon[27254]: 2.6 scrub starts 2024-09-11T03:42:49.297 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:48 smithi084 ceph-mon[27254]: 2.6 scrub ok 2024-09-11T03:42:49.297 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:48 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:42:49.297 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:48 smithi084 ceph-mon[27254]: 2.15 scrub ok 2024-09-11T03:42:49.297 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:48 smithi084 ceph-mon[27254]: osdmap e352: 12 total, 12 up, 10 in 2024-09-11T03:42:49.368 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:49.393 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:49.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:49.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:49 smithi175 ceph-mon[30894]: 2.3 scrub starts 2024-09-11T03:42:49.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:49 smithi175 ceph-mon[30894]: 2.3 scrub ok 2024-09-11T03:42:49.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:49 smithi175 ceph-mon[30894]: 2.6 scrub starts 2024-09-11T03:42:49.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:49 smithi175 ceph-mon[30894]: 2.6 scrub ok 2024-09-11T03:42:49.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:49 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:42:49.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:49 smithi175 ceph-mon[30894]: 2.15 scrub ok 2024-09-11T03:42:49.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:49 smithi175 ceph-mon[30894]: osdmap e352: 12 total, 12 up, 10 in 2024-09-11T03:42:49.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:49.752 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:49.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:49.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:49.854 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:49.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T03:42:49.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:49.881 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:49.881 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:49.889 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: specified pg_num 51 < pg_num_max 32 2024-09-11T03:42:49.927 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:42:49.928 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:42:49.945 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:42:49.945 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:42:49.960 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:42:49.961 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:42:49.976 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:42:49.976 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:42:49.982 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:49.994 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:42:49.994 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:42:50.007 INFO:tasks.rados.rados.2.smithi175.stdout:1198: done (1 left) 2024-09-11T03:42:50.007 INFO:tasks.rados.rados.2.smithi175.stdout:1199: read oid 45 snap -1 2024-09-11T03:42:50.007 INFO:tasks.rados.rados.2.smithi175.stdout:1199: expect (ObjNum 416 snap 127 seq_num 416) 2024-09-11T03:42:50.007 INFO:tasks.rados.rados.2.smithi175.stdout:1200: snap_remove snap 120 2024-09-11T03:42:50.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:50.008 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:50.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:50.018 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:42:50.018 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:42:50.036 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:42:50.037 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:42:50.052 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:42:50.053 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:42:50.070 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:42:50.070 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:42:50.087 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:42:50.087 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:42:50.103 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:42:50.103 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:42:50.109 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:50.121 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:42:50.121 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:42:50.132 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:50.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:50.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:50.135 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:42:50.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:50 smithi084 ceph-mon[27254]: pgmap v682: 57 pgs: 57 active+clean; 209 MiB data, 811 MiB used, 893 GiB / 894 GiB avail; 19 MiB/s rd, 9.7 MiB/s wr, 36 op/s 2024-09-11T03:42:50.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:50 smithi084 ceph-mon[27254]: 1.c scrub starts 2024-09-11T03:42:50.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:50 smithi084 ceph-mon[27254]: 1.c scrub ok 2024-09-11T03:42:50.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:50 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3955269091' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-11T03:42:50.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:50 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/435087961' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:50.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:50 smithi121 ceph-mon[30729]: pgmap v682: 57 pgs: 57 active+clean; 209 MiB data, 811 MiB used, 893 GiB / 894 GiB avail; 19 MiB/s rd, 9.7 MiB/s wr, 36 op/s 2024-09-11T03:42:50.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:50 smithi121 ceph-mon[30729]: 1.c scrub starts 2024-09-11T03:42:50.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:50 smithi121 ceph-mon[30729]: 1.c scrub ok 2024-09-11T03:42:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:50 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3955269091' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-11T03:42:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:50 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/435087961' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:50.234 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:50.256 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:50.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:50.357 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:50.378 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:50.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:50.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:50.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:50 smithi175 ceph-mon[30894]: pgmap v682: 57 pgs: 57 active+clean; 209 MiB data, 811 MiB used, 893 GiB / 894 GiB avail; 19 MiB/s rd, 9.7 MiB/s wr, 36 op/s 2024-09-11T03:42:50.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:50 smithi175 ceph-mon[30894]: 1.c scrub starts 2024-09-11T03:42:50.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:50 smithi175 ceph-mon[30894]: 1.c scrub ok 2024-09-11T03:42:50.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:50 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3955269091' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-11T03:42:50.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:50 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/435087961' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:50.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:50.501 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:50.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:50.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:50.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:50.625 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:50.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:50.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:50.630 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:42:50.630 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:42:50.630 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:42:50.630 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:42:50.630 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:42:50.649 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:42:50.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:50.749 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:50.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:50.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:50.851 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:50.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:50.873 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:50.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:50.974 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:51.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:51.001 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:51.002 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:51.017 INFO:tasks.rados.rados.2.smithi175.stdout:1200: done (2 left) 2024-09-11T03:42:51.017 INFO:tasks.rados.rados.2.smithi175.stdout:1201: read oid 38 snap -1 2024-09-11T03:42:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:1201: expect (ObjNum 318 snap 86 seq_num 318) 2024-09-11T03:42:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:1202: snap_create 2024-09-11T03:42:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:1196: finishing write tid 2 to smithi17557870-44 2024-09-11T03:42:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:1196: finishing write tid 3 to smithi17557870-44 2024-09-11T03:42:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:1196: finishing write tid 4 to smithi17557870-44 2024-09-11T03:42:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:1196: oid 44 updating version 0 to 424 2024-09-11T03:42:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:1196: oid 44 updating version 424 to 425 2024-09-11T03:42:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:1196: oid 44 updating version 425 to 426 2024-09-11T03:42:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 426 (ObjNum 419 snap 128 seq_num 419) dirty exists 2024-09-11T03:42:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:1196: left oid 44 (ObjNum 419 snap 128 seq_num 419) 2024-09-11T03:42:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:1199: expect (ObjNum 416 snap 127 seq_num 416) 2024-09-11T03:42:51.042 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:42:51.102 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:51.123 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:51.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:51.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:51.190 INFO:tasks.rados.rados.2.smithi175.stdout:1201: expect (ObjNum 318 snap 86 seq_num 318) 2024-09-11T03:42:51.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:51.246 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:51.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:51.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:51.258 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:51 smithi121 ceph-mon[30729]: osdmap e353: 12 total, 12 up, 10 in 2024-09-11T03:42:51.259 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:51 smithi121 ceph-mon[30729]: 1.13 scrub starts 2024-09-11T03:42:51.259 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:51 smithi121 ceph-mon[30729]: 1.13 scrub ok 2024-09-11T03:42:51.259 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:51 smithi121 ceph-mon[30729]: 2.19 scrub starts 2024-09-11T03:42:51.259 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:51 smithi121 ceph-mon[30729]: 2.19 scrub ok 2024-09-11T03:42:51.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T03:42:51.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:51.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:51 smithi084 ceph-mon[27254]: osdmap e353: 12 total, 12 up, 10 in 2024-09-11T03:42:51.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:51 smithi084 ceph-mon[27254]: 1.13 scrub starts 2024-09-11T03:42:51.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:51 smithi084 ceph-mon[27254]: 1.13 scrub ok 2024-09-11T03:42:51.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:51 smithi084 ceph-mon[27254]: 2.19 scrub starts 2024-09-11T03:42:51.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:51 smithi084 ceph-mon[27254]: 2.19 scrub ok 2024-09-11T03:42:51.372 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:51.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:51.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:51.419 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:51.474 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:51.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:51 smithi175 ceph-mon[30894]: osdmap e353: 12 total, 12 up, 10 in 2024-09-11T03:42:51.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:51 smithi175 ceph-mon[30894]: 1.13 scrub starts 2024-09-11T03:42:51.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:51 smithi175 ceph-mon[30894]: 1.13 scrub ok 2024-09-11T03:42:51.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:51 smithi175 ceph-mon[30894]: 2.19 scrub starts 2024-09-11T03:42:51.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:51 smithi175 ceph-mon[30894]: 2.19 scrub ok 2024-09-11T03:42:51.495 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:51.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:51.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:51.568 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:42:51.597 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:51.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:51.618 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:51.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:51.719 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:51.742 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:51.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:51.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:51.844 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:51.865 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:51.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:51.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:51.896 INFO:tasks.rados.rados.2.smithi175.stdout:1196: done (3 left) 2024-09-11T03:42:51.896 INFO:tasks.rados.rados.2.smithi175.stdout:1199: done (2 left) 2024-09-11T03:42:51.896 INFO:tasks.rados.rados.2.smithi175.stdout:1201: done (1 left) 2024-09-11T03:42:51.896 INFO:tasks.rados.rados.2.smithi175.stdout:1202: done (0 left) 2024-09-11T03:42:51.896 INFO:tasks.rados.rados.2.smithi175.stdout:1203: read oid 47 snap -1 2024-09-11T03:42:51.896 INFO:tasks.rados.rados.2.smithi175.stdout:1203: expect (ObjNum 369 snap 113 seq_num 369) 2024-09-11T03:42:51.896 INFO:tasks.rados.rados.2.smithi175.stdout:1204: read oid 49 snap -1 2024-09-11T03:42:51.896 INFO:tasks.rados.rados.2.smithi175.stdout:1204: expect (ObjNum 338 snap 98 seq_num 338) 2024-09-11T03:42:51.896 INFO:tasks.rados.rados.2.smithi175.stdout:1205: read oid 17 snap -1 2024-09-11T03:42:51.896 INFO:tasks.rados.rados.2.smithi175.stdout:1205: expect (ObjNum 384 snap 117 seq_num 384) 2024-09-11T03:42:51.897 INFO:tasks.rados.rados.2.smithi175.stdout:1206: delete oid 26 current snap is 129 2024-09-11T03:42:51.898 INFO:tasks.rados.rados.2.smithi175.stdout:1206: done (3 left) 2024-09-11T03:42:51.898 INFO:tasks.rados.rados.2.smithi175.stdout:1207: snap_remove snap 128 2024-09-11T03:42:51.967 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:51.989 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:51.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:51.990 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:52.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:52.112 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:52.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:52.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:52.121 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:42:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:42:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:42:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:42:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:588 2.f e129/95 2:fda69313:::smithi17557870-16:head v 129'32, mlcod=124'28)", 2024-09-11T03:42:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:05.847680+0000", 2024-09-11T03:42:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 226.246391878, 2024-09-11T03:42:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075524320000000004, 2024-09-11T03:42:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.847680+0000", 2024-09-11T03:42:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.847680+0000", 2024-09-11T03:42:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.847681+0000", 2024-09-11T03:42:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.184e-06 2024-09-11T03:42:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.853139+0000", 2024-09-11T03:42:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054575960000000003 2024-09-11T03:42:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.853154+0000", 2024-09-11T03:42:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5285999999999999e-05 2024-09-11T03:42:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.853179+0000", 2024-09-11T03:42:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4380999999999999e-05 2024-09-11T03:42:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.853234+0000", 2024-09-11T03:42:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5164999999999998e-05 2024-09-11T03:42:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.853250+0000", 2024-09-11T03:42:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6541999999999998e-05 2024-09-11T03:42:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.855165+0000", 2024-09-11T03:42:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019147560000000001 2024-09-11T03:42:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.855233+0000", 2024-09-11T03:42:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7521999999999995e-05 2024-09-11T03:42:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:847 2.d e148/95 2:b1c0f730:::smithi17557870-20:head v 148'82, mlcod=136'79)", 2024-09-11T03:42:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:24.846640+0000", 2024-09-11T03:42:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 207.247431799, 2024-09-11T03:42:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053917130000000002, 2024-09-11T03:42:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.846640+0000", 2024-09-11T03:42:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.846640+0000", 2024-09-11T03:42:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.846641+0000", 2024-09-11T03:42:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3500000000000002e-07 2024-09-11T03:42:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.850322+0000", 2024-09-11T03:42:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036810390000000001 2024-09-11T03:42:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.850336+0000", 2024-09-11T03:42:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4688999999999999e-05 2024-09-11T03:42:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.850345+0000", 2024-09-11T03:42:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3010000000000007e-06 2024-09-11T03:42:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.850409+0000", 2024-09-11T03:42:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4367000000000002e-05 2024-09-11T03:42:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.850434+0000", 2024-09-11T03:42:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4589000000000001e-05 2024-09-11T03:42:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.851980+0000", 2024-09-11T03:42:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001546513 2024-09-11T03:42:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:24.852032+0000", 2024-09-11T03:42:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1879999999999998e-05 2024-09-11T03:42:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1290 2.d e196/95 2:b1c0f730:::smithi17557870-20:head v 196'156, mlcod=192'152)", 2024-09-11T03:42:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:12.850226+0000", 2024-09-11T03:42:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 159.243845771, 2024-09-11T03:42:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054838100000000004, 2024-09-11T03:42:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.850226+0000", 2024-09-11T03:42:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.850226+0000", 2024-09-11T03:42:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.850227+0000", 2024-09-11T03:42:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7700000000000004e-07 2024-09-11T03:42:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.853952+0000", 2024-09-11T03:42:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003724757 2024-09-11T03:42:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.853966+0000", 2024-09-11T03:42:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4469e-05 2024-09-11T03:42:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.853973+0000", 2024-09-11T03:42:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6669999999999999e-06 2024-09-11T03:42:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.854041+0000", 2024-09-11T03:42:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7669999999999994e-05 2024-09-11T03:42:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.854067+0000", 2024-09-11T03:42:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6363000000000002e-05 2024-09-11T03:42:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.855661+0000", 2024-09-11T03:42:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001594088 2024-09-11T03:42:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:12.855710+0000", 2024-09-11T03:42:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9119000000000002e-05 2024-09-11T03:42:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1516 2.d e212/95 2:b6a25a79:::smithi17557870-47:head v 212'170, mlcod=206'166)", 2024-09-11T03:42:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:28.923064+0000", 2024-09-11T03:42:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 143.17100792400001, 2024-09-11T03:42:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059113580000000002, 2024-09-11T03:42:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.923064+0000", 2024-09-11T03:42:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.923064+0000", 2024-09-11T03:42:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.923065+0000", 2024-09-11T03:42:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7599999999999998e-07 2024-09-11T03:42:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.927101+0000", 2024-09-11T03:42:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040361549999999996 2024-09-11T03:42:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.927112+0000", 2024-09-11T03:42:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.101e-05 2024-09-11T03:42:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.927119+0000", 2024-09-11T03:42:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5329999999999997e-06 2024-09-11T03:42:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.927197+0000", 2024-09-11T03:42:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7921000000000002e-05 2024-09-11T03:42:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.927216+0000", 2024-09-11T03:42:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8651000000000002e-05 2024-09-11T03:42:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.928941+0000", 2024-09-11T03:42:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00172554 2024-09-11T03:42:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.928976+0000", 2024-09-11T03:42:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4272e-05 2024-09-11T03:42:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1513 2.5 e212/95 2:a6e3bbd9:::smithi17557870-38:head v 212'178, mlcod=203'174)", 2024-09-11T03:42:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:28.928761+0000", 2024-09-11T03:42:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 143.165311277, 2024-09-11T03:42:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0094340099999999996, 2024-09-11T03:42:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.928761+0000", 2024-09-11T03:42:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.928761+0000", 2024-09-11T03:42:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.928761+0000", 2024-09-11T03:42:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.68e-07 2024-09-11T03:42:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.935532+0000", 2024-09-11T03:42:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0067704449999999999 2024-09-11T03:42:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.935544+0000", 2024-09-11T03:42:52.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2427e-05 2024-09-11T03:42:52.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.935569+0000", 2024-09-11T03:42:52.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5397000000000001e-05 2024-09-11T03:42:52.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.935616+0000", 2024-09-11T03:42:52.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6248999999999997e-05 2024-09-11T03:42:52.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.935633+0000", 2024-09-11T03:42:52.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7405e-05 2024-09-11T03:42:52.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.938161+0000", 2024-09-11T03:42:52.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002527928 2024-09-11T03:42:52.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.938195+0000", 2024-09-11T03:42:52.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3991e-05 2024-09-11T03:42:52.144 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.145 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.145 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.145 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1589 2.7 e218/191 2:e77ce595:::smithi17557870-31:head v 218'174, mlcod=218'168)", 2024-09-11T03:42:52.145 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:35.117324+0000", 2024-09-11T03:42:52.145 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 136.976748155, 2024-09-11T03:42:52.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0068407770000000001, 2024-09-11T03:42:52.145 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.145 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.145 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.145 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.117324+0000", 2024-09-11T03:42:52.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.117324+0000", 2024-09-11T03:42:52.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.117328+0000", 2024-09-11T03:42:52.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3669999999999999e-06 2024-09-11T03:42:52.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.120926+0000", 2024-09-11T03:42:52.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035978490000000002 2024-09-11T03:42:52.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.120944+0000", 2024-09-11T03:42:52.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.766e-05 2024-09-11T03:42:52.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.120952+0000", 2024-09-11T03:42:52.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7819999999999994e-06 2024-09-11T03:42:52.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.121018+0000", 2024-09-11T03:42:52.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6452999999999995e-05 2024-09-11T03:42:52.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.121041+0000", 2024-09-11T03:42:52.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3102999999999999e-05 2024-09-11T03:42:52.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.124136+0000", 2024-09-11T03:42:52.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030948220000000001 2024-09-11T03:42:52.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.124165+0000", 2024-09-11T03:42:52.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8741000000000001e-05 2024-09-11T03:42:52.149 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.149 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.149 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.149 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1634 2.7 e219/191 2:e23d1762:::smithi17557870-21:head v 219'181, mlcod=219'179)", 2024-09-11T03:42:52.149 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:36.217648+0000", 2024-09-11T03:42:52.149 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 135.87642418600001, 2024-09-11T03:42:52.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056020419999999998, 2024-09-11T03:42:52.150 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.150 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.150 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.150 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:36.217648+0000", 2024-09-11T03:42:52.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:36.217648+0000", 2024-09-11T03:42:52.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:36.217649+0000", 2024-09-11T03:42:52.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9599999999999999e-07 2024-09-11T03:42:52.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:36.220943+0000", 2024-09-11T03:42:52.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003294707 2024-09-11T03:42:52.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:36.220955+0000", 2024-09-11T03:42:52.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.151e-05 2024-09-11T03:42:52.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:36.220960+0000", 2024-09-11T03:42:52.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0039999999999999e-06 2024-09-11T03:42:52.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:36.221013+0000", 2024-09-11T03:42:52.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3596e-05 2024-09-11T03:42:52.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:36.221035+0000", 2024-09-11T03:42:52.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1588999999999999e-05 2024-09-11T03:42:52.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:36.223201+0000", 2024-09-11T03:42:52.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002165578 2024-09-11T03:42:52.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:36.223250+0000", 2024-09-11T03:42:52.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9361999999999999e-05 2024-09-11T03:42:52.153 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.153 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.154 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.154 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1816 2.d e243/95 2:b7bdfe52:::smithi17557870-10:head v 243'237, mlcod=243'233)", 2024-09-11T03:42:52.154 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:59.922654+0000", 2024-09-11T03:42:52.154 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 112.171418254, 2024-09-11T03:42:52.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092334370000000006, 2024-09-11T03:42:52.154 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.154 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.154 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.154 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.922654+0000", 2024-09-11T03:42:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.922654+0000", 2024-09-11T03:42:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.922656+0000", 2024-09-11T03:42:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8500000000000001e-06 2024-09-11T03:42:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.929356+0000", 2024-09-11T03:42:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0066998589999999998 2024-09-11T03:42:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.929367+0000", 2024-09-11T03:42:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.182e-05 2024-09-11T03:42:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.929374+0000", 2024-09-11T03:42:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4649999999999999e-06 2024-09-11T03:42:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.929468+0000", 2024-09-11T03:42:52.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4549000000000006e-05 2024-09-11T03:42:52.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.929487+0000", 2024-09-11T03:42:52.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8870000000000001e-05 2024-09-11T03:42:52.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.931844+0000", 2024-09-11T03:42:52.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023565840000000001 2024-09-11T03:42:52.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.931887+0000", 2024-09-11T03:42:52.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3439999999999997e-05 2024-09-11T03:42:52.158 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.158 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.158 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.158 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:2115 2.5 e268/95 2:a6e3bbd9:::smithi17557870-38:head v 268'256, mlcod=260'252)", 2024-09-11T03:42:52.158 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:24.876823+0000", 2024-09-11T03:42:52.158 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.217249140000007, 2024-09-11T03:42:52.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075190609999999996, 2024-09-11T03:42:52.158 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.159 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.159 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.159 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.876823+0000", 2024-09-11T03:42:52.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.876823+0000", 2024-09-11T03:42:52.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.876835+0000", 2024-09-11T03:42:52.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1647e-05 2024-09-11T03:42:52.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.881215+0000", 2024-09-11T03:42:52.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043805720000000001 2024-09-11T03:42:52.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.881229+0000", 2024-09-11T03:42:52.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4088e-05 2024-09-11T03:42:52.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.881257+0000", 2024-09-11T03:42:52.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7937e-05 2024-09-11T03:42:52.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.881355+0000", 2024-09-11T03:42:52.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8159999999999995e-05 2024-09-11T03:42:52.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.881377+0000", 2024-09-11T03:42:52.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.19e-05 2024-09-11T03:42:52.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.884304+0000", 2024-09-11T03:42:52.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002926452 2024-09-11T03:42:52.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.884342+0000", 2024-09-11T03:42:52.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8305e-05 2024-09-11T03:42:52.162 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.162 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.162 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.163 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:2421 2.18 e294/281 2:1bdf1a36:::smithi17557870-36:head v 294'198, mlcod=0'0)", 2024-09-11T03:42:52.163 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:50.964382+0000", 2024-09-11T03:42:52.163 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.129690582000002, 2024-09-11T03:42:52.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056538329999999996, 2024-09-11T03:42:52.163 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.163 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.163 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.163 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:50.964382+0000", 2024-09-11T03:42:52.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:50.964382+0000", 2024-09-11T03:42:52.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:50.964382+0000", 2024-09-11T03:42:52.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4300000000000003e-07 2024-09-11T03:42:52.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:50.967595+0000", 2024-09-11T03:42:52.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032131149999999999 2024-09-11T03:42:52.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:50.967607+0000", 2024-09-11T03:42:52.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2255999999999999e-05 2024-09-11T03:42:52.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:50.967631+0000", 2024-09-11T03:42:52.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3519e-05 2024-09-11T03:42:52.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:50.967697+0000", 2024-09-11T03:42:52.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6278000000000005e-05 2024-09-11T03:42:52.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:50.967721+0000", 2024-09-11T03:42:52.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4034999999999999e-05 2024-09-11T03:42:52.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:50.969995+0000", 2024-09-11T03:42:52.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022739959999999999 2024-09-11T03:42:52.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:50.970035+0000", 2024-09-11T03:42:52.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0091e-05 2024-09-11T03:42:52.169 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.169 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.169 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.169 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:2492 2.5 e302/276 2:a1a99782:::smithi17557870-1:head v 302'331, mlcod=297'328)", 2024-09-11T03:42:52.169 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:58.902222+0000", 2024-09-11T03:42:52.169 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.191850172999999, 2024-09-11T03:42:52.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.007615704, 2024-09-11T03:42:52.170 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.170 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.170 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.170 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.902222+0000", 2024-09-11T03:42:52.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.902222+0000", 2024-09-11T03:42:52.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.902223+0000", 2024-09-11T03:42:52.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0999999999999999e-07 2024-09-11T03:42:52.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.907105+0000", 2024-09-11T03:42:52.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048825120000000003 2024-09-11T03:42:52.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.907120+0000", 2024-09-11T03:42:52.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5362999999999998e-05 2024-09-11T03:42:52.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.907145+0000", 2024-09-11T03:42:52.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4941999999999999e-05 2024-09-11T03:42:52.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.907217+0000", 2024-09-11T03:42:52.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1585e-05 2024-09-11T03:42:52.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.907233+0000", 2024-09-11T03:42:52.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5684000000000001e-05 2024-09-11T03:42:52.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.909765+0000", 2024-09-11T03:42:52.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025323889999999999 2024-09-11T03:42:52.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.909838+0000", 2024-09-11T03:42:52.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2719000000000006e-05 2024-09-11T03:42:52.174 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.174 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.174 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.174 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:2511 2.d e305/95 2:b6a25a79:::smithi17557870-47:head v 305'368, mlcod=304'363)", 2024-09-11T03:42:52.174 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:01.878380+0000", 2024-09-11T03:42:52.174 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.215691761000002, 2024-09-11T03:42:52.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065527210000000001, 2024-09-11T03:42:52.174 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.174 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.174 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.174 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.878380+0000", 2024-09-11T03:42:52.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.878380+0000", 2024-09-11T03:42:52.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.878381+0000", 2024-09-11T03:42:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0599999999999999e-07 2024-09-11T03:42:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.883199+0000", 2024-09-11T03:42:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048174150000000002 2024-09-11T03:42:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.883211+0000", 2024-09-11T03:42:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2095e-05 2024-09-11T03:42:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.883218+0000", 2024-09-11T03:42:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5820000000000004e-06 2024-09-11T03:42:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.883261+0000", 2024-09-11T03:42:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3163e-05 2024-09-11T03:42:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.883275+0000", 2024-09-11T03:42:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.344e-05 2024-09-11T03:42:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.884892+0000", 2024-09-11T03:42:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016175580000000001 2024-09-11T03:42:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.884933+0000", 2024-09-11T03:42:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0661999999999998e-05 2024-09-11T03:42:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:2568 2.d e314/95 2:b6a25a79:::smithi17557870-47:head v 314'392, mlcod=314'388)", 2024-09-11T03:42:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:10.885931+0000", 2024-09-11T03:42:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.208141451000003, 2024-09-11T03:42:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0089540990000000001, 2024-09-11T03:42:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.885931+0000", 2024-09-11T03:42:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.885931+0000", 2024-09-11T03:42:52.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.885932+0000", 2024-09-11T03:42:52.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3900000000000003e-07 2024-09-11T03:42:52.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.892118+0000", 2024-09-11T03:42:52.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061860530000000004 2024-09-11T03:42:52.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.892130+0000", 2024-09-11T03:42:52.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2381999999999999e-05 2024-09-11T03:42:52.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.892136+0000", 2024-09-11T03:42:52.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2820000000000002e-06 2024-09-11T03:42:52.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.892219+0000", 2024-09-11T03:42:52.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3115000000000006e-05 2024-09-11T03:42:52.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.892236+0000", 2024-09-11T03:42:52.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6864e-05 2024-09-11T03:42:52.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.894845+0000", 2024-09-11T03:42:52.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002608806 2024-09-11T03:42:52.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.894885+0000", 2024-09-11T03:42:52.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9657999999999999e-05 2024-09-11T03:42:52.183 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.183 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.183 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.183 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:2931 2.5 e346/276 2:a6d17fc0:::smithi17557870-13:head v 346'401, mlcod=340'397)", 2024-09-11T03:42:52.183 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:43.125695+0000", 2024-09-11T03:42:52.183 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.9683772340000001, 2024-09-11T03:42:52.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070637080000000001, 2024-09-11T03:42:52.184 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.184 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.184 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.184 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.125695+0000", 2024-09-11T03:42:52.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.125695+0000", 2024-09-11T03:42:52.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.125696+0000", 2024-09-11T03:42:52.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1800000000000004e-07 2024-09-11T03:42:52.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.130541+0000", 2024-09-11T03:42:52.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048448349999999996 2024-09-11T03:42:52.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.130555+0000", 2024-09-11T03:42:52.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4122e-05 2024-09-11T03:42:52.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.130562+0000", 2024-09-11T03:42:52.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1489999999999997e-06 2024-09-11T03:42:52.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.130646+0000", 2024-09-11T03:42:52.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3823e-05 2024-09-11T03:42:52.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.130663+0000", 2024-09-11T03:42:52.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7385e-05 2024-09-11T03:42:52.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.132716+0000", 2024-09-11T03:42:52.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020528759999999999 2024-09-11T03:42:52.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.132759+0000", 2024-09-11T03:42:52.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2599999999999999e-05 2024-09-11T03:42:52.187 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.187 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.187 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.188 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:2933 2.d e346/333 2:b6874adf:::smithi17557870-17:head v 346'440, mlcod=334'436)", 2024-09-11T03:42:52.188 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:43.239992+0000", 2024-09-11T03:42:52.188 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.8540803090000004, 2024-09-11T03:42:52.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048373210000000003, 2024-09-11T03:42:52.188 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.188 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.188 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.188 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.239992+0000", 2024-09-11T03:42:52.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.239992+0000", 2024-09-11T03:42:52.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.239992+0000", 2024-09-11T03:42:52.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2399999999999999e-07 2024-09-11T03:42:52.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.241991+0000", 2024-09-11T03:42:52.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019987849999999999 2024-09-11T03:42:52.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.242021+0000", 2024-09-11T03:42:52.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9981999999999999e-05 2024-09-11T03:42:52.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.242036+0000", 2024-09-11T03:42:52.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5328e-05 2024-09-11T03:42:52.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.242114+0000", 2024-09-11T03:42:52.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7951e-05 2024-09-11T03:42:52.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.242138+0000", 2024-09-11T03:42:52.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3947999999999998e-05 2024-09-11T03:42:52.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.244771+0000", 2024-09-11T03:42:52.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026324370000000001 2024-09-11T03:42:52.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.244829+0000", 2024-09-11T03:42:52.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8566000000000001e-05 2024-09-11T03:42:52.192 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.192 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.192 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.192 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:2970 2.d e348/333 2:b1c0f730:::smithi17557870-20:head v 348'452, mlcod=348'448)", 2024-09-11T03:42:52.192 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:44.989767+0000", 2024-09-11T03:42:52.193 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 7.104305063, 2024-09-11T03:42:52.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006454502, 2024-09-11T03:42:52.193 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.193 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.193 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.193 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.989767+0000", 2024-09-11T03:42:52.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.989767+0000", 2024-09-11T03:42:52.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.989768+0000", 2024-09-11T03:42:52.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5099999999999999e-07 2024-09-11T03:42:52.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.991761+0000", 2024-09-11T03:42:52.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019935859999999999 2024-09-11T03:42:52.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.991795+0000", 2024-09-11T03:42:52.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3954000000000001e-05 2024-09-11T03:42:52.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.991810+0000", 2024-09-11T03:42:52.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.502e-05 2024-09-11T03:42:52.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.991908+0000", 2024-09-11T03:42:52.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7943999999999996e-05 2024-09-11T03:42:52.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.991931+0000", 2024-09-11T03:42:52.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2572999999999999e-05 2024-09-11T03:42:52.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.996168+0000", 2024-09-11T03:42:52.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042373039999999999 2024-09-11T03:42:52.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.996222+0000", 2024-09-11T03:42:52.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3369999999999999e-05 2024-09-11T03:42:52.196 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.197 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.197 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.197 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(osd.8.0:622 2.5 e353/276 2:a6e3bbd9:::smithi17557870-38:7a v 353'425, mlcod=352'421)", 2024-09-11T03:42:52.197 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:50.005197+0000", 2024-09-11T03:42:52.197 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 2.0888751459999999, 2024-09-11T03:42:52.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078228199999999994, 2024-09-11T03:42:52.197 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.197 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.197 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.197 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005197+0000", 2024-09-11T03:42:52.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005197+0000", 2024-09-11T03:42:52.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005197+0000", 2024-09-11T03:42:52.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3300000000000001e-07 2024-09-11T03:42:52.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005208+0000", 2024-09-11T03:42:52.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0366e-05 2024-09-11T03:42:52.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005227+0000", 2024-09-11T03:42:52.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8953999999999998e-05 2024-09-11T03:42:52.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005235+0000", 2024-09-11T03:42:52.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7369999999999995e-06 2024-09-11T03:42:52.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005320+0000", 2024-09-11T03:42:52.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4956000000000004e-05 2024-09-11T03:42:52.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005357+0000", 2024-09-11T03:42:52.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6640999999999997e-05 2024-09-11T03:42:52.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.012805+0000", 2024-09-11T03:42:52.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0074477349999999996 2024-09-11T03:42:52.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.013020+0000", 2024-09-11T03:42:52.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000215198 2024-09-11T03:42:52.201 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.201 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.201 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.201 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(osd.8.0:624 2.5 e353/276 2:a6d17fc0:::smithi17557870-13:80 v 353'427, mlcod=352'421)", 2024-09-11T03:42:52.201 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:50.005913+0000", 2024-09-11T03:42:52.201 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 2.0881588519999998, 2024-09-11T03:42:52.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076474830000000001, 2024-09-11T03:42:52.202 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.202 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.202 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.202 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005913+0000", 2024-09-11T03:42:52.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005913+0000", 2024-09-11T03:42:52.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005914+0000", 2024-09-11T03:42:52.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0999999999999998e-07 2024-09-11T03:42:52.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005921+0000", 2024-09-11T03:42:52.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5849999999999996e-06 2024-09-11T03:42:52.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005929+0000", 2024-09-11T03:42:52.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3210000000000005e-06 2024-09-11T03:42:52.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005935+0000", 2024-09-11T03:42:52.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9560000000000002e-06 2024-09-11T03:42:52.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005981+0000", 2024-09-11T03:42:52.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6323999999999999e-05 2024-09-11T03:42:52.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005993+0000", 2024-09-11T03:42:52.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2211e-05 2024-09-11T03:42:52.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.013516+0000", 2024-09-11T03:42:52.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075221560000000003 2024-09-11T03:42:52.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.013561+0000", 2024-09-11T03:42:52.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5219999999999997e-05 2024-09-11T03:42:52.205 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.205 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.205 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(osd.9.0:306 2.18 e353/312 2:1984a3a9:::smithi17557870-23:80 v 353'246, mlcod=352'242)", 2024-09-11T03:42:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:50.007690+0000", 2024-09-11T03:42:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 2.0863819530000001, 2024-09-11T03:42:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052879249999999997, 2024-09-11T03:42:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.007690+0000", 2024-09-11T03:42:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.007690+0000", 2024-09-11T03:42:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.007691+0000", 2024-09-11T03:42:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1800000000000002e-07 2024-09-11T03:42:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.007694+0000", 2024-09-11T03:42:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6229999999999999e-06 2024-09-11T03:42:52.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.007711+0000", 2024-09-11T03:42:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7283999999999999e-05 2024-09-11T03:42:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.007723+0000", 2024-09-11T03:42:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1556000000000001e-05 2024-09-11T03:42:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.007785+0000", 2024-09-11T03:42:52.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2163999999999996e-05 2024-09-11T03:42:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.007806+0000", 2024-09-11T03:42:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0653e-05 2024-09-11T03:42:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.012942+0000", 2024-09-11T03:42:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005135896 2024-09-11T03:42:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.012978+0000", 2024-09-11T03:42:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6430999999999997e-05 2024-09-11T03:42:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(osd.9.0:307 2.18 e353/312 2:1bdf1a36:::smithi17557870-36:80 v 353'248, mlcod=352'242)", 2024-09-11T03:42:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:50.008234+0000", 2024-09-11T03:42:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 2.0858380510000001, 2024-09-11T03:42:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052512460000000002, 2024-09-11T03:42:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:42:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:42:52.210 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:42:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:42:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:42:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.008234+0000", 2024-09-11T03:42:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:42:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.008234+0000", 2024-09-11T03:42:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:42:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:42:52.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.008235+0000", 2024-09-11T03:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5300000000000003e-07 2024-09-11T03:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.008241+0000", 2024-09-11T03:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.755e-06 2024-09-11T03:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.008254+0000", 2024-09-11T03:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3852999999999999e-05 2024-09-11T03:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.008261+0000", 2024-09-11T03:42:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.37e-06 2024-09-11T03:42:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:42:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.008950+0000", 2024-09-11T03:42:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00068876899999999999 2024-09-11T03:42:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:42:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.008970+0000", 2024-09-11T03:42:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0327999999999999e-05 2024-09-11T03:42:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:42:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.013458+0000", 2024-09-11T03:42:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044884080000000002 2024-09-11T03:42:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:42:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:42:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:42:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.013485+0000", 2024-09-11T03:42:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7010000000000001e-05 2024-09-11T03:42:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:42:52.214 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:42:52.214 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:42:52.215 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:52.236 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:52.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:52.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:52.338 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:52.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:52.360 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:52.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:52 smithi084 ceph-mon[27254]: 2.f scrub starts 2024-09-11T03:42:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:52 smithi084 ceph-mon[27254]: 2.f scrub ok 2024-09-11T03:42:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:52 smithi084 ceph-mon[27254]: 1.e scrub starts 2024-09-11T03:42:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:52 smithi084 ceph-mon[27254]: 1.e scrub ok 2024-09-11T03:42:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:52 smithi084 ceph-mon[27254]: pgmap v685: 57 pgs: 57 active+clean; 209 MiB data, 806 MiB used, 893 GiB / 894 GiB avail; 8.7 MiB/s rd, 5.4 MiB/s wr, 17 op/s 2024-09-11T03:42:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:52 smithi084 ceph-mon[27254]: 1.14 scrub starts 2024-09-11T03:42:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:52 smithi084 ceph-mon[27254]: 1.14 scrub ok 2024-09-11T03:42:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:52 smithi084 ceph-mon[27254]: osdmap e354: 12 total, 12 up, 10 in 2024-09-11T03:42:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:52 smithi084 ceph-mon[27254]: osdmap e355: 12 total, 12 up, 10 in 2024-09-11T03:42:52.407 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:52 smithi121 ceph-mon[30729]: 2.f scrub starts 2024-09-11T03:42:52.407 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:52 smithi121 ceph-mon[30729]: 2.f scrub ok 2024-09-11T03:42:52.407 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:52 smithi121 ceph-mon[30729]: 1.e scrub starts 2024-09-11T03:42:52.407 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:52 smithi121 ceph-mon[30729]: 1.e scrub ok 2024-09-11T03:42:52.407 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:52 smithi121 ceph-mon[30729]: pgmap v685: 57 pgs: 57 active+clean; 209 MiB data, 806 MiB used, 893 GiB / 894 GiB avail; 8.7 MiB/s rd, 5.4 MiB/s wr, 17 op/s 2024-09-11T03:42:52.408 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:52 smithi121 ceph-mon[30729]: 1.14 scrub starts 2024-09-11T03:42:52.408 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:52 smithi121 ceph-mon[30729]: 1.14 scrub ok 2024-09-11T03:42:52.408 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:52 smithi121 ceph-mon[30729]: osdmap e354: 12 total, 12 up, 10 in 2024-09-11T03:42:52.408 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:52 smithi121 ceph-mon[30729]: osdmap e355: 12 total, 12 up, 10 in 2024-09-11T03:42:52.461 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:52 smithi175 ceph-mon[30894]: 2.f scrub starts 2024-09-11T03:42:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:52 smithi175 ceph-mon[30894]: 2.f scrub ok 2024-09-11T03:42:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:52 smithi175 ceph-mon[30894]: 1.e scrub starts 2024-09-11T03:42:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:52 smithi175 ceph-mon[30894]: 1.e scrub ok 2024-09-11T03:42:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:52 smithi175 ceph-mon[30894]: pgmap v685: 57 pgs: 57 active+clean; 209 MiB data, 806 MiB used, 893 GiB / 894 GiB avail; 8.7 MiB/s rd, 5.4 MiB/s wr, 17 op/s 2024-09-11T03:42:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:52 smithi175 ceph-mon[30894]: 1.14 scrub starts 2024-09-11T03:42:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:52 smithi175 ceph-mon[30894]: 1.14 scrub ok 2024-09-11T03:42:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:52 smithi175 ceph-mon[30894]: osdmap e354: 12 total, 12 up, 10 in 2024-09-11T03:42:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:52 smithi175 ceph-mon[30894]: osdmap e355: 12 total, 12 up, 10 in 2024-09-11T03:42:52.485 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:52.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:52.487 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:52.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:52.612 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:52.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:52.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:52.714 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:52.736 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:52.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:52.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:52.861 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:52.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:52.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:52.886 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T03:42:52.894 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:42:52.898 INFO:tasks.rados.rados.2.smithi175.stdout:1207: done (3 left) 2024-09-11T03:42:52.898 INFO:tasks.rados.rados.2.smithi175.stdout:1208: read oid 44 snap -1 2024-09-11T03:42:52.898 INFO:tasks.rados.rados.2.smithi175.stdout:1208: expect (ObjNum 419 snap 128 seq_num 419) 2024-09-11T03:42:52.898 INFO:tasks.rados.rados.2.smithi175.stdout:1209: copy_from oid 14 from oid 27 current snap is 129 2024-09-11T03:42:52.898 INFO:tasks.rados.rados.2.smithi175.stdout:1204: expect (ObjNum 338 snap 98 seq_num 338) 2024-09-11T03:42:52.962 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:52.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:52.985 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:52.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:53.085 INFO:tasks.rados.rados.2.smithi175.stdout:1203: expect (ObjNum 369 snap 113 seq_num 369) 2024-09-11T03:42:53.085 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:53.107 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:53.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:53.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:53.169 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:42:53.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:53 smithi121 ceph-mon[30729]: 1.d scrub starts 2024-09-11T03:42:53.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:53 smithi121 ceph-mon[30729]: 1.d scrub ok 2024-09-11T03:42:53.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:53 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3948385525' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:42:53.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:53 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:53.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:53 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3948385525' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:42:53.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:53 smithi121 ceph-mon[30729]: osdmap e356: 12 total, 12 up, 10 in 2024-09-11T03:42:53.208 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:53.215 INFO:tasks.rados.rados.2.smithi175.stdout:1205: expect (ObjNum 384 snap 117 seq_num 384) 2024-09-11T03:42:53.229 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:53.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:53.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:53.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:53 smithi084 ceph-mon[27254]: 1.d scrub starts 2024-09-11T03:42:53.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:53 smithi084 ceph-mon[27254]: 1.d scrub ok 2024-09-11T03:42:53.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:53 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3948385525' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:42:53.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:53 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:53.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:53 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3948385525' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:42:53.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:53 smithi084 ceph-mon[27254]: osdmap e356: 12 total, 12 up, 10 in 2024-09-11T03:42:53.323 INFO:tasks.rados.rados.2.smithi175.stdout:1209: finishing copy_from racing read to smithi17557870-14 2024-09-11T03:42:53.323 INFO:tasks.rados.rados.2.smithi175.stdout:1209: finishing copy_from to smithi17557870-14 2024-09-11T03:42:53.323 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 182 (ObjNum 380 snap 117 seq_num 380) dirty exists 2024-09-11T03:42:53.323 INFO:tasks.rados.rados.2.smithi175.stdout:1208: expect (ObjNum 419 snap 128 seq_num 419) 2024-09-11T03:42:53.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:53.352 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:53.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:53.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:53.388 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:42:53.454 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:53.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:53.476 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:53.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:53.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:53 smithi175 ceph-mon[30894]: 1.d scrub starts 2024-09-11T03:42:53.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:53 smithi175 ceph-mon[30894]: 1.d scrub ok 2024-09-11T03:42:53.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:53 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3948385525' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:42:53.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:53 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:53.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:53 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3948385525' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:42:53.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:53 smithi175 ceph-mon[30894]: osdmap e356: 12 total, 12 up, 10 in 2024-09-11T03:42:53.478 INFO:tasks.rados.rados.2.smithi175.stdout:1203: done (4 left) 2024-09-11T03:42:53.478 INFO:tasks.rados.rados.2.smithi175.stdout:1204: done (3 left) 2024-09-11T03:42:53.478 INFO:tasks.rados.rados.2.smithi175.stdout:1205: done (2 left) 2024-09-11T03:42:53.478 INFO:tasks.rados.rados.2.smithi175.stdout:1208: done (1 left) 2024-09-11T03:42:53.478 INFO:tasks.rados.rados.2.smithi175.stdout:1209: done (0 left) 2024-09-11T03:42:53.478 INFO:tasks.rados.rados.2.smithi175.stdout:1210: write oid 40 current snap is 129 2024-09-11T03:42:53.478 INFO:tasks.rados.rados.2.smithi175.stdout:1210: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2024-09-11T03:42:53.484 INFO:tasks.rados.rados.2.smithi175.stdout:1210: writing smithi17557870-40 from 692554 to 1247047 tid 1 2024-09-11T03:42:53.491 INFO:tasks.rados.rados.2.smithi175.stdout:1210: writing smithi17557870-40 from 1778107 to 2457912 tid 2 2024-09-11T03:42:53.495 INFO:tasks.rados.rados.2.smithi175.stdout:1210: writing smithi17557870-40 from 2889444 to 3092554 tid 3 2024-09-11T03:42:53.495 INFO:tasks.rados.rados.2.smithi175.stdout:1211: read oid 46 snap -1 2024-09-11T03:42:53.495 INFO:tasks.rados.rados.2.smithi175.stdout:1211: expect (ObjNum 414 snap 127 seq_num 414) 2024-09-11T03:42:53.495 INFO:tasks.rados.rados.2.smithi175.stdout:1212: rollback oid 49 current snap is 129 2024-09-11T03:42:53.495 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 49 to 115 2024-09-11T03:42:53.495 INFO:tasks.rados.rados.2.smithi175.stdout:1210: finishing write tid 1 to smithi17557870-40 2024-09-11T03:42:53.495 INFO:tasks.rados.rados.2.smithi175.stdout:1213: copy_from oid 16 from oid 30 current snap is 129 2024-09-11T03:42:53.496 INFO:tasks.rados.rados.2.smithi175.stdout:1214: read oid 35 snap -1 2024-09-11T03:42:53.496 INFO:tasks.rados.rados.2.smithi175.stdout:1214: expect (ObjNum 411 snap 127 seq_num 411) 2024-09-11T03:42:53.496 INFO:tasks.rados.rados.2.smithi175.stdout:1215: write (excl) oid 27 current snap is 129 2024-09-11T03:42:53.496 INFO:tasks.rados.rados.2.smithi175.stdout:1210: finishing write tid 2 to smithi17557870-40 2024-09-11T03:42:53.496 INFO:tasks.rados.rados.2.smithi175.stdout:1215: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2024-09-11T03:42:53.502 INFO:tasks.rados.rados.2.smithi175.stdout:1215: writing smithi17557870-27 from 709361 to 1339103 tid 1 2024-09-11T03:42:53.508 INFO:tasks.rados.rados.2.smithi175.stdout:1215: writing smithi17557870-27 from 1836589 to 2376405 tid 2 2024-09-11T03:42:53.512 INFO:tasks.rados.rados.2.smithi175.stdout:1215: writing smithi17557870-27 from 3033220 to 3109361 tid 3 2024-09-11T03:42:53.512 INFO:tasks.rados.rados.2.smithi175.stdout:1216: copy_from oid 17 from oid 5 current snap is 129 2024-09-11T03:42:53.512 INFO:tasks.rados.rados.2.smithi175.stdout:1210: finishing write tid 3 to smithi17557870-40 2024-09-11T03:42:53.512 INFO:tasks.rados.rados.2.smithi175.stdout:1210: finishing write tid 4 to smithi17557870-40 2024-09-11T03:42:53.512 INFO:tasks.rados.rados.2.smithi175.stdout:1210: finishing write tid 5 to smithi17557870-40 2024-09-11T03:42:53.512 INFO:tasks.rados.rados.2.smithi175.stdout:1210: oid 40 updating version 0 to 373 2024-09-11T03:42:53.513 INFO:tasks.rados.rados.2.smithi175.stdout:1210: oid 40 version 373 is already newer than 371 2024-09-11T03:42:53.513 INFO:tasks.rados.rados.2.smithi175.stdout:1210: oid 40 version 373 is already newer than 372 2024-09-11T03:42:53.513 INFO:tasks.rados.rados.2.smithi175.stdout:1210: oid 40 updating version 373 to 374 2024-09-11T03:42:53.513 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 374 (ObjNum 422 snap 129 seq_num 422) dirty exists 2024-09-11T03:42:53.513 INFO:tasks.rados.rados.2.smithi175.stdout:1210: left oid 40 (ObjNum 422 snap 129 seq_num 422) 2024-09-11T03:42:53.513 INFO:tasks.rados.rados.2.smithi175.stdout:1211: expect (ObjNum 414 snap 127 seq_num 414) 2024-09-11T03:42:53.577 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:53.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:53.601 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:53.602 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:53.636 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:42:53.637 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:42:53.637 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:42:53.637 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:42:53.695 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:53.702 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:53.705 INFO:tasks.rados.rados.2.smithi175.stdout:1212: finishing rollback tid 0 to smithi17557870-49 2024-09-11T03:42:53.705 INFO:tasks.rados.rados.2.smithi175.stdout:1212: finishing rollback tid 1 to smithi17557870-49 2024-09-11T03:42:53.705 INFO:tasks.rados.rados.2.smithi175.stdout:1212: finishing rollback tid 2 to smithi17557870-49 2024-09-11T03:42:53.705 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 448 (ObjNum 338 snap 98 seq_num 338) dirty exists 2024-09-11T03:42:53.705 INFO:tasks.rados.rados.2.smithi175.stdout:1213: finishing copy_from racing read to smithi17557870-16 2024-09-11T03:42:53.705 INFO:tasks.rados.rados.2.smithi175.stdout:1214: expect (ObjNum 411 snap 127 seq_num 411) 2024-09-11T03:42:53.723 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:53.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:53.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:53.826 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:53.837 INFO:tasks.rados.rados.2.smithi175.stdout:1215: finishing write tid 1 to smithi17557870-27 2024-09-11T03:42:53.837 INFO:tasks.rados.rados.2.smithi175.stdout:1213: finishing copy_from to smithi17557870-16 2024-09-11T03:42:53.838 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 442 (ObjNum 396 snap 123 seq_num 396) dirty exists 2024-09-11T03:42:53.838 INFO:tasks.rados.rados.2.smithi175.stdout:1215: finishing write tid 2 to smithi17557870-27 2024-09-11T03:42:53.838 INFO:tasks.rados.rados.2.smithi175.stdout:1210: done (6 left) 2024-09-11T03:42:53.838 INFO:tasks.rados.rados.2.smithi175.stdout:1211: done (5 left) 2024-09-11T03:42:53.838 INFO:tasks.rados.rados.2.smithi175.stdout:1212: done (4 left) 2024-09-11T03:42:53.838 INFO:tasks.rados.rados.2.smithi175.stdout:1213: done (3 left) 2024-09-11T03:42:53.838 INFO:tasks.rados.rados.2.smithi175.stdout:1214: done (2 left) 2024-09-11T03:42:53.838 INFO:tasks.rados.rados.2.smithi175.stdout:1217: write (excl) oid 28 current snap is 129 2024-09-11T03:42:53.838 INFO:tasks.rados.rados.2.smithi175.stdout:1216: finishing copy_from to smithi17557870-17 2024-09-11T03:42:53.838 INFO:tasks.rados.rados.2.smithi175.stdout:1216: got expected ENOENT (src dne) 2024-09-11T03:42:53.838 INFO:tasks.rados.rados.2.smithi175.stdout:1216: finishing copy_from racing read to smithi17557870-17 2024-09-11T03:42:53.838 INFO:tasks.rados.rados.2.smithi175.stdout:1215: finishing write tid 3 to smithi17557870-27 2024-09-11T03:42:53.838 INFO:tasks.rados.rados.2.smithi175.stdout:1215: finishing write tid 4 to smithi17557870-27 2024-09-11T03:42:53.838 INFO:tasks.rados.rados.2.smithi175.stdout:1217: seq_num 424 ranges {726168=704991,1895071=683474} 2024-09-11T03:42:53.845 INFO:tasks.rados.rados.2.smithi175.stdout:1217: writing smithi17557870-28 from 726168 to 1431159 tid 1 2024-09-11T03:42:53.847 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:53.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:53.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:53.851 INFO:tasks.rados.rados.2.smithi175.stdout:1217: writing smithi17557870-28 from 1895071 to 2578545 tid 2 2024-09-11T03:42:53.851 INFO:tasks.rados.rados.2.smithi175.stdout:1216: done (2 left) 2024-09-11T03:42:53.851 INFO:tasks.rados.rados.2.smithi175.stdout:1218: write (excl) oid 17 current snap is 129 2024-09-11T03:42:53.852 INFO:tasks.rados.rados.2.smithi175.stdout:1218: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2024-09-11T03:42:53.859 INFO:tasks.rados.rados.2.smithi175.stdout:1218: writing smithi17557870-17 from 742975 to 1523215 tid 1 2024-09-11T03:42:53.864 INFO:tasks.rados.rados.2.smithi175.stdout:1218: writing smithi17557870-17 from 1953553 to 2380685 tid 2 2024-09-11T03:42:53.869 INFO:tasks.rados.rados.2.smithi175.stdout:1218: writing smithi17557870-17 from 2971713 to 3142975 tid 3 2024-09-11T03:42:53.869 INFO:tasks.rados.rados.2.smithi175.stdout:1219: write oid 23 current snap is 129 2024-09-11T03:42:53.869 INFO:tasks.rados.rados.2.smithi175.stdout:1215: finishing write tid 5 to smithi17557870-27 2024-09-11T03:42:53.869 INFO:tasks.rados.rados.2.smithi175.stdout:1215: oid 27 updating version 0 to 436 2024-09-11T03:42:53.869 INFO:tasks.rados.rados.2.smithi175.stdout:1215: oid 27 updating version 436 to 439 2024-09-11T03:42:53.869 INFO:tasks.rados.rados.2.smithi175.stdout:1215: oid 27 version 439 is already newer than 437 2024-09-11T03:42:53.869 INFO:tasks.rados.rados.2.smithi175.stdout:1215: oid 27 version 439 is already newer than 438 2024-09-11T03:42:53.869 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 439 (ObjNum 423 snap 129 seq_num 423) dirty exists 2024-09-11T03:42:53.869 INFO:tasks.rados.rados.2.smithi175.stdout:1215: left oid 27 (ObjNum 423 snap 129 seq_num 423) 2024-09-11T03:42:53.869 INFO:tasks.rados.rados.2.smithi175.stdout:1217: finishing write tid 1 to smithi17557870-28 2024-09-11T03:42:53.869 INFO:tasks.rados.rados.2.smithi175.stdout:1217: finishing write tid 2 to smithi17557870-28 2024-09-11T03:42:53.870 INFO:tasks.rados.rados.2.smithi175.stdout:1217: finishing write tid 3 to smithi17557870-28 2024-09-11T03:42:53.870 INFO:tasks.rados.rados.2.smithi175.stdout:1219: seq_num 426 ranges {759782=571842,2012035=687143} 2024-09-11T03:42:53.876 INFO:tasks.rados.rados.2.smithi175.stdout:1219: writing smithi17557870-23 from 759782 to 1331624 tid 1 2024-09-11T03:42:53.884 INFO:tasks.rados.rados.2.smithi175.stdout:1219: writing smithi17557870-23 from 2012035 to 2699178 tid 2 2024-09-11T03:42:53.884 INFO:tasks.rados.rados.2.smithi175.stdout:1215: done (3 left) 2024-09-11T03:42:53.884 INFO:tasks.rados.rados.2.smithi175.stdout:1220: snap_create 2024-09-11T03:42:53.884 INFO:tasks.rados.rados.2.smithi175.stdout:1217: finishing write tid 4 to smithi17557870-28 2024-09-11T03:42:53.884 INFO:tasks.rados.rados.2.smithi175.stdout:1217: oid 28 updating version 0 to 471 2024-09-11T03:42:53.884 INFO:tasks.rados.rados.2.smithi175.stdout:1217: oid 28 updating version 471 to 472 2024-09-11T03:42:53.884 INFO:tasks.rados.rados.2.smithi175.stdout:1217: oid 28 version 472 is already newer than 470 2024-09-11T03:42:53.884 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 472 (ObjNum 424 snap 129 seq_num 424) dirty exists 2024-09-11T03:42:53.884 INFO:tasks.rados.rados.2.smithi175.stdout:1217: left oid 28 (ObjNum 424 snap 129 seq_num 424) 2024-09-11T03:42:53.884 INFO:tasks.rados.rados.2.smithi175.stdout:1218: finishing write tid 1 to smithi17557870-17 2024-09-11T03:42:53.884 INFO:tasks.rados.rados.2.smithi175.stdout:1218: finishing write tid 2 to smithi17557870-17 2024-09-11T03:42:53.884 INFO:tasks.rados.rados.2.smithi175.stdout:1218: finishing write tid 3 to smithi17557870-17 2024-09-11T03:42:53.885 INFO:tasks.rados.rados.2.smithi175.stdout:1218: finishing write tid 4 to smithi17557870-17 2024-09-11T03:42:53.885 INFO:tasks.rados.rados.2.smithi175.stdout:1218: finishing write tid 5 to smithi17557870-17 2024-09-11T03:42:53.885 INFO:tasks.rados.rados.2.smithi175.stdout:1218: oid 17 updating version 0 to 474 2024-09-11T03:42:53.885 INFO:tasks.rados.rados.2.smithi175.stdout:1218: oid 17 updating version 474 to 475 2024-09-11T03:42:53.885 INFO:tasks.rados.rados.2.smithi175.stdout:1218: oid 17 updating version 475 to 476 2024-09-11T03:42:53.885 INFO:tasks.rados.rados.2.smithi175.stdout:1218: oid 17 updating version 476 to 477 2024-09-11T03:42:53.885 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 477 (ObjNum 425 snap 129 seq_num 425) dirty exists 2024-09-11T03:42:53.885 INFO:tasks.rados.rados.2.smithi175.stdout:1218: left oid 17 (ObjNum 425 snap 129 seq_num 425) 2024-09-11T03:42:53.885 INFO:tasks.rados.rados.2.smithi175.stdout:1219: finishing write tid 1 to smithi17557870-23 2024-09-11T03:42:53.886 INFO:tasks.rados.rados.2.smithi175.stdout:1219: finishing write tid 2 to smithi17557870-23 2024-09-11T03:42:53.886 INFO:tasks.rados.rados.2.smithi175.stdout:1219: finishing write tid 3 to smithi17557870-23 2024-09-11T03:42:53.887 INFO:tasks.rados.rados.2.smithi175.stdout:1219: finishing write tid 4 to smithi17557870-23 2024-09-11T03:42:53.887 INFO:tasks.rados.rados.2.smithi175.stdout:1219: oid 23 updating version 0 to 256 2024-09-11T03:42:53.887 INFO:tasks.rados.rados.2.smithi175.stdout:1219: oid 23 version 256 is already newer than 255 2024-09-11T03:42:53.887 INFO:tasks.rados.rados.2.smithi175.stdout:1219: oid 23 version 256 is already newer than 254 2024-09-11T03:42:53.887 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 256 (ObjNum 426 snap 129 seq_num 426) dirty exists 2024-09-11T03:42:53.887 INFO:tasks.rados.rados.2.smithi175.stdout:1219: left oid 23 (ObjNum 426 snap 129 seq_num 426) 2024-09-11T03:42:53.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:53.970 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:53.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:54.021 INFO:tasks.rados.rados.2.smithi175.stdout:1217: done (3 left) 2024-09-11T03:42:54.021 INFO:tasks.rados.rados.2.smithi175.stdout:1218: done (2 left) 2024-09-11T03:42:54.021 INFO:tasks.rados.rados.2.smithi175.stdout:1219: done (1 left) 2024-09-11T03:42:54.021 INFO:tasks.rados.rados.2.smithi175.stdout:1220: done (0 left) 2024-09-11T03:42:54.021 INFO:tasks.rados.rados.2.smithi175.stdout:1221: read oid 12 snap -1 2024-09-11T03:42:54.021 INFO:tasks.rados.rados.2.smithi175.stdout:1221: expect (ObjNum 335 snap 95 seq_num 335) 2024-09-11T03:42:54.022 INFO:tasks.rados.rados.2.smithi175.stdout:1222: rollback oid 38 current snap is 130 2024-09-11T03:42:54.022 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 38 to 129 2024-09-11T03:42:54.022 INFO:tasks.rados.rados.2.smithi175.stdout:1223: snap_remove snap 115 2024-09-11T03:42:54.072 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:54.093 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:54.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:54.094 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:54.195 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:54.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:54.217 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:54.218 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:54.227 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:42:54.227 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:42:54.243 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:42:54.244 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:42:54.244 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:54 smithi084 ceph-mon[27254]: pgmap v688: 57 pgs: 57 active+clean; 190 MiB data, 794 MiB used, 893 GiB / 894 GiB avail; 14 MiB/s rd, 308 KiB/s wr, 12 op/s 2024-09-11T03:42:54.244 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:54 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:42:54.244 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:54 smithi084 ceph-mon[27254]: 2.1 scrub ok 2024-09-11T03:42:54.249 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:54 smithi121 ceph-mon[30729]: pgmap v688: 57 pgs: 57 active+clean; 190 MiB data, 794 MiB used, 893 GiB / 894 GiB avail; 14 MiB/s rd, 308 KiB/s wr, 12 op/s 2024-09-11T03:42:54.249 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:54 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:42:54.249 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:54 smithi121 ceph-mon[30729]: 2.1 scrub ok 2024-09-11T03:42:54.259 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:42:54.259 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:42:54.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T03:42:54.276 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:42:54.276 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:42:54.292 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:42:54.292 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:42:54.308 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:42:54.309 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:42:54.318 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:54.325 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:42:54.325 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:42:54.341 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:54.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:54.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:54.343 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:42:54.343 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:42:54.360 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:42:54.360 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:42:54.376 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:42:54.377 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:42:54.393 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:42:54.393 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:42:54.409 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:42:54.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:42:54.443 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:54.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:54.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:54.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:54 smithi175 ceph-mon[30894]: pgmap v688: 57 pgs: 57 active+clean; 190 MiB data, 794 MiB used, 893 GiB / 894 GiB avail; 14 MiB/s rd, 308 KiB/s wr, 12 op/s 2024-09-11T03:42:54.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:54 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:42:54.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:54 smithi175 ceph-mon[30894]: 2.1 scrub ok 2024-09-11T03:42:54.515 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:42:54.565 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:54.587 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:54.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:54.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:54.689 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:42:54.710 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:54.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:42:54.812 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:54.833 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:54.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:54.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:54.935 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:54.959 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:54.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:54.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:55.008 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:42:55.009 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:42:55.034 INFO:tasks.rados.rados.2.smithi175.stdout:1223: done (2 left) 2024-09-11T03:42:55.035 INFO:tasks.rados.rados.2.smithi175.stdout:1224: rollback oid 31 current snap is 130 2024-09-11T03:42:55.035 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 31 to 127 2024-09-11T03:42:55.035 INFO:tasks.rados.rados.2.smithi175.stdout:1221: expect (ObjNum 335 snap 95 seq_num 335) 2024-09-11T03:42:55.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:55.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:55.083 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:55.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:55.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:55 smithi084 ceph-mon[27254]: osdmap e357: 12 total, 12 up, 10 in 2024-09-11T03:42:55.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:55 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2027577453' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:55.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:55 smithi121 ceph-mon[30729]: osdmap e357: 12 total, 12 up, 10 in 2024-09-11T03:42:55.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:55 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2027577453' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:55.184 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:55.205 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:55.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:55.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:55.252 INFO:tasks.rados.rados.2.smithi175.stdout:1222: finishing rollback tid 0 to smithi17557870-38 2024-09-11T03:42:55.252 INFO:tasks.rados.rados.2.smithi175.stdout:1222: finishing rollback tid 1 to smithi17557870-38 2024-09-11T03:42:55.252 INFO:tasks.rados.rados.2.smithi175.stdout:1222: finishing rollback tid 2 to smithi17557870-38 2024-09-11T03:42:55.252 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 439 (ObjNum 318 snap 86 seq_num 318) dirty exists 2024-09-11T03:42:55.252 INFO:tasks.rados.rados.2.smithi175.stdout:1221: done (2 left) 2024-09-11T03:42:55.252 INFO:tasks.rados.rados.2.smithi175.stdout:1222: done (1 left) 2024-09-11T03:42:55.253 INFO:tasks.rados.rados.2.smithi175.stdout:1225: read oid 10 snap -1 2024-09-11T03:42:55.253 INFO:tasks.rados.rados.2.smithi175.stdout:1225: expect (ObjNum 310 snap 85 seq_num 310) 2024-09-11T03:42:55.253 INFO:tasks.rados.rados.2.smithi175.stdout:1226: read oid 3 snap -1 2024-09-11T03:42:55.253 INFO:tasks.rados.rados.2.smithi175.stdout:1226: expect (ObjNum 384 snap 117 seq_num 384) 2024-09-11T03:42:55.253 INFO:tasks.rados.rados.2.smithi175.stdout:1227: snap_remove snap 129 2024-09-11T03:42:55.306 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:55.327 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:55.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:55.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:55.429 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:55.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:55.452 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:55.452 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:55.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:55 smithi175 ceph-mon[30894]: osdmap e357: 12 total, 12 up, 10 in 2024-09-11T03:42:55.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:55 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2027577453' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:55.553 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:55.576 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:55.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:55.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:55.650 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [6, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:42:55.650 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:42:55.651 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-11T03:42:55.651 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight-by-utilization 1000 0.05 2024-09-11T03:42:55.660 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:42:55.679 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:42:55.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T03:42:55.700 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:55.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:55.701 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:42:55.803 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:55.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:55.826 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:55.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:55.916 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:55.923 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:42:55.927 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:55.948 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:55.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:55.949 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:56.043 INFO:tasks.rados.rados.2.smithi175.stdout:1227: done (3 left) 2024-09-11T03:42:56.043 INFO:tasks.rados.rados.2.smithi175.stdout:1228: rollback oid 41 current snap is 130 2024-09-11T03:42:56.043 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 41 to 127 2024-09-11T03:42:56.043 INFO:tasks.rados.rados.2.smithi175.stdout:1224: finishing rollback tid 0 to smithi17557870-31 2024-09-11T03:42:56.043 INFO:tasks.rados.rados.2.smithi175.stdout:1224: finishing rollback tid 1 to smithi17557870-31 2024-09-11T03:42:56.043 INFO:tasks.rados.rados.2.smithi175.stdout:1224: finishing rollback tid 2 to smithi17557870-31 2024-09-11T03:42:56.043 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 447 (ObjNum 362 snap 107 seq_num 362) dirty exists 2024-09-11T03:42:56.043 INFO:tasks.rados.rados.2.smithi175.stdout:1225: expect (ObjNum 310 snap 85 seq_num 310) 2024-09-11T03:42:56.051 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:56.072 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:56.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:56.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:56.171 INFO:tasks.rados.rados.2.smithi175.stdout:1226: expect (ObjNum 384 snap 117 seq_num 384) 2024-09-11T03:42:56.174 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:56.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:56.195 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:56.195 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:56.284 INFO:tasks.rados.rados.2.smithi175.stdout:1228: finishing rollback tid 0 to smithi17557870-41 2024-09-11T03:42:56.285 INFO:tasks.rados.rados.2.smithi175.stdout:1228: finishing rollback tid 1 to smithi17557870-41 2024-09-11T03:42:56.285 INFO:tasks.rados.rados.2.smithi175.stdout:1224: done (3 left) 2024-09-11T03:42:56.285 INFO:tasks.rados.rados.2.smithi175.stdout:1225: done (2 left) 2024-09-11T03:42:56.285 INFO:tasks.rados.rados.2.smithi175.stdout:1226: done (1 left) 2024-09-11T03:42:56.285 INFO:tasks.rados.rados.2.smithi175.stdout:1229: write oid 18 current snap is 130 2024-09-11T03:42:56.285 INFO:tasks.rados.rados.2.smithi175.stdout:1228: finishing rollback tid 2 to smithi17557870-41 2024-09-11T03:42:56.285 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 356 (ObjNum 413 snap 127 seq_num 413) dirty exists 2024-09-11T03:42:56.285 INFO:tasks.rados.rados.2.smithi175.stdout:1229: seq_num 427 ranges {776589=647091,2070517=430801,3026559=150030} 2024-09-11T03:42:56.291 INFO:tasks.rados.rados.2.smithi175.stdout:1229: writing smithi17557870-18 from 776589 to 1423680 tid 1 2024-09-11T03:42:56.295 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:56 smithi084 ceph-mon[27254]: pgmap v691: 57 pgs: 57 active+clean; 190 MiB data, 783 MiB used, 893 GiB / 894 GiB avail; 14 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-11T03:42:56.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:56 smithi084 ceph-mon[27254]: osdmap e358: 12 total, 12 up, 10 in 2024-09-11T03:42:56.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:56.297 INFO:tasks.rados.rados.2.smithi175.stdout:1229: writing smithi17557870-18 from 2070517 to 2501318 tid 2 2024-09-11T03:42:56.301 INFO:tasks.rados.rados.2.smithi175.stdout:1229: writing smithi17557870-18 from 3026559 to 3176589 tid 3 2024-09-11T03:42:56.301 INFO:tasks.rados.rados.2.smithi175.stdout:1228: done (1 left) 2024-09-11T03:42:56.302 INFO:tasks.rados.rados.2.smithi175.stdout:1230: read oid 47 snap -1 2024-09-11T03:42:56.302 INFO:tasks.rados.rados.2.smithi175.stdout:1230: expect (ObjNum 369 snap 113 seq_num 369) 2024-09-11T03:42:56.302 INFO:tasks.rados.rados.2.smithi175.stdout:1231: read oid 11 snap -1 2024-09-11T03:42:56.302 INFO:tasks.rados.rados.2.smithi175.stdout:1231: expect (ObjNum 371 snap 113 seq_num 371) 2024-09-11T03:42:56.302 INFO:tasks.rados.rados.2.smithi175.stdout:1232: read oid 30 snap -1 2024-09-11T03:42:56.302 INFO:tasks.rados.rados.2.smithi175.stdout:1232: expect (ObjNum 396 snap 123 seq_num 396) 2024-09-11T03:42:56.302 INFO:tasks.rados.rados.2.smithi175.stdout:1233: write (excl) oid 10 current snap is 130 2024-09-11T03:42:56.302 INFO:tasks.rados.rados.2.smithi175.stdout:1229: finishing write tid 1 to smithi17557870-18 2024-09-11T03:42:56.302 INFO:tasks.rados.rados.2.smithi175.stdout:1229: finishing write tid 2 to smithi17557870-18 2024-09-11T03:42:56.302 INFO:tasks.rados.rados.2.smithi175.stdout:1233: seq_num 428 ranges {793396=722340,2128999=690812} 2024-09-11T03:42:56.309 INFO:tasks.rados.rados.2.smithi175.stdout:1233: writing smithi17557870-10 from 793396 to 1515736 tid 1 2024-09-11T03:42:56.317 INFO:tasks.rados.rados.2.smithi175.stdout:1233: writing smithi17557870-10 from 2128999 to 2819811 tid 2 2024-09-11T03:42:56.317 INFO:tasks.rados.rados.2.smithi175.stdout:1234: delete oid 34 current snap is 130 2024-09-11T03:42:56.317 INFO:tasks.rados.rados.2.smithi175.stdout:1229: finishing write tid 3 to smithi17557870-18 2024-09-11T03:42:56.317 INFO:tasks.rados.rados.2.smithi175.stdout:1229: finishing write tid 4 to smithi17557870-18 2024-09-11T03:42:56.317 INFO:tasks.rados.rados.2.smithi175.stdout:1229: finishing write tid 5 to smithi17557870-18 2024-09-11T03:42:56.317 INFO:tasks.rados.rados.2.smithi175.stdout:1229: oid 18 updating version 0 to 246 2024-09-11T03:42:56.317 INFO:tasks.rados.rados.2.smithi175.stdout:1229: oid 18 version 246 is already newer than 245 2024-09-11T03:42:56.318 INFO:tasks.rados.rados.2.smithi175.stdout:1229: oid 18 version 246 is already newer than 244 2024-09-11T03:42:56.318 INFO:tasks.rados.rados.2.smithi175.stdout:1229: oid 18 version 246 is already newer than 243 2024-09-11T03:42:56.318 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 246 (ObjNum 427 snap 130 seq_num 427) dirty exists 2024-09-11T03:42:56.318 INFO:tasks.rados.rados.2.smithi175.stdout:1229: left oid 18 (ObjNum 427 snap 130 seq_num 427) 2024-09-11T03:42:56.318 INFO:tasks.rados.rados.2.smithi175.stdout:1230: expect (ObjNum 369 snap 113 seq_num 369) 2024-09-11T03:42:56.321 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:56.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:56.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:56.342 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:56 smithi121 ceph-mon[30729]: pgmap v691: 57 pgs: 57 active+clean; 190 MiB data, 783 MiB used, 893 GiB / 894 GiB avail; 14 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-11T03:42:56.343 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:56 smithi121 ceph-mon[30729]: osdmap e358: 12 total, 12 up, 10 in 2024-09-11T03:42:56.417 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:56.423 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:56.441 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:42:56.441 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:42:56.445 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:56.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:56.448 INFO:tasks.rados.rados.2.smithi175.stdout:1231: expect (ObjNum 371 snap 113 seq_num 371) 2024-09-11T03:42:56.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:56 smithi175 ceph-mon[30894]: pgmap v691: 57 pgs: 57 active+clean; 190 MiB data, 783 MiB used, 893 GiB / 894 GiB avail; 14 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-09-11T03:42:56.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:56 smithi175 ceph-mon[30894]: osdmap e358: 12 total, 12 up, 10 in 2024-09-11T03:42:56.546 INFO:tasks.rados.rados.2.smithi175.stdout:1233: finishing write tid 1 to smithi17557870-10 2024-09-11T03:42:56.547 INFO:tasks.rados.rados.2.smithi175.stdout:1232: expect (ObjNum 396 snap 123 seq_num 396) 2024-09-11T03:42:56.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:56.568 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:56.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:56.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:56.579 INFO:teuthology.orchestra.run.smithi084.stdout:moved 0 / 171 (0%) 2024-09-11T03:42:56.579 INFO:teuthology.orchestra.run.smithi084.stdout:avg 17.1 2024-09-11T03:42:56.579 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 2.62488 -> 2.62488 (expected baseline 3.92301) 2024-09-11T03:42:56.579 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.2 with 14 -> 14 pgs (0.818713 -> 0.818713 * mean) 2024-09-11T03:42:56.579 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.7 with 22 -> 22 pgs (1.28655 -> 1.28655 * mean) 2024-09-11T03:42:56.580 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:42:56.580 INFO:teuthology.orchestra.run.smithi084.stdout:oload 1000 2024-09-11T03:42:56.580 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 0.05 2024-09-11T03:42:56.580 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2024-09-11T03:42:56.580 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0008 2024-09-11T03:42:56.580 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0083 2024-09-11T03:42:56.580 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2024-09-11T03:42:56.629 INFO:tasks.rados.rados.2.smithi175.stdout:1233: finishing write tid 2 to smithi17557870-10 2024-09-11T03:42:56.629 INFO:tasks.rados.rados.2.smithi175.stdout:1233: finishing write tid 3 to smithi17557870-10 2024-09-11T03:42:56.629 INFO:tasks.rados.rados.2.smithi175.stdout:1229: done (5 left) 2024-09-11T03:42:56.629 INFO:tasks.rados.rados.2.smithi175.stdout:1230: done (4 left) 2024-09-11T03:42:56.629 INFO:tasks.rados.rados.2.smithi175.stdout:1231: done (3 left) 2024-09-11T03:42:56.629 INFO:tasks.rados.rados.2.smithi175.stdout:1232: done (2 left) 2024-09-11T03:42:56.629 INFO:tasks.rados.rados.2.smithi175.stdout:1234: done (1 left) 2024-09-11T03:42:56.629 INFO:tasks.rados.rados.2.smithi175.stdout:1235: write (excl) oid 40 current snap is 130 2024-09-11T03:42:56.630 INFO:tasks.rados.rados.2.smithi175.stdout:1233: finishing write tid 4 to smithi17557870-10 2024-09-11T03:42:56.630 INFO:tasks.rados.rados.2.smithi175.stdout:1233: oid 10 updating version 0 to 484 2024-09-11T03:42:56.630 INFO:tasks.rados.rados.2.smithi175.stdout:1233: oid 10 version 484 is already newer than 483 2024-09-11T03:42:56.630 INFO:tasks.rados.rados.2.smithi175.stdout:1233: oid 10 updating version 484 to 485 2024-09-11T03:42:56.630 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 485 (ObjNum 428 snap 130 seq_num 428) dirty exists 2024-09-11T03:42:56.630 INFO:tasks.rados.rados.2.smithi175.stdout:1233: left oid 10 (ObjNum 428 snap 130 seq_num 428) 2024-09-11T03:42:56.630 INFO:tasks.rados.rados.2.smithi175.stdout:1235: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2024-09-11T03:42:56.633 INFO:tasks.rados.rados.2.smithi175.stdout:1235: writing smithi17557870-40 from 427010 to 899848 tid 1 2024-09-11T03:42:56.639 INFO:tasks.rados.rados.2.smithi175.stdout:1235: writing smithi17557870-40 from 1329610 to 2024091 tid 2 2024-09-11T03:42:56.646 INFO:tasks.rados.rados.2.smithi175.stdout:1235: writing smithi17557870-40 from 2708828 to 3227010 tid 3 2024-09-11T03:42:56.646 INFO:tasks.rados.rados.2.smithi175.stdout:1233: done (1 left) 2024-09-11T03:42:56.646 INFO:tasks.rados.rados.2.smithi175.stdout:1236: rollback oid 25 current snap is 130 2024-09-11T03:42:56.647 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 25 to 127 2024-09-11T03:42:56.647 INFO:tasks.rados.rados.2.smithi175.stdout:1235: finishing write tid 1 to smithi17557870-40 2024-09-11T03:42:56.647 INFO:tasks.rados.rados.2.smithi175.stdout:1235: finishing write tid 2 to smithi17557870-40 2024-09-11T03:42:56.647 INFO:tasks.rados.rados.2.smithi175.stdout:1237: rollback oid 31 current snap is 130 2024-09-11T03:42:56.647 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 31 to 127 2024-09-11T03:42:56.647 INFO:tasks.rados.rados.2.smithi175.stdout:1238: rollback oid 19 current snap is 130 2024-09-11T03:42:56.647 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 19 to 127 2024-09-11T03:42:56.647 INFO:tasks.rados.rados.2.smithi175.stdout:1239: write oid 46 current snap is 130 2024-09-11T03:42:56.647 INFO:tasks.rados.rados.2.smithi175.stdout:1239: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2024-09-11T03:42:56.651 INFO:tasks.rados.rados.2.smithi175.stdout:1239: writing smithi17557870-46 from 443817 to 991904 tid 1 2024-09-11T03:42:56.658 INFO:tasks.rados.rados.2.smithi175.stdout:1239: writing smithi17557870-46 from 1788092 to 2226231 tid 2 2024-09-11T03:42:56.664 INFO:tasks.rados.rados.2.smithi175.stdout:1239: writing smithi17557870-46 from 3019898 to 3243817 tid 3 2024-09-11T03:42:56.664 INFO:tasks.rados.rados.2.smithi175.stdout:1240: read oid 47 snap -1 2024-09-11T03:42:56.664 INFO:tasks.rados.rados.2.smithi175.stdout:1240: expect (ObjNum 369 snap 113 seq_num 369) 2024-09-11T03:42:56.664 INFO:tasks.rados.rados.2.smithi175.stdout:1241: read oid 43 snap -1 2024-09-11T03:42:56.664 INFO:tasks.rados.rados.2.smithi175.stdout:1241: expect (ObjNum 412 snap 127 seq_num 412) 2024-09-11T03:42:56.665 INFO:tasks.rados.rados.2.smithi175.stdout:1242: delete oid 6 current snap is 130 2024-09-11T03:42:56.665 INFO:tasks.rados.rados.2.smithi175.stdout:1238: finishing rollback tid 1 to smithi17557870-19 2024-09-11T03:42:56.665 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T03:42:56.665 INFO:tasks.rados.rados.2.smithi175.stdout:1236: finishing rollback tid 0 to smithi17557870-25 2024-09-11T03:42:56.665 INFO:tasks.rados.rados.2.smithi175.stdout:1237: finishing rollback tid 0 to smithi17557870-31 2024-09-11T03:42:56.665 INFO:tasks.rados.rados.2.smithi175.stdout:1236: finishing rollback tid 1 to smithi17557870-25 2024-09-11T03:42:56.665 INFO:tasks.rados.rados.2.smithi175.stdout:1237: finishing rollback tid 1 to smithi17557870-31 2024-09-11T03:42:56.665 INFO:tasks.rados.rados.2.smithi175.stdout:1235: finishing write tid 3 to smithi17557870-40 2024-09-11T03:42:56.665 INFO:tasks.rados.rados.2.smithi175.stdout:1236: finishing rollback tid 2 to smithi17557870-25 2024-09-11T03:42:56.665 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 228 (ObjNum 344 snap 99 seq_num 344) dirty exists 2024-09-11T03:42:56.665 INFO:tasks.rados.rados.2.smithi175.stdout:1237: finishing rollback tid 2 to smithi17557870-31 2024-09-11T03:42:56.665 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 452 (ObjNum 362 snap 107 seq_num 362) dirty exists 2024-09-11T03:42:56.666 INFO:tasks.rados.rados.2.smithi175.stdout:1235: finishing write tid 4 to smithi17557870-40 2024-09-11T03:42:56.666 INFO:tasks.rados.rados.2.smithi175.stdout:1235: finishing write tid 5 to smithi17557870-40 2024-09-11T03:42:56.666 INFO:tasks.rados.rados.2.smithi175.stdout:1235: oid 40 updating version 0 to 378 2024-09-11T03:42:56.666 INFO:tasks.rados.rados.2.smithi175.stdout:1235: oid 40 version 378 is already newer than 376 2024-09-11T03:42:56.666 INFO:tasks.rados.rados.2.smithi175.stdout:1235: oid 40 version 378 is already newer than 377 2024-09-11T03:42:56.666 INFO:tasks.rados.rados.2.smithi175.stdout:1235: oid 40 version 378 is already newer than 375 2024-09-11T03:42:56.666 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 378 (ObjNum 430 snap 130 seq_num 430) dirty exists 2024-09-11T03:42:56.666 INFO:tasks.rados.rados.2.smithi175.stdout:1235: left oid 40 (ObjNum 430 snap 130 seq_num 430) 2024-09-11T03:42:56.666 INFO:tasks.rados.rados.2.smithi175.stdout:1239: finishing write tid 1 to smithi17557870-46 2024-09-11T03:42:56.666 INFO:tasks.rados.rados.2.smithi175.stdout:1239: finishing write tid 2 to smithi17557870-46 2024-09-11T03:42:56.667 INFO:tasks.rados.rados.2.smithi175.stdout:1235: done (7 left) 2024-09-11T03:42:56.667 INFO:tasks.rados.rados.2.smithi175.stdout:1236: done (6 left) 2024-09-11T03:42:56.667 INFO:tasks.rados.rados.2.smithi175.stdout:1237: done (5 left) 2024-09-11T03:42:56.667 INFO:tasks.rados.rados.2.smithi175.stdout:1238: done (4 left) 2024-09-11T03:42:56.667 INFO:tasks.rados.rados.2.smithi175.stdout:1242: done (3 left) 2024-09-11T03:42:56.667 INFO:tasks.rados.rados.2.smithi175.stdout:1243: read oid 22 snap 127 2024-09-11T03:42:56.668 INFO:tasks.rados.rados.2.smithi175.stdout:1243: expect (ObjNum 342 snap 99 seq_num 342) 2024-09-11T03:42:56.668 INFO:tasks.rados.rados.2.smithi175.stdout:1244: delete oid 1 current snap is 130 2024-09-11T03:42:56.668 INFO:tasks.rados.rados.2.smithi175.stdout:1244: done (4 left) 2024-09-11T03:42:56.668 INFO:tasks.rados.rados.2.smithi175.stdout:1245: read oid 34 snap -1 2024-09-11T03:42:56.669 INFO:tasks.rados.rados.2.smithi175.stdout:1245: expect deleted 2024-09-11T03:42:56.669 INFO:tasks.rados.rados.2.smithi175.stdout:1246: write oid 48 current snap is 130 2024-09-11T03:42:56.669 INFO:tasks.rados.rados.2.smithi175.stdout:1246: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2024-09-11T03:42:56.670 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:56.674 INFO:tasks.rados.rados.2.smithi175.stdout:1246: writing smithi17557870-48 from 494238 to 984425 tid 1 2024-09-11T03:42:56.681 INFO:tasks.rados.rados.2.smithi175.stdout:1246: writing smithi17557870-48 from 1563538 to 2265357 tid 2 2024-09-11T03:42:56.687 INFO:tasks.rados.rados.2.smithi175.stdout:1246: writing smithi17557870-48 from 2818520 to 3294238 tid 3 2024-09-11T03:42:56.687 INFO:tasks.rados.rados.2.smithi175.stdout:1247: write (excl) oid 30 current snap is 130 2024-09-11T03:42:56.687 INFO:tasks.rados.rados.2.smithi175.stdout:1239: finishing write tid 3 to smithi17557870-46 2024-09-11T03:42:56.687 INFO:tasks.rados.rados.2.smithi175.stdout:1247: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2024-09-11T03:42:56.692 INFO:tasks.rados.rados.2.smithi175.stdout:1247: writing smithi17557870-30 from 511045 to 1076481 tid 1 2024-09-11T03:42:56.695 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:56.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:56.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:56.698 INFO:tasks.rados.rados.2.smithi175.stdout:1247: writing smithi17557870-30 from 1622020 to 2067497 tid 2 2024-09-11T03:42:56.706 INFO:tasks.rados.rados.2.smithi175.stdout:1247: writing smithi17557870-30 from 2729590 to 3286445 tid 3 2024-09-11T03:42:56.706 INFO:tasks.rados.rados.2.smithi175.stdout:1248: read oid 9 snap 127 2024-09-11T03:42:56.706 INFO:tasks.rados.rados.2.smithi175.stdout:1248: expect (ObjNum 277 snap 73 seq_num 277) 2024-09-11T03:42:56.706 INFO:tasks.rados.rados.2.smithi175.stdout:1249: read oid 15 snap 127 2024-09-11T03:42:56.706 INFO:tasks.rados.rados.2.smithi175.stdout:1249: expect (ObjNum 390 snap 121 seq_num 390) 2024-09-11T03:42:56.706 INFO:tasks.rados.rados.2.smithi175.stdout:1250: read oid 31 snap -1 2024-09-11T03:42:56.706 INFO:tasks.rados.rados.2.smithi175.stdout:1250: expect (ObjNum 362 snap 107 seq_num 362) 2024-09-11T03:42:56.706 INFO:tasks.rados.rados.2.smithi175.stdout:1251: delete oid 49 current snap is 130 2024-09-11T03:42:56.706 INFO:tasks.rados.rados.2.smithi175.stdout:1239: finishing write tid 4 to smithi17557870-46 2024-09-11T03:42:56.706 INFO:tasks.rados.rados.2.smithi175.stdout:1239: finishing write tid 5 to smithi17557870-46 2024-09-11T03:42:56.707 INFO:tasks.rados.rados.2.smithi175.stdout:1239: oid 46 updating version 0 to 175 2024-09-11T03:42:56.707 INFO:tasks.rados.rados.2.smithi175.stdout:1239: oid 46 updating version 175 to 178 2024-09-11T03:42:56.707 INFO:tasks.rados.rados.2.smithi175.stdout:1239: oid 46 version 178 is already newer than 177 2024-09-11T03:42:56.707 INFO:tasks.rados.rados.2.smithi175.stdout:1239: oid 46 version 178 is already newer than 176 2024-09-11T03:42:56.707 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 178 (ObjNum 431 snap 130 seq_num 431) dirty exists 2024-09-11T03:42:56.707 INFO:tasks.rados.rados.2.smithi175.stdout:1239: left oid 46 (ObjNum 431 snap 130 seq_num 431) 2024-09-11T03:42:56.707 INFO:tasks.rados.rados.2.smithi175.stdout:1240: expect (ObjNum 369 snap 113 seq_num 369) 2024-09-11T03:42:56.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:56.819 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:56.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:56.837 INFO:tasks.rados.rados.2.smithi175.stdout:1246: finishing write tid 1 to smithi17557870-48 2024-09-11T03:42:56.837 INFO:tasks.rados.rados.2.smithi175.stdout:1243: expect (ObjNum 342 snap 99 seq_num 342) 2024-09-11T03:42:56.920 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:42:56.942 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:56.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:56.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:42:57.002 INFO:tasks.rados.rados.2.smithi175.stdout:1241: expect (ObjNum 412 snap 127 seq_num 412) 2024-09-11T03:42:57.043 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:57.064 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:57.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:57.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:57.080 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:57 smithi121 ceph-mon[30729]: osdmap e359: 12 total, 12 up, 10 in 2024-09-11T03:42:57.080 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:57 smithi121 ceph-mon[30729]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:42:57.080 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:57 smithi121 ceph-mon[30729]: osdmap e360: 12 total, 12 up, 10 in 2024-09-11T03:42:57.097 INFO:tasks.rados.rados.2.smithi175.stdout:1246: finishing write tid 2 to smithi17557870-48 2024-09-11T03:42:57.097 INFO:tasks.rados.rados.2.smithi175.stdout:1246: finishing write tid 3 to smithi17557870-48 2024-09-11T03:42:57.097 INFO:tasks.rados.rados.2.smithi175.stdout:1246: finishing write tid 4 to smithi17557870-48 2024-09-11T03:42:57.097 INFO:tasks.rados.rados.2.smithi175.stdout:1246: finishing write tid 5 to smithi17557870-48 2024-09-11T03:42:57.097 INFO:tasks.rados.rados.2.smithi175.stdout:1246: oid 48 updating version 0 to 443 2024-09-11T03:42:57.097 INFO:tasks.rados.rados.2.smithi175.stdout:1246: oid 48 updating version 443 to 445 2024-09-11T03:42:57.097 INFO:tasks.rados.rados.2.smithi175.stdout:1246: oid 48 version 445 is already newer than 444 2024-09-11T03:42:57.097 INFO:tasks.rados.rados.2.smithi175.stdout:1246: oid 48 updating version 445 to 446 2024-09-11T03:42:57.097 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 446 (ObjNum 434 snap 130 seq_num 434) dirty exists 2024-09-11T03:42:57.097 INFO:tasks.rados.rados.2.smithi175.stdout:1246: left oid 48 (ObjNum 434 snap 130 seq_num 434) 2024-09-11T03:42:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:1247: finishing write tid 1 to smithi17557870-30 2024-09-11T03:42:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:1239: done (10 left) 2024-09-11T03:42:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:1240: done (9 left) 2024-09-11T03:42:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:1241: done (8 left) 2024-09-11T03:42:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:1243: done (7 left) 2024-09-11T03:42:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:1245: done (6 left) 2024-09-11T03:42:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:1246: done (5 left) 2024-09-11T03:42:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:1251: done (4 left) 2024-09-11T03:42:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:1252: rollback oid 21 current snap is 130 2024-09-11T03:42:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 21 to 127 2024-09-11T03:42:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:1247: finishing write tid 2 to smithi17557870-30 2024-09-11T03:42:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:1249: expect (ObjNum 390 snap 121 seq_num 390) 2024-09-11T03:42:57.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T03:42:57.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:57.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:42:57.188 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:57.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:57.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:57.253 INFO:tasks.rados.rados.2.smithi175.stdout:1248: expect (ObjNum 277 snap 73 seq_num 277) 2024-09-11T03:42:57.289 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:57.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:57.311 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:57.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:57.340 INFO:tasks.rados.rados.2.smithi175.stdout:1247: finishing write tid 3 to smithi17557870-30 2024-09-11T03:42:57.341 INFO:tasks.rados.rados.2.smithi175.stdout:1247: finishing write tid 4 to smithi17557870-30 2024-09-11T03:42:57.341 INFO:tasks.rados.rados.2.smithi175.stdout:1247: finishing write tid 5 to smithi17557870-30 2024-09-11T03:42:57.341 INFO:tasks.rados.rados.2.smithi175.stdout:1247: oid 30 updating version 0 to 491 2024-09-11T03:42:57.341 INFO:tasks.rados.rados.2.smithi175.stdout:1247: oid 30 updating version 491 to 492 2024-09-11T03:42:57.341 INFO:tasks.rados.rados.2.smithi175.stdout:1247: oid 30 updating version 492 to 493 2024-09-11T03:42:57.341 INFO:tasks.rados.rados.2.smithi175.stdout:1247: oid 30 version 493 is already newer than 490 2024-09-11T03:42:57.341 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 493 (ObjNum 435 snap 130 seq_num 435) dirty exists 2024-09-11T03:42:57.341 INFO:tasks.rados.rados.2.smithi175.stdout:1247: left oid 30 (ObjNum 435 snap 130 seq_num 435) 2024-09-11T03:42:57.341 INFO:tasks.rados.rados.2.smithi175.stdout:1250: expect (ObjNum 362 snap 107 seq_num 362) 2024-09-11T03:42:57.350 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:42:57.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:57 smithi084 ceph-mon[27254]: osdmap e359: 12 total, 12 up, 10 in 2024-09-11T03:42:57.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:57 smithi084 ceph-mon[27254]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:42:57.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:57 smithi084 ceph-mon[27254]: osdmap e360: 12 total, 12 up, 10 in 2024-09-11T03:42:57.388 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-11T03:42:57.388 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-09-11T03:42:57.412 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:57.453 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:57.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:57.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:57 smithi175 ceph-mon[30894]: osdmap e359: 12 total, 12 up, 10 in 2024-09-11T03:42:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:57 smithi175 ceph-mon[30894]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:42:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:57 smithi175 ceph-mon[30894]: osdmap e360: 12 total, 12 up, 10 in 2024-09-11T03:42:57.482 INFO:tasks.rados.rados.2.smithi175.stdout:1252: finishing rollback tid 0 to smithi17557870-21 2024-09-11T03:42:57.483 INFO:tasks.rados.rados.2.smithi175.stdout:1252: finishing rollback tid 1 to smithi17557870-21 2024-09-11T03:42:57.483 INFO:tasks.rados.rados.2.smithi175.stdout:1252: finishing rollback tid 2 to smithi17557870-21 2024-09-11T03:42:57.483 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 456 (ObjNum 266 snap 67 seq_num 266) dirty exists 2024-09-11T03:42:57.483 INFO:tasks.rados.rados.2.smithi175.stdout:1247: done (4 left) 2024-09-11T03:42:57.483 INFO:tasks.rados.rados.2.smithi175.stdout:1248: done (3 left) 2024-09-11T03:42:57.483 INFO:tasks.rados.rados.2.smithi175.stdout:1249: done (2 left) 2024-09-11T03:42:57.483 INFO:tasks.rados.rados.2.smithi175.stdout:1250: done (1 left) 2024-09-11T03:42:57.483 INFO:tasks.rados.rados.2.smithi175.stdout:1252: done (0 left) 2024-09-11T03:42:57.483 INFO:tasks.rados.rados.2.smithi175.stdout:1253: read oid 38 snap -1 2024-09-11T03:42:57.483 INFO:tasks.rados.rados.2.smithi175.stdout:1253: expect (ObjNum 318 snap 86 seq_num 318) 2024-09-11T03:42:57.483 INFO:tasks.rados.rados.2.smithi175.stdout:1254: read oid 42 snap -1 2024-09-11T03:42:57.483 INFO:tasks.rados.rados.2.smithi175.stdout:1254: expect (ObjNum 409 snap 127 seq_num 409) 2024-09-11T03:42:57.483 INFO:tasks.rados.rados.2.smithi175.stdout:1255: copy_from oid 33 from oid 8 current snap is 130 2024-09-11T03:42:57.484 INFO:tasks.rados.rados.2.smithi175.stdout:1256: write oid 28 current snap is 130 2024-09-11T03:42:57.484 INFO:tasks.rados.rados.2.smithi175.stdout:1256: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2024-09-11T03:42:57.489 INFO:tasks.rados.rados.2.smithi175.stdout:1256: writing smithi17557870-28 from 544659 to 1260593 tid 1 2024-09-11T03:42:57.494 INFO:tasks.rados.rados.2.smithi175.stdout:1256: writing smithi17557870-28 from 1738984 to 2188130 tid 2 2024-09-11T03:42:57.500 INFO:tasks.rados.rados.2.smithi175.stdout:1256: writing smithi17557870-28 from 2784436 to 3344659 tid 3 2024-09-11T03:42:57.501 INFO:tasks.rados.rados.2.smithi175.stdout:1257: delete oid 31 current snap is 130 2024-09-11T03:42:57.501 INFO:tasks.rados.rados.2.smithi175.stdout:1253: expect (ObjNum 318 snap 86 seq_num 318) 2024-09-11T03:42:57.555 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:57.576 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:57.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:57.577 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:57.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:57.685 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:57.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:57.701 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:57.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:57.714 INFO:tasks.rados.rados.2.smithi175.stdout:1254: expect (ObjNum 409 snap 127 seq_num 409) 2024-09-11T03:42:57.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:42:57.823 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:57.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:57.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:42:57.825 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:42:57.825 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:42:57.869 INFO:tasks.rados.rados.2.smithi175.stdout:1255: finishing copy_from racing read to smithi17557870-33 2024-09-11T03:42:57.869 INFO:tasks.rados.rados.2.smithi175.stdout:1255: finishing copy_from to smithi17557870-33 2024-09-11T03:42:57.869 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 468 (ObjNum 415 snap 127 seq_num 415) dirty exists 2024-09-11T03:42:57.869 INFO:tasks.rados.rados.2.smithi175.stdout:1256: finishing write tid 1 to smithi17557870-28 2024-09-11T03:42:57.869 INFO:tasks.rados.rados.2.smithi175.stdout:1256: finishing write tid 2 to smithi17557870-28 2024-09-11T03:42:57.869 INFO:tasks.rados.rados.2.smithi175.stdout:1253: done (4 left) 2024-09-11T03:42:57.869 INFO:tasks.rados.rados.2.smithi175.stdout:1254: done (3 left) 2024-09-11T03:42:57.869 INFO:tasks.rados.rados.2.smithi175.stdout:1255: done (2 left) 2024-09-11T03:42:57.870 INFO:tasks.rados.rados.2.smithi175.stdout:1257: done (1 left) 2024-09-11T03:42:57.870 INFO:tasks.rados.rados.2.smithi175.stdout:1258: snap_remove snap 127 2024-09-11T03:42:57.925 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:57.947 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:57.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:57.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:58.046 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:42:58.049 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:58.054 INFO:tasks.rados.rados.2.smithi175.stdout:1258: done (1 left) 2024-09-11T03:42:58.054 INFO:tasks.rados.rados.2.smithi175.stdout:1259: delete oid 29 current snap is 130 2024-09-11T03:42:58.054 INFO:tasks.rados.rados.2.smithi175.stdout:1256: finishing write tid 3 to smithi17557870-28 2024-09-11T03:42:58.055 INFO:tasks.rados.rados.2.smithi175.stdout:1256: finishing write tid 4 to smithi17557870-28 2024-09-11T03:42:58.055 INFO:tasks.rados.rados.2.smithi175.stdout:1256: finishing write tid 5 to smithi17557870-28 2024-09-11T03:42:58.055 INFO:tasks.rados.rados.2.smithi175.stdout:1256: oid 28 updating version 0 to 483 2024-09-11T03:42:58.055 INFO:tasks.rados.rados.2.smithi175.stdout:1256: oid 28 version 483 is already newer than 482 2024-09-11T03:42:58.055 INFO:tasks.rados.rados.2.smithi175.stdout:1256: oid 28 version 483 is already newer than 481 2024-09-11T03:42:58.055 INFO:tasks.rados.rados.2.smithi175.stdout:1256: oid 28 version 483 is already newer than 480 2024-09-11T03:42:58.055 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 483 (ObjNum 437 snap 130 seq_num 437) dirty exists 2024-09-11T03:42:58.055 INFO:tasks.rados.rados.2.smithi175.stdout:1256: left oid 28 (ObjNum 437 snap 130 seq_num 437) 2024-09-11T03:42:58.056 INFO:tasks.rados.rados.2.smithi175.stdout:1256: done (1 left) 2024-09-11T03:42:58.056 INFO:tasks.rados.rados.2.smithi175.stdout:1259: done (0 left) 2024-09-11T03:42:58.056 INFO:tasks.rados.rados.2.smithi175.stdout:1260: rollback oid 2 current snap is 130 2024-09-11T03:42:58.056 INFO:tasks.rados.rados.2.smithi175.stdout:1260: done (0 left) 2024-09-11T03:42:58.056 INFO:tasks.rados.rados.2.smithi175.stdout:1261: write oid 12 current snap is 130 2024-09-11T03:42:58.056 INFO:tasks.rados.rados.2.smithi175.stdout:1261: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2024-09-11T03:42:58.062 INFO:tasks.rados.rados.2.smithi175.stdout:1261: writing smithi17557870-12 from 595080 to 1136761 tid 1 2024-09-11T03:42:58.069 INFO:tasks.rados.rados.2.smithi175.stdout:1261: writing smithi17557870-12 from 1798077 to 2394550 tid 2 2024-09-11T03:42:58.072 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:58.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:58.073 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:58.075 INFO:tasks.rados.rados.2.smithi175.stdout:1261: writing smithi17557870-12 from 3150352 to 3395080 tid 3 2024-09-11T03:42:58.075 INFO:tasks.rados.rados.2.smithi175.stdout:1262: delete oid 13 current snap is 130 2024-09-11T03:42:58.075 INFO:tasks.rados.rados.2.smithi175.stdout:1261: finishing write tid 1 to smithi17557870-12 2024-09-11T03:42:58.075 INFO:tasks.rados.rados.2.smithi175.stdout:1261: finishing write tid 2 to smithi17557870-12 2024-09-11T03:42:58.076 INFO:tasks.rados.rados.2.smithi175.stdout:1261: finishing write tid 3 to smithi17557870-12 2024-09-11T03:42:58.077 INFO:tasks.rados.rados.2.smithi175.stdout:1261: finishing write tid 4 to smithi17557870-12 2024-09-11T03:42:58.078 INFO:tasks.rados.rados.2.smithi175.stdout:1262: done (1 left) 2024-09-11T03:42:58.078 INFO:tasks.rados.rados.2.smithi175.stdout:1263: delete oid 28 current snap is 130 2024-09-11T03:42:58.078 INFO:tasks.rados.rados.2.smithi175.stdout:1261: finishing write tid 5 to smithi17557870-12 2024-09-11T03:42:58.078 INFO:tasks.rados.rados.2.smithi175.stdout:1261: oid 12 updating version 0 to 453 2024-09-11T03:42:58.078 INFO:tasks.rados.rados.2.smithi175.stdout:1261: oid 12 updating version 453 to 456 2024-09-11T03:42:58.078 INFO:tasks.rados.rados.2.smithi175.stdout:1261: oid 12 version 456 is already newer than 455 2024-09-11T03:42:58.078 INFO:tasks.rados.rados.2.smithi175.stdout:1261: oid 12 version 456 is already newer than 454 2024-09-11T03:42:58.078 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 456 (ObjNum 440 snap 130 seq_num 440) dirty exists 2024-09-11T03:42:58.078 INFO:tasks.rados.rados.2.smithi175.stdout:1261: left oid 12 (ObjNum 440 snap 130 seq_num 440) 2024-09-11T03:42:58.079 INFO:tasks.rados.rados.2.smithi175.stdout:1261: done (1 left) 2024-09-11T03:42:58.079 INFO:tasks.rados.rados.2.smithi175.stdout:1263: done (0 left) 2024-09-11T03:42:58.079 INFO:tasks.rados.rados.2.smithi175.stdout:1264: write (excl) oid 50 current snap is 130 2024-09-11T03:42:58.079 INFO:tasks.rados.rados.2.smithi175.stdout:1264: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2024-09-11T03:42:58.085 INFO:tasks.rados.rados.2.smithi175.stdout:1264: writing smithi17557870-50 from 645501 to 1129282 tid 1 2024-09-11T03:42:58.089 INFO:tasks.rados.rados.2.smithi175.stdout:1264: writing smithi17557870-50 from 1573523 to 2033676 tid 2 2024-09-11T03:42:58.097 INFO:tasks.rados.rados.2.smithi175.stdout:1264: writing smithi17557870-50 from 2548974 to 3312358 tid 3 2024-09-11T03:42:58.097 INFO:tasks.rados.rados.2.smithi175.stdout:1265: copy_from oid 31 from oid 39 current snap is 130 2024-09-11T03:42:58.097 INFO:tasks.rados.rados.2.smithi175.stdout:1264: finishing write tid 1 to smithi17557870-50 2024-09-11T03:42:58.097 INFO:tasks.rados.rados.2.smithi175.stdout:1264: finishing write tid 2 to smithi17557870-50 2024-09-11T03:42:58.097 INFO:tasks.rados.rados.2.smithi175.stdout:1266: rollback oid 10 current snap is 130 2024-09-11T03:42:58.097 INFO:tasks.rados.rados.2.smithi175.stdout:1266: done (2 left) 2024-09-11T03:42:58.097 INFO:tasks.rados.rados.2.smithi175.stdout:1267: read oid 13 snap -1 2024-09-11T03:42:58.098 INFO:tasks.rados.rados.2.smithi175.stdout:1267: expect deleted 2024-09-11T03:42:58.098 INFO:tasks.rados.rados.2.smithi175.stdout:1268: write oid 36 current snap is 130 2024-09-11T03:42:58.098 INFO:tasks.rados.rados.2.smithi175.stdout:1268: seq_num 444 ranges {662308=559030,1632005=603811,2860044=602264} 2024-09-11T03:42:58.103 INFO:tasks.rados.rados.2.smithi175.stdout:1268: writing smithi17557870-36 from 662308 to 1221338 tid 1 2024-09-11T03:42:58.109 INFO:tasks.rados.rados.2.smithi175.stdout:1268: writing smithi17557870-36 from 1632005 to 2235816 tid 2 2024-09-11T03:42:58.116 INFO:tasks.rados.rados.2.smithi175.stdout:1268: writing smithi17557870-36 from 2860044 to 3462308 tid 3 2024-09-11T03:42:58.116 INFO:tasks.rados.rados.2.smithi175.stdout:1269: write oid 49 current snap is 130 2024-09-11T03:42:58.116 INFO:tasks.rados.rados.2.smithi175.stdout:1269: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2024-09-11T03:42:58.123 INFO:tasks.rados.rados.2.smithi175.stdout:1269: writing smithi17557870-49 from 679115 to 1313394 tid 1 2024-09-11T03:42:58.129 INFO:tasks.rados.rados.2.smithi175.stdout:1269: writing smithi17557870-49 from 2090487 to 2554309 tid 2 2024-09-11T03:42:58.137 INFO:tasks.rados.rados.2.smithi175.stdout:1269: writing smithi17557870-49 from 3003820 to 3479115 tid 3 2024-09-11T03:42:58.137 INFO:tasks.rados.rados.2.smithi175.stdout:1270: copy_from oid 7 from oid 12 current snap is 130 2024-09-11T03:42:58.137 INFO:tasks.rados.rados.2.smithi175.stdout:1264: finishing write tid 3 to smithi17557870-50 2024-09-11T03:42:58.137 INFO:tasks.rados.rados.2.smithi175.stdout:1264: finishing write tid 4 to smithi17557870-50 2024-09-11T03:42:58.137 INFO:tasks.rados.rados.2.smithi175.stdout:1264: finishing write tid 5 to smithi17557870-50 2024-09-11T03:42:58.138 INFO:tasks.rados.rados.2.smithi175.stdout:1264: oid 50 updating version 0 to 489 2024-09-11T03:42:58.138 INFO:tasks.rados.rados.2.smithi175.stdout:1264: oid 50 version 489 is already newer than 487 2024-09-11T03:42:58.138 INFO:tasks.rados.rados.2.smithi175.stdout:1264: oid 50 updating version 489 to 490 2024-09-11T03:42:58.138 INFO:tasks.rados.rados.2.smithi175.stdout:1264: oid 50 version 490 is already newer than 488 2024-09-11T03:42:58.138 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 490 (ObjNum 443 snap 130 seq_num 443) dirty exists 2024-09-11T03:42:58.138 INFO:tasks.rados.rados.2.smithi175.stdout:1264: left oid 50 (ObjNum 443 snap 130 seq_num 443) 2024-09-11T03:42:58.138 INFO:tasks.rados.rados.2.smithi175.stdout:1265: finishing copy_from racing read to smithi17557870-31 2024-09-11T03:42:58.138 INFO:tasks.rados.rados.2.smithi175.stdout:1265: finishing copy_from to smithi17557870-31 2024-09-11T03:42:58.138 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 311 (ObjNum 388 snap 117 seq_num 388) dirty exists 2024-09-11T03:42:58.138 INFO:tasks.rados.rados.2.smithi175.stdout:1268: finishing write tid 1 to smithi17557870-36 2024-09-11T03:42:58.139 INFO:tasks.rados.rados.2.smithi175.stdout:1268: finishing write tid 2 to smithi17557870-36 2024-09-11T03:42:58.139 INFO:tasks.rados.rados.2.smithi175.stdout:1268: finishing write tid 3 to smithi17557870-36 2024-09-11T03:42:58.139 INFO:tasks.rados.rados.2.smithi175.stdout:1268: finishing write tid 4 to smithi17557870-36 2024-09-11T03:42:58.139 INFO:tasks.rados.rados.2.smithi175.stdout:1264: done (5 left) 2024-09-11T03:42:58.139 INFO:tasks.rados.rados.2.smithi175.stdout:1265: done (4 left) 2024-09-11T03:42:58.139 INFO:tasks.rados.rados.2.smithi175.stdout:1267: done (3 left) 2024-09-11T03:42:58.139 INFO:tasks.rados.rados.2.smithi175.stdout:1271: read oid 47 snap -1 2024-09-11T03:42:58.139 INFO:tasks.rados.rados.2.smithi175.stdout:1271: expect (ObjNum 369 snap 113 seq_num 369) 2024-09-11T03:42:58.139 INFO:tasks.rados.rados.2.smithi175.stdout:1272: copy_from oid 6 from oid 20 current snap is 130 2024-09-11T03:42:58.139 INFO:tasks.rados.rados.2.smithi175.stdout:1268: finishing write tid 5 to smithi17557870-36 2024-09-11T03:42:58.139 INFO:tasks.rados.rados.2.smithi175.stdout:1268: oid 36 updating version 0 to 265 2024-09-11T03:42:58.139 INFO:tasks.rados.rados.2.smithi175.stdout:1268: oid 36 version 265 is already newer than 263 2024-09-11T03:42:58.140 INFO:tasks.rados.rados.2.smithi175.stdout:1268: oid 36 updating version 265 to 266 2024-09-11T03:42:58.140 INFO:tasks.rados.rados.2.smithi175.stdout:1268: oid 36 version 266 is already newer than 264 2024-09-11T03:42:58.140 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 266 (ObjNum 444 snap 130 seq_num 444) dirty exists 2024-09-11T03:42:58.140 INFO:tasks.rados.rados.2.smithi175.stdout:1268: left oid 36 (ObjNum 444 snap 130 seq_num 444) 2024-09-11T03:42:58.140 INFO:tasks.rados.rados.2.smithi175.stdout:1269: finishing write tid 1 to smithi17557870-49 2024-09-11T03:42:58.140 INFO:tasks.rados.rados.2.smithi175.stdout:1269: finishing write tid 2 to smithi17557870-49 2024-09-11T03:42:58.140 INFO:tasks.rados.rados.2.smithi175.stdout:1268: done (4 left) 2024-09-11T03:42:58.140 INFO:tasks.rados.rados.2.smithi175.stdout:1273: read oid 44 snap -1 2024-09-11T03:42:58.140 INFO:tasks.rados.rados.2.smithi175.stdout:1273: expect (ObjNum 419 snap 128 seq_num 419) 2024-09-11T03:42:58.140 INFO:tasks.rados.rados.2.smithi175.stdout:1274: copy_from oid 15 from oid 34 current snap is 130 2024-09-11T03:42:58.140 INFO:tasks.rados.rados.2.smithi175.stdout:1275: write (excl) oid 46 current snap is 130 2024-09-11T03:42:58.140 INFO:tasks.rados.rados.2.smithi175.stdout:1275: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2024-09-11T03:42:58.143 INFO:tasks.rados.rados.2.smithi175.stdout:1275: writing smithi17557870-46 from 695922 to 1405450 tid 1 2024-09-11T03:42:58.150 INFO:tasks.rados.rados.2.smithi175.stdout:1275: writing smithi17557870-46 from 2032616 to 2640096 tid 2 2024-09-11T03:42:58.151 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:42:58.152 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:42:58.156 INFO:tasks.rados.rados.2.smithi175.stdout:1275: writing smithi17557870-46 from 3198537 to 3495922 tid 3 2024-09-11T03:42:58.156 INFO:tasks.rados.rados.2.smithi175.stdout:1276: read oid 19 snap -1 2024-09-11T03:42:58.156 INFO:tasks.rados.rados.2.smithi175.stdout:1276: expect deleted 2024-09-11T03:42:58.156 INFO:tasks.rados.rados.2.smithi175.stdout:1277: write oid 3 current snap is 130 2024-09-11T03:42:58.156 INFO:tasks.rados.rados.2.smithi175.stdout:1274: finishing copy_from to smithi17557870-15 2024-09-11T03:42:58.156 INFO:tasks.rados.rados.2.smithi175.stdout:1274: got expected ENOENT (src dne) 2024-09-11T03:42:58.156 INFO:tasks.rados.rados.2.smithi175.stdout:1277: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2024-09-11T03:42:58.164 INFO:tasks.rados.rados.2.smithi175.stdout:1277: writing smithi17557870-3 from 712729 to 1497506 tid 1 2024-09-11T03:42:58.170 INFO:tasks.rados.rados.2.smithi175.stdout:1277: writing smithi17557870-3 from 2091098 to 2558589 tid 2 2024-09-11T03:42:58.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:42:58.171 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:42:58.174 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:42:58.176 INFO:tasks.rados.rados.2.smithi175.stdout:1277: writing smithi17557870-3 from 3342313 to 3512729 tid 3 2024-09-11T03:42:58.176 INFO:tasks.rados.rados.2.smithi175.stdout:1278: read oid 36 snap -1 2024-09-11T03:42:58.177 INFO:tasks.rados.rados.2.smithi175.stdout:1278: expect (ObjNum 444 snap 130 seq_num 444) 2024-09-11T03:42:58.177 INFO:tasks.rados.rados.2.smithi175.stdout:1279: write oid 1 current snap is 130 2024-09-11T03:42:58.177 INFO:tasks.rados.rados.2.smithi175.stdout:1274: finishing copy_from racing read to smithi17557870-15 2024-09-11T03:42:58.177 INFO:tasks.rados.rados.2.smithi175.stdout:1269: finishing write tid 3 to smithi17557870-49 2024-09-11T03:42:58.177 INFO:tasks.rados.rados.2.smithi175.stdout:1269: finishing write tid 4 to smithi17557870-49 2024-09-11T03:42:58.177 INFO:tasks.rados.rados.2.smithi175.stdout:1269: finishing write tid 5 to smithi17557870-49 2024-09-11T03:42:58.177 INFO:tasks.rados.rados.2.smithi175.stdout:1269: oid 49 updating version 0 to 453 2024-09-11T03:42:58.177 INFO:tasks.rados.rados.2.smithi175.stdout:1269: oid 49 version 453 is already newer than 452 2024-09-11T03:42:58.177 INFO:tasks.rados.rados.2.smithi175.stdout:1269: oid 49 version 453 is already newer than 450 2024-09-11T03:42:58.177 INFO:tasks.rados.rados.2.smithi175.stdout:1269: oid 49 version 453 is already newer than 451 2024-09-11T03:42:58.178 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 453 (ObjNum 445 snap 130 seq_num 445) dirty exists 2024-09-11T03:42:58.178 INFO:tasks.rados.rados.2.smithi175.stdout:1269: left oid 49 (ObjNum 445 snap 130 seq_num 445) 2024-09-11T03:42:58.178 INFO:tasks.rados.rados.2.smithi175.stdout:1279: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2024-09-11T03:42:58.182 INFO:tasks.rados.rados.2.smithi175.stdout:1279: writing smithi17557870-1 from 729536 to 1189562 tid 1 2024-09-11T03:42:58.188 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:42:58.188 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:42:58.189 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:58 smithi084 ceph-mon[27254]: pgmap v694: 57 pgs: 57 active+clean; 168 MiB data, 719 MiB used, 893 GiB / 894 GiB avail; 22 MiB/s rd, 2.8 MiB/s wr, 29 op/s 2024-09-11T03:42:58.189 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:58 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1966474106' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:58.189 INFO:tasks.rados.rados.2.smithi175.stdout:1279: writing smithi17557870-1 from 1749580 to 2360729 tid 2 2024-09-11T03:42:58.194 INFO:tasks.rados.rados.2.smithi175.stdout:1279: writing smithi17557870-1 from 2853383 to 3305833 tid 3 2024-09-11T03:42:58.194 INFO:tasks.rados.rados.2.smithi175.stdout:1269: done (10 left) 2024-09-11T03:42:58.194 INFO:tasks.rados.rados.2.smithi175.stdout:1274: done (9 left) 2024-09-11T03:42:58.195 INFO:tasks.rados.rados.2.smithi175.stdout:1280: delete oid 29 current snap is 130 2024-09-11T03:42:58.195 INFO:tasks.rados.rados.2.smithi175.stdout:1275: finishing write tid 1 to smithi17557870-46 2024-09-11T03:42:58.195 INFO:tasks.rados.rados.2.smithi175.stdout:1271: expect (ObjNum 369 snap 113 seq_num 369) 2024-09-11T03:42:58.197 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:58.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:58.198 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:42:58.206 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:42:58.206 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:42:58.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:42:58.222 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:42:58.239 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:42:58.239 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:42:58.255 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:42:58.256 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:42:58.272 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:42:58.272 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:42:58.288 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:42:58.289 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:42:58.298 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:58.304 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:42:58.305 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:42:58.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:58.320 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:58.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:58.322 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:42:58.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:42:58.327 INFO:tasks.rados.rados.2.smithi175.stdout:1272: finishing copy_from racing read to smithi17557870-6 2024-09-11T03:42:58.328 INFO:tasks.rados.rados.2.smithi175.stdout:1271: done (9 left) 2024-09-11T03:42:58.328 INFO:tasks.rados.rados.2.smithi175.stdout:1280: done (8 left) 2024-09-11T03:42:58.328 INFO:tasks.rados.rados.2.smithi175.stdout:1281: rollback oid 37 current snap is 130 2024-09-11T03:42:58.328 INFO:tasks.rados.rados.2.smithi175.stdout:1281: done (8 left) 2024-09-11T03:42:58.328 INFO:tasks.rados.rados.2.smithi175.stdout:1282: write oid 43 current snap is 130 2024-09-11T03:42:58.328 INFO:tasks.rados.rados.2.smithi175.stdout:1275: finishing write tid 2 to smithi17557870-46 2024-09-11T03:42:58.328 INFO:tasks.rados.rados.2.smithi175.stdout:1273: expect (ObjNum 419 snap 128 seq_num 419) 2024-09-11T03:42:58.338 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:42:58.339 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:42:58.421 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:58 smithi121 ceph-mon[30729]: pgmap v694: 57 pgs: 57 active+clean; 168 MiB data, 719 MiB used, 893 GiB / 894 GiB avail; 22 MiB/s rd, 2.8 MiB/s wr, 29 op/s 2024-09-11T03:42:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:58 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1966474106' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:58.443 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:58.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:58.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:58.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T03:42:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:58 smithi175 ceph-mon[30894]: pgmap v694: 57 pgs: 57 active+clean; 168 MiB data, 719 MiB used, 893 GiB / 894 GiB avail; 22 MiB/s rd, 2.8 MiB/s wr, 29 op/s 2024-09-11T03:42:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:58 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1966474106' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:42:58.482 INFO:tasks.rados.rados.2.smithi175.stdout:1270: finishing copy_from racing read to smithi17557870-7 2024-09-11T03:42:58.482 INFO:tasks.rados.rados.2.smithi175.stdout:1282: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2024-09-11T03:42:58.490 INFO:tasks.rados.rados.2.smithi175.stdout:1282: writing smithi17557870-43 from 763150 to 1490027 tid 1 2024-09-11T03:42:58.496 INFO:tasks.rados.rados.2.smithi175.stdout:1282: writing smithi17557870-43 from 1982897 to 2597715 tid 2 2024-09-11T03:42:58.502 INFO:tasks.rados.rados.2.smithi175.stdout:1282: writing smithi17557870-43 from 3024582 to 3499576 tid 3 2024-09-11T03:42:58.502 INFO:tasks.rados.rados.2.smithi175.stdout:1273: done (8 left) 2024-09-11T03:42:58.502 INFO:tasks.rados.rados.2.smithi175.stdout:1283: read oid 48 snap -1 2024-09-11T03:42:58.502 INFO:tasks.rados.rados.2.smithi175.stdout:1283: expect (ObjNum 434 snap 130 seq_num 434) 2024-09-11T03:42:58.502 INFO:tasks.rados.rados.2.smithi175.stdout:1284: write (excl) oid 2 current snap is 130 2024-09-11T03:42:58.502 INFO:tasks.rados.rados.2.smithi175.stdout:1277: finishing write tid 1 to smithi17557870-3 2024-09-11T03:42:58.502 INFO:tasks.rados.rados.2.smithi175.stdout:1275: finishing write tid 3 to smithi17557870-46 2024-09-11T03:42:58.503 INFO:tasks.rados.rados.2.smithi175.stdout:1277: finishing write tid 2 to smithi17557870-3 2024-09-11T03:42:58.503 INFO:tasks.rados.rados.2.smithi175.stdout:1275: finishing write tid 4 to smithi17557870-46 2024-09-11T03:42:58.503 INFO:tasks.rados.rados.2.smithi175.stdout:1272: finishing copy_from to smithi17557870-6 2024-09-11T03:42:58.503 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 292 (ObjNum 335 snap 95 seq_num 335) dirty exists 2024-09-11T03:42:58.503 INFO:tasks.rados.rados.2.smithi175.stdout:1270: finishing copy_from to smithi17557870-7 2024-09-11T03:42:58.503 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 456 (ObjNum 440 snap 130 seq_num 440) dirty exists 2024-09-11T03:42:58.503 INFO:tasks.rados.rados.2.smithi175.stdout:1275: finishing write tid 5 to smithi17557870-46 2024-09-11T03:42:58.503 INFO:tasks.rados.rados.2.smithi175.stdout:1275: oid 46 updating version 0 to 181 2024-09-11T03:42:58.503 INFO:tasks.rados.rados.2.smithi175.stdout:1275: oid 46 version 181 is already newer than 180 2024-09-11T03:42:58.503 INFO:tasks.rados.rados.2.smithi175.stdout:1275: oid 46 updating version 181 to 182 2024-09-11T03:42:58.503 INFO:tasks.rados.rados.2.smithi175.stdout:1275: oid 46 version 182 is already newer than 179 2024-09-11T03:42:58.503 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 182 (ObjNum 446 snap 130 seq_num 446) dirty exists 2024-09-11T03:42:58.503 INFO:tasks.rados.rados.2.smithi175.stdout:1275: left oid 46 (ObjNum 446 snap 130 seq_num 446) 2024-09-11T03:42:58.503 INFO:tasks.rados.rados.2.smithi175.stdout:1277: finishing write tid 3 to smithi17557870-3 2024-09-11T03:42:58.504 INFO:tasks.rados.rados.2.smithi175.stdout:1277: finishing write tid 4 to smithi17557870-3 2024-09-11T03:42:58.504 INFO:tasks.rados.rados.2.smithi175.stdout:1277: finishing write tid 5 to smithi17557870-3 2024-09-11T03:42:58.504 INFO:tasks.rados.rados.2.smithi175.stdout:1277: oid 3 updating version 0 to 493 2024-09-11T03:42:58.504 INFO:tasks.rados.rados.2.smithi175.stdout:1277: oid 3 updating version 493 to 494 2024-09-11T03:42:58.504 INFO:tasks.rados.rados.2.smithi175.stdout:1277: oid 3 version 494 is already newer than 492 2024-09-11T03:42:58.504 INFO:tasks.rados.rados.2.smithi175.stdout:1277: oid 3 version 494 is already newer than 491 2024-09-11T03:42:58.504 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 494 (ObjNum 447 snap 130 seq_num 447) dirty exists 2024-09-11T03:42:58.504 INFO:tasks.rados.rados.2.smithi175.stdout:1277: left oid 3 (ObjNum 447 snap 130 seq_num 447) 2024-09-11T03:42:58.504 INFO:tasks.rados.rados.2.smithi175.stdout:1284: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2024-09-11T03:42:58.508 INFO:tasks.rados.rados.2.smithi175.stdout:1284: writing smithi17557870-2 from 779957 to 1182083 tid 1 2024-09-11T03:42:58.515 INFO:tasks.rados.rados.2.smithi175.stdout:1284: writing smithi17557870-2 from 1925026 to 2683502 tid 2 2024-09-11T03:42:58.521 INFO:tasks.rados.rados.2.smithi175.stdout:1284: writing smithi17557870-2 from 3335652 to 3579957 tid 3 2024-09-11T03:42:58.521 INFO:tasks.rados.rados.2.smithi175.stdout:1270: done (9 left) 2024-09-11T03:42:58.521 INFO:tasks.rados.rados.2.smithi175.stdout:1272: done (8 left) 2024-09-11T03:42:58.521 INFO:tasks.rados.rados.2.smithi175.stdout:1275: done (7 left) 2024-09-11T03:42:58.521 INFO:tasks.rados.rados.2.smithi175.stdout:1276: done (6 left) 2024-09-11T03:42:58.521 INFO:tasks.rados.rados.2.smithi175.stdout:1277: done (5 left) 2024-09-11T03:42:58.521 INFO:tasks.rados.rados.2.smithi175.stdout:1285: read oid 44 snap -1 2024-09-11T03:42:58.521 INFO:tasks.rados.rados.2.smithi175.stdout:1285: expect (ObjNum 419 snap 128 seq_num 419) 2024-09-11T03:42:58.521 INFO:tasks.rados.rados.2.smithi175.stdout:1286: read oid 20 snap -1 2024-09-11T03:42:58.521 INFO:tasks.rados.rados.2.smithi175.stdout:1286: expect (ObjNum 335 snap 95 seq_num 335) 2024-09-11T03:42:58.521 INFO:tasks.rados.rados.2.smithi175.stdout:1287: delete oid 28 current snap is 130 2024-09-11T03:42:58.521 INFO:tasks.rados.rados.2.smithi175.stdout:1278: expect (ObjNum 444 snap 130 seq_num 444) 2024-09-11T03:42:58.545 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:58.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:58.567 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:58.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:42:58.667 INFO:tasks.rados.rados.2.smithi175.stdout:1279: finishing write tid 1 to smithi17557870-1 2024-09-11T03:42:58.667 INFO:tasks.rados.rados.2.smithi175.stdout:1279: finishing write tid 2 to smithi17557870-1 2024-09-11T03:42:58.667 INFO:tasks.rados.rados.2.smithi175.stdout:1279: finishing write tid 3 to smithi17557870-1 2024-09-11T03:42:58.667 INFO:tasks.rados.rados.2.smithi175.stdout:1279: finishing write tid 4 to smithi17557870-1 2024-09-11T03:42:58.667 INFO:tasks.rados.rados.2.smithi175.stdout:1278: done (7 left) 2024-09-11T03:42:58.667 INFO:tasks.rados.rados.2.smithi175.stdout:1287: done (6 left) 2024-09-11T03:42:58.667 INFO:tasks.rados.rados.2.smithi175.stdout:1288: read oid 5 snap -1 2024-09-11T03:42:58.667 INFO:tasks.rados.rados.2.smithi175.stdout:1288: expect deleted 2024-09-11T03:42:58.667 INFO:tasks.rados.rados.2.smithi175.stdout:1289: read oid 25 snap -1 2024-09-11T03:42:58.668 INFO:tasks.rados.rados.2.smithi175.stdout:1289: expect (ObjNum 344 snap 99 seq_num 344) 2024-09-11T03:42:58.668 INFO:tasks.rados.rados.2.smithi175.stdout:1290: delete oid 22 current snap is 130 2024-09-11T03:42:58.668 INFO:tasks.rados.rados.2.smithi175.stdout:1279: finishing write tid 5 to smithi17557870-1 2024-09-11T03:42:58.668 INFO:tasks.rados.rados.2.smithi175.stdout:1279: oid 1 updating version 0 to 459 2024-09-11T03:42:58.668 INFO:tasks.rados.rados.2.smithi175.stdout:1279: oid 1 updating version 459 to 460 2024-09-11T03:42:58.668 INFO:tasks.rados.rados.2.smithi175.stdout:1279: oid 1 updating version 460 to 461 2024-09-11T03:42:58.668 INFO:tasks.rados.rados.2.smithi175.stdout:1279: oid 1 version 461 is already newer than 458 2024-09-11T03:42:58.668 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 461 (ObjNum 448 snap 130 seq_num 448) dirty exists 2024-09-11T03:42:58.668 INFO:tasks.rados.rados.2.smithi175.stdout:1279: left oid 1 (ObjNum 448 snap 130 seq_num 448) 2024-09-11T03:42:58.668 INFO:tasks.rados.rados.2.smithi175.stdout:1282: finishing write tid 1 to smithi17557870-43 2024-09-11T03:42:58.668 INFO:tasks.rados.rados.2.smithi175.stdout:1282: finishing write tid 2 to smithi17557870-43 2024-09-11T03:42:58.668 INFO:tasks.rados.rados.2.smithi175.stdout:1282: finishing write tid 3 to smithi17557870-43 2024-09-11T03:42:58.668 INFO:tasks.rados.rados.2.smithi175.stdout:1282: finishing write tid 4 to smithi17557870-43 2024-09-11T03:42:58.668 INFO:tasks.rados.rados.2.smithi175.stdout:1282: finishing write tid 5 to smithi17557870-43 2024-09-11T03:42:58.669 INFO:tasks.rados.rados.2.smithi175.stdout:1282: oid 43 updating version 0 to 380 2024-09-11T03:42:58.669 INFO:tasks.rados.rados.2.smithi175.stdout:1282: oid 43 updating version 380 to 382 2024-09-11T03:42:58.669 INFO:tasks.rados.rados.2.smithi175.stdout:1282: oid 43 version 382 is already newer than 381 2024-09-11T03:42:58.669 INFO:tasks.rados.rados.2.smithi175.stdout:1282: oid 43 version 382 is already newer than 379 2024-09-11T03:42:58.669 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 382 (ObjNum 450 snap 130 seq_num 450) dirty exists 2024-09-11T03:42:58.669 INFO:tasks.rados.rados.2.smithi175.stdout:1282: left oid 43 (ObjNum 450 snap 130 seq_num 450) 2024-09-11T03:42:58.669 INFO:tasks.rados.rados.2.smithi175.stdout:1283: expect (ObjNum 434 snap 130 seq_num 434) 2024-09-11T03:42:58.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:58.694 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:58.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:58.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:58.752 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:42:58.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:58.817 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:58.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:58.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:58.863 INFO:tasks.rados.rados.2.smithi175.stdout:1284: finishing write tid 1 to smithi17557870-2 2024-09-11T03:42:58.864 INFO:tasks.rados.rados.2.smithi175.stdout:1284: finishing write tid 2 to smithi17557870-2 2024-09-11T03:42:58.864 INFO:tasks.rados.rados.2.smithi175.stdout:1279: done (8 left) 2024-09-11T03:42:58.864 INFO:tasks.rados.rados.2.smithi175.stdout:1282: done (7 left) 2024-09-11T03:42:58.864 INFO:tasks.rados.rados.2.smithi175.stdout:1283: done (6 left) 2024-09-11T03:42:58.864 INFO:tasks.rados.rados.2.smithi175.stdout:1290: done (5 left) 2024-09-11T03:42:58.864 INFO:tasks.rados.rados.2.smithi175.stdout:1291: write (excl) oid 36 current snap is 130 2024-09-11T03:42:58.864 INFO:tasks.rados.rados.2.smithi175.stdout:1284: finishing write tid 3 to smithi17557870-2 2024-09-11T03:42:58.864 INFO:tasks.rados.rados.2.smithi175.stdout:1284: finishing write tid 4 to smithi17557870-2 2024-09-11T03:42:58.864 INFO:tasks.rados.rados.2.smithi175.stdout:1291: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2024-09-11T03:42:58.869 INFO:tasks.rados.rados.2.smithi175.stdout:1291: writing smithi17557870-36 from 430378 to 1058251 tid 1 2024-09-11T03:42:58.876 INFO:tasks.rados.rados.2.smithi175.stdout:1291: writing smithi17557870-36 from 1700472 to 2322628 tid 2 2024-09-11T03:42:58.883 INFO:tasks.rados.rados.2.smithi175.stdout:1291: writing smithi17557870-36 from 3017921 to 3630378 tid 3 2024-09-11T03:42:58.883 INFO:tasks.rados.rados.2.smithi175.stdout:1292: write oid 15 current snap is 130 2024-09-11T03:42:58.883 INFO:tasks.rados.rados.2.smithi175.stdout:1284: finishing write tid 5 to smithi17557870-2 2024-09-11T03:42:58.883 INFO:tasks.rados.rados.2.smithi175.stdout:1284: oid 2 updating version 0 to 468 2024-09-11T03:42:58.883 INFO:tasks.rados.rados.2.smithi175.stdout:1284: oid 2 version 468 is already newer than 467 2024-09-11T03:42:58.884 INFO:tasks.rados.rados.2.smithi175.stdout:1284: oid 2 version 468 is already newer than 465 2024-09-11T03:42:58.884 INFO:tasks.rados.rados.2.smithi175.stdout:1284: oid 2 version 468 is already newer than 466 2024-09-11T03:42:58.884 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 468 (ObjNum 451 snap 130 seq_num 451) dirty exists 2024-09-11T03:42:58.884 INFO:tasks.rados.rados.2.smithi175.stdout:1284: left oid 2 (ObjNum 451 snap 130 seq_num 451) 2024-09-11T03:42:58.884 INFO:tasks.rados.rados.2.smithi175.stdout:1292: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2024-09-11T03:42:58.889 INFO:tasks.rados.rados.2.smithi175.stdout:1292: writing smithi17557870-15 from 447185 to 1150307 tid 1 2024-09-11T03:42:58.896 INFO:tasks.rados.rados.2.smithi175.stdout:1292: writing smithi17557870-15 from 1642601 to 2408415 tid 2 2024-09-11T03:42:58.901 INFO:tasks.rados.rados.2.smithi175.stdout:1292: writing smithi17557870-15 from 2928991 to 3376698 tid 3 2024-09-11T03:42:58.902 INFO:tasks.rados.rados.2.smithi175.stdout:1284: done (6 left) 2024-09-11T03:42:58.902 INFO:tasks.rados.rados.2.smithi175.stdout:1293: read oid 39 snap -1 2024-09-11T03:42:58.902 INFO:tasks.rados.rados.2.smithi175.stdout:1293: expect (ObjNum 388 snap 117 seq_num 388) 2024-09-11T03:42:58.902 INFO:tasks.rados.rados.2.smithi175.stdout:1294: delete oid 24 current snap is 130 2024-09-11T03:42:58.902 INFO:tasks.rados.rados.2.smithi175.stdout:1285: expect (ObjNum 419 snap 128 seq_num 419) 2024-09-11T03:42:58.919 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:58.940 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:58.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:58.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:59.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:59.059 INFO:tasks.rados.rados.2.smithi175.stdout:1286: expect (ObjNum 335 snap 95 seq_num 335) 2024-09-11T03:42:59.063 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:59.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:59.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:59 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:59.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:59 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1966474106' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:59.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:59 smithi121 ceph-mon[30729]: osdmap e361: 12 total, 12 up, 10 in 2024-09-11T03:42:59.141 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:42:59 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2633633229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:59.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:59 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:59.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:59 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1966474106' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:59.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:59 smithi084 ceph-mon[27254]: osdmap e361: 12 total, 12 up, 10 in 2024-09-11T03:42:59.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:42:59 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2633633229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:59.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:42:59.186 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:59.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:42:59.239 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:42:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:42:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:42:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:42:59.240 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:42:59.273 INFO:tasks.rados.rados.2.smithi175.stdout:1289: expect (ObjNum 344 snap 99 seq_num 344) 2024-09-11T03:42:59.289 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:42:59.310 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:59.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:59.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:42:59.412 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:42:59.434 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:59.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:42:59.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:59 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:42:59.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:59 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1966474106' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:42:59.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:59 smithi175 ceph-mon[30894]: osdmap e361: 12 total, 12 up, 10 in 2024-09-11T03:42:59.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:42:59 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2633633229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:42:59.505 INFO:tasks.rados.rados.2.smithi175.stdout:1291: finishing write tid 1 to smithi17557870-36 2024-09-11T03:42:59.505 INFO:tasks.rados.rados.2.smithi175.stdout:1291: finishing write tid 2 to smithi17557870-36 2024-09-11T03:42:59.505 INFO:tasks.rados.rados.2.smithi175.stdout:1291: finishing write tid 3 to smithi17557870-36 2024-09-11T03:42:59.505 INFO:tasks.rados.rados.2.smithi175.stdout:1291: finishing write tid 4 to smithi17557870-36 2024-09-11T03:42:59.505 INFO:tasks.rados.rados.2.smithi175.stdout:1291: finishing write tid 5 to smithi17557870-36 2024-09-11T03:42:59.506 INFO:tasks.rados.rados.2.smithi175.stdout:1291: oid 36 updating version 0 to 267 2024-09-11T03:42:59.506 INFO:tasks.rados.rados.2.smithi175.stdout:1291: oid 36 updating version 267 to 270 2024-09-11T03:42:59.506 INFO:tasks.rados.rados.2.smithi175.stdout:1291: oid 36 version 270 is already newer than 268 2024-09-11T03:42:59.506 INFO:tasks.rados.rados.2.smithi175.stdout:1291: oid 36 version 270 is already newer than 269 2024-09-11T03:42:59.506 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 270 (ObjNum 454 snap 130 seq_num 454) dirty exists 2024-09-11T03:42:59.506 INFO:tasks.rados.rados.2.smithi175.stdout:1291: left oid 36 (ObjNum 454 snap 130 seq_num 454) 2024-09-11T03:42:59.506 INFO:tasks.rados.rados.2.smithi175.stdout:1292: finishing write tid 1 to smithi17557870-15 2024-09-11T03:42:59.506 INFO:tasks.rados.rados.2.smithi175.stdout:1285: done (7 left) 2024-09-11T03:42:59.506 INFO:tasks.rados.rados.2.smithi175.stdout:1286: done (6 left) 2024-09-11T03:42:59.506 INFO:tasks.rados.rados.2.smithi175.stdout:1288: done (5 left) 2024-09-11T03:42:59.506 INFO:tasks.rados.rados.2.smithi175.stdout:1289: done (4 left) 2024-09-11T03:42:59.506 INFO:tasks.rados.rados.2.smithi175.stdout:1291: done (3 left) 2024-09-11T03:42:59.506 INFO:tasks.rados.rados.2.smithi175.stdout:1294: done (2 left) 2024-09-11T03:42:59.507 INFO:tasks.rados.rados.2.smithi175.stdout:1295: read oid 32 snap -1 2024-09-11T03:42:59.507 INFO:tasks.rados.rados.2.smithi175.stdout:1295: expect (ObjNum 277 snap 73 seq_num 277) 2024-09-11T03:42:59.507 INFO:tasks.rados.rados.2.smithi175.stdout:1296: delete oid 16 current snap is 130 2024-09-11T03:42:59.507 INFO:tasks.rados.rados.2.smithi175.stdout:1292: finishing write tid 2 to smithi17557870-15 2024-09-11T03:42:59.507 INFO:tasks.rados.rados.2.smithi175.stdout:1292: finishing write tid 3 to smithi17557870-15 2024-09-11T03:42:59.507 INFO:tasks.rados.rados.2.smithi175.stdout:1292: finishing write tid 4 to smithi17557870-15 2024-09-11T03:42:59.507 INFO:tasks.rados.rados.2.smithi175.stdout:1292: finishing write tid 5 to smithi17557870-15 2024-09-11T03:42:59.507 INFO:tasks.rados.rados.2.smithi175.stdout:1292: oid 15 updating version 0 to 196 2024-09-11T03:42:59.507 INFO:tasks.rados.rados.2.smithi175.stdout:1292: oid 15 updating version 196 to 198 2024-09-11T03:42:59.507 INFO:tasks.rados.rados.2.smithi175.stdout:1292: oid 15 version 198 is already newer than 197 2024-09-11T03:42:59.507 INFO:tasks.rados.rados.2.smithi175.stdout:1292: oid 15 version 198 is already newer than 195 2024-09-11T03:42:59.507 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 198 (ObjNum 455 snap 130 seq_num 455) dirty exists 2024-09-11T03:42:59.507 INFO:tasks.rados.rados.2.smithi175.stdout:1292: left oid 15 (ObjNum 455 snap 130 seq_num 455) 2024-09-11T03:42:59.507 INFO:tasks.rados.rados.2.smithi175.stdout:1293: expect (ObjNum 388 snap 117 seq_num 388) 2024-09-11T03:42:59.536 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:42:59.557 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:59.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:59.558 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:42:59.593 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:42:59.633 INFO:tasks.rados.rados.2.smithi175.stdout:1295: expect (ObjNum 277 snap 73 seq_num 277) 2024-09-11T03:42:59.659 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:42:59.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:59.681 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:42:59.682 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:42:59.720 INFO:tasks.rados.rados.2.smithi175.stdout:1292: done (3 left) 2024-09-11T03:42:59.720 INFO:tasks.rados.rados.2.smithi175.stdout:1293: done (2 left) 2024-09-11T03:42:59.720 INFO:tasks.rados.rados.2.smithi175.stdout:1295: done (1 left) 2024-09-11T03:42:59.720 INFO:tasks.rados.rados.2.smithi175.stdout:1296: done (0 left) 2024-09-11T03:42:59.720 INFO:tasks.rados.rados.2.smithi175.stdout:1297: delete oid 42 current snap is 130 2024-09-11T03:42:59.721 INFO:tasks.rados.rados.2.smithi175.stdout:1297: done (0 left) 2024-09-11T03:42:59.721 INFO:tasks.rados.rados.2.smithi175.stdout:1298: write oid 2 current snap is 130 2024-09-11T03:42:59.721 INFO:tasks.rados.rados.2.smithi175.stdout:1298: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2024-09-11T03:42:59.728 INFO:tasks.rados.rados.2.smithi175.stdout:1298: writing smithi17557870-2 from 514413 to 1234884 tid 1 2024-09-11T03:42:59.735 INFO:tasks.rados.rados.2.smithi175.stdout:1298: writing smithi17557870-2 from 1876529 to 2649681 tid 2 2024-09-11T03:42:59.741 INFO:tasks.rados.rados.2.smithi175.stdout:1298: writing smithi17557870-2 from 3438683 to 3714413 tid 3 2024-09-11T03:42:59.741 INFO:tasks.rados.rados.2.smithi175.stdout:1299: read oid 46 snap -1 2024-09-11T03:42:59.741 INFO:tasks.rados.rados.2.smithi175.stdout:1299: expect (ObjNum 446 snap 130 seq_num 446) 2024-09-11T03:42:59.742 INFO:tasks.rados.rados.2.smithi175.stdout:1300: write oid 26 current snap is 130 2024-09-11T03:42:59.742 INFO:tasks.rados.rados.2.smithi175.stdout:1298: finishing write tid 1 to smithi17557870-2 2024-09-11T03:42:59.742 INFO:tasks.rados.rados.2.smithi175.stdout:1298: finishing write tid 2 to smithi17557870-2 2024-09-11T03:42:59.742 INFO:tasks.rados.rados.2.smithi175.stdout:1300: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2024-09-11T03:42:59.748 INFO:tasks.rados.rados.2.smithi175.stdout:1300: writing smithi17557870-26 from 531220 to 1326940 tid 1 2024-09-11T03:42:59.754 INFO:tasks.rados.rados.2.smithi175.stdout:1300: writing smithi17557870-26 from 1935011 to 2451821 tid 2 2024-09-11T03:42:59.760 INFO:tasks.rados.rados.2.smithi175.stdout:1300: writing smithi17557870-26 from 3066106 to 3486526 tid 3 2024-09-11T03:42:59.760 INFO:tasks.rados.rados.2.smithi175.stdout:1301: read oid 21 snap -1 2024-09-11T03:42:59.760 INFO:tasks.rados.rados.2.smithi175.stdout:1301: expect (ObjNum 266 snap 67 seq_num 266) 2024-09-11T03:42:59.760 INFO:tasks.rados.rados.2.smithi175.stdout:1302: snap_create 2024-09-11T03:42:59.760 INFO:tasks.rados.rados.2.smithi175.stdout:1298: finishing write tid 3 to smithi17557870-2 2024-09-11T03:42:59.760 INFO:tasks.rados.rados.2.smithi175.stdout:1298: finishing write tid 4 to smithi17557870-2 2024-09-11T03:42:59.760 INFO:tasks.rados.rados.2.smithi175.stdout:1298: finishing write tid 5 to smithi17557870-2 2024-09-11T03:42:59.760 INFO:tasks.rados.rados.2.smithi175.stdout:1298: oid 2 updating version 0 to 472 2024-09-11T03:42:59.761 INFO:tasks.rados.rados.2.smithi175.stdout:1298: oid 2 version 472 is already newer than 471 2024-09-11T03:42:59.761 INFO:tasks.rados.rados.2.smithi175.stdout:1298: oid 2 version 472 is already newer than 470 2024-09-11T03:42:59.761 INFO:tasks.rados.rados.2.smithi175.stdout:1298: oid 2 version 472 is already newer than 469 2024-09-11T03:42:59.761 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 472 (ObjNum 459 snap 130 seq_num 459) dirty exists 2024-09-11T03:42:59.761 INFO:tasks.rados.rados.2.smithi175.stdout:1298: left oid 2 (ObjNum 459 snap 130 seq_num 459) 2024-09-11T03:42:59.761 INFO:tasks.rados.rados.2.smithi175.stdout:1299: expect (ObjNum 446 snap 130 seq_num 446) 2024-09-11T03:42:59.782 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:42:59.806 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:42:59.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:59.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:42:59.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T03:42:59.908 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:42:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:42:59.957 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:42:59.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:00.037 INFO:tasks.rados.rados.2.smithi175.stdout:1300: finishing write tid 1 to smithi17557870-26 2024-09-11T03:43:00.038 INFO:tasks.rados.rados.2.smithi175.stdout:1300: finishing write tid 2 to smithi17557870-26 2024-09-11T03:43:00.038 INFO:tasks.rados.rados.2.smithi175.stdout:1300: finishing write tid 3 to smithi17557870-26 2024-09-11T03:43:00.038 INFO:tasks.rados.rados.2.smithi175.stdout:1300: finishing write tid 4 to smithi17557870-26 2024-09-11T03:43:00.038 INFO:tasks.rados.rados.2.smithi175.stdout:1300: finishing write tid 5 to smithi17557870-26 2024-09-11T03:43:00.038 INFO:tasks.rados.rados.2.smithi175.stdout:1300: oid 26 updating version 0 to 235 2024-09-11T03:43:00.038 INFO:tasks.rados.rados.2.smithi175.stdout:1300: oid 26 version 235 is already newer than 232 2024-09-11T03:43:00.038 INFO:tasks.rados.rados.2.smithi175.stdout:1300: oid 26 version 235 is already newer than 233 2024-09-11T03:43:00.038 INFO:tasks.rados.rados.2.smithi175.stdout:1300: oid 26 version 235 is already newer than 234 2024-09-11T03:43:00.038 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 235 (ObjNum 460 snap 130 seq_num 460) dirty exists 2024-09-11T03:43:00.038 INFO:tasks.rados.rados.2.smithi175.stdout:1300: left oid 26 (ObjNum 460 snap 130 seq_num 460) 2024-09-11T03:43:00.038 INFO:tasks.rados.rados.2.smithi175.stdout:1301: expect (ObjNum 266 snap 67 seq_num 266) 2024-09-11T03:43:00.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:00.081 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:00.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:00.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:00.104 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:43:00.111 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:00 smithi084 ceph-mon[27254]: pgmap v697: 57 pgs: 57 active+clean; 132 MiB data, 689 MiB used, 893 GiB / 894 GiB avail; 43 MiB/s rd, 5.3 MiB/s wr, 59 op/s 2024-09-11T03:43:00.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:00 smithi121 ceph-mon[30729]: pgmap v697: 57 pgs: 57 active+clean; 132 MiB data, 689 MiB used, 893 GiB / 894 GiB avail; 43 MiB/s rd, 5.3 MiB/s wr, 59 op/s 2024-09-11T03:43:00.183 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:00.186 INFO:tasks.rados.rados.2.smithi175.stdout:1298: done (4 left) 2024-09-11T03:43:00.186 INFO:tasks.rados.rados.2.smithi175.stdout:1299: done (3 left) 2024-09-11T03:43:00.186 INFO:tasks.rados.rados.2.smithi175.stdout:1300: done (2 left) 2024-09-11T03:43:00.186 INFO:tasks.rados.rados.2.smithi175.stdout:1301: done (1 left) 2024-09-11T03:43:00.186 INFO:tasks.rados.rados.2.smithi175.stdout:1302: done (0 left) 2024-09-11T03:43:00.187 INFO:tasks.rados.rados.2.smithi175.stdout:1303: snap_create 2024-09-11T03:43:00.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:00.205 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:00.205 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:00.278 INFO:teuthology.orchestra.run.smithi084.stdout:moved 0 / 171 (0%) 2024-09-11T03:43:00.278 INFO:teuthology.orchestra.run.smithi084.stdout:avg 17.1 2024-09-11T03:43:00.278 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 2.62488 -> 2.62488 (expected baseline 3.92301) 2024-09-11T03:43:00.278 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.2 with 14 -> 14 pgs (0.818713 -> 0.818713 * mean) 2024-09-11T03:43:00.278 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.7 with 22 -> 22 pgs (1.28655 -> 1.28655 * mean) 2024-09-11T03:43:00.278 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:43:00.278 INFO:teuthology.orchestra.run.smithi084.stdout:oload 1000 2024-09-11T03:43:00.278 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 3 2024-09-11T03:43:00.278 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2024-09-11T03:43:00.279 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0008 2024-09-11T03:43:00.279 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0075 2024-09-11T03:43:00.279 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2024-09-11T03:43:00.305 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:00.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:00.327 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:00.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:00.428 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:00.448 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:00.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:00.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:00 smithi175 ceph-mon[30894]: pgmap v697: 57 pgs: 57 active+clean; 132 MiB data, 689 MiB used, 893 GiB / 894 GiB avail; 43 MiB/s rd, 5.3 MiB/s wr, 59 op/s 2024-09-11T03:43:00.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:00.571 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:00.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:00.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:00.639 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:43:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:43:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:43:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:43:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDScrubReserve(1.0,QREQUEST,e:65,from:7,reservation_nonce:2)", 2024-09-11T03:43:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:36:27.978100+0000", 2024-09-11T03:43:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 392.65997673499999, 2024-09-11T03:43:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012126909999999999, 2024-09-11T03:43:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:43:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.978100+0000", 2024-09-11T03:43:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.978100+0000", 2024-09-11T03:43:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.978101+0000", 2024-09-11T03:43:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.029e-06 2024-09-11T03:43:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.978106+0000", 2024-09-11T03:43:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5029999999999997e-06 2024-09-11T03:43:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.978121+0000", 2024-09-11T03:43:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4329e-05 2024-09-11T03:43:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.978130+0000", 2024-09-11T03:43:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9579999999999996e-06 2024-09-11T03:43:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.978198+0000", 2024-09-11T03:43:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8762000000000003e-05 2024-09-11T03:43:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:43:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.978204+0000", 2024-09-11T03:43:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2939999999999996e-06 2024-09-11T03:43:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.979261+0000", 2024-09-11T03:43:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010568929999999999 2024-09-11T03:43:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.979279+0000", 2024-09-11T03:43:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8453000000000001e-05 2024-09-11T03:43:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:36:27.979313+0000", 2024-09-11T03:43:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3470000000000003e-05 2024-09-11T03:43:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:225 2.2 e103/95 2:43d79ac9:::smithi17557870-37:head v 103'11, mlcod=100'8)", 2024-09-11T03:43:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.943277+0000", 2024-09-11T03:43:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 260.69479915199997, 2024-09-11T03:43:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000975161, 2024-09-11T03:43:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943277+0000", 2024-09-11T03:43:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943277+0000", 2024-09-11T03:43:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943278+0000", 2024-09-11T03:43:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5800000000000001e-07 2024-09-11T03:43:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943283+0000", 2024-09-11T03:43:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8640000000000004e-06 2024-09-11T03:43:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943298+0000", 2024-09-11T03:43:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.506e-05 2024-09-11T03:43:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943308+0000", 2024-09-11T03:43:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0821e-05 2024-09-11T03:43:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943381+0000", 2024-09-11T03:43:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3028000000000006e-05 2024-09-11T03:43:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943406+0000", 2024-09-11T03:43:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4119999999999999e-05 2024-09-11T03:43:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:43:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.944218+0000", 2024-09-11T03:43:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081275499999999997 2024-09-11T03:43:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.944253+0000", 2024-09-11T03:43:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4255000000000003e-05 2024-09-11T03:43:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(osd.5.0:18 2.2 e104/95 2:43d79ac9:::smithi17557870-37:3 v 104'13, mlcod=100'9)", 2024-09-11T03:43:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:41.272262+0000", 2024-09-11T03:43:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 259.36581425200001, 2024-09-11T03:43:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00099929800000000003, 2024-09-11T03:43:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.272262+0000", 2024-09-11T03:43:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.272262+0000", 2024-09-11T03:43:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.272263+0000", 2024-09-11T03:43:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0500000000000004e-07 2024-09-11T03:43:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.272270+0000", 2024-09-11T03:43:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8340000000000001e-06 2024-09-11T03:43:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.272283+0000", 2024-09-11T03:43:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2981e-05 2024-09-11T03:43:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.272295+0000", 2024-09-11T03:43:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2116999999999999e-05 2024-09-11T03:43:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.272383+0000", 2024-09-11T03:43:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7831999999999995e-05 2024-09-11T03:43:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.272406+0000", 2024-09-11T03:43:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3887e-05 2024-09-11T03:43:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:43:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.273199+0000", 2024-09-11T03:43:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079215399999999999 2024-09-11T03:43:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.273262+0000", 2024-09-11T03:43:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2988000000000001e-05 2024-09-11T03:43:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:343 2.2 e113/95 2:494ba932:::smithi17557870-43:head v 113'20, mlcod=107'17)", 2024-09-11T03:43:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.831626+0000", 2024-09-11T03:43:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 250.806450856, 2024-09-11T03:43:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035769930000000001, 2024-09-11T03:43:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.831626+0000", 2024-09-11T03:43:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.831626+0000", 2024-09-11T03:43:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.831626+0000", 2024-09-11T03:43:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2999999999999999e-07 2024-09-11T03:43:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833432+0000", 2024-09-11T03:43:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018063350000000001 2024-09-11T03:43:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833447+0000", 2024-09-11T03:43:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5065999999999999e-05 2024-09-11T03:43:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833455+0000", 2024-09-11T03:43:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2930000000000003e-06 2024-09-11T03:43:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833512+0000", 2024-09-11T03:43:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7648000000000001e-05 2024-09-11T03:43:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833538+0000", 2024-09-11T03:43:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5962e-05 2024-09-11T03:43:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:43:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.835162+0000", 2024-09-11T03:43:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016233580000000001 2024-09-11T03:43:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.835203+0000", 2024-09-11T03:43:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1100999999999998e-05 2024-09-11T03:43:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:366 2.7 2:e0eabcfb:::smithi17557870-9:head [write 410104~465105,stat] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:43:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:50.854890+0000", 2024-09-11T03:43:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 249.78318656600001, 2024-09-11T03:43:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040276269999999998, 2024-09-11T03:43:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:43:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:43:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 366 2024-09-11T03:43:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.854890+0000", 2024-09-11T03:43:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.854890+0000", 2024-09-11T03:43:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.854892+0000", 2024-09-11T03:43:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.629e-06 2024-09-11T03:43:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.855622+0000", 2024-09-11T03:43:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073058800000000001 2024-09-11T03:43:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.855638+0000", 2024-09-11T03:43:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5648e-05 2024-09-11T03:43:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.855662+0000", 2024-09-11T03:43:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4620000000000001e-05 2024-09-11T03:43:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.855717+0000", 2024-09-11T03:43:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4342000000000002e-05 2024-09-11T03:43:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.855789+0000", 2024-09-11T03:43:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2550000000000002e-05 2024-09-11T03:43:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:43:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.855901+0000", 2024-09-11T03:43:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111352 2024-09-11T03:43:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:43:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.857414+0000", 2024-09-11T03:43:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015131159999999999 2024-09-11T03:43:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.858043+0000", 2024-09-11T03:43:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00062914299999999998 2024-09-11T03:43:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.858836+0000", 2024-09-11T03:43:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079319899999999997 2024-09-11T03:43:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:43:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.858871+0000", 2024-09-11T03:43:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4523999999999997e-05 2024-09-11T03:43:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.858918+0000", 2024-09-11T03:43:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6916000000000003e-05 2024-09-11T03:43:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:367 2.7 2:e0eabcfb:::smithi17557870-9:head [write 1563527~583025] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:43:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:50.861918+0000", 2024-09-11T03:43:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 249.776159009, 2024-09-11T03:43:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035522819999999999, 2024-09-11T03:43:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:43:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:43:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 367 2024-09-11T03:43:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.861918+0000", 2024-09-11T03:43:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.861918+0000", 2024-09-11T03:43:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.861919+0000", 2024-09-11T03:43:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0300000000000001e-06 2024-09-11T03:43:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.862732+0000", 2024-09-11T03:43:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081320100000000005 2024-09-11T03:43:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.862745+0000", 2024-09-11T03:43:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2801e-05 2024-09-11T03:43:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.862750+0000", 2024-09-11T03:43:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5729999999999998e-06 2024-09-11T03:43:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.862843+0000", 2024-09-11T03:43:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3033000000000002e-05 2024-09-11T03:43:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.862912+0000", 2024-09-11T03:43:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8616999999999997e-05 2024-09-11T03:43:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:43:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.862997+0000", 2024-09-11T03:43:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4975000000000005e-05 2024-09-11T03:43:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:43:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.864288+0000", 2024-09-11T03:43:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001291237 2024-09-11T03:43:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.865104+0000", 2024-09-11T03:43:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081632099999999995 2024-09-11T03:43:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.865395+0000", 2024-09-11T03:43:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00029032800000000002 2024-09-11T03:43:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:43:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.865428+0000", 2024-09-11T03:43:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3327000000000001e-05 2024-09-11T03:43:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.865470+0000", 2024-09-11T03:43:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1838999999999997e-05 2024-09-11T03:43:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:368 2.7 2:e0eabcfb:::smithi17557870-9:head [write 2810926~399178] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:43:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:50.867602+0000", 2024-09-11T03:43:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 249.77047453099999, 2024-09-11T03:43:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024245920000000002, 2024-09-11T03:43:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:43:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:43:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 368 2024-09-11T03:43:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.867602+0000", 2024-09-11T03:43:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.867602+0000", 2024-09-11T03:43:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.867603+0000", 2024-09-11T03:43:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0410000000000001e-06 2024-09-11T03:43:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868107+0000", 2024-09-11T03:43:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050433300000000004 2024-09-11T03:43:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868118+0000", 2024-09-11T03:43:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0798999999999999e-05 2024-09-11T03:43:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868123+0000", 2024-09-11T03:43:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2429999999999996e-06 2024-09-11T03:43:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868196+0000", 2024-09-11T03:43:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2529999999999998e-05 2024-09-11T03:43:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868260+0000", 2024-09-11T03:43:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3569000000000001e-05 2024-09-11T03:43:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:43:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868344+0000", 2024-09-11T03:43:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4608e-05 2024-09-11T03:43:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:43:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869414+0000", 2024-09-11T03:43:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010696029999999999 2024-09-11T03:43:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869737+0000", 2024-09-11T03:43:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032352800000000001 2024-09-11T03:43:00.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869952+0000", 2024-09-11T03:43:00.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021448800000000001 2024-09-11T03:43:00.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:43:00.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869984+0000", 2024-09-11T03:43:00.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2406000000000001e-05 2024-09-11T03:43:00.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.870027+0000", 2024-09-11T03:43:00.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2444000000000002e-05 2024-09-11T03:43:00.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.681 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.681 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:369 2.7 2:e0eabcfb:::smithi17557870-9:head [setxattr _header (55) in=7b,truncate 3210104] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:43:00.681 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:50.868138+0000", 2024-09-11T03:43:00.681 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 249.76993887200001, 2024-09-11T03:43:00.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019888039999999998, 2024-09-11T03:43:00.681 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:43:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:43:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 369 2024-09-11T03:43:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868138+0000", 2024-09-11T03:43:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868138+0000", 2024-09-11T03:43:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868138+0000", 2024-09-11T03:43:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0200000000000001e-07 2024-09-11T03:43:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868159+0000", 2024-09-11T03:43:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0874e-05 2024-09-11T03:43:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868173+0000", 2024-09-11T03:43:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3801999999999999e-05 2024-09-11T03:43:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868180+0000", 2024-09-11T03:43:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4960000000000001e-06 2024-09-11T03:43:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868995+0000", 2024-09-11T03:43:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081590999999999999 2024-09-11T03:43:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869060+0000", 2024-09-11T03:43:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4568999999999998e-05 2024-09-11T03:43:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:43:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869147+0000", 2024-09-11T03:43:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6543000000000001e-05 2024-09-11T03:43:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:43:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869416+0000", 2024-09-11T03:43:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00026988700000000001 2024-09-11T03:43:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869773+0000", 2024-09-11T03:43:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035655000000000002 2024-09-11T03:43:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.870057+0000", 2024-09-11T03:43:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028380700000000002 2024-09-11T03:43:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:43:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.870090+0000", 2024-09-11T03:43:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3311e-05 2024-09-11T03:43:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.870127+0000", 2024-09-11T03:43:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6353000000000002e-05 2024-09-11T03:43:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:370 2.7 2:e0eabcfb:::smithi17557870-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:43:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:50.868190+0000", 2024-09-11T03:43:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 249.76988660999999, 2024-09-11T03:43:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020750069999999998, 2024-09-11T03:43:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:43:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:43:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:43:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 370 2024-09-11T03:43:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868190+0000", 2024-09-11T03:43:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868190+0000", 2024-09-11T03:43:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868190+0000", 2024-09-11T03:43:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3399999999999999e-07 2024-09-11T03:43:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868193+0000", 2024-09-11T03:43:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.227e-06 2024-09-11T03:43:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868199+0000", 2024-09-11T03:43:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7730000000000002e-06 2024-09-11T03:43:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.868205+0000", 2024-09-11T03:43:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9390000000000002e-06 2024-09-11T03:43:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869359+0000", 2024-09-11T03:43:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011531169999999999 2024-09-11T03:43:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:43:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.869405+0000", 2024-09-11T03:43:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6925999999999998e-05 2024-09-11T03:43:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.870161+0000", 2024-09-11T03:43:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075579900000000003 2024-09-11T03:43:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.870205+0000", 2024-09-11T03:43:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3662000000000003e-05 2024-09-11T03:43:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:50.870265+0000", 2024-09-11T03:43:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.003e-05 2024-09-11T03:43:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:402 2.7 2:e23d1762:::smithi17557870-21:head [copy-from ver 9] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:43:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:51.085659+0000", 2024-09-11T03:43:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 249.55241773099999, 2024-09-11T03:43:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012118869000000001, 2024-09-11T03:43:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:43:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:43:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 402 2024-09-11T03:43:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085659+0000", 2024-09-11T03:43:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085659+0000", 2024-09-11T03:43:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085661+0000", 2024-09-11T03:43:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1220000000000002e-06 2024-09-11T03:43:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085669+0000", 2024-09-11T03:43:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2779999999999993e-06 2024-09-11T03:43:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085689+0000", 2024-09-11T03:43:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9639999999999998e-05 2024-09-11T03:43:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085699+0000", 2024-09-11T03:43:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0524e-05 2024-09-11T03:43:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085789+0000", 2024-09-11T03:43:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9215000000000006e-05 2024-09-11T03:43:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085880+0000", 2024-09-11T03:43:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1187999999999995e-05 2024-09-11T03:43:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:43:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.090839+0000", 2024-09-11T03:43:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004958863 2024-09-11T03:43:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:43:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.093676+0000", 2024-09-11T03:43:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028368930000000001 2024-09-11T03:43:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.097457+0000", 2024-09-11T03:43:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003781197 2024-09-11T03:43:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.097707+0000", 2024-09-11T03:43:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024975900000000001 2024-09-11T03:43:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:43:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.097745+0000", 2024-09-11T03:43:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8596000000000001e-05 2024-09-11T03:43:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.097778+0000", 2024-09-11T03:43:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2594e-05 2024-09-11T03:43:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:403 2.7 2:e23d1762:::smithi17557870-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:43:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:51.085720+0000", 2024-09-11T03:43:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 249.552356911, 2024-09-11T03:43:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076054490000000002, 2024-09-11T03:43:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:43:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:43:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:43:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 403 2024-09-11T03:43:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085720+0000", 2024-09-11T03:43:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085720+0000", 2024-09-11T03:43:00.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085720+0000", 2024-09-11T03:43:00.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5599999999999995e-07 2024-09-11T03:43:00.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085742+0000", 2024-09-11T03:43:00.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1968e-05 2024-09-11T03:43:00.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085750+0000", 2024-09-11T03:43:00.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8699999999999992e-06 2024-09-11T03:43:00.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085756+0000", 2024-09-11T03:43:00.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-09-11T03:43:00.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.086016+0000", 2024-09-11T03:43:00.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00026035099999999997 2024-09-11T03:43:00.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:00.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.086071+0000", 2024-09-11T03:43:00.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4614999999999998e-05 2024-09-11T03:43:00.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.093154+0000", 2024-09-11T03:43:00.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070835050000000004 2024-09-11T03:43:00.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.093239+0000", 2024-09-11T03:43:00.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5309999999999995e-05 2024-09-11T03:43:00.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.093325+0000", 2024-09-11T03:43:00.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5654000000000004e-05 2024-09-11T03:43:00.706 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.706 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.706 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.706 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:405 2.7 2:e6da0655:::smithi17557870-27:head [delete] snapc 9=[9] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115)", 2024-09-11T03:43:00.706 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:51.814169+0000", 2024-09-11T03:43:00.706 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 248.823907496, 2024-09-11T03:43:00.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030515849999999999, 2024-09-11T03:43:00.706 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.706 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.706 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:00.706 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:43:00.707 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:43:00.707 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:00.707 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 405 2024-09-11T03:43:00.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.707 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.814169+0000", 2024-09-11T03:43:00.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.814169+0000", 2024-09-11T03:43:00.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.814171+0000", 2024-09-11T03:43:00.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.844e-06 2024-09-11T03:43:00.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.814178+0000", 2024-09-11T03:43:00.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-09-11T03:43:00.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.814194+0000", 2024-09-11T03:43:00.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5943e-05 2024-09-11T03:43:00.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.814221+0000", 2024-09-11T03:43:00.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6659999999999999e-05 2024-09-11T03:43:00.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.814291+0000", 2024-09-11T03:43:00.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0388000000000004e-05 2024-09-11T03:43:00.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:43:00.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.814298+0000", 2024-09-11T03:43:00.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4659999999999996e-06 2024-09-11T03:43:00.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.815820+0000", 2024-09-11T03:43:00.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015221639999999999 2024-09-11T03:43:00.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.815911+0000", 2024-09-11T03:43:00.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0989000000000006e-05 2024-09-11T03:43:00.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:43:00.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.816069+0000", 2024-09-11T03:43:00.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000158436 2024-09-11T03:43:00.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:43:00.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.816751+0000", 2024-09-11T03:43:00.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00068151400000000003 2024-09-11T03:43:00.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.816971+0000", 2024-09-11T03:43:00.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022063399999999999 2024-09-11T03:43:00.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.817149+0000", 2024-09-11T03:43:00.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017768100000000001 2024-09-11T03:43:00.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:43:00.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.817187+0000", 2024-09-11T03:43:00.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7889999999999998e-05 2024-09-11T03:43:00.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.817221+0000", 2024-09-11T03:43:00.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3640000000000003e-05 2024-09-11T03:43:00.713 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.713 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.713 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.714 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:2 2.7 2:e0ec5763:::smithi17557870-2:head [assert-version v3,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e119)", 2024-09-11T03:43:00.714 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:55.859247+0000", 2024-09-11T03:43:00.714 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 244.77883006900001, 2024-09-11T03:43:00.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045129180000000003, 2024-09-11T03:43:00.714 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.714 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.714 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:43:00.714 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.859247+0000", 2024-09-11T03:43:00.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.859247+0000", 2024-09-11T03:43:00.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.859248+0000", 2024-09-11T03:43:00.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6899999999999999e-06 2024-09-11T03:43:00.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.859254+0000", 2024-09-11T03:43:00.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7649999999999999e-06 2024-09-11T03:43:00.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.859264+0000", 2024-09-11T03:43:00.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0244000000000001e-05 2024-09-11T03:43:00.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.859274+0000", 2024-09-11T03:43:00.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7839999999999998e-06 2024-09-11T03:43:00.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.859345+0000", 2024-09-11T03:43:00.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0584e-05 2024-09-11T03:43:00.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:43:00.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.859351+0000", 2024-09-11T03:43:00.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2319999999999998e-06 2024-09-11T03:43:00.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.860628+0000", 2024-09-11T03:43:00.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001277542 2024-09-11T03:43:00.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.860700+0000", 2024-09-11T03:43:00.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1524000000000002e-05 2024-09-11T03:43:00.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:55.863759+0000", 2024-09-11T03:43:00.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003059553 2024-09-11T03:43:00.718 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.718 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.718 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.719 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:583 2.2 e128/95 2:494ba932:::smithi17557870-43:head v 128'41, mlcod=124'37)", 2024-09-11T03:43:00.719 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:04.877155+0000", 2024-09-11T03:43:00.719 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 235.76092116999999, 2024-09-11T03:43:00.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00099352800000000003, 2024-09-11T03:43:00.719 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.719 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.719 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:00.719 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.877155+0000", 2024-09-11T03:43:00.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.877155+0000", 2024-09-11T03:43:00.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.877156+0000", 2024-09-11T03:43:00.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2799999999999996e-07 2024-09-11T03:43:00.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.877171+0000", 2024-09-11T03:43:00.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.507e-05 2024-09-11T03:43:00.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.877199+0000", 2024-09-11T03:43:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8407000000000001e-05 2024-09-11T03:43:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.877209+0000", 2024-09-11T03:43:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-09-11T03:43:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.877287+0000", 2024-09-11T03:43:00.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8559999999999993e-05 2024-09-11T03:43:00.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.877310+0000", 2024-09-11T03:43:00.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2872000000000001e-05 2024-09-11T03:43:00.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:43:00.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.878111+0000", 2024-09-11T03:43:00.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00080112399999999996 2024-09-11T03:43:00.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:04.878149+0000", 2024-09-11T03:43:00.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7755e-05 2024-09-11T03:43:00.724 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.724 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.724 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.724 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(osd.5.0:45 2.2 e130/95 2:494ba932:::smithi17557870-43:11 v 130'45, mlcod=128'41)", 2024-09-11T03:43:00.724 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:06.816260+0000", 2024-09-11T03:43:00.724 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 233.821816144, 2024-09-11T03:43:00.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001130898, 2024-09-11T03:43:00.724 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.724 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.725 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:00.725 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.816260+0000", 2024-09-11T03:43:00.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.816260+0000", 2024-09-11T03:43:00.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.816261+0000", 2024-09-11T03:43:00.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1900000000000002e-07 2024-09-11T03:43:00.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.816268+0000", 2024-09-11T03:43:00.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0090000000000001e-06 2024-09-11T03:43:00.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.816280+0000", 2024-09-11T03:43:00.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1783999999999999e-05 2024-09-11T03:43:00.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.816288+0000", 2024-09-11T03:43:00.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1969999999999996e-06 2024-09-11T03:43:00.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.816357+0000", 2024-09-11T03:43:00.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8779e-05 2024-09-11T03:43:00.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.816381+0000", 2024-09-11T03:43:00.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3634000000000001e-05 2024-09-11T03:43:00.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:43:00.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.817334+0000", 2024-09-11T03:43:00.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00095372000000000002 2024-09-11T03:43:00.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:06.817391+0000", 2024-09-11T03:43:00.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7055999999999997e-05 2024-09-11T03:43:00.728 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.728 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.728 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.729 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:613 2.7 2:e23d1762:::smithi17557870-21:head [stat,delete] snapc 11=[f,e] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-11T03:43:00.729 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:07.043261+0000", 2024-09-11T03:43:00.729 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 233.59481566100001, 2024-09-11T03:43:00.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040861639999999998, 2024-09-11T03:43:00.729 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.729 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.729 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:00.729 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:43:00.729 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:43:00.729 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:00.729 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 613 2024-09-11T03:43:00.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.730 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043261+0000", 2024-09-11T03:43:00.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043261+0000", 2024-09-11T03:43:00.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043264+0000", 2024-09-11T03:43:00.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7769999999999998e-06 2024-09-11T03:43:00.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043275+0000", 2024-09-11T03:43:00.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1063000000000001e-05 2024-09-11T03:43:00.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043306+0000", 2024-09-11T03:43:00.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1327e-05 2024-09-11T03:43:00.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043322+0000", 2024-09-11T03:43:00.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5602000000000001e-05 2024-09-11T03:43:00.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043458+0000", 2024-09-11T03:43:00.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000136036 2024-09-11T03:43:00.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043552+0000", 2024-09-11T03:43:00.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4058e-05 2024-09-11T03:43:00.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:43:00.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.043693+0000", 2024-09-11T03:43:00.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014124 2024-09-11T03:43:00.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:43:00.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.044599+0000", 2024-09-11T03:43:00.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090568099999999998 2024-09-11T03:43:00.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.045227+0000", 2024-09-11T03:43:00.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00062812899999999995 2024-09-11T03:43:00.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.047232+0000", 2024-09-11T03:43:00.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020049759999999999 2024-09-11T03:43:00.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:43:00.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.047299+0000", 2024-09-11T03:43:00.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6795e-05 2024-09-11T03:43:00.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.047347+0000", 2024-09-11T03:43:00.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8479999999999997e-05 2024-09-11T03:43:00.735 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.735 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.735 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.735 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:631 2.7 2:e6da0655:::smithi17557870-27:head [append 0~0] snapc 11=[e] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:43:00.735 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:07.867549+0000", 2024-09-11T03:43:00.735 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 232.77052745399999, 2024-09-11T03:43:00.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024185919999999998, 2024-09-11T03:43:00.736 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.736 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.736 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:00.736 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:43:00.736 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:43:00.736 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:00.736 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 631 2024-09-11T03:43:00.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.736 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867549+0000", 2024-09-11T03:43:00.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867549+0000", 2024-09-11T03:43:00.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867552+0000", 2024-09-11T03:43:00.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7e-06 2024-09-11T03:43:00.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867562+0000", 2024-09-11T03:43:00.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0226e-05 2024-09-11T03:43:00.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867593+0000", 2024-09-11T03:43:00.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1235999999999997e-05 2024-09-11T03:43:00.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867609+0000", 2024-09-11T03:43:00.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5324000000000001e-05 2024-09-11T03:43:00.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867761+0000", 2024-09-11T03:43:00.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015238799999999999 2024-09-11T03:43:00.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867971+0000", 2024-09-11T03:43:00.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00020959900000000001 2024-09-11T03:43:00.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:43:00.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.868164+0000", 2024-09-11T03:43:00.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019307800000000001 2024-09-11T03:43:00.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869787+0000", 2024-09-11T03:43:00.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016228600000000001 2024-09-11T03:43:00.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:43:00.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869801+0000", 2024-09-11T03:43:00.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4637999999999999e-05 2024-09-11T03:43:00.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869895+0000", 2024-09-11T03:43:00.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3615000000000004e-05 2024-09-11T03:43:00.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:43:00.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869933+0000", 2024-09-11T03:43:00.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8189000000000003e-05 2024-09-11T03:43:00.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869968+0000", 2024-09-11T03:43:00.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4739000000000001e-05 2024-09-11T03:43:00.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.742 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.742 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:632 2.7 2:e6da0655:::smithi17557870-27:head [rollback e] snapc 11=[e] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:43:00.742 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:07.867642+0000", 2024-09-11T03:43:00.742 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 232.77043450599999, 2024-09-11T03:43:00.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024234339999999999, 2024-09-11T03:43:00.742 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.742 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.742 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:00.742 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:43:00.743 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:43:00.743 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:00.743 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 632 2024-09-11T03:43:00.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.743 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867642+0000", 2024-09-11T03:43:00.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867642+0000", 2024-09-11T03:43:00.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867643+0000", 2024-09-11T03:43:00.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.28e-06 2024-09-11T03:43:00.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867649+0000", 2024-09-11T03:43:00.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5600000000000001e-06 2024-09-11T03:43:00.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867667+0000", 2024-09-11T03:43:00.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8406999999999998e-05 2024-09-11T03:43:00.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867681+0000", 2024-09-11T03:43:00.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3973e-05 2024-09-11T03:43:00.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.868700+0000", 2024-09-11T03:43:00.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010191779999999999 2024-09-11T03:43:00.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.868760+0000", 2024-09-11T03:43:00.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9339999999999998e-05 2024-09-11T03:43:00.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:43:00.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.868900+0000", 2024-09-11T03:43:00.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014041999999999999 2024-09-11T03:43:00.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869845+0000", 2024-09-11T03:43:00.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00094485599999999999 2024-09-11T03:43:00.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:43:00.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869977+0000", 2024-09-11T03:43:00.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013179500000000001 2024-09-11T03:43:00.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.870003+0000", 2024-09-11T03:43:00.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6064e-05 2024-09-11T03:43:00.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:43:00.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.870040+0000", 2024-09-11T03:43:00.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7076999999999998e-05 2024-09-11T03:43:00.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.870066+0000", 2024-09-11T03:43:00.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5483999999999999e-05 2024-09-11T03:43:00.748 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.749 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.749 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.749 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:633 2.7 2:e6da0655:::smithi17557870-27:head [append 0~0] snapc 11=[e] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:43:00.749 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:07.867728+0000", 2024-09-11T03:43:00.749 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 232.77034859700001, 2024-09-11T03:43:00.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024618119999999999, 2024-09-11T03:43:00.749 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.749 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.749 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:00.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:43:00.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:43:00.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:00.750 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 633 2024-09-11T03:43:00.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.750 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867728+0000", 2024-09-11T03:43:00.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867728+0000", 2024-09-11T03:43:00.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867729+0000", 2024-09-11T03:43:00.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3069999999999999e-06 2024-09-11T03:43:00.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867777+0000", 2024-09-11T03:43:00.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.757e-05 2024-09-11T03:43:00.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867809+0000", 2024-09-11T03:43:00.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2209000000000003e-05 2024-09-11T03:43:00.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.867819+0000", 2024-09-11T03:43:00.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7860000000000008e-06 2024-09-11T03:43:00.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869311+0000", 2024-09-11T03:43:00.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001491722 2024-09-11T03:43:00.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869406+0000", 2024-09-11T03:43:00.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5612999999999994e-05 2024-09-11T03:43:00.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:43:00.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.869547+0000", 2024-09-11T03:43:00.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000141175 2024-09-11T03:43:00.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:43:00.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.870077+0000", 2024-09-11T03:43:00.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052995199999999998 2024-09-11T03:43:00.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.870108+0000", 2024-09-11T03:43:00.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0657000000000001e-05 2024-09-11T03:43:00.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:00.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.870135+0000", 2024-09-11T03:43:00.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7506999999999999e-05 2024-09-11T03:43:00.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:43:00.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.870164+0000", 2024-09-11T03:43:00.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8504e-05 2024-09-11T03:43:00.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.870190+0000", 2024-09-11T03:43:00.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5809999999999999e-05 2024-09-11T03:43:00.755 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.755 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.755 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.756 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.11.83:3 2.7 2:e77ce595:::smithi17557870-31:head [assert-version v23,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:43:00.756 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:08.111371+0000", 2024-09-11T03:43:00.756 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 232.526705566, 2024-09-11T03:43:00.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023068860000000002, 2024-09-11T03:43:00.756 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:43:00.756 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:43:00.756 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:43:00.756 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:43:00.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:43:00.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.111371+0000", 2024-09-11T03:43:00.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:43:00.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.111371+0000", 2024-09-11T03:43:00.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:43:00.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:43:00.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.111373+0000", 2024-09-11T03:43:00.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.716e-06 2024-09-11T03:43:00.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:43:00.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.111378+0000", 2024-09-11T03:43:00.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2240000000000003e-06 2024-09-11T03:43:00.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:43:00.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.111388+0000", 2024-09-11T03:43:00.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9129999999999998e-06 2024-09-11T03:43:00.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:43:00.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.111394+0000", 2024-09-11T03:43:00.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1659999999999999e-06 2024-09-11T03:43:00.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:43:00.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.111455+0000", 2024-09-11T03:43:00.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0525000000000001e-05 2024-09-11T03:43:00.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:43:00.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.111522+0000", 2024-09-11T03:43:00.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7010999999999999e-05 2024-09-11T03:43:00.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:43:00.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:43:00.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:43:00.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:08.113678+0000", 2024-09-11T03:43:00.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002156331 2024-09-11T03:43:00.760 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.760 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.760 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.760 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:43:00.760 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:43:00.760 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:43:00.760 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:00.783 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:00.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:00.784 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:00.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:43:00.885 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:00.908 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:00.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:00.909 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:00.937 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-11T03:43:00.937 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-11T03:43:01.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:01.031 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:01.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:01.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:01.063 INFO:tasks.rados.rados.2.smithi175.stdout:1303: done (0 left) 2024-09-11T03:43:01.063 INFO:tasks.rados.rados.2.smithi175.stdout:1304: write oid 35 current snap is 132 2024-09-11T03:43:01.064 INFO:tasks.rados.rados.2.smithi175.stdout:1304: seq_num 461 ranges {548027=470969,1593493=776821,3093529=631692} 2024-09-11T03:43:01.069 INFO:tasks.rados.rados.2.smithi175.stdout:1304: writing smithi17557870-35 from 548027 to 1018996 tid 1 2024-09-11T03:43:01.076 INFO:tasks.rados.rados.2.smithi175.stdout:1304: writing smithi17557870-35 from 1593493 to 2370314 tid 2 2024-09-11T03:43:01.084 INFO:tasks.rados.rados.2.smithi175.stdout:1304: writing smithi17557870-35 from 3093529 to 3725221 tid 3 2024-09-11T03:43:01.084 INFO:tasks.rados.rados.2.smithi175.stdout:1305: write (excl) oid 20 current snap is 132 2024-09-11T03:43:01.084 INFO:tasks.rados.rados.2.smithi175.stdout:1305: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2024-09-11T03:43:01.090 INFO:tasks.rados.rados.2.smithi175.stdout:1305: writing smithi17557870-20 from 564834 to 1111052 tid 1 2024-09-11T03:43:01.095 INFO:tasks.rados.rados.2.smithi175.stdout:1305: writing smithi17557870-20 from 1651975 to 2172454 tid 2 2024-09-11T03:43:01.101 INFO:tasks.rados.rados.2.smithi175.stdout:1305: writing smithi17557870-20 from 2720952 to 3280269 tid 3 2024-09-11T03:43:01.101 INFO:tasks.rados.rados.2.smithi175.stdout:1306: snap_create 2024-09-11T03:43:01.102 INFO:tasks.rados.rados.2.smithi175.stdout:1304: finishing write tid 1 to smithi17557870-35 2024-09-11T03:43:01.102 INFO:tasks.rados.rados.2.smithi175.stdout:1304: finishing write tid 2 to smithi17557870-35 2024-09-11T03:43:01.102 INFO:tasks.rados.rados.2.smithi175.stdout:1304: finishing write tid 3 to smithi17557870-35 2024-09-11T03:43:01.102 INFO:tasks.rados.rados.2.smithi175.stdout:1304: finishing write tid 4 to smithi17557870-35 2024-09-11T03:43:01.102 INFO:tasks.rados.rados.2.smithi175.stdout:1304: finishing write tid 5 to smithi17557870-35 2024-09-11T03:43:01.102 INFO:tasks.rados.rados.2.smithi175.stdout:1304: oid 35 updating version 0 to 458 2024-09-11T03:43:01.102 INFO:tasks.rados.rados.2.smithi175.stdout:1304: oid 35 version 458 is already newer than 457 2024-09-11T03:43:01.102 INFO:tasks.rados.rados.2.smithi175.stdout:1304: oid 35 version 458 is already newer than 456 2024-09-11T03:43:01.102 INFO:tasks.rados.rados.2.smithi175.stdout:1304: oid 35 version 458 is already newer than 455 2024-09-11T03:43:01.102 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 458 (ObjNum 461 snap 132 seq_num 461) dirty exists 2024-09-11T03:43:01.102 INFO:tasks.rados.rados.2.smithi175.stdout:1304: left oid 35 (ObjNum 461 snap 132 seq_num 461) 2024-09-11T03:43:01.102 INFO:tasks.rados.rados.2.smithi175.stdout:1305: finishing write tid 1 to smithi17557870-20 2024-09-11T03:43:01.102 INFO:tasks.rados.rados.2.smithi175.stdout:1305: finishing write tid 2 to smithi17557870-20 2024-09-11T03:43:01.104 INFO:tasks.rados.rados.2.smithi175.stdout:1305: finishing write tid 3 to smithi17557870-20 2024-09-11T03:43:01.104 INFO:tasks.rados.rados.2.smithi175.stdout:1305: finishing write tid 4 to smithi17557870-20 2024-09-11T03:43:01.104 INFO:tasks.rados.rados.2.smithi175.stdout:1305: finishing write tid 5 to smithi17557870-20 2024-09-11T03:43:01.104 INFO:tasks.rados.rados.2.smithi175.stdout:1305: oid 20 updating version 0 to 491 2024-09-11T03:43:01.105 INFO:tasks.rados.rados.2.smithi175.stdout:1305: oid 20 updating version 491 to 493 2024-09-11T03:43:01.105 INFO:tasks.rados.rados.2.smithi175.stdout:1305: oid 20 updating version 493 to 494 2024-09-11T03:43:01.105 INFO:tasks.rados.rados.2.smithi175.stdout:1305: oid 20 version 494 is already newer than 492 2024-09-11T03:43:01.105 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 494 (ObjNum 462 snap 132 seq_num 462) dirty exists 2024-09-11T03:43:01.105 INFO:tasks.rados.rados.2.smithi175.stdout:1305: left oid 20 (ObjNum 462 snap 132 seq_num 462) 2024-09-11T03:43:01.114 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:01.132 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:01.155 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:01.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:01.156 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:01.235 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:01 smithi084 ceph-mon[27254]: osdmap e362: 12 total, 12 up, 10 in 2024-09-11T03:43:01.235 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:01 smithi084 ceph-mon[27254]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:43:01.236 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:01 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:43:01.236 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:01 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:43:01.256 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:01.277 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:01.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:01.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:01.379 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:01.400 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:01.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:01.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:01.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:01 smithi121 ceph-mon[30729]: osdmap e362: 12 total, 12 up, 10 in 2024-09-11T03:43:01.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:01 smithi121 ceph-mon[30729]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:43:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:01 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:43:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:01 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:43:01.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T03:43:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:01 smithi175 ceph-mon[30894]: osdmap e362: 12 total, 12 up, 10 in 2024-09-11T03:43:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:01 smithi175 ceph-mon[30894]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:43:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:01 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:43:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:01 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:43:01.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:01.535 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:01.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:01.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:01.552 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:43:01.636 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:01.659 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:01.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:01.665 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:01.754 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:43:01.761 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:01.782 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:01.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:01.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:01.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:01.892 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:43:01.897 INFO:tasks.rados.rados.2.smithi175.stdout:1304: done (2 left) 2024-09-11T03:43:01.897 INFO:tasks.rados.rados.2.smithi175.stdout:1305: done (1 left) 2024-09-11T03:43:01.897 INFO:tasks.rados.rados.2.smithi175.stdout:1306: done (0 left) 2024-09-11T03:43:01.897 INFO:tasks.rados.rados.2.smithi175.stdout:1307: read oid 3 snap -1 2024-09-11T03:43:01.897 INFO:tasks.rados.rados.2.smithi175.stdout:1307: expect (ObjNum 447 snap 130 seq_num 447) 2024-09-11T03:43:01.897 INFO:tasks.rados.rados.2.smithi175.stdout:1308: delete oid 33 current snap is 133 2024-09-11T03:43:01.906 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:01.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:01.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:01.910 INFO:tasks.rados.rados.2.smithi175.stdout:1307: expect (ObjNum 447 snap 130 seq_num 447) 2024-09-11T03:43:02.008 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:02.029 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:02.030 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:02.061 INFO:tasks.rados.rados.2.smithi175.stdout:1307: done (1 left) 2024-09-11T03:43:02.061 INFO:tasks.rados.rados.2.smithi175.stdout:1308: done (0 left) 2024-09-11T03:43:02.061 INFO:tasks.rados.rados.2.smithi175.stdout:1309: write oid 46 current snap is 133 2024-09-11T03:43:02.061 INFO:tasks.rados.rados.2.smithi175.stdout:1309: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2024-09-11T03:43:02.066 INFO:tasks.rados.rados.2.smithi175.stdout:1309: writing smithi17557870-46 from 598448 to 1011517 tid 1 2024-09-11T03:43:02.073 INFO:tasks.rados.rados.2.smithi175.stdout:1309: writing smithi17557870-46 from 1768939 to 2293087 tid 2 2024-09-11T03:43:02.079 INFO:tasks.rados.rados.2.smithi175.stdout:1309: writing smithi17557870-46 from 2775798 to 3357659 tid 3 2024-09-11T03:43:02.082 INFO:tasks.rados.rados.2.smithi175.stdout:1309: writing smithi17557870-46 from 3781488 to 3798448 tid 4 2024-09-11T03:43:02.082 INFO:tasks.rados.rados.2.smithi175.stdout:1310: read oid 34 snap -1 2024-09-11T03:43:02.082 INFO:tasks.rados.rados.2.smithi175.stdout:1310: expect deleted 2024-09-11T03:43:02.082 INFO:tasks.rados.rados.2.smithi175.stdout:1311: read oid 32 snap -1 2024-09-11T03:43:02.082 INFO:tasks.rados.rados.2.smithi175.stdout:1311: expect (ObjNum 277 snap 73 seq_num 277) 2024-09-11T03:43:02.082 INFO:tasks.rados.rados.2.smithi175.stdout:1309: finishing write tid 1 to smithi17557870-46 2024-09-11T03:43:02.082 INFO:tasks.rados.rados.2.smithi175.stdout:1309: finishing write tid 2 to smithi17557870-46 2024-09-11T03:43:02.082 INFO:tasks.rados.rados.2.smithi175.stdout:1309: finishing write tid 3 to smithi17557870-46 2024-09-11T03:43:02.082 INFO:tasks.rados.rados.2.smithi175.stdout:1312: delete oid 14 current snap is 133 2024-09-11T03:43:02.083 INFO:tasks.rados.rados.2.smithi175.stdout:1309: finishing write tid 4 to smithi17557870-46 2024-09-11T03:43:02.083 INFO:tasks.rados.rados.2.smithi175.stdout:1311: expect (ObjNum 277 snap 73 seq_num 277) 2024-09-11T03:43:02.131 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:02.152 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:02.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:02.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:02.166 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:02 smithi121 ceph-mon[30729]: pgmap v699: 57 pgs: 57 active+clean; 132 MiB data, 672 MiB used, 893 GiB / 894 GiB avail; 36 MiB/s rd, 4.5 MiB/s wr, 50 op/s 2024-09-11T03:43:02.166 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:02 smithi121 ceph-mon[30729]: osdmap e363: 12 total, 12 up, 10 in 2024-09-11T03:43:02.166 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:02 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/328461103' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:43:02.166 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:02 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/328461103' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:43:02.166 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:02 smithi121 ceph-mon[30729]: osdmap e364: 12 total, 12 up, 10 in 2024-09-11T03:43:02.168 INFO:tasks.rados.rados.2.smithi175.stdout:1309: finishing write tid 5 to smithi17557870-46 2024-09-11T03:43:02.168 INFO:tasks.rados.rados.2.smithi175.stdout:1309: finishing write tid 6 to smithi17557870-46 2024-09-11T03:43:02.168 INFO:tasks.rados.rados.2.smithi175.stdout:1309: oid 46 updating version 0 to 185 2024-09-11T03:43:02.168 INFO:tasks.rados.rados.2.smithi175.stdout:1309: oid 46 version 185 is already newer than 183 2024-09-11T03:43:02.168 INFO:tasks.rados.rados.2.smithi175.stdout:1309: oid 46 updating version 185 to 186 2024-09-11T03:43:02.169 INFO:tasks.rados.rados.2.smithi175.stdout:1309: oid 46 updating version 186 to 187 2024-09-11T03:43:02.169 INFO:tasks.rados.rados.2.smithi175.stdout:1309: oid 46 version 187 is already newer than 184 2024-09-11T03:43:02.169 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 187 (ObjNum 464 snap 133 seq_num 464) dirty exists 2024-09-11T03:43:02.169 INFO:tasks.rados.rados.2.smithi175.stdout:1309: left oid 46 (ObjNum 464 snap 133 seq_num 464) 2024-09-11T03:43:02.169 INFO:tasks.rados.rados.2.smithi175.stdout:1309: done (3 left) 2024-09-11T03:43:02.169 INFO:tasks.rados.rados.2.smithi175.stdout:1310: done (2 left) 2024-09-11T03:43:02.169 INFO:tasks.rados.rados.2.smithi175.stdout:1311: done (1 left) 2024-09-11T03:43:02.169 INFO:tasks.rados.rados.2.smithi175.stdout:1312: done (0 left) 2024-09-11T03:43:02.169 INFO:tasks.rados.rados.2.smithi175.stdout:1313: rollback oid 12 current snap is 133 2024-09-11T03:43:02.169 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 12 to 131 2024-09-11T03:43:02.169 INFO:tasks.rados.rados.2.smithi175.stdout:1314: delete oid 35 current snap is 133 2024-09-11T03:43:02.170 INFO:tasks.rados.rados.2.smithi175.stdout:1314: done (1 left) 2024-09-11T03:43:02.170 INFO:tasks.rados.rados.2.smithi175.stdout:1315: snap_create 2024-09-11T03:43:02.171 INFO:tasks.rados.rados.2.smithi175.stdout:1313: finishing rollback tid 0 to smithi17557870-12 2024-09-11T03:43:02.171 INFO:tasks.rados.rados.2.smithi175.stdout:1313: finishing rollback tid 1 to smithi17557870-12 2024-09-11T03:43:02.171 INFO:tasks.rados.rados.2.smithi175.stdout:1313: finishing rollback tid 2 to smithi17557870-12 2024-09-11T03:43:02.171 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 466 (ObjNum 440 snap 130 seq_num 440) dirty exists 2024-09-11T03:43:02.255 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:02.276 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:02.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:02.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:02.281 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:43:02.281 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:43:02.281 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:43:02.281 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:43:02.352 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:02 smithi084 ceph-mon[27254]: pgmap v699: 57 pgs: 57 active+clean; 132 MiB data, 672 MiB used, 893 GiB / 894 GiB avail; 36 MiB/s rd, 4.5 MiB/s wr, 50 op/s 2024-09-11T03:43:02.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:02 smithi084 ceph-mon[27254]: osdmap e363: 12 total, 12 up, 10 in 2024-09-11T03:43:02.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:02 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/328461103' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:43:02.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:02 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/328461103' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:43:02.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:02 smithi084 ceph-mon[27254]: osdmap e364: 12 total, 12 up, 10 in 2024-09-11T03:43:02.378 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:02.399 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:02.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:02.400 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:02 smithi175 ceph-mon[30894]: pgmap v699: 57 pgs: 57 active+clean; 132 MiB data, 672 MiB used, 893 GiB / 894 GiB avail; 36 MiB/s rd, 4.5 MiB/s wr, 50 op/s 2024-09-11T03:43:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:02 smithi175 ceph-mon[30894]: osdmap e363: 12 total, 12 up, 10 in 2024-09-11T03:43:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:02 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/328461103' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:43:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:02 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/328461103' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:43:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:02 smithi175 ceph-mon[30894]: osdmap e364: 12 total, 12 up, 10 in 2024-09-11T03:43:02.501 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:02.522 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:02.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:02.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:02.534 INFO:teuthology.orchestra.run.smithi084.stdout:moved 0 / 171 (0%) 2024-09-11T03:43:02.534 INFO:teuthology.orchestra.run.smithi084.stdout:avg 17.1 2024-09-11T03:43:02.534 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 2.62488 -> 2.62488 (expected baseline 3.92301) 2024-09-11T03:43:02.535 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.2 with 14 -> 14 pgs (0.818713 -> 0.818713 * mean) 2024-09-11T03:43:02.535 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.7 with 22 -> 22 pgs (1.28655 -> 1.28655 * mean) 2024-09-11T03:43:02.535 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:43:02.535 INFO:teuthology.orchestra.run.smithi084.stdout:oload 1000 2024-09-11T03:43:02.535 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 1 2024-09-11T03:43:02.535 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2024-09-11T03:43:02.535 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0007 2024-09-11T03:43:02.535 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0072 2024-09-11T03:43:02.535 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2024-09-11T03:43:02.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:02.646 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:02.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:02.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:02.748 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:02.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:02.771 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:02.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:02.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:02.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T03:43:02.894 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:02.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:02.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:02.995 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:03.016 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:03.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:03.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:03.071 INFO:tasks.rados.rados.2.smithi175.stdout:1313: done (1 left) 2024-09-11T03:43:03.071 INFO:tasks.rados.rados.2.smithi175.stdout:1315: done (0 left) 2024-09-11T03:43:03.071 INFO:tasks.rados.rados.2.smithi175.stdout:1316: write oid 24 current snap is 134 2024-09-11T03:43:03.071 INFO:tasks.rados.rados.2.smithi175.stdout:1316: seq_num 467 ranges {648869=638816,1828032=787828,3141714=532030} 2024-09-11T03:43:03.077 INFO:tasks.rados.rados.2.smithi175.stdout:1316: writing smithi17557870-24 from 648869 to 1287685 tid 1 2024-09-11T03:43:03.085 INFO:tasks.rados.rados.2.smithi175.stdout:1316: writing smithi17557870-24 from 1828032 to 2615860 tid 2 2024-09-11T03:43:03.091 INFO:tasks.rados.rados.2.smithi175.stdout:1316: writing smithi17557870-24 from 3141714 to 3673744 tid 3 2024-09-11T03:43:03.091 INFO:tasks.rados.rados.2.smithi175.stdout:1317: snap_create 2024-09-11T03:43:03.091 INFO:tasks.rados.rados.2.smithi175.stdout:1316: finishing write tid 1 to smithi17557870-24 2024-09-11T03:43:03.091 INFO:tasks.rados.rados.2.smithi175.stdout:1316: finishing write tid 2 to smithi17557870-24 2024-09-11T03:43:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1316: finishing write tid 3 to smithi17557870-24 2024-09-11T03:43:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1316: finishing write tid 4 to smithi17557870-24 2024-09-11T03:43:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1316: finishing write tid 5 to smithi17557870-24 2024-09-11T03:43:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1316: oid 24 updating version 0 to 502 2024-09-11T03:43:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1316: oid 24 version 502 is already newer than 501 2024-09-11T03:43:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1316: oid 24 updating version 502 to 504 2024-09-11T03:43:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1316: oid 24 version 504 is already newer than 503 2024-09-11T03:43:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 504 (ObjNum 467 snap 134 seq_num 467) dirty exists 2024-09-11T03:43:03.094 INFO:tasks.rados.rados.2.smithi175.stdout:1316: left oid 24 (ObjNum 467 snap 134 seq_num 467) 2024-09-11T03:43:03.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:03.125 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:43:03.139 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:03.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:03.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:03.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:03 smithi121 ceph-mon[30729]: 2.16 scrub starts 2024-09-11T03:43:03.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:03 smithi121 ceph-mon[30729]: 2.16 scrub ok 2024-09-11T03:43:03.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:03 smithi121 ceph-mon[30729]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:43:03.241 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:03.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:03.266 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:03.266 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:03.288 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:03 smithi084 ceph-mon[27254]: 2.16 scrub starts 2024-09-11T03:43:03.288 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:03 smithi084 ceph-mon[27254]: 2.16 scrub ok 2024-09-11T03:43:03.289 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:03 smithi084 ceph-mon[27254]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:43:03.367 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:03.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:03.389 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:03.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:03.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:03 smithi175 ceph-mon[30894]: 2.16 scrub starts 2024-09-11T03:43:03.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:03 smithi175 ceph-mon[30894]: 2.16 scrub ok 2024-09-11T03:43:03.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:03 smithi175 ceph-mon[30894]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:43:03.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:03.512 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:03.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:03.613 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:43:03.613 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:43:03.613 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:43:03.614 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:43:03.614 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:43:03.615 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:03.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:03.638 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:03.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:03.671 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-11T03:43:03.671 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-09-11T03:43:03.727 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:03.739 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:03.760 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:03.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:03.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:03.862 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:03.883 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:03.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:03.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:03.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:04.006 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:04.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:04.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:04.077 INFO:tasks.rados.rados.2.smithi175.stdout:1316: done (1 left) 2024-09-11T03:43:04.078 INFO:tasks.rados.rados.2.smithi175.stdout:1317: done (0 left) 2024-09-11T03:43:04.078 INFO:tasks.rados.rados.2.smithi175.stdout:1318: snap_remove snap 134 2024-09-11T03:43:04.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:04.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:04.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:04.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:04.240 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:43:04.240 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:43:04.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:04 smithi084 ceph-mon[27254]: 2.11 scrub starts 2024-09-11T03:43:04.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:04 smithi084 ceph-mon[27254]: 2.11 scrub ok 2024-09-11T03:43:04.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:04 smithi084 ceph-mon[27254]: pgmap v702: 57 pgs: 57 active+clean; 119 MiB data, 662 MiB used, 893 GiB / 894 GiB avail; 45 MiB/s rd, 7.6 MiB/s wr, 61 op/s 2024-09-11T03:43:04.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:04 smithi084 ceph-mon[27254]: 2.4 scrub starts 2024-09-11T03:43:04.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:04 smithi084 ceph-mon[27254]: 2.4 scrub ok 2024-09-11T03:43:04.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:04 smithi084 ceph-mon[27254]: osdmap e365: 12 total, 12 up, 10 in 2024-09-11T03:43:04.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:04 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:43:04.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:04 smithi084 ceph-mon[27254]: 2.2 scrub ok 2024-09-11T03:43:04.243 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:04.253 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:04.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:04.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:04.257 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:43:04.257 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:43:04.276 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:43:04.276 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:43:04.293 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:43:04.293 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:43:04.298 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T03:43:04.309 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:43:04.310 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:43:04.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:04 smithi121 ceph-mon[30729]: 2.11 scrub starts 2024-09-11T03:43:04.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:04 smithi121 ceph-mon[30729]: 2.11 scrub ok 2024-09-11T03:43:04.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:04 smithi121 ceph-mon[30729]: pgmap v702: 57 pgs: 57 active+clean; 119 MiB data, 662 MiB used, 893 GiB / 894 GiB avail; 45 MiB/s rd, 7.6 MiB/s wr, 61 op/s 2024-09-11T03:43:04.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:04 smithi121 ceph-mon[30729]: 2.4 scrub starts 2024-09-11T03:43:04.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:04 smithi121 ceph-mon[30729]: 2.4 scrub ok 2024-09-11T03:43:04.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:04 smithi121 ceph-mon[30729]: osdmap e365: 12 total, 12 up, 10 in 2024-09-11T03:43:04.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:04 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:43:04.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:04 smithi121 ceph-mon[30729]: 2.2 scrub ok 2024-09-11T03:43:04.325 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:43:04.326 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:43:04.342 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:43:04.342 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:43:04.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:04.358 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:43:04.359 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:43:04.375 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:04.376 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:43:04.376 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:43:04.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:04.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:04.393 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:43:04.394 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:43:04.410 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:43:04.411 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:43:04.427 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:43:04.427 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:43:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:04 smithi175 ceph-mon[30894]: 2.11 scrub starts 2024-09-11T03:43:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:04 smithi175 ceph-mon[30894]: 2.11 scrub ok 2024-09-11T03:43:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:04 smithi175 ceph-mon[30894]: pgmap v702: 57 pgs: 57 active+clean; 119 MiB data, 662 MiB used, 893 GiB / 894 GiB avail; 45 MiB/s rd, 7.6 MiB/s wr, 61 op/s 2024-09-11T03:43:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:04 smithi175 ceph-mon[30894]: 2.4 scrub starts 2024-09-11T03:43:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:04 smithi175 ceph-mon[30894]: 2.4 scrub ok 2024-09-11T03:43:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:04 smithi175 ceph-mon[30894]: osdmap e365: 12 total, 12 up, 10 in 2024-09-11T03:43:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:04 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:43:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:04 smithi175 ceph-mon[30894]: 2.2 scrub ok 2024-09-11T03:43:04.478 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:04.499 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:04.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:04.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:04.541 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:43:04.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:04.623 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:04.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:04.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:04.725 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:04.751 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:04.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:04.752 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:04.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:04.873 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:04.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:04.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:04.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:43:04.975 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:05.001 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:05.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:05.002 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:05.070 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:43:05.070 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:43:05.090 INFO:tasks.rados.rados.2.smithi175.stdout:1318: done (0 left) 2024-09-11T03:43:05.090 INFO:tasks.rados.rados.2.smithi175.stdout:1319: snap_remove snap 133 2024-09-11T03:43:05.103 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:05.124 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:05.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:05.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:05.226 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:05.247 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:05.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:05.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:05.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:05 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:43:05.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:05 smithi084 ceph-mon[27254]: 2.8 scrub ok 2024-09-11T03:43:05.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:05 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:43:05.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:05 smithi084 ceph-mon[27254]: 2.9 scrub ok 2024-09-11T03:43:05.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:05 smithi084 ceph-mon[27254]: osdmap e366: 12 total, 12 up, 10 in 2024-09-11T03:43:05.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:05 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/446795586' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:05.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:05 smithi084 ceph-mon[27254]: 2.5 scrub starts 2024-09-11T03:43:05.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:05 smithi084 ceph-mon[27254]: 2.5 scrub ok 2024-09-11T03:43:05.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:05 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:43:05.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:05 smithi084 ceph-mon[27254]: 2.14 scrub ok 2024-09-11T03:43:05.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:05 smithi084 ceph-mon[27254]: 2.12 scrub starts 2024-09-11T03:43:05.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:05 smithi084 ceph-mon[27254]: 2.12 scrub ok 2024-09-11T03:43:05.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:05.370 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:05.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:05.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:05.385 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:05.416 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:05 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:43:05.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:05 smithi121 ceph-mon[30729]: 2.8 scrub ok 2024-09-11T03:43:05.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:05 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:43:05.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:05 smithi121 ceph-mon[30729]: 2.9 scrub ok 2024-09-11T03:43:05.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:05 smithi121 ceph-mon[30729]: osdmap e366: 12 total, 12 up, 10 in 2024-09-11T03:43:05.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:05 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/446795586' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:05.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:05 smithi121 ceph-mon[30729]: 2.5 scrub starts 2024-09-11T03:43:05.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:05 smithi121 ceph-mon[30729]: 2.5 scrub ok 2024-09-11T03:43:05.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:05 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:43:05.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:05 smithi121 ceph-mon[30729]: 2.14 scrub ok 2024-09-11T03:43:05.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:05 smithi121 ceph-mon[30729]: 2.12 scrub starts 2024-09-11T03:43:05.417 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:05 smithi121 ceph-mon[30729]: 2.12 scrub ok 2024-09-11T03:43:05.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:05 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:43:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:05 smithi175 ceph-mon[30894]: 2.8 scrub ok 2024-09-11T03:43:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:05 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:43:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:05 smithi175 ceph-mon[30894]: 2.9 scrub ok 2024-09-11T03:43:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:05 smithi175 ceph-mon[30894]: osdmap e366: 12 total, 12 up, 10 in 2024-09-11T03:43:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:05 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/446795586' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:05 smithi175 ceph-mon[30894]: 2.5 scrub starts 2024-09-11T03:43:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:05 smithi175 ceph-mon[30894]: 2.5 scrub ok 2024-09-11T03:43:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:05 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:43:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:05 smithi175 ceph-mon[30894]: 2.14 scrub ok 2024-09-11T03:43:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:05 smithi175 ceph-mon[30894]: 2.12 scrub starts 2024-09-11T03:43:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:05 smithi175 ceph-mon[30894]: 2.12 scrub ok 2024-09-11T03:43:05.494 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:05.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:05.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:05.534 INFO:teuthology.orchestra.run.smithi084.stdout:moved 0 / 171 (0%) 2024-09-11T03:43:05.535 INFO:teuthology.orchestra.run.smithi084.stdout:avg 17.1 2024-09-11T03:43:05.535 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 2.62488 -> 2.62488 (expected baseline 3.92301) 2024-09-11T03:43:05.535 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.2 with 14 -> 14 pgs (0.818713 -> 0.818713 * mean) 2024-09-11T03:43:05.535 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.7 with 22 -> 22 pgs (1.28655 -> 1.28655 * mean) 2024-09-11T03:43:05.535 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:43:05.535 INFO:teuthology.orchestra.run.smithi084.stdout:oload 1000 2024-09-11T03:43:05.535 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 0.05 2024-09-11T03:43:05.535 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2024-09-11T03:43:05.535 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0007 2024-09-11T03:43:05.535 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0070 2024-09-11T03:43:05.536 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2024-09-11T03:43:05.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:05.616 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:05.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:05.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:05.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:05.740 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:05.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:05.760 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-11T03:43:05.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:05.865 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:05.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:05.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:05.967 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:05.990 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:05.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:06.015 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:43:06.087 INFO:tasks.rados.rados.2.smithi175.stdout:1319: done (0 left) 2024-09-11T03:43:06.088 INFO:tasks.rados.rados.2.smithi175.stdout:1320: write (excl) oid 48 current snap is 135 2024-09-11T03:43:06.088 INFO:tasks.rados.rados.2.smithi175.stdout:1320: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2024-09-11T03:43:06.090 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:06.095 INFO:tasks.rados.rados.2.smithi175.stdout:1320: writing smithi17557870-48 from 665676 to 1379741 tid 1 2024-09-11T03:43:06.101 INFO:tasks.rados.rados.2.smithi175.stdout:1320: writing smithi17557870-48 from 1886514 to 2418000 tid 2 2024-09-11T03:43:06.109 INFO:tasks.rados.rados.2.smithi175.stdout:1320: writing smithi17557870-48 from 3169137 to 3865676 tid 3 2024-09-11T03:43:06.109 INFO:tasks.rados.rados.2.smithi175.stdout:1321: delete oid 20 current snap is 135 2024-09-11T03:43:06.109 INFO:tasks.rados.rados.2.smithi175.stdout:1320: finishing write tid 1 to smithi17557870-48 2024-09-11T03:43:06.109 INFO:tasks.rados.rados.2.smithi175.stdout:1320: finishing write tid 2 to smithi17557870-48 2024-09-11T03:43:06.112 INFO:tasks.rados.rados.2.smithi175.stdout:1320: finishing write tid 3 to smithi17557870-48 2024-09-11T03:43:06.112 INFO:tasks.rados.rados.2.smithi175.stdout:1320: finishing write tid 4 to smithi17557870-48 2024-09-11T03:43:06.112 INFO:tasks.rados.rados.2.smithi175.stdout:1320: finishing write tid 5 to smithi17557870-48 2024-09-11T03:43:06.113 INFO:tasks.rados.rados.2.smithi175.stdout:1320: oid 48 updating version 0 to 449 2024-09-11T03:43:06.113 INFO:tasks.rados.rados.2.smithi175.stdout:1320: oid 48 updating version 449 to 450 2024-09-11T03:43:06.113 INFO:tasks.rados.rados.2.smithi175.stdout:1320: oid 48 updating version 450 to 451 2024-09-11T03:43:06.113 INFO:tasks.rados.rados.2.smithi175.stdout:1320: oid 48 version 451 is already newer than 448 2024-09-11T03:43:06.113 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 451 (ObjNum 468 snap 135 seq_num 468) dirty exists 2024-09-11T03:43:06.113 INFO:tasks.rados.rados.2.smithi175.stdout:1320: left oid 48 (ObjNum 468 snap 135 seq_num 468) 2024-09-11T03:43:06.113 INFO:tasks.rados.rados.2.smithi175.stdout:1320: done (1 left) 2024-09-11T03:43:06.113 INFO:tasks.rados.rados.2.smithi175.stdout:1321: done (0 left) 2024-09-11T03:43:06.113 INFO:tasks.rados.rados.2.smithi175.stdout:1322: snap_remove snap 131 2024-09-11T03:43:06.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:06.114 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:06.114 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:06.215 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:06.236 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:06.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:06.338 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:06.360 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:06.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:06.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:06 smithi084 ceph-mon[27254]: pgmap v705: 57 pgs: 57 active+clean; 121 MiB data, 644 MiB used, 893 GiB / 894 GiB avail; 39 MiB/s rd, 8.0 MiB/s wr, 56 op/s 2024-09-11T03:43:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:06 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:43:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:06 smithi084 ceph-mon[27254]: 2.7 scrub ok 2024-09-11T03:43:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:06 smithi084 ceph-mon[27254]: osdmap e367: 12 total, 12 up, 10 in 2024-09-11T03:43:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:06 smithi084 ceph-mon[27254]: from='client.15702 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:43:06.412 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:06 smithi175 ceph-mon[30894]: pgmap v705: 57 pgs: 57 active+clean; 121 MiB data, 644 MiB used, 893 GiB / 894 GiB avail; 39 MiB/s rd, 8.0 MiB/s wr, 56 op/s 2024-09-11T03:43:06.412 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:06 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:43:06.412 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:06 smithi175 ceph-mon[30894]: 2.7 scrub ok 2024-09-11T03:43:06.413 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:06 smithi175 ceph-mon[30894]: osdmap e367: 12 total, 12 up, 10 in 2024-09-11T03:43:06.413 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:06 smithi175 ceph-mon[30894]: from='client.15702 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:43:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:06 smithi121 ceph-mon[30729]: pgmap v705: 57 pgs: 57 active+clean; 121 MiB data, 644 MiB used, 893 GiB / 894 GiB avail; 39 MiB/s rd, 8.0 MiB/s wr, 56 op/s 2024-09-11T03:43:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:06 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:43:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:06 smithi121 ceph-mon[30729]: 2.7 scrub ok 2024-09-11T03:43:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:06 smithi121 ceph-mon[30729]: osdmap e367: 12 total, 12 up, 10 in 2024-09-11T03:43:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:06 smithi121 ceph-mon[30729]: from='client.15702 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:43:06.462 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:06.485 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:06.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:06.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:06.520 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:43:06.521 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:43:06.587 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:06.609 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:06.610 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:06.711 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:06.733 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:06.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:06.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:06.811 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-11T03:43:06.811 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd test-reweight-by-utilization 110 1.0 2024-09-11T03:43:06.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:06.861 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:06.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:06.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:06.893 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:43:06.898 INFO:tasks.rados.rados.2.smithi175.stdout:1322: done (0 left) 2024-09-11T03:43:06.898 INFO:tasks.rados.rados.2.smithi175.stdout:1323: snap_create 2024-09-11T03:43:06.963 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:06.985 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:06.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:06.986 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:07.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:07.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:07.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:07.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:07 smithi084 ceph-mon[27254]: 1.0 scrub starts 2024-09-11T03:43:07.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:07 smithi084 ceph-mon[27254]: 1.0 scrub ok 2024-09-11T03:43:07.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:07 smithi084 ceph-mon[27254]: 2.17 scrub starts 2024-09-11T03:43:07.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:07 smithi084 ceph-mon[27254]: 2.17 scrub ok 2024-09-11T03:43:07.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:07 smithi084 ceph-mon[27254]: osdmap e368: 12 total, 12 up, 10 in 2024-09-11T03:43:07.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/921804624' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:07.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:07 smithi084 ceph-mon[27254]: 2.13 scrub starts 2024-09-11T03:43:07.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:07 smithi084 ceph-mon[27254]: 2.13 scrub ok 2024-09-11T03:43:07.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:07 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:07.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:07 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:43:07.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:07 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:43:07.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/921804624' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:07.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:07 smithi084 ceph-mon[27254]: osdmap e369: 12 total, 12 up, 10 in 2024-09-11T03:43:07.166 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:43:07.214 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:07.227 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-11T03:43:07.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:07.236 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:07.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:07.337 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:07.358 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:07.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:07.359 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:07 smithi121 ceph-mon[30729]: 1.0 scrub starts 2024-09-11T03:43:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:07 smithi121 ceph-mon[30729]: 1.0 scrub ok 2024-09-11T03:43:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:07 smithi121 ceph-mon[30729]: 2.17 scrub starts 2024-09-11T03:43:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:07 smithi121 ceph-mon[30729]: 2.17 scrub ok 2024-09-11T03:43:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:07 smithi121 ceph-mon[30729]: osdmap e368: 12 total, 12 up, 10 in 2024-09-11T03:43:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/921804624' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:07 smithi121 ceph-mon[30729]: 2.13 scrub starts 2024-09-11T03:43:07.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:07 smithi121 ceph-mon[30729]: 2.13 scrub ok 2024-09-11T03:43:07.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:07 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:07.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:07 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:43:07.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:07 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:43:07.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/921804624' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:07.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:07 smithi121 ceph-mon[30729]: osdmap e369: 12 total, 12 up, 10 in 2024-09-11T03:43:07.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:07 smithi175 ceph-mon[30894]: 1.0 scrub starts 2024-09-11T03:43:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:07 smithi175 ceph-mon[30894]: 1.0 scrub ok 2024-09-11T03:43:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:07 smithi175 ceph-mon[30894]: 2.17 scrub starts 2024-09-11T03:43:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:07 smithi175 ceph-mon[30894]: 2.17 scrub ok 2024-09-11T03:43:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:07 smithi175 ceph-mon[30894]: osdmap e368: 12 total, 12 up, 10 in 2024-09-11T03:43:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/921804624' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:07 smithi175 ceph-mon[30894]: 2.13 scrub starts 2024-09-11T03:43:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:07 smithi175 ceph-mon[30894]: 2.13 scrub ok 2024-09-11T03:43:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:07 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:07 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:43:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:07 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:43:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/921804624' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:07 smithi175 ceph-mon[30894]: osdmap e369: 12 total, 12 up, 10 in 2024-09-11T03:43:07.481 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:43:07.485 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:07.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:07.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:07.587 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:07.607 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:07.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:07.609 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:07.627 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:07.643 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:07.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:07.735 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:07.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:07.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:07.837 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:07.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:07.859 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:07.860 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:07.900 INFO:tasks.rados.rados.2.smithi175.stdout:1323: done (0 left) 2024-09-11T03:43:07.900 INFO:tasks.rados.rados.2.smithi175.stdout:1324: read oid 22 snap -1 2024-09-11T03:43:07.900 INFO:tasks.rados.rados.2.smithi175.stdout:1324: expect deleted 2024-09-11T03:43:07.900 INFO:tasks.rados.rados.2.smithi175.stdout:1325: read oid 17 snap -1 2024-09-11T03:43:07.900 INFO:tasks.rados.rados.2.smithi175.stdout:1325: expect (ObjNum 425 snap 129 seq_num 425) 2024-09-11T03:43:07.900 INFO:tasks.rados.rados.2.smithi175.stdout:1326: snap_create 2024-09-11T03:43:07.910 INFO:tasks.rados.rados.2.smithi175.stdout:1325: expect (ObjNum 425 snap 129 seq_num 425) 2024-09-11T03:43:07.961 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:07.965 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:43:07.965 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:43:07.983 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:07.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:07.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:08.084 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:08.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:08.107 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:08.107 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:08.209 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:08 smithi175 ceph-mon[30894]: pgmap v708: 57 pgs: 57 active+clean; 123 MiB data, 646 MiB used, 893 GiB / 894 GiB avail; 4.7 MiB/s rd, 2.7 MiB/s wr, 16 op/s 2024-09-11T03:43:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:08 smithi175 ceph-mon[30894]: osdmap e370: 12 total, 12 up, 10 in 2024-09-11T03:43:08.230 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:08.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:08.231 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:08.332 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:08.343 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:43:08.343 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:43:08.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:08.355 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:08.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:08.360 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:43:08.360 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:43:08.361 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:08 smithi084 ceph-mon[27254]: pgmap v708: 57 pgs: 57 active+clean; 123 MiB data, 646 MiB used, 893 GiB / 894 GiB avail; 4.7 MiB/s rd, 2.7 MiB/s wr, 16 op/s 2024-09-11T03:43:08.361 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:08 smithi084 ceph-mon[27254]: osdmap e370: 12 total, 12 up, 10 in 2024-09-11T03:43:08.377 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:43:08.377 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:43:08.392 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:43:08.393 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:43:08.409 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:43:08.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:43:08.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:08 smithi121 ceph-mon[30729]: pgmap v708: 57 pgs: 57 active+clean; 123 MiB data, 646 MiB used, 893 GiB / 894 GiB avail; 4.7 MiB/s rd, 2.7 MiB/s wr, 16 op/s 2024-09-11T03:43:08.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:08 smithi121 ceph-mon[30729]: osdmap e370: 12 total, 12 up, 10 in 2024-09-11T03:43:08.426 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:43:08.426 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:43:08.442 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:43:08.442 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:43:08.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:08.459 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:43:08.459 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:43:08.475 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:43:08.476 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:43:08.478 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:08.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:08.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:08.492 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:43:08.493 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:43:08.510 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:43:08.527 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:43:08.528 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:43:08.579 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:08.601 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:08.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:08.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:08.701 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-11T03:43:08.703 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:08.722 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:08.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:08.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:08.761 INFO:teuthology.orchestra.run.smithi084.stdout:moved 4 / 171 (2.33918%) 2024-09-11T03:43:08.761 INFO:teuthology.orchestra.run.smithi084.stdout:avg 17.1 2024-09-11T03:43:08.761 INFO:teuthology.orchestra.run.smithi084.stdout:stddev 2.62488 -> 2.11896 (expected baseline 3.92301) 2024-09-11T03:43:08.761 INFO:teuthology.orchestra.run.smithi084.stdout:min osd.2 with 14 -> 14 pgs (0.818713 -> 0.818713 * mean) 2024-09-11T03:43:08.761 INFO:teuthology.orchestra.run.smithi084.stdout:max osd.7 with 22 -> 19 pgs (1.28655 -> 1.11111 * mean) 2024-09-11T03:43:08.761 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:43:08.761 INFO:teuthology.orchestra.run.smithi084.stdout:oload 110 2024-09-11T03:43:08.761 INFO:teuthology.orchestra.run.smithi084.stdout:max_change 1 2024-09-11T03:43:08.761 INFO:teuthology.orchestra.run.smithi084.stdout:max_change_osds 4 2024-09-11T03:43:08.762 INFO:teuthology.orchestra.run.smithi084.stdout:average_utilization 0.0007 2024-09-11T03:43:08.762 INFO:teuthology.orchestra.run.smithi084.stdout:overload_utilization 0.0008 2024-09-11T03:43:08.762 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3 weight 1.0000 -> 0.7578 2024-09-11T03:43:08.762 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7 weight 1.0000 -> 0.8933 2024-09-11T03:43:08.762 INFO:teuthology.orchestra.run.smithi084.stderr:no change 2024-09-11T03:43:08.824 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:08.845 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:08.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:08.846 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:08.901 INFO:tasks.rados.rados.2.smithi175.stdout:1324: done (2 left) 2024-09-11T03:43:08.901 INFO:tasks.rados.rados.2.smithi175.stdout:1325: done (1 left) 2024-09-11T03:43:08.901 INFO:tasks.rados.rados.2.smithi175.stdout:1326: done (0 left) 2024-09-11T03:43:08.902 INFO:tasks.rados.rados.2.smithi175.stdout:1327: read oid 17 snap -1 2024-09-11T03:43:08.902 INFO:tasks.rados.rados.2.smithi175.stdout:1327: expect (ObjNum 425 snap 129 seq_num 425) 2024-09-11T03:43:08.902 INFO:tasks.rados.rados.2.smithi175.stdout:1328: delete oid 25 current snap is 137 2024-09-11T03:43:08.906 INFO:tasks.rados.rados.2.smithi175.stdout:1328: done (1 left) 2024-09-11T03:43:08.906 INFO:tasks.rados.rados.2.smithi175.stdout:1329: snap_remove snap 130 2024-09-11T03:43:08.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:08.948 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:43:08.969 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:08.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:08.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:09.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:09.092 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:09.092 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:43:08 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: 2024-09-11T03:43:08.760+0000 7fab40ce9640 -1 mgr.server reply reply (2) No such file or directory no change 2024-09-11T03:43:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:09.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:09.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:09.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:09.216 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:09.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:09.317 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:09.339 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:09.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:09.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:09.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:09 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/345853125' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:09.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:09 smithi084 ceph-mon[27254]: 1.7 scrub starts 2024-09-11T03:43:09.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:09 smithi084 ceph-mon[27254]: 1.7 scrub ok 2024-09-11T03:43:09.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:09 smithi084 ceph-mon[27254]: osdmap e371: 12 total, 12 up, 10 in 2024-09-11T03:43:09.403 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:09 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/345853125' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:09.403 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:09 smithi175 ceph-mon[30894]: 1.7 scrub starts 2024-09-11T03:43:09.403 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:09 smithi175 ceph-mon[30894]: 1.7 scrub ok 2024-09-11T03:43:09.403 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:09 smithi175 ceph-mon[30894]: osdmap e371: 12 total, 12 up, 10 in 2024-09-11T03:43:09.405 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:43:09.405 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:43:09.406 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:43:09.406 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:43:09.406 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.406 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:588 2.f 2:fda69313:::smithi17557870-16:head [copy-from ver 31] snapc 11=[11,f,e] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-11T03:43:09.406 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:05.838050+0000", 2024-09-11T03:43:09.406 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 243.56678822800001, 2024-09-11T03:43:09.406 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017475905, 2024-09-11T03:43:09.406 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.406 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.406 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:09.406 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.407 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.407 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.407 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 588 2024-09-11T03:43:09.407 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.407 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.407 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.407 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.407 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838050+0000", 2024-09-11T03:43:09.407 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.407 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.407 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.408 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.408 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838050+0000", 2024-09-11T03:43:09.408 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.408 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.408 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.408 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.408 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838050+0000", 2024-09-11T03:43:09.408 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2399999999999998e-07 2024-09-11T03:43:09.408 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.408 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.408 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.409 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838054+0000", 2024-09-11T03:43:09.409 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8519999999999997e-06 2024-09-11T03:43:09.409 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.409 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.409 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.409 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838064+0000", 2024-09-11T03:43:09.409 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7559999999999994e-06 2024-09-11T03:43:09.409 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.409 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.409 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.409 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838069+0000", 2024-09-11T03:43:09.409 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4759999999999997e-06 2024-09-11T03:43:09.410 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.410 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.410 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.410 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838118+0000", 2024-09-11T03:43:09.410 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8881999999999997e-05 2024-09-11T03:43:09.410 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.410 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.410 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.410 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838180+0000", 2024-09-11T03:43:09.410 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1569000000000006e-05 2024-09-11T03:43:09.410 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.410 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.411 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,9", 2024-09-11T03:43:09.411 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.847578+0000", 2024-09-11T03:43:09.411 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0093983880000000006 2024-09-11T03:43:09.411 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.411 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.411 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:09.411 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.851581+0000", 2024-09-11T03:43:09.411 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0040029929999999998 2024-09-11T03:43:09.411 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.411 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.411 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.411 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.853070+0000", 2024-09-11T03:43:09.412 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014892130000000001 2024-09-11T03:43:09.412 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.412 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.412 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.412 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.855406+0000", 2024-09-11T03:43:09.412 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00233533 2024-09-11T03:43:09.412 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.412 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.412 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:09.412 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.855473+0000", 2024-09-11T03:43:09.412 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7726999999999997e-05 2024-09-11T03:43:09.412 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.413 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.413 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.413 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.855526+0000", 2024-09-11T03:43:09.413 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2095000000000002e-05 2024-09-11T03:43:09.413 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.413 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.413 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.413 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.413 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.413 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:589 2.f 2:fda69313:::smithi17557870-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e129)", 2024-09-11T03:43:09.413 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:05.838080+0000", 2024-09-11T03:43:09.413 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 243.5667579, 2024-09-11T03:43:09.414 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013050325, 2024-09-11T03:43:09.414 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.414 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.414 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:09.414 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.414 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.414 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.414 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 589 2024-09-11T03:43:09.414 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.414 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.414 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.414 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.415 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838080+0000", 2024-09-11T03:43:09.415 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.415 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.415 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.415 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.415 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838080+0000", 2024-09-11T03:43:09.415 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.415 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.415 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.415 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.415 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838081+0000", 2024-09-11T03:43:09.415 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8699999999999995e-07 2024-09-11T03:43:09.415 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.416 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.416 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.416 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838083+0000", 2024-09-11T03:43:09.416 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3690000000000001e-06 2024-09-11T03:43:09.416 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.416 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.416 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.416 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838090+0000", 2024-09-11T03:43:09.416 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.669e-06 2024-09-11T03:43:09.416 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.416 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.416 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.417 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838095+0000", 2024-09-11T03:43:09.417 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1850000000000003e-06 2024-09-11T03:43:09.417 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.417 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.417 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.417 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838299+0000", 2024-09-11T03:43:09.417 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00020415400000000001 2024-09-11T03:43:09.417 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.417 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.417 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:09.417 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838327+0000", 2024-09-11T03:43:09.417 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8045e-05 2024-09-11T03:43:09.417 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.417 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.418 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.418 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.850913+0000", 2024-09-11T03:43:09.418 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012586409 2024-09-11T03:43:09.418 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.418 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.418 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.418 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.851013+0000", 2024-09-11T03:43:09.418 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9287e-05 2024-09-11T03:43:09.418 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.418 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.418 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.418 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.851130+0000", 2024-09-11T03:43:09.419 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011772 2024-09-11T03:43:09.419 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.419 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.419 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.419 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.419 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.419 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:985 2.b 2:dde92fc6:::smithi17557870-41:head [copy-from ver 46] snapc 24=[24,23,22,20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-11T03:43:09.419 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:39.929714+0000", 2024-09-11T03:43:09.419 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 209.47512391000001, 2024-09-11T03:43:09.419 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012702392999999999, 2024-09-11T03:43:09.419 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.419 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.419 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:09.420 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.420 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.420 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.420 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 985 2024-09-11T03:43:09.420 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.420 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.420 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.420 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.420 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.929714+0000", 2024-09-11T03:43:09.420 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.420 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.420 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.421 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.421 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.929714+0000", 2024-09-11T03:43:09.421 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.421 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.421 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.421 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.421 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.929718+0000", 2024-09-11T03:43:09.421 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7500000000000001e-06 2024-09-11T03:43:09.421 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.421 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.421 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:09 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/345853125' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:09 smithi121 ceph-mon[30729]: 1.7 scrub starts 2024-09-11T03:43:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:09 smithi121 ceph-mon[30729]: 1.7 scrub ok 2024-09-11T03:43:09.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:09 smithi121 ceph-mon[30729]: osdmap e371: 12 total, 12 up, 10 in 2024-09-11T03:43:09.423 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.929732+0000", 2024-09-11T03:43:09.423 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4392999999999999e-05 2024-09-11T03:43:09.423 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.423 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.423 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.423 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.929762+0000", 2024-09-11T03:43:09.424 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0204000000000001e-05 2024-09-11T03:43:09.424 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.424 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.424 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.424 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.929775+0000", 2024-09-11T03:43:09.424 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2758000000000001e-05 2024-09-11T03:43:09.424 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.424 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.424 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.425 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.929843+0000", 2024-09-11T03:43:09.425 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8251999999999996e-05 2024-09-11T03:43:09.425 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.425 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.425 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.425 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.929911+0000", 2024-09-11T03:43:09.425 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7805000000000006e-05 2024-09-11T03:43:09.425 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.425 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.425 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,11", 2024-09-11T03:43:09.426 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932521+0000", 2024-09-11T03:43:09.426 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0026096679999999999 2024-09-11T03:43:09.426 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.426 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.426 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.426 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.938393+0000", 2024-09-11T03:43:09.426 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0058723569999999999 2024-09-11T03:43:09.426 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.426 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.426 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:09.426 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.942180+0000", 2024-09-11T03:43:09.426 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0037872610000000001 2024-09-11T03:43:09.426 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.426 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.427 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.427 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.942321+0000", 2024-09-11T03:43:09.427 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00014012399999999999 2024-09-11T03:43:09.427 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.427 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.427 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:09.427 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.942360+0000", 2024-09-11T03:43:09.427 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9505000000000002e-05 2024-09-11T03:43:09.427 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.427 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.427 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.427 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.942416+0000", 2024-09-11T03:43:09.428 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6316000000000001e-05 2024-09-11T03:43:09.428 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.428 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.428 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.428 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.428 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.428 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1152 2.b 2:de6b1316:::smithi17557870-32:head [copy-from ver 77] snapc 2d=[2d,2c,2b,28,26] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-11T03:43:09.428 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:59.869443+0000", 2024-09-11T03:43:09.428 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 189.53539524199999, 2024-09-11T03:43:09.428 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015544123999999999, 2024-09-11T03:43:09.428 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.428 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.429 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:09.429 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.429 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.429 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.429 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1152 2024-09-11T03:43:09.429 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.429 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.429 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.429 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.429 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869443+0000", 2024-09-11T03:43:09.429 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.429 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.429 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.430 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.430 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869443+0000", 2024-09-11T03:43:09.430 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.430 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.430 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.430 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.430 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869444+0000", 2024-09-11T03:43:09.430 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3829999999999999e-06 2024-09-11T03:43:09.430 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.430 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.430 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.430 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869454+0000", 2024-09-11T03:43:09.431 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0166999999999999e-05 2024-09-11T03:43:09.431 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.431 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.431 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.431 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869467+0000", 2024-09-11T03:43:09.431 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3243e-05 2024-09-11T03:43:09.431 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.431 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.431 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.431 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869475+0000", 2024-09-11T03:43:09.431 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7209999999999994e-06 2024-09-11T03:43:09.431 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.431 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.431 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.432 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869537+0000", 2024-09-11T03:43:09.432 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2075000000000005e-05 2024-09-11T03:43:09.432 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.432 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.432 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.432 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869625+0000", 2024-09-11T03:43:09.432 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7736000000000002e-05 2024-09-11T03:43:09.432 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.432 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.432 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,11", 2024-09-11T03:43:09.432 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.878093+0000", 2024-09-11T03:43:09.432 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0084675710000000001 2024-09-11T03:43:09.433 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.433 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.433 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:09.433 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.882577+0000", 2024-09-11T03:43:09.433 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0044839349999999997 2024-09-11T03:43:09.433 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.433 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.433 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.433 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.883819+0000", 2024-09-11T03:43:09.433 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012421680000000001 2024-09-11T03:43:09.433 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.433 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.433 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.434 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.884915+0000", 2024-09-11T03:43:09.434 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0010967559999999999 2024-09-11T03:43:09.434 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.434 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.434 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:09.434 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.884952+0000", 2024-09-11T03:43:09.434 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6835999999999998e-05 2024-09-11T03:43:09.434 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.434 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.434 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.434 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.884987+0000", 2024-09-11T03:43:09.434 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4533000000000003e-05 2024-09-11T03:43:09.434 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.434 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.435 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.435 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.435 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.435 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1153 2.b 2:de6b1316:::smithi17557870-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e183)", 2024-09-11T03:43:09.435 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:59.869502+0000", 2024-09-11T03:43:09.435 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 189.53533553899999, 2024-09-11T03:43:09.435 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012503205, 2024-09-11T03:43:09.435 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.435 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.435 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:09.435 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.435 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.435 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.436 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1153 2024-09-11T03:43:09.436 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.436 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.436 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.436 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.436 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869502+0000", 2024-09-11T03:43:09.436 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.436 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.436 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.436 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.436 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869502+0000", 2024-09-11T03:43:09.436 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.437 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.437 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.437 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.437 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869503+0000", 2024-09-11T03:43:09.437 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.32e-07 2024-09-11T03:43:09.437 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.437 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.437 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.437 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869505+0000", 2024-09-11T03:43:09.437 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2809999999999998e-06 2024-09-11T03:43:09.437 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.437 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.437 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.438 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869511+0000", 2024-09-11T03:43:09.438 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6890000000000001e-06 2024-09-11T03:43:09.438 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.438 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.438 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.438 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869516+0000", 2024-09-11T03:43:09.438 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3600000000000004e-06 2024-09-11T03:43:09.438 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.438 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.438 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.438 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869774+0000", 2024-09-11T03:43:09.438 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00025756299999999999 2024-09-11T03:43:09.438 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.439 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.439 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:09.439 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869841+0000", 2024-09-11T03:43:09.439 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7557000000000004e-05 2024-09-11T03:43:09.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.439 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.439 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.439 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.881848+0000", 2024-09-11T03:43:09.439 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012007085000000001 2024-09-11T03:43:09.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.439 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.439 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.439 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.881923+0000", 2024-09-11T03:43:09.440 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4932999999999996e-05 2024-09-11T03:43:09.440 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.440 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.440 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.440 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.882006+0000", 2024-09-11T03:43:09.440 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2305000000000006e-05 2024-09-11T03:43:09.440 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.440 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.440 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.440 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.440 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.440 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1513 2.5 2:a6e3bbd9:::smithi17557870-38:head [copy-from ver 208] snapc 3b=[3a,38,37,35] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-09-11T03:43:09.440 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:28.917633+0000", 2024-09-11T03:43:09.441 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 160.48720508700001, 2024-09-11T03:43:09.441 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021282327, 2024-09-11T03:43:09.441 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.441 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.441 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:09.441 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.441 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.441 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.441 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1513 2024-09-11T03:43:09.441 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.441 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.441 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.442 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.442 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917633+0000", 2024-09-11T03:43:09.442 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.442 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.442 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.442 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.442 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917633+0000", 2024-09-11T03:43:09.442 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.442 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.442 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.442 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.442 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917635+0000", 2024-09-11T03:43:09.442 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6390000000000001e-06 2024-09-11T03:43:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917649+0000", 2024-09-11T03:43:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3721e-05 2024-09-11T03:43:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917669+0000", 2024-09-11T03:43:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9423999999999999e-05 2024-09-11T03:43:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917676+0000", 2024-09-11T03:43:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7330000000000003e-06 2024-09-11T03:43:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917747+0000", 2024-09-11T03:43:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1081999999999995e-05 2024-09-11T03:43:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917816+0000", 2024-09-11T03:43:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8504999999999995e-05 2024-09-11T03:43:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.445 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.445 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:43:09.445 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.928464+0000", 2024-09-11T03:43:09.445 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.01064815 2024-09-11T03:43:09.445 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.445 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.445 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:09.445 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.937323+0000", 2024-09-11T03:43:09.445 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0088585160000000003 2024-09-11T03:43:09.445 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.445 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.445 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.445 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.938350+0000", 2024-09-11T03:43:09.446 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00102728 2024-09-11T03:43:09.446 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.446 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.446 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.446 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.938848+0000", 2024-09-11T03:43:09.446 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00049766199999999995 2024-09-11T03:43:09.446 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.446 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.446 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:09.446 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.938884+0000", 2024-09-11T03:43:09.446 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6916999999999999e-05 2024-09-11T03:43:09.446 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.446 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.447 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.447 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.938915+0000", 2024-09-11T03:43:09.447 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0698000000000003e-05 2024-09-11T03:43:09.447 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.447 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.447 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.447 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.447 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.447 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1514 2.5 2:a6e3bbd9:::smithi17557870-38:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2024-09-11T03:43:09.447 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:28.917699+0000", 2024-09-11T03:43:09.447 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 160.487138727, 2024-09-11T03:43:09.447 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017658610000000002, 2024-09-11T03:43:09.447 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.448 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.448 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:09.448 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.448 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.448 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.448 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1514 2024-09-11T03:43:09.448 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.448 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.448 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.448 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.448 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917699+0000", 2024-09-11T03:43:09.449 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.449 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.449 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.449 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.449 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917699+0000", 2024-09-11T03:43:09.449 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.449 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.449 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.449 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.449 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917700+0000", 2024-09-11T03:43:09.449 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.99e-07 2024-09-11T03:43:09.449 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.449 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.450 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.450 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917703+0000", 2024-09-11T03:43:09.450 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8260000000000001e-06 2024-09-11T03:43:09.450 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.450 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.450 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.450 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917713+0000", 2024-09-11T03:43:09.450 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0539e-05 2024-09-11T03:43:09.450 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.450 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.450 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.451 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917719+0000", 2024-09-11T03:43:09.451 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1500000000000004e-06 2024-09-11T03:43:09.451 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.451 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.451 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.451 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917971+0000", 2024-09-11T03:43:09.451 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00025184900000000001 2024-09-11T03:43:09.451 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.451 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.451 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:09.451 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.918032+0000", 2024-09-11T03:43:09.452 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0449999999999999e-05 2024-09-11T03:43:09.452 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.452 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.452 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.452 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.935206+0000", 2024-09-11T03:43:09.452 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017174405 2024-09-11T03:43:09.452 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.452 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.452 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.452 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.935277+0000", 2024-09-11T03:43:09.452 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0862e-05 2024-09-11T03:43:09.452 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.452 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.453 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.453 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.935358+0000", 2024-09-11T03:43:09.453 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0929999999999999e-05 2024-09-11T03:43:09.453 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.453 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.453 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.453 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.453 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.453 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1583 2.b 2:de6b1316:::smithi17557870-32:head [copy-from ver 82] snapc 3f=[3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-09-11T03:43:09.453 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:34.845880+0000", 2024-09-11T03:43:09.453 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 154.558958308, 2024-09-11T03:43:09.453 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020941156999999998, 2024-09-11T03:43:09.454 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.454 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.454 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:09.454 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.454 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.454 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.454 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1583 2024-09-11T03:43:09.454 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.454 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.454 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.454 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.454 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845880+0000", 2024-09-11T03:43:09.455 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.455 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.455 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.455 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.455 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845880+0000", 2024-09-11T03:43:09.455 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.455 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.455 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.455 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.455 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845881+0000", 2024-09-11T03:43:09.455 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.871e-06 2024-09-11T03:43:09.455 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.455 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.456 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.456 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845891+0000", 2024-09-11T03:43:09.456 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2410000000000001e-06 2024-09-11T03:43:09.456 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.456 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.456 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.456 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845909+0000", 2024-09-11T03:43:09.456 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8162999999999999e-05 2024-09-11T03:43:09.456 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.456 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.456 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.456 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845919+0000", 2024-09-11T03:43:09.457 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9869999999999994e-06 2024-09-11T03:43:09.457 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.457 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.457 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.457 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845994+0000", 2024-09-11T03:43:09.457 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4757000000000003e-05 2024-09-11T03:43:09.457 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.457 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.457 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:43:09.457 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845999+0000", 2024-09-11T03:43:09.457 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.434e-06 2024-09-11T03:43:09.457 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.458 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.458 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.458 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.847498+0000", 2024-09-11T03:43:09.458 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014989370000000001 2024-09-11T03:43:09.458 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.458 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.458 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.458 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.847585+0000", 2024-09-11T03:43:09.458 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7058999999999994e-05 2024-09-11T03:43:09.458 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.458 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.458 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,11", 2024-09-11T03:43:09.458 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.860229+0000", 2024-09-11T03:43:09.458 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012643765 2024-09-11T03:43:09.459 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.459 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.459 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:09.459 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.864139+0000", 2024-09-11T03:43:09.459 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039105019999999997 2024-09-11T03:43:09.459 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.459 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.459 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.459 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.865334+0000", 2024-09-11T03:43:09.459 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001194938 2024-09-11T03:43:09.459 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.459 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.459 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.460 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.866748+0000", 2024-09-11T03:43:09.460 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001413873 2024-09-11T03:43:09.460 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.460 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.460 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:09.460 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.866787+0000", 2024-09-11T03:43:09.460 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8571000000000001e-05 2024-09-11T03:43:09.460 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.460 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.460 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.460 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.866821+0000", 2024-09-11T03:43:09.460 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4059e-05 2024-09-11T03:43:09.460 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.461 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.461 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.461 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.461 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.461 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1584 2.b 2:de6b1316:::smithi17557870-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e218)", 2024-09-11T03:43:09.461 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:34.845942+0000", 2024-09-11T03:43:09.461 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 154.55889636000001, 2024-09-11T03:43:09.461 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017600596999999999, 2024-09-11T03:43:09.461 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.461 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.461 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:09.461 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.462 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.462 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.462 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1584 2024-09-11T03:43:09.462 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.462 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.462 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.462 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.462 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845942+0000", 2024-09-11T03:43:09.462 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.462 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.462 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.462 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.463 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845942+0000", 2024-09-11T03:43:09.463 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.463 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.463 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.463 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.463 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845942+0000", 2024-09-11T03:43:09.463 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8100000000000003e-07 2024-09-11T03:43:09.463 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.463 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.463 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.463 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845944+0000", 2024-09-11T03:43:09.463 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0899999999999999e-06 2024-09-11T03:43:09.464 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.464 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.464 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.464 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845949+0000", 2024-09-11T03:43:09.464 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3129999999999998e-06 2024-09-11T03:43:09.464 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.464 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.464 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.464 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845953+0000", 2024-09-11T03:43:09.464 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8060000000000001e-06 2024-09-11T03:43:09.464 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.465 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.465 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.465 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.846055+0000", 2024-09-11T03:43:09.465 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010187999999999999 2024-09-11T03:43:09.465 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.465 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.465 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:43:09.465 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.846062+0000", 2024-09-11T03:43:09.465 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6239999999999996e-06 2024-09-11T03:43:09.465 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.465 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.466 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.466 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.847720+0000", 2024-09-11T03:43:09.466 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001658681 2024-09-11T03:43:09.466 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.466 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.466 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:09.466 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.847773+0000", 2024-09-11T03:43:09.466 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2710999999999997e-05 2024-09-11T03:43:09.466 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.466 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.466 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.466 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.863362+0000", 2024-09-11T03:43:09.467 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015588493 2024-09-11T03:43:09.467 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.467 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.467 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.467 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.863447+0000", 2024-09-11T03:43:09.467 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5682999999999999e-05 2024-09-11T03:43:09.467 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.467 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.467 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.467 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.863542+0000", 2024-09-11T03:43:09.467 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4834999999999996e-05 2024-09-11T03:43:09.468 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.468 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.468 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.468 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.468 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.468 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1607 2.f 2:fa239c32:::smithi17557870-4:head [copy-from ver 82] snapc 40=[40,3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:43:09.468 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:35.897501+0000", 2024-09-11T03:43:09.468 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 153.50733666299999, 2024-09-11T03:43:09.468 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018113787999999999, 2024-09-11T03:43:09.468 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.469 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.469 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:09.469 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.469 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.469 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.469 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1607 2024-09-11T03:43:09.469 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.470 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.470 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.470 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.470 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897501+0000", 2024-09-11T03:43:09.470 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.470 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.470 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.470 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.470 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897501+0000", 2024-09-11T03:43:09.470 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.471 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.471 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.471 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.471 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897503+0000", 2024-09-11T03:43:09.471 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8840000000000001e-06 2024-09-11T03:43:09.471 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.471 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.471 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.471 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897510+0000", 2024-09-11T03:43:09.471 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4740000000000002e-06 2024-09-11T03:43:09.471 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.471 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.471 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.472 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897525+0000", 2024-09-11T03:43:09.472 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5702e-05 2024-09-11T03:43:09.472 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.472 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.472 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.472 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897532+0000", 2024-09-11T03:43:09.472 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7329999999999997e-06 2024-09-11T03:43:09.472 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.472 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.472 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.472 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897610+0000", 2024-09-11T03:43:09.472 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8153999999999997e-05 2024-09-11T03:43:09.472 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.473 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.473 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:43:09.473 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897618+0000", 2024-09-11T03:43:09.473 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5630000000000003e-06 2024-09-11T03:43:09.473 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.473 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.473 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.473 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.898813+0000", 2024-09-11T03:43:09.473 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0011955819999999999 2024-09-11T03:43:09.473 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.473 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.473 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.474 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.898858+0000", 2024-09-11T03:43:09.474 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5092000000000001e-05 2024-09-11T03:43:09.474 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.474 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.474 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,9", 2024-09-11T03:43:09.474 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.908861+0000", 2024-09-11T03:43:09.474 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010003025 2024-09-11T03:43:09.474 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.474 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.474 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:09.474 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.912807+0000", 2024-09-11T03:43:09.474 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039458339999999996 2024-09-11T03:43:09.474 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.475 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.475 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.475 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.914051+0000", 2024-09-11T03:43:09.475 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012439040000000001 2024-09-11T03:43:09.475 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.475 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.475 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.475 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.915500+0000", 2024-09-11T03:43:09.475 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014488839999999999 2024-09-11T03:43:09.475 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.475 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.475 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:09.476 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.915561+0000", 2024-09-11T03:43:09.476 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1290000000000004e-05 2024-09-11T03:43:09.476 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.476 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.476 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.476 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.915615+0000", 2024-09-11T03:43:09.476 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3667e-05 2024-09-11T03:43:09.476 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.476 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.476 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.476 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.476 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.476 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1608 2.f 2:fa239c32:::smithi17557870-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:43:09.477 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:35.897552+0000", 2024-09-11T03:43:09.477 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 153.50728635499999, 2024-09-11T03:43:09.477 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014781580000000001, 2024-09-11T03:43:09.477 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.477 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.477 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:09.477 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.477 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.477 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.477 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1608 2024-09-11T03:43:09.478 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.478 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.478 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.478 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.478 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897552+0000", 2024-09-11T03:43:09.478 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.478 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.478 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.478 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.478 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897552+0000", 2024-09-11T03:43:09.478 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.478 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.478 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.479 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.479 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897552+0000", 2024-09-11T03:43:09.479 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3399999999999999e-07 2024-09-11T03:43:09.479 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.479 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.479 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.479 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897558+0000", 2024-09-11T03:43:09.479 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7769999999999999e-06 2024-09-11T03:43:09.479 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.479 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.479 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.479 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897565+0000", 2024-09-11T03:43:09.480 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7410000000000003e-06 2024-09-11T03:43:09.480 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.480 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.480 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.480 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897568+0000", 2024-09-11T03:43:09.480 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6799999999999999e-06 2024-09-11T03:43:09.480 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.480 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.480 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.480 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897671+0000", 2024-09-11T03:43:09.480 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000102291 2024-09-11T03:43:09.480 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.480 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.481 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:43:09.481 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897674+0000", 2024-09-11T03:43:09.481 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6890000000000002e-06 2024-09-11T03:43:09.481 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.481 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.481 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.481 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.898967+0000", 2024-09-11T03:43:09.481 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001292996 2024-09-11T03:43:09.481 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.481 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.481 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:09.481 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.899052+0000", 2024-09-11T03:43:09.481 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5112999999999996e-05 2024-09-11T03:43:09.482 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.482 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.482 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.482 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.912099+0000", 2024-09-11T03:43:09.482 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013046775 2024-09-11T03:43:09.482 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.482 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.482 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.482 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.912198+0000", 2024-09-11T03:43:09.482 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8659000000000005e-05 2024-09-11T03:43:09.482 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.482 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.482 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.483 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.912333+0000", 2024-09-11T03:43:09.483 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000135325 2024-09-11T03:43:09.483 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.483 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.483 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.483 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.483 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.483 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1877 2.b 2:de6b1316:::smithi17557870-32:head [copy-from ver 223] snapc 4d=[4d,4b,49,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-11T03:43:09.484 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:03.907218+0000", 2024-09-11T03:43:09.484 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 125.497619758, 2024-09-11T03:43:09.484 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019706267999999999, 2024-09-11T03:43:09.484 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.484 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.484 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:09.484 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.484 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.484 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.484 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1877 2024-09-11T03:43:09.484 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.485 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.485 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.485 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.485 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907218+0000", 2024-09-11T03:43:09.485 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.485 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.485 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.485 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.485 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907218+0000", 2024-09-11T03:43:09.485 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.485 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.486 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.486 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.486 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907219+0000", 2024-09-11T03:43:09.486 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.221e-06 2024-09-11T03:43:09.486 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.486 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.486 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.486 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907226+0000", 2024-09-11T03:43:09.486 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1580000000000002e-06 2024-09-11T03:43:09.486 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.486 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.487 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:09.489 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.489 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907239+0000", 2024-09-11T03:43:09.489 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3865e-05 2024-09-11T03:43:09.489 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.489 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.489 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.490 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907247+0000", 2024-09-11T03:43:09.490 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8520000000000004e-06 2024-09-11T03:43:09.490 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.490 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.490 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.490 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907312+0000", 2024-09-11T03:43:09.490 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4437000000000007e-05 2024-09-11T03:43:09.490 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.490 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.490 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.490 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907417+0000", 2024-09-11T03:43:09.490 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000105698 2024-09-11T03:43:09.490 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.491 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.491 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,11", 2024-09-11T03:43:09.491 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.915919+0000", 2024-09-11T03:43:09.491 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.008501794 2024-09-11T03:43:09.491 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.491 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.491 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:09.491 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.920523+0000", 2024-09-11T03:43:09.491 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0046041220000000004 2024-09-11T03:43:09.491 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.491 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.491 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.491 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.920865+0000", 2024-09-11T03:43:09.492 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00034196500000000001 2024-09-11T03:43:09.492 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.492 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.492 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.492 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.926838+0000", 2024-09-11T03:43:09.492 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0059723420000000003 2024-09-11T03:43:09.492 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.492 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.492 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:09.492 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.926882+0000", 2024-09-11T03:43:09.492 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4199999999999997e-05 2024-09-11T03:43:09.492 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.492 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.492 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.492 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.926924+0000", 2024-09-11T03:43:09.493 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2614000000000002e-05 2024-09-11T03:43:09.493 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.493 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.493 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.493 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.493 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.493 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1878 2.b 2:de6b1316:::smithi17557870-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e247)", 2024-09-11T03:43:09.493 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:03.907266+0000", 2024-09-11T03:43:09.493 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 125.497572222, 2024-09-11T03:43:09.494 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012700637000000001, 2024-09-11T03:43:09.494 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.494 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.494 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:09.494 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.494 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.494 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.494 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1878 2024-09-11T03:43:09.494 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907266+0000", 2024-09-11T03:43:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907266+0000", 2024-09-11T03:43:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.496 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.496 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907266+0000", 2024-09-11T03:43:09.496 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3399999999999999e-07 2024-09-11T03:43:09.496 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.496 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.496 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.496 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907269+0000", 2024-09-11T03:43:09.496 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4739999999999999e-06 2024-09-11T03:43:09.496 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.496 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.496 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.496 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907276+0000", 2024-09-11T03:43:09.496 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2830000000000004e-06 2024-09-11T03:43:09.497 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.497 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.497 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.497 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907281+0000", 2024-09-11T03:43:09.497 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4430000000000002e-06 2024-09-11T03:43:09.497 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.497 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.497 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.497 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907582+0000", 2024-09-11T03:43:09.497 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00030033099999999998 2024-09-11T03:43:09.497 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.497 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.498 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:09.498 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907629+0000", 2024-09-11T03:43:09.498 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7308999999999997e-05 2024-09-11T03:43:09.498 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.498 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.498 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.498 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.919825+0000", 2024-09-11T03:43:09.498 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012196039000000001 2024-09-11T03:43:09.498 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.498 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.498 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.498 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.919892+0000", 2024-09-11T03:43:09.498 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7198000000000003e-05 2024-09-11T03:43:09.499 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.499 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.499 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.499 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.919966+0000", 2024-09-11T03:43:09.499 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4025999999999999e-05 2024-09-11T03:43:09.499 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.499 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.499 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.499 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.499 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.499 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2115 2.5 2:a6e3bbd9:::smithi17557870-38:head [copy-from ver 221] snapc 57=[57,56,51,4e] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-11T03:43:09.499 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:24.863058+0000", 2024-09-11T03:43:09.499 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 104.541779447, 2024-09-11T03:43:09.500 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.025181835999999999, 2024-09-11T03:43:09.500 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.500 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.500 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:09.500 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.500 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.500 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.500 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2115 2024-09-11T03:43:09.500 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.500 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.500 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.500 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.501 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863058+0000", 2024-09-11T03:43:09.501 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.501 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.501 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.501 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.501 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863058+0000", 2024-09-11T03:43:09.501 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.501 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.501 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.501 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.501 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863061+0000", 2024-09-11T03:43:09.501 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2649999999999999e-06 2024-09-11T03:43:09.502 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.502 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.502 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.502 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863072+0000", 2024-09-11T03:43:09.502 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1409999999999999e-05 2024-09-11T03:43:09.502 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.502 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.502 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.502 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863091+0000", 2024-09-11T03:43:09.502 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8825e-05 2024-09-11T03:43:09.503 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.503 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.503 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.503 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863099+0000", 2024-09-11T03:43:09.503 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7849999999999992e-06 2024-09-11T03:43:09.503 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.503 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.503 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.503 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863164+0000", 2024-09-11T03:43:09.503 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5078000000000003e-05 2024-09-11T03:43:09.503 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.504 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.504 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:43:09.504 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863169+0000", 2024-09-11T03:43:09.504 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1649999999999997e-06 2024-09-11T03:43:09.504 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.504 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.504 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.504 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.864677+0000", 2024-09-11T03:43:09.504 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001507928 2024-09-11T03:43:09.504 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.504 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.505 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.505 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.864744+0000", 2024-09-11T03:43:09.505 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6990999999999996e-05 2024-09-11T03:43:09.505 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.505 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.505 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:43:09.505 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.876551+0000", 2024-09-11T03:43:09.505 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011806891 2024-09-11T03:43:09.505 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.505 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.505 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:09.506 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.882152+0000", 2024-09-11T03:43:09.506 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0056009240000000002 2024-09-11T03:43:09.506 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.506 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.506 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.506 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.884525+0000", 2024-09-11T03:43:09.506 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002373436 2024-09-11T03:43:09.506 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.506 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.506 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.506 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.888133+0000", 2024-09-11T03:43:09.506 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003608207 2024-09-11T03:43:09.506 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.507 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.507 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:09.507 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.888194+0000", 2024-09-11T03:43:09.507 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1007000000000001e-05 2024-09-11T03:43:09.507 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.507 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.507 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.507 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.888240+0000", 2024-09-11T03:43:09.507 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5924000000000003e-05 2024-09-11T03:43:09.507 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.507 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.507 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.507 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.507 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.508 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2116 2.5 2:a6e3bbd9:::smithi17557870-38:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e268)", 2024-09-11T03:43:09.508 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:24.863121+0000", 2024-09-11T03:43:09.508 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 104.541716928, 2024-09-11T03:43:09.508 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018358349999999999, 2024-09-11T03:43:09.508 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.508 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.508 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:09.508 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.508 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.508 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.508 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2116 2024-09-11T03:43:09.508 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.509 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.509 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.509 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.509 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863121+0000", 2024-09-11T03:43:09.509 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.509 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.509 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.509 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.509 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863121+0000", 2024-09-11T03:43:09.509 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.509 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.509 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.509 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.510 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863121+0000", 2024-09-11T03:43:09.510 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5900000000000002e-07 2024-09-11T03:43:09.510 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.510 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.510 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.510 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863126+0000", 2024-09-11T03:43:09.510 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8910000000000003e-06 2024-09-11T03:43:09.510 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.510 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.510 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.510 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863133+0000", 2024-09-11T03:43:09.510 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4489999999999996e-06 2024-09-11T03:43:09.510 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.511 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.511 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.511 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863137+0000", 2024-09-11T03:43:09.511 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4549999999999997e-06 2024-09-11T03:43:09.511 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.511 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.511 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.511 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863232+0000", 2024-09-11T03:43:09.511 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4308999999999994e-05 2024-09-11T03:43:09.511 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.511 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.511 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:43:09.512 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863238+0000", 2024-09-11T03:43:09.512 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6279999999999999e-06 2024-09-11T03:43:09.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.512 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.512 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.512 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.864857+0000", 2024-09-11T03:43:09.512 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0016184330000000001 2024-09-11T03:43:09.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.512 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.512 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:09.512 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.864900+0000", 2024-09-11T03:43:09.513 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3745000000000002e-05 2024-09-11T03:43:09.513 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.513 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.513 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.513 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.881241+0000", 2024-09-11T03:43:09.513 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016340232999999999 2024-09-11T03:43:09.513 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.513 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.513 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.513 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.881352+0000", 2024-09-11T03:43:09.514 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000110945 2024-09-11T03:43:09.514 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.514 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.514 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.514 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.881479+0000", 2024-09-11T03:43:09.514 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012780300000000001 2024-09-11T03:43:09.514 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.514 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.514 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.514 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.514 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.514 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2165 2.b 2:dde92fc6:::smithi17557870-41:head [copy-from ver 101] snapc 58=[58,57,56,51] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-09-11T03:43:09.515 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:26.857685+0000", 2024-09-11T03:43:09.515 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 102.547152536, 2024-09-11T03:43:09.515 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022414478000000002, 2024-09-11T03:43:09.515 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.515 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.515 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:09.515 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.515 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.515 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.515 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2165 2024-09-11T03:43:09.515 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.515 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.515 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.516 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.516 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857685+0000", 2024-09-11T03:43:09.516 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.516 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.516 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.516 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.516 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857685+0000", 2024-09-11T03:43:09.516 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.516 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.516 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.516 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.516 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857687+0000", 2024-09-11T03:43:09.517 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.716e-06 2024-09-11T03:43:09.517 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.517 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.517 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.517 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857694+0000", 2024-09-11T03:43:09.517 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9909999999999996e-06 2024-09-11T03:43:09.517 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.517 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.517 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.517 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857731+0000", 2024-09-11T03:43:09.517 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7083999999999999e-05 2024-09-11T03:43:09.517 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.517 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.518 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.518 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857741+0000", 2024-09-11T03:43:09.518 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0054999999999999e-05 2024-09-11T03:43:09.518 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.518 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.518 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.518 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857802+0000", 2024-09-11T03:43:09.518 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0402000000000002e-05 2024-09-11T03:43:09.518 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.518 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.518 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.518 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857878+0000", 2024-09-11T03:43:09.518 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5926000000000005e-05 2024-09-11T03:43:09.518 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.519 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.519 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,11", 2024-09-11T03:43:09.519 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.871599+0000", 2024-09-11T03:43:09.519 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013721492 2024-09-11T03:43:09.519 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.519 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.519 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.519 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.876926+0000", 2024-09-11T03:43:09.519 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0053270949999999996 2024-09-11T03:43:09.519 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.519 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.519 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:09.519 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.877015+0000", 2024-09-11T03:43:09.520 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.9315999999999997e-05 2024-09-11T03:43:09.520 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.520 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.520 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.520 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.880031+0000", 2024-09-11T03:43:09.520 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030155830000000001 2024-09-11T03:43:09.520 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.520 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.520 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:09.520 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.880069+0000", 2024-09-11T03:43:09.520 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8349999999999997e-05 2024-09-11T03:43:09.521 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.521 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.521 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.521 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.880100+0000", 2024-09-11T03:43:09.521 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0468e-05 2024-09-11T03:43:09.521 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.521 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.521 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.521 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.521 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.521 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2166 2.b 2:dde92fc6:::smithi17557870-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e270)", 2024-09-11T03:43:09.521 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:26.857761+0000", 2024-09-11T03:43:09.521 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 102.547076541, 2024-09-11T03:43:09.522 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018486750999999999, 2024-09-11T03:43:09.522 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.522 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.522 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:09.522 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.522 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.522 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.522 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2166 2024-09-11T03:43:09.522 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.522 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.522 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.522 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.522 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857761+0000", 2024-09-11T03:43:09.523 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.523 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.523 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.523 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.523 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857761+0000", 2024-09-11T03:43:09.523 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.523 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.523 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.523 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.523 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857762+0000", 2024-09-11T03:43:09.523 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.51e-07 2024-09-11T03:43:09.523 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.523 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.524 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.524 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857765+0000", 2024-09-11T03:43:09.524 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0029999999999999e-06 2024-09-11T03:43:09.524 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.524 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.524 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.524 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857772+0000", 2024-09-11T03:43:09.524 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1060000000000002e-06 2024-09-11T03:43:09.524 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.524 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.524 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.524 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857777+0000", 2024-09-11T03:43:09.524 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6290000000000001e-06 2024-09-11T03:43:09.525 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.525 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.525 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.525 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857989+0000", 2024-09-11T03:43:09.525 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00021282300000000001 2024-09-11T03:43:09.525 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.525 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.525 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:09.525 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.858027+0000", 2024-09-11T03:43:09.525 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7326999999999997e-05 2024-09-11T03:43:09.525 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.525 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.525 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.526 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.876066+0000", 2024-09-11T03:43:09.526 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018039276999999999 2024-09-11T03:43:09.526 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.526 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.526 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.526 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.876155+0000", 2024-09-11T03:43:09.526 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.9062999999999998e-05 2024-09-11T03:43:09.526 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.526 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.526 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.526 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.876248+0000", 2024-09-11T03:43:09.526 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2972000000000004e-05 2024-09-11T03:43:09.526 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2492 2.5 2:a1a99782:::smithi17557870-1:head [copy-from ver 279] snapc 6b=[6b,6a,69,68,67,65,64,63,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-11T03:43:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:58.892917+0000", 2024-09-11T03:43:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 70.511921028000003, 2024-09-11T03:43:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018454061000000001, 2024-09-11T03:43:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2492 2024-09-11T03:43:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892917+0000", 2024-09-11T03:43:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892917+0000", 2024-09-11T03:43:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892918+0000", 2024-09-11T03:43:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0899999999999999e-06 2024-09-11T03:43:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892925+0000", 2024-09-11T03:43:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5470000000000002e-06 2024-09-11T03:43:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892937+0000", 2024-09-11T03:43:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.269e-05 2024-09-11T03:43:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892945+0000", 2024-09-11T03:43:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7270000000000007e-06 2024-09-11T03:43:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.893002+0000", 2024-09-11T03:43:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6650999999999997e-05 2024-09-11T03:43:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.893112+0000", 2024-09-11T03:43:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011042100000000001 2024-09-11T03:43:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:43:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.901793+0000", 2024-09-11T03:43:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0086810189999999999 2024-09-11T03:43:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.905968+0000", 2024-09-11T03:43:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0041751239999999997 2024-09-11T03:43:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.910080+0000", 2024-09-11T03:43:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0041120799999999997 2024-09-11T03:43:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.911159+0000", 2024-09-11T03:43:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0010783609999999999 2024-09-11T03:43:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.911257+0000", 2024-09-11T03:43:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8167999999999999e-05 2024-09-11T03:43:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.534 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.911371+0000", 2024-09-11T03:43:09.534 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114183 2024-09-11T03:43:09.534 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.534 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.534 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.534 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.534 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.534 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2931 2.5 2:a6d17fc0:::smithi17557870-13:head [copy-from ver 419] snapc 80=[80,7b,7a,79,77,75,6c,65] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-11T03:43:09.534 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:43.119642+0000", 2024-09-11T03:43:09.534 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 26.285196051, 2024-09-11T03:43:09.534 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013261583, 2024-09-11T03:43:09.534 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.534 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.535 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:09.535 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.535 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.535 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.535 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2931 2024-09-11T03:43:09.535 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.535 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.535 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.535 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.535 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.119642+0000", 2024-09-11T03:43:09.535 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.535 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.535 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.535 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.536 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.119642+0000", 2024-09-11T03:43:09.536 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.536 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.536 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.536 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.536 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.119644+0000", 2024-09-11T03:43:09.536 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0889999999999998e-06 2024-09-11T03:43:09.536 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.536 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.536 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.536 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.119652+0000", 2024-09-11T03:43:09.536 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1159999999999999e-06 2024-09-11T03:43:09.537 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.537 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.537 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.537 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.119670+0000", 2024-09-11T03:43:09.537 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7903000000000001e-05 2024-09-11T03:43:09.537 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.537 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.537 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.537 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.119677+0000", 2024-09-11T03:43:09.537 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2520000000000002e-06 2024-09-11T03:43:09.537 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.537 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.537 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.537 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.119752+0000", 2024-09-11T03:43:09.538 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5092000000000005e-05 2024-09-11T03:43:09.538 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.538 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.538 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:43:09.538 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.119759+0000", 2024-09-11T03:43:09.538 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.165e-06 2024-09-11T03:43:09.538 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.538 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.538 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.538 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.121986+0000", 2024-09-11T03:43:09.538 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022266159999999998 2024-09-11T03:43:09.538 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.538 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.539 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.539 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.122074+0000", 2024-09-11T03:43:09.539 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8010999999999995e-05 2024-09-11T03:43:09.539 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.539 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.539 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:43:09.539 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.125529+0000", 2024-09-11T03:43:09.539 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0034549709999999998 2024-09-11T03:43:09.539 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.539 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.539 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:09.539 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.128468+0000", 2024-09-11T03:43:09.540 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0029391209999999998 2024-09-11T03:43:09.540 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.540 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.540 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.540 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.131454+0000", 2024-09-11T03:43:09.540 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002986135 2024-09-11T03:43:09.540 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.540 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.540 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.540 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.132843+0000", 2024-09-11T03:43:09.540 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001388762 2024-09-11T03:43:09.540 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.540 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.541 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:09.541 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.132876+0000", 2024-09-11T03:43:09.541 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.2740999999999997e-05 2024-09-11T03:43:09.541 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.541 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.541 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.541 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.132903+0000", 2024-09-11T03:43:09.541 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7608999999999999e-05 2024-09-11T03:43:09.541 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.541 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.541 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.541 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.541 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.542 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:3067 2.f 2:fda69313:::smithi17557870-16:head [copy-from ver 442] snapc 82=[81,75] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-11T03:43:09.542 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:53.495971+0000", 2024-09-11T03:43:09.542 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 15.908866979000001, 2024-09-11T03:43:09.542 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012726619, 2024-09-11T03:43:09.542 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:09.542 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:09.542 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:09.542 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:09.542 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:09.542 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:09.542 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 3067 2024-09-11T03:43:09.542 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.542 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:09.543 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.543 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:09.543 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.495971+0000", 2024-09-11T03:43:09.543 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.543 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.543 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.543 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:09.543 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.495971+0000", 2024-09-11T03:43:09.543 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:09.543 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.543 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.544 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:09.544 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.495972+0000", 2024-09-11T03:43:09.544 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.471e-06 2024-09-11T03:43:09.544 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.544 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.544 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:09.544 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.495978+0000", 2024-09-11T03:43:09.544 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1490000000000002e-06 2024-09-11T03:43:09.545 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.545 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.545 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:09.545 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.495991+0000", 2024-09-11T03:43:09.545 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3196e-05 2024-09-11T03:43:09.545 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.545 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.545 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:09.545 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.495998+0000", 2024-09-11T03:43:09.545 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9700000000000002e-06 2024-09-11T03:43:09.545 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.546 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.546 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:09.546 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.496167+0000", 2024-09-11T03:43:09.546 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000168787 2024-09-11T03:43:09.546 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.546 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.546 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:09.546 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.496256+0000", 2024-09-11T03:43:09.546 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.9257000000000003e-05 2024-09-11T03:43:09.546 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.546 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.546 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,9", 2024-09-11T03:43:09.546 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.502295+0000", 2024-09-11T03:43:09.547 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0060393139999999996 2024-09-11T03:43:09.547 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.547 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.547 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:09.547 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.506401+0000", 2024-09-11T03:43:09.547 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0041057899999999998 2024-09-11T03:43:09.547 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.547 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.547 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.547 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.507258+0000", 2024-09-11T03:43:09.547 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00085705999999999998 2024-09-11T03:43:09.547 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.547 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.547 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:09.547 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.508621+0000", 2024-09-11T03:43:09.548 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013631610000000001 2024-09-11T03:43:09.548 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.548 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.548 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:09.548 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.508660+0000", 2024-09-11T03:43:09.548 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8418999999999999e-05 2024-09-11T03:43:09.548 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:09.548 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:09.548 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:09.548 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.508698+0000", 2024-09-11T03:43:09.548 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8044999999999999e-05 2024-09-11T03:43:09.548 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.548 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.549 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.549 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:09.549 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:09.549 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:43:09.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:09.571 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:09.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:09.575 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:43:09.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:09.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:09.694 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:09.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:09.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:09.817 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:09.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:09.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:09.904 INFO:tasks.rados.rados.2.smithi175.stdout:1329: done (1 left) 2024-09-11T03:43:09.905 INFO:tasks.rados.rados.2.smithi175.stdout:1330: delete oid 40 current snap is 137 2024-09-11T03:43:09.905 INFO:tasks.rados.rados.2.smithi175.stdout:1327: expect (ObjNum 425 snap 129 seq_num 425) 2024-09-11T03:43:09.919 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:09.941 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:09.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:09.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:09.943 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:10.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:10.065 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:10.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:10.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:10.082 INFO:tasks.rados.rados.2.smithi175.stdout:1327: done (1 left) 2024-09-11T03:43:10.082 INFO:tasks.rados.rados.2.smithi175.stdout:1330: done (0 left) 2024-09-11T03:43:10.082 INFO:tasks.rados.rados.2.smithi175.stdout:1331: write (excl) oid 48 current snap is 137 2024-09-11T03:43:10.083 INFO:tasks.rados.rados.2.smithi175.stdout:1331: seq_num 472 ranges {732904=731414,2120442=538824,3278829=504743} 2024-09-11T03:43:10.092 INFO:tasks.rados.rados.2.smithi175.stdout:1331: writing smithi17557870-48 from 732904 to 1464318 tid 1 2024-09-11T03:43:10.100 INFO:tasks.rados.rados.2.smithi175.stdout:1331: writing smithi17557870-48 from 2120442 to 2659266 tid 2 2024-09-11T03:43:10.107 INFO:tasks.rados.rados.2.smithi175.stdout:1331: writing smithi17557870-48 from 3278829 to 3783572 tid 3 2024-09-11T03:43:10.107 INFO:tasks.rados.rados.2.smithi175.stdout:1332: copy_from oid 24 from oid 33 current snap is 137 2024-09-11T03:43:10.108 INFO:tasks.rados.rados.2.smithi175.stdout:1331: finishing write tid 1 to smithi17557870-48 2024-09-11T03:43:10.108 INFO:tasks.rados.rados.2.smithi175.stdout:1331: finishing write tid 2 to smithi17557870-48 2024-09-11T03:43:10.108 INFO:tasks.rados.rados.2.smithi175.stdout:1333: write oid 43 current snap is 137 2024-09-11T03:43:10.108 INFO:tasks.rados.rados.2.smithi175.stdout:1333: seq_num 473 ranges {749711=406663,1778924=682482,3189899=716015} 2024-09-11T03:43:10.114 INFO:tasks.rados.rados.2.smithi175.stdout:1333: writing smithi17557870-43 from 749711 to 1156374 tid 1 2024-09-11T03:43:10.115 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-11T03:43:10.121 INFO:tasks.rados.rados.2.smithi175.stdout:1333: writing smithi17557870-43 from 1778924 to 2461406 tid 2 2024-09-11T03:43:10.130 INFO:tasks.rados.rados.2.smithi175.stdout:1333: writing smithi17557870-43 from 3189899 to 3905914 tid 3 2024-09-11T03:43:10.130 INFO:tasks.rados.rados.2.smithi175.stdout:1334: rollback oid 19 current snap is 137 2024-09-11T03:43:10.130 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 19 to 136 2024-09-11T03:43:10.130 INFO:tasks.rados.rados.2.smithi175.stdout:1332: finishing copy_from to smithi17557870-24 2024-09-11T03:43:10.130 INFO:tasks.rados.rados.2.smithi175.stdout:1332: got expected ENOENT (src dne) 2024-09-11T03:43:10.130 INFO:tasks.rados.rados.2.smithi175.stdout:1332: finishing copy_from racing read to smithi17557870-24 2024-09-11T03:43:10.130 INFO:tasks.rados.rados.2.smithi175.stdout:1332: done (3 left) 2024-09-11T03:43:10.130 INFO:tasks.rados.rados.2.smithi175.stdout:1335: read oid 27 snap 136 2024-09-11T03:43:10.130 INFO:tasks.rados.rados.2.smithi175.stdout:1335: expect (ObjNum 423 snap 129 seq_num 423) 2024-09-11T03:43:10.130 INFO:tasks.rados.rados.2.smithi175.stdout:1336: delete oid 11 current snap is 137 2024-09-11T03:43:10.130 INFO:tasks.rados.rados.2.smithi175.stdout:1331: finishing write tid 3 to smithi17557870-48 2024-09-11T03:43:10.130 INFO:tasks.rados.rados.2.smithi175.stdout:1331: finishing write tid 4 to smithi17557870-48 2024-09-11T03:43:10.130 INFO:tasks.rados.rados.2.smithi175.stdout:1331: finishing write tid 5 to smithi17557870-48 2024-09-11T03:43:10.131 INFO:tasks.rados.rados.2.smithi175.stdout:1331: oid 48 updating version 0 to 453 2024-09-11T03:43:10.131 INFO:tasks.rados.rados.2.smithi175.stdout:1331: oid 48 version 453 is already newer than 452 2024-09-11T03:43:10.131 INFO:tasks.rados.rados.2.smithi175.stdout:1331: oid 48 updating version 453 to 455 2024-09-11T03:43:10.131 INFO:tasks.rados.rados.2.smithi175.stdout:1331: oid 48 version 455 is already newer than 454 2024-09-11T03:43:10.131 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 455 (ObjNum 472 snap 137 seq_num 472) dirty exists 2024-09-11T03:43:10.131 INFO:tasks.rados.rados.2.smithi175.stdout:1331: left oid 48 (ObjNum 472 snap 137 seq_num 472) 2024-09-11T03:43:10.131 INFO:tasks.rados.rados.2.smithi175.stdout:1333: finishing write tid 1 to smithi17557870-43 2024-09-11T03:43:10.131 INFO:tasks.rados.rados.2.smithi175.stdout:1333: finishing write tid 2 to smithi17557870-43 2024-09-11T03:43:10.132 INFO:tasks.rados.rados.2.smithi175.stdout:1331: done (4 left) 2024-09-11T03:43:10.132 INFO:tasks.rados.rados.2.smithi175.stdout:1336: done (3 left) 2024-09-11T03:43:10.132 INFO:tasks.rados.rados.2.smithi175.stdout:1337: snap_create 2024-09-11T03:43:10.132 INFO:tasks.rados.rados.2.smithi175.stdout:1334: finishing rollback tid 1 to smithi17557870-19 2024-09-11T03:43:10.132 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T03:43:10.133 INFO:tasks.rados.rados.2.smithi175.stdout:1333: finishing write tid 3 to smithi17557870-43 2024-09-11T03:43:10.133 INFO:tasks.rados.rados.2.smithi175.stdout:1333: finishing write tid 4 to smithi17557870-43 2024-09-11T03:43:10.133 INFO:tasks.rados.rados.2.smithi175.stdout:1333: finishing write tid 5 to smithi17557870-43 2024-09-11T03:43:10.133 INFO:tasks.rados.rados.2.smithi175.stdout:1333: oid 43 updating version 0 to 387 2024-09-11T03:43:10.134 INFO:tasks.rados.rados.2.smithi175.stdout:1333: oid 43 version 387 is already newer than 386 2024-09-11T03:43:10.134 INFO:tasks.rados.rados.2.smithi175.stdout:1333: oid 43 version 387 is already newer than 385 2024-09-11T03:43:10.134 INFO:tasks.rados.rados.2.smithi175.stdout:1333: oid 43 version 387 is already newer than 384 2024-09-11T03:43:10.134 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 387 (ObjNum 473 snap 137 seq_num 473) dirty exists 2024-09-11T03:43:10.134 INFO:tasks.rados.rados.2.smithi175.stdout:1333: left oid 43 (ObjNum 473 snap 137 seq_num 473) 2024-09-11T03:43:10.140 INFO:tasks.rados.rados.2.smithi175.stdout:1335: expect (ObjNum 423 snap 129 seq_num 423) 2024-09-11T03:43:10.167 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:10.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:10.188 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:10.189 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:10.209 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:10 smithi084 ceph-mon[27254]: pgmap v711: 57 pgs: 57 active+clean; 128 MiB data, 653 MiB used, 893 GiB / 894 GiB avail; 3.6 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-09-11T03:43:10.209 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:10 smithi084 ceph-mon[27254]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:43:10.209 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:10 smithi084 ceph-mon[27254]: osdmap e372: 12 total, 12 up, 10 in 2024-09-11T03:43:10.209 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:10 smithi084 ceph-mon[27254]: 1.1e scrub starts 2024-09-11T03:43:10.210 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:10 smithi084 ceph-mon[27254]: 1.1e scrub ok 2024-09-11T03:43:10.289 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:10.312 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:10.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:10.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:10.361 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:43:10.415 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:10.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:10 smithi121 ceph-mon[30729]: pgmap v711: 57 pgs: 57 active+clean; 128 MiB data, 653 MiB used, 893 GiB / 894 GiB avail; 3.6 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-09-11T03:43:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:10 smithi121 ceph-mon[30729]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:43:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:10 smithi121 ceph-mon[30729]: osdmap e372: 12 total, 12 up, 10 in 2024-09-11T03:43:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:10 smithi121 ceph-mon[30729]: 1.1e scrub starts 2024-09-11T03:43:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:10 smithi121 ceph-mon[30729]: 1.1e scrub ok 2024-09-11T03:43:10.436 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:10.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:10.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:10.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:10 smithi175 ceph-mon[30894]: pgmap v711: 57 pgs: 57 active+clean; 128 MiB data, 653 MiB used, 893 GiB / 894 GiB avail; 3.6 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2024-09-11T03:43:10.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:10 smithi175 ceph-mon[30894]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T03:43:10.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:10 smithi175 ceph-mon[30894]: osdmap e372: 12 total, 12 up, 10 in 2024-09-11T03:43:10.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:10 smithi175 ceph-mon[30894]: 1.1e scrub starts 2024-09-11T03:43:10.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:10 smithi175 ceph-mon[30894]: 1.1e scrub ok 2024-09-11T03:43:10.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:10.561 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:10.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:10.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:10.663 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:10.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:10.717 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:10.718 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:10.819 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:10.840 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:10.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:10.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:10.868 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:43:10.868 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:43:10.868 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:43:10.868 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:43:10.942 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:10.964 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:10.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:11.065 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:11.087 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:11.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:11.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:11.112 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:43:11.115 INFO:tasks.rados.rados.2.smithi175.stdout:1333: done (3 left) 2024-09-11T03:43:11.115 INFO:tasks.rados.rados.2.smithi175.stdout:1334: done (2 left) 2024-09-11T03:43:11.116 INFO:tasks.rados.rados.2.smithi175.stdout:1335: done (1 left) 2024-09-11T03:43:11.116 INFO:tasks.rados.rados.2.smithi175.stdout:1337: done (0 left) 2024-09-11T03:43:11.116 INFO:tasks.rados.rados.2.smithi175.stdout:1338: write (excl) oid 31 current snap is 138 2024-09-11T03:43:11.116 INFO:tasks.rados.rados.2.smithi175.stdout:1338: seq_num 475 ranges {783325=557161,1779535=686151,3244745=454912} 2024-09-11T03:43:11.122 INFO:tasks.rados.rados.2.smithi175.stdout:1338: writing smithi17557870-31 from 783325 to 1340486 tid 1 2024-09-11T03:43:11.128 INFO:tasks.rados.rados.2.smithi175.stdout:1338: writing smithi17557870-31 from 1779535 to 2465686 tid 2 2024-09-11T03:43:11.136 INFO:tasks.rados.rados.2.smithi175.stdout:1338: writing smithi17557870-31 from 3244745 to 3699657 tid 3 2024-09-11T03:43:11.136 INFO:tasks.rados.rados.2.smithi175.stdout:1339: copy_from oid 40 from oid 11 current snap is 138 2024-09-11T03:43:11.136 INFO:tasks.rados.rados.2.smithi175.stdout:1340: rollback oid 50 current snap is 138 2024-09-11T03:43:11.136 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 50 to 135 2024-09-11T03:43:11.136 INFO:tasks.rados.rados.2.smithi175.stdout:1338: finishing write tid 1 to smithi17557870-31 2024-09-11T03:43:11.136 INFO:tasks.rados.rados.2.smithi175.stdout:1338: finishing write tid 2 to smithi17557870-31 2024-09-11T03:43:11.136 INFO:tasks.rados.rados.2.smithi175.stdout:1341: write oid 38 current snap is 138 2024-09-11T03:43:11.136 INFO:tasks.rados.rados.2.smithi175.stdout:1341: seq_num 476 ranges {400132=632410,1438017=546162} 2024-09-11T03:43:11.142 INFO:tasks.rados.rados.2.smithi175.stdout:1341: writing smithi17557870-38 from 400132 to 1032542 tid 1 2024-09-11T03:43:11.147 INFO:tasks.rados.rados.2.smithi175.stdout:1341: writing smithi17557870-38 from 1438017 to 1984179 tid 2 2024-09-11T03:43:11.147 INFO:tasks.rados.rados.2.smithi175.stdout:1342: snap_create 2024-09-11T03:43:11.147 INFO:tasks.rados.rados.2.smithi175.stdout:1339: finishing copy_from to smithi17557870-40 2024-09-11T03:43:11.147 INFO:tasks.rados.rados.2.smithi175.stdout:1339: got expected ENOENT (src dne) 2024-09-11T03:43:11.147 INFO:tasks.rados.rados.2.smithi175.stdout:1340: finishing rollback tid 0 to smithi17557870-50 2024-09-11T03:43:11.148 INFO:tasks.rados.rados.2.smithi175.stdout:1339: finishing copy_from racing read to smithi17557870-40 2024-09-11T03:43:11.148 INFO:tasks.rados.rados.2.smithi175.stdout:1338: finishing write tid 3 to smithi17557870-31 2024-09-11T03:43:11.148 INFO:tasks.rados.rados.2.smithi175.stdout:1338: finishing write tid 4 to smithi17557870-31 2024-09-11T03:43:11.148 INFO:tasks.rados.rados.2.smithi175.stdout:1338: finishing write tid 5 to smithi17557870-31 2024-09-11T03:43:11.148 INFO:tasks.rados.rados.2.smithi175.stdout:1338: oid 31 updating version 0 to 476 2024-09-11T03:43:11.148 INFO:tasks.rados.rados.2.smithi175.stdout:1338: oid 31 updating version 476 to 477 2024-09-11T03:43:11.148 INFO:tasks.rados.rados.2.smithi175.stdout:1338: oid 31 version 477 is already newer than 474 2024-09-11T03:43:11.148 INFO:tasks.rados.rados.2.smithi175.stdout:1338: oid 31 version 477 is already newer than 475 2024-09-11T03:43:11.148 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 477 (ObjNum 475 snap 138 seq_num 475) dirty exists 2024-09-11T03:43:11.148 INFO:tasks.rados.rados.2.smithi175.stdout:1338: left oid 31 (ObjNum 475 snap 138 seq_num 475) 2024-09-11T03:43:11.148 INFO:tasks.rados.rados.2.smithi175.stdout:1340: finishing rollback tid 1 to smithi17557870-50 2024-09-11T03:43:11.148 INFO:tasks.rados.rados.2.smithi175.stdout:1340: finishing rollback tid 2 to smithi17557870-50 2024-09-11T03:43:11.148 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 506 (ObjNum 443 snap 130 seq_num 443) dirty exists 2024-09-11T03:43:11.149 INFO:tasks.rados.rados.2.smithi175.stdout:1341: finishing write tid 1 to smithi17557870-38 2024-09-11T03:43:11.149 INFO:tasks.rados.rados.2.smithi175.stdout:1341: finishing write tid 2 to smithi17557870-38 2024-09-11T03:43:11.149 INFO:tasks.rados.rados.2.smithi175.stdout:1341: finishing write tid 3 to smithi17557870-38 2024-09-11T03:43:11.150 INFO:tasks.rados.rados.2.smithi175.stdout:1341: finishing write tid 4 to smithi17557870-38 2024-09-11T03:43:11.150 INFO:tasks.rados.rados.2.smithi175.stdout:1341: oid 38 updating version 0 to 472 2024-09-11T03:43:11.150 INFO:tasks.rados.rados.2.smithi175.stdout:1341: oid 38 updating version 472 to 474 2024-09-11T03:43:11.150 INFO:tasks.rados.rados.2.smithi175.stdout:1341: oid 38 version 474 is already newer than 473 2024-09-11T03:43:11.150 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 474 (ObjNum 476 snap 138 seq_num 476) dirty exists 2024-09-11T03:43:11.150 INFO:tasks.rados.rados.2.smithi175.stdout:1341: left oid 38 (ObjNum 476 snap 138 seq_num 476) 2024-09-11T03:43:11.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:11.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:11.214 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:11.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:11.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:11.340 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:11.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:11.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:11.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:11 smithi084 ceph-mon[27254]: 1.15 scrub starts 2024-09-11T03:43:11.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:11 smithi084 ceph-mon[27254]: 1.15 scrub ok 2024-09-11T03:43:11.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:11 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1652467626' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:43:11.415 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:11 smithi175 ceph-mon[30894]: 1.15 scrub starts 2024-09-11T03:43:11.415 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:11 smithi175 ceph-mon[30894]: 1.15 scrub ok 2024-09-11T03:43:11.415 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:11 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1652467626' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:43:11.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:11 smithi121 ceph-mon[30729]: 1.15 scrub starts 2024-09-11T03:43:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:11 smithi121 ceph-mon[30729]: 1.15 scrub ok 2024-09-11T03:43:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:11 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1652467626' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:43:11.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:11.443 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-11T03:43:11.467 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:11.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:11.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:11.569 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:11.591 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:11.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:11.693 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:11.716 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:11.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:11.723 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:43:11.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:11.844 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:11.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:11.898 INFO:tasks.rados.rados.2.smithi175.stdout:1338: done (4 left) 2024-09-11T03:43:11.898 INFO:tasks.rados.rados.2.smithi175.stdout:1339: done (3 left) 2024-09-11T03:43:11.898 INFO:tasks.rados.rados.2.smithi175.stdout:1340: done (2 left) 2024-09-11T03:43:11.898 INFO:tasks.rados.rados.2.smithi175.stdout:1341: done (1 left) 2024-09-11T03:43:11.898 INFO:tasks.rados.rados.2.smithi175.stdout:1342: done (0 left) 2024-09-11T03:43:11.898 INFO:tasks.rados.rados.2.smithi175.stdout:1343: write (excl) oid 38 current snap is 139 2024-09-11T03:43:11.898 INFO:tasks.rados.rados.2.smithi175.stdout:1343: seq_num 477 ranges {416939=707659,1896499=120440} 2024-09-11T03:43:11.903 INFO:tasks.rados.rados.2.smithi175.stdout:1343: writing smithi17557870-38 from 416939 to 1124598 tid 1 2024-09-11T03:43:11.908 INFO:tasks.rados.rados.2.smithi175.stdout:1343: writing smithi17557870-38 from 1896499 to 2016939 tid 2 2024-09-11T03:43:11.908 INFO:tasks.rados.rados.2.smithi175.stdout:1344: write oid 24 current snap is 139 2024-09-11T03:43:11.908 INFO:tasks.rados.rados.2.smithi175.stdout:1344: seq_num 478 ranges {433746=782908,1954981=78765} 2024-09-11T03:43:11.915 INFO:tasks.rados.rados.2.smithi175.stdout:1344: writing smithi17557870-24 from 433746 to 1216654 tid 1 2024-09-11T03:43:11.920 INFO:tasks.rados.rados.2.smithi175.stdout:1344: writing smithi17557870-24 from 1954981 to 2033746 tid 2 2024-09-11T03:43:11.920 INFO:tasks.rados.rados.2.smithi175.stdout:1345: read oid 14 snap 132 2024-09-11T03:43:11.920 INFO:tasks.rados.rados.2.smithi175.stdout:1345: expect (ObjNum 380 snap 117 seq_num 380) 2024-09-11T03:43:11.920 INFO:tasks.rados.rados.2.smithi175.stdout:1346: rollback oid 8 current snap is 139 2024-09-11T03:43:11.920 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 8 to 132 2024-09-11T03:43:11.920 INFO:tasks.rados.rados.2.smithi175.stdout:1343: finishing write tid 1 to smithi17557870-38 2024-09-11T03:43:11.921 INFO:tasks.rados.rados.2.smithi175.stdout:1347: rollback oid 15 current snap is 139 2024-09-11T03:43:11.921 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 15 to 138 2024-09-11T03:43:11.921 INFO:tasks.rados.rados.2.smithi175.stdout:1343: finishing write tid 2 to smithi17557870-38 2024-09-11T03:43:11.921 INFO:tasks.rados.rados.2.smithi175.stdout:1343: finishing write tid 3 to smithi17557870-38 2024-09-11T03:43:11.921 INFO:tasks.rados.rados.2.smithi175.stdout:1343: finishing write tid 4 to smithi17557870-38 2024-09-11T03:43:11.921 INFO:tasks.rados.rados.2.smithi175.stdout:1343: oid 38 updating version 0 to 476 2024-09-11T03:43:11.921 INFO:tasks.rados.rados.2.smithi175.stdout:1343: oid 38 updating version 476 to 477 2024-09-11T03:43:11.921 INFO:tasks.rados.rados.2.smithi175.stdout:1343: oid 38 updating version 477 to 478 2024-09-11T03:43:11.921 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 478 (ObjNum 477 snap 139 seq_num 477) dirty exists 2024-09-11T03:43:11.921 INFO:tasks.rados.rados.2.smithi175.stdout:1343: left oid 38 (ObjNum 477 snap 139 seq_num 477) 2024-09-11T03:43:11.921 INFO:tasks.rados.rados.2.smithi175.stdout:1343: done (4 left) 2024-09-11T03:43:11.921 INFO:tasks.rados.rados.2.smithi175.stdout:1348: copy_from oid 37 from oid 5 current snap is 139 2024-09-11T03:43:11.921 INFO:tasks.rados.rados.2.smithi175.stdout:1344: finishing write tid 1 to smithi17557870-24 2024-09-11T03:43:11.922 INFO:tasks.rados.rados.2.smithi175.stdout:1349: delete oid 23 current snap is 139 2024-09-11T03:43:11.922 INFO:tasks.rados.rados.2.smithi175.stdout:1344: finishing write tid 2 to smithi17557870-24 2024-09-11T03:43:11.922 INFO:tasks.rados.rados.2.smithi175.stdout:1348: finishing copy_from to smithi17557870-37 2024-09-11T03:43:11.922 INFO:tasks.rados.rados.2.smithi175.stdout:1348: got expected ENOENT (src dne) 2024-09-11T03:43:11.922 INFO:tasks.rados.rados.2.smithi175.stdout:1348: finishing copy_from racing read to smithi17557870-37 2024-09-11T03:43:11.923 INFO:tasks.rados.rados.2.smithi175.stdout:1348: done (5 left) 2024-09-11T03:43:11.923 INFO:tasks.rados.rados.2.smithi175.stdout:1349: done (4 left) 2024-09-11T03:43:11.923 INFO:tasks.rados.rados.2.smithi175.stdout:1350: snap_create 2024-09-11T03:43:11.923 INFO:tasks.rados.rados.2.smithi175.stdout:1347: finishing rollback tid 0 to smithi17557870-15 2024-09-11T03:43:11.923 INFO:tasks.rados.rados.2.smithi175.stdout:1347: finishing rollback tid 1 to smithi17557870-15 2024-09-11T03:43:11.924 INFO:tasks.rados.rados.2.smithi175.stdout:1347: finishing rollback tid 2 to smithi17557870-15 2024-09-11T03:43:11.924 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 202 (ObjNum 455 snap 130 seq_num 455) dirty exists 2024-09-11T03:43:11.925 INFO:tasks.rados.rados.2.smithi175.stdout:1344: finishing write tid 3 to smithi17557870-24 2024-09-11T03:43:11.926 INFO:tasks.rados.rados.2.smithi175.stdout:1344: finishing write tid 4 to smithi17557870-24 2024-09-11T03:43:11.926 INFO:tasks.rados.rados.2.smithi175.stdout:1344: oid 24 updating version 0 to 511 2024-09-11T03:43:11.926 INFO:tasks.rados.rados.2.smithi175.stdout:1344: oid 24 updating version 511 to 512 2024-09-11T03:43:11.926 INFO:tasks.rados.rados.2.smithi175.stdout:1344: oid 24 version 512 is already newer than 510 2024-09-11T03:43:11.926 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 512 (ObjNum 478 snap 139 seq_num 478) dirty exists 2024-09-11T03:43:11.926 INFO:tasks.rados.rados.2.smithi175.stdout:1344: left oid 24 (ObjNum 478 snap 139 seq_num 478) 2024-09-11T03:43:11.929 INFO:tasks.rados.rados.2.smithi175.stdout:1345: expect (ObjNum 380 snap 117 seq_num 380) 2024-09-11T03:43:11.935 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:43:11.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:11.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:11.971 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:11.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:12.044 INFO:tasks.rados.rados.2.smithi175.stdout:1346: finishing rollback tid 0 to smithi17557870-8 2024-09-11T03:43:12.044 INFO:tasks.rados.rados.2.smithi175.stdout:1346: finishing rollback tid 1 to smithi17557870-8 2024-09-11T03:43:12.044 INFO:tasks.rados.rados.2.smithi175.stdout:1346: finishing rollback tid 2 to smithi17557870-8 2024-09-11T03:43:12.044 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 516 (ObjNum 415 snap 127 seq_num 415) dirty exists 2024-09-11T03:43:12.073 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:12.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:12.096 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:12.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:12.177 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:12.194 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:43:12.194 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:43:12.194 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:43:12.194 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:43:12.194 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:43:12.197 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:12.220 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:12.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:12.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:12.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:12.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:12.349 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:12.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:12 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:43:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:12 smithi084 ceph-mon[27254]: 2.0 scrub ok 2024-09-11T03:43:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:12 smithi084 ceph-mon[27254]: 1.1a scrub starts 2024-09-11T03:43:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:12 smithi084 ceph-mon[27254]: 1.1a scrub ok 2024-09-11T03:43:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:12 smithi084 ceph-mon[27254]: pgmap v714: 57 pgs: 57 active+clean; 128 MiB data, 659 MiB used, 893 GiB / 894 GiB avail; 511 B/s rd, 854 KiB/s wr, 3 op/s 2024-09-11T03:43:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:12 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:12 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1652467626' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:43:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:12 smithi084 ceph-mon[27254]: osdmap e373: 12 total, 12 up, 10 in 2024-09-11T03:43:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:12 smithi084 ceph-mon[27254]: osdmap e374: 12 total, 12 up, 10 in 2024-09-11T03:43:12.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:12 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:43:12.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:12 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:43:12.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:12 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:43:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:12 smithi121 ceph-mon[30729]: 2.0 scrub ok 2024-09-11T03:43:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:12 smithi121 ceph-mon[30729]: 1.1a scrub starts 2024-09-11T03:43:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:12 smithi121 ceph-mon[30729]: 1.1a scrub ok 2024-09-11T03:43:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:12 smithi121 ceph-mon[30729]: pgmap v714: 57 pgs: 57 active+clean; 128 MiB data, 659 MiB used, 893 GiB / 894 GiB avail; 511 B/s rd, 854 KiB/s wr, 3 op/s 2024-09-11T03:43:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:12 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:12 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1652467626' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:43:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:12 smithi121 ceph-mon[30729]: osdmap e373: 12 total, 12 up, 10 in 2024-09-11T03:43:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:12 smithi121 ceph-mon[30729]: osdmap e374: 12 total, 12 up, 10 in 2024-09-11T03:43:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:12 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:43:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:12 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:43:12.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:12.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:12.475 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:12.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:12 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:43:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:12 smithi175 ceph-mon[30894]: 2.0 scrub ok 2024-09-11T03:43:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:12 smithi175 ceph-mon[30894]: 1.1a scrub starts 2024-09-11T03:43:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:12 smithi175 ceph-mon[30894]: 1.1a scrub ok 2024-09-11T03:43:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:12 smithi175 ceph-mon[30894]: pgmap v714: 57 pgs: 57 active+clean; 128 MiB data, 659 MiB used, 893 GiB / 894 GiB avail; 511 B/s rd, 854 KiB/s wr, 3 op/s 2024-09-11T03:43:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:12 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:12 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1652467626' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:43:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:12 smithi175 ceph-mon[30894]: osdmap e373: 12 total, 12 up, 10 in 2024-09-11T03:43:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:12 smithi175 ceph-mon[30894]: osdmap e374: 12 total, 12 up, 10 in 2024-09-11T03:43:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:12 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:43:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:12 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:43:12.576 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:12.598 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:12.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:12.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:12.690 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:43:12.690 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:43:12.701 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:12.706 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:43:12.706 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:43:12.722 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:43:12.722 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:43:12.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:12.723 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:12.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:12.738 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:43:12.739 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:43:12.755 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:43:12.756 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:43:12.773 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:43:12.773 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:43:12.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:43:12.790 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:43:12.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:43:12.807 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:43:12.824 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:12.825 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:43:12.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:43:12.834 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-11T03:43:12.841 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:43:12.841 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:43:12.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:12.846 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:12.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:12.858 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:43:12.858 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:43:12.874 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:43:12.875 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:43:12.899 INFO:tasks.rados.rados.2.smithi175.stdout:1344: done (4 left) 2024-09-11T03:43:12.900 INFO:tasks.rados.rados.2.smithi175.stdout:1345: done (3 left) 2024-09-11T03:43:12.900 INFO:tasks.rados.rados.2.smithi175.stdout:1346: done (2 left) 2024-09-11T03:43:12.900 INFO:tasks.rados.rados.2.smithi175.stdout:1347: done (1 left) 2024-09-11T03:43:12.900 INFO:tasks.rados.rados.2.smithi175.stdout:1350: done (0 left) 2024-09-11T03:43:12.900 INFO:tasks.rados.rados.2.smithi175.stdout:1351: read oid 36 snap -1 2024-09-11T03:43:12.900 INFO:tasks.rados.rados.2.smithi175.stdout:1351: expect (ObjNum 454 snap 130 seq_num 454) 2024-09-11T03:43:12.900 INFO:tasks.rados.rados.2.smithi175.stdout:1352: read oid 5 snap -1 2024-09-11T03:43:12.900 INFO:tasks.rados.rados.2.smithi175.stdout:1352: expect deleted 2024-09-11T03:43:12.900 INFO:tasks.rados.rados.2.smithi175.stdout:1353: delete oid 37 current snap is 140 2024-09-11T03:43:12.903 INFO:tasks.rados.rados.2.smithi175.stdout:1352: done (2 left) 2024-09-11T03:43:12.903 INFO:tasks.rados.rados.2.smithi175.stdout:1353: done (1 left) 2024-09-11T03:43:12.903 INFO:tasks.rados.rados.2.smithi175.stdout:1354: delete oid 33 current snap is 140 2024-09-11T03:43:12.905 INFO:tasks.rados.rados.2.smithi175.stdout:1354: done (1 left) 2024-09-11T03:43:12.905 INFO:tasks.rados.rados.2.smithi175.stdout:1355: write oid 24 current snap is 140 2024-09-11T03:43:12.905 INFO:tasks.rados.rados.2.smithi175.stdout:1355: seq_num 482 ranges {500974=400257,1388909=557169} 2024-09-11T03:43:12.910 INFO:tasks.rados.rados.2.smithi175.stdout:1355: writing smithi17557870-24 from 500974 to 901231 tid 1 2024-09-11T03:43:12.916 INFO:tasks.rados.rados.2.smithi175.stdout:1355: writing smithi17557870-24 from 1388909 to 1946078 tid 2 2024-09-11T03:43:12.916 INFO:tasks.rados.rados.2.smithi175.stdout:1356: write oid 39 current snap is 140 2024-09-11T03:43:12.916 INFO:tasks.rados.rados.2.smithi175.stdout:1356: seq_num 483 ranges {517781=475506,1731038=386743} 2024-09-11T03:43:12.921 INFO:tasks.rados.rados.2.smithi175.stdout:1356: writing smithi17557870-39 from 517781 to 993287 tid 1 2024-09-11T03:43:12.927 INFO:tasks.rados.rados.2.smithi175.stdout:1356: writing smithi17557870-39 from 1731038 to 2117781 tid 2 2024-09-11T03:43:12.928 INFO:tasks.rados.rados.2.smithi175.stdout:1357: read oid 37 snap 137 2024-09-11T03:43:12.928 INFO:tasks.rados.rados.2.smithi175.stdout:1357: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:43:12.928 INFO:tasks.rados.rados.2.smithi175.stdout:1358: snap_create 2024-09-11T03:43:12.928 INFO:tasks.rados.rados.2.smithi175.stdout:1351: expect (ObjNum 454 snap 130 seq_num 454) 2024-09-11T03:43:12.947 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:12.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:12.969 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:12.969 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:13.070 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:13.093 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:13.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:13.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:13.097 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:43:13.132 INFO:tasks.rados.rados.2.smithi175.stdout:1355: finishing write tid 1 to smithi17557870-24 2024-09-11T03:43:13.132 INFO:tasks.rados.rados.2.smithi175.stdout:1355: finishing write tid 2 to smithi17557870-24 2024-09-11T03:43:13.132 INFO:tasks.rados.rados.2.smithi175.stdout:1355: finishing write tid 3 to smithi17557870-24 2024-09-11T03:43:13.132 INFO:tasks.rados.rados.2.smithi175.stdout:1355: finishing write tid 4 to smithi17557870-24 2024-09-11T03:43:13.132 INFO:tasks.rados.rados.2.smithi175.stdout:1355: oid 24 updating version 0 to 519 2024-09-11T03:43:13.132 INFO:tasks.rados.rados.2.smithi175.stdout:1355: oid 24 updating version 519 to 520 2024-09-11T03:43:13.132 INFO:tasks.rados.rados.2.smithi175.stdout:1355: oid 24 version 520 is already newer than 518 2024-09-11T03:43:13.132 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 520 (ObjNum 482 snap 140 seq_num 482) dirty exists 2024-09-11T03:43:13.132 INFO:tasks.rados.rados.2.smithi175.stdout:1355: left oid 24 (ObjNum 482 snap 140 seq_num 482) 2024-09-11T03:43:13.132 INFO:tasks.rados.rados.2.smithi175.stdout:1356: finishing write tid 1 to smithi17557870-39 2024-09-11T03:43:13.132 INFO:tasks.rados.rados.2.smithi175.stdout:1356: finishing write tid 2 to smithi17557870-39 2024-09-11T03:43:13.132 INFO:tasks.rados.rados.2.smithi175.stdout:1356: finishing write tid 3 to smithi17557870-39 2024-09-11T03:43:13.132 INFO:tasks.rados.rados.2.smithi175.stdout:1356: finishing write tid 4 to smithi17557870-39 2024-09-11T03:43:13.132 INFO:tasks.rados.rados.2.smithi175.stdout:1356: oid 39 updating version 0 to 360 2024-09-11T03:43:13.133 INFO:tasks.rados.rados.2.smithi175.stdout:1356: oid 39 updating version 360 to 362 2024-09-11T03:43:13.133 INFO:tasks.rados.rados.2.smithi175.stdout:1356: oid 39 version 362 is already newer than 361 2024-09-11T03:43:13.133 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 362 (ObjNum 483 snap 140 seq_num 483) dirty exists 2024-09-11T03:43:13.133 INFO:tasks.rados.rados.2.smithi175.stdout:1356: left oid 39 (ObjNum 483 snap 140 seq_num 483) 2024-09-11T03:43:13.133 INFO:tasks.rados.rados.2.smithi175.stdout:1357: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-11T03:43:13.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:13.217 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:13.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:13.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:13.319 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:13.340 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:13.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:13.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:13.383 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:13 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1961430373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:13.383 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:13 smithi175 ceph-mon[30894]: osdmap e375: 12 total, 12 up, 10 in 2024-09-11T03:43:13.383 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:13 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:43:13.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:13 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1961430373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:13.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:13 smithi084 ceph-mon[27254]: osdmap e375: 12 total, 12 up, 10 in 2024-09-11T03:43:13.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:13 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:43:13.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:13 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1961430373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:13.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:13 smithi121 ceph-mon[30729]: osdmap e375: 12 total, 12 up, 10 in 2024-09-11T03:43:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:13 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:43:13.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:13.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:13.465 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:13.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:13.566 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:13.587 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:13.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:13.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:13.689 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:13.711 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:13.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:13.712 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:13.744 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:43:13.744 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:43:13.797 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:43:13.813 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:13.834 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:13.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:13.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:13.900 INFO:tasks.rados.rados.2.smithi175.stdout:1351: done (4 left) 2024-09-11T03:43:13.901 INFO:tasks.rados.rados.2.smithi175.stdout:1355: done (3 left) 2024-09-11T03:43:13.901 INFO:tasks.rados.rados.2.smithi175.stdout:1356: done (2 left) 2024-09-11T03:43:13.901 INFO:tasks.rados.rados.2.smithi175.stdout:1357: done (1 left) 2024-09-11T03:43:13.901 INFO:tasks.rados.rados.2.smithi175.stdout:1358: done (0 left) 2024-09-11T03:43:13.901 INFO:tasks.rados.rados.2.smithi175.stdout:1359: write oid 10 current snap is 141 2024-09-11T03:43:13.901 INFO:tasks.rados.rados.2.smithi175.stdout:1359: seq_num 484 ranges {534588=550755,1789520=345068} 2024-09-11T03:43:13.906 INFO:tasks.rados.rados.2.smithi175.stdout:1359: writing smithi17557870-10 from 534588 to 1085343 tid 1 2024-09-11T03:43:13.912 INFO:tasks.rados.rados.2.smithi175.stdout:1359: writing smithi17557870-10 from 1789520 to 2134588 tid 2 2024-09-11T03:43:13.912 INFO:tasks.rados.rados.2.smithi175.stdout:1360: read oid 37 snap -1 2024-09-11T03:43:13.912 INFO:tasks.rados.rados.2.smithi175.stdout:1360: expect deleted 2024-09-11T03:43:13.912 INFO:tasks.rados.rados.2.smithi175.stdout:1361: read oid 16 snap -1 2024-09-11T03:43:13.912 INFO:tasks.rados.rados.2.smithi175.stdout:1361: expect deleted 2024-09-11T03:43:13.912 INFO:tasks.rados.rados.2.smithi175.stdout:1362: delete oid 15 current snap is 141 2024-09-11T03:43:13.912 INFO:tasks.rados.rados.2.smithi175.stdout:1359: finishing write tid 1 to smithi17557870-10 2024-09-11T03:43:13.914 INFO:tasks.rados.rados.2.smithi175.stdout:1359: finishing write tid 2 to smithi17557870-10 2024-09-11T03:43:13.914 INFO:tasks.rados.rados.2.smithi175.stdout:1359: finishing write tid 3 to smithi17557870-10 2024-09-11T03:43:13.915 INFO:tasks.rados.rados.2.smithi175.stdout:1359: finishing write tid 4 to smithi17557870-10 2024-09-11T03:43:13.915 INFO:tasks.rados.rados.2.smithi175.stdout:1359: oid 10 updating version 0 to 504 2024-09-11T03:43:13.915 INFO:tasks.rados.rados.2.smithi175.stdout:1359: oid 10 version 504 is already newer than 503 2024-09-11T03:43:13.915 INFO:tasks.rados.rados.2.smithi175.stdout:1359: oid 10 version 504 is already newer than 502 2024-09-11T03:43:13.915 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 504 (ObjNum 484 snap 141 seq_num 484) dirty exists 2024-09-11T03:43:13.915 INFO:tasks.rados.rados.2.smithi175.stdout:1359: left oid 10 (ObjNum 484 snap 141 seq_num 484) 2024-09-11T03:43:13.915 INFO:tasks.rados.rados.2.smithi175.stdout:1359: done (3 left) 2024-09-11T03:43:13.915 INFO:tasks.rados.rados.2.smithi175.stdout:1360: done (2 left) 2024-09-11T03:43:13.915 INFO:tasks.rados.rados.2.smithi175.stdout:1361: done (1 left) 2024-09-11T03:43:13.915 INFO:tasks.rados.rados.2.smithi175.stdout:1362: done (0 left) 2024-09-11T03:43:13.915 INFO:tasks.rados.rados.2.smithi175.stdout:1363: delete oid 5 current snap is 141 2024-09-11T03:43:13.917 INFO:tasks.rados.rados.2.smithi175.stdout:1363: done (0 left) 2024-09-11T03:43:13.917 INFO:tasks.rados.rados.2.smithi175.stdout:1364: snap_remove snap 132 2024-09-11T03:43:13.936 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:13.957 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:13.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:13.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:14.046 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:14.058 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:14.079 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:14.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:14.080 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:14.182 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:14.203 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:14.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:14.204 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:14.305 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:14.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:14.328 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:14.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:14.380 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:14 smithi175 ceph-mon[30894]: pgmap v717: 57 pgs: 57 active+clean; 129 MiB data, 661 MiB used, 893 GiB / 894 GiB avail; 4.9 MiB/s rd, 722 KiB/s wr, 8 op/s 2024-09-11T03:43:14.380 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:14 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:43:14.380 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:14 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:43:14.380 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:14 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:43:14.381 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:14 smithi175 ceph-mon[30894]: osdmap e376: 12 total, 12 up, 10 in 2024-09-11T03:43:14.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:14 smithi084 ceph-mon[27254]: pgmap v717: 57 pgs: 57 active+clean; 129 MiB data, 661 MiB used, 893 GiB / 894 GiB avail; 4.9 MiB/s rd, 722 KiB/s wr, 8 op/s 2024-09-11T03:43:14.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:14 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:43:14.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:14 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:43:14.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:14 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:43:14.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:14 smithi084 ceph-mon[27254]: osdmap e376: 12 total, 12 up, 10 in 2024-09-11T03:43:14.407 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-11T03:43:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:14 smithi121 ceph-mon[30729]: pgmap v717: 57 pgs: 57 active+clean; 129 MiB data, 661 MiB used, 893 GiB / 894 GiB avail; 4.9 MiB/s rd, 722 KiB/s wr, 8 op/s 2024-09-11T03:43:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:14 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:43:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:14 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:43:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:14 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:43:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:14 smithi121 ceph-mon[30729]: osdmap e376: 12 total, 12 up, 10 in 2024-09-11T03:43:14.428 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:14.452 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:14.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:14.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:14.550 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [6, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:43:14.550 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:43:14.550 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-11T03:43:14.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-11T03:43:14.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:14.578 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:14.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:14.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:14.646 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:43:14.680 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:14.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:14.703 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:14.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:14.797 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:14.804 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:14.829 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:14.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:14.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:14.898 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:43:14.902 INFO:tasks.rados.rados.2.smithi175.stdout:1364: done (0 left) 2024-09-11T03:43:14.902 INFO:tasks.rados.rados.2.smithi175.stdout:1365: write (excl) oid 40 current snap is 141 2024-09-11T03:43:14.902 INFO:tasks.rados.rados.2.smithi175.stdout:1365: seq_num 487 ranges {585009=492855,1564966=620043} 2024-09-11T03:43:14.908 INFO:tasks.rados.rados.2.smithi175.stdout:1365: writing smithi17557870-40 from 585009 to 1077864 tid 1 2024-09-11T03:43:14.916 INFO:tasks.rados.rados.2.smithi175.stdout:1365: writing smithi17557870-40 from 1564966 to 2185009 tid 2 2024-09-11T03:43:14.916 INFO:tasks.rados.rados.2.smithi175.stdout:1366: read oid 33 snap -1 2024-09-11T03:43:14.917 INFO:tasks.rados.rados.2.smithi175.stdout:1366: expect deleted 2024-09-11T03:43:14.917 INFO:tasks.rados.rados.2.smithi175.stdout:1367: copy_from oid 2 from oid 6 current snap is 141 2024-09-11T03:43:14.917 INFO:tasks.rados.rados.2.smithi175.stdout:1365: finishing write tid 1 to smithi17557870-40 2024-09-11T03:43:14.917 INFO:tasks.rados.rados.2.smithi175.stdout:1368: read oid 17 snap -1 2024-09-11T03:43:14.917 INFO:tasks.rados.rados.2.smithi175.stdout:1368: expect (ObjNum 425 snap 129 seq_num 425) 2024-09-11T03:43:14.917 INFO:tasks.rados.rados.2.smithi175.stdout:1369: read oid 3 snap -1 2024-09-11T03:43:14.917 INFO:tasks.rados.rados.2.smithi175.stdout:1369: expect (ObjNum 447 snap 130 seq_num 447) 2024-09-11T03:43:14.917 INFO:tasks.rados.rados.2.smithi175.stdout:1370: read oid 14 snap -1 2024-09-11T03:43:14.917 INFO:tasks.rados.rados.2.smithi175.stdout:1370: expect deleted 2024-09-11T03:43:14.917 INFO:tasks.rados.rados.2.smithi175.stdout:1371: copy_from oid 36 from oid 35 current snap is 141 2024-09-11T03:43:14.917 INFO:tasks.rados.rados.2.smithi175.stdout:1372: read oid 13 snap -1 2024-09-11T03:43:14.917 INFO:tasks.rados.rados.2.smithi175.stdout:1372: expect deleted 2024-09-11T03:43:14.917 INFO:tasks.rados.rados.2.smithi175.stdout:1373: copy_from oid 7 from oid 41 current snap is 141 2024-09-11T03:43:14.918 INFO:tasks.rados.rados.2.smithi175.stdout:1374: read oid 4 snap -1 2024-09-11T03:43:14.918 INFO:tasks.rados.rados.2.smithi175.stdout:1374: expect deleted 2024-09-11T03:43:14.918 INFO:tasks.rados.rados.2.smithi175.stdout:1375: copy_from oid 19 from oid 23 current snap is 141 2024-09-11T03:43:14.918 INFO:tasks.rados.rados.2.smithi175.stdout:1376: write (excl) oid 10 current snap is 141 2024-09-11T03:43:14.918 INFO:tasks.rados.rados.2.smithi175.stdout:1376: seq_num 488 ranges {601816=568104,1623448=451823} 2024-09-11T03:43:14.923 INFO:tasks.rados.rados.2.smithi175.stdout:1376: writing smithi17557870-10 from 601816 to 1169920 tid 1 2024-09-11T03:43:14.928 INFO:tasks.rados.rados.2.smithi175.stdout:1376: writing smithi17557870-10 from 1623448 to 2075271 tid 2 2024-09-11T03:43:14.928 INFO:tasks.rados.rados.2.smithi175.stdout:1377: write (excl) oid 48 current snap is 141 2024-09-11T03:43:14.928 INFO:tasks.rados.rados.2.smithi175.stdout:1377: seq_num 489 ranges {618623=643353,1681930=536693} 2024-09-11T03:43:14.931 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:14.935 INFO:tasks.rados.rados.2.smithi175.stdout:1377: writing smithi17557870-48 from 618623 to 1261976 tid 1 2024-09-11T03:43:14.941 INFO:tasks.rados.rados.2.smithi175.stdout:1377: writing smithi17557870-48 from 1681930 to 2218623 tid 2 2024-09-11T03:43:14.941 INFO:tasks.rados.rados.2.smithi175.stdout:1378: delete oid 18 current snap is 141 2024-09-11T03:43:14.941 INFO:tasks.rados.rados.2.smithi175.stdout:1371: finishing copy_from to smithi17557870-36 2024-09-11T03:43:14.941 INFO:tasks.rados.rados.2.smithi175.stdout:1371: got expected ENOENT (src dne) 2024-09-11T03:43:14.941 INFO:tasks.rados.rados.2.smithi175.stdout:1371: finishing copy_from racing read to smithi17557870-36 2024-09-11T03:43:14.942 INFO:tasks.rados.rados.2.smithi175.stdout:1365: finishing write tid 2 to smithi17557870-40 2024-09-11T03:43:14.942 INFO:tasks.rados.rados.2.smithi175.stdout:1365: finishing write tid 3 to smithi17557870-40 2024-09-11T03:43:14.942 INFO:tasks.rados.rados.2.smithi175.stdout:1365: finishing write tid 4 to smithi17557870-40 2024-09-11T03:43:14.942 INFO:tasks.rados.rados.2.smithi175.stdout:1365: oid 40 updating version 0 to 390 2024-09-11T03:43:14.942 INFO:tasks.rados.rados.2.smithi175.stdout:1365: oid 40 version 390 is already newer than 388 2024-09-11T03:43:14.942 INFO:tasks.rados.rados.2.smithi175.stdout:1365: oid 40 version 390 is already newer than 389 2024-09-11T03:43:14.942 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 390 (ObjNum 487 snap 141 seq_num 487) dirty exists 2024-09-11T03:43:14.942 INFO:tasks.rados.rados.2.smithi175.stdout:1365: left oid 40 (ObjNum 487 snap 141 seq_num 487) 2024-09-11T03:43:14.943 INFO:tasks.rados.rados.2.smithi175.stdout:1368: expect (ObjNum 425 snap 129 seq_num 425) 2024-09-11T03:43:14.953 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:14.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:14.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:15.055 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:15.078 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:15.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:15.079 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:15.118 INFO:tasks.rados.rados.2.smithi175.stdout:1375: finishing copy_from to smithi17557870-19 2024-09-11T03:43:15.118 INFO:tasks.rados.rados.2.smithi175.stdout:1375: got expected ENOENT (src dne) 2024-09-11T03:43:15.118 INFO:tasks.rados.rados.2.smithi175.stdout:1375: finishing copy_from racing read to smithi17557870-19 2024-09-11T03:43:15.118 INFO:tasks.rados.rados.2.smithi175.stdout:1367: finishing copy_from racing read to smithi17557870-2 2024-09-11T03:43:15.119 INFO:tasks.rados.rados.2.smithi175.stdout:1373: finishing copy_from racing read to smithi17557870-7 2024-09-11T03:43:15.119 INFO:tasks.rados.rados.2.smithi175.stdout:1376: finishing write tid 1 to smithi17557870-10 2024-09-11T03:43:15.119 INFO:tasks.rados.rados.2.smithi175.stdout:1376: finishing write tid 2 to smithi17557870-10 2024-09-11T03:43:15.119 INFO:tasks.rados.rados.2.smithi175.stdout:1365: done (13 left) 2024-09-11T03:43:15.119 INFO:tasks.rados.rados.2.smithi175.stdout:1366: done (12 left) 2024-09-11T03:43:15.119 INFO:tasks.rados.rados.2.smithi175.stdout:1368: done (11 left) 2024-09-11T03:43:15.119 INFO:tasks.rados.rados.2.smithi175.stdout:1370: done (10 left) 2024-09-11T03:43:15.119 INFO:tasks.rados.rados.2.smithi175.stdout:1371: done (9 left) 2024-09-11T03:43:15.119 INFO:tasks.rados.rados.2.smithi175.stdout:1372: done (8 left) 2024-09-11T03:43:15.119 INFO:tasks.rados.rados.2.smithi175.stdout:1374: done (7 left) 2024-09-11T03:43:15.120 INFO:tasks.rados.rados.2.smithi175.stdout:1375: done (6 left) 2024-09-11T03:43:15.120 INFO:tasks.rados.rados.2.smithi175.stdout:1378: done (5 left) 2024-09-11T03:43:15.120 INFO:tasks.rados.rados.2.smithi175.stdout:1379: snap_remove snap 135 2024-09-11T03:43:15.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:15.183 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:43:15.184 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:43:15.201 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:15.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:15.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:15.304 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:15.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:15.326 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:15.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:15.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:15 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:43:15.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:15 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:43:15.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:15 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1587683695' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:15.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:15 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1587683695' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:15.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:15 smithi084 ceph-mon[27254]: osdmap e377: 12 total, 12 up, 10 in 2024-09-11T03:43:15.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:15 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:43:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:15 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:43:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:15 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1587683695' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:15 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1587683695' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:15 smithi121 ceph-mon[30729]: osdmap e377: 12 total, 12 up, 10 in 2024-09-11T03:43:15.427 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:15.449 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:15.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:15.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:15.461 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:15 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:43:15.462 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:15 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:43:15.462 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:15 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1587683695' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:15.462 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:15 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1587683695' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:15.462 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:15 smithi175 ceph-mon[30894]: osdmap e377: 12 total, 12 up, 10 in 2024-09-11T03:43:15.552 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:15.578 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:15.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:15.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:15.622 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:43:15.680 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:15.701 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:15.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:15.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:15.748 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-11T03:43:15.803 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:15.827 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:15.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:15.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:15.904 INFO:tasks.rados.rados.2.smithi175.stdout:1379: done (5 left) 2024-09-11T03:43:15.905 INFO:tasks.rados.rados.2.smithi175.stdout:1380: snap_create 2024-09-11T03:43:15.905 INFO:tasks.rados.rados.2.smithi175.stdout:1376: finishing write tid 3 to smithi17557870-10 2024-09-11T03:43:15.905 INFO:tasks.rados.rados.2.smithi175.stdout:1369: expect (ObjNum 447 snap 130 seq_num 447) 2024-09-11T03:43:15.929 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:15.951 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:15.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:16.034 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:43:16.052 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:16.061 INFO:tasks.rados.rados.2.smithi175.stdout:1376: finishing write tid 4 to smithi17557870-10 2024-09-11T03:43:16.061 INFO:tasks.rados.rados.2.smithi175.stdout:1376: oid 10 updating version 0 to 509 2024-09-11T03:43:16.061 INFO:tasks.rados.rados.2.smithi175.stdout:1376: oid 10 updating version 509 to 510 2024-09-11T03:43:16.061 INFO:tasks.rados.rados.2.smithi175.stdout:1376: oid 10 updating version 510 to 511 2024-09-11T03:43:16.062 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 511 (ObjNum 488 snap 141 seq_num 488) dirty exists 2024-09-11T03:43:16.062 INFO:tasks.rados.rados.2.smithi175.stdout:1376: left oid 10 (ObjNum 488 snap 141 seq_num 488) 2024-09-11T03:43:16.062 INFO:tasks.rados.rados.2.smithi175.stdout:1367: finishing copy_from to smithi17557870-2 2024-09-11T03:43:16.062 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 292 (ObjNum 335 snap 95 seq_num 335) dirty exists 2024-09-11T03:43:16.062 INFO:tasks.rados.rados.2.smithi175.stdout:1377: finishing write tid 1 to smithi17557870-48 2024-09-11T03:43:16.062 INFO:tasks.rados.rados.2.smithi175.stdout:1373: finishing copy_from to smithi17557870-7 2024-09-11T03:43:16.062 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 356 (ObjNum 413 snap 127 seq_num 413) dirty exists 2024-09-11T03:43:16.062 INFO:tasks.rados.rados.2.smithi175.stdout:1377: finishing write tid 2 to smithi17557870-48 2024-09-11T03:43:16.062 INFO:tasks.rados.rados.2.smithi175.stdout:1377: finishing write tid 3 to smithi17557870-48 2024-09-11T03:43:16.062 INFO:tasks.rados.rados.2.smithi175.stdout:1377: finishing write tid 4 to smithi17557870-48 2024-09-11T03:43:16.062 INFO:tasks.rados.rados.2.smithi175.stdout:1377: oid 48 updating version 0 to 457 2024-09-11T03:43:16.062 INFO:tasks.rados.rados.2.smithi175.stdout:1377: oid 48 version 457 is already newer than 456 2024-09-11T03:43:16.062 INFO:tasks.rados.rados.2.smithi175.stdout:1377: oid 48 updating version 457 to 458 2024-09-11T03:43:16.063 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 458 (ObjNum 489 snap 141 seq_num 489) dirty exists 2024-09-11T03:43:16.063 INFO:tasks.rados.rados.2.smithi175.stdout:1377: left oid 48 (ObjNum 489 snap 141 seq_num 489) 2024-09-11T03:43:16.076 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:16.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:16.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:16.102 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:16.178 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:16.200 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:16.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:16.201 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:16.217 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:43:16.217 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:43:16.217 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 378, 2024-09-11T03:43:16.217 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "10fe7946-6fee-11ef-bcea-c7b262605968", 2024-09-11T03:43:16.217 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-09-11T03:30:30.532652+0000", 2024-09-11T03:43:16.218 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-09-11T03:43:15.899638+0000", 2024-09-11T03:43:16.218 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-09-11T03:37:26.787638+0000", 2024-09-11T03:43:16.218 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-09-11T03:40:04.952764+0000", 2024-09-11T03:43:16.218 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T03:43:16.218 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5806080, 2024-09-11T03:43:16.218 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-09-11T03:43:16.218 INFO:teuthology.orchestra.run.smithi084.stdout: "nodeep-scrub", 2024-09-11T03:43:16.218 INFO:teuthology.orchestra.run.smithi084.stdout: "noscrub", 2024-09-11T03:43:16.218 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-09-11T03:43:16.218 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-09-11T03:43:16.219 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-09-11T03:43:16.219 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-09-11T03:43:16.219 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:43:16.219 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 26, 2024-09-11T03:43:16.219 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T03:43:16.219 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T03:43:16.219 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T03:43:16.219 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-09-11T03:43:16.219 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 2, 2024-09-11T03:43:16.219 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 12, 2024-09-11T03:43:16.219 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-09-11T03:43:16.220 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-09-11T03:43:16.220 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-09-11T03:43:16.220 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-09-11T03:43:16.220 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-09-11T03:43:16.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.220 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-09-11T03:43:16.220 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-09-11T03:43:16.220 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-11T03:33:42.916933+0000", 2024-09-11T03:43:16.220 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-11T03:43:16.220 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-11T03:43:16.220 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-11T03:43:16.221 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-11T03:43:16.221 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-11T03:43:16.221 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-11T03:43:16.221 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-11T03:43:16.221 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-11T03:43:16.221 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T03:43:16.221 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T03:43:16.221 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-11T03:43:16.221 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-11T03:43:16.221 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-11T03:43:16.221 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 31, 2024-09-11T03:43:16.222 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 31, 2024-09-11T03:43:16.222 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 31, 2024-09-11T03:43:16.222 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 31, 2024-09-11T03:43:16.222 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 31, 2024-09-11T03:43:16.222 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-11T03:43:16.222 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-11T03:43:16.222 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-11T03:43:16.222 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-11T03:43:16.222 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-11T03:43:16.222 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-11T03:43:16.222 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-11T03:43:16.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.223 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "222", 2024-09-11T03:43:16.223 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-11T03:43:16.223 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T03:43:16.223 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "220", 2024-09-11T03:43:16.223 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-11T03:43:16.223 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-11T03:43:16.223 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-11T03:43:16.223 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-11T03:43:16.223 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-11T03:43:16.223 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-11T03:43:16.224 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-11T03:43:16.224 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-11T03:43:16.224 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-11T03:43:16.224 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-11T03:43:16.224 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-11T03:43:16.224 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-11T03:43:16.224 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-11T03:43:16.224 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-11T03:43:16.224 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-11T03:43:16.224 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T03:43:16.225 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T03:43:16.225 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T03:43:16.225 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-11T03:43:16.225 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-11T03:43:16.225 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-11T03:43:16.226 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-11T03:43:16.226 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-11T03:43:16.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.226 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-11T03:43:16.226 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-11T03:43:16.226 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-11T03:43:16.226 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-11T03:43:16.226 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-11T03:43:16.226 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T03:43:16.226 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-11T03:43:16.226 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-11T03:43:16.227 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-11T03:43:16.227 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-11T03:43:16.227 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-11T03:43:16.227 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-11T03:43:16.227 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-09-11T03:43:16.227 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-09-11T03:43:16.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.228 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-11T03:43:16.228 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-09-11T03:43:16.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.228 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-11T03:43:16.228 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-11T03:43:16.228 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.940000057220459, 2024-09-11T03:43:16.228 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.940000057220459, 2024-09-11T03:43:16.228 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.82999998331069946, 2024-09-11T03:43:16.228 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.6100000143051147, 2024-09-11T03:43:16.228 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.6100000143051147, 2024-09-11T03:43:16.229 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-11T03:43:16.229 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-11T03:43:16.229 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-11T03:43:16.229 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.229 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-11T03:43:16.229 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_0", 2024-09-11T03:43:16.229 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-11T03:38:28.163534+0000", 2024-09-11T03:43:16.229 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-11T03:43:16.229 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T03:43:16.230 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-11T03:43:16.230 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-11T03:43:16.230 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-11T03:43:16.230 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-11T03:43:16.230 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-11T03:43:16.230 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-11T03:43:16.230 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T03:43:16.230 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T03:43:16.230 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-11T03:43:16.230 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-11T03:43:16.230 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-11T03:43:16.230 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 26, 2024-09-11T03:43:16.231 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 26, 2024-09-11T03:43:16.231 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 26, 2024-09-11T03:43:16.231 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 26, 2024-09-11T03:43:16.231 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 26, 2024-09-11T03:43:16.231 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-11T03:43:16.231 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-11T03:43:16.231 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-11T03:43:16.231 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-11T03:43:16.231 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-11T03:43:16.231 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-11T03:43:16.231 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-11T03:43:16.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.232 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "378", 2024-09-11T03:43:16.232 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-11T03:43:16.232 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T03:43:16.232 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "276", 2024-09-11T03:43:16.232 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-11T03:43:16.232 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-11T03:43:16.232 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 142, 2024-09-11T03:43:16.232 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 378, 2024-09-11T03:43:16.232 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-11T03:43:16.232 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-11T03:43:16.232 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-11T03:43:16.233 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-11T03:43:16.233 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-11T03:43:16.233 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-11T03:43:16.233 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-11T03:43:16.233 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-11T03:43:16.233 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-11T03:43:16.233 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-11T03:43:16.233 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-11T03:43:16.233 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T03:43:16.233 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T03:43:16.233 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T03:43:16.234 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-11T03:43:16.234 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-11T03:43:16.234 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-11T03:43:16.234 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-11T03:43:16.234 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-11T03:43:16.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.234 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-11T03:43:16.234 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-11T03:43:16.234 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-11T03:43:16.234 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-11T03:43:16.234 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-11T03:43:16.235 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T03:43:16.235 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-11T03:43:16.235 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-11T03:43:16.235 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-11T03:43:16.235 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-11T03:43:16.235 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-11T03:43:16.235 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-11T03:43:16.235 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-11T03:43:16.235 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-09-11T03:43:16.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.235 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-11T03:43:16.236 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-11T03:43:16.236 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.8600000143051147, 2024-09-11T03:43:16.236 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.8600000143051147, 2024-09-11T03:43:16.236 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.82999998331069946, 2024-09-11T03:43:16.236 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.5399999618530273, 2024-09-11T03:43:16.236 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.5399999618530273, 2024-09-11T03:43:16.236 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-11T03:43:16.236 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-11T03:43:16.236 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-11T03:43:16.236 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.237 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.237 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:43:16.237 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-11T03:43:16.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.237 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-11T03:43:16.237 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "9a431e48-e46b-44f9-bfa7-cb56c1484e1c", 2024-09-11T03:43:16.237 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:43:16.237 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:43:16.237 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:43:16.237 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:43:16.237 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:43:16.238 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:43:16.238 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 8, 2024-09-11T03:43:16.238 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 280, 2024-09-11T03:43:16.238 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:43:16.238 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:43:16.238 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:43:16.238 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.238 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.238 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-09-11T03:43:16.238 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:43:16.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.239 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.239 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-09-11T03:43:16.239 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:43:16.239 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.239 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.240 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:43:16.240 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.240 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.241 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-09-11T03:43:16.241 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:43:16.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.241 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.241 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-09-11T03:43:16.241 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:43:16.241 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.241 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.242 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:43:16.242 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.242 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.242 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-09-11T03:43:16.242 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:43:16.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.242 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.242 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-09-11T03:43:16.242 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:43:16.243 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.243 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.243 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:43:16.243 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.243 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.243 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-09-11T03:43:16.243 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:43:16.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.244 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.244 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-09-11T03:43:16.244 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:43:16.244 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.244 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.244 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6803/1376667411", 2024-09-11T03:43:16.244 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6805/1376667411", 2024-09-11T03:43:16.244 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6809/1376667411", 2024-09-11T03:43:16.244 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/1376667411", 2024-09-11T03:43:16.244 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:43:16.245 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:43:16.245 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:43:16.245 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.245 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-11T03:43:16.245 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "62fd6e25-3508-40b4-a0f4-f7de137e3bb5", 2024-09-11T03:43:16.245 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:43:16.245 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:43:16.246 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:43:16.246 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:43:16.246 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:43:16.246 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:43:16.246 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-09-11T03:43:16.246 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 280, 2024-09-11T03:43:16.246 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:43:16.246 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:43:16.246 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:43:16.246 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.247 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.247 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-09-11T03:43:16.247 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:43:16.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.247 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.247 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-09-11T03:43:16.247 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:43:16.247 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.247 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.248 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:43:16.248 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.248 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.248 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-09-11T03:43:16.248 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:43:16.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.248 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.249 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-09-11T03:43:16.249 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:43:16.249 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.249 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.249 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:43:16.249 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.249 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.249 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-09-11T03:43:16.249 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:43:16.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.250 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.250 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6817", 2024-09-11T03:43:16.250 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:43:16.250 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.250 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.250 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:43:16.250 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.251 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.251 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-09-11T03:43:16.251 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:43:16.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.251 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.251 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-09-11T03:43:16.251 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:43:16.251 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.251 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.251 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6811/3138939249", 2024-09-11T03:43:16.252 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6813/3138939249", 2024-09-11T03:43:16.252 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6817/3138939249", 2024-09-11T03:43:16.252 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/3138939249", 2024-09-11T03:43:16.252 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:43:16.252 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:43:16.252 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:43:16.252 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.252 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-11T03:43:16.253 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ff351425-4715-49dc-be49-f27bf205d57e", 2024-09-11T03:43:16.253 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:43:16.253 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:43:16.253 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:43:16.253 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:43:16.253 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:43:16.253 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:43:16.253 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 18, 2024-09-11T03:43:16.253 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 281, 2024-09-11T03:43:16.253 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:43:16.253 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:43:16.254 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:43:16.254 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.254 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.254 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6818", 2024-09-11T03:43:16.254 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:43:16.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.254 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.254 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6819", 2024-09-11T03:43:16.254 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:43:16.254 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.255 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.255 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:43:16.255 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.255 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.255 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6820", 2024-09-11T03:43:16.255 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:43:16.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.255 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.255 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6821", 2024-09-11T03:43:16.255 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:43:16.256 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.256 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.256 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:43:16.256 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.256 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.256 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6824", 2024-09-11T03:43:16.256 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:43:16.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.257 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.257 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6825", 2024-09-11T03:43:16.257 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:43:16.257 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.257 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.257 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:43:16.257 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.257 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.258 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6822", 2024-09-11T03:43:16.258 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:43:16.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.258 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.258 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6823", 2024-09-11T03:43:16.258 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:43:16.258 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.258 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.258 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6819/3001811591", 2024-09-11T03:43:16.258 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6821/3001811591", 2024-09-11T03:43:16.259 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6825/3001811591", 2024-09-11T03:43:16.259 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6823/3001811591", 2024-09-11T03:43:16.259 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:43:16.259 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:43:16.259 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:43:16.259 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.259 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-11T03:43:16.259 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "2518a93e-c612-4731-91ce-5685c25ffa9f", 2024-09-11T03:43:16.259 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:43:16.260 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:43:16.260 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:43:16.260 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:43:16.260 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:43:16.260 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:43:16.260 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 26, 2024-09-11T03:43:16.260 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 332, 2024-09-11T03:43:16.260 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:43:16.260 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:43:16.260 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:43:16.260 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.261 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.261 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6826", 2024-09-11T03:43:16.261 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:43:16.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.261 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.261 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6827", 2024-09-11T03:43:16.261 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:43:16.261 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.261 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.262 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:43:16.262 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.262 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.262 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6828", 2024-09-11T03:43:16.262 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:43:16.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.262 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.262 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6829", 2024-09-11T03:43:16.262 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:43:16.262 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.263 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.263 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:43:16.263 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.263 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.263 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6832", 2024-09-11T03:43:16.263 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:43:16.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.263 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.264 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6833", 2024-09-11T03:43:16.264 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:43:16.264 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.264 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.264 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:43:16.264 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.264 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.264 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6830", 2024-09-11T03:43:16.264 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:43:16.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.265 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.265 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6831", 2024-09-11T03:43:16.265 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:43:16.265 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.265 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.265 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6827/302512129", 2024-09-11T03:43:16.265 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6829/302512129", 2024-09-11T03:43:16.266 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6833/302512129", 2024-09-11T03:43:16.266 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6831/302512129", 2024-09-11T03:43:16.266 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:43:16.266 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:43:16.266 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:43:16.266 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.266 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-11T03:43:16.266 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d12f3c2d-95cc-4276-a7e1-bee982d58948", 2024-09-11T03:43:16.266 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:43:16.267 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:43:16.267 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:43:16.267 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:43:16.267 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:43:16.267 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:43:16.267 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-09-11T03:43:16.267 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 281, 2024-09-11T03:43:16.267 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:43:16.267 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:43:16.267 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:43:16.267 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.268 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.268 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6800", 2024-09-11T03:43:16.268 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:43:16.268 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.268 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.268 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6801", 2024-09-11T03:43:16.268 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:43:16.268 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.269 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.269 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:43:16.269 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.269 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.269 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6802", 2024-09-11T03:43:16.269 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:43:16.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.269 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.270 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6803", 2024-09-11T03:43:16.270 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:43:16.270 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.270 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.270 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:43:16.270 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.270 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.270 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6806", 2024-09-11T03:43:16.270 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:43:16.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.271 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.271 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6807", 2024-09-11T03:43:16.271 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:43:16.271 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.271 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.271 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:43:16.271 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.271 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.272 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6804", 2024-09-11T03:43:16.272 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:43:16.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.272 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.272 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6805", 2024-09-11T03:43:16.272 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:43:16.272 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.272 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.272 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.121:6801/3873214513", 2024-09-11T03:43:16.273 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.121:6803/3873214513", 2024-09-11T03:43:16.273 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.121:6807/3873214513", 2024-09-11T03:43:16.273 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.121:6805/3873214513", 2024-09-11T03:43:16.273 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:43:16.273 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:43:16.273 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:43:16.273 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.273 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-11T03:43:16.273 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "32164dd5-7e8f-4925-8526-bbb1e33e3364", 2024-09-11T03:43:16.274 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:43:16.274 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:43:16.274 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:43:16.274 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:43:16.274 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:43:16.274 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:43:16.274 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 38, 2024-09-11T03:43:16.274 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 333, 2024-09-11T03:43:16.274 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:43:16.274 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:43:16.274 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:43:16.274 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.275 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.275 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6808", 2024-09-11T03:43:16.275 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:43:16.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.275 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.275 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6809", 2024-09-11T03:43:16.275 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:43:16.275 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.276 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.276 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:43:16.276 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.276 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.276 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6810", 2024-09-11T03:43:16.276 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:43:16.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.276 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.277 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6811", 2024-09-11T03:43:16.277 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:43:16.277 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.277 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.277 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:43:16.277 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.277 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.277 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6814", 2024-09-11T03:43:16.278 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:43:16.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.278 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.278 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6815", 2024-09-11T03:43:16.278 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:43:16.278 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.278 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.278 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:43:16.278 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.278 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.279 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6812", 2024-09-11T03:43:16.279 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:43:16.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.279 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.279 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6813", 2024-09-11T03:43:16.279 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:43:16.279 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.279 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.279 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.121:6809/918612194", 2024-09-11T03:43:16.279 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.121:6811/918612194", 2024-09-11T03:43:16.279 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.121:6815/918612194", 2024-09-11T03:43:16.280 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.121:6813/918612194", 2024-09-11T03:43:16.280 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:43:16.280 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:43:16.280 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:43:16.280 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.280 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-11T03:43:16.280 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "3f107a88-803f-4cd8-804a-241310c282ea", 2024-09-11T03:43:16.280 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:43:16.280 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-09-11T03:43:16.280 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-09-11T03:43:16.281 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:43:16.281 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:43:16.281 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:43:16.281 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 45, 2024-09-11T03:43:16.281 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 95, 2024-09-11T03:43:16.281 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:43:16.281 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:43:16.281 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:43:16.281 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.281 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.282 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6816", 2024-09-11T03:43:16.282 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:43:16.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.282 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.282 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6817", 2024-09-11T03:43:16.282 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:43:16.282 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.282 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.282 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:43:16.282 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.283 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.283 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6818", 2024-09-11T03:43:16.283 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:43:16.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.283 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.283 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6819", 2024-09-11T03:43:16.283 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:43:16.283 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.283 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.283 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:43:16.284 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.284 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.284 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6822", 2024-09-11T03:43:16.284 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:43:16.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.284 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.284 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6823", 2024-09-11T03:43:16.284 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:43:16.284 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.284 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.285 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:43:16.285 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.285 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.285 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6820", 2024-09-11T03:43:16.285 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:43:16.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.285 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.285 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6821", 2024-09-11T03:43:16.285 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:43:16.286 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.286 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.286 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.121:6817/3384813363", 2024-09-11T03:43:16.286 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.121:6819/3384813363", 2024-09-11T03:43:16.286 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.121:6823/3384813363", 2024-09-11T03:43:16.286 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.121:6821/3384813363", 2024-09-11T03:43:16.286 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:43:16.286 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:43:16.286 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:43:16.286 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.287 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-11T03:43:16.287 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "dfee7f95-1fb9-4493-9907-cb10d1837bdb", 2024-09-11T03:43:16.287 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:43:16.287 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:43:16.287 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:43:16.287 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:43:16.287 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:43:16.287 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:43:16.288 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 52, 2024-09-11T03:43:16.288 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 309, 2024-09-11T03:43:16.288 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:43:16.288 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:43:16.288 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:43:16.288 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.288 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.288 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6824", 2024-09-11T03:43:16.288 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:43:16.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.289 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.289 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6825", 2024-09-11T03:43:16.289 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:43:16.289 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.289 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.289 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:43:16.289 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.290 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.290 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6826", 2024-09-11T03:43:16.290 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:43:16.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.290 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.290 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6827", 2024-09-11T03:43:16.290 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:43:16.290 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.290 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.291 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:43:16.291 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.291 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.291 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6830", 2024-09-11T03:43:16.291 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:43:16.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.291 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.291 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6831", 2024-09-11T03:43:16.291 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:43:16.291 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.292 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.292 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:43:16.292 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.292 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.292 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6828", 2024-09-11T03:43:16.292 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:43:16.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.292 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.293 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6829", 2024-09-11T03:43:16.293 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:43:16.293 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.293 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.293 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.121:6825/449222054", 2024-09-11T03:43:16.293 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.121:6827/449222054", 2024-09-11T03:43:16.293 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.121:6831/449222054", 2024-09-11T03:43:16.293 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.121:6829/449222054", 2024-09-11T03:43:16.293 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:43:16.293 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:43:16.293 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:43:16.294 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.294 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-09-11T03:43:16.294 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4", 2024-09-11T03:43:16.294 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:43:16.294 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:43:16.294 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:43:16.294 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:43:16.294 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:43:16.294 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:43:16.295 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 60, 2024-09-11T03:43:16.295 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 278, 2024-09-11T03:43:16.295 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:43:16.295 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:43:16.295 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:43:16.295 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.295 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.295 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6800", 2024-09-11T03:43:16.295 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:43:16.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.296 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.296 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6801", 2024-09-11T03:43:16.296 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:43:16.296 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.296 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.296 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:43:16.296 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.297 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.297 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6802", 2024-09-11T03:43:16.297 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:43:16.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.297 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.297 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6803", 2024-09-11T03:43:16.297 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:43:16.297 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.297 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.297 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:43:16.298 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.298 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.298 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6806", 2024-09-11T03:43:16.298 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:43:16.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.298 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.298 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6807", 2024-09-11T03:43:16.298 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:43:16.298 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.299 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.299 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:43:16.299 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.299 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.299 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6804", 2024-09-11T03:43:16.299 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:43:16.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.299 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.300 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6805", 2024-09-11T03:43:16.300 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:43:16.300 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.300 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.300 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.175:6801/2660128384", 2024-09-11T03:43:16.300 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.175:6803/2660128384", 2024-09-11T03:43:16.300 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.175:6807/2660128384", 2024-09-11T03:43:16.300 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.175:6805/2660128384", 2024-09-11T03:43:16.300 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:43:16.301 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:43:16.301 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:43:16.301 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.301 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-09-11T03:43:16.301 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "50162e42-0443-41d1-8709-a0db61c087da", 2024-09-11T03:43:16.301 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:43:16.301 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:43:16.301 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:43:16.301 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:43:16.302 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:43:16.302 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:43:16.302 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 69, 2024-09-11T03:43:16.302 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 312, 2024-09-11T03:43:16.302 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:43:16.302 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:43:16.302 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:43:16.302 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.302 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.302 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6808", 2024-09-11T03:43:16.303 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:43:16.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.303 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.303 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6809", 2024-09-11T03:43:16.303 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:43:16.303 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.303 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.303 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:43:16.303 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.304 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.304 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6810", 2024-09-11T03:43:16.304 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:43:16.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.304 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.304 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6811", 2024-09-11T03:43:16.304 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:43:16.304 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.305 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.305 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:43:16.305 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.305 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.305 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6814", 2024-09-11T03:43:16.305 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:43:16.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.305 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.306 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6815", 2024-09-11T03:43:16.306 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:43:16.306 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.306 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.306 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:43:16.306 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.306 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.306 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6812", 2024-09-11T03:43:16.306 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:43:16.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.307 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.307 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6813", 2024-09-11T03:43:16.307 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:43:16.307 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.307 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.307 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.175:6809/1270991649", 2024-09-11T03:43:16.307 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.175:6811/1270991649", 2024-09-11T03:43:16.307 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.175:6815/1270991649", 2024-09-11T03:43:16.307 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.175:6813/1270991649", 2024-09-11T03:43:16.308 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:43:16.308 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:43:16.308 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:43:16.308 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.308 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-09-11T03:43:16.308 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4b9426e1-bac8-47d3-ad58-54b89e8d1eb1", 2024-09-11T03:43:16.308 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:43:16.308 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-09-11T03:43:16.309 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-09-11T03:43:16.309 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:43:16.309 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:43:16.309 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:43:16.309 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 76, 2024-09-11T03:43:16.309 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 182, 2024-09-11T03:43:16.309 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:43:16.309 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:43:16.309 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:43:16.309 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.310 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.310 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6816", 2024-09-11T03:43:16.310 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:43:16.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.310 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.310 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6817", 2024-09-11T03:43:16.310 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:43:16.310 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.310 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.311 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:43:16.311 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.311 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.311 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6818", 2024-09-11T03:43:16.311 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:43:16.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.311 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.311 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6819", 2024-09-11T03:43:16.312 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:43:16.312 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.312 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.312 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:43:16.312 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.312 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.312 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6822", 2024-09-11T03:43:16.312 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:43:16.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.313 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.313 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6823", 2024-09-11T03:43:16.313 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:43:16.313 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.313 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.313 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:43:16.313 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.313 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.314 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6820", 2024-09-11T03:43:16.314 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:43:16.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.314 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.314 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6821", 2024-09-11T03:43:16.314 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:43:16.314 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.314 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.315 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.175:6817/3877631021", 2024-09-11T03:43:16.315 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.175:6819/3877631021", 2024-09-11T03:43:16.315 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.175:6823/3877631021", 2024-09-11T03:43:16.315 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.175:6821/3877631021", 2024-09-11T03:43:16.315 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:43:16.315 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:43:16.315 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:43:16.315 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.315 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-09-11T03:43:16.316 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "cd1a28a1-413c-4274-9876-e1bdb7703be6", 2024-09-11T03:43:16.316 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:43:16.316 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:43:16.316 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:43:16.316 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:43:16.316 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:43:16.316 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:43:16.316 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 83, 2024-09-11T03:43:16.316 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 222, 2024-09-11T03:43:16.316 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:43:16.317 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:43:16.317 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:43:16.317 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.317 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.317 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6824", 2024-09-11T03:43:16.317 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:43:16.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.317 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.317 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6825", 2024-09-11T03:43:16.318 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:43:16.318 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.318 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.318 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:43:16.318 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.318 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.318 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6826", 2024-09-11T03:43:16.318 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:43:16.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.319 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.319 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6827", 2024-09-11T03:43:16.319 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:43:16.319 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.319 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.319 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:43:16.319 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.319 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.320 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6830", 2024-09-11T03:43:16.320 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:43:16.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.320 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.320 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6831", 2024-09-11T03:43:16.320 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:43:16.320 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.320 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.320 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:43:16.320 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:43:16.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.321 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:43:16.321 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6828", 2024-09-11T03:43:16.321 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:43:16.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.321 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:43:16.321 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6829", 2024-09-11T03:43:16.321 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:43:16.321 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.321 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.322 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.175:6825/1150051236", 2024-09-11T03:43:16.322 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.175:6827/1150051236", 2024-09-11T03:43:16.322 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.175:6831/1150051236", 2024-09-11T03:43:16.322 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.175:6829/1150051236", 2024-09-11T03:43:16.322 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:43:16.322 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:43:16.322 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:43:16.322 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.322 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.322 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:43:16.322 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-09-11T03:43:16.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.323 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-11T03:43:16.323 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:43:16.323 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:43:16.323 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:43:16.323 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:43:16.323 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:43:16.323 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:42:19.799941+0000", 2024-09-11T03:43:16.323 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:43:16.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-11T03:43:16.324 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:43:16.324 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:43:16.324 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:43:16.324 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:43:16.324 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:43:16.324 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:43:14.014218+0000", 2024-09-11T03:43:16.324 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:43:16.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-11T03:43:16.325 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:43:16.325 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:43:16.325 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:43:16.325 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:43:16.325 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:43:16.325 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:42:02.898695+0000", 2024-09-11T03:43:16.325 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:43:16.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-11T03:43:16.325 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:43:16.326 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:43:16.326 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:43:16.326 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:43:16.326 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:43:16.326 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:42:41.819237+0000", 2024-09-11T03:43:16.326 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:43:16.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.326 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-11T03:43:16.326 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:43:16.326 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:43:16.326 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:43:16.327 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:43:16.327 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:43:16.327 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:42:08.783166+0000", 2024-09-11T03:43:16.327 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:43:16.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.327 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-11T03:43:16.327 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:43:16.327 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:43:16.327 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:43:16.327 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:43:16.328 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:43:16.328 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:42:25.432246+0000", 2024-09-11T03:43:16.328 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:43:16.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.328 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-11T03:43:16.328 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:43:16.328 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:43:16.328 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:43:16.328 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:43:16.328 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-09-11T03:43:16.329 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:42:13.715916+0000", 2024-09-11T03:43:16.329 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:43:16.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.329 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-11T03:43:16.329 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:43:16.329 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:43:16.329 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:43:16.329 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:43:16.329 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:43:16.329 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:42:46.980292+0000", 2024-09-11T03:43:16.329 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:43:16.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.330 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-09-11T03:43:16.330 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:43:16.330 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:43:16.330 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:43:16.330 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:43:16.330 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:43:16.330 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:43:00.798033+0000", 2024-09-11T03:43:16.330 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:43:16.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.331 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-09-11T03:43:16.331 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:43:16.331 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:16.332 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:43:16.333 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:43:16.333 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:43:16.333 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:43:16.333 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:42:42.240655+0000", 2024-09-11T03:43:16.333 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:43:16.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.333 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-09-11T03:43:16.333 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:43:16.333 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:43:16.334 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:43:16.334 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:43:16.334 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-09-11T03:43:16.334 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:43:11.996326+0000", 2024-09-11T03:43:16.334 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:43:16.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.334 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-09-11T03:43:16.334 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:43:16.334 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:43:16.334 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:43:16.335 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:43:16.335 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:43:16.335 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:42:15.880811+0000", 2024-09-11T03:43:16.335 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:43:16.335 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.335 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:43:16.335 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [], 2024-09-11T03:43:16.335 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [ 2024-09-11T03:43:16.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.335 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "1.a", 2024-09-11T03:43:16.335 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-11T03:43:16.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.336 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 4, 2024-09-11T03:43:16.336 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 9 2024-09-11T03:43:16.336 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.336 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.336 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.2", 2024-09-11T03:43:16.336 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-11T03:43:16.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.336 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 9, 2024-09-11T03:43:16.336 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 11 2024-09-11T03:43:16.337 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.337 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.337 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.d", 2024-09-11T03:43:16.337 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-11T03:43:16.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.337 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 9, 2024-09-11T03:43:16.337 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-11T03:43:16.337 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.337 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.338 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.338 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:43:16.338 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-09-11T03:43:16.338 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-09-11T03:43:16.338 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-09-11T03:43:16.338 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-09-11T03:43:16.338 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/108136515": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:43:16.338 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3411515187": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:43:16.338 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6801/3365920551": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:43:16.338 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/3365920551": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:43:16.338 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3885577441": "2024-09-12T03:31:03.077186+0000", 2024-09-11T03:43:16.338 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/22039254": "2024-09-12T03:31:03.077186+0000", 2024-09-11T03:43:16.339 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6801/3496645948": "2024-09-12T03:31:03.077186+0000", 2024-09-11T03:43:16.339 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4206337830": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:43:16.339 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/3496645948": "2024-09-12T03:31:03.077186+0000", 2024-09-11T03:43:16.339 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1989775875": "2024-09-12T03:31:03.077186+0000" 2024-09-11T03:43:16.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.339 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-09-11T03:43:16.339 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-09-11T03:43:16.339 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-09-11T03:43:16.339 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-09-11T03:43:16.339 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-09-11T03:43:16.339 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-09-11T03:43:16.340 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-09-11T03:43:16.340 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-09-11T03:43:16.340 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.340 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [ 2024-09-11T03:43:16.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.340 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-11T03:43:16.340 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-11T03:43:16.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.340 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 132, 2024-09-11T03:43:16.340 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 3 2024-09-11T03:43:16.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:16.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.341 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 137, 2024-09-11T03:43:16.341 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-11T03:43:16.341 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.341 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.341 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.341 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:43:16.341 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [ 2024-09-11T03:43:16.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.341 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-11T03:43:16.341 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-11T03:43:16.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:16.342 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 137, 2024-09-11T03:43:16.342 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-11T03:43:16.342 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.342 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:16.342 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.342 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:43:16.342 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-09-11T03:43:16.342 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-09-11T03:43:16.342 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-09-11T03:43:16.342 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-09-11T03:43:16.342 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-09-11T03:43:16.343 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-09-11T03:43:16.343 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-09-11T03:43:16.343 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-09-11T03:43:16.343 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-09-11T03:43:16.343 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:16.343 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:43:16.352 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:16.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:16.356 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:16.407 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:16 smithi084 ceph-mon[27254]: pgmap v720: 57 pgs: 57 active+clean; 135 MiB data, 670 MiB used, 893 GiB / 894 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 22 op/s 2024-09-11T03:43:16.407 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:16 smithi084 ceph-mon[27254]: osdmap e378: 12 total, 12 up, 10 in 2024-09-11T03:43:16.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:16 smithi121 ceph-mon[30729]: pgmap v720: 57 pgs: 57 active+clean; 135 MiB data, 670 MiB used, 893 GiB / 894 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 22 op/s 2024-09-11T03:43:16.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:16 smithi121 ceph-mon[30729]: osdmap e378: 12 total, 12 up, 10 in 2024-09-11T03:43:16.432 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:16 smithi175 ceph-mon[30894]: pgmap v720: 57 pgs: 57 active+clean; 135 MiB data, 670 MiB used, 893 GiB / 894 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 22 op/s 2024-09-11T03:43:16.432 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:16 smithi175 ceph-mon[30894]: osdmap e378: 12 total, 12 up, 10 in 2024-09-11T03:43:16.457 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:16.483 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:16.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:16.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:16.531 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:43:16.531 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:43:16.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:16.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:16.607 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:16.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:16.708 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:16.730 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:16.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:16.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:16.833 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:16.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:16.855 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:16.855 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:16.902 INFO:tasks.rados.rados.2.smithi175.stdout:1367: done (5 left) 2024-09-11T03:43:16.902 INFO:tasks.rados.rados.2.smithi175.stdout:1369: done (4 left) 2024-09-11T03:43:16.902 INFO:tasks.rados.rados.2.smithi175.stdout:1373: done (3 left) 2024-09-11T03:43:16.902 INFO:tasks.rados.rados.2.smithi175.stdout:1376: done (2 left) 2024-09-11T03:43:16.902 INFO:tasks.rados.rados.2.smithi175.stdout:1377: done (1 left) 2024-09-11T03:43:16.902 INFO:tasks.rados.rados.2.smithi175.stdout:1380: done (0 left) 2024-09-11T03:43:16.902 INFO:tasks.rados.rados.2.smithi175.stdout:1381: copy_from oid 2 from oid 16 current snap is 142 2024-09-11T03:43:16.902 INFO:tasks.rados.rados.2.smithi175.stdout:1382: copy_from oid 24 from oid 7 current snap is 142 2024-09-11T03:43:16.903 INFO:tasks.rados.rados.2.smithi175.stdout:1383: snap_create 2024-09-11T03:43:16.906 INFO:tasks.rados.rados.2.smithi175.stdout:1381: finishing copy_from to smithi17557870-2 2024-09-11T03:43:16.906 INFO:tasks.rados.rados.2.smithi175.stdout:1381: got expected ENOENT (src dne) 2024-09-11T03:43:16.906 INFO:tasks.rados.rados.2.smithi175.stdout:1381: finishing copy_from racing read to smithi17557870-2 2024-09-11T03:43:16.917 INFO:tasks.rados.rados.2.smithi175.stdout:1382: finishing copy_from racing read to smithi17557870-24 2024-09-11T03:43:16.920 INFO:tasks.rados.rados.2.smithi175.stdout:1382: finishing copy_from to smithi17557870-24 2024-09-11T03:43:16.920 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 356 (ObjNum 413 snap 127 seq_num 413) dirty exists 2024-09-11T03:43:16.956 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:16.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:16.978 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:16.978 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:17.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:17.105 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:17.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:17.113 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-11T03:43:17.207 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:17.231 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:17.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:17.263 INFO:tasks.thrashosds.thrasher:j is {'epoch': 378, 'fsid': '10fe7946-6fee-11ef-bcea-c7b262605968', 'created': '2024-09-11T03:30:30.532652+0000', 'modified': '2024-09-11T03:43:15.899638+0000', 'last_up_change': '2024-09-11T03:37:26.787638+0000', 'last_in_change': '2024-09-11T03:40:04.952764+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T03:33:42.916933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '222', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '220', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.940000057220459, 'score_stable': 1.940000057220459, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.6100000143051147, 'raw_score_stable': 1.6100000143051147, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-11T03:38:28.163534+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '378', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '276', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 142, 'snap_epoch': 378, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8600000143051147, 'score_stable': 1.8600000143051147, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.5399999618530273, 'raw_score_stable': 1.5399999618530273, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '9a431e48-e46b-44f9-bfa7-cb56c1484e1c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 280, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 1376667411}, {'type': 'v1', 'addr': '172.21.15.84:6803', 'nonce': 1376667411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 1376667411}, {'type': 'v1', 'addr': '172.21.15.84:6805', 'nonce': 1376667411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 1376667411}, {'type': 'v1', 'addr': '172.21.15.84:6809', 'nonce': 1376667411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 1376667411}, {'type': 'v1', 'addr': '172.21.15.84:6807', 'nonce': 1376667411}]}, 'public_addr': '172.21.15.84:6803/1376667411', 'cluster_addr': '172.21.15.84:6805/1376667411', 'heartbeat_back_addr': '172.21.15.84:6809/1376667411', 'heartbeat_front_addr': '172.21.15.84:6807/1376667411', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '62fd6e25-3508-40b4-a0f4-f7de137e3bb5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 280, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3138939249}, {'type': 'v1', 'addr': '172.21.15.84:6811', 'nonce': 3138939249}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 3138939249}, {'type': 'v1', 'addr': '172.21.15.84:6813', 'nonce': 3138939249}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6816', 'nonce': 3138939249}, {'type': 'v1', 'addr': '172.21.15.84:6817', 'nonce': 3138939249}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 3138939249}, {'type': 'v1', 'addr': '172.21.15.84:6815', 'nonce': 3138939249}]}, 'public_addr': '172.21.15.84:6811/3138939249', 'cluster_addr': '172.21.15.84:6813/3138939249', 'heartbeat_back_addr': '172.21.15.84:6817/3138939249', 'heartbeat_front_addr': '172.21.15.84:6815/3138939249', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ff351425-4715-49dc-be49-f27bf205d57e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 281, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6818', 'nonce': 3001811591}, {'type': 'v1', 'addr': '172.21.15.84:6819', 'nonce': 3001811591}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6820', 'nonce': 3001811591}, {'type': 'v1', 'addr': '172.21.15.84:6821', 'nonce': 3001811591}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6824', 'nonce': 3001811591}, {'type': 'v1', 'addr': '172.21.15.84:6825', 'nonce': 3001811591}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6822', 'nonce': 3001811591}, {'type': 'v1', 'addr': '172.21.15.84:6823', 'nonce': 3001811591}]}, 'public_addr': '172.21.15.84:6819/3001811591', 'cluster_addr': '172.21.15.84:6821/3001811591', 'heartbeat_back_addr': '172.21.15.84:6825/3001811591', 'heartbeat_front_addr': '172.21.15.84:6823/3001811591', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2518a93e-c612-4731-91ce-5685c25ffa9f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 332, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6826', 'nonce': 302512129}, {'type': 'v1', 'addr': '172.21.15.84:6827', 'nonce': 302512129}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6828', 'nonce': 302512129}, {'type': 'v1', 'addr': '172.21.15.84:6829', 'nonce': 302512129}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6832', 'nonce': 302512129}, {'type': 'v1', 'addr': '172.21.15.84:6833', 'nonce': 302512129}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6830', 'nonce': 302512129}, {'type': 'v1', 'addr': '172.21.15.84:6831', 'nonce': 302512129}]}, 'public_addr': '172.21.15.84:6827/302512129', 'cluster_addr': '172.21.15.84:6829/302512129', 'heartbeat_back_addr': '172.21.15.84:6833/302512129', 'heartbeat_front_addr': '172.21.15.84:6831/302512129', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd12f3c2d-95cc-4276-a7e1-bee982d58948', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 281, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6800', 'nonce': 3873214513}, {'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3873214513}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6802', 'nonce': 3873214513}, {'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3873214513}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6806', 'nonce': 3873214513}, {'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 3873214513}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6804', 'nonce': 3873214513}, {'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 3873214513}]}, 'public_addr': '172.21.15.121:6801/3873214513', 'cluster_addr': '172.21.15.121:6803/3873214513', 'heartbeat_back_addr': '172.21.15.121:6807/3873214513', 'heartbeat_front_addr': '172.21.15.121:6805/3873214513', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '32164dd5-7e8f-4925-8526-bbb1e33e3364', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 333, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6808', 'nonce': 918612194}, {'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 918612194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6810', 'nonce': 918612194}, {'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 918612194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6814', 'nonce': 918612194}, {'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 918612194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6812', 'nonce': 918612194}, {'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 918612194}]}, 'public_addr': '172.21.15.121:6809/918612194', 'cluster_addr': '172.21.15.121:6811/918612194', 'heartbeat_back_addr': '172.21.15.121:6815/918612194', 'heartbeat_front_addr': '172.21.15.121:6813/918612194', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '3f107a88-803f-4cd8-804a-241310c282ea', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6816', 'nonce': 3384813363}, {'type': 'v1', 'addr': '172.21.15.121:6817', 'nonce': 3384813363}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6818', 'nonce': 3384813363}, {'type': 'v1', 'addr': '172.21.15.121:6819', 'nonce': 3384813363}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6822', 'nonce': 3384813363}, {'type': 'v1', 'addr': '172.21.15.121:6823', 'nonce': 3384813363}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6820', 'nonce': 3384813363}, {'type': 'v1', 'addr': '172.21.15.121:6821', 'nonce': 3384813363}]}, 'public_addr': '172.21.15.121:6817/3384813363', 'cluster_addr': '172.21.15.121:6819/3384813363', 'heartbeat_back_addr': '172.21.15.121:6823/3384813363', 'heartbeat_front_addr': '172.21.15.121:6821/3384813363', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dfee7f95-1fb9-4493-9907-cb10d1837bdb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 309, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6824', 'nonce': 449222054}, {'type': 'v1', 'addr': '172.21.15.121:6825', 'nonce': 449222054}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6826', 'nonce': 449222054}, {'type': 'v1', 'addr': '172.21.15.121:6827', 'nonce': 449222054}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6830', 'nonce': 449222054}, {'type': 'v1', 'addr': '172.21.15.121:6831', 'nonce': 449222054}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6828', 'nonce': 449222054}, {'type': 'v1', 'addr': '172.21.15.121:6829', 'nonce': 449222054}]}, 'public_addr': '172.21.15.121:6825/449222054', 'cluster_addr': '172.21.15.121:6827/449222054', 'heartbeat_back_addr': '172.21.15.121:6831/449222054', 'heartbeat_front_addr': '172.21.15.121:6829/449222054', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 278, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6800', 'nonce': 2660128384}, {'type': 'v1', 'addr': '172.21.15.175:6801', 'nonce': 2660128384}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6802', 'nonce': 2660128384}, {'type': 'v1', 'addr': '172.21.15.175:6803', 'nonce': 2660128384}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6806', 'nonce': 2660128384}, {'type': 'v1', 'addr': '172.21.15.175:6807', 'nonce': 2660128384}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6804', 'nonce': 2660128384}, {'type': 'v1', 'addr': '172.21.15.175:6805', 'nonce': 2660128384}]}, 'public_addr': '172.21.15.175:6801/2660128384', 'cluster_addr': '172.21.15.175:6803/2660128384', 'heartbeat_back_addr': '172.21.15.175:6807/2660128384', 'heartbeat_front_addr': '172.21.15.175:6805/2660128384', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '50162e42-0443-41d1-8709-a0db61c087da', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 312, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6808', 'nonce': 1270991649}, {'type': 'v1', 'addr': '172.21.15.175:6809', 'nonce': 1270991649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6810', 'nonce': 1270991649}, {'type': 'v1', 'addr': '172.21.15.175:6811', 'nonce': 1270991649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6814', 'nonce': 1270991649}, {'type': 'v1', 'addr': '172.21.15.175:6815', 'nonce': 1270991649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6812', 'nonce': 1270991649}, {'type': 'v1', 'addr': '172.21.15.175:6813', 'nonce': 1270991649}]}, 'public_addr': '172.21.15.175:6809/1270991649', 'cluster_addr': '172.21.15.175:6811/1270991649', 'heartbeat_back_addr': '172.21.15.175:6815/1270991649', 'heartbeat_front_addr': '172.21.15.175:6813/1270991649', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '4b9426e1-bac8-47d3-ad58-54b89e8d1eb1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 182, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6816', 'nonce': 3877631021}, {'type': 'v1', 'addr': '172.21.15.175:6817', 'nonce': 3877631021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6818', 'nonce': 3877631021}, {'type': 'v1', 'addr': '172.21.15.175:6819', 'nonce': 3877631021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6822', 'nonce': 3877631021}, {'type': 'v1', 'addr': '172.21.15.175:6823', 'nonce': 3877631021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6820', 'nonce': 3877631021}, {'type': 'v1', 'addr': '172.21.15.175:6821', 'nonce': 3877631021}]}, 'public_addr': '172.21.15.175:6817/3877631021', 'cluster_addr': '172.21.15.175:6819/3877631021', 'heartbeat_back_addr': '172.21.15.175:6823/3877631021', 'heartbeat_front_addr': '172.21.15.175:6821/3877631021', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'cd1a28a1-413c-4274-9876-e1bdb7703be6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6824', 'nonce': 1150051236}, {'type': 'v1', 'addr': '172.21.15.175:6825', 'nonce': 1150051236}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6826', 'nonce': 1150051236}, {'type': 'v1', 'addr': '172.21.15.175:6827', 'nonce': 1150051236}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6830', 'nonce': 1150051236}, {'type': 'v1', 'addr': '172.21.15.175:6831', 'nonce': 1150051236}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6828', 'nonce': 1150051236}, {'type': 'v1', 'addr': '172.21.15.175:6829', 'nonce': 1150051236}]}, 'public_addr': '172.21.15.175:6825/1150051236', 'cluster_addr': '172.21.15.175:6827/1150051236', 'heartbeat_back_addr': '172.21.15.175:6831/1150051236', 'heartbeat_front_addr': '172.21.15.175:6829/1150051236', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:42:19.799941+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:43:14.014218+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:42:02.898695+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:42:41.819237+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:42:08.783166+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:42:25.432246+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-11T03:42:13.715916+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:42:46.980292+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:43:00.798033+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:42:42.240655+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-11T03:43:11.996326+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:42:15.880811+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '1.a', 'mappings': [{'from': 4, 'to': 9}]}, {'pgid': '2.2', 'mappings': [{'from': 9, 'to': 11}]}, {'pgid': '2.d', 'mappings': [{'from': 9, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/108136515': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:0/3411515187': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:6801/3365920551': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:6800/3365920551': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:0/3885577441': '2024-09-12T03:31:03.077186+0000', '172.21.15.84:0/22039254': '2024-09-12T03:31:03.077186+0000', '172.21.15.84:6801/3496645948': '2024-09-12T03:31:03.077186+0000', '172.21.15.84:0/4206337830': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:6800/3496645948': '2024-09-12T03:31:03.077186+0000', '172.21.15.84:0/1989775875': '2024-09-12T03:31:03.077186+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 132, 'length': 3}, {'begin': 137, 'length': 1}]}], 'new_removed_snaps': [{'pool': 2, 'snaps': [{'begin': 137, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T03:43:17.263 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 1.a 2024-09-11T03:43:17.264 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd rm-pg-upmap-items 1.a 2024-09-11T03:43:17.333 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:17.353 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:43:17.359 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:17.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:17.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1322173222' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-11T03:43:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:17 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:17 smithi121 ceph-mon[30729]: osdmap e379: 12 total, 12 up, 10 in 2024-09-11T03:43:17.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1322173222' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-11T03:43:17.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:17 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:17.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:17 smithi084 ceph-mon[27254]: osdmap e379: 12 total, 12 up, 10 in 2024-09-11T03:43:17.461 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:17.474 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1322173222' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-11T03:43:17.474 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:17 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:17.474 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:17 smithi175 ceph-mon[30894]: osdmap e379: 12 total, 12 up, 10 in 2024-09-11T03:43:17.484 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:17.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:17.527 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:43:17.527 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:43:17.545 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:43:17.545 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:43:17.562 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:43:17.562 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:43:17.579 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:43:17.579 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:43:17.581 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:17.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:17.597 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:43:17.597 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:43:17.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:17.612 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:17.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:17.615 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:43:17.615 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:43:17.632 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:43:17.633 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:43:17.649 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:43:17.650 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:43:17.677 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:43:17.677 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:43:17.693 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:43:17.694 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:43:17.710 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:43:17.710 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:43:17.713 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:17.728 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:43:17.728 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:43:17.734 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:17.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:17.735 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:17.823 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:43:17.823 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:43:17.823 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:43:17.824 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:43:17.824 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.824 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:366 2.7 e114/95 2:e0eabcfb:::smithi17557870-9:head v 114'24, mlcod=112'22)", 2024-09-11T03:43:17.824 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.856154+0000", 2024-09-11T03:43:17.824 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 266.96633666000002, 2024-09-11T03:43:17.824 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025307979999999999, 2024-09-11T03:43:17.824 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.824 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.824 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:17.824 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.824 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.824 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.825 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.856154+0000", 2024-09-11T03:43:17.825 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.825 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.825 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.825 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.825 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.856154+0000", 2024-09-11T03:43:17.825 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.825 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.825 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.825 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.825 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.856154+0000", 2024-09-11T03:43:17.825 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1e-07 2024-09-11T03:43:17.825 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.826 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.826 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.826 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.857067+0000", 2024-09-11T03:43:17.826 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00091319100000000005 2024-09-11T03:43:17.826 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.826 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.826 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.826 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.857088+0000", 2024-09-11T03:43:17.826 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0264000000000001e-05 2024-09-11T03:43:17.826 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.826 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.826 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.826 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.857101+0000", 2024-09-11T03:43:17.826 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3645000000000001e-05 2024-09-11T03:43:17.827 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.827 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.827 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.827 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.857189+0000", 2024-09-11T03:43:17.827 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8090000000000005e-05 2024-09-11T03:43:17.827 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.827 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.827 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.827 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.857219+0000", 2024-09-11T03:43:17.827 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9893999999999999e-05 2024-09-11T03:43:17.827 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.827 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.828 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:17.828 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.858638+0000", 2024-09-11T03:43:17.828 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001419182 2024-09-11T03:43:17.828 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.828 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.828 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.828 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.858685+0000", 2024-09-11T03:43:17.828 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6322000000000001e-05 2024-09-11T03:43:17.828 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.828 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.828 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.828 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.829 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.829 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:386 2.0 e114/95 2:0762776a:::smithi17557870-25:head v 114'11, mlcod=113'8)", 2024-09-11T03:43:17.829 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.988307+0000", 2024-09-11T03:43:17.829 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 266.834183709, 2024-09-11T03:43:17.829 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025481689999999999, 2024-09-11T03:43:17.829 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.829 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.829 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:17.829 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.829 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.829 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.829 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.988307+0000", 2024-09-11T03:43:17.829 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.830 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.830 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.830 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.830 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.988307+0000", 2024-09-11T03:43:17.830 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.830 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.830 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.830 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.830 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.988309+0000", 2024-09-11T03:43:17.830 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9089999999999998e-06 2024-09-11T03:43:17.830 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.830 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.830 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.831 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.989577+0000", 2024-09-11T03:43:17.831 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012685940000000001 2024-09-11T03:43:17.831 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.831 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.831 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.831 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.989603+0000", 2024-09-11T03:43:17.831 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5398999999999999e-05 2024-09-11T03:43:17.831 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.831 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.831 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.831 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.989613+0000", 2024-09-11T03:43:17.831 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0519e-05 2024-09-11T03:43:17.831 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.832 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.832 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.832 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.989691+0000", 2024-09-11T03:43:17.832 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8112e-05 2024-09-11T03:43:17.832 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.832 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.832 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.832 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.989724+0000", 2024-09-11T03:43:17.832 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.2438999999999999e-05 2024-09-11T03:43:17.832 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.832 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.832 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:17.832 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.990816+0000", 2024-09-11T03:43:17.833 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001091854 2024-09-11T03:43:17.833 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.833 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.833 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.833 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.990855+0000", 2024-09-11T03:43:17.833 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9342999999999999e-05 2024-09-11T03:43:17.833 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.833 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.833 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.833 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.833 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.833 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:387 2.0 e114/95 2:0762776a:::smithi17557870-25:head v 114'12, mlcod=114'10)", 2024-09-11T03:43:17.833 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.995117+0000", 2024-09-11T03:43:17.834 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 266.82737386899998, 2024-09-11T03:43:17.834 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023580089999999999, 2024-09-11T03:43:17.834 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.834 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.834 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:17.834 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.834 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.834 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.834 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.995117+0000", 2024-09-11T03:43:17.834 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.834 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.834 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.834 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.834 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.995117+0000", 2024-09-11T03:43:17.835 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.835 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.835 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.835 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.835 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.995118+0000", 2024-09-11T03:43:17.835 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4900000000000005e-07 2024-09-11T03:43:17.835 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.835 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.835 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.835 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.996445+0000", 2024-09-11T03:43:17.835 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001327607 2024-09-11T03:43:17.835 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.836 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.836 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.836 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.996460+0000", 2024-09-11T03:43:17.836 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4572e-05 2024-09-11T03:43:17.836 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.836 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.836 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.836 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.996467+0000", 2024-09-11T03:43:17.836 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2760000000000003e-06 2024-09-11T03:43:17.836 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.836 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.836 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.837 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.996529+0000", 2024-09-11T03:43:17.837 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1883000000000004e-05 2024-09-11T03:43:17.837 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.837 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.837 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.837 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.996552+0000", 2024-09-11T03:43:17.837 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3258e-05 2024-09-11T03:43:17.837 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.837 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.837 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:17.837 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.997444+0000", 2024-09-11T03:43:17.837 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00089145800000000003 2024-09-11T03:43:17.837 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.838 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.838 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.838 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.997475+0000", 2024-09-11T03:43:17.838 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1205999999999999e-05 2024-09-11T03:43:17.838 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.838 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.838 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.838 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.838 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.838 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:402 2.7 e114/95 2:e23d1762:::smithi17557870-21:head v 114'28, mlcod=114'26)", 2024-09-11T03:43:17.838 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:51.091054+0000", 2024-09-11T03:43:17.838 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 266.73143615800001, 2024-09-11T03:43:17.838 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0062707429999999996, 2024-09-11T03:43:17.838 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.839 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.839 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:17.840 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:17.840 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.840 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.841 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.841 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.091054+0000", 2024-09-11T03:43:17.841 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.841 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.841 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.841 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.841 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.091054+0000", 2024-09-11T03:43:17.841 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.841 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.841 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.841 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.841 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.091056+0000", 2024-09-11T03:43:17.842 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.128e-06 2024-09-11T03:43:17.842 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.842 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.842 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.842 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.095985+0000", 2024-09-11T03:43:17.842 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0049292549999999996 2024-09-11T03:43:17.842 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.842 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.842 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.842 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.096004+0000", 2024-09-11T03:43:17.842 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8859e-05 2024-09-11T03:43:17.842 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.843 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.843 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.843 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.096013+0000", 2024-09-11T03:43:17.843 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6169999999999996e-06 2024-09-11T03:43:17.843 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.843 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.843 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.843 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.096101+0000", 2024-09-11T03:43:17.843 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7331999999999997e-05 2024-09-11T03:43:17.843 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.843 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.843 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.843 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.096120+0000", 2024-09-11T03:43:17.843 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9783999999999999e-05 2024-09-11T03:43:17.844 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.844 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.844 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:17.844 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.097285+0000", 2024-09-11T03:43:17.844 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001164237 2024-09-11T03:43:17.844 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.844 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.844 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.844 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:51.097325+0000", 2024-09-11T03:43:17.844 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0531000000000002e-05 2024-09-11T03:43:17.844 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.844 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.844 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.844 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.844 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.845 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:851 2.7 2:e6da0655:::smithi17557870-27:head [write 736173~553965,stat] snapc 1a=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-11T03:43:17.845 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:26.831961+0000", 2024-09-11T03:43:17.845 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 230.990529377, 2024-09-11T03:43:17.845 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0048828370000000001, 2024-09-11T03:43:17.845 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.845 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.845 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:17.845 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:17.845 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:17.845 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:17.845 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 851 2024-09-11T03:43:17.845 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.846 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.846 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.846 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.846 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.831961+0000", 2024-09-11T03:43:17.846 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.846 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.846 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.846 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.846 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.831961+0000", 2024-09-11T03:43:17.846 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.846 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.846 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.847 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.847 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.831963+0000", 2024-09-11T03:43:17.847 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.86e-06 2024-09-11T03:43:17.847 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.847 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.847 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.847 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.832372+0000", 2024-09-11T03:43:17.847 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00040875299999999998 2024-09-11T03:43:17.847 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.847 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.847 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.847 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.832385+0000", 2024-09-11T03:43:17.848 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3244999999999999e-05 2024-09-11T03:43:17.848 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.848 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.848 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.848 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.832397+0000", 2024-09-11T03:43:17.848 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2342e-05 2024-09-11T03:43:17.848 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.848 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.848 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.848 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.832498+0000", 2024-09-11T03:43:17.849 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000100732 2024-09-11T03:43:17.849 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.849 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.849 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:43:17.849 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.832507+0000", 2024-09-11T03:43:17.849 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.9309999999999997e-06 2024-09-11T03:43:17.849 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.849 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.849 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.849 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.833851+0000", 2024-09-11T03:43:17.849 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001343894 2024-09-11T03:43:17.849 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.850 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.850 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.850 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.833959+0000", 2024-09-11T03:43:17.850 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010828 2024-09-11T03:43:17.850 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.850 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.850 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:43:17.850 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.834088+0000", 2024-09-11T03:43:17.850 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000128639 2024-09-11T03:43:17.850 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.850 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.850 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:17.851 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.835317+0000", 2024-09-11T03:43:17.851 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012290000000000001 2024-09-11T03:43:17.851 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.851 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.851 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:17.851 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.836679+0000", 2024-09-11T03:43:17.851 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013623590000000001 2024-09-11T03:43:17.851 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.851 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.851 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:17.851 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.836779+0000", 2024-09-11T03:43:17.851 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9427999999999997e-05 2024-09-11T03:43:17.851 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.852 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.852 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:17.852 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.836815+0000", 2024-09-11T03:43:17.852 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6057999999999999e-05 2024-09-11T03:43:17.852 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.852 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.852 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.852 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.836844+0000", 2024-09-11T03:43:17.852 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9315999999999998e-05 2024-09-11T03:43:17.852 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.852 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.852 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.852 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.853 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.853 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:852 2.7 2:e6da0655:::smithi17557870-27:head [write 1867350~468823] snapc 1a=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-11T03:43:17.853 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:26.837175+0000", 2024-09-11T03:43:17.853 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 230.985316017, 2024-09-11T03:43:17.853 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003236287, 2024-09-11T03:43:17.853 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.853 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.853 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:17.853 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:17.853 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:17.853 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:17.853 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 852 2024-09-11T03:43:17.854 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.854 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.854 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.854 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.854 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837175+0000", 2024-09-11T03:43:17.854 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.854 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.854 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.854 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.854 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837175+0000", 2024-09-11T03:43:17.855 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.855 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.855 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.855 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.855 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837176+0000", 2024-09-11T03:43:17.855 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2669999999999999e-06 2024-09-11T03:43:17.855 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.855 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.855 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.855 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837474+0000", 2024-09-11T03:43:17.855 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00029848599999999999 2024-09-11T03:43:17.855 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.855 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.856 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.856 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837486+0000", 2024-09-11T03:43:17.856 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2009e-05 2024-09-11T03:43:17.856 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.856 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.856 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.856 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837492+0000", 2024-09-11T03:43:17.856 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8599999999999998e-06 2024-09-11T03:43:17.856 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.856 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.856 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.856 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837545+0000", 2024-09-11T03:43:17.857 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3139000000000001e-05 2024-09-11T03:43:17.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.857 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.857 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.857 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837643+0000", 2024-09-11T03:43:17.857 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.789e-05 2024-09-11T03:43:17.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.857 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.857 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:43:17.857 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837812+0000", 2024-09-11T03:43:17.857 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016848199999999999 2024-09-11T03:43:17.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.858 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.858 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:17.858 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.839259+0000", 2024-09-11T03:43:17.858 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001447453 2024-09-11T03:43:17.858 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.858 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.858 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:17.858 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840221+0000", 2024-09-11T03:43:17.858 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00096196400000000005 2024-09-11T03:43:17.858 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.858 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.858 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:17.858 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840328+0000", 2024-09-11T03:43:17.859 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000106511 2024-09-11T03:43:17.859 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.859 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.859 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:17.859 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840367+0000", 2024-09-11T03:43:17.859 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9501e-05 2024-09-11T03:43:17.859 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.859 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.859 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.859 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840411+0000", 2024-09-11T03:43:17.859 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3724999999999999e-05 2024-09-11T03:43:17.859 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.860 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.860 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.860 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.860 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.860 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:853 2.7 2:e6da0655:::smithi17557870-27:head [setxattr _header (57) in=7b,truncate 2336173] snapc 1a=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-11T03:43:17.860 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:26.837507+0000", 2024-09-11T03:43:17.860 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 230.98498351999999, 2024-09-11T03:43:17.860 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030766439999999999, 2024-09-11T03:43:17.860 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.860 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.860 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:17.860 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:17.860 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:17.861 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:17.861 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 853 2024-09-11T03:43:17.861 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.861 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.861 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.861 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.861 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837507+0000", 2024-09-11T03:43:17.861 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.861 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.861 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.861 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.861 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837507+0000", 2024-09-11T03:43:17.861 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.862 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.862 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.862 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.862 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837508+0000", 2024-09-11T03:43:17.862 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7299999999999995e-07 2024-09-11T03:43:17.862 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.862 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.862 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.862 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837514+0000", 2024-09-11T03:43:17.862 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7450000000000001e-06 2024-09-11T03:43:17.862 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.862 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.862 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.863 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837520+0000", 2024-09-11T03:43:17.863 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7490000000000001e-06 2024-09-11T03:43:17.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.863 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.863 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.863 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837524+0000", 2024-09-11T03:43:17.863 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0319999999999997e-06 2024-09-11T03:43:17.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.863 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.863 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.863 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.838759+0000", 2024-09-11T03:43:17.863 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001234726 2024-09-11T03:43:17.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.864 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.864 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.864 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.838825+0000", 2024-09-11T03:43:17.864 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6124000000000006e-05 2024-09-11T03:43:17.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.864 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.864 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:43:17.864 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.838909+0000", 2024-09-11T03:43:17.864 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4254999999999999e-05 2024-09-11T03:43:17.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.864 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.864 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:17.864 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.839304+0000", 2024-09-11T03:43:17.865 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00039483699999999999 2024-09-11T03:43:17.865 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.865 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.865 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:17.865 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840463+0000", 2024-09-11T03:43:17.865 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001158493 2024-09-11T03:43:17.865 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.865 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.865 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:17.865 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840500+0000", 2024-09-11T03:43:17.865 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7020999999999997e-05 2024-09-11T03:43:17.866 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.866 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.866 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:17.866 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840532+0000", 2024-09-11T03:43:17.866 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1810000000000002e-05 2024-09-11T03:43:17.866 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.866 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.866 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.866 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840584+0000", 2024-09-11T03:43:17.866 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2179000000000003e-05 2024-09-11T03:43:17.866 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.866 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.866 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.867 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.867 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.867 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:854 2.7 2:e6da0655:::smithi17557870-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e150)", 2024-09-11T03:43:17.867 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:26.837532+0000", 2024-09-11T03:43:17.867 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 230.98495846099999, 2024-09-11T03:43:17.867 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032582370000000002, 2024-09-11T03:43:17.867 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.867 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.867 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:17.867 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:17.867 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:17.867 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:17.867 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 854 2024-09-11T03:43:17.867 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.868 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.868 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.868 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.868 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837532+0000", 2024-09-11T03:43:17.868 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.868 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.868 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.868 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.868 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837532+0000", 2024-09-11T03:43:17.868 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.868 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.868 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.869 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.869 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837533+0000", 2024-09-11T03:43:17.869 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0800000000000005e-07 2024-09-11T03:43:17.869 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.869 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.869 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.869 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837535+0000", 2024-09-11T03:43:17.869 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.001e-06 2024-09-11T03:43:17.869 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.869 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.869 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.869 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837543+0000", 2024-09-11T03:43:17.869 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3080000000000008e-06 2024-09-11T03:43:17.870 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.870 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.870 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.870 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.837548+0000", 2024-09-11T03:43:17.870 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6689999999999997e-06 2024-09-11T03:43:17.870 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.870 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.870 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.870 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.839166+0000", 2024-09-11T03:43:17.870 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001618565 2024-09-11T03:43:17.870 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.870 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.870 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:43:17.871 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.839246+0000", 2024-09-11T03:43:17.871 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9297999999999999e-05 2024-09-11T03:43:17.871 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.871 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.871 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840622+0000", 2024-09-11T03:43:17.871 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013767790000000001 2024-09-11T03:43:17.871 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.871 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.872 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840692+0000", 2024-09-11T03:43:17.872 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0020999999999999e-05 2024-09-11T03:43:17.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.872 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.872 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:26.840790+0000", 2024-09-11T03:43:17.872 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8088e-05 2024-09-11T03:43:17.872 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.872 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.872 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.873 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:898 2.7 2:e23d1762:::smithi17557870-21:head [delete] snapc 20=[20,1f,1e,1d,1c] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e157)", 2024-09-11T03:43:17.873 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:33.891697+0000", 2024-09-11T03:43:17.873 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 223.930794123, 2024-09-11T03:43:17.873 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022672870000000002, 2024-09-11T03:43:17.873 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.873 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.873 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:17.873 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:17.873 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:17.873 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:17.873 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 898 2024-09-11T03:43:17.873 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.873 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.874 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.874 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.874 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.891697+0000", 2024-09-11T03:43:17.874 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.874 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.874 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.874 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.874 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.891697+0000", 2024-09-11T03:43:17.874 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.874 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.874 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.874 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.875 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.891698+0000", 2024-09-11T03:43:17.875 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8920000000000001e-06 2024-09-11T03:43:17.875 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.875 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.875 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.875 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.891706+0000", 2024-09-11T03:43:17.875 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4490000000000002e-06 2024-09-11T03:43:17.875 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.875 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.875 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.875 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.891724+0000", 2024-09-11T03:43:17.875 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8219999999999998e-05 2024-09-11T03:43:17.875 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.876 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.876 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.876 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.891732+0000", 2024-09-11T03:43:17.876 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3410000000000003e-06 2024-09-11T03:43:17.876 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.876 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.876 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.876 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.891799+0000", 2024-09-11T03:43:17.876 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6711000000000006e-05 2024-09-11T03:43:17.876 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.876 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.876 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:43:17.876 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.891803+0000", 2024-09-11T03:43:17.877 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3549999999999998e-06 2024-09-11T03:43:17.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.877 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.877 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.877 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.892785+0000", 2024-09-11T03:43:17.877 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00098169699999999991 2024-09-11T03:43:17.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.877 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.878 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.878 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.892906+0000", 2024-09-11T03:43:17.878 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012052700000000001 2024-09-11T03:43:17.878 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.878 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.878 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.893964+0000", 2024-09-11T03:43:17.878 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001058095 2024-09-11T03:43:17.878 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.878 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.878 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.878 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.879 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.879 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:899 2.7 2:e0ec5763:::smithi17557870-2:head [append 0~0] snapc 20=[20,1f,1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-11T03:43:17.879 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:33.894175+0000", 2024-09-11T03:43:17.879 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 223.92831550899999, 2024-09-11T03:43:17.879 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0021720340000000002, 2024-09-11T03:43:17.879 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.879 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.879 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:17.879 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:17.879 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:17.880 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:17.880 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 899 2024-09-11T03:43:17.880 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.880 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.880 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.880 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.880 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894175+0000", 2024-09-11T03:43:17.880 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.880 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.880 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.880 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.880 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894175+0000", 2024-09-11T03:43:17.880 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.881 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.881 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.881 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.881 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894177+0000", 2024-09-11T03:43:17.881 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6029999999999999e-06 2024-09-11T03:43:17.881 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.881 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.881 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.881 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894183+0000", 2024-09-11T03:43:17.882 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5889999999999999e-06 2024-09-11T03:43:17.882 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.882 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.882 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.882 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894196+0000", 2024-09-11T03:43:17.882 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3155e-05 2024-09-11T03:43:17.882 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.882 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.882 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.882 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894205+0000", 2024-09-11T03:43:17.882 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1489999999999994e-06 2024-09-11T03:43:17.883 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.883 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.883 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.883 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894264+0000", 2024-09-11T03:43:17.883 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9771000000000001e-05 2024-09-11T03:43:17.883 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.883 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.883 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.883 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894421+0000", 2024-09-11T03:43:17.883 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015673099999999999 2024-09-11T03:43:17.883 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.883 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.884 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:43:17.884 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894653+0000", 2024-09-11T03:43:17.884 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00023214700000000001 2024-09-11T03:43:17.884 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.884 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.884 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:17.884 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896217+0000", 2024-09-11T03:43:17.884 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015637019999999999 2024-09-11T03:43:17.884 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.884 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.884 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:17.884 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896256+0000", 2024-09-11T03:43:17.884 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9187999999999998e-05 2024-09-11T03:43:17.885 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.885 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.885 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:17.885 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896272+0000", 2024-09-11T03:43:17.885 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6013999999999999e-05 2024-09-11T03:43:17.885 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.885 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.885 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:17.885 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896313+0000", 2024-09-11T03:43:17.886 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0728e-05 2024-09-11T03:43:17.886 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.886 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.886 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.886 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896347+0000", 2024-09-11T03:43:17.886 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4257000000000001e-05 2024-09-11T03:43:17.886 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.886 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.886 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.886 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.886 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.886 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:900 2.7 2:e0ec5763:::smithi17557870-2:head [rollback 20] snapc 20=[20,1f,1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-11T03:43:17.887 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:33.894221+0000", 2024-09-11T03:43:17.887 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 223.928269629, 2024-09-11T03:43:17.887 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022939969999999999, 2024-09-11T03:43:17.887 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.887 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.887 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:17.887 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:17.887 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:17.887 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:17.887 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 900 2024-09-11T03:43:17.887 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.887 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.887 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.887 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.888 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894221+0000", 2024-09-11T03:43:17.888 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.888 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.888 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.888 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.888 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894221+0000", 2024-09-11T03:43:17.888 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.888 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.888 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.888 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.888 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894221+0000", 2024-09-11T03:43:17.889 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.51e-07 2024-09-11T03:43:17.889 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.889 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.889 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.889 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894224+0000", 2024-09-11T03:43:17.889 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4490000000000002e-06 2024-09-11T03:43:17.889 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.889 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.889 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.889 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894230+0000", 2024-09-11T03:43:17.889 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5359999999999998e-06 2024-09-11T03:43:17.889 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.890 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.890 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.890 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894236+0000", 2024-09-11T03:43:17.890 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1889999999999998e-06 2024-09-11T03:43:17.890 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.890 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.890 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.890 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.895141+0000", 2024-09-11T03:43:17.890 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00090550899999999998 2024-09-11T03:43:17.890 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.890 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.890 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.890 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.895196+0000", 2024-09-11T03:43:17.890 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4657999999999997e-05 2024-09-11T03:43:17.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.891 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:43:17.891 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.895310+0000", 2024-09-11T03:43:17.891 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114274 2024-09-11T03:43:17.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.891 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:17.891 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896394+0000", 2024-09-11T03:43:17.891 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001083908 2024-09-11T03:43:17.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.891 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:17.892 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896445+0000", 2024-09-11T03:43:17.892 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0793e-05 2024-09-11T03:43:17.892 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.892 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.892 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:17.892 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896457+0000", 2024-09-11T03:43:17.892 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2061e-05 2024-09-11T03:43:17.892 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.892 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.892 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:17.892 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896491+0000", 2024-09-11T03:43:17.892 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4208000000000002e-05 2024-09-11T03:43:17.892 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.893 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.893 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896515+0000", 2024-09-11T03:43:17.893 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3961e-05 2024-09-11T03:43:17.893 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.893 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.893 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.893 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.893 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:901 2.7 2:e0ec5763:::smithi17557870-2:head [append 0~0] snapc 20=[20,1f,1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-11T03:43:17.893 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:33.894258+0000", 2024-09-11T03:43:17.893 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 223.92823259299999, 2024-09-11T03:43:17.893 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002369674, 2024-09-11T03:43:17.894 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.894 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.894 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:17.894 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:17.894 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:17.894 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:17.894 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 901 2024-09-11T03:43:17.894 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.894 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.894 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.894 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.894 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894258+0000", 2024-09-11T03:43:17.894 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.895 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.895 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.895 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.895 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894258+0000", 2024-09-11T03:43:17.895 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.895 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.895 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.895 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.895 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894258+0000", 2024-09-11T03:43:17.895 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9000000000000002e-07 2024-09-11T03:43:17.895 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.895 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.895 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.896 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894262+0000", 2024-09-11T03:43:17.896 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0690000000000002e-06 2024-09-11T03:43:17.896 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.896 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.896 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.896 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894269+0000", 2024-09-11T03:43:17.896 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2910000000000002e-06 2024-09-11T03:43:17.896 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.897 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.897 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.897 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.894283+0000", 2024-09-11T03:43:17.897 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4022999999999999e-05 2024-09-11T03:43:17.897 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.897 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.897 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.897 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.895708+0000", 2024-09-11T03:43:17.897 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014255209999999999 2024-09-11T03:43:17.897 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.897 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.897 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.898 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.895812+0000", 2024-09-11T03:43:17.898 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000103643 2024-09-11T03:43:17.898 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.898 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.898 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:43:17.898 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.895966+0000", 2024-09-11T03:43:17.898 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000154067 2024-09-11T03:43:17.898 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.898 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.898 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:17.898 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896523+0000", 2024-09-11T03:43:17.898 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00055707700000000001 2024-09-11T03:43:17.898 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.899 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.899 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:17.899 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896563+0000", 2024-09-11T03:43:17.899 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9473e-05 2024-09-11T03:43:17.899 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.899 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.899 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:17.899 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896586+0000", 2024-09-11T03:43:17.899 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2900000000000001e-05 2024-09-11T03:43:17.900 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.900 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.900 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:17.900 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896610+0000", 2024-09-11T03:43:17.900 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4411000000000001e-05 2024-09-11T03:43:17.900 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.900 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.900 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.900 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:33.896628+0000", 2024-09-11T03:43:17.900 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7808999999999998e-05 2024-09-11T03:43:17.901 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.901 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.901 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.901 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.901 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.901 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(osd.7.0:69 2.0 e172/95 2:0762776a:::smithi17557870-25:26 v 172'20, mlcod=170'16)", 2024-09-11T03:43:17.901 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:48.860013+0000", 2024-09-11T03:43:17.901 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 208.962477662, 2024-09-11T03:43:17.901 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0028445179999999999, 2024-09-11T03:43:17.902 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.902 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.902 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:17.902 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.902 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.902 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.902 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:48.860013+0000", 2024-09-11T03:43:17.902 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.902 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.902 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.902 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.902 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:48.860013+0000", 2024-09-11T03:43:17.902 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.903 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.903 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:48.860014+0000", 2024-09-11T03:43:17.903 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1399999999999996e-07 2024-09-11T03:43:17.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.903 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.903 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:48.860021+0000", 2024-09-11T03:43:17.903 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9399999999999996e-06 2024-09-11T03:43:17.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.904 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.904 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:48.860040+0000", 2024-09-11T03:43:17.904 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8867000000000001e-05 2024-09-11T03:43:17.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.904 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.904 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:48.860048+0000", 2024-09-11T03:43:17.905 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0309999999999998e-06 2024-09-11T03:43:17.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.905 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:43:17.906 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.907 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:48.860145+0000", 2024-09-11T03:43:17.907 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7789999999999997e-05 2024-09-11T03:43:17.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.907 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.907 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:48.860169+0000", 2024-09-11T03:43:17.907 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4015e-05 2024-09-11T03:43:17.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.907 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:17.907 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:48.862796+0000", 2024-09-11T03:43:17.907 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0026262640000000001 2024-09-11T03:43:17.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.908 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.908 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:48.862858+0000", 2024-09-11T03:43:17.908 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1896999999999994e-05 2024-09-11T03:43:17.908 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.908 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.908 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.908 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(osd.1.13:10 2.7 2:e0ec5763:::smithi17557870-2:head [assert-version v73,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e174)", 2024-09-11T03:43:17.908 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:50.874784+0000", 2024-09-11T03:43:17.909 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 206.94770704699999, 2024-09-11T03:43:17.909 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035152500000000001, 2024-09-11T03:43:17.909 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.909 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.909 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:17.909 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.909 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.909 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:50.874784+0000", 2024-09-11T03:43:17.909 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.909 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.909 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.909 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:50.874784+0000", 2024-09-11T03:43:17.910 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.910 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.910 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.910 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:50.874786+0000", 2024-09-11T03:43:17.910 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8540000000000001e-06 2024-09-11T03:43:17.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.910 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.910 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.910 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:50.874795+0000", 2024-09-11T03:43:17.910 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8089999999999999e-06 2024-09-11T03:43:17.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.910 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.911 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.911 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:50.874810+0000", 2024-09-11T03:43:17.911 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4328000000000001e-05 2024-09-11T03:43:17.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.911 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.911 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.911 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:50.874819+0000", 2024-09-11T03:43:17.911 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7089999999999997e-06 2024-09-11T03:43:17.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.911 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.911 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.911 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:50.874891+0000", 2024-09-11T03:43:17.912 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2117000000000001e-05 2024-09-11T03:43:17.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.912 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:43:17.912 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:50.874897+0000", 2024-09-11T03:43:17.912 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1919999999999996e-06 2024-09-11T03:43:17.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.912 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.912 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:50.876524+0000", 2024-09-11T03:43:17.912 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0016269450000000001 2024-09-11T03:43:17.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.913 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.913 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:50.876609+0000", 2024-09-11T03:43:17.913 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5858000000000004e-05 2024-09-11T03:43:17.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.913 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.914 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:50.878299+0000", 2024-09-11T03:43:17.914 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0016894379999999999 2024-09-11T03:43:17.914 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.914 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.914 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.914 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.914 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1064 2.7 2:e6da0655:::smithi17557870-27:26 [sparse-read 0~1287597,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:43:17.914 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:51.860323+0000", 2024-09-11T03:43:17.914 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 205.962167494, 2024-09-11T03:43:17.914 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002557311, 2024-09-11T03:43:17.914 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.915 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.915 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:17.915 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:17.915 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:17.915 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:17.915 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1064 2024-09-11T03:43:17.915 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.915 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.915 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.916 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860323+0000", 2024-09-11T03:43:17.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.916 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.916 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860323+0000", 2024-09-11T03:43:17.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.916 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.916 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860326+0000", 2024-09-11T03:43:17.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9909999999999998e-06 2024-09-11T03:43:17.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860338+0000", 2024-09-11T03:43:17.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1357e-05 2024-09-11T03:43:17.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860358+0000", 2024-09-11T03:43:17.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0271999999999998e-05 2024-09-11T03:43:17.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860366+0000", 2024-09-11T03:43:17.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5120000000000008e-06 2024-09-11T03:43:17.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.918 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860435+0000", 2024-09-11T03:43:17.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9109000000000005e-05 2024-09-11T03:43:17.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.918 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860564+0000", 2024-09-11T03:43:17.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012859500000000001 2024-09-11T03:43:17.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.919 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.919 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.862880+0000", 2024-09-11T03:43:17.919 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023164750000000001 2024-09-11T03:43:17.919 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.919 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.919 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.920 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1065 2.7 2:e6da0655:::smithi17557870-27:26 [read 0~2336173,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:43:17.920 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:51.860386+0000", 2024-09-11T03:43:17.920 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 205.962105057, 2024-09-11T03:43:17.920 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0038817930000000001, 2024-09-11T03:43:17.920 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.920 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.920 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:17.920 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:17.920 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:17.920 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:17.920 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1065 2024-09-11T03:43:17.920 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.920 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.920 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.921 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860386+0000", 2024-09-11T03:43:17.921 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860386+0000", 2024-09-11T03:43:17.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860386+0000", 2024-09-11T03:43:17.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5499999999999998e-07 2024-09-11T03:43:17.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.923 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.923 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860390+0000", 2024-09-11T03:43:17.923 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.534e-06 2024-09-11T03:43:17.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.923 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.923 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.923 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860401+0000", 2024-09-11T03:43:17.923 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1375999999999999e-05 2024-09-11T03:43:17.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.923 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.923 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.923 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860405+0000", 2024-09-11T03:43:17.924 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2549999999999999e-06 2024-09-11T03:43:17.924 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.924 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.924 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.924 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.862931+0000", 2024-09-11T03:43:17.924 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025256860000000001 2024-09-11T03:43:17.924 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.924 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.924 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.924 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.863035+0000", 2024-09-11T03:43:17.924 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010397 2024-09-11T03:43:17.924 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.925 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.925 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.925 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.864267+0000", 2024-09-11T03:43:17.925 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001232417 2024-09-11T03:43:17.925 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.925 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.925 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.925 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.925 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1066 2.7 2:e6da0655:::smithi17557870-27:26 [sparse-read 0~1287597] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:43:17.925 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:51.860415+0000", 2024-09-11T03:43:17.925 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 205.962076013, 2024-09-11T03:43:17.925 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0041463150000000002, 2024-09-11T03:43:17.926 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.926 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.926 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:17.926 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:17.926 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:17.926 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:17.926 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1066 2024-09-11T03:43:17.926 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.926 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.926 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.926 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860415+0000", 2024-09-11T03:43:17.926 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.927 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.927 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860415+0000", 2024-09-11T03:43:17.927 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.927 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.927 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860415+0000", 2024-09-11T03:43:17.927 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3499999999999996e-07 2024-09-11T03:43:17.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.928 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.928 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860418+0000", 2024-09-11T03:43:17.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3599999999999999e-06 2024-09-11T03:43:17.928 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.928 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.928 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.928 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860424+0000", 2024-09-11T03:43:17.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2940000000000002e-06 2024-09-11T03:43:17.928 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.928 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.928 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.928 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.860429+0000", 2024-09-11T03:43:17.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8590000000000004e-06 2024-09-11T03:43:17.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.929 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.929 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.929 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.864312+0000", 2024-09-11T03:43:17.929 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00388376 2024-09-11T03:43:17.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.929 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.929 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.929 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.864417+0000", 2024-09-11T03:43:17.929 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000104786 2024-09-11T03:43:17.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.929 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.930 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.930 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:51.864561+0000", 2024-09-11T03:43:17.930 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000143721 2024-09-11T03:43:17.930 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.930 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.930 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.930 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.930 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.930 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1165 2.7 2:e6da0655:::smithi17557870-27:head [write 726102~788786] snapc 2e=[2e,2c,2b,28,26] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:43:17.930 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:01.847791+0000", 2024-09-11T03:43:17.930 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 195.97469924699999, 2024-09-11T03:43:17.930 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0052458820000000003, 2024-09-11T03:43:17.930 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.931 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.931 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:17.931 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:17.931 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:17.931 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:17.931 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1165 2024-09-11T03:43:17.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.931 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.931 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.931 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.847791+0000", 2024-09-11T03:43:17.931 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.932 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.932 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.847791+0000", 2024-09-11T03:43:17.932 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.934 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.934 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.847794+0000", 2024-09-11T03:43:17.934 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.593e-06 2024-09-11T03:43:17.934 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.934 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.934 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.848200+0000", 2024-09-11T03:43:17.934 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00040596300000000002 2024-09-11T03:43:17.934 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.934 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.935 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.848220+0000", 2024-09-11T03:43:17.935 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0095e-05 2024-09-11T03:43:17.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.935 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.935 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.848229+0000", 2024-09-11T03:43:17.935 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5120000000000008e-06 2024-09-11T03:43:17.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.935 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.935 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.848292+0000", 2024-09-11T03:43:17.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3112000000000002e-05 2024-09-11T03:43:17.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.936 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:43:17.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.848298+0000", 2024-09-11T03:43:17.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0090000000000004e-06 2024-09-11T03:43:17.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.936 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.849865+0000", 2024-09-11T03:43:17.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001567494 2024-09-11T03:43:17.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.849979+0000", 2024-09-11T03:43:17.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114082 2024-09-11T03:43:17.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:43:17.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.850148+0000", 2024-09-11T03:43:17.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016826800000000001 2024-09-11T03:43:17.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:17.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.851402+0000", 2024-09-11T03:43:17.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012549169999999999 2024-09-11T03:43:17.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:17.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.852584+0000", 2024-09-11T03:43:17.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001181323 2024-09-11T03:43:17.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:17.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.852952+0000", 2024-09-11T03:43:17.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00036795600000000001 2024-09-11T03:43:17.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:17.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.852991+0000", 2024-09-11T03:43:17.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9406000000000002e-05 2024-09-11T03:43:17.939 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.939 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.939 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.853037+0000", 2024-09-11T03:43:17.939 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6152000000000001e-05 2024-09-11T03:43:17.939 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.939 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.939 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.939 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.939 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1166 2.7 2:e6da0655:::smithi17557870-27:head [write 1917886~610393] snapc 2e=[2e,2c,2b,28,26] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:43:17.939 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:01.853392+0000", 2024-09-11T03:43:17.939 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 195.96909825099999, 2024-09-11T03:43:17.939 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0028818210000000001, 2024-09-11T03:43:17.940 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.940 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.940 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:17.940 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:17.940 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:17.940 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:17.940 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1166 2024-09-11T03:43:17.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.940 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.940 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.853392+0000", 2024-09-11T03:43:17.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.853392+0000", 2024-09-11T03:43:17.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.853393+0000", 2024-09-11T03:43:17.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7400000000000002e-07 2024-09-11T03:43:17.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.853547+0000", 2024-09-11T03:43:17.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000153342 2024-09-11T03:43:17.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.853556+0000", 2024-09-11T03:43:17.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8919999999999995e-06 2024-09-11T03:43:17.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.943 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.853562+0000", 2024-09-11T03:43:17.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2120000000000002e-06 2024-09-11T03:43:17.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.943 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.943 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.853596+0000", 2024-09-11T03:43:17.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3887000000000003e-05 2024-09-11T03:43:17.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.943 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.943 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.853657+0000", 2024-09-11T03:43:17.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1111000000000005e-05 2024-09-11T03:43:17.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:43:17.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.853750+0000", 2024-09-11T03:43:17.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3051000000000001e-05 2024-09-11T03:43:17.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:17.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.854531+0000", 2024-09-11T03:43:17.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00078159200000000001 2024-09-11T03:43:17.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:17.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.856114+0000", 2024-09-11T03:43:17.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001582457 2024-09-11T03:43:17.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:17.945 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.856209+0000", 2024-09-11T03:43:17.945 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4903999999999998e-05 2024-09-11T03:43:17.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:17.945 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.856246+0000", 2024-09-11T03:43:17.945 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6974999999999998e-05 2024-09-11T03:43:17.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.945 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:01.856274+0000", 2024-09-11T03:43:17.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8623999999999999e-05 2024-09-11T03:43:17.946 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.946 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.946 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.946 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(osd.1.0:291 2.4 e186/95 2:25f5f8b9:::smithi17557870-26:2e v 186'72, mlcod=185'67)", 2024-09-11T03:43:17.946 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:02.950727+0000", 2024-09-11T03:43:17.946 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 194.87176385500001, 2024-09-11T03:43:17.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002109741, 2024-09-11T03:43:17.946 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:17.946 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:17.946 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:17.946 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:17.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:17.947 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.950727+0000", 2024-09-11T03:43:17.947 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:17.947 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.950727+0000", 2024-09-11T03:43:17.947 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:17.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:17.947 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.950727+0000", 2024-09-11T03:43:17.947 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.15e-07 2024-09-11T03:43:17.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:17.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.950737+0000", 2024-09-11T03:43:17.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4099999999999997e-06 2024-09-11T03:43:17.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:17.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.950744+0000", 2024-09-11T03:43:17.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4089999999999997e-06 2024-09-11T03:43:17.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:17.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.950749+0000", 2024-09-11T03:43:17.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5090000000000001e-06 2024-09-11T03:43:17.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:17.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.951242+0000", 2024-09-11T03:43:17.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00049238099999999996 2024-09-11T03:43:17.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:17.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.951268+0000", 2024-09-11T03:43:17.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.656e-05 2024-09-11T03:43:17.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.950 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:17.950 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.952799+0000", 2024-09-11T03:43:17.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015302149999999999 2024-09-11T03:43:17.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:17.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:17.950 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:17.950 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.952837+0000", 2024-09-11T03:43:17.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7941999999999997e-05 2024-09-11T03:43:17.950 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.950 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.950 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.950 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:17.951 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:17.951 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:43:17.953 INFO:tasks.rados.rados.2.smithi175.stdout:1381: done (2 left) 2024-09-11T03:43:17.953 INFO:tasks.rados.rados.2.smithi175.stdout:1382: done (1 left) 2024-09-11T03:43:17.953 INFO:tasks.rados.rados.2.smithi175.stdout:1383: done (0 left) 2024-09-11T03:43:17.953 INFO:tasks.rados.rados.2.smithi175.stdout:1384: write (excl) oid 3 current snap is 143 2024-09-11T03:43:17.953 INFO:tasks.rados.rados.2.smithi175.stdout:1384: seq_num 491 ranges {652237=793851,2198894=53343} 2024-09-11T03:43:17.953 INFO:tasks.rados.rados.2.smithi175.stdout:1384: writing smithi17557870-3 from 652237 to 1446088 tid 1 2024-09-11T03:43:17.953 INFO:tasks.rados.rados.2.smithi175.stdout:1384: writing smithi17557870-3 from 2198894 to 2252237 tid 2 2024-09-11T03:43:17.953 INFO:tasks.rados.rados.2.smithi175.stdout:1385: delete oid 20 current snap is 143 2024-09-11T03:43:17.953 INFO:tasks.rados.rados.2.smithi175.stdout:1384: finishing write tid 1 to smithi17557870-3 2024-09-11T03:43:17.953 INFO:tasks.rados.rados.2.smithi175.stdout:1384: finishing write tid 2 to smithi17557870-3 2024-09-11T03:43:17.953 INFO:tasks.rados.rados.2.smithi175.stdout:1384: finishing write tid 3 to smithi17557870-3 2024-09-11T03:43:17.953 INFO:tasks.rados.rados.2.smithi175.stdout:1384: finishing write tid 4 to smithi17557870-3 2024-09-11T03:43:17.954 INFO:tasks.rados.rados.2.smithi175.stdout:1384: oid 3 updating version 0 to 516 2024-09-11T03:43:17.954 INFO:tasks.rados.rados.2.smithi175.stdout:1384: oid 3 updating version 516 to 517 2024-09-11T03:43:17.954 INFO:tasks.rados.rados.2.smithi175.stdout:1384: oid 3 version 517 is already newer than 515 2024-09-11T03:43:17.954 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 517 (ObjNum 491 snap 143 seq_num 491) dirty exists 2024-09-11T03:43:17.954 INFO:tasks.rados.rados.2.smithi175.stdout:1384: left oid 3 (ObjNum 491 snap 143 seq_num 491) 2024-09-11T03:43:17.954 INFO:tasks.rados.rados.2.smithi175.stdout:1384: done (1 left) 2024-09-11T03:43:17.954 INFO:tasks.rados.rados.2.smithi175.stdout:1385: done (0 left) 2024-09-11T03:43:17.954 INFO:tasks.rados.rados.2.smithi175.stdout:1386: write oid 37 current snap is 143 2024-09-11T03:43:17.954 INFO:tasks.rados.rados.2.smithi175.stdout:1386: seq_num 493 ranges {685851=544349,1799505=486346} 2024-09-11T03:43:17.954 INFO:tasks.rados.rados.2.smithi175.stdout:1386: writing smithi17557870-37 from 685851 to 1230200 tid 1 2024-09-11T03:43:17.954 INFO:tasks.rados.rados.2.smithi175.stdout:1386: writing smithi17557870-37 from 1799505 to 2285851 tid 2 2024-09-11T03:43:17.954 INFO:tasks.rados.rados.2.smithi175.stdout:1387: write (excl) oid 25 current snap is 143 2024-09-11T03:43:17.954 INFO:tasks.rados.rados.2.smithi175.stdout:1386: finishing write tid 1 to smithi17557870-37 2024-09-11T03:43:17.954 INFO:tasks.rados.rados.2.smithi175.stdout:1387: seq_num 494 ranges {702658=619598,1857987=444671} 2024-09-11T03:43:17.955 INFO:tasks.rados.rados.2.smithi175.stdout:1387: writing smithi17557870-25 from 702658 to 1322256 tid 1 2024-09-11T03:43:17.955 INFO:tasks.rados.rados.2.smithi175.stdout:1387: writing smithi17557870-25 from 1857987 to 2302658 tid 2 2024-09-11T03:43:17.955 INFO:tasks.rados.rados.2.smithi175.stdout:1388: snap_remove snap 140 2024-09-11T03:43:17.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:17.956 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:17.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:18.057 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:18.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:18.079 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:18.079 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:18.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:18.204 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:18.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:18.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:18.307 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:18.329 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:18.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:18.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:18 smithi084 ceph-mon[27254]: pgmap v723: 57 pgs: 57 active+clean; 134 MiB data, 674 MiB used, 893 GiB / 894 GiB avail; 17 MiB/s rd, 3.1 MiB/s wr, 29 op/s 2024-09-11T03:43:18.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:18 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4243587578' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:18.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:18 smithi084 ceph-mon[27254]: osdmap e380: 12 total, 12 up, 10 in 2024-09-11T03:43:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:18 smithi121 ceph-mon[30729]: pgmap v723: 57 pgs: 57 active+clean; 134 MiB data, 674 MiB used, 893 GiB / 894 GiB avail; 17 MiB/s rd, 3.1 MiB/s wr, 29 op/s 2024-09-11T03:43:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:18 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4243587578' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:18 smithi121 ceph-mon[30729]: osdmap e380: 12 total, 12 up, 10 in 2024-09-11T03:43:18.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:18.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:18.457 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:18.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:18 smithi175 ceph-mon[30894]: pgmap v723: 57 pgs: 57 active+clean; 134 MiB data, 674 MiB used, 893 GiB / 894 GiB avail; 17 MiB/s rd, 3.1 MiB/s wr, 29 op/s 2024-09-11T03:43:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:18 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4243587578' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:18 smithi175 ceph-mon[30894]: osdmap e380: 12 total, 12 up, 10 in 2024-09-11T03:43:18.559 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:18.582 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:18.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:18.583 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:18.592 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-11T03:43:18.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:18.705 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:18.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:18.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:18.807 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:18.816 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:18.830 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:18.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:18.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:18.871 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:43:18.903 INFO:tasks.rados.rados.2.smithi175.stdout:1388: done (2 left) 2024-09-11T03:43:18.903 INFO:tasks.rados.rados.2.smithi175.stdout:1389: copy_from oid 6 from oid 47 current snap is 143 2024-09-11T03:43:18.903 INFO:tasks.rados.rados.2.smithi175.stdout:1386: finishing write tid 2 to smithi17557870-37 2024-09-11T03:43:18.903 INFO:tasks.rados.rados.2.smithi175.stdout:1386: finishing write tid 3 to smithi17557870-37 2024-09-11T03:43:18.903 INFO:tasks.rados.rados.2.smithi175.stdout:1386: finishing write tid 4 to smithi17557870-37 2024-09-11T03:43:18.903 INFO:tasks.rados.rados.2.smithi175.stdout:1386: oid 37 updating version 0 to 148 2024-09-11T03:43:18.903 INFO:tasks.rados.rados.2.smithi175.stdout:1386: oid 37 version 148 is already newer than 146 2024-09-11T03:43:18.903 INFO:tasks.rados.rados.2.smithi175.stdout:1386: oid 37 version 148 is already newer than 147 2024-09-11T03:43:18.903 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 148 (ObjNum 493 snap 143 seq_num 493) dirty exists 2024-09-11T03:43:18.904 INFO:tasks.rados.rados.2.smithi175.stdout:1386: left oid 37 (ObjNum 493 snap 143 seq_num 493) 2024-09-11T03:43:18.904 INFO:tasks.rados.rados.2.smithi175.stdout:1387: finishing write tid 1 to smithi17557870-25 2024-09-11T03:43:18.904 INFO:tasks.rados.rados.2.smithi175.stdout:1387: finishing write tid 2 to smithi17557870-25 2024-09-11T03:43:18.904 INFO:tasks.rados.rados.2.smithi175.stdout:1387: finishing write tid 3 to smithi17557870-25 2024-09-11T03:43:18.904 INFO:tasks.rados.rados.2.smithi175.stdout:1387: finishing write tid 4 to smithi17557870-25 2024-09-11T03:43:18.904 INFO:tasks.rados.rados.2.smithi175.stdout:1387: oid 25 updating version 0 to 232 2024-09-11T03:43:18.904 INFO:tasks.rados.rados.2.smithi175.stdout:1387: oid 25 version 232 is already newer than 231 2024-09-11T03:43:18.904 INFO:tasks.rados.rados.2.smithi175.stdout:1387: oid 25 version 232 is already newer than 230 2024-09-11T03:43:18.904 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 232 (ObjNum 494 snap 143 seq_num 494) dirty exists 2024-09-11T03:43:18.904 INFO:tasks.rados.rados.2.smithi175.stdout:1387: left oid 25 (ObjNum 494 snap 143 seq_num 494) 2024-09-11T03:43:18.904 INFO:tasks.rados.rados.2.smithi175.stdout:1386: done (2 left) 2024-09-11T03:43:18.904 INFO:tasks.rados.rados.2.smithi175.stdout:1387: done (1 left) 2024-09-11T03:43:18.904 INFO:tasks.rados.rados.2.smithi175.stdout:1390: write oid 33 current snap is 143 2024-09-11T03:43:18.904 INFO:tasks.rados.rados.2.smithi175.stdout:1390: seq_num 495 ranges {719465=411200,1632822=686643} 2024-09-11T03:43:18.909 INFO:tasks.rados.rados.2.smithi175.stdout:1390: writing smithi17557870-33 from 719465 to 1130665 tid 1 2024-09-11T03:43:18.916 INFO:tasks.rados.rados.2.smithi175.stdout:1390: writing smithi17557870-33 from 1632822 to 2319465 tid 2 2024-09-11T03:43:18.917 INFO:tasks.rados.rados.2.smithi175.stdout:1391: rollback oid 7 current snap is 143 2024-09-11T03:43:18.917 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 7 to 139 2024-09-11T03:43:18.917 INFO:tasks.rados.rados.2.smithi175.stdout:1390: finishing write tid 1 to smithi17557870-33 2024-09-11T03:43:18.917 INFO:tasks.rados.rados.2.smithi175.stdout:1389: finishing copy_from racing read to smithi17557870-6 2024-09-11T03:43:18.917 INFO:tasks.rados.rados.2.smithi175.stdout:1389: finishing copy_from to smithi17557870-6 2024-09-11T03:43:18.917 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 408 (ObjNum 369 snap 113 seq_num 369) dirty exists 2024-09-11T03:43:18.917 INFO:tasks.rados.rados.2.smithi175.stdout:1389: done (2 left) 2024-09-11T03:43:18.917 INFO:tasks.rados.rados.2.smithi175.stdout:1392: snap_create 2024-09-11T03:43:18.920 INFO:tasks.rados.rados.2.smithi175.stdout:1391: finishing rollback tid 0 to smithi17557870-7 2024-09-11T03:43:18.920 INFO:tasks.rados.rados.2.smithi175.stdout:1391: finishing rollback tid 1 to smithi17557870-7 2024-09-11T03:43:18.920 INFO:tasks.rados.rados.2.smithi175.stdout:1390: finishing write tid 2 to smithi17557870-33 2024-09-11T03:43:18.921 INFO:tasks.rados.rados.2.smithi175.stdout:1390: finishing write tid 3 to smithi17557870-33 2024-09-11T03:43:18.921 INFO:tasks.rados.rados.2.smithi175.stdout:1391: finishing rollback tid 2 to smithi17557870-7 2024-09-11T03:43:18.921 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 459 (ObjNum 440 snap 130 seq_num 440) dirty exists 2024-09-11T03:43:18.921 INFO:tasks.rados.rados.2.smithi175.stdout:1390: finishing write tid 4 to smithi17557870-33 2024-09-11T03:43:18.921 INFO:tasks.rados.rados.2.smithi175.stdout:1390: oid 33 updating version 0 to 522 2024-09-11T03:43:18.921 INFO:tasks.rados.rados.2.smithi175.stdout:1390: oid 33 version 522 is already newer than 520 2024-09-11T03:43:18.922 INFO:tasks.rados.rados.2.smithi175.stdout:1390: oid 33 version 522 is already newer than 521 2024-09-11T03:43:18.922 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 522 (ObjNum 495 snap 143 seq_num 495) dirty exists 2024-09-11T03:43:18.922 INFO:tasks.rados.rados.2.smithi175.stdout:1390: left oid 33 (ObjNum 495 snap 143 seq_num 495) 2024-09-11T03:43:18.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:18.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:18.958 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:18.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:19.060 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:19.082 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:19.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:19.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:19.207 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:19.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:19.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:19.308 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:43:19.308 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:43:19.308 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:43:19.308 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:43:19.308 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:19.332 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:19.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:19.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:19.456 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:19.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:19.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:19.558 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:19.580 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:19.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:19.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:19.682 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:19.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:19.705 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:19.705 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:19.806 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:19.828 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:19.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:19.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:19.906 INFO:teuthology.orchestra.run.smithi084.stderr:clear 1.a pg_upmap_items mapping 2024-09-11T03:43:19.906 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:43:19.911 INFO:tasks.rados.rados.2.smithi175.stdout:1390: done (2 left) 2024-09-11T03:43:19.911 INFO:tasks.rados.rados.2.smithi175.stdout:1391: done (1 left) 2024-09-11T03:43:19.912 INFO:tasks.rados.rados.2.smithi175.stdout:1392: done (0 left) 2024-09-11T03:43:19.912 INFO:tasks.rados.rados.2.smithi175.stdout:1393: write oid 25 current snap is 144 2024-09-11T03:43:19.912 INFO:tasks.rados.rados.2.smithi175.stdout:1393: seq_num 496 ranges {736272=486449,1974951=361321} 2024-09-11T03:43:19.918 INFO:tasks.rados.rados.2.smithi175.stdout:1393: writing smithi17557870-25 from 736272 to 1222721 tid 1 2024-09-11T03:43:19.925 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-11T03:43:19.926 INFO:tasks.rados.rados.2.smithi175.stdout:1393: writing smithi17557870-25 from 1974951 to 2336272 tid 2 2024-09-11T03:43:19.926 INFO:tasks.rados.rados.2.smithi175.stdout:1394: write (excl) oid 46 current snap is 144 2024-09-11T03:43:19.927 INFO:tasks.rados.rados.2.smithi175.stdout:1393: finishing write tid 1 to smithi17557870-25 2024-09-11T03:43:19.927 INFO:tasks.rados.rados.2.smithi175.stdout:1394: seq_num 497 ranges {753079=561698,2033433=319646} 2024-09-11T03:43:19.930 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:19.932 INFO:tasks.rados.rados.2.smithi175.stdout:1394: writing smithi17557870-46 from 753079 to 1314777 tid 1 2024-09-11T03:43:19.938 INFO:tasks.rados.rados.2.smithi175.stdout:1394: writing smithi17557870-46 from 2033433 to 2353079 tid 2 2024-09-11T03:43:19.938 INFO:tasks.rados.rados.2.smithi175.stdout:1395: write oid 22 current snap is 144 2024-09-11T03:43:19.938 INFO:tasks.rados.rados.2.smithi175.stdout:1393: finishing write tid 2 to smithi17557870-25 2024-09-11T03:43:19.938 INFO:tasks.rados.rados.2.smithi175.stdout:1395: seq_num 498 ranges {769886=636947,2091915=277971} 2024-09-11T03:43:19.945 INFO:tasks.rados.rados.2.smithi175.stdout:1395: writing smithi17557870-22 from 769886 to 1406833 tid 1 2024-09-11T03:43:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:1395: writing smithi17557870-22 from 2091915 to 2369886 tid 2 2024-09-11T03:43:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:1396: snap_create 2024-09-11T03:43:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:1393: finishing write tid 3 to smithi17557870-25 2024-09-11T03:43:19.952 INFO:tasks.rados.rados.2.smithi175.stdout:1393: finishing write tid 4 to smithi17557870-25 2024-09-11T03:43:19.953 INFO:tasks.rados.rados.2.smithi175.stdout:1393: oid 25 updating version 0 to 235 2024-09-11T03:43:19.953 INFO:tasks.rados.rados.2.smithi175.stdout:1393: oid 25 version 235 is already newer than 234 2024-09-11T03:43:19.953 INFO:tasks.rados.rados.2.smithi175.stdout:1393: oid 25 version 235 is already newer than 233 2024-09-11T03:43:19.953 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 235 (ObjNum 496 snap 144 seq_num 496) dirty exists 2024-09-11T03:43:19.953 INFO:tasks.rados.rados.2.smithi175.stdout:1393: left oid 25 (ObjNum 496 snap 144 seq_num 496) 2024-09-11T03:43:19.953 INFO:tasks.rados.rados.2.smithi175.stdout:1394: finishing write tid 1 to smithi17557870-46 2024-09-11T03:43:19.954 INFO:tasks.rados.rados.2.smithi175.stdout:1394: finishing write tid 2 to smithi17557870-46 2024-09-11T03:43:19.954 INFO:tasks.rados.rados.2.smithi175.stdout:1394: finishing write tid 3 to smithi17557870-46 2024-09-11T03:43:19.954 INFO:tasks.rados.rados.2.smithi175.stdout:1394: finishing write tid 4 to smithi17557870-46 2024-09-11T03:43:19.954 INFO:tasks.rados.rados.2.smithi175.stdout:1394: oid 46 updating version 0 to 189 2024-09-11T03:43:19.955 INFO:tasks.rados.rados.2.smithi175.stdout:1394: oid 46 updating version 189 to 190 2024-09-11T03:43:19.955 INFO:tasks.rados.rados.2.smithi175.stdout:1394: oid 46 version 190 is already newer than 188 2024-09-11T03:43:19.955 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 190 (ObjNum 497 snap 144 seq_num 497) dirty exists 2024-09-11T03:43:19.956 INFO:tasks.rados.rados.2.smithi175.stdout:1394: left oid 46 (ObjNum 497 snap 144 seq_num 497) 2024-09-11T03:43:19.956 INFO:tasks.rados.rados.2.smithi175.stdout:1395: finishing write tid 1 to smithi17557870-22 2024-09-11T03:43:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:19.957 INFO:tasks.rados.rados.2.smithi175.stdout:1395: finishing write tid 2 to smithi17557870-22 2024-09-11T03:43:19.957 INFO:tasks.rados.rados.2.smithi175.stdout:1395: finishing write tid 3 to smithi17557870-22 2024-09-11T03:43:19.958 INFO:tasks.rados.rados.2.smithi175.stdout:1395: finishing write tid 4 to smithi17557870-22 2024-09-11T03:43:19.958 INFO:tasks.rados.rados.2.smithi175.stdout:1395: oid 22 updating version 0 to 523 2024-09-11T03:43:19.958 INFO:tasks.rados.rados.2.smithi175.stdout:1395: oid 22 updating version 523 to 525 2024-09-11T03:43:19.958 INFO:tasks.rados.rados.2.smithi175.stdout:1395: oid 22 version 525 is already newer than 524 2024-09-11T03:43:19.958 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 525 (ObjNum 498 snap 144 seq_num 498) dirty exists 2024-09-11T03:43:19.958 INFO:tasks.rados.rados.2.smithi175.stdout:1395: left oid 22 (ObjNum 498 snap 144 seq_num 498) 2024-09-11T03:43:19.958 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:19.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:20.060 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:20.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:20.082 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:20.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:20.169 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:43:20.248 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:20.250 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:19 smithi121 ceph-mon[30729]: pgmap v726: 57 pgs: 57 active+clean; 135 MiB data, 679 MiB used, 893 GiB / 894 GiB avail; 8.2 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-11T03:43:20.250 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:19 smithi121 ceph-mon[30729]: osdmap e381: 12 total, 12 up, 10 in 2024-09-11T03:43:20.250 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:19 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2161659455' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "1.a"} : dispatch 2024-09-11T03:43:20.250 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:19 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1986230479' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:43:20.250 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:19 smithi175 ceph-mon[30894]: pgmap v726: 57 pgs: 57 active+clean; 135 MiB data, 679 MiB used, 893 GiB / 894 GiB avail; 8.2 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-11T03:43:20.250 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:19 smithi175 ceph-mon[30894]: osdmap e381: 12 total, 12 up, 10 in 2024-09-11T03:43:20.250 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:19 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2161659455' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "1.a"} : dispatch 2024-09-11T03:43:20.250 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:19 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1986230479' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:43:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:20.276 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:20.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:20.377 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:19 smithi084 ceph-mon[27254]: pgmap v726: 57 pgs: 57 active+clean; 135 MiB data, 679 MiB used, 893 GiB / 894 GiB avail; 8.2 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-11T03:43:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:19 smithi084 ceph-mon[27254]: osdmap e381: 12 total, 12 up, 10 in 2024-09-11T03:43:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:19 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2161659455' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "1.a"} : dispatch 2024-09-11T03:43:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:19 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1986230479' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:43:20.399 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:20.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:20.502 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:20.523 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:20.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:20.524 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:20.527 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:43:20.625 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:20.646 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:20.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:20.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:20.731 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:43:20.731 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:43:20.731 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:43:20.731 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:43:20.731 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:43:20.748 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:20.774 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:20.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:20.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:20.876 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:20.898 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:20.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:20.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:20.915 INFO:tasks.rados.rados.2.smithi175.stdout:1393: done (3 left) 2024-09-11T03:43:20.915 INFO:tasks.rados.rados.2.smithi175.stdout:1394: done (2 left) 2024-09-11T03:43:20.915 INFO:tasks.rados.rados.2.smithi175.stdout:1395: done (1 left) 2024-09-11T03:43:20.915 INFO:tasks.rados.rados.2.smithi175.stdout:1396: done (0 left) 2024-09-11T03:43:20.915 INFO:tasks.rados.rados.2.smithi175.stdout:1397: snap_remove snap 139 2024-09-11T03:43:21.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:21.026 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:21.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:21.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:21.127 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:21.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:20 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2161659455' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.a"}]': finished 2024-09-11T03:43:21.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:20 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1986230479' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:43:21.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:20 smithi084 ceph-mon[27254]: osdmap e382: 12 total, 12 up, 10 in 2024-09-11T03:43:21.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:20 smithi084 ceph-mon[27254]: 1.16 scrub starts 2024-09-11T03:43:21.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:20 smithi084 ceph-mon[27254]: 1.16 scrub ok 2024-09-11T03:43:21.150 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:21.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:21.154 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:21.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:20 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2161659455' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.a"}]': finished 2024-09-11T03:43:21.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:20 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1986230479' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:43:21.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:20 smithi121 ceph-mon[30729]: osdmap e382: 12 total, 12 up, 10 in 2024-09-11T03:43:21.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:20 smithi121 ceph-mon[30729]: 1.16 scrub starts 2024-09-11T03:43:21.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:20 smithi121 ceph-mon[30729]: 1.16 scrub ok 2024-09-11T03:43:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:20 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2161659455' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.a"}]': finished 2024-09-11T03:43:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:20 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1986230479' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:43:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:20 smithi175 ceph-mon[30894]: osdmap e382: 12 total, 12 up, 10 in 2024-09-11T03:43:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:20 smithi175 ceph-mon[30894]: 1.16 scrub starts 2024-09-11T03:43:21.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:20 smithi175 ceph-mon[30894]: 1.16 scrub ok 2024-09-11T03:43:21.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:21.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:21.282 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:21.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:21.353 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-11T03:43:21.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:21.408 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:21.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:21.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:21.484 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:21.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:21.535 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:21.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:21.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:21.593 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:43:21.637 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:21.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:21.660 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:21.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:21.761 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:21.782 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:21.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:21.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:21.884 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:21.904 INFO:tasks.rados.rados.2.smithi175.stdout:1397: done (0 left) 2024-09-11T03:43:21.904 INFO:tasks.rados.rados.2.smithi175.stdout:1398: copy_from oid 20 from oid 4 current snap is 145 2024-09-11T03:43:21.905 INFO:tasks.rados.rados.2.smithi175.stdout:1399: read oid 46 snap -1 2024-09-11T03:43:21.905 INFO:tasks.rados.rados.2.smithi175.stdout:1399: expect (ObjNum 497 snap 144 seq_num 497) 2024-09-11T03:43:21.905 INFO:tasks.rados.rados.2.smithi175.stdout:1400: copy_from oid 37 from oid 39 current snap is 145 2024-09-11T03:43:21.905 INFO:tasks.rados.rados.2.smithi175.stdout:1401: read oid 19 snap -1 2024-09-11T03:43:21.905 INFO:tasks.rados.rados.2.smithi175.stdout:1401: expect deleted 2024-09-11T03:43:21.905 INFO:tasks.rados.rados.2.smithi175.stdout:1402: write oid 13 current snap is 145 2024-09-11T03:43:21.905 INFO:tasks.rados.rados.2.smithi175.stdout:1402: seq_num 499 ranges {786693=712196,2150397=236296} 2024-09-11T03:43:21.909 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:21.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:21.910 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:21.913 INFO:tasks.rados.rados.2.smithi175.stdout:1402: writing smithi17557870-13 from 786693 to 1498889 tid 1 2024-09-11T03:43:21.918 INFO:tasks.rados.rados.2.smithi175.stdout:1402: writing smithi17557870-13 from 2150397 to 2386693 tid 2 2024-09-11T03:43:21.918 INFO:tasks.rados.rados.2.smithi175.stdout:1403: snap_remove snap 141 2024-09-11T03:43:21.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:21 smithi175 ceph-mon[30894]: pgmap v729: 57 pgs: 57 active+clean; 135 MiB data, 679 MiB used, 893 GiB / 894 GiB avail; 255 B/s rd, 1.8 MiB/s wr, 3 op/s 2024-09-11T03:43:21.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:21 smithi175 ceph-mon[30894]: osdmap e383: 12 total, 12 up, 10 in 2024-09-11T03:43:21.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:21 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:21.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:21 smithi175 ceph-mon[30894]: osdmap e384: 12 total, 12 up, 10 in 2024-09-11T03:43:21.995 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:43:21.995 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:43:22.012 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:22.012 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:43:22.013 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:43:22.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:21 smithi084 ceph-mon[27254]: pgmap v729: 57 pgs: 57 active+clean; 135 MiB data, 679 MiB used, 893 GiB / 894 GiB avail; 255 B/s rd, 1.8 MiB/s wr, 3 op/s 2024-09-11T03:43:22.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:21 smithi084 ceph-mon[27254]: osdmap e383: 12 total, 12 up, 10 in 2024-09-11T03:43:22.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:21 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:22.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:21 smithi084 ceph-mon[27254]: osdmap e384: 12 total, 12 up, 10 in 2024-09-11T03:43:22.028 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:43:22.029 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:43:22.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:22.036 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:22.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:22.044 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:43:22.044 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:43:22.061 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:43:22.061 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:43:22.064 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:43:22.064 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:43:22.064 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:43:22.064 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:43:22.064 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.065 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:365 2.c 2:327469ff:::smithi17557870-49:head [read 0~2806736,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:43:22.065 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.850028+0000", 2024-09-11T03:43:22.065 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 271.19983092299998, 2024-09-11T03:43:22.065 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0077480070000000003, 2024-09-11T03:43:22.065 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.065 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.065 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:22.065 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:22.065 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:22.065 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:22.065 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 365 2024-09-11T03:43:22.065 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.066 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.066 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.066 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.066 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850028+0000", 2024-09-11T03:43:22.066 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.066 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.066 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.066 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.066 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850028+0000", 2024-09-11T03:43:22.066 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.066 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.066 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.067 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.067 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850029+0000", 2024-09-11T03:43:22.067 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1200000000000003e-07 2024-09-11T03:43:22.067 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.067 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.067 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.067 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850033+0000", 2024-09-11T03:43:22.067 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4789999999999998e-06 2024-09-11T03:43:22.067 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.067 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.067 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.067 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850043+0000", 2024-09-11T03:43:22.068 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.38e-06 2024-09-11T03:43:22.068 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.068 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.068 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.068 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850056+0000", 2024-09-11T03:43:22.068 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3179999999999999e-05 2024-09-11T03:43:22.068 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.068 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.068 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.068 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850150+0000", 2024-09-11T03:43:22.068 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4549000000000006e-05 2024-09-11T03:43:22.068 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.069 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.069 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:43:22.069 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850155+0000", 2024-09-11T03:43:22.069 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0440000000000003e-06 2024-09-11T03:43:22.069 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.069 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.069 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.069 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.855967+0000", 2024-09-11T03:43:22.069 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0058113289999999996 2024-09-11T03:43:22.069 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.069 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.069 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.069 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.856051+0000", 2024-09-11T03:43:22.070 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3840999999999999e-05 2024-09-11T03:43:22.070 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.070 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.070 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.070 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.857776+0000", 2024-09-11T03:43:22.070 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0017255930000000001 2024-09-11T03:43:22.070 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.070 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.070 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.070 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.070 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.070 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:640 2.a 2:5014e305:::smithi17557870-46:head [copy-from ver 23] snapc 11=[e] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:43:22.070 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:43:22.071 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 253.93956593499999, 2024-09-11T03:43:22.071 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019221736, 2024-09-11T03:43:22.071 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.071 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.071 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:22.071 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:22.071 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:22.071 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:22.071 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 640 2024-09-11T03:43:22.071 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.071 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.071 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.071 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.071 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:43:22.072 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.072 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.072 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.072 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.072 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:43:22.072 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.072 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.072 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.072 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.072 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110295+0000", 2024-09-11T03:43:22.072 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0609999999999998e-06 2024-09-11T03:43:22.072 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.073 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.073 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.073 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110305+0000", 2024-09-11T03:43:22.073 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4169999999999998e-06 2024-09-11T03:43:22.073 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.073 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.073 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.073 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110321+0000", 2024-09-11T03:43:22.073 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6288000000000001e-05 2024-09-11T03:43:22.073 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.073 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.074 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.074 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110328+0000", 2024-09-11T03:43:22.074 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3950000000000003e-06 2024-09-11T03:43:22.074 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.074 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.074 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.074 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110396+0000", 2024-09-11T03:43:22.074 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7164999999999998e-05 2024-09-11T03:43:22.074 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.074 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.074 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.074 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110507+0000", 2024-09-11T03:43:22.074 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000111217 2024-09-11T03:43:22.074 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.075 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.075 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:43:22.075 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.118213+0000", 2024-09-11T03:43:22.075 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0077066249999999999 2024-09-11T03:43:22.075 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.075 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.075 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:22.075 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125103+0000", 2024-09-11T03:43:22.075 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068894359999999997 2024-09-11T03:43:22.075 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.075 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.075 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:22.075 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125459+0000", 2024-09-11T03:43:22.075 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000356146 2024-09-11T03:43:22.076 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.076 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.076 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:22.076 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129369+0000", 2024-09-11T03:43:22.076 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039099249999999999 2024-09-11T03:43:22.076 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.076 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.076 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:22.076 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129428+0000", 2024-09-11T03:43:22.076 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9113000000000001e-05 2024-09-11T03:43:22.077 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.077 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.077 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.077 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129515+0000", 2024-09-11T03:43:22.077 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6947999999999995e-05 2024-09-11T03:43:22.077 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.077 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.077 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.077 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.077 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.077 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:641 2.a 2:5014e305:::smithi17557870-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:43:22.077 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:43:22.077 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 253.93951454399999, 2024-09-11T03:43:22.078 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011882876000000001, 2024-09-11T03:43:22.078 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.078 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.078 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:22.078 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:22.078 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:22.078 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:22.078 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 641 2024-09-11T03:43:22.078 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.078 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.078 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.078 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.078 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:43:22.079 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.079 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.079 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.079 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.079 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:43:22.079 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.079 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.079 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.079 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.079 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:43:22.079 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7599999999999998e-07 2024-09-11T03:43:22.079 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.079 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.080 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.080 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110347+0000", 2024-09-11T03:43:22.080 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.685e-06 2024-09-11T03:43:22.080 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.080 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.080 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.080 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110351+0000", 2024-09-11T03:43:22.080 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4889999999999997e-06 2024-09-11T03:43:22.080 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.080 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.080 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.081 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110354+0000", 2024-09-11T03:43:22.081 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0510000000000001e-06 2024-09-11T03:43:22.081 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.081 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.081 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.081 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110644+0000", 2024-09-11T03:43:22.081 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00028940200000000002 2024-09-11T03:43:22.081 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.081 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.081 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:22.081 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110673+0000", 2024-09-11T03:43:22.081 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9833e-05 2024-09-11T03:43:22.081 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.082 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.082 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.082 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122038+0000", 2024-09-11T03:43:22.082 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011364915 2024-09-11T03:43:22.082 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.082 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.082 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.082 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122127+0000", 2024-09-11T03:43:22.082 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8987999999999996e-05 2024-09-11T03:43:22.082 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.082 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.082 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.082 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122227+0000", 2024-09-11T03:43:22.082 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010013699999999999 2024-09-11T03:43:22.083 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.083 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.083 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.083 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.083 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.083 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:752 2.c 2:3c007163:::smithi17557870-35:head [copy-from ver 39] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:43:22.083 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:43:22.083 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 247.04084326200001, 2024-09-11T03:43:22.083 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017596783000000001, 2024-09-11T03:43:22.083 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.083 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.083 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:22.083 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:22.083 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:22.084 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:22.084 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 752 2024-09-11T03:43:22.084 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.084 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.084 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.084 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.084 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:43:22.084 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.084 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.084 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.084 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.084 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:43:22.084 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.085 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.085 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.085 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.085 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009018+0000", 2024-09-11T03:43:22.085 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.48e-06 2024-09-11T03:43:22.085 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.085 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.085 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.085 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009029+0000", 2024-09-11T03:43:22.085 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0377000000000001e-05 2024-09-11T03:43:22.085 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.085 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.085 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.086 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009050+0000", 2024-09-11T03:43:22.086 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0849e-05 2024-09-11T03:43:22.086 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.086 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.086 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.086 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009058+0000", 2024-09-11T03:43:22.086 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8019999999999998e-06 2024-09-11T03:43:22.086 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.086 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.086 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.086 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009125+0000", 2024-09-11T03:43:22.086 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6388000000000002e-05 2024-09-11T03:43:22.087 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.087 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.087 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:43:22.087 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009130+0000", 2024-09-11T03:43:22.087 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0900000000000004e-06 2024-09-11T03:43:22.087 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.087 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.087 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.087 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010518+0000", 2024-09-11T03:43:22.087 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013880030000000001 2024-09-11T03:43:22.087 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.087 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.087 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.088 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010582+0000", 2024-09-11T03:43:22.088 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.368e-05 2024-09-11T03:43:22.088 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.088 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.088 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:43:22.088 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.015350+0000", 2024-09-11T03:43:22.088 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0047682870000000004 2024-09-11T03:43:22.088 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.088 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.088 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:22.088 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.019005+0000", 2024-09-11T03:43:22.088 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0036549040000000001 2024-09-11T03:43:22.088 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.089 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.089 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:22.089 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.024096+0000", 2024-09-11T03:43:22.089 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050917439999999996 2024-09-11T03:43:22.089 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.089 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.089 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:22.089 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026521+0000", 2024-09-11T03:43:22.089 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024247399999999999 2024-09-11T03:43:22.089 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.089 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.089 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:22.090 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026559+0000", 2024-09-11T03:43:22.090 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8232999999999998e-05 2024-09-11T03:43:22.090 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.090 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.090 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.090 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026613+0000", 2024-09-11T03:43:22.090 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3205999999999998e-05 2024-09-11T03:43:22.090 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.090 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.090 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.090 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.090 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.090 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:753 2.c 2:3c007163:::smithi17557870-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:43:22.091 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:43:22.091 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 247.04077617499999, 2024-09-11T03:43:22.091 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0094619260000000007, 2024-09-11T03:43:22.091 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.091 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.091 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:22.091 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:22.091 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:22.091 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:22.091 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 753 2024-09-11T03:43:22.091 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.091 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.092 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.092 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.092 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:43:22.092 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.092 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.092 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.092 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.092 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:43:22.092 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.092 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.092 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.092 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.092 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:43:22.093 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2100000000000002e-07 2024-09-11T03:43:22.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.093 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.093 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009085+0000", 2024-09-11T03:43:22.093 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5910000000000001e-06 2024-09-11T03:43:22.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.093 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.093 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009090+0000", 2024-09-11T03:43:22.093 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5180000000000004e-06 2024-09-11T03:43:22.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.094 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.094 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009094+0000", 2024-09-11T03:43:22.094 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0620000000000001e-06 2024-09-11T03:43:22.094 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.094 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.094 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.094 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009178+0000", 2024-09-11T03:43:22.094 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4525999999999997e-05 2024-09-11T03:43:22.094 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.094 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.094 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:43:22.094 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009182+0000", 2024-09-11T03:43:22.094 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4640000000000002e-06 2024-09-11T03:43:22.095 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.095 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.095 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.095 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010677+0000", 2024-09-11T03:43:22.095 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014957569999999999 2024-09-11T03:43:22.095 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.095 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.095 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:22.095 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010706+0000", 2024-09-11T03:43:22.095 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8873999999999998e-05 2024-09-11T03:43:22.095 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.095 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.095 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.096 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018385+0000", 2024-09-11T03:43:22.096 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0076790490000000003 2024-09-11T03:43:22.096 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.096 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.096 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.096 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018466+0000", 2024-09-11T03:43:22.096 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0483999999999997e-05 2024-09-11T03:43:22.096 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.096 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.096 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.096 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018545+0000", 2024-09-11T03:43:22.096 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9179999999999997e-05 2024-09-11T03:43:22.097 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.097 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.097 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.097 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.097 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.097 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:985 2.b e163/95 2:dde92fc6:::smithi17557870-41:head v 163'52, mlcod=158'48)", 2024-09-11T03:43:22.097 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:43:22.097 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 222.11690863000001, 2024-09-11T03:43:22.097 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0092288890000000005, 2024-09-11T03:43:22.097 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.097 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.097 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:22.097 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.098 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.098 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.098 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:43:22.098 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.098 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.098 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.098 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.098 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:43:22.098 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.098 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.098 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.098 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.098 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:43:22.098 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7799999999999997e-07 2024-09-11T03:43:22.099 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.099 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.099 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.099 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935287+0000", 2024-09-11T03:43:22.099 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023361599999999999 2024-09-11T03:43:22.099 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.099 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.099 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.099 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935321+0000", 2024-09-11T03:43:22.099 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3840000000000001e-05 2024-09-11T03:43:22.099 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.100 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.100 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.100 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935335+0000", 2024-09-11T03:43:22.100 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4184e-05 2024-09-11T03:43:22.100 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.100 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.100 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.100 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935428+0000", 2024-09-11T03:43:22.100 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3144000000000001e-05 2024-09-11T03:43:22.100 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.100 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.100 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.100 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935503+0000", 2024-09-11T03:43:22.100 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4697999999999995e-05 2024-09-11T03:43:22.101 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.101 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.101 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:22.101 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.942117+0000", 2024-09-11T03:43:22.101 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0066141350000000002 2024-09-11T03:43:22.101 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.101 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.101 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.101 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.942179+0000", 2024-09-11T03:43:22.101 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.245e-05 2024-09-11T03:43:22.101 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.101 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.101 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.102 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.102 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.102 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:990 2.a 2:5014e305:::smithi17557870-46:head [write 638699~541153] snapc 24=[24,23,22,20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-11T03:43:22.102 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:43:22.102 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 222.11292737799999, 2024-09-11T03:43:22.102 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0080995760000000007, 2024-09-11T03:43:22.102 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.102 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.102 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:22.102 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:22.102 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:22.102 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:22.102 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 990 2024-09-11T03:43:22.103 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.103 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.103 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.103 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.103 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:43:22.103 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.103 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.103 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.103 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.103 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:43:22.103 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.103 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.104 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.104 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.104 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936935+0000", 2024-09-11T03:43:22.104 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7800000000000001e-06 2024-09-11T03:43:22.104 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.104 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.104 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.104 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937090+0000", 2024-09-11T03:43:22.104 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000155857 2024-09-11T03:43:22.104 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.104 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.104 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.104 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937109+0000", 2024-09-11T03:43:22.105 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8216e-05 2024-09-11T03:43:22.105 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.105 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.105 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.105 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937119+0000", 2024-09-11T03:43:22.105 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0406e-05 2024-09-11T03:43:22.105 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.105 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.105 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.105 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937195+0000", 2024-09-11T03:43:22.105 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6372999999999995e-05 2024-09-11T03:43:22.105 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.105 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.106 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.106 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937287+0000", 2024-09-11T03:43:22.106 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1434000000000006e-05 2024-09-11T03:43:22.106 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.106 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.106 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:43:22.106 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937454+0000", 2024-09-11T03:43:22.106 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000167262 2024-09-11T03:43:22.106 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.106 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.106 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:22.106 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.943162+0000", 2024-09-11T03:43:22.106 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0057082230000000001 2024-09-11T03:43:22.106 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.107 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.107 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:22.107 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.943196+0000", 2024-09-11T03:43:22.107 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4025e-05 2024-09-11T03:43:22.107 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.107 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.108 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:22.109 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.944922+0000", 2024-09-11T03:43:22.109 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001726102 2024-09-11T03:43:22.109 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.109 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.109 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:22.109 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.944977+0000", 2024-09-11T03:43:22.109 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4849999999999998e-05 2024-09-11T03:43:22.109 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.109 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.109 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.109 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.945031+0000", 2024-09-11T03:43:22.110 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4048000000000002e-05 2024-09-11T03:43:22.110 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.110 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.110 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.110 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.110 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.110 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1175 2.c 2:32ade24e:::smithi17557870-45:head [copy-from ver 130] snapc 2e=[2e,2c,2b,28,26] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:43:22.110 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:43:22.110 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 199.94771948600001, 2024-09-11T03:43:22.110 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015807878000000001, 2024-09-11T03:43:22.110 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.110 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.110 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:22.110 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:22.111 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:22.111 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:22.111 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1175 2024-09-11T03:43:22.111 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.111 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.111 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.111 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.111 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:43:22.111 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.111 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.111 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.112 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.112 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:43:22.112 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.112 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.112 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.112 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.112 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102142+0000", 2024-09-11T03:43:22.112 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3369999999999998e-06 2024-09-11T03:43:22.112 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.112 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.112 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.112 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102152+0000", 2024-09-11T03:43:22.113 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8400000000000007e-06 2024-09-11T03:43:22.113 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.113 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.113 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.113 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102172+0000", 2024-09-11T03:43:22.113 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0537e-05 2024-09-11T03:43:22.113 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.113 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.113 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.113 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102183+0000", 2024-09-11T03:43:22.113 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0423e-05 2024-09-11T03:43:22.113 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.113 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.114 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.114 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102253+0000", 2024-09-11T03:43:22.114 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0528e-05 2024-09-11T03:43:22.114 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.114 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.114 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:43:22.114 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102259+0000", 2024-09-11T03:43:22.114 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1589999999999998e-06 2024-09-11T03:43:22.114 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.114 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.114 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.114 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103477+0000", 2024-09-11T03:43:22.114 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012174550000000001 2024-09-11T03:43:22.114 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.115 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.115 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.115 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103545+0000", 2024-09-11T03:43:22.115 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7595999999999995e-05 2024-09-11T03:43:22.115 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.115 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.115 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:43:22.115 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.110323+0000", 2024-09-11T03:43:22.115 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0067785730000000004 2024-09-11T03:43:22.115 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.115 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.115 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:22.115 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.113411+0000", 2024-09-11T03:43:22.115 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030882769999999999 2024-09-11T03:43:22.116 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.116 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.116 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:22.116 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117742+0000", 2024-09-11T03:43:22.116 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0043301920000000001 2024-09-11T03:43:22.116 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.116 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.116 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:22.116 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117835+0000", 2024-09-11T03:43:22.116 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3848e-05 2024-09-11T03:43:22.116 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.116 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.117 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:22.117 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117916+0000", 2024-09-11T03:43:22.117 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0505000000000003e-05 2024-09-11T03:43:22.117 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.117 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.117 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.117 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117948+0000", 2024-09-11T03:43:22.117 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1608e-05 2024-09-11T03:43:22.117 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.117 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.117 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.117 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.118 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.118 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1176 2.c 2:32ade24e:::smithi17557870-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:43:22.118 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:43:22.118 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 199.94765522200001, 2024-09-11T03:43:22.118 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010847406, 2024-09-11T03:43:22.118 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.118 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.118 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:22.118 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:22.118 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:22.118 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:22.118 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1176 2024-09-11T03:43:22.118 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.119 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.119 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.119 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.119 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:43:22.119 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.119 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.119 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.119 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.119 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:43:22.119 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.119 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.119 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.119 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.119 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:43:22.120 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4999999999999998e-07 2024-09-11T03:43:22.120 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.120 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.120 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.120 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102206+0000", 2024-09-11T03:43:22.120 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9429999999999999e-06 2024-09-11T03:43:22.120 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.120 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.120 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.120 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102214+0000", 2024-09-11T03:43:22.120 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0399999999999993e-06 2024-09-11T03:43:22.120 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.120 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.120 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.121 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102220+0000", 2024-09-11T03:43:22.121 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2989999999999996e-06 2024-09-11T03:43:22.121 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.121 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.121 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.121 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102307+0000", 2024-09-11T03:43:22.121 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7635999999999999e-05 2024-09-11T03:43:22.121 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.121 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.121 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:43:22.121 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102310+0000", 2024-09-11T03:43:22.121 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.2160000000000002e-06 2024-09-11T03:43:22.121 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.122 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.122 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.122 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103661+0000", 2024-09-11T03:43:22.122 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013507860000000001 2024-09-11T03:43:22.122 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.122 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.122 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:22.122 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103686+0000", 2024-09-11T03:43:22.122 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.44e-05 2024-09-11T03:43:22.122 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.122 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.122 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.122 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.112884+0000", 2024-09-11T03:43:22.122 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0091986110000000006 2024-09-11T03:43:22.123 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.123 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.123 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.123 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.112967+0000", 2024-09-11T03:43:22.123 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2478000000000006e-05 2024-09-11T03:43:22.123 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.123 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.123 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.124 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.113051+0000", 2024-09-11T03:43:22.124 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4647000000000005e-05 2024-09-11T03:43:22.124 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.125 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.125 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.125 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.125 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.125 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1719 2.c 2:327469ff:::smithi17557870-49:head [write 487469~563838,stat] snapc 45=[45,44,43,3e,3d,3c,3a] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-11T03:43:22.125 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:45.859743+0000", 2024-09-11T03:43:22.126 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 156.19011644400001, 2024-09-11T03:43:22.126 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0078899869999999993, 2024-09-11T03:43:22.126 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.126 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.127 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:22.127 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:22.127 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:22.127 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:22.127 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1719 2024-09-11T03:43:22.128 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.128 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.128 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.128 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.128 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.859743+0000", 2024-09-11T03:43:22.128 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.129 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.129 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.129 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.129 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.859743+0000", 2024-09-11T03:43:22.129 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.129 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.129 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.129 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.129 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.859745+0000", 2024-09-11T03:43:22.130 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.712e-06 2024-09-11T03:43:22.130 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.130 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.130 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.130 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860066+0000", 2024-09-11T03:43:22.130 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00032013499999999998 2024-09-11T03:43:22.130 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.130 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.130 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.130 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860090+0000", 2024-09-11T03:43:22.130 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4590999999999999e-05 2024-09-11T03:43:22.130 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.130 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.131 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.131 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860102+0000", 2024-09-11T03:43:22.131 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2269999999999999e-05 2024-09-11T03:43:22.131 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.131 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.132 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.132 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860167+0000", 2024-09-11T03:43:22.132 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4806000000000003e-05 2024-09-11T03:43:22.132 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.132 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.132 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:43:22.132 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860172+0000", 2024-09-11T03:43:22.132 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7219999999999999e-06 2024-09-11T03:43:22.132 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.132 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.132 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.132 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.862974+0000", 2024-09-11T03:43:22.132 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0028017480000000002 2024-09-11T03:43:22.133 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.133 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.133 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.133 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.863071+0000", 2024-09-11T03:43:22.133 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7738000000000005e-05 2024-09-11T03:43:22.133 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.133 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.133 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:43:22.133 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.863253+0000", 2024-09-11T03:43:22.133 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00018170999999999999 2024-09-11T03:43:22.133 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.133 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.133 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:22.134 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.864451+0000", 2024-09-11T03:43:22.134 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00119782 2024-09-11T03:43:22.134 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.134 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.134 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:22.134 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.865698+0000", 2024-09-11T03:43:22.134 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001246973 2024-09-11T03:43:22.134 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.134 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.134 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:22.134 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.867563+0000", 2024-09-11T03:43:22.134 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001865189 2024-09-11T03:43:22.134 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.135 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.135 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:22.135 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.867601+0000", 2024-09-11T03:43:22.135 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8333e-05 2024-09-11T03:43:22.135 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.135 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.135 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.135 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.867633+0000", 2024-09-11T03:43:22.135 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1239999999999999e-05 2024-09-11T03:43:22.135 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.135 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.135 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.135 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.136 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.136 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1863 2.c 2:3c007163:::smithi17557870-35:head [read 0~3613505,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:43:22.136 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:01.850063+0000", 2024-09-11T03:43:22.136 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 140.19979636400001, 2024-09-11T03:43:22.136 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0078079589999999997, 2024-09-11T03:43:22.136 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.136 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.136 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:22.136 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:22.136 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:22.136 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:22.136 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1863 2024-09-11T03:43:22.136 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.136 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.137 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.137 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.137 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850063+0000", 2024-09-11T03:43:22.137 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.137 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.137 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.137 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.137 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850063+0000", 2024-09-11T03:43:22.137 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.137 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.137 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.137 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.137 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850066+0000", 2024-09-11T03:43:22.137 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8700000000000001e-06 2024-09-11T03:43:22.138 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.138 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.138 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.138 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850074+0000", 2024-09-11T03:43:22.138 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3380000000000005e-06 2024-09-11T03:43:22.138 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.138 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.138 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.138 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850096+0000", 2024-09-11T03:43:22.138 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1753e-05 2024-09-11T03:43:22.138 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.139 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.139 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.139 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850108+0000", 2024-09-11T03:43:22.139 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2300000000000001e-05 2024-09-11T03:43:22.139 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.139 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.139 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.139 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850235+0000", 2024-09-11T03:43:22.139 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000126604 2024-09-11T03:43:22.139 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.139 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.139 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:43:22.139 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850244+0000", 2024-09-11T03:43:22.140 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.0790000000000007e-06 2024-09-11T03:43:22.140 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.140 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.140 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.140 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.852514+0000", 2024-09-11T03:43:22.140 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022705199999999998 2024-09-11T03:43:22.140 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.140 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.140 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.140 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.852621+0000", 2024-09-11T03:43:22.140 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000106649 2024-09-11T03:43:22.140 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.140 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.140 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.141 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.857871+0000", 2024-09-11T03:43:22.141 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0052498459999999999 2024-09-11T03:43:22.141 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.141 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.141 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.142 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.142 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.142 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1864 2.c 2:3c007163:::smithi17557870-35:head [sparse-read 0~1995320] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:43:22.142 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:43:22.142 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 140.19972588799999, 2024-09-11T03:43:22.142 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0079536510000000008, 2024-09-11T03:43:22.142 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.142 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.142 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:22.142 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:22.143 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:22.143 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:22.143 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1864 2024-09-11T03:43:22.143 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.143 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.143 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.143 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.143 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:43:22.143 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.143 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.143 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.143 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.143 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:43:22.143 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.144 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.144 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.144 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.144 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850134+0000", 2024-09-11T03:43:22.144 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2399999999999998e-07 2024-09-11T03:43:22.144 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.144 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.144 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.144 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850137+0000", 2024-09-11T03:43:22.144 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4010000000000001e-06 2024-09-11T03:43:22.145 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.145 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.145 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.145 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850164+0000", 2024-09-11T03:43:22.145 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6616999999999999e-05 2024-09-11T03:43:22.145 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.145 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.145 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.145 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850171+0000", 2024-09-11T03:43:22.145 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7209999999999997e-06 2024-09-11T03:43:22.145 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.145 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.145 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.146 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850311+0000", 2024-09-11T03:43:22.146 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00014029900000000001 2024-09-11T03:43:22.146 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.146 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:22.147 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:43:22.148 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:43:22.148 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.148 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:43:22.148 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850317+0000", 2024-09-11T03:43:22.148 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7799999999999997e-06 2024-09-11T03:43:22.148 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.148 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.149 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.149 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.857902+0000", 2024-09-11T03:43:22.149 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0075851759999999999 2024-09-11T03:43:22.149 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.149 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.149 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.149 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.857978+0000", 2024-09-11T03:43:22.149 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6272000000000004e-05 2024-09-11T03:43:22.149 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.149 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.149 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.149 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.858087+0000", 2024-09-11T03:43:22.149 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108761 2024-09-11T03:43:22.150 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.150 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.150 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.150 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.150 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.150 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1865 2.c 2:3c007163:::smithi17557870-35:head [sparse-read 0~1995320] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:43:22.150 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:01.850224+0000", 2024-09-11T03:43:22.150 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 140.19963553100001, 2024-09-11T03:43:22.150 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.008147797, 2024-09-11T03:43:22.150 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.150 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.150 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:22.150 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:22.150 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:22.151 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:22.151 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1865 2024-09-11T03:43:22.151 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.151 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.151 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.151 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.151 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850224+0000", 2024-09-11T03:43:22.151 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.151 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.151 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.151 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.151 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850224+0000", 2024-09-11T03:43:22.151 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.151 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.152 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.152 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.152 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850224+0000", 2024-09-11T03:43:22.152 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6099999999999999e-07 2024-09-11T03:43:22.152 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.152 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.152 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.152 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850229+0000", 2024-09-11T03:43:22.152 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5449999999999997e-06 2024-09-11T03:43:22.153 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.153 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.153 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.153 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850244+0000", 2024-09-11T03:43:22.153 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5048999999999999e-05 2024-09-11T03:43:22.153 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.153 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.153 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.153 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850258+0000", 2024-09-11T03:43:22.153 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3505e-05 2024-09-11T03:43:22.153 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.153 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.153 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.154 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850344+0000", 2024-09-11T03:43:22.154 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6329999999999995e-05 2024-09-11T03:43:22.154 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.154 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.154 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:43:22.154 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850349+0000", 2024-09-11T03:43:22.154 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0610000000000003e-06 2024-09-11T03:43:22.154 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.154 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.154 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.154 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.858124+0000", 2024-09-11T03:43:22.154 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.007774792 2024-09-11T03:43:22.155 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.155 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.155 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.155 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.858236+0000", 2024-09-11T03:43:22.155 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011264000000000001 2024-09-11T03:43:22.155 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.155 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.155 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.155 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.858371+0000", 2024-09-11T03:43:22.155 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013501400000000001 2024-09-11T03:43:22.155 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.155 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.155 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.155 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.156 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.156 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:1868 2.0 e245/188 2:0762776a:::smithi17557870-25:head v 245'39, mlcod=242'35)", 2024-09-11T03:43:22.156 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:01.863124+0000", 2024-09-11T03:43:22.156 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 140.18673558500001, 2024-09-11T03:43:22.156 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0073720210000000003, 2024-09-11T03:43:22.156 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.156 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.156 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:22.156 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.156 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.156 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.156 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.863124+0000", 2024-09-11T03:43:22.156 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.157 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.157 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.157 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.157 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.863124+0000", 2024-09-11T03:43:22.157 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.157 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.157 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.157 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.157 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.863124+0000", 2024-09-11T03:43:22.157 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.91e-07 2024-09-11T03:43:22.157 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.157 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.157 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.157 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.868459+0000", 2024-09-11T03:43:22.158 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0053354500000000003 2024-09-11T03:43:22.158 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.158 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.158 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.158 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.868473+0000", 2024-09-11T03:43:22.158 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3930999999999999e-05 2024-09-11T03:43:22.158 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.158 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.158 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.158 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.868481+0000", 2024-09-11T03:43:22.158 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9030000000000004e-06 2024-09-11T03:43:22.158 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.158 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.158 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.159 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.868545+0000", 2024-09-11T03:43:22.159 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3975999999999999e-05 2024-09-11T03:43:22.159 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.159 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.159 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.159 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.868565+0000", 2024-09-11T03:43:22.159 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9551e-05 2024-09-11T03:43:22.159 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.159 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.159 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:22.159 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.870455+0000", 2024-09-11T03:43:22.159 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001890624 2024-09-11T03:43:22.159 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.159 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.160 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.160 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.870496+0000", 2024-09-11T03:43:22.160 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0295e-05 2024-09-11T03:43:22.160 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.160 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.160 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.160 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.160 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.160 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2140 2.c 2:3c007163:::smithi17557870-35:head [copy-from ver 101] snapc 57=[57,56,51,4e] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-11T03:43:22.160 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:25.064188+0000", 2024-09-11T03:43:22.160 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 116.98567106500001, 2024-09-11T03:43:22.160 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013777315, 2024-09-11T03:43:22.160 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.161 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.161 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:22.161 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:22.161 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:22.161 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:22.161 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2140 2024-09-11T03:43:22.161 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.161 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.161 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.161 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.161 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064188+0000", 2024-09-11T03:43:22.161 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.161 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.161 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.162 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.162 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064188+0000", 2024-09-11T03:43:22.162 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.162 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.162 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.162 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.162 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064190+0000", 2024-09-11T03:43:22.162 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9700000000000002e-06 2024-09-11T03:43:22.162 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.162 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.163 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.163 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064198+0000", 2024-09-11T03:43:22.163 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3610000000000001e-06 2024-09-11T03:43:22.163 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.163 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.163 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.163 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064216+0000", 2024-09-11T03:43:22.163 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7768999999999999e-05 2024-09-11T03:43:22.163 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.163 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.163 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.163 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064225+0000", 2024-09-11T03:43:22.163 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7099999999999996e-06 2024-09-11T03:43:22.164 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.164 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.164 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.164 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064291+0000", 2024-09-11T03:43:22.164 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5634000000000002e-05 2024-09-11T03:43:22.164 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.164 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.164 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.164 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064367+0000", 2024-09-11T03:43:22.164 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6687999999999995e-05 2024-09-11T03:43:22.164 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.164 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.164 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:43:22.165 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.066720+0000", 2024-09-11T03:43:22.165 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023526010000000002 2024-09-11T03:43:22.165 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.165 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.165 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:22.165 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.069243+0000", 2024-09-11T03:43:22.165 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025227320000000002 2024-09-11T03:43:22.165 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.165 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.165 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:22.166 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.073940+0000", 2024-09-11T03:43:22.166 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0046973930000000002 2024-09-11T03:43:22.166 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.166 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.166 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:22.166 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.077850+0000", 2024-09-11T03:43:22.166 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039098279999999997 2024-09-11T03:43:22.166 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.166 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.166 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:22.166 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.077915+0000", 2024-09-11T03:43:22.166 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5160000000000006e-05 2024-09-11T03:43:22.166 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.167 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.167 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.167 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.077965+0000", 2024-09-11T03:43:22.167 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0469000000000001e-05 2024-09-11T03:43:22.167 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.167 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.167 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.167 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.167 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.167 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:2421 2.18 e294/281 2:1bdf1a36:::smithi17557870-36:head v 294'198, mlcod=0'0)", 2024-09-11T03:43:22.167 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:50.964269+0000", 2024-09-11T03:43:22.167 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 91.085590166000003, 2024-09-11T03:43:22.167 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013043020000000001, 2024-09-11T03:43:22.168 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.168 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.168 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:22.168 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.168 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.168 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.168 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964269+0000", 2024-09-11T03:43:22.168 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.168 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.168 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.168 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.168 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964269+0000", 2024-09-11T03:43:22.168 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.168 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.169 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.169 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.169 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964270+0000", 2024-09-11T03:43:22.169 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3510000000000001e-06 2024-09-11T03:43:22.169 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.169 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.169 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.169 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966680+0000", 2024-09-11T03:43:22.169 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002409698 2024-09-11T03:43:22.169 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.169 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.169 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.169 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966720+0000", 2024-09-11T03:43:22.170 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0265999999999997e-05 2024-09-11T03:43:22.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.170 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.170 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.170 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966738+0000", 2024-09-11T03:43:22.170 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7540000000000001e-05 2024-09-11T03:43:22.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.170 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.170 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.170 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966835+0000", 2024-09-11T03:43:22.170 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7059999999999996e-05 2024-09-11T03:43:22.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.170 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.170 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.171 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966862+0000", 2024-09-11T03:43:22.171 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7107e-05 2024-09-11T03:43:22.171 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.171 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.171 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:22.171 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977265+0000", 2024-09-11T03:43:22.171 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.01040258 2024-09-11T03:43:22.171 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.171 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.171 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.171 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977312+0000", 2024-09-11T03:43:22.171 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7417999999999999e-05 2024-09-11T03:43:22.171 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.172 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.172 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.172 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.172 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.172 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:2443 2.12 e296/281 2:494ba932:::smithi17557870-43:head v 296'131, mlcod=291'128)", 2024-09-11T03:43:22.172 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:52.895921+0000", 2024-09-11T03:43:22.172 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 89.153937744000004, 2024-09-11T03:43:22.172 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0084055459999999999, 2024-09-11T03:43:22.172 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.172 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.172 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:22.172 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.172 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.895921+0000", 2024-09-11T03:43:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.895921+0000", 2024-09-11T03:43:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.895922+0000", 2024-09-11T03:43:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1699999999999998e-07 2024-09-11T03:43:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.174 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.174 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.174 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902380+0000", 2024-09-11T03:43:22.174 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0064576629999999998 2024-09-11T03:43:22.174 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.174 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.174 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.174 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902391+0000", 2024-09-11T03:43:22.174 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1246999999999999e-05 2024-09-11T03:43:22.174 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.175 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.175 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.175 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902398+0000", 2024-09-11T03:43:22.175 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7560000000000002e-06 2024-09-11T03:43:22.175 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.175 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.175 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.175 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902482+0000", 2024-09-11T03:43:22.175 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4881000000000003e-05 2024-09-11T03:43:22.175 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.175 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.175 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.175 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902500+0000", 2024-09-11T03:43:22.175 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7464000000000001e-05 2024-09-11T03:43:22.176 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.176 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.176 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:22.176 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.904289+0000", 2024-09-11T03:43:22.176 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0017895599999999999 2024-09-11T03:43:22.176 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.176 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.176 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.176 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.904327+0000", 2024-09-11T03:43:22.176 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7457999999999999e-05 2024-09-11T03:43:22.176 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.176 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.176 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.176 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.177 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.177 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2972 2.c 2:32ade24e:::smithi17557870-45:head [copy-from ver 171] snapc 80=[80,7b,7a,79,77,75] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-11T03:43:22.177 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:43:22.177 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 36.975765191000001, 2024-09-11T03:43:22.177 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021036193000000002, 2024-09-11T03:43:22.177 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.177 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.177 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:22.177 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:22.177 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:22.177 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:22.177 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2972 2024-09-11T03:43:22.177 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.178 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.178 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.178 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.178 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:43:22.178 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.178 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.178 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.178 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.178 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:43:22.178 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.178 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.178 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.178 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.179 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074096+0000", 2024-09-11T03:43:22.179 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.685e-06 2024-09-11T03:43:22.179 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.179 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.179 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.179 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074104+0000", 2024-09-11T03:43:22.179 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9950000000000005e-06 2024-09-11T03:43:22.179 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.179 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.179 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.179 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074123+0000", 2024-09-11T03:43:22.179 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9587999999999999e-05 2024-09-11T03:43:22.179 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.180 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.180 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.180 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074132+0000", 2024-09-11T03:43:22.180 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1579999999999994e-06 2024-09-11T03:43:22.180 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.180 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.180 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.180 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074204+0000", 2024-09-11T03:43:22.180 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2072999999999999e-05 2024-09-11T03:43:22.180 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.180 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.180 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.180 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074302+0000", 2024-09-11T03:43:22.181 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7706000000000003e-05 2024-09-11T03:43:22.181 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.181 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.181 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:43:22.181 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.087397+0000", 2024-09-11T03:43:22.181 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013095292999999999 2024-09-11T03:43:22.181 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.181 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.181 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:43:22.181 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090766+0000", 2024-09-11T03:43:22.181 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0033682109999999999 2024-09-11T03:43:22.181 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.181 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.182 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:22.182 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.094977+0000", 2024-09-11T03:43:22.182 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0042108759999999997 2024-09-11T03:43:22.182 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.182 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.182 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:22.182 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095061+0000", 2024-09-11T03:43:22.182 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4827999999999995e-05 2024-09-11T03:43:22.182 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.182 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.182 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:43:22.182 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095099+0000", 2024-09-11T03:43:22.182 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7546999999999998e-05 2024-09-11T03:43:22.183 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.183 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.183 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.183 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095130+0000", 2024-09-11T03:43:22.183 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1232999999999997e-05 2024-09-11T03:43:22.183 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.183 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.183 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.183 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.183 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.183 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2973 2.c 2:32ade24e:::smithi17557870-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e348)", 2024-09-11T03:43:22.183 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:43:22.183 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 36.975700072999999, 2024-09-11T03:43:22.183 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016062098, 2024-09-11T03:43:22.184 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.184 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.184 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:22.184 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:22.184 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:22.184 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:22.184 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2973 2024-09-11T03:43:22.184 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.184 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.184 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.184 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.184 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:43:22.184 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.184 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.185 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.185 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.185 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:43:22.185 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.185 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.185 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.185 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.185 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074160+0000", 2024-09-11T03:43:22.185 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.46e-07 2024-09-11T03:43:22.185 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.185 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.185 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.185 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074162+0000", 2024-09-11T03:43:22.186 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5220000000000002e-06 2024-09-11T03:43:22.186 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.186 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.186 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.186 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074178+0000", 2024-09-11T03:43:22.186 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.609e-05 2024-09-11T03:43:22.186 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.186 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.186 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.186 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074183+0000", 2024-09-11T03:43:22.186 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8280000000000002e-06 2024-09-11T03:43:22.186 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.186 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.186 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.187 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074428+0000", 2024-09-11T03:43:22.187 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00024529599999999999 2024-09-11T03:43:22.187 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.187 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.187 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:22.187 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074469+0000", 2024-09-11T03:43:22.187 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0463000000000002e-05 2024-09-11T03:43:22.187 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.187 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.187 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.187 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090093+0000", 2024-09-11T03:43:22.187 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015624449 2024-09-11T03:43:22.187 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.187 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.188 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.188 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090155+0000", 2024-09-11T03:43:22.188 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1618999999999994e-05 2024-09-11T03:43:22.188 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.188 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.188 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.188 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090221+0000", 2024-09-11T03:43:22.188 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6384999999999995e-05 2024-09-11T03:43:22.188 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.188 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.188 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.188 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.188 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.189 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:3071 2.c 2:3c007163:::smithi17557870-35:head [read 0~2907677,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e356)", 2024-09-11T03:43:22.189 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:53.496172+0000", 2024-09-11T03:43:22.189 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 28.553686776999999, 2024-09-11T03:43:22.189 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.007229358, 2024-09-11T03:43:22.189 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:43:22.189 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:43:22.189 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:43:22.189 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:43:22.189 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:43:22.189 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:22.189 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 3071 2024-09-11T03:43:22.189 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.189 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:43:22.189 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.190 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:43:22.190 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.496172+0000", 2024-09-11T03:43:22.190 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.190 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.190 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.190 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:43:22.190 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.496172+0000", 2024-09-11T03:43:22.190 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:43:22.190 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.190 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.190 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:43:22.190 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.496173+0000", 2024-09-11T03:43:22.190 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9900000000000001e-07 2024-09-11T03:43:22.190 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.191 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.191 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:43:22.191 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.496175+0000", 2024-09-11T03:43:22.191 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3580000000000001e-06 2024-09-11T03:43:22.191 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.191 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.191 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:43:22.191 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.496181+0000", 2024-09-11T03:43:22.191 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8470000000000001e-06 2024-09-11T03:43:22.191 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.191 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.191 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:43:22.191 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.496185+0000", 2024-09-11T03:43:22.192 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3970000000000004e-06 2024-09-11T03:43:22.192 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.192 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.192 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:43:22.192 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.502331+0000", 2024-09-11T03:43:22.192 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0061454350000000003 2024-09-11T03:43:22.192 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.192 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.192 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:43:22.192 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.502402+0000", 2024-09-11T03:43:22.192 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0749999999999999e-05 2024-09-11T03:43:22.192 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:43:22.192 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:43:22.193 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:43:22.193 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:53.503402+0000", 2024-09-11T03:43:22.193 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0010001719999999999 2024-09-11T03:43:22.193 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.193 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.193 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.193 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:43:22.193 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:43:22.193 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:43:22.193 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:43:22.194 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:43:22.194 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:43:22.194 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:43:22.194 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:43:22.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:43:22.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:43:22.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:43:22.196 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:43:22.196 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:43:22.197 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:21 smithi121 ceph-mon[30729]: pgmap v729: 57 pgs: 57 active+clean; 135 MiB data, 679 MiB used, 893 GiB / 894 GiB avail; 255 B/s rd, 1.8 MiB/s wr, 3 op/s 2024-09-11T03:43:22.197 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:21 smithi121 ceph-mon[30729]: osdmap e383: 12 total, 12 up, 10 in 2024-09-11T03:43:22.197 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:21 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:22.197 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:21 smithi121 ceph-mon[30729]: osdmap e384: 12 total, 12 up, 10 in 2024-09-11T03:43:22.197 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:43:22.197 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:43:22.217 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:22.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:22.319 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:22.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:22.341 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:22.341 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:22.442 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:22.465 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:22.466 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:22.567 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:22.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:22.589 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:22.589 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:22.691 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:22.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:22.712 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:22.712 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:22.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:22.840 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T03:43:22.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:22.841 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:22.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:22.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:43:22.924 INFO:tasks.rados.rados.2.smithi175.stdout:1403: done (5 left) 2024-09-11T03:43:22.924 INFO:tasks.rados.rados.2.smithi175.stdout:1404: read oid 8 snap 142 2024-09-11T03:43:22.924 INFO:tasks.rados.rados.2.smithi175.stdout:1404: expect (ObjNum 415 snap 127 seq_num 415) 2024-09-11T03:43:22.924 INFO:tasks.rados.rados.2.smithi175.stdout:1405: read oid 45 snap -1 2024-09-11T03:43:22.924 INFO:tasks.rados.rados.2.smithi175.stdout:1405: expect (ObjNum 416 snap 127 seq_num 416) 2024-09-11T03:43:22.924 INFO:tasks.rados.rados.2.smithi175.stdout:1406: read oid 6 snap -1 2024-09-11T03:43:22.924 INFO:tasks.rados.rados.2.smithi175.stdout:1406: expect (ObjNum 369 snap 113 seq_num 369) 2024-09-11T03:43:22.924 INFO:tasks.rados.rados.2.smithi175.stdout:1407: read oid 24 snap -1 2024-09-11T03:43:22.924 INFO:tasks.rados.rados.2.smithi175.stdout:1407: expect (ObjNum 413 snap 127 seq_num 413) 2024-09-11T03:43:22.925 INFO:tasks.rados.rados.2.smithi175.stdout:1408: delete oid 34 current snap is 145 2024-09-11T03:43:22.925 INFO:tasks.rados.rados.2.smithi175.stdout:1398: finishing copy_from to smithi17557870-20 2024-09-11T03:43:22.925 INFO:tasks.rados.rados.2.smithi175.stdout:1398: got expected ENOENT (src dne) 2024-09-11T03:43:22.925 INFO:tasks.rados.rados.2.smithi175.stdout:1398: finishing copy_from racing read to smithi17557870-20 2024-09-11T03:43:22.925 INFO:tasks.rados.rados.2.smithi175.stdout:1399: expect (ObjNum 497 snap 144 seq_num 497) 2024-09-11T03:43:22.942 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:22.963 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:22.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:22.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:22.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:22 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4113620044' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:23.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:23.086 INFO:tasks.rados.rados.2.smithi175.stdout:1400: finishing copy_from racing read to smithi17557870-37 2024-09-11T03:43:23.086 INFO:tasks.rados.rados.2.smithi175.stdout:1402: finishing write tid 1 to smithi17557870-13 2024-09-11T03:43:23.086 INFO:tasks.rados.rados.2.smithi175.stdout:1400: finishing copy_from to smithi17557870-37 2024-09-11T03:43:23.086 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 362 (ObjNum 483 snap 140 seq_num 483) dirty exists 2024-09-11T03:43:23.086 INFO:tasks.rados.rados.2.smithi175.stdout:1402: finishing write tid 2 to smithi17557870-13 2024-09-11T03:43:23.086 INFO:tasks.rados.rados.2.smithi175.stdout:1402: finishing write tid 3 to smithi17557870-13 2024-09-11T03:43:23.086 INFO:tasks.rados.rados.2.smithi175.stdout:1402: finishing write tid 4 to smithi17557870-13 2024-09-11T03:43:23.086 INFO:tasks.rados.rados.2.smithi175.stdout:1402: oid 13 updating version 0 to 485 2024-09-11T03:43:23.086 INFO:tasks.rados.rados.2.smithi175.stdout:1402: oid 13 updating version 485 to 486 2024-09-11T03:43:23.086 INFO:tasks.rados.rados.2.smithi175.stdout:1402: oid 13 updating version 486 to 487 2024-09-11T03:43:23.087 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 487 (ObjNum 499 snap 145 seq_num 499) dirty exists 2024-09-11T03:43:23.087 INFO:tasks.rados.rados.2.smithi175.stdout:1402: left oid 13 (ObjNum 499 snap 145 seq_num 499) 2024-09-11T03:43:23.087 INFO:tasks.rados.rados.2.smithi175.stdout:1405: expect (ObjNum 416 snap 127 seq_num 416) 2024-09-11T03:43:23.091 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:23.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:23.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:23.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:22 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4113620044' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:23.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:22 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4113620044' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:23.193 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:23.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:23.215 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:23.215 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:23.257 INFO:tasks.rados.rados.2.smithi175.stdout:1406: expect (ObjNum 369 snap 113 seq_num 369) 2024-09-11T03:43:23.317 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:23.338 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:23.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:23.339 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:23.339 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:23.343 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:23.389 INFO:tasks.rados.rados.2.smithi175.stdout:1404: expect (ObjNum 415 snap 127 seq_num 415) 2024-09-11T03:43:23.440 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:23.465 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:23.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:23.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:23.568 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:23.569 INFO:tasks.rados.rados.2.smithi175.stdout:1407: expect (ObjNum 413 snap 127 seq_num 413) 2024-09-11T03:43:23.589 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:23.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:23.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:23.691 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:23.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:23.714 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:23.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:23.785 INFO:tasks.rados.rados.2.smithi175.stdout:1398: done (9 left) 2024-09-11T03:43:23.785 INFO:tasks.rados.rados.2.smithi175.stdout:1399: done (8 left) 2024-09-11T03:43:23.785 INFO:tasks.rados.rados.2.smithi175.stdout:1400: done (7 left) 2024-09-11T03:43:23.785 INFO:tasks.rados.rados.2.smithi175.stdout:1401: done (6 left) 2024-09-11T03:43:23.785 INFO:tasks.rados.rados.2.smithi175.stdout:1402: done (5 left) 2024-09-11T03:43:23.785 INFO:tasks.rados.rados.2.smithi175.stdout:1404: done (4 left) 2024-09-11T03:43:23.785 INFO:tasks.rados.rados.2.smithi175.stdout:1405: done (3 left) 2024-09-11T03:43:23.785 INFO:tasks.rados.rados.2.smithi175.stdout:1406: done (2 left) 2024-09-11T03:43:23.785 INFO:tasks.rados.rados.2.smithi175.stdout:1407: done (1 left) 2024-09-11T03:43:23.785 INFO:tasks.rados.rados.2.smithi175.stdout:1408: done (0 left) 2024-09-11T03:43:23.785 INFO:tasks.rados.rados.2.smithi175.stdout:1409: read oid 23 snap -1 2024-09-11T03:43:23.785 INFO:tasks.rados.rados.2.smithi175.stdout:1409: expect deleted 2024-09-11T03:43:23.786 INFO:tasks.rados.rados.2.smithi175.stdout:1410: snap_remove snap 138 2024-09-11T03:43:23.815 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:23.837 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:23.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:23.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:23.926 INFO:tasks.rados.rados.2.smithi175.stdout:1410: done (1 left) 2024-09-11T03:43:23.926 INFO:tasks.rados.rados.2.smithi175.stdout:1411: read oid 17 snap -1 2024-09-11T03:43:23.926 INFO:tasks.rados.rados.2.smithi175.stdout:1411: expect (ObjNum 425 snap 129 seq_num 425) 2024-09-11T03:43:23.926 INFO:tasks.rados.rados.2.smithi175.stdout:1412: read oid 20 snap -1 2024-09-11T03:43:23.927 INFO:tasks.rados.rados.2.smithi175.stdout:1412: expect deleted 2024-09-11T03:43:23.927 INFO:tasks.rados.rados.2.smithi175.stdout:1413: rollback oid 11 current snap is 145 2024-09-11T03:43:23.927 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 11 to 142 2024-09-11T03:43:23.927 INFO:tasks.rados.rados.2.smithi175.stdout:1414: write (excl) oid 18 current snap is 145 2024-09-11T03:43:23.927 INFO:tasks.rados.rados.2.smithi175.stdout:1414: seq_num 501 ranges {420307=462694,1351008=617495} 2024-09-11T03:43:23.931 INFO:tasks.rados.rados.2.smithi175.stdout:1414: writing smithi17557870-18 from 420307 to 883001 tid 1 2024-09-11T03:43:23.937 INFO:tasks.rados.rados.2.smithi175.stdout:1414: writing smithi17557870-18 from 1351008 to 1968503 tid 2 2024-09-11T03:43:23.937 INFO:tasks.rados.rados.2.smithi175.stdout:1415: rollback oid 12 current snap is 145 2024-09-11T03:43:23.937 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 12 to 137 2024-09-11T03:43:23.938 INFO:tasks.rados.rados.2.smithi175.stdout:1413: finishing rollback tid 1 to smithi17557870-11 2024-09-11T03:43:23.938 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 427 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-11T03:43:23.938 INFO:tasks.rados.rados.2.smithi175.stdout:1414: finishing write tid 1 to smithi17557870-18 2024-09-11T03:43:23.938 INFO:tasks.rados.rados.2.smithi175.stdout:1409: done (5 left) 2024-09-11T03:43:23.938 INFO:tasks.rados.rados.2.smithi175.stdout:1413: done (4 left) 2024-09-11T03:43:23.938 INFO:tasks.rados.rados.2.smithi175.stdout:1416: delete oid 21 current snap is 145 2024-09-11T03:43:23.938 INFO:tasks.rados.rados.2.smithi175.stdout:1411: expect (ObjNum 425 snap 129 seq_num 425) 2024-09-11T03:43:23.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:23.961 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:23.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:24.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:24.087 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:24.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:24.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:24.109 INFO:tasks.rados.rados.2.smithi175.stdout:1415: finishing rollback tid 0 to smithi17557870-12 2024-09-11T03:43:24.109 INFO:tasks.rados.rados.2.smithi175.stdout:1415: finishing rollback tid 1 to smithi17557870-12 2024-09-11T03:43:24.109 INFO:tasks.rados.rados.2.smithi175.stdout:1411: done (4 left) 2024-09-11T03:43:24.110 INFO:tasks.rados.rados.2.smithi175.stdout:1412: done (3 left) 2024-09-11T03:43:24.110 INFO:tasks.rados.rados.2.smithi175.stdout:1416: done (2 left) 2024-09-11T03:43:24.110 INFO:tasks.rados.rados.2.smithi175.stdout:1417: snap_create 2024-09-11T03:43:24.110 INFO:tasks.rados.rados.2.smithi175.stdout:1415: finishing rollback tid 2 to smithi17557870-12 2024-09-11T03:43:24.110 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 493 (ObjNum 440 snap 130 seq_num 440) dirty exists 2024-09-11T03:43:24.110 INFO:tasks.rados.rados.2.smithi175.stdout:1414: finishing write tid 2 to smithi17557870-18 2024-09-11T03:43:24.110 INFO:tasks.rados.rados.2.smithi175.stdout:1414: finishing write tid 3 to smithi17557870-18 2024-09-11T03:43:24.110 INFO:tasks.rados.rados.2.smithi175.stdout:1414: finishing write tid 4 to smithi17557870-18 2024-09-11T03:43:24.110 INFO:tasks.rados.rados.2.smithi175.stdout:1414: oid 18 updating version 0 to 261 2024-09-11T03:43:24.110 INFO:tasks.rados.rados.2.smithi175.stdout:1414: oid 18 version 261 is already newer than 260 2024-09-11T03:43:24.110 INFO:tasks.rados.rados.2.smithi175.stdout:1414: oid 18 version 261 is already newer than 259 2024-09-11T03:43:24.110 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 261 (ObjNum 501 snap 145 seq_num 501) dirty exists 2024-09-11T03:43:24.110 INFO:tasks.rados.rados.2.smithi175.stdout:1414: left oid 18 (ObjNum 501 snap 145 seq_num 501) 2024-09-11T03:43:24.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:23 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:43:24.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:23 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:43:24.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:23 smithi084 ceph-mon[27254]: pgmap v732: 57 pgs: 57 active+clean; 149 MiB data, 695 MiB used, 893 GiB / 894 GiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-11T03:43:24.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:23 smithi084 ceph-mon[27254]: osdmap e385: 12 total, 12 up, 10 in 2024-09-11T03:43:24.189 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:24.209 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:24.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:24.210 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:23 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:43:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:23 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:43:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:23 smithi175 ceph-mon[30894]: pgmap v732: 57 pgs: 57 active+clean; 149 MiB data, 695 MiB used, 893 GiB / 894 GiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-11T03:43:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:23 smithi175 ceph-mon[30894]: osdmap e385: 12 total, 12 up, 10 in 2024-09-11T03:43:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:43:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:43:24.312 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:24.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:24.334 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:24.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:23 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:43:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:23 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:43:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:23 smithi121 ceph-mon[30729]: pgmap v732: 57 pgs: 57 active+clean; 149 MiB data, 695 MiB used, 893 GiB / 894 GiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-11T03:43:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:23 smithi121 ceph-mon[30729]: osdmap e385: 12 total, 12 up, 10 in 2024-09-11T03:43:24.435 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:24.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:24.457 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:24.457 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:24.558 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:24.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:24.580 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:24.580 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:24.681 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:24.702 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:24.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:24.703 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:24.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:24.827 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:24.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:24.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:24.929 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:24.951 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:24.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:24.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:25.019 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T03:43:25.053 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:25.084 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:25.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:25.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:25.134 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:43:25.138 INFO:tasks.rados.rados.2.smithi175.stdout:1414: done (2 left) 2024-09-11T03:43:25.138 INFO:tasks.rados.rados.2.smithi175.stdout:1415: done (1 left) 2024-09-11T03:43:25.138 INFO:tasks.rados.rados.2.smithi175.stdout:1417: done (0 left) 2024-09-11T03:43:25.138 INFO:tasks.rados.rados.2.smithi175.stdout:1418: delete oid 14 current snap is 146 2024-09-11T03:43:25.142 INFO:tasks.rados.rados.2.smithi175.stdout:1418: done (0 left) 2024-09-11T03:43:25.142 INFO:tasks.rados.rados.2.smithi175.stdout:1419: copy_from oid 1 from oid 4 current snap is 146 2024-09-11T03:43:25.142 INFO:tasks.rados.rados.2.smithi175.stdout:1420: write oid 7 current snap is 146 2024-09-11T03:43:25.142 INFO:tasks.rados.rados.2.smithi175.stdout:1420: seq_num 504 ranges {470728=404794,1526454=481175} 2024-09-11T03:43:25.147 INFO:tasks.rados.rados.2.smithi175.stdout:1420: writing smithi17557870-7 from 470728 to 875522 tid 1 2024-09-11T03:43:25.153 INFO:tasks.rados.rados.2.smithi175.stdout:1420: writing smithi17557870-7 from 1526454 to 2007629 tid 2 2024-09-11T03:43:25.154 INFO:tasks.rados.rados.2.smithi175.stdout:1421: read oid 28 snap -1 2024-09-11T03:43:25.154 INFO:tasks.rados.rados.2.smithi175.stdout:1421: expect deleted 2024-09-11T03:43:25.154 INFO:tasks.rados.rados.2.smithi175.stdout:1422: snap_remove snap 145 2024-09-11T03:43:25.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:25.211 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:25.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:25.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:25.275 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:43:25.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:25 smithi084 ceph-mon[27254]: 2.10 scrub starts 2024-09-11T03:43:25.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:25 smithi084 ceph-mon[27254]: 2.10 scrub ok 2024-09-11T03:43:25.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:25 smithi084 ceph-mon[27254]: 1.2 scrub starts 2024-09-11T03:43:25.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:25 smithi084 ceph-mon[27254]: 1.2 scrub ok 2024-09-11T03:43:25.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:25 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:43:25.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:25 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:43:25.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:25 smithi084 ceph-mon[27254]: osdmap e386: 12 total, 12 up, 10 in 2024-09-11T03:43:25.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:25 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1695957281' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:25.313 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:25.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:25.334 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:25.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:25 smithi121 ceph-mon[30729]: 2.10 scrub starts 2024-09-11T03:43:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:25 smithi121 ceph-mon[30729]: 2.10 scrub ok 2024-09-11T03:43:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:25 smithi121 ceph-mon[30729]: 1.2 scrub starts 2024-09-11T03:43:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:25 smithi121 ceph-mon[30729]: 1.2 scrub ok 2024-09-11T03:43:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:25 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:43:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:25 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:43:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:25 smithi121 ceph-mon[30729]: osdmap e386: 12 total, 12 up, 10 in 2024-09-11T03:43:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:25 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1695957281' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:25.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:25.460 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:25.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:25.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:25 smithi175 ceph-mon[30894]: 2.10 scrub starts 2024-09-11T03:43:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:25 smithi175 ceph-mon[30894]: 2.10 scrub ok 2024-09-11T03:43:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:25 smithi175 ceph-mon[30894]: 1.2 scrub starts 2024-09-11T03:43:25.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:25 smithi175 ceph-mon[30894]: 1.2 scrub ok 2024-09-11T03:43:25.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:25 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:43:25.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:25 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:43:25.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:25 smithi175 ceph-mon[30894]: osdmap e386: 12 total, 12 up, 10 in 2024-09-11T03:43:25.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:25 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1695957281' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:25.562 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:25.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:25.585 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:25.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:25.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:25.712 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:25.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:25.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:25.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:25.839 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:25.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:25.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:25.883 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:25.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:25.967 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:25.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:25.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:26.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:26.094 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:26.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:26.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:26.149 INFO:tasks.rados.rados.2.smithi175.stdout:1422: done (3 left) 2024-09-11T03:43:26.149 INFO:tasks.rados.rados.2.smithi175.stdout:1423: rollback oid 26 current snap is 146 2024-09-11T03:43:26.149 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 26 to 142 2024-09-11T03:43:26.149 INFO:tasks.rados.rados.2.smithi175.stdout:1419: finishing copy_from to smithi17557870-1 2024-09-11T03:43:26.149 INFO:tasks.rados.rados.2.smithi175.stdout:1419: got expected ENOENT (src dne) 2024-09-11T03:43:26.149 INFO:tasks.rados.rados.2.smithi175.stdout:1419: finishing copy_from racing read to smithi17557870-1 2024-09-11T03:43:26.149 INFO:tasks.rados.rados.2.smithi175.stdout:1420: finishing write tid 1 to smithi17557870-7 2024-09-11T03:43:26.149 INFO:tasks.rados.rados.2.smithi175.stdout:1420: finishing write tid 2 to smithi17557870-7 2024-09-11T03:43:26.149 INFO:tasks.rados.rados.2.smithi175.stdout:1420: finishing write tid 3 to smithi17557870-7 2024-09-11T03:43:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:1420: finishing write tid 4 to smithi17557870-7 2024-09-11T03:43:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:1420: oid 7 updating version 0 to 462 2024-09-11T03:43:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:1420: oid 7 version 462 is already newer than 460 2024-09-11T03:43:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:1420: oid 7 version 462 is already newer than 461 2024-09-11T03:43:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 462 (ObjNum 504 snap 146 seq_num 504) dirty exists 2024-09-11T03:43:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:1420: left oid 7 (ObjNum 504 snap 146 seq_num 504) 2024-09-11T03:43:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:1419: done (3 left) 2024-09-11T03:43:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:1420: done (2 left) 2024-09-11T03:43:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:1421: done (1 left) 2024-09-11T03:43:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:1424: write (excl) oid 24 current snap is 146 2024-09-11T03:43:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:1424: seq_num 505 ranges {487535=480043,1584936=624833} 2024-09-11T03:43:26.153 INFO:tasks.rados.rados.2.smithi175.stdout:1424: writing smithi17557870-24 from 487535 to 967578 tid 1 2024-09-11T03:43:26.160 INFO:tasks.rados.rados.2.smithi175.stdout:1424: writing smithi17557870-24 from 1584936 to 2209769 tid 2 2024-09-11T03:43:26.160 INFO:tasks.rados.rados.2.smithi175.stdout:1425: snap_remove snap 137 2024-09-11T03:43:26.196 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:26.218 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:26.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:26.319 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:26.340 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:26.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:26.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:26.381 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:43:26.381 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:43:26.393 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:26 smithi084 ceph-mon[27254]: pgmap v735: 57 pgs: 57 active+clean; 150 MiB data, 702 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 2.1 MiB/s wr, 19 op/s 2024-09-11T03:43:26.393 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:26 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:43:26.393 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:26 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:43:26.393 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1695957281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:26.393 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:26 smithi084 ceph-mon[27254]: osdmap e387: 12 total, 12 up, 10 in 2024-09-11T03:43:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:26 smithi121 ceph-mon[30729]: pgmap v735: 57 pgs: 57 active+clean; 150 MiB data, 702 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 2.1 MiB/s wr, 19 op/s 2024-09-11T03:43:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:26 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:43:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:26 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:43:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1695957281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:26 smithi121 ceph-mon[30729]: osdmap e387: 12 total, 12 up, 10 in 2024-09-11T03:43:26.424 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [6, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:43:26.424 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:43:26.425 INFO:tasks.thrashosds.thrasher:inject_pause on osd.11 2024-09-11T03:43:26.425 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T03:43:26.425 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T03:43:26.425 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 config set filestore_inject_stall 3 2024-09-11T03:43:26.442 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:26.452 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:26 smithi175 ceph-mon[30894]: pgmap v735: 57 pgs: 57 active+clean; 150 MiB data, 702 MiB used, 893 GiB / 894 GiB avail; 12 MiB/s rd, 2.1 MiB/s wr, 19 op/s 2024-09-11T03:43:26.452 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:26 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:43:26.452 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:26 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:43:26.453 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1695957281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:26.453 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:26 smithi175 ceph-mon[30894]: osdmap e387: 12 total, 12 up, 10 in 2024-09-11T03:43:26.463 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:26.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:26.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:26.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:26.578 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:26.592 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:26.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:26.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:26.678 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:43:26.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:26.718 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:26.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:26.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:26.820 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:26.845 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:26.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:26.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:26.906 INFO:tasks.rados.rados.2.smithi175.stdout:1425: done (2 left) 2024-09-11T03:43:26.906 INFO:tasks.rados.rados.2.smithi175.stdout:1426: snap_create 2024-09-11T03:43:26.906 INFO:tasks.rados.rados.2.smithi175.stdout:1423: finishing rollback tid 0 to smithi17557870-26 2024-09-11T03:43:26.906 INFO:tasks.rados.rados.2.smithi175.stdout:1423: finishing rollback tid 1 to smithi17557870-26 2024-09-11T03:43:26.906 INFO:tasks.rados.rados.2.smithi175.stdout:1423: finishing rollback tid 2 to smithi17557870-26 2024-09-11T03:43:26.906 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 240 (ObjNum 460 snap 130 seq_num 460) dirty exists 2024-09-11T03:43:26.906 INFO:tasks.rados.rados.2.smithi175.stdout:1424: finishing write tid 1 to smithi17557870-24 2024-09-11T03:43:26.906 INFO:tasks.rados.rados.2.smithi175.stdout:1424: finishing write tid 2 to smithi17557870-24 2024-09-11T03:43:26.906 INFO:tasks.rados.rados.2.smithi175.stdout:1424: finishing write tid 3 to smithi17557870-24 2024-09-11T03:43:26.907 INFO:tasks.rados.rados.2.smithi175.stdout:1424: finishing write tid 4 to smithi17557870-24 2024-09-11T03:43:26.907 INFO:tasks.rados.rados.2.smithi175.stdout:1424: oid 24 updating version 0 to 545 2024-09-11T03:43:26.907 INFO:tasks.rados.rados.2.smithi175.stdout:1424: oid 24 version 545 is already newer than 544 2024-09-11T03:43:26.907 INFO:tasks.rados.rados.2.smithi175.stdout:1424: oid 24 version 545 is already newer than 543 2024-09-11T03:43:26.907 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 545 (ObjNum 505 snap 146 seq_num 505) dirty exists 2024-09-11T03:43:26.907 INFO:tasks.rados.rados.2.smithi175.stdout:1424: left oid 24 (ObjNum 505 snap 146 seq_num 505) 2024-09-11T03:43:26.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:26.970 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:26.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:26.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:27.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T03:43:27.072 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:27.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:27.095 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:27.095 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:27.181 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:43:27.181 INFO:teuthology.orchestra.run.smithi175.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T03:43:27.181 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:43:27.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:27.221 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:27.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:27.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:27.274 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:27 smithi084 ceph-mon[27254]: osdmap e388: 12 total, 12 up, 10 in 2024-09-11T03:43:27.274 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:27 smithi084 ceph-mon[27254]: osdmap e389: 12 total, 12 up, 10 in 2024-09-11T03:43:27.324 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:27.346 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:27.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:27.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:27 smithi121 ceph-mon[30729]: osdmap e388: 12 total, 12 up, 10 in 2024-09-11T03:43:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:27 smithi121 ceph-mon[30729]: osdmap e389: 12 total, 12 up, 10 in 2024-09-11T03:43:27.448 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:27.470 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:27.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:27.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:27 smithi175 ceph-mon[30894]: osdmap e388: 12 total, 12 up, 10 in 2024-09-11T03:43:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:27 smithi175 ceph-mon[30894]: osdmap e389: 12 total, 12 up, 10 in 2024-09-11T03:43:27.531 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:43:27.532 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:43:27.548 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:43:27.548 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:43:27.567 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:43:27.567 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:43:27.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:27.572 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:27.584 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:43:27.584 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:43:27.594 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:27.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:27.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:27.603 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:43:27.603 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:43:27.620 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:43:27.620 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:43:27.637 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:43:27.638 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:43:27.654 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:43:27.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:43:27.671 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:43:27.671 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:43:27.687 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:43:27.687 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:43:27.696 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:27.704 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:43:27.705 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:43:27.717 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:27.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:27.718 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:27.720 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:43:27.721 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:43:27.818 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:27.839 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:27.840 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:27.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:43:27.905 INFO:tasks.rados.rados.2.smithi175.stdout:1423: done (2 left) 2024-09-11T03:43:27.905 INFO:tasks.rados.rados.2.smithi175.stdout:1424: done (1 left) 2024-09-11T03:43:27.906 INFO:tasks.rados.rados.2.smithi175.stdout:1426: done (0 left) 2024-09-11T03:43:27.906 INFO:tasks.rados.rados.2.smithi175.stdout:1427: snap_create 2024-09-11T03:43:27.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:27.964 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:27.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:28.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:28.092 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:28.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:28.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:28.194 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:28.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:28.217 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:28.217 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:28.318 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:28.344 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:28.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:28.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:28.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:28 smithi084 ceph-mon[27254]: pgmap v738: 57 pgs: 57 active+clean; 156 MiB data, 710 MiB used, 893 GiB / 894 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2024-09-11T03:43:28.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:28 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/303305344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:28.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:28 smithi084 ceph-mon[27254]: osdmap e390: 12 total, 12 up, 10 in 2024-09-11T03:43:28.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:28 smithi121 ceph-mon[30729]: pgmap v738: 57 pgs: 57 active+clean; 156 MiB data, 710 MiB used, 893 GiB / 894 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2024-09-11T03:43:28.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:28 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/303305344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:28.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:28 smithi121 ceph-mon[30729]: osdmap e390: 12 total, 12 up, 10 in 2024-09-11T03:43:28.446 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:28.468 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:28.469 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:28 smithi175 ceph-mon[30894]: pgmap v738: 57 pgs: 57 active+clean; 156 MiB data, 710 MiB used, 893 GiB / 894 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2024-09-11T03:43:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:28 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/303305344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:28 smithi175 ceph-mon[30894]: osdmap e390: 12 total, 12 up, 10 in 2024-09-11T03:43:28.569 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:28.591 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:28.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:28.592 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:28.693 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:28.715 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:28.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:28.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:28.787 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:43:28.787 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:43:28.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:43:28.787 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:43:28.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.787 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:752 2.c e138/95 2:3c007163:::smithi17557870-35:head v 138'40, mlcod=135'37)", 2024-09-11T03:43:28.787 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:43:28.788 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 253.77093895199999, 2024-09-11T03:43:28.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084184749999999999, 2024-09-11T03:43:28.788 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.788 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.788 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:28.788 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:43:28.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:43:28.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:43:28.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8400000000000001e-07 2024-09-11T03:43:28.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021686+0000", 2024-09-11T03:43:28.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061118680000000003 2024-09-11T03:43:28.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021706+0000", 2024-09-11T03:43:28.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9942999999999999e-05 2024-09-11T03:43:28.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021717+0000", 2024-09-11T03:43:28.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.098e-05 2024-09-11T03:43:28.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021806+0000", 2024-09-11T03:43:28.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9197999999999995e-05 2024-09-11T03:43:28.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021834+0000", 2024-09-11T03:43:28.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8251000000000001e-05 2024-09-11T03:43:28.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:28.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.023931+0000", 2024-09-11T03:43:28.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020966639999999998 2024-09-11T03:43:28.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.023992+0000", 2024-09-11T03:43:28.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1387e-05 2024-09-11T03:43:28.792 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.792 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.792 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.792 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1175 2.c e185/95 2:32ade24e:::smithi17557870-45:head v 185'66, mlcod=182'62)", 2024-09-11T03:43:28.793 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:02.110649+0000", 2024-09-11T03:43:28.793 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 206.67586396799999, 2024-09-11T03:43:28.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00675758, 2024-09-11T03:43:28.793 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.793 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.793 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:28.793 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.110649+0000", 2024-09-11T03:43:28.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.110649+0000", 2024-09-11T03:43:28.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.110649+0000", 2024-09-11T03:43:28.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4699999999999998e-07 2024-09-11T03:43:28.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.115096+0000", 2024-09-11T03:43:28.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044473220000000001 2024-09-11T03:43:28.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.115114+0000", 2024-09-11T03:43:28.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7825e-05 2024-09-11T03:43:28.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.115124+0000", 2024-09-11T03:43:28.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5200000000000003e-06 2024-09-11T03:43:28.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.115202+0000", 2024-09-11T03:43:28.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8102000000000005e-05 2024-09-11T03:43:28.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.115231+0000", 2024-09-11T03:43:28.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8765e-05 2024-09-11T03:43:28.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:28.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.117364+0000", 2024-09-11T03:43:28.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021330630000000001 2024-09-11T03:43:28.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.117406+0000", 2024-09-11T03:43:28.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2735999999999999e-05 2024-09-11T03:43:28.797 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.797 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.797 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.797 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1285 2.1 e196/95 2:8638b65e:::smithi17557870-5:head v 196'52, mlcod=173'49)", 2024-09-11T03:43:28.798 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:12.850227+0000", 2024-09-11T03:43:28.798 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 195.93628533399999, 2024-09-11T03:43:28.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059143149999999999, 2024-09-11T03:43:28.798 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.798 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.798 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:28.798 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.850227+0000", 2024-09-11T03:43:28.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.850227+0000", 2024-09-11T03:43:28.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.850228+0000", 2024-09-11T03:43:28.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6199999999999997e-07 2024-09-11T03:43:28.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853054+0000", 2024-09-11T03:43:28.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028262000000000001 2024-09-11T03:43:28.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853087+0000", 2024-09-11T03:43:28.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2724e-05 2024-09-11T03:43:28.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853100+0000", 2024-09-11T03:43:28.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3456e-05 2024-09-11T03:43:28.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853187+0000", 2024-09-11T03:43:28.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6211999999999994e-05 2024-09-11T03:43:28.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853216+0000", 2024-09-11T03:43:28.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9825999999999999e-05 2024-09-11T03:43:28.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:28.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856078+0000", 2024-09-11T03:43:28.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028619689999999998 2024-09-11T03:43:28.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856142+0000", 2024-09-11T03:43:28.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3266e-05 2024-09-11T03:43:28.802 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.802 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.802 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.802 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1513 2.5 e212/95 2:a6e3bbd9:::smithi17557870-38:head v 212'178, mlcod=203'174)", 2024-09-11T03:43:28.802 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:28.928696+0000", 2024-09-11T03:43:28.802 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 179.85781704499999, 2024-09-11T03:43:28.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0099738110000000008, 2024-09-11T03:43:28.803 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.803 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.803 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:28.803 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.928696+0000", 2024-09-11T03:43:28.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.928696+0000", 2024-09-11T03:43:28.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.928696+0000", 2024-09-11T03:43:28.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9500000000000001e-07 2024-09-11T03:43:28.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935839+0000", 2024-09-11T03:43:28.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.007143473 2024-09-11T03:43:28.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935850+0000", 2024-09-11T03:43:28.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.026e-05 2024-09-11T03:43:28.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935854+0000", 2024-09-11T03:43:28.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8080000000000004e-06 2024-09-11T03:43:28.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935903+0000", 2024-09-11T03:43:28.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8380000000000001e-05 2024-09-11T03:43:28.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935917+0000", 2024-09-11T03:43:28.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4192999999999999e-05 2024-09-11T03:43:28.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:28.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.938638+0000", 2024-09-11T03:43:28.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002721298 2024-09-11T03:43:28.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.938669+0000", 2024-09-11T03:43:28.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1204000000000001e-05 2024-09-11T03:43:28.808 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.808 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.808 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.808 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2112 2.4 e268/188 2:2ef19c88:::smithi17557870-15:head v 268'153, mlcod=264'150)", 2024-09-11T03:43:28.809 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:24.873716+0000", 2024-09-11T03:43:28.809 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 123.912796915, 2024-09-11T03:43:28.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065806770000000001, 2024-09-11T03:43:28.809 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.809 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.809 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:28.809 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.873716+0000", 2024-09-11T03:43:28.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.873716+0000", 2024-09-11T03:43:28.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.873716+0000", 2024-09-11T03:43:28.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0699999999999998e-07 2024-09-11T03:43:28.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.875592+0000", 2024-09-11T03:43:28.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001875413 2024-09-11T03:43:28.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.875615+0000", 2024-09-11T03:43:28.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3884999999999999e-05 2024-09-11T03:43:28.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.875629+0000", 2024-09-11T03:43:28.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.366e-05 2024-09-11T03:43:28.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.875707+0000", 2024-09-11T03:43:28.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8301999999999997e-05 2024-09-11T03:43:28.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.875733+0000", 2024-09-11T03:43:28.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5307999999999999e-05 2024-09-11T03:43:28.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:28.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.880233+0000", 2024-09-11T03:43:28.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00450052 2024-09-11T03:43:28.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.880296+0000", 2024-09-11T03:43:28.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3281999999999995e-05 2024-09-11T03:43:28.816 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.816 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.816 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.816 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2115 2.5 e268/95 2:a6e3bbd9:::smithi17557870-38:head v 268'256, mlcod=260'252)", 2024-09-11T03:43:28.816 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:24.876744+0000", 2024-09-11T03:43:28.816 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 123.909768378, 2024-09-11T03:43:28.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01118442, 2024-09-11T03:43:28.817 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.817 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.817 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:28.817 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.876744+0000", 2024-09-11T03:43:28.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.876744+0000", 2024-09-11T03:43:28.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.876745+0000", 2024-09-11T03:43:28.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8999999999999998e-07 2024-09-11T03:43:28.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.885165+0000", 2024-09-11T03:43:28.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084204060000000001 2024-09-11T03:43:28.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.885185+0000", 2024-09-11T03:43:28.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0432999999999999e-05 2024-09-11T03:43:28.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.885203+0000", 2024-09-11T03:43:28.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7569000000000001e-05 2024-09-11T03:43:28.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.885268+0000", 2024-09-11T03:43:28.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5104999999999994e-05 2024-09-11T03:43:28.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.885289+0000", 2024-09-11T03:43:28.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0800000000000001e-05 2024-09-11T03:43:28.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:28.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.887888+0000", 2024-09-11T03:43:28.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025986999999999998 2024-09-11T03:43:28.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.887929+0000", 2024-09-11T03:43:28.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1117e-05 2024-09-11T03:43:28.823 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.823 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.823 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.823 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2127 2.4 e268/188 2:2ef19c88:::smithi17557870-15:head v 268'154, mlcod=265'152)", 2024-09-11T03:43:28.823 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:25.035224+0000", 2024-09-11T03:43:28.823 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 123.751289054, 2024-09-11T03:43:28.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048299429999999997, 2024-09-11T03:43:28.823 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.824 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.824 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:28.824 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.035224+0000", 2024-09-11T03:43:28.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.035224+0000", 2024-09-11T03:43:28.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:28.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.035226+0000", 2024-09-11T03:43:28.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.024e-06 2024-09-11T03:43:28.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.035794+0000", 2024-09-11T03:43:28.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00056879499999999998 2024-09-11T03:43:28.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.035810+0000", 2024-09-11T03:43:28.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5361000000000001e-05 2024-09-11T03:43:28.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.035816+0000", 2024-09-11T03:43:28.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3790000000000003e-06 2024-09-11T03:43:28.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.035876+0000", 2024-09-11T03:43:28.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9392999999999999e-05 2024-09-11T03:43:28.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.035899+0000", 2024-09-11T03:43:28.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3706e-05 2024-09-11T03:43:28.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:28.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.039991+0000", 2024-09-11T03:43:28.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040916629999999997 2024-09-11T03:43:28.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.040054+0000", 2024-09-11T03:43:28.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2621999999999998e-05 2024-09-11T03:43:28.830 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.830 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.830 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.830 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2140 2.c e268/95 2:3c007163:::smithi17557870-35:head v 268'210, mlcod=267'207)", 2024-09-11T03:43:28.830 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:25.066817+0000", 2024-09-11T03:43:28.830 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 123.71969616200001, 2024-09-11T03:43:28.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069600249999999999, 2024-09-11T03:43:28.830 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.830 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.830 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:28.831 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.066817+0000", 2024-09-11T03:43:28.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.066817+0000", 2024-09-11T03:43:28.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.066817+0000", 2024-09-11T03:43:28.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1099999999999995e-07 2024-09-11T03:43:28.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.072136+0000", 2024-09-11T03:43:28.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053183930000000003 2024-09-11T03:43:28.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.072164+0000", 2024-09-11T03:43:28.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8603999999999999e-05 2024-09-11T03:43:28.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.072178+0000", 2024-09-11T03:43:28.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3733e-05 2024-09-11T03:43:28.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.072237+0000", 2024-09-11T03:43:28.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9072e-05 2024-09-11T03:43:28.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.072255+0000", 2024-09-11T03:43:28.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7625000000000002e-05 2024-09-11T03:43:28.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:28.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.073708+0000", 2024-09-11T03:43:28.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001453668 2024-09-11T03:43:28.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.073777+0000", 2024-09-11T03:43:28.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8318999999999994e-05 2024-09-11T03:43:28.835 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.835 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.835 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.835 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.8.0:443 2.5 e279/276 2:a6e3bbd9:::smithi17557870-38:5d v 279'283, mlcod=278'279)", 2024-09-11T03:43:28.835 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:35.865183+0000", 2024-09-11T03:43:28.835 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 112.92133013900001, 2024-09-11T03:43:28.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011482019, 2024-09-11T03:43:28.835 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.835 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.835 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:28.835 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865183+0000", 2024-09-11T03:43:28.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865183+0000", 2024-09-11T03:43:28.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865183+0000", 2024-09-11T03:43:28.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0399999999999996e-07 2024-09-11T03:43:28.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865190+0000", 2024-09-11T03:43:28.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9460000000000001e-06 2024-09-11T03:43:28.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865199+0000", 2024-09-11T03:43:28.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7979999999999995e-06 2024-09-11T03:43:28.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865214+0000", 2024-09-11T03:43:28.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5184999999999999e-05 2024-09-11T03:43:28.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865278+0000", 2024-09-11T03:43:28.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4086999999999998e-05 2024-09-11T03:43:28.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865303+0000", 2024-09-11T03:43:28.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.446e-05 2024-09-11T03:43:28.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:28.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.876628+0000", 2024-09-11T03:43:28.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011325574 2024-09-11T03:43:28.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.876665+0000", 2024-09-11T03:43:28.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6464999999999997e-05 2024-09-11T03:43:28.839 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.839 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.839 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.839 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2241 2.14 2:2ef19c88:::smithi17557870-15:58 [read 0~3327819,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:43:28.840 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:36.886222+0000", 2024-09-11T03:43:28.840 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 111.900291174, 2024-09-11T03:43:28.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012098761, 2024-09-11T03:43:28.840 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.840 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.840 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:43:28.840 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:43:28.840 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:43:28.840 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:28.840 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2241 2024-09-11T03:43:28.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.840 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886222+0000", 2024-09-11T03:43:28.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886222+0000", 2024-09-11T03:43:28.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886223+0000", 2024-09-11T03:43:28.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4529999999999999e-06 2024-09-11T03:43:28.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886228+0000", 2024-09-11T03:43:28.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9869999999999999e-06 2024-09-11T03:43:28.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886239+0000", 2024-09-11T03:43:28.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.08e-05 2024-09-11T03:43:28.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886249+0000", 2024-09-11T03:43:28.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0322e-05 2024-09-11T03:43:28.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886366+0000", 2024-09-11T03:43:28.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000116549 2024-09-11T03:43:28.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:43:28.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886375+0000", 2024-09-11T03:43:28.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3870000000000001e-06 2024-09-11T03:43:28.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.893120+0000", 2024-09-11T03:43:28.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067446340000000002 2024-09-11T03:43:28.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.893275+0000", 2024-09-11T03:43:28.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000155079 2024-09-11T03:43:28.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.898320+0000", 2024-09-11T03:43:28.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0050455500000000002 2024-09-11T03:43:28.845 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.845 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.845 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.845 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.8.0:445 2.5 e280/276 2:a286c222:::smithi17557870-29:5b v 280'285, mlcod=278'281)", 2024-09-11T03:43:28.846 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:36.886383+0000", 2024-09-11T03:43:28.846 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 111.900129775, 2024-09-11T03:43:28.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.004457323, 2024-09-11T03:43:28.846 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.846 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.846 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:28.847 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886383+0000", 2024-09-11T03:43:28.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886383+0000", 2024-09-11T03:43:28.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886383+0000", 2024-09-11T03:43:28.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.51e-07 2024-09-11T03:43:28.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886389+0000", 2024-09-11T03:43:28.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7379999999999999e-06 2024-09-11T03:43:28.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886398+0000", 2024-09-11T03:43:28.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1959999999999997e-06 2024-09-11T03:43:28.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886404+0000", 2024-09-11T03:43:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6749999999999999e-06 2024-09-11T03:43:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886471+0000", 2024-09-11T03:43:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7044000000000004e-05 2024-09-11T03:43:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886493+0000", 2024-09-11T03:43:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1956000000000001e-05 2024-09-11T03:43:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:28.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.890814+0000", 2024-09-11T03:43:28.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043206260000000002 2024-09-11T03:43:28.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.890840+0000", 2024-09-11T03:43:28.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6537e-05 2024-09-11T03:43:28.850 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.850 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.850 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.851 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2242 2.14 2:2ef19c88:::smithi17557870-15:58 [sparse-read 0~2125189] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:43:28.851 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:36.886426+0000", 2024-09-11T03:43:28.851 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 111.900086226, 2024-09-11T03:43:28.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012666422, 2024-09-11T03:43:28.851 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.851 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.851 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:43:28.851 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:43:28.851 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:43:28.851 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:28.851 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2242 2024-09-11T03:43:28.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.852 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886426+0000", 2024-09-11T03:43:28.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886426+0000", 2024-09-11T03:43:28.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886427+0000", 2024-09-11T03:43:28.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2799999999999996e-07 2024-09-11T03:43:28.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886431+0000", 2024-09-11T03:43:28.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7910000000000002e-06 2024-09-11T03:43:28.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886439+0000", 2024-09-11T03:43:28.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.038e-06 2024-09-11T03:43:28.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886445+0000", 2024-09-11T03:43:28.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3799999999999999e-06 2024-09-11T03:43:28.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886527+0000", 2024-09-11T03:43:28.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1797000000000003e-05 2024-09-11T03:43:28.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:43:28.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886534+0000", 2024-09-11T03:43:28.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2599999999999999e-06 2024-09-11T03:43:28.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.898657+0000", 2024-09-11T03:43:28.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012123073 2024-09-11T03:43:28.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.898831+0000", 2024-09-11T03:43:28.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000173931 2024-09-11T03:43:28.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.899093+0000", 2024-09-11T03:43:28.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026152400000000001 2024-09-11T03:43:28.856 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.856 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.856 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.856 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2243 2.14 2:2ef19c88:::smithi17557870-15:58 [read 0~3327819,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:43:28.856 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:36.886461+0000", 2024-09-11T03:43:28.856 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 111.90005145799999, 2024-09-11T03:43:28.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014874943999999999, 2024-09-11T03:43:28.856 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.856 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.857 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:43:28.857 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:43:28.857 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:43:28.857 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:28.857 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2243 2024-09-11T03:43:28.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.857 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886461+0000", 2024-09-11T03:43:28.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886461+0000", 2024-09-11T03:43:28.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886462+0000", 2024-09-11T03:43:28.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1899999999999998e-07 2024-09-11T03:43:28.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886464+0000", 2024-09-11T03:43:28.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.632e-06 2024-09-11T03:43:28.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886470+0000", 2024-09-11T03:43:28.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.772e-06 2024-09-11T03:43:28.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886474+0000", 2024-09-11T03:43:28.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2549999999999999e-06 2024-09-11T03:43:28.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886554+0000", 2024-09-11T03:43:28.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9851999999999994e-05 2024-09-11T03:43:28.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:43:28.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886560+0000", 2024-09-11T03:43:28.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.117e-06 2024-09-11T03:43:28.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.899326+0000", 2024-09-11T03:43:28.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012765841999999999 2024-09-11T03:43:28.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.899468+0000", 2024-09-11T03:43:28.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014217099999999999 2024-09-11T03:43:28.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.901336+0000", 2024-09-11T03:43:28.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001867884 2024-09-11T03:43:28.862 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.863 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.863 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.863 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.8.0:446 2.5 e280/276 2:a6e3bbd9:::smithi17557870-38:5d v 280'287, mlcod=278'281)", 2024-09-11T03:43:28.863 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:36.886860+0000", 2024-09-11T03:43:28.863 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 111.899652233, 2024-09-11T03:43:28.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043266240000000003, 2024-09-11T03:43:28.863 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.863 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.863 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:28.863 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886860+0000", 2024-09-11T03:43:28.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886860+0000", 2024-09-11T03:43:28.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886861+0000", 2024-09-11T03:43:28.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0800000000000005e-07 2024-09-11T03:43:28.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886866+0000", 2024-09-11T03:43:28.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6800000000000001e-06 2024-09-11T03:43:28.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886875+0000", 2024-09-11T03:43:28.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9779999999999994e-06 2024-09-11T03:43:28.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886881+0000", 2024-09-11T03:43:28.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0129999999999999e-06 2024-09-11T03:43:28.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886931+0000", 2024-09-11T03:43:28.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9962e-05 2024-09-11T03:43:28.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886949+0000", 2024-09-11T03:43:28.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8615999999999999e-05 2024-09-11T03:43:28.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:28.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.891157+0000", 2024-09-11T03:43:28.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042076609999999997 2024-09-11T03:43:28.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.891187+0000", 2024-09-11T03:43:28.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0205999999999998e-05 2024-09-11T03:43:28.871 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.871 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.871 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.871 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2443 2.12 e296/281 2:494ba932:::smithi17557870-43:head v 296'131, mlcod=291'128)", 2024-09-11T03:43:28.871 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:52.895764+0000", 2024-09-11T03:43:28.871 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 95.890749181999993, 2024-09-11T03:43:28.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068629390000000002, 2024-09-11T03:43:28.871 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.871 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.871 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:28.872 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.895764+0000", 2024-09-11T03:43:28.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.895764+0000", 2024-09-11T03:43:28.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.895764+0000", 2024-09-11T03:43:28.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2099999999999998e-07 2024-09-11T03:43:28.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.898075+0000", 2024-09-11T03:43:28.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0023108790000000001 2024-09-11T03:43:28.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.898091+0000", 2024-09-11T03:43:28.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6293000000000001e-05 2024-09-11T03:43:28.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.898104+0000", 2024-09-11T03:43:28.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2539e-05 2024-09-11T03:43:28.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.898160+0000", 2024-09-11T03:43:28.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.643e-05 2024-09-11T03:43:28.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.898186+0000", 2024-09-11T03:43:28.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5666000000000001e-05 2024-09-11T03:43:28.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:28.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.902581+0000", 2024-09-11T03:43:28.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043955089999999997 2024-09-11T03:43:28.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.902626+0000", 2024-09-11T03:43:28.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5302000000000001e-05 2024-09-11T03:43:28.876 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.876 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.876 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.876 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2492 2.5 e302/276 2:a1a99782:::smithi17557870-1:head v 302'331, mlcod=297'328)", 2024-09-11T03:43:28.876 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:58.902093+0000", 2024-09-11T03:43:28.876 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 89.884419778999998, 2024-09-11T03:43:28.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087348040000000005, 2024-09-11T03:43:28.876 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.876 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.876 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:28.877 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.902093+0000", 2024-09-11T03:43:28.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.902093+0000", 2024-09-11T03:43:28.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.902093+0000", 2024-09-11T03:43:28.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.11e-07 2024-09-11T03:43:28.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.908258+0000", 2024-09-11T03:43:28.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061649770000000003 2024-09-11T03:43:28.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.908276+0000", 2024-09-11T03:43:28.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7737e-05 2024-09-11T03:43:28.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.908284+0000", 2024-09-11T03:43:28.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2490000000000001e-06 2024-09-11T03:43:28.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.908348+0000", 2024-09-11T03:43:28.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3712000000000003e-05 2024-09-11T03:43:28.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.908367+0000", 2024-09-11T03:43:28.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9633e-05 2024-09-11T03:43:28.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:28.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.910624+0000", 2024-09-11T03:43:28.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022564680000000002 2024-09-11T03:43:28.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.910828+0000", 2024-09-11T03:43:28.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00020381699999999999 2024-09-11T03:43:28.880 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.880 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.881 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.881 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2931 2.5 e346/276 2:a6d17fc0:::smithi17557870-13:head v 346'401, mlcod=340'397)", 2024-09-11T03:43:28.881 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:43.125646+0000", 2024-09-11T03:43:28.881 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 45.660866630999998, 2024-09-11T03:43:28.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056773580000000004, 2024-09-11T03:43:28.881 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.881 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.881 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:28.881 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.125646+0000", 2024-09-11T03:43:28.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.125646+0000", 2024-09-11T03:43:28.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.125646+0000", 2024-09-11T03:43:28.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6899999999999999e-07 2024-09-11T03:43:28.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.129356+0000", 2024-09-11T03:43:28.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037093819999999998 2024-09-11T03:43:28.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.129373+0000", 2024-09-11T03:43:28.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7028e-05 2024-09-11T03:43:28.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.129381+0000", 2024-09-11T03:43:28.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3380000000000005e-06 2024-09-11T03:43:28.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.129437+0000", 2024-09-11T03:43:28.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6001000000000001e-05 2024-09-11T03:43:28.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.129456+0000", 2024-09-11T03:43:28.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8916999999999999e-05 2024-09-11T03:43:28.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:28.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.131287+0000", 2024-09-11T03:43:28.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018311340000000001 2024-09-11T03:43:28.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.131323+0000", 2024-09-11T03:43:28.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6389e-05 2024-09-11T03:43:28.885 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.885 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.885 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.886 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2944 2.12 e347/312 2:494ba932:::smithi17557870-43:head v 347'195, mlcod=346'189)", 2024-09-11T03:43:28.886 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:43.972394+0000", 2024-09-11T03:43:28.886 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 44.814118737999998, 2024-09-11T03:43:28.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.004314666, 2024-09-11T03:43:28.886 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.886 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.886 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:28.886 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.972394+0000", 2024-09-11T03:43:28.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.972394+0000", 2024-09-11T03:43:28.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.972395+0000", 2024-09-11T03:43:28.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.438e-06 2024-09-11T03:43:28.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.974058+0000", 2024-09-11T03:43:28.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001662321 2024-09-11T03:43:28.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.974081+0000", 2024-09-11T03:43:28.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3351000000000001e-05 2024-09-11T03:43:28.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.974090+0000", 2024-09-11T03:43:28.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1560000000000001e-06 2024-09-11T03:43:28.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.974169+0000", 2024-09-11T03:43:28.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8671999999999995e-05 2024-09-11T03:43:28.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.974195+0000", 2024-09-11T03:43:28.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5675999999999999e-05 2024-09-11T03:43:28.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:28.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.976673+0000", 2024-09-11T03:43:28.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002478619 2024-09-11T03:43:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.976709+0000", 2024-09-11T03:43:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5432999999999998e-05 2024-09-11T03:43:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.1.0:635 2.4 e353/276 2:25f5f8b9:::smithi17557870-26:80 v 353'215, mlcod=352'209)", 2024-09-11T03:43:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:50.003105+0000", 2024-09-11T03:43:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 38.783407736999997, 2024-09-11T03:43:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.004696581, 2024-09-11T03:43:28.891 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.891 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.891 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:28.891 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.003105+0000", 2024-09-11T03:43:28.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.003105+0000", 2024-09-11T03:43:28.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.003106+0000", 2024-09-11T03:43:28.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.04e-06 2024-09-11T03:43:28.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.003113+0000", 2024-09-11T03:43:28.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7270000000000001e-06 2024-09-11T03:43:28.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.003135+0000", 2024-09-11T03:43:28.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2558999999999999e-05 2024-09-11T03:43:28.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.003152+0000", 2024-09-11T03:43:28.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6245000000000001e-05 2024-09-11T03:43:28.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.003200+0000", 2024-09-11T03:43:28.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8381999999999999e-05 2024-09-11T03:43:28.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.003220+0000", 2024-09-11T03:43:28.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0230000000000001e-05 2024-09-11T03:43:28.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:28.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.007275+0000", 2024-09-11T03:43:28.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040553179999999996 2024-09-11T03:43:28.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.007802+0000", 2024-09-11T03:43:28.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052607999999999995 2024-09-11T03:43:28.895 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.895 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.895 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.895 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:3234 2.19 e361/281 2:98bb65f5:::smithi17557870-6:head v 361'386, mlcod=359'383)", 2024-09-11T03:43:28.895 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:58.150912+0000", 2024-09-11T03:43:28.895 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 30.635600999000001, 2024-09-11T03:43:28.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0057244130000000002, 2024-09-11T03:43:28.896 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:28.896 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:28.896 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:28.896 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:28.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:28.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.150912+0000", 2024-09-11T03:43:28.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:28.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.150912+0000", 2024-09-11T03:43:28.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:28.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:28.897 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.150913+0000", 2024-09-11T03:43:28.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.06e-06 2024-09-11T03:43:28.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:28.897 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.153675+0000", 2024-09-11T03:43:28.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002761955 2024-09-11T03:43:28.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:28.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.153697+0000", 2024-09-11T03:43:28.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1784999999999999e-05 2024-09-11T03:43:28.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:28.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.153713+0000", 2024-09-11T03:43:28.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6560000000000001e-05 2024-09-11T03:43:28.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:28.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.153804+0000", 2024-09-11T03:43:28.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0946999999999995e-05 2024-09-11T03:43:28.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:28.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.153846+0000", 2024-09-11T03:43:28.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2348000000000002e-05 2024-09-11T03:43:28.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:28.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.156588+0000", 2024-09-11T03:43:28.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027418350000000002 2024-09-11T03:43:28.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:28.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:28.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:28.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.156636+0000", 2024-09-11T03:43:28.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7923000000000002e-05 2024-09-11T03:43:28.900 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.900 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.900 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.900 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:28.900 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:28.900 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:43:28.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:28.902 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:28.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:28.905 INFO:tasks.rados.rados.2.smithi175.stdout:1427: done (0 left) 2024-09-11T03:43:28.905 INFO:tasks.rados.rados.2.smithi175.stdout:1428: rollback oid 10 current snap is 148 2024-09-11T03:43:28.905 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 10 to 147 2024-09-11T03:43:28.905 INFO:tasks.rados.rados.2.smithi175.stdout:1429: write oid 22 current snap is 148 2024-09-11T03:43:28.906 INFO:tasks.rados.rados.2.smithi175.stdout:1429: seq_num 506 ranges {504342=555292,1643418=484844} 2024-09-11T03:43:28.910 INFO:tasks.rados.rados.2.smithi175.stdout:1429: writing smithi17557870-22 from 504342 to 1059634 tid 1 2024-09-11T03:43:28.916 INFO:tasks.rados.rados.2.smithi175.stdout:1429: writing smithi17557870-22 from 1643418 to 2128262 tid 2 2024-09-11T03:43:28.916 INFO:tasks.rados.rados.2.smithi175.stdout:1430: snap_remove snap 144 2024-09-11T03:43:29.003 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:29.026 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:29.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:29.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:29.045 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:29.127 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:29.149 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:29.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:29.151 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:29.252 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:29.278 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:29.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:29.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:29.380 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:29.403 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:29.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:29.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:29.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:29.531 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:29.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:29.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:29.633 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:29.658 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:29.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:29.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:29.760 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:29.788 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:29.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:29.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:29.795 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T03:43:29.891 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:29.913 INFO:tasks.rados.rados.2.smithi175.stdout:1430: done (2 left) 2024-09-11T03:43:29.913 INFO:tasks.rados.rados.2.smithi175.stdout:1431: snap_remove snap 143 2024-09-11T03:43:29.917 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:29.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:29.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:30.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:30.045 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:30.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:30.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:30.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:29 smithi084 ceph-mon[27254]: pgmap v741: 57 pgs: 57 active+clean; 157 MiB data, 714 MiB used, 893 GiB / 894 GiB avail; 6.8 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-11T03:43:30.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:29 smithi084 ceph-mon[27254]: osdmap e391: 12 total, 12 up, 10 in 2024-09-11T03:43:30.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:29 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:43:30.064 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:29 smithi084 ceph-mon[27254]: 2.a scrub ok 2024-09-11T03:43:30.147 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:30.169 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:30.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:30.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:29 smithi121 ceph-mon[30729]: pgmap v741: 57 pgs: 57 active+clean; 157 MiB data, 714 MiB used, 893 GiB / 894 GiB avail; 6.8 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-11T03:43:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:29 smithi121 ceph-mon[30729]: osdmap e391: 12 total, 12 up, 10 in 2024-09-11T03:43:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:29 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:43:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:29 smithi121 ceph-mon[30729]: 2.a scrub ok 2024-09-11T03:43:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:29 smithi175 ceph-mon[30894]: pgmap v741: 57 pgs: 57 active+clean; 157 MiB data, 714 MiB used, 893 GiB / 894 GiB avail; 6.8 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-11T03:43:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:29 smithi175 ceph-mon[30894]: osdmap e391: 12 total, 12 up, 10 in 2024-09-11T03:43:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:29 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:43:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:29 smithi175 ceph-mon[30894]: 2.a scrub ok 2024-09-11T03:43:30.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:30.298 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:30.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:30.359 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:30.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:30.424 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:30.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:30.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:30.526 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:30.533 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:43:30.551 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:30.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:30.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:30.654 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:30.679 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:30.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:30.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:30.804 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:43:30.804 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:43:30.804 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:43:30.804 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:43:30.807 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:30.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:30.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:30.909 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:30.912 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:43:30.920 INFO:tasks.rados.rados.2.smithi175.stdout:1431: done (2 left) 2024-09-11T03:43:30.920 INFO:tasks.rados.rados.2.smithi175.stdout:1432: write oid 7 current snap is 148 2024-09-11T03:43:30.921 INFO:tasks.rados.rados.2.smithi175.stdout:1428: finishing rollback tid 0 to smithi17557870-10 2024-09-11T03:43:30.921 INFO:tasks.rados.rados.2.smithi175.stdout:1428: finishing rollback tid 1 to smithi17557870-10 2024-09-11T03:43:30.921 INFO:tasks.rados.rados.2.smithi175.stdout:1428: finishing rollback tid 2 to smithi17557870-10 2024-09-11T03:43:30.921 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 526 (ObjNum 488 snap 141 seq_num 488) dirty exists 2024-09-11T03:43:30.921 INFO:tasks.rados.rados.2.smithi175.stdout:1429: finishing write tid 1 to smithi17557870-22 2024-09-11T03:43:30.921 INFO:tasks.rados.rados.2.smithi175.stdout:1429: finishing write tid 2 to smithi17557870-22 2024-09-11T03:43:30.921 INFO:tasks.rados.rados.2.smithi175.stdout:1429: finishing write tid 3 to smithi17557870-22 2024-09-11T03:43:30.921 INFO:tasks.rados.rados.2.smithi175.stdout:1429: finishing write tid 4 to smithi17557870-22 2024-09-11T03:43:30.921 INFO:tasks.rados.rados.2.smithi175.stdout:1429: oid 22 updating version 0 to 539 2024-09-11T03:43:30.921 INFO:tasks.rados.rados.2.smithi175.stdout:1429: oid 22 version 539 is already newer than 537 2024-09-11T03:43:30.921 INFO:tasks.rados.rados.2.smithi175.stdout:1429: oid 22 version 539 is already newer than 538 2024-09-11T03:43:30.921 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 539 (ObjNum 506 snap 148 seq_num 506) dirty exists 2024-09-11T03:43:30.921 INFO:tasks.rados.rados.2.smithi175.stdout:1429: left oid 22 (ObjNum 506 snap 148 seq_num 506) 2024-09-11T03:43:30.922 INFO:tasks.rados.rados.2.smithi175.stdout:1432: seq_num 507 ranges {521149=630541,1701900=628502} 2024-09-11T03:43:30.927 INFO:tasks.rados.rados.2.smithi175.stdout:1432: writing smithi17557870-7 from 521149 to 1151690 tid 1 2024-09-11T03:43:30.933 INFO:tasks.rados.rados.2.smithi175.stdout:1432: writing smithi17557870-7 from 1701900 to 2330402 tid 2 2024-09-11T03:43:30.933 INFO:tasks.rados.rados.2.smithi175.stdout:1428: done (2 left) 2024-09-11T03:43:30.934 INFO:tasks.rados.rados.2.smithi175.stdout:1429: done (1 left) 2024-09-11T03:43:30.934 INFO:tasks.rados.rados.2.smithi175.stdout:1433: rollback oid 12 current snap is 148 2024-09-11T03:43:30.934 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 12 to 136 2024-09-11T03:43:30.934 INFO:tasks.rados.rados.2.smithi175.stdout:1434: snap_create 2024-09-11T03:43:30.934 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:30.934 INFO:tasks.rados.rados.2.smithi175.stdout:1432: finishing write tid 1 to smithi17557870-7 2024-09-11T03:43:30.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:30.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:30.938 INFO:tasks.rados.rados.2.smithi175.stdout:1432: finishing write tid 2 to smithi17557870-7 2024-09-11T03:43:30.939 INFO:tasks.rados.rados.2.smithi175.stdout:1432: finishing write tid 3 to smithi17557870-7 2024-09-11T03:43:30.939 INFO:tasks.rados.rados.2.smithi175.stdout:1432: finishing write tid 4 to smithi17557870-7 2024-09-11T03:43:30.939 INFO:tasks.rados.rados.2.smithi175.stdout:1432: oid 7 updating version 0 to 464 2024-09-11T03:43:30.939 INFO:tasks.rados.rados.2.smithi175.stdout:1432: oid 7 updating version 464 to 465 2024-09-11T03:43:30.939 INFO:tasks.rados.rados.2.smithi175.stdout:1432: oid 7 version 465 is already newer than 463 2024-09-11T03:43:30.939 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 465 (ObjNum 507 snap 148 seq_num 507) dirty exists 2024-09-11T03:43:30.939 INFO:tasks.rados.rados.2.smithi175.stdout:1432: left oid 7 (ObjNum 507 snap 148 seq_num 507) 2024-09-11T03:43:30.940 INFO:tasks.rados.rados.2.smithi175.stdout:1433: finishing rollback tid 0 to smithi17557870-12 2024-09-11T03:43:30.940 INFO:tasks.rados.rados.2.smithi175.stdout:1433: finishing rollback tid 1 to smithi17557870-12 2024-09-11T03:43:30.940 INFO:tasks.rados.rados.2.smithi175.stdout:1433: finishing rollback tid 2 to smithi17557870-12 2024-09-11T03:43:30.940 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 503 (ObjNum 440 snap 130 seq_num 440) dirty exists 2024-09-11T03:43:31.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:31.062 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:31.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:31.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:31.164 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:31.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:30 smithi121 ceph-mon[30729]: osdmap e392: 12 total, 12 up, 10 in 2024-09-11T03:43:31.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:30 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3718917036' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:43:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:31.186 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:31.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:31.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:30 smithi175 ceph-mon[30894]: osdmap e392: 12 total, 12 up, 10 in 2024-09-11T03:43:31.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:30 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3718917036' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:43:31.287 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:31.309 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:31.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:31.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:31.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:30 smithi084 ceph-mon[27254]: osdmap e392: 12 total, 12 up, 10 in 2024-09-11T03:43:31.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:30 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3718917036' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:43:31.410 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:31.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:31.437 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:31.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:31.539 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:31.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:31.561 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:31.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:31.662 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:31.689 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:31.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:31.690 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:31.791 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:31.814 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:31.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:31.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:31.819 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:31.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T03:43:31.905 INFO:tasks.rados.rados.2.smithi175.stdout:1432: done (2 left) 2024-09-11T03:43:31.905 INFO:tasks.rados.rados.2.smithi175.stdout:1433: done (1 left) 2024-09-11T03:43:31.905 INFO:tasks.rados.rados.2.smithi175.stdout:1434: done (0 left) 2024-09-11T03:43:31.905 INFO:tasks.rados.rados.2.smithi175.stdout:1435: read oid 28 snap 146 2024-09-11T03:43:31.905 INFO:tasks.rados.rados.2.smithi175.stdout:1435: expect deleted 2024-09-11T03:43:31.905 INFO:tasks.rados.rados.2.smithi175.stdout:1436: delete oid 47 current snap is 149 2024-09-11T03:43:31.909 INFO:tasks.rados.rados.2.smithi175.stdout:1435: done (1 left) 2024-09-11T03:43:31.909 INFO:tasks.rados.rados.2.smithi175.stdout:1436: done (0 left) 2024-09-11T03:43:31.909 INFO:tasks.rados.rados.2.smithi175.stdout:1437: rollback oid 31 current snap is 149 2024-09-11T03:43:31.909 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 31 to 142 2024-09-11T03:43:31.909 INFO:tasks.rados.rados.2.smithi175.stdout:1438: snap_create 2024-09-11T03:43:31.913 INFO:tasks.rados.rados.2.smithi175.stdout:1437: finishing rollback tid 0 to smithi17557870-31 2024-09-11T03:43:31.914 INFO:tasks.rados.rados.2.smithi175.stdout:1437: finishing rollback tid 1 to smithi17557870-31 2024-09-11T03:43:31.914 INFO:tasks.rados.rados.2.smithi175.stdout:1437: finishing rollback tid 2 to smithi17557870-31 2024-09-11T03:43:31.914 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 507 (ObjNum 475 snap 138 seq_num 475) dirty exists 2024-09-11T03:43:31.916 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:31.937 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:31.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:32.039 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:32.065 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:32.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:32.066 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:32.081 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:31 smithi084 ceph-mon[27254]: 1.18 scrub starts 2024-09-11T03:43:32.081 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:31 smithi084 ceph-mon[27254]: 1.18 scrub ok 2024-09-11T03:43:32.081 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:31 smithi084 ceph-mon[27254]: pgmap v744: 57 pgs: 57 active+clean; 157 MiB data, 713 MiB used, 893 GiB / 894 GiB avail; 3.4 MiB/s rd, 270 KiB/s wr, 4 op/s 2024-09-11T03:43:32.081 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:31 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:32.081 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:31 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3718917036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:43:32.081 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:31 smithi084 ceph-mon[27254]: osdmap e393: 12 total, 12 up, 10 in 2024-09-11T03:43:32.081 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:31 smithi084 ceph-mon[27254]: osdmap e394: 12 total, 12 up, 10 in 2024-09-11T03:43:32.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:32.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:31 smithi121 ceph-mon[30729]: 1.18 scrub starts 2024-09-11T03:43:32.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:31 smithi121 ceph-mon[30729]: 1.18 scrub ok 2024-09-11T03:43:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:31 smithi121 ceph-mon[30729]: pgmap v744: 57 pgs: 57 active+clean; 157 MiB data, 713 MiB used, 893 GiB / 894 GiB avail; 3.4 MiB/s rd, 270 KiB/s wr, 4 op/s 2024-09-11T03:43:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:31 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:31 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3718917036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:43:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:31 smithi121 ceph-mon[30729]: osdmap e393: 12 total, 12 up, 10 in 2024-09-11T03:43:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:31 smithi121 ceph-mon[30729]: osdmap e394: 12 total, 12 up, 10 in 2024-09-11T03:43:32.192 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:32.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:32.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:31 smithi175 ceph-mon[30894]: 1.18 scrub starts 2024-09-11T03:43:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:31 smithi175 ceph-mon[30894]: 1.18 scrub ok 2024-09-11T03:43:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:31 smithi175 ceph-mon[30894]: pgmap v744: 57 pgs: 57 active+clean; 157 MiB data, 713 MiB used, 893 GiB / 894 GiB avail; 3.4 MiB/s rd, 270 KiB/s wr, 4 op/s 2024-09-11T03:43:32.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:31 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:32.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:31 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3718917036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:43:32.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:31 smithi175 ceph-mon[30894]: osdmap e393: 12 total, 12 up, 10 in 2024-09-11T03:43:32.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:31 smithi175 ceph-mon[30894]: osdmap e394: 12 total, 12 up, 10 in 2024-09-11T03:43:32.293 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:32.316 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:32.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:32.418 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:32.444 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:32.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:32.445 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:32.546 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:32.568 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:32.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:32.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:32.670 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:32.696 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:32.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:32.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:32.798 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:32.811 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [6, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:43:32.812 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:43:32.812 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T03:43:32.812 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.646394 2024-09-11T03:43:32.812 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 1 0.6463937508416715 2024-09-11T03:43:32.824 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:32.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:32.825 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:32.840 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:43:32.841 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:43:32.863 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:43:32.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:43:32.867 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:32.881 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:43:32.882 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:43:32.901 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:43:32.901 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:43:32.905 INFO:tasks.rados.rados.2.smithi175.stdout:1437: done (1 left) 2024-09-11T03:43:32.905 INFO:tasks.rados.rados.2.smithi175.stdout:1438: done (0 left) 2024-09-11T03:43:32.905 INFO:tasks.rados.rados.2.smithi175.stdout:1439: write (excl) oid 16 current snap is 150 2024-09-11T03:43:32.905 INFO:tasks.rados.rados.2.smithi175.stdout:1439: seq_num 509 ranges {554763=781039,2102511=452252} 2024-09-11T03:43:32.912 INFO:tasks.rados.rados.2.smithi175.stdout:1439: writing smithi17557870-16 from 554763 to 1335802 tid 1 2024-09-11T03:43:32.919 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:43:32.920 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:43:32.920 INFO:tasks.rados.rados.2.smithi175.stdout:1439: writing smithi17557870-16 from 2102511 to 2554763 tid 2 2024-09-11T03:43:32.921 INFO:tasks.rados.rados.2.smithi175.stdout:1440: read oid 40 snap -1 2024-09-11T03:43:32.921 INFO:tasks.rados.rados.2.smithi175.stdout:1440: expect (ObjNum 487 snap 141 seq_num 487) 2024-09-11T03:43:32.921 INFO:tasks.rados.rados.2.smithi175.stdout:1441: read oid 4 snap -1 2024-09-11T03:43:32.921 INFO:tasks.rados.rados.2.smithi175.stdout:1441: expect deleted 2024-09-11T03:43:32.921 INFO:tasks.rados.rados.2.smithi175.stdout:1442: read oid 31 snap -1 2024-09-11T03:43:32.921 INFO:tasks.rados.rados.2.smithi175.stdout:1442: expect (ObjNum 475 snap 138 seq_num 475) 2024-09-11T03:43:32.921 INFO:tasks.rados.rados.2.smithi175.stdout:1443: snap_remove snap 146 2024-09-11T03:43:32.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:32.939 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:43:32.940 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:43:32.947 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:32.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:32.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:32.957 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:43:32.957 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:43:32.973 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:43:32.974 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:43:32.991 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:43:32.991 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:43:33.008 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:43:33.008 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:43:33.028 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:43:33.028 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:43:33.046 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:43:33.047 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:43:33.049 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:33.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:33.073 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:33.074 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:33.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:32 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:43:33.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:32 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:43:33.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:32 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:43:33.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:32 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:43:33.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:32 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4192460530' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:33.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:32 smithi084 ceph-mon[27254]: osdmap e395: 12 total, 12 up, 10 in 2024-09-11T03:43:33.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:32 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:43:33.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:32 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:43:33.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:32 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:43:33.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:32 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:43:33.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:32 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4192460530' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:33.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:32 smithi121 ceph-mon[30729]: osdmap e395: 12 total, 12 up, 10 in 2024-09-11T03:43:33.174 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:33.200 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:33.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:33.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:32 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:43:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:32 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:43:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:32 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:43:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:32 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:43:33.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:32 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4192460530' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:33.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:32 smithi175 ceph-mon[30894]: osdmap e395: 12 total, 12 up, 10 in 2024-09-11T03:43:33.302 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:33.325 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:33.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:33.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:33.427 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:33.452 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:33.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:33.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:33.580 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:33.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:33.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:33.682 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:33.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:33.706 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:33.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:33.807 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:33.833 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:33.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:33.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:33.924 INFO:tasks.rados.rados.2.smithi175.stdout:1443: done (4 left) 2024-09-11T03:43:33.925 INFO:tasks.rados.rados.2.smithi175.stdout:1444: copy_from oid 28 from oid 20 current snap is 150 2024-09-11T03:43:33.925 INFO:tasks.rados.rados.2.smithi175.stdout:1439: finishing write tid 1 to smithi17557870-16 2024-09-11T03:43:33.925 INFO:tasks.rados.rados.2.smithi175.stdout:1439: finishing write tid 2 to smithi17557870-16 2024-09-11T03:43:33.925 INFO:tasks.rados.rados.2.smithi175.stdout:1439: finishing write tid 3 to smithi17557870-16 2024-09-11T03:43:33.925 INFO:tasks.rados.rados.2.smithi175.stdout:1439: finishing write tid 4 to smithi17557870-16 2024-09-11T03:43:33.925 INFO:tasks.rados.rados.2.smithi175.stdout:1439: oid 16 updating version 0 to 460 2024-09-11T03:43:33.925 INFO:tasks.rados.rados.2.smithi175.stdout:1439: oid 16 version 460 is already newer than 459 2024-09-11T03:43:33.925 INFO:tasks.rados.rados.2.smithi175.stdout:1439: oid 16 updating version 460 to 461 2024-09-11T03:43:33.925 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 461 (ObjNum 509 snap 150 seq_num 509) dirty exists 2024-09-11T03:43:33.925 INFO:tasks.rados.rados.2.smithi175.stdout:1439: left oid 16 (ObjNum 509 snap 150 seq_num 509) 2024-09-11T03:43:33.925 INFO:tasks.rados.rados.2.smithi175.stdout:1440: expect (ObjNum 487 snap 141 seq_num 487) 2024-09-11T03:43:33.935 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:33.959 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:33.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:33.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:33.995 INFO:tasks.rados.rados.2.smithi175.stdout:1442: expect (ObjNum 475 snap 138 seq_num 475) 2024-09-11T03:43:34.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:34.086 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:34.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:34.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:34.107 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:33 smithi084 ceph-mon[27254]: pgmap v747: 57 pgs: 57 active+clean; 154 MiB data, 713 MiB used, 893 GiB / 894 GiB avail; 511 B/s rd, 562 KiB/s wr, 3 op/s 2024-09-11T03:43:34.128 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:43:34.128 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:43:34.187 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:34.209 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:34.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:34.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:34.223 INFO:tasks.rados.rados.2.smithi175.stdout:1444: finishing copy_from to smithi17557870-28 2024-09-11T03:43:34.223 INFO:tasks.rados.rados.2.smithi175.stdout:1444: got expected ENOENT (src dne) 2024-09-11T03:43:34.223 INFO:tasks.rados.rados.2.smithi175.stdout:1444: finishing copy_from racing read to smithi17557870-28 2024-09-11T03:43:34.223 INFO:tasks.rados.rados.2.smithi175.stdout:1439: done (4 left) 2024-09-11T03:43:34.223 INFO:tasks.rados.rados.2.smithi175.stdout:1440: done (3 left) 2024-09-11T03:43:34.224 INFO:tasks.rados.rados.2.smithi175.stdout:1441: done (2 left) 2024-09-11T03:43:34.224 INFO:tasks.rados.rados.2.smithi175.stdout:1442: done (1 left) 2024-09-11T03:43:34.224 INFO:tasks.rados.rados.2.smithi175.stdout:1444: done (0 left) 2024-09-11T03:43:34.224 INFO:tasks.rados.rados.2.smithi175.stdout:1445: rollback oid 44 current snap is 150 2024-09-11T03:43:34.224 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 44 to 147 2024-09-11T03:43:34.224 INFO:tasks.rados.rados.2.smithi175.stdout:1446: read oid 26 snap 148 2024-09-11T03:43:34.224 INFO:tasks.rados.rados.2.smithi175.stdout:1446: expect (ObjNum 460 snap 130 seq_num 460) 2024-09-11T03:43:34.224 INFO:tasks.rados.rados.2.smithi175.stdout:1447: read oid 50 snap -1 2024-09-11T03:43:34.224 INFO:tasks.rados.rados.2.smithi175.stdout:1447: expect (ObjNum 443 snap 130 seq_num 443) 2024-09-11T03:43:34.224 INFO:tasks.rados.rados.2.smithi175.stdout:1448: copy_from oid 49 from oid 42 current snap is 150 2024-09-11T03:43:34.224 INFO:tasks.rados.rados.2.smithi175.stdout:1449: snap_create 2024-09-11T03:43:34.224 INFO:tasks.rados.rados.2.smithi175.stdout:1448: finishing copy_from to smithi17557870-49 2024-09-11T03:43:34.224 INFO:tasks.rados.rados.2.smithi175.stdout:1448: got expected ENOENT (src dne) 2024-09-11T03:43:34.225 INFO:tasks.rados.rados.2.smithi175.stdout:1448: finishing copy_from racing read to smithi17557870-49 2024-09-11T03:43:34.225 INFO:tasks.rados.rados.2.smithi175.stdout:1445: finishing rollback tid 0 to smithi17557870-44 2024-09-11T03:43:34.226 INFO:tasks.rados.rados.2.smithi175.stdout:1445: finishing rollback tid 1 to smithi17557870-44 2024-09-11T03:43:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:33 smithi175 ceph-mon[30894]: pgmap v747: 57 pgs: 57 active+clean; 154 MiB data, 713 MiB used, 893 GiB / 894 GiB avail; 511 B/s rd, 562 KiB/s wr, 3 op/s 2024-09-11T03:43:34.227 INFO:tasks.rados.rados.2.smithi175.stdout:1445: finishing rollback tid 2 to smithi17557870-44 2024-09-11T03:43:34.227 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 430 (ObjNum 419 snap 128 seq_num 419) dirty exists 2024-09-11T03:43:34.240 INFO:tasks.rados.rados.2.smithi175.stdout:1447: expect (ObjNum 443 snap 130 seq_num 443) 2024-09-11T03:43:34.311 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:34.332 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:34.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:34.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:34.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:33 smithi121 ceph-mon[30729]: pgmap v747: 57 pgs: 57 active+clean; 154 MiB data, 713 MiB used, 893 GiB / 894 GiB avail; 511 B/s rd, 562 KiB/s wr, 3 op/s 2024-09-11T03:43:34.434 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:34.460 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:34.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:34.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:34.474 INFO:tasks.rados.rados.2.smithi175.stdout:1446: expect (ObjNum 460 snap 130 seq_num 460) 2024-09-11T03:43:34.562 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:34.583 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:34.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:34.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:34.599 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:43:34.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:34.706 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:34.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:34.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:34.808 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:34.829 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:34.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:34.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:34.847 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T03:43:34.875 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:34.907 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:34.931 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:34.933 INFO:tasks.rados.rados.2.smithi175.stdout:1445: done (4 left) 2024-09-11T03:43:34.933 INFO:tasks.rados.rados.2.smithi175.stdout:1446: done (3 left) 2024-09-11T03:43:34.933 INFO:tasks.rados.rados.2.smithi175.stdout:1447: done (2 left) 2024-09-11T03:43:34.933 INFO:tasks.rados.rados.2.smithi175.stdout:1448: done (1 left) 2024-09-11T03:43:34.933 INFO:tasks.rados.rados.2.smithi175.stdout:1449: done (0 left) 2024-09-11T03:43:34.933 INFO:tasks.rados.rados.2.smithi175.stdout:1450: write (excl) oid 4 current snap is 151 2024-09-11T03:43:34.933 INFO:tasks.rados.rados.2.smithi175.stdout:1450: seq_num 510 ranges {571570=572641,1877346=694224} 2024-09-11T03:43:34.938 INFO:tasks.rados.rados.2.smithi175.stdout:1450: writing smithi17557870-4 from 571570 to 1144211 tid 1 2024-09-11T03:43:34.946 INFO:tasks.rados.rados.2.smithi175.stdout:1450: writing smithi17557870-4 from 1877346 to 2571570 tid 2 2024-09-11T03:43:34.947 INFO:tasks.rados.rados.2.smithi175.stdout:1451: delete oid 5 current snap is 151 2024-09-11T03:43:34.947 INFO:tasks.rados.rados.2.smithi175.stdout:1450: finishing write tid 1 to smithi17557870-4 2024-09-11T03:43:34.950 INFO:tasks.rados.rados.2.smithi175.stdout:1450: finishing write tid 2 to smithi17557870-4 2024-09-11T03:43:34.950 INFO:tasks.rados.rados.2.smithi175.stdout:1450: finishing write tid 3 to smithi17557870-4 2024-09-11T03:43:34.950 INFO:tasks.rados.rados.2.smithi175.stdout:1451: done (1 left) 2024-09-11T03:43:34.950 INFO:tasks.rados.rados.2.smithi175.stdout:1452: read oid 46 snap -1 2024-09-11T03:43:34.950 INFO:tasks.rados.rados.2.smithi175.stdout:1452: expect (ObjNum 497 snap 144 seq_num 497) 2024-09-11T03:43:34.950 INFO:tasks.rados.rados.2.smithi175.stdout:1453: rollback oid 42 current snap is 151 2024-09-11T03:43:34.950 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 42 to 136 2024-09-11T03:43:34.951 INFO:tasks.rados.rados.2.smithi175.stdout:1450: finishing write tid 4 to smithi17557870-4 2024-09-11T03:43:34.951 INFO:tasks.rados.rados.2.smithi175.stdout:1450: oid 4 updating version 0 to 464 2024-09-11T03:43:34.951 INFO:tasks.rados.rados.2.smithi175.stdout:1450: oid 4 version 464 is already newer than 463 2024-09-11T03:43:34.951 INFO:tasks.rados.rados.2.smithi175.stdout:1450: oid 4 version 464 is already newer than 462 2024-09-11T03:43:34.951 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 464 (ObjNum 510 snap 151 seq_num 510) dirty exists 2024-09-11T03:43:34.951 INFO:tasks.rados.rados.2.smithi175.stdout:1450: left oid 4 (ObjNum 510 snap 151 seq_num 510) 2024-09-11T03:43:34.951 INFO:tasks.rados.rados.2.smithi175.stdout:1450: done (2 left) 2024-09-11T03:43:34.951 INFO:tasks.rados.rados.2.smithi175.stdout:1454: read oid 35 snap 149 2024-09-11T03:43:34.951 INFO:tasks.rados.rados.2.smithi175.stdout:1454: expect deleted 2024-09-11T03:43:34.951 INFO:tasks.rados.rados.2.smithi175.stdout:1455: snap_create 2024-09-11T03:43:34.952 INFO:tasks.rados.rados.2.smithi175.stdout:1453: finishing rollback tid 1 to smithi17557870-42 2024-09-11T03:43:34.952 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 0 (ObjNum 145 snap 0 seq_num 1745212672) dirty dne 2024-09-11T03:43:34.953 INFO:tasks.rados.rados.2.smithi175.stdout:1452: expect (ObjNum 497 snap 144 seq_num 497) 2024-09-11T03:43:34.955 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:34.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:34.956 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:35.057 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:35.078 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:35.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:35.079 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:35.142 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:34 smithi084 ceph-mon[27254]: osdmap e396: 12 total, 12 up, 10 in 2024-09-11T03:43:35.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:35.200 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:35.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:35.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:34 smithi175 ceph-mon[30894]: osdmap e396: 12 total, 12 up, 10 in 2024-09-11T03:43:35.302 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:35.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:35.324 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:35.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:35.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:34 smithi121 ceph-mon[30729]: osdmap e396: 12 total, 12 up, 10 in 2024-09-11T03:43:35.425 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:35.445 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:35.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:35.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:35.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:35.573 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:35.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:35.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:35.675 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:35.696 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:35.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:35.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:35.824 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:35.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:35.838 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:43:35.855 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:35.926 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:35.935 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.1 primary-affinity to 0.646394 (8423622) 2024-09-11T03:43:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:1452: done (3 left) 2024-09-11T03:43:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:1453: done (2 left) 2024-09-11T03:43:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:1454: done (1 left) 2024-09-11T03:43:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:1455: done (0 left) 2024-09-11T03:43:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:1456: delete oid 14 current snap is 152 2024-09-11T03:43:35.944 INFO:tasks.rados.rados.2.smithi175.stdout:1456: done (0 left) 2024-09-11T03:43:35.944 INFO:tasks.rados.rados.2.smithi175.stdout:1457: rollback oid 36 current snap is 152 2024-09-11T03:43:35.944 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 36 to 151 2024-09-11T03:43:35.944 INFO:tasks.rados.rados.2.smithi175.stdout:1458: write oid 20 current snap is 152 2024-09-11T03:43:35.944 INFO:tasks.rados.rados.2.smithi175.stdout:1458: seq_num 513 ranges {621991=798388,1936439=639509} 2024-09-11T03:43:35.949 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:35.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:35.950 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:35.952 INFO:tasks.rados.rados.2.smithi175.stdout:1458: writing smithi17557870-20 from 621991 to 1420379 tid 1 2024-09-11T03:43:35.958 INFO:tasks.rados.rados.2.smithi175.stdout:1458: writing smithi17557870-20 from 1936439 to 2575948 tid 2 2024-09-11T03:43:35.958 INFO:tasks.rados.rados.2.smithi175.stdout:1459: write oid 4 current snap is 152 2024-09-11T03:43:35.958 INFO:tasks.rados.rados.2.smithi175.stdout:1459: seq_num 514 ranges {638798=473637,1594921=783167} 2024-09-11T03:43:35.964 INFO:tasks.rados.rados.2.smithi175.stdout:1459: writing smithi17557870-4 from 638798 to 1112435 tid 1 2024-09-11T03:43:35.971 INFO:tasks.rados.rados.2.smithi175.stdout:1459: writing smithi17557870-4 from 1594921 to 2378088 tid 2 2024-09-11T03:43:35.971 INFO:tasks.rados.rados.2.smithi175.stdout:1460: write oid 8 current snap is 152 2024-09-11T03:43:35.971 INFO:tasks.rados.rados.2.smithi175.stdout:1460: seq_num 515 ranges {655605=548886,1653403=643178} 2024-09-11T03:43:35.977 INFO:tasks.rados.rados.2.smithi175.stdout:1460: writing smithi17557870-8 from 655605 to 1204491 tid 1 2024-09-11T03:43:35.984 INFO:tasks.rados.rados.2.smithi175.stdout:1460: writing smithi17557870-8 from 1653403 to 2296581 tid 2 2024-09-11T03:43:35.984 INFO:tasks.rados.rados.2.smithi175.stdout:1461: delete oid 13 current snap is 152 2024-09-11T03:43:35.984 INFO:tasks.rados.rados.2.smithi175.stdout:1457: finishing rollback tid 0 to smithi17557870-36 2024-09-11T03:43:35.984 INFO:tasks.rados.rados.2.smithi175.stdout:1457: finishing rollback tid 1 to smithi17557870-36 2024-09-11T03:43:35.984 INFO:tasks.rados.rados.2.smithi175.stdout:1457: finishing rollback tid 2 to smithi17557870-36 2024-09-11T03:43:35.984 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 284 (ObjNum 454 snap 130 seq_num 454) dirty exists 2024-09-11T03:43:35.985 INFO:tasks.rados.rados.2.smithi175.stdout:1458: finishing write tid 1 to smithi17557870-20 2024-09-11T03:43:35.985 INFO:tasks.rados.rados.2.smithi175.stdout:1458: finishing write tid 2 to smithi17557870-20 2024-09-11T03:43:35.985 INFO:tasks.rados.rados.2.smithi175.stdout:1458: finishing write tid 3 to smithi17557870-20 2024-09-11T03:43:35.985 INFO:tasks.rados.rados.2.smithi175.stdout:1458: finishing write tid 4 to smithi17557870-20 2024-09-11T03:43:35.985 INFO:tasks.rados.rados.2.smithi175.stdout:1458: oid 20 updating version 0 to 538 2024-09-11T03:43:35.985 INFO:tasks.rados.rados.2.smithi175.stdout:1458: oid 20 updating version 538 to 539 2024-09-11T03:43:35.985 INFO:tasks.rados.rados.2.smithi175.stdout:1458: oid 20 version 539 is already newer than 537 2024-09-11T03:43:35.985 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 539 (ObjNum 513 snap 152 seq_num 513) dirty exists 2024-09-11T03:43:35.985 INFO:tasks.rados.rados.2.smithi175.stdout:1458: left oid 20 (ObjNum 513 snap 152 seq_num 513) 2024-09-11T03:43:35.985 INFO:tasks.rados.rados.2.smithi175.stdout:1459: finishing write tid 1 to smithi17557870-4 2024-09-11T03:43:35.985 INFO:tasks.rados.rados.2.smithi175.stdout:1459: finishing write tid 2 to smithi17557870-4 2024-09-11T03:43:35.985 INFO:tasks.rados.rados.2.smithi175.stdout:1459: finishing write tid 3 to smithi17557870-4 2024-09-11T03:43:35.986 INFO:tasks.rados.rados.2.smithi175.stdout:1459: finishing write tid 4 to smithi17557870-4 2024-09-11T03:43:35.986 INFO:tasks.rados.rados.2.smithi175.stdout:1459: oid 4 updating version 0 to 467 2024-09-11T03:43:35.986 INFO:tasks.rados.rados.2.smithi175.stdout:1459: oid 4 version 467 is already newer than 465 2024-09-11T03:43:35.986 INFO:tasks.rados.rados.2.smithi175.stdout:1459: oid 4 version 467 is already newer than 466 2024-09-11T03:43:35.986 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 467 (ObjNum 514 snap 152 seq_num 514) dirty exists 2024-09-11T03:43:35.986 INFO:tasks.rados.rados.2.smithi175.stdout:1459: left oid 4 (ObjNum 514 snap 152 seq_num 514) 2024-09-11T03:43:35.986 INFO:tasks.rados.rados.2.smithi175.stdout:1460: finishing write tid 1 to smithi17557870-8 2024-09-11T03:43:35.986 INFO:tasks.rados.rados.2.smithi175.stdout:1457: done (4 left) 2024-09-11T03:43:35.987 INFO:tasks.rados.rados.2.smithi175.stdout:1458: done (3 left) 2024-09-11T03:43:35.987 INFO:tasks.rados.rados.2.smithi175.stdout:1459: done (2 left) 2024-09-11T03:43:35.987 INFO:tasks.rados.rados.2.smithi175.stdout:1461: done (1 left) 2024-09-11T03:43:35.987 INFO:tasks.rados.rados.2.smithi175.stdout:1462: snap_remove snap 148 2024-09-11T03:43:36.051 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:36.076 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:36.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:36.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:36.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:35 smithi084 ceph-mon[27254]: pgmap v750: 57 pgs: 57 active+clean; 154 MiB data, 714 MiB used, 893 GiB / 894 GiB avail; 3.9 MiB/s rd, 562 KiB/s wr, 7 op/s 2024-09-11T03:43:36.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:35 smithi084 ceph-mon[27254]: osdmap e397: 12 total, 12 up, 10 in 2024-09-11T03:43:36.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:35 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3432283783' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.6463937508416715} : dispatch 2024-09-11T03:43:36.178 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:36.200 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:36.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:36.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:35 smithi175 ceph-mon[30894]: pgmap v750: 57 pgs: 57 active+clean; 154 MiB data, 714 MiB used, 893 GiB / 894 GiB avail; 3.9 MiB/s rd, 562 KiB/s wr, 7 op/s 2024-09-11T03:43:36.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:35 smithi175 ceph-mon[30894]: osdmap e397: 12 total, 12 up, 10 in 2024-09-11T03:43:36.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:35 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3432283783' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.6463937508416715} : dispatch 2024-09-11T03:43:36.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:36.329 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:36.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:36.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:35 smithi121 ceph-mon[30729]: pgmap v750: 57 pgs: 57 active+clean; 154 MiB data, 714 MiB used, 893 GiB / 894 GiB avail; 3.9 MiB/s rd, 562 KiB/s wr, 7 op/s 2024-09-11T03:43:36.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:35 smithi121 ceph-mon[30729]: osdmap e397: 12 total, 12 up, 10 in 2024-09-11T03:43:36.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:35 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3432283783' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.6463937508416715} : dispatch 2024-09-11T03:43:36.432 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:36.453 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:36.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:36.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:36.555 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:36.556 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:43:36.557 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:43:36.581 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:36.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:36.683 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:36.707 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:36.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:36.708 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:36.809 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:36.832 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:36.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:36.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:36.903 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:43:36.906 INFO:tasks.rados.rados.2.smithi175.stdout:1462: done (1 left) 2024-09-11T03:43:36.906 INFO:tasks.rados.rados.2.smithi175.stdout:1463: copy_from oid 18 from oid 16 current snap is 152 2024-09-11T03:43:36.906 INFO:tasks.rados.rados.2.smithi175.stdout:1460: finishing write tid 2 to smithi17557870-8 2024-09-11T03:43:36.906 INFO:tasks.rados.rados.2.smithi175.stdout:1460: finishing write tid 3 to smithi17557870-8 2024-09-11T03:43:36.907 INFO:tasks.rados.rados.2.smithi175.stdout:1460: finishing write tid 4 to smithi17557870-8 2024-09-11T03:43:36.907 INFO:tasks.rados.rados.2.smithi175.stdout:1460: oid 8 updating version 0 to 557 2024-09-11T03:43:36.907 INFO:tasks.rados.rados.2.smithi175.stdout:1460: oid 8 version 557 is already newer than 556 2024-09-11T03:43:36.907 INFO:tasks.rados.rados.2.smithi175.stdout:1460: oid 8 updating version 557 to 558 2024-09-11T03:43:36.907 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 558 (ObjNum 515 snap 152 seq_num 515) dirty exists 2024-09-11T03:43:36.907 INFO:tasks.rados.rados.2.smithi175.stdout:1460: left oid 8 (ObjNum 515 snap 152 seq_num 515) 2024-09-11T03:43:36.907 INFO:tasks.rados.rados.2.smithi175.stdout:1460: done (1 left) 2024-09-11T03:43:36.907 INFO:tasks.rados.rados.2.smithi175.stdout:1464: rollback oid 45 current snap is 152 2024-09-11T03:43:36.907 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 45 to 149 2024-09-11T03:43:36.907 INFO:tasks.rados.rados.2.smithi175.stdout:1465: snap_remove snap 147 2024-09-11T03:43:36.934 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:36.956 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:36.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:36.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:37.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:37.083 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:37.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:37.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:37.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:37.209 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:37.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3432283783' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.6463937508416715}]': finished 2024-09-11T03:43:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:36 smithi175 ceph-mon[30894]: osdmap e398: 12 total, 12 up, 10 in 2024-09-11T03:43:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1440083238' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:36 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1440083238' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:36 smithi175 ceph-mon[30894]: osdmap e399: 12 total, 12 up, 10 in 2024-09-11T03:43:37.311 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:37.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:37.335 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:37.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:37.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3432283783' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.6463937508416715}]': finished 2024-09-11T03:43:37.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:36 smithi084 ceph-mon[27254]: osdmap e398: 12 total, 12 up, 10 in 2024-09-11T03:43:37.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1440083238' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:37.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:36 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:37.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1440083238' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:37.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:36 smithi084 ceph-mon[27254]: osdmap e399: 12 total, 12 up, 10 in 2024-09-11T03:43:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3432283783' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.6463937508416715}]': finished 2024-09-11T03:43:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:36 smithi121 ceph-mon[30729]: osdmap e398: 12 total, 12 up, 10 in 2024-09-11T03:43:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1440083238' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:36 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1440083238' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:36 smithi121 ceph-mon[30729]: osdmap e399: 12 total, 12 up, 10 in 2024-09-11T03:43:37.437 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:37.462 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:37.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:37.564 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:37.590 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:37.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:37.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:37.692 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:37.718 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:37.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:37.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:37.769 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T03:43:37.821 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:37.846 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:37.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:37.847 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:37.909 INFO:tasks.rados.rados.2.smithi175.stdout:1465: done (2 left) 2024-09-11T03:43:37.909 INFO:tasks.rados.rados.2.smithi175.stdout:1466: read oid 2 snap 136 2024-09-11T03:43:37.909 INFO:tasks.rados.rados.2.smithi175.stdout:1466: expect (ObjNum 459 snap 130 seq_num 459) 2024-09-11T03:43:37.909 INFO:tasks.rados.rados.2.smithi175.stdout:1467: write (excl) oid 41 current snap is 152 2024-09-11T03:43:37.910 INFO:tasks.rados.rados.2.smithi175.stdout:1464: finishing rollback tid 0 to smithi17557870-45 2024-09-11T03:43:37.910 INFO:tasks.rados.rados.2.smithi175.stdout:1464: finishing rollback tid 1 to smithi17557870-45 2024-09-11T03:43:37.910 INFO:tasks.rados.rados.2.smithi175.stdout:1464: finishing rollback tid 2 to smithi17557870-45 2024-09-11T03:43:37.910 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 462 (ObjNum 416 snap 127 seq_num 416) dirty exists 2024-09-11T03:43:37.910 INFO:tasks.rados.rados.2.smithi175.stdout:1463: finishing copy_from racing read to smithi17557870-18 2024-09-11T03:43:37.910 INFO:tasks.rados.rados.2.smithi175.stdout:1463: finishing copy_from to smithi17557870-18 2024-09-11T03:43:37.910 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 461 (ObjNum 509 snap 150 seq_num 509) dirty exists 2024-09-11T03:43:37.910 INFO:tasks.rados.rados.2.smithi175.stdout:1467: seq_num 517 ranges {689219=415737,1770367=646847} 2024-09-11T03:43:37.914 INFO:tasks.rados.rados.2.smithi175.stdout:1467: writing smithi17557870-41 from 689219 to 1104956 tid 1 2024-09-11T03:43:37.923 INFO:tasks.rados.rados.2.smithi175.stdout:1467: writing smithi17557870-41 from 1770367 to 2417214 tid 2 2024-09-11T03:43:37.923 INFO:tasks.rados.rados.2.smithi175.stdout:1463: done (3 left) 2024-09-11T03:43:37.923 INFO:tasks.rados.rados.2.smithi175.stdout:1464: done (2 left) 2024-09-11T03:43:37.923 INFO:tasks.rados.rados.2.smithi175.stdout:1468: write oid 46 current snap is 152 2024-09-11T03:43:37.923 INFO:tasks.rados.rados.2.smithi175.stdout:1468: seq_num 518 ranges {706026=490986,1828849=790505} 2024-09-11T03:43:37.929 INFO:tasks.rados.rados.2.smithi175.stdout:1468: writing smithi17557870-46 from 706026 to 1197012 tid 1 2024-09-11T03:43:37.937 INFO:tasks.rados.rados.2.smithi175.stdout:1468: writing smithi17557870-46 from 1828849 to 2619354 tid 2 2024-09-11T03:43:37.937 INFO:tasks.rados.rados.2.smithi175.stdout:1469: snap_create 2024-09-11T03:43:37.937 INFO:tasks.rados.rados.2.smithi175.stdout:1466: expect (ObjNum 459 snap 130 seq_num 459) 2024-09-11T03:43:37.948 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:37.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:37.971 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:37.971 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:38.042 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:37 smithi084 ceph-mon[27254]: pgmap v753: 57 pgs: 1 peering, 56 active+clean; 162 MiB data, 723 MiB used, 893 GiB / 894 GiB avail; 7.8 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-11T03:43:38.042 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:37 smithi084 ceph-mon[27254]: osdmap e400: 12 total, 12 up, 10 in 2024-09-11T03:43:38.072 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:38.096 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:38.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:38.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:38.198 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:38.205 INFO:tasks.rados.rados.2.smithi175.stdout:1467: finishing write tid 1 to smithi17557870-41 2024-09-11T03:43:38.205 INFO:tasks.rados.rados.2.smithi175.stdout:1467: finishing write tid 2 to smithi17557870-41 2024-09-11T03:43:38.205 INFO:tasks.rados.rados.2.smithi175.stdout:1467: finishing write tid 3 to smithi17557870-41 2024-09-11T03:43:38.205 INFO:tasks.rados.rados.2.smithi175.stdout:1467: finishing write tid 4 to smithi17557870-41 2024-09-11T03:43:38.205 INFO:tasks.rados.rados.2.smithi175.stdout:1467: oid 41 updating version 0 to 374 2024-09-11T03:43:38.205 INFO:tasks.rados.rados.2.smithi175.stdout:1467: oid 41 updating version 374 to 375 2024-09-11T03:43:38.205 INFO:tasks.rados.rados.2.smithi175.stdout:1467: oid 41 updating version 375 to 376 2024-09-11T03:43:38.206 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 376 (ObjNum 517 snap 152 seq_num 517) dirty exists 2024-09-11T03:43:38.206 INFO:tasks.rados.rados.2.smithi175.stdout:1467: left oid 41 (ObjNum 517 snap 152 seq_num 517) 2024-09-11T03:43:38.206 INFO:tasks.rados.rados.2.smithi175.stdout:1468: finishing write tid 1 to smithi17557870-46 2024-09-11T03:43:38.206 INFO:tasks.rados.rados.2.smithi175.stdout:1468: finishing write tid 2 to smithi17557870-46 2024-09-11T03:43:38.206 INFO:tasks.rados.rados.2.smithi175.stdout:1468: finishing write tid 3 to smithi17557870-46 2024-09-11T03:43:38.206 INFO:tasks.rados.rados.2.smithi175.stdout:1468: finishing write tid 4 to smithi17557870-46 2024-09-11T03:43:38.206 INFO:tasks.rados.rados.2.smithi175.stdout:1468: oid 46 updating version 0 to 193 2024-09-11T03:43:38.206 INFO:tasks.rados.rados.2.smithi175.stdout:1468: oid 46 version 193 is already newer than 191 2024-09-11T03:43:38.206 INFO:tasks.rados.rados.2.smithi175.stdout:1468: oid 46 version 193 is already newer than 192 2024-09-11T03:43:38.206 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 193 (ObjNum 518 snap 152 seq_num 518) dirty exists 2024-09-11T03:43:38.206 INFO:tasks.rados.rados.2.smithi175.stdout:1468: left oid 46 (ObjNum 518 snap 152 seq_num 518) 2024-09-11T03:43:38.218 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:38.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:38.219 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:38.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:37 smithi175 ceph-mon[30894]: pgmap v753: 57 pgs: 1 peering, 56 active+clean; 162 MiB data, 723 MiB used, 893 GiB / 894 GiB avail; 7.8 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-11T03:43:38.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:37 smithi175 ceph-mon[30894]: osdmap e400: 12 total, 12 up, 10 in 2024-09-11T03:43:38.320 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:38.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:37 smithi121 ceph-mon[30729]: pgmap v753: 57 pgs: 1 peering, 56 active+clean; 162 MiB data, 723 MiB used, 893 GiB / 894 GiB avail; 7.8 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-11T03:43:38.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:37 smithi121 ceph-mon[30729]: osdmap e400: 12 total, 12 up, 10 in 2024-09-11T03:43:38.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:38.345 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:38.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:38.446 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:38.467 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:38.468 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:38.511 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:38.533 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:38.568 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:38.590 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:38.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:38.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:38.692 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:38.713 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:38.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:38.714 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:38.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:38.837 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:38.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:38.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:38.909 INFO:tasks.rados.rados.2.smithi175.stdout:1466: done (3 left) 2024-09-11T03:43:38.909 INFO:tasks.rados.rados.2.smithi175.stdout:1467: done (2 left) 2024-09-11T03:43:38.909 INFO:tasks.rados.rados.2.smithi175.stdout:1468: done (1 left) 2024-09-11T03:43:38.909 INFO:tasks.rados.rados.2.smithi175.stdout:1469: done (0 left) 2024-09-11T03:43:38.909 INFO:tasks.rados.rados.2.smithi175.stdout:1470: rollback oid 24 current snap is 153 2024-09-11T03:43:38.909 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 149 2024-09-11T03:43:38.909 INFO:tasks.rados.rados.2.smithi175.stdout:1471: copy_from oid 8 from oid 47 current snap is 153 2024-09-11T03:43:38.909 INFO:tasks.rados.rados.2.smithi175.stdout:1472: snap_create 2024-09-11T03:43:38.912 INFO:tasks.rados.rados.2.smithi175.stdout:1470: finishing rollback tid 0 to smithi17557870-24 2024-09-11T03:43:38.913 INFO:tasks.rados.rados.2.smithi175.stdout:1470: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:43:38.913 INFO:tasks.rados.rados.2.smithi175.stdout:1470: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:43:38.913 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 566 (ObjNum 505 snap 146 seq_num 505) dirty exists 2024-09-11T03:43:38.913 INFO:tasks.rados.rados.2.smithi175.stdout:1471: finishing copy_from to smithi17557870-8 2024-09-11T03:43:38.913 INFO:tasks.rados.rados.2.smithi175.stdout:1471: got expected ENOENT (src dne) 2024-09-11T03:43:38.913 INFO:tasks.rados.rados.2.smithi175.stdout:1471: finishing copy_from racing read to smithi17557870-8 2024-09-11T03:43:38.938 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:38.961 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:38.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:38.962 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:39.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:38 smithi084 ceph-mon[27254]: 1.3 scrub starts 2024-09-11T03:43:39.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:38 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:43:39.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:38 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:43:39.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:38 smithi084 ceph-mon[27254]: osdmap e401: 12 total, 12 up, 10 in 2024-09-11T03:43:39.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:39.084 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:39.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:39.186 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:39.209 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:39.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:39.210 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:39.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:38 smithi175 ceph-mon[30894]: 1.3 scrub starts 2024-09-11T03:43:39.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:38 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:43:39.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:38 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:43:39.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:38 smithi175 ceph-mon[30894]: osdmap e401: 12 total, 12 up, 10 in 2024-09-11T03:43:39.311 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:39.315 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:43:39.315 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:43:39.315 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:43:39.315 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:43:39.321 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:43:39.321 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:43:39.332 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:39.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:39.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:43:39.338 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:43:39.355 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:43:39.355 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:43:39.371 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:43:39.371 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:43:39.387 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:43:39.388 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:43:39.404 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:43:39.404 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:43:39.421 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:43:39.421 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:43:39.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:38 smithi121 ceph-mon[30729]: 1.3 scrub starts 2024-09-11T03:43:39.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:38 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:43:39.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:38 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:43:39.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:38 smithi121 ceph-mon[30729]: osdmap e401: 12 total, 12 up, 10 in 2024-09-11T03:43:39.434 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:39.438 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:43:39.438 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:43:39.455 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:39.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:39.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:39.471 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:43:39.471 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:43:39.488 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:43:39.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:43:39.546 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:43:39.546 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:43:39.546 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:43:39.546 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:43:39.557 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:39.578 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:39.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:39.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:39.680 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:39.706 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:39.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:39.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:39.809 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:39.833 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:39.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:39.834 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:39.913 INFO:tasks.rados.rados.2.smithi175.stdout:1470: done (2 left) 2024-09-11T03:43:39.913 INFO:tasks.rados.rados.2.smithi175.stdout:1471: done (1 left) 2024-09-11T03:43:39.913 INFO:tasks.rados.rados.2.smithi175.stdout:1472: done (0 left) 2024-09-11T03:43:39.913 INFO:tasks.rados.rados.2.smithi175.stdout:1473: delete oid 48 current snap is 154 2024-09-11T03:43:39.913 INFO:tasks.rados.rados.2.smithi175.stdout:1473: done (0 left) 2024-09-11T03:43:39.914 INFO:tasks.rados.rados.2.smithi175.stdout:1474: write oid 1 current snap is 154 2024-09-11T03:43:39.914 INFO:tasks.rados.rados.2.smithi175.stdout:1474: seq_num 520 ranges {739640=641484,1829460=794174} 2024-09-11T03:43:39.920 INFO:tasks.rados.rados.2.smithi175.stdout:1474: writing smithi17557870-1 from 739640 to 1381124 tid 1 2024-09-11T03:43:39.927 INFO:tasks.rados.rados.2.smithi175.stdout:1474: writing smithi17557870-1 from 1829460 to 2623634 tid 2 2024-09-11T03:43:39.927 INFO:tasks.rados.rados.2.smithi175.stdout:1475: delete oid 12 current snap is 154 2024-09-11T03:43:39.928 INFO:tasks.rados.rados.2.smithi175.stdout:1474: finishing write tid 1 to smithi17557870-1 2024-09-11T03:43:39.932 INFO:tasks.rados.rados.2.smithi175.stdout:1474: finishing write tid 2 to smithi17557870-1 2024-09-11T03:43:39.932 INFO:tasks.rados.rados.2.smithi175.stdout:1474: finishing write tid 3 to smithi17557870-1 2024-09-11T03:43:39.932 INFO:tasks.rados.rados.2.smithi175.stdout:1474: finishing write tid 4 to smithi17557870-1 2024-09-11T03:43:39.932 INFO:tasks.rados.rados.2.smithi175.stdout:1474: oid 1 updating version 0 to 518 2024-09-11T03:43:39.932 INFO:tasks.rados.rados.2.smithi175.stdout:1474: oid 1 version 518 is already newer than 517 2024-09-11T03:43:39.932 INFO:tasks.rados.rados.2.smithi175.stdout:1474: oid 1 updating version 518 to 519 2024-09-11T03:43:39.933 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 519 (ObjNum 520 snap 154 seq_num 520) dirty exists 2024-09-11T03:43:39.933 INFO:tasks.rados.rados.2.smithi175.stdout:1474: left oid 1 (ObjNum 520 snap 154 seq_num 520) 2024-09-11T03:43:39.933 INFO:tasks.rados.rados.2.smithi175.stdout:1474: done (1 left) 2024-09-11T03:43:39.933 INFO:tasks.rados.rados.2.smithi175.stdout:1475: done (0 left) 2024-09-11T03:43:39.933 INFO:tasks.rados.rados.2.smithi175.stdout:1476: read oid 50 snap -1 2024-09-11T03:43:39.933 INFO:tasks.rados.rados.2.smithi175.stdout:1476: expect (ObjNum 443 snap 130 seq_num 443) 2024-09-11T03:43:39.933 INFO:tasks.rados.rados.2.smithi175.stdout:1477: read oid 31 snap 149 2024-09-11T03:43:39.933 INFO:tasks.rados.rados.2.smithi175.stdout:1477: expect (ObjNum 475 snap 138 seq_num 475) 2024-09-11T03:43:39.933 INFO:tasks.rados.rados.2.smithi175.stdout:1478: write oid 18 current snap is 154 2024-09-11T03:43:39.933 INFO:tasks.rados.rados.2.smithi175.stdout:1478: seq_num 522 ranges {773254=791982,2346424=426830} 2024-09-11T03:43:39.935 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:39.940 INFO:tasks.rados.rados.2.smithi175.stdout:1478: writing smithi17557870-18 from 773254 to 1565236 tid 1 2024-09-11T03:43:39.947 INFO:tasks.rados.rados.2.smithi175.stdout:1478: writing smithi17557870-18 from 2346424 to 2773254 tid 2 2024-09-11T03:43:39.947 INFO:tasks.rados.rados.2.smithi175.stdout:1479: snap_create 2024-09-11T03:43:39.947 INFO:tasks.rados.rados.2.smithi175.stdout:1476: expect (ObjNum 443 snap 130 seq_num 443) 2024-09-11T03:43:39.958 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:39.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:39.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:40.060 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:40.081 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:40.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:40.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:40.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:39 smithi084 ceph-mon[27254]: pgmap v756: 57 pgs: 1 peering, 56 active+clean; 163 MiB data, 717 MiB used, 893 GiB / 894 GiB avail; 10 MiB/s rd, 1.5 MiB/s wr, 15 op/s 2024-09-11T03:43:40.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:39 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3596350903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:40.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:39 smithi084 ceph-mon[27254]: osdmap e402: 12 total, 12 up, 10 in 2024-09-11T03:43:40.180 INFO:tasks.rados.rados.2.smithi175.stdout:1477: expect (ObjNum 475 snap 138 seq_num 475) 2024-09-11T03:43:40.182 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:40.203 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:40.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:40.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:40.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:39 smithi175 ceph-mon[30894]: pgmap v756: 57 pgs: 1 peering, 56 active+clean; 163 MiB data, 717 MiB used, 893 GiB / 894 GiB avail; 10 MiB/s rd, 1.5 MiB/s wr, 15 op/s 2024-09-11T03:43:40.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:39 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3596350903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:40.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:39 smithi175 ceph-mon[30894]: osdmap e402: 12 total, 12 up, 10 in 2024-09-11T03:43:40.305 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:40.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:40.328 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:40.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:40.408 INFO:tasks.rados.rados.2.smithi175.stdout:1478: finishing write tid 1 to smithi17557870-18 2024-09-11T03:43:40.408 INFO:tasks.rados.rados.2.smithi175.stdout:1478: finishing write tid 2 to smithi17557870-18 2024-09-11T03:43:40.408 INFO:tasks.rados.rados.2.smithi175.stdout:1478: finishing write tid 3 to smithi17557870-18 2024-09-11T03:43:40.408 INFO:tasks.rados.rados.2.smithi175.stdout:1478: finishing write tid 4 to smithi17557870-18 2024-09-11T03:43:40.408 INFO:tasks.rados.rados.2.smithi175.stdout:1478: oid 18 updating version 0 to 463 2024-09-11T03:43:40.408 INFO:tasks.rados.rados.2.smithi175.stdout:1478: oid 18 updating version 463 to 464 2024-09-11T03:43:40.409 INFO:tasks.rados.rados.2.smithi175.stdout:1478: oid 18 version 464 is already newer than 462 2024-09-11T03:43:40.409 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 464 (ObjNum 522 snap 154 seq_num 522) dirty exists 2024-09-11T03:43:40.409 INFO:tasks.rados.rados.2.smithi175.stdout:1478: left oid 18 (ObjNum 522 snap 154 seq_num 522) 2024-09-11T03:43:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:39 smithi121 ceph-mon[30729]: pgmap v756: 57 pgs: 1 peering, 56 active+clean; 163 MiB data, 717 MiB used, 893 GiB / 894 GiB avail; 10 MiB/s rd, 1.5 MiB/s wr, 15 op/s 2024-09-11T03:43:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:39 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3596350903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:39 smithi121 ceph-mon[30729]: osdmap e402: 12 total, 12 up, 10 in 2024-09-11T03:43:40.429 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:40.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:43:40.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:40.451 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:40.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:40.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:40.576 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:40.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:40.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:40.700 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:40.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:40.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:40.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:40.824 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:40.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:40.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:40.868 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T03:43:40.926 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:40.929 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:40.947 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:40.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:40.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:40.956 INFO:tasks.rados.rados.2.smithi175.stdout:1476: done (3 left) 2024-09-11T03:43:40.956 INFO:tasks.rados.rados.2.smithi175.stdout:1477: done (2 left) 2024-09-11T03:43:40.956 INFO:tasks.rados.rados.2.smithi175.stdout:1478: done (1 left) 2024-09-11T03:43:40.956 INFO:tasks.rados.rados.2.smithi175.stdout:1479: done (0 left) 2024-09-11T03:43:40.956 INFO:tasks.rados.rados.2.smithi175.stdout:1480: read oid 13 snap -1 2024-09-11T03:43:40.956 INFO:tasks.rados.rados.2.smithi175.stdout:1480: expect deleted 2024-09-11T03:43:40.956 INFO:tasks.rados.rados.2.smithi175.stdout:1481: delete oid 27 current snap is 155 2024-09-11T03:43:40.958 INFO:tasks.rados.rados.2.smithi175.stdout:1480: done (1 left) 2024-09-11T03:43:40.958 INFO:tasks.rados.rados.2.smithi175.stdout:1481: done (0 left) 2024-09-11T03:43:40.958 INFO:tasks.rados.rados.2.smithi175.stdout:1482: write oid 11 current snap is 155 2024-09-11T03:43:40.959 INFO:tasks.rados.rados.2.smithi175.stdout:1482: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2024-09-11T03:43:40.963 INFO:tasks.rados.rados.2.smithi175.stdout:1482: writing smithi17557870-11 from 406868 to 949348 tid 1 2024-09-11T03:43:40.969 INFO:tasks.rados.rados.2.smithi175.stdout:1482: writing smithi17557870-11 from 1547035 to 1948547 tid 2 2024-09-11T03:43:40.973 INFO:tasks.rados.rados.2.smithi175.stdout:1482: writing smithi17557870-11 from 2690354 to 2806868 tid 3 2024-09-11T03:43:40.973 INFO:tasks.rados.rados.2.smithi175.stdout:1483: snap_create 2024-09-11T03:43:40.974 INFO:tasks.rados.rados.2.smithi175.stdout:1482: finishing write tid 1 to smithi17557870-11 2024-09-11T03:43:40.974 INFO:tasks.rados.rados.2.smithi175.stdout:1482: finishing write tid 2 to smithi17557870-11 2024-09-11T03:43:40.975 INFO:tasks.rados.rados.2.smithi175.stdout:1482: finishing write tid 3 to smithi17557870-11 2024-09-11T03:43:40.975 INFO:tasks.rados.rados.2.smithi175.stdout:1482: finishing write tid 4 to smithi17557870-11 2024-09-11T03:43:40.975 INFO:tasks.rados.rados.2.smithi175.stdout:1482: finishing write tid 5 to smithi17557870-11 2024-09-11T03:43:40.976 INFO:tasks.rados.rados.2.smithi175.stdout:1482: oid 11 updating version 0 to 433 2024-09-11T03:43:40.976 INFO:tasks.rados.rados.2.smithi175.stdout:1482: oid 11 version 433 is already newer than 431 2024-09-11T03:43:40.976 INFO:tasks.rados.rados.2.smithi175.stdout:1482: oid 11 updating version 433 to 434 2024-09-11T03:43:40.976 INFO:tasks.rados.rados.2.smithi175.stdout:1482: oid 11 version 434 is already newer than 432 2024-09-11T03:43:40.976 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 434 (ObjNum 524 snap 155 seq_num 524) dirty exists 2024-09-11T03:43:40.976 INFO:tasks.rados.rados.2.smithi175.stdout:1482: left oid 11 (ObjNum 524 snap 155 seq_num 524) 2024-09-11T03:43:41.049 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:41.070 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:41.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:41.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:41.172 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:41.194 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:41.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:41.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:41.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:41.318 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:41.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:41.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:41.420 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:41.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:41.443 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:41.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:41.545 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:41.567 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:41.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:41.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:41.576 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:41.670 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:41.692 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:41.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:41.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:41.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:41.817 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:41.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:41.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:41.904 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:43:41.909 INFO:tasks.rados.rados.2.smithi175.stdout:1482: done (1 left) 2024-09-11T03:43:41.909 INFO:tasks.rados.rados.2.smithi175.stdout:1483: done (0 left) 2024-09-11T03:43:41.909 INFO:tasks.rados.rados.2.smithi175.stdout:1484: snap_remove snap 151 2024-09-11T03:43:41.919 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:41.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:41.941 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:41.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:41.970 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:43:41.970 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:43:41.970 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:43:41.970 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:43:41.970 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:43:42.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:42.066 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:42.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:42.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:42.168 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:42.173 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:43:42.193 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:42.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:42.194 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:41 smithi175 ceph-mon[30894]: pgmap v759: 57 pgs: 1 peering, 56 active+clean; 163 MiB data, 721 MiB used, 893 GiB / 894 GiB avail; 6.3 MiB/s rd, 308 KiB/s wr, 8 op/s; 16 KiB/s, 0 objects/s recovering 2024-09-11T03:43:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:41 smithi175 ceph-mon[30894]: osdmap e403: 12 total, 12 up, 10 in 2024-09-11T03:43:42.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:41 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1150810201' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:43:42.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:41 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:42.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:41 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1150810201' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:43:42.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:41 smithi175 ceph-mon[30894]: osdmap e404: 12 total, 12 up, 10 in 2024-09-11T03:43:42.296 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:42.321 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:42.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:42.322 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:42.385 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [6, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:43:42.385 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:43:42.385 INFO:tasks.thrashosds.thrasher:inject_pause on osd.11 2024-09-11T03:43:42.385 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T03:43:42.385 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T03:43:42.386 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 config set filestore_inject_stall 3 2024-09-11T03:43:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:41 smithi084 ceph-mon[27254]: pgmap v759: 57 pgs: 1 peering, 56 active+clean; 163 MiB data, 721 MiB used, 893 GiB / 894 GiB avail; 6.3 MiB/s rd, 308 KiB/s wr, 8 op/s; 16 KiB/s, 0 objects/s recovering 2024-09-11T03:43:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:41 smithi084 ceph-mon[27254]: osdmap e403: 12 total, 12 up, 10 in 2024-09-11T03:43:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:41 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1150810201' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:43:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:41 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:41 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1150810201' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:43:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:41 smithi084 ceph-mon[27254]: osdmap e404: 12 total, 12 up, 10 in 2024-09-11T03:43:42.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:41 smithi121 ceph-mon[30729]: pgmap v759: 57 pgs: 1 peering, 56 active+clean; 163 MiB data, 721 MiB used, 893 GiB / 894 GiB avail; 6.3 MiB/s rd, 308 KiB/s wr, 8 op/s; 16 KiB/s, 0 objects/s recovering 2024-09-11T03:43:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:41 smithi121 ceph-mon[30729]: osdmap e403: 12 total, 12 up, 10 in 2024-09-11T03:43:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:41 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1150810201' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:43:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:41 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:41 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1150810201' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:43:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:41 smithi121 ceph-mon[30729]: osdmap e404: 12 total, 12 up, 10 in 2024-09-11T03:43:42.424 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:42.453 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:42.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:42.454 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:42.555 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:42.581 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:42.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:42.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:42.640 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:43:42.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:42.704 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:42.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:42.807 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:42.828 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:42.829 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:42.913 INFO:tasks.rados.rados.2.smithi175.stdout:1484: done (0 left) 2024-09-11T03:43:42.913 INFO:tasks.rados.rados.2.smithi175.stdout:1485: copy_from oid 32 from oid 13 current snap is 156 2024-09-11T03:43:42.913 INFO:tasks.rados.rados.2.smithi175.stdout:1486: delete oid 5 current snap is 156 2024-09-11T03:43:42.914 INFO:tasks.rados.rados.2.smithi175.stdout:1485: finishing copy_from to smithi17557870-32 2024-09-11T03:43:42.914 INFO:tasks.rados.rados.2.smithi175.stdout:1485: got expected ENOENT (src dne) 2024-09-11T03:43:42.914 INFO:tasks.rados.rados.2.smithi175.stdout:1485: finishing copy_from racing read to smithi17557870-32 2024-09-11T03:43:42.916 INFO:tasks.rados.rados.2.smithi175.stdout:1485: done (1 left) 2024-09-11T03:43:42.916 INFO:tasks.rados.rados.2.smithi175.stdout:1486: done (0 left) 2024-09-11T03:43:42.917 INFO:tasks.rados.rados.2.smithi175.stdout:1487: write oid 39 current snap is 156 2024-09-11T03:43:42.917 INFO:tasks.rados.rados.2.smithi175.stdout:1487: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2024-09-11T03:43:42.921 INFO:tasks.rados.rados.2.smithi175.stdout:1487: writing smithi17557870-39 from 440482 to 849813 tid 1 2024-09-11T03:43:42.926 INFO:tasks.rados.rados.2.smithi175.stdout:1487: writing smithi17557870-39 from 1380352 to 1785533 tid 2 2024-09-11T03:43:42.930 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:42.933 INFO:tasks.rados.rados.2.smithi175.stdout:1487: writing smithi17557870-39 from 2461553 to 2840482 tid 3 2024-09-11T03:43:42.933 INFO:tasks.rados.rados.2.smithi175.stdout:1488: write oid 11 current snap is 156 2024-09-11T03:43:42.933 INFO:tasks.rados.rados.2.smithi175.stdout:1487: finishing write tid 1 to smithi17557870-39 2024-09-11T03:43:42.933 INFO:tasks.rados.rados.2.smithi175.stdout:1488: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2024-09-11T03:43:42.935 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T03:43:42.938 INFO:tasks.rados.rados.2.smithi175.stdout:1488: writing smithi17557870-11 from 457289 to 941869 tid 1 2024-09-11T03:43:42.943 INFO:tasks.rados.rados.2.smithi175.stdout:1488: writing smithi17557870-11 from 1438834 to 1987673 tid 2 2024-09-11T03:43:42.948 INFO:tasks.rados.rados.2.smithi175.stdout:1488: writing smithi17557870-11 from 2772623 to 2857289 tid 3 2024-09-11T03:43:42.948 INFO:tasks.rados.rados.2.smithi175.stdout:1489: copy_from oid 50 from oid 44 current snap is 156 2024-09-11T03:43:42.953 INFO:tasks.rados.rados.2.smithi175.stdout:1487: finishing write tid 2 to smithi17557870-39 2024-09-11T03:43:42.953 INFO:tasks.rados.rados.2.smithi175.stdout:1487: finishing write tid 3 to smithi17557870-39 2024-09-11T03:43:42.954 INFO:tasks.rados.rados.2.smithi175.stdout:1487: finishing write tid 4 to smithi17557870-39 2024-09-11T03:43:42.954 INFO:tasks.rados.rados.2.smithi175.stdout:1487: finishing write tid 5 to smithi17557870-39 2024-09-11T03:43:42.954 INFO:tasks.rados.rados.2.smithi175.stdout:1487: oid 39 updating version 0 to 383 2024-09-11T03:43:42.954 INFO:tasks.rados.rados.2.smithi175.stdout:1487: oid 39 version 383 is already newer than 381 2024-09-11T03:43:42.954 INFO:tasks.rados.rados.2.smithi175.stdout:1487: oid 39 version 383 is already newer than 382 2024-09-11T03:43:42.954 INFO:tasks.rados.rados.2.smithi175.stdout:1487: oid 39 version 383 is already newer than 380 2024-09-11T03:43:42.954 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 383 (ObjNum 526 snap 156 seq_num 526) dirty exists 2024-09-11T03:43:42.954 INFO:tasks.rados.rados.2.smithi175.stdout:1487: left oid 39 (ObjNum 526 snap 156 seq_num 526) 2024-09-11T03:43:42.954 INFO:tasks.rados.rados.2.smithi175.stdout:1488: finishing write tid 1 to smithi17557870-11 2024-09-11T03:43:42.954 INFO:tasks.rados.rados.2.smithi175.stdout:1488: finishing write tid 2 to smithi17557870-11 2024-09-11T03:43:42.954 INFO:tasks.rados.rados.2.smithi175.stdout:1487: done (2 left) 2024-09-11T03:43:42.955 INFO:tasks.rados.rados.2.smithi175.stdout:1490: read oid 25 snap -1 2024-09-11T03:43:42.955 INFO:tasks.rados.rados.2.smithi175.stdout:1490: expect (ObjNum 496 snap 144 seq_num 496) 2024-09-11T03:43:42.955 INFO:tasks.rados.rados.2.smithi175.stdout:1491: write oid 6 current snap is 156 2024-09-11T03:43:42.955 INFO:tasks.rados.rados.2.smithi175.stdout:1491: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2024-09-11T03:43:42.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:42.956 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:42.956 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:42.956 INFO:tasks.rados.rados.2.smithi175.stdout:1491: writing smithi17557870-6 from 474096 to 1033925 tid 1 2024-09-11T03:43:42.960 INFO:tasks.rados.rados.2.smithi175.stdout:1491: writing smithi17557870-6 from 1780963 to 2189813 tid 2 2024-09-11T03:43:42.964 INFO:tasks.rados.rados.2.smithi175.stdout:1491: writing smithi17557870-6 from 2800046 to 2874096 tid 3 2024-09-11T03:43:42.964 INFO:tasks.rados.rados.2.smithi175.stdout:1492: write oid 26 current snap is 156 2024-09-11T03:43:42.964 INFO:tasks.rados.rados.2.smithi175.stdout:1488: finishing write tid 3 to smithi17557870-11 2024-09-11T03:43:42.964 INFO:tasks.rados.rados.2.smithi175.stdout:1492: seq_num 529 ranges {490903=635078,1839445=552508} 2024-09-11T03:43:42.969 INFO:tasks.rados.rados.2.smithi175.stdout:1492: writing smithi17557870-26 from 490903 to 1125981 tid 1 2024-09-11T03:43:42.977 INFO:tasks.rados.rados.2.smithi175.stdout:1492: writing smithi17557870-26 from 1839445 to 2391953 tid 2 2024-09-11T03:43:42.977 INFO:tasks.rados.rados.2.smithi175.stdout:1493: snap_remove snap 154 2024-09-11T03:43:43.057 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:43.078 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:43.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:43.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:43.129 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:43:43.129 INFO:teuthology.orchestra.run.smithi175.stdout: "success": "" 2024-09-11T03:43:43.129 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:43:43.181 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:43.205 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:43.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:43.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:43.307 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:43.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:43.329 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:43.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:43.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:43.440 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:43.456 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:43.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:43.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:43.558 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:43.579 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:43.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:43.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:43.682 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:43.705 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:43.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:43.706 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:43.807 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:43.829 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:43.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:43.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:43.931 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:43.944 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:43:43.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:43:43.944 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:43 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:43:43.945 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:43 smithi084 ceph-mon[27254]: 2.b scrub ok 2024-09-11T03:43:43.945 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:43 smithi084 ceph-mon[27254]: pgmap v762: 57 pgs: 57 active+clean; 171 MiB data, 734 MiB used, 893 GiB / 894 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 11 op/s; 16 KiB/s, 0 objects/s recovering 2024-09-11T03:43:43.945 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:43 smithi084 ceph-mon[27254]: osdmap e405: 12 total, 12 up, 10 in 2024-09-11T03:43:43.953 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:43.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:43.954 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:43.954 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:43:43.955 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:43:43.961 INFO:tasks.rados.rados.2.smithi175.stdout:1493: done (5 left) 2024-09-11T03:43:43.961 INFO:tasks.rados.rados.2.smithi175.stdout:1494: read oid 39 snap -1 2024-09-11T03:43:43.961 INFO:tasks.rados.rados.2.smithi175.stdout:1494: expect (ObjNum 526 snap 156 seq_num 526) 2024-09-11T03:43:43.961 INFO:tasks.rados.rados.2.smithi175.stdout:1495: rollback oid 20 current snap is 156 2024-09-11T03:43:43.961 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 20 to 152 2024-09-11T03:43:43.961 INFO:tasks.rados.rados.2.smithi175.stdout:1488: finishing write tid 4 to smithi17557870-11 2024-09-11T03:43:43.961 INFO:tasks.rados.rados.2.smithi175.stdout:1488: finishing write tid 5 to smithi17557870-11 2024-09-11T03:43:43.961 INFO:tasks.rados.rados.2.smithi175.stdout:1488: oid 11 updating version 0 to 438 2024-09-11T03:43:43.961 INFO:tasks.rados.rados.2.smithi175.stdout:1488: oid 11 version 438 is already newer than 435 2024-09-11T03:43:43.961 INFO:tasks.rados.rados.2.smithi175.stdout:1488: oid 11 version 438 is already newer than 436 2024-09-11T03:43:43.962 INFO:tasks.rados.rados.2.smithi175.stdout:1488: oid 11 version 438 is already newer than 437 2024-09-11T03:43:43.962 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 438 (ObjNum 527 snap 156 seq_num 527) dirty exists 2024-09-11T03:43:43.962 INFO:tasks.rados.rados.2.smithi175.stdout:1488: left oid 11 (ObjNum 527 snap 156 seq_num 527) 2024-09-11T03:43:43.962 INFO:tasks.rados.rados.2.smithi175.stdout:1490: expect (ObjNum 496 snap 144 seq_num 496) 2024-09-11T03:43:43.979 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:43:43.979 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:43:43.984 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:44.002 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:43:44.002 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:43:44.020 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:43:44.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:43:44.037 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:43:44.037 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:43:44.053 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:43:44.054 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:43:44.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:44.065 INFO:tasks.rados.rados.2.smithi175.stdout:1491: finishing write tid 1 to smithi17557870-6 2024-09-11T03:43:44.065 INFO:tasks.rados.rados.2.smithi175.stdout:1489: finishing copy_from racing read to smithi17557870-50 2024-09-11T03:43:44.066 INFO:tasks.rados.rados.2.smithi175.stdout:1491: finishing write tid 2 to smithi17557870-6 2024-09-11T03:43:44.066 INFO:tasks.rados.rados.2.smithi175.stdout:1491: finishing write tid 3 to smithi17557870-6 2024-09-11T03:43:44.066 INFO:tasks.rados.rados.2.smithi175.stdout:1489: finishing copy_from to smithi17557870-50 2024-09-11T03:43:44.066 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 430 (ObjNum 419 snap 128 seq_num 419) dirty exists 2024-09-11T03:43:44.066 INFO:tasks.rados.rados.2.smithi175.stdout:1491: finishing write tid 4 to smithi17557870-6 2024-09-11T03:43:44.066 INFO:tasks.rados.rados.2.smithi175.stdout:1488: done (6 left) 2024-09-11T03:43:44.066 INFO:tasks.rados.rados.2.smithi175.stdout:1489: done (5 left) 2024-09-11T03:43:44.066 INFO:tasks.rados.rados.2.smithi175.stdout:1490: done (4 left) 2024-09-11T03:43:44.066 INFO:tasks.rados.rados.2.smithi175.stdout:1496: write (excl) oid 7 current snap is 156 2024-09-11T03:43:44.066 INFO:tasks.rados.rados.2.smithi175.stdout:1491: finishing write tid 5 to smithi17557870-6 2024-09-11T03:43:44.066 INFO:tasks.rados.rados.2.smithi175.stdout:1491: oid 6 updating version 0 to 409 2024-09-11T03:43:44.066 INFO:tasks.rados.rados.2.smithi175.stdout:1491: oid 6 updating version 409 to 412 2024-09-11T03:43:44.066 INFO:tasks.rados.rados.2.smithi175.stdout:1491: oid 6 version 412 is already newer than 411 2024-09-11T03:43:44.067 INFO:tasks.rados.rados.2.smithi175.stdout:1491: oid 6 version 412 is already newer than 410 2024-09-11T03:43:44.067 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 412 (ObjNum 528 snap 156 seq_num 528) dirty exists 2024-09-11T03:43:44.067 INFO:tasks.rados.rados.2.smithi175.stdout:1491: left oid 6 (ObjNum 528 snap 156 seq_num 528) 2024-09-11T03:43:44.067 INFO:tasks.rados.rados.2.smithi175.stdout:1492: finishing write tid 1 to smithi17557870-26 2024-09-11T03:43:44.067 INFO:tasks.rados.rados.2.smithi175.stdout:1496: seq_num 530 ranges {507710=710327,1897927=412519,2854892=52818} 2024-09-11T03:43:44.070 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:43:44.071 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:43:44.071 INFO:tasks.rados.rados.2.smithi175.stdout:1496: writing smithi17557870-7 from 507710 to 1218037 tid 1 2024-09-11T03:43:44.077 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:44.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:44.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:44.078 INFO:tasks.rados.rados.2.smithi175.stdout:1496: writing smithi17557870-7 from 1897927 to 2310446 tid 2 2024-09-11T03:43:44.082 INFO:tasks.rados.rados.2.smithi175.stdout:1496: writing smithi17557870-7 from 2854892 to 2907710 tid 3 2024-09-11T03:43:44.082 INFO:tasks.rados.rados.2.smithi175.stdout:1491: done (4 left) 2024-09-11T03:43:44.082 INFO:tasks.rados.rados.2.smithi175.stdout:1497: snap_create 2024-09-11T03:43:44.082 INFO:tasks.rados.rados.2.smithi175.stdout:1492: finishing write tid 2 to smithi17557870-26 2024-09-11T03:43:44.082 INFO:tasks.rados.rados.2.smithi175.stdout:1492: finishing write tid 3 to smithi17557870-26 2024-09-11T03:43:44.082 INFO:tasks.rados.rados.2.smithi175.stdout:1492: finishing write tid 4 to smithi17557870-26 2024-09-11T03:43:44.082 INFO:tasks.rados.rados.2.smithi175.stdout:1492: oid 26 updating version 0 to 248 2024-09-11T03:43:44.082 INFO:tasks.rados.rados.2.smithi175.stdout:1492: oid 26 updating version 248 to 249 2024-09-11T03:43:44.082 INFO:tasks.rados.rados.2.smithi175.stdout:1492: oid 26 updating version 249 to 250 2024-09-11T03:43:44.082 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 250 (ObjNum 529 snap 156 seq_num 529) dirty exists 2024-09-11T03:43:44.082 INFO:tasks.rados.rados.2.smithi175.stdout:1492: left oid 26 (ObjNum 529 snap 156 seq_num 529) 2024-09-11T03:43:44.082 INFO:tasks.rados.rados.2.smithi175.stdout:1495: finishing rollback tid 0 to smithi17557870-20 2024-09-11T03:43:44.083 INFO:tasks.rados.rados.2.smithi175.stdout:1495: finishing rollback tid 1 to smithi17557870-20 2024-09-11T03:43:44.083 INFO:tasks.rados.rados.2.smithi175.stdout:1494: expect (ObjNum 526 snap 156 seq_num 526) 2024-09-11T03:43:44.088 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:43:44.089 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:43:44.106 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:43:44.106 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:43:44.122 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:43:44.123 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:43:44.139 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:43:44.139 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:43:44.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:43 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:43:44.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:43 smithi121 ceph-mon[30729]: 2.b scrub ok 2024-09-11T03:43:44.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:43 smithi121 ceph-mon[30729]: pgmap v762: 57 pgs: 57 active+clean; 171 MiB data, 734 MiB used, 893 GiB / 894 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 11 op/s; 16 KiB/s, 0 objects/s recovering 2024-09-11T03:43:44.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:43 smithi121 ceph-mon[30729]: osdmap e405: 12 total, 12 up, 10 in 2024-09-11T03:43:44.178 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:44.200 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:44.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:44.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:43 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:43:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:43 smithi175 ceph-mon[30894]: 2.b scrub ok 2024-09-11T03:43:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:43 smithi175 ceph-mon[30894]: pgmap v762: 57 pgs: 57 active+clean; 171 MiB data, 734 MiB used, 893 GiB / 894 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 11 op/s; 16 KiB/s, 0 objects/s recovering 2024-09-11T03:43:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:43 smithi175 ceph-mon[30894]: osdmap e405: 12 total, 12 up, 10 in 2024-09-11T03:43:44.248 INFO:tasks.rados.rados.2.smithi175.stdout:1495: finishing rollback tid 2 to smithi17557870-20 2024-09-11T03:43:44.249 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 549 (ObjNum 513 snap 152 seq_num 513) dirty exists 2024-09-11T03:43:44.249 INFO:tasks.rados.rados.2.smithi175.stdout:1496: finishing write tid 1 to smithi17557870-7 2024-09-11T03:43:44.249 INFO:tasks.rados.rados.2.smithi175.stdout:1496: finishing write tid 2 to smithi17557870-7 2024-09-11T03:43:44.249 INFO:tasks.rados.rados.2.smithi175.stdout:1496: finishing write tid 3 to smithi17557870-7 2024-09-11T03:43:44.249 INFO:tasks.rados.rados.2.smithi175.stdout:1496: finishing write tid 4 to smithi17557870-7 2024-09-11T03:43:44.249 INFO:tasks.rados.rados.2.smithi175.stdout:1496: finishing write tid 5 to smithi17557870-7 2024-09-11T03:43:44.249 INFO:tasks.rados.rados.2.smithi175.stdout:1496: oid 7 updating version 0 to 468 2024-09-11T03:43:44.249 INFO:tasks.rados.rados.2.smithi175.stdout:1496: oid 7 version 468 is already newer than 467 2024-09-11T03:43:44.249 INFO:tasks.rados.rados.2.smithi175.stdout:1496: oid 7 updating version 468 to 469 2024-09-11T03:43:44.249 INFO:tasks.rados.rados.2.smithi175.stdout:1496: oid 7 version 469 is already newer than 466 2024-09-11T03:43:44.249 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 469 (ObjNum 530 snap 156 seq_num 530) dirty exists 2024-09-11T03:43:44.250 INFO:tasks.rados.rados.2.smithi175.stdout:1496: left oid 7 (ObjNum 530 snap 156 seq_num 530) 2024-09-11T03:43:44.301 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:44.323 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:44.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:44.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:44.425 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:44.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:44.448 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:44.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:44.549 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:44.571 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:44.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:44.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:44.673 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:44.696 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:44.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:44.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:44.798 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:44.824 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:44.825 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:44.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:44.947 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:44.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:44.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:44.964 INFO:tasks.rados.rados.2.smithi175.stdout:1492: done (4 left) 2024-09-11T03:43:44.964 INFO:tasks.rados.rados.2.smithi175.stdout:1494: done (3 left) 2024-09-11T03:43:44.964 INFO:tasks.rados.rados.2.smithi175.stdout:1495: done (2 left) 2024-09-11T03:43:44.964 INFO:tasks.rados.rados.2.smithi175.stdout:1496: done (1 left) 2024-09-11T03:43:44.964 INFO:tasks.rados.rados.2.smithi175.stdout:1497: done (0 left) 2024-09-11T03:43:44.964 INFO:tasks.rados.rados.2.smithi175.stdout:1498: snap_create 2024-09-11T03:43:44.985 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:44 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/100631810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:44.985 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:44 smithi084 ceph-mon[27254]: osdmap e406: 12 total, 12 up, 10 in 2024-09-11T03:43:45.050 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:45.073 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:45.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:45.074 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:45.093 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:43:45.094 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:43:45.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:44 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/100631810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:45.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:44 smithi121 ceph-mon[30729]: osdmap e406: 12 total, 12 up, 10 in 2024-09-11T03:43:45.174 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:45.197 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:45.197 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:45.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:44 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/100631810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:45.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:44 smithi175 ceph-mon[30894]: osdmap e406: 12 total, 12 up, 10 in 2024-09-11T03:43:45.298 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:45.320 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:45.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:45.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:45.444 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:45.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:45.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:45.456 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:43:45.546 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:45.570 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:45.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:45.572 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:45.672 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:45.697 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:45.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:45.698 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:45.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:45.826 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:45.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:45.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:45.840 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T03:43:45.903 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:45.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:45.953 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:45.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:45.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:45.966 INFO:tasks.rados.rados.2.smithi175.stdout:1498: done (0 left) 2024-09-11T03:43:45.967 INFO:tasks.rados.rados.2.smithi175.stdout:1499: write (excl) oid 37 current snap is 158 2024-09-11T03:43:45.967 INFO:tasks.rados.rados.2.smithi175.stdout:1499: seq_num 531 ranges {524517=785576,1956409=556177} 2024-09-11T03:43:45.973 INFO:tasks.rados.rados.2.smithi175.stdout:1499: writing smithi17557870-37 from 524517 to 1310093 tid 1 2024-09-11T03:43:45.980 INFO:tasks.rados.rados.2.smithi175.stdout:1499: writing smithi17557870-37 from 1956409 to 2512586 tid 2 2024-09-11T03:43:45.980 INFO:tasks.rados.rados.2.smithi175.stdout:1500: delete oid 48 current snap is 158 2024-09-11T03:43:45.980 INFO:tasks.rados.rados.2.smithi175.stdout:1499: finishing write tid 1 to smithi17557870-37 2024-09-11T03:43:45.984 INFO:tasks.rados.rados.2.smithi175.stdout:1500: done (1 left) 2024-09-11T03:43:45.984 INFO:tasks.rados.rados.2.smithi175.stdout:1501: write (excl) oid 21 current snap is 158 2024-09-11T03:43:45.984 INFO:tasks.rados.rados.2.smithi175.stdout:1501: seq_num 533 ranges {558131=652427,1673373=559846,2820808=137323} 2024-09-11T03:43:45.990 INFO:tasks.rados.rados.2.smithi175.stdout:1501: writing smithi17557870-21 from 558131 to 1210558 tid 1 2024-09-11T03:43:45.996 INFO:tasks.rados.rados.2.smithi175.stdout:1501: writing smithi17557870-21 from 1673373 to 2233219 tid 2 2024-09-11T03:43:46.000 INFO:tasks.rados.rados.2.smithi175.stdout:1501: writing smithi17557870-21 from 2820808 to 2958131 tid 3 2024-09-11T03:43:46.000 INFO:tasks.rados.rados.2.smithi175.stdout:1502: read oid 50 snap -1 2024-09-11T03:43:46.000 INFO:tasks.rados.rados.2.smithi175.stdout:1502: expect (ObjNum 419 snap 128 seq_num 419) 2024-09-11T03:43:46.000 INFO:tasks.rados.rados.2.smithi175.stdout:1503: delete oid 39 current snap is 158 2024-09-11T03:43:46.000 INFO:tasks.rados.rados.2.smithi175.stdout:1499: finishing write tid 2 to smithi17557870-37 2024-09-11T03:43:46.000 INFO:tasks.rados.rados.2.smithi175.stdout:1499: finishing write tid 3 to smithi17557870-37 2024-09-11T03:43:46.000 INFO:tasks.rados.rados.2.smithi175.stdout:1499: finishing write tid 4 to smithi17557870-37 2024-09-11T03:43:46.000 INFO:tasks.rados.rados.2.smithi175.stdout:1499: oid 37 updating version 0 to 363 2024-09-11T03:43:46.001 INFO:tasks.rados.rados.2.smithi175.stdout:1499: oid 37 updating version 363 to 365 2024-09-11T03:43:46.001 INFO:tasks.rados.rados.2.smithi175.stdout:1499: oid 37 version 365 is already newer than 364 2024-09-11T03:43:46.001 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 365 (ObjNum 531 snap 158 seq_num 531) dirty exists 2024-09-11T03:43:46.001 INFO:tasks.rados.rados.2.smithi175.stdout:1499: left oid 37 (ObjNum 531 snap 158 seq_num 531) 2024-09-11T03:43:46.001 INFO:tasks.rados.rados.2.smithi175.stdout:1501: finishing write tid 1 to smithi17557870-21 2024-09-11T03:43:46.001 INFO:tasks.rados.rados.2.smithi175.stdout:1501: finishing write tid 2 to smithi17557870-21 2024-09-11T03:43:46.002 INFO:tasks.rados.rados.2.smithi175.stdout:1499: done (3 left) 2024-09-11T03:43:46.002 INFO:tasks.rados.rados.2.smithi175.stdout:1503: done (2 left) 2024-09-11T03:43:46.002 INFO:tasks.rados.rados.2.smithi175.stdout:1504: delete oid 4 current snap is 158 2024-09-11T03:43:46.003 INFO:tasks.rados.rados.2.smithi175.stdout:1501: finishing write tid 3 to smithi17557870-21 2024-09-11T03:43:46.003 INFO:tasks.rados.rados.2.smithi175.stdout:1501: finishing write tid 4 to smithi17557870-21 2024-09-11T03:43:46.003 INFO:tasks.rados.rados.2.smithi175.stdout:1501: finishing write tid 5 to smithi17557870-21 2024-09-11T03:43:46.003 INFO:tasks.rados.rados.2.smithi175.stdout:1501: oid 21 updating version 0 to 521 2024-09-11T03:43:46.003 INFO:tasks.rados.rados.2.smithi175.stdout:1501: oid 21 updating version 521 to 523 2024-09-11T03:43:46.003 INFO:tasks.rados.rados.2.smithi175.stdout:1501: oid 21 version 523 is already newer than 522 2024-09-11T03:43:46.003 INFO:tasks.rados.rados.2.smithi175.stdout:1501: oid 21 version 523 is already newer than 520 2024-09-11T03:43:46.003 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 523 (ObjNum 533 snap 158 seq_num 533) dirty exists 2024-09-11T03:43:46.004 INFO:tasks.rados.rados.2.smithi175.stdout:1501: left oid 21 (ObjNum 533 snap 158 seq_num 533) 2024-09-11T03:43:46.004 INFO:tasks.rados.rados.2.smithi175.stdout:1501: done (2 left) 2024-09-11T03:43:46.005 INFO:tasks.rados.rados.2.smithi175.stdout:1504: done (1 left) 2024-09-11T03:43:46.005 INFO:tasks.rados.rados.2.smithi175.stdout:1505: read oid 40 snap -1 2024-09-11T03:43:46.005 INFO:tasks.rados.rados.2.smithi175.stdout:1505: expect (ObjNum 487 snap 141 seq_num 487) 2024-09-11T03:43:46.005 INFO:tasks.rados.rados.2.smithi175.stdout:1506: read oid 18 snap -1 2024-09-11T03:43:46.005 INFO:tasks.rados.rados.2.smithi175.stdout:1506: expect (ObjNum 522 snap 154 seq_num 522) 2024-09-11T03:43:46.005 INFO:tasks.rados.rados.2.smithi175.stdout:1507: read oid 41 snap -1 2024-09-11T03:43:46.005 INFO:tasks.rados.rados.2.smithi175.stdout:1507: expect (ObjNum 517 snap 152 seq_num 517) 2024-09-11T03:43:46.005 INFO:tasks.rados.rados.2.smithi175.stdout:1508: write oid 43 current snap is 158 2024-09-11T03:43:46.005 INFO:tasks.rados.rados.2.smithi175.stdout:1508: seq_num 536 ranges {608552=478174,1848819=707173} 2024-09-11T03:43:46.011 INFO:tasks.rados.rados.2.smithi175.stdout:1508: writing smithi17557870-43 from 608552 to 1086726 tid 1 2024-09-11T03:43:46.020 INFO:tasks.rados.rados.2.smithi175.stdout:1508: writing smithi17557870-43 from 1848819 to 2555992 tid 2 2024-09-11T03:43:46.020 INFO:tasks.rados.rados.2.smithi175.stdout:1509: snap_remove snap 152 2024-09-11T03:43:46.055 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:46.078 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:46.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:46.079 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:46.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:45 smithi084 ceph-mon[27254]: pgmap v765: 57 pgs: 57 active+clean; 173 MiB data, 740 MiB used, 893 GiB / 894 GiB avail; 6.0 MiB/s rd, 2.3 MiB/s wr, 14 op/s 2024-09-11T03:43:46.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:45 smithi084 ceph-mon[27254]: osdmap e407: 12 total, 12 up, 10 in 2024-09-11T03:43:46.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:46.204 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:46.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:46.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:45 smithi175 ceph-mon[30894]: pgmap v765: 57 pgs: 57 active+clean; 173 MiB data, 740 MiB used, 893 GiB / 894 GiB avail; 6.0 MiB/s rd, 2.3 MiB/s wr, 14 op/s 2024-09-11T03:43:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:45 smithi175 ceph-mon[30894]: osdmap e407: 12 total, 12 up, 10 in 2024-09-11T03:43:46.306 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:46.327 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:46.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:46.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:46.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:45 smithi121 ceph-mon[30729]: pgmap v765: 57 pgs: 57 active+clean; 173 MiB data, 740 MiB used, 893 GiB / 894 GiB avail; 6.0 MiB/s rd, 2.3 MiB/s wr, 14 op/s 2024-09-11T03:43:46.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:45 smithi121 ceph-mon[30729]: osdmap e407: 12 total, 12 up, 10 in 2024-09-11T03:43:46.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:46.430 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:46.455 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:46.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:46.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:46.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:46.582 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:46.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:46.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:46.684 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:46.709 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:46.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:46.711 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:46.812 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:46.837 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:46.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:46.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:46.886 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:43:46.905 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:43:46.911 INFO:tasks.rados.rados.2.smithi175.stdout:1509: done (5 left) 2024-09-11T03:43:46.911 INFO:tasks.rados.rados.2.smithi175.stdout:1510: read oid 19 snap -1 2024-09-11T03:43:46.911 INFO:tasks.rados.rados.2.smithi175.stdout:1510: expect deleted 2024-09-11T03:43:46.911 INFO:tasks.rados.rados.2.smithi175.stdout:1511: snap_create 2024-09-11T03:43:46.911 INFO:tasks.rados.rados.2.smithi175.stdout:1507: expect (ObjNum 517 snap 152 seq_num 517) 2024-09-11T03:43:46.915 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:43:46.915 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:43:46.939 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:46.962 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:46.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:46.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:47.064 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:47.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:47.086 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:47.086 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:47.103 INFO:tasks.rados.rados.2.smithi175.stdout:1505: expect (ObjNum 487 snap 141 seq_num 487) 2024-09-11T03:43:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:46 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:43:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:46 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:43:47.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:46 smithi084 ceph-mon[27254]: osdmap e408: 12 total, 12 up, 10 in 2024-09-11T03:43:47.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2688786969' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:47.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:46 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:43:47.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:46 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:43:47.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2688786969' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:47.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:46 smithi084 ceph-mon[27254]: osdmap e409: 12 total, 12 up, 10 in 2024-09-11T03:43:47.172 INFO:tasks.rados.rados.2.smithi175.stdout:1508: finishing write tid 1 to smithi17557870-43 2024-09-11T03:43:47.172 INFO:tasks.rados.rados.2.smithi175.stdout:1506: expect (ObjNum 522 snap 154 seq_num 522) 2024-09-11T03:43:47.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:47.208 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:47.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:47.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:46 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:43:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:46 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:43:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:46 smithi175 ceph-mon[30894]: osdmap e408: 12 total, 12 up, 10 in 2024-09-11T03:43:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2688786969' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:46 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:43:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:46 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:43:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2688786969' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:46 smithi175 ceph-mon[30894]: osdmap e409: 12 total, 12 up, 10 in 2024-09-11T03:43:47.291 INFO:tasks.rados.rados.2.smithi175.stdout:1502: expect (ObjNum 419 snap 128 seq_num 419) 2024-09-11T03:43:47.310 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:47.331 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:47.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:47.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:47.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:46 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:43:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:46 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:43:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:46 smithi121 ceph-mon[30729]: osdmap e408: 12 total, 12 up, 10 in 2024-09-11T03:43:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2688786969' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:46 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:43:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:46 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:43:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2688786969' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:46 smithi121 ceph-mon[30729]: osdmap e409: 12 total, 12 up, 10 in 2024-09-11T03:43:47.433 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:47.448 INFO:tasks.rados.rados.2.smithi175.stdout:1508: finishing write tid 2 to smithi17557870-43 2024-09-11T03:43:47.448 INFO:tasks.rados.rados.2.smithi175.stdout:1508: finishing write tid 3 to smithi17557870-43 2024-09-11T03:43:47.448 INFO:tasks.rados.rados.2.smithi175.stdout:1508: finishing write tid 4 to smithi17557870-43 2024-09-11T03:43:47.448 INFO:tasks.rados.rados.2.smithi175.stdout:1508: oid 43 updating version 0 to 393 2024-09-11T03:43:47.448 INFO:tasks.rados.rados.2.smithi175.stdout:1508: oid 43 version 393 is already newer than 392 2024-09-11T03:43:47.448 INFO:tasks.rados.rados.2.smithi175.stdout:1508: oid 43 version 393 is already newer than 391 2024-09-11T03:43:47.448 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 393 (ObjNum 536 snap 158 seq_num 536) dirty exists 2024-09-11T03:43:47.448 INFO:tasks.rados.rados.2.smithi175.stdout:1508: left oid 43 (ObjNum 536 snap 158 seq_num 536) 2024-09-11T03:43:47.454 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:47.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:47.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:47.556 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:47.577 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:47.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:47.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:47.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T03:43:47.679 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:47.700 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:47.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:47.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:47.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:47.828 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:47.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:47.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:47.912 INFO:tasks.rados.rados.2.smithi175.stdout:1502: done (6 left) 2024-09-11T03:43:47.912 INFO:tasks.rados.rados.2.smithi175.stdout:1505: done (5 left) 2024-09-11T03:43:47.913 INFO:tasks.rados.rados.2.smithi175.stdout:1506: done (4 left) 2024-09-11T03:43:47.913 INFO:tasks.rados.rados.2.smithi175.stdout:1507: done (3 left) 2024-09-11T03:43:47.913 INFO:tasks.rados.rados.2.smithi175.stdout:1508: done (2 left) 2024-09-11T03:43:47.913 INFO:tasks.rados.rados.2.smithi175.stdout:1510: done (1 left) 2024-09-11T03:43:47.913 INFO:tasks.rados.rados.2.smithi175.stdout:1511: done (0 left) 2024-09-11T03:43:47.913 INFO:tasks.rados.rados.2.smithi175.stdout:1512: write (excl) oid 37 current snap is 159 2024-09-11T03:43:47.913 INFO:tasks.rados.rados.2.smithi175.stdout:1512: seq_num 537 ranges {625359=553423,1790948=567184,2930500=94859} 2024-09-11T03:43:47.918 INFO:tasks.rados.rados.2.smithi175.stdout:1512: writing smithi17557870-37 from 625359 to 1178782 tid 1 2024-09-11T03:43:47.925 INFO:tasks.rados.rados.2.smithi175.stdout:1512: writing smithi17557870-37 from 1790948 to 2358132 tid 2 2024-09-11T03:43:47.929 INFO:tasks.rados.rados.2.smithi175.stdout:1512: writing smithi17557870-37 from 2930500 to 3025359 tid 3 2024-09-11T03:43:47.929 INFO:tasks.rados.rados.2.smithi175.stdout:1513: copy_from oid 35 from oid 46 current snap is 159 2024-09-11T03:43:47.929 INFO:tasks.rados.rados.2.smithi175.stdout:1512: finishing write tid 1 to smithi17557870-37 2024-09-11T03:43:47.929 INFO:tasks.rados.rados.2.smithi175.stdout:1512: finishing write tid 2 to smithi17557870-37 2024-09-11T03:43:47.929 INFO:tasks.rados.rados.2.smithi175.stdout:1514: delete oid 17 current snap is 159 2024-09-11T03:43:47.930 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:47.931 INFO:tasks.rados.rados.2.smithi175.stdout:1512: finishing write tid 3 to smithi17557870-37 2024-09-11T03:43:47.931 INFO:tasks.rados.rados.2.smithi175.stdout:1512: finishing write tid 4 to smithi17557870-37 2024-09-11T03:43:47.931 INFO:tasks.rados.rados.2.smithi175.stdout:1514: done (2 left) 2024-09-11T03:43:47.931 INFO:tasks.rados.rados.2.smithi175.stdout:1515: delete oid 13 current snap is 159 2024-09-11T03:43:47.931 INFO:tasks.rados.rados.2.smithi175.stdout:1512: finishing write tid 5 to smithi17557870-37 2024-09-11T03:43:47.931 INFO:tasks.rados.rados.2.smithi175.stdout:1512: oid 37 updating version 0 to 368 2024-09-11T03:43:47.932 INFO:tasks.rados.rados.2.smithi175.stdout:1512: oid 37 version 368 is already newer than 367 2024-09-11T03:43:47.932 INFO:tasks.rados.rados.2.smithi175.stdout:1512: oid 37 version 368 is already newer than 366 2024-09-11T03:43:47.932 INFO:tasks.rados.rados.2.smithi175.stdout:1512: oid 37 updating version 368 to 369 2024-09-11T03:43:47.932 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 369 (ObjNum 537 snap 159 seq_num 537) dirty exists 2024-09-11T03:43:47.932 INFO:tasks.rados.rados.2.smithi175.stdout:1512: left oid 37 (ObjNum 537 snap 159 seq_num 537) 2024-09-11T03:43:47.934 INFO:tasks.rados.rados.2.smithi175.stdout:1512: done (2 left) 2024-09-11T03:43:47.934 INFO:tasks.rados.rados.2.smithi175.stdout:1515: done (1 left) 2024-09-11T03:43:47.934 INFO:tasks.rados.rados.2.smithi175.stdout:1516: snap_remove snap 158 2024-09-11T03:43:47.951 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:47.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:47.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:48.053 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:48.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:48.076 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:48.076 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:48.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:48.203 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:48.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:48.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:48.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:47 smithi175 ceph-mon[30894]: pgmap v768: 57 pgs: 57 active+clean; 181 MiB data, 760 MiB used, 893 GiB / 894 GiB avail; 17 MiB/s rd, 2.4 MiB/s wr, 24 op/s 2024-09-11T03:43:48.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:47 smithi175 ceph-mon[30894]: osdmap e410: 12 total, 12 up, 10 in 2024-09-11T03:43:48.305 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:48.330 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:48.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:48.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:48.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:47 smithi084 ceph-mon[27254]: pgmap v768: 57 pgs: 57 active+clean; 181 MiB data, 760 MiB used, 893 GiB / 894 GiB avail; 17 MiB/s rd, 2.4 MiB/s wr, 24 op/s 2024-09-11T03:43:48.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:47 smithi084 ceph-mon[27254]: osdmap e410: 12 total, 12 up, 10 in 2024-09-11T03:43:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:47 smithi121 ceph-mon[30729]: pgmap v768: 57 pgs: 57 active+clean; 181 MiB data, 760 MiB used, 893 GiB / 894 GiB avail; 17 MiB/s rd, 2.4 MiB/s wr, 24 op/s 2024-09-11T03:43:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:47 smithi121 ceph-mon[30729]: osdmap e410: 12 total, 12 up, 10 in 2024-09-11T03:43:48.423 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:48.432 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:48.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:48.455 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:48.455 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:48.556 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:48.578 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:48.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:48.580 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:48.650 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:48.680 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:48.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:48.708 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:48.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:48.770 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11] out_osds: [6, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:43:48.770 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:43:48.770 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-09-11T03:43:48.771 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd in 10 2024-09-11T03:43:48.809 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:48.831 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:48.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:48.915 INFO:tasks.rados.rados.2.smithi175.stdout:1516: done (1 left) 2024-09-11T03:43:48.915 INFO:tasks.rados.rados.2.smithi175.stdout:1517: read oid 43 snap -1 2024-09-11T03:43:48.915 INFO:tasks.rados.rados.2.smithi175.stdout:1517: expect (ObjNum 536 snap 158 seq_num 536) 2024-09-11T03:43:48.915 INFO:tasks.rados.rados.2.smithi175.stdout:1518: read oid 27 snap -1 2024-09-11T03:43:48.915 INFO:tasks.rados.rados.2.smithi175.stdout:1518: expect deleted 2024-09-11T03:43:48.915 INFO:tasks.rados.rados.2.smithi175.stdout:1519: read oid 18 snap 155 2024-09-11T03:43:48.915 INFO:tasks.rados.rados.2.smithi175.stdout:1519: expect (ObjNum 522 snap 154 seq_num 522) 2024-09-11T03:43:48.915 INFO:tasks.rados.rados.2.smithi175.stdout:1520: snap_remove snap 153 2024-09-11T03:43:48.933 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:48.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:48.958 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:48.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:49.060 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:49.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:49.084 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:49.085 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:49.094 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:43:49.094 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:43:49.115 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:43:49.115 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:43:49.131 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:43:49.131 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:43:49.147 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:43:49.147 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:43:49.163 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:43:49.163 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:43:49.180 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:43:49.180 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:43:49.185 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:49.196 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:43:49.196 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:43:49.208 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:49.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:49.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:49.213 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:43:49.213 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:43:49.230 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:43:49.230 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:43:49.247 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:43:49.247 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:43:49.264 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:43:49.264 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:43:49.281 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:43:49.282 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:43:49.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:49.332 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:49.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:49.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:49.337 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:43:49.337 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:43:49.337 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:43:49.337 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:43:49.337 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:43:49.434 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:49.460 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:49.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:49.461 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:49.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:49.586 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:49.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:49.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:49.688 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:49.711 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:49.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:49.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:49.838 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:49.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:49.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:49.924 INFO:tasks.rados.rados.2.smithi175.stdout:1520: done (4 left) 2024-09-11T03:43:49.924 INFO:tasks.rados.rados.2.smithi175.stdout:1521: copy_from oid 40 from oid 12 current snap is 159 2024-09-11T03:43:49.924 INFO:tasks.rados.rados.2.smithi175.stdout:1513: finishing copy_from racing read to smithi17557870-35 2024-09-11T03:43:49.924 INFO:tasks.rados.rados.2.smithi175.stdout:1513: finishing copy_from to smithi17557870-35 2024-09-11T03:43:49.924 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 193 (ObjNum 518 snap 152 seq_num 518) dirty exists 2024-09-11T03:43:49.924 INFO:tasks.rados.rados.2.smithi175.stdout:1519: expect (ObjNum 522 snap 154 seq_num 522) 2024-09-11T03:43:49.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:49.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:49.964 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:49.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:50.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:49 smithi084 ceph-mon[27254]: pgmap v771: 57 pgs: 57 active+clean; 185 MiB data, 770 MiB used, 893 GiB / 894 GiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2024-09-11T03:43:50.017 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:49 smithi084 ceph-mon[27254]: osdmap e411: 12 total, 12 up, 10 in 2024-09-11T03:43:50.017 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/742685590' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:50.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T03:43:50.045 INFO:tasks.rados.rados.2.smithi175.stdout:1517: expect (ObjNum 536 snap 158 seq_num 536) 2024-09-11T03:43:50.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:50.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:50.089 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:50.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:49 smithi121 ceph-mon[30729]: pgmap v771: 57 pgs: 57 active+clean; 185 MiB data, 770 MiB used, 893 GiB / 894 GiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2024-09-11T03:43:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:49 smithi121 ceph-mon[30729]: osdmap e411: 12 total, 12 up, 10 in 2024-09-11T03:43:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/742685590' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:50.190 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:50.211 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:50.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:50.212 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:49 smithi175 ceph-mon[30894]: pgmap v771: 57 pgs: 57 active+clean; 185 MiB data, 770 MiB used, 893 GiB / 894 GiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2024-09-11T03:43:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:49 smithi175 ceph-mon[30894]: osdmap e411: 12 total, 12 up, 10 in 2024-09-11T03:43:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/742685590' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:50.249 INFO:tasks.rados.rados.2.smithi175.stdout:1521: finishing copy_from to smithi17557870-40 2024-09-11T03:43:50.249 INFO:tasks.rados.rados.2.smithi175.stdout:1521: got expected ENOENT (src dne) 2024-09-11T03:43:50.249 INFO:tasks.rados.rados.2.smithi175.stdout:1521: finishing copy_from racing read to smithi17557870-40 2024-09-11T03:43:50.249 INFO:tasks.rados.rados.2.smithi175.stdout:1513: done (4 left) 2024-09-11T03:43:50.249 INFO:tasks.rados.rados.2.smithi175.stdout:1517: done (3 left) 2024-09-11T03:43:50.249 INFO:tasks.rados.rados.2.smithi175.stdout:1518: done (2 left) 2024-09-11T03:43:50.250 INFO:tasks.rados.rados.2.smithi175.stdout:1519: done (1 left) 2024-09-11T03:43:50.250 INFO:tasks.rados.rados.2.smithi175.stdout:1521: done (0 left) 2024-09-11T03:43:50.250 INFO:tasks.rados.rados.2.smithi175.stdout:1522: write (excl) oid 3 current snap is 159 2024-09-11T03:43:50.250 INFO:tasks.rados.rados.2.smithi175.stdout:1522: seq_num 540 ranges {675780=495523,1682747=714511,3012769=63011} 2024-09-11T03:43:50.254 INFO:tasks.rados.rados.2.smithi175.stdout:1522: writing smithi17557870-3 from 675780 to 1171303 tid 1 2024-09-11T03:43:50.261 INFO:tasks.rados.rados.2.smithi175.stdout:1522: writing smithi17557870-3 from 1682747 to 2397258 tid 2 2024-09-11T03:43:50.265 INFO:tasks.rados.rados.2.smithi175.stdout:1522: writing smithi17557870-3 from 3012769 to 3075780 tid 3 2024-09-11T03:43:50.265 INFO:tasks.rados.rados.2.smithi175.stdout:1523: read oid 30 snap -1 2024-09-11T03:43:50.265 INFO:tasks.rados.rados.2.smithi175.stdout:1523: expect (ObjNum 435 snap 130 seq_num 435) 2024-09-11T03:43:50.265 INFO:tasks.rados.rados.2.smithi175.stdout:1524: write (excl) oid 34 current snap is 159 2024-09-11T03:43:50.265 INFO:tasks.rados.rados.2.smithi175.stdout:1522: finishing write tid 1 to smithi17557870-3 2024-09-11T03:43:50.265 INFO:tasks.rados.rados.2.smithi175.stdout:1522: finishing write tid 2 to smithi17557870-3 2024-09-11T03:43:50.266 INFO:tasks.rados.rados.2.smithi175.stdout:1524: seq_num 541 ranges {692587=570772,2024876=574522,3040192=52395} 2024-09-11T03:43:50.272 INFO:tasks.rados.rados.2.smithi175.stdout:1524: writing smithi17557870-34 from 692587 to 1263359 tid 1 2024-09-11T03:43:50.279 INFO:tasks.rados.rados.2.smithi175.stdout:1524: writing smithi17557870-34 from 2024876 to 2599398 tid 2 2024-09-11T03:43:50.283 INFO:tasks.rados.rados.2.smithi175.stdout:1524: writing smithi17557870-34 from 3040192 to 3092587 tid 3 2024-09-11T03:43:50.283 INFO:tasks.rados.rados.2.smithi175.stdout:1525: snap_create 2024-09-11T03:43:50.283 INFO:tasks.rados.rados.2.smithi175.stdout:1522: finishing write tid 3 to smithi17557870-3 2024-09-11T03:43:50.283 INFO:tasks.rados.rados.2.smithi175.stdout:1522: finishing write tid 4 to smithi17557870-3 2024-09-11T03:43:50.283 INFO:tasks.rados.rados.2.smithi175.stdout:1522: finishing write tid 5 to smithi17557870-3 2024-09-11T03:43:50.283 INFO:tasks.rados.rados.2.smithi175.stdout:1522: oid 3 updating version 0 to 556 2024-09-11T03:43:50.283 INFO:tasks.rados.rados.2.smithi175.stdout:1522: oid 3 updating version 556 to 558 2024-09-11T03:43:50.283 INFO:tasks.rados.rados.2.smithi175.stdout:1522: oid 3 version 558 is already newer than 557 2024-09-11T03:43:50.283 INFO:tasks.rados.rados.2.smithi175.stdout:1522: oid 3 version 558 is already newer than 555 2024-09-11T03:43:50.284 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 558 (ObjNum 540 snap 159 seq_num 540) dirty exists 2024-09-11T03:43:50.284 INFO:tasks.rados.rados.2.smithi175.stdout:1522: left oid 3 (ObjNum 540 snap 159 seq_num 540) 2024-09-11T03:43:50.284 INFO:tasks.rados.rados.2.smithi175.stdout:1524: finishing write tid 1 to smithi17557870-34 2024-09-11T03:43:50.284 INFO:tasks.rados.rados.2.smithi175.stdout:1523: expect (ObjNum 435 snap 130 seq_num 435) 2024-09-11T03:43:50.313 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:50.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:50.335 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:50.335 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:50.376 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:43:50.430 INFO:tasks.rados.rados.2.smithi175.stdout:1524: finishing write tid 2 to smithi17557870-34 2024-09-11T03:43:50.431 INFO:tasks.rados.rados.2.smithi175.stdout:1524: finishing write tid 3 to smithi17557870-34 2024-09-11T03:43:50.431 INFO:tasks.rados.rados.2.smithi175.stdout:1524: finishing write tid 4 to smithi17557870-34 2024-09-11T03:43:50.431 INFO:tasks.rados.rados.2.smithi175.stdout:1524: finishing write tid 5 to smithi17557870-34 2024-09-11T03:43:50.431 INFO:tasks.rados.rados.2.smithi175.stdout:1524: oid 34 updating version 0 to 257 2024-09-11T03:43:50.431 INFO:tasks.rados.rados.2.smithi175.stdout:1524: oid 34 updating version 257 to 259 2024-09-11T03:43:50.432 INFO:tasks.rados.rados.2.smithi175.stdout:1524: oid 34 updating version 259 to 260 2024-09-11T03:43:50.432 INFO:tasks.rados.rados.2.smithi175.stdout:1524: oid 34 version 260 is already newer than 258 2024-09-11T03:43:50.432 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 260 (ObjNum 541 snap 159 seq_num 541) dirty exists 2024-09-11T03:43:50.432 INFO:tasks.rados.rados.2.smithi175.stdout:1524: left oid 34 (ObjNum 541 snap 159 seq_num 541) 2024-09-11T03:43:50.437 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:50.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:50.460 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:50.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:50.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:50.582 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:50.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:50.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:50.684 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:50.706 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:50.707 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:50.791 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:50.807 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:50.830 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:50.830 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:50.833 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:50.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:50.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:50.924 INFO:tasks.rados.rados.2.smithi175.stdout:1522: done (3 left) 2024-09-11T03:43:50.924 INFO:tasks.rados.rados.2.smithi175.stdout:1523: done (2 left) 2024-09-11T03:43:50.924 INFO:tasks.rados.rados.2.smithi175.stdout:1524: done (1 left) 2024-09-11T03:43:50.924 INFO:tasks.rados.rados.2.smithi175.stdout:1525: done (0 left) 2024-09-11T03:43:50.924 INFO:tasks.rados.rados.2.smithi175.stdout:1526: snap_create 2024-09-11T03:43:50.935 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:50.959 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:50.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:50.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:51.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:50 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:43:51.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:50 smithi084 ceph-mon[27254]: 2.c scrub ok 2024-09-11T03:43:51.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:50 smithi084 ceph-mon[27254]: osdmap e412: 12 total, 12 up, 10 in 2024-09-11T03:43:51.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:50 smithi084 ceph-mon[27254]: 1.c scrub starts 2024-09-11T03:43:51.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:50 smithi084 ceph-mon[27254]: 1.c scrub ok 2024-09-11T03:43:51.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:50 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:43:51.061 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:50 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:43:51.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:51.084 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:51.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:51.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:51.211 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:51.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:51.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:50 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:43:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:50 smithi175 ceph-mon[30894]: 2.c scrub ok 2024-09-11T03:43:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:50 smithi175 ceph-mon[30894]: osdmap e412: 12 total, 12 up, 10 in 2024-09-11T03:43:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:50 smithi175 ceph-mon[30894]: 1.c scrub starts 2024-09-11T03:43:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:50 smithi175 ceph-mon[30894]: 1.c scrub ok 2024-09-11T03:43:51.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:50 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:43:51.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:50 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:43:51.313 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:51.334 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:51.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:51.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:50 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:43:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:50 smithi121 ceph-mon[30729]: 2.c scrub ok 2024-09-11T03:43:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:50 smithi121 ceph-mon[30729]: osdmap e412: 12 total, 12 up, 10 in 2024-09-11T03:43:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:50 smithi121 ceph-mon[30729]: 1.c scrub starts 2024-09-11T03:43:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:50 smithi121 ceph-mon[30729]: 1.c scrub ok 2024-09-11T03:43:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:50 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:43:51.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:50 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:43:51.435 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:51.459 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:51.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:51.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:51.562 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:51.588 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:51.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:51.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:51.632 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:51 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a[27250]: 2024-09-11T03:43:51.508+0000 7fae0fb7d640 -1 mon.a@0(leader).osd e413 definitely_dead 0 2024-09-11T03:43:51.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:51.716 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:51.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:51.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:51.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:51.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:51.840 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:51.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:51.906 INFO:teuthology.orchestra.run.smithi084.stderr:marked in osd.10. 2024-09-11T03:43:51.911 INFO:tasks.rados.rados.2.smithi175.stdout:1526: done (0 left) 2024-09-11T03:43:51.911 INFO:tasks.rados.rados.2.smithi175.stdout:1527: snap_remove snap 150 2024-09-11T03:43:51.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:51.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:51.964 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:51.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:51.969 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:43:51.969 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:43:51.969 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:43:51.969 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:43:51.969 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.970 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:640 2.a e131/95 2:5014e305:::smithi17557870-46:head v 131'5, mlcod=100'2)", 2024-09-11T03:43:51.970 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:43:51.970 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 283.85048804299998, 2024-09-11T03:43:51.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010809757999999999, 2024-09-11T03:43:51.970 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:51.970 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:51.970 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:51.970 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:51.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:51.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:43:51.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:51.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:51.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:43:51.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:51.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:51.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:43:51.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6100000000000002e-07 2024-09-11T03:43:51.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:51.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122019+0000", 2024-09-11T03:43:51.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0036004240000000001 2024-09-11T03:43:51.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:51.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122067+0000", 2024-09-11T03:43:51.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7589000000000001e-05 2024-09-11T03:43:51.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:51.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122077+0000", 2024-09-11T03:43:51.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0036e-05 2024-09-11T03:43:51.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:51.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122133+0000", 2024-09-11T03:43:51.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6350000000000001e-05 2024-09-11T03:43:51.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:51.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122152+0000", 2024-09-11T03:43:51.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8595999999999999e-05 2024-09-11T03:43:51.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:51.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129179+0000", 2024-09-11T03:43:51.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0070268079999999998 2024-09-11T03:43:51.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:51.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129228+0000", 2024-09-11T03:43:51.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9594e-05 2024-09-11T03:43:51.974 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:51.974 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:51.974 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:51.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.974 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:649 2.6 e131/95 2:6dbd1e81:::smithi17557870-7:head v 131'25, mlcod=123'21)", 2024-09-11T03:43:51.974 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:08.118731+0000", 2024-09-11T03:43:51.974 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 283.85017624099999, 2024-09-11T03:43:51.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01067091, 2024-09-11T03:43:51.975 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:51.975 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:51.975 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:51.975 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:51.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:51.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118731+0000", 2024-09-11T03:43:51.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:51.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:51.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118731+0000", 2024-09-11T03:43:51.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:51.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:51.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118731+0000", 2024-09-11T03:43:51.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4399999999999998e-07 2024-09-11T03:43:51.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:51.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.121866+0000", 2024-09-11T03:43:51.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031341720000000002 2024-09-11T03:43:51.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:51.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.121900+0000", 2024-09-11T03:43:51.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4266e-05 2024-09-11T03:43:51.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:51.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.121914+0000", 2024-09-11T03:43:51.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3668e-05 2024-09-11T03:43:51.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:51.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122002+0000", 2024-09-11T03:43:51.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7995e-05 2024-09-11T03:43:51.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:51.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122029+0000", 2024-09-11T03:43:51.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7199000000000002e-05 2024-09-11T03:43:51.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:51.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129354+0000", 2024-09-11T03:43:51.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073255899999999999 2024-09-11T03:43:51.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.979 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:51.979 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129401+0000", 2024-09-11T03:43:51.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7076000000000001e-05 2024-09-11T03:43:51.979 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:51.979 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:51.979 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:51.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.979 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1516 2.d e212/95 2:b6a25a79:::smithi17557870-47:head v 212'170, mlcod=206'166)", 2024-09-11T03:43:51.979 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:28.923076+0000", 2024-09-11T03:43:51.979 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 203.045831115, 2024-09-11T03:43:51.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081530539999999999, 2024-09-11T03:43:51.979 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:51.980 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:51.980 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:51.980 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:51.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:51.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.923076+0000", 2024-09-11T03:43:51.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:51.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:51.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.923076+0000", 2024-09-11T03:43:51.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:51.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:51.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.923076+0000", 2024-09-11T03:43:51.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1799999999999999e-07 2024-09-11T03:43:51.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:51.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927708+0000", 2024-09-11T03:43:51.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046320040000000003 2024-09-11T03:43:51.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:51.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927726+0000", 2024-09-11T03:43:51.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7730000000000001e-05 2024-09-11T03:43:51.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:51.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927734+0000", 2024-09-11T03:43:51.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2989999999999997e-06 2024-09-11T03:43:51.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:51.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927816+0000", 2024-09-11T03:43:51.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2256999999999995e-05 2024-09-11T03:43:51.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:51.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927842+0000", 2024-09-11T03:43:51.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5369999999999999e-05 2024-09-11T03:43:51.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:51.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.931178+0000", 2024-09-11T03:43:51.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033369039999999999 2024-09-11T03:43:51.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:51.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.931229+0000", 2024-09-11T03:43:51.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0272000000000003e-05 2024-09-11T03:43:51.983 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:51.983 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:51.983 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:51.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.984 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1524 2.6 e212/95 2:6dbd1e81:::smithi17557870-7:head v 212'123, mlcod=204'119)", 2024-09-11T03:43:51.984 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:28.927804+0000", 2024-09-11T03:43:51.984 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 203.041102663, 2024-09-11T03:43:51.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087752009999999998, 2024-09-11T03:43:51.984 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:51.984 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:51.985 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:51.985 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:51.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.985 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:51.985 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927804+0000", 2024-09-11T03:43:51.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:51.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.986 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:51.986 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927804+0000", 2024-09-11T03:43:51.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:51.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.986 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:51.986 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927804+0000", 2024-09-11T03:43:51.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8799999999999998e-07 2024-09-11T03:43:51.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:51.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929502+0000", 2024-09-11T03:43:51.987 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016974690000000001 2024-09-11T03:43:51.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:51.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929523+0000", 2024-09-11T03:43:51.987 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1373999999999999e-05 2024-09-11T03:43:51.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:51.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929543+0000", 2024-09-11T03:43:51.987 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9950999999999999e-05 2024-09-11T03:43:51.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.988 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:51.988 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929633+0000", 2024-09-11T03:43:51.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0158e-05 2024-09-11T03:43:51.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.988 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:51.988 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929657+0000", 2024-09-11T03:43:51.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3737999999999999e-05 2024-09-11T03:43:51.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.988 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:51.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.936522+0000", 2024-09-11T03:43:51.989 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068650589999999997 2024-09-11T03:43:51.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:51.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.936579+0000", 2024-09-11T03:43:51.989 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7163999999999997e-05 2024-09-11T03:43:51.989 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:51.989 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:51.989 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:51.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.989 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1589 2.7 e218/191 2:e77ce595:::smithi17557870-31:head v 218'174, mlcod=218'168)", 2024-09-11T03:43:51.989 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:35.116398+0000", 2024-09-11T03:43:51.990 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 196.85250833200001, 2024-09-11T03:43:51.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0089646050000000005, 2024-09-11T03:43:51.990 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:51.990 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:51.990 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:51.990 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:51.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.990 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:51.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116398+0000", 2024-09-11T03:43:51.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:51.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.990 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:51.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116398+0000", 2024-09-11T03:43:51.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:51.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:51.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116400+0000", 2024-09-11T03:43:51.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6109999999999999e-06 2024-09-11T03:43:51.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:51.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119261+0000", 2024-09-11T03:43:51.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028605309999999999 2024-09-11T03:43:51.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:51.992 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119281+0000", 2024-09-11T03:43:51.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0491000000000001e-05 2024-09-11T03:43:51.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:51.992 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119292+0000", 2024-09-11T03:43:51.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0759e-05 2024-09-11T03:43:51.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:51.992 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119350+0000", 2024-09-11T03:43:51.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8588999999999998e-05 2024-09-11T03:43:51.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:51.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119369+0000", 2024-09-11T03:43:51.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8227e-05 2024-09-11T03:43:51.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:51.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125320+0000", 2024-09-11T03:43:51.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059510409999999998 2024-09-11T03:43:51.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.994 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:51.994 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125363+0000", 2024-09-11T03:43:51.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3356000000000003e-05 2024-09-11T03:43:51.994 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:51.994 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:51.994 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:51.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.994 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1816 2.d e243/95 2:b7bdfe52:::smithi17557870-10:head v 243'237, mlcod=243'233)", 2024-09-11T03:43:51.994 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:59.922567+0000", 2024-09-11T03:43:51.994 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 172.04633998099999, 2024-09-11T03:43:51.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094601100000000007, 2024-09-11T03:43:51.995 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:51.995 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:51.995 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:51.995 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:51.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.995 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:51.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.922567+0000", 2024-09-11T03:43:51.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:51.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.995 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:51.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.922567+0000", 2024-09-11T03:43:51.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:51.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:51.996 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.922568+0000", 2024-09-11T03:43:51.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.345e-06 2024-09-11T03:43:51.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:51.996 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929634+0000", 2024-09-11T03:43:51.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0070657130000000004 2024-09-11T03:43:51.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:51.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929654+0000", 2024-09-11T03:43:51.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9972000000000001e-05 2024-09-11T03:43:51.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:51.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929663+0000", 2024-09-11T03:43:51.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3770000000000002e-06 2024-09-11T03:43:51.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:51.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929731+0000", 2024-09-11T03:43:51.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8095000000000004e-05 2024-09-11T03:43:51.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:51.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929759+0000", 2024-09-11T03:43:51.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7248999999999999e-05 2024-09-11T03:43:51.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:51.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.931991+0000", 2024-09-11T03:43:51.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022320679999999998 2024-09-11T03:43:51.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:51.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.932027+0000", 2024-09-11T03:43:51.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6291000000000002e-05 2024-09-11T03:43:51.999 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:51.999 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:51.999 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:51.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:51.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:51.999 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.11.0:202 2.a e279/95 2:5014e305:::smithi17557870-46:5c v 279'82, mlcod=276'78)", 2024-09-11T03:43:51.999 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:35.868642+0000", 2024-09-11T03:43:51.999 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 136.100264802, 2024-09-11T03:43:51.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084062019999999998, 2024-09-11T03:43:51.999 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:51.999 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:51.999 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:51.999 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:52.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:52.000 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.868642+0000", 2024-09-11T03:43:52.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:52.000 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.868642+0000", 2024-09-11T03:43:52.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:52.001 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.868642+0000", 2024-09-11T03:43:52.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.72e-07 2024-09-11T03:43:52.001 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.001 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.001 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:52.001 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.868646+0000", 2024-09-11T03:43:52.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.309e-06 2024-09-11T03:43:52.001 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.001 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.001 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:52.001 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.868654+0000", 2024-09-11T03:43:52.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7169999999999997e-06 2024-09-11T03:43:52.001 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.002 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.002 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:52.002 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.868660+0000", 2024-09-11T03:43:52.002 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5350000000000004e-06 2024-09-11T03:43:52.002 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.002 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.002 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.002 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.876082+0000", 2024-09-11T03:43:52.002 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074219009999999998 2024-09-11T03:43:52.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:52.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.876104+0000", 2024-09-11T03:43:52.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2568000000000001e-05 2024-09-11T03:43:52.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:52.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.876990+0000", 2024-09-11T03:43:52.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00088600599999999995 2024-09-11T03:43:52.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:52.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.877048+0000", 2024-09-11T03:43:52.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7893999999999998e-05 2024-09-11T03:43:52.005 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.005 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:52.005 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.005 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2431 2.12 2:4e5019e5:::smithi17557870-40:head [copy-from ver 279] snapc 66=[66,65,64,63,61,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-09-11T03:43:52.006 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:51.877192+0000", 2024-09-11T03:43:52.006 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 120.091714689, 2024-09-11T03:43:52.006 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013330089, 2024-09-11T03:43:52.006 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:52.006 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:52.006 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:52.006 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:43:52.006 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:43:52.006 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:52.006 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2431 2024-09-11T03:43:52.006 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.007 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:52.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:52.007 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877192+0000", 2024-09-11T03:43:52.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.007 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:52.007 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877192+0000", 2024-09-11T03:43:52.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.007 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:52.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877194+0000", 2024-09-11T03:43:52.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.767e-06 2024-09-11T03:43:52.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:52.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877202+0000", 2024-09-11T03:43:52.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0770000000000008e-06 2024-09-11T03:43:52.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:52.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877220+0000", 2024-09-11T03:43:52.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8561000000000001e-05 2024-09-11T03:43:52.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:52.009 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877230+0000", 2024-09-11T03:43:52.009 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2820000000000002e-06 2024-09-11T03:43:52.009 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.009 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.009 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877290+0000", 2024-09-11T03:43:52.009 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0025999999999998e-05 2024-09-11T03:43:52.009 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.009 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:52.009 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877404+0000", 2024-09-11T03:43:52.009 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011425999999999999 2024-09-11T03:43:52.009 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.010 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.010 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,11", 2024-09-11T03:43:52.010 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.884630+0000", 2024-09-11T03:43:52.010 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072256769999999998 2024-09-11T03:43:52.010 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.010 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.010 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:43:52.010 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.888559+0000", 2024-09-11T03:43:52.010 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039288750000000001 2024-09-11T03:43:52.010 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.010 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.010 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:52.010 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.888615+0000", 2024-09-11T03:43:52.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6609e-05 2024-09-11T03:43:52.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.011 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.011 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:52.011 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.890459+0000", 2024-09-11T03:43:52.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001844116 2024-09-11T03:43:52.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.011 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.011 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:52.011 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.890494+0000", 2024-09-11T03:43:52.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4921e-05 2024-09-11T03:43:52.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.011 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.012 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:52.012 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.890522+0000", 2024-09-11T03:43:52.012 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7917999999999999e-05 2024-09-11T03:43:52.012 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.012 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:52.012 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.012 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.012 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.012 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2432 2.12 2:4e5019e5:::smithi17557870-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e295)", 2024-09-11T03:43:52.012 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:51.877251+0000", 2024-09-11T03:43:52.012 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 120.09165603700001, 2024-09-11T03:43:52.012 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010600994000000001, 2024-09-11T03:43:52.012 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:52.013 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:52.013 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:43:52.013 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:43:52.013 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:43:52.013 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:52.013 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2432 2024-09-11T03:43:52.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.013 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:52.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.013 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:52.013 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877251+0000", 2024-09-11T03:43:52.013 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:52.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877251+0000", 2024-09-11T03:43:52.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:52.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877251+0000", 2024-09-11T03:43:52.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3399999999999999e-07 2024-09-11T03:43:52.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:52.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877254+0000", 2024-09-11T03:43:52.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.379e-06 2024-09-11T03:43:52.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:52.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877262+0000", 2024-09-11T03:43:52.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0779999999999996e-06 2024-09-11T03:43:52.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:52.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877265+0000", 2024-09-11T03:43:52.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.641e-06 2024-09-11T03:43:52.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877539+0000", 2024-09-11T03:43:52.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027397000000000001 2024-09-11T03:43:52.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:52.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877571+0000", 2024-09-11T03:43:52.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2122999999999997e-05 2024-09-11T03:43:52.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.887709+0000", 2024-09-11T03:43:52.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010137922000000001 2024-09-11T03:43:52.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.017 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:52.017 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.887781+0000", 2024-09-11T03:43:52.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1384999999999995e-05 2024-09-11T03:43:52.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.017 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:52.017 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.887852+0000", 2024-09-11T03:43:52.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0962000000000003e-05 2024-09-11T03:43:52.017 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.017 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:52.018 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.018 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.018 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.018 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2434 2.9 e295/276 2:939cf94d:::smithi17557870-28:head v 295'353, mlcod=295'349)", 2024-09-11T03:43:52.018 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:52.055871+0000", 2024-09-11T03:43:52.018 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 119.913035874, 2024-09-11T03:43:52.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083876999999999997, 2024-09-11T03:43:52.018 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:52.018 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:52.018 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:52.018 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:52.018 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.018 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:52.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.055871+0000", 2024-09-11T03:43:52.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:52.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.055871+0000", 2024-09-11T03:43:52.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:52.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.055873+0000", 2024-09-11T03:43:52.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9539999999999998e-06 2024-09-11T03:43:52.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:52.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.058061+0000", 2024-09-11T03:43:52.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002188204 2024-09-11T03:43:52.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:52.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.058085+0000", 2024-09-11T03:43:52.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4450000000000001e-05 2024-09-11T03:43:52.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:52.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.058096+0000", 2024-09-11T03:43:52.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.096e-05 2024-09-11T03:43:52.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.058183+0000", 2024-09-11T03:43:52.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6235000000000004e-05 2024-09-11T03:43:52.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:52.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.058220+0000", 2024-09-11T03:43:52.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7651000000000003e-05 2024-09-11T03:43:52.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:52.022 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.064221+0000", 2024-09-11T03:43:52.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060009599999999996 2024-09-11T03:43:52.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.022 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:52.022 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.064259+0000", 2024-09-11T03:43:52.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7286000000000002e-05 2024-09-11T03:43:52.022 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.022 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:52.022 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.022 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2443 2.12 2:494ba932:::smithi17557870-43:head [copy-from ver 342] snapc 66=[65,64,63,61,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-11T03:43:52.023 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:52.884558+0000", 2024-09-11T03:43:52.023 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 119.08434832, 2024-09-11T03:43:52.023 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019937844999999999, 2024-09-11T03:43:52.023 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:52.023 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:52.023 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:52.023 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:43:52.023 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:43:52.023 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:52.023 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2443 2024-09-11T03:43:52.023 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.023 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:52.023 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.023 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:52.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884558+0000", 2024-09-11T03:43:52.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:52.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884558+0000", 2024-09-11T03:43:52.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:52.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884560+0000", 2024-09-11T03:43:52.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7e-06 2024-09-11T03:43:52.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:52.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884568+0000", 2024-09-11T03:43:52.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0940000000000008e-06 2024-09-11T03:43:52.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:52.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884584+0000", 2024-09-11T03:43:52.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6277e-05 2024-09-11T03:43:52.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:52.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884593+0000", 2024-09-11T03:43:52.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1810000000000001e-06 2024-09-11T03:43:52.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884660+0000", 2024-09-11T03:43:52.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7362999999999999e-05 2024-09-11T03:43:52.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:52.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884731+0000", 2024-09-11T03:43:52.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1316999999999995e-05 2024-09-11T03:43:52.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,11", 2024-09-11T03:43:52.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.895614+0000", 2024-09-11T03:43:52.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010882911 2024-09-11T03:43:52.027 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.027 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:43:52.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.900723+0000", 2024-09-11T03:43:52.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005109143 2024-09-11T03:43:52.027 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.027 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:52.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.902758+0000", 2024-09-11T03:43:52.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020342870000000001 2024-09-11T03:43:52.027 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.028 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:52.028 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.904417+0000", 2024-09-11T03:43:52.028 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001658875 2024-09-11T03:43:52.028 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.028 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.028 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:52.028 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.904454+0000", 2024-09-11T03:43:52.028 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7883999999999998e-05 2024-09-11T03:43:52.028 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.028 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.028 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:52.028 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.904496+0000", 2024-09-11T03:43:52.028 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1813000000000001e-05 2024-09-11T03:43:52.029 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.029 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:52.029 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.029 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.029 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.029 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2444 2.12 2:494ba932:::smithi17557870-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e296)", 2024-09-11T03:43:52.029 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:52.884615+0000", 2024-09-11T03:43:52.029 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 119.084291759, 2024-09-11T03:43:52.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015042823, 2024-09-11T03:43:52.029 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:52.029 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:52.029 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:43:52.029 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:43:52.029 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:43:52.030 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:52.030 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2444 2024-09-11T03:43:52.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.030 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:52.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:52.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884615+0000", 2024-09-11T03:43:52.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:52.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884615+0000", 2024-09-11T03:43:52.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:52.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884616+0000", 2024-09-11T03:43:52.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5499999999999998e-07 2024-09-11T03:43:52.031 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:52.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884618+0000", 2024-09-11T03:43:52.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.818e-06 2024-09-11T03:43:52.031 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:52.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884632+0000", 2024-09-11T03:43:52.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3219e-05 2024-09-11T03:43:52.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:52.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884639+0000", 2024-09-11T03:43:52.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9990000000000002e-06 2024-09-11T03:43:52.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884918+0000", 2024-09-11T03:43:52.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027906699999999998 2024-09-11T03:43:52.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.033 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:52.033 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884959+0000", 2024-09-11T03:43:52.033 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1844999999999997e-05 2024-09-11T03:43:52.033 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.033 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.033 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.033 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.899489+0000", 2024-09-11T03:43:52.033 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014529034999999999 2024-09-11T03:43:52.033 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.033 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.033 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:52.033 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.899577+0000", 2024-09-11T03:43:52.033 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8512999999999998e-05 2024-09-11T03:43:52.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.034 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:52.034 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.899658+0000", 2024-09-11T03:43:52.034 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0772000000000005e-05 2024-09-11T03:43:52.034 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.034 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:52.034 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.034 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2513 2.19 2:98bb65f5:::smithi17557870-6:head [write 467327~633480] snapc 6c=[6c,6b,6a,69,68,65,63,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-09-11T03:43:52.034 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:01.874110+0000", 2024-09-11T03:43:52.034 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 110.09479639, 2024-09-11T03:43:52.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084441010000000007, 2024-09-11T03:43:52.035 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:52.035 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:52.035 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:52.035 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:43:52.035 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:43:52.035 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:52.035 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2513 2024-09-11T03:43:52.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.035 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:52.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:52.035 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.874110+0000", 2024-09-11T03:43:52.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.036 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:52.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.874110+0000", 2024-09-11T03:43:52.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.036 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:52.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.874112+0000", 2024-09-11T03:43:52.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.728e-06 2024-09-11T03:43:52.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:52.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.875529+0000", 2024-09-11T03:43:52.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001417153 2024-09-11T03:43:52.037 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.037 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:52.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.875542+0000", 2024-09-11T03:43:52.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3156000000000001e-05 2024-09-11T03:43:52.037 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.037 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:52.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.875551+0000", 2024-09-11T03:43:52.038 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3080000000000008e-06 2024-09-11T03:43:52.038 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.038 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.875732+0000", 2024-09-11T03:43:52.038 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018141199999999999 2024-09-11T03:43:52.038 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:52.038 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.875856+0000", 2024-09-11T03:43:52.038 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012354299999999999 2024-09-11T03:43:52.038 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,8", 2024-09-11T03:43:52.039 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.876030+0000", 2024-09-11T03:43:52.039 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000174079 2024-09-11T03:43:52.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.039 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.039 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:43:52.039 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.878440+0000", 2024-09-11T03:43:52.039 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024106829999999998 2024-09-11T03:43:52.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.039 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.039 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:52.039 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.880223+0000", 2024-09-11T03:43:52.039 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017821639999999999 2024-09-11T03:43:52.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.040 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.040 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:52.040 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.882437+0000", 2024-09-11T03:43:52.040 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022147030000000002 2024-09-11T03:43:52.040 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.040 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.040 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:52.040 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.882499+0000", 2024-09-11T03:43:52.040 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1747000000000004e-05 2024-09-11T03:43:52.040 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.040 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.040 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:52.040 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.882554+0000", 2024-09-11T03:43:52.041 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5424999999999999e-05 2024-09-11T03:43:52.041 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.041 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:52.041 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.041 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.041 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2568 2.d e314/95 2:b6a25a79:::smithi17557870-47:head v 314'392, mlcod=314'388)", 2024-09-11T03:43:52.041 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:10.885879+0000", 2024-09-11T03:43:52.041 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 101.083027915, 2024-09-11T03:43:52.041 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086201669999999998, 2024-09-11T03:43:52.041 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:52.041 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:52.041 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:52.041 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:52.042 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.042 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:52.042 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.885879+0000", 2024-09-11T03:43:52.042 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.042 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.042 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.042 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:52.042 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.885879+0000", 2024-09-11T03:43:52.042 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.042 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.042 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.042 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:52.043 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.885880+0000", 2024-09-11T03:43:52.043 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1900000000000002e-07 2024-09-11T03:43:52.043 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.043 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.043 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:52.043 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.892125+0000", 2024-09-11T03:43:52.043 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062450869999999999 2024-09-11T03:43:52.043 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.043 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.043 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:52.043 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.892137+0000", 2024-09-11T03:43:52.043 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2224e-05 2024-09-11T03:43:52.043 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.043 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.044 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:52.044 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.892143+0000", 2024-09-11T03:43:52.044 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.426e-06 2024-09-11T03:43:52.044 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.044 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.044 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.044 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.892200+0000", 2024-09-11T03:43:52.044 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6511999999999997e-05 2024-09-11T03:43:52.044 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.044 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.044 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:52.044 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.892218+0000", 2024-09-11T03:43:52.044 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.77e-05 2024-09-11T03:43:52.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:52.045 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.894465+0000", 2024-09-11T03:43:52.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022472159999999998 2024-09-11T03:43:52.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:52.045 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.894499+0000", 2024-09-11T03:43:52.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4282999999999997e-05 2024-09-11T03:43:52.045 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.045 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:52.045 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.046 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.046 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.046 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2641 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 350] snapc 75=[75,74,73,71,6f,6c,68,65,5d] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-11T03:43:52.046 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:19.887918+0000", 2024-09-11T03:43:52.046 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 92.080988465000004, 2024-09-11T03:43:52.046 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011895905, 2024-09-11T03:43:52.047 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:52.047 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:52.047 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:52.047 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:43:52.047 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:43:52.047 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:52.047 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2641 2024-09-11T03:43:52.047 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.047 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:52.048 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.048 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:52.048 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887918+0000", 2024-09-11T03:43:52.048 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.048 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.048 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.048 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:52.048 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887918+0000", 2024-09-11T03:43:52.048 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.048 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.048 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.048 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:52.048 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887920+0000", 2024-09-11T03:43:52.049 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.762e-06 2024-09-11T03:43:52.049 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.049 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.049 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:52.049 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887928+0000", 2024-09-11T03:43:52.049 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8900000000000007e-06 2024-09-11T03:43:52.049 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.049 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.049 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:52.049 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887945+0000", 2024-09-11T03:43:52.049 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6707000000000001e-05 2024-09-11T03:43:52.049 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.050 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.050 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:52.050 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887953+0000", 2024-09-11T03:43:52.050 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5620000000000004e-06 2024-09-11T03:43:52.050 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.050 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.050 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.050 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.888014+0000", 2024-09-11T03:43:52.050 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0798000000000003e-05 2024-09-11T03:43:52.050 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.050 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:43:52.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.888019+0000", 2024-09-11T03:43:52.051 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5990000000000004e-06 2024-09-11T03:43:52.051 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.051 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.889290+0000", 2024-09-11T03:43:52.051 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012718009999999999 2024-09-11T03:43:52.051 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.051 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:52.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.889393+0000", 2024-09-11T03:43:52.051 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102298 2024-09-11T03:43:52.051 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.052 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.052 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,8", 2024-09-11T03:43:52.052 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.894120+0000", 2024-09-11T03:43:52.052 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047277259999999998 2024-09-11T03:43:52.052 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.052 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.052 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:43:52.052 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.897232+0000", 2024-09-11T03:43:52.052 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031110859999999999 2024-09-11T03:43:52.052 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.052 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.052 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:52.052 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.897530+0000", 2024-09-11T03:43:52.053 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000298342 2024-09-11T03:43:52.053 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.053 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.053 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:52.053 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.899728+0000", 2024-09-11T03:43:52.053 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021984819999999999 2024-09-11T03:43:52.053 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.053 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.053 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:52.053 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.899782+0000", 2024-09-11T03:43:52.053 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3922999999999999e-05 2024-09-11T03:43:52.053 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.053 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.054 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:52.054 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.899814+0000", 2024-09-11T03:43:52.054 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1928999999999999e-05 2024-09-11T03:43:52.054 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.054 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:52.054 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.054 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.054 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.054 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2642 2.19 2:98bb65f5:::smithi17557870-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e323)", 2024-09-11T03:43:52.054 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:19.887977+0000", 2024-09-11T03:43:52.054 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 92.080929913999995, 2024-09-11T03:43:52.054 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085743450000000006, 2024-09-11T03:43:52.054 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:52.054 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:52.055 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:43:52.055 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:43:52.055 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:43:52.055 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:52.055 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2642 2024-09-11T03:43:52.055 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.055 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:52.055 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.055 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:52.055 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887977+0000", 2024-09-11T03:43:52.056 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.056 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.056 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.056 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:52.056 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887977+0000", 2024-09-11T03:43:52.056 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.056 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.056 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.056 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:52.057 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887977+0000", 2024-09-11T03:43:52.057 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9699999999999996e-07 2024-09-11T03:43:52.057 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.057 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.057 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:52.057 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887980+0000", 2024-09-11T03:43:52.057 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4700000000000001e-06 2024-09-11T03:43:52.057 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.057 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.057 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:52.057 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887986+0000", 2024-09-11T03:43:52.057 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1140000000000002e-06 2024-09-11T03:43:52.057 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.057 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.058 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:52.058 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887991+0000", 2024-09-11T03:43:52.058 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3730000000000001e-06 2024-09-11T03:43:52.058 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.058 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.058 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.058 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.888058+0000", 2024-09-11T03:43:52.058 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6322999999999995e-05 2024-09-11T03:43:52.058 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.058 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.058 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:43:52.058 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.888062+0000", 2024-09-11T03:43:52.058 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7600000000000002e-06 2024-09-11T03:43:52.059 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.059 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.059 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.059 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.889535+0000", 2024-09-11T03:43:52.059 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001472391 2024-09-11T03:43:52.059 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.059 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.059 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:52.059 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.889580+0000", 2024-09-11T03:43:52.059 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5139000000000003e-05 2024-09-11T03:43:52.059 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.059 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.059 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.059 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.896410+0000", 2024-09-11T03:43:52.060 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068302379999999998 2024-09-11T03:43:52.060 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.060 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.060 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:52.060 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.896478+0000", 2024-09-11T03:43:52.060 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8159999999999998e-05 2024-09-11T03:43:52.060 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.060 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.060 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:52.060 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.896551+0000", 2024-09-11T03:43:52.060 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2780000000000005e-05 2024-09-11T03:43:52.060 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.060 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:52.061 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.061 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.061 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.061 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2944 2.12 2:494ba932:::smithi17557870-43:head [copy-from ver 311] snapc 80=[80,7b,7a,79,77,75,6c] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-11T03:43:52.061 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:43.962947+0000", 2024-09-11T03:43:52.061 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 68.005959265000001, 2024-09-11T03:43:52.061 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014367049, 2024-09-11T03:43:52.061 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:52.061 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:52.061 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:52.061 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:43:52.062 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:43:52.062 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:52.062 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2944 2024-09-11T03:43:52.062 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.062 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:52.062 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.062 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:52.062 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962947+0000", 2024-09-11T03:43:52.063 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.063 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.063 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.063 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:52.063 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962947+0000", 2024-09-11T03:43:52.063 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.063 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.063 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.063 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:52.063 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962949+0000", 2024-09-11T03:43:52.063 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.689e-06 2024-09-11T03:43:52.063 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.064 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.064 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:52.064 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962955+0000", 2024-09-11T03:43:52.064 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.297e-06 2024-09-11T03:43:52.064 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.064 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.064 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:52.064 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962967+0000", 2024-09-11T03:43:52.064 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1517e-05 2024-09-11T03:43:52.064 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.064 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.064 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:52.065 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962975+0000", 2024-09-11T03:43:52.065 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2889999999999998e-06 2024-09-11T03:43:52.065 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.065 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.065 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.065 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963875+0000", 2024-09-11T03:43:52.065 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00089981200000000005 2024-09-11T03:43:52.065 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.065 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.065 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:52.065 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.964025+0000", 2024-09-11T03:43:52.065 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000149624 2024-09-11T03:43:52.066 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.066 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.066 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,7", 2024-09-11T03:43:52.066 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.971400+0000", 2024-09-11T03:43:52.066 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073748349999999997 2024-09-11T03:43:52.066 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.066 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.066 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:43:52.066 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.975907+0000", 2024-09-11T03:43:52.066 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045076159999999999 2024-09-11T03:43:52.066 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.066 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.066 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:52.066 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.976808+0000", 2024-09-11T03:43:52.067 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00090078700000000001 2024-09-11T03:43:52.067 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.067 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.067 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:52.067 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.977244+0000", 2024-09-11T03:43:52.067 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00043565800000000001 2024-09-11T03:43:52.067 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.067 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.067 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:52.067 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.977281+0000", 2024-09-11T03:43:52.067 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7150999999999997e-05 2024-09-11T03:43:52.067 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.067 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.068 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:52.068 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.977315+0000", 2024-09-11T03:43:52.068 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3773999999999999e-05 2024-09-11T03:43:52.068 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.068 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:52.068 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.068 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.068 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.068 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2945 2.12 2:494ba932:::smithi17557870-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e347)", 2024-09-11T03:43:52.068 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:43.963009+0000", 2024-09-11T03:43:52.068 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 68.005897665000006, 2024-09-11T03:43:52.068 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012192595000000001, 2024-09-11T03:43:52.069 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:52.069 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:52.069 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:43:52.069 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:43:52.069 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:43:52.069 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:52.069 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2945 2024-09-11T03:43:52.069 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.069 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:52.069 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.069 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:52.069 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963009+0000", 2024-09-11T03:43:52.069 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.070 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.070 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.070 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:52.070 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963009+0000", 2024-09-11T03:43:52.070 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.070 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.070 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.070 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:52.070 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963010+0000", 2024-09-11T03:43:52.070 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0699999999999997e-07 2024-09-11T03:43:52.070 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.070 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.070 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:52.071 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963012+0000", 2024-09-11T03:43:52.071 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6599999999999999e-06 2024-09-11T03:43:52.071 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.071 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.071 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:52.071 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963020+0000", 2024-09-11T03:43:52.071 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2649999999999999e-06 2024-09-11T03:43:52.071 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.071 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.071 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:52.071 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963025+0000", 2024-09-11T03:43:52.071 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6860000000000003e-06 2024-09-11T03:43:52.071 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.072 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.072 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.072 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.964185+0000", 2024-09-11T03:43:52.072 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011599329999999999 2024-09-11T03:43:52.072 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.072 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.072 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:52.072 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.964230+0000", 2024-09-11T03:43:52.072 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4336999999999999e-05 2024-09-11T03:43:52.072 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.072 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.072 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.072 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.974986+0000", 2024-09-11T03:43:52.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010755928999999999 2024-09-11T03:43:52.073 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.073 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.073 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:52.073 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.975091+0000", 2024-09-11T03:43:52.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105701 2024-09-11T03:43:52.073 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.073 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.073 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:52.073 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.975202+0000", 2024-09-11T03:43:52.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011047700000000001 2024-09-11T03:43:52.073 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.074 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:52.074 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.074 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:3536 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 408] snapc 90=[90,8f,8d,8c,8b,8a] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-11T03:43:52.074 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:43:18.903548+0000", 2024-09-11T03:43:52.074 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 33.065358332999999, 2024-09-11T03:43:52.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014015196000000001, 2024-09-11T03:43:52.074 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:52.074 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:52.074 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:43:52.074 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:43:52.074 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:43:52.074 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:52.075 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3536 2024-09-11T03:43:52.075 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.075 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:52.075 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.075 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:52.075 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903548+0000", 2024-09-11T03:43:52.075 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.075 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.075 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.075 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:52.075 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903548+0000", 2024-09-11T03:43:52.075 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.075 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.076 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.076 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:52.076 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903550+0000", 2024-09-11T03:43:52.076 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.499e-06 2024-09-11T03:43:52.076 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.076 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.076 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:52.076 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903555+0000", 2024-09-11T03:43:52.076 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6890000000000003e-06 2024-09-11T03:43:52.076 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.076 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.077 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:52.077 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903567+0000", 2024-09-11T03:43:52.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1912999999999999e-05 2024-09-11T03:43:52.077 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.077 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.077 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:52.077 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903574+0000", 2024-09-11T03:43:52.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7260000000000002e-06 2024-09-11T03:43:52.077 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.077 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.077 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.077 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904043+0000", 2024-09-11T03:43:52.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00046877799999999999 2024-09-11T03:43:52.078 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.078 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.078 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:52.078 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904136+0000", 2024-09-11T03:43:52.078 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2745999999999996e-05 2024-09-11T03:43:52.078 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.078 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.078 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,8", 2024-09-11T03:43:52.078 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.912436+0000", 2024-09-11T03:43:52.078 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083000049999999992 2024-09-11T03:43:52.078 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.078 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.078 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:43:52.078 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.915813+0000", 2024-09-11T03:43:52.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033772839999999999 2024-09-11T03:43:52.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:52.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.916094+0000", 2024-09-11T03:43:52.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028135600000000002 2024-09-11T03:43:52.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:43:52.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.917498+0000", 2024-09-11T03:43:52.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014033680000000001 2024-09-11T03:43:52.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:43:52.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.917534+0000", 2024-09-11T03:43:52.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6180000000000003e-05 2024-09-11T03:43:52.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.080 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.080 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:52.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.917564+0000", 2024-09-11T03:43:52.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9652e-05 2024-09-11T03:43:52.080 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.080 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:52.080 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.080 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.080 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:3537 2.19 2:98bb65f5:::smithi17557870-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e381)", 2024-09-11T03:43:52.081 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:43:52.081 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 33.065310861999997, 2024-09-11T03:43:52.081 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011559349999999999, 2024-09-11T03:43:52.081 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:43:52.081 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:43:52.081 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:43:52.081 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:43:52.081 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:43:52.081 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:43:52.081 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3537 2024-09-11T03:43:52.081 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.081 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:43:52.081 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:43:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:43:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:43:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:43:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:43:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:43:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:43:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8899999999999999e-07 2024-09-11T03:43:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:43:52.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903599+0000", 2024-09-11T03:43:52.083 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3750000000000001e-06 2024-09-11T03:43:52.083 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:43:52.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903606+0000", 2024-09-11T03:43:52.083 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1969999999999999e-06 2024-09-11T03:43:52.083 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:43:52.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903611+0000", 2024-09-11T03:43:52.083 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3680000000000001e-06 2024-09-11T03:43:52.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.084 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.084 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.084 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904244+0000", 2024-09-11T03:43:52.084 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00063279999999999999 2024-09-11T03:43:52.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.084 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.084 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:43:52.084 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904291+0000", 2024-09-11T03:43:52.084 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6755999999999998e-05 2024-09-11T03:43:52.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.084 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.085 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:43:52.085 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.914973+0000", 2024-09-11T03:43:52.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010682199999999999 2024-09-11T03:43:52.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.085 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.085 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:43:52.085 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.915043+0000", 2024-09-11T03:43:52.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0297999999999997e-05 2024-09-11T03:43:52.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:43:52.085 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:43:52.085 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:43:52.085 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.915155+0000", 2024-09-11T03:43:52.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000111767 2024-09-11T03:43:52.086 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.086 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:52.086 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.086 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:43:52.086 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:43:52.086 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:43:52.086 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:52.107 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:52.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:52.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:52.209 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:51 smithi175 ceph-mon[30894]: pgmap v774: 57 pgs: 57 active+clean; 185 MiB data, 771 MiB used, 893 GiB / 894 GiB avail; 1.6 MiB/s rd, 619 KiB/s wr, 4 op/s 2024-09-11T03:43:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:51 smithi175 ceph-mon[30894]: osdmap e413: 12 total, 12 up, 10 in 2024-09-11T03:43:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1904437874' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-09-11T03:43:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1904437874' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-09-11T03:43:52.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:51 smithi175 ceph-mon[30894]: osdmap e414: 12 total, 12 up, 11 in 2024-09-11T03:43:52.232 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:52.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:52.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:52.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:51 smithi084 ceph-mon[27254]: pgmap v774: 57 pgs: 57 active+clean; 185 MiB data, 771 MiB used, 893 GiB / 894 GiB avail; 1.6 MiB/s rd, 619 KiB/s wr, 4 op/s 2024-09-11T03:43:52.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:51 smithi084 ceph-mon[27254]: osdmap e413: 12 total, 12 up, 10 in 2024-09-11T03:43:52.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1904437874' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-09-11T03:43:52.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1904437874' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-09-11T03:43:52.303 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:51 smithi084 ceph-mon[27254]: osdmap e414: 12 total, 12 up, 11 in 2024-09-11T03:43:52.334 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:52.359 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:52.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:52.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:52.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:51 smithi121 ceph-mon[30729]: pgmap v774: 57 pgs: 57 active+clean; 185 MiB data, 771 MiB used, 893 GiB / 894 GiB avail; 1.6 MiB/s rd, 619 KiB/s wr, 4 op/s 2024-09-11T03:43:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:51 smithi121 ceph-mon[30729]: osdmap e413: 12 total, 12 up, 10 in 2024-09-11T03:43:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1904437874' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-09-11T03:43:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1904437874' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-09-11T03:43:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:51 smithi121 ceph-mon[30729]: osdmap e414: 12 total, 12 up, 11 in 2024-09-11T03:43:52.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:52.487 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:52.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:52.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:52.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:52.615 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:52.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:52.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:52.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:52.742 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:52.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:52.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:52.775 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:43:52.844 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:52.871 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:52.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:52.872 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:52.880 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-09-11T03:43:52.917 INFO:tasks.rados.rados.2.smithi175.stdout:1527: done (0 left) 2024-09-11T03:43:52.917 INFO:tasks.rados.rados.2.smithi175.stdout:1528: rollback oid 5 current snap is 161 2024-09-11T03:43:52.917 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 5 to 156 2024-09-11T03:43:52.917 INFO:tasks.rados.rados.2.smithi175.stdout:1529: rollback oid 12 current snap is 161 2024-09-11T03:43:52.917 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 12 to 156 2024-09-11T03:43:52.917 INFO:tasks.rados.rados.2.smithi175.stdout:1530: copy_from oid 37 from oid 16 current snap is 161 2024-09-11T03:43:52.917 INFO:tasks.rados.rados.2.smithi175.stdout:1531: snap_create 2024-09-11T03:43:52.917 INFO:tasks.rados.rados.2.smithi175.stdout:1528: finishing rollback tid 1 to smithi17557870-5 2024-09-11T03:43:52.918 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 0 (ObjNum 113 snap 0 seq_num 1745059696) dirty dne 2024-09-11T03:43:52.923 INFO:tasks.rados.rados.2.smithi175.stdout:1529: finishing rollback tid 1 to smithi17557870-12 2024-09-11T03:43:52.923 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 521 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T03:43:52.927 INFO:tasks.rados.rados.2.smithi175.stdout:1530: finishing copy_from racing read to smithi17557870-37 2024-09-11T03:43:52.931 INFO:tasks.rados.rados.2.smithi175.stdout:1530: finishing copy_from to smithi17557870-37 2024-09-11T03:43:52.931 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 461 (ObjNum 509 snap 150 seq_num 509) dirty exists 2024-09-11T03:43:52.973 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:52.993 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:52.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:52.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:53.095 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:53.121 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:53.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:53.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:53.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:52 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:43:53.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:52 smithi084 ceph-mon[27254]: 2.d scrub ok 2024-09-11T03:43:53.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:52 smithi084 ceph-mon[27254]: osdmap e415: 12 total, 12 up, 11 in 2024-09-11T03:43:53.223 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:52 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:43:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:52 smithi175 ceph-mon[30894]: 2.d scrub ok 2024-09-11T03:43:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:52 smithi175 ceph-mon[30894]: osdmap e415: 12 total, 12 up, 11 in 2024-09-11T03:43:53.244 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:53.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:53.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:53.346 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:53.372 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:53.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:53.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:53.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:52 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:43:53.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:52 smithi121 ceph-mon[30729]: 2.d scrub ok 2024-09-11T03:43:53.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:52 smithi121 ceph-mon[30729]: osdmap e415: 12 total, 12 up, 11 in 2024-09-11T03:43:53.473 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:53.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:53.495 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:53.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:53.597 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:53.623 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:53.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:53.624 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:53.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:53.751 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:53.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:53.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:53.853 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:53.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:53.879 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:53.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:53.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T03:43:53.903 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:53.910 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:43:53.914 INFO:tasks.rados.rados.2.smithi175.stdout:1528: done (3 left) 2024-09-11T03:43:53.914 INFO:tasks.rados.rados.2.smithi175.stdout:1529: done (2 left) 2024-09-11T03:43:53.914 INFO:tasks.rados.rados.2.smithi175.stdout:1530: done (1 left) 2024-09-11T03:43:53.914 INFO:tasks.rados.rados.2.smithi175.stdout:1531: done (0 left) 2024-09-11T03:43:53.914 INFO:tasks.rados.rados.2.smithi175.stdout:1532: read oid 3 snap -1 2024-09-11T03:43:53.914 INFO:tasks.rados.rados.2.smithi175.stdout:1532: expect (ObjNum 540 snap 159 seq_num 540) 2024-09-11T03:43:53.914 INFO:tasks.rados.rados.2.smithi175.stdout:1533: snap_remove snap 136 2024-09-11T03:43:53.981 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:54.003 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:54.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:54.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:54.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:54.126 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:54.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:54.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:54.140 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:43:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:53 smithi084 ceph-mon[27254]: pgmap v777: 57 pgs: 3 unknown, 5 peering, 49 active+clean; 158 MiB data, 815 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:43:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:53 smithi084 ceph-mon[27254]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-11T03:43:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:53 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/474771966' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:43:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:53 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:53 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/474771966' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:43:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:53 smithi084 ceph-mon[27254]: osdmap e416: 12 total, 12 up, 11 in 2024-09-11T03:43:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:53 smithi121 ceph-mon[30729]: pgmap v777: 57 pgs: 3 unknown, 5 peering, 49 active+clean; 158 MiB data, 815 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:43:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:53 smithi121 ceph-mon[30729]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-11T03:43:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:53 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/474771966' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:43:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:53 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:53 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/474771966' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:43:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:53 smithi121 ceph-mon[30729]: osdmap e416: 12 total, 12 up, 11 in 2024-09-11T03:43:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:53 smithi175 ceph-mon[30894]: pgmap v777: 57 pgs: 3 unknown, 5 peering, 49 active+clean; 158 MiB data, 815 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:43:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:53 smithi175 ceph-mon[30894]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-11T03:43:54.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:53 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/474771966' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:43:54.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:53 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:43:54.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:53 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/474771966' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:43:54.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:53 smithi175 ceph-mon[30894]: osdmap e416: 12 total, 12 up, 11 in 2024-09-11T03:43:54.228 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:54.250 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:54.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:54.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:54.352 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:54.376 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:54.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:54.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:54.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:54.502 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:54.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:54.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:54.604 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:54.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:54.629 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:54.629 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:54.649 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:43:54.649 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:43:54.649 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:43:54.649 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:43:54.730 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:54.751 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:54.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:54.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:54.854 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:54.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:54.877 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:54.878 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:54.918 INFO:tasks.rados.rados.2.smithi175.stdout:1533: done (1 left) 2024-09-11T03:43:54.919 INFO:tasks.rados.rados.2.smithi175.stdout:1534: read oid 12 snap 155 2024-09-11T03:43:54.919 INFO:tasks.rados.rados.2.smithi175.stdout:1534: expect deleted 2024-09-11T03:43:54.919 INFO:tasks.rados.rados.2.smithi175.stdout:1535: delete oid 2 current snap is 162 2024-09-11T03:43:54.919 INFO:tasks.rados.rados.2.smithi175.stdout:1532: expect (ObjNum 540 snap 159 seq_num 540) 2024-09-11T03:43:54.979 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:55.000 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:55.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:55.001 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:55.036 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:43:55.036 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:43:55.037 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:43:55.037 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:43:55.037 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:43:55.037 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:43:55.037 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:43:55.037 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:43:55.037 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:43:55.037 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:43:55.037 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:43:55.037 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:43:55.037 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:43:55.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:43:55.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:43:55.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:43:55.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:43:55.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:43:55.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:43:55.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:43:55.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:43:55.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:43:55.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:43:55.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:43:55.102 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:55.124 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:55.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:55.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:55.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:54 smithi084 ceph-mon[27254]: 1.1 scrub starts 2024-09-11T03:43:55.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:54 smithi084 ceph-mon[27254]: 1.1 scrub ok 2024-09-11T03:43:55.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:54 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1657911965' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:55.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:54 smithi084 ceph-mon[27254]: osdmap e417: 12 total, 12 up, 11 in 2024-09-11T03:43:55.226 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:55.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:54 smithi175 ceph-mon[30894]: 1.1 scrub starts 2024-09-11T03:43:55.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:54 smithi175 ceph-mon[30894]: 1.1 scrub ok 2024-09-11T03:43:55.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:54 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1657911965' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:55.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:54 smithi175 ceph-mon[30894]: osdmap e417: 12 total, 12 up, 11 in 2024-09-11T03:43:55.247 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:55.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:55.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:55.311 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:54 smithi121 ceph-mon[30729]: 1.1 scrub starts 2024-09-11T03:43:55.311 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:54 smithi121 ceph-mon[30729]: 1.1 scrub ok 2024-09-11T03:43:55.313 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:54 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1657911965' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:43:55.313 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:54 smithi121 ceph-mon[30729]: osdmap e417: 12 total, 12 up, 11 in 2024-09-11T03:43:55.339 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T03:43:55.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:55.370 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:55.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:55.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:55.472 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:55.495 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:55.495 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:55.580 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:43:55.596 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:55.618 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:55.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:55.720 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:55.742 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:55.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:55.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:55.844 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:55.865 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:55.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:55.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:55.922 INFO:tasks.rados.rados.2.smithi175.stdout:1532: done (2 left) 2024-09-11T03:43:55.922 INFO:tasks.rados.rados.2.smithi175.stdout:1534: done (1 left) 2024-09-11T03:43:55.922 INFO:tasks.rados.rados.2.smithi175.stdout:1535: done (0 left) 2024-09-11T03:43:55.922 INFO:tasks.rados.rados.2.smithi175.stdout:1536: copy_from oid 3 from oid 10 current snap is 162 2024-09-11T03:43:55.922 INFO:tasks.rados.rados.2.smithi175.stdout:1537: write (excl) oid 11 current snap is 162 2024-09-11T03:43:55.922 INFO:tasks.rados.rados.2.smithi175.stdout:1537: seq_num 543 ranges {726201=721270,2141840=578191} 2024-09-11T03:43:55.930 INFO:tasks.rados.rados.2.smithi175.stdout:1537: writing smithi17557870-11 from 726201 to 1447471 tid 1 2024-09-11T03:43:55.937 INFO:tasks.rados.rados.2.smithi175.stdout:1537: writing smithi17557870-11 from 2141840 to 2720031 tid 2 2024-09-11T03:43:55.937 INFO:tasks.rados.rados.2.smithi175.stdout:1538: snap_create 2024-09-11T03:43:55.937 INFO:tasks.rados.rados.2.smithi175.stdout:1536: finishing copy_from racing read to smithi17557870-3 2024-09-11T03:43:55.937 INFO:tasks.rados.rados.2.smithi175.stdout:1537: finishing write tid 1 to smithi17557870-11 2024-09-11T03:43:55.937 INFO:tasks.rados.rados.2.smithi175.stdout:1536: finishing copy_from to smithi17557870-3 2024-09-11T03:43:55.937 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 526 (ObjNum 488 snap 141 seq_num 488) dirty exists 2024-09-11T03:43:55.939 INFO:tasks.rados.rados.2.smithi175.stdout:1537: finishing write tid 2 to smithi17557870-11 2024-09-11T03:43:55.940 INFO:tasks.rados.rados.2.smithi175.stdout:1537: finishing write tid 3 to smithi17557870-11 2024-09-11T03:43:55.940 INFO:tasks.rados.rados.2.smithi175.stdout:1537: finishing write tid 4 to smithi17557870-11 2024-09-11T03:43:55.940 INFO:tasks.rados.rados.2.smithi175.stdout:1537: oid 11 updating version 0 to 440 2024-09-11T03:43:55.940 INFO:tasks.rados.rados.2.smithi175.stdout:1537: oid 11 updating version 440 to 441 2024-09-11T03:43:55.940 INFO:tasks.rados.rados.2.smithi175.stdout:1537: oid 11 version 441 is already newer than 439 2024-09-11T03:43:55.940 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 441 (ObjNum 543 snap 162 seq_num 543) dirty exists 2024-09-11T03:43:55.940 INFO:tasks.rados.rados.2.smithi175.stdout:1537: left oid 11 (ObjNum 543 snap 162 seq_num 543) 2024-09-11T03:43:55.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:55.989 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:55.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:55.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:56.011 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:55 smithi121 ceph-mon[30729]: 1.7 scrub starts 2024-09-11T03:43:56.011 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:55 smithi121 ceph-mon[30729]: 1.7 scrub ok 2024-09-11T03:43:56.011 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:55 smithi121 ceph-mon[30729]: 1.10 scrub starts 2024-09-11T03:43:56.011 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:55 smithi121 ceph-mon[30729]: pgmap v780: 57 pgs: 3 unknown, 5 peering, 49 active+clean; 158 MiB data, 817 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:43:56.012 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:55 smithi121 ceph-mon[30729]: osdmap e418: 12 total, 12 up, 11 in 2024-09-11T03:43:56.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:56.111 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:56.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:56.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:56.114 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:43:56.114 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:43:56.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:55 smithi084 ceph-mon[27254]: 1.7 scrub starts 2024-09-11T03:43:56.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:55 smithi084 ceph-mon[27254]: 1.7 scrub ok 2024-09-11T03:43:56.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:55 smithi084 ceph-mon[27254]: 1.10 scrub starts 2024-09-11T03:43:56.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:55 smithi084 ceph-mon[27254]: pgmap v780: 57 pgs: 3 unknown, 5 peering, 49 active+clean; 158 MiB data, 817 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:43:56.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:55 smithi084 ceph-mon[27254]: osdmap e418: 12 total, 12 up, 11 in 2024-09-11T03:43:56.214 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:55 smithi175 ceph-mon[30894]: 1.7 scrub starts 2024-09-11T03:43:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:55 smithi175 ceph-mon[30894]: 1.7 scrub ok 2024-09-11T03:43:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:55 smithi175 ceph-mon[30894]: 1.10 scrub starts 2024-09-11T03:43:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:55 smithi175 ceph-mon[30894]: pgmap v780: 57 pgs: 3 unknown, 5 peering, 49 active+clean; 158 MiB data, 817 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:43:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:55 smithi175 ceph-mon[30894]: osdmap e418: 12 total, 12 up, 11 in 2024-09-11T03:43:56.236 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:56.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:56.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:56.338 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:56.362 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:56.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:56.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:56.464 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:43:56.488 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:56.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:56.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:43:56.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:56.613 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:56.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:56.715 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:56.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T03:43:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:56.740 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:56.740 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:56.841 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:56.864 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:56.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:56.905 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:43:56.946 INFO:tasks.rados.rados.2.smithi175.stdout:1536: done (2 left) 2024-09-11T03:43:56.947 INFO:tasks.rados.rados.2.smithi175.stdout:1537: done (1 left) 2024-09-11T03:43:56.947 INFO:tasks.rados.rados.2.smithi175.stdout:1538: done (0 left) 2024-09-11T03:43:56.947 INFO:tasks.rados.rados.2.smithi175.stdout:1539: copy_from oid 14 from oid 49 current snap is 163 2024-09-11T03:43:56.947 INFO:tasks.rados.rados.2.smithi175.stdout:1540: read oid 29 snap -1 2024-09-11T03:43:56.947 INFO:tasks.rados.rados.2.smithi175.stdout:1540: expect deleted 2024-09-11T03:43:56.947 INFO:tasks.rados.rados.2.smithi175.stdout:1541: snap_create 2024-09-11T03:43:56.961 INFO:tasks.rados.rados.2.smithi175.stdout:1539: finishing copy_from racing read to smithi17557870-14 2024-09-11T03:43:56.963 INFO:tasks.rados.rados.2.smithi175.stdout:1539: finishing copy_from to smithi17557870-14 2024-09-11T03:43:56.963 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 453 (ObjNum 445 snap 130 seq_num 445) dirty exists 2024-09-11T03:43:56.966 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:56.975 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:43:56.988 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:56.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:56.990 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:57.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:57.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:56 smithi121 ceph-mon[30729]: 1.5 scrub starts 2024-09-11T03:43:57.112 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:57.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:57.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:57.160 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:57.214 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:57.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:56 smithi175 ceph-mon[30894]: 1.5 scrub starts 2024-09-11T03:43:57.236 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:57.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:57.237 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:57.280 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:56 smithi084 ceph-mon[27254]: 1.5 scrub starts 2024-09-11T03:43:57.338 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:57.359 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:57.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:57.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:57.462 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:57.469 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:43:57.469 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:43:57.483 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:57.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:57.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:57.585 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:57.608 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:57.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:57.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:57.731 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:57.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:57.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:57.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:57.855 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:57.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:57.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:57.878 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:43:57.880 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:43:57.880 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:43:57.880 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-11T03:43:57.881 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:43:57.943 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:43:57.948 INFO:tasks.rados.rados.2.smithi175.stdout:1539: done (2 left) 2024-09-11T03:43:57.948 INFO:tasks.rados.rados.2.smithi175.stdout:1540: done (1 left) 2024-09-11T03:43:57.948 INFO:tasks.rados.rados.2.smithi175.stdout:1541: done (0 left) 2024-09-11T03:43:57.948 INFO:tasks.rados.rados.2.smithi175.stdout:1542: rollback oid 20 current snap is 164 2024-09-11T03:43:57.948 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 20 to 162 2024-09-11T03:43:57.948 INFO:tasks.rados.rados.2.smithi175.stdout:1543: snap_remove snap 160 2024-09-11T03:43:57.957 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:57.979 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:57.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:57.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:58.041 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:57 smithi121 ceph-mon[30729]: 1.6 scrub starts 2024-09-11T03:43:58.041 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:57 smithi121 ceph-mon[30729]: pgmap v783: 57 pgs: 3 unknown, 5 peering, 49 active+clean; 141 MiB data, 799 MiB used, 983 GiB / 983 GiB avail; 9.1 MiB/s rd, 1.4 MiB/s wr, 7 op/s; 2.0 MiB/s, 7 objects/s recovering 2024-09-11T03:43:58.041 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:57 smithi121 ceph-mon[30729]: osdmap e419: 12 total, 12 up, 11 in 2024-09-11T03:43:58.041 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:57 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/66106274' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:58.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T03:43:58.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:58.134 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:58.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:58.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:57 smithi175 ceph-mon[30894]: 1.6 scrub starts 2024-09-11T03:43:58.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:57 smithi175 ceph-mon[30894]: pgmap v783: 57 pgs: 3 unknown, 5 peering, 49 active+clean; 141 MiB data, 799 MiB used, 983 GiB / 983 GiB avail; 9.1 MiB/s rd, 1.4 MiB/s wr, 7 op/s; 2.0 MiB/s, 7 objects/s recovering 2024-09-11T03:43:58.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:57 smithi175 ceph-mon[30894]: osdmap e419: 12 total, 12 up, 11 in 2024-09-11T03:43:58.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:57 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/66106274' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:58.236 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:58.257 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:58.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:58.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:58.311 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:43:58.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:57 smithi084 ceph-mon[27254]: 1.6 scrub starts 2024-09-11T03:43:58.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:57 smithi084 ceph-mon[27254]: pgmap v783: 57 pgs: 3 unknown, 5 peering, 49 active+clean; 141 MiB data, 799 MiB used, 983 GiB / 983 GiB avail; 9.1 MiB/s rd, 1.4 MiB/s wr, 7 op/s; 2.0 MiB/s, 7 objects/s recovering 2024-09-11T03:43:58.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:57 smithi084 ceph-mon[27254]: osdmap e419: 12 total, 12 up, 11 in 2024-09-11T03:43:58.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:57 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/66106274' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:43:58.359 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:43:58.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:58.381 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:58.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:43:58.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:43:58.504 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:58.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:58.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:43:58.606 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:43:58.632 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:58.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:58.633 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:43:58.733 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:43:58.757 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:58.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:58.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:43:58.796 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:58.797 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:43:58.850 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:43:58.850 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:43:58.859 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:58.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:58.885 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:58.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:58.960 INFO:tasks.rados.rados.2.smithi175.stdout:1543: done (1 left) 2024-09-11T03:43:58.960 INFO:tasks.rados.rados.2.smithi175.stdout:1544: snap_create 2024-09-11T03:43:58.960 INFO:tasks.rados.rados.2.smithi175.stdout:1542: finishing rollback tid 0 to smithi17557870-20 2024-09-11T03:43:58.960 INFO:tasks.rados.rados.2.smithi175.stdout:1542: finishing rollback tid 1 to smithi17557870-20 2024-09-11T03:43:58.961 INFO:tasks.rados.rados.2.smithi175.stdout:1542: finishing rollback tid 2 to smithi17557870-20 2024-09-11T03:43:58.961 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 571 (ObjNum 513 snap 152 seq_num 513) dirty exists 2024-09-11T03:43:58.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:43:59.008 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:59.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:59.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:43:59.111 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:43:59.137 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:59.138 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:43:59.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:58 smithi121 ceph-mon[30729]: 2.17 scrub starts 2024-09-11T03:43:59.139 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:58 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/66106274' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:59.140 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:58 smithi121 ceph-mon[30729]: osdmap e420: 12 total, 12 up, 11 in 2024-09-11T03:43:59.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:58 smithi175 ceph-mon[30894]: 2.17 scrub starts 2024-09-11T03:43:59.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:58 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/66106274' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:59.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:58 smithi175 ceph-mon[30894]: osdmap e420: 12 total, 12 up, 11 in 2024-09-11T03:43:59.239 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:43:59.243 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:58 smithi084 ceph-mon[27254]: 2.17 scrub starts 2024-09-11T03:43:59.243 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:58 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/66106274' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:43:59.243 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:58 smithi084 ceph-mon[27254]: osdmap e420: 12 total, 12 up, 11 in 2024-09-11T03:43:59.261 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:59.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:43:59.363 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:43:59.388 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:59.389 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:43:59.490 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:59.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T03:43:59.515 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:43:59.516 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:43:59.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:59.519 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:59.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:59.620 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:43:59.642 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:43:59.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:59.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:43:59.736 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:43:59.744 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:43:59.765 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:43:59.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:59.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:43:59.867 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:43:59.893 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:43:59.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:43:59.894 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:43:59.960 INFO:tasks.rados.rados.2.smithi175.stdout:1542: done (1 left) 2024-09-11T03:43:59.960 INFO:tasks.rados.rados.2.smithi175.stdout:1544: done (0 left) 2024-09-11T03:43:59.960 INFO:tasks.rados.rados.2.smithi175.stdout:1545: write oid 41 current snap is 165 2024-09-11T03:43:59.960 INFO:tasks.rados.rados.2.smithi175.stdout:1545: seq_num 544 ranges {743008=796519,2200322=721849} 2024-09-11T03:43:59.968 INFO:tasks.rados.rados.2.smithi175.stdout:1545: writing smithi17557870-41 from 743008 to 1539527 tid 1 2024-09-11T03:43:59.976 INFO:tasks.rados.rados.2.smithi175.stdout:1545: writing smithi17557870-41 from 2200322 to 2922171 tid 2 2024-09-11T03:43:59.976 INFO:tasks.rados.rados.2.smithi175.stdout:1546: read oid 39 snap -1 2024-09-11T03:43:59.977 INFO:tasks.rados.rados.2.smithi175.stdout:1546: expect deleted 2024-09-11T03:43:59.977 INFO:tasks.rados.rados.2.smithi175.stdout:1545: finishing write tid 1 to smithi17557870-41 2024-09-11T03:43:59.977 INFO:tasks.rados.rados.2.smithi175.stdout:1547: read oid 9 snap 149 2024-09-11T03:43:59.977 INFO:tasks.rados.rados.2.smithi175.stdout:1547: expect (ObjNum 277 snap 73 seq_num 277) 2024-09-11T03:43:59.977 INFO:tasks.rados.rados.2.smithi175.stdout:1548: copy_from oid 11 from oid 18 current snap is 165 2024-09-11T03:43:59.977 INFO:tasks.rados.rados.2.smithi175.stdout:1549: snap_create 2024-09-11T03:43:59.980 INFO:tasks.rados.rados.2.smithi175.stdout:1545: finishing write tid 2 to smithi17557870-41 2024-09-11T03:43:59.980 INFO:tasks.rados.rados.2.smithi175.stdout:1545: finishing write tid 3 to smithi17557870-41 2024-09-11T03:43:59.980 INFO:tasks.rados.rados.2.smithi175.stdout:1545: finishing write tid 4 to smithi17557870-41 2024-09-11T03:43:59.980 INFO:tasks.rados.rados.2.smithi175.stdout:1545: oid 41 updating version 0 to 401 2024-09-11T03:43:59.981 INFO:tasks.rados.rados.2.smithi175.stdout:1545: oid 41 updating version 401 to 402 2024-09-11T03:43:59.981 INFO:tasks.rados.rados.2.smithi175.stdout:1545: oid 41 updating version 402 to 403 2024-09-11T03:43:59.981 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 403 (ObjNum 544 snap 165 seq_num 544) dirty exists 2024-09-11T03:43:59.981 INFO:tasks.rados.rados.2.smithi175.stdout:1545: left oid 41 (ObjNum 544 snap 165 seq_num 544) 2024-09-11T03:43:59.986 INFO:tasks.rados.rados.2.smithi175.stdout:1547: expect (ObjNum 277 snap 73 seq_num 277) 2024-09-11T03:43:59.995 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:00.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:00.017 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:00.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:00.070 INFO:tasks.rados.rados.2.smithi175.stdout:1548: finishing copy_from racing read to smithi17557870-11 2024-09-11T03:44:00.070 INFO:tasks.rados.rados.2.smithi175.stdout:1548: finishing copy_from to smithi17557870-11 2024-09-11T03:44:00.070 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 464 (ObjNum 522 snap 154 seq_num 522) dirty exists 2024-09-11T03:44:00.119 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:00.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:59 smithi121 ceph-mon[30729]: 2.17 scrub starts 2024-09-11T03:44:00.127 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:59 smithi121 ceph-mon[30729]: pgmap v786: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 168 MiB data, 792 MiB used, 983 GiB / 983 GiB avail; 60 MiB/s rd, 21 MiB/s wr, 115 op/s; 9.8 MiB/s, 30 objects/s recovering 2024-09-11T03:44:00.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:59 smithi121 ceph-mon[30729]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-11T03:44:00.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:59 smithi121 ceph-mon[30729]: osdmap e421: 12 total, 12 up, 11 in 2024-09-11T03:44:00.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:43:59 smithi121 ceph-mon[30729]: from='client.15864 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:44:00.140 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:00.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:00.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:59 smithi084 ceph-mon[27254]: 2.17 scrub starts 2024-09-11T03:44:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:59 smithi084 ceph-mon[27254]: pgmap v786: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 168 MiB data, 792 MiB used, 983 GiB / 983 GiB avail; 60 MiB/s rd, 21 MiB/s wr, 115 op/s; 9.8 MiB/s, 30 objects/s recovering 2024-09-11T03:44:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:59 smithi084 ceph-mon[27254]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-11T03:44:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:59 smithi084 ceph-mon[27254]: osdmap e421: 12 total, 12 up, 11 in 2024-09-11T03:44:00.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:43:59 smithi084 ceph-mon[27254]: from='client.15864 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:44:00.226 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:44:00.226 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:44:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:59 smithi175 ceph-mon[30894]: 2.17 scrub starts 2024-09-11T03:44:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:59 smithi175 ceph-mon[30894]: pgmap v786: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 168 MiB data, 792 MiB used, 983 GiB / 983 GiB avail; 60 MiB/s rd, 21 MiB/s wr, 115 op/s; 9.8 MiB/s, 30 objects/s recovering 2024-09-11T03:44:00.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:59 smithi175 ceph-mon[30894]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-11T03:44:00.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:59 smithi175 ceph-mon[30894]: osdmap e421: 12 total, 12 up, 11 in 2024-09-11T03:44:00.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:43:59 smithi175 ceph-mon[30894]: from='client.15864 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:44:00.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:00.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:00.266 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:00.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:00.354 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":787,"stamp":"2024-09-11T03:43:58.955444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":176631109,"num_objects":90,"num_object_clones":43,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":9,"num_read":1991,"num_read_kb":1833946,"num_write":1444,"num_write_kb":561208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":65202606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10681,"ondisk_log_size":10681,"up":171,"acting":171,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":175,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":811464,"kb_used_data":410112,"kb_used_omap":17,"kb_used_meta":401070,"kb_avail":1030385208,"statfs":{"total":1055945392128,"available":1055114452992,"internally_reserved":0,"allocated":419954688,"data_stored":491809181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":410696565},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11319007,"num_objects":4,"num_object_clones":-4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":270,"num_read_kb":245776,"num_write":193,"num_write_kb":85806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":41076203,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.043018"},"pg_stats":[{"pgid":"1.1e","version":"20'92","reported_seq":415,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-11T03:43:56.951187+0000","last_change":"2024-09-11T03:43:09.915818+0000","last_active":"2024-09-11T03:43:56.951187+0000","last_peered":"2024-09-11T03:43:56.951187+0000","last_clean":"2024-09-11T03:43:56.951187+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:43:56.951187+0000","last_undegraded":"2024-09-11T03:43:56.951187+0000","last_fullsized":"2024-09-11T03:43:56.951187+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:09.915771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:09.915771+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:22.261961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'92","reported_seq":410,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.916903+0000","last_change":"2024-09-11T03:40:41.348889+0000","last_active":"2024-09-11T03:43:55.916903+0000","last_peered":"2024-09-11T03:43:55.916903+0000","last_clean":"2024-09-11T03:43:55.916903+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:43:55.916903+0000","last_undegraded":"2024-09-11T03:43:55.916903+0000","last_fullsized":"2024-09-11T03:43:55.916903+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:41.348828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:41.348828+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:16.941879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'92","reported_seq":410,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.915458+0000","last_change":"2024-09-11T03:40:40.188174+0000","last_active":"2024-09-11T03:43:55.915458+0000","last_peered":"2024-09-11T03:43:55.915458+0000","last_clean":"2024-09-11T03:43:55.915458+0000","last_became_active":"2024-09-11T03:40:39.848230+0000","last_became_peered":"2024-09-11T03:40:39.848230+0000","last_unstale":"2024-09-11T03:43:55.915458+0000","last_undegraded":"2024-09-11T03:43:55.915458+0000","last_fullsized":"2024-09-11T03:43:55.915458+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.188124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.188124+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:13.535618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"417'296","reported_seq":1998,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-11T03:43:58.346983+0000","last_change":"2024-09-11T03:43:54.926437+0000","last_active":"2024-09-11T03:43:58.346983+0000","last_peered":"2024-09-11T03:43:58.346983+0000","last_clean":"2024-09-11T03:43:58.346983+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:43:58.346983+0000","last_undegraded":"2024-09-11T03:43:58.346983+0000","last_fullsized":"2024-09-11T03:43:58.346983+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"348'232","last_scrub_stamp":"2024-09-11T03:42:45.411870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:42:45.411870+0000","objects_scrubbed":3,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":2,"snaptrim_duration":0.0079015739999999998,"stat_sum":{"num_bytes":3630378,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":99500,"num_write":25,"num_write_kb":11732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4865740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.1b","version":"20'92","reported_seq":410,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.915611+0000","last_change":"2024-09-11T03:40:42.166354+0000","last_active":"2024-09-11T03:43:55.915611+0000","last_peered":"2024-09-11T03:43:55.915611+0000","last_clean":"2024-09-11T03:43:55.915611+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:43:55.915611+0000","last_undegraded":"2024-09-11T03:43:55.915611+0000","last_fullsized":"2024-09-11T03:43:55.915611+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:42.166295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:42.166295+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:15.513587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.19","version":"417'411","reported_seq":2116,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-11T03:43:57.957412+0000","last_change":"2024-09-11T03:43:54.919701+0000","last_active":"2024-09-11T03:43:57.957412+0000","last_peered":"2024-09-11T03:43:57.957412+0000","last_clean":"2024-09-11T03:43:57.957412+0000","last_became_active":"2024-09-11T03:41:38.886787+0000","last_became_peered":"2024-09-11T03:41:38.886787+0000","last_unstale":"2024-09-11T03:43:57.957412+0000","last_undegraded":"2024-09-11T03:43:57.957412+0000","last_fullsized":"2024-09-11T03:43:57.957412+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"353'377","last_scrub_stamp":"2024-09-11T03:42:50.509630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:42:50.509630+0000","objects_scrubbed":3,"log_size":411,"log_dups_size":0,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:17.168275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0045299069999999997,"stat_sum":{"num_bytes":7473120,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":10058,"num_write":19,"num_write_kb":10805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.1a","version":"20'92","reported_seq":657,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-11T03:43:54.913516+0000","last_change":"2024-09-11T03:43:10.631373+0000","last_active":"2024-09-11T03:43:54.913516+0000","last_peered":"2024-09-11T03:43:54.913516+0000","last_clean":"2024-09-11T03:43:54.913516+0000","last_became_active":"2024-09-11T03:40:39.848781+0000","last_became_peered":"2024-09-11T03:40:39.848781+0000","last_unstale":"2024-09-11T03:43:54.913516+0000","last_undegraded":"2024-09-11T03:43:54.913516+0000","last_fullsized":"2024-09-11T03:43:54.913516+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:10.631316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:10.631316+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:32.934757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"20'92","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-11T03:43:57.151915+0000","last_change":"2024-09-11T03:43:52.921672+0000","last_active":"2024-09-11T03:43:57.151915+0000","last_peered":"2024-09-11T03:43:57.151915+0000","last_clean":"2024-09-11T03:43:57.151915+0000","last_became_active":"2024-09-11T03:43:52.920761+0000","last_became_peered":"2024-09-11T03:43:52.920761+0000","last_unstale":"2024-09-11T03:43:57.151915+0000","last_undegraded":"2024-09-11T03:43:57.151915+0000","last_fullsized":"2024-09-11T03:43:57.151915+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.718503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.718503+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'92","reported_seq":538,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.924769+0000","last_change":"2024-09-11T03:43:30.252705+0000","last_active":"2024-09-11T03:43:55.924769+0000","last_peered":"2024-09-11T03:43:55.924769+0000","last_clean":"2024-09-11T03:43:55.924769+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:43:55.924769+0000","last_undegraded":"2024-09-11T03:43:55.924769+0000","last_fullsized":"2024-09-11T03:43:55.924769+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:30.252631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:30.252631+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:43.831297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"417'218","reported_seq":1869,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-11T03:43:58.347775+0000","last_change":"2024-09-11T03:43:54.922934+0000","last_active":"2024-09-11T03:43:58.347775+0000","last_peered":"2024-09-11T03:43:58.347775+0000","last_clean":"2024-09-11T03:43:58.347775+0000","last_became_active":"2024-09-11T03:41:38.890690+0000","last_became_peered":"2024-09-11T03:41:38.890690+0000","last_unstale":"2024-09-11T03:43:58.347775+0000","last_undegraded":"2024-09-11T03:43:58.347775+0000","last_fullsized":"2024-09-11T03:43:58.347775+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"361'198","last_scrub_stamp":"2024-09-11T03:43:04.629608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:04.629608+0000","objects_scrubbed":1,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:34.077474+0000","scrub_duration":2,"objects_trimmed":2,"snaptrim_duration":0.0048185809999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":8230,"num_write":14,"num_write_kb":3064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[9,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":425,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.915743+0000","last_change":"2024-09-11T03:43:52.928549+0000","last_active":"2024-09-11T03:43:55.915743+0000","last_peered":"2024-09-11T03:43:55.915743+0000","last_clean":"2024-09-11T03:43:55.915743+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:43:55.915743+0000","last_undegraded":"2024-09-11T03:43:55.915743+0000","last_fullsized":"2024-09-11T03:43:55.915743+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:57.260664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:57.260664+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":1939,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-11T03:43:58.347316+0000","last_change":"2024-09-11T03:43:54.921634+0000","last_active":"2024-09-11T03:43:58.347316+0000","last_peered":"2024-09-11T03:43:58.347316+0000","last_clean":"2024-09-11T03:43:58.347316+0000","last_became_active":"2024-09-11T03:41:38.888228+0000","last_became_peered":"2024-09-11T03:41:38.888228+0000","last_unstale":"2024-09-11T03:43:58.347316+0000","last_undegraded":"2024-09-11T03:43:58.347316+0000","last_fullsized":"2024-09-11T03:43:58.347316+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:42:48.441294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:42:48.441294+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:17.997630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034213799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":424,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.916996+0000","last_change":"2024-09-11T03:43:19.971243+0000","last_active":"2024-09-11T03:43:55.916996+0000","last_peered":"2024-09-11T03:43:55.916996+0000","last_clean":"2024-09-11T03:43:55.916996+0000","last_became_active":"2024-09-11T03:40:39.849950+0000","last_became_peered":"2024-09-11T03:40:39.849950+0000","last_unstale":"2024-09-11T03:43:55.916996+0000","last_undegraded":"2024-09-11T03:43:55.916996+0000","last_fullsized":"2024-09-11T03:43:55.916996+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:19.971197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:19.971197+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:23.410394+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"417'281","reported_seq":1895,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.924992+0000","last_change":"2024-09-11T03:43:54.926574+0000","last_active":"2024-09-11T03:43:55.924992+0000","last_peered":"2024-09-11T03:43:55.924992+0000","last_clean":"2024-09-11T03:43:55.924992+0000","last_became_active":"2024-09-11T03:41:38.887856+0000","last_became_peered":"2024-09-11T03:41:38.887856+0000","last_unstale":"2024-09-11T03:43:55.924992+0000","last_undegraded":"2024-09-11T03:43:55.924992+0000","last_fullsized":"2024-09-11T03:43:55.924992+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"361'231","last_scrub_stamp":"2024-09-11T03:43:02.213189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:02.213189+0000","objects_scrubbed":1,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:03.943820+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0044170549999999996,"stat_sum":{"num_bytes":6971847,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":37,"num_write_kb":16303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,9],"acting":[5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":417,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.917677+0000","last_change":"2024-09-11T03:43:09.565121+0000","last_active":"2024-09-11T03:43:55.917677+0000","last_peered":"2024-09-11T03:43:55.917677+0000","last_clean":"2024-09-11T03:43:55.917677+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:43:55.917677+0000","last_undegraded":"2024-09-11T03:43:55.917677+0000","last_fullsized":"2024-09-11T03:43:55.917677+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:09.565070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:09.565070+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:16.696567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":1331,"reported_epoch":419,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T03:43:57.342757+0000","last_change":"2024-09-11T03:43:57.342580+0000","last_active":"2024-09-11T03:43:57.342757+0000","last_peered":"2024-09-11T03:43:57.342757+0000","last_clean":"2024-09-11T03:43:57.342757+0000","last_became_active":"2024-09-11T03:43:55.918743+0000","last_became_peered":"2024-09-11T03:43:55.918743+0000","last_unstale":"2024-09-11T03:43:57.342757+0000","last_undegraded":"2024-09-11T03:43:57.342757+0000","last_fullsized":"2024-09-11T03:43:57.342757+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:43:05.941234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:05.941234+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0047020209999999998,"stat_sum":{"num_bytes":10375123,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":67,"num_read_kb":44307,"num_write":75,"num_write_kb":27773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3216701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[10,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":450,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-11T03:43:56.949131+0000","last_change":"2024-09-11T03:42:50.826054+0000","last_active":"2024-09-11T03:43:56.949131+0000","last_peered":"2024-09-11T03:43:56.949131+0000","last_clean":"2024-09-11T03:43:56.949131+0000","last_became_active":"2024-09-11T03:40:27.847297+0000","last_became_peered":"2024-09-11T03:40:27.847297+0000","last_unstale":"2024-09-11T03:43:56.949131+0000","last_undegraded":"2024-09-11T03:43:56.949131+0000","last_fullsized":"2024-09-11T03:43:56.949131+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:50.825993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:50.825993+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:11.309951+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":727,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.915507+0000","last_change":"2024-09-11T03:43:54.914680+0000","last_active":"2024-09-11T03:43:55.915507+0000","last_peered":"2024-09-11T03:43:55.915507+0000","last_clean":"2024-09-11T03:43:55.915507+0000","last_became_active":"2024-09-11T03:41:35.874295+0000","last_became_peered":"2024-09-11T03:41:35.874295+0000","last_unstale":"2024-09-11T03:43:55.915507+0000","last_undegraded":"2024-09-11T03:43:55.915507+0000","last_fullsized":"2024-09-11T03:43:55.915507+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:43:23.459989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:23.459989+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:24.086793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00076525699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.13","version":"20'92","reported_seq":540,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.916737+0000","last_change":"2024-09-11T03:43:52.923463+0000","last_active":"2024-09-11T03:43:55.916737+0000","last_peered":"2024-09-11T03:43:55.916737+0000","last_clean":"2024-09-11T03:43:55.916737+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:43:55.916737+0000","last_undegraded":"2024-09-11T03:43:55.916737+0000","last_fullsized":"2024-09-11T03:43:55.916737+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:50.118049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:50.118049+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:07.455624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":1716,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-11T03:43:54.914393+0000","last_change":"2024-09-11T03:43:54.914393+0000","last_active":"2024-09-11T03:43:54.914393+0000","last_peered":"2024-09-11T03:43:54.914393+0000","last_clean":"2024-09-11T03:43:54.914393+0000","last_became_active":"2024-09-11T03:41:38.886813+0000","last_became_peered":"2024-09-11T03:41:38.886813+0000","last_unstale":"2024-09-11T03:43:54.914393+0000","last_undegraded":"2024-09-11T03:43:54.914393+0000","last_fullsized":"2024-09-11T03:43:54.914393+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:43:02.615095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:02.615095+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:31.986363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078221300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,9],"acting":[4,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":435,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.924812+0000","last_change":"2024-09-11T03:40:37.284871+0000","last_active":"2024-09-11T03:43:55.924812+0000","last_peered":"2024-09-11T03:43:55.924812+0000","last_clean":"2024-09-11T03:43:55.924812+0000","last_became_active":"2024-09-11T03:40:27.845249+0000","last_became_peered":"2024-09-11T03:40:27.845249+0000","last_unstale":"2024-09-11T03:43:55.924812+0000","last_undegraded":"2024-09-11T03:43:55.924812+0000","last_fullsized":"2024-09-11T03:43:55.924812+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:37.284800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:37.284800+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:56.216518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"418'262","reported_seq":1906,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-11T03:43:57.152075+0000","last_change":"2024-09-11T03:43:55.925155+0000","last_active":"2024-09-11T03:43:57.152075+0000","last_peered":"2024-09-11T03:43:57.152075+0000","last_clean":"2024-09-11T03:43:57.152075+0000","last_became_active":"2024-09-11T03:43:55.918817+0000","last_became_peered":"2024-09-11T03:43:55.918817+0000","last_unstale":"2024-09-11T03:43:57.152075+0000","last_undegraded":"2024-09-11T03:43:57.152075+0000","last_fullsized":"2024-09-11T03:43:57.152075+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"361'230","last_scrub_stamp":"2024-09-11T03:43:04.715581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:04.715581+0000","objects_scrubbed":2,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:17.748872+0000","scrub_duration":1,"objects_trimmed":2,"snaptrim_duration":0.0056255009999999998,"stat_sum":{"num_bytes":7320067,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":55376,"num_write":65,"num_write_kb":26837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":15216236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":441,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.915731+0000","last_change":"2024-09-11T03:40:35.703292+0000","last_active":"2024-09-11T03:43:55.915731+0000","last_peered":"2024-09-11T03:43:55.915731+0000","last_clean":"2024-09-11T03:43:55.915731+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:43:55.915731+0000","last_undegraded":"2024-09-11T03:43:55.915731+0000","last_fullsized":"2024-09-11T03:43:55.915731+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.703203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.703203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:54.072667+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":1727,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-11T03:43:58.347255+0000","last_change":"2024-09-11T03:43:54.923396+0000","last_active":"2024-09-11T03:43:58.347255+0000","last_peered":"2024-09-11T03:43:58.347255+0000","last_clean":"2024-09-11T03:43:58.347255+0000","last_became_active":"2024-09-11T03:41:38.886834+0000","last_became_peered":"2024-09-11T03:41:38.886834+0000","last_unstale":"2024-09-11T03:43:58.347255+0000","last_undegraded":"2024-09-11T03:43:58.347255+0000","last_fullsized":"2024-09-11T03:43:58.347255+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:43:06.659184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:06.659184+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:33.670984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0034139029999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":453,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.924773+0000","last_change":"2024-09-11T03:43:54.921928+0000","last_active":"2024-09-11T03:43:55.924773+0000","last_peered":"2024-09-11T03:43:55.924773+0000","last_clean":"2024-09-11T03:43:55.924773+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:43:55.924773+0000","last_undegraded":"2024-09-11T03:43:55.924773+0000","last_fullsized":"2024-09-11T03:43:55.924773+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:03.151323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:03.151323+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:24.434126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"417'399","reported_seq":2031,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.915502+0000","last_change":"2024-09-11T03:43:54.919094+0000","last_active":"2024-09-11T03:43:55.915502+0000","last_peered":"2024-09-11T03:43:55.915502+0000","last_clean":"2024-09-11T03:43:55.915502+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:43:55.915502+0000","last_undegraded":"2024-09-11T03:43:55.915502+0000","last_fullsized":"2024-09-11T03:43:55.915502+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'376","last_scrub_stamp":"2024-09-11T03:43:42.475580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:42.475580+0000","objects_scrubbed":5,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:50.434871+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0050256140000000003,"stat_sum":{"num_bytes":10314408,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":130,"num_read_kb":88280,"num_write":132,"num_write_kb":59525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":519,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.924913+0000","last_change":"2024-09-11T03:42:05.135033+0000","last_active":"2024-09-11T03:43:55.924913+0000","last_peered":"2024-09-11T03:43:55.924913+0000","last_clean":"2024-09-11T03:43:55.924913+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:43:55.924913+0000","last_undegraded":"2024-09-11T03:43:55.924913+0000","last_fullsized":"2024-09-11T03:43:55.924913+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:05.134977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:05.134977+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:16.379482+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"417'186","reported_seq":1827,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-11T03:43:56.951170+0000","last_change":"2024-09-11T03:43:54.922987+0000","last_active":"2024-09-11T03:43:56.951170+0000","last_peered":"2024-09-11T03:43:56.951170+0000","last_clean":"2024-09-11T03:43:56.951170+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:43:56.951170+0000","last_undegraded":"2024-09-11T03:43:56.951170+0000","last_fullsized":"2024-09-11T03:43:56.951170+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'174","last_scrub_stamp":"2024-09-11T03:43:28.950646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:28.950646+0000","objects_scrubbed":2,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:37.166113+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0059305160000000003,"stat_sum":{"num_bytes":6047955,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":129,"num_read_kb":136006,"num_write":70,"num_write_kb":25584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":846,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.917551+0000","last_change":"2024-09-11T03:42:00.816905+0000","last_active":"2024-09-11T03:43:55.917551+0000","last_peered":"2024-09-11T03:43:55.917551+0000","last_clean":"2024-09-11T03:43:55.917551+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:43:55.917551+0000","last_undegraded":"2024-09-11T03:43:55.917551+0000","last_fullsized":"2024-09-11T03:43:55.917551+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:00.816850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:00.816850+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:11.566006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"418'568","reported_seq":2307,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.934672+0000","last_change":"2024-09-11T03:43:54.917551+0000","last_active":"2024-09-11T03:43:55.934672+0000","last_peered":"2024-09-11T03:43:55.934672+0000","last_clean":"2024-09-11T03:43:55.934672+0000","last_became_active":"2024-09-11T03:41:33.865638+0000","last_became_peered":"2024-09-11T03:41:33.865638+0000","last_unstale":"2024-09-11T03:43:55.934672+0000","last_undegraded":"2024-09-11T03:43:55.934672+0000","last_fullsized":"2024-09-11T03:43:55.934672+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'498","last_scrub_stamp":"2024-09-11T03:43:03.902049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:03.902049+0000","objects_scrubbed":4,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:15.093949+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0043153430000000001,"stat_sum":{"num_bytes":19761736,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":188,"num_read_kb":220944,"num_write":93,"num_write_kb":33645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":762,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-11T03:43:58.347552+0000","last_change":"2024-09-11T03:41:45.227630+0000","last_active":"2024-09-11T03:43:58.347552+0000","last_peered":"2024-09-11T03:43:58.347552+0000","last_clean":"2024-09-11T03:43:58.347552+0000","last_became_active":"2024-09-11T03:40:37.849988+0000","last_became_peered":"2024-09-11T03:40:37.849988+0000","last_unstale":"2024-09-11T03:43:58.347552+0000","last_undegraded":"2024-09-11T03:43:58.347552+0000","last_fullsized":"2024-09-11T03:43:58.347552+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:45.227581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:45.227581+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:11.694058+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"417'281","reported_seq":1929,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.917880+0000","last_change":"2024-09-11T03:43:54.925438+0000","last_active":"2024-09-11T03:43:55.917880+0000","last_peered":"2024-09-11T03:43:55.917880+0000","last_clean":"2024-09-11T03:43:55.917880+0000","last_became_active":"2024-09-11T03:41:33.863875+0000","last_became_peered":"2024-09-11T03:41:33.863875+0000","last_unstale":"2024-09-11T03:43:55.917880+0000","last_undegraded":"2024-09-11T03:43:55.917880+0000","last_fullsized":"2024-09-11T03:43:55.917880+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'248","last_scrub_stamp":"2024-09-11T03:43:03.673970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:03.673970+0000","objects_scrubbed":1,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:32.644805+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0097910080000000003,"stat_sum":{"num_bytes":5193561,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":13815,"num_write":25,"num_write_kb":7046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":451,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-11T03:43:56.950993+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:43:56.950993+0000","last_peered":"2024-09-11T03:43:56.950993+0000","last_clean":"2024-09-11T03:43:56.950993+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:43:56.950993+0000","last_undegraded":"2024-09-11T03:43:56.950993+0000","last_fullsized":"2024-09-11T03:43:56.950993+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:43:56.995481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"418'537","reported_seq":2262,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-11T03:43:57.152109+0000","last_change":"2024-09-11T03:43:55.925767+0000","last_active":"2024-09-11T03:43:57.152109+0000","last_peered":"2024-09-11T03:43:57.152109+0000","last_clean":"2024-09-11T03:43:57.152109+0000","last_became_active":"2024-09-11T03:43:55.918858+0000","last_became_peered":"2024-09-11T03:43:55.918858+0000","last_unstale":"2024-09-11T03:43:57.152109+0000","last_undegraded":"2024-09-11T03:43:57.152109+0000","last_fullsized":"2024-09-11T03:43:57.152109+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'472","last_scrub_stamp":"2024-09-11T03:43:04.939811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:04.939811+0000","objects_scrubbed":5,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:12.416940+0000","scrub_duration":3,"objects_trimmed":2,"snaptrim_duration":0.0059832540000000003,"stat_sum":{"num_bytes":18807363,"num_objects":9,"num_object_clones":4,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":121,"num_read_kb":128068,"num_write":54,"num_write_kb":22028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":14282259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,3],"acting":[10,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":617,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-11T03:43:56.951062+0000","last_change":"2024-09-11T03:42:00.181197+0000","last_active":"2024-09-11T03:43:56.951062+0000","last_peered":"2024-09-11T03:43:56.951062+0000","last_clean":"2024-09-11T03:43:56.951062+0000","last_became_active":"2024-09-11T03:40:25.858281+0000","last_became_peered":"2024-09-11T03:40:25.858281+0000","last_unstale":"2024-09-11T03:43:56.951062+0000","last_undegraded":"2024-09-11T03:43:56.951062+0000","last_fullsized":"2024-09-11T03:43:56.951062+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:00.181139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:00.181139+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:00.932653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":1790,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.917523+0000","last_change":"2024-09-11T03:43:54.914373+0000","last_active":"2024-09-11T03:43:55.917523+0000","last_peered":"2024-09-11T03:43:55.917523+0000","last_clean":"2024-09-11T03:43:55.917523+0000","last_became_active":"2024-09-11T03:41:33.865499+0000","last_became_peered":"2024-09-11T03:41:33.865499+0000","last_unstale":"2024-09-11T03:43:55.917523+0000","last_undegraded":"2024-09-11T03:43:55.917523+0000","last_fullsized":"2024-09-11T03:43:55.917523+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:42:48.051821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:42:48.051821+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:16.734812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00068083399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":23,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-11T03:43:57.151964+0000","last_change":"2024-09-11T03:43:55.916457+0000","last_active":"2024-09-11T03:43:57.151964+0000","last_peered":"2024-09-11T03:43:57.151964+0000","last_clean":"2024-09-11T03:43:57.151964+0000","last_became_active":"2024-09-11T03:43:52.924325+0000","last_became_peered":"2024-09-11T03:43:52.924325+0000","last_unstale":"2024-09-11T03:43:57.151964+0000","last_undegraded":"2024-09-11T03:43:57.151964+0000","last_fullsized":"2024-09-11T03:43:57.151964+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:38.224231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:38.224231+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:44:25.348193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"417'549","reported_seq":2238,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.915606+0000","last_change":"2024-09-11T03:43:54.922977+0000","last_active":"2024-09-11T03:43:55.915606+0000","last_peered":"2024-09-11T03:43:55.915606+0000","last_clean":"2024-09-11T03:43:55.915606+0000","last_became_active":"2024-09-11T03:41:33.866316+0000","last_became_peered":"2024-09-11T03:41:33.866316+0000","last_unstale":"2024-09-11T03:43:55.915606+0000","last_undegraded":"2024-09-11T03:43:55.915606+0000","last_fullsized":"2024-09-11T03:43:55.915606+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'466","last_scrub_stamp":"2024-09-11T03:43:04.384893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:04.384893+0000","objects_scrubbed":4,"log_size":549,"log_dups_size":0,"ondisk_log_size":549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:04.851714+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0089069760000000005,"stat_sum":{"num_bytes":12763906,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":116,"num_read_kb":140670,"num_write":60,"num_write_kb":18898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":23,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-11T03:43:57.151979+0000","last_change":"2024-09-11T03:43:57.151979+0000","last_active":"2024-09-11T03:43:57.151979+0000","last_peered":"2024-09-11T03:43:57.151979+0000","last_clean":"2024-09-11T03:43:57.151979+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:43:57.151979+0000","last_undegraded":"2024-09-11T03:43:57.151979+0000","last_fullsized":"2024-09-11T03:43:57.151979+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:05.222462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:05.222462+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:44:26.305790+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":36063,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"418'264","reported_seq":1962,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.922843+0000","last_change":"2024-09-11T03:43:55.923413+0000","last_active":"2024-09-11T03:43:55.922843+0000","last_peered":"2024-09-11T03:43:55.922843+0000","last_clean":"2024-09-11T03:43:55.922843+0000","last_became_active":"2024-09-11T03:43:55.918081+0000","last_became_peered":"2024-09-11T03:43:55.918081+0000","last_unstale":"2024-09-11T03:43:55.922843+0000","last_undegraded":"2024-09-11T03:43:55.922843+0000","last_fullsized":"2024-09-11T03:43:55.922843+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'235","last_scrub_stamp":"2024-09-11T03:43:02.929206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:02.929206+0000","objects_scrubbed":1,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:15.839685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0047206569999999996,"stat_sum":{"num_bytes":8011393,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":33842,"num_write":35,"num_write_kb":7432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":4118225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,10],"acting":[0,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":28,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-11T03:43:57.152078+0000","last_change":"2024-09-11T03:43:54.313880+0000","last_active":"2024-09-11T03:43:57.152078+0000","last_peered":"2024-09-11T03:43:57.152078+0000","last_clean":"2024-09-11T03:43:57.152078+0000","last_became_active":"2024-09-11T03:43:52.923384+0000","last_became_peered":"2024-09-11T03:43:52.923384+0000","last_unstale":"2024-09-11T03:43:57.152078+0000","last_undegraded":"2024-09-11T03:43:57.152078+0000","last_fullsized":"2024-09-11T03:43:57.152078+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:54.313794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:54.313794+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:09.663654+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,5],"acting":[10,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"417'181","reported_seq":1868,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.924851+0000","last_change":"2024-09-11T03:43:54.924984+0000","last_active":"2024-09-11T03:43:55.924851+0000","last_peered":"2024-09-11T03:43:55.924851+0000","last_clean":"2024-09-11T03:43:55.924851+0000","last_became_active":"2024-09-11T03:42:30.902838+0000","last_became_peered":"2024-09-11T03:42:30.902838+0000","last_unstale":"2024-09-11T03:43:55.924851+0000","last_undegraded":"2024-09-11T03:43:55.924851+0000","last_fullsized":"2024-09-11T03:43:55.924851+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'139","last_scrub_stamp":"2024-09-11T03:43:03.241260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:03.241260+0000","objects_scrubbed":1,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:32.761644+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0029232329999999999,"stat_sum":{"num_bytes":6692531,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":34,"num_read_kb":27471,"num_write":18,"num_write_kb":8071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,11],"acting":[5,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":28,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-11T03:43:57.151915+0000","last_change":"2024-09-11T03:43:53.274298+0000","last_active":"2024-09-11T03:43:57.151915+0000","last_peered":"2024-09-11T03:43:57.151915+0000","last_clean":"2024-09-11T03:43:57.151915+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:43:57.151915+0000","last_undegraded":"2024-09-11T03:43:57.151915+0000","last_fullsized":"2024-09-11T03:43:57.151915+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:53.274236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:53.274236+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:18.478310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"405'143","reported_seq":1794,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-11T03:43:56.949242+0000","last_change":"2024-09-11T03:43:54.918625+0000","last_active":"2024-09-11T03:43:56.949242+0000","last_peered":"2024-09-11T03:43:56.949242+0000","last_clean":"2024-09-11T03:43:56.949242+0000","last_became_active":"2024-09-11T03:41:33.866217+0000","last_became_peered":"2024-09-11T03:41:33.866217+0000","last_unstale":"2024-09-11T03:43:56.949242+0000","last_undegraded":"2024-09-11T03:43:56.949242+0000","last_fullsized":"2024-09-11T03:43:56.949242+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"348'138","last_scrub_stamp":"2024-09-11T03:42:52.852516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:42:52.852516+0000","objects_scrubbed":2,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:06.069536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001340438,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":1777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":235,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.917725+0000","last_change":"2024-09-11T03:43:23.494479+0000","last_active":"2024-09-11T03:43:55.917725+0000","last_peered":"2024-09-11T03:43:55.917725+0000","last_clean":"2024-09-11T03:43:55.917725+0000","last_became_active":"2024-09-11T03:42:06.870500+0000","last_became_peered":"2024-09-11T03:42:06.870500+0000","last_unstale":"2024-09-11T03:43:55.917725+0000","last_undegraded":"2024-09-11T03:43:55.917725+0000","last_fullsized":"2024-09-11T03:43:55.917725+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:23.494418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:23.494418+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:51.064032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"418'113","reported_seq":1801,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.922591+0000","last_change":"2024-09-11T03:43:55.922919+0000","last_active":"2024-09-11T03:43:55.922591+0000","last_peered":"2024-09-11T03:43:55.922591+0000","last_clean":"2024-09-11T03:43:55.922591+0000","last_became_active":"2024-09-11T03:43:55.920265+0000","last_became_peered":"2024-09-11T03:43:55.920265+0000","last_unstale":"2024-09-11T03:43:55.922591+0000","last_undegraded":"2024-09-11T03:43:55.922591+0000","last_fullsized":"2024-09-11T03:43:55.922591+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'98","last_scrub_stamp":"2024-09-11T03:43:10.533291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:10.533291+0000","objects_scrubbed":2,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:15.965508+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0023081349999999998,"stat_sum":{"num_bytes":2336272,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":20533,"num_write":16,"num_write_kb":3640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4387035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":53,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-11T03:43:54.913632+0000","last_change":"2024-09-11T03:43:37.909230+0000","last_active":"2024-09-11T03:43:54.913632+0000","last_peered":"2024-09-11T03:43:54.913632+0000","last_clean":"2024-09-11T03:43:54.913632+0000","last_became_active":"2024-09-11T03:43:36.914119+0000","last_became_peered":"2024-09-11T03:43:36.914119+0000","last_unstale":"2024-09-11T03:43:54.913632+0000","last_undegraded":"2024-09-11T03:43:54.913632+0000","last_fullsized":"2024-09-11T03:43:54.913632+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:57.122773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:57.122773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:44:07.437742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":64112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"418'219","reported_seq":1870,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.941019+0000","last_change":"2024-09-11T03:43:54.924535+0000","last_active":"2024-09-11T03:43:55.941019+0000","last_peered":"2024-09-11T03:43:55.941019+0000","last_clean":"2024-09-11T03:43:55.941019+0000","last_became_active":"2024-09-11T03:41:33.862271+0000","last_became_peered":"2024-09-11T03:41:33.862271+0000","last_unstale":"2024-09-11T03:43:55.941019+0000","last_undegraded":"2024-09-11T03:43:55.941019+0000","last_fullsized":"2024-09-11T03:43:55.941019+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'173","last_scrub_stamp":"2024-09-11T03:42:47.254016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:42:47.254016+0000","objects_scrubbed":4,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:07.192977+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.002435085,"stat_sum":{"num_bytes":8535459,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":69,"num_read_kb":65698,"num_write":32,"num_write_kb":9028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":852,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.917963+0000","last_change":"2024-09-11T03:43:05.695738+0000","last_active":"2024-09-11T03:43:55.917963+0000","last_peered":"2024-09-11T03:43:55.917963+0000","last_clean":"2024-09-11T03:43:55.917963+0000","last_became_active":"2024-09-11T03:40:25.854624+0000","last_became_peered":"2024-09-11T03:40:25.854624+0000","last_unstale":"2024-09-11T03:43:55.917963+0000","last_undegraded":"2024-09-11T03:43:55.917963+0000","last_fullsized":"2024-09-11T03:43:55.917963+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:05.695673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:05.695673+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:12.649757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"20'92","reported_seq":437,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.925038+0000","last_change":"2024-09-11T03:40:35.282811+0000","last_active":"2024-09-11T03:43:55.925038+0000","last_peered":"2024-09-11T03:43:55.925038+0000","last_clean":"2024-09-11T03:43:55.925038+0000","last_became_active":"2024-09-11T03:40:27.844262+0000","last_became_peered":"2024-09-11T03:40:27.844262+0000","last_unstale":"2024-09-11T03:43:55.925038+0000","last_undegraded":"2024-09-11T03:43:55.925038+0000","last_fullsized":"2024-09-11T03:43:55.925038+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.282762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.282762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"417'418","reported_seq":2063,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-11T03:43:56.951271+0000","last_change":"2024-09-11T03:43:54.921377+0000","last_active":"2024-09-11T03:43:56.951271+0000","last_peered":"2024-09-11T03:43:56.951271+0000","last_clean":"2024-09-11T03:43:56.951271+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:43:56.951271+0000","last_undegraded":"2024-09-11T03:43:56.951271+0000","last_fullsized":"2024-09-11T03:43:56.951271+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'414","last_scrub_stamp":"2024-09-11T03:43:48.982261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:48.982261+0000","objects_scrubbed":4,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:12.112898+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0050341020000000004,"stat_sum":{"num_bytes":9176853,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":211,"num_read_kb":169481,"num_write":177,"num_write_kb":76051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":701,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-11T03:43:54.913595+0000","last_change":"2024-09-11T03:42:50.644051+0000","last_active":"2024-09-11T03:43:54.913595+0000","last_peered":"2024-09-11T03:43:54.913595+0000","last_clean":"2024-09-11T03:43:54.913595+0000","last_became_active":"2024-09-11T03:40:37.850623+0000","last_became_peered":"2024-09-11T03:40:37.850623+0000","last_unstale":"2024-09-11T03:43:54.913595+0000","last_undegraded":"2024-09-11T03:43:54.913595+0000","last_fullsized":"2024-09-11T03:43:54.913595+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:50.644003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:50.644003+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:11.778523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"417'567","reported_seq":2196,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.925081+0000","last_change":"2024-09-11T03:43:54.928239+0000","last_active":"2024-09-11T03:43:55.925081+0000","last_peered":"2024-09-11T03:43:55.925081+0000","last_clean":"2024-09-11T03:43:55.925081+0000","last_became_active":"2024-09-11T03:42:30.902959+0000","last_became_peered":"2024-09-11T03:42:30.902959+0000","last_unstale":"2024-09-11T03:43:55.925081+0000","last_undegraded":"2024-09-11T03:43:55.925081+0000","last_fullsized":"2024-09-11T03:43:55.925081+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'559","last_scrub_stamp":"2024-09-11T03:43:51.375966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:51.375966+0000","objects_scrubbed":9,"log_size":567,"log_dups_size":0,"ondisk_log_size":567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:19.752532+0000","scrub_duration":4,"objects_trimmed":1,"snaptrim_duration":0.0061012740000000003,"stat_sum":{"num_bytes":10168565,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":290,"num_read_kb":270537,"num_write":183,"num_write_kb":72739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17460823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":461,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.917694+0000","last_change":"2024-09-11T03:42:51.733829+0000","last_active":"2024-09-11T03:43:55.917694+0000","last_peered":"2024-09-11T03:43:55.917694+0000","last_clean":"2024-09-11T03:43:55.917694+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:43:55.917694+0000","last_undegraded":"2024-09-11T03:43:55.917694+0000","last_fullsized":"2024-09-11T03:43:55.917694+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:51.733762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:51.733762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:03.246292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"419'581","reported_seq":2196,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-11T03:43:56.962426+0000","last_change":"2024-09-11T03:43:54.924942+0000","last_active":"2024-09-11T03:43:56.962426+0000","last_peered":"2024-09-11T03:43:56.962426+0000","last_clean":"2024-09-11T03:43:56.962426+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:43:56.962426+0000","last_undegraded":"2024-09-11T03:43:56.962426+0000","last_fullsized":"2024-09-11T03:43:56.962426+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'435","last_scrub_stamp":"2024-09-11T03:42:28.893309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:42:28.893309+0000","objects_scrubbed":8,"log_size":581,"log_dups_size":0,"ondisk_log_size":581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":2,"snaptrim_duration":0.0076928059999999999,"stat_sum":{"num_bytes":15003296,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":230,"num_read_kb":203860,"num_write":184,"num_write_kb":74113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"2.f","version":"417'314","reported_seq":1950,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-11T03:43:55.915805+0000","last_change":"2024-09-11T03:43:54.923960+0000","last_active":"2024-09-11T03:43:55.915805+0000","last_peered":"2024-09-11T03:43:55.915805+0000","last_clean":"2024-09-11T03:43:55.915805+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:43:55.915805+0000","last_undegraded":"2024-09-11T03:43:55.915805+0000","last_fullsized":"2024-09-11T03:43:55.915805+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'238","last_scrub_stamp":"2024-09-11T03:42:50.411357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:42:50.411357+0000","objects_scrubbed":5,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:18.541578+0000","scrub_duration":16,"objects_trimmed":1,"snaptrim_duration":0.0096043280000000005,"stat_sum":{"num_bytes":7412184,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":131,"num_read_kb":97259,"num_write":115,"num_write_kb":44955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"8a","length":"1"},{"start":"98","length":"1"},{"start":"9a","length":"2"},{"start":"a0","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":482,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-11T03:43:57.957119+0000","last_change":"2024-09-11T03:43:50.727772+0000","last_active":"2024-09-11T03:43:57.957119+0000","last_peered":"2024-09-11T03:43:57.957119+0000","last_clean":"2024-09-11T03:43:57.957119+0000","last_became_active":"2024-09-11T03:40:37.847449+0000","last_became_peered":"2024-09-11T03:40:37.847449+0000","last_unstale":"2024-09-11T03:43:57.957119+0000","last_undegraded":"2024-09-11T03:43:57.957119+0000","last_fullsized":"2024-09-11T03:43:57.957119+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:50.727710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:50.727710+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:01.411341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":175996017,"num_objects":88,"num_object_clones":43,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":9,"num_read":1987,"num_read_kb":1833942,"num_write":1437,"num_write_kb":561046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":65054347,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":343076864,"data_stored":433287274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7829,"ondisk_log_size":7829,"up":78,"acting":78,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":635092,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":7,"num_write_kb":162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":148259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285649,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71772,"kb_used_data":36420,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93673380,"statfs":{"total":95995035648,"available":95921541120,"internally_reserved":0,"allocated":37294080,"data_stored":42127283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514581,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56880,"kb_used_data":25684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93688272,"statfs":{"total":95995035648,"available":95936790528,"internally_reserved":0,"allocated":26300416,"data_stored":29861859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743515,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75892,"kb_used_data":36388,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93669260,"statfs":{"total":95995035648,"available":95917322240,"internally_reserved":0,"allocated":37261312,"data_stored":43842451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68344,"kb_used_data":28832,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93676808,"statfs":{"total":95995035648,"available":95925051392,"internally_reserved":0,"allocated":29523968,"data_stored":32250710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299493,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85784,"kb_used_data":46272,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93659368,"statfs":{"total":95995035648,"available":95907192832,"internally_reserved":0,"allocated":47382528,"data_stored":55312104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":193273528426,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757360,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61736,"kb_used_data":26380,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93683416,"statfs":{"total":95995035648,"available":95931817984,"internally_reserved":0,"allocated":27013120,"data_stored":32438226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986293,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87872,"kb_used_data":48360,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93657280,"statfs":{"total":95995035648,"available":95905054720,"internally_reserved":0,"allocated":49520640,"data_stored":60127633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87580,"kb_used_data":60544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93657572,"statfs":{"total":95995035648,"available":95905353728,"internally_reserved":0,"allocated":61997056,"data_stored":73535424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411456,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66724,"kb_used_data":31376,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93678428,"statfs":{"total":95995035648,"available":95926710272,"internally_reserved":0,"allocated":32129024,"data_stored":39418179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70028,"kb_used_data":30516,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93675124,"statfs":{"total":95995035648,"available":95923326976,"internally_reserved":0,"allocated":31248384,"data_stored":35265210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738506,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78852,"kb_used_data":39340,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93666300,"statfs":{"total":95995035648,"available":95914291200,"internally_reserved":0,"allocated":40284160,"data_stored":47630102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25698304,"data_stored":32689287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28033024,"data_stored":33896881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":26914816,"data_stored":36063018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":56659968,"data_stored":71491728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36155392,"data_stored":47576271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":21811200,"data_stored":29091279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4079616,"data_stored":4066399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":38092800,"data_stored":47628251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24539136,"data_stored":29112909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":31924224,"data_stored":40360858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":21291008,"data_stored":26707946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":27877376,"data_stored":34602447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:44:00.358 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pgp_num 51 2024-09-11T03:44:00.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:00.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:00.390 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:00.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:00.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:00.509 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:44:00.509 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:44:00.513 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:00.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:00.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:00.526 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:44:00.527 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:44:00.544 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:44:00.544 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:44:00.566 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:44:00.566 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:44:00.584 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:44:00.584 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:44:00.601 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:44:00.602 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:44:00.614 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:00.615 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:00.619 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:44:00.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:44:00.637 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:44:00.637 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:44:00.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:00.638 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:00.639 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:00.654 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:44:00.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:44:00.671 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:44:00.671 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:44:00.688 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:44:00.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:44:00.691 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:44:00.705 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:44:00.705 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:44:00.740 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:00.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:00.763 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:00.763 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:00.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:00.886 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:00.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:00.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:00.894 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T03:44:00.972 INFO:tasks.rados.rados.2.smithi175.stdout:1545: done (4 left) 2024-09-11T03:44:00.972 INFO:tasks.rados.rados.2.smithi175.stdout:1546: done (3 left) 2024-09-11T03:44:00.972 INFO:tasks.rados.rados.2.smithi175.stdout:1547: done (2 left) 2024-09-11T03:44:00.972 INFO:tasks.rados.rados.2.smithi175.stdout:1548: done (1 left) 2024-09-11T03:44:00.972 INFO:tasks.rados.rados.2.smithi175.stdout:1549: done (0 left) 2024-09-11T03:44:00.973 INFO:tasks.rados.rados.2.smithi175.stdout:1550: write oid 27 current snap is 166 2024-09-11T03:44:00.973 INFO:tasks.rados.rados.2.smithi175.stdout:1550: seq_num 545 ranges {759815=471768,1742451=465507,2917178=242637} 2024-09-11T03:44:00.978 INFO:tasks.rados.rados.2.smithi175.stdout:1550: writing smithi17557870-27 from 759815 to 1231583 tid 1 2024-09-11T03:44:00.983 INFO:tasks.rados.rados.2.smithi175.stdout:1550: writing smithi17557870-27 from 1742451 to 2207958 tid 2 2024-09-11T03:44:00.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:00.989 INFO:tasks.rados.rados.2.smithi175.stdout:1550: writing smithi17557870-27 from 2917178 to 3159815 tid 3 2024-09-11T03:44:00.989 INFO:tasks.rados.rados.2.smithi175.stdout:1551: write oid 30 current snap is 166 2024-09-11T03:44:00.989 INFO:tasks.rados.rados.2.smithi175.stdout:1550: finishing write tid 1 to smithi17557870-27 2024-09-11T03:44:00.989 INFO:tasks.rados.rados.2.smithi175.stdout:1550: finishing write tid 2 to smithi17557870-27 2024-09-11T03:44:00.990 INFO:tasks.rados.rados.2.smithi175.stdout:1551: seq_num 546 ranges {776622=547017,1800933=725518,2944601=232021} 2024-09-11T03:44:00.996 INFO:tasks.rados.rados.2.smithi175.stdout:1551: writing smithi17557870-30 from 776622 to 1323639 tid 1 2024-09-11T03:44:01.004 INFO:tasks.rados.rados.2.smithi175.stdout:1551: writing smithi17557870-30 from 1800933 to 2526451 tid 2 2024-09-11T03:44:01.007 INFO:tasks.rados.rados.2.smithi175.stdout:1551: writing smithi17557870-30 from 2944601 to 3176622 tid 3 2024-09-11T03:44:01.007 INFO:tasks.rados.rados.2.smithi175.stdout:1552: snap_create 2024-09-11T03:44:01.007 INFO:tasks.rados.rados.2.smithi175.stdout:1550: finishing write tid 3 to smithi17557870-27 2024-09-11T03:44:01.007 INFO:tasks.rados.rados.2.smithi175.stdout:1550: finishing write tid 4 to smithi17557870-27 2024-09-11T03:44:01.007 INFO:tasks.rados.rados.2.smithi175.stdout:1550: finishing write tid 5 to smithi17557870-27 2024-09-11T03:44:01.007 INFO:tasks.rados.rados.2.smithi175.stdout:1550: oid 27 updating version 0 to 542 2024-09-11T03:44:01.007 INFO:tasks.rados.rados.2.smithi175.stdout:1550: oid 27 version 542 is already newer than 541 2024-09-11T03:44:01.007 INFO:tasks.rados.rados.2.smithi175.stdout:1550: oid 27 updating version 542 to 543 2024-09-11T03:44:01.007 INFO:tasks.rados.rados.2.smithi175.stdout:1550: oid 27 version 543 is already newer than 540 2024-09-11T03:44:01.007 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 543 (ObjNum 545 snap 166 seq_num 545) dirty exists 2024-09-11T03:44:01.007 INFO:tasks.rados.rados.2.smithi175.stdout:1550: left oid 27 (ObjNum 545 snap 166 seq_num 545) 2024-09-11T03:44:01.008 INFO:tasks.rados.rados.2.smithi175.stdout:1551: finishing write tid 1 to smithi17557870-30 2024-09-11T03:44:01.008 INFO:tasks.rados.rados.2.smithi175.stdout:1551: finishing write tid 2 to smithi17557870-30 2024-09-11T03:44:01.009 INFO:tasks.rados.rados.2.smithi175.stdout:1551: finishing write tid 3 to smithi17557870-30 2024-09-11T03:44:01.009 INFO:tasks.rados.rados.2.smithi175.stdout:1551: finishing write tid 4 to smithi17557870-30 2024-09-11T03:44:01.009 INFO:tasks.rados.rados.2.smithi175.stdout:1551: finishing write tid 5 to smithi17557870-30 2024-09-11T03:44:01.009 INFO:tasks.rados.rados.2.smithi175.stdout:1551: oid 30 updating version 0 to 586 2024-09-11T03:44:01.009 INFO:tasks.rados.rados.2.smithi175.stdout:1551: oid 30 version 586 is already newer than 583 2024-09-11T03:44:01.009 INFO:tasks.rados.rados.2.smithi175.stdout:1551: oid 30 version 586 is already newer than 584 2024-09-11T03:44:01.009 INFO:tasks.rados.rados.2.smithi175.stdout:1551: oid 30 version 586 is already newer than 585 2024-09-11T03:44:01.009 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 586 (ObjNum 546 snap 166 seq_num 546) dirty exists 2024-09-11T03:44:01.009 INFO:tasks.rados.rados.2.smithi175.stdout:1551: left oid 30 (ObjNum 546 snap 166 seq_num 546) 2024-09-11T03:44:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:01.010 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:01.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:01.111 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:01.133 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:01.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:01.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:00 smithi121 ceph-mon[30729]: osdmap e422: 12 total, 12 up, 11 in 2024-09-11T03:44:01.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:00 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/275263301' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:01.178 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:44:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:00 smithi175 ceph-mon[30894]: osdmap e422: 12 total, 12 up, 11 in 2024-09-11T03:44:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:00 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/275263301' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:01.235 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:01.255 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:01.256 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:01.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:01.379 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:01.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:01.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:00 smithi084 ceph-mon[27254]: osdmap e422: 12 total, 12 up, 11 in 2024-09-11T03:44:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:00 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/275263301' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:01.481 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:01.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:01.503 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:01.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:01.605 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:01.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:01.627 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:01.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:01.641 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:44:01.642 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:44:01.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:44:01.642 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:44:01.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.642 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:221 2.3 2:c6c49f2c:::smithi17557870-44:head [copy-from ver 9] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:44:01.643 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941456+0000", 2024-09-11T03:44:01.643 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 321.69909064199999, 2024-09-11T03:44:01.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015312882, 2024-09-11T03:44:01.643 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.643 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.643 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:01.643 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.644 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.644 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.644 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 221 2024-09-11T03:44:01.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.644 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941456+0000", 2024-09-11T03:44:01.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941456+0000", 2024-09-11T03:44:01.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941459+0000", 2024-09-11T03:44:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9189999999999999e-06 2024-09-11T03:44:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941469+0000", 2024-09-11T03:44:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0069e-05 2024-09-11T03:44:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941490+0000", 2024-09-11T03:44:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0947000000000001e-05 2024-09-11T03:44:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941501+0000", 2024-09-11T03:44:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0169e-05 2024-09-11T03:44:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941585+0000", 2024-09-11T03:44:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4022000000000003e-05 2024-09-11T03:44:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:44:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941590+0000", 2024-09-11T03:44:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4929999999999998e-06 2024-09-11T03:44:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.942889+0000", 2024-09-11T03:44:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001298539 2024-09-11T03:44:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.942984+0000", 2024-09-11T03:44:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5513999999999994e-05 2024-09-11T03:44:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:44:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.950816+0000", 2024-09-11T03:44:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078315850000000003 2024-09-11T03:44:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956638+0000", 2024-09-11T03:44:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058225769999999998 2024-09-11T03:44:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:44:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956649+0000", 2024-09-11T03:44:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0492000000000001e-05 2024-09-11T03:44:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956690+0000", 2024-09-11T03:44:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.142e-05 2024-09-11T03:44:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956728+0000", 2024-09-11T03:44:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8195000000000003e-05 2024-09-11T03:44:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956769+0000", 2024-09-11T03:44:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0941e-05 2024-09-11T03:44:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:222 2.3 2:c6c49f2c:::smithi17557870-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:44:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:44:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 321.699012563, 2024-09-11T03:44:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015032419, 2024-09-11T03:44:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:44:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 222 2024-09-11T03:44:01.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.654 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:44:01.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:44:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:44:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.51e-07 2024-09-11T03:44:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941539+0000", 2024-09-11T03:44:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0060000000000003e-06 2024-09-11T03:44:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941547+0000", 2024-09-11T03:44:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9009999999999994e-06 2024-09-11T03:44:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941553+0000", 2024-09-11T03:44:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1519999999999997e-06 2024-09-11T03:44:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941691+0000", 2024-09-11T03:44:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013752600000000001 2024-09-11T03:44:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941697+0000", 2024-09-11T03:44:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3770000000000001e-06 2024-09-11T03:44:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943170+0000", 2024-09-11T03:44:01.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001473504 2024-09-11T03:44:01.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:01.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943225+0000", 2024-09-11T03:44:01.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4101000000000003e-05 2024-09-11T03:44:01.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956388+0000", 2024-09-11T03:44:01.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013163223999999999 2024-09-11T03:44:01.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956456+0000", 2024-09-11T03:44:01.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8563000000000001e-05 2024-09-11T03:44:01.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956567+0000", 2024-09-11T03:44:01.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011061399999999999 2024-09-11T03:44:01.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.660 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.660 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.660 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:223 2.3 2:c3768452:::smithi17557870-11:head [copy-from ver 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:44:01.660 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:44:01.660 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 321.69897251800001, 2024-09-11T03:44:01.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016727301999999999, 2024-09-11T03:44:01.660 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.660 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.660 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:01.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.661 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.661 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.661 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 223 2024-09-11T03:44:01.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.661 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:44:01.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:44:01.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:44:01.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0999999999999999e-07 2024-09-11T03:44:01.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941578+0000", 2024-09-11T03:44:01.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.021e-06 2024-09-11T03:44:01.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941583+0000", 2024-09-11T03:44:01.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0520000000000001e-06 2024-09-11T03:44:01.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941586+0000", 2024-09-11T03:44:01.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-09-11T03:44:01.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941750+0000", 2024-09-11T03:44:01.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016353200000000001 2024-09-11T03:44:01.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:01.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941756+0000", 2024-09-11T03:44:01.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.75e-06 2024-09-11T03:44:01.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943289+0000", 2024-09-11T03:44:01.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001533747 2024-09-11T03:44:01.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943383+0000", 2024-09-11T03:44:01.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3655999999999999e-05 2024-09-11T03:44:01.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:44:01.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.954899+0000", 2024-09-11T03:44:01.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011515463 2024-09-11T03:44:01.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:44:01.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.957952+0000", 2024-09-11T03:44:01.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030533560000000001 2024-09-11T03:44:01.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958200+0000", 2024-09-11T03:44:01.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024759100000000002 2024-09-11T03:44:01.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958229+0000", 2024-09-11T03:44:01.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9862e-05 2024-09-11T03:44:01.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:01.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958262+0000", 2024-09-11T03:44:01.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2267000000000001e-05 2024-09-11T03:44:01.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958302+0000", 2024-09-11T03:44:01.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0204000000000003e-05 2024-09-11T03:44:01.668 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.668 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.668 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.669 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:224 2.3 2:c3768452:::smithi17557870-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:44:01.669 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941599+0000", 2024-09-11T03:44:01.669 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 321.69894796599999, 2024-09-11T03:44:01.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014750406000000001, 2024-09-11T03:44:01.669 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.669 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.669 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:44:01.669 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.669 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.669 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.670 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 224 2024-09-11T03:44:01.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.670 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941599+0000", 2024-09-11T03:44:01.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941599+0000", 2024-09-11T03:44:01.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941599+0000", 2024-09-11T03:44:01.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2599999999999998e-07 2024-09-11T03:44:01.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941601+0000", 2024-09-11T03:44:01.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6249999999999999e-06 2024-09-11T03:44:01.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941605+0000", 2024-09-11T03:44:01.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3059999999999999e-06 2024-09-11T03:44:01.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941609+0000", 2024-09-11T03:44:01.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.506e-06 2024-09-11T03:44:01.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941789+0000", 2024-09-11T03:44:01.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000180261 2024-09-11T03:44:01.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:01.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941795+0000", 2024-09-11T03:44:01.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4970000000000001e-06 2024-09-11T03:44:01.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943572+0000", 2024-09-11T03:44:01.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001777725 2024-09-11T03:44:01.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:01.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943616+0000", 2024-09-11T03:44:01.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3550999999999997e-05 2024-09-11T03:44:01.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956192+0000", 2024-09-11T03:44:01.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012576517000000001 2024-09-11T03:44:01.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956267+0000", 2024-09-11T03:44:01.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4956000000000005e-05 2024-09-11T03:44:01.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956350+0000", 2024-09-11T03:44:01.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2136000000000001e-05 2024-09-11T03:44:01.676 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.676 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.676 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.676 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:343 2.2 2:494ba932:::smithi17557870-43:head [copy-from ver 6] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:44:01.676 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.820776+0000", 2024-09-11T03:44:01.676 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 311.81977137600001, 2024-09-11T03:44:01.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017461323000000001, 2024-09-11T03:44:01.676 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.676 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.676 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:01.676 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.677 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.677 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.677 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 343 2024-09-11T03:44:01.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.677 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820776+0000", 2024-09-11T03:44:01.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820776+0000", 2024-09-11T03:44:01.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820779+0000", 2024-09-11T03:44:01.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3000000000000002e-06 2024-09-11T03:44:01.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820797+0000", 2024-09-11T03:44:01.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7730000000000001e-05 2024-09-11T03:44:01.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820853+0000", 2024-09-11T03:44:01.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5909000000000003e-05 2024-09-11T03:44:01.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820872+0000", 2024-09-11T03:44:01.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9709999999999999e-05 2024-09-11T03:44:01.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820976+0000", 2024-09-11T03:44:01.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010374099999999999 2024-09-11T03:44:01.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:44:01.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820982+0000", 2024-09-11T03:44:01.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4389999999999999e-06 2024-09-11T03:44:01.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825547+0000", 2024-09-11T03:44:01.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045658260000000003 2024-09-11T03:44:01.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825626+0000", 2024-09-11T03:44:01.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8289999999999998e-05 2024-09-11T03:44:01.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,6", 2024-09-11T03:44:01.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830895+0000", 2024-09-11T03:44:01.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052695060000000002 2024-09-11T03:44:01.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:44:01.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.835040+0000", 2024-09-11T03:44:01.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041446549999999997 2024-09-11T03:44:01.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.835364+0000", 2024-09-11T03:44:01.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032384200000000002 2024-09-11T03:44:01.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838162+0000", 2024-09-11T03:44:01.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027987509999999999 2024-09-11T03:44:01.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:01.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838203+0000", 2024-09-11T03:44:01.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0209000000000001e-05 2024-09-11T03:44:01.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838237+0000", 2024-09-11T03:44:01.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4415000000000002e-05 2024-09-11T03:44:01.685 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.685 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.685 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.685 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:344 2.2 2:494ba932:::smithi17557870-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:44:01.686 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.820917+0000", 2024-09-11T03:44:01.686 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 311.81962974999999, 2024-09-11T03:44:01.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013455992, 2024-09-11T03:44:01.686 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.686 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.686 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:44:01.686 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.686 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.686 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.687 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 344 2024-09-11T03:44:01.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.687 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820917+0000", 2024-09-11T03:44:01.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820917+0000", 2024-09-11T03:44:01.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820918+0000", 2024-09-11T03:44:01.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.34e-07 2024-09-11T03:44:01.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820920+0000", 2024-09-11T03:44:01.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.48e-06 2024-09-11T03:44:01.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820927+0000", 2024-09-11T03:44:01.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.187e-06 2024-09-11T03:44:01.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820931+0000", 2024-09-11T03:44:01.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9349999999999996e-06 2024-09-11T03:44:01.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.821034+0000", 2024-09-11T03:44:01.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010277600000000001 2024-09-11T03:44:01.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:01.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.821040+0000", 2024-09-11T03:44:01.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6389999999999997e-06 2024-09-11T03:44:01.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825767+0000", 2024-09-11T03:44:01.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047268060000000001 2024-09-11T03:44:01.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:01.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825844+0000", 2024-09-11T03:44:01.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7066999999999999e-05 2024-09-11T03:44:01.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834208+0000", 2024-09-11T03:44:01.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083646199999999997 2024-09-11T03:44:01.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834284+0000", 2024-09-11T03:44:01.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5765000000000004e-05 2024-09-11T03:44:01.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834373+0000", 2024-09-11T03:44:01.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9283000000000006e-05 2024-09-11T03:44:01.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.693 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.693 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1068 2.3 2:c3768452:::smithi17557870-11:head [write 507611~777843,stat] snapc 28=[28,27,26] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:44:01.693 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.871196+0000", 2024-09-11T03:44:01.693 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 249.769351328, 2024-09-11T03:44:01.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.036362196999999999, 2024-09-11T03:44:01.693 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.694 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.694 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:01.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.694 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1068 2024-09-11T03:44:01.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.694 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.871196+0000", 2024-09-11T03:44:01.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.871196+0000", 2024-09-11T03:44:01.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.871198+0000", 2024-09-11T03:44:01.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5000000000000002e-06 2024-09-11T03:44:01.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873403+0000", 2024-09-11T03:44:01.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022048739999999999 2024-09-11T03:44:01.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873438+0000", 2024-09-11T03:44:01.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4518999999999999e-05 2024-09-11T03:44:01.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873453+0000", 2024-09-11T03:44:01.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5733999999999999e-05 2024-09-11T03:44:01.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873597+0000", 2024-09-11T03:44:01.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143702 2024-09-11T03:44:01.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873668+0000", 2024-09-11T03:44:01.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0996999999999998e-05 2024-09-11T03:44:01.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:44:01.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873819+0000", 2024-09-11T03:44:01.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015105300000000001 2024-09-11T03:44:01.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:44:01.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.875607+0000", 2024-09-11T03:44:01.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017877419999999999 2024-09-11T03:44:01.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.875920+0000", 2024-09-11T03:44:01.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031312799999999997 2024-09-11T03:44:01.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907471+0000", 2024-09-11T03:44:01.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.031551348 2024-09-11T03:44:01.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:01.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907514+0000", 2024-09-11T03:44:01.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2873000000000001e-05 2024-09-11T03:44:01.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907558+0000", 2024-09-11T03:44:01.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3727000000000003e-05 2024-09-11T03:44:01.703 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.703 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.703 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.704 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1069 2.3 2:c3768452:::smithi17557870-11:head [write 2073973~444721] snapc 28=[28,27,26] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:44:01.704 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.877745+0000", 2024-09-11T03:44:01.704 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 249.76280185100001, 2024-09-11T03:44:01.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029936307999999998, 2024-09-11T03:44:01.704 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.704 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.704 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:01.704 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.705 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.705 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.705 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1069 2024-09-11T03:44:01.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.705 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.877745+0000", 2024-09-11T03:44:01.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.877745+0000", 2024-09-11T03:44:01.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.877747+0000", 2024-09-11T03:44:01.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.61e-06 2024-09-11T03:44:01.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878169+0000", 2024-09-11T03:44:01.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00042167499999999999 2024-09-11T03:44:01.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878181+0000", 2024-09-11T03:44:01.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2714999999999999e-05 2024-09-11T03:44:01.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878187+0000", 2024-09-11T03:44:01.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2389999999999999e-06 2024-09-11T03:44:01.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878284+0000", 2024-09-11T03:44:01.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6632000000000006e-05 2024-09-11T03:44:01.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878363+0000", 2024-09-11T03:44:01.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9369000000000005e-05 2024-09-11T03:44:01.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:44:01.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878472+0000", 2024-09-11T03:44:01.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108995 2024-09-11T03:44:01.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:44:01.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879469+0000", 2024-09-11T03:44:01.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00099629100000000006 2024-09-11T03:44:01.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879835+0000", 2024-09-11T03:44:01.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036609700000000002 2024-09-11T03:44:01.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907609+0000", 2024-09-11T03:44:01.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027773923999999998 2024-09-11T03:44:01.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:01.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907647+0000", 2024-09-11T03:44:01.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8124000000000003e-05 2024-09-11T03:44:01.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907682+0000", 2024-09-11T03:44:01.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4637000000000001e-05 2024-09-11T03:44:01.711 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.711 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.711 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.711 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1070 2.3 2:c3768452:::smithi17557870-11:head [setxattr _header (57) in=7b,truncate 2907611] snapc 28=[28,27,26] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:44:01.711 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.878201+0000", 2024-09-11T03:44:01.711 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 249.76234605499999, 2024-09-11T03:44:01.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029581774000000002, 2024-09-11T03:44:01.712 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.712 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.712 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:01.712 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.712 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.712 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.712 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1070 2024-09-11T03:44:01.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.712 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878201+0000", 2024-09-11T03:44:01.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878201+0000", 2024-09-11T03:44:01.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878201+0000", 2024-09-11T03:44:01.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4299999999999998e-07 2024-09-11T03:44:01.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878222+0000", 2024-09-11T03:44:01.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0591999999999999e-05 2024-09-11T03:44:01.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878228+0000", 2024-09-11T03:44:01.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.857e-06 2024-09-11T03:44:01.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878232+0000", 2024-09-11T03:44:01.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6459999999999999e-06 2024-09-11T03:44:01.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879123+0000", 2024-09-11T03:44:01.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089174700000000003 2024-09-11T03:44:01.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879183+0000", 2024-09-11T03:44:01.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9502000000000001e-05 2024-09-11T03:44:01.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:44:01.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879287+0000", 2024-09-11T03:44:01.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103952 2024-09-11T03:44:01.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:44:01.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879606+0000", 2024-09-11T03:44:01.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031872600000000002 2024-09-11T03:44:01.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879933+0000", 2024-09-11T03:44:01.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032754700000000003 2024-09-11T03:44:01.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907711+0000", 2024-09-11T03:44:01.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027778265999999999 2024-09-11T03:44:01.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:01.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907746+0000", 2024-09-11T03:44:01.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4403999999999998e-05 2024-09-11T03:44:01.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907783+0000", 2024-09-11T03:44:01.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7092000000000003e-05 2024-09-11T03:44:01.718 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.718 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.718 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.718 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1071 2.3 2:c3768452:::smithi17557870-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:44:01.718 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.878273+0000", 2024-09-11T03:44:01.718 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 249.7622744, 2024-09-11T03:44:01.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.02966705, 2024-09-11T03:44:01.718 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.719 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.719 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:44:01.719 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.719 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.719 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.719 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1071 2024-09-11T03:44:01.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.719 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878273+0000", 2024-09-11T03:44:01.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878273+0000", 2024-09-11T03:44:01.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878273+0000", 2024-09-11T03:44:01.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1600000000000002e-07 2024-09-11T03:44:01.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878275+0000", 2024-09-11T03:44:01.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0949999999999998e-06 2024-09-11T03:44:01.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878281+0000", 2024-09-11T03:44:01.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6550000000000001e-06 2024-09-11T03:44:01.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878284+0000", 2024-09-11T03:44:01.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5370000000000002e-06 2024-09-11T03:44:01.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879510+0000", 2024-09-11T03:44:01.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001225638 2024-09-11T03:44:01.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:44:01.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879586+0000", 2024-09-11T03:44:01.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6058999999999998e-05 2024-09-11T03:44:01.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907811+0000", 2024-09-11T03:44:01.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028224775000000001 2024-09-11T03:44:01.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907873+0000", 2024-09-11T03:44:01.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2310000000000005e-05 2024-09-11T03:44:01.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907940+0000", 2024-09-11T03:44:01.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6564999999999997e-05 2024-09-11T03:44:01.723 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.724 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.724 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.724 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1555 2.3 2:c6c49f2c:::smithi17557870-44:head [copy-from ver 82] snapc 3c=[3c,3a,38,37] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-11T03:44:01.724 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:30.880293+0000", 2024-09-11T03:44:01.724 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 210.760254377, 2024-09-11T03:44:01.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020483438999999999, 2024-09-11T03:44:01.724 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.724 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.724 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:01.724 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.724 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.725 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.725 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1555 2024-09-11T03:44:01.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.725 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880293+0000", 2024-09-11T03:44:01.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880293+0000", 2024-09-11T03:44:01.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880295+0000", 2024-09-11T03:44:01.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2409999999999998e-06 2024-09-11T03:44:01.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880305+0000", 2024-09-11T03:44:01.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5999999999999996e-06 2024-09-11T03:44:01.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880326+0000", 2024-09-11T03:44:01.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1248999999999999e-05 2024-09-11T03:44:01.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880335+0000", 2024-09-11T03:44:01.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4369999999999996e-06 2024-09-11T03:44:01.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880414+0000", 2024-09-11T03:44:01.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8560999999999995e-05 2024-09-11T03:44:01.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:44:01.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880422+0000", 2024-09-11T03:44:01.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.712e-06 2024-09-11T03:44:01.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887559+0000", 2024-09-11T03:44:01.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071376920000000002 2024-09-11T03:44:01.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887625+0000", 2024-09-11T03:44:01.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6256999999999999e-05 2024-09-11T03:44:01.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:44:01.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.893371+0000", 2024-09-11T03:44:01.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0057455579999999996 2024-09-11T03:44:01.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.899577+0000", 2024-09-11T03:44:01.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062060600000000002 2024-09-11T03:44:01.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:44:01.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900444+0000", 2024-09-11T03:44:01.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000867307 2024-09-11T03:44:01.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900703+0000", 2024-09-11T03:44:01.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00025901299999999999 2024-09-11T03:44:01.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:01.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900743+0000", 2024-09-11T03:44:01.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9471000000000002e-05 2024-09-11T03:44:01.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900776+0000", 2024-09-11T03:44:01.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3281000000000002e-05 2024-09-11T03:44:01.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.731 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.732 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1556 2.3 2:c6c49f2c:::smithi17557870-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e214)", 2024-09-11T03:44:01.732 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:30.880359+0000", 2024-09-11T03:44:01.732 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 210.76018768, 2024-09-11T03:44:01.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017105065999999999, 2024-09-11T03:44:01.732 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.732 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.732 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:44:01.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.732 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1556 2024-09-11T03:44:01.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.733 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880359+0000", 2024-09-11T03:44:01.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880359+0000", 2024-09-11T03:44:01.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880360+0000", 2024-09-11T03:44:01.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.75e-07 2024-09-11T03:44:01.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880362+0000", 2024-09-11T03:44:01.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4959999999999999e-06 2024-09-11T03:44:01.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880370+0000", 2024-09-11T03:44:01.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.153e-06 2024-09-11T03:44:01.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880376+0000", 2024-09-11T03:44:01.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0900000000000001e-06 2024-09-11T03:44:01.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880478+0000", 2024-09-11T03:44:01.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102482 2024-09-11T03:44:01.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:01.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880487+0000", 2024-09-11T03:44:01.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8109999999999992e-06 2024-09-11T03:44:01.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887725+0000", 2024-09-11T03:44:01.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072383930000000001 2024-09-11T03:44:01.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:01.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887757+0000", 2024-09-11T03:44:01.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1692000000000001e-05 2024-09-11T03:44:01.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.897328+0000", 2024-09-11T03:44:01.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095712149999999992 2024-09-11T03:44:01.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.897393+0000", 2024-09-11T03:44:01.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5005000000000005e-05 2024-09-11T03:44:01.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.897464+0000", 2024-09-11T03:44:01.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1254000000000006e-05 2024-09-11T03:44:01.738 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.738 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.738 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.738 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3040 2.d 2:b6a25a79:::smithi17557870-47:head [read 0~2201783,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:44:01.738 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:44:01.738 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.743567452999997, 2024-09-11T03:44:01.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020060432, 2024-09-11T03:44:01.739 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.739 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.739 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:44:01.739 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.739 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.739 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.739 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3040 2024-09-11T03:44:01.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.739 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:44:01.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:44:01.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896981+0000", 2024-09-11T03:44:01.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.747e-06 2024-09-11T03:44:01.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896989+0000", 2024-09-11T03:44:01.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9559999999999997e-06 2024-09-11T03:44:01.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897005+0000", 2024-09-11T03:44:01.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5704000000000001e-05 2024-09-11T03:44:01.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897013+0000", 2024-09-11T03:44:01.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.028e-06 2024-09-11T03:44:01.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897078+0000", 2024-09-11T03:44:01.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4813000000000004e-05 2024-09-11T03:44:01.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:44:01.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897083+0000", 2024-09-11T03:44:01.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1109999999999999e-06 2024-09-11T03:44:01.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.914637+0000", 2024-09-11T03:44:01.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017553701000000001 2024-09-11T03:44:01.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.914744+0000", 2024-09-11T03:44:01.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107131 2024-09-11T03:44:01.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917040+0000", 2024-09-11T03:44:01.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022962410000000001 2024-09-11T03:44:01.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.743 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.744 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3041 2.d 2:b6a25a79:::smithi17557870-47:head [sparse-read 0~1480887] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:44:01.744 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:44:01.744 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.743501897000002, 2024-09-11T03:44:01.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020163026000000001, 2024-09-11T03:44:01.744 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.744 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.744 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:44:01.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.744 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3041 2024-09-11T03:44:01.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.745 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:44:01.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:44:01.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897046+0000", 2024-09-11T03:44:01.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7e-07 2024-09-11T03:44:01.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897048+0000", 2024-09-11T03:44:01.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0700000000000001e-06 2024-09-11T03:44:01.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897058+0000", 2024-09-11T03:44:01.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0151e-05 2024-09-11T03:44:01.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897068+0000", 2024-09-11T03:44:01.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0329000000000001e-05 2024-09-11T03:44:01.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897128+0000", 2024-09-11T03:44:01.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0208999999999999e-05 2024-09-11T03:44:01.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:01.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897132+0000", 2024-09-11T03:44:01.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7290000000000002e-06 2024-09-11T03:44:01.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917070+0000", 2024-09-11T03:44:01.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019937951999999998 2024-09-11T03:44:01.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917123+0000", 2024-09-11T03:44:01.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3149000000000002e-05 2024-09-11T03:44:01.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917208+0000", 2024-09-11T03:44:01.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5067000000000004e-05 2024-09-11T03:44:01.749 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.749 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.749 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.749 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3042 2.d 2:b6a25a79:::smithi17557870-47:head [sparse-read 0~1480887] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:44:01.749 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:44:01.749 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.743467718000005, 2024-09-11T03:44:01.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020302042999999999, 2024-09-11T03:44:01.750 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.750 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.750 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:44:01.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.750 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3042 2024-09-11T03:44:01.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.750 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:44:01.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:44:01.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897080+0000", 2024-09-11T03:44:01.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.82e-07 2024-09-11T03:44:01.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897082+0000", 2024-09-11T03:44:01.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.418e-06 2024-09-11T03:44:01.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897089+0000", 2024-09-11T03:44:01.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8090000000000004e-06 2024-09-11T03:44:01.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897094+0000", 2024-09-11T03:44:01.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2399999999999998e-06 2024-09-11T03:44:01.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:44:01.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0928999999999999e-05 2024-09-11T03:44:01.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:01.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897158+0000", 2024-09-11T03:44:01.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9890000000000001e-06 2024-09-11T03:44:01.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917227+0000", 2024-09-11T03:44:01.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020069019 2024-09-11T03:44:01.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917290+0000", 2024-09-11T03:44:01.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2848999999999994e-05 2024-09-11T03:44:01.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917381+0000", 2024-09-11T03:44:01.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1308000000000001e-05 2024-09-11T03:44:01.755 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.755 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.755 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.755 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3046 2.d 2:b6874adf:::smithi17557870-17:head [sparse-read 0~1769472,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:44:01.755 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:44:01.755 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.743445757000003, 2024-09-11T03:44:01.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021672665000000001, 2024-09-11T03:44:01.755 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.756 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.756 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:44:01.756 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.756 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.756 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.756 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3046 2024-09-11T03:44:01.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.756 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:44:01.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:44:01.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897102+0000", 2024-09-11T03:44:01.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.65e-07 2024-09-11T03:44:01.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897105+0000", 2024-09-11T03:44:01.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1820000000000002e-06 2024-09-11T03:44:01.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897111+0000", 2024-09-11T03:44:01.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4330000000000001e-06 2024-09-11T03:44:01.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897117+0000", 2024-09-11T03:44:01.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2979999999999999e-06 2024-09-11T03:44:01.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897179+0000", 2024-09-11T03:44:01.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2825999999999998e-05 2024-09-11T03:44:01.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:01.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897182+0000", 2024-09-11T03:44:01.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7769999999999998e-06 2024-09-11T03:44:01.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917399+0000", 2024-09-11T03:44:01.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020216622 2024-09-11T03:44:01.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917444+0000", 2024-09-11T03:44:01.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5558e-05 2024-09-11T03:44:01.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918774+0000", 2024-09-11T03:44:01.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013296040000000001 2024-09-11T03:44:01.760 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.760 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.760 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.761 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3047 2.d 2:b6874adf:::smithi17557870-17:head [read 0~2453888,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:44:01.761 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:44:01.761 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.743416585000006, 2024-09-11T03:44:01.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022691901, 2024-09-11T03:44:01.761 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.761 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.761 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:44:01.761 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.761 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.761 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.761 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3047 2024-09-11T03:44:01.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.762 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:44:01.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:44:01.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:44:01.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8599999999999998e-07 2024-09-11T03:44:01.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897134+0000", 2024-09-11T03:44:01.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1470000000000001e-06 2024-09-11T03:44:01.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897141+0000", 2024-09-11T03:44:01.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8029999999999999e-06 2024-09-11T03:44:01.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897147+0000", 2024-09-11T03:44:01.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4520000000000002e-06 2024-09-11T03:44:01.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897202+0000", 2024-09-11T03:44:01.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4255999999999997e-05 2024-09-11T03:44:01.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:01.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897204+0000", 2024-09-11T03:44:01.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6649999999999999e-06 2024-09-11T03:44:01.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918807+0000", 2024-09-11T03:44:01.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021602962999999999 2024-09-11T03:44:01.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918887+0000", 2024-09-11T03:44:01.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9550999999999998e-05 2024-09-11T03:44:01.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919822+0000", 2024-09-11T03:44:01.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093557799999999995 2024-09-11T03:44:01.766 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.766 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.766 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.766 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3048 2.d 2:b6874adf:::smithi17557870-17:head [read 0~2453888,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:44:01.766 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:44:01.766 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.743391760999998, 2024-09-11T03:44:01.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.023274353000000001, 2024-09-11T03:44:01.766 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.767 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.767 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:44:01.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.767 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3048 2024-09-11T03:44:01.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.767 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:44:01.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:44:01.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897156+0000", 2024-09-11T03:44:01.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1199999999999998e-07 2024-09-11T03:44:01.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897159+0000", 2024-09-11T03:44:01.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7439999999999999e-06 2024-09-11T03:44:01.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897165+0000", 2024-09-11T03:44:01.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6749999999999996e-06 2024-09-11T03:44:01.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897170+0000", 2024-09-11T03:44:01.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9420000000000003e-06 2024-09-11T03:44:01.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897226+0000", 2024-09-11T03:44:01.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6026999999999997e-05 2024-09-11T03:44:01.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:01.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897232+0000", 2024-09-11T03:44:01.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2609999999999999e-06 2024-09-11T03:44:01.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919841+0000", 2024-09-11T03:44:01.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022609054999999999 2024-09-11T03:44:01.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919885+0000", 2024-09-11T03:44:01.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3383999999999997e-05 2024-09-11T03:44:01.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.920430+0000", 2024-09-11T03:44:01.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054485299999999998 2024-09-11T03:44:01.771 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.771 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.772 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.772 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3565 2.2 2:43d79ac9:::smithi17557870-37:head [copy-from ver 362] snapc 92=[92,91,90,8f,8c,8b,8a] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-11T03:44:01.772 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:43:21.906736+0000", 2024-09-11T03:44:01.772 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 39.73381148, 2024-09-11T03:44:01.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014013827, 2024-09-11T03:44:01.772 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.772 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.772 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:01.772 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.772 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.772 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.772 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3565 2024-09-11T03:44:01.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.773 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:21.906736+0000", 2024-09-11T03:44:01.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:21.906736+0000", 2024-09-11T03:44:01.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:21.906737+0000", 2024-09-11T03:44:01.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.384e-06 2024-09-11T03:44:01.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:21.906750+0000", 2024-09-11T03:44:01.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3200000000000001e-05 2024-09-11T03:44:01.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:21.906776+0000", 2024-09-11T03:44:01.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5718999999999999e-05 2024-09-11T03:44:01.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:21.906785+0000", 2024-09-11T03:44:01.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6410000000000008e-06 2024-09-11T03:44:01.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:21.906883+0000", 2024-09-11T03:44:01.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8133000000000004e-05 2024-09-11T03:44:01.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:44:01.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:21.906890+0000", 2024-09-11T03:44:01.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3220000000000004e-06 2024-09-11T03:44:01.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:21.908660+0000", 2024-09-11T03:44:01.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017701220000000001 2024-09-11T03:44:01.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:21.908767+0000", 2024-09-11T03:44:01.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106885 2024-09-11T03:44:01.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,11", 2024-09-11T03:44:01.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:21.914190+0000", 2024-09-11T03:44:01.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054230930000000004 2024-09-11T03:44:01.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:44:01.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:21.918453+0000", 2024-09-11T03:44:01.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042631609999999997 2024-09-11T03:44:01.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:21.920230+0000", 2024-09-11T03:44:01.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001777172 2024-09-11T03:44:01.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:21.920667+0000", 2024-09-11T03:44:01.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00043681599999999997 2024-09-11T03:44:01.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:01.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:21.920708+0000", 2024-09-11T03:44:01.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0324000000000002e-05 2024-09-11T03:44:01.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:21.920749+0000", 2024-09-11T03:44:01.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-09-11T03:44:01.779 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.779 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.779 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.779 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3879 2.2 2:43d79ac9:::smithi17557870-37:head [copy-from ver 461] snapc a2=[a2,a1,9f,9e,9d,97,90,8a] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-11T03:44:01.779 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:43:52.918307+0000", 2024-09-11T03:44:01.779 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.7222398200000004, 2024-09-11T03:44:01.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014449813000000001, 2024-09-11T03:44:01.779 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:01.779 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:01.780 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:01.780 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:01.780 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:01.780 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:01.780 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3879 2024-09-11T03:44:01.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.780 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:01.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:01.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.918307+0000", 2024-09-11T03:44:01.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:01.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.918307+0000", 2024-09-11T03:44:01.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:01.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:01.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.918309+0000", 2024-09-11T03:44:01.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.61e-06 2024-09-11T03:44:01.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:01.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.918315+0000", 2024-09-11T03:44:01.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9909999999999999e-06 2024-09-11T03:44:01.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:01.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.918328+0000", 2024-09-11T03:44:01.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3429e-05 2024-09-11T03:44:01.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:01.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.918337+0000", 2024-09-11T03:44:01.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7309999999999999e-06 2024-09-11T03:44:01.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:01.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.918415+0000", 2024-09-11T03:44:01.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7922000000000004e-05 2024-09-11T03:44:01.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:01.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.918554+0000", 2024-09-11T03:44:01.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000138591 2024-09-11T03:44:01.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,11", 2024-09-11T03:44:01.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.924780+0000", 2024-09-11T03:44:01.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062259560000000004 2024-09-11T03:44:01.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:44:01.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.929082+0000", 2024-09-11T03:44:01.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043024020000000003 2024-09-11T03:44:01.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.930686+0000", 2024-09-11T03:44:01.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016045359999999999 2024-09-11T03:44:01.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:01.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.932683+0000", 2024-09-11T03:44:01.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019961810000000001 2024-09-11T03:44:01.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:01.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.932723+0000", 2024-09-11T03:44:01.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0046999999999998e-05 2024-09-11T03:44:01.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:01.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:01.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:01.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.932757+0000", 2024-09-11T03:44:01.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4416999999999999e-05 2024-09-11T03:44:01.786 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.786 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.786 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.786 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:01.786 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:01.786 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:44:01.787 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:01.808 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:01.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:01.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:01.910 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:01.912 INFO:tasks.rados.rados.2.smithi175.stdout:1550: done (2 left) 2024-09-11T03:44:01.912 INFO:tasks.rados.rados.2.smithi175.stdout:1551: done (1 left) 2024-09-11T03:44:01.912 INFO:tasks.rados.rados.2.smithi175.stdout:1552: done (0 left) 2024-09-11T03:44:01.912 INFO:tasks.rados.rados.2.smithi175.stdout:1553: snap_remove snap 162 2024-09-11T03:44:01.915 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:01.932 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:01.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:01.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:02.035 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:02.040 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:01 smithi084 ceph-mon[27254]: 2.17 scrub starts 2024-09-11T03:44:02.040 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:01 smithi084 ceph-mon[27254]: pgmap v789: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 168 MiB data, 771 MiB used, 983 GiB / 983 GiB avail; 56 MiB/s rd, 20 MiB/s wr, 110 op/s; 7.8 MiB/s, 23 objects/s recovering 2024-09-11T03:44:02.040 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:01 smithi084 ceph-mon[27254]: osdmap e423: 12 total, 12 up, 11 in 2024-09-11T03:44:02.040 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:01 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:02.040 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:01 smithi084 ceph-mon[27254]: osdmap e424: 12 total, 12 up, 11 in 2024-09-11T03:44:02.056 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:02.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:02.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:02.058 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: specified pgp_num 51 > pg_num 31 2024-09-11T03:44:02.158 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:02.180 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:02.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:02.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:01 smithi175 ceph-mon[30894]: 2.17 scrub starts 2024-09-11T03:44:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:01 smithi175 ceph-mon[30894]: pgmap v789: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 168 MiB data, 771 MiB used, 983 GiB / 983 GiB avail; 56 MiB/s rd, 20 MiB/s wr, 110 op/s; 7.8 MiB/s, 23 objects/s recovering 2024-09-11T03:44:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:01 smithi175 ceph-mon[30894]: osdmap e423: 12 total, 12 up, 11 in 2024-09-11T03:44:02.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:01 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:02.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:01 smithi175 ceph-mon[30894]: osdmap e424: 12 total, 12 up, 11 in 2024-09-11T03:44:02.281 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:02.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:02.304 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:02.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:02.405 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:02.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:01 smithi121 ceph-mon[30729]: 2.17 scrub starts 2024-09-11T03:44:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:01 smithi121 ceph-mon[30729]: pgmap v789: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 168 MiB data, 771 MiB used, 983 GiB / 983 GiB avail; 56 MiB/s rd, 20 MiB/s wr, 110 op/s; 7.8 MiB/s, 23 objects/s recovering 2024-09-11T03:44:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:01 smithi121 ceph-mon[30729]: osdmap e423: 12 total, 12 up, 11 in 2024-09-11T03:44:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:01 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:01 smithi121 ceph-mon[30729]: osdmap e424: 12 total, 12 up, 11 in 2024-09-11T03:44:02.426 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:02.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:02.525 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T03:44:02.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:02.547 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:02.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:02.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:02.649 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:02.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:02.671 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:02.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:02.766 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:44:02.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:02.774 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:44:02.794 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:02.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:02.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:02.896 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:02.917 INFO:tasks.rados.rados.2.smithi175.stdout:1553: done (0 left) 2024-09-11T03:44:02.917 INFO:tasks.rados.rados.2.smithi175.stdout:1554: rollback oid 48 current snap is 167 2024-09-11T03:44:02.917 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 48 to 157 2024-09-11T03:44:02.917 INFO:tasks.rados.rados.2.smithi175.stdout:1555: snap_create 2024-09-11T03:44:02.920 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:02.920 INFO:tasks.rados.rados.2.smithi175.stdout:1554: finishing rollback tid 1 to smithi17557870-48 2024-09-11T03:44:02.920 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 468 (ObjNum 881 snap 0 seq_num 1745306800) dirty dne 2024-09-11T03:44:02.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:02.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:03.022 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:03.043 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:03.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:03.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:03.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:02 smithi084 ceph-mon[27254]: 2.17 scrub starts 2024-09-11T03:44:03.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:02 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2564983812' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "51"} : dispatch 2024-09-11T03:44:03.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:02 smithi084 ceph-mon[27254]: osdmap e425: 12 total, 12 up, 11 in 2024-09-11T03:44:03.144 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:03.171 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:03.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:03.172 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:03.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:02 smithi121 ceph-mon[30729]: 2.17 scrub starts 2024-09-11T03:44:03.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:02 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2564983812' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "51"} : dispatch 2024-09-11T03:44:03.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:02 smithi121 ceph-mon[30729]: osdmap e425: 12 total, 12 up, 11 in 2024-09-11T03:44:03.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:02 smithi175 ceph-mon[30894]: 2.17 scrub starts 2024-09-11T03:44:03.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:02 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2564983812' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "51"} : dispatch 2024-09-11T03:44:03.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:02 smithi175 ceph-mon[30894]: osdmap e425: 12 total, 12 up, 11 in 2024-09-11T03:44:03.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:03.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:03.295 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:03.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:03.318 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:44:03.318 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:44:03.318 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:44:03.318 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:44:03.395 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:03.421 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:03.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:03.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:03.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:44:03.523 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:03.549 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:03.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:03.550 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:03.650 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:03.676 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:03.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:03.677 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:03.696 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:03.778 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:03.800 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:03.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:03.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:03.902 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:03.911 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:44:03.918 INFO:tasks.rados.rados.2.smithi175.stdout:1554: done (1 left) 2024-09-11T03:44:03.918 INFO:tasks.rados.rados.2.smithi175.stdout:1555: done (0 left) 2024-09-11T03:44:03.918 INFO:tasks.rados.rados.2.smithi175.stdout:1556: read oid 43 snap -1 2024-09-11T03:44:03.918 INFO:tasks.rados.rados.2.smithi175.stdout:1556: expect (ObjNum 536 snap 158 seq_num 536) 2024-09-11T03:44:03.918 INFO:tasks.rados.rados.2.smithi175.stdout:1557: delete oid 15 current snap is 168 2024-09-11T03:44:03.921 INFO:tasks.rados.rados.2.smithi175.stdout:1557: done (1 left) 2024-09-11T03:44:03.921 INFO:tasks.rados.rados.2.smithi175.stdout:1558: snap_remove snap 142 2024-09-11T03:44:03.927 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:03.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:03.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T03:44:04.029 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:04.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:04.051 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:04.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:03 smithi084 ceph-mon[27254]: pgmap v792: 57 pgs: 57 active+clean; 176 MiB data, 765 MiB used, 983 GiB / 983 GiB avail; 3.8 MiB/s rd, 1.3 MiB/s wr, 6 op/s 2024-09-11T03:44:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:03 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2789305707' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:44:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:03 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2789305707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:44:04.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:03 smithi084 ceph-mon[27254]: osdmap e426: 12 total, 12 up, 11 in 2024-09-11T03:44:04.152 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:04.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:03 smithi121 ceph-mon[30729]: pgmap v792: 57 pgs: 57 active+clean; 176 MiB data, 765 MiB used, 983 GiB / 983 GiB avail; 3.8 MiB/s rd, 1.3 MiB/s wr, 6 op/s 2024-09-11T03:44:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:03 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2789305707' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:44:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:03 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2789305707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:44:04.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:03 smithi121 ceph-mon[30729]: osdmap e426: 12 total, 12 up, 11 in 2024-09-11T03:44:04.178 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:04.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:04.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:04.223 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:44:04.280 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:04.308 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:04.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:04.309 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:04.345 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:44:04.345 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:44:04.358 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:44:04.359 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:44:04.374 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:44:04.374 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:44:04.391 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:44:04.391 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:44:04.407 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:44:04.407 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:44:04.410 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:04.424 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:44:04.424 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:44:04.431 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:04.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:04.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:04.440 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:44:04.441 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:44:04.457 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:44:04.458 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:44:04.474 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:44:04.475 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:44:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:03 smithi175 ceph-mon[30894]: pgmap v792: 57 pgs: 57 active+clean; 176 MiB data, 765 MiB used, 983 GiB / 983 GiB avail; 3.8 MiB/s rd, 1.3 MiB/s wr, 6 op/s 2024-09-11T03:44:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:03 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2789305707' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:44:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:03 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2789305707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:44:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:03 smithi175 ceph-mon[30894]: osdmap e426: 12 total, 12 up, 11 in 2024-09-11T03:44:04.492 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:44:04.492 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:44:04.508 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:44:04.509 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:44:04.525 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:44:04.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:44:04.533 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:04.554 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:04.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:04.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:04.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:04.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:04.679 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:04.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:04.683 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:44:04.683 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:44:04.781 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:04.802 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:04.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:04.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:04.904 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:04.920 INFO:tasks.rados.rados.2.smithi175.stdout:1558: done (1 left) 2024-09-11T03:44:04.920 INFO:tasks.rados.rados.2.smithi175.stdout:1559: rollback oid 1 current snap is 168 2024-09-11T03:44:04.920 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 1 to 163 2024-09-11T03:44:04.920 INFO:tasks.rados.rados.2.smithi175.stdout:1560: write (excl) oid 38 current snap is 168 2024-09-11T03:44:04.920 INFO:tasks.rados.rados.2.smithi175.stdout:1560: seq_num 548 ranges {410236=413868,1234250=729187,2715800=423063} 2024-09-11T03:44:04.924 INFO:tasks.rados.rados.2.smithi175.stdout:1560: writing smithi17557870-38 from 410236 to 824104 tid 1 2024-09-11T03:44:04.930 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:04.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:04.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:04.931 INFO:tasks.rados.rados.2.smithi175.stdout:1560: writing smithi17557870-38 from 1234250 to 1963437 tid 2 2024-09-11T03:44:04.938 INFO:tasks.rados.rados.2.smithi175.stdout:1560: writing smithi17557870-38 from 2715800 to 3138863 tid 3 2024-09-11T03:44:04.938 INFO:tasks.rados.rados.2.smithi175.stdout:1561: read oid 25 snap 155 2024-09-11T03:44:04.938 INFO:tasks.rados.rados.2.smithi175.stdout:1561: expect (ObjNum 496 snap 144 seq_num 496) 2024-09-11T03:44:04.938 INFO:tasks.rados.rados.2.smithi175.stdout:1562: snap_remove snap 165 2024-09-11T03:44:05.032 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:05.054 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:05.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:05.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:04 smithi084 ceph-mon[27254]: 2.17 scrub starts 2024-09-11T03:44:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:04 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3540279846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:04 smithi084 ceph-mon[27254]: 2.19 scrub starts 2024-09-11T03:44:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:04 smithi084 ceph-mon[27254]: 2.19 scrub ok 2024-09-11T03:44:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:04 smithi084 ceph-mon[27254]: osdmap e427: 12 total, 12 up, 11 in 2024-09-11T03:44:05.156 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:05.180 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:05.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:05.181 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:05.261 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:04 smithi121 ceph-mon[30729]: 2.17 scrub starts 2024-09-11T03:44:05.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:04 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3540279846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:05.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:04 smithi121 ceph-mon[30729]: 2.19 scrub starts 2024-09-11T03:44:05.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:04 smithi121 ceph-mon[30729]: 2.19 scrub ok 2024-09-11T03:44:05.262 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:04 smithi121 ceph-mon[30729]: osdmap e427: 12 total, 12 up, 11 in 2024-09-11T03:44:05.263 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T03:44:05.283 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:05.309 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:05.310 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:05.411 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:05.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:05.433 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:05.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:04 smithi175 ceph-mon[30894]: 2.17 scrub starts 2024-09-11T03:44:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:04 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3540279846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:04 smithi175 ceph-mon[30894]: 2.19 scrub starts 2024-09-11T03:44:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:04 smithi175 ceph-mon[30894]: 2.19 scrub ok 2024-09-11T03:44:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:04 smithi175 ceph-mon[30894]: osdmap e427: 12 total, 12 up, 11 in 2024-09-11T03:44:05.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:05.547 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:44:05.555 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:05.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:05.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:05.657 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:05.683 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:05.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:05.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:05.784 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:05.806 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:05.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:05.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:05.908 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:05.920 INFO:tasks.rados.rados.2.smithi175.stdout:1562: done (4 left) 2024-09-11T03:44:05.921 INFO:tasks.rados.rados.2.smithi175.stdout:1563: write (excl) oid 48 current snap is 168 2024-09-11T03:44:05.921 INFO:tasks.rados.rados.2.smithi175.stdout:1556: expect (ObjNum 536 snap 158 seq_num 536) 2024-09-11T03:44:05.931 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:05.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:05.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:06.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:06.041 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:44:06.041 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:44:06.054 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:06.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:06.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:06.124 INFO:tasks.rados.rados.2.smithi175.stdout:1559: finishing rollback tid 0 to smithi17557870-1 2024-09-11T03:44:06.124 INFO:tasks.rados.rados.2.smithi175.stdout:1559: finishing rollback tid 1 to smithi17557870-1 2024-09-11T03:44:06.125 INFO:tasks.rados.rados.2.smithi175.stdout:1559: finishing rollback tid 2 to smithi17557870-1 2024-09-11T03:44:06.125 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 557 (ObjNum 520 snap 154 seq_num 520) dirty exists 2024-09-11T03:44:06.125 INFO:tasks.rados.rados.2.smithi175.stdout:1560: finishing write tid 1 to smithi17557870-38 2024-09-11T03:44:06.125 INFO:tasks.rados.rados.2.smithi175.stdout:1560: finishing write tid 2 to smithi17557870-38 2024-09-11T03:44:06.125 INFO:tasks.rados.rados.2.smithi175.stdout:1560: finishing write tid 3 to smithi17557870-38 2024-09-11T03:44:06.125 INFO:tasks.rados.rados.2.smithi175.stdout:1560: finishing write tid 4 to smithi17557870-38 2024-09-11T03:44:06.125 INFO:tasks.rados.rados.2.smithi175.stdout:1560: finishing write tid 5 to smithi17557870-38 2024-09-11T03:44:06.125 INFO:tasks.rados.rados.2.smithi175.stdout:1560: oid 38 updating version 0 to 559 2024-09-11T03:44:06.125 INFO:tasks.rados.rados.2.smithi175.stdout:1560: oid 38 updating version 559 to 562 2024-09-11T03:44:06.125 INFO:tasks.rados.rados.2.smithi175.stdout:1560: oid 38 version 562 is already newer than 561 2024-09-11T03:44:06.125 INFO:tasks.rados.rados.2.smithi175.stdout:1560: oid 38 version 562 is already newer than 560 2024-09-11T03:44:06.125 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 562 (ObjNum 548 snap 168 seq_num 548) dirty exists 2024-09-11T03:44:06.125 INFO:tasks.rados.rados.2.smithi175.stdout:1560: left oid 38 (ObjNum 548 snap 168 seq_num 548) 2024-09-11T03:44:06.126 INFO:tasks.rados.rados.2.smithi175.stdout:1561: expect (ObjNum 496 snap 144 seq_num 496) 2024-09-11T03:44:06.156 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:06.178 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:06.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:06.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:06.226 INFO:tasks.rados.rados.2.smithi175.stdout:1563: seq_num 549 ranges {427043=489117,1576379=472845,2626870=600173} 2024-09-11T03:44:06.230 INFO:tasks.rados.rados.2.smithi175.stdout:1563: writing smithi17557870-48 from 427043 to 916160 tid 1 2024-09-11T03:44:06.237 INFO:tasks.rados.rados.2.smithi175.stdout:1563: writing smithi17557870-48 from 1576379 to 2049224 tid 2 2024-09-11T03:44:06.244 INFO:tasks.rados.rados.2.smithi175.stdout:1563: writing smithi17557870-48 from 2626870 to 3227043 tid 3 2024-09-11T03:44:06.244 INFO:tasks.rados.rados.2.smithi175.stdout:1556: done (4 left) 2024-09-11T03:44:06.244 INFO:tasks.rados.rados.2.smithi175.stdout:1559: done (3 left) 2024-09-11T03:44:06.244 INFO:tasks.rados.rados.2.smithi175.stdout:1560: done (2 left) 2024-09-11T03:44:06.244 INFO:tasks.rados.rados.2.smithi175.stdout:1561: done (1 left) 2024-09-11T03:44:06.244 INFO:tasks.rados.rados.2.smithi175.stdout:1564: snap_remove snap 163 2024-09-11T03:44:06.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:06.302 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:06.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:05 smithi084 ceph-mon[27254]: 2.17 scrub starts 2024-09-11T03:44:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:05 smithi084 ceph-mon[27254]: 2.17 scrub ok 2024-09-11T03:44:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:05 smithi084 ceph-mon[27254]: pgmap v795: 57 pgs: 57 active+clean; 177 MiB data, 773 MiB used, 983 GiB / 983 GiB avail; 3.8 MiB/s rd, 1.7 MiB/s wr, 7 op/s 2024-09-11T03:44:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:05 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:44:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:05 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:44:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:05 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:44:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:05 smithi084 ceph-mon[27254]: 2.15 scrub ok 2024-09-11T03:44:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:05 smithi084 ceph-mon[27254]: osdmap e428: 12 total, 12 up, 11 in 2024-09-11T03:44:06.404 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:06.410 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:05 smithi121 ceph-mon[30729]: 2.17 scrub starts 2024-09-11T03:44:06.411 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:05 smithi121 ceph-mon[30729]: 2.17 scrub ok 2024-09-11T03:44:06.411 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:05 smithi121 ceph-mon[30729]: pgmap v795: 57 pgs: 57 active+clean; 177 MiB data, 773 MiB used, 983 GiB / 983 GiB avail; 3.8 MiB/s rd, 1.7 MiB/s wr, 7 op/s 2024-09-11T03:44:06.411 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:05 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:44:06.411 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:05 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:44:06.411 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:05 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:44:06.411 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:05 smithi121 ceph-mon[30729]: 2.15 scrub ok 2024-09-11T03:44:06.411 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:05 smithi121 ceph-mon[30729]: osdmap e428: 12 total, 12 up, 11 in 2024-09-11T03:44:06.426 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:06.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:06.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:06.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:05 smithi175 ceph-mon[30894]: 2.17 scrub starts 2024-09-11T03:44:06.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:05 smithi175 ceph-mon[30894]: 2.17 scrub ok 2024-09-11T03:44:06.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:05 smithi175 ceph-mon[30894]: pgmap v795: 57 pgs: 57 active+clean; 177 MiB data, 773 MiB used, 983 GiB / 983 GiB avail; 3.8 MiB/s rd, 1.7 MiB/s wr, 7 op/s 2024-09-11T03:44:06.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:05 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:44:06.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:05 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:44:06.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:05 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:44:06.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:05 smithi175 ceph-mon[30894]: 2.15 scrub ok 2024-09-11T03:44:06.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:05 smithi175 ceph-mon[30894]: osdmap e428: 12 total, 12 up, 11 in 2024-09-11T03:44:06.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:06.548 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:06.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:06.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:06.650 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:06.672 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:06.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:06.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:06.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T03:44:06.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:06.801 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:06.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:06.892 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:44:06.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:06.917 INFO:tasks.rados.rados.2.smithi175.stdout:1564: done (1 left) 2024-09-11T03:44:06.917 INFO:tasks.rados.rados.2.smithi175.stdout:1565: snap_remove snap 157 2024-09-11T03:44:06.926 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:06.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:06.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:06.984 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:44:07.028 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:07.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:07.052 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:07.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:07.151 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:07.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:07.175 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:07.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:07.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:07.251 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:06 smithi121 ceph-mon[30729]: 1.19 scrub starts 2024-09-11T03:44:07.252 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:06 smithi121 ceph-mon[30729]: 1.19 scrub ok 2024-09-11T03:44:07.252 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:06 smithi121 ceph-mon[30729]: 2.4 scrub starts 2024-09-11T03:44:07.252 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:06 smithi121 ceph-mon[30729]: 2.4 scrub ok 2024-09-11T03:44:07.252 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:06 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:07.252 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:06 smithi121 ceph-mon[30729]: osdmap e429: 12 total, 12 up, 11 in 2024-09-11T03:44:07.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:06 smithi084 ceph-mon[27254]: 1.19 scrub starts 2024-09-11T03:44:07.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:06 smithi084 ceph-mon[27254]: 1.19 scrub ok 2024-09-11T03:44:07.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:06 smithi084 ceph-mon[27254]: 2.4 scrub starts 2024-09-11T03:44:07.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:06 smithi084 ceph-mon[27254]: 2.4 scrub ok 2024-09-11T03:44:07.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:06 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:07.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:06 smithi084 ceph-mon[27254]: osdmap e429: 12 total, 12 up, 11 in 2024-09-11T03:44:07.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:07.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:07.298 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:07.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:07.399 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:07.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:07.427 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:07.427 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:07.455 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:44:07.455 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:44:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:06 smithi175 ceph-mon[30894]: 1.19 scrub starts 2024-09-11T03:44:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:06 smithi175 ceph-mon[30894]: 1.19 scrub ok 2024-09-11T03:44:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:06 smithi175 ceph-mon[30894]: 2.4 scrub starts 2024-09-11T03:44:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:06 smithi175 ceph-mon[30894]: 2.4 scrub ok 2024-09-11T03:44:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:06 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:06 smithi175 ceph-mon[30894]: osdmap e429: 12 total, 12 up, 11 in 2024-09-11T03:44:07.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:07.549 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:07.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:07.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:07.651 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:07.676 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:07.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:07.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:07.721 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:44:07.775 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:44:07.775 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:44:07.775 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T03:44:07.775 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.871683 2024-09-11T03:44:07.775 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 1 0.8716834513998815 2024-09-11T03:44:07.779 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:07.841 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:07.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:07.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:07.912 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:44:07.919 INFO:tasks.rados.rados.2.smithi175.stdout:1565: done (1 left) 2024-09-11T03:44:07.919 INFO:tasks.rados.rados.2.smithi175.stdout:1566: snap_create 2024-09-11T03:44:07.919 INFO:tasks.rados.rados.2.smithi175.stdout:1563: finishing write tid 1 to smithi17557870-48 2024-09-11T03:44:07.919 INFO:tasks.rados.rados.2.smithi175.stdout:1563: finishing write tid 2 to smithi17557870-48 2024-09-11T03:44:07.919 INFO:tasks.rados.rados.2.smithi175.stdout:1563: finishing write tid 3 to smithi17557870-48 2024-09-11T03:44:07.919 INFO:tasks.rados.rados.2.smithi175.stdout:1563: finishing write tid 4 to smithi17557870-48 2024-09-11T03:44:07.920 INFO:tasks.rados.rados.2.smithi175.stdout:1563: finishing write tid 5 to smithi17557870-48 2024-09-11T03:44:07.920 INFO:tasks.rados.rados.2.smithi175.stdout:1563: oid 48 updating version 0 to 471 2024-09-11T03:44:07.920 INFO:tasks.rados.rados.2.smithi175.stdout:1563: oid 48 updating version 471 to 473 2024-09-11T03:44:07.920 INFO:tasks.rados.rados.2.smithi175.stdout:1563: oid 48 version 473 is already newer than 470 2024-09-11T03:44:07.920 INFO:tasks.rados.rados.2.smithi175.stdout:1563: oid 48 version 473 is already newer than 472 2024-09-11T03:44:07.920 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 473 (ObjNum 549 snap 168 seq_num 549) dirty exists 2024-09-11T03:44:07.920 INFO:tasks.rados.rados.2.smithi175.stdout:1563: left oid 48 (ObjNum 549 snap 168 seq_num 549) 2024-09-11T03:44:07.942 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:07.969 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:07.969 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:08.051 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:07 smithi121 ceph-mon[30729]: 1.12 scrub starts 2024-09-11T03:44:08.051 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:07 smithi121 ceph-mon[30729]: 1.12 scrub ok 2024-09-11T03:44:08.051 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:07 smithi121 ceph-mon[30729]: pgmap v798: 57 pgs: 57 active+clean; 176 MiB data, 768 MiB used, 983 GiB / 983 GiB avail; 2.5 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2024-09-11T03:44:08.051 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:07 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:44:08.051 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:07 smithi121 ceph-mon[30729]: 2.9 scrub ok 2024-09-11T03:44:08.051 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1272893328' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:08.051 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:07 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:44:08.052 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:07 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:44:08.052 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1272893328' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:08.052 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:07 smithi121 ceph-mon[30729]: osdmap e430: 12 total, 12 up, 11 in 2024-09-11T03:44:08.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:08.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T03:44:08.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:08.093 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:08.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:08.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:08.215 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:08.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:08.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:08.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:07 smithi084 ceph-mon[27254]: 1.12 scrub starts 2024-09-11T03:44:08.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:07 smithi084 ceph-mon[27254]: 1.12 scrub ok 2024-09-11T03:44:08.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:07 smithi084 ceph-mon[27254]: pgmap v798: 57 pgs: 57 active+clean; 176 MiB data, 768 MiB used, 983 GiB / 983 GiB avail; 2.5 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2024-09-11T03:44:08.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:07 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:44:08.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:07 smithi084 ceph-mon[27254]: 2.9 scrub ok 2024-09-11T03:44:08.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1272893328' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:08.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:07 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:44:08.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:07 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:44:08.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1272893328' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:08.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:07 smithi084 ceph-mon[27254]: osdmap e430: 12 total, 12 up, 11 in 2024-09-11T03:44:08.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:08.336 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:44:08.340 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:08.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:08.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:08.443 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:08.465 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:08.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:08.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:07 smithi175 ceph-mon[30894]: 1.12 scrub starts 2024-09-11T03:44:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:07 smithi175 ceph-mon[30894]: 1.12 scrub ok 2024-09-11T03:44:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:07 smithi175 ceph-mon[30894]: pgmap v798: 57 pgs: 57 active+clean; 176 MiB data, 768 MiB used, 983 GiB / 983 GiB avail; 2.5 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2024-09-11T03:44:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:07 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:44:08.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:07 smithi175 ceph-mon[30894]: 2.9 scrub ok 2024-09-11T03:44:08.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1272893328' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:08.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:07 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:44:08.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:07 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:44:08.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1272893328' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:08.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:07 smithi175 ceph-mon[30894]: osdmap e430: 12 total, 12 up, 11 in 2024-09-11T03:44:08.567 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:08.589 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:08.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:08.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:08.712 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:08.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:08.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:08.742 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:08.747 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:08.788 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:44:08.789 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:44:08.814 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:08.840 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:08.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:08.841 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:08.920 INFO:tasks.rados.rados.2.smithi175.stdout:1563: done (1 left) 2024-09-11T03:44:08.920 INFO:tasks.rados.rados.2.smithi175.stdout:1566: done (0 left) 2024-09-11T03:44:08.920 INFO:tasks.rados.rados.2.smithi175.stdout:1567: rollback oid 24 current snap is 169 2024-09-11T03:44:08.920 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 161 2024-09-11T03:44:08.920 INFO:tasks.rados.rados.2.smithi175.stdout:1568: read oid 42 snap 164 2024-09-11T03:44:08.921 INFO:tasks.rados.rados.2.smithi175.stdout:1568: expect deleted 2024-09-11T03:44:08.921 INFO:tasks.rados.rados.2.smithi175.stdout:1569: snap_create 2024-09-11T03:44:08.924 INFO:tasks.rados.rados.2.smithi175.stdout:1567: finishing rollback tid 0 to smithi17557870-24 2024-09-11T03:44:08.924 INFO:tasks.rados.rados.2.smithi175.stdout:1567: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:44:08.924 INFO:tasks.rados.rados.2.smithi175.stdout:1567: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:44:08.925 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 604 (ObjNum 505 snap 146 seq_num 505) dirty exists 2024-09-11T03:44:08.942 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:08.967 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:08.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:08.968 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:09.068 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:09.092 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:09.093 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:09.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:09 smithi121 ceph-mon[30729]: 1.3 scrub starts 2024-09-11T03:44:09.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:09 smithi121 ceph-mon[30729]: 1.3 scrub ok 2024-09-11T03:44:09.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:09 smithi121 ceph-mon[30729]: 2.3 scrub starts 2024-09-11T03:44:09.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:09 smithi121 ceph-mon[30729]: 2.3 scrub ok 2024-09-11T03:44:09.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:09 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:44:09.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:09 smithi121 ceph-mon[30729]: 2.e scrub ok 2024-09-11T03:44:09.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:09 smithi121 ceph-mon[30729]: osdmap e431: 12 total, 12 up, 11 in 2024-09-11T03:44:09.194 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:09.218 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:09.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:09.219 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:09.255 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:09 smithi084 ceph-mon[27254]: 1.3 scrub starts 2024-09-11T03:44:09.255 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:09 smithi084 ceph-mon[27254]: 1.3 scrub ok 2024-09-11T03:44:09.255 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:09 smithi084 ceph-mon[27254]: 2.3 scrub starts 2024-09-11T03:44:09.255 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:09 smithi084 ceph-mon[27254]: 2.3 scrub ok 2024-09-11T03:44:09.255 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:09 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:44:09.255 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:09 smithi084 ceph-mon[27254]: 2.e scrub ok 2024-09-11T03:44:09.255 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:09 smithi084 ceph-mon[27254]: osdmap e431: 12 total, 12 up, 11 in 2024-09-11T03:44:09.320 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:09.342 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:09.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:09.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:09.378 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:09 smithi175 ceph-mon[30894]: 1.3 scrub starts 2024-09-11T03:44:09.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:09 smithi175 ceph-mon[30894]: 1.3 scrub ok 2024-09-11T03:44:09.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:09 smithi175 ceph-mon[30894]: 2.3 scrub starts 2024-09-11T03:44:09.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:09 smithi175 ceph-mon[30894]: 2.3 scrub ok 2024-09-11T03:44:09.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:09 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:44:09.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:09 smithi175 ceph-mon[30894]: 2.e scrub ok 2024-09-11T03:44:09.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:09 smithi175 ceph-mon[30894]: osdmap e431: 12 total, 12 up, 11 in 2024-09-11T03:44:09.437 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:44:09.437 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:44:09.444 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:09.455 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:44:09.455 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:44:09.467 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:09.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:09.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:09.474 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:44:09.474 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:44:09.489 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:44:09.489 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:44:09.506 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:44:09.506 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:44:09.523 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:44:09.523 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:44:09.540 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:44:09.541 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:44:09.557 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:44:09.558 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:44:09.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:09.574 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:44:09.574 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:44:09.591 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:44:09.591 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:44:09.592 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:09.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:09.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:09.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T03:44:09.609 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:44:09.609 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:44:09.629 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:44:09.630 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:44:09.693 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:09.715 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:09.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:09.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:09.817 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:09.840 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:09.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:09.841 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:09.898 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:44:09.915 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.1 primary-affinity to 0.871683 (8571262) 2024-09-11T03:44:09.920 INFO:tasks.rados.rados.2.smithi175.stdout:1567: done (2 left) 2024-09-11T03:44:09.920 INFO:tasks.rados.rados.2.smithi175.stdout:1568: done (1 left) 2024-09-11T03:44:09.920 INFO:tasks.rados.rados.2.smithi175.stdout:1569: done (0 left) 2024-09-11T03:44:09.920 INFO:tasks.rados.rados.2.smithi175.stdout:1570: delete oid 32 current snap is 170 2024-09-11T03:44:09.923 INFO:tasks.rados.rados.2.smithi175.stdout:1570: done (0 left) 2024-09-11T03:44:09.923 INFO:tasks.rados.rados.2.smithi175.stdout:1571: snap_create 2024-09-11T03:44:09.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:09.963 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:09.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:09.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:10.065 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:10.086 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:10.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:10.087 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:10.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:10.209 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:10.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:10.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:10.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:10 smithi084 ceph-mon[27254]: pgmap v801: 57 pgs: 57 active+clean; 156 MiB data, 747 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:44:10.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:10 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:44:10.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:10 smithi084 ceph-mon[27254]: 2.1 scrub ok 2024-09-11T03:44:10.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:10 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3730086312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:10.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:10 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/518801985' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.8716834513998815} : dispatch 2024-09-11T03:44:10.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:10 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/518801985' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.8716834513998815}]': finished 2024-09-11T03:44:10.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:10 smithi084 ceph-mon[27254]: osdmap e432: 12 total, 12 up, 11 in 2024-09-11T03:44:10.303 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:10 smithi121 ceph-mon[30729]: pgmap v801: 57 pgs: 57 active+clean; 156 MiB data, 747 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:44:10.304 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:10 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:44:10.304 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:10 smithi121 ceph-mon[30729]: 2.1 scrub ok 2024-09-11T03:44:10.304 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3730086312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:10.304 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/518801985' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.8716834513998815} : dispatch 2024-09-11T03:44:10.304 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/518801985' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.8716834513998815}]': finished 2024-09-11T03:44:10.304 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:10 smithi121 ceph-mon[30729]: osdmap e432: 12 total, 12 up, 11 in 2024-09-11T03:44:10.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:10.334 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:10.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:10.412 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:44:10.412 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:44:10.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:44:10.413 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:44:10.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.413 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:228 2.9 e105/95 2:95fa802a:::smithi17557870-22:head v 105'20, mlcod=100'17)", 2024-09-11T03:44:10.413 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:44:10.413 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 328.43860059299999, 2024-09-11T03:44:10.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01695953, 2024-09-11T03:44:10.413 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.413 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.413 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:10.413 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:44:10.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:44:10.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:44:10.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2600000000000002e-07 2024-09-11T03:44:10.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982417+0000", 2024-09-11T03:44:10.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01547431 2024-09-11T03:44:10.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982440+0000", 2024-09-11T03:44:10.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2506000000000001e-05 2024-09-11T03:44:10.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982448+0000", 2024-09-11T03:44:10.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2989999999999997e-06 2024-09-11T03:44:10.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982562+0000", 2024-09-11T03:44:10.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114084 2024-09-11T03:44:10.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982583+0000", 2024-09-11T03:44:10.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1160999999999999e-05 2024-09-11T03:44:10.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:10.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.983852+0000", 2024-09-11T03:44:10.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001268953 2024-09-11T03:44:10.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.983902+0000", 2024-09-11T03:44:10.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9691000000000002e-05 2024-09-11T03:44:10.418 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.418 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.418 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.418 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:345 2.0 2:0762776a:::smithi17557870-25:head [write 742876~447756] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:44:10.418 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.827240+0000", 2024-09-11T03:44:10.418 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 320.57830327800002, 2024-09-11T03:44:10.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076328890000000003, 2024-09-11T03:44:10.419 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.419 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.419 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:10.419 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:10.419 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:10.419 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:10.419 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 345 2024-09-11T03:44:10.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.419 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.827240+0000", 2024-09-11T03:44:10.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.827240+0000", 2024-09-11T03:44:10.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.827243+0000", 2024-09-11T03:44:10.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0110000000000001e-06 2024-09-11T03:44:10.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828270+0000", 2024-09-11T03:44:10.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001026838 2024-09-11T03:44:10.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828298+0000", 2024-09-11T03:44:10.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8561999999999999e-05 2024-09-11T03:44:10.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828312+0000", 2024-09-11T03:44:10.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3781000000000001e-05 2024-09-11T03:44:10.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828398+0000", 2024-09-11T03:44:10.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6336999999999997e-05 2024-09-11T03:44:10.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:44:10.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828405+0000", 2024-09-11T03:44:10.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9079999999999998e-06 2024-09-11T03:44:10.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830015+0000", 2024-09-11T03:44:10.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001609344 2024-09-11T03:44:10.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830095+0000", 2024-09-11T03:44:10.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0259000000000005e-05 2024-09-11T03:44:10.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10", 2024-09-11T03:44:10.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830248+0000", 2024-09-11T03:44:10.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015351500000000001 2024-09-11T03:44:10.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:44:10.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.831760+0000", 2024-09-11T03:44:10.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015116540000000001 2024-09-11T03:44:10.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:10.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.832800+0000", 2024-09-11T03:44:10.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010400800000000001 2024-09-11T03:44:10.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:10.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834760+0000", 2024-09-11T03:44:10.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019600809999999998 2024-09-11T03:44:10.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:10.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834804+0000", 2024-09-11T03:44:10.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3922999999999999e-05 2024-09-11T03:44:10.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834873+0000", 2024-09-11T03:44:10.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8596000000000005e-05 2024-09-11T03:44:10.426 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.426 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.426 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.426 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:346 2.0 2:0762776a:::smithi17557870-25:head [write 1729599~575687] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:44:10.426 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.833957+0000", 2024-09-11T03:44:10.426 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 320.57158655299997, 2024-09-11T03:44:10.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00864616, 2024-09-11T03:44:10.426 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.426 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.427 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:10.427 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:10.427 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:10.427 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:10.427 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 346 2024-09-11T03:44:10.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.427 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833957+0000", 2024-09-11T03:44:10.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833957+0000", 2024-09-11T03:44:10.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833958+0000", 2024-09-11T03:44:10.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.674e-06 2024-09-11T03:44:10.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838969+0000", 2024-09-11T03:44:10.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005010993 2024-09-11T03:44:10.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839016+0000", 2024-09-11T03:44:10.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6576000000000003e-05 2024-09-11T03:44:10.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839037+0000", 2024-09-11T03:44:10.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1356e-05 2024-09-11T03:44:10.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839197+0000", 2024-09-11T03:44:10.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000159338 2024-09-11T03:44:10.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839266+0000", 2024-09-11T03:44:10.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9758999999999994e-05 2024-09-11T03:44:10.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10", 2024-09-11T03:44:10.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839374+0000", 2024-09-11T03:44:10.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010791899999999999 2024-09-11T03:44:10.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:44:10.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.840763+0000", 2024-09-11T03:44:10.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013883999999999999 2024-09-11T03:44:10.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:10.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842254+0000", 2024-09-11T03:44:10.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001491409 2024-09-11T03:44:10.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:10.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842538+0000", 2024-09-11T03:44:10.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028420099999999998 2024-09-11T03:44:10.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:10.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842573+0000", 2024-09-11T03:44:10.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4381000000000002e-05 2024-09-11T03:44:10.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842603+0000", 2024-09-11T03:44:10.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0154e-05 2024-09-11T03:44:10.433 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.433 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.433 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.434 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:623 2.8 2:162357fa:::smithi17557870-18:head [read 0~3563051,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e130)", 2024-09-11T03:44:10.434 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:07.066819+0000", 2024-09-11T03:44:10.434 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 303.33872404300001, 2024-09-11T03:44:10.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058140739999999998, 2024-09-11T03:44:10.434 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.434 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.434 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:44:10.434 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:10.434 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:10.434 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:10.434 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 623 2024-09-11T03:44:10.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.435 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066819+0000", 2024-09-11T03:44:10.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066819+0000", 2024-09-11T03:44:10.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066820+0000", 2024-09-11T03:44:10.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3900000000000005e-07 2024-09-11T03:44:10.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066823+0000", 2024-09-11T03:44:10.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4750000000000002e-06 2024-09-11T03:44:10.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066840+0000", 2024-09-11T03:44:10.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6603999999999999e-05 2024-09-11T03:44:10.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066846+0000", 2024-09-11T03:44:10.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5409999999999997e-06 2024-09-11T03:44:10.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.071239+0000", 2024-09-11T03:44:10.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043923039999999997 2024-09-11T03:44:10.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.071311+0000", 2024-09-11T03:44:10.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2408000000000002e-05 2024-09-11T03:44:10.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.072633+0000", 2024-09-11T03:44:10.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013222030000000001 2024-09-11T03:44:10.439 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.439 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.439 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.439 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:624 2.8 2:162357fa:::smithi17557870-18:head [sparse-read 0~1957854] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e130)", 2024-09-11T03:44:10.439 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:07.066857+0000", 2024-09-11T03:44:10.439 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 303.33868579300002, 2024-09-11T03:44:10.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059990820000000002, 2024-09-11T03:44:10.439 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.439 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.439 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:44:10.440 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:10.440 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:10.440 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:10.440 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 624 2024-09-11T03:44:10.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.440 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066857+0000", 2024-09-11T03:44:10.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066857+0000", 2024-09-11T03:44:10.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066858+0000", 2024-09-11T03:44:10.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1699999999999998e-07 2024-09-11T03:44:10.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066861+0000", 2024-09-11T03:44:10.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2210000000000002e-06 2024-09-11T03:44:10.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066870+0000", 2024-09-11T03:44:10.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8149999999999995e-06 2024-09-11T03:44:10.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.066883+0000", 2024-09-11T03:44:10.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3220999999999999e-05 2024-09-11T03:44:10.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.072671+0000", 2024-09-11T03:44:10.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0057879910000000001 2024-09-11T03:44:10.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.072735+0000", 2024-09-11T03:44:10.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3414e-05 2024-09-11T03:44:10.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:07.072856+0000", 2024-09-11T03:44:10.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000121803 2024-09-11T03:44:10.443 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.443 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.443 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.444 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:746 2.0 2:0762776a:::smithi17557870-25:head [sparse-read 0~1885763] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e137)", 2024-09-11T03:44:10.444 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:13.831294+0000", 2024-09-11T03:44:10.444 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 296.57424879799999, 2024-09-11T03:44:10.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0057346979999999999, 2024-09-11T03:44:10.444 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.444 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.444 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:44:10.444 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:10.444 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:10.444 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:10.444 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 746 2024-09-11T03:44:10.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.445 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831294+0000", 2024-09-11T03:44:10.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831294+0000", 2024-09-11T03:44:10.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831295+0000", 2024-09-11T03:44:10.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7899999999999999e-07 2024-09-11T03:44:10.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831297+0000", 2024-09-11T03:44:10.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2019999999999998e-06 2024-09-11T03:44:10.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831303+0000", 2024-09-11T03:44:10.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.139e-06 2024-09-11T03:44:10.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831312+0000", 2024-09-11T03:44:10.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1379999999999996e-06 2024-09-11T03:44:10.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831406+0000", 2024-09-11T03:44:10.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4178999999999994e-05 2024-09-11T03:44:10.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:10.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.831411+0000", 2024-09-11T03:44:10.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7890000000000002e-06 2024-09-11T03:44:10.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.836793+0000", 2024-09-11T03:44:10.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053814850000000001 2024-09-11T03:44:10.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.836892+0000", 2024-09-11T03:44:10.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8987999999999995e-05 2024-09-11T03:44:10.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:13.837029+0000", 2024-09-11T03:44:10.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013729899999999999 2024-09-11T03:44:10.449 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.449 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.449 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.450 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:985 2.b e163/95 2:dde92fc6:::smithi17557870-41:head v 163'52, mlcod=158'48)", 2024-09-11T03:44:10.450 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:39.932630+0000", 2024-09-11T03:44:10.450 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 270.472913586, 2024-09-11T03:44:10.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056057540000000001, 2024-09-11T03:44:10.450 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.450 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.450 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:10.450 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.932630+0000", 2024-09-11T03:44:10.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.932630+0000", 2024-09-11T03:44:10.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.932630+0000", 2024-09-11T03:44:10.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7000000000000001e-07 2024-09-11T03:44:10.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.935963+0000", 2024-09-11T03:44:10.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033333339999999999 2024-09-11T03:44:10.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.935973+0000", 2024-09-11T03:44:10.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9690000000000006e-06 2024-09-11T03:44:10.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.935980+0000", 2024-09-11T03:44:10.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.596e-06 2024-09-11T03:44:10.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.936026+0000", 2024-09-11T03:44:10.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5850000000000003e-05 2024-09-11T03:44:10.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.936038+0000", 2024-09-11T03:44:10.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2408999999999999e-05 2024-09-11T03:44:10.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:10.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.938200+0000", 2024-09-11T03:44:10.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021619260000000002 2024-09-11T03:44:10.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:39.938235+0000", 2024-09-11T03:44:10.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.54e-05 2024-09-11T03:44:10.454 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.454 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.454 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.454 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1048 2.9 e175/95 2:98bb65f5:::smithi17557870-6:head v 175'134, mlcod=168'130)", 2024-09-11T03:44:10.454 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.832944+0000", 2024-09-11T03:44:10.454 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 258.572599599, 2024-09-11T03:44:10.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058696390000000003, 2024-09-11T03:44:10.454 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.455 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.455 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:10.455 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.832944+0000", 2024-09-11T03:44:10.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.832944+0000", 2024-09-11T03:44:10.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.832944+0000", 2024-09-11T03:44:10.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.72e-07 2024-09-11T03:44:10.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.836610+0000", 2024-09-11T03:44:10.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036663709999999999 2024-09-11T03:44:10.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.836624+0000", 2024-09-11T03:44:10.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3419e-05 2024-09-11T03:44:10.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.836634+0000", 2024-09-11T03:44:10.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0193e-05 2024-09-11T03:44:10.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.836684+0000", 2024-09-11T03:44:10.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0188000000000001e-05 2024-09-11T03:44:10.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.836702+0000", 2024-09-11T03:44:10.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8567999999999999e-05 2024-09-11T03:44:10.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:10.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.838767+0000", 2024-09-11T03:44:10.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002064986 2024-09-11T03:44:10.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.838813+0000", 2024-09-11T03:44:10.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5741999999999997e-05 2024-09-11T03:44:10.458 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.459 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.459 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.459 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1152 2.b e183/95 2:de6b1316:::smithi17557870-32:head v 183'96, mlcod=182'92)", 2024-09-11T03:44:10.459 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:59.878348+0000", 2024-09-11T03:44:10.459 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 250.52719526600001, 2024-09-11T03:44:10.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064294529999999999, 2024-09-11T03:44:10.459 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.459 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.459 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:10.459 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.878348+0000", 2024-09-11T03:44:10.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.460 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:10.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.878348+0000", 2024-09-11T03:44:10.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.878348+0000", 2024-09-11T03:44:10.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8900000000000001e-07 2024-09-11T03:44:10.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.882607+0000", 2024-09-11T03:44:10.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042589139999999999 2024-09-11T03:44:10.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.882621+0000", 2024-09-11T03:44:10.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3925e-05 2024-09-11T03:44:10.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.882628+0000", 2024-09-11T03:44:10.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6000000000000003e-06 2024-09-11T03:44:10.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.882686+0000", 2024-09-11T03:44:10.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8579999999999998e-05 2024-09-11T03:44:10.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.882703+0000", 2024-09-11T03:44:10.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7167999999999999e-05 2024-09-11T03:44:10.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:10.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.884729+0000", 2024-09-11T03:44:10.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020253720000000001 2024-09-11T03:44:10.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.884777+0000", 2024-09-11T03:44:10.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8705000000000002e-05 2024-09-11T03:44:10.466 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.466 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.466 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.466 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1161 2.8 2:1bdf1a36:::smithi17557870-36:head [delete] snapc 2e=[2e,2c,2b,28,26] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:44:10.466 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:01.831716+0000", 2024-09-11T03:44:10.466 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 248.57382687699999, 2024-09-11T03:44:10.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065461870000000002, 2024-09-11T03:44:10.466 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.466 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.466 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:10.466 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:10.466 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:10.467 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:10.467 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1161 2024-09-11T03:44:10.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.467 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831716+0000", 2024-09-11T03:44:10.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831716+0000", 2024-09-11T03:44:10.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831718+0000", 2024-09-11T03:44:10.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1890000000000001e-06 2024-09-11T03:44:10.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831726+0000", 2024-09-11T03:44:10.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5190000000000003e-06 2024-09-11T03:44:10.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831741+0000", 2024-09-11T03:44:10.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5407e-05 2024-09-11T03:44:10.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831749+0000", 2024-09-11T03:44:10.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0129999999999993e-06 2024-09-11T03:44:10.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831837+0000", 2024-09-11T03:44:10.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7558000000000004e-05 2024-09-11T03:44:10.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:44:10.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831845+0000", 2024-09-11T03:44:10.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3799999999999994e-06 2024-09-11T03:44:10.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.834626+0000", 2024-09-11T03:44:10.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027804140000000001 2024-09-11T03:44:10.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.834743+0000", 2024-09-11T03:44:10.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117569 2024-09-11T03:44:10.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,9", 2024-09-11T03:44:10.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.834902+0000", 2024-09-11T03:44:10.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015838700000000001 2024-09-11T03:44:10.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:44:10.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.835777+0000", 2024-09-11T03:44:10.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087511700000000002 2024-09-11T03:44:10.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:10.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.836462+0000", 2024-09-11T03:44:10.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000684686 2024-09-11T03:44:10.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:10.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.838188+0000", 2024-09-11T03:44:10.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001726193 2024-09-11T03:44:10.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:10.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.838229+0000", 2024-09-11T03:44:10.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0991000000000001e-05 2024-09-11T03:44:10.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.838262+0000", 2024-09-11T03:44:10.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3763999999999997e-05 2024-09-11T03:44:10.473 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.473 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.473 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.473 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1555 2.3 e214/95 2:c6c49f2c:::smithi17557870-44:head v 214'74, mlcod=198'70)", 2024-09-11T03:44:10.473 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:30.893545+0000", 2024-09-11T03:44:10.473 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 219.511997675, 2024-09-11T03:44:10.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070433960000000004, 2024-09-11T03:44:10.474 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.474 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.474 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:10.474 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.893545+0000", 2024-09-11T03:44:10.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.893545+0000", 2024-09-11T03:44:10.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.893546+0000", 2024-09-11T03:44:10.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7500000000000001e-07 2024-09-11T03:44:10.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895656+0000", 2024-09-11T03:44:10.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002110476 2024-09-11T03:44:10.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895679+0000", 2024-09-11T03:44:10.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2795000000000001e-05 2024-09-11T03:44:10.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895692+0000", 2024-09-11T03:44:10.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3141e-05 2024-09-11T03:44:10.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895758+0000", 2024-09-11T03:44:10.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5871999999999995e-05 2024-09-11T03:44:10.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895786+0000", 2024-09-11T03:44:10.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8391e-05 2024-09-11T03:44:10.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:10.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900550+0000", 2024-09-11T03:44:10.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047632869999999997 2024-09-11T03:44:10.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900589+0000", 2024-09-11T03:44:10.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9159000000000002e-05 2024-09-11T03:44:10.478 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.478 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.478 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.478 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1609 2.8 2:1bdf1a36:::smithi17557870-36:head [write 618557~727148] snapc 40=[40,3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:44:10.478 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:35.905306+0000", 2024-09-11T03:44:10.478 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 214.50023760600001, 2024-09-11T03:44:10.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069719600000000001, 2024-09-11T03:44:10.478 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.478 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.478 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:10.479 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:10.479 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:10.479 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:10.479 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1609 2024-09-11T03:44:10.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.479 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.905306+0000", 2024-09-11T03:44:10.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.905306+0000", 2024-09-11T03:44:10.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.905307+0000", 2024-09-11T03:44:10.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.482e-06 2024-09-11T03:44:10.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907145+0000", 2024-09-11T03:44:10.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001838412 2024-09-11T03:44:10.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907157+0000", 2024-09-11T03:44:10.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-09-11T03:44:10.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907165+0000", 2024-09-11T03:44:10.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6340000000000001e-06 2024-09-11T03:44:10.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907215+0000", 2024-09-11T03:44:10.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0760999999999997e-05 2024-09-11T03:44:10.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907282+0000", 2024-09-11T03:44:10.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6934000000000007e-05 2024-09-11T03:44:10.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,9", 2024-09-11T03:44:10.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907400+0000", 2024-09-11T03:44:10.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117625 2024-09-11T03:44:10.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:44:10.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.908820+0000", 2024-09-11T03:44:10.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0014204140000000001 2024-09-11T03:44:10.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:10.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.910145+0000", 2024-09-11T03:44:10.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013246250000000001 2024-09-11T03:44:10.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:10.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.912194+0000", 2024-09-11T03:44:10.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020485640000000001 2024-09-11T03:44:10.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:10.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.912229+0000", 2024-09-11T03:44:10.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5128e-05 2024-09-11T03:44:10.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.912277+0000", 2024-09-11T03:44:10.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8746999999999999e-05 2024-09-11T03:44:10.485 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.485 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.485 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.485 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1868 2.0 2:0762776a:::smithi17557870-25:head [copy-from ver 218] snapc 4c=[4c,4b,49,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:44:10.485 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:01.850162+0000", 2024-09-11T03:44:10.485 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 188.55538127700001, 2024-09-11T03:44:10.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020677853999999999, 2024-09-11T03:44:10.486 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.486 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.486 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:10.486 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:10.486 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:10.486 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:10.487 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1868 2024-09-11T03:44:10.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.487 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850162+0000", 2024-09-11T03:44:10.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850162+0000", 2024-09-11T03:44:10.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850164+0000", 2024-09-11T03:44:10.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.588e-06 2024-09-11T03:44:10.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850174+0000", 2024-09-11T03:44:10.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8300000000000008e-06 2024-09-11T03:44:10.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850193+0000", 2024-09-11T03:44:10.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8346e-05 2024-09-11T03:44:10.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850204+0000", 2024-09-11T03:44:10.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1144e-05 2024-09-11T03:44:10.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850285+0000", 2024-09-11T03:44:10.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1525000000000003e-05 2024-09-11T03:44:10.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:44:10.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850293+0000", 2024-09-11T03:44:10.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.072e-06 2024-09-11T03:44:10.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.852390+0000", 2024-09-11T03:44:10.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002096601 2024-09-11T03:44:10.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.852527+0000", 2024-09-11T03:44:10.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013703899999999999 2024-09-11T03:44:10.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,11", 2024-09-11T03:44:10.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.862911+0000", 2024-09-11T03:44:10.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01038392 2024-09-11T03:44:10.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:44:10.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.867006+0000", 2024-09-11T03:44:10.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004095291 2024-09-11T03:44:10.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:10.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.868758+0000", 2024-09-11T03:44:10.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017514869999999999 2024-09-11T03:44:10.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:10.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.870714+0000", 2024-09-11T03:44:10.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019563229999999998 2024-09-11T03:44:10.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:10.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.870780+0000", 2024-09-11T03:44:10.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6459999999999997e-05 2024-09-11T03:44:10.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.870840+0000", 2024-09-11T03:44:10.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9228000000000003e-05 2024-09-11T03:44:10.493 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.493 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.493 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.494 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1869 2.0 2:0762776a:::smithi17557870-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:44:10.494 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:01.850232+0000", 2024-09-11T03:44:10.494 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 188.555310693, 2024-09-11T03:44:10.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016419715000000001, 2024-09-11T03:44:10.494 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.494 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.494 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:44:10.494 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:10.494 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:10.494 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:10.494 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1869 2024-09-11T03:44:10.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.494 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850232+0000", 2024-09-11T03:44:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850232+0000", 2024-09-11T03:44:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850233+0000", 2024-09-11T03:44:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5300000000000004e-07 2024-09-11T03:44:10.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850237+0000", 2024-09-11T03:44:10.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3660000000000002e-06 2024-09-11T03:44:10.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850247+0000", 2024-09-11T03:44:10.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.38e-06 2024-09-11T03:44:10.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850254+0000", 2024-09-11T03:44:10.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2099999999999996e-06 2024-09-11T03:44:10.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850365+0000", 2024-09-11T03:44:10.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110499 2024-09-11T03:44:10.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:10.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850369+0000", 2024-09-11T03:44:10.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.454e-06 2024-09-11T03:44:10.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.852747+0000", 2024-09-11T03:44:10.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023778480000000001 2024-09-11T03:44:10.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:10.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.852806+0000", 2024-09-11T03:44:10.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8931999999999998e-05 2024-09-11T03:44:10.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.866380+0000", 2024-09-11T03:44:10.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013574693000000001 2024-09-11T03:44:10.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.866512+0000", 2024-09-11T03:44:10.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013140999999999999 2024-09-11T03:44:10.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.866652+0000", 2024-09-11T03:44:10.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014027000000000001 2024-09-11T03:44:10.500 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.500 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.500 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.500 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1877 2.b e247/95 2:de6b1316:::smithi17557870-32:head v 247'232, mlcod=243'228)", 2024-09-11T03:44:10.500 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:03.916185+0000", 2024-09-11T03:44:10.500 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 186.48935816299999, 2024-09-11T03:44:10.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010307996, 2024-09-11T03:44:10.500 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.500 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.500 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:10.501 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.916185+0000", 2024-09-11T03:44:10.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.916185+0000", 2024-09-11T03:44:10.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.916185+0000", 2024-09-11T03:44:10.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1699999999999998e-07 2024-09-11T03:44:10.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924225+0000", 2024-09-11T03:44:10.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080395750000000002 2024-09-11T03:44:10.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924243+0000", 2024-09-11T03:44:10.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7955e-05 2024-09-11T03:44:10.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924271+0000", 2024-09-11T03:44:10.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7797000000000002e-05 2024-09-11T03:44:10.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924349+0000", 2024-09-11T03:44:10.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8137000000000001e-05 2024-09-11T03:44:10.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924370+0000", 2024-09-11T03:44:10.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1226e-05 2024-09-11T03:44:10.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:10.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.926454+0000", 2024-09-11T03:44:10.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002083438 2024-09-11T03:44:10.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.926493+0000", 2024-09-11T03:44:10.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9351000000000003e-05 2024-09-11T03:44:10.504 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.504 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.504 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.505 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1989 2.0 2:0762776a:::smithi17557870-25:head [copy-from ver 162] snapc 52=[52,51,4e,3d] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-11T03:44:10.505 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:14.949188+0000", 2024-09-11T03:44:10.505 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 175.456355097, 2024-09-11T03:44:10.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014886257999999999, 2024-09-11T03:44:10.505 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.505 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.505 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:10.505 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:10.505 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:10.505 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:10.505 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1989 2024-09-11T03:44:10.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.506 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949188+0000", 2024-09-11T03:44:10.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949188+0000", 2024-09-11T03:44:10.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949191+0000", 2024-09-11T03:44:10.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.706e-06 2024-09-11T03:44:10.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949204+0000", 2024-09-11T03:44:10.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3142999999999999e-05 2024-09-11T03:44:10.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949227+0000", 2024-09-11T03:44:10.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3320999999999999e-05 2024-09-11T03:44:10.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949238+0000", 2024-09-11T03:44:10.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0352e-05 2024-09-11T03:44:10.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949313+0000", 2024-09-11T03:44:10.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5450000000000004e-05 2024-09-11T03:44:10.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:44:10.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949319+0000", 2024-09-11T03:44:10.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9279999999999998e-06 2024-09-11T03:44:10.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.950847+0000", 2024-09-11T03:44:10.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015282690000000001 2024-09-11T03:44:10.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.950916+0000", 2024-09-11T03:44:10.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8559000000000005e-05 2024-09-11T03:44:10.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,11", 2024-09-11T03:44:10.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.956873+0000", 2024-09-11T03:44:10.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059576359999999997 2024-09-11T03:44:10.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:44:10.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.960470+0000", 2024-09-11T03:44:10.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035970749999999999 2024-09-11T03:44:10.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:10.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.963875+0000", 2024-09-11T03:44:10.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034049050000000002 2024-09-11T03:44:10.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:10.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.963958+0000", 2024-09-11T03:44:10.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2769999999999995e-05 2024-09-11T03:44:10.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:10.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.964021+0000", 2024-09-11T03:44:10.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2442999999999998e-05 2024-09-11T03:44:10.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.964074+0000", 2024-09-11T03:44:10.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3701e-05 2024-09-11T03:44:10.512 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.512 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.512 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.512 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1990 2.0 2:0762776a:::smithi17557870-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e258)", 2024-09-11T03:44:10.512 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:14.949265+0000", 2024-09-11T03:44:10.512 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 175.456278056, 2024-09-11T03:44:10.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010821580000000001, 2024-09-11T03:44:10.513 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.513 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.513 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:44:10.513 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:10.513 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:10.513 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:10.513 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1990 2024-09-11T03:44:10.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.513 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949265+0000", 2024-09-11T03:44:10.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949265+0000", 2024-09-11T03:44:10.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949266+0000", 2024-09-11T03:44:10.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3000000000000001e-07 2024-09-11T03:44:10.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949268+0000", 2024-09-11T03:44:10.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2249999999999999e-06 2024-09-11T03:44:10.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949273+0000", 2024-09-11T03:44:10.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6389999999999997e-06 2024-09-11T03:44:10.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949277+0000", 2024-09-11T03:44:10.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4630000000000001e-06 2024-09-11T03:44:10.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949380+0000", 2024-09-11T03:44:10.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103099 2024-09-11T03:44:10.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:10.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949387+0000", 2024-09-11T03:44:10.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9290000000000001e-06 2024-09-11T03:44:10.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.951035+0000", 2024-09-11T03:44:10.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001647995 2024-09-11T03:44:10.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:10.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.951082+0000", 2024-09-11T03:44:10.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7231999999999998e-05 2024-09-11T03:44:10.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.959865+0000", 2024-09-11T03:44:10.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087832770000000008 2024-09-11T03:44:10.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.959967+0000", 2024-09-11T03:44:10.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010153 2024-09-11T03:44:10.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.960087+0000", 2024-09-11T03:44:10.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011976099999999999 2024-09-11T03:44:10.518 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.518 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.518 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.519 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:2434 2.9 e295/276 2:939cf94d:::smithi17557870-28:head v 295'353, mlcod=295'349)", 2024-09-11T03:44:10.519 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:52.055937+0000", 2024-09-11T03:44:10.519 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 138.34960588000001, 2024-09-11T03:44:10.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071747089999999996, 2024-09-11T03:44:10.519 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.519 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.519 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:10.519 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.055937+0000", 2024-09-11T03:44:10.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.055937+0000", 2024-09-11T03:44:10.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.055938+0000", 2024-09-11T03:44:10.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.043e-06 2024-09-11T03:44:10.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.059618+0000", 2024-09-11T03:44:10.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003679435 2024-09-11T03:44:10.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.059642+0000", 2024-09-11T03:44:10.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4485999999999999e-05 2024-09-11T03:44:10.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.059671+0000", 2024-09-11T03:44:10.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8560000000000001e-05 2024-09-11T03:44:10.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.059750+0000", 2024-09-11T03:44:10.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9170000000000003e-05 2024-09-11T03:44:10.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.059780+0000", 2024-09-11T03:44:10.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9558999999999999e-05 2024-09-11T03:44:10.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:10.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.063060+0000", 2024-09-11T03:44:10.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003280383 2024-09-11T03:44:10.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.063112+0000", 2024-09-11T03:44:10.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2073000000000001e-05 2024-09-11T03:44:10.523 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.523 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.523 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.523 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3749 2.8 2:162357fa:::smithi17557870-18:head [write 773254~791982] snapc 9b=[9b,9a,99,98,97,90,8a] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-11T03:44:10.523 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:43:39.942169+0000", 2024-09-11T03:44:10.523 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.463373739000001, 2024-09-11T03:44:10.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0082841789999999992, 2024-09-11T03:44:10.523 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.523 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.523 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:10.524 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:44:10.524 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:44:10.524 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:10.524 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3749 2024-09-11T03:44:10.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.524 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.942169+0000", 2024-09-11T03:44:10.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.942169+0000", 2024-09-11T03:44:10.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.942171+0000", 2024-09-11T03:44:10.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0470000000000001e-06 2024-09-11T03:44:10.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.943249+0000", 2024-09-11T03:44:10.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010774459999999999 2024-09-11T03:44:10.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.943269+0000", 2024-09-11T03:44:10.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0063000000000001e-05 2024-09-11T03:44:10.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.943279+0000", 2024-09-11T03:44:10.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0010000000000001e-05 2024-09-11T03:44:10.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.943372+0000", 2024-09-11T03:44:10.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3196999999999996e-05 2024-09-11T03:44:10.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:44:10.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.943377+0000", 2024-09-11T03:44:10.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6990000000000002e-06 2024-09-11T03:44:10.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.944823+0000", 2024-09-11T03:44:10.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001446084 2024-09-11T03:44:10.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.944931+0000", 2024-09-11T03:44:10.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107786 2024-09-11T03:44:10.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,9", 2024-09-11T03:44:10.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.945043+0000", 2024-09-11T03:44:10.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000112083 2024-09-11T03:44:10.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:44:10.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.946640+0000", 2024-09-11T03:44:10.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015976409999999999 2024-09-11T03:44:10.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:10.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.948221+0000", 2024-09-11T03:44:10.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015802559999999999 2024-09-11T03:44:10.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:10.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.950386+0000", 2024-09-11T03:44:10.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021654869999999998 2024-09-11T03:44:10.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:10.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.950416+0000", 2024-09-11T03:44:10.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0086e-05 2024-09-11T03:44:10.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.950454+0000", 2024-09-11T03:44:10.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7293999999999999e-05 2024-09-11T03:44:10.531 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.531 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.531 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.531 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:3778 2.9 e405/276 2:924204df:::smithi17557870-50:head v 405'547, mlcod=396'543)", 2024-09-11T03:44:10.531 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:43:42.960014+0000", 2024-09-11T03:44:10.531 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.445529583999999, 2024-09-11T03:44:10.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006334765, 2024-09-11T03:44:10.531 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:44:10.531 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:44:10.531 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:10.532 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:44:10.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:44:10.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.960014+0000", 2024-09-11T03:44:10.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:44:10.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.960014+0000", 2024-09-11T03:44:10.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:44:10.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:44:10.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.960014+0000", 2024-09-11T03:44:10.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8199999999999999e-07 2024-09-11T03:44:10.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:44:10.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.963105+0000", 2024-09-11T03:44:10.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030911129999999999 2024-09-11T03:44:10.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:44:10.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.963118+0000", 2024-09-11T03:44:10.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2773e-05 2024-09-11T03:44:10.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:44:10.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.963128+0000", 2024-09-11T03:44:10.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0321e-05 2024-09-11T03:44:10.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:44:10.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.963223+0000", 2024-09-11T03:44:10.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5358000000000004e-05 2024-09-11T03:44:10.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:44:10.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.963245+0000", 2024-09-11T03:44:10.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1308e-05 2024-09-11T03:44:10.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:44:10.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.966304+0000", 2024-09-11T03:44:10.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030595240000000001 2024-09-11T03:44:10.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:44:10.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:44:10.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:44:10.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.966348+0000", 2024-09-11T03:44:10.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4186e-05 2024-09-11T03:44:10.535 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.535 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.535 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.536 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:44:10.536 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:44:10.536 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:44:10.537 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:10 smithi175 ceph-mon[30894]: pgmap v801: 57 pgs: 57 active+clean; 156 MiB data, 747 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:44:10.537 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:10 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:44:10.537 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:10 smithi175 ceph-mon[30894]: 2.1 scrub ok 2024-09-11T03:44:10.537 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:10 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3730086312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:10.537 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:10 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/518801985' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.8716834513998815} : dispatch 2024-09-11T03:44:10.537 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:10 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/518801985' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.8716834513998815}]': finished 2024-09-11T03:44:10.537 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:10 smithi175 ceph-mon[30894]: osdmap e432: 12 total, 12 up, 11 in 2024-09-11T03:44:10.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:10.538 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:10.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:10.639 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:10.661 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:10.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:10.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:10.704 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:44:10.763 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:10.785 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:10.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:10.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:10.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:10.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:10.910 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:10.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:10.920 INFO:tasks.rados.rados.2.smithi175.stdout:1571: done (0 left) 2024-09-11T03:44:10.920 INFO:tasks.rados.rados.2.smithi175.stdout:1572: rollback oid 24 current snap is 171 2024-09-11T03:44:10.920 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 155 2024-09-11T03:44:10.920 INFO:tasks.rados.rados.2.smithi175.stdout:1573: write (excl) oid 26 current snap is 171 2024-09-11T03:44:10.920 INFO:tasks.rados.rados.2.smithi175.stdout:1573: seq_num 551 ranges {460657=639615,1693343=476514,2681716=578941} 2024-09-11T03:44:10.926 INFO:tasks.rados.rados.2.smithi175.stdout:1573: writing smithi17557870-26 from 460657 to 1100272 tid 1 2024-09-11T03:44:10.932 INFO:tasks.rados.rados.2.smithi175.stdout:1573: writing smithi17557870-26 from 1693343 to 2169857 tid 2 2024-09-11T03:44:10.938 INFO:tasks.rados.rados.2.smithi175.stdout:1573: writing smithi17557870-26 from 2681716 to 3260657 tid 3 2024-09-11T03:44:10.938 INFO:tasks.rados.rados.2.smithi175.stdout:1574: write (excl) oid 7 current snap is 171 2024-09-11T03:44:10.938 INFO:tasks.rados.rados.2.smithi175.stdout:1574: seq_num 552 ranges {477464=714864,1751825=736525,3225492=51972} 2024-09-11T03:44:10.944 INFO:tasks.rados.rados.2.smithi175.stdout:1574: writing smithi17557870-7 from 477464 to 1192328 tid 1 2024-09-11T03:44:10.951 INFO:tasks.rados.rados.2.smithi175.stdout:1574: writing smithi17557870-7 from 1751825 to 2488350 tid 2 2024-09-11T03:44:10.956 INFO:tasks.rados.rados.2.smithi175.stdout:1574: writing smithi17557870-7 from 3225492 to 3277464 tid 3 2024-09-11T03:44:10.956 INFO:tasks.rados.rados.2.smithi175.stdout:1575: snap_remove snap 168 2024-09-11T03:44:11.010 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:11.031 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:11.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:11.032 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:11.133 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:11.154 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:11.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:11.155 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:11.256 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:11.277 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:11.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:11.280 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:11.379 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:11.400 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:11.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:11.401 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:11.502 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:11.511 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-11T03:44:11.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:11.524 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:11.524 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:11.625 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:11.647 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:11.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:11.748 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:11.764 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:44:11.770 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:11.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:11.872 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:11.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:11.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:11.918 INFO:tasks.rados.rados.2.smithi175.stdout:1575: done (3 left) 2024-09-11T03:44:11.918 INFO:tasks.rados.rados.2.smithi175.stdout:1576: write (excl) oid 4 current snap is 171 2024-09-11T03:44:11.918 INFO:tasks.rados.rados.2.smithi175.stdout:1576: seq_num 553 ranges {494271=790113,1693954=480183,2620209=512129} 2024-09-11T03:44:11.924 INFO:tasks.rados.rados.2.smithi175.stdout:1576: writing smithi17557870-4 from 494271 to 1284384 tid 1 2024-09-11T03:44:11.930 INFO:tasks.rados.rados.2.smithi175.stdout:1576: writing smithi17557870-4 from 1693954 to 2174137 tid 2 2024-09-11T03:44:11.935 INFO:tasks.rados.rados.2.smithi175.stdout:1576: writing smithi17557870-4 from 2620209 to 3132338 tid 3 2024-09-11T03:44:11.935 INFO:tasks.rados.rados.2.smithi175.stdout:1577: snap_remove snap 149 2024-09-11T03:44:11.996 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:12.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:12.019 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:12.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:12.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:12.142 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:12.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:12.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:12.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:11 smithi121 ceph-mon[30729]: pgmap v804: 57 pgs: 57 active+clean; 156 MiB data, 749 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:44:12.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:11 smithi121 ceph-mon[30729]: osdmap e433: 12 total, 12 up, 11 in 2024-09-11T03:44:12.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:11 smithi121 ceph-mon[30729]: 1.13 scrub starts 2024-09-11T03:44:12.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:11 smithi121 ceph-mon[30729]: 1.13 scrub ok 2024-09-11T03:44:12.194 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:11 smithi175 ceph-mon[30894]: pgmap v804: 57 pgs: 57 active+clean; 156 MiB data, 749 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:44:12.194 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:11 smithi175 ceph-mon[30894]: osdmap e433: 12 total, 12 up, 11 in 2024-09-11T03:44:12.194 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:11 smithi175 ceph-mon[30894]: 1.13 scrub starts 2024-09-11T03:44:12.194 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:11 smithi175 ceph-mon[30894]: 1.13 scrub ok 2024-09-11T03:44:12.220 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:44:12.220 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:44:12.220 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:44:12.221 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:44:12.244 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:12.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:12.267 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:12.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:12.368 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:12.391 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:12.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:12.392 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:11 smithi084 ceph-mon[27254]: pgmap v804: 57 pgs: 57 active+clean; 156 MiB data, 749 MiB used, 983 GiB / 983 GiB avail 2024-09-11T03:44:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:11 smithi084 ceph-mon[27254]: osdmap e433: 12 total, 12 up, 11 in 2024-09-11T03:44:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:11 smithi084 ceph-mon[27254]: 1.13 scrub starts 2024-09-11T03:44:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:11 smithi084 ceph-mon[27254]: 1.13 scrub ok 2024-09-11T03:44:12.493 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:12.515 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:12.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:12.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:12.537 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:44:12.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:12.641 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:12.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:12.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:12.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:12.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:12.768 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:12.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:12.782 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:12.859 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-11T03:44:12.869 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:12.890 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:12.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:12.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:12.928 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:44:12.932 INFO:tasks.rados.rados.2.smithi175.stdout:1577: done (4 left) 2024-09-11T03:44:12.932 INFO:tasks.rados.rados.2.smithi175.stdout:1578: read oid 35 snap 161 2024-09-11T03:44:12.932 INFO:tasks.rados.rados.2.smithi175.stdout:1578: expect (ObjNum 518 snap 152 seq_num 518) 2024-09-11T03:44:12.932 INFO:tasks.rados.rados.2.smithi175.stdout:1579: write oid 31 current snap is 171 2024-09-11T03:44:12.932 INFO:tasks.rados.rados.2.smithi175.stdout:1579: seq_num 554 ranges {511078=465362,1752436=740194,3163985=147093} 2024-09-11T03:44:12.938 INFO:tasks.rados.rados.2.smithi175.stdout:1579: writing smithi17557870-31 from 511078 to 976440 tid 1 2024-09-11T03:44:12.947 INFO:tasks.rados.rados.2.smithi175.stdout:1579: writing smithi17557870-31 from 1752436 to 2492630 tid 2 2024-09-11T03:44:12.953 INFO:tasks.rados.rados.2.smithi175.stdout:1579: writing smithi17557870-31 from 3163985 to 3311078 tid 3 2024-09-11T03:44:12.953 INFO:tasks.rados.rados.2.smithi175.stdout:1580: read oid 9 snap -1 2024-09-11T03:44:12.953 INFO:tasks.rados.rados.2.smithi175.stdout:1580: expect (ObjNum 277 snap 73 seq_num 277) 2024-09-11T03:44:12.953 INFO:tasks.rados.rados.2.smithi175.stdout:1581: delete oid 11 current snap is 171 2024-09-11T03:44:12.953 INFO:tasks.rados.rados.2.smithi175.stdout:1572: finishing rollback tid 0 to smithi17557870-24 2024-09-11T03:44:12.953 INFO:tasks.rados.rados.2.smithi175.stdout:1572: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:44:12.953 INFO:tasks.rados.rados.2.smithi175.stdout:1572: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:44:12.953 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 608 (ObjNum 505 snap 146 seq_num 505) dirty exists 2024-09-11T03:44:12.954 INFO:tasks.rados.rados.2.smithi175.stdout:1573: finishing write tid 1 to smithi17557870-26 2024-09-11T03:44:12.954 INFO:tasks.rados.rados.2.smithi175.stdout:1573: finishing write tid 2 to smithi17557870-26 2024-09-11T03:44:12.954 INFO:tasks.rados.rados.2.smithi175.stdout:1573: finishing write tid 3 to smithi17557870-26 2024-09-11T03:44:12.954 INFO:tasks.rados.rados.2.smithi175.stdout:1573: finishing write tid 4 to smithi17557870-26 2024-09-11T03:44:12.954 INFO:tasks.rados.rados.2.smithi175.stdout:1573: finishing write tid 5 to smithi17557870-26 2024-09-11T03:44:12.954 INFO:tasks.rados.rados.2.smithi175.stdout:1573: oid 26 updating version 0 to 268 2024-09-11T03:44:12.954 INFO:tasks.rados.rados.2.smithi175.stdout:1573: oid 26 updating version 268 to 269 2024-09-11T03:44:12.954 INFO:tasks.rados.rados.2.smithi175.stdout:1573: oid 26 updating version 269 to 270 2024-09-11T03:44:12.954 INFO:tasks.rados.rados.2.smithi175.stdout:1573: oid 26 updating version 270 to 271 2024-09-11T03:44:12.954 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 271 (ObjNum 551 snap 171 seq_num 551) dirty exists 2024-09-11T03:44:12.954 INFO:tasks.rados.rados.2.smithi175.stdout:1573: left oid 26 (ObjNum 551 snap 171 seq_num 551) 2024-09-11T03:44:12.954 INFO:tasks.rados.rados.2.smithi175.stdout:1574: finishing write tid 1 to smithi17557870-7 2024-09-11T03:44:12.954 INFO:tasks.rados.rados.2.smithi175.stdout:1574: finishing write tid 2 to smithi17557870-7 2024-09-11T03:44:12.955 INFO:tasks.rados.rados.2.smithi175.stdout:1574: finishing write tid 3 to smithi17557870-7 2024-09-11T03:44:12.955 INFO:tasks.rados.rados.2.smithi175.stdout:1574: finishing write tid 4 to smithi17557870-7 2024-09-11T03:44:12.955 INFO:tasks.rados.rados.2.smithi175.stdout:1574: finishing write tid 5 to smithi17557870-7 2024-09-11T03:44:12.955 INFO:tasks.rados.rados.2.smithi175.stdout:1574: oid 7 updating version 0 to 470 2024-09-11T03:44:12.955 INFO:tasks.rados.rados.2.smithi175.stdout:1574: oid 7 updating version 470 to 471 2024-09-11T03:44:12.955 INFO:tasks.rados.rados.2.smithi175.stdout:1574: oid 7 updating version 471 to 472 2024-09-11T03:44:12.955 INFO:tasks.rados.rados.2.smithi175.stdout:1574: oid 7 updating version 472 to 473 2024-09-11T03:44:12.955 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 473 (ObjNum 552 snap 171 seq_num 552) dirty exists 2024-09-11T03:44:12.955 INFO:tasks.rados.rados.2.smithi175.stdout:1574: left oid 7 (ObjNum 552 snap 171 seq_num 552) 2024-09-11T03:44:12.955 INFO:tasks.rados.rados.2.smithi175.stdout:1576: finishing write tid 1 to smithi17557870-4 2024-09-11T03:44:12.955 INFO:tasks.rados.rados.2.smithi175.stdout:1576: finishing write tid 2 to smithi17557870-4 2024-09-11T03:44:12.955 INFO:tasks.rados.rados.2.smithi175.stdout:1576: finishing write tid 3 to smithi17557870-4 2024-09-11T03:44:12.955 INFO:tasks.rados.rados.2.smithi175.stdout:1576: finishing write tid 4 to smithi17557870-4 2024-09-11T03:44:12.956 INFO:tasks.rados.rados.2.smithi175.stdout:1576: finishing write tid 5 to smithi17557870-4 2024-09-11T03:44:12.956 INFO:tasks.rados.rados.2.smithi175.stdout:1576: oid 4 updating version 0 to 475 2024-09-11T03:44:12.956 INFO:tasks.rados.rados.2.smithi175.stdout:1576: oid 4 version 475 is already newer than 474 2024-09-11T03:44:12.956 INFO:tasks.rados.rados.2.smithi175.stdout:1576: oid 4 updating version 475 to 476 2024-09-11T03:44:12.956 INFO:tasks.rados.rados.2.smithi175.stdout:1576: oid 4 updating version 476 to 477 2024-09-11T03:44:12.956 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 477 (ObjNum 553 snap 171 seq_num 553) dirty exists 2024-09-11T03:44:12.956 INFO:tasks.rados.rados.2.smithi175.stdout:1576: left oid 4 (ObjNum 553 snap 171 seq_num 553) 2024-09-11T03:44:12.956 INFO:tasks.rados.rados.2.smithi175.stdout:1578: expect (ObjNum 518 snap 152 seq_num 518) 2024-09-11T03:44:12.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:13.013 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:13.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:13.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:13.115 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:13.125 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:44:13.139 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:13.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:13.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:13.170 INFO:tasks.rados.rados.2.smithi175.stdout:1579: finishing write tid 1 to smithi17557870-31 2024-09-11T03:44:13.170 INFO:tasks.rados.rados.2.smithi175.stdout:1572: done (7 left) 2024-09-11T03:44:13.170 INFO:tasks.rados.rados.2.smithi175.stdout:1573: done (6 left) 2024-09-11T03:44:13.170 INFO:tasks.rados.rados.2.smithi175.stdout:1574: done (5 left) 2024-09-11T03:44:13.170 INFO:tasks.rados.rados.2.smithi175.stdout:1576: done (4 left) 2024-09-11T03:44:13.170 INFO:tasks.rados.rados.2.smithi175.stdout:1578: done (3 left) 2024-09-11T03:44:13.170 INFO:tasks.rados.rados.2.smithi175.stdout:1581: done (2 left) 2024-09-11T03:44:13.170 INFO:tasks.rados.rados.2.smithi175.stdout:1582: read oid 8 snap -1 2024-09-11T03:44:13.170 INFO:tasks.rados.rados.2.smithi175.stdout:1582: expect (ObjNum 515 snap 152 seq_num 515) 2024-09-11T03:44:13.170 INFO:tasks.rados.rados.2.smithi175.stdout:1583: write oid 26 current snap is 171 2024-09-11T03:44:13.170 INFO:tasks.rados.rados.2.smithi175.stdout:1579: finishing write tid 2 to smithi17557870-31 2024-09-11T03:44:13.170 INFO:tasks.rados.rados.2.smithi175.stdout:1583: seq_num 556 ranges {544692=732213,1985753=743863,3335184=9508} 2024-09-11T03:44:13.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:12 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:44:13.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:12 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:44:13.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:12 smithi121 ceph-mon[30729]: osdmap e434: 12 total, 12 up, 11 in 2024-09-11T03:44:13.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:12 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3137049399' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:44:13.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:12 smithi121 ceph-mon[30729]: 1.14 scrub starts 2024-09-11T03:44:13.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:12 smithi121 ceph-mon[30729]: 1.14 scrub ok 2024-09-11T03:44:13.176 INFO:tasks.rados.rados.2.smithi175.stdout:1583: writing smithi17557870-26 from 544692 to 1276905 tid 1 2024-09-11T03:44:13.184 INFO:tasks.rados.rados.2.smithi175.stdout:1583: writing smithi17557870-26 from 1985753 to 2729616 tid 2 2024-09-11T03:44:13.188 INFO:tasks.rados.rados.2.smithi175.stdout:1583: writing smithi17557870-26 from 3335184 to 3344692 tid 3 2024-09-11T03:44:13.188 INFO:tasks.rados.rados.2.smithi175.stdout:1584: copy_from oid 44 from oid 1 current snap is 171 2024-09-11T03:44:13.188 INFO:tasks.rados.rados.2.smithi175.stdout:1580: expect (ObjNum 277 snap 73 seq_num 277) 2024-09-11T03:44:13.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:12 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:44:13.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:12 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:44:13.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:12 smithi175 ceph-mon[30894]: osdmap e434: 12 total, 12 up, 11 in 2024-09-11T03:44:13.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:12 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3137049399' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:44:13.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:12 smithi175 ceph-mon[30894]: 1.14 scrub starts 2024-09-11T03:44:13.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:12 smithi175 ceph-mon[30894]: 1.14 scrub ok 2024-09-11T03:44:13.242 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:13.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:13.264 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:13.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:13.276 INFO:tasks.rados.rados.2.smithi175.stdout:1579: finishing write tid 3 to smithi17557870-31 2024-09-11T03:44:13.276 INFO:tasks.rados.rados.2.smithi175.stdout:1579: finishing write tid 4 to smithi17557870-31 2024-09-11T03:44:13.277 INFO:tasks.rados.rados.2.smithi175.stdout:1579: finishing write tid 5 to smithi17557870-31 2024-09-11T03:44:13.277 INFO:tasks.rados.rados.2.smithi175.stdout:1579: oid 31 updating version 0 to 561 2024-09-11T03:44:13.278 INFO:tasks.rados.rados.2.smithi175.stdout:1579: oid 31 version 561 is already newer than 560 2024-09-11T03:44:13.278 INFO:tasks.rados.rados.2.smithi175.stdout:1579: oid 31 updating version 561 to 562 2024-09-11T03:44:13.278 INFO:tasks.rados.rados.2.smithi175.stdout:1579: oid 31 version 562 is already newer than 559 2024-09-11T03:44:13.278 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 562 (ObjNum 554 snap 171 seq_num 554) dirty exists 2024-09-11T03:44:13.278 INFO:tasks.rados.rados.2.smithi175.stdout:1579: left oid 31 (ObjNum 554 snap 171 seq_num 554) 2024-09-11T03:44:13.278 INFO:tasks.rados.rados.2.smithi175.stdout:1579: done (4 left) 2024-09-11T03:44:13.278 INFO:tasks.rados.rados.2.smithi175.stdout:1580: done (3 left) 2024-09-11T03:44:13.278 INFO:tasks.rados.rados.2.smithi175.stdout:1585: copy_from oid 46 from oid 47 current snap is 171 2024-09-11T03:44:13.278 INFO:tasks.rados.rados.2.smithi175.stdout:1582: expect (ObjNum 515 snap 152 seq_num 515) 2024-09-11T03:44:13.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:13.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:12 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:44:13.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:12 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:44:13.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:12 smithi084 ceph-mon[27254]: osdmap e434: 12 total, 12 up, 11 in 2024-09-11T03:44:13.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:12 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3137049399' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:44:13.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:12 smithi084 ceph-mon[27254]: 1.14 scrub starts 2024-09-11T03:44:13.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:12 smithi084 ceph-mon[27254]: 1.14 scrub ok 2024-09-11T03:44:13.389 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:13.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:13.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:13.465 INFO:tasks.rados.rados.2.smithi175.stdout:1583: finishing write tid 1 to smithi17557870-26 2024-09-11T03:44:13.465 INFO:tasks.rados.rados.2.smithi175.stdout:1583: finishing write tid 2 to smithi17557870-26 2024-09-11T03:44:13.465 INFO:tasks.rados.rados.2.smithi175.stdout:1583: finishing write tid 3 to smithi17557870-26 2024-09-11T03:44:13.465 INFO:tasks.rados.rados.2.smithi175.stdout:1583: finishing write tid 4 to smithi17557870-26 2024-09-11T03:44:13.465 INFO:tasks.rados.rados.2.smithi175.stdout:1583: finishing write tid 5 to smithi17557870-26 2024-09-11T03:44:13.465 INFO:tasks.rados.rados.2.smithi175.stdout:1583: oid 26 updating version 0 to 279 2024-09-11T03:44:13.465 INFO:tasks.rados.rados.2.smithi175.stdout:1583: oid 26 version 279 is already newer than 276 2024-09-11T03:44:13.465 INFO:tasks.rados.rados.2.smithi175.stdout:1583: oid 26 version 279 is already newer than 277 2024-09-11T03:44:13.465 INFO:tasks.rados.rados.2.smithi175.stdout:1583: oid 26 version 279 is already newer than 278 2024-09-11T03:44:13.466 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 279 (ObjNum 556 snap 171 seq_num 556) dirty exists 2024-09-11T03:44:13.466 INFO:tasks.rados.rados.2.smithi175.stdout:1583: left oid 26 (ObjNum 556 snap 171 seq_num 556) 2024-09-11T03:44:13.466 INFO:tasks.rados.rados.2.smithi175.stdout:1584: finishing copy_from racing read to smithi17557870-44 2024-09-11T03:44:13.466 INFO:tasks.rados.rados.2.smithi175.stdout:1584: finishing copy_from to smithi17557870-44 2024-09-11T03:44:13.466 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 557 (ObjNum 520 snap 154 seq_num 520) dirty exists 2024-09-11T03:44:13.466 INFO:tasks.rados.rados.2.smithi175.stdout:1582: done (3 left) 2024-09-11T03:44:13.466 INFO:tasks.rados.rados.2.smithi175.stdout:1583: done (2 left) 2024-09-11T03:44:13.466 INFO:tasks.rados.rados.2.smithi175.stdout:1584: done (1 left) 2024-09-11T03:44:13.466 INFO:tasks.rados.rados.2.smithi175.stdout:1586: delete oid 36 current snap is 171 2024-09-11T03:44:13.466 INFO:tasks.rados.rados.2.smithi175.stdout:1585: finishing copy_from to smithi17557870-46 2024-09-11T03:44:13.466 INFO:tasks.rados.rados.2.smithi175.stdout:1585: got expected ENOENT (src dne) 2024-09-11T03:44:13.466 INFO:tasks.rados.rados.2.smithi175.stdout:1585: finishing copy_from racing read to smithi17557870-46 2024-09-11T03:44:13.467 INFO:tasks.rados.rados.2.smithi175.stdout:1585: done (1 left) 2024-09-11T03:44:13.467 INFO:tasks.rados.rados.2.smithi175.stdout:1586: done (0 left) 2024-09-11T03:44:13.467 INFO:tasks.rados.rados.2.smithi175.stdout:1587: delete oid 46 current snap is 171 2024-09-11T03:44:13.468 INFO:tasks.rados.rados.2.smithi175.stdout:1587: done (0 left) 2024-09-11T03:44:13.468 INFO:tasks.rados.rados.2.smithi175.stdout:1588: write oid 20 current snap is 171 2024-09-11T03:44:13.469 INFO:tasks.rados.rados.2.smithi175.stdout:1588: seq_num 559 ranges {595113=557960,1644846=491190,2784747=610366} 2024-09-11T03:44:13.474 INFO:tasks.rados.rados.2.smithi175.stdout:1588: writing smithi17557870-20 from 595113 to 1153073 tid 1 2024-09-11T03:44:13.479 INFO:tasks.rados.rados.2.smithi175.stdout:1588: writing smithi17557870-20 from 1644846 to 2136036 tid 2 2024-09-11T03:44:13.487 INFO:tasks.rados.rados.2.smithi175.stdout:1588: writing smithi17557870-20 from 2784747 to 3395113 tid 3 2024-09-11T03:44:13.487 INFO:tasks.rados.rados.2.smithi175.stdout:1589: delete oid 14 current snap is 171 2024-09-11T03:44:13.487 INFO:tasks.rados.rados.2.smithi175.stdout:1588: finishing write tid 1 to smithi17557870-20 2024-09-11T03:44:13.487 INFO:tasks.rados.rados.2.smithi175.stdout:1588: finishing write tid 2 to smithi17557870-20 2024-09-11T03:44:13.489 INFO:tasks.rados.rados.2.smithi175.stdout:1589: done (1 left) 2024-09-11T03:44:13.489 INFO:tasks.rados.rados.2.smithi175.stdout:1590: copy_from oid 31 from oid 7 current snap is 171 2024-09-11T03:44:13.489 INFO:tasks.rados.rados.2.smithi175.stdout:1591: copy_from oid 2 from oid 39 current snap is 171 2024-09-11T03:44:13.489 INFO:tasks.rados.rados.2.smithi175.stdout:1592: write oid 47 current snap is 171 2024-09-11T03:44:13.489 INFO:tasks.rados.rados.2.smithi175.stdout:1592: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2024-09-11T03:44:13.490 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:13.496 INFO:tasks.rados.rados.2.smithi175.stdout:1592: writing smithi17557870-47 from 628727 to 1337185 tid 1 2024-09-11T03:44:13.502 INFO:tasks.rados.rados.2.smithi175.stdout:1592: writing smithi17557870-47 from 2045457 to 2540316 tid 2 2024-09-11T03:44:13.508 INFO:tasks.rados.rados.2.smithi175.stdout:1592: writing smithi17557870-47 from 3123240 to 3428727 tid 3 2024-09-11T03:44:13.508 INFO:tasks.rados.rados.2.smithi175.stdout:1593: read oid 8 snap -1 2024-09-11T03:44:13.508 INFO:tasks.rados.rados.2.smithi175.stdout:1593: expect (ObjNum 515 snap 152 seq_num 515) 2024-09-11T03:44:13.508 INFO:tasks.rados.rados.2.smithi175.stdout:1594: write oid 48 current snap is 171 2024-09-11T03:44:13.508 INFO:tasks.rados.rados.2.smithi175.stdout:1588: finishing write tid 3 to smithi17557870-20 2024-09-11T03:44:13.508 INFO:tasks.rados.rados.2.smithi175.stdout:1594: seq_num 562 ranges {645534=783707,2103939=638517,3150663=294871} 2024-09-11T03:44:13.515 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:13.515 INFO:tasks.rados.rados.2.smithi175.stdout:1594: writing smithi17557870-48 from 645534 to 1429241 tid 1 2024-09-11T03:44:13.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:13.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:13.524 INFO:tasks.rados.rados.2.smithi175.stdout:1594: writing smithi17557870-48 from 2103939 to 2742456 tid 2 2024-09-11T03:44:13.528 INFO:tasks.rados.rados.2.smithi175.stdout:1594: writing smithi17557870-48 from 3150663 to 3445534 tid 3 2024-09-11T03:44:13.528 INFO:tasks.rados.rados.2.smithi175.stdout:1595: snap_create 2024-09-11T03:44:13.528 INFO:tasks.rados.rados.2.smithi175.stdout:1588: finishing write tid 4 to smithi17557870-20 2024-09-11T03:44:13.529 INFO:tasks.rados.rados.2.smithi175.stdout:1588: finishing write tid 5 to smithi17557870-20 2024-09-11T03:44:13.529 INFO:tasks.rados.rados.2.smithi175.stdout:1588: oid 20 updating version 0 to 591 2024-09-11T03:44:13.529 INFO:tasks.rados.rados.2.smithi175.stdout:1588: oid 20 updating version 591 to 593 2024-09-11T03:44:13.529 INFO:tasks.rados.rados.2.smithi175.stdout:1588: oid 20 version 593 is already newer than 592 2024-09-11T03:44:13.529 INFO:tasks.rados.rados.2.smithi175.stdout:1588: oid 20 updating version 593 to 594 2024-09-11T03:44:13.529 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 594 (ObjNum 559 snap 171 seq_num 559) dirty exists 2024-09-11T03:44:13.529 INFO:tasks.rados.rados.2.smithi175.stdout:1588: left oid 20 (ObjNum 559 snap 171 seq_num 559) 2024-09-11T03:44:13.529 INFO:tasks.rados.rados.2.smithi175.stdout:1591: finishing copy_from to smithi17557870-2 2024-09-11T03:44:13.529 INFO:tasks.rados.rados.2.smithi175.stdout:1591: got expected ENOENT (src dne) 2024-09-11T03:44:13.529 INFO:tasks.rados.rados.2.smithi175.stdout:1591: finishing copy_from racing read to smithi17557870-2 2024-09-11T03:44:13.529 INFO:tasks.rados.rados.2.smithi175.stdout:1590: finishing copy_from racing read to smithi17557870-31 2024-09-11T03:44:13.529 INFO:tasks.rados.rados.2.smithi175.stdout:1592: finishing write tid 1 to smithi17557870-47 2024-09-11T03:44:13.529 INFO:tasks.rados.rados.2.smithi175.stdout:1592: finishing write tid 2 to smithi17557870-47 2024-09-11T03:44:13.529 INFO:tasks.rados.rados.2.smithi175.stdout:1590: finishing copy_from to smithi17557870-31 2024-09-11T03:44:13.530 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 473 (ObjNum 552 snap 171 seq_num 552) dirty exists 2024-09-11T03:44:13.530 INFO:tasks.rados.rados.2.smithi175.stdout:1592: finishing write tid 3 to smithi17557870-47 2024-09-11T03:44:13.530 INFO:tasks.rados.rados.2.smithi175.stdout:1592: finishing write tid 4 to smithi17557870-47 2024-09-11T03:44:13.530 INFO:tasks.rados.rados.2.smithi175.stdout:1592: finishing write tid 5 to smithi17557870-47 2024-09-11T03:44:13.530 INFO:tasks.rados.rados.2.smithi175.stdout:1592: oid 47 updating version 0 to 596 2024-09-11T03:44:13.530 INFO:tasks.rados.rados.2.smithi175.stdout:1592: oid 47 updating version 596 to 598 2024-09-11T03:44:13.530 INFO:tasks.rados.rados.2.smithi175.stdout:1592: oid 47 version 598 is already newer than 597 2024-09-11T03:44:13.530 INFO:tasks.rados.rados.2.smithi175.stdout:1592: oid 47 version 598 is already newer than 595 2024-09-11T03:44:13.530 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 598 (ObjNum 561 snap 171 seq_num 561) dirty exists 2024-09-11T03:44:13.530 INFO:tasks.rados.rados.2.smithi175.stdout:1592: left oid 47 (ObjNum 561 snap 171 seq_num 561) 2024-09-11T03:44:13.530 INFO:tasks.rados.rados.2.smithi175.stdout:1593: expect (ObjNum 515 snap 152 seq_num 515) 2024-09-11T03:44:13.617 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:13.629 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:44:13.629 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:44:13.629 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:44:13.629 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:44:13.629 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:44:13.638 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:13.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:13.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:13.724 INFO:tasks.rados.rados.2.smithi175.stdout:1594: finishing write tid 1 to smithi17557870-48 2024-09-11T03:44:13.724 INFO:tasks.rados.rados.2.smithi175.stdout:1594: finishing write tid 2 to smithi17557870-48 2024-09-11T03:44:13.724 INFO:tasks.rados.rados.2.smithi175.stdout:1594: finishing write tid 3 to smithi17557870-48 2024-09-11T03:44:13.724 INFO:tasks.rados.rados.2.smithi175.stdout:1594: finishing write tid 4 to smithi17557870-48 2024-09-11T03:44:13.724 INFO:tasks.rados.rados.2.smithi175.stdout:1594: finishing write tid 5 to smithi17557870-48 2024-09-11T03:44:13.724 INFO:tasks.rados.rados.2.smithi175.stdout:1594: oid 48 updating version 0 to 479 2024-09-11T03:44:13.724 INFO:tasks.rados.rados.2.smithi175.stdout:1594: oid 48 updating version 479 to 480 2024-09-11T03:44:13.725 INFO:tasks.rados.rados.2.smithi175.stdout:1594: oid 48 updating version 480 to 481 2024-09-11T03:44:13.725 INFO:tasks.rados.rados.2.smithi175.stdout:1594: oid 48 version 481 is already newer than 478 2024-09-11T03:44:13.725 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 481 (ObjNum 562 snap 171 seq_num 562) dirty exists 2024-09-11T03:44:13.725 INFO:tasks.rados.rados.2.smithi175.stdout:1594: left oid 48 (ObjNum 562 snap 171 seq_num 562) 2024-09-11T03:44:13.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:13.764 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:13.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:13.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:13.865 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:13.887 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:13.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:13.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:13.931 INFO:tasks.rados.rados.2.smithi175.stdout:1588: done (6 left) 2024-09-11T03:44:13.931 INFO:tasks.rados.rados.2.smithi175.stdout:1590: done (5 left) 2024-09-11T03:44:13.931 INFO:tasks.rados.rados.2.smithi175.stdout:1591: done (4 left) 2024-09-11T03:44:13.931 INFO:tasks.rados.rados.2.smithi175.stdout:1592: done (3 left) 2024-09-11T03:44:13.931 INFO:tasks.rados.rados.2.smithi175.stdout:1593: done (2 left) 2024-09-11T03:44:13.932 INFO:tasks.rados.rados.2.smithi175.stdout:1594: done (1 left) 2024-09-11T03:44:13.932 INFO:tasks.rados.rados.2.smithi175.stdout:1595: done (0 left) 2024-09-11T03:44:13.932 INFO:tasks.rados.rados.2.smithi175.stdout:1596: rollback oid 43 current snap is 172 2024-09-11T03:44:13.932 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 43 to 169 2024-09-11T03:44:13.932 INFO:tasks.rados.rados.2.smithi175.stdout:1597: read oid 40 snap -1 2024-09-11T03:44:13.932 INFO:tasks.rados.rados.2.smithi175.stdout:1597: expect (ObjNum 487 snap 141 seq_num 487) 2024-09-11T03:44:13.932 INFO:tasks.rados.rados.2.smithi175.stdout:1598: write oid 20 current snap is 172 2024-09-11T03:44:13.932 INFO:tasks.rados.rados.2.smithi175.stdout:1598: seq_num 563 ranges {662341=575309,1878774=498528,2894439=457555} 2024-09-11T03:44:13.937 INFO:tasks.rados.rados.2.smithi175.stdout:1598: writing smithi17557870-20 from 662341 to 1237650 tid 1 2024-09-11T03:44:13.944 INFO:tasks.rados.rados.2.smithi175.stdout:1598: writing smithi17557870-20 from 1878774 to 2377302 tid 2 2024-09-11T03:44:13.950 INFO:tasks.rados.rados.2.smithi175.stdout:1598: writing smithi17557870-20 from 2894439 to 3351994 tid 3 2024-09-11T03:44:13.950 INFO:tasks.rados.rados.2.smithi175.stdout:1599: copy_from oid 39 from oid 36 current snap is 172 2024-09-11T03:44:13.950 INFO:tasks.rados.rados.2.smithi175.stdout:1596: finishing rollback tid 0 to smithi17557870-43 2024-09-11T03:44:13.950 INFO:tasks.rados.rados.2.smithi175.stdout:1596: finishing rollback tid 1 to smithi17557870-43 2024-09-11T03:44:13.950 INFO:tasks.rados.rados.2.smithi175.stdout:1596: finishing rollback tid 2 to smithi17557870-43 2024-09-11T03:44:13.950 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 396 (ObjNum 536 snap 158 seq_num 536) dirty exists 2024-09-11T03:44:13.950 INFO:tasks.rados.rados.2.smithi175.stdout:1597: expect (ObjNum 487 snap 141 seq_num 487) 2024-09-11T03:44:13.989 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:14.011 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:14.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:14.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:14.023 INFO:tasks.rados.rados.2.smithi175.stdout:1598: finishing write tid 1 to smithi17557870-20 2024-09-11T03:44:14.023 INFO:tasks.rados.rados.2.smithi175.stdout:1598: finishing write tid 2 to smithi17557870-20 2024-09-11T03:44:14.023 INFO:tasks.rados.rados.2.smithi175.stdout:1596: done (3 left) 2024-09-11T03:44:14.023 INFO:tasks.rados.rados.2.smithi175.stdout:1597: done (2 left) 2024-09-11T03:44:14.023 INFO:tasks.rados.rados.2.smithi175.stdout:1600: delete oid 40 current snap is 172 2024-09-11T03:44:14.023 INFO:tasks.rados.rados.2.smithi175.stdout:1599: finishing copy_from to smithi17557870-39 2024-09-11T03:44:14.023 INFO:tasks.rados.rados.2.smithi175.stdout:1599: got expected ENOENT (src dne) 2024-09-11T03:44:14.023 INFO:tasks.rados.rados.2.smithi175.stdout:1599: finishing copy_from racing read to smithi17557870-39 2024-09-11T03:44:14.023 INFO:tasks.rados.rados.2.smithi175.stdout:1598: finishing write tid 3 to smithi17557870-20 2024-09-11T03:44:14.023 INFO:tasks.rados.rados.2.smithi175.stdout:1598: finishing write tid 4 to smithi17557870-20 2024-09-11T03:44:14.024 INFO:tasks.rados.rados.2.smithi175.stdout:1598: finishing write tid 5 to smithi17557870-20 2024-09-11T03:44:14.024 INFO:tasks.rados.rados.2.smithi175.stdout:1598: oid 20 updating version 0 to 602 2024-09-11T03:44:14.024 INFO:tasks.rados.rados.2.smithi175.stdout:1598: oid 20 updating version 602 to 603 2024-09-11T03:44:14.024 INFO:tasks.rados.rados.2.smithi175.stdout:1598: oid 20 version 603 is already newer than 601 2024-09-11T03:44:14.024 INFO:tasks.rados.rados.2.smithi175.stdout:1598: oid 20 version 603 is already newer than 600 2024-09-11T03:44:14.024 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 603 (ObjNum 563 snap 172 seq_num 563) dirty exists 2024-09-11T03:44:14.024 INFO:tasks.rados.rados.2.smithi175.stdout:1598: left oid 20 (ObjNum 563 snap 172 seq_num 563) 2024-09-11T03:44:14.025 INFO:tasks.rados.rados.2.smithi175.stdout:1598: done (2 left) 2024-09-11T03:44:14.025 INFO:tasks.rados.rados.2.smithi175.stdout:1599: done (1 left) 2024-09-11T03:44:14.025 INFO:tasks.rados.rados.2.smithi175.stdout:1600: done (0 left) 2024-09-11T03:44:14.025 INFO:tasks.rados.rados.2.smithi175.stdout:1601: rollback oid 29 current snap is 172 2024-09-11T03:44:14.025 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 29 to 156 2024-09-11T03:44:14.025 INFO:tasks.rados.rados.2.smithi175.stdout:1602: write oid 49 current snap is 172 2024-09-11T03:44:14.025 INFO:tasks.rados.rados.2.smithi175.stdout:1602: seq_num 565 ranges {695955=725807,1879385=502197,2949285=480099} 2024-09-11T03:44:14.026 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:44:14.026 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:44:14.032 INFO:tasks.rados.rados.2.smithi175.stdout:1602: writing smithi17557870-49 from 695955 to 1421762 tid 1 2024-09-11T03:44:14.037 INFO:tasks.rados.rados.2.smithi175.stdout:1602: writing smithi17557870-49 from 1879385 to 2381582 tid 2 2024-09-11T03:44:14.043 INFO:tasks.rados.rados.2.smithi175.stdout:1602: writing smithi17557870-49 from 2949285 to 3429384 tid 3 2024-09-11T03:44:14.043 INFO:tasks.rados.rados.2.smithi175.stdout:1603: snap_remove snap 161 2024-09-11T03:44:14.044 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:44:14.045 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:44:14.045 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:13 smithi084 ceph-mon[27254]: 2.12 scrub starts 2024-09-11T03:44:14.045 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:13 smithi084 ceph-mon[27254]: 2.12 scrub ok 2024-09-11T03:44:14.045 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:13 smithi084 ceph-mon[27254]: 1.e scrub starts 2024-09-11T03:44:14.045 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:13 smithi084 ceph-mon[27254]: 1.e scrub ok 2024-09-11T03:44:14.045 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:13 smithi084 ceph-mon[27254]: pgmap v807: 57 pgs: 57 active+clean; 160 MiB data, 755 MiB used, 983 GiB / 983 GiB avail; 0 B/s rd, 765 KiB/s wr, 2 op/s 2024-09-11T03:44:14.045 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:13 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:44:14.045 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:13 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:44:14.045 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:13 smithi084 ceph-mon[27254]: 1.a scrub starts 2024-09-11T03:44:14.046 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:13 smithi084 ceph-mon[27254]: 1.a scrub ok 2024-09-11T03:44:14.046 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:13 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:14.046 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:13 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3137049399' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:44:14.046 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:13 smithi084 ceph-mon[27254]: osdmap e435: 12 total, 12 up, 11 in 2024-09-11T03:44:14.046 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:13 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:44:14.061 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:44:14.062 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:44:14.078 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:44:14.078 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:44:14.095 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:44:14.095 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:44:14.111 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:44:14.111 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:44:14.113 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:14.128 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:44:14.128 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:44:14.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:14.136 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:14.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:14.146 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:44:14.146 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:44:14.163 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:44:14.164 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:44:14.186 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:44:14.187 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:44:14.207 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:44:14.207 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:44:14.223 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:44:14.223 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:44:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:13 smithi175 ceph-mon[30894]: 2.12 scrub starts 2024-09-11T03:44:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:13 smithi175 ceph-mon[30894]: 2.12 scrub ok 2024-09-11T03:44:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:13 smithi175 ceph-mon[30894]: 1.e scrub starts 2024-09-11T03:44:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:13 smithi175 ceph-mon[30894]: 1.e scrub ok 2024-09-11T03:44:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:13 smithi175 ceph-mon[30894]: pgmap v807: 57 pgs: 57 active+clean; 160 MiB data, 755 MiB used, 983 GiB / 983 GiB avail; 0 B/s rd, 765 KiB/s wr, 2 op/s 2024-09-11T03:44:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:13 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:44:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:13 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:44:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:13 smithi175 ceph-mon[30894]: 1.a scrub starts 2024-09-11T03:44:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:13 smithi175 ceph-mon[30894]: 1.a scrub ok 2024-09-11T03:44:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:13 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:13 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3137049399' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:44:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:13 smithi175 ceph-mon[30894]: osdmap e435: 12 total, 12 up, 11 in 2024-09-11T03:44:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:13 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:44:14.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:14.260 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:14.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:14.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:14.289 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-11T03:44:14.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:14.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:14.384 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:14.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:14.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:13 smithi121 ceph-mon[30729]: 2.12 scrub starts 2024-09-11T03:44:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:13 smithi121 ceph-mon[30729]: 2.12 scrub ok 2024-09-11T03:44:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:13 smithi121 ceph-mon[30729]: 1.e scrub starts 2024-09-11T03:44:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:13 smithi121 ceph-mon[30729]: 1.e scrub ok 2024-09-11T03:44:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:13 smithi121 ceph-mon[30729]: pgmap v807: 57 pgs: 57 active+clean; 160 MiB data, 755 MiB used, 983 GiB / 983 GiB avail; 0 B/s rd, 765 KiB/s wr, 2 op/s 2024-09-11T03:44:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:13 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:44:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:13 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:44:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:13 smithi121 ceph-mon[30729]: 1.a scrub starts 2024-09-11T03:44:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:13 smithi121 ceph-mon[30729]: 1.a scrub ok 2024-09-11T03:44:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:13 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:13 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3137049399' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:44:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:13 smithi121 ceph-mon[30729]: osdmap e435: 12 total, 12 up, 11 in 2024-09-11T03:44:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:13 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:44:14.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:14.507 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:14.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:14.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:14.529 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:44:14.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:14.630 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:14.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:14.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:14.732 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:14.754 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:14.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:14.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:14.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:14.877 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:14.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:14.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:14.923 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:44:14.924 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:44:14.936 INFO:tasks.rados.rados.2.smithi175.stdout:1603: done (2 left) 2024-09-11T03:44:14.936 INFO:tasks.rados.rados.2.smithi175.stdout:1604: write (excl) oid 15 current snap is 172 2024-09-11T03:44:14.936 INFO:tasks.rados.rados.2.smithi175.stdout:1601: finishing rollback tid 1 to smithi17557870-29 2024-09-11T03:44:14.936 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 0 (ObjNum 129 snap 0 seq_num 1745270504) dirty dne 2024-09-11T03:44:14.936 INFO:tasks.rados.rados.2.smithi175.stdout:1604: seq_num 566 ranges {712762=401056,1537867=645855,2860355=652407} 2024-09-11T03:44:14.942 INFO:tasks.rados.rados.2.smithi175.stdout:1604: writing smithi17557870-15 from 712762 to 1113818 tid 1 2024-09-11T03:44:14.949 INFO:tasks.rados.rados.2.smithi175.stdout:1604: writing smithi17557870-15 from 1537867 to 2183722 tid 2 2024-09-11T03:44:14.956 INFO:tasks.rados.rados.2.smithi175.stdout:1604: writing smithi17557870-15 from 2860355 to 3512762 tid 3 2024-09-11T03:44:14.957 INFO:tasks.rados.rados.2.smithi175.stdout:1601: done (2 left) 2024-09-11T03:44:14.957 INFO:tasks.rados.rados.2.smithi175.stdout:1605: write oid 44 current snap is 172 2024-09-11T03:44:14.957 INFO:tasks.rados.rados.2.smithi175.stdout:1602: finishing write tid 1 to smithi17557870-49 2024-09-11T03:44:14.957 INFO:tasks.rados.rados.2.smithi175.stdout:1602: finishing write tid 2 to smithi17557870-49 2024-09-11T03:44:14.957 INFO:tasks.rados.rados.2.smithi175.stdout:1602: finishing write tid 3 to smithi17557870-49 2024-09-11T03:44:14.957 INFO:tasks.rados.rados.2.smithi175.stdout:1602: finishing write tid 4 to smithi17557870-49 2024-09-11T03:44:14.957 INFO:tasks.rados.rados.2.smithi175.stdout:1602: finishing write tid 5 to smithi17557870-49 2024-09-11T03:44:14.957 INFO:tasks.rados.rados.2.smithi175.stdout:1602: oid 49 updating version 0 to 465 2024-09-11T03:44:14.957 INFO:tasks.rados.rados.2.smithi175.stdout:1602: oid 49 updating version 465 to 466 2024-09-11T03:44:14.957 INFO:tasks.rados.rados.2.smithi175.stdout:1602: oid 49 version 466 is already newer than 463 2024-09-11T03:44:14.957 INFO:tasks.rados.rados.2.smithi175.stdout:1602: oid 49 version 466 is already newer than 464 2024-09-11T03:44:14.957 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 466 (ObjNum 565 snap 172 seq_num 565) dirty exists 2024-09-11T03:44:14.958 INFO:tasks.rados.rados.2.smithi175.stdout:1602: left oid 49 (ObjNum 565 snap 172 seq_num 565) 2024-09-11T03:44:14.958 INFO:tasks.rados.rados.2.smithi175.stdout:1605: seq_num 567 ranges {729569=476305,1996349=505866,3004131=525438} 2024-09-11T03:44:14.963 INFO:tasks.rados.rados.2.smithi175.stdout:1605: writing smithi17557870-44 from 729569 to 1205874 tid 1 2024-09-11T03:44:14.970 INFO:tasks.rados.rados.2.smithi175.stdout:1605: writing smithi17557870-44 from 1996349 to 2502215 tid 2 2024-09-11T03:44:14.976 INFO:tasks.rados.rados.2.smithi175.stdout:1605: writing smithi17557870-44 from 3004131 to 3529569 tid 3 2024-09-11T03:44:14.976 INFO:tasks.rados.rados.2.smithi175.stdout:1602: done (2 left) 2024-09-11T03:44:14.976 INFO:tasks.rados.rados.2.smithi175.stdout:1606: delete oid 36 current snap is 172 2024-09-11T03:44:14.976 INFO:tasks.rados.rados.2.smithi175.stdout:1604: finishing write tid 1 to smithi17557870-15 2024-09-11T03:44:14.976 INFO:tasks.rados.rados.2.smithi175.stdout:1604: finishing write tid 2 to smithi17557870-15 2024-09-11T03:44:14.977 INFO:tasks.rados.rados.2.smithi175.stdout:1604: finishing write tid 3 to smithi17557870-15 2024-09-11T03:44:14.977 INFO:tasks.rados.rados.2.smithi175.stdout:1604: finishing write tid 4 to smithi17557870-15 2024-09-11T03:44:14.977 INFO:tasks.rados.rados.2.smithi175.stdout:1604: finishing write tid 5 to smithi17557870-15 2024-09-11T03:44:14.977 INFO:tasks.rados.rados.2.smithi175.stdout:1604: oid 15 updating version 0 to 220 2024-09-11T03:44:14.977 INFO:tasks.rados.rados.2.smithi175.stdout:1604: oid 15 updating version 220 to 222 2024-09-11T03:44:14.977 INFO:tasks.rados.rados.2.smithi175.stdout:1604: oid 15 version 222 is already newer than 221 2024-09-11T03:44:14.977 INFO:tasks.rados.rados.2.smithi175.stdout:1604: oid 15 updating version 222 to 223 2024-09-11T03:44:14.977 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 223 (ObjNum 566 snap 172 seq_num 566) dirty exists 2024-09-11T03:44:14.977 INFO:tasks.rados.rados.2.smithi175.stdout:1604: left oid 15 (ObjNum 566 snap 172 seq_num 566) 2024-09-11T03:44:14.977 INFO:tasks.rados.rados.2.smithi175.stdout:1605: finishing write tid 1 to smithi17557870-44 2024-09-11T03:44:14.977 INFO:tasks.rados.rados.2.smithi175.stdout:1605: finishing write tid 2 to smithi17557870-44 2024-09-11T03:44:14.978 INFO:tasks.rados.rados.2.smithi175.stdout:1604: done (2 left) 2024-09-11T03:44:14.978 INFO:tasks.rados.rados.2.smithi175.stdout:1606: done (1 left) 2024-09-11T03:44:14.978 INFO:tasks.rados.rados.2.smithi175.stdout:1607: snap_create 2024-09-11T03:44:14.978 INFO:tasks.rados.rados.2.smithi175.stdout:1605: finishing write tid 3 to smithi17557870-44 2024-09-11T03:44:14.979 INFO:tasks.rados.rados.2.smithi175.stdout:1605: finishing write tid 4 to smithi17557870-44 2024-09-11T03:44:14.979 INFO:tasks.rados.rados.2.smithi175.stdout:1605: finishing write tid 5 to smithi17557870-44 2024-09-11T03:44:14.979 INFO:tasks.rados.rados.2.smithi175.stdout:1605: oid 44 updating version 0 to 559 2024-09-11T03:44:14.979 INFO:tasks.rados.rados.2.smithi175.stdout:1605: oid 44 version 559 is already newer than 558 2024-09-11T03:44:14.979 INFO:tasks.rados.rados.2.smithi175.stdout:1605: oid 44 updating version 559 to 561 2024-09-11T03:44:14.979 INFO:tasks.rados.rados.2.smithi175.stdout:1605: oid 44 version 561 is already newer than 560 2024-09-11T03:44:14.979 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 561 (ObjNum 567 snap 172 seq_num 567) dirty exists 2024-09-11T03:44:14.979 INFO:tasks.rados.rados.2.smithi175.stdout:1605: left oid 44 (ObjNum 567 snap 172 seq_num 567) 2024-09-11T03:44:14.979 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:15.000 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:15.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:15.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:15.102 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:15.124 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:15.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:15.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:15.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:15.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:14 smithi175 ceph-mon[30894]: osdmap e436: 12 total, 12 up, 11 in 2024-09-11T03:44:15.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:14 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/918851361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:15.248 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:15.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:15.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:15.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:15.371 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:15.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:15.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:15.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:14 smithi084 ceph-mon[27254]: osdmap e436: 12 total, 12 up, 11 in 2024-09-11T03:44:15.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:14 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/918851361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:15.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:14 smithi121 ceph-mon[30729]: osdmap e436: 12 total, 12 up, 11 in 2024-09-11T03:44:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:14 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/918851361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:15.473 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:15.495 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:15.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:15.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:15.528 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-11T03:44:15.597 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:15.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:15.619 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:15.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:15.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:15.743 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:15.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:15.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:15.772 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:44:15.815 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:44:15.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:15.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:15.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:15.940 INFO:tasks.rados.rados.2.smithi175.stdout:1605: done (1 left) 2024-09-11T03:44:15.940 INFO:tasks.rados.rados.2.smithi175.stdout:1607: done (0 left) 2024-09-11T03:44:15.940 INFO:tasks.rados.rados.2.smithi175.stdout:1608: snap_create 2024-09-11T03:44:15.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:15.990 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:15.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:15.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:16.069 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:16.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:16.112 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:16.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:16.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:16.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:15 smithi084 ceph-mon[27254]: pgmap v810: 57 pgs: 57 active+clean; 158 MiB data, 760 MiB used, 983 GiB / 983 GiB avail; 0 B/s rd, 765 KiB/s wr, 3 op/s 2024-09-11T03:44:16.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:15 smithi084 ceph-mon[27254]: osdmap e437: 12 total, 12 up, 11 in 2024-09-11T03:44:16.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:44:16.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:15 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:44:16.152 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:44:16.153 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:44:16.153 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T03:44:16.153 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 1.000000 2024-09-11T03:44:16.153 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-11T03:44:16.160 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:15 smithi175 ceph-mon[30894]: pgmap v810: 57 pgs: 57 active+clean; 158 MiB data, 760 MiB used, 983 GiB / 983 GiB avail; 0 B/s rd, 765 KiB/s wr, 3 op/s 2024-09-11T03:44:16.161 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:15 smithi175 ceph-mon[30894]: osdmap e437: 12 total, 12 up, 11 in 2024-09-11T03:44:16.161 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:44:16.161 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:15 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:44:16.214 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:16.236 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:16.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:16.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:16.257 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:44:16.257 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:44:16.338 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:16.361 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:16.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:16.364 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:16.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:15 smithi121 ceph-mon[30729]: pgmap v810: 57 pgs: 57 active+clean; 158 MiB data, 760 MiB used, 983 GiB / 983 GiB avail; 0 B/s rd, 765 KiB/s wr, 3 op/s 2024-09-11T03:44:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:15 smithi121 ceph-mon[30729]: osdmap e437: 12 total, 12 up, 11 in 2024-09-11T03:44:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:44:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:15 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:44:16.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:16.487 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:16.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:16.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:16.589 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:16.612 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:16.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:16.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:16.614 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:16.715 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:16.737 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:16.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:16.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:16.770 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:44:16.838 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:16.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:16.862 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:16.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:16.939 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:44:16.944 INFO:tasks.rados.rados.2.smithi175.stdout:1608: done (0 left) 2024-09-11T03:44:16.945 INFO:tasks.rados.rados.2.smithi175.stdout:1609: rollback oid 2 current snap is 174 2024-09-11T03:44:16.945 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 2 to 171 2024-09-11T03:44:16.945 INFO:tasks.rados.rados.2.smithi175.stdout:1610: write oid 28 current snap is 174 2024-09-11T03:44:16.945 INFO:tasks.rados.rados.2.smithi175.stdout:1610: seq_num 569 ranges {763183=626803,1996960=793182,3226271=336912} 2024-09-11T03:44:16.952 INFO:tasks.rados.rados.2.smithi175.stdout:1610: writing smithi17557870-28 from 763183 to 1389986 tid 1 2024-09-11T03:44:16.960 INFO:tasks.rados.rados.2.smithi175.stdout:1610: writing smithi17557870-28 from 1996960 to 2790142 tid 2 2024-09-11T03:44:16.963 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:16.964 INFO:tasks.rados.rados.2.smithi175.stdout:1610: writing smithi17557870-28 from 3226271 to 3563183 tid 3 2024-09-11T03:44:16.964 INFO:tasks.rados.rados.2.smithi175.stdout:1611: write oid 24 current snap is 174 2024-09-11T03:44:16.964 INFO:tasks.rados.rados.2.smithi175.stdout:1611: seq_num 570 ranges {779990=702052,2055442=653193,3253694=326296} 2024-09-11T03:44:16.972 INFO:tasks.rados.rados.2.smithi175.stdout:1611: writing smithi17557870-24 from 779990 to 1482042 tid 1 2024-09-11T03:44:16.979 INFO:tasks.rados.rados.2.smithi175.stdout:1611: writing smithi17557870-24 from 2055442 to 2708635 tid 2 2024-09-11T03:44:16.984 INFO:tasks.rados.rados.2.smithi175.stdout:1611: writing smithi17557870-24 from 3253694 to 3579990 tid 3 2024-09-11T03:44:16.984 INFO:tasks.rados.rados.2.smithi175.stdout:1612: delete oid 46 current snap is 174 2024-09-11T03:44:16.984 INFO:tasks.rados.rados.2.smithi175.stdout:1609: finishing rollback tid 1 to smithi17557870-2 2024-09-11T03:44:16.984 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 529 (ObjNum 1745304224 snap 22076 seq_num 304) dirty dne 2024-09-11T03:44:16.985 INFO:tasks.rados.rados.2.smithi175.stdout:1610: finishing write tid 1 to smithi17557870-28 2024-09-11T03:44:16.985 INFO:tasks.rados.rados.2.smithi175.stdout:1610: finishing write tid 2 to smithi17557870-28 2024-09-11T03:44:16.985 INFO:tasks.rados.rados.2.smithi175.stdout:1610: finishing write tid 3 to smithi17557870-28 2024-09-11T03:44:16.985 INFO:tasks.rados.rados.2.smithi175.stdout:1610: finishing write tid 4 to smithi17557870-28 2024-09-11T03:44:16.985 INFO:tasks.rados.rados.2.smithi175.stdout:1610: finishing write tid 5 to smithi17557870-28 2024-09-11T03:44:16.985 INFO:tasks.rados.rados.2.smithi175.stdout:1610: oid 28 updating version 0 to 586 2024-09-11T03:44:16.985 INFO:tasks.rados.rados.2.smithi175.stdout:1610: oid 28 version 586 is already newer than 584 2024-09-11T03:44:16.985 INFO:tasks.rados.rados.2.smithi175.stdout:1610: oid 28 version 586 is already newer than 583 2024-09-11T03:44:16.985 INFO:tasks.rados.rados.2.smithi175.stdout:1610: oid 28 version 586 is already newer than 585 2024-09-11T03:44:16.985 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 586 (ObjNum 569 snap 174 seq_num 569) dirty exists 2024-09-11T03:44:16.985 INFO:tasks.rados.rados.2.smithi175.stdout:1610: left oid 28 (ObjNum 569 snap 174 seq_num 569) 2024-09-11T03:44:16.985 INFO:tasks.rados.rados.2.smithi175.stdout:1611: finishing write tid 1 to smithi17557870-24 2024-09-11T03:44:16.985 INFO:tasks.rados.rados.2.smithi175.stdout:1611: finishing write tid 2 to smithi17557870-24 2024-09-11T03:44:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:16.987 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:16.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:16.988 INFO:tasks.rados.rados.2.smithi175.stdout:1611: finishing write tid 3 to smithi17557870-24 2024-09-11T03:44:16.988 INFO:tasks.rados.rados.2.smithi175.stdout:1611: finishing write tid 4 to smithi17557870-24 2024-09-11T03:44:16.988 INFO:tasks.rados.rados.2.smithi175.stdout:1611: finishing write tid 5 to smithi17557870-24 2024-09-11T03:44:16.988 INFO:tasks.rados.rados.2.smithi175.stdout:1611: oid 24 updating version 0 to 627 2024-09-11T03:44:16.988 INFO:tasks.rados.rados.2.smithi175.stdout:1611: oid 24 version 627 is already newer than 626 2024-09-11T03:44:16.988 INFO:tasks.rados.rados.2.smithi175.stdout:1611: oid 24 version 627 is already newer than 625 2024-09-11T03:44:16.988 INFO:tasks.rados.rados.2.smithi175.stdout:1611: oid 24 version 627 is already newer than 624 2024-09-11T03:44:16.988 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 627 (ObjNum 570 snap 174 seq_num 570) dirty exists 2024-09-11T03:44:16.988 INFO:tasks.rados.rados.2.smithi175.stdout:1611: left oid 24 (ObjNum 570 snap 174 seq_num 570) 2024-09-11T03:44:16.989 INFO:tasks.rados.rados.2.smithi175.stdout:1609: done (3 left) 2024-09-11T03:44:16.989 INFO:tasks.rados.rados.2.smithi175.stdout:1610: done (2 left) 2024-09-11T03:44:16.989 INFO:tasks.rados.rados.2.smithi175.stdout:1611: done (1 left) 2024-09-11T03:44:16.989 INFO:tasks.rados.rados.2.smithi175.stdout:1612: done (0 left) 2024-09-11T03:44:16.989 INFO:tasks.rados.rados.2.smithi175.stdout:1613: copy_from oid 46 from oid 21 current snap is 174 2024-09-11T03:44:16.989 INFO:tasks.rados.rados.2.smithi175.stdout:1614: read oid 1 snap -1 2024-09-11T03:44:16.989 INFO:tasks.rados.rados.2.smithi175.stdout:1614: expect (ObjNum 520 snap 154 seq_num 520) 2024-09-11T03:44:16.989 INFO:tasks.rados.rados.2.smithi175.stdout:1615: write oid 31 current snap is 174 2024-09-11T03:44:16.990 INFO:tasks.rados.rados.2.smithi175.stdout:1615: seq_num 572 ranges {413604=568903,1488759=656862,2624893=591709} 2024-09-11T03:44:16.994 INFO:tasks.rados.rados.2.smithi175.stdout:1615: writing smithi17557870-31 from 413604 to 982507 tid 1 2024-09-11T03:44:17.000 INFO:tasks.rados.rados.2.smithi175.stdout:1615: writing smithi17557870-31 from 1488759 to 2145621 tid 2 2024-09-11T03:44:17.003 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:16 smithi084 ceph-mon[27254]: osdmap e438: 12 total, 12 up, 11 in 2024-09-11T03:44:17.004 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:44:17.004 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:44:17.004 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:44:17.004 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:16 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/529617913' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:17.006 INFO:tasks.rados.rados.2.smithi175.stdout:1615: writing smithi17557870-31 from 2624893 to 3216602 tid 3 2024-09-11T03:44:17.007 INFO:tasks.rados.rados.2.smithi175.stdout:1616: snap_create 2024-09-11T03:44:17.007 INFO:tasks.rados.rados.2.smithi175.stdout:1614: expect (ObjNum 520 snap 154 seq_num 520) 2024-09-11T03:44:17.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:17.098 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-11T03:44:17.109 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:17.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:17.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:17.122 INFO:tasks.rados.rados.2.smithi175.stdout:1613: finishing copy_from racing read to smithi17557870-46 2024-09-11T03:44:17.122 INFO:tasks.rados.rados.2.smithi175.stdout:1615: finishing write tid 1 to smithi17557870-31 2024-09-11T03:44:17.122 INFO:tasks.rados.rados.2.smithi175.stdout:1613: finishing copy_from to smithi17557870-46 2024-09-11T03:44:17.123 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 523 (ObjNum 533 snap 158 seq_num 533) dirty exists 2024-09-11T03:44:17.123 INFO:tasks.rados.rados.2.smithi175.stdout:1615: finishing write tid 2 to smithi17557870-31 2024-09-11T03:44:17.123 INFO:tasks.rados.rados.2.smithi175.stdout:1615: finishing write tid 3 to smithi17557870-31 2024-09-11T03:44:17.123 INFO:tasks.rados.rados.2.smithi175.stdout:1615: finishing write tid 4 to smithi17557870-31 2024-09-11T03:44:17.123 INFO:tasks.rados.rados.2.smithi175.stdout:1615: finishing write tid 5 to smithi17557870-31 2024-09-11T03:44:17.123 INFO:tasks.rados.rados.2.smithi175.stdout:1615: oid 31 updating version 0 to 572 2024-09-11T03:44:17.123 INFO:tasks.rados.rados.2.smithi175.stdout:1615: oid 31 version 572 is already newer than 571 2024-09-11T03:44:17.123 INFO:tasks.rados.rados.2.smithi175.stdout:1615: oid 31 version 572 is already newer than 570 2024-09-11T03:44:17.123 INFO:tasks.rados.rados.2.smithi175.stdout:1615: oid 31 updating version 572 to 573 2024-09-11T03:44:17.123 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 573 (ObjNum 572 snap 174 seq_num 572) dirty exists 2024-09-11T03:44:17.123 INFO:tasks.rados.rados.2.smithi175.stdout:1615: left oid 31 (ObjNum 572 snap 174 seq_num 572) 2024-09-11T03:44:17.211 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:16 smithi175 ceph-mon[30894]: osdmap e438: 12 total, 12 up, 11 in 2024-09-11T03:44:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:44:17.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:44:17.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:44:17.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:16 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/529617913' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:17.234 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:17.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:17.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:17.361 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:17.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:17.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:17.365 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:44:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:16 smithi121 ceph-mon[30729]: osdmap e438: 12 total, 12 up, 11 in 2024-09-11T03:44:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:44:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:44:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:44:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:16 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/529617913' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:17.464 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:17.486 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:17.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:17.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:17.614 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:17.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:17.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:17.716 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:17.742 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:17.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:17.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:17.796 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:44:17.797 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:44:17.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:17.868 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:17.869 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:17.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:17.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:17.953 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-11T03:44:17.959 INFO:tasks.rados.rados.2.smithi175.stdout:1613: done (3 left) 2024-09-11T03:44:17.959 INFO:tasks.rados.rados.2.smithi175.stdout:1614: done (2 left) 2024-09-11T03:44:17.959 INFO:tasks.rados.rados.2.smithi175.stdout:1615: done (1 left) 2024-09-11T03:44:17.959 INFO:tasks.rados.rados.2.smithi175.stdout:1616: done (0 left) 2024-09-11T03:44:17.959 INFO:tasks.rados.rados.2.smithi175.stdout:1617: copy_from oid 32 from oid 16 current snap is 175 2024-09-11T03:44:17.959 INFO:tasks.rados.rados.2.smithi175.stdout:1618: delete oid 50 current snap is 175 2024-09-11T03:44:17.962 INFO:tasks.rados.rados.2.smithi175.stdout:1618: done (1 left) 2024-09-11T03:44:17.962 INFO:tasks.rados.rados.2.smithi175.stdout:1619: write (excl) oid 43 current snap is 175 2024-09-11T03:44:17.962 INFO:tasks.rados.rados.2.smithi175.stdout:1619: seq_num 574 ranges {447218=719401,1889370=660531,2963386=614253} 2024-09-11T03:44:17.969 INFO:tasks.rados.rados.2.smithi175.stdout:1619: writing smithi17557870-43 from 447218 to 1166619 tid 1 2024-09-11T03:44:17.971 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:17.976 INFO:tasks.rados.rados.2.smithi175.stdout:1619: writing smithi17557870-43 from 1889370 to 2549901 tid 2 2024-09-11T03:44:17.982 INFO:tasks.rados.rados.2.smithi175.stdout:1619: writing smithi17557870-43 from 2963386 to 3577639 tid 3 2024-09-11T03:44:17.982 INFO:tasks.rados.rados.2.smithi175.stdout:1620: delete oid 29 current snap is 175 2024-09-11T03:44:17.982 INFO:tasks.rados.rados.2.smithi175.stdout:1617: finishing copy_from racing read to smithi17557870-32 2024-09-11T03:44:17.983 INFO:tasks.rados.rados.2.smithi175.stdout:1617: finishing copy_from to smithi17557870-32 2024-09-11T03:44:17.983 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 461 (ObjNum 509 snap 150 seq_num 509) dirty exists 2024-09-11T03:44:17.983 INFO:tasks.rados.rados.2.smithi175.stdout:1619: finishing write tid 1 to smithi17557870-43 2024-09-11T03:44:17.983 INFO:tasks.rados.rados.2.smithi175.stdout:1619: finishing write tid 2 to smithi17557870-43 2024-09-11T03:44:17.984 INFO:tasks.rados.rados.2.smithi175.stdout:1617: done (2 left) 2024-09-11T03:44:17.984 INFO:tasks.rados.rados.2.smithi175.stdout:1620: done (1 left) 2024-09-11T03:44:17.984 INFO:tasks.rados.rados.2.smithi175.stdout:1621: snap_create 2024-09-11T03:44:17.985 INFO:tasks.rados.rados.2.smithi175.stdout:1619: finishing write tid 3 to smithi17557870-43 2024-09-11T03:44:17.985 INFO:tasks.rados.rados.2.smithi175.stdout:1619: finishing write tid 4 to smithi17557870-43 2024-09-11T03:44:17.985 INFO:tasks.rados.rados.2.smithi175.stdout:1619: finishing write tid 5 to smithi17557870-43 2024-09-11T03:44:17.985 INFO:tasks.rados.rados.2.smithi175.stdout:1619: oid 43 updating version 0 to 398 2024-09-11T03:44:17.985 INFO:tasks.rados.rados.2.smithi175.stdout:1619: oid 43 updating version 398 to 401 2024-09-11T03:44:17.985 INFO:tasks.rados.rados.2.smithi175.stdout:1619: oid 43 version 401 is already newer than 399 2024-09-11T03:44:17.985 INFO:tasks.rados.rados.2.smithi175.stdout:1619: oid 43 version 401 is already newer than 400 2024-09-11T03:44:17.986 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 401 (ObjNum 574 snap 175 seq_num 574) dirty exists 2024-09-11T03:44:17.986 INFO:tasks.rados.rados.2.smithi175.stdout:1619: left oid 43 (ObjNum 574 snap 175 seq_num 574) 2024-09-11T03:44:17.994 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:17.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:17.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:18.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:17 smithi084 ceph-mon[27254]: pgmap v813: 57 pgs: 57 active+clean; 161 MiB data, 771 MiB used, 983 GiB / 983 GiB avail; 1.7 KiB/s rd, 3.3 MiB/s wr, 12 op/s 2024-09-11T03:44:18.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/529617913' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:18.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:17 smithi084 ceph-mon[27254]: osdmap e439: 12 total, 12 up, 11 in 2024-09-11T03:44:18.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2056896357' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:44:18.096 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:18.117 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:18.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:18.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:18.127 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:17 smithi175 ceph-mon[30894]: pgmap v813: 57 pgs: 57 active+clean; 161 MiB data, 771 MiB used, 983 GiB / 983 GiB avail; 1.7 KiB/s rd, 3.3 MiB/s wr, 12 op/s 2024-09-11T03:44:18.128 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/529617913' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:18.128 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:17 smithi175 ceph-mon[30894]: osdmap e439: 12 total, 12 up, 11 in 2024-09-11T03:44:18.128 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2056896357' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:44:18.219 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:18.241 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:18.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:18.343 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:18.365 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:18.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:18.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:17 smithi121 ceph-mon[30729]: pgmap v813: 57 pgs: 57 active+clean; 161 MiB data, 771 MiB used, 983 GiB / 983 GiB avail; 1.7 KiB/s rd, 3.3 MiB/s wr, 12 op/s 2024-09-11T03:44:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/529617913' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:17 smithi121 ceph-mon[30729]: osdmap e439: 12 total, 12 up, 11 in 2024-09-11T03:44:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2056896357' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:44:18.438 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-11T03:44:18.466 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:18.503 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:18.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:18.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:18.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:18.626 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:18.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:18.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:18.681 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:44:18.728 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:18.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:18.750 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:18.751 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:18.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:18.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:18.879 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:18.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:18.968 INFO:tasks.rados.rados.2.smithi175.stdout:1619: done (1 left) 2024-09-11T03:44:18.968 INFO:tasks.rados.rados.2.smithi175.stdout:1621: done (0 left) 2024-09-11T03:44:18.968 INFO:tasks.rados.rados.2.smithi175.stdout:1622: rollback oid 22 current snap is 176 2024-09-11T03:44:18.969 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 22 to 175 2024-09-11T03:44:18.969 INFO:tasks.rados.rados.2.smithi175.stdout:1623: snap_remove snap 156 2024-09-11T03:44:18.979 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:19.001 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:19.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:19.002 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:19.016 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:44:19.016 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:44:19.031 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:44:19.032 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:44:19.047 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:44:19.047 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:44:19.063 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:44:19.063 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:44:19.073 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:18 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:44:19.073 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:18 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:44:19.073 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:18 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2056896357' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:44:19.073 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:18 smithi175 ceph-mon[30894]: osdmap e440: 12 total, 12 up, 11 in 2024-09-11T03:44:19.081 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:44:19.081 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:44:19.098 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:44:19.098 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:44:19.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:19.115 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:44:19.115 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:44:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:19.130 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:19.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:19.131 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:44:19.132 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:44:19.148 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:44:19.148 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:44:19.164 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:44:19.165 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:44:19.173 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:44:19.173 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:44:19.174 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:44:19.174 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:44:19.174 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.174 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:847 2.d 2:b1c0f730:::smithi17557870-20:head [copy-from ver 90] snapc 1a=[1a,19] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-11T03:44:19.174 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:44:19.174 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 294.33345530399998, 2024-09-11T03:44:19.175 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013054913, 2024-09-11T03:44:19.175 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.175 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.175 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:19.175 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.175 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.176 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.176 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 847 2024-09-11T03:44:19.176 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.176 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.176 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.177 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.177 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:44:19.177 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.177 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.177 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.177 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.178 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:44:19.178 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.178 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.178 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.178 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.178 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839297+0000", 2024-09-11T03:44:19.178 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0049999999999999e-06 2024-09-11T03:44:19.178 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.179 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.179 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.179 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839309+0000", 2024-09-11T03:44:19.179 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1531e-05 2024-09-11T03:44:19.179 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.180 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.180 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.180 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839326+0000", 2024-09-11T03:44:19.180 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7788e-05 2024-09-11T03:44:19.180 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.180 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.181 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.181 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839336+0000", 2024-09-11T03:44:19.181 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5619999999999993e-06 2024-09-11T03:44:19.181 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.181 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.181 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.181 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839423+0000", 2024-09-11T03:44:19.181 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6790000000000001e-05 2024-09-11T03:44:19.181 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.181 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.181 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:44:19.182 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839434+0000", 2024-09-11T03:44:19.182 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0971e-05 2024-09-11T03:44:19.182 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.182 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.182 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.182 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841520+0000", 2024-09-11T03:44:19.182 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0020866299999999999 2024-09-11T03:44:19.183 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.183 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.183 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.183 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841590+0000", 2024-09-11T03:44:19.183 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9288000000000005e-05 2024-09-11T03:44:19.183 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.183 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.183 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:44:19.183 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.846430+0000", 2024-09-11T03:44:19.184 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0048408349999999999 2024-09-11T03:44:19.184 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.184 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.184 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:19.184 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849726+0000", 2024-09-11T03:44:19.184 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032955390000000001 2024-09-11T03:44:19.185 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.185 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.185 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.185 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852240+0000", 2024-09-11T03:44:19.185 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002513492 2024-09-11T03:44:19.185 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.185 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.185 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.185 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852278+0000", 2024-09-11T03:44:19.185 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7988999999999998e-05 2024-09-11T03:44:19.186 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.186 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.186 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:19.186 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852317+0000", 2024-09-11T03:44:19.186 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9310000000000001e-05 2024-09-11T03:44:19.186 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.187 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.187 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.187 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852350+0000", 2024-09-11T03:44:19.187 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3182999999999997e-05 2024-09-11T03:44:19.187 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.187 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.187 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.187 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.187 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.187 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:848 2.d 2:b1c0f730:::smithi17557870-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2024-09-11T03:44:19.187 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:24.839354+0000", 2024-09-11T03:44:19.187 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 294.33339590700001, 2024-09-11T03:44:19.187 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010055476000000001, 2024-09-11T03:44:19.188 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.188 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.188 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:44:19.188 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.188 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.188 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.188 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 848 2024-09-11T03:44:19.188 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.188 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.188 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839354+0000", 2024-09-11T03:44:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839354+0000", 2024-09-11T03:44:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839355+0000", 2024-09-11T03:44:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6899999999999996e-07 2024-09-11T03:44:19.190 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.190 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.190 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.190 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839358+0000", 2024-09-11T03:44:19.190 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8729999999999999e-06 2024-09-11T03:44:19.190 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.190 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.190 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839367+0000", 2024-09-11T03:44:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3999999999999992e-06 2024-09-11T03:44:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839371+0000", 2024-09-11T03:44:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6059999999999996e-06 2024-09-11T03:44:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.192 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839509+0000", 2024-09-11T03:44:19.192 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013766700000000001 2024-09-11T03:44:19.192 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.192 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.192 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:19.192 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839515+0000", 2024-09-11T03:44:19.193 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4740000000000002e-06 2024-09-11T03:44:19.193 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.193 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.193 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.193 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841883+0000", 2024-09-11T03:44:19.193 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023678319999999998 2024-09-11T03:44:19.194 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.194 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.194 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:19.194 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841941+0000", 2024-09-11T03:44:19.194 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8241e-05 2024-09-11T03:44:19.194 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.195 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.195 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.195 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849208+0000", 2024-09-11T03:44:19.195 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0072661690000000003 2024-09-11T03:44:19.195 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.196 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.196 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.196 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849295+0000", 2024-09-11T03:44:19.196 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7582000000000003e-05 2024-09-11T03:44:19.196 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.196 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.196 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.196 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849410+0000", 2024-09-11T03:44:19.196 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114863 2024-09-11T03:44:19.197 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.197 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.197 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.197 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.197 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.197 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:907 2.d 2:b6a25a79:::smithi17557870-47:head [copy-from ver 47] snapc 20=[20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-11T03:44:19.198 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:34.902174+0000", 2024-09-11T03:44:19.198 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 284.27057684900001, 2024-09-11T03:44:19.198 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011234144, 2024-09-11T03:44:19.198 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.198 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.198 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:19.199 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:44:19.200 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:44:19.200 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:44:19.201 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:44:19.201 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.201 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.201 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.201 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 907 2024-09-11T03:44:19.201 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.201 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.201 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.201 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.202 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902174+0000", 2024-09-11T03:44:19.202 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.202 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.202 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.202 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.202 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902174+0000", 2024-09-11T03:44:19.202 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.202 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.202 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.202 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.202 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902175+0000", 2024-09-11T03:44:19.202 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0189999999999999e-06 2024-09-11T03:44:19.202 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.203 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.203 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.203 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902178+0000", 2024-09-11T03:44:19.203 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4410000000000002e-06 2024-09-11T03:44:19.203 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.203 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.203 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.203 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902188+0000", 2024-09-11T03:44:19.203 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0023999999999999e-05 2024-09-11T03:44:19.203 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.203 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.203 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.203 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902193+0000", 2024-09-11T03:44:19.203 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.31e-06 2024-09-11T03:44:19.204 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.204 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.204 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.204 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902248+0000", 2024-09-11T03:44:19.204 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4562999999999999e-05 2024-09-11T03:44:19.204 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.204 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.204 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.204 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902344+0000", 2024-09-11T03:44:19.204 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5841999999999995e-05 2024-09-11T03:44:19.204 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.204 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.204 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:44:19.205 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.907645+0000", 2024-09-11T03:44:19.205 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0053015579999999996 2024-09-11T03:44:19.205 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.205 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.205 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:19.205 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.909768+0000", 2024-09-11T03:44:19.205 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0021225020000000001 2024-09-11T03:44:19.205 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.205 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.205 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.205 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.912827+0000", 2024-09-11T03:44:19.205 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003059375 2024-09-11T03:44:19.205 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.205 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.206 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.206 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.913334+0000", 2024-09-11T03:44:19.206 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00050710499999999999 2024-09-11T03:44:19.206 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.206 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.206 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:19.206 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.913372+0000", 2024-09-11T03:44:19.206 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7486e-05 2024-09-11T03:44:19.206 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.206 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.206 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.206 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.913408+0000", 2024-09-11T03:44:19.206 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5918999999999999e-05 2024-09-11T03:44:19.207 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.207 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.207 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.207 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.207 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.207 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1290 2.d 2:b1c0f730:::smithi17557870-20:head [copy-from ver 48] snapc 33=[32,30,2c,2b] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:44:19.207 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:12.836022+0000", 2024-09-11T03:44:19.207 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 246.33672802300001, 2024-09-11T03:44:19.207 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022892863999999999, 2024-09-11T03:44:19.207 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.207 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.207 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:19.207 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.207 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.208 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.208 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1290 2024-09-11T03:44:19.208 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.208 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.208 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.208 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.208 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836022+0000", 2024-09-11T03:44:19.208 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.208 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.208 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.208 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.208 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836022+0000", 2024-09-11T03:44:19.208 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.208 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.209 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.209 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.209 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836025+0000", 2024-09-11T03:44:19.209 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6690000000000002e-06 2024-09-11T03:44:19.209 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.209 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.209 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.209 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836033+0000", 2024-09-11T03:44:19.209 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0809999999999994e-06 2024-09-11T03:44:19.209 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.209 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.209 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.209 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836053+0000", 2024-09-11T03:44:19.210 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9429999999999999e-05 2024-09-11T03:44:19.210 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.210 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.210 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.210 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836063+0000", 2024-09-11T03:44:19.210 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0438000000000001e-05 2024-09-11T03:44:19.210 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.210 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.210 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.210 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836165+0000", 2024-09-11T03:44:19.210 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000101523 2024-09-11T03:44:19.210 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.210 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.210 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:44:19.211 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836172+0000", 2024-09-11T03:44:19.211 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6499999999999996e-06 2024-09-11T03:44:19.211 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.211 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.211 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.211 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839394+0000", 2024-09-11T03:44:19.211 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032215590000000001 2024-09-11T03:44:19.211 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.211 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.211 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.211 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839507+0000", 2024-09-11T03:44:19.211 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000113557 2024-09-11T03:44:19.211 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.211 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.212 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:44:19.212 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.849905+0000", 2024-09-11T03:44:19.212 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010398174 2024-09-11T03:44:19.212 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.212 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.212 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:19.212 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.853428+0000", 2024-09-11T03:44:19.212 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035222869999999998 2024-09-11T03:44:19.212 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.212 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.212 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.212 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.855923+0000", 2024-09-11T03:44:19.212 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024951840000000001 2024-09-11T03:44:19.212 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.213 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.213 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.213 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.858781+0000", 2024-09-11T03:44:19.213 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002858368 2024-09-11T03:44:19.213 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.213 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.213 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:19.213 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.858881+0000", 2024-09-11T03:44:19.213 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9500999999999995e-05 2024-09-11T03:44:19.213 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.213 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.213 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.213 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.858915+0000", 2024-09-11T03:44:19.214 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4443000000000002e-05 2024-09-11T03:44:19.214 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.214 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.214 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.214 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.214 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.214 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1291 2.d 2:b1c0f730:::smithi17557870-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:44:19.214 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:12.836223+0000", 2024-09-11T03:44:19.214 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 246.33652727699999, 2024-09-11T03:44:19.214 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016933050000000002, 2024-09-11T03:44:19.214 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.214 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.214 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:44:19.214 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.215 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.215 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.215 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1291 2024-09-11T03:44:19.215 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.215 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.215 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.215 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.215 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836223+0000", 2024-09-11T03:44:19.215 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.215 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.215 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.215 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.215 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836223+0000", 2024-09-11T03:44:19.215 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.216 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.216 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.216 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.216 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836224+0000", 2024-09-11T03:44:19.216 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0020000000000001e-06 2024-09-11T03:44:19.216 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.216 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.216 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.216 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836229+0000", 2024-09-11T03:44:19.216 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7180000000000004e-06 2024-09-11T03:44:19.216 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.216 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.216 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.217 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836245+0000", 2024-09-11T03:44:19.217 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5981000000000002e-05 2024-09-11T03:44:19.217 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.217 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.217 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.217 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836253+0000", 2024-09-11T03:44:19.217 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1720000000000007e-06 2024-09-11T03:44:19.217 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.217 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.217 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.217 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836317+0000", 2024-09-11T03:44:19.217 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4189999999999994e-05 2024-09-11T03:44:19.217 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.217 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.218 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:19.218 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836324+0000", 2024-09-11T03:44:19.218 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9249999999999998e-06 2024-09-11T03:44:19.218 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.218 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.218 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.218 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839738+0000", 2024-09-11T03:44:19.218 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0034135580000000001 2024-09-11T03:44:19.218 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.218 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.218 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:19.218 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839804+0000", 2024-09-11T03:44:19.219 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5974000000000002e-05 2024-09-11T03:44:19.219 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.219 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.219 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.219 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.852933+0000", 2024-09-11T03:44:19.219 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013129429999999999 2024-09-11T03:44:19.219 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.219 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.219 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.219 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.853032+0000", 2024-09-11T03:44:19.219 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8592999999999996e-05 2024-09-11T03:44:19.219 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.219 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.220 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.220 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.853156+0000", 2024-09-11T03:44:19.220 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012450699999999999 2024-09-11T03:44:19.220 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.220 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.220 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.220 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.220 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.220 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1440 2.d 2:b6a25a79:::smithi17557870-47:35 [read 0~3042067,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e206)", 2024-09-11T03:44:19.220 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:22.839838+0000", 2024-09-11T03:44:19.220 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 236.33291208200001, 2024-09-11T03:44:19.220 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0089542019999999996, 2024-09-11T03:44:19.220 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.221 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.221 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:44:19.221 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.221 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.221 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.221 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1440 2024-09-11T03:44:19.221 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.221 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.221 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.221 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.221 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839838+0000", 2024-09-11T03:44:19.221 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.221 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.221 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.222 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.222 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839838+0000", 2024-09-11T03:44:19.222 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.222 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.222 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.222 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.222 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839839+0000", 2024-09-11T03:44:19.222 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9500000000000002e-07 2024-09-11T03:44:19.222 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.222 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.222 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.222 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839843+0000", 2024-09-11T03:44:19.222 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3200000000000001e-06 2024-09-11T03:44:19.222 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.223 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.223 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.223 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839851+0000", 2024-09-11T03:44:19.223 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5170000000000002e-06 2024-09-11T03:44:19.223 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.223 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.223 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.223 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839856+0000", 2024-09-11T03:44:19.223 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5799999999999999e-06 2024-09-11T03:44:19.223 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.223 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.223 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.223 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.840764+0000", 2024-09-11T03:44:19.223 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00090780899999999998 2024-09-11T03:44:19.224 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.224 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.224 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:44:19.224 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.840857+0000", 2024-09-11T03:44:19.224 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2707999999999994e-05 2024-09-11T03:44:19.224 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.224 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.224 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.224 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.847171+0000", 2024-09-11T03:44:19.224 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0063140920000000003 2024-09-11T03:44:19.224 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.224 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.224 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.225 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.847241+0000", 2024-09-11T03:44:19.225 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0392e-05 2024-09-11T03:44:19.225 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.225 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.225 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.225 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.848793+0000", 2024-09-11T03:44:19.225 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015510890000000001 2024-09-11T03:44:19.225 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.225 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.225 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.225 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.225 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.225 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1441 2.d 2:b6a25a79:::smithi17557870-47:35 [sparse-read 0~1600344] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e206)", 2024-09-11T03:44:19.225 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:22.839865+0000", 2024-09-11T03:44:19.226 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 236.33288505799999, 2024-09-11T03:44:19.226 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0091540579999999996, 2024-09-11T03:44:19.226 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.226 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.226 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:44:19.226 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.226 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.226 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.226 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1441 2024-09-11T03:44:19.226 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.226 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.226 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.226 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.226 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839865+0000", 2024-09-11T03:44:19.227 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.227 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.227 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.227 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.227 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839865+0000", 2024-09-11T03:44:19.227 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.227 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.227 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.227 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.227 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839866+0000", 2024-09-11T03:44:19.227 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4499999999999997e-07 2024-09-11T03:44:19.227 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.227 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.227 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.227 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839869+0000", 2024-09-11T03:44:19.228 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.055e-06 2024-09-11T03:44:19.228 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.228 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.228 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.228 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839875+0000", 2024-09-11T03:44:19.228 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3319999999999997e-06 2024-09-11T03:44:19.228 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.228 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.228 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.228 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.839880+0000", 2024-09-11T03:44:19.228 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0340000000000004e-06 2024-09-11T03:44:19.228 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.228 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.229 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.229 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.841228+0000", 2024-09-11T03:44:19.229 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001347521 2024-09-11T03:44:19.229 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.229 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.229 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:44:19.232 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.841306+0000", 2024-09-11T03:44:19.232 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8182000000000005e-05 2024-09-11T03:44:19.232 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.232 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.232 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.232 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.848858+0000", 2024-09-11T03:44:19.232 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0075521039999999996 2024-09-11T03:44:19.232 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.232 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.233 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.233 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.848919+0000", 2024-09-11T03:44:19.233 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.05e-05 2024-09-11T03:44:19.233 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.233 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.233 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.233 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:22.849019+0000", 2024-09-11T03:44:19.233 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000100785 2024-09-11T03:44:19.233 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.233 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.233 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.233 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.233 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.234 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1516 2.d 2:b6a25a79:::smithi17557870-47:head [copy-from ver 97] snapc 3b=[3a,38,37,35] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-09-11T03:44:19.234 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:28.919589+0000", 2024-09-11T03:44:19.234 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 230.25316114699999, 2024-09-11T03:44:19.234 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011956165, 2024-09-11T03:44:19.234 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.234 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.234 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:19.234 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.234 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.234 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.234 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1516 2024-09-11T03:44:19.234 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.234 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.234 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.235 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.235 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919589+0000", 2024-09-11T03:44:19.235 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.235 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.235 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.235 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.235 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919589+0000", 2024-09-11T03:44:19.235 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.235 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.235 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.235 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.235 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919591+0000", 2024-09-11T03:44:19.235 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5409999999999999e-06 2024-09-11T03:44:19.235 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.236 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.236 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.236 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919596+0000", 2024-09-11T03:44:19.236 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9280000000000001e-06 2024-09-11T03:44:19.236 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.236 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.236 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.236 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919610+0000", 2024-09-11T03:44:19.236 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4285e-05 2024-09-11T03:44:19.236 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.236 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.236 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.236 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919617+0000", 2024-09-11T03:44:19.237 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.435e-06 2024-09-11T03:44:19.237 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.237 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.237 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.237 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919688+0000", 2024-09-11T03:44:19.237 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0263000000000001e-05 2024-09-11T03:44:19.237 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.237 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.237 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.237 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919794+0000", 2024-09-11T03:44:19.237 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000106055 2024-09-11T03:44:19.237 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.237 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.237 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:44:19.238 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.922849+0000", 2024-09-11T03:44:19.238 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030557330000000001 2024-09-11T03:44:19.238 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.238 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.238 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:19.238 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.926556+0000", 2024-09-11T03:44:19.238 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0037066669999999999 2024-09-11T03:44:19.238 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.239 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.239 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.239 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.929197+0000", 2024-09-11T03:44:19.239 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002640529 2024-09-11T03:44:19.239 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.239 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.239 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.239 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.931432+0000", 2024-09-11T03:44:19.239 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022350339999999999 2024-09-11T03:44:19.239 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.239 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.239 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:19.239 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.931501+0000", 2024-09-11T03:44:19.239 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9584000000000003e-05 2024-09-11T03:44:19.240 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.240 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.240 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.240 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.931545+0000", 2024-09-11T03:44:19.240 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4110999999999998e-05 2024-09-11T03:44:19.240 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.240 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.240 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.240 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.240 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.241 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2254 2.18 2:1984a3a9:::smithi17557870-23:head [write 746310~635349,stat] snapc 5d=[5d,5a,59,58,56] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:44:19.241 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.883765+0000", 2024-09-11T03:44:19.241 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 162.28898503400001, 2024-09-11T03:44:19.241 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022154396, 2024-09-11T03:44:19.241 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.241 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.241 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:19.241 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.241 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.241 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.241 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2254 2024-09-11T03:44:19.241 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.241 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.241 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.242 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.242 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.883765+0000", 2024-09-11T03:44:19.242 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.242 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.242 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.242 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.242 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.883765+0000", 2024-09-11T03:44:19.242 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.242 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.242 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.242 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.242 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.883768+0000", 2024-09-11T03:44:19.242 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4080000000000001e-06 2024-09-11T03:44:19.243 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.243 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.243 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.243 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884378+0000", 2024-09-11T03:44:19.243 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00061034200000000005 2024-09-11T03:44:19.243 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.243 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.243 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.243 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884399+0000", 2024-09-11T03:44:19.243 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1175e-05 2024-09-11T03:44:19.243 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.243 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.243 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.243 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884412+0000", 2024-09-11T03:44:19.244 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3015e-05 2024-09-11T03:44:19.244 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.244 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.244 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.244 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884520+0000", 2024-09-11T03:44:19.244 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000107706 2024-09-11T03:44:19.244 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.244 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.244 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:44:19.244 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884534+0000", 2024-09-11T03:44:19.244 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3923e-05 2024-09-11T03:44:19.244 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.244 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.245 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.245 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.895937+0000", 2024-09-11T03:44:19.245 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011403207 2024-09-11T03:44:19.245 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.245 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.245 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:44:19.245 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896424+0000", 2024-09-11T03:44:19.245 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00048641199999999998 2024-09-11T03:44:19.245 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.245 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.245 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.245 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.898138+0000", 2024-09-11T03:44:19.245 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001714272 2024-09-11T03:44:19.245 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.246 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.246 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.246 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.898309+0000", 2024-09-11T03:44:19.246 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017075200000000001 2024-09-11T03:44:19.246 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.246 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.246 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:44:19.246 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.898561+0000", 2024-09-11T03:44:19.246 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00025223500000000001 2024-09-11T03:44:19.246 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.246 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.246 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.246 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903048+0000", 2024-09-11T03:44:19.246 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0044876320000000001 2024-09-11T03:44:19.247 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.247 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.247 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.247 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903118+0000", 2024-09-11T03:44:19.247 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9608000000000002e-05 2024-09-11T03:44:19.247 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.247 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.247 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:19.247 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905694+0000", 2024-09-11T03:44:19.247 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025761650000000001 2024-09-11T03:44:19.247 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.247 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.247 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:19.248 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905749+0000", 2024-09-11T03:44:19.248 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5037999999999997e-05 2024-09-11T03:44:19.248 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.248 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.248 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.248 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905920+0000", 2024-09-11T03:44:19.248 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017050600000000001 2024-09-11T03:44:19.248 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.248 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.248 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.248 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.248 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.248 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2255 2.18 2:1984a3a9:::smithi17557870-23:head [write 2077646~576443] snapc 5d=[5d,5a,59,58,56] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:44:19.249 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.884463+0000", 2024-09-11T03:44:19.249 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 162.288287303, 2024-09-11T03:44:19.249 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022426167, 2024-09-11T03:44:19.249 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.249 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.249 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:19.249 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.249 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.249 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.249 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2255 2024-09-11T03:44:19.249 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.249 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.249 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.250 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.250 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884463+0000", 2024-09-11T03:44:19.250 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.250 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.250 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.250 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.250 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884463+0000", 2024-09-11T03:44:19.250 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.250 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.250 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.250 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.250 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884464+0000", 2024-09-11T03:44:19.250 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.64e-07 2024-09-11T03:44:19.250 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.251 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.251 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.251 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885204+0000", 2024-09-11T03:44:19.251 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00074007200000000004 2024-09-11T03:44:19.251 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.251 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.251 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.251 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885217+0000", 2024-09-11T03:44:19.251 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2547e-05 2024-09-11T03:44:19.251 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.251 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.251 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.251 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885226+0000", 2024-09-11T03:44:19.252 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1849999999999993e-06 2024-09-11T03:44:19.252 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.252 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.252 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.252 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885295+0000", 2024-09-11T03:44:19.252 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8657000000000004e-05 2024-09-11T03:44:19.252 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.252 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.252 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:44:19.252 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885307+0000", 2024-09-11T03:44:19.252 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2529e-05 2024-09-11T03:44:19.252 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.252 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.253 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.253 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896570+0000", 2024-09-11T03:44:19.253 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011262529 2024-09-11T03:44:19.253 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.253 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.253 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:44:19.253 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896624+0000", 2024-09-11T03:44:19.253 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4614000000000003e-05 2024-09-11T03:44:19.253 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.253 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.253 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.253 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.900547+0000", 2024-09-11T03:44:19.254 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039228850000000001 2024-09-11T03:44:19.254 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.254 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.254 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.254 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.900706+0000", 2024-09-11T03:44:19.254 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015907000000000001 2024-09-11T03:44:19.254 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.254 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.254 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:44:19.254 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.900901+0000", 2024-09-11T03:44:19.254 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00019514399999999999 2024-09-11T03:44:19.255 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.255 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.255 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.255 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903192+0000", 2024-09-11T03:44:19.255 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022906519999999998 2024-09-11T03:44:19.255 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.255 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.255 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.255 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.904724+0000", 2024-09-11T03:44:19.255 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015319520000000001 2024-09-11T03:44:19.255 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.255 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.255 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:19.255 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906254+0000", 2024-09-11T03:44:19.256 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015302759999999999 2024-09-11T03:44:19.256 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.256 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.256 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:19.256 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906358+0000", 2024-09-11T03:44:19.256 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010404300000000001 2024-09-11T03:44:19.256 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.256 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.256 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.256 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906889+0000", 2024-09-11T03:44:19.256 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000531048 2024-09-11T03:44:19.256 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.256 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.257 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.257 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.257 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.257 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2256 2.18 2:1984a3a9:::smithi17557870-23:head [write 3362764~183546] snapc 5d=[5d,5a,59,58,56] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:44:19.257 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.885408+0000", 2024-09-11T03:44:19.257 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 162.28734196799999, 2024-09-11T03:44:19.257 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021926139000000001, 2024-09-11T03:44:19.257 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.257 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.257 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:19.257 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.257 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.257 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.258 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2256 2024-09-11T03:44:19.258 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.258 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.258 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.258 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.258 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885408+0000", 2024-09-11T03:44:19.258 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.258 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.258 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.258 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.258 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885408+0000", 2024-09-11T03:44:19.258 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.258 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.258 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.258 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.259 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885410+0000", 2024-09-11T03:44:19.259 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4270000000000001e-06 2024-09-11T03:44:19.259 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.259 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.259 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.259 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885561+0000", 2024-09-11T03:44:19.259 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000151254 2024-09-11T03:44:19.259 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.259 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.259 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.259 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885573+0000", 2024-09-11T03:44:19.259 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1897e-05 2024-09-11T03:44:19.259 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.259 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.260 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.260 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885581+0000", 2024-09-11T03:44:19.260 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6290000000000001e-06 2024-09-11T03:44:19.260 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.260 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.260 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.260 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885643+0000", 2024-09-11T03:44:19.260 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2215e-05 2024-09-11T03:44:19.260 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.260 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.260 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:44:19.260 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885652+0000", 2024-09-11T03:44:19.260 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5580000000000006e-06 2024-09-11T03:44:19.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.261 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.261 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896678+0000", 2024-09-11T03:44:19.261 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011025956 2024-09-11T03:44:19.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.261 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:44:19.261 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896729+0000", 2024-09-11T03:44:19.261 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1019000000000001e-05 2024-09-11T03:44:19.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.261 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.262 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903250+0000", 2024-09-11T03:44:19.262 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0065205109999999997 2024-09-11T03:44:19.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.262 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.262 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903415+0000", 2024-09-11T03:44:19.262 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016545399999999999 2024-09-11T03:44:19.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.262 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:44:19.262 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903701+0000", 2024-09-11T03:44:19.262 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00028586399999999999 2024-09-11T03:44:19.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.263 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.263 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905681+0000", 2024-09-11T03:44:19.263 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0019798950000000002 2024-09-11T03:44:19.263 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.263 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.263 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.263 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905999+0000", 2024-09-11T03:44:19.263 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00031789999999999998 2024-09-11T03:44:19.263 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.263 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.263 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:19.263 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906904+0000", 2024-09-11T03:44:19.263 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00090535599999999996 2024-09-11T03:44:19.263 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.264 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.264 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:19.264 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907002+0000", 2024-09-11T03:44:19.264 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7653999999999997e-05 2024-09-11T03:44:19.264 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.264 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.264 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.264 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907335+0000", 2024-09-11T03:44:19.264 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00033254999999999998 2024-09-11T03:44:19.264 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.264 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.264 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.264 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.265 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.265 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2257 2.18 2:1984a3a9:::smithi17557870-23:head [setxattr _header (57) in=7b,truncate 3546310] snapc 5d=[5d,5a,59,58,56] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:44:19.265 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.885667+0000", 2024-09-11T03:44:19.265 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 162.28708294699999, 2024-09-11T03:44:19.265 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022144878999999999, 2024-09-11T03:44:19.265 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.265 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.265 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:19.265 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.265 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.265 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.265 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2257 2024-09-11T03:44:19.265 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.266 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.266 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.266 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.266 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885667+0000", 2024-09-11T03:44:19.266 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.266 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.266 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.266 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885667+0000", 2024-09-11T03:44:19.266 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.266 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.266 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.266 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885668+0000", 2024-09-11T03:44:19.267 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5600000000000005e-07 2024-09-11T03:44:19.267 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.267 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.267 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.267 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885673+0000", 2024-09-11T03:44:19.267 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4970000000000003e-06 2024-09-11T03:44:19.267 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.267 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.267 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.267 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885680+0000", 2024-09-11T03:44:19.267 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3780000000000003e-06 2024-09-11T03:44:19.267 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.267 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.267 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.268 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885686+0000", 2024-09-11T03:44:19.268 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0789999999999997e-06 2024-09-11T03:44:19.268 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.268 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.268 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.268 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886080+0000", 2024-09-11T03:44:19.268 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00039392399999999998 2024-09-11T03:44:19.268 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.268 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.268 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:44:19.268 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886091+0000", 2024-09-11T03:44:19.268 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0472000000000001e-05 2024-09-11T03:44:19.268 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.269 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.269 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.269 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896776+0000", 2024-09-11T03:44:19.269 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010684953000000001 2024-09-11T03:44:19.269 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.269 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.269 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:44:19.269 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896839+0000", 2024-09-11T03:44:19.269 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3293999999999994e-05 2024-09-11T03:44:19.269 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.269 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.269 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.269 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.904784+0000", 2024-09-11T03:44:19.269 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.007944843 2024-09-11T03:44:19.270 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.270 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.270 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.270 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.904909+0000", 2024-09-11T03:44:19.270 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012491600000000001 2024-09-11T03:44:19.270 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.270 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.270 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:44:19.270 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905105+0000", 2024-09-11T03:44:19.270 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000196336 2024-09-11T03:44:19.270 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.270 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.270 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.271 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906057+0000", 2024-09-11T03:44:19.271 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00095239800000000002 2024-09-11T03:44:19.271 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.271 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.271 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:19.271 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907348+0000", 2024-09-11T03:44:19.271 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012904489999999999 2024-09-11T03:44:19.271 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.271 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.271 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.271 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907457+0000", 2024-09-11T03:44:19.271 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108869 2024-09-11T03:44:19.271 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.271 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907543+0000", 2024-09-11T03:44:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6212999999999996e-05 2024-09-11T03:44:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907812+0000", 2024-09-11T03:44:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00026950200000000002 2024-09-11T03:44:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2258 2.18 2:1984a3a9:::smithi17557870-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:44:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.886095+0000", 2024-09-11T03:44:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 162.286655182, 2024-09-11T03:44:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022147336, 2024-09-11T03:44:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:44:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2258 2024-09-11T03:44:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886095+0000", 2024-09-11T03:44:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886095+0000", 2024-09-11T03:44:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886096+0000", 2024-09-11T03:44:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0200000000000001e-07 2024-09-11T03:44:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886100+0000", 2024-09-11T03:44:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9859999999999996e-06 2024-09-11T03:44:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886109+0000", 2024-09-11T03:44:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6349999999999995e-06 2024-09-11T03:44:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886116+0000", 2024-09-11T03:44:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0890000000000002e-06 2024-09-11T03:44:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886389+0000", 2024-09-11T03:44:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00027359200000000002 2024-09-11T03:44:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:44:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886399+0000", 2024-09-11T03:44:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9599999999999995e-06 2024-09-11T03:44:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896922+0000", 2024-09-11T03:44:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010522567 2024-09-11T03:44:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:44:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896977+0000", 2024-09-11T03:44:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5003999999999997e-05 2024-09-11T03:44:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905540+0000", 2024-09-11T03:44:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0085636329999999993 2024-09-11T03:44:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.278 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:44:19.278 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905625+0000", 2024-09-11T03:44:19.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4468999999999994e-05 2024-09-11T03:44:19.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.278 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.278 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.278 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907869+0000", 2024-09-11T03:44:19.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022438290000000001 2024-09-11T03:44:19.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.278 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.278 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.278 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.908014+0000", 2024-09-11T03:44:19.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00014511799999999999 2024-09-11T03:44:19.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.279 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.279 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.279 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.908243+0000", 2024-09-11T03:44:19.279 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00022865200000000001 2024-09-11T03:44:19.279 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.279 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.279 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.279 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.279 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.279 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2421 2.18 2:1bdf1a36:::smithi17557870-36:head [copy-from ver 157] snapc 65=[65,64,63,61,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-11T03:44:19.279 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:50.954849+0000", 2024-09-11T03:44:19.279 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 148.217901152, 2024-09-11T03:44:19.279 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022711387999999999, 2024-09-11T03:44:19.280 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.280 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.280 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:19.280 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.280 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.280 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.280 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2421 2024-09-11T03:44:19.280 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.280 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.280 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.280 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.280 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954849+0000", 2024-09-11T03:44:19.280 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.280 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.281 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.281 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.281 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954849+0000", 2024-09-11T03:44:19.281 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.281 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.281 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.281 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.281 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954851+0000", 2024-09-11T03:44:19.281 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3090000000000001e-06 2024-09-11T03:44:19.281 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.281 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.281 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.281 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954856+0000", 2024-09-11T03:44:19.282 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9059999999999999e-06 2024-09-11T03:44:19.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.282 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.282 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954867+0000", 2024-09-11T03:44:19.282 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0879999999999999e-05 2024-09-11T03:44:19.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.282 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.282 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954873+0000", 2024-09-11T03:44:19.282 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.203e-06 2024-09-11T03:44:19.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.282 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.283 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954921+0000", 2024-09-11T03:44:19.283 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8180000000000003e-05 2024-09-11T03:44:19.283 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.283 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.283 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.955004+0000", 2024-09-11T03:44:19.283 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2824000000000005e-05 2024-09-11T03:44:19.283 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.283 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:44:19.283 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964099+0000", 2024-09-11T03:44:19.283 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0090952499999999992 2024-09-11T03:44:19.283 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.284 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.284 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.970182+0000", 2024-09-11T03:44:19.284 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0060831510000000002 2024-09-11T03:44:19.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.284 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:19.284 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977039+0000", 2024-09-11T03:44:19.284 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068571980000000001 2024-09-11T03:44:19.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.284 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.284 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977432+0000", 2024-09-11T03:44:19.284 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00039304900000000002 2024-09-11T03:44:19.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.285 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:19.285 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977504+0000", 2024-09-11T03:44:19.285 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1863e-05 2024-09-11T03:44:19.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.285 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.285 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977561+0000", 2024-09-11T03:44:19.285 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6575e-05 2024-09-11T03:44:19.285 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.285 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.285 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.286 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2422 2.18 2:1bdf1a36:::smithi17557870-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e294)", 2024-09-11T03:44:19.286 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:50.954885+0000", 2024-09-11T03:44:19.286 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 148.217865115, 2024-09-11T03:44:19.286 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012999881, 2024-09-11T03:44:19.286 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.286 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.286 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:44:19.286 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.286 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.286 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.286 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2422 2024-09-11T03:44:19.286 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.286 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.287 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.287 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.287 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954885+0000", 2024-09-11T03:44:19.287 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.287 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.287 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.287 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.287 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954885+0000", 2024-09-11T03:44:19.287 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.287 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.287 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.287 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.287 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954886+0000", 2024-09-11T03:44:19.287 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.82e-07 2024-09-11T03:44:19.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.288 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.288 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.288 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954888+0000", 2024-09-11T03:44:19.288 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1909999999999999e-06 2024-09-11T03:44:19.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.288 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.288 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.288 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954899+0000", 2024-09-11T03:44:19.288 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0963e-05 2024-09-11T03:44:19.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.288 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.288 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.289 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954904+0000", 2024-09-11T03:44:19.289 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3539999999999999e-06 2024-09-11T03:44:19.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.289 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.289 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.955109+0000", 2024-09-11T03:44:19.289 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000204494 2024-09-11T03:44:19.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.289 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:19.289 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.955158+0000", 2024-09-11T03:44:19.290 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:19.291 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:18 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:44:19.291 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:18 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:44:19.291 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:18 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2056896357' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:44:19.291 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:18 smithi084 ceph-mon[27254]: osdmap e440: 12 total, 12 up, 11 in 2024-09-11T03:44:19.291 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9502000000000001e-05 2024-09-11T03:44:19.291 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.292 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.292 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.292 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.967733+0000", 2024-09-11T03:44:19.292 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012574571999999999 2024-09-11T03:44:19.292 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.292 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.292 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.292 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.967803+0000", 2024-09-11T03:44:19.292 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0160999999999995e-05 2024-09-11T03:44:19.292 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.292 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.292 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.292 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.967885+0000", 2024-09-11T03:44:19.293 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2162000000000004e-05 2024-09-11T03:44:19.293 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.293 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.293 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.293 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.293 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.293 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2511 2.d 2:b6a25a79:::smithi17557870-47:head [copy-from ver 327] snapc 6c=[6c,6b,6a,69,68,65,63,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-09-11T03:44:19.293 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:01.867547+0000", 2024-09-11T03:44:19.293 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 137.30520316499999, 2024-09-11T03:44:19.293 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017562828999999999, 2024-09-11T03:44:19.293 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.293 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.294 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:19.294 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.294 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.294 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.294 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2511 2024-09-11T03:44:19.294 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.294 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.294 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.294 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.294 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867547+0000", 2024-09-11T03:44:19.294 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.294 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.294 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.294 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.295 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867547+0000", 2024-09-11T03:44:19.295 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.295 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.295 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.295 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.295 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867549+0000", 2024-09-11T03:44:19.295 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4100000000000001e-06 2024-09-11T03:44:19.295 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.295 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.295 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.295 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867554+0000", 2024-09-11T03:44:19.295 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.463e-06 2024-09-11T03:44:19.295 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.295 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.296 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.296 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867565+0000", 2024-09-11T03:44:19.296 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0531e-05 2024-09-11T03:44:19.296 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.296 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.296 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.296 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867574+0000", 2024-09-11T03:44:19.296 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7850000000000003e-06 2024-09-11T03:44:19.296 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.296 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.296 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.296 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867625+0000", 2024-09-11T03:44:19.296 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0958999999999998e-05 2024-09-11T03:44:19.297 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.297 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.297 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:44:19.297 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867632+0000", 2024-09-11T03:44:19.297 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4150000000000004e-06 2024-09-11T03:44:19.297 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.297 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.297 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.297 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.870960+0000", 2024-09-11T03:44:19.297 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0033279199999999998 2024-09-11T03:44:19.297 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.297 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.297 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.297 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.871059+0000", 2024-09-11T03:44:19.298 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9423e-05 2024-09-11T03:44:19.298 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.298 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.298 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:44:19.298 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.878007+0000", 2024-09-11T03:44:19.298 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.006947796 2024-09-11T03:44:19.298 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.298 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.298 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:19.298 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.881858+0000", 2024-09-11T03:44:19.298 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003850647 2024-09-11T03:44:19.298 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.298 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.299 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.299 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.884846+0000", 2024-09-11T03:44:19.299 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0029880309999999999 2024-09-11T03:44:19.299 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.299 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.299 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.299 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.885043+0000", 2024-09-11T03:44:19.299 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00019722899999999999 2024-09-11T03:44:19.299 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.299 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.299 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:19.299 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.885080+0000", 2024-09-11T03:44:19.299 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7106e-05 2024-09-11T03:44:19.299 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.300 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.300 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.300 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.885110+0000", 2024-09-11T03:44:19.300 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0114e-05 2024-09-11T03:44:19.300 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.300 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.300 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.300 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.300 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.300 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2512 2.d 2:b6a25a79:::smithi17557870-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e305)", 2024-09-11T03:44:19.300 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:01.867594+0000", 2024-09-11T03:44:19.300 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 137.305156896, 2024-09-11T03:44:19.300 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013722218, 2024-09-11T03:44:19.300 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.301 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.301 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:44:19.301 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.301 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.301 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.301 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2512 2024-09-11T03:44:19.301 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.301 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.301 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.301 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.301 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867594+0000", 2024-09-11T03:44:19.301 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.301 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.301 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.302 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.302 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867594+0000", 2024-09-11T03:44:19.302 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.302 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.302 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.302 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.302 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867594+0000", 2024-09-11T03:44:19.302 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9100000000000004e-07 2024-09-11T03:44:19.302 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.302 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.302 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.302 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867597+0000", 2024-09-11T03:44:19.302 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6369999999999999e-06 2024-09-11T03:44:19.302 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.303 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.303 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.303 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867604+0000", 2024-09-11T03:44:19.303 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5610000000000001e-06 2024-09-11T03:44:19.303 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.303 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.303 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.303 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867610+0000", 2024-09-11T03:44:19.303 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1249999999999998e-06 2024-09-11T03:44:19.303 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.303 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.303 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.303 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867674+0000", 2024-09-11T03:44:19.303 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3969999999999999e-05 2024-09-11T03:44:19.304 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.304 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.304 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:19.304 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867679+0000", 2024-09-11T03:44:19.304 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5929999999999999e-06 2024-09-11T03:44:19.304 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.304 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.304 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.304 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.871237+0000", 2024-09-11T03:44:19.304 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035579689999999998 2024-09-11T03:44:19.304 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.304 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.304 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:19.305 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.871280+0000", 2024-09-11T03:44:19.305 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2917000000000003e-05 2024-09-11T03:44:19.305 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.305 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.305 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.305 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.881114+0000", 2024-09-11T03:44:19.305 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0098343660000000006 2024-09-11T03:44:19.305 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.305 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.305 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.305 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.881210+0000", 2024-09-11T03:44:19.305 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5730000000000007e-05 2024-09-11T03:44:19.305 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.305 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.306 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.306 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.881316+0000", 2024-09-11T03:44:19.306 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000105759 2024-09-11T03:44:19.306 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.306 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.306 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.306 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.306 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.306 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2568 2.d 2:b6a25a79:::smithi17557870-47:head [copy-from ver 157] snapc 70=[70,6f,6e,6c,68,65,63,5d] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-11T03:44:19.306 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:10.876872+0000", 2024-09-11T03:44:19.306 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 128.29587802699999, 2024-09-11T03:44:19.306 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018302771999999998, 2024-09-11T03:44:19.306 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.307 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.307 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:19.307 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.307 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.307 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.307 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2568 2024-09-11T03:44:19.307 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.307 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.307 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.307 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.307 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876872+0000", 2024-09-11T03:44:19.307 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.307 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.307 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.308 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.308 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876872+0000", 2024-09-11T03:44:19.308 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.308 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.308 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.308 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.308 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876874+0000", 2024-09-11T03:44:19.308 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5570000000000001e-06 2024-09-11T03:44:19.308 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.308 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.308 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.308 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876880+0000", 2024-09-11T03:44:19.308 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8799999999999996e-06 2024-09-11T03:44:19.308 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.309 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.309 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.309 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876901+0000", 2024-09-11T03:44:19.309 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0942999999999999e-05 2024-09-11T03:44:19.309 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.309 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.309 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.309 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876909+0000", 2024-09-11T03:44:19.309 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3410000000000003e-06 2024-09-11T03:44:19.309 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.309 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.309 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.309 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876966+0000", 2024-09-11T03:44:19.309 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7179999999999998e-05 2024-09-11T03:44:19.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.310 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.310 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.877099+0000", 2024-09-11T03:44:19.310 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013234499999999999 2024-09-11T03:44:19.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.310 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:44:19.310 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.885734+0000", 2024-09-11T03:44:19.310 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0086350730000000001 2024-09-11T03:44:19.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.310 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:19.310 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.890016+0000", 2024-09-11T03:44:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.004282213 2024-09-11T03:44:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.894661+0000", 2024-09-11T03:44:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0046455059999999998 2024-09-11T03:44:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.895046+0000", 2024-09-11T03:44:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00038434999999999999 2024-09-11T03:44:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.895116+0000", 2024-09-11T03:44:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0536000000000004e-05 2024-09-11T03:44:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.895175+0000", 2024-09-11T03:44:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8848000000000003e-05 2024-09-11T03:44:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2569 2.d 2:b6a25a79:::smithi17557870-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e314)", 2024-09-11T03:44:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:10.876927+0000", 2024-09-11T03:44:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 128.29582322799999, 2024-09-11T03:44:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012434861, 2024-09-11T03:44:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:44:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2569 2024-09-11T03:44:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876927+0000", 2024-09-11T03:44:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876927+0000", 2024-09-11T03:44:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876928+0000", 2024-09-11T03:44:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9200000000000001e-07 2024-09-11T03:44:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876931+0000", 2024-09-11T03:44:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.881e-06 2024-09-11T03:44:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876938+0000", 2024-09-11T03:44:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7090000000000002e-06 2024-09-11T03:44:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876944+0000", 2024-09-11T03:44:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7599999999999999e-06 2024-09-11T03:44:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.877267+0000", 2024-09-11T03:44:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00032272300000000002 2024-09-11T03:44:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.877312+0000", 2024-09-11T03:44:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4981999999999997e-05 2024-09-11T03:44:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.889216+0000", 2024-09-11T03:44:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011904246 2024-09-11T03:44:19.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.317 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.317 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.317 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.889286+0000", 2024-09-11T03:44:19.317 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9628000000000005e-05 2024-09-11T03:44:19.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.317 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.317 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.317 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.889362+0000", 2024-09-11T03:44:19.317 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6340000000000004e-05 2024-09-11T03:44:19.317 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.317 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.317 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.318 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.318 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(osd.2.0:700 2.e e415/95 2:750de3b7:::smithi17557870-8:99 v 415'576, mlcod=405'570)", 2024-09-11T03:44:19.318 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:43:52.912988+0000", 2024-09-11T03:44:19.318 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 26.259762070000001, 2024-09-11T03:44:19.318 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00915879, 2024-09-11T03:44:19.318 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:19.318 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:19.318 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:19.318 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:19.318 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.318 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:19.318 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.912988+0000", 2024-09-11T03:44:19.318 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.318 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.319 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.319 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:19.319 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.912988+0000", 2024-09-11T03:44:19.319 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:19.319 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.319 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.319 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:19.319 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.912989+0000", 2024-09-11T03:44:19.319 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8599999999999997e-07 2024-09-11T03:44:19.319 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.319 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.319 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:19.319 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.912998+0000", 2024-09-11T03:44:19.319 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6209999999999993e-06 2024-09-11T03:44:19.320 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.320 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.320 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:19.320 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.913011+0000", 2024-09-11T03:44:19.320 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3179999999999999e-05 2024-09-11T03:44:19.320 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.320 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.320 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:19.320 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.913020+0000", 2024-09-11T03:44:19.320 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4499999999999993e-06 2024-09-11T03:44:19.320 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.320 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.320 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:19.320 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.913086+0000", 2024-09-11T03:44:19.321 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5727000000000003e-05 2024-09-11T03:44:19.321 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.321 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.321 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:19.321 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.913115+0000", 2024-09-11T03:44:19.321 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8688999999999999e-05 2024-09-11T03:44:19.321 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.321 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.321 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:19.321 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.922075+0000", 2024-09-11T03:44:19.321 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0089597180000000002 2024-09-11T03:44:19.321 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:19.321 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:19.321 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:19.322 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.922147+0000", 2024-09-11T03:44:19.322 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2519000000000001e-05 2024-09-11T03:44:19.322 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.322 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.322 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.322 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:19.322 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:19.322 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:44:19.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:19.343 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:19.343 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:19.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:18 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:44:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:18 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:44:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:18 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2056896357' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:44:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:18 smithi121 ceph-mon[30729]: osdmap e440: 12 total, 12 up, 11 in 2024-09-11T03:44:19.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:19.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:19.473 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:19.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:19.575 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:19.596 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:19.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:19.597 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:19.698 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:19.721 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:19.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:19.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:19.802 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:44:19.823 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:19.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:19.846 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:19.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:19.947 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:19.969 INFO:tasks.rados.rados.2.smithi175.stdout:1623: done (1 left) 2024-09-11T03:44:19.969 INFO:tasks.rados.rados.2.smithi175.stdout:1624: snap_remove snap 171 2024-09-11T03:44:19.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:19.974 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:19.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:20.053 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:20.063 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:19 smithi175 ceph-mon[30894]: pgmap v816: 57 pgs: 57 active+clean; 172 MiB data, 794 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 6.8 MiB/s wr, 32 op/s 2024-09-11T03:44:20.064 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:19 smithi175 ceph-mon[30894]: osdmap e441: 12 total, 12 up, 11 in 2024-09-11T03:44:20.064 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:19 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/852636516' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:20.064 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:19 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:44:20.064 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:19 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:44:20.075 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:20.090 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-11T03:44:20.097 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:20.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:20.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:20.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:19 smithi084 ceph-mon[27254]: pgmap v816: 57 pgs: 57 active+clean; 172 MiB data, 794 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 6.8 MiB/s wr, 32 op/s 2024-09-11T03:44:20.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:19 smithi084 ceph-mon[27254]: osdmap e441: 12 total, 12 up, 11 in 2024-09-11T03:44:20.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:19 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/852636516' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:20.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:19 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:44:20.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:19 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:44:20.199 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:20.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:20.222 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:20.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:20.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:20.327 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:44:20.349 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:20.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:20.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:19 smithi121 ceph-mon[30729]: pgmap v816: 57 pgs: 57 active+clean; 172 MiB data, 794 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 6.8 MiB/s wr, 32 op/s 2024-09-11T03:44:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:19 smithi121 ceph-mon[30729]: osdmap e441: 12 total, 12 up, 11 in 2024-09-11T03:44:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:19 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/852636516' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:19 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:44:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:19 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:44:20.451 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:20.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:20.473 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:20.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:20.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:20.600 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:20.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:20.702 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:20.725 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:20.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:20.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:20.806 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:44:20.806 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:44:20.807 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:44:20.807 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:44:20.827 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:20.848 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:20.849 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:20.950 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:20.969 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:44:20.973 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:20.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:20.975 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:20.975 INFO:tasks.rados.rados.2.smithi175.stdout:1624: done (1 left) 2024-09-11T03:44:20.975 INFO:tasks.rados.rados.2.smithi175.stdout:1625: delete oid 37 current snap is 176 2024-09-11T03:44:20.975 INFO:tasks.rados.rados.2.smithi175.stdout:1622: finishing rollback tid 0 to smithi17557870-22 2024-09-11T03:44:20.975 INFO:tasks.rados.rados.2.smithi175.stdout:1622: finishing rollback tid 1 to smithi17557870-22 2024-09-11T03:44:20.975 INFO:tasks.rados.rados.2.smithi175.stdout:1622: finishing rollback tid 2 to smithi17557870-22 2024-09-11T03:44:20.975 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 592 (ObjNum 506 snap 148 seq_num 506) dirty exists 2024-09-11T03:44:20.977 INFO:tasks.rados.rados.2.smithi175.stdout:1622: done (1 left) 2024-09-11T03:44:20.977 INFO:tasks.rados.rados.2.smithi175.stdout:1625: done (0 left) 2024-09-11T03:44:20.977 INFO:tasks.rados.rados.2.smithi175.stdout:1626: rollback oid 4 current snap is 176 2024-09-11T03:44:20.977 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 4 to 175 2024-09-11T03:44:20.977 INFO:tasks.rados.rados.2.smithi175.stdout:1627: snap_create 2024-09-11T03:44:20.980 INFO:tasks.rados.rados.2.smithi175.stdout:1626: finishing rollback tid 0 to smithi17557870-4 2024-09-11T03:44:20.981 INFO:tasks.rados.rados.2.smithi175.stdout:1626: finishing rollback tid 1 to smithi17557870-4 2024-09-11T03:44:20.981 INFO:tasks.rados.rados.2.smithi175.stdout:1626: finishing rollback tid 2 to smithi17557870-4 2024-09-11T03:44:20.981 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 484 (ObjNum 553 snap 171 seq_num 553) dirty exists 2024-09-11T03:44:21.075 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:21.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:21.098 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:21.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:21.199 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:21.221 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:21.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:21.222 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:20 smithi175 ceph-mon[30894]: osdmap e442: 12 total, 12 up, 11 in 2024-09-11T03:44:21.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:20 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2261472434' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:44:21.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:21.346 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:21.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:21.360 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:20 smithi084 ceph-mon[27254]: osdmap e442: 12 total, 12 up, 11 in 2024-09-11T03:44:21.360 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:20 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2261472434' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:44:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:20 smithi121 ceph-mon[30729]: osdmap e442: 12 total, 12 up, 11 in 2024-09-11T03:44:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:20 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2261472434' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:44:21.447 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:21.472 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-11T03:44:21.473 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:21.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:21.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:21.575 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:21.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:21.663 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:21.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:21.753 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:44:21.765 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:21.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:21.787 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:21.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:21.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:44:21.888 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:21.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:21.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:21.928 INFO:tasks.rados.rados.2.smithi175.stdout:1626: done (1 left) 2024-09-11T03:44:21.928 INFO:tasks.rados.rados.2.smithi175.stdout:1627: done (0 left) 2024-09-11T03:44:21.928 INFO:tasks.rados.rados.2.smithi175.stdout:1628: write (excl) oid 27 current snap is 177 2024-09-11T03:44:21.928 INFO:tasks.rados.rados.2.smithi175.stdout:1628: seq_num 577 ranges {497639=545148,1548463=407858,2529302=564422,3528143=169496} 2024-09-11T03:44:21.933 INFO:tasks.rados.rados.2.smithi175.stdout:1628: writing smithi17557870-27 from 497639 to 1042787 tid 1 2024-09-11T03:44:21.938 INFO:tasks.rados.rados.2.smithi175.stdout:1628: writing smithi17557870-27 from 1548463 to 1956321 tid 2 2024-09-11T03:44:21.945 INFO:tasks.rados.rados.2.smithi175.stdout:1628: writing smithi17557870-27 from 2529302 to 3093724 tid 3 2024-09-11T03:44:21.948 INFO:tasks.rados.rados.2.smithi175.stdout:1628: writing smithi17557870-27 from 3528143 to 3697639 tid 4 2024-09-11T03:44:21.948 INFO:tasks.rados.rados.2.smithi175.stdout:1629: delete oid 4 current snap is 177 2024-09-11T03:44:21.949 INFO:tasks.rados.rados.2.smithi175.stdout:1628: finishing write tid 1 to smithi17557870-27 2024-09-11T03:44:21.949 INFO:tasks.rados.rados.2.smithi175.stdout:1628: finishing write tid 2 to smithi17557870-27 2024-09-11T03:44:21.949 INFO:tasks.rados.rados.2.smithi175.stdout:1628: finishing write tid 3 to smithi17557870-27 2024-09-11T03:44:21.951 INFO:tasks.rados.rados.2.smithi175.stdout:1628: finishing write tid 4 to smithi17557870-27 2024-09-11T03:44:21.951 INFO:tasks.rados.rados.2.smithi175.stdout:1628: finishing write tid 5 to smithi17557870-27 2024-09-11T03:44:21.951 INFO:tasks.rados.rados.2.smithi175.stdout:1629: done (1 left) 2024-09-11T03:44:21.951 INFO:tasks.rados.rados.2.smithi175.stdout:1630: read oid 23 snap -1 2024-09-11T03:44:21.952 INFO:tasks.rados.rados.2.smithi175.stdout:1630: expect deleted 2024-09-11T03:44:21.952 INFO:tasks.rados.rados.2.smithi175.stdout:1631: snap_create 2024-09-11T03:44:21.952 INFO:tasks.rados.rados.2.smithi175.stdout:1628: finishing write tid 6 to smithi17557870-27 2024-09-11T03:44:21.952 INFO:tasks.rados.rados.2.smithi175.stdout:1628: oid 27 updating version 0 to 585 2024-09-11T03:44:21.952 INFO:tasks.rados.rados.2.smithi175.stdout:1628: oid 27 version 585 is already newer than 582 2024-09-11T03:44:21.952 INFO:tasks.rados.rados.2.smithi175.stdout:1628: oid 27 version 585 is already newer than 584 2024-09-11T03:44:21.952 INFO:tasks.rados.rados.2.smithi175.stdout:1628: oid 27 version 585 is already newer than 583 2024-09-11T03:44:21.952 INFO:tasks.rados.rados.2.smithi175.stdout:1628: oid 27 version 585 is already newer than 581 2024-09-11T03:44:21.952 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 585 (ObjNum 577 snap 177 seq_num 577) dirty exists 2024-09-11T03:44:21.952 INFO:tasks.rados.rados.2.smithi175.stdout:1628: left oid 27 (ObjNum 577 snap 177 seq_num 577) 2024-09-11T03:44:22.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:22.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:22.037 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:22.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:22.120 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:22.138 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:22.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:21 smithi084 ceph-mon[27254]: pgmap v819: 57 pgs: 57 active+clean; 172 MiB data, 794 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 3.5 MiB/s wr, 21 op/s 2024-09-11T03:44:22.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:21 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:22.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2261472434' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:44:22.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:21 smithi084 ceph-mon[27254]: osdmap e443: 12 total, 12 up, 11 in 2024-09-11T03:44:22.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:21 smithi084 ceph-mon[27254]: osdmap e444: 12 total, 12 up, 11 in 2024-09-11T03:44:22.163 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:22.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:22.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:22.182 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:21 smithi175 ceph-mon[30894]: pgmap v819: 57 pgs: 57 active+clean; 172 MiB data, 794 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 3.5 MiB/s wr, 21 op/s 2024-09-11T03:44:22.182 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:21 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:22.182 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:21 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2261472434' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:44:22.183 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:21 smithi175 ceph-mon[30894]: osdmap e443: 12 total, 12 up, 11 in 2024-09-11T03:44:22.183 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:21 smithi175 ceph-mon[30894]: osdmap e444: 12 total, 12 up, 11 in 2024-09-11T03:44:22.265 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:22.287 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:22.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:22.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:22.288 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:44:22.288 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:44:22.288 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:44:22.289 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:44:22.289 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:44:22.388 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:22.410 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:22.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:22.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:22.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:21 smithi121 ceph-mon[30729]: pgmap v819: 57 pgs: 57 active+clean; 172 MiB data, 794 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 3.5 MiB/s wr, 21 op/s 2024-09-11T03:44:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:21 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:21 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2261472434' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:44:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:21 smithi121 ceph-mon[30729]: osdmap e443: 12 total, 12 up, 11 in 2024-09-11T03:44:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:21 smithi121 ceph-mon[30729]: osdmap e444: 12 total, 12 up, 11 in 2024-09-11T03:44:22.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:22.533 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:22.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:22.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:22.635 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:22.661 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:22.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:22.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:22.662 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:44:22.663 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:44:22.679 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:44:22.679 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:44:22.695 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:44:22.695 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:44:22.710 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:44:22.711 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:44:22.727 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:44:22.728 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:44:22.744 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:44:22.744 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:44:22.761 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:44:22.761 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:44:22.763 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:22.778 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:44:22.778 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:44:22.785 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:22.786 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:22.795 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:44:22.795 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:44:22.811 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:44:22.811 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:44:22.828 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:44:22.828 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:44:22.844 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:44:22.845 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:44:22.887 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:22.908 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:22.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:22.909 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:22.928 INFO:tasks.rados.rados.2.smithi175.stdout:1628: done (2 left) 2024-09-11T03:44:22.928 INFO:tasks.rados.rados.2.smithi175.stdout:1630: done (1 left) 2024-09-11T03:44:22.928 INFO:tasks.rados.rados.2.smithi175.stdout:1631: done (0 left) 2024-09-11T03:44:22.929 INFO:tasks.rados.rados.2.smithi175.stdout:1632: write oid 34 current snap is 178 2024-09-11T03:44:22.929 INFO:tasks.rados.rados.2.smithi175.stdout:1632: seq_num 579 ranges {531253=411999,1381780=411527,2300501=586966,3376974=354279} 2024-09-11T03:44:22.932 INFO:tasks.rados.rados.2.smithi175.stdout:1632: writing smithi17557870-34 from 531253 to 943252 tid 1 2024-09-11T03:44:22.936 INFO:tasks.rados.rados.2.smithi175.stdout:1632: writing smithi17557870-34 from 1381780 to 1793307 tid 2 2024-09-11T03:44:22.942 INFO:tasks.rados.rados.2.smithi175.stdout:1632: writing smithi17557870-34 from 2300501 to 2887467 tid 3 2024-09-11T03:44:22.948 INFO:tasks.rados.rados.2.smithi175.stdout:1632: writing smithi17557870-34 from 3376974 to 3731253 tid 4 2024-09-11T03:44:22.948 INFO:tasks.rados.rados.2.smithi175.stdout:1633: rollback oid 39 current snap is 178 2024-09-11T03:44:22.948 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 39 to 166 2024-09-11T03:44:22.948 INFO:tasks.rados.rados.2.smithi175.stdout:1632: finishing write tid 1 to smithi17557870-34 2024-09-11T03:44:22.948 INFO:tasks.rados.rados.2.smithi175.stdout:1632: finishing write tid 2 to smithi17557870-34 2024-09-11T03:44:22.948 INFO:tasks.rados.rados.2.smithi175.stdout:1632: finishing write tid 3 to smithi17557870-34 2024-09-11T03:44:22.948 INFO:tasks.rados.rados.2.smithi175.stdout:1634: read oid 48 snap -1 2024-09-11T03:44:22.948 INFO:tasks.rados.rados.2.smithi175.stdout:1634: expect (ObjNum 562 snap 171 seq_num 562) 2024-09-11T03:44:22.948 INFO:tasks.rados.rados.2.smithi175.stdout:1635: read oid 14 snap -1 2024-09-11T03:44:22.948 INFO:tasks.rados.rados.2.smithi175.stdout:1635: expect deleted 2024-09-11T03:44:22.949 INFO:tasks.rados.rados.2.smithi175.stdout:1636: snap_remove snap 177 2024-09-11T03:44:23.006 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-11T03:44:23.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:23.036 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:23.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:23.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:23.138 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:23.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:22 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:44:23.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:22 smithi084 ceph-mon[27254]: 2.2 scrub ok 2024-09-11T03:44:23.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:22 smithi084 ceph-mon[27254]: 2.11 scrub starts 2024-09-11T03:44:23.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:22 smithi084 ceph-mon[27254]: 2.11 scrub ok 2024-09-11T03:44:23.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:22 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/705648011' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:23.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:22 smithi084 ceph-mon[27254]: osdmap e445: 12 total, 12 up, 11 in 2024-09-11T03:44:23.163 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:23.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:23.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:23.266 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:23.285 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:44:23.287 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:23.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:23.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:23.389 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:23.396 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:22 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:44:23.396 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:22 smithi175 ceph-mon[30894]: 2.2 scrub ok 2024-09-11T03:44:23.396 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:22 smithi175 ceph-mon[30894]: 2.11 scrub starts 2024-09-11T03:44:23.396 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:22 smithi175 ceph-mon[30894]: 2.11 scrub ok 2024-09-11T03:44:23.396 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:22 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/705648011' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:23.396 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:22 smithi175 ceph-mon[30894]: osdmap e445: 12 total, 12 up, 11 in 2024-09-11T03:44:23.411 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:23.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:23.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:22 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:44:23.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:22 smithi121 ceph-mon[30729]: 2.2 scrub ok 2024-09-11T03:44:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:22 smithi121 ceph-mon[30729]: 2.11 scrub starts 2024-09-11T03:44:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:22 smithi121 ceph-mon[30729]: 2.11 scrub ok 2024-09-11T03:44:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:22 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/705648011' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:22 smithi121 ceph-mon[30729]: osdmap e445: 12 total, 12 up, 11 in 2024-09-11T03:44:23.513 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:23.534 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:23.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:23.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:23.635 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:23.661 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:23.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:23.721 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:44:23.721 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:44:23.725 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:44:23.725 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:44:23.726 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T03:44:23.726 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.791361 2024-09-11T03:44:23.726 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 7 0.7913607814791547 2024-09-11T03:44:23.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:23.773 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:44:23.788 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:23.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:23.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:23.890 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:23.914 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:23.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:23.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:23.930 INFO:tasks.rados.rados.2.smithi175.stdout:1636: done (4 left) 2024-09-11T03:44:23.930 INFO:tasks.rados.rados.2.smithi175.stdout:1637: read oid 6 snap -1 2024-09-11T03:44:23.930 INFO:tasks.rados.rados.2.smithi175.stdout:1637: expect (ObjNum 528 snap 156 seq_num 528) 2024-09-11T03:44:23.930 INFO:tasks.rados.rados.2.smithi175.stdout:1638: snap_remove snap 159 2024-09-11T03:44:23.984 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:24.017 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:24.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:24.039 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:24.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:24.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:23 smithi084 ceph-mon[27254]: 2.16 scrub starts 2024-09-11T03:44:24.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:23 smithi084 ceph-mon[27254]: 2.16 scrub ok 2024-09-11T03:44:24.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:23 smithi084 ceph-mon[27254]: pgmap v822: 57 pgs: 57 active+clean; 179 MiB data, 802 MiB used, 983 GiB / 983 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2024-09-11T03:44:24.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:23 smithi084 ceph-mon[27254]: 2.6 scrub starts 2024-09-11T03:44:24.088 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:23 smithi084 ceph-mon[27254]: 2.6 scrub ok 2024-09-11T03:44:24.088 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:23 smithi084 ceph-mon[27254]: osdmap e446: 12 total, 12 up, 11 in 2024-09-11T03:44:24.139 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:24.161 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:24.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:24.162 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:24.261 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:23 smithi175 ceph-mon[30894]: 2.16 scrub starts 2024-09-11T03:44:24.261 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:23 smithi175 ceph-mon[30894]: 2.16 scrub ok 2024-09-11T03:44:24.261 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:23 smithi175 ceph-mon[30894]: pgmap v822: 57 pgs: 57 active+clean; 179 MiB data, 802 MiB used, 983 GiB / 983 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2024-09-11T03:44:24.261 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:23 smithi175 ceph-mon[30894]: 2.6 scrub starts 2024-09-11T03:44:24.261 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:23 smithi175 ceph-mon[30894]: 2.6 scrub ok 2024-09-11T03:44:24.261 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:23 smithi175 ceph-mon[30894]: osdmap e446: 12 total, 12 up, 11 in 2024-09-11T03:44:24.263 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:24.285 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:24.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:24.286 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:24.291 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-11T03:44:24.387 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:24.409 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:24.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:23 smithi121 ceph-mon[30729]: 2.16 scrub starts 2024-09-11T03:44:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:23 smithi121 ceph-mon[30729]: 2.16 scrub ok 2024-09-11T03:44:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:23 smithi121 ceph-mon[30729]: pgmap v822: 57 pgs: 57 active+clean; 179 MiB data, 802 MiB used, 983 GiB / 983 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2024-09-11T03:44:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:23 smithi121 ceph-mon[30729]: 2.6 scrub starts 2024-09-11T03:44:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:23 smithi121 ceph-mon[30729]: 2.6 scrub ok 2024-09-11T03:44:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:23 smithi121 ceph-mon[30729]: osdmap e446: 12 total, 12 up, 11 in 2024-09-11T03:44:24.445 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:24.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:24.532 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:24.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:24.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:24.574 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:44:24.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:24.655 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:24.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:24.757 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:24.779 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:24.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:24.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:24.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:24.907 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:24.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:24.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:24.927 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.7 primary-affinity to 0.791361 (8518622) 2024-09-11T03:44:24.931 INFO:tasks.rados.rados.2.smithi175.stdout:1638: done (5 left) 2024-09-11T03:44:24.931 INFO:tasks.rados.rados.2.smithi175.stdout:1639: read oid 44 snap -1 2024-09-11T03:44:24.931 INFO:tasks.rados.rados.2.smithi175.stdout:1639: expect (ObjNum 567 snap 172 seq_num 567) 2024-09-11T03:44:24.931 INFO:tasks.rados.rados.2.smithi175.stdout:1640: snap_remove snap 174 2024-09-11T03:44:25.008 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:25.015 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:44:25.015 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:44:25.030 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:25.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:25.031 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:25.132 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:25.155 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:25.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:25.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:25.257 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:25.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:25.280 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:25.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:25.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:24 smithi084 ceph-mon[27254]: 1.16 scrub starts 2024-09-11T03:44:25.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:24 smithi084 ceph-mon[27254]: 1.16 scrub ok 2024-09-11T03:44:25.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:24 smithi084 ceph-mon[27254]: 1.1e scrub starts 2024-09-11T03:44:25.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:24 smithi084 ceph-mon[27254]: 1.1e scrub ok 2024-09-11T03:44:25.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:24 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3910469067' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.7913607814791547} : dispatch 2024-09-11T03:44:25.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:24 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3910469067' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7913607814791547}]': finished 2024-09-11T03:44:25.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:24 smithi084 ceph-mon[27254]: osdmap e447: 12 total, 12 up, 11 in 2024-09-11T03:44:25.381 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:25.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:25.404 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:25.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:24 smithi121 ceph-mon[30729]: 1.16 scrub starts 2024-09-11T03:44:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:24 smithi121 ceph-mon[30729]: 1.16 scrub ok 2024-09-11T03:44:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:24 smithi121 ceph-mon[30729]: 1.1e scrub starts 2024-09-11T03:44:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:24 smithi121 ceph-mon[30729]: 1.1e scrub ok 2024-09-11T03:44:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:24 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3910469067' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.7913607814791547} : dispatch 2024-09-11T03:44:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:24 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3910469067' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7913607814791547}]': finished 2024-09-11T03:44:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:24 smithi121 ceph-mon[30729]: osdmap e447: 12 total, 12 up, 11 in 2024-09-11T03:44:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:24 smithi175 ceph-mon[30894]: 1.16 scrub starts 2024-09-11T03:44:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:24 smithi175 ceph-mon[30894]: 1.16 scrub ok 2024-09-11T03:44:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:24 smithi175 ceph-mon[30894]: 1.1e scrub starts 2024-09-11T03:44:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:24 smithi175 ceph-mon[30894]: 1.1e scrub ok 2024-09-11T03:44:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:24 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3910469067' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.7913607814791547} : dispatch 2024-09-11T03:44:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:24 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3910469067' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7913607814791547}]': finished 2024-09-11T03:44:25.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:24 smithi175 ceph-mon[30894]: osdmap e447: 12 total, 12 up, 11 in 2024-09-11T03:44:25.506 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:25.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:25.529 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:25.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:25.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:44:25.629 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-11T03:44:25.630 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:25.651 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:25.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:25.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:25.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:25.775 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:25.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:25.804 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:25.867 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:44:25.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:25.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:25.905 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:25.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:25.928 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:44:25.933 INFO:tasks.rados.rados.2.smithi175.stdout:1640: done (6 left) 2024-09-11T03:44:25.933 INFO:tasks.rados.rados.2.smithi175.stdout:1641: copy_from oid 4 from oid 37 current snap is 178 2024-09-11T03:44:25.933 INFO:tasks.rados.rados.2.smithi175.stdout:1633: finishing rollback tid 1 to smithi17557870-39 2024-09-11T03:44:25.933 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 387 (ObjNum 321 snap 0 seq_num 3351724244) dirty dne 2024-09-11T03:44:25.933 INFO:tasks.rados.rados.2.smithi175.stdout:1632: finishing write tid 4 to smithi17557870-34 2024-09-11T03:44:25.933 INFO:tasks.rados.rados.2.smithi175.stdout:1632: finishing write tid 5 to smithi17557870-34 2024-09-11T03:44:25.933 INFO:tasks.rados.rados.2.smithi175.stdout:1632: finishing write tid 6 to smithi17557870-34 2024-09-11T03:44:25.933 INFO:tasks.rados.rados.2.smithi175.stdout:1632: oid 34 updating version 0 to 287 2024-09-11T03:44:25.933 INFO:tasks.rados.rados.2.smithi175.stdout:1632: oid 34 updating version 287 to 288 2024-09-11T03:44:25.933 INFO:tasks.rados.rados.2.smithi175.stdout:1632: oid 34 version 288 is already newer than 286 2024-09-11T03:44:25.933 INFO:tasks.rados.rados.2.smithi175.stdout:1632: oid 34 updating version 288 to 289 2024-09-11T03:44:25.934 INFO:tasks.rados.rados.2.smithi175.stdout:1632: oid 34 version 289 is already newer than 285 2024-09-11T03:44:25.934 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 289 (ObjNum 579 snap 178 seq_num 579) dirty exists 2024-09-11T03:44:25.934 INFO:tasks.rados.rados.2.smithi175.stdout:1632: left oid 34 (ObjNum 579 snap 178 seq_num 579) 2024-09-11T03:44:25.934 INFO:tasks.rados.rados.2.smithi175.stdout:1634: expect (ObjNum 562 snap 171 seq_num 562) 2024-09-11T03:44:26.006 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:26.027 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:26.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:26.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:26.082 INFO:tasks.rados.rados.2.smithi175.stdout:1637: expect (ObjNum 528 snap 156 seq_num 528) 2024-09-11T03:44:26.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:26.155 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:26.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:26.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:26.257 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:26.263 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:25 smithi175 ceph-mon[30894]: 2.5 scrub starts 2024-09-11T03:44:26.263 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:25 smithi175 ceph-mon[30894]: 2.5 scrub ok 2024-09-11T03:44:26.263 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:25 smithi175 ceph-mon[30894]: pgmap v825: 57 pgs: 57 active+clean; 179 MiB data, 804 MiB used, 983 GiB / 983 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2024-09-11T03:44:26.263 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:25 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2109962593' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:26.263 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:25 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:44:26.263 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:25 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:44:26.263 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:25 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2109962593' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:26.263 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:25 smithi175 ceph-mon[30894]: osdmap e448: 12 total, 12 up, 11 in 2024-09-11T03:44:26.265 INFO:tasks.rados.rados.2.smithi175.stdout:1639: expect (ObjNum 567 snap 172 seq_num 567) 2024-09-11T03:44:26.279 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:26.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:26.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:26.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:25 smithi084 ceph-mon[27254]: 2.5 scrub starts 2024-09-11T03:44:26.352 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:25 smithi084 ceph-mon[27254]: 2.5 scrub ok 2024-09-11T03:44:26.352 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:25 smithi084 ceph-mon[27254]: pgmap v825: 57 pgs: 57 active+clean; 179 MiB data, 804 MiB used, 983 GiB / 983 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2024-09-11T03:44:26.352 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:25 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2109962593' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:26.352 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:25 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:44:26.352 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:25 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:44:26.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:25 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2109962593' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:26.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:25 smithi084 ceph-mon[27254]: osdmap e448: 12 total, 12 up, 11 in 2024-09-11T03:44:26.361 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:44:26.361 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:44:26.381 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:26.403 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:26.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:26.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:25 smithi121 ceph-mon[30729]: 2.5 scrub starts 2024-09-11T03:44:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:25 smithi121 ceph-mon[30729]: 2.5 scrub ok 2024-09-11T03:44:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:25 smithi121 ceph-mon[30729]: pgmap v825: 57 pgs: 57 active+clean; 179 MiB data, 804 MiB used, 983 GiB / 983 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 9 op/s 2024-09-11T03:44:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:25 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2109962593' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:25 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:44:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:25 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:44:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:25 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2109962593' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:25 smithi121 ceph-mon[30729]: osdmap e448: 12 total, 12 up, 11 in 2024-09-11T03:44:26.453 INFO:tasks.rados.rados.2.smithi175.stdout:1641: finishing copy_from to smithi17557870-4 2024-09-11T03:44:26.453 INFO:tasks.rados.rados.2.smithi175.stdout:1641: got expected ENOENT (src dne) 2024-09-11T03:44:26.453 INFO:tasks.rados.rados.2.smithi175.stdout:1641: finishing copy_from racing read to smithi17557870-4 2024-09-11T03:44:26.453 INFO:tasks.rados.rados.2.smithi175.stdout:1632: done (6 left) 2024-09-11T03:44:26.453 INFO:tasks.rados.rados.2.smithi175.stdout:1633: done (5 left) 2024-09-11T03:44:26.453 INFO:tasks.rados.rados.2.smithi175.stdout:1634: done (4 left) 2024-09-11T03:44:26.453 INFO:tasks.rados.rados.2.smithi175.stdout:1635: done (3 left) 2024-09-11T03:44:26.453 INFO:tasks.rados.rados.2.smithi175.stdout:1637: done (2 left) 2024-09-11T03:44:26.453 INFO:tasks.rados.rados.2.smithi175.stdout:1639: done (1 left) 2024-09-11T03:44:26.453 INFO:tasks.rados.rados.2.smithi175.stdout:1641: done (0 left) 2024-09-11T03:44:26.453 INFO:tasks.rados.rados.2.smithi175.stdout:1642: snap_remove snap 164 2024-09-11T03:44:26.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:26.526 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:26.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:26.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:26.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:26.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:26.655 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:26.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:26.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:26.781 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:26.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:26.883 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:26.905 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:26.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:26.906 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:26.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:44:26.913 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:44:26.920 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:44:26.921 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:44:26.930 INFO:tasks.rados.rados.2.smithi175.stdout:1642: done (0 left) 2024-09-11T03:44:26.930 INFO:tasks.rados.rados.2.smithi175.stdout:1643: write (excl) oid 9 current snap is 178 2024-09-11T03:44:26.930 INFO:tasks.rados.rados.2.smithi175.stdout:1643: seq_num 580 ranges {548060=487248,1440262=671538,2844277=798238} 2024-09-11T03:44:26.936 INFO:tasks.rados.rados.2.smithi175.stdout:1643: writing smithi17557870-9 from 548060 to 1035308 tid 1 2024-09-11T03:44:26.942 INFO:tasks.rados.rados.2.smithi175.stdout:1643: writing smithi17557870-9 from 1440262 to 2111800 tid 2 2024-09-11T03:44:26.944 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:44:26.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:44:26.951 INFO:tasks.rados.rados.2.smithi175.stdout:1643: writing smithi17557870-9 from 2844277 to 3642515 tid 3 2024-09-11T03:44:26.951 INFO:tasks.rados.rados.2.smithi175.stdout:1644: read oid 26 snap -1 2024-09-11T03:44:26.951 INFO:tasks.rados.rados.2.smithi175.stdout:1644: expect (ObjNum 556 snap 171 seq_num 556) 2024-09-11T03:44:26.951 INFO:tasks.rados.rados.2.smithi175.stdout:1645: rollback oid 35 current snap is 178 2024-09-11T03:44:26.951 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 35 to 175 2024-09-11T03:44:26.951 INFO:tasks.rados.rados.2.smithi175.stdout:1643: finishing write tid 1 to smithi17557870-9 2024-09-11T03:44:26.951 INFO:tasks.rados.rados.2.smithi175.stdout:1646: snap_create 2024-09-11T03:44:26.951 INFO:tasks.rados.rados.2.smithi175.stdout:1643: finishing write tid 2 to smithi17557870-9 2024-09-11T03:44:26.955 INFO:tasks.rados.rados.2.smithi175.stdout:1643: finishing write tid 3 to smithi17557870-9 2024-09-11T03:44:26.956 INFO:tasks.rados.rados.2.smithi175.stdout:1643: finishing write tid 4 to smithi17557870-9 2024-09-11T03:44:26.956 INFO:tasks.rados.rados.2.smithi175.stdout:1643: finishing write tid 5 to smithi17557870-9 2024-09-11T03:44:26.956 INFO:tasks.rados.rados.2.smithi175.stdout:1643: oid 9 updating version 0 to 597 2024-09-11T03:44:26.956 INFO:tasks.rados.rados.2.smithi175.stdout:1643: oid 9 version 597 is already newer than 596 2024-09-11T03:44:26.956 INFO:tasks.rados.rados.2.smithi175.stdout:1643: oid 9 updating version 597 to 598 2024-09-11T03:44:26.956 INFO:tasks.rados.rados.2.smithi175.stdout:1643: oid 9 version 598 is already newer than 595 2024-09-11T03:44:26.956 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 598 (ObjNum 580 snap 178 seq_num 580) dirty exists 2024-09-11T03:44:26.956 INFO:tasks.rados.rados.2.smithi175.stdout:1643: left oid 9 (ObjNum 580 snap 178 seq_num 580) 2024-09-11T03:44:26.956 INFO:tasks.rados.rados.2.smithi175.stdout:1645: finishing rollback tid 0 to smithi17557870-35 2024-09-11T03:44:26.957 INFO:tasks.rados.rados.2.smithi175.stdout:1645: finishing rollback tid 1 to smithi17557870-35 2024-09-11T03:44:26.957 INFO:tasks.rados.rados.2.smithi175.stdout:1645: finishing rollback tid 2 to smithi17557870-35 2024-09-11T03:44:26.957 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 469 (ObjNum 518 snap 152 seq_num 518) dirty exists 2024-09-11T03:44:26.963 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:44:26.963 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:44:26.965 INFO:tasks.rados.rados.2.smithi175.stdout:1644: expect (ObjNum 556 snap 171 seq_num 556) 2024-09-11T03:44:26.981 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:44:26.981 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:44:26.999 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:44:27.000 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:44:27.007 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:27.016 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:44:27.017 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:44:27.028 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:27.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:27.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:27.034 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:44:27.034 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:44:27.051 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:44:27.051 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:44:27.067 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:44:27.068 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:44:27.078 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:26 smithi175 ceph-mon[30894]: 2.f scrub starts 2024-09-11T03:44:27.078 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:26 smithi175 ceph-mon[30894]: 2.f scrub ok 2024-09-11T03:44:27.078 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:26 smithi175 ceph-mon[30894]: 1.5 scrub starts 2024-09-11T03:44:27.078 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:26 smithi175 ceph-mon[30894]: 1.5 scrub ok 2024-09-11T03:44:27.078 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2678724145' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:27.078 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.4", "id": [7, 2]} : dispatch 2024-09-11T03:44:27.078 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.4", "id": [7, 2]}]': finished 2024-09-11T03:44:27.078 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:26 smithi175 ceph-mon[30894]: osdmap e449: 12 total, 12 up, 11 in 2024-09-11T03:44:27.084 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:44:27.084 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:44:27.101 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:44:27.102 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:44:27.106 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-11T03:44:27.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:27.160 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:27.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:27.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:27.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:27.283 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:27.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:27.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:27.385 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:27.386 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:44:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:26 smithi084 ceph-mon[27254]: 2.f scrub starts 2024-09-11T03:44:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:26 smithi084 ceph-mon[27254]: 2.f scrub ok 2024-09-11T03:44:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:26 smithi084 ceph-mon[27254]: 1.5 scrub starts 2024-09-11T03:44:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:26 smithi084 ceph-mon[27254]: 1.5 scrub ok 2024-09-11T03:44:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2678724145' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.4", "id": [7, 2]} : dispatch 2024-09-11T03:44:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.4", "id": [7, 2]}]': finished 2024-09-11T03:44:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:26 smithi084 ceph-mon[27254]: osdmap e449: 12 total, 12 up, 11 in 2024-09-11T03:44:27.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:27.407 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:27.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:26 smithi121 ceph-mon[30729]: 2.f scrub starts 2024-09-11T03:44:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:26 smithi121 ceph-mon[30729]: 2.f scrub ok 2024-09-11T03:44:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:26 smithi121 ceph-mon[30729]: 1.5 scrub starts 2024-09-11T03:44:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:26 smithi121 ceph-mon[30729]: 1.5 scrub ok 2024-09-11T03:44:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2678724145' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.4", "id": [7, 2]} : dispatch 2024-09-11T03:44:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.4", "id": [7, 2]}]': finished 2024-09-11T03:44:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:26 smithi121 ceph-mon[30729]: osdmap e449: 12 total, 12 up, 11 in 2024-09-11T03:44:27.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:27.529 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:27.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:27.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:27.657 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:27.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:27.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:27.759 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:27.781 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:27.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:27.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:27.839 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:44:27.840 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:44:27.840 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:44:27.840 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:44:27.840 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.840 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:365 2.c 2:327469ff:::smithi17557870-49:head [read 0~2806736,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:44:27.840 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:38:50.850028+0000", 2024-09-11T03:44:27.840 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 336.98291846299998, 2024-09-11T03:44:27.840 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0077480070000000003, 2024-09-11T03:44:27.840 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.840 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.840 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:44:27.840 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:27.841 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:27.841 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:27.841 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 365 2024-09-11T03:44:27.841 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.841 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.841 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.841 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.841 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850028+0000", 2024-09-11T03:44:27.841 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.841 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.841 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.841 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.841 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850028+0000", 2024-09-11T03:44:27.842 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.842 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.842 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.842 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.842 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850029+0000", 2024-09-11T03:44:27.842 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1200000000000003e-07 2024-09-11T03:44:27.842 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.842 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.842 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.842 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850033+0000", 2024-09-11T03:44:27.842 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4789999999999998e-06 2024-09-11T03:44:27.842 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.842 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.843 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.843 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850043+0000", 2024-09-11T03:44:27.843 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.38e-06 2024-09-11T03:44:27.843 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.843 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.843 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.843 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850056+0000", 2024-09-11T03:44:27.843 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3179999999999999e-05 2024-09-11T03:44:27.843 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.843 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.843 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.843 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850150+0000", 2024-09-11T03:44:27.843 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4549000000000006e-05 2024-09-11T03:44:27.843 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.844 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.844 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:27.844 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.850155+0000", 2024-09-11T03:44:27.844 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0440000000000003e-06 2024-09-11T03:44:27.844 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.844 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.844 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.844 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.855967+0000", 2024-09-11T03:44:27.844 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0058113289999999996 2024-09-11T03:44:27.844 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.844 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.844 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.844 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.856051+0000", 2024-09-11T03:44:27.845 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3840999999999999e-05 2024-09-11T03:44:27.845 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.845 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.845 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.845 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:38:50.857776+0000", 2024-09-11T03:44:27.845 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0017255930000000001 2024-09-11T03:44:27.845 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.845 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.845 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.845 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.845 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.845 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:640 2.a 2:5014e305:::smithi17557870-46:head [copy-from ver 23] snapc 11=[e] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:44:27.845 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:44:27.846 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 319.72265347500002, 2024-09-11T03:44:27.846 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019221736, 2024-09-11T03:44:27.846 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.846 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.846 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:27.846 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:27.846 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:27.846 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:27.846 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 640 2024-09-11T03:44:27.846 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.846 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.846 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.846 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.846 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:44:27.847 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.847 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.847 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.847 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.847 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:44:27.847 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.847 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.847 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.847 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.847 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110295+0000", 2024-09-11T03:44:27.847 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0609999999999998e-06 2024-09-11T03:44:27.847 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.847 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.848 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.848 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110305+0000", 2024-09-11T03:44:27.848 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4169999999999998e-06 2024-09-11T03:44:27.848 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.848 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.848 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.848 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110321+0000", 2024-09-11T03:44:27.848 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6288000000000001e-05 2024-09-11T03:44:27.848 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.848 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.848 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.848 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110328+0000", 2024-09-11T03:44:27.848 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3950000000000003e-06 2024-09-11T03:44:27.849 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.849 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.849 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.849 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110396+0000", 2024-09-11T03:44:27.849 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7164999999999998e-05 2024-09-11T03:44:27.849 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.849 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.850 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.850 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110507+0000", 2024-09-11T03:44:27.850 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000111217 2024-09-11T03:44:27.850 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.850 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.850 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:44:27.850 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.118213+0000", 2024-09-11T03:44:27.851 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0077066249999999999 2024-09-11T03:44:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125103+0000", 2024-09-11T03:44:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068894359999999997 2024-09-11T03:44:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125459+0000", 2024-09-11T03:44:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000356146 2024-09-11T03:44:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129369+0000", 2024-09-11T03:44:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039099249999999999 2024-09-11T03:44:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129428+0000", 2024-09-11T03:44:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9113000000000001e-05 2024-09-11T03:44:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129515+0000", 2024-09-11T03:44:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6947999999999995e-05 2024-09-11T03:44:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:641 2.a 2:5014e305:::smithi17557870-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:44:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:44:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 319.72260208400002, 2024-09-11T03:44:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011882876000000001, 2024-09-11T03:44:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:44:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 641 2024-09-11T03:44:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:44:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:44:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:44:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7599999999999998e-07 2024-09-11T03:44:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110347+0000", 2024-09-11T03:44:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.685e-06 2024-09-11T03:44:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110351+0000", 2024-09-11T03:44:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4889999999999997e-06 2024-09-11T03:44:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110354+0000", 2024-09-11T03:44:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0510000000000001e-06 2024-09-11T03:44:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110644+0000", 2024-09-11T03:44:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00028940200000000002 2024-09-11T03:44:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110673+0000", 2024-09-11T03:44:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9833e-05 2024-09-11T03:44:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122038+0000", 2024-09-11T03:44:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011364915 2024-09-11T03:44:27.859 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.859 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.859 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.859 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122127+0000", 2024-09-11T03:44:27.859 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8987999999999996e-05 2024-09-11T03:44:27.859 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.859 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.859 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.859 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122227+0000", 2024-09-11T03:44:27.859 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010013699999999999 2024-09-11T03:44:27.859 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.859 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.859 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.859 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.860 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.860 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:752 2.c 2:3c007163:::smithi17557870-35:head [copy-from ver 39] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:44:27.860 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:44:27.860 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 312.82393080200001, 2024-09-11T03:44:27.860 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017596783000000001, 2024-09-11T03:44:27.860 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.860 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.860 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:27.860 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:27.860 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:27.860 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:27.860 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 752 2024-09-11T03:44:27.860 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.861 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.861 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.861 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.861 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:44:27.861 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.861 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.861 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.861 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.861 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:44:27.861 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.861 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.861 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.861 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.862 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009018+0000", 2024-09-11T03:44:27.862 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.48e-06 2024-09-11T03:44:27.862 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.862 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.862 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.862 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009029+0000", 2024-09-11T03:44:27.862 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0377000000000001e-05 2024-09-11T03:44:27.862 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.862 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.862 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.862 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009050+0000", 2024-09-11T03:44:27.862 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0849e-05 2024-09-11T03:44:27.862 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.863 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.863 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.863 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009058+0000", 2024-09-11T03:44:27.863 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8019999999999998e-06 2024-09-11T03:44:27.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.863 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.863 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.863 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009125+0000", 2024-09-11T03:44:27.863 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6388000000000002e-05 2024-09-11T03:44:27.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.863 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.863 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:44:27.863 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009130+0000", 2024-09-11T03:44:27.863 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0900000000000004e-06 2024-09-11T03:44:27.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.864 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.864 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.864 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010518+0000", 2024-09-11T03:44:27.864 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013880030000000001 2024-09-11T03:44:27.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.864 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.864 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.864 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010582+0000", 2024-09-11T03:44:27.864 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.368e-05 2024-09-11T03:44:27.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.865 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.865 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:44:27.865 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.015350+0000", 2024-09-11T03:44:27.865 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0047682870000000004 2024-09-11T03:44:27.865 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.865 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.865 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:27.865 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.019005+0000", 2024-09-11T03:44:27.865 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0036549040000000001 2024-09-11T03:44:27.865 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.865 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.865 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:27.865 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.024096+0000", 2024-09-11T03:44:27.865 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050917439999999996 2024-09-11T03:44:27.866 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.866 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.866 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:27.866 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026521+0000", 2024-09-11T03:44:27.866 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024247399999999999 2024-09-11T03:44:27.866 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.866 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.866 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:27.866 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026559+0000", 2024-09-11T03:44:27.866 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8232999999999998e-05 2024-09-11T03:44:27.866 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.867 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.867 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.867 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026613+0000", 2024-09-11T03:44:27.867 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3205999999999998e-05 2024-09-11T03:44:27.867 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.867 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.867 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.867 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.867 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.867 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:753 2.c 2:3c007163:::smithi17557870-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:44:27.867 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:44:27.868 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 312.82386371500002, 2024-09-11T03:44:27.868 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0094619260000000007, 2024-09-11T03:44:27.868 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.868 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.868 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:44:27.868 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:27.868 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:27.868 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:27.868 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 753 2024-09-11T03:44:27.869 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.869 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.869 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.869 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.869 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:44:27.869 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.869 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.869 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.869 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.869 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:44:27.869 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.869 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.869 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.870 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.870 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:44:27.870 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2100000000000002e-07 2024-09-11T03:44:27.870 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.870 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.870 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.870 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009085+0000", 2024-09-11T03:44:27.870 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5910000000000001e-06 2024-09-11T03:44:27.870 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.870 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.870 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.870 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009090+0000", 2024-09-11T03:44:27.870 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5180000000000004e-06 2024-09-11T03:44:27.870 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.871 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.871 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009094+0000", 2024-09-11T03:44:27.871 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0620000000000001e-06 2024-09-11T03:44:27.871 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.871 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.871 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009178+0000", 2024-09-11T03:44:27.871 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4525999999999997e-05 2024-09-11T03:44:27.871 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.871 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:27.871 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009182+0000", 2024-09-11T03:44:27.872 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4640000000000002e-06 2024-09-11T03:44:27.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.872 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.872 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010677+0000", 2024-09-11T03:44:27.872 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014957569999999999 2024-09-11T03:44:27.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.872 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:27.872 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010706+0000", 2024-09-11T03:44:27.872 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8873999999999998e-05 2024-09-11T03:44:27.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.873 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.873 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018385+0000", 2024-09-11T03:44:27.873 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0076790490000000003 2024-09-11T03:44:27.873 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.873 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.873 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.873 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018466+0000", 2024-09-11T03:44:27.873 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0483999999999997e-05 2024-09-11T03:44:27.873 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.873 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.873 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.873 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018545+0000", 2024-09-11T03:44:27.873 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9179999999999997e-05 2024-09-11T03:44:27.874 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.874 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.874 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.874 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.874 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.874 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:985 2.b e163/95 2:dde92fc6:::smithi17557870-41:head v 163'52, mlcod=158'48)", 2024-09-11T03:44:27.874 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:44:27.874 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 287.89999617000001, 2024-09-11T03:44:27.874 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0092288890000000005, 2024-09-11T03:44:27.874 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.874 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.874 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:27.874 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.874 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.875 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.875 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:44:27.875 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.875 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.875 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.875 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.875 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:44:27.875 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.875 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.875 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.875 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.875 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:44:27.875 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7799999999999997e-07 2024-09-11T03:44:27.876 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.876 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.876 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.876 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935287+0000", 2024-09-11T03:44:27.876 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023361599999999999 2024-09-11T03:44:27.876 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.876 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.876 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.876 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935321+0000", 2024-09-11T03:44:27.876 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3840000000000001e-05 2024-09-11T03:44:27.876 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.876 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.876 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.877 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935335+0000", 2024-09-11T03:44:27.877 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4184e-05 2024-09-11T03:44:27.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.877 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.877 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.877 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935428+0000", 2024-09-11T03:44:27.877 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3144000000000001e-05 2024-09-11T03:44:27.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.877 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.877 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.877 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935503+0000", 2024-09-11T03:44:27.877 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4697999999999995e-05 2024-09-11T03:44:27.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.878 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:27.878 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.942117+0000", 2024-09-11T03:44:27.878 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0066141350000000002 2024-09-11T03:44:27.878 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.878 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.878 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.942179+0000", 2024-09-11T03:44:27.878 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.245e-05 2024-09-11T03:44:27.878 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.878 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.878 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.878 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.879 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:990 2.a 2:5014e305:::smithi17557870-46:head [write 638699~541153] snapc 24=[24,23,22,20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-11T03:44:27.879 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:44:27.879 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 287.89601491799999, 2024-09-11T03:44:27.879 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0080995760000000007, 2024-09-11T03:44:27.879 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.879 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.879 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:27.879 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:27.879 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:27.879 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:27.880 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 990 2024-09-11T03:44:27.880 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.880 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.880 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.880 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.880 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:44:27.880 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.880 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.880 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.880 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.880 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:44:27.880 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.881 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.881 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.881 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.881 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936935+0000", 2024-09-11T03:44:27.881 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7800000000000001e-06 2024-09-11T03:44:27.881 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.881 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.881 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.881 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937090+0000", 2024-09-11T03:44:27.881 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000155857 2024-09-11T03:44:27.881 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.881 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.881 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.882 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937109+0000", 2024-09-11T03:44:27.882 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8216e-05 2024-09-11T03:44:27.882 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.882 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.882 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.882 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937119+0000", 2024-09-11T03:44:27.882 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0406e-05 2024-09-11T03:44:27.882 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.882 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.882 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.882 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937195+0000", 2024-09-11T03:44:27.882 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6372999999999995e-05 2024-09-11T03:44:27.882 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.882 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.883 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.883 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937287+0000", 2024-09-11T03:44:27.883 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1434000000000006e-05 2024-09-11T03:44:27.883 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.883 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.883 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:44:27.883 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937454+0000", 2024-09-11T03:44:27.883 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000167262 2024-09-11T03:44:27.883 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.883 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.883 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:27.883 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.943162+0000", 2024-09-11T03:44:27.883 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0057082230000000001 2024-09-11T03:44:27.883 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.884 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.884 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:27.884 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.943196+0000", 2024-09-11T03:44:27.884 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4025e-05 2024-09-11T03:44:27.884 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.884 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.884 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:27.884 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.944922+0000", 2024-09-11T03:44:27.884 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001726102 2024-09-11T03:44:27.884 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.884 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.884 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:27.884 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.944977+0000", 2024-09-11T03:44:27.885 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4849999999999998e-05 2024-09-11T03:44:27.885 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.885 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.885 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.885 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.945031+0000", 2024-09-11T03:44:27.885 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4048000000000002e-05 2024-09-11T03:44:27.885 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.885 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.885 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.885 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.885 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.885 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1175 2.c 2:32ade24e:::smithi17557870-45:head [copy-from ver 130] snapc 2e=[2e,2c,2b,28,26] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:44:27.885 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:44:27.886 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 265.73080702599998, 2024-09-11T03:44:27.886 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015807878000000001, 2024-09-11T03:44:27.886 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.886 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.886 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:27.886 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:27.886 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:27.886 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:27.886 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1175 2024-09-11T03:44:27.886 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.886 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.886 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:27.889 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.889 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:44:27.889 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.889 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.889 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.890 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.890 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:44:27.890 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.890 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.890 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.890 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.890 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102142+0000", 2024-09-11T03:44:27.890 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3369999999999998e-06 2024-09-11T03:44:27.890 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.890 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.890 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.890 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102152+0000", 2024-09-11T03:44:27.890 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8400000000000007e-06 2024-09-11T03:44:27.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.891 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.891 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102172+0000", 2024-09-11T03:44:27.891 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0537e-05 2024-09-11T03:44:27.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.891 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.891 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102183+0000", 2024-09-11T03:44:27.891 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0423e-05 2024-09-11T03:44:27.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.891 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.892 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102253+0000", 2024-09-11T03:44:27.892 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0528e-05 2024-09-11T03:44:27.892 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.892 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.892 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:44:27.892 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102259+0000", 2024-09-11T03:44:27.892 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1589999999999998e-06 2024-09-11T03:44:27.892 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.892 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.892 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.892 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103477+0000", 2024-09-11T03:44:27.892 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012174550000000001 2024-09-11T03:44:27.892 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.893 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.893 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103545+0000", 2024-09-11T03:44:27.893 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7595999999999995e-05 2024-09-11T03:44:27.893 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.893 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:44:27.893 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.110323+0000", 2024-09-11T03:44:27.893 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0067785730000000004 2024-09-11T03:44:27.893 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.893 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:27.893 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.113411+0000", 2024-09-11T03:44:27.894 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030882769999999999 2024-09-11T03:44:27.894 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.894 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.894 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:27.894 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117742+0000", 2024-09-11T03:44:27.894 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0043301920000000001 2024-09-11T03:44:27.894 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.894 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.894 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:27.894 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117835+0000", 2024-09-11T03:44:27.894 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3848e-05 2024-09-11T03:44:27.894 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.894 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.895 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:27.895 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117916+0000", 2024-09-11T03:44:27.895 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0505000000000003e-05 2024-09-11T03:44:27.895 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.895 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.895 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.895 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117948+0000", 2024-09-11T03:44:27.895 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1608e-05 2024-09-11T03:44:27.895 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.895 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.895 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.895 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.895 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.895 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1176 2.c 2:32ade24e:::smithi17557870-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:44:27.896 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:44:27.896 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 265.73074276199998, 2024-09-11T03:44:27.896 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010847406, 2024-09-11T03:44:27.896 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.896 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.896 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:44:27.896 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:27.896 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:27.896 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:27.896 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1176 2024-09-11T03:44:27.896 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.896 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.896 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.897 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.897 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:44:27.897 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.897 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.897 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.897 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.897 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:44:27.897 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.897 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.897 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.898 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.898 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:44:27.898 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4999999999999998e-07 2024-09-11T03:44:27.898 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.898 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.898 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.898 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102206+0000", 2024-09-11T03:44:27.898 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9429999999999999e-06 2024-09-11T03:44:27.898 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.898 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.898 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.898 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102214+0000", 2024-09-11T03:44:27.898 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0399999999999993e-06 2024-09-11T03:44:27.899 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.899 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.899 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.899 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102220+0000", 2024-09-11T03:44:27.899 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2989999999999996e-06 2024-09-11T03:44:27.899 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.899 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.899 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.899 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102307+0000", 2024-09-11T03:44:27.899 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7635999999999999e-05 2024-09-11T03:44:27.899 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.899 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.899 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:27.899 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102310+0000", 2024-09-11T03:44:27.900 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.2160000000000002e-06 2024-09-11T03:44:27.900 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.900 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.900 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.900 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103661+0000", 2024-09-11T03:44:27.900 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013507860000000001 2024-09-11T03:44:27.900 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.900 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.900 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:27.900 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103686+0000", 2024-09-11T03:44:27.900 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.44e-05 2024-09-11T03:44:27.900 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.900 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.901 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.901 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.112884+0000", 2024-09-11T03:44:27.901 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0091986110000000006 2024-09-11T03:44:27.901 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.901 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.901 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.901 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.112967+0000", 2024-09-11T03:44:27.901 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2478000000000006e-05 2024-09-11T03:44:27.901 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.901 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.901 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.901 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.113051+0000", 2024-09-11T03:44:27.901 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4647000000000005e-05 2024-09-11T03:44:27.902 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.902 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.902 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.902 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.902 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.902 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1719 2.c 2:327469ff:::smithi17557870-49:head [write 487469~563838,stat] snapc 45=[45,44,43,3e,3d,3c,3a] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-11T03:44:27.902 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:45.859743+0000", 2024-09-11T03:44:27.902 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 221.97320398400001, 2024-09-11T03:44:27.902 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0078899869999999993, 2024-09-11T03:44:27.902 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.902 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.902 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:27.902 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:27.902 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:27.903 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:27.903 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1719 2024-09-11T03:44:27.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.903 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.903 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.903 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.859743+0000", 2024-09-11T03:44:27.903 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.903 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.903 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.859743+0000", 2024-09-11T03:44:27.903 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.904 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.904 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.859745+0000", 2024-09-11T03:44:27.904 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.712e-06 2024-09-11T03:44:27.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.904 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.904 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860066+0000", 2024-09-11T03:44:27.904 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00032013499999999998 2024-09-11T03:44:27.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.904 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860090+0000", 2024-09-11T03:44:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4590999999999999e-05 2024-09-11T03:44:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860102+0000", 2024-09-11T03:44:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2269999999999999e-05 2024-09-11T03:44:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860167+0000", 2024-09-11T03:44:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4806000000000003e-05 2024-09-11T03:44:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.906 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.906 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:44:27.906 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.860172+0000", 2024-09-11T03:44:27.906 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7219999999999999e-06 2024-09-11T03:44:27.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.906 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.906 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.906 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.862974+0000", 2024-09-11T03:44:27.906 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0028017480000000002 2024-09-11T03:44:27.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.906 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.906 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.906 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.863071+0000", 2024-09-11T03:44:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7738000000000005e-05 2024-09-11T03:44:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:44:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.863253+0000", 2024-09-11T03:44:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00018170999999999999 2024-09-11T03:44:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.864451+0000", 2024-09-11T03:44:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00119782 2024-09-11T03:44:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.865698+0000", 2024-09-11T03:44:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001246973 2024-09-11T03:44:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.867563+0000", 2024-09-11T03:44:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001865189 2024-09-11T03:44:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.867601+0000", 2024-09-11T03:44:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8333e-05 2024-09-11T03:44:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:45.867633+0000", 2024-09-11T03:44:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1239999999999999e-05 2024-09-11T03:44:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1863 2.c 2:3c007163:::smithi17557870-35:head [read 0~3613505,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:44:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:01.850063+0000", 2024-09-11T03:44:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 205.982883904, 2024-09-11T03:44:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0078079589999999997, 2024-09-11T03:44:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:44:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1863 2024-09-11T03:44:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850063+0000", 2024-09-11T03:44:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850063+0000", 2024-09-11T03:44:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850066+0000", 2024-09-11T03:44:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8700000000000001e-06 2024-09-11T03:44:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850074+0000", 2024-09-11T03:44:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3380000000000005e-06 2024-09-11T03:44:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850096+0000", 2024-09-11T03:44:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1753e-05 2024-09-11T03:44:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850108+0000", 2024-09-11T03:44:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2300000000000001e-05 2024-09-11T03:44:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850235+0000", 2024-09-11T03:44:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000126604 2024-09-11T03:44:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:44:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850244+0000", 2024-09-11T03:44:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.0790000000000007e-06 2024-09-11T03:44:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.852514+0000", 2024-09-11T03:44:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022705199999999998 2024-09-11T03:44:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.852621+0000", 2024-09-11T03:44:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000106649 2024-09-11T03:44:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.857871+0000", 2024-09-11T03:44:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0052498459999999999 2024-09-11T03:44:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.915 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.915 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1864 2.c 2:3c007163:::smithi17557870-35:head [sparse-read 0~1995320] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:44:27.915 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:44:27.915 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 205.98281342799999, 2024-09-11T03:44:27.915 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0079536510000000008, 2024-09-11T03:44:27.915 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.915 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.915 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:44:27.915 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:27.915 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:27.915 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:27.915 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1864 2024-09-11T03:44:27.915 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.916 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.916 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.916 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:44:27.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.916 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.916 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:44:27.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.916 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850134+0000", 2024-09-11T03:44:27.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2399999999999998e-07 2024-09-11T03:44:27.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850137+0000", 2024-09-11T03:44:27.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4010000000000001e-06 2024-09-11T03:44:27.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850164+0000", 2024-09-11T03:44:27.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6616999999999999e-05 2024-09-11T03:44:27.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.918 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850171+0000", 2024-09-11T03:44:27.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7209999999999997e-06 2024-09-11T03:44:27.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.918 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850311+0000", 2024-09-11T03:44:27.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00014029900000000001 2024-09-11T03:44:27.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:27.918 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850317+0000", 2024-09-11T03:44:27.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7799999999999997e-06 2024-09-11T03:44:27.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.919 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.919 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.857902+0000", 2024-09-11T03:44:27.919 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0075851759999999999 2024-09-11T03:44:27.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.919 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.919 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.857978+0000", 2024-09-11T03:44:27.919 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6272000000000004e-05 2024-09-11T03:44:27.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.919 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.920 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.858087+0000", 2024-09-11T03:44:27.920 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108761 2024-09-11T03:44:27.920 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.920 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.920 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.920 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.920 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.920 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1865 2.c 2:3c007163:::smithi17557870-35:head [sparse-read 0~1995320] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:44:27.920 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:01.850224+0000", 2024-09-11T03:44:27.920 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 205.98272307100001, 2024-09-11T03:44:27.920 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.008147797, 2024-09-11T03:44:27.920 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.920 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.921 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:44:27.921 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:27.921 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:27.921 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:27.921 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1865 2024-09-11T03:44:27.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.921 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.921 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850224+0000", 2024-09-11T03:44:27.921 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850224+0000", 2024-09-11T03:44:27.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850224+0000", 2024-09-11T03:44:27.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6099999999999999e-07 2024-09-11T03:44:27.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850229+0000", 2024-09-11T03:44:27.923 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5449999999999997e-06 2024-09-11T03:44:27.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.923 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.923 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.923 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850244+0000", 2024-09-11T03:44:27.923 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5048999999999999e-05 2024-09-11T03:44:27.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.923 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.923 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.923 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850258+0000", 2024-09-11T03:44:27.923 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3505e-05 2024-09-11T03:44:27.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.923 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.924 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.924 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850344+0000", 2024-09-11T03:44:27.924 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6329999999999995e-05 2024-09-11T03:44:27.924 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.924 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.924 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:27.924 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850349+0000", 2024-09-11T03:44:27.924 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0610000000000003e-06 2024-09-11T03:44:27.924 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.924 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.924 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.924 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.858124+0000", 2024-09-11T03:44:27.924 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.007774792 2024-09-11T03:44:27.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.925 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.925 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.925 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.858236+0000", 2024-09-11T03:44:27.925 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011264000000000001 2024-09-11T03:44:27.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.925 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.925 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.925 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.858371+0000", 2024-09-11T03:44:27.925 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013501400000000001 2024-09-11T03:44:27.925 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.925 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.925 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.926 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:1868 2.0 e245/188 2:0762776a:::smithi17557870-25:head v 245'39, mlcod=242'35)", 2024-09-11T03:44:27.926 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:01.863124+0000", 2024-09-11T03:44:27.926 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 205.969823125, 2024-09-11T03:44:27.926 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0073720210000000003, 2024-09-11T03:44:27.926 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.926 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.926 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:27.926 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.926 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.926 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.863124+0000", 2024-09-11T03:44:27.926 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.927 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.927 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.863124+0000", 2024-09-11T03:44:27.927 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.927 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.927 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.863124+0000", 2024-09-11T03:44:27.927 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.91e-07 2024-09-11T03:44:27.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.927 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.928 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.868459+0000", 2024-09-11T03:44:27.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0053354500000000003 2024-09-11T03:44:27.928 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.928 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.928 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.928 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.868473+0000", 2024-09-11T03:44:27.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3930999999999999e-05 2024-09-11T03:44:27.928 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.928 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.928 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.928 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.868481+0000", 2024-09-11T03:44:27.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9030000000000004e-06 2024-09-11T03:44:27.928 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.928 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.929 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.929 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.868545+0000", 2024-09-11T03:44:27.929 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3975999999999999e-05 2024-09-11T03:44:27.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.929 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.929 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.929 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.868565+0000", 2024-09-11T03:44:27.929 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9551e-05 2024-09-11T03:44:27.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.929 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.929 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:27.929 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.870455+0000", 2024-09-11T03:44:27.929 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001890624 2024-09-11T03:44:27.930 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.930 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.930 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.930 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.870496+0000", 2024-09-11T03:44:27.930 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0295e-05 2024-09-11T03:44:27.930 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.930 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.930 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.930 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.930 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.930 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2140 2.c 2:3c007163:::smithi17557870-35:head [copy-from ver 101] snapc 57=[57,56,51,4e] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-11T03:44:27.930 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:25.064188+0000", 2024-09-11T03:44:27.930 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 182.76875860499999, 2024-09-11T03:44:27.930 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013777315, 2024-09-11T03:44:27.931 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.931 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.931 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:27.931 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:27.931 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:27.931 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:27.931 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2140 2024-09-11T03:44:27.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.931 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.931 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.931 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064188+0000", 2024-09-11T03:44:27.931 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.932 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.932 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064188+0000", 2024-09-11T03:44:27.932 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.932 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.932 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064190+0000", 2024-09-11T03:44:27.932 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9700000000000002e-06 2024-09-11T03:44:27.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.932 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.932 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064198+0000", 2024-09-11T03:44:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3610000000000001e-06 2024-09-11T03:44:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064216+0000", 2024-09-11T03:44:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7768999999999999e-05 2024-09-11T03:44:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064225+0000", 2024-09-11T03:44:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7099999999999996e-06 2024-09-11T03:44:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064291+0000", 2024-09-11T03:44:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5634000000000002e-05 2024-09-11T03:44:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064367+0000", 2024-09-11T03:44:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6687999999999995e-05 2024-09-11T03:44:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:44:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.066720+0000", 2024-09-11T03:44:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023526010000000002 2024-09-11T03:44:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.069243+0000", 2024-09-11T03:44:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025227320000000002 2024-09-11T03:44:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.073940+0000", 2024-09-11T03:44:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0046973930000000002 2024-09-11T03:44:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.077850+0000", 2024-09-11T03:44:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039098279999999997 2024-09-11T03:44:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.077915+0000", 2024-09-11T03:44:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5160000000000006e-05 2024-09-11T03:44:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.077965+0000", 2024-09-11T03:44:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0469000000000001e-05 2024-09-11T03:44:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:2421 2.18 e294/281 2:1bdf1a36:::smithi17557870-36:head v 294'198, mlcod=0'0)", 2024-09-11T03:44:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:50.964269+0000", 2024-09-11T03:44:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 156.868677706, 2024-09-11T03:44:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013043020000000001, 2024-09-11T03:44:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964269+0000", 2024-09-11T03:44:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964269+0000", 2024-09-11T03:44:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964270+0000", 2024-09-11T03:44:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3510000000000001e-06 2024-09-11T03:44:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966680+0000", 2024-09-11T03:44:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002409698 2024-09-11T03:44:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966720+0000", 2024-09-11T03:44:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0265999999999997e-05 2024-09-11T03:44:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966738+0000", 2024-09-11T03:44:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7540000000000001e-05 2024-09-11T03:44:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966835+0000", 2024-09-11T03:44:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7059999999999996e-05 2024-09-11T03:44:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966862+0000", 2024-09-11T03:44:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7107e-05 2024-09-11T03:44:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:27.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977265+0000", 2024-09-11T03:44:27.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.01040258 2024-09-11T03:44:27.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977312+0000", 2024-09-11T03:44:27.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7417999999999999e-05 2024-09-11T03:44:27.941 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.941 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.941 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.941 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:2443 2.12 e296/281 2:494ba932:::smithi17557870-43:head v 296'131, mlcod=291'128)", 2024-09-11T03:44:27.942 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:52.895921+0000", 2024-09-11T03:44:27.942 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 154.93702528399999, 2024-09-11T03:44:27.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0084055459999999999, 2024-09-11T03:44:27.942 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.942 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.942 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:27.942 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.895921+0000", 2024-09-11T03:44:27.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.943 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.895921+0000", 2024-09-11T03:44:27.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.943 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.943 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.895922+0000", 2024-09-11T03:44:27.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1699999999999998e-07 2024-09-11T03:44:27.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.943 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.943 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902380+0000", 2024-09-11T03:44:27.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0064576629999999998 2024-09-11T03:44:27.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902391+0000", 2024-09-11T03:44:27.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1246999999999999e-05 2024-09-11T03:44:27.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902398+0000", 2024-09-11T03:44:27.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7560000000000002e-06 2024-09-11T03:44:27.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902482+0000", 2024-09-11T03:44:27.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4881000000000003e-05 2024-09-11T03:44:27.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.945 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902500+0000", 2024-09-11T03:44:27.945 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7464000000000001e-05 2024-09-11T03:44:27.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:27.945 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.904289+0000", 2024-09-11T03:44:27.945 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0017895599999999999 2024-09-11T03:44:27.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.946 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.904327+0000", 2024-09-11T03:44:27.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7457999999999999e-05 2024-09-11T03:44:27.946 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.946 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.946 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.946 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2972 2.c 2:32ade24e:::smithi17557870-45:head [copy-from ver 171] snapc 80=[80,7b,7a,79,77,75] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-11T03:44:27.946 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:44:27.946 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 102.758852731, 2024-09-11T03:44:27.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021036193000000002, 2024-09-11T03:44:27.946 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.946 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.946 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:27.947 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:27.947 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:27.947 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:27.947 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2972 2024-09-11T03:44:27.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.947 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.947 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:44:27.947 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:44:27.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074096+0000", 2024-09-11T03:44:27.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.685e-06 2024-09-11T03:44:27.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074104+0000", 2024-09-11T03:44:27.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9950000000000005e-06 2024-09-11T03:44:27.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074123+0000", 2024-09-11T03:44:27.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9587999999999999e-05 2024-09-11T03:44:27.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074132+0000", 2024-09-11T03:44:27.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1579999999999994e-06 2024-09-11T03:44:27.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074204+0000", 2024-09-11T03:44:27.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2072999999999999e-05 2024-09-11T03:44:27.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.950 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.950 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074302+0000", 2024-09-11T03:44:27.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7706000000000003e-05 2024-09-11T03:44:27.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.950 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:44:27.950 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.087397+0000", 2024-09-11T03:44:27.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013095292999999999 2024-09-11T03:44:27.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.950 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:44:27.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090766+0000", 2024-09-11T03:44:27.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0033682109999999999 2024-09-11T03:44:27.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:27.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.094977+0000", 2024-09-11T03:44:27.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0042108759999999997 2024-09-11T03:44:27.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:27.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095061+0000", 2024-09-11T03:44:27.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4827999999999995e-05 2024-09-11T03:44:27.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.952 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:27.952 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095099+0000", 2024-09-11T03:44:27.952 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7546999999999998e-05 2024-09-11T03:44:27.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.952 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.952 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095130+0000", 2024-09-11T03:44:27.952 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1232999999999997e-05 2024-09-11T03:44:27.952 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.952 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.952 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.953 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.953 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2973 2.c 2:32ade24e:::smithi17557870-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e348)", 2024-09-11T03:44:27.953 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:44:27.953 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 102.758787613, 2024-09-11T03:44:27.953 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016062098, 2024-09-11T03:44:27.953 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.953 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.953 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:44:27.953 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:44:27.953 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:44:27.953 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:27.953 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2973 2024-09-11T03:44:27.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.954 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.954 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:44:27.954 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.954 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:44:27.954 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.955 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074160+0000", 2024-09-11T03:44:27.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.46e-07 2024-09-11T03:44:27.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.955 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.955 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074162+0000", 2024-09-11T03:44:27.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5220000000000002e-06 2024-09-11T03:44:27.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.955 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.955 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074178+0000", 2024-09-11T03:44:27.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.609e-05 2024-09-11T03:44:27.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.956 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.956 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074183+0000", 2024-09-11T03:44:27.956 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8280000000000002e-06 2024-09-11T03:44:27.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.956 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.956 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074428+0000", 2024-09-11T03:44:27.956 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00024529599999999999 2024-09-11T03:44:27.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.956 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:27.956 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074469+0000", 2024-09-11T03:44:27.956 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0463000000000002e-05 2024-09-11T03:44:27.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.957 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090093+0000", 2024-09-11T03:44:27.957 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015624449 2024-09-11T03:44:27.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.957 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090155+0000", 2024-09-11T03:44:27.957 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1618999999999994e-05 2024-09-11T03:44:27.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090221+0000", 2024-09-11T03:44:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6384999999999995e-05 2024-09-11T03:44:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:3879 2.2 e415/333 2:43d79ac9:::smithi17557870-37:head v 415'179, mlcod=412'175)", 2024-09-11T03:44:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:43:52.925191+0000", 2024-09-11T03:44:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 34.907755477000002, 2024-09-11T03:44:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0073536000000000001, 2024-09-11T03:44:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:44:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:44:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:44:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:44:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.925191+0000", 2024-09-11T03:44:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:44:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.925191+0000", 2024-09-11T03:44:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:44:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:44:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.925192+0000", 2024-09-11T03:44:27.960 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3200000000000004e-07 2024-09-11T03:44:27.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.960 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:44:27.960 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.930567+0000", 2024-09-11T03:44:27.960 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0053747209999999998 2024-09-11T03:44:27.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.960 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:44:27.960 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.930584+0000", 2024-09-11T03:44:27.960 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7108e-05 2024-09-11T03:44:27.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.961 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:44:27.961 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.930596+0000", 2024-09-11T03:44:27.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1919999999999999e-05 2024-09-11T03:44:27.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.961 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:44:27.961 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.930667+0000", 2024-09-11T03:44:27.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1427000000000006e-05 2024-09-11T03:44:27.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.961 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:44:27.961 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.930692+0000", 2024-09-11T03:44:27.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5133000000000001e-05 2024-09-11T03:44:27.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.962 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:44:27.962 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.932502+0000", 2024-09-11T03:44:27.962 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00180963 2024-09-11T03:44:27.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:44:27.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:44:27.962 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:44:27.962 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:43:52.932545+0000", 2024-09-11T03:44:27.962 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2828999999999999e-05 2024-09-11T03:44:27.962 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.962 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.962 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.962 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:44:27.963 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:44:27.963 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:44:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:27.965 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:27.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:27.966 INFO:tasks.rados.rados.2.smithi175.stdout:1643: done (3 left) 2024-09-11T03:44:27.966 INFO:tasks.rados.rados.2.smithi175.stdout:1644: done (2 left) 2024-09-11T03:44:27.966 INFO:tasks.rados.rados.2.smithi175.stdout:1645: done (1 left) 2024-09-11T03:44:27.966 INFO:tasks.rados.rados.2.smithi175.stdout:1646: done (0 left) 2024-09-11T03:44:27.966 INFO:tasks.rados.rados.2.smithi175.stdout:1647: copy_from oid 41 from oid 15 current snap is 179 2024-09-11T03:44:27.966 INFO:tasks.rados.rados.2.smithi175.stdout:1648: delete oid 36 current snap is 179 2024-09-11T03:44:27.966 INFO:tasks.rados.rados.2.smithi175.stdout:1648: done (1 left) 2024-09-11T03:44:27.966 INFO:tasks.rados.rados.2.smithi175.stdout:1649: read oid 8 snap 169 2024-09-11T03:44:27.966 INFO:tasks.rados.rados.2.smithi175.stdout:1649: expect (ObjNum 515 snap 152 seq_num 515) 2024-09-11T03:44:27.966 INFO:tasks.rados.rados.2.smithi175.stdout:1650: snap_create 2024-09-11T03:44:27.966 INFO:tasks.rados.rados.2.smithi175.stdout:1647: finishing copy_from racing read to smithi17557870-41 2024-09-11T03:44:27.966 INFO:tasks.rados.rados.2.smithi175.stdout:1649: expect (ObjNum 515 snap 152 seq_num 515) 2024-09-11T03:44:28.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:28.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:28.091 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:28.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:28.131 INFO:tasks.rados.rados.2.smithi175.stdout:1647: finishing copy_from to smithi17557870-41 2024-09-11T03:44:28.131 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 223 (ObjNum 566 snap 172 seq_num 566) dirty exists 2024-09-11T03:44:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:27 smithi084 ceph-mon[27254]: 1.10 scrub starts 2024-09-11T03:44:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:27 smithi084 ceph-mon[27254]: 1.10 scrub ok 2024-09-11T03:44:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:27 smithi084 ceph-mon[27254]: 1.6 scrub starts 2024-09-11T03:44:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:27 smithi084 ceph-mon[27254]: 1.6 scrub ok 2024-09-11T03:44:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:27 smithi084 ceph-mon[27254]: pgmap v828: 57 pgs: 57 active+clean; 177 MiB data, 803 MiB used, 983 GiB / 983 GiB avail; 5.6 MiB/s rd, 431 KiB/s wr, 6 op/s 2024-09-11T03:44:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:27 smithi084 ceph-mon[27254]: osdmap e450: 12 total, 12 up, 11 in 2024-09-11T03:44:28.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:28.215 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:28.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:28.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:27 smithi175 ceph-mon[30894]: 1.10 scrub starts 2024-09-11T03:44:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:27 smithi175 ceph-mon[30894]: 1.10 scrub ok 2024-09-11T03:44:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:27 smithi175 ceph-mon[30894]: 1.6 scrub starts 2024-09-11T03:44:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:27 smithi175 ceph-mon[30894]: 1.6 scrub ok 2024-09-11T03:44:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:27 smithi175 ceph-mon[30894]: pgmap v828: 57 pgs: 57 active+clean; 177 MiB data, 803 MiB used, 983 GiB / 983 GiB avail; 5.6 MiB/s rd, 431 KiB/s wr, 6 op/s 2024-09-11T03:44:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:27 smithi175 ceph-mon[30894]: osdmap e450: 12 total, 12 up, 11 in 2024-09-11T03:44:28.317 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:28.338 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:28.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:28.339 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:27 smithi121 ceph-mon[30729]: 1.10 scrub starts 2024-09-11T03:44:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:27 smithi121 ceph-mon[30729]: 1.10 scrub ok 2024-09-11T03:44:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:27 smithi121 ceph-mon[30729]: 1.6 scrub starts 2024-09-11T03:44:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:27 smithi121 ceph-mon[30729]: 1.6 scrub ok 2024-09-11T03:44:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:27 smithi121 ceph-mon[30729]: pgmap v828: 57 pgs: 57 active+clean; 177 MiB data, 803 MiB used, 983 GiB / 983 GiB avail; 5.6 MiB/s rd, 431 KiB/s wr, 6 op/s 2024-09-11T03:44:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:27 smithi121 ceph-mon[30729]: osdmap e450: 12 total, 12 up, 11 in 2024-09-11T03:44:28.440 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:28.466 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:28.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:28.568 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:28.593 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:28.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:28.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:28.665 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:44:28.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:28.721 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:28.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:28.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:28.747 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T03:44:28.823 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:28.844 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:28.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:28.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:28.923 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:28.932 INFO:tasks.rados.rados.2.smithi175.stdout:1647: done (2 left) 2024-09-11T03:44:28.932 INFO:tasks.rados.rados.2.smithi175.stdout:1649: done (1 left) 2024-09-11T03:44:28.932 INFO:tasks.rados.rados.2.smithi175.stdout:1650: done (0 left) 2024-09-11T03:44:28.932 INFO:tasks.rados.rados.2.smithi175.stdout:1651: rollback oid 40 current snap is 180 2024-09-11T03:44:28.933 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 40 to 170 2024-09-11T03:44:28.933 INFO:tasks.rados.rados.2.smithi175.stdout:1652: rollback oid 45 current snap is 180 2024-09-11T03:44:28.933 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 45 to 173 2024-09-11T03:44:28.933 INFO:tasks.rados.rados.2.smithi175.stdout:1653: copy_from oid 44 from oid 6 current snap is 180 2024-09-11T03:44:28.933 INFO:tasks.rados.rados.2.smithi175.stdout:1654: copy_from oid 15 from oid 11 current snap is 180 2024-09-11T03:44:28.933 INFO:tasks.rados.rados.2.smithi175.stdout:1655: delete oid 38 current snap is 180 2024-09-11T03:44:28.935 INFO:tasks.rados.rados.2.smithi175.stdout:1654: finishing copy_from to smithi17557870-15 2024-09-11T03:44:28.936 INFO:tasks.rados.rados.2.smithi175.stdout:1654: got expected ENOENT (src dne) 2024-09-11T03:44:28.936 INFO:tasks.rados.rados.2.smithi175.stdout:1655: done (4 left) 2024-09-11T03:44:28.936 INFO:tasks.rados.rados.2.smithi175.stdout:1656: rollback oid 35 current snap is 180 2024-09-11T03:44:28.936 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 35 to 173 2024-09-11T03:44:28.936 INFO:tasks.rados.rados.2.smithi175.stdout:1654: finishing copy_from racing read to smithi17557870-15 2024-09-11T03:44:28.936 INFO:tasks.rados.rados.2.smithi175.stdout:1654: done (4 left) 2024-09-11T03:44:28.936 INFO:tasks.rados.rados.2.smithi175.stdout:1657: read oid 11 snap -1 2024-09-11T03:44:28.936 INFO:tasks.rados.rados.2.smithi175.stdout:1657: expect deleted 2024-09-11T03:44:28.936 INFO:tasks.rados.rados.2.smithi175.stdout:1658: read oid 8 snap -1 2024-09-11T03:44:28.936 INFO:tasks.rados.rados.2.smithi175.stdout:1658: expect (ObjNum 515 snap 152 seq_num 515) 2024-09-11T03:44:28.936 INFO:tasks.rados.rados.2.smithi175.stdout:1659: write oid 30 current snap is 180 2024-09-11T03:44:28.936 INFO:tasks.rados.rados.2.smithi175.stdout:1659: seq_num 583 ranges {598481=712995,1899355=418865,3093840=704641} 2024-09-11T03:44:28.942 INFO:tasks.rados.rados.2.smithi175.stdout:1659: writing smithi17557870-30 from 598481 to 1311476 tid 1 2024-09-11T03:44:28.946 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:28.948 INFO:tasks.rados.rados.2.smithi175.stdout:1659: writing smithi17557870-30 from 1899355 to 2318220 tid 2 2024-09-11T03:44:28.956 INFO:tasks.rados.rados.2.smithi175.stdout:1659: writing smithi17557870-30 from 3093840 to 3798481 tid 3 2024-09-11T03:44:28.956 INFO:tasks.rados.rados.2.smithi175.stdout:1660: snap_remove snap 178 2024-09-11T03:44:28.968 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:28.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:28.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:29.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:29.095 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:29.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:29.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:29.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:29.222 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:29.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:29.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:29.324 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:29.346 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:29.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:29.347 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:29.439 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:29.447 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:29.473 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:29.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:29.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:29.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:29.601 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:29.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:29.703 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:29.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:29.725 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:29.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:29.827 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:29.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:29.848 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:29.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:29.890 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:44:29.921 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:44:29.921 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:44:29.921 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:44:29.921 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:44:29.934 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:44:29.938 INFO:tasks.rados.rados.2.smithi175.stdout:1660: done (7 left) 2024-09-11T03:44:29.938 INFO:tasks.rados.rados.2.smithi175.stdout:1661: snap_remove snap 176 2024-09-11T03:44:29.949 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:29.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:29.972 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:29.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:30.074 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:30.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:30.096 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:30.096 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:30.197 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:30.213 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:29 smithi084 ceph-mon[27254]: 1.4 scrub starts 2024-09-11T03:44:30.213 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:29 smithi084 ceph-mon[27254]: 1.4 scrub ok 2024-09-11T03:44:30.213 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:29 smithi084 ceph-mon[27254]: pgmap v831: 57 pgs: 1 peering, 56 active+clean; 179 MiB data, 808 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 885 KiB/s wr, 11 op/s 2024-09-11T03:44:30.213 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:29 smithi084 ceph-mon[27254]: osdmap e451: 12 total, 12 up, 11 in 2024-09-11T03:44:30.213 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:29 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2908641155' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:44:30.220 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:30.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:30.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:29 smithi175 ceph-mon[30894]: 1.4 scrub starts 2024-09-11T03:44:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:29 smithi175 ceph-mon[30894]: 1.4 scrub ok 2024-09-11T03:44:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:29 smithi175 ceph-mon[30894]: pgmap v831: 57 pgs: 1 peering, 56 active+clean; 179 MiB data, 808 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 885 KiB/s wr, 11 op/s 2024-09-11T03:44:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:29 smithi175 ceph-mon[30894]: osdmap e451: 12 total, 12 up, 11 in 2024-09-11T03:44:30.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:29 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2908641155' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:44:30.322 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:30.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:30.345 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:30.345 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:29 smithi121 ceph-mon[30729]: 1.4 scrub starts 2024-09-11T03:44:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:29 smithi121 ceph-mon[30729]: 1.4 scrub ok 2024-09-11T03:44:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:29 smithi121 ceph-mon[30729]: pgmap v831: 57 pgs: 1 peering, 56 active+clean; 179 MiB data, 808 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 885 KiB/s wr, 11 op/s 2024-09-11T03:44:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:29 smithi121 ceph-mon[30729]: osdmap e451: 12 total, 12 up, 11 in 2024-09-11T03:44:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:29 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2908641155' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:44:30.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:30.467 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:30.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:30.569 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:30.591 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:30.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:30.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T03:44:30.693 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:44:30.693 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:44:30.694 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-11T03:44:30.694 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-11T03:44:30.694 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:30.718 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:30.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:30.719 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:30.820 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:30.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:30.847 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:30.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:30.945 INFO:tasks.rados.rados.2.smithi175.stdout:1661: done (7 left) 2024-09-11T03:44:30.945 INFO:tasks.rados.rados.2.smithi175.stdout:1662: read oid 5 snap -1 2024-09-11T03:44:30.945 INFO:tasks.rados.rados.2.smithi175.stdout:1662: expect deleted 2024-09-11T03:44:30.945 INFO:tasks.rados.rados.2.smithi175.stdout:1663: snap_remove snap 155 2024-09-11T03:44:30.949 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:30.973 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:30.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:30.974 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:31.075 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:31.100 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:31.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:31.101 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:31.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:30 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:44:31.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:30 smithi084 ceph-mon[27254]: 2.8 scrub ok 2024-09-11T03:44:31.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:30 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:31.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:30 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2908641155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:44:31.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:30 smithi084 ceph-mon[27254]: osdmap e452: 12 total, 12 up, 11 in 2024-09-11T03:44:31.130 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:31.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:31.227 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:31.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:30 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:44:31.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:30 smithi175 ceph-mon[30894]: 2.8 scrub ok 2024-09-11T03:44:31.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:30 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:31.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:30 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2908641155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:44:31.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:30 smithi175 ceph-mon[30894]: osdmap e452: 12 total, 12 up, 11 in 2024-09-11T03:44:31.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:31.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:31.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:31.354 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:31.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:31.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:31.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:30 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:44:31.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:30 smithi121 ceph-mon[30729]: 2.8 scrub ok 2024-09-11T03:44:31.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:30 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:31.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:30 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2908641155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:44:31.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:30 smithi121 ceph-mon[30729]: osdmap e452: 12 total, 12 up, 11 in 2024-09-11T03:44:31.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:31.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:31.480 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:31.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:31.581 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:31.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:31.605 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:31.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:31.620 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:44:31.620 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:44:31.627 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:44:31.627 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:44:31.643 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:44:31.644 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:44:31.660 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:44:31.661 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:44:31.678 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:31.679 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:44:31.679 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:44:31.701 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:44:31.701 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:44:31.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:31.718 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:44:31.719 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:44:31.727 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:31.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:31.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:31.735 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:44:31.736 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:44:31.753 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:44:31.753 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:44:31.770 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:44:31.770 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:44:31.787 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:44:31.787 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:44:31.804 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:44:31.804 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:44:31.829 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:31.855 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:31.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:31.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:31.930 INFO:tasks.rados.rados.2.smithi175.stdout:1663: done (8 left) 2024-09-11T03:44:31.930 INFO:tasks.rados.rados.2.smithi175.stdout:1664: snap_remove snap 169 2024-09-11T03:44:31.956 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:31.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:31.982 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:31.982 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:32.083 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:32.107 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:32.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:32.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:32.209 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:31 smithi175 ceph-mon[30894]: pgmap v834: 57 pgs: 1 peering, 56 active+clean; 179 MiB data, 808 MiB used, 983 GiB / 983 GiB avail; 6.9 MiB/s rd, 454 KiB/s wr, 5 op/s 2024-09-11T03:44:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:31 smithi175 ceph-mon[30894]: osdmap e453: 12 total, 12 up, 11 in 2024-09-11T03:44:32.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:31 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/670078157' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:32.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:31 smithi175 ceph-mon[30894]: osdmap e454: 12 total, 12 up, 11 in 2024-09-11T03:44:32.233 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:32.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:32.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:32.261 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:31 smithi084 ceph-mon[27254]: pgmap v834: 57 pgs: 1 peering, 56 active+clean; 179 MiB data, 808 MiB used, 983 GiB / 983 GiB avail; 6.9 MiB/s rd, 454 KiB/s wr, 5 op/s 2024-09-11T03:44:32.261 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:31 smithi084 ceph-mon[27254]: osdmap e453: 12 total, 12 up, 11 in 2024-09-11T03:44:32.262 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:31 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/670078157' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:32.262 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:31 smithi084 ceph-mon[27254]: osdmap e454: 12 total, 12 up, 11 in 2024-09-11T03:44:32.335 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:32.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:32.363 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:32.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:31 smithi121 ceph-mon[30729]: pgmap v834: 57 pgs: 1 peering, 56 active+clean; 179 MiB data, 808 MiB used, 983 GiB / 983 GiB avail; 6.9 MiB/s rd, 454 KiB/s wr, 5 op/s 2024-09-11T03:44:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:31 smithi121 ceph-mon[30729]: osdmap e453: 12 total, 12 up, 11 in 2024-09-11T03:44:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:31 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/670078157' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:31 smithi121 ceph-mon[30729]: osdmap e454: 12 total, 12 up, 11 in 2024-09-11T03:44:32.464 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:32.489 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:32.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:32.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:32.591 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:32.616 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:32.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:32.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:32.718 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:32.745 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:32.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:32.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:32.847 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:32.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:32.874 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:32.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:32.881 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:44:32.881 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:44:32.886 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:32.931 INFO:tasks.rados.rados.2.smithi175.stdout:1664: done (8 left) 2024-09-11T03:44:32.931 INFO:tasks.rados.rados.2.smithi175.stdout:1665: snap_create 2024-09-11T03:44:32.931 INFO:tasks.rados.rados.2.smithi175.stdout:1651: finishing rollback tid 1 to smithi17557870-40 2024-09-11T03:44:32.932 INFO:tasks.rados.rados.2.smithi175.stdout:1651: finishing rollback tid 2 to smithi17557870-40 2024-09-11T03:44:32.932 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 403 (ObjNum 487 snap 141 seq_num 487) dirty exists 2024-09-11T03:44:32.932 INFO:tasks.rados.rados.2.smithi175.stdout:1652: finishing rollback tid 0 to smithi17557870-45 2024-09-11T03:44:32.932 INFO:tasks.rados.rados.2.smithi175.stdout:1652: finishing rollback tid 1 to smithi17557870-45 2024-09-11T03:44:32.932 INFO:tasks.rados.rados.2.smithi175.stdout:1652: finishing rollback tid 2 to smithi17557870-45 2024-09-11T03:44:32.932 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 472 (ObjNum 416 snap 127 seq_num 416) dirty exists 2024-09-11T03:44:32.932 INFO:tasks.rados.rados.2.smithi175.stdout:1656: finishing rollback tid 0 to smithi17557870-35 2024-09-11T03:44:32.932 INFO:tasks.rados.rados.2.smithi175.stdout:1656: finishing rollback tid 1 to smithi17557870-35 2024-09-11T03:44:32.932 INFO:tasks.rados.rados.2.smithi175.stdout:1656: finishing rollback tid 2 to smithi17557870-35 2024-09-11T03:44:32.932 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 475 (ObjNum 518 snap 152 seq_num 518) dirty exists 2024-09-11T03:44:32.932 INFO:tasks.rados.rados.2.smithi175.stdout:1658: expect (ObjNum 515 snap 152 seq_num 515) 2024-09-11T03:44:32.977 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:32.998 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:32.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:32.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:33.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:33.116 INFO:tasks.rados.rados.2.smithi175.stdout:1653: finishing copy_from racing read to smithi17557870-44 2024-09-11T03:44:33.116 INFO:tasks.rados.rados.2.smithi175.stdout:1659: finishing write tid 1 to smithi17557870-30 2024-09-11T03:44:33.116 INFO:tasks.rados.rados.2.smithi175.stdout:1653: finishing copy_from to smithi17557870-44 2024-09-11T03:44:33.116 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 412 (ObjNum 528 snap 156 seq_num 528) dirty exists 2024-09-11T03:44:33.117 INFO:tasks.rados.rados.2.smithi175.stdout:1659: finishing write tid 2 to smithi17557870-30 2024-09-11T03:44:33.117 INFO:tasks.rados.rados.2.smithi175.stdout:1659: finishing write tid 3 to smithi17557870-30 2024-09-11T03:44:33.117 INFO:tasks.rados.rados.2.smithi175.stdout:1659: finishing write tid 4 to smithi17557870-30 2024-09-11T03:44:33.117 INFO:tasks.rados.rados.2.smithi175.stdout:1659: finishing write tid 5 to smithi17557870-30 2024-09-11T03:44:33.117 INFO:tasks.rados.rados.2.smithi175.stdout:1659: oid 30 updating version 0 to 648 2024-09-11T03:44:33.117 INFO:tasks.rados.rados.2.smithi175.stdout:1659: oid 30 version 648 is already newer than 645 2024-09-11T03:44:33.117 INFO:tasks.rados.rados.2.smithi175.stdout:1659: oid 30 version 648 is already newer than 647 2024-09-11T03:44:33.117 INFO:tasks.rados.rados.2.smithi175.stdout:1659: oid 30 version 648 is already newer than 646 2024-09-11T03:44:33.117 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 648 (ObjNum 583 snap 180 seq_num 583) dirty exists 2024-09-11T03:44:33.117 INFO:tasks.rados.rados.2.smithi175.stdout:1659: left oid 30 (ObjNum 583 snap 180 seq_num 583) 2024-09-11T03:44:33.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:33.128 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:33.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:33.150 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:44:33.229 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:33.250 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:33.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:33.251 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:33.352 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:33.373 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:33.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:33.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:33.476 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:33.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:33.498 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:33.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:33.530 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T03:44:33.573 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:33.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:33.636 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:33.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:33.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:33.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:33.761 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:33.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:33.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:33.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:33.889 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:33.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:33.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:33.932 INFO:tasks.rados.rados.2.smithi175.stdout:1651: done (8 left) 2024-09-11T03:44:33.933 INFO:tasks.rados.rados.2.smithi175.stdout:1652: done (7 left) 2024-09-11T03:44:33.933 INFO:tasks.rados.rados.2.smithi175.stdout:1653: done (6 left) 2024-09-11T03:44:33.933 INFO:tasks.rados.rados.2.smithi175.stdout:1656: done (5 left) 2024-09-11T03:44:33.933 INFO:tasks.rados.rados.2.smithi175.stdout:1657: done (4 left) 2024-09-11T03:44:33.933 INFO:tasks.rados.rados.2.smithi175.stdout:1658: done (3 left) 2024-09-11T03:44:33.933 INFO:tasks.rados.rados.2.smithi175.stdout:1659: done (2 left) 2024-09-11T03:44:33.933 INFO:tasks.rados.rados.2.smithi175.stdout:1662: done (1 left) 2024-09-11T03:44:33.933 INFO:tasks.rados.rados.2.smithi175.stdout:1665: done (0 left) 2024-09-11T03:44:33.933 INFO:tasks.rados.rados.2.smithi175.stdout:1666: rollback oid 12 current snap is 181 2024-09-11T03:44:33.933 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 12 to 175 2024-09-11T03:44:33.934 INFO:tasks.rados.rados.2.smithi175.stdout:1667: write (excl) oid 21 current snap is 181 2024-09-11T03:44:33.934 INFO:tasks.rados.rados.2.smithi175.stdout:1667: seq_num 584 ranges {615288=788244,1957837=562523,3121263=559679} 2024-09-11T03:44:33.940 INFO:tasks.rados.rados.2.smithi175.stdout:1667: writing smithi17557870-21 from 615288 to 1403532 tid 1 2024-09-11T03:44:33.946 INFO:tasks.rados.rados.2.smithi175.stdout:1667: writing smithi17557870-21 from 1957837 to 2520360 tid 2 2024-09-11T03:44:33.952 INFO:tasks.rados.rados.2.smithi175.stdout:1667: writing smithi17557870-21 from 3121263 to 3680942 tid 3 2024-09-11T03:44:33.952 INFO:tasks.rados.rados.2.smithi175.stdout:1668: snap_create 2024-09-11T03:44:33.952 INFO:tasks.rados.rados.2.smithi175.stdout:1666: finishing rollback tid 1 to smithi17557870-12 2024-09-11T03:44:33.952 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 0 (ObjNum 129 snap 0 seq_num 1745447160) dirty dne 2024-09-11T03:44:33.952 INFO:tasks.rados.rados.2.smithi175.stdout:1667: finishing write tid 1 to smithi17557870-21 2024-09-11T03:44:33.953 INFO:tasks.rados.rados.2.smithi175.stdout:1667: finishing write tid 2 to smithi17557870-21 2024-09-11T03:44:33.955 INFO:tasks.rados.rados.2.smithi175.stdout:1667: finishing write tid 3 to smithi17557870-21 2024-09-11T03:44:33.956 INFO:tasks.rados.rados.2.smithi175.stdout:1667: finishing write tid 4 to smithi17557870-21 2024-09-11T03:44:33.956 INFO:tasks.rados.rados.2.smithi175.stdout:1667: finishing write tid 5 to smithi17557870-21 2024-09-11T03:44:33.956 INFO:tasks.rados.rados.2.smithi175.stdout:1667: oid 21 updating version 0 to 617 2024-09-11T03:44:33.956 INFO:tasks.rados.rados.2.smithi175.stdout:1667: oid 21 updating version 617 to 619 2024-09-11T03:44:33.956 INFO:tasks.rados.rados.2.smithi175.stdout:1667: oid 21 version 619 is already newer than 618 2024-09-11T03:44:33.956 INFO:tasks.rados.rados.2.smithi175.stdout:1667: oid 21 version 619 is already newer than 616 2024-09-11T03:44:33.956 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 619 (ObjNum 584 snap 181 seq_num 584) dirty exists 2024-09-11T03:44:33.956 INFO:tasks.rados.rados.2.smithi175.stdout:1667: left oid 21 (ObjNum 584 snap 181 seq_num 584) 2024-09-11T03:44:33.991 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:34.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:34.014 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:34.015 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:34.030 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:33 smithi084 ceph-mon[27254]: pgmap v837: 57 pgs: 57 active+clean; 179 MiB data, 814 MiB used, 983 GiB / 983 GiB avail; 3.8 MiB/s rd, 1.5 MiB/s wr, 4 op/s 2024-09-11T03:44:34.030 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:33 smithi084 ceph-mon[27254]: osdmap e455: 12 total, 12 up, 11 in 2024-09-11T03:44:34.115 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:34.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:34.138 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:34.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:33 smithi175 ceph-mon[30894]: pgmap v837: 57 pgs: 57 active+clean; 179 MiB data, 814 MiB used, 983 GiB / 983 GiB avail; 3.8 MiB/s rd, 1.5 MiB/s wr, 4 op/s 2024-09-11T03:44:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:33 smithi175 ceph-mon[30894]: osdmap e455: 12 total, 12 up, 11 in 2024-09-11T03:44:34.239 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:34.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:34.262 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:34.262 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:34.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:34.389 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:34.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:34.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:33 smithi121 ceph-mon[30729]: pgmap v837: 57 pgs: 57 active+clean; 179 MiB data, 814 MiB used, 983 GiB / 983 GiB avail; 3.8 MiB/s rd, 1.5 MiB/s wr, 4 op/s 2024-09-11T03:44:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:33 smithi121 ceph-mon[30729]: osdmap e455: 12 total, 12 up, 11 in 2024-09-11T03:44:34.445 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:44:34.446 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:44:34.446 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 456, 2024-09-11T03:44:34.446 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "10fe7946-6fee-11ef-bcea-c7b262605968", 2024-09-11T03:44:34.446 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-09-11T03:30:30.532652+0000", 2024-09-11T03:44:34.446 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-09-11T03:44:33.927895+0000", 2024-09-11T03:44:34.446 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-09-11T03:37:26.787638+0000", 2024-09-11T03:44:34.446 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-09-11T03:43:51.906016+0000", 2024-09-11T03:44:34.446 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T03:44:34.446 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5801984, 2024-09-11T03:44:34.446 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-09-11T03:44:34.446 INFO:teuthology.orchestra.run.smithi084.stdout: "noscrub", 2024-09-11T03:44:34.447 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-09-11T03:44:34.447 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-09-11T03:44:34.447 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-09-11T03:44:34.447 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-09-11T03:44:34.447 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:44:34.447 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 26, 2024-09-11T03:44:34.447 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T03:44:34.447 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T03:44:34.447 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T03:44:34.447 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-09-11T03:44:34.447 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 2, 2024-09-11T03:44:34.448 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 12, 2024-09-11T03:44:34.448 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-09-11T03:44:34.448 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-09-11T03:44:34.448 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-09-11T03:44:34.448 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-09-11T03:44:34.448 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-09-11T03:44:34.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.448 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-09-11T03:44:34.448 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-09-11T03:44:34.448 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-11T03:33:42.916933+0000", 2024-09-11T03:44:34.448 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-11T03:44:34.448 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-11T03:44:34.448 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-11T03:44:34.448 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-11T03:44:34.449 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-11T03:44:34.449 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-11T03:44:34.449 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-11T03:44:34.449 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-11T03:44:34.449 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T03:44:34.449 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T03:44:34.449 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-11T03:44:34.449 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-11T03:44:34.449 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-11T03:44:34.449 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 31, 2024-09-11T03:44:34.449 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 31, 2024-09-11T03:44:34.449 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 31, 2024-09-11T03:44:34.449 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 31, 2024-09-11T03:44:34.450 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 31, 2024-09-11T03:44:34.450 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-11T03:44:34.450 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-11T03:44:34.450 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-11T03:44:34.450 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-11T03:44:34.450 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-11T03:44:34.450 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-11T03:44:34.450 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-11T03:44:34.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.450 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "222", 2024-09-11T03:44:34.450 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-11T03:44:34.450 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T03:44:34.450 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "220", 2024-09-11T03:44:34.451 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-11T03:44:34.451 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-11T03:44:34.451 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-11T03:44:34.451 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-11T03:44:34.451 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-11T03:44:34.451 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-11T03:44:34.451 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-11T03:44:34.451 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-11T03:44:34.451 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-11T03:44:34.451 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-11T03:44:34.451 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-11T03:44:34.451 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-11T03:44:34.451 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-11T03:44:34.452 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-11T03:44:34.452 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-11T03:44:34.452 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T03:44:34.452 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T03:44:34.452 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T03:44:34.452 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-11T03:44:34.452 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-11T03:44:34.452 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-11T03:44:34.452 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-11T03:44:34.452 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-11T03:44:34.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.452 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-11T03:44:34.452 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-11T03:44:34.453 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-11T03:44:34.453 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-11T03:44:34.453 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-11T03:44:34.453 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T03:44:34.453 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-11T03:44:34.453 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-11T03:44:34.453 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-11T03:44:34.453 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-11T03:44:34.453 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-11T03:44:34.453 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-11T03:44:34.453 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-09-11T03:44:34.453 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-09-11T03:44:34.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.454 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-11T03:44:34.454 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-09-11T03:44:34.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.454 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-11T03:44:34.454 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-11T03:44:34.454 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.9199999570846558, 2024-09-11T03:44:34.454 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.9199999570846558, 2024-09-11T03:44:34.454 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.92000001668930054, 2024-09-11T03:44:34.454 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.7699999809265137, 2024-09-11T03:44:34.454 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.7699999809265137, 2024-09-11T03:44:34.454 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.97191870212554932, 2024-09-11T03:44:34.454 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-11T03:44:34.454 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-09-11T03:44:34.455 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.455 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-11T03:44:34.455 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_0", 2024-09-11T03:44:34.455 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-11T03:38:28.163534+0000", 2024-09-11T03:44:34.455 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-11T03:44:34.455 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T03:44:34.455 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-11T03:44:34.455 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-11T03:44:34.455 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-11T03:44:34.456 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-11T03:44:34.456 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-11T03:44:34.456 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-11T03:44:34.456 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T03:44:34.456 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T03:44:34.456 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-11T03:44:34.456 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-11T03:44:34.456 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-11T03:44:34.456 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 26, 2024-09-11T03:44:34.456 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 26, 2024-09-11T03:44:34.456 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 26, 2024-09-11T03:44:34.456 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 26, 2024-09-11T03:44:34.456 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 26, 2024-09-11T03:44:34.457 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-11T03:44:34.457 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-11T03:44:34.457 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-11T03:44:34.457 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-11T03:44:34.457 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-11T03:44:34.457 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-11T03:44:34.458 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-11T03:44:34.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.458 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "456", 2024-09-11T03:44:34.459 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-11T03:44:34.459 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T03:44:34.459 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "276", 2024-09-11T03:44:34.459 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-11T03:44:34.459 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-11T03:44:34.459 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 182, 2024-09-11T03:44:34.459 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 456, 2024-09-11T03:44:34.459 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-11T03:44:34.459 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-11T03:44:34.459 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-11T03:44:34.459 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-11T03:44:34.459 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-11T03:44:34.459 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-11T03:44:34.460 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-11T03:44:34.460 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-11T03:44:34.460 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-11T03:44:34.460 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-11T03:44:34.460 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-11T03:44:34.460 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T03:44:34.460 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T03:44:34.460 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T03:44:34.460 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-11T03:44:34.460 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-11T03:44:34.460 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-11T03:44:34.460 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-11T03:44:34.460 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-11T03:44:34.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.461 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-11T03:44:34.461 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-11T03:44:34.461 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-11T03:44:34.461 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-11T03:44:34.461 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-11T03:44:34.461 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T03:44:34.461 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-11T03:44:34.461 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-11T03:44:34.461 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-11T03:44:34.461 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-11T03:44:34.461 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-11T03:44:34.461 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-11T03:44:34.462 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-11T03:44:34.462 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-09-11T03:44:34.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.462 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-11T03:44:34.462 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-11T03:44:34.462 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.8400000333786011, 2024-09-11T03:44:34.462 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.8400000333786011, 2024-09-11T03:44:34.462 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.92000001668930054, 2024-09-11T03:44:34.462 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.690000057220459, 2024-09-11T03:44:34.462 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.690000057220459, 2024-09-11T03:44:34.462 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.97191870212554932, 2024-09-11T03:44:34.462 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-11T03:44:34.462 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-09-11T03:44:34.462 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.463 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.463 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:44:34.463 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-11T03:44:34.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.463 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-11T03:44:34.463 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "9a431e48-e46b-44f9-bfa7-cb56c1484e1c", 2024-09-11T03:44:34.463 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:34.463 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:34.463 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:34.463 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:34.463 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:34.463 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:34.463 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 8, 2024-09-11T03:44:34.463 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 447, 2024-09-11T03:44:34.464 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:34.464 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:34.464 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:34.464 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.464 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.464 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-09-11T03:44:34.464 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:44:34.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.464 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.464 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-09-11T03:44:34.464 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:44:34.465 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.465 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.465 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:34.465 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.466 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.466 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-09-11T03:44:34.466 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:44:34.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.466 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.466 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-09-11T03:44:34.466 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:44:34.466 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.466 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.466 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:34.466 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.467 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.467 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-09-11T03:44:34.467 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:44:34.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.467 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.467 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-09-11T03:44:34.467 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:44:34.467 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.467 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.467 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:34.468 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.468 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.468 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-09-11T03:44:34.468 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:44:34.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.468 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.468 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-09-11T03:44:34.468 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:44:34.468 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.468 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.469 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6803/1376667411", 2024-09-11T03:44:34.469 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6805/1376667411", 2024-09-11T03:44:34.469 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6809/1376667411", 2024-09-11T03:44:34.469 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/1376667411", 2024-09-11T03:44:34.469 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:34.469 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:34.469 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:34.469 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.469 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-11T03:44:34.469 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "62fd6e25-3508-40b4-a0f4-f7de137e3bb5", 2024-09-11T03:44:34.470 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:34.470 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:34.470 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:34.470 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 0.871673583984375, 2024-09-11T03:44:34.470 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:34.470 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:34.470 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-09-11T03:44:34.470 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 432, 2024-09-11T03:44:34.470 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:34.470 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:34.470 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:34.470 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.471 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.471 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-09-11T03:44:34.471 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:44:34.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.471 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.471 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-09-11T03:44:34.471 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:44:34.471 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.471 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.471 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:34.472 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.472 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.472 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-09-11T03:44:34.472 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:44:34.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.472 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.472 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-09-11T03:44:34.472 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:44:34.472 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.472 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.473 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:34.473 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.473 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.473 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-09-11T03:44:34.473 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:44:34.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.473 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.473 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6817", 2024-09-11T03:44:34.473 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:44:34.473 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.474 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.474 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:34.474 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.474 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.474 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-09-11T03:44:34.474 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:44:34.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.474 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.475 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-09-11T03:44:34.475 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:44:34.475 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.475 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.475 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6811/3138939249", 2024-09-11T03:44:34.475 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6813/3138939249", 2024-09-11T03:44:34.475 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6817/3138939249", 2024-09-11T03:44:34.475 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/3138939249", 2024-09-11T03:44:34.475 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:34.475 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:34.475 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:34.475 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.476 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-11T03:44:34.476 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ff351425-4715-49dc-be49-f27bf205d57e", 2024-09-11T03:44:34.476 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:34.476 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:34.476 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:34.476 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:34.476 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:34.476 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:34.476 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 18, 2024-09-11T03:44:34.476 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 416, 2024-09-11T03:44:34.476 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:34.477 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:34.477 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:34.477 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.477 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.477 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6818", 2024-09-11T03:44:34.477 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:44:34.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.477 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.477 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6819", 2024-09-11T03:44:34.477 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:44:34.477 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.478 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.478 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:34.478 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.478 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.478 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6820", 2024-09-11T03:44:34.478 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:44:34.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.478 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.478 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6821", 2024-09-11T03:44:34.478 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:44:34.478 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.479 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.479 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:34.479 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.479 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.479 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6824", 2024-09-11T03:44:34.479 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:44:34.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.479 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.479 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6825", 2024-09-11T03:44:34.479 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:44:34.479 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.480 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.480 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:34.480 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.480 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.480 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6822", 2024-09-11T03:44:34.480 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:44:34.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.480 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.480 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6823", 2024-09-11T03:44:34.480 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:44:34.480 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.481 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.481 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6819/3001811591", 2024-09-11T03:44:34.481 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6821/3001811591", 2024-09-11T03:44:34.481 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6825/3001811591", 2024-09-11T03:44:34.481 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6823/3001811591", 2024-09-11T03:44:34.481 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:34.481 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:34.481 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:34.481 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.481 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-11T03:44:34.481 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "2518a93e-c612-4731-91ce-5685c25ffa9f", 2024-09-11T03:44:34.482 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:34.482 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:34.482 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:34.482 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:34.482 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:34.482 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:34.482 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 26, 2024-09-11T03:44:34.482 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 332, 2024-09-11T03:44:34.482 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:34.482 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:34.482 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:34.482 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.482 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.483 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6826", 2024-09-11T03:44:34.483 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:44:34.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.483 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.483 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6827", 2024-09-11T03:44:34.483 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:44:34.483 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.483 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.483 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:34.483 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.483 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.484 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6828", 2024-09-11T03:44:34.484 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:44:34.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.484 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.484 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6829", 2024-09-11T03:44:34.484 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:44:34.484 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.484 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.485 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:34.485 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.485 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:34.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.486 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.486 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6832", 2024-09-11T03:44:34.486 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:44:34.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.486 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.486 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6833", 2024-09-11T03:44:34.486 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:44:34.486 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.487 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.487 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:34.487 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.487 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.487 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6830", 2024-09-11T03:44:34.487 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:44:34.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.487 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.487 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6831", 2024-09-11T03:44:34.487 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:44:34.487 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.488 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.488 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6827/302512129", 2024-09-11T03:44:34.488 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6829/302512129", 2024-09-11T03:44:34.488 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6833/302512129", 2024-09-11T03:44:34.489 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6831/302512129", 2024-09-11T03:44:34.489 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:34.489 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:34.489 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:34.489 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.489 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-11T03:44:34.489 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d12f3c2d-95cc-4276-a7e1-bee982d58948", 2024-09-11T03:44:34.489 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:34.489 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:34.489 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:34.490 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:34.490 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:34.490 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:34.490 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-09-11T03:44:34.490 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 398, 2024-09-11T03:44:34.490 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:34.490 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:34.490 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:34.490 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.490 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.491 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6800", 2024-09-11T03:44:34.491 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:44:34.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.491 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.491 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6801", 2024-09-11T03:44:34.491 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:44:34.491 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.491 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.491 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:34.491 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.491 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.492 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6802", 2024-09-11T03:44:34.492 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:44:34.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.492 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.492 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6803", 2024-09-11T03:44:34.492 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:44:34.492 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.493 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.493 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:34.493 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.493 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.493 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6806", 2024-09-11T03:44:34.493 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:44:34.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.494 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.494 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6807", 2024-09-11T03:44:34.494 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:44:34.494 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.494 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.494 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:34.494 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.494 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.495 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6804", 2024-09-11T03:44:34.495 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:44:34.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.495 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.495 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6805", 2024-09-11T03:44:34.495 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:44:34.496 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.496 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.496 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.121:6801/3873214513", 2024-09-11T03:44:34.496 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.121:6803/3873214513", 2024-09-11T03:44:34.496 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.121:6807/3873214513", 2024-09-11T03:44:34.496 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.121:6805/3873214513", 2024-09-11T03:44:34.496 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:34.496 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:34.496 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:34.496 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.497 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-11T03:44:34.497 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "32164dd5-7e8f-4925-8526-bbb1e33e3364", 2024-09-11T03:44:34.497 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:34.497 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:34.497 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:34.497 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:34.497 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:34.497 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:34.497 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 38, 2024-09-11T03:44:34.497 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 414, 2024-09-11T03:44:34.497 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:34.497 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:34.497 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:34.497 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.498 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.498 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6808", 2024-09-11T03:44:34.498 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:44:34.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.498 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.498 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6809", 2024-09-11T03:44:34.498 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:44:34.498 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.498 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.498 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:34.498 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.499 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.499 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6810", 2024-09-11T03:44:34.499 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:44:34.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.499 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.499 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6811", 2024-09-11T03:44:34.499 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:44:34.499 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.499 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.499 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:34.500 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.500 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.500 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6814", 2024-09-11T03:44:34.500 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:44:34.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.500 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.500 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6815", 2024-09-11T03:44:34.500 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:44:34.500 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.500 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.500 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:34.501 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.501 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.501 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6812", 2024-09-11T03:44:34.501 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:44:34.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.501 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.501 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6813", 2024-09-11T03:44:34.501 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:44:34.501 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.501 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.502 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.121:6809/918612194", 2024-09-11T03:44:34.502 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.121:6811/918612194", 2024-09-11T03:44:34.502 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.121:6815/918612194", 2024-09-11T03:44:34.502 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.121:6813/918612194", 2024-09-11T03:44:34.502 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:34.502 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:34.502 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:34.502 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.502 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-11T03:44:34.502 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "3f107a88-803f-4cd8-804a-241310c282ea", 2024-09-11T03:44:34.502 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:34.503 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-09-11T03:44:34.503 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-09-11T03:44:34.503 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:34.503 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:34.503 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:34.503 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 45, 2024-09-11T03:44:34.503 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 95, 2024-09-11T03:44:34.503 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:34.503 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:34.503 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:34.504 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.504 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.504 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6816", 2024-09-11T03:44:34.504 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:44:34.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.504 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.504 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6817", 2024-09-11T03:44:34.504 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:44:34.504 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.504 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.505 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:34.505 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.505 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.505 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6818", 2024-09-11T03:44:34.505 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:44:34.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.505 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.505 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6819", 2024-09-11T03:44:34.506 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:44:34.506 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.506 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.506 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:34.506 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.506 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.506 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6822", 2024-09-11T03:44:34.506 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:44:34.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.506 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.506 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6823", 2024-09-11T03:44:34.507 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:44:34.507 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.507 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.507 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:34.507 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.507 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.507 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6820", 2024-09-11T03:44:34.507 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:44:34.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.507 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.507 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6821", 2024-09-11T03:44:34.508 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:44:34.508 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.508 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.508 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.121:6817/3384813363", 2024-09-11T03:44:34.508 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.121:6819/3384813363", 2024-09-11T03:44:34.508 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.121:6823/3384813363", 2024-09-11T03:44:34.508 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.121:6821/3384813363", 2024-09-11T03:44:34.508 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:34.508 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:34.508 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:34.508 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.509 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-11T03:44:34.509 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "dfee7f95-1fb9-4493-9907-cb10d1837bdb", 2024-09-11T03:44:34.509 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:34.509 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:34.509 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:34.509 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 0.791351318359375, 2024-09-11T03:44:34.509 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:34.509 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:34.509 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 52, 2024-09-11T03:44:34.509 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 417, 2024-09-11T03:44:34.509 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:34.509 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:34.510 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:34.510 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.510 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.510 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6824", 2024-09-11T03:44:34.510 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:44:34.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.510 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.510 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6825", 2024-09-11T03:44:34.510 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:44:34.510 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.510 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.511 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:34.511 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.511 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.511 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6826", 2024-09-11T03:44:34.511 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:44:34.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.511 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.511 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6827", 2024-09-11T03:44:34.511 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:44:34.511 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.511 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.512 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:34.512 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.512 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.512 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6830", 2024-09-11T03:44:34.512 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:44:34.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.512 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.513 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6831", 2024-09-11T03:44:34.513 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:44:34.513 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.513 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.513 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:34.513 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.513 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.513 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6828", 2024-09-11T03:44:34.513 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:44:34.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.514 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.514 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6829", 2024-09-11T03:44:34.514 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:44:34.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.514 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.514 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.121:6825/449222054", 2024-09-11T03:44:34.514 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.121:6827/449222054", 2024-09-11T03:44:34.514 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.121:6831/449222054", 2024-09-11T03:44:34.514 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.121:6829/449222054", 2024-09-11T03:44:34.514 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:34.514 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:34.514 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:34.515 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.515 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-09-11T03:44:34.515 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4", 2024-09-11T03:44:34.515 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:34.515 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:34.515 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:34.515 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:34.515 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:34.515 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:34.515 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 60, 2024-09-11T03:44:34.515 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 414, 2024-09-11T03:44:34.515 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:34.516 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:34.516 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:34.516 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.516 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.516 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6800", 2024-09-11T03:44:34.516 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:44:34.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.516 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.516 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6801", 2024-09-11T03:44:34.516 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:44:34.516 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.517 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.517 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:34.517 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.517 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.517 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6802", 2024-09-11T03:44:34.517 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:44:34.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.517 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.517 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6803", 2024-09-11T03:44:34.517 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:44:34.517 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.518 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.518 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:34.518 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.518 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.518 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6806", 2024-09-11T03:44:34.518 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:44:34.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.518 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.518 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6807", 2024-09-11T03:44:34.518 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:44:34.519 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.519 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.519 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:34.519 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.519 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.519 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6804", 2024-09-11T03:44:34.519 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:44:34.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.519 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.519 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6805", 2024-09-11T03:44:34.519 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:44:34.520 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.520 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.520 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.175:6801/2660128384", 2024-09-11T03:44:34.520 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.175:6803/2660128384", 2024-09-11T03:44:34.520 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.175:6807/2660128384", 2024-09-11T03:44:34.520 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.175:6805/2660128384", 2024-09-11T03:44:34.520 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:34.520 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:34.520 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:34.520 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.520 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-09-11T03:44:34.521 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "50162e42-0443-41d1-8709-a0db61c087da", 2024-09-11T03:44:34.521 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:34.521 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:34.521 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:34.521 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:34.521 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:34.521 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:34.521 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 69, 2024-09-11T03:44:34.521 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 312, 2024-09-11T03:44:34.521 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:34.521 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:34.521 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:34.521 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.522 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.522 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6808", 2024-09-11T03:44:34.522 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:44:34.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.522 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.522 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6809", 2024-09-11T03:44:34.522 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:44:34.522 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.522 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.522 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:34.522 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.523 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.523 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6810", 2024-09-11T03:44:34.523 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:44:34.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.523 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.523 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6811", 2024-09-11T03:44:34.523 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:44:34.523 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.523 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.523 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:34.523 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.524 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.524 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6814", 2024-09-11T03:44:34.524 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:44:34.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.524 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.524 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6815", 2024-09-11T03:44:34.524 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:44:34.524 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.524 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.524 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:34.524 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.525 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.525 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6812", 2024-09-11T03:44:34.525 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:44:34.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.525 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.525 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6813", 2024-09-11T03:44:34.525 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:44:34.525 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.525 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.525 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.175:6809/1270991649", 2024-09-11T03:44:34.526 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.175:6811/1270991649", 2024-09-11T03:44:34.526 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.175:6815/1270991649", 2024-09-11T03:44:34.526 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.175:6813/1270991649", 2024-09-11T03:44:34.526 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:34.526 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:34.526 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:34.526 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.526 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-09-11T03:44:34.526 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4b9426e1-bac8-47d3-ad58-54b89e8d1eb1", 2024-09-11T03:44:34.526 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:34.526 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:34.527 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:34.527 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:34.527 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:34.527 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:34.527 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 76, 2024-09-11T03:44:34.527 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 417, 2024-09-11T03:44:34.527 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:34.527 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:34.527 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:34.527 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.527 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.527 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6816", 2024-09-11T03:44:34.528 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:44:34.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.528 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.528 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6817", 2024-09-11T03:44:34.528 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:44:34.528 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.528 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.528 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:34.528 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.528 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.528 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6818", 2024-09-11T03:44:34.529 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:44:34.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.529 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.529 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6819", 2024-09-11T03:44:34.529 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:44:34.529 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.529 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.529 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:34.529 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.530 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.530 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6822", 2024-09-11T03:44:34.530 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:44:34.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.530 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.530 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6823", 2024-09-11T03:44:34.530 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:44:34.530 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.530 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.530 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:34.530 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.531 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.531 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6820", 2024-09-11T03:44:34.531 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:44:34.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.531 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.531 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6821", 2024-09-11T03:44:34.531 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:44:34.531 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.531 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.531 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.175:6817/3877631021", 2024-09-11T03:44:34.531 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.175:6819/3877631021", 2024-09-11T03:44:34.532 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.175:6823/3877631021", 2024-09-11T03:44:34.532 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.175:6821/3877631021", 2024-09-11T03:44:34.532 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:34.532 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:34.532 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:34.532 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.532 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-09-11T03:44:34.532 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "cd1a28a1-413c-4274-9876-e1bdb7703be6", 2024-09-11T03:44:34.532 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:34.532 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:34.533 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:34.533 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:34.533 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:34.533 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:34.533 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 83, 2024-09-11T03:44:34.533 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 449, 2024-09-11T03:44:34.533 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:34.533 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:34.533 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:34.533 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.533 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.533 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6824", 2024-09-11T03:44:34.534 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:44:34.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.534 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.534 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6825", 2024-09-11T03:44:34.534 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:44:34.534 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.534 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.534 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:34.534 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.534 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.534 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6826", 2024-09-11T03:44:34.535 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:44:34.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.535 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.535 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6827", 2024-09-11T03:44:34.535 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:44:34.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.535 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.535 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:34.535 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.535 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.535 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6830", 2024-09-11T03:44:34.536 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:44:34.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.536 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.536 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6831", 2024-09-11T03:44:34.536 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:44:34.536 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.536 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.536 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:34.536 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:34.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.537 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:34.537 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6828", 2024-09-11T03:44:34.537 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:44:34.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.537 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:34.537 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6829", 2024-09-11T03:44:34.537 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:44:34.537 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.537 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.537 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.175:6825/1150051236", 2024-09-11T03:44:34.537 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.175:6827/1150051236", 2024-09-11T03:44:34.538 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.175:6831/1150051236", 2024-09-11T03:44:34.538 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.175:6829/1150051236", 2024-09-11T03:44:34.538 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:34.538 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:34.538 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:34.538 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.538 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.538 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:44:34.538 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-09-11T03:44:34.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.538 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-11T03:44:34.538 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:34.538 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:34.538 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:34.539 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:34.539 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:34.539 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:43:37.917862+0000", 2024-09-11T03:44:34.539 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:34.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.539 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-11T03:44:34.539 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:34.539 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:34.539 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:34.539 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:34.539 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:34.539 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:44:19.071328+0000", 2024-09-11T03:44:34.540 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:34.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.540 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-11T03:44:34.540 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:34.540 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:34.540 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:34.540 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:34.540 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:34.540 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:43:23.497634+0000", 2024-09-11T03:44:34.540 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:34.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.540 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-11T03:44:34.541 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:34.541 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:34.541 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:34.541 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:34.541 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:34.541 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:43:50.817861+0000", 2024-09-11T03:44:34.541 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:34.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.541 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-11T03:44:34.541 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:34.541 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:34.541 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:34.542 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:34.542 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:34.542 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:43:22.721092+0000", 2024-09-11T03:44:34.542 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:34.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.542 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-11T03:44:34.542 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:34.542 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:34.542 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:34.542 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:34.542 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:34.542 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:43:31.672879+0000", 2024-09-11T03:44:34.542 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:34.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.543 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-11T03:44:34.543 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:34.543 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:34.543 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:34.543 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:34.543 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-09-11T03:44:34.543 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:43:31.134086+0000", 2024-09-11T03:44:34.543 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:34.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.543 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-11T03:44:34.544 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:34.544 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:34.544 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:34.544 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:34.544 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:34.544 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:44:12.815905+0000", 2024-09-11T03:44:34.544 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:34.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.544 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-09-11T03:44:34.544 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:34.544 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:34.544 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:34.544 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:34.545 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:34.545 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:44:11.646575+0000", 2024-09-11T03:44:34.545 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:34.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.545 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-09-11T03:44:34.545 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:34.545 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:34.545 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:34.545 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:34.545 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:34.545 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:44:05.536067+0000", 2024-09-11T03:44:34.546 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:34.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.546 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-09-11T03:44:34.546 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:34.546 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:34.546 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:34.546 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:34.546 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:34.546 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:44:17.122767+0000", 2024-09-11T03:44:34.546 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:34.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.546 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-09-11T03:44:34.547 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:34.547 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:34.547 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:34.547 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:34.547 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:34.547 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:43:45.212841+0000", 2024-09-11T03:44:34.547 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:34.547 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.547 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:44:34.547 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [], 2024-09-11T03:44:34.547 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [ 2024-09-11T03:44:34.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.547 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "1.4", 2024-09-11T03:44:34.548 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-11T03:44:34.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.548 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 7, 2024-09-11T03:44:34.548 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-11T03:44:34.548 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.548 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.548 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.2", 2024-09-11T03:44:34.548 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-11T03:44:34.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.548 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 9, 2024-09-11T03:44:34.548 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 11 2024-09-11T03:44:34.548 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.549 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.549 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.d", 2024-09-11T03:44:34.549 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-11T03:44:34.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.549 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 9, 2024-09-11T03:44:34.549 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-11T03:44:34.549 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.549 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.549 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.549 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:44:34.550 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-09-11T03:44:34.550 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-09-11T03:44:34.550 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-09-11T03:44:34.550 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-09-11T03:44:34.550 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/108136515": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:44:34.550 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3411515187": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:44:34.550 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6801/3365920551": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:44:34.550 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/3365920551": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:44:34.550 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3885577441": "2024-09-12T03:31:03.077186+0000", 2024-09-11T03:44:34.550 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/22039254": "2024-09-12T03:31:03.077186+0000", 2024-09-11T03:44:34.550 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6801/3496645948": "2024-09-12T03:31:03.077186+0000", 2024-09-11T03:44:34.550 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4206337830": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:44:34.550 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/3496645948": "2024-09-12T03:31:03.077186+0000", 2024-09-11T03:44:34.550 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1989775875": "2024-09-12T03:31:03.077186+0000" 2024-09-11T03:44:34.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.551 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-09-11T03:44:34.551 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-09-11T03:44:34.551 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-09-11T03:44:34.551 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-09-11T03:44:34.551 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-09-11T03:44:34.551 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-09-11T03:44:34.551 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-09-11T03:44:34.551 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-09-11T03:44:34.551 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.551 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [ 2024-09-11T03:44:34.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.551 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-11T03:44:34.552 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-11T03:44:34.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.552 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 157, 2024-09-11T03:44:34.552 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-11T03:44:34.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.552 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 166, 2024-09-11T03:44:34.552 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-11T03:44:34.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.552 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 171, 2024-09-11T03:44:34.552 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-11T03:44:34.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 178, 2024-09-11T03:44:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-11T03:44:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 180, 2024-09-11T03:44:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-11T03:44:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:44:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2024-09-11T03:44:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-09-11T03:44:34.553 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-09-11T03:44:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-09-11T03:44:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-09-11T03:44:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-09-11T03:44:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-09-11T03:44:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-09-11T03:44:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-09-11T03:44:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-09-11T03:44:34.554 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:34.554 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:44:34.554 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:34.579 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:34.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:34.680 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:34.682 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:44:34.706 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:34.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:34.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:34.808 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:34.833 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:34.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:34.834 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:34.935 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:34.938 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:44:34.941 INFO:tasks.rados.rados.2.smithi175.stdout:1666: done (2 left) 2024-09-11T03:44:34.941 INFO:tasks.rados.rados.2.smithi175.stdout:1667: done (1 left) 2024-09-11T03:44:34.941 INFO:tasks.rados.rados.2.smithi175.stdout:1668: done (0 left) 2024-09-11T03:44:34.941 INFO:tasks.rados.rados.2.smithi175.stdout:1669: copy_from oid 31 from oid 33 current snap is 182 2024-09-11T03:44:34.941 INFO:tasks.rados.rados.2.smithi175.stdout:1670: snap_remove snap 166 2024-09-11T03:44:34.962 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:34.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:34.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:35.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:35.085 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:35.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:35.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:35.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:34 smithi084 ceph-mon[27254]: osdmap e456: 12 total, 12 up, 11 in 2024-09-11T03:44:35.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:34 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2129045555' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-11T03:44:35.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:34 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/375532163' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:35.187 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:35.213 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:35.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:35.215 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:34 smithi175 ceph-mon[30894]: osdmap e456: 12 total, 12 up, 11 in 2024-09-11T03:44:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:34 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2129045555' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-11T03:44:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:34 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/375532163' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:35.262 INFO:tasks.thrashosds.thrasher:j is {'epoch': 456, 'fsid': '10fe7946-6fee-11ef-bcea-c7b262605968', 'created': '2024-09-11T03:30:30.532652+0000', 'modified': '2024-09-11T03:44:33.927895+0000', 'last_up_change': '2024-09-11T03:37:26.787638+0000', 'last_in_change': '2024-09-11T03:43:51.906016+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T03:33:42.916933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '222', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '220', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9199999570846558, 'score_stable': 1.9199999570846558, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 1.7699999809265137, 'raw_score_stable': 1.7699999809265137, 'primary_affinity_weighted': 0.9719187021255493, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-11T03:38:28.163534+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '456', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '276', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 182, 'snap_epoch': 456, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.840000033378601, 'score_stable': 1.840000033378601, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.9719187021255493, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}], 'osds': [{'osd': 0, 'uuid': '9a431e48-e46b-44f9-bfa7-cb56c1484e1c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 447, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 1376667411}, {'type': 'v1', 'addr': '172.21.15.84:6803', 'nonce': 1376667411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 1376667411}, {'type': 'v1', 'addr': '172.21.15.84:6805', 'nonce': 1376667411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 1376667411}, {'type': 'v1', 'addr': '172.21.15.84:6809', 'nonce': 1376667411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 1376667411}, {'type': 'v1', 'addr': '172.21.15.84:6807', 'nonce': 1376667411}]}, 'public_addr': '172.21.15.84:6803/1376667411', 'cluster_addr': '172.21.15.84:6805/1376667411', 'heartbeat_back_addr': '172.21.15.84:6809/1376667411', 'heartbeat_front_addr': '172.21.15.84:6807/1376667411', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '62fd6e25-3508-40b4-a0f4-f7de137e3bb5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.871673583984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 432, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3138939249}, {'type': 'v1', 'addr': '172.21.15.84:6811', 'nonce': 3138939249}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 3138939249}, {'type': 'v1', 'addr': '172.21.15.84:6813', 'nonce': 3138939249}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6816', 'nonce': 3138939249}, {'type': 'v1', 'addr': '172.21.15.84:6817', 'nonce': 3138939249}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 3138939249}, {'type': 'v1', 'addr': '172.21.15.84:6815', 'nonce': 3138939249}]}, 'public_addr': '172.21.15.84:6811/3138939249', 'cluster_addr': '172.21.15.84:6813/3138939249', 'heartbeat_back_addr': '172.21.15.84:6817/3138939249', 'heartbeat_front_addr': '172.21.15.84:6815/3138939249', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ff351425-4715-49dc-be49-f27bf205d57e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 416, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6818', 'nonce': 3001811591}, {'type': 'v1', 'addr': '172.21.15.84:6819', 'nonce': 3001811591}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6820', 'nonce': 3001811591}, {'type': 'v1', 'addr': '172.21.15.84:6821', 'nonce': 3001811591}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6824', 'nonce': 3001811591}, {'type': 'v1', 'addr': '172.21.15.84:6825', 'nonce': 3001811591}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6822', 'nonce': 3001811591}, {'type': 'v1', 'addr': '172.21.15.84:6823', 'nonce': 3001811591}]}, 'public_addr': '172.21.15.84:6819/3001811591', 'cluster_addr': '172.21.15.84:6821/3001811591', 'heartbeat_back_addr': '172.21.15.84:6825/3001811591', 'heartbeat_front_addr': '172.21.15.84:6823/3001811591', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2518a93e-c612-4731-91ce-5685c25ffa9f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 332, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6826', 'nonce': 302512129}, {'type': 'v1', 'addr': '172.21.15.84:6827', 'nonce': 302512129}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6828', 'nonce': 302512129}, {'type': 'v1', 'addr': '172.21.15.84:6829', 'nonce': 302512129}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6832', 'nonce': 302512129}, {'type': 'v1', 'addr': '172.21.15.84:6833', 'nonce': 302512129}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6830', 'nonce': 302512129}, {'type': 'v1', 'addr': '172.21.15.84:6831', 'nonce': 302512129}]}, 'public_addr': '172.21.15.84:6827/302512129', 'cluster_addr': '172.21.15.84:6829/302512129', 'heartbeat_back_addr': '172.21.15.84:6833/302512129', 'heartbeat_front_addr': '172.21.15.84:6831/302512129', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd12f3c2d-95cc-4276-a7e1-bee982d58948', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 398, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6800', 'nonce': 3873214513}, {'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3873214513}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6802', 'nonce': 3873214513}, {'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3873214513}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6806', 'nonce': 3873214513}, {'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 3873214513}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6804', 'nonce': 3873214513}, {'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 3873214513}]}, 'public_addr': '172.21.15.121:6801/3873214513', 'cluster_addr': '172.21.15.121:6803/3873214513', 'heartbeat_back_addr': '172.21.15.121:6807/3873214513', 'heartbeat_front_addr': '172.21.15.121:6805/3873214513', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '32164dd5-7e8f-4925-8526-bbb1e33e3364', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 414, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6808', 'nonce': 918612194}, {'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 918612194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6810', 'nonce': 918612194}, {'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 918612194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6814', 'nonce': 918612194}, {'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 918612194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6812', 'nonce': 918612194}, {'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 918612194}]}, 'public_addr': '172.21.15.121:6809/918612194', 'cluster_addr': '172.21.15.121:6811/918612194', 'heartbeat_back_addr': '172.21.15.121:6815/918612194', 'heartbeat_front_addr': '172.21.15.121:6813/918612194', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '3f107a88-803f-4cd8-804a-241310c282ea', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6816', 'nonce': 3384813363}, {'type': 'v1', 'addr': '172.21.15.121:6817', 'nonce': 3384813363}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6818', 'nonce': 3384813363}, {'type': 'v1', 'addr': '172.21.15.121:6819', 'nonce': 3384813363}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6822', 'nonce': 3384813363}, {'type': 'v1', 'addr': '172.21.15.121:6823', 'nonce': 3384813363}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6820', 'nonce': 3384813363}, {'type': 'v1', 'addr': '172.21.15.121:6821', 'nonce': 3384813363}]}, 'public_addr': '172.21.15.121:6817/3384813363', 'cluster_addr': '172.21.15.121:6819/3384813363', 'heartbeat_back_addr': '172.21.15.121:6823/3384813363', 'heartbeat_front_addr': '172.21.15.121:6821/3384813363', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dfee7f95-1fb9-4493-9907-cb10d1837bdb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.791351318359375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6824', 'nonce': 449222054}, {'type': 'v1', 'addr': '172.21.15.121:6825', 'nonce': 449222054}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6826', 'nonce': 449222054}, {'type': 'v1', 'addr': '172.21.15.121:6827', 'nonce': 449222054}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6830', 'nonce': 449222054}, {'type': 'v1', 'addr': '172.21.15.121:6831', 'nonce': 449222054}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6828', 'nonce': 449222054}, {'type': 'v1', 'addr': '172.21.15.121:6829', 'nonce': 449222054}]}, 'public_addr': '172.21.15.121:6825/449222054', 'cluster_addr': '172.21.15.121:6827/449222054', 'heartbeat_back_addr': '172.21.15.121:6831/449222054', 'heartbeat_front_addr': '172.21.15.121:6829/449222054', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 414, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6800', 'nonce': 2660128384}, {'type': 'v1', 'addr': '172.21.15.175:6801', 'nonce': 2660128384}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6802', 'nonce': 2660128384}, {'type': 'v1', 'addr': '172.21.15.175:6803', 'nonce': 2660128384}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6806', 'nonce': 2660128384}, {'type': 'v1', 'addr': '172.21.15.175:6807', 'nonce': 2660128384}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6804', 'nonce': 2660128384}, {'type': 'v1', 'addr': '172.21.15.175:6805', 'nonce': 2660128384}]}, 'public_addr': '172.21.15.175:6801/2660128384', 'cluster_addr': '172.21.15.175:6803/2660128384', 'heartbeat_back_addr': '172.21.15.175:6807/2660128384', 'heartbeat_front_addr': '172.21.15.175:6805/2660128384', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '50162e42-0443-41d1-8709-a0db61c087da', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 312, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6808', 'nonce': 1270991649}, {'type': 'v1', 'addr': '172.21.15.175:6809', 'nonce': 1270991649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6810', 'nonce': 1270991649}, {'type': 'v1', 'addr': '172.21.15.175:6811', 'nonce': 1270991649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6814', 'nonce': 1270991649}, {'type': 'v1', 'addr': '172.21.15.175:6815', 'nonce': 1270991649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6812', 'nonce': 1270991649}, {'type': 'v1', 'addr': '172.21.15.175:6813', 'nonce': 1270991649}]}, 'public_addr': '172.21.15.175:6809/1270991649', 'cluster_addr': '172.21.15.175:6811/1270991649', 'heartbeat_back_addr': '172.21.15.175:6815/1270991649', 'heartbeat_front_addr': '172.21.15.175:6813/1270991649', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '4b9426e1-bac8-47d3-ad58-54b89e8d1eb1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6816', 'nonce': 3877631021}, {'type': 'v1', 'addr': '172.21.15.175:6817', 'nonce': 3877631021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6818', 'nonce': 3877631021}, {'type': 'v1', 'addr': '172.21.15.175:6819', 'nonce': 3877631021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6822', 'nonce': 3877631021}, {'type': 'v1', 'addr': '172.21.15.175:6823', 'nonce': 3877631021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6820', 'nonce': 3877631021}, {'type': 'v1', 'addr': '172.21.15.175:6821', 'nonce': 3877631021}]}, 'public_addr': '172.21.15.175:6817/3877631021', 'cluster_addr': '172.21.15.175:6819/3877631021', 'heartbeat_back_addr': '172.21.15.175:6823/3877631021', 'heartbeat_front_addr': '172.21.15.175:6821/3877631021', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'cd1a28a1-413c-4274-9876-e1bdb7703be6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 449, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6824', 'nonce': 1150051236}, {'type': 'v1', 'addr': '172.21.15.175:6825', 'nonce': 1150051236}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6826', 'nonce': 1150051236}, {'type': 'v1', 'addr': '172.21.15.175:6827', 'nonce': 1150051236}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6830', 'nonce': 1150051236}, {'type': 'v1', 'addr': '172.21.15.175:6831', 'nonce': 1150051236}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6828', 'nonce': 1150051236}, {'type': 'v1', 'addr': '172.21.15.175:6829', 'nonce': 1150051236}]}, 'public_addr': '172.21.15.175:6825/1150051236', 'cluster_addr': '172.21.15.175:6827/1150051236', 'heartbeat_back_addr': '172.21.15.175:6831/1150051236', 'heartbeat_front_addr': '172.21.15.175:6829/1150051236', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:43:37.917862+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:44:19.071328+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:43:23.497634+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:43:50.817861+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:43:22.721092+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:43:31.672879+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-11T03:43:31.134086+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:44:12.815905+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:44:11.646575+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:44:05.536067+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:44:17.122767+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:43:45.212841+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '1.4', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '2.2', 'mappings': [{'from': 9, 'to': 11}]}, {'pgid': '2.d', 'mappings': [{'from': 9, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/108136515': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:0/3411515187': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:6801/3365920551': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:6800/3365920551': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:0/3885577441': '2024-09-12T03:31:03.077186+0000', '172.21.15.84:0/22039254': '2024-09-12T03:31:03.077186+0000', '172.21.15.84:6801/3496645948': '2024-09-12T03:31:03.077186+0000', '172.21.15.84:0/4206337830': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:6800/3496645948': '2024-09-12T03:31:03.077186+0000', '172.21.15.84:0/1989775875': '2024-09-12T03:31:03.077186+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 157, 'length': 1}, {'begin': 166, 'length': 1}, {'begin': 171, 'length': 1}, {'begin': 178, 'length': 1}, {'begin': 180, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T03:44:35.262 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-11T03:44:35.316 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:35.340 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:35.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:34 smithi121 ceph-mon[30729]: osdmap e456: 12 total, 12 up, 11 in 2024-09-11T03:44:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:34 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2129045555' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-11T03:44:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:34 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/375532163' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:35.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:35.468 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:35.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:35.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:35.570 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:35.591 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:35.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:35.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:35.694 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:35.715 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:35.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:35.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:35.817 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:35.838 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:35.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:35.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:35.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:35.944 INFO:tasks.rados.rados.2.smithi175.stdout:1670: done (1 left) 2024-09-11T03:44:35.944 INFO:tasks.rados.rados.2.smithi175.stdout:1671: snap_remove snap 167 2024-09-11T03:44:35.966 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:35.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:35.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:36.068 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:36.087 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:36.091 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:36.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:36.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:36.193 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:36.207 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:35 smithi084 ceph-mon[27254]: pgmap v840: 57 pgs: 57 active+clean; 178 MiB data, 798 MiB used, 983 GiB / 983 GiB avail; 6.1 MiB/s rd, 2.0 MiB/s wr, 8 op/s 2024-09-11T03:44:36.207 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:35 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:36.207 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:35 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/375532163' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:36.208 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:35 smithi084 ceph-mon[27254]: osdmap e457: 12 total, 12 up, 11 in 2024-09-11T03:44:36.219 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:36.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:36.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:35 smithi175 ceph-mon[30894]: pgmap v840: 57 pgs: 57 active+clean; 178 MiB data, 798 MiB used, 983 GiB / 983 GiB avail; 6.1 MiB/s rd, 2.0 MiB/s wr, 8 op/s 2024-09-11T03:44:36.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:35 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:36.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:35 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/375532163' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:36.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:35 smithi175 ceph-mon[30894]: osdmap e457: 12 total, 12 up, 11 in 2024-09-11T03:44:36.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:36.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:36.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:36.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:35 smithi121 ceph-mon[30729]: pgmap v840: 57 pgs: 57 active+clean; 178 MiB data, 798 MiB used, 983 GiB / 983 GiB avail; 6.1 MiB/s rd, 2.0 MiB/s wr, 8 op/s 2024-09-11T03:44:36.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:36 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:36.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/375532163' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:36.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:36 smithi121 ceph-mon[30729]: osdmap e457: 12 total, 12 up, 11 in 2024-09-11T03:44:36.447 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:36.472 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:36.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:36.575 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:36.600 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:36.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:36.601 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:36.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:36.729 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:36.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:36.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:36.748 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:44:36.749 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:44:36.755 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:44:36.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:44:36.771 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:44:36.771 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:44:36.787 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:44:36.787 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:44:36.804 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:44:36.805 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:44:36.821 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:44:36.821 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:44:36.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:36.838 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:44:36.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:44:36.852 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:36.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:36.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:36.856 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:44:36.856 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:44:36.874 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:44:36.875 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:44:36.891 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:44:36.891 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:44:36.908 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:44:36.908 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:44:36.925 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:44:36.925 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:44:36.933 INFO:tasks.rados.rados.2.smithi175.stdout:1671: done (1 left) 2024-09-11T03:44:36.934 INFO:tasks.rados.rados.2.smithi175.stdout:1672: rollback oid 45 current snap is 182 2024-09-11T03:44:36.934 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 45 to 181 2024-09-11T03:44:36.934 INFO:tasks.rados.rados.2.smithi175.stdout:1669: finishing copy_from racing read to smithi17557870-31 2024-09-11T03:44:36.934 INFO:tasks.rados.rados.2.smithi175.stdout:1669: finishing copy_from to smithi17557870-31 2024-09-11T03:44:36.934 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 522 (ObjNum 495 snap 143 seq_num 495) dirty exists 2024-09-11T03:44:36.934 INFO:tasks.rados.rados.2.smithi175.stdout:1669: done (1 left) 2024-09-11T03:44:36.934 INFO:tasks.rados.rados.2.smithi175.stdout:1673: write oid 26 current snap is 182 2024-09-11T03:44:36.934 INFO:tasks.rados.rados.2.smithi175.stdout:1673: seq_num 585 ranges {632095=463493,1616319=422534,2748686=487304} 2024-09-11T03:44:36.939 INFO:tasks.rados.rados.2.smithi175.stdout:1673: writing smithi17557870-26 from 632095 to 1095588 tid 1 2024-09-11T03:44:36.945 INFO:tasks.rados.rados.2.smithi175.stdout:1673: writing smithi17557870-26 from 1616319 to 2038853 tid 2 2024-09-11T03:44:36.952 INFO:tasks.rados.rados.2.smithi175.stdout:1673: writing smithi17557870-26 from 2748686 to 3235990 tid 3 2024-09-11T03:44:36.952 INFO:tasks.rados.rados.2.smithi175.stdout:1674: read oid 32 snap 172 2024-09-11T03:44:36.952 INFO:tasks.rados.rados.2.smithi175.stdout:1674: expect deleted 2024-09-11T03:44:36.953 INFO:tasks.rados.rados.2.smithi175.stdout:1675: snap_create 2024-09-11T03:44:36.953 INFO:tasks.rados.rados.2.smithi175.stdout:1672: finishing rollback tid 0 to smithi17557870-45 2024-09-11T03:44:36.953 INFO:tasks.rados.rados.2.smithi175.stdout:1672: finishing rollback tid 1 to smithi17557870-45 2024-09-11T03:44:36.953 INFO:tasks.rados.rados.2.smithi175.stdout:1672: finishing rollback tid 2 to smithi17557870-45 2024-09-11T03:44:36.953 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 481 (ObjNum 416 snap 127 seq_num 416) dirty exists 2024-09-11T03:44:36.953 INFO:tasks.rados.rados.2.smithi175.stdout:1673: finishing write tid 1 to smithi17557870-26 2024-09-11T03:44:36.953 INFO:tasks.rados.rados.2.smithi175.stdout:1673: finishing write tid 2 to smithi17557870-26 2024-09-11T03:44:36.955 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:36.955 INFO:tasks.rados.rados.2.smithi175.stdout:1673: finishing write tid 3 to smithi17557870-26 2024-09-11T03:44:36.956 INFO:tasks.rados.rados.2.smithi175.stdout:1673: finishing write tid 4 to smithi17557870-26 2024-09-11T03:44:36.956 INFO:tasks.rados.rados.2.smithi175.stdout:1673: finishing write tid 5 to smithi17557870-26 2024-09-11T03:44:36.956 INFO:tasks.rados.rados.2.smithi175.stdout:1673: oid 26 updating version 0 to 319 2024-09-11T03:44:36.956 INFO:tasks.rados.rados.2.smithi175.stdout:1673: oid 26 updating version 319 to 322 2024-09-11T03:44:36.956 INFO:tasks.rados.rados.2.smithi175.stdout:1673: oid 26 version 322 is already newer than 320 2024-09-11T03:44:36.956 INFO:tasks.rados.rados.2.smithi175.stdout:1673: oid 26 version 322 is already newer than 321 2024-09-11T03:44:36.956 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 322 (ObjNum 585 snap 182 seq_num 585) dirty exists 2024-09-11T03:44:36.956 INFO:tasks.rados.rados.2.smithi175.stdout:1673: left oid 26 (ObjNum 585 snap 182 seq_num 585) 2024-09-11T03:44:36.970 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:44:36.970 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:44:36.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:44:36.971 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:44:36.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.971 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:752 2.c e138/95 2:3c007163:::smithi17557870-35:head v 138'40, mlcod=135'37)", 2024-09-11T03:44:36.971 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:44:36.971 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 321.954707536, 2024-09-11T03:44:36.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084184749999999999, 2024-09-11T03:44:36.971 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:36.971 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:36.971 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:36.971 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:36.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:36.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:44:36.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:36.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:36.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:44:36.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:36.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:36.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:44:36.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8400000000000001e-07 2024-09-11T03:44:36.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:36.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021686+0000", 2024-09-11T03:44:36.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061118680000000003 2024-09-11T03:44:36.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:36.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021706+0000", 2024-09-11T03:44:36.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9942999999999999e-05 2024-09-11T03:44:36.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:36.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021717+0000", 2024-09-11T03:44:36.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.098e-05 2024-09-11T03:44:36.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:36.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021806+0000", 2024-09-11T03:44:36.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9197999999999995e-05 2024-09-11T03:44:36.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:36.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021834+0000", 2024-09-11T03:44:36.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8251000000000001e-05 2024-09-11T03:44:36.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:36.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.023931+0000", 2024-09-11T03:44:36.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020966639999999998 2024-09-11T03:44:36.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:36.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.023992+0000", 2024-09-11T03:44:36.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1387e-05 2024-09-11T03:44:36.975 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:36.975 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:36.975 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:36.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.976 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1175 2.c e185/95 2:32ade24e:::smithi17557870-45:head v 185'66, mlcod=182'62)", 2024-09-11T03:44:36.976 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:02.110649+0000", 2024-09-11T03:44:36.976 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 274.85963255199999, 2024-09-11T03:44:36.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00675758, 2024-09-11T03:44:36.976 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:36.976 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:36.976 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:36.976 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:36.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:36.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.110649+0000", 2024-09-11T03:44:36.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:36.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:36.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.110649+0000", 2024-09-11T03:44:36.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:36.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:36.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.110649+0000", 2024-09-11T03:44:36.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4699999999999998e-07 2024-09-11T03:44:36.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:36.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.115096+0000", 2024-09-11T03:44:36.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044473220000000001 2024-09-11T03:44:36.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:36.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.115114+0000", 2024-09-11T03:44:36.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7825e-05 2024-09-11T03:44:36.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:36.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.115124+0000", 2024-09-11T03:44:36.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5200000000000003e-06 2024-09-11T03:44:36.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.979 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:36.979 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.115202+0000", 2024-09-11T03:44:36.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8102000000000005e-05 2024-09-11T03:44:36.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.979 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:36.979 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.115231+0000", 2024-09-11T03:44:36.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8765e-05 2024-09-11T03:44:36.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:36.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.117364+0000", 2024-09-11T03:44:36.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021330630000000001 2024-09-11T03:44:36.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:36.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:02.117406+0000", 2024-09-11T03:44:36.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2735999999999999e-05 2024-09-11T03:44:36.980 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:36.980 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:36.980 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:36.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.981 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1285 2.1 e196/95 2:8638b65e:::smithi17557870-5:head v 196'52, mlcod=173'49)", 2024-09-11T03:44:36.981 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:12.850227+0000", 2024-09-11T03:44:36.981 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 264.120053918, 2024-09-11T03:44:36.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059143149999999999, 2024-09-11T03:44:36.981 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:36.981 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:36.981 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:36.981 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:36.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:36.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.850227+0000", 2024-09-11T03:44:36.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:36.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:36.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.850227+0000", 2024-09-11T03:44:36.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:36.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:36.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.850228+0000", 2024-09-11T03:44:36.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6199999999999997e-07 2024-09-11T03:44:36.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:36.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853054+0000", 2024-09-11T03:44:36.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028262000000000001 2024-09-11T03:44:36.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:36.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853087+0000", 2024-09-11T03:44:36.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2724e-05 2024-09-11T03:44:36.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:36.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853100+0000", 2024-09-11T03:44:36.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3456e-05 2024-09-11T03:44:36.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:36.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853187+0000", 2024-09-11T03:44:36.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6211999999999994e-05 2024-09-11T03:44:36.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:36.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.853216+0000", 2024-09-11T03:44:36.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9825999999999999e-05 2024-09-11T03:44:36.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.985 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:36.985 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856078+0000", 2024-09-11T03:44:36.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028619689999999998 2024-09-11T03:44:36.985 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.985 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:36.985 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856142+0000", 2024-09-11T03:44:36.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3266e-05 2024-09-11T03:44:36.985 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:36.985 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:36.985 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:36.985 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.986 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1513 2.5 e212/95 2:a6e3bbd9:::smithi17557870-38:head v 212'178, mlcod=203'174)", 2024-09-11T03:44:36.986 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:28.928696+0000", 2024-09-11T03:44:36.986 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 248.041585629, 2024-09-11T03:44:36.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0099738110000000008, 2024-09-11T03:44:36.986 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:36.986 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:36.986 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:36.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:36.988 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:36.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.988 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:36.988 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.928696+0000", 2024-09-11T03:44:36.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:36.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.988 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:36.988 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.928696+0000", 2024-09-11T03:44:36.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:36.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:36.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.928696+0000", 2024-09-11T03:44:36.989 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9500000000000001e-07 2024-09-11T03:44:36.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:36.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935839+0000", 2024-09-11T03:44:36.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.007143473 2024-09-11T03:44:36.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.990 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:36.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935850+0000", 2024-09-11T03:44:36.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.026e-05 2024-09-11T03:44:36.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.990 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:36.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935854+0000", 2024-09-11T03:44:36.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8080000000000004e-06 2024-09-11T03:44:36.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:36.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935903+0000", 2024-09-11T03:44:36.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8380000000000001e-05 2024-09-11T03:44:36.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:36.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935917+0000", 2024-09-11T03:44:36.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4192999999999999e-05 2024-09-11T03:44:36.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:36.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.938638+0000", 2024-09-11T03:44:36.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002721298 2024-09-11T03:44:36.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:36.992 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.938669+0000", 2024-09-11T03:44:36.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1204000000000001e-05 2024-09-11T03:44:36.992 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:36.992 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:36.992 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:36.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.992 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2112 2.4 e268/188 2:2ef19c88:::smithi17557870-15:head v 268'153, mlcod=264'150)", 2024-09-11T03:44:36.992 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:24.873716+0000", 2024-09-11T03:44:36.992 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 192.09656549900001, 2024-09-11T03:44:36.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065806770000000001, 2024-09-11T03:44:36.993 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:36.993 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:36.993 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:36.993 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:36.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:36.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.873716+0000", 2024-09-11T03:44:36.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:36.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:36.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.873716+0000", 2024-09-11T03:44:36.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:36.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.994 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:36.994 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.873716+0000", 2024-09-11T03:44:36.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0699999999999998e-07 2024-09-11T03:44:36.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.994 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:36.994 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.875592+0000", 2024-09-11T03:44:36.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001875413 2024-09-11T03:44:36.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.994 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:36.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.875615+0000", 2024-09-11T03:44:36.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3884999999999999e-05 2024-09-11T03:44:36.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.995 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:36.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.875629+0000", 2024-09-11T03:44:36.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.366e-05 2024-09-11T03:44:36.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.995 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:36.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.875707+0000", 2024-09-11T03:44:36.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8301999999999997e-05 2024-09-11T03:44:36.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:36.996 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.875733+0000", 2024-09-11T03:44:36.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5307999999999999e-05 2024-09-11T03:44:36.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:36.996 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.880233+0000", 2024-09-11T03:44:36.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00450052 2024-09-11T03:44:36.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:36.996 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.880296+0000", 2024-09-11T03:44:36.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3281999999999995e-05 2024-09-11T03:44:36.997 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:36.997 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:36.997 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:36.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.997 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2115 2.5 e268/95 2:a6e3bbd9:::smithi17557870-38:head v 268'256, mlcod=260'252)", 2024-09-11T03:44:36.997 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:24.876744+0000", 2024-09-11T03:44:36.997 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 192.093536962, 2024-09-11T03:44:36.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01118442, 2024-09-11T03:44:36.997 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:36.997 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:36.997 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:36.998 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:36.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:36.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.876744+0000", 2024-09-11T03:44:36.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:36.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:36.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.876744+0000", 2024-09-11T03:44:36.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:36.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:36.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.876745+0000", 2024-09-11T03:44:36.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8999999999999998e-07 2024-09-11T03:44:36.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:36.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.885165+0000", 2024-09-11T03:44:36.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084204060000000001 2024-09-11T03:44:36.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:36.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:36.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:37.000 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.885185+0000", 2024-09-11T03:44:37.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0432999999999999e-05 2024-09-11T03:44:37.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:37.000 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.885203+0000", 2024-09-11T03:44:37.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7569000000000001e-05 2024-09-11T03:44:37.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.000 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.885268+0000", 2024-09-11T03:44:37.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5104999999999994e-05 2024-09-11T03:44:37.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.001 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.001 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:37.001 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.885289+0000", 2024-09-11T03:44:37.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0800000000000001e-05 2024-09-11T03:44:37.001 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.001 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.001 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:37.001 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.887888+0000", 2024-09-11T03:44:37.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025986999999999998 2024-09-11T03:44:37.001 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.001 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.001 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:37.001 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.887929+0000", 2024-09-11T03:44:37.002 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1117e-05 2024-09-11T03:44:37.002 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.002 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:37.002 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.002 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.002 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.002 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2127 2.4 e268/188 2:2ef19c88:::smithi17557870-15:head v 268'154, mlcod=265'152)", 2024-09-11T03:44:37.002 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:25.035224+0000", 2024-09-11T03:44:37.002 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 191.93505763799999, 2024-09-11T03:44:37.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048299429999999997, 2024-09-11T03:44:37.003 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:37.003 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:37.003 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:37.003 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:37.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.003 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:37.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.035224+0000", 2024-09-11T03:44:37.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:37.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.035224+0000", 2024-09-11T03:44:37.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:37.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.035226+0000", 2024-09-11T03:44:37.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.024e-06 2024-09-11T03:44:37.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.006 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.006 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:37.006 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.035794+0000", 2024-09-11T03:44:37.006 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00056879499999999998 2024-09-11T03:44:37.006 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.006 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.006 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:37.006 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.035810+0000", 2024-09-11T03:44:37.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5361000000000001e-05 2024-09-11T03:44:37.007 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:37.007 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.035816+0000", 2024-09-11T03:44:37.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3790000000000003e-06 2024-09-11T03:44:37.007 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.007 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.035876+0000", 2024-09-11T03:44:37.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9392999999999999e-05 2024-09-11T03:44:37.007 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:37.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.035899+0000", 2024-09-11T03:44:37.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3706e-05 2024-09-11T03:44:37.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:37.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.039991+0000", 2024-09-11T03:44:37.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040916629999999997 2024-09-11T03:44:37.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:37.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.040054+0000", 2024-09-11T03:44:37.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2621999999999998e-05 2024-09-11T03:44:37.008 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.009 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:37.009 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.009 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.009 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2140 2.c e268/95 2:3c007163:::smithi17557870-35:head v 268'210, mlcod=267'207)", 2024-09-11T03:44:37.009 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:25.066817+0000", 2024-09-11T03:44:37.009 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 191.903464746, 2024-09-11T03:44:37.009 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069600249999999999, 2024-09-11T03:44:37.009 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:37.009 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:37.009 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:37.009 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:37.010 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.010 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:37.010 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.066817+0000", 2024-09-11T03:44:37.010 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.010 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.010 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.010 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:37.010 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.066817+0000", 2024-09-11T03:44:37.010 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.010 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.010 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.011 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:37.011 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.066817+0000", 2024-09-11T03:44:37.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1099999999999995e-07 2024-09-11T03:44:37.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.011 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.011 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:37.011 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.072136+0000", 2024-09-11T03:44:37.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053183930000000003 2024-09-11T03:44:37.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.012 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.012 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:37.012 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.072164+0000", 2024-09-11T03:44:37.012 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8603999999999999e-05 2024-09-11T03:44:37.012 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.012 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.012 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:37.012 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.072178+0000", 2024-09-11T03:44:37.012 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3733e-05 2024-09-11T03:44:37.012 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.012 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.012 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.012 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.072237+0000", 2024-09-11T03:44:37.013 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9072e-05 2024-09-11T03:44:37.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.013 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:37.013 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.072255+0000", 2024-09-11T03:44:37.013 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7625000000000002e-05 2024-09-11T03:44:37.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:37.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.073708+0000", 2024-09-11T03:44:37.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001453668 2024-09-11T03:44:37.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:37.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:25.073777+0000", 2024-09-11T03:44:37.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8318999999999994e-05 2024-09-11T03:44:37.015 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.015 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:37.015 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.016 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.8.0:443 2.5 e279/276 2:a6e3bbd9:::smithi17557870-38:5d v 279'283, mlcod=278'279)", 2024-09-11T03:44:37.016 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:35.865183+0000", 2024-09-11T03:44:37.016 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 181.105098723, 2024-09-11T03:44:37.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011482019, 2024-09-11T03:44:37.016 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:37.016 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:37.016 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:37.016 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:37.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:37.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865183+0000", 2024-09-11T03:44:37.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.017 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:37.017 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865183+0000", 2024-09-11T03:44:37.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.017 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:37.017 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865183+0000", 2024-09-11T03:44:37.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0399999999999996e-07 2024-09-11T03:44:37.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.017 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:37.018 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865190+0000", 2024-09-11T03:44:37.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9460000000000001e-06 2024-09-11T03:44:37.018 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.018 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.018 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:37.018 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865199+0000", 2024-09-11T03:44:37.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7979999999999995e-06 2024-09-11T03:44:37.018 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.018 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.018 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:37.018 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865214+0000", 2024-09-11T03:44:37.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5184999999999999e-05 2024-09-11T03:44:37.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865278+0000", 2024-09-11T03:44:37.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4086999999999998e-05 2024-09-11T03:44:37.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:37.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865303+0000", 2024-09-11T03:44:37.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.446e-05 2024-09-11T03:44:37.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:37.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.876628+0000", 2024-09-11T03:44:37.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011325574 2024-09-11T03:44:37.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:37.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.876665+0000", 2024-09-11T03:44:37.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6464999999999997e-05 2024-09-11T03:44:37.021 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.021 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:37.022 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.022 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2241 2.14 2:2ef19c88:::smithi17557870-15:58 [read 0~3327819,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:44:37.022 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:36.886222+0000", 2024-09-11T03:44:37.022 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 180.084059758, 2024-09-11T03:44:37.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012098761, 2024-09-11T03:44:37.022 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:37.022 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:37.022 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:44:37.022 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:37.022 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:37.023 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:37.023 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2241 2024-09-11T03:44:37.023 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.023 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:37.023 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.023 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:37.023 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886222+0000", 2024-09-11T03:44:37.023 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.023 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.023 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.023 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:37.023 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886222+0000", 2024-09-11T03:44:37.023 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.023 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:37.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886223+0000", 2024-09-11T03:44:37.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4529999999999999e-06 2024-09-11T03:44:37.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:37.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886228+0000", 2024-09-11T03:44:37.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9869999999999999e-06 2024-09-11T03:44:37.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:37.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886239+0000", 2024-09-11T03:44:37.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.08e-05 2024-09-11T03:44:37.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:37.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886249+0000", 2024-09-11T03:44:37.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0322e-05 2024-09-11T03:44:37.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886366+0000", 2024-09-11T03:44:37.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000116549 2024-09-11T03:44:37.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:44:37.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886375+0000", 2024-09-11T03:44:37.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3870000000000001e-06 2024-09-11T03:44:37.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.893120+0000", 2024-09-11T03:44:37.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067446340000000002 2024-09-11T03:44:37.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:37.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.893275+0000", 2024-09-11T03:44:37.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000155079 2024-09-11T03:44:37.027 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.027 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:37.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.898320+0000", 2024-09-11T03:44:37.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0050455500000000002 2024-09-11T03:44:37.027 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.027 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:37.027 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.027 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.027 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.8.0:445 2.5 e280/276 2:a286c222:::smithi17557870-29:5b v 280'285, mlcod=278'281)", 2024-09-11T03:44:37.027 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:36.886383+0000", 2024-09-11T03:44:37.028 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 180.08389835899999, 2024-09-11T03:44:37.028 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.004457323, 2024-09-11T03:44:37.028 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:37.028 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:37.028 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:37.028 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:37.028 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.028 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:37.028 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886383+0000", 2024-09-11T03:44:37.028 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.028 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.028 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.028 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:37.029 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886383+0000", 2024-09-11T03:44:37.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.029 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.029 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.029 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:37.029 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886383+0000", 2024-09-11T03:44:37.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.51e-07 2024-09-11T03:44:37.029 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.029 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.029 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:37.029 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886389+0000", 2024-09-11T03:44:37.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7379999999999999e-06 2024-09-11T03:44:37.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:37.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886398+0000", 2024-09-11T03:44:37.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1959999999999997e-06 2024-09-11T03:44:37.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:37.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886404+0000", 2024-09-11T03:44:37.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6749999999999999e-06 2024-09-11T03:44:37.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886471+0000", 2024-09-11T03:44:37.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7044000000000004e-05 2024-09-11T03:44:37.031 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:37.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886493+0000", 2024-09-11T03:44:37.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1956000000000001e-05 2024-09-11T03:44:37.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:37.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.890814+0000", 2024-09-11T03:44:37.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043206260000000002 2024-09-11T03:44:37.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:37.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.890840+0000", 2024-09-11T03:44:37.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6537e-05 2024-09-11T03:44:37.033 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.033 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:37.033 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.033 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.033 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.033 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2242 2.14 2:2ef19c88:::smithi17557870-15:58 [sparse-read 0~2125189] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:44:37.033 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:36.886426+0000", 2024-09-11T03:44:37.033 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 180.08385480999999, 2024-09-11T03:44:37.033 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012666422, 2024-09-11T03:44:37.033 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:37.033 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:37.033 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:44:37.033 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:37.034 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:37.034 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:37.034 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2242 2024-09-11T03:44:37.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.034 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:37.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.034 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:37.034 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886426+0000", 2024-09-11T03:44:37.034 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.034 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:37.034 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886426+0000", 2024-09-11T03:44:37.034 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:37.035 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886427+0000", 2024-09-11T03:44:37.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2799999999999996e-07 2024-09-11T03:44:37.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:37.035 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886431+0000", 2024-09-11T03:44:37.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7910000000000002e-06 2024-09-11T03:44:37.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:37.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886439+0000", 2024-09-11T03:44:37.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.038e-06 2024-09-11T03:44:37.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.036 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:37.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886445+0000", 2024-09-11T03:44:37.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3799999999999999e-06 2024-09-11T03:44:37.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.036 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886527+0000", 2024-09-11T03:44:37.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1797000000000003e-05 2024-09-11T03:44:37.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:44:37.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886534+0000", 2024-09-11T03:44:37.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2599999999999999e-06 2024-09-11T03:44:37.037 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.037 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.898657+0000", 2024-09-11T03:44:37.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012123073 2024-09-11T03:44:37.037 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.037 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:37.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.898831+0000", 2024-09-11T03:44:37.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000173931 2024-09-11T03:44:37.038 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:37.038 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.899093+0000", 2024-09-11T03:44:37.038 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026152400000000001 2024-09-11T03:44:37.038 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.038 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:37.038 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.038 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.038 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2243 2.14 2:2ef19c88:::smithi17557870-15:58 [read 0~3327819,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:44:37.038 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:36.886461+0000", 2024-09-11T03:44:37.038 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 180.08382004200001, 2024-09-11T03:44:37.038 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014874943999999999, 2024-09-11T03:44:37.039 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:37.039 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:37.039 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:44:37.039 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:37.039 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:37.039 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:37.039 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2243 2024-09-11T03:44:37.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.039 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:37.039 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.039 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:37.039 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886461+0000", 2024-09-11T03:44:37.039 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.040 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.040 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.040 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:37.040 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886461+0000", 2024-09-11T03:44:37.040 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.040 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.040 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.040 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:37.040 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886462+0000", 2024-09-11T03:44:37.040 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1899999999999998e-07 2024-09-11T03:44:37.040 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.040 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.040 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:37.040 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886464+0000", 2024-09-11T03:44:37.041 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.632e-06 2024-09-11T03:44:37.041 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.041 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:37.041 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886470+0000", 2024-09-11T03:44:37.041 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.772e-06 2024-09-11T03:44:37.041 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.041 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:37.041 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886474+0000", 2024-09-11T03:44:37.041 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2549999999999999e-06 2024-09-11T03:44:37.041 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.042 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.042 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886554+0000", 2024-09-11T03:44:37.042 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9851999999999994e-05 2024-09-11T03:44:37.042 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.042 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.042 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:44:37.042 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886560+0000", 2024-09-11T03:44:37.042 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.117e-06 2024-09-11T03:44:37.042 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.042 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.042 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.042 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.899326+0000", 2024-09-11T03:44:37.042 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012765841999999999 2024-09-11T03:44:37.042 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.043 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.043 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:37.043 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.899468+0000", 2024-09-11T03:44:37.043 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014217099999999999 2024-09-11T03:44:37.043 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.043 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.043 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:37.043 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.901336+0000", 2024-09-11T03:44:37.043 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001867884 2024-09-11T03:44:37.043 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.043 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:37.043 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.043 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.043 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.044 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.8.0:446 2.5 e280/276 2:a6e3bbd9:::smithi17557870-38:5d v 280'287, mlcod=278'281)", 2024-09-11T03:44:37.044 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:36.886860+0000", 2024-09-11T03:44:37.044 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 180.08342081699999, 2024-09-11T03:44:37.044 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043266240000000003, 2024-09-11T03:44:37.044 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:37.044 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:37.044 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:37.044 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:37.044 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.044 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:37.044 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886860+0000", 2024-09-11T03:44:37.044 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.044 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:37.045 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886860+0000", 2024-09-11T03:44:37.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:37.045 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886861+0000", 2024-09-11T03:44:37.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0800000000000005e-07 2024-09-11T03:44:37.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:37.045 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886866+0000", 2024-09-11T03:44:37.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6800000000000001e-06 2024-09-11T03:44:37.046 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.046 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.046 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:37.046 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886875+0000", 2024-09-11T03:44:37.046 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9779999999999994e-06 2024-09-11T03:44:37.046 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.046 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.046 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:37.046 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886881+0000", 2024-09-11T03:44:37.046 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0129999999999999e-06 2024-09-11T03:44:37.046 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.046 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.046 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.046 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886931+0000", 2024-09-11T03:44:37.047 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9962e-05 2024-09-11T03:44:37.047 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.047 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.047 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:37.047 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886949+0000", 2024-09-11T03:44:37.047 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8615999999999999e-05 2024-09-11T03:44:37.047 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.047 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.047 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:37.047 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.891157+0000", 2024-09-11T03:44:37.047 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042076609999999997 2024-09-11T03:44:37.047 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.047 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.048 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:37.048 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.891187+0000", 2024-09-11T03:44:37.048 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0205999999999998e-05 2024-09-11T03:44:37.048 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.048 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:37.048 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.048 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.048 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.048 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2443 2.12 e296/281 2:494ba932:::smithi17557870-43:head v 296'131, mlcod=291'128)", 2024-09-11T03:44:37.048 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:52.895764+0000", 2024-09-11T03:44:37.048 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 164.07451776600001, 2024-09-11T03:44:37.048 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068629390000000002, 2024-09-11T03:44:37.048 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:37.048 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:37.049 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:37.049 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:37.049 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.049 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:37.049 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.895764+0000", 2024-09-11T03:44:37.049 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.049 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.049 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.049 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:37.049 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.895764+0000", 2024-09-11T03:44:37.049 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.049 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.049 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.050 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:37.050 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.895764+0000", 2024-09-11T03:44:37.050 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2099999999999998e-07 2024-09-11T03:44:37.050 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.050 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.050 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:37.050 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.898075+0000", 2024-09-11T03:44:37.050 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0023108790000000001 2024-09-11T03:44:37.050 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.050 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.050 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:37.050 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.898091+0000", 2024-09-11T03:44:37.050 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6293000000000001e-05 2024-09-11T03:44:37.050 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.051 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:37.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.898104+0000", 2024-09-11T03:44:37.051 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2539e-05 2024-09-11T03:44:37.051 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.051 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.898160+0000", 2024-09-11T03:44:37.051 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.643e-05 2024-09-11T03:44:37.051 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.051 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:37.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.898186+0000", 2024-09-11T03:44:37.052 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5666000000000001e-05 2024-09-11T03:44:37.052 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.052 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.052 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:37.052 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.902581+0000", 2024-09-11T03:44:37.052 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043955089999999997 2024-09-11T03:44:37.052 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.052 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.052 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:37.052 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.902626+0000", 2024-09-11T03:44:37.052 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5302000000000001e-05 2024-09-11T03:44:37.052 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.052 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:37.052 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.053 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.053 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.053 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2492 2.5 e302/276 2:a1a99782:::smithi17557870-1:head v 302'331, mlcod=297'328)", 2024-09-11T03:44:37.053 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:58.902093+0000", 2024-09-11T03:44:37.053 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 158.06818836299999, 2024-09-11T03:44:37.053 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087348040000000005, 2024-09-11T03:44:37.053 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:37.053 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:37.053 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:37.053 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:37.053 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.053 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:37.053 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.902093+0000", 2024-09-11T03:44:37.053 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.054 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.054 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.054 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:37.054 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.902093+0000", 2024-09-11T03:44:37.054 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:37.059 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:37.059 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.060 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.060 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.060 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:37.060 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.902093+0000", 2024-09-11T03:44:37.060 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.11e-07 2024-09-11T03:44:37.060 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.060 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.060 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:37.060 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.908258+0000", 2024-09-11T03:44:37.060 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061649770000000003 2024-09-11T03:44:37.060 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.060 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.061 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:37.061 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.908276+0000", 2024-09-11T03:44:37.061 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7737e-05 2024-09-11T03:44:37.061 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.061 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.061 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:37.061 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.908284+0000", 2024-09-11T03:44:37.061 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2490000000000001e-06 2024-09-11T03:44:37.061 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.061 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.061 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.061 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.908348+0000", 2024-09-11T03:44:37.061 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3712000000000003e-05 2024-09-11T03:44:37.062 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.062 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.062 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:37.062 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.908367+0000", 2024-09-11T03:44:37.062 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9633e-05 2024-09-11T03:44:37.062 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.062 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.062 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:37.062 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.910624+0000", 2024-09-11T03:44:37.062 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022564680000000002 2024-09-11T03:44:37.062 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.062 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.062 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:37.062 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.910828+0000", 2024-09-11T03:44:37.063 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00020381699999999999 2024-09-11T03:44:37.063 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.063 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:37.063 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.063 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.063 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.063 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2931 2.5 e346/276 2:a6d17fc0:::smithi17557870-13:head v 346'401, mlcod=340'397)", 2024-09-11T03:44:37.063 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:43.125646+0000", 2024-09-11T03:44:37.063 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 113.844635215, 2024-09-11T03:44:37.063 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056773580000000004, 2024-09-11T03:44:37.063 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:37.063 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:37.063 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:37.063 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:37.064 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.064 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:37.064 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.125646+0000", 2024-09-11T03:44:37.064 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.064 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.064 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.064 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:37.064 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.125646+0000", 2024-09-11T03:44:37.064 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.064 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.064 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.064 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:37.064 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.125646+0000", 2024-09-11T03:44:37.065 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6899999999999999e-07 2024-09-11T03:44:37.065 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.065 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.065 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:37.065 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.129356+0000", 2024-09-11T03:44:37.065 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037093819999999998 2024-09-11T03:44:37.065 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.065 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.065 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:37.065 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.129373+0000", 2024-09-11T03:44:37.065 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7028e-05 2024-09-11T03:44:37.065 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.065 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.065 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:37.066 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.129381+0000", 2024-09-11T03:44:37.066 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3380000000000005e-06 2024-09-11T03:44:37.066 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.066 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.066 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.066 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.129437+0000", 2024-09-11T03:44:37.066 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6001000000000001e-05 2024-09-11T03:44:37.066 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.066 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.066 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:37.066 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.129456+0000", 2024-09-11T03:44:37.066 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8916999999999999e-05 2024-09-11T03:44:37.066 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.067 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.067 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:37.067 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.131287+0000", 2024-09-11T03:44:37.067 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018311340000000001 2024-09-11T03:44:37.067 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.067 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.067 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:37.067 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.131323+0000", 2024-09-11T03:44:37.067 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6389e-05 2024-09-11T03:44:37.067 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.067 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:37.067 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.067 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.067 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.068 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.1.0:635 2.4 e353/276 2:25f5f8b9:::smithi17557870-26:80 v 353'215, mlcod=352'209)", 2024-09-11T03:44:37.068 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:50.003105+0000", 2024-09-11T03:44:37.068 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 106.967176321, 2024-09-11T03:44:37.068 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.004696581, 2024-09-11T03:44:37.068 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:37.068 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:37.068 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:37.068 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:37.068 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.069 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:37.069 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.003105+0000", 2024-09-11T03:44:37.069 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.069 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.069 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.069 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:37.069 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.003105+0000", 2024-09-11T03:44:37.069 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.069 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.069 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.069 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:37.069 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.003106+0000", 2024-09-11T03:44:37.069 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.04e-06 2024-09-11T03:44:37.069 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.070 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.070 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:37.070 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.003113+0000", 2024-09-11T03:44:37.070 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7270000000000001e-06 2024-09-11T03:44:37.070 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.070 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.070 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:37.070 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.003135+0000", 2024-09-11T03:44:37.070 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2558999999999999e-05 2024-09-11T03:44:37.070 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.070 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.070 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:37.070 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.003152+0000", 2024-09-11T03:44:37.070 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6245000000000001e-05 2024-09-11T03:44:37.071 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.071 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.071 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.071 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.003200+0000", 2024-09-11T03:44:37.071 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8381999999999999e-05 2024-09-11T03:44:37.071 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.071 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.071 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:37.071 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.003220+0000", 2024-09-11T03:44:37.071 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0230000000000001e-05 2024-09-11T03:44:37.071 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.071 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.071 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:37.071 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.007275+0000", 2024-09-11T03:44:37.072 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040553179999999996 2024-09-11T03:44:37.072 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.072 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.072 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:37.072 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:50.007802+0000", 2024-09-11T03:44:37.072 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00052607999999999995 2024-09-11T03:44:37.072 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.072 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:37.072 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.072 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.072 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.072 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:3234 2.19 e361/281 2:98bb65f5:::smithi17557870-6:head v 361'386, mlcod=359'383)", 2024-09-11T03:44:37.072 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:58.150912+0000", 2024-09-11T03:44:37.072 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 98.819369582999997, 2024-09-11T03:44:37.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0057244130000000002, 2024-09-11T03:44:37.073 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:37.073 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:37.073 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:37.073 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:37.073 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.073 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:37.073 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.150912+0000", 2024-09-11T03:44:37.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.073 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.073 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.073 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:37.073 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.150912+0000", 2024-09-11T03:44:37.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:37.074 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.150913+0000", 2024-09-11T03:44:37.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.06e-06 2024-09-11T03:44:37.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:37.074 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.153675+0000", 2024-09-11T03:44:37.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002761955 2024-09-11T03:44:37.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:37.075 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.153697+0000", 2024-09-11T03:44:37.075 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1784999999999999e-05 2024-09-11T03:44:37.075 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.075 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.075 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:37.075 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.153713+0000", 2024-09-11T03:44:37.075 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6560000000000001e-05 2024-09-11T03:44:37.075 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.075 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.075 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.075 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.153804+0000", 2024-09-11T03:44:37.075 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0946999999999995e-05 2024-09-11T03:44:37.075 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.075 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.076 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:37.076 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.153846+0000", 2024-09-11T03:44:37.076 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2348000000000002e-05 2024-09-11T03:44:37.076 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.076 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.076 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:37.076 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.156588+0000", 2024-09-11T03:44:37.076 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027418350000000002 2024-09-11T03:44:37.076 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.076 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.076 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:37.077 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:58.156636+0000", 2024-09-11T03:44:37.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7923000000000002e-05 2024-09-11T03:44:37.077 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.077 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:37.077 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.077 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.077 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.077 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.11.0:417 2.c e452/95 2:3c007163:::smithi17557870-35:b5 v 452'451, mlcod=451'448)", 2024-09-11T03:44:37.077 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:44:29.938400+0000", 2024-09-11T03:44:37.077 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 7.0318813789999997, 2024-09-11T03:44:37.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045257279999999997, 2024-09-11T03:44:37.077 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:37.077 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:37.077 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:37.078 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:37.078 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.078 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:37.078 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:29.938400+0000", 2024-09-11T03:44:37.078 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.078 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.078 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.078 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:37.078 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:29.938400+0000", 2024-09-11T03:44:37.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:37.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:37.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:29.938400+0000", 2024-09-11T03:44:37.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.58e-07 2024-09-11T03:44:37.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:37.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:29.938405+0000", 2024-09-11T03:44:37.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7380000000000002e-06 2024-09-11T03:44:37.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:37.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:29.938421+0000", 2024-09-11T03:44:37.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6416999999999999e-05 2024-09-11T03:44:37.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.080 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.080 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:37.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:29.938430+0000", 2024-09-11T03:44:37.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6929999999999996e-06 2024-09-11T03:44:37.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.080 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.080 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:37.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:29.939494+0000", 2024-09-11T03:44:37.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001063523 2024-09-11T03:44:37.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.080 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.081 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:37.081 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:29.939516+0000", 2024-09-11T03:44:37.081 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.196e-05 2024-09-11T03:44:37.081 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.081 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.081 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:37.081 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:29.942869+0000", 2024-09-11T03:44:37.081 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033532000000000002 2024-09-11T03:44:37.081 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:37.081 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:37.081 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:37.081 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:29.942926+0000", 2024-09-11T03:44:37.081 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6839000000000002e-05 2024-09-11T03:44:37.081 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.082 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:37.082 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.082 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:37.082 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:37.082 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:44:37.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:36 smithi084 ceph-mon[27254]: osdmap e458: 12 total, 12 up, 11 in 2024-09-11T03:44:37.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2790903624' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:37.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:36 smithi084 ceph-mon[27254]: osdmap e459: 12 total, 12 up, 11 in 2024-09-11T03:44:37.159 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:37.185 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:37.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:37.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:36 smithi175 ceph-mon[30894]: osdmap e458: 12 total, 12 up, 11 in 2024-09-11T03:44:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2790903624' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:36 smithi175 ceph-mon[30894]: osdmap e459: 12 total, 12 up, 11 in 2024-09-11T03:44:37.287 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:37.313 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:37.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:37.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:37.415 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:37.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:36 smithi121 ceph-mon[30729]: osdmap e458: 12 total, 12 up, 11 in 2024-09-11T03:44:37.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2790903624' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:36 smithi121 ceph-mon[30729]: osdmap e459: 12 total, 12 up, 11 in 2024-09-11T03:44:37.441 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:37.442 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:37.543 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:37.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:37.570 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:37.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:37.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:37.696 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:37.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:37.798 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:37.824 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:37.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:37.825 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:37.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:37.952 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:37.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:37.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:37.965 INFO:tasks.rados.rados.2.smithi175.stdout:1672: done (3 left) 2024-09-11T03:44:37.965 INFO:tasks.rados.rados.2.smithi175.stdout:1673: done (2 left) 2024-09-11T03:44:37.965 INFO:tasks.rados.rados.2.smithi175.stdout:1674: done (1 left) 2024-09-11T03:44:37.965 INFO:tasks.rados.rados.2.smithi175.stdout:1675: done (0 left) 2024-09-11T03:44:37.965 INFO:tasks.rados.rados.2.smithi175.stdout:1676: delete oid 47 current snap is 183 2024-09-11T03:44:37.969 INFO:tasks.rados.rados.2.smithi175.stdout:1676: done (0 left) 2024-09-11T03:44:37.969 INFO:tasks.rados.rados.2.smithi175.stdout:1677: read oid 13 snap 172 2024-09-11T03:44:37.969 INFO:tasks.rados.rados.2.smithi175.stdout:1677: expect deleted 2024-09-11T03:44:37.969 INFO:tasks.rados.rados.2.smithi175.stdout:1678: read oid 40 snap -1 2024-09-11T03:44:37.969 INFO:tasks.rados.rados.2.smithi175.stdout:1678: expect (ObjNum 487 snap 141 seq_num 487) 2024-09-11T03:44:37.969 INFO:tasks.rados.rados.2.smithi175.stdout:1679: write oid 14 current snap is 183 2024-09-11T03:44:37.969 INFO:tasks.rados.rados.2.smithi175.stdout:1679: seq_num 587 ranges {665709=730344,2133283=426203,3203532=509848} 2024-09-11T03:44:37.976 INFO:tasks.rados.rados.2.smithi175.stdout:1679: writing smithi17557870-14 from 665709 to 1396053 tid 1 2024-09-11T03:44:37.983 INFO:tasks.rados.rados.2.smithi175.stdout:1679: writing smithi17557870-14 from 2133283 to 2559486 tid 2 2024-09-11T03:44:37.989 INFO:tasks.rados.rados.2.smithi175.stdout:1679: writing smithi17557870-14 from 3203532 to 3713380 tid 3 2024-09-11T03:44:37.989 INFO:tasks.rados.rados.2.smithi175.stdout:1680: delete oid 16 current snap is 183 2024-09-11T03:44:37.989 INFO:tasks.rados.rados.2.smithi175.stdout:1678: expect (ObjNum 487 snap 141 seq_num 487) 2024-09-11T03:44:38.054 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:38.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:44:38.059 INFO:tasks.rados.rados.2.smithi175.stdout:1679: finishing write tid 1 to smithi17557870-14 2024-09-11T03:44:38.059 INFO:tasks.rados.rados.2.smithi175.stdout:1679: finishing write tid 2 to smithi17557870-14 2024-09-11T03:44:38.059 INFO:tasks.rados.rados.2.smithi175.stdout:1679: finishing write tid 3 to smithi17557870-14 2024-09-11T03:44:38.060 INFO:tasks.rados.rados.2.smithi175.stdout:1679: finishing write tid 4 to smithi17557870-14 2024-09-11T03:44:38.060 INFO:tasks.rados.rados.2.smithi175.stdout:1679: finishing write tid 5 to smithi17557870-14 2024-09-11T03:44:38.060 INFO:tasks.rados.rados.2.smithi175.stdout:1679: oid 14 updating version 0 to 676 2024-09-11T03:44:38.060 INFO:tasks.rados.rados.2.smithi175.stdout:1679: oid 14 updating version 676 to 677 2024-09-11T03:44:38.060 INFO:tasks.rados.rados.2.smithi175.stdout:1679: oid 14 version 677 is already newer than 675 2024-09-11T03:44:38.060 INFO:tasks.rados.rados.2.smithi175.stdout:1679: oid 14 updating version 677 to 678 2024-09-11T03:44:38.060 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 678 (ObjNum 587 snap 183 seq_num 587) dirty exists 2024-09-11T03:44:38.060 INFO:tasks.rados.rados.2.smithi175.stdout:1679: left oid 14 (ObjNum 587 snap 183 seq_num 587) 2024-09-11T03:44:38.060 INFO:tasks.rados.rados.2.smithi175.stdout:1677: done (3 left) 2024-09-11T03:44:38.060 INFO:tasks.rados.rados.2.smithi175.stdout:1678: done (2 left) 2024-09-11T03:44:38.060 INFO:tasks.rados.rados.2.smithi175.stdout:1679: done (1 left) 2024-09-11T03:44:38.060 INFO:tasks.rados.rados.2.smithi175.stdout:1680: done (0 left) 2024-09-11T03:44:38.060 INFO:tasks.rados.rados.2.smithi175.stdout:1681: read oid 2 snap -1 2024-09-11T03:44:38.061 INFO:tasks.rados.rados.2.smithi175.stdout:1681: expect deleted 2024-09-11T03:44:38.061 INFO:tasks.rados.rados.2.smithi175.stdout:1682: read oid 13 snap -1 2024-09-11T03:44:38.061 INFO:tasks.rados.rados.2.smithi175.stdout:1682: expect deleted 2024-09-11T03:44:38.061 INFO:tasks.rados.rados.2.smithi175.stdout:1683: write oid 26 current snap is 183 2024-09-11T03:44:38.061 INFO:tasks.rados.rados.2.smithi175.stdout:1683: seq_num 589 ranges {699323=480842,1850247=429872,2858378=532392} 2024-09-11T03:44:38.065 INFO:tasks.rados.rados.2.smithi175.stdout:1683: writing smithi17557870-26 from 699323 to 1180165 tid 1 2024-09-11T03:44:38.071 INFO:tasks.rados.rados.2.smithi175.stdout:1683: writing smithi17557870-26 from 1850247 to 2280119 tid 2 2024-09-11T03:44:38.078 INFO:tasks.rados.rados.2.smithi175.stdout:1683: writing smithi17557870-26 from 2858378 to 3390770 tid 3 2024-09-11T03:44:38.078 INFO:tasks.rados.rados.2.smithi175.stdout:1684: rollback oid 16 current snap is 183 2024-09-11T03:44:38.078 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 16 to 179 2024-09-11T03:44:38.078 INFO:tasks.rados.rados.2.smithi175.stdout:1683: finishing write tid 1 to smithi17557870-26 2024-09-11T03:44:38.078 INFO:tasks.rados.rados.2.smithi175.stdout:1683: finishing write tid 2 to smithi17557870-26 2024-09-11T03:44:38.078 INFO:tasks.rados.rados.2.smithi175.stdout:1681: done (3 left) 2024-09-11T03:44:38.078 INFO:tasks.rados.rados.2.smithi175.stdout:1682: done (2 left) 2024-09-11T03:44:38.078 INFO:tasks.rados.rados.2.smithi175.stdout:1685: read oid 20 snap -1 2024-09-11T03:44:38.078 INFO:tasks.rados.rados.2.smithi175.stdout:1685: expect (ObjNum 563 snap 172 seq_num 563) 2024-09-11T03:44:38.078 INFO:tasks.rados.rados.2.smithi175.stdout:1686: write oid 23 current snap is 183 2024-09-11T03:44:38.078 INFO:tasks.rados.rados.2.smithi175.stdout:1686: seq_num 590 ranges {716130=556091,1792376=573530,2769448=460017,3821956=94174} 2024-09-11T03:44:38.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:38.079 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:38.080 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:38.084 INFO:tasks.rados.rados.2.smithi175.stdout:1686: writing smithi17557870-23 from 716130 to 1272221 tid 1 2024-09-11T03:44:38.090 INFO:tasks.rados.rados.2.smithi175.stdout:1686: writing smithi17557870-23 from 1792376 to 2365906 tid 2 2024-09-11T03:44:38.096 INFO:tasks.rados.rados.2.smithi175.stdout:1686: writing smithi17557870-23 from 2769448 to 3229465 tid 3 2024-09-11T03:44:38.100 INFO:tasks.rados.rados.2.smithi175.stdout:1686: writing smithi17557870-23 from 3821956 to 3916130 tid 4 2024-09-11T03:44:38.100 INFO:tasks.rados.rados.2.smithi175.stdout:1687: write (excl) oid 42 current snap is 183 2024-09-11T03:44:38.100 INFO:tasks.rados.rados.2.smithi175.stdout:1684: finishing rollback tid 1 to smithi17557870-16 2024-09-11T03:44:38.100 INFO:tasks.rados.rados.2.smithi175.stdout:1684: finishing rollback tid 2 to smithi17557870-16 2024-09-11T03:44:38.100 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 488 (ObjNum 509 snap 150 seq_num 509) dirty exists 2024-09-11T03:44:38.100 INFO:tasks.rados.rados.2.smithi175.stdout:1683: finishing write tid 3 to smithi17557870-26 2024-09-11T03:44:38.100 INFO:tasks.rados.rados.2.smithi175.stdout:1683: finishing write tid 4 to smithi17557870-26 2024-09-11T03:44:38.100 INFO:tasks.rados.rados.2.smithi175.stdout:1683: finishing write tid 5 to smithi17557870-26 2024-09-11T03:44:38.100 INFO:tasks.rados.rados.2.smithi175.stdout:1683: oid 26 updating version 0 to 326 2024-09-11T03:44:38.101 INFO:tasks.rados.rados.2.smithi175.stdout:1683: oid 26 version 326 is already newer than 325 2024-09-11T03:44:38.101 INFO:tasks.rados.rados.2.smithi175.stdout:1683: oid 26 updating version 326 to 327 2024-09-11T03:44:38.101 INFO:tasks.rados.rados.2.smithi175.stdout:1683: oid 26 version 327 is already newer than 324 2024-09-11T03:44:38.101 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 327 (ObjNum 589 snap 183 seq_num 589) dirty exists 2024-09-11T03:44:38.101 INFO:tasks.rados.rados.2.smithi175.stdout:1683: left oid 26 (ObjNum 589 snap 183 seq_num 589) 2024-09-11T03:44:38.101 INFO:tasks.rados.rados.2.smithi175.stdout:1686: finishing write tid 1 to smithi17557870-23 2024-09-11T03:44:38.101 INFO:tasks.rados.rados.2.smithi175.stdout:1685: expect (ObjNum 563 snap 172 seq_num 563) 2024-09-11T03:44:38.181 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:38.202 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:38.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:38.203 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:38.219 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:37 smithi084 ceph-mon[27254]: pgmap v843: 57 pgs: 57 active+clean; 156 MiB data, 765 MiB used, 983 GiB / 983 GiB avail; 2.3 MiB/s rd, 449 KiB/s wr, 3 op/s 2024-09-11T03:44:38.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:37 smithi175 ceph-mon[30894]: pgmap v843: 57 pgs: 57 active+clean; 156 MiB data, 765 MiB used, 983 GiB / 983 GiB avail; 2.3 MiB/s rd, 449 KiB/s wr, 3 op/s 2024-09-11T03:44:38.297 INFO:tasks.rados.rados.2.smithi175.stdout:1686: finishing write tid 2 to smithi17557870-23 2024-09-11T03:44:38.297 INFO:tasks.rados.rados.2.smithi175.stdout:1686: finishing write tid 3 to smithi17557870-23 2024-09-11T03:44:38.297 INFO:tasks.rados.rados.2.smithi175.stdout:1686: finishing write tid 4 to smithi17557870-23 2024-09-11T03:44:38.297 INFO:tasks.rados.rados.2.smithi175.stdout:1686: finishing write tid 5 to smithi17557870-23 2024-09-11T03:44:38.297 INFO:tasks.rados.rados.2.smithi175.stdout:1686: finishing write tid 6 to smithi17557870-23 2024-09-11T03:44:38.297 INFO:tasks.rados.rados.2.smithi175.stdout:1686: oid 23 updating version 0 to 323 2024-09-11T03:44:38.297 INFO:tasks.rados.rados.2.smithi175.stdout:1686: oid 23 version 323 is already newer than 322 2024-09-11T03:44:38.297 INFO:tasks.rados.rados.2.smithi175.stdout:1686: oid 23 updating version 323 to 325 2024-09-11T03:44:38.298 INFO:tasks.rados.rados.2.smithi175.stdout:1686: oid 23 version 325 is already newer than 321 2024-09-11T03:44:38.298 INFO:tasks.rados.rados.2.smithi175.stdout:1686: oid 23 version 325 is already newer than 324 2024-09-11T03:44:38.298 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 325 (ObjNum 590 snap 183 seq_num 590) dirty exists 2024-09-11T03:44:38.298 INFO:tasks.rados.rados.2.smithi175.stdout:1686: left oid 23 (ObjNum 590 snap 183 seq_num 590) 2024-09-11T03:44:38.298 INFO:tasks.rados.rados.2.smithi175.stdout:1687: seq_num 591 ranges {732937=631340,1850858=433541,2796871=671289,3888195=44742} 2024-09-11T03:44:38.303 INFO:tasks.rados.rados.2.smithi175.stdout:1687: writing smithi17557870-42 from 732937 to 1364277 tid 1 2024-09-11T03:44:38.304 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:38.309 INFO:tasks.rados.rados.2.smithi175.stdout:1687: writing smithi17557870-42 from 1850858 to 2284399 tid 2 2024-09-11T03:44:38.315 INFO:tasks.rados.rados.2.smithi175.stdout:1687: writing smithi17557870-42 from 2796871 to 3468160 tid 3 2024-09-11T03:44:38.318 INFO:tasks.rados.rados.2.smithi175.stdout:1687: writing smithi17557870-42 from 3888195 to 3932937 tid 4 2024-09-11T03:44:38.318 INFO:tasks.rados.rados.2.smithi175.stdout:1683: done (4 left) 2024-09-11T03:44:38.318 INFO:tasks.rados.rados.2.smithi175.stdout:1684: done (3 left) 2024-09-11T03:44:38.319 INFO:tasks.rados.rados.2.smithi175.stdout:1685: done (2 left) 2024-09-11T03:44:38.319 INFO:tasks.rados.rados.2.smithi175.stdout:1686: done (1 left) 2024-09-11T03:44:38.319 INFO:tasks.rados.rados.2.smithi175.stdout:1688: write oid 49 current snap is 183 2024-09-11T03:44:38.319 INFO:tasks.rados.rados.2.smithi175.stdout:1687: finishing write tid 1 to smithi17557870-42 2024-09-11T03:44:38.319 INFO:tasks.rados.rados.2.smithi175.stdout:1687: finishing write tid 2 to smithi17557870-42 2024-09-11T03:44:38.319 INFO:tasks.rados.rados.2.smithi175.stdout:1688: seq_num 592 ranges {749744=706589,1909340=577199,3224294=482561} 2024-09-11T03:44:38.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:37 smithi121 ceph-mon[30729]: pgmap v843: 57 pgs: 57 active+clean; 156 MiB data, 765 MiB used, 983 GiB / 983 GiB avail; 2.3 MiB/s rd, 449 KiB/s wr, 3 op/s 2024-09-11T03:44:38.325 INFO:tasks.rados.rados.2.smithi175.stdout:1688: writing smithi17557870-49 from 749744 to 1456333 tid 1 2024-09-11T03:44:38.328 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:38.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:38.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:38.331 INFO:tasks.rados.rados.2.smithi175.stdout:1688: writing smithi17557870-49 from 1909340 to 2486539 tid 2 2024-09-11T03:44:38.338 INFO:tasks.rados.rados.2.smithi175.stdout:1688: writing smithi17557870-49 from 3224294 to 3706855 tid 3 2024-09-11T03:44:38.338 INFO:tasks.rados.rados.2.smithi175.stdout:1689: copy_from oid 30 from oid 10 current snap is 183 2024-09-11T03:44:38.339 INFO:tasks.rados.rados.2.smithi175.stdout:1687: finishing write tid 3 to smithi17557870-42 2024-09-11T03:44:38.339 INFO:tasks.rados.rados.2.smithi175.stdout:1687: finishing write tid 4 to smithi17557870-42 2024-09-11T03:44:38.339 INFO:tasks.rados.rados.2.smithi175.stdout:1687: finishing write tid 5 to smithi17557870-42 2024-09-11T03:44:38.339 INFO:tasks.rados.rados.2.smithi175.stdout:1687: finishing write tid 6 to smithi17557870-42 2024-09-11T03:44:38.339 INFO:tasks.rados.rados.2.smithi175.stdout:1687: oid 42 updating version 0 to 488 2024-09-11T03:44:38.339 INFO:tasks.rados.rados.2.smithi175.stdout:1687: oid 42 version 488 is already newer than 486 2024-09-11T03:44:38.339 INFO:tasks.rados.rados.2.smithi175.stdout:1687: oid 42 version 488 is already newer than 484 2024-09-11T03:44:38.339 INFO:tasks.rados.rados.2.smithi175.stdout:1687: oid 42 version 488 is already newer than 485 2024-09-11T03:44:38.339 INFO:tasks.rados.rados.2.smithi175.stdout:1687: oid 42 version 488 is already newer than 487 2024-09-11T03:44:38.339 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 488 (ObjNum 591 snap 183 seq_num 591) dirty exists 2024-09-11T03:44:38.339 INFO:tasks.rados.rados.2.smithi175.stdout:1687: left oid 42 (ObjNum 591 snap 183 seq_num 591) 2024-09-11T03:44:38.339 INFO:tasks.rados.rados.2.smithi175.stdout:1688: finishing write tid 1 to smithi17557870-49 2024-09-11T03:44:38.339 INFO:tasks.rados.rados.2.smithi175.stdout:1688: finishing write tid 2 to smithi17557870-49 2024-09-11T03:44:38.339 INFO:tasks.rados.rados.2.smithi175.stdout:1687: done (2 left) 2024-09-11T03:44:38.340 INFO:tasks.rados.rados.2.smithi175.stdout:1690: write (excl) oid 1 current snap is 183 2024-09-11T03:44:38.340 INFO:tasks.rados.rados.2.smithi175.stdout:1690: seq_num 593 ranges {766551=498191,1684175=720857,2851717=693833} 2024-09-11T03:44:38.345 INFO:tasks.rados.rados.2.smithi175.stdout:1690: writing smithi17557870-1 from 766551 to 1264742 tid 1 2024-09-11T03:44:38.351 INFO:tasks.rados.rados.2.smithi175.stdout:1690: writing smithi17557870-1 from 1684175 to 2405032 tid 2 2024-09-11T03:44:38.358 INFO:tasks.rados.rados.2.smithi175.stdout:1690: writing smithi17557870-1 from 2851717 to 3545550 tid 3 2024-09-11T03:44:38.358 INFO:tasks.rados.rados.2.smithi175.stdout:1691: read oid 26 snap -1 2024-09-11T03:44:38.358 INFO:tasks.rados.rados.2.smithi175.stdout:1691: expect (ObjNum 589 snap 183 seq_num 589) 2024-09-11T03:44:38.358 INFO:tasks.rados.rados.2.smithi175.stdout:1692: snap_create 2024-09-11T03:44:38.358 INFO:tasks.rados.rados.2.smithi175.stdout:1688: finishing write tid 3 to smithi17557870-49 2024-09-11T03:44:38.358 INFO:tasks.rados.rados.2.smithi175.stdout:1688: finishing write tid 4 to smithi17557870-49 2024-09-11T03:44:38.358 INFO:tasks.rados.rados.2.smithi175.stdout:1688: finishing write tid 5 to smithi17557870-49 2024-09-11T03:44:38.359 INFO:tasks.rados.rados.2.smithi175.stdout:1688: oid 49 updating version 0 to 492 2024-09-11T03:44:38.359 INFO:tasks.rados.rados.2.smithi175.stdout:1688: oid 49 updating version 492 to 493 2024-09-11T03:44:38.359 INFO:tasks.rados.rados.2.smithi175.stdout:1688: oid 49 version 493 is already newer than 490 2024-09-11T03:44:38.359 INFO:tasks.rados.rados.2.smithi175.stdout:1688: oid 49 version 493 is already newer than 491 2024-09-11T03:44:38.359 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 493 (ObjNum 592 snap 183 seq_num 592) dirty exists 2024-09-11T03:44:38.359 INFO:tasks.rados.rados.2.smithi175.stdout:1688: left oid 49 (ObjNum 592 snap 183 seq_num 592) 2024-09-11T03:44:38.359 INFO:tasks.rados.rados.2.smithi175.stdout:1689: finishing copy_from racing read to smithi17557870-30 2024-09-11T03:44:38.359 INFO:tasks.rados.rados.2.smithi175.stdout:1689: finishing copy_from to smithi17557870-30 2024-09-11T03:44:38.359 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 526 (ObjNum 488 snap 141 seq_num 488) dirty exists 2024-09-11T03:44:38.359 INFO:tasks.rados.rados.2.smithi175.stdout:1690: finishing write tid 1 to smithi17557870-1 2024-09-11T03:44:38.359 INFO:tasks.rados.rados.2.smithi175.stdout:1690: finishing write tid 2 to smithi17557870-1 2024-09-11T03:44:38.361 INFO:tasks.rados.rados.2.smithi175.stdout:1690: finishing write tid 3 to smithi17557870-1 2024-09-11T03:44:38.361 INFO:tasks.rados.rados.2.smithi175.stdout:1690: finishing write tid 4 to smithi17557870-1 2024-09-11T03:44:38.361 INFO:tasks.rados.rados.2.smithi175.stdout:1690: finishing write tid 5 to smithi17557870-1 2024-09-11T03:44:38.361 INFO:tasks.rados.rados.2.smithi175.stdout:1690: oid 1 updating version 0 to 624 2024-09-11T03:44:38.361 INFO:tasks.rados.rados.2.smithi175.stdout:1690: oid 1 version 624 is already newer than 621 2024-09-11T03:44:38.361 INFO:tasks.rados.rados.2.smithi175.stdout:1690: oid 1 version 624 is already newer than 623 2024-09-11T03:44:38.361 INFO:tasks.rados.rados.2.smithi175.stdout:1690: oid 1 version 624 is already newer than 622 2024-09-11T03:44:38.361 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 624 (ObjNum 593 snap 183 seq_num 593) dirty exists 2024-09-11T03:44:38.362 INFO:tasks.rados.rados.2.smithi175.stdout:1690: left oid 1 (ObjNum 593 snap 183 seq_num 593) 2024-09-11T03:44:38.372 INFO:tasks.rados.rados.2.smithi175.stdout:1691: expect (ObjNum 589 snap 183 seq_num 589) 2024-09-11T03:44:38.430 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:38.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:38.452 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:38.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:38.554 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:38.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:38.575 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:38.575 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:38.665 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T03:44:38.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:38.698 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:38.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:38.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:38.741 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:38.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:38.820 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:38.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:38.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:38.922 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:38.943 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:38.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:38.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:38.970 INFO:tasks.rados.rados.2.smithi175.stdout:1688: done (4 left) 2024-09-11T03:44:38.970 INFO:tasks.rados.rados.2.smithi175.stdout:1689: done (3 left) 2024-09-11T03:44:38.970 INFO:tasks.rados.rados.2.smithi175.stdout:1690: done (2 left) 2024-09-11T03:44:38.970 INFO:tasks.rados.rados.2.smithi175.stdout:1691: done (1 left) 2024-09-11T03:44:38.970 INFO:tasks.rados.rados.2.smithi175.stdout:1692: done (0 left) 2024-09-11T03:44:38.970 INFO:tasks.rados.rados.2.smithi175.stdout:1693: read oid 47 snap -1 2024-09-11T03:44:38.970 INFO:tasks.rados.rados.2.smithi175.stdout:1693: expect deleted 2024-09-11T03:44:38.970 INFO:tasks.rados.rados.2.smithi175.stdout:1694: snap_create 2024-09-11T03:44:39.045 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:39.068 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:39.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:39.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:39.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:38 smithi084 ceph-mon[27254]: osdmap e460: 12 total, 12 up, 11 in 2024-09-11T03:44:39.170 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:39.193 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:39.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:39.194 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:39.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:38 smithi175 ceph-mon[30894]: osdmap e460: 12 total, 12 up, 11 in 2024-09-11T03:44:39.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:39.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:39.317 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:39.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:39.336 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:39.418 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:39.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:38 smithi121 ceph-mon[30729]: osdmap e460: 12 total, 12 up, 11 in 2024-09-11T03:44:39.439 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:39.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:39.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:39.542 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:39.565 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:39.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:39.666 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:39.687 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:39.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:39.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:39.789 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:39.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:39.811 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:39.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:39.826 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:44:39.826 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:44:39.826 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:44:39.826 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:44:39.912 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:39.933 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:39.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:39.934 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:39.936 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:44:39.965 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:44:39.968 INFO:tasks.rados.rados.2.smithi175.stdout:1693: done (1 left) 2024-09-11T03:44:39.968 INFO:tasks.rados.rados.2.smithi175.stdout:1694: done (0 left) 2024-09-11T03:44:39.968 INFO:tasks.rados.rados.2.smithi175.stdout:1695: delete oid 44 current snap is 185 2024-09-11T03:44:39.972 INFO:tasks.rados.rados.2.smithi175.stdout:1695: done (0 left) 2024-09-11T03:44:39.972 INFO:tasks.rados.rados.2.smithi175.stdout:1696: copy_from oid 25 from oid 42 current snap is 185 2024-09-11T03:44:39.972 INFO:tasks.rados.rados.2.smithi175.stdout:1697: write (excl) oid 26 current snap is 185 2024-09-11T03:44:39.972 INFO:tasks.rados.rados.2.smithi175.stdout:1697: seq_num 595 ranges {400165=648689,1684786=315379} 2024-09-11T03:44:39.977 INFO:tasks.rados.rados.2.smithi175.stdout:1697: writing smithi17557870-26 from 400165 to 1048854 tid 1 2024-09-11T03:44:39.983 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:39 smithi084 ceph-mon[27254]: pgmap v846: 57 pgs: 57 active+clean; 168 MiB data, 765 MiB used, 983 GiB / 983 GiB avail; 767 B/s rd, 1.8 MiB/s wr, 5 op/s 2024-09-11T03:44:39.983 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:39 smithi084 ceph-mon[27254]: osdmap e461: 12 total, 12 up, 11 in 2024-09-11T03:44:39.983 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:39 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1609245777' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:44:39.985 INFO:tasks.rados.rados.2.smithi175.stdout:1697: writing smithi17557870-26 from 1684786 to 2000165 tid 2 2024-09-11T03:44:39.985 INFO:tasks.rados.rados.2.smithi175.stdout:1697: finishing write tid 1 to smithi17557870-26 2024-09-11T03:44:39.985 INFO:tasks.rados.rados.2.smithi175.stdout:1698: write oid 32 current snap is 185 2024-09-11T03:44:39.985 INFO:tasks.rados.rados.2.smithi175.stdout:1698: seq_num 596 ranges {416972=723938,1743268=273704} 2024-09-11T03:44:39.990 INFO:tasks.rados.rados.2.smithi175.stdout:1698: writing smithi17557870-32 from 416972 to 1140910 tid 1 2024-09-11T03:44:39.996 INFO:tasks.rados.rados.2.smithi175.stdout:1698: writing smithi17557870-32 from 1743268 to 2016972 tid 2 2024-09-11T03:44:39.996 INFO:tasks.rados.rados.2.smithi175.stdout:1699: snap_create 2024-09-11T03:44:39.996 INFO:tasks.rados.rados.2.smithi175.stdout:1697: finishing write tid 2 to smithi17557870-26 2024-09-11T03:44:39.996 INFO:tasks.rados.rados.2.smithi175.stdout:1697: finishing write tid 3 to smithi17557870-26 2024-09-11T03:44:39.996 INFO:tasks.rados.rados.2.smithi175.stdout:1697: finishing write tid 4 to smithi17557870-26 2024-09-11T03:44:39.996 INFO:tasks.rados.rados.2.smithi175.stdout:1697: oid 26 updating version 0 to 330 2024-09-11T03:44:39.997 INFO:tasks.rados.rados.2.smithi175.stdout:1697: oid 26 updating version 330 to 331 2024-09-11T03:44:39.997 INFO:tasks.rados.rados.2.smithi175.stdout:1697: oid 26 version 331 is already newer than 329 2024-09-11T03:44:39.997 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 331 (ObjNum 595 snap 185 seq_num 595) dirty exists 2024-09-11T03:44:39.997 INFO:tasks.rados.rados.2.smithi175.stdout:1697: left oid 26 (ObjNum 595 snap 185 seq_num 595) 2024-09-11T03:44:39.997 INFO:tasks.rados.rados.2.smithi175.stdout:1696: finishing copy_from racing read to smithi17557870-25 2024-09-11T03:44:39.997 INFO:tasks.rados.rados.2.smithi175.stdout:1698: finishing write tid 1 to smithi17557870-32 2024-09-11T03:44:39.998 INFO:tasks.rados.rados.2.smithi175.stdout:1698: finishing write tid 2 to smithi17557870-32 2024-09-11T03:44:39.998 INFO:tasks.rados.rados.2.smithi175.stdout:1698: finishing write tid 3 to smithi17557870-32 2024-09-11T03:44:39.998 INFO:tasks.rados.rados.2.smithi175.stdout:1698: finishing write tid 4 to smithi17557870-32 2024-09-11T03:44:39.998 INFO:tasks.rados.rados.2.smithi175.stdout:1698: oid 32 updating version 0 to 473 2024-09-11T03:44:39.999 INFO:tasks.rados.rados.2.smithi175.stdout:1698: oid 32 version 473 is already newer than 472 2024-09-11T03:44:39.999 INFO:tasks.rados.rados.2.smithi175.stdout:1698: oid 32 version 473 is already newer than 471 2024-09-11T03:44:39.999 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 473 (ObjNum 596 snap 185 seq_num 596) dirty exists 2024-09-11T03:44:39.999 INFO:tasks.rados.rados.2.smithi175.stdout:1698: left oid 32 (ObjNum 596 snap 185 seq_num 596) 2024-09-11T03:44:40.000 INFO:tasks.rados.rados.2.smithi175.stdout:1696: finishing copy_from to smithi17557870-25 2024-09-11T03:44:40.000 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 488 (ObjNum 591 snap 183 seq_num 591) dirty exists 2024-09-11T03:44:40.035 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:40.056 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:40.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:40.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:40.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:40.180 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:40.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:40.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:40.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:39 smithi175 ceph-mon[30894]: pgmap v846: 57 pgs: 57 active+clean; 168 MiB data, 765 MiB used, 983 GiB / 983 GiB avail; 767 B/s rd, 1.8 MiB/s wr, 5 op/s 2024-09-11T03:44:40.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:39 smithi175 ceph-mon[30894]: osdmap e461: 12 total, 12 up, 11 in 2024-09-11T03:44:40.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:39 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1609245777' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:44:40.263 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:44:40.263 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:44:40.263 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T03:44:40.263 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.000000 2024-09-11T03:44:40.263 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 4 0 2024-09-11T03:44:40.282 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:40.328 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:40.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:40.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:39 smithi121 ceph-mon[30729]: pgmap v846: 57 pgs: 57 active+clean; 168 MiB data, 765 MiB used, 983 GiB / 983 GiB avail; 767 B/s rd, 1.8 MiB/s wr, 5 op/s 2024-09-11T03:44:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:39 smithi121 ceph-mon[30729]: osdmap e461: 12 total, 12 up, 11 in 2024-09-11T03:44:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:39 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1609245777' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:44:40.430 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:40.451 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:40.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:40.452 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:40.553 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:40.574 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:40.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:40.575 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:40.676 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:40.698 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:40.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:40.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:40.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:40.825 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:40.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:40.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:40.927 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:40.949 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:40.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:40.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:40.977 INFO:tasks.rados.rados.2.smithi175.stdout:1696: done (3 left) 2024-09-11T03:44:40.977 INFO:tasks.rados.rados.2.smithi175.stdout:1697: done (2 left) 2024-09-11T03:44:40.977 INFO:tasks.rados.rados.2.smithi175.stdout:1698: done (1 left) 2024-09-11T03:44:40.977 INFO:tasks.rados.rados.2.smithi175.stdout:1699: done (0 left) 2024-09-11T03:44:40.977 INFO:tasks.rados.rados.2.smithi175.stdout:1700: write (excl) oid 16 current snap is 186 2024-09-11T03:44:40.977 INFO:tasks.rados.rados.2.smithi175.stdout:1700: seq_num 597 ranges {433779=799187,1801750=232029} 2024-09-11T03:44:40.983 INFO:tasks.rados.rados.2.smithi175.stdout:1700: writing smithi17557870-16 from 433779 to 1232966 tid 1 2024-09-11T03:44:40.988 INFO:tasks.rados.rados.2.smithi175.stdout:1700: writing smithi17557870-16 from 1801750 to 2033779 tid 2 2024-09-11T03:44:40.988 INFO:tasks.rados.rados.2.smithi175.stdout:1701: snap_remove snap 180 2024-09-11T03:44:41.051 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:41.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:41.078 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:41.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:41.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:41.204 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:41.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:41.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:41.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:40 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:41.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:40 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1609245777' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:44:41.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:40 smithi175 ceph-mon[30894]: osdmap e462: 12 total, 12 up, 11 in 2024-09-11T03:44:41.306 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:41.327 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:41.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:41.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:41.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:40 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:41.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:40 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1609245777' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:44:41.353 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:40 smithi084 ceph-mon[27254]: osdmap e462: 12 total, 12 up, 11 in 2024-09-11T03:44:41.381 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T03:44:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:40 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:40 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1609245777' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:44:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:40 smithi121 ceph-mon[30729]: osdmap e462: 12 total, 12 up, 11 in 2024-09-11T03:44:41.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:41.455 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:41.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:41.461 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:41.468 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:41.557 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:41.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:41.580 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:41.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:41.681 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:41.705 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:41.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:41.807 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:41.829 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:41.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:41.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:41.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:41.933 INFO:tasks.rados.rados.2.smithi175.stdout:1701: done (1 left) 2024-09-11T03:44:41.933 INFO:tasks.rados.rados.2.smithi175.stdout:1702: snap_remove snap 179 2024-09-11T03:44:41.956 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:41.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:42.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:42.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:42.081 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:42.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:42.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:42.207 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:42.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:42.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:41 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:44:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:41 smithi175 ceph-mon[30894]: 2.0 scrub ok 2024-09-11T03:44:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:41 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:44:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:41 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:44:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:41 smithi175 ceph-mon[30894]: pgmap v849: 57 pgs: 57 active+clean; 168 MiB data, 767 MiB used, 983 GiB / 983 GiB avail; 767 B/s rd, 1.8 MiB/s wr, 5 op/s 2024-09-11T03:44:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:41 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:44:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:41 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:44:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:41 smithi175 ceph-mon[30894]: osdmap e463: 12 total, 12 up, 11 in 2024-09-11T03:44:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:41 smithi175 ceph-mon[30894]: osdmap e464: 12 total, 12 up, 11 in 2024-09-11T03:44:42.276 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:41 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:44:42.277 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:41 smithi084 ceph-mon[27254]: 2.0 scrub ok 2024-09-11T03:44:42.277 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:41 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:44:42.277 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:41 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:44:42.277 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:41 smithi084 ceph-mon[27254]: pgmap v849: 57 pgs: 57 active+clean; 168 MiB data, 767 MiB used, 983 GiB / 983 GiB avail; 767 B/s rd, 1.8 MiB/s wr, 5 op/s 2024-09-11T03:44:42.277 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:41 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:44:42.277 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:41 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:44:42.277 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:41 smithi084 ceph-mon[27254]: osdmap e463: 12 total, 12 up, 11 in 2024-09-11T03:44:42.277 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:41 smithi084 ceph-mon[27254]: osdmap e464: 12 total, 12 up, 11 in 2024-09-11T03:44:42.309 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:42.330 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:42.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:42.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:42.376 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:44:42.376 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:44:42.394 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:44:42.411 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:44:42.412 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:44:42.421 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:41 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:44:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:41 smithi121 ceph-mon[30729]: 2.0 scrub ok 2024-09-11T03:44:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:41 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:44:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:41 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:44:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:41 smithi121 ceph-mon[30729]: pgmap v849: 57 pgs: 57 active+clean; 168 MiB data, 767 MiB used, 983 GiB / 983 GiB avail; 767 B/s rd, 1.8 MiB/s wr, 5 op/s 2024-09-11T03:44:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:41 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:44:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:41 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:44:42.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:41 smithi121 ceph-mon[30729]: osdmap e463: 12 total, 12 up, 11 in 2024-09-11T03:44:42.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:41 smithi121 ceph-mon[30729]: osdmap e464: 12 total, 12 up, 11 in 2024-09-11T03:44:42.431 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:44:42.431 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:44:42.432 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:42.449 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:44:42.449 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:44:42.454 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:42.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:42.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:42.467 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:44:42.467 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:44:42.484 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:44:42.485 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:44:42.502 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:44:42.502 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:44:42.519 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:44:42.519 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:44:42.535 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:44:42.536 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:44:42.552 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:44:42.552 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:44:42.556 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:42.569 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:44:42.569 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:44:42.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:42.583 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:42.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:42.684 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:42.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:42.725 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:42.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:42.726 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:44:42.826 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:42.852 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:42.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:42.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:42.930 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.4 primary-affinity to 0 (802) 2024-09-11T03:44:42.935 INFO:tasks.rados.rados.2.smithi175.stdout:1702: done (1 left) 2024-09-11T03:44:42.935 INFO:tasks.rados.rados.2.smithi175.stdout:1703: read oid 9 snap -1 2024-09-11T03:44:42.935 INFO:tasks.rados.rados.2.smithi175.stdout:1703: expect (ObjNum 580 snap 178 seq_num 580) 2024-09-11T03:44:42.935 INFO:tasks.rados.rados.2.smithi175.stdout:1704: snap_remove snap 182 2024-09-11T03:44:42.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:42.980 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:42.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:42.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:43.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:43.106 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:43.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:43.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:43.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:42 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:44:43.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:42 smithi084 ceph-mon[27254]: 2.7 scrub ok 2024-09-11T03:44:43.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:42 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:44:43.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:42 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:44:43.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:42 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2545899225' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-11T03:44:43.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:42 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/53634437' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:43.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:42 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2545899225' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-11T03:44:43.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:42 smithi084 ceph-mon[27254]: osdmap e465: 12 total, 12 up, 11 in 2024-09-11T03:44:43.208 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:43.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:43.233 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:43.233 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:43.334 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:43.356 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:43.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:43.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:42 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:44:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:42 smithi121 ceph-mon[30729]: 2.7 scrub ok 2024-09-11T03:44:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:42 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:44:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:42 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:44:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:42 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2545899225' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-11T03:44:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:42 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/53634437' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:42 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2545899225' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-11T03:44:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:42 smithi121 ceph-mon[30729]: osdmap e465: 12 total, 12 up, 11 in 2024-09-11T03:44:43.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:42 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:44:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:42 smithi175 ceph-mon[30894]: 2.7 scrub ok 2024-09-11T03:44:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:42 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:44:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:42 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:44:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:42 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2545899225' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-11T03:44:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:42 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/53634437' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:42 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2545899225' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-11T03:44:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:42 smithi175 ceph-mon[30894]: osdmap e465: 12 total, 12 up, 11 in 2024-09-11T03:44:43.479 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:43.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:43.581 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:43.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:43.603 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:43.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:43.704 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:43.730 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:43.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:43.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:43.832 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:43.853 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:43.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:43.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:43.879 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:44:43.879 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:44:43.937 INFO:tasks.rados.rados.2.smithi175.stdout:1704: done (2 left) 2024-09-11T03:44:43.938 INFO:tasks.rados.rados.2.smithi175.stdout:1705: rollback oid 49 current snap is 186 2024-09-11T03:44:43.938 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 49 to 172 2024-09-11T03:44:43.938 INFO:tasks.rados.rados.2.smithi175.stdout:1700: finishing write tid 1 to smithi17557870-16 2024-09-11T03:44:43.938 INFO:tasks.rados.rados.2.smithi175.stdout:1700: finishing write tid 2 to smithi17557870-16 2024-09-11T03:44:43.938 INFO:tasks.rados.rados.2.smithi175.stdout:1700: finishing write tid 3 to smithi17557870-16 2024-09-11T03:44:43.938 INFO:tasks.rados.rados.2.smithi175.stdout:1700: finishing write tid 4 to smithi17557870-16 2024-09-11T03:44:43.938 INFO:tasks.rados.rados.2.smithi175.stdout:1700: oid 16 updating version 0 to 490 2024-09-11T03:44:43.938 INFO:tasks.rados.rados.2.smithi175.stdout:1700: oid 16 updating version 490 to 491 2024-09-11T03:44:43.938 INFO:tasks.rados.rados.2.smithi175.stdout:1700: oid 16 version 491 is already newer than 489 2024-09-11T03:44:43.938 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 491 (ObjNum 597 snap 186 seq_num 597) dirty exists 2024-09-11T03:44:43.938 INFO:tasks.rados.rados.2.smithi175.stdout:1700: left oid 16 (ObjNum 597 snap 186 seq_num 597) 2024-09-11T03:44:43.939 INFO:tasks.rados.rados.2.smithi175.stdout:1703: expect (ObjNum 580 snap 178 seq_num 580) 2024-09-11T03:44:43.955 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:43.980 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:43.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:43.981 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:44.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:44.108 INFO:tasks.rados.rados.2.smithi175.stdout:1705: finishing rollback tid 0 to smithi17557870-49 2024-09-11T03:44:44.108 INFO:tasks.rados.rados.2.smithi175.stdout:1705: finishing rollback tid 1 to smithi17557870-49 2024-09-11T03:44:44.108 INFO:tasks.rados.rados.2.smithi175.stdout:1705: finishing rollback tid 2 to smithi17557870-49 2024-09-11T03:44:44.108 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 509 (ObjNum 565 snap 172 seq_num 565) dirty exists 2024-09-11T03:44:44.108 INFO:tasks.rados.rados.2.smithi175.stdout:1700: done (2 left) 2024-09-11T03:44:44.108 INFO:tasks.rados.rados.2.smithi175.stdout:1703: done (1 left) 2024-09-11T03:44:44.109 INFO:tasks.rados.rados.2.smithi175.stdout:1705: done (0 left) 2024-09-11T03:44:44.109 INFO:tasks.rados.rados.2.smithi175.stdout:1706: rollback oid 10 current snap is 186 2024-09-11T03:44:44.109 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 10 to 183 2024-09-11T03:44:44.109 INFO:tasks.rados.rados.2.smithi175.stdout:1707: rollback oid 25 current snap is 186 2024-09-11T03:44:44.109 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 25 to 184 2024-09-11T03:44:44.109 INFO:tasks.rados.rados.2.smithi175.stdout:1708: snap_remove snap 181 2024-09-11T03:44:44.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:44.110 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:44.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:44.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:43 smithi084 ceph-mon[27254]: pgmap v852: 57 pgs: 57 active+clean; 185 MiB data, 777 MiB used, 983 GiB / 983 GiB avail; 8.7 MiB/s rd, 3.6 MiB/s wr, 20 op/s 2024-09-11T03:44:44.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:43 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:44:44.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:43 smithi084 ceph-mon[27254]: 2.14 scrub ok 2024-09-11T03:44:44.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:43 smithi084 ceph-mon[27254]: osdmap e466: 12 total, 12 up, 11 in 2024-09-11T03:44:44.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:44.236 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:44.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:44.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:44.337 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:44.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:44.360 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:44.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:43 smithi121 ceph-mon[30729]: pgmap v852: 57 pgs: 57 active+clean; 185 MiB data, 777 MiB used, 983 GiB / 983 GiB avail; 8.7 MiB/s rd, 3.6 MiB/s wr, 20 op/s 2024-09-11T03:44:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:43 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:44:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:43 smithi121 ceph-mon[30729]: 2.14 scrub ok 2024-09-11T03:44:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:43 smithi121 ceph-mon[30729]: osdmap e466: 12 total, 12 up, 11 in 2024-09-11T03:44:44.461 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:43 smithi175 ceph-mon[30894]: pgmap v852: 57 pgs: 57 active+clean; 185 MiB data, 777 MiB used, 983 GiB / 983 GiB avail; 8.7 MiB/s rd, 3.6 MiB/s wr, 20 op/s 2024-09-11T03:44:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:43 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:44:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:43 smithi175 ceph-mon[30894]: 2.14 scrub ok 2024-09-11T03:44:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:43 smithi175 ceph-mon[30894]: osdmap e466: 12 total, 12 up, 11 in 2024-09-11T03:44:44.482 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:44.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:44.483 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:44.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:44.607 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:44.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:44.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:44.627 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T03:44:44.709 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:44.731 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:44.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:44.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:44.833 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:44.854 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:44.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:44.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:44.956 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:44.978 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:44.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:44.979 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:44.990 INFO:tasks.rados.rados.2.smithi175.stdout:1708: done (2 left) 2024-09-11T03:44:44.991 INFO:tasks.rados.rados.2.smithi175.stdout:1709: read oid 28 snap -1 2024-09-11T03:44:44.991 INFO:tasks.rados.rados.2.smithi175.stdout:1709: expect (ObjNum 569 snap 174 seq_num 569) 2024-09-11T03:44:44.991 INFO:tasks.rados.rados.2.smithi175.stdout:1710: copy_from oid 5 from oid 47 current snap is 186 2024-09-11T03:44:44.991 INFO:tasks.rados.rados.2.smithi175.stdout:1711: copy_from oid 43 from oid 36 current snap is 186 2024-09-11T03:44:44.991 INFO:tasks.rados.rados.2.smithi175.stdout:1712: read oid 12 snap 170 2024-09-11T03:44:44.991 INFO:tasks.rados.rados.2.smithi175.stdout:1712: expect deleted 2024-09-11T03:44:44.991 INFO:tasks.rados.rados.2.smithi175.stdout:1713: read oid 21 snap -1 2024-09-11T03:44:44.991 INFO:tasks.rados.rados.2.smithi175.stdout:1713: expect (ObjNum 584 snap 181 seq_num 584) 2024-09-11T03:44:44.991 INFO:tasks.rados.rados.2.smithi175.stdout:1714: read oid 11 snap 170 2024-09-11T03:44:44.991 INFO:tasks.rados.rados.2.smithi175.stdout:1714: expect (ObjNum 522 snap 154 seq_num 522) 2024-09-11T03:44:44.991 INFO:tasks.rados.rados.2.smithi175.stdout:1715: rollback oid 23 current snap is 186 2024-09-11T03:44:44.991 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 23 to 173 2024-09-11T03:44:44.991 INFO:tasks.rados.rados.2.smithi175.stdout:1716: snap_remove snap 172 2024-09-11T03:44:45.080 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:45.103 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:45.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:45.147 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:45.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:45.230 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:45.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:45.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:45.266 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:44 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:44:45.267 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:44 smithi084 ceph-mon[27254]: 2.a scrub ok 2024-09-11T03:44:45.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:45.358 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:45.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:45.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:45.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:44 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:44:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:44 smithi121 ceph-mon[30729]: 2.a scrub ok 2024-09-11T03:44:45.460 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:45.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:44 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:44:45.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:44 smithi175 ceph-mon[30894]: 2.a scrub ok 2024-09-11T03:44:45.482 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:45.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:45.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:45.585 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:45.611 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:45.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:45.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:45.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:45.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:45.741 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:45.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:45.778 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:45.843 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:45.870 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:45.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:45.871 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:45.972 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:45.994 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:45.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:45.997 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:44:46.001 INFO:tasks.rados.rados.2.smithi175.stdout:1716: done (9 left) 2024-09-11T03:44:46.001 INFO:tasks.rados.rados.2.smithi175.stdout:1717: write (excl) oid 44 current snap is 186 2024-09-11T03:44:46.001 INFO:tasks.rados.rados.2.smithi175.stdout:1717: seq_num 598 ranges {450586=474436,1343879=588206} 2024-09-11T03:44:46.007 INFO:tasks.rados.rados.2.smithi175.stdout:1717: writing smithi17557870-44 from 450586 to 925022 tid 1 2024-09-11T03:44:46.014 INFO:tasks.rados.rados.2.smithi175.stdout:1717: writing smithi17557870-44 from 1343879 to 1932085 tid 2 2024-09-11T03:44:46.014 INFO:tasks.rados.rados.2.smithi175.stdout:1718: rollback oid 46 current snap is 186 2024-09-11T03:44:46.014 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 46 to 173 2024-09-11T03:44:46.014 INFO:tasks.rados.rados.2.smithi175.stdout:1707: finishing rollback tid 0 to smithi17557870-25 2024-09-11T03:44:46.014 INFO:tasks.rados.rados.2.smithi175.stdout:1707: finishing rollback tid 1 to smithi17557870-25 2024-09-11T03:44:46.014 INFO:tasks.rados.rados.2.smithi175.stdout:1707: finishing rollback tid 2 to smithi17557870-25 2024-09-11T03:44:46.014 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 491 (ObjNum 496 snap 144 seq_num 496) dirty exists 2024-09-11T03:44:46.014 INFO:tasks.rados.rados.2.smithi175.stdout:1706: finishing rollback tid 0 to smithi17557870-10 2024-09-11T03:44:46.014 INFO:tasks.rados.rados.2.smithi175.stdout:1706: finishing rollback tid 1 to smithi17557870-10 2024-09-11T03:44:46.014 INFO:tasks.rados.rados.2.smithi175.stdout:1706: finishing rollback tid 2 to smithi17557870-10 2024-09-11T03:44:46.014 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 635 (ObjNum 488 snap 141 seq_num 488) dirty exists 2024-09-11T03:44:46.015 INFO:tasks.rados.rados.2.smithi175.stdout:1715: finishing rollback tid 0 to smithi17557870-23 2024-09-11T03:44:46.015 INFO:tasks.rados.rados.2.smithi175.stdout:1706: done (10 left) 2024-09-11T03:44:46.015 INFO:tasks.rados.rados.2.smithi175.stdout:1707: done (9 left) 2024-09-11T03:44:46.015 INFO:tasks.rados.rados.2.smithi175.stdout:1712: done (8 left) 2024-09-11T03:44:46.015 INFO:tasks.rados.rados.2.smithi175.stdout:1719: snap_create 2024-09-11T03:44:46.015 INFO:tasks.rados.rados.2.smithi175.stdout:1715: finishing rollback tid 1 to smithi17557870-23 2024-09-11T03:44:46.015 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 328 (ObjNum 129 snap 0 seq_num 1745352552) dirty dne 2024-09-11T03:44:46.015 INFO:tasks.rados.rados.2.smithi175.stdout:1711: finishing copy_from to smithi17557870-43 2024-09-11T03:44:46.015 INFO:tasks.rados.rados.2.smithi175.stdout:1711: got expected ENOENT (src dne) 2024-09-11T03:44:46.015 INFO:tasks.rados.rados.2.smithi175.stdout:1711: finishing copy_from racing read to smithi17557870-43 2024-09-11T03:44:46.015 INFO:tasks.rados.rados.2.smithi175.stdout:1710: finishing copy_from to smithi17557870-5 2024-09-11T03:44:46.015 INFO:tasks.rados.rados.2.smithi175.stdout:1710: got expected ENOENT (src dne) 2024-09-11T03:44:46.015 INFO:tasks.rados.rados.2.smithi175.stdout:1710: finishing copy_from racing read to smithi17557870-5 2024-09-11T03:44:46.015 INFO:tasks.rados.rados.2.smithi175.stdout:1713: expect (ObjNum 584 snap 181 seq_num 584) 2024-09-11T03:44:46.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:46.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:46.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:46.186 INFO:tasks.rados.rados.2.smithi175.stdout:1714: expect (ObjNum 522 snap 154 seq_num 522) 2024-09-11T03:44:46.224 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:46.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:46.246 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:46.246 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:46.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:45 smithi084 ceph-mon[27254]: 1.e scrub starts 2024-09-11T03:44:46.304 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:45 smithi084 ceph-mon[27254]: 1.e scrub ok 2024-09-11T03:44:46.304 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:45 smithi084 ceph-mon[27254]: 1.1a scrub starts 2024-09-11T03:44:46.304 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:45 smithi084 ceph-mon[27254]: 1.1a scrub ok 2024-09-11T03:44:46.304 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:45 smithi084 ceph-mon[27254]: pgmap v855: 57 pgs: 57 active+clean; 187 MiB data, 792 MiB used, 983 GiB / 983 GiB avail; 8.7 MiB/s rd, 4.2 MiB/s wr, 20 op/s 2024-09-11T03:44:46.304 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:45 smithi084 ceph-mon[27254]: osdmap e467: 12 total, 12 up, 11 in 2024-09-11T03:44:46.304 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:45 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1326307384' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:46.307 INFO:tasks.rados.rados.2.smithi175.stdout:1709: expect (ObjNum 569 snap 174 seq_num 569) 2024-09-11T03:44:46.347 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:46.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:46.369 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:46.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:46.418 INFO:tasks.rados.rados.2.smithi175.stdout:1717: finishing write tid 1 to smithi17557870-44 2024-09-11T03:44:46.418 INFO:tasks.rados.rados.2.smithi175.stdout:1718: finishing rollback tid 0 to smithi17557870-46 2024-09-11T03:44:46.418 INFO:tasks.rados.rados.2.smithi175.stdout:1718: finishing rollback tid 1 to smithi17557870-46 2024-09-11T03:44:46.418 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 525 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T03:44:46.418 INFO:tasks.rados.rados.2.smithi175.stdout:1717: finishing write tid 2 to smithi17557870-44 2024-09-11T03:44:46.418 INFO:tasks.rados.rados.2.smithi175.stdout:1717: finishing write tid 3 to smithi17557870-44 2024-09-11T03:44:46.419 INFO:tasks.rados.rados.2.smithi175.stdout:1717: finishing write tid 4 to smithi17557870-44 2024-09-11T03:44:46.419 INFO:tasks.rados.rados.2.smithi175.stdout:1717: oid 44 updating version 0 to 564 2024-09-11T03:44:46.419 INFO:tasks.rados.rados.2.smithi175.stdout:1717: oid 44 updating version 564 to 565 2024-09-11T03:44:46.419 INFO:tasks.rados.rados.2.smithi175.stdout:1717: oid 44 version 565 is already newer than 563 2024-09-11T03:44:46.419 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 565 (ObjNum 598 snap 186 seq_num 598) dirty exists 2024-09-11T03:44:46.419 INFO:tasks.rados.rados.2.smithi175.stdout:1717: left oid 44 (ObjNum 598 snap 186 seq_num 598) 2024-09-11T03:44:46.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:45 smithi121 ceph-mon[30729]: 1.e scrub starts 2024-09-11T03:44:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:45 smithi121 ceph-mon[30729]: 1.e scrub ok 2024-09-11T03:44:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:45 smithi121 ceph-mon[30729]: 1.1a scrub starts 2024-09-11T03:44:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:45 smithi121 ceph-mon[30729]: 1.1a scrub ok 2024-09-11T03:44:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:45 smithi121 ceph-mon[30729]: pgmap v855: 57 pgs: 57 active+clean; 187 MiB data, 792 MiB used, 983 GiB / 983 GiB avail; 8.7 MiB/s rd, 4.2 MiB/s wr, 20 op/s 2024-09-11T03:44:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:45 smithi121 ceph-mon[30729]: osdmap e467: 12 total, 12 up, 11 in 2024-09-11T03:44:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:45 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1326307384' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:46.435 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:44:46.435 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:44:46.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:45 smithi175 ceph-mon[30894]: 1.e scrub starts 2024-09-11T03:44:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:45 smithi175 ceph-mon[30894]: 1.e scrub ok 2024-09-11T03:44:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:45 smithi175 ceph-mon[30894]: 1.1a scrub starts 2024-09-11T03:44:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:45 smithi175 ceph-mon[30894]: 1.1a scrub ok 2024-09-11T03:44:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:45 smithi175 ceph-mon[30894]: pgmap v855: 57 pgs: 57 active+clean; 187 MiB data, 792 MiB used, 983 GiB / 983 GiB avail; 8.7 MiB/s rd, 4.2 MiB/s wr, 20 op/s 2024-09-11T03:44:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:45 smithi175 ceph-mon[30894]: osdmap e467: 12 total, 12 up, 11 in 2024-09-11T03:44:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:45 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1326307384' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:46.492 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:46.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:46.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:46.594 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:46.615 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:46.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:46.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:46.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:46.743 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:46.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:46.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:46.845 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:46.866 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:46.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:46.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:46.935 INFO:tasks.rados.rados.2.smithi175.stdout:1709: done (8 left) 2024-09-11T03:44:46.935 INFO:tasks.rados.rados.2.smithi175.stdout:1710: done (7 left) 2024-09-11T03:44:46.935 INFO:tasks.rados.rados.2.smithi175.stdout:1711: done (6 left) 2024-09-11T03:44:46.935 INFO:tasks.rados.rados.2.smithi175.stdout:1713: done (5 left) 2024-09-11T03:44:46.935 INFO:tasks.rados.rados.2.smithi175.stdout:1714: done (4 left) 2024-09-11T03:44:46.935 INFO:tasks.rados.rados.2.smithi175.stdout:1715: done (3 left) 2024-09-11T03:44:46.935 INFO:tasks.rados.rados.2.smithi175.stdout:1717: done (2 left) 2024-09-11T03:44:46.935 INFO:tasks.rados.rados.2.smithi175.stdout:1718: done (1 left) 2024-09-11T03:44:46.936 INFO:tasks.rados.rados.2.smithi175.stdout:1719: done (0 left) 2024-09-11T03:44:46.936 INFO:tasks.rados.rados.2.smithi175.stdout:1720: copy_from oid 41 from oid 45 current snap is 187 2024-09-11T03:44:46.936 INFO:tasks.rados.rados.2.smithi175.stdout:1721: snap_create 2024-09-11T03:44:46.947 INFO:tasks.rados.rados.2.smithi175.stdout:1720: finishing copy_from racing read to smithi17557870-41 2024-09-11T03:44:46.951 INFO:tasks.rados.rados.2.smithi175.stdout:1720: finishing copy_from to smithi17557870-41 2024-09-11T03:44:46.951 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 481 (ObjNum 416 snap 127 seq_num 416) dirty exists 2024-09-11T03:44:46.968 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:46.990 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:46.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:46.991 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:47.092 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:47.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:47.120 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:47.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:47.147 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:44:47.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:46 smithi084 ceph-mon[27254]: 2.11 scrub starts 2024-09-11T03:44:47.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:46 smithi084 ceph-mon[27254]: 2.11 scrub ok 2024-09-11T03:44:47.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:46 smithi084 ceph-mon[27254]: 2.18 scrub starts 2024-09-11T03:44:47.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:46 smithi084 ceph-mon[27254]: 2.18 scrub ok 2024-09-11T03:44:47.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:46 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:44:47.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:46 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:44:47.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1326307384' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:47.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:46 smithi084 ceph-mon[27254]: osdmap e468: 12 total, 12 up, 11 in 2024-09-11T03:44:47.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:46 smithi084 ceph-mon[27254]: osdmap e469: 12 total, 12 up, 11 in 2024-09-11T03:44:47.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T03:44:47.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:47.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:47.249 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:47.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:47.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:47.376 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:47.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:47.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:46 smithi121 ceph-mon[30729]: 2.11 scrub starts 2024-09-11T03:44:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:46 smithi121 ceph-mon[30729]: 2.11 scrub ok 2024-09-11T03:44:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:46 smithi121 ceph-mon[30729]: 2.18 scrub starts 2024-09-11T03:44:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:46 smithi121 ceph-mon[30729]: 2.18 scrub ok 2024-09-11T03:44:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:46 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:44:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:46 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:44:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1326307384' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:46 smithi121 ceph-mon[30729]: osdmap e468: 12 total, 12 up, 11 in 2024-09-11T03:44:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:46 smithi121 ceph-mon[30729]: osdmap e469: 12 total, 12 up, 11 in 2024-09-11T03:44:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:46 smithi175 ceph-mon[30894]: 2.11 scrub starts 2024-09-11T03:44:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:46 smithi175 ceph-mon[30894]: 2.11 scrub ok 2024-09-11T03:44:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:46 smithi175 ceph-mon[30894]: 2.18 scrub starts 2024-09-11T03:44:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:46 smithi175 ceph-mon[30894]: 2.18 scrub ok 2024-09-11T03:44:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:46 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:44:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:46 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:44:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1326307384' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:46 smithi175 ceph-mon[30894]: osdmap e468: 12 total, 12 up, 11 in 2024-09-11T03:44:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:46 smithi175 ceph-mon[30894]: osdmap e469: 12 total, 12 up, 11 in 2024-09-11T03:44:47.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:47.500 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:47.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:47.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:47.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:47.628 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:47.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:47.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:47.730 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:47.733 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:47.744 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:47.756 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:47.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:47.757 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:47.858 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:47.883 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:47.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:47.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:47.936 INFO:tasks.rados.rados.2.smithi175.stdout:1720: done (1 left) 2024-09-11T03:44:47.936 INFO:tasks.rados.rados.2.smithi175.stdout:1721: done (0 left) 2024-09-11T03:44:47.936 INFO:tasks.rados.rados.2.smithi175.stdout:1722: delete oid 10 current snap is 188 2024-09-11T03:44:47.938 INFO:tasks.rados.rados.2.smithi175.stdout:1722: done (0 left) 2024-09-11T03:44:47.938 INFO:tasks.rados.rados.2.smithi175.stdout:1723: rollback oid 41 current snap is 188 2024-09-11T03:44:47.938 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 41 to 185 2024-09-11T03:44:47.938 INFO:tasks.rados.rados.2.smithi175.stdout:1724: read oid 11 snap -1 2024-09-11T03:44:47.938 INFO:tasks.rados.rados.2.smithi175.stdout:1724: expect deleted 2024-09-11T03:44:47.938 INFO:tasks.rados.rados.2.smithi175.stdout:1725: snap_remove snap 173 2024-09-11T03:44:47.985 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:48.010 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:48.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:48.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:48.113 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:48.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:48.141 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:48.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:48.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:47 smithi084 ceph-mon[27254]: 1.18 scrub starts 2024-09-11T03:44:48.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:47 smithi084 ceph-mon[27254]: 1.18 scrub ok 2024-09-11T03:44:48.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:48 smithi084 ceph-mon[27254]: pgmap v858: 57 pgs: 57 active+clean; 190 MiB data, 793 MiB used, 983 GiB / 983 GiB avail; 3.8 MiB/s rd, 797 KiB/s wr, 6 op/s 2024-09-11T03:44:48.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:48 smithi084 ceph-mon[27254]: 2.13 scrub starts 2024-09-11T03:44:48.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:48 smithi084 ceph-mon[27254]: 2.13 scrub ok 2024-09-11T03:44:48.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:48 smithi084 ceph-mon[27254]: osdmap e470: 12 total, 12 up, 11 in 2024-09-11T03:44:48.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:48.267 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:48.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:48.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:48.369 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:48.391 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:48.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:48.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:48 smithi121 ceph-mon[30729]: 1.18 scrub starts 2024-09-11T03:44:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:48 smithi121 ceph-mon[30729]: 1.18 scrub ok 2024-09-11T03:44:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:48 smithi121 ceph-mon[30729]: pgmap v858: 57 pgs: 57 active+clean; 190 MiB data, 793 MiB used, 983 GiB / 983 GiB avail; 3.8 MiB/s rd, 797 KiB/s wr, 6 op/s 2024-09-11T03:44:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:48 smithi121 ceph-mon[30729]: 2.13 scrub starts 2024-09-11T03:44:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:48 smithi121 ceph-mon[30729]: 2.13 scrub ok 2024-09-11T03:44:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:48 smithi121 ceph-mon[30729]: osdmap e470: 12 total, 12 up, 11 in 2024-09-11T03:44:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:48 smithi175 ceph-mon[30894]: 1.18 scrub starts 2024-09-11T03:44:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:48 smithi175 ceph-mon[30894]: 1.18 scrub ok 2024-09-11T03:44:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:48 smithi175 ceph-mon[30894]: pgmap v858: 57 pgs: 57 active+clean; 190 MiB data, 793 MiB used, 983 GiB / 983 GiB avail; 3.8 MiB/s rd, 797 KiB/s wr, 6 op/s 2024-09-11T03:44:48.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:48 smithi175 ceph-mon[30894]: 2.13 scrub starts 2024-09-11T03:44:48.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:48 smithi175 ceph-mon[30894]: 2.13 scrub ok 2024-09-11T03:44:48.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:48 smithi175 ceph-mon[30894]: osdmap e470: 12 total, 12 up, 11 in 2024-09-11T03:44:48.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:48.519 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:48.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:48.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:48.566 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:44:48.566 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:44:48.621 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:48.643 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:48.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:48.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:48.671 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:44:48.671 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:44:48.671 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-11T03:44:48.671 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-11T03:44:48.745 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:48.772 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:48.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:48.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:48.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:48.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:48.901 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:48.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:48.938 INFO:tasks.rados.rados.2.smithi175.stdout:1725: done (2 left) 2024-09-11T03:44:48.938 INFO:tasks.rados.rados.2.smithi175.stdout:1726: delete oid 39 current snap is 188 2024-09-11T03:44:48.938 INFO:tasks.rados.rados.2.smithi175.stdout:1723: finishing rollback tid 0 to smithi17557870-41 2024-09-11T03:44:48.938 INFO:tasks.rados.rados.2.smithi175.stdout:1723: finishing rollback tid 1 to smithi17557870-41 2024-09-11T03:44:48.938 INFO:tasks.rados.rados.2.smithi175.stdout:1723: finishing rollback tid 2 to smithi17557870-41 2024-09-11T03:44:48.939 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 489 (ObjNum 566 snap 172 seq_num 566) dirty exists 2024-09-11T03:44:48.941 INFO:tasks.rados.rados.2.smithi175.stdout:1723: done (2 left) 2024-09-11T03:44:48.941 INFO:tasks.rados.rados.2.smithi175.stdout:1724: done (1 left) 2024-09-11T03:44:48.942 INFO:tasks.rados.rados.2.smithi175.stdout:1726: done (0 left) 2024-09-11T03:44:48.942 INFO:tasks.rados.rados.2.smithi175.stdout:1727: copy_from oid 38 from oid 49 current snap is 188 2024-09-11T03:44:48.942 INFO:tasks.rados.rados.2.smithi175.stdout:1728: snap_remove snap 185 2024-09-11T03:44:49.003 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:49.026 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:49.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:49.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:49.128 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:49.150 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:49.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:49.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:49.252 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:49.276 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:49.277 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:49.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:49 smithi084 ceph-mon[27254]: 2.10 scrub starts 2024-09-11T03:44:49.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:49 smithi084 ceph-mon[27254]: 2.10 scrub ok 2024-09-11T03:44:49.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:49 smithi084 ceph-mon[27254]: osdmap e471: 12 total, 12 up, 11 in 2024-09-11T03:44:49.378 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:49.401 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:49.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:49.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:49.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:49 smithi121 ceph-mon[30729]: 2.10 scrub starts 2024-09-11T03:44:49.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:49 smithi121 ceph-mon[30729]: 2.10 scrub ok 2024-09-11T03:44:49.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:49 smithi121 ceph-mon[30729]: osdmap e471: 12 total, 12 up, 11 in 2024-09-11T03:44:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:49 smithi175 ceph-mon[30894]: 2.10 scrub starts 2024-09-11T03:44:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:49 smithi175 ceph-mon[30894]: 2.10 scrub ok 2024-09-11T03:44:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:49 smithi175 ceph-mon[30894]: osdmap e471: 12 total, 12 up, 11 in 2024-09-11T03:44:49.498 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:44:49.498 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:44:49.503 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:49.529 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:44:49.529 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:44:49.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:49.530 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:49.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:49.547 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:49.549 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:44:49.550 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:44:49.578 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T03:44:49.581 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:44:49.581 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:44:49.598 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:44:49.599 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:44:49.616 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:44:49.616 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:44:49.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:49.632 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:44:49.633 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:44:49.651 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:44:49.651 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:44:49.653 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:49.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:49.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:49.668 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:44:49.668 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:44:49.680 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:44:49.686 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:44:49.686 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:44:49.719 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:44:49.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:44:49.753 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:44:49.753 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:44:49.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:49.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:49.779 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:49.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:49.880 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:49.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:49.903 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:49.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:49.939 INFO:tasks.rados.rados.2.smithi175.stdout:1728: done (1 left) 2024-09-11T03:44:49.939 INFO:tasks.rados.rados.2.smithi175.stdout:1729: delete oid 11 current snap is 188 2024-09-11T03:44:49.939 INFO:tasks.rados.rados.2.smithi175.stdout:1727: finishing copy_from racing read to smithi17557870-38 2024-09-11T03:44:49.939 INFO:tasks.rados.rados.2.smithi175.stdout:1727: finishing copy_from to smithi17557870-38 2024-09-11T03:44:49.939 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 509 (ObjNum 565 snap 172 seq_num 565) dirty exists 2024-09-11T03:44:49.940 INFO:tasks.rados.rados.2.smithi175.stdout:1727: done (1 left) 2024-09-11T03:44:49.940 INFO:tasks.rados.rados.2.smithi175.stdout:1729: done (0 left) 2024-09-11T03:44:49.940 INFO:tasks.rados.rados.2.smithi175.stdout:1730: snap_remove snap 184 2024-09-11T03:44:50.004 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:50.025 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:50.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:50.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:50.128 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:50.153 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:50.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:50.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:50.192 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:50 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:44:50.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:50 smithi175 ceph-mon[30894]: 2.b scrub ok 2024-09-11T03:44:50.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:50 smithi175 ceph-mon[30894]: pgmap v861: 57 pgs: 57 active+clean; 190 MiB data, 796 MiB used, 983 GiB / 983 GiB avail; 9.7 MiB/s rd, 260 KiB/s wr, 11 op/s 2024-09-11T03:44:50.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:50 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2611522659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:50.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:50 smithi175 ceph-mon[30894]: osdmap e472: 12 total, 12 up, 11 in 2024-09-11T03:44:50.255 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:50.281 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:50.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:50.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:50.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:50 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:44:50.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:50 smithi084 ceph-mon[27254]: 2.b scrub ok 2024-09-11T03:44:50.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:50 smithi084 ceph-mon[27254]: pgmap v861: 57 pgs: 57 active+clean; 190 MiB data, 796 MiB used, 983 GiB / 983 GiB avail; 9.7 MiB/s rd, 260 KiB/s wr, 11 op/s 2024-09-11T03:44:50.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:50 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2611522659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:50.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:50 smithi084 ceph-mon[27254]: osdmap e472: 12 total, 12 up, 11 in 2024-09-11T03:44:50.357 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:44:50.357 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:44:50.357 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 472, 2024-09-11T03:44:50.357 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "10fe7946-6fee-11ef-bcea-c7b262605968", 2024-09-11T03:44:50.357 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-09-11T03:30:30.532652+0000", 2024-09-11T03:44:50.357 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-09-11T03:44:49.933276+0000", 2024-09-11T03:44:50.357 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-09-11T03:37:26.787638+0000", 2024-09-11T03:44:50.357 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-09-11T03:43:51.906016+0000", 2024-09-11T03:44:50.357 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T03:44:50.358 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5799936, 2024-09-11T03:44:50.358 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-09-11T03:44:50.358 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-09-11T03:44:50.358 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-09-11T03:44:50.358 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-09-11T03:44:50.358 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-09-11T03:44:50.358 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:44:50.358 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 26, 2024-09-11T03:44:50.358 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T03:44:50.358 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T03:44:50.358 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T03:44:50.358 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-09-11T03:44:50.358 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 2, 2024-09-11T03:44:50.359 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 12, 2024-09-11T03:44:50.359 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-09-11T03:44:50.359 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-09-11T03:44:50.359 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-09-11T03:44:50.359 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-09-11T03:44:50.359 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-09-11T03:44:50.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.359 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-09-11T03:44:50.359 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-09-11T03:44:50.359 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-11T03:33:42.916933+0000", 2024-09-11T03:44:50.359 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-11T03:44:50.359 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-11T03:44:50.359 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-11T03:44:50.360 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-11T03:44:50.360 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-11T03:44:50.360 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-11T03:44:50.360 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-11T03:44:50.360 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-11T03:44:50.360 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T03:44:50.360 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T03:44:50.360 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-11T03:44:50.360 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-11T03:44:50.360 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-11T03:44:50.360 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 31, 2024-09-11T03:44:50.360 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 31, 2024-09-11T03:44:50.361 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 31, 2024-09-11T03:44:50.361 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 31, 2024-09-11T03:44:50.361 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 31, 2024-09-11T03:44:50.361 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-11T03:44:50.361 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-11T03:44:50.361 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-11T03:44:50.361 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-11T03:44:50.361 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-11T03:44:50.361 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-11T03:44:50.361 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-11T03:44:50.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.361 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "222", 2024-09-11T03:44:50.361 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-11T03:44:50.362 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T03:44:50.362 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "220", 2024-09-11T03:44:50.362 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-11T03:44:50.362 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-11T03:44:50.362 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-11T03:44:50.362 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-11T03:44:50.362 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-11T03:44:50.362 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-11T03:44:50.362 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-11T03:44:50.362 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-11T03:44:50.362 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-11T03:44:50.362 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-11T03:44:50.363 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-11T03:44:50.363 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-11T03:44:50.363 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-11T03:44:50.363 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-11T03:44:50.363 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-11T03:44:50.363 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T03:44:50.363 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T03:44:50.363 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T03:44:50.363 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-11T03:44:50.363 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-11T03:44:50.363 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-11T03:44:50.363 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-11T03:44:50.364 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-11T03:44:50.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.364 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-11T03:44:50.364 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-11T03:44:50.364 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-11T03:44:50.364 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-11T03:44:50.364 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-11T03:44:50.364 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T03:44:50.364 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-11T03:44:50.364 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-11T03:44:50.364 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-11T03:44:50.364 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-11T03:44:50.364 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-11T03:44:50.365 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-11T03:44:50.365 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-09-11T03:44:50.365 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-09-11T03:44:50.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.365 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-11T03:44:50.365 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-09-11T03:44:50.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.365 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-11T03:44:50.365 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-11T03:44:50.365 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.7699999809265137, 2024-09-11T03:44:50.365 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.7699999809265137, 2024-09-11T03:44:50.365 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-11T03:44:50.365 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.7699999809265137, 2024-09-11T03:44:50.366 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.7699999809265137, 2024-09-11T03:44:50.366 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.88858538866043091, 2024-09-11T03:44:50.366 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-11T03:44:50.366 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-09-11T03:44:50.366 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.366 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-11T03:44:50.366 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_0", 2024-09-11T03:44:50.366 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-11T03:38:28.163534+0000", 2024-09-11T03:44:50.366 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-11T03:44:50.366 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T03:44:50.366 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-11T03:44:50.367 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-11T03:44:50.367 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-11T03:44:50.367 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-11T03:44:50.367 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-11T03:44:50.367 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-11T03:44:50.367 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T03:44:50.367 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T03:44:50.367 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-11T03:44:50.367 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-11T03:44:50.367 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-11T03:44:50.367 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 26, 2024-09-11T03:44:50.367 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 26, 2024-09-11T03:44:50.367 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 26, 2024-09-11T03:44:50.368 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 26, 2024-09-11T03:44:50.368 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 26, 2024-09-11T03:44:50.368 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-11T03:44:50.368 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-11T03:44:50.368 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-11T03:44:50.368 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-11T03:44:50.368 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-11T03:44:50.368 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-11T03:44:50.368 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-11T03:44:50.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.368 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "472", 2024-09-11T03:44:50.368 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-11T03:44:50.368 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T03:44:50.369 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "276", 2024-09-11T03:44:50.369 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-11T03:44:50.369 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-11T03:44:50.369 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 189, 2024-09-11T03:44:50.369 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 472, 2024-09-11T03:44:50.369 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-11T03:44:50.369 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-11T03:44:50.369 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-11T03:44:50.369 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-11T03:44:50.369 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-11T03:44:50.369 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-11T03:44:50.369 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-11T03:44:50.369 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-11T03:44:50.369 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-11T03:44:50.370 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-11T03:44:50.370 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-11T03:44:50.370 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T03:44:50.370 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T03:44:50.370 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T03:44:50.370 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-11T03:44:50.370 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-11T03:44:50.370 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-11T03:44:50.370 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-11T03:44:50.370 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-11T03:44:50.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.370 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-11T03:44:50.370 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-11T03:44:50.371 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-11T03:44:50.371 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-11T03:44:50.371 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-11T03:44:50.371 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T03:44:50.371 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-11T03:44:50.371 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-11T03:44:50.371 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-11T03:44:50.371 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-11T03:44:50.371 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-11T03:44:50.371 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-11T03:44:50.371 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-11T03:44:50.371 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-09-11T03:44:50.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.371 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-11T03:44:50.372 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-11T03:44:50.372 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.690000057220459, 2024-09-11T03:44:50.372 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.690000057220459, 2024-09-11T03:44:50.372 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-11T03:44:50.372 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.690000057220459, 2024-09-11T03:44:50.372 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.690000057220459, 2024-09-11T03:44:50.372 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.88858538866043091, 2024-09-11T03:44:50.372 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-11T03:44:50.372 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-09-11T03:44:50.372 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.372 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.372 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:44:50.372 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-11T03:44:50.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.373 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-11T03:44:50.373 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "9a431e48-e46b-44f9-bfa7-cb56c1484e1c", 2024-09-11T03:44:50.373 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:50.373 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:50.373 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:50.373 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:50.373 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:50.373 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:50.373 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 8, 2024-09-11T03:44:50.373 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 447, 2024-09-11T03:44:50.373 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:50.373 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:50.374 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:50.374 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.374 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.374 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-09-11T03:44:50.374 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:44:50.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.374 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.374 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-09-11T03:44:50.374 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:44:50.374 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.374 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.375 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:50.375 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.375 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.375 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-09-11T03:44:50.375 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:44:50.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.375 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.375 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-09-11T03:44:50.375 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:44:50.375 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.375 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.376 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:50.376 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.376 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.376 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-09-11T03:44:50.376 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:44:50.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.376 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.376 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-09-11T03:44:50.376 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:44:50.376 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.377 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.377 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:50.377 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.377 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.377 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-09-11T03:44:50.377 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:44:50.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.377 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.377 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-09-11T03:44:50.377 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1376667411 2024-09-11T03:44:50.378 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.378 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.378 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6803/1376667411", 2024-09-11T03:44:50.378 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6805/1376667411", 2024-09-11T03:44:50.378 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6809/1376667411", 2024-09-11T03:44:50.378 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/1376667411", 2024-09-11T03:44:50.378 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:50.378 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:50.378 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:50.378 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.378 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-11T03:44:50.379 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "62fd6e25-3508-40b4-a0f4-f7de137e3bb5", 2024-09-11T03:44:50.379 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:50.379 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:50.379 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:50.379 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 0.871673583984375, 2024-09-11T03:44:50.379 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:50.379 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:50.379 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-09-11T03:44:50.379 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 465, 2024-09-11T03:44:50.379 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:50.379 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:50.379 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:50.379 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.380 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.380 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-09-11T03:44:50.380 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:44:50.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.380 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.380 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-09-11T03:44:50.380 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:44:50.380 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.380 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.380 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:50.380 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.381 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.381 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-09-11T03:44:50.381 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:44:50.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.381 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.381 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-09-11T03:44:50.381 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:44:50.382 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.382 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.382 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:50.382 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.382 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.382 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-09-11T03:44:50.382 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:44:50.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.382 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.382 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6817", 2024-09-11T03:44:50.383 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:44:50.383 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.383 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.383 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:50.383 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.383 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.383 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-09-11T03:44:50.384 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:44:50.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.384 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.384 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-09-11T03:44:50.384 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3138939249 2024-09-11T03:44:50.384 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.384 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.384 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6811/3138939249", 2024-09-11T03:44:50.384 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6813/3138939249", 2024-09-11T03:44:50.384 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6817/3138939249", 2024-09-11T03:44:50.385 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/3138939249", 2024-09-11T03:44:50.385 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:50.385 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:50.385 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:50.385 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.385 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-11T03:44:50.385 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ff351425-4715-49dc-be49-f27bf205d57e", 2024-09-11T03:44:50.385 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:50.385 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:50.385 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:50.385 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:50.386 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:50.386 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:50.386 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 18, 2024-09-11T03:44:50.386 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 416, 2024-09-11T03:44:50.386 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:50.386 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:50.386 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:50.386 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.386 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.386 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6818", 2024-09-11T03:44:50.386 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:44:50.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.387 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.387 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6819", 2024-09-11T03:44:50.387 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:44:50.387 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.387 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.387 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:50.387 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.387 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.387 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6820", 2024-09-11T03:44:50.387 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:44:50.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.388 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.388 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6821", 2024-09-11T03:44:50.388 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:44:50.388 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.388 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.388 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:50.388 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.388 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.388 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6824", 2024-09-11T03:44:50.389 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:44:50.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.389 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.389 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6825", 2024-09-11T03:44:50.389 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:44:50.389 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.389 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.389 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:50.389 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.392 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.392 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6822", 2024-09-11T03:44:50.392 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:44:50.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.392 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.392 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6823", 2024-09-11T03:44:50.392 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3001811591 2024-09-11T03:44:50.392 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.392 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.392 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6819/3001811591", 2024-09-11T03:44:50.392 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6821/3001811591", 2024-09-11T03:44:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6825/3001811591", 2024-09-11T03:44:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6823/3001811591", 2024-09-11T03:44:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-11T03:44:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "2518a93e-c612-4731-91ce-5685c25ffa9f", 2024-09-11T03:44:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:50.393 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 26, 2024-09-11T03:44:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 332, 2024-09-11T03:44:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6826", 2024-09-11T03:44:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:44:50.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6827", 2024-09-11T03:44:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:44:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6828", 2024-09-11T03:44:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:44:50.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6829", 2024-09-11T03:44:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:44:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.396 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6832", 2024-09-11T03:44:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:44:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6833", 2024-09-11T03:44:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:44:50.398 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6830", 2024-09-11T03:44:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:44:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6831", 2024-09-11T03:44:50.399 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 302512129 2024-09-11T03:44:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6827/302512129", 2024-09-11T03:44:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6829/302512129", 2024-09-11T03:44:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6833/302512129", 2024-09-11T03:44:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6831/302512129", 2024-09-11T03:44:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-11T03:44:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "d12f3c2d-95cc-4276-a7e1-bee982d58948", 2024-09-11T03:44:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 0, 2024-09-11T03:44:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-09-11T03:44:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 398, 2024-09-11T03:44:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:50.401 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6800", 2024-09-11T03:44:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:44:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6801", 2024-09-11T03:44:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:44:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.402 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6802", 2024-09-11T03:44:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:44:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6803", 2024-09-11T03:44:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:44:50.403 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6806", 2024-09-11T03:44:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:44:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6807", 2024-09-11T03:44:50.404 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:44:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6804", 2024-09-11T03:44:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:44:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6805", 2024-09-11T03:44:50.405 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3873214513 2024-09-11T03:44:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.121:6801/3873214513", 2024-09-11T03:44:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.121:6803/3873214513", 2024-09-11T03:44:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.121:6807/3873214513", 2024-09-11T03:44:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.121:6805/3873214513", 2024-09-11T03:44:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.406 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-11T03:44:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "32164dd5-7e8f-4925-8526-bbb1e33e3364", 2024-09-11T03:44:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 38, 2024-09-11T03:44:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 414, 2024-09-11T03:44:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6808", 2024-09-11T03:44:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:44:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6809", 2024-09-11T03:44:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:44:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:50.408 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6810", 2024-09-11T03:44:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:44:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6811", 2024-09-11T03:44:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:44:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.409 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6814", 2024-09-11T03:44:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:44:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6815", 2024-09-11T03:44:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:44:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.410 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6812", 2024-09-11T03:44:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:44:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6813", 2024-09-11T03:44:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 918612194 2024-09-11T03:44:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.411 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.121:6809/918612194", 2024-09-11T03:44:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.121:6811/918612194", 2024-09-11T03:44:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.121:6815/918612194", 2024-09-11T03:44:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.121:6813/918612194", 2024-09-11T03:44:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-11T03:44:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "3f107a88-803f-4cd8-804a-241310c282ea", 2024-09-11T03:44:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:50.412 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-09-11T03:44:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-09-11T03:44:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 45, 2024-09-11T03:44:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 95, 2024-09-11T03:44:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6816", 2024-09-11T03:44:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:44:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6817", 2024-09-11T03:44:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:44:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6818", 2024-09-11T03:44:50.414 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:44:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6819", 2024-09-11T03:44:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:44:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6822", 2024-09-11T03:44:50.415 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:44:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6823", 2024-09-11T03:44:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:44:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.416 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6820", 2024-09-11T03:44:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:44:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6821", 2024-09-11T03:44:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3384813363 2024-09-11T03:44:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.121:6817/3384813363", 2024-09-11T03:44:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.121:6819/3384813363", 2024-09-11T03:44:50.417 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.121:6823/3384813363", 2024-09-11T03:44:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.121:6821/3384813363", 2024-09-11T03:44:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-11T03:44:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "dfee7f95-1fb9-4493-9907-cb10d1837bdb", 2024-09-11T03:44:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 0.791351318359375, 2024-09-11T03:44:50.418 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 52, 2024-09-11T03:44:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 465, 2024-09-11T03:44:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6824", 2024-09-11T03:44:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:44:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6825", 2024-09-11T03:44:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:44:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6826", 2024-09-11T03:44:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:44:50.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6827", 2024-09-11T03:44:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:44:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6830", 2024-09-11T03:44:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:44:50.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6831", 2024-09-11T03:44:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:44:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6828", 2024-09-11T03:44:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:44:50.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.121:6829", 2024-09-11T03:44:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 449222054 2024-09-11T03:44:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.121:6825/449222054", 2024-09-11T03:44:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.121:6827/449222054", 2024-09-11T03:44:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.121:6831/449222054", 2024-09-11T03:44:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.121:6829/449222054", 2024-09-11T03:44:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:50.423 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-09-11T03:44:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4", 2024-09-11T03:44:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 60, 2024-09-11T03:44:50.424 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 414, 2024-09-11T03:44:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6800", 2024-09-11T03:44:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:44:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6801", 2024-09-11T03:44:50.425 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:44:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6802", 2024-09-11T03:44:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:44:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6803", 2024-09-11T03:44:50.426 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:44:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6806", 2024-09-11T03:44:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:44:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.427 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6807", 2024-09-11T03:44:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:44:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6804", 2024-09-11T03:44:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:44:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.428 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6805", 2024-09-11T03:44:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2660128384 2024-09-11T03:44:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.175:6801/2660128384", 2024-09-11T03:44:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.175:6803/2660128384", 2024-09-11T03:44:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.175:6807/2660128384", 2024-09-11T03:44:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.175:6805/2660128384", 2024-09-11T03:44:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:50.429 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-09-11T03:44:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "50162e42-0443-41d1-8709-a0db61c087da", 2024-09-11T03:44:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 69, 2024-09-11T03:44:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 312, 2024-09-11T03:44:50.430 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6808", 2024-09-11T03:44:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:44:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6809", 2024-09-11T03:44:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:44:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.431 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6810", 2024-09-11T03:44:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:44:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6811", 2024-09-11T03:44:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:44:50.432 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6814", 2024-09-11T03:44:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:44:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6815", 2024-09-11T03:44:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:44:50.433 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6812", 2024-09-11T03:44:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:44:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6813", 2024-09-11T03:44:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1270991649 2024-09-11T03:44:50.434 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.175:6809/1270991649", 2024-09-11T03:44:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.175:6811/1270991649", 2024-09-11T03:44:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.175:6815/1270991649", 2024-09-11T03:44:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.175:6813/1270991649", 2024-09-11T03:44:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-09-11T03:44:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4b9426e1-bac8-47d3-ad58-54b89e8d1eb1", 2024-09-11T03:44:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 76, 2024-09-11T03:44:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 417, 2024-09-11T03:44:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:50.436 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6816", 2024-09-11T03:44:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:44:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6817", 2024-09-11T03:44:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:44:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:50.437 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6818", 2024-09-11T03:44:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:44:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6819", 2024-09-11T03:44:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:44:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.438 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6822", 2024-09-11T03:44:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:44:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6823", 2024-09-11T03:44:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:44:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.439 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6820", 2024-09-11T03:44:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:44:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6821", 2024-09-11T03:44:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3877631021 2024-09-11T03:44:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.175:6817/3877631021", 2024-09-11T03:44:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.175:6819/3877631021", 2024-09-11T03:44:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.175:6823/3877631021", 2024-09-11T03:44:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.175:6821/3877631021", 2024-09-11T03:44:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-09-11T03:44:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "cd1a28a1-413c-4274-9876-e1bdb7703be6", 2024-09-11T03:44:50.441 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-11T03:44:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-11T03:44:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-11T03:44:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-11T03:44:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-11T03:44:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-11T03:44:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 83, 2024-09-11T03:44:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 449, 2024-09-11T03:44:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-11T03:44:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-11T03:44:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-11T03:44:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.442 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6824", 2024-09-11T03:44:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:44:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6825", 2024-09-11T03:44:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:44:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-11T03:44:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.443 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6826", 2024-09-11T03:44:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:44:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6827", 2024-09-11T03:44:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:44:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-11T03:44:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.444 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6830", 2024-09-11T03:44:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:44:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6831", 2024-09-11T03:44:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:44:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-11T03:44:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-11T03:44:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.445 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-11T03:44:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6828", 2024-09-11T03:44:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:44:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-09-11T03:44:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.175:6829", 2024-09-11T03:44:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1150051236 2024-09-11T03:44:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.175:6825/1150051236", 2024-09-11T03:44:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.175:6827/1150051236", 2024-09-11T03:44:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.175:6831/1150051236", 2024-09-11T03:44:50.446 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.175:6829/1150051236", 2024-09-11T03:44:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-11T03:44:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-11T03:44:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-11T03:44:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:44:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-09-11T03:44:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-11T03:44:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:50.447 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:43:37.917862+0000", 2024-09-11T03:44:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-11T03:44:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:44:19.071328+0000", 2024-09-11T03:44:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:50.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-11T03:44:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:44:40.814709+0000", 2024-09-11T03:44:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-11T03:44:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:50.449 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:50.450 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:50.450 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:50.450 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:50.450 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:43:50.817861+0000", 2024-09-11T03:44:50.450 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:50.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.450 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-11T03:44:50.450 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:50.450 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:50.450 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:50.450 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:50.450 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:44:40.596330+0000", 2024-09-11T03:44:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-11T03:44:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:44:41.447558+0000", 2024-09-11T03:44:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:50.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-11T03:44:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-09-11T03:44:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:43:31.134086+0000", 2024-09-11T03:44:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-11T03:44:50.452 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:44:12.815905+0000", 2024-09-11T03:44:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 8, 2024-09-11T03:44:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:44:11.646575+0000", 2024-09-11T03:44:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 9, 2024-09-11T03:44:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:44:05.536067+0000", 2024-09-11T03:44:50.454 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 10, 2024-09-11T03:44:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:44:17.122767+0000", 2024-09-11T03:44:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.455 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 11, 2024-09-11T03:44:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-11T03:44:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-11T03:44:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-11T03:44:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540719139924082687, 2024-09-11T03:44:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-11T03:44:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-11T03:43:45.212841+0000", 2024-09-11T03:44:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-11T03:44:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:44:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [], 2024-09-11T03:44:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [ 2024-09-11T03:44:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.456 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "1.4", 2024-09-11T03:44:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-11T03:44:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 7, 2024-09-11T03:44:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-11T03:44:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.2", 2024-09-11T03:44:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-11T03:44:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 9, 2024-09-11T03:44:50.457 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 11 2024-09-11T03:44:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "2.d", 2024-09-11T03:44:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-11T03:44:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 9, 2024-09-11T03:44:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-11T03:44:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.458 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:44:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-09-11T03:44:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-09-11T03:44:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-09-11T03:44:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-09-11T03:44:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/108136515": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:44:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3411515187": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:44:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6801/3365920551": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:44:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/3365920551": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:44:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3885577441": "2024-09-12T03:31:03.077186+0000", 2024-09-11T03:44:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/22039254": "2024-09-12T03:31:03.077186+0000", 2024-09-11T03:44:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6801/3496645948": "2024-09-12T03:31:03.077186+0000", 2024-09-11T03:44:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/4206337830": "2024-09-12T03:31:26.598965+0000", 2024-09-11T03:44:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/3496645948": "2024-09-12T03:31:03.077186+0000", 2024-09-11T03:44:50.459 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1989775875": "2024-09-12T03:31:03.077186+0000" 2024-09-11T03:44:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-09-11T03:44:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-09-11T03:44:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-09-11T03:44:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-09-11T03:44:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-09-11T03:44:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-09-11T03:44:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-09-11T03:44:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-09-11T03:44:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [ 2024-09-11T03:44:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.460 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-11T03:44:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-11T03:44:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 174, 2024-09-11T03:44:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 2 2024-09-11T03:44:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 181, 2024-09-11T03:44:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 4 2024-09-11T03:44:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 187, 2024-09-11T03:44:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-11T03:44:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:44:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [ 2024-09-11T03:44:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-11T03:44:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-11T03:44:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 187, 2024-09-11T03:44:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-11T03:44:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-11T03:44:50.462 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-09-11T03:44:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-09-11T03:44:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-09-11T03:44:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-09-11T03:44:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-09-11T03:44:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-09-11T03:44:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-09-11T03:44:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-09-11T03:44:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-09-11T03:44:50.463 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:50.463 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:44:50.464 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:50 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:44:50.464 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:50 smithi121 ceph-mon[30729]: 2.b scrub ok 2024-09-11T03:44:50.464 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:50 smithi121 ceph-mon[30729]: pgmap v861: 57 pgs: 57 active+clean; 190 MiB data, 796 MiB used, 983 GiB / 983 GiB avail; 9.7 MiB/s rd, 260 KiB/s wr, 11 op/s 2024-09-11T03:44:50.464 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:50 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2611522659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:50.464 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:50 smithi121 ceph-mon[30729]: osdmap e472: 12 total, 12 up, 11 in 2024-09-11T03:44:50.488 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:50.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:50.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:50.590 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:50.616 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:50.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:50.617 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:50.718 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:50.744 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:50.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:50.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:50.840 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:44:50.840 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:44:50.840 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:44:50.841 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:44:50.841 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:44:50.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:50.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:50.873 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:50.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:50.941 INFO:tasks.rados.rados.2.smithi175.stdout:1730: done (0 left) 2024-09-11T03:44:50.941 INFO:tasks.rados.rados.2.smithi175.stdout:1731: read oid 11 snap -1 2024-09-11T03:44:50.942 INFO:tasks.rados.rados.2.smithi175.stdout:1731: expect deleted 2024-09-11T03:44:50.942 INFO:tasks.rados.rados.2.smithi175.stdout:1732: read oid 49 snap 186 2024-09-11T03:44:50.942 INFO:tasks.rados.rados.2.smithi175.stdout:1732: expect (ObjNum 565 snap 172 seq_num 565) 2024-09-11T03:44:50.942 INFO:tasks.rados.rados.2.smithi175.stdout:1733: read oid 38 snap 170 2024-09-11T03:44:50.942 INFO:tasks.rados.rados.2.smithi175.stdout:1733: expect (ObjNum 548 snap 168 seq_num 548) 2024-09-11T03:44:50.942 INFO:tasks.rados.rados.2.smithi175.stdout:1734: read oid 34 snap -1 2024-09-11T03:44:50.942 INFO:tasks.rados.rados.2.smithi175.stdout:1734: expect (ObjNum 579 snap 178 seq_num 579) 2024-09-11T03:44:50.942 INFO:tasks.rados.rados.2.smithi175.stdout:1735: snap_create 2024-09-11T03:44:50.949 INFO:tasks.rados.rados.2.smithi175.stdout:1733: expect (ObjNum 548 snap 168 seq_num 548) 2024-09-11T03:44:50.974 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:50.995 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:50.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:50.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:51.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:51.122 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:51.122 INFO:tasks.rados.rados.2.smithi175.stdout:1732: expect (ObjNum 565 snap 172 seq_num 565) 2024-09-11T03:44:51.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:51.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:51.224 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:51.245 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:51.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:51.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:51.284 INFO:tasks.rados.rados.2.smithi175.stdout:1734: expect (ObjNum 579 snap 178 seq_num 579) 2024-09-11T03:44:51.347 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:51.368 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:51.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:51.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/19187089' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-11T03:44:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:51 smithi084 ceph-mon[27254]: osdmap e473: 12 total, 12 up, 11 in 2024-09-11T03:44:51.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/19187089' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-11T03:44:51.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:51 smithi121 ceph-mon[30729]: osdmap e473: 12 total, 12 up, 11 in 2024-09-11T03:44:51.470 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:51.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/19187089' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-11T03:44:51.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:51 smithi175 ceph-mon[30894]: osdmap e473: 12 total, 12 up, 11 in 2024-09-11T03:44:51.491 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:51.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:51.492 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:51.593 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:51.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:51.616 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:51.617 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:51.718 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:51.738 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:51.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:51.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:51.840 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:51.862 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:51.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:51.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:51.935 INFO:tasks.rados.rados.2.smithi175.stdout:1731: done (4 left) 2024-09-11T03:44:51.935 INFO:tasks.rados.rados.2.smithi175.stdout:1732: done (3 left) 2024-09-11T03:44:51.935 INFO:tasks.rados.rados.2.smithi175.stdout:1733: done (2 left) 2024-09-11T03:44:51.935 INFO:tasks.rados.rados.2.smithi175.stdout:1734: done (1 left) 2024-09-11T03:44:51.935 INFO:tasks.rados.rados.2.smithi175.stdout:1735: done (0 left) 2024-09-11T03:44:51.935 INFO:tasks.rados.rados.2.smithi175.stdout:1736: snap_remove snap 188 2024-09-11T03:44:51.964 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:51.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:51.988 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:51.988 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:52.089 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:52.113 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:52.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:52.114 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:52.160 INFO:tasks.thrashosds.thrasher:j is {'epoch': 472, 'fsid': '10fe7946-6fee-11ef-bcea-c7b262605968', 'created': '2024-09-11T03:30:30.532652+0000', 'modified': '2024-09-11T03:44:49.933276+0000', 'last_up_change': '2024-09-11T03:37:26.787638+0000', 'last_in_change': '2024-09-11T03:43:51.906016+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T03:33:42.916933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '222', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '220', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7699999809265137, 'score_stable': 1.7699999809265137, 'optimal_score': 1, 'raw_score_acting': 1.7699999809265137, 'raw_score_stable': 1.7699999809265137, 'primary_affinity_weighted': 0.8885853886604309, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-11T03:38:28.163534+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '472', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '276', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 189, 'snap_epoch': 472, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.8885853886604309, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': '9a431e48-e46b-44f9-bfa7-cb56c1484e1c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 447, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 1376667411}, {'type': 'v1', 'addr': '172.21.15.84:6803', 'nonce': 1376667411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 1376667411}, {'type': 'v1', 'addr': '172.21.15.84:6805', 'nonce': 1376667411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 1376667411}, {'type': 'v1', 'addr': '172.21.15.84:6809', 'nonce': 1376667411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 1376667411}, {'type': 'v1', 'addr': '172.21.15.84:6807', 'nonce': 1376667411}]}, 'public_addr': '172.21.15.84:6803/1376667411', 'cluster_addr': '172.21.15.84:6805/1376667411', 'heartbeat_back_addr': '172.21.15.84:6809/1376667411', 'heartbeat_front_addr': '172.21.15.84:6807/1376667411', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '62fd6e25-3508-40b4-a0f4-f7de137e3bb5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.871673583984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 465, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3138939249}, {'type': 'v1', 'addr': '172.21.15.84:6811', 'nonce': 3138939249}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 3138939249}, {'type': 'v1', 'addr': '172.21.15.84:6813', 'nonce': 3138939249}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6816', 'nonce': 3138939249}, {'type': 'v1', 'addr': '172.21.15.84:6817', 'nonce': 3138939249}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 3138939249}, {'type': 'v1', 'addr': '172.21.15.84:6815', 'nonce': 3138939249}]}, 'public_addr': '172.21.15.84:6811/3138939249', 'cluster_addr': '172.21.15.84:6813/3138939249', 'heartbeat_back_addr': '172.21.15.84:6817/3138939249', 'heartbeat_front_addr': '172.21.15.84:6815/3138939249', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ff351425-4715-49dc-be49-f27bf205d57e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 416, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6818', 'nonce': 3001811591}, {'type': 'v1', 'addr': '172.21.15.84:6819', 'nonce': 3001811591}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6820', 'nonce': 3001811591}, {'type': 'v1', 'addr': '172.21.15.84:6821', 'nonce': 3001811591}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6824', 'nonce': 3001811591}, {'type': 'v1', 'addr': '172.21.15.84:6825', 'nonce': 3001811591}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6822', 'nonce': 3001811591}, {'type': 'v1', 'addr': '172.21.15.84:6823', 'nonce': 3001811591}]}, 'public_addr': '172.21.15.84:6819/3001811591', 'cluster_addr': '172.21.15.84:6821/3001811591', 'heartbeat_back_addr': '172.21.15.84:6825/3001811591', 'heartbeat_front_addr': '172.21.15.84:6823/3001811591', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2518a93e-c612-4731-91ce-5685c25ffa9f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 332, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6826', 'nonce': 302512129}, {'type': 'v1', 'addr': '172.21.15.84:6827', 'nonce': 302512129}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6828', 'nonce': 302512129}, {'type': 'v1', 'addr': '172.21.15.84:6829', 'nonce': 302512129}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6832', 'nonce': 302512129}, {'type': 'v1', 'addr': '172.21.15.84:6833', 'nonce': 302512129}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6830', 'nonce': 302512129}, {'type': 'v1', 'addr': '172.21.15.84:6831', 'nonce': 302512129}]}, 'public_addr': '172.21.15.84:6827/302512129', 'cluster_addr': '172.21.15.84:6829/302512129', 'heartbeat_back_addr': '172.21.15.84:6833/302512129', 'heartbeat_front_addr': '172.21.15.84:6831/302512129', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd12f3c2d-95cc-4276-a7e1-bee982d58948', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 398, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6800', 'nonce': 3873214513}, {'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 3873214513}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6802', 'nonce': 3873214513}, {'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 3873214513}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6806', 'nonce': 3873214513}, {'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 3873214513}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6804', 'nonce': 3873214513}, {'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 3873214513}]}, 'public_addr': '172.21.15.121:6801/3873214513', 'cluster_addr': '172.21.15.121:6803/3873214513', 'heartbeat_back_addr': '172.21.15.121:6807/3873214513', 'heartbeat_front_addr': '172.21.15.121:6805/3873214513', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '32164dd5-7e8f-4925-8526-bbb1e33e3364', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 414, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6808', 'nonce': 918612194}, {'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 918612194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6810', 'nonce': 918612194}, {'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 918612194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6814', 'nonce': 918612194}, {'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 918612194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6812', 'nonce': 918612194}, {'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 918612194}]}, 'public_addr': '172.21.15.121:6809/918612194', 'cluster_addr': '172.21.15.121:6811/918612194', 'heartbeat_back_addr': '172.21.15.121:6815/918612194', 'heartbeat_front_addr': '172.21.15.121:6813/918612194', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '3f107a88-803f-4cd8-804a-241310c282ea', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6816', 'nonce': 3384813363}, {'type': 'v1', 'addr': '172.21.15.121:6817', 'nonce': 3384813363}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6818', 'nonce': 3384813363}, {'type': 'v1', 'addr': '172.21.15.121:6819', 'nonce': 3384813363}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6822', 'nonce': 3384813363}, {'type': 'v1', 'addr': '172.21.15.121:6823', 'nonce': 3384813363}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6820', 'nonce': 3384813363}, {'type': 'v1', 'addr': '172.21.15.121:6821', 'nonce': 3384813363}]}, 'public_addr': '172.21.15.121:6817/3384813363', 'cluster_addr': '172.21.15.121:6819/3384813363', 'heartbeat_back_addr': '172.21.15.121:6823/3384813363', 'heartbeat_front_addr': '172.21.15.121:6821/3384813363', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dfee7f95-1fb9-4493-9907-cb10d1837bdb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.791351318359375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 465, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6824', 'nonce': 449222054}, {'type': 'v1', 'addr': '172.21.15.121:6825', 'nonce': 449222054}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6826', 'nonce': 449222054}, {'type': 'v1', 'addr': '172.21.15.121:6827', 'nonce': 449222054}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6830', 'nonce': 449222054}, {'type': 'v1', 'addr': '172.21.15.121:6831', 'nonce': 449222054}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6828', 'nonce': 449222054}, {'type': 'v1', 'addr': '172.21.15.121:6829', 'nonce': 449222054}]}, 'public_addr': '172.21.15.121:6825/449222054', 'cluster_addr': '172.21.15.121:6827/449222054', 'heartbeat_back_addr': '172.21.15.121:6831/449222054', 'heartbeat_front_addr': '172.21.15.121:6829/449222054', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 414, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6800', 'nonce': 2660128384}, {'type': 'v1', 'addr': '172.21.15.175:6801', 'nonce': 2660128384}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6802', 'nonce': 2660128384}, {'type': 'v1', 'addr': '172.21.15.175:6803', 'nonce': 2660128384}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6806', 'nonce': 2660128384}, {'type': 'v1', 'addr': '172.21.15.175:6807', 'nonce': 2660128384}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6804', 'nonce': 2660128384}, {'type': 'v1', 'addr': '172.21.15.175:6805', 'nonce': 2660128384}]}, 'public_addr': '172.21.15.175:6801/2660128384', 'cluster_addr': '172.21.15.175:6803/2660128384', 'heartbeat_back_addr': '172.21.15.175:6807/2660128384', 'heartbeat_front_addr': '172.21.15.175:6805/2660128384', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '50162e42-0443-41d1-8709-a0db61c087da', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 312, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6808', 'nonce': 1270991649}, {'type': 'v1', 'addr': '172.21.15.175:6809', 'nonce': 1270991649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6810', 'nonce': 1270991649}, {'type': 'v1', 'addr': '172.21.15.175:6811', 'nonce': 1270991649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6814', 'nonce': 1270991649}, {'type': 'v1', 'addr': '172.21.15.175:6815', 'nonce': 1270991649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6812', 'nonce': 1270991649}, {'type': 'v1', 'addr': '172.21.15.175:6813', 'nonce': 1270991649}]}, 'public_addr': '172.21.15.175:6809/1270991649', 'cluster_addr': '172.21.15.175:6811/1270991649', 'heartbeat_back_addr': '172.21.15.175:6815/1270991649', 'heartbeat_front_addr': '172.21.15.175:6813/1270991649', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '4b9426e1-bac8-47d3-ad58-54b89e8d1eb1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6816', 'nonce': 3877631021}, {'type': 'v1', 'addr': '172.21.15.175:6817', 'nonce': 3877631021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6818', 'nonce': 3877631021}, {'type': 'v1', 'addr': '172.21.15.175:6819', 'nonce': 3877631021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6822', 'nonce': 3877631021}, {'type': 'v1', 'addr': '172.21.15.175:6823', 'nonce': 3877631021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6820', 'nonce': 3877631021}, {'type': 'v1', 'addr': '172.21.15.175:6821', 'nonce': 3877631021}]}, 'public_addr': '172.21.15.175:6817/3877631021', 'cluster_addr': '172.21.15.175:6819/3877631021', 'heartbeat_back_addr': '172.21.15.175:6823/3877631021', 'heartbeat_front_addr': '172.21.15.175:6821/3877631021', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'cd1a28a1-413c-4274-9876-e1bdb7703be6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 449, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6824', 'nonce': 1150051236}, {'type': 'v1', 'addr': '172.21.15.175:6825', 'nonce': 1150051236}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6826', 'nonce': 1150051236}, {'type': 'v1', 'addr': '172.21.15.175:6827', 'nonce': 1150051236}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6830', 'nonce': 1150051236}, {'type': 'v1', 'addr': '172.21.15.175:6831', 'nonce': 1150051236}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6828', 'nonce': 1150051236}, {'type': 'v1', 'addr': '172.21.15.175:6829', 'nonce': 1150051236}]}, 'public_addr': '172.21.15.175:6825/1150051236', 'cluster_addr': '172.21.15.175:6827/1150051236', 'heartbeat_back_addr': '172.21.15.175:6831/1150051236', 'heartbeat_front_addr': '172.21.15.175:6829/1150051236', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:43:37.917862+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:44:19.071328+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:44:40.814709+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:43:50.817861+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:44:40.596330+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:44:41.447558+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-11T03:43:31.134086+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:44:12.815905+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:44:11.646575+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:44:05.536067+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:44:17.122767+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T03:43:45.212841+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '1.4', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '2.2', 'mappings': [{'from': 9, 'to': 11}]}, {'pgid': '2.d', 'mappings': [{'from': 9, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/108136515': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:0/3411515187': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:6801/3365920551': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:6800/3365920551': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:0/3885577441': '2024-09-12T03:31:03.077186+0000', '172.21.15.84:0/22039254': '2024-09-12T03:31:03.077186+0000', '172.21.15.84:6801/3496645948': '2024-09-12T03:31:03.077186+0000', '172.21.15.84:0/4206337830': '2024-09-12T03:31:26.598965+0000', '172.21.15.84:6800/3496645948': '2024-09-12T03:31:03.077186+0000', '172.21.15.84:0/1989775875': '2024-09-12T03:31:03.077186+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 174, 'length': 2}, {'begin': 181, 'length': 4}, {'begin': 187, 'length': 1}]}], 'new_removed_snaps': [{'pool': 2, 'snaps': [{'begin': 187, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T03:44:52.160 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:44:52.215 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:52.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:52.239 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:52.239 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:52.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:52 smithi084 ceph-mon[27254]: pgmap v864: 57 pgs: 57 active+clean; 190 MiB data, 798 MiB used, 983 GiB / 983 GiB avail; 5.9 MiB/s rd, 0 B/s wr, 5 op/s 2024-09-11T03:44:52.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:52 smithi084 ceph-mon[27254]: osdmap e474: 12 total, 12 up, 11 in 2024-09-11T03:44:52.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:52.365 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:52.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:52.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:52 smithi121 ceph-mon[30729]: pgmap v864: 57 pgs: 57 active+clean; 190 MiB data, 798 MiB used, 983 GiB / 983 GiB avail; 5.9 MiB/s rd, 0 B/s wr, 5 op/s 2024-09-11T03:44:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:52 smithi121 ceph-mon[30729]: osdmap e474: 12 total, 12 up, 11 in 2024-09-11T03:44:52.467 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:52 smithi175 ceph-mon[30894]: pgmap v864: 57 pgs: 57 active+clean; 190 MiB data, 798 MiB used, 983 GiB / 983 GiB avail; 5.9 MiB/s rd, 0 B/s wr, 5 op/s 2024-09-11T03:44:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:52 smithi175 ceph-mon[30894]: osdmap e474: 12 total, 12 up, 11 in 2024-09-11T03:44:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:52.493 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:52.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:52.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:52.620 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:52.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:52.705 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T03:44:52.710 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:52.722 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:52.743 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:52.747 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:52.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:52.748 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:52.849 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:52.875 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:52.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:52.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:52.939 INFO:tasks.rados.rados.2.smithi175.stdout:1736: done (0 left) 2024-09-11T03:44:52.939 INFO:tasks.rados.rados.2.smithi175.stdout:1737: rollback oid 33 current snap is 189 2024-09-11T03:44:52.939 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 33 to 175 2024-09-11T03:44:52.939 INFO:tasks.rados.rados.2.smithi175.stdout:1738: read oid 50 snap -1 2024-09-11T03:44:52.939 INFO:tasks.rados.rados.2.smithi175.stdout:1738: expect deleted 2024-09-11T03:44:52.939 INFO:tasks.rados.rados.2.smithi175.stdout:1739: read oid 34 snap -1 2024-09-11T03:44:52.940 INFO:tasks.rados.rados.2.smithi175.stdout:1739: expect (ObjNum 579 snap 178 seq_num 579) 2024-09-11T03:44:52.940 INFO:tasks.rados.rados.2.smithi175.stdout:1740: read oid 31 snap -1 2024-09-11T03:44:52.940 INFO:tasks.rados.rados.2.smithi175.stdout:1740: expect (ObjNum 495 snap 143 seq_num 495) 2024-09-11T03:44:52.940 INFO:tasks.rados.rados.2.smithi175.stdout:1741: rollback oid 29 current snap is 189 2024-09-11T03:44:52.940 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 29 to 183 2024-09-11T03:44:52.940 INFO:tasks.rados.rados.2.smithi175.stdout:1742: read oid 14 snap -1 2024-09-11T03:44:52.940 INFO:tasks.rados.rados.2.smithi175.stdout:1742: expect (ObjNum 587 snap 183 seq_num 587) 2024-09-11T03:44:52.940 INFO:tasks.rados.rados.2.smithi175.stdout:1743: delete oid 10 current snap is 189 2024-09-11T03:44:52.942 INFO:tasks.rados.rados.2.smithi175.stdout:1741: finishing rollback tid 1 to smithi17557870-29 2024-09-11T03:44:52.942 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T03:44:52.943 INFO:tasks.rados.rados.2.smithi175.stdout:1741: done (6 left) 2024-09-11T03:44:52.943 INFO:tasks.rados.rados.2.smithi175.stdout:1743: done (5 left) 2024-09-11T03:44:52.943 INFO:tasks.rados.rados.2.smithi175.stdout:1744: snap_remove snap 187 2024-09-11T03:44:52.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:53.003 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:53.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:53.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:53.105 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:53.128 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:53.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:53.129 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:53.230 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:53.253 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:53.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:53.254 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:53.355 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:53.381 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:53.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:53.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:53.508 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:53.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:53.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:53.632 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:53.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:53.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:53.641 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:53.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:53.759 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:53.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:53.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:53.766 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:44:53.832 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:44:53.833 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:44:53.861 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:53.888 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:53.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:53.963 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:44:53.969 INFO:tasks.rados.rados.2.smithi175.stdout:1744: done (5 left) 2024-09-11T03:44:53.969 INFO:tasks.rados.rados.2.smithi175.stdout:1745: snap_remove snap 170 2024-09-11T03:44:53.989 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:54.010 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:54.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:54.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:54.106 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:53 smithi084 ceph-mon[27254]: pgmap v867: 57 pgs: 57 active+clean; 194 MiB data, 805 MiB used, 983 GiB / 983 GiB avail; 8.0 MiB/s rd, 1.5 MiB/s wr, 7 op/s 2024-09-11T03:44:54.107 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:53 smithi084 ceph-mon[27254]: osdmap e475: 12 total, 12 up, 11 in 2024-09-11T03:44:54.107 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:53 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3033361904' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:44:54.112 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:54.138 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:54.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:53 smithi175 ceph-mon[30894]: pgmap v867: 57 pgs: 57 active+clean; 194 MiB data, 805 MiB used, 983 GiB / 983 GiB avail; 8.0 MiB/s rd, 1.5 MiB/s wr, 7 op/s 2024-09-11T03:44:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:53 smithi175 ceph-mon[30894]: osdmap e475: 12 total, 12 up, 11 in 2024-09-11T03:44:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:53 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3033361904' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:44:54.240 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:54.266 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:54.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:54.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:54.368 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:54.390 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:54.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:54.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:53 smithi121 ceph-mon[30729]: pgmap v867: 57 pgs: 57 active+clean; 194 MiB data, 805 MiB used, 983 GiB / 983 GiB avail; 8.0 MiB/s rd, 1.5 MiB/s wr, 7 op/s 2024-09-11T03:44:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:53 smithi121 ceph-mon[30729]: osdmap e475: 12 total, 12 up, 11 in 2024-09-11T03:44:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:53 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3033361904' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:44:54.491 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:54.513 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:54.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:54.614 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:54.636 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:54.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:54.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:54.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:54.760 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:54.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:54.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:54.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:54.888 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:54.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:54.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:54.977 INFO:tasks.rados.rados.2.smithi175.stdout:1745: done (5 left) 2024-09-11T03:44:54.977 INFO:tasks.rados.rados.2.smithi175.stdout:1746: read oid 6 snap -1 2024-09-11T03:44:54.977 INFO:tasks.rados.rados.2.smithi175.stdout:1746: expect (ObjNum 528 snap 156 seq_num 528) 2024-09-11T03:44:54.977 INFO:tasks.rados.rados.2.smithi175.stdout:1747: rollback oid 3 current snap is 189 2024-09-11T03:44:54.977 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 3 to 175 2024-09-11T03:44:54.977 INFO:tasks.rados.rados.2.smithi175.stdout:1740: expect (ObjNum 495 snap 143 seq_num 495) 2024-09-11T03:44:54.990 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:44:55.011 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:55.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:44:55.053 INFO:tasks.rados.rados.2.smithi175.stdout:1739: expect (ObjNum 579 snap 178 seq_num 579) 2024-09-11T03:44:55.113 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:44:55.134 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:55.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:55.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:44:55.164 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:44:55.164 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:44:55.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:44:55.164 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:44:55.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.164 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:877 2.e 2:77dadb97:::smithi17557870-30:head [copy-from ver 12] snapc 1e=[1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-11T03:44:55.164 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:31.988317+0000", 2024-09-11T03:44:55.164 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 323.17472067599999, 2024-09-11T03:44:55.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011419110999999999, 2024-09-11T03:44:55.165 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.165 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.165 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:55.165 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.165 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.165 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.165 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 877 2024-09-11T03:44:55.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.165 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.988317+0000", 2024-09-11T03:44:55.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.988317+0000", 2024-09-11T03:44:55.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.988319+0000", 2024-09-11T03:44:55.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.866e-06 2024-09-11T03:44:55.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.988329+0000", 2024-09-11T03:44:55.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0132000000000001e-05 2024-09-11T03:44:55.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.988349+0000", 2024-09-11T03:44:55.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9400999999999999e-05 2024-09-11T03:44:55.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.988375+0000", 2024-09-11T03:44:55.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6156000000000002e-05 2024-09-11T03:44:55.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.988447+0000", 2024-09-11T03:44:55.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2559999999999996e-05 2024-09-11T03:44:55.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.988554+0000", 2024-09-11T03:44:55.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000106699 2024-09-11T03:44:55.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:44:55.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.994152+0000", 2024-09-11T03:44:55.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055978130000000001 2024-09-11T03:44:55.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:44:55.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.997930+0000", 2024-09-11T03:44:55.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037786780000000002 2024-09-11T03:44:55.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.998218+0000", 2024-09-11T03:44:55.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000287335 2024-09-11T03:44:55.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.999572+0000", 2024-09-11T03:44:55.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001354657 2024-09-11T03:44:55.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:55.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.999667+0000", 2024-09-11T03:44:55.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4435e-05 2024-09-11T03:44:55.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.999736+0000", 2024-09-11T03:44:55.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9379000000000001e-05 2024-09-11T03:44:55.171 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.171 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.171 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.172 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1068 2.3 e175/95 2:c3768452:::smithi17557870-11:head v 175'55, mlcod=165'51)", 2024-09-11T03:44:55.172 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:51.874076+0000", 2024-09-11T03:44:55.172 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 303.28896197699999, 2024-09-11T03:44:55.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033166769999999998, 2024-09-11T03:44:55.172 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.172 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.172 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:55.172 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.874076+0000", 2024-09-11T03:44:55.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.874076+0000", 2024-09-11T03:44:55.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.874076+0000", 2024-09-11T03:44:55.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.66e-07 2024-09-11T03:44:55.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.906033+0000", 2024-09-11T03:44:55.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.031956971000000001 2024-09-11T03:44:55.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.906049+0000", 2024-09-11T03:44:55.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6503000000000001e-05 2024-09-11T03:44:55.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.906058+0000", 2024-09-11T03:44:55.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3860000000000007e-06 2024-09-11T03:44:55.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.906111+0000", 2024-09-11T03:44:55.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3053999999999997e-05 2024-09-11T03:44:55.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.906129+0000", 2024-09-11T03:44:55.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8071999999999999e-05 2024-09-11T03:44:55.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:55.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.907203+0000", 2024-09-11T03:44:55.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010736960000000001 2024-09-11T03:44:55.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.907243+0000", 2024-09-11T03:44:55.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9922000000000002e-05 2024-09-11T03:44:55.176 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.176 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.176 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.176 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1159 2.e 2:750de3b7:::smithi17557870-8:head [copy-from ver 77] snapc 2e=[2e,2c,2b,28,26] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:44:55.176 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:01.831728+0000", 2024-09-11T03:44:55.177 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.33130982599999, 2024-09-11T03:44:55.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016271035999999999, 2024-09-11T03:44:55.177 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.177 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.177 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:55.177 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.177 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.177 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.177 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1159 2024-09-11T03:44:55.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.177 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831728+0000", 2024-09-11T03:44:55.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831728+0000", 2024-09-11T03:44:55.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831730+0000", 2024-09-11T03:44:55.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.283e-06 2024-09-11T03:44:55.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831741+0000", 2024-09-11T03:44:55.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0475000000000001e-05 2024-09-11T03:44:55.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831760+0000", 2024-09-11T03:44:55.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9034000000000001e-05 2024-09-11T03:44:55.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831776+0000", 2024-09-11T03:44:55.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6106000000000001e-05 2024-09-11T03:44:55.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831836+0000", 2024-09-11T03:44:55.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0420000000000001e-05 2024-09-11T03:44:55.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:44:55.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831841+0000", 2024-09-11T03:44:55.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8770000000000001e-06 2024-09-11T03:44:55.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.833455+0000", 2024-09-11T03:44:55.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016140639999999999 2024-09-11T03:44:55.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.833568+0000", 2024-09-11T03:44:55.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011259700000000001 2024-09-11T03:44:55.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:44:55.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.840230+0000", 2024-09-11T03:44:55.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066619189999999997 2024-09-11T03:44:55.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:44:55.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.845836+0000", 2024-09-11T03:44:55.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056066459999999999 2024-09-11T03:44:55.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.846094+0000", 2024-09-11T03:44:55.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025752300000000001 2024-09-11T03:44:55.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.847897+0000", 2024-09-11T03:44:55.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018033529999999999 2024-09-11T03:44:55.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:55.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.847954+0000", 2024-09-11T03:44:55.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6436e-05 2024-09-11T03:44:55.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.847999+0000", 2024-09-11T03:44:55.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5303000000000003e-05 2024-09-11T03:44:55.185 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.185 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.186 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.186 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1160 2.e 2:750de3b7:::smithi17557870-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:44:55.186 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:01.831798+0000", 2024-09-11T03:44:55.186 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.331240106, 2024-09-11T03:44:55.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012910331000000001, 2024-09-11T03:44:55.186 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.186 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.186 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:44:55.186 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.186 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.187 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.187 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1160 2024-09-11T03:44:55.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.187 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831798+0000", 2024-09-11T03:44:55.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831798+0000", 2024-09-11T03:44:55.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831798+0000", 2024-09-11T03:44:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0699999999999998e-07 2024-09-11T03:44:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831800+0000", 2024-09-11T03:44:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8679999999999999e-06 2024-09-11T03:44:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831806+0000", 2024-09-11T03:44:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6339999999999998e-06 2024-09-11T03:44:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831809+0000", 2024-09-11T03:44:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8650000000000003e-06 2024-09-11T03:44:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831896+0000", 2024-09-11T03:44:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6700000000000007e-05 2024-09-11T03:44:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831904+0000", 2024-09-11T03:44:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8099999999999998e-06 2024-09-11T03:44:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.833745+0000", 2024-09-11T03:44:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018409979999999999 2024-09-11T03:44:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.833795+0000", 2024-09-11T03:44:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0413e-05 2024-09-11T03:44:55.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.844574+0000", 2024-09-11T03:44:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010778993000000001 2024-09-11T03:44:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.844637+0000", 2024-09-11T03:44:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3002000000000004e-05 2024-09-11T03:44:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.844708+0000", 2024-09-11T03:44:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0641000000000003e-05 2024-09-11T03:44:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1285 2.1 2:8638b65e:::smithi17557870-5:head [copy-from ver 153] snapc 33=[32,30,2c,2b] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:44:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:12.836052+0000", 2024-09-11T03:44:55.192 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 282.326985509, 2024-09-11T03:44:55.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020441596999999999, 2024-09-11T03:44:55.193 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.193 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.193 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:55.193 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.193 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1285 2024-09-11T03:44:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836052+0000", 2024-09-11T03:44:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836052+0000", 2024-09-11T03:44:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836053+0000", 2024-09-11T03:44:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.34e-07 2024-09-11T03:44:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836055+0000", 2024-09-11T03:44:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7549999999999999e-06 2024-09-11T03:44:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836061+0000", 2024-09-11T03:44:55.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6579999999999999e-06 2024-09-11T03:44:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836071+0000", 2024-09-11T03:44:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0141e-05 2024-09-11T03:44:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836118+0000", 2024-09-11T03:44:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7200999999999998e-05 2024-09-11T03:44:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:44:55.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836124+0000", 2024-09-11T03:44:55.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8150000000000002e-06 2024-09-11T03:44:55.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.837592+0000", 2024-09-11T03:44:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014672509999999999 2024-09-11T03:44:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.837642+0000", 2024-09-11T03:44:55.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0145999999999997e-05 2024-09-11T03:44:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,3", 2024-09-11T03:44:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.849946+0000", 2024-09-11T03:44:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012303869 2024-09-11T03:44:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:44:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.852055+0000", 2024-09-11T03:44:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002109835 2024-09-11T03:44:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856314+0000", 2024-09-11T03:44:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042587700000000003 2024-09-11T03:44:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856368+0000", 2024-09-11T03:44:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3538999999999997e-05 2024-09-11T03:44:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856435+0000", 2024-09-11T03:44:55.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7470000000000003e-05 2024-09-11T03:44:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856494+0000", 2024-09-11T03:44:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8712999999999998e-05 2024-09-11T03:44:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.204 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1286 2.1 2:8638b65e:::smithi17557870-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:44:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:12.836083+0000", 2024-09-11T03:44:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 282.32695463900001, 2024-09-11T03:44:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015739692, 2024-09-11T03:44:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:44:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1286 2024-09-11T03:44:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836083+0000", 2024-09-11T03:44:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836083+0000", 2024-09-11T03:44:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836084+0000", 2024-09-11T03:44:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2500000000000001e-07 2024-09-11T03:44:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836085+0000", 2024-09-11T03:44:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8020000000000001e-06 2024-09-11T03:44:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836092+0000", 2024-09-11T03:44:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3169999999999998e-06 2024-09-11T03:44:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836096+0000", 2024-09-11T03:44:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.634e-06 2024-09-11T03:44:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836147+0000", 2024-09-11T03:44:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0374999999999998e-05 2024-09-11T03:44:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836150+0000", 2024-09-11T03:44:55.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.072e-06 2024-09-11T03:44:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.837748+0000", 2024-09-11T03:44:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001597867 2024-09-11T03:44:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.837783+0000", 2024-09-11T03:44:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5395000000000002e-05 2024-09-11T03:44:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.851559+0000", 2024-09-11T03:44:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013776343999999999 2024-09-11T03:44:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.851665+0000", 2024-09-11T03:44:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010521199999999999 2024-09-11T03:44:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.851823+0000", 2024-09-11T03:44:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015824900000000001 2024-09-11T03:44:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.212 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1434 2.e 2:7efbbea8:::smithi17557870-24:head [copy-from ver 75] snapc 36=[35,2b] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-11T03:44:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:20.864420+0000", 2024-09-11T03:44:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 274.298618288, 2024-09-11T03:44:55.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012389887, 2024-09-11T03:44:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.214 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1434 2024-09-11T03:44:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864420+0000", 2024-09-11T03:44:55.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864420+0000", 2024-09-11T03:44:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864421+0000", 2024-09-11T03:44:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8369999999999999e-06 2024-09-11T03:44:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864430+0000", 2024-09-11T03:44:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.132e-06 2024-09-11T03:44:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864447+0000", 2024-09-11T03:44:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6305999999999999e-05 2024-09-11T03:44:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864454+0000", 2024-09-11T03:44:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1230000000000002e-06 2024-09-11T03:44:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864527+0000", 2024-09-11T03:44:55.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2606000000000002e-05 2024-09-11T03:44:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:44:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864536+0000", 2024-09-11T03:44:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7790000000000007e-06 2024-09-11T03:44:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.865844+0000", 2024-09-11T03:44:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307473 2024-09-11T03:44:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.865934+0000", 2024-09-11T03:44:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9785999999999997e-05 2024-09-11T03:44:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:44:55.224 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:54 smithi084 ceph-mon[27254]: from='client.16041 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:44:55.224 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:54 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:55.225 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:54 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3033361904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:44:55.225 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:54 smithi084 ceph-mon[27254]: osdmap e476: 12 total, 12 up, 11 in 2024-09-11T03:44:55.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.869567+0000", 2024-09-11T03:44:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0036332859999999999 2024-09-11T03:44:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:44:55.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.874154+0000", 2024-09-11T03:44:55.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.004586957 2024-09-11T03:44:55.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.875683+0000", 2024-09-11T03:44:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015289920000000001 2024-09-11T03:44:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.876715+0000", 2024-09-11T03:44:55.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010321620000000001 2024-09-11T03:44:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.876752+0000", 2024-09-11T03:44:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6820000000000003e-05 2024-09-11T03:44:55.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.876809+0000", 2024-09-11T03:44:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7627999999999998e-05 2024-09-11T03:44:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.231 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1589 2.7 2:e77ce595:::smithi17557870-31:head [copy-from ver 148] snapc 3f=[3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-09-11T03:44:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:35.104207+0000", 2024-09-11T03:44:55.232 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 260.05883063499999, 2024-09-11T03:44:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021309244000000001, 2024-09-11T03:44:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.233 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1589 2024-09-11T03:44:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104207+0000", 2024-09-11T03:44:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104207+0000", 2024-09-11T03:44:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104208+0000", 2024-09-11T03:44:55.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2780000000000001e-06 2024-09-11T03:44:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104213+0000", 2024-09-11T03:44:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5979999999999999e-06 2024-09-11T03:44:55.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104225+0000", 2024-09-11T03:44:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1819e-05 2024-09-11T03:44:55.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104249+0000", 2024-09-11T03:44:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4117999999999998e-05 2024-09-11T03:44:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104305+0000", 2024-09-11T03:44:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6069999999999997e-05 2024-09-11T03:44:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104375+0000", 2024-09-11T03:44:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0156999999999999e-05 2024-09-11T03:44:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:44:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116294+0000", 2024-09-11T03:44:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011918676 2024-09-11T03:44:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:44:55.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.123116+0000", 2024-09-11T03:44:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068219259999999999 2024-09-11T03:44:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.124470+0000", 2024-09-11T03:44:55.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001354079 2024-09-11T03:44:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125455+0000", 2024-09-11T03:44:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00098535300000000001 2024-09-11T03:44:55.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125487+0000", 2024-09-11T03:44:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2200999999999999e-05 2024-09-11T03:44:55.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125516+0000", 2024-09-11T03:44:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8969e-05 2024-09-11T03:44:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.246 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1590 2.7 2:e77ce595:::smithi17557870-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e218)", 2024-09-11T03:44:55.247 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:35.104267+0000", 2024-09-11T03:44:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 260.05877105500002, 2024-09-11T03:44:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017183714999999999, 2024-09-11T03:44:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:44:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1590 2024-09-11T03:44:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104267+0000", 2024-09-11T03:44:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104267+0000", 2024-09-11T03:44:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104267+0000", 2024-09-11T03:44:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.77e-07 2024-09-11T03:44:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104269+0000", 2024-09-11T03:44:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8199999999999999e-06 2024-09-11T03:44:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104274+0000", 2024-09-11T03:44:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2839999999999997e-06 2024-09-11T03:44:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104277+0000", 2024-09-11T03:44:55.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0149999999999999e-06 2024-09-11T03:44:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104478+0000", 2024-09-11T03:44:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200256 2024-09-11T03:44:55.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104511+0000", 2024-09-11T03:44:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3155999999999999e-05 2024-09-11T03:44:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.121250+0000", 2024-09-11T03:44:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016738843999999999 2024-09-11T03:44:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.121352+0000", 2024-09-11T03:44:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102384 2024-09-11T03:44:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.121450+0000", 2024-09-11T03:44:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8579000000000006e-05 2024-09-11T03:44:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1634 2.7 2:e23d1762:::smithi17557870-21:head [copy-from ver 214] snapc 40=[40,3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:44:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:36.206869+0000", 2024-09-11T03:44:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 258.956168343, 2024-09-11T03:44:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017521808999999999, 2024-09-11T03:44:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.260 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1634 2024-09-11T03:44:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206869+0000", 2024-09-11T03:44:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206869+0000", 2024-09-11T03:44:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206872+0000", 2024-09-11T03:44:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.926e-06 2024-09-11T03:44:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206898+0000", 2024-09-11T03:44:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5987e-05 2024-09-11T03:44:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206986+0000", 2024-09-11T03:44:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7714999999999996e-05 2024-09-11T03:44:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207030+0000", 2024-09-11T03:44:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3455999999999999e-05 2024-09-11T03:44:55.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207171+0000", 2024-09-11T03:44:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014184700000000001 2024-09-11T03:44:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207283+0000", 2024-09-11T03:44:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000111976 2024-09-11T03:44:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:44:55.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.216671+0000", 2024-09-11T03:44:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0093877200000000004 2024-09-11T03:44:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.223379+0000", 2024-09-11T03:44:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067082289999999996 2024-09-11T03:44:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:44:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224223+0000", 2024-09-11T03:44:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000843771 2024-09-11T03:44:55.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224293+0000", 2024-09-11T03:44:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0339000000000005e-05 2024-09-11T03:44:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224347+0000", 2024-09-11T03:44:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3288999999999998e-05 2024-09-11T03:44:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224391+0000", 2024-09-11T03:44:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4554000000000001e-05 2024-09-11T03:44:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1635 2.7 2:e23d1762:::smithi17557870-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:44:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:36.207063+0000", 2024-09-11T03:44:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 258.95597456600001, 2024-09-11T03:44:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013462717000000001, 2024-09-11T03:44:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:44:55.267 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1635 2024-09-11T03:44:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.268 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207063+0000", 2024-09-11T03:44:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207063+0000", 2024-09-11T03:44:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207064+0000", 2024-09-11T03:44:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.043e-06 2024-09-11T03:44:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207071+0000", 2024-09-11T03:44:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8929999999999999e-06 2024-09-11T03:44:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207086+0000", 2024-09-11T03:44:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4596999999999999e-05 2024-09-11T03:44:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207095+0000", 2024-09-11T03:44:55.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9930000000000001e-06 2024-09-11T03:44:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207386+0000", 2024-09-11T03:44:55.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029111399999999998 2024-09-11T03:44:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207423+0000", 2024-09-11T03:44:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6825000000000001e-05 2024-09-11T03:44:55.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.220376+0000", 2024-09-11T03:44:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012953549 2024-09-11T03:44:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.220446+0000", 2024-09-11T03:44:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9578000000000004e-05 2024-09-11T03:44:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.220526+0000", 2024-09-11T03:44:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0124999999999996e-05 2024-09-11T03:44:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.278 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1866 2.e 2:7b0c3397:::smithi17557870-14:head [copy-from ver 238] snapc 4c=[4c,4b,49,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:44:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:44:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 233.312904729, 2024-09-11T03:44:55.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012778199000000001, 2024-09-11T03:44:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.280 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1866 2024-09-11T03:44:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:44:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:44:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.281 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850135+0000", 2024-09-11T03:44:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5850000000000001e-06 2024-09-11T03:44:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850142+0000", 2024-09-11T03:44:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2010000000000002e-06 2024-09-11T03:44:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850160+0000", 2024-09-11T03:44:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7858000000000001e-05 2024-09-11T03:44:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850170+0000", 2024-09-11T03:44:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9219999999999992e-06 2024-09-11T03:44:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850235+0000", 2024-09-11T03:44:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4849000000000002e-05 2024-09-11T03:44:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:44:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850246+0000", 2024-09-11T03:44:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1938e-05 2024-09-11T03:44:55.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.851390+0000", 2024-09-11T03:44:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011435219999999999 2024-09-11T03:44:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.851491+0000", 2024-09-11T03:44:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000100868 2024-09-11T03:44:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:44:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.857380+0000", 2024-09-11T03:44:55.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058892839999999998 2024-09-11T03:44:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:44:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.861564+0000", 2024-09-11T03:44:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041843369999999998 2024-09-11T03:44:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.861883+0000", 2024-09-11T03:44:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031904399999999999 2024-09-11T03:44:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.862846+0000", 2024-09-11T03:44:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096234000000000001 2024-09-11T03:44:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.862883+0000", 2024-09-11T03:44:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7132000000000003e-05 2024-09-11T03:44:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.862911+0000", 2024-09-11T03:44:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8319000000000001e-05 2024-09-11T03:44:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.286 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1867 2.e 2:7b0c3397:::smithi17557870-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:44:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:01.850191+0000", 2024-09-11T03:44:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 233.312847164, 2024-09-11T03:44:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010604511, 2024-09-11T03:44:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:44:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.287 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1867 2024-09-11T03:44:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850191+0000", 2024-09-11T03:44:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850191+0000", 2024-09-11T03:44:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850191+0000", 2024-09-11T03:44:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0399999999999996e-07 2024-09-11T03:44:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850194+0000", 2024-09-11T03:44:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6850000000000001e-06 2024-09-11T03:44:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850202+0000", 2024-09-11T03:44:55.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1489999999999994e-06 2024-09-11T03:44:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850208+0000", 2024-09-11T03:44:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2140000000000001e-06 2024-09-11T03:44:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850303+0000", 2024-09-11T03:44:55.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4745000000000002e-05 2024-09-11T03:44:55.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850309+0000", 2024-09-11T03:44:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0719999999999996e-06 2024-09-11T03:44:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.851644+0000", 2024-09-11T03:44:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001335319 2024-09-11T03:44:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.851686+0000", 2024-09-11T03:44:55.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1387000000000002e-05 2024-09-11T03:44:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.860644+0000", 2024-09-11T03:44:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0089587669999999994 2024-09-11T03:44:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.860709+0000", 2024-09-11T03:44:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4651000000000002e-05 2024-09-11T03:44:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.860795+0000", 2024-09-11T03:44:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6018000000000002e-05 2024-09-11T03:44:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1895 2.e 2:7efbbea8:::smithi17557870-24:head [copy-from ver 160] snapc 4e=[4e,4d,49,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-09-11T03:44:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:05.924255+0000", 2024-09-11T03:44:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 229.23878273299999, 2024-09-11T03:44:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016112593000000001, 2024-09-11T03:44:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.294 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1895 2024-09-11T03:44:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924255+0000", 2024-09-11T03:44:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924255+0000", 2024-09-11T03:44:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924257+0000", 2024-09-11T03:44:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.742e-06 2024-09-11T03:44:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924264+0000", 2024-09-11T03:44:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6090000000000003e-06 2024-09-11T03:44:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924281+0000", 2024-09-11T03:44:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6157000000000001e-05 2024-09-11T03:44:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924290+0000", 2024-09-11T03:44:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0499999999999997e-06 2024-09-11T03:44:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924352+0000", 2024-09-11T03:44:55.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1968e-05 2024-09-11T03:44:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:44:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924358+0000", 2024-09-11T03:44:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1380000000000004e-06 2024-09-11T03:44:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.925553+0000", 2024-09-11T03:44:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001194802 2024-09-11T03:44:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.925622+0000", 2024-09-11T03:44:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9598999999999996e-05 2024-09-11T03:44:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:44:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.931889+0000", 2024-09-11T03:44:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062668419999999999 2024-09-11T03:44:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:44:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.937406+0000", 2024-09-11T03:44:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055165659999999997 2024-09-11T03:44:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.937537+0000", 2024-09-11T03:44:55.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000131941 2024-09-11T03:44:55.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.940300+0000", 2024-09-11T03:44:55.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027624720000000002 2024-09-11T03:44:55.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:55.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.940337+0000", 2024-09-11T03:44:55.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6869999999999998e-05 2024-09-11T03:44:55.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.940368+0000", 2024-09-11T03:44:55.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0837000000000003e-05 2024-09-11T03:44:55.301 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.301 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.301 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.301 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1896 2.e 2:7efbbea8:::smithi17557870-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e249)", 2024-09-11T03:44:55.301 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:05.924309+0000", 2024-09-11T03:44:55.301 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 229.23872889200001, 2024-09-11T03:44:55.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010556994, 2024-09-11T03:44:55.302 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.302 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.302 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:44:55.302 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.302 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.302 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.302 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1896 2024-09-11T03:44:55.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.302 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924309+0000", 2024-09-11T03:44:55.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924309+0000", 2024-09-11T03:44:55.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924309+0000", 2024-09-11T03:44:55.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6800000000000001e-07 2024-09-11T03:44:55.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924312+0000", 2024-09-11T03:44:55.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6690000000000002e-06 2024-09-11T03:44:55.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924319+0000", 2024-09-11T03:44:55.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3930000000000002e-06 2024-09-11T03:44:55.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924325+0000", 2024-09-11T03:44:55.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5770000000000001e-06 2024-09-11T03:44:55.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924400+0000", 2024-09-11T03:44:55.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5271000000000005e-05 2024-09-11T03:44:55.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:44:55.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924403+0000", 2024-09-11T03:44:55.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.101e-06 2024-09-11T03:44:55.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.925723+0000", 2024-09-11T03:44:55.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001319305 2024-09-11T03:44:55.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:55.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.925754+0000", 2024-09-11T03:44:55.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1161999999999997e-05 2024-09-11T03:44:55.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.934670+0000", 2024-09-11T03:44:55.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0089165860000000007 2024-09-11T03:44:55.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.934759+0000", 2024-09-11T03:44:55.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8437999999999996e-05 2024-09-11T03:44:55.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.934866+0000", 2024-09-11T03:44:55.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107024 2024-09-11T03:44:55.307 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.307 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.307 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.307 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2565 2.7 2:e0eabcfb:::smithi17557870-9:head [copy-from ver 223] snapc 70=[70,6f,6e,6c,6a,68,65,63,5d] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-11T03:44:55.307 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:09.876844+0000", 2024-09-11T03:44:55.308 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 165.28619419699999, 2024-09-11T03:44:55.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015411238000000001, 2024-09-11T03:44:55.308 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.308 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.308 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:55.308 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.308 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.308 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.308 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2565 2024-09-11T03:44:55.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.308 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876844+0000", 2024-09-11T03:44:55.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876844+0000", 2024-09-11T03:44:55.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876845+0000", 2024-09-11T03:44:55.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.78e-07 2024-09-11T03:44:55.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876850+0000", 2024-09-11T03:44:55.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9540000000000003e-06 2024-09-11T03:44:55.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876863+0000", 2024-09-11T03:44:55.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3349e-05 2024-09-11T03:44:55.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876873+0000", 2024-09-11T03:44:55.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9590000000000007e-06 2024-09-11T03:44:55.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876919+0000", 2024-09-11T03:44:55.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5720000000000003e-05 2024-09-11T03:44:55.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876984+0000", 2024-09-11T03:44:55.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5437000000000004e-05 2024-09-11T03:44:55.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:44:55.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.887314+0000", 2024-09-11T03:44:55.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0103298 2024-09-11T03:44:55.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:44:55.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.891590+0000", 2024-09-11T03:44:55.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042757380000000003 2024-09-11T03:44:55.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.891885+0000", 2024-09-11T03:44:55.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029539200000000001 2024-09-11T03:44:55.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.892177+0000", 2024-09-11T03:44:55.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029223699999999999 2024-09-11T03:44:55.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:55.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.892215+0000", 2024-09-11T03:44:55.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7740000000000001e-05 2024-09-11T03:44:55.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.892255+0000", 2024-09-11T03:44:55.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9934000000000001e-05 2024-09-11T03:44:55.313 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.313 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.313 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.314 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2566 2.7 2:e0eabcfb:::smithi17557870-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e313)", 2024-09-11T03:44:55.314 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:09.876886+0000", 2024-09-11T03:44:55.314 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 165.28615200900001, 2024-09-11T03:44:55.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014077338, 2024-09-11T03:44:55.314 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.314 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.314 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:44:55.314 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.314 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.314 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.314 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2566 2024-09-11T03:44:55.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.315 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876886+0000", 2024-09-11T03:44:55.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876886+0000", 2024-09-11T03:44:55.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876886+0000", 2024-09-11T03:44:55.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7000000000000005e-07 2024-09-11T03:44:55.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876889+0000", 2024-09-11T03:44:55.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5730000000000002e-06 2024-09-11T03:44:55.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876895+0000", 2024-09-11T03:44:55.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4160000000000001e-06 2024-09-11T03:44:55.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876899+0000", 2024-09-11T03:44:55.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3060000000000002e-06 2024-09-11T03:44:55.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.877088+0000", 2024-09-11T03:44:55.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018935499999999999 2024-09-11T03:44:55.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:44:55.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.877121+0000", 2024-09-11T03:44:55.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2566e-05 2024-09-11T03:44:55.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.890821+0000", 2024-09-11T03:44:55.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013699932999999999 2024-09-11T03:44:55.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.890886+0000", 2024-09-11T03:44:55.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5129999999999995e-05 2024-09-11T03:44:55.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.319 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.890963+0000", 2024-09-11T03:44:55.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7489000000000003e-05 2024-09-11T03:44:55.319 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.319 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.319 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.319 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2846 2.7 2:e6da0655:::smithi17557870-27:head [copy-from ver 182] snapc 7d=[7d,7c,7b,7a,79,78,77,75,6c,65] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-09-11T03:44:55.319 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:34.923113+0000", 2024-09-11T03:44:55.319 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 140.239925153, 2024-09-11T03:44:55.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011447241, 2024-09-11T03:44:55.319 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.319 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.319 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:55.320 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.320 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.320 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.320 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2846 2024-09-11T03:44:55.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.320 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.923113+0000", 2024-09-11T03:44:55.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.923113+0000", 2024-09-11T03:44:55.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.923115+0000", 2024-09-11T03:44:55.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9099999999999999e-06 2024-09-11T03:44:55.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.923123+0000", 2024-09-11T03:44:55.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4500000000000004e-06 2024-09-11T03:44:55.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.923141+0000", 2024-09-11T03:44:55.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8314000000000001e-05 2024-09-11T03:44:55.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.923152+0000", 2024-09-11T03:44:55.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0336000000000001e-05 2024-09-11T03:44:55.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.923208+0000", 2024-09-11T03:44:55.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6702e-05 2024-09-11T03:44:55.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.923276+0000", 2024-09-11T03:44:55.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7743000000000005e-05 2024-09-11T03:44:55.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:44:55.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.929687+0000", 2024-09-11T03:44:55.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0064108740000000004 2024-09-11T03:44:55.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:44:55.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.933037+0000", 2024-09-11T03:44:55.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003349781 2024-09-11T03:44:55.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.933319+0000", 2024-09-11T03:44:55.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028245800000000003 2024-09-11T03:44:55.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.934433+0000", 2024-09-11T03:44:55.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011135299999999999 2024-09-11T03:44:55.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:55.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.934504+0000", 2024-09-11T03:44:55.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1067000000000002e-05 2024-09-11T03:44:55.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.934560+0000", 2024-09-11T03:44:55.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6076000000000003e-05 2024-09-11T03:44:55.325 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.325 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.325 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.325 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:3054 2.e 2:7b0c3397:::smithi17557870-14:head [copy-from ver 182] snapc 82=[81,75] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-11T03:44:55.325 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:52.899536+0000", 2024-09-11T03:44:55.326 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 122.26350226, 2024-09-11T03:44:55.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011625175, 2024-09-11T03:44:55.326 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.326 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.326 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:55.326 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:44:55.326 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:44:55.326 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:44:55.326 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3054 2024-09-11T03:44:55.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.326 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.899536+0000", 2024-09-11T03:44:55.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.899536+0000", 2024-09-11T03:44:55.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.899538+0000", 2024-09-11T03:44:55.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.142e-06 2024-09-11T03:44:55.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.899546+0000", 2024-09-11T03:44:55.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8990000000000001e-06 2024-09-11T03:44:55.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.899564+0000", 2024-09-11T03:44:55.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8706999999999999e-05 2024-09-11T03:44:55.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.899576+0000", 2024-09-11T03:44:55.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1221e-05 2024-09-11T03:44:55.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.899633+0000", 2024-09-11T03:44:55.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7630999999999998e-05 2024-09-11T03:44:55.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.899711+0000", 2024-09-11T03:44:55.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7574e-05 2024-09-11T03:44:55.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:44:55.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.902898+0000", 2024-09-11T03:44:55.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031874999999999998 2024-09-11T03:44:55.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:44:55.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.907012+0000", 2024-09-11T03:44:55.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041138909999999997 2024-09-11T03:44:55.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.907303+0000", 2024-09-11T03:44:55.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029084 2024-09-11T03:44:55.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:44:55.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.911086+0000", 2024-09-11T03:44:55.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037825810000000001 2024-09-11T03:44:55.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:55.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.911124+0000", 2024-09-11T03:44:55.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8399999999999998e-05 2024-09-11T03:44:55.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.911161+0000", 2024-09-11T03:44:55.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6789000000000003e-05 2024-09-11T03:44:55.331 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.332 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.332 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.332 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:4021 2.7 e435/417 2:e77ce595:::smithi17557870-31:head v 435'563, mlcod=435'561)", 2024-09-11T03:44:55.332 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:44:13.499280+0000", 2024-09-11T03:44:55.332 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 41.663758113, 2024-09-11T03:44:55.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011214442999999999, 2024-09-11T03:44:55.332 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:44:55.332 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:44:55.332 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:44:55.332 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:44:55.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:44:55.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.499280+0000", 2024-09-11T03:44:55.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:44:55.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.499280+0000", 2024-09-11T03:44:55.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:44:55.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:44:55.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.499281+0000", 2024-09-11T03:44:55.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8400000000000002e-07 2024-09-11T03:44:55.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:44:55.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.506341+0000", 2024-09-11T03:44:55.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0070602399999999997 2024-09-11T03:44:55.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:44:55.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.506370+0000", 2024-09-11T03:44:55.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9048999999999999e-05 2024-09-11T03:44:55.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:44:55.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.506381+0000", 2024-09-11T03:44:55.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1440000000000001e-05 2024-09-11T03:44:55.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:44:55.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.506458+0000", 2024-09-11T03:44:55.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6941999999999996e-05 2024-09-11T03:44:55.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:44:55.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.506487+0000", 2024-09-11T03:44:55.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9104000000000001e-05 2024-09-11T03:44:55.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:44:55.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.510453+0000", 2024-09-11T03:44:55.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039655039999999999 2024-09-11T03:44:55.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:44:55.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:44:55.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:44:55.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.510494+0000", 2024-09-11T03:44:55.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1180000000000002e-05 2024-09-11T03:44:55.336 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.336 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.336 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.336 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:44:55.336 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:44:55.336 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:44:55.337 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:55.338 INFO:tasks.rados.rados.2.smithi175.stdout:1738: done (6 left) 2024-09-11T03:44:55.338 INFO:tasks.rados.rados.2.smithi175.stdout:1739: done (5 left) 2024-09-11T03:44:55.338 INFO:tasks.rados.rados.2.smithi175.stdout:1740: done (4 left) 2024-09-11T03:44:55.338 INFO:tasks.rados.rados.2.smithi175.stdout:1748: snap_remove snap 183 2024-09-11T03:44:55.338 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:54 smithi175 ceph-mon[30894]: from='client.16041 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:44:55.338 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:54 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:55.338 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:54 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3033361904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:44:55.338 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:54 smithi175 ceph-mon[30894]: osdmap e476: 12 total, 12 up, 11 in 2024-09-11T03:44:55.363 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:55.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:55.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:55.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:54 smithi121 ceph-mon[30729]: from='client.16041 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:44:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:54 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:44:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:54 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3033361904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:44:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:54 smithi121 ceph-mon[30729]: osdmap e476: 12 total, 12 up, 11 in 2024-09-11T03:44:55.466 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:55.491 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:55.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:55.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:55.593 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:55.615 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:55.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:55.616 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:55.717 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:55.738 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:55.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:55.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:55.839 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:55.865 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:55.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:55.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:55.874 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":868,"stamp":"2024-09-11T03:44:52.937033+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":203235287,"num_objects":91,"num_object_clones":47,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":9,"num_read":2203,"num_read_kb":2069796,"num_write":1608,"num_write_kb":625327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":61837646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11830,"ondisk_log_size":11830,"up":171,"acting":171,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":171,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":824648,"kb_used_data":423288,"kb_used_omap":17,"kb_used_meta":401070,"kb_avail":1030372024,"statfs":{"total":1055945392128,"available":1055100952576,"internally_reserved":0,"allocated":433446912,"data_stored":487779007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":410696565},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4250018,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":32919,"num_write":5,"num_write_kb":6337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.003396"},"pg_stats":[{"pgid":"1.1e","version":"20'92","reported_seq":532,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.941894+0000","last_change":"2024-09-11T03:44:24.235660+0000","last_active":"2024-09-11T03:44:50.941894+0000","last_peered":"2024-09-11T03:44:50.941894+0000","last_clean":"2024-09-11T03:44:50.941894+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:44:50.941894+0000","last_undegraded":"2024-09-11T03:44:50.941894+0000","last_fullsized":"2024-09-11T03:44:50.941894+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:24.235616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:24.235616+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:52.500250+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'92","reported_seq":520,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.943992+0000","last_change":"2024-09-11T03:40:41.348889+0000","last_active":"2024-09-11T03:44:50.943992+0000","last_peered":"2024-09-11T03:44:50.943992+0000","last_clean":"2024-09-11T03:44:50.943992+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:44:50.943992+0000","last_undegraded":"2024-09-11T03:44:50.943992+0000","last_fullsized":"2024-09-11T03:44:50.943992+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:41.348828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:41.348828+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:19.829110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'92","reported_seq":520,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.938397+0000","last_change":"2024-09-11T03:40:40.188174+0000","last_active":"2024-09-11T03:44:50.938397+0000","last_peered":"2024-09-11T03:44:50.938397+0000","last_clean":"2024-09-11T03:44:50.938397+0000","last_became_active":"2024-09-11T03:40:39.848230+0000","last_became_peered":"2024-09-11T03:40:39.848230+0000","last_unstale":"2024-09-11T03:44:50.938397+0000","last_undegraded":"2024-09-11T03:44:50.938397+0000","last_fullsized":"2024-09-11T03:44:50.938397+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.188124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.188124+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:16.276276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"467'328","reported_seq":2292,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T03:44:48.347745+0000","last_change":"2024-09-11T03:44:46.003937+0000","last_active":"2024-09-11T03:44:48.347745+0000","last_peered":"2024-09-11T03:44:48.347745+0000","last_clean":"2024-09-11T03:44:48.347745+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:44:48.347745+0000","last_undegraded":"2024-09-11T03:44:48.347745+0000","last_fullsized":"2024-09-11T03:44:48.347745+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"467'328","last_scrub_stamp":"2024-09-11T03:44:45.863666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:45.863666+0000","objects_scrubbed":4,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:08.456641+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0019832930000000001,"stat_sum":{"num_bytes":7546508,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":79,"num_read_kb":99501,"num_write":34,"num_write_kb":13379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4865740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b5","length":"4"}]},{"pgid":"1.1b","version":"20'92","reported_seq":520,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.938342+0000","last_change":"2024-09-11T03:40:42.166354+0000","last_active":"2024-09-11T03:44:50.938342+0000","last_peered":"2024-09-11T03:44:50.938342+0000","last_clean":"2024-09-11T03:44:50.938342+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:44:50.938342+0000","last_undegraded":"2024-09-11T03:44:50.938342+0000","last_fullsized":"2024-09-11T03:44:50.938342+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:42.166295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:42.166295+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:17.298159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.19","version":"454'417","reported_seq":2389,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T03:44:47.938136+0000","last_change":"2024-09-11T03:44:46.004756+0000","last_active":"2024-09-11T03:44:47.938136+0000","last_peered":"2024-09-11T03:44:47.938136+0000","last_clean":"2024-09-11T03:44:47.938136+0000","last_became_active":"2024-09-11T03:41:38.886787+0000","last_became_peered":"2024-09-11T03:41:38.886787+0000","last_unstale":"2024-09-11T03:44:47.938136+0000","last_undegraded":"2024-09-11T03:44:47.938136+0000","last_fullsized":"2024-09-11T03:44:47.938136+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"417'411","last_scrub_stamp":"2024-09-11T03:44:04.649503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:04.649503+0000","objects_scrubbed":3,"log_size":417,"log_dups_size":0,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:33.247797+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00065563600000000002,"stat_sum":{"num_bytes":2874096,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":22809,"num_write":19,"num_write_kb":10805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b5","length":"4"}]},{"pgid":"1.1a","version":"20'92","reported_seq":279,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.943763+0000","last_change":"2024-09-11T03:44:44.639175+0000","last_active":"2024-09-11T03:44:50.943763+0000","last_peered":"2024-09-11T03:44:50.943763+0000","last_clean":"2024-09-11T03:44:50.943763+0000","last_became_active":"2024-09-11T03:44:43.938873+0000","last_became_peered":"2024-09-11T03:44:43.938873+0000","last_unstale":"2024-09-11T03:44:50.943763+0000","last_undegraded":"2024-09-11T03:44:50.943763+0000","last_fullsized":"2024-09-11T03:44:50.943763+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:44.639128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:44.639128+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:52.231401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"20'92","reported_seq":138,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-11T03:44:51.943788+0000","last_change":"2024-09-11T03:44:05.420726+0000","last_active":"2024-09-11T03:44:51.943788+0000","last_peered":"2024-09-11T03:44:51.943788+0000","last_clean":"2024-09-11T03:44:51.943788+0000","last_became_active":"2024-09-11T03:43:52.920761+0000","last_became_peered":"2024-09-11T03:43:52.920761+0000","last_unstale":"2024-09-11T03:44:51.943788+0000","last_undegraded":"2024-09-11T03:44:51.943788+0000","last_fullsized":"2024-09-11T03:44:51.943788+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:05.420660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:05.420660+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:18.483482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'92","reported_seq":654,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.943754+0000","last_change":"2024-09-11T03:44:46.470994+0000","last_active":"2024-09-11T03:44:50.943754+0000","last_peered":"2024-09-11T03:44:50.943754+0000","last_clean":"2024-09-11T03:44:50.943754+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:44:50.943754+0000","last_undegraded":"2024-09-11T03:44:50.943754+0000","last_fullsized":"2024-09-11T03:44:50.943754+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:46.470921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:46.470921+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:13.212885+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"437'223","reported_seq":2139,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T03:44:48.348092+0000","last_change":"2024-09-11T03:44:46.005497+0000","last_active":"2024-09-11T03:44:48.348092+0000","last_peered":"2024-09-11T03:44:48.348092+0000","last_clean":"2024-09-11T03:44:48.348092+0000","last_became_active":"2024-09-11T03:41:38.890690+0000","last_became_peered":"2024-09-11T03:41:38.890690+0000","last_unstale":"2024-09-11T03:44:48.348092+0000","last_undegraded":"2024-09-11T03:44:48.348092+0000","last_fullsized":"2024-09-11T03:44:48.348092+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"437'223","last_scrub_stamp":"2024-09-11T03:44:42.964833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:42.964833+0000","objects_scrubbed":1,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:52.093388+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002216525,"stat_sum":{"num_bytes":3512762,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":8231,"num_write":19,"num_write_kb":4725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[9,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b5","length":"4"}]},{"pgid":"1.17","version":"20'92","reported_seq":535,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.938716+0000","last_change":"2024-09-11T03:43:52.928549+0000","last_active":"2024-09-11T03:44:50.938716+0000","last_peered":"2024-09-11T03:44:50.938716+0000","last_clean":"2024-09-11T03:44:50.938716+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:44:50.938716+0000","last_undegraded":"2024-09-11T03:44:50.938716+0000","last_fullsized":"2024-09-11T03:44:50.938716+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:41:57.260664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:41:57.260664+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:57.337816+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":2202,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T03:44:48.348064+0000","last_change":"2024-09-11T03:44:46.004875+0000","last_active":"2024-09-11T03:44:48.348064+0000","last_peered":"2024-09-11T03:44:48.348064+0000","last_clean":"2024-09-11T03:44:48.348064+0000","last_became_active":"2024-09-11T03:41:38.888228+0000","last_became_peered":"2024-09-11T03:41:38.888228+0000","last_unstale":"2024-09-11T03:44:48.348064+0000","last_undegraded":"2024-09-11T03:44:48.348064+0000","last_fullsized":"2024-09-11T03:44:48.348064+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:44:05.787400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:05.787400+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:33.964647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0028040830000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b5","length":"4"}]},{"pgid":"1.16","version":"20'92","reported_seq":542,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.943878+0000","last_change":"2024-09-11T03:44:23.840266+0000","last_active":"2024-09-11T03:44:50.943878+0000","last_peered":"2024-09-11T03:44:50.943878+0000","last_clean":"2024-09-11T03:44:50.943878+0000","last_became_active":"2024-09-11T03:40:39.849950+0000","last_became_peered":"2024-09-11T03:40:39.849950+0000","last_unstale":"2024-09-11T03:44:50.943878+0000","last_undegraded":"2024-09-11T03:44:50.943878+0000","last_fullsized":"2024-09-11T03:44:50.943878+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:23.840184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:23.840184+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:30.631786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"459'308","reported_seq":2209,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.947059+0000","last_change":"2024-09-11T03:44:50.947428+0000","last_active":"2024-09-11T03:44:50.947059+0000","last_peered":"2024-09-11T03:44:50.947059+0000","last_clean":"2024-09-11T03:44:50.947059+0000","last_became_active":"2024-09-11T03:41:38.887856+0000","last_became_peered":"2024-09-11T03:41:38.887856+0000","last_unstale":"2024-09-11T03:44:50.947059+0000","last_undegraded":"2024-09-11T03:44:50.947059+0000","last_fullsized":"2024-09-11T03:44:50.947059+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"442'296","last_scrub_stamp":"2024-09-11T03:44:22.311265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:22.311265+0000","objects_scrubbed":3,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:31.298601+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0032111560000000002,"stat_sum":{"num_bytes":4728853,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":42,"num_write_kb":17773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,9],"acting":[5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ae","length":"2"},{"start":"b5","length":"4"},{"start":"ba","length":"2"}]},{"pgid":"1.15","version":"20'92","reported_seq":527,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.941117+0000","last_change":"2024-09-11T03:43:09.565121+0000","last_active":"2024-09-11T03:44:50.941117+0000","last_peered":"2024-09-11T03:44:50.941117+0000","last_clean":"2024-09-11T03:44:50.941117+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:44:50.941117+0000","last_undegraded":"2024-09-11T03:44:50.941117+0000","last_fullsized":"2024-09-11T03:44:50.941117+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:09.565070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:09.565070+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:31.533657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":1632,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-11T03:44:51.944312+0000","last_change":"2024-09-11T03:44:50.946328+0000","last_active":"2024-09-11T03:44:51.944312+0000","last_peered":"2024-09-11T03:44:51.944312+0000","last_clean":"2024-09-11T03:44:51.944312+0000","last_became_active":"2024-09-11T03:43:55.918743+0000","last_became_peered":"2024-09-11T03:43:55.918743+0000","last_unstale":"2024-09-11T03:44:51.944312+0000","last_undegraded":"2024-09-11T03:44:51.944312+0000","last_fullsized":"2024-09-11T03:44:51.944312+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:44:04.380239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:04.380239+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:05.769751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000489801,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[10,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"af","length":"1"},{"start":"ba","length":"2"}]},{"pgid":"1.14","version":"20'92","reported_seq":557,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-11T03:44:47.233491+0000","last_change":"2024-09-11T03:44:12.182364+0000","last_active":"2024-09-11T03:44:47.233491+0000","last_peered":"2024-09-11T03:44:47.233491+0000","last_clean":"2024-09-11T03:44:47.233491+0000","last_became_active":"2024-09-11T03:40:27.847297+0000","last_became_peered":"2024-09-11T03:40:27.847297+0000","last_unstale":"2024-09-11T03:44:47.233491+0000","last_undegraded":"2024-09-11T03:44:47.233491+0000","last_fullsized":"2024-09-11T03:44:47.233491+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:12.182320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:12.182320+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:29.690356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":1016,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.938790+0000","last_change":"2024-09-11T03:44:50.939381+0000","last_active":"2024-09-11T03:44:50.938790+0000","last_peered":"2024-09-11T03:44:50.938790+0000","last_clean":"2024-09-11T03:44:50.938790+0000","last_became_active":"2024-09-11T03:41:35.874295+0000","last_became_peered":"2024-09-11T03:41:35.874295+0000","last_unstale":"2024-09-11T03:44:50.938790+0000","last_undegraded":"2024-09-11T03:44:50.938790+0000","last_fullsized":"2024-09-11T03:44:50.938790+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:44:47.465238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:47.465238+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:52.069778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000895202,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ae","length":"2"},{"start":"b5","length":"4"},{"start":"ba","length":"2"}]},{"pgid":"1.13","version":"20'92","reported_seq":658,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.943891+0000","last_change":"2024-09-11T03:44:10.979219+0000","last_active":"2024-09-11T03:44:50.943891+0000","last_peered":"2024-09-11T03:44:50.943891+0000","last_clean":"2024-09-11T03:44:50.943891+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:44:50.943891+0000","last_undegraded":"2024-09-11T03:44:50.943891+0000","last_fullsized":"2024-09-11T03:44:50.943891+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:10.979164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:10.979164+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:39.950458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":1079,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.944113+0000","last_change":"2024-09-11T03:44:50.944616+0000","last_active":"2024-09-11T03:44:50.944113+0000","last_peered":"2024-09-11T03:44:50.944113+0000","last_clean":"2024-09-11T03:44:50.944113+0000","last_became_active":"2024-09-11T03:44:43.939694+0000","last_became_peered":"2024-09-11T03:44:43.939694+0000","last_unstale":"2024-09-11T03:44:50.944113+0000","last_undegraded":"2024-09-11T03:44:50.944113+0000","last_fullsized":"2024-09-11T03:44:50.944113+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:44:45.591530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:45.591530+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:52.101612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000745661,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,9],"acting":[1,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ae","length":"2"},{"start":"b5","length":"4"},{"start":"ba","length":"2"}]},{"pgid":"1.12","version":"20'92","reported_seq":551,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.943763+0000","last_change":"2024-09-11T03:44:06.520517+0000","last_active":"2024-09-11T03:44:50.943763+0000","last_peered":"2024-09-11T03:44:50.943763+0000","last_clean":"2024-09-11T03:44:50.943763+0000","last_became_active":"2024-09-11T03:40:27.845249+0000","last_became_peered":"2024-09-11T03:40:27.845249+0000","last_unstale":"2024-09-11T03:44:50.943763+0000","last_undegraded":"2024-09-11T03:44:50.943763+0000","last_fullsized":"2024-09-11T03:44:50.943763+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:06.520457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:06.520457+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:08.245361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"471'323","reported_seq":2255,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-11T03:44:51.943906+0000","last_change":"2024-09-11T03:44:50.945580+0000","last_active":"2024-09-11T03:44:51.943906+0000","last_peered":"2024-09-11T03:44:51.943906+0000","last_clean":"2024-09-11T03:44:51.943906+0000","last_became_active":"2024-09-11T03:43:55.918817+0000","last_became_peered":"2024-09-11T03:43:55.918817+0000","last_unstale":"2024-09-11T03:44:51.943906+0000","last_undegraded":"2024-09-11T03:44:51.943906+0000","last_fullsized":"2024-09-11T03:44:51.943906+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"430'266","last_scrub_stamp":"2024-09-11T03:44:12.392724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:12.392724+0000","objects_scrubbed":3,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:16.149709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00066300400000000002,"stat_sum":{"num_bytes":9442334,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":102,"num_read_kb":88496,"num_write":74,"num_write_kb":28786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":15216236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"af","length":"1"},{"start":"ba","length":"2"}]},{"pgid":"1.11","version":"20'92","reported_seq":551,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.945039+0000","last_change":"2024-09-11T03:40:35.703292+0000","last_active":"2024-09-11T03:44:50.945039+0000","last_peered":"2024-09-11T03:44:50.945039+0000","last_clean":"2024-09-11T03:44:50.945039+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:44:50.945039+0000","last_undegraded":"2024-09-11T03:44:50.945039+0000","last_fullsized":"2024-09-11T03:44:50.945039+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.703203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.703203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:54.072667+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":1989,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T03:44:48.347902+0000","last_change":"2024-09-11T03:44:46.858699+0000","last_active":"2024-09-11T03:44:48.347902+0000","last_peered":"2024-09-11T03:44:48.347902+0000","last_clean":"2024-09-11T03:44:48.347902+0000","last_became_active":"2024-09-11T03:41:38.886834+0000","last_became_peered":"2024-09-11T03:41:38.886834+0000","last_unstale":"2024-09-11T03:44:48.347902+0000","last_undegraded":"2024-09-11T03:44:48.347902+0000","last_fullsized":"2024-09-11T03:44:48.347902+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:44:46.858645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:46.858645+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:12.426307+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0024269959999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b5","length":"4"}]},{"pgid":"1.10","version":"20'92","reported_seq":569,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.943757+0000","last_change":"2024-09-11T03:44:26.391068+0000","last_active":"2024-09-11T03:44:50.943757+0000","last_peered":"2024-09-11T03:44:50.943757+0000","last_clean":"2024-09-11T03:44:50.943757+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:44:50.943757+0000","last_undegraded":"2024-09-11T03:44:50.943757+0000","last_fullsized":"2024-09-11T03:44:50.943757+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:26.391001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:26.391001+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:29.074745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"473'498","reported_seq":2406,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.944054+0000","last_change":"2024-09-11T03:44:50.944583+0000","last_active":"2024-09-11T03:44:50.944054+0000","last_peered":"2024-09-11T03:44:50.944054+0000","last_clean":"2024-09-11T03:44:50.944054+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:44:50.944054+0000","last_undegraded":"2024-09-11T03:44:50.944054+0000","last_fullsized":"2024-09-11T03:44:50.944054+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'489","last_scrub_stamp":"2024-09-11T03:44:48.449022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:48.449022+0000","objects_scrubbed":6,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:12.361551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0058852619999999996,"stat_sum":{"num_bytes":16712649,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":136,"num_read_kb":88282,"num_write":146,"num_write_kb":70831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ae","length":"2"},{"start":"b5","length":"4"},{"start":"ba","length":"2"}]},{"pgid":"1.8","version":"20'92","reported_seq":628,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.943943+0000","last_change":"2024-09-11T03:42:05.135033+0000","last_active":"2024-09-11T03:44:50.943943+0000","last_peered":"2024-09-11T03:44:50.943943+0000","last_clean":"2024-09-11T03:44:50.943943+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:44:50.943943+0000","last_undegraded":"2024-09-11T03:44:50.943943+0000","last_fullsized":"2024-09-11T03:44:50.943943+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:05.134977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:05.134977+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:17.456475+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"473'217","reported_seq":2144,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.945547+0000","last_change":"2024-09-11T03:44:50.945907+0000","last_active":"2024-09-11T03:44:50.945547+0000","last_peered":"2024-09-11T03:44:50.945547+0000","last_clean":"2024-09-11T03:44:50.945547+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:44:50.945547+0000","last_undegraded":"2024-09-11T03:44:50.945547+0000","last_fullsized":"2024-09-11T03:44:50.945547+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'210","last_scrub_stamp":"2024-09-11T03:44:44.275761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:44.275761+0000","objects_scrubbed":2,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:46.252070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003751084,"stat_sum":{"num_bytes":5664157,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":131,"num_read_kb":136006,"num_write":74,"num_write_kb":28473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ae","length":"2"},{"start":"b5","length":"4"},{"start":"ba","length":"2"}]},{"pgid":"1.9","version":"20'92","reported_seq":957,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.940842+0000","last_change":"2024-09-11T03:42:00.816905+0000","last_active":"2024-09-11T03:44:50.940842+0000","last_peered":"2024-09-11T03:44:50.940842+0000","last_clean":"2024-09-11T03:44:50.940842+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:44:50.940842+0000","last_undegraded":"2024-09-11T03:44:50.940842+0000","last_fullsized":"2024-09-11T03:44:50.940842+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:00.816850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:00.816850+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:15.510201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"471'642","reported_seq":2660,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.944367+0000","last_change":"2024-09-11T03:44:50.944586+0000","last_active":"2024-09-11T03:44:50.944367+0000","last_peered":"2024-09-11T03:44:50.944367+0000","last_clean":"2024-09-11T03:44:50.944367+0000","last_became_active":"2024-09-11T03:41:33.865638+0000","last_became_peered":"2024-09-11T03:41:33.865638+0000","last_unstale":"2024-09-11T03:44:50.944367+0000","last_undegraded":"2024-09-11T03:44:50.944367+0000","last_fullsized":"2024-09-11T03:44:50.944367+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"427'574","last_scrub_stamp":"2024-09-11T03:44:06.983705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:06.983705+0000","objects_scrubbed":7,"log_size":642,"log_dups_size":0,"ondisk_log_size":642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:10.875787+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00043973500000000002,"stat_sum":{"num_bytes":13630939,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":197,"num_read_kb":231341,"num_write":101,"num_write_kb":35363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ae","length":"2"},{"start":"b5","length":"4"},{"start":"ba","length":"2"}]},{"pgid":"1.a","version":"20'92","reported_seq":871,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T03:44:48.347795+0000","last_change":"2024-09-11T03:44:12.896244+0000","last_active":"2024-09-11T03:44:48.347795+0000","last_peered":"2024-09-11T03:44:48.347795+0000","last_clean":"2024-09-11T03:44:48.347795+0000","last_became_active":"2024-09-11T03:40:37.849988+0000","last_became_peered":"2024-09-11T03:40:37.849988+0000","last_unstale":"2024-09-11T03:44:48.347795+0000","last_undegraded":"2024-09-11T03:44:48.347795+0000","last_fullsized":"2024-09-11T03:44:48.347795+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:12.896197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:12.896197+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:15.260075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"435'283","reported_seq":2221,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.941370+0000","last_change":"2024-09-11T03:44:50.941751+0000","last_active":"2024-09-11T03:44:50.941370+0000","last_peered":"2024-09-11T03:44:50.941370+0000","last_clean":"2024-09-11T03:44:50.941370+0000","last_became_active":"2024-09-11T03:41:33.863875+0000","last_became_peered":"2024-09-11T03:41:33.863875+0000","last_unstale":"2024-09-11T03:44:50.941370+0000","last_undegraded":"2024-09-11T03:44:50.941370+0000","last_fullsized":"2024-09-11T03:44:50.941370+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'283","last_scrub_stamp":"2024-09-11T03:44:29.473127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:29.473127+0000","objects_scrubbed":1,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:55.613869+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074734400000000005,"stat_sum":{"num_bytes":2773254,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":13815,"num_write":25,"num_write_kb":7046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ae","length":"2"},{"start":"b5","length":"4"},{"start":"ba","length":"2"}]},{"pgid":"1.b","version":"20'92","reported_seq":559,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.941610+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:44:50.941610+0000","last_peered":"2024-09-11T03:44:50.941610+0000","last_clean":"2024-09-11T03:44:50.941610+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:44:50.941610+0000","last_undegraded":"2024-09-11T03:44:50.941610+0000","last_fullsized":"2024-09-11T03:44:50.941610+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:59.023943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"471'663","reported_seq":2673,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-11T03:44:51.944174+0000","last_change":"2024-09-11T03:44:50.946138+0000","last_active":"2024-09-11T03:44:51.944174+0000","last_peered":"2024-09-11T03:44:51.944174+0000","last_clean":"2024-09-11T03:44:51.944174+0000","last_became_active":"2024-09-11T03:43:55.918858+0000","last_became_peered":"2024-09-11T03:43:55.918858+0000","last_unstale":"2024-09-11T03:44:51.944174+0000","last_undegraded":"2024-09-11T03:44:51.944174+0000","last_fullsized":"2024-09-11T03:44:51.944174+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'637","last_scrub_stamp":"2024-09-11T03:44:41.365645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:41.365645+0000","objects_scrubbed":10,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:03.222178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00063506599999999999,"stat_sum":{"num_bytes":25204362,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":165,"num_read_kb":171754,"num_write":87,"num_write_kb":37175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":14282259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,3],"acting":[10,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"af","length":"1"},{"start":"ba","length":"2"}]},{"pgid":"1.4","version":"20'92","reported_seq":741,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.941672+0000","last_change":"2024-09-11T03:44:28.192468+0000","last_active":"2024-09-11T03:44:50.941672+0000","last_peered":"2024-09-11T03:44:50.941672+0000","last_clean":"2024-09-11T03:44:50.941672+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:44:50.941672+0000","last_undegraded":"2024-09-11T03:44:50.941672+0000","last_fullsized":"2024-09-11T03:44:50.941672+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:28.192403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:28.192403+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:49.633126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":2079,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.951087+0000","last_change":"2024-09-11T03:44:50.951295+0000","last_active":"2024-09-11T03:44:50.951087+0000","last_peered":"2024-09-11T03:44:50.951087+0000","last_clean":"2024-09-11T03:44:50.951087+0000","last_became_active":"2024-09-11T03:41:33.865499+0000","last_became_peered":"2024-09-11T03:41:33.865499+0000","last_unstale":"2024-09-11T03:44:50.951087+0000","last_undegraded":"2024-09-11T03:44:50.951087+0000","last_fullsized":"2024-09-11T03:44:50.951087+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:44:22.878667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:22.878667+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:44.361872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070874500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ae","length":"2"},{"start":"b5","length":"4"},{"start":"ba","length":"2"}]},{"pgid":"1.5","version":"20'92","reported_seq":141,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-11T03:44:51.943888+0000","last_change":"2024-09-11T03:44:25.511501+0000","last_active":"2024-09-11T03:44:51.943888+0000","last_peered":"2024-09-11T03:44:51.943888+0000","last_clean":"2024-09-11T03:44:51.943888+0000","last_became_active":"2024-09-11T03:43:52.924325+0000","last_became_peered":"2024-09-11T03:43:52.924325+0000","last_unstale":"2024-09-11T03:44:51.943888+0000","last_undegraded":"2024-09-11T03:44:51.943888+0000","last_fullsized":"2024-09-11T03:44:51.943888+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:25.511435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:25.511435+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:32.421579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"471'643","reported_seq":2614,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.947996+0000","last_change":"2024-09-11T03:44:50.939279+0000","last_active":"2024-09-11T03:44:50.947996+0000","last_peered":"2024-09-11T03:44:50.947996+0000","last_clean":"2024-09-11T03:44:50.947996+0000","last_became_active":"2024-09-11T03:41:33.866316+0000","last_became_peered":"2024-09-11T03:41:33.866316+0000","last_unstale":"2024-09-11T03:44:50.947996+0000","last_undegraded":"2024-09-11T03:44:50.947996+0000","last_fullsized":"2024-09-11T03:44:50.947996+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'591","last_scrub_stamp":"2024-09-11T03:44:24.345630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:24.345630+0000","objects_scrubbed":4,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:29.195204+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00062498199999999995,"stat_sum":{"num_bytes":11891790,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":137,"num_read_kb":159280,"num_write":74,"num_write_kb":25714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ae","length":"2"},{"start":"b5","length":"4"},{"start":"ba","length":"2"}]},{"pgid":"1.6","version":"20'92","reported_seq":141,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-11T03:44:51.943893+0000","last_change":"2024-09-11T03:44:26.477597+0000","last_active":"2024-09-11T03:44:51.943893+0000","last_peered":"2024-09-11T03:44:51.943893+0000","last_clean":"2024-09-11T03:44:51.943893+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:44:51.943893+0000","last_undegraded":"2024-09-11T03:44:51.943893+0000","last_fullsized":"2024-09-11T03:44:51.943893+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:26.477545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:26.477545+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:55.012137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"473'349","reported_seq":2310,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.952847+0000","last_change":"2024-09-11T03:44:50.953042+0000","last_active":"2024-09-11T03:44:50.952847+0000","last_peered":"2024-09-11T03:44:50.952847+0000","last_clean":"2024-09-11T03:44:50.952847+0000","last_became_active":"2024-09-11T03:44:10.921135+0000","last_became_peered":"2024-09-11T03:44:10.921135+0000","last_unstale":"2024-09-11T03:44:50.952847+0000","last_undegraded":"2024-09-11T03:44:50.952847+0000","last_fullsized":"2024-09-11T03:44:50.952847+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"427'266","last_scrub_stamp":"2024-09-11T03:44:06.079088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:06.079088+0000","objects_scrubbed":1,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:13.552584+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0088559280000000008,"stat_sum":{"num_bytes":13826820,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":71,"num_read_kb":84058,"num_write":65,"num_write_kb":15961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":4118225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ae","length":"2"},{"start":"b5","length":"4"},{"start":"ba","length":"2"}]},{"pgid":"1.7","version":"20'92","reported_seq":138,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-11T03:44:51.944094+0000","last_change":"2024-09-11T03:43:54.313880+0000","last_active":"2024-09-11T03:44:51.944094+0000","last_peered":"2024-09-11T03:44:51.944094+0000","last_clean":"2024-09-11T03:44:51.944094+0000","last_became_active":"2024-09-11T03:43:52.923384+0000","last_became_peered":"2024-09-11T03:43:52.923384+0000","last_unstale":"2024-09-11T03:44:51.944094+0000","last_undegraded":"2024-09-11T03:44:51.944094+0000","last_fullsized":"2024-09-11T03:44:51.944094+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:54.313794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:54.313794+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:09.663654+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,5],"acting":[10,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"471'209","reported_seq":2181,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.944262+0000","last_change":"2024-09-11T03:44:50.945192+0000","last_active":"2024-09-11T03:44:50.944262+0000","last_peered":"2024-09-11T03:44:50.944262+0000","last_clean":"2024-09-11T03:44:50.944262+0000","last_became_active":"2024-09-11T03:42:30.902838+0000","last_became_peered":"2024-09-11T03:42:30.902838+0000","last_unstale":"2024-09-11T03:44:50.944262+0000","last_undegraded":"2024-09-11T03:44:50.944262+0000","last_fullsized":"2024-09-11T03:44:50.944262+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'191","last_scrub_stamp":"2024-09-11T03:44:21.347407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:21.347407+0000","objects_scrubbed":4,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:44.315538+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012298579999999999,"stat_sum":{"num_bytes":2554763,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":34,"num_read_kb":27471,"num_write":19,"num_write_kb":8071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,11],"acting":[5,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ae","length":"2"},{"start":"b5","length":"4"},{"start":"ba","length":"2"}]},{"pgid":"1.1","version":"20'92","reported_seq":138,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-11T03:44:51.943780+0000","last_change":"2024-09-11T03:43:53.274298+0000","last_active":"2024-09-11T03:44:51.943780+0000","last_peered":"2024-09-11T03:44:51.943780+0000","last_clean":"2024-09-11T03:44:51.943780+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:44:51.943780+0000","last_undegraded":"2024-09-11T03:44:51.943780+0000","last_fullsized":"2024-09-11T03:44:51.943780+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:53.274236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:53.274236+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:18.478310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"405'143","reported_seq":2055,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-11T03:44:47.233647+0000","last_change":"2024-09-11T03:44:46.009726+0000","last_active":"2024-09-11T03:44:47.233647+0000","last_peered":"2024-09-11T03:44:47.233647+0000","last_clean":"2024-09-11T03:44:47.233647+0000","last_became_active":"2024-09-11T03:41:33.866217+0000","last_became_peered":"2024-09-11T03:41:33.866217+0000","last_unstale":"2024-09-11T03:44:47.233647+0000","last_undegraded":"2024-09-11T03:44:47.233647+0000","last_fullsized":"2024-09-11T03:44:47.233647+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'143","last_scrub_stamp":"2024-09-11T03:44:09.120125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:09.120125+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:17.820026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00074707699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":1777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b5","length":"4"}]},{"pgid":"1.2","version":"20'92","reported_seq":539,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.945039+0000","last_change":"2024-09-11T03:44:25.934869+0000","last_active":"2024-09-11T03:44:50.945039+0000","last_peered":"2024-09-11T03:44:50.945039+0000","last_clean":"2024-09-11T03:44:50.945039+0000","last_became_active":"2024-09-11T03:44:25.934580+0000","last_became_peered":"2024-09-11T03:44:25.934580+0000","last_unstale":"2024-09-11T03:44:50.945039+0000","last_undegraded":"2024-09-11T03:44:50.945039+0000","last_fullsized":"2024-09-11T03:44:50.945039+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:23.494418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:23.494418+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:36.969904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"473'135","reported_seq":2115,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.945792+0000","last_change":"2024-09-11T03:44:50.946266+0000","last_active":"2024-09-11T03:44:50.945792+0000","last_peered":"2024-09-11T03:44:50.945792+0000","last_clean":"2024-09-11T03:44:50.945792+0000","last_became_active":"2024-09-11T03:43:55.920265+0000","last_became_peered":"2024-09-11T03:43:55.920265+0000","last_unstale":"2024-09-11T03:44:50.945792+0000","last_undegraded":"2024-09-11T03:44:50.945792+0000","last_fullsized":"2024-09-11T03:44:50.945792+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"462'115","last_scrub_stamp":"2024-09-11T03:44:40.414111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:40.414111+0000","objects_scrubbed":2,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:47.560106+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0050642090000000001,"stat_sum":{"num_bytes":4672544,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":30738,"num_write":19,"num_write_kb":7481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4387035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ae","length":"2"},{"start":"b5","length":"4"},{"start":"ba","length":"2"}]},{"pgid":"1.3","version":"20'92","reported_seq":573,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.944010+0000","last_change":"2024-09-11T03:44:10.921140+0000","last_active":"2024-09-11T03:44:50.944010+0000","last_peered":"2024-09-11T03:44:50.944010+0000","last_clean":"2024-09-11T03:44:50.944010+0000","last_became_active":"2024-09-11T03:44:10.920828+0000","last_became_peered":"2024-09-11T03:44:10.920828+0000","last_unstale":"2024-09-11T03:44:50.944010+0000","last_undegraded":"2024-09-11T03:44:50.944010+0000","last_fullsized":"2024-09-11T03:44:50.944010+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:07.508193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:07.508193+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:16.182235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"473'302","reported_seq":2236,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.947232+0000","last_change":"2024-09-11T03:44:50.947543+0000","last_active":"2024-09-11T03:44:50.947232+0000","last_peered":"2024-09-11T03:44:50.947232+0000","last_clean":"2024-09-11T03:44:50.947232+0000","last_became_active":"2024-09-11T03:41:33.862271+0000","last_became_peered":"2024-09-11T03:41:33.862271+0000","last_unstale":"2024-09-11T03:44:50.947232+0000","last_undegraded":"2024-09-11T03:44:50.947232+0000","last_fullsized":"2024-09-11T03:44:50.947232+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'229","last_scrub_stamp":"2024-09-11T03:44:07.554423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:07.554423+0000","objects_scrubbed":6,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:12.167368+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0035675049999999999,"stat_sum":{"num_bytes":14269638,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":90,"num_read_kb":77297,"num_write":46,"num_write_kb":19734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ae","length":"2"},{"start":"b5","length":"4"},{"start":"ba","length":"2"}]},{"pgid":"1.0","version":"20'92","reported_seq":962,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.941343+0000","last_change":"2024-09-11T03:43:05.695738+0000","last_active":"2024-09-11T03:44:50.941343+0000","last_peered":"2024-09-11T03:44:50.941343+0000","last_clean":"2024-09-11T03:44:50.941343+0000","last_became_active":"2024-09-11T03:40:25.854624+0000","last_became_peered":"2024-09-11T03:40:25.854624+0000","last_unstale":"2024-09-11T03:44:50.941343+0000","last_undegraded":"2024-09-11T03:44:50.941343+0000","last_fullsized":"2024-09-11T03:44:50.941343+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:05.695673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:05.695673+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:16.473462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"20'92","reported_seq":545,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.944243+0000","last_change":"2024-09-11T03:40:35.282811+0000","last_active":"2024-09-11T03:44:50.944243+0000","last_peered":"2024-09-11T03:44:50.944243+0000","last_clean":"2024-09-11T03:44:50.944243+0000","last_became_active":"2024-09-11T03:40:27.844262+0000","last_became_peered":"2024-09-11T03:40:27.844262+0000","last_unstale":"2024-09-11T03:44:50.944243+0000","last_undegraded":"2024-09-11T03:44:50.944243+0000","last_fullsized":"2024-09-11T03:44:50.944243+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.282762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.282762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:44:58.420352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"473'529","reported_seq":2446,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.953592+0000","last_change":"2024-09-11T03:44:50.953907+0000","last_active":"2024-09-11T03:44:50.953592+0000","last_peered":"2024-09-11T03:44:50.953592+0000","last_clean":"2024-09-11T03:44:50.953592+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:44:50.953592+0000","last_undegraded":"2024-09-11T03:44:50.953592+0000","last_fullsized":"2024-09-11T03:44:50.953592+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'414","last_scrub_stamp":"2024-09-11T03:43:48.982261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:48.982261+0000","objects_scrubbed":4,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:17.026692+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.011927138,"stat_sum":{"num_bytes":18399420,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":230,"num_read_kb":184708,"num_write":203,"num_write_kb":81188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ae","length":"2"},{"start":"b5","length":"4"},{"start":"ba","length":"2"}]},{"pgid":"1.e","version":"20'92","reported_seq":279,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.941277+0000","last_change":"2024-09-11T03:44:44.403660+0000","last_active":"2024-09-11T03:44:50.941277+0000","last_peered":"2024-09-11T03:44:50.941277+0000","last_clean":"2024-09-11T03:44:50.941277+0000","last_became_active":"2024-09-11T03:44:43.940022+0000","last_became_peered":"2024-09-11T03:44:43.940022+0000","last_unstale":"2024-09-11T03:44:50.941277+0000","last_undegraded":"2024-09-11T03:44:50.941277+0000","last_fullsized":"2024-09-11T03:44:50.941277+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:44.403586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:44.403586+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:52.781784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"473'655","reported_seq":2556,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.946752+0000","last_change":"2024-09-11T03:44:50.946992+0000","last_active":"2024-09-11T03:44:50.946752+0000","last_peered":"2024-09-11T03:44:50.946752+0000","last_clean":"2024-09-11T03:44:50.946752+0000","last_became_active":"2024-09-11T03:42:30.902959+0000","last_became_peered":"2024-09-11T03:42:30.902959+0000","last_unstale":"2024-09-11T03:44:50.946752+0000","last_undegraded":"2024-09-11T03:44:50.946752+0000","last_fullsized":"2024-09-11T03:44:50.946752+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'559","last_scrub_stamp":"2024-09-11T03:43:51.375966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:43:51.375966+0000","objects_scrubbed":9,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:09.820964+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0030209780000000001,"stat_sum":{"num_bytes":10467877,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":303,"num_read_kb":281932,"num_write":204,"num_write_kb":77332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17460823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ae","length":"2"},{"start":"b5","length":"4"},{"start":"ba","length":"2"}]},{"pgid":"1.d","version":"20'92","reported_seq":571,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.941148+0000","last_change":"2024-09-11T03:42:51.733829+0000","last_active":"2024-09-11T03:44:50.941148+0000","last_peered":"2024-09-11T03:44:50.941148+0000","last_clean":"2024-09-11T03:44:50.941148+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:44:50.941148+0000","last_undegraded":"2024-09-11T03:44:50.941148+0000","last_fullsized":"2024-09-11T03:44:50.941148+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:51.733762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:51.733762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:03.542258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"468'692","reported_seq":2566,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-11T03:44:47.233780+0000","last_change":"2024-09-11T03:44:46.013128+0000","last_active":"2024-09-11T03:44:47.233780+0000","last_peered":"2024-09-11T03:44:47.233780+0000","last_clean":"2024-09-11T03:44:47.233780+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:44:47.233780+0000","last_undegraded":"2024-09-11T03:44:47.233780+0000","last_fullsized":"2024-09-11T03:44:47.233780+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'600","last_scrub_stamp":"2024-09-11T03:44:08.151567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:08.151567+0000","objects_scrubbed":7,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:19.151258+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0038337850000000001,"stat_sum":{"num_bytes":20093002,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":270,"num_read_kb":250591,"num_write":210,"num_write_kb":82803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b5","length":"4"}]},{"pgid":"2.f","version":"473'378","reported_seq":2304,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-11T03:44:50.941626+0000","last_change":"2024-09-11T03:44:50.941872+0000","last_active":"2024-09-11T03:44:50.941626+0000","last_peered":"2024-09-11T03:44:50.941626+0000","last_clean":"2024-09-11T03:44:50.941626+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:44:50.941626+0000","last_undegraded":"2024-09-11T03:44:50.941626+0000","last_fullsized":"2024-09-11T03:44:50.941626+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'342","last_scrub_stamp":"2024-09-11T03:44:25.345256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:25.345256+0000","objects_scrubbed":7,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:31.915926+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002986542,"stat_sum":{"num_bytes":14379151,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":148,"num_read_kb":113478,"num_write":139,"num_write_kb":50910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ae","length":"2"},{"start":"b5","length":"4"},{"start":"ba","length":"2"}]},{"pgid":"1.c","version":"20'92","reported_seq":582,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T03:44:47.938092+0000","last_change":"2024-09-11T03:43:50.727772+0000","last_active":"2024-09-11T03:44:47.938092+0000","last_peered":"2024-09-11T03:44:47.938092+0000","last_clean":"2024-09-11T03:44:47.938092+0000","last_became_active":"2024-09-11T03:40:37.847449+0000","last_became_peered":"2024-09-11T03:40:37.847449+0000","last_unstale":"2024-09-11T03:44:47.938092+0000","last_undegraded":"2024-09-11T03:44:47.938092+0000","last_fullsized":"2024-09-11T03:44:47.938092+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:50.727710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:50.727710+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:45:01.411341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,9],"acting":[3,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":202644919,"num_objects":89,"num_object_clones":47,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":9,"num_read":2203,"num_read_kb":2069796,"num_write":1608,"num_write_kb":625327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":61837646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":373710848,"data_stored":450769750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8978,"ondisk_log_size":8978,"up":78,"acting":78,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285660,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68644,"kb_used_data":33296,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93676508,"statfs":{"total":95995035648,"available":95924744192,"internally_reserved":0,"allocated":34095104,"data_stored":41308572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514592,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67952,"kb_used_data":36756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93677200,"statfs":{"total":95995035648,"available":95925452800,"internally_reserved":0,"allocated":37638144,"data_stored":42011376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743525,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87132,"kb_used_data":43460,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93658020,"statfs":{"total":95995035648,"available":95905812480,"internally_reserved":0,"allocated":44503040,"data_stored":48845029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":44693952},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":69872,"kb_used_data":30356,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93675280,"statfs":{"total":95995035648,"available":95923486720,"internally_reserved":0,"allocated":31084544,"data_stored":33768654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299504,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71980,"kb_used_data":44952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93673172,"statfs":{"total":95995035648,"available":95921328128,"internally_reserved":0,"allocated":46030848,"data_stored":50553166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":193273528437,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757371,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62292,"kb_used_data":26936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93682860,"statfs":{"total":95995035648,"available":95931248640,"internally_reserved":0,"allocated":27582464,"data_stored":27759967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986304,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70392,"kb_used_data":30876,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93674760,"statfs":{"total":95995035648,"available":95922954240,"internally_reserved":0,"allocated":31617024,"data_stored":34159375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91664,"kb_used_data":60472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93653488,"statfs":{"total":95995035648,"available":95901171712,"internally_reserved":0,"allocated":61923328,"data_stored":72482490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411466,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83428,"kb_used_data":43920,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93661724,"statfs":{"total":95995035648,"available":95909605376,"internally_reserved":0,"allocated":44974080,"data_stored":47910603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74868,"kb_used_data":35348,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93670284,"statfs":{"total":95995035648,"available":95918370816,"internally_reserved":0,"allocated":36196352,"data_stored":44342873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738517,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76424,"kb_used_data":36916,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93668728,"statfs":{"total":95995035648,"available":95916777472,"internally_reserved":0,"allocated":37801984,"data_stored":44636902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31801344,"data_stored":40708055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30535680,"data_stored":40749989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":39145472,"data_stored":44141232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":55971840,"data_stored":68586687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25968640,"data_stored":30574774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":21704704,"data_stored":23957552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4079616,"data_stored":4066399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40153088,"data_stored":46750751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":25423872,"data_stored":30175770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":38551552,"data_stored":44949226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":31952896,"data_stored":38401910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28422144,"data_stored":37707405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:44:55.877 INFO:tasks.thrashosds.thrasher:Setting 1.16 to [11, 2, 0] 2024-09-11T03:44:55.877 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.16', '11', '2', '0'] 2024-09-11T03:44:55.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pg-upmap 1.16 11 2 0 2024-09-11T03:44:55.967 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:55.977 INFO:tasks.rados.rados.2.smithi175.stdout:1748: done (4 left) 2024-09-11T03:44:55.977 INFO:tasks.rados.rados.2.smithi175.stdout:1749: write (excl) oid 19 current snap is 189 2024-09-11T03:44:55.977 INFO:tasks.rados.rados.2.smithi175.stdout:1749: seq_num 603 ranges {534621=567034,1636289=498332} 2024-09-11T03:44:55.983 INFO:tasks.rados.rados.2.smithi175.stdout:1749: writing smithi17557870-19 from 534621 to 1101655 tid 1 2024-09-11T03:44:55.990 INFO:tasks.rados.rados.2.smithi175.stdout:1749: writing smithi17557870-19 from 1636289 to 2134621 tid 2 2024-09-11T03:44:55.990 INFO:tasks.rados.rados.2.smithi175.stdout:1750: write (excl) oid 46 current snap is 189 2024-09-11T03:44:55.990 INFO:tasks.rados.rados.2.smithi175.stdout:1737: finishing rollback tid 0 to smithi17557870-33 2024-09-11T03:44:55.990 INFO:tasks.rados.rados.2.smithi175.stdout:1750: seq_num 604 ranges {551428=642283,1694771=456657} 2024-09-11T03:44:55.991 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:55.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:55.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:55.996 INFO:tasks.rados.rados.2.smithi175.stdout:1750: writing smithi17557870-46 from 551428 to 1193711 tid 1 2024-09-11T03:44:56.001 INFO:tasks.rados.rados.2.smithi175.stdout:1750: writing smithi17557870-46 from 1694771 to 2151428 tid 2 2024-09-11T03:44:56.002 INFO:tasks.rados.rados.2.smithi175.stdout:1751: read oid 22 snap -1 2024-09-11T03:44:56.002 INFO:tasks.rados.rados.2.smithi175.stdout:1751: expect (ObjNum 506 snap 148 seq_num 506) 2024-09-11T03:44:56.002 INFO:tasks.rados.rados.2.smithi175.stdout:1752: snap_create 2024-09-11T03:44:56.002 INFO:tasks.rados.rados.2.smithi175.stdout:1737: finishing rollback tid 1 to smithi17557870-33 2024-09-11T03:44:56.002 INFO:tasks.rados.rados.2.smithi175.stdout:1737: finishing rollback tid 2 to smithi17557870-33 2024-09-11T03:44:56.002 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 646 (ObjNum 495 snap 143 seq_num 495) dirty exists 2024-09-11T03:44:56.002 INFO:tasks.rados.rados.2.smithi175.stdout:1742: expect (ObjNum 587 snap 183 seq_num 587) 2024-09-11T03:44:56.094 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:56.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:56.117 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:56.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:56.127 INFO:tasks.rados.rados.2.smithi175.stdout:1747: finishing rollback tid 0 to smithi17557870-3 2024-09-11T03:44:56.127 INFO:tasks.rados.rados.2.smithi175.stdout:1747: finishing rollback tid 1 to smithi17557870-3 2024-09-11T03:44:56.127 INFO:tasks.rados.rados.2.smithi175.stdout:1747: finishing rollback tid 2 to smithi17557870-3 2024-09-11T03:44:56.127 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 652 (ObjNum 488 snap 141 seq_num 488) dirty exists 2024-09-11T03:44:56.127 INFO:tasks.rados.rados.2.smithi175.stdout:1746: expect (ObjNum 528 snap 156 seq_num 528) 2024-09-11T03:44:56.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:55 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:44:56.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:55 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:44:56.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:55 smithi084 ceph-mon[27254]: pgmap v870: 57 pgs: 57 active+clean; 194 MiB data, 801 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2024-09-11T03:44:56.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:55 smithi084 ceph-mon[27254]: osdmap e477: 12 total, 12 up, 11 in 2024-09-11T03:44:56.218 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:56.238 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:56.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:56.239 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:56.309 INFO:tasks.rados.rados.2.smithi175.stdout:1749: finishing write tid 1 to smithi17557870-19 2024-09-11T03:44:56.309 INFO:tasks.rados.rados.2.smithi175.stdout:1749: finishing write tid 2 to smithi17557870-19 2024-09-11T03:44:56.309 INFO:tasks.rados.rados.2.smithi175.stdout:1749: finishing write tid 3 to smithi17557870-19 2024-09-11T03:44:56.310 INFO:tasks.rados.rados.2.smithi175.stdout:1749: finishing write tid 4 to smithi17557870-19 2024-09-11T03:44:56.310 INFO:tasks.rados.rados.2.smithi175.stdout:1749: oid 19 updating version 0 to 474 2024-09-11T03:44:56.310 INFO:tasks.rados.rados.2.smithi175.stdout:1749: oid 19 updating version 474 to 476 2024-09-11T03:44:56.310 INFO:tasks.rados.rados.2.smithi175.stdout:1749: oid 19 version 476 is already newer than 475 2024-09-11T03:44:56.310 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 476 (ObjNum 603 snap 189 seq_num 603) dirty exists 2024-09-11T03:44:56.310 INFO:tasks.rados.rados.2.smithi175.stdout:1749: left oid 19 (ObjNum 603 snap 189 seq_num 603) 2024-09-11T03:44:56.310 INFO:tasks.rados.rados.2.smithi175.stdout:1750: finishing write tid 1 to smithi17557870-46 2024-09-11T03:44:56.310 INFO:tasks.rados.rados.2.smithi175.stdout:1750: finishing write tid 2 to smithi17557870-46 2024-09-11T03:44:56.310 INFO:tasks.rados.rados.2.smithi175.stdout:1750: finishing write tid 3 to smithi17557870-46 2024-09-11T03:44:56.310 INFO:tasks.rados.rados.2.smithi175.stdout:1750: finishing write tid 4 to smithi17557870-46 2024-09-11T03:44:56.310 INFO:tasks.rados.rados.2.smithi175.stdout:1750: oid 46 updating version 0 to 526 2024-09-11T03:44:56.310 INFO:tasks.rados.rados.2.smithi175.stdout:1750: oid 46 updating version 526 to 527 2024-09-11T03:44:56.310 INFO:tasks.rados.rados.2.smithi175.stdout:1750: oid 46 updating version 527 to 528 2024-09-11T03:44:56.311 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 528 (ObjNum 604 snap 189 seq_num 604) dirty exists 2024-09-11T03:44:56.311 INFO:tasks.rados.rados.2.smithi175.stdout:1750: left oid 46 (ObjNum 604 snap 189 seq_num 604) 2024-09-11T03:44:56.311 INFO:tasks.rados.rados.2.smithi175.stdout:1751: expect (ObjNum 506 snap 148 seq_num 506) 2024-09-11T03:44:56.340 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:56.362 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:56.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:56.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:56.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:55 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:44:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:55 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:44:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:55 smithi121 ceph-mon[30729]: pgmap v870: 57 pgs: 57 active+clean; 194 MiB data, 801 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2024-09-11T03:44:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:55 smithi121 ceph-mon[30729]: osdmap e477: 12 total, 12 up, 11 in 2024-09-11T03:44:56.463 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:56.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:55 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:44:56.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:55 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:44:56.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:55 smithi175 ceph-mon[30894]: pgmap v870: 57 pgs: 57 active+clean; 194 MiB data, 801 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2024-09-11T03:44:56.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:55 smithi175 ceph-mon[30894]: osdmap e477: 12 total, 12 up, 11 in 2024-09-11T03:44:56.489 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:56.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:56.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:56.591 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:56.613 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:56.614 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:56.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T03:44:56.692 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:56.715 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:56.736 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:56.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:56.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:56.838 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:56.859 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:56.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:56.860 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:56.939 INFO:tasks.rados.rados.2.smithi175.stdout:1737: done (7 left) 2024-09-11T03:44:56.939 INFO:tasks.rados.rados.2.smithi175.stdout:1742: done (6 left) 2024-09-11T03:44:56.939 INFO:tasks.rados.rados.2.smithi175.stdout:1746: done (5 left) 2024-09-11T03:44:56.940 INFO:tasks.rados.rados.2.smithi175.stdout:1747: done (4 left) 2024-09-11T03:44:56.940 INFO:tasks.rados.rados.2.smithi175.stdout:1749: done (3 left) 2024-09-11T03:44:56.940 INFO:tasks.rados.rados.2.smithi175.stdout:1750: done (2 left) 2024-09-11T03:44:56.940 INFO:tasks.rados.rados.2.smithi175.stdout:1751: done (1 left) 2024-09-11T03:44:56.940 INFO:tasks.rados.rados.2.smithi175.stdout:1752: done (0 left) 2024-09-11T03:44:56.940 INFO:tasks.rados.rados.2.smithi175.stdout:1753: read oid 26 snap 186 2024-09-11T03:44:56.940 INFO:tasks.rados.rados.2.smithi175.stdout:1753: expect (ObjNum 595 snap 185 seq_num 595) 2024-09-11T03:44:56.940 INFO:tasks.rados.rados.2.smithi175.stdout:1754: delete oid 39 current snap is 190 2024-09-11T03:44:56.940 INFO:tasks.rados.rados.2.smithi175.stdout:1754: done (1 left) 2024-09-11T03:44:56.941 INFO:tasks.rados.rados.2.smithi175.stdout:1755: snap_create 2024-09-11T03:44:56.946 INFO:tasks.rados.rados.2.smithi175.stdout:1753: expect (ObjNum 595 snap 185 seq_num 595) 2024-09-11T03:44:56.949 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:44:56.960 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:56.982 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:56.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:56.983 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:57.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:57.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:57.110 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:57.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:57.194 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:44:57.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:44:57.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:57.219 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:44:57.219 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:44:57.219 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:57 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:44:57.219 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:57 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:44:57.219 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:57 smithi084 ceph-mon[27254]: osdmap e478: 12 total, 12 up, 11 in 2024-09-11T03:44:57.219 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:57 smithi084 ceph-mon[27254]: osdmap e479: 12 total, 12 up, 11 in 2024-09-11T03:44:57.236 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:57.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:57.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:57.237 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:44:57.238 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:44:57.246 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:57.252 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:57.254 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:44:57.255 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:44:57.280 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:44:57.280 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:44:57.299 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:44:57.299 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:44:57.316 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:44:57.316 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:44:57.332 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:44:57.333 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:44:57.337 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:44:57.349 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:44:57.349 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:44:57.360 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:57.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:57.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:44:57.368 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:44:57.368 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:44:57.385 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:44:57.385 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:44:57.401 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:44:57.401 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:44:57.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:57 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:44:57.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:57 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:44:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:57 smithi121 ceph-mon[30729]: osdmap e478: 12 total, 12 up, 11 in 2024-09-11T03:44:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:57 smithi121 ceph-mon[30729]: osdmap e479: 12 total, 12 up, 11 in 2024-09-11T03:44:57.463 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:57 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:44:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:57 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:44:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:57 smithi175 ceph-mon[30894]: osdmap e478: 12 total, 12 up, 11 in 2024-09-11T03:44:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:57 smithi175 ceph-mon[30894]: osdmap e479: 12 total, 12 up, 11 in 2024-09-11T03:44:57.486 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:57.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:57.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:57.611 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:57.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:57.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:57.713 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:57.736 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:57.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:57.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:57.838 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:44:57.859 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:57.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:57.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:44:57.938 INFO:tasks.rados.rados.2.smithi175.stdout:1753: done (1 left) 2024-09-11T03:44:57.938 INFO:tasks.rados.rados.2.smithi175.stdout:1755: done (0 left) 2024-09-11T03:44:57.938 INFO:tasks.rados.rados.2.smithi175.stdout:1756: write (excl) oid 43 current snap is 191 2024-09-11T03:44:57.938 INFO:tasks.rados.rados.2.smithi175.stdout:1756: seq_num 606 ranges {585042=792781,2095382=89660} 2024-09-11T03:44:57.945 INFO:tasks.rados.rados.2.smithi175.stdout:1756: writing smithi17557870-43 from 585042 to 1377823 tid 1 2024-09-11T03:44:57.949 INFO:tasks.rados.rados.2.smithi175.stdout:1756: writing smithi17557870-43 from 2095382 to 2185042 tid 2 2024-09-11T03:44:57.949 INFO:tasks.rados.rados.2.smithi175.stdout:1757: write oid 11 current snap is 191 2024-09-11T03:44:57.950 INFO:tasks.rados.rados.2.smithi175.stdout:1757: seq_num 607 ranges {601849=468030,1753864=447985} 2024-09-11T03:44:57.955 INFO:tasks.rados.rados.2.smithi175.stdout:1757: writing smithi17557870-11 from 601849 to 1069879 tid 1 2024-09-11T03:44:57.961 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:44:57.962 INFO:tasks.rados.rados.2.smithi175.stdout:1757: writing smithi17557870-11 from 1753864 to 2201849 tid 2 2024-09-11T03:44:57.962 INFO:tasks.rados.rados.2.smithi175.stdout:1758: delete oid 9 current snap is 191 2024-09-11T03:44:57.962 INFO:tasks.rados.rados.2.smithi175.stdout:1756: finishing write tid 1 to smithi17557870-43 2024-09-11T03:44:57.962 INFO:tasks.rados.rados.2.smithi175.stdout:1756: finishing write tid 2 to smithi17557870-43 2024-09-11T03:44:57.962 INFO:tasks.rados.rados.2.smithi175.stdout:1756: finishing write tid 3 to smithi17557870-43 2024-09-11T03:44:57.962 INFO:tasks.rados.rados.2.smithi175.stdout:1756: finishing write tid 4 to smithi17557870-43 2024-09-11T03:44:57.962 INFO:tasks.rados.rados.2.smithi175.stdout:1756: oid 43 updating version 0 to 404 2024-09-11T03:44:57.962 INFO:tasks.rados.rados.2.smithi175.stdout:1756: oid 43 updating version 404 to 406 2024-09-11T03:44:57.962 INFO:tasks.rados.rados.2.smithi175.stdout:1756: oid 43 version 406 is already newer than 405 2024-09-11T03:44:57.963 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 406 (ObjNum 606 snap 191 seq_num 606) dirty exists 2024-09-11T03:44:57.963 INFO:tasks.rados.rados.2.smithi175.stdout:1756: left oid 43 (ObjNum 606 snap 191 seq_num 606) 2024-09-11T03:44:57.963 INFO:tasks.rados.rados.2.smithi175.stdout:1757: finishing write tid 1 to smithi17557870-11 2024-09-11T03:44:57.964 INFO:tasks.rados.rados.2.smithi175.stdout:1756: done (2 left) 2024-09-11T03:44:57.964 INFO:tasks.rados.rados.2.smithi175.stdout:1758: done (1 left) 2024-09-11T03:44:57.964 INFO:tasks.rados.rados.2.smithi175.stdout:1759: read oid 42 snap -1 2024-09-11T03:44:57.964 INFO:tasks.rados.rados.2.smithi175.stdout:1759: expect (ObjNum 591 snap 183 seq_num 591) 2024-09-11T03:44:57.964 INFO:tasks.rados.rados.2.smithi175.stdout:1760: read oid 2 snap -1 2024-09-11T03:44:57.964 INFO:tasks.rados.rados.2.smithi175.stdout:1760: expect deleted 2024-09-11T03:44:57.964 INFO:tasks.rados.rados.2.smithi175.stdout:1761: write oid 18 current snap is 191 2024-09-11T03:44:57.964 INFO:tasks.rados.rados.2.smithi175.stdout:1761: seq_num 609 ranges {635463=734881,1987181=248282} 2024-09-11T03:44:57.971 INFO:tasks.rados.rados.2.smithi175.stdout:1761: writing smithi17557870-18 from 635463 to 1370344 tid 1 2024-09-11T03:44:57.976 INFO:tasks.rados.rados.2.smithi175.stdout:1761: writing smithi17557870-18 from 1987181 to 2235463 tid 2 2024-09-11T03:44:57.976 INFO:tasks.rados.rados.2.smithi175.stdout:1762: snap_create 2024-09-11T03:44:57.977 INFO:tasks.rados.rados.2.smithi175.stdout:1757: finishing write tid 2 to smithi17557870-11 2024-09-11T03:44:57.977 INFO:tasks.rados.rados.2.smithi175.stdout:1757: finishing write tid 3 to smithi17557870-11 2024-09-11T03:44:57.977 INFO:tasks.rados.rados.2.smithi175.stdout:1757: finishing write tid 4 to smithi17557870-11 2024-09-11T03:44:57.977 INFO:tasks.rados.rados.2.smithi175.stdout:1757: oid 11 updating version 0 to 567 2024-09-11T03:44:57.977 INFO:tasks.rados.rados.2.smithi175.stdout:1757: oid 11 updating version 567 to 568 2024-09-11T03:44:57.977 INFO:tasks.rados.rados.2.smithi175.stdout:1757: oid 11 version 568 is already newer than 566 2024-09-11T03:44:57.977 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 568 (ObjNum 607 snap 191 seq_num 607) dirty exists 2024-09-11T03:44:57.977 INFO:tasks.rados.rados.2.smithi175.stdout:1757: left oid 11 (ObjNum 607 snap 191 seq_num 607) 2024-09-11T03:44:57.977 INFO:tasks.rados.rados.2.smithi175.stdout:1759: expect (ObjNum 591 snap 183 seq_num 591) 2024-09-11T03:44:57.986 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:57.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:57.987 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:44:58.088 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:58.105 INFO:tasks.rados.rados.2.smithi175.stdout:1761: finishing write tid 1 to smithi17557870-18 2024-09-11T03:44:58.106 INFO:tasks.rados.rados.2.smithi175.stdout:1761: finishing write tid 2 to smithi17557870-18 2024-09-11T03:44:58.106 INFO:tasks.rados.rados.2.smithi175.stdout:1761: finishing write tid 3 to smithi17557870-18 2024-09-11T03:44:58.106 INFO:tasks.rados.rados.2.smithi175.stdout:1761: finishing write tid 4 to smithi17557870-18 2024-09-11T03:44:58.106 INFO:tasks.rados.rados.2.smithi175.stdout:1761: oid 18 updating version 0 to 465 2024-09-11T03:44:58.106 INFO:tasks.rados.rados.2.smithi175.stdout:1761: oid 18 updating version 465 to 466 2024-09-11T03:44:58.106 INFO:tasks.rados.rados.2.smithi175.stdout:1761: oid 18 updating version 466 to 467 2024-09-11T03:44:58.106 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 467 (ObjNum 609 snap 191 seq_num 609) dirty exists 2024-09-11T03:44:58.106 INFO:tasks.rados.rados.2.smithi175.stdout:1761: left oid 18 (ObjNum 609 snap 191 seq_num 609) 2024-09-11T03:44:58.109 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:58.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:58.110 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:58 smithi084 ceph-mon[27254]: pgmap v873: 57 pgs: 57 active+clean; 174 MiB data, 775 MiB used, 983 GiB / 983 GiB avail; 9.8 MiB/s rd, 260 KiB/s wr, 8 op/s 2024-09-11T03:44:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:58 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/209606353' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:58 smithi084 ceph-mon[27254]: osdmap e480: 12 total, 12 up, 11 in 2024-09-11T03:44:58.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:58.236 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:58.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:58.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:58.338 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:58.362 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:58.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:58 smithi121 ceph-mon[30729]: pgmap v873: 57 pgs: 57 active+clean; 174 MiB data, 775 MiB used, 983 GiB / 983 GiB avail; 9.8 MiB/s rd, 260 KiB/s wr, 8 op/s 2024-09-11T03:44:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:58 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/209606353' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:58 smithi121 ceph-mon[30729]: osdmap e480: 12 total, 12 up, 11 in 2024-09-11T03:44:58.464 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:44:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:58 smithi175 ceph-mon[30894]: pgmap v873: 57 pgs: 57 active+clean; 174 MiB data, 775 MiB used, 983 GiB / 983 GiB avail; 9.8 MiB/s rd, 260 KiB/s wr, 8 op/s 2024-09-11T03:44:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:58 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/209606353' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:44:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:58 smithi175 ceph-mon[30894]: osdmap e480: 12 total, 12 up, 11 in 2024-09-11T03:44:58.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:58.486 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:58.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:44:58.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:58.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:58.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:58.715 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:58.738 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:58.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:58.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:58.840 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:58.862 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:58.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:58.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:58.936 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:44:58.939 INFO:tasks.rados.rados.2.smithi175.stdout:1757: done (4 left) 2024-09-11T03:44:58.939 INFO:tasks.rados.rados.2.smithi175.stdout:1759: done (3 left) 2024-09-11T03:44:58.939 INFO:tasks.rados.rados.2.smithi175.stdout:1760: done (2 left) 2024-09-11T03:44:58.940 INFO:tasks.rados.rados.2.smithi175.stdout:1761: done (1 left) 2024-09-11T03:44:58.940 INFO:tasks.rados.rados.2.smithi175.stdout:1762: done (0 left) 2024-09-11T03:44:58.940 INFO:tasks.rados.rados.2.smithi175.stdout:1763: snap_create 2024-09-11T03:44:58.964 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:58.969 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:44:58.969 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:44:58.989 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:58.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:58.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:59.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:44:59.117 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:59.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:59.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:44:59.220 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:44:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:59.247 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:59.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:44:59.325 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:44:59.348 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:44:59.373 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:44:59.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:59.374 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:44:59.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:59 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3897637901' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:59.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:59 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3897637901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:59.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:44:59 smithi084 ceph-mon[27254]: osdmap e481: 12 total, 12 up, 11 in 2024-09-11T03:44:59.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:59 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3897637901' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:59.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:59 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3897637901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:59.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:44:59 smithi121 ceph-mon[30729]: osdmap e481: 12 total, 12 up, 11 in 2024-09-11T03:44:59.474 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:59.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:59 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3897637901' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:44:59.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:59 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3897637901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:44:59.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:44:59 smithi175 ceph-mon[30894]: osdmap e481: 12 total, 12 up, 11 in 2024-09-11T03:44:59.496 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:59.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:59.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:59.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:44:59.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:59.622 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:59.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:44:59.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:44:59.744 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:44:59.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:59.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:44:59.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:44:59.872 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:44:59.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:44:59.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:44:59.876 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T03:44:59.944 INFO:tasks.rados.rados.2.smithi175.stdout:1763: done (0 left) 2024-09-11T03:44:59.944 INFO:tasks.rados.rados.2.smithi175.stdout:1764: snap_create 2024-09-11T03:44:59.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:00.000 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:00.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:00.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:00.103 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:00.129 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:00.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:00 smithi084 ceph-mon[27254]: pgmap v876: 57 pgs: 57 active+clean; 151 MiB data, 733 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 528 KiB/s wr, 12 op/s 2024-09-11T03:45:00.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:00 smithi084 ceph-mon[27254]: osdmap e482: 12 total, 12 up, 11 in 2024-09-11T03:45:00.230 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:00.256 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:00.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:00.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:00.358 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:00.384 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:00.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:00.385 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:00 smithi121 ceph-mon[30729]: pgmap v876: 57 pgs: 57 active+clean; 151 MiB data, 733 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 528 KiB/s wr, 12 op/s 2024-09-11T03:45:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:00 smithi121 ceph-mon[30729]: osdmap e482: 12 total, 12 up, 11 in 2024-09-11T03:45:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:00 smithi175 ceph-mon[30894]: pgmap v876: 57 pgs: 57 active+clean; 151 MiB data, 733 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 528 KiB/s wr, 12 op/s 2024-09-11T03:45:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:00 smithi175 ceph-mon[30894]: osdmap e482: 12 total, 12 up, 11 in 2024-09-11T03:45:00.478 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:45:00.485 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:00.541 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:00.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:00.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:00.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:00.669 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:00.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:00.771 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:00.793 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:00.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:00.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:00.921 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:00.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:00.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:00.939 INFO:teuthology.orchestra.run.smithi084.stderr:set 1.16 pg_upmap mapping to [11,2,0] 2024-09-11T03:45:00.945 INFO:tasks.rados.rados.2.smithi175.stdout:1764: done (0 left) 2024-09-11T03:45:00.945 INFO:tasks.rados.rados.2.smithi175.stdout:1765: write (excl) oid 11 current snap is 194 2024-09-11T03:45:00.945 INFO:tasks.rados.rados.2.smithi175.stdout:1765: seq_num 610 ranges {652270=410130,1529310=493867} 2024-09-11T03:45:00.947 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:00.951 INFO:tasks.rados.rados.2.smithi175.stdout:1765: writing smithi17557870-11 from 652270 to 1062400 tid 1 2024-09-11T03:45:00.956 INFO:tasks.rados.rados.2.smithi175.stdout:1765: writing smithi17557870-11 from 1529310 to 2023177 tid 2 2024-09-11T03:45:00.956 INFO:tasks.rados.rados.2.smithi175.stdout:1766: write (excl) oid 3 current snap is 194 2024-09-11T03:45:00.956 INFO:tasks.rados.rados.2.smithi175.stdout:1766: seq_num 611 ranges {669077=485379,1587792=681285} 2024-09-11T03:45:00.962 INFO:tasks.rados.rados.2.smithi175.stdout:1766: writing smithi17557870-3 from 669077 to 1154456 tid 1 2024-09-11T03:45:00.970 INFO:tasks.rados.rados.2.smithi175.stdout:1766: writing smithi17557870-3 from 1587792 to 2269077 tid 2 2024-09-11T03:45:00.970 INFO:tasks.rados.rados.2.smithi175.stdout:1767: read oid 31 snap -1 2024-09-11T03:45:00.970 INFO:tasks.rados.rados.2.smithi175.stdout:1767: expect (ObjNum 495 snap 143 seq_num 495) 2024-09-11T03:45:00.970 INFO:tasks.rados.rados.2.smithi175.stdout:1768: rollback oid 17 current snap is 194 2024-09-11T03:45:00.970 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 17 to 175 2024-09-11T03:45:00.970 INFO:tasks.rados.rados.2.smithi175.stdout:1765: finishing write tid 1 to smithi17557870-11 2024-09-11T03:45:00.970 INFO:tasks.rados.rados.2.smithi175.stdout:1769: rollback oid 44 current snap is 194 2024-09-11T03:45:00.970 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 44 to 193 2024-09-11T03:45:00.970 INFO:tasks.rados.rados.2.smithi175.stdout:1765: finishing write tid 2 to smithi17557870-11 2024-09-11T03:45:00.970 INFO:tasks.rados.rados.2.smithi175.stdout:1765: finishing write tid 3 to smithi17557870-11 2024-09-11T03:45:00.970 INFO:tasks.rados.rados.2.smithi175.stdout:1765: finishing write tid 4 to smithi17557870-11 2024-09-11T03:45:00.970 INFO:tasks.rados.rados.2.smithi175.stdout:1765: oid 11 updating version 0 to 570 2024-09-11T03:45:00.971 INFO:tasks.rados.rados.2.smithi175.stdout:1765: oid 11 updating version 570 to 571 2024-09-11T03:45:00.971 INFO:tasks.rados.rados.2.smithi175.stdout:1765: oid 11 version 571 is already newer than 569 2024-09-11T03:45:00.971 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 571 (ObjNum 610 snap 194 seq_num 610) dirty exists 2024-09-11T03:45:00.971 INFO:tasks.rados.rados.2.smithi175.stdout:1765: left oid 11 (ObjNum 610 snap 194 seq_num 610) 2024-09-11T03:45:00.971 INFO:tasks.rados.rados.2.smithi175.stdout:1766: finishing write tid 1 to smithi17557870-3 2024-09-11T03:45:00.971 INFO:tasks.rados.rados.2.smithi175.stdout:1765: done (4 left) 2024-09-11T03:45:00.971 INFO:tasks.rados.rados.2.smithi175.stdout:1770: read oid 43 snap 186 2024-09-11T03:45:00.971 INFO:tasks.rados.rados.2.smithi175.stdout:1770: expect (ObjNum 574 snap 175 seq_num 574) 2024-09-11T03:45:00.971 INFO:tasks.rados.rados.2.smithi175.stdout:1771: copy_from oid 27 from oid 2 current snap is 194 2024-09-11T03:45:00.971 INFO:tasks.rados.rados.2.smithi175.stdout:1772: read oid 48 snap 192 2024-09-11T03:45:00.971 INFO:tasks.rados.rados.2.smithi175.stdout:1772: expect (ObjNum 562 snap 171 seq_num 562) 2024-09-11T03:45:00.971 INFO:tasks.rados.rados.2.smithi175.stdout:1773: copy_from oid 10 from oid 12 current snap is 194 2024-09-11T03:45:00.971 INFO:tasks.rados.rados.2.smithi175.stdout:1774: read oid 37 snap 192 2024-09-11T03:45:00.971 INFO:tasks.rados.rados.2.smithi175.stdout:1774: expect deleted 2024-09-11T03:45:00.972 INFO:tasks.rados.rados.2.smithi175.stdout:1775: rollback oid 15 current snap is 194 2024-09-11T03:45:00.972 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 15 to 191 2024-09-11T03:45:00.972 INFO:tasks.rados.rados.2.smithi175.stdout:1776: read oid 16 snap -1 2024-09-11T03:45:00.972 INFO:tasks.rados.rados.2.smithi175.stdout:1776: expect (ObjNum 597 snap 186 seq_num 597) 2024-09-11T03:45:00.972 INFO:tasks.rados.rados.2.smithi175.stdout:1777: rollback oid 26 current snap is 194 2024-09-11T03:45:00.972 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 26 to 191 2024-09-11T03:45:00.972 INFO:tasks.rados.rados.2.smithi175.stdout:1778: rollback oid 6 current snap is 194 2024-09-11T03:45:00.972 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 6 to 189 2024-09-11T03:45:00.972 INFO:tasks.rados.rados.2.smithi175.stdout:1779: snap_create 2024-09-11T03:45:00.972 INFO:tasks.rados.rados.2.smithi175.stdout:1768: finishing rollback tid 1 to smithi17557870-17 2024-09-11T03:45:00.972 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T03:45:00.973 INFO:tasks.rados.rados.2.smithi175.stdout:1769: finishing rollback tid 0 to smithi17557870-44 2024-09-11T03:45:00.973 INFO:tasks.rados.rados.2.smithi175.stdout:1773: finishing copy_from to smithi17557870-10 2024-09-11T03:45:00.973 INFO:tasks.rados.rados.2.smithi175.stdout:1773: got expected ENOENT (src dne) 2024-09-11T03:45:00.973 INFO:tasks.rados.rados.2.smithi175.stdout:1773: finishing copy_from racing read to smithi17557870-10 2024-09-11T03:45:00.973 INFO:tasks.rados.rados.2.smithi175.stdout:1769: finishing rollback tid 1 to smithi17557870-44 2024-09-11T03:45:00.973 INFO:tasks.rados.rados.2.smithi175.stdout:1769: finishing rollback tid 2 to smithi17557870-44 2024-09-11T03:45:00.973 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 574 (ObjNum 598 snap 186 seq_num 598) dirty exists 2024-09-11T03:45:00.973 INFO:tasks.rados.rados.2.smithi175.stdout:1775: finishing rollback tid 0 to smithi17557870-15 2024-09-11T03:45:00.974 INFO:tasks.rados.rados.2.smithi175.stdout:1778: finishing rollback tid 0 to smithi17557870-6 2024-09-11T03:45:00.974 INFO:tasks.rados.rados.2.smithi175.stdout:1775: finishing rollback tid 1 to smithi17557870-15 2024-09-11T03:45:00.975 INFO:tasks.rados.rados.2.smithi175.stdout:1766: finishing write tid 2 to smithi17557870-3 2024-09-11T03:45:00.975 INFO:tasks.rados.rados.2.smithi175.stdout:1766: finishing write tid 3 to smithi17557870-3 2024-09-11T03:45:00.975 INFO:tasks.rados.rados.2.smithi175.stdout:1775: finishing rollback tid 2 to smithi17557870-15 2024-09-11T03:45:00.975 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 227 (ObjNum 566 snap 172 seq_num 566) dirty exists 2024-09-11T03:45:00.975 INFO:tasks.rados.rados.2.smithi175.stdout:1778: finishing rollback tid 1 to smithi17557870-6 2024-09-11T03:45:00.975 INFO:tasks.rados.rados.2.smithi175.stdout:1766: finishing write tid 4 to smithi17557870-3 2024-09-11T03:45:00.975 INFO:tasks.rados.rados.2.smithi175.stdout:1766: oid 3 updating version 0 to 665 2024-09-11T03:45:00.975 INFO:tasks.rados.rados.2.smithi175.stdout:1766: oid 3 updating version 665 to 666 2024-09-11T03:45:00.975 INFO:tasks.rados.rados.2.smithi175.stdout:1766: oid 3 version 666 is already newer than 664 2024-09-11T03:45:00.975 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 666 (ObjNum 611 snap 194 seq_num 611) dirty exists 2024-09-11T03:45:00.975 INFO:tasks.rados.rados.2.smithi175.stdout:1766: left oid 3 (ObjNum 611 snap 194 seq_num 611) 2024-09-11T03:45:00.975 INFO:tasks.rados.rados.2.smithi175.stdout:1777: finishing rollback tid 0 to smithi17557870-26 2024-09-11T03:45:00.975 INFO:tasks.rados.rados.2.smithi175.stdout:1778: finishing rollback tid 2 to smithi17557870-6 2024-09-11T03:45:00.975 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 421 (ObjNum 528 snap 156 seq_num 528) dirty exists 2024-09-11T03:45:00.976 INFO:tasks.rados.rados.2.smithi175.stdout:1777: finishing rollback tid 1 to smithi17557870-26 2024-09-11T03:45:00.976 INFO:tasks.rados.rados.2.smithi175.stdout:1777: finishing rollback tid 2 to smithi17557870-26 2024-09-11T03:45:00.976 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 359 (ObjNum 595 snap 185 seq_num 595) dirty exists 2024-09-11T03:45:00.977 INFO:tasks.rados.rados.2.smithi175.stdout:1767: expect (ObjNum 495 snap 143 seq_num 495) 2024-09-11T03:45:01.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:01.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:01.047 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:01.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:01.055 INFO:tasks.rados.rados.2.smithi175.stdout:1771: finishing copy_from to smithi17557870-27 2024-09-11T03:45:01.056 INFO:tasks.rados.rados.2.smithi175.stdout:1771: got expected ENOENT (src dne) 2024-09-11T03:45:01.056 INFO:tasks.rados.rados.2.smithi175.stdout:1771: finishing copy_from racing read to smithi17557870-27 2024-09-11T03:45:01.056 INFO:tasks.rados.rados.2.smithi175.stdout:1772: expect (ObjNum 562 snap 171 seq_num 562) 2024-09-11T03:45:01.148 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:01.171 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:01.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:01.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:01.197 INFO:tasks.rados.rados.2.smithi175.stdout:1776: expect (ObjNum 597 snap 186 seq_num 597) 2024-09-11T03:45:01.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3861752359' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.16", "id": [11, 2, 0]} : dispatch 2024-09-11T03:45:01.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3861752359' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.16", "id": [11, 2, 0]}]': finished 2024-09-11T03:45:01.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:01 smithi084 ceph-mon[27254]: osdmap e483: 12 total, 12 up, 11 in 2024-09-11T03:45:01.273 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:01.280 INFO:tasks.rados.rados.2.smithi175.stdout:1770: expect (ObjNum 574 snap 175 seq_num 574) 2024-09-11T03:45:01.293 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:01.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:01.294 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:01.396 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:01.418 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:01.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3861752359' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.16", "id": [11, 2, 0]} : dispatch 2024-09-11T03:45:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3861752359' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.16", "id": [11, 2, 0]}]': finished 2024-09-11T03:45:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:01 smithi121 ceph-mon[30729]: osdmap e483: 12 total, 12 up, 11 in 2024-09-11T03:45:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3861752359' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.16", "id": [11, 2, 0]} : dispatch 2024-09-11T03:45:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3861752359' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.16", "id": [11, 2, 0]}]': finished 2024-09-11T03:45:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:01 smithi175 ceph-mon[30894]: osdmap e483: 12 total, 12 up, 11 in 2024-09-11T03:45:01.519 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:01.540 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:01.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:01.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:01.642 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:01.664 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:01.664 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:01.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:01.791 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:01.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:01.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:01.863 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:45:01.863 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:45:01.892 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:01.915 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:01.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:01.938 INFO:tasks.rados.rados.2.smithi175.stdout:1766: done (13 left) 2024-09-11T03:45:01.938 INFO:tasks.rados.rados.2.smithi175.stdout:1767: done (12 left) 2024-09-11T03:45:01.939 INFO:tasks.rados.rados.2.smithi175.stdout:1768: done (11 left) 2024-09-11T03:45:01.939 INFO:tasks.rados.rados.2.smithi175.stdout:1769: done (10 left) 2024-09-11T03:45:01.939 INFO:tasks.rados.rados.2.smithi175.stdout:1770: done (9 left) 2024-09-11T03:45:01.939 INFO:tasks.rados.rados.2.smithi175.stdout:1771: done (8 left) 2024-09-11T03:45:01.939 INFO:tasks.rados.rados.2.smithi175.stdout:1772: done (7 left) 2024-09-11T03:45:01.939 INFO:tasks.rados.rados.2.smithi175.stdout:1773: done (6 left) 2024-09-11T03:45:01.939 INFO:tasks.rados.rados.2.smithi175.stdout:1774: done (5 left) 2024-09-11T03:45:01.939 INFO:tasks.rados.rados.2.smithi175.stdout:1775: done (4 left) 2024-09-11T03:45:01.939 INFO:tasks.rados.rados.2.smithi175.stdout:1776: done (3 left) 2024-09-11T03:45:01.939 INFO:tasks.rados.rados.2.smithi175.stdout:1777: done (2 left) 2024-09-11T03:45:01.939 INFO:tasks.rados.rados.2.smithi175.stdout:1778: done (1 left) 2024-09-11T03:45:01.939 INFO:tasks.rados.rados.2.smithi175.stdout:1779: done (0 left) 2024-09-11T03:45:01.940 INFO:tasks.rados.rados.2.smithi175.stdout:1780: rollback oid 38 current snap is 195 2024-09-11T03:45:01.940 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 38 to 189 2024-09-11T03:45:01.940 INFO:tasks.rados.rados.2.smithi175.stdout:1781: write (excl) oid 5 current snap is 195 2024-09-11T03:45:01.940 INFO:tasks.rados.rados.2.smithi175.stdout:1781: seq_num 612 ranges {685884=560628,2046274=239610} 2024-09-11T03:45:01.945 INFO:tasks.rados.rados.2.smithi175.stdout:1781: writing smithi17557870-5 from 685884 to 1246512 tid 1 2024-09-11T03:45:01.951 INFO:tasks.rados.rados.2.smithi175.stdout:1781: writing smithi17557870-5 from 2046274 to 2285884 tid 2 2024-09-11T03:45:01.951 INFO:tasks.rados.rados.2.smithi175.stdout:1782: rollback oid 41 current snap is 195 2024-09-11T03:45:01.952 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 41 to 192 2024-09-11T03:45:01.952 INFO:tasks.rados.rados.2.smithi175.stdout:1780: finishing rollback tid 0 to smithi17557870-38 2024-09-11T03:45:01.952 INFO:tasks.rados.rados.2.smithi175.stdout:1780: finishing rollback tid 1 to smithi17557870-38 2024-09-11T03:45:01.952 INFO:tasks.rados.rados.2.smithi175.stdout:1780: finishing rollback tid 2 to smithi17557870-38 2024-09-11T03:45:01.952 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 652 (ObjNum 565 snap 172 seq_num 565) dirty exists 2024-09-11T03:45:01.952 INFO:tasks.rados.rados.2.smithi175.stdout:1781: finishing write tid 1 to smithi17557870-5 2024-09-11T03:45:01.952 INFO:tasks.rados.rados.2.smithi175.stdout:1780: done (2 left) 2024-09-11T03:45:01.952 INFO:tasks.rados.rados.2.smithi175.stdout:1783: snap_remove snap 186 2024-09-11T03:45:02.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:02.042 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:02.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:02.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:02.144 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:02.165 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:02.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:02.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:02.267 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:02.288 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:02.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:02.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:02.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:02 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:45:02.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:02 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:45:02.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:02 smithi084 ceph-mon[27254]: pgmap v879: 57 pgs: 57 active+clean; 151 MiB data, 728 MiB used, 983 GiB / 983 GiB avail; 5.6 MiB/s rd, 268 KiB/s wr, 5 op/s 2024-09-11T03:45:02.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:02 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:02.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:02 smithi084 ceph-mon[27254]: osdmap e484: 12 total, 12 up, 11 in 2024-09-11T03:45:02.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:02.417 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:02.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:02.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:02 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:45:02.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:02 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:45:02.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:02 smithi121 ceph-mon[30729]: pgmap v879: 57 pgs: 57 active+clean; 151 MiB data, 728 MiB used, 983 GiB / 983 GiB avail; 5.6 MiB/s rd, 268 KiB/s wr, 5 op/s 2024-09-11T03:45:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:02 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:02 smithi121 ceph-mon[30729]: osdmap e484: 12 total, 12 up, 11 in 2024-09-11T03:45:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:02 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:45:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:02 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:45:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:02 smithi175 ceph-mon[30894]: pgmap v879: 57 pgs: 57 active+clean; 151 MiB data, 728 MiB used, 983 GiB / 983 GiB avail; 5.6 MiB/s rd, 268 KiB/s wr, 5 op/s 2024-09-11T03:45:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:02 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:02 smithi175 ceph-mon[30894]: osdmap e484: 12 total, 12 up, 11 in 2024-09-11T03:45:02.519 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:02.544 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:02.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:02.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:02.647 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:02.671 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:02.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:02.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:02.729 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T03:45:02.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:02.783 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:02.799 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:02.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:02.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:02.901 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:02.927 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:02.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:02.928 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:02.941 INFO:tasks.rados.rados.2.smithi175.stdout:1781: finishing write tid 2 to smithi17557870-5 2024-09-11T03:45:02.941 INFO:tasks.rados.rados.2.smithi175.stdout:1781: finishing write tid 3 to smithi17557870-5 2024-09-11T03:45:02.941 INFO:tasks.rados.rados.2.smithi175.stdout:1781: finishing write tid 4 to smithi17557870-5 2024-09-11T03:45:02.941 INFO:tasks.rados.rados.2.smithi175.stdout:1781: oid 5 updating version 0 to 273 2024-09-11T03:45:02.941 INFO:tasks.rados.rados.2.smithi175.stdout:1781: oid 5 version 273 is already newer than 272 2024-09-11T03:45:02.941 INFO:tasks.rados.rados.2.smithi175.stdout:1781: oid 5 updating version 273 to 274 2024-09-11T03:45:02.942 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 274 (ObjNum 612 snap 195 seq_num 612) dirty exists 2024-09-11T03:45:02.942 INFO:tasks.rados.rados.2.smithi175.stdout:1781: left oid 5 (ObjNum 612 snap 195 seq_num 612) 2024-09-11T03:45:02.942 INFO:tasks.rados.rados.2.smithi175.stdout:1782: finishing rollback tid 0 to smithi17557870-41 2024-09-11T03:45:02.942 INFO:tasks.rados.rados.2.smithi175.stdout:1782: finishing rollback tid 1 to smithi17557870-41 2024-09-11T03:45:02.942 INFO:tasks.rados.rados.2.smithi175.stdout:1782: finishing rollback tid 2 to smithi17557870-41 2024-09-11T03:45:02.942 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 511 (ObjNum 566 snap 172 seq_num 566) dirty exists 2024-09-11T03:45:02.942 INFO:tasks.rados.rados.2.smithi175.stdout:1781: done (2 left) 2024-09-11T03:45:02.942 INFO:tasks.rados.rados.2.smithi175.stdout:1782: done (1 left) 2024-09-11T03:45:02.942 INFO:tasks.rados.rados.2.smithi175.stdout:1783: done (0 left) 2024-09-11T03:45:02.942 INFO:tasks.rados.rados.2.smithi175.stdout:1784: snap_create 2024-09-11T03:45:02.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:45:03.030 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:03.052 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:03.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:03.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:03.154 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:03.176 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:03.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:03.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:03.297 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:45:03.297 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:45:03.304 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:03.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:03.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:03.313 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:45:03.313 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:45:03.332 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:03.334 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:45:03.334 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:45:03.352 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:45:03.352 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:45:03.372 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:45:03.372 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:45:03.389 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:45:03.389 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:45:03.406 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:03.406 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:45:03.407 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:45:03.423 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:45:03.423 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:45:03.432 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:03.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:03.433 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:03.439 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:45:03.440 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:45:03.456 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:45:03.457 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:45:03.473 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:45:03.473 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:45:03.491 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:45:03.491 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:45:03.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:03.560 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:03.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:03.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:03.662 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:03.688 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:03.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:03.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:03.790 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:03.812 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:03.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:03.914 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:03.941 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:03.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:03.942 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:03.944 INFO:tasks.rados.rados.2.smithi175.stdout:1784: done (0 left) 2024-09-11T03:45:03.944 INFO:tasks.rados.rados.2.smithi175.stdout:1785: write oid 35 current snap is 196 2024-09-11T03:45:03.944 INFO:tasks.rados.rados.2.smithi175.stdout:1785: seq_num 613 ranges {702691=635877,2104756=197935} 2024-09-11T03:45:03.951 INFO:tasks.rados.rados.2.smithi175.stdout:1785: writing smithi17557870-35 from 702691 to 1338568 tid 1 2024-09-11T03:45:03.957 INFO:tasks.rados.rados.2.smithi175.stdout:1785: writing smithi17557870-35 from 2104756 to 2302691 tid 2 2024-09-11T03:45:03.957 INFO:tasks.rados.rados.2.smithi175.stdout:1786: copy_from oid 14 from oid 25 current snap is 196 2024-09-11T03:45:03.957 INFO:tasks.rados.rados.2.smithi175.stdout:1787: snap_create 2024-09-11T03:45:03.959 INFO:tasks.rados.rados.2.smithi175.stdout:1785: finishing write tid 1 to smithi17557870-35 2024-09-11T03:45:03.959 INFO:tasks.rados.rados.2.smithi175.stdout:1785: finishing write tid 2 to smithi17557870-35 2024-09-11T03:45:03.959 INFO:tasks.rados.rados.2.smithi175.stdout:1785: finishing write tid 3 to smithi17557870-35 2024-09-11T03:45:03.959 INFO:tasks.rados.rados.2.smithi175.stdout:1785: finishing write tid 4 to smithi17557870-35 2024-09-11T03:45:03.960 INFO:tasks.rados.rados.2.smithi175.stdout:1785: oid 35 updating version 0 to 540 2024-09-11T03:45:03.960 INFO:tasks.rados.rados.2.smithi175.stdout:1785: oid 35 version 540 is already newer than 539 2024-09-11T03:45:03.960 INFO:tasks.rados.rados.2.smithi175.stdout:1785: oid 35 updating version 540 to 541 2024-09-11T03:45:03.960 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 541 (ObjNum 613 snap 196 seq_num 613) dirty exists 2024-09-11T03:45:03.960 INFO:tasks.rados.rados.2.smithi175.stdout:1785: left oid 35 (ObjNum 613 snap 196 seq_num 613) 2024-09-11T03:45:03.969 INFO:tasks.rados.rados.2.smithi175.stdout:1786: finishing copy_from racing read to smithi17557870-14 2024-09-11T03:45:03.971 INFO:tasks.rados.rados.2.smithi175.stdout:1786: finishing copy_from to smithi17557870-14 2024-09-11T03:45:03.971 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 491 (ObjNum 496 snap 144 seq_num 496) dirty exists 2024-09-11T03:45:04.043 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:04.064 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:04.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:04.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:04.166 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:04.179 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:03 smithi084 ceph-mon[27254]: 1.16 scrub starts 2024-09-11T03:45:04.179 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:03 smithi084 ceph-mon[27254]: pgmap v882: 57 pgs: 57 active+clean; 155 MiB data, 725 MiB used, 983 GiB / 983 GiB avail; 20 MiB/s rd, 1.2 MiB/s wr, 23 op/s; 8.8 KiB/s, 0 objects/s recovering 2024-09-11T03:45:04.179 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:03 smithi084 ceph-mon[27254]: osdmap e485: 12 total, 12 up, 11 in 2024-09-11T03:45:04.179 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:03 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3770677155' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:04.192 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:04.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:04.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:03 smithi175 ceph-mon[30894]: 1.16 scrub starts 2024-09-11T03:45:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:03 smithi175 ceph-mon[30894]: pgmap v882: 57 pgs: 57 active+clean; 155 MiB data, 725 MiB used, 983 GiB / 983 GiB avail; 20 MiB/s rd, 1.2 MiB/s wr, 23 op/s; 8.8 KiB/s, 0 objects/s recovering 2024-09-11T03:45:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:03 smithi175 ceph-mon[30894]: osdmap e485: 12 total, 12 up, 11 in 2024-09-11T03:45:04.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:03 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3770677155' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:04.294 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:04.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:04.316 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:04.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:04.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:04.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:03 smithi121 ceph-mon[30729]: 1.16 scrub starts 2024-09-11T03:45:04.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:03 smithi121 ceph-mon[30729]: pgmap v882: 57 pgs: 57 active+clean; 155 MiB data, 725 MiB used, 983 GiB / 983 GiB avail; 20 MiB/s rd, 1.2 MiB/s wr, 23 op/s; 8.8 KiB/s, 0 objects/s recovering 2024-09-11T03:45:04.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:03 smithi121 ceph-mon[30729]: osdmap e485: 12 total, 12 up, 11 in 2024-09-11T03:45:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:03 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3770677155' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:04.443 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:04.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:04.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:04.446 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:45:04.446 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:45:04.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:45:04.446 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:45:04.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.447 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:640 2.a e131/95 2:5014e305:::smithi17557870-46:head v 131'5, mlcod=100'2)", 2024-09-11T03:45:04.447 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:45:04.447 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 356.32386816000002, 2024-09-11T03:45:04.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010809757999999999, 2024-09-11T03:45:04.447 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.447 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.447 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:04.447 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:45:04.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:45:04.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:45:04.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6100000000000002e-07 2024-09-11T03:45:04.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122019+0000", 2024-09-11T03:45:04.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0036004240000000001 2024-09-11T03:45:04.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122067+0000", 2024-09-11T03:45:04.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7589000000000001e-05 2024-09-11T03:45:04.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122077+0000", 2024-09-11T03:45:04.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0036e-05 2024-09-11T03:45:04.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122133+0000", 2024-09-11T03:45:04.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6350000000000001e-05 2024-09-11T03:45:04.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122152+0000", 2024-09-11T03:45:04.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8595999999999999e-05 2024-09-11T03:45:04.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:04.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129179+0000", 2024-09-11T03:45:04.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0070268079999999998 2024-09-11T03:45:04.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129228+0000", 2024-09-11T03:45:04.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9594e-05 2024-09-11T03:45:04.451 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.451 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.451 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.451 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:649 2.6 e131/95 2:6dbd1e81:::smithi17557870-7:head v 131'25, mlcod=123'21)", 2024-09-11T03:45:04.451 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:08.118731+0000", 2024-09-11T03:45:04.451 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 356.32355635800002, 2024-09-11T03:45:04.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01067091, 2024-09-11T03:45:04.452 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.452 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.452 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:04.452 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118731+0000", 2024-09-11T03:45:04.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118731+0000", 2024-09-11T03:45:04.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118731+0000", 2024-09-11T03:45:04.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4399999999999998e-07 2024-09-11T03:45:04.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.121866+0000", 2024-09-11T03:45:04.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031341720000000002 2024-09-11T03:45:04.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.121900+0000", 2024-09-11T03:45:04.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4266e-05 2024-09-11T03:45:04.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.121914+0000", 2024-09-11T03:45:04.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3668e-05 2024-09-11T03:45:04.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122002+0000", 2024-09-11T03:45:04.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7995e-05 2024-09-11T03:45:04.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122029+0000", 2024-09-11T03:45:04.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7199000000000002e-05 2024-09-11T03:45:04.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:04.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129354+0000", 2024-09-11T03:45:04.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073255899999999999 2024-09-11T03:45:04.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129401+0000", 2024-09-11T03:45:04.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7076000000000001e-05 2024-09-11T03:45:04.456 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.456 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.456 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.456 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1524 2.6 e212/95 2:6dbd1e81:::smithi17557870-7:head v 212'123, mlcod=204'119)", 2024-09-11T03:45:04.457 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:28.927804+0000", 2024-09-11T03:45:04.457 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 275.51448277999998, 2024-09-11T03:45:04.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087752009999999998, 2024-09-11T03:45:04.457 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.457 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.457 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:04.457 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927804+0000", 2024-09-11T03:45:04.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927804+0000", 2024-09-11T03:45:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927804+0000", 2024-09-11T03:45:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8799999999999998e-07 2024-09-11T03:45:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929502+0000", 2024-09-11T03:45:04.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016974690000000001 2024-09-11T03:45:04.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929523+0000", 2024-09-11T03:45:04.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1373999999999999e-05 2024-09-11T03:45:04.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929543+0000", 2024-09-11T03:45:04.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9950999999999999e-05 2024-09-11T03:45:04.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929633+0000", 2024-09-11T03:45:04.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0158e-05 2024-09-11T03:45:04.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929657+0000", 2024-09-11T03:45:04.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3737999999999999e-05 2024-09-11T03:45:04.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:04.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.936522+0000", 2024-09-11T03:45:04.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068650589999999997 2024-09-11T03:45:04.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.936579+0000", 2024-09-11T03:45:04.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7163999999999997e-05 2024-09-11T03:45:04.461 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.462 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.462 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.462 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1589 2.7 e218/191 2:e77ce595:::smithi17557870-31:head v 218'174, mlcod=218'168)", 2024-09-11T03:45:04.462 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:35.116398+0000", 2024-09-11T03:45:04.462 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 269.32588844899999, 2024-09-11T03:45:04.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0089646050000000005, 2024-09-11T03:45:04.462 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.462 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.462 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:04.462 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116398+0000", 2024-09-11T03:45:04.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116398+0000", 2024-09-11T03:45:04.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116400+0000", 2024-09-11T03:45:04.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6109999999999999e-06 2024-09-11T03:45:04.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119261+0000", 2024-09-11T03:45:04.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028605309999999999 2024-09-11T03:45:04.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119281+0000", 2024-09-11T03:45:04.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0491000000000001e-05 2024-09-11T03:45:04.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119292+0000", 2024-09-11T03:45:04.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0759e-05 2024-09-11T03:45:04.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119350+0000", 2024-09-11T03:45:04.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8588999999999998e-05 2024-09-11T03:45:04.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119369+0000", 2024-09-11T03:45:04.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8227e-05 2024-09-11T03:45:04.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:04.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125320+0000", 2024-09-11T03:45:04.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059510409999999998 2024-09-11T03:45:04.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125363+0000", 2024-09-11T03:45:04.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3356000000000003e-05 2024-09-11T03:45:04.467 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.467 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.467 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.467 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1816 2.d e243/95 2:b7bdfe52:::smithi17557870-10:head v 243'237, mlcod=243'233)", 2024-09-11T03:45:04.467 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:59.922567+0000", 2024-09-11T03:45:04.467 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 244.51972009799999, 2024-09-11T03:45:04.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094601100000000007, 2024-09-11T03:45:04.467 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.468 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.468 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:04.468 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.922567+0000", 2024-09-11T03:45:04.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.922567+0000", 2024-09-11T03:45:04.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.922568+0000", 2024-09-11T03:45:04.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.345e-06 2024-09-11T03:45:04.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929634+0000", 2024-09-11T03:45:04.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0070657130000000004 2024-09-11T03:45:04.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929654+0000", 2024-09-11T03:45:04.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9972000000000001e-05 2024-09-11T03:45:04.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929663+0000", 2024-09-11T03:45:04.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3770000000000002e-06 2024-09-11T03:45:04.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929731+0000", 2024-09-11T03:45:04.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8095000000000004e-05 2024-09-11T03:45:04.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929759+0000", 2024-09-11T03:45:04.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7248999999999999e-05 2024-09-11T03:45:04.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:04.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.931991+0000", 2024-09-11T03:45:04.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022320679999999998 2024-09-11T03:45:04.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.932027+0000", 2024-09-11T03:45:04.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6291000000000002e-05 2024-09-11T03:45:04.471 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.471 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.471 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.472 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.11.0:202 2.a e279/95 2:5014e305:::smithi17557870-46:5c v 279'82, mlcod=276'78)", 2024-09-11T03:45:04.472 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:35.868642+0000", 2024-09-11T03:45:04.472 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 208.573644919, 2024-09-11T03:45:04.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084062019999999998, 2024-09-11T03:45:04.472 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.472 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.472 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:04.472 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.868642+0000", 2024-09-11T03:45:04.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.868642+0000", 2024-09-11T03:45:04.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.868642+0000", 2024-09-11T03:45:04.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.72e-07 2024-09-11T03:45:04.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.868646+0000", 2024-09-11T03:45:04.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.309e-06 2024-09-11T03:45:04.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.868654+0000", 2024-09-11T03:45:04.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7169999999999997e-06 2024-09-11T03:45:04.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.868660+0000", 2024-09-11T03:45:04.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5350000000000004e-06 2024-09-11T03:45:04.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.876082+0000", 2024-09-11T03:45:04.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074219009999999998 2024-09-11T03:45:04.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.876104+0000", 2024-09-11T03:45:04.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2568000000000001e-05 2024-09-11T03:45:04.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:04.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.876990+0000", 2024-09-11T03:45:04.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00088600599999999995 2024-09-11T03:45:04.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.877048+0000", 2024-09-11T03:45:04.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7893999999999998e-05 2024-09-11T03:45:04.476 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.476 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.476 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.476 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2431 2.12 2:4e5019e5:::smithi17557870-40:head [copy-from ver 279] snapc 66=[66,65,64,63,61,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-09-11T03:45:04.476 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:51.877192+0000", 2024-09-11T03:45:04.476 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 192.56509480599999, 2024-09-11T03:45:04.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013330089, 2024-09-11T03:45:04.476 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.477 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.477 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:04.477 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:04.477 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:04.477 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:04.477 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2431 2024-09-11T03:45:04.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.477 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877192+0000", 2024-09-11T03:45:04.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877192+0000", 2024-09-11T03:45:04.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877194+0000", 2024-09-11T03:45:04.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.767e-06 2024-09-11T03:45:04.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877202+0000", 2024-09-11T03:45:04.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0770000000000008e-06 2024-09-11T03:45:04.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877220+0000", 2024-09-11T03:45:04.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8561000000000001e-05 2024-09-11T03:45:04.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877230+0000", 2024-09-11T03:45:04.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2820000000000002e-06 2024-09-11T03:45:04.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877290+0000", 2024-09-11T03:45:04.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0025999999999998e-05 2024-09-11T03:45:04.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877404+0000", 2024-09-11T03:45:04.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011425999999999999 2024-09-11T03:45:04.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,11", 2024-09-11T03:45:04.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.884630+0000", 2024-09-11T03:45:04.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072256769999999998 2024-09-11T03:45:04.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:45:04.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.888559+0000", 2024-09-11T03:45:04.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039288750000000001 2024-09-11T03:45:04.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:04.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.888615+0000", 2024-09-11T03:45:04.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6609e-05 2024-09-11T03:45:04.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:04.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.890459+0000", 2024-09-11T03:45:04.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001844116 2024-09-11T03:45:04.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:04.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.890494+0000", 2024-09-11T03:45:04.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4921e-05 2024-09-11T03:45:04.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.890522+0000", 2024-09-11T03:45:04.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7917999999999999e-05 2024-09-11T03:45:04.482 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.482 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.482 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.483 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2432 2.12 2:4e5019e5:::smithi17557870-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e295)", 2024-09-11T03:45:04.483 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:51.877251+0000", 2024-09-11T03:45:04.483 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 192.56503615400001, 2024-09-11T03:45:04.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010600994000000001, 2024-09-11T03:45:04.483 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.483 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.483 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:45:04.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:04.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:04.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:04.483 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2432 2024-09-11T03:45:04.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.483 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877251+0000", 2024-09-11T03:45:04.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877251+0000", 2024-09-11T03:45:04.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877251+0000", 2024-09-11T03:45:04.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3399999999999999e-07 2024-09-11T03:45:04.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877254+0000", 2024-09-11T03:45:04.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.379e-06 2024-09-11T03:45:04.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877262+0000", 2024-09-11T03:45:04.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0779999999999996e-06 2024-09-11T03:45:04.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877265+0000", 2024-09-11T03:45:04.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.641e-06 2024-09-11T03:45:04.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877539+0000", 2024-09-11T03:45:04.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027397000000000001 2024-09-11T03:45:04.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:04.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877571+0000", 2024-09-11T03:45:04.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2122999999999997e-05 2024-09-11T03:45:04.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.887709+0000", 2024-09-11T03:45:04.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010137922000000001 2024-09-11T03:45:04.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.887781+0000", 2024-09-11T03:45:04.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1384999999999995e-05 2024-09-11T03:45:04.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.887852+0000", 2024-09-11T03:45:04.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0962000000000003e-05 2024-09-11T03:45:04.487 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.487 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.487 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.488 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2434 2.9 e295/276 2:939cf94d:::smithi17557870-28:head v 295'353, mlcod=295'349)", 2024-09-11T03:45:04.488 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:52.055871+0000", 2024-09-11T03:45:04.488 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 192.38641599100001, 2024-09-11T03:45:04.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083876999999999997, 2024-09-11T03:45:04.488 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.488 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.488 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:04.488 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.055871+0000", 2024-09-11T03:45:04.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.055871+0000", 2024-09-11T03:45:04.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.055873+0000", 2024-09-11T03:45:04.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9539999999999998e-06 2024-09-11T03:45:04.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.058061+0000", 2024-09-11T03:45:04.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002188204 2024-09-11T03:45:04.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.058085+0000", 2024-09-11T03:45:04.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4450000000000001e-05 2024-09-11T03:45:04.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.058096+0000", 2024-09-11T03:45:04.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.096e-05 2024-09-11T03:45:04.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.058183+0000", 2024-09-11T03:45:04.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6235000000000004e-05 2024-09-11T03:45:04.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.058220+0000", 2024-09-11T03:45:04.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7651000000000003e-05 2024-09-11T03:45:04.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:04.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.064221+0000", 2024-09-11T03:45:04.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060009599999999996 2024-09-11T03:45:04.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.064259+0000", 2024-09-11T03:45:04.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7286000000000002e-05 2024-09-11T03:45:04.492 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.492 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.493 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.493 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2443 2.12 2:494ba932:::smithi17557870-43:head [copy-from ver 342] snapc 66=[65,64,63,61,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-11T03:45:04.493 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:52.884558+0000", 2024-09-11T03:45:04.493 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 191.55772843700001, 2024-09-11T03:45:04.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019937844999999999, 2024-09-11T03:45:04.493 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.493 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.493 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:04.493 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:04.493 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:04.493 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:04.493 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2443 2024-09-11T03:45:04.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.494 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884558+0000", 2024-09-11T03:45:04.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884558+0000", 2024-09-11T03:45:04.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884560+0000", 2024-09-11T03:45:04.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7e-06 2024-09-11T03:45:04.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884568+0000", 2024-09-11T03:45:04.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0940000000000008e-06 2024-09-11T03:45:04.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884584+0000", 2024-09-11T03:45:04.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6277e-05 2024-09-11T03:45:04.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884593+0000", 2024-09-11T03:45:04.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1810000000000001e-06 2024-09-11T03:45:04.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884660+0000", 2024-09-11T03:45:04.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7362999999999999e-05 2024-09-11T03:45:04.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884731+0000", 2024-09-11T03:45:04.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1316999999999995e-05 2024-09-11T03:45:04.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,11", 2024-09-11T03:45:04.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.895614+0000", 2024-09-11T03:45:04.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010882911 2024-09-11T03:45:04.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:45:04.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.900723+0000", 2024-09-11T03:45:04.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005109143 2024-09-11T03:45:04.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:04.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.902758+0000", 2024-09-11T03:45:04.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020342870000000001 2024-09-11T03:45:04.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:04.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.904417+0000", 2024-09-11T03:45:04.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001658875 2024-09-11T03:45:04.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:04.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.904454+0000", 2024-09-11T03:45:04.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7883999999999998e-05 2024-09-11T03:45:04.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.904496+0000", 2024-09-11T03:45:04.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1813000000000001e-05 2024-09-11T03:45:04.499 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.499 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.499 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.499 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2444 2.12 2:494ba932:::smithi17557870-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e296)", 2024-09-11T03:45:04.499 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:52.884615+0000", 2024-09-11T03:45:04.499 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 191.557671876, 2024-09-11T03:45:04.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015042823, 2024-09-11T03:45:04.499 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.500 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.500 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:45:04.500 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:04.500 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:04.500 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:04.500 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2444 2024-09-11T03:45:04.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.500 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884615+0000", 2024-09-11T03:45:04.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884615+0000", 2024-09-11T03:45:04.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884616+0000", 2024-09-11T03:45:04.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5499999999999998e-07 2024-09-11T03:45:04.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884618+0000", 2024-09-11T03:45:04.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.818e-06 2024-09-11T03:45:04.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884632+0000", 2024-09-11T03:45:04.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3219e-05 2024-09-11T03:45:04.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884639+0000", 2024-09-11T03:45:04.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9990000000000002e-06 2024-09-11T03:45:04.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884918+0000", 2024-09-11T03:45:04.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027906699999999998 2024-09-11T03:45:04.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:04.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884959+0000", 2024-09-11T03:45:04.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1844999999999997e-05 2024-09-11T03:45:04.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.899489+0000", 2024-09-11T03:45:04.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014529034999999999 2024-09-11T03:45:04.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.899577+0000", 2024-09-11T03:45:04.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8512999999999998e-05 2024-09-11T03:45:04.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.899658+0000", 2024-09-11T03:45:04.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0772000000000005e-05 2024-09-11T03:45:04.504 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.504 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.504 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.504 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2513 2.19 2:98bb65f5:::smithi17557870-6:head [write 467327~633480] snapc 6c=[6c,6b,6a,69,68,65,63,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-09-11T03:45:04.504 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:01.874110+0000", 2024-09-11T03:45:04.505 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 182.568176507, 2024-09-11T03:45:04.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084441010000000007, 2024-09-11T03:45:04.505 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.505 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.505 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:04.505 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:04.505 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:04.505 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:04.505 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2513 2024-09-11T03:45:04.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.505 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.874110+0000", 2024-09-11T03:45:04.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.874110+0000", 2024-09-11T03:45:04.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.506 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:04.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.874112+0000", 2024-09-11T03:45:04.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.728e-06 2024-09-11T03:45:04.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.875529+0000", 2024-09-11T03:45:04.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001417153 2024-09-11T03:45:04.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.875542+0000", 2024-09-11T03:45:04.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3156000000000001e-05 2024-09-11T03:45:04.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.875551+0000", 2024-09-11T03:45:04.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3080000000000008e-06 2024-09-11T03:45:04.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.875732+0000", 2024-09-11T03:45:04.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018141199999999999 2024-09-11T03:45:04.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.875856+0000", 2024-09-11T03:45:04.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012354299999999999 2024-09-11T03:45:04.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,8", 2024-09-11T03:45:04.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.876030+0000", 2024-09-11T03:45:04.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000174079 2024-09-11T03:45:04.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:45:04.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.878440+0000", 2024-09-11T03:45:04.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024106829999999998 2024-09-11T03:45:04.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:04.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.880223+0000", 2024-09-11T03:45:04.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017821639999999999 2024-09-11T03:45:04.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:04.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.882437+0000", 2024-09-11T03:45:04.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022147030000000002 2024-09-11T03:45:04.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:04.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.882499+0000", 2024-09-11T03:45:04.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1747000000000004e-05 2024-09-11T03:45:04.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:01.882554+0000", 2024-09-11T03:45:04.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5424999999999999e-05 2024-09-11T03:45:04.511 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.511 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.511 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.512 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2568 2.d e314/95 2:b6a25a79:::smithi17557870-47:head v 314'392, mlcod=314'388)", 2024-09-11T03:45:04.512 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:10.885879+0000", 2024-09-11T03:45:04.512 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 173.55640803200001, 2024-09-11T03:45:04.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086201669999999998, 2024-09-11T03:45:04.512 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.512 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.512 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:04.512 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.885879+0000", 2024-09-11T03:45:04.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.885879+0000", 2024-09-11T03:45:04.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.885880+0000", 2024-09-11T03:45:04.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1900000000000002e-07 2024-09-11T03:45:04.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.892125+0000", 2024-09-11T03:45:04.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062450869999999999 2024-09-11T03:45:04.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.892137+0000", 2024-09-11T03:45:04.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2224e-05 2024-09-11T03:45:04.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.892143+0000", 2024-09-11T03:45:04.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.426e-06 2024-09-11T03:45:04.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.892200+0000", 2024-09-11T03:45:04.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6511999999999997e-05 2024-09-11T03:45:04.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.892218+0000", 2024-09-11T03:45:04.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.77e-05 2024-09-11T03:45:04.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:04.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.894465+0000", 2024-09-11T03:45:04.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022472159999999998 2024-09-11T03:45:04.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.894499+0000", 2024-09-11T03:45:04.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4282999999999997e-05 2024-09-11T03:45:04.516 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.516 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.516 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.516 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2641 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 350] snapc 75=[75,74,73,71,6f,6c,68,65,5d] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-11T03:45:04.516 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:19.887918+0000", 2024-09-11T03:45:04.516 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 164.554368582, 2024-09-11T03:45:04.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011895905, 2024-09-11T03:45:04.516 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.517 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.517 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:04.517 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:04.517 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:04.517 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:04.517 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2641 2024-09-11T03:45:04.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.517 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887918+0000", 2024-09-11T03:45:04.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887918+0000", 2024-09-11T03:45:04.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887920+0000", 2024-09-11T03:45:04.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.762e-06 2024-09-11T03:45:04.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887928+0000", 2024-09-11T03:45:04.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8900000000000007e-06 2024-09-11T03:45:04.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887945+0000", 2024-09-11T03:45:04.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6707000000000001e-05 2024-09-11T03:45:04.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887953+0000", 2024-09-11T03:45:04.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5620000000000004e-06 2024-09-11T03:45:04.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.888014+0000", 2024-09-11T03:45:04.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0798000000000003e-05 2024-09-11T03:45:04.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:45:04.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.888019+0000", 2024-09-11T03:45:04.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5990000000000004e-06 2024-09-11T03:45:04.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.889290+0000", 2024-09-11T03:45:04.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012718009999999999 2024-09-11T03:45:04.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.889393+0000", 2024-09-11T03:45:04.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102298 2024-09-11T03:45:04.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,8", 2024-09-11T03:45:04.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.894120+0000", 2024-09-11T03:45:04.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047277259999999998 2024-09-11T03:45:04.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:45:04.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.897232+0000", 2024-09-11T03:45:04.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031110859999999999 2024-09-11T03:45:04.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:04.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.897530+0000", 2024-09-11T03:45:04.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000298342 2024-09-11T03:45:04.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:04.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.899728+0000", 2024-09-11T03:45:04.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021984819999999999 2024-09-11T03:45:04.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:04.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.899782+0000", 2024-09-11T03:45:04.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3922999999999999e-05 2024-09-11T03:45:04.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.899814+0000", 2024-09-11T03:45:04.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1928999999999999e-05 2024-09-11T03:45:04.523 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.523 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.523 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.523 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2642 2.19 2:98bb65f5:::smithi17557870-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e323)", 2024-09-11T03:45:04.523 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:19.887977+0000", 2024-09-11T03:45:04.523 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 164.554310031, 2024-09-11T03:45:04.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085743450000000006, 2024-09-11T03:45:04.523 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.523 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.523 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:45:04.524 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:04.524 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:04.524 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:04.524 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2642 2024-09-11T03:45:04.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.524 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887977+0000", 2024-09-11T03:45:04.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887977+0000", 2024-09-11T03:45:04.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887977+0000", 2024-09-11T03:45:04.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9699999999999996e-07 2024-09-11T03:45:04.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887980+0000", 2024-09-11T03:45:04.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4700000000000001e-06 2024-09-11T03:45:04.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887986+0000", 2024-09-11T03:45:04.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1140000000000002e-06 2024-09-11T03:45:04.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887991+0000", 2024-09-11T03:45:04.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3730000000000001e-06 2024-09-11T03:45:04.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.888058+0000", 2024-09-11T03:45:04.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6322999999999995e-05 2024-09-11T03:45:04.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:04.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.888062+0000", 2024-09-11T03:45:04.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7600000000000002e-06 2024-09-11T03:45:04.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.889535+0000", 2024-09-11T03:45:04.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001472391 2024-09-11T03:45:04.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:04.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.889580+0000", 2024-09-11T03:45:04.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5139000000000003e-05 2024-09-11T03:45:04.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.896410+0000", 2024-09-11T03:45:04.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068302379999999998 2024-09-11T03:45:04.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.896478+0000", 2024-09-11T03:45:04.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8159999999999998e-05 2024-09-11T03:45:04.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.896551+0000", 2024-09-11T03:45:04.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2780000000000005e-05 2024-09-11T03:45:04.529 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.529 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.529 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.529 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2944 2.12 2:494ba932:::smithi17557870-43:head [copy-from ver 311] snapc 80=[80,7b,7a,79,77,75,6c] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-11T03:45:04.529 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:43.962947+0000", 2024-09-11T03:45:04.529 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 140.47933938200001, 2024-09-11T03:45:04.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014367049, 2024-09-11T03:45:04.529 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.529 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.529 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:04.530 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:04.530 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:04.530 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:04.530 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2944 2024-09-11T03:45:04.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.530 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962947+0000", 2024-09-11T03:45:04.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962947+0000", 2024-09-11T03:45:04.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962949+0000", 2024-09-11T03:45:04.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.689e-06 2024-09-11T03:45:04.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962955+0000", 2024-09-11T03:45:04.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.297e-06 2024-09-11T03:45:04.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962967+0000", 2024-09-11T03:45:04.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1517e-05 2024-09-11T03:45:04.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962975+0000", 2024-09-11T03:45:04.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2889999999999998e-06 2024-09-11T03:45:04.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963875+0000", 2024-09-11T03:45:04.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00089981200000000005 2024-09-11T03:45:04.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.964025+0000", 2024-09-11T03:45:04.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000149624 2024-09-11T03:45:04.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,7", 2024-09-11T03:45:04.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.971400+0000", 2024-09-11T03:45:04.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073748349999999997 2024-09-11T03:45:04.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:45:04.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.975907+0000", 2024-09-11T03:45:04.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045076159999999999 2024-09-11T03:45:04.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:04.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.976808+0000", 2024-09-11T03:45:04.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00090078700000000001 2024-09-11T03:45:04.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:04.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.977244+0000", 2024-09-11T03:45:04.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00043565800000000001 2024-09-11T03:45:04.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:04.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.977281+0000", 2024-09-11T03:45:04.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7150999999999997e-05 2024-09-11T03:45:04.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.977315+0000", 2024-09-11T03:45:04.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3773999999999999e-05 2024-09-11T03:45:04.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.535 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.535 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2945 2.12 2:494ba932:::smithi17557870-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e347)", 2024-09-11T03:45:04.535 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:43.963009+0000", 2024-09-11T03:45:04.535 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 140.479277782, 2024-09-11T03:45:04.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012192595000000001, 2024-09-11T03:45:04.536 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.536 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.536 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:45:04.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:04.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:04.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:04.536 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2945 2024-09-11T03:45:04.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.536 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963009+0000", 2024-09-11T03:45:04.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963009+0000", 2024-09-11T03:45:04.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963010+0000", 2024-09-11T03:45:04.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0699999999999997e-07 2024-09-11T03:45:04.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963012+0000", 2024-09-11T03:45:04.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6599999999999999e-06 2024-09-11T03:45:04.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963020+0000", 2024-09-11T03:45:04.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2649999999999999e-06 2024-09-11T03:45:04.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963025+0000", 2024-09-11T03:45:04.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6860000000000003e-06 2024-09-11T03:45:04.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.964185+0000", 2024-09-11T03:45:04.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011599329999999999 2024-09-11T03:45:04.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:04.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.964230+0000", 2024-09-11T03:45:04.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4336999999999999e-05 2024-09-11T03:45:04.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.974986+0000", 2024-09-11T03:45:04.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010755928999999999 2024-09-11T03:45:04.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.975091+0000", 2024-09-11T03:45:04.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105701 2024-09-11T03:45:04.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.975202+0000", 2024-09-11T03:45:04.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011047700000000001 2024-09-11T03:45:04.540 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.540 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.540 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.541 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:3536 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 408] snapc 90=[90,8f,8d,8c,8b,8a] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-11T03:45:04.541 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:43:18.903548+0000", 2024-09-11T03:45:04.541 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 105.53873845, 2024-09-11T03:45:04.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014015196000000001, 2024-09-11T03:45:04.541 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.541 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.541 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:04.541 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:04.541 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:04.541 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:04.541 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3536 2024-09-11T03:45:04.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.541 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903548+0000", 2024-09-11T03:45:04.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903548+0000", 2024-09-11T03:45:04.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903550+0000", 2024-09-11T03:45:04.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.499e-06 2024-09-11T03:45:04.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903555+0000", 2024-09-11T03:45:04.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6890000000000003e-06 2024-09-11T03:45:04.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903567+0000", 2024-09-11T03:45:04.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1912999999999999e-05 2024-09-11T03:45:04.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903574+0000", 2024-09-11T03:45:04.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7260000000000002e-06 2024-09-11T03:45:04.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904043+0000", 2024-09-11T03:45:04.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00046877799999999999 2024-09-11T03:45:04.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904136+0000", 2024-09-11T03:45:04.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2745999999999996e-05 2024-09-11T03:45:04.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,8", 2024-09-11T03:45:04.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.912436+0000", 2024-09-11T03:45:04.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083000049999999992 2024-09-11T03:45:04.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:45:04.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.915813+0000", 2024-09-11T03:45:04.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033772839999999999 2024-09-11T03:45:04.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:04.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.916094+0000", 2024-09-11T03:45:04.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028135600000000002 2024-09-11T03:45:04.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:04.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.917498+0000", 2024-09-11T03:45:04.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014033680000000001 2024-09-11T03:45:04.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:04.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.917534+0000", 2024-09-11T03:45:04.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6180000000000003e-05 2024-09-11T03:45:04.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.917564+0000", 2024-09-11T03:45:04.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9652e-05 2024-09-11T03:45:04.547 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.547 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.547 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.547 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:3537 2.19 2:98bb65f5:::smithi17557870-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e381)", 2024-09-11T03:45:04.547 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:45:04.547 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 105.53869097899999, 2024-09-11T03:45:04.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011559349999999999, 2024-09-11T03:45:04.547 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.547 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.547 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:45:04.547 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:04.547 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:04.547 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:04.548 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3537 2024-09-11T03:45:04.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.548 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:45:04.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:45:04.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:45:04.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8899999999999999e-07 2024-09-11T03:45:04.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903599+0000", 2024-09-11T03:45:04.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3750000000000001e-06 2024-09-11T03:45:04.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903606+0000", 2024-09-11T03:45:04.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1969999999999999e-06 2024-09-11T03:45:04.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903611+0000", 2024-09-11T03:45:04.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3680000000000001e-06 2024-09-11T03:45:04.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904244+0000", 2024-09-11T03:45:04.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00063279999999999999 2024-09-11T03:45:04.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:04.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904291+0000", 2024-09-11T03:45:04.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6755999999999998e-05 2024-09-11T03:45:04.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.914973+0000", 2024-09-11T03:45:04.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010682199999999999 2024-09-11T03:45:04.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.915043+0000", 2024-09-11T03:45:04.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0297999999999997e-05 2024-09-11T03:45:04.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.915155+0000", 2024-09-11T03:45:04.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000111767 2024-09-11T03:45:04.552 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.552 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.552 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.552 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:4021 2.7 e435/417 2:e77ce595:::smithi17557870-31:head v 435'563, mlcod=435'561)", 2024-09-11T03:45:04.552 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:44:13.499341+0000", 2024-09-11T03:45:04.552 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 50.942945430000002, 2024-09-11T03:45:04.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082180559999999996, 2024-09-11T03:45:04.552 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:04.552 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:04.552 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:04.552 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:04.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:04.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.499341+0000", 2024-09-11T03:45:04.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:04.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.499341+0000", 2024-09-11T03:45:04.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:04.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:04.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.499342+0000", 2024-09-11T03:45:04.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0270000000000001e-06 2024-09-11T03:45:04.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:04.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.504715+0000", 2024-09-11T03:45:04.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053727810000000001 2024-09-11T03:45:04.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:04.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.504732+0000", 2024-09-11T03:45:04.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6317e-05 2024-09-11T03:45:04.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:04.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.504739+0000", 2024-09-11T03:45:04.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8399999999999995e-06 2024-09-11T03:45:04.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:04.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.504806+0000", 2024-09-11T03:45:04.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6249999999999998e-05 2024-09-11T03:45:04.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:04.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.504825+0000", 2024-09-11T03:45:04.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8870000000000001e-05 2024-09-11T03:45:04.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:04.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.507520+0000", 2024-09-11T03:45:04.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026951280000000002 2024-09-11T03:45:04.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:04.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:04.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:04.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.507559+0000", 2024-09-11T03:45:04.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9842999999999998e-05 2024-09-11T03:45:04.556 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.556 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.556 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.556 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:04.556 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:04.556 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:45:04.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:04.582 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:04.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:04.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:04.684 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:04.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:04.706 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:04.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:04.807 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:04.828 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:04.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:04.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:04.930 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:04.950 INFO:tasks.rados.rados.2.smithi175.stdout:1785: done (2 left) 2024-09-11T03:45:04.950 INFO:tasks.rados.rados.2.smithi175.stdout:1786: done (1 left) 2024-09-11T03:45:04.950 INFO:tasks.rados.rados.2.smithi175.stdout:1787: done (0 left) 2024-09-11T03:45:04.950 INFO:tasks.rados.rados.2.smithi175.stdout:1788: snap_create 2024-09-11T03:45:04.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:04.952 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:04.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:05.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:05.080 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:05.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:05.182 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:05.202 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:04 smithi084 ceph-mon[27254]: osdmap e486: 12 total, 12 up, 11 in 2024-09-11T03:45:05.208 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:05.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:05.209 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:05.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:04 smithi175 ceph-mon[30894]: osdmap e486: 12 total, 12 up, 11 in 2024-09-11T03:45:05.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:05.331 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:05.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:05.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:05.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:04 smithi121 ceph-mon[30729]: osdmap e486: 12 total, 12 up, 11 in 2024-09-11T03:45:05.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:05.459 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:05.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:05.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:05.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:05.583 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:05.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:05.685 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:05.710 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:05.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:05.711 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:05.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:05.838 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:05.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:05.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:05.948 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:45:05.951 INFO:tasks.rados.rados.2.smithi175.stdout:1788: done (0 left) 2024-09-11T03:45:05.952 INFO:tasks.rados.rados.2.smithi175.stdout:1789: snap_create 2024-09-11T03:45:05.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T03:45:05.964 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:05.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:05.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:06.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:06.088 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:06.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:06.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:06.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:06.206 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:45:06.210 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:45:06.212 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:06.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:06.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:05 smithi175 ceph-mon[30894]: pgmap v885: 57 pgs: 57 active+clean; 157 MiB data, 715 MiB used, 983 GiB / 983 GiB avail; 20 MiB/s rd, 1.4 MiB/s wr, 25 op/s; 8.8 KiB/s, 0 objects/s recovering 2024-09-11T03:45:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:05 smithi175 ceph-mon[30894]: osdmap e487: 12 total, 12 up, 11 in 2024-09-11T03:45:06.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:05 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2575989552' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:45:06.228 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:05 smithi084 ceph-mon[27254]: pgmap v885: 57 pgs: 57 active+clean; 157 MiB data, 715 MiB used, 983 GiB / 983 GiB avail; 20 MiB/s rd, 1.4 MiB/s wr, 25 op/s; 8.8 KiB/s, 0 objects/s recovering 2024-09-11T03:45:06.228 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:05 smithi084 ceph-mon[27254]: osdmap e487: 12 total, 12 up, 11 in 2024-09-11T03:45:06.228 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:05 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2575989552' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:45:06.314 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:06.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:05 smithi121 ceph-mon[30729]: pgmap v885: 57 pgs: 57 active+clean; 157 MiB data, 715 MiB used, 983 GiB / 983 GiB avail; 20 MiB/s rd, 1.4 MiB/s wr, 25 op/s; 8.8 KiB/s, 0 objects/s recovering 2024-09-11T03:45:06.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:05 smithi121 ceph-mon[30729]: osdmap e487: 12 total, 12 up, 11 in 2024-09-11T03:45:06.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:05 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2575989552' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:45:06.340 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:06.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:06.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:06.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:06.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:06.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:06.566 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:06.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:06.593 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:06.593 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:06.679 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:06.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:06.718 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:06.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:06.811 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:45:06.811 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:45:06.811 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:45:06.811 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:45:06.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:06.840 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:06.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:06.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:06.939 INFO:tasks.rados.rados.2.smithi175.stdout:1789: done (0 left) 2024-09-11T03:45:06.939 INFO:tasks.rados.rados.2.smithi175.stdout:1790: read oid 26 snap 198 2024-09-11T03:45:06.939 INFO:tasks.rados.rados.2.smithi175.stdout:1790: expect (ObjNum 595 snap 185 seq_num 595) 2024-09-11T03:45:06.939 INFO:tasks.rados.rados.2.smithi175.stdout:1791: delete oid 19 current snap is 199 2024-09-11T03:45:06.941 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:06.944 INFO:tasks.rados.rados.2.smithi175.stdout:1791: done (1 left) 2024-09-11T03:45:06.944 INFO:tasks.rados.rados.2.smithi175.stdout:1792: snap_remove snap 190 2024-09-11T03:45:06.967 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:06.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:06.968 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:07.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:07.090 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:07.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:07.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:07.161 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:45:07.161 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:45:07.177 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:45:07.177 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:45:07.192 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:07.194 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:45:07.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:45:07.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:06 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2575989552' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:45:07.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:06 smithi084 ceph-mon[27254]: osdmap e488: 12 total, 12 up, 11 in 2024-09-11T03:45:07.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:06 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:07.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:06 smithi084 ceph-mon[27254]: osdmap e489: 12 total, 12 up, 11 in 2024-09-11T03:45:07.210 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:45:07.210 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:45:07.212 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:07.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:07.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:07.227 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:45:07.227 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:45:07.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:06 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2575989552' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:45:07.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:06 smithi175 ceph-mon[30894]: osdmap e488: 12 total, 12 up, 11 in 2024-09-11T03:45:07.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:06 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:07.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:06 smithi175 ceph-mon[30894]: osdmap e489: 12 total, 12 up, 11 in 2024-09-11T03:45:07.243 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:45:07.243 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:45:07.244 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:45:07.244 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:45:07.245 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T03:45:07.245 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.000000 2024-09-11T03:45:07.245 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 3 0 2024-09-11T03:45:07.261 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:45:07.261 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:45:07.277 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:45:07.277 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:45:07.294 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:45:07.295 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:45:07.312 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:45:07.312 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:45:07.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:07.329 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:45:07.330 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:45:07.335 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:07.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:07.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:07.359 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:45:07.360 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:45:07.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:06 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2575989552' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:45:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:06 smithi121 ceph-mon[30729]: osdmap e488: 12 total, 12 up, 11 in 2024-09-11T03:45:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:06 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:06 smithi121 ceph-mon[30729]: osdmap e489: 12 total, 12 up, 11 in 2024-09-11T03:45:07.437 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:07.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:07.460 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:07.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:07.560 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:07.575 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T03:45:07.587 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:07.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:07.588 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:07.689 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:07.715 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:07.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:07.816 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:45:07.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:07.838 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:07.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:07.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:07.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:07.943 INFO:tasks.rados.rados.2.smithi175.stdout:1792: done (1 left) 2024-09-11T03:45:07.943 INFO:tasks.rados.rados.2.smithi175.stdout:1793: write (excl) oid 5 current snap is 199 2024-09-11T03:45:07.944 INFO:tasks.rados.rados.2.smithi175.stdout:1793: seq_num 615 ranges {736305=786375,2105367=230938} 2024-09-11T03:45:07.953 INFO:tasks.rados.rados.2.smithi175.stdout:1793: writing smithi17557870-5 from 736305 to 1522680 tid 1 2024-09-11T03:45:07.958 INFO:tasks.rados.rados.2.smithi175.stdout:1793: writing smithi17557870-5 from 2105367 to 2336305 tid 2 2024-09-11T03:45:07.958 INFO:tasks.rados.rados.2.smithi175.stdout:1794: snap_remove snap 194 2024-09-11T03:45:07.963 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:07.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:07.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:08.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:08.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:08.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:08.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:07 smithi084 ceph-mon[27254]: pgmap v888: 57 pgs: 57 active+clean; 154 MiB data, 719 MiB used, 983 GiB / 983 GiB avail; 255 B/s rd, 195 KiB/s wr, 3 op/s 2024-09-11T03:45:08.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/157733284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:08.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:07 smithi084 ceph-mon[27254]: osdmap e490: 12 total, 12 up, 11 in 2024-09-11T03:45:08.170 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:08.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:08.204 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:07 smithi121 ceph-mon[30729]: pgmap v888: 57 pgs: 57 active+clean; 154 MiB data, 719 MiB used, 983 GiB / 983 GiB avail; 255 B/s rd, 195 KiB/s wr, 3 op/s 2024-09-11T03:45:08.204 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/157733284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:08.204 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:07 smithi121 ceph-mon[30729]: osdmap e490: 12 total, 12 up, 11 in 2024-09-11T03:45:08.209 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:08.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:07 smithi175 ceph-mon[30894]: pgmap v888: 57 pgs: 57 active+clean; 154 MiB data, 719 MiB used, 983 GiB / 983 GiB avail; 255 B/s rd, 195 KiB/s wr, 3 op/s 2024-09-11T03:45:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/157733284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:08.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:07 smithi175 ceph-mon[30894]: osdmap e490: 12 total, 12 up, 11 in 2024-09-11T03:45:08.303 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:45:08.304 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:45:08.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:08.331 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:08.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:08.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:08.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:08.456 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:08.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:08.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:08.558 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:08.579 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:08.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:08.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:08.611 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:45:08.681 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:08.702 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:08.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:08.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:08.804 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:08.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:08.826 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:08.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:08.868 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:08.927 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:08.953 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:08.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:08.954 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:08.958 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T03:45:08.973 INFO:tasks.rados.rados.2.smithi175.stdout:1794: done (2 left) 2024-09-11T03:45:08.974 INFO:tasks.rados.rados.2.smithi175.stdout:1795: copy_from oid 42 from oid 38 current snap is 199 2024-09-11T03:45:08.974 INFO:tasks.rados.rados.2.smithi175.stdout:1790: expect (ObjNum 595 snap 185 seq_num 595) 2024-09-11T03:45:09.055 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:09.077 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:09.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:09.078 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:09.124 INFO:tasks.rados.rados.2.smithi175.stdout:1793: finishing write tid 1 to smithi17557870-5 2024-09-11T03:45:09.124 INFO:tasks.rados.rados.2.smithi175.stdout:1793: finishing write tid 2 to smithi17557870-5 2024-09-11T03:45:09.124 INFO:tasks.rados.rados.2.smithi175.stdout:1793: finishing write tid 3 to smithi17557870-5 2024-09-11T03:45:09.124 INFO:tasks.rados.rados.2.smithi175.stdout:1793: finishing write tid 4 to smithi17557870-5 2024-09-11T03:45:09.124 INFO:tasks.rados.rados.2.smithi175.stdout:1793: oid 5 updating version 0 to 276 2024-09-11T03:45:09.124 INFO:tasks.rados.rados.2.smithi175.stdout:1793: oid 5 updating version 276 to 277 2024-09-11T03:45:09.124 INFO:tasks.rados.rados.2.smithi175.stdout:1793: oid 5 version 277 is already newer than 275 2024-09-11T03:45:09.125 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 277 (ObjNum 615 snap 199 seq_num 615) dirty exists 2024-09-11T03:45:09.125 INFO:tasks.rados.rados.2.smithi175.stdout:1793: left oid 5 (ObjNum 615 snap 199 seq_num 615) 2024-09-11T03:45:09.125 INFO:tasks.rados.rados.2.smithi175.stdout:1790: done (2 left) 2024-09-11T03:45:09.125 INFO:tasks.rados.rados.2.smithi175.stdout:1793: done (1 left) 2024-09-11T03:45:09.125 INFO:tasks.rados.rados.2.smithi175.stdout:1796: write (excl) oid 1 current snap is 199 2024-09-11T03:45:09.125 INFO:tasks.rados.rados.2.smithi175.stdout:1795: finishing copy_from racing read to smithi17557870-42 2024-09-11T03:45:09.125 INFO:tasks.rados.rados.2.smithi175.stdout:1795: finishing copy_from to smithi17557870-42 2024-09-11T03:45:09.125 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 652 (ObjNum 565 snap 172 seq_num 565) dirty exists 2024-09-11T03:45:09.125 INFO:tasks.rados.rados.2.smithi175.stdout:1796: seq_num 616 ranges {753112=577977,1880202=472910} 2024-09-11T03:45:09.131 INFO:tasks.rados.rados.2.smithi175.stdout:1796: writing smithi17557870-1 from 753112 to 1331089 tid 1 2024-09-11T03:45:09.137 INFO:tasks.rados.rados.2.smithi175.stdout:1796: writing smithi17557870-1 from 1880202 to 2353112 tid 2 2024-09-11T03:45:09.138 INFO:tasks.rados.rados.2.smithi175.stdout:1795: done (1 left) 2024-09-11T03:45:09.138 INFO:tasks.rados.rados.2.smithi175.stdout:1797: write oid 39 current snap is 199 2024-09-11T03:45:09.138 INFO:tasks.rados.rados.2.smithi175.stdout:1796: finishing write tid 1 to smithi17557870-1 2024-09-11T03:45:09.138 INFO:tasks.rados.rados.2.smithi175.stdout:1797: seq_num 617 ranges {769919=653226,1938684=431235} 2024-09-11T03:45:09.145 INFO:tasks.rados.rados.2.smithi175.stdout:1797: writing smithi17557870-39 from 769919 to 1423145 tid 1 2024-09-11T03:45:09.150 INFO:tasks.rados.rados.2.smithi175.stdout:1797: writing smithi17557870-39 from 1938684 to 2369919 tid 2 2024-09-11T03:45:09.150 INFO:tasks.rados.rados.2.smithi175.stdout:1798: write (excl) oid 16 current snap is 199 2024-09-11T03:45:09.150 INFO:tasks.rados.rados.2.smithi175.stdout:1796: finishing write tid 2 to smithi17557870-1 2024-09-11T03:45:09.150 INFO:tasks.rados.rados.2.smithi175.stdout:1796: finishing write tid 3 to smithi17557870-1 2024-09-11T03:45:09.150 INFO:tasks.rados.rados.2.smithi175.stdout:1798: seq_num 618 ranges {786726=728475,2280813=105913} 2024-09-11T03:45:09.158 INFO:tasks.rados.rados.2.smithi175.stdout:1798: writing smithi17557870-16 from 786726 to 1515201 tid 1 2024-09-11T03:45:09.163 INFO:tasks.rados.rados.2.smithi175.stdout:1798: writing smithi17557870-16 from 2280813 to 2386726 tid 2 2024-09-11T03:45:09.163 INFO:tasks.rados.rados.2.smithi175.stdout:1799: snap_remove snap 191 2024-09-11T03:45:09.179 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:09.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:09.202 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:09.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:09.203 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:45:09.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:08 smithi175 ceph-mon[30894]: 1.17 scrub starts 2024-09-11T03:45:09.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:08 smithi175 ceph-mon[30894]: 1.17 scrub ok 2024-09-11T03:45:09.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:08 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:45:09.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:08 smithi175 ceph-mon[30894]: 2.d scrub ok 2024-09-11T03:45:09.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:08 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1314369143' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.0} : dispatch 2024-09-11T03:45:09.266 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:08 smithi084 ceph-mon[27254]: 1.17 scrub starts 2024-09-11T03:45:09.266 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:08 smithi084 ceph-mon[27254]: 1.17 scrub ok 2024-09-11T03:45:09.267 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:08 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:45:09.267 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:08 smithi084 ceph-mon[27254]: 2.d scrub ok 2024-09-11T03:45:09.267 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:08 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1314369143' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.0} : dispatch 2024-09-11T03:45:09.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:08 smithi121 ceph-mon[30729]: 1.17 scrub starts 2024-09-11T03:45:09.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:08 smithi121 ceph-mon[30729]: 1.17 scrub ok 2024-09-11T03:45:09.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:08 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:45:09.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:08 smithi121 ceph-mon[30729]: 2.d scrub ok 2024-09-11T03:45:09.293 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:08 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1314369143' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.0} : dispatch 2024-09-11T03:45:09.304 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:09.324 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:09.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:09.426 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:09.449 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:09.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:09.551 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:09.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:09.572 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:09.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:09.674 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:09.695 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:09.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:09.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:09.701 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:45:09.701 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:45:09.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:09.821 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:09.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:09.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:09.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:09.944 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:09.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:09.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:09.966 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:45:09.966 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.3 primary-affinity to 0 (802) 2024-09-11T03:45:09.972 INFO:tasks.rados.rados.2.smithi175.stdout:1799: done (3 left) 2024-09-11T03:45:09.972 INFO:tasks.rados.rados.2.smithi175.stdout:1800: write oid 3 current snap is 199 2024-09-11T03:45:09.972 INFO:tasks.rados.rados.2.smithi175.stdout:1800: seq_num 619 ranges {403533=403724,1539295=652201} 2024-09-11T03:45:09.976 INFO:tasks.rados.rados.2.smithi175.stdout:1800: writing smithi17557870-3 from 403533 to 807257 tid 1 2024-09-11T03:45:09.985 INFO:tasks.rados.rados.2.smithi175.stdout:1800: writing smithi17557870-3 from 1539295 to 2191496 tid 2 2024-09-11T03:45:09.985 INFO:tasks.rados.rados.2.smithi175.stdout:1801: copy_from oid 43 from oid 26 current snap is 199 2024-09-11T03:45:09.985 INFO:tasks.rados.rados.2.smithi175.stdout:1796: finishing write tid 4 to smithi17557870-1 2024-09-11T03:45:09.985 INFO:tasks.rados.rados.2.smithi175.stdout:1796: oid 1 updating version 0 to 658 2024-09-11T03:45:09.985 INFO:tasks.rados.rados.2.smithi175.stdout:1796: oid 1 updating version 658 to 660 2024-09-11T03:45:09.985 INFO:tasks.rados.rados.2.smithi175.stdout:1796: oid 1 version 660 is already newer than 659 2024-09-11T03:45:09.985 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 660 (ObjNum 616 snap 199 seq_num 616) dirty exists 2024-09-11T03:45:09.985 INFO:tasks.rados.rados.2.smithi175.stdout:1796: left oid 1 (ObjNum 616 snap 199 seq_num 616) 2024-09-11T03:45:09.985 INFO:tasks.rados.rados.2.smithi175.stdout:1797: finishing write tid 1 to smithi17557870-39 2024-09-11T03:45:09.985 INFO:tasks.rados.rados.2.smithi175.stdout:1796: done (4 left) 2024-09-11T03:45:09.985 INFO:tasks.rados.rados.2.smithi175.stdout:1802: snap_remove snap 198 2024-09-11T03:45:10.046 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:10.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:10.067 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:10.068 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:10.121 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:45:10.168 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:10.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:10.190 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:10.190 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:10.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:09 smithi175 ceph-mon[30894]: pgmap v891: 57 pgs: 57 active+clean; 158 MiB data, 724 MiB used, 983 GiB / 983 GiB avail; 511 B/s rd, 1023 KiB/s wr, 4 op/s 2024-09-11T03:45:10.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:09 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1314369143' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]': finished 2024-09-11T03:45:10.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:09 smithi175 ceph-mon[30894]: osdmap e491: 12 total, 12 up, 11 in 2024-09-11T03:45:10.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:09 smithi175 ceph-mon[30894]: from='client.16083 172.21.15.84:0/1314369143' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.0} : dispatch 2024-09-11T03:45:10.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:09 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/230431579' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:45:10.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:09 smithi084 ceph-mon[27254]: pgmap v891: 57 pgs: 57 active+clean; 158 MiB data, 724 MiB used, 983 GiB / 983 GiB avail; 511 B/s rd, 1023 KiB/s wr, 4 op/s 2024-09-11T03:45:10.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:09 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1314369143' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]': finished 2024-09-11T03:45:10.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:09 smithi084 ceph-mon[27254]: osdmap e491: 12 total, 12 up, 11 in 2024-09-11T03:45:10.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:09 smithi084 ceph-mon[27254]: from='client.16083 172.21.15.84:0/1314369143' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.0} : dispatch 2024-09-11T03:45:10.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:09 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/230431579' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:45:10.291 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:10.312 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:10.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:10.313 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:10.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:09 smithi121 ceph-mon[30729]: pgmap v891: 57 pgs: 57 active+clean; 158 MiB data, 724 MiB used, 983 GiB / 983 GiB avail; 511 B/s rd, 1023 KiB/s wr, 4 op/s 2024-09-11T03:45:10.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:09 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1314369143' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]': finished 2024-09-11T03:45:10.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:09 smithi121 ceph-mon[30729]: osdmap e491: 12 total, 12 up, 11 in 2024-09-11T03:45:10.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:09 smithi121 ceph-mon[30729]: from='client.16083 172.21.15.84:0/1314369143' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.0} : dispatch 2024-09-11T03:45:10.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:09 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/230431579' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:45:10.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T03:45:10.414 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:10.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:10.437 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:10.437 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:10.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:10.560 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:10.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:10.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:10.598 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:45:10.662 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:10.684 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:10.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:10.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:10.786 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:10.808 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:10.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:10.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:10.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:10.934 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:10.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:10.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:10.982 INFO:tasks.rados.rados.2.smithi175.stdout:1802: done (4 left) 2024-09-11T03:45:10.982 INFO:tasks.rados.rados.2.smithi175.stdout:1803: write (excl) oid 29 current snap is 199 2024-09-11T03:45:10.982 INFO:tasks.rados.rados.2.smithi175.stdout:1803: seq_num 620 ranges {420340=478973,1597777=512212} 2024-09-11T03:45:10.987 INFO:tasks.rados.rados.2.smithi175.stdout:1803: writing smithi17557870-29 from 420340 to 899313 tid 1 2024-09-11T03:45:10.994 INFO:tasks.rados.rados.2.smithi175.stdout:1803: writing smithi17557870-29 from 1597777 to 2109989 tid 2 2024-09-11T03:45:10.994 INFO:tasks.rados.rados.2.smithi175.stdout:1804: copy_from oid 34 from oid 18 current snap is 199 2024-09-11T03:45:10.994 INFO:tasks.rados.rados.2.smithi175.stdout:1797: finishing write tid 2 to smithi17557870-39 2024-09-11T03:45:10.994 INFO:tasks.rados.rados.2.smithi175.stdout:1797: finishing write tid 3 to smithi17557870-39 2024-09-11T03:45:10.994 INFO:tasks.rados.rados.2.smithi175.stdout:1797: finishing write tid 4 to smithi17557870-39 2024-09-11T03:45:10.995 INFO:tasks.rados.rados.2.smithi175.stdout:1797: oid 39 updating version 0 to 519 2024-09-11T03:45:10.995 INFO:tasks.rados.rados.2.smithi175.stdout:1797: oid 39 version 519 is already newer than 518 2024-09-11T03:45:10.995 INFO:tasks.rados.rados.2.smithi175.stdout:1797: oid 39 updating version 519 to 520 2024-09-11T03:45:10.995 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 520 (ObjNum 617 snap 199 seq_num 617) dirty exists 2024-09-11T03:45:10.995 INFO:tasks.rados.rados.2.smithi175.stdout:1797: left oid 39 (ObjNum 617 snap 199 seq_num 617) 2024-09-11T03:45:10.995 INFO:tasks.rados.rados.2.smithi175.stdout:1798: finishing write tid 1 to smithi17557870-16 2024-09-11T03:45:10.995 INFO:tasks.rados.rados.2.smithi175.stdout:1797: done (5 left) 2024-09-11T03:45:10.995 INFO:tasks.rados.rados.2.smithi175.stdout:1805: read oid 9 snap -1 2024-09-11T03:45:10.995 INFO:tasks.rados.rados.2.smithi175.stdout:1805: expect deleted 2024-09-11T03:45:10.995 INFO:tasks.rados.rados.2.smithi175.stdout:1806: read oid 27 snap -1 2024-09-11T03:45:10.995 INFO:tasks.rados.rados.2.smithi175.stdout:1806: expect (ObjNum 577 snap 177 seq_num 577) 2024-09-11T03:45:10.995 INFO:tasks.rados.rados.2.smithi175.stdout:1807: write oid 23 current snap is 199 2024-09-11T03:45:10.995 INFO:tasks.rados.rados.2.smithi175.stdout:1798: finishing write tid 2 to smithi17557870-16 2024-09-11T03:45:10.995 INFO:tasks.rados.rados.2.smithi175.stdout:1807: seq_num 621 ranges {437147=554222,1656259=655870} 2024-09-11T03:45:10.999 INFO:tasks.rados.rados.2.smithi175.stdout:1807: writing smithi17557870-23 from 437147 to 991369 tid 1 2024-09-11T03:45:11.009 INFO:tasks.rados.rados.2.smithi175.stdout:1807: writing smithi17557870-23 from 1656259 to 2312129 tid 2 2024-09-11T03:45:11.009 INFO:tasks.rados.rados.2.smithi175.stdout:1808: delete oid 17 current snap is 199 2024-09-11T03:45:11.009 INFO:tasks.rados.rados.2.smithi175.stdout:1798: finishing write tid 3 to smithi17557870-16 2024-09-11T03:45:11.009 INFO:tasks.rados.rados.2.smithi175.stdout:1798: finishing write tid 4 to smithi17557870-16 2024-09-11T03:45:11.009 INFO:tasks.rados.rados.2.smithi175.stdout:1798: oid 16 updating version 0 to 492 2024-09-11T03:45:11.009 INFO:tasks.rados.rados.2.smithi175.stdout:1798: oid 16 updating version 492 to 493 2024-09-11T03:45:11.009 INFO:tasks.rados.rados.2.smithi175.stdout:1798: oid 16 updating version 493 to 494 2024-09-11T03:45:11.010 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 494 (ObjNum 618 snap 199 seq_num 618) dirty exists 2024-09-11T03:45:11.010 INFO:tasks.rados.rados.2.smithi175.stdout:1798: left oid 16 (ObjNum 618 snap 199 seq_num 618) 2024-09-11T03:45:11.010 INFO:tasks.rados.rados.2.smithi175.stdout:1800: finishing write tid 1 to smithi17557870-3 2024-09-11T03:45:11.010 INFO:tasks.rados.rados.2.smithi175.stdout:1800: finishing write tid 2 to smithi17557870-3 2024-09-11T03:45:11.010 INFO:tasks.rados.rados.2.smithi175.stdout:1800: finishing write tid 3 to smithi17557870-3 2024-09-11T03:45:11.010 INFO:tasks.rados.rados.2.smithi175.stdout:1800: finishing write tid 4 to smithi17557870-3 2024-09-11T03:45:11.010 INFO:tasks.rados.rados.2.smithi175.stdout:1800: oid 3 updating version 0 to 674 2024-09-11T03:45:11.010 INFO:tasks.rados.rados.2.smithi175.stdout:1800: oid 3 updating version 674 to 678 2024-09-11T03:45:11.010 INFO:tasks.rados.rados.2.smithi175.stdout:1800: oid 3 version 678 is already newer than 677 2024-09-11T03:45:11.010 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 678 (ObjNum 619 snap 199 seq_num 619) dirty exists 2024-09-11T03:45:11.010 INFO:tasks.rados.rados.2.smithi175.stdout:1800: left oid 3 (ObjNum 619 snap 199 seq_num 619) 2024-09-11T03:45:11.010 INFO:tasks.rados.rados.2.smithi175.stdout:1801: finishing copy_from racing read to smithi17557870-43 2024-09-11T03:45:11.010 INFO:tasks.rados.rados.2.smithi175.stdout:1801: finishing copy_from to smithi17557870-43 2024-09-11T03:45:11.010 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 359 (ObjNum 595 snap 185 seq_num 595) dirty exists 2024-09-11T03:45:11.011 INFO:tasks.rados.rados.2.smithi175.stdout:1803: finishing write tid 1 to smithi17557870-29 2024-09-11T03:45:11.011 INFO:tasks.rados.rados.2.smithi175.stdout:1803: finishing write tid 2 to smithi17557870-29 2024-09-11T03:45:11.011 INFO:tasks.rados.rados.2.smithi175.stdout:1803: finishing write tid 3 to smithi17557870-29 2024-09-11T03:45:11.011 INFO:tasks.rados.rados.2.smithi175.stdout:1803: finishing write tid 4 to smithi17557870-29 2024-09-11T03:45:11.011 INFO:tasks.rados.rados.2.smithi175.stdout:1803: oid 29 updating version 0 to 668 2024-09-11T03:45:11.011 INFO:tasks.rados.rados.2.smithi175.stdout:1803: oid 29 version 668 is already newer than 667 2024-09-11T03:45:11.011 INFO:tasks.rados.rados.2.smithi175.stdout:1803: oid 29 updating version 668 to 669 2024-09-11T03:45:11.011 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 669 (ObjNum 620 snap 199 seq_num 620) dirty exists 2024-09-11T03:45:11.011 INFO:tasks.rados.rados.2.smithi175.stdout:1803: left oid 29 (ObjNum 620 snap 199 seq_num 620) 2024-09-11T03:45:11.011 INFO:tasks.rados.rados.2.smithi175.stdout:1807: finishing write tid 1 to smithi17557870-23 2024-09-11T03:45:11.011 INFO:tasks.rados.rados.2.smithi175.stdout:1806: expect (ObjNum 577 snap 177 seq_num 577) 2024-09-11T03:45:11.036 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:11.057 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:11.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:11.058 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:11.059 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:45:11.059 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:45:11.159 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:11.174 INFO:tasks.rados.rados.2.smithi175.stdout:1804: finishing copy_from racing read to smithi17557870-34 2024-09-11T03:45:11.174 INFO:tasks.rados.rados.2.smithi175.stdout:1804: finishing copy_from to smithi17557870-34 2024-09-11T03:45:11.174 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 467 (ObjNum 609 snap 191 seq_num 609) dirty exists 2024-09-11T03:45:11.174 INFO:tasks.rados.rados.2.smithi175.stdout:1807: finishing write tid 2 to smithi17557870-23 2024-09-11T03:45:11.174 INFO:tasks.rados.rados.2.smithi175.stdout:1807: finishing write tid 3 to smithi17557870-23 2024-09-11T03:45:11.174 INFO:tasks.rados.rados.2.smithi175.stdout:1807: finishing write tid 4 to smithi17557870-23 2024-09-11T03:45:11.174 INFO:tasks.rados.rados.2.smithi175.stdout:1807: oid 23 updating version 0 to 339 2024-09-11T03:45:11.174 INFO:tasks.rados.rados.2.smithi175.stdout:1807: oid 23 version 339 is already newer than 338 2024-09-11T03:45:11.174 INFO:tasks.rados.rados.2.smithi175.stdout:1807: oid 23 version 339 is already newer than 337 2024-09-11T03:45:11.174 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 339 (ObjNum 621 snap 199 seq_num 621) dirty exists 2024-09-11T03:45:11.174 INFO:tasks.rados.rados.2.smithi175.stdout:1807: left oid 23 (ObjNum 621 snap 199 seq_num 621) 2024-09-11T03:45:11.175 INFO:tasks.rados.rados.2.smithi175.stdout:1798: done (8 left) 2024-09-11T03:45:11.175 INFO:tasks.rados.rados.2.smithi175.stdout:1800: done (7 left) 2024-09-11T03:45:11.175 INFO:tasks.rados.rados.2.smithi175.stdout:1801: done (6 left) 2024-09-11T03:45:11.175 INFO:tasks.rados.rados.2.smithi175.stdout:1803: done (5 left) 2024-09-11T03:45:11.175 INFO:tasks.rados.rados.2.smithi175.stdout:1804: done (4 left) 2024-09-11T03:45:11.175 INFO:tasks.rados.rados.2.smithi175.stdout:1805: done (3 left) 2024-09-11T03:45:11.175 INFO:tasks.rados.rados.2.smithi175.stdout:1806: done (2 left) 2024-09-11T03:45:11.175 INFO:tasks.rados.rados.2.smithi175.stdout:1807: done (1 left) 2024-09-11T03:45:11.175 INFO:tasks.rados.rados.2.smithi175.stdout:1808: done (0 left) 2024-09-11T03:45:11.175 INFO:tasks.rados.rados.2.smithi175.stdout:1809: read oid 41 snap 175 2024-09-11T03:45:11.175 INFO:tasks.rados.rados.2.smithi175.stdout:1809: expect (ObjNum 544 snap 165 seq_num 544) 2024-09-11T03:45:11.175 INFO:tasks.rados.rados.2.smithi175.stdout:1810: read oid 25 snap -1 2024-09-11T03:45:11.175 INFO:tasks.rados.rados.2.smithi175.stdout:1810: expect (ObjNum 496 snap 144 seq_num 496) 2024-09-11T03:45:11.175 INFO:tasks.rados.rados.2.smithi175.stdout:1811: write oid 18 current snap is 199 2024-09-11T03:45:11.176 INFO:tasks.rados.rados.2.smithi175.stdout:1811: seq_num 623 ranges {470761=704720,1656870=659539} 2024-09-11T03:45:11.179 INFO:tasks.rados.rados.2.smithi175.stdout:1811: writing smithi17557870-18 from 470761 to 1175481 tid 1 2024-09-11T03:45:11.182 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:11.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:11.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:11.186 INFO:tasks.rados.rados.2.smithi175.stdout:1811: writing smithi17557870-18 from 1656870 to 2316409 tid 2 2024-09-11T03:45:11.186 INFO:tasks.rados.rados.2.smithi175.stdout:1812: copy_from oid 19 from oid 49 current snap is 199 2024-09-11T03:45:11.186 INFO:tasks.rados.rados.2.smithi175.stdout:1809: expect (ObjNum 544 snap 165 seq_num 544) 2024-09-11T03:45:11.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:10 smithi084 ceph-mon[27254]: 1.12 scrub starts 2024-09-11T03:45:11.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:10 smithi084 ceph-mon[27254]: 1.12 scrub ok 2024-09-11T03:45:11.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:10 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:45:11.250 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:10 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:45:11.251 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:10 smithi084 ceph-mon[27254]: from='client.16083 172.21.15.84:0/1314369143' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]': finished 2024-09-11T03:45:11.251 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:10 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/230431579' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:45:11.251 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:10 smithi084 ceph-mon[27254]: osdmap e492: 12 total, 12 up, 11 in 2024-09-11T03:45:11.251 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:10 smithi084 ceph-mon[27254]: 1.c scrub starts 2024-09-11T03:45:11.251 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:10 smithi084 ceph-mon[27254]: 1.c scrub ok 2024-09-11T03:45:11.285 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:11.306 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:11.307 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:11.307 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:11.380 INFO:tasks.rados.rados.2.smithi175.stdout:1810: expect (ObjNum 496 snap 144 seq_num 496) 2024-09-11T03:45:11.409 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:11.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:10 smithi121 ceph-mon[30729]: 1.12 scrub starts 2024-09-11T03:45:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:10 smithi121 ceph-mon[30729]: 1.12 scrub ok 2024-09-11T03:45:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:10 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:45:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:10 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:45:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:10 smithi121 ceph-mon[30729]: from='client.16083 172.21.15.84:0/1314369143' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]': finished 2024-09-11T03:45:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/230431579' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:45:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:10 smithi121 ceph-mon[30729]: osdmap e492: 12 total, 12 up, 11 in 2024-09-11T03:45:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:10 smithi121 ceph-mon[30729]: 1.c scrub starts 2024-09-11T03:45:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:10 smithi121 ceph-mon[30729]: 1.c scrub ok 2024-09-11T03:45:11.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:11.432 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:11.432 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:10 smithi175 ceph-mon[30894]: 1.12 scrub starts 2024-09-11T03:45:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:10 smithi175 ceph-mon[30894]: 1.12 scrub ok 2024-09-11T03:45:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:10 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:45:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:10 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:45:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:10 smithi175 ceph-mon[30894]: from='client.16083 172.21.15.84:0/1314369143' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]': finished 2024-09-11T03:45:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:10 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/230431579' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:45:11.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:10 smithi175 ceph-mon[30894]: osdmap e492: 12 total, 12 up, 11 in 2024-09-11T03:45:11.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:10 smithi175 ceph-mon[30894]: 1.c scrub starts 2024-09-11T03:45:11.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:10 smithi175 ceph-mon[30894]: 1.c scrub ok 2024-09-11T03:45:11.487 INFO:tasks.rados.rados.2.smithi175.stdout:1811: finishing write tid 1 to smithi17557870-18 2024-09-11T03:45:11.487 INFO:tasks.rados.rados.2.smithi175.stdout:1809: done (3 left) 2024-09-11T03:45:11.487 INFO:tasks.rados.rados.2.smithi175.stdout:1810: done (2 left) 2024-09-11T03:45:11.488 INFO:tasks.rados.rados.2.smithi175.stdout:1813: read oid 44 snap -1 2024-09-11T03:45:11.488 INFO:tasks.rados.rados.2.smithi175.stdout:1813: expect (ObjNum 598 snap 186 seq_num 598) 2024-09-11T03:45:11.488 INFO:tasks.rados.rados.2.smithi175.stdout:1814: write oid 26 current snap is 199 2024-09-11T03:45:11.488 INFO:tasks.rados.rados.2.smithi175.stdout:1811: finishing write tid 2 to smithi17557870-18 2024-09-11T03:45:11.488 INFO:tasks.rados.rados.2.smithi175.stdout:1811: finishing write tid 3 to smithi17557870-18 2024-09-11T03:45:11.488 INFO:tasks.rados.rados.2.smithi175.stdout:1814: seq_num 624 ranges {487568=496322,1431705=519550} 2024-09-11T03:45:11.492 INFO:tasks.rados.rados.2.smithi175.stdout:1814: writing smithi17557870-26 from 487568 to 983890 tid 1 2024-09-11T03:45:11.498 INFO:tasks.rados.rados.2.smithi175.stdout:1814: writing smithi17557870-26 from 1431705 to 1951255 tid 2 2024-09-11T03:45:11.498 INFO:tasks.rados.rados.2.smithi175.stdout:1815: write (excl) oid 28 current snap is 199 2024-09-11T03:45:11.498 INFO:tasks.rados.rados.2.smithi175.stdout:1811: finishing write tid 4 to smithi17557870-18 2024-09-11T03:45:11.498 INFO:tasks.rados.rados.2.smithi175.stdout:1811: oid 18 updating version 0 to 470 2024-09-11T03:45:11.498 INFO:tasks.rados.rados.2.smithi175.stdout:1811: oid 18 version 470 is already newer than 468 2024-09-11T03:45:11.498 INFO:tasks.rados.rados.2.smithi175.stdout:1811: oid 18 version 470 is already newer than 469 2024-09-11T03:45:11.498 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 470 (ObjNum 623 snap 199 seq_num 623) dirty exists 2024-09-11T03:45:11.498 INFO:tasks.rados.rados.2.smithi175.stdout:1811: left oid 18 (ObjNum 623 snap 199 seq_num 623) 2024-09-11T03:45:11.498 INFO:tasks.rados.rados.2.smithi175.stdout:1812: finishing copy_from racing read to smithi17557870-19 2024-09-11T03:45:11.498 INFO:tasks.rados.rados.2.smithi175.stdout:1815: seq_num 625 ranges {504375=571571,1490187=663208} 2024-09-11T03:45:11.503 INFO:tasks.rados.rados.2.smithi175.stdout:1815: writing smithi17557870-28 from 504375 to 1075946 tid 1 2024-09-11T03:45:11.509 INFO:tasks.rados.rados.2.smithi175.stdout:1815: writing smithi17557870-28 from 1490187 to 2153395 tid 2 2024-09-11T03:45:11.510 INFO:tasks.rados.rados.2.smithi175.stdout:1811: done (4 left) 2024-09-11T03:45:11.510 INFO:tasks.rados.rados.2.smithi175.stdout:1816: snap_create 2024-09-11T03:45:11.510 INFO:tasks.rados.rados.2.smithi175.stdout:1812: finishing copy_from to smithi17557870-19 2024-09-11T03:45:11.510 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 509 (ObjNum 565 snap 172 seq_num 565) dirty exists 2024-09-11T03:45:11.510 INFO:tasks.rados.rados.2.smithi175.stdout:1813: expect (ObjNum 598 snap 186 seq_num 598) 2024-09-11T03:45:11.533 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:11.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:11.556 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:11.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:11.573 INFO:tasks.rados.rados.2.smithi175.stdout:1814: finishing write tid 1 to smithi17557870-26 2024-09-11T03:45:11.573 INFO:tasks.rados.rados.2.smithi175.stdout:1814: finishing write tid 2 to smithi17557870-26 2024-09-11T03:45:11.573 INFO:tasks.rados.rados.2.smithi175.stdout:1814: finishing write tid 3 to smithi17557870-26 2024-09-11T03:45:11.573 INFO:tasks.rados.rados.2.smithi175.stdout:1814: finishing write tid 4 to smithi17557870-26 2024-09-11T03:45:11.573 INFO:tasks.rados.rados.2.smithi175.stdout:1814: oid 26 updating version 0 to 470 2024-09-11T03:45:11.574 INFO:tasks.rados.rados.2.smithi175.stdout:1814: oid 26 version 470 is already newer than 469 2024-09-11T03:45:11.574 INFO:tasks.rados.rados.2.smithi175.stdout:1814: oid 26 version 470 is already newer than 468 2024-09-11T03:45:11.574 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 470 (ObjNum 624 snap 199 seq_num 624) dirty exists 2024-09-11T03:45:11.574 INFO:tasks.rados.rados.2.smithi175.stdout:1814: left oid 26 (ObjNum 624 snap 199 seq_num 624) 2024-09-11T03:45:11.574 INFO:tasks.rados.rados.2.smithi175.stdout:1815: finishing write tid 1 to smithi17557870-28 2024-09-11T03:45:11.574 INFO:tasks.rados.rados.2.smithi175.stdout:1815: finishing write tid 2 to smithi17557870-28 2024-09-11T03:45:11.575 INFO:tasks.rados.rados.2.smithi175.stdout:1815: finishing write tid 3 to smithi17557870-28 2024-09-11T03:45:11.575 INFO:tasks.rados.rados.2.smithi175.stdout:1815: finishing write tid 4 to smithi17557870-28 2024-09-11T03:45:11.575 INFO:tasks.rados.rados.2.smithi175.stdout:1815: oid 28 updating version 0 to 684 2024-09-11T03:45:11.575 INFO:tasks.rados.rados.2.smithi175.stdout:1815: oid 28 version 684 is already newer than 683 2024-09-11T03:45:11.575 INFO:tasks.rados.rados.2.smithi175.stdout:1815: oid 28 version 684 is already newer than 682 2024-09-11T03:45:11.575 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 684 (ObjNum 625 snap 199 seq_num 625) dirty exists 2024-09-11T03:45:11.575 INFO:tasks.rados.rados.2.smithi175.stdout:1815: left oid 28 (ObjNum 625 snap 199 seq_num 625) 2024-09-11T03:45:11.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:11.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:11.681 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:11.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:11.769 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:45:11.769 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:45:11.782 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:11.785 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:45:11.785 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:45:11.801 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:45:11.801 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:45:11.803 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:11.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:11.804 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:11.817 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:45:11.817 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:45:11.833 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:45:11.833 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:45:11.849 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:45:11.849 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:45:11.856 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T03:45:11.865 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:45:11.866 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:45:11.883 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:45:11.883 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:45:11.900 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:45:11.900 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:45:11.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:11.916 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:45:11.916 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:45:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:11.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:11.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:11.933 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:45:11.933 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:45:11.941 INFO:tasks.rados.rados.2.smithi175.stdout:1812: done (4 left) 2024-09-11T03:45:11.941 INFO:tasks.rados.rados.2.smithi175.stdout:1813: done (3 left) 2024-09-11T03:45:11.941 INFO:tasks.rados.rados.2.smithi175.stdout:1814: done (2 left) 2024-09-11T03:45:11.941 INFO:tasks.rados.rados.2.smithi175.stdout:1815: done (1 left) 2024-09-11T03:45:11.941 INFO:tasks.rados.rados.2.smithi175.stdout:1816: done (0 left) 2024-09-11T03:45:11.941 INFO:tasks.rados.rados.2.smithi175.stdout:1817: rollback oid 19 current snap is 200 2024-09-11T03:45:11.941 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 19 to 199 2024-09-11T03:45:11.941 INFO:tasks.rados.rados.2.smithi175.stdout:1818: write (excl) oid 33 current snap is 200 2024-09-11T03:45:11.941 INFO:tasks.rados.rados.2.smithi175.stdout:1818: seq_num 626 ranges {521182=646820,1948669=523219} 2024-09-11T03:45:11.947 INFO:tasks.rados.rados.2.smithi175.stdout:1818: writing smithi17557870-33 from 521182 to 1168002 tid 1 2024-09-11T03:45:11.953 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:45:11.953 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:45:11.955 INFO:tasks.rados.rados.2.smithi175.stdout:1818: writing smithi17557870-33 from 1948669 to 2471888 tid 2 2024-09-11T03:45:11.955 INFO:tasks.rados.rados.2.smithi175.stdout:1819: write oid 17 current snap is 200 2024-09-11T03:45:11.955 INFO:tasks.rados.rados.2.smithi175.stdout:1817: finishing rollback tid 0 to smithi17557870-19 2024-09-11T03:45:11.955 INFO:tasks.rados.rados.2.smithi175.stdout:1817: finishing rollback tid 1 to smithi17557870-19 2024-09-11T03:45:11.955 INFO:tasks.rados.rados.2.smithi175.stdout:1819: seq_num 627 ranges {537989=722069,1890798=647191} 2024-09-11T03:45:11.961 INFO:tasks.rados.rados.2.smithi175.stdout:1819: writing smithi17557870-17 from 537989 to 1260058 tid 1 2024-09-11T03:45:11.969 INFO:tasks.rados.rados.2.smithi175.stdout:1819: writing smithi17557870-17 from 1890798 to 2537989 tid 2 2024-09-11T03:45:11.969 INFO:tasks.rados.rados.2.smithi175.stdout:1820: snap_create 2024-09-11T03:45:11.969 INFO:tasks.rados.rados.2.smithi175.stdout:1817: finishing rollback tid 2 to smithi17557870-19 2024-09-11T03:45:11.969 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 512 (ObjNum 565 snap 172 seq_num 565) dirty exists 2024-09-11T03:45:11.969 INFO:tasks.rados.rados.2.smithi175.stdout:1818: finishing write tid 1 to smithi17557870-33 2024-09-11T03:45:11.969 INFO:tasks.rados.rados.2.smithi175.stdout:1818: finishing write tid 2 to smithi17557870-33 2024-09-11T03:45:11.969 INFO:tasks.rados.rados.2.smithi175.stdout:1818: finishing write tid 3 to smithi17557870-33 2024-09-11T03:45:11.969 INFO:tasks.rados.rados.2.smithi175.stdout:1818: finishing write tid 4 to smithi17557870-33 2024-09-11T03:45:11.969 INFO:tasks.rados.rados.2.smithi175.stdout:1818: oid 33 updating version 0 to 686 2024-09-11T03:45:11.969 INFO:tasks.rados.rados.2.smithi175.stdout:1818: oid 33 updating version 686 to 688 2024-09-11T03:45:11.969 INFO:tasks.rados.rados.2.smithi175.stdout:1818: oid 33 version 688 is already newer than 687 2024-09-11T03:45:11.969 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 688 (ObjNum 626 snap 200 seq_num 626) dirty exists 2024-09-11T03:45:11.969 INFO:tasks.rados.rados.2.smithi175.stdout:1818: left oid 33 (ObjNum 626 snap 200 seq_num 626) 2024-09-11T03:45:11.970 INFO:tasks.rados.rados.2.smithi175.stdout:1819: finishing write tid 1 to smithi17557870-17 2024-09-11T03:45:11.973 INFO:tasks.rados.rados.2.smithi175.stdout:1819: finishing write tid 2 to smithi17557870-17 2024-09-11T03:45:11.973 INFO:tasks.rados.rados.2.smithi175.stdout:1819: finishing write tid 3 to smithi17557870-17 2024-09-11T03:45:11.973 INFO:tasks.rados.rados.2.smithi175.stdout:1819: finishing write tid 4 to smithi17557870-17 2024-09-11T03:45:11.973 INFO:tasks.rados.rados.2.smithi175.stdout:1819: oid 17 updating version 0 to 669 2024-09-11T03:45:11.974 INFO:tasks.rados.rados.2.smithi175.stdout:1819: oid 17 version 669 is already newer than 668 2024-09-11T03:45:11.974 INFO:tasks.rados.rados.2.smithi175.stdout:1819: oid 17 updating version 669 to 670 2024-09-11T03:45:11.974 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 670 (ObjNum 627 snap 200 seq_num 627) dirty exists 2024-09-11T03:45:11.974 INFO:tasks.rados.rados.2.smithi175.stdout:1819: left oid 17 (ObjNum 627 snap 200 seq_num 627) 2024-09-11T03:45:12.028 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:12.049 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:12.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:12.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:12.104 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:45:12.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:11 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:45:12.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:11 smithi084 ceph-mon[27254]: 2.c scrub ok 2024-09-11T03:45:12.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:11 smithi084 ceph-mon[27254]: pgmap v894: 57 pgs: 57 active+clean; 158 MiB data, 726 MiB used, 983 GiB / 983 GiB avail; 511 B/s rd, 1023 KiB/s wr, 3 op/s 2024-09-11T03:45:12.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:11 smithi084 ceph-mon[27254]: osdmap e493: 12 total, 12 up, 11 in 2024-09-11T03:45:12.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:11 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3801731448' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:12.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:11 smithi084 ceph-mon[27254]: osdmap e494: 12 total, 12 up, 11 in 2024-09-11T03:45:12.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:12.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:11 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:45:12.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:11 smithi121 ceph-mon[30729]: 2.c scrub ok 2024-09-11T03:45:12.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:11 smithi121 ceph-mon[30729]: pgmap v894: 57 pgs: 57 active+clean; 158 MiB data, 726 MiB used, 983 GiB / 983 GiB avail; 511 B/s rd, 1023 KiB/s wr, 3 op/s 2024-09-11T03:45:12.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:11 smithi121 ceph-mon[30729]: osdmap e493: 12 total, 12 up, 11 in 2024-09-11T03:45:12.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:11 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3801731448' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:12.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:11 smithi121 ceph-mon[30729]: osdmap e494: 12 total, 12 up, 11 in 2024-09-11T03:45:12.172 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:12.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:12.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:12.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:12.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:12.297 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:12.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:12.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:12.419 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:12.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:12.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:11 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:45:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:11 smithi175 ceph-mon[30894]: 2.c scrub ok 2024-09-11T03:45:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:11 smithi175 ceph-mon[30894]: pgmap v894: 57 pgs: 57 active+clean; 158 MiB data, 726 MiB used, 983 GiB / 983 GiB avail; 511 B/s rd, 1023 KiB/s wr, 3 op/s 2024-09-11T03:45:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:11 smithi175 ceph-mon[30894]: osdmap e493: 12 total, 12 up, 11 in 2024-09-11T03:45:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:11 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3801731448' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:11 smithi175 ceph-mon[30894]: osdmap e494: 12 total, 12 up, 11 in 2024-09-11T03:45:12.521 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:12.543 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:12.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:12.597 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:45:12.597 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:45:12.597 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:45:12.597 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:45:12.598 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:45:12.645 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:12.670 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:12.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:12.671 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:12.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:45:12.771 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:12.793 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:12.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:12.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:12.895 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:12.921 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:12.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:12.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:12.932 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:12.944 INFO:tasks.rados.rados.2.smithi175.stdout:1817: done (3 left) 2024-09-11T03:45:12.944 INFO:tasks.rados.rados.2.smithi175.stdout:1818: done (2 left) 2024-09-11T03:45:12.944 INFO:tasks.rados.rados.2.smithi175.stdout:1819: done (1 left) 2024-09-11T03:45:12.944 INFO:tasks.rados.rados.2.smithi175.stdout:1820: done (0 left) 2024-09-11T03:45:12.944 INFO:tasks.rados.rados.2.smithi175.stdout:1821: rollback oid 48 current snap is 201 2024-09-11T03:45:12.944 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 48 to 197 2024-09-11T03:45:12.944 INFO:tasks.rados.rados.2.smithi175.stdout:1822: rollback oid 46 current snap is 201 2024-09-11T03:45:12.944 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 46 to 195 2024-09-11T03:45:12.944 INFO:tasks.rados.rados.2.smithi175.stdout:1823: delete oid 15 current snap is 201 2024-09-11T03:45:12.947 INFO:tasks.rados.rados.2.smithi175.stdout:1821: finishing rollback tid 0 to smithi17557870-48 2024-09-11T03:45:12.948 INFO:tasks.rados.rados.2.smithi175.stdout:1822: finishing rollback tid 0 to smithi17557870-46 2024-09-11T03:45:12.948 INFO:tasks.rados.rados.2.smithi175.stdout:1823: done (2 left) 2024-09-11T03:45:12.948 INFO:tasks.rados.rados.2.smithi175.stdout:1824: read oid 3 snap 196 2024-09-11T03:45:12.949 INFO:tasks.rados.rados.2.smithi175.stdout:1824: expect (ObjNum 611 snap 194 seq_num 611) 2024-09-11T03:45:12.949 INFO:tasks.rados.rados.2.smithi175.stdout:1825: read oid 27 snap -1 2024-09-11T03:45:12.949 INFO:tasks.rados.rados.2.smithi175.stdout:1825: expect (ObjNum 577 snap 177 seq_num 577) 2024-09-11T03:45:12.949 INFO:tasks.rados.rados.2.smithi175.stdout:1826: snap_create 2024-09-11T03:45:12.949 INFO:tasks.rados.rados.2.smithi175.stdout:1821: finishing rollback tid 1 to smithi17557870-48 2024-09-11T03:45:12.949 INFO:tasks.rados.rados.2.smithi175.stdout:1821: finishing rollback tid 2 to smithi17557870-48 2024-09-11T03:45:12.949 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 497 (ObjNum 562 snap 171 seq_num 562) dirty exists 2024-09-11T03:45:12.949 INFO:tasks.rados.rados.2.smithi175.stdout:1822: finishing rollback tid 1 to smithi17557870-46 2024-09-11T03:45:12.949 INFO:tasks.rados.rados.2.smithi175.stdout:1822: finishing rollback tid 2 to smithi17557870-46 2024-09-11T03:45:12.949 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 531 (ObjNum 604 snap 189 seq_num 604) dirty exists 2024-09-11T03:45:12.955 INFO:tasks.rados.rados.2.smithi175.stdout:1825: expect (ObjNum 577 snap 177 seq_num 577) 2024-09-11T03:45:13.023 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:13.045 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:13.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:13.119 INFO:tasks.rados.rados.2.smithi175.stdout:1824: expect (ObjNum 611 snap 194 seq_num 611) 2024-09-11T03:45:13.146 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:13.168 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:13.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:13.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:13.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T03:45:13.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:13.329 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:13.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:13.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:13.431 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:13.454 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:13.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:13.508 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:45:13.556 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:13.577 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:13.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:13.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:13.679 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:13.700 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:13.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:13.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:13.802 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:13.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:13.826 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:13.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:13.927 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:13.944 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:45:13.949 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:13.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:13.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:13.950 INFO:tasks.rados.rados.2.smithi175.stdout:1821: done (4 left) 2024-09-11T03:45:13.950 INFO:tasks.rados.rados.2.smithi175.stdout:1822: done (3 left) 2024-09-11T03:45:13.950 INFO:tasks.rados.rados.2.smithi175.stdout:1824: done (2 left) 2024-09-11T03:45:13.951 INFO:tasks.rados.rados.2.smithi175.stdout:1825: done (1 left) 2024-09-11T03:45:13.951 INFO:tasks.rados.rados.2.smithi175.stdout:1826: done (0 left) 2024-09-11T03:45:13.951 INFO:tasks.rados.rados.2.smithi175.stdout:1827: delete oid 11 current snap is 202 2024-09-11T03:45:13.953 INFO:tasks.rados.rados.2.smithi175.stdout:1827: done (0 left) 2024-09-11T03:45:13.954 INFO:tasks.rados.rados.2.smithi175.stdout:1828: snap_remove snap 201 2024-09-11T03:45:14.008 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:45:14.008 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:45:14.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:45:14.008 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:45:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:221 2.3 2:c6c49f2c:::smithi17557870-44:head [copy-from ver 9] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:45:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941456+0000", 2024-09-11T03:45:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 394.06573575300001, 2024-09-11T03:45:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015312882, 2024-09-11T03:45:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 221 2024-09-11T03:45:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941456+0000", 2024-09-11T03:45:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941456+0000", 2024-09-11T03:45:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941459+0000", 2024-09-11T03:45:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9189999999999999e-06 2024-09-11T03:45:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941469+0000", 2024-09-11T03:45:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0069e-05 2024-09-11T03:45:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941490+0000", 2024-09-11T03:45:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0947000000000001e-05 2024-09-11T03:45:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941501+0000", 2024-09-11T03:45:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0169e-05 2024-09-11T03:45:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941585+0000", 2024-09-11T03:45:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4022000000000003e-05 2024-09-11T03:45:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:45:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941590+0000", 2024-09-11T03:45:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4929999999999998e-06 2024-09-11T03:45:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.942889+0000", 2024-09-11T03:45:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001298539 2024-09-11T03:45:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.942984+0000", 2024-09-11T03:45:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5513999999999994e-05 2024-09-11T03:45:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:45:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.950816+0000", 2024-09-11T03:45:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078315850000000003 2024-09-11T03:45:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956638+0000", 2024-09-11T03:45:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058225769999999998 2024-09-11T03:45:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:45:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956649+0000", 2024-09-11T03:45:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0492000000000001e-05 2024-09-11T03:45:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956690+0000", 2024-09-11T03:45:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.142e-05 2024-09-11T03:45:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956728+0000", 2024-09-11T03:45:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8195000000000003e-05 2024-09-11T03:45:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956769+0000", 2024-09-11T03:45:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0941e-05 2024-09-11T03:45:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:222 2.3 2:c6c49f2c:::smithi17557870-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:45:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:45:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 394.06565767400002, 2024-09-11T03:45:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015032419, 2024-09-11T03:45:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:45:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 222 2024-09-11T03:45:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:45:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:45:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:45:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.51e-07 2024-09-11T03:45:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941539+0000", 2024-09-11T03:45:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0060000000000003e-06 2024-09-11T03:45:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941547+0000", 2024-09-11T03:45:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9009999999999994e-06 2024-09-11T03:45:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941553+0000", 2024-09-11T03:45:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1519999999999997e-06 2024-09-11T03:45:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941691+0000", 2024-09-11T03:45:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013752600000000001 2024-09-11T03:45:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941697+0000", 2024-09-11T03:45:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3770000000000001e-06 2024-09-11T03:45:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943170+0000", 2024-09-11T03:45:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001473504 2024-09-11T03:45:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943225+0000", 2024-09-11T03:45:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4101000000000003e-05 2024-09-11T03:45:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956388+0000", 2024-09-11T03:45:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013163223999999999 2024-09-11T03:45:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956456+0000", 2024-09-11T03:45:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8563000000000001e-05 2024-09-11T03:45:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956567+0000", 2024-09-11T03:45:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011061399999999999 2024-09-11T03:45:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:223 2.3 2:c3768452:::smithi17557870-11:head [copy-from ver 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:45:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:45:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 394.06561762899997, 2024-09-11T03:45:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016727301999999999, 2024-09-11T03:45:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 223 2024-09-11T03:45:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:45:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:45:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:45:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0999999999999999e-07 2024-09-11T03:45:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941578+0000", 2024-09-11T03:45:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.021e-06 2024-09-11T03:45:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941583+0000", 2024-09-11T03:45:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0520000000000001e-06 2024-09-11T03:45:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941586+0000", 2024-09-11T03:45:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-09-11T03:45:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941750+0000", 2024-09-11T03:45:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016353200000000001 2024-09-11T03:45:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941756+0000", 2024-09-11T03:45:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.75e-06 2024-09-11T03:45:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943289+0000", 2024-09-11T03:45:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001533747 2024-09-11T03:45:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943383+0000", 2024-09-11T03:45:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3655999999999999e-05 2024-09-11T03:45:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:45:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.954899+0000", 2024-09-11T03:45:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011515463 2024-09-11T03:45:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:45:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.957952+0000", 2024-09-11T03:45:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030533560000000001 2024-09-11T03:45:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958200+0000", 2024-09-11T03:45:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024759100000000002 2024-09-11T03:45:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958229+0000", 2024-09-11T03:45:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9862e-05 2024-09-11T03:45:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958262+0000", 2024-09-11T03:45:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2267000000000001e-05 2024-09-11T03:45:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958302+0000", 2024-09-11T03:45:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0204000000000003e-05 2024-09-11T03:45:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:224 2.3 2:c3768452:::smithi17557870-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:45:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941599+0000", 2024-09-11T03:45:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 394.06559307700002, 2024-09-11T03:45:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014750406000000001, 2024-09-11T03:45:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:45:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 224 2024-09-11T03:45:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941599+0000", 2024-09-11T03:45:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941599+0000", 2024-09-11T03:45:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941599+0000", 2024-09-11T03:45:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2599999999999998e-07 2024-09-11T03:45:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941601+0000", 2024-09-11T03:45:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6249999999999999e-06 2024-09-11T03:45:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941605+0000", 2024-09-11T03:45:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3059999999999999e-06 2024-09-11T03:45:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941609+0000", 2024-09-11T03:45:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.506e-06 2024-09-11T03:45:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941789+0000", 2024-09-11T03:45:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000180261 2024-09-11T03:45:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941795+0000", 2024-09-11T03:45:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4970000000000001e-06 2024-09-11T03:45:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943572+0000", 2024-09-11T03:45:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001777725 2024-09-11T03:45:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943616+0000", 2024-09-11T03:45:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3550999999999997e-05 2024-09-11T03:45:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956192+0000", 2024-09-11T03:45:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012576517000000001 2024-09-11T03:45:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956267+0000", 2024-09-11T03:45:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4956000000000005e-05 2024-09-11T03:45:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956350+0000", 2024-09-11T03:45:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2136000000000001e-05 2024-09-11T03:45:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:343 2.2 2:494ba932:::smithi17557870-43:head [copy-from ver 6] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:45:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.820776+0000", 2024-09-11T03:45:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 384.18641648699997, 2024-09-11T03:45:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017461323000000001, 2024-09-11T03:45:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 343 2024-09-11T03:45:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820776+0000", 2024-09-11T03:45:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820776+0000", 2024-09-11T03:45:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820779+0000", 2024-09-11T03:45:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3000000000000002e-06 2024-09-11T03:45:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820797+0000", 2024-09-11T03:45:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7730000000000001e-05 2024-09-11T03:45:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820853+0000", 2024-09-11T03:45:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5909000000000003e-05 2024-09-11T03:45:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820872+0000", 2024-09-11T03:45:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9709999999999999e-05 2024-09-11T03:45:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820976+0000", 2024-09-11T03:45:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010374099999999999 2024-09-11T03:45:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:45:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820982+0000", 2024-09-11T03:45:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4389999999999999e-06 2024-09-11T03:45:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825547+0000", 2024-09-11T03:45:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045658260000000003 2024-09-11T03:45:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825626+0000", 2024-09-11T03:45:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8289999999999998e-05 2024-09-11T03:45:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,6", 2024-09-11T03:45:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830895+0000", 2024-09-11T03:45:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052695060000000002 2024-09-11T03:45:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:45:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.835040+0000", 2024-09-11T03:45:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041446549999999997 2024-09-11T03:45:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.835364+0000", 2024-09-11T03:45:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032384200000000002 2024-09-11T03:45:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838162+0000", 2024-09-11T03:45:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027987509999999999 2024-09-11T03:45:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838203+0000", 2024-09-11T03:45:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0209000000000001e-05 2024-09-11T03:45:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838237+0000", 2024-09-11T03:45:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4415000000000002e-05 2024-09-11T03:45:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1068 2.3 2:c3768452:::smithi17557870-11:head [write 507611~777843,stat] snapc 28=[28,27,26] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:45:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.871196+0000", 2024-09-11T03:45:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 322.135996439, 2024-09-11T03:45:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.036362196999999999, 2024-09-11T03:45:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1068 2024-09-11T03:45:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.871196+0000", 2024-09-11T03:45:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.871196+0000", 2024-09-11T03:45:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.871198+0000", 2024-09-11T03:45:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5000000000000002e-06 2024-09-11T03:45:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873403+0000", 2024-09-11T03:45:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022048739999999999 2024-09-11T03:45:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873438+0000", 2024-09-11T03:45:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4518999999999999e-05 2024-09-11T03:45:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873453+0000", 2024-09-11T03:45:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5733999999999999e-05 2024-09-11T03:45:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873597+0000", 2024-09-11T03:45:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143702 2024-09-11T03:45:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873668+0000", 2024-09-11T03:45:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0996999999999998e-05 2024-09-11T03:45:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:45:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873819+0000", 2024-09-11T03:45:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015105300000000001 2024-09-11T03:45:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:45:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.875607+0000", 2024-09-11T03:45:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017877419999999999 2024-09-11T03:45:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.875920+0000", 2024-09-11T03:45:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031312799999999997 2024-09-11T03:45:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907471+0000", 2024-09-11T03:45:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.031551348 2024-09-11T03:45:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907514+0000", 2024-09-11T03:45:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2873000000000001e-05 2024-09-11T03:45:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907558+0000", 2024-09-11T03:45:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3727000000000003e-05 2024-09-11T03:45:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1069 2.3 2:c3768452:::smithi17557870-11:head [write 2073973~444721] snapc 28=[28,27,26] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:45:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.877745+0000", 2024-09-11T03:45:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 322.12944696199997, 2024-09-11T03:45:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029936307999999998, 2024-09-11T03:45:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1069 2024-09-11T03:45:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.877745+0000", 2024-09-11T03:45:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.877745+0000", 2024-09-11T03:45:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.877747+0000", 2024-09-11T03:45:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.61e-06 2024-09-11T03:45:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878169+0000", 2024-09-11T03:45:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00042167499999999999 2024-09-11T03:45:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878181+0000", 2024-09-11T03:45:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2714999999999999e-05 2024-09-11T03:45:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878187+0000", 2024-09-11T03:45:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2389999999999999e-06 2024-09-11T03:45:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878284+0000", 2024-09-11T03:45:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6632000000000006e-05 2024-09-11T03:45:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878363+0000", 2024-09-11T03:45:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9369000000000005e-05 2024-09-11T03:45:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:45:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878472+0000", 2024-09-11T03:45:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108995 2024-09-11T03:45:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:45:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879469+0000", 2024-09-11T03:45:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00099629100000000006 2024-09-11T03:45:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879835+0000", 2024-09-11T03:45:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036609700000000002 2024-09-11T03:45:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907609+0000", 2024-09-11T03:45:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027773923999999998 2024-09-11T03:45:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907647+0000", 2024-09-11T03:45:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8124000000000003e-05 2024-09-11T03:45:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907682+0000", 2024-09-11T03:45:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4637000000000001e-05 2024-09-11T03:45:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1070 2.3 2:c3768452:::smithi17557870-11:head [setxattr _header (57) in=7b,truncate 2907611] snapc 28=[28,27,26] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:45:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.878201+0000", 2024-09-11T03:45:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 322.12899116599999, 2024-09-11T03:45:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029581774000000002, 2024-09-11T03:45:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1070 2024-09-11T03:45:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878201+0000", 2024-09-11T03:45:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878201+0000", 2024-09-11T03:45:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878201+0000", 2024-09-11T03:45:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4299999999999998e-07 2024-09-11T03:45:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878222+0000", 2024-09-11T03:45:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0591999999999999e-05 2024-09-11T03:45:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878228+0000", 2024-09-11T03:45:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.857e-06 2024-09-11T03:45:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878232+0000", 2024-09-11T03:45:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6459999999999999e-06 2024-09-11T03:45:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879123+0000", 2024-09-11T03:45:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089174700000000003 2024-09-11T03:45:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879183+0000", 2024-09-11T03:45:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9502000000000001e-05 2024-09-11T03:45:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:45:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879287+0000", 2024-09-11T03:45:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103952 2024-09-11T03:45:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:45:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879606+0000", 2024-09-11T03:45:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031872600000000002 2024-09-11T03:45:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879933+0000", 2024-09-11T03:45:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032754700000000003 2024-09-11T03:45:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907711+0000", 2024-09-11T03:45:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027778265999999999 2024-09-11T03:45:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907746+0000", 2024-09-11T03:45:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4403999999999998e-05 2024-09-11T03:45:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907783+0000", 2024-09-11T03:45:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7092000000000003e-05 2024-09-11T03:45:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1071 2.3 2:c3768452:::smithi17557870-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:45:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.878273+0000", 2024-09-11T03:45:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 322.12891951099999, 2024-09-11T03:45:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.02966705, 2024-09-11T03:45:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:45:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1071 2024-09-11T03:45:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878273+0000", 2024-09-11T03:45:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878273+0000", 2024-09-11T03:45:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878273+0000", 2024-09-11T03:45:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1600000000000002e-07 2024-09-11T03:45:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878275+0000", 2024-09-11T03:45:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0949999999999998e-06 2024-09-11T03:45:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878281+0000", 2024-09-11T03:45:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6550000000000001e-06 2024-09-11T03:45:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878284+0000", 2024-09-11T03:45:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5370000000000002e-06 2024-09-11T03:45:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879510+0000", 2024-09-11T03:45:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001225638 2024-09-11T03:45:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:45:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879586+0000", 2024-09-11T03:45:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6058999999999998e-05 2024-09-11T03:45:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907811+0000", 2024-09-11T03:45:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028224775000000001 2024-09-11T03:45:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907873+0000", 2024-09-11T03:45:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2310000000000005e-05 2024-09-11T03:45:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907940+0000", 2024-09-11T03:45:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6564999999999997e-05 2024-09-11T03:45:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1555 2.3 2:c6c49f2c:::smithi17557870-44:head [copy-from ver 82] snapc 3c=[3c,3a,38,37] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-11T03:45:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:30.880293+0000", 2024-09-11T03:45:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 283.12689948799999, 2024-09-11T03:45:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020483438999999999, 2024-09-11T03:45:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1555 2024-09-11T03:45:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880293+0000", 2024-09-11T03:45:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880293+0000", 2024-09-11T03:45:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880295+0000", 2024-09-11T03:45:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2409999999999998e-06 2024-09-11T03:45:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880305+0000", 2024-09-11T03:45:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5999999999999996e-06 2024-09-11T03:45:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880326+0000", 2024-09-11T03:45:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1248999999999999e-05 2024-09-11T03:45:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880335+0000", 2024-09-11T03:45:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4369999999999996e-06 2024-09-11T03:45:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880414+0000", 2024-09-11T03:45:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8560999999999995e-05 2024-09-11T03:45:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:45:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880422+0000", 2024-09-11T03:45:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.712e-06 2024-09-11T03:45:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887559+0000", 2024-09-11T03:45:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071376920000000002 2024-09-11T03:45:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887625+0000", 2024-09-11T03:45:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6256999999999999e-05 2024-09-11T03:45:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:45:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.893371+0000", 2024-09-11T03:45:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0057455579999999996 2024-09-11T03:45:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.899577+0000", 2024-09-11T03:45:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062060600000000002 2024-09-11T03:45:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:45:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900444+0000", 2024-09-11T03:45:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000867307 2024-09-11T03:45:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900703+0000", 2024-09-11T03:45:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00025901299999999999 2024-09-11T03:45:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900743+0000", 2024-09-11T03:45:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9471000000000002e-05 2024-09-11T03:45:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900776+0000", 2024-09-11T03:45:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3281000000000002e-05 2024-09-11T03:45:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1556 2.3 2:c6c49f2c:::smithi17557870-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e214)", 2024-09-11T03:45:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:30.880359+0000", 2024-09-11T03:45:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 283.12683279100003, 2024-09-11T03:45:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017105065999999999, 2024-09-11T03:45:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:45:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1556 2024-09-11T03:45:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880359+0000", 2024-09-11T03:45:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880359+0000", 2024-09-11T03:45:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880360+0000", 2024-09-11T03:45:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.75e-07 2024-09-11T03:45:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880362+0000", 2024-09-11T03:45:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4959999999999999e-06 2024-09-11T03:45:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880370+0000", 2024-09-11T03:45:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.153e-06 2024-09-11T03:45:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880376+0000", 2024-09-11T03:45:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0900000000000001e-06 2024-09-11T03:45:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880478+0000", 2024-09-11T03:45:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102482 2024-09-11T03:45:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880487+0000", 2024-09-11T03:45:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8109999999999992e-06 2024-09-11T03:45:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887725+0000", 2024-09-11T03:45:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072383930000000001 2024-09-11T03:45:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887757+0000", 2024-09-11T03:45:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1692000000000001e-05 2024-09-11T03:45:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.897328+0000", 2024-09-11T03:45:14.083 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:13 smithi121 ceph-mon[30729]: 1.7 scrub starts 2024-09-11T03:45:14.084 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:13 smithi121 ceph-mon[30729]: 1.7 scrub ok 2024-09-11T03:45:14.084 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:13 smithi121 ceph-mon[30729]: pgmap v897: 57 pgs: 57 active+clean; 181 MiB data, 752 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 4.7 MiB/s wr, 24 op/s 2024-09-11T03:45:14.084 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:13 smithi121 ceph-mon[30729]: 2.3 scrub starts 2024-09-11T03:45:14.084 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:13 smithi121 ceph-mon[30729]: 2.3 scrub ok 2024-09-11T03:45:14.084 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:13 smithi121 ceph-mon[30729]: osdmap e495: 12 total, 12 up, 11 in 2024-09-11T03:45:14.084 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:13 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3176185444' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:45:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095712149999999992 2024-09-11T03:45:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.897393+0000", 2024-09-11T03:45:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5005000000000005e-05 2024-09-11T03:45:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.897464+0000", 2024-09-11T03:45:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1254000000000006e-05 2024-09-11T03:45:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3040 2.d 2:b6a25a79:::smithi17557870-47:head [read 0~2201783,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:45:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:45:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 142.11021256399999, 2024-09-11T03:45:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020060432, 2024-09-11T03:45:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:45:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.087 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.087 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.087 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3040 2024-09-11T03:45:14.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.087 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:45:14.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:45:14.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896981+0000", 2024-09-11T03:45:14.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.747e-06 2024-09-11T03:45:14.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896989+0000", 2024-09-11T03:45:14.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9559999999999997e-06 2024-09-11T03:45:14.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897005+0000", 2024-09-11T03:45:14.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5704000000000001e-05 2024-09-11T03:45:14.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897013+0000", 2024-09-11T03:45:14.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.028e-06 2024-09-11T03:45:14.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897078+0000", 2024-09-11T03:45:14.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4813000000000004e-05 2024-09-11T03:45:14.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:45:14.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897083+0000", 2024-09-11T03:45:14.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1109999999999999e-06 2024-09-11T03:45:14.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.914637+0000", 2024-09-11T03:45:14.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017553701000000001 2024-09-11T03:45:14.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.914744+0000", 2024-09-11T03:45:14.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107131 2024-09-11T03:45:14.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917040+0000", 2024-09-11T03:45:14.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022962410000000001 2024-09-11T03:45:14.091 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.091 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.091 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.091 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3041 2.d 2:b6a25a79:::smithi17557870-47:head [sparse-read 0~1480887] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:45:14.091 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:45:14.091 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 142.11014700800001, 2024-09-11T03:45:14.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020163026000000001, 2024-09-11T03:45:14.092 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.092 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.092 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:45:14.092 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.092 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.092 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.092 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3041 2024-09-11T03:45:14.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.092 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:45:14.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:45:14.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897046+0000", 2024-09-11T03:45:14.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7e-07 2024-09-11T03:45:14.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897048+0000", 2024-09-11T03:45:14.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0700000000000001e-06 2024-09-11T03:45:14.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897058+0000", 2024-09-11T03:45:14.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0151e-05 2024-09-11T03:45:14.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897068+0000", 2024-09-11T03:45:14.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0329000000000001e-05 2024-09-11T03:45:14.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897128+0000", 2024-09-11T03:45:14.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0208999999999999e-05 2024-09-11T03:45:14.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:14.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897132+0000", 2024-09-11T03:45:14.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7290000000000002e-06 2024-09-11T03:45:14.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917070+0000", 2024-09-11T03:45:14.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019937951999999998 2024-09-11T03:45:14.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917123+0000", 2024-09-11T03:45:14.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3149000000000002e-05 2024-09-11T03:45:14.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917208+0000", 2024-09-11T03:45:14.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5067000000000004e-05 2024-09-11T03:45:14.096 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.096 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.096 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.096 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3042 2.d 2:b6a25a79:::smithi17557870-47:head [sparse-read 0~1480887] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:45:14.096 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:45:14.097 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 142.110112829, 2024-09-11T03:45:14.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020302042999999999, 2024-09-11T03:45:14.097 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.097 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.097 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:45:14.097 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.097 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.097 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.097 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3042 2024-09-11T03:45:14.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.097 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:45:14.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:45:14.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897080+0000", 2024-09-11T03:45:14.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.82e-07 2024-09-11T03:45:14.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897082+0000", 2024-09-11T03:45:14.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.418e-06 2024-09-11T03:45:14.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897089+0000", 2024-09-11T03:45:14.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8090000000000004e-06 2024-09-11T03:45:14.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897094+0000", 2024-09-11T03:45:14.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2399999999999998e-06 2024-09-11T03:45:14.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:45:14.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0928999999999999e-05 2024-09-11T03:45:14.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:14.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897158+0000", 2024-09-11T03:45:14.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9890000000000001e-06 2024-09-11T03:45:14.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917227+0000", 2024-09-11T03:45:14.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020069019 2024-09-11T03:45:14.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917290+0000", 2024-09-11T03:45:14.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2848999999999994e-05 2024-09-11T03:45:14.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917381+0000", 2024-09-11T03:45:14.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1308000000000001e-05 2024-09-11T03:45:14.101 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.101 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.101 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.102 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3046 2.d 2:b6874adf:::smithi17557870-17:head [sparse-read 0~1769472,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:45:14.102 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:45:14.102 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 142.11009086799999, 2024-09-11T03:45:14.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021672665000000001, 2024-09-11T03:45:14.102 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.102 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.102 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:45:14.102 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.102 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.102 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.102 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3046 2024-09-11T03:45:14.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.102 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:45:14.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:45:14.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897102+0000", 2024-09-11T03:45:14.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.65e-07 2024-09-11T03:45:14.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897105+0000", 2024-09-11T03:45:14.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1820000000000002e-06 2024-09-11T03:45:14.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897111+0000", 2024-09-11T03:45:14.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4330000000000001e-06 2024-09-11T03:45:14.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897117+0000", 2024-09-11T03:45:14.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2979999999999999e-06 2024-09-11T03:45:14.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897179+0000", 2024-09-11T03:45:14.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2825999999999998e-05 2024-09-11T03:45:14.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:14.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897182+0000", 2024-09-11T03:45:14.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7769999999999998e-06 2024-09-11T03:45:14.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917399+0000", 2024-09-11T03:45:14.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020216622 2024-09-11T03:45:14.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917444+0000", 2024-09-11T03:45:14.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5558e-05 2024-09-11T03:45:14.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918774+0000", 2024-09-11T03:45:14.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013296040000000001 2024-09-11T03:45:14.107 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.107 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.107 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.107 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3047 2.d 2:b6874adf:::smithi17557870-17:head [read 0~2453888,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:45:14.107 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:45:14.107 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 142.110061696, 2024-09-11T03:45:14.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022691901, 2024-09-11T03:45:14.107 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.107 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.108 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:45:14.108 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.108 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.108 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.108 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3047 2024-09-11T03:45:14.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.108 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:45:14.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:45:14.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:45:14.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8599999999999998e-07 2024-09-11T03:45:14.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897134+0000", 2024-09-11T03:45:14.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1470000000000001e-06 2024-09-11T03:45:14.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897141+0000", 2024-09-11T03:45:14.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8029999999999999e-06 2024-09-11T03:45:14.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897147+0000", 2024-09-11T03:45:14.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4520000000000002e-06 2024-09-11T03:45:14.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897202+0000", 2024-09-11T03:45:14.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4255999999999997e-05 2024-09-11T03:45:14.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:14.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897204+0000", 2024-09-11T03:45:14.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6649999999999999e-06 2024-09-11T03:45:14.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918807+0000", 2024-09-11T03:45:14.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021602962999999999 2024-09-11T03:45:14.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918887+0000", 2024-09-11T03:45:14.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9550999999999998e-05 2024-09-11T03:45:14.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919822+0000", 2024-09-11T03:45:14.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093557799999999995 2024-09-11T03:45:14.112 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.112 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.112 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.112 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3048 2.d 2:b6874adf:::smithi17557870-17:head [read 0~2453888,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:45:14.112 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:45:14.112 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 142.11003687199999, 2024-09-11T03:45:14.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.023274353000000001, 2024-09-11T03:45:14.113 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.113 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.113 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:45:14.113 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.113 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.113 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.113 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3048 2024-09-11T03:45:14.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.113 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:45:14.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:45:14.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897156+0000", 2024-09-11T03:45:14.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1199999999999998e-07 2024-09-11T03:45:14.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897159+0000", 2024-09-11T03:45:14.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7439999999999999e-06 2024-09-11T03:45:14.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897165+0000", 2024-09-11T03:45:14.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6749999999999996e-06 2024-09-11T03:45:14.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897170+0000", 2024-09-11T03:45:14.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9420000000000003e-06 2024-09-11T03:45:14.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897226+0000", 2024-09-11T03:45:14.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6026999999999997e-05 2024-09-11T03:45:14.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:14.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897232+0000", 2024-09-11T03:45:14.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2609999999999999e-06 2024-09-11T03:45:14.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919841+0000", 2024-09-11T03:45:14.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022609054999999999 2024-09-11T03:45:14.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919885+0000", 2024-09-11T03:45:14.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3383999999999997e-05 2024-09-11T03:45:14.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.920430+0000", 2024-09-11T03:45:14.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054485299999999998 2024-09-11T03:45:14.117 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.117 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.117 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.117 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3879 2.2 2:43d79ac9:::smithi17557870-37:head [copy-from ver 461] snapc a2=[a2,a1,9f,9e,9d,97,90,8a] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-11T03:45:14.118 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:43:52.918307+0000", 2024-09-11T03:45:14.118 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 81.088884930999996, 2024-09-11T03:45:14.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014449813000000001, 2024-09-11T03:45:14.118 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.118 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.118 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:14.118 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.118 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.118 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.118 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3879 2024-09-11T03:45:14.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.118 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.918307+0000", 2024-09-11T03:45:14.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.918307+0000", 2024-09-11T03:45:14.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.918309+0000", 2024-09-11T03:45:14.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.61e-06 2024-09-11T03:45:14.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.918315+0000", 2024-09-11T03:45:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9909999999999999e-06 2024-09-11T03:45:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.918328+0000", 2024-09-11T03:45:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3429e-05 2024-09-11T03:45:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.918337+0000", 2024-09-11T03:45:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7309999999999999e-06 2024-09-11T03:45:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.918415+0000", 2024-09-11T03:45:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7922000000000004e-05 2024-09-11T03:45:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.918554+0000", 2024-09-11T03:45:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000138591 2024-09-11T03:45:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,11", 2024-09-11T03:45:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.924780+0000", 2024-09-11T03:45:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062259560000000004 2024-09-11T03:45:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:45:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.929082+0000", 2024-09-11T03:45:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043024020000000003 2024-09-11T03:45:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.930686+0000", 2024-09-11T03:45:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016045359999999999 2024-09-11T03:45:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.932683+0000", 2024-09-11T03:45:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019961810000000001 2024-09-11T03:45:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.932723+0000", 2024-09-11T03:45:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0046999999999998e-05 2024-09-11T03:45:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:52.932757+0000", 2024-09-11T03:45:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4416999999999999e-05 2024-09-11T03:45:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4176 2.3 2:c6c49f2c:::smithi17557870-44:head [copy-from ver 412] snapc b5=[b5,b4,b2,b1,af,ae,ac,ab,a9,a8,9d] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-11T03:45:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:28.933487+0000", 2024-09-11T03:45:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 45.073705664999999, 2024-09-11T03:45:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018627511999999999, 2024-09-11T03:45:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4176 2024-09-11T03:45:14.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.125 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933487+0000", 2024-09-11T03:45:14.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933487+0000", 2024-09-11T03:45:14.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933489+0000", 2024-09-11T03:45:14.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2510000000000001e-06 2024-09-11T03:45:14.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933498+0000", 2024-09-11T03:45:14.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3219999999999999e-06 2024-09-11T03:45:14.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933516+0000", 2024-09-11T03:45:14.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7971000000000001e-05 2024-09-11T03:45:14.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933529+0000", 2024-09-11T03:45:14.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2476e-05 2024-09-11T03:45:14.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933657+0000", 2024-09-11T03:45:14.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000128842 2024-09-11T03:45:14.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:45:14.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933663+0000", 2024-09-11T03:45:14.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9580000000000004e-06 2024-09-11T03:45:14.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935429+0000", 2024-09-11T03:45:14.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001765146 2024-09-11T03:45:14.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935549+0000", 2024-09-11T03:45:14.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120504 2024-09-11T03:45:14.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:45:14.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.944235+0000", 2024-09-11T03:45:14.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086856499999999996 2024-09-11T03:45:14.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:45:14.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948823+0000", 2024-09-11T03:45:14.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045879570000000001 2024-09-11T03:45:14.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.951466+0000", 2024-09-11T03:45:14.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026434739999999998 2024-09-11T03:45:14.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:14.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.952026+0000", 2024-09-11T03:45:14.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055985599999999996 2024-09-11T03:45:14.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:14.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.952068+0000", 2024-09-11T03:45:14.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1918000000000001e-05 2024-09-11T03:45:14.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.952114+0000", 2024-09-11T03:45:14.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6186999999999997e-05 2024-09-11T03:45:14.130 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.130 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.130 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.131 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4177 2.3 2:c6c49f2c:::smithi17557870-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e451)", 2024-09-11T03:45:14.131 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:28.933561+0000", 2024-09-11T03:45:14.131 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 45.073630852000001, 2024-09-11T03:45:14.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014798686, 2024-09-11T03:45:14.131 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:14.131 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:14.131 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:45:14.131 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:14.131 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:14.131 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:14.131 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4177 2024-09-11T03:45:14.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.132 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:14.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:14.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933561+0000", 2024-09-11T03:45:14.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:14.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933561+0000", 2024-09-11T03:45:14.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:14.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:14.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933562+0000", 2024-09-11T03:45:14.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4600000000000005e-07 2024-09-11T03:45:14.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:14.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933565+0000", 2024-09-11T03:45:14.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8729999999999999e-06 2024-09-11T03:45:14.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:14.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933572+0000", 2024-09-11T03:45:14.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6090000000000003e-06 2024-09-11T03:45:14.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:14.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933579+0000", 2024-09-11T03:45:14.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4339999999999997e-06 2024-09-11T03:45:14.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933717+0000", 2024-09-11T03:45:14.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000137995 2024-09-11T03:45:14.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:14.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933721+0000", 2024-09-11T03:45:14.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6940000000000001e-06 2024-09-11T03:45:14.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935773+0000", 2024-09-11T03:45:14.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002052398 2024-09-11T03:45:14.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:14.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935834+0000", 2024-09-11T03:45:14.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1146e-05 2024-09-11T03:45:14.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:14.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948174+0000", 2024-09-11T03:45:14.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012339786 2024-09-11T03:45:14.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:14.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948263+0000", 2024-09-11T03:45:14.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8838999999999994e-05 2024-09-11T03:45:14.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:14.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:14.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:14.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948360+0000", 2024-09-11T03:45:14.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7366000000000003e-05 2024-09-11T03:45:14.136 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.136 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.136 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.136 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:14.136 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:14.137 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:45:14.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:14.160 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:14.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:13 smithi175 ceph-mon[30894]: 1.7 scrub starts 2024-09-11T03:45:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:13 smithi175 ceph-mon[30894]: 1.7 scrub ok 2024-09-11T03:45:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:13 smithi175 ceph-mon[30894]: pgmap v897: 57 pgs: 57 active+clean; 181 MiB data, 752 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 4.7 MiB/s wr, 24 op/s 2024-09-11T03:45:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:13 smithi175 ceph-mon[30894]: 2.3 scrub starts 2024-09-11T03:45:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:13 smithi175 ceph-mon[30894]: 2.3 scrub ok 2024-09-11T03:45:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:13 smithi175 ceph-mon[30894]: osdmap e495: 12 total, 12 up, 11 in 2024-09-11T03:45:14.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:13 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3176185444' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:45:14.261 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:14.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:14.285 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:14.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:14.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:13 smithi084 ceph-mon[27254]: 1.7 scrub starts 2024-09-11T03:45:14.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:13 smithi084 ceph-mon[27254]: 1.7 scrub ok 2024-09-11T03:45:14.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:13 smithi084 ceph-mon[27254]: pgmap v897: 57 pgs: 57 active+clean; 181 MiB data, 752 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 4.7 MiB/s wr, 24 op/s 2024-09-11T03:45:14.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:13 smithi084 ceph-mon[27254]: 2.3 scrub starts 2024-09-11T03:45:14.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:13 smithi084 ceph-mon[27254]: 2.3 scrub ok 2024-09-11T03:45:14.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:13 smithi084 ceph-mon[27254]: osdmap e495: 12 total, 12 up, 11 in 2024-09-11T03:45:14.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:13 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3176185444' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:45:14.386 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:14.410 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:14.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:14.411 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:14.512 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:14.535 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:14.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:14.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:14.637 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:14.662 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:14.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:14.663 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:14.721 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:45:14.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:14.786 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:14.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:14.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:14.856 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T03:45:14.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:14.911 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:14.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:14.959 INFO:tasks.rados.rados.2.smithi175.stdout:1828: done (0 left) 2024-09-11T03:45:14.959 INFO:tasks.rados.rados.2.smithi175.stdout:1829: write oid 33 current snap is 202 2024-09-11T03:45:14.959 INFO:tasks.rados.rados.2.smithi175.stdout:1829: seq_num 630 ranges {588410=547816,1666244=414204,2484603=103807} 2024-09-11T03:45:14.964 INFO:tasks.rados.rados.2.smithi175.stdout:1829: writing smithi17557870-33 from 588410 to 1136226 tid 1 2024-09-11T03:45:14.969 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:14.970 INFO:tasks.rados.rados.2.smithi175.stdout:1829: writing smithi17557870-33 from 1666244 to 2080448 tid 2 2024-09-11T03:45:14.973 INFO:tasks.rados.rados.2.smithi175.stdout:1829: writing smithi17557870-33 from 2484603 to 2588410 tid 3 2024-09-11T03:45:14.973 INFO:tasks.rados.rados.2.smithi175.stdout:1830: rollback oid 31 current snap is 202 2024-09-11T03:45:14.973 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 31 to 199 2024-09-11T03:45:14.973 INFO:tasks.rados.rados.2.smithi175.stdout:1829: finishing write tid 1 to smithi17557870-33 2024-09-11T03:45:14.973 INFO:tasks.rados.rados.2.smithi175.stdout:1829: finishing write tid 2 to smithi17557870-33 2024-09-11T03:45:14.973 INFO:tasks.rados.rados.2.smithi175.stdout:1831: snap_remove snap 193 2024-09-11T03:45:15.011 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:15.033 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:15.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:15.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:15.099 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:45:15.134 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:15.156 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:15.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:15.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:14 smithi121 ceph-mon[30729]: 2.17 scrub starts 2024-09-11T03:45:15.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:14 smithi121 ceph-mon[30729]: 2.17 scrub ok 2024-09-11T03:45:15.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:14 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:15.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:14 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3176185444' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:45:15.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:14 smithi121 ceph-mon[30729]: osdmap e496: 12 total, 12 up, 11 in 2024-09-11T03:45:15.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:14 smithi175 ceph-mon[30894]: 2.17 scrub starts 2024-09-11T03:45:15.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:14 smithi175 ceph-mon[30894]: 2.17 scrub ok 2024-09-11T03:45:15.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:14 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:15.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:14 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3176185444' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:45:15.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:14 smithi175 ceph-mon[30894]: osdmap e496: 12 total, 12 up, 11 in 2024-09-11T03:45:15.244 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:14 smithi084 ceph-mon[27254]: 2.17 scrub starts 2024-09-11T03:45:15.244 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:14 smithi084 ceph-mon[27254]: 2.17 scrub ok 2024-09-11T03:45:15.244 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:14 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:15.244 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:14 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3176185444' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:45:15.244 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:14 smithi084 ceph-mon[27254]: osdmap e496: 12 total, 12 up, 11 in 2024-09-11T03:45:15.257 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:15.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:15.284 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:15.284 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:15.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:15.406 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:15.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:15.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:15.508 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:15.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:15.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:15.541 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:45:15.541 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:45:15.541 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:45:15.541 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:45:15.541 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:45:15.541 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:45:15.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:45:15.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:45:15.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:45:15.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:45:15.550 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:45:15.550 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:45:15.567 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:45:15.567 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:45:15.568 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:45:15.568 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:45:15.583 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:45:15.584 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:45:15.601 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:45:15.601 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:45:15.617 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:45:15.618 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:45:15.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:15.634 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:45:15.635 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:45:15.652 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:45:15.652 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:45:15.653 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:15.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:15.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:15.755 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:15.781 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:15.782 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:15.883 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:15.904 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:15.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:15.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:15.964 INFO:tasks.rados.rados.2.smithi175.stdout:1831: done (2 left) 2024-09-11T03:45:15.965 INFO:tasks.rados.rados.2.smithi175.stdout:1832: copy_from oid 39 from oid 6 current snap is 202 2024-09-11T03:45:15.965 INFO:tasks.rados.rados.2.smithi175.stdout:1833: write oid 38 current snap is 202 2024-09-11T03:45:15.965 INFO:tasks.rados.rados.2.smithi175.stdout:1833: seq_num 631 ranges {605217=739418,2124726=480491} 2024-09-11T03:45:15.972 INFO:tasks.rados.rados.2.smithi175.stdout:1833: writing smithi17557870-38 from 605217 to 1344635 tid 1 2024-09-11T03:45:15.981 INFO:tasks.rados.rados.2.smithi175.stdout:1833: writing smithi17557870-38 from 2124726 to 2605217 tid 2 2024-09-11T03:45:15.981 INFO:tasks.rados.rados.2.smithi175.stdout:1834: read oid 49 snap -1 2024-09-11T03:45:15.981 INFO:tasks.rados.rados.2.smithi175.stdout:1834: expect (ObjNum 565 snap 172 seq_num 565) 2024-09-11T03:45:15.981 INFO:tasks.rados.rados.2.smithi175.stdout:1835: write oid 20 current snap is 202 2024-09-11T03:45:15.981 INFO:tasks.rados.rados.2.smithi175.stdout:1829: finishing write tid 3 to smithi17557870-33 2024-09-11T03:45:15.981 INFO:tasks.rados.rados.2.smithi175.stdout:1835: seq_num 632 ranges {622024=414667,1783208=417873} 2024-09-11T03:45:15.985 INFO:tasks.rados.rados.2.smithi175.stdout:1835: writing smithi17557870-20 from 622024 to 1036691 tid 1 2024-09-11T03:45:15.993 INFO:tasks.rados.rados.2.smithi175.stdout:1835: writing smithi17557870-20 from 1783208 to 2201081 tid 2 2024-09-11T03:45:15.993 INFO:tasks.rados.rados.2.smithi175.stdout:1836: read oid 3 snap -1 2024-09-11T03:45:15.993 INFO:tasks.rados.rados.2.smithi175.stdout:1836: expect (ObjNum 619 snap 199 seq_num 619) 2024-09-11T03:45:15.993 INFO:tasks.rados.rados.2.smithi175.stdout:1837: write oid 32 current snap is 202 2024-09-11T03:45:15.993 INFO:tasks.rados.rados.2.smithi175.stdout:1829: finishing write tid 4 to smithi17557870-33 2024-09-11T03:45:15.993 INFO:tasks.rados.rados.2.smithi175.stdout:1829: finishing write tid 5 to smithi17557870-33 2024-09-11T03:45:15.993 INFO:tasks.rados.rados.2.smithi175.stdout:1829: oid 33 updating version 0 to 692 2024-09-11T03:45:15.994 INFO:tasks.rados.rados.2.smithi175.stdout:1829: oid 33 updating version 692 to 693 2024-09-11T03:45:15.994 INFO:tasks.rados.rados.2.smithi175.stdout:1829: oid 33 version 693 is already newer than 691 2024-09-11T03:45:15.994 INFO:tasks.rados.rados.2.smithi175.stdout:1829: oid 33 version 693 is already newer than 690 2024-09-11T03:45:15.994 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 693 (ObjNum 630 snap 202 seq_num 630) dirty exists 2024-09-11T03:45:15.994 INFO:tasks.rados.rados.2.smithi175.stdout:1829: left oid 33 (ObjNum 630 snap 202 seq_num 630) 2024-09-11T03:45:15.994 INFO:tasks.rados.rados.2.smithi175.stdout:1830: finishing rollback tid 0 to smithi17557870-31 2024-09-11T03:45:15.994 INFO:tasks.rados.rados.2.smithi175.stdout:1830: finishing rollback tid 1 to smithi17557870-31 2024-09-11T03:45:15.994 INFO:tasks.rados.rados.2.smithi175.stdout:1830: finishing rollback tid 2 to smithi17557870-31 2024-09-11T03:45:15.994 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 681 (ObjNum 495 snap 143 seq_num 495) dirty exists 2024-09-11T03:45:15.994 INFO:tasks.rados.rados.2.smithi175.stdout:1833: finishing write tid 1 to smithi17557870-38 2024-09-11T03:45:15.994 INFO:tasks.rados.rados.2.smithi175.stdout:1837: seq_num 633 ranges {638831=489916,1841690=677884} 2024-09-11T03:45:15.998 INFO:tasks.rados.rados.2.smithi175.stdout:1837: writing smithi17557870-32 from 638831 to 1128747 tid 1 2024-09-11T03:45:16.006 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:16.007 INFO:tasks.rados.rados.2.smithi175.stdout:1837: writing smithi17557870-32 from 1841690 to 2519574 tid 2 2024-09-11T03:45:16.007 INFO:tasks.rados.rados.2.smithi175.stdout:1829: done (7 left) 2024-09-11T03:45:16.007 INFO:tasks.rados.rados.2.smithi175.stdout:1830: done (6 left) 2024-09-11T03:45:16.007 INFO:tasks.rados.rados.2.smithi175.stdout:1838: write oid 23 current snap is 202 2024-09-11T03:45:16.007 INFO:tasks.rados.rados.2.smithi175.stdout:1832: finishing copy_from racing read to smithi17557870-39 2024-09-11T03:45:16.007 INFO:tasks.rados.rados.2.smithi175.stdout:1833: finishing write tid 2 to smithi17557870-38 2024-09-11T03:45:16.007 INFO:tasks.rados.rados.2.smithi175.stdout:1838: seq_num 634 ranges {655638=565165,1900172=421542} 2024-09-11T03:45:16.013 INFO:tasks.rados.rados.2.smithi175.stdout:1838: writing smithi17557870-23 from 655638 to 1220803 tid 1 2024-09-11T03:45:16.019 INFO:tasks.rados.rados.2.smithi175.stdout:1838: writing smithi17557870-23 from 1900172 to 2321714 tid 2 2024-09-11T03:45:16.020 INFO:tasks.rados.rados.2.smithi175.stdout:1839: delete oid 15 current snap is 202 2024-09-11T03:45:16.020 INFO:tasks.rados.rados.2.smithi175.stdout:1833: finishing write tid 3 to smithi17557870-38 2024-09-11T03:45:16.020 INFO:tasks.rados.rados.2.smithi175.stdout:1832: finishing copy_from to smithi17557870-39 2024-09-11T03:45:16.020 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 421 (ObjNum 528 snap 156 seq_num 528) dirty exists 2024-09-11T03:45:16.020 INFO:tasks.rados.rados.2.smithi175.stdout:1833: finishing write tid 4 to smithi17557870-38 2024-09-11T03:45:16.020 INFO:tasks.rados.rados.2.smithi175.stdout:1833: oid 38 updating version 0 to 675 2024-09-11T03:45:16.020 INFO:tasks.rados.rados.2.smithi175.stdout:1833: oid 38 updating version 675 to 676 2024-09-11T03:45:16.020 INFO:tasks.rados.rados.2.smithi175.stdout:1833: oid 38 updating version 676 to 677 2024-09-11T03:45:16.020 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 677 (ObjNum 631 snap 202 seq_num 631) dirty exists 2024-09-11T03:45:16.020 INFO:tasks.rados.rados.2.smithi175.stdout:1833: left oid 38 (ObjNum 631 snap 202 seq_num 631) 2024-09-11T03:45:16.020 INFO:tasks.rados.rados.2.smithi175.stdout:1834: expect (ObjNum 565 snap 172 seq_num 565) 2024-09-11T03:45:16.029 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:16.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:16.030 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:16.131 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:16.153 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:16.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:16.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:16.185 INFO:tasks.rados.rados.2.smithi175.stdout:1835: finishing write tid 1 to smithi17557870-20 2024-09-11T03:45:16.185 INFO:tasks.rados.rados.2.smithi175.stdout:1835: finishing write tid 2 to smithi17557870-20 2024-09-11T03:45:16.185 INFO:tasks.rados.rados.2.smithi175.stdout:1835: finishing write tid 3 to smithi17557870-20 2024-09-11T03:45:16.185 INFO:tasks.rados.rados.2.smithi175.stdout:1835: finishing write tid 4 to smithi17557870-20 2024-09-11T03:45:16.185 INFO:tasks.rados.rados.2.smithi175.stdout:1835: oid 20 updating version 0 to 674 2024-09-11T03:45:16.186 INFO:tasks.rados.rados.2.smithi175.stdout:1835: oid 20 version 674 is already newer than 672 2024-09-11T03:45:16.186 INFO:tasks.rados.rados.2.smithi175.stdout:1835: oid 20 version 674 is already newer than 673 2024-09-11T03:45:16.186 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 674 (ObjNum 632 snap 202 seq_num 632) dirty exists 2024-09-11T03:45:16.186 INFO:tasks.rados.rados.2.smithi175.stdout:1835: left oid 20 (ObjNum 632 snap 202 seq_num 632) 2024-09-11T03:45:16.186 INFO:tasks.rados.rados.2.smithi175.stdout:1837: finishing write tid 1 to smithi17557870-32 2024-09-11T03:45:16.186 INFO:tasks.rados.rados.2.smithi175.stdout:1836: expect (ObjNum 619 snap 199 seq_num 619) 2024-09-11T03:45:16.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:15 smithi175 ceph-mon[30894]: pgmap v900: 57 pgs: 57 active+clean; 183 MiB data, 765 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 5.0 MiB/s wr, 25 op/s 2024-09-11T03:45:16.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:15 smithi175 ceph-mon[30894]: osdmap e497: 12 total, 12 up, 11 in 2024-09-11T03:45:16.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:15 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:45:16.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:15 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:45:16.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:15 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1132193790' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:16.233 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:15 smithi121 ceph-mon[30729]: pgmap v900: 57 pgs: 57 active+clean; 183 MiB data, 765 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 5.0 MiB/s wr, 25 op/s 2024-09-11T03:45:16.233 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:15 smithi121 ceph-mon[30729]: osdmap e497: 12 total, 12 up, 11 in 2024-09-11T03:45:16.233 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:15 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:45:16.233 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:15 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:45:16.233 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:15 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1132193790' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:16.252 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:45:16.253 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:45:16.253 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2024-09-11T03:45:16.253 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T03:45:16.253 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T03:45:16.253 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 config set filestore_inject_stall 3 2024-09-11T03:45:16.255 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:16.276 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:16.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:16.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:16.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T03:45:16.338 INFO:tasks.rados.rados.2.smithi175.stdout:1837: finishing write tid 2 to smithi17557870-32 2024-09-11T03:45:16.338 INFO:tasks.rados.rados.2.smithi175.stdout:1837: finishing write tid 3 to smithi17557870-32 2024-09-11T03:45:16.338 INFO:tasks.rados.rados.2.smithi175.stdout:1837: finishing write tid 4 to smithi17557870-32 2024-09-11T03:45:16.338 INFO:tasks.rados.rados.2.smithi175.stdout:1837: oid 32 updating version 0 to 529 2024-09-11T03:45:16.338 INFO:tasks.rados.rados.2.smithi175.stdout:1837: oid 32 version 529 is already newer than 528 2024-09-11T03:45:16.338 INFO:tasks.rados.rados.2.smithi175.stdout:1837: oid 32 updating version 529 to 530 2024-09-11T03:45:16.338 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 530 (ObjNum 633 snap 202 seq_num 633) dirty exists 2024-09-11T03:45:16.338 INFO:tasks.rados.rados.2.smithi175.stdout:1837: left oid 32 (ObjNum 633 snap 202 seq_num 633) 2024-09-11T03:45:16.338 INFO:tasks.rados.rados.2.smithi175.stdout:1838: finishing write tid 1 to smithi17557870-23 2024-09-11T03:45:16.338 INFO:tasks.rados.rados.2.smithi175.stdout:1832: done (7 left) 2024-09-11T03:45:16.338 INFO:tasks.rados.rados.2.smithi175.stdout:1833: done (6 left) 2024-09-11T03:45:16.338 INFO:tasks.rados.rados.2.smithi175.stdout:1834: done (5 left) 2024-09-11T03:45:16.339 INFO:tasks.rados.rados.2.smithi175.stdout:1835: done (4 left) 2024-09-11T03:45:16.339 INFO:tasks.rados.rados.2.smithi175.stdout:1836: done (3 left) 2024-09-11T03:45:16.339 INFO:tasks.rados.rados.2.smithi175.stdout:1837: done (2 left) 2024-09-11T03:45:16.339 INFO:tasks.rados.rados.2.smithi175.stdout:1839: done (1 left) 2024-09-11T03:45:16.339 INFO:tasks.rados.rados.2.smithi175.stdout:1840: copy_from oid 3 from oid 45 current snap is 202 2024-09-11T03:45:16.339 INFO:tasks.rados.rados.2.smithi175.stdout:1838: finishing write tid 2 to smithi17557870-23 2024-09-11T03:45:16.339 INFO:tasks.rados.rados.2.smithi175.stdout:1838: finishing write tid 3 to smithi17557870-23 2024-09-11T03:45:16.339 INFO:tasks.rados.rados.2.smithi175.stdout:1838: finishing write tid 4 to smithi17557870-23 2024-09-11T03:45:16.339 INFO:tasks.rados.rados.2.smithi175.stdout:1838: oid 23 updating version 0 to 342 2024-09-11T03:45:16.339 INFO:tasks.rados.rados.2.smithi175.stdout:1838: oid 23 updating version 342 to 343 2024-09-11T03:45:16.339 INFO:tasks.rados.rados.2.smithi175.stdout:1838: oid 23 version 343 is already newer than 341 2024-09-11T03:45:16.339 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 343 (ObjNum 634 snap 202 seq_num 634) dirty exists 2024-09-11T03:45:16.339 INFO:tasks.rados.rados.2.smithi175.stdout:1838: left oid 23 (ObjNum 634 snap 202 seq_num 634) 2024-09-11T03:45:16.339 INFO:tasks.rados.rados.2.smithi175.stdout:1838: done (1 left) 2024-09-11T03:45:16.340 INFO:tasks.rados.rados.2.smithi175.stdout:1841: delete oid 36 current snap is 202 2024-09-11T03:45:16.340 INFO:tasks.rados.rados.2.smithi175.stdout:1841: done (1 left) 2024-09-11T03:45:16.340 INFO:tasks.rados.rados.2.smithi175.stdout:1842: read oid 32 snap 196 2024-09-11T03:45:16.340 INFO:tasks.rados.rados.2.smithi175.stdout:1842: expect (ObjNum 596 snap 185 seq_num 596) 2024-09-11T03:45:16.340 INFO:tasks.rados.rados.2.smithi175.stdout:1843: delete oid 30 current snap is 202 2024-09-11T03:45:16.341 INFO:tasks.rados.rados.2.smithi175.stdout:1843: done (2 left) 2024-09-11T03:45:16.341 INFO:tasks.rados.rados.2.smithi175.stdout:1844: snap_remove snap 199 2024-09-11T03:45:16.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:15 smithi084 ceph-mon[27254]: pgmap v900: 57 pgs: 57 active+clean; 183 MiB data, 765 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 5.0 MiB/s wr, 25 op/s 2024-09-11T03:45:16.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:15 smithi084 ceph-mon[27254]: osdmap e497: 12 total, 12 up, 11 in 2024-09-11T03:45:16.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:15 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:45:16.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:15 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:45:16.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:15 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1132193790' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:16.379 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:16.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:16.402 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:16.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:16.503 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:16.513 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:45:16.524 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:16.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:16.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:16.544 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:45:16.626 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:16.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:16.649 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:16.649 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:16.703 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:45:16.750 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:16.771 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:16.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:16.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:16.896 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:16.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:16.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:16.947 INFO:tasks.rados.rados.2.smithi175.stdout:1844: done (2 left) 2024-09-11T03:45:16.947 INFO:tasks.rados.rados.2.smithi175.stdout:1845: copy_from oid 28 from oid 34 current snap is 202 2024-09-11T03:45:16.947 INFO:tasks.rados.rados.2.smithi175.stdout:1842: expect (ObjNum 596 snap 185 seq_num 596) 2024-09-11T03:45:16.950 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:16.993 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:45:16.994 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:45:17.000 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:17.021 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:17.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:17.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:17.034 INFO:tasks.rados.rados.2.smithi175.stdout:1840: finishing copy_from racing read to smithi17557870-3 2024-09-11T03:45:17.034 INFO:tasks.rados.rados.2.smithi175.stdout:1840: finishing copy_from to smithi17557870-3 2024-09-11T03:45:17.034 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 481 (ObjNum 416 snap 127 seq_num 416) dirty exists 2024-09-11T03:45:17.034 INFO:tasks.rados.rados.2.smithi175.stdout:1845: finishing copy_from racing read to smithi17557870-28 2024-09-11T03:45:17.034 INFO:tasks.rados.rados.2.smithi175.stdout:1845: finishing copy_from to smithi17557870-28 2024-09-11T03:45:17.034 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 467 (ObjNum 609 snap 191 seq_num 609) dirty exists 2024-09-11T03:45:17.034 INFO:tasks.rados.rados.2.smithi175.stdout:1840: done (2 left) 2024-09-11T03:45:17.034 INFO:tasks.rados.rados.2.smithi175.stdout:1842: done (1 left) 2024-09-11T03:45:17.034 INFO:tasks.rados.rados.2.smithi175.stdout:1845: done (0 left) 2024-09-11T03:45:17.034 INFO:tasks.rados.rados.2.smithi175.stdout:1846: snap_remove snap 197 2024-09-11T03:45:17.044 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:45:17.044 INFO:teuthology.orchestra.run.smithi175.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T03:45:17.044 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:45:17.074 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:16 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:45:17.074 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:16 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:45:17.074 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:16 smithi084 ceph-mon[27254]: osdmap e498: 12 total, 12 up, 11 in 2024-09-11T03:45:17.074 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:45:17.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:45:17.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:45:17.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:45:17.075 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:16 smithi084 ceph-mon[27254]: osdmap e499: 12 total, 12 up, 11 in 2024-09-11T03:45:17.123 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:17.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:17.146 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:17.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:17.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:16 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:45:17.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:16 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:45:17.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:16 smithi175 ceph-mon[30894]: osdmap e498: 12 total, 12 up, 11 in 2024-09-11T03:45:17.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:45:17.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:45:17.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:45:17.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:45:17.229 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:16 smithi175 ceph-mon[30894]: osdmap e499: 12 total, 12 up, 11 in 2024-09-11T03:45:17.247 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:17.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:17.270 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:17.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:17.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:16 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:45:17.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:16 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:45:17.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:16 smithi121 ceph-mon[30729]: osdmap e498: 12 total, 12 up, 11 in 2024-09-11T03:45:17.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:45:17.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:45:17.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:45:17.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:45:17.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:16 smithi121 ceph-mon[30729]: osdmap e499: 12 total, 12 up, 11 in 2024-09-11T03:45:17.371 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:17.394 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:17.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:17.395 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:17.496 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:17.518 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:17.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:17.519 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:17.620 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:17.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T03:45:17.642 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:17.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:17.643 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:17.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:17.765 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:17.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:17.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:17.889 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:17.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:17.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:17.908 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:45:17.964 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:45:17.969 INFO:tasks.rados.rados.2.smithi175.stdout:1846: done (0 left) 2024-09-11T03:45:17.969 INFO:tasks.rados.rados.2.smithi175.stdout:1847: copy_from oid 2 from oid 33 current snap is 202 2024-09-11T03:45:17.969 INFO:tasks.rados.rados.2.smithi175.stdout:1848: rollback oid 20 current snap is 202 2024-09-11T03:45:17.969 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 20 to 195 2024-09-11T03:45:17.969 INFO:tasks.rados.rados.2.smithi175.stdout:1849: copy_from oid 41 from oid 28 current snap is 202 2024-09-11T03:45:17.969 INFO:tasks.rados.rados.2.smithi175.stdout:1850: read oid 3 snap 175 2024-09-11T03:45:17.969 INFO:tasks.rados.rados.2.smithi175.stdout:1850: expect (ObjNum 488 snap 141 seq_num 488) 2024-09-11T03:45:17.969 INFO:tasks.rados.rados.2.smithi175.stdout:1851: read oid 25 snap -1 2024-09-11T03:45:17.969 INFO:tasks.rados.rados.2.smithi175.stdout:1851: expect (ObjNum 496 snap 144 seq_num 496) 2024-09-11T03:45:17.970 INFO:tasks.rados.rados.2.smithi175.stdout:1852: read oid 22 snap -1 2024-09-11T03:45:17.970 INFO:tasks.rados.rados.2.smithi175.stdout:1852: expect (ObjNum 506 snap 148 seq_num 506) 2024-09-11T03:45:17.970 INFO:tasks.rados.rados.2.smithi175.stdout:1853: rollback oid 1 current snap is 202 2024-09-11T03:45:17.970 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 1 to 200 2024-09-11T03:45:17.970 INFO:tasks.rados.rados.2.smithi175.stdout:1854: snap_remove snap 196 2024-09-11T03:45:17.972 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:17 smithi175 ceph-mon[30894]: pgmap v903: 57 pgs: 57 active+clean; 196 MiB data, 793 MiB used, 983 GiB / 983 GiB avail; 5.3 MiB/s rd, 3.3 MiB/s wr, 16 op/s 2024-09-11T03:45:17.972 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:17 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:45:17.972 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:17 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:45:17.972 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/495851656' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:45:17.972 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:17 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:45:17.972 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:17 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:45:17.991 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:18.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:18.015 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:18.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:18.116 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:18.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:18.139 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:18.139 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:18.240 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:18.241 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:17 smithi121 ceph-mon[30729]: pgmap v903: 57 pgs: 57 active+clean; 196 MiB data, 793 MiB used, 983 GiB / 983 GiB avail; 5.3 MiB/s rd, 3.3 MiB/s wr, 16 op/s 2024-09-11T03:45:18.241 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:17 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:45:18.241 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:17 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:45:18.241 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/495851656' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:45:18.241 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:17 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:45:18.242 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:17 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:45:18.262 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:18.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:18.263 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:18.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:17 smithi084 ceph-mon[27254]: pgmap v903: 57 pgs: 57 active+clean; 196 MiB data, 793 MiB used, 983 GiB / 983 GiB avail; 5.3 MiB/s rd, 3.3 MiB/s wr, 16 op/s 2024-09-11T03:45:18.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:17 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:45:18.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:17 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:45:18.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/495851656' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:45:18.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:17 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:45:18.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:17 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:45:18.339 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:45:18.339 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:45:18.364 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:18.385 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:18.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:18.386 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:18.400 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:45:18.487 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:18.508 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:18.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:18.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:18.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:18.632 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:18.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:18.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:18.756 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:18.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:18.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:18.804 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:18.858 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:18.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:18.882 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:18.882 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:18.983 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:18.985 INFO:tasks.rados.rados.2.smithi175.stdout:1854: done (7 left) 2024-09-11T03:45:18.985 INFO:tasks.rados.rados.2.smithi175.stdout:1855: read oid 13 snap -1 2024-09-11T03:45:18.985 INFO:tasks.rados.rados.2.smithi175.stdout:1855: expect deleted 2024-09-11T03:45:18.985 INFO:tasks.rados.rados.2.smithi175.stdout:1856: read oid 32 snap -1 2024-09-11T03:45:18.985 INFO:tasks.rados.rados.2.smithi175.stdout:1856: expect (ObjNum 633 snap 202 seq_num 633) 2024-09-11T03:45:18.986 INFO:tasks.rados.rados.2.smithi175.stdout:1857: read oid 27 snap -1 2024-09-11T03:45:18.986 INFO:tasks.rados.rados.2.smithi175.stdout:1857: expect (ObjNum 577 snap 177 seq_num 577) 2024-09-11T03:45:18.986 INFO:tasks.rados.rados.2.smithi175.stdout:1858: write (excl) oid 11 current snap is 202 2024-09-11T03:45:18.986 INFO:tasks.rados.rados.2.smithi175.stdout:1858: seq_num 638 ranges {722866=466161,1617747=428880,2587634=135232} 2024-09-11T03:45:18.992 INFO:tasks.rados.rados.2.smithi175.stdout:1858: writing smithi17557870-11 from 722866 to 1189027 tid 1 2024-09-11T03:45:18.997 INFO:tasks.rados.rados.2.smithi175.stdout:1858: writing smithi17557870-11 from 1617747 to 2046627 tid 2 2024-09-11T03:45:19.001 INFO:tasks.rados.rados.2.smithi175.stdout:1858: writing smithi17557870-11 from 2587634 to 2722866 tid 3 2024-09-11T03:45:19.001 INFO:tasks.rados.rados.2.smithi175.stdout:1859: snap_create 2024-09-11T03:45:19.001 INFO:tasks.rados.rados.2.smithi175.stdout:1853: finishing rollback tid 0 to smithi17557870-1 2024-09-11T03:45:19.001 INFO:tasks.rados.rados.2.smithi175.stdout:1853: finishing rollback tid 1 to smithi17557870-1 2024-09-11T03:45:19.001 INFO:tasks.rados.rados.2.smithi175.stdout:1853: finishing rollback tid 2 to smithi17557870-1 2024-09-11T03:45:19.001 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 687 (ObjNum 616 snap 199 seq_num 616) dirty exists 2024-09-11T03:45:19.001 INFO:tasks.rados.rados.2.smithi175.stdout:1848: finishing rollback tid 0 to smithi17557870-20 2024-09-11T03:45:19.002 INFO:tasks.rados.rados.2.smithi175.stdout:1848: finishing rollback tid 1 to smithi17557870-20 2024-09-11T03:45:19.002 INFO:tasks.rados.rados.2.smithi175.stdout:1848: finishing rollback tid 2 to smithi17557870-20 2024-09-11T03:45:19.002 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 679 (ObjNum 563 snap 172 seq_num 563) dirty exists 2024-09-11T03:45:19.002 INFO:tasks.rados.rados.2.smithi175.stdout:1851: expect (ObjNum 496 snap 144 seq_num 496) 2024-09-11T03:45:19.006 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:19.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:19.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T03:45:19.052 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:18 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/495851656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:45:19.052 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:18 smithi084 ceph-mon[27254]: osdmap e500: 12 total, 12 up, 11 in 2024-09-11T03:45:19.107 INFO:tasks.rados.rados.2.smithi175.stdout:1850: expect (ObjNum 488 snap 141 seq_num 488) 2024-09-11T03:45:19.108 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:19.129 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:19.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:19.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:19.234 INFO:tasks.rados.rados.2.smithi175.stdout:1847: finishing copy_from racing read to smithi17557870-2 2024-09-11T03:45:19.234 INFO:tasks.rados.rados.2.smithi175.stdout:1849: finishing copy_from racing read to smithi17557870-41 2024-09-11T03:45:19.234 INFO:tasks.rados.rados.2.smithi175.stdout:1852: expect (ObjNum 506 snap 148 seq_num 506) 2024-09-11T03:45:19.253 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:19.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:19.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:19.285 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:45:19.292 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:45:19.293 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:45:19.309 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:45:19.309 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:45:19.325 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:45:19.325 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:45:19.337 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:18 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/495851656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:45:19.337 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:18 smithi121 ceph-mon[30729]: osdmap e500: 12 total, 12 up, 11 in 2024-09-11T03:45:19.341 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:45:19.341 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:45:19.343 INFO:tasks.rados.rados.2.smithi175.stdout:1849: finishing copy_from to smithi17557870-41 2024-09-11T03:45:19.344 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 467 (ObjNum 609 snap 191 seq_num 609) dirty exists 2024-09-11T03:45:19.344 INFO:tasks.rados.rados.2.smithi175.stdout:1847: finishing copy_from to smithi17557870-2 2024-09-11T03:45:19.344 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 693 (ObjNum 630 snap 202 seq_num 630) dirty exists 2024-09-11T03:45:19.344 INFO:tasks.rados.rados.2.smithi175.stdout:1857: expect (ObjNum 577 snap 177 seq_num 577) 2024-09-11T03:45:19.355 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:19.357 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:45:19.358 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:45:19.374 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:45:19.375 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:45:19.378 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:19.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:19.391 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:45:19.391 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:45:19.408 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:45:19.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:45:19.424 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:45:19.424 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:45:19.441 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:45:19.441 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:45:19.458 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:45:19.458 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:45:19.475 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:45:19.475 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:45:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:18 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/495851656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:45:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:18 smithi175 ceph-mon[30894]: osdmap e500: 12 total, 12 up, 11 in 2024-09-11T03:45:19.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:19.503 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:19.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:19.505 INFO:tasks.rados.rados.2.smithi175.stdout:1856: expect (ObjNum 633 snap 202 seq_num 633) 2024-09-11T03:45:19.603 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:19.624 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:19.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:19.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:19.644 INFO:tasks.rados.rados.2.smithi175.stdout:1858: finishing write tid 1 to smithi17557870-11 2024-09-11T03:45:19.644 INFO:tasks.rados.rados.2.smithi175.stdout:1858: finishing write tid 2 to smithi17557870-11 2024-09-11T03:45:19.644 INFO:tasks.rados.rados.2.smithi175.stdout:1858: finishing write tid 3 to smithi17557870-11 2024-09-11T03:45:19.644 INFO:tasks.rados.rados.2.smithi175.stdout:1858: finishing write tid 4 to smithi17557870-11 2024-09-11T03:45:19.645 INFO:tasks.rados.rados.2.smithi175.stdout:1858: finishing write tid 5 to smithi17557870-11 2024-09-11T03:45:19.645 INFO:tasks.rados.rados.2.smithi175.stdout:1858: oid 11 updating version 0 to 579 2024-09-11T03:45:19.645 INFO:tasks.rados.rados.2.smithi175.stdout:1858: oid 11 version 579 is already newer than 576 2024-09-11T03:45:19.645 INFO:tasks.rados.rados.2.smithi175.stdout:1858: oid 11 version 579 is already newer than 578 2024-09-11T03:45:19.645 INFO:tasks.rados.rados.2.smithi175.stdout:1858: oid 11 version 579 is already newer than 577 2024-09-11T03:45:19.645 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 579 (ObjNum 638 snap 202 seq_num 638) dirty exists 2024-09-11T03:45:19.645 INFO:tasks.rados.rados.2.smithi175.stdout:1858: left oid 11 (ObjNum 638 snap 202 seq_num 638) 2024-09-11T03:45:19.726 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:19.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:19.748 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:19.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:19.772 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:45:19.772 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:45:19.772 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:45:19.772 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:45:19.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:19.870 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:19.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:19.973 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:19.985 INFO:tasks.rados.rados.2.smithi175.stdout:1847: done (11 left) 2024-09-11T03:45:19.985 INFO:tasks.rados.rados.2.smithi175.stdout:1848: done (10 left) 2024-09-11T03:45:19.985 INFO:tasks.rados.rados.2.smithi175.stdout:1849: done (9 left) 2024-09-11T03:45:19.985 INFO:tasks.rados.rados.2.smithi175.stdout:1850: done (8 left) 2024-09-11T03:45:19.986 INFO:tasks.rados.rados.2.smithi175.stdout:1851: done (7 left) 2024-09-11T03:45:19.986 INFO:tasks.rados.rados.2.smithi175.stdout:1852: done (6 left) 2024-09-11T03:45:19.986 INFO:tasks.rados.rados.2.smithi175.stdout:1853: done (5 left) 2024-09-11T03:45:19.986 INFO:tasks.rados.rados.2.smithi175.stdout:1855: done (4 left) 2024-09-11T03:45:19.986 INFO:tasks.rados.rados.2.smithi175.stdout:1856: done (3 left) 2024-09-11T03:45:19.986 INFO:tasks.rados.rados.2.smithi175.stdout:1857: done (2 left) 2024-09-11T03:45:19.986 INFO:tasks.rados.rados.2.smithi175.stdout:1858: done (1 left) 2024-09-11T03:45:19.986 INFO:tasks.rados.rados.2.smithi175.stdout:1859: done (0 left) 2024-09-11T03:45:19.986 INFO:tasks.rados.rados.2.smithi175.stdout:1860: copy_from oid 5 from oid 14 current snap is 203 2024-09-11T03:45:19.986 INFO:tasks.rados.rados.2.smithi175.stdout:1861: snap_create 2024-09-11T03:45:19.992 INFO:tasks.rados.rados.2.smithi175.stdout:1860: finishing copy_from racing read to smithi17557870-5 2024-09-11T03:45:19.994 INFO:tasks.rados.rados.2.smithi175.stdout:1860: finishing copy_from to smithi17557870-5 2024-09-11T03:45:19.994 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 491 (ObjNum 496 snap 144 seq_num 496) dirty exists 2024-09-11T03:45:19.997 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:19.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:19.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:20.015 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:19 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:45:20.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:19 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:45:20.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:19 smithi084 ceph-mon[27254]: pgmap v906: 57 pgs: 57 active+clean; 197 MiB data, 806 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 3.3 MiB/s wr, 21 op/s 2024-09-11T03:45:20.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:19 smithi084 ceph-mon[27254]: osdmap e501: 12 total, 12 up, 11 in 2024-09-11T03:45:20.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:19 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/156213875' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:20.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:20.122 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:20.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:20.224 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:19 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:45:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:19 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:45:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:19 smithi175 ceph-mon[30894]: pgmap v906: 57 pgs: 57 active+clean; 197 MiB data, 806 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 3.3 MiB/s wr, 21 op/s 2024-09-11T03:45:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:19 smithi175 ceph-mon[30894]: osdmap e501: 12 total, 12 up, 11 in 2024-09-11T03:45:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:19 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/156213875' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:20.245 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:20.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:20.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:20.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:20.370 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:20.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:20.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:19 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:45:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:19 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:45:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:19 smithi121 ceph-mon[30729]: pgmap v906: 57 pgs: 57 active+clean; 197 MiB data, 806 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 3.3 MiB/s wr, 21 op/s 2024-09-11T03:45:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:19 smithi121 ceph-mon[30729]: osdmap e501: 12 total, 12 up, 11 in 2024-09-11T03:45:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:19 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/156213875' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:20.472 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:20.488 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T03:45:20.493 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:20.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:20.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:20.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:20.617 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:20.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:20.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:20.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:20.728 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:45:20.741 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:20.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:20.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:20.779 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:45:20.843 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:20.864 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:20.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:20.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:20.966 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:20.988 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:20.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:20.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:20.994 INFO:tasks.rados.rados.2.smithi175.stdout:1860: done (1 left) 2024-09-11T03:45:20.994 INFO:tasks.rados.rados.2.smithi175.stdout:1861: done (0 left) 2024-09-11T03:45:20.995 INFO:tasks.rados.rados.2.smithi175.stdout:1862: write (excl) oid 23 current snap is 204 2024-09-11T03:45:20.995 INFO:tasks.rados.rados.2.smithi175.stdout:1862: seq_num 639 ranges {739673=657763,2076229=663444} 2024-09-11T03:45:21.001 INFO:tasks.rados.rados.2.smithi175.stdout:1862: writing smithi17557870-23 from 739673 to 1397436 tid 1 2024-09-11T03:45:21.009 INFO:tasks.rados.rados.2.smithi175.stdout:1862: writing smithi17557870-23 from 2076229 to 2739673 tid 2 2024-09-11T03:45:21.009 INFO:tasks.rados.rados.2.smithi175.stdout:1863: delete oid 48 current snap is 204 2024-09-11T03:45:21.012 INFO:tasks.rados.rados.2.smithi175.stdout:1863: done (1 left) 2024-09-11T03:45:21.012 INFO:tasks.rados.rados.2.smithi175.stdout:1864: snap_remove snap 192 2024-09-11T03:45:21.028 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:21.090 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:21.101 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:20 smithi121 ceph-mon[30729]: osdmap e502: 12 total, 12 up, 11 in 2024-09-11T03:45:21.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:21.113 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:21.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:21.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:20 smithi084 ceph-mon[27254]: osdmap e502: 12 total, 12 up, 11 in 2024-09-11T03:45:21.205 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:45:21.205 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:45:21.205 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:45:21.205 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:45:21.205 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:45:21.214 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:21.236 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:21.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:21.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:21.338 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:21.359 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:21.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:21.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:21.461 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:21.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:20 smithi175 ceph-mon[30894]: osdmap e502: 12 total, 12 up, 11 in 2024-09-11T03:45:21.483 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:21.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:21.484 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:21.585 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:21.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:21.608 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:21.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:21.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:21.730 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:21.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:21.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:21.832 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:21.854 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:21.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:21.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:21.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T03:45:21.940 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:45:21.946 INFO:tasks.rados.rados.2.smithi175.stdout:1864: done (1 left) 2024-09-11T03:45:21.946 INFO:tasks.rados.rados.2.smithi175.stdout:1865: write (excl) oid 14 current snap is 204 2024-09-11T03:45:21.946 INFO:tasks.rados.rados.2.smithi175.stdout:1865: seq_num 641 ranges {773287=408261,1793193=576207} 2024-09-11T03:45:21.952 INFO:tasks.rados.rados.2.smithi175.stdout:1865: writing smithi17557870-14 from 773287 to 1181548 tid 1 2024-09-11T03:45:21.956 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:21.960 INFO:tasks.rados.rados.2.smithi175.stdout:1865: writing smithi17557870-14 from 1793193 to 2369400 tid 2 2024-09-11T03:45:21.960 INFO:tasks.rados.rados.2.smithi175.stdout:1866: rollback oid 25 current snap is 204 2024-09-11T03:45:21.960 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 25 to 203 2024-09-11T03:45:21.960 INFO:tasks.rados.rados.2.smithi175.stdout:1862: finishing write tid 1 to smithi17557870-23 2024-09-11T03:45:21.960 INFO:tasks.rados.rados.2.smithi175.stdout:1862: finishing write tid 2 to smithi17557870-23 2024-09-11T03:45:21.960 INFO:tasks.rados.rados.2.smithi175.stdout:1862: finishing write tid 3 to smithi17557870-23 2024-09-11T03:45:21.960 INFO:tasks.rados.rados.2.smithi175.stdout:1862: finishing write tid 4 to smithi17557870-23 2024-09-11T03:45:21.960 INFO:tasks.rados.rados.2.smithi175.stdout:1862: oid 23 updating version 0 to 350 2024-09-11T03:45:21.961 INFO:tasks.rados.rados.2.smithi175.stdout:1862: oid 23 version 350 is already newer than 349 2024-09-11T03:45:21.961 INFO:tasks.rados.rados.2.smithi175.stdout:1862: oid 23 version 350 is already newer than 348 2024-09-11T03:45:21.961 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 350 (ObjNum 639 snap 204 seq_num 639) dirty exists 2024-09-11T03:45:21.961 INFO:tasks.rados.rados.2.smithi175.stdout:1862: left oid 23 (ObjNum 639 snap 204 seq_num 639) 2024-09-11T03:45:21.961 INFO:tasks.rados.rados.2.smithi175.stdout:1865: finishing write tid 1 to smithi17557870-14 2024-09-11T03:45:21.961 INFO:tasks.rados.rados.2.smithi175.stdout:1862: done (2 left) 2024-09-11T03:45:21.961 INFO:tasks.rados.rados.2.smithi175.stdout:1867: copy_from oid 1 from oid 35 current snap is 204 2024-09-11T03:45:21.961 INFO:tasks.rados.rados.2.smithi175.stdout:1868: write oid 27 current snap is 204 2024-09-11T03:45:21.961 INFO:tasks.rados.rados.2.smithi175.stdout:1868: seq_num 642 ranges {790094=483510,1851675=436218} 2024-09-11T03:45:21.966 INFO:tasks.rados.rados.2.smithi175.stdout:1868: writing smithi17557870-27 from 790094 to 1273604 tid 1 2024-09-11T03:45:21.973 INFO:tasks.rados.rados.2.smithi175.stdout:1868: writing smithi17557870-27 from 1851675 to 2287893 tid 2 2024-09-11T03:45:21.973 INFO:tasks.rados.rados.2.smithi175.stdout:1869: read oid 42 snap 203 2024-09-11T03:45:21.973 INFO:tasks.rados.rados.2.smithi175.stdout:1869: expect (ObjNum 565 snap 172 seq_num 565) 2024-09-11T03:45:21.973 INFO:tasks.rados.rados.2.smithi175.stdout:1870: read oid 39 snap -1 2024-09-11T03:45:21.973 INFO:tasks.rados.rados.2.smithi175.stdout:1870: expect (ObjNum 528 snap 156 seq_num 528) 2024-09-11T03:45:21.973 INFO:tasks.rados.rados.2.smithi175.stdout:1871: delete oid 12 current snap is 204 2024-09-11T03:45:21.973 INFO:tasks.rados.rados.2.smithi175.stdout:1866: finishing rollback tid 0 to smithi17557870-25 2024-09-11T03:45:21.973 INFO:tasks.rados.rados.2.smithi175.stdout:1866: finishing rollback tid 1 to smithi17557870-25 2024-09-11T03:45:21.973 INFO:tasks.rados.rados.2.smithi175.stdout:1866: finishing rollback tid 2 to smithi17557870-25 2024-09-11T03:45:21.973 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 494 (ObjNum 496 snap 144 seq_num 496) dirty exists 2024-09-11T03:45:21.973 INFO:tasks.rados.rados.2.smithi175.stdout:1865: finishing write tid 2 to smithi17557870-14 2024-09-11T03:45:21.974 INFO:tasks.rados.rados.2.smithi175.stdout:1865: finishing write tid 3 to smithi17557870-14 2024-09-11T03:45:21.974 INFO:tasks.rados.rados.2.smithi175.stdout:1865: finishing write tid 4 to smithi17557870-14 2024-09-11T03:45:21.974 INFO:tasks.rados.rados.2.smithi175.stdout:1865: oid 14 updating version 0 to 727 2024-09-11T03:45:21.974 INFO:tasks.rados.rados.2.smithi175.stdout:1865: oid 14 version 727 is already newer than 726 2024-09-11T03:45:21.974 INFO:tasks.rados.rados.2.smithi175.stdout:1865: oid 14 updating version 727 to 728 2024-09-11T03:45:21.974 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 728 (ObjNum 641 snap 204 seq_num 641) dirty exists 2024-09-11T03:45:21.974 INFO:tasks.rados.rados.2.smithi175.stdout:1865: left oid 14 (ObjNum 641 snap 204 seq_num 641) 2024-09-11T03:45:21.974 INFO:tasks.rados.rados.2.smithi175.stdout:1867: finishing copy_from racing read to smithi17557870-1 2024-09-11T03:45:21.974 INFO:tasks.rados.rados.2.smithi175.stdout:1868: finishing write tid 1 to smithi17557870-27 2024-09-11T03:45:21.975 INFO:tasks.rados.rados.2.smithi175.stdout:1867: finishing copy_from to smithi17557870-1 2024-09-11T03:45:21.975 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 541 (ObjNum 613 snap 196 seq_num 613) dirty exists 2024-09-11T03:45:21.975 INFO:tasks.rados.rados.2.smithi175.stdout:1865: done (6 left) 2024-09-11T03:45:21.975 INFO:tasks.rados.rados.2.smithi175.stdout:1866: done (5 left) 2024-09-11T03:45:21.975 INFO:tasks.rados.rados.2.smithi175.stdout:1867: done (4 left) 2024-09-11T03:45:21.975 INFO:tasks.rados.rados.2.smithi175.stdout:1871: done (3 left) 2024-09-11T03:45:21.975 INFO:tasks.rados.rados.2.smithi175.stdout:1872: snap_remove snap 195 2024-09-11T03:45:21.982 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:21.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:21.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:22.043 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:45:22.084 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:22.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:22.107 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:22.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:22.151 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:45:22.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:21 smithi121 ceph-mon[30729]: pgmap v909: 57 pgs: 57 active+clean; 197 MiB data, 808 MiB used, 983 GiB / 983 GiB avail; 7.8 MiB/s rd, 241 KiB/s wr, 7 op/s 2024-09-11T03:45:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:21 smithi121 ceph-mon[30729]: osdmap e503: 12 total, 12 up, 11 in 2024-09-11T03:45:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:21 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/726585534' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:45:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:21 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:21 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/726585534' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:45:22.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:21 smithi121 ceph-mon[30729]: osdmap e504: 12 total, 12 up, 11 in 2024-09-11T03:45:22.208 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:22.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:22.231 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:22.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:22.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:21 smithi084 ceph-mon[27254]: pgmap v909: 57 pgs: 57 active+clean; 197 MiB data, 808 MiB used, 983 GiB / 983 GiB avail; 7.8 MiB/s rd, 241 KiB/s wr, 7 op/s 2024-09-11T03:45:22.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:21 smithi084 ceph-mon[27254]: osdmap e503: 12 total, 12 up, 11 in 2024-09-11T03:45:22.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/726585534' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:45:22.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:21 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:22.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/726585534' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:45:22.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:21 smithi084 ceph-mon[27254]: osdmap e504: 12 total, 12 up, 11 in 2024-09-11T03:45:22.332 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:22.354 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:22.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:22.455 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:22.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:22.477 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:22.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:21 smithi175 ceph-mon[30894]: pgmap v909: 57 pgs: 57 active+clean; 197 MiB data, 808 MiB used, 983 GiB / 983 GiB avail; 7.8 MiB/s rd, 241 KiB/s wr, 7 op/s 2024-09-11T03:45:22.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:21 smithi175 ceph-mon[30894]: osdmap e503: 12 total, 12 up, 11 in 2024-09-11T03:45:22.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:21 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/726585534' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:45:22.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:21 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:22.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:21 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/726585534' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:45:22.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:21 smithi175 ceph-mon[30894]: osdmap e504: 12 total, 12 up, 11 in 2024-09-11T03:45:22.578 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:22.605 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:22.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:22.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:22.612 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:45:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:45:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:45:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:45:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:228 2.9 e105/95 2:95fa802a:::smithi17557870-22:head v 105'20, mlcod=100'17)", 2024-09-11T03:45:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:45:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 400.64472430400002, 2024-09-11T03:45:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01695953, 2024-09-11T03:45:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:45:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:45:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:45:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2600000000000002e-07 2024-09-11T03:45:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982417+0000", 2024-09-11T03:45:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01547431 2024-09-11T03:45:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982440+0000", 2024-09-11T03:45:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2506000000000001e-05 2024-09-11T03:45:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982448+0000", 2024-09-11T03:45:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2989999999999997e-06 2024-09-11T03:45:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982562+0000", 2024-09-11T03:45:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114084 2024-09-11T03:45:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982583+0000", 2024-09-11T03:45:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1160999999999999e-05 2024-09-11T03:45:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.983852+0000", 2024-09-11T03:45:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001268953 2024-09-11T03:45:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.983902+0000", 2024-09-11T03:45:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9691000000000002e-05 2024-09-11T03:45:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:345 2.0 2:0762776a:::smithi17557870-25:head [write 742876~447756] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:45:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.827240+0000", 2024-09-11T03:45:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 392.784426989, 2024-09-11T03:45:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076328890000000003, 2024-09-11T03:45:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 345 2024-09-11T03:45:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.827240+0000", 2024-09-11T03:45:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.827240+0000", 2024-09-11T03:45:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.827243+0000", 2024-09-11T03:45:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0110000000000001e-06 2024-09-11T03:45:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828270+0000", 2024-09-11T03:45:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001026838 2024-09-11T03:45:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828298+0000", 2024-09-11T03:45:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8561999999999999e-05 2024-09-11T03:45:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828312+0000", 2024-09-11T03:45:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3781000000000001e-05 2024-09-11T03:45:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828398+0000", 2024-09-11T03:45:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6336999999999997e-05 2024-09-11T03:45:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:45:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828405+0000", 2024-09-11T03:45:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9079999999999998e-06 2024-09-11T03:45:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830015+0000", 2024-09-11T03:45:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001609344 2024-09-11T03:45:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830095+0000", 2024-09-11T03:45:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0259000000000005e-05 2024-09-11T03:45:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10", 2024-09-11T03:45:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830248+0000", 2024-09-11T03:45:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015351500000000001 2024-09-11T03:45:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:45:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.831760+0000", 2024-09-11T03:45:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015116540000000001 2024-09-11T03:45:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.832800+0000", 2024-09-11T03:45:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010400800000000001 2024-09-11T03:45:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834760+0000", 2024-09-11T03:45:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019600809999999998 2024-09-11T03:45:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834804+0000", 2024-09-11T03:45:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3922999999999999e-05 2024-09-11T03:45:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834873+0000", 2024-09-11T03:45:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8596000000000005e-05 2024-09-11T03:45:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:346 2.0 2:0762776a:::smithi17557870-25:head [write 1729599~575687] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:45:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.833957+0000", 2024-09-11T03:45:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 392.77771026400001, 2024-09-11T03:45:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00864616, 2024-09-11T03:45:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 346 2024-09-11T03:45:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833957+0000", 2024-09-11T03:45:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833957+0000", 2024-09-11T03:45:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833958+0000", 2024-09-11T03:45:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.674e-06 2024-09-11T03:45:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838969+0000", 2024-09-11T03:45:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005010993 2024-09-11T03:45:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839016+0000", 2024-09-11T03:45:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6576000000000003e-05 2024-09-11T03:45:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839037+0000", 2024-09-11T03:45:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1356e-05 2024-09-11T03:45:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839197+0000", 2024-09-11T03:45:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000159338 2024-09-11T03:45:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839266+0000", 2024-09-11T03:45:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9758999999999994e-05 2024-09-11T03:45:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10", 2024-09-11T03:45:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839374+0000", 2024-09-11T03:45:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010791899999999999 2024-09-11T03:45:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:45:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.840763+0000", 2024-09-11T03:45:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013883999999999999 2024-09-11T03:45:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842254+0000", 2024-09-11T03:45:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001491409 2024-09-11T03:45:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842538+0000", 2024-09-11T03:45:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028420099999999998 2024-09-11T03:45:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842573+0000", 2024-09-11T03:45:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4381000000000002e-05 2024-09-11T03:45:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842603+0000", 2024-09-11T03:45:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0154e-05 2024-09-11T03:45:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1152 2.b e183/95 2:de6b1316:::smithi17557870-32:head v 183'96, mlcod=182'92)", 2024-09-11T03:45:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:59.878348+0000", 2024-09-11T03:45:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 322.73331897700001, 2024-09-11T03:45:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064294529999999999, 2024-09-11T03:45:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.878348+0000", 2024-09-11T03:45:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.878348+0000", 2024-09-11T03:45:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.878348+0000", 2024-09-11T03:45:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8900000000000001e-07 2024-09-11T03:45:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.882607+0000", 2024-09-11T03:45:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042589139999999999 2024-09-11T03:45:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.882621+0000", 2024-09-11T03:45:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3925e-05 2024-09-11T03:45:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.882628+0000", 2024-09-11T03:45:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6000000000000003e-06 2024-09-11T03:45:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.882686+0000", 2024-09-11T03:45:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8579999999999998e-05 2024-09-11T03:45:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.882703+0000", 2024-09-11T03:45:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7167999999999999e-05 2024-09-11T03:45:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.884729+0000", 2024-09-11T03:45:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020253720000000001 2024-09-11T03:45:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:59.884777+0000", 2024-09-11T03:45:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8705000000000002e-05 2024-09-11T03:45:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1161 2.8 2:1bdf1a36:::smithi17557870-36:head [delete] snapc 2e=[2e,2c,2b,28,26] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:45:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:01.831716+0000", 2024-09-11T03:45:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 320.77995058800002, 2024-09-11T03:45:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065461870000000002, 2024-09-11T03:45:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1161 2024-09-11T03:45:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831716+0000", 2024-09-11T03:45:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831716+0000", 2024-09-11T03:45:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831718+0000", 2024-09-11T03:45:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1890000000000001e-06 2024-09-11T03:45:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831726+0000", 2024-09-11T03:45:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5190000000000003e-06 2024-09-11T03:45:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831741+0000", 2024-09-11T03:45:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5407e-05 2024-09-11T03:45:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831749+0000", 2024-09-11T03:45:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0129999999999993e-06 2024-09-11T03:45:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831837+0000", 2024-09-11T03:45:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7558000000000004e-05 2024-09-11T03:45:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:45:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.831845+0000", 2024-09-11T03:45:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3799999999999994e-06 2024-09-11T03:45:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.834626+0000", 2024-09-11T03:45:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027804140000000001 2024-09-11T03:45:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.834743+0000", 2024-09-11T03:45:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117569 2024-09-11T03:45:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,9", 2024-09-11T03:45:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.834902+0000", 2024-09-11T03:45:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015838700000000001 2024-09-11T03:45:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:45:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.835777+0000", 2024-09-11T03:45:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087511700000000002 2024-09-11T03:45:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.836462+0000", 2024-09-11T03:45:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000684686 2024-09-11T03:45:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.838188+0000", 2024-09-11T03:45:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001726193 2024-09-11T03:45:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.838229+0000", 2024-09-11T03:45:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0991000000000001e-05 2024-09-11T03:45:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:01.838262+0000", 2024-09-11T03:45:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3763999999999997e-05 2024-09-11T03:45:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1555 2.3 e214/95 2:c6c49f2c:::smithi17557870-44:head v 214'74, mlcod=198'70)", 2024-09-11T03:45:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:30.893545+0000", 2024-09-11T03:45:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 291.71812138600001, 2024-09-11T03:45:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070433960000000004, 2024-09-11T03:45:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.893545+0000", 2024-09-11T03:45:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.893545+0000", 2024-09-11T03:45:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.893546+0000", 2024-09-11T03:45:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7500000000000001e-07 2024-09-11T03:45:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895656+0000", 2024-09-11T03:45:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002110476 2024-09-11T03:45:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895679+0000", 2024-09-11T03:45:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2795000000000001e-05 2024-09-11T03:45:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895692+0000", 2024-09-11T03:45:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3141e-05 2024-09-11T03:45:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895758+0000", 2024-09-11T03:45:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5871999999999995e-05 2024-09-11T03:45:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895786+0000", 2024-09-11T03:45:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8391e-05 2024-09-11T03:45:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900550+0000", 2024-09-11T03:45:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047632869999999997 2024-09-11T03:45:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900589+0000", 2024-09-11T03:45:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9159000000000002e-05 2024-09-11T03:45:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1609 2.8 2:1bdf1a36:::smithi17557870-36:head [write 618557~727148] snapc 40=[40,3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:45:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:35.905306+0000", 2024-09-11T03:45:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 286.70636131700002, 2024-09-11T03:45:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069719600000000001, 2024-09-11T03:45:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1609 2024-09-11T03:45:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.905306+0000", 2024-09-11T03:45:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.905306+0000", 2024-09-11T03:45:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.905307+0000", 2024-09-11T03:45:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.482e-06 2024-09-11T03:45:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907145+0000", 2024-09-11T03:45:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001838412 2024-09-11T03:45:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907157+0000", 2024-09-11T03:45:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-09-11T03:45:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907165+0000", 2024-09-11T03:45:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6340000000000001e-06 2024-09-11T03:45:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907215+0000", 2024-09-11T03:45:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0760999999999997e-05 2024-09-11T03:45:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907282+0000", 2024-09-11T03:45:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6934000000000007e-05 2024-09-11T03:45:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,9", 2024-09-11T03:45:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907400+0000", 2024-09-11T03:45:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117625 2024-09-11T03:45:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:45:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.908820+0000", 2024-09-11T03:45:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0014204140000000001 2024-09-11T03:45:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.910145+0000", 2024-09-11T03:45:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013246250000000001 2024-09-11T03:45:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.912194+0000", 2024-09-11T03:45:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020485640000000001 2024-09-11T03:45:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.912229+0000", 2024-09-11T03:45:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5128e-05 2024-09-11T03:45:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.912277+0000", 2024-09-11T03:45:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8746999999999999e-05 2024-09-11T03:45:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1868 2.0 2:0762776a:::smithi17557870-25:head [copy-from ver 218] snapc 4c=[4c,4b,49,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:45:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:01.850162+0000", 2024-09-11T03:45:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 260.76150498800001, 2024-09-11T03:45:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020677853999999999, 2024-09-11T03:45:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1868 2024-09-11T03:45:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850162+0000", 2024-09-11T03:45:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850162+0000", 2024-09-11T03:45:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850164+0000", 2024-09-11T03:45:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.588e-06 2024-09-11T03:45:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850174+0000", 2024-09-11T03:45:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8300000000000008e-06 2024-09-11T03:45:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850193+0000", 2024-09-11T03:45:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8346e-05 2024-09-11T03:45:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850204+0000", 2024-09-11T03:45:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1144e-05 2024-09-11T03:45:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850285+0000", 2024-09-11T03:45:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1525000000000003e-05 2024-09-11T03:45:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:45:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850293+0000", 2024-09-11T03:45:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.072e-06 2024-09-11T03:45:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.852390+0000", 2024-09-11T03:45:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002096601 2024-09-11T03:45:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.852527+0000", 2024-09-11T03:45:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013703899999999999 2024-09-11T03:45:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,11", 2024-09-11T03:45:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.862911+0000", 2024-09-11T03:45:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01038392 2024-09-11T03:45:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:45:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.867006+0000", 2024-09-11T03:45:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004095291 2024-09-11T03:45:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.868758+0000", 2024-09-11T03:45:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017514869999999999 2024-09-11T03:45:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.870714+0000", 2024-09-11T03:45:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019563229999999998 2024-09-11T03:45:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.870780+0000", 2024-09-11T03:45:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6459999999999997e-05 2024-09-11T03:45:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.870840+0000", 2024-09-11T03:45:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9228000000000003e-05 2024-09-11T03:45:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1869 2.0 2:0762776a:::smithi17557870-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:45:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:01.850232+0000", 2024-09-11T03:45:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 260.761434404, 2024-09-11T03:45:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016419715000000001, 2024-09-11T03:45:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:45:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1869 2024-09-11T03:45:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850232+0000", 2024-09-11T03:45:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850232+0000", 2024-09-11T03:45:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850233+0000", 2024-09-11T03:45:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5300000000000004e-07 2024-09-11T03:45:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850237+0000", 2024-09-11T03:45:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3660000000000002e-06 2024-09-11T03:45:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850247+0000", 2024-09-11T03:45:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.38e-06 2024-09-11T03:45:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850254+0000", 2024-09-11T03:45:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2099999999999996e-06 2024-09-11T03:45:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850365+0000", 2024-09-11T03:45:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110499 2024-09-11T03:45:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850369+0000", 2024-09-11T03:45:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.454e-06 2024-09-11T03:45:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.852747+0000", 2024-09-11T03:45:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023778480000000001 2024-09-11T03:45:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.852806+0000", 2024-09-11T03:45:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8931999999999998e-05 2024-09-11T03:45:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.866380+0000", 2024-09-11T03:45:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013574693000000001 2024-09-11T03:45:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.866512+0000", 2024-09-11T03:45:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013140999999999999 2024-09-11T03:45:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.866652+0000", 2024-09-11T03:45:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014027000000000001 2024-09-11T03:45:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1877 2.b e247/95 2:de6b1316:::smithi17557870-32:head v 247'232, mlcod=243'228)", 2024-09-11T03:45:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:03.916185+0000", 2024-09-11T03:45:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 258.695481874, 2024-09-11T03:45:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010307996, 2024-09-11T03:45:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.916185+0000", 2024-09-11T03:45:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.916185+0000", 2024-09-11T03:45:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.916185+0000", 2024-09-11T03:45:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1699999999999998e-07 2024-09-11T03:45:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924225+0000", 2024-09-11T03:45:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080395750000000002 2024-09-11T03:45:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924243+0000", 2024-09-11T03:45:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7955e-05 2024-09-11T03:45:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924271+0000", 2024-09-11T03:45:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7797000000000002e-05 2024-09-11T03:45:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924349+0000", 2024-09-11T03:45:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8137000000000001e-05 2024-09-11T03:45:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924370+0000", 2024-09-11T03:45:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1226e-05 2024-09-11T03:45:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.926454+0000", 2024-09-11T03:45:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002083438 2024-09-11T03:45:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.926493+0000", 2024-09-11T03:45:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9351000000000003e-05 2024-09-11T03:45:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1989 2.0 2:0762776a:::smithi17557870-25:head [copy-from ver 162] snapc 52=[52,51,4e,3d] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-11T03:45:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:14.949188+0000", 2024-09-11T03:45:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 247.662478808, 2024-09-11T03:45:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014886257999999999, 2024-09-11T03:45:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1989 2024-09-11T03:45:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949188+0000", 2024-09-11T03:45:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949188+0000", 2024-09-11T03:45:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949191+0000", 2024-09-11T03:45:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.706e-06 2024-09-11T03:45:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949204+0000", 2024-09-11T03:45:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3142999999999999e-05 2024-09-11T03:45:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949227+0000", 2024-09-11T03:45:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3320999999999999e-05 2024-09-11T03:45:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949238+0000", 2024-09-11T03:45:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0352e-05 2024-09-11T03:45:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949313+0000", 2024-09-11T03:45:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5450000000000004e-05 2024-09-11T03:45:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:45:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949319+0000", 2024-09-11T03:45:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9279999999999998e-06 2024-09-11T03:45:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.950847+0000", 2024-09-11T03:45:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015282690000000001 2024-09-11T03:45:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.950916+0000", 2024-09-11T03:45:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8559000000000005e-05 2024-09-11T03:45:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,11", 2024-09-11T03:45:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.956873+0000", 2024-09-11T03:45:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059576359999999997 2024-09-11T03:45:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:45:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.960470+0000", 2024-09-11T03:45:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035970749999999999 2024-09-11T03:45:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.963875+0000", 2024-09-11T03:45:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034049050000000002 2024-09-11T03:45:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.963958+0000", 2024-09-11T03:45:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2769999999999995e-05 2024-09-11T03:45:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.964021+0000", 2024-09-11T03:45:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2442999999999998e-05 2024-09-11T03:45:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.964074+0000", 2024-09-11T03:45:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3701e-05 2024-09-11T03:45:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1990 2.0 2:0762776a:::smithi17557870-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e258)", 2024-09-11T03:45:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:14.949265+0000", 2024-09-11T03:45:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 247.66240176700001, 2024-09-11T03:45:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010821580000000001, 2024-09-11T03:45:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:45:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1990 2024-09-11T03:45:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949265+0000", 2024-09-11T03:45:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949265+0000", 2024-09-11T03:45:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949266+0000", 2024-09-11T03:45:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3000000000000001e-07 2024-09-11T03:45:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949268+0000", 2024-09-11T03:45:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2249999999999999e-06 2024-09-11T03:45:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949273+0000", 2024-09-11T03:45:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6389999999999997e-06 2024-09-11T03:45:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949277+0000", 2024-09-11T03:45:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4630000000000001e-06 2024-09-11T03:45:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949380+0000", 2024-09-11T03:45:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103099 2024-09-11T03:45:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949387+0000", 2024-09-11T03:45:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9290000000000001e-06 2024-09-11T03:45:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.951035+0000", 2024-09-11T03:45:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001647995 2024-09-11T03:45:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.951082+0000", 2024-09-11T03:45:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7231999999999998e-05 2024-09-11T03:45:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.959865+0000", 2024-09-11T03:45:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087832770000000008 2024-09-11T03:45:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.959967+0000", 2024-09-11T03:45:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010153 2024-09-11T03:45:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.960087+0000", 2024-09-11T03:45:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011976099999999999 2024-09-11T03:45:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:2434 2.9 e295/276 2:939cf94d:::smithi17557870-28:head v 295'353, mlcod=295'349)", 2024-09-11T03:45:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:52.055937+0000", 2024-09-11T03:45:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 210.55572959099999, 2024-09-11T03:45:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071747089999999996, 2024-09-11T03:45:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.055937+0000", 2024-09-11T03:45:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.055937+0000", 2024-09-11T03:45:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.055938+0000", 2024-09-11T03:45:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.043e-06 2024-09-11T03:45:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.059618+0000", 2024-09-11T03:45:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003679435 2024-09-11T03:45:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.059642+0000", 2024-09-11T03:45:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4485999999999999e-05 2024-09-11T03:45:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.059671+0000", 2024-09-11T03:45:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8560000000000001e-05 2024-09-11T03:45:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.059750+0000", 2024-09-11T03:45:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9170000000000003e-05 2024-09-11T03:45:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.059780+0000", 2024-09-11T03:45:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9558999999999999e-05 2024-09-11T03:45:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.063060+0000", 2024-09-11T03:45:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003280383 2024-09-11T03:45:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.063112+0000", 2024-09-11T03:45:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2073000000000001e-05 2024-09-11T03:45:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3749 2.8 2:162357fa:::smithi17557870-18:head [write 773254~791982] snapc 9b=[9b,9a,99,98,97,90,8a] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-11T03:45:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:43:39.942169+0000", 2024-09-11T03:45:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 102.66949744999999, 2024-09-11T03:45:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0082841789999999992, 2024-09-11T03:45:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3749 2024-09-11T03:45:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.942169+0000", 2024-09-11T03:45:22.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.942169+0000", 2024-09-11T03:45:22.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.942171+0000", 2024-09-11T03:45:22.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0470000000000001e-06 2024-09-11T03:45:22.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.943249+0000", 2024-09-11T03:45:22.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010774459999999999 2024-09-11T03:45:22.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.943269+0000", 2024-09-11T03:45:22.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0063000000000001e-05 2024-09-11T03:45:22.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.943279+0000", 2024-09-11T03:45:22.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0010000000000001e-05 2024-09-11T03:45:22.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.943372+0000", 2024-09-11T03:45:22.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3196999999999996e-05 2024-09-11T03:45:22.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:45:22.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.943377+0000", 2024-09-11T03:45:22.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6990000000000002e-06 2024-09-11T03:45:22.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.944823+0000", 2024-09-11T03:45:22.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001446084 2024-09-11T03:45:22.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.944931+0000", 2024-09-11T03:45:22.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107786 2024-09-11T03:45:22.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,9", 2024-09-11T03:45:22.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.945043+0000", 2024-09-11T03:45:22.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000112083 2024-09-11T03:45:22.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:45:22.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.946640+0000", 2024-09-11T03:45:22.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015976409999999999 2024-09-11T03:45:22.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:22.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.948221+0000", 2024-09-11T03:45:22.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015802559999999999 2024-09-11T03:45:22.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:22.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.950386+0000", 2024-09-11T03:45:22.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021654869999999998 2024-09-11T03:45:22.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:22.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.950416+0000", 2024-09-11T03:45:22.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0086e-05 2024-09-11T03:45:22.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.950454+0000", 2024-09-11T03:45:22.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7293999999999999e-05 2024-09-11T03:45:22.709 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.709 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.709 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.710 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:3778 2.9 e405/276 2:924204df:::smithi17557870-50:head v 405'547, mlcod=396'543)", 2024-09-11T03:45:22.710 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:43:42.960014+0000", 2024-09-11T03:45:22.710 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 99.651653295000003, 2024-09-11T03:45:22.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006334765, 2024-09-11T03:45:22.710 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.710 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.710 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:22.710 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.960014+0000", 2024-09-11T03:45:22.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.960014+0000", 2024-09-11T03:45:22.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.960014+0000", 2024-09-11T03:45:22.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8199999999999999e-07 2024-09-11T03:45:22.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.963105+0000", 2024-09-11T03:45:22.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030911129999999999 2024-09-11T03:45:22.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.963118+0000", 2024-09-11T03:45:22.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2773e-05 2024-09-11T03:45:22.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.963128+0000", 2024-09-11T03:45:22.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0321e-05 2024-09-11T03:45:22.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.963223+0000", 2024-09-11T03:45:22.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5358000000000004e-05 2024-09-11T03:45:22.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.963245+0000", 2024-09-11T03:45:22.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1308e-05 2024-09-11T03:45:22.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:22.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.966304+0000", 2024-09-11T03:45:22.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030595240000000001 2024-09-11T03:45:22.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:42.966348+0000", 2024-09-11T03:45:22.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4186e-05 2024-09-11T03:45:22.714 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.714 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.714 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.715 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:4164 2.b e450/95 2:dde92fc6:::smithi17557870-41:head v 450'447, mlcod=449'443)", 2024-09-11T03:45:22.715 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:27.936778+0000", 2024-09-11T03:45:22.715 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 54.674889180000001, 2024-09-11T03:45:22.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069376189999999999, 2024-09-11T03:45:22.715 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.715 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.715 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:22.715 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.936778+0000", 2024-09-11T03:45:22.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.936778+0000", 2024-09-11T03:45:22.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.936778+0000", 2024-09-11T03:45:22.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8199999999999999e-07 2024-09-11T03:45:22.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.940955+0000", 2024-09-11T03:45:22.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041768120000000002 2024-09-11T03:45:22.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.940968+0000", 2024-09-11T03:45:22.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3757999999999999e-05 2024-09-11T03:45:22.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.940977+0000", 2024-09-11T03:45:22.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0310000000000004e-06 2024-09-11T03:45:22.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.941083+0000", 2024-09-11T03:45:22.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105611 2024-09-11T03:45:22.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.941103+0000", 2024-09-11T03:45:22.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9854999999999999e-05 2024-09-11T03:45:22.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:22.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.943669+0000", 2024-09-11T03:45:22.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025656979999999999 2024-09-11T03:45:22.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.943715+0000", 2024-09-11T03:45:22.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6672000000000003e-05 2024-09-11T03:45:22.719 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.719 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.719 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.719 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:4176 2.3 e451/276 2:c6c49f2c:::smithi17557870-44:head v 451'262, mlcod=449'258)", 2024-09-11T03:45:22.719 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:28.944463+0000", 2024-09-11T03:45:22.719 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.667203870999998, 2024-09-11T03:45:22.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0068327259999999999, 2024-09-11T03:45:22.719 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.719 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.719 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:22.719 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.944463+0000", 2024-09-11T03:45:22.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.944463+0000", 2024-09-11T03:45:22.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.944464+0000", 2024-09-11T03:45:22.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3200000000000003e-07 2024-09-11T03:45:22.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.946801+0000", 2024-09-11T03:45:22.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023370729999999998 2024-09-11T03:45:22.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.946844+0000", 2024-09-11T03:45:22.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3098e-05 2024-09-11T03:45:22.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.946861+0000", 2024-09-11T03:45:22.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6501e-05 2024-09-11T03:45:22.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.946957+0000", 2024-09-11T03:45:22.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6558000000000006e-05 2024-09-11T03:45:22.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.946984+0000", 2024-09-11T03:45:22.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6421999999999999e-05 2024-09-11T03:45:22.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:22.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.951247+0000", 2024-09-11T03:45:22.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042632119999999997 2024-09-11T03:45:22.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.951296+0000", 2024-09-11T03:45:22.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8930000000000001e-05 2024-09-11T03:45:22.724 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.725 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.725 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.725 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4288 2.0 2:0762776a:::smithi17557870-25:head [copy-from ver 488] snapc ba=[ba,b9,b8,b7,b6,b5,b1,af,ae,ac] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-11T03:45:22.725 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:39.973423+0000", 2024-09-11T03:45:22.725 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 42.638244161999999, 2024-09-11T03:45:22.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028521689999999999, 2024-09-11T03:45:22.725 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.726 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.726 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:22.726 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:22.726 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:22.726 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:22.726 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4288 2024-09-11T03:45:22.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.726 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973423+0000", 2024-09-11T03:45:22.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973423+0000", 2024-09-11T03:45:22.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973424+0000", 2024-09-11T03:45:22.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3599999999999999e-06 2024-09-11T03:45:22.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973431+0000", 2024-09-11T03:45:22.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.55e-06 2024-09-11T03:45:22.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973441+0000", 2024-09-11T03:45:22.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0383e-05 2024-09-11T03:45:22.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973446+0000", 2024-09-11T03:45:22.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3689999999999998e-06 2024-09-11T03:45:22.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973498+0000", 2024-09-11T03:45:22.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1412000000000002e-05 2024-09-11T03:45:22.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973592+0000", 2024-09-11T03:45:22.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4557999999999998e-05 2024-09-11T03:45:22.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10", 2024-09-11T03:45:22.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.987776+0000", 2024-09-11T03:45:22.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014183553 2024-09-11T03:45:22.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:45:22.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.991768+0000", 2024-09-11T03:45:22.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039917620000000003 2024-09-11T03:45:22.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:22.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.999033+0000", 2024-09-11T03:45:22.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072650620000000001 2024-09-11T03:45:22.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:22.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:40.001866+0000", 2024-09-11T03:45:22.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028328160000000002 2024-09-11T03:45:22.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:22.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:40.001905+0000", 2024-09-11T03:45:22.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9737999999999998e-05 2024-09-11T03:45:22.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:40.001944+0000", 2024-09-11T03:45:22.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9127e-05 2024-09-11T03:45:22.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.732 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.732 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.732 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4289 2.0 2:0762776a:::smithi17557870-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e462)", 2024-09-11T03:45:22.732 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:39.973461+0000", 2024-09-11T03:45:22.732 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 42.638206144000002, 2024-09-11T03:45:22.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017766608, 2024-09-11T03:45:22.732 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.732 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.732 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:45:22.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:45:22.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:45:22.733 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:22.733 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4289 2024-09-11T03:45:22.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.733 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973461+0000", 2024-09-11T03:45:22.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973461+0000", 2024-09-11T03:45:22.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973461+0000", 2024-09-11T03:45:22.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0399999999999996e-07 2024-09-11T03:45:22.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973467+0000", 2024-09-11T03:45:22.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8780000000000003e-06 2024-09-11T03:45:22.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973472+0000", 2024-09-11T03:45:22.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3009999999999997e-06 2024-09-11T03:45:22.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973477+0000", 2024-09-11T03:45:22.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7570000000000004e-06 2024-09-11T03:45:22.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973749+0000", 2024-09-11T03:45:22.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027214900000000001 2024-09-11T03:45:22.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:22.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973782+0000", 2024-09-11T03:45:22.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2543000000000003e-05 2024-09-11T03:45:22.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.991066+0000", 2024-09-11T03:45:22.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017284568 2024-09-11T03:45:22.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.991145+0000", 2024-09-11T03:45:22.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8498000000000006e-05 2024-09-11T03:45:22.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.991227+0000", 2024-09-11T03:45:22.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2410000000000005e-05 2024-09-11T03:45:22.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.737 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.737 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:4688 2.18 e503/312 2:1984a3a9:::smithi17557870-23:head v 503'349, mlcod=498'344)", 2024-09-11T03:45:22.737 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:45:21.011490+0000", 2024-09-11T03:45:22.738 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.600176467, 2024-09-11T03:45:22.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019963814, 2024-09-11T03:45:22.738 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:45:22.738 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:45:22.738 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:22.738 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:45:22.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:45:22.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.011490+0000", 2024-09-11T03:45:22.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:45:22.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.011490+0000", 2024-09-11T03:45:22.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:45:22.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:45:22.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.011491+0000", 2024-09-11T03:45:22.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7000000000000003e-07 2024-09-11T03:45:22.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:45:22.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030626+0000", 2024-09-11T03:45:22.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019134880999999999 2024-09-11T03:45:22.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:45:22.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030636+0000", 2024-09-11T03:45:22.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0159e-05 2024-09-11T03:45:22.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:45:22.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030642+0000", 2024-09-11T03:45:22.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4070000000000001e-06 2024-09-11T03:45:22.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:45:22.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030686+0000", 2024-09-11T03:45:22.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4320999999999998e-05 2024-09-11T03:45:22.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:45:22.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030710+0000", 2024-09-11T03:45:22.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3495000000000001e-05 2024-09-11T03:45:22.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:45:22.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.031422+0000", 2024-09-11T03:45:22.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000712588 2024-09-11T03:45:22.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:45:22.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:45:22.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:45:22.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.031454+0000", 2024-09-11T03:45:22.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2092999999999999e-05 2024-09-11T03:45:22.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.742 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:45:22.742 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:45:22.742 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:45:22.743 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:22.769 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:22.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:22.770 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:22.777 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:45:22.777 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-11T03:45:22.819 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:22.871 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:22.894 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:22.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:22.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:22.948 INFO:tasks.rados.rados.2.smithi175.stdout:1872: done (3 left) 2024-09-11T03:45:22.949 INFO:tasks.rados.rados.2.smithi175.stdout:1873: snap_create 2024-09-11T03:45:22.949 INFO:tasks.rados.rados.2.smithi175.stdout:1868: finishing write tid 2 to smithi17557870-27 2024-09-11T03:45:22.949 INFO:tasks.rados.rados.2.smithi175.stdout:1868: finishing write tid 3 to smithi17557870-27 2024-09-11T03:45:22.949 INFO:tasks.rados.rados.2.smithi175.stdout:1868: finishing write tid 4 to smithi17557870-27 2024-09-11T03:45:22.949 INFO:tasks.rados.rados.2.smithi175.stdout:1868: oid 27 updating version 0 to 696 2024-09-11T03:45:22.949 INFO:tasks.rados.rados.2.smithi175.stdout:1868: oid 27 version 696 is already newer than 695 2024-09-11T03:45:22.949 INFO:tasks.rados.rados.2.smithi175.stdout:1868: oid 27 version 696 is already newer than 694 2024-09-11T03:45:22.949 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 696 (ObjNum 642 snap 204 seq_num 642) dirty exists 2024-09-11T03:45:22.949 INFO:tasks.rados.rados.2.smithi175.stdout:1868: left oid 27 (ObjNum 642 snap 204 seq_num 642) 2024-09-11T03:45:22.949 INFO:tasks.rados.rados.2.smithi175.stdout:1870: expect (ObjNum 528 snap 156 seq_num 528) 2024-09-11T03:45:22.995 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:23.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:23.064 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:23.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:23.127 INFO:tasks.rados.rados.2.smithi175.stdout:1869: expect (ObjNum 565 snap 172 seq_num 565) 2024-09-11T03:45:23.166 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:23.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:23.188 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:23.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:23.289 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:23.302 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:45:23.303 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:45:23.310 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:23.313 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:23.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:23.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:23.319 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:45:23.320 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:45:23.342 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:45:23.342 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:45:23.358 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:45:23.359 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:45:23.376 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:45:23.377 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:45:23.392 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:45:23.392 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:45:23.409 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:45:23.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:45:23.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:23.425 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:45:23.425 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:45:23.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:23.437 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:23.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:23.441 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:45:23.442 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:45:23.458 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:45:23.459 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:45:23.475 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:45:23.475 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:45:23.491 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:45:23.492 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:45:23.538 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:23.545 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-11T03:45:23.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:23.562 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:23.562 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:23.663 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:23.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:23.685 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:23.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:23.786 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:23.806 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:45:23.808 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:23.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:23.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:23.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:23.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:23.933 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:23.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:23.948 INFO:tasks.rados.rados.2.smithi175.stdout:1868: done (3 left) 2024-09-11T03:45:23.948 INFO:tasks.rados.rados.2.smithi175.stdout:1869: done (2 left) 2024-09-11T03:45:23.948 INFO:tasks.rados.rados.2.smithi175.stdout:1870: done (1 left) 2024-09-11T03:45:23.949 INFO:tasks.rados.rados.2.smithi175.stdout:1873: done (0 left) 2024-09-11T03:45:23.949 INFO:tasks.rados.rados.2.smithi175.stdout:1874: rollback oid 26 current snap is 205 2024-09-11T03:45:23.949 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 26 to 189 2024-09-11T03:45:23.949 INFO:tasks.rados.rados.2.smithi175.stdout:1875: delete oid 1 current snap is 205 2024-09-11T03:45:23.951 INFO:tasks.rados.rados.2.smithi175.stdout:1874: finishing rollback tid 0 to smithi17557870-26 2024-09-11T03:45:23.951 INFO:tasks.rados.rados.2.smithi175.stdout:1874: finishing rollback tid 1 to smithi17557870-26 2024-09-11T03:45:23.952 INFO:tasks.rados.rados.2.smithi175.stdout:1874: finishing rollback tid 2 to smithi17557870-26 2024-09-11T03:45:23.952 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 473 (ObjNum 595 snap 185 seq_num 595) dirty exists 2024-09-11T03:45:23.952 INFO:tasks.rados.rados.2.smithi175.stdout:1874: done (1 left) 2024-09-11T03:45:23.952 INFO:tasks.rados.rados.2.smithi175.stdout:1875: done (0 left) 2024-09-11T03:45:23.952 INFO:tasks.rados.rados.2.smithi175.stdout:1876: rollback oid 2 current snap is 205 2024-09-11T03:45:23.952 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 2 to 189 2024-09-11T03:45:23.952 INFO:tasks.rados.rados.2.smithi175.stdout:1877: read oid 33 snap -1 2024-09-11T03:45:23.952 INFO:tasks.rados.rados.2.smithi175.stdout:1877: expect (ObjNum 630 snap 202 seq_num 630) 2024-09-11T03:45:23.952 INFO:tasks.rados.rados.2.smithi175.stdout:1878: copy_from oid 5 from oid 31 current snap is 205 2024-09-11T03:45:23.952 INFO:tasks.rados.rados.2.smithi175.stdout:1879: snap_remove snap 204 2024-09-11T03:45:24.035 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:24.057 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:24.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:24.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:23 smithi084 ceph-mon[27254]: pgmap v912: 57 pgs: 57 active+clean; 208 MiB data, 834 MiB used, 983 GiB / 983 GiB avail; 19 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2024-09-11T03:45:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:23 smithi084 ceph-mon[27254]: osdmap e505: 12 total, 12 up, 11 in 2024-09-11T03:45:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:23 smithi084 ceph-mon[27254]: 2.19 scrub starts 2024-09-11T03:45:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:23 smithi084 ceph-mon[27254]: 2.19 scrub ok 2024-09-11T03:45:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:23 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/635785710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:24.159 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:24.180 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:24.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:24.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:23 smithi175 ceph-mon[30894]: pgmap v912: 57 pgs: 57 active+clean; 208 MiB data, 834 MiB used, 983 GiB / 983 GiB avail; 19 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2024-09-11T03:45:24.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:23 smithi175 ceph-mon[30894]: osdmap e505: 12 total, 12 up, 11 in 2024-09-11T03:45:24.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:23 smithi175 ceph-mon[30894]: 2.19 scrub starts 2024-09-11T03:45:24.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:23 smithi175 ceph-mon[30894]: 2.19 scrub ok 2024-09-11T03:45:24.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:23 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/635785710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:24.259 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:45:24.259 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:45:24.283 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:24.305 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:24.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:24.307 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:24.408 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:24.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:23 smithi121 ceph-mon[30729]: pgmap v912: 57 pgs: 57 active+clean; 208 MiB data, 834 MiB used, 983 GiB / 983 GiB avail; 19 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2024-09-11T03:45:24.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:23 smithi121 ceph-mon[30729]: osdmap e505: 12 total, 12 up, 11 in 2024-09-11T03:45:24.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:23 smithi121 ceph-mon[30729]: 2.19 scrub starts 2024-09-11T03:45:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:23 smithi121 ceph-mon[30729]: 2.19 scrub ok 2024-09-11T03:45:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:23 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/635785710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:24.429 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:24.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:24.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:24.531 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:24.552 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:24.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:24.553 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:24.654 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:24.676 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:24.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:24.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:24.745 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:45:24.778 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:24.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:24.801 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:24.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:24.902 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:24.925 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:24.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:24.951 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T03:45:24.960 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-11T03:45:24.963 INFO:tasks.rados.rados.2.smithi175.stdout:1879: done (3 left) 2024-09-11T03:45:24.963 INFO:tasks.rados.rados.2.smithi175.stdout:1880: read oid 44 snap -1 2024-09-11T03:45:24.963 INFO:tasks.rados.rados.2.smithi175.stdout:1880: expect (ObjNum 598 snap 186 seq_num 598) 2024-09-11T03:45:24.963 INFO:tasks.rados.rados.2.smithi175.stdout:1881: read oid 39 snap -1 2024-09-11T03:45:24.963 INFO:tasks.rados.rados.2.smithi175.stdout:1881: expect (ObjNum 528 snap 156 seq_num 528) 2024-09-11T03:45:24.963 INFO:tasks.rados.rados.2.smithi175.stdout:1882: snap_create 2024-09-11T03:45:24.963 INFO:tasks.rados.rados.2.smithi175.stdout:1876: finishing rollback tid 0 to smithi17557870-2 2024-09-11T03:45:24.963 INFO:tasks.rados.rados.2.smithi175.stdout:1876: finishing rollback tid 1 to smithi17557870-2 2024-09-11T03:45:24.963 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 703 (ObjNum 241 snap 0 seq_num 1745294416) dirty dne 2024-09-11T03:45:24.963 INFO:tasks.rados.rados.2.smithi175.stdout:1877: expect (ObjNum 630 snap 202 seq_num 630) 2024-09-11T03:45:25.027 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:25.053 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:25.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:25.054 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:25.125 INFO:tasks.rados.rados.2.smithi175.stdout:1878: finishing copy_from racing read to smithi17557870-5 2024-09-11T03:45:25.125 INFO:tasks.rados.rados.2.smithi175.stdout:1878: finishing copy_from to smithi17557870-5 2024-09-11T03:45:25.125 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 681 (ObjNum 495 snap 143 seq_num 495) dirty exists 2024-09-11T03:45:25.125 INFO:tasks.rados.rados.2.smithi175.stdout:1881: expect (ObjNum 528 snap 156 seq_num 528) 2024-09-11T03:45:25.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:24 smithi084 ceph-mon[27254]: 1.0 scrub starts 2024-09-11T03:45:25.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:24 smithi084 ceph-mon[27254]: 1.0 scrub ok 2024-09-11T03:45:25.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:24 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:45:25.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:24 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:45:25.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:24 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:45:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:24 smithi084 ceph-mon[27254]: 2.9 scrub ok 2024-09-11T03:45:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:24 smithi084 ceph-mon[27254]: osdmap e506: 12 total, 12 up, 11 in 2024-09-11T03:45:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:24 smithi084 ceph-mon[27254]: 1.a scrub starts 2024-09-11T03:45:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:24 smithi084 ceph-mon[27254]: 1.a scrub ok 2024-09-11T03:45:25.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:24 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2642104943' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:45:25.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:25.176 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:25.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:25.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:25.219 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:45:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:24 smithi175 ceph-mon[30894]: 1.0 scrub starts 2024-09-11T03:45:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:24 smithi175 ceph-mon[30894]: 1.0 scrub ok 2024-09-11T03:45:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:24 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:45:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:24 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:45:25.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:24 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:45:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:24 smithi175 ceph-mon[30894]: 2.9 scrub ok 2024-09-11T03:45:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:24 smithi175 ceph-mon[30894]: osdmap e506: 12 total, 12 up, 11 in 2024-09-11T03:45:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:24 smithi175 ceph-mon[30894]: 1.a scrub starts 2024-09-11T03:45:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:24 smithi175 ceph-mon[30894]: 1.a scrub ok 2024-09-11T03:45:25.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:24 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2642104943' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:45:25.278 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:25.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:25.300 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:25.300 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:25.318 INFO:tasks.rados.rados.2.smithi175.stdout:1880: expect (ObjNum 598 snap 186 seq_num 598) 2024-09-11T03:45:25.402 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:24 smithi121 ceph-mon[30729]: 1.0 scrub starts 2024-09-11T03:45:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:24 smithi121 ceph-mon[30729]: 1.0 scrub ok 2024-09-11T03:45:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:24 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:45:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:24 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:45:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:24 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:45:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:24 smithi121 ceph-mon[30729]: 2.9 scrub ok 2024-09-11T03:45:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:24 smithi121 ceph-mon[30729]: osdmap e506: 12 total, 12 up, 11 in 2024-09-11T03:45:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:24 smithi121 ceph-mon[30729]: 1.a scrub starts 2024-09-11T03:45:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:24 smithi121 ceph-mon[30729]: 1.a scrub ok 2024-09-11T03:45:25.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:24 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2642104943' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:45:25.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:25.424 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:25.424 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:25.525 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:25.546 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:25.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:25.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:25.648 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:25.658 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:45:25.658 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:45:25.669 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:25.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:25.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:25.744 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:25.771 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:25.775 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-11T03:45:25.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:25.794 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:25.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:25.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:25.918 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:25.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:25.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:25.959 INFO:tasks.rados.rados.2.smithi175.stdout:1876: done (5 left) 2024-09-11T03:45:25.959 INFO:tasks.rados.rados.2.smithi175.stdout:1877: done (4 left) 2024-09-11T03:45:25.959 INFO:tasks.rados.rados.2.smithi175.stdout:1878: done (3 left) 2024-09-11T03:45:25.959 INFO:tasks.rados.rados.2.smithi175.stdout:1880: done (2 left) 2024-09-11T03:45:25.959 INFO:tasks.rados.rados.2.smithi175.stdout:1881: done (1 left) 2024-09-11T03:45:25.959 INFO:tasks.rados.rados.2.smithi175.stdout:1882: done (0 left) 2024-09-11T03:45:25.959 INFO:tasks.rados.rados.2.smithi175.stdout:1883: snap_create 2024-09-11T03:45:26.010 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:25 smithi084 ceph-mon[27254]: pgmap v915: 57 pgs: 57 active+clean; 212 MiB data, 844 MiB used, 983 GiB / 983 GiB avail; 19 MiB/s rd, 3.3 MiB/s wr, 24 op/s 2024-09-11T03:45:26.010 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:25 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2642104943' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:45:26.010 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:25 smithi084 ceph-mon[27254]: osdmap e507: 12 total, 12 up, 11 in 2024-09-11T03:45:26.020 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:26.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:26.042 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:26.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:26.144 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:26.167 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:26.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:26.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:25 smithi175 ceph-mon[30894]: pgmap v915: 57 pgs: 57 active+clean; 212 MiB data, 844 MiB used, 983 GiB / 983 GiB avail; 19 MiB/s rd, 3.3 MiB/s wr, 24 op/s 2024-09-11T03:45:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:25 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2642104943' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:45:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:25 smithi175 ceph-mon[30894]: osdmap e507: 12 total, 12 up, 11 in 2024-09-11T03:45:26.268 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:26.280 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:26.286 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:45:26.290 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:26.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:26.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:26.300 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-11T03:45:26.392 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:26.413 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:26.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:26.414 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:25 smithi121 ceph-mon[30729]: pgmap v915: 57 pgs: 57 active+clean; 212 MiB data, 844 MiB used, 983 GiB / 983 GiB avail; 19 MiB/s rd, 3.3 MiB/s wr, 24 op/s 2024-09-11T03:45:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:25 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2642104943' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:45:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:25 smithi121 ceph-mon[30729]: osdmap e507: 12 total, 12 up, 11 in 2024-09-11T03:45:26.515 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:26.536 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:26.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:26.537 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:26.543 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:45:26.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:26.667 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:26.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:26.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:26.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:26.778 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:26.794 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:26.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:26.896 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:26.918 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:26.919 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:26.941 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:45:26.945 INFO:tasks.rados.rados.2.smithi175.stdout:1883: done (0 left) 2024-09-11T03:45:26.945 INFO:tasks.rados.rados.2.smithi175.stdout:1884: snap_create 2024-09-11T03:45:27.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:27.025 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:45:27.026 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:45:27.045 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:27.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:27.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:27.147 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:27.170 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:27.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:27.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:26 smithi175 ceph-mon[30894]: osdmap e508: 12 total, 12 up, 11 in 2024-09-11T03:45:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2153860704' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:45:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:26 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:26 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:45:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:26 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:45:27.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2153860704' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:45:27.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:26 smithi175 ceph-mon[30894]: osdmap e509: 12 total, 12 up, 11 in 2024-09-11T03:45:27.271 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:27.293 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:27.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:27.294 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:27.334 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:26 smithi084 ceph-mon[27254]: osdmap e508: 12 total, 12 up, 11 in 2024-09-11T03:45:27.334 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2153860704' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:45:27.334 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:26 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:27.334 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:26 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:45:27.334 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:26 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:45:27.334 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2153860704' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:45:27.334 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:26 smithi084 ceph-mon[27254]: osdmap e509: 12 total, 12 up, 11 in 2024-09-11T03:45:27.395 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:27.421 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:27.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:27.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:26 smithi121 ceph-mon[30729]: osdmap e508: 12 total, 12 up, 11 in 2024-09-11T03:45:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2153860704' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:45:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:26 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:26 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:45:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:26 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:45:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2153860704' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:45:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:26 smithi121 ceph-mon[30729]: osdmap e509: 12 total, 12 up, 11 in 2024-09-11T03:45:27.450 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:45:27.450 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:45:27.451 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:45:27.451 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:45:27.465 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:45:27.465 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:45:27.481 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:45:27.481 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:45:27.498 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:45:27.498 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:45:27.515 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:45:27.515 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:45:27.522 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:27.532 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:45:27.532 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:45:27.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:27.545 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:27.545 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:27.549 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:45:27.549 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:45:27.566 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:45:27.566 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:45:27.582 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:45:27.583 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:45:27.599 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:45:27.599 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:45:27.615 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:45:27.616 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:45:27.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:27.650 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-11T03:45:27.672 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:27.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:27.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:27.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:27.799 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:27.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:27.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:27.889 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:45:27.901 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:27.923 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:27.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:27.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:27.943 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T03:45:27.949 INFO:tasks.rados.rados.2.smithi175.stdout:1884: done (0 left) 2024-09-11T03:45:27.949 INFO:tasks.rados.rados.2.smithi175.stdout:1885: read oid 7 snap -1 2024-09-11T03:45:27.949 INFO:tasks.rados.rados.2.smithi175.stdout:1885: expect (ObjNum 552 snap 171 seq_num 552) 2024-09-11T03:45:27.949 INFO:tasks.rados.rados.2.smithi175.stdout:1886: copy_from oid 17 from oid 47 current snap is 208 2024-09-11T03:45:27.950 INFO:tasks.rados.rados.2.smithi175.stdout:1887: copy_from oid 29 from oid 45 current snap is 208 2024-09-11T03:45:27.950 INFO:tasks.rados.rados.2.smithi175.stdout:1888: write oid 32 current snap is 208 2024-09-11T03:45:27.950 INFO:tasks.rados.rados.2.smithi175.stdout:1888: seq_num 645 ranges {440515=709257,1910768=583545} 2024-09-11T03:45:27.956 INFO:tasks.rados.rados.2.smithi175.stdout:1888: writing smithi17557870-32 from 440515 to 1149772 tid 1 2024-09-11T03:45:27.964 INFO:tasks.rados.rados.2.smithi175.stdout:1888: writing smithi17557870-32 from 1910768 to 2494313 tid 2 2024-09-11T03:45:27.964 INFO:tasks.rados.rados.2.smithi175.stdout:1889: write (excl) oid 40 current snap is 208 2024-09-11T03:45:27.964 INFO:tasks.rados.rados.2.smithi175.stdout:1887: finishing copy_from racing read to smithi17557870-29 2024-09-11T03:45:27.964 INFO:tasks.rados.rados.2.smithi175.stdout:1889: seq_num 646 ranges {457322=784506,1969250=443556} 2024-09-11T03:45:27.971 INFO:tasks.rados.rados.2.smithi175.stdout:1889: writing smithi17557870-40 from 457322 to 1241828 tid 1 2024-09-11T03:45:27.977 INFO:tasks.rados.rados.2.smithi175.stdout:1889: writing smithi17557870-40 from 1969250 to 2412806 tid 2 2024-09-11T03:45:27.978 INFO:tasks.rados.rados.2.smithi175.stdout:1890: delete oid 18 current snap is 208 2024-09-11T03:45:27.978 INFO:tasks.rados.rados.2.smithi175.stdout:1888: finishing write tid 1 to smithi17557870-32 2024-09-11T03:45:27.978 INFO:tasks.rados.rados.2.smithi175.stdout:1886: finishing copy_from to smithi17557870-17 2024-09-11T03:45:27.978 INFO:tasks.rados.rados.2.smithi175.stdout:1886: got expected ENOENT (src dne) 2024-09-11T03:45:27.978 INFO:tasks.rados.rados.2.smithi175.stdout:1887: finishing copy_from to smithi17557870-29 2024-09-11T03:45:27.978 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 481 (ObjNum 416 snap 127 seq_num 416) dirty exists 2024-09-11T03:45:27.978 INFO:tasks.rados.rados.2.smithi175.stdout:1885: expect (ObjNum 552 snap 171 seq_num 552) 2024-09-11T03:45:28.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:28.046 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:28.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:28.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:28.149 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:28.170 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:28.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:28.171 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:27 smithi175 ceph-mon[30894]: 1.1 scrub starts 2024-09-11T03:45:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:27 smithi175 ceph-mon[30894]: 1.1 scrub ok 2024-09-11T03:45:28.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:27 smithi175 ceph-mon[30894]: pgmap v918: 57 pgs: 57 active+clean; 211 MiB data, 844 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2024-09-11T03:45:28.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:27 smithi175 ceph-mon[30894]: 2.4 scrub starts 2024-09-11T03:45:28.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:27 smithi175 ceph-mon[30894]: 2.4 scrub ok 2024-09-11T03:45:28.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:27 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1912583132' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:45:28.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:27 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1913656705' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:28.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:27 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1912583132' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:45:28.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:27 smithi175 ceph-mon[30894]: osdmap e510: 12 total, 12 up, 11 in 2024-09-11T03:45:28.237 INFO:tasks.rados.rados.2.smithi175.stdout:1886: finishing copy_from racing read to smithi17557870-17 2024-09-11T03:45:28.237 INFO:tasks.rados.rados.2.smithi175.stdout:1888: finishing write tid 2 to smithi17557870-32 2024-09-11T03:45:28.237 INFO:tasks.rados.rados.2.smithi175.stdout:1888: finishing write tid 3 to smithi17557870-32 2024-09-11T03:45:28.237 INFO:tasks.rados.rados.2.smithi175.stdout:1888: finishing write tid 4 to smithi17557870-32 2024-09-11T03:45:28.237 INFO:tasks.rados.rados.2.smithi175.stdout:1888: oid 32 updating version 0 to 554 2024-09-11T03:45:28.237 INFO:tasks.rados.rados.2.smithi175.stdout:1888: oid 32 version 554 is already newer than 552 2024-09-11T03:45:28.237 INFO:tasks.rados.rados.2.smithi175.stdout:1888: oid 32 version 554 is already newer than 553 2024-09-11T03:45:28.237 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 554 (ObjNum 645 snap 208 seq_num 645) dirty exists 2024-09-11T03:45:28.237 INFO:tasks.rados.rados.2.smithi175.stdout:1888: left oid 32 (ObjNum 645 snap 208 seq_num 645) 2024-09-11T03:45:28.237 INFO:tasks.rados.rados.2.smithi175.stdout:1889: finishing write tid 1 to smithi17557870-40 2024-09-11T03:45:28.237 INFO:tasks.rados.rados.2.smithi175.stdout:1885: done (5 left) 2024-09-11T03:45:28.238 INFO:tasks.rados.rados.2.smithi175.stdout:1886: done (4 left) 2024-09-11T03:45:28.238 INFO:tasks.rados.rados.2.smithi175.stdout:1887: done (3 left) 2024-09-11T03:45:28.238 INFO:tasks.rados.rados.2.smithi175.stdout:1888: done (2 left) 2024-09-11T03:45:28.238 INFO:tasks.rados.rados.2.smithi175.stdout:1890: done (1 left) 2024-09-11T03:45:28.238 INFO:tasks.rados.rados.2.smithi175.stdout:1891: delete oid 44 current snap is 208 2024-09-11T03:45:28.238 INFO:tasks.rados.rados.2.smithi175.stdout:1889: finishing write tid 2 to smithi17557870-40 2024-09-11T03:45:28.238 INFO:tasks.rados.rados.2.smithi175.stdout:1889: finishing write tid 3 to smithi17557870-40 2024-09-11T03:45:28.238 INFO:tasks.rados.rados.2.smithi175.stdout:1889: finishing write tid 4 to smithi17557870-40 2024-09-11T03:45:28.238 INFO:tasks.rados.rados.2.smithi175.stdout:1889: oid 40 updating version 0 to 408 2024-09-11T03:45:28.238 INFO:tasks.rados.rados.2.smithi175.stdout:1889: oid 40 version 408 is already newer than 407 2024-09-11T03:45:28.238 INFO:tasks.rados.rados.2.smithi175.stdout:1889: oid 40 updating version 408 to 409 2024-09-11T03:45:28.238 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 409 (ObjNum 646 snap 208 seq_num 646) dirty exists 2024-09-11T03:45:28.239 INFO:tasks.rados.rados.2.smithi175.stdout:1889: left oid 40 (ObjNum 646 snap 208 seq_num 646) 2024-09-11T03:45:28.239 INFO:tasks.rados.rados.2.smithi175.stdout:1889: done (1 left) 2024-09-11T03:45:28.239 INFO:tasks.rados.rados.2.smithi175.stdout:1891: done (0 left) 2024-09-11T03:45:28.239 INFO:tasks.rados.rados.2.smithi175.stdout:1892: snap_remove snap 205 2024-09-11T03:45:28.273 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:28.293 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:28.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:28.294 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:28.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:27 smithi084 ceph-mon[27254]: 1.1 scrub starts 2024-09-11T03:45:28.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:27 smithi084 ceph-mon[27254]: 1.1 scrub ok 2024-09-11T03:45:28.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:27 smithi084 ceph-mon[27254]: pgmap v918: 57 pgs: 57 active+clean; 211 MiB data, 844 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2024-09-11T03:45:28.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:27 smithi084 ceph-mon[27254]: 2.4 scrub starts 2024-09-11T03:45:28.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:27 smithi084 ceph-mon[27254]: 2.4 scrub ok 2024-09-11T03:45:28.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:27 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1912583132' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:45:28.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:27 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1913656705' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:28.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:27 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1912583132' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:45:28.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:27 smithi084 ceph-mon[27254]: osdmap e510: 12 total, 12 up, 11 in 2024-09-11T03:45:28.366 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:45:28.366 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:45:28.366 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:45:28.366 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:45:28.395 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:28.417 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:28.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:28.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:28.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:27 smithi121 ceph-mon[30729]: 1.1 scrub starts 2024-09-11T03:45:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:27 smithi121 ceph-mon[30729]: 1.1 scrub ok 2024-09-11T03:45:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:27 smithi121 ceph-mon[30729]: pgmap v918: 57 pgs: 57 active+clean; 211 MiB data, 844 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2024-09-11T03:45:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:27 smithi121 ceph-mon[30729]: 2.4 scrub starts 2024-09-11T03:45:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:27 smithi121 ceph-mon[30729]: 2.4 scrub ok 2024-09-11T03:45:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:27 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1912583132' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:45:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:27 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1913656705' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:27 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1912583132' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:45:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:27 smithi121 ceph-mon[30729]: osdmap e510: 12 total, 12 up, 11 in 2024-09-11T03:45:28.519 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:28.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:28.541 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:28.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:28.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:28.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:28.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:28.770 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:28.792 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:28.793 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:28.894 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:28.920 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:28.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:28.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:28.975 INFO:tasks.rados.rados.2.smithi175.stdout:1892: done (0 left) 2024-09-11T03:45:28.975 INFO:tasks.rados.rados.2.smithi175.stdout:1893: delete oid 27 current snap is 208 2024-09-11T03:45:28.979 INFO:tasks.rados.rados.2.smithi175.stdout:1893: done (0 left) 2024-09-11T03:45:28.980 INFO:tasks.rados.rados.2.smithi175.stdout:1894: delete oid 29 current snap is 208 2024-09-11T03:45:28.981 INFO:tasks.rados.rados.2.smithi175.stdout:1894: done (0 left) 2024-09-11T03:45:28.981 INFO:tasks.rados.rados.2.smithi175.stdout:1895: snap_remove snap 203 2024-09-11T03:45:29.021 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:29.043 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:29.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:29.044 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:29.114 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-11T03:45:29.145 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:29.170 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:29.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:29.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:29.272 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:29.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:29.295 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:29.295 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:29.352 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:45:29.389 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-11T03:45:29.395 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:29.416 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:29.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:29.417 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:29.519 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:29.540 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:29.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:29.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:29.638 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:29.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:29.668 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:29.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:29.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:29.771 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:29.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:29.792 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:29.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:29.822 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:45:29.822 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:45:29.822 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:45:29.822 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:45:29.822 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:45:29.893 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:29.916 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:29.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:29.982 INFO:tasks.rados.rados.2.smithi175.stdout:1895: done (0 left) 2024-09-11T03:45:29.982 INFO:tasks.rados.rados.2.smithi175.stdout:1896: rollback oid 2 current snap is 208 2024-09-11T03:45:29.983 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 2 to 200 2024-09-11T03:45:29.983 INFO:tasks.rados.rados.2.smithi175.stdout:1897: snap_remove snap 175 2024-09-11T03:45:30.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:30.043 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:30.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:30.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:30.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:29 smithi084 ceph-mon[27254]: pgmap v921: 57 pgs: 57 active+clean; 212 MiB data, 843 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2024-09-11T03:45:30.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:29 smithi084 ceph-mon[27254]: osdmap e511: 12 total, 12 up, 11 in 2024-09-11T03:45:30.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:29 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:45:30.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:29 smithi084 ceph-mon[27254]: 2.15 scrub ok 2024-09-11T03:45:30.099 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:45:30.145 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:30.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:30.167 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:30.167 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:30.184 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:29 smithi175 ceph-mon[30894]: pgmap v921: 57 pgs: 57 active+clean; 212 MiB data, 843 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2024-09-11T03:45:30.185 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:29 smithi175 ceph-mon[30894]: osdmap e511: 12 total, 12 up, 11 in 2024-09-11T03:45:30.185 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:29 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:45:30.185 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:29 smithi175 ceph-mon[30894]: 2.15 scrub ok 2024-09-11T03:45:30.268 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:30.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:30.290 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:30.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:30.344 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:30.390 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:30.412 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:30.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:29 smithi121 ceph-mon[30729]: pgmap v921: 57 pgs: 57 active+clean; 212 MiB data, 843 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2024-09-11T03:45:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:29 smithi121 ceph-mon[30729]: osdmap e511: 12 total, 12 up, 11 in 2024-09-11T03:45:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:29 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:45:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:29 smithi121 ceph-mon[30729]: 2.15 scrub ok 2024-09-11T03:45:30.512 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-11T03:45:30.513 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:30.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:30.535 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:30.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:30.636 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:30.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:30.658 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:30.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:30.751 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:45:30.759 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:30.781 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:30.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:30.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:30.883 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:30.909 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:30.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:30.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:30.979 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T03:45:30.985 INFO:tasks.rados.rados.2.smithi175.stdout:1897: done (1 left) 2024-09-11T03:45:30.986 INFO:tasks.rados.rados.2.smithi175.stdout:1898: read oid 12 snap -1 2024-09-11T03:45:30.986 INFO:tasks.rados.rados.2.smithi175.stdout:1898: expect deleted 2024-09-11T03:45:30.986 INFO:tasks.rados.rados.2.smithi175.stdout:1899: snap_remove snap 202 2024-09-11T03:45:31.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:31.035 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:31.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:31.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:31.137 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:31.138 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:30 smithi175 ceph-mon[30894]: osdmap e512: 12 total, 12 up, 11 in 2024-09-11T03:45:31.138 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:30 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4035001911' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:45:31.159 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:31.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:31.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:31.238 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:45:31.238 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:45:31.238 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:45:31.238 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:45:31.238 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.239 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:847 2.d 2:b1c0f730:::smithi17557870-20:head [copy-from ver 90] snapc 1a=[1a,19] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-11T03:45:31.239 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:45:31.239 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 366.39795064800001, 2024-09-11T03:45:31.239 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013054913, 2024-09-11T03:45:31.239 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.239 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.239 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:31.239 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.239 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.239 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.239 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 847 2024-09-11T03:45:31.239 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.240 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.240 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.240 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.240 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:45:31.240 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.240 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.240 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.241 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.241 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:45:31.241 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.241 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.241 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.241 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.241 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839297+0000", 2024-09-11T03:45:31.241 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0049999999999999e-06 2024-09-11T03:45:31.241 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.241 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.241 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.241 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839309+0000", 2024-09-11T03:45:31.242 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1531e-05 2024-09-11T03:45:31.242 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.242 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.242 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.242 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839326+0000", 2024-09-11T03:45:31.242 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7788e-05 2024-09-11T03:45:31.242 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.242 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.242 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.242 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839336+0000", 2024-09-11T03:45:31.242 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5619999999999993e-06 2024-09-11T03:45:31.243 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.243 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.243 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.243 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839423+0000", 2024-09-11T03:45:31.243 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6790000000000001e-05 2024-09-11T03:45:31.243 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.243 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.243 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:45:31.243 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839434+0000", 2024-09-11T03:45:31.243 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0971e-05 2024-09-11T03:45:31.243 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.243 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.244 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.244 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841520+0000", 2024-09-11T03:45:31.244 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0020866299999999999 2024-09-11T03:45:31.244 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.244 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.244 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.244 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841590+0000", 2024-09-11T03:45:31.244 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9288000000000005e-05 2024-09-11T03:45:31.244 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.244 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.244 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:45:31.244 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.846430+0000", 2024-09-11T03:45:31.244 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0048408349999999999 2024-09-11T03:45:31.245 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.245 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.245 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:31.245 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849726+0000", 2024-09-11T03:45:31.245 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032955390000000001 2024-09-11T03:45:31.245 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.245 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.245 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.245 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852240+0000", 2024-09-11T03:45:31.245 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002513492 2024-09-11T03:45:31.245 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.245 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.245 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.246 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852278+0000", 2024-09-11T03:45:31.246 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7988999999999998e-05 2024-09-11T03:45:31.246 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.246 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.246 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:31.246 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852317+0000", 2024-09-11T03:45:31.246 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9310000000000001e-05 2024-09-11T03:45:31.246 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.246 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.246 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.246 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852350+0000", 2024-09-11T03:45:31.246 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3182999999999997e-05 2024-09-11T03:45:31.246 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.247 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.247 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.247 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.247 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.247 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:907 2.d 2:b6a25a79:::smithi17557870-47:head [copy-from ver 47] snapc 20=[20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-11T03:45:31.247 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:34.902174+0000", 2024-09-11T03:45:31.247 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 356.33507219299997, 2024-09-11T03:45:31.247 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011234144, 2024-09-11T03:45:31.247 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.247 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.247 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:31.247 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.247 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.247 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.248 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 907 2024-09-11T03:45:31.248 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.248 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.248 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.248 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.248 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902174+0000", 2024-09-11T03:45:31.248 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.248 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.248 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.248 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.248 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902174+0000", 2024-09-11T03:45:31.248 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.248 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.249 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.249 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.249 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902175+0000", 2024-09-11T03:45:31.249 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0189999999999999e-06 2024-09-11T03:45:31.249 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.249 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.249 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.249 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902178+0000", 2024-09-11T03:45:31.249 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4410000000000002e-06 2024-09-11T03:45:31.249 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.249 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.249 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.250 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902188+0000", 2024-09-11T03:45:31.250 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0023999999999999e-05 2024-09-11T03:45:31.250 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.250 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.250 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.250 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902193+0000", 2024-09-11T03:45:31.250 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.31e-06 2024-09-11T03:45:31.250 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.250 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.250 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.250 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902248+0000", 2024-09-11T03:45:31.250 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4562999999999999e-05 2024-09-11T03:45:31.250 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.250 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.251 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.251 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.902344+0000", 2024-09-11T03:45:31.251 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5841999999999995e-05 2024-09-11T03:45:31.251 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.251 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.251 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:45:31.251 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.907645+0000", 2024-09-11T03:45:31.251 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0053015579999999996 2024-09-11T03:45:31.251 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.251 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.251 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:31.251 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.909768+0000", 2024-09-11T03:45:31.251 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0021225020000000001 2024-09-11T03:45:31.252 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.252 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.252 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.252 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.912827+0000", 2024-09-11T03:45:31.252 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003059375 2024-09-11T03:45:31.252 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.252 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.252 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.252 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.913334+0000", 2024-09-11T03:45:31.252 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00050710499999999999 2024-09-11T03:45:31.252 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.252 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.252 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:31.252 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.913372+0000", 2024-09-11T03:45:31.253 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7486e-05 2024-09-11T03:45:31.253 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.253 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.253 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.253 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:34.913408+0000", 2024-09-11T03:45:31.253 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5918999999999999e-05 2024-09-11T03:45:31.253 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.253 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.253 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.253 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.253 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.253 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1290 2.d 2:b1c0f730:::smithi17557870-20:head [copy-from ver 48] snapc 33=[32,30,2c,2b] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:45:31.254 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:45:31.255 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:12.836022+0000", 2024-09-11T03:45:31.255 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 318.401223367, 2024-09-11T03:45:31.255 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022892863999999999, 2024-09-11T03:45:31.255 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.255 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.255 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:31.255 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.255 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.256 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.256 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1290 2024-09-11T03:45:31.256 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.256 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.256 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.256 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.256 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836022+0000", 2024-09-11T03:45:31.256 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.256 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.256 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.256 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.256 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836022+0000", 2024-09-11T03:45:31.257 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.257 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.257 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.257 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.257 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836025+0000", 2024-09-11T03:45:31.257 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6690000000000002e-06 2024-09-11T03:45:31.257 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.257 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.257 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.257 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836033+0000", 2024-09-11T03:45:31.257 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0809999999999994e-06 2024-09-11T03:45:31.257 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.258 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.258 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.258 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836053+0000", 2024-09-11T03:45:31.258 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9429999999999999e-05 2024-09-11T03:45:31.258 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.258 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.258 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.258 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836063+0000", 2024-09-11T03:45:31.258 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0438000000000001e-05 2024-09-11T03:45:31.258 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.258 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.259 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.259 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836165+0000", 2024-09-11T03:45:31.259 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000101523 2024-09-11T03:45:31.259 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.259 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.259 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:45:31.259 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836172+0000", 2024-09-11T03:45:31.259 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6499999999999996e-06 2024-09-11T03:45:31.259 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.259 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.259 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.259 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839394+0000", 2024-09-11T03:45:31.259 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032215590000000001 2024-09-11T03:45:31.260 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.260 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.260 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.260 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839507+0000", 2024-09-11T03:45:31.260 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000113557 2024-09-11T03:45:31.260 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.260 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.260 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:45:31.260 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.849905+0000", 2024-09-11T03:45:31.260 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010398174 2024-09-11T03:45:31.260 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.260 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.260 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.853428+0000", 2024-09-11T03:45:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035222869999999998 2024-09-11T03:45:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.855923+0000", 2024-09-11T03:45:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024951840000000001 2024-09-11T03:45:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.858781+0000", 2024-09-11T03:45:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002858368 2024-09-11T03:45:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.858881+0000", 2024-09-11T03:45:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9500999999999995e-05 2024-09-11T03:45:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.858915+0000", 2024-09-11T03:45:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4443000000000002e-05 2024-09-11T03:45:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1291 2.d 2:b1c0f730:::smithi17557870-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:45:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:12.836223+0000", 2024-09-11T03:45:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 318.40102262099998, 2024-09-11T03:45:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016933050000000002, 2024-09-11T03:45:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:45:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1291 2024-09-11T03:45:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836223+0000", 2024-09-11T03:45:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836223+0000", 2024-09-11T03:45:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836224+0000", 2024-09-11T03:45:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0020000000000001e-06 2024-09-11T03:45:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836229+0000", 2024-09-11T03:45:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7180000000000004e-06 2024-09-11T03:45:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836245+0000", 2024-09-11T03:45:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5981000000000002e-05 2024-09-11T03:45:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836253+0000", 2024-09-11T03:45:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1720000000000007e-06 2024-09-11T03:45:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836317+0000", 2024-09-11T03:45:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4189999999999994e-05 2024-09-11T03:45:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836324+0000", 2024-09-11T03:45:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9249999999999998e-06 2024-09-11T03:45:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.267 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839738+0000", 2024-09-11T03:45:31.267 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0034135580000000001 2024-09-11T03:45:31.267 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.267 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.267 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:31.267 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839804+0000", 2024-09-11T03:45:31.267 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5974000000000002e-05 2024-09-11T03:45:31.267 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.267 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.267 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.267 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.852933+0000", 2024-09-11T03:45:31.267 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013129429999999999 2024-09-11T03:45:31.267 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.267 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.268 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.268 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.853032+0000", 2024-09-11T03:45:31.268 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8592999999999996e-05 2024-09-11T03:45:31.268 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.268 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.268 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.268 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.853156+0000", 2024-09-11T03:45:31.268 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012450699999999999 2024-09-11T03:45:31.268 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.268 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.268 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.268 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.268 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.269 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1516 2.d 2:b6a25a79:::smithi17557870-47:head [copy-from ver 97] snapc 3b=[3a,38,37,35] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-09-11T03:45:31.269 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:28.919589+0000", 2024-09-11T03:45:31.269 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 302.31765649099998, 2024-09-11T03:45:31.269 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011956165, 2024-09-11T03:45:31.269 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.269 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.269 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:31.269 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.269 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.269 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.269 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1516 2024-09-11T03:45:31.269 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.269 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.270 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.270 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.270 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919589+0000", 2024-09-11T03:45:31.270 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.270 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.270 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.270 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.270 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919589+0000", 2024-09-11T03:45:31.270 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.270 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.270 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.270 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.270 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919591+0000", 2024-09-11T03:45:31.271 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5409999999999999e-06 2024-09-11T03:45:31.271 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.271 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.271 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.271 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919596+0000", 2024-09-11T03:45:31.271 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9280000000000001e-06 2024-09-11T03:45:31.271 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.271 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.271 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.271 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919610+0000", 2024-09-11T03:45:31.271 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4285e-05 2024-09-11T03:45:31.271 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.271 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.271 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.272 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919617+0000", 2024-09-11T03:45:31.272 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.435e-06 2024-09-11T03:45:31.272 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.272 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.272 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.272 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919688+0000", 2024-09-11T03:45:31.272 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0263000000000001e-05 2024-09-11T03:45:31.272 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.272 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.272 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.272 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919794+0000", 2024-09-11T03:45:31.272 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000106055 2024-09-11T03:45:31.273 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.273 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.273 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:45:31.273 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.922849+0000", 2024-09-11T03:45:31.273 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030557330000000001 2024-09-11T03:45:31.273 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.273 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.273 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:31.273 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.926556+0000", 2024-09-11T03:45:31.273 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0037066669999999999 2024-09-11T03:45:31.273 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.273 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.273 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.274 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.929197+0000", 2024-09-11T03:45:31.274 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002640529 2024-09-11T03:45:31.274 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.274 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.274 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.274 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.931432+0000", 2024-09-11T03:45:31.274 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022350339999999999 2024-09-11T03:45:31.274 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.274 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.274 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:31.274 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.931501+0000", 2024-09-11T03:45:31.274 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9584000000000003e-05 2024-09-11T03:45:31.274 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.275 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.275 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.275 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.931545+0000", 2024-09-11T03:45:31.275 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4110999999999998e-05 2024-09-11T03:45:31.275 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.275 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.275 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.275 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.275 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.275 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2254 2.18 2:1984a3a9:::smithi17557870-23:head [write 746310~635349,stat] snapc 5d=[5d,5a,59,58,56] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:45:31.275 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.883765+0000", 2024-09-11T03:45:31.275 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 234.353480378, 2024-09-11T03:45:31.276 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022154396, 2024-09-11T03:45:31.276 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.276 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.276 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:31.276 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.276 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.276 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.276 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2254 2024-09-11T03:45:31.276 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.276 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.276 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.276 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.276 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.883765+0000", 2024-09-11T03:45:31.276 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.277 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.277 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.277 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.277 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.883765+0000", 2024-09-11T03:45:31.277 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.277 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.277 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.277 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.277 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.883768+0000", 2024-09-11T03:45:31.277 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4080000000000001e-06 2024-09-11T03:45:31.277 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.277 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.277 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.277 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884378+0000", 2024-09-11T03:45:31.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00061034200000000005 2024-09-11T03:45:31.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.278 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.278 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.278 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884399+0000", 2024-09-11T03:45:31.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1175e-05 2024-09-11T03:45:31.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.278 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.278 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.278 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884412+0000", 2024-09-11T03:45:31.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3015e-05 2024-09-11T03:45:31.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.278 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.279 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.279 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884520+0000", 2024-09-11T03:45:31.279 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000107706 2024-09-11T03:45:31.279 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.279 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.279 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:45:31.279 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884534+0000", 2024-09-11T03:45:31.279 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3923e-05 2024-09-11T03:45:31.279 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.279 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.279 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.279 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.895937+0000", 2024-09-11T03:45:31.279 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011403207 2024-09-11T03:45:31.279 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.280 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.280 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:45:31.280 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896424+0000", 2024-09-11T03:45:31.280 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00048641199999999998 2024-09-11T03:45:31.280 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.280 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.280 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.280 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.898138+0000", 2024-09-11T03:45:31.280 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001714272 2024-09-11T03:45:31.280 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.280 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.280 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.280 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.898309+0000", 2024-09-11T03:45:31.280 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017075200000000001 2024-09-11T03:45:31.281 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.281 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.281 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:45:31.281 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.898561+0000", 2024-09-11T03:45:31.281 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00025223500000000001 2024-09-11T03:45:31.281 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.281 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.281 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.281 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903048+0000", 2024-09-11T03:45:31.281 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0044876320000000001 2024-09-11T03:45:31.281 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.281 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.281 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.281 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903118+0000", 2024-09-11T03:45:31.282 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9608000000000002e-05 2024-09-11T03:45:31.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.282 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:31.282 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905694+0000", 2024-09-11T03:45:31.282 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025761650000000001 2024-09-11T03:45:31.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.282 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:31.282 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905749+0000", 2024-09-11T03:45:31.282 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5037999999999997e-05 2024-09-11T03:45:31.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.283 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.283 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905920+0000", 2024-09-11T03:45:31.283 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017050600000000001 2024-09-11T03:45:31.283 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.283 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.283 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.283 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.283 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2255 2.18 2:1984a3a9:::smithi17557870-23:head [write 2077646~576443] snapc 5d=[5d,5a,59,58,56] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:45:31.283 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.884463+0000", 2024-09-11T03:45:31.283 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 234.352782647, 2024-09-11T03:45:31.283 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022426167, 2024-09-11T03:45:31.283 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.284 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.284 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:31.284 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.284 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.284 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.284 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2255 2024-09-11T03:45:31.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.284 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.284 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.284 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884463+0000", 2024-09-11T03:45:31.284 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.285 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.285 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884463+0000", 2024-09-11T03:45:31.285 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.285 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.285 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884464+0000", 2024-09-11T03:45:31.285 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.64e-07 2024-09-11T03:45:31.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.285 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.285 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885204+0000", 2024-09-11T03:45:31.285 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00074007200000000004 2024-09-11T03:45:31.286 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.286 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.286 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885217+0000", 2024-09-11T03:45:31.286 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2547e-05 2024-09-11T03:45:31.286 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.286 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.286 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885226+0000", 2024-09-11T03:45:31.286 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1849999999999993e-06 2024-09-11T03:45:31.286 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.286 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.286 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885295+0000", 2024-09-11T03:45:31.287 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8657000000000004e-05 2024-09-11T03:45:31.287 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.287 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.287 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:45:31.287 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885307+0000", 2024-09-11T03:45:31.287 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2529e-05 2024-09-11T03:45:31.287 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.287 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.287 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.287 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896570+0000", 2024-09-11T03:45:31.287 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011262529 2024-09-11T03:45:31.287 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.288 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.288 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:45:31.288 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896624+0000", 2024-09-11T03:45:31.288 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4614000000000003e-05 2024-09-11T03:45:31.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.288 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.288 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.288 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.900547+0000", 2024-09-11T03:45:31.288 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039228850000000001 2024-09-11T03:45:31.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.288 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.288 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.288 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.900706+0000", 2024-09-11T03:45:31.289 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015907000000000001 2024-09-11T03:45:31.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.289 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:45:31.289 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.900901+0000", 2024-09-11T03:45:31.289 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00019514399999999999 2024-09-11T03:45:31.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.289 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.289 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903192+0000", 2024-09-11T03:45:31.289 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022906519999999998 2024-09-11T03:45:31.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.289 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.290 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.904724+0000", 2024-09-11T03:45:31.290 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015319520000000001 2024-09-11T03:45:31.290 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.290 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.290 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:31.290 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906254+0000", 2024-09-11T03:45:31.290 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015302759999999999 2024-09-11T03:45:31.290 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.290 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.290 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:31.290 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906358+0000", 2024-09-11T03:45:31.290 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010404300000000001 2024-09-11T03:45:31.290 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.291 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.291 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.291 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906889+0000", 2024-09-11T03:45:31.291 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000531048 2024-09-11T03:45:31.291 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.291 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.291 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.291 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.291 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.291 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2256 2.18 2:1984a3a9:::smithi17557870-23:head [write 3362764~183546] snapc 5d=[5d,5a,59,58,56] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:45:31.291 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.885408+0000", 2024-09-11T03:45:31.291 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 234.35183731199999, 2024-09-11T03:45:31.291 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021926139000000001, 2024-09-11T03:45:31.292 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.292 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.292 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:31.292 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.292 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.292 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.292 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2256 2024-09-11T03:45:31.292 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.293 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.293 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.293 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.293 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885408+0000", 2024-09-11T03:45:31.293 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.293 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.293 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.293 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.293 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885408+0000", 2024-09-11T03:45:31.293 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.293 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.293 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.293 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.293 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885410+0000", 2024-09-11T03:45:31.294 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4270000000000001e-06 2024-09-11T03:45:31.294 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.294 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.294 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.294 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885561+0000", 2024-09-11T03:45:31.294 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000151254 2024-09-11T03:45:31.294 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.294 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.294 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.294 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885573+0000", 2024-09-11T03:45:31.294 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1897e-05 2024-09-11T03:45:31.294 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.294 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.294 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.295 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885581+0000", 2024-09-11T03:45:31.295 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6290000000000001e-06 2024-09-11T03:45:31.295 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.295 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.295 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.295 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885643+0000", 2024-09-11T03:45:31.295 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2215e-05 2024-09-11T03:45:31.295 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.295 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.295 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:45:31.296 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885652+0000", 2024-09-11T03:45:31.296 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5580000000000006e-06 2024-09-11T03:45:31.296 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.296 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.296 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.296 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896678+0000", 2024-09-11T03:45:31.296 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011025956 2024-09-11T03:45:31.296 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.296 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.296 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:45:31.296 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896729+0000", 2024-09-11T03:45:31.296 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1019000000000001e-05 2024-09-11T03:45:31.296 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.296 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.297 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.297 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903250+0000", 2024-09-11T03:45:31.297 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0065205109999999997 2024-09-11T03:45:31.297 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.297 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.297 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.297 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903415+0000", 2024-09-11T03:45:31.297 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016545399999999999 2024-09-11T03:45:31.297 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.297 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.297 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:45:31.297 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903701+0000", 2024-09-11T03:45:31.297 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00028586399999999999 2024-09-11T03:45:31.297 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.298 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.298 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.298 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905681+0000", 2024-09-11T03:45:31.298 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0019798950000000002 2024-09-11T03:45:31.298 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.298 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.298 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.298 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905999+0000", 2024-09-11T03:45:31.298 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00031789999999999998 2024-09-11T03:45:31.298 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.298 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.298 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:31.298 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906904+0000", 2024-09-11T03:45:31.298 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00090535599999999996 2024-09-11T03:45:31.299 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.299 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.299 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:31.299 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907002+0000", 2024-09-11T03:45:31.299 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7653999999999997e-05 2024-09-11T03:45:31.299 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.299 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.299 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.299 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907335+0000", 2024-09-11T03:45:31.299 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00033254999999999998 2024-09-11T03:45:31.299 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.300 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.300 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.300 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.300 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.300 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2257 2.18 2:1984a3a9:::smithi17557870-23:head [setxattr _header (57) in=7b,truncate 3546310] snapc 5d=[5d,5a,59,58,56] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:45:31.300 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.885667+0000", 2024-09-11T03:45:31.300 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 234.35157829100001, 2024-09-11T03:45:31.300 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022144878999999999, 2024-09-11T03:45:31.300 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.300 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.300 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:31.300 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.300 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.301 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.301 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2257 2024-09-11T03:45:31.301 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.301 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.301 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.301 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.301 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885667+0000", 2024-09-11T03:45:31.301 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.301 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.301 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.301 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.301 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885667+0000", 2024-09-11T03:45:31.301 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.301 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.302 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.302 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.302 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885668+0000", 2024-09-11T03:45:31.302 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5600000000000005e-07 2024-09-11T03:45:31.302 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.302 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.302 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.302 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885673+0000", 2024-09-11T03:45:31.302 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4970000000000003e-06 2024-09-11T03:45:31.302 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.302 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.302 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.302 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885680+0000", 2024-09-11T03:45:31.302 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3780000000000003e-06 2024-09-11T03:45:31.303 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.303 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.303 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.303 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885686+0000", 2024-09-11T03:45:31.303 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0789999999999997e-06 2024-09-11T03:45:31.303 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.303 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.303 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.303 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886080+0000", 2024-09-11T03:45:31.303 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00039392399999999998 2024-09-11T03:45:31.303 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.303 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.303 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:45:31.304 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886091+0000", 2024-09-11T03:45:31.304 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0472000000000001e-05 2024-09-11T03:45:31.304 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.304 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.304 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.304 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896776+0000", 2024-09-11T03:45:31.304 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010684953000000001 2024-09-11T03:45:31.304 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.304 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.304 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:45:31.304 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896839+0000", 2024-09-11T03:45:31.304 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3293999999999994e-05 2024-09-11T03:45:31.304 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.305 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.305 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.305 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.904784+0000", 2024-09-11T03:45:31.305 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.007944843 2024-09-11T03:45:31.305 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.305 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.305 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.305 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.904909+0000", 2024-09-11T03:45:31.305 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012491600000000001 2024-09-11T03:45:31.305 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.305 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.305 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:45:31.305 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905105+0000", 2024-09-11T03:45:31.306 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000196336 2024-09-11T03:45:31.306 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.306 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.306 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.306 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906057+0000", 2024-09-11T03:45:31.306 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00095239800000000002 2024-09-11T03:45:31.306 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.306 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.306 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:31.306 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907348+0000", 2024-09-11T03:45:31.306 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012904489999999999 2024-09-11T03:45:31.306 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.306 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.306 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.307 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907457+0000", 2024-09-11T03:45:31.307 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108869 2024-09-11T03:45:31.307 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.307 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.307 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:31.307 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907543+0000", 2024-09-11T03:45:31.307 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6212999999999996e-05 2024-09-11T03:45:31.307 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.307 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.307 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.307 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907812+0000", 2024-09-11T03:45:31.307 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00026950200000000002 2024-09-11T03:45:31.307 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.307 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.308 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.308 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.308 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.308 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2258 2.18 2:1984a3a9:::smithi17557870-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:45:31.308 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.886095+0000", 2024-09-11T03:45:31.308 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 234.351150526, 2024-09-11T03:45:31.308 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022147336, 2024-09-11T03:45:31.308 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.308 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.308 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:45:31.308 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.308 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.308 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.309 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2258 2024-09-11T03:45:31.309 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.309 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.309 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.309 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.309 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886095+0000", 2024-09-11T03:45:31.309 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.309 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.309 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.309 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.309 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886095+0000", 2024-09-11T03:45:31.309 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.310 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.310 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886096+0000", 2024-09-11T03:45:31.310 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0200000000000001e-07 2024-09-11T03:45:31.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.310 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.310 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886100+0000", 2024-09-11T03:45:31.310 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9859999999999996e-06 2024-09-11T03:45:31.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.310 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.311 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886109+0000", 2024-09-11T03:45:31.311 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6349999999999995e-06 2024-09-11T03:45:31.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.311 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.311 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886116+0000", 2024-09-11T03:45:31.311 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0890000000000002e-06 2024-09-11T03:45:31.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.311 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.311 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886389+0000", 2024-09-11T03:45:31.311 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00027359200000000002 2024-09-11T03:45:31.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.312 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.312 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:45:31.312 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886399+0000", 2024-09-11T03:45:31.312 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9599999999999995e-06 2024-09-11T03:45:31.312 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.312 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.312 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.312 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896922+0000", 2024-09-11T03:45:31.312 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010522567 2024-09-11T03:45:31.312 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.312 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.312 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:45:31.312 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896977+0000", 2024-09-11T03:45:31.313 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5003999999999997e-05 2024-09-11T03:45:31.313 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.313 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.313 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.313 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905540+0000", 2024-09-11T03:45:31.313 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0085636329999999993 2024-09-11T03:45:31.313 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.313 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.313 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:45:31.313 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:31.315 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905625+0000", 2024-09-11T03:45:31.316 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4468999999999994e-05 2024-09-11T03:45:31.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.316 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.316 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907869+0000", 2024-09-11T03:45:31.316 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022438290000000001 2024-09-11T03:45:31.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.316 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.316 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.908014+0000", 2024-09-11T03:45:31.316 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00014511799999999999 2024-09-11T03:45:31.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.317 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.317 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.317 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.908243+0000", 2024-09-11T03:45:31.317 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00022865200000000001 2024-09-11T03:45:31.317 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.317 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.317 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.317 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.317 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2421 2.18 2:1bdf1a36:::smithi17557870-36:head [copy-from ver 157] snapc 65=[65,64,63,61,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-11T03:45:31.317 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:50.954849+0000", 2024-09-11T03:45:31.318 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 220.28239649599999, 2024-09-11T03:45:31.318 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022711387999999999, 2024-09-11T03:45:31.318 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.318 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.318 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:31.318 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.318 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.318 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.318 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2421 2024-09-11T03:45:31.318 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.318 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.318 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.318 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.318 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954849+0000", 2024-09-11T03:45:31.319 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.319 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.319 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.319 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.319 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954849+0000", 2024-09-11T03:45:31.319 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.319 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.319 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.319 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.319 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954851+0000", 2024-09-11T03:45:31.319 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3090000000000001e-06 2024-09-11T03:45:31.319 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.319 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.320 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.320 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954856+0000", 2024-09-11T03:45:31.320 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9059999999999999e-06 2024-09-11T03:45:31.320 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.320 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.320 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.320 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954867+0000", 2024-09-11T03:45:31.320 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0879999999999999e-05 2024-09-11T03:45:31.320 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.320 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.320 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.320 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954873+0000", 2024-09-11T03:45:31.320 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.203e-06 2024-09-11T03:45:31.321 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.321 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.321 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.321 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954921+0000", 2024-09-11T03:45:31.321 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8180000000000003e-05 2024-09-11T03:45:31.321 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.321 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.321 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.321 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.955004+0000", 2024-09-11T03:45:31.321 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2824000000000005e-05 2024-09-11T03:45:31.321 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.321 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.321 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:45:31.322 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964099+0000", 2024-09-11T03:45:31.322 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0090952499999999992 2024-09-11T03:45:31.322 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.322 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.322 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.322 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.970182+0000", 2024-09-11T03:45:31.322 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0060831510000000002 2024-09-11T03:45:31.322 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.322 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.322 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:31.322 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977039+0000", 2024-09-11T03:45:31.322 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068571980000000001 2024-09-11T03:45:31.322 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.322 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.323 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.323 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977432+0000", 2024-09-11T03:45:31.323 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00039304900000000002 2024-09-11T03:45:31.323 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.323 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.323 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:31.323 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977504+0000", 2024-09-11T03:45:31.323 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1863e-05 2024-09-11T03:45:31.323 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.323 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.323 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.323 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977561+0000", 2024-09-11T03:45:31.324 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6575e-05 2024-09-11T03:45:31.324 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.324 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.324 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.324 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.324 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2422 2.18 2:1bdf1a36:::smithi17557870-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e294)", 2024-09-11T03:45:31.324 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:50.954885+0000", 2024-09-11T03:45:31.324 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 220.28236045899999, 2024-09-11T03:45:31.325 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012999881, 2024-09-11T03:45:31.325 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.325 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.325 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:45:31.325 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.325 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.325 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.325 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2422 2024-09-11T03:45:31.326 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.326 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.326 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.326 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.326 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954885+0000", 2024-09-11T03:45:31.326 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.326 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.326 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.326 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.326 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954885+0000", 2024-09-11T03:45:31.326 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.326 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.326 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.326 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.327 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954886+0000", 2024-09-11T03:45:31.327 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.82e-07 2024-09-11T03:45:31.327 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.327 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.327 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.327 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954888+0000", 2024-09-11T03:45:31.327 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1909999999999999e-06 2024-09-11T03:45:31.327 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.327 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.327 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.327 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954899+0000", 2024-09-11T03:45:31.327 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0963e-05 2024-09-11T03:45:31.327 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.327 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.328 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.328 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954904+0000", 2024-09-11T03:45:31.328 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3539999999999999e-06 2024-09-11T03:45:31.328 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.328 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.328 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.328 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.955109+0000", 2024-09-11T03:45:31.328 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000204494 2024-09-11T03:45:31.328 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.329 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.329 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:31.329 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.955158+0000", 2024-09-11T03:45:31.329 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9502000000000001e-05 2024-09-11T03:45:31.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.329 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.329 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.329 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.967733+0000", 2024-09-11T03:45:31.329 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012574571999999999 2024-09-11T03:45:31.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.329 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.329 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.329 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.967803+0000", 2024-09-11T03:45:31.330 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0160999999999995e-05 2024-09-11T03:45:31.330 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.330 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.330 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.330 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.967885+0000", 2024-09-11T03:45:31.330 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2162000000000004e-05 2024-09-11T03:45:31.330 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.330 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.330 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.330 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.330 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.330 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2511 2.d 2:b6a25a79:::smithi17557870-47:head [copy-from ver 327] snapc 6c=[6c,6b,6a,69,68,65,63,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-09-11T03:45:31.331 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:01.867547+0000", 2024-09-11T03:45:31.331 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 209.36969850899999, 2024-09-11T03:45:31.331 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017562828999999999, 2024-09-11T03:45:31.331 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.331 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.331 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:31.331 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.331 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.331 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.331 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2511 2024-09-11T03:45:31.331 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.331 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.331 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.331 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.332 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867547+0000", 2024-09-11T03:45:31.332 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.332 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.332 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.332 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.332 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867547+0000", 2024-09-11T03:45:31.332 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.332 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.332 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.332 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.332 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867549+0000", 2024-09-11T03:45:31.332 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4100000000000001e-06 2024-09-11T03:45:31.333 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.333 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.333 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.333 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867554+0000", 2024-09-11T03:45:31.333 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.463e-06 2024-09-11T03:45:31.333 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.333 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.333 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.333 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867565+0000", 2024-09-11T03:45:31.333 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0531e-05 2024-09-11T03:45:31.333 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.333 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.333 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.334 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867574+0000", 2024-09-11T03:45:31.334 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7850000000000003e-06 2024-09-11T03:45:31.334 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.334 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.334 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.334 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867625+0000", 2024-09-11T03:45:31.334 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0958999999999998e-05 2024-09-11T03:45:31.334 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.334 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.334 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:45:31.334 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867632+0000", 2024-09-11T03:45:31.334 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4150000000000004e-06 2024-09-11T03:45:31.334 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.335 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.335 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.335 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.870960+0000", 2024-09-11T03:45:31.335 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0033279199999999998 2024-09-11T03:45:31.335 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.335 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.335 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.335 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.871059+0000", 2024-09-11T03:45:31.335 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9423e-05 2024-09-11T03:45:31.335 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.335 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.335 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:45:31.335 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.878007+0000", 2024-09-11T03:45:31.336 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.006947796 2024-09-11T03:45:31.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.336 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.336 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:31.336 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.881858+0000", 2024-09-11T03:45:31.336 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003850647 2024-09-11T03:45:31.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.336 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.336 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.336 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.884846+0000", 2024-09-11T03:45:31.336 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0029880309999999999 2024-09-11T03:45:31.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.336 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.337 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.337 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.885043+0000", 2024-09-11T03:45:31.337 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00019722899999999999 2024-09-11T03:45:31.337 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.337 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.337 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:31.337 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.885080+0000", 2024-09-11T03:45:31.337 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7106e-05 2024-09-11T03:45:31.337 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.337 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.337 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.337 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.885110+0000", 2024-09-11T03:45:31.337 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0114e-05 2024-09-11T03:45:31.337 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.338 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.338 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.338 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.338 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.338 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2512 2.d 2:b6a25a79:::smithi17557870-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e305)", 2024-09-11T03:45:31.338 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:01.867594+0000", 2024-09-11T03:45:31.338 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 209.36965223999999, 2024-09-11T03:45:31.338 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013722218, 2024-09-11T03:45:31.338 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.338 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.338 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:45:31.338 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.338 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.339 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.339 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2512 2024-09-11T03:45:31.339 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.339 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.339 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.339 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.339 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867594+0000", 2024-09-11T03:45:31.339 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.339 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.339 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.339 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.339 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867594+0000", 2024-09-11T03:45:31.339 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867594+0000", 2024-09-11T03:45:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9100000000000004e-07 2024-09-11T03:45:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867597+0000", 2024-09-11T03:45:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6369999999999999e-06 2024-09-11T03:45:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867604+0000", 2024-09-11T03:45:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5610000000000001e-06 2024-09-11T03:45:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867610+0000", 2024-09-11T03:45:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1249999999999998e-06 2024-09-11T03:45:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867674+0000", 2024-09-11T03:45:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3969999999999999e-05 2024-09-11T03:45:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867679+0000", 2024-09-11T03:45:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5929999999999999e-06 2024-09-11T03:45:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.871237+0000", 2024-09-11T03:45:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035579689999999998 2024-09-11T03:45:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.871280+0000", 2024-09-11T03:45:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2917000000000003e-05 2024-09-11T03:45:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.881114+0000", 2024-09-11T03:45:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0098343660000000006 2024-09-11T03:45:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.881210+0000", 2024-09-11T03:45:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5730000000000007e-05 2024-09-11T03:45:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.881316+0000", 2024-09-11T03:45:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000105759 2024-09-11T03:45:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2568 2.d 2:b6a25a79:::smithi17557870-47:head [copy-from ver 157] snapc 70=[70,6f,6e,6c,68,65,63,5d] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-11T03:45:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:10.876872+0000", 2024-09-11T03:45:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 200.36037337100001, 2024-09-11T03:45:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018302771999999998, 2024-09-11T03:45:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2568 2024-09-11T03:45:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876872+0000", 2024-09-11T03:45:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876872+0000", 2024-09-11T03:45:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876874+0000", 2024-09-11T03:45:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5570000000000001e-06 2024-09-11T03:45:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876880+0000", 2024-09-11T03:45:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8799999999999996e-06 2024-09-11T03:45:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.347 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876901+0000", 2024-09-11T03:45:31.347 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0942999999999999e-05 2024-09-11T03:45:31.347 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.347 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.347 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.347 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876909+0000", 2024-09-11T03:45:31.347 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3410000000000003e-06 2024-09-11T03:45:31.347 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.347 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.347 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.347 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876966+0000", 2024-09-11T03:45:31.347 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7179999999999998e-05 2024-09-11T03:45:31.347 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.348 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.348 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.348 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.877099+0000", 2024-09-11T03:45:31.348 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013234499999999999 2024-09-11T03:45:31.348 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.348 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.348 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:45:31.348 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.885734+0000", 2024-09-11T03:45:31.348 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0086350730000000001 2024-09-11T03:45:31.348 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.348 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.348 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:31.348 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.890016+0000", 2024-09-11T03:45:31.349 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.004282213 2024-09-11T03:45:31.349 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.349 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.349 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.349 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.894661+0000", 2024-09-11T03:45:31.349 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0046455059999999998 2024-09-11T03:45:31.349 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.349 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.349 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.349 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.895046+0000", 2024-09-11T03:45:31.349 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00038434999999999999 2024-09-11T03:45:31.349 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.349 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.349 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:31.350 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.895116+0000", 2024-09-11T03:45:31.350 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0536000000000004e-05 2024-09-11T03:45:31.350 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.350 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.350 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.350 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.895175+0000", 2024-09-11T03:45:31.350 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8848000000000003e-05 2024-09-11T03:45:31.350 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.350 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.350 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.350 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.350 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.350 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2569 2.d 2:b6a25a79:::smithi17557870-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e314)", 2024-09-11T03:45:31.350 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:10.876927+0000", 2024-09-11T03:45:31.351 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 200.36031857200001, 2024-09-11T03:45:31.351 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012434861, 2024-09-11T03:45:31.351 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.351 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.351 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:45:31.351 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.351 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.351 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.351 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2569 2024-09-11T03:45:31.351 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.351 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.351 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.351 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.352 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876927+0000", 2024-09-11T03:45:31.352 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.352 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.352 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.352 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.352 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876927+0000", 2024-09-11T03:45:31.352 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.352 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.352 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.352 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.352 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876928+0000", 2024-09-11T03:45:31.352 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9200000000000001e-07 2024-09-11T03:45:31.352 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.353 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.353 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.353 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876931+0000", 2024-09-11T03:45:31.353 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.881e-06 2024-09-11T03:45:31.353 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.353 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.353 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.353 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876938+0000", 2024-09-11T03:45:31.353 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7090000000000002e-06 2024-09-11T03:45:31.353 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.353 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.353 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.353 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876944+0000", 2024-09-11T03:45:31.354 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7599999999999999e-06 2024-09-11T03:45:31.354 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.354 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.354 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.354 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.877267+0000", 2024-09-11T03:45:31.354 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00032272300000000002 2024-09-11T03:45:31.354 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.354 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.354 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:31.354 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.877312+0000", 2024-09-11T03:45:31.354 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4981999999999997e-05 2024-09-11T03:45:31.354 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.354 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.354 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.355 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.889216+0000", 2024-09-11T03:45:31.355 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011904246 2024-09-11T03:45:31.355 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.355 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.355 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.355 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.889286+0000", 2024-09-11T03:45:31.355 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9628000000000005e-05 2024-09-11T03:45:31.355 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.355 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.355 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.355 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.889362+0000", 2024-09-11T03:45:31.356 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6340000000000004e-05 2024-09-11T03:45:31.356 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.356 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.356 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.356 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.356 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.356 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4687 2.18 2:1984a3a9:::smithi17557870-23:head [write 739673~657763,stat] snapc cd=[cd,cc,ca,c5,c2,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-11T03:45:31.356 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.003513+0000", 2024-09-11T03:45:31.356 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 10.233733188, 2024-09-11T03:45:31.356 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.028701775999999998, 2024-09-11T03:45:31.356 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.356 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.356 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:31.356 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.357 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.357 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.357 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4687 2024-09-11T03:45:31.357 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.357 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.357 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.357 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.357 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003513+0000", 2024-09-11T03:45:31.357 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.357 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.357 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.357 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.357 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003513+0000", 2024-09-11T03:45:31.358 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.358 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.358 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003515+0000", 2024-09-11T03:45:31.358 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5600000000000001e-06 2024-09-11T03:45:31.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.358 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.358 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003862+0000", 2024-09-11T03:45:31.358 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00034641300000000001 2024-09-11T03:45:31.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.358 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.359 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003880+0000", 2024-09-11T03:45:31.359 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8669e-05 2024-09-11T03:45:31.359 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.359 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.359 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.359 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003906+0000", 2024-09-11T03:45:31.359 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5432999999999999e-05 2024-09-11T03:45:31.359 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.359 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.359 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.359 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003972+0000", 2024-09-11T03:45:31.359 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6729000000000004e-05 2024-09-11T03:45:31.359 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.359 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.360 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:45:31.360 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003982+0000", 2024-09-11T03:45:31.360 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2979999999999997e-06 2024-09-11T03:45:31.360 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.360 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.360 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.360 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.005521+0000", 2024-09-11T03:45:31.360 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015392959999999999 2024-09-11T03:45:31.360 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.360 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.360 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.360 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.005611+0000", 2024-09-11T03:45:31.361 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.0373999999999999e-05 2024-09-11T03:45:31.361 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.361 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.361 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,7", 2024-09-11T03:45:31.361 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.005729+0000", 2024-09-11T03:45:31.361 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000117834 2024-09-11T03:45:31.361 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.361 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.361 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:31.361 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.007175+0000", 2024-09-11T03:45:31.361 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014461140000000001 2024-09-11T03:45:31.361 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.361 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.362 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.362 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.008417+0000", 2024-09-11T03:45:31.362 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012421120000000001 2024-09-11T03:45:31.362 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.362 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.362 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.362 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032091+0000", 2024-09-11T03:45:31.362 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.023674066000000001 2024-09-11T03:45:31.362 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.362 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.362 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:31.362 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032146+0000", 2024-09-11T03:45:31.363 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4786e-05 2024-09-11T03:45:31.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.363 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.363 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.363 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032214+0000", 2024-09-11T03:45:31.363 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8091999999999998e-05 2024-09-11T03:45:31.363 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.363 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.363 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.363 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.363 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4688 2.18 2:1984a3a9:::smithi17557870-23:head [write 2076229~663444] snapc cd=[cd,cc,ca,c5,c2,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-11T03:45:31.363 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.010935+0000", 2024-09-11T03:45:31.364 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 10.226311128000001, 2024-09-11T03:45:31.364 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021404982999999999, 2024-09-11T03:45:31.364 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.364 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.364 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:31.364 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.364 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.364 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.364 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4688 2024-09-11T03:45:31.364 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.364 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.364 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.364 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.365 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.010935+0000", 2024-09-11T03:45:31.365 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.365 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.365 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.365 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.365 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.010935+0000", 2024-09-11T03:45:31.365 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.365 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.365 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.365 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.365 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.010936+0000", 2024-09-11T03:45:31.365 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1689999999999999e-06 2024-09-11T03:45:31.365 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.365 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.366 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.366 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011065+0000", 2024-09-11T03:45:31.366 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000128997 2024-09-11T03:45:31.366 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.366 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.366 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.366 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011076+0000", 2024-09-11T03:45:31.366 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1561e-05 2024-09-11T03:45:31.366 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.366 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.366 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.366 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011082+0000", 2024-09-11T03:45:31.366 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2599999999999996e-06 2024-09-11T03:45:31.367 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.367 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.367 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.367 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011127+0000", 2024-09-11T03:45:31.367 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5509000000000001e-05 2024-09-11T03:45:31.367 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.367 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.367 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.367 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011212+0000", 2024-09-11T03:45:31.367 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5173999999999995e-05 2024-09-11T03:45:31.367 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.367 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.367 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,7", 2024-09-11T03:45:31.367 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011309+0000", 2024-09-11T03:45:31.368 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6980999999999999e-05 2024-09-11T03:45:31.368 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.368 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.368 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:31.368 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012804+0000", 2024-09-11T03:45:31.368 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001495019 2024-09-11T03:45:31.368 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.368 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.368 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.368 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.031586+0000", 2024-09-11T03:45:31.368 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018782099 2024-09-11T03:45:31.368 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.368 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.369 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.369 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032261+0000", 2024-09-11T03:45:31.369 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000674846 2024-09-11T03:45:31.369 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.369 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.369 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:31.369 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032297+0000", 2024-09-11T03:45:31.369 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6010000000000003e-05 2024-09-11T03:45:31.369 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.369 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.369 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.369 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032340+0000", 2024-09-11T03:45:31.369 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2357999999999997e-05 2024-09-11T03:45:31.369 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.370 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.370 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.370 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.370 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.370 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4689 2.18 2:1984a3a9:::smithi17557870-23:head [setxattr _header (58) in=7b,truncate 2739673] snapc cd=[cd,cc,ca,c5,c2,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-11T03:45:31.370 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.011093+0000", 2024-09-11T03:45:31.370 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 10.226152652, 2024-09-11T03:45:31.370 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.02140688, 2024-09-11T03:45:31.370 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.370 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.370 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:31.370 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.370 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.370 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.371 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4689 2024-09-11T03:45:31.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.371 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.371 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.371 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.371 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011093+0000", 2024-09-11T03:45:31.371 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.371 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.371 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.371 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011093+0000", 2024-09-11T03:45:31.371 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.372 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.372 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.372 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011094+0000", 2024-09-11T03:45:31.372 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9999999999999997e-07 2024-09-11T03:45:31.372 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.372 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.372 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.372 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011100+0000", 2024-09-11T03:45:31.372 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1369999999999999e-06 2024-09-11T03:45:31.372 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.372 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.372 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.372 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011106+0000", 2024-09-11T03:45:31.373 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.421e-06 2024-09-11T03:45:31.373 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.373 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.373 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.373 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011117+0000", 2024-09-11T03:45:31.373 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1012999999999999e-05 2024-09-11T03:45:31.373 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.373 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.373 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.373 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012151+0000", 2024-09-11T03:45:31.373 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0010333880000000001 2024-09-11T03:45:31.373 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.373 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.374 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.374 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012265+0000", 2024-09-11T03:45:31.374 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114141 2024-09-11T03:45:31.374 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.374 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.374 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,7", 2024-09-11T03:45:31.374 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012418+0000", 2024-09-11T03:45:31.374 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015286499999999999 2024-09-11T03:45:31.374 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.374 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.374 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:31.374 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012807+0000", 2024-09-11T03:45:31.374 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00038979800000000001 2024-09-11T03:45:31.375 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.375 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.375 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.375 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.031717+0000", 2024-09-11T03:45:31.375 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018909888 2024-09-11T03:45:31.375 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.375 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.375 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:31.375 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032383+0000", 2024-09-11T03:45:31.375 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00066515900000000002 2024-09-11T03:45:31.375 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.375 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.375 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:31.376 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032447+0000", 2024-09-11T03:45:31.376 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4958999999999999e-05 2024-09-11T03:45:31.376 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.376 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.376 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.376 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032500+0000", 2024-09-11T03:45:31.376 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2510999999999999e-05 2024-09-11T03:45:31.376 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.376 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.376 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.376 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.376 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.376 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4690 2.18 2:1984a3a9:::smithi17557870-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e503)", 2024-09-11T03:45:31.377 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.011126+0000", 2024-09-11T03:45:31.377 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 10.226119424, 2024-09-11T03:45:31.377 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021513529, 2024-09-11T03:45:31.377 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:31.377 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:31.377 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:45:31.377 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:31.377 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:31.377 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:31.377 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4690 2024-09-11T03:45:31.377 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.377 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:31.377 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.377 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:31.378 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011126+0000", 2024-09-11T03:45:31.378 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.378 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.378 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.378 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:31.378 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011126+0000", 2024-09-11T03:45:31.378 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:31.378 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.378 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.379 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:31.379 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011127+0000", 2024-09-11T03:45:31.379 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8800000000000003e-07 2024-09-11T03:45:31.379 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.379 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.379 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:31.379 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011129+0000", 2024-09-11T03:45:31.379 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0090000000000002e-06 2024-09-11T03:45:31.379 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.379 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.380 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:31.380 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011146+0000", 2024-09-11T03:45:31.380 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7209000000000001e-05 2024-09-11T03:45:31.380 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.380 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.380 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:31.380 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011152+0000", 2024-09-11T03:45:31.380 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5160000000000002e-06 2024-09-11T03:45:31.380 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.380 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.380 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.381 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012718+0000", 2024-09-11T03:45:31.381 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001566413 2024-09-11T03:45:31.381 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.381 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.381 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:45:31.381 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012793+0000", 2024-09-11T03:45:31.381 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5197999999999994e-05 2024-09-11T03:45:31.381 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.381 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.381 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:31.381 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032530+0000", 2024-09-11T03:45:31.382 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019736848000000001 2024-09-11T03:45:31.382 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.382 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.382 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:31.382 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032577+0000", 2024-09-11T03:45:31.382 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6878000000000001e-05 2024-09-11T03:45:31.382 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:31.382 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:31.382 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:31.382 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032640+0000", 2024-09-11T03:45:31.382 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2970000000000002e-05 2024-09-11T03:45:31.382 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.383 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.383 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.383 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:31.383 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:31.383 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:45:31.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:30 smithi084 ceph-mon[27254]: osdmap e512: 12 total, 12 up, 11 in 2024-09-11T03:45:31.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:30 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4035001911' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:45:31.404 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:31.405 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:31.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:30 smithi121 ceph-mon[30729]: osdmap e512: 12 total, 12 up, 11 in 2024-09-11T03:45:31.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:30 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4035001911' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:45:31.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:31.532 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:31.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:31.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:31.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:31.656 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:31.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:31.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:31.707 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-11T03:45:31.740 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:31.758 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:31.785 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:31.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:31.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:31.887 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:31.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:31.909 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:31.909 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:31.943 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:45:31.946 INFO:tasks.rados.rados.2.smithi175.stdout:1899: done (2 left) 2024-09-11T03:45:31.947 INFO:tasks.rados.rados.2.smithi175.stdout:1900: write oid 24 current snap is 208 2024-09-11T03:45:31.947 INFO:tasks.rados.rados.2.smithi175.stdout:1900: seq_num 651 ranges {541357=477104,1745307=594552} 2024-09-11T03:45:31.953 INFO:tasks.rados.rados.2.smithi175.stdout:1900: writing smithi17557870-24 from 541357 to 1018461 tid 1 2024-09-11T03:45:31.961 INFO:tasks.rados.rados.2.smithi175.stdout:1900: writing smithi17557870-24 from 1745307 to 2339859 tid 2 2024-09-11T03:45:31.961 INFO:tasks.rados.rados.2.smithi175.stdout:1901: read oid 29 snap -1 2024-09-11T03:45:31.961 INFO:tasks.rados.rados.2.smithi175.stdout:1901: expect deleted 2024-09-11T03:45:31.961 INFO:tasks.rados.rados.2.smithi175.stdout:1902: read oid 26 snap -1 2024-09-11T03:45:31.961 INFO:tasks.rados.rados.2.smithi175.stdout:1902: expect (ObjNum 595 snap 185 seq_num 595) 2024-09-11T03:45:31.961 INFO:tasks.rados.rados.2.smithi175.stdout:1903: snap_create 2024-09-11T03:45:31.961 INFO:tasks.rados.rados.2.smithi175.stdout:1896: finishing rollback tid 1 to smithi17557870-2 2024-09-11T03:45:31.961 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 703 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T03:45:31.961 INFO:tasks.rados.rados.2.smithi175.stdout:1900: finishing write tid 1 to smithi17557870-24 2024-09-11T03:45:31.967 INFO:tasks.rados.rados.2.smithi175.stdout:1900: finishing write tid 2 to smithi17557870-24 2024-09-11T03:45:31.967 INFO:tasks.rados.rados.2.smithi175.stdout:1900: finishing write tid 3 to smithi17557870-24 2024-09-11T03:45:31.967 INFO:tasks.rados.rados.2.smithi175.stdout:1900: finishing write tid 4 to smithi17557870-24 2024-09-11T03:45:31.967 INFO:tasks.rados.rados.2.smithi175.stdout:1900: oid 24 updating version 0 to 742 2024-09-11T03:45:31.967 INFO:tasks.rados.rados.2.smithi175.stdout:1900: oid 24 version 742 is already newer than 741 2024-09-11T03:45:31.967 INFO:tasks.rados.rados.2.smithi175.stdout:1900: oid 24 version 742 is already newer than 740 2024-09-11T03:45:31.967 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 742 (ObjNum 651 snap 208 seq_num 651) dirty exists 2024-09-11T03:45:31.967 INFO:tasks.rados.rados.2.smithi175.stdout:1900: left oid 24 (ObjNum 651 snap 208 seq_num 651) 2024-09-11T03:45:31.976 INFO:tasks.rados.rados.2.smithi175.stdout:1902: expect (ObjNum 595 snap 185 seq_num 595) 2024-09-11T03:45:32.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:32.031 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:32.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:32.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:32.133 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:32.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:32.155 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:32.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:32.183 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-11T03:45:32.245 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:31 smithi175 ceph-mon[30894]: 1.19 scrub starts 2024-09-11T03:45:32.245 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:31 smithi175 ceph-mon[30894]: 1.19 scrub ok 2024-09-11T03:45:32.245 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:31 smithi175 ceph-mon[30894]: pgmap v924: 57 pgs: 57 active+clean; 212 MiB data, 842 MiB used, 983 GiB / 983 GiB avail; 0 B/s rd, 566 KiB/s wr, 0 op/s 2024-09-11T03:45:32.245 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:31 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4035001911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:45:32.245 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:31 smithi175 ceph-mon[30894]: osdmap e513: 12 total, 12 up, 11 in 2024-09-11T03:45:32.246 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:31 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1743949823' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:45:32.246 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:31 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:32.246 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:31 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1743949823' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:45:32.246 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:31 smithi175 ceph-mon[30894]: osdmap e514: 12 total, 12 up, 11 in 2024-09-11T03:45:32.256 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:32.280 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:31 smithi084 ceph-mon[27254]: 1.19 scrub starts 2024-09-11T03:45:32.280 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:31 smithi084 ceph-mon[27254]: 1.19 scrub ok 2024-09-11T03:45:32.280 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:31 smithi084 ceph-mon[27254]: pgmap v924: 57 pgs: 57 active+clean; 212 MiB data, 842 MiB used, 983 GiB / 983 GiB avail; 0 B/s rd, 566 KiB/s wr, 0 op/s 2024-09-11T03:45:32.280 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:31 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4035001911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:45:32.280 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:31 smithi084 ceph-mon[27254]: osdmap e513: 12 total, 12 up, 11 in 2024-09-11T03:45:32.280 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:31 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1743949823' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:45:32.280 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:31 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:32.280 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:31 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1743949823' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:45:32.280 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:31 smithi084 ceph-mon[27254]: osdmap e514: 12 total, 12 up, 11 in 2024-09-11T03:45:32.282 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:32.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:32.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:32.384 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:32.405 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:32.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:32.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:31 smithi121 ceph-mon[30729]: 1.19 scrub starts 2024-09-11T03:45:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:31 smithi121 ceph-mon[30729]: 1.19 scrub ok 2024-09-11T03:45:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:31 smithi121 ceph-mon[30729]: pgmap v924: 57 pgs: 57 active+clean; 212 MiB data, 842 MiB used, 983 GiB / 983 GiB avail; 0 B/s rd, 566 KiB/s wr, 0 op/s 2024-09-11T03:45:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:31 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4035001911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:45:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:31 smithi121 ceph-mon[30729]: osdmap e513: 12 total, 12 up, 11 in 2024-09-11T03:45:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:31 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1743949823' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:45:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:31 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:31 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1743949823' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:45:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:31 smithi121 ceph-mon[30729]: osdmap e514: 12 total, 12 up, 11 in 2024-09-11T03:45:32.463 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:45:32.507 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:32.528 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:32.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:32.529 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:32.630 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:32.651 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:32.652 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:32.752 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:32.774 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:32.774 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:32.874 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:32.895 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:32.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:32.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:32.932 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:45:32.932 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:45:32.947 INFO:tasks.rados.rados.2.smithi175.stdout:1896: done (5 left) 2024-09-11T03:45:32.947 INFO:tasks.rados.rados.2.smithi175.stdout:1898: done (4 left) 2024-09-11T03:45:32.948 INFO:tasks.rados.rados.2.smithi175.stdout:1900: done (3 left) 2024-09-11T03:45:32.948 INFO:tasks.rados.rados.2.smithi175.stdout:1901: done (2 left) 2024-09-11T03:45:32.948 INFO:tasks.rados.rados.2.smithi175.stdout:1902: done (1 left) 2024-09-11T03:45:32.948 INFO:tasks.rados.rados.2.smithi175.stdout:1903: done (0 left) 2024-09-11T03:45:32.948 INFO:tasks.rados.rados.2.smithi175.stdout:1904: delete oid 38 current snap is 209 2024-09-11T03:45:32.949 INFO:tasks.rados.rados.2.smithi175.stdout:1904: done (0 left) 2024-09-11T03:45:32.949 INFO:tasks.rados.rados.2.smithi175.stdout:1905: write (excl) oid 46 current snap is 209 2024-09-11T03:45:32.949 INFO:tasks.rados.rados.2.smithi175.stdout:1905: seq_num 653 ranges {574971=627602,1862271=598221} 2024-09-11T03:45:32.949 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:45:32.950 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:45:32.956 INFO:tasks.rados.rados.2.smithi175.stdout:1905: writing smithi17557870-46 from 574971 to 1202573 tid 1 2024-09-11T03:45:32.963 INFO:tasks.rados.rados.2.smithi175.stdout:1905: writing smithi17557870-46 from 1862271 to 2460492 tid 2 2024-09-11T03:45:32.963 INFO:tasks.rados.rados.2.smithi175.stdout:1906: write oid 41 current snap is 209 2024-09-11T03:45:32.963 INFO:tasks.rados.rados.2.smithi175.stdout:1905: finishing write tid 1 to smithi17557870-46 2024-09-11T03:45:32.963 INFO:tasks.rados.rados.2.smithi175.stdout:1906: seq_num 654 ranges {591778=419204,1637106=741879,2910049=81729} 2024-09-11T03:45:32.966 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:45:32.967 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:45:32.968 INFO:tasks.rados.rados.2.smithi175.stdout:1906: writing smithi17557870-41 from 591778 to 1010982 tid 1 2024-09-11T03:45:32.975 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:45:32.975 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:45:32.975 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:45:32.976 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:45:32.976 INFO:tasks.rados.rados.2.smithi175.stdout:1906: writing smithi17557870-41 from 1637106 to 2378985 tid 2 2024-09-11T03:45:32.980 INFO:tasks.rados.rados.2.smithi175.stdout:1906: writing smithi17557870-41 from 2910049 to 2991778 tid 3 2024-09-11T03:45:32.980 INFO:tasks.rados.rados.2.smithi175.stdout:1907: copy_from oid 22 from oid 31 current snap is 209 2024-09-11T03:45:32.980 INFO:tasks.rados.rados.2.smithi175.stdout:1905: finishing write tid 2 to smithi17557870-46 2024-09-11T03:45:32.980 INFO:tasks.rados.rados.2.smithi175.stdout:1905: finishing write tid 3 to smithi17557870-46 2024-09-11T03:45:32.980 INFO:tasks.rados.rados.2.smithi175.stdout:1905: finishing write tid 4 to smithi17557870-46 2024-09-11T03:45:32.980 INFO:tasks.rados.rados.2.smithi175.stdout:1905: oid 46 updating version 0 to 534 2024-09-11T03:45:32.980 INFO:tasks.rados.rados.2.smithi175.stdout:1905: oid 46 version 534 is already newer than 533 2024-09-11T03:45:32.980 INFO:tasks.rados.rados.2.smithi175.stdout:1905: oid 46 version 534 is already newer than 532 2024-09-11T03:45:32.980 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 534 (ObjNum 653 snap 209 seq_num 653) dirty exists 2024-09-11T03:45:32.980 INFO:tasks.rados.rados.2.smithi175.stdout:1905: left oid 46 (ObjNum 653 snap 209 seq_num 653) 2024-09-11T03:45:32.980 INFO:tasks.rados.rados.2.smithi175.stdout:1906: finishing write tid 1 to smithi17557870-41 2024-09-11T03:45:32.981 INFO:tasks.rados.rados.2.smithi175.stdout:1906: finishing write tid 2 to smithi17557870-41 2024-09-11T03:45:32.981 INFO:tasks.rados.rados.2.smithi175.stdout:1905: done (2 left) 2024-09-11T03:45:32.981 INFO:tasks.rados.rados.2.smithi175.stdout:1908: snap_remove snap 189 2024-09-11T03:45:32.985 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:45:32.985 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:45:32.997 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:33.004 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:45:33.004 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:45:33.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:33.019 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:33.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:33.020 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:33.023 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:45:33.024 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:45:33.042 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:45:33.042 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:45:33.060 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:45:33.060 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:45:33.077 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:45:33.077 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:45:33.094 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:45:33.094 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:45:33.110 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:45:33.110 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:45:33.121 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:33.128 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:45:33.128 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:45:33.144 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:33.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:33.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:33.218 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:32 smithi084 ceph-mon[27254]: 2.12 scrub starts 2024-09-11T03:45:33.218 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:32 smithi084 ceph-mon[27254]: 2.12 scrub ok 2024-09-11T03:45:33.218 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:32 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2037734212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:33.218 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:32 smithi084 ceph-mon[27254]: osdmap e515: 12 total, 12 up, 11 in 2024-09-11T03:45:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:32 smithi175 ceph-mon[30894]: 2.12 scrub starts 2024-09-11T03:45:33.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:32 smithi175 ceph-mon[30894]: 2.12 scrub ok 2024-09-11T03:45:33.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:32 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2037734212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:33.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:32 smithi175 ceph-mon[30894]: osdmap e515: 12 total, 12 up, 11 in 2024-09-11T03:45:33.246 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:33.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:33.268 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:33.268 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:33.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:33.395 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:33.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:33.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:33.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:32 smithi121 ceph-mon[30729]: 2.12 scrub starts 2024-09-11T03:45:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:32 smithi121 ceph-mon[30729]: 2.12 scrub ok 2024-09-11T03:45:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:32 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2037734212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:32 smithi121 ceph-mon[30729]: osdmap e515: 12 total, 12 up, 11 in 2024-09-11T03:45:33.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:33.518 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:33.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:33.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:33.603 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-11T03:45:33.620 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:33.641 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:33.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:33.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:33.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:33.769 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:33.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:33.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:33.842 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:45:33.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:33.896 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:33.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:33.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:33.950 INFO:tasks.rados.rados.2.smithi175.stdout:1908: done (2 left) 2024-09-11T03:45:33.950 INFO:tasks.rados.rados.2.smithi175.stdout:1909: snap_create 2024-09-11T03:45:33.950 INFO:tasks.rados.rados.2.smithi175.stdout:1906: finishing write tid 3 to smithi17557870-41 2024-09-11T03:45:33.950 INFO:tasks.rados.rados.2.smithi175.stdout:1906: finishing write tid 4 to smithi17557870-41 2024-09-11T03:45:33.951 INFO:tasks.rados.rados.2.smithi175.stdout:1906: finishing write tid 5 to smithi17557870-41 2024-09-11T03:45:33.951 INFO:tasks.rados.rados.2.smithi175.stdout:1906: oid 41 updating version 0 to 568 2024-09-11T03:45:33.951 INFO:tasks.rados.rados.2.smithi175.stdout:1906: oid 41 version 568 is already newer than 567 2024-09-11T03:45:33.951 INFO:tasks.rados.rados.2.smithi175.stdout:1906: oid 41 updating version 568 to 569 2024-09-11T03:45:33.951 INFO:tasks.rados.rados.2.smithi175.stdout:1906: oid 41 version 569 is already newer than 566 2024-09-11T03:45:33.951 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 569 (ObjNum 654 snap 209 seq_num 654) dirty exists 2024-09-11T03:45:33.951 INFO:tasks.rados.rados.2.smithi175.stdout:1906: left oid 41 (ObjNum 654 snap 209 seq_num 654) 2024-09-11T03:45:33.951 INFO:tasks.rados.rados.2.smithi175.stdout:1907: finishing copy_from racing read to smithi17557870-22 2024-09-11T03:45:33.951 INFO:tasks.rados.rados.2.smithi175.stdout:1907: finishing copy_from to smithi17557870-22 2024-09-11T03:45:33.951 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 681 (ObjNum 495 snap 143 seq_num 495) dirty exists 2024-09-11T03:45:33.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:34.022 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:34.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:34.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:34.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:34.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:34.145 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:34.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:34.206 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:33 smithi084 ceph-mon[27254]: pgmap v927: 57 pgs: 57 active+clean; 170 MiB data, 828 MiB used, 983 GiB / 983 GiB avail; 2.8 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-11T03:45:34.206 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:33 smithi084 ceph-mon[27254]: 1.16 scrub starts 2024-09-11T03:45:34.206 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:33 smithi084 ceph-mon[27254]: osdmap e516: 12 total, 12 up, 11 in 2024-09-11T03:45:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:33 smithi175 ceph-mon[30894]: pgmap v927: 57 pgs: 57 active+clean; 170 MiB data, 828 MiB used, 983 GiB / 983 GiB avail; 2.8 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-11T03:45:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:33 smithi175 ceph-mon[30894]: 1.16 scrub starts 2024-09-11T03:45:34.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:33 smithi175 ceph-mon[30894]: osdmap e516: 12 total, 12 up, 11 in 2024-09-11T03:45:34.247 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:34.271 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:34.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:34.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:34.295 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:45:34.295 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:45:34.373 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:34.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:34.396 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:34.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:33 smithi121 ceph-mon[30729]: pgmap v927: 57 pgs: 57 active+clean; 170 MiB data, 828 MiB used, 983 GiB / 983 GiB avail; 2.8 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-11T03:45:34.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:33 smithi121 ceph-mon[30729]: 1.16 scrub starts 2024-09-11T03:45:34.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:33 smithi121 ceph-mon[30729]: osdmap e516: 12 total, 12 up, 11 in 2024-09-11T03:45:34.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:34.518 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:34.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:34.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:34.620 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:34.646 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:34.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:34.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:34.692 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:45:34.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:34.773 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:34.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:34.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:34.875 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:34.893 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-11T03:45:34.897 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:34.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:34.938 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:34.947 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T03:45:34.952 INFO:tasks.rados.rados.2.smithi175.stdout:1906: done (2 left) 2024-09-11T03:45:34.953 INFO:tasks.rados.rados.2.smithi175.stdout:1907: done (1 left) 2024-09-11T03:45:34.953 INFO:tasks.rados.rados.2.smithi175.stdout:1909: done (0 left) 2024-09-11T03:45:34.953 INFO:tasks.rados.rados.2.smithi175.stdout:1910: delete oid 29 current snap is 210 2024-09-11T03:45:34.956 INFO:tasks.rados.rados.2.smithi175.stdout:1910: done (0 left) 2024-09-11T03:45:34.956 INFO:tasks.rados.rados.2.smithi175.stdout:1911: write oid 3 current snap is 210 2024-09-11T03:45:34.956 INFO:tasks.rados.rados.2.smithi175.stdout:1911: seq_num 656 ranges {625392=569702,1637717=745548,2848542=176850} 2024-09-11T03:45:34.962 INFO:tasks.rados.rados.2.smithi175.stdout:1911: writing smithi17557870-3 from 625392 to 1195094 tid 1 2024-09-11T03:45:34.968 INFO:tasks.rados.rados.2.smithi175.stdout:1911: writing smithi17557870-3 from 1637717 to 2383265 tid 2 2024-09-11T03:45:34.972 INFO:tasks.rados.rados.2.smithi175.stdout:1911: writing smithi17557870-3 from 2848542 to 3025392 tid 3 2024-09-11T03:45:34.972 INFO:tasks.rados.rados.2.smithi175.stdout:1912: write (excl) oid 31 current snap is 210 2024-09-11T03:45:34.972 INFO:tasks.rados.rados.2.smithi175.stdout:1912: seq_num 657 ranges {642199=644951,1696199=605559,2992318=49881} 2024-09-11T03:45:34.978 INFO:tasks.rados.rados.2.smithi175.stdout:1912: writing smithi17557870-31 from 642199 to 1287150 tid 1 2024-09-11T03:45:34.984 INFO:tasks.rados.rados.2.smithi175.stdout:1912: writing smithi17557870-31 from 1696199 to 2301758 tid 2 2024-09-11T03:45:34.989 INFO:tasks.rados.rados.2.smithi175.stdout:1912: writing smithi17557870-31 from 2992318 to 3042199 tid 3 2024-09-11T03:45:34.989 INFO:tasks.rados.rados.2.smithi175.stdout:1913: copy_from oid 18 from oid 4 current snap is 210 2024-09-11T03:45:34.989 INFO:tasks.rados.rados.2.smithi175.stdout:1911: finishing write tid 1 to smithi17557870-3 2024-09-11T03:45:34.989 INFO:tasks.rados.rados.2.smithi175.stdout:1911: finishing write tid 2 to smithi17557870-3 2024-09-11T03:45:34.989 INFO:tasks.rados.rados.2.smithi175.stdout:1911: finishing write tid 3 to smithi17557870-3 2024-09-11T03:45:34.990 INFO:tasks.rados.rados.2.smithi175.stdout:1911: finishing write tid 4 to smithi17557870-3 2024-09-11T03:45:34.990 INFO:tasks.rados.rados.2.smithi175.stdout:1911: finishing write tid 5 to smithi17557870-3 2024-09-11T03:45:34.990 INFO:tasks.rados.rados.2.smithi175.stdout:1911: oid 3 updating version 0 to 752 2024-09-11T03:45:34.990 INFO:tasks.rados.rados.2.smithi175.stdout:1911: oid 3 version 752 is already newer than 751 2024-09-11T03:45:34.990 INFO:tasks.rados.rados.2.smithi175.stdout:1911: oid 3 updating version 752 to 754 2024-09-11T03:45:34.990 INFO:tasks.rados.rados.2.smithi175.stdout:1911: oid 3 version 754 is already newer than 753 2024-09-11T03:45:34.990 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 754 (ObjNum 656 snap 210 seq_num 656) dirty exists 2024-09-11T03:45:34.990 INFO:tasks.rados.rados.2.smithi175.stdout:1911: left oid 3 (ObjNum 656 snap 210 seq_num 656) 2024-09-11T03:45:34.990 INFO:tasks.rados.rados.2.smithi175.stdout:1912: finishing write tid 1 to smithi17557870-31 2024-09-11T03:45:34.990 INFO:tasks.rados.rados.2.smithi175.stdout:1912: finishing write tid 2 to smithi17557870-31 2024-09-11T03:45:34.990 INFO:tasks.rados.rados.2.smithi175.stdout:1911: done (2 left) 2024-09-11T03:45:34.990 INFO:tasks.rados.rados.2.smithi175.stdout:1914: delete oid 35 current snap is 210 2024-09-11T03:45:34.991 INFO:tasks.rados.rados.2.smithi175.stdout:1913: finishing copy_from to smithi17557870-18 2024-09-11T03:45:34.991 INFO:tasks.rados.rados.2.smithi175.stdout:1913: got expected ENOENT (src dne) 2024-09-11T03:45:34.991 INFO:tasks.rados.rados.2.smithi175.stdout:1913: finishing copy_from racing read to smithi17557870-18 2024-09-11T03:45:34.991 INFO:tasks.rados.rados.2.smithi175.stdout:1912: finishing write tid 3 to smithi17557870-31 2024-09-11T03:45:34.991 INFO:tasks.rados.rados.2.smithi175.stdout:1912: finishing write tid 4 to smithi17557870-31 2024-09-11T03:45:34.991 INFO:tasks.rados.rados.2.smithi175.stdout:1912: finishing write tid 5 to smithi17557870-31 2024-09-11T03:45:34.992 INFO:tasks.rados.rados.2.smithi175.stdout:1912: oid 31 updating version 0 to 734 2024-09-11T03:45:34.992 INFO:tasks.rados.rados.2.smithi175.stdout:1912: oid 31 version 734 is already newer than 732 2024-09-11T03:45:34.992 INFO:tasks.rados.rados.2.smithi175.stdout:1912: oid 31 updating version 734 to 735 2024-09-11T03:45:34.992 INFO:tasks.rados.rados.2.smithi175.stdout:1912: oid 31 version 735 is already newer than 733 2024-09-11T03:45:34.992 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 735 (ObjNum 657 snap 210 seq_num 657) dirty exists 2024-09-11T03:45:34.992 INFO:tasks.rados.rados.2.smithi175.stdout:1912: left oid 31 (ObjNum 657 snap 210 seq_num 657) 2024-09-11T03:45:34.993 INFO:tasks.rados.rados.2.smithi175.stdout:1912: done (2 left) 2024-09-11T03:45:34.993 INFO:tasks.rados.rados.2.smithi175.stdout:1913: done (1 left) 2024-09-11T03:45:34.994 INFO:tasks.rados.rados.2.smithi175.stdout:1914: done (0 left) 2024-09-11T03:45:34.994 INFO:tasks.rados.rados.2.smithi175.stdout:1915: snap_create 2024-09-11T03:45:34.999 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:35.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:35.021 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:35.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:35.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:35.141 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:45:35.148 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:35.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:35.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:34 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1528725967' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:45:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:34 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1528725967' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:45:35.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:34 smithi175 ceph-mon[30894]: osdmap e517: 12 total, 12 up, 11 in 2024-09-11T03:45:35.250 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:35.272 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:35.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:35.291 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:34 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1528725967' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:45:35.291 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:34 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1528725967' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:45:35.292 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:34 smithi084 ceph-mon[27254]: osdmap e517: 12 total, 12 up, 11 in 2024-09-11T03:45:35.374 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:35.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:35.396 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:35.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:35.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:34 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1528725967' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:45:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:34 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1528725967' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:45:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:34 smithi121 ceph-mon[30729]: osdmap e517: 12 total, 12 up, 11 in 2024-09-11T03:45:35.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:35.518 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:35.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:35.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:35.609 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:45:35.609 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:45:35.620 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:35.646 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:35.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:35.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:35.748 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:35.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:35.769 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:35.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:35.871 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:35.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:35.894 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:35.894 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:35.983 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:45:35.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:35.995 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:45:36.001 INFO:tasks.rados.rados.2.smithi175.stdout:1915: done (0 left) 2024-09-11T03:45:36.001 INFO:tasks.rados.rados.2.smithi175.stdout:1916: copy_from oid 29 from oid 23 current snap is 211 2024-09-11T03:45:36.001 INFO:tasks.rados.rados.2.smithi175.stdout:1917: read oid 10 snap -1 2024-09-11T03:45:36.001 INFO:tasks.rados.rados.2.smithi175.stdout:1917: expect deleted 2024-09-11T03:45:36.001 INFO:tasks.rados.rados.2.smithi175.stdout:1918: write oid 6 current snap is 211 2024-09-11T03:45:36.001 INFO:tasks.rados.rados.2.smithi175.stdout:1918: seq_num 659 ranges {675813=795449,2096810=609228} 2024-09-11T03:45:36.010 INFO:tasks.rados.rados.2.smithi175.stdout:1918: writing smithi17557870-6 from 675813 to 1471262 tid 1 2024-09-11T03:45:36.017 INFO:tasks.rados.rados.2.smithi175.stdout:1918: writing smithi17557870-6 from 2096810 to 2706038 tid 2 2024-09-11T03:45:36.017 INFO:tasks.rados.rados.2.smithi175.stdout:1919: read oid 37 snap -1 2024-09-11T03:45:36.017 INFO:tasks.rados.rados.2.smithi175.stdout:1919: expect deleted 2024-09-11T03:45:36.018 INFO:tasks.rados.rados.2.smithi175.stdout:1920: delete oid 22 current snap is 211 2024-09-11T03:45:36.018 INFO:tasks.rados.rados.2.smithi175.stdout:1916: finishing copy_from racing read to smithi17557870-29 2024-09-11T03:45:36.018 INFO:tasks.rados.rados.2.smithi175.stdout:1916: finishing copy_from to smithi17557870-29 2024-09-11T03:45:36.018 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 350 (ObjNum 639 snap 204 seq_num 639) dirty exists 2024-09-11T03:45:36.019 INFO:tasks.rados.rados.2.smithi175.stdout:1918: finishing write tid 1 to smithi17557870-6 2024-09-11T03:45:36.021 INFO:tasks.rados.rados.2.smithi175.stdout:1918: finishing write tid 2 to smithi17557870-6 2024-09-11T03:45:36.021 INFO:tasks.rados.rados.2.smithi175.stdout:1918: finishing write tid 3 to smithi17557870-6 2024-09-11T03:45:36.021 INFO:tasks.rados.rados.2.smithi175.stdout:1916: done (4 left) 2024-09-11T03:45:36.021 INFO:tasks.rados.rados.2.smithi175.stdout:1917: done (3 left) 2024-09-11T03:45:36.021 INFO:tasks.rados.rados.2.smithi175.stdout:1919: done (2 left) 2024-09-11T03:45:36.021 INFO:tasks.rados.rados.2.smithi175.stdout:1920: done (1 left) 2024-09-11T03:45:36.021 INFO:tasks.rados.rados.2.smithi175.stdout:1921: delete oid 49 current snap is 211 2024-09-11T03:45:36.021 INFO:tasks.rados.rados.2.smithi175.stdout:1918: finishing write tid 4 to smithi17557870-6 2024-09-11T03:45:36.021 INFO:tasks.rados.rados.2.smithi175.stdout:1918: oid 6 updating version 0 to 437 2024-09-11T03:45:36.021 INFO:tasks.rados.rados.2.smithi175.stdout:1918: oid 6 updating version 437 to 438 2024-09-11T03:45:36.022 INFO:tasks.rados.rados.2.smithi175.stdout:1918: oid 6 updating version 438 to 439 2024-09-11T03:45:36.022 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 439 (ObjNum 659 snap 211 seq_num 659) dirty exists 2024-09-11T03:45:36.022 INFO:tasks.rados.rados.2.smithi175.stdout:1918: left oid 6 (ObjNum 659 snap 211 seq_num 659) 2024-09-11T03:45:36.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:36.023 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:36.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:36.023 INFO:tasks.rados.rados.2.smithi175.stdout:1918: done (1 left) 2024-09-11T03:45:36.024 INFO:tasks.rados.rados.2.smithi175.stdout:1921: done (0 left) 2024-09-11T03:45:36.024 INFO:tasks.rados.rados.2.smithi175.stdout:1922: snap_create 2024-09-11T03:45:36.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:36.127 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-11T03:45:36.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:36.145 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:36.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:36.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:36 smithi175 ceph-mon[30894]: pgmap v930: 57 pgs: 57 active+clean; 165 MiB data, 799 MiB used, 983 GiB / 983 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 10 op/s 2024-09-11T03:45:36.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1808209047' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:45:36.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:36.269 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-11T03:45:36.273 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:36.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:36.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:36.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:36 smithi084 ceph-mon[27254]: pgmap v930: 57 pgs: 57 active+clean; 165 MiB data, 799 MiB used, 983 GiB / 983 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 10 op/s 2024-09-11T03:45:36.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1808209047' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:45:36.375 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:36.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:36.397 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:36.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:36.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:36 smithi121 ceph-mon[30729]: pgmap v930: 57 pgs: 57 active+clean; 165 MiB data, 799 MiB used, 983 GiB / 983 GiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 10 op/s 2024-09-11T03:45:36.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1808209047' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:45:36.499 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:36.513 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:45:36.521 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:36.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:36.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:36.623 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:36.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:36.645 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:36.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:36.746 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:36.771 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:36.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:36.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:36.780 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:36.784 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:36.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:36.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:36.895 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:36.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:36.947 INFO:tasks.rados.rados.2.smithi175.stdout:1922: done (0 left) 2024-09-11T03:45:36.947 INFO:tasks.rados.rados.2.smithi175.stdout:1923: write (excl) oid 12 current snap is 212 2024-09-11T03:45:36.947 INFO:tasks.rados.rados.2.smithi175.stdout:1923: seq_num 662 ranges {726234=737549,1988609=756555} 2024-09-11T03:45:36.955 INFO:tasks.rados.rados.2.smithi175.stdout:1923: writing smithi17557870-12 from 726234 to 1463783 tid 1 2024-09-11T03:45:36.962 INFO:tasks.rados.rados.2.smithi175.stdout:1923: writing smithi17557870-12 from 1988609 to 2745164 tid 2 2024-09-11T03:45:36.962 INFO:tasks.rados.rados.2.smithi175.stdout:1924: write oid 3 current snap is 212 2024-09-11T03:45:36.962 INFO:tasks.rados.rados.2.smithi175.stdout:1923: finishing write tid 1 to smithi17557870-12 2024-09-11T03:45:36.962 INFO:tasks.rados.rados.2.smithi175.stdout:1924: seq_num 663 ranges {743041=412798,1930738=616566,3040503=102538} 2024-09-11T03:45:36.968 INFO:tasks.rados.rados.2.smithi175.stdout:1924: writing smithi17557870-3 from 743041 to 1155839 tid 1 2024-09-11T03:45:36.976 INFO:tasks.rados.rados.2.smithi175.stdout:1924: writing smithi17557870-3 from 1930738 to 2547304 tid 2 2024-09-11T03:45:36.980 INFO:tasks.rados.rados.2.smithi175.stdout:1924: writing smithi17557870-3 from 3040503 to 3143041 tid 3 2024-09-11T03:45:36.980 INFO:tasks.rados.rados.2.smithi175.stdout:1925: write oid 9 current snap is 212 2024-09-11T03:45:36.980 INFO:tasks.rados.rados.2.smithi175.stdout:1923: finishing write tid 2 to smithi17557870-12 2024-09-11T03:45:36.980 INFO:tasks.rados.rados.2.smithi175.stdout:1923: finishing write tid 3 to smithi17557870-12 2024-09-11T03:45:36.980 INFO:tasks.rados.rados.2.smithi175.stdout:1923: finishing write tid 4 to smithi17557870-12 2024-09-11T03:45:36.980 INFO:tasks.rados.rados.2.smithi175.stdout:1923: oid 12 updating version 0 to 732 2024-09-11T03:45:36.980 INFO:tasks.rados.rados.2.smithi175.stdout:1923: oid 12 updating version 732 to 733 2024-09-11T03:45:36.981 INFO:tasks.rados.rados.2.smithi175.stdout:1923: oid 12 updating version 733 to 734 2024-09-11T03:45:36.981 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 734 (ObjNum 662 snap 212 seq_num 662) dirty exists 2024-09-11T03:45:36.981 INFO:tasks.rados.rados.2.smithi175.stdout:1923: left oid 12 (ObjNum 662 snap 212 seq_num 662) 2024-09-11T03:45:36.981 INFO:tasks.rados.rados.2.smithi175.stdout:1924: finishing write tid 1 to smithi17557870-3 2024-09-11T03:45:36.981 INFO:tasks.rados.rados.2.smithi175.stdout:1925: seq_num 664 ranges {759848=488047,1989220=760224} 2024-09-11T03:45:36.982 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:45:36.983 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:45:36.986 INFO:tasks.rados.rados.2.smithi175.stdout:1925: writing smithi17557870-9 from 759848 to 1247895 tid 1 2024-09-11T03:45:36.996 INFO:tasks.rados.rados.2.smithi175.stdout:1925: writing smithi17557870-9 from 1989220 to 2749444 tid 2 2024-09-11T03:45:36.996 INFO:tasks.rados.rados.2.smithi175.stdout:1923: done (2 left) 2024-09-11T03:45:36.996 INFO:tasks.rados.rados.2.smithi175.stdout:1926: read oid 48 snap -1 2024-09-11T03:45:36.996 INFO:tasks.rados.rados.2.smithi175.stdout:1926: expect deleted 2024-09-11T03:45:36.996 INFO:tasks.rados.rados.2.smithi175.stdout:1927: read oid 45 snap -1 2024-09-11T03:45:36.996 INFO:tasks.rados.rados.2.smithi175.stdout:1927: expect (ObjNum 416 snap 127 seq_num 416) 2024-09-11T03:45:36.996 INFO:tasks.rados.rados.2.smithi175.stdout:1928: read oid 22 snap 208 2024-09-11T03:45:36.996 INFO:tasks.rados.rados.2.smithi175.stdout:1928: expect (ObjNum 506 snap 148 seq_num 506) 2024-09-11T03:45:36.996 INFO:tasks.rados.rados.2.smithi175.stdout:1929: write oid 21 current snap is 212 2024-09-11T03:45:36.996 INFO:tasks.rados.rados.2.smithi175.stdout:1929: seq_num 665 ranges {776655=563296,2047702=620235,3095349=81306} 2024-09-11T03:45:36.997 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:37.003 INFO:tasks.rados.rados.2.smithi175.stdout:1929: writing smithi17557870-21 from 776655 to 1339951 tid 1 2024-09-11T03:45:37.011 INFO:tasks.rados.rados.2.smithi175.stdout:1929: writing smithi17557870-21 from 2047702 to 2667937 tid 2 2024-09-11T03:45:37.015 INFO:tasks.rados.rados.2.smithi175.stdout:1929: writing smithi17557870-21 from 3095349 to 3176655 tid 3 2024-09-11T03:45:37.015 INFO:tasks.rados.rados.2.smithi175.stdout:1930: delete oid 5 current snap is 212 2024-09-11T03:45:37.015 INFO:tasks.rados.rados.2.smithi175.stdout:1924: finishing write tid 2 to smithi17557870-3 2024-09-11T03:45:37.015 INFO:tasks.rados.rados.2.smithi175.stdout:1924: finishing write tid 3 to smithi17557870-3 2024-09-11T03:45:37.015 INFO:tasks.rados.rados.2.smithi175.stdout:1924: finishing write tid 4 to smithi17557870-3 2024-09-11T03:45:37.015 INFO:tasks.rados.rados.2.smithi175.stdout:1924: finishing write tid 5 to smithi17557870-3 2024-09-11T03:45:37.015 INFO:tasks.rados.rados.2.smithi175.stdout:1924: oid 3 updating version 0 to 758 2024-09-11T03:45:37.015 INFO:tasks.rados.rados.2.smithi175.stdout:1924: oid 3 updating version 758 to 759 2024-09-11T03:45:37.015 INFO:tasks.rados.rados.2.smithi175.stdout:1924: oid 3 updating version 759 to 760 2024-09-11T03:45:37.015 INFO:tasks.rados.rados.2.smithi175.stdout:1924: oid 3 updating version 760 to 761 2024-09-11T03:45:37.015 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 761 (ObjNum 663 snap 212 seq_num 663) dirty exists 2024-09-11T03:45:37.015 INFO:tasks.rados.rados.2.smithi175.stdout:1924: left oid 3 (ObjNum 663 snap 212 seq_num 663) 2024-09-11T03:45:37.016 INFO:tasks.rados.rados.2.smithi175.stdout:1925: finishing write tid 1 to smithi17557870-9 2024-09-11T03:45:37.016 INFO:tasks.rados.rados.2.smithi175.stdout:1925: finishing write tid 2 to smithi17557870-9 2024-09-11T03:45:37.016 INFO:tasks.rados.rados.2.smithi175.stdout:1925: finishing write tid 3 to smithi17557870-9 2024-09-11T03:45:37.016 INFO:tasks.rados.rados.2.smithi175.stdout:1925: finishing write tid 4 to smithi17557870-9 2024-09-11T03:45:37.016 INFO:tasks.rados.rados.2.smithi175.stdout:1925: oid 9 updating version 0 to 737 2024-09-11T03:45:37.016 INFO:tasks.rados.rados.2.smithi175.stdout:1925: oid 9 updating version 737 to 738 2024-09-11T03:45:37.016 INFO:tasks.rados.rados.2.smithi175.stdout:1925: oid 9 version 738 is already newer than 736 2024-09-11T03:45:37.016 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 738 (ObjNum 664 snap 212 seq_num 664) dirty exists 2024-09-11T03:45:37.016 INFO:tasks.rados.rados.2.smithi175.stdout:1925: left oid 9 (ObjNum 664 snap 212 seq_num 664) 2024-09-11T03:45:37.016 INFO:tasks.rados.rados.2.smithi175.stdout:1927: expect (ObjNum 416 snap 127 seq_num 416) 2024-09-11T03:45:37.024 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:37.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:37.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:37.126 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:37.148 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:37.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:37.188 INFO:tasks.rados.rados.2.smithi175.stdout:1928: expect (ObjNum 506 snap 148 seq_num 506) 2024-09-11T03:45:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:37 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1808209047' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:45:37.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:37 smithi175 ceph-mon[30894]: osdmap e518: 12 total, 12 up, 11 in 2024-09-11T03:45:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:37 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:37.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:37 smithi175 ceph-mon[30894]: osdmap e519: 12 total, 12 up, 11 in 2024-09-11T03:45:37.250 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:37.269 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1808209047' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:45:37.269 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:37 smithi084 ceph-mon[27254]: osdmap e518: 12 total, 12 up, 11 in 2024-09-11T03:45:37.269 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:37 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:37.269 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:37 smithi084 ceph-mon[27254]: osdmap e519: 12 total, 12 up, 11 in 2024-09-11T03:45:37.270 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:37.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:37.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:37.301 INFO:tasks.rados.rados.2.smithi175.stdout:1929: finishing write tid 1 to smithi17557870-21 2024-09-11T03:45:37.301 INFO:tasks.rados.rados.2.smithi175.stdout:1929: finishing write tid 2 to smithi17557870-21 2024-09-11T03:45:37.302 INFO:tasks.rados.rados.2.smithi175.stdout:1929: finishing write tid 3 to smithi17557870-21 2024-09-11T03:45:37.302 INFO:tasks.rados.rados.2.smithi175.stdout:1929: finishing write tid 4 to smithi17557870-21 2024-09-11T03:45:37.302 INFO:tasks.rados.rados.2.smithi175.stdout:1929: finishing write tid 5 to smithi17557870-21 2024-09-11T03:45:37.302 INFO:tasks.rados.rados.2.smithi175.stdout:1929: oid 21 updating version 0 to 742 2024-09-11T03:45:37.302 INFO:tasks.rados.rados.2.smithi175.stdout:1929: oid 21 version 742 is already newer than 741 2024-09-11T03:45:37.302 INFO:tasks.rados.rados.2.smithi175.stdout:1929: oid 21 updating version 742 to 743 2024-09-11T03:45:37.302 INFO:tasks.rados.rados.2.smithi175.stdout:1929: oid 21 version 743 is already newer than 740 2024-09-11T03:45:37.302 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 743 (ObjNum 665 snap 212 seq_num 665) dirty exists 2024-09-11T03:45:37.302 INFO:tasks.rados.rados.2.smithi175.stdout:1929: left oid 21 (ObjNum 665 snap 212 seq_num 665) 2024-09-11T03:45:37.302 INFO:tasks.rados.rados.2.smithi175.stdout:1924: done (6 left) 2024-09-11T03:45:37.302 INFO:tasks.rados.rados.2.smithi175.stdout:1925: done (5 left) 2024-09-11T03:45:37.302 INFO:tasks.rados.rados.2.smithi175.stdout:1926: done (4 left) 2024-09-11T03:45:37.302 INFO:tasks.rados.rados.2.smithi175.stdout:1927: done (3 left) 2024-09-11T03:45:37.303 INFO:tasks.rados.rados.2.smithi175.stdout:1928: done (2 left) 2024-09-11T03:45:37.303 INFO:tasks.rados.rados.2.smithi175.stdout:1929: done (1 left) 2024-09-11T03:45:37.303 INFO:tasks.rados.rados.2.smithi175.stdout:1930: done (0 left) 2024-09-11T03:45:37.303 INFO:tasks.rados.rados.2.smithi175.stdout:1931: read oid 48 snap -1 2024-09-11T03:45:37.303 INFO:tasks.rados.rados.2.smithi175.stdout:1931: expect deleted 2024-09-11T03:45:37.303 INFO:tasks.rados.rados.2.smithi175.stdout:1932: read oid 21 snap -1 2024-09-11T03:45:37.303 INFO:tasks.rados.rados.2.smithi175.stdout:1932: expect (ObjNum 665 snap 212 seq_num 665) 2024-09-11T03:45:37.303 INFO:tasks.rados.rados.2.smithi175.stdout:1933: write oid 7 current snap is 212 2024-09-11T03:45:37.303 INFO:tasks.rados.rados.2.smithi175.stdout:1933: seq_num 667 ranges {410269=713794,1648313=507551,2917489=292780} 2024-09-11T03:45:37.307 INFO:tasks.rados.rados.2.smithi175.stdout:1933: writing smithi17557870-7 from 410269 to 1124063 tid 1 2024-09-11T03:45:37.313 INFO:tasks.rados.rados.2.smithi175.stdout:1933: writing smithi17557870-7 from 1648313 to 2155864 tid 2 2024-09-11T03:45:37.320 INFO:tasks.rados.rados.2.smithi175.stdout:1933: writing smithi17557870-7 from 2917489 to 3210269 tid 3 2024-09-11T03:45:37.320 INFO:tasks.rados.rados.2.smithi175.stdout:1934: rollback oid 25 current snap is 212 2024-09-11T03:45:37.320 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 25 to 209 2024-09-11T03:45:37.320 INFO:tasks.rados.rados.2.smithi175.stdout:1932: expect (ObjNum 665 snap 212 seq_num 665) 2024-09-11T03:45:37.372 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:37.393 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:37.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:37.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:37 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1808209047' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:45:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:37 smithi121 ceph-mon[30729]: osdmap e518: 12 total, 12 up, 11 in 2024-09-11T03:45:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:37 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:37 smithi121 ceph-mon[30729]: osdmap e519: 12 total, 12 up, 11 in 2024-09-11T03:45:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:1933: finishing write tid 1 to smithi17557870-7 2024-09-11T03:45:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:1933: finishing write tid 2 to smithi17557870-7 2024-09-11T03:45:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:1933: finishing write tid 3 to smithi17557870-7 2024-09-11T03:45:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:1934: finishing rollback tid 0 to smithi17557870-25 2024-09-11T03:45:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:1933: finishing write tid 4 to smithi17557870-7 2024-09-11T03:45:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:1934: finishing rollback tid 1 to smithi17557870-25 2024-09-11T03:45:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:1933: finishing write tid 5 to smithi17557870-7 2024-09-11T03:45:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:1933: oid 7 updating version 0 to 513 2024-09-11T03:45:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:1933: oid 7 updating version 513 to 516 2024-09-11T03:45:37.494 INFO:tasks.rados.rados.2.smithi175.stdout:1933: oid 7 version 516 is already newer than 515 2024-09-11T03:45:37.495 INFO:tasks.rados.rados.2.smithi175.stdout:1933: oid 7 version 516 is already newer than 514 2024-09-11T03:45:37.495 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 516 (ObjNum 667 snap 212 seq_num 667) dirty exists 2024-09-11T03:45:37.495 INFO:tasks.rados.rados.2.smithi175.stdout:1933: left oid 7 (ObjNum 667 snap 212 seq_num 667) 2024-09-11T03:45:37.495 INFO:tasks.rados.rados.2.smithi175.stdout:1934: finishing rollback tid 2 to smithi17557870-25 2024-09-11T03:45:37.495 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 497 (ObjNum 496 snap 144 seq_num 496) dirty exists 2024-09-11T03:45:37.495 INFO:tasks.rados.rados.2.smithi175.stdout:1931: done (3 left) 2024-09-11T03:45:37.495 INFO:tasks.rados.rados.2.smithi175.stdout:1932: done (2 left) 2024-09-11T03:45:37.495 INFO:tasks.rados.rados.2.smithi175.stdout:1933: done (1 left) 2024-09-11T03:45:37.495 INFO:tasks.rados.rados.2.smithi175.stdout:1934: done (0 left) 2024-09-11T03:45:37.495 INFO:tasks.rados.rados.2.smithi175.stdout:1935: write (excl) oid 14 current snap is 212 2024-09-11T03:45:37.495 INFO:tasks.rados.rados.2.smithi175.stdout:1935: seq_num 668 ranges {427076=789043,1706795=767562,2944912=282164} 2024-09-11T03:45:37.496 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:37.499 INFO:tasks.rados.rados.2.smithi175.stdout:1935: writing smithi17557870-14 from 427076 to 1216119 tid 1 2024-09-11T03:45:37.506 INFO:tasks.rados.rados.2.smithi175.stdout:1935: writing smithi17557870-14 from 1706795 to 2474357 tid 2 2024-09-11T03:45:37.510 INFO:tasks.rados.rados.2.smithi175.stdout:1935: writing smithi17557870-14 from 2944912 to 3227076 tid 3 2024-09-11T03:45:37.511 INFO:tasks.rados.rados.2.smithi175.stdout:1936: snap_create 2024-09-11T03:45:37.511 INFO:tasks.rados.rados.2.smithi175.stdout:1935: finishing write tid 1 to smithi17557870-14 2024-09-11T03:45:37.511 INFO:tasks.rados.rados.2.smithi175.stdout:1935: finishing write tid 2 to smithi17557870-14 2024-09-11T03:45:37.513 INFO:tasks.rados.rados.2.smithi175.stdout:1935: finishing write tid 3 to smithi17557870-14 2024-09-11T03:45:37.514 INFO:tasks.rados.rados.2.smithi175.stdout:1935: finishing write tid 4 to smithi17557870-14 2024-09-11T03:45:37.514 INFO:tasks.rados.rados.2.smithi175.stdout:1935: finishing write tid 5 to smithi17557870-14 2024-09-11T03:45:37.514 INFO:tasks.rados.rados.2.smithi175.stdout:1935: oid 14 updating version 0 to 753 2024-09-11T03:45:37.514 INFO:tasks.rados.rados.2.smithi175.stdout:1935: oid 14 version 753 is already newer than 752 2024-09-11T03:45:37.514 INFO:tasks.rados.rados.2.smithi175.stdout:1935: oid 14 version 753 is already newer than 751 2024-09-11T03:45:37.514 INFO:tasks.rados.rados.2.smithi175.stdout:1935: oid 14 version 753 is already newer than 750 2024-09-11T03:45:37.514 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 753 (ObjNum 668 snap 212 seq_num 668) dirty exists 2024-09-11T03:45:37.514 INFO:tasks.rados.rados.2.smithi175.stdout:1935: left oid 14 (ObjNum 668 snap 212 seq_num 668) 2024-09-11T03:45:37.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:37.519 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:37.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:37.621 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:37.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:37.643 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:37.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:37.733 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:45:37.733 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:45:37.740 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:45:37.740 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:45:37.744 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:37.757 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-11T03:45:37.758 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:45:37.758 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:45:37.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:37.766 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:37.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:37.773 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:45:37.773 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:45:37.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:45:37.791 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:45:37.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:45:37.808 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:45:37.825 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:45:37.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:45:37.841 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:45:37.841 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:45:37.858 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:45:37.858 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:45:37.867 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:37.874 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:45:37.874 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:45:37.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:37.889 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:37.889 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:37.891 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:45:37.891 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:45:37.907 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:45:37.907 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:45:37.990 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:38.000 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:45:38.011 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:38.012 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:38.015 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 1 (10000) 2024-09-11T03:45:38.019 INFO:tasks.rados.rados.2.smithi175.stdout:1935: done (1 left) 2024-09-11T03:45:38.019 INFO:tasks.rados.rados.2.smithi175.stdout:1936: done (0 left) 2024-09-11T03:45:38.019 INFO:tasks.rados.rados.2.smithi175.stdout:1937: rollback oid 20 current snap is 213 2024-09-11T03:45:38.019 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 20 to 211 2024-09-11T03:45:38.019 INFO:tasks.rados.rados.2.smithi175.stdout:1938: rollback oid 39 current snap is 213 2024-09-11T03:45:38.019 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 39 to 208 2024-09-11T03:45:38.019 INFO:tasks.rados.rados.2.smithi175.stdout:1939: read oid 38 snap -1 2024-09-11T03:45:38.020 INFO:tasks.rados.rados.2.smithi175.stdout:1939: expect deleted 2024-09-11T03:45:38.020 INFO:tasks.rados.rados.2.smithi175.stdout:1940: snap_remove snap 212 2024-09-11T03:45:38.113 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:38.135 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:38.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:38.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:38.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:38 smithi084 ceph-mon[27254]: pgmap v933: 57 pgs: 57 active+clean; 158 MiB data, 727 MiB used, 983 GiB / 983 GiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 9 op/s 2024-09-11T03:45:38.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:38 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1163825320' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:45:38.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:38 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2150784101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:38.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:38.259 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:38.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:38.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:38.263 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:38 smithi175 ceph-mon[30894]: pgmap v933: 57 pgs: 57 active+clean; 158 MiB data, 727 MiB used, 983 GiB / 983 GiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 9 op/s 2024-09-11T03:45:38.263 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:38 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1163825320' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:45:38.263 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:38 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2150784101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:38.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:38 smithi121 ceph-mon[30729]: pgmap v933: 57 pgs: 57 active+clean; 158 MiB data, 727 MiB used, 983 GiB / 983 GiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 9 op/s 2024-09-11T03:45:38.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:38 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1163825320' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:45:38.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:38 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2150784101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:38.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:38.387 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:38.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:38.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:38.475 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:45:38.475 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:45:38.475 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:45:38.475 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:45:38.475 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:45:38.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:38.511 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:38.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:38.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:38.613 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:38.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:38.635 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:38.636 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:38.721 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:45:38.737 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:38.757 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:38.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:38.759 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:38.860 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:38.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-11T03:45:38.881 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:38.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:38.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:38.983 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:39.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:39.005 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:39.006 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:1940: done (3 left) 2024-09-11T03:45:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:1941: read oid 7 snap -1 2024-09-11T03:45:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:1941: expect (ObjNum 667 snap 212 seq_num 667) 2024-09-11T03:45:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:1942: delete oid 8 current snap is 213 2024-09-11T03:45:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:1938: finishing rollback tid 0 to smithi17557870-39 2024-09-11T03:45:39.025 INFO:tasks.rados.rados.2.smithi175.stdout:1938: finishing rollback tid 1 to smithi17557870-39 2024-09-11T03:45:39.026 INFO:tasks.rados.rados.2.smithi175.stdout:1938: finishing rollback tid 2 to smithi17557870-39 2024-09-11T03:45:39.026 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 577 (ObjNum 528 snap 156 seq_num 528) dirty exists 2024-09-11T03:45:39.026 INFO:tasks.rados.rados.2.smithi175.stdout:1937: finishing rollback tid 0 to smithi17557870-20 2024-09-11T03:45:39.026 INFO:tasks.rados.rados.2.smithi175.stdout:1937: finishing rollback tid 1 to smithi17557870-20 2024-09-11T03:45:39.026 INFO:tasks.rados.rados.2.smithi175.stdout:1937: finishing rollback tid 2 to smithi17557870-20 2024-09-11T03:45:39.026 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 699 (ObjNum 563 snap 172 seq_num 563) dirty exists 2024-09-11T03:45:39.027 INFO:tasks.rados.rados.2.smithi175.stdout:1937: done (4 left) 2024-09-11T03:45:39.027 INFO:tasks.rados.rados.2.smithi175.stdout:1938: done (3 left) 2024-09-11T03:45:39.027 INFO:tasks.rados.rados.2.smithi175.stdout:1939: done (2 left) 2024-09-11T03:45:39.027 INFO:tasks.rados.rados.2.smithi175.stdout:1942: done (1 left) 2024-09-11T03:45:39.027 INFO:tasks.rados.rados.2.smithi175.stdout:1943: snap_create 2024-09-11T03:45:39.044 INFO:tasks.rados.rados.2.smithi175.stdout:1941: expect (ObjNum 667 snap 212 seq_num 667) 2024-09-11T03:45:39.106 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:39.122 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-11T03:45:39.127 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:39.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:39.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:39.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:39.250 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:39.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:39.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:39.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:39 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1163825320' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:45:39.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:39 smithi084 ceph-mon[27254]: osdmap e520: 12 total, 12 up, 11 in 2024-09-11T03:45:39.352 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:39.374 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:39.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:39.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:39.405 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:39.406 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:45:39.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:39 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1163825320' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:45:39.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:39 smithi121 ceph-mon[30729]: osdmap e520: 12 total, 12 up, 11 in 2024-09-11T03:45:39.476 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:39.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:39 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1163825320' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:45:39.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:39 smithi175 ceph-mon[30894]: osdmap e520: 12 total, 12 up, 11 in 2024-09-11T03:45:39.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:39.498 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:39.498 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:39.599 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:39.621 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:39.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:39.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:39.723 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:39.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:39.747 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:39.748 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:39.848 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:39.850 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:45:39.850 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:45:39.850 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:45:39.850 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:45:39.850 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.850 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:640 2.a 2:5014e305:::smithi17557870-46:head [copy-from ver 23] snapc 11=[e] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:45:39.851 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:45:39.851 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 391.72924174299999, 2024-09-11T03:45:39.851 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019221736, 2024-09-11T03:45:39.851 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.851 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.851 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:39.851 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:39.851 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:39.851 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:39.851 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 640 2024-09-11T03:45:39.851 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.851 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.851 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.851 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:45:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:45:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110295+0000", 2024-09-11T03:45:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0609999999999998e-06 2024-09-11T03:45:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110305+0000", 2024-09-11T03:45:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4169999999999998e-06 2024-09-11T03:45:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110321+0000", 2024-09-11T03:45:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6288000000000001e-05 2024-09-11T03:45:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110328+0000", 2024-09-11T03:45:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3950000000000003e-06 2024-09-11T03:45:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110396+0000", 2024-09-11T03:45:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7164999999999998e-05 2024-09-11T03:45:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110507+0000", 2024-09-11T03:45:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000111217 2024-09-11T03:45:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:45:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.118213+0000", 2024-09-11T03:45:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0077066249999999999 2024-09-11T03:45:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125103+0000", 2024-09-11T03:45:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068894359999999997 2024-09-11T03:45:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125459+0000", 2024-09-11T03:45:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000356146 2024-09-11T03:45:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129369+0000", 2024-09-11T03:45:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039099249999999999 2024-09-11T03:45:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129428+0000", 2024-09-11T03:45:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9113000000000001e-05 2024-09-11T03:45:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129515+0000", 2024-09-11T03:45:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6947999999999995e-05 2024-09-11T03:45:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:641 2.a 2:5014e305:::smithi17557870-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:45:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:45:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 391.72919035199999, 2024-09-11T03:45:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011882876000000001, 2024-09-11T03:45:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.858 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.858 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:45:39.858 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:39.858 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:39.858 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:39.858 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 641 2024-09-11T03:45:39.858 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.858 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.858 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.858 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.858 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:45:39.858 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.858 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.858 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.859 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.859 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:45:39.859 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.859 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.859 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.859 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.859 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:45:39.859 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7599999999999998e-07 2024-09-11T03:45:39.859 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.859 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.859 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.859 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110347+0000", 2024-09-11T03:45:39.859 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.685e-06 2024-09-11T03:45:39.859 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.860 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.860 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.860 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110351+0000", 2024-09-11T03:45:39.860 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4889999999999997e-06 2024-09-11T03:45:39.860 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.860 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.860 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.860 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110354+0000", 2024-09-11T03:45:39.860 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0510000000000001e-06 2024-09-11T03:45:39.860 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.860 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.860 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.860 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110644+0000", 2024-09-11T03:45:39.860 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00028940200000000002 2024-09-11T03:45:39.861 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.861 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.861 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:39.861 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110673+0000", 2024-09-11T03:45:39.861 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9833e-05 2024-09-11T03:45:39.861 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.861 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.861 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.861 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122038+0000", 2024-09-11T03:45:39.861 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011364915 2024-09-11T03:45:39.861 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.861 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.861 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.862 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122127+0000", 2024-09-11T03:45:39.862 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8987999999999996e-05 2024-09-11T03:45:39.862 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.862 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.862 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.862 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122227+0000", 2024-09-11T03:45:39.862 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010013699999999999 2024-09-11T03:45:39.862 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.862 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.862 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.862 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.862 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.862 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:752 2.c 2:3c007163:::smithi17557870-35:head [copy-from ver 39] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:45:39.862 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:45:39.863 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 384.83051906999998, 2024-09-11T03:45:39.863 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017596783000000001, 2024-09-11T03:45:39.863 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.863 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.863 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:39.863 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:39.863 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:39.863 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:39.863 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 752 2024-09-11T03:45:39.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.863 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.863 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.863 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.864 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:45:39.864 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.864 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.864 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.864 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:45:39.864 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.864 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.864 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.864 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009018+0000", 2024-09-11T03:45:39.864 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.48e-06 2024-09-11T03:45:39.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.864 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009029+0000", 2024-09-11T03:45:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0377000000000001e-05 2024-09-11T03:45:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009050+0000", 2024-09-11T03:45:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0849e-05 2024-09-11T03:45:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009058+0000", 2024-09-11T03:45:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8019999999999998e-06 2024-09-11T03:45:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009125+0000", 2024-09-11T03:45:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6388000000000002e-05 2024-09-11T03:45:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:45:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009130+0000", 2024-09-11T03:45:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0900000000000004e-06 2024-09-11T03:45:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010518+0000", 2024-09-11T03:45:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013880030000000001 2024-09-11T03:45:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010582+0000", 2024-09-11T03:45:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.368e-05 2024-09-11T03:45:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:45:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.015350+0000", 2024-09-11T03:45:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0047682870000000004 2024-09-11T03:45:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.019005+0000", 2024-09-11T03:45:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0036549040000000001 2024-09-11T03:45:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.024096+0000", 2024-09-11T03:45:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050917439999999996 2024-09-11T03:45:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026521+0000", 2024-09-11T03:45:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024247399999999999 2024-09-11T03:45:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026559+0000", 2024-09-11T03:45:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8232999999999998e-05 2024-09-11T03:45:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026613+0000", 2024-09-11T03:45:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3205999999999998e-05 2024-09-11T03:45:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:753 2.c 2:3c007163:::smithi17557870-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:45:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:45:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 384.83045198299999, 2024-09-11T03:45:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0094619260000000007, 2024-09-11T03:45:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:45:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 753 2024-09-11T03:45:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.871 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.871 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:45:39.871 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.871 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.871 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.871 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:45:39.871 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.872 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.872 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009083+0000", 2024-09-11T03:45:39.872 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2100000000000002e-07 2024-09-11T03:45:39.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.872 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.872 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009085+0000", 2024-09-11T03:45:39.872 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5910000000000001e-06 2024-09-11T03:45:39.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.872 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.873 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009090+0000", 2024-09-11T03:45:39.873 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5180000000000004e-06 2024-09-11T03:45:39.873 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.873 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.873 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.873 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009094+0000", 2024-09-11T03:45:39.873 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0620000000000001e-06 2024-09-11T03:45:39.873 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.873 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.873 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.873 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009178+0000", 2024-09-11T03:45:39.873 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4525999999999997e-05 2024-09-11T03:45:39.873 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.874 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.874 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:39.874 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009182+0000", 2024-09-11T03:45:39.874 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4640000000000002e-06 2024-09-11T03:45:39.874 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.874 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.874 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.874 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010677+0000", 2024-09-11T03:45:39.874 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014957569999999999 2024-09-11T03:45:39.874 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.874 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.874 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:39.874 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010706+0000", 2024-09-11T03:45:39.874 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8873999999999998e-05 2024-09-11T03:45:39.875 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.875 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.875 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.875 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018385+0000", 2024-09-11T03:45:39.875 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0076790490000000003 2024-09-11T03:45:39.875 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.875 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.875 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.875 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018466+0000", 2024-09-11T03:45:39.875 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0483999999999997e-05 2024-09-11T03:45:39.875 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.875 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.875 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.875 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.018545+0000", 2024-09-11T03:45:39.876 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9179999999999997e-05 2024-09-11T03:45:39.876 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.876 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.876 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.876 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.876 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.876 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:985 2.b e163/95 2:dde92fc6:::smithi17557870-41:head v 163'52, mlcod=158'48)", 2024-09-11T03:45:39.876 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:45:39.876 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 359.90658443799998, 2024-09-11T03:45:39.876 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0092288890000000005, 2024-09-11T03:45:39.876 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.876 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.876 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:39.877 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.877 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.877 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.877 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:45:39.877 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.877 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.877 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.877 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:45:39.877 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.877 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.877 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.877 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.932951+0000", 2024-09-11T03:45:39.878 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7799999999999997e-07 2024-09-11T03:45:39.878 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.878 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.878 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935287+0000", 2024-09-11T03:45:39.878 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023361599999999999 2024-09-11T03:45:39.878 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.878 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.878 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935321+0000", 2024-09-11T03:45:39.878 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3840000000000001e-05 2024-09-11T03:45:39.878 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.878 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.879 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935335+0000", 2024-09-11T03:45:39.879 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4184e-05 2024-09-11T03:45:39.879 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.879 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.879 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.879 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935428+0000", 2024-09-11T03:45:39.879 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3144000000000001e-05 2024-09-11T03:45:39.879 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.879 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.879 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.879 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.935503+0000", 2024-09-11T03:45:39.879 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4697999999999995e-05 2024-09-11T03:45:39.879 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.879 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.880 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:39.880 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.942117+0000", 2024-09-11T03:45:39.880 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0066141350000000002 2024-09-11T03:45:39.880 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.880 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.880 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.880 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.942179+0000", 2024-09-11T03:45:39.880 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.245e-05 2024-09-11T03:45:39.880 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.880 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.880 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.880 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.880 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.881 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:990 2.a 2:5014e305:::smithi17557870-46:head [write 638699~541153] snapc 24=[24,23,22,20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-11T03:45:39.881 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:45:39.881 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 359.90260318600002, 2024-09-11T03:45:39.881 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0080995760000000007, 2024-09-11T03:45:39.881 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.881 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.881 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:39.881 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:39.881 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:39.881 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:39.881 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 990 2024-09-11T03:45:39.881 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.881 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.882 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.882 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.882 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:45:39.882 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.882 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.882 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.882 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.882 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936932+0000", 2024-09-11T03:45:39.882 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.882 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.882 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.882 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.882 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.936935+0000", 2024-09-11T03:45:39.882 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7800000000000001e-06 2024-09-11T03:45:39.883 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.883 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.883 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.883 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937090+0000", 2024-09-11T03:45:39.883 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000155857 2024-09-11T03:45:39.883 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.883 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.883 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.883 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937109+0000", 2024-09-11T03:45:39.883 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8216e-05 2024-09-11T03:45:39.883 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.883 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.883 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.883 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937119+0000", 2024-09-11T03:45:39.884 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0406e-05 2024-09-11T03:45:39.884 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.884 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.884 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.884 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937195+0000", 2024-09-11T03:45:39.884 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6372999999999995e-05 2024-09-11T03:45:39.884 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.884 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.884 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.884 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937287+0000", 2024-09-11T03:45:39.884 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1434000000000006e-05 2024-09-11T03:45:39.884 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.884 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.885 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:45:39.885 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.937454+0000", 2024-09-11T03:45:39.885 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000167262 2024-09-11T03:45:39.885 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.885 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.885 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:39.885 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.943162+0000", 2024-09-11T03:45:39.885 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0057082230000000001 2024-09-11T03:45:39.885 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.885 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.885 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:39.885 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.943196+0000", 2024-09-11T03:45:39.885 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4025e-05 2024-09-11T03:45:39.886 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.886 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.886 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:39.886 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.944922+0000", 2024-09-11T03:45:39.886 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001726102 2024-09-11T03:45:39.886 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.886 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.886 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:39.886 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.944977+0000", 2024-09-11T03:45:39.886 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4849999999999998e-05 2024-09-11T03:45:39.886 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.886 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.886 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.886 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:39.945031+0000", 2024-09-11T03:45:39.887 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4048000000000002e-05 2024-09-11T03:45:39.887 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.887 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.887 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.887 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.887 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.887 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1175 2.c 2:32ade24e:::smithi17557870-45:head [copy-from ver 130] snapc 2e=[2e,2c,2b,28,26] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:45:39.887 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:45:39.887 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 337.73739529400001, 2024-09-11T03:45:39.887 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015807878000000001, 2024-09-11T03:45:39.887 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.887 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.887 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:39.887 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:39.888 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:39.888 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:39.888 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1175 2024-09-11T03:45:39.888 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.888 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.888 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.888 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.888 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:45:39.888 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.888 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.888 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.888 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.888 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:45:39.888 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.889 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.889 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.889 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.889 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102142+0000", 2024-09-11T03:45:39.889 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3369999999999998e-06 2024-09-11T03:45:39.889 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.889 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.889 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.889 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102152+0000", 2024-09-11T03:45:39.889 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8400000000000007e-06 2024-09-11T03:45:39.889 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.889 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.889 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.889 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102172+0000", 2024-09-11T03:45:39.890 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0537e-05 2024-09-11T03:45:39.890 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.890 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.890 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.890 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102183+0000", 2024-09-11T03:45:39.890 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0423e-05 2024-09-11T03:45:39.890 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.890 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.890 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.890 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102253+0000", 2024-09-11T03:45:39.890 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0528e-05 2024-09-11T03:45:39.890 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.890 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.890 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:45:39.891 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102259+0000", 2024-09-11T03:45:39.891 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1589999999999998e-06 2024-09-11T03:45:39.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.891 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.891 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103477+0000", 2024-09-11T03:45:39.891 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012174550000000001 2024-09-11T03:45:39.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.891 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.891 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103545+0000", 2024-09-11T03:45:39.891 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7595999999999995e-05 2024-09-11T03:45:39.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.892 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:45:39.892 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.110323+0000", 2024-09-11T03:45:39.892 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0067785730000000004 2024-09-11T03:45:39.892 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.892 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.892 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:39.892 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.113411+0000", 2024-09-11T03:45:39.892 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030882769999999999 2024-09-11T03:45:39.892 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.892 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.892 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:39.892 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117742+0000", 2024-09-11T03:45:39.892 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0043301920000000001 2024-09-11T03:45:39.893 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.893 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:39.893 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117835+0000", 2024-09-11T03:45:39.893 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3848e-05 2024-09-11T03:45:39.893 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.893 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:39.893 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117916+0000", 2024-09-11T03:45:39.893 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0505000000000003e-05 2024-09-11T03:45:39.893 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.893 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117948+0000", 2024-09-11T03:45:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1608e-05 2024-09-11T03:45:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1176 2.c 2:32ade24e:::smithi17557870-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:45:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:45:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 337.73733103000001, 2024-09-11T03:45:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010847406, 2024-09-11T03:45:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:45:39.895 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:39.895 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:39.895 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:39.895 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1176 2024-09-11T03:45:39.895 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.895 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.895 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.895 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.895 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:45:39.895 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.895 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.895 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.895 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.895 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:45:39.896 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.896 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.896 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.896 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.896 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:45:39.896 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4999999999999998e-07 2024-09-11T03:45:39.896 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.896 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.896 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.896 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102206+0000", 2024-09-11T03:45:39.896 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9429999999999999e-06 2024-09-11T03:45:39.896 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.896 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.897 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.897 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102214+0000", 2024-09-11T03:45:39.897 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0399999999999993e-06 2024-09-11T03:45:39.897 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.897 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.897 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.897 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102220+0000", 2024-09-11T03:45:39.897 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2989999999999996e-06 2024-09-11T03:45:39.897 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.897 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.897 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.897 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102307+0000", 2024-09-11T03:45:39.897 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7635999999999999e-05 2024-09-11T03:45:39.898 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.898 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.898 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:39.898 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102310+0000", 2024-09-11T03:45:39.898 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.2160000000000002e-06 2024-09-11T03:45:39.898 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.898 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.898 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.898 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103661+0000", 2024-09-11T03:45:39.898 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013507860000000001 2024-09-11T03:45:39.898 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.898 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.898 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:39.898 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103686+0000", 2024-09-11T03:45:39.899 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.44e-05 2024-09-11T03:45:39.899 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.899 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.899 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.899 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.112884+0000", 2024-09-11T03:45:39.899 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0091986110000000006 2024-09-11T03:45:39.899 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.899 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.899 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.899 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.112967+0000", 2024-09-11T03:45:39.899 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2478000000000006e-05 2024-09-11T03:45:39.899 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.899 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.899 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.900 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.113051+0000", 2024-09-11T03:45:39.900 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4647000000000005e-05 2024-09-11T03:45:39.900 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.900 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.900 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.900 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.900 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.900 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1864 2.c 2:3c007163:::smithi17557870-35:head [sparse-read 0~1995320] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:45:39.900 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:45:39.900 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 277.98940169600002, 2024-09-11T03:45:39.900 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0079536510000000008, 2024-09-11T03:45:39.900 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.900 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.901 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:45:39.901 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:39.901 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:39.901 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:39.901 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1864 2024-09-11T03:45:39.901 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.901 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.901 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.901 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.901 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:45:39.901 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.901 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.901 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.901 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.902 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:45:39.902 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.902 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.902 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.902 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.902 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850134+0000", 2024-09-11T03:45:39.902 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2399999999999998e-07 2024-09-11T03:45:39.902 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.902 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.902 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.902 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850137+0000", 2024-09-11T03:45:39.902 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4010000000000001e-06 2024-09-11T03:45:39.902 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.903 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.903 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850164+0000", 2024-09-11T03:45:39.903 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6616999999999999e-05 2024-09-11T03:45:39.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.903 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.903 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850171+0000", 2024-09-11T03:45:39.903 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7209999999999997e-06 2024-09-11T03:45:39.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.903 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.903 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850311+0000", 2024-09-11T03:45:39.903 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00014029900000000001 2024-09-11T03:45:39.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.904 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:39.904 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850317+0000", 2024-09-11T03:45:39.904 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7799999999999997e-06 2024-09-11T03:45:39.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.904 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.904 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.857902+0000", 2024-09-11T03:45:39.904 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0075851759999999999 2024-09-11T03:45:39.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.904 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.905 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.857978+0000", 2024-09-11T03:45:39.905 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6272000000000004e-05 2024-09-11T03:45:39.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.905 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.905 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.858087+0000", 2024-09-11T03:45:39.905 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108761 2024-09-11T03:45:39.905 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.905 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.905 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.905 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1865 2.c 2:3c007163:::smithi17557870-35:head [sparse-read 0~1995320] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:45:39.905 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:01.850224+0000", 2024-09-11T03:45:39.906 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 277.98931133899998, 2024-09-11T03:45:39.906 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.008147797, 2024-09-11T03:45:39.906 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.906 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.906 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:45:39.906 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:39.906 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:39.906 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:39.906 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1865 2024-09-11T03:45:39.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.906 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.906 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.906 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.906 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850224+0000", 2024-09-11T03:45:39.907 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.907 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.907 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850224+0000", 2024-09-11T03:45:39.907 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.907 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.907 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850224+0000", 2024-09-11T03:45:39.907 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6099999999999999e-07 2024-09-11T03:45:39.909 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:39.909 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.909 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.909 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850229+0000", 2024-09-11T03:45:39.910 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5449999999999997e-06 2024-09-11T03:45:39.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.910 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.910 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.910 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850244+0000", 2024-09-11T03:45:39.910 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5048999999999999e-05 2024-09-11T03:45:39.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.910 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.910 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.910 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850258+0000", 2024-09-11T03:45:39.910 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3505e-05 2024-09-11T03:45:39.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.910 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.911 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.911 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850344+0000", 2024-09-11T03:45:39.911 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6329999999999995e-05 2024-09-11T03:45:39.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.911 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.911 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:39.911 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.850349+0000", 2024-09-11T03:45:39.911 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0610000000000003e-06 2024-09-11T03:45:39.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.911 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.911 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.911 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.858124+0000", 2024-09-11T03:45:39.911 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.007774792 2024-09-11T03:45:39.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.912 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.912 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.858236+0000", 2024-09-11T03:45:39.912 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011264000000000001 2024-09-11T03:45:39.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.912 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.912 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:01.858371+0000", 2024-09-11T03:45:39.912 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013501400000000001 2024-09-11T03:45:39.912 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.912 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.912 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.913 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2140 2.c 2:3c007163:::smithi17557870-35:head [copy-from ver 101] snapc 57=[57,56,51,4e] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-11T03:45:39.913 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:25.064188+0000", 2024-09-11T03:45:39.913 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 254.77534687299999, 2024-09-11T03:45:39.913 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013777315, 2024-09-11T03:45:39.913 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.913 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.913 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:39.913 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:39.913 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:39.913 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:39.913 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2140 2024-09-11T03:45:39.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.913 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.914 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.914 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.914 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064188+0000", 2024-09-11T03:45:39.914 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.914 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.914 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.914 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064188+0000", 2024-09-11T03:45:39.914 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.914 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.914 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.914 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064190+0000", 2024-09-11T03:45:39.914 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9700000000000002e-06 2024-09-11T03:45:39.915 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.915 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.915 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064198+0000", 2024-09-11T03:45:39.915 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3610000000000001e-06 2024-09-11T03:45:39.915 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.915 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.915 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064216+0000", 2024-09-11T03:45:39.915 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7768999999999999e-05 2024-09-11T03:45:39.915 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.915 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.915 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064225+0000", 2024-09-11T03:45:39.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7099999999999996e-06 2024-09-11T03:45:39.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.916 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.916 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064291+0000", 2024-09-11T03:45:39.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5634000000000002e-05 2024-09-11T03:45:39.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.916 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.916 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064367+0000", 2024-09-11T03:45:39.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6687999999999995e-05 2024-09-11T03:45:39.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:45:39.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.066720+0000", 2024-09-11T03:45:39.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023526010000000002 2024-09-11T03:45:39.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:39.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.069243+0000", 2024-09-11T03:45:39.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025227320000000002 2024-09-11T03:45:39.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:39.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.073940+0000", 2024-09-11T03:45:39.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0046973930000000002 2024-09-11T03:45:39.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:39.918 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.077850+0000", 2024-09-11T03:45:39.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039098279999999997 2024-09-11T03:45:39.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:39.918 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.077915+0000", 2024-09-11T03:45:39.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5160000000000006e-05 2024-09-11T03:45:39.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.918 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.077965+0000", 2024-09-11T03:45:39.919 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0469000000000001e-05 2024-09-11T03:45:39.919 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.919 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.919 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.919 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:2421 2.18 e294/281 2:1bdf1a36:::smithi17557870-36:head v 294'198, mlcod=0'0)", 2024-09-11T03:45:39.919 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:50.964269+0000", 2024-09-11T03:45:39.919 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 228.875265974, 2024-09-11T03:45:39.919 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013043020000000001, 2024-09-11T03:45:39.919 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.919 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.919 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:39.919 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.920 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.920 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.920 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964269+0000", 2024-09-11T03:45:39.920 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.920 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.920 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.920 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.920 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964269+0000", 2024-09-11T03:45:39.920 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.920 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.920 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.920 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.920 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964270+0000", 2024-09-11T03:45:39.920 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3510000000000001e-06 2024-09-11T03:45:39.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.921 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966680+0000", 2024-09-11T03:45:39.921 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002409698 2024-09-11T03:45:39.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.921 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966720+0000", 2024-09-11T03:45:39.921 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0265999999999997e-05 2024-09-11T03:45:39.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.921 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966738+0000", 2024-09-11T03:45:39.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7540000000000001e-05 2024-09-11T03:45:39.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966835+0000", 2024-09-11T03:45:39.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7059999999999996e-05 2024-09-11T03:45:39.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966862+0000", 2024-09-11T03:45:39.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7107e-05 2024-09-11T03:45:39.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:39.923 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977265+0000", 2024-09-11T03:45:39.923 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.01040258 2024-09-11T03:45:39.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.923 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.923 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.923 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977312+0000", 2024-09-11T03:45:39.923 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7417999999999999e-05 2024-09-11T03:45:39.923 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.923 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.923 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.923 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.923 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:2443 2.12 e296/281 2:494ba932:::smithi17557870-43:head v 296'131, mlcod=291'128)", 2024-09-11T03:45:39.923 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:52.895921+0000", 2024-09-11T03:45:39.924 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 226.94361355199999, 2024-09-11T03:45:39.924 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0084055459999999999, 2024-09-11T03:45:39.924 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.924 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.924 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:39.924 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.924 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.924 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.924 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.895921+0000", 2024-09-11T03:45:39.924 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.924 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.924 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.924 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.925 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.895921+0000", 2024-09-11T03:45:39.925 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.925 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.925 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.925 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.895922+0000", 2024-09-11T03:45:39.925 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1699999999999998e-07 2024-09-11T03:45:39.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.925 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.925 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.925 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902380+0000", 2024-09-11T03:45:39.925 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0064576629999999998 2024-09-11T03:45:39.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.925 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.926 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.926 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902391+0000", 2024-09-11T03:45:39.926 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1246999999999999e-05 2024-09-11T03:45:39.926 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.926 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.926 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902398+0000", 2024-09-11T03:45:39.926 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7560000000000002e-06 2024-09-11T03:45:39.926 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.926 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.926 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902482+0000", 2024-09-11T03:45:39.926 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4881000000000003e-05 2024-09-11T03:45:39.926 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.927 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.927 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.902500+0000", 2024-09-11T03:45:39.927 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7464000000000001e-05 2024-09-11T03:45:39.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.927 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:39.927 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.904289+0000", 2024-09-11T03:45:39.927 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0017895599999999999 2024-09-11T03:45:39.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.927 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.927 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:52.904327+0000", 2024-09-11T03:45:39.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7457999999999999e-05 2024-09-11T03:45:39.928 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.928 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.928 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.928 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.928 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.928 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2972 2.c 2:32ade24e:::smithi17557870-45:head [copy-from ver 171] snapc 80=[80,7b,7a,79,77,75] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-11T03:45:39.928 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:45:39.928 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 174.76544099899999, 2024-09-11T03:45:39.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021036193000000002, 2024-09-11T03:45:39.928 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.928 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.928 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:39.928 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:39.929 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:39.929 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:39.929 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2972 2024-09-11T03:45:39.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.929 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.929 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.929 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.929 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:45:39.929 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.929 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.929 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.929 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:45:39.929 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.930 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.930 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.930 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.930 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074096+0000", 2024-09-11T03:45:39.930 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.685e-06 2024-09-11T03:45:39.930 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.930 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.930 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.930 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074104+0000", 2024-09-11T03:45:39.930 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9950000000000005e-06 2024-09-11T03:45:39.930 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.930 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.930 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.930 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074123+0000", 2024-09-11T03:45:39.931 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9587999999999999e-05 2024-09-11T03:45:39.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.931 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.931 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074132+0000", 2024-09-11T03:45:39.931 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1579999999999994e-06 2024-09-11T03:45:39.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.931 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.931 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074204+0000", 2024-09-11T03:45:39.931 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2072999999999999e-05 2024-09-11T03:45:39.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.931 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.932 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074302+0000", 2024-09-11T03:45:39.932 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7706000000000003e-05 2024-09-11T03:45:39.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.932 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:45:39.932 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.087397+0000", 2024-09-11T03:45:39.932 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013095292999999999 2024-09-11T03:45:39.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.932 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:45:39.932 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090766+0000", 2024-09-11T03:45:39.932 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0033682109999999999 2024-09-11T03:45:39.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.933 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.933 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:39.933 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.094977+0000", 2024-09-11T03:45:39.933 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0042108759999999997 2024-09-11T03:45:39.933 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.933 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.933 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:39.933 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095061+0000", 2024-09-11T03:45:39.933 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4827999999999995e-05 2024-09-11T03:45:39.933 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.933 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.933 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:45:39.933 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095099+0000", 2024-09-11T03:45:39.933 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7546999999999998e-05 2024-09-11T03:45:39.934 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.934 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.934 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095130+0000", 2024-09-11T03:45:39.934 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1232999999999997e-05 2024-09-11T03:45:39.934 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.934 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.934 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.934 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.934 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2973 2.c 2:32ade24e:::smithi17557870-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e348)", 2024-09-11T03:45:39.934 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:45:39.934 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 174.76537588100001, 2024-09-11T03:45:39.934 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016062098, 2024-09-11T03:45:39.935 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.935 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.935 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:45:39.935 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:39.935 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:39.935 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:39.935 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2973 2024-09-11T03:45:39.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.935 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.935 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.935 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:45:39.935 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.936 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:45:39.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.936 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074160+0000", 2024-09-11T03:45:39.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.46e-07 2024-09-11T03:45:39.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.936 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074162+0000", 2024-09-11T03:45:39.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5220000000000002e-06 2024-09-11T03:45:39.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074178+0000", 2024-09-11T03:45:39.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.609e-05 2024-09-11T03:45:39.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074183+0000", 2024-09-11T03:45:39.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8280000000000002e-06 2024-09-11T03:45:39.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074428+0000", 2024-09-11T03:45:39.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00024529599999999999 2024-09-11T03:45:39.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:39.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074469+0000", 2024-09-11T03:45:39.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0463000000000002e-05 2024-09-11T03:45:39.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090093+0000", 2024-09-11T03:45:39.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015624449 2024-09-11T03:45:39.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.939 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090155+0000", 2024-09-11T03:45:39.939 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1618999999999994e-05 2024-09-11T03:45:39.939 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.939 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.939 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090221+0000", 2024-09-11T03:45:39.939 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6384999999999995e-05 2024-09-11T03:45:39.939 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.939 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.939 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.939 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.939 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4365 2.c 2:327469ff:::smithi17557870-49:bc [sparse-read 0~1947401,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e473)", 2024-09-11T03:45:39.939 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:50.942868+0000", 2024-09-11T03:45:39.940 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 48.896666768999999, 2024-09-11T03:45:39.940 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010056181000000001, 2024-09-11T03:45:39.940 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.940 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.940 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:45:39.940 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:39.940 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:39.940 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:39.940 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4365 2024-09-11T03:45:39.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.940 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.940 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.940 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.942868+0000", 2024-09-11T03:45:39.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.942868+0000", 2024-09-11T03:45:39.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.942870+0000", 2024-09-11T03:45:39.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1909999999999999e-06 2024-09-11T03:45:39.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.942879+0000", 2024-09-11T03:45:39.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3070000000000003e-06 2024-09-11T03:45:39.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.942970+0000", 2024-09-11T03:45:39.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1734e-05 2024-09-11T03:45:39.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.942982+0000", 2024-09-11T03:45:39.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1984000000000001e-05 2024-09-11T03:45:39.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.943 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.943 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943388+0000", 2024-09-11T03:45:39.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00040549800000000001 2024-09-11T03:45:39.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.943 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:45:39.943 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943504+0000", 2024-09-11T03:45:39.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000116611 2024-09-11T03:45:39.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.943 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.943 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.952686+0000", 2024-09-11T03:45:39.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0091813880000000004 2024-09-11T03:45:39.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.952765+0000", 2024-09-11T03:45:39.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9109000000000004e-05 2024-09-11T03:45:39.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.952924+0000", 2024-09-11T03:45:39.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000159359 2024-09-11T03:45:39.944 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.944 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.944 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.945 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4366 2.c 2:327469ff:::smithi17557870-49:bc [sparse-read 0~1947401] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e473)", 2024-09-11T03:45:39.945 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:50.943008+0000", 2024-09-11T03:45:39.945 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 48.896526778000002, 2024-09-11T03:45:39.945 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010277684, 2024-09-11T03:45:39.945 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.945 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.945 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:45:39.945 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:39.945 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:39.945 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:39.945 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4366 2024-09-11T03:45:39.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.945 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.946 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.946 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943008+0000", 2024-09-11T03:45:39.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.946 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.946 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943008+0000", 2024-09-11T03:45:39.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.946 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.946 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943009+0000", 2024-09-11T03:45:39.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5000000000000001e-07 2024-09-11T03:45:39.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.947 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943014+0000", 2024-09-11T03:45:39.947 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8720000000000001e-06 2024-09-11T03:45:39.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.947 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943024+0000", 2024-09-11T03:45:39.947 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6250000000000002e-06 2024-09-11T03:45:39.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.947 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943030+0000", 2024-09-11T03:45:39.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.224e-06 2024-09-11T03:45:39.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943556+0000", 2024-09-11T03:45:39.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00052628699999999996 2024-09-11T03:45:39.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:45:39.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943635+0000", 2024-09-11T03:45:39.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9152000000000004e-05 2024-09-11T03:45:39.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.952995+0000", 2024-09-11T03:45:39.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.009360007 2024-09-11T03:45:39.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.953133+0000", 2024-09-11T03:45:39.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013800999999999999 2024-09-11T03:45:39.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.953286+0000", 2024-09-11T03:45:39.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000152657 2024-09-11T03:45:39.949 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.950 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.950 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.950 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4367 2.c 2:327469ff:::smithi17557870-49:bc [sparse-read 0~1947401] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e473)", 2024-09-11T03:45:39.950 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:50.943104+0000", 2024-09-11T03:45:39.950 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 48.896430840000001, 2024-09-11T03:45:39.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010538881999999999, 2024-09-11T03:45:39.950 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.950 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.950 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:45:39.950 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:39.950 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:39.950 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:39.951 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4367 2024-09-11T03:45:39.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.951 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943104+0000", 2024-09-11T03:45:39.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943104+0000", 2024-09-11T03:45:39.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.952 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.952 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943105+0000", 2024-09-11T03:45:39.952 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1900000000000002e-07 2024-09-11T03:45:39.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.952 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.952 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943109+0000", 2024-09-11T03:45:39.952 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8720000000000004e-06 2024-09-11T03:45:39.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.952 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.952 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943118+0000", 2024-09-11T03:45:39.952 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5999999999999996e-06 2024-09-11T03:45:39.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.953 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.953 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.953 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943126+0000", 2024-09-11T03:45:39.953 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2980000000000003e-06 2024-09-11T03:45:39.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.953 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.953 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.953 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943674+0000", 2024-09-11T03:45:39.953 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00054807100000000002 2024-09-11T03:45:39.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.953 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.953 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:45:39.953 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943762+0000", 2024-09-11T03:45:39.954 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8702000000000006e-05 2024-09-11T03:45:39.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.954 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.953343+0000", 2024-09-11T03:45:39.954 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0095801959999999992 2024-09-11T03:45:39.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.954 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.953457+0000", 2024-09-11T03:45:39.954 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114234 2024-09-11T03:45:39.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.955 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.953643+0000", 2024-09-11T03:45:39.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00018619 2024-09-11T03:45:39.955 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.955 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.955 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.955 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4707 2.c 2:3603e322:::smithi17557870-42:cd [read 0~3495955,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e504)", 2024-09-11T03:45:39.955 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.973854+0000", 2024-09-11T03:45:39.955 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 17.865681210999998, 2024-09-11T03:45:39.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0080577340000000004, 2024-09-11T03:45:39.955 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.955 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.955 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:45:39.956 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:39.956 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:39.956 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:39.956 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4707 2024-09-11T03:45:39.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.956 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.956 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.956 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.973854+0000", 2024-09-11T03:45:39.956 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.956 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.957 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.973854+0000", 2024-09-11T03:45:39.957 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.957 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.973854+0000", 2024-09-11T03:45:39.957 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8400000000000004e-07 2024-09-11T03:45:39.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.957 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.973858+0000", 2024-09-11T03:45:39.957 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0720000000000001e-06 2024-09-11T03:45:39.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.958 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.973864+0000", 2024-09-11T03:45:39.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5380000000000002e-06 2024-09-11T03:45:39.958 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.958 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.958 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.959 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.973868+0000", 2024-09-11T03:45:39.960 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.811e-06 2024-09-11T03:45:39.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.960 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.960 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.980261+0000", 2024-09-11T03:45:39.960 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0063937389999999998 2024-09-11T03:45:39.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.960 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.960 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.980393+0000", 2024-09-11T03:45:39.960 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013167300000000001 2024-09-11T03:45:39.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.960 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.961 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.981911+0000", 2024-09-11T03:45:39.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015183169999999999 2024-09-11T03:45:39.961 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.961 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.961 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.961 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4708 2.c 2:3603e322:::smithi17557870-42:cd [sparse-read 0~2052096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e504)", 2024-09-11T03:45:39.961 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.973893+0000", 2024-09-11T03:45:39.961 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 17.865642149999999, 2024-09-11T03:45:39.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0083488299999999998, 2024-09-11T03:45:39.961 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:45:39.961 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:45:39.961 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:45:39.961 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:45:39.962 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:45:39.962 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:39.962 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4708 2024-09-11T03:45:39.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.962 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:45:39.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.962 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:45:39.962 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.973893+0000", 2024-09-11T03:45:39.962 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.962 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:45:39.962 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.973893+0000", 2024-09-11T03:45:39.962 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:45:39.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.963 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:45:39.963 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.973893+0000", 2024-09-11T03:45:39.963 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2e-07 2024-09-11T03:45:39.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.963 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:45:39.963 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.973896+0000", 2024-09-11T03:45:39.963 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9560000000000002e-06 2024-09-11T03:45:39.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.963 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:45:39.963 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.973903+0000", 2024-09-11T03:45:39.963 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7429999999999996e-06 2024-09-11T03:45:39.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.964 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.964 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:45:39.964 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.973908+0000", 2024-09-11T03:45:39.964 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0969999999999996e-06 2024-09-11T03:45:39.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.964 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.964 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:45:39.964 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.981955+0000", 2024-09-11T03:45:39.964 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0080472170000000006 2024-09-11T03:45:39.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.964 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.964 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:45:39.964 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.982076+0000", 2024-09-11T03:45:39.965 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000120549 2024-09-11T03:45:39.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:45:39.965 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:45:39.965 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:45:39.965 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.982242+0000", 2024-09-11T03:45:39.965 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016574800000000001 2024-09-11T03:45:39.965 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.965 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.965 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.965 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:45:39.965 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:45:39.965 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:45:39.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:39.966 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:39.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:40.032 INFO:tasks.rados.rados.2.smithi175.stdout:1941: done (1 left) 2024-09-11T03:45:40.032 INFO:tasks.rados.rados.2.smithi175.stdout:1943: done (0 left) 2024-09-11T03:45:40.032 INFO:tasks.rados.rados.2.smithi175.stdout:1944: write (excl) oid 29 current snap is 214 2024-09-11T03:45:40.033 INFO:tasks.rados.rados.2.smithi175.stdout:1944: seq_num 670 ranges {460690=655894,1540112=771231,2716111=539778} 2024-09-11T03:45:40.038 INFO:tasks.rados.rados.2.smithi175.stdout:1944: writing smithi17557870-29 from 460690 to 1116584 tid 1 2024-09-11T03:45:40.044 INFO:tasks.rados.rados.2.smithi175.stdout:1944: writing smithi17557870-29 from 1540112 to 2311343 tid 2 2024-09-11T03:45:40.050 INFO:tasks.rados.rados.2.smithi175.stdout:1944: writing smithi17557870-29 from 2716111 to 3255889 tid 3 2024-09-11T03:45:40.050 INFO:tasks.rados.rados.2.smithi175.stdout:1945: write oid 31 current snap is 214 2024-09-11T03:45:40.050 INFO:tasks.rados.rados.2.smithi175.stdout:1944: finishing write tid 1 to smithi17557870-29 2024-09-11T03:45:40.050 INFO:tasks.rados.rados.2.smithi175.stdout:1944: finishing write tid 2 to smithi17557870-29 2024-09-11T03:45:40.050 INFO:tasks.rados.rados.2.smithi175.stdout:1945: seq_num 671 ranges {477497=731143,1998594=514889,3143534=133963} 2024-09-11T03:45:40.056 INFO:tasks.rados.rados.2.smithi175.stdout:1945: writing smithi17557870-31 from 477497 to 1208640 tid 1 2024-09-11T03:45:40.063 INFO:tasks.rados.rados.2.smithi175.stdout:1945: writing smithi17557870-31 from 1998594 to 2513483 tid 2 2024-09-11T03:45:40.067 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:40.069 INFO:tasks.rados.rados.2.smithi175.stdout:1945: writing smithi17557870-31 from 3143534 to 3277497 tid 3 2024-09-11T03:45:40.069 INFO:tasks.rados.rados.2.smithi175.stdout:1946: rollback oid 21 current snap is 214 2024-09-11T03:45:40.069 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 21 to 206 2024-09-11T03:45:40.069 INFO:tasks.rados.rados.2.smithi175.stdout:1944: finishing write tid 3 to smithi17557870-29 2024-09-11T03:45:40.069 INFO:tasks.rados.rados.2.smithi175.stdout:1944: finishing write tid 4 to smithi17557870-29 2024-09-11T03:45:40.069 INFO:tasks.rados.rados.2.smithi175.stdout:1944: finishing write tid 5 to smithi17557870-29 2024-09-11T03:45:40.069 INFO:tasks.rados.rados.2.smithi175.stdout:1944: oid 29 updating version 0 to 739 2024-09-11T03:45:40.069 INFO:tasks.rados.rados.2.smithi175.stdout:1944: oid 29 version 739 is already newer than 738 2024-09-11T03:45:40.069 INFO:tasks.rados.rados.2.smithi175.stdout:1944: oid 29 version 739 is already newer than 737 2024-09-11T03:45:40.070 INFO:tasks.rados.rados.2.smithi175.stdout:1944: oid 29 version 739 is already newer than 736 2024-09-11T03:45:40.070 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 739 (ObjNum 670 snap 214 seq_num 670) dirty exists 2024-09-11T03:45:40.070 INFO:tasks.rados.rados.2.smithi175.stdout:1944: left oid 29 (ObjNum 670 snap 214 seq_num 670) 2024-09-11T03:45:40.070 INFO:tasks.rados.rados.2.smithi175.stdout:1945: finishing write tid 1 to smithi17557870-31 2024-09-11T03:45:40.070 INFO:tasks.rados.rados.2.smithi175.stdout:1945: finishing write tid 2 to smithi17557870-31 2024-09-11T03:45:40.070 INFO:tasks.rados.rados.2.smithi175.stdout:1944: done (2 left) 2024-09-11T03:45:40.070 INFO:tasks.rados.rados.2.smithi175.stdout:1947: read oid 9 snap -1 2024-09-11T03:45:40.070 INFO:tasks.rados.rados.2.smithi175.stdout:1947: expect (ObjNum 664 snap 212 seq_num 664) 2024-09-11T03:45:40.070 INFO:tasks.rados.rados.2.smithi175.stdout:1948: write (excl) oid 5 current snap is 214 2024-09-11T03:45:40.070 INFO:tasks.rados.rados.2.smithi175.stdout:1948: seq_num 672 ranges {494304=406392,1540723=774900,2770957=523347} 2024-09-11T03:45:40.073 INFO:tasks.rados.rados.2.smithi175.stdout:1948: writing smithi17557870-5 from 494304 to 900696 tid 1 2024-09-11T03:45:40.081 INFO:tasks.rados.rados.2.smithi175.stdout:1948: writing smithi17557870-5 from 1540723 to 2315623 tid 2 2024-09-11T03:45:40.087 INFO:tasks.rados.rados.2.smithi175.stdout:1948: writing smithi17557870-5 from 2770957 to 3294304 tid 3 2024-09-11T03:45:40.087 INFO:tasks.rados.rados.2.smithi175.stdout:1949: write (excl) oid 24 current snap is 214 2024-09-11T03:45:40.087 INFO:tasks.rados.rados.2.smithi175.stdout:1945: finishing write tid 3 to smithi17557870-31 2024-09-11T03:45:40.087 INFO:tasks.rados.rados.2.smithi175.stdout:1949: seq_num 673 ranges {511111=481641,1599205=518558,2682027=489947} 2024-09-11T03:45:40.092 INFO:tasks.rados.rados.2.smithi175.stdout:1949: writing smithi17557870-24 from 511111 to 992752 tid 1 2024-09-11T03:45:40.093 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:40.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:40.094 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:40.098 INFO:tasks.rados.rados.2.smithi175.stdout:1949: writing smithi17557870-24 from 1599205 to 2117763 tid 2 2024-09-11T03:45:40.105 INFO:tasks.rados.rados.2.smithi175.stdout:1949: writing smithi17557870-24 from 2682027 to 3171974 tid 3 2024-09-11T03:45:40.105 INFO:tasks.rados.rados.2.smithi175.stdout:1950: copy_from oid 2 from oid 3 current snap is 214 2024-09-11T03:45:40.105 INFO:tasks.rados.rados.2.smithi175.stdout:1945: finishing write tid 4 to smithi17557870-31 2024-09-11T03:45:40.105 INFO:tasks.rados.rados.2.smithi175.stdout:1945: finishing write tid 5 to smithi17557870-31 2024-09-11T03:45:40.105 INFO:tasks.rados.rados.2.smithi175.stdout:1945: oid 31 updating version 0 to 745 2024-09-11T03:45:40.105 INFO:tasks.rados.rados.2.smithi175.stdout:1945: oid 31 updating version 745 to 747 2024-09-11T03:45:40.105 INFO:tasks.rados.rados.2.smithi175.stdout:1945: oid 31 updating version 747 to 748 2024-09-11T03:45:40.105 INFO:tasks.rados.rados.2.smithi175.stdout:1945: oid 31 version 748 is already newer than 746 2024-09-11T03:45:40.105 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 748 (ObjNum 671 snap 214 seq_num 671) dirty exists 2024-09-11T03:45:40.105 INFO:tasks.rados.rados.2.smithi175.stdout:1945: left oid 31 (ObjNum 671 snap 214 seq_num 671) 2024-09-11T03:45:40.105 INFO:tasks.rados.rados.2.smithi175.stdout:1946: finishing rollback tid 0 to smithi17557870-21 2024-09-11T03:45:40.105 INFO:tasks.rados.rados.2.smithi175.stdout:1946: finishing rollback tid 1 to smithi17557870-21 2024-09-11T03:45:40.106 INFO:tasks.rados.rados.2.smithi175.stdout:1945: done (5 left) 2024-09-11T03:45:40.106 INFO:tasks.rados.rados.2.smithi175.stdout:1951: snap_remove snap 209 2024-09-11T03:45:40.147 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:40 smithi084 ceph-mon[27254]: pgmap v936: 57 pgs: 57 active+clean; 152 MiB data, 720 MiB used, 983 GiB / 983 GiB avail; 7.6 MiB/s rd, 3.8 MiB/s wr, 21 op/s 2024-09-11T03:45:40.147 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:40 smithi084 ceph-mon[27254]: osdmap e521: 12 total, 12 up, 11 in 2024-09-11T03:45:40.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:40.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:40.217 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:40.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:40.278 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:40 smithi175 ceph-mon[30894]: pgmap v936: 57 pgs: 57 active+clean; 152 MiB data, 720 MiB used, 983 GiB / 983 GiB avail; 7.6 MiB/s rd, 3.8 MiB/s wr, 21 op/s 2024-09-11T03:45:40.279 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:40 smithi175 ceph-mon[30894]: osdmap e521: 12 total, 12 up, 11 in 2024-09-11T03:45:40.318 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:40.340 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:40.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:40.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:40.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:40 smithi121 ceph-mon[30729]: pgmap v936: 57 pgs: 57 active+clean; 152 MiB data, 720 MiB used, 983 GiB / 983 GiB avail; 7.6 MiB/s rd, 3.8 MiB/s wr, 21 op/s 2024-09-11T03:45:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:40 smithi121 ceph-mon[30729]: osdmap e521: 12 total, 12 up, 11 in 2024-09-11T03:45:40.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:40.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:40.466 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:40.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:40.567 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:40.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:40.591 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:40.591 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:40.692 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:40.714 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:40.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:40.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:40.816 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:40.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:40.838 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:40.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:40.911 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T03:45:40.939 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:40.974 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:40.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:41.031 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:45:41.031 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T03:45:41.040 INFO:tasks.rados.rados.2.smithi175.stdout:1951: done (5 left) 2024-09-11T03:45:41.040 INFO:tasks.rados.rados.2.smithi175.stdout:1952: read oid 4 snap -1 2024-09-11T03:45:41.040 INFO:tasks.rados.rados.2.smithi175.stdout:1952: expect deleted 2024-09-11T03:45:41.040 INFO:tasks.rados.rados.2.smithi175.stdout:1953: read oid 6 snap -1 2024-09-11T03:45:41.040 INFO:tasks.rados.rados.2.smithi175.stdout:1953: expect (ObjNum 659 snap 211 seq_num 659) 2024-09-11T03:45:41.040 INFO:tasks.rados.rados.2.smithi175.stdout:1954: read oid 25 snap -1 2024-09-11T03:45:41.040 INFO:tasks.rados.rados.2.smithi175.stdout:1954: expect (ObjNum 496 snap 144 seq_num 496) 2024-09-11T03:45:41.040 INFO:tasks.rados.rados.2.smithi175.stdout:1955: delete oid 10 current snap is 214 2024-09-11T03:45:41.040 INFO:tasks.rados.rados.2.smithi175.stdout:1948: finishing write tid 1 to smithi17557870-5 2024-09-11T03:45:41.040 INFO:tasks.rados.rados.2.smithi175.stdout:1946: finishing rollback tid 2 to smithi17557870-21 2024-09-11T03:45:41.041 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 752 (ObjNum 584 snap 181 seq_num 584) dirty exists 2024-09-11T03:45:41.041 INFO:tasks.rados.rados.2.smithi175.stdout:1947: expect (ObjNum 664 snap 212 seq_num 664) 2024-09-11T03:45:41.076 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:41.098 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:41.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:41.148 INFO:tasks.rados.rados.2.smithi175.stdout:1948: finishing write tid 2 to smithi17557870-5 2024-09-11T03:45:41.148 INFO:tasks.rados.rados.2.smithi175.stdout:1948: finishing write tid 3 to smithi17557870-5 2024-09-11T03:45:41.148 INFO:tasks.rados.rados.2.smithi175.stdout:1948: finishing write tid 4 to smithi17557870-5 2024-09-11T03:45:41.148 INFO:tasks.rados.rados.2.smithi175.stdout:1948: finishing write tid 5 to smithi17557870-5 2024-09-11T03:45:41.148 INFO:tasks.rados.rados.2.smithi175.stdout:1948: oid 5 updating version 0 to 683 2024-09-11T03:45:41.148 INFO:tasks.rados.rados.2.smithi175.stdout:1948: oid 5 updating version 683 to 686 2024-09-11T03:45:41.148 INFO:tasks.rados.rados.2.smithi175.stdout:1948: oid 5 version 686 is already newer than 685 2024-09-11T03:45:41.148 INFO:tasks.rados.rados.2.smithi175.stdout:1948: oid 5 version 686 is already newer than 684 2024-09-11T03:45:41.148 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 686 (ObjNum 672 snap 214 seq_num 672) dirty exists 2024-09-11T03:45:41.149 INFO:tasks.rados.rados.2.smithi175.stdout:1948: left oid 5 (ObjNum 672 snap 214 seq_num 672) 2024-09-11T03:45:41.149 INFO:tasks.rados.rados.2.smithi175.stdout:1949: finishing write tid 1 to smithi17557870-24 2024-09-11T03:45:41.149 INFO:tasks.rados.rados.2.smithi175.stdout:1949: finishing write tid 2 to smithi17557870-24 2024-09-11T03:45:41.149 INFO:tasks.rados.rados.2.smithi175.stdout:1949: finishing write tid 3 to smithi17557870-24 2024-09-11T03:45:41.149 INFO:tasks.rados.rados.2.smithi175.stdout:1949: finishing write tid 4 to smithi17557870-24 2024-09-11T03:45:41.149 INFO:tasks.rados.rados.2.smithi175.stdout:1949: finishing write tid 5 to smithi17557870-24 2024-09-11T03:45:41.149 INFO:tasks.rados.rados.2.smithi175.stdout:1949: oid 24 updating version 0 to 758 2024-09-11T03:45:41.149 INFO:tasks.rados.rados.2.smithi175.stdout:1949: oid 24 version 758 is already newer than 757 2024-09-11T03:45:41.149 INFO:tasks.rados.rados.2.smithi175.stdout:1949: oid 24 updating version 758 to 760 2024-09-11T03:45:41.149 INFO:tasks.rados.rados.2.smithi175.stdout:1949: oid 24 version 760 is already newer than 759 2024-09-11T03:45:41.149 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 760 (ObjNum 673 snap 214 seq_num 673) dirty exists 2024-09-11T03:45:41.149 INFO:tasks.rados.rados.2.smithi175.stdout:1949: left oid 24 (ObjNum 673 snap 214 seq_num 673) 2024-09-11T03:45:41.149 INFO:tasks.rados.rados.2.smithi175.stdout:1950: finishing copy_from racing read to smithi17557870-2 2024-09-11T03:45:41.149 INFO:tasks.rados.rados.2.smithi175.stdout:1950: finishing copy_from to smithi17557870-2 2024-09-11T03:45:41.150 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 761 (ObjNum 663 snap 212 seq_num 663) dirty exists 2024-09-11T03:45:41.150 INFO:tasks.rados.rados.2.smithi175.stdout:1946: done (8 left) 2024-09-11T03:45:41.150 INFO:tasks.rados.rados.2.smithi175.stdout:1947: done (7 left) 2024-09-11T03:45:41.150 INFO:tasks.rados.rados.2.smithi175.stdout:1948: done (6 left) 2024-09-11T03:45:41.150 INFO:tasks.rados.rados.2.smithi175.stdout:1949: done (5 left) 2024-09-11T03:45:41.150 INFO:tasks.rados.rados.2.smithi175.stdout:1950: done (4 left) 2024-09-11T03:45:41.150 INFO:tasks.rados.rados.2.smithi175.stdout:1955: done (3 left) 2024-09-11T03:45:41.150 INFO:tasks.rados.rados.2.smithi175.stdout:1956: read oid 46 snap -1 2024-09-11T03:45:41.150 INFO:tasks.rados.rados.2.smithi175.stdout:1956: expect (ObjNum 653 snap 209 seq_num 653) 2024-09-11T03:45:41.150 INFO:tasks.rados.rados.2.smithi175.stdout:1957: snap_remove snap 213 2024-09-11T03:45:41.199 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:41.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:41.221 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:41.222 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:41.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:41 smithi175 ceph-mon[30894]: osdmap e522: 12 total, 12 up, 11 in 2024-09-11T03:45:41.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:41 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1008229043' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:45:41.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:41 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/93169038' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:45:41.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:41.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:45:41.348 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:41.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:41.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:41 smithi084 ceph-mon[27254]: osdmap e522: 12 total, 12 up, 11 in 2024-09-11T03:45:41.358 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:41 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1008229043' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:45:41.358 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:41 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/93169038' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:45:41.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:41 smithi121 ceph-mon[30729]: osdmap e522: 12 total, 12 up, 11 in 2024-09-11T03:45:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:41 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1008229043' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:45:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:41 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/93169038' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:45:41.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:41.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:41.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:41.577 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:41.602 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:41.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:41.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:41.704 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:41.726 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:41.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:41.780 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-11T03:45:41.828 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:41.853 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:41.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:41.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:41.951 INFO:tasks.rados.rados.2.smithi175.stdout:1957: done (4 left) 2024-09-11T03:45:41.951 INFO:tasks.rados.rados.2.smithi175.stdout:1958: snap_create 2024-09-11T03:45:41.952 INFO:tasks.rados.rados.2.smithi175.stdout:1954: expect (ObjNum 496 snap 144 seq_num 496) 2024-09-11T03:45:41.955 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:41.979 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:41.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:41.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:42.033 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:42 smithi084 ceph-mon[27254]: pgmap v939: 57 pgs: 57 active+clean; 152 MiB data, 724 MiB used, 983 GiB / 983 GiB avail; 6.0 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2024-09-11T03:45:42.033 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:42 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1008229043' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:45:42.033 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:42 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/93169038' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:45:42.033 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:42 smithi084 ceph-mon[27254]: osdmap e523: 12 total, 12 up, 11 in 2024-09-11T03:45:42.033 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:42 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:45:42.033 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:42 smithi084 ceph-mon[27254]: 2.1 scrub ok 2024-09-11T03:45:42.033 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:42 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:42.034 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:42 smithi084 ceph-mon[27254]: osdmap e524: 12 total, 12 up, 11 in 2024-09-11T03:45:42.058 INFO:tasks.rados.rados.2.smithi175.stdout:1953: expect (ObjNum 659 snap 211 seq_num 659) 2024-09-11T03:45:42.081 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:42.102 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:42.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:42.103 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:42.204 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:42.225 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:42.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:42.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:42.273 INFO:tasks.rados.rados.2.smithi175.stdout:1956: expect (ObjNum 653 snap 209 seq_num 653) 2024-09-11T03:45:42.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:42.348 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:42.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:42.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:42 smithi121 ceph-mon[30729]: pgmap v939: 57 pgs: 57 active+clean; 152 MiB data, 724 MiB used, 983 GiB / 983 GiB avail; 6.0 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2024-09-11T03:45:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:42 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1008229043' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:45:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:42 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/93169038' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:45:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:42 smithi121 ceph-mon[30729]: osdmap e523: 12 total, 12 up, 11 in 2024-09-11T03:45:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:42 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:45:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:42 smithi121 ceph-mon[30729]: 2.1 scrub ok 2024-09-11T03:45:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:42 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:42 smithi121 ceph-mon[30729]: osdmap e524: 12 total, 12 up, 11 in 2024-09-11T03:45:42.451 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:42.472 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:42.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:42.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:42 smithi175 ceph-mon[30894]: pgmap v939: 57 pgs: 57 active+clean; 152 MiB data, 724 MiB used, 983 GiB / 983 GiB avail; 6.0 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2024-09-11T03:45:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:42 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1008229043' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:45:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:42 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/93169038' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:45:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:42 smithi175 ceph-mon[30894]: osdmap e523: 12 total, 12 up, 11 in 2024-09-11T03:45:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:42 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:45:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:42 smithi175 ceph-mon[30894]: 2.1 scrub ok 2024-09-11T03:45:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:42 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:42 smithi175 ceph-mon[30894]: osdmap e524: 12 total, 12 up, 11 in 2024-09-11T03:45:42.513 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:42.574 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:42.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:42.596 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:42.596 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:42.697 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:42.718 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:42.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:42.719 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:42.820 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:42.846 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:42.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:42.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:42.948 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:42.950 INFO:tasks.rados.rados.2.smithi175.stdout:1952: done (4 left) 2024-09-11T03:45:42.950 INFO:tasks.rados.rados.2.smithi175.stdout:1953: done (3 left) 2024-09-11T03:45:42.950 INFO:tasks.rados.rados.2.smithi175.stdout:1954: done (2 left) 2024-09-11T03:45:42.950 INFO:tasks.rados.rados.2.smithi175.stdout:1956: done (1 left) 2024-09-11T03:45:42.950 INFO:tasks.rados.rados.2.smithi175.stdout:1958: done (0 left) 2024-09-11T03:45:42.950 INFO:tasks.rados.rados.2.smithi175.stdout:1959: read oid 4 snap -1 2024-09-11T03:45:42.950 INFO:tasks.rados.rados.2.smithi175.stdout:1959: expect deleted 2024-09-11T03:45:42.951 INFO:tasks.rados.rados.2.smithi175.stdout:1960: read oid 18 snap -1 2024-09-11T03:45:42.951 INFO:tasks.rados.rados.2.smithi175.stdout:1960: expect deleted 2024-09-11T03:45:42.951 INFO:tasks.rados.rados.2.smithi175.stdout:1961: snap_create 2024-09-11T03:45:42.970 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:42.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:42.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:43.015 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:45:43.015 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:45:43.015 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:45:43.015 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:45:43.016 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:43.072 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:43.093 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:43.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:43.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:43.195 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:43.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:43.217 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:43.217 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:43.318 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:43.339 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:43.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:43.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:43.441 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:43.462 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:43.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:43.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:43.564 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:43.590 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:43.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:43.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:43.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:43.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:43.717 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:43.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:43.819 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:43.840 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:43.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:43.942 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:43.948 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2024-09-11T03:45:43.952 INFO:tasks.rados.rados.2.smithi175.stdout:1959: done (2 left) 2024-09-11T03:45:43.953 INFO:tasks.rados.rados.2.smithi175.stdout:1960: done (1 left) 2024-09-11T03:45:43.953 INFO:tasks.rados.rados.2.smithi175.stdout:1961: done (0 left) 2024-09-11T03:45:43.953 INFO:tasks.rados.rados.2.smithi175.stdout:1962: read oid 25 snap -1 2024-09-11T03:45:43.953 INFO:tasks.rados.rados.2.smithi175.stdout:1962: expect (ObjNum 496 snap 144 seq_num 496) 2024-09-11T03:45:43.953 INFO:tasks.rados.rados.2.smithi175.stdout:1963: read oid 21 snap -1 2024-09-11T03:45:43.953 INFO:tasks.rados.rados.2.smithi175.stdout:1963: expect (ObjNum 584 snap 181 seq_num 584) 2024-09-11T03:45:43.953 INFO:tasks.rados.rados.2.smithi175.stdout:1964: delete oid 35 current snap is 216 2024-09-11T03:45:43.956 INFO:tasks.rados.rados.2.smithi175.stdout:1964: done (2 left) 2024-09-11T03:45:43.956 INFO:tasks.rados.rados.2.smithi175.stdout:1965: read oid 2 snap 211 2024-09-11T03:45:43.956 INFO:tasks.rados.rados.2.smithi175.stdout:1965: expect deleted 2024-09-11T03:45:43.956 INFO:tasks.rados.rados.2.smithi175.stdout:1966: snap_create 2024-09-11T03:45:43.960 INFO:tasks.rados.rados.2.smithi175.stdout:1962: expect (ObjNum 496 snap 144 seq_num 496) 2024-09-11T03:45:43.968 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:43.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:43.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:44.067 INFO:tasks.rados.rados.2.smithi175.stdout:1963: expect (ObjNum 584 snap 181 seq_num 584) 2024-09-11T03:45:44.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:44.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:44.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:44.198 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:44.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:44.221 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:44.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:43 smithi175 ceph-mon[30894]: 2.f scrub starts 2024-09-11T03:45:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:43 smithi175 ceph-mon[30894]: 2.f scrub ok 2024-09-11T03:45:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:43 smithi175 ceph-mon[30894]: pgmap v942: 57 pgs: 57 active+clean; 164 MiB data, 749 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 2.5 MiB/s wr, 20 op/s 2024-09-11T03:45:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:43 smithi175 ceph-mon[30894]: osdmap e525: 12 total, 12 up, 11 in 2024-09-11T03:45:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:43 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1399692864' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:45:44.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:43 smithi084 ceph-mon[27254]: 2.f scrub starts 2024-09-11T03:45:44.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:43 smithi084 ceph-mon[27254]: 2.f scrub ok 2024-09-11T03:45:44.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:43 smithi084 ceph-mon[27254]: pgmap v942: 57 pgs: 57 active+clean; 164 MiB data, 749 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 2.5 MiB/s wr, 20 op/s 2024-09-11T03:45:44.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:43 smithi084 ceph-mon[27254]: osdmap e525: 12 total, 12 up, 11 in 2024-09-11T03:45:44.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:43 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1399692864' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:45:44.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:44.337 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T03:45:44.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:44.349 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:44.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:43 smithi121 ceph-mon[30729]: 2.f scrub starts 2024-09-11T03:45:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:43 smithi121 ceph-mon[30729]: 2.f scrub ok 2024-09-11T03:45:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:43 smithi121 ceph-mon[30729]: pgmap v942: 57 pgs: 57 active+clean; 164 MiB data, 749 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 2.5 MiB/s wr, 20 op/s 2024-09-11T03:45:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:43 smithi121 ceph-mon[30729]: osdmap e525: 12 total, 12 up, 11 in 2024-09-11T03:45:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:43 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1399692864' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:45:44.450 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:44.471 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:44.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:44.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:44.507 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:45:44.574 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:44.575 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:44.597 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:44.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:44.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:44.699 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:44.720 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:44.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:44.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:44.822 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:44.846 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:44.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:44.847 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:44.948 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:44.965 INFO:tasks.rados.rados.2.smithi175.stdout:1962: done (3 left) 2024-09-11T03:45:44.965 INFO:tasks.rados.rados.2.smithi175.stdout:1963: done (2 left) 2024-09-11T03:45:44.965 INFO:tasks.rados.rados.2.smithi175.stdout:1965: done (1 left) 2024-09-11T03:45:44.965 INFO:tasks.rados.rados.2.smithi175.stdout:1966: done (0 left) 2024-09-11T03:45:44.965 INFO:tasks.rados.rados.2.smithi175.stdout:1967: snap_create 2024-09-11T03:45:44.970 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:44.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:44.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:45.072 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:45.096 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:45.097 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:45.198 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:45.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:45.221 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:45.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:45.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:44 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1399692864' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:45:45.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:44 smithi175 ceph-mon[30894]: osdmap e526: 12 total, 12 up, 11 in 2024-09-11T03:45:45.322 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:45.345 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:45.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:45.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:44 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1399692864' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:45:45.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:44 smithi084 ceph-mon[27254]: osdmap e526: 12 total, 12 up, 11 in 2024-09-11T03:45:45.377 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 8 1 2024-09-11T03:45:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:44 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1399692864' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:45:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:44 smithi121 ceph-mon[30729]: osdmap e526: 12 total, 12 up, 11 in 2024-09-11T03:45:45.447 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:45.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:45.469 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:45.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:45.571 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:45.592 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:45.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:45.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:45.653 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:45:45.653 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:45:45.657 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:45.671 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:45:45.671 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:45:45.694 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:45.696 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:45:45.696 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:45:45.712 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:45:45.712 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:45:45.719 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:45.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:45.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:45.729 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:45:45.729 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:45:45.745 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:45:45.746 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:45:45.763 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:45:45.763 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:45:45.779 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:45:45.780 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:45:45.797 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:45:45.797 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:45:45.814 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:45:45.814 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:45:45.821 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:45.832 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:45:45.832 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:45:45.847 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:45.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:45.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:45.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:45:45.849 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:45:45.948 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:45.985 INFO:tasks.rados.rados.2.smithi175.stdout:1967: done (0 left) 2024-09-11T03:45:45.985 INFO:tasks.rados.rados.2.smithi175.stdout:1968: snap_remove snap 208 2024-09-11T03:45:46.000 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:46.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:46.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:46.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:46.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:46.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:46.208 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:45 smithi084 ceph-mon[27254]: pgmap v945: 57 pgs: 57 active+clean; 169 MiB data, 764 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 3.3 MiB/s wr, 24 op/s 2024-09-11T03:45:46.208 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:45 smithi084 ceph-mon[27254]: osdmap e527: 12 total, 12 up, 11 in 2024-09-11T03:45:46.208 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:45 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3901098271' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:45 smithi175 ceph-mon[30894]: pgmap v945: 57 pgs: 57 active+clean; 169 MiB data, 764 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 3.3 MiB/s wr, 24 op/s 2024-09-11T03:45:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:45 smithi175 ceph-mon[30894]: osdmap e527: 12 total, 12 up, 11 in 2024-09-11T03:45:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:45 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3901098271' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:46.231 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:46.257 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:46.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:46.258 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:46.359 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:46.383 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:46.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:46.384 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:46.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:45 smithi121 ceph-mon[30729]: pgmap v945: 57 pgs: 57 active+clean; 169 MiB data, 764 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 3.3 MiB/s wr, 24 op/s 2024-09-11T03:45:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:45 smithi121 ceph-mon[30729]: osdmap e527: 12 total, 12 up, 11 in 2024-09-11T03:45:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:45 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3901098271' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:46.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:46.509 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:46.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:46.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:46.611 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:46.636 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:46.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:46.637 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:46.738 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:46.763 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:46.764 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:46.802 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:45:46.802 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:45:46.865 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:46.887 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:46.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:46.954 INFO:tasks.rados.rados.2.smithi175.stdout:1968: done (0 left) 2024-09-11T03:45:46.954 INFO:tasks.rados.rados.2.smithi175.stdout:1969: read oid 22 snap 210 2024-09-11T03:45:46.954 INFO:tasks.rados.rados.2.smithi175.stdout:1969: expect (ObjNum 495 snap 143 seq_num 495) 2024-09-11T03:45:46.954 INFO:tasks.rados.rados.2.smithi175.stdout:1970: rollback oid 5 current snap is 218 2024-09-11T03:45:46.954 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 5 to 200 2024-09-11T03:45:46.954 INFO:tasks.rados.rados.2.smithi175.stdout:1971: copy_from oid 37 from oid 24 current snap is 218 2024-09-11T03:45:46.954 INFO:tasks.rados.rados.2.smithi175.stdout:1972: snap_remove snap 215 2024-09-11T03:45:46.989 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:47.010 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:47.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:47.111 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:47.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:47.136 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:47.136 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:47.237 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:47.260 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:47.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:47.261 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:47.362 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:47.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:47.385 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:47.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:47.389 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:46 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:45:47.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:46 smithi084 ceph-mon[27254]: 2.7 scrub ok 2024-09-11T03:45:47.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:46 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:45:47.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:46 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:45:47.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:46 smithi084 ceph-mon[27254]: osdmap e528: 12 total, 12 up, 11 in 2024-09-11T03:45:47.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:46 smithi084 ceph-mon[27254]: osdmap e529: 12 total, 12 up, 11 in 2024-09-11T03:45:47.404 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:47.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:47 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:45:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:47 smithi121 ceph-mon[30729]: 2.7 scrub ok 2024-09-11T03:45:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:47 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:45:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:47 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:45:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:47 smithi121 ceph-mon[30729]: osdmap e528: 12 total, 12 up, 11 in 2024-09-11T03:45:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:47 smithi121 ceph-mon[30729]: osdmap e529: 12 total, 12 up, 11 in 2024-09-11T03:45:47.426 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T03:45:47.439 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:46 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:45:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:46 smithi175 ceph-mon[30894]: 2.7 scrub ok 2024-09-11T03:45:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:46 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:45:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:46 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:45:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:46 smithi175 ceph-mon[30894]: osdmap e528: 12 total, 12 up, 11 in 2024-09-11T03:45:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:46 smithi175 ceph-mon[30894]: osdmap e529: 12 total, 12 up, 11 in 2024-09-11T03:45:47.486 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:47.512 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:47.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:47.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:47.614 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:47.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:47.637 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:47.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:47.738 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:47.766 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:47.766 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:47.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:47.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:47.889 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:47.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:47.961 INFO:tasks.rados.rados.2.smithi175.stdout:1972: done (3 left) 2024-09-11T03:45:47.961 INFO:tasks.rados.rados.2.smithi175.stdout:1973: write oid 1 current snap is 218 2024-09-11T03:45:47.962 INFO:tasks.rados.rados.2.smithi175.stdout:1970: finishing rollback tid 0 to smithi17557870-5 2024-09-11T03:45:47.962 INFO:tasks.rados.rados.2.smithi175.stdout:1973: seq_num 676 ranges {561532=707388,2058298=782238} 2024-09-11T03:45:47.967 INFO:tasks.rados.rados.2.smithi175.stdout:1973: writing smithi17557870-1 from 561532 to 1268920 tid 1 2024-09-11T03:45:47.976 INFO:tasks.rados.rados.2.smithi175.stdout:1973: writing smithi17557870-1 from 2058298 to 2840536 tid 2 2024-09-11T03:45:47.977 INFO:tasks.rados.rados.2.smithi175.stdout:1974: write (excl) oid 16 current snap is 218 2024-09-11T03:45:47.977 INFO:tasks.rados.rados.2.smithi175.stdout:1970: finishing rollback tid 1 to smithi17557870-5 2024-09-11T03:45:47.977 INFO:tasks.rados.rados.2.smithi175.stdout:1974: seq_num 677 ranges {578339=498990,1833133=525896,2791719=586620} 2024-09-11T03:45:47.982 INFO:tasks.rados.rados.2.smithi175.stdout:1974: writing smithi17557870-16 from 578339 to 1077329 tid 1 2024-09-11T03:45:47.989 INFO:tasks.rados.rados.2.smithi175.stdout:1974: writing smithi17557870-16 from 1833133 to 2359029 tid 2 2024-09-11T03:45:47.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:47.995 INFO:tasks.rados.rados.2.smithi175.stdout:1974: writing smithi17557870-16 from 2791719 to 3378339 tid 3 2024-09-11T03:45:47.996 INFO:tasks.rados.rados.2.smithi175.stdout:1975: read oid 50 snap -1 2024-09-11T03:45:47.996 INFO:tasks.rados.rados.2.smithi175.stdout:1975: expect deleted 2024-09-11T03:45:47.996 INFO:tasks.rados.rados.2.smithi175.stdout:1976: delete oid 48 current snap is 218 2024-09-11T03:45:47.996 INFO:tasks.rados.rados.2.smithi175.stdout:1970: finishing rollback tid 2 to smithi17557870-5 2024-09-11T03:45:47.996 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 689 (ObjNum 615 snap 199 seq_num 615) dirty exists 2024-09-11T03:45:47.996 INFO:tasks.rados.rados.2.smithi175.stdout:1969: expect (ObjNum 495 snap 143 seq_num 495) 2024-09-11T03:45:48.015 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:48.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:48.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:48.072 INFO:tasks.rados.rados.2.smithi175.stdout:1971: finishing copy_from racing read to smithi17557870-37 2024-09-11T03:45:48.072 INFO:tasks.rados.rados.2.smithi175.stdout:1971: finishing copy_from to smithi17557870-37 2024-09-11T03:45:48.072 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 760 (ObjNum 673 snap 214 seq_num 673) dirty exists 2024-09-11T03:45:48.072 INFO:tasks.rados.rados.2.smithi175.stdout:1973: finishing write tid 1 to smithi17557870-1 2024-09-11T03:45:48.072 INFO:tasks.rados.rados.2.smithi175.stdout:1973: finishing write tid 2 to smithi17557870-1 2024-09-11T03:45:48.072 INFO:tasks.rados.rados.2.smithi175.stdout:1973: finishing write tid 3 to smithi17557870-1 2024-09-11T03:45:48.072 INFO:tasks.rados.rados.2.smithi175.stdout:1973: finishing write tid 4 to smithi17557870-1 2024-09-11T03:45:48.072 INFO:tasks.rados.rados.2.smithi175.stdout:1973: oid 1 updating version 0 to 745 2024-09-11T03:45:48.072 INFO:tasks.rados.rados.2.smithi175.stdout:1973: oid 1 version 745 is already newer than 744 2024-09-11T03:45:48.073 INFO:tasks.rados.rados.2.smithi175.stdout:1973: oid 1 updating version 745 to 746 2024-09-11T03:45:48.073 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 746 (ObjNum 676 snap 218 seq_num 676) dirty exists 2024-09-11T03:45:48.073 INFO:tasks.rados.rados.2.smithi175.stdout:1973: left oid 1 (ObjNum 676 snap 218 seq_num 676) 2024-09-11T03:45:48.073 INFO:tasks.rados.rados.2.smithi175.stdout:1974: finishing write tid 1 to smithi17557870-16 2024-09-11T03:45:48.073 INFO:tasks.rados.rados.2.smithi175.stdout:1974: finishing write tid 2 to smithi17557870-16 2024-09-11T03:45:48.073 INFO:tasks.rados.rados.2.smithi175.stdout:1974: finishing write tid 3 to smithi17557870-16 2024-09-11T03:45:48.073 INFO:tasks.rados.rados.2.smithi175.stdout:1974: finishing write tid 4 to smithi17557870-16 2024-09-11T03:45:48.073 INFO:tasks.rados.rados.2.smithi175.stdout:1974: finishing write tid 5 to smithi17557870-16 2024-09-11T03:45:48.073 INFO:tasks.rados.rados.2.smithi175.stdout:1974: oid 16 updating version 0 to 501 2024-09-11T03:45:48.073 INFO:tasks.rados.rados.2.smithi175.stdout:1974: oid 16 version 501 is already newer than 500 2024-09-11T03:45:48.073 INFO:tasks.rados.rados.2.smithi175.stdout:1974: oid 16 updating version 501 to 502 2024-09-11T03:45:48.073 INFO:tasks.rados.rados.2.smithi175.stdout:1974: oid 16 version 502 is already newer than 499 2024-09-11T03:45:48.074 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 502 (ObjNum 677 snap 218 seq_num 677) dirty exists 2024-09-11T03:45:48.074 INFO:tasks.rados.rados.2.smithi175.stdout:1974: left oid 16 (ObjNum 677 snap 218 seq_num 677) 2024-09-11T03:45:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:1969: done (6 left) 2024-09-11T03:45:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:1970: done (5 left) 2024-09-11T03:45:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:1971: done (4 left) 2024-09-11T03:45:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:1973: done (3 left) 2024-09-11T03:45:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:1974: done (2 left) 2024-09-11T03:45:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:1975: done (1 left) 2024-09-11T03:45:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:1976: done (0 left) 2024-09-11T03:45:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:1977: rollback oid 7 current snap is 218 2024-09-11T03:45:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 7 to 207 2024-09-11T03:45:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:1978: copy_from oid 44 from oid 22 current snap is 218 2024-09-11T03:45:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:1979: write (excl) oid 3 current snap is 218 2024-09-11T03:45:48.075 INFO:tasks.rados.rados.2.smithi175.stdout:1979: seq_num 679 ranges {611953=649488,1950097=529565,3246565=165388} 2024-09-11T03:45:48.080 INFO:tasks.rados.rados.2.smithi175.stdout:1979: writing smithi17557870-3 from 611953 to 1261441 tid 1 2024-09-11T03:45:48.086 INFO:tasks.rados.rados.2.smithi175.stdout:1979: writing smithi17557870-3 from 1950097 to 2479662 tid 2 2024-09-11T03:45:48.092 INFO:tasks.rados.rados.2.smithi175.stdout:1979: writing smithi17557870-3 from 3246565 to 3411953 tid 3 2024-09-11T03:45:48.092 INFO:tasks.rados.rados.2.smithi175.stdout:1980: write (excl) oid 2 current snap is 218 2024-09-11T03:45:48.092 INFO:tasks.rados.rados.2.smithi175.stdout:1978: finishing copy_from to smithi17557870-44 2024-09-11T03:45:48.092 INFO:tasks.rados.rados.2.smithi175.stdout:1978: got expected ENOENT (src dne) 2024-09-11T03:45:48.092 INFO:tasks.rados.rados.2.smithi175.stdout:1978: finishing copy_from racing read to smithi17557870-44 2024-09-11T03:45:48.092 INFO:tasks.rados.rados.2.smithi175.stdout:1977: finishing rollback tid 0 to smithi17557870-7 2024-09-11T03:45:48.092 INFO:tasks.rados.rados.2.smithi175.stdout:1977: finishing rollback tid 1 to smithi17557870-7 2024-09-11T03:45:48.093 INFO:tasks.rados.rados.2.smithi175.stdout:1980: seq_num 680 ranges {628760=724737,1892226=673223,3157635=271125} 2024-09-11T03:45:48.099 INFO:tasks.rados.rados.2.smithi175.stdout:1980: writing smithi17557870-2 from 628760 to 1353497 tid 1 2024-09-11T03:45:48.105 INFO:tasks.rados.rados.2.smithi175.stdout:1980: writing smithi17557870-2 from 1892226 to 2565449 tid 2 2024-09-11T03:45:48.110 INFO:tasks.rados.rados.2.smithi175.stdout:1980: writing smithi17557870-2 from 3157635 to 3428760 tid 3 2024-09-11T03:45:48.110 INFO:tasks.rados.rados.2.smithi175.stdout:1978: done (3 left) 2024-09-11T03:45:48.111 INFO:tasks.rados.rados.2.smithi175.stdout:1981: snap_remove snap 214 2024-09-11T03:45:48.117 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:48.138 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:48.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:48.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:48.240 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:48.261 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:48.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:48.262 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:48.263 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:47 smithi084 ceph-mon[27254]: pgmap v948: 57 pgs: 57 active+clean; 169 MiB data, 772 MiB used, 983 GiB / 983 GiB avail; 3.3 MiB/s rd, 781 KiB/s wr, 7 op/s 2024-09-11T03:45:48.263 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:47 smithi084 ceph-mon[27254]: osdmap e530: 12 total, 12 up, 11 in 2024-09-11T03:45:48.343 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:48.362 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:48.384 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:48.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:48.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:48.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:47 smithi121 ceph-mon[30729]: pgmap v948: 57 pgs: 57 active+clean; 169 MiB data, 772 MiB used, 983 GiB / 983 GiB avail; 3.3 MiB/s rd, 781 KiB/s wr, 7 op/s 2024-09-11T03:45:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:47 smithi121 ceph-mon[30729]: osdmap e530: 12 total, 12 up, 11 in 2024-09-11T03:45:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:48 smithi175 ceph-mon[30894]: pgmap v948: 57 pgs: 57 active+clean; 169 MiB data, 772 MiB used, 983 GiB / 983 GiB avail; 3.3 MiB/s rd, 781 KiB/s wr, 7 op/s 2024-09-11T03:45:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:48 smithi175 ceph-mon[30894]: osdmap e530: 12 total, 12 up, 11 in 2024-09-11T03:45:48.487 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:48.508 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:48.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:48.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:48.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:45:48.610 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:48.633 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:48.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:48.634 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:48.735 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:48.757 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:48.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:48.758 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:48.833 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:45:48.833 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:45:48.859 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:48.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:48.882 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:48.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:48.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:49.004 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.8 to 1 (10000) 2024-09-11T03:45:49.004 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:45:49.008 INFO:tasks.rados.rados.2.smithi175.stdout:1981: done (3 left) 2024-09-11T03:45:49.008 INFO:tasks.rados.rados.2.smithi175.stdout:1982: write (excl) oid 25 current snap is 218 2024-09-11T03:45:49.008 INFO:tasks.rados.rados.2.smithi175.stdout:1977: finishing rollback tid 2 to smithi17557870-7 2024-09-11T03:45:49.008 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 519 (ObjNum 552 snap 171 seq_num 552) dirty exists 2024-09-11T03:45:49.008 INFO:tasks.rados.rados.2.smithi175.stdout:1979: finishing write tid 1 to smithi17557870-3 2024-09-11T03:45:49.008 INFO:tasks.rados.rados.2.smithi175.stdout:1979: finishing write tid 2 to smithi17557870-3 2024-09-11T03:45:49.009 INFO:tasks.rados.rados.2.smithi175.stdout:1979: finishing write tid 3 to smithi17557870-3 2024-09-11T03:45:49.009 INFO:tasks.rados.rados.2.smithi175.stdout:1982: seq_num 681 ranges {645567=799986,1950708=533234,3185058=260509} 2024-09-11T03:45:49.010 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:49.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:49.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:49.015 INFO:tasks.rados.rados.2.smithi175.stdout:1982: writing smithi17557870-25 from 645567 to 1445553 tid 1 2024-09-11T03:45:49.022 INFO:tasks.rados.rados.2.smithi175.stdout:1982: writing smithi17557870-25 from 1950708 to 2483942 tid 2 2024-09-11T03:45:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:1982: writing smithi17557870-25 from 3185058 to 3445567 tid 3 2024-09-11T03:45:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:1977: done (3 left) 2024-09-11T03:45:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:1983: copy_from oid 34 from oid 5 current snap is 218 2024-09-11T03:45:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:1979: finishing write tid 4 to smithi17557870-3 2024-09-11T03:45:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:1979: finishing write tid 5 to smithi17557870-3 2024-09-11T03:45:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:1979: oid 3 updating version 0 to 771 2024-09-11T03:45:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:1979: oid 3 updating version 771 to 773 2024-09-11T03:45:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:1979: oid 3 updating version 773 to 774 2024-09-11T03:45:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:1979: oid 3 version 774 is already newer than 772 2024-09-11T03:45:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 774 (ObjNum 679 snap 218 seq_num 679) dirty exists 2024-09-11T03:45:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:1979: left oid 3 (ObjNum 679 snap 218 seq_num 679) 2024-09-11T03:45:49.030 INFO:tasks.rados.rados.2.smithi175.stdout:1980: finishing write tid 1 to smithi17557870-2 2024-09-11T03:45:49.030 INFO:tasks.rados.rados.2.smithi175.stdout:1980: finishing write tid 2 to smithi17557870-2 2024-09-11T03:45:49.030 INFO:tasks.rados.rados.2.smithi175.stdout:1980: finishing write tid 3 to smithi17557870-2 2024-09-11T03:45:49.030 INFO:tasks.rados.rados.2.smithi175.stdout:1980: finishing write tid 4 to smithi17557870-2 2024-09-11T03:45:49.030 INFO:tasks.rados.rados.2.smithi175.stdout:1979: done (3 left) 2024-09-11T03:45:49.030 INFO:tasks.rados.rados.2.smithi175.stdout:1984: copy_from oid 9 from oid 12 current snap is 218 2024-09-11T03:45:49.030 INFO:tasks.rados.rados.2.smithi175.stdout:1980: finishing write tid 5 to smithi17557870-2 2024-09-11T03:45:49.030 INFO:tasks.rados.rados.2.smithi175.stdout:1980: oid 2 updating version 0 to 768 2024-09-11T03:45:49.030 INFO:tasks.rados.rados.2.smithi175.stdout:1980: oid 2 updating version 768 to 770 2024-09-11T03:45:49.030 INFO:tasks.rados.rados.2.smithi175.stdout:1980: oid 2 version 770 is already newer than 769 2024-09-11T03:45:49.030 INFO:tasks.rados.rados.2.smithi175.stdout:1980: oid 2 version 770 is already newer than 767 2024-09-11T03:45:49.030 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 770 (ObjNum 680 snap 218 seq_num 680) dirty exists 2024-09-11T03:45:49.030 INFO:tasks.rados.rados.2.smithi175.stdout:1980: left oid 2 (ObjNum 680 snap 218 seq_num 680) 2024-09-11T03:45:49.031 INFO:tasks.rados.rados.2.smithi175.stdout:1982: finishing write tid 1 to smithi17557870-25 2024-09-11T03:45:49.031 INFO:tasks.rados.rados.2.smithi175.stdout:1980: done (3 left) 2024-09-11T03:45:49.031 INFO:tasks.rados.rados.2.smithi175.stdout:1985: copy_from oid 44 from oid 50 current snap is 218 2024-09-11T03:45:49.031 INFO:tasks.rados.rados.2.smithi175.stdout:1982: finishing write tid 2 to smithi17557870-25 2024-09-11T03:45:49.031 INFO:tasks.rados.rados.2.smithi175.stdout:1986: write (excl) oid 41 current snap is 218 2024-09-11T03:45:49.031 INFO:tasks.rados.rados.2.smithi175.stdout:1986: seq_num 682 ranges {662374=475235,1609190=676892,2812481=624101} 2024-09-11T03:45:49.034 INFO:tasks.rados.rados.2.smithi175.stdout:1986: writing smithi17557870-41 from 662374 to 1137609 tid 1 2024-09-11T03:45:49.041 INFO:tasks.rados.rados.2.smithi175.stdout:1986: writing smithi17557870-41 from 1609190 to 2286082 tid 2 2024-09-11T03:45:49.047 INFO:tasks.rados.rados.2.smithi175.stdout:1986: writing smithi17557870-41 from 2812481 to 3436582 tid 3 2024-09-11T03:45:49.048 INFO:tasks.rados.rados.2.smithi175.stdout:1987: rollback oid 37 current snap is 218 2024-09-11T03:45:49.048 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 37 to 210 2024-09-11T03:45:49.048 INFO:tasks.rados.rados.2.smithi175.stdout:1982: finishing write tid 3 to smithi17557870-25 2024-09-11T03:45:49.048 INFO:tasks.rados.rados.2.smithi175.stdout:1982: finishing write tid 4 to smithi17557870-25 2024-09-11T03:45:49.048 INFO:tasks.rados.rados.2.smithi175.stdout:1982: finishing write tid 5 to smithi17557870-25 2024-09-11T03:45:49.048 INFO:tasks.rados.rados.2.smithi175.stdout:1982: oid 25 updating version 0 to 501 2024-09-11T03:45:49.048 INFO:tasks.rados.rados.2.smithi175.stdout:1982: oid 25 version 501 is already newer than 498 2024-09-11T03:45:49.048 INFO:tasks.rados.rados.2.smithi175.stdout:1982: oid 25 version 501 is already newer than 499 2024-09-11T03:45:49.048 INFO:tasks.rados.rados.2.smithi175.stdout:1982: oid 25 version 501 is already newer than 500 2024-09-11T03:45:49.048 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 501 (ObjNum 681 snap 218 seq_num 681) dirty exists 2024-09-11T03:45:49.048 INFO:tasks.rados.rados.2.smithi175.stdout:1982: left oid 25 (ObjNum 681 snap 218 seq_num 681) 2024-09-11T03:45:49.048 INFO:tasks.rados.rados.2.smithi175.stdout:1985: finishing copy_from to smithi17557870-44 2024-09-11T03:45:49.049 INFO:tasks.rados.rados.2.smithi175.stdout:1985: got expected ENOENT (src dne) 2024-09-11T03:45:49.049 INFO:tasks.rados.rados.2.smithi175.stdout:1982: done (5 left) 2024-09-11T03:45:49.049 INFO:tasks.rados.rados.2.smithi175.stdout:1988: snap_create 2024-09-11T03:45:49.049 INFO:tasks.rados.rados.2.smithi175.stdout:1985: finishing copy_from racing read to smithi17557870-44 2024-09-11T03:45:49.049 INFO:tasks.rados.rados.2.smithi175.stdout:1984: finishing copy_from racing read to smithi17557870-9 2024-09-11T03:45:49.049 INFO:tasks.rados.rados.2.smithi175.stdout:1986: finishing write tid 1 to smithi17557870-41 2024-09-11T03:45:49.049 INFO:tasks.rados.rados.2.smithi175.stdout:1983: finishing copy_from racing read to smithi17557870-34 2024-09-11T03:45:49.049 INFO:tasks.rados.rados.2.smithi175.stdout:1983: finishing copy_from to smithi17557870-34 2024-09-11T03:45:49.049 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 689 (ObjNum 615 snap 199 seq_num 615) dirty exists 2024-09-11T03:45:49.049 INFO:tasks.rados.rados.2.smithi175.stdout:1986: finishing write tid 2 to smithi17557870-41 2024-09-11T03:45:49.049 INFO:tasks.rados.rados.2.smithi175.stdout:1984: finishing copy_from to smithi17557870-9 2024-09-11T03:45:49.050 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 734 (ObjNum 662 snap 212 seq_num 662) dirty exists 2024-09-11T03:45:49.050 INFO:tasks.rados.rados.2.smithi175.stdout:1987: finishing rollback tid 0 to smithi17557870-37 2024-09-11T03:45:49.050 INFO:tasks.rados.rados.2.smithi175.stdout:1987: finishing rollback tid 1 to smithi17557870-37 2024-09-11T03:45:49.050 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 762 (ObjNum 289 snap 0 seq_num 2931793392) dirty dne 2024-09-11T03:45:49.051 INFO:tasks.rados.rados.2.smithi175.stdout:1986: finishing write tid 3 to smithi17557870-41 2024-09-11T03:45:49.051 INFO:tasks.rados.rados.2.smithi175.stdout:1986: finishing write tid 4 to smithi17557870-41 2024-09-11T03:45:49.051 INFO:tasks.rados.rados.2.smithi175.stdout:1986: finishing write tid 5 to smithi17557870-41 2024-09-11T03:45:49.051 INFO:tasks.rados.rados.2.smithi175.stdout:1986: oid 41 updating version 0 to 589 2024-09-11T03:45:49.051 INFO:tasks.rados.rados.2.smithi175.stdout:1986: oid 41 updating version 589 to 590 2024-09-11T03:45:49.051 INFO:tasks.rados.rados.2.smithi175.stdout:1986: oid 41 version 590 is already newer than 587 2024-09-11T03:45:49.051 INFO:tasks.rados.rados.2.smithi175.stdout:1986: oid 41 version 590 is already newer than 588 2024-09-11T03:45:49.051 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 590 (ObjNum 682 snap 218 seq_num 682) dirty exists 2024-09-11T03:45:49.051 INFO:tasks.rados.rados.2.smithi175.stdout:1986: left oid 41 (ObjNum 682 snap 218 seq_num 682) 2024-09-11T03:45:49.112 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:49.133 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:49.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:49.134 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:49.235 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:49.257 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:49.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:49.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:49.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:49 smithi084 ceph-mon[27254]: 1.13 scrub starts 2024-09-11T03:45:49.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:49 smithi084 ceph-mon[27254]: 1.13 scrub ok 2024-09-11T03:45:49.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3733292133' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:45:49.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/913229732' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:45:49.360 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:49.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:49.382 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:49.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:49.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:49 smithi121 ceph-mon[30729]: 1.13 scrub starts 2024-09-11T03:45:49.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:49 smithi121 ceph-mon[30729]: 1.13 scrub ok 2024-09-11T03:45:49.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3733292133' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:45:49.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/913229732' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:45:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:49 smithi175 ceph-mon[30894]: 1.13 scrub starts 2024-09-11T03:45:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:49 smithi175 ceph-mon[30894]: 1.13 scrub ok 2024-09-11T03:45:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3733292133' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:45:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/913229732' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:45:49.483 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:49.506 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:49.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:49.607 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:49.632 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:49.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:49.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:49.734 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:49.758 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:49.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:49.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:49.860 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:49.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:49.884 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:49.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:49.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:50.008 INFO:tasks.rados.rados.2.smithi175.stdout:1983: done (5 left) 2024-09-11T03:45:50.009 INFO:tasks.rados.rados.2.smithi175.stdout:1984: done (4 left) 2024-09-11T03:45:50.009 INFO:tasks.rados.rados.2.smithi175.stdout:1985: done (3 left) 2024-09-11T03:45:50.009 INFO:tasks.rados.rados.2.smithi175.stdout:1986: done (2 left) 2024-09-11T03:45:50.009 INFO:tasks.rados.rados.2.smithi175.stdout:1987: done (1 left) 2024-09-11T03:45:50.009 INFO:tasks.rados.rados.2.smithi175.stdout:1988: done (0 left) 2024-09-11T03:45:50.009 INFO:tasks.rados.rados.2.smithi175.stdout:1989: rollback oid 24 current snap is 219 2024-09-11T03:45:50.009 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 206 2024-09-11T03:45:50.009 INFO:tasks.rados.rados.2.smithi175.stdout:1990: write oid 22 current snap is 219 2024-09-11T03:45:50.009 INFO:tasks.rados.rados.2.smithi175.stdout:1990: seq_num 683 ranges {679181=550484,1667672=536903,2839904=435373} 2024-09-11T03:45:50.010 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:50.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:50.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:50.015 INFO:tasks.rados.rados.2.smithi175.stdout:1990: writing smithi17557870-22 from 679181 to 1229665 tid 1 2024-09-11T03:45:50.020 INFO:tasks.rados.rados.2.smithi175.stdout:1990: writing smithi17557870-22 from 1667672 to 2204575 tid 2 2024-09-11T03:45:50.026 INFO:tasks.rados.rados.2.smithi175.stdout:1990: writing smithi17557870-22 from 2839904 to 3275277 tid 3 2024-09-11T03:45:50.026 INFO:tasks.rados.rados.2.smithi175.stdout:1991: snap_create 2024-09-11T03:45:50.026 INFO:tasks.rados.rados.2.smithi175.stdout:1989: finishing rollback tid 0 to smithi17557870-24 2024-09-11T03:45:50.027 INFO:tasks.rados.rados.2.smithi175.stdout:1989: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:45:50.027 INFO:tasks.rados.rados.2.smithi175.stdout:1989: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:45:50.027 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 778 (ObjNum 570 snap 174 seq_num 570) dirty exists 2024-09-11T03:45:50.027 INFO:tasks.rados.rados.2.smithi175.stdout:1990: finishing write tid 1 to smithi17557870-22 2024-09-11T03:45:50.027 INFO:tasks.rados.rados.2.smithi175.stdout:1990: finishing write tid 2 to smithi17557870-22 2024-09-11T03:45:50.029 INFO:tasks.rados.rados.2.smithi175.stdout:1990: finishing write tid 3 to smithi17557870-22 2024-09-11T03:45:50.029 INFO:tasks.rados.rados.2.smithi175.stdout:1990: finishing write tid 4 to smithi17557870-22 2024-09-11T03:45:50.029 INFO:tasks.rados.rados.2.smithi175.stdout:1990: finishing write tid 5 to smithi17557870-22 2024-09-11T03:45:50.029 INFO:tasks.rados.rados.2.smithi175.stdout:1990: oid 22 updating version 0 to 777 2024-09-11T03:45:50.029 INFO:tasks.rados.rados.2.smithi175.stdout:1990: oid 22 updating version 777 to 779 2024-09-11T03:45:50.030 INFO:tasks.rados.rados.2.smithi175.stdout:1990: oid 22 updating version 779 to 780 2024-09-11T03:45:50.030 INFO:tasks.rados.rados.2.smithi175.stdout:1990: oid 22 version 780 is already newer than 778 2024-09-11T03:45:50.030 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 780 (ObjNum 683 snap 219 seq_num 683) dirty exists 2024-09-11T03:45:50.030 INFO:tasks.rados.rados.2.smithi175.stdout:1990: left oid 22 (ObjNum 683 snap 219 seq_num 683) 2024-09-11T03:45:50.111 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:50.134 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:50.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:50.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:50.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:50.260 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:50.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:50.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:50.310 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:50 smithi084 ceph-mon[27254]: pgmap v951: 57 pgs: 57 active+clean; 171 MiB data, 772 MiB used, 983 GiB / 983 GiB avail; 7.5 MiB/s rd, 0 B/s wr, 6 op/s 2024-09-11T03:45:50.310 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:50 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:45:50.310 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:50 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:45:50.310 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:50 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3733292133' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:45:50.310 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:50 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/913229732' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:45:50.310 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:50 smithi084 ceph-mon[27254]: osdmap e531: 12 total, 12 up, 11 in 2024-09-11T03:45:50.339 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 9 1 2024-09-11T03:45:50.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:50.387 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:50.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:50.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:50.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:50 smithi121 ceph-mon[30729]: pgmap v951: 57 pgs: 57 active+clean; 171 MiB data, 772 MiB used, 983 GiB / 983 GiB avail; 7.5 MiB/s rd, 0 B/s wr, 6 op/s 2024-09-11T03:45:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:50 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:45:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:50 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:45:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:50 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3733292133' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:45:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:50 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/913229732' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:45:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:50 smithi121 ceph-mon[30729]: osdmap e531: 12 total, 12 up, 11 in 2024-09-11T03:45:50.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:50 smithi175 ceph-mon[30894]: pgmap v951: 57 pgs: 57 active+clean; 171 MiB data, 772 MiB used, 983 GiB / 983 GiB avail; 7.5 MiB/s rd, 0 B/s wr, 6 op/s 2024-09-11T03:45:50.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:50 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:45:50.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:50 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:45:50.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:50 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3733292133' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:45:50.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:50 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/913229732' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:45:50.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:50 smithi175 ceph-mon[30894]: osdmap e531: 12 total, 12 up, 11 in 2024-09-11T03:45:50.489 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:50.515 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:50.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:50.516 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:50.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:50.642 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:50.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:50.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:50.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:50.768 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:50.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:50.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:50.790 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T03:45:50.853 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:50.869 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:50.894 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:50.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:50.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:50.996 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:1989: done (2 left) 2024-09-11T03:45:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:1990: done (1 left) 2024-09-11T03:45:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:1991: done (0 left) 2024-09-11T03:45:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:1992: rollback oid 9 current snap is 220 2024-09-11T03:45:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 9 to 211 2024-09-11T03:45:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:1993: write (excl) oid 2 current snap is 220 2024-09-11T03:45:51.018 INFO:tasks.rados.rados.2.smithi175.stdout:1993: seq_num 684 ranges {695988=625733,2009801=680561,3150974=345014} 2024-09-11T03:45:51.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:51.020 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:51.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:51.025 INFO:tasks.rados.rados.2.smithi175.stdout:1993: writing smithi17557870-2 from 695988 to 1321721 tid 1 2024-09-11T03:45:51.032 INFO:tasks.rados.rados.2.smithi175.stdout:1993: writing smithi17557870-2 from 2009801 to 2690362 tid 2 2024-09-11T03:45:51.038 INFO:tasks.rados.rados.2.smithi175.stdout:1993: writing smithi17557870-2 from 3150974 to 3495988 tid 3 2024-09-11T03:45:51.038 INFO:tasks.rados.rados.2.smithi175.stdout:1994: copy_from oid 16 from oid 27 current snap is 220 2024-09-11T03:45:51.038 INFO:tasks.rados.rados.2.smithi175.stdout:1992: finishing rollback tid 0 to smithi17557870-9 2024-09-11T03:45:51.038 INFO:tasks.rados.rados.2.smithi175.stdout:1992: finishing rollback tid 1 to smithi17557870-9 2024-09-11T03:45:51.038 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 777 (ObjNum 593 snap 0 seq_num 2931793696) dirty dne 2024-09-11T03:45:51.038 INFO:tasks.rados.rados.2.smithi175.stdout:1993: finishing write tid 1 to smithi17557870-2 2024-09-11T03:45:51.038 INFO:tasks.rados.rados.2.smithi175.stdout:1993: finishing write tid 2 to smithi17557870-2 2024-09-11T03:45:51.038 INFO:tasks.rados.rados.2.smithi175.stdout:1992: done (2 left) 2024-09-11T03:45:51.038 INFO:tasks.rados.rados.2.smithi175.stdout:1995: snap_create 2024-09-11T03:45:51.040 INFO:tasks.rados.rados.2.smithi175.stdout:1994: finishing copy_from to smithi17557870-16 2024-09-11T03:45:51.040 INFO:tasks.rados.rados.2.smithi175.stdout:1994: got expected ENOENT (src dne) 2024-09-11T03:45:51.040 INFO:tasks.rados.rados.2.smithi175.stdout:1993: finishing write tid 3 to smithi17557870-2 2024-09-11T03:45:51.040 INFO:tasks.rados.rados.2.smithi175.stdout:1994: finishing copy_from racing read to smithi17557870-16 2024-09-11T03:45:51.041 INFO:tasks.rados.rados.2.smithi175.stdout:1993: finishing write tid 4 to smithi17557870-2 2024-09-11T03:45:51.041 INFO:tasks.rados.rados.2.smithi175.stdout:1993: finishing write tid 5 to smithi17557870-2 2024-09-11T03:45:51.041 INFO:tasks.rados.rados.2.smithi175.stdout:1993: oid 2 updating version 0 to 782 2024-09-11T03:45:51.041 INFO:tasks.rados.rados.2.smithi175.stdout:1993: oid 2 version 782 is already newer than 779 2024-09-11T03:45:51.041 INFO:tasks.rados.rados.2.smithi175.stdout:1993: oid 2 version 782 is already newer than 780 2024-09-11T03:45:51.041 INFO:tasks.rados.rados.2.smithi175.stdout:1993: oid 2 version 782 is already newer than 781 2024-09-11T03:45:51.042 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 782 (ObjNum 684 snap 220 seq_num 684) dirty exists 2024-09-11T03:45:51.042 INFO:tasks.rados.rados.2.smithi175.stdout:1993: left oid 2 (ObjNum 684 snap 220 seq_num 684) 2024-09-11T03:45:51.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:51.146 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:51.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:51.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:51 smithi175 ceph-mon[30894]: 2.6 scrub starts 2024-09-11T03:45:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:51 smithi175 ceph-mon[30894]: 2.6 scrub ok 2024-09-11T03:45:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:51 smithi175 ceph-mon[30894]: osdmap e532: 12 total, 12 up, 11 in 2024-09-11T03:45:51.248 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:51.270 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:51.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:51.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:51.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:51 smithi084 ceph-mon[27254]: 2.6 scrub starts 2024-09-11T03:45:51.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:51 smithi084 ceph-mon[27254]: 2.6 scrub ok 2024-09-11T03:45:51.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:51 smithi084 ceph-mon[27254]: osdmap e532: 12 total, 12 up, 11 in 2024-09-11T03:45:51.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:51.398 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:51.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:51.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:51.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:51 smithi121 ceph-mon[30729]: 2.6 scrub starts 2024-09-11T03:45:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:51 smithi121 ceph-mon[30729]: 2.6 scrub ok 2024-09-11T03:45:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:51 smithi121 ceph-mon[30729]: osdmap e532: 12 total, 12 up, 11 in 2024-09-11T03:45:51.447 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:45:51.447 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:45:51.466 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:45:51.466 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:45:51.467 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:51.471 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:51.483 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:45:51.484 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:45:51.500 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:51.501 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:45:51.502 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:45:51.520 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:45:51.521 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:45:51.522 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:51.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:51.538 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:45:51.538 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:45:51.555 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:45:51.555 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:45:51.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:45:51.572 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:45:51.589 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:45:51.589 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:45:51.606 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:45:51.606 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:45:51.624 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:45:51.624 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:45:51.625 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:51.640 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:45:51.640 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:45:51.646 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:51.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:51.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:51.726 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:45:51.748 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:51.774 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:51.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:51.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:51.877 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:51.903 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:51.904 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:51.956 INFO:tasks.rados.rados.2.smithi175.stdout:1993: done (2 left) 2024-09-11T03:45:51.956 INFO:tasks.rados.rados.2.smithi175.stdout:1994: done (1 left) 2024-09-11T03:45:51.956 INFO:tasks.rados.rados.2.smithi175.stdout:1995: done (0 left) 2024-09-11T03:45:51.956 INFO:tasks.rados.rados.2.smithi175.stdout:1996: rollback oid 31 current snap is 221 2024-09-11T03:45:51.956 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 31 to 206 2024-09-11T03:45:51.957 INFO:tasks.rados.rados.2.smithi175.stdout:1997: snap_remove snap 220 2024-09-11T03:45:52.004 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:52.030 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:52.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:52.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:52.132 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:52.158 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:52.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:52.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:52.261 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:52.286 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:52.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:52.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:52.388 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:52.389 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:52 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:45:52.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:52 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:45:52.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:52 smithi084 ceph-mon[27254]: 2.5 scrub starts 2024-09-11T03:45:52.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:52 smithi084 ceph-mon[27254]: 2.5 scrub ok 2024-09-11T03:45:52.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:52 smithi084 ceph-mon[27254]: pgmap v954: 57 pgs: 57 active+clean; 171 MiB data, 775 MiB used, 983 GiB / 983 GiB avail; 4.2 MiB/s rd, 0 B/s wr, 2 op/s 2024-09-11T03:45:52.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:52 smithi084 ceph-mon[27254]: osdmap e533: 12 total, 12 up, 11 in 2024-09-11T03:45:52.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:52 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:45:52.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:52 smithi084 ceph-mon[27254]: 2.e scrub ok 2024-09-11T03:45:52.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:52 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4284058300' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:52.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:52 smithi084 ceph-mon[27254]: osdmap e534: 12 total, 12 up, 11 in 2024-09-11T03:45:52.415 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:52.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:52.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:52 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:45:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:52 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:45:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:52 smithi121 ceph-mon[30729]: 2.5 scrub starts 2024-09-11T03:45:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:52 smithi121 ceph-mon[30729]: 2.5 scrub ok 2024-09-11T03:45:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:52 smithi121 ceph-mon[30729]: pgmap v954: 57 pgs: 57 active+clean; 171 MiB data, 775 MiB used, 983 GiB / 983 GiB avail; 4.2 MiB/s rd, 0 B/s wr, 2 op/s 2024-09-11T03:45:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:52 smithi121 ceph-mon[30729]: osdmap e533: 12 total, 12 up, 11 in 2024-09-11T03:45:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:52 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:45:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:52 smithi121 ceph-mon[30729]: 2.e scrub ok 2024-09-11T03:45:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:52 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4284058300' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:52 smithi121 ceph-mon[30729]: osdmap e534: 12 total, 12 up, 11 in 2024-09-11T03:45:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:52 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:45:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:52 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:45:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:52 smithi175 ceph-mon[30894]: 2.5 scrub starts 2024-09-11T03:45:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:52 smithi175 ceph-mon[30894]: 2.5 scrub ok 2024-09-11T03:45:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:52 smithi175 ceph-mon[30894]: pgmap v954: 57 pgs: 57 active+clean; 171 MiB data, 775 MiB used, 983 GiB / 983 GiB avail; 4.2 MiB/s rd, 0 B/s wr, 2 op/s 2024-09-11T03:45:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:52 smithi175 ceph-mon[30894]: osdmap e533: 12 total, 12 up, 11 in 2024-09-11T03:45:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:52 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:45:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:52 smithi175 ceph-mon[30894]: 2.e scrub ok 2024-09-11T03:45:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:52 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4284058300' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:52 smithi175 ceph-mon[30894]: osdmap e534: 12 total, 12 up, 11 in 2024-09-11T03:45:52.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:52.542 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:52.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:52.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:52.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:52.668 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:52.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:52.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:52.688 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:45:52.689 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:45:52.689 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:45:52.689 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:45:52.770 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:52.796 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:52.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:52.797 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:52.899 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:52.925 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:52.925 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:52.957 INFO:tasks.rados.rados.2.smithi175.stdout:1997: done (1 left) 2024-09-11T03:45:52.957 INFO:tasks.rados.rados.2.smithi175.stdout:1998: write (excl) oid 46 current snap is 221 2024-09-11T03:45:52.957 INFO:tasks.rados.rados.2.smithi175.stdout:1996: finishing rollback tid 0 to smithi17557870-31 2024-09-11T03:45:52.957 INFO:tasks.rados.rados.2.smithi175.stdout:1998: seq_num 685 ranges {712795=417335,1784636=540572,2894750=457917} 2024-09-11T03:45:52.963 INFO:tasks.rados.rados.2.smithi175.stdout:1998: writing smithi17557870-46 from 712795 to 1130130 tid 1 2024-09-11T03:45:52.970 INFO:tasks.rados.rados.2.smithi175.stdout:1998: writing smithi17557870-46 from 1784636 to 2325208 tid 2 2024-09-11T03:45:52.976 INFO:tasks.rados.rados.2.smithi175.stdout:1998: writing smithi17557870-46 from 2894750 to 3352667 tid 3 2024-09-11T03:45:52.976 INFO:tasks.rados.rados.2.smithi175.stdout:1999: rollback oid 41 current snap is 221 2024-09-11T03:45:52.976 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 41 to 206 2024-09-11T03:45:52.976 INFO:tasks.rados.rados.2.smithi175.stdout:1996: finishing rollback tid 1 to smithi17557870-31 2024-09-11T03:45:52.976 INFO:tasks.rados.rados.2.smithi175.stdout:1996: finishing rollback tid 2 to smithi17557870-31 2024-09-11T03:45:52.976 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 786 (ObjNum 495 snap 143 seq_num 495) dirty exists 2024-09-11T03:45:52.976 INFO:tasks.rados.rados.2.smithi175.stdout:1998: finishing write tid 1 to smithi17557870-46 2024-09-11T03:45:52.976 INFO:tasks.rados.rados.2.smithi175.stdout:1998: finishing write tid 2 to smithi17557870-46 2024-09-11T03:45:52.976 INFO:tasks.rados.rados.2.smithi175.stdout:1996: done (2 left) 2024-09-11T03:45:52.976 INFO:tasks.rados.rados.2.smithi175.stdout:2000: write oid 17 current snap is 221 2024-09-11T03:45:52.977 INFO:tasks.rados.rados.2.smithi175.stdout:2000: seq_num 686 ranges {729602=492584,1843118=684230,3322173=207429} 2024-09-11T03:45:52.982 INFO:tasks.rados.rados.2.smithi175.stdout:2000: writing smithi17557870-17 from 729602 to 1222186 tid 1 2024-09-11T03:45:52.989 INFO:tasks.rados.rados.2.smithi175.stdout:2000: writing smithi17557870-17 from 1843118 to 2527348 tid 2 2024-09-11T03:45:52.995 INFO:tasks.rados.rados.2.smithi175.stdout:2000: writing smithi17557870-17 from 3322173 to 3529602 tid 3 2024-09-11T03:45:52.995 INFO:tasks.rados.rados.2.smithi175.stdout:2001: read oid 21 snap -1 2024-09-11T03:45:52.995 INFO:tasks.rados.rados.2.smithi175.stdout:2001: expect (ObjNum 584 snap 181 seq_num 584) 2024-09-11T03:45:52.996 INFO:tasks.rados.rados.2.smithi175.stdout:2002: snap_create 2024-09-11T03:45:52.996 INFO:tasks.rados.rados.2.smithi175.stdout:1998: finishing write tid 3 to smithi17557870-46 2024-09-11T03:45:52.996 INFO:tasks.rados.rados.2.smithi175.stdout:1998: finishing write tid 4 to smithi17557870-46 2024-09-11T03:45:52.996 INFO:tasks.rados.rados.2.smithi175.stdout:1998: finishing write tid 5 to smithi17557870-46 2024-09-11T03:45:52.996 INFO:tasks.rados.rados.2.smithi175.stdout:1998: oid 46 updating version 0 to 537 2024-09-11T03:45:52.996 INFO:tasks.rados.rados.2.smithi175.stdout:1998: oid 46 version 537 is already newer than 536 2024-09-11T03:45:52.996 INFO:tasks.rados.rados.2.smithi175.stdout:1998: oid 46 updating version 537 to 538 2024-09-11T03:45:52.996 INFO:tasks.rados.rados.2.smithi175.stdout:1998: oid 46 version 538 is already newer than 535 2024-09-11T03:45:52.996 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 538 (ObjNum 685 snap 221 seq_num 685) dirty exists 2024-09-11T03:45:52.996 INFO:tasks.rados.rados.2.smithi175.stdout:1998: left oid 46 (ObjNum 685 snap 221 seq_num 685) 2024-09-11T03:45:52.996 INFO:tasks.rados.rados.2.smithi175.stdout:1999: finishing rollback tid 0 to smithi17557870-41 2024-09-11T03:45:52.996 INFO:tasks.rados.rados.2.smithi175.stdout:1999: finishing rollback tid 1 to smithi17557870-41 2024-09-11T03:45:52.996 INFO:tasks.rados.rados.2.smithi175.stdout:1999: finishing rollback tid 2 to smithi17557870-41 2024-09-11T03:45:52.996 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 594 (ObjNum 609 snap 191 seq_num 609) dirty exists 2024-09-11T03:45:52.997 INFO:tasks.rados.rados.2.smithi175.stdout:2000: finishing write tid 1 to smithi17557870-17 2024-09-11T03:45:52.997 INFO:tasks.rados.rados.2.smithi175.stdout:2000: finishing write tid 2 to smithi17557870-17 2024-09-11T03:45:52.997 INFO:tasks.rados.rados.2.smithi175.stdout:2000: finishing write tid 3 to smithi17557870-17 2024-09-11T03:45:52.997 INFO:tasks.rados.rados.2.smithi175.stdout:2000: finishing write tid 4 to smithi17557870-17 2024-09-11T03:45:52.998 INFO:tasks.rados.rados.2.smithi175.stdout:2000: finishing write tid 5 to smithi17557870-17 2024-09-11T03:45:52.998 INFO:tasks.rados.rados.2.smithi175.stdout:2000: oid 17 updating version 0 to 711 2024-09-11T03:45:52.998 INFO:tasks.rados.rados.2.smithi175.stdout:2000: oid 17 version 711 is already newer than 709 2024-09-11T03:45:52.998 INFO:tasks.rados.rados.2.smithi175.stdout:2000: oid 17 version 711 is already newer than 710 2024-09-11T03:45:52.998 INFO:tasks.rados.rados.2.smithi175.stdout:2000: oid 17 version 711 is already newer than 708 2024-09-11T03:45:52.998 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 711 (ObjNum 686 snap 221 seq_num 686) dirty exists 2024-09-11T03:45:52.998 INFO:tasks.rados.rados.2.smithi175.stdout:2000: left oid 17 (ObjNum 686 snap 221 seq_num 686) 2024-09-11T03:45:53.001 INFO:tasks.rados.rados.2.smithi175.stdout:2001: expect (ObjNum 584 snap 181 seq_num 584) 2024-09-11T03:45:53.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:53.047 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:53.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:53.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:53.149 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:53.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:53.171 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:53.172 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:53.273 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:53.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:53.295 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:53.295 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:53.315 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T03:45:53.396 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:53.422 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:53.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:53.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:53.524 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:53.546 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:53.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:53.547 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:53.648 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:53.669 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:53.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:53.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:53.771 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:53.796 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:53.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:53.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:53.838 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:53.898 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:53.921 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:53.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:53.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:53.960 INFO:tasks.rados.rados.2.smithi175.stdout:1998: done (4 left) 2024-09-11T03:45:53.960 INFO:tasks.rados.rados.2.smithi175.stdout:1999: done (3 left) 2024-09-11T03:45:53.960 INFO:tasks.rados.rados.2.smithi175.stdout:2000: done (2 left) 2024-09-11T03:45:53.960 INFO:tasks.rados.rados.2.smithi175.stdout:2001: done (1 left) 2024-09-11T03:45:53.960 INFO:tasks.rados.rados.2.smithi175.stdout:2002: done (0 left) 2024-09-11T03:45:53.960 INFO:tasks.rados.rados.2.smithi175.stdout:2003: snap_remove snap 211 2024-09-11T03:45:54.024 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:54.046 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:54.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:54.047 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:54.148 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:54.170 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:54.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:54.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:53 smithi175 ceph-mon[30894]: pgmap v957: 57 pgs: 57 active+clean; 196 MiB data, 802 MiB used, 983 GiB / 983 GiB avail; 3.3 MiB/s rd, 5.1 MiB/s wr, 19 op/s 2024-09-11T03:45:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:53 smithi175 ceph-mon[30894]: 1.1a scrub starts 2024-09-11T03:45:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:53 smithi175 ceph-mon[30894]: 1.1a scrub ok 2024-09-11T03:45:54.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:53 smithi175 ceph-mon[30894]: osdmap e535: 12 total, 12 up, 11 in 2024-09-11T03:45:54.269 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:53 smithi084 ceph-mon[27254]: pgmap v957: 57 pgs: 57 active+clean; 196 MiB data, 802 MiB used, 983 GiB / 983 GiB avail; 3.3 MiB/s rd, 5.1 MiB/s wr, 19 op/s 2024-09-11T03:45:54.269 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:53 smithi084 ceph-mon[27254]: 1.1a scrub starts 2024-09-11T03:45:54.269 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:53 smithi084 ceph-mon[27254]: 1.1a scrub ok 2024-09-11T03:45:54.269 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:53 smithi084 ceph-mon[27254]: osdmap e535: 12 total, 12 up, 11 in 2024-09-11T03:45:54.273 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:54.273 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:54.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:54.297 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:54.297 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:54.299 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:45:54.299 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:45:54.299 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:45:54.299 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:45:54.300 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:45:54.397 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:54.421 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:54.421 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:53 smithi121 ceph-mon[30729]: pgmap v957: 57 pgs: 57 active+clean; 196 MiB data, 802 MiB used, 983 GiB / 983 GiB avail; 3.3 MiB/s rd, 5.1 MiB/s wr, 19 op/s 2024-09-11T03:45:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:53 smithi121 ceph-mon[30729]: 1.1a scrub starts 2024-09-11T03:45:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:53 smithi121 ceph-mon[30729]: 1.1a scrub ok 2024-09-11T03:45:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:53 smithi121 ceph-mon[30729]: osdmap e535: 12 total, 12 up, 11 in 2024-09-11T03:45:54.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:45:54.522 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:54.553 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:54.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:54.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:54.655 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:54.681 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:54.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:54.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:54.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:54.809 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:54.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:54.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:54.912 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:54.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:54.938 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:54.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:54.962 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.9 to 1 (10000) 2024-09-11T03:45:54.967 INFO:tasks.rados.rados.2.smithi175.stdout:2003: done (0 left) 2024-09-11T03:45:54.968 INFO:tasks.rados.rados.2.smithi175.stdout:2004: snap_create 2024-09-11T03:45:55.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:55.064 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:55.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:55.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:55.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:55.190 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:55.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:55.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:55.221 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:54 smithi084 ceph-mon[27254]: 2.10 scrub starts 2024-09-11T03:45:55.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:54 smithi084 ceph-mon[27254]: 2.10 scrub ok 2024-09-11T03:45:55.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:54 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:45:55.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:54 smithi084 ceph-mon[27254]: 1.e scrub starts 2024-09-11T03:45:55.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:54 smithi084 ceph-mon[27254]: 1.e scrub ok 2024-09-11T03:45:55.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:54 smithi084 ceph-mon[27254]: 2.2 scrub ok 2024-09-11T03:45:55.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:54 smithi084 ceph-mon[27254]: 2.11 scrub starts 2024-09-11T03:45:55.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:54 smithi084 ceph-mon[27254]: 2.11 scrub ok 2024-09-11T03:45:55.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:54 smithi084 ceph-mon[27254]: osdmap e536: 12 total, 12 up, 11 in 2024-09-11T03:45:55.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:54 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3481820590' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:45:55.223 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T03:45:55.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:54 smithi175 ceph-mon[30894]: 2.10 scrub starts 2024-09-11T03:45:55.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:54 smithi175 ceph-mon[30894]: 2.10 scrub ok 2024-09-11T03:45:55.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:54 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:45:55.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:54 smithi175 ceph-mon[30894]: 1.e scrub starts 2024-09-11T03:45:55.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:54 smithi175 ceph-mon[30894]: 1.e scrub ok 2024-09-11T03:45:55.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:54 smithi175 ceph-mon[30894]: 2.2 scrub ok 2024-09-11T03:45:55.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:54 smithi175 ceph-mon[30894]: 2.11 scrub starts 2024-09-11T03:45:55.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:54 smithi175 ceph-mon[30894]: 2.11 scrub ok 2024-09-11T03:45:55.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:54 smithi175 ceph-mon[30894]: osdmap e536: 12 total, 12 up, 11 in 2024-09-11T03:45:55.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:54 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3481820590' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:45:55.291 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:55.312 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:55.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:55.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:55.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:45:55.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:54 smithi121 ceph-mon[30729]: 2.10 scrub starts 2024-09-11T03:45:55.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:54 smithi121 ceph-mon[30729]: 2.10 scrub ok 2024-09-11T03:45:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:54 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:45:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:54 smithi121 ceph-mon[30729]: 1.e scrub starts 2024-09-11T03:45:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:54 smithi121 ceph-mon[30729]: 1.e scrub ok 2024-09-11T03:45:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:54 smithi121 ceph-mon[30729]: 2.2 scrub ok 2024-09-11T03:45:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:54 smithi121 ceph-mon[30729]: 2.11 scrub starts 2024-09-11T03:45:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:54 smithi121 ceph-mon[30729]: 2.11 scrub ok 2024-09-11T03:45:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:54 smithi121 ceph-mon[30729]: osdmap e536: 12 total, 12 up, 11 in 2024-09-11T03:45:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:54 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3481820590' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:45:55.438 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:55.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:45:55.541 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:55.562 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:55.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:55.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:55.664 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:55.689 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:55.690 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:55.765 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:55.791 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:55.817 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:55.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:55.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:55.919 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:55.945 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:55.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:55.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:55.963 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:45:55.969 INFO:tasks.rados.rados.2.smithi175.stdout:2004: done (0 left) 2024-09-11T03:45:55.969 INFO:tasks.rados.rados.2.smithi175.stdout:2005: delete oid 4 current snap is 223 2024-09-11T03:45:55.972 INFO:tasks.rados.rados.2.smithi175.stdout:2005: done (0 left) 2024-09-11T03:45:55.972 INFO:tasks.rados.rados.2.smithi175.stdout:2006: read oid 33 snap -1 2024-09-11T03:45:55.972 INFO:tasks.rados.rados.2.smithi175.stdout:2006: expect (ObjNum 630 snap 202 seq_num 630) 2024-09-11T03:45:55.972 INFO:tasks.rados.rados.2.smithi175.stdout:2007: snap_remove snap 217 2024-09-11T03:45:56.047 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:56.068 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:56.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:56.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:56.170 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:56.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:56.192 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:56.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:55 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:45:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:55 smithi175 ceph-mon[30894]: 2.a scrub ok 2024-09-11T03:45:56.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:55 smithi175 ceph-mon[30894]: 2.16 scrub starts 2024-09-11T03:45:56.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:55 smithi175 ceph-mon[30894]: 2.16 scrub ok 2024-09-11T03:45:56.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:55 smithi175 ceph-mon[30894]: pgmap v960: 57 pgs: 57 active+clean; 198 MiB data, 835 MiB used, 983 GiB / 983 GiB avail; 3.3 MiB/s rd, 5.1 MiB/s wr, 19 op/s 2024-09-11T03:45:56.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:55 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3481820590' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:45:56.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:55 smithi175 ceph-mon[30894]: osdmap e537: 12 total, 12 up, 11 in 2024-09-11T03:45:56.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:55 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1646377269' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:45:56.293 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:56.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:56.315 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:56.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:56.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:55 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:45:56.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:55 smithi084 ceph-mon[27254]: 2.a scrub ok 2024-09-11T03:45:56.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:55 smithi084 ceph-mon[27254]: 2.16 scrub starts 2024-09-11T03:45:56.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:55 smithi084 ceph-mon[27254]: 2.16 scrub ok 2024-09-11T03:45:56.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:55 smithi084 ceph-mon[27254]: pgmap v960: 57 pgs: 57 active+clean; 198 MiB data, 835 MiB used, 983 GiB / 983 GiB avail; 3.3 MiB/s rd, 5.1 MiB/s wr, 19 op/s 2024-09-11T03:45:56.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:55 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3481820590' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:45:56.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:55 smithi084 ceph-mon[27254]: osdmap e537: 12 total, 12 up, 11 in 2024-09-11T03:45:56.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:55 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1646377269' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:45:56.365 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 11 1 2024-09-11T03:45:56.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:45:56.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:55 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:45:56.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:55 smithi121 ceph-mon[30729]: 2.a scrub ok 2024-09-11T03:45:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:55 smithi121 ceph-mon[30729]: 2.16 scrub starts 2024-09-11T03:45:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:55 smithi121 ceph-mon[30729]: 2.16 scrub ok 2024-09-11T03:45:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:55 smithi121 ceph-mon[30729]: pgmap v960: 57 pgs: 57 active+clean; 198 MiB data, 835 MiB used, 983 GiB / 983 GiB avail; 3.3 MiB/s rd, 5.1 MiB/s wr, 19 op/s 2024-09-11T03:45:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:55 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3481820590' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:45:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:55 smithi121 ceph-mon[30729]: osdmap e537: 12 total, 12 up, 11 in 2024-09-11T03:45:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:55 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1646377269' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:45:56.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:56.445 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:56.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:45:56.546 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:56.572 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:56.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:56.573 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:56.578 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:56.674 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:56.699 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:56.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:56.700 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:56.801 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:56.826 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:56.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:56.828 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:56.929 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:45:56.951 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:56.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:45:56.957 INFO:tasks.rados.rados.2.smithi175.stdout:2007: done (1 left) 2024-09-11T03:45:56.957 INFO:tasks.rados.rados.2.smithi175.stdout:2008: snap_create 2024-09-11T03:45:56.958 INFO:tasks.rados.rados.2.smithi175.stdout:2006: expect (ObjNum 630 snap 202 seq_num 630) 2024-09-11T03:45:57.053 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:57.073 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:57.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:57.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:57.175 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:57.197 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:57.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:57.198 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:57.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:56 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:45:57.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:56 smithi175 ceph-mon[30894]: 2.0 scrub ok 2024-09-11T03:45:57.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:56 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:57.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:56 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1646377269' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:45:57.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:56 smithi175 ceph-mon[30894]: osdmap e538: 12 total, 12 up, 11 in 2024-09-11T03:45:57.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:56 smithi175 ceph-mon[30894]: osdmap e539: 12 total, 12 up, 11 in 2024-09-11T03:45:57.299 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:57.320 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:57.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:57.321 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:56 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:45:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:56 smithi084 ceph-mon[27254]: 2.0 scrub ok 2024-09-11T03:45:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:56 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1646377269' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:45:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:56 smithi084 ceph-mon[27254]: osdmap e538: 12 total, 12 up, 11 in 2024-09-11T03:45:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:56 smithi084 ceph-mon[27254]: osdmap e539: 12 total, 12 up, 11 in 2024-09-11T03:45:57.421 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:56 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:45:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:56 smithi121 ceph-mon[30729]: 2.0 scrub ok 2024-09-11T03:45:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:56 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:45:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:56 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1646377269' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:45:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:56 smithi121 ceph-mon[30729]: osdmap e538: 12 total, 12 up, 11 in 2024-09-11T03:45:57.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:56 smithi121 ceph-mon[30729]: osdmap e539: 12 total, 12 up, 11 in 2024-09-11T03:45:57.444 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:57.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:57.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:57.546 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:57.567 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:57.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:57.569 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:57.670 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:57.697 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:57.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:57.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:57.744 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:45:57.744 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:45:57.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:45:57.744 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:45:57.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:228 2.9 2:95fa802a:::smithi17557870-22:head [write 490771~686315,stat] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-11T03:45:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:41.965436+0000", 2024-09-11T03:45:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 435.77750607399997, 2024-09-11T03:45:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018731718000000001, 2024-09-11T03:45:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 228 2024-09-11T03:45:57.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.965436+0000", 2024-09-11T03:45:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.965436+0000", 2024-09-11T03:45:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.965439+0000", 2024-09-11T03:45:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2879999999999999e-06 2024-09-11T03:45:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.966276+0000", 2024-09-11T03:45:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00083751099999999998 2024-09-11T03:45:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.966299+0000", 2024-09-11T03:45:57.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2532999999999999e-05 2024-09-11T03:45:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.966308+0000", 2024-09-11T03:45:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4839999999999993e-06 2024-09-11T03:45:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.966374+0000", 2024-09-11T03:45:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5534e-05 2024-09-11T03:45:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.966442+0000", 2024-09-11T03:45:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8368999999999995e-05 2024-09-11T03:45:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:45:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.966571+0000", 2024-09-11T03:45:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012857999999999999 2024-09-11T03:45:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:45:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.968401+0000", 2024-09-11T03:45:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001830304 2024-09-11T03:45:57.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.968632+0000", 2024-09-11T03:45:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000230752 2024-09-11T03:45:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984080+0000", 2024-09-11T03:45:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015448066999999999 2024-09-11T03:45:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984117+0000", 2024-09-11T03:45:57.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7150000000000002e-05 2024-09-11T03:45:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984168+0000", 2024-09-11T03:45:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1146000000000001e-05 2024-09-11T03:45:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:229 2.9 2:95fa802a:::smithi17557870-22:head [write 1885075~406346] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-11T03:45:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:41.971562+0000", 2024-09-11T03:45:57.751 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 435.771380384, 2024-09-11T03:45:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012698844000000001, 2024-09-11T03:45:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 229 2024-09-11T03:45:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971562+0000", 2024-09-11T03:45:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971562+0000", 2024-09-11T03:45:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971563+0000", 2024-09-11T03:45:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1200000000000001e-06 2024-09-11T03:45:57.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971924+0000", 2024-09-11T03:45:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00036046400000000001 2024-09-11T03:45:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971934+0000", 2024-09-11T03:45:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0744000000000001e-05 2024-09-11T03:45:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971942+0000", 2024-09-11T03:45:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1489999999999997e-06 2024-09-11T03:45:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.971984+0000", 2024-09-11T03:45:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2178000000000002e-05 2024-09-11T03:45:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.972080+0000", 2024-09-11T03:45:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6362999999999999e-05 2024-09-11T03:45:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:45:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.972181+0000", 2024-09-11T03:45:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000100406 2024-09-11T03:45:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:45:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.973247+0000", 2024-09-11T03:45:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001066949 2024-09-11T03:45:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.973329+0000", 2024-09-11T03:45:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1897000000000005e-05 2024-09-11T03:45:57.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984202+0000", 2024-09-11T03:45:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010872728999999999 2024-09-11T03:45:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984235+0000", 2024-09-11T03:45:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3011e-05 2024-09-11T03:45:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:41.984261+0000", 2024-09-11T03:45:57.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5834e-05 2024-09-11T03:45:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:341 2.9 2:9718db5a:::smithi17557870-3:head [copy-from ver 6] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:45:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:49.820733+0000", 2024-09-11T03:45:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 427.92220911700002, 2024-09-11T03:45:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018872793999999998, 2024-09-11T03:45:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:57.758 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 341 2024-09-11T03:45:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.759 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820733+0000", 2024-09-11T03:45:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820733+0000", 2024-09-11T03:45:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820736+0000", 2024-09-11T03:45:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9500000000000001e-06 2024-09-11T03:45:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820748+0000", 2024-09-11T03:45:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1516e-05 2024-09-11T03:45:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820778+0000", 2024-09-11T03:45:57.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9827000000000001e-05 2024-09-11T03:45:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820800+0000", 2024-09-11T03:45:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2265000000000001e-05 2024-09-11T03:45:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820873+0000", 2024-09-11T03:45:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3386999999999994e-05 2024-09-11T03:45:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:45:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820880+0000", 2024-09-11T03:45:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6800000000000004e-06 2024-09-11T03:45:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.821954+0000", 2024-09-11T03:45:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010737500000000001 2024-09-11T03:45:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.822016+0000", 2024-09-11T03:45:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1929999999999998e-05 2024-09-11T03:45:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:45:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.832720+0000", 2024-09-11T03:45:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010704709 2024-09-11T03:45:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:45:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.837688+0000", 2024-09-11T03:45:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049680089999999998 2024-09-11T03:45:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.837979+0000", 2024-09-11T03:45:57.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029093800000000003 2024-09-11T03:45:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.839534+0000", 2024-09-11T03:45:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015545170000000001 2024-09-11T03:45:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.839569+0000", 2024-09-11T03:45:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5386999999999998e-05 2024-09-11T03:45:57.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.839606+0000", 2024-09-11T03:45:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6928999999999998e-05 2024-09-11T03:45:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.768 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:342 2.9 2:9718db5a:::smithi17557870-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:45:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:49.820827+0000", 2024-09-11T03:45:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 427.92211572899998, 2024-09-11T03:45:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016114883999999999, 2024-09-11T03:45:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:45:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 342 2024-09-11T03:45:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820827+0000", 2024-09-11T03:45:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820827+0000", 2024-09-11T03:45:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820827+0000", 2024-09-11T03:45:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4600000000000005e-07 2024-09-11T03:45:57.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820831+0000", 2024-09-11T03:45:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.219e-06 2024-09-11T03:45:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820840+0000", 2024-09-11T03:45:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4399999999999994e-06 2024-09-11T03:45:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820847+0000", 2024-09-11T03:45:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4789999999999999e-06 2024-09-11T03:45:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820919+0000", 2024-09-11T03:45:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1223999999999994e-05 2024-09-11T03:45:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.820922+0000", 2024-09-11T03:45:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9270000000000001e-06 2024-09-11T03:45:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.822138+0000", 2024-09-11T03:45:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001215982 2024-09-11T03:45:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.822186+0000", 2024-09-11T03:45:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8905e-05 2024-09-11T03:45:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.836672+0000", 2024-09-11T03:45:57.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014485434 2024-09-11T03:45:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.836817+0000", 2024-09-11T03:45:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014537900000000001 2024-09-11T03:45:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:49.836942+0000", 2024-09-11T03:45:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000124349 2024-09-11T03:45:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.774 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:428 2.9 2:95fa802a:::smithi17557870-22:head [copy-from ver 3] snapc b=[b] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-11T03:45:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:55.852514+0000", 2024-09-11T03:45:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 421.89042867299997, 2024-09-11T03:45:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020845635000000001, 2024-09-11T03:45:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 428 2024-09-11T03:45:57.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852514+0000", 2024-09-11T03:45:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852514+0000", 2024-09-11T03:45:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852514+0000", 2024-09-11T03:45:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0399999999999995e-07 2024-09-11T03:45:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852521+0000", 2024-09-11T03:45:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3879999999999997e-06 2024-09-11T03:45:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852533+0000", 2024-09-11T03:45:57.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2028000000000001e-05 2024-09-11T03:45:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852541+0000", 2024-09-11T03:45:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7000000000000008e-06 2024-09-11T03:45:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.856017+0000", 2024-09-11T03:45:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034762170000000002 2024-09-11T03:45:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.856106+0000", 2024-09-11T03:45:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9219000000000001e-05 2024-09-11T03:45:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:45:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.867527+0000", 2024-09-11T03:45:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011420625 2024-09-11T03:45:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:45:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.871293+0000", 2024-09-11T03:45:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037662189999999999 2024-09-11T03:45:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.871599+0000", 2024-09-11T03:45:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00030639900000000001 2024-09-11T03:45:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.873296+0000", 2024-09-11T03:45:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016962170000000001 2024-09-11T03:45:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.873330+0000", 2024-09-11T03:45:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4644000000000002e-05 2024-09-11T03:45:57.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.873359+0000", 2024-09-11T03:45:57.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9275e-05 2024-09-11T03:45:57.781 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.781 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.781 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.781 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:429 2.9 2:95fa802a:::smithi17557870-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e119)", 2024-09-11T03:45:57.781 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:38:55.852553+0000", 2024-09-11T03:45:57.781 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 421.890389347, 2024-09-11T03:45:57.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018332642, 2024-09-11T03:45:57.781 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.782 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.782 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:45:57.782 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.782 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.782 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.782 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 429 2024-09-11T03:45:57.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.782 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852553+0000", 2024-09-11T03:45:57.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852553+0000", 2024-09-11T03:45:57.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852554+0000", 2024-09-11T03:45:57.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2600000000000002e-07 2024-09-11T03:45:57.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852556+0000", 2024-09-11T03:45:57.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.373e-06 2024-09-11T03:45:57.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852564+0000", 2024-09-11T03:45:57.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4440000000000008e-06 2024-09-11T03:45:57.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.852570+0000", 2024-09-11T03:45:57.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7459999999999997e-06 2024-09-11T03:45:57.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.856347+0000", 2024-09-11T03:45:57.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037767970000000001 2024-09-11T03:45:57.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:57.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.856411+0000", 2024-09-11T03:45:57.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3522000000000006e-05 2024-09-11T03:45:57.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.870676+0000", 2024-09-11T03:45:57.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014265582000000001 2024-09-11T03:45:57.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.870780+0000", 2024-09-11T03:45:57.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000103786 2024-09-11T03:45:57.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:38:55.870886+0000", 2024-09-11T03:45:57.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010586600000000001 2024-09-11T03:45:57.786 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.786 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.786 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.786 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:649 2.6 2:6dbd1e81:::smithi17557870-7:head [copy-from ver 82] snapc 11=[e] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:45:57.787 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:08.111685+0000", 2024-09-11T03:45:57.787 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 409.63125759000002, 2024-09-11T03:45:57.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017973807000000001, 2024-09-11T03:45:57.787 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.787 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.787 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:57.787 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.787 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.787 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.787 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 649 2024-09-11T03:45:57.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.787 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111685+0000", 2024-09-11T03:45:57.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111685+0000", 2024-09-11T03:45:57.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111686+0000", 2024-09-11T03:45:57.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.339e-06 2024-09-11T03:45:57.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111692+0000", 2024-09-11T03:45:57.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4759999999999997e-06 2024-09-11T03:45:57.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111706+0000", 2024-09-11T03:45:57.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4388e-05 2024-09-11T03:45:57.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111714+0000", 2024-09-11T03:45:57.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8730000000000007e-06 2024-09-11T03:45:57.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111782+0000", 2024-09-11T03:45:57.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7984000000000005e-05 2024-09-11T03:45:57.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.111857+0000", 2024-09-11T03:45:57.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4741999999999997e-05 2024-09-11T03:45:57.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,8", 2024-09-11T03:45:57.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118631+0000", 2024-09-11T03:45:57.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067745890000000001 2024-09-11T03:45:57.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:45:57.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.123831+0000", 2024-09-11T03:45:57.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051996869999999997 2024-09-11T03:45:57.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.127180+0000", 2024-09-11T03:45:57.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003348668 2024-09-11T03:45:57.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129539+0000", 2024-09-11T03:45:57.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0023596239999999998 2024-09-11T03:45:57.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:57.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129607+0000", 2024-09-11T03:45:57.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7556000000000002e-05 2024-09-11T03:45:57.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129659+0000", 2024-09-11T03:45:57.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1881e-05 2024-09-11T03:45:57.793 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.793 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.793 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.793 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:941 2.6 2:6dbd1e81:::smithi17557870-7:head [copy-from ver 44] snapc 24=[24,23,22,20,1f,1e,1c] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-11T03:45:57.793 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:39.829469+0000", 2024-09-11T03:45:57.793 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 377.91347328699999, 2024-09-11T03:45:57.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016117155000000001, 2024-09-11T03:45:57.793 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.793 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.793 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:57.793 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.794 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.794 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.794 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 941 2024-09-11T03:45:57.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.794 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829469+0000", 2024-09-11T03:45:57.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829469+0000", 2024-09-11T03:45:57.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829471+0000", 2024-09-11T03:45:57.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2060000000000001e-06 2024-09-11T03:45:57.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829480+0000", 2024-09-11T03:45:57.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3739999999999998e-06 2024-09-11T03:45:57.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829496+0000", 2024-09-11T03:45:57.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6728e-05 2024-09-11T03:45:57.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829504+0000", 2024-09-11T03:45:57.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5170000000000002e-06 2024-09-11T03:45:57.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829576+0000", 2024-09-11T03:45:57.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1734000000000001e-05 2024-09-11T03:45:57.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:45:57.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829582+0000", 2024-09-11T03:45:57.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7340000000000002e-06 2024-09-11T03:45:57.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.831253+0000", 2024-09-11T03:45:57.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016706379999999999 2024-09-11T03:45:57.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.831319+0000", 2024-09-11T03:45:57.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5893000000000001e-05 2024-09-11T03:45:57.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,8", 2024-09-11T03:45:57.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.839473+0000", 2024-09-11T03:45:57.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081536119999999993 2024-09-11T03:45:57.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:45:57.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.843019+0000", 2024-09-11T03:45:57.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0035461310000000001 2024-09-11T03:45:57.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.844908+0000", 2024-09-11T03:45:57.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018889460000000001 2024-09-11T03:45:57.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.845471+0000", 2024-09-11T03:45:57.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00056334899999999999 2024-09-11T03:45:57.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:57.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.845531+0000", 2024-09-11T03:45:57.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0361e-05 2024-09-11T03:45:57.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.845586+0000", 2024-09-11T03:45:57.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4932000000000002e-05 2024-09-11T03:45:57.800 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.800 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.800 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.800 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:942 2.6 2:6dbd1e81:::smithi17557870-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e163)", 2024-09-11T03:45:57.800 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:39.829523+0000", 2024-09-11T03:45:57.800 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 377.91341958200002, 2024-09-11T03:45:57.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013066657000000001, 2024-09-11T03:45:57.800 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.801 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.801 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:45:57.801 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.801 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.801 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.801 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 942 2024-09-11T03:45:57.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.801 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829523+0000", 2024-09-11T03:45:57.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829523+0000", 2024-09-11T03:45:57.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829523+0000", 2024-09-11T03:45:57.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1099999999999996e-07 2024-09-11T03:45:57.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829526+0000", 2024-09-11T03:45:57.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4609999999999998e-06 2024-09-11T03:45:57.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829615+0000", 2024-09-11T03:45:57.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9015000000000001e-05 2024-09-11T03:45:57.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829624+0000", 2024-09-11T03:45:57.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5470000000000002e-06 2024-09-11T03:45:57.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829692+0000", 2024-09-11T03:45:57.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7605000000000001e-05 2024-09-11T03:45:57.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:57.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.829698+0000", 2024-09-11T03:45:57.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4990000000000002e-06 2024-09-11T03:45:57.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.831462+0000", 2024-09-11T03:45:57.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001764278 2024-09-11T03:45:57.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:57.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.831504+0000", 2024-09-11T03:45:57.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2524000000000001e-05 2024-09-11T03:45:57.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.842403+0000", 2024-09-11T03:45:57.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:57.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010899030000000001 2024-09-11T03:45:57.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.842492+0000", 2024-09-11T03:45:57.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8579999999999996e-05 2024-09-11T03:45:57.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:39.842590+0000", 2024-09-11T03:45:57.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7607000000000002e-05 2024-09-11T03:45:57.808 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.808 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.808 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.808 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1524 2.6 2:6dbd1e81:::smithi17557870-7:head [copy-from ver 123] snapc 3b=[3a,38,37,35] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-09-11T03:45:57.808 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:28.919673+0000", 2024-09-11T03:45:57.808 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 328.82326975699999, 2024-09-11T03:45:57.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017353795000000002, 2024-09-11T03:45:57.808 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.809 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.809 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:57.809 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.809 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.809 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.809 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1524 2024-09-11T03:45:57.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.809 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.919673+0000", 2024-09-11T03:45:57.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.919673+0000", 2024-09-11T03:45:57.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.919674+0000", 2024-09-11T03:45:57.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9500000000000001e-07 2024-09-11T03:45:57.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.919678+0000", 2024-09-11T03:45:57.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0929999999999996e-06 2024-09-11T03:45:57.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.919688+0000", 2024-09-11T03:45:57.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0012e-05 2024-09-11T03:45:57.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.919693+0000", 2024-09-11T03:45:57.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0420000000000002e-06 2024-09-11T03:45:57.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.919736+0000", 2024-09-11T03:45:57.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3714999999999997e-05 2024-09-11T03:45:57.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.919805+0000", 2024-09-11T03:45:57.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8703999999999998e-05 2024-09-11T03:45:57.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,8", 2024-09-11T03:45:57.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927449+0000", 2024-09-11T03:45:57.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0076441089999999996 2024-09-11T03:45:57.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.934909+0000", 2024-09-11T03:45:57.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074598859999999998 2024-09-11T03:45:57.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.936725+0000", 2024-09-11T03:45:57.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001815934 2024-09-11T03:45:57.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:45:57.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.936898+0000", 2024-09-11T03:45:57.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017337 2024-09-11T03:45:57.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:57.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.936965+0000", 2024-09-11T03:45:57.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6964999999999993e-05 2024-09-11T03:45:57.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.937027+0000", 2024-09-11T03:45:57.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1069999999999996e-05 2024-09-11T03:45:57.814 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.814 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.814 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.815 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2112 2.4 2:2ef19c88:::smithi17557870-15:head [copy-from ver 101] snapc 57=[57,56,51,4e] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-11T03:45:57.815 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:24.862975+0000", 2024-09-11T03:45:57.815 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 272.879967728, 2024-09-11T03:45:57.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017653314999999999, 2024-09-11T03:45:57.815 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.815 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.815 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:57.815 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.815 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.815 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.815 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2112 2024-09-11T03:45:57.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.816 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.862975+0000", 2024-09-11T03:45:57.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.862975+0000", 2024-09-11T03:45:57.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.862976+0000", 2024-09-11T03:45:57.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.511e-06 2024-09-11T03:45:57.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.862980+0000", 2024-09-11T03:45:57.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0709999999999996e-06 2024-09-11T03:45:57.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.862990+0000", 2024-09-11T03:45:57.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0083e-05 2024-09-11T03:45:57.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.862998+0000", 2024-09-11T03:45:57.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3080000000000002e-06 2024-09-11T03:45:57.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.863087+0000", 2024-09-11T03:45:57.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8898000000000002e-05 2024-09-11T03:45:57.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.863182+0000", 2024-09-11T03:45:57.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5585e-05 2024-09-11T03:45:57.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,2", 2024-09-11T03:45:57.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.873576+0000", 2024-09-11T03:45:57.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010394007 2024-09-11T03:45:57.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:45:57.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.880299+0000", 2024-09-11T03:45:57.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067223329999999996 2024-09-11T03:45:57.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.880455+0000", 2024-09-11T03:45:57.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015622800000000001 2024-09-11T03:45:57.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.880546+0000", 2024-09-11T03:45:57.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1131000000000006e-05 2024-09-11T03:45:57.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:57.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.880584+0000", 2024-09-11T03:45:57.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8297999999999998e-05 2024-09-11T03:45:57.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.880628+0000", 2024-09-11T03:45:57.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3862000000000001e-05 2024-09-11T03:45:57.821 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.821 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.821 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.821 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2113 2.4 2:2ef19c88:::smithi17557870-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e268)", 2024-09-11T03:45:57.821 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:24.863010+0000", 2024-09-11T03:45:57.821 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 272.87993272599999, 2024-09-11T03:45:57.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014261305, 2024-09-11T03:45:57.821 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.821 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.822 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:45:57.822 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.822 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.822 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.822 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2113 2024-09-11T03:45:57.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.822 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.863010+0000", 2024-09-11T03:45:57.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.863010+0000", 2024-09-11T03:45:57.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.863010+0000", 2024-09-11T03:45:57.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2200000000000004e-07 2024-09-11T03:45:57.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.863013+0000", 2024-09-11T03:45:57.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.356e-06 2024-09-11T03:45:57.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.863020+0000", 2024-09-11T03:45:57.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2910000000000002e-06 2024-09-11T03:45:57.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.863025+0000", 2024-09-11T03:45:57.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4600000000000002e-06 2024-09-11T03:45:57.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.863301+0000", 2024-09-11T03:45:57.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027531799999999998 2024-09-11T03:45:57.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:57.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.863332+0000", 2024-09-11T03:45:57.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1285000000000003e-05 2024-09-11T03:45:57.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.877067+0000", 2024-09-11T03:45:57.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013735511000000001 2024-09-11T03:45:57.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.877169+0000", 2024-09-11T03:45:57.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000101406 2024-09-11T03:45:57.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.877271+0000", 2024-09-11T03:45:57.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102156 2024-09-11T03:45:57.826 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.826 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.826 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.827 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:3778 2.9 2:924204df:::smithi17557870-50:head [copy-from ver 430] snapc 9d=[9d,9c,9b,9a,98,97,90,8a] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-11T03:45:57.827 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:43:42.949380+0000", 2024-09-11T03:45:57.827 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 134.79356209400001, 2024-09-11T03:45:57.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018153696, 2024-09-11T03:45:57.827 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.827 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.827 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:57.827 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.827 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.827 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.827 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3778 2024-09-11T03:45:57.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.827 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949380+0000", 2024-09-11T03:45:57.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949380+0000", 2024-09-11T03:45:57.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949382+0000", 2024-09-11T03:45:57.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.652e-06 2024-09-11T03:45:57.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949389+0000", 2024-09-11T03:45:57.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9569999999999996e-06 2024-09-11T03:45:57.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949405+0000", 2024-09-11T03:45:57.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6212e-05 2024-09-11T03:45:57.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949412+0000", 2024-09-11T03:45:57.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2099999999999996e-06 2024-09-11T03:45:57.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949483+0000", 2024-09-11T03:45:57.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0253000000000007e-05 2024-09-11T03:45:57.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949603+0000", 2024-09-11T03:45:57.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120538 2024-09-11T03:45:57.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:45:57.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.959706+0000", 2024-09-11T03:45:57.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010102629 2024-09-11T03:45:57.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:45:57.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.965469+0000", 2024-09-11T03:45:57.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005762784 2024-09-11T03:45:57.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.966495+0000", 2024-09-11T03:45:57.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001026622 2024-09-11T03:45:57.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.967289+0000", 2024-09-11T03:45:57.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00079390799999999996 2024-09-11T03:45:57.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:57.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.967429+0000", 2024-09-11T03:45:57.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013990899999999999 2024-09-11T03:45:57.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.967534+0000", 2024-09-11T03:45:57.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105022 2024-09-11T03:45:57.833 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.833 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.833 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.833 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:3779 2.9 2:924204df:::smithi17557870-50:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e405)", 2024-09-11T03:45:57.833 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:43:42.949428+0000", 2024-09-11T03:45:57.833 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 134.793514036, 2024-09-11T03:45:57.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014699339000000001, 2024-09-11T03:45:57.833 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.833 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.834 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:45:57.834 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.834 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.834 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.834 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3779 2024-09-11T03:45:57.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.834 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949428+0000", 2024-09-11T03:45:57.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949428+0000", 2024-09-11T03:45:57.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949429+0000", 2024-09-11T03:45:57.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5699999999999998e-07 2024-09-11T03:45:57.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949431+0000", 2024-09-11T03:45:57.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7940000000000001e-06 2024-09-11T03:45:57.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949436+0000", 2024-09-11T03:45:57.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.682e-06 2024-09-11T03:45:57.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949440+0000", 2024-09-11T03:45:57.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8249999999999998e-06 2024-09-11T03:45:57.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949740+0000", 2024-09-11T03:45:57.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00030021799999999999 2024-09-11T03:45:57.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:45:57.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949779+0000", 2024-09-11T03:45:57.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8679e-05 2024-09-11T03:45:57.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.964002+0000", 2024-09-11T03:45:57.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014222692 2024-09-11T03:45:57.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.964062+0000", 2024-09-11T03:45:57.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0118000000000003e-05 2024-09-11T03:45:57.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.964128+0000", 2024-09-11T03:45:57.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5974000000000002e-05 2024-09-11T03:45:57.838 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.838 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.838 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.839 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:4372 2.4 2:220bb244:::smithi17557870-34:head [read 0~3731253,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e473)", 2024-09-11T03:45:57.839 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:44:50.942772+0000", 2024-09-11T03:45:57.839 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 66.800170741000002, 2024-09-11T03:45:57.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081756590000000001, 2024-09-11T03:45:57.839 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.839 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.839 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:45:57.839 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.839 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.839 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.839 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4372 2024-09-11T03:45:57.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.839 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.942772+0000", 2024-09-11T03:45:57.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.942772+0000", 2024-09-11T03:45:57.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.942772+0000", 2024-09-11T03:45:57.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.96e-07 2024-09-11T03:45:57.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.942775+0000", 2024-09-11T03:45:57.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4099999999999998e-06 2024-09-11T03:45:57.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.942780+0000", 2024-09-11T03:45:57.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9599999999999997e-06 2024-09-11T03:45:57.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.942784+0000", 2024-09-11T03:45:57.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3000000000000002e-06 2024-09-11T03:45:57.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.942864+0000", 2024-09-11T03:45:57.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9950000000000005e-05 2024-09-11T03:45:57.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:57.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.942867+0000", 2024-09-11T03:45:57.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1190000000000001e-06 2024-09-11T03:45:57.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.950012+0000", 2024-09-11T03:45:57.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071455809999999998 2024-09-11T03:45:57.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.950080+0000", 2024-09-11T03:45:57.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7069000000000004e-05 2024-09-11T03:45:57.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.950947+0000", 2024-09-11T03:45:57.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00086787400000000003 2024-09-11T03:45:57.844 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.844 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.844 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.844 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:4373 2.4 2:220bb244:::smithi17557870-34:head [sparse-read 0~2412898] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e473)", 2024-09-11T03:45:57.844 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:44:50.942798+0000", 2024-09-11T03:45:57.844 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 66.800144286000005, 2024-09-11T03:45:57.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084339080000000004, 2024-09-11T03:45:57.844 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.844 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.844 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:45:57.845 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.845 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.845 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.845 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4373 2024-09-11T03:45:57.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.845 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.942798+0000", 2024-09-11T03:45:57.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.942798+0000", 2024-09-11T03:45:57.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.942798+0000", 2024-09-11T03:45:57.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.91e-07 2024-09-11T03:45:57.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.942800+0000", 2024-09-11T03:45:57.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6610000000000001e-06 2024-09-11T03:45:57.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.942808+0000", 2024-09-11T03:45:57.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4889999999999998e-06 2024-09-11T03:45:57.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.942816+0000", 2024-09-11T03:45:57.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0989999999999999e-06 2024-09-11T03:45:57.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.942881+0000", 2024-09-11T03:45:57.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5324e-05 2024-09-11T03:45:57.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:57.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.942884+0000", 2024-09-11T03:45:57.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3390000000000001e-06 2024-09-11T03:45:57.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.951066+0000", 2024-09-11T03:45:57.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081816930000000003 2024-09-11T03:45:57.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.951137+0000", 2024-09-11T03:45:57.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0456000000000004e-05 2024-09-11T03:45:57.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.951232+0000", 2024-09-11T03:45:57.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5556000000000005e-05 2024-09-11T03:45:57.849 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.849 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.849 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.849 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.11.0:470 2.c e473/95 2:327469ff:::smithi17557870-49:bb v 473'529, mlcod=471'525)", 2024-09-11T03:45:57.849 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:44:50.943349+0000", 2024-09-11T03:45:57.849 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 66.799593935999994, 2024-09-11T03:45:57.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087381230000000004, 2024-09-11T03:45:57.850 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.850 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.850 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:57.850 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.943349+0000", 2024-09-11T03:45:57.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.943349+0000", 2024-09-11T03:45:57.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.943349+0000", 2024-09-11T03:45:57.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.22e-07 2024-09-11T03:45:57.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.943353+0000", 2024-09-11T03:45:57.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5820000000000003e-06 2024-09-11T03:45:57.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.943374+0000", 2024-09-11T03:45:57.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0948e-05 2024-09-11T03:45:57.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.943382+0000", 2024-09-11T03:45:57.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1420000000000004e-06 2024-09-11T03:45:57.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.943435+0000", 2024-09-11T03:45:57.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3489000000000002e-05 2024-09-11T03:45:57.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.943456+0000", 2024-09-11T03:45:57.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1294999999999999e-05 2024-09-11T03:45:57.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:57.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.952049+0000", 2024-09-11T03:45:57.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085924699999999996 2024-09-11T03:45:57.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:50.952087+0000", 2024-09-11T03:45:57.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7874999999999999e-05 2024-09-11T03:45:57.854 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.854 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.854 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.854 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:4399 2.9 2:9718db5a:::smithi17557870-3:head [rollback b1] snapc be=[bc,b9,b1] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-09-11T03:45:57.854 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:44:54.978342+0000", 2024-09-11T03:45:57.854 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 62.764600352999999, 2024-09-11T03:45:57.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0077034740000000001, 2024-09-11T03:45:57.854 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.854 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.854 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:57.854 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.855 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.855 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.855 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4399 2024-09-11T03:45:57.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.855 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.978342+0000", 2024-09-11T03:45:57.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.978342+0000", 2024-09-11T03:45:57.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.978343+0000", 2024-09-11T03:45:57.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1399999999999997e-07 2024-09-11T03:45:57.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.978345+0000", 2024-09-11T03:45:57.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7049999999999999e-06 2024-09-11T03:45:57.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.978352+0000", 2024-09-11T03:45:57.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9090000000000003e-06 2024-09-11T03:45:57.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.978358+0000", 2024-09-11T03:45:57.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4149999999999998e-06 2024-09-11T03:45:57.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.978451+0000", 2024-09-11T03:45:57.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2868000000000006e-05 2024-09-11T03:45:57.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:57.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.978454+0000", 2024-09-11T03:45:57.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2310000000000001e-06 2024-09-11T03:45:57.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.981049+0000", 2024-09-11T03:45:57.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025956379999999999 2024-09-11T03:45:57.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.981143+0000", 2024-09-11T03:45:57.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3158000000000005e-05 2024-09-11T03:45:57.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:45:57.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.981362+0000", 2024-09-11T03:45:57.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021966600000000001 2024-09-11T03:45:57.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.983170+0000", 2024-09-11T03:45:57.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018080940000000001 2024-09-11T03:45:57.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:45:57.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.983184+0000", 2024-09-11T03:45:57.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3244999999999999e-05 2024-09-11T03:45:57.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.985946+0000", 2024-09-11T03:45:57.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002762764 2024-09-11T03:45:57.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:57.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.986002+0000", 2024-09-11T03:45:57.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5455999999999999e-05 2024-09-11T03:45:57.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.986046+0000", 2024-09-11T03:45:57.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3810999999999998e-05 2024-09-11T03:45:57.861 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.861 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.861 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.861 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:4400 2.9 2:9718db5a:::smithi17557870-3:head [append 0~0] snapc be=[bc,b9,b1] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-09-11T03:45:57.861 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:44:54.978437+0000", 2024-09-11T03:45:57.861 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 62.764505256, 2024-09-11T03:45:57.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00774255, 2024-09-11T03:45:57.861 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.861 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.861 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:45:57.861 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.861 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.862 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.862 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4400 2024-09-11T03:45:57.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.862 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.978437+0000", 2024-09-11T03:45:57.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.978437+0000", 2024-09-11T03:45:57.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.978438+0000", 2024-09-11T03:45:57.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4199999999999995e-07 2024-09-11T03:45:57.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.978441+0000", 2024-09-11T03:45:57.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5980000000000001e-06 2024-09-11T03:45:57.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.978451+0000", 2024-09-11T03:45:57.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7210000000000006e-06 2024-09-11T03:45:57.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.978458+0000", 2024-09-11T03:45:57.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4389999999999997e-06 2024-09-11T03:45:57.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.978513+0000", 2024-09-11T03:45:57.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5835999999999999e-05 2024-09-11T03:45:57.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:57.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.978519+0000", 2024-09-11T03:45:57.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8089999999999998e-06 2024-09-11T03:45:57.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.981948+0000", 2024-09-11T03:45:57.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034291489999999998 2024-09-11T03:45:57.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.982030+0000", 2024-09-11T03:45:57.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1747000000000002e-05 2024-09-11T03:45:57.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:45:57.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.982286+0000", 2024-09-11T03:45:57.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000255591 2024-09-11T03:45:57.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:45:57.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.985677+0000", 2024-09-11T03:45:57.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003391607 2024-09-11T03:45:57.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.985725+0000", 2024-09-11T03:45:57.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7568999999999998e-05 2024-09-11T03:45:57.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:45:57.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.986081+0000", 2024-09-11T03:45:57.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00035615400000000002 2024-09-11T03:45:57.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:45:57.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.986131+0000", 2024-09-11T03:45:57.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0015000000000001e-05 2024-09-11T03:45:57.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:54.986180+0000", 2024-09-11T03:45:57.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8673e-05 2024-09-11T03:45:57.867 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.868 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.868 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.868 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:4772 2.4 2:25f5f8b9:::smithi17557870-26:head [read 0~2000165,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e514)", 2024-09-11T03:45:57.868 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:45:31.962181+0000", 2024-09-11T03:45:57.868 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 25.780761777999999, 2024-09-11T03:45:57.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084290460000000008, 2024-09-11T03:45:57.868 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:45:57.868 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:45:57.868 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:45:57.868 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:45:57.868 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:45:57.868 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:45:57.869 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4772 2024-09-11T03:45:57.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.869 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:45:57.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:45:57.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:31.962181+0000", 2024-09-11T03:45:57.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:45:57.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:31.962181+0000", 2024-09-11T03:45:57.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:45:57.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:45:57.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:31.962181+0000", 2024-09-11T03:45:57.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4499999999999997e-07 2024-09-11T03:45:57.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:45:57.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:31.962185+0000", 2024-09-11T03:45:57.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5530000000000001e-06 2024-09-11T03:45:57.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:45:57.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:31.962201+0000", 2024-09-11T03:45:57.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6177000000000001e-05 2024-09-11T03:45:57.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:45:57.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:31.962207+0000", 2024-09-11T03:45:57.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.964e-06 2024-09-11T03:45:57.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:31.962293+0000", 2024-09-11T03:45:57.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5609e-05 2024-09-11T03:45:57.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:45:57.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:31.962298+0000", 2024-09-11T03:45:57.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.164e-06 2024-09-11T03:45:57.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:45:57.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:31.969725+0000", 2024-09-11T03:45:57.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074276990000000003 2024-09-11T03:45:57.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:45:57.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:31.969844+0000", 2024-09-11T03:45:57.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000118676 2024-09-11T03:45:57.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:45:57.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:45:57.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:45:57.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:31.970610+0000", 2024-09-11T03:45:57.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00076565900000000002 2024-09-11T03:45:57.873 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.873 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.873 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.873 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:45:57.873 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:45:57.873 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:45:57.894 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:57.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:57.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:57.956 INFO:tasks.rados.rados.2.smithi175.stdout:2006: done (1 left) 2024-09-11T03:45:57.957 INFO:tasks.rados.rados.2.smithi175.stdout:2008: done (0 left) 2024-09-11T03:45:57.957 INFO:tasks.rados.rados.2.smithi175.stdout:2009: read oid 43 snap 207 2024-09-11T03:45:57.957 INFO:tasks.rados.rados.2.smithi175.stdout:2009: expect (ObjNum 595 snap 185 seq_num 595) 2024-09-11T03:45:57.957 INFO:tasks.rados.rados.2.smithi175.stdout:2010: read oid 34 snap -1 2024-09-11T03:45:57.957 INFO:tasks.rados.rados.2.smithi175.stdout:2010: expect (ObjNum 615 snap 199 seq_num 615) 2024-09-11T03:45:57.957 INFO:tasks.rados.rados.2.smithi175.stdout:2011: snap_remove snap 200 2024-09-11T03:45:57.995 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:58.021 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:58.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:58.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:58.123 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:45:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:57 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:45:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:57 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:45:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:57 smithi084 ceph-mon[27254]: pgmap v963: 57 pgs: 57 active+clean; 200 MiB data, 840 MiB used, 983 GiB / 983 GiB avail; 2.8 MiB/s rd, 338 KiB/s wr, 4 op/s 2024-09-11T03:45:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:57 smithi084 ceph-mon[27254]: osdmap e540: 12 total, 12 up, 11 in 2024-09-11T03:45:58.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:58.145 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:58.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:45:58.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:57 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:45:58.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:57 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:45:58.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:57 smithi175 ceph-mon[30894]: pgmap v963: 57 pgs: 57 active+clean; 200 MiB data, 840 MiB used, 983 GiB / 983 GiB avail; 2.8 MiB/s rd, 338 KiB/s wr, 4 op/s 2024-09-11T03:45:58.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:57 smithi175 ceph-mon[30894]: osdmap e540: 12 total, 12 up, 11 in 2024-09-11T03:45:58.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:58.271 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:58.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:58.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:58.373 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:58.400 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:58.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:58.401 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:58.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:57 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:45:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:57 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:45:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:57 smithi121 ceph-mon[30729]: pgmap v963: 57 pgs: 57 active+clean; 200 MiB data, 840 MiB used, 983 GiB / 983 GiB avail; 2.8 MiB/s rd, 338 KiB/s wr, 4 op/s 2024-09-11T03:45:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:57 smithi121 ceph-mon[30729]: osdmap e540: 12 total, 12 up, 11 in 2024-09-11T03:45:58.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:58.527 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:58.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:58.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:58.629 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:45:58.651 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:58.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:58.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:45:58.699 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T03:45:58.753 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:45:58.779 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:58.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:58.780 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:45:58.881 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:58.906 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:58.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:58.907 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:58.912 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:45:58.921 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:45:58.921 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:45:58.946 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:45:58.947 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:45:58.959 INFO:tasks.rados.rados.2.smithi175.stdout:2011: done (2 left) 2024-09-11T03:45:58.959 INFO:tasks.rados.rados.2.smithi175.stdout:2012: snap_create 2024-09-11T03:45:58.959 INFO:tasks.rados.rados.2.smithi175.stdout:2009: expect (ObjNum 595 snap 185 seq_num 595) 2024-09-11T03:45:58.973 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:45:58.973 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:45:58.990 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:45:58.991 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:45:59.008 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:45:59.008 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:45:59.008 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:45:59.025 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:45:59.025 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:45:59.029 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:45:59.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:59.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:45:59.042 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:45:59.042 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:45:59.060 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:45:59.061 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:45:59.077 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:45:59.077 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:45:59.094 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:45:59.094 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:45:59.109 INFO:tasks.rados.rados.2.smithi175.stdout:2010: expect (ObjNum 615 snap 199 seq_num 615) 2024-09-11T03:45:59.110 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:45:59.111 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:45:59.127 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:45:59.127 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:45:59.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:45:59.155 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:59.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:59.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:45:59.257 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:45:59.278 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:59.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:59.279 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:45:59.380 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:45:59.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:58 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3861156188' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:59.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:58 smithi084 ceph-mon[27254]: osdmap e541: 12 total, 12 up, 11 in 2024-09-11T03:45:59.401 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:59.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:59.402 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:45:59.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:58 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3861156188' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:59.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:58 smithi121 ceph-mon[30729]: osdmap e541: 12 total, 12 up, 11 in 2024-09-11T03:45:59.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:58 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3861156188' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:45:59.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:58 smithi175 ceph-mon[30894]: osdmap e541: 12 total, 12 up, 11 in 2024-09-11T03:45:59.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:59.528 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:59.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:59.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:59.568 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:45:59.630 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:45:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:59.655 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:45:59.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:45:59.757 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:59.779 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:59.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:59.780 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:59.880 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:45:59.902 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:45:59.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:45:59.903 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:45:59.961 INFO:tasks.rados.rados.2.smithi175.stdout:2009: done (2 left) 2024-09-11T03:45:59.962 INFO:tasks.rados.rados.2.smithi175.stdout:2010: done (1 left) 2024-09-11T03:45:59.962 INFO:tasks.rados.rados.2.smithi175.stdout:2012: done (0 left) 2024-09-11T03:45:59.962 INFO:tasks.rados.rados.2.smithi175.stdout:2013: rollback oid 18 current snap is 225 2024-09-11T03:45:59.962 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 18 to 219 2024-09-11T03:45:59.962 INFO:tasks.rados.rados.2.smithi175.stdout:2014: snap_remove snap 224 2024-09-11T03:46:00.004 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:00.006 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:00.026 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:00.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:00.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:00.044 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:00.128 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:00.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:59 smithi084 ceph-mon[27254]: pgmap v966: 57 pgs: 57 active+clean; 202 MiB data, 845 MiB used, 983 GiB / 983 GiB avail; 7.1 MiB/s rd, 684 KiB/s wr, 8 op/s 2024-09-11T03:46:00.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:45:59 smithi084 ceph-mon[27254]: osdmap e542: 12 total, 12 up, 11 in 2024-09-11T03:46:00.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:00.152 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:00.152 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:00.253 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:00.274 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:00.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:00.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:00.376 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:00.398 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:00.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:00.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:00.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:59 smithi121 ceph-mon[30729]: pgmap v966: 57 pgs: 57 active+clean; 202 MiB data, 845 MiB used, 983 GiB / 983 GiB avail; 7.1 MiB/s rd, 684 KiB/s wr, 8 op/s 2024-09-11T03:46:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:45:59 smithi121 ceph-mon[30729]: osdmap e542: 12 total, 12 up, 11 in 2024-09-11T03:46:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:59 smithi175 ceph-mon[30894]: pgmap v966: 57 pgs: 57 active+clean; 202 MiB data, 845 MiB used, 983 GiB / 983 GiB avail; 7.1 MiB/s rd, 684 KiB/s wr, 8 op/s 2024-09-11T03:46:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:45:59 smithi175 ceph-mon[30894]: osdmap e542: 12 total, 12 up, 11 in 2024-09-11T03:46:00.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:00.521 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:00.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:00.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:00.622 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:00.648 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:00.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:00.649 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:00.750 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:00.755 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:46:00.755 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:46:00.771 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:00.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:00.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:00.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:00.899 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:00.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:00.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:00.957 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.11 to 1 (10000) 2024-09-11T03:46:00.963 INFO:tasks.rados.rados.2.smithi175.stdout:2014: done (1 left) 2024-09-11T03:46:00.963 INFO:tasks.rados.rados.2.smithi175.stdout:2015: rollback oid 12 current snap is 225 2024-09-11T03:46:00.964 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 12 to 223 2024-09-11T03:46:00.964 INFO:tasks.rados.rados.2.smithi175.stdout:2013: finishing rollback tid 1 to smithi17557870-18 2024-09-11T03:46:00.964 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 471 (ObjNum 129 snap 0 seq_num 1745240768) dirty dne 2024-09-11T03:46:00.964 INFO:tasks.rados.rados.2.smithi175.stdout:2013: done (1 left) 2024-09-11T03:46:00.964 INFO:tasks.rados.rados.2.smithi175.stdout:2016: snap_remove snap 218 2024-09-11T03:46:01.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:01.025 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:01.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:01.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:01.127 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:01.149 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:01.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:01.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:01.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:01.273 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:01.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:01.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:01.276 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:00 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1984649675' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:46:01.276 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:00 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1984649675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:46:01.276 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:00 smithi084 ceph-mon[27254]: osdmap e543: 12 total, 12 up, 11 in 2024-09-11T03:46:01.375 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:01.401 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:01.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:01.402 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:01.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:00 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1984649675' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:46:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:00 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1984649675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:46:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:00 smithi121 ceph-mon[30729]: osdmap e543: 12 total, 12 up, 11 in 2024-09-11T03:46:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:00 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1984649675' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:46:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:00 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1984649675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:46:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:00 smithi175 ceph-mon[30894]: osdmap e543: 12 total, 12 up, 11 in 2024-09-11T03:46:01.503 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:01.528 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:01.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:01.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:01.630 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:01.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 10 1 2024-09-11T03:46:01.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:01.653 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:01.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:01.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:01.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:01.776 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:01.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:01.790 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:46:01.877 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:01.902 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:01.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:01.957 INFO:tasks.rados.rados.2.smithi175.stdout:2016: done (1 left) 2024-09-11T03:46:01.958 INFO:tasks.rados.rados.2.smithi175.stdout:2017: copy_from oid 30 from oid 29 current snap is 225 2024-09-11T03:46:01.958 INFO:tasks.rados.rados.2.smithi175.stdout:2015: finishing rollback tid 0 to smithi17557870-12 2024-09-11T03:46:01.958 INFO:tasks.rados.rados.2.smithi175.stdout:2015: finishing rollback tid 1 to smithi17557870-12 2024-09-11T03:46:01.958 INFO:tasks.rados.rados.2.smithi175.stdout:2015: finishing rollback tid 2 to smithi17557870-12 2024-09-11T03:46:01.958 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 758 (ObjNum 662 snap 212 seq_num 662) dirty exists 2024-09-11T03:46:01.958 INFO:tasks.rados.rados.2.smithi175.stdout:2015: done (1 left) 2024-09-11T03:46:01.958 INFO:tasks.rados.rados.2.smithi175.stdout:2018: copy_from oid 32 from oid 35 current snap is 225 2024-09-11T03:46:01.958 INFO:tasks.rados.rados.2.smithi175.stdout:2019: snap_create 2024-09-11T03:46:01.965 INFO:tasks.rados.rados.2.smithi175.stdout:2018: finishing copy_from to smithi17557870-32 2024-09-11T03:46:01.965 INFO:tasks.rados.rados.2.smithi175.stdout:2018: got expected ENOENT (src dne) 2024-09-11T03:46:01.966 INFO:tasks.rados.rados.2.smithi175.stdout:2018: finishing copy_from racing read to smithi17557870-32 2024-09-11T03:46:01.974 INFO:tasks.rados.rados.2.smithi175.stdout:2017: finishing copy_from racing read to smithi17557870-30 2024-09-11T03:46:01.977 INFO:tasks.rados.rados.2.smithi175.stdout:2017: finishing copy_from to smithi17557870-30 2024-09-11T03:46:01.978 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 739 (ObjNum 670 snap 214 seq_num 670) dirty exists 2024-09-11T03:46:02.004 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:02.027 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:02.029 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:02.129 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:02.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:01 smithi084 ceph-mon[27254]: pgmap v969: 57 pgs: 57 active+clean; 202 MiB data, 838 MiB used, 983 GiB / 983 GiB avail; 4.2 MiB/s rd, 346 KiB/s wr, 3 op/s 2024-09-11T03:46:02.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:01 smithi084 ceph-mon[27254]: osdmap e544: 12 total, 12 up, 11 in 2024-09-11T03:46:02.151 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:02.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:02.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:02.279 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:02.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:02.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:02.358 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T03:46:02.377 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:02.381 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:02.405 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:02.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:02.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:02.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:01 smithi121 ceph-mon[30729]: pgmap v969: 57 pgs: 57 active+clean; 202 MiB data, 838 MiB used, 983 GiB / 983 GiB avail; 4.2 MiB/s rd, 346 KiB/s wr, 3 op/s 2024-09-11T03:46:02.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:01 smithi121 ceph-mon[30729]: osdmap e544: 12 total, 12 up, 11 in 2024-09-11T03:46:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:01 smithi175 ceph-mon[30894]: pgmap v969: 57 pgs: 57 active+clean; 202 MiB data, 838 MiB used, 983 GiB / 983 GiB avail; 4.2 MiB/s rd, 346 KiB/s wr, 3 op/s 2024-09-11T03:46:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:01 smithi175 ceph-mon[30894]: osdmap e544: 12 total, 12 up, 11 in 2024-09-11T03:46:02.507 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:02.533 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:02.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:02.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:02.635 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:02.661 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:02.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:02.662 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:02.763 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:02.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:02.795 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:02.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:02.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:02.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:02.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:02.954 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:46:02.959 INFO:tasks.rados.rados.2.smithi175.stdout:2017: done (2 left) 2024-09-11T03:46:02.959 INFO:tasks.rados.rados.2.smithi175.stdout:2018: done (1 left) 2024-09-11T03:46:02.959 INFO:tasks.rados.rados.2.smithi175.stdout:2019: done (0 left) 2024-09-11T03:46:02.959 INFO:tasks.rados.rados.2.smithi175.stdout:2020: read oid 5 snap 206 2024-09-11T03:46:02.959 INFO:tasks.rados.rados.2.smithi175.stdout:2020: expect (ObjNum 495 snap 143 seq_num 495) 2024-09-11T03:46:02.959 INFO:tasks.rados.rados.2.smithi175.stdout:2021: read oid 21 snap -1 2024-09-11T03:46:02.959 INFO:tasks.rados.rados.2.smithi175.stdout:2021: expect (ObjNum 584 snap 181 seq_num 584) 2024-09-11T03:46:02.960 INFO:tasks.rados.rados.2.smithi175.stdout:2022: read oid 28 snap -1 2024-09-11T03:46:02.960 INFO:tasks.rados.rados.2.smithi175.stdout:2022: expect (ObjNum 609 snap 191 seq_num 609) 2024-09-11T03:46:02.960 INFO:tasks.rados.rados.2.smithi175.stdout:2023: copy_from oid 29 from oid 47 current snap is 226 2024-09-11T03:46:02.960 INFO:tasks.rados.rados.2.smithi175.stdout:2024: read oid 32 snap -1 2024-09-11T03:46:02.960 INFO:tasks.rados.rados.2.smithi175.stdout:2024: expect (ObjNum 645 snap 208 seq_num 645) 2024-09-11T03:46:02.960 INFO:tasks.rados.rados.2.smithi175.stdout:2025: rollback oid 39 current snap is 226 2024-09-11T03:46:02.960 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 39 to 222 2024-09-11T03:46:02.960 INFO:tasks.rados.rados.2.smithi175.stdout:2026: read oid 25 snap -1 2024-09-11T03:46:02.960 INFO:tasks.rados.rados.2.smithi175.stdout:2026: expect (ObjNum 681 snap 218 seq_num 681) 2024-09-11T03:46:02.961 INFO:tasks.rados.rados.2.smithi175.stdout:2027: write (excl) oid 12 current snap is 226 2024-09-11T03:46:02.961 INFO:tasks.rados.rados.2.smithi175.stdout:2027: seq_num 688 ranges {763216=643082,1843729=687899,3260666=302550} 2024-09-11T03:46:02.961 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:02.967 INFO:tasks.rados.rados.2.smithi175.stdout:2027: writing smithi17557870-12 from 763216 to 1406298 tid 1 2024-09-11T03:46:02.967 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:02.974 INFO:tasks.rados.rados.2.smithi175.stdout:2027: writing smithi17557870-12 from 1843729 to 2531628 tid 2 2024-09-11T03:46:02.980 INFO:tasks.rados.rados.2.smithi175.stdout:2027: writing smithi17557870-12 from 3260666 to 3563216 tid 3 2024-09-11T03:46:02.980 INFO:tasks.rados.rados.2.smithi175.stdout:2028: delete oid 43 current snap is 226 2024-09-11T03:46:02.980 INFO:tasks.rados.rados.2.smithi175.stdout:2023: finishing copy_from to smithi17557870-29 2024-09-11T03:46:02.980 INFO:tasks.rados.rados.2.smithi175.stdout:2023: got expected ENOENT (src dne) 2024-09-11T03:46:02.980 INFO:tasks.rados.rados.2.smithi175.stdout:2023: finishing copy_from racing read to smithi17557870-29 2024-09-11T03:46:02.980 INFO:tasks.rados.rados.2.smithi175.stdout:2024: expect (ObjNum 645 snap 208 seq_num 645) 2024-09-11T03:46:03.024 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:03.045 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:03.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:03.046 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:03.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:03.172 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:03.172 INFO:tasks.rados.rados.2.smithi175.stdout:2021: expect (ObjNum 584 snap 181 seq_num 584) 2024-09-11T03:46:03.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:03.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:03.274 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:03.297 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:03.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:03.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:02 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1664122253' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:03.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:02 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:03.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:02 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1664122253' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:03.316 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:02 smithi084 ceph-mon[27254]: osdmap e545: 12 total, 12 up, 11 in 2024-09-11T03:46:03.342 INFO:tasks.rados.rados.2.smithi175.stdout:2025: finishing rollback tid 0 to smithi17557870-39 2024-09-11T03:46:03.342 INFO:tasks.rados.rados.2.smithi175.stdout:2025: finishing rollback tid 1 to smithi17557870-39 2024-09-11T03:46:03.342 INFO:tasks.rados.rados.2.smithi175.stdout:2025: finishing rollback tid 2 to smithi17557870-39 2024-09-11T03:46:03.342 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 612 (ObjNum 528 snap 156 seq_num 528) dirty exists 2024-09-11T03:46:03.342 INFO:tasks.rados.rados.2.smithi175.stdout:2027: finishing write tid 1 to smithi17557870-12 2024-09-11T03:46:03.343 INFO:tasks.rados.rados.2.smithi175.stdout:2022: expect (ObjNum 609 snap 191 seq_num 609) 2024-09-11T03:46:03.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:03.421 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:02 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1664122253' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:02 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:02 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1664122253' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:02 smithi121 ceph-mon[30729]: osdmap e545: 12 total, 12 up, 11 in 2024-09-11T03:46:03.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:03.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:03.443 INFO:tasks.rados.rados.2.smithi175.stdout:2020: expect (ObjNum 495 snap 143 seq_num 495) 2024-09-11T03:46:03.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:02 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1664122253' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:03.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:02 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:03.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:02 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1664122253' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:03.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:02 smithi175 ceph-mon[30894]: osdmap e545: 12 total, 12 up, 11 in 2024-09-11T03:46:03.517 INFO:tasks.rados.rados.2.smithi175.stdout:2027: finishing write tid 2 to smithi17557870-12 2024-09-11T03:46:03.517 INFO:tasks.rados.rados.2.smithi175.stdout:2026: expect (ObjNum 681 snap 218 seq_num 681) 2024-09-11T03:46:03.524 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:03.545 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:03.546 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:03.647 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:03.670 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:03.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:03.698 INFO:tasks.rados.rados.2.smithi175.stdout:2027: finishing write tid 3 to smithi17557870-12 2024-09-11T03:46:03.698 INFO:tasks.rados.rados.2.smithi175.stdout:2027: finishing write tid 4 to smithi17557870-12 2024-09-11T03:46:03.698 INFO:tasks.rados.rados.2.smithi175.stdout:2027: finishing write tid 5 to smithi17557870-12 2024-09-11T03:46:03.699 INFO:tasks.rados.rados.2.smithi175.stdout:2027: oid 12 updating version 0 to 765 2024-09-11T03:46:03.699 INFO:tasks.rados.rados.2.smithi175.stdout:2027: oid 12 version 765 is already newer than 764 2024-09-11T03:46:03.699 INFO:tasks.rados.rados.2.smithi175.stdout:2027: oid 12 version 765 is already newer than 763 2024-09-11T03:46:03.699 INFO:tasks.rados.rados.2.smithi175.stdout:2027: oid 12 version 765 is already newer than 762 2024-09-11T03:46:03.699 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 765 (ObjNum 688 snap 226 seq_num 688) dirty exists 2024-09-11T03:46:03.699 INFO:tasks.rados.rados.2.smithi175.stdout:2027: left oid 12 (ObjNum 688 snap 226 seq_num 688) 2024-09-11T03:46:03.699 INFO:tasks.rados.rados.2.smithi175.stdout:2020: done (8 left) 2024-09-11T03:46:03.699 INFO:tasks.rados.rados.2.smithi175.stdout:2021: done (7 left) 2024-09-11T03:46:03.699 INFO:tasks.rados.rados.2.smithi175.stdout:2022: done (6 left) 2024-09-11T03:46:03.699 INFO:tasks.rados.rados.2.smithi175.stdout:2023: done (5 left) 2024-09-11T03:46:03.699 INFO:tasks.rados.rados.2.smithi175.stdout:2024: done (4 left) 2024-09-11T03:46:03.699 INFO:tasks.rados.rados.2.smithi175.stdout:2025: done (3 left) 2024-09-11T03:46:03.699 INFO:tasks.rados.rados.2.smithi175.stdout:2026: done (2 left) 2024-09-11T03:46:03.699 INFO:tasks.rados.rados.2.smithi175.stdout:2027: done (1 left) 2024-09-11T03:46:03.700 INFO:tasks.rados.rados.2.smithi175.stdout:2028: done (0 left) 2024-09-11T03:46:03.700 INFO:tasks.rados.rados.2.smithi175.stdout:2029: rollback oid 22 current snap is 226 2024-09-11T03:46:03.700 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 22 to 210 2024-09-11T03:46:03.700 INFO:tasks.rados.rados.2.smithi175.stdout:2030: rollback oid 24 current snap is 226 2024-09-11T03:46:03.700 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 221 2024-09-11T03:46:03.700 INFO:tasks.rados.rados.2.smithi175.stdout:2031: write oid 42 current snap is 226 2024-09-11T03:46:03.700 INFO:tasks.rados.rados.2.smithi175.stdout:2031: seq_num 690 ranges {796830=793580,2360693=691568} 2024-09-11T03:46:03.706 INFO:tasks.rados.rados.2.smithi175.stdout:2031: writing smithi17557870-42 from 796830 to 1590410 tid 1 2024-09-11T03:46:03.714 INFO:tasks.rados.rados.2.smithi175.stdout:2031: writing smithi17557870-42 from 2360693 to 3052261 tid 2 2024-09-11T03:46:03.714 INFO:tasks.rados.rados.2.smithi175.stdout:2032: read oid 41 snap -1 2024-09-11T03:46:03.714 INFO:tasks.rados.rados.2.smithi175.stdout:2032: expect (ObjNum 609 snap 191 seq_num 609) 2024-09-11T03:46:03.715 INFO:tasks.rados.rados.2.smithi175.stdout:2033: rollback oid 25 current snap is 226 2024-09-11T03:46:03.715 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 25 to 223 2024-09-11T03:46:03.715 INFO:tasks.rados.rados.2.smithi175.stdout:2030: finishing rollback tid 0 to smithi17557870-24 2024-09-11T03:46:03.715 INFO:tasks.rados.rados.2.smithi175.stdout:2034: snap_create 2024-09-11T03:46:03.717 INFO:tasks.rados.rados.2.smithi175.stdout:2033: finishing rollback tid 0 to smithi17557870-25 2024-09-11T03:46:03.717 INFO:tasks.rados.rados.2.smithi175.stdout:2030: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:46:03.717 INFO:tasks.rados.rados.2.smithi175.stdout:2033: finishing rollback tid 1 to smithi17557870-25 2024-09-11T03:46:03.717 INFO:tasks.rados.rados.2.smithi175.stdout:2030: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:46:03.717 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 801 (ObjNum 570 snap 174 seq_num 570) dirty exists 2024-09-11T03:46:03.718 INFO:tasks.rados.rados.2.smithi175.stdout:2032: expect (ObjNum 609 snap 191 seq_num 609) 2024-09-11T03:46:03.771 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:03.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:03.798 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:03.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:03.821 INFO:tasks.rados.rados.2.smithi175.stdout:2033: finishing rollback tid 2 to smithi17557870-25 2024-09-11T03:46:03.821 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 504 (ObjNum 681 snap 218 seq_num 681) dirty exists 2024-09-11T03:46:03.821 INFO:tasks.rados.rados.2.smithi175.stdout:2029: finishing rollback tid 0 to smithi17557870-22 2024-09-11T03:46:03.821 INFO:tasks.rados.rados.2.smithi175.stdout:2029: finishing rollback tid 1 to smithi17557870-22 2024-09-11T03:46:03.821 INFO:tasks.rados.rados.2.smithi175.stdout:2029: finishing rollback tid 2 to smithi17557870-22 2024-09-11T03:46:03.821 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 796 (ObjNum 495 snap 143 seq_num 495) dirty exists 2024-09-11T03:46:03.821 INFO:tasks.rados.rados.2.smithi175.stdout:2031: finishing write tid 1 to smithi17557870-42 2024-09-11T03:46:03.821 INFO:tasks.rados.rados.2.smithi175.stdout:2031: finishing write tid 2 to smithi17557870-42 2024-09-11T03:46:03.821 INFO:tasks.rados.rados.2.smithi175.stdout:2031: finishing write tid 3 to smithi17557870-42 2024-09-11T03:46:03.821 INFO:tasks.rados.rados.2.smithi175.stdout:2031: finishing write tid 4 to smithi17557870-42 2024-09-11T03:46:03.821 INFO:tasks.rados.rados.2.smithi175.stdout:2031: oid 42 updating version 0 to 655 2024-09-11T03:46:03.821 INFO:tasks.rados.rados.2.smithi175.stdout:2031: oid 42 updating version 655 to 657 2024-09-11T03:46:03.822 INFO:tasks.rados.rados.2.smithi175.stdout:2031: oid 42 version 657 is already newer than 656 2024-09-11T03:46:03.822 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 657 (ObjNum 690 snap 226 seq_num 690) dirty exists 2024-09-11T03:46:03.822 INFO:tasks.rados.rados.2.smithi175.stdout:2031: left oid 42 (ObjNum 690 snap 226 seq_num 690) 2024-09-11T03:46:03.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:03.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:03.925 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:03.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:04.001 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.10 to 1 (10000) 2024-09-11T03:46:04.003 INFO:tasks.rados.rados.2.smithi175.stdout:2029: done (5 left) 2024-09-11T03:46:04.003 INFO:tasks.rados.rados.2.smithi175.stdout:2030: done (4 left) 2024-09-11T03:46:04.003 INFO:tasks.rados.rados.2.smithi175.stdout:2031: done (3 left) 2024-09-11T03:46:04.004 INFO:tasks.rados.rados.2.smithi175.stdout:2032: done (2 left) 2024-09-11T03:46:04.004 INFO:tasks.rados.rados.2.smithi175.stdout:2033: done (1 left) 2024-09-11T03:46:04.004 INFO:tasks.rados.rados.2.smithi175.stdout:2034: done (0 left) 2024-09-11T03:46:04.004 INFO:tasks.rados.rados.2.smithi175.stdout:2035: read oid 30 snap -1 2024-09-11T03:46:04.004 INFO:tasks.rados.rados.2.smithi175.stdout:2035: expect (ObjNum 670 snap 214 seq_num 670) 2024-09-11T03:46:04.004 INFO:tasks.rados.rados.2.smithi175.stdout:2036: copy_from oid 4 from oid 1 current snap is 227 2024-09-11T03:46:04.004 INFO:tasks.rados.rados.2.smithi175.stdout:2037: delete oid 42 current snap is 227 2024-09-11T03:46:04.007 INFO:tasks.rados.rados.2.smithi175.stdout:2037: done (2 left) 2024-09-11T03:46:04.007 INFO:tasks.rados.rados.2.smithi175.stdout:2038: delete oid 45 current snap is 227 2024-09-11T03:46:04.009 INFO:tasks.rados.rados.2.smithi175.stdout:2038: done (2 left) 2024-09-11T03:46:04.010 INFO:tasks.rados.rados.2.smithi175.stdout:2039: write (excl) oid 42 current snap is 227 2024-09-11T03:46:04.010 INFO:tasks.rados.rados.2.smithi175.stdout:2039: seq_num 693 ranges {447251=735680,1736139=438895,2597781=780799} 2024-09-11T03:46:04.017 INFO:tasks.rados.rados.2.smithi175.stdout:2039: writing smithi17557870-42 from 447251 to 1182931 tid 1 2024-09-11T03:46:04.022 INFO:tasks.rados.rados.2.smithi175.stdout:2039: writing smithi17557870-42 from 1736139 to 2175034 tid 2 2024-09-11T03:46:04.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:04.029 INFO:tasks.rados.rados.2.smithi175.stdout:2039: writing smithi17557870-42 from 2597781 to 3378580 tid 3 2024-09-11T03:46:04.030 INFO:tasks.rados.rados.2.smithi175.stdout:2040: read oid 34 snap -1 2024-09-11T03:46:04.030 INFO:tasks.rados.rados.2.smithi175.stdout:2040: expect (ObjNum 615 snap 199 seq_num 615) 2024-09-11T03:46:04.030 INFO:tasks.rados.rados.2.smithi175.stdout:2041: rollback oid 49 current snap is 227 2024-09-11T03:46:04.030 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 49 to 221 2024-09-11T03:46:04.030 INFO:tasks.rados.rados.2.smithi175.stdout:2042: delete oid 31 current snap is 227 2024-09-11T03:46:04.030 INFO:tasks.rados.rados.2.smithi175.stdout:2036: finishing copy_from racing read to smithi17557870-4 2024-09-11T03:46:04.030 INFO:tasks.rados.rados.2.smithi175.stdout:2036: finishing copy_from to smithi17557870-4 2024-09-11T03:46:04.030 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 746 (ObjNum 676 snap 218 seq_num 676) dirty exists 2024-09-11T03:46:04.030 INFO:tasks.rados.rados.2.smithi175.stdout:2035: expect (ObjNum 670 snap 214 seq_num 670) 2024-09-11T03:46:04.049 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:04.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:04.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:04.150 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:04.172 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:04.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:04.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:04.177 INFO:tasks.rados.rados.2.smithi175.stdout:2039: finishing write tid 1 to smithi17557870-42 2024-09-11T03:46:04.177 INFO:tasks.rados.rados.2.smithi175.stdout:2039: finishing write tid 2 to smithi17557870-42 2024-09-11T03:46:04.177 INFO:tasks.rados.rados.2.smithi175.stdout:2039: finishing write tid 3 to smithi17557870-42 2024-09-11T03:46:04.177 INFO:tasks.rados.rados.2.smithi175.stdout:2039: finishing write tid 4 to smithi17557870-42 2024-09-11T03:46:04.177 INFO:tasks.rados.rados.2.smithi175.stdout:2040: expect (ObjNum 615 snap 199 seq_num 615) 2024-09-11T03:46:04.274 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:04.284 INFO:tasks.rados.rados.2.smithi175.stdout:2039: finishing write tid 5 to smithi17557870-42 2024-09-11T03:46:04.284 INFO:tasks.rados.rados.2.smithi175.stdout:2039: oid 42 updating version 0 to 660 2024-09-11T03:46:04.284 INFO:tasks.rados.rados.2.smithi175.stdout:2039: oid 42 updating version 660 to 663 2024-09-11T03:46:04.284 INFO:tasks.rados.rados.2.smithi175.stdout:2039: oid 42 version 663 is already newer than 662 2024-09-11T03:46:04.284 INFO:tasks.rados.rados.2.smithi175.stdout:2039: oid 42 version 663 is already newer than 661 2024-09-11T03:46:04.284 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 663 (ObjNum 693 snap 227 seq_num 693) dirty exists 2024-09-11T03:46:04.284 INFO:tasks.rados.rados.2.smithi175.stdout:2039: left oid 42 (ObjNum 693 snap 227 seq_num 693) 2024-09-11T03:46:04.284 INFO:tasks.rados.rados.2.smithi175.stdout:2041: finishing rollback tid 1 to smithi17557870-49 2024-09-11T03:46:04.285 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 654 (ObjNum 33 snap 0 seq_num 1713067256) dirty dne 2024-09-11T03:46:04.286 INFO:tasks.rados.rados.2.smithi175.stdout:2035: done (5 left) 2024-09-11T03:46:04.286 INFO:tasks.rados.rados.2.smithi175.stdout:2036: done (4 left) 2024-09-11T03:46:04.286 INFO:tasks.rados.rados.2.smithi175.stdout:2039: done (3 left) 2024-09-11T03:46:04.286 INFO:tasks.rados.rados.2.smithi175.stdout:2040: done (2 left) 2024-09-11T03:46:04.286 INFO:tasks.rados.rados.2.smithi175.stdout:2041: done (1 left) 2024-09-11T03:46:04.286 INFO:tasks.rados.rados.2.smithi175.stdout:2042: done (0 left) 2024-09-11T03:46:04.286 INFO:tasks.rados.rados.2.smithi175.stdout:2043: snap_remove snap 219 2024-09-11T03:46:04.294 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:04.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:04.295 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:04.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:03 smithi084 ceph-mon[27254]: pgmap v972: 57 pgs: 57 active+clean; 175 MiB data, 804 MiB used, 983 GiB / 983 GiB avail; 5.4 MiB/s rd, 0 B/s wr, 5 op/s 2024-09-11T03:46:04.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:03 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/42490827' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:46:04.396 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:04.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:03 smithi121 ceph-mon[30729]: pgmap v972: 57 pgs: 57 active+clean; 175 MiB data, 804 MiB used, 983 GiB / 983 GiB avail; 5.4 MiB/s rd, 0 B/s wr, 5 op/s 2024-09-11T03:46:04.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:03 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/42490827' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:46:04.422 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:04.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:04.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:03 smithi175 ceph-mon[30894]: pgmap v972: 57 pgs: 57 active+clean; 175 MiB data, 804 MiB used, 983 GiB / 983 GiB avail; 5.4 MiB/s rd, 0 B/s wr, 5 op/s 2024-09-11T03:46:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:03 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/42490827' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:46:04.524 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:04.549 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:04.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:04.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:04.651 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:04.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:04.673 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:04.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:04.774 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:04.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:04.797 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:04.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:04.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:04.924 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:04.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:04.929 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T03:46:04.930 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:46:05.000 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:46:05.000 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:46:05.011 INFO:tasks.rados.rados.2.smithi175.stdout:2043: done (0 left) 2024-09-11T03:46:05.011 INFO:tasks.rados.rados.2.smithi175.stdout:2044: read oid 19 snap 221 2024-09-11T03:46:05.011 INFO:tasks.rados.rados.2.smithi175.stdout:2044: expect (ObjNum 565 snap 172 seq_num 565) 2024-09-11T03:46:05.011 INFO:tasks.rados.rados.2.smithi175.stdout:2045: copy_from oid 30 from oid 35 current snap is 227 2024-09-11T03:46:05.012 INFO:tasks.rados.rados.2.smithi175.stdout:2046: read oid 28 snap -1 2024-09-11T03:46:05.012 INFO:tasks.rados.rados.2.smithi175.stdout:2046: expect (ObjNum 609 snap 191 seq_num 609) 2024-09-11T03:46:05.012 INFO:tasks.rados.rados.2.smithi175.stdout:2047: write oid 22 current snap is 227 2024-09-11T03:46:05.012 INFO:tasks.rados.rados.2.smithi175.stdout:2047: seq_num 695 ranges {480865=486178,1453103=442564,2652627=519696} 2024-09-11T03:46:05.017 INFO:tasks.rados.rados.2.smithi175.stdout:2047: writing smithi17557870-22 from 480865 to 967043 tid 1 2024-09-11T03:46:05.022 INFO:tasks.rados.rados.2.smithi175.stdout:2047: writing smithi17557870-22 from 1453103 to 1895667 tid 2 2024-09-11T03:46:05.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:05.029 INFO:tasks.rados.rados.2.smithi175.stdout:2047: writing smithi17557870-22 from 2652627 to 3172323 tid 3 2024-09-11T03:46:05.029 INFO:tasks.rados.rados.2.smithi175.stdout:2048: delete oid 3 current snap is 227 2024-09-11T03:46:05.029 INFO:tasks.rados.rados.2.smithi175.stdout:2045: finishing copy_from to smithi17557870-30 2024-09-11T03:46:05.029 INFO:tasks.rados.rados.2.smithi175.stdout:2045: got expected ENOENT (src dne) 2024-09-11T03:46:05.030 INFO:tasks.rados.rados.2.smithi175.stdout:2045: finishing copy_from racing read to smithi17557870-30 2024-09-11T03:46:05.030 INFO:tasks.rados.rados.2.smithi175.stdout:2046: expect (ObjNum 609 snap 191 seq_num 609) 2024-09-11T03:46:05.048 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:05.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:05.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:05.133 INFO:tasks.rados.rados.2.smithi175.stdout:2047: finishing write tid 1 to smithi17557870-22 2024-09-11T03:46:05.133 INFO:tasks.rados.rados.2.smithi175.stdout:2047: finishing write tid 2 to smithi17557870-22 2024-09-11T03:46:05.133 INFO:tasks.rados.rados.2.smithi175.stdout:2044: expect (ObjNum 565 snap 172 seq_num 565) 2024-09-11T03:46:05.150 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:05.174 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:05.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:05.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:05.276 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:05.293 INFO:tasks.rados.rados.2.smithi175.stdout:2047: finishing write tid 3 to smithi17557870-22 2024-09-11T03:46:05.293 INFO:tasks.rados.rados.2.smithi175.stdout:2047: finishing write tid 4 to smithi17557870-22 2024-09-11T03:46:05.293 INFO:tasks.rados.rados.2.smithi175.stdout:2047: finishing write tid 5 to smithi17557870-22 2024-09-11T03:46:05.293 INFO:tasks.rados.rados.2.smithi175.stdout:2047: oid 22 updating version 0 to 800 2024-09-11T03:46:05.293 INFO:tasks.rados.rados.2.smithi175.stdout:2047: oid 22 updating version 800 to 802 2024-09-11T03:46:05.293 INFO:tasks.rados.rados.2.smithi175.stdout:2047: oid 22 updating version 802 to 803 2024-09-11T03:46:05.294 INFO:tasks.rados.rados.2.smithi175.stdout:2047: oid 22 version 803 is already newer than 801 2024-09-11T03:46:05.294 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 803 (ObjNum 695 snap 227 seq_num 695) dirty exists 2024-09-11T03:46:05.294 INFO:tasks.rados.rados.2.smithi175.stdout:2047: left oid 22 (ObjNum 695 snap 227 seq_num 695) 2024-09-11T03:46:05.294 INFO:tasks.rados.rados.2.smithi175.stdout:2044: done (4 left) 2024-09-11T03:46:05.294 INFO:tasks.rados.rados.2.smithi175.stdout:2045: done (3 left) 2024-09-11T03:46:05.294 INFO:tasks.rados.rados.2.smithi175.stdout:2046: done (2 left) 2024-09-11T03:46:05.294 INFO:tasks.rados.rados.2.smithi175.stdout:2047: done (1 left) 2024-09-11T03:46:05.294 INFO:tasks.rados.rados.2.smithi175.stdout:2048: done (0 left) 2024-09-11T03:46:05.294 INFO:tasks.rados.rados.2.smithi175.stdout:2049: copy_from oid 29 from oid 25 current snap is 227 2024-09-11T03:46:05.294 INFO:tasks.rados.rados.2.smithi175.stdout:2050: write oid 40 current snap is 227 2024-09-11T03:46:05.294 INFO:tasks.rados.rados.2.smithi175.stdout:2050: seq_num 697 ranges {514479=636676,1853714=446233,2991120=542240} 2024-09-11T03:46:05.298 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:05.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:05.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:05.299 INFO:tasks.rados.rados.2.smithi175.stdout:2050: writing smithi17557870-40 from 514479 to 1151155 tid 1 2024-09-11T03:46:05.305 INFO:tasks.rados.rados.2.smithi175.stdout:2050: writing smithi17557870-40 from 1853714 to 2299947 tid 2 2024-09-11T03:46:05.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:05 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/42490827' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:46:05.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:05 smithi084 ceph-mon[27254]: osdmap e546: 12 total, 12 up, 11 in 2024-09-11T03:46:05.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:05 smithi084 ceph-mon[27254]: 1.16 scrub starts 2024-09-11T03:46:05.312 INFO:tasks.rados.rados.2.smithi175.stdout:2050: writing smithi17557870-40 from 2991120 to 3533360 tid 3 2024-09-11T03:46:05.313 INFO:tasks.rados.rados.2.smithi175.stdout:2051: copy_from oid 24 from oid 35 current snap is 227 2024-09-11T03:46:05.313 INFO:tasks.rados.rados.2.smithi175.stdout:2050: finishing write tid 1 to smithi17557870-40 2024-09-11T03:46:05.313 INFO:tasks.rados.rados.2.smithi175.stdout:2049: finishing copy_from racing read to smithi17557870-29 2024-09-11T03:46:05.313 INFO:tasks.rados.rados.2.smithi175.stdout:2052: rollback oid 33 current snap is 227 2024-09-11T03:46:05.313 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 33 to 225 2024-09-11T03:46:05.313 INFO:tasks.rados.rados.2.smithi175.stdout:2050: finishing write tid 2 to smithi17557870-40 2024-09-11T03:46:05.313 INFO:tasks.rados.rados.2.smithi175.stdout:2053: write (excl) oid 5 current snap is 227 2024-09-11T03:46:05.313 INFO:tasks.rados.rados.2.smithi175.stdout:2053: seq_num 698 ranges {531286=711925,1912196=706244,3018543=712743} 2024-09-11T03:46:05.319 INFO:tasks.rados.rados.2.smithi175.stdout:2053: writing smithi17557870-5 from 531286 to 1243211 tid 1 2024-09-11T03:46:05.326 INFO:tasks.rados.rados.2.smithi175.stdout:2053: writing smithi17557870-5 from 1912196 to 2618440 tid 2 2024-09-11T03:46:05.333 INFO:tasks.rados.rados.2.smithi175.stdout:2053: writing smithi17557870-5 from 3018543 to 3731286 tid 3 2024-09-11T03:46:05.333 INFO:tasks.rados.rados.2.smithi175.stdout:2054: read oid 32 snap 210 2024-09-11T03:46:05.333 INFO:tasks.rados.rados.2.smithi175.stdout:2054: expect (ObjNum 645 snap 208 seq_num 645) 2024-09-11T03:46:05.333 INFO:tasks.rados.rados.2.smithi175.stdout:2055: read oid 2 snap -1 2024-09-11T03:46:05.333 INFO:tasks.rados.rados.2.smithi175.stdout:2055: expect (ObjNum 684 snap 220 seq_num 684) 2024-09-11T03:46:05.333 INFO:tasks.rados.rados.2.smithi175.stdout:2056: delete oid 36 current snap is 227 2024-09-11T03:46:05.333 INFO:tasks.rados.rados.2.smithi175.stdout:2049: finishing copy_from to smithi17557870-29 2024-09-11T03:46:05.334 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 504 (ObjNum 681 snap 218 seq_num 681) dirty exists 2024-09-11T03:46:05.334 INFO:tasks.rados.rados.2.smithi175.stdout:2051: finishing copy_from to smithi17557870-24 2024-09-11T03:46:05.334 INFO:tasks.rados.rados.2.smithi175.stdout:2051: got expected ENOENT (src dne) 2024-09-11T03:46:05.334 INFO:tasks.rados.rados.2.smithi175.stdout:2051: finishing copy_from racing read to smithi17557870-24 2024-09-11T03:46:05.334 INFO:tasks.rados.rados.2.smithi175.stdout:2050: finishing write tid 3 to smithi17557870-40 2024-09-11T03:46:05.334 INFO:tasks.rados.rados.2.smithi175.stdout:2050: finishing write tid 4 to smithi17557870-40 2024-09-11T03:46:05.334 INFO:tasks.rados.rados.2.smithi175.stdout:2052: finishing rollback tid 0 to smithi17557870-33 2024-09-11T03:46:05.334 INFO:tasks.rados.rados.2.smithi175.stdout:2050: finishing write tid 5 to smithi17557870-40 2024-09-11T03:46:05.334 INFO:tasks.rados.rados.2.smithi175.stdout:2050: oid 40 updating version 0 to 412 2024-09-11T03:46:05.334 INFO:tasks.rados.rados.2.smithi175.stdout:2050: oid 40 updating version 412 to 413 2024-09-11T03:46:05.334 INFO:tasks.rados.rados.2.smithi175.stdout:2050: oid 40 version 413 is already newer than 411 2024-09-11T03:46:05.334 INFO:tasks.rados.rados.2.smithi175.stdout:2050: oid 40 updating version 413 to 414 2024-09-11T03:46:05.334 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 414 (ObjNum 697 snap 227 seq_num 697) dirty exists 2024-09-11T03:46:05.334 INFO:tasks.rados.rados.2.smithi175.stdout:2050: left oid 40 (ObjNum 697 snap 227 seq_num 697) 2024-09-11T03:46:05.334 INFO:tasks.rados.rados.2.smithi175.stdout:2052: finishing rollback tid 1 to smithi17557870-33 2024-09-11T03:46:05.335 INFO:tasks.rados.rados.2.smithi175.stdout:2052: finishing rollback tid 2 to smithi17557870-33 2024-09-11T03:46:05.335 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 809 (ObjNum 630 snap 202 seq_num 630) dirty exists 2024-09-11T03:46:05.335 INFO:tasks.rados.rados.2.smithi175.stdout:2053: finishing write tid 1 to smithi17557870-5 2024-09-11T03:46:05.335 INFO:tasks.rados.rados.2.smithi175.stdout:2053: finishing write tid 2 to smithi17557870-5 2024-09-11T03:46:05.336 INFO:tasks.rados.rados.2.smithi175.stdout:2049: done (7 left) 2024-09-11T03:46:05.336 INFO:tasks.rados.rados.2.smithi175.stdout:2050: done (6 left) 2024-09-11T03:46:05.336 INFO:tasks.rados.rados.2.smithi175.stdout:2051: done (5 left) 2024-09-11T03:46:05.336 INFO:tasks.rados.rados.2.smithi175.stdout:2052: done (4 left) 2024-09-11T03:46:05.336 INFO:tasks.rados.rados.2.smithi175.stdout:2056: done (3 left) 2024-09-11T03:46:05.336 INFO:tasks.rados.rados.2.smithi175.stdout:2057: snap_create 2024-09-11T03:46:05.336 INFO:tasks.rados.rados.2.smithi175.stdout:2053: finishing write tid 3 to smithi17557870-5 2024-09-11T03:46:05.336 INFO:tasks.rados.rados.2.smithi175.stdout:2053: finishing write tid 4 to smithi17557870-5 2024-09-11T03:46:05.336 INFO:tasks.rados.rados.2.smithi175.stdout:2053: finishing write tid 5 to smithi17557870-5 2024-09-11T03:46:05.336 INFO:tasks.rados.rados.2.smithi175.stdout:2053: oid 5 updating version 0 to 690 2024-09-11T03:46:05.336 INFO:tasks.rados.rados.2.smithi175.stdout:2053: oid 5 updating version 690 to 693 2024-09-11T03:46:05.336 INFO:tasks.rados.rados.2.smithi175.stdout:2053: oid 5 version 693 is already newer than 692 2024-09-11T03:46:05.337 INFO:tasks.rados.rados.2.smithi175.stdout:2053: oid 5 version 693 is already newer than 691 2024-09-11T03:46:05.337 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 693 (ObjNum 698 snap 227 seq_num 698) dirty exists 2024-09-11T03:46:05.337 INFO:tasks.rados.rados.2.smithi175.stdout:2053: left oid 5 (ObjNum 698 snap 227 seq_num 698) 2024-09-11T03:46:05.337 INFO:tasks.rados.rados.2.smithi175.stdout:2054: expect (ObjNum 645 snap 208 seq_num 645) 2024-09-11T03:46:05.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:05.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:05 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/42490827' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:46:05.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:05 smithi121 ceph-mon[30729]: osdmap e546: 12 total, 12 up, 11 in 2024-09-11T03:46:05.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:05 smithi121 ceph-mon[30729]: 1.16 scrub starts 2024-09-11T03:46:05.424 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:05.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:05.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:05 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/42490827' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:46:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:05 smithi175 ceph-mon[30894]: osdmap e546: 12 total, 12 up, 11 in 2024-09-11T03:46:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:05 smithi175 ceph-mon[30894]: 1.16 scrub starts 2024-09-11T03:46:05.524 INFO:tasks.rados.rados.2.smithi175.stdout:2055: expect (ObjNum 684 snap 220 seq_num 684) 2024-09-11T03:46:05.525 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:05.548 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:05.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:05.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:05.650 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:05.672 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:05.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:05.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:05.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:05.778 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T03:46:05.799 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:05.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:05.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:05.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:05.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:05.927 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:05.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:06.013 INFO:tasks.rados.rados.2.smithi175.stdout:2053: done (3 left) 2024-09-11T03:46:06.013 INFO:tasks.rados.rados.2.smithi175.stdout:2054: done (2 left) 2024-09-11T03:46:06.014 INFO:tasks.rados.rados.2.smithi175.stdout:2055: done (1 left) 2024-09-11T03:46:06.014 INFO:tasks.rados.rados.2.smithi175.stdout:2057: done (0 left) 2024-09-11T03:46:06.014 INFO:tasks.rados.rados.2.smithi175.stdout:2058: snap_remove snap 225 2024-09-11T03:46:06.028 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:06.052 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:06.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:06.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:06.056 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:46:06.056 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:46:06.075 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:46:06.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:46:06.080 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:46:06.098 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:46:06.098 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:46:06.100 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:06.117 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:46:06.117 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:46:06.136 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:06.139 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:46:06.140 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:46:06.154 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:06.164 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:46:06.164 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:46:06.175 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:06.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:06.184 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:46:06.184 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:46:06.199 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:46:06.199 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:46:06.216 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:46:06.216 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:46:06.233 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:46:06.233 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:46:06.249 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:46:06.249 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:46:06.262 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:06 smithi084 ceph-mon[27254]: pgmap v975: 57 pgs: 57 active+clean; 169 MiB data, 791 MiB used, 983 GiB / 983 GiB avail; 7.1 MiB/s rd, 796 KiB/s wr, 7 op/s 2024-09-11T03:46:06.262 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:06 smithi084 ceph-mon[27254]: osdmap e547: 12 total, 12 up, 11 in 2024-09-11T03:46:06.267 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:46:06.267 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:46:06.277 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:06.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:06.300 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:06.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:06.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:06.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:06 smithi121 ceph-mon[30729]: pgmap v975: 57 pgs: 57 active+clean; 169 MiB data, 791 MiB used, 983 GiB / 983 GiB avail; 7.1 MiB/s rd, 796 KiB/s wr, 7 op/s 2024-09-11T03:46:06.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:06 smithi121 ceph-mon[30729]: osdmap e547: 12 total, 12 up, 11 in 2024-09-11T03:46:06.427 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:06.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:06.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:06.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:06 smithi175 ceph-mon[30894]: pgmap v975: 57 pgs: 57 active+clean; 169 MiB data, 791 MiB used, 983 GiB / 983 GiB avail; 7.1 MiB/s rd, 796 KiB/s wr, 7 op/s 2024-09-11T03:46:06.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:06 smithi175 ceph-mon[30894]: osdmap e547: 12 total, 12 up, 11 in 2024-09-11T03:46:06.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:06.555 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:06.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:06.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:06.682 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:06.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:06.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:06.784 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:06.807 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:06.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:06.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:06.909 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:06.920 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:46:06.921 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:46:06.935 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:06.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:06.937 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:06.960 INFO:tasks.rados.rados.2.smithi175.stdout:2058: done (0 left) 2024-09-11T03:46:06.960 INFO:tasks.rados.rados.2.smithi175.stdout:2059: snap_remove snap 210 2024-09-11T03:46:07.037 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:07.085 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:07.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:07.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:07.187 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:07.208 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:07.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:07.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:07.306 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:07 smithi084 ceph-mon[27254]: osdmap e548: 12 total, 12 up, 11 in 2024-09-11T03:46:07.307 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/512239304' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:07.307 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:07 smithi084 ceph-mon[27254]: osdmap e549: 12 total, 12 up, 11 in 2024-09-11T03:46:07.313 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:07.339 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:07.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:07.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:07 smithi121 ceph-mon[30729]: osdmap e548: 12 total, 12 up, 11 in 2024-09-11T03:46:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/512239304' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:07 smithi121 ceph-mon[30729]: osdmap e549: 12 total, 12 up, 11 in 2024-09-11T03:46:07.441 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:07.464 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:07.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:07 smithi175 ceph-mon[30894]: osdmap e548: 12 total, 12 up, 11 in 2024-09-11T03:46:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/512239304' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:07 smithi175 ceph-mon[30894]: osdmap e549: 12 total, 12 up, 11 in 2024-09-11T03:46:07.565 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:07.591 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:07.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:07.592 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:07.693 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:07.715 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:07.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:07.730 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":978,"stamp":"2024-09-11T03:46:06.789134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":188233860,"num_objects":96,"num_object_clones":52,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":9,"num_read":2844,"num_read_kb":2728967,"num_write":1982,"num_write_kb":761100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":61873709,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13207,"ondisk_log_size":13207,"up":171,"acting":171,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":171,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":798900,"kb_used_data":412768,"kb_used_omap":17,"kb_used_meta":385838,"kb_avail":1030397772,"statfs":{"total":1055945392128,"available":1055127318528,"internally_reserved":0,"allocated":422674432,"data_stored":500123192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":395098997},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4362343,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":-1,"num_read":86,"num_read_kb":93298,"num_write":22,"num_write_kb":12846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000772"},"pg_stats":[{"pgid":"1.1e","version":"20'92","reported_seq":672,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-09-11T03:46:01.964798+0000","last_change":"2024-09-11T03:44:24.235660+0000","last_active":"2024-09-11T03:46:01.964798+0000","last_peered":"2024-09-11T03:46:01.964798+0000","last_clean":"2024-09-11T03:46:01.964798+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:46:01.964798+0000","last_undegraded":"2024-09-11T03:46:01.964798+0000","last_fullsized":"2024-09-11T03:46:01.964798+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:24.235616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:24.235616+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:57.069193+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'92","reported_seq":668,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365675+0000","last_change":"2024-09-11T03:40:41.348889+0000","last_active":"2024-09-11T03:46:06.365675+0000","last_peered":"2024-09-11T03:46:06.365675+0000","last_clean":"2024-09-11T03:46:06.365675+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:46:06.365675+0000","last_undegraded":"2024-09-11T03:46:06.365675+0000","last_fullsized":"2024-09-11T03:46:06.365675+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:41.348828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:41.348828+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:20.867494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'92","reported_seq":671,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013215+0000","last_change":"2024-09-11T03:40:40.188174+0000","last_active":"2024-09-11T03:46:06.013215+0000","last_peered":"2024-09-11T03:46:06.013215+0000","last_clean":"2024-09-11T03:46:06.013215+0000","last_became_active":"2024-09-11T03:40:39.848230+0000","last_became_peered":"2024-09-11T03:40:39.848230+0000","last_unstale":"2024-09-11T03:46:06.013215+0000","last_undegraded":"2024-09-11T03:46:06.013215+0000","last_fullsized":"2024-09-11T03:46:06.013215+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.188124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.188124+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:19.492283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"541'356","reported_seq":2693,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-11T03:46:03.044893+0000","last_change":"2024-09-11T03:46:01.964215+0000","last_active":"2024-09-11T03:46:03.044893+0000","last_peered":"2024-09-11T03:46:03.044893+0000","last_clean":"2024-09-11T03:46:03.044893+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:46:03.044893+0000","last_undegraded":"2024-09-11T03:46:03.044893+0000","last_fullsized":"2024-09-11T03:46:03.044893+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"467'328","last_scrub_stamp":"2024-09-11T03:44:45.863666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:45.863666+0000","objects_scrubbed":4,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001139482,"stat_sum":{"num_bytes":2739673,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":99504,"num_write":46,"num_write_kb":16817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4865740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"2"},{"start":"e2","length":"1"}]},{"pgid":"1.1b","version":"20'92","reported_seq":671,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013245+0000","last_change":"2024-09-11T03:40:42.166354+0000","last_active":"2024-09-11T03:46:06.013245+0000","last_peered":"2024-09-11T03:46:06.013245+0000","last_clean":"2024-09-11T03:46:06.013245+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:46:06.013245+0000","last_undegraded":"2024-09-11T03:46:06.013245+0000","last_fullsized":"2024-09-11T03:46:06.013245+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:42.166295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:42.166295+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:21.519903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.19","version":"541'445","reported_seq":2811,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-09-11T03:46:05.020101+0000","last_change":"2024-09-11T03:46:05.020677+0000","last_active":"2024-09-11T03:46:05.020101+0000","last_peered":"2024-09-11T03:46:05.020101+0000","last_clean":"2024-09-11T03:46:05.020101+0000","last_became_active":"2024-09-11T03:45:09.975629+0000","last_became_peered":"2024-09-11T03:45:09.975629+0000","last_unstale":"2024-09-11T03:46:05.020101+0000","last_undegraded":"2024-09-11T03:46:05.020101+0000","last_fullsized":"2024-09-11T03:46:05.020101+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"504'431","last_scrub_stamp":"2024-09-11T03:45:23.194125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:23.194125+0000","objects_scrubbed":2,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:45.065837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00081455700000000004,"stat_sum":{"num_bytes":4480490,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":45909,"num_write":25,"num_write_kb":12177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,8],"acting":[0,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.1a","version":"20'92","reported_seq":434,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365236+0000","last_change":"2024-09-11T03:45:52.914136+0000","last_active":"2024-09-11T03:46:06.365236+0000","last_peered":"2024-09-11T03:46:06.365236+0000","last_clean":"2024-09-11T03:46:06.365236+0000","last_became_active":"2024-09-11T03:44:43.938873+0000","last_became_peered":"2024-09-11T03:44:43.938873+0000","last_unstale":"2024-09-11T03:46:06.365236+0000","last_undegraded":"2024-09-11T03:46:06.365236+0000","last_fullsized":"2024-09-11T03:46:06.365236+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:52.914088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:52.914088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:10.451317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"20'92","reported_seq":286,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-09-11T03:46:01.956054+0000","last_change":"2024-09-11T03:45:30.424485+0000","last_active":"2024-09-11T03:46:01.956054+0000","last_peered":"2024-09-11T03:46:01.956054+0000","last_clean":"2024-09-11T03:46:01.956054+0000","last_became_active":"2024-09-11T03:43:52.920761+0000","last_became_peered":"2024-09-11T03:43:52.920761+0000","last_unstale":"2024-09-11T03:46:01.956054+0000","last_undegraded":"2024-09-11T03:46:01.956054+0000","last_fullsized":"2024-09-11T03:46:01.956054+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:30.424428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:30.424428+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:57.891319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'92","reported_seq":803,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365642+0000","last_change":"2024-09-11T03:44:46.470994+0000","last_active":"2024-09-11T03:46:06.365642+0000","last_peered":"2024-09-11T03:46:06.365642+0000","last_clean":"2024-09-11T03:46:06.365642+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:46:06.365642+0000","last_undegraded":"2024-09-11T03:46:06.365642+0000","last_fullsized":"2024-09-11T03:46:06.365642+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:46.470921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:46.470921+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:13.212885+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"541'254","reported_seq":2539,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-11T03:46:03.045317+0000","last_change":"2024-09-11T03:46:01.965340+0000","last_active":"2024-09-11T03:46:03.045317+0000","last_peered":"2024-09-11T03:46:03.045317+0000","last_clean":"2024-09-11T03:46:03.045317+0000","last_became_active":"2024-09-11T03:41:38.890690+0000","last_became_peered":"2024-09-11T03:41:38.890690+0000","last_unstale":"2024-09-11T03:46:03.045317+0000","last_undegraded":"2024-09-11T03:46:03.045317+0000","last_fullsized":"2024-09-11T03:46:03.045317+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"437'223","last_scrub_stamp":"2024-09-11T03:44:42.964833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:42.964833+0000","objects_scrubbed":1,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:05.794222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001743872,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":8231,"num_write":22,"num_write_kb":4725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[9,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"2"},{"start":"e2","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":694,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013484+0000","last_change":"2024-09-11T03:45:07.446084+0000","last_active":"2024-09-11T03:46:06.013484+0000","last_peered":"2024-09-11T03:46:06.013484+0000","last_clean":"2024-09-11T03:46:06.013484+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:46:06.013484+0000","last_undegraded":"2024-09-11T03:46:06.013484+0000","last_fullsized":"2024-09-11T03:46:06.013484+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:07.445986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:07.445986+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:11.836438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":2581,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-11T03:46:03.045357+0000","last_change":"2024-09-11T03:46:01.964398+0000","last_active":"2024-09-11T03:46:03.045357+0000","last_peered":"2024-09-11T03:46:03.045357+0000","last_clean":"2024-09-11T03:46:03.045357+0000","last_became_active":"2024-09-11T03:41:38.888228+0000","last_became_peered":"2024-09-11T03:41:38.888228+0000","last_unstale":"2024-09-11T03:46:03.045357+0000","last_undegraded":"2024-09-11T03:46:03.045357+0000","last_fullsized":"2024-09-11T03:46:03.045357+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:45:28.993767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:28.993767+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:55.236497+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0011299330000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"2"},{"start":"e2","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":137,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-09-11T03:46:01.964652+0000","last_change":"2024-09-11T03:45:33.951473+0000","last_active":"2024-09-11T03:46:01.964652+0000","last_peered":"2024-09-11T03:46:01.964652+0000","last_clean":"2024-09-11T03:46:01.964652+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:46:01.964652+0000","last_undegraded":"2024-09-11T03:46:01.964652+0000","last_fullsized":"2024-09-11T03:46:01.964652+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:23.840184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:23.840184+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:46:03.322520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":36063,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.16","version":"541'363","reported_seq":2644,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365979+0000","last_change":"2024-09-11T03:46:05.019193+0000","last_active":"2024-09-11T03:46:06.365979+0000","last_peered":"2024-09-11T03:46:06.365979+0000","last_clean":"2024-09-11T03:46:06.365979+0000","last_became_active":"2024-09-11T03:41:38.887856+0000","last_became_peered":"2024-09-11T03:41:38.887856+0000","last_unstale":"2024-09-11T03:46:06.365979+0000","last_undegraded":"2024-09-11T03:46:06.365979+0000","last_fullsized":"2024-09-11T03:46:06.365979+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"531'357","last_scrub_stamp":"2024-09-11T03:45:54.655442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:54.655442+0000","objects_scrubbed":4,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:11.754612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0051539560000000003,"stat_sum":{"num_bytes":9868864,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":50,"num_read_kb":36893,"num_write":57,"num_write_kb":23709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,9],"acting":[5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":676,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013544+0000","last_change":"2024-09-11T03:43:09.565121+0000","last_active":"2024-09-11T03:46:06.013544+0000","last_peered":"2024-09-11T03:46:06.013544+0000","last_clean":"2024-09-11T03:46:06.013544+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:46:06.013544+0000","last_undegraded":"2024-09-11T03:46:06.013544+0000","last_fullsized":"2024-09-11T03:46:06.013544+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:09.565070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:09.565070+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:32.580107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":1986,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-09-11T03:46:01.957184+0000","last_change":"2024-09-11T03:46:01.957474+0000","last_active":"2024-09-11T03:46:01.957184+0000","last_peered":"2024-09-11T03:46:01.957184+0000","last_clean":"2024-09-11T03:46:01.957184+0000","last_became_active":"2024-09-11T03:43:55.918743+0000","last_became_peered":"2024-09-11T03:43:55.918743+0000","last_unstale":"2024-09-11T03:46:01.957184+0000","last_undegraded":"2024-09-11T03:46:01.957184+0000","last_fullsized":"2024-09-11T03:46:01.957184+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:45:13.319584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:13.319584+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:36.452484+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00088453200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[10,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"2"},{"start":"e2","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":708,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-11T03:46:02.956325+0000","last_change":"2024-09-11T03:44:12.182364+0000","last_active":"2024-09-11T03:46:02.956325+0000","last_peered":"2024-09-11T03:46:02.956325+0000","last_clean":"2024-09-11T03:46:02.956325+0000","last_became_active":"2024-09-11T03:40:27.847297+0000","last_became_peered":"2024-09-11T03:40:27.847297+0000","last_unstale":"2024-09-11T03:46:02.956325+0000","last_undegraded":"2024-09-11T03:46:02.956325+0000","last_fullsized":"2024-09-11T03:46:02.956325+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:12.182320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:12.182320+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:00.159862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":1389,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013267+0000","last_change":"2024-09-11T03:46:05.059273+0000","last_active":"2024-09-11T03:46:06.013267+0000","last_peered":"2024-09-11T03:46:06.013267+0000","last_clean":"2024-09-11T03:46:06.013267+0000","last_became_active":"2024-09-11T03:41:35.874295+0000","last_became_peered":"2024-09-11T03:41:35.874295+0000","last_unstale":"2024-09-11T03:46:06.013267+0000","last_undegraded":"2024-09-11T03:46:06.013267+0000","last_fullsized":"2024-09-11T03:46:06.013267+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:45:53.606467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:53.606467+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:13.208602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069043000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.13","version":"20'92","reported_seq":814,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365381+0000","last_change":"2024-09-11T03:45:47.911982+0000","last_active":"2024-09-11T03:46:06.365381+0000","last_peered":"2024-09-11T03:46:06.365381+0000","last_clean":"2024-09-11T03:46:06.365381+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:46:06.365381+0000","last_undegraded":"2024-09-11T03:46:06.365381+0000","last_fullsized":"2024-09-11T03:46:06.365381+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:47.911918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:47.911918+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:14.917780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":1446,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365939+0000","last_change":"2024-09-11T03:46:05.016763+0000","last_active":"2024-09-11T03:46:06.365939+0000","last_peered":"2024-09-11T03:46:06.365939+0000","last_clean":"2024-09-11T03:46:06.365939+0000","last_became_active":"2024-09-11T03:44:43.939694+0000","last_became_peered":"2024-09-11T03:44:43.939694+0000","last_unstale":"2024-09-11T03:46:06.365939+0000","last_undegraded":"2024-09-11T03:46:06.365939+0000","last_fullsized":"2024-09-11T03:46:06.365939+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:45:53.909148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:53.909148+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:20.606178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002806144,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,9],"acting":[1,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":708,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365671+0000","last_change":"2024-09-11T03:45:09.782668+0000","last_active":"2024-09-11T03:46:06.365671+0000","last_peered":"2024-09-11T03:46:06.365671+0000","last_clean":"2024-09-11T03:46:06.365671+0000","last_became_active":"2024-09-11T03:40:27.845249+0000","last_became_peered":"2024-09-11T03:40:27.845249+0000","last_unstale":"2024-09-11T03:46:06.365671+0000","last_undegraded":"2024-09-11T03:46:06.365671+0000","last_fullsized":"2024-09-11T03:46:06.365671+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:09.782621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:09.782621+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:33.934257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"541'367","reported_seq":2657,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-09-11T03:46:01.956684+0000","last_change":"2024-09-11T03:46:01.957123+0000","last_active":"2024-09-11T03:46:01.956684+0000","last_peered":"2024-09-11T03:46:01.956684+0000","last_clean":"2024-09-11T03:46:01.956684+0000","last_became_active":"2024-09-11T03:43:55.918817+0000","last_became_peered":"2024-09-11T03:43:55.918817+0000","last_unstale":"2024-09-11T03:46:01.956684+0000","last_undegraded":"2024-09-11T03:46:01.956684+0000","last_fullsized":"2024-09-11T03:46:01.956684+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"513'359","last_scrub_stamp":"2024-09-11T03:45:31.423318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:31.423318+0000","objects_scrubbed":4,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:39.585130+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00080269800000000002,"stat_sum":{"num_bytes":5857752,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":127,"num_read_kb":121602,"num_write":83,"num_write_kb":32804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":15216236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"2"},{"start":"e2","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":697,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-09-11T03:46:05.020184+0000","last_change":"2024-09-11T03:40:35.703292+0000","last_active":"2024-09-11T03:46:05.020184+0000","last_peered":"2024-09-11T03:46:05.020184+0000","last_clean":"2024-09-11T03:46:05.020184+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:46:05.020184+0000","last_undegraded":"2024-09-11T03:46:05.020184+0000","last_fullsized":"2024-09-11T03:46:05.020184+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.703203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.703203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:00.137853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":2360,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-11T03:46:03.045131+0000","last_change":"2024-09-11T03:46:01.963509+0000","last_active":"2024-09-11T03:46:03.045131+0000","last_peered":"2024-09-11T03:46:03.045131+0000","last_clean":"2024-09-11T03:46:03.045131+0000","last_became_active":"2024-09-11T03:41:38.886834+0000","last_became_peered":"2024-09-11T03:41:38.886834+0000","last_unstale":"2024-09-11T03:46:03.045131+0000","last_undegraded":"2024-09-11T03:46:03.045131+0000","last_fullsized":"2024-09-11T03:46:03.045131+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:44:46.858645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:46.858645+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:03.619771+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00099416799999999992,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"2"},{"start":"e2","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":718,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365498+0000","last_change":"2024-09-11T03:44:26.391068+0000","last_active":"2024-09-11T03:46:06.365498+0000","last_peered":"2024-09-11T03:46:06.365498+0000","last_clean":"2024-09-11T03:46:06.365498+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:46:06.365498+0000","last_undegraded":"2024-09-11T03:46:06.365498+0000","last_fullsized":"2024-09-11T03:46:06.365498+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:26.391001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:26.391001+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:32.609928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"547'616","reported_seq":2901,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013420+0000","last_change":"2024-09-11T03:46:05.061054+0000","last_active":"2024-09-11T03:46:06.013420+0000","last_peered":"2024-09-11T03:46:06.013420+0000","last_clean":"2024-09-11T03:46:06.013420+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:46:06.013420+0000","last_undegraded":"2024-09-11T03:46:06.013420+0000","last_fullsized":"2024-09-11T03:46:06.013420+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'489","last_scrub_stamp":"2024-09-11T03:44:48.449022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:48.449022+0000","objects_scrubbed":6,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:08.216315+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0025474349999999998,"stat_sum":{"num_bytes":12188856,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":215,"num_read_kb":178596,"num_write":178,"num_write_kb":82238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":776,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365967+0000","last_change":"2024-09-11T03:42:05.135033+0000","last_active":"2024-09-11T03:46:06.365967+0000","last_peered":"2024-09-11T03:46:06.365967+0000","last_clean":"2024-09-11T03:46:06.365967+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:46:06.365967+0000","last_undegraded":"2024-09-11T03:46:06.365967+0000","last_fullsized":"2024-09-11T03:46:06.365967+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:05.134977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:05.134977+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:20.668974+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"544'263","reported_seq":2547,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-09-11T03:46:01.967611+0000","last_change":"2024-09-11T03:46:01.967821+0000","last_active":"2024-09-11T03:46:01.967611+0000","last_peered":"2024-09-11T03:46:01.967611+0000","last_clean":"2024-09-11T03:46:01.967611+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:46:01.967611+0000","last_undegraded":"2024-09-11T03:46:01.967611+0000","last_fullsized":"2024-09-11T03:46:01.967611+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"535'255","last_scrub_stamp":"2024-09-11T03:45:54.125664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:54.125664+0000","objects_scrubbed":4,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:55.966587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0028400589999999998,"stat_sum":{"num_bytes":5467682,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":146,"num_read_kb":149021,"num_write":89,"num_write_kb":32129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"2"},{"start":"e2","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":1105,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013408+0000","last_change":"2024-09-11T03:42:00.816905+0000","last_active":"2024-09-11T03:46:06.013408+0000","last_peered":"2024-09-11T03:46:06.013408+0000","last_clean":"2024-09-11T03:46:06.013408+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:46:06.013408+0000","last_undegraded":"2024-09-11T03:46:06.013408+0000","last_fullsized":"2024-09-11T03:46:06.013408+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:00.816850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:00.816850+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:21.512672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"547'809","reported_seq":3209,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365626+0000","last_change":"2024-09-11T03:46:05.021312+0000","last_active":"2024-09-11T03:46:06.365626+0000","last_peered":"2024-09-11T03:46:06.365626+0000","last_clean":"2024-09-11T03:46:06.365626+0000","last_became_active":"2024-09-11T03:41:33.865638+0000","last_became_peered":"2024-09-11T03:41:33.865638+0000","last_unstale":"2024-09-11T03:46:06.365626+0000","last_undegraded":"2024-09-11T03:46:06.365626+0000","last_fullsized":"2024-09-11T03:46:06.365626+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'731","last_scrub_stamp":"2024-09-11T03:45:23.948691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:23.948691+0000","objects_scrubbed":13,"log_size":809,"log_dups_size":0,"ondisk_log_size":809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:51.368410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0072290139999999998,"stat_sum":{"num_bytes":24815258,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":314,"num_read_kb":347374,"num_write":160,"num_write_kb":55087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":1028,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-11T03:46:03.045017+0000","last_change":"2024-09-11T03:45:24.066287+0000","last_active":"2024-09-11T03:46:03.045017+0000","last_peered":"2024-09-11T03:46:03.045017+0000","last_clean":"2024-09-11T03:46:03.045017+0000","last_became_active":"2024-09-11T03:40:37.849988+0000","last_became_peered":"2024-09-11T03:40:37.849988+0000","last_unstale":"2024-09-11T03:46:03.045017+0000","last_undegraded":"2024-09-11T03:46:03.045017+0000","last_fullsized":"2024-09-11T03:46:03.045017+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:24.066237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:24.066237+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:29.248184+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"542'320","reported_seq":2620,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013540+0000","last_change":"2024-09-11T03:46:05.009515+0000","last_active":"2024-09-11T03:46:06.013540+0000","last_peered":"2024-09-11T03:46:06.013540+0000","last_clean":"2024-09-11T03:46:06.013540+0000","last_became_active":"2024-09-11T03:41:33.863875+0000","last_became_peered":"2024-09-11T03:41:33.863875+0000","last_unstale":"2024-09-11T03:46:06.013540+0000","last_undegraded":"2024-09-11T03:46:06.013540+0000","last_fullsized":"2024-09-11T03:46:06.013540+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'283","last_scrub_stamp":"2024-09-11T03:44:29.473127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:29.473127+0000","objects_scrubbed":1,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00061925399999999996,"stat_sum":{"num_bytes":2470761,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":25,"num_read_kb":13817,"num_write":34,"num_write_kb":9341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":700,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-09-11T03:46:01.964711+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:46:01.964711+0000","last_peered":"2024-09-11T03:46:01.964711+0000","last_clean":"2024-09-11T03:46:01.964711+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:46:01.964711+0000","last_undegraded":"2024-09-11T03:46:01.964711+0000","last_fullsized":"2024-09-11T03:46:01.964711+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:04.324886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"544'810","reported_seq":3185,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-09-11T03:46:01.962875+0000","last_change":"2024-09-11T03:46:01.962875+0000","last_active":"2024-09-11T03:46:01.962875+0000","last_peered":"2024-09-11T03:46:01.962875+0000","last_clean":"2024-09-11T03:46:01.962875+0000","last_became_active":"2024-09-11T03:43:55.918858+0000","last_became_peered":"2024-09-11T03:43:55.918858+0000","last_unstale":"2024-09-11T03:46:01.962875+0000","last_undegraded":"2024-09-11T03:46:01.962875+0000","last_fullsized":"2024-09-11T03:46:01.962875+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"524'761","last_scrub_stamp":"2024-09-11T03:45:45.237733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:45.237733+0000","objects_scrubbed":11,"log_size":810,"log_dups_size":0,"ondisk_log_size":810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:04.001236+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0063596369999999996,"stat_sum":{"num_bytes":28479920,"num_objects":14,"num_object_clones":9,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":259,"num_read_kb":294602,"num_write":135,"num_write_kb":55047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":14282259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,3],"acting":[10,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"2"},{"start":"e2","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":881,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-09-11T03:46:01.964590+0000","last_change":"2024-09-11T03:44:28.192468+0000","last_active":"2024-09-11T03:46:01.964590+0000","last_peered":"2024-09-11T03:46:01.964590+0000","last_clean":"2024-09-11T03:46:01.964590+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:46:01.964590+0000","last_undegraded":"2024-09-11T03:46:01.964590+0000","last_fullsized":"2024-09-11T03:46:01.964590+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:28.192403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:28.192403+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:56.078747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":2447,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365504+0000","last_change":"2024-09-11T03:46:05.019458+0000","last_active":"2024-09-11T03:46:06.365504+0000","last_peered":"2024-09-11T03:46:06.365504+0000","last_clean":"2024-09-11T03:46:06.365504+0000","last_became_active":"2024-09-11T03:41:33.865499+0000","last_became_peered":"2024-09-11T03:41:33.865499+0000","last_unstale":"2024-09-11T03:46:06.365504+0000","last_undegraded":"2024-09-11T03:46:06.365504+0000","last_fullsized":"2024-09-11T03:46:06.365504+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:45:49.918830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:49.918830+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:55.119040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0048807099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":282,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-09-11T03:46:01.956097+0000","last_change":"2024-09-11T03:44:25.511501+0000","last_active":"2024-09-11T03:46:01.956097+0000","last_peered":"2024-09-11T03:46:01.956097+0000","last_clean":"2024-09-11T03:46:01.956097+0000","last_became_active":"2024-09-11T03:43:52.924325+0000","last_became_peered":"2024-09-11T03:43:52.924325+0000","last_unstale":"2024-09-11T03:46:01.956097+0000","last_undegraded":"2024-09-11T03:46:01.956097+0000","last_fullsized":"2024-09-11T03:46:01.956097+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:25.511435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:25.511435+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:32.454502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"547'769","reported_seq":3105,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013436+0000","last_change":"2024-09-11T03:46:05.061158+0000","last_active":"2024-09-11T03:46:06.013436+0000","last_peered":"2024-09-11T03:46:06.013436+0000","last_clean":"2024-09-11T03:46:06.013436+0000","last_became_active":"2024-09-11T03:41:33.866316+0000","last_became_peered":"2024-09-11T03:41:33.866316+0000","last_unstale":"2024-09-11T03:46:06.013436+0000","last_undegraded":"2024-09-11T03:46:06.013436+0000","last_fullsized":"2024-09-11T03:46:06.013436+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"530'746","last_scrub_stamp":"2024-09-11T03:45:50.568426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:50.568426+0000","objects_scrubbed":9,"log_size":769,"log_dups_size":0,"ondisk_log_size":769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:17.296595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024829510000000002,"stat_sum":{"num_bytes":19987543,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":156,"num_read_kb":159287,"num_write":117,"num_write_kb":46550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":282,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-09-11T03:46:01.956347+0000","last_change":"2024-09-11T03:44:26.477597+0000","last_active":"2024-09-11T03:46:01.956347+0000","last_peered":"2024-09-11T03:46:01.956347+0000","last_clean":"2024-09-11T03:46:01.956347+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:46:01.956347+0000","last_undegraded":"2024-09-11T03:46:01.956347+0000","last_fullsized":"2024-09-11T03:46:01.956347+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:26.477545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:26.477545+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:57.218670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"541'419","reported_seq":2756,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365984+0000","last_change":"2024-09-11T03:46:05.019454+0000","last_active":"2024-09-11T03:46:06.365984+0000","last_peered":"2024-09-11T03:46:06.365984+0000","last_clean":"2024-09-11T03:46:06.365984+0000","last_became_active":"2024-09-11T03:44:10.921135+0000","last_became_peered":"2024-09-11T03:44:10.921135+0000","last_unstale":"2024-09-11T03:46:06.365984+0000","last_undegraded":"2024-09-11T03:46:06.365984+0000","last_fullsized":"2024-09-11T03:46:06.365984+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'397","last_scrub_stamp":"2024-09-11T03:45:26.963120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:26.963120+0000","objects_scrubbed":7,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:34.917273+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0033009860000000001,"stat_sum":{"num_bytes":6571933,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":135391,"num_write":75,"num_write_kb":21420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":4118225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":287,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-09-11T03:46:01.956419+0000","last_change":"2024-09-11T03:45:12.360874+0000","last_active":"2024-09-11T03:46:01.956419+0000","last_peered":"2024-09-11T03:46:01.956419+0000","last_clean":"2024-09-11T03:46:01.956419+0000","last_became_active":"2024-09-11T03:43:52.923384+0000","last_became_peered":"2024-09-11T03:43:52.923384+0000","last_unstale":"2024-09-11T03:46:01.956419+0000","last_undegraded":"2024-09-11T03:46:01.956419+0000","last_fullsized":"2024-09-11T03:46:01.956419+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:12.360829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:12.360829+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:31.665882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,5],"acting":[10,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"531'216","reported_seq":2558,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365800+0000","last_change":"2024-09-11T03:46:05.014589+0000","last_active":"2024-09-11T03:46:06.365800+0000","last_peered":"2024-09-11T03:46:06.365800+0000","last_clean":"2024-09-11T03:46:06.365800+0000","last_became_active":"2024-09-11T03:42:30.902838+0000","last_became_peered":"2024-09-11T03:42:30.902838+0000","last_unstale":"2024-09-11T03:46:06.365800+0000","last_undegraded":"2024-09-11T03:46:06.365800+0000","last_fullsized":"2024-09-11T03:46:06.365800+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:45:53.642117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:53.642117+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:06.475100+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0.00069119299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":27471,"num_write":22,"num_write_kb":11305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,11],"acting":[5,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":287,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-09-11T03:46:01.956065+0000","last_change":"2024-09-11T03:45:26.373529+0000","last_active":"2024-09-11T03:46:01.956065+0000","last_peered":"2024-09-11T03:46:01.956065+0000","last_clean":"2024-09-11T03:46:01.956065+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:46:01.956065+0000","last_undegraded":"2024-09-11T03:46:01.956065+0000","last_fullsized":"2024-09-11T03:46:01.956065+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:26.373478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:26.373478+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:27.036677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"541'192","reported_seq":2488,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-11T03:46:02.961540+0000","last_change":"2024-09-11T03:46:01.960446+0000","last_active":"2024-09-11T03:46:02.961540+0000","last_peered":"2024-09-11T03:46:02.961540+0000","last_clean":"2024-09-11T03:46:02.961540+0000","last_became_active":"2024-09-11T03:41:33.866217+0000","last_became_peered":"2024-09-11T03:41:33.866217+0000","last_unstale":"2024-09-11T03:46:02.961540+0000","last_undegraded":"2024-09-11T03:46:02.961540+0000","last_fullsized":"2024-09-11T03:46:02.961540+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'176","last_scrub_stamp":"2024-09-11T03:45:41.180057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:41.180057+0000","objects_scrubbed":3,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:43.565099+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001254107,"stat_sum":{"num_bytes":7950074,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":7024,"num_write":26,"num_write_kb":9767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"2"},{"start":"e2","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":684,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-09-11T03:46:05.019253+0000","last_change":"2024-09-11T03:44:25.934869+0000","last_active":"2024-09-11T03:46:05.019253+0000","last_peered":"2024-09-11T03:46:05.019253+0000","last_clean":"2024-09-11T03:46:05.019253+0000","last_became_active":"2024-09-11T03:44:25.934580+0000","last_became_peered":"2024-09-11T03:44:25.934580+0000","last_unstale":"2024-09-11T03:46:05.019253+0000","last_undegraded":"2024-09-11T03:46:05.019253+0000","last_fullsized":"2024-09-11T03:46:05.019253+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:23.494418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:23.494418+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:37.110438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"547'178","reported_seq":2541,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013805+0000","last_change":"2024-09-11T03:46:05.011485+0000","last_active":"2024-09-11T03:46:06.013805+0000","last_peered":"2024-09-11T03:46:06.013805+0000","last_clean":"2024-09-11T03:46:06.013805+0000","last_became_active":"2024-09-11T03:43:55.920265+0000","last_became_peered":"2024-09-11T03:43:55.920265+0000","last_unstale":"2024-09-11T03:46:06.013805+0000","last_undegraded":"2024-09-11T03:46:06.013805+0000","last_fullsized":"2024-09-11T03:46:06.013805+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"537'168","last_scrub_stamp":"2024-09-11T03:45:55.666981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:55.666981+0000","objects_scrubbed":4,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:07.930280+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0024739089999999998,"stat_sum":{"num_bytes":4631117,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":76,"num_read_kb":79688,"num_write":30,"num_write_kb":9039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4387035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":720,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365486+0000","last_change":"2024-09-11T03:44:10.921140+0000","last_active":"2024-09-11T03:46:06.365486+0000","last_peered":"2024-09-11T03:46:06.365486+0000","last_clean":"2024-09-11T03:46:06.365486+0000","last_became_active":"2024-09-11T03:44:10.920828+0000","last_became_peered":"2024-09-11T03:44:10.920828+0000","last_unstale":"2024-09-11T03:46:06.365486+0000","last_undegraded":"2024-09-11T03:46:06.365486+0000","last_fullsized":"2024-09-11T03:46:06.365486+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:07.508193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:07.508193+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:18.921093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"541'367","reported_seq":2670,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365926+0000","last_change":"2024-09-11T03:46:05.019467+0000","last_active":"2024-09-11T03:46:06.365926+0000","last_peered":"2024-09-11T03:46:06.365926+0000","last_clean":"2024-09-11T03:46:06.365926+0000","last_became_active":"2024-09-11T03:41:33.862271+0000","last_became_peered":"2024-09-11T03:41:33.862271+0000","last_unstale":"2024-09-11T03:46:06.365926+0000","last_undegraded":"2024-09-11T03:46:06.365926+0000","last_fullsized":"2024-09-11T03:46:06.365926+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'327","last_scrub_stamp":"2024-09-11T03:45:12.798880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:12.798880+0000","objects_scrubbed":5,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:15.146868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0053604739999999996,"stat_sum":{"num_bytes":4773452,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":114,"num_read_kb":95408,"num_write":63,"num_write_kb":22522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":1119,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013854+0000","last_change":"2024-09-11T03:45:23.491671+0000","last_active":"2024-09-11T03:46:06.013854+0000","last_peered":"2024-09-11T03:46:06.013854+0000","last_clean":"2024-09-11T03:46:06.013854+0000","last_became_active":"2024-09-11T03:40:25.854624+0000","last_became_peered":"2024-09-11T03:40:25.854624+0000","last_unstale":"2024-09-11T03:46:06.013854+0000","last_undegraded":"2024-09-11T03:46:06.013854+0000","last_fullsized":"2024-09-11T03:46:06.013854+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:23.491587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:23.491587+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:42.891922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"20'92","reported_seq":694,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365759+0000","last_change":"2024-09-11T03:40:35.282811+0000","last_active":"2024-09-11T03:46:06.365759+0000","last_peered":"2024-09-11T03:46:06.365759+0000","last_clean":"2024-09-11T03:46:06.365759+0000","last_became_active":"2024-09-11T03:40:27.844262+0000","last_became_peered":"2024-09-11T03:40:27.844262+0000","last_unstale":"2024-09-11T03:46:06.365759+0000","last_undegraded":"2024-09-11T03:46:06.365759+0000","last_fullsized":"2024-09-11T03:46:06.365759+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.282762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.282762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:00.679637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"541'596","reported_seq":2871,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-09-11T03:46:01.965523+0000","last_change":"2024-09-11T03:46:01.966113+0000","last_active":"2024-09-11T03:46:01.965523+0000","last_peered":"2024-09-11T03:46:01.965523+0000","last_clean":"2024-09-11T03:46:01.965523+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:46:01.965523+0000","last_undegraded":"2024-09-11T03:46:01.965523+0000","last_fullsized":"2024-09-11T03:46:01.965523+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'551","last_scrub_stamp":"2024-09-11T03:45:10.285456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:10.285456+0000","objects_scrubbed":9,"log_size":596,"log_dups_size":0,"ondisk_log_size":596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:16.115202+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011339830000000001,"stat_sum":{"num_bytes":12286313,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":268,"num_read_kb":237147,"num_write":210,"num_write_kb":85418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"2"},{"start":"e2","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":435,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013701+0000","last_change":"2024-09-11T03:45:53.637950+0000","last_active":"2024-09-11T03:46:06.013701+0000","last_peered":"2024-09-11T03:46:06.013701+0000","last_clean":"2024-09-11T03:46:06.013701+0000","last_became_active":"2024-09-11T03:44:43.940022+0000","last_became_peered":"2024-09-11T03:44:43.940022+0000","last_unstale":"2024-09-11T03:46:06.013701+0000","last_undegraded":"2024-09-11T03:46:06.013701+0000","last_fullsized":"2024-09-11T03:46:06.013701+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:53.637891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:53.637891+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:14.116074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"547'725","reported_seq":2994,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365964+0000","last_change":"2024-09-11T03:46:05.020197+0000","last_active":"2024-09-11T03:46:06.365964+0000","last_peered":"2024-09-11T03:46:06.365964+0000","last_clean":"2024-09-11T03:46:06.365964+0000","last_became_active":"2024-09-11T03:42:30.902959+0000","last_became_peered":"2024-09-11T03:42:30.902959+0000","last_unstale":"2024-09-11T03:46:06.365964+0000","last_undegraded":"2024-09-11T03:46:06.365964+0000","last_fullsized":"2024-09-11T03:46:06.365964+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"485'666","last_scrub_stamp":"2024-09-11T03:45:07.788521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:07.788521+0000","objects_scrubbed":5,"log_size":725,"log_dups_size":0,"ondisk_log_size":725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:24.818995+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0061154139999999996,"stat_sum":{"num_bytes":8168757,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":308,"num_read_kb":281935,"num_write":221,"num_write_kb":80839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17460823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":720,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013651+0000","last_change":"2024-09-11T03:42:51.733829+0000","last_active":"2024-09-11T03:46:06.013651+0000","last_peered":"2024-09-11T03:46:06.013651+0000","last_clean":"2024-09-11T03:46:06.013651+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:46:06.013651+0000","last_undegraded":"2024-09-11T03:46:06.013651+0000","last_fullsized":"2024-09-11T03:46:06.013651+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:51.733762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:51.733762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"544'797","reported_seq":3045,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-11T03:46:02.956514+0000","last_change":"2024-09-11T03:46:01.964808+0000","last_active":"2024-09-11T03:46:02.956514+0000","last_peered":"2024-09-11T03:46:02.956514+0000","last_clean":"2024-09-11T03:46:02.956514+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:46:02.956514+0000","last_undegraded":"2024-09-11T03:46:02.956514+0000","last_fullsized":"2024-09-11T03:46:02.956514+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"532'778","last_scrub_stamp":"2024-09-11T03:45:51.172946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:51.172946+0000","objects_scrubbed":11,"log_size":797,"log_dups_size":0,"ondisk_log_size":797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:06.173002+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0054067910000000002,"stat_sum":{"num_bytes":19126872,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":290,"num_read_kb":267331,"num_write":234,"num_write_kb":93533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"2"},{"start":"e2","length":"1"}]},{"pgid":"2.f","version":"546'452","reported_seq":2749,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013530+0000","last_change":"2024-09-11T03:46:05.061334+0000","last_active":"2024-09-11T03:46:06.013530+0000","last_peered":"2024-09-11T03:46:06.013530+0000","last_clean":"2024-09-11T03:46:06.013530+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:46:06.013530+0000","last_undegraded":"2024-09-11T03:46:06.013530+0000","last_fullsized":"2024-09-11T03:46:06.013530+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'434","last_scrub_stamp":"2024-09-11T03:45:42.488889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:42.488889+0000","objects_scrubbed":3,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:06.542862+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0025501759999999999,"stat_sum":{"num_bytes":7764757,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":174,"num_read_kb":142734,"num_write":152,"num_write_kb":56584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":130,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-09-11T03:46:05.019255+0000","last_change":"2024-09-11T03:45:10.103917+0000","last_active":"2024-09-11T03:46:05.019255+0000","last_peered":"2024-09-11T03:46:05.019255+0000","last_clean":"2024-09-11T03:46:05.019255+0000","last_became_active":"2024-09-11T03:45:09.973577+0000","last_became_peered":"2024-09-11T03:45:09.973577+0000","last_unstale":"2024-09-11T03:46:05.019255+0000","last_undegraded":"2024-09-11T03:46:05.019255+0000","last_fullsized":"2024-09-11T03:46:05.019255+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:10.103870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:10.103870+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:30.592332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,9],"acting":[0,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":187630074,"num_objects":94,"num_object_clones":52,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":9,"num_read":2842,"num_read_kb":2728965,"num_write":1979,"num_write_kb":761051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":61837646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":358858752,"data_stored":466331767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10355,"ondisk_log_size":10355,"up":78,"acting":78,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":603786,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":36063,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285674,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71672,"kb_used_data":32164,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93673480,"statfs":{"total":95995035648,"available":95921643520,"internally_reserved":0,"allocated":32935936,"data_stored":38338889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514606,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68392,"kb_used_data":33044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93676760,"statfs":{"total":95995035648,"available":95925002240,"internally_reserved":0,"allocated":33837056,"data_stored":39718014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743540,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65196,"kb_used_data":38164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93679956,"statfs":{"total":95995035648,"available":95928274944,"internally_reserved":0,"allocated":39079936,"data_stored":46559731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":59708,"kb_used_data":32672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93685444,"statfs":{"total":95995035648,"available":95933894656,"internally_reserved":0,"allocated":33456128,"data_stored":39822007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299519,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73400,"kb_used_data":42204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93671752,"statfs":{"total":95995035648,"available":95919874048,"internally_reserved":0,"allocated":43216896,"data_stored":57162471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":193273528451,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757386,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58692,"kb_used_data":19176,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93686460,"statfs":{"total":95995035648,"available":95934935040,"internally_reserved":0,"allocated":19636224,"data_stored":22351907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986319,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54916,"kb_used_data":27884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93690236,"statfs":{"total":95995035648,"available":95938801664,"internally_reserved":0,"allocated":28553216,"data_stored":31071099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149843,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":118104,"kb_used_data":68864,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93627048,"statfs":{"total":95995035648,"available":95874097152,"internally_reserved":0,"allocated":70516736,"data_stored":87643191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411481,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81676,"kb_used_data":42164,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93663476,"statfs":{"total":95995035648,"available":95911399424,"internally_reserved":0,"allocated":43175936,"data_stored":52417562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575007,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63204,"kb_used_data":36168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93681948,"statfs":{"total":95995035648,"available":95930314752,"internally_reserved":0,"allocated":37036032,"data_stored":42693752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738532,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83940,"kb_used_data":40264,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93661212,"statfs":{"total":95995035648,"available":95909081088,"internally_reserved":0,"allocated":41230336,"data_stored":42344569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38342656,"data_stored":42083830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30453760,"data_stored":38477609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36425728,"data_stored":48025015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":63643648,"data_stored":86933922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":23859200,"data_stored":28735975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12836864,"data_stored":17926256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4079616,"data_stored":4066399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36429824,"data_stored":52745080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":26873856,"data_stored":35605864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":32206848,"data_stored":42040752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":27291648,"data_stored":35526715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":26415104,"data_stored":34164350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:46:07.734 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:46:07.817 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:07.838 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:07.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:07.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:07.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:07.960 INFO:tasks.rados.rados.2.smithi175.stdout:2059: done (0 left) 2024-09-11T03:46:07.960 INFO:tasks.rados.rados.2.smithi175.stdout:2060: snap_create 2024-09-11T03:46:07.965 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:07.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:08.067 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:08.090 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:08.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:08.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:08.193 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:08.219 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:08.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:08.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:08.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:08 smithi084 ceph-mon[27254]: pgmap v978: 57 pgs: 57 active+clean; 180 MiB data, 780 MiB used, 983 GiB / 983 GiB avail; 23 MiB/s rd, 3.1 MiB/s wr, 26 op/s 2024-09-11T03:46:08.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:08 smithi084 ceph-mon[27254]: from='client.16287 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:46:08.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:08 smithi084 ceph-mon[27254]: osdmap e550: 12 total, 12 up, 11 in 2024-09-11T03:46:08.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:08 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:46:08.278 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:08 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:46:08.320 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:08.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:08 smithi121 ceph-mon[30729]: pgmap v978: 57 pgs: 57 active+clean; 180 MiB data, 780 MiB used, 983 GiB / 983 GiB avail; 23 MiB/s rd, 3.1 MiB/s wr, 26 op/s 2024-09-11T03:46:08.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:08 smithi121 ceph-mon[30729]: from='client.16287 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:46:08.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:08 smithi121 ceph-mon[30729]: osdmap e550: 12 total, 12 up, 11 in 2024-09-11T03:46:08.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:08 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:46:08.324 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:08 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:46:08.346 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:08.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:08.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:08.383 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:46:08.383 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:46:08.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:46:08.384 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:46:08.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.384 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:877 2.e 2:77dadb97:::smithi17557870-30:head [copy-from ver 12] snapc 1e=[1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-11T03:46:08.384 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:31.988317+0000", 2024-09-11T03:46:08.384 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 396.39392939499999, 2024-09-11T03:46:08.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011419110999999999, 2024-09-11T03:46:08.384 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.384 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.384 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:08.384 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.384 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.385 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.385 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 877 2024-09-11T03:46:08.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.385 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.988317+0000", 2024-09-11T03:46:08.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.988317+0000", 2024-09-11T03:46:08.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.988319+0000", 2024-09-11T03:46:08.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.866e-06 2024-09-11T03:46:08.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.988329+0000", 2024-09-11T03:46:08.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0132000000000001e-05 2024-09-11T03:46:08.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.988349+0000", 2024-09-11T03:46:08.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9400999999999999e-05 2024-09-11T03:46:08.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.988375+0000", 2024-09-11T03:46:08.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6156000000000002e-05 2024-09-11T03:46:08.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.988447+0000", 2024-09-11T03:46:08.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2559999999999996e-05 2024-09-11T03:46:08.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.988554+0000", 2024-09-11T03:46:08.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000106699 2024-09-11T03:46:08.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:46:08.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.994152+0000", 2024-09-11T03:46:08.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055978130000000001 2024-09-11T03:46:08.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:46:08.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.997930+0000", 2024-09-11T03:46:08.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037786780000000002 2024-09-11T03:46:08.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.998218+0000", 2024-09-11T03:46:08.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000287335 2024-09-11T03:46:08.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.999572+0000", 2024-09-11T03:46:08.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001354657 2024-09-11T03:46:08.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:46:08.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.999667+0000", 2024-09-11T03:46:08.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4435e-05 2024-09-11T03:46:08.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:31.999736+0000", 2024-09-11T03:46:08.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9379000000000001e-05 2024-09-11T03:46:08.390 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.390 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.391 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.391 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1068 2.3 e175/95 2:c3768452:::smithi17557870-11:head v 175'55, mlcod=165'51)", 2024-09-11T03:46:08.391 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:51.874076+0000", 2024-09-11T03:46:08.391 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 376.50817069599998, 2024-09-11T03:46:08.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033166769999999998, 2024-09-11T03:46:08.391 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.391 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.391 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:08.391 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.874076+0000", 2024-09-11T03:46:08.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.874076+0000", 2024-09-11T03:46:08.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.874076+0000", 2024-09-11T03:46:08.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.66e-07 2024-09-11T03:46:08.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.906033+0000", 2024-09-11T03:46:08.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.031956971000000001 2024-09-11T03:46:08.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.906049+0000", 2024-09-11T03:46:08.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6503000000000001e-05 2024-09-11T03:46:08.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.906058+0000", 2024-09-11T03:46:08.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3860000000000007e-06 2024-09-11T03:46:08.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.906111+0000", 2024-09-11T03:46:08.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3053999999999997e-05 2024-09-11T03:46:08.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.906129+0000", 2024-09-11T03:46:08.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8071999999999999e-05 2024-09-11T03:46:08.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:46:08.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.907203+0000", 2024-09-11T03:46:08.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010736960000000001 2024-09-11T03:46:08.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.907243+0000", 2024-09-11T03:46:08.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9922000000000002e-05 2024-09-11T03:46:08.395 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.395 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.395 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.395 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1159 2.e 2:750de3b7:::smithi17557870-8:head [copy-from ver 77] snapc 2e=[2e,2c,2b,28,26] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:46:08.395 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:01.831728+0000", 2024-09-11T03:46:08.396 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 366.55051854499999, 2024-09-11T03:46:08.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016271035999999999, 2024-09-11T03:46:08.396 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.396 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.396 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:08.396 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.396 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.396 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.396 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1159 2024-09-11T03:46:08.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.396 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831728+0000", 2024-09-11T03:46:08.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831728+0000", 2024-09-11T03:46:08.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831730+0000", 2024-09-11T03:46:08.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.283e-06 2024-09-11T03:46:08.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831741+0000", 2024-09-11T03:46:08.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0475000000000001e-05 2024-09-11T03:46:08.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831760+0000", 2024-09-11T03:46:08.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9034000000000001e-05 2024-09-11T03:46:08.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831776+0000", 2024-09-11T03:46:08.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6106000000000001e-05 2024-09-11T03:46:08.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831836+0000", 2024-09-11T03:46:08.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0420000000000001e-05 2024-09-11T03:46:08.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:46:08.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831841+0000", 2024-09-11T03:46:08.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8770000000000001e-06 2024-09-11T03:46:08.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.833455+0000", 2024-09-11T03:46:08.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016140639999999999 2024-09-11T03:46:08.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.833568+0000", 2024-09-11T03:46:08.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011259700000000001 2024-09-11T03:46:08.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:46:08.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.840230+0000", 2024-09-11T03:46:08.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066619189999999997 2024-09-11T03:46:08.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:46:08.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.845836+0000", 2024-09-11T03:46:08.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056066459999999999 2024-09-11T03:46:08.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.846094+0000", 2024-09-11T03:46:08.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025752300000000001 2024-09-11T03:46:08.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.847897+0000", 2024-09-11T03:46:08.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018033529999999999 2024-09-11T03:46:08.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:46:08.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.847954+0000", 2024-09-11T03:46:08.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6436e-05 2024-09-11T03:46:08.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.847999+0000", 2024-09-11T03:46:08.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5303000000000003e-05 2024-09-11T03:46:08.403 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.403 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.403 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.403 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1160 2.e 2:750de3b7:::smithi17557870-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:46:08.403 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:01.831798+0000", 2024-09-11T03:46:08.404 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 366.55044882499999, 2024-09-11T03:46:08.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012910331000000001, 2024-09-11T03:46:08.404 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.404 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.404 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:46:08.404 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.404 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.404 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.404 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1160 2024-09-11T03:46:08.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.404 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831798+0000", 2024-09-11T03:46:08.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831798+0000", 2024-09-11T03:46:08.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831798+0000", 2024-09-11T03:46:08.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0699999999999998e-07 2024-09-11T03:46:08.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831800+0000", 2024-09-11T03:46:08.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8679999999999999e-06 2024-09-11T03:46:08.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831806+0000", 2024-09-11T03:46:08.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6339999999999998e-06 2024-09-11T03:46:08.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831809+0000", 2024-09-11T03:46:08.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8650000000000003e-06 2024-09-11T03:46:08.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831896+0000", 2024-09-11T03:46:08.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6700000000000007e-05 2024-09-11T03:46:08.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:46:08.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831904+0000", 2024-09-11T03:46:08.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8099999999999998e-06 2024-09-11T03:46:08.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.833745+0000", 2024-09-11T03:46:08.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018409979999999999 2024-09-11T03:46:08.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:46:08.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.833795+0000", 2024-09-11T03:46:08.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0413e-05 2024-09-11T03:46:08.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.844574+0000", 2024-09-11T03:46:08.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010778993000000001 2024-09-11T03:46:08.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.844637+0000", 2024-09-11T03:46:08.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3002000000000004e-05 2024-09-11T03:46:08.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.844708+0000", 2024-09-11T03:46:08.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0641000000000003e-05 2024-09-11T03:46:08.409 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.409 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.409 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.409 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1285 2.1 2:8638b65e:::smithi17557870-5:head [copy-from ver 153] snapc 33=[32,30,2c,2b] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:46:08.409 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:12.836052+0000", 2024-09-11T03:46:08.410 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 355.54619422799999, 2024-09-11T03:46:08.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020441596999999999, 2024-09-11T03:46:08.410 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.410 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.410 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:08.410 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.410 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.410 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.410 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1285 2024-09-11T03:46:08.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.410 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836052+0000", 2024-09-11T03:46:08.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836052+0000", 2024-09-11T03:46:08.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836053+0000", 2024-09-11T03:46:08.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.34e-07 2024-09-11T03:46:08.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836055+0000", 2024-09-11T03:46:08.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7549999999999999e-06 2024-09-11T03:46:08.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836061+0000", 2024-09-11T03:46:08.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6579999999999999e-06 2024-09-11T03:46:08.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836071+0000", 2024-09-11T03:46:08.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0141e-05 2024-09-11T03:46:08.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836118+0000", 2024-09-11T03:46:08.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7200999999999998e-05 2024-09-11T03:46:08.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:46:08.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836124+0000", 2024-09-11T03:46:08.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8150000000000002e-06 2024-09-11T03:46:08.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.837592+0000", 2024-09-11T03:46:08.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014672509999999999 2024-09-11T03:46:08.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.837642+0000", 2024-09-11T03:46:08.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0145999999999997e-05 2024-09-11T03:46:08.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,3", 2024-09-11T03:46:08.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.849946+0000", 2024-09-11T03:46:08.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012303869 2024-09-11T03:46:08.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:46:08.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.852055+0000", 2024-09-11T03:46:08.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002109835 2024-09-11T03:46:08.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856314+0000", 2024-09-11T03:46:08.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042587700000000003 2024-09-11T03:46:08.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856368+0000", 2024-09-11T03:46:08.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3538999999999997e-05 2024-09-11T03:46:08.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:46:08.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856435+0000", 2024-09-11T03:46:08.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7470000000000003e-05 2024-09-11T03:46:08.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856494+0000", 2024-09-11T03:46:08.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8712999999999998e-05 2024-09-11T03:46:08.416 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.416 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.416 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.417 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1286 2.1 2:8638b65e:::smithi17557870-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:46:08.417 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:12.836083+0000", 2024-09-11T03:46:08.417 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 355.546163358, 2024-09-11T03:46:08.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015739692, 2024-09-11T03:46:08.417 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.417 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.417 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:46:08.417 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.417 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.417 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.417 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1286 2024-09-11T03:46:08.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.418 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836083+0000", 2024-09-11T03:46:08.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836083+0000", 2024-09-11T03:46:08.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836084+0000", 2024-09-11T03:46:08.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2500000000000001e-07 2024-09-11T03:46:08.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836085+0000", 2024-09-11T03:46:08.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8020000000000001e-06 2024-09-11T03:46:08.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836092+0000", 2024-09-11T03:46:08.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3169999999999998e-06 2024-09-11T03:46:08.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836096+0000", 2024-09-11T03:46:08.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.634e-06 2024-09-11T03:46:08.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836147+0000", 2024-09-11T03:46:08.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0374999999999998e-05 2024-09-11T03:46:08.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:46:08.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836150+0000", 2024-09-11T03:46:08.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.072e-06 2024-09-11T03:46:08.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.837748+0000", 2024-09-11T03:46:08.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001597867 2024-09-11T03:46:08.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:46:08.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.837783+0000", 2024-09-11T03:46:08.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5395000000000002e-05 2024-09-11T03:46:08.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.851559+0000", 2024-09-11T03:46:08.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013776343999999999 2024-09-11T03:46:08.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.851665+0000", 2024-09-11T03:46:08.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010521199999999999 2024-09-11T03:46:08.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.851823+0000", 2024-09-11T03:46:08.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015824900000000001 2024-09-11T03:46:08.422 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.423 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.423 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.423 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1434 2.e 2:7efbbea8:::smithi17557870-24:head [copy-from ver 75] snapc 36=[35,2b] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-11T03:46:08.423 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:20.864420+0000", 2024-09-11T03:46:08.423 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 347.51782700699999, 2024-09-11T03:46:08.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012389887, 2024-09-11T03:46:08.423 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.423 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.423 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:08.423 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.423 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.423 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.424 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1434 2024-09-11T03:46:08.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.424 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864420+0000", 2024-09-11T03:46:08.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864420+0000", 2024-09-11T03:46:08.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864421+0000", 2024-09-11T03:46:08.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8369999999999999e-06 2024-09-11T03:46:08.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864430+0000", 2024-09-11T03:46:08.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.132e-06 2024-09-11T03:46:08.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864447+0000", 2024-09-11T03:46:08.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6305999999999999e-05 2024-09-11T03:46:08.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864454+0000", 2024-09-11T03:46:08.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1230000000000002e-06 2024-09-11T03:46:08.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864527+0000", 2024-09-11T03:46:08.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2606000000000002e-05 2024-09-11T03:46:08.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:46:08.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864536+0000", 2024-09-11T03:46:08.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7790000000000007e-06 2024-09-11T03:46:08.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.865844+0000", 2024-09-11T03:46:08.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307473 2024-09-11T03:46:08.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.865934+0000", 2024-09-11T03:46:08.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9785999999999997e-05 2024-09-11T03:46:08.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:46:08.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.869567+0000", 2024-09-11T03:46:08.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0036332859999999999 2024-09-11T03:46:08.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:46:08.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.874154+0000", 2024-09-11T03:46:08.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.004586957 2024-09-11T03:46:08.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.875683+0000", 2024-09-11T03:46:08.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015289920000000001 2024-09-11T03:46:08.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.876715+0000", 2024-09-11T03:46:08.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010321620000000001 2024-09-11T03:46:08.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:46:08.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.876752+0000", 2024-09-11T03:46:08.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6820000000000003e-05 2024-09-11T03:46:08.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.876809+0000", 2024-09-11T03:46:08.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7627999999999998e-05 2024-09-11T03:46:08.430 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.430 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.430 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.430 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1589 2.7 2:e77ce595:::smithi17557870-31:head [copy-from ver 148] snapc 3f=[3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-09-11T03:46:08.430 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:35.104207+0000", 2024-09-11T03:46:08.430 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 333.27803935399999, 2024-09-11T03:46:08.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021309244000000001, 2024-09-11T03:46:08.430 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.430 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.431 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:08.431 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.431 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.431 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.431 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1589 2024-09-11T03:46:08.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.431 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104207+0000", 2024-09-11T03:46:08.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104207+0000", 2024-09-11T03:46:08.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104208+0000", 2024-09-11T03:46:08.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2780000000000001e-06 2024-09-11T03:46:08.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104213+0000", 2024-09-11T03:46:08.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5979999999999999e-06 2024-09-11T03:46:08.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104225+0000", 2024-09-11T03:46:08.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1819e-05 2024-09-11T03:46:08.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104249+0000", 2024-09-11T03:46:08.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4117999999999998e-05 2024-09-11T03:46:08.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104305+0000", 2024-09-11T03:46:08.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6069999999999997e-05 2024-09-11T03:46:08.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104375+0000", 2024-09-11T03:46:08.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0156999999999999e-05 2024-09-11T03:46:08.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:46:08.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116294+0000", 2024-09-11T03:46:08.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011918676 2024-09-11T03:46:08.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:46:08.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.123116+0000", 2024-09-11T03:46:08.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068219259999999999 2024-09-11T03:46:08.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.124470+0000", 2024-09-11T03:46:08.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001354079 2024-09-11T03:46:08.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125455+0000", 2024-09-11T03:46:08.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00098535300000000001 2024-09-11T03:46:08.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:46:08.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125487+0000", 2024-09-11T03:46:08.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2200999999999999e-05 2024-09-11T03:46:08.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125516+0000", 2024-09-11T03:46:08.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8969e-05 2024-09-11T03:46:08.436 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.436 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.436 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.437 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1590 2.7 2:e77ce595:::smithi17557870-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e218)", 2024-09-11T03:46:08.437 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:35.104267+0000", 2024-09-11T03:46:08.437 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 333.27797977400002, 2024-09-11T03:46:08.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017183714999999999, 2024-09-11T03:46:08.437 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.437 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.437 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:46:08.437 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.437 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.437 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.437 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1590 2024-09-11T03:46:08.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.438 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104267+0000", 2024-09-11T03:46:08.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104267+0000", 2024-09-11T03:46:08.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104267+0000", 2024-09-11T03:46:08.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.77e-07 2024-09-11T03:46:08.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104269+0000", 2024-09-11T03:46:08.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8199999999999999e-06 2024-09-11T03:46:08.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104274+0000", 2024-09-11T03:46:08.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2839999999999997e-06 2024-09-11T03:46:08.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104277+0000", 2024-09-11T03:46:08.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0149999999999999e-06 2024-09-11T03:46:08.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104478+0000", 2024-09-11T03:46:08.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200256 2024-09-11T03:46:08.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:46:08.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104511+0000", 2024-09-11T03:46:08.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3155999999999999e-05 2024-09-11T03:46:08.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.121250+0000", 2024-09-11T03:46:08.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016738843999999999 2024-09-11T03:46:08.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.121352+0000", 2024-09-11T03:46:08.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102384 2024-09-11T03:46:08.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.121450+0000", 2024-09-11T03:46:08.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8579000000000006e-05 2024-09-11T03:46:08.441 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.442 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.442 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.442 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1634 2.7 2:e23d1762:::smithi17557870-21:head [copy-from ver 214] snapc 40=[40,3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:46:08.442 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:36.206869+0000", 2024-09-11T03:46:08.442 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 332.175377062, 2024-09-11T03:46:08.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017521808999999999, 2024-09-11T03:46:08.442 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.442 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.442 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:08.442 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.442 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.443 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.443 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1634 2024-09-11T03:46:08.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.443 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206869+0000", 2024-09-11T03:46:08.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206869+0000", 2024-09-11T03:46:08.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206872+0000", 2024-09-11T03:46:08.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.926e-06 2024-09-11T03:46:08.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206898+0000", 2024-09-11T03:46:08.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5987e-05 2024-09-11T03:46:08.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206986+0000", 2024-09-11T03:46:08.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7714999999999996e-05 2024-09-11T03:46:08.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207030+0000", 2024-09-11T03:46:08.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3455999999999999e-05 2024-09-11T03:46:08.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207171+0000", 2024-09-11T03:46:08.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014184700000000001 2024-09-11T03:46:08.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207283+0000", 2024-09-11T03:46:08.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000111976 2024-09-11T03:46:08.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:46:08.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.216671+0000", 2024-09-11T03:46:08.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0093877200000000004 2024-09-11T03:46:08.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.223379+0000", 2024-09-11T03:46:08.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067082289999999996 2024-09-11T03:46:08.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:46:08.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224223+0000", 2024-09-11T03:46:08.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000843771 2024-09-11T03:46:08.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224293+0000", 2024-09-11T03:46:08.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0339000000000005e-05 2024-09-11T03:46:08.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:46:08.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224347+0000", 2024-09-11T03:46:08.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3288999999999998e-05 2024-09-11T03:46:08.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224391+0000", 2024-09-11T03:46:08.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4554000000000001e-05 2024-09-11T03:46:08.448 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.448 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.448 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.448 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1635 2.7 2:e23d1762:::smithi17557870-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:46:08.448 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:36.207063+0000", 2024-09-11T03:46:08.448 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 332.175183285, 2024-09-11T03:46:08.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013462717000000001, 2024-09-11T03:46:08.449 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.449 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.449 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:46:08.449 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.449 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.449 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.449 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1635 2024-09-11T03:46:08.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.449 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207063+0000", 2024-09-11T03:46:08.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207063+0000", 2024-09-11T03:46:08.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207064+0000", 2024-09-11T03:46:08.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.043e-06 2024-09-11T03:46:08.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207071+0000", 2024-09-11T03:46:08.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8929999999999999e-06 2024-09-11T03:46:08.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207086+0000", 2024-09-11T03:46:08.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4596999999999999e-05 2024-09-11T03:46:08.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207095+0000", 2024-09-11T03:46:08.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9930000000000001e-06 2024-09-11T03:46:08.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207386+0000", 2024-09-11T03:46:08.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029111399999999998 2024-09-11T03:46:08.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:46:08.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207423+0000", 2024-09-11T03:46:08.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6825000000000001e-05 2024-09-11T03:46:08.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.220376+0000", 2024-09-11T03:46:08.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012953549 2024-09-11T03:46:08.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.220446+0000", 2024-09-11T03:46:08.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9578000000000004e-05 2024-09-11T03:46:08.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.220526+0000", 2024-09-11T03:46:08.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0124999999999996e-05 2024-09-11T03:46:08.453 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.453 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.454 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.454 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1866 2.e 2:7b0c3397:::smithi17557870-14:head [copy-from ver 238] snapc 4c=[4c,4b,49,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:46:08.454 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:46:08.454 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 306.53211344800002, 2024-09-11T03:46:08.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012778199000000001, 2024-09-11T03:46:08.454 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.454 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.454 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:08.454 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.454 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.454 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.454 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1866 2024-09-11T03:46:08.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.455 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:46:08.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:46:08.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850135+0000", 2024-09-11T03:46:08.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5850000000000001e-06 2024-09-11T03:46:08.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850142+0000", 2024-09-11T03:46:08.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2010000000000002e-06 2024-09-11T03:46:08.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.456 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:08.457 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:08.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850160+0000", 2024-09-11T03:46:08.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7858000000000001e-05 2024-09-11T03:46:08.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850170+0000", 2024-09-11T03:46:08.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9219999999999992e-06 2024-09-11T03:46:08.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850235+0000", 2024-09-11T03:46:08.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4849000000000002e-05 2024-09-11T03:46:08.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:46:08.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850246+0000", 2024-09-11T03:46:08.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1938e-05 2024-09-11T03:46:08.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.851390+0000", 2024-09-11T03:46:08.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011435219999999999 2024-09-11T03:46:08.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.851491+0000", 2024-09-11T03:46:08.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000100868 2024-09-11T03:46:08.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:46:08.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.857380+0000", 2024-09-11T03:46:08.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058892839999999998 2024-09-11T03:46:08.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:46:08.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.861564+0000", 2024-09-11T03:46:08.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041843369999999998 2024-09-11T03:46:08.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.861883+0000", 2024-09-11T03:46:08.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031904399999999999 2024-09-11T03:46:08.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.862846+0000", 2024-09-11T03:46:08.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096234000000000001 2024-09-11T03:46:08.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:46:08.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.862883+0000", 2024-09-11T03:46:08.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7132000000000003e-05 2024-09-11T03:46:08.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.862911+0000", 2024-09-11T03:46:08.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8319000000000001e-05 2024-09-11T03:46:08.462 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.463 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.463 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.463 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1895 2.e 2:7efbbea8:::smithi17557870-24:head [copy-from ver 160] snapc 4e=[4e,4d,49,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-09-11T03:46:08.463 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:05.924255+0000", 2024-09-11T03:46:08.463 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 302.45799145199999, 2024-09-11T03:46:08.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016112593000000001, 2024-09-11T03:46:08.463 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.463 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.463 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:08.463 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.463 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.464 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.464 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1895 2024-09-11T03:46:08.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.464 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924255+0000", 2024-09-11T03:46:08.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924255+0000", 2024-09-11T03:46:08.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924257+0000", 2024-09-11T03:46:08.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.742e-06 2024-09-11T03:46:08.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924264+0000", 2024-09-11T03:46:08.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6090000000000003e-06 2024-09-11T03:46:08.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924281+0000", 2024-09-11T03:46:08.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6157000000000001e-05 2024-09-11T03:46:08.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924290+0000", 2024-09-11T03:46:08.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0499999999999997e-06 2024-09-11T03:46:08.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924352+0000", 2024-09-11T03:46:08.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1968e-05 2024-09-11T03:46:08.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:46:08.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924358+0000", 2024-09-11T03:46:08.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1380000000000004e-06 2024-09-11T03:46:08.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.925553+0000", 2024-09-11T03:46:08.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001194802 2024-09-11T03:46:08.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.925622+0000", 2024-09-11T03:46:08.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9598999999999996e-05 2024-09-11T03:46:08.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:46:08.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.931889+0000", 2024-09-11T03:46:08.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062668419999999999 2024-09-11T03:46:08.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:46:08.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.937406+0000", 2024-09-11T03:46:08.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055165659999999997 2024-09-11T03:46:08.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.937537+0000", 2024-09-11T03:46:08.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000131941 2024-09-11T03:46:08.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.940300+0000", 2024-09-11T03:46:08.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027624720000000002 2024-09-11T03:46:08.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:46:08.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.940337+0000", 2024-09-11T03:46:08.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6869999999999998e-05 2024-09-11T03:46:08.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.940368+0000", 2024-09-11T03:46:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0837000000000003e-05 2024-09-11T03:46:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2565 2.7 2:e0eabcfb:::smithi17557870-9:head [copy-from ver 223] snapc 70=[70,6f,6e,6c,6a,68,65,63,5d] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-11T03:46:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:09.876844+0000", 2024-09-11T03:46:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 238.50540291600001, 2024-09-11T03:46:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015411238000000001, 2024-09-11T03:46:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.470 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:08.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.471 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2565 2024-09-11T03:46:08.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.471 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876844+0000", 2024-09-11T03:46:08.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876844+0000", 2024-09-11T03:46:08.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876845+0000", 2024-09-11T03:46:08.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.78e-07 2024-09-11T03:46:08.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876850+0000", 2024-09-11T03:46:08.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9540000000000003e-06 2024-09-11T03:46:08.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876863+0000", 2024-09-11T03:46:08.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3349e-05 2024-09-11T03:46:08.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876873+0000", 2024-09-11T03:46:08.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9590000000000007e-06 2024-09-11T03:46:08.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876919+0000", 2024-09-11T03:46:08.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5720000000000003e-05 2024-09-11T03:46:08.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876984+0000", 2024-09-11T03:46:08.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5437000000000004e-05 2024-09-11T03:46:08.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:46:08.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.887314+0000", 2024-09-11T03:46:08.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0103298 2024-09-11T03:46:08.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:46:08.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.891590+0000", 2024-09-11T03:46:08.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042757380000000003 2024-09-11T03:46:08.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.891885+0000", 2024-09-11T03:46:08.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029539200000000001 2024-09-11T03:46:08.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.892177+0000", 2024-09-11T03:46:08.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029223699999999999 2024-09-11T03:46:08.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:46:08.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.892215+0000", 2024-09-11T03:46:08.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7740000000000001e-05 2024-09-11T03:46:08.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.892255+0000", 2024-09-11T03:46:08.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9934000000000001e-05 2024-09-11T03:46:08.476 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.476 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.476 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.477 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2566 2.7 2:e0eabcfb:::smithi17557870-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e313)", 2024-09-11T03:46:08.477 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:09.876886+0000", 2024-09-11T03:46:08.477 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 238.505360728, 2024-09-11T03:46:08.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014077338, 2024-09-11T03:46:08.477 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.477 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.477 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:46:08.477 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.477 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.477 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.477 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2566 2024-09-11T03:46:08.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.478 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876886+0000", 2024-09-11T03:46:08.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876886+0000", 2024-09-11T03:46:08.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876886+0000", 2024-09-11T03:46:08.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7000000000000005e-07 2024-09-11T03:46:08.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876889+0000", 2024-09-11T03:46:08.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5730000000000002e-06 2024-09-11T03:46:08.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876895+0000", 2024-09-11T03:46:08.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4160000000000001e-06 2024-09-11T03:46:08.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876899+0000", 2024-09-11T03:46:08.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3060000000000002e-06 2024-09-11T03:46:08.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.877088+0000", 2024-09-11T03:46:08.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018935499999999999 2024-09-11T03:46:08.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:46:08.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.877121+0000", 2024-09-11T03:46:08.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2566e-05 2024-09-11T03:46:08.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.890821+0000", 2024-09-11T03:46:08.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013699932999999999 2024-09-11T03:46:08.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.890886+0000", 2024-09-11T03:46:08.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5129999999999995e-05 2024-09-11T03:46:08.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.890963+0000", 2024-09-11T03:46:08.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7489000000000003e-05 2024-09-11T03:46:08.485 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.485 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.485 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.485 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2846 2.7 2:e6da0655:::smithi17557870-27:head [copy-from ver 182] snapc 7d=[7d,7c,7b,7a,79,78,77,75,6c,65] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-09-11T03:46:08.485 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:34.923113+0000", 2024-09-11T03:46:08.485 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 213.459133872, 2024-09-11T03:46:08.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011447241, 2024-09-11T03:46:08.485 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.485 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.485 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:08.486 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.486 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.486 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.486 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2846 2024-09-11T03:46:08.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.486 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.923113+0000", 2024-09-11T03:46:08.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.923113+0000", 2024-09-11T03:46:08.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.923115+0000", 2024-09-11T03:46:08.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9099999999999999e-06 2024-09-11T03:46:08.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.923123+0000", 2024-09-11T03:46:08.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4500000000000004e-06 2024-09-11T03:46:08.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.923141+0000", 2024-09-11T03:46:08.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8314000000000001e-05 2024-09-11T03:46:08.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.923152+0000", 2024-09-11T03:46:08.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0336000000000001e-05 2024-09-11T03:46:08.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.923208+0000", 2024-09-11T03:46:08.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6702e-05 2024-09-11T03:46:08.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.923276+0000", 2024-09-11T03:46:08.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7743000000000005e-05 2024-09-11T03:46:08.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:46:08.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.929687+0000", 2024-09-11T03:46:08.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0064108740000000004 2024-09-11T03:46:08.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:46:08.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.933037+0000", 2024-09-11T03:46:08.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003349781 2024-09-11T03:46:08.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.933319+0000", 2024-09-11T03:46:08.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028245800000000003 2024-09-11T03:46:08.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.934433+0000", 2024-09-11T03:46:08.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011135299999999999 2024-09-11T03:46:08.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:46:08.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.934504+0000", 2024-09-11T03:46:08.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1067000000000002e-05 2024-09-11T03:46:08.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:34.934560+0000", 2024-09-11T03:46:08.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6076000000000003e-05 2024-09-11T03:46:08.491 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.491 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.491 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.491 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:3054 2.e 2:7b0c3397:::smithi17557870-14:head [copy-from ver 182] snapc 82=[81,75] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-11T03:46:08.491 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:52.899536+0000", 2024-09-11T03:46:08.492 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 195.48271097899999, 2024-09-11T03:46:08.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011625175, 2024-09-11T03:46:08.492 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.492 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.492 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:08.492 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.492 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.492 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.492 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3054 2024-09-11T03:46:08.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.492 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.899536+0000", 2024-09-11T03:46:08.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.899536+0000", 2024-09-11T03:46:08.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.899538+0000", 2024-09-11T03:46:08.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.142e-06 2024-09-11T03:46:08.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.899546+0000", 2024-09-11T03:46:08.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8990000000000001e-06 2024-09-11T03:46:08.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.899564+0000", 2024-09-11T03:46:08.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8706999999999999e-05 2024-09-11T03:46:08.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.899576+0000", 2024-09-11T03:46:08.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1221e-05 2024-09-11T03:46:08.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.899633+0000", 2024-09-11T03:46:08.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7630999999999998e-05 2024-09-11T03:46:08.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.899711+0000", 2024-09-11T03:46:08.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7574e-05 2024-09-11T03:46:08.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:46:08.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.902898+0000", 2024-09-11T03:46:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031874999999999998 2024-09-11T03:46:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:46:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.907012+0000", 2024-09-11T03:46:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041138909999999997 2024-09-11T03:46:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.907303+0000", 2024-09-11T03:46:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029084 2024-09-11T03:46:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.911086+0000", 2024-09-11T03:46:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037825810000000001 2024-09-11T03:46:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:46:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.911124+0000", 2024-09-11T03:46:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8399999999999998e-05 2024-09-11T03:46:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:52.911161+0000", 2024-09-11T03:46:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6789000000000003e-05 2024-09-11T03:46:08.497 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:4021 2.7 e435/417 2:e77ce595:::smithi17557870-31:head v 435'563, mlcod=435'561)", 2024-09-11T03:46:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:44:13.499280+0000", 2024-09-11T03:46:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 114.88296683199999, 2024-09-11T03:46:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011214442999999999, 2024-09-11T03:46:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.499280+0000", 2024-09-11T03:46:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.499280+0000", 2024-09-11T03:46:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.499281+0000", 2024-09-11T03:46:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8400000000000002e-07 2024-09-11T03:46:08.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.506341+0000", 2024-09-11T03:46:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0070602399999999997 2024-09-11T03:46:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.506370+0000", 2024-09-11T03:46:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9048999999999999e-05 2024-09-11T03:46:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.506381+0000", 2024-09-11T03:46:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1440000000000001e-05 2024-09-11T03:46:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.506458+0000", 2024-09-11T03:46:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6941999999999996e-05 2024-09-11T03:46:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.506487+0000", 2024-09-11T03:46:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9104000000000001e-05 2024-09-11T03:46:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:46:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.510453+0000", 2024-09-11T03:46:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039655039999999999 2024-09-11T03:46:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:44:13.510494+0000", 2024-09-11T03:46:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1180000000000002e-05 2024-09-11T03:46:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.502 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:4498 2.e 2:7b0c3397:::smithi17557870-14:head [copy-from ver 491] snapc c5=[c5,c4,c3,c2,c1,c0,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-09-11T03:46:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:45:03.958685+0000", 2024-09-11T03:46:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 64.423561077000002, 2024-09-11T03:46:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014061295999999999, 2024-09-11T03:46:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4498 2024-09-11T03:46:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958685+0000", 2024-09-11T03:46:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958685+0000", 2024-09-11T03:46:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958687+0000", 2024-09-11T03:46:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.609e-06 2024-09-11T03:46:08.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958695+0000", 2024-09-11T03:46:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5410000000000003e-06 2024-09-11T03:46:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958711+0000", 2024-09-11T03:46:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6719999999999999e-05 2024-09-11T03:46:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958718+0000", 2024-09-11T03:46:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9720000000000003e-06 2024-09-11T03:46:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958804+0000", 2024-09-11T03:46:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5483999999999997e-05 2024-09-11T03:46:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:46:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958811+0000", 2024-09-11T03:46:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4059999999999999e-06 2024-09-11T03:46:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.960161+0000", 2024-09-11T03:46:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001349772 2024-09-11T03:46:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.960244+0000", 2024-09-11T03:46:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3461999999999995e-05 2024-09-11T03:46:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:46:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.967582+0000", 2024-09-11T03:46:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073378480000000001 2024-09-11T03:46:08.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:46:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.970934+0000", 2024-09-11T03:46:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003351812 2024-09-11T03:46:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.971224+0000", 2024-09-11T03:46:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029005399999999998 2024-09-11T03:46:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.972661+0000", 2024-09-11T03:46:08.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001436866 2024-09-11T03:46:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:46:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.972700+0000", 2024-09-11T03:46:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9341000000000002e-05 2024-09-11T03:46:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.972747+0000", 2024-09-11T03:46:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6409000000000002e-05 2024-09-11T03:46:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.509 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:4499 2.e 2:7b0c3397:::smithi17557870-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e486)", 2024-09-11T03:46:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:45:03.958737+0000", 2024-09-11T03:46:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 64.423509537000001, 2024-09-11T03:46:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011700933, 2024-09-11T03:46:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:46:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:46:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:46:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:46:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:46:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4499 2024-09-11T03:46:08.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:46:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:46:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958737+0000", 2024-09-11T03:46:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:46:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958737+0000", 2024-09-11T03:46:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:46:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:46:08.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958737+0000", 2024-09-11T03:46:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9399999999999995e-07 2024-09-11T03:46:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:46:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958740+0000", 2024-09-11T03:46:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6970000000000002e-06 2024-09-11T03:46:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:46:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958754+0000", 2024-09-11T03:46:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4151999999999999e-05 2024-09-11T03:46:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:46:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958759+0000", 2024-09-11T03:46:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4839999999999998e-06 2024-09-11T03:46:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958882+0000", 2024-09-11T03:46:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012276799999999999 2024-09-11T03:46:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:46:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958887+0000", 2024-09-11T03:46:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5149999999999997e-06 2024-09-11T03:46:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.960350+0000", 2024-09-11T03:46:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001463245 2024-09-11T03:46:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:46:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.960391+0000", 2024-09-11T03:46:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0261e-05 2024-09-11T03:46:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:46:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.970279+0000", 2024-09-11T03:46:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0098882150000000005 2024-09-11T03:46:08.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:46:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.970358+0000", 2024-09-11T03:46:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9448000000000002e-05 2024-09-11T03:46:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:46:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:46:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:46:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.970438+0000", 2024-09-11T03:46:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9653999999999993e-05 2024-09-11T03:46:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:46:08.515 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:46:08.516 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:46:08.516 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:08 smithi175 ceph-mon[30894]: pgmap v978: 57 pgs: 57 active+clean; 180 MiB data, 780 MiB used, 983 GiB / 983 GiB avail; 23 MiB/s rd, 3.1 MiB/s wr, 26 op/s 2024-09-11T03:46:08.516 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:08 smithi175 ceph-mon[30894]: from='client.16287 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:46:08.517 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:08 smithi175 ceph-mon[30894]: osdmap e550: 12 total, 12 up, 11 in 2024-09-11T03:46:08.517 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:08 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:46:08.517 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:08 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:46:08.541 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:08.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:08.542 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:08.644 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:08.665 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:08.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:08.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:08.767 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:08.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:08.789 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:08.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:08.890 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:08.916 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:08.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:08.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:08.960 INFO:tasks.rados.rados.2.smithi175.stdout:2060: done (0 left) 2024-09-11T03:46:08.960 INFO:tasks.rados.rados.2.smithi175.stdout:2061: write oid 42 current snap is 229 2024-09-11T03:46:08.960 INFO:tasks.rados.rados.2.smithi175.stdout:2061: seq_num 700 ranges {564900=578776,1629160=709913,3073389=492409} 2024-09-11T03:46:08.966 INFO:tasks.rados.rados.2.smithi175.stdout:2061: writing smithi17557870-42 from 564900 to 1143676 tid 1 2024-09-11T03:46:08.973 INFO:tasks.rados.rados.2.smithi175.stdout:2061: writing smithi17557870-42 from 1629160 to 2339073 tid 2 2024-09-11T03:46:08.980 INFO:tasks.rados.rados.2.smithi175.stdout:2061: writing smithi17557870-42 from 3073389 to 3565798 tid 3 2024-09-11T03:46:08.980 INFO:tasks.rados.rados.2.smithi175.stdout:2062: snap_remove snap 226 2024-09-11T03:46:09.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:09.044 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:09.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:09.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:09.170 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:09.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:09.272 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:09.293 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:09.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:09.395 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:09.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:09.417 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:09.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:09.513 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T03:46:09.518 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:09.578 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:09.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:09.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:09.680 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:09.706 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:09.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:09.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:09.808 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:09.833 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:09.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:09.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:09.935 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:09.957 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:09.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:09.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:09.972 INFO:tasks.rados.rados.2.smithi175.stdout:2062: done (1 left) 2024-09-11T03:46:09.972 INFO:tasks.rados.rados.2.smithi175.stdout:2063: read oid 35 snap -1 2024-09-11T03:46:09.972 INFO:tasks.rados.rados.2.smithi175.stdout:2063: expect deleted 2024-09-11T03:46:09.972 INFO:tasks.rados.rados.2.smithi175.stdout:2064: write oid 5 current snap is 229 2024-09-11T03:46:09.972 INFO:tasks.rados.rados.2.smithi175.stdout:2061: finishing write tid 1 to smithi17557870-42 2024-09-11T03:46:09.972 INFO:tasks.rados.rados.2.smithi175.stdout:2064: seq_num 701 ranges {581707=654025,1687642=453571,2700812=703681} 2024-09-11T03:46:09.978 INFO:tasks.rados.rados.2.smithi175.stdout:2064: writing smithi17557870-5 from 581707 to 1235732 tid 1 2024-09-11T03:46:09.983 INFO:tasks.rados.rados.2.smithi175.stdout:2064: writing smithi17557870-5 from 1687642 to 2141213 tid 2 2024-09-11T03:46:09.990 INFO:tasks.rados.rados.2.smithi175.stdout:2064: writing smithi17557870-5 from 2700812 to 3404493 tid 3 2024-09-11T03:46:09.991 INFO:tasks.rados.rados.2.smithi175.stdout:2065: read oid 41 snap -1 2024-09-11T03:46:09.991 INFO:tasks.rados.rados.2.smithi175.stdout:2065: expect (ObjNum 609 snap 191 seq_num 609) 2024-09-11T03:46:09.991 INFO:tasks.rados.rados.2.smithi175.stdout:2066: snap_remove snap 221 2024-09-11T03:46:10.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:10.060 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:10.085 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:10.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:10.095 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:10.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:10.190 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:46:10.205 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:09 smithi084 ceph-mon[27254]: pgmap v981: 57 pgs: 57 active+clean; 184 MiB data, 782 MiB used, 983 GiB / 983 GiB avail; 33 MiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-11T03:46:10.205 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:09 smithi084 ceph-mon[27254]: osdmap e551: 12 total, 12 up, 11 in 2024-09-11T03:46:10.210 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:10.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:10.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:10.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:09 smithi175 ceph-mon[30894]: pgmap v981: 57 pgs: 57 active+clean; 184 MiB data, 782 MiB used, 983 GiB / 983 GiB avail; 33 MiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-11T03:46:10.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:09 smithi175 ceph-mon[30894]: osdmap e551: 12 total, 12 up, 11 in 2024-09-11T03:46:10.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:10.336 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:10.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:10.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:10.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:09 smithi121 ceph-mon[30729]: pgmap v981: 57 pgs: 57 active+clean; 184 MiB data, 782 MiB used, 983 GiB / 983 GiB avail; 33 MiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-09-11T03:46:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:09 smithi121 ceph-mon[30729]: osdmap e551: 12 total, 12 up, 11 in 2024-09-11T03:46:10.438 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:10.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:10.463 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:10.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:10.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:10.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:10.590 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:10.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:10.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:10.715 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:10.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:10.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:10.727 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:46:10.727 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:46:10.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:10.841 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:10.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:10.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:10.943 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:10.970 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:46:10.982 INFO:tasks.rados.rados.2.smithi175.stdout:2061: finishing write tid 2 to smithi17557870-42 2024-09-11T03:46:10.982 INFO:tasks.rados.rados.2.smithi175.stdout:2061: finishing write tid 3 to smithi17557870-42 2024-09-11T03:46:10.982 INFO:tasks.rados.rados.2.smithi175.stdout:2061: finishing write tid 4 to smithi17557870-42 2024-09-11T03:46:10.982 INFO:tasks.rados.rados.2.smithi175.stdout:2061: finishing write tid 5 to smithi17557870-42 2024-09-11T03:46:10.982 INFO:tasks.rados.rados.2.smithi175.stdout:2061: oid 42 updating version 0 to 665 2024-09-11T03:46:10.982 INFO:tasks.rados.rados.2.smithi175.stdout:2061: oid 42 updating version 665 to 666 2024-09-11T03:46:10.982 INFO:tasks.rados.rados.2.smithi175.stdout:2061: oid 42 updating version 666 to 667 2024-09-11T03:46:10.982 INFO:tasks.rados.rados.2.smithi175.stdout:2061: oid 42 version 667 is already newer than 664 2024-09-11T03:46:10.982 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 667 (ObjNum 700 snap 229 seq_num 700) dirty exists 2024-09-11T03:46:10.982 INFO:tasks.rados.rados.2.smithi175.stdout:2061: left oid 42 (ObjNum 700 snap 229 seq_num 700) 2024-09-11T03:46:10.982 INFO:tasks.rados.rados.2.smithi175.stdout:2064: finishing write tid 1 to smithi17557870-5 2024-09-11T03:46:10.982 INFO:tasks.rados.rados.2.smithi175.stdout:2064: finishing write tid 2 to smithi17557870-5 2024-09-11T03:46:10.983 INFO:tasks.rados.rados.2.smithi175.stdout:2065: expect (ObjNum 609 snap 191 seq_num 609) 2024-09-11T03:46:10.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:10 smithi084 ceph-mon[27254]: osdmap e552: 12 total, 12 up, 11 in 2024-09-11T03:46:10.988 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:10 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4269410590' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:46:10.988 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:10 smithi084 ceph-mon[27254]: from='client.16299 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:46:11.000 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:11.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:11.086 INFO:tasks.rados.rados.2.smithi175.stdout:2064: finishing write tid 3 to smithi17557870-5 2024-09-11T03:46:11.086 INFO:tasks.rados.rados.2.smithi175.stdout:2064: finishing write tid 4 to smithi17557870-5 2024-09-11T03:46:11.086 INFO:tasks.rados.rados.2.smithi175.stdout:2064: finishing write tid 5 to smithi17557870-5 2024-09-11T03:46:11.086 INFO:tasks.rados.rados.2.smithi175.stdout:2064: oid 5 updating version 0 to 696 2024-09-11T03:46:11.086 INFO:tasks.rados.rados.2.smithi175.stdout:2064: oid 5 updating version 696 to 697 2024-09-11T03:46:11.087 INFO:tasks.rados.rados.2.smithi175.stdout:2064: oid 5 version 697 is already newer than 695 2024-09-11T03:46:11.087 INFO:tasks.rados.rados.2.smithi175.stdout:2064: oid 5 version 697 is already newer than 694 2024-09-11T03:46:11.087 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 697 (ObjNum 701 snap 229 seq_num 701) dirty exists 2024-09-11T03:46:11.087 INFO:tasks.rados.rados.2.smithi175.stdout:2064: left oid 5 (ObjNum 701 snap 229 seq_num 701) 2024-09-11T03:46:11.087 INFO:tasks.rados.rados.2.smithi175.stdout:2061: done (4 left) 2024-09-11T03:46:11.087 INFO:tasks.rados.rados.2.smithi175.stdout:2063: done (3 left) 2024-09-11T03:46:11.087 INFO:tasks.rados.rados.2.smithi175.stdout:2064: done (2 left) 2024-09-11T03:46:11.087 INFO:tasks.rados.rados.2.smithi175.stdout:2065: done (1 left) 2024-09-11T03:46:11.087 INFO:tasks.rados.rados.2.smithi175.stdout:2066: done (0 left) 2024-09-11T03:46:11.087 INFO:tasks.rados.rados.2.smithi175.stdout:2067: snap_remove snap 206 2024-09-11T03:46:11.102 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:11.123 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:11.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:11.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:11.225 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:11.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:10 smithi175 ceph-mon[30894]: osdmap e552: 12 total, 12 up, 11 in 2024-09-11T03:46:11.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:10 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4269410590' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:46:11.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:10 smithi175 ceph-mon[30894]: from='client.16299 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:46:11.247 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:11.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:11.248 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:11.349 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:11.370 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:11.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:11.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:10 smithi121 ceph-mon[30729]: osdmap e552: 12 total, 12 up, 11 in 2024-09-11T03:46:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4269410590' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:46:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:10 smithi121 ceph-mon[30729]: from='client.16299 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:46:11.471 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:11.493 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:11.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:11.494 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:11.575 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":983,"stamp":"2024-09-11T03:46:09.969439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":192528963,"num_objects":101,"num_object_clones":57,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":12,"num_read":2881,"num_read_kb":2778553,"num_write":2007,"num_write_kb":768133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":61873709,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13273,"ondisk_log_size":13273,"up":171,"acting":171,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":171,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":800740,"kb_used_data":414592,"kb_used_omap":17,"kb_used_meta":385838,"kb_avail":1030395932,"statfs":{"total":1055945392128,"available":1055125434368,"internally_reserved":0,"allocated":424542208,"data_stored":513972368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":395098997},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15677752,"num_objects":13,"num_object_clones":12,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":4,"num_read":114,"num_read_kb":135864,"num_write":46,"num_write_kb":16694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.957276"},"pg_stats":[{"pgid":"1.1e","version":"20'92","reported_seq":680,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365923+0000","last_change":"2024-09-11T03:44:24.235660+0000","last_active":"2024-09-11T03:46:06.365923+0000","last_peered":"2024-09-11T03:46:06.365923+0000","last_clean":"2024-09-11T03:46:06.365923+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:46:06.365923+0000","last_undegraded":"2024-09-11T03:46:06.365923+0000","last_fullsized":"2024-09-11T03:46:06.365923+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:24.235616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:24.235616+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:57.069193+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'92","reported_seq":668,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365675+0000","last_change":"2024-09-11T03:40:41.348889+0000","last_active":"2024-09-11T03:46:06.365675+0000","last_peered":"2024-09-11T03:46:06.365675+0000","last_clean":"2024-09-11T03:46:06.365675+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:46:06.365675+0000","last_undegraded":"2024-09-11T03:46:06.365675+0000","last_fullsized":"2024-09-11T03:46:06.365675+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:41.348828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:41.348828+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:20.867494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'92","reported_seq":671,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013215+0000","last_change":"2024-09-11T03:40:40.188174+0000","last_active":"2024-09-11T03:46:06.013215+0000","last_peered":"2024-09-11T03:46:06.013215+0000","last_clean":"2024-09-11T03:46:06.013215+0000","last_became_active":"2024-09-11T03:40:39.848230+0000","last_became_peered":"2024-09-11T03:40:39.848230+0000","last_unstale":"2024-09-11T03:46:06.013215+0000","last_undegraded":"2024-09-11T03:46:06.013215+0000","last_fullsized":"2024-09-11T03:46:06.013215+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:40.188124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:40.188124+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:19.492283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,11],"acting":[8,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"547'357","reported_seq":2721,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-09-11T03:46:07.965172+0000","last_change":"2024-09-11T03:46:07.965743+0000","last_active":"2024-09-11T03:46:07.965172+0000","last_peered":"2024-09-11T03:46:07.965172+0000","last_clean":"2024-09-11T03:46:07.965172+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:46:07.965172+0000","last_undegraded":"2024-09-11T03:46:07.965172+0000","last_fullsized":"2024-09-11T03:46:07.965172+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"467'328","last_scrub_stamp":"2024-09-11T03:44:45.863666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:45.863666+0000","objects_scrubbed":4,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017644850000000001,"stat_sum":{"num_bytes":2739673,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":99504,"num_write":46,"num_write_kb":16817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4865740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d4","length":"1"},{"start":"dc","length":"2"},{"start":"e2","length":"2"}]},{"pgid":"1.1b","version":"20'92","reported_seq":671,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013245+0000","last_change":"2024-09-11T03:40:42.166354+0000","last_active":"2024-09-11T03:46:06.013245+0000","last_peered":"2024-09-11T03:46:06.013245+0000","last_clean":"2024-09-11T03:46:06.013245+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:46:06.013245+0000","last_undegraded":"2024-09-11T03:46:06.013245+0000","last_fullsized":"2024-09-11T03:46:06.013245+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:42.166295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:42.166295+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:21.519903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.19","version":"541'445","reported_seq":2811,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-09-11T03:46:05.020101+0000","last_change":"2024-09-11T03:46:05.020677+0000","last_active":"2024-09-11T03:46:05.020101+0000","last_peered":"2024-09-11T03:46:05.020101+0000","last_clean":"2024-09-11T03:46:05.020101+0000","last_became_active":"2024-09-11T03:45:09.975629+0000","last_became_peered":"2024-09-11T03:45:09.975629+0000","last_unstale":"2024-09-11T03:46:05.020101+0000","last_undegraded":"2024-09-11T03:46:05.020101+0000","last_fullsized":"2024-09-11T03:46:05.020101+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"504'431","last_scrub_stamp":"2024-09-11T03:45:23.194125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:23.194125+0000","objects_scrubbed":2,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:45.065837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00081455700000000004,"stat_sum":{"num_bytes":4480490,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":45909,"num_write":25,"num_write_kb":12177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,8],"acting":[0,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.1a","version":"20'92","reported_seq":434,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365236+0000","last_change":"2024-09-11T03:45:52.914136+0000","last_active":"2024-09-11T03:46:06.365236+0000","last_peered":"2024-09-11T03:46:06.365236+0000","last_clean":"2024-09-11T03:46:06.365236+0000","last_became_active":"2024-09-11T03:44:43.938873+0000","last_became_peered":"2024-09-11T03:44:43.938873+0000","last_unstale":"2024-09-11T03:46:06.365236+0000","last_undegraded":"2024-09-11T03:46:06.365236+0000","last_fullsized":"2024-09-11T03:46:06.365236+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:52.914088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:52.914088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:10.451317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"20'92","reported_seq":296,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.963871+0000","last_change":"2024-09-11T03:45:30.424485+0000","last_active":"2024-09-11T03:46:06.963871+0000","last_peered":"2024-09-11T03:46:06.963871+0000","last_clean":"2024-09-11T03:46:06.963871+0000","last_became_active":"2024-09-11T03:43:52.920761+0000","last_became_peered":"2024-09-11T03:43:52.920761+0000","last_unstale":"2024-09-11T03:46:06.963871+0000","last_undegraded":"2024-09-11T03:46:06.963871+0000","last_fullsized":"2024-09-11T03:46:06.963871+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:30.424428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:30.424428+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:57.891319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'92","reported_seq":803,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365642+0000","last_change":"2024-09-11T03:44:46.470994+0000","last_active":"2024-09-11T03:46:06.365642+0000","last_peered":"2024-09-11T03:46:06.365642+0000","last_clean":"2024-09-11T03:46:06.365642+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:46:06.365642+0000","last_undegraded":"2024-09-11T03:46:06.365642+0000","last_fullsized":"2024-09-11T03:46:06.365642+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:46.470921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:46.470921+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:13.212885+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"541'254","reported_seq":2566,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-09-11T03:46:07.965106+0000","last_change":"2024-09-11T03:46:07.965573+0000","last_active":"2024-09-11T03:46:07.965106+0000","last_peered":"2024-09-11T03:46:07.965106+0000","last_clean":"2024-09-11T03:46:07.965106+0000","last_became_active":"2024-09-11T03:41:38.890690+0000","last_became_peered":"2024-09-11T03:41:38.890690+0000","last_unstale":"2024-09-11T03:46:07.965106+0000","last_undegraded":"2024-09-11T03:46:07.965106+0000","last_fullsized":"2024-09-11T03:46:07.965106+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"437'223","last_scrub_stamp":"2024-09-11T03:44:42.964833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:42.964833+0000","objects_scrubbed":1,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001630016,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":8231,"num_write":22,"num_write_kb":4725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[9,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d4","length":"1"},{"start":"dc","length":"2"},{"start":"e2","length":"2"}]},{"pgid":"1.17","version":"20'92","reported_seq":694,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013484+0000","last_change":"2024-09-11T03:45:07.446084+0000","last_active":"2024-09-11T03:46:06.013484+0000","last_peered":"2024-09-11T03:46:06.013484+0000","last_clean":"2024-09-11T03:46:06.013484+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:46:06.013484+0000","last_undegraded":"2024-09-11T03:46:06.013484+0000","last_fullsized":"2024-09-11T03:46:06.013484+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:07.445986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:07.445986+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:11.836438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":2608,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-09-11T03:46:07.964383+0000","last_change":"2024-09-11T03:46:07.965002+0000","last_active":"2024-09-11T03:46:07.964383+0000","last_peered":"2024-09-11T03:46:07.964383+0000","last_clean":"2024-09-11T03:46:07.964383+0000","last_became_active":"2024-09-11T03:41:38.888228+0000","last_became_peered":"2024-09-11T03:41:38.888228+0000","last_unstale":"2024-09-11T03:46:07.964383+0000","last_undegraded":"2024-09-11T03:46:07.964383+0000","last_fullsized":"2024-09-11T03:46:07.964383+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:45:28.993767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:28.993767+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:56.989926+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0011417210000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d4","length":"1"},{"start":"dc","length":"2"},{"start":"e2","length":"2"}]},{"pgid":"1.16","version":"20'92","reported_seq":148,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365794+0000","last_change":"2024-09-11T03:46:05.016475+0000","last_active":"2024-09-11T03:46:06.365794+0000","last_peered":"2024-09-11T03:46:06.365794+0000","last_clean":"2024-09-11T03:46:06.365794+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:46:06.365794+0000","last_undegraded":"2024-09-11T03:46:06.365794+0000","last_fullsized":"2024-09-11T03:46:06.365794+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:23.840184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:23.840184+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:46:34.095648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":36063,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.16","version":"541'363","reported_seq":2644,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365979+0000","last_change":"2024-09-11T03:46:05.019193+0000","last_active":"2024-09-11T03:46:06.365979+0000","last_peered":"2024-09-11T03:46:06.365979+0000","last_clean":"2024-09-11T03:46:06.365979+0000","last_became_active":"2024-09-11T03:41:38.887856+0000","last_became_peered":"2024-09-11T03:41:38.887856+0000","last_unstale":"2024-09-11T03:46:06.365979+0000","last_undegraded":"2024-09-11T03:46:06.365979+0000","last_fullsized":"2024-09-11T03:46:06.365979+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"531'357","last_scrub_stamp":"2024-09-11T03:45:54.655442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:54.655442+0000","objects_scrubbed":4,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:11.754612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0051539560000000003,"stat_sum":{"num_bytes":9868864,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":50,"num_read_kb":36893,"num_write":57,"num_write_kb":23709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,9],"acting":[5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":676,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013544+0000","last_change":"2024-09-11T03:43:09.565121+0000","last_active":"2024-09-11T03:46:06.013544+0000","last_peered":"2024-09-11T03:46:06.013544+0000","last_clean":"2024-09-11T03:46:06.013544+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:46:06.013544+0000","last_undegraded":"2024-09-11T03:46:06.013544+0000","last_fullsized":"2024-09-11T03:46:06.013544+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:09.565070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:09.565070+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:32.580107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":2009,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.966593+0000","last_change":"2024-09-11T03:46:06.966891+0000","last_active":"2024-09-11T03:46:06.966593+0000","last_peered":"2024-09-11T03:46:06.966593+0000","last_clean":"2024-09-11T03:46:06.966593+0000","last_became_active":"2024-09-11T03:43:55.918743+0000","last_became_peered":"2024-09-11T03:43:55.918743+0000","last_unstale":"2024-09-11T03:46:06.966593+0000","last_undegraded":"2024-09-11T03:46:06.966593+0000","last_fullsized":"2024-09-11T03:46:06.966593+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:45:13.319584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:13.319584+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:36.932063+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0023772620000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[10,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"dc","length":"2"},{"start":"e2","length":"2"}]},{"pgid":"1.14","version":"20'92","reported_seq":718,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-09-11T03:46:07.967199+0000","last_change":"2024-09-11T03:44:12.182364+0000","last_active":"2024-09-11T03:46:07.967199+0000","last_peered":"2024-09-11T03:46:07.967199+0000","last_clean":"2024-09-11T03:46:07.967199+0000","last_became_active":"2024-09-11T03:40:27.847297+0000","last_became_peered":"2024-09-11T03:40:27.847297+0000","last_unstale":"2024-09-11T03:46:07.967199+0000","last_undegraded":"2024-09-11T03:46:07.967199+0000","last_fullsized":"2024-09-11T03:46:07.967199+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:12.182320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:12.182320+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:00.159862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":1389,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013267+0000","last_change":"2024-09-11T03:46:05.059273+0000","last_active":"2024-09-11T03:46:06.013267+0000","last_peered":"2024-09-11T03:46:06.013267+0000","last_clean":"2024-09-11T03:46:06.013267+0000","last_became_active":"2024-09-11T03:41:35.874295+0000","last_became_peered":"2024-09-11T03:41:35.874295+0000","last_unstale":"2024-09-11T03:46:06.013267+0000","last_undegraded":"2024-09-11T03:46:06.013267+0000","last_fullsized":"2024-09-11T03:46:06.013267+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:45:53.606467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:53.606467+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:13.208602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069043000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.13","version":"20'92","reported_seq":814,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365381+0000","last_change":"2024-09-11T03:45:47.911982+0000","last_active":"2024-09-11T03:46:06.365381+0000","last_peered":"2024-09-11T03:46:06.365381+0000","last_clean":"2024-09-11T03:46:06.365381+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:46:06.365381+0000","last_undegraded":"2024-09-11T03:46:06.365381+0000","last_fullsized":"2024-09-11T03:46:06.365381+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:47.911918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:47.911918+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:14.917780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":1446,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365939+0000","last_change":"2024-09-11T03:46:05.016763+0000","last_active":"2024-09-11T03:46:06.365939+0000","last_peered":"2024-09-11T03:46:06.365939+0000","last_clean":"2024-09-11T03:46:06.365939+0000","last_became_active":"2024-09-11T03:44:43.939694+0000","last_became_peered":"2024-09-11T03:44:43.939694+0000","last_unstale":"2024-09-11T03:46:06.365939+0000","last_undegraded":"2024-09-11T03:46:06.365939+0000","last_fullsized":"2024-09-11T03:46:06.365939+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:45:53.909148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:53.909148+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:20.606178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002806144,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,9],"acting":[1,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":708,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365671+0000","last_change":"2024-09-11T03:45:09.782668+0000","last_active":"2024-09-11T03:46:06.365671+0000","last_peered":"2024-09-11T03:46:06.365671+0000","last_clean":"2024-09-11T03:46:06.365671+0000","last_became_active":"2024-09-11T03:40:27.845249+0000","last_became_peered":"2024-09-11T03:40:27.845249+0000","last_unstale":"2024-09-11T03:46:06.365671+0000","last_undegraded":"2024-09-11T03:46:06.365671+0000","last_fullsized":"2024-09-11T03:46:06.365671+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:09.782621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:09.782621+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:33.934257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"549'380","reported_seq":2691,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.966398+0000","last_change":"2024-09-11T03:46:06.966756+0000","last_active":"2024-09-11T03:46:06.966398+0000","last_peered":"2024-09-11T03:46:06.966398+0000","last_clean":"2024-09-11T03:46:06.966398+0000","last_became_active":"2024-09-11T03:43:55.918817+0000","last_became_peered":"2024-09-11T03:43:55.918817+0000","last_unstale":"2024-09-11T03:46:06.966398+0000","last_undegraded":"2024-09-11T03:46:06.966398+0000","last_fullsized":"2024-09-11T03:46:06.966398+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"513'359","last_scrub_stamp":"2024-09-11T03:45:31.423318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:31.423318+0000","objects_scrubbed":4,"log_size":380,"log_dups_size":0,"ondisk_log_size":380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:44.164014+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0026271900000000002,"stat_sum":{"num_bytes":7797818,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":129,"num_read_kb":121603,"num_write":89,"num_write_kb":34392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":15216236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"dc","length":"2"},{"start":"e2","length":"2"}]},{"pgid":"1.11","version":"20'92","reported_seq":697,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-09-11T03:46:05.020184+0000","last_change":"2024-09-11T03:40:35.703292+0000","last_active":"2024-09-11T03:46:05.020184+0000","last_peered":"2024-09-11T03:46:05.020184+0000","last_clean":"2024-09-11T03:46:05.020184+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:46:05.020184+0000","last_undegraded":"2024-09-11T03:46:05.020184+0000","last_fullsized":"2024-09-11T03:46:05.020184+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.703203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.703203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:00.137853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":2387,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-09-11T03:46:07.964463+0000","last_change":"2024-09-11T03:46:07.965025+0000","last_active":"2024-09-11T03:46:07.964463+0000","last_peered":"2024-09-11T03:46:07.964463+0000","last_clean":"2024-09-11T03:46:07.964463+0000","last_became_active":"2024-09-11T03:41:38.886834+0000","last_became_peered":"2024-09-11T03:41:38.886834+0000","last_unstale":"2024-09-11T03:46:07.964463+0000","last_undegraded":"2024-09-11T03:46:07.964463+0000","last_fullsized":"2024-09-11T03:46:07.964463+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:44:46.858645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:46.858645+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011406070000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d4","length":"1"},{"start":"dc","length":"2"},{"start":"e2","length":"2"}]},{"pgid":"1.10","version":"20'92","reported_seq":718,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365498+0000","last_change":"2024-09-11T03:44:26.391068+0000","last_active":"2024-09-11T03:46:06.365498+0000","last_peered":"2024-09-11T03:46:06.365498+0000","last_clean":"2024-09-11T03:46:06.365498+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:46:06.365498+0000","last_undegraded":"2024-09-11T03:46:06.365498+0000","last_fullsized":"2024-09-11T03:46:06.365498+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:26.391001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:26.391001+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:32.609928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"547'616","reported_seq":2901,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013420+0000","last_change":"2024-09-11T03:46:05.061054+0000","last_active":"2024-09-11T03:46:06.013420+0000","last_peered":"2024-09-11T03:46:06.013420+0000","last_clean":"2024-09-11T03:46:06.013420+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:46:06.013420+0000","last_undegraded":"2024-09-11T03:46:06.013420+0000","last_fullsized":"2024-09-11T03:46:06.013420+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'489","last_scrub_stamp":"2024-09-11T03:44:48.449022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:48.449022+0000","objects_scrubbed":6,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:08.216315+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0025474349999999998,"stat_sum":{"num_bytes":12188856,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":215,"num_read_kb":178596,"num_write":178,"num_write_kb":82238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":776,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365967+0000","last_change":"2024-09-11T03:42:05.135033+0000","last_active":"2024-09-11T03:46:06.365967+0000","last_peered":"2024-09-11T03:46:06.365967+0000","last_clean":"2024-09-11T03:46:06.365967+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:46:06.365967+0000","last_undegraded":"2024-09-11T03:46:06.365967+0000","last_fullsized":"2024-09-11T03:46:06.365967+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:05.134977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:05.134977+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:20.668974+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"547'265","reported_seq":2565,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365681+0000","last_change":"2024-09-11T03:46:05.020409+0000","last_active":"2024-09-11T03:46:06.365681+0000","last_peered":"2024-09-11T03:46:06.365681+0000","last_clean":"2024-09-11T03:46:06.365681+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:46:06.365681+0000","last_undegraded":"2024-09-11T03:46:06.365681+0000","last_fullsized":"2024-09-11T03:46:06.365681+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"535'255","last_scrub_stamp":"2024-09-11T03:45:54.125664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:54.125664+0000","objects_scrubbed":4,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:19.394574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0037024979999999998,"stat_sum":{"num_bytes":5467682,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":146,"num_read_kb":149021,"num_write":89,"num_write_kb":32129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":1105,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013408+0000","last_change":"2024-09-11T03:42:00.816905+0000","last_active":"2024-09-11T03:46:06.013408+0000","last_peered":"2024-09-11T03:46:06.013408+0000","last_clean":"2024-09-11T03:46:06.013408+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:46:06.013408+0000","last_undegraded":"2024-09-11T03:46:06.013408+0000","last_fullsized":"2024-09-11T03:46:06.013408+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:00.816850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:00.816850+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:21.512672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"547'809","reported_seq":3209,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365626+0000","last_change":"2024-09-11T03:46:05.021312+0000","last_active":"2024-09-11T03:46:06.365626+0000","last_peered":"2024-09-11T03:46:06.365626+0000","last_clean":"2024-09-11T03:46:06.365626+0000","last_became_active":"2024-09-11T03:41:33.865638+0000","last_became_peered":"2024-09-11T03:41:33.865638+0000","last_unstale":"2024-09-11T03:46:06.365626+0000","last_undegraded":"2024-09-11T03:46:06.365626+0000","last_fullsized":"2024-09-11T03:46:06.365626+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'731","last_scrub_stamp":"2024-09-11T03:45:23.948691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:23.948691+0000","objects_scrubbed":13,"log_size":809,"log_dups_size":0,"ondisk_log_size":809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:51.368410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0072290139999999998,"stat_sum":{"num_bytes":24815258,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":314,"num_read_kb":347374,"num_write":160,"num_write_kb":55087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":1038,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-09-11T03:46:07.963699+0000","last_change":"2024-09-11T03:45:24.066287+0000","last_active":"2024-09-11T03:46:07.963699+0000","last_peered":"2024-09-11T03:46:07.963699+0000","last_clean":"2024-09-11T03:46:07.963699+0000","last_became_active":"2024-09-11T03:40:37.849988+0000","last_became_peered":"2024-09-11T03:40:37.849988+0000","last_unstale":"2024-09-11T03:46:07.963699+0000","last_undegraded":"2024-09-11T03:46:07.963699+0000","last_fullsized":"2024-09-11T03:46:07.963699+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:24.066237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:24.066237+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:29.248184+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,0],"acting":[9,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"542'320","reported_seq":2620,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013540+0000","last_change":"2024-09-11T03:46:05.009515+0000","last_active":"2024-09-11T03:46:06.013540+0000","last_peered":"2024-09-11T03:46:06.013540+0000","last_clean":"2024-09-11T03:46:06.013540+0000","last_became_active":"2024-09-11T03:41:33.863875+0000","last_became_peered":"2024-09-11T03:41:33.863875+0000","last_unstale":"2024-09-11T03:46:06.013540+0000","last_undegraded":"2024-09-11T03:46:06.013540+0000","last_fullsized":"2024-09-11T03:46:06.013540+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'283","last_scrub_stamp":"2024-09-11T03:44:29.473127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:29.473127+0000","objects_scrubbed":1,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00061925399999999996,"stat_sum":{"num_bytes":2470761,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":25,"num_read_kb":13817,"num_write":34,"num_write_kb":9341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":708,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365616+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:46:06.365616+0000","last_peered":"2024-09-11T03:46:06.365616+0000","last_clean":"2024-09-11T03:46:06.365616+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:46:06.365616+0000","last_undegraded":"2024-09-11T03:46:06.365616+0000","last_fullsized":"2024-09-11T03:46:06.365616+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:05.070582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"549'820","reported_seq":3222,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.966296+0000","last_change":"2024-09-11T03:46:06.966547+0000","last_active":"2024-09-11T03:46:06.966296+0000","last_peered":"2024-09-11T03:46:06.966296+0000","last_clean":"2024-09-11T03:46:06.966296+0000","last_became_active":"2024-09-11T03:43:55.918858+0000","last_became_peered":"2024-09-11T03:43:55.918858+0000","last_unstale":"2024-09-11T03:46:06.966296+0000","last_undegraded":"2024-09-11T03:46:06.966296+0000","last_fullsized":"2024-09-11T03:46:06.966296+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"524'761","last_scrub_stamp":"2024-09-11T03:45:45.237733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:45.237733+0000","objects_scrubbed":11,"log_size":820,"log_dups_size":0,"ondisk_log_size":820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:47.750112+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0022320899999999999,"stat_sum":{"num_bytes":23894519,"num_objects":13,"num_object_clones":8,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":279,"num_read_kb":332338,"num_write":136,"num_write_kb":55047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":14282259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,3],"acting":[10,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"dc","length":"2"},{"start":"e2","length":"2"}]},{"pgid":"1.4","version":"20'92","reported_seq":889,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365866+0000","last_change":"2024-09-11T03:44:28.192468+0000","last_active":"2024-09-11T03:46:06.365866+0000","last_peered":"2024-09-11T03:46:06.365866+0000","last_clean":"2024-09-11T03:46:06.365866+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:46:06.365866+0000","last_undegraded":"2024-09-11T03:46:06.365866+0000","last_fullsized":"2024-09-11T03:46:06.365866+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:28.192403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:28.192403+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:56.078747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":2447,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365504+0000","last_change":"2024-09-11T03:46:05.019458+0000","last_active":"2024-09-11T03:46:06.365504+0000","last_peered":"2024-09-11T03:46:06.365504+0000","last_clean":"2024-09-11T03:46:06.365504+0000","last_became_active":"2024-09-11T03:41:33.865499+0000","last_became_peered":"2024-09-11T03:41:33.865499+0000","last_unstale":"2024-09-11T03:46:06.365504+0000","last_undegraded":"2024-09-11T03:46:06.365504+0000","last_fullsized":"2024-09-11T03:46:06.365504+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:45:49.918830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:49.918830+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:55.119040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0048807099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":292,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.963947+0000","last_change":"2024-09-11T03:44:25.511501+0000","last_active":"2024-09-11T03:46:06.963947+0000","last_peered":"2024-09-11T03:46:06.963947+0000","last_clean":"2024-09-11T03:46:06.963947+0000","last_became_active":"2024-09-11T03:43:52.924325+0000","last_became_peered":"2024-09-11T03:43:52.924325+0000","last_unstale":"2024-09-11T03:46:06.963947+0000","last_undegraded":"2024-09-11T03:46:06.963947+0000","last_fullsized":"2024-09-11T03:46:06.963947+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:25.511435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:25.511435+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:32.454502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"547'769","reported_seq":3105,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013436+0000","last_change":"2024-09-11T03:46:05.061158+0000","last_active":"2024-09-11T03:46:06.013436+0000","last_peered":"2024-09-11T03:46:06.013436+0000","last_clean":"2024-09-11T03:46:06.013436+0000","last_became_active":"2024-09-11T03:41:33.866316+0000","last_became_peered":"2024-09-11T03:41:33.866316+0000","last_unstale":"2024-09-11T03:46:06.013436+0000","last_undegraded":"2024-09-11T03:46:06.013436+0000","last_fullsized":"2024-09-11T03:46:06.013436+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"530'746","last_scrub_stamp":"2024-09-11T03:45:50.568426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:50.568426+0000","objects_scrubbed":9,"log_size":769,"log_dups_size":0,"ondisk_log_size":769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:17.296595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024829510000000002,"stat_sum":{"num_bytes":19987543,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":156,"num_read_kb":159287,"num_write":117,"num_write_kb":46550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":292,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.963971+0000","last_change":"2024-09-11T03:44:26.477597+0000","last_active":"2024-09-11T03:46:06.963971+0000","last_peered":"2024-09-11T03:46:06.963971+0000","last_clean":"2024-09-11T03:46:06.963971+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:46:06.963971+0000","last_undegraded":"2024-09-11T03:46:06.963971+0000","last_fullsized":"2024-09-11T03:46:06.963971+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:26.477545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:26.477545+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:57.218670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"541'419","reported_seq":2756,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365984+0000","last_change":"2024-09-11T03:46:05.019454+0000","last_active":"2024-09-11T03:46:06.365984+0000","last_peered":"2024-09-11T03:46:06.365984+0000","last_clean":"2024-09-11T03:46:06.365984+0000","last_became_active":"2024-09-11T03:44:10.921135+0000","last_became_peered":"2024-09-11T03:44:10.921135+0000","last_unstale":"2024-09-11T03:46:06.365984+0000","last_undegraded":"2024-09-11T03:46:06.365984+0000","last_fullsized":"2024-09-11T03:46:06.365984+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'397","last_scrub_stamp":"2024-09-11T03:45:26.963120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:26.963120+0000","objects_scrubbed":7,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:34.917273+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0033009860000000001,"stat_sum":{"num_bytes":6571933,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":135391,"num_write":75,"num_write_kb":21420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":4118225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":297,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.964174+0000","last_change":"2024-09-11T03:45:12.360874+0000","last_active":"2024-09-11T03:46:06.964174+0000","last_peered":"2024-09-11T03:46:06.964174+0000","last_clean":"2024-09-11T03:46:06.964174+0000","last_became_active":"2024-09-11T03:43:52.923384+0000","last_became_peered":"2024-09-11T03:43:52.923384+0000","last_unstale":"2024-09-11T03:46:06.964174+0000","last_undegraded":"2024-09-11T03:46:06.964174+0000","last_fullsized":"2024-09-11T03:46:06.964174+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:12.360829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:12.360829+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:31.665882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,5],"acting":[10,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"531'216","reported_seq":2558,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365800+0000","last_change":"2024-09-11T03:46:05.014589+0000","last_active":"2024-09-11T03:46:06.365800+0000","last_peered":"2024-09-11T03:46:06.365800+0000","last_clean":"2024-09-11T03:46:06.365800+0000","last_became_active":"2024-09-11T03:42:30.902838+0000","last_became_peered":"2024-09-11T03:42:30.902838+0000","last_unstale":"2024-09-11T03:46:06.365800+0000","last_undegraded":"2024-09-11T03:46:06.365800+0000","last_fullsized":"2024-09-11T03:46:06.365800+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:45:53.642117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:53.642117+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:06.475100+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0.00069119299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":27471,"num_write":22,"num_write_kb":11305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,11],"acting":[5,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":297,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.963885+0000","last_change":"2024-09-11T03:45:26.373529+0000","last_active":"2024-09-11T03:46:06.963885+0000","last_peered":"2024-09-11T03:46:06.963885+0000","last_clean":"2024-09-11T03:46:06.963885+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:46:06.963885+0000","last_undegraded":"2024-09-11T03:46:06.963885+0000","last_fullsized":"2024-09-11T03:46:06.963885+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:26.373478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:26.373478+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:27.036677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"550'201","reported_seq":2524,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-09-11T03:46:07.969144+0000","last_change":"2024-09-11T03:46:07.969487+0000","last_active":"2024-09-11T03:46:07.969144+0000","last_peered":"2024-09-11T03:46:07.969144+0000","last_clean":"2024-09-11T03:46:07.969144+0000","last_became_active":"2024-09-11T03:41:33.866217+0000","last_became_peered":"2024-09-11T03:41:33.866217+0000","last_unstale":"2024-09-11T03:46:07.969144+0000","last_undegraded":"2024-09-11T03:46:07.969144+0000","last_fullsized":"2024-09-11T03:46:07.969144+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'176","last_scrub_stamp":"2024-09-11T03:45:41.180057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:41.180057+0000","objects_scrubbed":3,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:48.698814+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018188659999999999,"stat_sum":{"num_bytes":10481089,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":7025,"num_write":31,"num_write_kb":11850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d4","length":"1"},{"start":"dc","length":"2"},{"start":"e2","length":"2"}]},{"pgid":"1.2","version":"20'92","reported_seq":684,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-09-11T03:46:05.019253+0000","last_change":"2024-09-11T03:44:25.934869+0000","last_active":"2024-09-11T03:46:05.019253+0000","last_peered":"2024-09-11T03:46:05.019253+0000","last_clean":"2024-09-11T03:46:05.019253+0000","last_became_active":"2024-09-11T03:44:25.934580+0000","last_became_peered":"2024-09-11T03:44:25.934580+0000","last_unstale":"2024-09-11T03:46:05.019253+0000","last_undegraded":"2024-09-11T03:46:05.019253+0000","last_fullsized":"2024-09-11T03:46:05.019253+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:23.494418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:23.494418+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:37.110438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"547'178","reported_seq":2541,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013805+0000","last_change":"2024-09-11T03:46:05.011485+0000","last_active":"2024-09-11T03:46:06.013805+0000","last_peered":"2024-09-11T03:46:06.013805+0000","last_clean":"2024-09-11T03:46:06.013805+0000","last_became_active":"2024-09-11T03:43:55.920265+0000","last_became_peered":"2024-09-11T03:43:55.920265+0000","last_unstale":"2024-09-11T03:46:06.013805+0000","last_undegraded":"2024-09-11T03:46:06.013805+0000","last_fullsized":"2024-09-11T03:46:06.013805+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"537'168","last_scrub_stamp":"2024-09-11T03:45:55.666981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:55.666981+0000","objects_scrubbed":4,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:07.930280+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0024739089999999998,"stat_sum":{"num_bytes":4631117,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":76,"num_read_kb":79688,"num_write":30,"num_write_kb":9039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4387035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":720,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365486+0000","last_change":"2024-09-11T03:44:10.921140+0000","last_active":"2024-09-11T03:46:06.365486+0000","last_peered":"2024-09-11T03:46:06.365486+0000","last_clean":"2024-09-11T03:46:06.365486+0000","last_became_active":"2024-09-11T03:44:10.920828+0000","last_became_peered":"2024-09-11T03:44:10.920828+0000","last_unstale":"2024-09-11T03:46:06.365486+0000","last_undegraded":"2024-09-11T03:46:06.365486+0000","last_fullsized":"2024-09-11T03:46:06.365486+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:07.508193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:07.508193+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:18.921093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"541'367","reported_seq":2670,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365926+0000","last_change":"2024-09-11T03:46:05.019467+0000","last_active":"2024-09-11T03:46:06.365926+0000","last_peered":"2024-09-11T03:46:06.365926+0000","last_clean":"2024-09-11T03:46:06.365926+0000","last_became_active":"2024-09-11T03:41:33.862271+0000","last_became_peered":"2024-09-11T03:41:33.862271+0000","last_unstale":"2024-09-11T03:46:06.365926+0000","last_undegraded":"2024-09-11T03:46:06.365926+0000","last_fullsized":"2024-09-11T03:46:06.365926+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'327","last_scrub_stamp":"2024-09-11T03:45:12.798880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:12.798880+0000","objects_scrubbed":5,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:15.146868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0053604739999999996,"stat_sum":{"num_bytes":4773452,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":114,"num_read_kb":95408,"num_write":63,"num_write_kb":22522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.0","version":"20'92","reported_seq":1119,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013854+0000","last_change":"2024-09-11T03:45:23.491671+0000","last_active":"2024-09-11T03:46:06.013854+0000","last_peered":"2024-09-11T03:46:06.013854+0000","last_clean":"2024-09-11T03:46:06.013854+0000","last_became_active":"2024-09-11T03:40:25.854624+0000","last_became_peered":"2024-09-11T03:40:25.854624+0000","last_unstale":"2024-09-11T03:46:06.013854+0000","last_undegraded":"2024-09-11T03:46:06.013854+0000","last_fullsized":"2024-09-11T03:46:06.013854+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:23.491587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:23.491587+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:42.891922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"20'92","reported_seq":694,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365759+0000","last_change":"2024-09-11T03:40:35.282811+0000","last_active":"2024-09-11T03:46:06.365759+0000","last_peered":"2024-09-11T03:46:06.365759+0000","last_clean":"2024-09-11T03:46:06.365759+0000","last_became_active":"2024-09-11T03:40:27.844262+0000","last_became_peered":"2024-09-11T03:40:27.844262+0000","last_unstale":"2024-09-11T03:46:06.365759+0000","last_undegraded":"2024-09-11T03:46:06.365759+0000","last_fullsized":"2024-09-11T03:46:06.365759+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.282762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.282762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:00.679637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"547'613","reported_seq":2902,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365945+0000","last_change":"2024-09-11T03:46:05.022495+0000","last_active":"2024-09-11T03:46:06.365945+0000","last_peered":"2024-09-11T03:46:06.365945+0000","last_clean":"2024-09-11T03:46:06.365945+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:46:06.365945+0000","last_undegraded":"2024-09-11T03:46:06.365945+0000","last_fullsized":"2024-09-11T03:46:06.365945+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'551","last_scrub_stamp":"2024-09-11T03:45:10.285456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:10.285456+0000","objects_scrubbed":9,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:35.987887+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0056519079999999998,"stat_sum":{"num_bytes":17519587,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":3,"num_read":271,"num_read_kb":237149,"num_write":221,"num_write_kb":88780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":435,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013701+0000","last_change":"2024-09-11T03:45:53.637950+0000","last_active":"2024-09-11T03:46:06.013701+0000","last_peered":"2024-09-11T03:46:06.013701+0000","last_clean":"2024-09-11T03:46:06.013701+0000","last_became_active":"2024-09-11T03:44:43.940022+0000","last_became_peered":"2024-09-11T03:44:43.940022+0000","last_unstale":"2024-09-11T03:46:06.013701+0000","last_undegraded":"2024-09-11T03:46:06.013701+0000","last_fullsized":"2024-09-11T03:46:06.013701+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:53.637891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:53.637891+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:14.116074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"547'725","reported_seq":2994,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.365964+0000","last_change":"2024-09-11T03:46:05.020197+0000","last_active":"2024-09-11T03:46:06.365964+0000","last_peered":"2024-09-11T03:46:06.365964+0000","last_clean":"2024-09-11T03:46:06.365964+0000","last_became_active":"2024-09-11T03:42:30.902959+0000","last_became_peered":"2024-09-11T03:42:30.902959+0000","last_unstale":"2024-09-11T03:46:06.365964+0000","last_undegraded":"2024-09-11T03:46:06.365964+0000","last_fullsized":"2024-09-11T03:46:06.365964+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"485'666","last_scrub_stamp":"2024-09-11T03:45:07.788521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:07.788521+0000","objects_scrubbed":5,"log_size":725,"log_dups_size":0,"ondisk_log_size":725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:24.818995+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0061154139999999996,"stat_sum":{"num_bytes":8168757,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":308,"num_read_kb":281935,"num_write":221,"num_write_kb":80839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17460823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":720,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013651+0000","last_change":"2024-09-11T03:42:51.733829+0000","last_active":"2024-09-11T03:46:06.013651+0000","last_peered":"2024-09-11T03:46:06.013651+0000","last_clean":"2024-09-11T03:46:06.013651+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:46:06.013651+0000","last_undegraded":"2024-09-11T03:46:06.013651+0000","last_fullsized":"2024-09-11T03:46:06.013651+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:51.733762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:51.733762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"550'811","reported_seq":3089,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-09-11T03:46:07.970994+0000","last_change":"2024-09-11T03:46:07.971158+0000","last_active":"2024-09-11T03:46:07.970994+0000","last_peered":"2024-09-11T03:46:07.970994+0000","last_clean":"2024-09-11T03:46:07.970994+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:46:07.970994+0000","last_undegraded":"2024-09-11T03:46:07.970994+0000","last_fullsized":"2024-09-11T03:46:07.970994+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"532'778","last_scrub_stamp":"2024-09-11T03:45:51.172946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:51.172946+0000","objects_scrubbed":11,"log_size":811,"log_dups_size":0,"ondisk_log_size":811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:20.146991+0000","scrub_duration":5,"objects_trimmed":1,"snaptrim_duration":0.003549394,"stat_sum":{"num_bytes":18303021,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":300,"num_read_kb":279177,"num_write":236,"num_write_kb":93533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d4","length":"1"},{"start":"dc","length":"2"},{"start":"e2","length":"2"}]},{"pgid":"2.f","version":"546'452","reported_seq":2749,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-09-11T03:46:06.013530+0000","last_change":"2024-09-11T03:46:05.061334+0000","last_active":"2024-09-11T03:46:06.013530+0000","last_peered":"2024-09-11T03:46:06.013530+0000","last_clean":"2024-09-11T03:46:06.013530+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:46:06.013530+0000","last_undegraded":"2024-09-11T03:46:06.013530+0000","last_fullsized":"2024-09-11T03:46:06.013530+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'434","last_scrub_stamp":"2024-09-11T03:45:42.488889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:42.488889+0000","objects_scrubbed":3,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:06.542862+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0025501759999999999,"stat_sum":{"num_bytes":7764757,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":174,"num_read_kb":142734,"num_write":152,"num_write_kb":56584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ca","length":"1"},{"start":"db","length":"3"},{"start":"e2","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":130,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-09-11T03:46:05.019255+0000","last_change":"2024-09-11T03:45:10.103917+0000","last_active":"2024-09-11T03:46:05.019255+0000","last_peered":"2024-09-11T03:46:05.019255+0000","last_clean":"2024-09-11T03:46:05.019255+0000","last_became_active":"2024-09-11T03:45:09.973577+0000","last_became_peered":"2024-09-11T03:45:09.973577+0000","last_unstale":"2024-09-11T03:46:05.019255+0000","last_undegraded":"2024-09-11T03:46:05.019255+0000","last_fullsized":"2024-09-11T03:46:05.019255+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:10.103870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:10.103870+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:30.592332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,9],"acting":[0,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":191925177,"num_objects":99,"num_object_clones":57,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":12,"num_read":2879,"num_read_kb":2778551,"num_write":2004,"num_write_kb":768084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":61837646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":357027840,"data_stored":466612899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10421,"ondisk_log_size":10421,"up":78,"acting":78,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":603786,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":36063,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285675,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73368,"kb_used_data":33852,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93671784,"statfs":{"total":95995035648,"available":95919906816,"internally_reserved":0,"allocated":34664448,"data_stored":42116237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514607,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67100,"kb_used_data":31744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93678052,"statfs":{"total":95995035648,"available":95926325248,"internally_reserved":0,"allocated":32505856,"data_stored":39223336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743541,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66856,"kb_used_data":39824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93678296,"statfs":{"total":95995035648,"available":95926575104,"internally_reserved":0,"allocated":40779776,"data_stored":52464894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":59708,"kb_used_data":32672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93685444,"statfs":{"total":95995035648,"available":95933894656,"internally_reserved":0,"allocated":33456128,"data_stored":39822007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299519,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73400,"kb_used_data":42204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93671752,"statfs":{"total":95995035648,"available":95919874048,"internally_reserved":0,"allocated":43216896,"data_stored":57162471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":193273528452,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757386,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58692,"kb_used_data":19176,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93686460,"statfs":{"total":95995035648,"available":95934935040,"internally_reserved":0,"allocated":19636224,"data_stored":22351907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986319,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54916,"kb_used_data":27884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93690236,"statfs":{"total":95995035648,"available":95938801664,"internally_reserved":0,"allocated":28553216,"data_stored":31071099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149844,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":120324,"kb_used_data":71088,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93624828,"statfs":{"total":95995035648,"available":95871823872,"internally_reserved":0,"allocated":72794112,"data_stored":95356688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411482,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79232,"kb_used_data":39716,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93665920,"statfs":{"total":95995035648,"available":95913902080,"internally_reserved":0,"allocated":40669184,"data_stored":49365408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575007,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63204,"kb_used_data":36168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93681948,"statfs":{"total":95995035648,"available":95930314752,"internally_reserved":0,"allocated":37036032,"data_stored":42693752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738532,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83940,"kb_used_data":40264,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93661212,"statfs":{"total":95995035648,"available":95909081088,"internally_reserved":0,"allocated":41230336,"data_stored":42344569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38342656,"data_stored":42083830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30453760,"data_stored":38477609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":33845248,"data_stored":44923053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":63303680,"data_stored":83466296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":23859200,"data_stored":28735975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12836864,"data_stored":17926256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4079616,"data_stored":4066399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36429824,"data_stored":52745080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":26873856,"data_stored":35605864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":33021952,"data_stored":45691898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":25899008,"data_stored":34990545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28082176,"data_stored":37900094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:46:11.578 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-11T03:46:11.578 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:46:11.579 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-09-11T03:46:11.579 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd in 6 2024-09-11T03:46:11.594 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:11.642 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:11.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:11.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:11.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:11.758 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:46:11.758 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:46:11.758 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:46:11.758 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:46:11.770 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:11.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:11.872 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:11.893 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:11.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:11.894 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:11.964 INFO:tasks.rados.rados.2.smithi175.stdout:2067: done (0 left) 2024-09-11T03:46:11.965 INFO:tasks.rados.rados.2.smithi175.stdout:2068: snap_remove snap 228 2024-09-11T03:46:11.995 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:12.020 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:12.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:12.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:12.122 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:12.143 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:12.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:12.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:11 smithi175 ceph-mon[30894]: pgmap v984: 57 pgs: 57 active+clean; 184 MiB data, 784 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2024-09-11T03:46:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:11 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:11 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4269410590' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:46:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:11 smithi175 ceph-mon[30894]: osdmap e553: 12 total, 12 up, 11 in 2024-09-11T03:46:12.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:11 smithi175 ceph-mon[30894]: osdmap e554: 12 total, 12 up, 11 in 2024-09-11T03:46:12.245 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:12.268 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:12.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:12.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:12.369 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:12.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:12.391 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:12.391 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:11 smithi084 ceph-mon[27254]: pgmap v984: 57 pgs: 57 active+clean; 184 MiB data, 784 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2024-09-11T03:46:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:11 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:11 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4269410590' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:46:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:11 smithi084 ceph-mon[27254]: osdmap e553: 12 total, 12 up, 11 in 2024-09-11T03:46:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:11 smithi084 ceph-mon[27254]: osdmap e554: 12 total, 12 up, 11 in 2024-09-11T03:46:12.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:11 smithi121 ceph-mon[30729]: pgmap v984: 57 pgs: 57 active+clean; 184 MiB data, 784 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2024-09-11T03:46:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:11 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:11 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4269410590' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:46:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:11 smithi121 ceph-mon[30729]: osdmap e553: 12 total, 12 up, 11 in 2024-09-11T03:46:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:11 smithi121 ceph-mon[30729]: osdmap e554: 12 total, 12 up, 11 in 2024-09-11T03:46:12.492 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:12.517 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:12.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:12.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:12.620 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:12.642 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:12.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:12.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:12.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:12.770 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:12.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:12.855 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:12.872 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:12.895 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:12.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:12.896 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:12.960 INFO:tasks.rados.rados.2.smithi175.stdout:2068: done (0 left) 2024-09-11T03:46:12.961 INFO:tasks.rados.rados.2.smithi175.stdout:2069: read oid 4 snap 227 2024-09-11T03:46:12.961 INFO:tasks.rados.rados.2.smithi175.stdout:2069: expect (ObjNum 676 snap 218 seq_num 676) 2024-09-11T03:46:12.961 INFO:tasks.rados.rados.2.smithi175.stdout:2070: read oid 29 snap -1 2024-09-11T03:46:12.961 INFO:tasks.rados.rados.2.smithi175.stdout:2070: expect (ObjNum 681 snap 218 seq_num 681) 2024-09-11T03:46:12.961 INFO:tasks.rados.rados.2.smithi175.stdout:2071: read oid 5 snap -1 2024-09-11T03:46:12.961 INFO:tasks.rados.rados.2.smithi175.stdout:2071: expect (ObjNum 701 snap 229 seq_num 701) 2024-09-11T03:46:12.961 INFO:tasks.rados.rados.2.smithi175.stdout:2072: copy_from oid 8 from oid 19 current snap is 229 2024-09-11T03:46:12.961 INFO:tasks.rados.rados.2.smithi175.stdout:2073: delete oid 2 current snap is 229 2024-09-11T03:46:12.965 INFO:tasks.rados.rados.2.smithi175.stdout:2073: done (4 left) 2024-09-11T03:46:12.965 INFO:tasks.rados.rados.2.smithi175.stdout:2074: read oid 39 snap -1 2024-09-11T03:46:12.966 INFO:tasks.rados.rados.2.smithi175.stdout:2074: expect (ObjNum 528 snap 156 seq_num 528) 2024-09-11T03:46:12.966 INFO:tasks.rados.rados.2.smithi175.stdout:2075: snap_remove snap 216 2024-09-11T03:46:12.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:13.022 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:13.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:13.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:13.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:13.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:12 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:46:13.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:12 smithi084 ceph-mon[27254]: 2.b scrub ok 2024-09-11T03:46:13.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:12 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:46:13.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:12 smithi084 ceph-mon[27254]: 2.14 scrub ok 2024-09-11T03:46:13.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:12 smithi084 ceph-mon[27254]: osdmap e555: 12 total, 12 up, 11 in 2024-09-11T03:46:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:13.147 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:13.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:13.248 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:13.270 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:13.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:13.271 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:13.372 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:13.385 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T03:46:13.395 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:13.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:13.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:12 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:46:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:12 smithi121 ceph-mon[30729]: 2.b scrub ok 2024-09-11T03:46:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:12 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:46:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:12 smithi121 ceph-mon[30729]: 2.14 scrub ok 2024-09-11T03:46:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:12 smithi121 ceph-mon[30729]: osdmap e555: 12 total, 12 up, 11 in 2024-09-11T03:46:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:12 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:46:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:12 smithi175 ceph-mon[30894]: 2.b scrub ok 2024-09-11T03:46:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:12 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:46:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:12 smithi175 ceph-mon[30894]: 2.14 scrub ok 2024-09-11T03:46:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:12 smithi175 ceph-mon[30894]: osdmap e555: 12 total, 12 up, 11 in 2024-09-11T03:46:13.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:13.519 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:13.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:13.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:13.621 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:13.645 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:13.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:13.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:13.652 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:13.747 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:13.773 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:13.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:13.774 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:13.874 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:13.897 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:13.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:13.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:13.965 INFO:tasks.rados.rados.2.smithi175.stdout:2075: done (5 left) 2024-09-11T03:46:13.965 INFO:tasks.rados.rados.2.smithi175.stdout:2076: snap_remove snap 207 2024-09-11T03:46:13.998 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:14.022 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:14.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:14.023 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:14.122 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:46:14.123 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:46:14.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:14.145 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:46:14.145 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:46:14.147 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:14.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:14.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:14.162 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:46:14.162 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:46:14.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:13 smithi084 ceph-mon[27254]: pgmap v987: 57 pgs: 57 active+clean; 179 MiB data, 787 MiB used, 983 GiB / 983 GiB avail; 967 KiB/s rd, 435 KiB/s wr, 3 op/s 2024-09-11T03:46:14.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:13 smithi084 ceph-mon[27254]: 1.17 scrub starts 2024-09-11T03:46:14.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:13 smithi084 ceph-mon[27254]: 1.17 scrub ok 2024-09-11T03:46:14.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:13 smithi084 ceph-mon[27254]: osdmap e556: 12 total, 12 up, 11 in 2024-09-11T03:46:14.178 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:46:14.178 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:46:14.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:46:14.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:46:14.211 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:46:14.212 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:46:14.228 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:46:14.229 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:46:14.245 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:46:14.246 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:46:14.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:14.263 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:46:14.263 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:46:14.275 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:14.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:14.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:14.280 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:46:14.281 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:46:14.298 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:46:14.298 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:46:14.314 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:46:14.315 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:46:14.376 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:14.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:14.399 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:14.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:14.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:13 smithi121 ceph-mon[30729]: pgmap v987: 57 pgs: 57 active+clean; 179 MiB data, 787 MiB used, 983 GiB / 983 GiB avail; 967 KiB/s rd, 435 KiB/s wr, 3 op/s 2024-09-11T03:46:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:13 smithi121 ceph-mon[30729]: 1.17 scrub starts 2024-09-11T03:46:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:13 smithi121 ceph-mon[30729]: 1.17 scrub ok 2024-09-11T03:46:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:13 smithi121 ceph-mon[30729]: osdmap e556: 12 total, 12 up, 11 in 2024-09-11T03:46:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:13 smithi175 ceph-mon[30894]: pgmap v987: 57 pgs: 57 active+clean; 179 MiB data, 787 MiB used, 983 GiB / 983 GiB avail; 967 KiB/s rd, 435 KiB/s wr, 3 op/s 2024-09-11T03:46:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:13 smithi175 ceph-mon[30894]: 1.17 scrub starts 2024-09-11T03:46:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:13 smithi175 ceph-mon[30894]: 1.17 scrub ok 2024-09-11T03:46:14.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:13 smithi175 ceph-mon[30894]: osdmap e556: 12 total, 12 up, 11 in 2024-09-11T03:46:14.500 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:14.519 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:14.526 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:14.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:14.529 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:14.630 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:14.652 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:14.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:14.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:14.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:14.780 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:14.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:14.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:14.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:46:14.842 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:14 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a[27250]: 2024-09-11T03:46:14.541+0000 7fae0fb7d640 -1 mon.a@0(leader).osd e556 definitely_dead 0 2024-09-11T03:46:14.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:14.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:14.905 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:14.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:14.960 INFO:teuthology.orchestra.run.smithi084.stderr:marked in osd.6. 2024-09-11T03:46:14.966 INFO:tasks.rados.rados.2.smithi175.stdout:2076: done (5 left) 2024-09-11T03:46:14.966 INFO:tasks.rados.rados.2.smithi175.stdout:2077: delete oid 48 current snap is 229 2024-09-11T03:46:14.966 INFO:tasks.rados.rados.2.smithi175.stdout:2070: expect (ObjNum 681 snap 218 seq_num 681) 2024-09-11T03:46:15.006 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:15.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:15.028 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:15.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:15.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:14 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3538305056' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:15.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:14 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4272661627' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-11T03:46:15.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:14 smithi084 ceph-mon[27254]: 2.13 scrub starts 2024-09-11T03:46:15.101 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:14 smithi084 ceph-mon[27254]: 2.13 scrub ok 2024-09-11T03:46:15.101 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:14 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4272661627' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-11T03:46:15.101 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:14 smithi084 ceph-mon[27254]: osdmap e557: 12 total, 12 up, 12 in 2024-09-11T03:46:15.129 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:15.148 INFO:tasks.rados.rados.2.smithi175.stdout:2069: expect (ObjNum 676 snap 218 seq_num 676) 2024-09-11T03:46:15.150 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:15.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:15.151 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:15.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:15.257 INFO:tasks.rados.rados.2.smithi175.stdout:2074: expect (ObjNum 528 snap 156 seq_num 528) 2024-09-11T03:46:15.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:15.275 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:15.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:15.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:15.402 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:15.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:15.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:14 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3538305056' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:14 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4272661627' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-11T03:46:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:14 smithi121 ceph-mon[30729]: 2.13 scrub starts 2024-09-11T03:46:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:14 smithi121 ceph-mon[30729]: 2.13 scrub ok 2024-09-11T03:46:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:14 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4272661627' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-11T03:46:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:14 smithi121 ceph-mon[30729]: osdmap e557: 12 total, 12 up, 12 in 2024-09-11T03:46:15.437 INFO:tasks.rados.rados.2.smithi175.stdout:2071: expect (ObjNum 701 snap 229 seq_num 701) 2024-09-11T03:46:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:14 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3538305056' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:15.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:14 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4272661627' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-11T03:46:15.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:14 smithi175 ceph-mon[30894]: 2.13 scrub starts 2024-09-11T03:46:15.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:14 smithi175 ceph-mon[30894]: 2.13 scrub ok 2024-09-11T03:46:15.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:14 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4272661627' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-11T03:46:15.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:14 smithi175 ceph-mon[30894]: osdmap e557: 12 total, 12 up, 12 in 2024-09-11T03:46:15.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:15.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:15.528 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:15.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:15.629 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:15.640 INFO:tasks.rados.rados.2.smithi175.stdout:2072: finishing copy_from racing read to smithi17557870-8 2024-09-11T03:46:15.640 INFO:tasks.rados.rados.2.smithi175.stdout:2072: finishing copy_from to smithi17557870-8 2024-09-11T03:46:15.640 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 512 (ObjNum 565 snap 172 seq_num 565) dirty exists 2024-09-11T03:46:15.640 INFO:tasks.rados.rados.2.smithi175.stdout:2069: done (5 left) 2024-09-11T03:46:15.640 INFO:tasks.rados.rados.2.smithi175.stdout:2070: done (4 left) 2024-09-11T03:46:15.640 INFO:tasks.rados.rados.2.smithi175.stdout:2071: done (3 left) 2024-09-11T03:46:15.641 INFO:tasks.rados.rados.2.smithi175.stdout:2072: done (2 left) 2024-09-11T03:46:15.641 INFO:tasks.rados.rados.2.smithi175.stdout:2074: done (1 left) 2024-09-11T03:46:15.641 INFO:tasks.rados.rados.2.smithi175.stdout:2077: done (0 left) 2024-09-11T03:46:15.641 INFO:tasks.rados.rados.2.smithi175.stdout:2078: write (excl) oid 48 current snap is 229 2024-09-11T03:46:15.641 INFO:tasks.rados.rados.2.smithi175.stdout:2078: seq_num 704 ranges {632128=479772,1746735=600898,3066728=653850} 2024-09-11T03:46:15.645 INFO:tasks.rados.rados.2.smithi175.stdout:2078: writing smithi17557870-48 from 632128 to 1111900 tid 1 2024-09-11T03:46:15.650 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:15.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:15.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:15.652 INFO:tasks.rados.rados.2.smithi175.stdout:2078: writing smithi17557870-48 from 1746735 to 2347633 tid 2 2024-09-11T03:46:15.660 INFO:tasks.rados.rados.2.smithi175.stdout:2078: writing smithi17557870-48 from 3066728 to 3720578 tid 3 2024-09-11T03:46:15.660 INFO:tasks.rados.rados.2.smithi175.stdout:2079: delete oid 44 current snap is 229 2024-09-11T03:46:15.660 INFO:tasks.rados.rados.2.smithi175.stdout:2078: finishing write tid 1 to smithi17557870-48 2024-09-11T03:46:15.660 INFO:tasks.rados.rados.2.smithi175.stdout:2078: finishing write tid 2 to smithi17557870-48 2024-09-11T03:46:15.662 INFO:tasks.rados.rados.2.smithi175.stdout:2079: done (1 left) 2024-09-11T03:46:15.662 INFO:tasks.rados.rados.2.smithi175.stdout:2080: snap_create 2024-09-11T03:46:15.663 INFO:tasks.rados.rados.2.smithi175.stdout:2078: finishing write tid 3 to smithi17557870-48 2024-09-11T03:46:15.663 INFO:tasks.rados.rados.2.smithi175.stdout:2078: finishing write tid 4 to smithi17557870-48 2024-09-11T03:46:15.663 INFO:tasks.rados.rados.2.smithi175.stdout:2078: finishing write tid 5 to smithi17557870-48 2024-09-11T03:46:15.663 INFO:tasks.rados.rados.2.smithi175.stdout:2078: oid 48 updating version 0 to 747 2024-09-11T03:46:15.663 INFO:tasks.rados.rados.2.smithi175.stdout:2078: oid 48 updating version 747 to 749 2024-09-11T03:46:15.663 INFO:tasks.rados.rados.2.smithi175.stdout:2078: oid 48 updating version 749 to 750 2024-09-11T03:46:15.663 INFO:tasks.rados.rados.2.smithi175.stdout:2078: oid 48 version 750 is already newer than 748 2024-09-11T03:46:15.663 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 750 (ObjNum 704 snap 229 seq_num 704) dirty exists 2024-09-11T03:46:15.663 INFO:tasks.rados.rados.2.smithi175.stdout:2078: left oid 48 (ObjNum 704 snap 229 seq_num 704) 2024-09-11T03:46:15.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:15.779 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:15.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:15.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:15.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:15.906 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:15.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:15.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:15.967 INFO:tasks.rados.rados.2.smithi175.stdout:2078: done (1 left) 2024-09-11T03:46:15.967 INFO:tasks.rados.rados.2.smithi175.stdout:2080: done (0 left) 2024-09-11T03:46:15.968 INFO:tasks.rados.rados.2.smithi175.stdout:2081: rollback oid 13 current snap is 230 2024-09-11T03:46:15.968 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 13 to 229 2024-09-11T03:46:15.968 INFO:tasks.rados.rados.2.smithi175.stdout:2082: copy_from oid 39 from oid 38 current snap is 230 2024-09-11T03:46:15.968 INFO:tasks.rados.rados.2.smithi175.stdout:2083: write oid 6 current snap is 230 2024-09-11T03:46:15.968 INFO:tasks.rados.rados.2.smithi175.stdout:2083: seq_num 706 ranges {665742=630270,1863699=604567,3121574=676394} 2024-09-11T03:46:15.975 INFO:tasks.rados.rados.2.smithi175.stdout:2083: writing smithi17557870-6 from 665742 to 1296012 tid 1 2024-09-11T03:46:15.981 INFO:tasks.rados.rados.2.smithi175.stdout:2083: writing smithi17557870-6 from 1863699 to 2468266 tid 2 2024-09-11T03:46:15.989 INFO:tasks.rados.rados.2.smithi175.stdout:2083: writing smithi17557870-6 from 3121574 to 3797968 tid 3 2024-09-11T03:46:15.989 INFO:tasks.rados.rados.2.smithi175.stdout:2084: rollback oid 5 current snap is 230 2024-09-11T03:46:15.989 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 5 to 222 2024-09-11T03:46:15.989 INFO:tasks.rados.rados.2.smithi175.stdout:2081: finishing rollback tid 1 to smithi17557870-13 2024-09-11T03:46:15.989 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 0 (ObjNum 1969 snap 0 seq_num 2931794352) dirty dne 2024-09-11T03:46:15.989 INFO:tasks.rados.rados.2.smithi175.stdout:2082: finishing copy_from to smithi17557870-39 2024-09-11T03:46:15.989 INFO:tasks.rados.rados.2.smithi175.stdout:2082: got expected ENOENT (src dne) 2024-09-11T03:46:15.989 INFO:tasks.rados.rados.2.smithi175.stdout:2082: finishing copy_from racing read to smithi17557870-39 2024-09-11T03:46:15.989 INFO:tasks.rados.rados.2.smithi175.stdout:2081: done (3 left) 2024-09-11T03:46:15.990 INFO:tasks.rados.rados.2.smithi175.stdout:2082: done (2 left) 2024-09-11T03:46:15.990 INFO:tasks.rados.rados.2.smithi175.stdout:2085: snap_remove snap 229 2024-09-11T03:46:16.008 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:16.029 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:16.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:16.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:16.110 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-09-11T03:46:16.131 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:16.154 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:16.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:16.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:16.256 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:16.282 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:16.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:16.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:16.284 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:15 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:46:16.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:15 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:46:16.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:15 smithi084 ceph-mon[27254]: pgmap v990: 57 pgs: 57 active+clean; 184 MiB data, 793 MiB used, 983 GiB / 983 GiB avail; 6.4 MiB/s rd, 1.7 MiB/s wr, 7 op/s 2024-09-11T03:46:16.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:15 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:46:16.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:15 smithi084 ceph-mon[27254]: 2.d scrub ok 2024-09-11T03:46:16.285 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:15 smithi084 ceph-mon[27254]: osdmap e558: 12 total, 12 up, 12 in 2024-09-11T03:46:16.293 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:46:16.293 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:46:16.384 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:16.410 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:16.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:16.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:15 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:46:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:15 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:46:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:15 smithi121 ceph-mon[30729]: pgmap v990: 57 pgs: 57 active+clean; 184 MiB data, 793 MiB used, 983 GiB / 983 GiB avail; 6.4 MiB/s rd, 1.7 MiB/s wr, 7 op/s 2024-09-11T03:46:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:15 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:46:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:15 smithi121 ceph-mon[30729]: 2.d scrub ok 2024-09-11T03:46:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:15 smithi121 ceph-mon[30729]: osdmap e558: 12 total, 12 up, 12 in 2024-09-11T03:46:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:15 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:46:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:15 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:46:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:15 smithi175 ceph-mon[30894]: pgmap v990: 57 pgs: 57 active+clean; 184 MiB data, 793 MiB used, 983 GiB / 983 GiB avail; 6.4 MiB/s rd, 1.7 MiB/s wr, 7 op/s 2024-09-11T03:46:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:15 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:46:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:15 smithi175 ceph-mon[30894]: 2.d scrub ok 2024-09-11T03:46:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:15 smithi175 ceph-mon[30894]: osdmap e558: 12 total, 12 up, 12 in 2024-09-11T03:46:16.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:16.538 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:16.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:16.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:16.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:16.666 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:16.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:16.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:16.768 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:16.790 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:16.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:16.891 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:16.913 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:16.914 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:16.963 INFO:tasks.rados.rados.2.smithi175.stdout:2085: done (2 left) 2024-09-11T03:46:16.963 INFO:tasks.rados.rados.2.smithi175.stdout:2086: delete oid 31 current snap is 230 2024-09-11T03:46:16.963 INFO:tasks.rados.rados.2.smithi175.stdout:2084: finishing rollback tid 0 to smithi17557870-5 2024-09-11T03:46:16.963 INFO:tasks.rados.rados.2.smithi175.stdout:2084: finishing rollback tid 1 to smithi17557870-5 2024-09-11T03:46:16.963 INFO:tasks.rados.rados.2.smithi175.stdout:2084: finishing rollback tid 2 to smithi17557870-5 2024-09-11T03:46:16.963 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 700 (ObjNum 615 snap 199 seq_num 615) dirty exists 2024-09-11T03:46:17.001 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:17.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T03:46:17.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:17.039 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:17.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:17.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:17.103 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:46:17.141 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:17.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:17.164 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:17.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:17.253 INFO:tasks.rados.rados.2.smithi175.stdout:2083: finishing write tid 1 to smithi17557870-6 2024-09-11T03:46:17.253 INFO:tasks.rados.rados.2.smithi175.stdout:2083: finishing write tid 2 to smithi17557870-6 2024-09-11T03:46:17.261 INFO:tasks.rados.rados.2.smithi175.stdout:2083: finishing write tid 3 to smithi17557870-6 2024-09-11T03:46:17.261 INFO:tasks.rados.rados.2.smithi175.stdout:2083: finishing write tid 4 to smithi17557870-6 2024-09-11T03:46:17.261 INFO:tasks.rados.rados.2.smithi175.stdout:2083: finishing write tid 5 to smithi17557870-6 2024-09-11T03:46:17.261 INFO:tasks.rados.rados.2.smithi175.stdout:2083: oid 6 updating version 0 to 454 2024-09-11T03:46:17.261 INFO:tasks.rados.rados.2.smithi175.stdout:2083: oid 6 version 454 is already newer than 452 2024-09-11T03:46:17.261 INFO:tasks.rados.rados.2.smithi175.stdout:2083: oid 6 version 454 is already newer than 453 2024-09-11T03:46:17.262 INFO:tasks.rados.rados.2.smithi175.stdout:2083: oid 6 version 454 is already newer than 451 2024-09-11T03:46:17.262 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 454 (ObjNum 706 snap 230 seq_num 706) dirty exists 2024-09-11T03:46:17.262 INFO:tasks.rados.rados.2.smithi175.stdout:2083: left oid 6 (ObjNum 706 snap 230 seq_num 706) 2024-09-11T03:46:17.262 INFO:tasks.rados.rados.2.smithi175.stdout:2083: done (2 left) 2024-09-11T03:46:17.262 INFO:tasks.rados.rados.2.smithi175.stdout:2084: done (1 left) 2024-09-11T03:46:17.262 INFO:tasks.rados.rados.2.smithi175.stdout:2086: done (0 left) 2024-09-11T03:46:17.262 INFO:tasks.rados.rados.2.smithi175.stdout:2087: read oid 49 snap -1 2024-09-11T03:46:17.262 INFO:tasks.rados.rados.2.smithi175.stdout:2087: expect deleted 2024-09-11T03:46:17.262 INFO:tasks.rados.rados.2.smithi175.stdout:2088: read oid 47 snap -1 2024-09-11T03:46:17.262 INFO:tasks.rados.rados.2.smithi175.stdout:2088: expect deleted 2024-09-11T03:46:17.262 INFO:tasks.rados.rados.2.smithi175.stdout:2089: snap_create 2024-09-11T03:46:17.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:16 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:46:17.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:16 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:46:17.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:16 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:46:17.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:16 smithi084 ceph-mon[27254]: osdmap e559: 12 total, 12 up, 12 in 2024-09-11T03:46:17.264 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:17.265 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:16 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:46:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:16 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:46:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:16 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:46:17.266 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:16 smithi121 ceph-mon[30729]: osdmap e559: 12 total, 12 up, 12 in 2024-09-11T03:46:17.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:17.312 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:17.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:17.414 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:17.436 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:17.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:17.437 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:16 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:46:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:16 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:46:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:16 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:46:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:16 smithi175 ceph-mon[30894]: osdmap e559: 12 total, 12 up, 12 in 2024-09-11T03:46:17.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:17.546 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:17.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:17.561 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:17.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:17.662 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:17.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:17.684 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:17.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:17.786 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:17.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:17.809 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:17.809 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:17.910 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:17.932 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:17.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:17.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:17.996 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:46:17.999 INFO:tasks.rados.rados.2.smithi175.stdout:2087: done (2 left) 2024-09-11T03:46:17.999 INFO:tasks.rados.rados.2.smithi175.stdout:2088: done (1 left) 2024-09-11T03:46:18.000 INFO:tasks.rados.rados.2.smithi175.stdout:2089: done (0 left) 2024-09-11T03:46:18.000 INFO:tasks.rados.rados.2.smithi175.stdout:2090: snap_create 2024-09-11T03:46:18.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:18.043 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:46:18.043 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:46:18.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:18.056 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:18.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:18.158 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:18.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:18.181 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:18.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:18.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:18 smithi084 ceph-mon[27254]: 1.f scrub starts 2024-09-11T03:46:18.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:18 smithi084 ceph-mon[27254]: 1.f scrub ok 2024-09-11T03:46:18.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:18 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:46:18.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:18 smithi084 ceph-mon[27254]: 2.8 scrub ok 2024-09-11T03:46:18.231 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:18 smithi084 ceph-mon[27254]: pgmap v993: 57 pgs: 5 remapped+peering, 52 active+clean; 163 MiB data, 754 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 1.7 MiB/s wr, 12 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-11T03:46:18.232 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:18 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1227538108' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:18.232 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:46:18.232 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:18 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:46:18.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:18.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:18.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:18.409 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:18.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:18 smithi121 ceph-mon[30729]: 1.f scrub starts 2024-09-11T03:46:18.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:18 smithi121 ceph-mon[30729]: 1.f scrub ok 2024-09-11T03:46:18.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:18 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:46:18.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:18 smithi121 ceph-mon[30729]: 2.8 scrub ok 2024-09-11T03:46:18.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:18 smithi121 ceph-mon[30729]: pgmap v993: 57 pgs: 5 remapped+peering, 52 active+clean; 163 MiB data, 754 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 1.7 MiB/s wr, 12 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-11T03:46:18.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:18 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1227538108' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:18.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:46:18.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:18 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:46:18.431 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:18.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:18.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:18.434 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:18 smithi175 ceph-mon[30894]: 1.f scrub starts 2024-09-11T03:46:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:18 smithi175 ceph-mon[30894]: 1.f scrub ok 2024-09-11T03:46:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:18 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:46:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:18 smithi175 ceph-mon[30894]: 2.8 scrub ok 2024-09-11T03:46:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:18 smithi175 ceph-mon[30894]: pgmap v993: 57 pgs: 5 remapped+peering, 52 active+clean; 163 MiB data, 754 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 1.7 MiB/s wr, 12 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-11T03:46:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:18 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1227538108' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:46:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:18 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:46:18.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:18.555 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:18.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:18.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:18.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:18.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:18.679 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:18.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:18.780 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:18.802 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:18.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:18.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:18.904 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:18.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:18.927 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:18.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:19.001 INFO:tasks.rados.rados.2.smithi175.stdout:2090: done (0 left) 2024-09-11T03:46:19.002 INFO:tasks.rados.rados.2.smithi175.stdout:2091: write oid 21 current snap is 232 2024-09-11T03:46:19.002 INFO:tasks.rados.rados.2.smithi175.stdout:2091: seq_num 708 ranges {699356=497121,1980663=608236,3176420=415291} 2024-09-11T03:46:19.008 INFO:tasks.rados.rados.2.smithi175.stdout:2091: writing smithi17557870-21 from 699356 to 1196477 tid 1 2024-09-11T03:46:19.016 INFO:tasks.rados.rados.2.smithi175.stdout:2091: writing smithi17557870-21 from 1980663 to 2588899 tid 2 2024-09-11T03:46:19.022 INFO:tasks.rados.rados.2.smithi175.stdout:2091: writing smithi17557870-21 from 3176420 to 3591711 tid 3 2024-09-11T03:46:19.022 INFO:tasks.rados.rados.2.smithi175.stdout:2092: write (excl) oid 7 current snap is 232 2024-09-11T03:46:19.022 INFO:tasks.rados.rados.2.smithi175.stdout:2091: finishing write tid 1 to smithi17557870-21 2024-09-11T03:46:19.022 INFO:tasks.rados.rados.2.smithi175.stdout:2091: finishing write tid 2 to smithi17557870-21 2024-09-11T03:46:19.022 INFO:tasks.rados.rados.2.smithi175.stdout:2092: seq_num 709 ranges {716163=572370,2039145=468247,3203843=626563} 2024-09-11T03:46:19.029 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:19.030 INFO:tasks.rados.rados.2.smithi175.stdout:2092: writing smithi17557870-7 from 716163 to 1288533 tid 1 2024-09-11T03:46:19.035 INFO:tasks.rados.rados.2.smithi175.stdout:2092: writing smithi17557870-7 from 2039145 to 2507392 tid 2 2024-09-11T03:46:19.050 INFO:tasks.rados.rados.2.smithi175.stdout:2092: writing smithi17557870-7 from 3203843 to 3830406 tid 3 2024-09-11T03:46:19.051 INFO:tasks.rados.rados.2.smithi175.stdout:2093: rollback oid 11 current snap is 232 2024-09-11T03:46:19.051 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 11 to 222 2024-09-11T03:46:19.051 INFO:tasks.rados.rados.2.smithi175.stdout:2091: finishing write tid 3 to smithi17557870-21 2024-09-11T03:46:19.051 INFO:tasks.rados.rados.2.smithi175.stdout:2091: finishing write tid 4 to smithi17557870-21 2024-09-11T03:46:19.051 INFO:tasks.rados.rados.2.smithi175.stdout:2091: finishing write tid 5 to smithi17557870-21 2024-09-11T03:46:19.051 INFO:tasks.rados.rados.2.smithi175.stdout:2091: oid 21 updating version 0 to 852 2024-09-11T03:46:19.051 INFO:tasks.rados.rados.2.smithi175.stdout:2091: oid 21 updating version 852 to 853 2024-09-11T03:46:19.051 INFO:tasks.rados.rados.2.smithi175.stdout:2091: oid 21 updating version 853 to 854 2024-09-11T03:46:19.051 INFO:tasks.rados.rados.2.smithi175.stdout:2091: oid 21 version 854 is already newer than 849 2024-09-11T03:46:19.051 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 854 (ObjNum 708 snap 232 seq_num 708) dirty exists 2024-09-11T03:46:19.051 INFO:tasks.rados.rados.2.smithi175.stdout:2091: left oid 21 (ObjNum 708 snap 232 seq_num 708) 2024-09-11T03:46:19.052 INFO:tasks.rados.rados.2.smithi175.stdout:2092: finishing write tid 1 to smithi17557870-7 2024-09-11T03:46:19.052 INFO:tasks.rados.rados.2.smithi175.stdout:2091: done (2 left) 2024-09-11T03:46:19.052 INFO:tasks.rados.rados.2.smithi175.stdout:2094: rollback oid 33 current snap is 232 2024-09-11T03:46:19.052 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 33 to 231 2024-09-11T03:46:19.052 INFO:tasks.rados.rados.2.smithi175.stdout:2092: finishing write tid 2 to smithi17557870-7 2024-09-11T03:46:19.052 INFO:tasks.rados.rados.2.smithi175.stdout:2095: read oid 21 snap -1 2024-09-11T03:46:19.052 INFO:tasks.rados.rados.2.smithi175.stdout:2095: expect (ObjNum 708 snap 232 seq_num 708) 2024-09-11T03:46:19.052 INFO:tasks.rados.rados.2.smithi175.stdout:2096: delete oid 9 current snap is 232 2024-09-11T03:46:19.052 INFO:tasks.rados.rados.2.smithi175.stdout:2094: finishing rollback tid 0 to smithi17557870-33 2024-09-11T03:46:19.052 INFO:tasks.rados.rados.2.smithi175.stdout:2092: finishing write tid 3 to smithi17557870-7 2024-09-11T03:46:19.052 INFO:tasks.rados.rados.2.smithi175.stdout:2094: finishing rollback tid 1 to smithi17557870-33 2024-09-11T03:46:19.052 INFO:tasks.rados.rados.2.smithi175.stdout:2093: finishing rollback tid 0 to smithi17557870-11 2024-09-11T03:46:19.053 INFO:tasks.rados.rados.2.smithi175.stdout:2092: finishing write tid 4 to smithi17557870-7 2024-09-11T03:46:19.053 INFO:tasks.rados.rados.2.smithi175.stdout:2093: finishing rollback tid 1 to smithi17557870-11 2024-09-11T03:46:19.053 INFO:tasks.rados.rados.2.smithi175.stdout:2094: finishing rollback tid 2 to smithi17557870-33 2024-09-11T03:46:19.053 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 853 (ObjNum 630 snap 202 seq_num 630) dirty exists 2024-09-11T03:46:19.053 INFO:tasks.rados.rados.2.smithi175.stdout:2092: finishing write tid 5 to smithi17557870-7 2024-09-11T03:46:19.053 INFO:tasks.rados.rados.2.smithi175.stdout:2092: oid 7 updating version 0 to 523 2024-09-11T03:46:19.053 INFO:tasks.rados.rados.2.smithi175.stdout:2092: oid 7 version 523 is already newer than 520 2024-09-11T03:46:19.053 INFO:tasks.rados.rados.2.smithi175.stdout:2092: oid 7 version 523 is already newer than 521 2024-09-11T03:46:19.053 INFO:tasks.rados.rados.2.smithi175.stdout:2092: oid 7 version 523 is already newer than 522 2024-09-11T03:46:19.053 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 523 (ObjNum 709 snap 232 seq_num 709) dirty exists 2024-09-11T03:46:19.053 INFO:tasks.rados.rados.2.smithi175.stdout:2092: left oid 7 (ObjNum 709 snap 232 seq_num 709) 2024-09-11T03:46:19.054 INFO:tasks.rados.rados.2.smithi175.stdout:2093: finishing rollback tid 2 to smithi17557870-11 2024-09-11T03:46:19.054 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 583 (ObjNum 638 snap 202 seq_num 638) dirty exists 2024-09-11T03:46:19.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:19.055 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:19.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:19.061 INFO:tasks.rados.rados.2.smithi175.stdout:2095: expect (ObjNum 708 snap 232 seq_num 708) 2024-09-11T03:46:19.156 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:19.178 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:19.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:19.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:19.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:19 smithi084 ceph-mon[27254]: 1.5 scrub starts 2024-09-11T03:46:19.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:19 smithi084 ceph-mon[27254]: 1.5 scrub ok 2024-09-11T03:46:19.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:19 smithi084 ceph-mon[27254]: 1.18 scrub starts 2024-09-11T03:46:19.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:19 smithi084 ceph-mon[27254]: 1.18 scrub ok 2024-09-11T03:46:19.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:19 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:46:19.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:19 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:46:19.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:19 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1227538108' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:19.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:19 smithi084 ceph-mon[27254]: osdmap e560: 12 total, 12 up, 12 in 2024-09-11T03:46:19.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:46:19.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:46:19.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:46:19.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:19 smithi084 ceph-mon[27254]: osdmap e561: 12 total, 12 up, 12 in 2024-09-11T03:46:19.279 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:19.297 INFO:tasks.rados.rados.2.smithi175.stdout:2092: done (4 left) 2024-09-11T03:46:19.297 INFO:tasks.rados.rados.2.smithi175.stdout:2093: done (3 left) 2024-09-11T03:46:19.297 INFO:tasks.rados.rados.2.smithi175.stdout:2094: done (2 left) 2024-09-11T03:46:19.297 INFO:tasks.rados.rados.2.smithi175.stdout:2095: done (1 left) 2024-09-11T03:46:19.297 INFO:tasks.rados.rados.2.smithi175.stdout:2096: done (0 left) 2024-09-11T03:46:19.297 INFO:tasks.rados.rados.2.smithi175.stdout:2097: read oid 10 snap 222 2024-09-11T03:46:19.297 INFO:tasks.rados.rados.2.smithi175.stdout:2097: expect deleted 2024-09-11T03:46:19.297 INFO:tasks.rados.rados.2.smithi175.stdout:2098: write oid 17 current snap is 232 2024-09-11T03:46:19.297 INFO:tasks.rados.rados.2.smithi175.stdout:2098: seq_num 711 ranges {749777=722868,2156109=471916,3258689=649107} 2024-09-11T03:46:19.300 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:19.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:19.305 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:19.305 INFO:tasks.rados.rados.2.smithi175.stdout:2098: writing smithi17557870-17 from 749777 to 1472645 tid 1 2024-09-11T03:46:19.311 INFO:tasks.rados.rados.2.smithi175.stdout:2098: writing smithi17557870-17 from 2156109 to 2628025 tid 2 2024-09-11T03:46:19.317 INFO:tasks.rados.rados.2.smithi175.stdout:2098: writing smithi17557870-17 from 3258689 to 3907796 tid 3 2024-09-11T03:46:19.317 INFO:tasks.rados.rados.2.smithi175.stdout:2099: snap_create 2024-09-11T03:46:19.317 INFO:tasks.rados.rados.2.smithi175.stdout:2098: finishing write tid 1 to smithi17557870-17 2024-09-11T03:46:19.318 INFO:tasks.rados.rados.2.smithi175.stdout:2098: finishing write tid 2 to smithi17557870-17 2024-09-11T03:46:19.320 INFO:tasks.rados.rados.2.smithi175.stdout:2098: finishing write tid 3 to smithi17557870-17 2024-09-11T03:46:19.320 INFO:tasks.rados.rados.2.smithi175.stdout:2098: finishing write tid 4 to smithi17557870-17 2024-09-11T03:46:19.321 INFO:tasks.rados.rados.2.smithi175.stdout:2098: finishing write tid 5 to smithi17557870-17 2024-09-11T03:46:19.321 INFO:tasks.rados.rados.2.smithi175.stdout:2098: oid 17 updating version 0 to 744 2024-09-11T03:46:19.321 INFO:tasks.rados.rados.2.smithi175.stdout:2098: oid 17 version 744 is already newer than 741 2024-09-11T03:46:19.321 INFO:tasks.rados.rados.2.smithi175.stdout:2098: oid 17 version 744 is already newer than 742 2024-09-11T03:46:19.321 INFO:tasks.rados.rados.2.smithi175.stdout:2098: oid 17 version 744 is already newer than 743 2024-09-11T03:46:19.321 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 744 (ObjNum 711 snap 232 seq_num 711) dirty exists 2024-09-11T03:46:19.321 INFO:tasks.rados.rados.2.smithi175.stdout:2098: left oid 17 (ObjNum 711 snap 232 seq_num 711) 2024-09-11T03:46:19.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:19 smithi121 ceph-mon[30729]: 1.5 scrub starts 2024-09-11T03:46:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:19 smithi121 ceph-mon[30729]: 1.5 scrub ok 2024-09-11T03:46:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:19 smithi121 ceph-mon[30729]: 1.18 scrub starts 2024-09-11T03:46:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:19 smithi121 ceph-mon[30729]: 1.18 scrub ok 2024-09-11T03:46:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:19 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:46:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:19 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:46:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:19 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1227538108' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:19 smithi121 ceph-mon[30729]: osdmap e560: 12 total, 12 up, 12 in 2024-09-11T03:46:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:46:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:46:19.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:46:19.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:19 smithi121 ceph-mon[30729]: osdmap e561: 12 total, 12 up, 12 in 2024-09-11T03:46:19.428 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:19.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:19.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:19.443 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:46:19.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:46:19.460 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:46:19.460 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:46:19.476 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:46:19.476 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:46:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:19 smithi175 ceph-mon[30894]: 1.5 scrub starts 2024-09-11T03:46:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:19 smithi175 ceph-mon[30894]: 1.5 scrub ok 2024-09-11T03:46:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:19 smithi175 ceph-mon[30894]: 1.18 scrub starts 2024-09-11T03:46:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:19 smithi175 ceph-mon[30894]: 1.18 scrub ok 2024-09-11T03:46:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:19 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:46:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:19 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:46:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:19 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1227538108' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:19 smithi175 ceph-mon[30894]: osdmap e560: 12 total, 12 up, 12 in 2024-09-11T03:46:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:46:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:46:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:46:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:19 smithi175 ceph-mon[30894]: osdmap e561: 12 total, 12 up, 12 in 2024-09-11T03:46:19.491 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:46:19.492 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:46:19.509 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:46:19.510 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:46:19.527 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:46:19.527 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:46:19.531 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:19.544 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:46:19.544 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:46:19.552 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:19.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:19.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:19.561 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:46:19.561 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:46:19.577 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:46:19.577 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:46:19.594 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:46:19.594 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:46:19.611 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:46:19.611 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:46:19.628 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:46:19.628 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:46:19.655 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:19.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:19.678 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:19.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:19.780 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:19.802 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:19.803 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:19.904 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:19.926 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:19.927 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:20.028 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:20.049 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:20.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:20.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:20.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:20.173 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:20.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:20.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:20.275 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:20.300 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:20.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:20.302 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:20.304 INFO:tasks.rados.rados.2.smithi175.stdout:2097: done (2 left) 2024-09-11T03:46:20.305 INFO:tasks.rados.rados.2.smithi175.stdout:2098: done (1 left) 2024-09-11T03:46:20.305 INFO:tasks.rados.rados.2.smithi175.stdout:2099: done (0 left) 2024-09-11T03:46:20.305 INFO:tasks.rados.rados.2.smithi175.stdout:2100: rollback oid 14 current snap is 233 2024-09-11T03:46:20.305 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 14 to 227 2024-09-11T03:46:20.305 INFO:tasks.rados.rados.2.smithi175.stdout:2101: write (excl) oid 5 current snap is 233 2024-09-11T03:46:20.306 INFO:tasks.rados.rados.2.smithi175.stdout:2101: seq_num 712 ranges {766584=798117,2098238=615574,3169759=460379} 2024-09-11T03:46:20.313 INFO:tasks.rados.rados.2.smithi175.stdout:2101: writing smithi17557870-5 from 766584 to 1564701 tid 1 2024-09-11T03:46:20.319 INFO:tasks.rados.rados.2.smithi175.stdout:2101: writing smithi17557870-5 from 2098238 to 2713812 tid 2 2024-09-11T03:46:20.324 INFO:tasks.rados.rados.2.smithi175.stdout:2101: writing smithi17557870-5 from 3169759 to 3630138 tid 3 2024-09-11T03:46:20.324 INFO:tasks.rados.rados.2.smithi175.stdout:2102: snap_remove snap 223 2024-09-11T03:46:20.403 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:20.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:20.427 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:20.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:20.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:20.551 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:20.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:20.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:20.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:20 smithi084 ceph-mon[27254]: pgmap v996: 57 pgs: 5 remapped+peering, 52 active+clean; 141 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 424 KiB/s wr, 8 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-11T03:46:20.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:20 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1150869164' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:20.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:20 smithi084 ceph-mon[27254]: 2.18 scrub starts 2024-09-11T03:46:20.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:20 smithi084 ceph-mon[27254]: 2.18 scrub ok 2024-09-11T03:46:20.647 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T03:46:20.653 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:20.669 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:20 smithi121 ceph-mon[30729]: pgmap v996: 57 pgs: 5 remapped+peering, 52 active+clean; 141 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 424 KiB/s wr, 8 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-11T03:46:20.670 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:20 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1150869164' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:20.670 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:20 smithi121 ceph-mon[30729]: 2.18 scrub starts 2024-09-11T03:46:20.670 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:20 smithi121 ceph-mon[30729]: 2.18 scrub ok 2024-09-11T03:46:20.690 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:20.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:20.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:20.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:20 smithi175 ceph-mon[30894]: pgmap v996: 57 pgs: 5 remapped+peering, 52 active+clean; 141 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 424 KiB/s wr, 8 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-11T03:46:20.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:20 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1150869164' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:20.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:20 smithi175 ceph-mon[30894]: 2.18 scrub starts 2024-09-11T03:46:20.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:20 smithi175 ceph-mon[30894]: 2.18 scrub ok 2024-09-11T03:46:20.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:20.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:20.814 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:20.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:20.893 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:46:20.915 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:20.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:20.938 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:20.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:20.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:46:21.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:21.061 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:21.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:21.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:21.111 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:46:21.111 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:46:21.112 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-11T03:46:21.112 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:46:21.163 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:21.189 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:21.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:21.191 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:21.238 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:21.291 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:21.314 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:21.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:21.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:21.322 INFO:tasks.rados.rados.2.smithi175.stdout:2102: done (2 left) 2024-09-11T03:46:21.322 INFO:tasks.rados.rados.2.smithi175.stdout:2103: write oid 1 current snap is 233 2024-09-11T03:46:21.322 INFO:tasks.rados.rados.2.smithi175.stdout:2100: finishing rollback tid 0 to smithi17557870-14 2024-09-11T03:46:21.322 INFO:tasks.rados.rados.2.smithi175.stdout:2100: finishing rollback tid 1 to smithi17557870-14 2024-09-11T03:46:21.322 INFO:tasks.rados.rados.2.smithi175.stdout:2100: finishing rollback tid 2 to smithi17557870-14 2024-09-11T03:46:21.322 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 832 (ObjNum 668 snap 212 seq_num 668) dirty exists 2024-09-11T03:46:21.322 INFO:tasks.rados.rados.2.smithi175.stdout:2101: finishing write tid 1 to smithi17557870-5 2024-09-11T03:46:21.322 INFO:tasks.rados.rados.2.smithi175.stdout:2101: finishing write tid 2 to smithi17557870-5 2024-09-11T03:46:21.322 INFO:tasks.rados.rados.2.smithi175.stdout:2103: seq_num 713 ranges {783391=473366,1756720=475585,2797182=788004} 2024-09-11T03:46:21.328 INFO:tasks.rados.rados.2.smithi175.stdout:2103: writing smithi17557870-1 from 783391 to 1256757 tid 1 2024-09-11T03:46:21.334 INFO:tasks.rados.rados.2.smithi175.stdout:2103: writing smithi17557870-1 from 1756720 to 2232305 tid 2 2024-09-11T03:46:21.342 INFO:tasks.rados.rados.2.smithi175.stdout:2103: writing smithi17557870-1 from 2797182 to 3585186 tid 3 2024-09-11T03:46:21.342 INFO:tasks.rados.rados.2.smithi175.stdout:2100: done (2 left) 2024-09-11T03:46:21.342 INFO:tasks.rados.rados.2.smithi175.stdout:2104: read oid 40 snap -1 2024-09-11T03:46:21.342 INFO:tasks.rados.rados.2.smithi175.stdout:2104: expect (ObjNum 697 snap 227 seq_num 697) 2024-09-11T03:46:21.342 INFO:tasks.rados.rados.2.smithi175.stdout:2105: copy_from oid 11 from oid 41 current snap is 233 2024-09-11T03:46:21.342 INFO:tasks.rados.rados.2.smithi175.stdout:2101: finishing write tid 3 to smithi17557870-5 2024-09-11T03:46:21.342 INFO:tasks.rados.rados.2.smithi175.stdout:2101: finishing write tid 4 to smithi17557870-5 2024-09-11T03:46:21.343 INFO:tasks.rados.rados.2.smithi175.stdout:2101: finishing write tid 5 to smithi17557870-5 2024-09-11T03:46:21.343 INFO:tasks.rados.rados.2.smithi175.stdout:2101: oid 5 updating version 0 to 702 2024-09-11T03:46:21.343 INFO:tasks.rados.rados.2.smithi175.stdout:2101: oid 5 updating version 702 to 704 2024-09-11T03:46:21.343 INFO:tasks.rados.rados.2.smithi175.stdout:2101: oid 5 version 704 is already newer than 703 2024-09-11T03:46:21.343 INFO:tasks.rados.rados.2.smithi175.stdout:2101: oid 5 version 704 is already newer than 701 2024-09-11T03:46:21.343 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 704 (ObjNum 712 snap 233 seq_num 712) dirty exists 2024-09-11T03:46:21.343 INFO:tasks.rados.rados.2.smithi175.stdout:2101: left oid 5 (ObjNum 712 snap 233 seq_num 712) 2024-09-11T03:46:21.343 INFO:tasks.rados.rados.2.smithi175.stdout:2103: finishing write tid 1 to smithi17557870-1 2024-09-11T03:46:21.343 INFO:tasks.rados.rados.2.smithi175.stdout:2103: finishing write tid 2 to smithi17557870-1 2024-09-11T03:46:21.343 INFO:tasks.rados.rados.2.smithi175.stdout:2101: done (3 left) 2024-09-11T03:46:21.343 INFO:tasks.rados.rados.2.smithi175.stdout:2106: delete oid 17 current snap is 233 2024-09-11T03:46:21.349 INFO:tasks.rados.rados.2.smithi175.stdout:2106: done (3 left) 2024-09-11T03:46:21.349 INFO:tasks.rados.rados.2.smithi175.stdout:2107: copy_from oid 4 from oid 37 current snap is 233 2024-09-11T03:46:21.349 INFO:tasks.rados.rados.2.smithi175.stdout:2103: finishing write tid 3 to smithi17557870-1 2024-09-11T03:46:21.349 INFO:tasks.rados.rados.2.smithi175.stdout:2108: delete oid 19 current snap is 233 2024-09-11T03:46:21.349 INFO:tasks.rados.rados.2.smithi175.stdout:2103: finishing write tid 4 to smithi17557870-1 2024-09-11T03:46:21.349 INFO:tasks.rados.rados.2.smithi175.stdout:2103: finishing write tid 5 to smithi17557870-1 2024-09-11T03:46:21.349 INFO:tasks.rados.rados.2.smithi175.stdout:2103: oid 1 updating version 0 to 798 2024-09-11T03:46:21.350 INFO:tasks.rados.rados.2.smithi175.stdout:2103: oid 1 version 798 is already newer than 797 2024-09-11T03:46:21.350 INFO:tasks.rados.rados.2.smithi175.stdout:2103: oid 1 updating version 798 to 799 2024-09-11T03:46:21.350 INFO:tasks.rados.rados.2.smithi175.stdout:2103: oid 1 updating version 799 to 800 2024-09-11T03:46:21.350 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 800 (ObjNum 713 snap 233 seq_num 713) dirty exists 2024-09-11T03:46:21.350 INFO:tasks.rados.rados.2.smithi175.stdout:2103: left oid 1 (ObjNum 713 snap 233 seq_num 713) 2024-09-11T03:46:21.350 INFO:tasks.rados.rados.2.smithi175.stdout:2107: finishing copy_from to smithi17557870-4 2024-09-11T03:46:21.350 INFO:tasks.rados.rados.2.smithi175.stdout:2107: got expected ENOENT (src dne) 2024-09-11T03:46:21.350 INFO:tasks.rados.rados.2.smithi175.stdout:2107: finishing copy_from racing read to smithi17557870-4 2024-09-11T03:46:21.350 INFO:tasks.rados.rados.2.smithi175.stdout:2104: expect (ObjNum 697 snap 227 seq_num 697) 2024-09-11T03:46:21.367 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:46:21.367 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:46:21.367 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:46:21.367 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:46:21.416 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:21.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:21.438 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:21.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:21.539 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:21.553 INFO:tasks.rados.rados.2.smithi175.stdout:2105: finishing copy_from racing read to smithi17557870-11 2024-09-11T03:46:21.554 INFO:tasks.rados.rados.2.smithi175.stdout:2105: finishing copy_from to smithi17557870-11 2024-09-11T03:46:21.554 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 594 (ObjNum 609 snap 191 seq_num 609) dirty exists 2024-09-11T03:46:21.554 INFO:tasks.rados.rados.2.smithi175.stdout:2103: done (4 left) 2024-09-11T03:46:21.554 INFO:tasks.rados.rados.2.smithi175.stdout:2104: done (3 left) 2024-09-11T03:46:21.554 INFO:tasks.rados.rados.2.smithi175.stdout:2105: done (2 left) 2024-09-11T03:46:21.554 INFO:tasks.rados.rados.2.smithi175.stdout:2107: done (1 left) 2024-09-11T03:46:21.554 INFO:tasks.rados.rados.2.smithi175.stdout:2108: done (0 left) 2024-09-11T03:46:21.554 INFO:tasks.rados.rados.2.smithi175.stdout:2109: snap_create 2024-09-11T03:46:21.561 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:21.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:21.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:21.583 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:21 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:46:21.583 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:21 smithi121 ceph-mon[30729]: 2.c scrub ok 2024-09-11T03:46:21.583 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:21 smithi121 ceph-mon[30729]: osdmap e562: 12 total, 12 up, 12 in 2024-09-11T03:46:21.583 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:21 smithi121 ceph-mon[30729]: 1.1c scrub starts 2024-09-11T03:46:21.583 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:21 smithi121 ceph-mon[30729]: 1.1c scrub ok 2024-09-11T03:46:21.583 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:21 smithi121 ceph-mon[30729]: 1.8 scrub starts 2024-09-11T03:46:21.583 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:21 smithi121 ceph-mon[30729]: 1.8 scrub ok 2024-09-11T03:46:21.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:21 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:46:21.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:21 smithi084 ceph-mon[27254]: 2.c scrub ok 2024-09-11T03:46:21.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:21 smithi084 ceph-mon[27254]: osdmap e562: 12 total, 12 up, 12 in 2024-09-11T03:46:21.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:21 smithi084 ceph-mon[27254]: 1.1c scrub starts 2024-09-11T03:46:21.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:21 smithi084 ceph-mon[27254]: 1.1c scrub ok 2024-09-11T03:46:21.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:21 smithi084 ceph-mon[27254]: 1.8 scrub starts 2024-09-11T03:46:21.623 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:21 smithi084 ceph-mon[27254]: 1.8 scrub ok 2024-09-11T03:46:21.664 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:21.685 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:21.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:21.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:21.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:21 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:46:21.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:21 smithi175 ceph-mon[30894]: 2.c scrub ok 2024-09-11T03:46:21.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:21 smithi175 ceph-mon[30894]: osdmap e562: 12 total, 12 up, 12 in 2024-09-11T03:46:21.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:21 smithi175 ceph-mon[30894]: 1.1c scrub starts 2024-09-11T03:46:21.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:21 smithi175 ceph-mon[30894]: 1.1c scrub ok 2024-09-11T03:46:21.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:21 smithi175 ceph-mon[30894]: 1.8 scrub starts 2024-09-11T03:46:21.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:21 smithi175 ceph-mon[30894]: 1.8 scrub ok 2024-09-11T03:46:21.783 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:21.787 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:21.809 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:21.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:21.810 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:21.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:21.932 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:21.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:21.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:21.957 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:46:21.961 INFO:tasks.rados.rados.2.smithi175.stdout:2109: done (0 left) 2024-09-11T03:46:21.964 INFO:tasks.rados.rados.2.smithi175.stdout:2110: delete oid 30 current snap is 234 2024-09-11T03:46:21.965 INFO:tasks.rados.rados.2.smithi175.stdout:2110: done (0 left) 2024-09-11T03:46:21.965 INFO:tasks.rados.rados.2.smithi175.stdout:2111: read oid 12 snap -1 2024-09-11T03:46:21.965 INFO:tasks.rados.rados.2.smithi175.stdout:2111: expect (ObjNum 688 snap 226 seq_num 688) 2024-09-11T03:46:21.965 INFO:tasks.rados.rados.2.smithi175.stdout:2112: read oid 44 snap -1 2024-09-11T03:46:21.965 INFO:tasks.rados.rados.2.smithi175.stdout:2112: expect deleted 2024-09-11T03:46:21.965 INFO:tasks.rados.rados.2.smithi175.stdout:2113: write oid 31 current snap is 234 2024-09-11T03:46:21.965 INFO:tasks.rados.rados.2.smithi175.stdout:2113: seq_num 717 ranges {450619=490715,1590648=459971} 2024-09-11T03:46:21.970 INFO:tasks.rados.rados.2.smithi175.stdout:2113: writing smithi17557870-31 from 450619 to 941334 tid 1 2024-09-11T03:46:21.977 INFO:tasks.rados.rados.2.smithi175.stdout:2113: writing smithi17557870-31 from 1590648 to 2050619 tid 2 2024-09-11T03:46:21.977 INFO:tasks.rados.rados.2.smithi175.stdout:2114: write (excl) oid 26 current snap is 234 2024-09-11T03:46:21.977 INFO:tasks.rados.rados.2.smithi175.stdout:2114: seq_num 718 ranges {467426=565964,1649130=418296} 2024-09-11T03:46:21.982 INFO:tasks.rados.rados.2.smithi175.stdout:2114: writing smithi17557870-26 from 467426 to 1033390 tid 1 2024-09-11T03:46:21.988 INFO:tasks.rados.rados.2.smithi175.stdout:2114: writing smithi17557870-26 from 1649130 to 2067426 tid 2 2024-09-11T03:46:21.988 INFO:tasks.rados.rados.2.smithi175.stdout:2115: snap_remove snap 227 2024-09-11T03:46:22.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:22.054 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:22.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:22.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:22.157 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:22.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T03:46:22.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:22.180 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:22.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:22.281 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:22.303 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:22.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:22.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:22.362 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:46:22.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:22.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:22.428 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:22.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:22.429 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:46:22.529 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:22.550 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:22.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:22.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:22.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:22 smithi121 ceph-mon[30729]: pgmap v999: 57 pgs: 5 remapped+peering, 52 active+clean; 141 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-11T03:46:22.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:22 smithi121 ceph-mon[30729]: osdmap e563: 12 total, 12 up, 12 in 2024-09-11T03:46:22.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:22 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2304699079' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:46:22.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:22 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:22.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:22 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2304699079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:46:22.568 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:22 smithi121 ceph-mon[30729]: osdmap e564: 12 total, 12 up, 12 in 2024-09-11T03:46:22.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:22 smithi084 ceph-mon[27254]: pgmap v999: 57 pgs: 5 remapped+peering, 52 active+clean; 141 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-11T03:46:22.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:22 smithi084 ceph-mon[27254]: osdmap e563: 12 total, 12 up, 12 in 2024-09-11T03:46:22.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:22 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2304699079' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:46:22.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:22 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:22.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:22 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2304699079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:46:22.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:22 smithi084 ceph-mon[27254]: osdmap e564: 12 total, 12 up, 12 in 2024-09-11T03:46:22.651 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:22.674 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:22.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:22.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:22 smithi175 ceph-mon[30894]: pgmap v999: 57 pgs: 5 remapped+peering, 52 active+clean; 141 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-11T03:46:22.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:22 smithi175 ceph-mon[30894]: osdmap e563: 12 total, 12 up, 12 in 2024-09-11T03:46:22.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:22 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2304699079' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:46:22.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:22 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:22.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:22 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2304699079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:46:22.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:22 smithi175 ceph-mon[30894]: osdmap e564: 12 total, 12 up, 12 in 2024-09-11T03:46:22.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:22.798 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:22.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:22.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:22.899 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:22.912 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:46:22.912 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:46:22.912 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:46:22.912 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:46:22.913 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:46:22.922 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:22.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:22.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:22.962 INFO:tasks.rados.rados.2.smithi175.stdout:2115: done (4 left) 2024-09-11T03:46:22.963 INFO:tasks.rados.rados.2.smithi175.stdout:2116: snap_create 2024-09-11T03:46:22.963 INFO:tasks.rados.rados.2.smithi175.stdout:2111: expect (ObjNum 688 snap 226 seq_num 688) 2024-09-11T03:46:23.024 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:23.042 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:23.046 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:23.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:23.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:23.124 INFO:tasks.rados.rados.2.smithi175.stdout:2113: finishing write tid 1 to smithi17557870-31 2024-09-11T03:46:23.124 INFO:tasks.rados.rados.2.smithi175.stdout:2113: finishing write tid 2 to smithi17557870-31 2024-09-11T03:46:23.124 INFO:tasks.rados.rados.2.smithi175.stdout:2113: finishing write tid 3 to smithi17557870-31 2024-09-11T03:46:23.124 INFO:tasks.rados.rados.2.smithi175.stdout:2113: finishing write tid 4 to smithi17557870-31 2024-09-11T03:46:23.124 INFO:tasks.rados.rados.2.smithi175.stdout:2113: oid 31 updating version 0 to 862 2024-09-11T03:46:23.124 INFO:tasks.rados.rados.2.smithi175.stdout:2113: oid 31 updating version 862 to 863 2024-09-11T03:46:23.125 INFO:tasks.rados.rados.2.smithi175.stdout:2113: oid 31 updating version 863 to 864 2024-09-11T03:46:23.125 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 864 (ObjNum 717 snap 234 seq_num 717) dirty exists 2024-09-11T03:46:23.125 INFO:tasks.rados.rados.2.smithi175.stdout:2113: left oid 31 (ObjNum 717 snap 234 seq_num 717) 2024-09-11T03:46:23.125 INFO:tasks.rados.rados.2.smithi175.stdout:2114: finishing write tid 1 to smithi17557870-26 2024-09-11T03:46:23.125 INFO:tasks.rados.rados.2.smithi175.stdout:2114: finishing write tid 2 to smithi17557870-26 2024-09-11T03:46:23.125 INFO:tasks.rados.rados.2.smithi175.stdout:2114: finishing write tid 3 to smithi17557870-26 2024-09-11T03:46:23.125 INFO:tasks.rados.rados.2.smithi175.stdout:2114: finishing write tid 4 to smithi17557870-26 2024-09-11T03:46:23.125 INFO:tasks.rados.rados.2.smithi175.stdout:2114: oid 26 updating version 0 to 692 2024-09-11T03:46:23.125 INFO:tasks.rados.rados.2.smithi175.stdout:2114: oid 26 version 692 is already newer than 691 2024-09-11T03:46:23.125 INFO:tasks.rados.rados.2.smithi175.stdout:2114: oid 26 version 692 is already newer than 690 2024-09-11T03:46:23.125 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 692 (ObjNum 718 snap 234 seq_num 718) dirty exists 2024-09-11T03:46:23.125 INFO:tasks.rados.rados.2.smithi175.stdout:2114: left oid 26 (ObjNum 718 snap 234 seq_num 718) 2024-09-11T03:46:23.148 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:23.170 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:23.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:23.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:23.206 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:46:23.207 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:46:23.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:23.292 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:23.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:23.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:23.395 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:23.417 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:23.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:23.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:23.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:23 smithi121 ceph-mon[30729]: 1.9 scrub starts 2024-09-11T03:46:23.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:23 smithi121 ceph-mon[30729]: 1.9 scrub ok 2024-09-11T03:46:23.474 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:23 smithi121 ceph-mon[30729]: osdmap e565: 12 total, 12 up, 12 in 2024-09-11T03:46:23.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T03:46:23.520 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:23.541 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:23.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:23.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:23.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:23 smithi084 ceph-mon[27254]: 1.9 scrub starts 2024-09-11T03:46:23.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:23 smithi084 ceph-mon[27254]: 1.9 scrub ok 2024-09-11T03:46:23.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:23 smithi084 ceph-mon[27254]: osdmap e565: 12 total, 12 up, 12 in 2024-09-11T03:46:23.643 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:23.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:23.665 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:23.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:23.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:23 smithi175 ceph-mon[30894]: 1.9 scrub starts 2024-09-11T03:46:23.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:23 smithi175 ceph-mon[30894]: 1.9 scrub ok 2024-09-11T03:46:23.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:23 smithi175 ceph-mon[30894]: osdmap e565: 12 total, 12 up, 12 in 2024-09-11T03:46:23.746 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:46:23.766 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:23.789 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:23.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:23.790 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:23.891 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:23.913 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:23.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:23.948 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1003,"stamp":"2024-09-11T03:46:22.960187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":142890214,"num_objects":64,"num_object_clones":26,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":6,"num_read":2953,"num_read_kb":2867670,"num_write":2057,"num_write_kb":786652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":83482233,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13647,"ondisk_log_size":13647,"up":171,"acting":171,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":772268,"kb_used_data":350792,"kb_used_omap":18,"kb_used_meta":421165,"kb_avail":1124169556,"statfs":{"total":1151940427776,"available":1151149625344,"internally_reserved":0,"allocated":359211008,"data_stored":383259925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":431273275},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4631779,"num_objects":0,"num_object_clones":2,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":29150,"num_write":32,"num_write_kb":9898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":21280844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.961553"},"pg_stats":[{"pgid":"1.1e","version":"20'92","reported_seq":710,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.316271+0000","last_change":"2024-09-11T03:44:24.235660+0000","last_active":"2024-09-11T03:46:21.316271+0000","last_peered":"2024-09-11T03:46:21.316271+0000","last_clean":"2024-09-11T03:46:21.316271+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:46:21.316271+0000","last_undegraded":"2024-09-11T03:46:21.316271+0000","last_fullsized":"2024-09-11T03:46:21.316271+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:24.235616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:24.235616+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:57.069193+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'92","reported_seq":698,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.319688+0000","last_change":"2024-09-11T03:40:41.348889+0000","last_active":"2024-09-11T03:46:21.319688+0000","last_peered":"2024-09-11T03:46:21.319688+0000","last_clean":"2024-09-11T03:46:21.319688+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:46:21.319688+0000","last_undegraded":"2024-09-11T03:46:21.319688+0000","last_fullsized":"2024-09-11T03:46:21.319688+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:41.348828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:41.348828+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'92","reported_seq":715,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.313619+0000","last_change":"2024-09-11T03:46:20.752592+0000","last_active":"2024-09-11T03:46:21.313619+0000","last_peered":"2024-09-11T03:46:21.313619+0000","last_clean":"2024-09-11T03:46:21.313619+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:46:21.313619+0000","last_undegraded":"2024-09-11T03:46:21.313619+0000","last_fullsized":"2024-09-11T03:46:21.313619+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:20.752526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:20.752526+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:32.553208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"547'357","reported_seq":2777,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-11T03:46:18.001450+0000","last_change":"2024-09-11T03:46:16.965612+0000","last_active":"2024-09-11T03:46:18.001450+0000","last_peered":"2024-09-11T03:46:18.001450+0000","last_clean":"2024-09-11T03:46:18.001450+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:46:18.001450+0000","last_undegraded":"2024-09-11T03:46:18.001450+0000","last_fullsized":"2024-09-11T03:46:18.001450+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"467'328","last_scrub_stamp":"2024-09-11T03:44:45.863666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:44:45.863666+0000","objects_scrubbed":4,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00053754699999999998,"stat_sum":{"num_bytes":2739673,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":99504,"num_write":46,"num_write_kb":16817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4865740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.1b","version":"20'92","reported_seq":700,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.313601+0000","last_change":"2024-09-11T03:40:42.166354+0000","last_active":"2024-09-11T03:46:21.313601+0000","last_peered":"2024-09-11T03:46:21.313601+0000","last_clean":"2024-09-11T03:46:21.313601+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:46:21.313601+0000","last_undegraded":"2024-09-11T03:46:21.313601+0000","last_fullsized":"2024-09-11T03:46:21.313601+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:42.166295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:42.166295+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:21.519903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.19","version":"563'458","reported_seq":2917,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.960512+0000","last_change":"2024-09-11T03:46:21.355552+0000","last_active":"2024-09-11T03:46:21.960512+0000","last_peered":"2024-09-11T03:46:21.960512+0000","last_clean":"2024-09-11T03:46:21.960512+0000","last_became_active":"2024-09-11T03:46:19.002002+0000","last_became_peered":"2024-09-11T03:46:19.002002+0000","last_unstale":"2024-09-11T03:46:21.960512+0000","last_undegraded":"2024-09-11T03:46:21.960512+0000","last_fullsized":"2024-09-11T03:46:21.960512+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"504'431","last_scrub_stamp":"2024-09-11T03:45:23.194125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:23.194125+0000","objects_scrubbed":2,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:29.635003+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058382180000000001,"stat_sum":{"num_bytes":5100579,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":45910,"num_write":30,"num_write_kb":14045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4870845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d1","length":"1"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.1a","version":"20'92","reported_seq":464,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.319258+0000","last_change":"2024-09-11T03:45:52.914136+0000","last_active":"2024-09-11T03:46:21.319258+0000","last_peered":"2024-09-11T03:46:21.319258+0000","last_clean":"2024-09-11T03:46:21.319258+0000","last_became_active":"2024-09-11T03:44:43.938873+0000","last_became_peered":"2024-09-11T03:44:43.938873+0000","last_unstale":"2024-09-11T03:46:21.319258+0000","last_undegraded":"2024-09-11T03:46:21.319258+0000","last_fullsized":"2024-09-11T03:46:21.319258+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:52.914088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:52.914088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:10.451317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"20'92","reported_seq":334,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.974164+0000","last_change":"2024-09-11T03:46:15.973211+0000","last_active":"2024-09-11T03:46:21.974164+0000","last_peered":"2024-09-11T03:46:21.974164+0000","last_clean":"2024-09-11T03:46:21.974164+0000","last_became_active":"2024-09-11T03:46:15.972841+0000","last_became_peered":"2024-09-11T03:46:15.972841+0000","last_unstale":"2024-09-11T03:46:21.974164+0000","last_undegraded":"2024-09-11T03:46:21.974164+0000","last_fullsized":"2024-09-11T03:46:21.974164+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:30.424428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:30.424428+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:42.733857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'92","reported_seq":840,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.343855+0000","last_change":"2024-09-11T03:46:17.873917+0000","last_active":"2024-09-11T03:46:21.343855+0000","last_peered":"2024-09-11T03:46:21.343855+0000","last_clean":"2024-09-11T03:46:21.343855+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:46:21.343855+0000","last_undegraded":"2024-09-11T03:46:21.343855+0000","last_fullsized":"2024-09-11T03:46:21.343855+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:17.873858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:17.873858+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:45.517409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"541'254","reported_seq":2553,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.960438+0000","last_change":"2024-09-11T03:46:21.353047+0000","last_active":"2024-09-11T03:46:21.960438+0000","last_peered":"2024-09-11T03:46:21.960438+0000","last_clean":"2024-09-11T03:46:21.960438+0000","last_became_active":"2024-09-11T03:46:19.004558+0000","last_became_peered":"2024-09-11T03:46:19.004558+0000","last_unstale":"2024-09-11T03:46:21.960438+0000","last_undegraded":"2024-09-11T03:46:21.960438+0000","last_fullsized":"2024-09-11T03:46:21.960438+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"541'254","last_scrub_stamp":"2024-09-11T03:46:12.749744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:12.749744+0000","objects_scrubbed":1,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:27.822380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012829269999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":8231,"num_write":22,"num_write_kb":4725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d1","length":"1"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.17","version":"20'92","reported_seq":731,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.313777+0000","last_change":"2024-09-11T03:46:12.911087+0000","last_active":"2024-09-11T03:46:21.313777+0000","last_peered":"2024-09-11T03:46:21.313777+0000","last_clean":"2024-09-11T03:46:21.313777+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:46:21.313777+0000","last_undegraded":"2024-09-11T03:46:21.313777+0000","last_fullsized":"2024-09-11T03:46:21.313777+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:12.911035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:12.911035+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:27.756515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":2664,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-11T03:46:18.001946+0000","last_change":"2024-09-11T03:46:16.967763+0000","last_active":"2024-09-11T03:46:18.001946+0000","last_peered":"2024-09-11T03:46:18.001946+0000","last_clean":"2024-09-11T03:46:18.001946+0000","last_became_active":"2024-09-11T03:41:38.888228+0000","last_became_peered":"2024-09-11T03:41:38.888228+0000","last_unstale":"2024-09-11T03:46:18.001946+0000","last_undegraded":"2024-09-11T03:46:18.001946+0000","last_fullsized":"2024-09-11T03:46:18.001946+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:45:28.993767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:28.993767+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:52.293465+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00044193800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.16","version":"20'92","reported_seq":178,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.316138+0000","last_change":"2024-09-11T03:46:05.016475+0000","last_active":"2024-09-11T03:46:21.316138+0000","last_peered":"2024-09-11T03:46:21.316138+0000","last_clean":"2024-09-11T03:46:21.316138+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:46:21.316138+0000","last_undegraded":"2024-09-11T03:46:21.316138+0000","last_fullsized":"2024-09-11T03:46:21.316138+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:23.840184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:23.840184+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:46:34.095648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":36063,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.16","version":"563'380","reported_seq":2815,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.353525+0000","last_change":"2024-09-11T03:46:21.353988+0000","last_active":"2024-09-11T03:46:21.353525+0000","last_peered":"2024-09-11T03:46:21.353525+0000","last_clean":"2024-09-11T03:46:21.353525+0000","last_became_active":"2024-09-11T03:46:19.007019+0000","last_became_peered":"2024-09-11T03:46:19.007019+0000","last_unstale":"2024-09-11T03:46:21.353525+0000","last_undegraded":"2024-09-11T03:46:21.353525+0000","last_fullsized":"2024-09-11T03:46:21.353525+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"531'357","last_scrub_stamp":"2024-09-11T03:45:54.655442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:54.655442+0000","objects_scrubbed":4,"log_size":380,"log_dups_size":0,"ondisk_log_size":380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:57.809736+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0095603679999999996,"stat_sum":{"num_bytes":8526356,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":52,"num_read_kb":36894,"num_write":63,"num_write_kb":25338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":5263193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.15","version":"20'92","reported_seq":705,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.316705+0000","last_change":"2024-09-11T03:43:09.565121+0000","last_active":"2024-09-11T03:46:21.316705+0000","last_peered":"2024-09-11T03:46:21.316705+0000","last_clean":"2024-09-11T03:46:21.316705+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:46:21.316705+0000","last_undegraded":"2024-09-11T03:46:21.316705+0000","last_fullsized":"2024-09-11T03:46:21.316705+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:09.565070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:09.565070+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:32.580107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":2030,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.960663+0000","last_change":"2024-09-11T03:46:21.355860+0000","last_active":"2024-09-11T03:46:21.960663+0000","last_peered":"2024-09-11T03:46:21.960663+0000","last_clean":"2024-09-11T03:46:21.960663+0000","last_became_active":"2024-09-11T03:46:19.003959+0000","last_became_peered":"2024-09-11T03:46:19.003959+0000","last_unstale":"2024-09-11T03:46:21.960663+0000","last_undegraded":"2024-09-11T03:46:21.960663+0000","last_fullsized":"2024-09-11T03:46:21.960663+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:45:13.319584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:13.319584+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:24.896644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003465264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d1","length":"1"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.14","version":"20'92","reported_seq":738,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-11T03:46:18.001851+0000","last_change":"2024-09-11T03:44:12.182364+0000","last_active":"2024-09-11T03:46:18.001851+0000","last_peered":"2024-09-11T03:46:18.001851+0000","last_clean":"2024-09-11T03:46:18.001851+0000","last_became_active":"2024-09-11T03:40:27.847297+0000","last_became_peered":"2024-09-11T03:40:27.847297+0000","last_unstale":"2024-09-11T03:46:18.001851+0000","last_undegraded":"2024-09-11T03:46:18.001851+0000","last_fullsized":"2024-09-11T03:46:18.001851+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:12.182320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:12.182320+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:00.159862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":1473,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.314160+0000","last_change":"2024-09-11T03:46:21.314596+0000","last_active":"2024-09-11T03:46:21.314160+0000","last_peered":"2024-09-11T03:46:21.314160+0000","last_clean":"2024-09-11T03:46:21.314160+0000","last_became_active":"2024-09-11T03:41:35.874295+0000","last_became_peered":"2024-09-11T03:41:35.874295+0000","last_unstale":"2024-09-11T03:46:21.314160+0000","last_undegraded":"2024-09-11T03:46:21.314160+0000","last_fullsized":"2024-09-11T03:46:21.314160+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:45:53.606467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:53.606467+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:14.634061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070632799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.13","version":"20'92","reported_seq":844,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.319334+0000","last_change":"2024-09-11T03:45:47.911982+0000","last_active":"2024-09-11T03:46:21.319334+0000","last_peered":"2024-09-11T03:46:21.319334+0000","last_clean":"2024-09-11T03:46:21.319334+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:46:21.319334+0000","last_undegraded":"2024-09-11T03:46:21.319334+0000","last_fullsized":"2024-09-11T03:46:21.319334+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:47.911918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:47.911918+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:14.917780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":1534,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.960266+0000","last_change":"2024-09-11T03:46:21.351705+0000","last_active":"2024-09-11T03:46:21.960266+0000","last_peered":"2024-09-11T03:46:21.960266+0000","last_clean":"2024-09-11T03:46:21.960266+0000","last_became_active":"2024-09-11T03:46:19.005117+0000","last_became_peered":"2024-09-11T03:46:19.005117+0000","last_unstale":"2024-09-11T03:46:21.960266+0000","last_undegraded":"2024-09-11T03:46:21.960266+0000","last_fullsized":"2024-09-11T03:46:21.960266+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:45:53.909148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:53.909148+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:03.743597+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001733905,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d1","length":"1"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.12","version":"20'92","reported_seq":744,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.343895+0000","last_change":"2024-09-11T03:46:15.974226+0000","last_active":"2024-09-11T03:46:21.343895+0000","last_peered":"2024-09-11T03:46:21.343895+0000","last_clean":"2024-09-11T03:46:21.343895+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:46:21.343895+0000","last_undegraded":"2024-09-11T03:46:21.343895+0000","last_fullsized":"2024-09-11T03:46:21.343895+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:09.782621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:09.782621+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"563'406","reported_seq":2796,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.974250+0000","last_change":"2024-09-11T03:46:21.322774+0000","last_active":"2024-09-11T03:46:21.974250+0000","last_peered":"2024-09-11T03:46:21.974250+0000","last_clean":"2024-09-11T03:46:21.974250+0000","last_became_active":"2024-09-11T03:43:55.918817+0000","last_became_peered":"2024-09-11T03:43:55.918817+0000","last_unstale":"2024-09-11T03:46:21.974250+0000","last_undegraded":"2024-09-11T03:46:21.974250+0000","last_fullsized":"2024-09-11T03:46:21.974250+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"513'359","last_scrub_stamp":"2024-09-11T03:45:31.423318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:31.423318+0000","objects_scrubbed":4,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:56.837974+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0041179429999999998,"stat_sum":{"num_bytes":6797553,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":137,"num_read_kb":133111,"num_write":89,"num_write_kb":34392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":15216236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"d1","length":"1"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.11","version":"20'92","reported_seq":727,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-09-11T03:46:20.316657+0000","last_change":"2024-09-11T03:40:35.703292+0000","last_active":"2024-09-11T03:46:20.316657+0000","last_peered":"2024-09-11T03:46:20.316657+0000","last_clean":"2024-09-11T03:46:20.316657+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:46:20.316657+0000","last_undegraded":"2024-09-11T03:46:20.316657+0000","last_fullsized":"2024-09-11T03:46:20.316657+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:35.703203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:35.703203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:00.137853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":2452,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-11T03:46:18.001786+0000","last_change":"2024-09-11T03:46:16.967225+0000","last_active":"2024-09-11T03:46:18.001786+0000","last_peered":"2024-09-11T03:46:18.001786+0000","last_clean":"2024-09-11T03:46:18.001786+0000","last_became_active":"2024-09-11T03:41:38.886834+0000","last_became_peered":"2024-09-11T03:41:38.886834+0000","last_unstale":"2024-09-11T03:46:18.001786+0000","last_undegraded":"2024-09-11T03:46:18.001786+0000","last_fullsized":"2024-09-11T03:46:18.001786+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:46:14.674306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:14.674306+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:36.373218+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00045930299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.10","version":"20'92","reported_seq":747,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.343857+0000","last_change":"2024-09-11T03:44:26.391068+0000","last_active":"2024-09-11T03:46:21.343857+0000","last_peered":"2024-09-11T03:46:21.343857+0000","last_clean":"2024-09-11T03:46:21.343857+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:46:21.343857+0000","last_undegraded":"2024-09-11T03:46:21.343857+0000","last_fullsized":"2024-09-11T03:46:21.343857+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:26.391001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:26.391001+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:32.609928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"563'642","reported_seq":3021,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.316852+0000","last_change":"2024-09-11T03:46:21.317110+0000","last_active":"2024-09-11T03:46:21.316852+0000","last_peered":"2024-09-11T03:46:21.316852+0000","last_clean":"2024-09-11T03:46:21.316852+0000","last_became_active":"2024-09-11T03:38:29.888685+0000","last_became_peered":"2024-09-11T03:38:29.888685+0000","last_unstale":"2024-09-11T03:46:21.316852+0000","last_undegraded":"2024-09-11T03:46:21.316852+0000","last_fullsized":"2024-09-11T03:46:21.316852+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"553'624","last_scrub_stamp":"2024-09-11T03:46:11.866762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:11.866762+0000","objects_scrubbed":8,"log_size":642,"log_dups_size":0,"ondisk_log_size":642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:30.790645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00320324,"stat_sum":{"num_bytes":7950074,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":231,"num_read_kb":191264,"num_write":178,"num_write_kb":82238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.8","version":"20'92","reported_seq":813,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.344127+0000","last_change":"2024-09-11T03:46:20.861683+0000","last_active":"2024-09-11T03:46:21.344127+0000","last_peered":"2024-09-11T03:46:21.344127+0000","last_clean":"2024-09-11T03:46:21.344127+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:46:21.344127+0000","last_undegraded":"2024-09-11T03:46:21.344127+0000","last_fullsized":"2024-09-11T03:46:21.344127+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:20.861635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:20.861635+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:27.499407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"557'273","reported_seq":2658,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.316486+0000","last_change":"2024-09-11T03:46:21.316824+0000","last_active":"2024-09-11T03:46:21.316486+0000","last_peered":"2024-09-11T03:46:21.316486+0000","last_clean":"2024-09-11T03:46:21.316486+0000","last_became_active":"2024-09-11T03:38:29.891669+0000","last_became_peered":"2024-09-11T03:38:29.891669+0000","last_unstale":"2024-09-11T03:46:21.316486+0000","last_undegraded":"2024-09-11T03:46:21.316486+0000","last_fullsized":"2024-09-11T03:46:21.316486+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"535'255","last_scrub_stamp":"2024-09-11T03:45:54.125664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:54.125664+0000","objects_scrubbed":4,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:06.966839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00064235900000000001,"stat_sum":{"num_bytes":3512795,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":146,"num_read_kb":149021,"num_write":89,"num_write_kb":32129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.9","version":"20'92","reported_seq":1134,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.316067+0000","last_change":"2024-09-11T03:42:00.816905+0000","last_active":"2024-09-11T03:46:21.316067+0000","last_peered":"2024-09-11T03:46:21.316067+0000","last_clean":"2024-09-11T03:46:21.316067+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:46:21.316067+0000","last_undegraded":"2024-09-11T03:46:21.316067+0000","last_fullsized":"2024-09-11T03:46:21.316067+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:00.816850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:00.816850+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:21.512672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"563'859","reported_seq":3335,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.325129+0000","last_change":"2024-09-11T03:46:21.325527+0000","last_active":"2024-09-11T03:46:21.325129+0000","last_peered":"2024-09-11T03:46:21.325129+0000","last_clean":"2024-09-11T03:46:21.325129+0000","last_became_active":"2024-09-11T03:41:33.865638+0000","last_became_peered":"2024-09-11T03:41:33.865638+0000","last_unstale":"2024-09-11T03:46:21.325129+0000","last_undegraded":"2024-09-11T03:46:21.325129+0000","last_fullsized":"2024-09-11T03:46:21.325129+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'731","last_scrub_stamp":"2024-09-11T03:45:23.948691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:23.948691+0000","objects_scrubbed":13,"log_size":859,"log_dups_size":0,"ondisk_log_size":859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:29.808162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0057016760000000001,"stat_sum":{"num_bytes":15395872,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":314,"num_read_kb":347374,"num_write":162,"num_write_kb":55087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.a","version":"20'92","reported_seq":1064,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-11T03:46:18.001387+0000","last_change":"2024-09-11T03:46:15.973700+0000","last_active":"2024-09-11T03:46:18.001387+0000","last_peered":"2024-09-11T03:46:18.001387+0000","last_clean":"2024-09-11T03:46:18.001387+0000","last_became_active":"2024-09-11T03:46:15.973271+0000","last_became_peered":"2024-09-11T03:46:15.973271+0000","last_unstale":"2024-09-11T03:46:18.001387+0000","last_undegraded":"2024-09-11T03:46:18.001387+0000","last_fullsized":"2024-09-11T03:46:18.001387+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:24.066237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:24.066237+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:25.003619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"557'324","reported_seq":2715,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.316763+0000","last_change":"2024-09-11T03:46:21.317132+0000","last_active":"2024-09-11T03:46:21.316763+0000","last_peered":"2024-09-11T03:46:21.316763+0000","last_clean":"2024-09-11T03:46:21.316763+0000","last_became_active":"2024-09-11T03:41:33.863875+0000","last_became_peered":"2024-09-11T03:41:33.863875+0000","last_unstale":"2024-09-11T03:46:21.316763+0000","last_undegraded":"2024-09-11T03:46:21.316763+0000","last_fullsized":"2024-09-11T03:46:21.316763+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'324","last_scrub_stamp":"2024-09-11T03:46:16.518345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:16.518345+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:42.901189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00067516699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":13817,"num_write":34,"num_write_kb":9341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.b","version":"20'92","reported_seq":738,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.316069+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:46:21.316069+0000","last_peered":"2024-09-11T03:46:21.316069+0000","last_clean":"2024-09-11T03:46:21.316069+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:46:21.316069+0000","last_undegraded":"2024-09-11T03:46:21.316069+0000","last_fullsized":"2024-09-11T03:46:21.316069+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:05.070582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"564'864","reported_seq":3341,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.981496+0000","last_change":"2024-09-11T03:46:21.357129+0000","last_active":"2024-09-11T03:46:21.981496+0000","last_peered":"2024-09-11T03:46:21.981496+0000","last_clean":"2024-09-11T03:46:21.981496+0000","last_became_active":"2024-09-11T03:46:19.004997+0000","last_became_peered":"2024-09-11T03:46:19.004997+0000","last_unstale":"2024-09-11T03:46:21.981496+0000","last_undegraded":"2024-09-11T03:46:21.981496+0000","last_fullsized":"2024-09-11T03:46:21.981496+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"524'761","last_scrub_stamp":"2024-09-11T03:45:45.237733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:45.237733+0000","objects_scrubbed":11,"log_size":864,"log_dups_size":0,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:56.600761+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0050338149999999996,"stat_sum":{"num_bytes":13286076,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":290,"num_read_kb":349977,"num_write":146,"num_write_kb":57463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":25429065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d1","length":"1"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.4","version":"20'92","reported_seq":920,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.316177+0000","last_change":"2024-09-11T03:44:28.192468+0000","last_active":"2024-09-11T03:46:21.316177+0000","last_peered":"2024-09-11T03:46:21.316177+0000","last_clean":"2024-09-11T03:46:21.316177+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:46:21.316177+0000","last_undegraded":"2024-09-11T03:46:21.316177+0000","last_fullsized":"2024-09-11T03:46:21.316177+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:28.192403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:28.192403+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:56.078747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":2532,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.322801+0000","last_change":"2024-09-11T03:46:21.324118+0000","last_active":"2024-09-11T03:46:21.322801+0000","last_peered":"2024-09-11T03:46:21.322801+0000","last_clean":"2024-09-11T03:46:21.322801+0000","last_became_active":"2024-09-11T03:41:33.865499+0000","last_became_peered":"2024-09-11T03:41:33.865499+0000","last_unstale":"2024-09-11T03:46:21.322801+0000","last_undegraded":"2024-09-11T03:46:21.322801+0000","last_fullsized":"2024-09-11T03:46:21.322801+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:45:49.918830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:49.918830+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:53.818652+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.004227799,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.5","version":"20'92","reported_seq":337,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.974394+0000","last_change":"2024-09-11T03:46:17.255120+0000","last_active":"2024-09-11T03:46:21.974394+0000","last_peered":"2024-09-11T03:46:21.974394+0000","last_clean":"2024-09-11T03:46:21.974394+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:46:21.974394+0000","last_undegraded":"2024-09-11T03:46:21.974394+0000","last_fullsized":"2024-09-11T03:46:21.974394+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:17.255073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:17.255073+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:31.448759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"563'795","reported_seq":3212,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.316886+0000","last_change":"2024-09-11T03:46:21.317134+0000","last_active":"2024-09-11T03:46:21.316886+0000","last_peered":"2024-09-11T03:46:21.316886+0000","last_clean":"2024-09-11T03:46:21.316886+0000","last_became_active":"2024-09-11T03:41:33.866316+0000","last_became_peered":"2024-09-11T03:41:33.866316+0000","last_unstale":"2024-09-11T03:46:21.316886+0000","last_undegraded":"2024-09-11T03:46:21.316886+0000","last_fullsized":"2024-09-11T03:46:21.316886+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"530'746","last_scrub_stamp":"2024-09-11T03:45:50.568426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:50.568426+0000","objects_scrubbed":9,"log_size":795,"log_dups_size":0,"ondisk_log_size":795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:10.700666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0033360289999999999,"stat_sum":{"num_bytes":14961986,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":164,"num_read_kb":169679,"num_write":117,"num_write_kb":46550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.6","version":"20'92","reported_seq":322,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.974389+0000","last_change":"2024-09-11T03:44:26.477597+0000","last_active":"2024-09-11T03:46:21.974389+0000","last_peered":"2024-09-11T03:46:21.974389+0000","last_clean":"2024-09-11T03:46:21.974389+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:46:21.974389+0000","last_undegraded":"2024-09-11T03:46:21.974389+0000","last_fullsized":"2024-09-11T03:46:21.974389+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:26.477545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:26.477545+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:57.218670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"557'427","reported_seq":2849,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.320128+0000","last_change":"2024-09-11T03:46:21.322681+0000","last_active":"2024-09-11T03:46:21.320128+0000","last_peered":"2024-09-11T03:46:21.320128+0000","last_clean":"2024-09-11T03:46:21.320128+0000","last_became_active":"2024-09-11T03:44:10.921135+0000","last_became_peered":"2024-09-11T03:44:10.921135+0000","last_unstale":"2024-09-11T03:46:21.320128+0000","last_undegraded":"2024-09-11T03:46:21.320128+0000","last_fullsized":"2024-09-11T03:46:21.320128+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'397","last_scrub_stamp":"2024-09-11T03:45:26.963120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:26.963120+0000","objects_scrubbed":7,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:29.887302+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.002727447,"stat_sum":{"num_bytes":4336470,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":135391,"num_write":75,"num_write_kb":21420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":4118225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.7","version":"20'92","reported_seq":24,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.960244+0000","last_change":"2024-09-11T03:46:15.972170+0000","last_active":"2024-09-11T03:46:21.960244+0000","last_peered":"2024-09-11T03:46:21.960244+0000","last_clean":"2024-09-11T03:46:21.960244+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:46:21.960244+0000","last_undegraded":"2024-09-11T03:46:21.960244+0000","last_fullsized":"2024-09-11T03:46:21.960244+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:12.360829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:12.360829+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:25.744381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"531'216","reported_seq":2672,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.344393+0000","last_change":"2024-09-11T03:46:21.345256+0000","last_active":"2024-09-11T03:46:21.344393+0000","last_peered":"2024-09-11T03:46:21.344393+0000","last_clean":"2024-09-11T03:46:21.344393+0000","last_became_active":"2024-09-11T03:46:19.005706+0000","last_became_peered":"2024-09-11T03:46:19.005706+0000","last_unstale":"2024-09-11T03:46:21.344393+0000","last_undegraded":"2024-09-11T03:46:21.344393+0000","last_fullsized":"2024-09-11T03:46:21.344393+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:45:53.642117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:53.642117+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:04.045954+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0.001148736,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":27471,"num_write":22,"num_write_kb":11305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.1","version":"20'92","reported_seq":327,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.974279+0000","last_change":"2024-09-11T03:45:26.373529+0000","last_active":"2024-09-11T03:46:21.974279+0000","last_peered":"2024-09-11T03:46:21.974279+0000","last_clean":"2024-09-11T03:46:21.974279+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:46:21.974279+0000","last_undegraded":"2024-09-11T03:46:21.974279+0000","last_fullsized":"2024-09-11T03:46:21.974279+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:26.373478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:26.373478+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:27.036677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"559'224","reported_seq":2607,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-11T03:46:18.003077+0000","last_change":"2024-09-11T03:46:16.965220+0000","last_active":"2024-09-11T03:46:18.003077+0000","last_peered":"2024-09-11T03:46:18.003077+0000","last_clean":"2024-09-11T03:46:18.003077+0000","last_became_active":"2024-09-11T03:41:33.866217+0000","last_became_peered":"2024-09-11T03:41:33.866217+0000","last_unstale":"2024-09-11T03:46:18.003077+0000","last_undegraded":"2024-09-11T03:46:18.003077+0000","last_fullsized":"2024-09-11T03:46:18.003077+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'176","last_scrub_stamp":"2024-09-11T03:45:41.180057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:41.180057+0000","objects_scrubbed":3,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:50.929123+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0052948789999999997,"stat_sum":{"num_bytes":6227908,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":29194,"num_write":38,"num_write_kb":13620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.2","version":"20'92","reported_seq":714,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-09-11T03:46:20.316387+0000","last_change":"2024-09-11T03:44:25.934869+0000","last_active":"2024-09-11T03:46:20.316387+0000","last_peered":"2024-09-11T03:46:20.316387+0000","last_clean":"2024-09-11T03:46:20.316387+0000","last_became_active":"2024-09-11T03:44:25.934580+0000","last_became_peered":"2024-09-11T03:44:25.934580+0000","last_unstale":"2024-09-11T03:46:20.316387+0000","last_undegraded":"2024-09-11T03:46:20.316387+0000","last_fullsized":"2024-09-11T03:46:20.316387+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:23.494418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:23.494418+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:37.110438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"563'192","reported_seq":2638,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.319581+0000","last_change":"2024-09-11T03:46:21.319840+0000","last_active":"2024-09-11T03:46:21.319581+0000","last_peered":"2024-09-11T03:46:21.319581+0000","last_clean":"2024-09-11T03:46:21.319581+0000","last_became_active":"2024-09-11T03:43:55.920265+0000","last_became_peered":"2024-09-11T03:43:55.920265+0000","last_unstale":"2024-09-11T03:46:21.319581+0000","last_undegraded":"2024-09-11T03:46:21.319581+0000","last_fullsized":"2024-09-11T03:46:21.319581+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"537'168","last_scrub_stamp":"2024-09-11T03:45:55.666981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:55.666981+0000","objects_scrubbed":4,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:04.099380+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0031930999999999999,"stat_sum":{"num_bytes":3445567,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":79688,"num_write":30,"num_write_kb":9039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4387035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.3","version":"20'92","reported_seq":758,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.319502+0000","last_change":"2024-09-11T03:46:15.969606+0000","last_active":"2024-09-11T03:46:21.319502+0000","last_peered":"2024-09-11T03:46:21.319502+0000","last_clean":"2024-09-11T03:46:21.319502+0000","last_became_active":"2024-09-11T03:46:15.968960+0000","last_became_peered":"2024-09-11T03:46:15.968960+0000","last_unstale":"2024-09-11T03:46:21.319502+0000","last_undegraded":"2024-09-11T03:46:21.319502+0000","last_fullsized":"2024-09-11T03:46:21.319502+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:44:07.508193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:44:07.508193+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"563'380","reported_seq":2765,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.353040+0000","last_change":"2024-09-11T03:46:21.348201+0000","last_active":"2024-09-11T03:46:21.353040+0000","last_peered":"2024-09-11T03:46:21.353040+0000","last_clean":"2024-09-11T03:46:21.353040+0000","last_became_active":"2024-09-11T03:41:33.862271+0000","last_became_peered":"2024-09-11T03:41:33.862271+0000","last_unstale":"2024-09-11T03:46:21.353040+0000","last_undegraded":"2024-09-11T03:46:21.353040+0000","last_fullsized":"2024-09-11T03:46:21.353040+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'327","last_scrub_stamp":"2024-09-11T03:45:12.798880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:12.798880+0000","objects_scrubbed":5,"log_size":380,"log_dups_size":0,"ondisk_log_size":380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.004068543,"stat_sum":{"num_bytes":4958329,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":95408,"num_write":66,"num_write_kb":24706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.0","version":"20'92","reported_seq":1155,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.316743+0000","last_change":"2024-09-11T03:46:15.975561+0000","last_active":"2024-09-11T03:46:21.316743+0000","last_peered":"2024-09-11T03:46:21.316743+0000","last_clean":"2024-09-11T03:46:21.316743+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:46:21.316743+0000","last_undegraded":"2024-09-11T03:46:21.316743+0000","last_fullsized":"2024-09-11T03:46:21.316743+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:23.491587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:23.491587+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:46:27.401841+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"20'92","reported_seq":32,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.960635+0000","last_change":"2024-09-11T03:46:16.476244+0000","last_active":"2024-09-11T03:46:21.960635+0000","last_peered":"2024-09-11T03:46:21.960635+0000","last_clean":"2024-09-11T03:46:21.960635+0000","last_became_active":"2024-09-11T03:46:15.973796+0000","last_became_peered":"2024-09-11T03:46:15.973796+0000","last_unstale":"2024-09-11T03:46:21.960635+0000","last_undegraded":"2024-09-11T03:46:21.960635+0000","last_fullsized":"2024-09-11T03:46:21.960635+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:16.476181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:16.476181+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:43.297927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"563'662","reported_seq":3034,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.320354+0000","last_change":"2024-09-11T03:46:21.320723+0000","last_active":"2024-09-11T03:46:21.320354+0000","last_peered":"2024-09-11T03:46:21.320354+0000","last_clean":"2024-09-11T03:46:21.320354+0000","last_became_active":"2024-09-11T03:38:30.163349+0000","last_became_peered":"2024-09-11T03:38:30.163349+0000","last_unstale":"2024-09-11T03:46:21.320354+0000","last_undegraded":"2024-09-11T03:46:21.320354+0000","last_fullsized":"2024-09-11T03:46:21.320354+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'658","last_scrub_stamp":"2024-09-11T03:46:20.155183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:20.155183+0000","objects_scrubbed":5,"log_size":662,"log_dups_size":0,"ondisk_log_size":662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:26.127450+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0041479480000000003,"stat_sum":{"num_bytes":12033665,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":272,"num_read_kb":237150,"num_write":226,"num_write_kb":90521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.e","version":"20'92","reported_seq":464,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.316522+0000","last_change":"2024-09-11T03:45:53.637950+0000","last_active":"2024-09-11T03:46:21.316522+0000","last_peered":"2024-09-11T03:46:21.316522+0000","last_clean":"2024-09-11T03:46:21.316522+0000","last_became_active":"2024-09-11T03:44:43.940022+0000","last_became_peered":"2024-09-11T03:44:43.940022+0000","last_unstale":"2024-09-11T03:46:21.316522+0000","last_undegraded":"2024-09-11T03:46:21.316522+0000","last_fullsized":"2024-09-11T03:46:21.316522+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:53.637891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:53.637891+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:14.116074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"563'748","reported_seq":3104,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.348087+0000","last_change":"2024-09-11T03:46:21.348301+0000","last_active":"2024-09-11T03:46:21.348087+0000","last_peered":"2024-09-11T03:46:21.348087+0000","last_clean":"2024-09-11T03:46:21.348087+0000","last_became_active":"2024-09-11T03:42:30.902959+0000","last_became_peered":"2024-09-11T03:42:30.902959+0000","last_unstale":"2024-09-11T03:46:21.348087+0000","last_undegraded":"2024-09-11T03:46:21.348087+0000","last_fullsized":"2024-09-11T03:46:21.348087+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'735","last_scrub_stamp":"2024-09-11T03:46:14.841436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:14.841436+0000","objects_scrubbed":4,"log_size":748,"log_dups_size":0,"ondisk_log_size":748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:37.960469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.004238782,"stat_sum":{"num_bytes":8877815,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":309,"num_read_kb":281936,"num_write":227,"num_write_kb":82640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17460823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.d","version":"20'92","reported_seq":749,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.316574+0000","last_change":"2024-09-11T03:42:51.733829+0000","last_active":"2024-09-11T03:46:21.316574+0000","last_peered":"2024-09-11T03:46:21.316574+0000","last_clean":"2024-09-11T03:46:21.316574+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:46:21.316574+0000","last_undegraded":"2024-09-11T03:46:21.316574+0000","last_fullsized":"2024-09-11T03:46:21.316574+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:51.733762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:51.733762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:06.611256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"557'828","reported_seq":3163,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-11T03:46:18.002686+0000","last_change":"2024-09-11T03:46:16.960970+0000","last_active":"2024-09-11T03:46:18.002686+0000","last_peered":"2024-09-11T03:46:18.002686+0000","last_clean":"2024-09-11T03:46:18.002686+0000","last_became_active":"2024-09-11T03:38:29.891761+0000","last_became_peered":"2024-09-11T03:38:29.891761+0000","last_unstale":"2024-09-11T03:46:18.002686+0000","last_undegraded":"2024-09-11T03:46:18.002686+0000","last_fullsized":"2024-09-11T03:46:18.002686+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"532'778","last_scrub_stamp":"2024-09-11T03:45:51.172946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:51.172946+0000","objects_scrubbed":11,"log_size":828,"log_dups_size":0,"ondisk_log_size":828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:10.043554+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00072996400000000005,"stat_sum":{"num_bytes":13563711,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":301,"num_read_kb":279177,"num_write":237,"num_write_kb":96948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"2.f","version":"557'465","reported_seq":2849,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-11T03:46:21.317201+0000","last_change":"2024-09-11T03:46:21.317513+0000","last_active":"2024-09-11T03:46:21.317201+0000","last_peered":"2024-09-11T03:46:21.317201+0000","last_clean":"2024-09-11T03:46:21.317201+0000","last_became_active":"2024-09-11T03:38:30.031486+0000","last_became_peered":"2024-09-11T03:38:30.031486+0000","last_unstale":"2024-09-11T03:46:21.317201+0000","last_undegraded":"2024-09-11T03:46:21.317201+0000","last_fullsized":"2024-09-11T03:46:21.317201+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'434","last_scrub_stamp":"2024-09-11T03:45:42.488889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:45:42.488889+0000","objects_scrubbed":3,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:11.318213+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0036101940000000002,"stat_sum":{"num_bytes":10571999,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185,"num_read_kb":157471,"num_write":157,"num_write_kb":58279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"d0","length":"2"},{"start":"da","length":"1"},{"start":"e1","length":"1"},{"start":"e6","length":"2"}]},{"pgid":"1.c","version":"20'92","reported_seq":171,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-09-11T03:46:20.316528+0000","last_change":"2024-09-11T03:46:15.978402+0000","last_active":"2024-09-11T03:46:20.316528+0000","last_peered":"2024-09-11T03:46:20.316528+0000","last_clean":"2024-09-11T03:46:20.316528+0000","last_became_active":"2024-09-11T03:46:15.974786+0000","last_became_peered":"2024-09-11T03:46:15.974786+0000","last_unstale":"2024-09-11T03:46:20.316528+0000","last_undegraded":"2024-09-11T03:46:20.316528+0000","last_fullsized":"2024-09-11T03:46:20.316528+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:10.103870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:10.103870+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":142286428,"num_objects":62,"num_object_clones":26,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":6,"num_read":2951,"num_read_kb":2867668,"num_write":2054,"num_write_kb":786603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":317,"num_bytes_recovered":83118490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":281980928,"data_stored":335215991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10795,"ondisk_log_size":10795,"up":78,"acting":78,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":603786,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":363743,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285678,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65944,"kb_used_data":26436,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93679208,"statfs":{"total":95995035648,"available":95927508992,"internally_reserved":0,"allocated":27070464,"data_stored":27658720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514610,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60668,"kb_used_data":25312,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93684484,"statfs":{"total":95995035648,"available":95932911616,"internally_reserved":0,"allocated":25919488,"data_stored":26895865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743543,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57836,"kb_used_data":30804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93687316,"statfs":{"total":95995035648,"available":95935811584,"internally_reserved":0,"allocated":31543296,"data_stored":35092743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037887,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52864,"kb_used_data":25828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93692288,"statfs":{"total":95995035648,"available":95940902912,"internally_reserved":0,"allocated":26447872,"data_stored":25841974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299522,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60956,"kb_used_data":29764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93684196,"statfs":{"total":95995035648,"available":95932616704,"internally_reserved":0,"allocated":30478336,"data_stored":35908794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528455,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":55760,"kb_used_data":24572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93689392,"statfs":{"total":95995035648,"available":95937937408,"internally_reserved":0,"allocated":25161728,"data_stored":25437333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757389,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58684,"kb_used_data":19172,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93686468,"statfs":{"total":95995035648,"available":95934943232,"internally_reserved":0,"allocated":19632128,"data_stored":19528548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986322,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56028,"kb_used_data":28996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93689124,"statfs":{"total":95995035648,"available":95937662976,"internally_reserved":0,"allocated":29691904,"data_stored":27925846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105288,"kb_used_data":51884,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93639864,"statfs":{"total":95995035648,"available":95887220736,"internally_reserved":0,"allocated":53129216,"data_stored":61843690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411484,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68016,"kb_used_data":28508,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93677136,"statfs":{"total":95995035648,"available":95925387264,"internally_reserved":0,"allocated":29192192,"data_stored":34215627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575010,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52996,"kb_used_data":25960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93692156,"statfs":{"total":95995035648,"available":95940767744,"internally_reserved":0,"allocated":26583040,"data_stored":27808379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738535,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77228,"kb_used_data":33556,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93667924,"statfs":{"total":95995035648,"available":95915954176,"internally_reserved":0,"allocated":34361344,"data_stored":35102406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29167616,"data_stored":32447348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19816448,"data_stored":23466036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":22245376,"data_stored":29688514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":46059520,"data_stored":57190145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":22937600,"data_stored":23591734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":13979648,"data_stored":16302916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":19009536,"data_stored":21709693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24838144,"data_stored":32691393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":19681280,"data_stored":21499631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24813568,"data_stored":30775288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":19128320,"data_stored":22536916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20303872,"data_stored":23316377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:46:23.951 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-11T03:46:23.952 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 36 2024-09-11T03:46:23.964 INFO:tasks.rados.rados.2.smithi175.stdout:2111: done (4 left) 2024-09-11T03:46:23.964 INFO:tasks.rados.rados.2.smithi175.stdout:2112: done (3 left) 2024-09-11T03:46:23.964 INFO:tasks.rados.rados.2.smithi175.stdout:2113: done (2 left) 2024-09-11T03:46:23.964 INFO:tasks.rados.rados.2.smithi175.stdout:2114: done (1 left) 2024-09-11T03:46:23.964 INFO:tasks.rados.rados.2.smithi175.stdout:2116: done (0 left) 2024-09-11T03:46:23.964 INFO:tasks.rados.rados.2.smithi175.stdout:2117: snap_remove snap 230 2024-09-11T03:46:24.015 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:24.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:24.037 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:24.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:24.138 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:24.156 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:46:24.156 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:46:24.158 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:24.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:24.174 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:46:24.175 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:46:24.191 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:46:24.191 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:46:24.209 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:46:24.209 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:46:24.226 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:46:24.226 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:46:24.227 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:46:24.228 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:46:24.240 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:24.244 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:46:24.244 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:46:24.260 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:24.262 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:46:24.263 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:46:24.281 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:24.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:24.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:24.282 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:46:24.283 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:46:24.299 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:46:24.299 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:46:24.316 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:46:24.316 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:46:24.333 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:46:24.334 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:46:24.350 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:46:24.351 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:46:24.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:24.404 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:24.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:24.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:24.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:24 smithi121 ceph-mon[30729]: pgmap v1002: 57 pgs: 57 active+clean; 136 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.4 MiB/s wr, 13 op/s; 5.1 MiB/s, 34 objects/s recovering 2024-09-11T03:46:24.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:24 smithi121 ceph-mon[30729]: from='client.16335 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:46:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:24 smithi121 ceph-mon[30729]: osdmap e566: 12 total, 12 up, 12 in 2024-09-11T03:46:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:24 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3994829294' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:24.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:24 smithi084 ceph-mon[27254]: pgmap v1002: 57 pgs: 57 active+clean; 136 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.4 MiB/s wr, 13 op/s; 5.1 MiB/s, 34 objects/s recovering 2024-09-11T03:46:24.493 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:24 smithi084 ceph-mon[27254]: from='client.16335 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:46:24.493 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:24 smithi084 ceph-mon[27254]: osdmap e566: 12 total, 12 up, 12 in 2024-09-11T03:46:24.493 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:24 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3994829294' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:24.506 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:24.532 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:24.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:24.533 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:24.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:24.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:24.657 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:24.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:24.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:24 smithi175 ceph-mon[30894]: pgmap v1002: 57 pgs: 57 active+clean; 136 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.4 MiB/s wr, 13 op/s; 5.1 MiB/s, 34 objects/s recovering 2024-09-11T03:46:24.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:24 smithi175 ceph-mon[30894]: from='client.16335 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:46:24.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:24 smithi175 ceph-mon[30894]: osdmap e566: 12 total, 12 up, 12 in 2024-09-11T03:46:24.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:24 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3994829294' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:24.758 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:24.784 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:24.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:24.785 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:24.806 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:46:24.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T03:46:24.886 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:24.939 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:24.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:24.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:24.967 INFO:tasks.rados.rados.2.smithi175.stdout:2117: done (0 left) 2024-09-11T03:46:24.967 INFO:tasks.rados.rados.2.smithi175.stdout:2118: write (excl) oid 31 current snap is 235 2024-09-11T03:46:24.967 INFO:tasks.rados.rados.2.smithi175.stdout:2118: seq_num 719 ranges {484233=641213,1707612=376621} 2024-09-11T03:46:24.973 INFO:tasks.rados.rados.2.smithi175.stdout:2118: writing smithi17557870-31 from 484233 to 1125446 tid 1 2024-09-11T03:46:24.978 INFO:tasks.rados.rados.2.smithi175.stdout:2118: writing smithi17557870-31 from 1707612 to 2084233 tid 2 2024-09-11T03:46:24.978 INFO:tasks.rados.rados.2.smithi175.stdout:2119: read oid 12 snap 231 2024-09-11T03:46:24.978 INFO:tasks.rados.rados.2.smithi175.stdout:2119: expect (ObjNum 688 snap 226 seq_num 688) 2024-09-11T03:46:24.978 INFO:tasks.rados.rados.2.smithi175.stdout:2120: copy_from oid 29 from oid 9 current snap is 235 2024-09-11T03:46:24.978 INFO:tasks.rados.rados.2.smithi175.stdout:2121: snap_create 2024-09-11T03:46:24.980 INFO:tasks.rados.rados.2.smithi175.stdout:2118: finishing write tid 1 to smithi17557870-31 2024-09-11T03:46:24.981 INFO:tasks.rados.rados.2.smithi175.stdout:2118: finishing write tid 2 to smithi17557870-31 2024-09-11T03:46:24.981 INFO:tasks.rados.rados.2.smithi175.stdout:2118: finishing write tid 3 to smithi17557870-31 2024-09-11T03:46:24.981 INFO:tasks.rados.rados.2.smithi175.stdout:2118: finishing write tid 4 to smithi17557870-31 2024-09-11T03:46:24.981 INFO:tasks.rados.rados.2.smithi175.stdout:2118: oid 31 updating version 0 to 873 2024-09-11T03:46:24.982 INFO:tasks.rados.rados.2.smithi175.stdout:2118: oid 31 updating version 873 to 874 2024-09-11T03:46:24.982 INFO:tasks.rados.rados.2.smithi175.stdout:2118: oid 31 version 874 is already newer than 870 2024-09-11T03:46:24.982 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 874 (ObjNum 719 snap 235 seq_num 719) dirty exists 2024-09-11T03:46:24.982 INFO:tasks.rados.rados.2.smithi175.stdout:2118: left oid 31 (ObjNum 719 snap 235 seq_num 719) 2024-09-11T03:46:24.984 INFO:tasks.rados.rados.2.smithi175.stdout:2119: expect (ObjNum 688 snap 226 seq_num 688) 2024-09-11T03:46:25.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:25.066 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:25.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:25.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:25.124 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:46:25.142 INFO:tasks.rados.rados.2.smithi175.stdout:2120: finishing copy_from to smithi17557870-29 2024-09-11T03:46:25.142 INFO:tasks.rados.rados.2.smithi175.stdout:2120: got expected ENOENT (src dne) 2024-09-11T03:46:25.142 INFO:tasks.rados.rados.2.smithi175.stdout:2120: finishing copy_from racing read to smithi17557870-29 2024-09-11T03:46:25.169 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:25.190 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:25.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:25.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:25.292 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:25.314 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:25.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:25.315 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:25.416 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:25.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:25.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:25.538 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:25.560 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:25.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:25.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:25.593 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:46:25.593 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:46:25.615 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:25.661 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:25.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:25.683 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:25.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:25.783 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:25.804 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:25.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:25.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:25.906 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:25.947 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:25.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:25.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:25.971 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 2 pg_num to 36 2024-09-11T03:46:25.974 INFO:tasks.rados.rados.2.smithi175.stdout:2118: done (3 left) 2024-09-11T03:46:25.974 INFO:tasks.rados.rados.2.smithi175.stdout:2119: done (2 left) 2024-09-11T03:46:25.974 INFO:tasks.rados.rados.2.smithi175.stdout:2120: done (1 left) 2024-09-11T03:46:25.975 INFO:tasks.rados.rados.2.smithi175.stdout:2121: done (0 left) 2024-09-11T03:46:25.975 INFO:tasks.rados.rados.2.smithi175.stdout:2122: rollback oid 31 current snap is 236 2024-09-11T03:46:25.975 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 31 to 222 2024-09-11T03:46:25.975 INFO:tasks.rados.rados.2.smithi175.stdout:2123: write (excl) oid 35 current snap is 236 2024-09-11T03:46:25.975 INFO:tasks.rados.rados.2.smithi175.stdout:2123: seq_num 720 ranges {501040=716462,1766094=334946} 2024-09-11T03:46:25.981 INFO:tasks.rados.rados.2.smithi175.stdout:2123: writing smithi17557870-35 from 501040 to 1217502 tid 1 2024-09-11T03:46:25.986 INFO:tasks.rados.rados.2.smithi175.stdout:2123: writing smithi17557870-35 from 1766094 to 2101040 tid 2 2024-09-11T03:46:25.986 INFO:tasks.rados.rados.2.smithi175.stdout:2124: write oid 32 current snap is 236 2024-09-11T03:46:25.986 INFO:tasks.rados.rados.2.smithi175.stdout:2122: finishing rollback tid 0 to smithi17557870-31 2024-09-11T03:46:25.987 INFO:tasks.rados.rados.2.smithi175.stdout:2122: finishing rollback tid 1 to smithi17557870-31 2024-09-11T03:46:25.987 INFO:tasks.rados.rados.2.smithi175.stdout:2124: seq_num 721 ranges {517847=791711,2108223=9624} 2024-09-11T03:46:25.993 INFO:tasks.rados.rados.2.smithi175.stdout:2124: writing smithi17557870-32 from 517847 to 1309558 tid 1 2024-09-11T03:46:25.998 INFO:tasks.rados.rados.2.smithi175.stdout:2124: writing smithi17557870-32 from 2108223 to 2117847 tid 2 2024-09-11T03:46:25.998 INFO:tasks.rados.rados.2.smithi175.stdout:2125: read oid 47 snap 222 2024-09-11T03:46:25.998 INFO:tasks.rados.rados.2.smithi175.stdout:2125: expect deleted 2024-09-11T03:46:25.998 INFO:tasks.rados.rados.2.smithi175.stdout:2126: write (excl) oid 36 current snap is 236 2024-09-11T03:46:25.999 INFO:tasks.rados.rados.2.smithi175.stdout:2122: finishing rollback tid 2 to smithi17557870-31 2024-09-11T03:46:25.999 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 878 (ObjNum 495 snap 143 seq_num 495) dirty exists 2024-09-11T03:46:25.999 INFO:tasks.rados.rados.2.smithi175.stdout:2126: seq_num 722 ranges {534654=466960,1766705=367949} 2024-09-11T03:46:26.003 INFO:tasks.rados.rados.2.smithi175.stdout:2126: writing smithi17557870-36 from 534654 to 1001614 tid 1 2024-09-11T03:46:26.010 INFO:tasks.rados.rados.2.smithi175.stdout:2126: writing smithi17557870-36 from 1766705 to 2134654 tid 2 2024-09-11T03:46:26.010 INFO:tasks.rados.rados.2.smithi175.stdout:2122: done (4 left) 2024-09-11T03:46:26.010 INFO:tasks.rados.rados.2.smithi175.stdout:2127: snap_create 2024-09-11T03:46:26.010 INFO:tasks.rados.rados.2.smithi175.stdout:2123: finishing write tid 1 to smithi17557870-35 2024-09-11T03:46:26.011 INFO:tasks.rados.rados.2.smithi175.stdout:2123: finishing write tid 2 to smithi17557870-35 2024-09-11T03:46:26.011 INFO:tasks.rados.rados.2.smithi175.stdout:2123: finishing write tid 3 to smithi17557870-35 2024-09-11T03:46:26.011 INFO:tasks.rados.rados.2.smithi175.stdout:2123: finishing write tid 4 to smithi17557870-35 2024-09-11T03:46:26.011 INFO:tasks.rados.rados.2.smithi175.stdout:2123: oid 35 updating version 0 to 668 2024-09-11T03:46:26.011 INFO:tasks.rados.rados.2.smithi175.stdout:2123: oid 35 updating version 668 to 669 2024-09-11T03:46:26.011 INFO:tasks.rados.rados.2.smithi175.stdout:2123: oid 35 updating version 669 to 670 2024-09-11T03:46:26.011 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 670 (ObjNum 720 snap 236 seq_num 720) dirty exists 2024-09-11T03:46:26.011 INFO:tasks.rados.rados.2.smithi175.stdout:2123: left oid 35 (ObjNum 720 snap 236 seq_num 720) 2024-09-11T03:46:26.011 INFO:tasks.rados.rados.2.smithi175.stdout:2124: finishing write tid 1 to smithi17557870-32 2024-09-11T03:46:26.011 INFO:tasks.rados.rados.2.smithi175.stdout:2124: finishing write tid 2 to smithi17557870-32 2024-09-11T03:46:26.011 INFO:tasks.rados.rados.2.smithi175.stdout:2124: finishing write tid 3 to smithi17557870-32 2024-09-11T03:46:26.011 INFO:tasks.rados.rados.2.smithi175.stdout:2124: finishing write tid 4 to smithi17557870-32 2024-09-11T03:46:26.011 INFO:tasks.rados.rados.2.smithi175.stdout:2124: oid 32 updating version 0 to 645 2024-09-11T03:46:26.012 INFO:tasks.rados.rados.2.smithi175.stdout:2124: oid 32 updating version 645 to 646 2024-09-11T03:46:26.012 INFO:tasks.rados.rados.2.smithi175.stdout:2124: oid 32 version 646 is already newer than 644 2024-09-11T03:46:26.012 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 646 (ObjNum 721 snap 236 seq_num 721) dirty exists 2024-09-11T03:46:26.012 INFO:tasks.rados.rados.2.smithi175.stdout:2124: left oid 32 (ObjNum 721 snap 236 seq_num 721) 2024-09-11T03:46:26.012 INFO:tasks.rados.rados.2.smithi175.stdout:2126: finishing write tid 1 to smithi17557870-36 2024-09-11T03:46:26.012 INFO:tasks.rados.rados.2.smithi175.stdout:2126: finishing write tid 2 to smithi17557870-36 2024-09-11T03:46:26.013 INFO:tasks.rados.rados.2.smithi175.stdout:2126: finishing write tid 3 to smithi17557870-36 2024-09-11T03:46:26.013 INFO:tasks.rados.rados.2.smithi175.stdout:2126: finishing write tid 4 to smithi17557870-36 2024-09-11T03:46:26.013 INFO:tasks.rados.rados.2.smithi175.stdout:2126: oid 36 updating version 0 to 360 2024-09-11T03:46:26.013 INFO:tasks.rados.rados.2.smithi175.stdout:2126: oid 36 version 360 is already newer than 358 2024-09-11T03:46:26.013 INFO:tasks.rados.rados.2.smithi175.stdout:2126: oid 36 version 360 is already newer than 359 2024-09-11T03:46:26.013 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 360 (ObjNum 722 snap 236 seq_num 722) dirty exists 2024-09-11T03:46:26.013 INFO:tasks.rados.rados.2.smithi175.stdout:2126: left oid 36 (ObjNum 722 snap 236 seq_num 722) 2024-09-11T03:46:26.048 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:26.070 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:26.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:26.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:26.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:25 smithi084 ceph-mon[27254]: pgmap v1005: 57 pgs: 57 active+clean; 139 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.9 MiB/s wr, 16 op/s; 5.1 MiB/s, 34 objects/s recovering 2024-09-11T03:46:26.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:25 smithi084 ceph-mon[27254]: osdmap e567: 12 total, 12 up, 12 in 2024-09-11T03:46:26.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:25 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2481167991' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"} : dispatch 2024-09-11T03:46:26.172 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:26.195 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:26.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:26.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:26.214 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:25 smithi121 ceph-mon[30729]: pgmap v1005: 57 pgs: 57 active+clean; 139 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.9 MiB/s wr, 16 op/s; 5.1 MiB/s, 34 objects/s recovering 2024-09-11T03:46:26.215 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:25 smithi121 ceph-mon[30729]: osdmap e567: 12 total, 12 up, 12 in 2024-09-11T03:46:26.215 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:25 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2481167991' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"} : dispatch 2024-09-11T03:46:26.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T03:46:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:25 smithi175 ceph-mon[30894]: pgmap v1005: 57 pgs: 57 active+clean; 139 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.9 MiB/s wr, 16 op/s; 5.1 MiB/s, 34 objects/s recovering 2024-09-11T03:46:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:25 smithi175 ceph-mon[30894]: osdmap e567: 12 total, 12 up, 12 in 2024-09-11T03:46:26.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:25 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2481167991' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"} : dispatch 2024-09-11T03:46:26.297 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:26.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:26.319 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:26.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:26.420 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:26.441 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:26.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:26.495 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:46:26.543 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:26.564 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:26.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:26.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:26.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:26.688 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:26.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:26.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:26.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:26.811 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:26.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:26.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:26.913 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:26.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:26.934 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:26.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:26.966 INFO:tasks.rados.rados.2.smithi175.stdout:2123: done (4 left) 2024-09-11T03:46:26.966 INFO:tasks.rados.rados.2.smithi175.stdout:2124: done (3 left) 2024-09-11T03:46:26.966 INFO:tasks.rados.rados.2.smithi175.stdout:2125: done (2 left) 2024-09-11T03:46:26.966 INFO:tasks.rados.rados.2.smithi175.stdout:2126: done (1 left) 2024-09-11T03:46:26.966 INFO:tasks.rados.rados.2.smithi175.stdout:2127: done (0 left) 2024-09-11T03:46:26.966 INFO:tasks.rados.rados.2.smithi175.stdout:2128: delete oid 46 current snap is 237 2024-09-11T03:46:26.983 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:46:26.983 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:46:27.036 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:27.055 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:46:27.059 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:27.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:27.060 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2481167991' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-09-11T03:46:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:26 smithi084 ceph-mon[27254]: osdmap e568: 12 total, 12 up, 12 in 2024-09-11T03:46:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1332227646' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-11T03:46:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:26 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:27.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1332227646' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:27.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-11T03:46:27.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:26 smithi084 ceph-mon[27254]: osdmap e569: 12 total, 12 up, 12 in 2024-09-11T03:46:27.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:27.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2481167991' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-09-11T03:46:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:26 smithi121 ceph-mon[30729]: osdmap e568: 12 total, 12 up, 12 in 2024-09-11T03:46:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1332227646' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-11T03:46:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:26 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1332227646' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-11T03:46:27.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:26 smithi121 ceph-mon[30729]: osdmap e569: 12 total, 12 up, 12 in 2024-09-11T03:46:27.182 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:27.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:27.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:27.208 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:46:27.285 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:27.306 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:27.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:27.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:27.408 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:27.429 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:27.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2481167991' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-09-11T03:46:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:26 smithi175 ceph-mon[30894]: osdmap e568: 12 total, 12 up, 12 in 2024-09-11T03:46:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1332227646' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-11T03:46:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:26 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1332227646' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-11T03:46:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:26 smithi175 ceph-mon[30894]: osdmap e569: 12 total, 12 up, 12 in 2024-09-11T03:46:27.531 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:27.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:27.553 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:27.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:27.655 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:27.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:27.682 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:27.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:27.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T03:46:27.783 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:27.804 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:27.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:27.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:27.906 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:27.927 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:27.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:27.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:27.967 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:46:27.972 INFO:tasks.rados.rados.2.smithi175.stdout:2128: done (0 left) 2024-09-11T03:46:27.972 INFO:tasks.rados.rados.2.smithi175.stdout:2129: snap_remove snap 233 2024-09-11T03:46:28.004 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:28.029 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:28.050 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:28.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:28.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:28.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:27 smithi084 ceph-mon[27254]: pgmap v1008: 57 pgs: 57 active+clean; 143 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2024-09-11T03:46:28.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:27 smithi084 ceph-mon[27254]: osdmap e570: 12 total, 12 up, 12 in 2024-09-11T03:46:28.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:28.173 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:28.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:28.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:28.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:28.296 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:28.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:28.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:28.336 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:27 smithi121 ceph-mon[30729]: pgmap v1008: 57 pgs: 57 active+clean; 143 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2024-09-11T03:46:28.336 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:27 smithi121 ceph-mon[30729]: osdmap e570: 12 total, 12 up, 12 in 2024-09-11T03:46:28.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:28.419 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:28.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:28.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:27 smithi175 ceph-mon[30894]: pgmap v1008: 57 pgs: 57 active+clean; 143 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2024-09-11T03:46:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:27 smithi175 ceph-mon[30894]: osdmap e570: 12 total, 12 up, 12 in 2024-09-11T03:46:28.510 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:46:28.510 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:46:28.521 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:28.527 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:46:28.527 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:46:28.543 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:46:28.543 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:46:28.546 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:28.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:28.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:28.559 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:46:28.559 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:46:28.575 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:46:28.575 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:46:28.591 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:46:28.591 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:46:28.592 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:46:28.593 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:46:28.609 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:46:28.609 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:46:28.625 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:46:28.625 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:46:28.642 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:46:28.643 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:46:28.648 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:28.660 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:46:28.660 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:46:28.671 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:28.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:28.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:28.678 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:46:28.678 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:46:28.695 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:46:28.695 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:46:28.772 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:28.799 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:28.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:28.800 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:28.901 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:28.923 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:28.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:28.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:28.973 INFO:tasks.rados.rados.2.smithi175.stdout:2129: done (0 left) 2024-09-11T03:46:28.973 INFO:tasks.rados.rados.2.smithi175.stdout:2130: write oid 29 current snap is 237 2024-09-11T03:46:28.973 INFO:tasks.rados.rados.2.smithi175.stdout:2130: seq_num 724 ranges {568268=733811,2000022=168246} 2024-09-11T03:46:28.977 INFO:tasks.rados.rados.2.smithi175.stdout:2130: writing smithi17557870-29 from 568268 to 1302079 tid 1 2024-09-11T03:46:28.982 INFO:tasks.rados.rados.2.smithi175.stdout:2130: writing smithi17557870-29 from 2000022 to 2168268 tid 2 2024-09-11T03:46:28.982 INFO:tasks.rados.rados.2.smithi175.stdout:2130: finishing write tid 1 to smithi17557870-29 2024-09-11T03:46:28.982 INFO:tasks.rados.rados.2.smithi175.stdout:2131: write oid 45 current snap is 237 2024-09-11T03:46:28.982 INFO:tasks.rados.rados.2.smithi175.stdout:2131: seq_num 725 ranges {585075=409060,1542151=642924} 2024-09-11T03:46:28.987 INFO:tasks.rados.rados.2.smithi175.stdout:2131: writing smithi17557870-45 from 585075 to 994135 tid 1 2024-09-11T03:46:28.995 INFO:tasks.rados.rados.2.smithi175.stdout:2131: writing smithi17557870-45 from 1542151 to 2185075 tid 2 2024-09-11T03:46:28.995 INFO:tasks.rados.rados.2.smithi175.stdout:2130: finishing write tid 2 to smithi17557870-29 2024-09-11T03:46:28.995 INFO:tasks.rados.rados.2.smithi175.stdout:2130: finishing write tid 3 to smithi17557870-29 2024-09-11T03:46:28.995 INFO:tasks.rados.rados.2.smithi175.stdout:2130: finishing write tid 4 to smithi17557870-29 2024-09-11T03:46:28.995 INFO:tasks.rados.rados.2.smithi175.stdout:2130: oid 29 updating version 0 to 808 2024-09-11T03:46:28.995 INFO:tasks.rados.rados.2.smithi175.stdout:2130: oid 29 version 808 is already newer than 807 2024-09-11T03:46:28.995 INFO:tasks.rados.rados.2.smithi175.stdout:2130: oid 29 version 808 is already newer than 806 2024-09-11T03:46:28.995 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 808 (ObjNum 724 snap 237 seq_num 724) dirty exists 2024-09-11T03:46:28.995 INFO:tasks.rados.rados.2.smithi175.stdout:2130: left oid 29 (ObjNum 724 snap 237 seq_num 724) 2024-09-11T03:46:28.996 INFO:tasks.rados.rados.2.smithi175.stdout:2131: finishing write tid 1 to smithi17557870-45 2024-09-11T03:46:28.996 INFO:tasks.rados.rados.2.smithi175.stdout:2132: read oid 41 snap -1 2024-09-11T03:46:28.996 INFO:tasks.rados.rados.2.smithi175.stdout:2132: expect (ObjNum 609 snap 191 seq_num 609) 2024-09-11T03:46:28.996 INFO:tasks.rados.rados.2.smithi175.stdout:2130: done (2 left) 2024-09-11T03:46:28.996 INFO:tasks.rados.rados.2.smithi175.stdout:2133: read oid 26 snap -1 2024-09-11T03:46:28.996 INFO:tasks.rados.rados.2.smithi175.stdout:2133: expect (ObjNum 718 snap 234 seq_num 718) 2024-09-11T03:46:28.996 INFO:tasks.rados.rados.2.smithi175.stdout:2134: read oid 19 snap 236 2024-09-11T03:46:28.996 INFO:tasks.rados.rados.2.smithi175.stdout:2134: expect deleted 2024-09-11T03:46:28.997 INFO:tasks.rados.rados.2.smithi175.stdout:2135: snap_create 2024-09-11T03:46:28.998 INFO:tasks.rados.rados.2.smithi175.stdout:2131: finishing write tid 2 to smithi17557870-45 2024-09-11T03:46:28.998 INFO:tasks.rados.rados.2.smithi175.stdout:2131: finishing write tid 3 to smithi17557870-45 2024-09-11T03:46:28.999 INFO:tasks.rados.rados.2.smithi175.stdout:2131: finishing write tid 4 to smithi17557870-45 2024-09-11T03:46:28.999 INFO:tasks.rados.rados.2.smithi175.stdout:2131: oid 45 updating version 0 to 672 2024-09-11T03:46:28.999 INFO:tasks.rados.rados.2.smithi175.stdout:2131: oid 45 version 672 is already newer than 671 2024-09-11T03:46:28.999 INFO:tasks.rados.rados.2.smithi175.stdout:2131: oid 45 updating version 672 to 673 2024-09-11T03:46:28.999 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 673 (ObjNum 725 snap 237 seq_num 725) dirty exists 2024-09-11T03:46:29.000 INFO:tasks.rados.rados.2.smithi175.stdout:2131: left oid 45 (ObjNum 725 snap 237 seq_num 725) 2024-09-11T03:46:29.001 INFO:tasks.rados.rados.2.smithi175.stdout:2133: expect (ObjNum 718 snap 234 seq_num 718) 2024-09-11T03:46:29.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:29.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:29.047 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:29.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:29.149 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:29.170 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:29.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:29.171 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:29.245 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:28 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:46:29.245 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:28 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:46:29.245 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:28 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:46:29.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:28 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2395303507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:29.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:28 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-11T03:46:29.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:28 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-11T03:46:29.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:28 smithi121 ceph-mon[30729]: osdmap e571: 12 total, 12 up, 12 in 2024-09-11T03:46:29.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T03:46:29.273 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:29.294 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:29.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:29.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:29.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:28 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:46:29.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:28 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:46:29.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:28 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:46:29.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:28 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2395303507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:29.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:28 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-11T03:46:29.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:28 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-11T03:46:29.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:28 smithi084 ceph-mon[27254]: osdmap e571: 12 total, 12 up, 12 in 2024-09-11T03:46:29.396 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:29.418 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:29.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:29.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:28 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:46:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:28 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:46:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:28 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:46:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:28 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2395303507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:28 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-09-11T03:46:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:28 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-11T03:46:29.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:28 smithi175 ceph-mon[30894]: osdmap e571: 12 total, 12 up, 12 in 2024-09-11T03:46:29.516 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:46:29.521 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:29.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:29.544 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:29.544 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:29.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:29.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:29.668 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:29.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:29.769 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:29.791 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:29.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:29.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:29.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:46:29.893 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:29.915 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:29.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:29.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:29.968 INFO:tasks.rados.rados.2.smithi175.stdout:2131: done (4 left) 2024-09-11T03:46:29.968 INFO:tasks.rados.rados.2.smithi175.stdout:2133: done (3 left) 2024-09-11T03:46:29.968 INFO:tasks.rados.rados.2.smithi175.stdout:2134: done (2 left) 2024-09-11T03:46:29.968 INFO:tasks.rados.rados.2.smithi175.stdout:2135: done (1 left) 2024-09-11T03:46:29.968 INFO:tasks.rados.rados.2.smithi175.stdout:2136: rollback oid 27 current snap is 238 2024-09-11T03:46:29.969 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 27 to 234 2024-09-11T03:46:29.969 INFO:tasks.rados.rados.2.smithi175.stdout:2137: copy_from oid 34 from oid 25 current snap is 238 2024-09-11T03:46:29.969 INFO:tasks.rados.rados.2.smithi175.stdout:2138: snap_remove snap 231 2024-09-11T03:46:30.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:30.026 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:46:30.027 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:46:30.027 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:46:30.027 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:46:30.042 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:30.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:30.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:30.123 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:30.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:30.165 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:30.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:30.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:30.243 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:29 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:46:30.243 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:29 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:46:30.243 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:29 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:46:30.243 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:29 smithi084 ceph-mon[27254]: pgmap v1011: 67 pgs: 2 peering, 8 unknown, 57 active+clean; 146 MiB data, 745 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 1.5 MiB/s wr, 14 op/s 2024-09-11T03:46:30.243 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:29 smithi084 ceph-mon[27254]: 2.f scrub starts 2024-09-11T03:46:30.244 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:29 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:46:30.244 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:29 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:46:30.244 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:29 smithi084 ceph-mon[27254]: osdmap e572: 12 total, 12 up, 12 in 2024-09-11T03:46:30.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:30.290 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:30.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:30.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:30.392 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:30.414 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:30.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:30.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:29 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:46:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:29 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:46:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:29 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:46:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:29 smithi121 ceph-mon[30729]: pgmap v1011: 67 pgs: 2 peering, 8 unknown, 57 active+clean; 146 MiB data, 745 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 1.5 MiB/s wr, 14 op/s 2024-09-11T03:46:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:29 smithi121 ceph-mon[30729]: 2.f scrub starts 2024-09-11T03:46:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:29 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:46:30.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:29 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:46:30.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:29 smithi121 ceph-mon[30729]: osdmap e572: 12 total, 12 up, 12 in 2024-09-11T03:46:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:29 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:46:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:29 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:46:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:29 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:46:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:29 smithi175 ceph-mon[30894]: pgmap v1011: 67 pgs: 2 peering, 8 unknown, 57 active+clean; 146 MiB data, 745 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 1.5 MiB/s wr, 14 op/s 2024-09-11T03:46:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:29 smithi175 ceph-mon[30894]: 2.f scrub starts 2024-09-11T03:46:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:29 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:46:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:29 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:46:30.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:29 smithi175 ceph-mon[30894]: osdmap e572: 12 total, 12 up, 12 in 2024-09-11T03:46:30.516 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:30.537 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:30.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:30.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:30.662 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:30.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:30.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:30.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:30.786 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:30.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:30.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:30.888 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:30.906 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T03:46:30.914 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:30.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:30.965 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:46:30.970 INFO:tasks.rados.rados.2.smithi175.stdout:2138: done (3 left) 2024-09-11T03:46:30.970 INFO:tasks.rados.rados.2.smithi175.stdout:2139: read oid 32 snap -1 2024-09-11T03:46:30.970 INFO:tasks.rados.rados.2.smithi175.stdout:2139: expect (ObjNum 721 snap 236 seq_num 721) 2024-09-11T03:46:30.970 INFO:tasks.rados.rados.2.smithi175.stdout:2140: read oid 28 snap -1 2024-09-11T03:46:30.970 INFO:tasks.rados.rados.2.smithi175.stdout:2140: expect (ObjNum 609 snap 191 seq_num 609) 2024-09-11T03:46:30.970 INFO:tasks.rados.rados.2.smithi175.stdout:2141: write (excl) oid 10 current snap is 238 2024-09-11T03:46:30.970 INFO:tasks.rados.rados.2.smithi175.stdout:2136: finishing rollback tid 1 to smithi17557870-27 2024-09-11T03:46:30.970 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T03:46:30.970 INFO:tasks.rados.rados.2.smithi175.stdout:2141: seq_num 726 ranges {601882=484309,1600633=524904} 2024-09-11T03:46:30.976 INFO:tasks.rados.rados.2.smithi175.stdout:2141: writing smithi17557870-10 from 601882 to 1086191 tid 1 2024-09-11T03:46:30.983 INFO:tasks.rados.rados.2.smithi175.stdout:2141: writing smithi17557870-10 from 1600633 to 2125537 tid 2 2024-09-11T03:46:30.983 INFO:tasks.rados.rados.2.smithi175.stdout:2136: done (5 left) 2024-09-11T03:46:30.983 INFO:tasks.rados.rados.2.smithi175.stdout:2142: snap_remove snap 235 2024-09-11T03:46:31.016 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:31.044 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:31.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:31.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:31.146 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:31.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:31.169 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:31.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:31.177 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:46:31.270 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:31.292 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:31.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:31.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:31.297 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:31 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:46:31.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:31 smithi121 ceph-mon[30729]: 2.3 scrub starts 2024-09-11T03:46:31.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:31 smithi121 ceph-mon[30729]: 2.f scrub starts 2024-09-11T03:46:31.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:31 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:46:31.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:31 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:46:31.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:31 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:46:31.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:31 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:46:31.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:31 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3600381953' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:46:31.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:31 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3600381953' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:46:31.298 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:31 smithi121 ceph-mon[30729]: osdmap e573: 12 total, 12 up, 12 in 2024-09-11T03:46:31.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:31 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:46:31.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:31 smithi084 ceph-mon[27254]: 2.3 scrub starts 2024-09-11T03:46:31.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:31 smithi084 ceph-mon[27254]: 2.f scrub starts 2024-09-11T03:46:31.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:31 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:46:31.334 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:31 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:46:31.334 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:31 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:46:31.334 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:31 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:46:31.334 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:31 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3600381953' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:46:31.334 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:31 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3600381953' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:46:31.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:31 smithi084 ceph-mon[27254]: osdmap e573: 12 total, 12 up, 12 in 2024-09-11T03:46:31.378 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:46:31.393 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:31.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:31.416 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:31.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:31 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:46:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:31 smithi175 ceph-mon[30894]: 2.3 scrub starts 2024-09-11T03:46:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:31 smithi175 ceph-mon[30894]: 2.f scrub starts 2024-09-11T03:46:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:31 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:46:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:31 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:46:31.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:31 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:46:31.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:31 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:46:31.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:31 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3600381953' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:46:31.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:31 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3600381953' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:46:31.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:31 smithi175 ceph-mon[30894]: osdmap e573: 12 total, 12 up, 12 in 2024-09-11T03:46:31.518 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:31.539 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:31.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:31.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:31.640 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:46:31.641 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:46:31.641 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:46:31.641 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:46:31.641 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:46:31.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:31.663 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:31.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:31.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:31.765 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:31.786 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:31.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:31.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:31.801 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:31.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:31.909 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:31.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:31.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:31.916 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:46:31.916 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:46:31.916 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T03:46:31.916 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.000000 2024-09-11T03:46:31.916 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 2 0 2024-09-11T03:46:31.966 INFO:tasks.rados.rados.2.smithi175.stdout:2142: done (5 left) 2024-09-11T03:46:31.967 INFO:tasks.rados.rados.2.smithi175.stdout:2143: snap_remove snap 234 2024-09-11T03:46:32.011 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:32.035 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:32.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:32.036 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:32.137 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:32.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:32.161 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:32.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:32.262 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:32.283 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:32.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:32.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:32 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:46:32.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:32 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:46:32.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:32 smithi084 ceph-mon[27254]: pgmap v1014: 67 pgs: 2 peering, 1 remapped+peering, 7 unknown, 57 active+clean; 151 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 7.8 MiB/s wr, 47 op/s 2024-09-11T03:46:32.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:32 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:46:32.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:32 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:46:32.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:32 smithi084 ceph-mon[27254]: 2.a scrub ok 2024-09-11T03:46:32.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:32 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:46:32.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:32 smithi084 ceph-mon[27254]: 2.1 scrub ok 2024-09-11T03:46:32.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:32 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:32.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:32 smithi084 ceph-mon[27254]: osdmap e574: 12 total, 12 up, 12 in 2024-09-11T03:46:32.321 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:46:32.321 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:46:32.335 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:46:32.336 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:46:32.353 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:46:32.353 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:46:32.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:32 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:46:32.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:32 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:46:32.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:32 smithi121 ceph-mon[30729]: pgmap v1014: 67 pgs: 2 peering, 1 remapped+peering, 7 unknown, 57 active+clean; 151 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 7.8 MiB/s wr, 47 op/s 2024-09-11T03:46:32.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:32 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:46:32.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:32 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:46:32.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:32 smithi121 ceph-mon[30729]: 2.a scrub ok 2024-09-11T03:46:32.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:32 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:46:32.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:32 smithi121 ceph-mon[30729]: 2.1 scrub ok 2024-09-11T03:46:32.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:32 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:32.362 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:32 smithi121 ceph-mon[30729]: osdmap e574: 12 total, 12 up, 12 in 2024-09-11T03:46:32.372 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:46:32.372 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:46:32.385 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:32.389 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T03:46:32.390 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:46:32.390 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:46:32.405 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:32.407 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:46:32.407 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:46:32.408 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:32.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:32.424 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:46:32.424 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:46:32.441 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:46:32.441 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:46:32.458 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:46:32.458 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:46:32.475 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:46:32.475 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:46:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:32 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:46:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:32 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:46:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:32 smithi175 ceph-mon[30894]: pgmap v1014: 67 pgs: 2 peering, 1 remapped+peering, 7 unknown, 57 active+clean; 151 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 7.8 MiB/s wr, 47 op/s 2024-09-11T03:46:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:32 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:46:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:32 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:46:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:32 smithi175 ceph-mon[30894]: 2.a scrub ok 2024-09-11T03:46:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:32 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:46:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:32 smithi175 ceph-mon[30894]: 2.1 scrub ok 2024-09-11T03:46:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:32 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:32 smithi175 ceph-mon[30894]: osdmap e574: 12 total, 12 up, 12 in 2024-09-11T03:46:32.491 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:46:32.491 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:46:32.507 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:46:32.508 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:46:32.510 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:32.532 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:32.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:32.533 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:32.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:32.639 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:46:32.657 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:32.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:32.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:32.759 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:32.782 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:32.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:32.884 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:32.906 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:32.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:32.967 INFO:tasks.rados.rados.2.smithi175.stdout:2143: done (5 left) 2024-09-11T03:46:32.967 INFO:tasks.rados.rados.2.smithi175.stdout:2144: snap_remove snap 237 2024-09-11T03:46:33.007 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:33.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:33.030 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:33.030 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:33.090 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:46:33.090 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:46:33.131 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:33.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:33.154 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:33.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:33.255 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:33.278 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:33.279 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:33.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:33 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:46:33.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:33 smithi084 ceph-mon[27254]: 2.0 scrub ok 2024-09-11T03:46:33.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:33 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:46:33.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:33 smithi084 ceph-mon[27254]: 2.d scrub ok 2024-09-11T03:46:33.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:33 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:46:33.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:33 smithi084 ceph-mon[27254]: 2.b scrub ok 2024-09-11T03:46:33.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:33 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:46:33.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:33 smithi084 ceph-mon[27254]: 2.c scrub ok 2024-09-11T03:46:33.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:33 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:46:33.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:33 smithi084 ceph-mon[27254]: 2.e scrub ok 2024-09-11T03:46:33.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:33 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/457905795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:33.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:33 smithi084 ceph-mon[27254]: osdmap e575: 12 total, 12 up, 12 in 2024-09-11T03:46:33.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:33 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:46:33.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:33 smithi121 ceph-mon[30729]: 2.0 scrub ok 2024-09-11T03:46:33.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:33 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:46:33.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:33 smithi121 ceph-mon[30729]: 2.d scrub ok 2024-09-11T03:46:33.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:33 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:46:33.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:33 smithi121 ceph-mon[30729]: 2.b scrub ok 2024-09-11T03:46:33.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:33 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:46:33.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:33 smithi121 ceph-mon[30729]: 2.c scrub ok 2024-09-11T03:46:33.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:33 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:46:33.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:33 smithi121 ceph-mon[30729]: 2.e scrub ok 2024-09-11T03:46:33.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:33 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/457905795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:33.378 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:33 smithi121 ceph-mon[30729]: osdmap e575: 12 total, 12 up, 12 in 2024-09-11T03:46:33.379 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:33.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:33.402 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:33.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:33.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:33 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:46:33.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:33 smithi175 ceph-mon[30894]: 2.0 scrub ok 2024-09-11T03:46:33.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:33 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:46:33.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:33 smithi175 ceph-mon[30894]: 2.d scrub ok 2024-09-11T03:46:33.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:33 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:46:33.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:33 smithi175 ceph-mon[30894]: 2.b scrub ok 2024-09-11T03:46:33.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:33 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:46:33.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:33 smithi175 ceph-mon[30894]: 2.c scrub ok 2024-09-11T03:46:33.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:33 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:46:33.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:33 smithi175 ceph-mon[30894]: 2.e scrub ok 2024-09-11T03:46:33.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:33 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/457905795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:33.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:33 smithi175 ceph-mon[30894]: osdmap e575: 12 total, 12 up, 12 in 2024-09-11T03:46:33.503 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:33.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:33.525 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:33.526 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:33.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:33.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:33.649 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:33.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:33.744 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:46:33.750 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:33.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T03:46:33.772 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:33.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:33.773 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:33.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:33.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:33.897 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:33.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:33.970 INFO:tasks.rados.rados.2.smithi175.stdout:2144: done (5 left) 2024-09-11T03:46:33.970 INFO:tasks.rados.rados.2.smithi175.stdout:2145: delete oid 29 current snap is 238 2024-09-11T03:46:33.970 INFO:tasks.rados.rados.2.smithi175.stdout:2140: expect (ObjNum 609 snap 191 seq_num 609) 2024-09-11T03:46:33.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:34.009 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:46:34.013 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:34.019 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:34.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:34.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:34.075 INFO:tasks.rados.rados.2.smithi175.stdout:2141: finishing write tid 1 to smithi17557870-10 2024-09-11T03:46:34.076 INFO:tasks.rados.rados.2.smithi175.stdout:2141: finishing write tid 2 to smithi17557870-10 2024-09-11T03:46:34.076 INFO:tasks.rados.rados.2.smithi175.stdout:2141: finishing write tid 3 to smithi17557870-10 2024-09-11T03:46:34.076 INFO:tasks.rados.rados.2.smithi175.stdout:2141: finishing write tid 4 to smithi17557870-10 2024-09-11T03:46:34.076 INFO:tasks.rados.rados.2.smithi175.stdout:2141: oid 10 updating version 0 to 755 2024-09-11T03:46:34.076 INFO:tasks.rados.rados.2.smithi175.stdout:2141: oid 10 updating version 755 to 756 2024-09-11T03:46:34.076 INFO:tasks.rados.rados.2.smithi175.stdout:2141: oid 10 updating version 756 to 757 2024-09-11T03:46:34.076 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 757 (ObjNum 726 snap 238 seq_num 726) dirty exists 2024-09-11T03:46:34.076 INFO:tasks.rados.rados.2.smithi175.stdout:2141: left oid 10 (ObjNum 726 snap 238 seq_num 726) 2024-09-11T03:46:34.076 INFO:tasks.rados.rados.2.smithi175.stdout:2140: done (5 left) 2024-09-11T03:46:34.076 INFO:tasks.rados.rados.2.smithi175.stdout:2141: done (4 left) 2024-09-11T03:46:34.076 INFO:tasks.rados.rados.2.smithi175.stdout:2145: done (3 left) 2024-09-11T03:46:34.077 INFO:tasks.rados.rados.2.smithi175.stdout:2146: read oid 18 snap 232 2024-09-11T03:46:34.077 INFO:tasks.rados.rados.2.smithi175.stdout:2146: expect deleted 2024-09-11T03:46:34.077 INFO:tasks.rados.rados.2.smithi175.stdout:2147: read oid 12 snap 236 2024-09-11T03:46:34.077 INFO:tasks.rados.rados.2.smithi175.stdout:2147: expect (ObjNum 688 snap 226 seq_num 688) 2024-09-11T03:46:34.077 INFO:tasks.rados.rados.2.smithi175.stdout:2148: delete oid 15 current snap is 238 2024-09-11T03:46:34.077 INFO:tasks.rados.rados.2.smithi175.stdout:2132: expect (ObjNum 609 snap 191 seq_num 609) 2024-09-11T03:46:34.122 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:34.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:34.145 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:34.145 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:34.182 INFO:tasks.rados.rados.2.smithi175.stdout:2137: finishing copy_from racing read to smithi17557870-34 2024-09-11T03:46:34.183 INFO:tasks.rados.rados.2.smithi175.stdout:2137: finishing copy_from to smithi17557870-34 2024-09-11T03:46:34.183 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 504 (ObjNum 681 snap 218 seq_num 681) dirty exists 2024-09-11T03:46:34.183 INFO:tasks.rados.rados.2.smithi175.stdout:2139: expect (ObjNum 721 snap 236 seq_num 721) 2024-09-11T03:46:34.247 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:34.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:34.270 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:34.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:34.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:34 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:46:34.270 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:34 smithi121 ceph-mon[30729]: 2.2 scrub ok 2024-09-11T03:46:34.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:34 smithi121 ceph-mon[30729]: pgmap v1017: 67 pgs: 5 peering, 1 active+undersized+remapped, 1 remapped+peering, 1 unknown, 59 active+clean; 157 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 15 objects/s recovering 2024-09-11T03:46:34.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:34 smithi121 ceph-mon[30729]: 2.f scrub starts 2024-09-11T03:46:34.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:34 smithi121 ceph-mon[30729]: 2.f scrub ok 2024-09-11T03:46:34.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:34 smithi121 ceph-mon[30729]: 2.1b scrub starts 2024-09-11T03:46:34.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:34 smithi121 ceph-mon[30729]: 2.1b scrub ok 2024-09-11T03:46:34.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:34 smithi121 ceph-mon[30729]: 2.21 scrub starts 2024-09-11T03:46:34.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:34 smithi121 ceph-mon[30729]: 2.21 scrub ok 2024-09-11T03:46:34.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:34 smithi121 ceph-mon[30729]: 2.1f scrub starts 2024-09-11T03:46:34.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:34 smithi121 ceph-mon[30729]: 2.1f scrub ok 2024-09-11T03:46:34.271 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:34 smithi121 ceph-mon[30729]: osdmap e576: 12 total, 12 up, 12 in 2024-09-11T03:46:34.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:34 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:46:34.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:34 smithi084 ceph-mon[27254]: 2.2 scrub ok 2024-09-11T03:46:34.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:34 smithi084 ceph-mon[27254]: pgmap v1017: 67 pgs: 5 peering, 1 active+undersized+remapped, 1 remapped+peering, 1 unknown, 59 active+clean; 157 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 15 objects/s recovering 2024-09-11T03:46:34.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:34 smithi084 ceph-mon[27254]: 2.f scrub starts 2024-09-11T03:46:34.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:34 smithi084 ceph-mon[27254]: 2.f scrub ok 2024-09-11T03:46:34.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:34 smithi084 ceph-mon[27254]: 2.1b scrub starts 2024-09-11T03:46:34.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:34 smithi084 ceph-mon[27254]: 2.1b scrub ok 2024-09-11T03:46:34.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:34 smithi084 ceph-mon[27254]: 2.21 scrub starts 2024-09-11T03:46:34.310 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:34 smithi084 ceph-mon[27254]: 2.21 scrub ok 2024-09-11T03:46:34.310 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:34 smithi084 ceph-mon[27254]: 2.1f scrub starts 2024-09-11T03:46:34.310 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:34 smithi084 ceph-mon[27254]: 2.1f scrub ok 2024-09-11T03:46:34.310 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:34 smithi084 ceph-mon[27254]: osdmap e576: 12 total, 12 up, 12 in 2024-09-11T03:46:34.357 INFO:tasks.rados.rados.2.smithi175.stdout:2147: expect (ObjNum 688 snap 226 seq_num 688) 2024-09-11T03:46:34.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:34.392 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:34.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:34.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:34.463 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:46:34.464 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:46:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:34 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:46:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:34 smithi175 ceph-mon[30894]: 2.2 scrub ok 2024-09-11T03:46:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:34 smithi175 ceph-mon[30894]: pgmap v1017: 67 pgs: 5 peering, 1 active+undersized+remapped, 1 remapped+peering, 1 unknown, 59 active+clean; 157 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 15 objects/s recovering 2024-09-11T03:46:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:34 smithi175 ceph-mon[30894]: 2.f scrub starts 2024-09-11T03:46:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:34 smithi175 ceph-mon[30894]: 2.f scrub ok 2024-09-11T03:46:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:34 smithi175 ceph-mon[30894]: 2.1b scrub starts 2024-09-11T03:46:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:34 smithi175 ceph-mon[30894]: 2.1b scrub ok 2024-09-11T03:46:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:34 smithi175 ceph-mon[30894]: 2.21 scrub starts 2024-09-11T03:46:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:34 smithi175 ceph-mon[30894]: 2.21 scrub ok 2024-09-11T03:46:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:34 smithi175 ceph-mon[30894]: 2.1f scrub starts 2024-09-11T03:46:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:34 smithi175 ceph-mon[30894]: 2.1f scrub ok 2024-09-11T03:46:34.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:34 smithi175 ceph-mon[30894]: osdmap e576: 12 total, 12 up, 12 in 2024-09-11T03:46:34.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:34.516 INFO:tasks.rados.rados.2.smithi175.stdout:2132: done (5 left) 2024-09-11T03:46:34.516 INFO:tasks.rados.rados.2.smithi175.stdout:2137: done (4 left) 2024-09-11T03:46:34.516 INFO:tasks.rados.rados.2.smithi175.stdout:2139: done (3 left) 2024-09-11T03:46:34.516 INFO:tasks.rados.rados.2.smithi175.stdout:2146: done (2 left) 2024-09-11T03:46:34.516 INFO:tasks.rados.rados.2.smithi175.stdout:2147: done (1 left) 2024-09-11T03:46:34.516 INFO:tasks.rados.rados.2.smithi175.stdout:2148: done (0 left) 2024-09-11T03:46:34.516 INFO:tasks.rados.rados.2.smithi175.stdout:2149: read oid 3 snap 232 2024-09-11T03:46:34.517 INFO:tasks.rados.rados.2.smithi175.stdout:2149: expect deleted 2024-09-11T03:46:34.517 INFO:tasks.rados.rados.2.smithi175.stdout:2150: rollback oid 24 current snap is 238 2024-09-11T03:46:34.517 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 236 2024-09-11T03:46:34.517 INFO:tasks.rados.rados.2.smithi175.stdout:2151: write oid 39 current snap is 238 2024-09-11T03:46:34.517 INFO:tasks.rados.rados.2.smithi175.stdout:2151: seq_num 729 ranges {652303=710056,2059726=192577} 2024-09-11T03:46:34.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:34.518 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:34.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:34.522 INFO:tasks.rados.rados.2.smithi175.stdout:2151: writing smithi17557870-39 from 652303 to 1362359 tid 1 2024-09-11T03:46:34.528 INFO:tasks.rados.rados.2.smithi175.stdout:2151: writing smithi17557870-39 from 2059726 to 2252303 tid 2 2024-09-11T03:46:34.528 INFO:tasks.rados.rados.2.smithi175.stdout:2152: copy_from oid 27 from oid 10 current snap is 238 2024-09-11T03:46:34.528 INFO:tasks.rados.rados.2.smithi175.stdout:2153: read oid 7 snap -1 2024-09-11T03:46:34.528 INFO:tasks.rados.rados.2.smithi175.stdout:2153: expect (ObjNum 709 snap 232 seq_num 709) 2024-09-11T03:46:34.529 INFO:tasks.rados.rados.2.smithi175.stdout:2154: read oid 23 snap -1 2024-09-11T03:46:34.529 INFO:tasks.rados.rados.2.smithi175.stdout:2154: expect (ObjNum 639 snap 204 seq_num 639) 2024-09-11T03:46:34.529 INFO:tasks.rados.rados.2.smithi175.stdout:2155: snap_remove snap 222 2024-09-11T03:46:34.618 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:34.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:34.640 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:34.641 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:34.742 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:34.763 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:34.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:34.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:34.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:34.887 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:34.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:34.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:34.989 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:35.010 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:35.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:35.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:35.013 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:46:35.017 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.2 primary-affinity to 0 (802) 2024-09-11T03:46:35.019 INFO:tasks.rados.rados.2.smithi175.stdout:2155: done (6 left) 2024-09-11T03:46:35.019 INFO:tasks.rados.rados.2.smithi175.stdout:2156: snap_create 2024-09-11T03:46:35.019 INFO:tasks.rados.rados.2.smithi175.stdout:2150: finishing rollback tid 0 to smithi17557870-24 2024-09-11T03:46:35.019 INFO:tasks.rados.rados.2.smithi175.stdout:2150: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:46:35.019 INFO:tasks.rados.rados.2.smithi175.stdout:2150: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:46:35.019 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 852 (ObjNum 570 snap 174 seq_num 570) dirty exists 2024-09-11T03:46:35.019 INFO:tasks.rados.rados.2.smithi175.stdout:2151: finishing write tid 1 to smithi17557870-39 2024-09-11T03:46:35.019 INFO:tasks.rados.rados.2.smithi175.stdout:2154: expect (ObjNum 639 snap 204 seq_num 639) 2024-09-11T03:46:35.112 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:35.134 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:35.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:35.161 INFO:tasks.rados.rados.2.smithi175.stdout:2151: finishing write tid 2 to smithi17557870-39 2024-09-11T03:46:35.161 INFO:tasks.rados.rados.2.smithi175.stdout:2151: finishing write tid 3 to smithi17557870-39 2024-09-11T03:46:35.161 INFO:tasks.rados.rados.2.smithi175.stdout:2151: finishing write tid 4 to smithi17557870-39 2024-09-11T03:46:35.161 INFO:tasks.rados.rados.2.smithi175.stdout:2151: oid 39 updating version 0 to 650 2024-09-11T03:46:35.161 INFO:tasks.rados.rados.2.smithi175.stdout:2151: oid 39 updating version 650 to 652 2024-09-11T03:46:35.161 INFO:tasks.rados.rados.2.smithi175.stdout:2151: oid 39 version 652 is already newer than 651 2024-09-11T03:46:35.161 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 652 (ObjNum 729 snap 238 seq_num 729) dirty exists 2024-09-11T03:46:35.161 INFO:tasks.rados.rados.2.smithi175.stdout:2151: left oid 39 (ObjNum 729 snap 238 seq_num 729) 2024-09-11T03:46:35.161 INFO:tasks.rados.rados.2.smithi175.stdout:2152: finishing copy_from racing read to smithi17557870-27 2024-09-11T03:46:35.161 INFO:tasks.rados.rados.2.smithi175.stdout:2152: finishing copy_from to smithi17557870-27 2024-09-11T03:46:35.161 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 757 (ObjNum 726 snap 238 seq_num 726) dirty exists 2024-09-11T03:46:35.162 INFO:tasks.rados.rados.2.smithi175.stdout:2153: expect (ObjNum 709 snap 232 seq_num 709) 2024-09-11T03:46:35.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T03:46:35.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:35.259 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:35.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:35.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:35.361 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:35 smithi084 ceph-mon[27254]: 2.1c scrub starts 2024-09-11T03:46:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:35 smithi084 ceph-mon[27254]: 2.1c scrub ok 2024-09-11T03:46:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:35 smithi084 ceph-mon[27254]: 2.22 scrub starts 2024-09-11T03:46:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:35 smithi084 ceph-mon[27254]: 2.22 scrub ok 2024-09-11T03:46:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:35 smithi084 ceph-mon[27254]: 2.20 scrub starts 2024-09-11T03:46:35.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:35 smithi084 ceph-mon[27254]: 2.20 scrub ok 2024-09-11T03:46:35.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:35 smithi084 ceph-mon[27254]: 2.3 scrub starts 2024-09-11T03:46:35.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:35 smithi084 ceph-mon[27254]: 2.3 scrub ok 2024-09-11T03:46:35.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:35 smithi084 ceph-mon[27254]: 2.23 scrub starts 2024-09-11T03:46:35.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:35 smithi084 ceph-mon[27254]: 2.23 scrub ok 2024-09-11T03:46:35.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:35 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/609859165' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-09-11T03:46:35.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:35 smithi084 ceph-mon[27254]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-09-11T03:46:35.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:35 smithi084 ceph-mon[27254]: 2.1e scrub starts 2024-09-11T03:46:35.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:35 smithi084 ceph-mon[27254]: 2.1e scrub ok 2024-09-11T03:46:35.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:35 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4095046450' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:35.383 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:35.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:35.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:35.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:35 smithi121 ceph-mon[30729]: 2.1c scrub starts 2024-09-11T03:46:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:35 smithi121 ceph-mon[30729]: 2.1c scrub ok 2024-09-11T03:46:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:35 smithi121 ceph-mon[30729]: 2.22 scrub starts 2024-09-11T03:46:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:35 smithi121 ceph-mon[30729]: 2.22 scrub ok 2024-09-11T03:46:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:35 smithi121 ceph-mon[30729]: 2.20 scrub starts 2024-09-11T03:46:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:35 smithi121 ceph-mon[30729]: 2.20 scrub ok 2024-09-11T03:46:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:35 smithi121 ceph-mon[30729]: 2.3 scrub starts 2024-09-11T03:46:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:35 smithi121 ceph-mon[30729]: 2.3 scrub ok 2024-09-11T03:46:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:35 smithi121 ceph-mon[30729]: 2.23 scrub starts 2024-09-11T03:46:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:35 smithi121 ceph-mon[30729]: 2.23 scrub ok 2024-09-11T03:46:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:35 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/609859165' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-09-11T03:46:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:35 smithi121 ceph-mon[30729]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-09-11T03:46:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:35 smithi121 ceph-mon[30729]: 2.1e scrub starts 2024-09-11T03:46:35.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:35 smithi121 ceph-mon[30729]: 2.1e scrub ok 2024-09-11T03:46:35.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:35 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4095046450' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:35.463 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:46:35.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:35 smithi175 ceph-mon[30894]: 2.1c scrub starts 2024-09-11T03:46:35.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:35 smithi175 ceph-mon[30894]: 2.1c scrub ok 2024-09-11T03:46:35.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:35 smithi175 ceph-mon[30894]: 2.22 scrub starts 2024-09-11T03:46:35.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:35 smithi175 ceph-mon[30894]: 2.22 scrub ok 2024-09-11T03:46:35.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:35 smithi175 ceph-mon[30894]: 2.20 scrub starts 2024-09-11T03:46:35.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:35 smithi175 ceph-mon[30894]: 2.20 scrub ok 2024-09-11T03:46:35.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:35 smithi175 ceph-mon[30894]: 2.3 scrub starts 2024-09-11T03:46:35.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:35 smithi175 ceph-mon[30894]: 2.3 scrub ok 2024-09-11T03:46:35.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:35 smithi175 ceph-mon[30894]: 2.23 scrub starts 2024-09-11T03:46:35.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:35 smithi175 ceph-mon[30894]: 2.23 scrub ok 2024-09-11T03:46:35.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:35 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/609859165' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-09-11T03:46:35.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:35 smithi175 ceph-mon[30894]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-09-11T03:46:35.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:35 smithi175 ceph-mon[30894]: 2.1e scrub starts 2024-09-11T03:46:35.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:35 smithi175 ceph-mon[30894]: 2.1e scrub ok 2024-09-11T03:46:35.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:35 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4095046450' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:35.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:35.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:35.507 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:35.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:35.608 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:35.624 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:46:35.630 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:35.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:35.631 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:35.732 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:35.755 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:35.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:35.857 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:35.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:35.879 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:35.880 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:35.905 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:46:35.906 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:46:35.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:36.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:36.003 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:36.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:36.022 INFO:tasks.rados.rados.2.smithi175.stdout:2149: done (6 left) 2024-09-11T03:46:36.023 INFO:tasks.rados.rados.2.smithi175.stdout:2150: done (5 left) 2024-09-11T03:46:36.023 INFO:tasks.rados.rados.2.smithi175.stdout:2151: done (4 left) 2024-09-11T03:46:36.023 INFO:tasks.rados.rados.2.smithi175.stdout:2152: done (3 left) 2024-09-11T03:46:36.023 INFO:tasks.rados.rados.2.smithi175.stdout:2153: done (2 left) 2024-09-11T03:46:36.023 INFO:tasks.rados.rados.2.smithi175.stdout:2154: done (1 left) 2024-09-11T03:46:36.023 INFO:tasks.rados.rados.2.smithi175.stdout:2156: done (0 left) 2024-09-11T03:46:36.023 INFO:tasks.rados.rados.2.smithi175.stdout:2157: write oid 28 current snap is 239 2024-09-11T03:46:36.023 INFO:tasks.rados.rados.2.smithi175.stdout:2157: seq_num 730 ranges {669110=501658,1834561=434549} 2024-09-11T03:46:36.028 INFO:tasks.rados.rados.2.smithi175.stdout:2157: writing smithi17557870-28 from 669110 to 1170768 tid 1 2024-09-11T03:46:36.035 INFO:tasks.rados.rados.2.smithi175.stdout:2157: writing smithi17557870-28 from 1834561 to 2269110 tid 2 2024-09-11T03:46:36.035 INFO:tasks.rados.rados.2.smithi175.stdout:2158: read oid 10 snap -1 2024-09-11T03:46:36.035 INFO:tasks.rados.rados.2.smithi175.stdout:2158: expect (ObjNum 726 snap 238 seq_num 726) 2024-09-11T03:46:36.035 INFO:tasks.rados.rados.2.smithi175.stdout:2159: delete oid 36 current snap is 239 2024-09-11T03:46:36.035 INFO:tasks.rados.rados.2.smithi175.stdout:2157: finishing write tid 1 to smithi17557870-28 2024-09-11T03:46:36.036 INFO:tasks.rados.rados.2.smithi175.stdout:2159: done (2 left) 2024-09-11T03:46:36.036 INFO:tasks.rados.rados.2.smithi175.stdout:2160: delete oid 45 current snap is 239 2024-09-11T03:46:36.037 INFO:tasks.rados.rados.2.smithi175.stdout:2157: finishing write tid 2 to smithi17557870-28 2024-09-11T03:46:36.037 INFO:tasks.rados.rados.2.smithi175.stdout:2157: finishing write tid 3 to smithi17557870-28 2024-09-11T03:46:36.037 INFO:tasks.rados.rados.2.smithi175.stdout:2157: finishing write tid 4 to smithi17557870-28 2024-09-11T03:46:36.037 INFO:tasks.rados.rados.2.smithi175.stdout:2157: oid 28 updating version 0 to 875 2024-09-11T03:46:36.037 INFO:tasks.rados.rados.2.smithi175.stdout:2157: oid 28 version 875 is already newer than 873 2024-09-11T03:46:36.038 INFO:tasks.rados.rados.2.smithi175.stdout:2157: oid 28 version 875 is already newer than 874 2024-09-11T03:46:36.038 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 875 (ObjNum 730 snap 239 seq_num 730) dirty exists 2024-09-11T03:46:36.038 INFO:tasks.rados.rados.2.smithi175.stdout:2157: left oid 28 (ObjNum 730 snap 239 seq_num 730) 2024-09-11T03:46:36.038 INFO:tasks.rados.rados.2.smithi175.stdout:2157: done (2 left) 2024-09-11T03:46:36.038 INFO:tasks.rados.rados.2.smithi175.stdout:2160: done (1 left) 2024-09-11T03:46:36.038 INFO:tasks.rados.rados.2.smithi175.stdout:2161: snap_create 2024-09-11T03:46:36.047 INFO:tasks.rados.rados.2.smithi175.stdout:2158: expect (ObjNum 726 snap 238 seq_num 726) 2024-09-11T03:46:36.104 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:36.125 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:36.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:36.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:36 smithi121 ceph-mon[30729]: 2.1d scrub starts 2024-09-11T03:46:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:36 smithi121 ceph-mon[30729]: 2.1d scrub ok 2024-09-11T03:46:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:36 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:46:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:36 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:46:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:36 smithi121 ceph-mon[30729]: 2.1a scrub starts 2024-09-11T03:46:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:36 smithi121 ceph-mon[30729]: 2.1a scrub ok 2024-09-11T03:46:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:36 smithi121 ceph-mon[30729]: pgmap v1020: 67 pgs: 5 peering, 1 active+undersized+remapped, 1 remapped+peering, 1 unknown, 59 active+clean; 154 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s, 20 objects/s recovering 2024-09-11T03:46:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:36 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:46:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:36 smithi121 ceph-mon[30729]: 2.9 scrub ok 2024-09-11T03:46:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:36 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:46:36.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:36 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:46:36.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:36 smithi121 ceph-mon[30729]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-09-11T03:46:36.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4095046450' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:36.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:36 smithi121 ceph-mon[30729]: osdmap e577: 12 total, 12 up, 12 in 2024-09-11T03:46:36.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:36 smithi121 ceph-mon[30729]: 1.16 scrub starts 2024-09-11T03:46:36.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:36 smithi121 ceph-mon[30729]: 1.16 scrub ok 2024-09-11T03:46:36.228 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:36.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:36.250 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:36.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:36.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:36 smithi084 ceph-mon[27254]: 2.1d scrub starts 2024-09-11T03:46:36.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:36 smithi084 ceph-mon[27254]: 2.1d scrub ok 2024-09-11T03:46:36.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:36 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:46:36.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:36 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:46:36.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:36 smithi084 ceph-mon[27254]: 2.1a scrub starts 2024-09-11T03:46:36.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:36 smithi084 ceph-mon[27254]: 2.1a scrub ok 2024-09-11T03:46:36.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:36 smithi084 ceph-mon[27254]: pgmap v1020: 67 pgs: 5 peering, 1 active+undersized+remapped, 1 remapped+peering, 1 unknown, 59 active+clean; 154 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s, 20 objects/s recovering 2024-09-11T03:46:36.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:36 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:46:36.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:36 smithi084 ceph-mon[27254]: 2.9 scrub ok 2024-09-11T03:46:36.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:36 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:46:36.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:36 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:46:36.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:36 smithi084 ceph-mon[27254]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-09-11T03:46:36.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4095046450' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:36.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:36 smithi084 ceph-mon[27254]: osdmap e577: 12 total, 12 up, 12 in 2024-09-11T03:46:36.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:36 smithi084 ceph-mon[27254]: 1.16 scrub starts 2024-09-11T03:46:36.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:36 smithi084 ceph-mon[27254]: 1.16 scrub ok 2024-09-11T03:46:36.351 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:36.401 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:36.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:36.416 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:36.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:36 smithi175 ceph-mon[30894]: 2.1d scrub starts 2024-09-11T03:46:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:36 smithi175 ceph-mon[30894]: 2.1d scrub ok 2024-09-11T03:46:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:36 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:46:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:36 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:46:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:36 smithi175 ceph-mon[30894]: 2.1a scrub starts 2024-09-11T03:46:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:36 smithi175 ceph-mon[30894]: 2.1a scrub ok 2024-09-11T03:46:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:36 smithi175 ceph-mon[30894]: pgmap v1020: 67 pgs: 5 peering, 1 active+undersized+remapped, 1 remapped+peering, 1 unknown, 59 active+clean; 154 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s, 20 objects/s recovering 2024-09-11T03:46:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:36 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:46:36.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:36 smithi175 ceph-mon[30894]: 2.9 scrub ok 2024-09-11T03:46:36.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:36 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:46:36.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:36 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:46:36.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:36 smithi175 ceph-mon[30894]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-09-11T03:46:36.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4095046450' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:36.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:36 smithi175 ceph-mon[30894]: osdmap e577: 12 total, 12 up, 12 in 2024-09-11T03:46:36.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:36 smithi175 ceph-mon[30894]: 1.16 scrub starts 2024-09-11T03:46:36.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:36 smithi175 ceph-mon[30894]: 1.16 scrub ok 2024-09-11T03:46:36.518 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:36.541 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:36.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:36.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:36.643 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:36.663 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T03:46:36.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:36.666 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:36.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:36.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:36.788 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:36.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:36.890 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:36.901 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:46:36.901 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:46:36.908 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:46:36.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:36.912 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:36.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:36.918 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:46:36.918 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:46:36.935 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:46:36.935 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:46:36.950 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:46:36.950 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:46:36.967 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:46:36.968 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:46:36.984 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:46:36.985 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:46:37.001 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:46:37.001 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:46:37.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:37.018 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:46:37.018 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:46:37.028 INFO:tasks.rados.rados.2.smithi175.stdout:2158: done (1 left) 2024-09-11T03:46:37.028 INFO:tasks.rados.rados.2.smithi175.stdout:2161: done (0 left) 2024-09-11T03:46:37.028 INFO:tasks.rados.rados.2.smithi175.stdout:2162: snap_create 2024-09-11T03:46:37.034 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:37.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:37.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:37.037 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:46:37.037 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:46:37.054 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:46:37.055 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:46:37.072 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:46:37.072 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:46:37.089 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:46:37.089 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:46:37.136 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:37.162 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:37.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:37.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:37.264 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:37.285 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:37.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:37.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:37.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:37 smithi121 ceph-mon[30729]: 2.17 scrub starts 2024-09-11T03:46:37.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:37 smithi121 ceph-mon[30729]: 2.17 scrub ok 2024-09-11T03:46:37.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:37 smithi121 ceph-mon[30729]: osdmap e578: 12 total, 12 up, 12 in 2024-09-11T03:46:37.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:37 smithi121 ceph-mon[30729]: 2.21 scrub starts 2024-09-11T03:46:37.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:37 smithi121 ceph-mon[30729]: 2.21 scrub ok 2024-09-11T03:46:37.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:37 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:46:37.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:37 smithi121 ceph-mon[30729]: 2.1 scrub ok 2024-09-11T03:46:37.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:37 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:46:37.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:37 smithi121 ceph-mon[30729]: 2.e scrub ok 2024-09-11T03:46:37.330 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:37 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2435943262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:37.345 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:46:37.345 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:46:37.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:46:37.345 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:46:37.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.345 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:228 2.9 e105/95 2:95fa802a:::smithi17557870-22:head v 105'20, mlcod=100'17)", 2024-09-11T03:46:37.345 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:46:37.346 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 475.37779274100001, 2024-09-11T03:46:37.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01695953, 2024-09-11T03:46:37.346 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.346 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.346 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:37.346 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:46:37.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:46:37.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.966943+0000", 2024-09-11T03:46:37.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2600000000000002e-07 2024-09-11T03:46:37.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982417+0000", 2024-09-11T03:46:37.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01547431 2024-09-11T03:46:37.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982440+0000", 2024-09-11T03:46:37.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2506000000000001e-05 2024-09-11T03:46:37.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982448+0000", 2024-09-11T03:46:37.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2989999999999997e-06 2024-09-11T03:46:37.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982562+0000", 2024-09-11T03:46:37.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114084 2024-09-11T03:46:37.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.982583+0000", 2024-09-11T03:46:37.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1160999999999999e-05 2024-09-11T03:46:37.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:46:37.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.983852+0000", 2024-09-11T03:46:37.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001268953 2024-09-11T03:46:37.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:41.983902+0000", 2024-09-11T03:46:37.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9691000000000002e-05 2024-09-11T03:46:37.350 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.350 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.350 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.350 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:345 2.0 2:0762776a:::smithi17557870-25:head [write 742876~447756] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:46:37.350 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.827240+0000", 2024-09-11T03:46:37.350 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 467.51749542599998, 2024-09-11T03:46:37.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076328890000000003, 2024-09-11T03:46:37.351 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.351 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.351 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:37.351 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:46:37.351 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:46:37.351 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:37.351 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 345 2024-09-11T03:46:37.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.351 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.827240+0000", 2024-09-11T03:46:37.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.827240+0000", 2024-09-11T03:46:37.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.827243+0000", 2024-09-11T03:46:37.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0110000000000001e-06 2024-09-11T03:46:37.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828270+0000", 2024-09-11T03:46:37.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001026838 2024-09-11T03:46:37.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828298+0000", 2024-09-11T03:46:37.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8561999999999999e-05 2024-09-11T03:46:37.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828312+0000", 2024-09-11T03:46:37.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3781000000000001e-05 2024-09-11T03:46:37.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828398+0000", 2024-09-11T03:46:37.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6336999999999997e-05 2024-09-11T03:46:37.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:46:37.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.828405+0000", 2024-09-11T03:46:37.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9079999999999998e-06 2024-09-11T03:46:37.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830015+0000", 2024-09-11T03:46:37.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001609344 2024-09-11T03:46:37.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830095+0000", 2024-09-11T03:46:37.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0259000000000005e-05 2024-09-11T03:46:37.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10", 2024-09-11T03:46:37.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830248+0000", 2024-09-11T03:46:37.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015351500000000001 2024-09-11T03:46:37.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:46:37.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.831760+0000", 2024-09-11T03:46:37.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015116540000000001 2024-09-11T03:46:37.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:37.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.832800+0000", 2024-09-11T03:46:37.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010400800000000001 2024-09-11T03:46:37.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:37.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834760+0000", 2024-09-11T03:46:37.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019600809999999998 2024-09-11T03:46:37.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:46:37.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834804+0000", 2024-09-11T03:46:37.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3922999999999999e-05 2024-09-11T03:46:37.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.834873+0000", 2024-09-11T03:46:37.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8596000000000005e-05 2024-09-11T03:46:37.358 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.358 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.358 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.358 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:346 2.0 2:0762776a:::smithi17557870-25:head [write 1729599~575687] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:46:37.358 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.833957+0000", 2024-09-11T03:46:37.358 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 467.51077870099999, 2024-09-11T03:46:37.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00864616, 2024-09-11T03:46:37.358 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.358 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.359 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:37.359 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:46:37.359 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:46:37.359 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:37.359 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 346 2024-09-11T03:46:37.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.359 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833957+0000", 2024-09-11T03:46:37.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833957+0000", 2024-09-11T03:46:37.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.833958+0000", 2024-09-11T03:46:37.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.674e-06 2024-09-11T03:46:37.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838969+0000", 2024-09-11T03:46:37.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005010993 2024-09-11T03:46:37.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839016+0000", 2024-09-11T03:46:37.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6576000000000003e-05 2024-09-11T03:46:37.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839037+0000", 2024-09-11T03:46:37.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1356e-05 2024-09-11T03:46:37.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839197+0000", 2024-09-11T03:46:37.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000159338 2024-09-11T03:46:37.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839266+0000", 2024-09-11T03:46:37.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9758999999999994e-05 2024-09-11T03:46:37.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10", 2024-09-11T03:46:37.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.839374+0000", 2024-09-11T03:46:37.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010791899999999999 2024-09-11T03:46:37.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:46:37.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.840763+0000", 2024-09-11T03:46:37.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013883999999999999 2024-09-11T03:46:37.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:37.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842254+0000", 2024-09-11T03:46:37.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001491409 2024-09-11T03:46:37.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:37.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842538+0000", 2024-09-11T03:46:37.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028420099999999998 2024-09-11T03:46:37.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:46:37.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842573+0000", 2024-09-11T03:46:37.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4381000000000002e-05 2024-09-11T03:46:37.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.842603+0000", 2024-09-11T03:46:37.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0154e-05 2024-09-11T03:46:37.366 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.366 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.366 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.366 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1555 2.3 e214/95 2:c6c49f2c:::smithi17557870-44:head v 214'74, mlcod=198'70)", 2024-09-11T03:46:37.366 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:30.893545+0000", 2024-09-11T03:46:37.367 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 366.45118982299999, 2024-09-11T03:46:37.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070433960000000004, 2024-09-11T03:46:37.367 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.367 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.367 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:37.367 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.893545+0000", 2024-09-11T03:46:37.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.893545+0000", 2024-09-11T03:46:37.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.893546+0000", 2024-09-11T03:46:37.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7500000000000001e-07 2024-09-11T03:46:37.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895656+0000", 2024-09-11T03:46:37.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002110476 2024-09-11T03:46:37.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895679+0000", 2024-09-11T03:46:37.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2795000000000001e-05 2024-09-11T03:46:37.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895692+0000", 2024-09-11T03:46:37.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3141e-05 2024-09-11T03:46:37.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895758+0000", 2024-09-11T03:46:37.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5871999999999995e-05 2024-09-11T03:46:37.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.895786+0000", 2024-09-11T03:46:37.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8391e-05 2024-09-11T03:46:37.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:46:37.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900550+0000", 2024-09-11T03:46:37.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047632869999999997 2024-09-11T03:46:37.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900589+0000", 2024-09-11T03:46:37.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9159000000000002e-05 2024-09-11T03:46:37.371 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.371 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.371 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.371 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1609 2.8 2:1bdf1a36:::smithi17557870-36:head [write 618557~727148] snapc 40=[40,3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:46:37.372 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:35.905306+0000", 2024-09-11T03:46:37.372 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 361.439429754, 2024-09-11T03:46:37.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069719600000000001, 2024-09-11T03:46:37.372 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.372 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.372 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:37.372 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:46:37.372 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:46:37.372 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:37.372 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1609 2024-09-11T03:46:37.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.372 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.905306+0000", 2024-09-11T03:46:37.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.905306+0000", 2024-09-11T03:46:37.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.905307+0000", 2024-09-11T03:46:37.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.482e-06 2024-09-11T03:46:37.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907145+0000", 2024-09-11T03:46:37.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001838412 2024-09-11T03:46:37.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907157+0000", 2024-09-11T03:46:37.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-09-11T03:46:37.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907165+0000", 2024-09-11T03:46:37.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6340000000000001e-06 2024-09-11T03:46:37.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907215+0000", 2024-09-11T03:46:37.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0760999999999997e-05 2024-09-11T03:46:37.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907282+0000", 2024-09-11T03:46:37.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6934000000000007e-05 2024-09-11T03:46:37.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,9", 2024-09-11T03:46:37.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.907400+0000", 2024-09-11T03:46:37.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117625 2024-09-11T03:46:37.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:46:37.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.908820+0000", 2024-09-11T03:46:37.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0014204140000000001 2024-09-11T03:46:37.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:37.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.910145+0000", 2024-09-11T03:46:37.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013246250000000001 2024-09-11T03:46:37.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:37.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.912194+0000", 2024-09-11T03:46:37.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020485640000000001 2024-09-11T03:46:37.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:46:37.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.912229+0000", 2024-09-11T03:46:37.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5128e-05 2024-09-11T03:46:37.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.912277+0000", 2024-09-11T03:46:37.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8746999999999999e-05 2024-09-11T03:46:37.379 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.379 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.379 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.379 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1868 2.0 2:0762776a:::smithi17557870-25:head [copy-from ver 218] snapc 4c=[4c,4b,49,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:46:37.379 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:01.850162+0000", 2024-09-11T03:46:37.379 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 335.494573425, 2024-09-11T03:46:37.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020677853999999999, 2024-09-11T03:46:37.379 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.379 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.379 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:37.380 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:46:37.380 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:46:37.380 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:37.380 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1868 2024-09-11T03:46:37.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.380 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850162+0000", 2024-09-11T03:46:37.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850162+0000", 2024-09-11T03:46:37.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850164+0000", 2024-09-11T03:46:37.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.588e-06 2024-09-11T03:46:37.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850174+0000", 2024-09-11T03:46:37.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8300000000000008e-06 2024-09-11T03:46:37.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850193+0000", 2024-09-11T03:46:37.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8346e-05 2024-09-11T03:46:37.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850204+0000", 2024-09-11T03:46:37.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1144e-05 2024-09-11T03:46:37.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850285+0000", 2024-09-11T03:46:37.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1525000000000003e-05 2024-09-11T03:46:37.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:46:37.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850293+0000", 2024-09-11T03:46:37.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.072e-06 2024-09-11T03:46:37.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.852390+0000", 2024-09-11T03:46:37.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002096601 2024-09-11T03:46:37.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.852527+0000", 2024-09-11T03:46:37.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013703899999999999 2024-09-11T03:46:37.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,11", 2024-09-11T03:46:37.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.862911+0000", 2024-09-11T03:46:37.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01038392 2024-09-11T03:46:37.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:46:37.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.867006+0000", 2024-09-11T03:46:37.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004095291 2024-09-11T03:46:37.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:37.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.868758+0000", 2024-09-11T03:46:37.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017514869999999999 2024-09-11T03:46:37.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:37.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.870714+0000", 2024-09-11T03:46:37.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019563229999999998 2024-09-11T03:46:37.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:46:37.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.870780+0000", 2024-09-11T03:46:37.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6459999999999997e-05 2024-09-11T03:46:37.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.870840+0000", 2024-09-11T03:46:37.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9228000000000003e-05 2024-09-11T03:46:37.386 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.386 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.386 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.386 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1869 2.0 2:0762776a:::smithi17557870-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:46:37.386 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:01.850232+0000", 2024-09-11T03:46:37.386 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 335.49450284099998, 2024-09-11T03:46:37.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016419715000000001, 2024-09-11T03:46:37.387 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.387 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.387 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:46:37.387 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:46:37.387 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:46:37.387 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:37.387 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1869 2024-09-11T03:46:37.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.387 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850232+0000", 2024-09-11T03:46:37.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850232+0000", 2024-09-11T03:46:37.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850233+0000", 2024-09-11T03:46:37.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5300000000000004e-07 2024-09-11T03:46:37.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850237+0000", 2024-09-11T03:46:37.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3660000000000002e-06 2024-09-11T03:46:37.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850247+0000", 2024-09-11T03:46:37.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.38e-06 2024-09-11T03:46:37.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850254+0000", 2024-09-11T03:46:37.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2099999999999996e-06 2024-09-11T03:46:37.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850365+0000", 2024-09-11T03:46:37.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110499 2024-09-11T03:46:37.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:46:37.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.850369+0000", 2024-09-11T03:46:37.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.454e-06 2024-09-11T03:46:37.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.852747+0000", 2024-09-11T03:46:37.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023778480000000001 2024-09-11T03:46:37.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:46:37.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.852806+0000", 2024-09-11T03:46:37.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8931999999999998e-05 2024-09-11T03:46:37.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.866380+0000", 2024-09-11T03:46:37.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013574693000000001 2024-09-11T03:46:37.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.866512+0000", 2024-09-11T03:46:37.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013140999999999999 2024-09-11T03:46:37.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:01.866652+0000", 2024-09-11T03:46:37.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014027000000000001 2024-09-11T03:46:37.392 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.393 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.393 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.393 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1877 2.b e247/95 2:de6b1316:::smithi17557870-32:head v 247'232, mlcod=243'228)", 2024-09-11T03:46:37.393 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:03.916185+0000", 2024-09-11T03:46:37.393 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 333.42855031099998, 2024-09-11T03:46:37.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010307996, 2024-09-11T03:46:37.393 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.393 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.393 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:37.393 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.916185+0000", 2024-09-11T03:46:37.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.916185+0000", 2024-09-11T03:46:37.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.916185+0000", 2024-09-11T03:46:37.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1699999999999998e-07 2024-09-11T03:46:37.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924225+0000", 2024-09-11T03:46:37.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080395750000000002 2024-09-11T03:46:37.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924243+0000", 2024-09-11T03:46:37.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7955e-05 2024-09-11T03:46:37.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924271+0000", 2024-09-11T03:46:37.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7797000000000002e-05 2024-09-11T03:46:37.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924349+0000", 2024-09-11T03:46:37.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8137000000000001e-05 2024-09-11T03:46:37.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.924370+0000", 2024-09-11T03:46:37.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1226e-05 2024-09-11T03:46:37.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:46:37.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.926454+0000", 2024-09-11T03:46:37.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002083438 2024-09-11T03:46:37.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:03.926493+0000", 2024-09-11T03:46:37.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9351000000000003e-05 2024-09-11T03:46:37.397 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.397 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.397 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.398 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1989 2.0 2:0762776a:::smithi17557870-25:head [copy-from ver 162] snapc 52=[52,51,4e,3d] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-11T03:46:37.398 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:14.949188+0000", 2024-09-11T03:46:37.398 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 322.39554724499999, 2024-09-11T03:46:37.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014886257999999999, 2024-09-11T03:46:37.398 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.398 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.398 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:37.398 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:46:37.398 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:46:37.398 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:37.399 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1989 2024-09-11T03:46:37.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.399 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949188+0000", 2024-09-11T03:46:37.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949188+0000", 2024-09-11T03:46:37.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949191+0000", 2024-09-11T03:46:37.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.706e-06 2024-09-11T03:46:37.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949204+0000", 2024-09-11T03:46:37.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3142999999999999e-05 2024-09-11T03:46:37.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949227+0000", 2024-09-11T03:46:37.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3320999999999999e-05 2024-09-11T03:46:37.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949238+0000", 2024-09-11T03:46:37.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0352e-05 2024-09-11T03:46:37.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949313+0000", 2024-09-11T03:46:37.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5450000000000004e-05 2024-09-11T03:46:37.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:46:37.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949319+0000", 2024-09-11T03:46:37.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9279999999999998e-06 2024-09-11T03:46:37.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.950847+0000", 2024-09-11T03:46:37.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015282690000000001 2024-09-11T03:46:37.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.950916+0000", 2024-09-11T03:46:37.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8559000000000005e-05 2024-09-11T03:46:37.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,11", 2024-09-11T03:46:37.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.956873+0000", 2024-09-11T03:46:37.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059576359999999997 2024-09-11T03:46:37.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:46:37.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.960470+0000", 2024-09-11T03:46:37.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035970749999999999 2024-09-11T03:46:37.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:37.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.963875+0000", 2024-09-11T03:46:37.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034049050000000002 2024-09-11T03:46:37.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:37.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.963958+0000", 2024-09-11T03:46:37.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2769999999999995e-05 2024-09-11T03:46:37.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:46:37.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.964021+0000", 2024-09-11T03:46:37.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2442999999999998e-05 2024-09-11T03:46:37.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.964074+0000", 2024-09-11T03:46:37.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3701e-05 2024-09-11T03:46:37.405 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.405 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.405 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.405 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1990 2.0 2:0762776a:::smithi17557870-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e258)", 2024-09-11T03:46:37.405 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:14.949265+0000", 2024-09-11T03:46:37.405 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 322.39547020399999, 2024-09-11T03:46:37.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010821580000000001, 2024-09-11T03:46:37.406 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.406 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.406 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:46:37.406 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:46:37.406 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:46:37.406 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:37.406 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1990 2024-09-11T03:46:37.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.406 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949265+0000", 2024-09-11T03:46:37.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949265+0000", 2024-09-11T03:46:37.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949266+0000", 2024-09-11T03:46:37.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3000000000000001e-07 2024-09-11T03:46:37.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949268+0000", 2024-09-11T03:46:37.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2249999999999999e-06 2024-09-11T03:46:37.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949273+0000", 2024-09-11T03:46:37.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6389999999999997e-06 2024-09-11T03:46:37.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949277+0000", 2024-09-11T03:46:37.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4630000000000001e-06 2024-09-11T03:46:37.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949380+0000", 2024-09-11T03:46:37.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103099 2024-09-11T03:46:37.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:46:37.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.949387+0000", 2024-09-11T03:46:37.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9290000000000001e-06 2024-09-11T03:46:37.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.951035+0000", 2024-09-11T03:46:37.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001647995 2024-09-11T03:46:37.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:46:37.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.951082+0000", 2024-09-11T03:46:37.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7231999999999998e-05 2024-09-11T03:46:37.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.959865+0000", 2024-09-11T03:46:37.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087832770000000008 2024-09-11T03:46:37.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.959967+0000", 2024-09-11T03:46:37.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010153 2024-09-11T03:46:37.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:14.960087+0000", 2024-09-11T03:46:37.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011976099999999999 2024-09-11T03:46:37.411 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.411 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.411 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.412 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:2434 2.9 e295/276 2:939cf94d:::smithi17557870-28:head v 295'353, mlcod=295'349)", 2024-09-11T03:46:37.412 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:52.055937+0000", 2024-09-11T03:46:37.412 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 285.28879802799997, 2024-09-11T03:46:37.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071747089999999996, 2024-09-11T03:46:37.412 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.412 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.412 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:37.412 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.055937+0000", 2024-09-11T03:46:37.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.055937+0000", 2024-09-11T03:46:37.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.055938+0000", 2024-09-11T03:46:37.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.043e-06 2024-09-11T03:46:37.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.059618+0000", 2024-09-11T03:46:37.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003679435 2024-09-11T03:46:37.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.059642+0000", 2024-09-11T03:46:37.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4485999999999999e-05 2024-09-11T03:46:37.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.059671+0000", 2024-09-11T03:46:37.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8560000000000001e-05 2024-09-11T03:46:37.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.059750+0000", 2024-09-11T03:46:37.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9170000000000003e-05 2024-09-11T03:46:37.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.059780+0000", 2024-09-11T03:46:37.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9558999999999999e-05 2024-09-11T03:46:37.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:46:37.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.063060+0000", 2024-09-11T03:46:37.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003280383 2024-09-11T03:46:37.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:52.063112+0000", 2024-09-11T03:46:37.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2073000000000001e-05 2024-09-11T03:46:37.416 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.416 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.416 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.416 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3749 2.8 2:162357fa:::smithi17557870-18:head [write 773254~791982] snapc 9b=[9b,9a,99,98,97,90,8a] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-11T03:46:37.416 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:43:39.942169+0000", 2024-09-11T03:46:37.417 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 177.40256588700001, 2024-09-11T03:46:37.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0082841789999999992, 2024-09-11T03:46:37.417 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.417 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.417 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:37.417 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:46:37.417 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:46:37.417 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:37.417 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3749 2024-09-11T03:46:37.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.417 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.942169+0000", 2024-09-11T03:46:37.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.942169+0000", 2024-09-11T03:46:37.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.942171+0000", 2024-09-11T03:46:37.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0470000000000001e-06 2024-09-11T03:46:37.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.943249+0000", 2024-09-11T03:46:37.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010774459999999999 2024-09-11T03:46:37.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.943269+0000", 2024-09-11T03:46:37.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0063000000000001e-05 2024-09-11T03:46:37.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.943279+0000", 2024-09-11T03:46:37.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0010000000000001e-05 2024-09-11T03:46:37.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.943372+0000", 2024-09-11T03:46:37.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3196999999999996e-05 2024-09-11T03:46:37.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:46:37.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.943377+0000", 2024-09-11T03:46:37.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6990000000000002e-06 2024-09-11T03:46:37.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.944823+0000", 2024-09-11T03:46:37.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001446084 2024-09-11T03:46:37.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.944931+0000", 2024-09-11T03:46:37.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107786 2024-09-11T03:46:37.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,9", 2024-09-11T03:46:37.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.945043+0000", 2024-09-11T03:46:37.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000112083 2024-09-11T03:46:37.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:46:37.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.946640+0000", 2024-09-11T03:46:37.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015976409999999999 2024-09-11T03:46:37.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:37.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.948221+0000", 2024-09-11T03:46:37.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015802559999999999 2024-09-11T03:46:37.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:37.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.950386+0000", 2024-09-11T03:46:37.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021654869999999998 2024-09-11T03:46:37.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.423 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:37.424 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:37 smithi084 ceph-mon[27254]: 2.17 scrub starts 2024-09-11T03:46:37.424 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:37 smithi084 ceph-mon[27254]: 2.17 scrub ok 2024-09-11T03:46:37.424 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:37 smithi084 ceph-mon[27254]: osdmap e578: 12 total, 12 up, 12 in 2024-09-11T03:46:37.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:37 smithi084 ceph-mon[27254]: 2.21 scrub starts 2024-09-11T03:46:37.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:37 smithi084 ceph-mon[27254]: 2.21 scrub ok 2024-09-11T03:46:37.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:37 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:46:37.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:37 smithi084 ceph-mon[27254]: 2.1 scrub ok 2024-09-11T03:46:37.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:37 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:46:37.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:37 smithi084 ceph-mon[27254]: 2.e scrub ok 2024-09-11T03:46:37.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2435943262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:37.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:46:37.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.950416+0000", 2024-09-11T03:46:37.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0086e-05 2024-09-11T03:46:37.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:43:39.950454+0000", 2024-09-11T03:46:37.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7293999999999999e-05 2024-09-11T03:46:37.426 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.426 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.426 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.427 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:4164 2.b e450/95 2:dde92fc6:::smithi17557870-41:head v 450'447, mlcod=449'443)", 2024-09-11T03:46:37.427 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:27.936778+0000", 2024-09-11T03:46:37.427 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.40795761699999, 2024-09-11T03:46:37.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069376189999999999, 2024-09-11T03:46:37.427 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.427 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.427 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:37.427 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.936778+0000", 2024-09-11T03:46:37.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.936778+0000", 2024-09-11T03:46:37.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.936778+0000", 2024-09-11T03:46:37.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8199999999999999e-07 2024-09-11T03:46:37.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.940955+0000", 2024-09-11T03:46:37.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041768120000000002 2024-09-11T03:46:37.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.940968+0000", 2024-09-11T03:46:37.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3757999999999999e-05 2024-09-11T03:46:37.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.940977+0000", 2024-09-11T03:46:37.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0310000000000004e-06 2024-09-11T03:46:37.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.941083+0000", 2024-09-11T03:46:37.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105611 2024-09-11T03:46:37.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.941103+0000", 2024-09-11T03:46:37.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9854999999999999e-05 2024-09-11T03:46:37.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:46:37.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.943669+0000", 2024-09-11T03:46:37.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025656979999999999 2024-09-11T03:46:37.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:27.943715+0000", 2024-09-11T03:46:37.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6672000000000003e-05 2024-09-11T03:46:37.431 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.431 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.431 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.431 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:4176 2.3 e451/276 2:c6c49f2c:::smithi17557870-44:head v 451'262, mlcod=449'258)", 2024-09-11T03:46:37.431 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:28.944463+0000", 2024-09-11T03:46:37.431 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 128.40027230800001, 2024-09-11T03:46:37.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0068327259999999999, 2024-09-11T03:46:37.432 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.432 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.432 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:37.432 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.944463+0000", 2024-09-11T03:46:37.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.944463+0000", 2024-09-11T03:46:37.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.944464+0000", 2024-09-11T03:46:37.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3200000000000003e-07 2024-09-11T03:46:37.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.946801+0000", 2024-09-11T03:46:37.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023370729999999998 2024-09-11T03:46:37.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.946844+0000", 2024-09-11T03:46:37.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3098e-05 2024-09-11T03:46:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.946861+0000", 2024-09-11T03:46:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6501e-05 2024-09-11T03:46:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.946957+0000", 2024-09-11T03:46:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6558000000000006e-05 2024-09-11T03:46:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.946984+0000", 2024-09-11T03:46:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6421999999999999e-05 2024-09-11T03:46:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:46:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.951247+0000", 2024-09-11T03:46:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042632119999999997 2024-09-11T03:46:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.951296+0000", 2024-09-11T03:46:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8930000000000001e-05 2024-09-11T03:46:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4288 2.0 2:0762776a:::smithi17557870-25:head [copy-from ver 488] snapc ba=[ba,b9,b8,b7,b6,b5,b1,af,ae,ac] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-11T03:46:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:39.973423+0000", 2024-09-11T03:46:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 117.37131259900001, 2024-09-11T03:46:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028521689999999999, 2024-09-11T03:46:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:46:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:46:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4288 2024-09-11T03:46:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973423+0000", 2024-09-11T03:46:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973423+0000", 2024-09-11T03:46:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973424+0000", 2024-09-11T03:46:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3599999999999999e-06 2024-09-11T03:46:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973431+0000", 2024-09-11T03:46:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.55e-06 2024-09-11T03:46:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973441+0000", 2024-09-11T03:46:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0383e-05 2024-09-11T03:46:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973446+0000", 2024-09-11T03:46:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3689999999999998e-06 2024-09-11T03:46:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973498+0000", 2024-09-11T03:46:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1412000000000002e-05 2024-09-11T03:46:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973592+0000", 2024-09-11T03:46:37.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4557999999999998e-05 2024-09-11T03:46:37.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10", 2024-09-11T03:46:37.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.987776+0000", 2024-09-11T03:46:37.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014183553 2024-09-11T03:46:37.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:46:37.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.991768+0000", 2024-09-11T03:46:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039917620000000003 2024-09-11T03:46:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.999033+0000", 2024-09-11T03:46:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072650620000000001 2024-09-11T03:46:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:40.001866+0000", 2024-09-11T03:46:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028328160000000002 2024-09-11T03:46:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:46:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:40.001905+0000", 2024-09-11T03:46:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9737999999999998e-05 2024-09-11T03:46:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:40.001944+0000", 2024-09-11T03:46:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9127e-05 2024-09-11T03:46:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4289 2.0 2:0762776a:::smithi17557870-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e462)", 2024-09-11T03:46:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:39.973461+0000", 2024-09-11T03:46:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 117.37127458099999, 2024-09-11T03:46:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017766608, 2024-09-11T03:46:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:46:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:46:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:46:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4289 2024-09-11T03:46:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973461+0000", 2024-09-11T03:46:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973461+0000", 2024-09-11T03:46:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973461+0000", 2024-09-11T03:46:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0399999999999996e-07 2024-09-11T03:46:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973467+0000", 2024-09-11T03:46:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8780000000000003e-06 2024-09-11T03:46:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973472+0000", 2024-09-11T03:46:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3009999999999997e-06 2024-09-11T03:46:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973477+0000", 2024-09-11T03:46:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7570000000000004e-06 2024-09-11T03:46:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973749+0000", 2024-09-11T03:46:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027214900000000001 2024-09-11T03:46:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:46:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973782+0000", 2024-09-11T03:46:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2543000000000003e-05 2024-09-11T03:46:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.991066+0000", 2024-09-11T03:46:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017284568 2024-09-11T03:46:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.991145+0000", 2024-09-11T03:46:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8498000000000006e-05 2024-09-11T03:46:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.991227+0000", 2024-09-11T03:46:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2410000000000005e-05 2024-09-11T03:46:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:4688 2.18 e503/312 2:1984a3a9:::smithi17557870-23:head v 503'349, mlcod=498'344)", 2024-09-11T03:46:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:45:21.011490+0000", 2024-09-11T03:46:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.333244903999997, 2024-09-11T03:46:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019963814, 2024-09-11T03:46:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.011490+0000", 2024-09-11T03:46:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.011490+0000", 2024-09-11T03:46:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.011491+0000", 2024-09-11T03:46:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7000000000000003e-07 2024-09-11T03:46:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030626+0000", 2024-09-11T03:46:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019134880999999999 2024-09-11T03:46:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030636+0000", 2024-09-11T03:46:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0159e-05 2024-09-11T03:46:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030642+0000", 2024-09-11T03:46:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4070000000000001e-06 2024-09-11T03:46:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030686+0000", 2024-09-11T03:46:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4320999999999998e-05 2024-09-11T03:46:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030710+0000", 2024-09-11T03:46:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3495000000000001e-05 2024-09-11T03:46:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:46:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.031422+0000", 2024-09-11T03:46:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000712588 2024-09-11T03:46:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.031454+0000", 2024-09-11T03:46:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2092999999999999e-05 2024-09-11T03:46:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4916 2.0 2:0762776a:::smithi17557870-25:head [read 0~2336272,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e523)", 2024-09-11T03:46:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:45:41.041017+0000", 2024-09-11T03:46:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 56.303717962, 2024-09-11T03:46:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087283320000000001, 2024-09-11T03:46:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:46:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:46:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:46:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4916 2024-09-11T03:46:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041017+0000", 2024-09-11T03:46:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041017+0000", 2024-09-11T03:46:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041019+0000", 2024-09-11T03:46:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0190000000000001e-06 2024-09-11T03:46:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041027+0000", 2024-09-11T03:46:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6599999999999995e-06 2024-09-11T03:46:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041053+0000", 2024-09-11T03:46:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5533000000000001e-05 2024-09-11T03:46:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041064+0000", 2024-09-11T03:46:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1080000000000001e-05 2024-09-11T03:46:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041459+0000", 2024-09-11T03:46:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039519099999999998 2024-09-11T03:46:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:46:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041531+0000", 2024-09-11T03:46:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2244999999999997e-05 2024-09-11T03:46:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.044916+0000", 2024-09-11T03:46:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003384691 2024-09-11T03:46:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.045007+0000", 2024-09-11T03:46:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1544000000000003e-05 2024-09-11T03:46:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.049746+0000", 2024-09-11T03:46:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047383690000000001 2024-09-11T03:46:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4917 2.0 2:0762776a:::smithi17557870-25:head [sparse-read 0~1097883] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e523)", 2024-09-11T03:46:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:45:41.041083+0000", 2024-09-11T03:46:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 56.303651973999997, 2024-09-11T03:46:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0088449059999999996, 2024-09-11T03:46:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:46:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:46:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:46:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4917 2024-09-11T03:46:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041083+0000", 2024-09-11T03:46:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041083+0000", 2024-09-11T03:46:37.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041084+0000", 2024-09-11T03:46:37.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8100000000000002e-07 2024-09-11T03:46:37.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041088+0000", 2024-09-11T03:46:37.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4180000000000001e-06 2024-09-11T03:46:37.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041096+0000", 2024-09-11T03:46:37.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0360000000000006e-06 2024-09-11T03:46:37.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041102+0000", 2024-09-11T03:46:37.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2639999999999997e-06 2024-09-11T03:46:37.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041576+0000", 2024-09-11T03:46:37.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000473888 2024-09-11T03:46:37.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:46:37.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041654+0000", 2024-09-11T03:46:37.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8444999999999999e-05 2024-09-11T03:46:37.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.049794+0000", 2024-09-11T03:46:37.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0081398319999999996 2024-09-11T03:46:37.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.049850+0000", 2024-09-11T03:46:37.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6498e-05 2024-09-11T03:46:37.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.049928+0000", 2024-09-11T03:46:37.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7744000000000006e-05 2024-09-11T03:46:37.463 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.463 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.463 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.463 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4918 2.0 2:0762776a:::smithi17557870-25:head [read 0~2336272,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e523)", 2024-09-11T03:46:37.463 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:45:41.041115+0000", 2024-09-11T03:46:37.464 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 56.303620058999996, 2024-09-11T03:46:37.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0098843630000000002, 2024-09-11T03:46:37.464 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:46:37.464 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:46:37.464 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:46:37.464 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:46:37.464 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:46:37.464 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:37.464 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4918 2024-09-11T03:46:37.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.464 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:46:37.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:46:37.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041115+0000", 2024-09-11T03:46:37.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:46:37.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041115+0000", 2024-09-11T03:46:37.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:46:37.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:46:37.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041116+0000", 2024-09-11T03:46:37.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6300000000000005e-07 2024-09-11T03:46:37.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:46:37.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041120+0000", 2024-09-11T03:46:37.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8349999999999997e-06 2024-09-11T03:46:37.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:46:37.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041127+0000", 2024-09-11T03:46:37.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0609999999999998e-06 2024-09-11T03:46:37.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:46:37.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041133+0000", 2024-09-11T03:46:37.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9089999999999997e-06 2024-09-11T03:46:37.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041703+0000", 2024-09-11T03:46:37.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057050799999999995 2024-09-11T03:46:37.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:46:37.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.041766+0000", 2024-09-11T03:46:37.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2815000000000001e-05 2024-09-11T03:46:37.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:46:37.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.049977+0000", 2024-09-11T03:46:37.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.008211065 2024-09-11T03:46:37.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:46:37.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.050032+0000", 2024-09-11T03:46:37.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4727e-05 2024-09-11T03:46:37.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:46:37.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:46:37.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:46:37.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:41.051000+0000", 2024-09-11T03:46:37.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00096787999999999996 2024-09-11T03:46:37.468 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.469 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.469 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.469 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:46:37.469 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:46:37.469 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:46:37.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:37.470 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:37.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:37 smithi175 ceph-mon[30894]: 2.17 scrub starts 2024-09-11T03:46:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:37 smithi175 ceph-mon[30894]: 2.17 scrub ok 2024-09-11T03:46:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:37 smithi175 ceph-mon[30894]: osdmap e578: 12 total, 12 up, 12 in 2024-09-11T03:46:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:37 smithi175 ceph-mon[30894]: 2.21 scrub starts 2024-09-11T03:46:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:37 smithi175 ceph-mon[30894]: 2.21 scrub ok 2024-09-11T03:46:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:37 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:46:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:37 smithi175 ceph-mon[30894]: 2.1 scrub ok 2024-09-11T03:46:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:37 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:46:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:37 smithi175 ceph-mon[30894]: 2.e scrub ok 2024-09-11T03:46:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:37 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2435943262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:37.572 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:37.598 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:37.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:37.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:37.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:37.721 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:37.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:37.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:37.824 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:37.847 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:37.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:37.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:37.948 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:37.971 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:37.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:37.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:38.032 INFO:tasks.rados.rados.2.smithi175.stdout:2162: done (0 left) 2024-09-11T03:46:38.032 INFO:tasks.rados.rados.2.smithi175.stdout:2163: delete oid 2 current snap is 241 2024-09-11T03:46:38.038 INFO:tasks.rados.rados.2.smithi175.stdout:2163: done (0 left) 2024-09-11T03:46:38.038 INFO:tasks.rados.rados.2.smithi175.stdout:2164: copy_from oid 44 from oid 22 current snap is 241 2024-09-11T03:46:38.038 INFO:tasks.rados.rados.2.smithi175.stdout:2165: copy_from oid 12 from oid 13 current snap is 241 2024-09-11T03:46:38.038 INFO:tasks.rados.rados.2.smithi175.stdout:2166: read oid 10 snap -1 2024-09-11T03:46:38.038 INFO:tasks.rados.rados.2.smithi175.stdout:2166: expect (ObjNum 726 snap 238 seq_num 726) 2024-09-11T03:46:38.038 INFO:tasks.rados.rados.2.smithi175.stdout:2167: rollback oid 41 current snap is 241 2024-09-11T03:46:38.038 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 41 to 238 2024-09-11T03:46:38.038 INFO:tasks.rados.rados.2.smithi175.stdout:2168: read oid 23 snap -1 2024-09-11T03:46:38.038 INFO:tasks.rados.rados.2.smithi175.stdout:2168: expect (ObjNum 639 snap 204 seq_num 639) 2024-09-11T03:46:38.039 INFO:tasks.rados.rados.2.smithi175.stdout:2169: write (excl) oid 7 current snap is 241 2024-09-11T03:46:38.039 INFO:tasks.rados.rados.2.smithi175.stdout:2169: seq_num 734 ranges {736338=402654,1552136=539580} 2024-09-11T03:46:38.045 INFO:tasks.rados.rados.2.smithi175.stdout:2169: writing smithi17557870-7 from 736338 to 1138992 tid 1 2024-09-11T03:46:38.052 INFO:tasks.rados.rados.2.smithi175.stdout:2169: writing smithi17557870-7 from 1552136 to 2091716 tid 2 2024-09-11T03:46:38.052 INFO:tasks.rados.rados.2.smithi175.stdout:2170: read oid 2 snap -1 2024-09-11T03:46:38.052 INFO:tasks.rados.rados.2.smithi175.stdout:2170: expect deleted 2024-09-11T03:46:38.052 INFO:tasks.rados.rados.2.smithi175.stdout:2171: rollback oid 46 current snap is 241 2024-09-11T03:46:38.052 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 46 to 239 2024-09-11T03:46:38.052 INFO:tasks.rados.rados.2.smithi175.stdout:2165: finishing copy_from to smithi17557870-12 2024-09-11T03:46:38.052 INFO:tasks.rados.rados.2.smithi175.stdout:2165: got expected ENOENT (src dne) 2024-09-11T03:46:38.052 INFO:tasks.rados.rados.2.smithi175.stdout:2165: finishing copy_from racing read to smithi17557870-12 2024-09-11T03:46:38.052 INFO:tasks.rados.rados.2.smithi175.stdout:2166: expect (ObjNum 726 snap 238 seq_num 726) 2024-09-11T03:46:38.073 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:38.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:38.095 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:38.095 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:38.123 INFO:tasks.rados.rados.2.smithi175.stdout:2167: finishing rollback tid 0 to smithi17557870-41 2024-09-11T03:46:38.124 INFO:tasks.rados.rados.2.smithi175.stdout:2167: finishing rollback tid 1 to smithi17557870-41 2024-09-11T03:46:38.124 INFO:tasks.rados.rados.2.smithi175.stdout:2168: expect (ObjNum 639 snap 204 seq_num 639) 2024-09-11T03:46:38.196 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:38.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:38.218 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:38.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:38.263 INFO:tasks.rados.rados.2.smithi175.stdout:2167: finishing rollback tid 2 to smithi17557870-41 2024-09-11T03:46:38.263 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 660 (ObjNum 609 snap 191 seq_num 609) dirty exists 2024-09-11T03:46:38.263 INFO:tasks.rados.rados.2.smithi175.stdout:2169: finishing write tid 1 to smithi17557870-7 2024-09-11T03:46:38.263 INFO:tasks.rados.rados.2.smithi175.stdout:2164: finishing copy_from racing read to smithi17557870-44 2024-09-11T03:46:38.263 INFO:tasks.rados.rados.2.smithi175.stdout:2164: finishing copy_from to smithi17557870-44 2024-09-11T03:46:38.263 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 803 (ObjNum 695 snap 227 seq_num 695) dirty exists 2024-09-11T03:46:38.263 INFO:tasks.rados.rados.2.smithi175.stdout:2171: finishing rollback tid 1 to smithi17557870-46 2024-09-11T03:46:38.264 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 539 (ObjNum 129 snap 0 seq_num 1746337064) dirty dne 2024-09-11T03:46:38.264 INFO:tasks.rados.rados.2.smithi175.stdout:2169: finishing write tid 2 to smithi17557870-7 2024-09-11T03:46:38.264 INFO:tasks.rados.rados.2.smithi175.stdout:2164: done (7 left) 2024-09-11T03:46:38.264 INFO:tasks.rados.rados.2.smithi175.stdout:2165: done (6 left) 2024-09-11T03:46:38.264 INFO:tasks.rados.rados.2.smithi175.stdout:2166: done (5 left) 2024-09-11T03:46:38.264 INFO:tasks.rados.rados.2.smithi175.stdout:2167: done (4 left) 2024-09-11T03:46:38.264 INFO:tasks.rados.rados.2.smithi175.stdout:2168: done (3 left) 2024-09-11T03:46:38.264 INFO:tasks.rados.rados.2.smithi175.stdout:2170: done (2 left) 2024-09-11T03:46:38.264 INFO:tasks.rados.rados.2.smithi175.stdout:2171: done (1 left) 2024-09-11T03:46:38.264 INFO:tasks.rados.rados.2.smithi175.stdout:2172: delete oid 39 current snap is 241 2024-09-11T03:46:38.264 INFO:tasks.rados.rados.2.smithi175.stdout:2169: finishing write tid 3 to smithi17557870-7 2024-09-11T03:46:38.264 INFO:tasks.rados.rados.2.smithi175.stdout:2169: finishing write tid 4 to smithi17557870-7 2024-09-11T03:46:38.264 INFO:tasks.rados.rados.2.smithi175.stdout:2169: oid 7 updating version 0 to 527 2024-09-11T03:46:38.265 INFO:tasks.rados.rados.2.smithi175.stdout:2169: oid 7 version 527 is already newer than 526 2024-09-11T03:46:38.265 INFO:tasks.rados.rados.2.smithi175.stdout:2169: oid 7 version 527 is already newer than 525 2024-09-11T03:46:38.265 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 527 (ObjNum 734 snap 241 seq_num 734) dirty exists 2024-09-11T03:46:38.265 INFO:tasks.rados.rados.2.smithi175.stdout:2169: left oid 7 (ObjNum 734 snap 241 seq_num 734) 2024-09-11T03:46:38.265 INFO:tasks.rados.rados.2.smithi175.stdout:2169: done (1 left) 2024-09-11T03:46:38.265 INFO:tasks.rados.rados.2.smithi175.stdout:2172: done (0 left) 2024-09-11T03:46:38.265 INFO:tasks.rados.rados.2.smithi175.stdout:2173: snap_create 2024-09-11T03:46:38.319 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:38.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:38.345 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:38.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:38.355 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:46:38.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:38 smithi084 ceph-mon[27254]: 1.14 scrub starts 2024-09-11T03:46:38.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:38 smithi084 ceph-mon[27254]: 1.14 scrub ok 2024-09-11T03:46:38.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:38 smithi084 ceph-mon[27254]: pgmap v1023: 67 pgs: 5 peering, 62 active+clean; 141 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:46:38.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:38 smithi084 ceph-mon[27254]: osdmap e579: 12 total, 12 up, 12 in 2024-09-11T03:46:38.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:38 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:46:38.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:38 smithi084 ceph-mon[27254]: 2.15 scrub ok 2024-09-11T03:46:38.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:38 smithi121 ceph-mon[30729]: 1.14 scrub starts 2024-09-11T03:46:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:38 smithi121 ceph-mon[30729]: 1.14 scrub ok 2024-09-11T03:46:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:38 smithi121 ceph-mon[30729]: pgmap v1023: 67 pgs: 5 peering, 62 active+clean; 141 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:46:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:38 smithi121 ceph-mon[30729]: osdmap e579: 12 total, 12 up, 12 in 2024-09-11T03:46:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:38 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:46:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:38 smithi121 ceph-mon[30729]: 2.15 scrub ok 2024-09-11T03:46:38.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:38.468 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-11T03:46:38.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:38.470 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:38.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:38 smithi175 ceph-mon[30894]: 1.14 scrub starts 2024-09-11T03:46:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:38 smithi175 ceph-mon[30894]: 1.14 scrub ok 2024-09-11T03:46:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:38 smithi175 ceph-mon[30894]: pgmap v1023: 67 pgs: 5 peering, 62 active+clean; 141 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:46:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:38 smithi175 ceph-mon[30894]: osdmap e579: 12 total, 12 up, 12 in 2024-09-11T03:46:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:38 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:46:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:38 smithi175 ceph-mon[30894]: 2.15 scrub ok 2024-09-11T03:46:38.571 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:38.594 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:38.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:38.595 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:38.634 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:38.696 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:38.713 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:46:38.717 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:38.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:38.721 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:38.822 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:38.843 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:38.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:38.844 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:38.945 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:38.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:38.966 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:38.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:39.045 INFO:tasks.rados.rados.2.smithi175.stdout:2173: done (0 left) 2024-09-11T03:46:39.045 INFO:tasks.rados.rados.2.smithi175.stdout:2174: snap_create 2024-09-11T03:46:39.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:39.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:39.091 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:39.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:39.156 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:39 smithi175 ceph-mon[30894]: 1.15 scrub starts 2024-09-11T03:46:39.156 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:39 smithi175 ceph-mon[30894]: 1.15 scrub ok 2024-09-11T03:46:39.156 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:39 smithi175 ceph-mon[30894]: 1.10 scrub starts 2024-09-11T03:46:39.156 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:39 smithi175 ceph-mon[30894]: 1.10 scrub ok 2024-09-11T03:46:39.156 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:39 smithi175 ceph-mon[30894]: osdmap e580: 12 total, 12 up, 12 in 2024-09-11T03:46:39.175 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:46:39.175 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:46:39.176 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:46:39.176 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:46:39.191 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:39.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:39.218 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:39.218 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:39.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:39 smithi084 ceph-mon[27254]: 1.15 scrub starts 2024-09-11T03:46:39.295 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:39 smithi084 ceph-mon[27254]: 1.15 scrub ok 2024-09-11T03:46:39.295 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:39 smithi084 ceph-mon[27254]: 1.10 scrub starts 2024-09-11T03:46:39.295 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:39 smithi084 ceph-mon[27254]: 1.10 scrub ok 2024-09-11T03:46:39.295 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:39 smithi084 ceph-mon[27254]: osdmap e580: 12 total, 12 up, 12 in 2024-09-11T03:46:39.315 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:39 smithi121 ceph-mon[30729]: 1.15 scrub starts 2024-09-11T03:46:39.315 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:39 smithi121 ceph-mon[30729]: 1.15 scrub ok 2024-09-11T03:46:39.315 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:39 smithi121 ceph-mon[30729]: 1.10 scrub starts 2024-09-11T03:46:39.315 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:39 smithi121 ceph-mon[30729]: 1.10 scrub ok 2024-09-11T03:46:39.315 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:39 smithi121 ceph-mon[30729]: osdmap e580: 12 total, 12 up, 12 in 2024-09-11T03:46:39.318 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:39.341 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:39.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:39.442 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:39.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:39.465 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:39.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:39.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:39.589 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:39.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:39.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:39.691 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:39.714 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:39.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:39.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:39.769 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:46:39.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:39.839 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:39.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:39.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:39.912 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-11T03:46:39.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:39.962 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:39.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:39.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:40.023 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:40.040 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:46:40.044 INFO:tasks.rados.rados.2.smithi175.stdout:2174: done (0 left) 2024-09-11T03:46:40.044 INFO:tasks.rados.rados.2.smithi175.stdout:2175: snap_remove snap 241 2024-09-11T03:46:40.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:40.086 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:40.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:40.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:40.169 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:46:40.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:40.209 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:40.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:40.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:40 smithi175 ceph-mon[30894]: pgmap v1026: 67 pgs: 67 active+clean; 125 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:46:40.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:40 smithi175 ceph-mon[30894]: osdmap e581: 12 total, 12 up, 12 in 2024-09-11T03:46:40.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:40 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2804011321' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:46:40.299 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:40 smithi084 ceph-mon[27254]: pgmap v1026: 67 pgs: 67 active+clean; 125 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:46:40.299 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:40 smithi084 ceph-mon[27254]: osdmap e581: 12 total, 12 up, 12 in 2024-09-11T03:46:40.299 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:40 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2804011321' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:46:40.299 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:40 smithi121 ceph-mon[30729]: pgmap v1026: 67 pgs: 67 active+clean; 125 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:46:40.299 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:40 smithi121 ceph-mon[30729]: osdmap e581: 12 total, 12 up, 12 in 2024-09-11T03:46:40.299 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:40 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2804011321' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:46:40.311 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:40.333 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:40.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:40.334 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:40.435 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:40.456 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:40.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:40.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:40.558 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:40.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:40.579 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:40.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:40.646 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:46:40.646 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:46:40.646 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:46:40.646 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:46:40.646 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:46:40.681 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:40.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:40.705 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:40.705 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:40.767 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:46:40.767 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-11T03:46:40.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:40.828 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:40.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:40.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:40.930 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:40.951 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:40.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:40.952 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:40.965 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:46:40.966 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:46:40.982 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:46:40.983 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:46:40.998 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:46:40.999 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:46:41.015 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:46:41.015 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:46:41.031 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:46:41.031 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:46:41.046 INFO:tasks.rados.rados.2.smithi175.stdout:2175: done (0 left) 2024-09-11T03:46:41.047 INFO:tasks.rados.rados.2.smithi175.stdout:2176: copy_from oid 19 from oid 36 current snap is 243 2024-09-11T03:46:41.047 INFO:tasks.rados.rados.2.smithi175.stdout:2177: write (excl) oid 29 current snap is 243 2024-09-11T03:46:41.047 INFO:tasks.rados.rados.2.smithi175.stdout:2177: seq_num 736 ranges {769952=553152,2069100=300852} 2024-09-11T03:46:41.052 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:46:41.053 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:46:41.053 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:41.054 INFO:tasks.rados.rados.2.smithi175.stdout:2177: writing smithi17557870-29 from 769952 to 1323104 tid 1 2024-09-11T03:46:41.060 INFO:tasks.rados.rados.2.smithi175.stdout:2177: writing smithi17557870-29 from 2069100 to 2369952 tid 2 2024-09-11T03:46:41.060 INFO:tasks.rados.rados.2.smithi175.stdout:2178: write (excl) oid 30 current snap is 243 2024-09-11T03:46:41.060 INFO:tasks.rados.rados.2.smithi175.stdout:2176: finishing copy_from to smithi17557870-19 2024-09-11T03:46:41.060 INFO:tasks.rados.rados.2.smithi175.stdout:2176: got expected ENOENT (src dne) 2024-09-11T03:46:41.060 INFO:tasks.rados.rados.2.smithi175.stdout:2177: finishing write tid 1 to smithi17557870-29 2024-09-11T03:46:41.060 INFO:tasks.rados.rados.2.smithi175.stdout:2176: finishing copy_from racing read to smithi17557870-19 2024-09-11T03:46:41.060 INFO:tasks.rados.rados.2.smithi175.stdout:2178: seq_num 737 ranges {786759=628401,2011229=375530} 2024-09-11T03:46:41.067 INFO:tasks.rados.rados.2.smithi175.stdout:2178: writing smithi17557870-30 from 786759 to 1415160 tid 1 2024-09-11T03:46:41.073 INFO:tasks.rados.rados.2.smithi175.stdout:2178: writing smithi17557870-30 from 2011229 to 2386759 tid 2 2024-09-11T03:46:41.073 INFO:tasks.rados.rados.2.smithi175.stdout:2176: done (2 left) 2024-09-11T03:46:41.073 INFO:tasks.rados.rados.2.smithi175.stdout:2179: snap_remove snap 232 2024-09-11T03:46:41.074 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:46:41.074 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:46:41.078 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:41.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:41.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:41.091 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:46:41.091 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:46:41.108 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:46:41.108 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:46:41.125 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:46:41.125 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:46:41.141 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:46:41.142 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:46:41.158 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:46:41.159 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:46:41.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:41.201 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:41.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:41.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:41.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:41 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:41.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:41 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2804011321' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:46:41.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:41 smithi175 ceph-mon[30894]: osdmap e582: 12 total, 12 up, 12 in 2024-09-11T03:46:41.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:41 smithi175 ceph-mon[30894]: 1.2 scrub starts 2024-09-11T03:46:41.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:41 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/314749792' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:41.276 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-11T03:46:41.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:41.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:41 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:41.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:41 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2804011321' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:46:41.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:41 smithi084 ceph-mon[27254]: osdmap e582: 12 total, 12 up, 12 in 2024-09-11T03:46:41.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:41 smithi084 ceph-mon[27254]: 1.2 scrub starts 2024-09-11T03:46:41.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:41 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/314749792' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:41.325 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:41.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:41.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:41 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:41 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2804011321' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:46:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:41 smithi121 ceph-mon[30729]: osdmap e582: 12 total, 12 up, 12 in 2024-09-11T03:46:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:41 smithi121 ceph-mon[30729]: 1.2 scrub starts 2024-09-11T03:46:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:41 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/314749792' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:41.427 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:41.449 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:41.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:41.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:41.515 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:46:41.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:41.573 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:41.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:41.675 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:41.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:41.697 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:41.698 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:41.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:41.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:41.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:41.927 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:41.949 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:41.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:41.950 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:41.973 INFO:tasks.rados.rados.2.smithi175.stdout:2179: done (2 left) 2024-09-11T03:46:41.973 INFO:tasks.rados.rados.2.smithi175.stdout:2180: write oid 44 current snap is 243 2024-09-11T03:46:41.974 INFO:tasks.rados.rados.2.smithi175.stdout:2177: finishing write tid 2 to smithi17557870-29 2024-09-11T03:46:41.974 INFO:tasks.rados.rados.2.smithi175.stdout:2177: finishing write tid 3 to smithi17557870-29 2024-09-11T03:46:41.974 INFO:tasks.rados.rados.2.smithi175.stdout:2177: finishing write tid 4 to smithi17557870-29 2024-09-11T03:46:41.974 INFO:tasks.rados.rados.2.smithi175.stdout:2177: oid 29 updating version 0 to 824 2024-09-11T03:46:41.974 INFO:tasks.rados.rados.2.smithi175.stdout:2177: oid 29 updating version 824 to 825 2024-09-11T03:46:41.974 INFO:tasks.rados.rados.2.smithi175.stdout:2177: oid 29 updating version 825 to 826 2024-09-11T03:46:41.974 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 826 (ObjNum 736 snap 243 seq_num 736) dirty exists 2024-09-11T03:46:41.974 INFO:tasks.rados.rados.2.smithi175.stdout:2177: left oid 29 (ObjNum 736 snap 243 seq_num 736) 2024-09-11T03:46:41.974 INFO:tasks.rados.rados.2.smithi175.stdout:2178: finishing write tid 1 to smithi17557870-30 2024-09-11T03:46:41.974 INFO:tasks.rados.rados.2.smithi175.stdout:2178: finishing write tid 2 to smithi17557870-30 2024-09-11T03:46:41.974 INFO:tasks.rados.rados.2.smithi175.stdout:2178: finishing write tid 3 to smithi17557870-30 2024-09-11T03:46:41.974 INFO:tasks.rados.rados.2.smithi175.stdout:2178: finishing write tid 4 to smithi17557870-30 2024-09-11T03:46:41.975 INFO:tasks.rados.rados.2.smithi175.stdout:2178: oid 30 updating version 0 to 851 2024-09-11T03:46:41.975 INFO:tasks.rados.rados.2.smithi175.stdout:2178: oid 30 updating version 851 to 853 2024-09-11T03:46:41.975 INFO:tasks.rados.rados.2.smithi175.stdout:2178: oid 30 version 853 is already newer than 852 2024-09-11T03:46:41.975 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 853 (ObjNum 737 snap 243 seq_num 737) dirty exists 2024-09-11T03:46:41.975 INFO:tasks.rados.rados.2.smithi175.stdout:2178: left oid 30 (ObjNum 737 snap 243 seq_num 737) 2024-09-11T03:46:41.975 INFO:tasks.rados.rados.2.smithi175.stdout:2180: seq_num 738 ranges {403566=420003,1386064=546918} 2024-09-11T03:46:41.978 INFO:tasks.rados.rados.2.smithi175.stdout:2180: writing smithi17557870-44 from 403566 to 823569 tid 1 2024-09-11T03:46:41.985 INFO:tasks.rados.rados.2.smithi175.stdout:2180: writing smithi17557870-44 from 1386064 to 1932982 tid 2 2024-09-11T03:46:41.985 INFO:tasks.rados.rados.2.smithi175.stdout:2177: done (2 left) 2024-09-11T03:46:41.985 INFO:tasks.rados.rados.2.smithi175.stdout:2178: done (1 left) 2024-09-11T03:46:41.985 INFO:tasks.rados.rados.2.smithi175.stdout:2181: copy_from oid 41 from oid 21 current snap is 243 2024-09-11T03:46:41.985 INFO:tasks.rados.rados.2.smithi175.stdout:2180: finishing write tid 1 to smithi17557870-44 2024-09-11T03:46:41.985 INFO:tasks.rados.rados.2.smithi175.stdout:2182: snap_remove snap 240 2024-09-11T03:46:41.995 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:46:41.995 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:46:42.051 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:42.075 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:42.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:42.082 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:42.177 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:42.198 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:42.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:42.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:42 smithi175 ceph-mon[30894]: pgmap v1029: 67 pgs: 67 active+clean; 125 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:46:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:42 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:46:42.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:42 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:46:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:42 smithi175 ceph-mon[30894]: osdmap e583: 12 total, 12 up, 12 in 2024-09-11T03:46:42.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:42 smithi175 ceph-mon[30894]: osdmap e584: 12 total, 12 up, 12 in 2024-09-11T03:46:42.300 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:42.321 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:42.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:42.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:42.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:42 smithi084 ceph-mon[27254]: pgmap v1029: 67 pgs: 67 active+clean; 125 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:46:42.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:42 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:46:42.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:42 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:46:42.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:42 smithi084 ceph-mon[27254]: osdmap e583: 12 total, 12 up, 12 in 2024-09-11T03:46:42.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:42 smithi084 ceph-mon[27254]: osdmap e584: 12 total, 12 up, 12 in 2024-09-11T03:46:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:42 smithi121 ceph-mon[30729]: pgmap v1029: 67 pgs: 67 active+clean; 125 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:46:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:42 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:46:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:42 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:46:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:42 smithi121 ceph-mon[30729]: osdmap e583: 12 total, 12 up, 12 in 2024-09-11T03:46:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:42 smithi121 ceph-mon[30729]: osdmap e584: 12 total, 12 up, 12 in 2024-09-11T03:46:42.423 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:42.444 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:42.445 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:42.546 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:42.567 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:42.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:42.568 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:42.609 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-11T03:46:42.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:42.695 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:42.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:42.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:42.797 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:42.819 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:42.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:42.889 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:46:42.920 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:42.941 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:42.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:42.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:42.965 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T03:46:42.969 INFO:tasks.rados.rados.2.smithi175.stdout:2182: done (2 left) 2024-09-11T03:46:42.969 INFO:tasks.rados.rados.2.smithi175.stdout:2183: delete oid 3 current snap is 243 2024-09-11T03:46:42.970 INFO:tasks.rados.rados.2.smithi175.stdout:2180: finishing write tid 2 to smithi17557870-44 2024-09-11T03:46:42.970 INFO:tasks.rados.rados.2.smithi175.stdout:2180: finishing write tid 3 to smithi17557870-44 2024-09-11T03:46:42.970 INFO:tasks.rados.rados.2.smithi175.stdout:2180: finishing write tid 4 to smithi17557870-44 2024-09-11T03:46:42.970 INFO:tasks.rados.rados.2.smithi175.stdout:2180: oid 44 updating version 0 to 805 2024-09-11T03:46:42.970 INFO:tasks.rados.rados.2.smithi175.stdout:2180: oid 44 updating version 805 to 806 2024-09-11T03:46:42.970 INFO:tasks.rados.rados.2.smithi175.stdout:2180: oid 44 version 806 is already newer than 804 2024-09-11T03:46:42.970 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 806 (ObjNum 738 snap 243 seq_num 738) dirty exists 2024-09-11T03:46:42.970 INFO:tasks.rados.rados.2.smithi175.stdout:2180: left oid 44 (ObjNum 738 snap 243 seq_num 738) 2024-09-11T03:46:42.970 INFO:tasks.rados.rados.2.smithi175.stdout:2181: finishing copy_from racing read to smithi17557870-41 2024-09-11T03:46:42.970 INFO:tasks.rados.rados.2.smithi175.stdout:2181: finishing copy_from to smithi17557870-41 2024-09-11T03:46:42.970 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 854 (ObjNum 708 snap 232 seq_num 708) dirty exists 2024-09-11T03:46:42.972 INFO:tasks.rados.rados.2.smithi175.stdout:2180: done (2 left) 2024-09-11T03:46:42.972 INFO:tasks.rados.rados.2.smithi175.stdout:2181: done (1 left) 2024-09-11T03:46:42.972 INFO:tasks.rados.rados.2.smithi175.stdout:2183: done (0 left) 2024-09-11T03:46:42.972 INFO:tasks.rados.rados.2.smithi175.stdout:2184: copy_from oid 37 from oid 1 current snap is 243 2024-09-11T03:46:42.972 INFO:tasks.rados.rados.2.smithi175.stdout:2185: delete oid 34 current snap is 243 2024-09-11T03:46:42.974 INFO:tasks.rados.rados.2.smithi175.stdout:2185: done (1 left) 2024-09-11T03:46:42.974 INFO:tasks.rados.rados.2.smithi175.stdout:2186: rollback oid 47 current snap is 243 2024-09-11T03:46:42.974 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 47 to 239 2024-09-11T03:46:42.975 INFO:tasks.rados.rados.2.smithi175.stdout:2187: read oid 2 snap 242 2024-09-11T03:46:42.975 INFO:tasks.rados.rados.2.smithi175.stdout:2187: expect deleted 2024-09-11T03:46:42.975 INFO:tasks.rados.rados.2.smithi175.stdout:2188: read oid 9 snap 239 2024-09-11T03:46:42.975 INFO:tasks.rados.rados.2.smithi175.stdout:2188: expect deleted 2024-09-11T03:46:42.975 INFO:tasks.rados.rados.2.smithi175.stdout:2189: copy_from oid 33 from oid 34 current snap is 243 2024-09-11T03:46:42.975 INFO:tasks.rados.rados.2.smithi175.stdout:2190: read oid 44 snap 238 2024-09-11T03:46:42.975 INFO:tasks.rados.rados.2.smithi175.stdout:2190: expect deleted 2024-09-11T03:46:42.975 INFO:tasks.rados.rados.2.smithi175.stdout:2191: read oid 22 snap -1 2024-09-11T03:46:42.975 INFO:tasks.rados.rados.2.smithi175.stdout:2191: expect (ObjNum 695 snap 227 seq_num 695) 2024-09-11T03:46:42.975 INFO:tasks.rados.rados.2.smithi175.stdout:2192: read oid 32 snap 242 2024-09-11T03:46:42.975 INFO:tasks.rados.rados.2.smithi175.stdout:2192: expect (ObjNum 721 snap 236 seq_num 721) 2024-09-11T03:46:42.975 INFO:tasks.rados.rados.2.smithi175.stdout:2193: write (excl) oid 3 current snap is 243 2024-09-11T03:46:42.975 INFO:tasks.rados.rados.2.smithi175.stdout:2193: seq_num 741 ranges {453987=645750,1845157=608830} 2024-09-11T03:46:42.981 INFO:tasks.rados.rados.2.smithi175.stdout:2193: writing smithi17557870-3 from 453987 to 1099737 tid 1 2024-09-11T03:46:42.989 INFO:tasks.rados.rados.2.smithi175.stdout:2193: writing smithi17557870-3 from 1845157 to 2453987 tid 2 2024-09-11T03:46:42.989 INFO:tasks.rados.rados.2.smithi175.stdout:2194: delete oid 4 current snap is 243 2024-09-11T03:46:42.990 INFO:tasks.rados.rados.2.smithi175.stdout:2186: finishing rollback tid 1 to smithi17557870-47 2024-09-11T03:46:42.990 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 0 (ObjNum 81 snap 0 seq_num 197157682) dirty dne 2024-09-11T03:46:42.990 INFO:tasks.rados.rados.2.smithi175.stdout:2192: expect (ObjNum 721 snap 236 seq_num 721) 2024-09-11T03:46:43.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:43.065 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:43.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:43.168 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:43.172 INFO:tasks.rados.rados.2.smithi175.stdout:2191: expect (ObjNum 695 snap 227 seq_num 695) 2024-09-11T03:46:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:43.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:43.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:43.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:43 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/729537126' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:46:43.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:43 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/729537126' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:46:43.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:43 smithi084 ceph-mon[27254]: osdmap e585: 12 total, 12 up, 12 in 2024-09-11T03:46:43.277 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:43 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/729537126' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:46:43.277 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:43 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/729537126' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:46:43.277 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:43 smithi175 ceph-mon[30894]: osdmap e585: 12 total, 12 up, 12 in 2024-09-11T03:46:43.291 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:43.313 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:43.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:43.314 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:43.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:46:43.350 INFO:tasks.rados.rados.2.smithi175.stdout:2189: finishing copy_from to smithi17557870-33 2024-09-11T03:46:43.350 INFO:tasks.rados.rados.2.smithi175.stdout:2189: got expected ENOENT (src dne) 2024-09-11T03:46:43.350 INFO:tasks.rados.rados.2.smithi175.stdout:2189: finishing copy_from racing read to smithi17557870-33 2024-09-11T03:46:43.350 INFO:tasks.rados.rados.2.smithi175.stdout:2193: finishing write tid 1 to smithi17557870-3 2024-09-11T03:46:43.350 INFO:tasks.rados.rados.2.smithi175.stdout:2184: finishing copy_from racing read to smithi17557870-37 2024-09-11T03:46:43.350 INFO:tasks.rados.rados.2.smithi175.stdout:2193: finishing write tid 2 to smithi17557870-3 2024-09-11T03:46:43.350 INFO:tasks.rados.rados.2.smithi175.stdout:2193: finishing write tid 3 to smithi17557870-3 2024-09-11T03:46:43.350 INFO:tasks.rados.rados.2.smithi175.stdout:2184: finishing copy_from to smithi17557870-37 2024-09-11T03:46:43.351 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 800 (ObjNum 713 snap 233 seq_num 713) dirty exists 2024-09-11T03:46:43.351 INFO:tasks.rados.rados.2.smithi175.stdout:2193: finishing write tid 4 to smithi17557870-3 2024-09-11T03:46:43.351 INFO:tasks.rados.rados.2.smithi175.stdout:2193: oid 3 updating version 0 to 880 2024-09-11T03:46:43.351 INFO:tasks.rados.rados.2.smithi175.stdout:2193: oid 3 version 880 is already newer than 879 2024-09-11T03:46:43.351 INFO:tasks.rados.rados.2.smithi175.stdout:2193: oid 3 updating version 880 to 881 2024-09-11T03:46:43.351 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 881 (ObjNum 741 snap 243 seq_num 741) dirty exists 2024-09-11T03:46:43.351 INFO:tasks.rados.rados.2.smithi175.stdout:2193: left oid 3 (ObjNum 741 snap 243 seq_num 741) 2024-09-11T03:46:43.351 INFO:tasks.rados.rados.2.smithi175.stdout:2184: done (9 left) 2024-09-11T03:46:43.351 INFO:tasks.rados.rados.2.smithi175.stdout:2186: done (8 left) 2024-09-11T03:46:43.351 INFO:tasks.rados.rados.2.smithi175.stdout:2187: done (7 left) 2024-09-11T03:46:43.351 INFO:tasks.rados.rados.2.smithi175.stdout:2188: done (6 left) 2024-09-11T03:46:43.351 INFO:tasks.rados.rados.2.smithi175.stdout:2189: done (5 left) 2024-09-11T03:46:43.351 INFO:tasks.rados.rados.2.smithi175.stdout:2190: done (4 left) 2024-09-11T03:46:43.351 INFO:tasks.rados.rados.2.smithi175.stdout:2191: done (3 left) 2024-09-11T03:46:43.352 INFO:tasks.rados.rados.2.smithi175.stdout:2192: done (2 left) 2024-09-11T03:46:43.352 INFO:tasks.rados.rados.2.smithi175.stdout:2193: done (1 left) 2024-09-11T03:46:43.352 INFO:tasks.rados.rados.2.smithi175.stdout:2194: done (0 left) 2024-09-11T03:46:43.352 INFO:tasks.rados.rados.2.smithi175.stdout:2195: rollback oid 17 current snap is 243 2024-09-11T03:46:43.352 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 17 to 238 2024-09-11T03:46:43.352 INFO:tasks.rados.rados.2.smithi175.stdout:2196: copy_from oid 37 from oid 44 current snap is 243 2024-09-11T03:46:43.352 INFO:tasks.rados.rados.2.smithi175.stdout:2197: rollback oid 15 current snap is 243 2024-09-11T03:46:43.352 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 15 to 236 2024-09-11T03:46:43.352 INFO:tasks.rados.rados.2.smithi175.stdout:2198: write (excl) oid 49 current snap is 243 2024-09-11T03:46:43.352 INFO:tasks.rados.rados.2.smithi175.stdout:2198: seq_num 743 ranges {487601=796248,1962121=525480} 2024-09-11T03:46:43.356 INFO:tasks.rados.rados.2.smithi175.stdout:2198: writing smithi17557870-49 from 487601 to 1283849 tid 1 2024-09-11T03:46:43.364 INFO:tasks.rados.rados.2.smithi175.stdout:2198: writing smithi17557870-49 from 1962121 to 2487601 tid 2 2024-09-11T03:46:43.364 INFO:tasks.rados.rados.2.smithi175.stdout:2199: rollback oid 40 current snap is 243 2024-09-11T03:46:43.364 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 40 to 238 2024-09-11T03:46:43.364 INFO:tasks.rados.rados.2.smithi175.stdout:2195: finishing rollback tid 1 to smithi17557870-17 2024-09-11T03:46:43.364 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 0 (ObjNum 81 snap 0 seq_num 1) dirty dne 2024-09-11T03:46:43.364 INFO:tasks.rados.rados.2.smithi175.stdout:2197: finishing rollback tid 1 to smithi17557870-15 2024-09-11T03:46:43.364 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 0 (ObjNum 129 snap 0 seq_num 1745049184) dirty dne 2024-09-11T03:46:43.364 INFO:tasks.rados.rados.2.smithi175.stdout:2196: finishing copy_from racing read to smithi17557870-37 2024-09-11T03:46:43.365 INFO:tasks.rados.rados.2.smithi175.stdout:2196: finishing copy_from to smithi17557870-37 2024-09-11T03:46:43.365 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 806 (ObjNum 738 snap 243 seq_num 738) dirty exists 2024-09-11T03:46:43.365 INFO:tasks.rados.rados.2.smithi175.stdout:2198: finishing write tid 1 to smithi17557870-49 2024-09-11T03:46:43.365 INFO:tasks.rados.rados.2.smithi175.stdout:2195: done (4 left) 2024-09-11T03:46:43.365 INFO:tasks.rados.rados.2.smithi175.stdout:2196: done (3 left) 2024-09-11T03:46:43.365 INFO:tasks.rados.rados.2.smithi175.stdout:2197: done (2 left) 2024-09-11T03:46:43.365 INFO:tasks.rados.rados.2.smithi175.stdout:2200: read oid 39 snap -1 2024-09-11T03:46:43.365 INFO:tasks.rados.rados.2.smithi175.stdout:2200: expect deleted 2024-09-11T03:46:43.365 INFO:tasks.rados.rados.2.smithi175.stdout:2201: read oid 10 snap 242 2024-09-11T03:46:43.365 INFO:tasks.rados.rados.2.smithi175.stdout:2201: expect (ObjNum 726 snap 238 seq_num 726) 2024-09-11T03:46:43.365 INFO:tasks.rados.rados.2.smithi175.stdout:2202: rollback oid 11 current snap is 243 2024-09-11T03:46:43.365 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 11 to 238 2024-09-11T03:46:43.365 INFO:tasks.rados.rados.2.smithi175.stdout:2203: snap_create 2024-09-11T03:46:43.366 INFO:tasks.rados.rados.2.smithi175.stdout:2199: finishing rollback tid 0 to smithi17557870-40 2024-09-11T03:46:43.366 INFO:tasks.rados.rados.2.smithi175.stdout:2199: finishing rollback tid 1 to smithi17557870-40 2024-09-11T03:46:43.367 INFO:tasks.rados.rados.2.smithi175.stdout:2199: finishing rollback tid 2 to smithi17557870-40 2024-09-11T03:46:43.367 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 417 (ObjNum 697 snap 227 seq_num 697) dirty exists 2024-09-11T03:46:43.367 INFO:tasks.rados.rados.2.smithi175.stdout:2198: finishing write tid 2 to smithi17557870-49 2024-09-11T03:46:43.367 INFO:tasks.rados.rados.2.smithi175.stdout:2198: finishing write tid 3 to smithi17557870-49 2024-09-11T03:46:43.367 INFO:tasks.rados.rados.2.smithi175.stdout:2198: finishing write tid 4 to smithi17557870-49 2024-09-11T03:46:43.367 INFO:tasks.rados.rados.2.smithi175.stdout:2198: oid 49 updating version 0 to 679 2024-09-11T03:46:43.367 INFO:tasks.rados.rados.2.smithi175.stdout:2198: oid 49 version 679 is already newer than 677 2024-09-11T03:46:43.367 INFO:tasks.rados.rados.2.smithi175.stdout:2198: oid 49 version 679 is already newer than 678 2024-09-11T03:46:43.367 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 679 (ObjNum 743 snap 243 seq_num 743) dirty exists 2024-09-11T03:46:43.367 INFO:tasks.rados.rados.2.smithi175.stdout:2198: left oid 49 (ObjNum 743 snap 243 seq_num 743) 2024-09-11T03:46:43.369 INFO:tasks.rados.rados.2.smithi175.stdout:2201: expect (ObjNum 726 snap 238 seq_num 726) 2024-09-11T03:46:43.385 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:46:43.385 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:46:43.416 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:43.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:43 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/729537126' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:46:43.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:43 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/729537126' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:46:43.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:43 smithi121 ceph-mon[30729]: osdmap e585: 12 total, 12 up, 12 in 2024-09-11T03:46:43.438 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:43.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:43.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:43.443 INFO:tasks.rados.rados.2.smithi175.stdout:2202: finishing rollback tid 0 to smithi17557870-11 2024-09-11T03:46:43.443 INFO:tasks.rados.rados.2.smithi175.stdout:2202: finishing rollback tid 1 to smithi17557870-11 2024-09-11T03:46:43.443 INFO:tasks.rados.rados.2.smithi175.stdout:2202: finishing rollback tid 2 to smithi17557870-11 2024-09-11T03:46:43.443 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 597 (ObjNum 609 snap 191 seq_num 609) dirty exists 2024-09-11T03:46:43.540 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:43.562 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:43.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:43.563 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:43.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:43.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:43.690 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:43.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:43.748 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-11T03:46:43.792 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:43.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:43.814 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:43.814 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:43.825 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:43.915 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:43.936 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:43.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:43.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:43.975 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-11T03:46:44.038 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:44.059 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:44.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:44.060 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:44.068 INFO:tasks.rados.rados.2.smithi175.stdout:2198: done (5 left) 2024-09-11T03:46:44.069 INFO:tasks.rados.rados.2.smithi175.stdout:2199: done (4 left) 2024-09-11T03:46:44.069 INFO:tasks.rados.rados.2.smithi175.stdout:2200: done (3 left) 2024-09-11T03:46:44.069 INFO:tasks.rados.rados.2.smithi175.stdout:2201: done (2 left) 2024-09-11T03:46:44.069 INFO:tasks.rados.rados.2.smithi175.stdout:2202: done (1 left) 2024-09-11T03:46:44.069 INFO:tasks.rados.rados.2.smithi175.stdout:2203: done (0 left) 2024-09-11T03:46:44.069 INFO:tasks.rados.rados.2.smithi175.stdout:2204: delete oid 6 current snap is 244 2024-09-11T03:46:44.070 INFO:tasks.rados.rados.2.smithi175.stdout:2204: done (0 left) 2024-09-11T03:46:44.070 INFO:tasks.rados.rados.2.smithi175.stdout:2205: snap_remove snap 243 2024-09-11T03:46:44.092 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:46:44.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:44.181 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:44.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:44.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:44.223 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:46:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:44 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:46:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:44 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:46:44.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:44 smithi175 ceph-mon[30894]: pgmap v1032: 67 pgs: 1 active+clean+scrubbing, 66 active+clean; 130 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:46:44.283 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:44.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:44.306 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:44.306 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:44.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:44 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:46:44.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:44 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:46:44.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:44 smithi084 ceph-mon[27254]: pgmap v1032: 67 pgs: 1 active+clean+scrubbing, 66 active+clean; 130 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:46:44.393 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:44.406 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:44 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:46:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:44 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:46:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:44 smithi121 ceph-mon[30729]: pgmap v1032: 67 pgs: 1 active+clean+scrubbing, 66 active+clean; 130 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:46:44.428 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:44.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:44.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:44.432 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:44.530 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:44.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:44.554 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:44.554 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:44.655 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:44.676 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:44.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:44.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:44.778 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:44.792 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:46:44.792 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:46:44.800 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:44.801 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:44.902 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:44.923 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:44.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:44.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:45.025 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:45.046 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:45.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:45.047 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:45.066 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:46:45.070 INFO:tasks.rados.rados.2.smithi175.stdout:2205: done (0 left) 2024-09-11T03:46:45.070 INFO:tasks.rados.rados.2.smithi175.stdout:2206: delete oid 35 current snap is 244 2024-09-11T03:46:45.073 INFO:tasks.rados.rados.2.smithi175.stdout:2206: done (0 left) 2024-09-11T03:46:45.073 INFO:tasks.rados.rados.2.smithi175.stdout:2207: snap_remove snap 239 2024-09-11T03:46:45.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:45 smithi084 ceph-mon[27254]: osdmap e586: 12 total, 12 up, 12 in 2024-09-11T03:46:45.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:45 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4119697325' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:45.148 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:45.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:45.171 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:45.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:45.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:45.299 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:45.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:45.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:45.401 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:45.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:45 smithi121 ceph-mon[30729]: osdmap e586: 12 total, 12 up, 12 in 2024-09-11T03:46:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:45 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4119697325' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:45.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:45.424 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:45.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:45.473 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:45 smithi175 ceph-mon[30894]: osdmap e586: 12 total, 12 up, 12 in 2024-09-11T03:46:45.473 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:45 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4119697325' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:45.501 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-11T03:46:45.526 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:45.552 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:45.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:45.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:45.558 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:46:45.558 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:46:45.574 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:46:45.574 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:46:45.590 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:46:45.590 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:46:45.606 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:46:45.606 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:46:45.623 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:46:45.623 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:46:45.641 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:46:45.641 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:46:45.654 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:45.658 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:46:45.658 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:46:45.674 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:46:45.674 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:46:45.679 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:45.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:45.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:45.691 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:46:45.691 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:46:45.708 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:46:45.708 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:46:45.724 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:46:45.725 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:46:45.741 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:46:45.741 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:46:45.742 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:46:45.781 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:45.802 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:45.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:45.803 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:45.904 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:45.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:45.927 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:45.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:46.028 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:46.049 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:46.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:46.050 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:46.085 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T03:46:46.089 INFO:tasks.rados.rados.2.smithi175.stdout:2207: done (0 left) 2024-09-11T03:46:46.089 INFO:tasks.rados.rados.2.smithi175.stdout:2208: write (excl) oid 21 current snap is 244 2024-09-11T03:46:46.089 INFO:tasks.rados.rados.2.smithi175.stdout:2208: seq_num 746 ranges {538022=738348,1737567=561594} 2024-09-11T03:46:46.096 INFO:tasks.rados.rados.2.smithi175.stdout:2208: writing smithi17557870-21 from 538022 to 1276370 tid 1 2024-09-11T03:46:46.103 INFO:tasks.rados.rados.2.smithi175.stdout:2208: writing smithi17557870-21 from 1737567 to 2299161 tid 2 2024-09-11T03:46:46.103 INFO:tasks.rados.rados.2.smithi175.stdout:2209: delete oid 1 current snap is 244 2024-09-11T03:46:46.104 INFO:tasks.rados.rados.2.smithi175.stdout:2209: done (1 left) 2024-09-11T03:46:46.104 INFO:tasks.rados.rados.2.smithi175.stdout:2210: snap_remove snap 236 2024-09-11T03:46:46.141 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:46 smithi175 ceph-mon[30894]: pgmap v1035: 67 pgs: 1 active+clean+scrubbing, 66 active+clean; 129 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:46:46.142 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:46 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:46.142 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4237160614' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:46:46.142 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4119697325' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:46.142 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:46 smithi175 ceph-mon[30894]: osdmap e587: 12 total, 12 up, 12 in 2024-09-11T03:46:46.142 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2479380750' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:46.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:46.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:46.173 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:46.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:46.242 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:46:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:46:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:46:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:46:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:847 2.d 2:b1c0f730:::smithi17557870-20:head [copy-from ver 90] snapc 1a=[1a,19] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-11T03:46:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:46:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 441.40223849199998, 2024-09-11T03:46:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013054913, 2024-09-11T03:46:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 847 2024-09-11T03:46:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:46:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839295+0000", 2024-09-11T03:46:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839297+0000", 2024-09-11T03:46:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0049999999999999e-06 2024-09-11T03:46:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839309+0000", 2024-09-11T03:46:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1531e-05 2024-09-11T03:46:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839326+0000", 2024-09-11T03:46:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7788e-05 2024-09-11T03:46:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839336+0000", 2024-09-11T03:46:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5619999999999993e-06 2024-09-11T03:46:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839423+0000", 2024-09-11T03:46:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6790000000000001e-05 2024-09-11T03:46:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:46:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.839434+0000", 2024-09-11T03:46:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0971e-05 2024-09-11T03:46:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841520+0000", 2024-09-11T03:46:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0020866299999999999 2024-09-11T03:46:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.841590+0000", 2024-09-11T03:46:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9288000000000005e-05 2024-09-11T03:46:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:46:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.846430+0000", 2024-09-11T03:46:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0048408349999999999 2024-09-11T03:46:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.849726+0000", 2024-09-11T03:46:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032955390000000001 2024-09-11T03:46:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852240+0000", 2024-09-11T03:46:46.249 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002513492 2024-09-11T03:46:46.249 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.249 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.249 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.249 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852278+0000", 2024-09-11T03:46:46.249 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7988999999999998e-05 2024-09-11T03:46:46.249 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.249 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.249 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:46.249 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852317+0000", 2024-09-11T03:46:46.249 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9310000000000001e-05 2024-09-11T03:46:46.249 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.249 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.250 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.250 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:24.852350+0000", 2024-09-11T03:46:46.250 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3182999999999997e-05 2024-09-11T03:46:46.250 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.250 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.250 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.250 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.250 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.250 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1290 2.d 2:b1c0f730:::smithi17557870-20:head [copy-from ver 48] snapc 33=[32,30,2c,2b] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:46:46.250 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:12.836022+0000", 2024-09-11T03:46:46.250 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 393.40551121099998, 2024-09-11T03:46:46.250 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022892863999999999, 2024-09-11T03:46:46.250 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.250 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.251 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:46.251 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.251 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.251 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.251 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1290 2024-09-11T03:46:46.251 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.251 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.251 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.251 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.251 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836022+0000", 2024-09-11T03:46:46.251 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.251 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.251 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.251 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.251 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836022+0000", 2024-09-11T03:46:46.252 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.252 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.252 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.252 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.252 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836025+0000", 2024-09-11T03:46:46.252 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6690000000000002e-06 2024-09-11T03:46:46.252 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.252 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.252 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.252 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836033+0000", 2024-09-11T03:46:46.252 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0809999999999994e-06 2024-09-11T03:46:46.252 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.252 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.252 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.253 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836053+0000", 2024-09-11T03:46:46.253 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9429999999999999e-05 2024-09-11T03:46:46.253 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.253 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.253 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.253 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836063+0000", 2024-09-11T03:46:46.253 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0438000000000001e-05 2024-09-11T03:46:46.253 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.253 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.253 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.253 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836165+0000", 2024-09-11T03:46:46.253 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000101523 2024-09-11T03:46:46.253 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.253 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.253 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:46:46.254 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836172+0000", 2024-09-11T03:46:46.254 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6499999999999996e-06 2024-09-11T03:46:46.254 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.254 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.254 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.254 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839394+0000", 2024-09-11T03:46:46.254 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032215590000000001 2024-09-11T03:46:46.254 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.254 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.254 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.254 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839507+0000", 2024-09-11T03:46:46.254 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000113557 2024-09-11T03:46:46.254 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.254 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.255 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:46:46.255 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.849905+0000", 2024-09-11T03:46:46.255 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010398174 2024-09-11T03:46:46.255 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.255 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.255 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:46.255 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.853428+0000", 2024-09-11T03:46:46.255 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035222869999999998 2024-09-11T03:46:46.255 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.255 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.255 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.255 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.855923+0000", 2024-09-11T03:46:46.255 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024951840000000001 2024-09-11T03:46:46.256 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.256 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.256 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.256 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.858781+0000", 2024-09-11T03:46:46.256 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002858368 2024-09-11T03:46:46.256 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.256 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.256 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:46.256 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.858881+0000", 2024-09-11T03:46:46.256 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9500999999999995e-05 2024-09-11T03:46:46.256 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.256 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.256 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.257 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.858915+0000", 2024-09-11T03:46:46.257 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4443000000000002e-05 2024-09-11T03:46:46.257 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.257 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.257 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.257 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.257 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.257 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1291 2.d 2:b1c0f730:::smithi17557870-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:46:46.257 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:12.836223+0000", 2024-09-11T03:46:46.257 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 393.40531046500001, 2024-09-11T03:46:46.257 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016933050000000002, 2024-09-11T03:46:46.257 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.257 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:46:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1291 2024-09-11T03:46:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836223+0000", 2024-09-11T03:46:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836223+0000", 2024-09-11T03:46:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836224+0000", 2024-09-11T03:46:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0020000000000001e-06 2024-09-11T03:46:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836229+0000", 2024-09-11T03:46:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7180000000000004e-06 2024-09-11T03:46:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836245+0000", 2024-09-11T03:46:46.260 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5981000000000002e-05 2024-09-11T03:46:46.260 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.260 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.260 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.260 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836253+0000", 2024-09-11T03:46:46.260 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1720000000000007e-06 2024-09-11T03:46:46.260 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.260 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.260 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.260 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836317+0000", 2024-09-11T03:46:46.260 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4189999999999994e-05 2024-09-11T03:46:46.260 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.260 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.260 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:46:46.261 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.836324+0000", 2024-09-11T03:46:46.261 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9249999999999998e-06 2024-09-11T03:46:46.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.261 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.261 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839738+0000", 2024-09-11T03:46:46.261 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0034135580000000001 2024-09-11T03:46:46.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.261 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:46:46.261 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.839804+0000", 2024-09-11T03:46:46.261 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5974000000000002e-05 2024-09-11T03:46:46.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.262 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.262 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.852933+0000", 2024-09-11T03:46:46.262 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013129429999999999 2024-09-11T03:46:46.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.262 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.262 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.853032+0000", 2024-09-11T03:46:46.262 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8592999999999996e-05 2024-09-11T03:46:46.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.262 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.263 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:12.853156+0000", 2024-09-11T03:46:46.263 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012450699999999999 2024-09-11T03:46:46.263 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.263 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.263 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.263 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.263 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.263 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1516 2.d 2:b6a25a79:::smithi17557870-47:head [copy-from ver 97] snapc 3b=[3a,38,37,35] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-09-11T03:46:46.263 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:28.919589+0000", 2024-09-11T03:46:46.263 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 377.32194433500001, 2024-09-11T03:46:46.263 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011956165, 2024-09-11T03:46:46.263 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.264 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.264 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:46.264 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.264 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.264 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.264 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1516 2024-09-11T03:46:46.264 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.264 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.264 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.264 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.264 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919589+0000", 2024-09-11T03:46:46.264 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.264 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.264 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.265 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.265 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919589+0000", 2024-09-11T03:46:46.265 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.265 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.265 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.265 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.265 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919591+0000", 2024-09-11T03:46:46.265 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5409999999999999e-06 2024-09-11T03:46:46.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.266 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.266 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.266 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919596+0000", 2024-09-11T03:46:46.266 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9280000000000001e-06 2024-09-11T03:46:46.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.266 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.266 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.266 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919610+0000", 2024-09-11T03:46:46.266 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4285e-05 2024-09-11T03:46:46.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.266 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.267 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.267 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919617+0000", 2024-09-11T03:46:46.267 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.435e-06 2024-09-11T03:46:46.267 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.267 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.267 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.267 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919688+0000", 2024-09-11T03:46:46.267 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0263000000000001e-05 2024-09-11T03:46:46.267 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.267 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.267 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.267 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.919794+0000", 2024-09-11T03:46:46.267 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000106055 2024-09-11T03:46:46.268 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.268 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.268 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:46:46.268 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.922849+0000", 2024-09-11T03:46:46.268 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030557330000000001 2024-09-11T03:46:46.268 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.268 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.268 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:46.268 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.926556+0000", 2024-09-11T03:46:46.268 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0037066669999999999 2024-09-11T03:46:46.268 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.268 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.268 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.268 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.929197+0000", 2024-09-11T03:46:46.269 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002640529 2024-09-11T03:46:46.269 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.269 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.269 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.269 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.931432+0000", 2024-09-11T03:46:46.269 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022350339999999999 2024-09-11T03:46:46.269 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.269 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.269 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:46.269 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.931501+0000", 2024-09-11T03:46:46.269 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9584000000000003e-05 2024-09-11T03:46:46.269 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.269 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.269 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.270 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.931545+0000", 2024-09-11T03:46:46.270 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4110999999999998e-05 2024-09-11T03:46:46.270 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.270 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.270 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.270 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.270 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.270 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2254 2.18 2:1984a3a9:::smithi17557870-23:head [write 746310~635349,stat] snapc 5d=[5d,5a,59,58,56] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:46:46.270 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.883765+0000", 2024-09-11T03:46:46.270 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 309.357768222, 2024-09-11T03:46:46.270 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022154396, 2024-09-11T03:46:46.270 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.271 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.271 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:46.271 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.271 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.271 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.271 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2254 2024-09-11T03:46:46.271 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.271 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.271 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.272 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.273 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.883765+0000", 2024-09-11T03:46:46.273 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.273 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.273 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.273 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.273 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.883765+0000", 2024-09-11T03:46:46.273 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.273 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.273 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.273 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.273 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.883768+0000", 2024-09-11T03:46:46.273 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4080000000000001e-06 2024-09-11T03:46:46.273 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.274 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.274 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.274 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884378+0000", 2024-09-11T03:46:46.274 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00061034200000000005 2024-09-11T03:46:46.274 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.274 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.274 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.274 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884399+0000", 2024-09-11T03:46:46.274 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1175e-05 2024-09-11T03:46:46.274 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.274 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.274 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.275 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884412+0000", 2024-09-11T03:46:46.275 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3015e-05 2024-09-11T03:46:46.275 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.275 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.275 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.275 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884520+0000", 2024-09-11T03:46:46.275 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000107706 2024-09-11T03:46:46.275 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.275 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.275 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:46:46.275 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884534+0000", 2024-09-11T03:46:46.275 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3923e-05 2024-09-11T03:46:46.276 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.276 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.276 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.276 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.895937+0000", 2024-09-11T03:46:46.276 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011403207 2024-09-11T03:46:46.276 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.276 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.276 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:46:46.276 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896424+0000", 2024-09-11T03:46:46.276 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00048641199999999998 2024-09-11T03:46:46.276 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.276 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.276 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.276 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.898138+0000", 2024-09-11T03:46:46.277 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001714272 2024-09-11T03:46:46.277 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.277 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.277 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.277 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.898309+0000", 2024-09-11T03:46:46.277 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017075200000000001 2024-09-11T03:46:46.277 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.277 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.277 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:46:46.277 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.898561+0000", 2024-09-11T03:46:46.277 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00025223500000000001 2024-09-11T03:46:46.277 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.277 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.278 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.278 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903048+0000", 2024-09-11T03:46:46.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0044876320000000001 2024-09-11T03:46:46.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.278 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.278 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.278 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903118+0000", 2024-09-11T03:46:46.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9608000000000002e-05 2024-09-11T03:46:46.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.278 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.278 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:46.278 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905694+0000", 2024-09-11T03:46:46.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025761650000000001 2024-09-11T03:46:46.279 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.279 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.279 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:46.279 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905749+0000", 2024-09-11T03:46:46.279 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5037999999999997e-05 2024-09-11T03:46:46.279 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.279 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.279 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.279 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905920+0000", 2024-09-11T03:46:46.279 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017050600000000001 2024-09-11T03:46:46.279 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.279 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.279 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.280 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.280 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.280 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2255 2.18 2:1984a3a9:::smithi17557870-23:head [write 2077646~576443] snapc 5d=[5d,5a,59,58,56] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:46:46.280 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.884463+0000", 2024-09-11T03:46:46.280 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 309.357070491, 2024-09-11T03:46:46.280 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022426167, 2024-09-11T03:46:46.280 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.280 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.280 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:46.280 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.280 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.280 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.280 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2255 2024-09-11T03:46:46.281 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.281 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.281 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.281 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.281 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884463+0000", 2024-09-11T03:46:46.281 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.281 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.281 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.281 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.281 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884463+0000", 2024-09-11T03:46:46.281 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.281 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.281 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.282 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.282 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884464+0000", 2024-09-11T03:46:46.282 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.64e-07 2024-09-11T03:46:46.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.282 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.282 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885204+0000", 2024-09-11T03:46:46.282 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00074007200000000004 2024-09-11T03:46:46.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.282 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.282 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885217+0000", 2024-09-11T03:46:46.282 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2547e-05 2024-09-11T03:46:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885226+0000", 2024-09-11T03:46:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1849999999999993e-06 2024-09-11T03:46:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885295+0000", 2024-09-11T03:46:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8657000000000004e-05 2024-09-11T03:46:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:46:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885307+0000", 2024-09-11T03:46:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2529e-05 2024-09-11T03:46:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896570+0000", 2024-09-11T03:46:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011262529 2024-09-11T03:46:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:46:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896624+0000", 2024-09-11T03:46:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4614000000000003e-05 2024-09-11T03:46:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.900547+0000", 2024-09-11T03:46:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039228850000000001 2024-09-11T03:46:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.900706+0000", 2024-09-11T03:46:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015907000000000001 2024-09-11T03:46:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:46:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.900901+0000", 2024-09-11T03:46:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00019514399999999999 2024-09-11T03:46:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903192+0000", 2024-09-11T03:46:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022906519999999998 2024-09-11T03:46:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.904724+0000", 2024-09-11T03:46:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015319520000000001 2024-09-11T03:46:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906254+0000", 2024-09-11T03:46:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015302759999999999 2024-09-11T03:46:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906358+0000", 2024-09-11T03:46:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010404300000000001 2024-09-11T03:46:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906889+0000", 2024-09-11T03:46:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000531048 2024-09-11T03:46:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2256 2.18 2:1984a3a9:::smithi17557870-23:head [write 3362764~183546] snapc 5d=[5d,5a,59,58,56] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:46:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.885408+0000", 2024-09-11T03:46:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 309.35612515600002, 2024-09-11T03:46:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021926139000000001, 2024-09-11T03:46:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2256 2024-09-11T03:46:46.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.289 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.289 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.289 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885408+0000", 2024-09-11T03:46:46.289 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.290 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.290 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885408+0000", 2024-09-11T03:46:46.290 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.290 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.290 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.290 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.290 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885410+0000", 2024-09-11T03:46:46.290 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4270000000000001e-06 2024-09-11T03:46:46.290 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.290 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.290 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.291 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885561+0000", 2024-09-11T03:46:46.291 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000151254 2024-09-11T03:46:46.291 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.291 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.291 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.291 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885573+0000", 2024-09-11T03:46:46.291 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1897e-05 2024-09-11T03:46:46.291 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.291 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.291 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.291 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885581+0000", 2024-09-11T03:46:46.291 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6290000000000001e-06 2024-09-11T03:46:46.291 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.291 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.292 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.292 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885643+0000", 2024-09-11T03:46:46.292 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2215e-05 2024-09-11T03:46:46.292 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.292 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.292 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:46:46.292 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885652+0000", 2024-09-11T03:46:46.292 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5580000000000006e-06 2024-09-11T03:46:46.292 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.292 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.292 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.292 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896678+0000", 2024-09-11T03:46:46.293 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011025956 2024-09-11T03:46:46.293 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.293 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.293 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:46:46.293 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896729+0000", 2024-09-11T03:46:46.293 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1019000000000001e-05 2024-09-11T03:46:46.293 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.293 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.293 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.293 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903250+0000", 2024-09-11T03:46:46.293 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0065205109999999997 2024-09-11T03:46:46.293 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.293 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.294 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.294 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903415+0000", 2024-09-11T03:46:46.294 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016545399999999999 2024-09-11T03:46:46.294 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.294 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.294 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:46:46.294 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903701+0000", 2024-09-11T03:46:46.294 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00028586399999999999 2024-09-11T03:46:46.294 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.294 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.294 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.294 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905681+0000", 2024-09-11T03:46:46.294 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0019798950000000002 2024-09-11T03:46:46.294 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.295 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.295 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.295 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905999+0000", 2024-09-11T03:46:46.295 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00031789999999999998 2024-09-11T03:46:46.295 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.295 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.295 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:46.295 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906904+0000", 2024-09-11T03:46:46.295 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00090535599999999996 2024-09-11T03:46:46.295 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.295 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.295 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:46.295 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907002+0000", 2024-09-11T03:46:46.296 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7653999999999997e-05 2024-09-11T03:46:46.296 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.296 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.296 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.296 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907335+0000", 2024-09-11T03:46:46.296 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00033254999999999998 2024-09-11T03:46:46.296 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.296 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.296 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.296 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.296 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.296 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2257 2.18 2:1984a3a9:::smithi17557870-23:head [setxattr _header (57) in=7b,truncate 3546310] snapc 5d=[5d,5a,59,58,56] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:46:46.297 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.885667+0000", 2024-09-11T03:46:46.297 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 309.35586613499999, 2024-09-11T03:46:46.297 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022144878999999999, 2024-09-11T03:46:46.297 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.297 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.297 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:46.297 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.297 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.297 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.297 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2257 2024-09-11T03:46:46.297 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.297 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.298 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.298 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.298 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885667+0000", 2024-09-11T03:46:46.298 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.298 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.298 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.298 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.298 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885667+0000", 2024-09-11T03:46:46.298 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.298 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.298 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.298 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.299 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885668+0000", 2024-09-11T03:46:46.299 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5600000000000005e-07 2024-09-11T03:46:46.299 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.299 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.299 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.299 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885673+0000", 2024-09-11T03:46:46.299 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4970000000000003e-06 2024-09-11T03:46:46.299 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.299 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.299 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.299 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885680+0000", 2024-09-11T03:46:46.299 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3780000000000003e-06 2024-09-11T03:46:46.299 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.300 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.300 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.300 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885686+0000", 2024-09-11T03:46:46.300 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0789999999999997e-06 2024-09-11T03:46:46.300 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.300 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.300 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.300 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886080+0000", 2024-09-11T03:46:46.300 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00039392399999999998 2024-09-11T03:46:46.300 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.300 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.300 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:46:46.301 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886091+0000", 2024-09-11T03:46:46.301 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0472000000000001e-05 2024-09-11T03:46:46.301 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.301 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.301 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.301 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896776+0000", 2024-09-11T03:46:46.301 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010684953000000001 2024-09-11T03:46:46.301 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.301 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.301 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:46:46.301 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896839+0000", 2024-09-11T03:46:46.301 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3293999999999994e-05 2024-09-11T03:46:46.302 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.302 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.302 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.302 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.904784+0000", 2024-09-11T03:46:46.302 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.007944843 2024-09-11T03:46:46.302 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.302 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.302 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.302 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.904909+0000", 2024-09-11T03:46:46.302 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012491600000000001 2024-09-11T03:46:46.302 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.302 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.303 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:46:46.303 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905105+0000", 2024-09-11T03:46:46.303 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000196336 2024-09-11T03:46:46.303 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.303 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.303 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.303 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906057+0000", 2024-09-11T03:46:46.303 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00095239800000000002 2024-09-11T03:46:46.303 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.303 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.303 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:46.303 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907348+0000", 2024-09-11T03:46:46.304 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012904489999999999 2024-09-11T03:46:46.304 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.304 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.304 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.304 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907457+0000", 2024-09-11T03:46:46.304 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108869 2024-09-11T03:46:46.304 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.304 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.304 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:46.304 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907543+0000", 2024-09-11T03:46:46.304 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6212999999999996e-05 2024-09-11T03:46:46.304 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.304 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.305 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.305 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907812+0000", 2024-09-11T03:46:46.305 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00026950200000000002 2024-09-11T03:46:46.305 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.305 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.305 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.305 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.305 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.305 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2258 2.18 2:1984a3a9:::smithi17557870-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:46:46.305 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.886095+0000", 2024-09-11T03:46:46.305 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 309.35543837, 2024-09-11T03:46:46.305 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022147336, 2024-09-11T03:46:46.306 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.306 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.306 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:46:46.306 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.306 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.306 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.306 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2258 2024-09-11T03:46:46.306 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.306 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.306 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.307 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.307 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886095+0000", 2024-09-11T03:46:46.307 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.307 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.307 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.307 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.307 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886095+0000", 2024-09-11T03:46:46.307 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.307 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.307 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.307 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.307 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886096+0000", 2024-09-11T03:46:46.307 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0200000000000001e-07 2024-09-11T03:46:46.308 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.308 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.308 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.308 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886100+0000", 2024-09-11T03:46:46.308 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9859999999999996e-06 2024-09-11T03:46:46.308 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.309 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.309 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.309 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886109+0000", 2024-09-11T03:46:46.309 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6349999999999995e-06 2024-09-11T03:46:46.309 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.309 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.309 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.309 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886116+0000", 2024-09-11T03:46:46.309 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0890000000000002e-06 2024-09-11T03:46:46.309 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.309 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.309 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.309 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886389+0000", 2024-09-11T03:46:46.310 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00027359200000000002 2024-09-11T03:46:46.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.310 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:46:46.310 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886399+0000", 2024-09-11T03:46:46.310 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9599999999999995e-06 2024-09-11T03:46:46.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.310 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.310 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896922+0000", 2024-09-11T03:46:46.310 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010522567 2024-09-11T03:46:46.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.311 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:46:46.311 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896977+0000", 2024-09-11T03:46:46.311 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5003999999999997e-05 2024-09-11T03:46:46.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.311 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.311 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905540+0000", 2024-09-11T03:46:46.311 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0085636329999999993 2024-09-11T03:46:46.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.311 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:46:46.312 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905625+0000", 2024-09-11T03:46:46.312 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4468999999999994e-05 2024-09-11T03:46:46.312 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.312 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.312 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.312 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907869+0000", 2024-09-11T03:46:46.312 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022438290000000001 2024-09-11T03:46:46.312 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.312 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.312 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.312 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.908014+0000", 2024-09-11T03:46:46.312 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00014511799999999999 2024-09-11T03:46:46.313 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.313 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.313 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.313 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.908243+0000", 2024-09-11T03:46:46.313 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00022865200000000001 2024-09-11T03:46:46.313 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.313 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.313 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.313 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.313 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.313 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2421 2.18 2:1bdf1a36:::smithi17557870-36:head [copy-from ver 157] snapc 65=[65,64,63,61,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-11T03:46:46.313 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:50.954849+0000", 2024-09-11T03:46:46.313 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 295.28668434000002, 2024-09-11T03:46:46.313 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022711387999999999, 2024-09-11T03:46:46.314 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.314 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.314 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:46.314 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.314 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.314 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.314 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2421 2024-09-11T03:46:46.314 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.314 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.314 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.314 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.314 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954849+0000", 2024-09-11T03:46:46.314 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.314 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.315 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.315 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.315 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954849+0000", 2024-09-11T03:46:46.315 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.315 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.315 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.315 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.315 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954851+0000", 2024-09-11T03:46:46.315 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3090000000000001e-06 2024-09-11T03:46:46.315 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.315 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.315 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.315 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954856+0000", 2024-09-11T03:46:46.315 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9059999999999999e-06 2024-09-11T03:46:46.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.316 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.316 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954867+0000", 2024-09-11T03:46:46.316 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0879999999999999e-05 2024-09-11T03:46:46.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.316 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.316 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954873+0000", 2024-09-11T03:46:46.316 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.203e-06 2024-09-11T03:46:46.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.316 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.317 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954921+0000", 2024-09-11T03:46:46.317 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8180000000000003e-05 2024-09-11T03:46:46.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.317 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.317 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.317 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.955004+0000", 2024-09-11T03:46:46.317 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2824000000000005e-05 2024-09-11T03:46:46.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.317 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.317 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:46:46.317 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964099+0000", 2024-09-11T03:46:46.317 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0090952499999999992 2024-09-11T03:46:46.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.317 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.318 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.318 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.970182+0000", 2024-09-11T03:46:46.318 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0060831510000000002 2024-09-11T03:46:46.318 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.318 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.318 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:46.318 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977039+0000", 2024-09-11T03:46:46.318 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068571980000000001 2024-09-11T03:46:46.318 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.318 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.318 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.318 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977432+0000", 2024-09-11T03:46:46.318 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00039304900000000002 2024-09-11T03:46:46.318 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.319 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.319 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:46.319 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977504+0000", 2024-09-11T03:46:46.319 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1863e-05 2024-09-11T03:46:46.319 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.319 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.319 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.319 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977561+0000", 2024-09-11T03:46:46.319 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6575e-05 2024-09-11T03:46:46.319 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.319 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.319 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.319 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.319 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.320 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2422 2.18 2:1bdf1a36:::smithi17557870-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e294)", 2024-09-11T03:46:46.320 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:50.954885+0000", 2024-09-11T03:46:46.320 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 295.28664830299999, 2024-09-11T03:46:46.320 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012999881, 2024-09-11T03:46:46.320 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.320 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.320 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:46:46.320 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.320 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.320 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.320 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2422 2024-09-11T03:46:46.320 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.320 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.320 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.321 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.321 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954885+0000", 2024-09-11T03:46:46.321 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.321 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.321 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.321 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.321 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954885+0000", 2024-09-11T03:46:46.321 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.321 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.321 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.321 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.321 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954886+0000", 2024-09-11T03:46:46.322 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.82e-07 2024-09-11T03:46:46.322 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.322 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.322 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.322 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954888+0000", 2024-09-11T03:46:46.322 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1909999999999999e-06 2024-09-11T03:46:46.322 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.322 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.322 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.322 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954899+0000", 2024-09-11T03:46:46.322 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0963e-05 2024-09-11T03:46:46.322 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.322 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.323 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.323 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954904+0000", 2024-09-11T03:46:46.323 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3539999999999999e-06 2024-09-11T03:46:46.323 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.323 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.323 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.323 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.955109+0000", 2024-09-11T03:46:46.323 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000204494 2024-09-11T03:46:46.323 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.323 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.323 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:46:46.323 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.955158+0000", 2024-09-11T03:46:46.323 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9502000000000001e-05 2024-09-11T03:46:46.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.324 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.324 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.324 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.967733+0000", 2024-09-11T03:46:46.324 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012574571999999999 2024-09-11T03:46:46.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.324 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.324 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.324 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.967803+0000", 2024-09-11T03:46:46.324 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0160999999999995e-05 2024-09-11T03:46:46.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.325 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.325 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.325 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.967885+0000", 2024-09-11T03:46:46.325 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2162000000000004e-05 2024-09-11T03:46:46.325 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.325 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.325 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.325 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.325 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.328 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2511 2.d 2:b6a25a79:::smithi17557870-47:head [copy-from ver 327] snapc 6c=[6c,6b,6a,69,68,65,63,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-09-11T03:46:46.328 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:01.867547+0000", 2024-09-11T03:46:46.328 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 284.37398635300002, 2024-09-11T03:46:46.328 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017562828999999999, 2024-09-11T03:46:46.328 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.328 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.328 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:46.328 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.328 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.328 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.328 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2511 2024-09-11T03:46:46.328 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.329 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.329 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.329 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.329 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867547+0000", 2024-09-11T03:46:46.329 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.329 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.329 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.329 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867547+0000", 2024-09-11T03:46:46.329 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.329 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.329 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.329 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867549+0000", 2024-09-11T03:46:46.330 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4100000000000001e-06 2024-09-11T03:46:46.330 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.330 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.330 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.330 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867554+0000", 2024-09-11T03:46:46.330 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.463e-06 2024-09-11T03:46:46.330 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.330 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.330 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.330 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867565+0000", 2024-09-11T03:46:46.330 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0531e-05 2024-09-11T03:46:46.330 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.330 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.330 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.330 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867574+0000", 2024-09-11T03:46:46.331 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7850000000000003e-06 2024-09-11T03:46:46.331 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.331 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.331 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.331 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867625+0000", 2024-09-11T03:46:46.331 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0958999999999998e-05 2024-09-11T03:46:46.331 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.331 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.331 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:46:46.331 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867632+0000", 2024-09-11T03:46:46.331 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4150000000000004e-06 2024-09-11T03:46:46.331 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.331 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.331 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.331 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.870960+0000", 2024-09-11T03:46:46.332 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0033279199999999998 2024-09-11T03:46:46.332 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.332 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.332 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.332 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.871059+0000", 2024-09-11T03:46:46.332 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9423e-05 2024-09-11T03:46:46.332 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.332 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.332 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:46:46.332 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.878007+0000", 2024-09-11T03:46:46.332 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.006947796 2024-09-11T03:46:46.332 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.332 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.332 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:46.333 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.881858+0000", 2024-09-11T03:46:46.333 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003850647 2024-09-11T03:46:46.333 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.333 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.333 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.333 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.884846+0000", 2024-09-11T03:46:46.333 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0029880309999999999 2024-09-11T03:46:46.333 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.333 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.333 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.333 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.885043+0000", 2024-09-11T03:46:46.333 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00019722899999999999 2024-09-11T03:46:46.333 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.333 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.334 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:46.334 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.885080+0000", 2024-09-11T03:46:46.334 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7106e-05 2024-09-11T03:46:46.334 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.334 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.334 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.334 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.885110+0000", 2024-09-11T03:46:46.334 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0114e-05 2024-09-11T03:46:46.334 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.334 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.334 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.334 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.334 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.334 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2512 2.d 2:b6a25a79:::smithi17557870-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e305)", 2024-09-11T03:46:46.335 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:01.867594+0000", 2024-09-11T03:46:46.335 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 284.37394008400003, 2024-09-11T03:46:46.335 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013722218, 2024-09-11T03:46:46.335 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.335 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.335 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:46:46.335 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.335 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.335 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.335 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2512 2024-09-11T03:46:46.335 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.335 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.335 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.335 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.336 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867594+0000", 2024-09-11T03:46:46.336 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.336 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.336 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.336 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867594+0000", 2024-09-11T03:46:46.336 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.336 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.336 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.336 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867594+0000", 2024-09-11T03:46:46.336 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9100000000000004e-07 2024-09-11T03:46:46.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.336 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.337 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.337 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867597+0000", 2024-09-11T03:46:46.337 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6369999999999999e-06 2024-09-11T03:46:46.337 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.337 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.337 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.337 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867604+0000", 2024-09-11T03:46:46.337 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5610000000000001e-06 2024-09-11T03:46:46.337 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.337 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.337 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.337 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867610+0000", 2024-09-11T03:46:46.337 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1249999999999998e-06 2024-09-11T03:46:46.337 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.338 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.338 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.338 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867674+0000", 2024-09-11T03:46:46.338 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3969999999999999e-05 2024-09-11T03:46:46.338 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.338 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.338 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:46:46.338 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.867679+0000", 2024-09-11T03:46:46.338 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5929999999999999e-06 2024-09-11T03:46:46.338 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.338 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.338 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.338 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.871237+0000", 2024-09-11T03:46:46.339 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035579689999999998 2024-09-11T03:46:46.339 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.339 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.339 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:46:46.339 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.871280+0000", 2024-09-11T03:46:46.339 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2917000000000003e-05 2024-09-11T03:46:46.339 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.339 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.339 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.339 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.881114+0000", 2024-09-11T03:46:46.339 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0098343660000000006 2024-09-11T03:46:46.339 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.339 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.339 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.340 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.881210+0000", 2024-09-11T03:46:46.340 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5730000000000007e-05 2024-09-11T03:46:46.340 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.340 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.340 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.340 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:01.881316+0000", 2024-09-11T03:46:46.340 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000105759 2024-09-11T03:46:46.340 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.340 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.340 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.340 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.340 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.341 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2568 2.d 2:b6a25a79:::smithi17557870-47:head [copy-from ver 157] snapc 70=[70,6f,6e,6c,68,65,63,5d] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-11T03:46:46.341 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:10.876872+0000", 2024-09-11T03:46:46.341 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 275.36466121500001, 2024-09-11T03:46:46.341 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018302771999999998, 2024-09-11T03:46:46.341 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.341 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.341 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:46.341 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.341 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.341 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.341 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2568 2024-09-11T03:46:46.341 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.341 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.342 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.342 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.342 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876872+0000", 2024-09-11T03:46:46.342 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.342 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.342 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.342 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.342 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876872+0000", 2024-09-11T03:46:46.342 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.342 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.342 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.342 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.343 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876874+0000", 2024-09-11T03:46:46.343 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5570000000000001e-06 2024-09-11T03:46:46.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.343 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.343 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.343 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876880+0000", 2024-09-11T03:46:46.343 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8799999999999996e-06 2024-09-11T03:46:46.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.343 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.343 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.343 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876901+0000", 2024-09-11T03:46:46.343 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0942999999999999e-05 2024-09-11T03:46:46.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.343 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.344 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.344 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876909+0000", 2024-09-11T03:46:46.344 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3410000000000003e-06 2024-09-11T03:46:46.344 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.344 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.344 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.344 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876966+0000", 2024-09-11T03:46:46.344 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7179999999999998e-05 2024-09-11T03:46:46.344 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.344 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.344 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.344 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.877099+0000", 2024-09-11T03:46:46.344 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013234499999999999 2024-09-11T03:46:46.345 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.345 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.345 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:46:46.345 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.885734+0000", 2024-09-11T03:46:46.345 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0086350730000000001 2024-09-11T03:46:46.345 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.345 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.345 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:46.345 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.890016+0000", 2024-09-11T03:46:46.345 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.004282213 2024-09-11T03:46:46.345 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.345 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.345 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.346 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.894661+0000", 2024-09-11T03:46:46.346 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0046455059999999998 2024-09-11T03:46:46.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.346 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.346 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.346 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.895046+0000", 2024-09-11T03:46:46.346 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00038434999999999999 2024-09-11T03:46:46.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.346 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.346 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:46.346 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.895116+0000", 2024-09-11T03:46:46.346 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0536000000000004e-05 2024-09-11T03:46:46.347 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.347 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.347 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.347 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.895175+0000", 2024-09-11T03:46:46.347 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8848000000000003e-05 2024-09-11T03:46:46.347 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.347 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.347 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.347 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.347 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.347 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2569 2.d 2:b6a25a79:::smithi17557870-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e314)", 2024-09-11T03:46:46.347 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:10.876927+0000", 2024-09-11T03:46:46.347 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 275.36460641600002, 2024-09-11T03:46:46.348 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012434861, 2024-09-11T03:46:46.348 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.348 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.348 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:46:46.348 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.348 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.348 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.348 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2569 2024-09-11T03:46:46.348 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.348 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.348 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.348 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.348 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876927+0000", 2024-09-11T03:46:46.349 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.349 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.349 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.349 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.349 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876927+0000", 2024-09-11T03:46:46.349 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.349 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.349 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.349 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.349 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876928+0000", 2024-09-11T03:46:46.349 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9200000000000001e-07 2024-09-11T03:46:46.349 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.349 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.350 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.350 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876931+0000", 2024-09-11T03:46:46.350 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.881e-06 2024-09-11T03:46:46.350 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.350 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.350 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.350 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876938+0000", 2024-09-11T03:46:46.350 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7090000000000002e-06 2024-09-11T03:46:46.350 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.350 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.350 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.350 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.876944+0000", 2024-09-11T03:46:46.350 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7599999999999999e-06 2024-09-11T03:46:46.351 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.351 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.351 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.351 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.877267+0000", 2024-09-11T03:46:46.351 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00032272300000000002 2024-09-11T03:46:46.351 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.351 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.351 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:46:46.351 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.877312+0000", 2024-09-11T03:46:46.351 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4981999999999997e-05 2024-09-11T03:46:46.351 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.351 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.351 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.351 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.889216+0000", 2024-09-11T03:46:46.352 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011904246 2024-09-11T03:46:46.352 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.352 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.352 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.352 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.889286+0000", 2024-09-11T03:46:46.352 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9628000000000005e-05 2024-09-11T03:46:46.352 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.352 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.352 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.352 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:10.889362+0000", 2024-09-11T03:46:46.352 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6340000000000004e-05 2024-09-11T03:46:46.352 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.352 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.353 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.353 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.353 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.353 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4687 2.18 2:1984a3a9:::smithi17557870-23:head [write 739673~657763,stat] snapc cd=[cd,cc,ca,c5,c2,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-11T03:46:46.353 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.003513+0000", 2024-09-11T03:46:46.353 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 85.238021032000006, 2024-09-11T03:46:46.353 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.028701775999999998, 2024-09-11T03:46:46.353 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.353 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.353 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:46.353 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.353 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.353 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.354 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4687 2024-09-11T03:46:46.354 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.354 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.354 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.354 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.354 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003513+0000", 2024-09-11T03:46:46.354 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.354 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.354 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.354 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.354 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003513+0000", 2024-09-11T03:46:46.354 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.354 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.354 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.355 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.355 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003515+0000", 2024-09-11T03:46:46.355 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5600000000000001e-06 2024-09-11T03:46:46.355 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.355 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.355 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.355 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003862+0000", 2024-09-11T03:46:46.355 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00034641300000000001 2024-09-11T03:46:46.355 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.355 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.355 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.355 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003880+0000", 2024-09-11T03:46:46.355 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8669e-05 2024-09-11T03:46:46.356 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.356 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.356 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.356 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003906+0000", 2024-09-11T03:46:46.356 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5432999999999999e-05 2024-09-11T03:46:46.356 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.356 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.356 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.356 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003972+0000", 2024-09-11T03:46:46.356 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6729000000000004e-05 2024-09-11T03:46:46.356 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.356 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.357 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:46:46.357 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003982+0000", 2024-09-11T03:46:46.357 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2979999999999997e-06 2024-09-11T03:46:46.357 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.357 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.357 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.357 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.005521+0000", 2024-09-11T03:46:46.357 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015392959999999999 2024-09-11T03:46:46.357 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.357 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.357 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.357 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.005611+0000", 2024-09-11T03:46:46.358 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.0373999999999999e-05 2024-09-11T03:46:46.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.358 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,7", 2024-09-11T03:46:46.358 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.005729+0000", 2024-09-11T03:46:46.358 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000117834 2024-09-11T03:46:46.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.358 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:46.358 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.007175+0000", 2024-09-11T03:46:46.358 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014461140000000001 2024-09-11T03:46:46.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.358 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.358 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.008417+0000", 2024-09-11T03:46:46.359 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012421120000000001 2024-09-11T03:46:46.359 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.359 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.359 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.359 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032091+0000", 2024-09-11T03:46:46.359 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.023674066000000001 2024-09-11T03:46:46.359 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.359 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.359 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:46.359 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032146+0000", 2024-09-11T03:46:46.359 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4786e-05 2024-09-11T03:46:46.359 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.359 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.359 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.360 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032214+0000", 2024-09-11T03:46:46.360 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8091999999999998e-05 2024-09-11T03:46:46.360 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.360 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.360 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.360 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.360 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.360 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4688 2.18 2:1984a3a9:::smithi17557870-23:head [write 2076229~663444] snapc cd=[cd,cc,ca,c5,c2,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-11T03:46:46.360 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.010935+0000", 2024-09-11T03:46:46.360 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 85.230598971999996, 2024-09-11T03:46:46.360 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021404982999999999, 2024-09-11T03:46:46.360 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.361 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.361 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:46.361 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.361 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.361 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.361 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4688 2024-09-11T03:46:46.361 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.361 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.361 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.361 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.361 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.010935+0000", 2024-09-11T03:46:46.361 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.361 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.362 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.362 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.362 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.010935+0000", 2024-09-11T03:46:46.362 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.362 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.362 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.362 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.362 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.010936+0000", 2024-09-11T03:46:46.362 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1689999999999999e-06 2024-09-11T03:46:46.362 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.362 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.362 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.362 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011065+0000", 2024-09-11T03:46:46.362 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000128997 2024-09-11T03:46:46.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.363 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.363 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.363 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011076+0000", 2024-09-11T03:46:46.363 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1561e-05 2024-09-11T03:46:46.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.363 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.363 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.363 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011082+0000", 2024-09-11T03:46:46.363 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2599999999999996e-06 2024-09-11T03:46:46.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.363 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.363 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.364 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:46.365 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011127+0000", 2024-09-11T03:46:46.365 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5509000000000001e-05 2024-09-11T03:46:46.365 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.365 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.365 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.365 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011212+0000", 2024-09-11T03:46:46.365 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5173999999999995e-05 2024-09-11T03:46:46.366 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.366 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.366 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,7", 2024-09-11T03:46:46.366 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011309+0000", 2024-09-11T03:46:46.366 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6980999999999999e-05 2024-09-11T03:46:46.366 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.366 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.366 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:46.366 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012804+0000", 2024-09-11T03:46:46.366 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001495019 2024-09-11T03:46:46.366 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.366 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.366 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.366 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.031586+0000", 2024-09-11T03:46:46.367 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018782099 2024-09-11T03:46:46.367 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.367 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.367 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.367 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032261+0000", 2024-09-11T03:46:46.367 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000674846 2024-09-11T03:46:46.367 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.367 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.367 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:46.367 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032297+0000", 2024-09-11T03:46:46.367 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6010000000000003e-05 2024-09-11T03:46:46.367 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.367 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.367 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.368 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032340+0000", 2024-09-11T03:46:46.368 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2357999999999997e-05 2024-09-11T03:46:46.368 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.368 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.368 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.368 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.368 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.368 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4689 2.18 2:1984a3a9:::smithi17557870-23:head [setxattr _header (58) in=7b,truncate 2739673] snapc cd=[cd,cc,ca,c5,c2,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-11T03:46:46.368 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.011093+0000", 2024-09-11T03:46:46.368 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 85.230440496, 2024-09-11T03:46:46.368 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.02140688, 2024-09-11T03:46:46.368 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.368 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.368 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:46.368 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.369 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.369 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.369 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4689 2024-09-11T03:46:46.369 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.369 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.369 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.369 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.369 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011093+0000", 2024-09-11T03:46:46.369 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.369 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.369 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.369 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.369 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011093+0000", 2024-09-11T03:46:46.369 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.370 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.370 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.370 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.370 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011094+0000", 2024-09-11T03:46:46.370 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9999999999999997e-07 2024-09-11T03:46:46.370 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.370 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.370 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.370 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011100+0000", 2024-09-11T03:46:46.370 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1369999999999999e-06 2024-09-11T03:46:46.370 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.370 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.370 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.370 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011106+0000", 2024-09-11T03:46:46.371 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.421e-06 2024-09-11T03:46:46.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.371 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.371 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.371 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011117+0000", 2024-09-11T03:46:46.371 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1012999999999999e-05 2024-09-11T03:46:46.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.371 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.371 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.371 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012151+0000", 2024-09-11T03:46:46.371 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0010333880000000001 2024-09-11T03:46:46.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.371 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.372 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.372 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012265+0000", 2024-09-11T03:46:46.372 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114141 2024-09-11T03:46:46.372 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.372 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.372 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,7", 2024-09-11T03:46:46.372 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012418+0000", 2024-09-11T03:46:46.372 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015286499999999999 2024-09-11T03:46:46.372 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.372 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.372 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:46.372 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012807+0000", 2024-09-11T03:46:46.373 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00038979800000000001 2024-09-11T03:46:46.373 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.373 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.373 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.373 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.031717+0000", 2024-09-11T03:46:46.373 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018909888 2024-09-11T03:46:46.373 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.373 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.373 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:46.373 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032383+0000", 2024-09-11T03:46:46.373 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00066515900000000002 2024-09-11T03:46:46.373 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.373 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.374 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:46.374 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032447+0000", 2024-09-11T03:46:46.374 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4958999999999999e-05 2024-09-11T03:46:46.374 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.374 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.374 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.374 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032500+0000", 2024-09-11T03:46:46.374 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2510999999999999e-05 2024-09-11T03:46:46.374 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.374 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.374 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.374 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.374 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.375 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4690 2.18 2:1984a3a9:::smithi17557870-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e503)", 2024-09-11T03:46:46.375 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.011126+0000", 2024-09-11T03:46:46.375 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 85.230407267999993, 2024-09-11T03:46:46.375 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021513529, 2024-09-11T03:46:46.375 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.375 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.375 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:46:46.375 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:46.375 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:46.375 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:46.375 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4690 2024-09-11T03:46:46.375 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.375 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.376 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.376 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.376 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011126+0000", 2024-09-11T03:46:46.376 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.376 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.376 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.376 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.376 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011126+0000", 2024-09-11T03:46:46.376 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.376 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.376 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.377 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.377 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011127+0000", 2024-09-11T03:46:46.377 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8800000000000003e-07 2024-09-11T03:46:46.377 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.377 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.377 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.377 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011129+0000", 2024-09-11T03:46:46.377 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0090000000000002e-06 2024-09-11T03:46:46.377 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.377 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.377 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.377 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011146+0000", 2024-09-11T03:46:46.378 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7209000000000001e-05 2024-09-11T03:46:46.378 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.378 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.378 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.378 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011152+0000", 2024-09-11T03:46:46.378 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5160000000000002e-06 2024-09-11T03:46:46.378 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.378 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.378 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.378 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012718+0000", 2024-09-11T03:46:46.378 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001566413 2024-09-11T03:46:46.378 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.379 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.379 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:46:46.379 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012793+0000", 2024-09-11T03:46:46.379 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5197999999999994e-05 2024-09-11T03:46:46.379 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.379 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.379 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.379 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032530+0000", 2024-09-11T03:46:46.379 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019736848000000001 2024-09-11T03:46:46.379 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.379 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.379 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.379 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032577+0000", 2024-09-11T03:46:46.379 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6878000000000001e-05 2024-09-11T03:46:46.379 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.380 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.380 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.380 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032640+0000", 2024-09-11T03:46:46.380 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2970000000000002e-05 2024-09-11T03:46:46.380 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.380 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.380 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.380 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.380 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.380 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:5201 2.e e555/95 2:750de3b7:::smithi17557870-8:head v 555'820, mlcod=554'817)", 2024-09-11T03:46:46.380 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:12.971902+0000", 2024-09-11T03:46:46.380 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 33.269631926000002, 2024-09-11T03:46:46.380 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012248102, 2024-09-11T03:46:46.380 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:46.381 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:46.381 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:46.381 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:46.381 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.381 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:46.381 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:12.971902+0000", 2024-09-11T03:46:46.381 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.381 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.381 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.381 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:46.381 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:12.971902+0000", 2024-09-11T03:46:46.381 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:46.381 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.381 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.382 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:46.382 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:12.971902+0000", 2024-09-11T03:46:46.382 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.85e-07 2024-09-11T03:46:46.382 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.382 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.382 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:46.382 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:12.981226+0000", 2024-09-11T03:46:46.382 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0093240170000000004 2024-09-11T03:46:46.382 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.382 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.382 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:46.382 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:12.981251+0000", 2024-09-11T03:46:46.382 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5593e-05 2024-09-11T03:46:46.382 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.383 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.383 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:46.383 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:12.981264+0000", 2024-09-11T03:46:46.383 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.253e-05 2024-09-11T03:46:46.383 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.383 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.383 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:46.383 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:12.981363+0000", 2024-09-11T03:46:46.383 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9134000000000003e-05 2024-09-11T03:46:46.383 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.383 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.383 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:46.383 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:12.981385+0000", 2024-09-11T03:46:46.383 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1458999999999999e-05 2024-09-11T03:46:46.384 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.384 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.384 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:46.384 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:12.984084+0000", 2024-09-11T03:46:46.384 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0026995769999999999 2024-09-11T03:46:46.384 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:46.384 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:46.384 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:46.384 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:12.984150+0000", 2024-09-11T03:46:46.384 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5606999999999997e-05 2024-09-11T03:46:46.384 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.384 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.384 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.384 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:46.385 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:46.385 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:46:46.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:46.386 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:46.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:46.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:46 smithi084 ceph-mon[27254]: pgmap v1035: 67 pgs: 1 active+clean+scrubbing, 66 active+clean; 129 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:46:46.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:46 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:46.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4237160614' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:46:46.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4119697325' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:46.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:46 smithi084 ceph-mon[27254]: osdmap e587: 12 total, 12 up, 12 in 2024-09-11T03:46:46.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2479380750' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:46.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:46 smithi121 ceph-mon[30729]: pgmap v1035: 67 pgs: 1 active+clean+scrubbing, 66 active+clean; 129 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:46:46.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:46 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4237160614' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:46:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4119697325' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:46 smithi121 ceph-mon[30729]: osdmap e587: 12 total, 12 up, 12 in 2024-09-11T03:46:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2479380750' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:46.488 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:46.509 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:46.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:46.510 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:46.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:46.632 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:46.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:46.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:46.734 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:46.756 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:46.756 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:46.857 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:46.878 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:46.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:46.879 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:46.980 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:47.006 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:47.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:47.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:47.101 INFO:tasks.rados.rados.2.smithi175.stdout:2210: done (1 left) 2024-09-11T03:46:47.101 INFO:tasks.rados.rados.2.smithi175.stdout:2211: snap_remove snap 242 2024-09-11T03:46:47.108 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:47.130 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:47.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:47.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:47.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:47.259 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:47.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:47.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:47.305 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:47 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4237160614' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:46:47.305 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:47 smithi084 ceph-mon[27254]: osdmap e588: 12 total, 12 up, 12 in 2024-09-11T03:46:47.309 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-11T03:46:47.332 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-11T03:46:47.361 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:47.383 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:47.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:47.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:47 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4237160614' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:46:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:47 smithi121 ceph-mon[30729]: osdmap e588: 12 total, 12 up, 12 in 2024-09-11T03:46:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:47 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4237160614' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:46:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:47 smithi175 ceph-mon[30894]: osdmap e588: 12 total, 12 up, 12 in 2024-09-11T03:46:47.485 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:47.506 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:47.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:47.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:47.552 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:46:47.577 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:47.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:47.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:47.635 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:47.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:47.736 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:47.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:47.761 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:47.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:47.862 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:47.884 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:47.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:47.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:47.986 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:48.007 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:48.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:48.008 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:48.017 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:46:48.017 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:46:48.017 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:46:48.017 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:46:48.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:46:48.097 INFO:tasks.rados.rados.2.smithi175.stdout:2211: done (1 left) 2024-09-11T03:46:48.097 INFO:tasks.rados.rados.2.smithi175.stdout:2212: read oid 46 snap -1 2024-09-11T03:46:48.098 INFO:tasks.rados.rados.2.smithi175.stdout:2212: expect deleted 2024-09-11T03:46:48.098 INFO:tasks.rados.rados.2.smithi175.stdout:2213: rollback oid 19 current snap is 244 2024-09-11T03:46:48.098 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 19 to 238 2024-09-11T03:46:48.098 INFO:tasks.rados.rados.2.smithi175.stdout:2208: finishing write tid 1 to smithi17557870-21 2024-09-11T03:46:48.098 INFO:tasks.rados.rados.2.smithi175.stdout:2208: finishing write tid 2 to smithi17557870-21 2024-09-11T03:46:48.098 INFO:tasks.rados.rados.2.smithi175.stdout:2208: finishing write tid 3 to smithi17557870-21 2024-09-11T03:46:48.098 INFO:tasks.rados.rados.2.smithi175.stdout:2208: finishing write tid 4 to smithi17557870-21 2024-09-11T03:46:48.098 INFO:tasks.rados.rados.2.smithi175.stdout:2208: oid 21 updating version 0 to 894 2024-09-11T03:46:48.098 INFO:tasks.rados.rados.2.smithi175.stdout:2208: oid 21 updating version 894 to 896 2024-09-11T03:46:48.098 INFO:tasks.rados.rados.2.smithi175.stdout:2208: oid 21 version 896 is already newer than 895 2024-09-11T03:46:48.098 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 896 (ObjNum 746 snap 244 seq_num 746) dirty exists 2024-09-11T03:46:48.098 INFO:tasks.rados.rados.2.smithi175.stdout:2208: left oid 21 (ObjNum 746 snap 244 seq_num 746) 2024-09-11T03:46:48.098 INFO:tasks.rados.rados.2.smithi175.stdout:2208: done (2 left) 2024-09-11T03:46:48.099 INFO:tasks.rados.rados.2.smithi175.stdout:2214: delete oid 9 current snap is 244 2024-09-11T03:46:48.104 INFO:tasks.rados.rados.2.smithi175.stdout:2213: finishing rollback tid 1 to smithi17557870-19 2024-09-11T03:46:48.104 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 0 (ObjNum 513 snap 0 seq_num 2931793616) dirty dne 2024-09-11T03:46:48.106 INFO:tasks.rados.rados.2.smithi175.stdout:2212: done (2 left) 2024-09-11T03:46:48.106 INFO:tasks.rados.rados.2.smithi175.stdout:2213: done (1 left) 2024-09-11T03:46:48.106 INFO:tasks.rados.rados.2.smithi175.stdout:2214: done (0 left) 2024-09-11T03:46:48.107 INFO:tasks.rados.rados.2.smithi175.stdout:2215: snap_create 2024-09-11T03:46:48.109 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:48.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:48.131 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:48.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:48.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:48.254 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:48.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:48.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:48.316 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:48.356 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:48.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:48.378 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:48.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:48 smithi084 ceph-mon[27254]: pgmap v1038: 67 pgs: 67 active+clean; 123 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.3 MiB/s wr, 10 op/s 2024-09-11T03:46:48.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:48 smithi084 ceph-mon[27254]: osdmap e589: 12 total, 12 up, 12 in 2024-09-11T03:46:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:48 smithi121 ceph-mon[30729]: pgmap v1038: 67 pgs: 67 active+clean; 123 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.3 MiB/s wr, 10 op/s 2024-09-11T03:46:48.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:48 smithi121 ceph-mon[30729]: osdmap e589: 12 total, 12 up, 12 in 2024-09-11T03:46:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:48 smithi175 ceph-mon[30894]: pgmap v1038: 67 pgs: 67 active+clean; 123 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.3 MiB/s wr, 10 op/s 2024-09-11T03:46:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:48 smithi175 ceph-mon[30894]: osdmap e589: 12 total, 12 up, 12 in 2024-09-11T03:46:48.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:48.500 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:48.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:48.601 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:48.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:48.624 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:48.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:48.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:46:48.725 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:48.746 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:48.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:48.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:48.848 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:48.870 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:48.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:48.874 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-11T03:46:48.937 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:48.973 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:48.994 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:48.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:48.995 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:49.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:49.105 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:46:49.105 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T03:46:49.109 INFO:tasks.rados.rados.2.smithi175.stdout:2215: done (0 left) 2024-09-11T03:46:49.109 INFO:tasks.rados.rados.2.smithi175.stdout:2216: snap_remove snap 244 2024-09-11T03:46:49.122 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:49.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:49.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:49.159 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:46:49.224 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:49.245 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:49.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:49.246 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:49.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:49.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:49.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:49 smithi084 ceph-mon[27254]: osdmap e590: 12 total, 12 up, 12 in 2024-09-11T03:46:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2924961676' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:46:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/283251795' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:46:49.417 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:49 smithi175 ceph-mon[30894]: osdmap e590: 12 total, 12 up, 12 in 2024-09-11T03:46:49.417 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2924961676' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:46:49.417 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/283251795' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:46:49.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:49 smithi121 ceph-mon[30729]: osdmap e590: 12 total, 12 up, 12 in 2024-09-11T03:46:49.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2924961676' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:46:49.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/283251795' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:46:49.475 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:49.497 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:49.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:49.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:49.599 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:49.605 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:46:49.605 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:46:49.606 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:46:49.606 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:46:49.606 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:46:49.620 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:49.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:49.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:49.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:49.748 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:49.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:49.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:49.850 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:49.873 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:49.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:49.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:49.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:50.000 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:50.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:50.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:50.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-11T03:46:50.102 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:50.112 INFO:tasks.rados.rados.2.smithi175.stdout:2216: done (0 left) 2024-09-11T03:46:50.112 INFO:tasks.rados.rados.2.smithi175.stdout:2217: delete oid 31 current snap is 245 2024-09-11T03:46:50.119 INFO:tasks.rados.rados.2.smithi175.stdout:2217: done (0 left) 2024-09-11T03:46:50.119 INFO:tasks.rados.rados.2.smithi175.stdout:2218: rollback oid 8 current snap is 245 2024-09-11T03:46:50.120 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 8 to 238 2024-09-11T03:46:50.120 INFO:tasks.rados.rados.2.smithi175.stdout:2219: snap_create 2024-09-11T03:46:50.123 INFO:tasks.rados.rados.2.smithi175.stdout:2218: finishing rollback tid 0 to smithi17557870-8 2024-09-11T03:46:50.124 INFO:tasks.rados.rados.2.smithi175.stdout:2218: finishing rollback tid 1 to smithi17557870-8 2024-09-11T03:46:50.124 INFO:tasks.rados.rados.2.smithi175.stdout:2218: finishing rollback tid 2 to smithi17557870-8 2024-09-11T03:46:50.124 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 859 (ObjNum 565 snap 172 seq_num 565) dirty exists 2024-09-11T03:46:50.126 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:50.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:50.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:50 smithi175 ceph-mon[30894]: pgmap v1041: 67 pgs: 67 active+clean; 126 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-11T03:46:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:50 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2924961676' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:46:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:50 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/283251795' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:46:50.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:50 smithi175 ceph-mon[30894]: osdmap e591: 12 total, 12 up, 12 in 2024-09-11T03:46:50.228 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:50.249 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:50.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:50.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:50.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:50 smithi084 ceph-mon[27254]: pgmap v1041: 67 pgs: 67 active+clean; 126 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-11T03:46:50.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:50 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2924961676' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:46:50.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:50 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/283251795' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:46:50.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:50 smithi084 ceph-mon[27254]: osdmap e591: 12 total, 12 up, 12 in 2024-09-11T03:46:50.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:50.364 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:50 smithi121 ceph-mon[30729]: pgmap v1041: 67 pgs: 67 active+clean; 126 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-11T03:46:50.364 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:50 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2924961676' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:46:50.364 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:50 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/283251795' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:46:50.364 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:50 smithi121 ceph-mon[30729]: osdmap e591: 12 total, 12 up, 12 in 2024-09-11T03:46:50.371 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-11T03:46:50.373 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:50.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:50.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:50.476 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:50.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:50.498 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:50.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:50.599 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:50.614 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:46:50.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:50.622 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:50.622 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:50.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:50.744 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:50.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:50.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:50.772 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:46:50.773 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:46:50.789 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:46:50.789 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:46:50.805 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:46:50.805 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:46:50.824 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:46:50.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:46:50.843 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:46:50.843 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:46:50.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:50.869 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:46:50.869 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:46:50.870 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:50.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:50.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:50.880 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:50.890 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:46:50.891 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:46:50.914 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:46:50.914 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:46:50.930 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:46:50.931 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:46:50.947 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:46:50.947 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:46:50.963 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:46:50.963 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:46:50.972 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:50.980 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:46:50.980 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:46:50.994 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:50.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:50.995 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:51.091 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:46:51.091 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:46:51.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:51.124 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:51.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:51.125 INFO:tasks.rados.rados.2.smithi175.stdout:2218: done (1 left) 2024-09-11T03:46:51.125 INFO:tasks.rados.rados.2.smithi175.stdout:2219: done (0 left) 2024-09-11T03:46:51.125 INFO:tasks.rados.rados.2.smithi175.stdout:2220: rollback oid 40 current snap is 246 2024-09-11T03:46:51.125 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 40 to 245 2024-09-11T03:46:51.126 INFO:tasks.rados.rados.2.smithi175.stdout:2221: copy_from oid 22 from oid 18 current snap is 246 2024-09-11T03:46:51.126 INFO:tasks.rados.rados.2.smithi175.stdout:2222: write (excl) oid 27 current snap is 246 2024-09-11T03:46:51.126 INFO:tasks.rados.rados.2.smithi175.stdout:2222: seq_num 750 ranges {605250=639344,1855142=452579} 2024-09-11T03:46:51.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:51.131 INFO:tasks.rados.rados.2.smithi175.stdout:2222: writing smithi17557870-27 from 605250 to 1244594 tid 1 2024-09-11T03:46:51.137 INFO:tasks.rados.rados.2.smithi175.stdout:2222: writing smithi17557870-27 from 1855142 to 2307721 tid 2 2024-09-11T03:46:51.137 INFO:tasks.rados.rados.2.smithi175.stdout:2223: snap_remove snap 238 2024-09-11T03:46:51.137 INFO:tasks.rados.rados.2.smithi175.stdout:2221: finishing copy_from to smithi17557870-22 2024-09-11T03:46:51.138 INFO:tasks.rados.rados.2.smithi175.stdout:2221: got expected ENOENT (src dne) 2024-09-11T03:46:51.138 INFO:tasks.rados.rados.2.smithi175.stdout:2221: finishing copy_from racing read to smithi17557870-22 2024-09-11T03:46:51.138 INFO:tasks.rados.rados.2.smithi175.stdout:2220: finishing rollback tid 0 to smithi17557870-40 2024-09-11T03:46:51.138 INFO:tasks.rados.rados.2.smithi175.stdout:2220: finishing rollback tid 1 to smithi17557870-40 2024-09-11T03:46:51.138 INFO:tasks.rados.rados.2.smithi175.stdout:2220: finishing rollback tid 2 to smithi17557870-40 2024-09-11T03:46:51.138 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 424 (ObjNum 697 snap 227 seq_num 697) dirty exists 2024-09-11T03:46:51.138 INFO:tasks.rados.rados.2.smithi175.stdout:2222: finishing write tid 1 to smithi17557870-27 2024-09-11T03:46:51.227 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:51.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:51.249 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:51.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:51.321 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:51 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:46:51.321 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:51 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:46:51.321 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:51 smithi175 ceph-mon[30894]: 2.4 scrub starts 2024-09-11T03:46:51.321 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:51 smithi175 ceph-mon[30894]: 2.4 scrub ok 2024-09-11T03:46:51.321 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:51 smithi175 ceph-mon[30894]: osdmap e592: 12 total, 12 up, 12 in 2024-09-11T03:46:51.322 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3495852710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:51.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:51.376 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:51.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:51.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:51 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:46:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:51 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:46:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:51 smithi084 ceph-mon[27254]: 2.4 scrub starts 2024-09-11T03:46:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:51 smithi084 ceph-mon[27254]: 2.4 scrub ok 2024-09-11T03:46:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:51 smithi084 ceph-mon[27254]: osdmap e592: 12 total, 12 up, 12 in 2024-09-11T03:46:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3495852710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:51 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:46:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:51 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:46:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:51 smithi121 ceph-mon[30729]: 2.4 scrub starts 2024-09-11T03:46:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:51 smithi121 ceph-mon[30729]: 2.4 scrub ok 2024-09-11T03:46:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:51 smithi121 ceph-mon[30729]: osdmap e592: 12 total, 12 up, 12 in 2024-09-11T03:46:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3495852710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:51.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:51.504 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:51.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:51.606 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:51.628 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:51.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:51.730 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:51.752 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:51.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:51.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:51.806 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-11T03:46:51.854 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:51.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:51.876 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:51.876 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:51.971 INFO:tasks.rados.rados.2.smithi175.stdout:2220: done (3 left) 2024-09-11T03:46:51.971 INFO:tasks.rados.rados.2.smithi175.stdout:2221: done (2 left) 2024-09-11T03:46:51.971 INFO:tasks.rados.rados.2.smithi175.stdout:2223: done (1 left) 2024-09-11T03:46:51.971 INFO:tasks.rados.rados.2.smithi175.stdout:2224: copy_from oid 28 from oid 49 current snap is 246 2024-09-11T03:46:51.971 INFO:tasks.rados.rados.2.smithi175.stdout:2222: finishing write tid 2 to smithi17557870-27 2024-09-11T03:46:51.971 INFO:tasks.rados.rados.2.smithi175.stdout:2222: finishing write tid 3 to smithi17557870-27 2024-09-11T03:46:51.972 INFO:tasks.rados.rados.2.smithi175.stdout:2222: finishing write tid 4 to smithi17557870-27 2024-09-11T03:46:51.972 INFO:tasks.rados.rados.2.smithi175.stdout:2222: oid 27 updating version 0 to 905 2024-09-11T03:46:51.972 INFO:tasks.rados.rados.2.smithi175.stdout:2222: oid 27 updating version 905 to 906 2024-09-11T03:46:51.972 INFO:tasks.rados.rados.2.smithi175.stdout:2222: oid 27 updating version 906 to 907 2024-09-11T03:46:51.972 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 907 (ObjNum 750 snap 246 seq_num 750) dirty exists 2024-09-11T03:46:51.972 INFO:tasks.rados.rados.2.smithi175.stdout:2222: left oid 27 (ObjNum 750 snap 246 seq_num 750) 2024-09-11T03:46:51.972 INFO:tasks.rados.rados.2.smithi175.stdout:2222: done (1 left) 2024-09-11T03:46:51.972 INFO:tasks.rados.rados.2.smithi175.stdout:2225: write oid 9 current snap is 246 2024-09-11T03:46:51.972 INFO:tasks.rados.rados.2.smithi175.stdout:2225: seq_num 751 ranges {622057=714593,1913624=708433} 2024-09-11T03:46:51.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:51.979 INFO:tasks.rados.rados.2.smithi175.stdout:2225: writing smithi17557870-9 from 622057 to 1336650 tid 1 2024-09-11T03:46:51.988 INFO:tasks.rados.rados.2.smithi175.stdout:2225: writing smithi17557870-9 from 1913624 to 2622057 tid 2 2024-09-11T03:46:51.988 INFO:tasks.rados.rados.2.smithi175.stdout:2226: snap_remove snap 245 2024-09-11T03:46:52.003 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:52.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:52.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:52.057 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:46:52.105 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:52.126 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:52.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:52.127 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:52.228 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:52.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:52.251 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:52.251 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:52.352 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:52.379 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:52.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:52.380 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:52 smithi084 ceph-mon[27254]: 2.12 scrub starts 2024-09-11T03:46:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:52 smithi084 ceph-mon[27254]: 2.12 scrub ok 2024-09-11T03:46:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:52 smithi084 ceph-mon[27254]: pgmap v1044: 67 pgs: 67 active+clean; 126 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 641 KiB/s wr, 6 op/s 2024-09-11T03:46:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:52 smithi084 ceph-mon[27254]: osdmap e593: 12 total, 12 up, 12 in 2024-09-11T03:46:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:52 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:52 smithi084 ceph-mon[27254]: osdmap e594: 12 total, 12 up, 12 in 2024-09-11T03:46:52.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:52 smithi121 ceph-mon[30729]: 2.12 scrub starts 2024-09-11T03:46:52.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:52 smithi121 ceph-mon[30729]: 2.12 scrub ok 2024-09-11T03:46:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:52 smithi121 ceph-mon[30729]: pgmap v1044: 67 pgs: 67 active+clean; 126 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 641 KiB/s wr, 6 op/s 2024-09-11T03:46:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:52 smithi121 ceph-mon[30729]: osdmap e593: 12 total, 12 up, 12 in 2024-09-11T03:46:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:52 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:52 smithi121 ceph-mon[30729]: osdmap e594: 12 total, 12 up, 12 in 2024-09-11T03:46:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:52 smithi175 ceph-mon[30894]: 2.12 scrub starts 2024-09-11T03:46:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:52 smithi175 ceph-mon[30894]: 2.12 scrub ok 2024-09-11T03:46:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:52 smithi175 ceph-mon[30894]: pgmap v1044: 67 pgs: 67 active+clean; 126 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 641 KiB/s wr, 6 op/s 2024-09-11T03:46:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:52 smithi175 ceph-mon[30894]: osdmap e593: 12 total, 12 up, 12 in 2024-09-11T03:46:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:52 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:46:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:52 smithi175 ceph-mon[30894]: osdmap e594: 12 total, 12 up, 12 in 2024-09-11T03:46:52.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:52.507 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:52.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:52.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:52.558 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:46:52.580 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:46:52.581 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:46:52.608 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:52.628 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:52.629 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:52.730 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:52.751 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:52.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:52.752 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:52.799 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:52.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:52.878 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:52.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:52.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:52.968 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T03:46:52.973 INFO:tasks.rados.rados.2.smithi175.stdout:2226: done (2 left) 2024-09-11T03:46:52.974 INFO:tasks.rados.rados.2.smithi175.stdout:2227: write oid 5 current snap is 246 2024-09-11T03:46:52.974 INFO:tasks.rados.rados.2.smithi175.stdout:2224: finishing copy_from racing read to smithi17557870-28 2024-09-11T03:46:52.974 INFO:tasks.rados.rados.2.smithi175.stdout:2227: seq_num 752 ranges {638864=789842,1972106=456248} 2024-09-11T03:46:52.980 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:52.981 INFO:tasks.rados.rados.2.smithi175.stdout:2227: writing smithi17557870-5 from 638864 to 1428706 tid 1 2024-09-11T03:46:52.988 INFO:tasks.rados.rados.2.smithi175.stdout:2227: writing smithi17557870-5 from 1972106 to 2428354 tid 2 2024-09-11T03:46:52.988 INFO:tasks.rados.rados.2.smithi175.stdout:2228: write oid 8 current snap is 246 2024-09-11T03:46:52.988 INFO:tasks.rados.rados.2.smithi175.stdout:2224: finishing copy_from to smithi17557870-28 2024-09-11T03:46:52.988 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 679 (ObjNum 743 snap 243 seq_num 743) dirty exists 2024-09-11T03:46:52.988 INFO:tasks.rados.rados.2.smithi175.stdout:2225: finishing write tid 1 to smithi17557870-9 2024-09-11T03:46:52.988 INFO:tasks.rados.rados.2.smithi175.stdout:2225: finishing write tid 2 to smithi17557870-9 2024-09-11T03:46:52.988 INFO:tasks.rados.rados.2.smithi175.stdout:2225: finishing write tid 3 to smithi17557870-9 2024-09-11T03:46:52.988 INFO:tasks.rados.rados.2.smithi175.stdout:2225: finishing write tid 4 to smithi17557870-9 2024-09-11T03:46:52.988 INFO:tasks.rados.rados.2.smithi175.stdout:2225: oid 9 updating version 0 to 915 2024-09-11T03:46:52.988 INFO:tasks.rados.rados.2.smithi175.stdout:2225: oid 9 version 915 is already newer than 914 2024-09-11T03:46:52.988 INFO:tasks.rados.rados.2.smithi175.stdout:2225: oid 9 updating version 915 to 916 2024-09-11T03:46:52.989 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 916 (ObjNum 751 snap 246 seq_num 751) dirty exists 2024-09-11T03:46:52.989 INFO:tasks.rados.rados.2.smithi175.stdout:2225: left oid 9 (ObjNum 751 snap 246 seq_num 751) 2024-09-11T03:46:52.989 INFO:tasks.rados.rados.2.smithi175.stdout:2228: seq_num 753 ranges {655671=581444,2030588=625083} 2024-09-11T03:46:52.994 INFO:tasks.rados.rados.2.smithi175.stdout:2228: writing smithi17557870-8 from 655671 to 1237115 tid 1 2024-09-11T03:46:53.002 INFO:tasks.rados.rados.2.smithi175.stdout:2228: writing smithi17557870-8 from 2030588 to 2655671 tid 2 2024-09-11T03:46:53.002 INFO:tasks.rados.rados.2.smithi175.stdout:2224: done (3 left) 2024-09-11T03:46:53.002 INFO:tasks.rados.rados.2.smithi175.stdout:2225: done (2 left) 2024-09-11T03:46:53.002 INFO:tasks.rados.rados.2.smithi175.stdout:2229: read oid 17 snap -1 2024-09-11T03:46:53.003 INFO:tasks.rados.rados.2.smithi175.stdout:2229: expect deleted 2024-09-11T03:46:53.003 INFO:tasks.rados.rados.2.smithi175.stdout:2230: write (excl) oid 2 current snap is 246 2024-09-11T03:46:53.003 INFO:tasks.rados.rados.2.smithi175.stdout:2227: finishing write tid 1 to smithi17557870-5 2024-09-11T03:46:53.003 INFO:tasks.rados.rados.2.smithi175.stdout:2230: seq_num 754 ranges {672478=656693,2089070=459917} 2024-09-11T03:46:53.004 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:53.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:53.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:53.009 INFO:tasks.rados.rados.2.smithi175.stdout:2230: writing smithi17557870-2 from 672478 to 1329171 tid 1 2024-09-11T03:46:53.017 INFO:tasks.rados.rados.2.smithi175.stdout:2230: writing smithi17557870-2 from 2089070 to 2548987 tid 2 2024-09-11T03:46:53.017 INFO:tasks.rados.rados.2.smithi175.stdout:2231: copy_from oid 29 from oid 50 current snap is 246 2024-09-11T03:46:53.017 INFO:tasks.rados.rados.2.smithi175.stdout:2227: finishing write tid 2 to smithi17557870-5 2024-09-11T03:46:53.017 INFO:tasks.rados.rados.2.smithi175.stdout:2227: finishing write tid 3 to smithi17557870-5 2024-09-11T03:46:53.017 INFO:tasks.rados.rados.2.smithi175.stdout:2227: finishing write tid 4 to smithi17557870-5 2024-09-11T03:46:53.018 INFO:tasks.rados.rados.2.smithi175.stdout:2227: oid 5 updating version 0 to 706 2024-09-11T03:46:53.018 INFO:tasks.rados.rados.2.smithi175.stdout:2227: oid 5 version 706 is already newer than 705 2024-09-11T03:46:53.018 INFO:tasks.rados.rados.2.smithi175.stdout:2227: oid 5 updating version 706 to 707 2024-09-11T03:46:53.018 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 707 (ObjNum 752 snap 246 seq_num 752) dirty exists 2024-09-11T03:46:53.018 INFO:tasks.rados.rados.2.smithi175.stdout:2227: left oid 5 (ObjNum 752 snap 246 seq_num 752) 2024-09-11T03:46:53.018 INFO:tasks.rados.rados.2.smithi175.stdout:2228: finishing write tid 1 to smithi17557870-8 2024-09-11T03:46:53.018 INFO:tasks.rados.rados.2.smithi175.stdout:2227: done (4 left) 2024-09-11T03:46:53.018 INFO:tasks.rados.rados.2.smithi175.stdout:2232: write (excl) oid 49 current snap is 246 2024-09-11T03:46:53.018 INFO:tasks.rados.rados.2.smithi175.stdout:2228: finishing write tid 2 to smithi17557870-8 2024-09-11T03:46:53.018 INFO:tasks.rados.rados.2.smithi175.stdout:2232: seq_num 755 ranges {689285=731942,2147552=541733} 2024-09-11T03:46:53.024 INFO:tasks.rados.rados.2.smithi175.stdout:2232: writing smithi17557870-49 from 689285 to 1421227 tid 1 2024-09-11T03:46:53.032 INFO:tasks.rados.rados.2.smithi175.stdout:2232: writing smithi17557870-49 from 2147552 to 2689285 tid 2 2024-09-11T03:46:53.032 INFO:tasks.rados.rados.2.smithi175.stdout:2229: done (4 left) 2024-09-11T03:46:53.032 INFO:tasks.rados.rados.2.smithi175.stdout:2233: rollback oid 6 current snap is 246 2024-09-11T03:46:53.032 INFO:tasks.rados.rados.2.smithi175.stdout:2233: done (4 left) 2024-09-11T03:46:53.032 INFO:tasks.rados.rados.2.smithi175.stdout:2234: read oid 20 snap -1 2024-09-11T03:46:53.032 INFO:tasks.rados.rados.2.smithi175.stdout:2234: expect (ObjNum 563 snap 172 seq_num 563) 2024-09-11T03:46:53.032 INFO:tasks.rados.rados.2.smithi175.stdout:2235: write (excl) oid 26 current snap is 246 2024-09-11T03:46:53.032 INFO:tasks.rados.rados.2.smithi175.stdout:2228: finishing write tid 3 to smithi17557870-8 2024-09-11T03:46:53.032 INFO:tasks.rados.rados.2.smithi175.stdout:2228: finishing write tid 4 to smithi17557870-8 2024-09-11T03:46:53.033 INFO:tasks.rados.rados.2.smithi175.stdout:2228: oid 8 updating version 0 to 862 2024-09-11T03:46:53.033 INFO:tasks.rados.rados.2.smithi175.stdout:2228: oid 8 updating version 862 to 863 2024-09-11T03:46:53.033 INFO:tasks.rados.rados.2.smithi175.stdout:2228: oid 8 updating version 863 to 864 2024-09-11T03:46:53.033 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 864 (ObjNum 753 snap 246 seq_num 753) dirty exists 2024-09-11T03:46:53.033 INFO:tasks.rados.rados.2.smithi175.stdout:2228: left oid 8 (ObjNum 753 snap 246 seq_num 753) 2024-09-11T03:46:53.033 INFO:tasks.rados.rados.2.smithi175.stdout:2230: finishing write tid 1 to smithi17557870-2 2024-09-11T03:46:53.033 INFO:tasks.rados.rados.2.smithi175.stdout:2235: seq_num 756 ranges {706092=407191,1806034=463586} 2024-09-11T03:46:53.037 INFO:tasks.rados.rados.2.smithi175.stdout:2235: writing smithi17557870-26 from 706092 to 1113283 tid 1 2024-09-11T03:46:53.044 INFO:tasks.rados.rados.2.smithi175.stdout:2235: writing smithi17557870-26 from 1806034 to 2269620 tid 2 2024-09-11T03:46:53.044 INFO:tasks.rados.rados.2.smithi175.stdout:2228: done (5 left) 2024-09-11T03:46:53.044 INFO:tasks.rados.rados.2.smithi175.stdout:2236: write oid 36 current snap is 246 2024-09-11T03:46:53.044 INFO:tasks.rados.rados.2.smithi175.stdout:2231: finishing copy_from to smithi17557870-29 2024-09-11T03:46:53.045 INFO:tasks.rados.rados.2.smithi175.stdout:2231: got expected ENOENT (src dne) 2024-09-11T03:46:53.045 INFO:tasks.rados.rados.2.smithi175.stdout:2236: seq_num 757 ranges {722899=482440,1748163=723597} 2024-09-11T03:46:53.050 INFO:tasks.rados.rados.2.smithi175.stdout:2236: writing smithi17557870-36 from 722899 to 1205339 tid 1 2024-09-11T03:46:53.058 INFO:tasks.rados.rados.2.smithi175.stdout:2236: writing smithi17557870-36 from 1748163 to 2471760 tid 2 2024-09-11T03:46:53.058 INFO:tasks.rados.rados.2.smithi175.stdout:2237: write oid 31 current snap is 246 2024-09-11T03:46:53.058 INFO:tasks.rados.rados.2.smithi175.stdout:2231: finishing copy_from racing read to smithi17557870-29 2024-09-11T03:46:53.058 INFO:tasks.rados.rados.2.smithi175.stdout:2230: finishing write tid 2 to smithi17557870-2 2024-09-11T03:46:53.058 INFO:tasks.rados.rados.2.smithi175.stdout:2230: finishing write tid 3 to smithi17557870-2 2024-09-11T03:46:53.058 INFO:tasks.rados.rados.2.smithi175.stdout:2230: finishing write tid 4 to smithi17557870-2 2024-09-11T03:46:53.058 INFO:tasks.rados.rados.2.smithi175.stdout:2230: oid 2 updating version 0 to 921 2024-09-11T03:46:53.058 INFO:tasks.rados.rados.2.smithi175.stdout:2230: oid 2 version 921 is already newer than 920 2024-09-11T03:46:53.058 INFO:tasks.rados.rados.2.smithi175.stdout:2230: oid 2 version 921 is already newer than 917 2024-09-11T03:46:53.058 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 921 (ObjNum 754 snap 246 seq_num 754) dirty exists 2024-09-11T03:46:53.059 INFO:tasks.rados.rados.2.smithi175.stdout:2230: left oid 2 (ObjNum 754 snap 246 seq_num 754) 2024-09-11T03:46:53.059 INFO:tasks.rados.rados.2.smithi175.stdout:2232: finishing write tid 1 to smithi17557870-49 2024-09-11T03:46:53.059 INFO:tasks.rados.rados.2.smithi175.stdout:2237: seq_num 758 ranges {739706=557689,1806645=467255} 2024-09-11T03:46:53.064 INFO:tasks.rados.rados.2.smithi175.stdout:2237: writing smithi17557870-31 from 739706 to 1297395 tid 1 2024-09-11T03:46:53.070 INFO:tasks.rados.rados.2.smithi175.stdout:2237: writing smithi17557870-31 from 1806645 to 2273900 tid 2 2024-09-11T03:46:53.070 INFO:tasks.rados.rados.2.smithi175.stdout:2230: done (6 left) 2024-09-11T03:46:53.070 INFO:tasks.rados.rados.2.smithi175.stdout:2231: done (5 left) 2024-09-11T03:46:53.070 INFO:tasks.rados.rados.2.smithi175.stdout:2238: delete oid 33 current snap is 246 2024-09-11T03:46:53.071 INFO:tasks.rados.rados.2.smithi175.stdout:2232: finishing write tid 2 to smithi17557870-49 2024-09-11T03:46:53.071 INFO:tasks.rados.rados.2.smithi175.stdout:2232: finishing write tid 3 to smithi17557870-49 2024-09-11T03:46:53.071 INFO:tasks.rados.rados.2.smithi175.stdout:2232: finishing write tid 4 to smithi17557870-49 2024-09-11T03:46:53.071 INFO:tasks.rados.rados.2.smithi175.stdout:2232: oid 49 updating version 0 to 684 2024-09-11T03:46:53.071 INFO:tasks.rados.rados.2.smithi175.stdout:2232: oid 49 version 684 is already newer than 682 2024-09-11T03:46:53.071 INFO:tasks.rados.rados.2.smithi175.stdout:2232: oid 49 version 684 is already newer than 683 2024-09-11T03:46:53.071 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 684 (ObjNum 755 snap 246 seq_num 755) dirty exists 2024-09-11T03:46:53.071 INFO:tasks.rados.rados.2.smithi175.stdout:2232: left oid 49 (ObjNum 755 snap 246 seq_num 755) 2024-09-11T03:46:53.071 INFO:tasks.rados.rados.2.smithi175.stdout:2235: finishing write tid 1 to smithi17557870-26 2024-09-11T03:46:53.071 INFO:tasks.rados.rados.2.smithi175.stdout:2234: expect (ObjNum 563 snap 172 seq_num 563) 2024-09-11T03:46:53.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:53.130 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:53.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:53.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:53.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:53.255 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:53.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:53.271 INFO:tasks.rados.rados.2.smithi175.stdout:2235: finishing write tid 2 to smithi17557870-26 2024-09-11T03:46:53.271 INFO:tasks.rados.rados.2.smithi175.stdout:2235: finishing write tid 3 to smithi17557870-26 2024-09-11T03:46:53.271 INFO:tasks.rados.rados.2.smithi175.stdout:2232: done (5 left) 2024-09-11T03:46:53.271 INFO:tasks.rados.rados.2.smithi175.stdout:2234: done (4 left) 2024-09-11T03:46:53.272 INFO:tasks.rados.rados.2.smithi175.stdout:2238: done (3 left) 2024-09-11T03:46:53.272 INFO:tasks.rados.rados.2.smithi175.stdout:2239: snap_create 2024-09-11T03:46:53.272 INFO:tasks.rados.rados.2.smithi175.stdout:2235: finishing write tid 4 to smithi17557870-26 2024-09-11T03:46:53.272 INFO:tasks.rados.rados.2.smithi175.stdout:2235: oid 26 updating version 0 to 694 2024-09-11T03:46:53.272 INFO:tasks.rados.rados.2.smithi175.stdout:2235: oid 26 updating version 694 to 695 2024-09-11T03:46:53.272 INFO:tasks.rados.rados.2.smithi175.stdout:2235: oid 26 updating version 695 to 696 2024-09-11T03:46:53.272 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 696 (ObjNum 756 snap 246 seq_num 756) dirty exists 2024-09-11T03:46:53.272 INFO:tasks.rados.rados.2.smithi175.stdout:2235: left oid 26 (ObjNum 756 snap 246 seq_num 756) 2024-09-11T03:46:53.272 INFO:tasks.rados.rados.2.smithi175.stdout:2236: finishing write tid 1 to smithi17557870-36 2024-09-11T03:46:53.273 INFO:tasks.rados.rados.2.smithi175.stdout:2236: finishing write tid 2 to smithi17557870-36 2024-09-11T03:46:53.273 INFO:tasks.rados.rados.2.smithi175.stdout:2236: finishing write tid 3 to smithi17557870-36 2024-09-11T03:46:53.273 INFO:tasks.rados.rados.2.smithi175.stdout:2236: finishing write tid 4 to smithi17557870-36 2024-09-11T03:46:53.273 INFO:tasks.rados.rados.2.smithi175.stdout:2236: oid 36 updating version 0 to 367 2024-09-11T03:46:53.273 INFO:tasks.rados.rados.2.smithi175.stdout:2236: oid 36 updating version 367 to 369 2024-09-11T03:46:53.273 INFO:tasks.rados.rados.2.smithi175.stdout:2236: oid 36 version 369 is already newer than 368 2024-09-11T03:46:53.273 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 369 (ObjNum 757 snap 246 seq_num 757) dirty exists 2024-09-11T03:46:53.273 INFO:tasks.rados.rados.2.smithi175.stdout:2236: left oid 36 (ObjNum 757 snap 246 seq_num 757) 2024-09-11T03:46:53.273 INFO:tasks.rados.rados.2.smithi175.stdout:2237: finishing write tid 1 to smithi17557870-31 2024-09-11T03:46:53.273 INFO:tasks.rados.rados.2.smithi175.stdout:2237: finishing write tid 2 to smithi17557870-31 2024-09-11T03:46:53.274 INFO:tasks.rados.rados.2.smithi175.stdout:2237: finishing write tid 3 to smithi17557870-31 2024-09-11T03:46:53.274 INFO:tasks.rados.rados.2.smithi175.stdout:2237: finishing write tid 4 to smithi17557870-31 2024-09-11T03:46:53.274 INFO:tasks.rados.rados.2.smithi175.stdout:2237: oid 31 updating version 0 to 924 2024-09-11T03:46:53.274 INFO:tasks.rados.rados.2.smithi175.stdout:2237: oid 31 version 924 is already newer than 922 2024-09-11T03:46:53.274 INFO:tasks.rados.rados.2.smithi175.stdout:2237: oid 31 version 924 is already newer than 923 2024-09-11T03:46:53.274 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 924 (ObjNum 758 snap 246 seq_num 758) dirty exists 2024-09-11T03:46:53.274 INFO:tasks.rados.rados.2.smithi175.stdout:2237: left oid 31 (ObjNum 758 snap 246 seq_num 758) 2024-09-11T03:46:53.355 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-11T03:46:53.355 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:53.363 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:53 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2423960466' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:46:53.363 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:53 smithi084 ceph-mon[27254]: 2.19 scrub starts 2024-09-11T03:46:53.363 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:53 smithi084 ceph-mon[27254]: 2.19 scrub ok 2024-09-11T03:46:53.363 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:53 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2423960466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:46:53.363 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:53 smithi084 ceph-mon[27254]: osdmap e595: 12 total, 12 up, 12 in 2024-09-11T03:46:53.376 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:53 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2423960466' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:46:53.376 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:53 smithi175 ceph-mon[30894]: 2.19 scrub starts 2024-09-11T03:46:53.376 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:53 smithi175 ceph-mon[30894]: 2.19 scrub ok 2024-09-11T03:46:53.376 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:53 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2423960466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:46:53.376 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:53 smithi175 ceph-mon[30894]: osdmap e595: 12 total, 12 up, 12 in 2024-09-11T03:46:53.379 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:53.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:53.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:53 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2423960466' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:46:53.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:53 smithi121 ceph-mon[30729]: 2.19 scrub starts 2024-09-11T03:46:53.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:53 smithi121 ceph-mon[30729]: 2.19 scrub ok 2024-09-11T03:46:53.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:53 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2423960466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:46:53.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:53 smithi121 ceph-mon[30729]: osdmap e595: 12 total, 12 up, 12 in 2024-09-11T03:46:53.481 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:53.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:53.503 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:53.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:53.596 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:46:53.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:53.628 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:53.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:53.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:53.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:53.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-11T03:46:53.755 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:53.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:53.857 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:53.874 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:46:53.879 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:53.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:53.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:53.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:54.006 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:54.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:54.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:54.028 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:54.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:54.133 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:54.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:54.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:54.145 INFO:tasks.rados.rados.2.smithi175.stdout:2235: done (3 left) 2024-09-11T03:46:54.145 INFO:tasks.rados.rados.2.smithi175.stdout:2236: done (2 left) 2024-09-11T03:46:54.145 INFO:tasks.rados.rados.2.smithi175.stdout:2237: done (1 left) 2024-09-11T03:46:54.145 INFO:tasks.rados.rados.2.smithi175.stdout:2239: done (0 left) 2024-09-11T03:46:54.145 INFO:tasks.rados.rados.2.smithi175.stdout:2240: write (excl) oid 17 current snap is 247 2024-09-11T03:46:54.145 INFO:tasks.rados.rados.2.smithi175.stdout:2240: seq_num 760 ranges {773320=708187,1923609=470924} 2024-09-11T03:46:54.153 INFO:tasks.rados.rados.2.smithi175.stdout:2240: writing smithi17557870-17 from 773320 to 1481507 tid 1 2024-09-11T03:46:54.158 INFO:tasks.rados.rados.2.smithi175.stdout:2240: writing smithi17557870-17 from 1923609 to 2394533 tid 2 2024-09-11T03:46:54.158 INFO:tasks.rados.rados.2.smithi175.stdout:2241: write (excl) oid 8 current snap is 247 2024-09-11T03:46:54.158 INFO:tasks.rados.rados.2.smithi175.stdout:2241: seq_num 761 ranges {790127=499789,1698444=730935} 2024-09-11T03:46:54.165 INFO:tasks.rados.rados.2.smithi175.stdout:2241: writing smithi17557870-8 from 790127 to 1289916 tid 1 2024-09-11T03:46:54.172 INFO:tasks.rados.rados.2.smithi175.stdout:2241: writing smithi17557870-8 from 1698444 to 2429379 tid 2 2024-09-11T03:46:54.172 INFO:tasks.rados.rados.2.smithi175.stdout:2242: write (excl) oid 47 current snap is 247 2024-09-11T03:46:54.172 INFO:tasks.rados.rados.2.smithi175.stdout:2240: finishing write tid 1 to smithi17557870-17 2024-09-11T03:46:54.172 INFO:tasks.rados.rados.2.smithi175.stdout:2240: finishing write tid 2 to smithi17557870-17 2024-09-11T03:46:54.172 INFO:tasks.rados.rados.2.smithi175.stdout:2240: finishing write tid 3 to smithi17557870-17 2024-09-11T03:46:54.172 INFO:tasks.rados.rados.2.smithi175.stdout:2240: finishing write tid 4 to smithi17557870-17 2024-09-11T03:46:54.172 INFO:tasks.rados.rados.2.smithi175.stdout:2240: oid 17 updating version 0 to 764 2024-09-11T03:46:54.172 INFO:tasks.rados.rados.2.smithi175.stdout:2240: oid 17 updating version 764 to 765 2024-09-11T03:46:54.172 INFO:tasks.rados.rados.2.smithi175.stdout:2240: oid 17 version 765 is already newer than 763 2024-09-11T03:46:54.172 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 765 (ObjNum 760 snap 247 seq_num 760) dirty exists 2024-09-11T03:46:54.172 INFO:tasks.rados.rados.2.smithi175.stdout:2240: left oid 17 (ObjNum 760 snap 247 seq_num 760) 2024-09-11T03:46:54.172 INFO:tasks.rados.rados.2.smithi175.stdout:2241: finishing write tid 1 to smithi17557870-8 2024-09-11T03:46:54.172 INFO:tasks.rados.rados.2.smithi175.stdout:2242: seq_num 762 ranges {406934=575038,1640573=474593,2642791=164143} 2024-09-11T03:46:54.177 INFO:tasks.rados.rados.2.smithi175.stdout:2242: writing smithi17557870-47 from 406934 to 981972 tid 1 2024-09-11T03:46:54.183 INFO:tasks.rados.rados.2.smithi175.stdout:2242: writing smithi17557870-47 from 1640573 to 2115166 tid 2 2024-09-11T03:46:54.188 INFO:tasks.rados.rados.2.smithi175.stdout:2242: writing smithi17557870-47 from 2642791 to 2806934 tid 3 2024-09-11T03:46:54.188 INFO:tasks.rados.rados.2.smithi175.stdout:2240: done (2 left) 2024-09-11T03:46:54.188 INFO:tasks.rados.rados.2.smithi175.stdout:2243: copy_from oid 33 from oid 27 current snap is 247 2024-09-11T03:46:54.189 INFO:tasks.rados.rados.2.smithi175.stdout:2241: finishing write tid 2 to smithi17557870-8 2024-09-11T03:46:54.189 INFO:tasks.rados.rados.2.smithi175.stdout:2241: finishing write tid 3 to smithi17557870-8 2024-09-11T03:46:54.189 INFO:tasks.rados.rados.2.smithi175.stdout:2241: finishing write tid 4 to smithi17557870-8 2024-09-11T03:46:54.189 INFO:tasks.rados.rados.2.smithi175.stdout:2241: oid 8 updating version 0 to 868 2024-09-11T03:46:54.189 INFO:tasks.rados.rados.2.smithi175.stdout:2241: oid 8 version 868 is already newer than 866 2024-09-11T03:46:54.190 INFO:tasks.rados.rados.2.smithi175.stdout:2241: oid 8 version 868 is already newer than 867 2024-09-11T03:46:54.190 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 868 (ObjNum 761 snap 247 seq_num 761) dirty exists 2024-09-11T03:46:54.190 INFO:tasks.rados.rados.2.smithi175.stdout:2241: left oid 8 (ObjNum 761 snap 247 seq_num 761) 2024-09-11T03:46:54.190 INFO:tasks.rados.rados.2.smithi175.stdout:2242: finishing write tid 1 to smithi17557870-47 2024-09-11T03:46:54.190 INFO:tasks.rados.rados.2.smithi175.stdout:2242: finishing write tid 2 to smithi17557870-47 2024-09-11T03:46:54.190 INFO:tasks.rados.rados.2.smithi175.stdout:2241: done (2 left) 2024-09-11T03:46:54.190 INFO:tasks.rados.rados.2.smithi175.stdout:2244: delete oid 7 current snap is 247 2024-09-11T03:46:54.192 INFO:tasks.rados.rados.2.smithi175.stdout:2242: finishing write tid 3 to smithi17557870-47 2024-09-11T03:46:54.192 INFO:tasks.rados.rados.2.smithi175.stdout:2242: finishing write tid 4 to smithi17557870-47 2024-09-11T03:46:54.192 INFO:tasks.rados.rados.2.smithi175.stdout:2242: finishing write tid 5 to smithi17557870-47 2024-09-11T03:46:54.192 INFO:tasks.rados.rados.2.smithi175.stdout:2242: oid 47 updating version 0 to 769 2024-09-11T03:46:54.192 INFO:tasks.rados.rados.2.smithi175.stdout:2242: oid 47 version 769 is already newer than 766 2024-09-11T03:46:54.193 INFO:tasks.rados.rados.2.smithi175.stdout:2242: oid 47 version 769 is already newer than 767 2024-09-11T03:46:54.193 INFO:tasks.rados.rados.2.smithi175.stdout:2242: oid 47 version 769 is already newer than 768 2024-09-11T03:46:54.193 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 769 (ObjNum 762 snap 247 seq_num 762) dirty exists 2024-09-11T03:46:54.193 INFO:tasks.rados.rados.2.smithi175.stdout:2242: left oid 47 (ObjNum 762 snap 247 seq_num 762) 2024-09-11T03:46:54.193 INFO:tasks.rados.rados.2.smithi175.stdout:2242: done (2 left) 2024-09-11T03:46:54.193 INFO:tasks.rados.rados.2.smithi175.stdout:2244: done (1 left) 2024-09-11T03:46:54.194 INFO:tasks.rados.rados.2.smithi175.stdout:2245: snap_remove snap 246 2024-09-11T03:46:54.194 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:46:54.194 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:46:54.235 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:54.257 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:54.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:54.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:54.359 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:54.383 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:54.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:54.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:54.393 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:54 smithi084 ceph-mon[27254]: pgmap v1047: 67 pgs: 67 active+clean; 101 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 614 KiB/s wr, 3 op/s 2024-09-11T03:46:54.393 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:54 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:46:54.393 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:54 smithi084 ceph-mon[27254]: 2.7 scrub ok 2024-09-11T03:46:54.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:54 smithi121 ceph-mon[30729]: pgmap v1047: 67 pgs: 67 active+clean; 101 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 614 KiB/s wr, 3 op/s 2024-09-11T03:46:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:54 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:46:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:54 smithi121 ceph-mon[30729]: 2.7 scrub ok 2024-09-11T03:46:54.476 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:54 smithi175 ceph-mon[30894]: pgmap v1047: 67 pgs: 67 active+clean; 101 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 614 KiB/s wr, 3 op/s 2024-09-11T03:46:54.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:54 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:46:54.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:54 smithi175 ceph-mon[30894]: 2.7 scrub ok 2024-09-11T03:46:54.485 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:54.506 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:54.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:54.507 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:54.547 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:54.607 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:54.628 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:54.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:54.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:54.731 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:54.752 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:54.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:54.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:54.854 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:54.875 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:54.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:54.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:54.887 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-11T03:46:54.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:55.003 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:55.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:55.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:55.017 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:46:55.017 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:46:55.034 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:46:55.034 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:46:55.049 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:46:55.050 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:46:55.065 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:46:55.066 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:46:55.084 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:46:55.084 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:46:55.101 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:46:55.101 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:46:55.105 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:46:55.118 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:46:55.118 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:46:55.124 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:46:55.126 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:55.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:55.127 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:46:55.136 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:46:55.136 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:46:55.147 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:46:55.147 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 1 (10000) 2024-09-11T03:46:55.154 INFO:tasks.rados.rados.2.smithi175.stdout:2245: done (1 left) 2024-09-11T03:46:55.154 INFO:tasks.rados.rados.2.smithi175.stdout:2246: delete oid 39 current snap is 247 2024-09-11T03:46:55.154 INFO:tasks.rados.rados.2.smithi175.stdout:2243: finishing copy_from racing read to smithi17557870-33 2024-09-11T03:46:55.154 INFO:tasks.rados.rados.2.smithi175.stdout:2243: finishing copy_from to smithi17557870-33 2024-09-11T03:46:55.154 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 907 (ObjNum 750 snap 246 seq_num 750) dirty exists 2024-09-11T03:46:55.157 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:46:55.157 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:46:55.158 INFO:tasks.rados.rados.2.smithi175.stdout:2243: done (1 left) 2024-09-11T03:46:55.159 INFO:tasks.rados.rados.2.smithi175.stdout:2246: done (0 left) 2024-09-11T03:46:55.159 INFO:tasks.rados.rados.2.smithi175.stdout:2247: write oid 2 current snap is 247 2024-09-11T03:46:55.159 INFO:tasks.rados.rados.2.smithi175.stdout:2247: seq_num 765 ranges {457355=400785,1416019=621920,2725060=132295} 2024-09-11T03:46:55.163 INFO:tasks.rados.rados.2.smithi175.stdout:2247: writing smithi17557870-2 from 457355 to 858140 tid 1 2024-09-11T03:46:55.170 INFO:tasks.rados.rados.2.smithi175.stdout:2247: writing smithi17557870-2 from 1416019 to 2037939 tid 2 2024-09-11T03:46:55.217 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:46:55.218 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:46:55.218 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:55 smithi084 ceph-mon[27254]: osdmap e596: 12 total, 12 up, 12 in 2024-09-11T03:46:55.218 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:55 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3261538401' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:55.218 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:55 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3615707684' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:46:55.218 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:55 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1656225490' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:55.219 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:46:55.219 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:46:55.220 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:46:55.220 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:46:55.222 INFO:tasks.rados.rados.2.smithi175.stdout:2247: writing smithi17557870-2 from 2725060 to 2857355 tid 3 2024-09-11T03:46:55.222 INFO:tasks.rados.rados.2.smithi175.stdout:2248: delete oid 30 current snap is 247 2024-09-11T03:46:55.222 INFO:tasks.rados.rados.2.smithi175.stdout:2247: finishing write tid 1 to smithi17557870-2 2024-09-11T03:46:55.223 INFO:tasks.rados.rados.2.smithi175.stdout:2247: finishing write tid 2 to smithi17557870-2 2024-09-11T03:46:55.223 INFO:tasks.rados.rados.2.smithi175.stdout:2248: done (1 left) 2024-09-11T03:46:55.223 INFO:tasks.rados.rados.2.smithi175.stdout:2249: delete oid 49 current snap is 247 2024-09-11T03:46:55.223 INFO:tasks.rados.rados.2.smithi175.stdout:2247: finishing write tid 3 to smithi17557870-2 2024-09-11T03:46:55.223 INFO:tasks.rados.rados.2.smithi175.stdout:2247: finishing write tid 4 to smithi17557870-2 2024-09-11T03:46:55.223 INFO:tasks.rados.rados.2.smithi175.stdout:2247: finishing write tid 5 to smithi17557870-2 2024-09-11T03:46:55.223 INFO:tasks.rados.rados.2.smithi175.stdout:2247: oid 2 updating version 0 to 928 2024-09-11T03:46:55.223 INFO:tasks.rados.rados.2.smithi175.stdout:2247: oid 2 version 928 is already newer than 927 2024-09-11T03:46:55.223 INFO:tasks.rados.rados.2.smithi175.stdout:2247: oid 2 version 928 is already newer than 926 2024-09-11T03:46:55.223 INFO:tasks.rados.rados.2.smithi175.stdout:2247: oid 2 version 928 is already newer than 925 2024-09-11T03:46:55.223 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 928 (ObjNum 765 snap 247 seq_num 765) dirty exists 2024-09-11T03:46:55.224 INFO:tasks.rados.rados.2.smithi175.stdout:2247: left oid 2 (ObjNum 765 snap 247 seq_num 765) 2024-09-11T03:46:55.224 INFO:tasks.rados.rados.2.smithi175.stdout:2247: done (1 left) 2024-09-11T03:46:55.224 INFO:tasks.rados.rados.2.smithi175.stdout:2249: done (0 left) 2024-09-11T03:46:55.224 INFO:tasks.rados.rados.2.smithi175.stdout:2250: write oid 14 current snap is 247 2024-09-11T03:46:55.224 INFO:tasks.rados.rados.2.smithi175.stdout:2250: seq_num 768 ranges {507776=742885,1991465=485600} 2024-09-11T03:46:55.224 INFO:tasks.rados.rados.2.smithi175.stdout:2250: writing smithi17557870-14 from 507776 to 1250661 tid 1 2024-09-11T03:46:55.224 INFO:tasks.rados.rados.2.smithi175.stdout:2250: writing smithi17557870-14 from 1991465 to 2477065 tid 2 2024-09-11T03:46:55.224 INFO:tasks.rados.rados.2.smithi175.stdout:2251: copy_from oid 1 from oid 8 current snap is 247 2024-09-11T03:46:55.224 INFO:tasks.rados.rados.2.smithi175.stdout:2250: finishing write tid 1 to smithi17557870-14 2024-09-11T03:46:55.224 INFO:tasks.rados.rados.2.smithi175.stdout:2252: rollback oid 10 current snap is 247 2024-09-11T03:46:55.224 INFO:tasks.rados.rados.2.smithi175.stdout:2252: done (2 left) 2024-09-11T03:46:55.224 INFO:tasks.rados.rados.2.smithi175.stdout:2253: delete oid 13 current snap is 247 2024-09-11T03:46:55.224 INFO:tasks.rados.rados.2.smithi175.stdout:2253: done (2 left) 2024-09-11T03:46:55.225 INFO:tasks.rados.rados.2.smithi175.stdout:2254: read oid 2 snap -1 2024-09-11T03:46:55.225 INFO:tasks.rados.rados.2.smithi175.stdout:2254: expect (ObjNum 765 snap 247 seq_num 765) 2024-09-11T03:46:55.225 INFO:tasks.rados.rados.2.smithi175.stdout:2255: delete oid 48 current snap is 247 2024-09-11T03:46:55.225 INFO:tasks.rados.rados.2.smithi175.stdout:2250: finishing write tid 2 to smithi17557870-14 2024-09-11T03:46:55.225 INFO:tasks.rados.rados.2.smithi175.stdout:2250: finishing write tid 3 to smithi17557870-14 2024-09-11T03:46:55.225 INFO:tasks.rados.rados.2.smithi175.stdout:2250: finishing write tid 4 to smithi17557870-14 2024-09-11T03:46:55.225 INFO:tasks.rados.rados.2.smithi175.stdout:2250: oid 14 updating version 0 to 865 2024-09-11T03:46:55.225 INFO:tasks.rados.rados.2.smithi175.stdout:2250: oid 14 version 865 is already newer than 864 2024-09-11T03:46:55.225 INFO:tasks.rados.rados.2.smithi175.stdout:2250: oid 14 version 865 is already newer than 863 2024-09-11T03:46:55.225 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 865 (ObjNum 768 snap 247 seq_num 768) dirty exists 2024-09-11T03:46:55.226 INFO:tasks.rados.rados.2.smithi175.stdout:2250: left oid 14 (ObjNum 768 snap 247 seq_num 768) 2024-09-11T03:46:55.226 INFO:tasks.rados.rados.2.smithi175.stdout:2250: done (3 left) 2024-09-11T03:46:55.226 INFO:tasks.rados.rados.2.smithi175.stdout:2255: done (2 left) 2024-09-11T03:46:55.226 INFO:tasks.rados.rados.2.smithi175.stdout:2256: rollback oid 7 current snap is 247 2024-09-11T03:46:55.226 INFO:tasks.rados.rados.2.smithi175.stdout:2256: done (2 left) 2024-09-11T03:46:55.226 INFO:tasks.rados.rados.2.smithi175.stdout:2257: snap_create 2024-09-11T03:46:55.226 INFO:tasks.rados.rados.2.smithi175.stdout:2251: finishing copy_from racing read to smithi17557870-1 2024-09-11T03:46:55.226 INFO:tasks.rados.rados.2.smithi175.stdout:2251: finishing copy_from to smithi17557870-1 2024-09-11T03:46:55.226 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 868 (ObjNum 761 snap 247 seq_num 761) dirty exists 2024-09-11T03:46:55.226 INFO:tasks.rados.rados.2.smithi175.stdout:2254: expect (ObjNum 765 snap 247 seq_num 765) 2024-09-11T03:46:55.228 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:55.251 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:55.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:55.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:55.353 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:46:55.374 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:55.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:55.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:46:55.399 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:55 smithi175 ceph-mon[30894]: osdmap e596: 12 total, 12 up, 12 in 2024-09-11T03:46:55.399 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:55 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3261538401' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:55.399 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:55 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3615707684' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:46:55.399 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:55 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1656225490' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:55 smithi121 ceph-mon[30729]: osdmap e596: 12 total, 12 up, 12 in 2024-09-11T03:46:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:55 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3261538401' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:46:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:55 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3615707684' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:46:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:55 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1656225490' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:55.476 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:55.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:55.499 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:55.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:55.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:55.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:55.626 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:55.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:55.634 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:46:55.634 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:46:55.634 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:46:55.634 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:46:55.634 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.634 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:640 2.a 2:5014e305:::smithi17557870-46:head [copy-from ver 23] snapc 11=[e] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:46:55.634 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:46:55.634 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 467.52287171299997, 2024-09-11T03:46:55.634 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019221736, 2024-09-11T03:46:55.634 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.634 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.634 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:55.635 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.635 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.635 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.635 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 640 2024-09-11T03:46:55.635 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.635 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.635 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.635 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.635 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:46:55.635 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.635 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.635 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.635 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.635 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:46:55.636 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.636 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.636 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.636 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.636 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110295+0000", 2024-09-11T03:46:55.636 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0609999999999998e-06 2024-09-11T03:46:55.636 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.636 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.636 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.636 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110305+0000", 2024-09-11T03:46:55.636 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4169999999999998e-06 2024-09-11T03:46:55.636 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.636 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.637 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.637 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110321+0000", 2024-09-11T03:46:55.637 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6288000000000001e-05 2024-09-11T03:46:55.637 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.637 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.637 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.637 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110328+0000", 2024-09-11T03:46:55.637 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3950000000000003e-06 2024-09-11T03:46:55.637 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.637 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.637 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.637 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110396+0000", 2024-09-11T03:46:55.637 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7164999999999998e-05 2024-09-11T03:46:55.638 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.638 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.638 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:55.638 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110507+0000", 2024-09-11T03:46:55.638 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000111217 2024-09-11T03:46:55.638 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.638 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.638 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:46:55.638 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.118213+0000", 2024-09-11T03:46:55.638 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0077066249999999999 2024-09-11T03:46:55.638 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.638 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.639 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:55.639 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125103+0000", 2024-09-11T03:46:55.639 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068894359999999997 2024-09-11T03:46:55.639 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.639 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.639 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:55.639 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125459+0000", 2024-09-11T03:46:55.639 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000356146 2024-09-11T03:46:55.639 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.639 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.639 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:55.639 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129369+0000", 2024-09-11T03:46:55.639 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039099249999999999 2024-09-11T03:46:55.639 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.639 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.640 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:55.640 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129428+0000", 2024-09-11T03:46:55.640 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9113000000000001e-05 2024-09-11T03:46:55.640 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.640 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.640 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.640 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129515+0000", 2024-09-11T03:46:55.640 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6947999999999995e-05 2024-09-11T03:46:55.640 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.640 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.640 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.640 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.642 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.642 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:641 2.a 2:5014e305:::smithi17557870-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:46:55.642 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:46:55.642 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 467.52282032199997, 2024-09-11T03:46:55.642 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011882876000000001, 2024-09-11T03:46:55.642 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.642 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.642 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:46:55.642 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.642 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.642 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.642 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 641 2024-09-11T03:46:55.643 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.643 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.643 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.643 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.643 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:46:55.643 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.643 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.643 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.643 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.643 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:46:55.643 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.643 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.644 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.644 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.644 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110345+0000", 2024-09-11T03:46:55.644 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7599999999999998e-07 2024-09-11T03:46:55.644 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.644 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.644 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.644 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110347+0000", 2024-09-11T03:46:55.644 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.685e-06 2024-09-11T03:46:55.644 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.644 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.644 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.644 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110351+0000", 2024-09-11T03:46:55.644 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4889999999999997e-06 2024-09-11T03:46:55.645 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.645 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.645 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.645 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110354+0000", 2024-09-11T03:46:55.645 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0510000000000001e-06 2024-09-11T03:46:55.645 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.645 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.645 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.645 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110644+0000", 2024-09-11T03:46:55.645 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00028940200000000002 2024-09-11T03:46:55.645 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.645 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.645 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:46:55.646 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110673+0000", 2024-09-11T03:46:55.646 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9833e-05 2024-09-11T03:46:55.646 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.646 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.646 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.646 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122038+0000", 2024-09-11T03:46:55.646 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011364915 2024-09-11T03:46:55.646 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.647 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.647 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:55.647 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122127+0000", 2024-09-11T03:46:55.647 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8987999999999996e-05 2024-09-11T03:46:55.647 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.647 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.647 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.647 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.122227+0000", 2024-09-11T03:46:55.647 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010013699999999999 2024-09-11T03:46:55.647 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.648 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.648 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.648 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.648 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.648 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:752 2.c 2:3c007163:::smithi17557870-35:head [copy-from ver 39] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:46:55.648 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:46:55.648 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 460.62414904000002, 2024-09-11T03:46:55.648 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017596783000000001, 2024-09-11T03:46:55.648 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.648 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.648 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:55.648 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.648 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.648 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.649 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 752 2024-09-11T03:46:55.649 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.649 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.649 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.649 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.649 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:46:55.649 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.649 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.649 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.649 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.649 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:46:55.649 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.649 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.650 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.650 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.650 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009018+0000", 2024-09-11T03:46:55.650 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.48e-06 2024-09-11T03:46:55.650 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.650 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.650 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.651 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009029+0000", 2024-09-11T03:46:55.651 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0377000000000001e-05 2024-09-11T03:46:55.651 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.651 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.651 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.651 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009050+0000", 2024-09-11T03:46:55.651 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0849e-05 2024-09-11T03:46:55.652 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.652 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.652 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.652 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009058+0000", 2024-09-11T03:46:55.652 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8019999999999998e-06 2024-09-11T03:46:55.652 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.652 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.652 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.652 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009125+0000", 2024-09-11T03:46:55.652 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6388000000000002e-05 2024-09-11T03:46:55.652 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.652 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.652 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:46:55.653 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009130+0000", 2024-09-11T03:46:55.653 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0900000000000004e-06 2024-09-11T03:46:55.653 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.653 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.653 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.653 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010518+0000", 2024-09-11T03:46:55.653 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013880030000000001 2024-09-11T03:46:55.653 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.653 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.653 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:55.653 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010582+0000", 2024-09-11T03:46:55.653 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.368e-05 2024-09-11T03:46:55.653 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.654 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.654 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:46:55.654 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.015350+0000", 2024-09-11T03:46:55.654 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0047682870000000004 2024-09-11T03:46:55.654 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.654 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.654 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:55.654 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.019005+0000", 2024-09-11T03:46:55.654 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0036549040000000001 2024-09-11T03:46:55.654 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.654 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.654 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:55.654 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.024096+0000", 2024-09-11T03:46:55.654 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050917439999999996 2024-09-11T03:46:55.655 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.655 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.655 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:55.655 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026521+0000", 2024-09-11T03:46:55.655 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024247399999999999 2024-09-11T03:46:55.655 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.655 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.655 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:55.655 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026559+0000", 2024-09-11T03:46:55.655 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8232999999999998e-05 2024-09-11T03:46:55.655 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.655 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.655 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.655 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026613+0000", 2024-09-11T03:46:55.656 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3205999999999998e-05 2024-09-11T03:46:55.656 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.656 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.656 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.656 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.656 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.656 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1175 2.c 2:32ade24e:::smithi17557870-45:head [copy-from ver 130] snapc 2e=[2e,2c,2b,28,26] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:46:55.656 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:46:55.656 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 413.53102526399999, 2024-09-11T03:46:55.656 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015807878000000001, 2024-09-11T03:46:55.656 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.656 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.656 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:55.657 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.657 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.657 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.657 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1175 2024-09-11T03:46:55.657 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.657 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.657 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.657 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.657 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:46:55.657 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.657 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.657 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.657 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.657 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:46:55.658 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.658 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.658 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.658 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.658 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102142+0000", 2024-09-11T03:46:55.658 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3369999999999998e-06 2024-09-11T03:46:55.658 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.658 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.658 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.658 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102152+0000", 2024-09-11T03:46:55.658 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8400000000000007e-06 2024-09-11T03:46:55.658 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.658 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.658 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102172+0000", 2024-09-11T03:46:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0537e-05 2024-09-11T03:46:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102183+0000", 2024-09-11T03:46:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0423e-05 2024-09-11T03:46:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102253+0000", 2024-09-11T03:46:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0528e-05 2024-09-11T03:46:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:46:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102259+0000", 2024-09-11T03:46:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1589999999999998e-06 2024-09-11T03:46:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103477+0000", 2024-09-11T03:46:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012174550000000001 2024-09-11T03:46:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103545+0000", 2024-09-11T03:46:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7595999999999995e-05 2024-09-11T03:46:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:46:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.110323+0000", 2024-09-11T03:46:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0067785730000000004 2024-09-11T03:46:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.113411+0000", 2024-09-11T03:46:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030882769999999999 2024-09-11T03:46:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117742+0000", 2024-09-11T03:46:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0043301920000000001 2024-09-11T03:46:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117835+0000", 2024-09-11T03:46:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3848e-05 2024-09-11T03:46:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117916+0000", 2024-09-11T03:46:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0505000000000003e-05 2024-09-11T03:46:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117948+0000", 2024-09-11T03:46:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1608e-05 2024-09-11T03:46:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1176 2.c 2:32ade24e:::smithi17557870-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:46:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:46:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 413.53096099999999, 2024-09-11T03:46:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010847406, 2024-09-11T03:46:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:46:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1176 2024-09-11T03:46:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.665 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:46:55.665 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.665 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.665 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.665 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.665 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:46:55.665 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.665 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.665 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.665 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.665 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102204+0000", 2024-09-11T03:46:55.665 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4999999999999998e-07 2024-09-11T03:46:55.665 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.666 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.666 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.666 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102206+0000", 2024-09-11T03:46:55.666 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9429999999999999e-06 2024-09-11T03:46:55.666 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.666 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.666 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.666 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102214+0000", 2024-09-11T03:46:55.666 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0399999999999993e-06 2024-09-11T03:46:55.666 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.666 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.666 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.666 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102220+0000", 2024-09-11T03:46:55.666 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2989999999999996e-06 2024-09-11T03:46:55.667 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.667 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.667 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.667 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102307+0000", 2024-09-11T03:46:55.667 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7635999999999999e-05 2024-09-11T03:46:55.667 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.667 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.667 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:46:55.667 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102310+0000", 2024-09-11T03:46:55.667 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.2160000000000002e-06 2024-09-11T03:46:55.667 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.667 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.667 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.667 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103661+0000", 2024-09-11T03:46:55.667 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013507860000000001 2024-09-11T03:46:55.668 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.668 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.668 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:46:55.668 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103686+0000", 2024-09-11T03:46:55.668 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.44e-05 2024-09-11T03:46:55.668 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.668 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.668 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.668 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.112884+0000", 2024-09-11T03:46:55.668 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0091986110000000006 2024-09-11T03:46:55.668 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.668 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.668 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:55.669 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.112967+0000", 2024-09-11T03:46:55.669 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2478000000000006e-05 2024-09-11T03:46:55.669 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.669 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.669 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.669 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.113051+0000", 2024-09-11T03:46:55.669 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4647000000000005e-05 2024-09-11T03:46:55.669 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.669 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.669 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.669 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.669 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.669 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2140 2.c 2:3c007163:::smithi17557870-35:head [copy-from ver 101] snapc 57=[57,56,51,4e] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-11T03:46:55.669 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:25.064188+0000", 2024-09-11T03:46:55.670 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 330.56897684299997, 2024-09-11T03:46:55.670 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013777315, 2024-09-11T03:46:55.670 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.670 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.670 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:55.670 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.670 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.670 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.670 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2140 2024-09-11T03:46:55.670 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.670 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.670 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.670 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.670 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064188+0000", 2024-09-11T03:46:55.670 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.671 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.671 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.671 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.671 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064188+0000", 2024-09-11T03:46:55.671 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.671 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.671 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.671 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.671 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064190+0000", 2024-09-11T03:46:55.671 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9700000000000002e-06 2024-09-11T03:46:55.671 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.671 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.671 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.672 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064198+0000", 2024-09-11T03:46:55.672 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3610000000000001e-06 2024-09-11T03:46:55.672 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.672 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.672 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.672 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064216+0000", 2024-09-11T03:46:55.672 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7768999999999999e-05 2024-09-11T03:46:55.672 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.672 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.672 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.672 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064225+0000", 2024-09-11T03:46:55.672 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7099999999999996e-06 2024-09-11T03:46:55.672 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.672 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.673 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.673 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064291+0000", 2024-09-11T03:46:55.673 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5634000000000002e-05 2024-09-11T03:46:55.673 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.673 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.673 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:55.673 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.064367+0000", 2024-09-11T03:46:55.673 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6687999999999995e-05 2024-09-11T03:46:55.673 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.673 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.673 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:46:55.673 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.066720+0000", 2024-09-11T03:46:55.673 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023526010000000002 2024-09-11T03:46:55.673 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.674 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.674 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:55.674 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.069243+0000", 2024-09-11T03:46:55.674 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025227320000000002 2024-09-11T03:46:55.674 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.674 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.674 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:55.674 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.073940+0000", 2024-09-11T03:46:55.674 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0046973930000000002 2024-09-11T03:46:55.674 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.674 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.674 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:55.674 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.077850+0000", 2024-09-11T03:46:55.674 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039098279999999997 2024-09-11T03:46:55.675 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.675 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.675 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:55.675 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.077915+0000", 2024-09-11T03:46:55.675 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5160000000000006e-05 2024-09-11T03:46:55.675 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.675 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.675 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.675 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:25.077965+0000", 2024-09-11T03:46:55.675 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0469000000000001e-05 2024-09-11T03:46:55.675 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.675 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.675 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.675 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.676 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.676 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:2421 2.18 e294/281 2:1bdf1a36:::smithi17557870-36:head v 294'198, mlcod=0'0)", 2024-09-11T03:46:55.676 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:50.964269+0000", 2024-09-11T03:46:55.676 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 304.66889594399998, 2024-09-11T03:46:55.676 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013043020000000001, 2024-09-11T03:46:55.676 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.676 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.676 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:55.676 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.676 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.676 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.676 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964269+0000", 2024-09-11T03:46:55.676 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.676 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.677 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.677 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.677 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964269+0000", 2024-09-11T03:46:55.677 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.677 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.677 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.677 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.677 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964270+0000", 2024-09-11T03:46:55.677 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3510000000000001e-06 2024-09-11T03:46:55.677 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.677 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.677 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.677 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966680+0000", 2024-09-11T03:46:55.677 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002409698 2024-09-11T03:46:55.678 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.678 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.678 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.678 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966720+0000", 2024-09-11T03:46:55.678 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0265999999999997e-05 2024-09-11T03:46:55.678 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.678 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.678 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.678 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966738+0000", 2024-09-11T03:46:55.678 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7540000000000001e-05 2024-09-11T03:46:55.678 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.678 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.678 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.678 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966835+0000", 2024-09-11T03:46:55.678 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7059999999999996e-05 2024-09-11T03:46:55.679 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.679 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.679 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:55.679 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.966862+0000", 2024-09-11T03:46:55.679 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7107e-05 2024-09-11T03:46:55.679 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.679 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.679 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:55.679 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977265+0000", 2024-09-11T03:46:55.679 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.01040258 2024-09-11T03:46:55.679 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.679 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.679 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.680 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977312+0000", 2024-09-11T03:46:55.680 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7417999999999999e-05 2024-09-11T03:46:55.680 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.680 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.680 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.680 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.680 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.680 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2972 2.c 2:32ade24e:::smithi17557870-45:head [copy-from ver 171] snapc 80=[80,7b,7a,79,77,75] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-11T03:46:55.680 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:46:55.680 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 250.559070969, 2024-09-11T03:46:55.680 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021036193000000002, 2024-09-11T03:46:55.680 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.680 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.681 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:46:55.681 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.681 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.681 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.681 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2972 2024-09-11T03:46:55.681 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.681 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.681 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.681 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.681 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:46:55.681 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.682 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.682 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.682 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.682 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:46:55.682 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.682 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.682 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.682 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.682 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074096+0000", 2024-09-11T03:46:55.682 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.685e-06 2024-09-11T03:46:55.682 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.683 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.683 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.683 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074104+0000", 2024-09-11T03:46:55.683 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9950000000000005e-06 2024-09-11T03:46:55.683 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.683 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.683 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.683 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074123+0000", 2024-09-11T03:46:55.683 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9587999999999999e-05 2024-09-11T03:46:55.683 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.683 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.683 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.684 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074132+0000", 2024-09-11T03:46:55.684 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1579999999999994e-06 2024-09-11T03:46:55.684 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.684 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.684 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.684 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074204+0000", 2024-09-11T03:46:55.684 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2072999999999999e-05 2024-09-11T03:46:55.684 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.684 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.685 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:55.685 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074302+0000", 2024-09-11T03:46:55.685 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7706000000000003e-05 2024-09-11T03:46:55.685 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.685 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.685 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:46:55.685 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.087397+0000", 2024-09-11T03:46:55.685 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013095292999999999 2024-09-11T03:46:55.685 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.685 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.685 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:46:55.685 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090766+0000", 2024-09-11T03:46:55.685 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0033682109999999999 2024-09-11T03:46:55.686 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.686 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.686 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:55.686 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.094977+0000", 2024-09-11T03:46:55.686 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0042108759999999997 2024-09-11T03:46:55.686 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.686 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.686 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:46:55.686 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095061+0000", 2024-09-11T03:46:55.686 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4827999999999995e-05 2024-09-11T03:46:55.686 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.686 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.686 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:46:55.686 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095099+0000", 2024-09-11T03:46:55.687 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7546999999999998e-05 2024-09-11T03:46:55.687 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.687 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.687 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.687 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095130+0000", 2024-09-11T03:46:55.687 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1232999999999997e-05 2024-09-11T03:46:55.687 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.687 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.687 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.687 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.687 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.687 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2973 2.c 2:32ade24e:::smithi17557870-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e348)", 2024-09-11T03:46:55.687 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:46:55.687 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 250.55900585099999, 2024-09-11T03:46:55.688 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016062098, 2024-09-11T03:46:55.688 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.688 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.688 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:46:55.688 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.688 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.688 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.688 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2973 2024-09-11T03:46:55.688 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.688 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.688 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.688 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.688 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:46:55.689 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.689 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.689 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.689 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.689 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:46:55.689 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.689 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.689 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.689 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.689 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074160+0000", 2024-09-11T03:46:55.689 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.46e-07 2024-09-11T03:46:55.689 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.689 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.689 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.690 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074162+0000", 2024-09-11T03:46:55.690 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5220000000000002e-06 2024-09-11T03:46:55.690 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.690 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.690 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.690 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074178+0000", 2024-09-11T03:46:55.690 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.609e-05 2024-09-11T03:46:55.690 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.690 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.690 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.690 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074183+0000", 2024-09-11T03:46:55.690 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8280000000000002e-06 2024-09-11T03:46:55.690 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.690 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.691 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.691 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074428+0000", 2024-09-11T03:46:55.691 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00024529599999999999 2024-09-11T03:46:55.691 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.691 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.691 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:46:55.691 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074469+0000", 2024-09-11T03:46:55.691 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0463000000000002e-05 2024-09-11T03:46:55.691 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.691 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.691 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.691 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090093+0000", 2024-09-11T03:46:55.691 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015624449 2024-09-11T03:46:55.691 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.692 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.692 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:55.692 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090155+0000", 2024-09-11T03:46:55.692 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1618999999999994e-05 2024-09-11T03:46:55.692 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.692 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.692 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.692 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090221+0000", 2024-09-11T03:46:55.692 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6384999999999995e-05 2024-09-11T03:46:55.692 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.692 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.692 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.692 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.692 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.693 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4366 2.c 2:327469ff:::smithi17557870-49:bc [sparse-read 0~1947401] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e473)", 2024-09-11T03:46:55.693 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:50.943008+0000", 2024-09-11T03:46:55.693 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 124.69015674800001, 2024-09-11T03:46:55.693 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010277684, 2024-09-11T03:46:55.693 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.693 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.693 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:46:55.693 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.693 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.693 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.693 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4366 2024-09-11T03:46:55.693 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.693 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.693 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.694 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.694 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943008+0000", 2024-09-11T03:46:55.694 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.694 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.694 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.694 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.694 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943008+0000", 2024-09-11T03:46:55.694 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.694 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.694 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.694 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.694 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943009+0000", 2024-09-11T03:46:55.694 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5000000000000001e-07 2024-09-11T03:46:55.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.695 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.695 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.695 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943014+0000", 2024-09-11T03:46:55.695 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8720000000000001e-06 2024-09-11T03:46:55.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.695 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.695 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.695 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943024+0000", 2024-09-11T03:46:55.695 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6250000000000002e-06 2024-09-11T03:46:55.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.695 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.695 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.696 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943030+0000", 2024-09-11T03:46:55.696 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.224e-06 2024-09-11T03:46:55.696 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.696 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.696 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.696 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943556+0000", 2024-09-11T03:46:55.696 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00052628699999999996 2024-09-11T03:46:55.696 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.696 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.696 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:46:55.696 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943635+0000", 2024-09-11T03:46:55.696 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9152000000000004e-05 2024-09-11T03:46:55.696 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.696 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.697 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.697 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.952995+0000", 2024-09-11T03:46:55.697 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.009360007 2024-09-11T03:46:55.697 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.697 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.697 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:55.697 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.953133+0000", 2024-09-11T03:46:55.697 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013800999999999999 2024-09-11T03:46:55.697 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.697 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.697 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.697 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.953286+0000", 2024-09-11T03:46:55.697 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000152657 2024-09-11T03:46:55.697 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.698 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.698 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.698 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.698 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.698 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4367 2.c 2:327469ff:::smithi17557870-49:bc [sparse-read 0~1947401] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e473)", 2024-09-11T03:46:55.698 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:50.943104+0000", 2024-09-11T03:46:55.698 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 124.69006081000001, 2024-09-11T03:46:55.698 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010538881999999999, 2024-09-11T03:46:55.698 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.698 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.698 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:46:55.698 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.698 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.699 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.699 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4367 2024-09-11T03:46:55.699 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.699 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.699 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.699 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.699 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943104+0000", 2024-09-11T03:46:55.699 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.699 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.699 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.700 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.700 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943104+0000", 2024-09-11T03:46:55.700 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.700 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.700 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.700 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.700 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943105+0000", 2024-09-11T03:46:55.700 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1900000000000002e-07 2024-09-11T03:46:55.700 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.700 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.700 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.700 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943109+0000", 2024-09-11T03:46:55.700 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8720000000000004e-06 2024-09-11T03:46:55.701 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.701 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.701 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.701 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943118+0000", 2024-09-11T03:46:55.701 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5999999999999996e-06 2024-09-11T03:46:55.701 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.701 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.701 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.701 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943126+0000", 2024-09-11T03:46:55.701 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2980000000000003e-06 2024-09-11T03:46:55.701 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.701 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.701 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.701 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943674+0000", 2024-09-11T03:46:55.702 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00054807100000000002 2024-09-11T03:46:55.702 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.702 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.702 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:46:55.702 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.943762+0000", 2024-09-11T03:46:55.702 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8702000000000006e-05 2024-09-11T03:46:55.702 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.702 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.702 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.702 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.953343+0000", 2024-09-11T03:46:55.702 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0095801959999999992 2024-09-11T03:46:55.702 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.702 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.702 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:55.703 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.953457+0000", 2024-09-11T03:46:55.703 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114234 2024-09-11T03:46:55.703 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.703 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.703 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.703 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:50.953643+0000", 2024-09-11T03:46:55.703 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00018619 2024-09-11T03:46:55.703 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.703 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.703 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.703 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.703 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.703 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5348 2.1b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e571)", 2024-09-11T03:46:55.703 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:28.996464+0000", 2024-09-11T03:46:55.704 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 26.636700435000002, 2024-09-11T03:46:55.704 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.973168225, 2024-09-11T03:46:55.704 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.704 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.704 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "reached pg", 2024-09-11T03:46:55.704 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.704 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.704 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.704 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5348 2024-09-11T03:46:55.704 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.704 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.704 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.704 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.704 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996464+0000", 2024-09-11T03:46:55.705 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.705 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.705 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.705 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.705 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996464+0000", 2024-09-11T03:46:55.705 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.705 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.705 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.705 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.705 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996465+0000", 2024-09-11T03:46:55.705 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3500000000000002e-07 2024-09-11T03:46:55.705 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.705 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.705 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.706 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996467+0000", 2024-09-11T03:46:55.706 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.576e-06 2024-09-11T03:46:55.706 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.706 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.706 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.706 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996475+0000", 2024-09-11T03:46:55.706 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1350000000000003e-06 2024-09-11T03:46:55.706 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.706 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.706 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.706 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996478+0000", 2024-09-11T03:46:55.706 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5240000000000001e-06 2024-09-11T03:46:55.706 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.707 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.707 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.707 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996545+0000", 2024-09-11T03:46:55.707 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7379000000000007e-05 2024-09-11T03:46:55.707 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.707 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.707 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:46:55.707 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996557+0000", 2024-09-11T03:46:55.707 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2019e-05 2024-09-11T03:46:55.707 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.707 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.707 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.707 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969589+0000", 2024-09-11T03:46:55.707 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97303155699999999 2024-09-11T03:46:55.708 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.708 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.708 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.708 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969633+0000", 2024-09-11T03:46:55.708 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3699999999999998e-05 2024-09-11T03:46:55.708 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.708 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.708 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.708 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.708 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.708 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5349 2.1b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e571)", 2024-09-11T03:46:55.708 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:46:55.708 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 26.636677306999999, 2024-09-11T03:46:55.708 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97318775899999999, 2024-09-11T03:46:55.709 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.709 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.710 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "reached pg", 2024-09-11T03:46:55.710 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.710 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.710 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.710 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5349 2024-09-11T03:46:55.710 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.710 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.710 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.710 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.710 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:46:55.710 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.710 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.710 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.711 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.711 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:46:55.711 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.711 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.711 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.711 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.711 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:46:55.711 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7300000000000002e-07 2024-09-11T03:46:55.711 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.711 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.711 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.711 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996490+0000", 2024-09-11T03:46:55.711 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8470000000000001e-06 2024-09-11T03:46:55.711 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.712 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.712 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.712 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996494+0000", 2024-09-11T03:46:55.712 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8419999999999998e-06 2024-09-11T03:46:55.712 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.712 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.712 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.712 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996497+0000", 2024-09-11T03:46:55.712 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8820000000000001e-06 2024-09-11T03:46:55.712 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.712 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.712 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.712 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996602+0000", 2024-09-11T03:46:55.713 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010565699999999999 2024-09-11T03:46:55.713 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.713 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.713 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:46:55.713 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996613+0000", 2024-09-11T03:46:55.713 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0359e-05 2024-09-11T03:46:55.713 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.713 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.713 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.713 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969667+0000", 2024-09-11T03:46:55.713 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97305483599999998 2024-09-11T03:46:55.713 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.713 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.714 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.714 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969675+0000", 2024-09-11T03:46:55.714 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9629999999999998e-06 2024-09-11T03:46:55.714 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.714 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.714 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.714 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.714 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.714 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5350 2.1b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e571)", 2024-09-11T03:46:55.714 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:46:55.714 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 26.636642151, 2024-09-11T03:46:55.714 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.973193639, 2024-09-11T03:46:55.715 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.715 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.715 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "reached pg", 2024-09-11T03:46:55.715 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.715 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.715 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.715 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5350 2024-09-11T03:46:55.715 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.715 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.715 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.715 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.715 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:46:55.715 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.716 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.716 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.716 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.716 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:46:55.716 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.716 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.716 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.716 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.716 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:46:55.716 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.27e-07 2024-09-11T03:46:55.717 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.717 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.717 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.717 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996525+0000", 2024-09-11T03:46:55.717 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.719e-06 2024-09-11T03:46:55.717 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.717 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.717 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.717 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996529+0000", 2024-09-11T03:46:55.717 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2690000000000001e-06 2024-09-11T03:46:55.717 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.717 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.717 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.718 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996532+0000", 2024-09-11T03:46:55.718 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4139999999999998e-06 2024-09-11T03:46:55.718 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.718 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.718 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.718 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996643+0000", 2024-09-11T03:46:55.718 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011041599999999999 2024-09-11T03:46:55.718 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.718 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.718 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:46:55.718 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996651+0000", 2024-09-11T03:46:55.718 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4309999999999994e-06 2024-09-11T03:46:55.718 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.719 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.719 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.719 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969706+0000", 2024-09-11T03:46:55.719 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97305510699999997 2024-09-11T03:46:55.719 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.719 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.719 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.719 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969716+0000", 2024-09-11T03:46:55.719 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9559999999999992e-06 2024-09-11T03:46:55.719 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.719 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.719 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.719 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.719 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.720 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5348 2.1b 2:dde92fc6:::smithi17557870-41:head [sparse-read 0~1317959,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:46:55.720 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973173+0000", 2024-09-11T03:46:55.720 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 24.659991973, 2024-09-11T03:46:55.720 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019341553000000001, 2024-09-11T03:46:55.720 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.720 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.720 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:46:55.720 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.720 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.720 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.720 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5348 2024-09-11T03:46:55.720 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.720 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.720 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.721 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.721 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973173+0000", 2024-09-11T03:46:55.721 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.721 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.721 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.721 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.721 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973173+0000", 2024-09-11T03:46:55.721 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.721 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.721 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.721 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.721 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973175+0000", 2024-09-11T03:46:55.721 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.074e-06 2024-09-11T03:46:55.721 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.722 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.722 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.722 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973183+0000", 2024-09-11T03:46:55.722 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3010000000000007e-06 2024-09-11T03:46:55.722 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.722 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.722 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.722 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973199+0000", 2024-09-11T03:46:55.722 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5279000000000001e-05 2024-09-11T03:46:55.722 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973214+0000", 2024-09-11T03:46:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5891000000000001e-05 2024-09-11T03:46:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973747+0000", 2024-09-11T03:46:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00053236600000000005 2024-09-11T03:46:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:46:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973760+0000", 2024-09-11T03:46:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2976000000000001e-05 2024-09-11T03:46:55.724 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.724 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.724 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.724 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.990782+0000", 2024-09-11T03:46:55.724 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017022371000000001 2024-09-11T03:46:55.724 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.724 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.724 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:55.724 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.990945+0000", 2024-09-11T03:46:55.724 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016245699999999999 2024-09-11T03:46:55.725 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.725 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.725 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.725 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.992514+0000", 2024-09-11T03:46:55.725 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001569838 2024-09-11T03:46:55.725 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.725 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.725 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.725 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.725 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.725 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5349 2.1b 2:dde92fc6:::smithi17557870-41:head [read 0~2235463,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:46:55.725 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:46:55.725 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 24.659927261, 2024-09-11T03:46:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020826846, 2024-09-11T03:46:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:46:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5349 2024-09-11T03:46:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:46:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:46:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:46:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2099999999999996e-07 2024-09-11T03:46:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973242+0000", 2024-09-11T03:46:55.728 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0489999999999997e-06 2024-09-11T03:46:55.728 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.728 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.728 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.728 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973250+0000", 2024-09-11T03:46:55.728 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0700000000000007e-06 2024-09-11T03:46:55.728 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.728 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.728 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.728 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973256+0000", 2024-09-11T03:46:55.728 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3979999999999998e-06 2024-09-11T03:46:55.728 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.728 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.728 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.729 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973794+0000", 2024-09-11T03:46:55.729 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00053782700000000005 2024-09-11T03:46:55.729 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.729 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.729 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:46:55.729 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973801+0000", 2024-09-11T03:46:55.729 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5299999999999999e-06 2024-09-11T03:46:55.729 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.729 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.729 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.729 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.992813+0000", 2024-09-11T03:46:55.729 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019011917999999999 2024-09-11T03:46:55.729 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.730 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.730 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:55.730 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.992923+0000", 2024-09-11T03:46:55.730 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000109874 2024-09-11T03:46:55.730 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.730 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.730 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.730 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994064+0000", 2024-09-11T03:46:55.730 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001141459 2024-09-11T03:46:55.730 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.730 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.730 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.730 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.730 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5350 2.1b 2:dde92fc6:::smithi17557870-41:head [sparse-read 0~1317959] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:46:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973264+0000", 2024-09-11T03:46:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 24.659900538999999, 2024-09-11T03:46:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021144998000000002, 2024-09-11T03:46:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:46:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5350 2024-09-11T03:46:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.732 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.732 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973264+0000", 2024-09-11T03:46:55.732 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.732 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.732 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.732 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.732 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973264+0000", 2024-09-11T03:46:55.732 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.732 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.732 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.732 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.732 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973265+0000", 2024-09-11T03:46:55.732 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5899999999999996e-07 2024-09-11T03:46:55.733 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.733 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.733 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.733 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973268+0000", 2024-09-11T03:46:55.733 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6970000000000002e-06 2024-09-11T03:46:55.733 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.733 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.733 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.733 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973274+0000", 2024-09-11T03:46:55.733 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6390000000000003e-06 2024-09-11T03:46:55.733 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.734 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.734 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.734 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973280+0000", 2024-09-11T03:46:55.734 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.451e-06 2024-09-11T03:46:55.734 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.734 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.734 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.734 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973888+0000", 2024-09-11T03:46:55.734 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00060798999999999998 2024-09-11T03:46:55.734 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.734 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.734 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:46:55.735 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973895+0000", 2024-09-11T03:46:55.735 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6199999999999999e-06 2024-09-11T03:46:55.735 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.735 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.735 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.735 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994115+0000", 2024-09-11T03:46:55.735 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020219487000000001 2024-09-11T03:46:55.735 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.735 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.735 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:55.735 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994236+0000", 2024-09-11T03:46:55.735 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000120715 2024-09-11T03:46:55.735 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.736 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.736 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.736 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994409+0000", 2024-09-11T03:46:55.736 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017384 2024-09-11T03:46:55.736 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.736 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.736 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.736 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.736 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.736 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5362 2.1b 2:de6b1316:::smithi17557870-32:head [read 0~2117847,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:46:55.736 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973757+0000", 2024-09-11T03:46:55.736 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 24.65940763, 2024-09-11T03:46:55.736 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.023415378000000001, 2024-09-11T03:46:55.737 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.737 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.737 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:46:55.737 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.737 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.737 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.737 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5362 2024-09-11T03:46:55.737 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.737 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.737 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.737 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.737 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973757+0000", 2024-09-11T03:46:55.737 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.737 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.738 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.738 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.738 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973757+0000", 2024-09-11T03:46:55.738 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.738 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.738 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.738 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.738 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973758+0000", 2024-09-11T03:46:55.738 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.088e-06 2024-09-11T03:46:55.738 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.738 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.738 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.738 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973765+0000", 2024-09-11T03:46:55.738 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1310000000000003e-06 2024-09-11T03:46:55.739 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.739 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.739 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.739 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973776+0000", 2024-09-11T03:46:55.739 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1762e-05 2024-09-11T03:46:55.739 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.739 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.739 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.739 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973785+0000", 2024-09-11T03:46:55.739 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8629999999999997e-06 2024-09-11T03:46:55.739 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.739 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.739 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.740 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973985+0000", 2024-09-11T03:46:55.740 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000200051 2024-09-11T03:46:55.740 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.740 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.740 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:46:55.740 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973995+0000", 2024-09-11T03:46:55.740 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0054000000000001e-05 2024-09-11T03:46:55.740 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.740 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.740 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.740 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994538+0000", 2024-09-11T03:46:55.740 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020542878000000001 2024-09-11T03:46:55.740 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.740 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.741 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:55.741 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994729+0000", 2024-09-11T03:46:55.741 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00019097100000000001 2024-09-11T03:46:55.741 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.741 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.741 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.741 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.997173+0000", 2024-09-11T03:46:55.741 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024435799999999999 2024-09-11T03:46:55.741 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.741 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.741 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.741 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.741 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.741 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5363 2.1b 2:de6b1316:::smithi17557870-32:head [read 0~2117847,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:46:55.742 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973799+0000", 2024-09-11T03:46:55.742 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 24.659365620999999, 2024-09-11T03:46:55.742 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.024619513999999999, 2024-09-11T03:46:55.742 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.742 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.742 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:46:55.742 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.742 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.742 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.742 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5363 2024-09-11T03:46:55.742 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.742 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.742 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.743 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.743 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973799+0000", 2024-09-11T03:46:55.743 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.743 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.743 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.743 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973799+0000", 2024-09-11T03:46:55.743 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.743 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.743 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.743 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973800+0000", 2024-09-11T03:46:55.743 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2500000000000005e-07 2024-09-11T03:46:55.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.744 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.744 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.744 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973804+0000", 2024-09-11T03:46:55.744 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6049999999999998e-06 2024-09-11T03:46:55.744 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.744 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.744 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.744 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973811+0000", 2024-09-11T03:46:55.744 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0350000000000004e-06 2024-09-11T03:46:55.744 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.744 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.744 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.744 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973816+0000", 2024-09-11T03:46:55.744 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.536e-06 2024-09-11T03:46:55.745 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.745 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.745 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.745 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974024+0000", 2024-09-11T03:46:55.745 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00020766099999999999 2024-09-11T03:46:55.745 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.745 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.745 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:46:55.745 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974032+0000", 2024-09-11T03:46:55.745 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3340000000000002e-06 2024-09-11T03:46:55.745 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.745 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.745 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.745 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.997304+0000", 2024-09-11T03:46:55.746 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.02327183 2024-09-11T03:46:55.746 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.746 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.746 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:55.746 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.997455+0000", 2024-09-11T03:46:55.746 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015147000000000001 2024-09-11T03:46:55.746 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.746 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.746 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.746 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.998419+0000", 2024-09-11T03:46:55.746 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00096341799999999996 2024-09-11T03:46:55.746 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.746 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.747 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.747 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.747 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.747 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5364 2.1b 2:de6b1316:::smithi17557870-32:head [read 0~2117847,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:46:55.747 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:46:55.747 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 24.659341901000001, 2024-09-11T03:46:55.747 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.025348697, 2024-09-11T03:46:55.747 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:46:55.747 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:46:55.747 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:46:55.747 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:46:55.747 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:46:55.747 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:46:55.747 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5364 2024-09-11T03:46:55.747 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.748 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:46:55.748 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.748 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:46:55.748 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:46:55.748 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.748 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.748 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.748 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:46:55.748 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:46:55.748 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:46:55.748 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.748 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.749 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:46:55.749 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:46:55.749 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4099999999999999e-07 2024-09-11T03:46:55.749 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.749 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.749 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:46:55.749 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973826+0000", 2024-09-11T03:46:55.749 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7690000000000001e-06 2024-09-11T03:46:55.749 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.749 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.749 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:46:55.749 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973833+0000", 2024-09-11T03:46:55.749 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4679999999999997e-06 2024-09-11T03:46:55.750 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.750 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.750 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:46:55.750 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973837+0000", 2024-09-11T03:46:55.750 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6070000000000001e-06 2024-09-11T03:46:55.750 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.750 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.750 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.750 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974062+0000", 2024-09-11T03:46:55.750 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00022431400000000001 2024-09-11T03:46:55.750 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.751 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.751 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:46:55.751 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974069+0000", 2024-09-11T03:46:55.751 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8599999999999993e-06 2024-09-11T03:46:55.751 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.751 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.751 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:46:55.751 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.998467+0000", 2024-09-11T03:46:55.751 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.024397723999999999 2024-09-11T03:46:55.751 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.751 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.751 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:46:55.752 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.998581+0000", 2024-09-11T03:46:55.752 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000113769 2024-09-11T03:46:55.752 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:46:55.752 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:46:55.752 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:46:55.752 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.999172+0000", 2024-09-11T03:46:55.752 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00059074500000000005 2024-09-11T03:46:55.752 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.752 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.752 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.752 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:46:55.752 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:46:55.752 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:46:55.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:55.777 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:55.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:55.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:55.879 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:55.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:55.902 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:55.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:56.003 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:56.025 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:56.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:56.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:56.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:46:56.153 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:56.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:56.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:46:56.155 INFO:tasks.rados.rados.2.smithi175.stdout:2251: done (2 left) 2024-09-11T03:46:56.155 INFO:tasks.rados.rados.2.smithi175.stdout:2254: done (1 left) 2024-09-11T03:46:56.155 INFO:tasks.rados.rados.2.smithi175.stdout:2257: done (0 left) 2024-09-11T03:46:56.155 INFO:tasks.rados.rados.2.smithi175.stdout:2258: read oid 11 snap -1 2024-09-11T03:46:56.155 INFO:tasks.rados.rados.2.smithi175.stdout:2258: expect (ObjNum 609 snap 191 seq_num 609) 2024-09-11T03:46:56.156 INFO:tasks.rados.rados.2.smithi175.stdout:2259: write (excl) oid 30 current snap is 248 2024-09-11T03:46:56.156 INFO:tasks.rados.rados.2.smithi175.stdout:2259: seq_num 771 ranges {558197=568632,1650558=632927,2773245=184952} 2024-09-11T03:46:56.160 INFO:tasks.rados.rados.2.smithi175.stdout:2259: writing smithi17557870-30 from 558197 to 1126829 tid 1 2024-09-11T03:46:56.167 INFO:tasks.rados.rados.2.smithi175.stdout:2259: writing smithi17557870-30 from 1650558 to 2283485 tid 2 2024-09-11T03:46:56.171 INFO:tasks.rados.rados.2.smithi175.stdout:2259: writing smithi17557870-30 from 2773245 to 2958197 tid 3 2024-09-11T03:46:56.171 INFO:tasks.rados.rados.2.smithi175.stdout:2260: write oid 43 current snap is 248 2024-09-11T03:46:56.171 INFO:tasks.rados.rados.2.smithi175.stdout:2260: seq_num 772 ranges {575004=643881,1709040=492938,2917021=57983} 2024-09-11T03:46:56.177 INFO:tasks.rados.rados.2.smithi175.stdout:2260: writing smithi17557870-43 from 575004 to 1218885 tid 1 2024-09-11T03:46:56.183 INFO:tasks.rados.rados.2.smithi175.stdout:2260: writing smithi17557870-43 from 1709040 to 2201978 tid 2 2024-09-11T03:46:56.188 INFO:tasks.rados.rados.2.smithi175.stdout:2260: writing smithi17557870-43 from 2917021 to 2975004 tid 3 2024-09-11T03:46:56.188 INFO:tasks.rados.rados.2.smithi175.stdout:2261: read oid 47 snap 247 2024-09-11T03:46:56.188 INFO:tasks.rados.rados.2.smithi175.stdout:2261: expect (ObjNum 762 snap 247 seq_num 762) 2024-09-11T03:46:56.188 INFO:tasks.rados.rados.2.smithi175.stdout:2262: write oid 49 current snap is 248 2024-09-11T03:46:56.188 INFO:tasks.rados.rados.2.smithi175.stdout:2258: expect (ObjNum 609 snap 191 seq_num 609) 2024-09-11T03:46:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:56 smithi175 ceph-mon[30894]: pgmap v1050: 67 pgs: 67 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-11T03:46:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:56 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:46:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:56 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:46:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:56 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3261538401' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:56 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3615707684' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:46:56.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:56 smithi175 ceph-mon[30894]: osdmap e597: 12 total, 12 up, 12 in 2024-09-11T03:46:56.255 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:56.276 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:56.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:56.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:56.293 INFO:tasks.rados.rados.2.smithi175.stdout:2259: finishing write tid 1 to smithi17557870-30 2024-09-11T03:46:56.293 INFO:tasks.rados.rados.2.smithi175.stdout:2259: finishing write tid 2 to smithi17557870-30 2024-09-11T03:46:56.293 INFO:tasks.rados.rados.2.smithi175.stdout:2259: finishing write tid 3 to smithi17557870-30 2024-09-11T03:46:56.293 INFO:tasks.rados.rados.2.smithi175.stdout:2262: seq_num 773 ranges {591811=719130,1767522=636596,2828091=163720} 2024-09-11T03:46:56.300 INFO:tasks.rados.rados.2.smithi175.stdout:2262: writing smithi17557870-49 from 591811 to 1310941 tid 1 2024-09-11T03:46:56.307 INFO:tasks.rados.rados.2.smithi175.stdout:2262: writing smithi17557870-49 from 1767522 to 2404118 tid 2 2024-09-11T03:46:56.310 INFO:tasks.rados.rados.2.smithi175.stdout:2262: writing smithi17557870-49 from 2828091 to 2991811 tid 3 2024-09-11T03:46:56.311 INFO:tasks.rados.rados.2.smithi175.stdout:2258: done (4 left) 2024-09-11T03:46:56.311 INFO:tasks.rados.rados.2.smithi175.stdout:2263: delete oid 12 current snap is 248 2024-09-11T03:46:56.311 INFO:tasks.rados.rados.2.smithi175.stdout:2259: finishing write tid 4 to smithi17557870-30 2024-09-11T03:46:56.311 INFO:tasks.rados.rados.2.smithi175.stdout:2259: finishing write tid 5 to smithi17557870-30 2024-09-11T03:46:56.311 INFO:tasks.rados.rados.2.smithi175.stdout:2259: oid 30 updating version 0 to 874 2024-09-11T03:46:56.311 INFO:tasks.rados.rados.2.smithi175.stdout:2259: oid 30 updating version 874 to 875 2024-09-11T03:46:56.311 INFO:tasks.rados.rados.2.smithi175.stdout:2259: oid 30 version 875 is already newer than 873 2024-09-11T03:46:56.311 INFO:tasks.rados.rados.2.smithi175.stdout:2259: oid 30 version 875 is already newer than 872 2024-09-11T03:46:56.311 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 875 (ObjNum 771 snap 248 seq_num 771) dirty exists 2024-09-11T03:46:56.311 INFO:tasks.rados.rados.2.smithi175.stdout:2259: left oid 30 (ObjNum 771 snap 248 seq_num 771) 2024-09-11T03:46:56.311 INFO:tasks.rados.rados.2.smithi175.stdout:2260: finishing write tid 1 to smithi17557870-43 2024-09-11T03:46:56.312 INFO:tasks.rados.rados.2.smithi175.stdout:2260: finishing write tid 2 to smithi17557870-43 2024-09-11T03:46:56.312 INFO:tasks.rados.rados.2.smithi175.stdout:2260: finishing write tid 3 to smithi17557870-43 2024-09-11T03:46:56.312 INFO:tasks.rados.rados.2.smithi175.stdout:2260: finishing write tid 4 to smithi17557870-43 2024-09-11T03:46:56.312 INFO:tasks.rados.rados.2.smithi175.stdout:2260: finishing write tid 5 to smithi17557870-43 2024-09-11T03:46:56.312 INFO:tasks.rados.rados.2.smithi175.stdout:2260: oid 43 updating version 0 to 429 2024-09-11T03:46:56.312 INFO:tasks.rados.rados.2.smithi175.stdout:2260: oid 43 updating version 429 to 431 2024-09-11T03:46:56.312 INFO:tasks.rados.rados.2.smithi175.stdout:2260: oid 43 version 431 is already newer than 430 2024-09-11T03:46:56.312 INFO:tasks.rados.rados.2.smithi175.stdout:2260: oid 43 updating version 431 to 432 2024-09-11T03:46:56.312 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 432 (ObjNum 772 snap 248 seq_num 772) dirty exists 2024-09-11T03:46:56.312 INFO:tasks.rados.rados.2.smithi175.stdout:2260: left oid 43 (ObjNum 772 snap 248 seq_num 772) 2024-09-11T03:46:56.312 INFO:tasks.rados.rados.2.smithi175.stdout:2261: expect (ObjNum 762 snap 247 seq_num 762) 2024-09-11T03:46:56.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:56 smithi084 ceph-mon[27254]: pgmap v1050: 67 pgs: 67 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-11T03:46:56.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:56 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:46:56.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:56 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:46:56.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3261538401' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:56.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3615707684' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:46:56.347 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:56 smithi084 ceph-mon[27254]: osdmap e597: 12 total, 12 up, 12 in 2024-09-11T03:46:56.378 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:56.399 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:56.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:56.405 INFO:tasks.rados.rados.2.smithi175.stdout:2262: finishing write tid 1 to smithi17557870-49 2024-09-11T03:46:56.405 INFO:tasks.rados.rados.2.smithi175.stdout:2262: finishing write tid 2 to smithi17557870-49 2024-09-11T03:46:56.405 INFO:tasks.rados.rados.2.smithi175.stdout:2262: finishing write tid 3 to smithi17557870-49 2024-09-11T03:46:56.405 INFO:tasks.rados.rados.2.smithi175.stdout:2262: finishing write tid 4 to smithi17557870-49 2024-09-11T03:46:56.405 INFO:tasks.rados.rados.2.smithi175.stdout:2262: finishing write tid 5 to smithi17557870-49 2024-09-11T03:46:56.405 INFO:tasks.rados.rados.2.smithi175.stdout:2262: oid 49 updating version 0 to 686 2024-09-11T03:46:56.405 INFO:tasks.rados.rados.2.smithi175.stdout:2262: oid 49 updating version 686 to 688 2024-09-11T03:46:56.405 INFO:tasks.rados.rados.2.smithi175.stdout:2262: oid 49 version 688 is already newer than 687 2024-09-11T03:46:56.405 INFO:tasks.rados.rados.2.smithi175.stdout:2262: oid 49 updating version 688 to 689 2024-09-11T03:46:56.405 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 689 (ObjNum 773 snap 248 seq_num 773) dirty exists 2024-09-11T03:46:56.405 INFO:tasks.rados.rados.2.smithi175.stdout:2262: left oid 49 (ObjNum 773 snap 248 seq_num 773) 2024-09-11T03:46:56.405 INFO:tasks.rados.rados.2.smithi175.stdout:2259: done (4 left) 2024-09-11T03:46:56.405 INFO:tasks.rados.rados.2.smithi175.stdout:2260: done (3 left) 2024-09-11T03:46:56.406 INFO:tasks.rados.rados.2.smithi175.stdout:2261: done (2 left) 2024-09-11T03:46:56.406 INFO:tasks.rados.rados.2.smithi175.stdout:2262: done (1 left) 2024-09-11T03:46:56.406 INFO:tasks.rados.rados.2.smithi175.stdout:2263: done (0 left) 2024-09-11T03:46:56.406 INFO:tasks.rados.rados.2.smithi175.stdout:2264: write (excl) oid 20 current snap is 248 2024-09-11T03:46:56.406 INFO:tasks.rados.rados.2.smithi175.stdout:2264: seq_num 775 ranges {625425=469628,1768133=640265} 2024-09-11T03:46:56.410 INFO:tasks.rados.rados.2.smithi175.stdout:2264: writing smithi17557870-20 from 625425 to 1095053 tid 1 2024-09-11T03:46:56.418 INFO:tasks.rados.rados.2.smithi175.stdout:2264: writing smithi17557870-20 from 1768133 to 2408398 tid 2 2024-09-11T03:46:56.418 INFO:tasks.rados.rados.2.smithi175.stdout:2265: snap_remove snap 247 2024-09-11T03:46:56.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:56 smithi121 ceph-mon[30729]: pgmap v1050: 67 pgs: 67 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-11T03:46:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:56 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:46:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:56 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:46:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:56 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3261538401' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:46:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:56 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3615707684' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:46:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:56 smithi121 ceph-mon[30729]: osdmap e597: 12 total, 12 up, 12 in 2024-09-11T03:46:56.501 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:56.523 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:56.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:56.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:56.625 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:56.628 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T03:46:56.646 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:56.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:56.647 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:56.748 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:56.769 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:56.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:56.770 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:56.871 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:56.892 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:56.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:56.893 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:56.973 INFO:tasks.rados.rados.2.smithi175.stdout:2265: done (1 left) 2024-09-11T03:46:56.974 INFO:tasks.rados.rados.2.smithi175.stdout:2266: copy_from oid 16 from oid 29 current snap is 248 2024-09-11T03:46:56.974 INFO:tasks.rados.rados.2.smithi175.stdout:2264: finishing write tid 1 to smithi17557870-20 2024-09-11T03:46:56.974 INFO:tasks.rados.rados.2.smithi175.stdout:2264: finishing write tid 2 to smithi17557870-20 2024-09-11T03:46:56.974 INFO:tasks.rados.rados.2.smithi175.stdout:2264: finishing write tid 3 to smithi17557870-20 2024-09-11T03:46:56.974 INFO:tasks.rados.rados.2.smithi175.stdout:2264: finishing write tid 4 to smithi17557870-20 2024-09-11T03:46:56.974 INFO:tasks.rados.rados.2.smithi175.stdout:2264: oid 20 updating version 0 to 772 2024-09-11T03:46:56.974 INFO:tasks.rados.rados.2.smithi175.stdout:2264: oid 20 updating version 772 to 773 2024-09-11T03:46:56.974 INFO:tasks.rados.rados.2.smithi175.stdout:2264: oid 20 version 773 is already newer than 771 2024-09-11T03:46:56.974 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 773 (ObjNum 775 snap 248 seq_num 775) dirty exists 2024-09-11T03:46:56.974 INFO:tasks.rados.rados.2.smithi175.stdout:2264: left oid 20 (ObjNum 775 snap 248 seq_num 775) 2024-09-11T03:46:56.974 INFO:tasks.rados.rados.2.smithi175.stdout:2264: done (1 left) 2024-09-11T03:46:56.974 INFO:tasks.rados.rados.2.smithi175.stdout:2267: copy_from oid 14 from oid 30 current snap is 248 2024-09-11T03:46:56.975 INFO:tasks.rados.rados.2.smithi175.stdout:2268: write oid 9 current snap is 248 2024-09-11T03:46:56.975 INFO:tasks.rados.rados.2.smithi175.stdout:2268: seq_num 776 ranges {642232=661230,1942968=783923} 2024-09-11T03:46:56.981 INFO:tasks.rados.rados.2.smithi175.stdout:2268: writing smithi17557870-9 from 642232 to 1303462 tid 1 2024-09-11T03:46:56.989 INFO:tasks.rados.rados.2.smithi175.stdout:2268: writing smithi17557870-9 from 1942968 to 2726891 tid 2 2024-09-11T03:46:56.990 INFO:tasks.rados.rados.2.smithi175.stdout:2269: rollback oid 34 current snap is 248 2024-09-11T03:46:56.990 INFO:tasks.rados.rados.2.smithi175.stdout:2269: done (3 left) 2024-09-11T03:46:56.990 INFO:tasks.rados.rados.2.smithi175.stdout:2270: read oid 25 snap -1 2024-09-11T03:46:56.990 INFO:tasks.rados.rados.2.smithi175.stdout:2270: expect (ObjNum 681 snap 218 seq_num 681) 2024-09-11T03:46:56.990 INFO:tasks.rados.rados.2.smithi175.stdout:2271: write (excl) oid 42 current snap is 248 2024-09-11T03:46:56.990 INFO:tasks.rados.rados.2.smithi175.stdout:2266: finishing copy_from racing read to smithi17557870-16 2024-09-11T03:46:56.990 INFO:tasks.rados.rados.2.smithi175.stdout:2271: seq_num 777 ranges {659039=736479,2001450=643934} 2024-09-11T03:46:56.994 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:56.998 INFO:tasks.rados.rados.2.smithi175.stdout:2271: writing smithi17557870-42 from 659039 to 1395518 tid 1 2024-09-11T03:46:57.005 INFO:tasks.rados.rados.2.smithi175.stdout:2271: writing smithi17557870-42 from 2001450 to 2645384 tid 2 2024-09-11T03:46:57.005 INFO:tasks.rados.rados.2.smithi175.stdout:2272: write (excl) oid 15 current snap is 248 2024-09-11T03:46:57.005 INFO:tasks.rados.rados.2.smithi175.stdout:2268: finishing write tid 1 to smithi17557870-9 2024-09-11T03:46:57.005 INFO:tasks.rados.rados.2.smithi175.stdout:2272: seq_num 778 ranges {675846=411728,1543579=787592,2848853=226993} 2024-09-11T03:46:57.011 INFO:tasks.rados.rados.2.smithi175.stdout:2272: writing smithi17557870-15 from 675846 to 1087574 tid 1 2024-09-11T03:46:57.018 INFO:tasks.rados.rados.2.smithi175.stdout:2272: writing smithi17557870-15 from 1543579 to 2331171 tid 2 2024-09-11T03:46:57.019 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:57.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:57.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:57.023 INFO:tasks.rados.rados.2.smithi175.stdout:2272: writing smithi17557870-15 from 2848853 to 3075846 tid 3 2024-09-11T03:46:57.023 INFO:tasks.rados.rados.2.smithi175.stdout:2273: snap_create 2024-09-11T03:46:57.023 INFO:tasks.rados.rados.2.smithi175.stdout:2267: finishing copy_from racing read to smithi17557870-14 2024-09-11T03:46:57.023 INFO:tasks.rados.rados.2.smithi175.stdout:2266: finishing copy_from to smithi17557870-16 2024-09-11T03:46:57.023 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 826 (ObjNum 736 snap 243 seq_num 736) dirty exists 2024-09-11T03:46:57.023 INFO:tasks.rados.rados.2.smithi175.stdout:2268: finishing write tid 2 to smithi17557870-9 2024-09-11T03:46:57.023 INFO:tasks.rados.rados.2.smithi175.stdout:2268: finishing write tid 3 to smithi17557870-9 2024-09-11T03:46:57.024 INFO:tasks.rados.rados.2.smithi175.stdout:2268: finishing write tid 4 to smithi17557870-9 2024-09-11T03:46:57.024 INFO:tasks.rados.rados.2.smithi175.stdout:2268: oid 9 updating version 0 to 930 2024-09-11T03:46:57.024 INFO:tasks.rados.rados.2.smithi175.stdout:2268: oid 9 updating version 930 to 931 2024-09-11T03:46:57.024 INFO:tasks.rados.rados.2.smithi175.stdout:2268: oid 9 version 931 is already newer than 929 2024-09-11T03:46:57.024 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 931 (ObjNum 776 snap 248 seq_num 776) dirty exists 2024-09-11T03:46:57.024 INFO:tasks.rados.rados.2.smithi175.stdout:2268: left oid 9 (ObjNum 776 snap 248 seq_num 776) 2024-09-11T03:46:57.024 INFO:tasks.rados.rados.2.smithi175.stdout:2267: finishing copy_from to smithi17557870-14 2024-09-11T03:46:57.024 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 875 (ObjNum 771 snap 248 seq_num 771) dirty exists 2024-09-11T03:46:57.024 INFO:tasks.rados.rados.2.smithi175.stdout:2270: expect (ObjNum 681 snap 218 seq_num 681) 2024-09-11T03:46:57.121 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:57.144 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:57.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:57.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:57.201 INFO:tasks.rados.rados.2.smithi175.stdout:2271: finishing write tid 1 to smithi17557870-42 2024-09-11T03:46:57.202 INFO:tasks.rados.rados.2.smithi175.stdout:2271: finishing write tid 2 to smithi17557870-42 2024-09-11T03:46:57.202 INFO:tasks.rados.rados.2.smithi175.stdout:2271: finishing write tid 3 to smithi17557870-42 2024-09-11T03:46:57.202 INFO:tasks.rados.rados.2.smithi175.stdout:2271: finishing write tid 4 to smithi17557870-42 2024-09-11T03:46:57.202 INFO:tasks.rados.rados.2.smithi175.stdout:2271: oid 42 updating version 0 to 692 2024-09-11T03:46:57.202 INFO:tasks.rados.rados.2.smithi175.stdout:2271: oid 42 version 692 is already newer than 691 2024-09-11T03:46:57.202 INFO:tasks.rados.rados.2.smithi175.stdout:2271: oid 42 version 692 is already newer than 690 2024-09-11T03:46:57.202 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 692 (ObjNum 777 snap 248 seq_num 777) dirty exists 2024-09-11T03:46:57.202 INFO:tasks.rados.rados.2.smithi175.stdout:2271: left oid 42 (ObjNum 777 snap 248 seq_num 777) 2024-09-11T03:46:57.202 INFO:tasks.rados.rados.2.smithi175.stdout:2272: finishing write tid 1 to smithi17557870-15 2024-09-11T03:46:57.202 INFO:tasks.rados.rados.2.smithi175.stdout:2272: finishing write tid 2 to smithi17557870-15 2024-09-11T03:46:57.202 INFO:tasks.rados.rados.2.smithi175.stdout:2272: finishing write tid 3 to smithi17557870-15 2024-09-11T03:46:57.202 INFO:tasks.rados.rados.2.smithi175.stdout:2272: finishing write tid 4 to smithi17557870-15 2024-09-11T03:46:57.203 INFO:tasks.rados.rados.2.smithi175.stdout:2272: finishing write tid 5 to smithi17557870-15 2024-09-11T03:46:57.203 INFO:tasks.rados.rados.2.smithi175.stdout:2272: oid 15 updating version 0 to 258 2024-09-11T03:46:57.203 INFO:tasks.rados.rados.2.smithi175.stdout:2272: oid 15 updating version 258 to 260 2024-09-11T03:46:57.203 INFO:tasks.rados.rados.2.smithi175.stdout:2272: oid 15 version 260 is already newer than 257 2024-09-11T03:46:57.203 INFO:tasks.rados.rados.2.smithi175.stdout:2272: oid 15 version 260 is already newer than 259 2024-09-11T03:46:57.203 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 260 (ObjNum 778 snap 248 seq_num 778) dirty exists 2024-09-11T03:46:57.203 INFO:tasks.rados.rados.2.smithi175.stdout:2272: left oid 15 (ObjNum 778 snap 248 seq_num 778) 2024-09-11T03:46:57.246 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:57.268 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:57.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:57.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:57.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:57 smithi084 ceph-mon[27254]: osdmap e598: 12 total, 12 up, 12 in 2024-09-11T03:46:57.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:57 smithi084 ceph-mon[27254]: osdmap e599: 12 total, 12 up, 12 in 2024-09-11T03:46:57.353 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-11T03:46:57.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:57.392 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:57.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:57.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:57.403 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:57.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:57 smithi121 ceph-mon[30729]: osdmap e598: 12 total, 12 up, 12 in 2024-09-11T03:46:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:57 smithi121 ceph-mon[30729]: osdmap e599: 12 total, 12 up, 12 in 2024-09-11T03:46:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:57 smithi175 ceph-mon[30894]: osdmap e598: 12 total, 12 up, 12 in 2024-09-11T03:46:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:57 smithi175 ceph-mon[30894]: osdmap e599: 12 total, 12 up, 12 in 2024-09-11T03:46:57.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:46:57.519 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:57.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:57.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:46:57.621 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:57.645 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:57.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:57.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:57.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:46:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:57.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:46:57.873 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:57.894 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:46:57.894 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:46:57.894 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:46:57.894 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:46:57.895 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:57.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:57.896 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:57.972 INFO:tasks.rados.rados.2.smithi175.stdout:2266: done (6 left) 2024-09-11T03:46:57.973 INFO:tasks.rados.rados.2.smithi175.stdout:2267: done (5 left) 2024-09-11T03:46:57.973 INFO:tasks.rados.rados.2.smithi175.stdout:2268: done (4 left) 2024-09-11T03:46:57.973 INFO:tasks.rados.rados.2.smithi175.stdout:2270: done (3 left) 2024-09-11T03:46:57.973 INFO:tasks.rados.rados.2.smithi175.stdout:2271: done (2 left) 2024-09-11T03:46:57.973 INFO:tasks.rados.rados.2.smithi175.stdout:2272: done (1 left) 2024-09-11T03:46:57.973 INFO:tasks.rados.rados.2.smithi175.stdout:2273: done (0 left) 2024-09-11T03:46:57.973 INFO:tasks.rados.rados.2.smithi175.stdout:2274: write (excl) oid 27 current snap is 249 2024-09-11T03:46:57.973 INFO:tasks.rados.rados.2.smithi175.stdout:2274: seq_num 779 ranges {692653=486977,1602061=647603,2992629=100024} 2024-09-11T03:46:57.978 INFO:tasks.rados.rados.2.smithi175.stdout:2274: writing smithi17557870-27 from 692653 to 1179630 tid 1 2024-09-11T03:46:57.984 INFO:tasks.rados.rados.2.smithi175.stdout:2274: writing smithi17557870-27 from 1602061 to 2249664 tid 2 2024-09-11T03:46:57.989 INFO:tasks.rados.rados.2.smithi175.stdout:2274: writing smithi17557870-27 from 2992629 to 3092653 tid 3 2024-09-11T03:46:57.989 INFO:tasks.rados.rados.2.smithi175.stdout:2275: write oid 16 current snap is 249 2024-09-11T03:46:57.989 INFO:tasks.rados.rados.2.smithi175.stdout:2274: finishing write tid 1 to smithi17557870-27 2024-09-11T03:46:57.989 INFO:tasks.rados.rados.2.smithi175.stdout:2274: finishing write tid 2 to smithi17557870-27 2024-09-11T03:46:57.989 INFO:tasks.rados.rados.2.smithi175.stdout:2275: seq_num 780 ranges {709460=562226,2060543=791261} 2024-09-11T03:46:57.996 INFO:tasks.rados.rados.2.smithi175.stdout:2275: writing smithi17557870-16 from 709460 to 1271686 tid 1 2024-09-11T03:46:57.997 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:58.005 INFO:tasks.rados.rados.2.smithi175.stdout:2275: writing smithi17557870-16 from 2060543 to 2851804 tid 2 2024-09-11T03:46:58.005 INFO:tasks.rados.rados.2.smithi175.stdout:2276: write (excl) oid 13 current snap is 249 2024-09-11T03:46:58.006 INFO:tasks.rados.rados.2.smithi175.stdout:2274: finishing write tid 3 to smithi17557870-27 2024-09-11T03:46:58.006 INFO:tasks.rados.rados.2.smithi175.stdout:2274: finishing write tid 4 to smithi17557870-27 2024-09-11T03:46:58.006 INFO:tasks.rados.rados.2.smithi175.stdout:2274: finishing write tid 5 to smithi17557870-27 2024-09-11T03:46:58.006 INFO:tasks.rados.rados.2.smithi175.stdout:2274: oid 27 updating version 0 to 935 2024-09-11T03:46:58.006 INFO:tasks.rados.rados.2.smithi175.stdout:2274: oid 27 version 935 is already newer than 934 2024-09-11T03:46:58.006 INFO:tasks.rados.rados.2.smithi175.stdout:2274: oid 27 version 935 is already newer than 933 2024-09-11T03:46:58.006 INFO:tasks.rados.rados.2.smithi175.stdout:2274: oid 27 updating version 935 to 936 2024-09-11T03:46:58.006 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 936 (ObjNum 779 snap 249 seq_num 779) dirty exists 2024-09-11T03:46:58.006 INFO:tasks.rados.rados.2.smithi175.stdout:2274: left oid 27 (ObjNum 779 snap 249 seq_num 779) 2024-09-11T03:46:58.006 INFO:tasks.rados.rados.2.smithi175.stdout:2275: finishing write tid 1 to smithi17557870-16 2024-09-11T03:46:58.006 INFO:tasks.rados.rados.2.smithi175.stdout:2276: seq_num 781 ranges {726267=637475,2119025=651272} 2024-09-11T03:46:58.012 INFO:tasks.rados.rados.2.smithi175.stdout:2276: writing smithi17557870-13 from 726267 to 1363742 tid 1 2024-09-11T03:46:58.019 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:58.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:58.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:58.021 INFO:tasks.rados.rados.2.smithi175.stdout:2276: writing smithi17557870-13 from 2119025 to 2770297 tid 2 2024-09-11T03:46:58.021 INFO:tasks.rados.rados.2.smithi175.stdout:2274: done (2 left) 2024-09-11T03:46:58.021 INFO:tasks.rados.rados.2.smithi175.stdout:2277: snap_remove snap 248 2024-09-11T03:46:58.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:46:58.122 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:58.145 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:58.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:58.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:58.151 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:58.247 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:58.268 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:58.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:58.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:58.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:58.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:58.392 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:58.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:58.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:58 smithi084 ceph-mon[27254]: pgmap v1053: 67 pgs: 67 active+clean; 91 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.8 MiB/s wr, 20 op/s 2024-09-11T03:46:58.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:58 smithi084 ceph-mon[27254]: osdmap e600: 12 total, 12 up, 12 in 2024-09-11T03:46:58.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:58 smithi121 ceph-mon[30729]: pgmap v1053: 67 pgs: 67 active+clean; 91 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.8 MiB/s wr, 20 op/s 2024-09-11T03:46:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:58 smithi121 ceph-mon[30729]: osdmap e600: 12 total, 12 up, 12 in 2024-09-11T03:46:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:58 smithi175 ceph-mon[30894]: pgmap v1053: 67 pgs: 67 active+clean; 91 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.8 MiB/s wr, 20 op/s 2024-09-11T03:46:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:58 smithi175 ceph-mon[30894]: osdmap e600: 12 total, 12 up, 12 in 2024-09-11T03:46:58.493 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:46:58.515 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:58.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:58.516 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:46:58.617 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:46:58.639 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:58.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:58.640 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:46:58.714 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T03:46:58.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:58.761 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:46:58.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:58.778 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:58.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:58.879 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:58.900 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:58.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:58.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:58.975 INFO:tasks.rados.rados.2.smithi175.stdout:2277: done (2 left) 2024-09-11T03:46:58.976 INFO:tasks.rados.rados.2.smithi175.stdout:2278: write oid 15 current snap is 249 2024-09-11T03:46:58.976 INFO:tasks.rados.rados.2.smithi175.stdout:2275: finishing write tid 2 to smithi17557870-16 2024-09-11T03:46:58.976 INFO:tasks.rados.rados.2.smithi175.stdout:2275: finishing write tid 3 to smithi17557870-16 2024-09-11T03:46:58.976 INFO:tasks.rados.rados.2.smithi175.stdout:2275: finishing write tid 4 to smithi17557870-16 2024-09-11T03:46:58.976 INFO:tasks.rados.rados.2.smithi175.stdout:2275: oid 16 updating version 0 to 828 2024-09-11T03:46:58.976 INFO:tasks.rados.rados.2.smithi175.stdout:2275: oid 16 version 828 is already newer than 827 2024-09-11T03:46:58.977 INFO:tasks.rados.rados.2.smithi175.stdout:2275: oid 16 updating version 828 to 829 2024-09-11T03:46:58.977 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 829 (ObjNum 780 snap 249 seq_num 780) dirty exists 2024-09-11T03:46:58.977 INFO:tasks.rados.rados.2.smithi175.stdout:2275: left oid 16 (ObjNum 780 snap 249 seq_num 780) 2024-09-11T03:46:58.977 INFO:tasks.rados.rados.2.smithi175.stdout:2276: finishing write tid 1 to smithi17557870-13 2024-09-11T03:46:58.977 INFO:tasks.rados.rados.2.smithi175.stdout:2278: seq_num 782 ranges {743074=712724,2061154=794930} 2024-09-11T03:46:58.983 INFO:tasks.rados.rados.2.smithi175.stdout:2278: writing smithi17557870-15 from 743074 to 1455798 tid 1 2024-09-11T03:46:58.992 INFO:tasks.rados.rados.2.smithi175.stdout:2278: writing smithi17557870-15 from 2061154 to 2856084 tid 2 2024-09-11T03:46:58.992 INFO:tasks.rados.rados.2.smithi175.stdout:2275: done (2 left) 2024-09-11T03:46:58.992 INFO:tasks.rados.rados.2.smithi175.stdout:2279: read oid 31 snap -1 2024-09-11T03:46:58.992 INFO:tasks.rados.rados.2.smithi175.stdout:2279: expect (ObjNum 758 snap 246 seq_num 758) 2024-09-11T03:46:58.992 INFO:tasks.rados.rados.2.smithi175.stdout:2280: write (excl) oid 30 current snap is 249 2024-09-11T03:46:58.992 INFO:tasks.rados.rados.2.smithi175.stdout:2276: finishing write tid 2 to smithi17557870-13 2024-09-11T03:46:58.992 INFO:tasks.rados.rados.2.smithi175.stdout:2276: finishing write tid 3 to smithi17557870-13 2024-09-11T03:46:58.993 INFO:tasks.rados.rados.2.smithi175.stdout:2276: finishing write tid 4 to smithi17557870-13 2024-09-11T03:46:58.993 INFO:tasks.rados.rados.2.smithi175.stdout:2276: oid 13 updating version 0 to 871 2024-09-11T03:46:58.993 INFO:tasks.rados.rados.2.smithi175.stdout:2276: oid 13 updating version 871 to 872 2024-09-11T03:46:58.993 INFO:tasks.rados.rados.2.smithi175.stdout:2276: oid 13 version 872 is already newer than 870 2024-09-11T03:46:58.993 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 872 (ObjNum 781 snap 249 seq_num 781) dirty exists 2024-09-11T03:46:58.993 INFO:tasks.rados.rados.2.smithi175.stdout:2276: left oid 13 (ObjNum 781 snap 249 seq_num 781) 2024-09-11T03:46:58.993 INFO:tasks.rados.rados.2.smithi175.stdout:2280: seq_num 783 ranges {759881=787973,2119636=654941} 2024-09-11T03:46:59.000 INFO:tasks.rados.rados.2.smithi175.stdout:2280: writing smithi17557870-30 from 759881 to 1547854 tid 1 2024-09-11T03:46:59.001 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:59.007 INFO:tasks.rados.rados.2.smithi175.stdout:2280: writing smithi17557870-30 from 2119636 to 2774577 tid 2 2024-09-11T03:46:59.008 INFO:tasks.rados.rados.2.smithi175.stdout:2276: done (3 left) 2024-09-11T03:46:59.008 INFO:tasks.rados.rados.2.smithi175.stdout:2281: write (excl) oid 46 current snap is 249 2024-09-11T03:46:59.008 INFO:tasks.rados.rados.2.smithi175.stdout:2281: seq_num 784 ranges {776688=579575,1894471=798599} 2024-09-11T03:46:59.015 INFO:tasks.rados.rados.2.smithi175.stdout:2281: writing smithi17557870-46 from 776688 to 1356263 tid 1 2024-09-11T03:46:59.023 INFO:tasks.rados.rados.2.smithi175.stdout:2281: writing smithi17557870-46 from 1894471 to 2693070 tid 2 2024-09-11T03:46:59.023 INFO:tasks.rados.rados.2.smithi175.stdout:2282: delete oid 28 current snap is 249 2024-09-11T03:46:59.023 INFO:tasks.rados.rados.2.smithi175.stdout:2278: finishing write tid 1 to smithi17557870-15 2024-09-11T03:46:59.023 INFO:tasks.rados.rados.2.smithi175.stdout:2278: finishing write tid 2 to smithi17557870-15 2024-09-11T03:46:59.024 INFO:tasks.rados.rados.2.smithi175.stdout:2278: finishing write tid 3 to smithi17557870-15 2024-09-11T03:46:59.024 INFO:tasks.rados.rados.2.smithi175.stdout:2278: finishing write tid 4 to smithi17557870-15 2024-09-11T03:46:59.024 INFO:tasks.rados.rados.2.smithi175.stdout:2278: oid 15 updating version 0 to 263 2024-09-11T03:46:59.024 INFO:tasks.rados.rados.2.smithi175.stdout:2278: oid 15 version 263 is already newer than 262 2024-09-11T03:46:59.024 INFO:tasks.rados.rados.2.smithi175.stdout:2278: oid 15 version 263 is already newer than 261 2024-09-11T03:46:59.024 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 263 (ObjNum 782 snap 249 seq_num 782) dirty exists 2024-09-11T03:46:59.024 INFO:tasks.rados.rados.2.smithi175.stdout:2278: left oid 15 (ObjNum 782 snap 249 seq_num 782) 2024-09-11T03:46:59.024 INFO:tasks.rados.rados.2.smithi175.stdout:2280: finishing write tid 1 to smithi17557870-30 2024-09-11T03:46:59.024 INFO:tasks.rados.rados.2.smithi175.stdout:2279: expect (ObjNum 758 snap 246 seq_num 758) 2024-09-11T03:46:59.025 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:59.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:59.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:59.116 INFO:tasks.rados.rados.2.smithi175.stdout:2281: finishing write tid 1 to smithi17557870-46 2024-09-11T03:46:59.116 INFO:tasks.rados.rados.2.smithi175.stdout:2281: finishing write tid 2 to smithi17557870-46 2024-09-11T03:46:59.116 INFO:tasks.rados.rados.2.smithi175.stdout:2281: finishing write tid 3 to smithi17557870-46 2024-09-11T03:46:59.116 INFO:tasks.rados.rados.2.smithi175.stdout:2281: finishing write tid 4 to smithi17557870-46 2024-09-11T03:46:59.117 INFO:tasks.rados.rados.2.smithi175.stdout:2281: oid 46 updating version 0 to 540 2024-09-11T03:46:59.117 INFO:tasks.rados.rados.2.smithi175.stdout:2281: oid 46 updating version 540 to 542 2024-09-11T03:46:59.117 INFO:tasks.rados.rados.2.smithi175.stdout:2281: oid 46 version 542 is already newer than 541 2024-09-11T03:46:59.117 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 542 (ObjNum 784 snap 249 seq_num 784) dirty exists 2024-09-11T03:46:59.117 INFO:tasks.rados.rados.2.smithi175.stdout:2281: left oid 46 (ObjNum 784 snap 249 seq_num 784) 2024-09-11T03:46:59.117 INFO:tasks.rados.rados.2.smithi175.stdout:2280: finishing write tid 2 to smithi17557870-30 2024-09-11T03:46:59.117 INFO:tasks.rados.rados.2.smithi175.stdout:2278: done (4 left) 2024-09-11T03:46:59.118 INFO:tasks.rados.rados.2.smithi175.stdout:2279: done (3 left) 2024-09-11T03:46:59.118 INFO:tasks.rados.rados.2.smithi175.stdout:2281: done (2 left) 2024-09-11T03:46:59.118 INFO:tasks.rados.rados.2.smithi175.stdout:2282: done (1 left) 2024-09-11T03:46:59.118 INFO:tasks.rados.rados.2.smithi175.stdout:2283: delete oid 50 current snap is 249 2024-09-11T03:46:59.118 INFO:tasks.rados.rados.2.smithi175.stdout:2280: finishing write tid 3 to smithi17557870-30 2024-09-11T03:46:59.118 INFO:tasks.rados.rados.2.smithi175.stdout:2280: finishing write tid 4 to smithi17557870-30 2024-09-11T03:46:59.118 INFO:tasks.rados.rados.2.smithi175.stdout:2280: oid 30 updating version 0 to 876 2024-09-11T03:46:59.118 INFO:tasks.rados.rados.2.smithi175.stdout:2280: oid 30 updating version 876 to 878 2024-09-11T03:46:59.119 INFO:tasks.rados.rados.2.smithi175.stdout:2280: oid 30 version 878 is already newer than 877 2024-09-11T03:46:59.119 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 878 (ObjNum 783 snap 249 seq_num 783) dirty exists 2024-09-11T03:46:59.119 INFO:tasks.rados.rados.2.smithi175.stdout:2280: left oid 30 (ObjNum 783 snap 249 seq_num 783) 2024-09-11T03:46:59.119 INFO:tasks.rados.rados.2.smithi175.stdout:2280: done (1 left) 2024-09-11T03:46:59.120 INFO:tasks.rados.rados.2.smithi175.stdout:2283: done (0 left) 2024-09-11T03:46:59.120 INFO:tasks.rados.rados.2.smithi175.stdout:2284: write (excl) oid 44 current snap is 249 2024-09-11T03:46:59.120 INFO:tasks.rados.rados.2.smithi175.stdout:2284: seq_num 787 ranges {427109=405322,1553564=545926,2579307=647802} 2024-09-11T03:46:59.122 INFO:tasks.rados.rados.2.smithi175.stdout:2284: writing smithi17557870-44 from 427109 to 832431 tid 1 2024-09-11T03:46:59.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:46:59.129 INFO:tasks.rados.rados.2.smithi175.stdout:2284: writing smithi17557870-44 from 1553564 to 2099490 tid 2 2024-09-11T03:46:59.135 INFO:tasks.rados.rados.2.smithi175.stdout:2284: writing smithi17557870-44 from 2579307 to 3227109 tid 3 2024-09-11T03:46:59.135 INFO:tasks.rados.rados.2.smithi175.stdout:2285: delete oid 13 current snap is 249 2024-09-11T03:46:59.135 INFO:tasks.rados.rados.2.smithi175.stdout:2284: finishing write tid 1 to smithi17557870-44 2024-09-11T03:46:59.135 INFO:tasks.rados.rados.2.smithi175.stdout:2284: finishing write tid 2 to smithi17557870-44 2024-09-11T03:46:59.136 INFO:tasks.rados.rados.2.smithi175.stdout:2285: done (1 left) 2024-09-11T03:46:59.137 INFO:tasks.rados.rados.2.smithi175.stdout:2286: write oid 4 current snap is 249 2024-09-11T03:46:59.137 INFO:tasks.rados.rados.2.smithi175.stdout:2286: seq_num 789 ranges {460723=555820,1670528=549595,2634153=626570} 2024-09-11T03:46:59.142 INFO:tasks.rados.rados.2.smithi175.stdout:2286: writing smithi17557870-4 from 460723 to 1016543 tid 1 2024-09-11T03:46:59.148 INFO:tasks.rados.rados.2.smithi175.stdout:2286: writing smithi17557870-4 from 1670528 to 2220123 tid 2 2024-09-11T03:46:59.150 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:59.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:59.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:46:59.154 INFO:tasks.rados.rados.2.smithi175.stdout:2286: writing smithi17557870-4 from 2634153 to 3260723 tid 3 2024-09-11T03:46:59.155 INFO:tasks.rados.rados.2.smithi175.stdout:2287: write (excl) oid 26 current snap is 249 2024-09-11T03:46:59.155 INFO:tasks.rados.rados.2.smithi175.stdout:2284: finishing write tid 3 to smithi17557870-44 2024-09-11T03:46:59.155 INFO:tasks.rados.rados.2.smithi175.stdout:2284: finishing write tid 4 to smithi17557870-44 2024-09-11T03:46:59.155 INFO:tasks.rados.rados.2.smithi175.stdout:2284: finishing write tid 5 to smithi17557870-44 2024-09-11T03:46:59.155 INFO:tasks.rados.rados.2.smithi175.stdout:2284: oid 44 updating version 0 to 808 2024-09-11T03:46:59.155 INFO:tasks.rados.rados.2.smithi175.stdout:2284: oid 44 version 808 is already newer than 807 2024-09-11T03:46:59.155 INFO:tasks.rados.rados.2.smithi175.stdout:2284: oid 44 updating version 808 to 809 2024-09-11T03:46:59.155 INFO:tasks.rados.rados.2.smithi175.stdout:2284: oid 44 updating version 809 to 810 2024-09-11T03:46:59.155 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 810 (ObjNum 787 snap 249 seq_num 787) dirty exists 2024-09-11T03:46:59.155 INFO:tasks.rados.rados.2.smithi175.stdout:2284: left oid 44 (ObjNum 787 snap 249 seq_num 787) 2024-09-11T03:46:59.155 INFO:tasks.rados.rados.2.smithi175.stdout:2286: finishing write tid 1 to smithi17557870-4 2024-09-11T03:46:59.155 INFO:tasks.rados.rados.2.smithi175.stdout:2286: finishing write tid 2 to smithi17557870-4 2024-09-11T03:46:59.155 INFO:tasks.rados.rados.2.smithi175.stdout:2287: seq_num 790 ranges {477530=631069,1612657=409606,2545223=517596} 2024-09-11T03:46:59.160 INFO:tasks.rados.rados.2.smithi175.stdout:2287: writing smithi17557870-26 from 477530 to 1108599 tid 1 2024-09-11T03:46:59.165 INFO:tasks.rados.rados.2.smithi175.stdout:2287: writing smithi17557870-26 from 1612657 to 2022263 tid 2 2024-09-11T03:46:59.171 INFO:tasks.rados.rados.2.smithi175.stdout:2287: writing smithi17557870-26 from 2545223 to 3062819 tid 3 2024-09-11T03:46:59.171 INFO:tasks.rados.rados.2.smithi175.stdout:2284: done (2 left) 2024-09-11T03:46:59.171 INFO:tasks.rados.rados.2.smithi175.stdout:2288: rollback oid 16 current snap is 249 2024-09-11T03:46:59.171 INFO:tasks.rados.rados.2.smithi175.stdout:2288: done (2 left) 2024-09-11T03:46:59.172 INFO:tasks.rados.rados.2.smithi175.stdout:2289: snap_create 2024-09-11T03:46:59.172 INFO:tasks.rados.rados.2.smithi175.stdout:2286: finishing write tid 3 to smithi17557870-4 2024-09-11T03:46:59.172 INFO:tasks.rados.rados.2.smithi175.stdout:2286: finishing write tid 4 to smithi17557870-4 2024-09-11T03:46:59.172 INFO:tasks.rados.rados.2.smithi175.stdout:2286: finishing write tid 5 to smithi17557870-4 2024-09-11T03:46:59.172 INFO:tasks.rados.rados.2.smithi175.stdout:2286: oid 4 updating version 0 to 830 2024-09-11T03:46:59.172 INFO:tasks.rados.rados.2.smithi175.stdout:2286: oid 4 updating version 830 to 831 2024-09-11T03:46:59.172 INFO:tasks.rados.rados.2.smithi175.stdout:2286: oid 4 updating version 831 to 832 2024-09-11T03:46:59.172 INFO:tasks.rados.rados.2.smithi175.stdout:2286: oid 4 updating version 832 to 833 2024-09-11T03:46:59.172 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 833 (ObjNum 789 snap 249 seq_num 789) dirty exists 2024-09-11T03:46:59.172 INFO:tasks.rados.rados.2.smithi175.stdout:2286: left oid 4 (ObjNum 789 snap 249 seq_num 789) 2024-09-11T03:46:59.172 INFO:tasks.rados.rados.2.smithi175.stdout:2287: finishing write tid 1 to smithi17557870-26 2024-09-11T03:46:59.172 INFO:tasks.rados.rados.2.smithi175.stdout:2287: finishing write tid 2 to smithi17557870-26 2024-09-11T03:46:59.174 INFO:tasks.rados.rados.2.smithi175.stdout:2287: finishing write tid 3 to smithi17557870-26 2024-09-11T03:46:59.174 INFO:tasks.rados.rados.2.smithi175.stdout:2287: finishing write tid 4 to smithi17557870-26 2024-09-11T03:46:59.174 INFO:tasks.rados.rados.2.smithi175.stdout:2287: finishing write tid 5 to smithi17557870-26 2024-09-11T03:46:59.174 INFO:tasks.rados.rados.2.smithi175.stdout:2287: oid 26 updating version 0 to 698 2024-09-11T03:46:59.174 INFO:tasks.rados.rados.2.smithi175.stdout:2287: oid 26 updating version 698 to 699 2024-09-11T03:46:59.174 INFO:tasks.rados.rados.2.smithi175.stdout:2287: oid 26 version 699 is already newer than 697 2024-09-11T03:46:59.174 INFO:tasks.rados.rados.2.smithi175.stdout:2287: oid 26 updating version 699 to 700 2024-09-11T03:46:59.174 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 700 (ObjNum 790 snap 249 seq_num 790) dirty exists 2024-09-11T03:46:59.174 INFO:tasks.rados.rados.2.smithi175.stdout:2287: left oid 26 (ObjNum 790 snap 249 seq_num 790) 2024-09-11T03:46:59.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:59.274 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:59.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:59.275 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:59.280 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:59.282 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:46:59.376 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:59.397 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:59.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:59.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:59.499 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:46:59.521 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:46:59.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:59.522 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:46:59.623 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:46:59.644 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:59.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:59.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:46:59.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:46:59.772 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:46:59.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:59.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:46:59.840 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:46:59.841 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:46:59.845 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:46:59.845 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:46:59.864 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:46:59.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:46:59.874 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:46:59.878 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:46:59.878 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:46:59.894 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:46:59.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:46:59.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:46:59.896 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:46:59.896 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:46:59.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:46:59.913 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:46:59.930 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:46:59.931 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:46:59.947 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:46:59.948 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:46:59.965 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:46:59.965 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:46:59.985 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:46:59.985 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:46:59.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:59 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:46:59.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:59 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:46:59.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:59 smithi084 ceph-mon[27254]: pgmap v1056: 67 pgs: 67 active+clean; 101 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.9 MiB/s wr, 38 op/s 2024-09-11T03:46:59.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:59 smithi084 ceph-mon[27254]: osdmap e601: 12 total, 12 up, 12 in 2024-09-11T03:46:59.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:59 smithi084 ceph-mon[27254]: 2.5 scrub starts 2024-09-11T03:46:59.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:59 smithi084 ceph-mon[27254]: 2.5 scrub ok 2024-09-11T03:46:59.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:59 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3155270664' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:46:59.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:46:59 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1761008279' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:46:59.996 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:00.001 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:47:00.001 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:47:00.019 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:47:00.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:47:00.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:00.020 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:00.020 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:00.121 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:00.143 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:00.144 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:00.161 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T03:47:00.164 INFO:tasks.rados.rados.2.smithi175.stdout:2286: done (2 left) 2024-09-11T03:47:00.164 INFO:tasks.rados.rados.2.smithi175.stdout:2287: done (1 left) 2024-09-11T03:47:00.165 INFO:tasks.rados.rados.2.smithi175.stdout:2289: done (0 left) 2024-09-11T03:47:00.165 INFO:tasks.rados.rados.2.smithi175.stdout:2290: snap_create 2024-09-11T03:47:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:59 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:47:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:59 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:47:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:59 smithi175 ceph-mon[30894]: pgmap v1056: 67 pgs: 67 active+clean; 101 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.9 MiB/s wr, 38 op/s 2024-09-11T03:47:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:59 smithi175 ceph-mon[30894]: osdmap e601: 12 total, 12 up, 12 in 2024-09-11T03:47:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:59 smithi175 ceph-mon[30894]: 2.5 scrub starts 2024-09-11T03:47:00.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:59 smithi175 ceph-mon[30894]: 2.5 scrub ok 2024-09-11T03:47:00.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:59 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3155270664' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:47:00.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:46:59 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1761008279' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:00.245 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:00.267 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:00.268 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:00.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:00.396 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:00.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:00.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:59 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:47:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:59 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:47:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:59 smithi121 ceph-mon[30729]: pgmap v1056: 67 pgs: 67 active+clean; 101 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.9 MiB/s wr, 38 op/s 2024-09-11T03:47:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:59 smithi121 ceph-mon[30729]: osdmap e601: 12 total, 12 up, 12 in 2024-09-11T03:47:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:59 smithi121 ceph-mon[30729]: 2.5 scrub starts 2024-09-11T03:47:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:59 smithi121 ceph-mon[30729]: 2.5 scrub ok 2024-09-11T03:47:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:59 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3155270664' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:47:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:46:59 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1761008279' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:00.498 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:00.520 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:00.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:00.521 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:00.622 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:00.644 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:00.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:00.645 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:00.746 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:00.767 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:00.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:00.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:00.869 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:00.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:00.893 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:00.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:00.994 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:01.017 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:01.018 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:01.077 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:00 smithi084 ceph-mon[27254]: 1.4 scrub starts 2024-09-11T03:47:01.077 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:00 smithi084 ceph-mon[27254]: 1.4 scrub ok 2024-09-11T03:47:01.077 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:00 smithi084 ceph-mon[27254]: 1.6 scrub starts 2024-09-11T03:47:01.077 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:00 smithi084 ceph-mon[27254]: 1.6 scrub ok 2024-09-11T03:47:01.077 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:00 smithi084 ceph-mon[27254]: 1.11 scrub starts 2024-09-11T03:47:01.077 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:00 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3155270664' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:47:01.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:00 smithi084 ceph-mon[27254]: 1.11 scrub ok 2024-09-11T03:47:01.078 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:00 smithi084 ceph-mon[27254]: osdmap e602: 12 total, 12 up, 12 in 2024-09-11T03:47:01.119 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:01.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:01.142 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:01.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:01.175 INFO:tasks.rados.rados.2.smithi175.stdout:2290: done (0 left) 2024-09-11T03:47:01.175 INFO:tasks.rados.rados.2.smithi175.stdout:2291: rollback oid 34 current snap is 251 2024-09-11T03:47:01.175 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 34 to 249 2024-09-11T03:47:01.175 INFO:tasks.rados.rados.2.smithi175.stdout:2292: snap_create 2024-09-11T03:47:01.175 INFO:tasks.rados.rados.2.smithi175.stdout:2291: finishing rollback tid 1 to smithi17557870-34 2024-09-11T03:47:01.175 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 0 (ObjNum 289 snap 0 seq_num 1746265552) dirty dne 2024-09-11T03:47:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:00 smithi175 ceph-mon[30894]: 1.4 scrub starts 2024-09-11T03:47:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:00 smithi175 ceph-mon[30894]: 1.4 scrub ok 2024-09-11T03:47:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:00 smithi175 ceph-mon[30894]: 1.6 scrub starts 2024-09-11T03:47:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:00 smithi175 ceph-mon[30894]: 1.6 scrub ok 2024-09-11T03:47:01.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:00 smithi175 ceph-mon[30894]: 1.11 scrub starts 2024-09-11T03:47:01.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:00 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3155270664' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:47:01.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:00 smithi175 ceph-mon[30894]: 1.11 scrub ok 2024-09-11T03:47:01.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:00 smithi175 ceph-mon[30894]: osdmap e602: 12 total, 12 up, 12 in 2024-09-11T03:47:01.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:01.269 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:01.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:01.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:01.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:01.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:01.398 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:01.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:00 smithi121 ceph-mon[30729]: 1.4 scrub starts 2024-09-11T03:47:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:00 smithi121 ceph-mon[30729]: 1.4 scrub ok 2024-09-11T03:47:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:00 smithi121 ceph-mon[30729]: 1.6 scrub starts 2024-09-11T03:47:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:00 smithi121 ceph-mon[30729]: 1.6 scrub ok 2024-09-11T03:47:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:00 smithi121 ceph-mon[30729]: 1.11 scrub starts 2024-09-11T03:47:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:00 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3155270664' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:47:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:00 smithi121 ceph-mon[30729]: 1.11 scrub ok 2024-09-11T03:47:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:00 smithi121 ceph-mon[30729]: osdmap e602: 12 total, 12 up, 12 in 2024-09-11T03:47:01.498 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:01.520 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:01.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:01.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:01.617 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:47:01.618 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:47:01.622 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:01.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:01.646 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:01.646 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:01.747 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:01.768 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:01.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:01.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:01.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:01.895 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:01.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:01.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:01.968 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:47:01.977 INFO:tasks.rados.rados.2.smithi175.stdout:2291: done (1 left) 2024-09-11T03:47:01.977 INFO:tasks.rados.rados.2.smithi175.stdout:2292: done (0 left) 2024-09-11T03:47:01.977 INFO:tasks.rados.rados.2.smithi175.stdout:2293: rollback oid 49 current snap is 252 2024-09-11T03:47:01.977 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 49 to 249 2024-09-11T03:47:01.977 INFO:tasks.rados.rados.2.smithi175.stdout:2294: read oid 16 snap -1 2024-09-11T03:47:01.977 INFO:tasks.rados.rados.2.smithi175.stdout:2294: expect (ObjNum 780 snap 249 seq_num 780) 2024-09-11T03:47:01.977 INFO:tasks.rados.rados.2.smithi175.stdout:2295: write (excl) oid 13 current snap is 252 2024-09-11T03:47:01.977 INFO:tasks.rados.rados.2.smithi175.stdout:2295: seq_num 791 ranges {494337=422671,1387492=553264,2688999=445221} 2024-09-11T03:47:01.982 INFO:tasks.rados.rados.2.smithi175.stdout:2295: writing smithi17557870-13 from 494337 to 917008 tid 1 2024-09-11T03:47:01.988 INFO:tasks.rados.rados.2.smithi175.stdout:2295: writing smithi17557870-13 from 1387492 to 1940756 tid 2 2024-09-11T03:47:01.995 INFO:tasks.rados.rados.2.smithi175.stdout:2295: writing smithi17557870-13 from 2688999 to 3134220 tid 3 2024-09-11T03:47:01.995 INFO:tasks.rados.rados.2.smithi175.stdout:2296: snap_remove snap 250 2024-09-11T03:47:01.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:02.017 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:02.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:02.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:02.119 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:02.140 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:02.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:02.242 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:02.263 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:02.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:02.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:02.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:01 smithi084 ceph-mon[27254]: 1.1e scrub starts 2024-09-11T03:47:02.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:01 smithi084 ceph-mon[27254]: 1.1e scrub ok 2024-09-11T03:47:02.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:01 smithi084 ceph-mon[27254]: 2.16 scrub starts 2024-09-11T03:47:02.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:01 smithi084 ceph-mon[27254]: 2.16 scrub ok 2024-09-11T03:47:02.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:01 smithi084 ceph-mon[27254]: pgmap v1059: 67 pgs: 67 active+clean; 101 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.7 MiB/s wr, 20 op/s 2024-09-11T03:47:02.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:01 smithi084 ceph-mon[27254]: osdmap e603: 12 total, 12 up, 12 in 2024-09-11T03:47:02.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4048304256' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:47:02.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:01 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:02.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4048304256' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:47:02.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:01 smithi084 ceph-mon[27254]: osdmap e604: 12 total, 12 up, 12 in 2024-09-11T03:47:02.366 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:02.377 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-11T03:47:02.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:02.388 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:02.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:01 smithi121 ceph-mon[30729]: 1.1e scrub starts 2024-09-11T03:47:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:01 smithi121 ceph-mon[30729]: 1.1e scrub ok 2024-09-11T03:47:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:01 smithi121 ceph-mon[30729]: 2.16 scrub starts 2024-09-11T03:47:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:01 smithi121 ceph-mon[30729]: 2.16 scrub ok 2024-09-11T03:47:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:01 smithi121 ceph-mon[30729]: pgmap v1059: 67 pgs: 67 active+clean; 101 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.7 MiB/s wr, 20 op/s 2024-09-11T03:47:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:01 smithi121 ceph-mon[30729]: osdmap e603: 12 total, 12 up, 12 in 2024-09-11T03:47:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4048304256' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:47:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:01 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4048304256' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:47:02.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:01 smithi121 ceph-mon[30729]: osdmap e604: 12 total, 12 up, 12 in 2024-09-11T03:47:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:01 smithi175 ceph-mon[30894]: 1.1e scrub starts 2024-09-11T03:47:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:01 smithi175 ceph-mon[30894]: 1.1e scrub ok 2024-09-11T03:47:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:01 smithi175 ceph-mon[30894]: 2.16 scrub starts 2024-09-11T03:47:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:01 smithi175 ceph-mon[30894]: 2.16 scrub ok 2024-09-11T03:47:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:01 smithi175 ceph-mon[30894]: pgmap v1059: 67 pgs: 67 active+clean; 101 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.7 MiB/s wr, 20 op/s 2024-09-11T03:47:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:01 smithi175 ceph-mon[30894]: osdmap e603: 12 total, 12 up, 12 in 2024-09-11T03:47:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4048304256' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:47:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:01 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4048304256' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:47:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:01 smithi175 ceph-mon[30894]: osdmap e604: 12 total, 12 up, 12 in 2024-09-11T03:47:02.489 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:02.515 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:02.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:02.516 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:02.616 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:02.642 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:02.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:02.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:02.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:02.769 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:02.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:02.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:02.795 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T03:47:02.826 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:47:02.870 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:02.903 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:02.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:02.986 INFO:tasks.rados.rados.2.smithi175.stdout:2296: done (3 left) 2024-09-11T03:47:02.987 INFO:tasks.rados.rados.2.smithi175.stdout:2297: write oid 6 current snap is 252 2024-09-11T03:47:02.987 INFO:tasks.rados.rados.2.smithi175.stdout:2297: seq_num 792 ranges {511144=497920,1445974=413275,2316422=656493} 2024-09-11T03:47:02.992 INFO:tasks.rados.rados.2.smithi175.stdout:2297: writing smithi17557870-6 from 511144 to 1009064 tid 1 2024-09-11T03:47:02.996 INFO:tasks.rados.rados.2.smithi175.stdout:2297: writing smithi17557870-6 from 1445974 to 1859249 tid 2 2024-09-11T03:47:03.002 INFO:tasks.rados.rados.2.smithi175.stdout:2297: writing smithi17557870-6 from 2316422 to 2972915 tid 3 2024-09-11T03:47:03.003 INFO:tasks.rados.rados.2.smithi175.stdout:2298: write oid 48 current snap is 252 2024-09-11T03:47:03.003 INFO:tasks.rados.rados.2.smithi175.stdout:2298: seq_num 793 ranges {527951=573169,1504456=556933,2743845=467765} 2024-09-11T03:47:03.004 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:03.008 INFO:tasks.rados.rados.2.smithi175.stdout:2298: writing smithi17557870-48 from 527951 to 1101120 tid 1 2024-09-11T03:47:03.014 INFO:tasks.rados.rados.2.smithi175.stdout:2298: writing smithi17557870-48 from 1504456 to 2061389 tid 2 2024-09-11T03:47:03.021 INFO:tasks.rados.rados.2.smithi175.stdout:2298: writing smithi17557870-48 from 2743845 to 3211610 tid 3 2024-09-11T03:47:03.021 INFO:tasks.rados.rados.2.smithi175.stdout:2299: write oid 4 current snap is 252 2024-09-11T03:47:03.021 INFO:tasks.rados.rados.2.smithi175.stdout:2299: seq_num 794 ranges {544758=648418,1846585=416944,2771268=573490} 2024-09-11T03:47:03.027 INFO:tasks.rados.rados.2.smithi175.stdout:2299: writing smithi17557870-4 from 544758 to 1193176 tid 1 2024-09-11T03:47:03.029 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:03.030 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:03.033 INFO:tasks.rados.rados.2.smithi175.stdout:2299: writing smithi17557870-4 from 1846585 to 2263529 tid 2 2024-09-11T03:47:03.039 INFO:tasks.rados.rados.2.smithi175.stdout:2299: writing smithi17557870-4 from 2771268 to 3344758 tid 3 2024-09-11T03:47:03.039 INFO:tasks.rados.rados.2.smithi175.stdout:2300: snap_create 2024-09-11T03:47:03.039 INFO:tasks.rados.rados.2.smithi175.stdout:2293: finishing rollback tid 0 to smithi17557870-49 2024-09-11T03:47:03.039 INFO:tasks.rados.rados.2.smithi175.stdout:2293: finishing rollback tid 1 to smithi17557870-49 2024-09-11T03:47:03.040 INFO:tasks.rados.rados.2.smithi175.stdout:2293: finishing rollback tid 2 to smithi17557870-49 2024-09-11T03:47:03.040 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 696 (ObjNum 773 snap 248 seq_num 773) dirty exists 2024-09-11T03:47:03.040 INFO:tasks.rados.rados.2.smithi175.stdout:2295: finishing write tid 1 to smithi17557870-13 2024-09-11T03:47:03.040 INFO:tasks.rados.rados.2.smithi175.stdout:2294: expect (ObjNum 780 snap 249 seq_num 780) 2024-09-11T03:47:03.131 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:03.156 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:03.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:03.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:03.185 INFO:tasks.rados.rados.2.smithi175.stdout:2295: finishing write tid 2 to smithi17557870-13 2024-09-11T03:47:03.185 INFO:tasks.rados.rados.2.smithi175.stdout:2295: finishing write tid 3 to smithi17557870-13 2024-09-11T03:47:03.185 INFO:tasks.rados.rados.2.smithi175.stdout:2295: finishing write tid 4 to smithi17557870-13 2024-09-11T03:47:03.185 INFO:tasks.rados.rados.2.smithi175.stdout:2295: finishing write tid 5 to smithi17557870-13 2024-09-11T03:47:03.185 INFO:tasks.rados.rados.2.smithi175.stdout:2295: oid 13 updating version 0 to 875 2024-09-11T03:47:03.185 INFO:tasks.rados.rados.2.smithi175.stdout:2295: oid 13 version 875 is already newer than 874 2024-09-11T03:47:03.185 INFO:tasks.rados.rados.2.smithi175.stdout:2295: oid 13 updating version 875 to 877 2024-09-11T03:47:03.185 INFO:tasks.rados.rados.2.smithi175.stdout:2295: oid 13 version 877 is already newer than 876 2024-09-11T03:47:03.186 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 877 (ObjNum 791 snap 252 seq_num 791) dirty exists 2024-09-11T03:47:03.186 INFO:tasks.rados.rados.2.smithi175.stdout:2295: left oid 13 (ObjNum 791 snap 252 seq_num 791) 2024-09-11T03:47:03.186 INFO:tasks.rados.rados.2.smithi175.stdout:2297: finishing write tid 1 to smithi17557870-6 2024-09-11T03:47:03.186 INFO:tasks.rados.rados.2.smithi175.stdout:2297: finishing write tid 2 to smithi17557870-6 2024-09-11T03:47:03.186 INFO:tasks.rados.rados.2.smithi175.stdout:2297: finishing write tid 3 to smithi17557870-6 2024-09-11T03:47:03.186 INFO:tasks.rados.rados.2.smithi175.stdout:2297: finishing write tid 4 to smithi17557870-6 2024-09-11T03:47:03.186 INFO:tasks.rados.rados.2.smithi175.stdout:2297: finishing write tid 5 to smithi17557870-6 2024-09-11T03:47:03.186 INFO:tasks.rados.rados.2.smithi175.stdout:2297: oid 6 updating version 0 to 477 2024-09-11T03:47:03.186 INFO:tasks.rados.rados.2.smithi175.stdout:2297: oid 6 version 477 is already newer than 476 2024-09-11T03:47:03.187 INFO:tasks.rados.rados.2.smithi175.stdout:2297: oid 6 version 477 is already newer than 475 2024-09-11T03:47:03.187 INFO:tasks.rados.rados.2.smithi175.stdout:2297: oid 6 updating version 477 to 478 2024-09-11T03:47:03.187 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 478 (ObjNum 792 snap 252 seq_num 792) dirty exists 2024-09-11T03:47:03.187 INFO:tasks.rados.rados.2.smithi175.stdout:2297: left oid 6 (ObjNum 792 snap 252 seq_num 792) 2024-09-11T03:47:03.187 INFO:tasks.rados.rados.2.smithi175.stdout:2298: finishing write tid 1 to smithi17557870-48 2024-09-11T03:47:03.187 INFO:tasks.rados.rados.2.smithi175.stdout:2298: finishing write tid 2 to smithi17557870-48 2024-09-11T03:47:03.187 INFO:tasks.rados.rados.2.smithi175.stdout:2298: finishing write tid 3 to smithi17557870-48 2024-09-11T03:47:03.187 INFO:tasks.rados.rados.2.smithi175.stdout:2298: finishing write tid 4 to smithi17557870-48 2024-09-11T03:47:03.187 INFO:tasks.rados.rados.2.smithi175.stdout:2298: finishing write tid 5 to smithi17557870-48 2024-09-11T03:47:03.187 INFO:tasks.rados.rados.2.smithi175.stdout:2298: oid 48 updating version 0 to 835 2024-09-11T03:47:03.187 INFO:tasks.rados.rados.2.smithi175.stdout:2298: oid 48 updating version 835 to 836 2024-09-11T03:47:03.187 INFO:tasks.rados.rados.2.smithi175.stdout:2298: oid 48 version 836 is already newer than 834 2024-09-11T03:47:03.187 INFO:tasks.rados.rados.2.smithi175.stdout:2298: oid 48 updating version 836 to 837 2024-09-11T03:47:03.187 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 837 (ObjNum 793 snap 252 seq_num 793) dirty exists 2024-09-11T03:47:03.188 INFO:tasks.rados.rados.2.smithi175.stdout:2298: left oid 48 (ObjNum 793 snap 252 seq_num 793) 2024-09-11T03:47:03.188 INFO:tasks.rados.rados.2.smithi175.stdout:2299: finishing write tid 1 to smithi17557870-4 2024-09-11T03:47:03.188 INFO:tasks.rados.rados.2.smithi175.stdout:2299: finishing write tid 2 to smithi17557870-4 2024-09-11T03:47:03.188 INFO:tasks.rados.rados.2.smithi175.stdout:2299: finishing write tid 3 to smithi17557870-4 2024-09-11T03:47:03.188 INFO:tasks.rados.rados.2.smithi175.stdout:2299: finishing write tid 4 to smithi17557870-4 2024-09-11T03:47:03.188 INFO:tasks.rados.rados.2.smithi175.stdout:2299: finishing write tid 5 to smithi17557870-4 2024-09-11T03:47:03.188 INFO:tasks.rados.rados.2.smithi175.stdout:2299: oid 4 updating version 0 to 839 2024-09-11T03:47:03.188 INFO:tasks.rados.rados.2.smithi175.stdout:2299: oid 4 updating version 839 to 840 2024-09-11T03:47:03.189 INFO:tasks.rados.rados.2.smithi175.stdout:2299: oid 4 updating version 840 to 841 2024-09-11T03:47:03.189 INFO:tasks.rados.rados.2.smithi175.stdout:2299: oid 4 version 841 is already newer than 838 2024-09-11T03:47:03.189 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 841 (ObjNum 794 snap 252 seq_num 794) dirty exists 2024-09-11T03:47:03.189 INFO:tasks.rados.rados.2.smithi175.stdout:2299: left oid 4 (ObjNum 794 snap 252 seq_num 794) 2024-09-11T03:47:03.259 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:03.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:03.282 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:03.282 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:03.383 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:03.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:03.405 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:03.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:03.494 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:03.506 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:03.527 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:03.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:03.528 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:03.530 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:03.628 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:03.650 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:03.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:03.651 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:03.752 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:03.774 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:03.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:03.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:03.876 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:03.902 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:03.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:03.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:03.993 INFO:tasks.rados.rados.2.smithi175.stdout:2293: done (6 left) 2024-09-11T03:47:03.993 INFO:tasks.rados.rados.2.smithi175.stdout:2294: done (5 left) 2024-09-11T03:47:03.993 INFO:tasks.rados.rados.2.smithi175.stdout:2295: done (4 left) 2024-09-11T03:47:03.993 INFO:tasks.rados.rados.2.smithi175.stdout:2297: done (3 left) 2024-09-11T03:47:03.993 INFO:tasks.rados.rados.2.smithi175.stdout:2298: done (2 left) 2024-09-11T03:47:03.993 INFO:tasks.rados.rados.2.smithi175.stdout:2299: done (1 left) 2024-09-11T03:47:03.993 INFO:tasks.rados.rados.2.smithi175.stdout:2300: done (0 left) 2024-09-11T03:47:03.994 INFO:tasks.rados.rados.2.smithi175.stdout:2301: read oid 18 snap -1 2024-09-11T03:47:03.994 INFO:tasks.rados.rados.2.smithi175.stdout:2301: expect deleted 2024-09-11T03:47:03.994 INFO:tasks.rados.rados.2.smithi175.stdout:2302: copy_from oid 39 from oid 37 current snap is 253 2024-09-11T03:47:03.994 INFO:tasks.rados.rados.2.smithi175.stdout:2303: snap_create 2024-09-11T03:47:04.004 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:04.005 INFO:tasks.rados.rados.2.smithi175.stdout:2302: finishing copy_from racing read to smithi17557870-39 2024-09-11T03:47:04.008 INFO:tasks.rados.rados.2.smithi175.stdout:2302: finishing copy_from to smithi17557870-39 2024-09-11T03:47:04.008 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 806 (ObjNum 738 snap 243 seq_num 738) dirty exists 2024-09-11T03:47:04.026 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:03 smithi084 ceph-mon[27254]: pgmap v1062: 67 pgs: 67 active+clean; 101 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.1 MiB/s wr, 19 op/s 2024-09-11T03:47:04.027 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:03 smithi084 ceph-mon[27254]: osdmap e605: 12 total, 12 up, 12 in 2024-09-11T03:47:04.028 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:04.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:04.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:04.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:04.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:04.153 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:04.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:04.254 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:04.276 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:04.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:04.378 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:04.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:04.400 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:04.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:04.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:03 smithi121 ceph-mon[30729]: pgmap v1062: 67 pgs: 67 active+clean; 101 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.1 MiB/s wr, 19 op/s 2024-09-11T03:47:04.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:03 smithi121 ceph-mon[30729]: osdmap e605: 12 total, 12 up, 12 in 2024-09-11T03:47:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:03 smithi175 ceph-mon[30894]: pgmap v1062: 67 pgs: 67 active+clean; 101 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.1 MiB/s wr, 19 op/s 2024-09-11T03:47:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:03 smithi175 ceph-mon[30894]: osdmap e605: 12 total, 12 up, 12 in 2024-09-11T03:47:04.501 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:04.521 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:04.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:04.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:04.579 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:47:04.579 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:47:04.585 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:04.595 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:47:04.596 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:47:04.618 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:47:04.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:47:04.623 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:04.635 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:47:04.635 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:47:04.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:04.645 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:04.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:04.652 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:47:04.653 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:47:04.670 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:47:04.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:47:04.687 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:47:04.687 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:47:04.704 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:47:04.705 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:47:04.721 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:47:04.721 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:47:04.737 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:47:04.738 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:47:04.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:04.755 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:47:04.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:47:04.772 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:04.773 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:47:04.773 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:47:04.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:04.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:04.875 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:04.899 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:04.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:04.900 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:04.994 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2024-09-11T03:47:05.001 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:05.003 INFO:tasks.rados.rados.2.smithi175.stdout:2301: done (2 left) 2024-09-11T03:47:05.003 INFO:tasks.rados.rados.2.smithi175.stdout:2302: done (1 left) 2024-09-11T03:47:05.003 INFO:tasks.rados.rados.2.smithi175.stdout:2303: done (0 left) 2024-09-11T03:47:05.003 INFO:tasks.rados.rados.2.smithi175.stdout:2304: rollback oid 11 current snap is 254 2024-09-11T03:47:05.003 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 11 to 252 2024-09-11T03:47:05.003 INFO:tasks.rados.rados.2.smithi175.stdout:2305: read oid 4 snap -1 2024-09-11T03:47:05.004 INFO:tasks.rados.rados.2.smithi175.stdout:2305: expect (ObjNum 794 snap 252 seq_num 794) 2024-09-11T03:47:05.004 INFO:tasks.rados.rados.2.smithi175.stdout:2306: write (excl) oid 38 current snap is 254 2024-09-11T03:47:05.004 INFO:tasks.rados.rados.2.smithi175.stdout:2306: seq_num 795 ranges {561565=723667,1905067=560602,3082338=279227} 2024-09-11T03:47:05.010 INFO:tasks.rados.rados.2.smithi175.stdout:2306: writing smithi17557870-38 from 561565 to 1285232 tid 1 2024-09-11T03:47:05.018 INFO:tasks.rados.rados.2.smithi175.stdout:2306: writing smithi17557870-38 from 1905067 to 2465669 tid 2 2024-09-11T03:47:05.023 INFO:tasks.rados.rados.2.smithi175.stdout:2306: writing smithi17557870-38 from 3082338 to 3361565 tid 3 2024-09-11T03:47:05.023 INFO:tasks.rados.rados.2.smithi175.stdout:2307: snap_remove snap 249 2024-09-11T03:47:05.023 INFO:tasks.rados.rados.2.smithi175.stdout:2304: finishing rollback tid 0 to smithi17557870-11 2024-09-11T03:47:05.023 INFO:tasks.rados.rados.2.smithi175.stdout:2304: finishing rollback tid 1 to smithi17557870-11 2024-09-11T03:47:05.023 INFO:tasks.rados.rados.2.smithi175.stdout:2304: finishing rollback tid 2 to smithi17557870-11 2024-09-11T03:47:05.023 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 600 (ObjNum 609 snap 191 seq_num 609) dirty exists 2024-09-11T03:47:05.024 INFO:tasks.rados.rados.2.smithi175.stdout:2306: finishing write tid 1 to smithi17557870-38 2024-09-11T03:47:05.024 INFO:tasks.rados.rados.2.smithi175.stdout:2305: expect (ObjNum 794 snap 252 seq_num 794) 2024-09-11T03:47:05.026 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:05.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:05.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:05.128 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:04 smithi084 ceph-mon[27254]: osdmap e606: 12 total, 12 up, 12 in 2024-09-11T03:47:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:04 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3068847600' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:47:05.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:04 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2688266243' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:05.150 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:05.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:05.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:05.166 INFO:tasks.rados.rados.2.smithi175.stdout:2306: finishing write tid 2 to smithi17557870-38 2024-09-11T03:47:05.166 INFO:tasks.rados.rados.2.smithi175.stdout:2306: finishing write tid 3 to smithi17557870-38 2024-09-11T03:47:05.166 INFO:tasks.rados.rados.2.smithi175.stdout:2306: finishing write tid 4 to smithi17557870-38 2024-09-11T03:47:05.166 INFO:tasks.rados.rados.2.smithi175.stdout:2306: finishing write tid 5 to smithi17557870-38 2024-09-11T03:47:05.166 INFO:tasks.rados.rados.2.smithi175.stdout:2306: oid 38 updating version 0 to 878 2024-09-11T03:47:05.167 INFO:tasks.rados.rados.2.smithi175.stdout:2306: oid 38 updating version 878 to 879 2024-09-11T03:47:05.167 INFO:tasks.rados.rados.2.smithi175.stdout:2306: oid 38 updating version 879 to 880 2024-09-11T03:47:05.167 INFO:tasks.rados.rados.2.smithi175.stdout:2306: oid 38 updating version 880 to 881 2024-09-11T03:47:05.167 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 881 (ObjNum 795 snap 254 seq_num 795) dirty exists 2024-09-11T03:47:05.167 INFO:tasks.rados.rados.2.smithi175.stdout:2306: left oid 38 (ObjNum 795 snap 254 seq_num 795) 2024-09-11T03:47:05.251 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:05.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:05.275 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:05.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:05.376 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:05.398 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:05.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:05.399 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:05.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:04 smithi121 ceph-mon[30729]: osdmap e606: 12 total, 12 up, 12 in 2024-09-11T03:47:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:04 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3068847600' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:47:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:04 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2688266243' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:04 smithi175 ceph-mon[30894]: osdmap e606: 12 total, 12 up, 12 in 2024-09-11T03:47:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:04 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3068847600' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:47:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:04 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2688266243' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:05.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:47:05.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:05.526 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:05.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:05.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:05.628 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:05.651 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:05.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:05.652 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:05.731 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 8 1 2024-09-11T03:47:05.753 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:05.775 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:05.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:05.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:05.877 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:05.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:05.899 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:05.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:05.999 INFO:tasks.rados.rados.2.smithi175.stdout:2304: done (3 left) 2024-09-11T03:47:05.999 INFO:tasks.rados.rados.2.smithi175.stdout:2305: done (2 left) 2024-09-11T03:47:05.999 INFO:tasks.rados.rados.2.smithi175.stdout:2306: done (1 left) 2024-09-11T03:47:06.000 INFO:tasks.rados.rados.2.smithi175.stdout:2307: done (0 left) 2024-09-11T03:47:06.000 INFO:tasks.rados.rados.2.smithi175.stdout:2308: write (excl) oid 5 current snap is 254 2024-09-11T03:47:06.000 INFO:tasks.rados.rados.2.smithi175.stdout:2308: seq_num 796 ranges {578372=798916,1963549=420613,2826114=417934} 2024-09-11T03:47:06.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:06.007 INFO:tasks.rados.rados.2.smithi175.stdout:2308: writing smithi17557870-5 from 578372 to 1377288 tid 1 2024-09-11T03:47:06.013 INFO:tasks.rados.rados.2.smithi175.stdout:2308: writing smithi17557870-5 from 1963549 to 2384162 tid 2 2024-09-11T03:47:06.018 INFO:tasks.rados.rados.2.smithi175.stdout:2308: writing smithi17557870-5 from 2826114 to 3244048 tid 3 2024-09-11T03:47:06.018 INFO:tasks.rados.rados.2.smithi175.stdout:2309: delete oid 33 current snap is 254 2024-09-11T03:47:06.018 INFO:tasks.rados.rados.2.smithi175.stdout:2308: finishing write tid 1 to smithi17557870-5 2024-09-11T03:47:06.018 INFO:tasks.rados.rados.2.smithi175.stdout:2308: finishing write tid 2 to smithi17557870-5 2024-09-11T03:47:06.020 INFO:tasks.rados.rados.2.smithi175.stdout:2308: finishing write tid 3 to smithi17557870-5 2024-09-11T03:47:06.020 INFO:tasks.rados.rados.2.smithi175.stdout:2309: done (1 left) 2024-09-11T03:47:06.020 INFO:tasks.rados.rados.2.smithi175.stdout:2310: rollback oid 29 current snap is 254 2024-09-11T03:47:06.021 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 29 to 253 2024-09-11T03:47:06.021 INFO:tasks.rados.rados.2.smithi175.stdout:2311: read oid 40 snap -1 2024-09-11T03:47:06.021 INFO:tasks.rados.rados.2.smithi175.stdout:2311: expect (ObjNum 697 snap 227 seq_num 697) 2024-09-11T03:47:06.021 INFO:tasks.rados.rados.2.smithi175.stdout:2312: copy_from oid 46 from oid 30 current snap is 254 2024-09-11T03:47:06.021 INFO:tasks.rados.rados.2.smithi175.stdout:2308: finishing write tid 4 to smithi17557870-5 2024-09-11T03:47:06.021 INFO:tasks.rados.rados.2.smithi175.stdout:2313: copy_from oid 7 from oid 33 current snap is 254 2024-09-11T03:47:06.021 INFO:tasks.rados.rados.2.smithi175.stdout:2314: write oid 14 current snap is 254 2024-09-11T03:47:06.021 INFO:tasks.rados.rados.2.smithi175.stdout:2314: seq_num 798 ranges {611986=549414,1564160=424282,2764607=440478} 2024-09-11T03:47:06.027 INFO:tasks.rados.rados.2.smithi175.stdout:2314: writing smithi17557870-14 from 611986 to 1161400 tid 1 2024-09-11T03:47:06.028 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:06.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:06.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:06.033 INFO:tasks.rados.rados.2.smithi175.stdout:2314: writing smithi17557870-14 from 1564160 to 1988442 tid 2 2024-09-11T03:47:06.040 INFO:tasks.rados.rados.2.smithi175.stdout:2314: writing smithi17557870-14 from 2764607 to 3205085 tid 3 2024-09-11T03:47:06.040 INFO:tasks.rados.rados.2.smithi175.stdout:2315: write (excl) oid 17 current snap is 254 2024-09-11T03:47:06.040 INFO:tasks.rados.rados.2.smithi175.stdout:2315: seq_num 799 ranges {628793=741016,2138995=567940,3192030=236763} 2024-09-11T03:47:06.047 INFO:tasks.rados.rados.2.smithi175.stdout:2315: writing smithi17557870-17 from 628793 to 1369809 tid 1 2024-09-11T03:47:06.054 INFO:tasks.rados.rados.2.smithi175.stdout:2315: writing smithi17557870-17 from 2138995 to 2706935 tid 2 2024-09-11T03:47:06.059 INFO:tasks.rados.rados.2.smithi175.stdout:2315: writing smithi17557870-17 from 3192030 to 3428793 tid 3 2024-09-11T03:47:06.059 INFO:tasks.rados.rados.2.smithi175.stdout:2316: snap_create 2024-09-11T03:47:06.059 INFO:tasks.rados.rados.2.smithi175.stdout:2308: finishing write tid 5 to smithi17557870-5 2024-09-11T03:47:06.059 INFO:tasks.rados.rados.2.smithi175.stdout:2308: oid 5 updating version 0 to 711 2024-09-11T03:47:06.059 INFO:tasks.rados.rados.2.smithi175.stdout:2308: oid 5 version 711 is already newer than 708 2024-09-11T03:47:06.060 INFO:tasks.rados.rados.2.smithi175.stdout:2308: oid 5 version 711 is already newer than 709 2024-09-11T03:47:06.060 INFO:tasks.rados.rados.2.smithi175.stdout:2308: oid 5 version 711 is already newer than 710 2024-09-11T03:47:06.060 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 711 (ObjNum 796 snap 254 seq_num 796) dirty exists 2024-09-11T03:47:06.060 INFO:tasks.rados.rados.2.smithi175.stdout:2308: left oid 5 (ObjNum 796 snap 254 seq_num 796) 2024-09-11T03:47:06.060 INFO:tasks.rados.rados.2.smithi175.stdout:2310: finishing rollback tid 0 to smithi17557870-29 2024-09-11T03:47:06.060 INFO:tasks.rados.rados.2.smithi175.stdout:2313: finishing copy_from to smithi17557870-7 2024-09-11T03:47:06.060 INFO:tasks.rados.rados.2.smithi175.stdout:2313: got expected ENOENT (src dne) 2024-09-11T03:47:06.060 INFO:tasks.rados.rados.2.smithi175.stdout:2313: finishing copy_from racing read to smithi17557870-7 2024-09-11T03:47:06.060 INFO:tasks.rados.rados.2.smithi175.stdout:2310: finishing rollback tid 1 to smithi17557870-29 2024-09-11T03:47:06.060 INFO:tasks.rados.rados.2.smithi175.stdout:2310: finishing rollback tid 2 to smithi17557870-29 2024-09-11T03:47:06.060 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 884 (ObjNum 736 snap 243 seq_num 736) dirty exists 2024-09-11T03:47:06.060 INFO:tasks.rados.rados.2.smithi175.stdout:2311: expect (ObjNum 697 snap 227 seq_num 697) 2024-09-11T03:47:06.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:06.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:06 smithi084 ceph-mon[27254]: pgmap v1065: 67 pgs: 67 active+clean; 101 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.4 MiB/s wr, 20 op/s 2024-09-11T03:47:06.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:06 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3068847600' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:47:06.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:06 smithi084 ceph-mon[27254]: osdmap e607: 12 total, 12 up, 12 in 2024-09-11T03:47:06.151 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:06.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:06.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:06.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:06.265 INFO:tasks.rados.rados.2.smithi175.stdout:2314: finishing write tid 1 to smithi17557870-14 2024-09-11T03:47:06.265 INFO:tasks.rados.rados.2.smithi175.stdout:2312: finishing copy_from racing read to smithi17557870-46 2024-09-11T03:47:06.265 INFO:tasks.rados.rados.2.smithi175.stdout:2312: finishing copy_from to smithi17557870-46 2024-09-11T03:47:06.265 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 878 (ObjNum 783 snap 249 seq_num 783) dirty exists 2024-09-11T03:47:06.266 INFO:tasks.rados.rados.2.smithi175.stdout:2314: finishing write tid 2 to smithi17557870-14 2024-09-11T03:47:06.266 INFO:tasks.rados.rados.2.smithi175.stdout:2314: finishing write tid 3 to smithi17557870-14 2024-09-11T03:47:06.266 INFO:tasks.rados.rados.2.smithi175.stdout:2314: finishing write tid 4 to smithi17557870-14 2024-09-11T03:47:06.266 INFO:tasks.rados.rados.2.smithi175.stdout:2314: finishing write tid 5 to smithi17557870-14 2024-09-11T03:47:06.266 INFO:tasks.rados.rados.2.smithi175.stdout:2314: oid 14 updating version 0 to 878 2024-09-11T03:47:06.266 INFO:tasks.rados.rados.2.smithi175.stdout:2314: oid 14 updating version 878 to 879 2024-09-11T03:47:06.266 INFO:tasks.rados.rados.2.smithi175.stdout:2314: oid 14 version 879 is already newer than 876 2024-09-11T03:47:06.266 INFO:tasks.rados.rados.2.smithi175.stdout:2314: oid 14 version 879 is already newer than 877 2024-09-11T03:47:06.266 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 879 (ObjNum 798 snap 254 seq_num 798) dirty exists 2024-09-11T03:47:06.266 INFO:tasks.rados.rados.2.smithi175.stdout:2314: left oid 14 (ObjNum 798 snap 254 seq_num 798) 2024-09-11T03:47:06.266 INFO:tasks.rados.rados.2.smithi175.stdout:2315: finishing write tid 1 to smithi17557870-17 2024-09-11T03:47:06.266 INFO:tasks.rados.rados.2.smithi175.stdout:2315: finishing write tid 2 to smithi17557870-17 2024-09-11T03:47:06.266 INFO:tasks.rados.rados.2.smithi175.stdout:2315: finishing write tid 3 to smithi17557870-17 2024-09-11T03:47:06.266 INFO:tasks.rados.rados.2.smithi175.stdout:2315: finishing write tid 4 to smithi17557870-17 2024-09-11T03:47:06.266 INFO:tasks.rados.rados.2.smithi175.stdout:2315: finishing write tid 5 to smithi17557870-17 2024-09-11T03:47:06.267 INFO:tasks.rados.rados.2.smithi175.stdout:2315: oid 17 updating version 0 to 777 2024-09-11T03:47:06.267 INFO:tasks.rados.rados.2.smithi175.stdout:2315: oid 17 updating version 777 to 779 2024-09-11T03:47:06.267 INFO:tasks.rados.rados.2.smithi175.stdout:2315: oid 17 version 779 is already newer than 778 2024-09-11T03:47:06.267 INFO:tasks.rados.rados.2.smithi175.stdout:2315: oid 17 updating version 779 to 780 2024-09-11T03:47:06.267 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 780 (ObjNum 799 snap 254 seq_num 799) dirty exists 2024-09-11T03:47:06.267 INFO:tasks.rados.rados.2.smithi175.stdout:2315: left oid 17 (ObjNum 799 snap 254 seq_num 799) 2024-09-11T03:47:06.279 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:06.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:06.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:06.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:06.407 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:06.408 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:47:06.408 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:47:06.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:47:06.408 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:47:06.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.408 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:752 2.c e138/95 2:3c007163:::smithi17557870-35:head v 138'40, mlcod=135'37)", 2024-09-11T03:47:06.408 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:47:06.408 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 471.39052328700001, 2024-09-11T03:47:06.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084184749999999999, 2024-09-11T03:47:06.409 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.409 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.409 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:06.409 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:47:06.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:47:06.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.015574+0000", 2024-09-11T03:47:06.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8400000000000001e-07 2024-09-11T03:47:06.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021686+0000", 2024-09-11T03:47:06.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061118680000000003 2024-09-11T03:47:06.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021706+0000", 2024-09-11T03:47:06.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9942999999999999e-05 2024-09-11T03:47:06.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021717+0000", 2024-09-11T03:47:06.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.098e-05 2024-09-11T03:47:06.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021806+0000", 2024-09-11T03:47:06.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9197999999999995e-05 2024-09-11T03:47:06.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.021834+0000", 2024-09-11T03:47:06.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8251000000000001e-05 2024-09-11T03:47:06.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:06.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.023931+0000", 2024-09-11T03:47:06.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020966639999999998 2024-09-11T03:47:06.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:15.023992+0000", 2024-09-11T03:47:06.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1387e-05 2024-09-11T03:47:06.417 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.417 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.417 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.417 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1513 2.5 e212/95 2:a6e3bbd9:::smithi17557870-38:head v 212'178, mlcod=203'174)", 2024-09-11T03:47:06.417 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:28.928696+0000", 2024-09-11T03:47:06.417 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 397.47740138, 2024-09-11T03:47:06.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0099738110000000008, 2024-09-11T03:47:06.417 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.417 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.417 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:06.418 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.928696+0000", 2024-09-11T03:47:06.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.928696+0000", 2024-09-11T03:47:06.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.928696+0000", 2024-09-11T03:47:06.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9500000000000001e-07 2024-09-11T03:47:06.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935839+0000", 2024-09-11T03:47:06.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.007143473 2024-09-11T03:47:06.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935850+0000", 2024-09-11T03:47:06.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.026e-05 2024-09-11T03:47:06.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935854+0000", 2024-09-11T03:47:06.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8080000000000004e-06 2024-09-11T03:47:06.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935903+0000", 2024-09-11T03:47:06.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8380000000000001e-05 2024-09-11T03:47:06.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.935917+0000", 2024-09-11T03:47:06.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4192999999999999e-05 2024-09-11T03:47:06.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:06.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.938638+0000", 2024-09-11T03:47:06.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002721298 2024-09-11T03:47:06.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.938669+0000", 2024-09-11T03:47:06.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1204000000000001e-05 2024-09-11T03:47:06.421 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.421 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.422 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.422 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2115 2.5 e268/95 2:a6e3bbd9:::smithi17557870-38:head v 268'256, mlcod=260'252)", 2024-09-11T03:47:06.422 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:24.876744+0000", 2024-09-11T03:47:06.422 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 341.52935271299998, 2024-09-11T03:47:06.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01118442, 2024-09-11T03:47:06.422 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.422 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.422 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:06.422 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.876744+0000", 2024-09-11T03:47:06.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.876744+0000", 2024-09-11T03:47:06.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.876745+0000", 2024-09-11T03:47:06.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8999999999999998e-07 2024-09-11T03:47:06.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.885165+0000", 2024-09-11T03:47:06.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084204060000000001 2024-09-11T03:47:06.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.885185+0000", 2024-09-11T03:47:06.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0432999999999999e-05 2024-09-11T03:47:06.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.885203+0000", 2024-09-11T03:47:06.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7569000000000001e-05 2024-09-11T03:47:06.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.885268+0000", 2024-09-11T03:47:06.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5104999999999994e-05 2024-09-11T03:47:06.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.885289+0000", 2024-09-11T03:47:06.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0800000000000001e-05 2024-09-11T03:47:06.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:06.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.887888+0000", 2024-09-11T03:47:06.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025986999999999998 2024-09-11T03:47:06.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:24.887929+0000", 2024-09-11T03:47:06.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1117e-05 2024-09-11T03:47:06.426 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.426 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.426 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.426 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.8.0:443 2.5 e279/276 2:a6e3bbd9:::smithi17557870-38:5d v 279'283, mlcod=278'279)", 2024-09-11T03:47:06.427 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:35.865183+0000", 2024-09-11T03:47:06.427 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 330.54091447399998, 2024-09-11T03:47:06.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011482019, 2024-09-11T03:47:06.427 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.427 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.427 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:06.427 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865183+0000", 2024-09-11T03:47:06.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865183+0000", 2024-09-11T03:47:06.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865183+0000", 2024-09-11T03:47:06.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0399999999999996e-07 2024-09-11T03:47:06.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865190+0000", 2024-09-11T03:47:06.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9460000000000001e-06 2024-09-11T03:47:06.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865199+0000", 2024-09-11T03:47:06.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7979999999999995e-06 2024-09-11T03:47:06.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865214+0000", 2024-09-11T03:47:06.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5184999999999999e-05 2024-09-11T03:47:06.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865278+0000", 2024-09-11T03:47:06.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4086999999999998e-05 2024-09-11T03:47:06.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.865303+0000", 2024-09-11T03:47:06.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.446e-05 2024-09-11T03:47:06.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:06.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.876628+0000", 2024-09-11T03:47:06.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011325574 2024-09-11T03:47:06.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:35.876665+0000", 2024-09-11T03:47:06.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6464999999999997e-05 2024-09-11T03:47:06.431 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.431 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.431 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.431 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2241 2.14 2:2ef19c88:::smithi17557870-15:58 [read 0~3327819,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:47:06.431 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:36.886222+0000", 2024-09-11T03:47:06.431 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 329.51987550899997, 2024-09-11T03:47:06.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012098761, 2024-09-11T03:47:06.431 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.432 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.432 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:47:06.432 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:06.432 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:06.432 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:06.432 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2241 2024-09-11T03:47:06.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.432 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886222+0000", 2024-09-11T03:47:06.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886222+0000", 2024-09-11T03:47:06.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886223+0000", 2024-09-11T03:47:06.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4529999999999999e-06 2024-09-11T03:47:06.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886228+0000", 2024-09-11T03:47:06.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9869999999999999e-06 2024-09-11T03:47:06.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886239+0000", 2024-09-11T03:47:06.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.08e-05 2024-09-11T03:47:06.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886249+0000", 2024-09-11T03:47:06.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0322e-05 2024-09-11T03:47:06.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886366+0000", 2024-09-11T03:47:06.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000116549 2024-09-11T03:47:06.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:47:06.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886375+0000", 2024-09-11T03:47:06.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3870000000000001e-06 2024-09-11T03:47:06.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.893120+0000", 2024-09-11T03:47:06.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067446340000000002 2024-09-11T03:47:06.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.893275+0000", 2024-09-11T03:47:06.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000155079 2024-09-11T03:47:06.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.898320+0000", 2024-09-11T03:47:06.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0050455500000000002 2024-09-11T03:47:06.436 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.436 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.437 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.437 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2242 2.14 2:2ef19c88:::smithi17557870-15:58 [sparse-read 0~2125189] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:47:06.437 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:36.886426+0000", 2024-09-11T03:47:06.437 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 329.519670561, 2024-09-11T03:47:06.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012666422, 2024-09-11T03:47:06.437 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.437 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.437 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:47:06.437 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:06.437 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:06.438 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:06.438 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2242 2024-09-11T03:47:06.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.438 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886426+0000", 2024-09-11T03:47:06.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886426+0000", 2024-09-11T03:47:06.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886427+0000", 2024-09-11T03:47:06.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2799999999999996e-07 2024-09-11T03:47:06.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886431+0000", 2024-09-11T03:47:06.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7910000000000002e-06 2024-09-11T03:47:06.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886439+0000", 2024-09-11T03:47:06.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.038e-06 2024-09-11T03:47:06.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886445+0000", 2024-09-11T03:47:06.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3799999999999999e-06 2024-09-11T03:47:06.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886527+0000", 2024-09-11T03:47:06.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1797000000000003e-05 2024-09-11T03:47:06.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:47:06.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886534+0000", 2024-09-11T03:47:06.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2599999999999999e-06 2024-09-11T03:47:06.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.898657+0000", 2024-09-11T03:47:06.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012123073 2024-09-11T03:47:06.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.898831+0000", 2024-09-11T03:47:06.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000173931 2024-09-11T03:47:06.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.899093+0000", 2024-09-11T03:47:06.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:06.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:06.443 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:06 smithi121 ceph-mon[30729]: pgmap v1065: 67 pgs: 67 active+clean; 101 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.4 MiB/s wr, 20 op/s 2024-09-11T03:47:06.444 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:06 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3068847600' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:47:06.444 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:06 smithi121 ceph-mon[30729]: osdmap e607: 12 total, 12 up, 12 in 2024-09-11T03:47:06.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026152400000000001 2024-09-11T03:47:06.445 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.445 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.445 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.445 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2243 2.14 2:2ef19c88:::smithi17557870-15:58 [read 0~3327819,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:47:06.445 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:36.886461+0000", 2024-09-11T03:47:06.445 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 329.51963579300002, 2024-09-11T03:47:06.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014874943999999999, 2024-09-11T03:47:06.445 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.445 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.445 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:47:06.446 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:06.446 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:06.446 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:06.446 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2243 2024-09-11T03:47:06.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.446 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886461+0000", 2024-09-11T03:47:06.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886461+0000", 2024-09-11T03:47:06.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886462+0000", 2024-09-11T03:47:06.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1899999999999998e-07 2024-09-11T03:47:06.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886464+0000", 2024-09-11T03:47:06.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.632e-06 2024-09-11T03:47:06.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886470+0000", 2024-09-11T03:47:06.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.772e-06 2024-09-11T03:47:06.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886474+0000", 2024-09-11T03:47:06.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2549999999999999e-06 2024-09-11T03:47:06.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886554+0000", 2024-09-11T03:47:06.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9851999999999994e-05 2024-09-11T03:47:06.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:47:06.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886560+0000", 2024-09-11T03:47:06.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.117e-06 2024-09-11T03:47:06.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.899326+0000", 2024-09-11T03:47:06.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012765841999999999 2024-09-11T03:47:06.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.899468+0000", 2024-09-11T03:47:06.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014217099999999999 2024-09-11T03:47:06.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.901336+0000", 2024-09-11T03:47:06.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001867884 2024-09-11T03:47:06.450 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.450 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.450 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.451 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2492 2.5 e302/276 2:a1a99782:::smithi17557870-1:head v 302'331, mlcod=297'328)", 2024-09-11T03:47:06.451 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:58.902093+0000", 2024-09-11T03:47:06.451 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 307.504004114, 2024-09-11T03:47:06.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087348040000000005, 2024-09-11T03:47:06.451 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.451 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.451 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:06.451 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.902093+0000", 2024-09-11T03:47:06.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.902093+0000", 2024-09-11T03:47:06.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.902093+0000", 2024-09-11T03:47:06.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.11e-07 2024-09-11T03:47:06.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.908258+0000", 2024-09-11T03:47:06.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061649770000000003 2024-09-11T03:47:06.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.908276+0000", 2024-09-11T03:47:06.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7737e-05 2024-09-11T03:47:06.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.908284+0000", 2024-09-11T03:47:06.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2490000000000001e-06 2024-09-11T03:47:06.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.908348+0000", 2024-09-11T03:47:06.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3712000000000003e-05 2024-09-11T03:47:06.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.908367+0000", 2024-09-11T03:47:06.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9633e-05 2024-09-11T03:47:06.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:06.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.910624+0000", 2024-09-11T03:47:06.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022564680000000002 2024-09-11T03:47:06.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:58.910828+0000", 2024-09-11T03:47:06.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00020381699999999999 2024-09-11T03:47:06.455 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.455 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.455 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.455 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:4700 2.5 e504/276 2:a1a99782:::smithi17557870-1:head v 504'697, mlcod=504'693)", 2024-09-11T03:47:06.455 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:45:21.967265+0000", 2024-09-11T03:47:06.456 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 104.438831636, 2024-09-11T03:47:06.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082476579999999997, 2024-09-11T03:47:06.456 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.456 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.456 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:06.456 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:21.967265+0000", 2024-09-11T03:47:06.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:21.967265+0000", 2024-09-11T03:47:06.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:21.967267+0000", 2024-09-11T03:47:06.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.584e-06 2024-09-11T03:47:06.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:21.972584+0000", 2024-09-11T03:47:06.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053167090000000002 2024-09-11T03:47:06.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:21.972601+0000", 2024-09-11T03:47:06.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7153999999999999e-05 2024-09-11T03:47:06.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:21.972610+0000", 2024-09-11T03:47:06.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7329999999999992e-06 2024-09-11T03:47:06.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:21.972689+0000", 2024-09-11T03:47:06.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8947000000000002e-05 2024-09-11T03:47:06.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:21.972719+0000", 2024-09-11T03:47:06.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0497e-05 2024-09-11T03:47:06.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:06.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:21.975467+0000", 2024-09-11T03:47:06.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027477460000000001 2024-09-11T03:47:06.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:21.975513+0000", 2024-09-11T03:47:06.460 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6288000000000001e-05 2024-09-11T03:47:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.461 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:4741 2.5 e510/276 2:a286c222:::smithi17557870-29:head v 510'708, mlcod=506'704)", 2024-09-11T03:47:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:45:27.954570+0000", 2024-09-11T03:47:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 98.451527517000002, 2024-09-11T03:47:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082970319999999993, 2024-09-11T03:47:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:27.954570+0000", 2024-09-11T03:47:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:27.954570+0000", 2024-09-11T03:47:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:27.954570+0000", 2024-09-11T03:47:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2399999999999999e-07 2024-09-11T03:47:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:27.960229+0000", 2024-09-11T03:47:06.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056591660000000002 2024-09-11T03:47:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:27.960246+0000", 2024-09-11T03:47:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6892e-05 2024-09-11T03:47:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:27.960253+0000", 2024-09-11T03:47:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6159999999999996e-06 2024-09-11T03:47:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:27.960306+0000", 2024-09-11T03:47:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2253000000000003e-05 2024-09-11T03:47:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:27.960324+0000", 2024-09-11T03:47:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7875000000000001e-05 2024-09-11T03:47:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:27.962819+0000", 2024-09-11T03:47:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024951840000000001 2024-09-11T03:47:06.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:27.962867+0000", 2024-09-11T03:47:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7821999999999997e-05 2024-09-11T03:47:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:4913 2.19 2:98bb65f5:::smithi17557870-6:head [sparse-read 0~2219734,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e523)", 2024-09-11T03:47:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:45:41.041241+0000", 2024-09-11T03:47:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 85.364855841999997, 2024-09-11T03:47:06.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013112360999999999, 2024-09-11T03:47:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:47:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4913 2024-09-11T03:47:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041241+0000", 2024-09-11T03:47:06.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041241+0000", 2024-09-11T03:47:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041243+0000", 2024-09-11T03:47:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7010000000000001e-06 2024-09-11T03:47:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041250+0000", 2024-09-11T03:47:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.674e-06 2024-09-11T03:47:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041263+0000", 2024-09-11T03:47:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2942000000000001e-05 2024-09-11T03:47:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041306+0000", 2024-09-11T03:47:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3622999999999999e-05 2024-09-11T03:47:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.042352+0000", 2024-09-11T03:47:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001045891 2024-09-11T03:47:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:47:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.042358+0000", 2024-09-11T03:47:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1460000000000001e-06 2024-09-11T03:47:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.051787+0000", 2024-09-11T03:47:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094287030000000001 2024-09-11T03:47:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.051892+0000", 2024-09-11T03:47:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000104706 2024-09-11T03:47:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.054354+0000", 2024-09-11T03:47:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024619749999999999 2024-09-11T03:47:06.471 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.472 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.472 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.472 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:4914 2.19 2:98bb65f5:::smithi17557870-6:head [read 0~3075813,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e523)", 2024-09-11T03:47:06.472 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:45:41.041333+0000", 2024-09-11T03:47:06.472 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 85.364763867999997, 2024-09-11T03:47:06.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014730996999999999, 2024-09-11T03:47:06.472 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.472 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.472 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:47:06.472 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:06.473 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:06.473 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:06.473 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4914 2024-09-11T03:47:06.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.473 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041333+0000", 2024-09-11T03:47:06.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041333+0000", 2024-09-11T03:47:06.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041334+0000", 2024-09-11T03:47:06.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8199999999999999e-07 2024-09-11T03:47:06.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041338+0000", 2024-09-11T03:47:06.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2590000000000002e-06 2024-09-11T03:47:06.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041347+0000", 2024-09-11T03:47:06.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1689999999999998e-06 2024-09-11T03:47:06.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041354+0000", 2024-09-11T03:47:06.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.365e-06 2024-09-11T03:47:06.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.042433+0000", 2024-09-11T03:47:06.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010788709999999999 2024-09-11T03:47:06.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:06.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.042439+0000", 2024-09-11T03:47:06.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2179999999999996e-06 2024-09-11T03:47:06.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.054401+0000", 2024-09-11T03:47:06.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011962494000000001 2024-09-11T03:47:06.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.054509+0000", 2024-09-11T03:47:06.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010773900000000001 2024-09-11T03:47:06.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.056064+0000", 2024-09-11T03:47:06.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015552000000000001 2024-09-11T03:47:06.477 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.477 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.477 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.477 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:4915 2.19 2:98bb65f5:::smithi17557870-6:head [sparse-read 0~2219734] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e523)", 2024-09-11T03:47:06.477 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:45:41.041363+0000", 2024-09-11T03:47:06.477 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 85.364734143999996, 2024-09-11T03:47:06.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015358287999999999, 2024-09-11T03:47:06.478 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.478 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.478 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:47:06.478 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:06.478 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:06.478 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:06.478 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4915 2024-09-11T03:47:06.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.478 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041363+0000", 2024-09-11T03:47:06.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041363+0000", 2024-09-11T03:47:06.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041363+0000", 2024-09-11T03:47:06.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8400000000000004e-07 2024-09-11T03:47:06.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041368+0000", 2024-09-11T03:47:06.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1060000000000002e-06 2024-09-11T03:47:06.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041374+0000", 2024-09-11T03:47:06.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.866e-06 2024-09-11T03:47:06.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041380+0000", 2024-09-11T03:47:06.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3720000000000004e-06 2024-09-11T03:47:06.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.042462+0000", 2024-09-11T03:47:06.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001082258 2024-09-11T03:47:06.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:06.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.042467+0000", 2024-09-11T03:47:06.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0130000000000001e-06 2024-09-11T03:47:06.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.056146+0000", 2024-09-11T03:47:06.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013679314999999999 2024-09-11T03:47:06.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.056249+0000", 2024-09-11T03:47:06.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102586 2024-09-11T03:47:06.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.056721+0000", 2024-09-11T03:47:06.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00047218800000000001 2024-09-11T03:47:06.482 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.482 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.483 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.483 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:5359 2.4 e573/432 2:220bb244:::smithi17557870-34:head v 573'441, mlcod=571'437)", 2024-09-11T03:47:06.483 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:30.991138+0000", 2024-09-11T03:47:06.483 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 35.414959314999997, 2024-09-11T03:47:06.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081793419999999992, 2024-09-11T03:47:06.483 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.483 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.483 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:06.483 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.991138+0000", 2024-09-11T03:47:06.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.991138+0000", 2024-09-11T03:47:06.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.991139+0000", 2024-09-11T03:47:06.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7199999999999997e-07 2024-09-11T03:47:06.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.993984+0000", 2024-09-11T03:47:06.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028451829999999998 2024-09-11T03:47:06.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.994006+0000", 2024-09-11T03:47:06.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1858e-05 2024-09-11T03:47:06.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.994014+0000", 2024-09-11T03:47:06.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4759999999999998e-06 2024-09-11T03:47:06.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.994063+0000", 2024-09-11T03:47:06.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9175999999999998e-05 2024-09-11T03:47:06.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.994085+0000", 2024-09-11T03:47:06.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1597e-05 2024-09-11T03:47:06.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.999268+0000", 2024-09-11T03:47:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051834079999999996 2024-09-11T03:47:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.999317+0000", 2024-09-11T03:47:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8772e-05 2024-09-11T03:47:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:5613 2.5 e597/276 2:a1a99782:::smithi17557870-1:head v 597'842, mlcod=594'840)", 2024-09-11T03:47:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:55.197549+0000", 2024-09-11T03:47:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 11.208548073999999, 2024-09-11T03:47:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082009619999999991, 2024-09-11T03:47:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:55.197549+0000", 2024-09-11T03:47:06.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:55.197549+0000", 2024-09-11T03:47:06.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:55.197549+0000", 2024-09-11T03:47:06.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5900000000000002e-07 2024-09-11T03:47:06.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:55.203779+0000", 2024-09-11T03:47:06.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062291259999999998 2024-09-11T03:47:06.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:55.203791+0000", 2024-09-11T03:47:06.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2916e-05 2024-09-11T03:47:06.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:55.203798+0000", 2024-09-11T03:47:06.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4729999999999997e-06 2024-09-11T03:47:06.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:55.203860+0000", 2024-09-11T03:47:06.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2025000000000003e-05 2024-09-11T03:47:06.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:55.203879+0000", 2024-09-11T03:47:06.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8975e-05 2024-09-11T03:47:06.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:06.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:55.205713+0000", 2024-09-11T03:47:06.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018341950000000001 2024-09-11T03:47:06.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:55.205750+0000", 2024-09-11T03:47:06.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6792999999999998e-05 2024-09-11T03:47:06.493 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.493 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.493 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.493 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5648 2.1f 2:fda69313:::smithi17557870-16:head [copy-from ver 826] snapc f9=[] ondisk+write+known_if_redirected+supports_pool_eio e599)", 2024-09-11T03:47:06.494 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:56.974414+0000", 2024-09-11T03:47:06.494 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.4316827150000009, 2024-09-11T03:47:06.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021325499000000001, 2024-09-11T03:47:06.494 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.494 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.494 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:06.494 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:06.494 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:06.494 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:06.494 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5648 2024-09-11T03:47:06.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.494 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974414+0000", 2024-09-11T03:47:06.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974414+0000", 2024-09-11T03:47:06.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974416+0000", 2024-09-11T03:47:06.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.666e-06 2024-09-11T03:47:06.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974422+0000", 2024-09-11T03:47:06.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2400000000000004e-06 2024-09-11T03:47:06.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974434+0000", 2024-09-11T03:47:06.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2111e-05 2024-09-11T03:47:06.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974442+0000", 2024-09-11T03:47:06.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9149999999999996e-06 2024-09-11T03:47:06.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974501+0000", 2024-09-11T03:47:06.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8369000000000003e-05 2024-09-11T03:47:06.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:47:06.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974509+0000", 2024-09-11T03:47:06.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2260000000000005e-06 2024-09-11T03:47:06.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976102+0000", 2024-09-11T03:47:06.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001593187 2024-09-11T03:47:06.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976234+0000", 2024-09-11T03:47:06.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000131903 2024-09-11T03:47:06.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7,8", 2024-09-11T03:47:06.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.986796+0000", 2024-09-11T03:47:06.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010562344 2024-09-11T03:47:06.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:47:06.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.989855+0000", 2024-09-11T03:47:06.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030587660000000001 2024-09-11T03:47:06.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:06.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.992580+0000", 2024-09-11T03:47:06.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027246950000000001 2024-09-11T03:47:06.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:06.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.995661+0000", 2024-09-11T03:47:06.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030807500000000002 2024-09-11T03:47:06.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:06.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.995708+0000", 2024-09-11T03:47:06.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7945999999999998e-05 2024-09-11T03:47:06.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.995740+0000", 2024-09-11T03:47:06.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1381000000000003e-05 2024-09-11T03:47:06.501 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.501 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.501 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.501 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5649 2.1f 2:fda69313:::smithi17557870-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e599)", 2024-09-11T03:47:06.501 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:56.974454+0000", 2024-09-11T03:47:06.501 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.4316431969999996, 2024-09-11T03:47:06.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015178341999999999, 2024-09-11T03:47:06.502 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.502 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.502 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:47:06.502 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:06.502 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:06.502 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:06.502 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5649 2024-09-11T03:47:06.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.502 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974454+0000", 2024-09-11T03:47:06.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974454+0000", 2024-09-11T03:47:06.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974454+0000", 2024-09-11T03:47:06.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8700000000000001e-07 2024-09-11T03:47:06.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974457+0000", 2024-09-11T03:47:06.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.61e-06 2024-09-11T03:47:06.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974462+0000", 2024-09-11T03:47:06.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6700000000000002e-06 2024-09-11T03:47:06.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974465+0000", 2024-09-11T03:47:06.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4829999999999999e-06 2024-09-11T03:47:06.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974546+0000", 2024-09-11T03:47:06.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0938000000000003e-05 2024-09-11T03:47:06.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:06.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974549+0000", 2024-09-11T03:47:06.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3830000000000001e-06 2024-09-11T03:47:06.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976401+0000", 2024-09-11T03:47:06.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018517480000000001 2024-09-11T03:47:06.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:06.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976445+0000", 2024-09-11T03:47:06.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3439000000000002e-05 2024-09-11T03:47:06.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.989435+0000", 2024-09-11T03:47:06.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012990714 2024-09-11T03:47:06.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.989531+0000", 2024-09-11T03:47:06.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5304999999999996e-05 2024-09-11T03:47:06.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.989632+0000", 2024-09-11T03:47:06.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000101665 2024-09-11T03:47:06.507 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.507 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.507 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.508 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5650 2.1e 2:7b0c3397:::smithi17557870-14:head [copy-from ver 875] snapc f9=[] ondisk+write+known_if_redirected+supports_pool_eio e599)", 2024-09-11T03:47:06.508 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:56.974491+0000", 2024-09-11T03:47:06.508 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.4316056869999993, 2024-09-11T03:47:06.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.029056842999999999, 2024-09-11T03:47:06.508 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.508 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.508 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:06.508 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:06.508 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:06.508 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:06.508 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5650 2024-09-11T03:47:06.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.509 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974491+0000", 2024-09-11T03:47:06.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974491+0000", 2024-09-11T03:47:06.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974492+0000", 2024-09-11T03:47:06.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.72e-07 2024-09-11T03:47:06.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974494+0000", 2024-09-11T03:47:06.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0480000000000001e-06 2024-09-11T03:47:06.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974498+0000", 2024-09-11T03:47:06.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1699999999999999e-06 2024-09-11T03:47:06.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974501+0000", 2024-09-11T03:47:06.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0479999999999999e-06 2024-09-11T03:47:06.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974537+0000", 2024-09-11T03:47:06.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6041000000000003e-05 2024-09-11T03:47:06.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:47:06.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974541+0000", 2024-09-11T03:47:06.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2010000000000001e-06 2024-09-11T03:47:06.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.975950+0000", 2024-09-11T03:47:06.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014090489999999999 2024-09-11T03:47:06.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976045+0000", 2024-09-11T03:47:06.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4704999999999995e-05 2024-09-11T03:47:06.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,10", 2024-09-11T03:47:06.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.992128+0000", 2024-09-11T03:47:06.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016083318999999999 2024-09-11T03:47:06.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:47:06.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.994884+0000", 2024-09-11T03:47:06.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027554860000000001 2024-09-11T03:47:06.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:06.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.997365+0000", 2024-09-11T03:47:06.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024811630000000002 2024-09-11T03:47:06.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:06.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:57.003332+0000", 2024-09-11T03:47:06.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059666720000000001 2024-09-11T03:47:06.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:06.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:57.003487+0000", 2024-09-11T03:47:06.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015519399999999999 2024-09-11T03:47:06.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:57.003548+0000", 2024-09-11T03:47:06.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1375000000000001e-05 2024-09-11T03:47:06.515 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.515 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.515 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.515 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5651 2.1e 2:7b0c3397:::smithi17557870-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e599)", 2024-09-11T03:47:06.515 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:56.974509+0000", 2024-09-11T03:47:06.515 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.4315877950000004, 2024-09-11T03:47:06.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.02013765, 2024-09-11T03:47:06.515 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.516 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.516 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:47:06.516 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:06.516 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:06.516 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:06.516 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5651 2024-09-11T03:47:06.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.516 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974509+0000", 2024-09-11T03:47:06.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974509+0000", 2024-09-11T03:47:06.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974510+0000", 2024-09-11T03:47:06.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5999999999999999e-07 2024-09-11T03:47:06.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974511+0000", 2024-09-11T03:47:06.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.72e-06 2024-09-11T03:47:06.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974517+0000", 2024-09-11T03:47:06.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0499999999999997e-06 2024-09-11T03:47:06.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974522+0000", 2024-09-11T03:47:06.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7770000000000002e-06 2024-09-11T03:47:06.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974570+0000", 2024-09-11T03:47:06.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7729000000000003e-05 2024-09-11T03:47:06.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:06.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974574+0000", 2024-09-11T03:47:06.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7929999999999999e-06 2024-09-11T03:47:06.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976323+0000", 2024-09-11T03:47:06.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017491760000000001 2024-09-11T03:47:06.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:06.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976367+0000", 2024-09-11T03:47:06.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3912000000000002e-05 2024-09-11T03:47:06.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.994395+0000", 2024-09-11T03:47:06.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018028512 2024-09-11T03:47:06.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.994492+0000", 2024-09-11T03:47:06.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6679000000000001e-05 2024-09-11T03:47:06.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.994647+0000", 2024-09-11T03:47:06.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015484200000000001 2024-09-11T03:47:06.521 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.521 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.521 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.521 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5783 2.1e 2:7b0c3397:::smithi17557870-14:head [write 1564160~424282] snapc ff=[ff,fe,fd] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-09-11T03:47:06.521 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:47:06.034510+0000", 2024-09-11T03:47:06.521 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.37158656899999998, 2024-09-11T03:47:06.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010361469, 2024-09-11T03:47:06.522 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:06.522 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:06.522 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:06.522 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:06.522 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:06.522 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:06.522 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5783 2024-09-11T03:47:06.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.522 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:06.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:06.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:06.034510+0000", 2024-09-11T03:47:06.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:06.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:06.034510+0000", 2024-09-11T03:47:06.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:06.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:06.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:06.034511+0000", 2024-09-11T03:47:06.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.003e-06 2024-09-11T03:47:06.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:06.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:06.035048+0000", 2024-09-11T03:47:06.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00053609999999999997 2024-09-11T03:47:06.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:06.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:06.035058+0000", 2024-09-11T03:47:06.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.01e-05 2024-09-11T03:47:06.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:06.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:06.035063+0000", 2024-09-11T03:47:06.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8670000000000002e-06 2024-09-11T03:47:06.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:06.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:06.035110+0000", 2024-09-11T03:47:06.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7071999999999999e-05 2024-09-11T03:47:06.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:06.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:06.035211+0000", 2024-09-11T03:47:06.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000101504 2024-09-11T03:47:06.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,10", 2024-09-11T03:47:06.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:06.035323+0000", 2024-09-11T03:47:06.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011178 2024-09-11T03:47:06.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:06.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:06.037007+0000", 2024-09-11T03:47:06.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016844329999999999 2024-09-11T03:47:06.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:47:06.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:06.037220+0000", 2024-09-11T03:47:06.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021269600000000001 2024-09-11T03:47:06.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:06.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:06.044760+0000", 2024-09-11T03:47:06.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0075397839999999999 2024-09-11T03:47:06.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:06.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:06.044824+0000", 2024-09-11T03:47:06.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4487999999999996e-05 2024-09-11T03:47:06.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:06.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:06.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:06.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:06.044872+0000", 2024-09-11T03:47:06.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7642000000000002e-05 2024-09-11T03:47:06.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.527 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.528 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:06.528 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:06.528 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:47:06.528 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:06 smithi175 ceph-mon[30894]: pgmap v1065: 67 pgs: 67 active+clean; 101 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.4 MiB/s wr, 20 op/s 2024-09-11T03:47:06.528 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:06 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3068847600' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:47:06.528 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:06 smithi175 ceph-mon[30894]: osdmap e607: 12 total, 12 up, 12 in 2024-09-11T03:47:06.544 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:06.564 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:06.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:06.666 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:06.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:06.691 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:06.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:06.791 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:06.817 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:06.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:06.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:06.919 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:06.940 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:06.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:06.974 INFO:tasks.rados.rados.2.smithi175.stdout:2308: done (7 left) 2024-09-11T03:47:06.975 INFO:tasks.rados.rados.2.smithi175.stdout:2310: done (6 left) 2024-09-11T03:47:06.975 INFO:tasks.rados.rados.2.smithi175.stdout:2311: done (5 left) 2024-09-11T03:47:06.975 INFO:tasks.rados.rados.2.smithi175.stdout:2312: done (4 left) 2024-09-11T03:47:06.975 INFO:tasks.rados.rados.2.smithi175.stdout:2313: done (3 left) 2024-09-11T03:47:06.975 INFO:tasks.rados.rados.2.smithi175.stdout:2314: done (2 left) 2024-09-11T03:47:06.975 INFO:tasks.rados.rados.2.smithi175.stdout:2315: done (1 left) 2024-09-11T03:47:06.975 INFO:tasks.rados.rados.2.smithi175.stdout:2316: done (0 left) 2024-09-11T03:47:06.975 INFO:tasks.rados.rados.2.smithi175.stdout:2317: read oid 18 snap -1 2024-09-11T03:47:06.975 INFO:tasks.rados.rados.2.smithi175.stdout:2317: expect deleted 2024-09-11T03:47:06.975 INFO:tasks.rados.rados.2.smithi175.stdout:2318: read oid 39 snap -1 2024-09-11T03:47:06.976 INFO:tasks.rados.rados.2.smithi175.stdout:2318: expect (ObjNum 738 snap 243 seq_num 738) 2024-09-11T03:47:06.976 INFO:tasks.rados.rados.2.smithi175.stdout:2319: write (excl) oid 15 current snap is 255 2024-09-11T03:47:06.976 INFO:tasks.rados.rados.2.smithi175.stdout:2319: seq_num 800 ranges {645600=416265,1797477=711598,3219453=226147} 2024-09-11T03:47:06.980 INFO:tasks.rados.rados.2.smithi175.stdout:2319: writing smithi17557870-15 from 645600 to 1061865 tid 1 2024-09-11T03:47:06.988 INFO:tasks.rados.rados.2.smithi175.stdout:2319: writing smithi17557870-15 from 1797477 to 2509075 tid 2 2024-09-11T03:47:06.993 INFO:tasks.rados.rados.2.smithi175.stdout:2319: writing smithi17557870-15 from 3219453 to 3445600 tid 3 2024-09-11T03:47:06.994 INFO:tasks.rados.rados.2.smithi175.stdout:2320: write (excl) oid 45 current snap is 255 2024-09-11T03:47:06.994 INFO:tasks.rados.rados.2.smithi175.stdout:2320: seq_num 801 ranges {662407=491514,1855959=571609,2846876=615531} 2024-09-11T03:47:07.000 INFO:tasks.rados.rados.2.smithi175.stdout:2320: writing smithi17557870-45 from 662407 to 1153921 tid 1 2024-09-11T03:47:07.006 INFO:tasks.rados.rados.2.smithi175.stdout:2320: writing smithi17557870-45 from 1855959 to 2427568 tid 2 2024-09-11T03:47:07.012 INFO:tasks.rados.rados.2.smithi175.stdout:2320: writing smithi17557870-45 from 2846876 to 3462407 tid 3 2024-09-11T03:47:07.012 INFO:tasks.rados.rados.2.smithi175.stdout:2321: write oid 14 current snap is 255 2024-09-11T03:47:07.013 INFO:tasks.rados.rados.2.smithi175.stdout:2321: seq_num 802 ranges {679214=566763,1798088=715267,3157946=321268} 2024-09-11T03:47:07.019 INFO:tasks.rados.rados.2.smithi175.stdout:2321: writing smithi17557870-14 from 679214 to 1245977 tid 1 2024-09-11T03:47:07.026 INFO:tasks.rados.rados.2.smithi175.stdout:2321: writing smithi17557870-14 from 1798088 to 2513355 tid 2 2024-09-11T03:47:07.032 INFO:tasks.rados.rados.2.smithi175.stdout:2321: writing smithi17557870-14 from 3157946 to 3479214 tid 3 2024-09-11T03:47:07.032 INFO:tasks.rados.rados.2.smithi175.stdout:2322: write oid 25 current snap is 255 2024-09-11T03:47:07.032 INFO:tasks.rados.rados.2.smithi175.stdout:2322: seq_num 803 ranges {696021=642012,1856570=575278,3185369=310652} 2024-09-11T03:47:07.039 INFO:tasks.rados.rados.2.smithi175.stdout:2322: writing smithi17557870-25 from 696021 to 1338033 tid 1 2024-09-11T03:47:07.041 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:07.045 INFO:tasks.rados.rados.2.smithi175.stdout:2322: writing smithi17557870-25 from 1856570 to 2431848 tid 2 2024-09-11T03:47:07.051 INFO:tasks.rados.rados.2.smithi175.stdout:2322: writing smithi17557870-25 from 3185369 to 3496021 tid 3 2024-09-11T03:47:07.051 INFO:tasks.rados.rados.2.smithi175.stdout:2323: write (excl) oid 7 current snap is 255 2024-09-11T03:47:07.051 INFO:tasks.rados.rados.2.smithi175.stdout:2323: seq_num 804 ranges {712828=717261,1915052=718936,3212792=300036} 2024-09-11T03:47:07.058 INFO:tasks.rados.rados.2.smithi175.stdout:2323: writing smithi17557870-7 from 712828 to 1430089 tid 1 2024-09-11T03:47:07.063 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:07.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:07.064 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:07.065 INFO:tasks.rados.rados.2.smithi175.stdout:2323: writing smithi17557870-7 from 1915052 to 2633988 tid 2 2024-09-11T03:47:07.070 INFO:tasks.rados.rados.2.smithi175.stdout:2323: writing smithi17557870-7 from 3212792 to 3512828 tid 3 2024-09-11T03:47:07.070 INFO:tasks.rados.rados.2.smithi175.stdout:2324: write oid 46 current snap is 255 2024-09-11T03:47:07.070 INFO:tasks.rados.rados.2.smithi175.stdout:2318: expect (ObjNum 738 snap 243 seq_num 738) 2024-09-11T03:47:07.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:07.188 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:07.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:07.192 INFO:tasks.rados.rados.2.smithi175.stdout:2319: finishing write tid 1 to smithi17557870-15 2024-09-11T03:47:07.193 INFO:tasks.rados.rados.2.smithi175.stdout:2319: finishing write tid 2 to smithi17557870-15 2024-09-11T03:47:07.193 INFO:tasks.rados.rados.2.smithi175.stdout:2319: finishing write tid 3 to smithi17557870-15 2024-09-11T03:47:07.193 INFO:tasks.rados.rados.2.smithi175.stdout:2319: finishing write tid 4 to smithi17557870-15 2024-09-11T03:47:07.193 INFO:tasks.rados.rados.2.smithi175.stdout:2319: finishing write tid 5 to smithi17557870-15 2024-09-11T03:47:07.193 INFO:tasks.rados.rados.2.smithi175.stdout:2319: oid 15 updating version 0 to 267 2024-09-11T03:47:07.193 INFO:tasks.rados.rados.2.smithi175.stdout:2319: oid 15 version 267 is already newer than 266 2024-09-11T03:47:07.193 INFO:tasks.rados.rados.2.smithi175.stdout:2319: oid 15 version 267 is already newer than 265 2024-09-11T03:47:07.193 INFO:tasks.rados.rados.2.smithi175.stdout:2319: oid 15 updating version 267 to 268 2024-09-11T03:47:07.193 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 268 (ObjNum 800 snap 255 seq_num 800) dirty exists 2024-09-11T03:47:07.193 INFO:tasks.rados.rados.2.smithi175.stdout:2319: left oid 15 (ObjNum 800 snap 255 seq_num 800) 2024-09-11T03:47:07.193 INFO:tasks.rados.rados.2.smithi175.stdout:2320: finishing write tid 1 to smithi17557870-45 2024-09-11T03:47:07.193 INFO:tasks.rados.rados.2.smithi175.stdout:2324: seq_num 805 ranges {729635=792510,1973534=578947,3240215=289420} 2024-09-11T03:47:07.200 INFO:tasks.rados.rados.2.smithi175.stdout:2324: writing smithi17557870-46 from 729635 to 1522145 tid 1 2024-09-11T03:47:07.206 INFO:tasks.rados.rados.2.smithi175.stdout:2324: writing smithi17557870-46 from 1973534 to 2552481 tid 2 2024-09-11T03:47:07.212 INFO:tasks.rados.rados.2.smithi175.stdout:2324: writing smithi17557870-46 from 3240215 to 3529635 tid 3 2024-09-11T03:47:07.212 INFO:tasks.rados.rados.2.smithi175.stdout:2317: done (7 left) 2024-09-11T03:47:07.212 INFO:tasks.rados.rados.2.smithi175.stdout:2318: done (6 left) 2024-09-11T03:47:07.212 INFO:tasks.rados.rados.2.smithi175.stdout:2319: done (5 left) 2024-09-11T03:47:07.212 INFO:tasks.rados.rados.2.smithi175.stdout:2325: read oid 13 snap -1 2024-09-11T03:47:07.212 INFO:tasks.rados.rados.2.smithi175.stdout:2325: expect (ObjNum 791 snap 252 seq_num 791) 2024-09-11T03:47:07.212 INFO:tasks.rados.rados.2.smithi175.stdout:2326: read oid 49 snap -1 2024-09-11T03:47:07.212 INFO:tasks.rados.rados.2.smithi175.stdout:2326: expect (ObjNum 773 snap 248 seq_num 773) 2024-09-11T03:47:07.213 INFO:tasks.rados.rados.2.smithi175.stdout:2327: snap_remove snap 252 2024-09-11T03:47:07.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:07 smithi084 ceph-mon[27254]: osdmap e608: 12 total, 12 up, 12 in 2024-09-11T03:47:07.240 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:07 smithi084 ceph-mon[27254]: osdmap e609: 12 total, 12 up, 12 in 2024-09-11T03:47:07.289 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:07.311 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:07.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:07.413 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:07.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:07 smithi121 ceph-mon[30729]: osdmap e608: 12 total, 12 up, 12 in 2024-09-11T03:47:07.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:07 smithi121 ceph-mon[30729]: osdmap e609: 12 total, 12 up, 12 in 2024-09-11T03:47:07.435 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:07.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:07.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:07 smithi175 ceph-mon[30894]: osdmap e608: 12 total, 12 up, 12 in 2024-09-11T03:47:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:07 smithi175 ceph-mon[30894]: osdmap e609: 12 total, 12 up, 12 in 2024-09-11T03:47:07.537 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:07.558 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:07.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:07.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:07.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:07.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:07.685 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:07.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:07.786 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:07.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:07.810 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:07.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:07.912 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:07.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T03:47:07.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:07.937 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:07.937 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:07.966 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:08.017 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:47:08.022 INFO:tasks.rados.rados.2.smithi175.stdout:2327: done (7 left) 2024-09-11T03:47:08.023 INFO:tasks.rados.rados.2.smithi175.stdout:2328: write (excl) oid 18 current snap is 255 2024-09-11T03:47:08.023 INFO:tasks.rados.rados.2.smithi175.stdout:2320: finishing write tid 2 to smithi17557870-45 2024-09-11T03:47:08.023 INFO:tasks.rados.rados.2.smithi175.stdout:2328: seq_num 806 ranges {746442=584112,1748369=722605,2983991=562451} 2024-09-11T03:47:08.030 INFO:tasks.rados.rados.2.smithi175.stdout:2328: writing smithi17557870-18 from 746442 to 1330554 tid 1 2024-09-11T03:47:08.036 INFO:tasks.rados.rados.2.smithi175.stdout:2328: writing smithi17557870-18 from 1748369 to 2470974 tid 2 2024-09-11T03:47:08.038 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:08.043 INFO:tasks.rados.rados.2.smithi175.stdout:2328: writing smithi17557870-18 from 2983991 to 3546442 tid 3 2024-09-11T03:47:08.044 INFO:tasks.rados.rados.2.smithi175.stdout:2329: copy_from oid 12 from oid 32 current snap is 255 2024-09-11T03:47:08.044 INFO:tasks.rados.rados.2.smithi175.stdout:2320: finishing write tid 3 to smithi17557870-45 2024-09-11T03:47:08.044 INFO:tasks.rados.rados.2.smithi175.stdout:2320: finishing write tid 4 to smithi17557870-45 2024-09-11T03:47:08.044 INFO:tasks.rados.rados.2.smithi175.stdout:2320: finishing write tid 5 to smithi17557870-45 2024-09-11T03:47:08.044 INFO:tasks.rados.rados.2.smithi175.stdout:2320: oid 45 updating version 0 to 703 2024-09-11T03:47:08.044 INFO:tasks.rados.rados.2.smithi175.stdout:2320: oid 45 updating version 703 to 704 2024-09-11T03:47:08.044 INFO:tasks.rados.rados.2.smithi175.stdout:2320: oid 45 version 704 is already newer than 702 2024-09-11T03:47:08.044 INFO:tasks.rados.rados.2.smithi175.stdout:2320: oid 45 version 704 is already newer than 701 2024-09-11T03:47:08.044 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 704 (ObjNum 801 snap 255 seq_num 801) dirty exists 2024-09-11T03:47:08.044 INFO:tasks.rados.rados.2.smithi175.stdout:2320: left oid 45 (ObjNum 801 snap 255 seq_num 801) 2024-09-11T03:47:08.044 INFO:tasks.rados.rados.2.smithi175.stdout:2321: finishing write tid 1 to smithi17557870-14 2024-09-11T03:47:08.044 INFO:tasks.rados.rados.2.smithi175.stdout:2321: finishing write tid 2 to smithi17557870-14 2024-09-11T03:47:08.044 INFO:tasks.rados.rados.2.smithi175.stdout:2321: finishing write tid 3 to smithi17557870-14 2024-09-11T03:47:08.044 INFO:tasks.rados.rados.2.smithi175.stdout:2321: finishing write tid 4 to smithi17557870-14 2024-09-11T03:47:08.045 INFO:tasks.rados.rados.2.smithi175.stdout:2321: finishing write tid 5 to smithi17557870-14 2024-09-11T03:47:08.045 INFO:tasks.rados.rados.2.smithi175.stdout:2321: oid 14 updating version 0 to 882 2024-09-11T03:47:08.045 INFO:tasks.rados.rados.2.smithi175.stdout:2321: oid 14 version 882 is already newer than 881 2024-09-11T03:47:08.045 INFO:tasks.rados.rados.2.smithi175.stdout:2321: oid 14 updating version 882 to 883 2024-09-11T03:47:08.045 INFO:tasks.rados.rados.2.smithi175.stdout:2321: oid 14 version 883 is already newer than 880 2024-09-11T03:47:08.045 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 883 (ObjNum 802 snap 255 seq_num 802) dirty exists 2024-09-11T03:47:08.045 INFO:tasks.rados.rados.2.smithi175.stdout:2321: left oid 14 (ObjNum 802 snap 255 seq_num 802) 2024-09-11T03:47:08.045 INFO:tasks.rados.rados.2.smithi175.stdout:2322: finishing write tid 1 to smithi17557870-25 2024-09-11T03:47:08.045 INFO:tasks.rados.rados.2.smithi175.stdout:2320: done (8 left) 2024-09-11T03:47:08.046 INFO:tasks.rados.rados.2.smithi175.stdout:2321: done (7 left) 2024-09-11T03:47:08.046 INFO:tasks.rados.rados.2.smithi175.stdout:2330: snap_remove snap 251 2024-09-11T03:47:08.062 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:08.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:08.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:08.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:08 smithi084 ceph-mon[27254]: pgmap v1068: 67 pgs: 67 active+clean; 115 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-11T03:47:08.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:08 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2780579760' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:47:08.163 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:08.190 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:08.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:08.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:08.199 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:47:08.291 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:08.312 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:08.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:08.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:08.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:08 smithi121 ceph-mon[30729]: pgmap v1068: 67 pgs: 67 active+clean; 115 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-11T03:47:08.323 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:08 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2780579760' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:47:08.413 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:08.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:08.440 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:08.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:08 smithi175 ceph-mon[30894]: pgmap v1068: 67 pgs: 67 active+clean; 115 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-11T03:47:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:08 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2780579760' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:47:08.541 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:08.566 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:08.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:08.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:08.669 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:08.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:08.696 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:08.697 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:08.798 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:08.824 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:08.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:08.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:08.927 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:08.949 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:08.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:09.027 INFO:tasks.rados.rados.2.smithi175.stdout:2330: done (7 left) 2024-09-11T03:47:09.028 INFO:tasks.rados.rados.2.smithi175.stdout:2331: copy_from oid 6 from oid 3 current snap is 255 2024-09-11T03:47:09.028 INFO:tasks.rados.rados.2.smithi175.stdout:2322: finishing write tid 2 to smithi17557870-25 2024-09-11T03:47:09.028 INFO:tasks.rados.rados.2.smithi175.stdout:2322: finishing write tid 3 to smithi17557870-25 2024-09-11T03:47:09.028 INFO:tasks.rados.rados.2.smithi175.stdout:2322: finishing write tid 4 to smithi17557870-25 2024-09-11T03:47:09.028 INFO:tasks.rados.rados.2.smithi175.stdout:2322: finishing write tid 5 to smithi17557870-25 2024-09-11T03:47:09.028 INFO:tasks.rados.rados.2.smithi175.stdout:2322: oid 25 updating version 0 to 507 2024-09-11T03:47:09.028 INFO:tasks.rados.rados.2.smithi175.stdout:2322: oid 25 version 507 is already newer than 505 2024-09-11T03:47:09.028 INFO:tasks.rados.rados.2.smithi175.stdout:2322: oid 25 version 507 is already newer than 506 2024-09-11T03:47:09.028 INFO:tasks.rados.rados.2.smithi175.stdout:2322: oid 25 updating version 507 to 508 2024-09-11T03:47:09.028 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 508 (ObjNum 803 snap 255 seq_num 803) dirty exists 2024-09-11T03:47:09.028 INFO:tasks.rados.rados.2.smithi175.stdout:2322: left oid 25 (ObjNum 803 snap 255 seq_num 803) 2024-09-11T03:47:09.029 INFO:tasks.rados.rados.2.smithi175.stdout:2323: finishing write tid 1 to smithi17557870-7 2024-09-11T03:47:09.029 INFO:tasks.rados.rados.2.smithi175.stdout:2323: finishing write tid 2 to smithi17557870-7 2024-09-11T03:47:09.029 INFO:tasks.rados.rados.2.smithi175.stdout:2323: finishing write tid 3 to smithi17557870-7 2024-09-11T03:47:09.029 INFO:tasks.rados.rados.2.smithi175.stdout:2323: finishing write tid 4 to smithi17557870-7 2024-09-11T03:47:09.029 INFO:tasks.rados.rados.2.smithi175.stdout:2323: finishing write tid 5 to smithi17557870-7 2024-09-11T03:47:09.029 INFO:tasks.rados.rados.2.smithi175.stdout:2323: oid 7 updating version 0 to 530 2024-09-11T03:47:09.029 INFO:tasks.rados.rados.2.smithi175.stdout:2323: oid 7 updating version 530 to 532 2024-09-11T03:47:09.029 INFO:tasks.rados.rados.2.smithi175.stdout:2323: oid 7 version 532 is already newer than 529 2024-09-11T03:47:09.029 INFO:tasks.rados.rados.2.smithi175.stdout:2323: oid 7 version 532 is already newer than 531 2024-09-11T03:47:09.029 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 532 (ObjNum 804 snap 255 seq_num 804) dirty exists 2024-09-11T03:47:09.030 INFO:tasks.rados.rados.2.smithi175.stdout:2323: left oid 7 (ObjNum 804 snap 255 seq_num 804) 2024-09-11T03:47:09.030 INFO:tasks.rados.rados.2.smithi175.stdout:2324: finishing write tid 1 to smithi17557870-46 2024-09-11T03:47:09.030 INFO:tasks.rados.rados.2.smithi175.stdout:2324: finishing write tid 2 to smithi17557870-46 2024-09-11T03:47:09.030 INFO:tasks.rados.rados.2.smithi175.stdout:2324: finishing write tid 3 to smithi17557870-46 2024-09-11T03:47:09.030 INFO:tasks.rados.rados.2.smithi175.stdout:2324: finishing write tid 4 to smithi17557870-46 2024-09-11T03:47:09.030 INFO:tasks.rados.rados.2.smithi175.stdout:2324: finishing write tid 5 to smithi17557870-46 2024-09-11T03:47:09.030 INFO:tasks.rados.rados.2.smithi175.stdout:2324: oid 46 updating version 0 to 879 2024-09-11T03:47:09.030 INFO:tasks.rados.rados.2.smithi175.stdout:2324: oid 46 updating version 879 to 880 2024-09-11T03:47:09.030 INFO:tasks.rados.rados.2.smithi175.stdout:2324: oid 46 updating version 880 to 881 2024-09-11T03:47:09.030 INFO:tasks.rados.rados.2.smithi175.stdout:2324: oid 46 updating version 881 to 882 2024-09-11T03:47:09.031 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 882 (ObjNum 805 snap 255 seq_num 805) dirty exists 2024-09-11T03:47:09.031 INFO:tasks.rados.rados.2.smithi175.stdout:2324: left oid 46 (ObjNum 805 snap 255 seq_num 805) 2024-09-11T03:47:09.031 INFO:tasks.rados.rados.2.smithi175.stdout:2325: expect (ObjNum 791 snap 252 seq_num 791) 2024-09-11T03:47:09.051 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:09.073 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:09.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:09.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:09.108 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:09 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:09.108 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:09 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2780579760' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:47:09.109 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:09 smithi084 ceph-mon[27254]: osdmap e610: 12 total, 12 up, 12 in 2024-09-11T03:47:09.136 INFO:tasks.rados.rados.2.smithi175.stdout:2326: expect (ObjNum 773 snap 248 seq_num 773) 2024-09-11T03:47:09.143 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:09.175 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:09.197 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:09.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:09.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:09.231 INFO:tasks.rados.rados.2.smithi175.stdout:2328: finishing write tid 1 to smithi17557870-18 2024-09-11T03:47:09.232 INFO:tasks.rados.rados.2.smithi175.stdout:2328: finishing write tid 2 to smithi17557870-18 2024-09-11T03:47:09.232 INFO:tasks.rados.rados.2.smithi175.stdout:2328: finishing write tid 3 to smithi17557870-18 2024-09-11T03:47:09.232 INFO:tasks.rados.rados.2.smithi175.stdout:2328: finishing write tid 4 to smithi17557870-18 2024-09-11T03:47:09.232 INFO:tasks.rados.rados.2.smithi175.stdout:2328: finishing write tid 5 to smithi17557870-18 2024-09-11T03:47:09.232 INFO:tasks.rados.rados.2.smithi175.stdout:2328: oid 18 updating version 0 to 475 2024-09-11T03:47:09.232 INFO:tasks.rados.rados.2.smithi175.stdout:2328: oid 18 version 475 is already newer than 473 2024-09-11T03:47:09.232 INFO:tasks.rados.rados.2.smithi175.stdout:2328: oid 18 version 475 is already newer than 474 2024-09-11T03:47:09.232 INFO:tasks.rados.rados.2.smithi175.stdout:2328: oid 18 version 475 is already newer than 472 2024-09-11T03:47:09.232 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 475 (ObjNum 806 snap 255 seq_num 806) dirty exists 2024-09-11T03:47:09.232 INFO:tasks.rados.rados.2.smithi175.stdout:2328: left oid 18 (ObjNum 806 snap 255 seq_num 806) 2024-09-11T03:47:09.232 INFO:tasks.rados.rados.2.smithi175.stdout:2329: finishing copy_from racing read to smithi17557870-12 2024-09-11T03:47:09.232 INFO:tasks.rados.rados.2.smithi175.stdout:2322: done (7 left) 2024-09-11T03:47:09.233 INFO:tasks.rados.rados.2.smithi175.stdout:2323: done (6 left) 2024-09-11T03:47:09.233 INFO:tasks.rados.rados.2.smithi175.stdout:2324: done (5 left) 2024-09-11T03:47:09.233 INFO:tasks.rados.rados.2.smithi175.stdout:2325: done (4 left) 2024-09-11T03:47:09.233 INFO:tasks.rados.rados.2.smithi175.stdout:2326: done (3 left) 2024-09-11T03:47:09.233 INFO:tasks.rados.rados.2.smithi175.stdout:2328: done (2 left) 2024-09-11T03:47:09.233 INFO:tasks.rados.rados.2.smithi175.stdout:2332: write (excl) oid 4 current snap is 255 2024-09-11T03:47:09.233 INFO:tasks.rados.rados.2.smithi175.stdout:2329: finishing copy_from to smithi17557870-12 2024-09-11T03:47:09.233 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 646 (ObjNum 721 snap 236 seq_num 721) dirty exists 2024-09-11T03:47:09.233 INFO:tasks.rados.rados.2.smithi175.stdout:2331: finishing copy_from racing read to smithi17557870-6 2024-09-11T03:47:09.233 INFO:tasks.rados.rados.2.smithi175.stdout:2331: finishing copy_from to smithi17557870-6 2024-09-11T03:47:09.233 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 881 (ObjNum 741 snap 243 seq_num 741) dirty exists 2024-09-11T03:47:09.234 INFO:tasks.rados.rados.2.smithi175.stdout:2332: seq_num 807 ranges {763249=659361,2090498=582616,3295061=268188} 2024-09-11T03:47:09.238 INFO:tasks.rados.rados.2.smithi175.stdout:2332: writing smithi17557870-4 from 763249 to 1422610 tid 1 2024-09-11T03:47:09.245 INFO:tasks.rados.rados.2.smithi175.stdout:2332: writing smithi17557870-4 from 2090498 to 2673114 tid 2 2024-09-11T03:47:09.250 INFO:tasks.rados.rados.2.smithi175.stdout:2332: writing smithi17557870-4 from 3295061 to 3563249 tid 3 2024-09-11T03:47:09.250 INFO:tasks.rados.rados.2.smithi175.stdout:2329: done (2 left) 2024-09-11T03:47:09.251 INFO:tasks.rados.rados.2.smithi175.stdout:2331: done (1 left) 2024-09-11T03:47:09.251 INFO:tasks.rados.rados.2.smithi175.stdout:2333: write oid 22 current snap is 255 2024-09-11T03:47:09.251 INFO:tasks.rados.rados.2.smithi175.stdout:2332: finishing write tid 1 to smithi17557870-4 2024-09-11T03:47:09.251 INFO:tasks.rados.rados.2.smithi175.stdout:2333: seq_num 808 ranges {780056=734610,2148980=726274,3322484=257572} 2024-09-11T03:47:09.258 INFO:tasks.rados.rados.2.smithi175.stdout:2333: writing smithi17557870-22 from 780056 to 1514666 tid 1 2024-09-11T03:47:09.265 INFO:tasks.rados.rados.2.smithi175.stdout:2333: writing smithi17557870-22 from 2148980 to 2875254 tid 2 2024-09-11T03:47:09.270 INFO:tasks.rados.rados.2.smithi175.stdout:2333: writing smithi17557870-22 from 3322484 to 3580056 tid 3 2024-09-11T03:47:09.270 INFO:tasks.rados.rados.2.smithi175.stdout:2334: write oid 24 current snap is 255 2024-09-11T03:47:09.270 INFO:tasks.rados.rados.2.smithi175.stdout:2332: finishing write tid 2 to smithi17557870-4 2024-09-11T03:47:09.271 INFO:tasks.rados.rados.2.smithi175.stdout:2332: finishing write tid 3 to smithi17557870-4 2024-09-11T03:47:09.271 INFO:tasks.rados.rados.2.smithi175.stdout:2332: finishing write tid 4 to smithi17557870-4 2024-09-11T03:47:09.271 INFO:tasks.rados.rados.2.smithi175.stdout:2332: finishing write tid 5 to smithi17557870-4 2024-09-11T03:47:09.271 INFO:tasks.rados.rados.2.smithi175.stdout:2332: oid 4 updating version 0 to 844 2024-09-11T03:47:09.271 INFO:tasks.rados.rados.2.smithi175.stdout:2332: oid 4 version 844 is already newer than 843 2024-09-11T03:47:09.271 INFO:tasks.rados.rados.2.smithi175.stdout:2332: oid 4 version 844 is already newer than 842 2024-09-11T03:47:09.272 INFO:tasks.rados.rados.2.smithi175.stdout:2332: oid 4 updating version 844 to 845 2024-09-11T03:47:09.272 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 845 (ObjNum 807 snap 255 seq_num 807) dirty exists 2024-09-11T03:47:09.272 INFO:tasks.rados.rados.2.smithi175.stdout:2332: left oid 4 (ObjNum 807 snap 255 seq_num 807) 2024-09-11T03:47:09.272 INFO:tasks.rados.rados.2.smithi175.stdout:2333: finishing write tid 1 to smithi17557870-22 2024-09-11T03:47:09.272 INFO:tasks.rados.rados.2.smithi175.stdout:2333: finishing write tid 2 to smithi17557870-22 2024-09-11T03:47:09.272 INFO:tasks.rados.rados.2.smithi175.stdout:2334: seq_num 809 ranges {796863=409859,1807462=586285,3066260=530603} 2024-09-11T03:47:09.277 INFO:tasks.rados.rados.2.smithi175.stdout:2334: writing smithi17557870-24 from 796863 to 1206722 tid 1 2024-09-11T03:47:09.282 INFO:tasks.rados.rados.2.smithi175.stdout:2334: writing smithi17557870-24 from 1807462 to 2393747 tid 2 2024-09-11T03:47:09.289 INFO:tasks.rados.rados.2.smithi175.stdout:2334: writing smithi17557870-24 from 3066260 to 3596863 tid 3 2024-09-11T03:47:09.289 INFO:tasks.rados.rados.2.smithi175.stdout:2332: done (2 left) 2024-09-11T03:47:09.289 INFO:tasks.rados.rados.2.smithi175.stdout:2335: snap_remove snap 253 2024-09-11T03:47:09.299 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:09.320 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:09.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:09.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:09.419 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:09 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:09 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2780579760' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:47:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:09 smithi121 ceph-mon[30729]: osdmap e610: 12 total, 12 up, 12 in 2024-09-11T03:47:09.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:09.445 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:09.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:09.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:09 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:09 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2780579760' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:47:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:09 smithi175 ceph-mon[30894]: osdmap e610: 12 total, 12 up, 12 in 2024-09-11T03:47:09.547 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:09.567 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:09.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:09.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:09.637 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:47:09.637 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:47:09.637 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:47:09.637 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:47:09.669 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:09.690 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:09.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:09.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:09.792 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:09.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:09.815 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:09.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:09.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:09.941 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:09.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:09.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:10.030 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.8 to 1 (10000) 2024-09-11T03:47:10.034 INFO:tasks.rados.rados.2.smithi175.stdout:2335: done (2 left) 2024-09-11T03:47:10.034 INFO:tasks.rados.rados.2.smithi175.stdout:2336: snap_create 2024-09-11T03:47:10.034 INFO:tasks.rados.rados.2.smithi175.stdout:2333: finishing write tid 3 to smithi17557870-22 2024-09-11T03:47:10.034 INFO:tasks.rados.rados.2.smithi175.stdout:2333: finishing write tid 4 to smithi17557870-22 2024-09-11T03:47:10.034 INFO:tasks.rados.rados.2.smithi175.stdout:2333: finishing write tid 5 to smithi17557870-22 2024-09-11T03:47:10.034 INFO:tasks.rados.rados.2.smithi175.stdout:2333: oid 22 updating version 0 to 912 2024-09-11T03:47:10.034 INFO:tasks.rados.rados.2.smithi175.stdout:2333: oid 22 updating version 912 to 913 2024-09-11T03:47:10.034 INFO:tasks.rados.rados.2.smithi175.stdout:2333: oid 22 version 913 is already newer than 911 2024-09-11T03:47:10.034 INFO:tasks.rados.rados.2.smithi175.stdout:2333: oid 22 version 913 is already newer than 910 2024-09-11T03:47:10.035 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 913 (ObjNum 808 snap 255 seq_num 808) dirty exists 2024-09-11T03:47:10.035 INFO:tasks.rados.rados.2.smithi175.stdout:2333: left oid 22 (ObjNum 808 snap 255 seq_num 808) 2024-09-11T03:47:10.035 INFO:tasks.rados.rados.2.smithi175.stdout:2334: finishing write tid 1 to smithi17557870-24 2024-09-11T03:47:10.035 INFO:tasks.rados.rados.2.smithi175.stdout:2334: finishing write tid 2 to smithi17557870-24 2024-09-11T03:47:10.035 INFO:tasks.rados.rados.2.smithi175.stdout:2334: finishing write tid 3 to smithi17557870-24 2024-09-11T03:47:10.035 INFO:tasks.rados.rados.2.smithi175.stdout:2334: finishing write tid 4 to smithi17557870-24 2024-09-11T03:47:10.035 INFO:tasks.rados.rados.2.smithi175.stdout:2334: finishing write tid 5 to smithi17557870-24 2024-09-11T03:47:10.035 INFO:tasks.rados.rados.2.smithi175.stdout:2334: oid 24 updating version 0 to 887 2024-09-11T03:47:10.035 INFO:tasks.rados.rados.2.smithi175.stdout:2334: oid 24 version 887 is already newer than 886 2024-09-11T03:47:10.035 INFO:tasks.rados.rados.2.smithi175.stdout:2334: oid 24 version 887 is already newer than 885 2024-09-11T03:47:10.035 INFO:tasks.rados.rados.2.smithi175.stdout:2334: oid 24 version 887 is already newer than 884 2024-09-11T03:47:10.035 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 887 (ObjNum 809 snap 255 seq_num 809) dirty exists 2024-09-11T03:47:10.035 INFO:tasks.rados.rados.2.smithi175.stdout:2334: left oid 24 (ObjNum 809 snap 255 seq_num 809) 2024-09-11T03:47:10.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:10.066 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:10.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:10.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:10.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:10.189 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:10.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:10.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:10.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:10 smithi084 ceph-mon[27254]: pgmap v1071: 67 pgs: 67 active+clean; 130 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.8 MiB/s wr, 25 op/s 2024-09-11T03:47:10.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:10 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:47:10.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:10 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:47:10.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:10 smithi084 ceph-mon[27254]: osdmap e611: 12 total, 12 up, 12 in 2024-09-11T03:47:10.276 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:10 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4133478622' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:47:10.291 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:10.313 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:10.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:10.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:10.378 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:10 smithi175 ceph-mon[30894]: pgmap v1071: 67 pgs: 67 active+clean; 130 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.8 MiB/s wr, 25 op/s 2024-09-11T03:47:10.378 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:10 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:47:10.378 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:10 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:47:10.378 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:10 smithi175 ceph-mon[30894]: osdmap e611: 12 total, 12 up, 12 in 2024-09-11T03:47:10.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:10 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4133478622' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:47:10.415 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:10 smithi121 ceph-mon[30729]: pgmap v1071: 67 pgs: 67 active+clean; 130 MiB data, 704 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.8 MiB/s wr, 25 op/s 2024-09-11T03:47:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:10 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:47:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:10 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:47:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:10 smithi121 ceph-mon[30729]: osdmap e611: 12 total, 12 up, 12 in 2024-09-11T03:47:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4133478622' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:47:10.437 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:10.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:10.438 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:10.539 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:10.560 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:10.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:10.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:10.662 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:10.688 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:10.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:10.790 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:10.811 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:10.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:10.913 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:10.915 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 9 1 2024-09-11T03:47:10.931 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:47:10.938 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:10.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:10.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:11.040 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:11.043 INFO:tasks.rados.rados.2.smithi175.stdout:2333: done (2 left) 2024-09-11T03:47:11.044 INFO:tasks.rados.rados.2.smithi175.stdout:2334: done (1 left) 2024-09-11T03:47:11.044 INFO:tasks.rados.rados.2.smithi175.stdout:2336: done (0 left) 2024-09-11T03:47:11.044 INFO:tasks.rados.rados.2.smithi175.stdout:2337: write (excl) oid 35 current snap is 256 2024-09-11T03:47:11.044 INFO:tasks.rados.rados.2.smithi175.stdout:2337: seq_num 810 ranges {413670=485108,1465944=729943,2977330=524801} 2024-09-11T03:47:11.048 INFO:tasks.rados.rados.2.smithi175.stdout:2337: writing smithi17557870-35 from 413670 to 898778 tid 1 2024-09-11T03:47:11.055 INFO:tasks.rados.rados.2.smithi175.stdout:2337: writing smithi17557870-35 from 1465944 to 2195887 tid 2 2024-09-11T03:47:11.062 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:11.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:11.064 INFO:tasks.rados.rados.2.smithi175.stdout:2337: writing smithi17557870-35 from 2977330 to 3502131 tid 3 2024-09-11T03:47:11.064 INFO:tasks.rados.rados.2.smithi175.stdout:2338: read oid 16 snap -1 2024-09-11T03:47:11.064 INFO:tasks.rados.rados.2.smithi175.stdout:2338: expect (ObjNum 780 snap 249 seq_num 780) 2024-09-11T03:47:11.064 INFO:tasks.rados.rados.2.smithi175.stdout:2339: write (excl) oid 24 current snap is 256 2024-09-11T03:47:11.064 INFO:tasks.rados.rados.2.smithi175.stdout:2337: finishing write tid 1 to smithi17557870-35 2024-09-11T03:47:11.064 INFO:tasks.rados.rados.2.smithi175.stdout:2337: finishing write tid 2 to smithi17557870-35 2024-09-11T03:47:11.064 INFO:tasks.rados.rados.2.smithi175.stdout:2339: seq_num 811 ranges {430477=560357,1408073=473601,2488400=736073} 2024-09-11T03:47:11.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:11.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:11 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4133478622' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:47:11.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:11 smithi084 ceph-mon[27254]: osdmap e612: 12 total, 12 up, 12 in 2024-09-11T03:47:11.067 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T03:47:11.068 INFO:tasks.rados.rados.2.smithi175.stdout:2339: writing smithi17557870-24 from 430477 to 990834 tid 1 2024-09-11T03:47:11.073 INFO:tasks.rados.rados.2.smithi175.stdout:2339: writing smithi17557870-24 from 1408073 to 1881674 tid 2 2024-09-11T03:47:11.081 INFO:tasks.rados.rados.2.smithi175.stdout:2339: writing smithi17557870-24 from 2488400 to 3224473 tid 3 2024-09-11T03:47:11.081 INFO:tasks.rados.rados.2.smithi175.stdout:2340: rollback oid 5 current snap is 256 2024-09-11T03:47:11.081 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 5 to 255 2024-09-11T03:47:11.081 INFO:tasks.rados.rados.2.smithi175.stdout:2337: finishing write tid 3 to smithi17557870-35 2024-09-11T03:47:11.081 INFO:tasks.rados.rados.2.smithi175.stdout:2337: finishing write tid 4 to smithi17557870-35 2024-09-11T03:47:11.081 INFO:tasks.rados.rados.2.smithi175.stdout:2337: finishing write tid 5 to smithi17557870-35 2024-09-11T03:47:11.082 INFO:tasks.rados.rados.2.smithi175.stdout:2337: oid 35 updating version 0 to 680 2024-09-11T03:47:11.082 INFO:tasks.rados.rados.2.smithi175.stdout:2337: oid 35 version 680 is already newer than 679 2024-09-11T03:47:11.082 INFO:tasks.rados.rados.2.smithi175.stdout:2337: oid 35 updating version 680 to 681 2024-09-11T03:47:11.082 INFO:tasks.rados.rados.2.smithi175.stdout:2337: oid 35 version 681 is already newer than 678 2024-09-11T03:47:11.082 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 681 (ObjNum 810 snap 256 seq_num 810) dirty exists 2024-09-11T03:47:11.082 INFO:tasks.rados.rados.2.smithi175.stdout:2337: left oid 35 (ObjNum 810 snap 256 seq_num 810) 2024-09-11T03:47:11.082 INFO:tasks.rados.rados.2.smithi175.stdout:2338: expect (ObjNum 780 snap 249 seq_num 780) 2024-09-11T03:47:11.166 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:11.186 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:11.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:11.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:11 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4133478622' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:47:11.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:11 smithi175 ceph-mon[30894]: osdmap e612: 12 total, 12 up, 12 in 2024-09-11T03:47:11.237 INFO:tasks.rados.rados.2.smithi175.stdout:2339: finishing write tid 1 to smithi17557870-24 2024-09-11T03:47:11.237 INFO:tasks.rados.rados.2.smithi175.stdout:2339: finishing write tid 2 to smithi17557870-24 2024-09-11T03:47:11.238 INFO:tasks.rados.rados.2.smithi175.stdout:2337: done (3 left) 2024-09-11T03:47:11.238 INFO:tasks.rados.rados.2.smithi175.stdout:2338: done (2 left) 2024-09-11T03:47:11.238 INFO:tasks.rados.rados.2.smithi175.stdout:2341: read oid 44 snap -1 2024-09-11T03:47:11.238 INFO:tasks.rados.rados.2.smithi175.stdout:2341: expect (ObjNum 787 snap 249 seq_num 787) 2024-09-11T03:47:11.238 INFO:tasks.rados.rados.2.smithi175.stdout:2342: read oid 9 snap 255 2024-09-11T03:47:11.238 INFO:tasks.rados.rados.2.smithi175.stdout:2342: expect (ObjNum 776 snap 248 seq_num 776) 2024-09-11T03:47:11.238 INFO:tasks.rados.rados.2.smithi175.stdout:2343: copy_from oid 30 from oid 46 current snap is 256 2024-09-11T03:47:11.238 INFO:tasks.rados.rados.2.smithi175.stdout:2340: finishing rollback tid 0 to smithi17557870-5 2024-09-11T03:47:11.238 INFO:tasks.rados.rados.2.smithi175.stdout:2339: finishing write tid 3 to smithi17557870-24 2024-09-11T03:47:11.238 INFO:tasks.rados.rados.2.smithi175.stdout:2340: finishing rollback tid 1 to smithi17557870-5 2024-09-11T03:47:11.238 INFO:tasks.rados.rados.2.smithi175.stdout:2340: finishing rollback tid 2 to smithi17557870-5 2024-09-11T03:47:11.238 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 714 (ObjNum 796 snap 254 seq_num 796) dirty exists 2024-09-11T03:47:11.238 INFO:tasks.rados.rados.2.smithi175.stdout:2339: finishing write tid 4 to smithi17557870-24 2024-09-11T03:47:11.238 INFO:tasks.rados.rados.2.smithi175.stdout:2339: finishing write tid 5 to smithi17557870-24 2024-09-11T03:47:11.238 INFO:tasks.rados.rados.2.smithi175.stdout:2339: oid 24 updating version 0 to 892 2024-09-11T03:47:11.239 INFO:tasks.rados.rados.2.smithi175.stdout:2339: oid 24 updating version 892 to 894 2024-09-11T03:47:11.239 INFO:tasks.rados.rados.2.smithi175.stdout:2339: oid 24 version 894 is already newer than 893 2024-09-11T03:47:11.239 INFO:tasks.rados.rados.2.smithi175.stdout:2339: oid 24 version 894 is already newer than 891 2024-09-11T03:47:11.239 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 894 (ObjNum 811 snap 256 seq_num 811) dirty exists 2024-09-11T03:47:11.239 INFO:tasks.rados.rados.2.smithi175.stdout:2339: left oid 24 (ObjNum 811 snap 256 seq_num 811) 2024-09-11T03:47:11.239 INFO:tasks.rados.rados.2.smithi175.stdout:2339: done (4 left) 2024-09-11T03:47:11.239 INFO:tasks.rados.rados.2.smithi175.stdout:2340: done (3 left) 2024-09-11T03:47:11.239 INFO:tasks.rados.rados.2.smithi175.stdout:2344: snap_remove snap 254 2024-09-11T03:47:11.289 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:11.310 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:11.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:11.311 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:11.412 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:11.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:11 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4133478622' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:47:11.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:11 smithi121 ceph-mon[30729]: osdmap e612: 12 total, 12 up, 12 in 2024-09-11T03:47:11.438 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:11.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:11.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:11.565 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:11.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:11.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:11.567 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:47:11.567 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:47:11.582 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:47:11.583 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:47:11.598 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:47:11.599 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:47:11.622 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:47:11.623 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:47:11.645 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:11.645 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:11.646 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:47:11.646 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:47:11.649 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:11.667 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:11.671 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:47:11.671 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:47:11.690 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:47:11.691 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:47:11.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:11.692 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:11.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:11.707 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:47:11.707 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:47:11.724 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:47:11.724 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:47:11.741 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:47:11.741 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:47:11.757 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:47:11.758 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:47:11.774 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:47:11.775 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:47:11.793 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:11.815 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:11.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:11.816 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:11.917 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:11.938 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:11.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:11.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:11.978 INFO:tasks.rados.rados.2.smithi175.stdout:2344: done (3 left) 2024-09-11T03:47:11.978 INFO:tasks.rados.rados.2.smithi175.stdout:2345: read oid 32 snap -1 2024-09-11T03:47:11.978 INFO:tasks.rados.rados.2.smithi175.stdout:2345: expect (ObjNum 721 snap 236 seq_num 721) 2024-09-11T03:47:11.978 INFO:tasks.rados.rados.2.smithi175.stdout:2346: copy_from oid 14 from oid 22 current snap is 256 2024-09-11T03:47:11.978 INFO:tasks.rados.rados.2.smithi175.stdout:2343: finishing copy_from racing read to smithi17557870-30 2024-09-11T03:47:11.978 INFO:tasks.rados.rados.2.smithi175.stdout:2343: finishing copy_from to smithi17557870-30 2024-09-11T03:47:11.978 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 882 (ObjNum 805 snap 255 seq_num 805) dirty exists 2024-09-11T03:47:11.978 INFO:tasks.rados.rados.2.smithi175.stdout:2342: expect (ObjNum 776 snap 248 seq_num 776) 2024-09-11T03:47:12.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:12.064 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:12.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:12.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:12.120 INFO:tasks.rados.rados.2.smithi175.stdout:2341: expect (ObjNum 787 snap 249 seq_num 787) 2024-09-11T03:47:12.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:12.190 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:12.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:12.292 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:12.301 INFO:tasks.rados.rados.2.smithi175.stdout:2345: expect (ObjNum 721 snap 236 seq_num 721) 2024-09-11T03:47:12.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:12 smithi084 ceph-mon[27254]: pgmap v1074: 67 pgs: 67 active+clean; 130 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.8 MiB/s wr, 11 op/s 2024-09-11T03:47:12.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:12 smithi084 ceph-mon[27254]: osdmap e613: 12 total, 12 up, 12 in 2024-09-11T03:47:12.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:12 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3573051799' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:12.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:12 smithi084 ceph-mon[27254]: osdmap e614: 12 total, 12 up, 12 in 2024-09-11T03:47:12.316 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:12.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:12.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:12.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:12.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:12 smithi121 ceph-mon[30729]: pgmap v1074: 67 pgs: 67 active+clean; 130 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.8 MiB/s wr, 11 op/s 2024-09-11T03:47:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:12 smithi121 ceph-mon[30729]: osdmap e613: 12 total, 12 up, 12 in 2024-09-11T03:47:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:12 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3573051799' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:12 smithi121 ceph-mon[30729]: osdmap e614: 12 total, 12 up, 12 in 2024-09-11T03:47:12.441 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:12.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:12.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:12.475 INFO:tasks.rados.rados.2.smithi175.stdout:2346: finishing copy_from racing read to smithi17557870-14 2024-09-11T03:47:12.476 INFO:tasks.rados.rados.2.smithi175.stdout:2346: finishing copy_from to smithi17557870-14 2024-09-11T03:47:12.476 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 913 (ObjNum 808 snap 255 seq_num 808) dirty exists 2024-09-11T03:47:12.476 INFO:tasks.rados.rados.2.smithi175.stdout:2341: done (4 left) 2024-09-11T03:47:12.476 INFO:tasks.rados.rados.2.smithi175.stdout:2342: done (3 left) 2024-09-11T03:47:12.476 INFO:tasks.rados.rados.2.smithi175.stdout:2343: done (2 left) 2024-09-11T03:47:12.476 INFO:tasks.rados.rados.2.smithi175.stdout:2345: done (1 left) 2024-09-11T03:47:12.476 INFO:tasks.rados.rados.2.smithi175.stdout:2346: done (0 left) 2024-09-11T03:47:12.476 INFO:tasks.rados.rados.2.smithi175.stdout:2347: snap_remove snap 255 2024-09-11T03:47:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:12 smithi175 ceph-mon[30894]: pgmap v1074: 67 pgs: 67 active+clean; 130 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.8 MiB/s wr, 11 op/s 2024-09-11T03:47:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:12 smithi175 ceph-mon[30894]: osdmap e613: 12 total, 12 up, 12 in 2024-09-11T03:47:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:12 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3573051799' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:12 smithi175 ceph-mon[30894]: osdmap e614: 12 total, 12 up, 12 in 2024-09-11T03:47:12.543 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:12.569 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:12.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:12.671 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:12.692 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:12.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:12.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:12.794 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:12.816 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:12.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:12.817 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:12.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:12.939 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:12.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:13.001 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:47:13.001 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:47:13.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:13.042 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.9 to 1 (10000) 2024-09-11T03:47:13.046 INFO:tasks.rados.rados.2.smithi175.stdout:2347: done (0 left) 2024-09-11T03:47:13.046 INFO:tasks.rados.rados.2.smithi175.stdout:2348: read oid 18 snap -1 2024-09-11T03:47:13.046 INFO:tasks.rados.rados.2.smithi175.stdout:2348: expect (ObjNum 806 snap 255 seq_num 806) 2024-09-11T03:47:13.046 INFO:tasks.rados.rados.2.smithi175.stdout:2349: write oid 28 current snap is 256 2024-09-11T03:47:13.046 INFO:tasks.rados.rados.2.smithi175.stdout:2349: seq_num 812 ranges {447284=635606,1866555=733612,3315823=331461} 2024-09-11T03:47:13.051 INFO:tasks.rados.rados.2.smithi175.stdout:2349: writing smithi17557870-28 from 447284 to 1082890 tid 1 2024-09-11T03:47:13.061 INFO:tasks.rados.rados.2.smithi175.stdout:2349: writing smithi17557870-28 from 1866555 to 2600167 tid 2 2024-09-11T03:47:13.067 INFO:tasks.rados.rados.2.smithi175.stdout:2349: writing smithi17557870-28 from 3315823 to 3647284 tid 3 2024-09-11T03:47:13.067 INFO:tasks.rados.rados.2.smithi175.stdout:2350: write (excl) oid 12 current snap is 256 2024-09-11T03:47:13.067 INFO:tasks.rados.rados.2.smithi175.stdout:2350: seq_num 813 ranges {464091=710855,1925037=477270,2943246=474970} 2024-09-11T03:47:13.068 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:13.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:13.073 INFO:tasks.rados.rados.2.smithi175.stdout:2350: writing smithi17557870-12 from 464091 to 1174946 tid 1 2024-09-11T03:47:13.079 INFO:tasks.rados.rados.2.smithi175.stdout:2350: writing smithi17557870-12 from 1925037 to 2402307 tid 2 2024-09-11T03:47:13.085 INFO:tasks.rados.rados.2.smithi175.stdout:2350: writing smithi17557870-12 from 2943246 to 3418216 tid 3 2024-09-11T03:47:13.086 INFO:tasks.rados.rados.2.smithi175.stdout:2351: delete oid 39 current snap is 256 2024-09-11T03:47:13.086 INFO:tasks.rados.rados.2.smithi175.stdout:2349: finishing write tid 1 to smithi17557870-28 2024-09-11T03:47:13.086 INFO:tasks.rados.rados.2.smithi175.stdout:2348: expect (ObjNum 806 snap 255 seq_num 806) 2024-09-11T03:47:13.170 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:13.191 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:13.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:13.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:13.198 INFO:tasks.rados.rados.2.smithi175.stdout:2349: finishing write tid 2 to smithi17557870-28 2024-09-11T03:47:13.198 INFO:tasks.rados.rados.2.smithi175.stdout:2349: finishing write tid 3 to smithi17557870-28 2024-09-11T03:47:13.198 INFO:tasks.rados.rados.2.smithi175.stdout:2349: finishing write tid 4 to smithi17557870-28 2024-09-11T03:47:13.198 INFO:tasks.rados.rados.2.smithi175.stdout:2349: finishing write tid 5 to smithi17557870-28 2024-09-11T03:47:13.198 INFO:tasks.rados.rados.2.smithi175.stdout:2349: oid 28 updating version 0 to 917 2024-09-11T03:47:13.198 INFO:tasks.rados.rados.2.smithi175.stdout:2349: oid 28 version 917 is already newer than 915 2024-09-11T03:47:13.199 INFO:tasks.rados.rados.2.smithi175.stdout:2349: oid 28 version 917 is already newer than 914 2024-09-11T03:47:13.199 INFO:tasks.rados.rados.2.smithi175.stdout:2349: oid 28 version 917 is already newer than 916 2024-09-11T03:47:13.199 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 917 (ObjNum 812 snap 256 seq_num 812) dirty exists 2024-09-11T03:47:13.199 INFO:tasks.rados.rados.2.smithi175.stdout:2349: left oid 28 (ObjNum 812 snap 256 seq_num 812) 2024-09-11T03:47:13.199 INFO:tasks.rados.rados.2.smithi175.stdout:2350: finishing write tid 1 to smithi17557870-12 2024-09-11T03:47:13.199 INFO:tasks.rados.rados.2.smithi175.stdout:2350: finishing write tid 2 to smithi17557870-12 2024-09-11T03:47:13.199 INFO:tasks.rados.rados.2.smithi175.stdout:2348: done (3 left) 2024-09-11T03:47:13.199 INFO:tasks.rados.rados.2.smithi175.stdout:2349: done (2 left) 2024-09-11T03:47:13.199 INFO:tasks.rados.rados.2.smithi175.stdout:2351: done (1 left) 2024-09-11T03:47:13.199 INFO:tasks.rados.rados.2.smithi175.stdout:2352: delete oid 16 current snap is 256 2024-09-11T03:47:13.199 INFO:tasks.rados.rados.2.smithi175.stdout:2350: finishing write tid 3 to smithi17557870-12 2024-09-11T03:47:13.199 INFO:tasks.rados.rados.2.smithi175.stdout:2350: finishing write tid 4 to smithi17557870-12 2024-09-11T03:47:13.199 INFO:tasks.rados.rados.2.smithi175.stdout:2350: finishing write tid 5 to smithi17557870-12 2024-09-11T03:47:13.199 INFO:tasks.rados.rados.2.smithi175.stdout:2350: oid 12 updating version 0 to 888 2024-09-11T03:47:13.200 INFO:tasks.rados.rados.2.smithi175.stdout:2350: oid 12 version 888 is already newer than 886 2024-09-11T03:47:13.200 INFO:tasks.rados.rados.2.smithi175.stdout:2350: oid 12 version 888 is already newer than 887 2024-09-11T03:47:13.200 INFO:tasks.rados.rados.2.smithi175.stdout:2350: oid 12 version 888 is already newer than 885 2024-09-11T03:47:13.200 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 888 (ObjNum 813 snap 256 seq_num 813) dirty exists 2024-09-11T03:47:13.200 INFO:tasks.rados.rados.2.smithi175.stdout:2350: left oid 12 (ObjNum 813 snap 256 seq_num 813) 2024-09-11T03:47:13.200 INFO:tasks.rados.rados.2.smithi175.stdout:2350: done (1 left) 2024-09-11T03:47:13.201 INFO:tasks.rados.rados.2.smithi175.stdout:2352: done (0 left) 2024-09-11T03:47:13.201 INFO:tasks.rados.rados.2.smithi175.stdout:2353: read oid 10 snap -1 2024-09-11T03:47:13.201 INFO:tasks.rados.rados.2.smithi175.stdout:2353: expect (ObjNum 726 snap 238 seq_num 726) 2024-09-11T03:47:13.201 INFO:tasks.rados.rados.2.smithi175.stdout:2354: rollback oid 28 current snap is 256 2024-09-11T03:47:13.201 INFO:tasks.rados.rados.2.smithi175.stdout:2354: done (1 left) 2024-09-11T03:47:13.201 INFO:tasks.rados.rados.2.smithi175.stdout:2355: write (excl) oid 43 current snap is 256 2024-09-11T03:47:13.201 INFO:tasks.rados.rados.2.smithi175.stdout:2355: seq_num 816 ranges {514512=652955,1700483=740950,3025515=688997} 2024-09-11T03:47:13.206 INFO:tasks.rados.rados.2.smithi175.stdout:2355: writing smithi17557870-43 from 514512 to 1167467 tid 1 2024-09-11T03:47:13.213 INFO:tasks.rados.rados.2.smithi175.stdout:2355: writing smithi17557870-43 from 1700483 to 2441433 tid 2 2024-09-11T03:47:13.220 INFO:tasks.rados.rados.2.smithi175.stdout:2355: writing smithi17557870-43 from 3025515 to 3714512 tid 3 2024-09-11T03:47:13.220 INFO:tasks.rados.rados.2.smithi175.stdout:2356: read oid 39 snap -1 2024-09-11T03:47:13.220 INFO:tasks.rados.rados.2.smithi175.stdout:2356: expect deleted 2024-09-11T03:47:13.220 INFO:tasks.rados.rados.2.smithi175.stdout:2357: snap_create 2024-09-11T03:47:13.220 INFO:tasks.rados.rados.2.smithi175.stdout:2353: expect (ObjNum 726 snap 238 seq_num 726) 2024-09-11T03:47:13.291 INFO:tasks.rados.rados.2.smithi175.stdout:2355: finishing write tid 1 to smithi17557870-43 2024-09-11T03:47:13.291 INFO:tasks.rados.rados.2.smithi175.stdout:2355: finishing write tid 2 to smithi17557870-43 2024-09-11T03:47:13.291 INFO:tasks.rados.rados.2.smithi175.stdout:2355: finishing write tid 3 to smithi17557870-43 2024-09-11T03:47:13.292 INFO:tasks.rados.rados.2.smithi175.stdout:2355: finishing write tid 4 to smithi17557870-43 2024-09-11T03:47:13.292 INFO:tasks.rados.rados.2.smithi175.stdout:2355: finishing write tid 5 to smithi17557870-43 2024-09-11T03:47:13.292 INFO:tasks.rados.rados.2.smithi175.stdout:2355: oid 43 updating version 0 to 435 2024-09-11T03:47:13.292 INFO:tasks.rados.rados.2.smithi175.stdout:2355: oid 43 version 435 is already newer than 433 2024-09-11T03:47:13.292 INFO:tasks.rados.rados.2.smithi175.stdout:2355: oid 43 updating version 435 to 436 2024-09-11T03:47:13.292 INFO:tasks.rados.rados.2.smithi175.stdout:2355: oid 43 version 436 is already newer than 434 2024-09-11T03:47:13.292 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 436 (ObjNum 816 snap 256 seq_num 816) dirty exists 2024-09-11T03:47:13.292 INFO:tasks.rados.rados.2.smithi175.stdout:2355: left oid 43 (ObjNum 816 snap 256 seq_num 816) 2024-09-11T03:47:13.292 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:13.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:13 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3219600715' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:47:13.315 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:13.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:13.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:13.417 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:13.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:13 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3219600715' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:47:13.439 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:13.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:13.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:13 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3219600715' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:47:13.541 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:13.562 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:13.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:13.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:13.664 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:13.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:13.686 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:13.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:13.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T03:47:13.788 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:13.810 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:13.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:13.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:13.912 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:13.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:13.934 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:13.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:14.035 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:14.053 INFO:tasks.rados.rados.2.smithi175.stdout:2353: done (3 left) 2024-09-11T03:47:14.053 INFO:tasks.rados.rados.2.smithi175.stdout:2355: done (2 left) 2024-09-11T03:47:14.053 INFO:tasks.rados.rados.2.smithi175.stdout:2356: done (1 left) 2024-09-11T03:47:14.053 INFO:tasks.rados.rados.2.smithi175.stdout:2357: done (0 left) 2024-09-11T03:47:14.053 INFO:tasks.rados.rados.2.smithi175.stdout:2358: snap_remove snap 256 2024-09-11T03:47:14.057 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:14.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:14.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:14.159 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:14.180 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:14.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:14.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:14.282 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:14.308 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:14.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:14.309 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:14.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:14 smithi084 ceph-mon[27254]: pgmap v1077: 67 pgs: 67 active+clean; 138 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-09-11T03:47:14.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:14 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3219600715' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:47:14.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:14 smithi084 ceph-mon[27254]: osdmap e615: 12 total, 12 up, 12 in 2024-09-11T03:47:14.409 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:14.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:14 smithi121 ceph-mon[30729]: pgmap v1077: 67 pgs: 67 active+clean; 138 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-09-11T03:47:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:14 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3219600715' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:47:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:14 smithi121 ceph-mon[30729]: osdmap e615: 12 total, 12 up, 12 in 2024-09-11T03:47:14.431 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:14.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:14.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:14 smithi175 ceph-mon[30894]: pgmap v1077: 67 pgs: 67 active+clean; 138 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-09-11T03:47:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:14 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3219600715' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:47:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:14 smithi175 ceph-mon[30894]: osdmap e615: 12 total, 12 up, 12 in 2024-09-11T03:47:14.532 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:14.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:14.554 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:14.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:14.655 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:14.676 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:14.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:14.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:14.745 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 11 1 2024-09-11T03:47:14.778 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:14.809 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:14.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:14.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:14.911 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:14.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:14.934 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:14.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:15.035 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:15.054 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:47:15.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:15 smithi084 ceph-mon[27254]: osdmap e616: 12 total, 12 up, 12 in 2024-09-11T03:47:15.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:15 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2665559627' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:47:15.058 INFO:tasks.rados.rados.2.smithi175.stdout:2358: done (0 left) 2024-09-11T03:47:15.058 INFO:tasks.rados.rados.2.smithi175.stdout:2359: delete oid 28 current snap is 257 2024-09-11T03:47:15.062 INFO:tasks.rados.rados.2.smithi175.stdout:2359: done (0 left) 2024-09-11T03:47:15.062 INFO:tasks.rados.rados.2.smithi175.stdout:2360: read oid 48 snap -1 2024-09-11T03:47:15.062 INFO:tasks.rados.rados.2.smithi175.stdout:2360: expect (ObjNum 793 snap 252 seq_num 793) 2024-09-11T03:47:15.062 INFO:tasks.rados.rados.2.smithi175.stdout:2361: snap_create 2024-09-11T03:47:15.063 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:15.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:15.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:15.073 INFO:tasks.rados.rados.2.smithi175.stdout:2360: expect (ObjNum 793 snap 252 seq_num 793) 2024-09-11T03:47:15.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:15.190 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:15.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:15.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:15.291 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:15.317 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:15.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:15.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:15.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:15 smithi121 ceph-mon[30729]: osdmap e616: 12 total, 12 up, 12 in 2024-09-11T03:47:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:15 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2665559627' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:47:15.443 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:15.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:15.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:15 smithi175 ceph-mon[30894]: osdmap e616: 12 total, 12 up, 12 in 2024-09-11T03:47:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:15 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2665559627' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:47:15.545 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:15.566 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:15.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:15.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:15.668 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:15.689 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:15.690 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:15.780 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:15.791 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:15.814 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:15.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:15.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:15.916 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:15.938 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:15.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:15.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:16.040 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:16.051 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:16.061 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:16.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:16.062 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:16.066 INFO:tasks.rados.rados.2.smithi175.stdout:2360: done (1 left) 2024-09-11T03:47:16.066 INFO:tasks.rados.rados.2.smithi175.stdout:2361: done (0 left) 2024-09-11T03:47:16.066 INFO:tasks.rados.rados.2.smithi175.stdout:2362: copy_from oid 26 from oid 1 current snap is 258 2024-09-11T03:47:16.066 INFO:tasks.rados.rados.2.smithi175.stdout:2363: delete oid 40 current snap is 258 2024-09-11T03:47:16.069 INFO:tasks.rados.rados.2.smithi175.stdout:2363: done (1 left) 2024-09-11T03:47:16.069 INFO:tasks.rados.rados.2.smithi175.stdout:2364: copy_from oid 23 from oid 4 current snap is 258 2024-09-11T03:47:16.070 INFO:tasks.rados.rados.2.smithi175.stdout:2365: delete oid 6 current snap is 258 2024-09-11T03:47:16.073 INFO:tasks.rados.rados.2.smithi175.stdout:2365: done (2 left) 2024-09-11T03:47:16.073 INFO:tasks.rados.rados.2.smithi175.stdout:2366: delete oid 39 current snap is 258 2024-09-11T03:47:16.076 INFO:tasks.rados.rados.2.smithi175.stdout:2366: done (2 left) 2024-09-11T03:47:16.076 INFO:tasks.rados.rados.2.smithi175.stdout:2367: write oid 14 current snap is 258 2024-09-11T03:47:16.076 INFO:tasks.rados.rados.2.smithi175.stdout:2367: seq_num 821 ranges {598547=629200,1876540=491946,3046277=681499} 2024-09-11T03:47:16.084 INFO:tasks.rados.rados.2.smithi175.stdout:2367: writing smithi17557870-14 from 598547 to 1227747 tid 1 2024-09-11T03:47:16.091 INFO:tasks.rados.rados.2.smithi175.stdout:2367: writing smithi17557870-14 from 1876540 to 2368486 tid 2 2024-09-11T03:47:16.098 INFO:tasks.rados.rados.2.smithi175.stdout:2367: writing smithi17557870-14 from 3046277 to 3727776 tid 3 2024-09-11T03:47:16.098 INFO:tasks.rados.rados.2.smithi175.stdout:2368: snap_create 2024-09-11T03:47:16.098 INFO:tasks.rados.rados.2.smithi175.stdout:2362: finishing copy_from racing read to smithi17557870-26 2024-09-11T03:47:16.098 INFO:tasks.rados.rados.2.smithi175.stdout:2362: finishing copy_from to smithi17557870-26 2024-09-11T03:47:16.098 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 868 (ObjNum 761 snap 247 seq_num 761) dirty exists 2024-09-11T03:47:16.099 INFO:tasks.rados.rados.2.smithi175.stdout:2367: finishing write tid 1 to smithi17557870-14 2024-09-11T03:47:16.099 INFO:tasks.rados.rados.2.smithi175.stdout:2364: finishing copy_from racing read to smithi17557870-23 2024-09-11T03:47:16.099 INFO:tasks.rados.rados.2.smithi175.stdout:2367: finishing write tid 2 to smithi17557870-14 2024-09-11T03:47:16.099 INFO:tasks.rados.rados.2.smithi175.stdout:2364: finishing copy_from to smithi17557870-23 2024-09-11T03:47:16.099 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 845 (ObjNum 807 snap 255 seq_num 807) dirty exists 2024-09-11T03:47:16.101 INFO:tasks.rados.rados.2.smithi175.stdout:2367: finishing write tid 3 to smithi17557870-14 2024-09-11T03:47:16.101 INFO:tasks.rados.rados.2.smithi175.stdout:2367: finishing write tid 4 to smithi17557870-14 2024-09-11T03:47:16.101 INFO:tasks.rados.rados.2.smithi175.stdout:2367: finishing write tid 5 to smithi17557870-14 2024-09-11T03:47:16.101 INFO:tasks.rados.rados.2.smithi175.stdout:2367: oid 14 updating version 0 to 916 2024-09-11T03:47:16.101 INFO:tasks.rados.rados.2.smithi175.stdout:2367: oid 14 version 916 is already newer than 914 2024-09-11T03:47:16.102 INFO:tasks.rados.rados.2.smithi175.stdout:2367: oid 14 updating version 916 to 917 2024-09-11T03:47:16.102 INFO:tasks.rados.rados.2.smithi175.stdout:2367: oid 14 version 917 is already newer than 915 2024-09-11T03:47:16.102 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 917 (ObjNum 821 snap 258 seq_num 821) dirty exists 2024-09-11T03:47:16.102 INFO:tasks.rados.rados.2.smithi175.stdout:2367: left oid 14 (ObjNum 821 snap 258 seq_num 821) 2024-09-11T03:47:16.163 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:16.188 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:16.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:16.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:16.291 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:16.313 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:16.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:16.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:16.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:47:16.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:47:16.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: pgmap v1080: 67 pgs: 67 active+clean; 139 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 5.7 MiB/s wr, 28 op/s 2024-09-11T03:47:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2665559627' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:47:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: osdmap e617: 12 total, 12 up, 12 in 2024-09-11T03:47:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: 2.6 scrub starts 2024-09-11T03:47:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: 2.6 scrub ok 2024-09-11T03:47:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=815733923,config=358384381,health=3122926494,kv=535792000}) 2024-09-11T03:47:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2302442024}) 2024-09-11T03:47:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=672841921}) 2024-09-11T03:47:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {kv=12,logm=88} crc {kv=460972491,logm=2997641686}) 2024-09-11T03:47:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3273538568}) 2024-09-11T03:47:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=45079649}) 2024-09-11T03:47:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3923531350}) 2024-09-11T03:47:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=310867574}) 2024-09-11T03:47:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3651530295}) 2024-09-11T03:47:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2173370324}) 2024-09-11T03:47:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3260737999}) 2024-09-11T03:47:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214616237}) 2024-09-11T03:47:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2165311186}) 2024-09-11T03:47:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=219309494}) 2024-09-11T03:47:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=596079086}) 2024-09-11T03:47:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1479922630}) 2024-09-11T03:47:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=394175278}) 2024-09-11T03:47:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1505254752}) 2024-09-11T03:47:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=190661897}) 2024-09-11T03:47:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1447619940}) 2024-09-11T03:47:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=833843382}) 2024-09-11T03:47:16.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3769935618}) 2024-09-11T03:47:16.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3907847984}) 2024-09-11T03:47:16.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2522260886}) 2024-09-11T03:47:16.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1862887304}) 2024-09-11T03:47:16.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=709075728}) 2024-09-11T03:47:16.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=520480488}) 2024-09-11T03:47:16.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1972221501}) 2024-09-11T03:47:16.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2578243330}) 2024-09-11T03:47:16.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3351499580}) 2024-09-11T03:47:16.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846810674}) 2024-09-11T03:47:16.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=415972015}) 2024-09-11T03:47:16.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3918824758}) 2024-09-11T03:47:16.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1801616326}) 2024-09-11T03:47:16.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1104892171}) 2024-09-11T03:47:16.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3119477199}) 2024-09-11T03:47:16.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2146836785}) 2024-09-11T03:47:16.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1142138332}) 2024-09-11T03:47:16.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1751979371}) 2024-09-11T03:47:16.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3865600578}) 2024-09-11T03:47:16.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3035528210}) 2024-09-11T03:47:16.327 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2283992338}) 2024-09-11T03:47:16.327 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=298396799}) 2024-09-11T03:47:16.327 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3895507629}) 2024-09-11T03:47:16.327 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3377800464}) 2024-09-11T03:47:16.327 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=900245296}) 2024-09-11T03:47:16.327 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=72,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=4} crc {logm=673645274,mdsmap=827622067,mgr=2786812108,mgr_command_descs=899306875,mgr_metadata=2702562194,mgrstat=261699561}) 2024-09-11T03:47:16.327 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=3316512334}) 2024-09-11T03:47:16.327 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=2588282169}) 2024-09-11T03:47:16.327 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=9,mon_config_key=91} crc {mgrstat=605345000,mon_config_key=2268427478}) 2024-09-11T03:47:16.327 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1089463772}) 2024-09-11T03:47:16.327 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=73,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osd_snap=6} crc {mon_config_key=3375284452,monmap=4220057526,nvmeofgw=598337686,osd_metadata=132375096,osd_pg_creating=2437009871,osd_snap=1577123235}) 2024-09-11T03:47:16.327 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=100} crc {osd_snap=3309974537}) 2024-09-11T03:47:16.327 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=3,osdmap=97} crc {osd_snap=3993163560,osdmap=764625503}) 2024-09-11T03:47:16.327 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4159111082}) 2024-09-11T03:47:16.328 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1427564552}) 2024-09-11T03:47:16.328 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2738037709}) 2024-09-11T03:47:16.328 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2086835996}) 2024-09-11T03:47:16.328 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1757530891}) 2024-09-11T03:47:16.328 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1971662565}) 2024-09-11T03:47:16.328 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1022250793}) 2024-09-11T03:47:16.328 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3050055340}) 2024-09-11T03:47:16.328 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3404259771}) 2024-09-11T03:47:16.328 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3788717077}) 2024-09-11T03:47:16.328 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2288196151}) 2024-09-11T03:47:16.328 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=40} crc {osdmap=478376831}) 2024-09-11T03:47:16.415 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:47:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:47:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: pgmap v1080: 67 pgs: 67 active+clean; 139 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 5.7 MiB/s wr, 28 op/s 2024-09-11T03:47:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2665559627' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:47:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: osdmap e617: 12 total, 12 up, 12 in 2024-09-11T03:47:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: 2.6 scrub starts 2024-09-11T03:47:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: 2.6 scrub ok 2024-09-11T03:47:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=815733923,config=358384381,health=3122926494,kv=535792000}) 2024-09-11T03:47:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2302442024}) 2024-09-11T03:47:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=672841921}) 2024-09-11T03:47:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {kv=12,logm=88} crc {kv=460972491,logm=2997641686}) 2024-09-11T03:47:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3273538568}) 2024-09-11T03:47:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=45079649}) 2024-09-11T03:47:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3923531350}) 2024-09-11T03:47:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=310867574}) 2024-09-11T03:47:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3651530295}) 2024-09-11T03:47:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2173370324}) 2024-09-11T03:47:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3260737999}) 2024-09-11T03:47:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214616237}) 2024-09-11T03:47:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2165311186}) 2024-09-11T03:47:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=219309494}) 2024-09-11T03:47:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=596079086}) 2024-09-11T03:47:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1479922630}) 2024-09-11T03:47:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=394175278}) 2024-09-11T03:47:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1505254752}) 2024-09-11T03:47:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=190661897}) 2024-09-11T03:47:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1447619940}) 2024-09-11T03:47:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=833843382}) 2024-09-11T03:47:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3769935618}) 2024-09-11T03:47:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3907847984}) 2024-09-11T03:47:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2522260886}) 2024-09-11T03:47:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1862887304}) 2024-09-11T03:47:16.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=709075728}) 2024-09-11T03:47:16.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=520480488}) 2024-09-11T03:47:16.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1972221501}) 2024-09-11T03:47:16.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2578243330}) 2024-09-11T03:47:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3351499580}) 2024-09-11T03:47:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846810674}) 2024-09-11T03:47:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=415972015}) 2024-09-11T03:47:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3918824758}) 2024-09-11T03:47:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1801616326}) 2024-09-11T03:47:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1104892171}) 2024-09-11T03:47:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3119477199}) 2024-09-11T03:47:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2146836785}) 2024-09-11T03:47:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1142138332}) 2024-09-11T03:47:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1751979371}) 2024-09-11T03:47:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3865600578}) 2024-09-11T03:47:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3035528210}) 2024-09-11T03:47:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2283992338}) 2024-09-11T03:47:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=298396799}) 2024-09-11T03:47:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3895507629}) 2024-09-11T03:47:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3377800464}) 2024-09-11T03:47:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=900245296}) 2024-09-11T03:47:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=72,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=4} crc {logm=673645274,mdsmap=827622067,mgr=2786812108,mgr_command_descs=899306875,mgr_metadata=2702562194,mgrstat=261699561}) 2024-09-11T03:47:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=3316512334}) 2024-09-11T03:47:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=2588282169}) 2024-09-11T03:47:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=9,mon_config_key=91} crc {mgrstat=605345000,mon_config_key=2268427478}) 2024-09-11T03:47:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1089463772}) 2024-09-11T03:47:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=73,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osd_snap=6} crc {mon_config_key=3375284452,monmap=4220057526,nvmeofgw=598337686,osd_metadata=132375096,osd_pg_creating=2437009871,osd_snap=1577123235}) 2024-09-11T03:47:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=100} crc {osd_snap=3309974537}) 2024-09-11T03:47:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=3,osdmap=97} crc {osd_snap=3993163560,osdmap=764625503}) 2024-09-11T03:47:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4159111082}) 2024-09-11T03:47:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1427564552}) 2024-09-11T03:47:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2738037709}) 2024-09-11T03:47:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2086835996}) 2024-09-11T03:47:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1757530891}) 2024-09-11T03:47:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1971662565}) 2024-09-11T03:47:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1022250793}) 2024-09-11T03:47:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3050055340}) 2024-09-11T03:47:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3404259771}) 2024-09-11T03:47:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3788717077}) 2024-09-11T03:47:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2288196151}) 2024-09-11T03:47:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=40} crc {osdmap=478376831}) 2024-09-11T03:47:16.437 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:16.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:16.438 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:47:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:47:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: pgmap v1080: 67 pgs: 67 active+clean; 139 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 5.7 MiB/s wr, 28 op/s 2024-09-11T03:47:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2665559627' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:47:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: osdmap e617: 12 total, 12 up, 12 in 2024-09-11T03:47:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: 2.6 scrub starts 2024-09-11T03:47:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: 2.6 scrub ok 2024-09-11T03:47:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=815733923,config=358384381,health=3122926494,kv=535792000}) 2024-09-11T03:47:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2302442024}) 2024-09-11T03:47:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=672841921}) 2024-09-11T03:47:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {kv=12,logm=88} crc {kv=460972491,logm=2997641686}) 2024-09-11T03:47:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3273538568}) 2024-09-11T03:47:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=45079649}) 2024-09-11T03:47:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3923531350}) 2024-09-11T03:47:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=310867574}) 2024-09-11T03:47:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3651530295}) 2024-09-11T03:47:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2173370324}) 2024-09-11T03:47:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3260737999}) 2024-09-11T03:47:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214616237}) 2024-09-11T03:47:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2165311186}) 2024-09-11T03:47:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=219309494}) 2024-09-11T03:47:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=596079086}) 2024-09-11T03:47:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1479922630}) 2024-09-11T03:47:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=394175278}) 2024-09-11T03:47:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1505254752}) 2024-09-11T03:47:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=190661897}) 2024-09-11T03:47:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1447619940}) 2024-09-11T03:47:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=833843382}) 2024-09-11T03:47:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3769935618}) 2024-09-11T03:47:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3907847984}) 2024-09-11T03:47:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2522260886}) 2024-09-11T03:47:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1862887304}) 2024-09-11T03:47:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=709075728}) 2024-09-11T03:47:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=520480488}) 2024-09-11T03:47:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1972221501}) 2024-09-11T03:47:16.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2578243330}) 2024-09-11T03:47:16.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3351499580}) 2024-09-11T03:47:16.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846810674}) 2024-09-11T03:47:16.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=415972015}) 2024-09-11T03:47:16.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3918824758}) 2024-09-11T03:47:16.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1801616326}) 2024-09-11T03:47:16.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1104892171}) 2024-09-11T03:47:16.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3119477199}) 2024-09-11T03:47:16.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2146836785}) 2024-09-11T03:47:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1142138332}) 2024-09-11T03:47:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1751979371}) 2024-09-11T03:47:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3865600578}) 2024-09-11T03:47:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3035528210}) 2024-09-11T03:47:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2283992338}) 2024-09-11T03:47:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=298396799}) 2024-09-11T03:47:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3895507629}) 2024-09-11T03:47:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3377800464}) 2024-09-11T03:47:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=900245296}) 2024-09-11T03:47:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=72,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=4} crc {logm=673645274,mdsmap=827622067,mgr=2786812108,mgr_command_descs=899306875,mgr_metadata=2702562194,mgrstat=261699561}) 2024-09-11T03:47:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=3316512334}) 2024-09-11T03:47:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=2588282169}) 2024-09-11T03:47:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=9,mon_config_key=91} crc {mgrstat=605345000,mon_config_key=2268427478}) 2024-09-11T03:47:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1089463772}) 2024-09-11T03:47:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=73,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osd_snap=6} crc {mon_config_key=3375284452,monmap=4220057526,nvmeofgw=598337686,osd_metadata=132375096,osd_pg_creating=2437009871,osd_snap=1577123235}) 2024-09-11T03:47:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=100} crc {osd_snap=3309974537}) 2024-09-11T03:47:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=3,osdmap=97} crc {osd_snap=3993163560,osdmap=764625503}) 2024-09-11T03:47:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4159111082}) 2024-09-11T03:47:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1427564552}) 2024-09-11T03:47:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2738037709}) 2024-09-11T03:47:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2086835996}) 2024-09-11T03:47:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1757530891}) 2024-09-11T03:47:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1971662565}) 2024-09-11T03:47:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1022250793}) 2024-09-11T03:47:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3050055340}) 2024-09-11T03:47:16.484 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3404259771}) 2024-09-11T03:47:16.484 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3788717077}) 2024-09-11T03:47:16.484 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2288196151}) 2024-09-11T03:47:16.484 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=40} crc {osdmap=478376831}) 2024-09-11T03:47:16.539 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:16.557 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:47:16.558 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:47:16.565 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:16.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:16.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:16.667 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:16.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:16.690 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:16.690 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:16.791 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:16.813 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:16.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:16.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:16.915 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:16.939 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:16.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:16.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:16.976 INFO:tasks.rados.rados.2.smithi175.stdout:2362: done (3 left) 2024-09-11T03:47:16.976 INFO:tasks.rados.rados.2.smithi175.stdout:2364: done (2 left) 2024-09-11T03:47:16.976 INFO:tasks.rados.rados.2.smithi175.stdout:2367: done (1 left) 2024-09-11T03:47:16.976 INFO:tasks.rados.rados.2.smithi175.stdout:2368: done (0 left) 2024-09-11T03:47:16.976 INFO:tasks.rados.rados.2.smithi175.stdout:2369: read oid 19 snap -1 2024-09-11T03:47:16.976 INFO:tasks.rados.rados.2.smithi175.stdout:2369: expect deleted 2024-09-11T03:47:16.977 INFO:tasks.rados.rados.2.smithi175.stdout:2370: snap_remove snap 257 2024-09-11T03:47:17.040 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:17.061 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:17.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:17.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:17.118 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:47:17.163 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:17.185 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:17.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:17.186 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:17.287 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:17.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:17 smithi084 ceph-mon[27254]: osdmap e618: 12 total, 12 up, 12 in 2024-09-11T03:47:17.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:17 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:47:17.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:17 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:47:17.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:17 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:47:17.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:17 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:47:17.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:17 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:47:17.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:17 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:47:17.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:17 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:47:17.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:17 smithi084 ceph-mon[27254]: osdmap e619: 12 total, 12 up, 12 in 2024-09-11T03:47:17.308 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:17.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:17.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:17.329 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T03:47:17.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:17.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:17 smithi121 ceph-mon[30729]: osdmap e618: 12 total, 12 up, 12 in 2024-09-11T03:47:17.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:17 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:47:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:17 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:47:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:17 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:47:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:17 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:47:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:17 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:47:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:17 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:47:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:17 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:47:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:17 smithi121 ceph-mon[30729]: osdmap e619: 12 total, 12 up, 12 in 2024-09-11T03:47:17.435 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:17.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:17.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:17 smithi175 ceph-mon[30894]: osdmap e618: 12 total, 12 up, 12 in 2024-09-11T03:47:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:17 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:47:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:17 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:47:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:17 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:47:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:17 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:47:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:17 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:47:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:17 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:47:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:17 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:47:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:17 smithi175 ceph-mon[30894]: osdmap e619: 12 total, 12 up, 12 in 2024-09-11T03:47:17.536 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:17.559 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:17.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:17.661 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:17.684 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:17.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:17.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:17.698 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:17.702 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:17.773 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:47:17.786 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:17.836 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:17.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:17.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:17.938 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:17.960 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:17.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:17.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:17.975 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.11 to 1 (10000) 2024-09-11T03:47:17.980 INFO:tasks.rados.rados.2.smithi175.stdout:2370: done (1 left) 2024-09-11T03:47:17.980 INFO:tasks.rados.rados.2.smithi175.stdout:2371: delete oid 50 current snap is 259 2024-09-11T03:47:17.985 INFO:tasks.rados.rados.2.smithi175.stdout:2369: done (1 left) 2024-09-11T03:47:17.985 INFO:tasks.rados.rados.2.smithi175.stdout:2371: done (0 left) 2024-09-11T03:47:17.985 INFO:tasks.rados.rados.2.smithi175.stdout:2372: snap_create 2024-09-11T03:47:18.061 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:18.086 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:18.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:18.087 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:18.130 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:18 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1826586336' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:47:18.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:18 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1826586336' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:47:18.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:18 smithi084 ceph-mon[27254]: osdmap e620: 12 total, 12 up, 12 in 2024-09-11T03:47:18.188 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:18.214 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:18.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:18.215 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:18.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:18.342 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:18.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:18.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:18.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:18 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1826586336' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:47:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:18 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1826586336' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:47:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:18 smithi121 ceph-mon[30729]: osdmap e620: 12 total, 12 up, 12 in 2024-09-11T03:47:18.443 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:18.469 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:18.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:18.470 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:18 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1826586336' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:47:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:18 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1826586336' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:47:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:18 smithi175 ceph-mon[30894]: osdmap e620: 12 total, 12 up, 12 in 2024-09-11T03:47:18.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:18.597 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:18.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:18.699 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:18.723 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:18.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:18.765 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 10 1 2024-09-11T03:47:18.825 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:18.851 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:18.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:18.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:18.953 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:18.971 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:47:18.972 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:47:18.978 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:18.980 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:18.981 INFO:tasks.rados.rados.2.smithi175.stdout:2372: done (0 left) 2024-09-11T03:47:18.981 INFO:tasks.rados.rados.2.smithi175.stdout:2373: snap_remove snap 258 2024-09-11T03:47:18.992 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:47:18.992 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:47:19.009 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:47:19.009 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:47:19.029 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:47:19.029 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:47:19.047 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:47:19.047 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:47:19.063 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:47:19.063 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:47:19.080 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:19.081 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:47:19.081 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:47:19.098 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:47:19.098 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:47:19.102 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:19.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:19.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:19.115 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:47:19.115 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:47:19.132 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:47:19.133 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:47:19.149 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:47:19.150 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:47:19.167 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:47:19.167 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:47:19.204 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:19.230 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:19.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:19.231 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:19.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:19 smithi084 ceph-mon[27254]: 1.13 scrub starts 2024-09-11T03:47:19.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:19 smithi084 ceph-mon[27254]: 1.13 scrub ok 2024-09-11T03:47:19.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:19 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3648762348' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:19.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:19 smithi084 ceph-mon[27254]: osdmap e621: 12 total, 12 up, 12 in 2024-09-11T03:47:19.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:19.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:19.360 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:19.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:19 smithi121 ceph-mon[30729]: 1.13 scrub starts 2024-09-11T03:47:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:19 smithi121 ceph-mon[30729]: 1.13 scrub ok 2024-09-11T03:47:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:19 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3648762348' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:19 smithi121 ceph-mon[30729]: osdmap e621: 12 total, 12 up, 12 in 2024-09-11T03:47:19.462 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:19 smithi175 ceph-mon[30894]: 1.13 scrub starts 2024-09-11T03:47:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:19 smithi175 ceph-mon[30894]: 1.13 scrub ok 2024-09-11T03:47:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:19 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3648762348' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:19 smithi175 ceph-mon[30894]: osdmap e621: 12 total, 12 up, 12 in 2024-09-11T03:47:19.484 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:19.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:19.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:19.586 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:19.607 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:19.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:19.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:19.733 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:19.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:19.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:19.835 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:19.861 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:19.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:19.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:19.963 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:19.983 INFO:tasks.rados.rados.2.smithi175.stdout:2373: done (0 left) 2024-09-11T03:47:19.983 INFO:tasks.rados.rados.2.smithi175.stdout:2374: copy_from oid 42 from oid 6 current snap is 260 2024-09-11T03:47:19.984 INFO:tasks.rados.rados.2.smithi175.stdout:2375: snap_remove snap 259 2024-09-11T03:47:19.991 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:19.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:19.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:20.092 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:20.108 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:47:20.108 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:47:20.108 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:47:20.109 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:47:20.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:20.117 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:20.117 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:20.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:20 smithi084 ceph-mon[27254]: pgmap v1086: 67 pgs: 67 active+clean; 132 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 18 op/s 2024-09-11T03:47:20.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:20 smithi084 ceph-mon[27254]: 2.10 scrub starts 2024-09-11T03:47:20.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:20 smithi084 ceph-mon[27254]: 2.10 scrub ok 2024-09-11T03:47:20.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:20 smithi084 ceph-mon[27254]: osdmap e622: 12 total, 12 up, 12 in 2024-09-11T03:47:20.120 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:20.218 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:20.242 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:20.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:20.243 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:20.344 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:20.354 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:20.370 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:20.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:20.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:20.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:20 smithi121 ceph-mon[30729]: pgmap v1086: 67 pgs: 67 active+clean; 132 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 18 op/s 2024-09-11T03:47:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:20 smithi121 ceph-mon[30729]: 2.10 scrub starts 2024-09-11T03:47:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:20 smithi121 ceph-mon[30729]: 2.10 scrub ok 2024-09-11T03:47:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:20 smithi121 ceph-mon[30729]: osdmap e622: 12 total, 12 up, 12 in 2024-09-11T03:47:20.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:20 smithi175 ceph-mon[30894]: pgmap v1086: 67 pgs: 67 active+clean; 132 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 18 op/s 2024-09-11T03:47:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:20 smithi175 ceph-mon[30894]: 2.10 scrub starts 2024-09-11T03:47:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:20 smithi175 ceph-mon[30894]: 2.10 scrub ok 2024-09-11T03:47:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:20 smithi175 ceph-mon[30894]: osdmap e622: 12 total, 12 up, 12 in 2024-09-11T03:47:20.496 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:20.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:20.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:20.598 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:20.624 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:20.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:20.625 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:20.726 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:20.753 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:20.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:20.754 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:20.855 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:20.879 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:20.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:20.889 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T03:47:20.982 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:20.983 INFO:tasks.rados.rados.2.smithi175.stdout:2375: done (1 left) 2024-09-11T03:47:20.983 INFO:tasks.rados.rados.2.smithi175.stdout:2376: delete oid 30 current snap is 260 2024-09-11T03:47:20.983 INFO:tasks.rados.rados.2.smithi175.stdout:2374: finishing copy_from to smithi17557870-42 2024-09-11T03:47:20.983 INFO:tasks.rados.rados.2.smithi175.stdout:2374: got expected ENOENT (src dne) 2024-09-11T03:47:20.983 INFO:tasks.rados.rados.2.smithi175.stdout:2374: finishing copy_from racing read to smithi17557870-42 2024-09-11T03:47:20.987 INFO:tasks.rados.rados.2.smithi175.stdout:2374: done (1 left) 2024-09-11T03:47:20.987 INFO:tasks.rados.rados.2.smithi175.stdout:2376: done (0 left) 2024-09-11T03:47:20.987 INFO:tasks.rados.rados.2.smithi175.stdout:2377: delete oid 21 current snap is 260 2024-09-11T03:47:20.991 INFO:tasks.rados.rados.2.smithi175.stdout:2377: done (0 left) 2024-09-11T03:47:20.991 INFO:tasks.rados.rados.2.smithi175.stdout:2378: delete oid 34 current snap is 260 2024-09-11T03:47:20.993 INFO:tasks.rados.rados.2.smithi175.stdout:2378: done (0 left) 2024-09-11T03:47:20.994 INFO:tasks.rados.rados.2.smithi175.stdout:2379: rollback oid 3 current snap is 260 2024-09-11T03:47:20.994 INFO:tasks.rados.rados.2.smithi175.stdout:2379: done (0 left) 2024-09-11T03:47:20.994 INFO:tasks.rados.rados.2.smithi175.stdout:2380: delete oid 26 current snap is 260 2024-09-11T03:47:20.995 INFO:tasks.rados.rados.2.smithi175.stdout:2380: done (0 left) 2024-09-11T03:47:20.996 INFO:tasks.rados.rados.2.smithi175.stdout:2381: read oid 35 snap -1 2024-09-11T03:47:20.996 INFO:tasks.rados.rados.2.smithi175.stdout:2381: expect (ObjNum 810 snap 256 seq_num 810) 2024-09-11T03:47:20.996 INFO:tasks.rados.rados.2.smithi175.stdout:2382: snap_create 2024-09-11T03:47:21.002 INFO:tasks.rados.rados.2.smithi175.stdout:2381: expect (ObjNum 810 snap 256 seq_num 810) 2024-09-11T03:47:21.004 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:21.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:21.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:21.106 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:21.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:21.128 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:21.129 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:21.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:21.255 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:21.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:21.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:21.357 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:21.379 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:21.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:21.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:21.507 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:21.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:21.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:21.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:21.635 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:21.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:21.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:21.737 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:21.759 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:21.761 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:21.854 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:21.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:21.886 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:21.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:21.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:21.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:47:21.974 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:47:21.980 INFO:tasks.rados.rados.2.smithi175.stdout:2381: done (1 left) 2024-09-11T03:47:21.980 INFO:tasks.rados.rados.2.smithi175.stdout:2382: done (0 left) 2024-09-11T03:47:21.980 INFO:tasks.rados.rados.2.smithi175.stdout:2383: snap_remove snap 260 2024-09-11T03:47:21.988 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:22.010 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:22.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:22.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:22.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:21 smithi084 ceph-mon[27254]: 2.11 scrub starts 2024-09-11T03:47:22.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:21 smithi084 ceph-mon[27254]: 2.11 scrub ok 2024-09-11T03:47:22.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:21 smithi084 ceph-mon[27254]: pgmap v1089: 67 pgs: 67 active+clean; 132 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.3 MiB/s wr, 3 op/s 2024-09-11T03:47:22.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:21 smithi084 ceph-mon[27254]: osdmap e623: 12 total, 12 up, 12 in 2024-09-11T03:47:22.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1492403458' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:47:22.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:21 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:47:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:21 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:47:22.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1492403458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:47:22.137 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:22.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:22.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:22.239 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:22.261 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:22.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:22.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:22.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:22.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:22.390 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:22.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:22.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:21 smithi121 ceph-mon[30729]: 2.11 scrub starts 2024-09-11T03:47:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:21 smithi121 ceph-mon[30729]: 2.11 scrub ok 2024-09-11T03:47:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:21 smithi121 ceph-mon[30729]: pgmap v1089: 67 pgs: 67 active+clean; 132 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.3 MiB/s wr, 3 op/s 2024-09-11T03:47:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:21 smithi121 ceph-mon[30729]: osdmap e623: 12 total, 12 up, 12 in 2024-09-11T03:47:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:21 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1492403458' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:47:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:21 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:47:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:21 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:47:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:21 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1492403458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:47:22.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:21 smithi175 ceph-mon[30894]: 2.11 scrub starts 2024-09-11T03:47:22.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:21 smithi175 ceph-mon[30894]: 2.11 scrub ok 2024-09-11T03:47:22.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:21 smithi175 ceph-mon[30894]: pgmap v1089: 67 pgs: 67 active+clean; 132 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.3 MiB/s wr, 3 op/s 2024-09-11T03:47:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:21 smithi175 ceph-mon[30894]: osdmap e623: 12 total, 12 up, 12 in 2024-09-11T03:47:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:21 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1492403458' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:47:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:21 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:47:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:21 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:47:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:21 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1492403458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:47:22.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:22.515 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:22.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:22.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:22.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:22.643 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:22.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:22.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:22.745 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:22.771 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:22.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:22.772 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:22.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:22.899 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:22.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:22.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:22.976 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.10 to 1 (10000) 2024-09-11T03:47:22.981 INFO:tasks.rados.rados.2.smithi175.stdout:2383: done (0 left) 2024-09-11T03:47:22.981 INFO:tasks.rados.rados.2.smithi175.stdout:2384: read oid 17 snap -1 2024-09-11T03:47:22.981 INFO:tasks.rados.rados.2.smithi175.stdout:2384: expect (ObjNum 799 snap 254 seq_num 799) 2024-09-11T03:47:22.981 INFO:tasks.rados.rados.2.smithi175.stdout:2385: write (excl) oid 23 current snap is 261 2024-09-11T03:47:22.981 INFO:tasks.rados.rados.2.smithi175.stdout:2385: seq_num 827 ranges {699389=797047,2111079=502953,3094462=581837} 2024-09-11T03:47:22.983 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:47:22.983 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:47:22.983 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:47:22.984 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:47:22.984 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:47:22.989 INFO:tasks.rados.rados.2.smithi175.stdout:2385: writing smithi17557870-23 from 699389 to 1496436 tid 1 2024-09-11T03:47:22.996 INFO:tasks.rados.rados.2.smithi175.stdout:2385: writing smithi17557870-23 from 2111079 to 2614032 tid 2 2024-09-11T03:47:23.000 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:23.002 INFO:tasks.rados.rados.2.smithi175.stdout:2385: writing smithi17557870-23 from 3094462 to 3676299 tid 3 2024-09-11T03:47:23.003 INFO:tasks.rados.rados.2.smithi175.stdout:2386: copy_from oid 10 from oid 29 current snap is 261 2024-09-11T03:47:23.003 INFO:tasks.rados.rados.2.smithi175.stdout:2384: expect (ObjNum 799 snap 254 seq_num 799) 2024-09-11T03:47:23.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:23.025 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:23.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:23.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:23.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:22 smithi084 ceph-mon[27254]: osdmap e624: 12 total, 12 up, 12 in 2024-09-11T03:47:23.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:22 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/85572833' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:47:23.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:22 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/85572833' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:47:23.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:22 smithi084 ceph-mon[27254]: osdmap e625: 12 total, 12 up, 12 in 2024-09-11T03:47:23.151 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:23.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:23.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:23.152 INFO:tasks.rados.rados.2.smithi175.stdout:2385: finishing write tid 1 to smithi17557870-23 2024-09-11T03:47:23.152 INFO:tasks.rados.rados.2.smithi175.stdout:2385: finishing write tid 2 to smithi17557870-23 2024-09-11T03:47:23.153 INFO:tasks.rados.rados.2.smithi175.stdout:2384: done (2 left) 2024-09-11T03:47:23.153 INFO:tasks.rados.rados.2.smithi175.stdout:2387: read oid 18 snap -1 2024-09-11T03:47:23.153 INFO:tasks.rados.rados.2.smithi175.stdout:2387: expect (ObjNum 806 snap 255 seq_num 806) 2024-09-11T03:47:23.153 INFO:tasks.rados.rados.2.smithi175.stdout:2388: read oid 47 snap -1 2024-09-11T03:47:23.153 INFO:tasks.rados.rados.2.smithi175.stdout:2388: expect (ObjNum 762 snap 247 seq_num 762) 2024-09-11T03:47:23.153 INFO:tasks.rados.rados.2.smithi175.stdout:2389: delete oid 5 current snap is 261 2024-09-11T03:47:23.153 INFO:tasks.rados.rados.2.smithi175.stdout:2385: finishing write tid 3 to smithi17557870-23 2024-09-11T03:47:23.153 INFO:tasks.rados.rados.2.smithi175.stdout:2385: finishing write tid 4 to smithi17557870-23 2024-09-11T03:47:23.153 INFO:tasks.rados.rados.2.smithi175.stdout:2385: finishing write tid 5 to smithi17557870-23 2024-09-11T03:47:23.153 INFO:tasks.rados.rados.2.smithi175.stdout:2385: oid 23 updating version 0 to 849 2024-09-11T03:47:23.153 INFO:tasks.rados.rados.2.smithi175.stdout:2385: oid 23 version 849 is already newer than 847 2024-09-11T03:47:23.153 INFO:tasks.rados.rados.2.smithi175.stdout:2385: oid 23 version 849 is already newer than 848 2024-09-11T03:47:23.153 INFO:tasks.rados.rados.2.smithi175.stdout:2385: oid 23 version 849 is already newer than 846 2024-09-11T03:47:23.154 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 849 (ObjNum 827 snap 261 seq_num 827) dirty exists 2024-09-11T03:47:23.154 INFO:tasks.rados.rados.2.smithi175.stdout:2385: left oid 23 (ObjNum 827 snap 261 seq_num 827) 2024-09-11T03:47:23.154 INFO:tasks.rados.rados.2.smithi175.stdout:2386: finishing copy_from racing read to smithi17557870-10 2024-09-11T03:47:23.154 INFO:tasks.rados.rados.2.smithi175.stdout:2386: finishing copy_from to smithi17557870-10 2024-09-11T03:47:23.154 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 884 (ObjNum 736 snap 243 seq_num 736) dirty exists 2024-09-11T03:47:23.154 INFO:tasks.rados.rados.2.smithi175.stdout:2385: done (4 left) 2024-09-11T03:47:23.154 INFO:tasks.rados.rados.2.smithi175.stdout:2386: done (3 left) 2024-09-11T03:47:23.154 INFO:tasks.rados.rados.2.smithi175.stdout:2389: done (2 left) 2024-09-11T03:47:23.155 INFO:tasks.rados.rados.2.smithi175.stdout:2390: copy_from oid 6 from oid 11 current snap is 261 2024-09-11T03:47:23.155 INFO:tasks.rados.rados.2.smithi175.stdout:2391: snap_create 2024-09-11T03:47:23.161 INFO:tasks.rados.rados.2.smithi175.stdout:2390: finishing copy_from racing read to smithi17557870-6 2024-09-11T03:47:23.162 INFO:tasks.rados.rados.2.smithi175.stdout:2388: expect (ObjNum 762 snap 247 seq_num 762) 2024-09-11T03:47:23.252 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:23.253 INFO:tasks.rados.rados.2.smithi175.stdout:2387: expect (ObjNum 806 snap 255 seq_num 806) 2024-09-11T03:47:23.273 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:23.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:23.274 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:23.365 INFO:tasks.rados.rados.2.smithi175.stdout:2390: finishing copy_from to smithi17557870-6 2024-09-11T03:47:23.365 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 600 (ObjNum 609 snap 191 seq_num 609) dirty exists 2024-09-11T03:47:23.375 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:23.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:23.398 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:23.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:23.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:22 smithi121 ceph-mon[30729]: osdmap e624: 12 total, 12 up, 12 in 2024-09-11T03:47:23.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:22 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/85572833' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:47:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:22 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/85572833' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:47:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:22 smithi121 ceph-mon[30729]: osdmap e625: 12 total, 12 up, 12 in 2024-09-11T03:47:23.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:22 smithi175 ceph-mon[30894]: osdmap e624: 12 total, 12 up, 12 in 2024-09-11T03:47:23.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:22 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/85572833' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:47:23.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:22 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/85572833' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:47:23.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:22 smithi175 ceph-mon[30894]: osdmap e625: 12 total, 12 up, 12 in 2024-09-11T03:47:23.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:23.520 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:23.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:23.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:23.622 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:23.643 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:23.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:23.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:23.661 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-11T03:47:23.744 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:23.765 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:23.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:23.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:23.868 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:23.894 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:23.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:23.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:23.994 INFO:tasks.rados.rados.2.smithi175.stdout:2387: done (3 left) 2024-09-11T03:47:23.994 INFO:tasks.rados.rados.2.smithi175.stdout:2388: done (2 left) 2024-09-11T03:47:23.994 INFO:tasks.rados.rados.2.smithi175.stdout:2390: done (1 left) 2024-09-11T03:47:23.994 INFO:tasks.rados.rados.2.smithi175.stdout:2391: done (0 left) 2024-09-11T03:47:23.994 INFO:tasks.rados.rados.2.smithi175.stdout:2392: snap_create 2024-09-11T03:47:23.996 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:24.017 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:24.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:24.018 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:24.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:24.144 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:24.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:24.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:24.246 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:24.268 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:24.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:24.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:24.339 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:23 smithi084 ceph-mon[27254]: 1.1d scrub starts 2024-09-11T03:47:24.339 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:23 smithi084 ceph-mon[27254]: 1.1d scrub ok 2024-09-11T03:47:24.339 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:23 smithi084 ceph-mon[27254]: 1.9 scrub starts 2024-09-11T03:47:24.339 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:23 smithi084 ceph-mon[27254]: 1.9 scrub ok 2024-09-11T03:47:24.339 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:23 smithi084 ceph-mon[27254]: pgmap v1092: 67 pgs: 67 active+clean; 118 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 440 KiB/s wr, 4 op/s 2024-09-11T03:47:24.366 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T03:47:24.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:24.396 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:24.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:24.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:24.413 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:23 smithi121 ceph-mon[30729]: 1.1d scrub starts 2024-09-11T03:47:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:23 smithi121 ceph-mon[30729]: 1.1d scrub ok 2024-09-11T03:47:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:23 smithi121 ceph-mon[30729]: 1.9 scrub starts 2024-09-11T03:47:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:23 smithi121 ceph-mon[30729]: 1.9 scrub ok 2024-09-11T03:47:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:23 smithi121 ceph-mon[30729]: pgmap v1092: 67 pgs: 67 active+clean; 118 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 440 KiB/s wr, 4 op/s 2024-09-11T03:47:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:23 smithi175 ceph-mon[30894]: 1.1d scrub starts 2024-09-11T03:47:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:23 smithi175 ceph-mon[30894]: 1.1d scrub ok 2024-09-11T03:47:24.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:23 smithi175 ceph-mon[30894]: 1.9 scrub starts 2024-09-11T03:47:24.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:23 smithi175 ceph-mon[30894]: 1.9 scrub ok 2024-09-11T03:47:24.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:23 smithi175 ceph-mon[30894]: pgmap v1092: 67 pgs: 67 active+clean; 118 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 440 KiB/s wr, 4 op/s 2024-09-11T03:47:24.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:24.523 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:24.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:24.625 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:24.647 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:24.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:24.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:24.748 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:24.774 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:24.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:24.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:24.816 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:47:24.875 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:24.897 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:24.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:24.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:24.918 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:47:24.918 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:47:24.936 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:47:24.936 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:47:24.969 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:47:24.970 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:47:24.981 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:24.981 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:24.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:25.003 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:47:25.003 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:47:25.009 INFO:tasks.rados.rados.2.smithi175.stdout:2392: done (0 left) 2024-09-11T03:47:25.009 INFO:tasks.rados.rados.2.smithi175.stdout:2393: read oid 2 snap 262 2024-09-11T03:47:25.009 INFO:tasks.rados.rados.2.smithi175.stdout:2393: expect (ObjNum 765 snap 247 seq_num 765) 2024-09-11T03:47:25.009 INFO:tasks.rados.rados.2.smithi175.stdout:2394: copy_from oid 32 from oid 42 current snap is 263 2024-09-11T03:47:25.010 INFO:tasks.rados.rados.2.smithi175.stdout:2395: read oid 36 snap -1 2024-09-11T03:47:25.010 INFO:tasks.rados.rados.2.smithi175.stdout:2395: expect (ObjNum 757 snap 246 seq_num 757) 2024-09-11T03:47:25.010 INFO:tasks.rados.rados.2.smithi175.stdout:2396: write oid 29 current snap is 263 2024-09-11T03:47:25.010 INFO:tasks.rados.rados.2.smithi175.stdout:2396: seq_num 829 ranges {733003=663898,1944396=506622,2865661=604381} 2024-09-11T03:47:25.020 INFO:tasks.rados.rados.2.smithi175.stdout:2396: writing smithi17557870-29 from 733003 to 1396901 tid 1 2024-09-11T03:47:25.025 INFO:tasks.rados.rados.2.smithi175.stdout:2396: writing smithi17557870-29 from 1944396 to 2451018 tid 2 2024-09-11T03:47:25.026 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:25.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:25.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:25.030 INFO:tasks.rados.rados.2.smithi175.stdout:2396: writing smithi17557870-29 from 2865661 to 3470042 tid 3 2024-09-11T03:47:25.031 INFO:tasks.rados.rados.2.smithi175.stdout:2397: copy_from oid 31 from oid 50 current snap is 263 2024-09-11T03:47:25.031 INFO:tasks.rados.rados.2.smithi175.stdout:2395: expect (ObjNum 757 snap 246 seq_num 757) 2024-09-11T03:47:25.037 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:47:25.037 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:47:25.058 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:47:25.059 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:47:25.075 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:47:25.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:47:25.092 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:47:25.092 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:47:25.119 INFO:tasks.rados.rados.2.smithi175.stdout:2394: finishing copy_from racing read to smithi17557870-32 2024-09-11T03:47:25.119 INFO:tasks.rados.rados.2.smithi175.stdout:2396: finishing write tid 1 to smithi17557870-29 2024-09-11T03:47:25.119 INFO:tasks.rados.rados.2.smithi175.stdout:2393: expect (ObjNum 765 snap 247 seq_num 765) 2024-09-11T03:47:25.128 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:25.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:25.149 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:25.149 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:25.232 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:25 smithi084 ceph-mon[27254]: 1.12 scrub starts 2024-09-11T03:47:25.232 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:25 smithi084 ceph-mon[27254]: 1.12 scrub ok 2024-09-11T03:47:25.232 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:25 smithi084 ceph-mon[27254]: osdmap e626: 12 total, 12 up, 12 in 2024-09-11T03:47:25.232 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:25 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/454550290' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:25.245 INFO:tasks.rados.rados.2.smithi175.stdout:2396: finishing write tid 2 to smithi17557870-29 2024-09-11T03:47:25.245 INFO:tasks.rados.rados.2.smithi175.stdout:2393: done (4 left) 2024-09-11T03:47:25.245 INFO:tasks.rados.rados.2.smithi175.stdout:2395: done (3 left) 2024-09-11T03:47:25.246 INFO:tasks.rados.rados.2.smithi175.stdout:2398: copy_from oid 30 from oid 48 current snap is 263 2024-09-11T03:47:25.246 INFO:tasks.rados.rados.2.smithi175.stdout:2394: finishing copy_from to smithi17557870-32 2024-09-11T03:47:25.246 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 692 (ObjNum 777 snap 248 seq_num 777) dirty exists 2024-09-11T03:47:25.246 INFO:tasks.rados.rados.2.smithi175.stdout:2397: finishing copy_from to smithi17557870-31 2024-09-11T03:47:25.246 INFO:tasks.rados.rados.2.smithi175.stdout:2397: got expected ENOENT (src dne) 2024-09-11T03:47:25.246 INFO:tasks.rados.rados.2.smithi175.stdout:2397: finishing copy_from racing read to smithi17557870-31 2024-09-11T03:47:25.246 INFO:tasks.rados.rados.2.smithi175.stdout:2396: finishing write tid 3 to smithi17557870-29 2024-09-11T03:47:25.246 INFO:tasks.rados.rados.2.smithi175.stdout:2396: finishing write tid 4 to smithi17557870-29 2024-09-11T03:47:25.246 INFO:tasks.rados.rados.2.smithi175.stdout:2396: finishing write tid 5 to smithi17557870-29 2024-09-11T03:47:25.246 INFO:tasks.rados.rados.2.smithi175.stdout:2396: oid 29 updating version 0 to 890 2024-09-11T03:47:25.246 INFO:tasks.rados.rados.2.smithi175.stdout:2396: oid 29 updating version 890 to 891 2024-09-11T03:47:25.246 INFO:tasks.rados.rados.2.smithi175.stdout:2396: oid 29 updating version 891 to 892 2024-09-11T03:47:25.246 INFO:tasks.rados.rados.2.smithi175.stdout:2396: oid 29 version 892 is already newer than 889 2024-09-11T03:47:25.246 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 892 (ObjNum 829 snap 263 seq_num 829) dirty exists 2024-09-11T03:47:25.247 INFO:tasks.rados.rados.2.smithi175.stdout:2396: left oid 29 (ObjNum 829 snap 263 seq_num 829) 2024-09-11T03:47:25.247 INFO:tasks.rados.rados.2.smithi175.stdout:2394: done (3 left) 2024-09-11T03:47:25.247 INFO:tasks.rados.rados.2.smithi175.stdout:2396: done (2 left) 2024-09-11T03:47:25.247 INFO:tasks.rados.rados.2.smithi175.stdout:2397: done (1 left) 2024-09-11T03:47:25.247 INFO:tasks.rados.rados.2.smithi175.stdout:2399: snap_remove snap 261 2024-09-11T03:47:25.250 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:25.272 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:25.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:25.273 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:25.311 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:47:25.311 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:47:25.327 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:47:25.328 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:47:25.345 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:47:25.345 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:47:25.361 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:47:25.361 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:47:25.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:25.399 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:25.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:25.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:25 smithi121 ceph-mon[30729]: 1.12 scrub starts 2024-09-11T03:47:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:25 smithi121 ceph-mon[30729]: 1.12 scrub ok 2024-09-11T03:47:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:25 smithi121 ceph-mon[30729]: osdmap e626: 12 total, 12 up, 12 in 2024-09-11T03:47:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:25 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/454550290' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:25 smithi175 ceph-mon[30894]: 1.12 scrub starts 2024-09-11T03:47:25.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:25 smithi175 ceph-mon[30894]: 1.12 scrub ok 2024-09-11T03:47:25.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:25 smithi175 ceph-mon[30894]: osdmap e626: 12 total, 12 up, 12 in 2024-09-11T03:47:25.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:25 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/454550290' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:25.501 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:25.524 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:25.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:25.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:25.626 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:25.648 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:25.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:25.750 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:25.775 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:25.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:25.782 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:47:25.782 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:47:25.877 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:25.899 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:25.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:25.934 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:26.008 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.6 to 1 (10000) 2024-09-11T03:47:26.013 INFO:tasks.rados.rados.2.smithi175.stdout:2399: done (1 left) 2024-09-11T03:47:26.013 INFO:tasks.rados.rados.2.smithi175.stdout:2400: read oid 43 snap -1 2024-09-11T03:47:26.013 INFO:tasks.rados.rados.2.smithi175.stdout:2400: expect (ObjNum 816 snap 256 seq_num 816) 2024-09-11T03:47:26.013 INFO:tasks.rados.rados.2.smithi175.stdout:2401: read oid 28 snap 262 2024-09-11T03:47:26.013 INFO:tasks.rados.rados.2.smithi175.stdout:2401: expect deleted 2024-09-11T03:47:26.013 INFO:tasks.rados.rados.2.smithi175.stdout:2402: write (excl) oid 36 current snap is 263 2024-09-11T03:47:26.014 INFO:tasks.rados.rados.2.smithi175.stdout:2398: finishing copy_from racing read to smithi17557870-30 2024-09-11T03:47:26.014 INFO:tasks.rados.rados.2.smithi175.stdout:2402: seq_num 830 ranges {749810=739147,2002878=650280,3293084=415653} 2024-09-11T03:47:26.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:26 smithi084 ceph-mon[27254]: pgmap v1095: 67 pgs: 67 active+clean; 115 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 900 KiB/s wr, 6 op/s 2024-09-11T03:47:26.021 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:26 smithi084 ceph-mon[27254]: osdmap e627: 12 total, 12 up, 12 in 2024-09-11T03:47:26.021 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:26 smithi084 ceph-mon[27254]: 1.1b scrub starts 2024-09-11T03:47:26.021 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:26 smithi084 ceph-mon[27254]: 1.1b scrub ok 2024-09-11T03:47:26.021 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/954029936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-11T03:47:26.021 INFO:tasks.rados.rados.2.smithi175.stdout:2402: writing smithi17557870-36 from 749810 to 1488957 tid 1 2024-09-11T03:47:26.027 INFO:tasks.rados.rados.2.smithi175.stdout:2402: writing smithi17557870-36 from 2002878 to 2653158 tid 2 2024-09-11T03:47:26.034 INFO:tasks.rados.rados.2.smithi175.stdout:2402: writing smithi17557870-36 from 3293084 to 3708737 tid 3 2024-09-11T03:47:26.034 INFO:tasks.rados.rados.2.smithi175.stdout:2403: read oid 37 snap -1 2024-09-11T03:47:26.034 INFO:tasks.rados.rados.2.smithi175.stdout:2403: expect (ObjNum 738 snap 243 seq_num 738) 2024-09-11T03:47:26.034 INFO:tasks.rados.rados.2.smithi175.stdout:2404: write (excl) oid 40 current snap is 263 2024-09-11T03:47:26.034 INFO:tasks.rados.rados.2.smithi175.stdout:2398: finishing copy_from to smithi17557870-30 2024-09-11T03:47:26.034 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 837 (ObjNum 793 snap 252 seq_num 793) dirty exists 2024-09-11T03:47:26.034 INFO:tasks.rados.rados.2.smithi175.stdout:2404: seq_num 831 ranges {766617=414396,1945007=510291,3204154=626925} 2024-09-11T03:47:26.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:26.040 INFO:tasks.rados.rados.2.smithi175.stdout:2404: writing smithi17557870-40 from 766617 to 1181013 tid 1 2024-09-11T03:47:26.048 INFO:tasks.rados.rados.2.smithi175.stdout:2404: writing smithi17557870-40 from 1945007 to 2455298 tid 2 2024-09-11T03:47:26.055 INFO:tasks.rados.rados.2.smithi175.stdout:2404: writing smithi17557870-40 from 3204154 to 3831079 tid 3 2024-09-11T03:47:26.055 INFO:tasks.rados.rados.2.smithi175.stdout:2398: done (5 left) 2024-09-11T03:47:26.055 INFO:tasks.rados.rados.2.smithi175.stdout:2405: delete oid 39 current snap is 263 2024-09-11T03:47:26.056 INFO:tasks.rados.rados.2.smithi175.stdout:2400: expect (ObjNum 816 snap 256 seq_num 816) 2024-09-11T03:47:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:26.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:26.163 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:26.184 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:26.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:26.217 INFO:tasks.rados.rados.2.smithi175.stdout:2402: finishing write tid 1 to smithi17557870-36 2024-09-11T03:47:26.217 INFO:tasks.rados.rados.2.smithi175.stdout:2402: finishing write tid 2 to smithi17557870-36 2024-09-11T03:47:26.217 INFO:tasks.rados.rados.2.smithi175.stdout:2400: done (5 left) 2024-09-11T03:47:26.218 INFO:tasks.rados.rados.2.smithi175.stdout:2401: done (4 left) 2024-09-11T03:47:26.218 INFO:tasks.rados.rados.2.smithi175.stdout:2405: done (3 left) 2024-09-11T03:47:26.218 INFO:tasks.rados.rados.2.smithi175.stdout:2406: delete oid 14 current snap is 263 2024-09-11T03:47:26.218 INFO:tasks.rados.rados.2.smithi175.stdout:2402: finishing write tid 3 to smithi17557870-36 2024-09-11T03:47:26.218 INFO:tasks.rados.rados.2.smithi175.stdout:2402: finishing write tid 4 to smithi17557870-36 2024-09-11T03:47:26.218 INFO:tasks.rados.rados.2.smithi175.stdout:2402: finishing write tid 5 to smithi17557870-36 2024-09-11T03:47:26.218 INFO:tasks.rados.rados.2.smithi175.stdout:2402: oid 36 updating version 0 to 852 2024-09-11T03:47:26.218 INFO:tasks.rados.rados.2.smithi175.stdout:2402: oid 36 version 852 is already newer than 850 2024-09-11T03:47:26.218 INFO:tasks.rados.rados.2.smithi175.stdout:2402: oid 36 updating version 852 to 853 2024-09-11T03:47:26.218 INFO:tasks.rados.rados.2.smithi175.stdout:2402: oid 36 version 853 is already newer than 851 2024-09-11T03:47:26.218 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 853 (ObjNum 830 snap 263 seq_num 830) dirty exists 2024-09-11T03:47:26.218 INFO:tasks.rados.rados.2.smithi175.stdout:2402: left oid 36 (ObjNum 830 snap 263 seq_num 830) 2024-09-11T03:47:26.218 INFO:tasks.rados.rados.2.smithi175.stdout:2403: expect (ObjNum 738 snap 243 seq_num 738) 2024-09-11T03:47:26.285 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:26.306 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:26.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:26.307 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:26.340 INFO:tasks.rados.rados.2.smithi175.stdout:2404: finishing write tid 1 to smithi17557870-40 2024-09-11T03:47:26.341 INFO:tasks.rados.rados.2.smithi175.stdout:2404: finishing write tid 2 to smithi17557870-40 2024-09-11T03:47:26.341 INFO:tasks.rados.rados.2.smithi175.stdout:2404: finishing write tid 3 to smithi17557870-40 2024-09-11T03:47:26.341 INFO:tasks.rados.rados.2.smithi175.stdout:2404: finishing write tid 4 to smithi17557870-40 2024-09-11T03:47:26.341 INFO:tasks.rados.rados.2.smithi175.stdout:2404: finishing write tid 5 to smithi17557870-40 2024-09-11T03:47:26.341 INFO:tasks.rados.rados.2.smithi175.stdout:2404: oid 40 updating version 0 to 441 2024-09-11T03:47:26.341 INFO:tasks.rados.rados.2.smithi175.stdout:2404: oid 40 updating version 441 to 444 2024-09-11T03:47:26.341 INFO:tasks.rados.rados.2.smithi175.stdout:2404: oid 40 version 444 is already newer than 442 2024-09-11T03:47:26.342 INFO:tasks.rados.rados.2.smithi175.stdout:2404: oid 40 version 444 is already newer than 443 2024-09-11T03:47:26.342 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 444 (ObjNum 831 snap 263 seq_num 831) dirty exists 2024-09-11T03:47:26.342 INFO:tasks.rados.rados.2.smithi175.stdout:2404: left oid 40 (ObjNum 831 snap 263 seq_num 831) 2024-09-11T03:47:26.342 INFO:tasks.rados.rados.2.smithi175.stdout:2402: done (3 left) 2024-09-11T03:47:26.342 INFO:tasks.rados.rados.2.smithi175.stdout:2403: done (2 left) 2024-09-11T03:47:26.342 INFO:tasks.rados.rados.2.smithi175.stdout:2404: done (1 left) 2024-09-11T03:47:26.342 INFO:tasks.rados.rados.2.smithi175.stdout:2406: done (0 left) 2024-09-11T03:47:26.342 INFO:tasks.rados.rados.2.smithi175.stdout:2407: read oid 47 snap -1 2024-09-11T03:47:26.342 INFO:tasks.rados.rados.2.smithi175.stdout:2407: expect (ObjNum 762 snap 247 seq_num 762) 2024-09-11T03:47:26.342 INFO:tasks.rados.rados.2.smithi175.stdout:2408: write (excl) oid 3 current snap is 263 2024-09-11T03:47:26.342 INFO:tasks.rados.rados.2.smithi175.stdout:2408: seq_num 834 ranges {417038=640143,1720453=296585} 2024-09-11T03:47:26.345 INFO:tasks.rados.rados.2.smithi175.stdout:2408: writing smithi17557870-3 from 417038 to 1057181 tid 1 2024-09-11T03:47:26.351 INFO:tasks.rados.rados.2.smithi175.stdout:2408: writing smithi17557870-3 from 1720453 to 2017038 tid 2 2024-09-11T03:47:26.351 INFO:tasks.rados.rados.2.smithi175.stdout:2409: read oid 30 snap 262 2024-09-11T03:47:26.352 INFO:tasks.rados.rados.2.smithi175.stdout:2409: expect deleted 2024-09-11T03:47:26.352 INFO:tasks.rados.rados.2.smithi175.stdout:2410: read oid 17 snap -1 2024-09-11T03:47:26.352 INFO:tasks.rados.rados.2.smithi175.stdout:2410: expect (ObjNum 799 snap 254 seq_num 799) 2024-09-11T03:47:26.352 INFO:tasks.rados.rados.2.smithi175.stdout:2411: rollback oid 25 current snap is 263 2024-09-11T03:47:26.352 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 25 to 262 2024-09-11T03:47:26.352 INFO:tasks.rados.rados.2.smithi175.stdout:2412: snap_create 2024-09-11T03:47:26.353 INFO:tasks.rados.rados.2.smithi175.stdout:2407: expect (ObjNum 762 snap 247 seq_num 762) 2024-09-11T03:47:26.408 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:26.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:26 smithi121 ceph-mon[30729]: pgmap v1095: 67 pgs: 67 active+clean; 115 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 900 KiB/s wr, 6 op/s 2024-09-11T03:47:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:26 smithi121 ceph-mon[30729]: osdmap e627: 12 total, 12 up, 12 in 2024-09-11T03:47:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:26 smithi121 ceph-mon[30729]: 1.1b scrub starts 2024-09-11T03:47:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:26 smithi121 ceph-mon[30729]: 1.1b scrub ok 2024-09-11T03:47:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/954029936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-11T03:47:26.429 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:26.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:26.430 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:26.447 INFO:tasks.rados.rados.2.smithi175.stdout:2411: finishing rollback tid 0 to smithi17557870-25 2024-09-11T03:47:26.447 INFO:tasks.rados.rados.2.smithi175.stdout:2408: finishing write tid 1 to smithi17557870-3 2024-09-11T03:47:26.447 INFO:tasks.rados.rados.2.smithi175.stdout:2411: finishing rollback tid 1 to smithi17557870-25 2024-09-11T03:47:26.447 INFO:tasks.rados.rados.2.smithi175.stdout:2411: finishing rollback tid 2 to smithi17557870-25 2024-09-11T03:47:26.447 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 511 (ObjNum 803 snap 255 seq_num 803) dirty exists 2024-09-11T03:47:26.448 INFO:tasks.rados.rados.2.smithi175.stdout:2408: finishing write tid 2 to smithi17557870-3 2024-09-11T03:47:26.448 INFO:tasks.rados.rados.2.smithi175.stdout:2408: finishing write tid 3 to smithi17557870-3 2024-09-11T03:47:26.448 INFO:tasks.rados.rados.2.smithi175.stdout:2408: finishing write tid 4 to smithi17557870-3 2024-09-11T03:47:26.448 INFO:tasks.rados.rados.2.smithi175.stdout:2408: oid 3 updating version 0 to 919 2024-09-11T03:47:26.448 INFO:tasks.rados.rados.2.smithi175.stdout:2408: oid 3 updating version 919 to 921 2024-09-11T03:47:26.448 INFO:tasks.rados.rados.2.smithi175.stdout:2408: oid 3 version 921 is already newer than 920 2024-09-11T03:47:26.448 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 921 (ObjNum 834 snap 263 seq_num 834) dirty exists 2024-09-11T03:47:26.448 INFO:tasks.rados.rados.2.smithi175.stdout:2408: left oid 3 (ObjNum 834 snap 263 seq_num 834) 2024-09-11T03:47:26.448 INFO:tasks.rados.rados.2.smithi175.stdout:2410: expect (ObjNum 799 snap 254 seq_num 799) 2024-09-11T03:47:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:26 smithi175 ceph-mon[30894]: pgmap v1095: 67 pgs: 67 active+clean; 115 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 900 KiB/s wr, 6 op/s 2024-09-11T03:47:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:26 smithi175 ceph-mon[30894]: osdmap e627: 12 total, 12 up, 12 in 2024-09-11T03:47:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:26 smithi175 ceph-mon[30894]: 1.1b scrub starts 2024-09-11T03:47:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:26 smithi175 ceph-mon[30894]: 1.1b scrub ok 2024-09-11T03:47:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/954029936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-11T03:47:26.531 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:26.553 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:26.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:26.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:26.655 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:26.681 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:26.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:26.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:26.725 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T03:47:26.726 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:47:26.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:26.808 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:26.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:26.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:26.910 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:26.932 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:26.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:26.933 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:26.982 INFO:tasks.rados.rados.2.smithi175.stdout:2407: done (5 left) 2024-09-11T03:47:26.982 INFO:tasks.rados.rados.2.smithi175.stdout:2408: done (4 left) 2024-09-11T03:47:26.982 INFO:tasks.rados.rados.2.smithi175.stdout:2409: done (3 left) 2024-09-11T03:47:26.982 INFO:tasks.rados.rados.2.smithi175.stdout:2410: done (2 left) 2024-09-11T03:47:26.983 INFO:tasks.rados.rados.2.smithi175.stdout:2411: done (1 left) 2024-09-11T03:47:26.983 INFO:tasks.rados.rados.2.smithi175.stdout:2412: done (0 left) 2024-09-11T03:47:26.983 INFO:tasks.rados.rados.2.smithi175.stdout:2413: write oid 35 current snap is 264 2024-09-11T03:47:26.983 INFO:tasks.rados.rados.2.smithi175.stdout:2413: seq_num 835 ranges {433845=715392,1662582=371263} 2024-09-11T03:47:26.988 INFO:tasks.rados.rados.2.smithi175.stdout:2413: writing smithi17557870-35 from 433845 to 1149237 tid 1 2024-09-11T03:47:26.993 INFO:tasks.rados.rados.2.smithi175.stdout:2413: writing smithi17557870-35 from 1662582 to 2033845 tid 2 2024-09-11T03:47:26.994 INFO:tasks.rados.rados.2.smithi175.stdout:2414: read oid 32 snap -1 2024-09-11T03:47:26.994 INFO:tasks.rados.rados.2.smithi175.stdout:2414: expect (ObjNum 777 snap 248 seq_num 777) 2024-09-11T03:47:26.994 INFO:tasks.rados.rados.2.smithi175.stdout:2415: delete oid 47 current snap is 264 2024-09-11T03:47:26.994 INFO:tasks.rados.rados.2.smithi175.stdout:2413: finishing write tid 1 to smithi17557870-35 2024-09-11T03:47:26.996 INFO:tasks.rados.rados.2.smithi175.stdout:2413: finishing write tid 2 to smithi17557870-35 2024-09-11T03:47:26.998 INFO:tasks.rados.rados.2.smithi175.stdout:2413: finishing write tid 3 to smithi17557870-35 2024-09-11T03:47:26.998 INFO:tasks.rados.rados.2.smithi175.stdout:2415: done (2 left) 2024-09-11T03:47:26.998 INFO:tasks.rados.rados.2.smithi175.stdout:2416: delete oid 33 current snap is 264 2024-09-11T03:47:26.998 INFO:tasks.rados.rados.2.smithi175.stdout:2413: finishing write tid 4 to smithi17557870-35 2024-09-11T03:47:26.998 INFO:tasks.rados.rados.2.smithi175.stdout:2413: oid 35 updating version 0 to 685 2024-09-11T03:47:26.998 INFO:tasks.rados.rados.2.smithi175.stdout:2413: oid 35 version 685 is already newer than 684 2024-09-11T03:47:26.998 INFO:tasks.rados.rados.2.smithi175.stdout:2413: oid 35 version 685 is already newer than 683 2024-09-11T03:47:26.998 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 685 (ObjNum 835 snap 264 seq_num 835) dirty exists 2024-09-11T03:47:26.998 INFO:tasks.rados.rados.2.smithi175.stdout:2413: left oid 35 (ObjNum 835 snap 264 seq_num 835) 2024-09-11T03:47:26.999 INFO:tasks.rados.rados.2.smithi175.stdout:2413: done (2 left) 2024-09-11T03:47:26.999 INFO:tasks.rados.rados.2.smithi175.stdout:2416: done (1 left) 2024-09-11T03:47:26.999 INFO:tasks.rados.rados.2.smithi175.stdout:2417: snap_remove snap 262 2024-09-11T03:47:27.034 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:27.055 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:27.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:27.056 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:27.157 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:27.179 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:27.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:27.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:27.281 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:27.307 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:27.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:27.360 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:27 smithi084 ceph-mon[27254]: 1.3 scrub starts 2024-09-11T03:47:27.361 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:27 smithi084 ceph-mon[27254]: 1.3 scrub ok 2024-09-11T03:47:27.361 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:27 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/954029936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T03:47:27.361 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:27 smithi084 ceph-mon[27254]: osdmap e628: 12 total, 12 up, 12 in 2024-09-11T03:47:27.361 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:27 smithi084 ceph-mon[27254]: 1.c scrub starts 2024-09-11T03:47:27.361 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:27 smithi084 ceph-mon[27254]: 1.c scrub ok 2024-09-11T03:47:27.361 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:27 smithi084 ceph-mon[27254]: osdmap e629: 12 total, 12 up, 12 in 2024-09-11T03:47:27.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:27.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:27 smithi121 ceph-mon[30729]: 1.3 scrub starts 2024-09-11T03:47:27.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:27 smithi121 ceph-mon[30729]: 1.3 scrub ok 2024-09-11T03:47:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:27 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/954029936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T03:47:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:27 smithi121 ceph-mon[30729]: osdmap e628: 12 total, 12 up, 12 in 2024-09-11T03:47:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:27 smithi121 ceph-mon[30729]: 1.c scrub starts 2024-09-11T03:47:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:27 smithi121 ceph-mon[30729]: 1.c scrub ok 2024-09-11T03:47:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:27 smithi121 ceph-mon[30729]: osdmap e629: 12 total, 12 up, 12 in 2024-09-11T03:47:27.434 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:27.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:27.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:27 smithi175 ceph-mon[30894]: 1.3 scrub starts 2024-09-11T03:47:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:27 smithi175 ceph-mon[30894]: 1.3 scrub ok 2024-09-11T03:47:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:27 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/954029936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T03:47:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:27 smithi175 ceph-mon[30894]: osdmap e628: 12 total, 12 up, 12 in 2024-09-11T03:47:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:27 smithi175 ceph-mon[30894]: 1.c scrub starts 2024-09-11T03:47:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:27 smithi175 ceph-mon[30894]: 1.c scrub ok 2024-09-11T03:47:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:27 smithi175 ceph-mon[30894]: osdmap e629: 12 total, 12 up, 12 in 2024-09-11T03:47:27.535 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:27.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:27.557 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:27.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:27.659 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:27.681 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:27.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:27.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:27.783 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:27.806 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:27.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:27.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:27.817 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T03:47:27.860 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:27.860 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:27.908 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:27.931 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:27.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:27.932 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:27.983 INFO:tasks.rados.rados.2.smithi175.stdout:2417: done (1 left) 2024-09-11T03:47:27.983 INFO:tasks.rados.rados.2.smithi175.stdout:2418: write oid 34 current snap is 264 2024-09-11T03:47:27.984 INFO:tasks.rados.rados.2.smithi175.stdout:2418: seq_num 838 ranges {484266=657492,1554381=529885} 2024-09-11T03:47:27.990 INFO:tasks.rados.rados.2.smithi175.stdout:2418: writing smithi17557870-34 from 484266 to 1141758 tid 1 2024-09-11T03:47:27.996 INFO:tasks.rados.rados.2.smithi175.stdout:2418: writing smithi17557870-34 from 1554381 to 2084266 tid 2 2024-09-11T03:47:27.996 INFO:tasks.rados.rados.2.smithi175.stdout:2419: read oid 33 snap -1 2024-09-11T03:47:27.996 INFO:tasks.rados.rados.2.smithi175.stdout:2419: expect deleted 2024-09-11T03:47:27.996 INFO:tasks.rados.rados.2.smithi175.stdout:2420: snap_create 2024-09-11T03:47:27.996 INFO:tasks.rados.rados.2.smithi175.stdout:2414: expect (ObjNum 777 snap 248 seq_num 777) 2024-09-11T03:47:28.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:28.057 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:28.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:28.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:28 smithi084 ceph-mon[27254]: pgmap v1098: 67 pgs: 67 active+clean; 119 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 1.7 MiB/s wr, 18 op/s 2024-09-11T03:47:28.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:28 smithi084 ceph-mon[27254]: osdmap e630: 12 total, 12 up, 12 in 2024-09-11T03:47:28.159 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:28.160 INFO:tasks.rados.rados.2.smithi175.stdout:2418: finishing write tid 1 to smithi17557870-34 2024-09-11T03:47:28.160 INFO:tasks.rados.rados.2.smithi175.stdout:2418: finishing write tid 2 to smithi17557870-34 2024-09-11T03:47:28.160 INFO:tasks.rados.rados.2.smithi175.stdout:2418: finishing write tid 3 to smithi17557870-34 2024-09-11T03:47:28.161 INFO:tasks.rados.rados.2.smithi175.stdout:2418: finishing write tid 4 to smithi17557870-34 2024-09-11T03:47:28.161 INFO:tasks.rados.rados.2.smithi175.stdout:2418: oid 34 updating version 0 to 870 2024-09-11T03:47:28.161 INFO:tasks.rados.rados.2.smithi175.stdout:2418: oid 34 updating version 870 to 872 2024-09-11T03:47:28.161 INFO:tasks.rados.rados.2.smithi175.stdout:2418: oid 34 version 872 is already newer than 871 2024-09-11T03:47:28.161 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 872 (ObjNum 838 snap 264 seq_num 838) dirty exists 2024-09-11T03:47:28.161 INFO:tasks.rados.rados.2.smithi175.stdout:2418: left oid 34 (ObjNum 838 snap 264 seq_num 838) 2024-09-11T03:47:28.182 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:28.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:28.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:28.284 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:28.307 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:28.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:28.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:28.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:28.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:28 smithi121 ceph-mon[30729]: pgmap v1098: 67 pgs: 67 active+clean; 119 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 1.7 MiB/s wr, 18 op/s 2024-09-11T03:47:28.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:28 smithi121 ceph-mon[30729]: osdmap e630: 12 total, 12 up, 12 in 2024-09-11T03:47:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:28.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:28.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:28 smithi175 ceph-mon[30894]: pgmap v1098: 67 pgs: 67 active+clean; 119 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 1.7 MiB/s wr, 18 op/s 2024-09-11T03:47:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:28 smithi175 ceph-mon[30894]: osdmap e630: 12 total, 12 up, 12 in 2024-09-11T03:47:28.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:28.560 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:28.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:28.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:28.662 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:28.668 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:47:28.669 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:47:28.684 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:28.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:28.786 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:28.812 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:28.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:28.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:28.914 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:28.937 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:28.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:28.938 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:28.982 INFO:tasks.rados.rados.2.smithi175.stdout:2414: done (3 left) 2024-09-11T03:47:28.982 INFO:tasks.rados.rados.2.smithi175.stdout:2418: done (2 left) 2024-09-11T03:47:28.982 INFO:tasks.rados.rados.2.smithi175.stdout:2419: done (1 left) 2024-09-11T03:47:28.982 INFO:tasks.rados.rados.2.smithi175.stdout:2420: done (0 left) 2024-09-11T03:47:28.982 INFO:tasks.rados.rados.2.smithi175.stdout:2421: snap_create 2024-09-11T03:47:29.039 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:29.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:29.063 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:29.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:29.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:29.190 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:29.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:29.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:29.292 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:29.318 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:29.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:29.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:29.387 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:47:29.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:29 smithi084 ceph-mon[27254]: 1.a scrub starts 2024-09-11T03:47:29.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:29 smithi084 ceph-mon[27254]: 1.a scrub ok 2024-09-11T03:47:29.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:29 smithi084 ceph-mon[27254]: from='client.16569 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:47:29.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:29 smithi084 ceph-mon[27254]: osdmap e631: 12 total, 12 up, 12 in 2024-09-11T03:47:29.419 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:29.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:29 smithi121 ceph-mon[30729]: 1.a scrub starts 2024-09-11T03:47:29.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:29 smithi121 ceph-mon[30729]: 1.a scrub ok 2024-09-11T03:47:29.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:29 smithi121 ceph-mon[30729]: from='client.16569 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:47:29.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:29 smithi121 ceph-mon[30729]: osdmap e631: 12 total, 12 up, 12 in 2024-09-11T03:47:29.436 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1100,"stamp":"2024-09-11T03:47:27.978854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131392485,"num_objects":46,"num_object_clones":5,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1771,"num_read_kb":1861180,"num_write":1188,"num_write_kb":446059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":244,"num_bytes_recovered":60090984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19214,"ondisk_log_size":19214,"up":201,"acting":201,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":729916,"kb_used_data":315096,"kb_used_omap":18,"kb_used_meta":412845,"kb_avail":1124211908,"statfs":{"total":1151940427776,"available":1151192993792,"internally_reserved":0,"allocated":322658304,"data_stored":290438871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":422753595},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10726705,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":91,"num_read_kb":129133,"num_write":24,"num_write_kb":12717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.986468"},"pg_stats":[{"pgid":"2.23","version":"601'395","reported_seq":3112,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.010784+0000","last_change":"2024-09-11T03:47:26.011406+0000","last_active":"2024-09-11T03:47:26.010784+0000","last_peered":"2024-09-11T03:47:26.010784+0000","last_clean":"2024-09-11T03:47:26.010784+0000","last_became_active":"2024-09-11T03:46:32.970100+0000","last_became_peered":"2024-09-11T03:46:32.970100+0000","last_unstale":"2024-09-11T03:47:26.010784+0000","last_undegraded":"2024-09-11T03:47:26.010784+0000","last_fullsized":"2024-09-11T03:47:26.010784+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"567'384","last_scrub_stamp":"2024-09-11T03:46:33.848050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.848050+0000","objects_scrubbed":0,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:00.904364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083002399999999995,"stat_sum":{"num_bytes":3227109,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":14955,"num_write":10,"num_write_kb":6104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"103","length":"5"}]},{"pgid":"2.22","version":"531'216","reported_seq":2930,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:27.095183+0000","last_change":"2024-09-11T03:47:26.157789+0000","last_active":"2024-09-11T03:47:27.095183+0000","last_peered":"2024-09-11T03:47:27.095183+0000","last_clean":"2024-09-11T03:47:27.095183+0000","last_became_active":"2024-09-11T03:46:32.970633+0000","last_became_peered":"2024-09-11T03:46:32.970633+0000","last_unstale":"2024-09-11T03:47:27.095183+0000","last_undegraded":"2024-09-11T03:47:27.095183+0000","last_fullsized":"2024-09-11T03:47:27.095183+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:46:33.448172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.448172+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:40.589871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00092271900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"105","length":"3"}]},{"pgid":"2.21","version":"625'264","reported_seq":3062,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.996898+0000","last_change":"2024-09-11T03:47:26.060556+0000","last_active":"2024-09-11T03:47:26.996898+0000","last_peered":"2024-09-11T03:47:26.996898+0000","last_clean":"2024-09-11T03:47:26.996898+0000","last_became_active":"2024-09-11T03:46:36.024928+0000","last_became_peered":"2024-09-11T03:46:36.024928+0000","last_unstale":"2024-09-11T03:47:26.996898+0000","last_undegraded":"2024-09-11T03:47:26.996898+0000","last_fullsized":"2024-09-11T03:47:26.996898+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":6,"last_scrub":"578'239","last_scrub_stamp":"2024-09-11T03:46:36.051463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:36.051463+0000","objects_scrubbed":2,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:03.682896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020913529999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":12,"num_write_kb":2819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,8],"acting":[11,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"105","length":"3"}]},{"pgid":"2.20","version":"628'211","reported_seq":3061,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.357254+0000","last_change":"2024-09-11T03:47:26.028802+0000","last_active":"2024-09-11T03:47:26.357254+0000","last_peered":"2024-09-11T03:47:26.357254+0000","last_clean":"2024-09-11T03:47:26.357254+0000","last_became_active":"2024-09-11T03:46:32.974375+0000","last_became_peered":"2024-09-11T03:46:32.974375+0000","last_unstale":"2024-09-11T03:47:26.357254+0000","last_undegraded":"2024-09-11T03:47:26.357254+0000","last_fullsized":"2024-09-11T03:47:26.357254+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:46:33.551541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.551541+0000","objects_scrubbed":2,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:36.382582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025659680000000001,"stat_sum":{"num_bytes":3496021,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":15112,"num_write":7,"num_write_kb":1493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"103","length":"5"}]},{"pgid":"2.1c","version":"629'685","reported_seq":3391,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.999168+0000","last_change":"2024-09-11T03:47:26.013448+0000","last_active":"2024-09-11T03:47:26.999168+0000","last_peered":"2024-09-11T03:47:26.999168+0000","last_clean":"2024-09-11T03:47:26.999168+0000","last_became_active":"2024-09-11T03:46:32.971036+0000","last_became_peered":"2024-09-11T03:46:32.971036+0000","last_unstale":"2024-09-11T03:47:26.999168+0000","last_undegraded":"2024-09-11T03:47:26.999168+0000","last_fullsized":"2024-09-11T03:47:26.999168+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"568'667","last_scrub_stamp":"2024-09-11T03:46:33.141943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.141943+0000","objects_scrubbed":1,"log_size":685,"log_dups_size":0,"ondisk_log_size":685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:38.878135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065484699999999996,"stat_sum":{"num_bytes":4700325,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10464,"num_write":10,"num_write_kb":2762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"105","length":"3"}]},{"pgid":"1.c","version":"20'92","reported_seq":301,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-11T03:47:25.021067+0000","last_change":"2024-09-11T03:46:15.978402+0000","last_active":"2024-09-11T03:47:25.021067+0000","last_peered":"2024-09-11T03:47:25.021067+0000","last_clean":"2024-09-11T03:47:25.021067+0000","last_became_active":"2024-09-11T03:46:15.974786+0000","last_became_peered":"2024-09-11T03:46:15.974786+0000","last_unstale":"2024-09-11T03:47:25.021067+0000","last_undegraded":"2024-09-11T03:47:25.021067+0000","last_fullsized":"2024-09-11T03:47:25.021067+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:10.103870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:10.103870+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.f","version":"557'465","reported_seq":3198,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.047962+0000","last_change":"2024-09-11T03:47:26.048350+0000","last_active":"2024-09-11T03:47:26.047962+0000","last_peered":"2024-09-11T03:47:26.047962+0000","last_clean":"2024-09-11T03:47:26.047962+0000","last_became_active":"2024-09-11T03:46:27.971779+0000","last_became_peered":"2024-09-11T03:46:27.971779+0000","last_unstale":"2024-09-11T03:47:26.047962+0000","last_undegraded":"2024-09-11T03:47:26.047962+0000","last_fullsized":"2024-09-11T03:47:26.047962+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:46:32.852908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:32.852908+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:00.596503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063914099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"103","length":"5"}]},{"pgid":"2.e","version":"623'885","reported_seq":3554,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-11T03:47:22.995653+0000","last_change":"2024-09-11T03:47:22.998043+0000","last_active":"2024-09-11T03:47:22.995653+0000","last_peered":"2024-09-11T03:47:22.995653+0000","last_clean":"2024-09-11T03:47:22.995653+0000","last_became_active":"2024-09-11T03:46:36.024843+0000","last_became_peered":"2024-09-11T03:46:36.024843+0000","last_unstale":"2024-09-11T03:47:22.995653+0000","last_undegraded":"2024-09-11T03:47:22.995653+0000","last_fullsized":"2024-09-11T03:47:22.995653+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'850","last_scrub_stamp":"2024-09-11T03:46:36.567100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:36.567100+0000","objects_scrubbed":3,"log_size":885,"log_dups_size":0,"ondisk_log_size":885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:48.243327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0038993890000000001,"stat_sum":{"num_bytes":2790127,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":26,"num_write_kb":9576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"103","length":"4"}]},{"pgid":"1.d","version":"20'92","reported_seq":879,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.027041+0000","last_change":"2024-09-11T03:42:51.733829+0000","last_active":"2024-09-11T03:47:26.027041+0000","last_peered":"2024-09-11T03:47:26.027041+0000","last_clean":"2024-09-11T03:47:26.027041+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:47:26.027041+0000","last_undegraded":"2024-09-11T03:47:26.027041+0000","last_fullsized":"2024-09-11T03:47:26.027041+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:51.733762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:51.733762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:11.597940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"625'787","reported_seq":3524,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.355976+0000","last_change":"2024-09-11T03:47:26.039870+0000","last_active":"2024-09-11T03:47:26.355976+0000","last_peered":"2024-09-11T03:47:26.355976+0000","last_clean":"2024-09-11T03:47:26.355976+0000","last_became_active":"2024-09-11T03:46:27.969385+0000","last_became_peered":"2024-09-11T03:46:27.969385+0000","last_unstale":"2024-09-11T03:47:26.355976+0000","last_undegraded":"2024-09-11T03:47:26.355976+0000","last_fullsized":"2024-09-11T03:47:26.355976+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'757","last_scrub_stamp":"2024-09-11T03:46:31.645504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:31.645504+0000","objects_scrubbed":5,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:49.956740+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002448689,"stat_sum":{"num_bytes":11631104,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":98,"num_read_kb":116232,"num_write":19,"num_write_kb":7250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"103","length":"5"}]},{"pgid":"1.e","version":"20'92","reported_seq":594,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.026963+0000","last_change":"2024-09-11T03:45:53.637950+0000","last_active":"2024-09-11T03:47:26.026963+0000","last_peered":"2024-09-11T03:47:26.026963+0000","last_clean":"2024-09-11T03:47:26.026963+0000","last_became_active":"2024-09-11T03:44:43.940022+0000","last_became_peered":"2024-09-11T03:44:43.940022+0000","last_unstale":"2024-09-11T03:47:26.026963+0000","last_undegraded":"2024-09-11T03:47:26.026963+0000","last_fullsized":"2024-09-11T03:47:26.026963+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:53.637891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:53.637891+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:14.519768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"611'706","reported_seq":3437,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.996720+0000","last_change":"2024-09-11T03:47:26.061357+0000","last_active":"2024-09-11T03:47:26.996720+0000","last_peered":"2024-09-11T03:47:26.996720+0000","last_clean":"2024-09-11T03:47:26.996720+0000","last_became_active":"2024-09-11T03:46:27.975493+0000","last_became_peered":"2024-09-11T03:46:27.975493+0000","last_unstale":"2024-09-11T03:47:26.996720+0000","last_undegraded":"2024-09-11T03:47:26.996720+0000","last_fullsized":"2024-09-11T03:47:26.996720+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"571'670","last_scrub_stamp":"2024-09-11T03:46:32.062033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:32.062033+0000","objects_scrubbed":4,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:44.451627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002730194,"stat_sum":{"num_bytes":9513257,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":8821,"num_write":26,"num_write_kb":7012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"105","length":"3"}]},{"pgid":"1.f","version":"20'92","reported_seq":162,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:27.095107+0000","last_change":"2024-09-11T03:46:16.476244+0000","last_active":"2024-09-11T03:47:27.095107+0000","last_peered":"2024-09-11T03:47:27.095107+0000","last_clean":"2024-09-11T03:47:27.095107+0000","last_became_active":"2024-09-11T03:46:15.973796+0000","last_became_peered":"2024-09-11T03:46:15.973796+0000","last_unstale":"2024-09-11T03:47:27.095107+0000","last_undegraded":"2024-09-11T03:47:27.095107+0000","last_fullsized":"2024-09-11T03:47:27.095107+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:16.476181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:16.476181+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:43.297927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":1284,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.026955+0000","last_change":"2024-09-11T03:46:15.975561+0000","last_active":"2024-09-11T03:47:26.026955+0000","last_peered":"2024-09-11T03:47:26.026955+0000","last_clean":"2024-09-11T03:47:26.026955+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:47:26.026955+0000","last_undegraded":"2024-09-11T03:47:26.026955+0000","last_fullsized":"2024-09-11T03:47:26.026955+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:23.491587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:23.491587+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:27.520369+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"612'414","reported_seq":3146,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.038866+0000","last_change":"2024-09-11T03:47:26.039435+0000","last_active":"2024-09-11T03:47:26.038866+0000","last_peered":"2024-09-11T03:47:26.038866+0000","last_clean":"2024-09-11T03:47:26.038866+0000","last_became_active":"2024-09-11T03:46:27.972329+0000","last_became_peered":"2024-09-11T03:46:27.972329+0000","last_unstale":"2024-09-11T03:47:26.038866+0000","last_undegraded":"2024-09-11T03:47:26.038866+0000","last_fullsized":"2024-09-11T03:47:26.038866+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'386","last_scrub_stamp":"2024-09-11T03:46:33.722283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.722283+0000","objects_scrubbed":3,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:53.755661+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0021067210000000002,"stat_sum":{"num_bytes":2235463,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":3868,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"103","length":"5"}]},{"pgid":"1.3","version":"20'92","reported_seq":896,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.010644+0000","last_change":"2024-09-11T03:47:25.344219+0000","last_active":"2024-09-11T03:47:26.010644+0000","last_peered":"2024-09-11T03:47:26.010644+0000","last_clean":"2024-09-11T03:47:26.010644+0000","last_became_active":"2024-09-11T03:46:15.968960+0000","last_became_peered":"2024-09-11T03:46:15.968960+0000","last_unstale":"2024-09-11T03:47:26.010644+0000","last_undegraded":"2024-09-11T03:47:26.010644+0000","last_fullsized":"2024-09-11T03:47:26.010644+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:25.344170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:25.344170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:39.221544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"563'192","reported_seq":2990,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.028880+0000","last_change":"2024-09-11T03:47:26.029416+0000","last_active":"2024-09-11T03:47:26.028880+0000","last_peered":"2024-09-11T03:47:26.028880+0000","last_clean":"2024-09-11T03:47:26.028880+0000","last_became_active":"2024-09-11T03:46:27.974405+0000","last_became_peered":"2024-09-11T03:46:27.974405+0000","last_unstale":"2024-09-11T03:47:26.028880+0000","last_undegraded":"2024-09-11T03:47:26.028880+0000","last_fullsized":"2024-09-11T03:47:26.028880+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:46:31.577153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:31.577153+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:35.636509+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024270630000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"103","length":"5"}]},{"pgid":"1.2","version":"20'92","reported_seq":847,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-11T03:47:25.020992+0000","last_change":"2024-09-11T03:46:41.056685+0000","last_active":"2024-09-11T03:47:25.020992+0000","last_peered":"2024-09-11T03:47:25.020992+0000","last_clean":"2024-09-11T03:47:25.020992+0000","last_became_active":"2024-09-11T03:44:25.934580+0000","last_became_peered":"2024-09-11T03:44:25.934580+0000","last_unstale":"2024-09-11T03:47:25.020992+0000","last_undegraded":"2024-09-11T03:47:25.020992+0000","last_fullsized":"2024-09-11T03:47:25.020992+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:23.494418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:23.494418+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:10.279429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"567'235","reported_seq":2917,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-11T03:47:25.021358+0000","last_change":"2024-09-11T03:47:23.162648+0000","last_active":"2024-09-11T03:47:25.021358+0000","last_peered":"2024-09-11T03:47:25.021358+0000","last_clean":"2024-09-11T03:47:25.021358+0000","last_became_active":"2024-09-11T03:46:36.024925+0000","last_became_peered":"2024-09-11T03:46:36.024925+0000","last_unstale":"2024-09-11T03:47:25.021358+0000","last_undegraded":"2024-09-11T03:47:25.021358+0000","last_fullsized":"2024-09-11T03:47:25.021358+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:46:36.189297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:36.189297+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:52.655690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013921770000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"103","length":"4"}]},{"pgid":"1.1","version":"20'92","reported_seq":456,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.991419+0000","last_change":"2024-09-11T03:45:26.373529+0000","last_active":"2024-09-11T03:47:26.991419+0000","last_peered":"2024-09-11T03:47:26.991419+0000","last_clean":"2024-09-11T03:47:26.991419+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:47:26.991419+0000","last_undegraded":"2024-09-11T03:47:26.991419+0000","last_fullsized":"2024-09-11T03:47:26.991419+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:26.373478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:26.373478+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:30.120533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"585'218","reported_seq":3025,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.038125+0000","last_change":"2024-09-11T03:47:26.038796+0000","last_active":"2024-09-11T03:47:26.038125+0000","last_peered":"2024-09-11T03:47:26.038125+0000","last_clean":"2024-09-11T03:47:26.038125+0000","last_became_active":"2024-09-11T03:46:27.971649+0000","last_became_peered":"2024-09-11T03:46:27.971649+0000","last_unstale":"2024-09-11T03:47:26.038125+0000","last_undegraded":"2024-09-11T03:47:26.038125+0000","last_fullsized":"2024-09-11T03:47:26.038125+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:46:32.681048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:32.681048+0000","objects_scrubbed":0,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:01.330583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017866239999999999,"stat_sum":{"num_bytes":2403566,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":5188,"num_write":2,"num_write_kb":6239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"103","length":"5"}]},{"pgid":"1.7","version":"20'92","reported_seq":154,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:27.094547+0000","last_change":"2024-09-11T03:46:15.972170+0000","last_active":"2024-09-11T03:47:27.094547+0000","last_peered":"2024-09-11T03:47:27.094547+0000","last_clean":"2024-09-11T03:47:27.094547+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:47:27.094547+0000","last_undegraded":"2024-09-11T03:47:27.094547+0000","last_fullsized":"2024-09-11T03:47:27.094547+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:12.360829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:12.360829+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:28.490489+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"623'478","reported_seq":3242,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.011553+0000","last_change":"2024-09-11T03:47:26.012358+0000","last_active":"2024-09-11T03:47:26.011553+0000","last_peered":"2024-09-11T03:47:26.011553+0000","last_clean":"2024-09-11T03:47:26.011553+0000","last_became_active":"2024-09-11T03:44:10.921135+0000","last_became_peered":"2024-09-11T03:44:10.921135+0000","last_unstale":"2024-09-11T03:47:26.011553+0000","last_undegraded":"2024-09-11T03:47:26.011553+0000","last_fullsized":"2024-09-11T03:47:26.011553+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"590'463","last_scrub_stamp":"2024-09-11T03:46:49.936885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:49.936885+0000","objects_scrubbed":3,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:04.884385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0014335159999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":141,"num_read_kb":139436,"num_write":92,"num_write_kb":30847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":4118225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"103","length":"5"}]},{"pgid":"1.6","version":"20'92","reported_seq":459,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.991795+0000","last_change":"2024-09-11T03:46:59.252925+0000","last_active":"2024-09-11T03:47:26.991795+0000","last_peered":"2024-09-11T03:47:26.991795+0000","last_clean":"2024-09-11T03:47:26.991795+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:47:26.991795+0000","last_undegraded":"2024-09-11T03:47:26.991795+0000","last_fullsized":"2024-09-11T03:47:26.991795+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:59.252877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:59.252877+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:07.389660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"628'880","reported_seq":3653,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.049636+0000","last_change":"2024-09-11T03:47:26.049897+0000","last_active":"2024-09-11T03:47:26.049636+0000","last_peered":"2024-09-11T03:47:26.049636+0000","last_clean":"2024-09-11T03:47:26.049636+0000","last_became_active":"2024-09-11T03:41:33.866316+0000","last_became_peered":"2024-09-11T03:41:33.866316+0000","last_unstale":"2024-09-11T03:47:26.049636+0000","last_undegraded":"2024-09-11T03:47:26.049636+0000","last_fullsized":"2024-09-11T03:47:26.049636+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'849","last_scrub_stamp":"2024-09-11T03:46:59.044884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:59.044884+0000","objects_scrubbed":3,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:16.303079+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0024115299999999998,"stat_sum":{"num_bytes":18132577,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":219,"num_read_kb":231414,"num_write":162,"num_write_kb":62297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"103","length":"5"}]},{"pgid":"1.5","version":"20'92","reported_seq":466,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.991511+0000","last_change":"2024-09-11T03:46:17.255120+0000","last_active":"2024-09-11T03:47:26.991511+0000","last_peered":"2024-09-11T03:47:26.991511+0000","last_clean":"2024-09-11T03:47:26.991511+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:47:26.991511+0000","last_undegraded":"2024-09-11T03:47:26.991511+0000","last_fullsized":"2024-09-11T03:47:26.991511+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:17.255073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:17.255073+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:31.448759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":2871,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.011497+0000","last_change":"2024-09-11T03:47:26.012317+0000","last_active":"2024-09-11T03:47:26.011497+0000","last_peered":"2024-09-11T03:47:26.011497+0000","last_clean":"2024-09-11T03:47:26.011497+0000","last_became_active":"2024-09-11T03:41:33.865499+0000","last_became_peered":"2024-09-11T03:41:33.865499+0000","last_unstale":"2024-09-11T03:47:26.011497+0000","last_undegraded":"2024-09-11T03:47:26.011497+0000","last_fullsized":"2024-09-11T03:47:26.011497+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:47:15.201479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:15.201479+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:22.676959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001604556,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"103","length":"5"}]},{"pgid":"1.4","version":"20'92","reported_seq":1059,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.996462+0000","last_change":"2024-09-11T03:46:59.164595+0000","last_active":"2024-09-11T03:47:26.996462+0000","last_peered":"2024-09-11T03:47:26.996462+0000","last_clean":"2024-09-11T03:47:26.996462+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:47:26.996462+0000","last_undegraded":"2024-09-11T03:47:26.996462+0000","last_fullsized":"2024-09-11T03:47:26.996462+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:59.164529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:59.164529+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:01.974929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"623'939","reported_seq":3771,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:27.094694+0000","last_change":"2024-09-11T03:47:26.157472+0000","last_active":"2024-09-11T03:47:27.094694+0000","last_peered":"2024-09-11T03:47:27.094694+0000","last_clean":"2024-09-11T03:47:27.094694+0000","last_became_active":"2024-09-11T03:46:19.004997+0000","last_became_peered":"2024-09-11T03:46:19.004997+0000","last_unstale":"2024-09-11T03:47:27.094694+0000","last_undegraded":"2024-09-11T03:47:27.094694+0000","last_fullsized":"2024-09-11T03:47:27.094694+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'924","last_scrub_stamp":"2024-09-11T03:46:53.434384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:53.434384+0000","objects_scrubbed":5,"log_size":939,"log_dups_size":0,"ondisk_log_size":939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:00.632515+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00084767200000000005,"stat_sum":{"num_bytes":11731946,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":344,"num_read_kb":409383,"num_write":189,"num_write_kb":70180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":25429065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"105","length":"3"}]},{"pgid":"1.b","version":"20'92","reported_seq":870,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.996815+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:47:26.996815+0000","last_peered":"2024-09-11T03:47:26.996815+0000","last_clean":"2024-09-11T03:47:26.996815+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:47:26.996815+0000","last_undegraded":"2024-09-11T03:47:26.996815+0000","last_fullsized":"2024-09-11T03:47:26.996815+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:06.223147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"610'328","reported_seq":3055,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.026886+0000","last_change":"2024-09-11T03:47:26.029262+0000","last_active":"2024-09-11T03:47:26.026886+0000","last_peered":"2024-09-11T03:47:26.026886+0000","last_clean":"2024-09-11T03:47:26.026886+0000","last_became_active":"2024-09-11T03:41:33.863875+0000","last_became_peered":"2024-09-11T03:41:33.863875+0000","last_unstale":"2024-09-11T03:47:26.026886+0000","last_undegraded":"2024-09-11T03:47:26.026886+0000","last_fullsized":"2024-09-11T03:47:26.026886+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'324","last_scrub_stamp":"2024-09-11T03:46:16.518345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:16.518345+0000","objects_scrubbed":0,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:45.906843+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0026140270000000001,"stat_sum":{"num_bytes":3546442,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":45190,"num_write":39,"num_write_kb":11168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"103","length":"5"}]},{"pgid":"1.a","version":"20'92","reported_seq":1195,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-11T03:47:22.993685+0000","last_change":"2024-09-11T03:46:15.973700+0000","last_active":"2024-09-11T03:47:22.993685+0000","last_peered":"2024-09-11T03:47:22.993685+0000","last_clean":"2024-09-11T03:47:22.993685+0000","last_became_active":"2024-09-11T03:46:15.973271+0000","last_became_peered":"2024-09-11T03:46:15.973271+0000","last_unstale":"2024-09-11T03:47:22.993685+0000","last_undegraded":"2024-09-11T03:47:22.993685+0000","last_fullsized":"2024-09-11T03:47:22.993685+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:24.066237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:24.066237+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:25.715302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"628'920","reported_seq":3746,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.357343+0000","last_change":"2024-09-11T03:47:26.011413+0000","last_active":"2024-09-11T03:47:26.357343+0000","last_peered":"2024-09-11T03:47:26.357343+0000","last_clean":"2024-09-11T03:47:26.357343+0000","last_became_active":"2024-09-11T03:41:33.865638+0000","last_became_peered":"2024-09-11T03:41:33.865638+0000","last_unstale":"2024-09-11T03:47:26.357343+0000","last_undegraded":"2024-09-11T03:47:26.357343+0000","last_fullsized":"2024-09-11T03:47:26.357343+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'865","last_scrub_stamp":"2024-09-11T03:46:34.843368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:34.843368+0000","objects_scrubbed":7,"log_size":920,"log_dups_size":0,"ondisk_log_size":920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:53.409033+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00096968499999999997,"stat_sum":{"num_bytes":6970771,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":346,"num_read_kb":371753,"num_write":190,"num_write_kb":66461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"103","length":"5"}]},{"pgid":"1.9","version":"20'92","reported_seq":1280,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.026253+0000","last_change":"2024-09-11T03:47:22.529885+0000","last_active":"2024-09-11T03:47:26.026253+0000","last_peered":"2024-09-11T03:47:26.026253+0000","last_clean":"2024-09-11T03:47:26.026253+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:47:26.026253+0000","last_undegraded":"2024-09-11T03:47:26.026253+0000","last_fullsized":"2024-09-11T03:47:26.026253+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:22.529805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:22.529805+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:34.199617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"614'308","reported_seq":3047,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.996625+0000","last_change":"2024-09-11T03:47:26.059717+0000","last_active":"2024-09-11T03:47:26.996625+0000","last_peered":"2024-09-11T03:47:26.996625+0000","last_clean":"2024-09-11T03:47:26.996625+0000","last_became_active":"2024-09-11T03:46:27.968719+0000","last_became_peered":"2024-09-11T03:46:27.968719+0000","last_unstale":"2024-09-11T03:47:26.996625+0000","last_undegraded":"2024-09-11T03:47:26.996625+0000","last_fullsized":"2024-09-11T03:47:26.996625+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'279","last_scrub_stamp":"2024-09-11T03:46:31.085715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:31.085715+0000","objects_scrubbed":2,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:42.744736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001553247,"stat_sum":{"num_bytes":3529635,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":10,"num_write_kb":6055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"105","length":"3"}]},{"pgid":"1.8","version":"20'92","reported_seq":942,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.037025+0000","last_change":"2024-09-11T03:46:20.861683+0000","last_active":"2024-09-11T03:47:26.037025+0000","last_peered":"2024-09-11T03:47:26.037025+0000","last_clean":"2024-09-11T03:47:26.037025+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:47:26.037025+0000","last_undegraded":"2024-09-11T03:47:26.037025+0000","last_fullsized":"2024-09-11T03:47:26.037025+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:20.861635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:20.861635+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:27.499407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"568'646","reported_seq":3371,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.047648+0000","last_change":"2024-09-11T03:47:26.047881+0000","last_active":"2024-09-11T03:47:26.047648+0000","last_peered":"2024-09-11T03:47:26.047648+0000","last_clean":"2024-09-11T03:47:26.047648+0000","last_became_active":"2024-09-11T03:46:27.974031+0000","last_became_peered":"2024-09-11T03:46:27.974031+0000","last_unstale":"2024-09-11T03:47:26.047648+0000","last_undegraded":"2024-09-11T03:47:26.047648+0000","last_fullsized":"2024-09-11T03:47:26.047648+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:46:31.818739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:31.818739+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:32.595903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046285299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"103","length":"5"}]},{"pgid":"1.10","version":"20'92","reported_seq":883,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.036985+0000","last_change":"2024-09-11T03:46:37.719201+0000","last_active":"2024-09-11T03:47:26.036985+0000","last_peered":"2024-09-11T03:47:26.036985+0000","last_clean":"2024-09-11T03:47:26.036985+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:47:26.036985+0000","last_undegraded":"2024-09-11T03:47:26.036985+0000","last_fullsized":"2024-09-11T03:47:26.036985+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:37.719151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:37.719151+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:41.060227+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":2782,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-11T03:47:22.995017+0000","last_change":"2024-09-11T03:47:22.996975+0000","last_active":"2024-09-11T03:47:22.995017+0000","last_peered":"2024-09-11T03:47:22.995017+0000","last_clean":"2024-09-11T03:47:22.995017+0000","last_became_active":"2024-09-11T03:41:38.886834+0000","last_became_peered":"2024-09-11T03:41:38.886834+0000","last_unstale":"2024-09-11T03:47:22.995017+0000","last_undegraded":"2024-09-11T03:47:22.995017+0000","last_fullsized":"2024-09-11T03:47:22.995017+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:46:14.674306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:14.674306+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:37.432719+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0028763590000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"103","length":"4"}]},{"pgid":"1.11","version":"20'92","reported_seq":864,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-11T03:47:25.021242+0000","last_change":"2024-09-11T03:47:00.162760+0000","last_active":"2024-09-11T03:47:25.021242+0000","last_peered":"2024-09-11T03:47:25.021242+0000","last_clean":"2024-09-11T03:47:25.021242+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:47:25.021242+0000","last_undegraded":"2024-09-11T03:47:25.021242+0000","last_fullsized":"2024-09-11T03:47:25.021242+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:00.162709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:00.162709+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:20.560991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"628'444","reported_seq":3175,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.991737+0000","last_change":"2024-09-11T03:47:26.013916+0000","last_active":"2024-09-11T03:47:26.991737+0000","last_peered":"2024-09-11T03:47:26.991737+0000","last_clean":"2024-09-11T03:47:26.991737+0000","last_became_active":"2024-09-11T03:43:55.918817+0000","last_became_peered":"2024-09-11T03:43:55.918817+0000","last_unstale":"2024-09-11T03:47:26.991737+0000","last_undegraded":"2024-09-11T03:47:26.991737+0000","last_fullsized":"2024-09-11T03:47:26.991737+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"590'420","last_scrub_stamp":"2024-09-11T03:46:50.265647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:50.265647+0000","objects_scrubbed":2,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:19.726336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011092389999999999,"stat_sum":{"num_bytes":7681129,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":166394,"num_write":109,"num_write_kb":39112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":15216236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"105","length":"3"}]},{"pgid":"1.12","version":"20'92","reported_seq":880,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.037070+0000","last_change":"2024-09-11T03:47:23.339646+0000","last_active":"2024-09-11T03:47:26.037070+0000","last_peered":"2024-09-11T03:47:26.037070+0000","last_clean":"2024-09-11T03:47:26.037070+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:47:26.037070+0000","last_undegraded":"2024-09-11T03:47:26.037070+0000","last_fullsized":"2024-09-11T03:47:26.037070+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:23.339595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:23.339595+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:46.448313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":1872,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:27.094483+0000","last_change":"2024-09-11T03:47:26.157075+0000","last_active":"2024-09-11T03:47:27.094483+0000","last_peered":"2024-09-11T03:47:27.094483+0000","last_clean":"2024-09-11T03:47:27.094483+0000","last_became_active":"2024-09-11T03:46:19.005117+0000","last_became_peered":"2024-09-11T03:46:19.005117+0000","last_unstale":"2024-09-11T03:47:27.094483+0000","last_undegraded":"2024-09-11T03:47:27.094483+0000","last_fullsized":"2024-09-11T03:47:27.094483+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:47:20.437167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:20.437167+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:27.629634+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00068093500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"105","length":"3"}]},{"pgid":"1.13","version":"20'92","reported_seq":983,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.010597+0000","last_change":"2024-09-11T03:47:18.217600+0000","last_active":"2024-09-11T03:47:26.010597+0000","last_peered":"2024-09-11T03:47:26.010597+0000","last_clean":"2024-09-11T03:47:26.010597+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:47:26.010597+0000","last_undegraded":"2024-09-11T03:47:26.010597+0000","last_fullsized":"2024-09-11T03:47:26.010597+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:18.217555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:18.217555+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:40.131743+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":1809,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.047503+0000","last_change":"2024-09-11T03:47:26.047879+0000","last_active":"2024-09-11T03:47:26.047503+0000","last_peered":"2024-09-11T03:47:26.047503+0000","last_clean":"2024-09-11T03:47:26.047503+0000","last_became_active":"2024-09-11T03:41:35.874295+0000","last_became_peered":"2024-09-11T03:41:35.874295+0000","last_unstale":"2024-09-11T03:47:26.047503+0000","last_undegraded":"2024-09-11T03:47:26.047503+0000","last_fullsized":"2024-09-11T03:47:26.047503+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:47:18.969898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:18.969898+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:30.487010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00062001899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"103","length":"5"}]},{"pgid":"1.14","version":"20'92","reported_seq":448,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.026483+0000","last_change":"2024-09-11T03:46:36.567028+0000","last_active":"2024-09-11T03:47:26.026483+0000","last_peered":"2024-09-11T03:47:26.026483+0000","last_clean":"2024-09-11T03:47:26.026483+0000","last_became_active":"2024-09-11T03:46:36.026459+0000","last_became_peered":"2024-09-11T03:46:36.026459+0000","last_unstale":"2024-09-11T03:47:26.026483+0000","last_undegraded":"2024-09-11T03:47:26.026483+0000","last_fullsized":"2024-09-11T03:47:26.026483+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:36.566974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:36.566974+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:51.906555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":2368,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:27.095146+0000","last_change":"2024-09-11T03:47:26.157689+0000","last_active":"2024-09-11T03:47:27.095146+0000","last_peered":"2024-09-11T03:47:27.095146+0000","last_clean":"2024-09-11T03:47:27.095146+0000","last_became_active":"2024-09-11T03:46:19.003959+0000","last_became_peered":"2024-09-11T03:46:19.003959+0000","last_unstale":"2024-09-11T03:47:27.095146+0000","last_undegraded":"2024-09-11T03:47:27.095146+0000","last_fullsized":"2024-09-11T03:47:27.095146+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:46:35.381114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:35.381114+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:57.567590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00089939800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"105","length":"3"}]},{"pgid":"1.15","version":"20'92","reported_seq":842,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.026643+0000","last_change":"2024-09-11T03:46:37.557063+0000","last_active":"2024-09-11T03:47:26.026643+0000","last_peered":"2024-09-11T03:47:26.026643+0000","last_clean":"2024-09-11T03:47:26.026643+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:47:26.026643+0000","last_undegraded":"2024-09-11T03:47:26.026643+0000","last_fullsized":"2024-09-11T03:47:26.026643+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:37.557020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:37.557020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:44.193179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"609'420","reported_seq":3188,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.037988+0000","last_change":"2024-09-11T03:47:26.038859+0000","last_active":"2024-09-11T03:47:26.037988+0000","last_peered":"2024-09-11T03:47:26.037988+0000","last_clean":"2024-09-11T03:47:26.037988+0000","last_became_active":"2024-09-11T03:46:19.007019+0000","last_became_peered":"2024-09-11T03:46:19.007019+0000","last_unstale":"2024-09-11T03:47:26.037988+0000","last_undegraded":"2024-09-11T03:47:26.037988+0000","last_fullsized":"2024-09-11T03:47:26.037988+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"597'416","last_scrub_stamp":"2024-09-11T03:47:00.394934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:00.394934+0000","objects_scrubbed":0,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:13.413942+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0015017310000000001,"stat_sum":{"num_bytes":3512828,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":65,"num_read_kb":54715,"num_write":73,"num_write_kb":27957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":5263193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"103","length":"5"}]},{"pgid":"1.16","version":"20'92","reported_seq":318,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.996907+0000","last_change":"2024-09-11T03:46:35.041806+0000","last_active":"2024-09-11T03:47:26.996907+0000","last_peered":"2024-09-11T03:47:26.996907+0000","last_clean":"2024-09-11T03:47:26.996907+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:47:26.996907+0000","last_undegraded":"2024-09-11T03:47:26.996907+0000","last_fullsized":"2024-09-11T03:47:26.996907+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:35.041709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:35.041709+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:45.999431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":3003,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-11T03:47:22.997079+0000","last_change":"2024-09-11T03:47:22.998318+0000","last_active":"2024-09-11T03:47:22.997079+0000","last_peered":"2024-09-11T03:47:22.997079+0000","last_clean":"2024-09-11T03:47:22.997079+0000","last_became_active":"2024-09-11T03:41:38.888228+0000","last_became_peered":"2024-09-11T03:41:38.888228+0000","last_unstale":"2024-09-11T03:47:22.997079+0000","last_undegraded":"2024-09-11T03:47:22.997079+0000","last_fullsized":"2024-09-11T03:47:22.997079+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:46:37.552437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:37.552437+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:38.773031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0039417009999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"103","length":"4"}]},{"pgid":"1.17","version":"20'92","reported_seq":860,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.047648+0000","last_change":"2024-09-11T03:46:12.911087+0000","last_active":"2024-09-11T03:47:26.047648+0000","last_peered":"2024-09-11T03:47:26.047648+0000","last_clean":"2024-09-11T03:47:26.047648+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:47:26.047648+0000","last_undegraded":"2024-09-11T03:47:26.047648+0000","last_fullsized":"2024-09-11T03:47:26.047648+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:12.911035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:12.911035+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:27.756515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"614'276","reported_seq":2907,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:27.094698+0000","last_change":"2024-09-11T03:47:26.157367+0000","last_active":"2024-09-11T03:47:27.094698+0000","last_peered":"2024-09-11T03:47:27.094698+0000","last_clean":"2024-09-11T03:47:27.094698+0000","last_became_active":"2024-09-11T03:46:19.004558+0000","last_became_peered":"2024-09-11T03:46:19.004558+0000","last_unstale":"2024-09-11T03:47:27.094698+0000","last_undegraded":"2024-09-11T03:47:27.094698+0000","last_fullsized":"2024-09-11T03:47:27.094698+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"541'254","last_scrub_stamp":"2024-09-11T03:46:12.749744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:12.749744+0000","objects_scrubbed":1,"log_size":276,"log_dups_size":0,"ondisk_log_size":276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00088591899999999999,"stat_sum":{"num_bytes":3445600,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":8234,"num_write":36,"num_write_kb":8917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"105","length":"3"}]},{"pgid":"1.18","version":"20'92","reported_seq":968,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.036889+0000","last_change":"2024-09-11T03:46:17.873917+0000","last_active":"2024-09-11T03:47:26.036889+0000","last_peered":"2024-09-11T03:47:26.036889+0000","last_clean":"2024-09-11T03:47:26.036889+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:47:26.036889+0000","last_undegraded":"2024-09-11T03:47:26.036889+0000","last_fullsized":"2024-09-11T03:47:26.036889+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:17.873858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:17.873858+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:45.517409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"628'697","reported_seq":3540,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:27.002013+0000","last_change":"2024-09-11T03:47:26.061539+0000","last_active":"2024-09-11T03:47:27.002013+0000","last_peered":"2024-09-11T03:47:27.002013+0000","last_clean":"2024-09-11T03:47:27.002013+0000","last_became_active":"2024-09-11T03:46:32.975567+0000","last_became_peered":"2024-09-11T03:46:32.975567+0000","last_unstale":"2024-09-11T03:47:27.002013+0000","last_undegraded":"2024-09-11T03:47:27.002013+0000","last_fullsized":"2024-09-11T03:47:27.002013+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"575'648","last_scrub_stamp":"2024-09-11T03:46:33.071702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.071702+0000","objects_scrubbed":5,"log_size":697,"log_dups_size":0,"ondisk_log_size":697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:39.747546+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003208116,"stat_sum":{"num_bytes":9076242,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":45255,"num_write":11,"num_write_kb":10027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7],"acting":[11,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"105","length":"3"}]},{"pgid":"1.19","version":"20'92","reported_seq":463,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.991371+0000","last_change":"2024-09-11T03:46:15.973211+0000","last_active":"2024-09-11T03:47:26.991371+0000","last_peered":"2024-09-11T03:47:26.991371+0000","last_clean":"2024-09-11T03:47:26.991371+0000","last_became_active":"2024-09-11T03:46:15.972841+0000","last_became_peered":"2024-09-11T03:46:15.972841+0000","last_unstale":"2024-09-11T03:47:26.991371+0000","last_undegraded":"2024-09-11T03:47:26.991371+0000","last_fullsized":"2024-09-11T03:47:26.991371+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:30.424428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:30.424428+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:43.252639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"557'273","reported_seq":2942,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:27.094669+0000","last_change":"2024-09-11T03:47:26.158001+0000","last_active":"2024-09-11T03:47:27.094669+0000","last_peered":"2024-09-11T03:47:27.094669+0000","last_clean":"2024-09-11T03:47:27.094669+0000","last_became_active":"2024-09-11T03:46:32.971003+0000","last_became_peered":"2024-09-11T03:46:32.971003+0000","last_unstale":"2024-09-11T03:47:27.094669+0000","last_undegraded":"2024-09-11T03:47:27.094669+0000","last_fullsized":"2024-09-11T03:47:27.094669+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:46:34.407744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:34.407744+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:03.771007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00106073,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,2],"acting":[6,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"105","length":"3"}]},{"pgid":"1.1a","version":"20'92","reported_seq":595,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.010342+0000","last_change":"2024-09-11T03:45:52.914136+0000","last_active":"2024-09-11T03:47:26.010342+0000","last_peered":"2024-09-11T03:47:26.010342+0000","last_clean":"2024-09-11T03:47:26.010342+0000","last_became_active":"2024-09-11T03:44:43.938873+0000","last_became_peered":"2024-09-11T03:44:43.938873+0000","last_unstale":"2024-09-11T03:47:26.010342+0000","last_undegraded":"2024-09-11T03:47:26.010342+0000","last_fullsized":"2024-09-11T03:47:26.010342+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:52.914088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:52.914088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:11.184197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"625'489","reported_seq":3286,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:27.094478+0000","last_change":"2024-09-11T03:47:26.157115+0000","last_active":"2024-09-11T03:47:27.094478+0000","last_peered":"2024-09-11T03:47:27.094478+0000","last_clean":"2024-09-11T03:47:27.094478+0000","last_became_active":"2024-09-11T03:46:19.002002+0000","last_became_peered":"2024-09-11T03:46:19.002002+0000","last_unstale":"2024-09-11T03:47:27.094478+0000","last_undegraded":"2024-09-11T03:47:27.094478+0000","last_fullsized":"2024-09-11T03:47:27.094478+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"594'474","last_scrub_stamp":"2024-09-11T03:46:52.412086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:52.412086+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:58.166522+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000795067,"stat_sum":{"num_bytes":2235463,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":45911,"num_write":39,"num_write_kb":20159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4870845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"105","length":"3"}]},{"pgid":"1.1b","version":"20'92","reported_seq":837,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.047224+0000","last_change":"2024-09-11T03:47:25.035842+0000","last_active":"2024-09-11T03:47:26.047224+0000","last_peered":"2024-09-11T03:47:26.047224+0000","last_clean":"2024-09-11T03:47:26.047224+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:47:26.047224+0000","last_undegraded":"2024-09-11T03:47:26.047224+0000","last_fullsized":"2024-09-11T03:47:26.047224+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:25.035793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:25.035793+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:30.302122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"625'377","reported_seq":3144,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-11T03:47:23.007465+0000","last_change":"2024-09-11T03:47:22.999823+0000","last_active":"2024-09-11T03:47:23.007465+0000","last_peered":"2024-09-11T03:47:23.007465+0000","last_clean":"2024-09-11T03:47:23.007465+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:47:23.007465+0000","last_undegraded":"2024-09-11T03:47:23.007465+0000","last_fullsized":"2024-09-11T03:47:23.007465+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"547'357","last_scrub_stamp":"2024-09-11T03:46:19.692855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:19.692855+0000","objects_scrubbed":1,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:45.653189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0053414220000000002,"stat_sum":{"num_bytes":6622288,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":121559,"num_write":61,"num_write_kb":24133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4865740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"103","length":"4"}]},{"pgid":"2.1f","version":"615'510","reported_seq":3183,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-11T03:47:25.021264+0000","last_change":"2024-09-11T03:47:23.163330+0000","last_active":"2024-09-11T03:47:25.021264+0000","last_peered":"2024-09-11T03:47:25.021264+0000","last_clean":"2024-09-11T03:47:25.021264+0000","last_became_active":"2024-09-11T03:46:32.968696+0000","last_became_peered":"2024-09-11T03:46:32.968696+0000","last_unstale":"2024-09-11T03:47:25.021264+0000","last_undegraded":"2024-09-11T03:47:25.021264+0000","last_fullsized":"2024-09-11T03:47:25.021264+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:46:33.227651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.227651+0000","objects_scrubbed":3,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:58.476088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019915269999999999,"stat_sum":{"num_bytes":6891200,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":47281,"num_write":28,"num_write_kb":9969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"103","length":"4"}]},{"pgid":"1.1c","version":"20'92","reported_seq":844,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.047368+0000","last_change":"2024-09-11T03:46:20.752592+0000","last_active":"2024-09-11T03:47:26.047368+0000","last_peered":"2024-09-11T03:47:26.047368+0000","last_clean":"2024-09-11T03:47:26.047368+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:47:26.047368+0000","last_undegraded":"2024-09-11T03:47:26.047368+0000","last_fullsized":"2024-09-11T03:47:26.047368+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:20.752526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:20.752526+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:32.553208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1e","version":"620'911","reported_seq":3586,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-11T03:47:25.021176+0000","last_change":"2024-09-11T03:47:23.162683+0000","last_active":"2024-09-11T03:47:25.021176+0000","last_peered":"2024-09-11T03:47:25.021176+0000","last_clean":"2024-09-11T03:47:25.021176+0000","last_became_active":"2024-09-11T03:46:32.970637+0000","last_became_peered":"2024-09-11T03:46:32.970637+0000","last_unstale":"2024-09-11T03:47:25.021176+0000","last_undegraded":"2024-09-11T03:47:25.021176+0000","last_fullsized":"2024-09-11T03:47:25.021176+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"575'848","last_scrub_stamp":"2024-09-11T03:46:34.276017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:34.276017+0000","objects_scrubbed":4,"log_size":911,"log_dups_size":0,"ondisk_log_size":911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:51.494141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001341189,"stat_sum":{"num_bytes":7429024,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":33,"num_write_kb":15522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"103","length":"4"}]},{"pgid":"1.1d","version":"20'92","reported_seq":836,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.010524+0000","last_change":"2024-09-11T03:47:22.312507+0000","last_active":"2024-09-11T03:47:26.010524+0000","last_peered":"2024-09-11T03:47:26.010524+0000","last_clean":"2024-09-11T03:47:26.010524+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:47:26.010524+0000","last_undegraded":"2024-09-11T03:47:26.010524+0000","last_fullsized":"2024-09-11T03:47:26.010524+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:22.312456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:22.312456+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:44.701057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"567'752","reported_seq":3436,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.991769+0000","last_change":"2024-09-11T03:47:26.013581+0000","last_active":"2024-09-11T03:47:26.991769+0000","last_peered":"2024-09-11T03:47:26.991769+0000","last_clean":"2024-09-11T03:47:26.991769+0000","last_became_active":"2024-09-11T03:46:32.972371+0000","last_became_peered":"2024-09-11T03:46:32.972371+0000","last_unstale":"2024-09-11T03:47:26.991769+0000","last_undegraded":"2024-09-11T03:47:26.991769+0000","last_fullsized":"2024-09-11T03:47:26.991769+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:46:34.138345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:34.138345+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:56.291291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081143399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,11],"acting":[10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"105","length":"3"}]},{"pgid":"1.1e","version":"20'92","reported_seq":850,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-11T03:47:26.996727+0000","last_change":"2024-09-11T03:47:00.206764+0000","last_active":"2024-09-11T03:47:26.996727+0000","last_peered":"2024-09-11T03:47:26.996727+0000","last_clean":"2024-09-11T03:47:26.996727+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:47:26.996727+0000","last_undegraded":"2024-09-11T03:47:26.996727+0000","last_fullsized":"2024-09-11T03:47:26.996727+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:00.206720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:00.206720+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:28.907706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":36,"stat_sum":{"num_bytes":130802117,"num_objects":44,"num_object_clones":5,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":1771,"num_read_kb":1861180,"num_write":1188,"num_write_kb":446059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":59763304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":238841856,"data_stored":242781722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16362,"ondisk_log_size":16362,"up":108,"acting":108,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285691,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63272,"kb_used_data":23756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93681880,"statfs":{"total":95995035648,"available":95930245120,"internally_reserved":0,"allocated":24326144,"data_stored":21537622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514623,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60576,"kb_used_data":25228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93684576,"statfs":{"total":95995035648,"available":95933005824,"internally_reserved":0,"allocated":25833472,"data_stored":22974606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743556,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57196,"kb_used_data":24336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93687956,"statfs":{"total":95995035648,"available":95936466944,"internally_reserved":0,"allocated":24920064,"data_stored":22128790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037900,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54132,"kb_used_data":22940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93691020,"statfs":{"total":95995035648,"available":95939604480,"internally_reserved":0,"allocated":23490560,"data_stored":20678775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299535,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70904,"kb_used_data":35552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93674248,"statfs":{"total":95995035648,"available":95922429952,"internally_reserved":0,"allocated":36405248,"data_stored":33499184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528468,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60460,"kb_used_data":25104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93684692,"statfs":{"total":95995035648,"available":95933124608,"internally_reserved":0,"allocated":25706496,"data_stored":24611882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757402,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62096,"kb_used_data":22584,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93683056,"statfs":{"total":95995035648,"available":95931449344,"internally_reserved":0,"allocated":23126016,"data_stored":20298929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986335,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52280,"kb_used_data":25252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93692872,"statfs":{"total":95995035648,"available":95941500928,"internally_reserved":0,"allocated":25858048,"data_stored":23062287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86520,"kb_used_data":33124,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93658632,"statfs":{"total":95995035648,"available":95906439168,"internally_reserved":0,"allocated":33918976,"data_stored":31000968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411497,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45696,"kb_used_data":18664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93699456,"statfs":{"total":95995035648,"available":95948242944,"internally_reserved":0,"allocated":19111936,"data_stored":16327688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575023,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49068,"kb_used_data":22032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93696084,"statfs":{"total":95995035648,"available":95944790016,"internally_reserved":0,"allocated":22560768,"data_stored":19815493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738548,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67716,"kb_used_data":36524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93677436,"statfs":{"total":95995035648,"available":95925694464,"internally_reserved":0,"allocated":37400576,"data_stored":34502647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29507584,"data_stored":29293465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14983168,"data_stored":14925728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11378688,"data_stored":11269599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27357184,"data_stored":27118889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20099072,"data_stored":19963885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":15343616,"data_stored":16946237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":17801216,"data_stored":19394437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32964608,"data_stored":34772357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":15925248,"data_stored":15797273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":17379328,"data_stored":17263686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":19353600,"data_stored":19388309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16748544,"data_stored":16647857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:47:29.440 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:47:29.443 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:29.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:29.444 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:29 smithi175 ceph-mon[30894]: 1.a scrub starts 2024-09-11T03:47:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:29 smithi175 ceph-mon[30894]: 1.a scrub ok 2024-09-11T03:47:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:29 smithi175 ceph-mon[30894]: from='client.16569 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:47:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:29 smithi175 ceph-mon[30894]: osdmap e631: 12 total, 12 up, 12 in 2024-09-11T03:47:29.545 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:29.568 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:29.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:29.569 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:29.654 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:29.669 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:29.701 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:29.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:29.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:29.804 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:29.828 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:29.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:29.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:29.931 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:29.955 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:29.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:29.956 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:29.980 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:47:29.984 INFO:tasks.rados.rados.2.smithi175.stdout:2421: done (0 left) 2024-09-11T03:47:29.984 INFO:tasks.rados.rados.2.smithi175.stdout:2422: read oid 14 snap -1 2024-09-11T03:47:29.984 INFO:tasks.rados.rados.2.smithi175.stdout:2422: expect deleted 2024-09-11T03:47:29.984 INFO:tasks.rados.rados.2.smithi175.stdout:2423: copy_from oid 8 from oid 21 current snap is 266 2024-09-11T03:47:29.984 INFO:tasks.rados.rados.2.smithi175.stdout:2424: write (excl) oid 34 current snap is 266 2024-09-11T03:47:29.984 INFO:tasks.rados.rados.2.smithi175.stdout:2424: seq_num 839 ranges {501073=732741,1896510=204563} 2024-09-11T03:47:29.990 INFO:tasks.rados.rados.2.smithi175.stdout:2424: writing smithi17557870-34 from 501073 to 1233814 tid 1 2024-09-11T03:47:29.995 INFO:tasks.rados.rados.2.smithi175.stdout:2424: writing smithi17557870-34 from 1896510 to 2101073 tid 2 2024-09-11T03:47:29.996 INFO:tasks.rados.rados.2.smithi175.stdout:2425: snap_remove snap 265 2024-09-11T03:47:30.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:30.083 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:30.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:30.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:30.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:30 smithi084 ceph-mon[27254]: 1.0 scrub starts 2024-09-11T03:47:30.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:30 smithi084 ceph-mon[27254]: 1.0 scrub ok 2024-09-11T03:47:30.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:30 smithi084 ceph-mon[27254]: pgmap v1101: 67 pgs: 67 active+clean; 130 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 4.3 MiB/s wr, 33 op/s 2024-09-11T03:47:30.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:30 smithi084 ceph-mon[27254]: 1.17 scrub starts 2024-09-11T03:47:30.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:30 smithi084 ceph-mon[27254]: 1.17 scrub ok 2024-09-11T03:47:30.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:30 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1679318684' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:47:30.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:30 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:30.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:30 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1679318684' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:47:30.087 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:30 smithi084 ceph-mon[27254]: osdmap e632: 12 total, 12 up, 12 in 2024-09-11T03:47:30.184 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:30.203 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:47:30.203 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:47:30.208 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:30.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:30.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:30.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:30.332 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:30.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:30.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:30 smithi121 ceph-mon[30729]: 1.0 scrub starts 2024-09-11T03:47:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:30 smithi121 ceph-mon[30729]: 1.0 scrub ok 2024-09-11T03:47:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:30 smithi121 ceph-mon[30729]: pgmap v1101: 67 pgs: 67 active+clean; 130 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 4.3 MiB/s wr, 33 op/s 2024-09-11T03:47:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:30 smithi121 ceph-mon[30729]: 1.17 scrub starts 2024-09-11T03:47:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:30 smithi121 ceph-mon[30729]: 1.17 scrub ok 2024-09-11T03:47:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:30 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1679318684' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:47:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:30 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:30 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1679318684' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:47:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:30 smithi121 ceph-mon[30729]: osdmap e632: 12 total, 12 up, 12 in 2024-09-11T03:47:30.434 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:30.458 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:30.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:30.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:30 smithi175 ceph-mon[30894]: 1.0 scrub starts 2024-09-11T03:47:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:30 smithi175 ceph-mon[30894]: 1.0 scrub ok 2024-09-11T03:47:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:30 smithi175 ceph-mon[30894]: pgmap v1101: 67 pgs: 67 active+clean; 130 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s rd, 4.3 MiB/s wr, 33 op/s 2024-09-11T03:47:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:30 smithi175 ceph-mon[30894]: 1.17 scrub starts 2024-09-11T03:47:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:30 smithi175 ceph-mon[30894]: 1.17 scrub ok 2024-09-11T03:47:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:30 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1679318684' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:47:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:30 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:30 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1679318684' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:47:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:30 smithi175 ceph-mon[30894]: osdmap e632: 12 total, 12 up, 12 in 2024-09-11T03:47:30.561 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:30.587 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:30.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:30.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:30.673 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:30.689 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:30.705 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:30.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:30.712 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:30.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:30.813 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:30.839 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:30.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:30.840 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:30.941 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:30.967 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:30.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:30.968 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:31.000 INFO:tasks.rados.rados.2.smithi175.stdout:2425: done (3 left) 2024-09-11T03:47:31.001 INFO:tasks.rados.rados.2.smithi175.stdout:2426: write (excl) oid 33 current snap is 266 2024-09-11T03:47:31.001 INFO:tasks.rados.rados.2.smithi175.stdout:2426: seq_num 840 ranges {517880=407990,1554992=562888} 2024-09-11T03:47:31.005 INFO:tasks.rados.rados.2.smithi175.stdout:2426: writing smithi17557870-33 from 517880 to 925870 tid 1 2024-09-11T03:47:31.012 INFO:tasks.rados.rados.2.smithi175.stdout:2426: writing smithi17557870-33 from 1554992 to 2117880 tid 2 2024-09-11T03:47:31.013 INFO:tasks.rados.rados.2.smithi175.stdout:2427: write oid 50 current snap is 266 2024-09-11T03:47:31.013 INFO:tasks.rados.rados.2.smithi175.stdout:2427: seq_num 841 ranges {534687=483239,1613474=412283} 2024-09-11T03:47:31.018 INFO:tasks.rados.rados.2.smithi175.stdout:2427: writing smithi17557870-50 from 534687 to 1017926 tid 1 2024-09-11T03:47:31.025 INFO:tasks.rados.rados.2.smithi175.stdout:2427: writing smithi17557870-50 from 1613474 to 2025757 tid 2 2024-09-11T03:47:31.025 INFO:tasks.rados.rados.2.smithi175.stdout:2428: read oid 16 snap 264 2024-09-11T03:47:31.025 INFO:tasks.rados.rados.2.smithi175.stdout:2428: expect deleted 2024-09-11T03:47:31.025 INFO:tasks.rados.rados.2.smithi175.stdout:2429: read oid 6 snap -1 2024-09-11T03:47:31.026 INFO:tasks.rados.rados.2.smithi175.stdout:2429: expect (ObjNum 609 snap 191 seq_num 609) 2024-09-11T03:47:31.026 INFO:tasks.rados.rados.2.smithi175.stdout:2430: copy_from oid 26 from oid 30 current snap is 266 2024-09-11T03:47:31.026 INFO:tasks.rados.rados.2.smithi175.stdout:2423: finishing copy_from to smithi17557870-8 2024-09-11T03:47:31.026 INFO:tasks.rados.rados.2.smithi175.stdout:2423: got expected ENOENT (src dne) 2024-09-11T03:47:31.026 INFO:tasks.rados.rados.2.smithi175.stdout:2423: finishing copy_from racing read to smithi17557870-8 2024-09-11T03:47:31.026 INFO:tasks.rados.rados.2.smithi175.stdout:2424: finishing write tid 1 to smithi17557870-34 2024-09-11T03:47:31.026 INFO:tasks.rados.rados.2.smithi175.stdout:2424: finishing write tid 2 to smithi17557870-34 2024-09-11T03:47:31.026 INFO:tasks.rados.rados.2.smithi175.stdout:2424: finishing write tid 3 to smithi17557870-34 2024-09-11T03:47:31.027 INFO:tasks.rados.rados.2.smithi175.stdout:2424: finishing write tid 4 to smithi17557870-34 2024-09-11T03:47:31.027 INFO:tasks.rados.rados.2.smithi175.stdout:2424: oid 34 updating version 0 to 875 2024-09-11T03:47:31.027 INFO:tasks.rados.rados.2.smithi175.stdout:2424: oid 34 version 875 is already newer than 874 2024-09-11T03:47:31.027 INFO:tasks.rados.rados.2.smithi175.stdout:2424: oid 34 version 875 is already newer than 873 2024-09-11T03:47:31.027 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 875 (ObjNum 839 snap 266 seq_num 839) dirty exists 2024-09-11T03:47:31.027 INFO:tasks.rados.rados.2.smithi175.stdout:2424: left oid 34 (ObjNum 839 snap 266 seq_num 839) 2024-09-11T03:47:31.027 INFO:tasks.rados.rados.2.smithi175.stdout:2426: finishing write tid 1 to smithi17557870-33 2024-09-11T03:47:31.027 INFO:tasks.rados.rados.2.smithi175.stdout:2426: finishing write tid 2 to smithi17557870-33 2024-09-11T03:47:31.027 INFO:tasks.rados.rados.2.smithi175.stdout:2426: finishing write tid 3 to smithi17557870-33 2024-09-11T03:47:31.028 INFO:tasks.rados.rados.2.smithi175.stdout:2426: finishing write tid 4 to smithi17557870-33 2024-09-11T03:47:31.028 INFO:tasks.rados.rados.2.smithi175.stdout:2426: oid 33 updating version 0 to 924 2024-09-11T03:47:31.028 INFO:tasks.rados.rados.2.smithi175.stdout:2426: oid 33 updating version 924 to 925 2024-09-11T03:47:31.028 INFO:tasks.rados.rados.2.smithi175.stdout:2426: oid 33 updating version 925 to 926 2024-09-11T03:47:31.028 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 926 (ObjNum 840 snap 266 seq_num 840) dirty exists 2024-09-11T03:47:31.028 INFO:tasks.rados.rados.2.smithi175.stdout:2426: left oid 33 (ObjNum 840 snap 266 seq_num 840) 2024-09-11T03:47:31.028 INFO:tasks.rados.rados.2.smithi175.stdout:2427: finishing write tid 1 to smithi17557870-50 2024-09-11T03:47:31.028 INFO:tasks.rados.rados.2.smithi175.stdout:2422: done (7 left) 2024-09-11T03:47:31.028 INFO:tasks.rados.rados.2.smithi175.stdout:2423: done (6 left) 2024-09-11T03:47:31.028 INFO:tasks.rados.rados.2.smithi175.stdout:2424: done (5 left) 2024-09-11T03:47:31.028 INFO:tasks.rados.rados.2.smithi175.stdout:2426: done (4 left) 2024-09-11T03:47:31.028 INFO:tasks.rados.rados.2.smithi175.stdout:2431: rollback oid 36 current snap is 266 2024-09-11T03:47:31.028 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 36 to 264 2024-09-11T03:47:31.028 INFO:tasks.rados.rados.2.smithi175.stdout:2432: read oid 13 snap -1 2024-09-11T03:47:31.029 INFO:tasks.rados.rados.2.smithi175.stdout:2432: expect (ObjNum 791 snap 252 seq_num 791) 2024-09-11T03:47:31.029 INFO:tasks.rados.rados.2.smithi175.stdout:2433: snap_remove snap 263 2024-09-11T03:47:31.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:31.093 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:31.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:31.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:31.181 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:47:31.182 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:47:31.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:31.198 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:47:31.199 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:47:31.214 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:47:31.214 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:47:31.219 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:31.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:31.230 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:47:31.230 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:47:31.247 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:47:31.247 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:47:31.264 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:47:31.264 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:47:31.279 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:47:31.280 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:47:31.296 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:47:31.296 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:47:31.313 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:47:31.313 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:47:31.321 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:31.330 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:47:31.331 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:47:31.342 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:31.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:31.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:31.348 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:47:31.348 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:47:31.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:47:31.366 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:47:31.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:31.469 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:31.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:31.571 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:31.592 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:31.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:31.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:31.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:31.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:31.721 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:31.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:31.822 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:31.849 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:31.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:31.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:31.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:31.977 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:31.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:31.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:31.984 INFO:tasks.rados.rados.2.smithi175.stdout:2433: done (6 left) 2024-09-11T03:47:31.984 INFO:tasks.rados.rados.2.smithi175.stdout:2434: delete oid 33 current snap is 266 2024-09-11T03:47:31.984 INFO:tasks.rados.rados.2.smithi175.stdout:2427: finishing write tid 2 to smithi17557870-50 2024-09-11T03:47:31.985 INFO:tasks.rados.rados.2.smithi175.stdout:2427: finishing write tid 3 to smithi17557870-50 2024-09-11T03:47:31.985 INFO:tasks.rados.rados.2.smithi175.stdout:2427: finishing write tid 4 to smithi17557870-50 2024-09-11T03:47:31.985 INFO:tasks.rados.rados.2.smithi175.stdout:2427: oid 50 updating version 0 to 927 2024-09-11T03:47:31.985 INFO:tasks.rados.rados.2.smithi175.stdout:2427: oid 50 updating version 927 to 928 2024-09-11T03:47:31.985 INFO:tasks.rados.rados.2.smithi175.stdout:2427: oid 50 updating version 928 to 929 2024-09-11T03:47:31.985 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 929 (ObjNum 841 snap 266 seq_num 841) dirty exists 2024-09-11T03:47:31.985 INFO:tasks.rados.rados.2.smithi175.stdout:2427: left oid 50 (ObjNum 841 snap 266 seq_num 841) 2024-09-11T03:47:31.985 INFO:tasks.rados.rados.2.smithi175.stdout:2431: finishing rollback tid 0 to smithi17557870-36 2024-09-11T03:47:31.985 INFO:tasks.rados.rados.2.smithi175.stdout:2431: finishing rollback tid 1 to smithi17557870-36 2024-09-11T03:47:31.985 INFO:tasks.rados.rados.2.smithi175.stdout:2431: finishing rollback tid 2 to smithi17557870-36 2024-09-11T03:47:31.985 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 856 (ObjNum 830 snap 263 seq_num 830) dirty exists 2024-09-11T03:47:31.985 INFO:tasks.rados.rados.2.smithi175.stdout:2432: expect (ObjNum 791 snap 252 seq_num 791) 2024-09-11T03:47:32.079 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:32.092 INFO:tasks.rados.rados.2.smithi175.stdout:2430: finishing copy_from racing read to smithi17557870-26 2024-09-11T03:47:32.092 INFO:tasks.rados.rados.2.smithi175.stdout:2429: expect (ObjNum 609 snap 191 seq_num 609) 2024-09-11T03:47:32.101 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:32.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:32.101 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:32.120 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T03:47:32.199 INFO:tasks.rados.rados.2.smithi175.stdout:2430: finishing copy_from to smithi17557870-26 2024-09-11T03:47:32.199 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 837 (ObjNum 793 snap 252 seq_num 793) dirty exists 2024-09-11T03:47:32.199 INFO:tasks.rados.rados.2.smithi175.stdout:2427: done (6 left) 2024-09-11T03:47:32.199 INFO:tasks.rados.rados.2.smithi175.stdout:2428: done (5 left) 2024-09-11T03:47:32.199 INFO:tasks.rados.rados.2.smithi175.stdout:2429: done (4 left) 2024-09-11T03:47:32.199 INFO:tasks.rados.rados.2.smithi175.stdout:2430: done (3 left) 2024-09-11T03:47:32.199 INFO:tasks.rados.rados.2.smithi175.stdout:2431: done (2 left) 2024-09-11T03:47:32.199 INFO:tasks.rados.rados.2.smithi175.stdout:2432: done (1 left) 2024-09-11T03:47:32.199 INFO:tasks.rados.rados.2.smithi175.stdout:2434: done (0 left) 2024-09-11T03:47:32.199 INFO:tasks.rados.rados.2.smithi175.stdout:2435: write oid 19 current snap is 266 2024-09-11T03:47:32.199 INFO:tasks.rados.rados.2.smithi175.stdout:2435: seq_num 843 ranges {568301=633737,1614085=415952} 2024-09-11T03:47:32.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:32.205 INFO:tasks.rados.rados.2.smithi175.stdout:2435: writing smithi17557870-19 from 568301 to 1202038 tid 1 2024-09-11T03:47:32.209 INFO:tasks.rados.rados.2.smithi175.stdout:2435: writing smithi17557870-19 from 1614085 to 2030037 tid 2 2024-09-11T03:47:32.210 INFO:tasks.rados.rados.2.smithi175.stdout:2436: delete oid 8 current snap is 266 2024-09-11T03:47:32.210 INFO:tasks.rados.rados.2.smithi175.stdout:2435: finishing write tid 1 to smithi17557870-19 2024-09-11T03:47:32.212 INFO:tasks.rados.rados.2.smithi175.stdout:2436: done (1 left) 2024-09-11T03:47:32.212 INFO:tasks.rados.rados.2.smithi175.stdout:2437: copy_from oid 24 from oid 8 current snap is 266 2024-09-11T03:47:32.212 INFO:tasks.rados.rados.2.smithi175.stdout:2438: write oid 11 current snap is 266 2024-09-11T03:47:32.212 INFO:tasks.rados.rados.2.smithi175.stdout:2438: seq_num 845 ranges {601915=500588,1847402=354513} 2024-09-11T03:47:32.217 INFO:tasks.rados.rados.2.smithi175.stdout:2438: writing smithi17557870-11 from 601915 to 1102503 tid 1 2024-09-11T03:47:32.223 INFO:tasks.rados.rados.2.smithi175.stdout:2438: writing smithi17557870-11 from 1847402 to 2201915 tid 2 2024-09-11T03:47:32.223 INFO:tasks.rados.rados.2.smithi175.stdout:2439: snap_remove snap 264 2024-09-11T03:47:32.224 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:32.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:32.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:32.325 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:32.347 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:32.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:32.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:32.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:31 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:47:32.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:31 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:47:32.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:31 smithi084 ceph-mon[27254]: pgmap v1104: 67 pgs: 67 active+clean; 130 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 3.1 MiB/s wr, 16 op/s 2024-09-11T03:47:32.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:31 smithi084 ceph-mon[27254]: osdmap e633: 12 total, 12 up, 12 in 2024-09-11T03:47:32.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:31 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3938890478' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:32.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:31 smithi084 ceph-mon[27254]: osdmap e634: 12 total, 12 up, 12 in 2024-09-11T03:47:32.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:31 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:47:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:31 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:47:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:31 smithi121 ceph-mon[30729]: pgmap v1104: 67 pgs: 67 active+clean; 130 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 3.1 MiB/s wr, 16 op/s 2024-09-11T03:47:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:31 smithi121 ceph-mon[30729]: osdmap e633: 12 total, 12 up, 12 in 2024-09-11T03:47:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:31 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3938890478' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:31 smithi121 ceph-mon[30729]: osdmap e634: 12 total, 12 up, 12 in 2024-09-11T03:47:32.448 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:32.469 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:32.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:32.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:31 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:47:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:31 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:47:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:31 smithi175 ceph-mon[30894]: pgmap v1104: 67 pgs: 67 active+clean; 130 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 3.1 MiB/s wr, 16 op/s 2024-09-11T03:47:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:31 smithi175 ceph-mon[30894]: osdmap e633: 12 total, 12 up, 12 in 2024-09-11T03:47:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:31 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3938890478' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:31 smithi175 ceph-mon[30894]: osdmap e634: 12 total, 12 up, 12 in 2024-09-11T03:47:32.570 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:32.592 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:32.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:32.593 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:32.694 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:32.715 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:32.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:32.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:32.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:32.843 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:32.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:32.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:32.945 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:32.966 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:32.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:32.967 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:33.002 INFO:tasks.rados.rados.2.smithi175.stdout:2439: done (3 left) 2024-09-11T03:47:33.003 INFO:tasks.rados.rados.2.smithi175.stdout:2440: rollback oid 33 current snap is 266 2024-09-11T03:47:33.003 INFO:tasks.rados.rados.2.smithi175.stdout:2440: done (3 left) 2024-09-11T03:47:33.003 INFO:tasks.rados.rados.2.smithi175.stdout:2441: rollback oid 23 current snap is 266 2024-09-11T03:47:33.003 INFO:tasks.rados.rados.2.smithi175.stdout:2441: done (3 left) 2024-09-11T03:47:33.003 INFO:tasks.rados.rados.2.smithi175.stdout:2442: snap_create 2024-09-11T03:47:33.003 INFO:tasks.rados.rados.2.smithi175.stdout:2435: finishing write tid 2 to smithi17557870-19 2024-09-11T03:47:33.003 INFO:tasks.rados.rados.2.smithi175.stdout:2435: finishing write tid 3 to smithi17557870-19 2024-09-11T03:47:33.003 INFO:tasks.rados.rados.2.smithi175.stdout:2437: finishing copy_from to smithi17557870-24 2024-09-11T03:47:33.003 INFO:tasks.rados.rados.2.smithi175.stdout:2437: got expected ENOENT (src dne) 2024-09-11T03:47:33.003 INFO:tasks.rados.rados.2.smithi175.stdout:2435: finishing write tid 4 to smithi17557870-19 2024-09-11T03:47:33.003 INFO:tasks.rados.rados.2.smithi175.stdout:2435: oid 19 updating version 0 to 534 2024-09-11T03:47:33.003 INFO:tasks.rados.rados.2.smithi175.stdout:2435: oid 19 version 534 is already newer than 533 2024-09-11T03:47:33.003 INFO:tasks.rados.rados.2.smithi175.stdout:2435: oid 19 updating version 534 to 535 2024-09-11T03:47:33.003 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 535 (ObjNum 843 snap 266 seq_num 843) dirty exists 2024-09-11T03:47:33.004 INFO:tasks.rados.rados.2.smithi175.stdout:2435: left oid 19 (ObjNum 843 snap 266 seq_num 843) 2024-09-11T03:47:33.004 INFO:tasks.rados.rados.2.smithi175.stdout:2437: finishing copy_from racing read to smithi17557870-24 2024-09-11T03:47:33.004 INFO:tasks.rados.rados.2.smithi175.stdout:2438: finishing write tid 1 to smithi17557870-11 2024-09-11T03:47:33.004 INFO:tasks.rados.rados.2.smithi175.stdout:2438: finishing write tid 2 to smithi17557870-11 2024-09-11T03:47:33.004 INFO:tasks.rados.rados.2.smithi175.stdout:2438: finishing write tid 3 to smithi17557870-11 2024-09-11T03:47:33.004 INFO:tasks.rados.rados.2.smithi175.stdout:2438: finishing write tid 4 to smithi17557870-11 2024-09-11T03:47:33.004 INFO:tasks.rados.rados.2.smithi175.stdout:2438: oid 11 updating version 0 to 603 2024-09-11T03:47:33.004 INFO:tasks.rados.rados.2.smithi175.stdout:2438: oid 11 version 603 is already newer than 602 2024-09-11T03:47:33.004 INFO:tasks.rados.rados.2.smithi175.stdout:2438: oid 11 version 603 is already newer than 601 2024-09-11T03:47:33.004 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 603 (ObjNum 845 snap 266 seq_num 845) dirty exists 2024-09-11T03:47:33.004 INFO:tasks.rados.rados.2.smithi175.stdout:2438: left oid 11 (ObjNum 845 snap 266 seq_num 845) 2024-09-11T03:47:33.068 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:33.089 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:33.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:33.091 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:33.192 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:33.213 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:33.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:33.316 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:33.338 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:33.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:33.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:33.440 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:33.463 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:33.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:33.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:33.485 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:33.566 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:33.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:33.589 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:33.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:33.657 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:47:33.658 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:47:33.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:33.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:33.712 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:33.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:33.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:33.840 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:33.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:33.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:33.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:33.964 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:33.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:33.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:34.005 INFO:tasks.rados.rados.2.smithi175.stdout:2435: done (3 left) 2024-09-11T03:47:34.005 INFO:tasks.rados.rados.2.smithi175.stdout:2437: done (2 left) 2024-09-11T03:47:34.005 INFO:tasks.rados.rados.2.smithi175.stdout:2438: done (1 left) 2024-09-11T03:47:34.005 INFO:tasks.rados.rados.2.smithi175.stdout:2442: done (0 left) 2024-09-11T03:47:34.005 INFO:tasks.rados.rados.2.smithi175.stdout:2443: write oid 27 current snap is 267 2024-09-11T03:47:34.005 INFO:tasks.rados.rados.2.smithi175.stdout:2443: seq_num 846 ranges {618722=575837,1905884=312838} 2024-09-11T03:47:34.011 INFO:tasks.rados.rados.2.smithi175.stdout:2443: writing smithi17557870-27 from 618722 to 1194559 tid 1 2024-09-11T03:47:34.017 INFO:tasks.rados.rados.2.smithi175.stdout:2443: writing smithi17557870-27 from 1905884 to 2218722 tid 2 2024-09-11T03:47:34.017 INFO:tasks.rados.rados.2.smithi175.stdout:2444: copy_from oid 5 from oid 11 current snap is 267 2024-09-11T03:47:34.017 INFO:tasks.rados.rados.2.smithi175.stdout:2445: delete oid 47 current snap is 267 2024-09-11T03:47:34.020 INFO:tasks.rados.rados.2.smithi175.stdout:2443: finishing write tid 1 to smithi17557870-27 2024-09-11T03:47:34.022 INFO:tasks.rados.rados.2.smithi175.stdout:2443: finishing write tid 2 to smithi17557870-27 2024-09-11T03:47:34.022 INFO:tasks.rados.rados.2.smithi175.stdout:2443: finishing write tid 3 to smithi17557870-27 2024-09-11T03:47:34.022 INFO:tasks.rados.rados.2.smithi175.stdout:2445: done (2 left) 2024-09-11T03:47:34.022 INFO:tasks.rados.rados.2.smithi175.stdout:2446: read oid 19 snap -1 2024-09-11T03:47:34.022 INFO:tasks.rados.rados.2.smithi175.stdout:2446: expect (ObjNum 843 snap 266 seq_num 843) 2024-09-11T03:47:34.022 INFO:tasks.rados.rados.2.smithi175.stdout:2447: read oid 32 snap -1 2024-09-11T03:47:34.022 INFO:tasks.rados.rados.2.smithi175.stdout:2447: expect (ObjNum 777 snap 248 seq_num 777) 2024-09-11T03:47:34.022 INFO:tasks.rados.rados.2.smithi175.stdout:2448: write (excl) oid 43 current snap is 267 2024-09-11T03:47:34.022 INFO:tasks.rados.rados.2.smithi175.stdout:2443: finishing write tid 4 to smithi17557870-27 2024-09-11T03:47:34.022 INFO:tasks.rados.rados.2.smithi175.stdout:2443: oid 27 updating version 0 to 942 2024-09-11T03:47:34.022 INFO:tasks.rados.rados.2.smithi175.stdout:2443: oid 27 version 942 is already newer than 941 2024-09-11T03:47:34.023 INFO:tasks.rados.rados.2.smithi175.stdout:2443: oid 27 updating version 942 to 943 2024-09-11T03:47:34.023 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 943 (ObjNum 846 snap 267 seq_num 846) dirty exists 2024-09-11T03:47:34.023 INFO:tasks.rados.rados.2.smithi175.stdout:2443: left oid 27 (ObjNum 846 snap 267 seq_num 846) 2024-09-11T03:47:34.023 INFO:tasks.rados.rados.2.smithi175.stdout:2448: seq_num 848 ranges {652336=726335,1906495=345841} 2024-09-11T03:47:34.030 INFO:tasks.rados.rados.2.smithi175.stdout:2448: writing smithi17557870-43 from 652336 to 1378671 tid 1 2024-09-11T03:47:34.035 INFO:tasks.rados.rados.2.smithi175.stdout:2448: writing smithi17557870-43 from 1906495 to 2252336 tid 2 2024-09-11T03:47:34.035 INFO:tasks.rados.rados.2.smithi175.stdout:2443: done (4 left) 2024-09-11T03:47:34.035 INFO:tasks.rados.rados.2.smithi175.stdout:2449: snap_remove snap 266 2024-09-11T03:47:34.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:34.091 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:34.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:34.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:34.112 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:47:34.112 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:47:34.112 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:47:34.112 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:47:34.113 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:47:34.193 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:34.217 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:34.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:34.218 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:34.319 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:34.341 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:34.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:34.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:34.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:34 smithi084 ceph-mon[27254]: pgmap v1107: 67 pgs: 67 active+clean; 132 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2024-09-11T03:47:34.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:34 smithi084 ceph-mon[27254]: osdmap e635: 12 total, 12 up, 12 in 2024-09-11T03:47:34.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:34 smithi084 ceph-mon[27254]: from='client.16587 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:47:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:34 smithi121 ceph-mon[30729]: pgmap v1107: 67 pgs: 67 active+clean; 132 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2024-09-11T03:47:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:34 smithi121 ceph-mon[30729]: osdmap e635: 12 total, 12 up, 12 in 2024-09-11T03:47:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:34 smithi121 ceph-mon[30729]: from='client.16587 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:47:34.442 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:34.465 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:34.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:34.467 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:34 smithi175 ceph-mon[30894]: pgmap v1107: 67 pgs: 67 active+clean; 132 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2024-09-11T03:47:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:34 smithi175 ceph-mon[30894]: osdmap e635: 12 total, 12 up, 12 in 2024-09-11T03:47:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:34 smithi175 ceph-mon[30894]: from='client.16587 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:47:34.567 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:34.589 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:34.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:34.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:34.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:34.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:34.718 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:34.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:34.812 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T03:47:34.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:34.844 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:34.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:34.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:47:34.945 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:34.970 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:34.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:35.017 INFO:tasks.rados.rados.2.smithi175.stdout:2449: done (4 left) 2024-09-11T03:47:35.017 INFO:tasks.rados.rados.2.smithi175.stdout:2450: read oid 13 snap -1 2024-09-11T03:47:35.017 INFO:tasks.rados.rados.2.smithi175.stdout:2450: expect (ObjNum 791 snap 252 seq_num 791) 2024-09-11T03:47:35.017 INFO:tasks.rados.rados.2.smithi175.stdout:2451: rollback oid 46 current snap is 267 2024-09-11T03:47:35.018 INFO:tasks.rados.rados.2.smithi175.stdout:2451: done (5 left) 2024-09-11T03:47:35.018 INFO:tasks.rados.rados.2.smithi175.stdout:2452: write (excl) oid 50 current snap is 267 2024-09-11T03:47:35.018 INFO:tasks.rados.rados.2.smithi175.stdout:2452: seq_num 849 ranges {669143=401584,1564977=426959} 2024-09-11T03:47:35.023 INFO:tasks.rados.rados.2.smithi175.stdout:2452: writing smithi17557870-50 from 669143 to 1070727 tid 1 2024-09-11T03:47:35.028 INFO:tasks.rados.rados.2.smithi175.stdout:2452: writing smithi17557870-50 from 1564977 to 1991936 tid 2 2024-09-11T03:47:35.028 INFO:tasks.rados.rados.2.smithi175.stdout:2453: rollback oid 39 current snap is 267 2024-09-11T03:47:35.028 INFO:tasks.rados.rados.2.smithi175.stdout:2453: done (6 left) 2024-09-11T03:47:35.029 INFO:tasks.rados.rados.2.smithi175.stdout:2454: rollback oid 48 current snap is 267 2024-09-11T03:47:35.029 INFO:tasks.rados.rados.2.smithi175.stdout:2454: done (6 left) 2024-09-11T03:47:35.029 INFO:tasks.rados.rados.2.smithi175.stdout:2455: read oid 41 snap -1 2024-09-11T03:47:35.029 INFO:tasks.rados.rados.2.smithi175.stdout:2455: expect (ObjNum 708 snap 232 seq_num 708) 2024-09-11T03:47:35.029 INFO:tasks.rados.rados.2.smithi175.stdout:2456: read oid 10 snap -1 2024-09-11T03:47:35.029 INFO:tasks.rados.rados.2.smithi175.stdout:2456: expect (ObjNum 736 snap 243 seq_num 736) 2024-09-11T03:47:35.029 INFO:tasks.rados.rados.2.smithi175.stdout:2457: delete oid 9 current snap is 267 2024-09-11T03:47:35.029 INFO:tasks.rados.rados.2.smithi175.stdout:2447: expect (ObjNum 777 snap 248 seq_num 777) 2024-09-11T03:47:35.072 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:35.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:35.093 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:35.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:35.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:35 smithi084 ceph-mon[27254]: osdmap e636: 12 total, 12 up, 12 in 2024-09-11T03:47:35.194 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:35.196 INFO:tasks.rados.rados.2.smithi175.stdout:2444: finishing copy_from racing read to smithi17557870-5 2024-09-11T03:47:35.197 INFO:tasks.rados.rados.2.smithi175.stdout:2444: finishing copy_from to smithi17557870-5 2024-09-11T03:47:35.197 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 603 (ObjNum 845 snap 266 seq_num 845) dirty exists 2024-09-11T03:47:35.197 INFO:tasks.rados.rados.2.smithi175.stdout:2446: expect (ObjNum 843 snap 266 seq_num 843) 2024-09-11T03:47:35.218 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:35.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:35.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:35.223 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1108,"stamp":"2024-09-11T03:47:33.001175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":138169910,"num_objects":48,"num_object_clones":3,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":1,"num_read":1810,"num_read_kb":1900379,"num_write":1213,"num_write_kb":458224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":244,"num_bytes_recovered":60090984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19257,"ondisk_log_size":19257,"up":201,"acting":201,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":773072,"kb_used_data":333568,"kb_used_omap":18,"kb_used_meta":439213,"kb_avail":1124168752,"statfs":{"total":1151940427776,"available":1151148802048,"internally_reserved":0,"allocated":341573632,"data_stored":311192857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":449754427},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2222563,"num_objects":1,"num_object_clones":-2,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":27,"num_read_kb":27376,"num_write":19,"num_write_kb":7151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020913"},"pg_stats":[{"pgid":"2.23","version":"601'395","reported_seq":3135,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.009904+0000","last_change":"2024-09-11T03:47:31.010298+0000","last_active":"2024-09-11T03:47:31.009904+0000","last_peered":"2024-09-11T03:47:31.009904+0000","last_clean":"2024-09-11T03:47:31.009904+0000","last_became_active":"2024-09-11T03:46:32.970100+0000","last_became_peered":"2024-09-11T03:46:32.970100+0000","last_unstale":"2024-09-11T03:47:31.009904+0000","last_undegraded":"2024-09-11T03:47:31.009904+0000","last_fullsized":"2024-09-11T03:47:31.009904+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"567'384","last_scrub_stamp":"2024-09-11T03:46:33.848050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.848050+0000","objects_scrubbed":0,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:34.854051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017134979999999999,"stat_sum":{"num_bytes":3227109,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":14955,"num_write":10,"num_write_kb":6104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"2.22","version":"531'216","reported_seq":2957,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-11T03:47:32.187969+0000","last_change":"2024-09-11T03:47:32.188321+0000","last_active":"2024-09-11T03:47:32.187969+0000","last_peered":"2024-09-11T03:47:32.187969+0000","last_clean":"2024-09-11T03:47:32.187969+0000","last_became_active":"2024-09-11T03:46:32.970633+0000","last_became_peered":"2024-09-11T03:46:32.970633+0000","last_unstale":"2024-09-11T03:47:32.187969+0000","last_undegraded":"2024-09-11T03:47:32.187969+0000","last_fullsized":"2024-09-11T03:47:32.187969+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:46:33.448172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.448172+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:50.134304+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012691239999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"107","length":"3"},{"start":"10b","length":"1"}]},{"pgid":"2.21","version":"625'264","reported_seq":3087,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.982907+0000","last_change":"2024-09-11T03:47:31.983723+0000","last_active":"2024-09-11T03:47:31.982907+0000","last_peered":"2024-09-11T03:47:31.982907+0000","last_clean":"2024-09-11T03:47:31.982907+0000","last_became_active":"2024-09-11T03:46:36.024928+0000","last_became_peered":"2024-09-11T03:46:36.024928+0000","last_unstale":"2024-09-11T03:47:31.982907+0000","last_undegraded":"2024-09-11T03:47:31.982907+0000","last_fullsized":"2024-09-11T03:47:31.982907+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":6,"last_scrub":"578'239","last_scrub_stamp":"2024-09-11T03:46:36.051463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:36.051463+0000","objects_scrubbed":2,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:46.312133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001442249,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":12,"num_write_kb":2819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,8],"acting":[11,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"3"},{"start":"10b","length":"1"}]},{"pgid":"2.20","version":"630'213","reported_seq":3086,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.010972+0000","last_change":"2024-09-11T03:47:31.011600+0000","last_active":"2024-09-11T03:47:31.010972+0000","last_peered":"2024-09-11T03:47:31.010972+0000","last_clean":"2024-09-11T03:47:31.010972+0000","last_became_active":"2024-09-11T03:46:32.974375+0000","last_became_peered":"2024-09-11T03:46:32.974375+0000","last_unstale":"2024-09-11T03:47:31.010972+0000","last_undegraded":"2024-09-11T03:47:31.010972+0000","last_fullsized":"2024-09-11T03:47:31.010972+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:46:33.551541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.551541+0000","objects_scrubbed":2,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:51.060436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001272556,"stat_sum":{"num_bytes":3496021,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":15112,"num_write":7,"num_write_kb":1493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"2.1c","version":"630'687","reported_seq":3413,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.012310+0000","last_change":"2024-09-11T03:47:31.012579+0000","last_active":"2024-09-11T03:47:31.012310+0000","last_peered":"2024-09-11T03:47:31.012310+0000","last_clean":"2024-09-11T03:47:31.012310+0000","last_became_active":"2024-09-11T03:46:32.971036+0000","last_became_peered":"2024-09-11T03:46:32.971036+0000","last_unstale":"2024-09-11T03:47:31.012310+0000","last_undegraded":"2024-09-11T03:47:31.012310+0000","last_fullsized":"2024-09-11T03:47:31.012310+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"568'667","last_scrub_stamp":"2024-09-11T03:46:33.141943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.141943+0000","objects_scrubbed":1,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:40.640221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057097500000000004,"stat_sum":{"num_bytes":4700325,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10464,"num_write":10,"num_write_kb":2762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":321,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:30.997342+0000","last_change":"2024-09-11T03:47:26.195120+0000","last_active":"2024-09-11T03:47:30.997342+0000","last_peered":"2024-09-11T03:47:30.997342+0000","last_clean":"2024-09-11T03:47:30.997342+0000","last_became_active":"2024-09-11T03:46:15.974786+0000","last_became_peered":"2024-09-11T03:46:15.974786+0000","last_unstale":"2024-09-11T03:47:30.997342+0000","last_undegraded":"2024-09-11T03:47:30.997342+0000","last_fullsized":"2024-09-11T03:47:30.997342+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:26.195070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:26.195070+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:44.774748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.f","version":"557'465","reported_seq":3222,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.000828+0000","last_change":"2024-09-11T03:47:31.000828+0000","last_active":"2024-09-11T03:47:31.000828+0000","last_peered":"2024-09-11T03:47:31.000828+0000","last_clean":"2024-09-11T03:47:31.000828+0000","last_became_active":"2024-09-11T03:46:27.971779+0000","last_became_peered":"2024-09-11T03:46:27.971779+0000","last_unstale":"2024-09-11T03:47:31.000828+0000","last_undegraded":"2024-09-11T03:47:31.000828+0000","last_fullsized":"2024-09-11T03:47:31.000828+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:46:32.852908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:32.852908+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:36.077220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075790500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"2.e","version":"627'886","reported_seq":3579,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-11T03:47:28.292696+0000","last_change":"2024-09-11T03:47:28.293331+0000","last_active":"2024-09-11T03:47:28.292696+0000","last_peered":"2024-09-11T03:47:28.292696+0000","last_clean":"2024-09-11T03:47:28.292696+0000","last_became_active":"2024-09-11T03:46:36.024843+0000","last_became_peered":"2024-09-11T03:46:36.024843+0000","last_unstale":"2024-09-11T03:47:28.292696+0000","last_undegraded":"2024-09-11T03:47:28.292696+0000","last_fullsized":"2024-09-11T03:47:28.292696+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'850","last_scrub_stamp":"2024-09-11T03:46:36.567100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:36.567100+0000","objects_scrubbed":3,"log_size":886,"log_dups_size":0,"ondisk_log_size":886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:03.554384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018846049999999999,"stat_sum":{"num_bytes":6118078,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":27,"num_write_kb":12826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"105","length":"4"}]},{"pgid":"1.d","version":"20'92","reported_seq":889,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.010774+0000","last_change":"2024-09-11T03:42:51.733829+0000","last_active":"2024-09-11T03:47:31.010774+0000","last_peered":"2024-09-11T03:47:31.010774+0000","last_clean":"2024-09-11T03:47:31.010774+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:47:31.010774+0000","last_undegraded":"2024-09-11T03:47:31.010774+0000","last_fullsized":"2024-09-11T03:47:31.010774+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:51.733762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:51.733762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:11.597940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"630'791","reported_seq":3550,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.156750+0000","last_change":"2024-09-11T03:47:31.157091+0000","last_active":"2024-09-11T03:47:31.156750+0000","last_peered":"2024-09-11T03:47:31.156750+0000","last_clean":"2024-09-11T03:47:31.156750+0000","last_became_active":"2024-09-11T03:46:27.969385+0000","last_became_peered":"2024-09-11T03:46:27.969385+0000","last_unstale":"2024-09-11T03:47:31.156750+0000","last_undegraded":"2024-09-11T03:47:31.156750+0000","last_fullsized":"2024-09-11T03:47:31.156750+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'757","last_scrub_stamp":"2024-09-11T03:46:31.645504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:31.645504+0000","objects_scrubbed":5,"log_size":791,"log_dups_size":0,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:34.906169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00081825699999999997,"stat_sum":{"num_bytes":11631104,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":99,"num_read_kb":116232,"num_write":20,"num_write_kb":7250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":604,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.010884+0000","last_change":"2024-09-11T03:45:53.637950+0000","last_active":"2024-09-11T03:47:31.010884+0000","last_peered":"2024-09-11T03:47:31.010884+0000","last_clean":"2024-09-11T03:47:31.010884+0000","last_became_active":"2024-09-11T03:44:43.940022+0000","last_became_peered":"2024-09-11T03:44:43.940022+0000","last_unstale":"2024-09-11T03:47:31.010884+0000","last_undegraded":"2024-09-11T03:47:31.010884+0000","last_fullsized":"2024-09-11T03:47:31.010884+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:53.637891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:53.637891+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:14.519768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"611'706","reported_seq":3463,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.984580+0000","last_change":"2024-09-11T03:47:31.984580+0000","last_active":"2024-09-11T03:47:31.984580+0000","last_peered":"2024-09-11T03:47:31.984580+0000","last_clean":"2024-09-11T03:47:31.984580+0000","last_became_active":"2024-09-11T03:46:27.975493+0000","last_became_peered":"2024-09-11T03:46:27.975493+0000","last_unstale":"2024-09-11T03:47:31.984580+0000","last_undegraded":"2024-09-11T03:47:31.984580+0000","last_fullsized":"2024-09-11T03:47:31.984580+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"571'670","last_scrub_stamp":"2024-09-11T03:46:32.062033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:32.062033+0000","objects_scrubbed":4,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:52.397326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001775733,"stat_sum":{"num_bytes":9513257,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":8821,"num_write":26,"num_write_kb":7012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"3"},{"start":"10b","length":"1"}]},{"pgid":"1.f","version":"20'92","reported_seq":172,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-11T03:47:32.186550+0000","last_change":"2024-09-11T03:46:16.476244+0000","last_active":"2024-09-11T03:47:32.186550+0000","last_peered":"2024-09-11T03:47:32.186550+0000","last_clean":"2024-09-11T03:47:32.186550+0000","last_became_active":"2024-09-11T03:46:15.973796+0000","last_became_peered":"2024-09-11T03:46:15.973796+0000","last_unstale":"2024-09-11T03:47:32.186550+0000","last_undegraded":"2024-09-11T03:47:32.186550+0000","last_fullsized":"2024-09-11T03:47:32.186550+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:16.476181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:16.476181+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:43.297927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":1302,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.010945+0000","last_change":"2024-09-11T03:47:28.495899+0000","last_active":"2024-09-11T03:47:31.010945+0000","last_peered":"2024-09-11T03:47:31.010945+0000","last_clean":"2024-09-11T03:47:31.010945+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:47:31.010945+0000","last_undegraded":"2024-09-11T03:47:31.010945+0000","last_fullsized":"2024-09-11T03:47:31.010945+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:28.495839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:28.495839+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:37.846037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"612'414","reported_seq":3169,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.156708+0000","last_change":"2024-09-11T03:47:31.157041+0000","last_active":"2024-09-11T03:47:31.156708+0000","last_peered":"2024-09-11T03:47:31.156708+0000","last_clean":"2024-09-11T03:47:31.156708+0000","last_became_active":"2024-09-11T03:46:27.972329+0000","last_became_peered":"2024-09-11T03:46:27.972329+0000","last_unstale":"2024-09-11T03:47:31.156708+0000","last_undegraded":"2024-09-11T03:47:31.156708+0000","last_fullsized":"2024-09-11T03:47:31.156708+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'386","last_scrub_stamp":"2024-09-11T03:46:33.722283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.722283+0000","objects_scrubbed":3,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:38.289898+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00081017999999999997,"stat_sum":{"num_bytes":2235463,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":3868,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":906,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.008859+0000","last_change":"2024-09-11T03:47:25.344219+0000","last_active":"2024-09-11T03:47:31.008859+0000","last_peered":"2024-09-11T03:47:31.008859+0000","last_clean":"2024-09-11T03:47:31.008859+0000","last_became_active":"2024-09-11T03:46:15.968960+0000","last_became_peered":"2024-09-11T03:46:15.968960+0000","last_unstale":"2024-09-11T03:47:31.008859+0000","last_undegraded":"2024-09-11T03:47:31.008859+0000","last_fullsized":"2024-09-11T03:47:31.008859+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:25.344170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:25.344170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:39.221544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"563'192","reported_seq":3013,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.011683+0000","last_change":"2024-09-11T03:47:31.012190+0000","last_active":"2024-09-11T03:47:31.011683+0000","last_peered":"2024-09-11T03:47:31.011683+0000","last_clean":"2024-09-11T03:47:31.011683+0000","last_became_active":"2024-09-11T03:46:27.974405+0000","last_became_peered":"2024-09-11T03:46:27.974405+0000","last_unstale":"2024-09-11T03:47:31.011683+0000","last_undegraded":"2024-09-11T03:47:31.011683+0000","last_fullsized":"2024-09-11T03:47:31.011683+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:46:31.577153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:31.577153+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:44.300023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011992050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":859,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:30.997214+0000","last_change":"2024-09-11T03:46:41.056685+0000","last_active":"2024-09-11T03:47:30.997214+0000","last_peered":"2024-09-11T03:47:30.997214+0000","last_clean":"2024-09-11T03:47:30.997214+0000","last_became_active":"2024-09-11T03:44:25.934580+0000","last_became_peered":"2024-09-11T03:44:25.934580+0000","last_unstale":"2024-09-11T03:47:30.997214+0000","last_undegraded":"2024-09-11T03:47:30.997214+0000","last_fullsized":"2024-09-11T03:47:30.997214+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:23.494418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:23.494418+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:10.279429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"567'235","reported_seq":2947,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:30.997608+0000","last_change":"2024-09-11T03:47:30.997939+0000","last_active":"2024-09-11T03:47:30.997608+0000","last_peered":"2024-09-11T03:47:30.997608+0000","last_clean":"2024-09-11T03:47:30.997608+0000","last_became_active":"2024-09-11T03:46:36.024925+0000","last_became_peered":"2024-09-11T03:46:36.024925+0000","last_unstale":"2024-09-11T03:47:30.997608+0000","last_undegraded":"2024-09-11T03:47:30.997608+0000","last_fullsized":"2024-09-11T03:47:30.997608+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:46:36.189297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:36.189297+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:05.586707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056960599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":464,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.011821+0000","last_change":"2024-09-11T03:45:26.373529+0000","last_active":"2024-09-11T03:47:31.011821+0000","last_peered":"2024-09-11T03:47:31.011821+0000","last_clean":"2024-09-11T03:47:31.011821+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:47:31.011821+0000","last_undegraded":"2024-09-11T03:47:31.011821+0000","last_fullsized":"2024-09-11T03:47:31.011821+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:26.373478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:26.373478+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"585'218","reported_seq":3048,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.156371+0000","last_change":"2024-09-11T03:47:31.156640+0000","last_active":"2024-09-11T03:47:31.156371+0000","last_peered":"2024-09-11T03:47:31.156371+0000","last_clean":"2024-09-11T03:47:31.156371+0000","last_became_active":"2024-09-11T03:46:27.971649+0000","last_became_peered":"2024-09-11T03:46:27.971649+0000","last_unstale":"2024-09-11T03:47:31.156371+0000","last_undegraded":"2024-09-11T03:47:31.156371+0000","last_fullsized":"2024-09-11T03:47:31.156371+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:46:32.681048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:32.681048+0000","objects_scrubbed":0,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:47.764558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054380499999999996,"stat_sum":{"num_bytes":2403566,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":5188,"num_write":2,"num_write_kb":6239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":164,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-11T03:47:32.186252+0000","last_change":"2024-09-11T03:46:15.972170+0000","last_active":"2024-09-11T03:47:32.186252+0000","last_peered":"2024-09-11T03:47:32.186252+0000","last_clean":"2024-09-11T03:47:32.186252+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:47:32.186252+0000","last_undegraded":"2024-09-11T03:47:32.186252+0000","last_fullsized":"2024-09-11T03:47:32.186252+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:12.360829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:12.360829+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"633'488","reported_seq":3277,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.038899+0000","last_change":"2024-09-11T03:47:31.012695+0000","last_active":"2024-09-11T03:47:31.038899+0000","last_peered":"2024-09-11T03:47:31.038899+0000","last_clean":"2024-09-11T03:47:31.038899+0000","last_became_active":"2024-09-11T03:44:10.921135+0000","last_became_peered":"2024-09-11T03:44:10.921135+0000","last_unstale":"2024-09-11T03:47:31.038899+0000","last_undegraded":"2024-09-11T03:47:31.038899+0000","last_fullsized":"2024-09-11T03:47:31.038899+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"590'463","last_scrub_stamp":"2024-09-11T03:46:49.936885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:49.936885+0000","objects_scrubbed":3,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:57.562538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0039979760000000003,"stat_sum":{"num_bytes":6349521,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":145,"num_read_kb":139438,"num_write":101,"num_write_kb":36174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":4118225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":467,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.012080+0000","last_change":"2024-09-11T03:46:59.252925+0000","last_active":"2024-09-11T03:47:31.012080+0000","last_peered":"2024-09-11T03:47:31.012080+0000","last_clean":"2024-09-11T03:47:31.012080+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:47:31.012080+0000","last_undegraded":"2024-09-11T03:47:31.012080+0000","last_fullsized":"2024-09-11T03:47:31.012080+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:59.252877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:59.252877+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:07.389660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"630'882","reported_seq":3681,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.028643+0000","last_change":"2024-09-11T03:47:31.000711+0000","last_active":"2024-09-11T03:47:31.028643+0000","last_peered":"2024-09-11T03:47:31.028643+0000","last_clean":"2024-09-11T03:47:31.028643+0000","last_became_active":"2024-09-11T03:41:33.866316+0000","last_became_peered":"2024-09-11T03:41:33.866316+0000","last_unstale":"2024-09-11T03:47:31.028643+0000","last_undegraded":"2024-09-11T03:47:31.028643+0000","last_fullsized":"2024-09-11T03:47:31.028643+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'849","last_scrub_stamp":"2024-09-11T03:46:59.044884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:59.044884+0000","objects_scrubbed":3,"log_size":882,"log_dups_size":0,"ondisk_log_size":882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:03.407233+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.00088914500000000004,"stat_sum":{"num_bytes":17043123,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":228,"num_read_kb":245678,"num_write":162,"num_write_kb":62297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":474,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.011955+0000","last_change":"2024-09-11T03:46:17.255120+0000","last_active":"2024-09-11T03:47:31.011955+0000","last_peered":"2024-09-11T03:47:31.011955+0000","last_clean":"2024-09-11T03:47:31.011955+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:47:31.011955+0000","last_undegraded":"2024-09-11T03:47:31.011955+0000","last_fullsized":"2024-09-11T03:47:31.011955+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:17.255073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:17.255073+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:31.448759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":2894,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.010203+0000","last_change":"2024-09-11T03:47:31.010417+0000","last_active":"2024-09-11T03:47:31.010203+0000","last_peered":"2024-09-11T03:47:31.010203+0000","last_clean":"2024-09-11T03:47:31.010203+0000","last_became_active":"2024-09-11T03:41:33.865499+0000","last_became_peered":"2024-09-11T03:41:33.865499+0000","last_unstale":"2024-09-11T03:47:31.010203+0000","last_undegraded":"2024-09-11T03:47:31.010203+0000","last_fullsized":"2024-09-11T03:47:31.010203+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:47:15.201479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:15.201479+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:15.452466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001760326,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":1068,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.982110+0000","last_change":"2024-09-11T03:46:59.164595+0000","last_active":"2024-09-11T03:47:31.982110+0000","last_peered":"2024-09-11T03:47:31.982110+0000","last_clean":"2024-09-11T03:47:31.982110+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:47:31.982110+0000","last_undegraded":"2024-09-11T03:47:31.982110+0000","last_fullsized":"2024-09-11T03:47:31.982110+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:59.164529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:59.164529+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:01.974929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"623'939","reported_seq":3798,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-11T03:47:32.187477+0000","last_change":"2024-09-11T03:47:32.187822+0000","last_active":"2024-09-11T03:47:32.187477+0000","last_peered":"2024-09-11T03:47:32.187477+0000","last_clean":"2024-09-11T03:47:32.187477+0000","last_became_active":"2024-09-11T03:46:19.004997+0000","last_became_peered":"2024-09-11T03:46:19.004997+0000","last_unstale":"2024-09-11T03:47:32.187477+0000","last_undegraded":"2024-09-11T03:47:32.187477+0000","last_fullsized":"2024-09-11T03:47:32.187477+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'924","last_scrub_stamp":"2024-09-11T03:46:53.434384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:53.434384+0000","objects_scrubbed":5,"log_size":939,"log_dups_size":0,"ondisk_log_size":939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:13.556133+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001511533,"stat_sum":{"num_bytes":11731946,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":344,"num_read_kb":409383,"num_write":189,"num_write_kb":70180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":25429065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"107","length":"3"},{"start":"10b","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":879,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.982330+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:47:31.982330+0000","last_peered":"2024-09-11T03:47:31.982330+0000","last_clean":"2024-09-11T03:47:31.982330+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:47:31.982330+0000","last_undegraded":"2024-09-11T03:47:31.982330+0000","last_fullsized":"2024-09-11T03:47:31.982330+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:06.223147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"610'328","reported_seq":3078,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.011009+0000","last_change":"2024-09-11T03:47:31.011621+0000","last_active":"2024-09-11T03:47:31.011009+0000","last_peered":"2024-09-11T03:47:31.011009+0000","last_clean":"2024-09-11T03:47:31.011009+0000","last_became_active":"2024-09-11T03:41:33.863875+0000","last_became_peered":"2024-09-11T03:41:33.863875+0000","last_unstale":"2024-09-11T03:47:31.011009+0000","last_undegraded":"2024-09-11T03:47:31.011009+0000","last_fullsized":"2024-09-11T03:47:31.011009+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'324","last_scrub_stamp":"2024-09-11T03:46:16.518345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:16.518345+0000","objects_scrubbed":0,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00097141800000000004,"stat_sum":{"num_bytes":3546442,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":45190,"num_write":39,"num_write_kb":11168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":1213,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-11T03:47:28.372829+0000","last_change":"2024-09-11T03:47:28.372829+0000","last_active":"2024-09-11T03:47:28.372829+0000","last_peered":"2024-09-11T03:47:28.372829+0000","last_clean":"2024-09-11T03:47:28.372829+0000","last_became_active":"2024-09-11T03:46:15.973271+0000","last_became_peered":"2024-09-11T03:46:15.973271+0000","last_unstale":"2024-09-11T03:47:28.372829+0000","last_undegraded":"2024-09-11T03:47:28.372829+0000","last_fullsized":"2024-09-11T03:47:28.372829+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:28.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:28.372745+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:58.221220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"633'929","reported_seq":3780,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.028179+0000","last_change":"2024-09-11T03:47:31.010149+0000","last_active":"2024-09-11T03:47:31.028179+0000","last_peered":"2024-09-11T03:47:31.028179+0000","last_clean":"2024-09-11T03:47:31.028179+0000","last_became_active":"2024-09-11T03:41:33.865638+0000","last_became_peered":"2024-09-11T03:41:33.865638+0000","last_unstale":"2024-09-11T03:47:31.028179+0000","last_undegraded":"2024-09-11T03:47:31.028179+0000","last_fullsized":"2024-09-11T03:47:31.028179+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'865","last_scrub_stamp":"2024-09-11T03:46:34.843368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:34.843368+0000","objects_scrubbed":7,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:46.887998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001623673,"stat_sum":{"num_bytes":9849661,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":349,"num_read_kb":371755,"num_write":198,"num_write_kb":68285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":1290,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.010280+0000","last_change":"2024-09-11T03:47:22.529885+0000","last_active":"2024-09-11T03:47:31.010280+0000","last_peered":"2024-09-11T03:47:31.010280+0000","last_clean":"2024-09-11T03:47:31.010280+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:47:31.010280+0000","last_undegraded":"2024-09-11T03:47:31.010280+0000","last_fullsized":"2024-09-11T03:47:31.010280+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:22.529805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:22.529805+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:34.199617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"614'308","reported_seq":3072,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.982540+0000","last_change":"2024-09-11T03:47:31.983620+0000","last_active":"2024-09-11T03:47:31.982540+0000","last_peered":"2024-09-11T03:47:31.982540+0000","last_clean":"2024-09-11T03:47:31.982540+0000","last_became_active":"2024-09-11T03:46:27.968719+0000","last_became_peered":"2024-09-11T03:46:27.968719+0000","last_unstale":"2024-09-11T03:47:31.982540+0000","last_undegraded":"2024-09-11T03:47:31.982540+0000","last_fullsized":"2024-09-11T03:47:31.982540+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'279","last_scrub_stamp":"2024-09-11T03:46:31.085715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:31.085715+0000","objects_scrubbed":2,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:40.825907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001415735,"stat_sum":{"num_bytes":3529635,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":10,"num_write_kb":6055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"3"},{"start":"10b","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":952,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.156131+0000","last_change":"2024-09-11T03:46:20.861683+0000","last_active":"2024-09-11T03:47:31.156131+0000","last_peered":"2024-09-11T03:47:31.156131+0000","last_clean":"2024-09-11T03:47:31.156131+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:47:31.156131+0000","last_undegraded":"2024-09-11T03:47:31.156131+0000","last_fullsized":"2024-09-11T03:47:31.156131+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:20.861635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:20.861635+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"568'646","reported_seq":3395,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.000387+0000","last_change":"2024-09-11T03:47:31.000387+0000","last_active":"2024-09-11T03:47:31.000387+0000","last_peered":"2024-09-11T03:47:31.000387+0000","last_clean":"2024-09-11T03:47:31.000387+0000","last_became_active":"2024-09-11T03:46:27.974031+0000","last_became_peered":"2024-09-11T03:46:27.974031+0000","last_unstale":"2024-09-11T03:47:31.000387+0000","last_undegraded":"2024-09-11T03:47:31.000387+0000","last_fullsized":"2024-09-11T03:47:31.000387+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:46:31.818739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:31.818739+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:43.194613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00063404200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":894,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.156114+0000","last_change":"2024-09-11T03:46:37.719201+0000","last_active":"2024-09-11T03:47:31.156114+0000","last_peered":"2024-09-11T03:47:31.156114+0000","last_clean":"2024-09-11T03:47:31.156114+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:47:31.156114+0000","last_undegraded":"2024-09-11T03:47:31.156114+0000","last_fullsized":"2024-09-11T03:47:31.156114+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:37.719151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:37.719151+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:41.060227+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":2805,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-11T03:47:28.293393+0000","last_change":"2024-09-11T03:47:28.295502+0000","last_active":"2024-09-11T03:47:28.293393+0000","last_peered":"2024-09-11T03:47:28.293393+0000","last_clean":"2024-09-11T03:47:28.293393+0000","last_became_active":"2024-09-11T03:41:38.886834+0000","last_became_peered":"2024-09-11T03:41:38.886834+0000","last_unstale":"2024-09-11T03:47:28.293393+0000","last_undegraded":"2024-09-11T03:47:28.293393+0000","last_fullsized":"2024-09-11T03:47:28.293393+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:46:14.674306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:14.674306+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:42.540411+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0039312599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"105","length":"4"}]},{"pgid":"1.11","version":"20'92","reported_seq":877,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:30.997277+0000","last_change":"2024-09-11T03:47:00.162760+0000","last_active":"2024-09-11T03:47:30.997277+0000","last_peered":"2024-09-11T03:47:30.997277+0000","last_clean":"2024-09-11T03:47:30.997277+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:47:30.997277+0000","last_undegraded":"2024-09-11T03:47:30.997277+0000","last_fullsized":"2024-09-11T03:47:30.997277+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:00.162709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:00.162709+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:20.560991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"628'444","reported_seq":3195,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.012690+0000","last_change":"2024-09-11T03:47:31.012949+0000","last_active":"2024-09-11T03:47:31.012690+0000","last_peered":"2024-09-11T03:47:31.012690+0000","last_clean":"2024-09-11T03:47:31.012690+0000","last_became_active":"2024-09-11T03:43:55.918817+0000","last_became_peered":"2024-09-11T03:43:55.918817+0000","last_unstale":"2024-09-11T03:47:31.012690+0000","last_undegraded":"2024-09-11T03:47:31.012690+0000","last_fullsized":"2024-09-11T03:47:31.012690+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"590'420","last_scrub_stamp":"2024-09-11T03:46:50.265647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:50.265647+0000","objects_scrubbed":2,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:18.050796+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00076052800000000001,"stat_sum":{"num_bytes":7681129,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":166394,"num_write":109,"num_write_kb":39112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":15216236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":891,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.156071+0000","last_change":"2024-09-11T03:47:23.339646+0000","last_active":"2024-09-11T03:47:31.156071+0000","last_peered":"2024-09-11T03:47:31.156071+0000","last_clean":"2024-09-11T03:47:31.156071+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:47:31.156071+0000","last_undegraded":"2024-09-11T03:47:31.156071+0000","last_fullsized":"2024-09-11T03:47:31.156071+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:23.339595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:23.339595+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:46.448313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":1899,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-11T03:47:32.186573+0000","last_change":"2024-09-11T03:47:32.187115+0000","last_active":"2024-09-11T03:47:32.186573+0000","last_peered":"2024-09-11T03:47:32.186573+0000","last_clean":"2024-09-11T03:47:32.186573+0000","last_became_active":"2024-09-11T03:46:19.005117+0000","last_became_peered":"2024-09-11T03:46:19.005117+0000","last_unstale":"2024-09-11T03:47:32.186573+0000","last_undegraded":"2024-09-11T03:47:32.186573+0000","last_fullsized":"2024-09-11T03:47:32.186573+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:47:20.437167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:20.437167+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:24.416671+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001044145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"107","length":"3"},{"start":"10b","length":"1"}]},{"pgid":"1.13","version":"20'92","reported_seq":993,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.008787+0000","last_change":"2024-09-11T03:47:18.217600+0000","last_active":"2024-09-11T03:47:31.008787+0000","last_peered":"2024-09-11T03:47:31.008787+0000","last_clean":"2024-09-11T03:47:31.008787+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:47:31.008787+0000","last_undegraded":"2024-09-11T03:47:31.008787+0000","last_fullsized":"2024-09-11T03:47:31.008787+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:18.217555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:18.217555+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:40.131743+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":1832,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:30.999787+0000","last_change":"2024-09-11T03:47:31.000232+0000","last_active":"2024-09-11T03:47:30.999787+0000","last_peered":"2024-09-11T03:47:30.999787+0000","last_clean":"2024-09-11T03:47:30.999787+0000","last_became_active":"2024-09-11T03:41:35.874295+0000","last_became_peered":"2024-09-11T03:41:35.874295+0000","last_unstale":"2024-09-11T03:47:30.999787+0000","last_undegraded":"2024-09-11T03:47:30.999787+0000","last_fullsized":"2024-09-11T03:47:30.999787+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:47:18.969898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:18.969898+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:44.193518+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066497699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":458,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.010638+0000","last_change":"2024-09-11T03:46:36.567028+0000","last_active":"2024-09-11T03:47:31.010638+0000","last_peered":"2024-09-11T03:47:31.010638+0000","last_clean":"2024-09-11T03:47:31.010638+0000","last_became_active":"2024-09-11T03:46:36.026459+0000","last_became_peered":"2024-09-11T03:46:36.026459+0000","last_unstale":"2024-09-11T03:47:31.010638+0000","last_undegraded":"2024-09-11T03:47:31.010638+0000","last_fullsized":"2024-09-11T03:47:31.010638+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:36.566974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:36.566974+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:51.906555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":2395,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-11T03:47:32.187904+0000","last_change":"2024-09-11T03:47:32.188328+0000","last_active":"2024-09-11T03:47:32.187904+0000","last_peered":"2024-09-11T03:47:32.187904+0000","last_clean":"2024-09-11T03:47:32.187904+0000","last_became_active":"2024-09-11T03:46:19.003959+0000","last_became_peered":"2024-09-11T03:46:19.003959+0000","last_unstale":"2024-09-11T03:47:32.187904+0000","last_undegraded":"2024-09-11T03:47:32.187904+0000","last_fullsized":"2024-09-11T03:47:32.187904+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:46:35.381114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:35.381114+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:39.015705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001810163,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"107","length":"3"},{"start":"10b","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":852,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.010697+0000","last_change":"2024-09-11T03:46:37.557063+0000","last_active":"2024-09-11T03:47:31.010697+0000","last_peered":"2024-09-11T03:47:31.010697+0000","last_clean":"2024-09-11T03:47:31.010697+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:47:31.010697+0000","last_undegraded":"2024-09-11T03:47:31.010697+0000","last_fullsized":"2024-09-11T03:47:31.010697+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:37.557020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:37.557020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:44.193179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"609'420","reported_seq":3211,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.156319+0000","last_change":"2024-09-11T03:47:31.156657+0000","last_active":"2024-09-11T03:47:31.156319+0000","last_peered":"2024-09-11T03:47:31.156319+0000","last_clean":"2024-09-11T03:47:31.156319+0000","last_became_active":"2024-09-11T03:46:19.007019+0000","last_became_peered":"2024-09-11T03:46:19.007019+0000","last_unstale":"2024-09-11T03:47:31.156319+0000","last_undegraded":"2024-09-11T03:47:31.156319+0000","last_fullsized":"2024-09-11T03:47:31.156319+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"597'416","last_scrub_stamp":"2024-09-11T03:47:00.394934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:00.394934+0000","objects_scrubbed":0,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:10.312854+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00064636399999999997,"stat_sum":{"num_bytes":3512828,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":65,"num_read_kb":54715,"num_write":73,"num_write_kb":27957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":5263193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":327,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.982792+0000","last_change":"2024-09-11T03:46:35.041806+0000","last_active":"2024-09-11T03:47:31.982792+0000","last_peered":"2024-09-11T03:47:31.982792+0000","last_clean":"2024-09-11T03:47:31.982792+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:47:31.982792+0000","last_undegraded":"2024-09-11T03:47:31.982792+0000","last_fullsized":"2024-09-11T03:47:31.982792+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:35.041709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:35.041709+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:45.999431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":3026,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-11T03:47:28.291573+0000","last_change":"2024-09-11T03:47:28.292641+0000","last_active":"2024-09-11T03:47:28.291573+0000","last_peered":"2024-09-11T03:47:28.291573+0000","last_clean":"2024-09-11T03:47:28.291573+0000","last_became_active":"2024-09-11T03:41:38.888228+0000","last_became_peered":"2024-09-11T03:41:38.888228+0000","last_unstale":"2024-09-11T03:47:28.291573+0000","last_undegraded":"2024-09-11T03:47:28.291573+0000","last_fullsized":"2024-09-11T03:47:28.291573+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:46:37.552437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:37.552437+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:06.030551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001339678,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"105","length":"4"}]},{"pgid":"1.17","version":"20'92","reported_seq":878,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:30.999875+0000","last_change":"2024-09-11T03:47:29.086383+0000","last_active":"2024-09-11T03:47:30.999875+0000","last_peered":"2024-09-11T03:47:30.999875+0000","last_clean":"2024-09-11T03:47:30.999875+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:47:30.999875+0000","last_undegraded":"2024-09-11T03:47:30.999875+0000","last_fullsized":"2024-09-11T03:47:30.999875+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:29.086335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:29.086335+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:40.330795+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"614'276","reported_seq":2934,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-11T03:47:32.187564+0000","last_change":"2024-09-11T03:47:32.187880+0000","last_active":"2024-09-11T03:47:32.187564+0000","last_peered":"2024-09-11T03:47:32.187564+0000","last_clean":"2024-09-11T03:47:32.187564+0000","last_became_active":"2024-09-11T03:46:19.004558+0000","last_became_peered":"2024-09-11T03:46:19.004558+0000","last_unstale":"2024-09-11T03:47:32.187564+0000","last_undegraded":"2024-09-11T03:47:32.187564+0000","last_fullsized":"2024-09-11T03:47:32.187564+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"541'254","last_scrub_stamp":"2024-09-11T03:46:12.749744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:12.749744+0000","objects_scrubbed":1,"log_size":276,"log_dups_size":0,"ondisk_log_size":276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:34.416989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001422651,"stat_sum":{"num_bytes":3445600,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":8234,"num_write":36,"num_write_kb":8917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"107","length":"3"},{"start":"10b","length":"1"}]},{"pgid":"1.18","version":"20'92","reported_seq":978,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.155984+0000","last_change":"2024-09-11T03:46:17.873917+0000","last_active":"2024-09-11T03:47:31.155984+0000","last_peered":"2024-09-11T03:47:31.155984+0000","last_clean":"2024-09-11T03:47:31.155984+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:47:31.155984+0000","last_undegraded":"2024-09-11T03:47:31.155984+0000","last_fullsized":"2024-09-11T03:47:31.155984+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:17.873858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:17.873858+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:45.517409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"630'699","reported_seq":3567,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.983693+0000","last_change":"2024-09-11T03:47:31.984450+0000","last_active":"2024-09-11T03:47:31.983693+0000","last_peered":"2024-09-11T03:47:31.983693+0000","last_clean":"2024-09-11T03:47:31.983693+0000","last_became_active":"2024-09-11T03:46:32.975567+0000","last_became_peered":"2024-09-11T03:46:32.975567+0000","last_unstale":"2024-09-11T03:47:31.983693+0000","last_undegraded":"2024-09-11T03:47:31.983693+0000","last_fullsized":"2024-09-11T03:47:31.983693+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"575'648","last_scrub_stamp":"2024-09-11T03:46:33.071702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.071702+0000","objects_scrubbed":5,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:47.436513+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001913831,"stat_sum":{"num_bytes":6958395,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":45255,"num_write":11,"num_write_kb":10027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7],"acting":[11,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"3"},{"start":"10b","length":"1"}]},{"pgid":"1.19","version":"20'92","reported_seq":471,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.011738+0000","last_change":"2024-09-11T03:46:15.973211+0000","last_active":"2024-09-11T03:47:31.011738+0000","last_peered":"2024-09-11T03:47:31.011738+0000","last_clean":"2024-09-11T03:47:31.011738+0000","last_became_active":"2024-09-11T03:46:15.972841+0000","last_became_peered":"2024-09-11T03:46:15.972841+0000","last_unstale":"2024-09-11T03:47:31.011738+0000","last_undegraded":"2024-09-11T03:47:31.011738+0000","last_fullsized":"2024-09-11T03:47:31.011738+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:30.424428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:30.424428+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:43.252639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"557'273","reported_seq":2969,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-11T03:47:32.188387+0000","last_change":"2024-09-11T03:47:32.188674+0000","last_active":"2024-09-11T03:47:32.188387+0000","last_peered":"2024-09-11T03:47:32.188387+0000","last_clean":"2024-09-11T03:47:32.188387+0000","last_became_active":"2024-09-11T03:46:32.971003+0000","last_became_peered":"2024-09-11T03:46:32.971003+0000","last_unstale":"2024-09-11T03:47:32.188387+0000","last_undegraded":"2024-09-11T03:47:32.188387+0000","last_fullsized":"2024-09-11T03:47:32.188387+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:46:34.407744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:34.407744+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:59.771789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013699039999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,2],"acting":[6,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"107","length":"3"},{"start":"10b","length":"1"}]},{"pgid":"1.1a","version":"20'92","reported_seq":605,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.008427+0000","last_change":"2024-09-11T03:45:52.914136+0000","last_active":"2024-09-11T03:47:31.008427+0000","last_peered":"2024-09-11T03:47:31.008427+0000","last_clean":"2024-09-11T03:47:31.008427+0000","last_became_active":"2024-09-11T03:44:43.938873+0000","last_became_peered":"2024-09-11T03:44:43.938873+0000","last_unstale":"2024-09-11T03:47:31.008427+0000","last_undegraded":"2024-09-11T03:47:31.008427+0000","last_fullsized":"2024-09-11T03:47:31.008427+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:52.914088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:52.914088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:11.184197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"625'489","reported_seq":3316,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-11T03:47:32.186817+0000","last_change":"2024-09-11T03:47:32.187266+0000","last_active":"2024-09-11T03:47:32.186817+0000","last_peered":"2024-09-11T03:47:32.186817+0000","last_clean":"2024-09-11T03:47:32.186817+0000","last_became_active":"2024-09-11T03:46:19.002002+0000","last_became_peered":"2024-09-11T03:46:19.002002+0000","last_unstale":"2024-09-11T03:47:32.186817+0000","last_undegraded":"2024-09-11T03:47:32.186817+0000","last_fullsized":"2024-09-11T03:47:32.186817+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"594'474","last_scrub_stamp":"2024-09-11T03:46:52.412086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:52.412086+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:21.875344+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001175274,"stat_sum":{"num_bytes":2235463,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":58,"num_read_kb":59019,"num_write":39,"num_write_kb":20159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4870845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"107","length":"3"},{"start":"10b","length":"1"}]},{"pgid":"1.1b","version":"20'92","reported_seq":847,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:30.999496+0000","last_change":"2024-09-11T03:47:25.035842+0000","last_active":"2024-09-11T03:47:30.999496+0000","last_peered":"2024-09-11T03:47:30.999496+0000","last_clean":"2024-09-11T03:47:30.999496+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:47:30.999496+0000","last_undegraded":"2024-09-11T03:47:30.999496+0000","last_fullsized":"2024-09-11T03:47:30.999496+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:25.035793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:25.035793+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:30.302122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"630'384","reported_seq":3177,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-09-11T03:47:28.293967+0000","last_change":"2024-09-11T03:47:28.294810+0000","last_active":"2024-09-11T03:47:28.293967+0000","last_peered":"2024-09-11T03:47:28.293967+0000","last_clean":"2024-09-11T03:47:28.293967+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:47:28.293967+0000","last_undegraded":"2024-09-11T03:47:28.293967+0000","last_fullsized":"2024-09-11T03:47:28.293967+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"547'357","last_scrub_stamp":"2024-09-11T03:46:19.692855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:19.692855+0000","objects_scrubbed":1,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0036585070000000001,"stat_sum":{"num_bytes":7849199,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":133382,"num_write":66,"num_write_kb":25897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4865740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"105","length":"4"}]},{"pgid":"2.1f","version":"615'510","reported_seq":3213,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:30.997756+0000","last_change":"2024-09-11T03:47:30.997928+0000","last_active":"2024-09-11T03:47:30.997756+0000","last_peered":"2024-09-11T03:47:30.997756+0000","last_clean":"2024-09-11T03:47:30.997756+0000","last_became_active":"2024-09-11T03:46:32.968696+0000","last_became_peered":"2024-09-11T03:46:32.968696+0000","last_unstale":"2024-09-11T03:47:30.997756+0000","last_undegraded":"2024-09-11T03:47:30.997756+0000","last_fullsized":"2024-09-11T03:47:30.997756+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:46:33.227651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.227651+0000","objects_scrubbed":3,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:41.360402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038689600000000002,"stat_sum":{"num_bytes":6891200,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":47281,"num_write":28,"num_write_kb":9969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.1c","version":"20'92","reported_seq":854,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:30.999553+0000","last_change":"2024-09-11T03:46:20.752592+0000","last_active":"2024-09-11T03:47:30.999553+0000","last_peered":"2024-09-11T03:47:30.999553+0000","last_clean":"2024-09-11T03:47:30.999553+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:47:30.999553+0000","last_undegraded":"2024-09-11T03:47:30.999553+0000","last_fullsized":"2024-09-11T03:47:30.999553+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:20.752526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:20.752526+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:32.553208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1e","version":"630'915","reported_seq":3619,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:30.997977+0000","last_change":"2024-09-11T03:47:30.998219+0000","last_active":"2024-09-11T03:47:30.997977+0000","last_peered":"2024-09-11T03:47:30.997977+0000","last_clean":"2024-09-11T03:47:30.997977+0000","last_became_active":"2024-09-11T03:46:32.970637+0000","last_became_peered":"2024-09-11T03:46:32.970637+0000","last_unstale":"2024-09-11T03:47:30.997977+0000","last_undegraded":"2024-09-11T03:47:30.997977+0000","last_fullsized":"2024-09-11T03:47:30.997977+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"575'848","last_scrub_stamp":"2024-09-11T03:46:34.276017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:34.276017+0000","objects_scrubbed":4,"log_size":915,"log_dups_size":0,"ondisk_log_size":915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:45.934784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00067380599999999999,"stat_sum":{"num_bytes":3630477,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":34,"num_write_kb":15522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.1d","version":"20'92","reported_seq":846,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.008587+0000","last_change":"2024-09-11T03:47:22.312507+0000","last_active":"2024-09-11T03:47:31.008587+0000","last_peered":"2024-09-11T03:47:31.008587+0000","last_clean":"2024-09-11T03:47:31.008587+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:47:31.008587+0000","last_undegraded":"2024-09-11T03:47:31.008587+0000","last_fullsized":"2024-09-11T03:47:31.008587+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:22.312456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:22.312456+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:44.701057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"567'752","reported_seq":3456,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.012240+0000","last_change":"2024-09-11T03:47:31.012537+0000","last_active":"2024-09-11T03:47:31.012240+0000","last_peered":"2024-09-11T03:47:31.012240+0000","last_clean":"2024-09-11T03:47:31.012240+0000","last_became_active":"2024-09-11T03:46:32.972371+0000","last_became_peered":"2024-09-11T03:46:32.972371+0000","last_unstale":"2024-09-11T03:47:31.012240+0000","last_undegraded":"2024-09-11T03:47:31.012240+0000","last_fullsized":"2024-09-11T03:47:31.012240+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:46:34.138345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:34.138345+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:59.036827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000598867,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,11],"acting":[10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"105","length":"4"},{"start":"10b","length":"1"}]},{"pgid":"1.1e","version":"20'92","reported_seq":859,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-11T03:47:31.982928+0000","last_change":"2024-09-11T03:47:00.206764+0000","last_active":"2024-09-11T03:47:31.982928+0000","last_peered":"2024-09-11T03:47:31.982928+0000","last_clean":"2024-09-11T03:47:31.982928+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:47:31.982928+0000","last_undegraded":"2024-09-11T03:47:31.982928+0000","last_fullsized":"2024-09-11T03:47:31.982928+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:00.206720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:00.206720+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:28.907706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":36,"stat_sum":{"num_bytes":137579542,"num_objects":46,"num_object_clones":3,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":1,"num_read":1810,"num_read_kb":1900379,"num_write":1213,"num_write_kb":458224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":59763304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":246857728,"data_stored":248910746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16405,"ondisk_log_size":16405,"up":108,"acting":108,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285692,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61988,"kb_used_data":22480,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93683164,"statfs":{"total":95995035648,"available":95931559936,"internally_reserved":0,"allocated":23019520,"data_stored":20244988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514624,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62376,"kb_used_data":27020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93682776,"statfs":{"total":95995035648,"available":95931162624,"internally_reserved":0,"allocated":27668480,"data_stored":25368904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743557,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":59400,"kb_used_data":28212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93685752,"statfs":{"total":95995035648,"available":95934210048,"internally_reserved":0,"allocated":28889088,"data_stored":26086526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037901,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54432,"kb_used_data":23240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93690720,"statfs":{"total":95995035648,"available":95939297280,"internally_reserved":0,"allocated":23797760,"data_stored":21177731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299536,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92736,"kb_used_data":39336,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93652416,"statfs":{"total":95995035648,"available":95900073984,"internally_reserved":0,"allocated":40280064,"data_stored":39628372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528469,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60516,"kb_used_data":25164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93684636,"statfs":{"total":95995035648,"available":95933067264,"internally_reserved":0,"allocated":25767936,"data_stored":22914790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757403,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62152,"kb_used_data":22644,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93683000,"statfs":{"total":95995035648,"available":95931392000,"internally_reserved":0,"allocated":23187456,"data_stored":20340158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986336,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58280,"kb_used_data":27092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93686872,"statfs":{"total":95995035648,"available":95935356928,"internally_reserved":0,"allocated":27742208,"data_stored":24905024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92096,"kb_used_data":38692,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93653056,"statfs":{"total":95995035648,"available":95900729344,"internally_reserved":0,"allocated":39620608,"data_stored":36710493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411498,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45488,"kb_used_data":18460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93699664,"statfs":{"total":95995035648,"available":95948455936,"internally_reserved":0,"allocated":18903040,"data_stored":16124297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575024,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57812,"kb_used_data":26620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93687340,"statfs":{"total":95995035648,"available":95935836160,"internally_reserved":0,"allocated":27258880,"data_stored":24787167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738549,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65796,"kb_used_data":34608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93679356,"statfs":{"total":95995035648,"available":95927660544,"internally_reserved":0,"allocated":35438592,"data_stored":32904407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29319168,"data_stored":29489062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19619840,"data_stored":19856173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11108352,"data_stored":11025029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27189248,"data_stored":26961189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20127744,"data_stored":19990550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":15343616,"data_stored":15207916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":18857984,"data_stored":18705805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32436224,"data_stored":34496130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16158720,"data_stored":16246737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":21286912,"data_stored":21180251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":20029440,"data_stored":20437910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":15380480,"data_stored":15313994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:47:35.226 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-11T03:47:35.226 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:47:35.227 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-11T03:47:35.227 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T03:47:35.227 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T03:47:35.227 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-11T03:47:35.254 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:35.265 INFO:tasks.rados.rados.2.smithi175.stdout:2448: finishing write tid 1 to smithi17557870-43 2024-09-11T03:47:35.265 INFO:tasks.rados.rados.2.smithi175.stdout:2448: finishing write tid 2 to smithi17557870-43 2024-09-11T03:47:35.265 INFO:tasks.rados.rados.2.smithi175.stdout:2448: finishing write tid 3 to smithi17557870-43 2024-09-11T03:47:35.265 INFO:tasks.rados.rados.2.smithi175.stdout:2448: finishing write tid 4 to smithi17557870-43 2024-09-11T03:47:35.265 INFO:tasks.rados.rados.2.smithi175.stdout:2448: oid 43 updating version 0 to 447 2024-09-11T03:47:35.265 INFO:tasks.rados.rados.2.smithi175.stdout:2448: oid 43 updating version 447 to 448 2024-09-11T03:47:35.265 INFO:tasks.rados.rados.2.smithi175.stdout:2448: oid 43 version 448 is already newer than 446 2024-09-11T03:47:35.265 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 448 (ObjNum 848 snap 267 seq_num 848) dirty exists 2024-09-11T03:47:35.265 INFO:tasks.rados.rados.2.smithi175.stdout:2448: left oid 43 (ObjNum 848 snap 267 seq_num 848) 2024-09-11T03:47:35.265 INFO:tasks.rados.rados.2.smithi175.stdout:2444: done (8 left) 2024-09-11T03:47:35.266 INFO:tasks.rados.rados.2.smithi175.stdout:2446: done (7 left) 2024-09-11T03:47:35.266 INFO:tasks.rados.rados.2.smithi175.stdout:2447: done (6 left) 2024-09-11T03:47:35.266 INFO:tasks.rados.rados.2.smithi175.stdout:2448: done (5 left) 2024-09-11T03:47:35.266 INFO:tasks.rados.rados.2.smithi175.stdout:2457: done (4 left) 2024-09-11T03:47:35.266 INFO:tasks.rados.rados.2.smithi175.stdout:2458: read oid 14 snap -1 2024-09-11T03:47:35.266 INFO:tasks.rados.rados.2.smithi175.stdout:2458: expect deleted 2024-09-11T03:47:35.266 INFO:tasks.rados.rados.2.smithi175.stdout:2459: write (excl) oid 12 current snap is 267 2024-09-11T03:47:35.266 INFO:tasks.rados.rados.2.smithi175.stdout:2450: expect (ObjNum 791 snap 252 seq_num 791) 2024-09-11T03:47:35.277 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:47:35.319 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:35.341 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:35.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:35.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:35.369 INFO:tasks.rados.rados.2.smithi175.stdout:2452: finishing write tid 1 to smithi17557870-50 2024-09-11T03:47:35.370 INFO:tasks.rados.rados.2.smithi175.stdout:2452: finishing write tid 2 to smithi17557870-50 2024-09-11T03:47:35.370 INFO:tasks.rados.rados.2.smithi175.stdout:2452: finishing write tid 3 to smithi17557870-50 2024-09-11T03:47:35.370 INFO:tasks.rados.rados.2.smithi175.stdout:2452: finishing write tid 4 to smithi17557870-50 2024-09-11T03:47:35.370 INFO:tasks.rados.rados.2.smithi175.stdout:2452: oid 50 updating version 0 to 931 2024-09-11T03:47:35.370 INFO:tasks.rados.rados.2.smithi175.stdout:2452: oid 50 updating version 931 to 933 2024-09-11T03:47:35.371 INFO:tasks.rados.rados.2.smithi175.stdout:2452: oid 50 version 933 is already newer than 932 2024-09-11T03:47:35.371 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 933 (ObjNum 849 snap 267 seq_num 849) dirty exists 2024-09-11T03:47:35.371 INFO:tasks.rados.rados.2.smithi175.stdout:2452: left oid 50 (ObjNum 849 snap 267 seq_num 849) 2024-09-11T03:47:35.371 INFO:tasks.rados.rados.2.smithi175.stdout:2456: expect (ObjNum 736 snap 243 seq_num 736) 2024-09-11T03:47:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:35 smithi121 ceph-mon[30729]: osdmap e636: 12 total, 12 up, 12 in 2024-09-11T03:47:35.442 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:35.447 INFO:tasks.rados.rados.2.smithi175.stdout:2455: expect (ObjNum 708 snap 232 seq_num 708) 2024-09-11T03:47:35.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:35.464 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:35.464 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:35.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:35 smithi175 ceph-mon[30894]: osdmap e636: 12 total, 12 up, 12 in 2024-09-11T03:47:35.481 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:47:35.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:35.587 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:35.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:35.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:35.680 INFO:tasks.rados.rados.2.smithi175.stdout:2459: seq_num 851 ranges {702757=552082,1681941=430628} 2024-09-11T03:47:35.686 INFO:tasks.rados.rados.2.smithi175.stdout:2459: writing smithi17557870-12 from 702757 to 1254839 tid 1 2024-09-11T03:47:35.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:35.692 INFO:tasks.rados.rados.2.smithi175.stdout:2459: writing smithi17557870-12 from 1681941 to 2112569 tid 2 2024-09-11T03:47:35.692 INFO:tasks.rados.rados.2.smithi175.stdout:2450: done (5 left) 2024-09-11T03:47:35.692 INFO:tasks.rados.rados.2.smithi175.stdout:2452: done (4 left) 2024-09-11T03:47:35.692 INFO:tasks.rados.rados.2.smithi175.stdout:2455: done (3 left) 2024-09-11T03:47:35.692 INFO:tasks.rados.rados.2.smithi175.stdout:2456: done (2 left) 2024-09-11T03:47:35.692 INFO:tasks.rados.rados.2.smithi175.stdout:2458: done (1 left) 2024-09-11T03:47:35.692 INFO:tasks.rados.rados.2.smithi175.stdout:2460: read oid 4 snap -1 2024-09-11T03:47:35.692 INFO:tasks.rados.rados.2.smithi175.stdout:2460: expect (ObjNum 807 snap 255 seq_num 807) 2024-09-11T03:47:35.692 INFO:tasks.rados.rados.2.smithi175.stdout:2461: delete oid 28 current snap is 267 2024-09-11T03:47:35.692 INFO:tasks.rados.rados.2.smithi175.stdout:2459: finishing write tid 1 to smithi17557870-12 2024-09-11T03:47:35.694 INFO:tasks.rados.rados.2.smithi175.stdout:2461: done (2 left) 2024-09-11T03:47:35.694 INFO:tasks.rados.rados.2.smithi175.stdout:2462: rollback oid 14 current snap is 267 2024-09-11T03:47:35.694 INFO:tasks.rados.rados.2.smithi175.stdout:2462: done (2 left) 2024-09-11T03:47:35.694 INFO:tasks.rados.rados.2.smithi175.stdout:2463: write oid 7 current snap is 267 2024-09-11T03:47:35.694 INFO:tasks.rados.rados.2.smithi175.stdout:2463: seq_num 853 ranges {736371=418933,1798905=434297} 2024-09-11T03:47:35.699 INFO:tasks.rados.rados.2.smithi175.stdout:2463: writing smithi17557870-7 from 736371 to 1155304 tid 1 2024-09-11T03:47:35.706 INFO:tasks.rados.rados.2.smithi175.stdout:2463: writing smithi17557870-7 from 1798905 to 2233202 tid 2 2024-09-11T03:47:35.706 INFO:tasks.rados.rados.2.smithi175.stdout:2464: read oid 14 snap -1 2024-09-11T03:47:35.706 INFO:tasks.rados.rados.2.smithi175.stdout:2464: expect deleted 2024-09-11T03:47:35.706 INFO:tasks.rados.rados.2.smithi175.stdout:2465: copy_from oid 30 from oid 15 current snap is 267 2024-09-11T03:47:35.706 INFO:tasks.rados.rados.2.smithi175.stdout:2459: finishing write tid 2 to smithi17557870-12 2024-09-11T03:47:35.707 INFO:tasks.rados.rados.2.smithi175.stdout:2459: finishing write tid 3 to smithi17557870-12 2024-09-11T03:47:35.707 INFO:tasks.rados.rados.2.smithi175.stdout:2459: finishing write tid 4 to smithi17557870-12 2024-09-11T03:47:35.707 INFO:tasks.rados.rados.2.smithi175.stdout:2459: oid 12 updating version 0 to 895 2024-09-11T03:47:35.707 INFO:tasks.rados.rados.2.smithi175.stdout:2459: oid 12 version 895 is already newer than 893 2024-09-11T03:47:35.707 INFO:tasks.rados.rados.2.smithi175.stdout:2459: oid 12 version 895 is already newer than 894 2024-09-11T03:47:35.707 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 895 (ObjNum 851 snap 267 seq_num 851) dirty exists 2024-09-11T03:47:35.707 INFO:tasks.rados.rados.2.smithi175.stdout:2459: left oid 12 (ObjNum 851 snap 267 seq_num 851) 2024-09-11T03:47:35.707 INFO:tasks.rados.rados.2.smithi175.stdout:2463: finishing write tid 1 to smithi17557870-7 2024-09-11T03:47:35.707 INFO:tasks.rados.rados.2.smithi175.stdout:2460: expect (ObjNum 807 snap 255 seq_num 807) 2024-09-11T03:47:35.712 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:35.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:35.812 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:47:35.812 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:47:35.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:47:35.812 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:47:35.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.812 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:640 2.a e131/95 2:5014e305:::smithi17557870-46:head v 131'5, mlcod=100'2)", 2024-09-11T03:47:35.812 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:47:35.812 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 507.69277368399997, 2024-09-11T03:47:35.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010809757999999999, 2024-09-11T03:47:35.813 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.813 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.813 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:35.813 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:47:35.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:47:35.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118419+0000", 2024-09-11T03:47:35.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6100000000000002e-07 2024-09-11T03:47:35.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122019+0000", 2024-09-11T03:47:35.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0036004240000000001 2024-09-11T03:47:35.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122067+0000", 2024-09-11T03:47:35.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7589000000000001e-05 2024-09-11T03:47:35.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122077+0000", 2024-09-11T03:47:35.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0036e-05 2024-09-11T03:47:35.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122133+0000", 2024-09-11T03:47:35.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6350000000000001e-05 2024-09-11T03:47:35.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122152+0000", 2024-09-11T03:47:35.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8595999999999999e-05 2024-09-11T03:47:35.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:35.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129179+0000", 2024-09-11T03:47:35.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0070268079999999998 2024-09-11T03:47:35.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129228+0000", 2024-09-11T03:47:35.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9594e-05 2024-09-11T03:47:35.817 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.817 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.817 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.817 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:649 2.6 e131/95 2:6dbd1e81:::smithi17557870-7:head v 131'25, mlcod=123'21)", 2024-09-11T03:47:35.818 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:08.118731+0000", 2024-09-11T03:47:35.818 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 507.69246188199998, 2024-09-11T03:47:35.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01067091, 2024-09-11T03:47:35.818 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.818 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.818 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:35.818 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118731+0000", 2024-09-11T03:47:35.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118731+0000", 2024-09-11T03:47:35.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.118731+0000", 2024-09-11T03:47:35.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4399999999999998e-07 2024-09-11T03:47:35.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.121866+0000", 2024-09-11T03:47:35.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031341720000000002 2024-09-11T03:47:35.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.121900+0000", 2024-09-11T03:47:35.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4266e-05 2024-09-11T03:47:35.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.121914+0000", 2024-09-11T03:47:35.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3668e-05 2024-09-11T03:47:35.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122002+0000", 2024-09-11T03:47:35.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7995e-05 2024-09-11T03:47:35.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.122029+0000", 2024-09-11T03:47:35.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7199000000000002e-05 2024-09-11T03:47:35.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:35.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129354+0000", 2024-09-11T03:47:35.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073255899999999999 2024-09-11T03:47:35.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:08.129401+0000", 2024-09-11T03:47:35.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7076000000000001e-05 2024-09-11T03:47:35.822 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.822 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.822 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.822 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1524 2.6 e212/95 2:6dbd1e81:::smithi17557870-7:head v 212'123, mlcod=204'119)", 2024-09-11T03:47:35.823 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:28.927804+0000", 2024-09-11T03:47:35.823 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 426.88338830399999, 2024-09-11T03:47:35.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087752009999999998, 2024-09-11T03:47:35.823 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.823 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.823 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:35.823 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927804+0000", 2024-09-11T03:47:35.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927804+0000", 2024-09-11T03:47:35.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.927804+0000", 2024-09-11T03:47:35.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8799999999999998e-07 2024-09-11T03:47:35.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929502+0000", 2024-09-11T03:47:35.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016974690000000001 2024-09-11T03:47:35.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929523+0000", 2024-09-11T03:47:35.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1373999999999999e-05 2024-09-11T03:47:35.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929543+0000", 2024-09-11T03:47:35.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9950999999999999e-05 2024-09-11T03:47:35.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929633+0000", 2024-09-11T03:47:35.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0158e-05 2024-09-11T03:47:35.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.929657+0000", 2024-09-11T03:47:35.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3737999999999999e-05 2024-09-11T03:47:35.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:35.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.936522+0000", 2024-09-11T03:47:35.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068650589999999997 2024-09-11T03:47:35.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:28.936579+0000", 2024-09-11T03:47:35.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7163999999999997e-05 2024-09-11T03:47:35.827 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.827 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.827 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.827 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1589 2.7 e218/191 2:e77ce595:::smithi17557870-31:head v 218'174, mlcod=218'168)", 2024-09-11T03:47:35.827 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:35.116398+0000", 2024-09-11T03:47:35.827 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 420.694793973, 2024-09-11T03:47:35.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0089646050000000005, 2024-09-11T03:47:35.828 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.828 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.828 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:35.828 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116398+0000", 2024-09-11T03:47:35.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116398+0000", 2024-09-11T03:47:35.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116400+0000", 2024-09-11T03:47:35.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6109999999999999e-06 2024-09-11T03:47:35.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119261+0000", 2024-09-11T03:47:35.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028605309999999999 2024-09-11T03:47:35.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119281+0000", 2024-09-11T03:47:35.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0491000000000001e-05 2024-09-11T03:47:35.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119292+0000", 2024-09-11T03:47:35.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0759e-05 2024-09-11T03:47:35.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119350+0000", 2024-09-11T03:47:35.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8588999999999998e-05 2024-09-11T03:47:35.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119369+0000", 2024-09-11T03:47:35.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8227e-05 2024-09-11T03:47:35.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:35.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125320+0000", 2024-09-11T03:47:35.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059510409999999998 2024-09-11T03:47:35.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125363+0000", 2024-09-11T03:47:35.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3356000000000003e-05 2024-09-11T03:47:35.832 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.832 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.832 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.832 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1816 2.d e243/95 2:b7bdfe52:::smithi17557870-10:head v 243'237, mlcod=243'233)", 2024-09-11T03:47:35.832 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:59.922567+0000", 2024-09-11T03:47:35.832 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 395.88862562200001, 2024-09-11T03:47:35.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094601100000000007, 2024-09-11T03:47:35.833 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.833 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.833 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:35.833 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.922567+0000", 2024-09-11T03:47:35.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.922567+0000", 2024-09-11T03:47:35.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.922568+0000", 2024-09-11T03:47:35.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.345e-06 2024-09-11T03:47:35.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929634+0000", 2024-09-11T03:47:35.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0070657130000000004 2024-09-11T03:47:35.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929654+0000", 2024-09-11T03:47:35.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9972000000000001e-05 2024-09-11T03:47:35.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929663+0000", 2024-09-11T03:47:35.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3770000000000002e-06 2024-09-11T03:47:35.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929731+0000", 2024-09-11T03:47:35.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8095000000000004e-05 2024-09-11T03:47:35.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929759+0000", 2024-09-11T03:47:35.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7248999999999999e-05 2024-09-11T03:47:35.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:35.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.931991+0000", 2024-09-11T03:47:35.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022320679999999998 2024-09-11T03:47:35.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.932027+0000", 2024-09-11T03:47:35.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6291000000000002e-05 2024-09-11T03:47:35.837 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.837 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.837 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.837 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2431 2.12 2:4e5019e5:::smithi17557870-40:head [copy-from ver 279] snapc 66=[66,65,64,63,61,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-09-11T03:47:35.837 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:51.877192+0000", 2024-09-11T03:47:35.837 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 343.93400033, 2024-09-11T03:47:35.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013330089, 2024-09-11T03:47:35.838 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.838 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.838 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:35.838 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:35.838 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:35.838 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:35.838 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2431 2024-09-11T03:47:35.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.838 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877192+0000", 2024-09-11T03:47:35.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877192+0000", 2024-09-11T03:47:35.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877194+0000", 2024-09-11T03:47:35.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.767e-06 2024-09-11T03:47:35.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877202+0000", 2024-09-11T03:47:35.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0770000000000008e-06 2024-09-11T03:47:35.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877220+0000", 2024-09-11T03:47:35.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8561000000000001e-05 2024-09-11T03:47:35.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877230+0000", 2024-09-11T03:47:35.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2820000000000002e-06 2024-09-11T03:47:35.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877290+0000", 2024-09-11T03:47:35.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0025999999999998e-05 2024-09-11T03:47:35.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877404+0000", 2024-09-11T03:47:35.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011425999999999999 2024-09-11T03:47:35.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,11", 2024-09-11T03:47:35.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.884630+0000", 2024-09-11T03:47:35.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072256769999999998 2024-09-11T03:47:35.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:47:35.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.888559+0000", 2024-09-11T03:47:35.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039288750000000001 2024-09-11T03:47:35.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:35.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.888615+0000", 2024-09-11T03:47:35.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6609e-05 2024-09-11T03:47:35.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:35.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.890459+0000", 2024-09-11T03:47:35.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001844116 2024-09-11T03:47:35.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:35.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.890494+0000", 2024-09-11T03:47:35.843 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:35.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4921e-05 2024-09-11T03:47:35.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.890522+0000", 2024-09-11T03:47:35.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7917999999999999e-05 2024-09-11T03:47:35.845 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.845 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.845 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.845 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2432 2.12 2:4e5019e5:::smithi17557870-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e295)", 2024-09-11T03:47:35.845 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:51.877251+0000", 2024-09-11T03:47:35.845 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 343.933941678, 2024-09-11T03:47:35.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010600994000000001, 2024-09-11T03:47:35.845 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.845 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.845 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:47:35.846 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:35.846 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:35.846 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:35.846 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2432 2024-09-11T03:47:35.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.846 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877251+0000", 2024-09-11T03:47:35.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877251+0000", 2024-09-11T03:47:35.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877251+0000", 2024-09-11T03:47:35.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3399999999999999e-07 2024-09-11T03:47:35.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877254+0000", 2024-09-11T03:47:35.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.379e-06 2024-09-11T03:47:35.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877262+0000", 2024-09-11T03:47:35.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0779999999999996e-06 2024-09-11T03:47:35.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877265+0000", 2024-09-11T03:47:35.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.641e-06 2024-09-11T03:47:35.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877539+0000", 2024-09-11T03:47:35.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027397000000000001 2024-09-11T03:47:35.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:35.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877571+0000", 2024-09-11T03:47:35.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2122999999999997e-05 2024-09-11T03:47:35.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.887709+0000", 2024-09-11T03:47:35.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010137922000000001 2024-09-11T03:47:35.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.887781+0000", 2024-09-11T03:47:35.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1384999999999995e-05 2024-09-11T03:47:35.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.887852+0000", 2024-09-11T03:47:35.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0962000000000003e-05 2024-09-11T03:47:35.850 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.850 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.850 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.850 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2443 2.12 2:494ba932:::smithi17557870-43:head [copy-from ver 342] snapc 66=[65,64,63,61,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-11T03:47:35.851 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:52.884558+0000", 2024-09-11T03:47:35.851 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 342.92663396099999, 2024-09-11T03:47:35.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019937844999999999, 2024-09-11T03:47:35.851 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.851 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.851 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:35.851 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:35.851 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:35.851 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:35.851 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2443 2024-09-11T03:47:35.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.851 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884558+0000", 2024-09-11T03:47:35.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884558+0000", 2024-09-11T03:47:35.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884560+0000", 2024-09-11T03:47:35.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7e-06 2024-09-11T03:47:35.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884568+0000", 2024-09-11T03:47:35.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0940000000000008e-06 2024-09-11T03:47:35.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884584+0000", 2024-09-11T03:47:35.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6277e-05 2024-09-11T03:47:35.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884593+0000", 2024-09-11T03:47:35.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1810000000000001e-06 2024-09-11T03:47:35.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884660+0000", 2024-09-11T03:47:35.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7362999999999999e-05 2024-09-11T03:47:35.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884731+0000", 2024-09-11T03:47:35.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1316999999999995e-05 2024-09-11T03:47:35.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,11", 2024-09-11T03:47:35.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.895614+0000", 2024-09-11T03:47:35.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010882911 2024-09-11T03:47:35.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:47:35.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.900723+0000", 2024-09-11T03:47:35.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005109143 2024-09-11T03:47:35.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:35.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.902758+0000", 2024-09-11T03:47:35.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020342870000000001 2024-09-11T03:47:35.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:35.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.904417+0000", 2024-09-11T03:47:35.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001658875 2024-09-11T03:47:35.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:35.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.904454+0000", 2024-09-11T03:47:35.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7883999999999998e-05 2024-09-11T03:47:35.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.904496+0000", 2024-09-11T03:47:35.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1813000000000001e-05 2024-09-11T03:47:35.857 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.857 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.857 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.857 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2444 2.12 2:494ba932:::smithi17557870-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e296)", 2024-09-11T03:47:35.857 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:52.884615+0000", 2024-09-11T03:47:35.857 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 342.92657739999999, 2024-09-11T03:47:35.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015042823, 2024-09-11T03:47:35.857 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.858 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.858 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:47:35.858 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:35.858 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:35.858 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:35.858 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2444 2024-09-11T03:47:35.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.858 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884615+0000", 2024-09-11T03:47:35.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884615+0000", 2024-09-11T03:47:35.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884616+0000", 2024-09-11T03:47:35.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5499999999999998e-07 2024-09-11T03:47:35.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884618+0000", 2024-09-11T03:47:35.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.818e-06 2024-09-11T03:47:35.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884632+0000", 2024-09-11T03:47:35.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3219e-05 2024-09-11T03:47:35.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884639+0000", 2024-09-11T03:47:35.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9990000000000002e-06 2024-09-11T03:47:35.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884918+0000", 2024-09-11T03:47:35.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027906699999999998 2024-09-11T03:47:35.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:35.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884959+0000", 2024-09-11T03:47:35.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1844999999999997e-05 2024-09-11T03:47:35.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.899489+0000", 2024-09-11T03:47:35.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014529034999999999 2024-09-11T03:47:35.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.899577+0000", 2024-09-11T03:47:35.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8512999999999998e-05 2024-09-11T03:47:35.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.899658+0000", 2024-09-11T03:47:35.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0772000000000005e-05 2024-09-11T03:47:35.862 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.862 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.862 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.862 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:2568 2.d e314/95 2:b6a25a79:::smithi17557870-47:head v 314'392, mlcod=314'388)", 2024-09-11T03:47:35.863 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:10.885879+0000", 2024-09-11T03:47:35.863 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 324.92531355599999, 2024-09-11T03:47:35.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086201669999999998, 2024-09-11T03:47:35.863 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.863 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.863 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:35.863 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.885879+0000", 2024-09-11T03:47:35.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.885879+0000", 2024-09-11T03:47:35.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.885880+0000", 2024-09-11T03:47:35.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1900000000000002e-07 2024-09-11T03:47:35.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.892125+0000", 2024-09-11T03:47:35.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062450869999999999 2024-09-11T03:47:35.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.892137+0000", 2024-09-11T03:47:35.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2224e-05 2024-09-11T03:47:35.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.892143+0000", 2024-09-11T03:47:35.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.426e-06 2024-09-11T03:47:35.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.892200+0000", 2024-09-11T03:47:35.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6511999999999997e-05 2024-09-11T03:47:35.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.892218+0000", 2024-09-11T03:47:35.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.77e-05 2024-09-11T03:47:35.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:35.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.894465+0000", 2024-09-11T03:47:35.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022472159999999998 2024-09-11T03:47:35.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:10.894499+0000", 2024-09-11T03:47:35.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4282999999999997e-05 2024-09-11T03:47:35.866 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.867 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.867 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.867 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2641 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 350] snapc 75=[75,74,73,71,6f,6c,68,65,5d] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-11T03:47:35.867 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:19.887918+0000", 2024-09-11T03:47:35.867 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 315.92327410600001, 2024-09-11T03:47:35.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011895905, 2024-09-11T03:47:35.867 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.867 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.867 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:35.867 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:35.867 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:35.867 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:35.867 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2641 2024-09-11T03:47:35.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.868 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887918+0000", 2024-09-11T03:47:35.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887918+0000", 2024-09-11T03:47:35.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887920+0000", 2024-09-11T03:47:35.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.762e-06 2024-09-11T03:47:35.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887928+0000", 2024-09-11T03:47:35.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8900000000000007e-06 2024-09-11T03:47:35.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887945+0000", 2024-09-11T03:47:35.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6707000000000001e-05 2024-09-11T03:47:35.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887953+0000", 2024-09-11T03:47:35.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5620000000000004e-06 2024-09-11T03:47:35.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.888014+0000", 2024-09-11T03:47:35.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0798000000000003e-05 2024-09-11T03:47:35.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:47:35.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.888019+0000", 2024-09-11T03:47:35.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5990000000000004e-06 2024-09-11T03:47:35.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.889290+0000", 2024-09-11T03:47:35.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012718009999999999 2024-09-11T03:47:35.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.889393+0000", 2024-09-11T03:47:35.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102298 2024-09-11T03:47:35.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,8", 2024-09-11T03:47:35.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.894120+0000", 2024-09-11T03:47:35.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047277259999999998 2024-09-11T03:47:35.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:47:35.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.897232+0000", 2024-09-11T03:47:35.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031110859999999999 2024-09-11T03:47:35.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:35.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.897530+0000", 2024-09-11T03:47:35.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000298342 2024-09-11T03:47:35.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:35.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.899728+0000", 2024-09-11T03:47:35.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021984819999999999 2024-09-11T03:47:35.876 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:35.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:35.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.899782+0000", 2024-09-11T03:47:35.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3922999999999999e-05 2024-09-11T03:47:35.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.899814+0000", 2024-09-11T03:47:35.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1928999999999999e-05 2024-09-11T03:47:35.877 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.877 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.877 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.878 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2642 2.19 2:98bb65f5:::smithi17557870-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e323)", 2024-09-11T03:47:35.878 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:19.887977+0000", 2024-09-11T03:47:35.878 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 315.92321555500001, 2024-09-11T03:47:35.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085743450000000006, 2024-09-11T03:47:35.878 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.878 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.878 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:47:35.878 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:35.878 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:35.878 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:35.878 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2642 2024-09-11T03:47:35.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.878 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887977+0000", 2024-09-11T03:47:35.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887977+0000", 2024-09-11T03:47:35.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887977+0000", 2024-09-11T03:47:35.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9699999999999996e-07 2024-09-11T03:47:35.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887980+0000", 2024-09-11T03:47:35.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4700000000000001e-06 2024-09-11T03:47:35.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887986+0000", 2024-09-11T03:47:35.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1140000000000002e-06 2024-09-11T03:47:35.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887991+0000", 2024-09-11T03:47:35.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3730000000000001e-06 2024-09-11T03:47:35.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.888058+0000", 2024-09-11T03:47:35.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6322999999999995e-05 2024-09-11T03:47:35.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:35.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.888062+0000", 2024-09-11T03:47:35.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7600000000000002e-06 2024-09-11T03:47:35.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.889535+0000", 2024-09-11T03:47:35.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001472391 2024-09-11T03:47:35.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:35.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.889580+0000", 2024-09-11T03:47:35.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5139000000000003e-05 2024-09-11T03:47:35.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.896410+0000", 2024-09-11T03:47:35.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068302379999999998 2024-09-11T03:47:35.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.896478+0000", 2024-09-11T03:47:35.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8159999999999998e-05 2024-09-11T03:47:35.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.896551+0000", 2024-09-11T03:47:35.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2780000000000005e-05 2024-09-11T03:47:35.884 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.884 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.884 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.884 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2944 2.12 2:494ba932:::smithi17557870-43:head [copy-from ver 311] snapc 80=[80,7b,7a,79,77,75,6c] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-11T03:47:35.884 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:43.962947+0000", 2024-09-11T03:47:35.884 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 291.84824490599999, 2024-09-11T03:47:35.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014367049, 2024-09-11T03:47:35.884 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.884 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.884 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:35.885 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:35.885 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:35.885 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:35.885 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2944 2024-09-11T03:47:35.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.885 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962947+0000", 2024-09-11T03:47:35.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962947+0000", 2024-09-11T03:47:35.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962949+0000", 2024-09-11T03:47:35.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.689e-06 2024-09-11T03:47:35.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962955+0000", 2024-09-11T03:47:35.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.297e-06 2024-09-11T03:47:35.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962967+0000", 2024-09-11T03:47:35.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1517e-05 2024-09-11T03:47:35.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962975+0000", 2024-09-11T03:47:35.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2889999999999998e-06 2024-09-11T03:47:35.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963875+0000", 2024-09-11T03:47:35.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00089981200000000005 2024-09-11T03:47:35.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.964025+0000", 2024-09-11T03:47:35.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000149624 2024-09-11T03:47:35.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,7", 2024-09-11T03:47:35.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.971400+0000", 2024-09-11T03:47:35.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073748349999999997 2024-09-11T03:47:35.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:47:35.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.975907+0000", 2024-09-11T03:47:35.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045076159999999999 2024-09-11T03:47:35.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:35.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.976808+0000", 2024-09-11T03:47:35.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00090078700000000001 2024-09-11T03:47:35.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:35.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.977244+0000", 2024-09-11T03:47:35.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00043565800000000001 2024-09-11T03:47:35.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:35.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.977281+0000", 2024-09-11T03:47:35.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7150999999999997e-05 2024-09-11T03:47:35.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.977315+0000", 2024-09-11T03:47:35.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3773999999999999e-05 2024-09-11T03:47:35.892 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.892 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.892 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.893 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2945 2.12 2:494ba932:::smithi17557870-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e347)", 2024-09-11T03:47:35.893 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:43.963009+0000", 2024-09-11T03:47:35.893 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 291.84818330600001, 2024-09-11T03:47:35.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012192595000000001, 2024-09-11T03:47:35.893 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.893 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.893 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:47:35.893 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:35.893 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:35.894 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:35.894 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2945 2024-09-11T03:47:35.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.894 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963009+0000", 2024-09-11T03:47:35.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963009+0000", 2024-09-11T03:47:35.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963010+0000", 2024-09-11T03:47:35.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0699999999999997e-07 2024-09-11T03:47:35.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963012+0000", 2024-09-11T03:47:35.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6599999999999999e-06 2024-09-11T03:47:35.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963020+0000", 2024-09-11T03:47:35.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2649999999999999e-06 2024-09-11T03:47:35.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963025+0000", 2024-09-11T03:47:35.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6860000000000003e-06 2024-09-11T03:47:35.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.897 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.964185+0000", 2024-09-11T03:47:35.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011599329999999999 2024-09-11T03:47:35.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:35.897 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.964230+0000", 2024-09-11T03:47:35.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4336999999999999e-05 2024-09-11T03:47:35.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.974986+0000", 2024-09-11T03:47:35.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010755928999999999 2024-09-11T03:47:35.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.975091+0000", 2024-09-11T03:47:35.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105701 2024-09-11T03:47:35.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.975202+0000", 2024-09-11T03:47:35.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011047700000000001 2024-09-11T03:47:35.898 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.899 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.899 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.899 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:3536 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 408] snapc 90=[90,8f,8d,8c,8b,8a] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-11T03:47:35.899 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:43:18.903548+0000", 2024-09-11T03:47:35.899 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 256.907643974, 2024-09-11T03:47:35.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014015196000000001, 2024-09-11T03:47:35.899 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.899 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.899 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:35.899 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:35.899 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:35.899 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:35.900 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3536 2024-09-11T03:47:35.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.900 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903548+0000", 2024-09-11T03:47:35.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903548+0000", 2024-09-11T03:47:35.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903550+0000", 2024-09-11T03:47:35.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.499e-06 2024-09-11T03:47:35.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903555+0000", 2024-09-11T03:47:35.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6890000000000003e-06 2024-09-11T03:47:35.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903567+0000", 2024-09-11T03:47:35.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1912999999999999e-05 2024-09-11T03:47:35.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903574+0000", 2024-09-11T03:47:35.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7260000000000002e-06 2024-09-11T03:47:35.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904043+0000", 2024-09-11T03:47:35.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00046877799999999999 2024-09-11T03:47:35.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904136+0000", 2024-09-11T03:47:35.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2745999999999996e-05 2024-09-11T03:47:35.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,8", 2024-09-11T03:47:35.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.912436+0000", 2024-09-11T03:47:35.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083000049999999992 2024-09-11T03:47:35.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:47:35.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.915813+0000", 2024-09-11T03:47:35.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033772839999999999 2024-09-11T03:47:35.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:35.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.916094+0000", 2024-09-11T03:47:35.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028135600000000002 2024-09-11T03:47:35.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:35.906 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.917498+0000", 2024-09-11T03:47:35.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014033680000000001 2024-09-11T03:47:35.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:35.906 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.917534+0000", 2024-09-11T03:47:35.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6180000000000003e-05 2024-09-11T03:47:35.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.917564+0000", 2024-09-11T03:47:35.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9652e-05 2024-09-11T03:47:35.907 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.907 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.907 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.907 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:3537 2.19 2:98bb65f5:::smithi17557870-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e381)", 2024-09-11T03:47:35.907 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:47:35.907 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 256.90759650299998, 2024-09-11T03:47:35.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011559349999999999, 2024-09-11T03:47:35.907 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.907 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.908 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:47:35.908 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:47:35.908 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:47:35.908 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:35.908 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3537 2024-09-11T03:47:35.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.908 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:47:35.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:47:35.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:47:35.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8899999999999999e-07 2024-09-11T03:47:35.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.911 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903599+0000", 2024-09-11T03:47:35.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3750000000000001e-06 2024-09-11T03:47:35.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.912 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903606+0000", 2024-09-11T03:47:35.912 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1969999999999999e-06 2024-09-11T03:47:35.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.912 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903611+0000", 2024-09-11T03:47:35.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3680000000000001e-06 2024-09-11T03:47:35.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904244+0000", 2024-09-11T03:47:35.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00063279999999999999 2024-09-11T03:47:35.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:35.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904291+0000", 2024-09-11T03:47:35.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6755999999999998e-05 2024-09-11T03:47:35.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.914973+0000", 2024-09-11T03:47:35.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010682199999999999 2024-09-11T03:47:35.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.915043+0000", 2024-09-11T03:47:35.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0297999999999997e-05 2024-09-11T03:47:35.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.915155+0000", 2024-09-11T03:47:35.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000111767 2024-09-11T03:47:35.915 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.915 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.916 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.916 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:4907 2.7 e522/417 2:e0ec5763:::smithi17557870-2:head v 522'753, mlcod=522'751)", 2024-09-11T03:47:35.916 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:45:40.117077+0000", 2024-09-11T03:47:35.916 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 115.694114998, 2024-09-11T03:47:35.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010151360999999999, 2024-09-11T03:47:35.917 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.917 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.917 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:35.917 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.917 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.117077+0000", 2024-09-11T03:47:35.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.117077+0000", 2024-09-11T03:47:35.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.117079+0000", 2024-09-11T03:47:35.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7799999999999999e-06 2024-09-11T03:47:35.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.122431+0000", 2024-09-11T03:47:35.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053519090000000002 2024-09-11T03:47:35.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.122448+0000", 2024-09-11T03:47:35.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6521e-05 2024-09-11T03:47:35.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.122455+0000", 2024-09-11T03:47:35.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3629999999999996e-06 2024-09-11T03:47:35.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.122511+0000", 2024-09-11T03:47:35.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5967000000000001e-05 2024-09-11T03:47:35.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.122528+0000", 2024-09-11T03:47:35.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6996000000000001e-05 2024-09-11T03:47:35.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:35.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.127188+0000", 2024-09-11T03:47:35.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046599509999999999 2024-09-11T03:47:35.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.127229+0000", 2024-09-11T03:47:35.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0874000000000002e-05 2024-09-11T03:47:35.921 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.921 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.922 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.922 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:5885 2.e e613/577 2:77dadb97:::smithi17557870-30:head v 613'880, mlcod=601'877)", 2024-09-11T03:47:35.922 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:47:11.245847+0000", 2024-09-11T03:47:35.922 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 24.565345654000001, 2024-09-11T03:47:35.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012027239, 2024-09-11T03:47:35.922 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.923 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.923 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:35.923 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.245847+0000", 2024-09-11T03:47:35.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.245847+0000", 2024-09-11T03:47:35.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.245847+0000", 2024-09-11T03:47:35.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7300000000000002e-07 2024-09-11T03:47:35.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.252621+0000", 2024-09-11T03:47:35.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067743339999999999 2024-09-11T03:47:35.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.925 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.252641+0000", 2024-09-11T03:47:35.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9734000000000001e-05 2024-09-11T03:47:35.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.925 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.252650+0000", 2024-09-11T03:47:35.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3400000000000004e-06 2024-09-11T03:47:35.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.252721+0000", 2024-09-11T03:47:35.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0767999999999997e-05 2024-09-11T03:47:35.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.252745+0000", 2024-09-11T03:47:35.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4278999999999999e-05 2024-09-11T03:47:35.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:35.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.257835+0000", 2024-09-11T03:47:35.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0050897060000000003 2024-09-11T03:47:35.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.257874+0000", 2024-09-11T03:47:35.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8804999999999999e-05 2024-09-11T03:47:35.927 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.927 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.927 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.927 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:5976 2.19 e625/560 2:98bb65f5:::smithi17557870-6:head v 625'489, mlcod=618'486)", 2024-09-11T03:47:35.927 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:47:23.160182+0000", 2024-09-11T03:47:35.927 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 12.651010703000001, 2024-09-11T03:47:35.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0089006399999999996, 2024-09-11T03:47:35.928 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.928 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.928 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:35.928 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:23.160182+0000", 2024-09-11T03:47:35.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:23.160182+0000", 2024-09-11T03:47:35.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:23.160182+0000", 2024-09-11T03:47:35.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.04e-07 2024-09-11T03:47:35.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:23.167777+0000", 2024-09-11T03:47:35.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0075952019999999997 2024-09-11T03:47:35.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:23.167790+0000", 2024-09-11T03:47:35.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2819e-05 2024-09-11T03:47:35.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:23.167797+0000", 2024-09-11T03:47:35.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.306e-06 2024-09-11T03:47:35.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:23.167845+0000", 2024-09-11T03:47:35.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7357000000000001e-05 2024-09-11T03:47:35.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:23.167861+0000", 2024-09-11T03:47:35.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6592e-05 2024-09-11T03:47:35.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:35.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:23.169052+0000", 2024-09-11T03:47:35.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011904750000000001 2024-09-11T03:47:35.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:23.169082+0000", 2024-09-11T03:47:35.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0685000000000002e-05 2024-09-11T03:47:35.932 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.932 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.932 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.932 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:6152 2.e e637/577 2:77dadb97:::smithi17557870-30:head v 637'891, mlcod=634'888)", 2024-09-11T03:47:35.932 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:47:35.714675+0000", 2024-09-11T03:47:35.932 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.096516975000000005, 2024-09-11T03:47:35.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011964542, 2024-09-11T03:47:35.932 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:47:35.932 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:47:35.932 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:35.932 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:47:35.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:47:35.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714675+0000", 2024-09-11T03:47:35.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:47:35.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714675+0000", 2024-09-11T03:47:35.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:47:35.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:47:35.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714676+0000", 2024-09-11T03:47:35.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0100000000000001e-07 2024-09-11T03:47:35.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:47:35.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.720928+0000", 2024-09-11T03:47:35.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062523839999999997 2024-09-11T03:47:35.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:47:35.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.720951+0000", 2024-09-11T03:47:35.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2751e-05 2024-09-11T03:47:35.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:47:35.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.720964+0000", 2024-09-11T03:47:35.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2673999999999999e-05 2024-09-11T03:47:35.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:47:35.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721056+0000", 2024-09-11T03:47:35.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2620000000000004e-05 2024-09-11T03:47:35.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:47:35.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721084+0000", 2024-09-11T03:47:35.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7974999999999999e-05 2024-09-11T03:47:35.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:47:35.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.726587+0000", 2024-09-11T03:47:35.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055026420000000003 2024-09-11T03:47:35.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:47:35.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:47:35.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:47:35.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.726640+0000", 2024-09-11T03:47:35.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3195000000000001e-05 2024-09-11T03:47:35.937 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.937 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.937 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.937 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:47:35.937 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:47:35.937 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:47:35.939 INFO:tasks.rados.rados.2.smithi175.stdout:2463: finishing write tid 2 to smithi17557870-7 2024-09-11T03:47:35.940 INFO:tasks.rados.rados.2.smithi175.stdout:2463: finishing write tid 3 to smithi17557870-7 2024-09-11T03:47:35.940 INFO:tasks.rados.rados.2.smithi175.stdout:2463: finishing write tid 4 to smithi17557870-7 2024-09-11T03:47:35.940 INFO:tasks.rados.rados.2.smithi175.stdout:2463: oid 7 updating version 0 to 538 2024-09-11T03:47:35.940 INFO:tasks.rados.rados.2.smithi175.stdout:2463: oid 7 version 538 is already newer than 536 2024-09-11T03:47:35.940 INFO:tasks.rados.rados.2.smithi175.stdout:2463: oid 7 version 538 is already newer than 537 2024-09-11T03:47:35.940 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 538 (ObjNum 853 snap 267 seq_num 853) dirty exists 2024-09-11T03:47:35.940 INFO:tasks.rados.rados.2.smithi175.stdout:2463: left oid 7 (ObjNum 853 snap 267 seq_num 853) 2024-09-11T03:47:35.940 INFO:tasks.rados.rados.2.smithi175.stdout:2465: finishing copy_from racing read to smithi17557870-30 2024-09-11T03:47:35.940 INFO:tasks.rados.rados.2.smithi175.stdout:2465: finishing copy_from to smithi17557870-30 2024-09-11T03:47:35.940 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 268 (ObjNum 800 snap 255 seq_num 800) dirty exists 2024-09-11T03:47:35.940 INFO:tasks.rados.rados.2.smithi175.stdout:2459: done (4 left) 2024-09-11T03:47:35.940 INFO:tasks.rados.rados.2.smithi175.stdout:2460: done (3 left) 2024-09-11T03:47:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:2463: done (2 left) 2024-09-11T03:47:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:2464: done (1 left) 2024-09-11T03:47:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:2465: done (0 left) 2024-09-11T03:47:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:2466: rollback oid 11 current snap is 267 2024-09-11T03:47:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:2466: done (0 left) 2024-09-11T03:47:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:2467: copy_from oid 8 from oid 9 current snap is 267 2024-09-11T03:47:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:2468: write (excl) oid 12 current snap is 267 2024-09-11T03:47:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:2468: seq_num 854 ranges {753178=494182,1857387=495791} 2024-09-11T03:47:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:2468: writing smithi17557870-12 from 753178 to 1247360 tid 1 2024-09-11T03:47:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:2468: writing smithi17557870-12 from 1857387 to 2353178 tid 2 2024-09-11T03:47:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:2467: finishing copy_from to smithi17557870-8 2024-09-11T03:47:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:2467: got expected ENOENT (src dne) 2024-09-11T03:47:35.941 INFO:tasks.rados.rados.2.smithi175.stdout:2467: finishing copy_from racing read to smithi17557870-8 2024-09-11T03:47:35.942 INFO:tasks.rados.rados.2.smithi175.stdout:2468: finishing write tid 1 to smithi17557870-12 2024-09-11T03:47:35.942 INFO:tasks.rados.rados.2.smithi175.stdout:2467: done (1 left) 2024-09-11T03:47:35.942 INFO:tasks.rados.rados.2.smithi175.stdout:2469: copy_from oid 30 from oid 39 current snap is 267 2024-09-11T03:47:35.942 INFO:tasks.rados.rados.2.smithi175.stdout:2470: rollback oid 11 current snap is 267 2024-09-11T03:47:35.942 INFO:tasks.rados.rados.2.smithi175.stdout:2470: done (2 left) 2024-09-11T03:47:35.942 INFO:tasks.rados.rados.2.smithi175.stdout:2471: snap_create 2024-09-11T03:47:35.942 INFO:tasks.rados.rados.2.smithi175.stdout:2469: finishing copy_from to smithi17557870-30 2024-09-11T03:47:35.942 INFO:tasks.rados.rados.2.smithi175.stdout:2469: got expected ENOENT (src dne) 2024-09-11T03:47:35.942 INFO:tasks.rados.rados.2.smithi175.stdout:2469: finishing copy_from racing read to smithi17557870-30 2024-09-11T03:47:35.942 INFO:tasks.rados.rados.2.smithi175.stdout:2468: finishing write tid 2 to smithi17557870-12 2024-09-11T03:47:35.942 INFO:tasks.rados.rados.2.smithi175.stdout:2468: finishing write tid 3 to smithi17557870-12 2024-09-11T03:47:35.942 INFO:tasks.rados.rados.2.smithi175.stdout:2468: finishing write tid 4 to smithi17557870-12 2024-09-11T03:47:35.942 INFO:tasks.rados.rados.2.smithi175.stdout:2468: oid 12 updating version 0 to 896 2024-09-11T03:47:35.942 INFO:tasks.rados.rados.2.smithi175.stdout:2468: oid 12 updating version 896 to 898 2024-09-11T03:47:35.943 INFO:tasks.rados.rados.2.smithi175.stdout:2468: oid 12 version 898 is already newer than 897 2024-09-11T03:47:35.943 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 898 (ObjNum 854 snap 267 seq_num 854) dirty exists 2024-09-11T03:47:35.943 INFO:tasks.rados.rados.2.smithi175.stdout:2468: left oid 12 (ObjNum 854 snap 267 seq_num 854) 2024-09-11T03:47:35.958 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:35.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:35.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:35.976 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:47:35.976 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T03:47:35.976 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:47:36.022 INFO:tasks.rados.rados.2.smithi175.stdout:2468: done (2 left) 2024-09-11T03:47:36.022 INFO:tasks.rados.rados.2.smithi175.stdout:2469: done (1 left) 2024-09-11T03:47:36.022 INFO:tasks.rados.rados.2.smithi175.stdout:2471: done (0 left) 2024-09-11T03:47:36.022 INFO:tasks.rados.rados.2.smithi175.stdout:2472: write oid 23 current snap is 268 2024-09-11T03:47:36.022 INFO:tasks.rados.rados.2.smithi175.stdout:2472: seq_num 855 ranges {769985=569431,1915869=437966} 2024-09-11T03:47:36.028 INFO:tasks.rados.rados.2.smithi175.stdout:2472: writing smithi17557870-23 from 769985 to 1339416 tid 1 2024-09-11T03:47:36.034 INFO:tasks.rados.rados.2.smithi175.stdout:2472: writing smithi17557870-23 from 1915869 to 2353835 tid 2 2024-09-11T03:47:36.034 INFO:tasks.rados.rados.2.smithi175.stdout:2473: snap_create 2024-09-11T03:47:36.034 INFO:tasks.rados.rados.2.smithi175.stdout:2472: finishing write tid 1 to smithi17557870-23 2024-09-11T03:47:36.037 INFO:tasks.rados.rados.2.smithi175.stdout:2472: finishing write tid 2 to smithi17557870-23 2024-09-11T03:47:36.037 INFO:tasks.rados.rados.2.smithi175.stdout:2472: finishing write tid 3 to smithi17557870-23 2024-09-11T03:47:36.038 INFO:tasks.rados.rados.2.smithi175.stdout:2472: finishing write tid 4 to smithi17557870-23 2024-09-11T03:47:36.038 INFO:tasks.rados.rados.2.smithi175.stdout:2472: oid 23 updating version 0 to 857 2024-09-11T03:47:36.038 INFO:tasks.rados.rados.2.smithi175.stdout:2472: oid 23 updating version 857 to 859 2024-09-11T03:47:36.038 INFO:tasks.rados.rados.2.smithi175.stdout:2472: oid 23 version 859 is already newer than 858 2024-09-11T03:47:36.038 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 859 (ObjNum 855 snap 268 seq_num 855) dirty exists 2024-09-11T03:47:36.038 INFO:tasks.rados.rados.2.smithi175.stdout:2472: left oid 23 (ObjNum 855 snap 268 seq_num 855) 2024-09-11T03:47:36.060 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:36.082 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:36.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:36.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:36.093 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:36 smithi084 ceph-mon[27254]: pgmap v1110: 67 pgs: 67 active+clean; 129 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2024-09-11T03:47:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:36 smithi084 ceph-mon[27254]: osdmap e637: 12 total, 12 up, 12 in 2024-09-11T03:47:36.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:36 smithi121 ceph-mon[30729]: pgmap v1110: 67 pgs: 67 active+clean; 129 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2024-09-11T03:47:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:36 smithi121 ceph-mon[30729]: osdmap e637: 12 total, 12 up, 12 in 2024-09-11T03:47:36.184 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:36.206 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:36.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:36.307 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:36.328 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:36.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:36.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:36.452 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:36.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:36.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:36 smithi175 ceph-mon[30894]: pgmap v1110: 67 pgs: 67 active+clean; 129 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2024-09-11T03:47:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:36 smithi175 ceph-mon[30894]: osdmap e637: 12 total, 12 up, 12 in 2024-09-11T03:47:36.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:36.575 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:36.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:36.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:36.677 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:36.688 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:47:36.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:47:36.691 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:47:36.691 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:47:36.700 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:36.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:36.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:36.707 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:47:36.707 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:47:36.723 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:47:36.723 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:47:36.739 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:47:36.740 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:47:36.756 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:47:36.756 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:47:36.774 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:47:36.774 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:47:36.791 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:47:36.791 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:47:36.801 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:36.808 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:47:36.808 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:47:36.822 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:36.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:36.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:36.825 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:47:36.825 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:47:36.843 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:47:36.843 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:47:36.860 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:47:36.860 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:47:36.924 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:36.947 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:36.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:36.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:36.981 INFO:tasks.rados.rados.2.smithi175.stdout:2472: done (1 left) 2024-09-11T03:47:36.982 INFO:tasks.rados.rados.2.smithi175.stdout:2473: done (0 left) 2024-09-11T03:47:36.982 INFO:tasks.rados.rados.2.smithi175.stdout:2474: snap_remove snap 268 2024-09-11T03:47:37.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:37.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:37.072 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:37.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:37.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:37 smithi084 ceph-mon[27254]: osdmap e638: 12 total, 12 up, 12 in 2024-09-11T03:47:37.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2602079678' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:37.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:37 smithi084 ceph-mon[27254]: osdmap e639: 12 total, 12 up, 12 in 2024-09-11T03:47:37.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:37 smithi121 ceph-mon[30729]: osdmap e638: 12 total, 12 up, 12 in 2024-09-11T03:47:37.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:37 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2602079678' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:37 smithi121 ceph-mon[30729]: osdmap e639: 12 total, 12 up, 12 in 2024-09-11T03:47:37.173 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:37.194 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:37.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:37.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:37.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T03:47:37.296 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:37.319 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:37.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:37.321 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:37.421 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:37.443 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:37.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:37.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:37 smithi175 ceph-mon[30894]: osdmap e638: 12 total, 12 up, 12 in 2024-09-11T03:47:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:37 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2602079678' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:37 smithi175 ceph-mon[30894]: osdmap e639: 12 total, 12 up, 12 in 2024-09-11T03:47:37.515 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:47:37.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:37.566 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:37.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:37.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:37.668 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:37.689 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:37.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:37.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:37.791 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:37.811 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:37.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:37.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:37.913 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:37.939 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:37.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:37.939 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:37.979 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:47:37.984 INFO:tasks.rados.rados.2.smithi175.stdout:2474: done (0 left) 2024-09-11T03:47:37.984 INFO:tasks.rados.rados.2.smithi175.stdout:2475: snap_remove snap 267 2024-09-11T03:47:38.000 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:47:38.000 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:47:38.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:38.062 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:38.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:38.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:38.163 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:38.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:38 smithi121 ceph-mon[30729]: pgmap v1113: 67 pgs: 67 active+clean; 123 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1.3 MiB/s wr, 18 op/s 2024-09-11T03:47:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:38 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:47:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:38 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:47:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:38 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/594806448' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:47:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:38 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:38 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/594806448' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:47:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:38 smithi121 ceph-mon[30729]: osdmap e640: 12 total, 12 up, 12 in 2024-09-11T03:47:38.184 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:38.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:38.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:38.286 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:38.307 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:38.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:38.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:38.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:38 smithi084 ceph-mon[27254]: pgmap v1113: 67 pgs: 67 active+clean; 123 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1.3 MiB/s wr, 18 op/s 2024-09-11T03:47:38.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:38 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:47:38.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:38 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:47:38.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:38 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/594806448' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:47:38.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:38 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:38.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:38 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/594806448' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:47:38.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:38 smithi084 ceph-mon[27254]: osdmap e640: 12 total, 12 up, 12 in 2024-09-11T03:47:38.408 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:38.434 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:38.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:38.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:38 smithi175 ceph-mon[30894]: pgmap v1113: 67 pgs: 67 active+clean; 123 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1.3 MiB/s wr, 18 op/s 2024-09-11T03:47:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:38 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:47:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:38 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:47:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:38 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/594806448' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:47:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:38 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:38 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/594806448' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:47:38.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:38 smithi175 ceph-mon[30894]: osdmap e640: 12 total, 12 up, 12 in 2024-09-11T03:47:38.536 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:38.558 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:38.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:38.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:38.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T03:47:38.659 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:38.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:38.695 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:38.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:38.796 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:38.822 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:38.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:38.875 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:47:38.924 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:38.945 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:38.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:38.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:38.987 INFO:tasks.rados.rados.2.smithi175.stdout:2475: done (0 left) 2024-09-11T03:47:38.988 INFO:tasks.rados.rados.2.smithi175.stdout:2476: write oid 18 current snap is 269 2024-09-11T03:47:38.988 INFO:tasks.rados.rados.2.smithi175.stdout:2476: seq_num 856 ranges {786792=644680,1857998=528794} 2024-09-11T03:47:38.995 INFO:tasks.rados.rados.2.smithi175.stdout:2476: writing smithi17557870-18 from 786792 to 1431472 tid 1 2024-09-11T03:47:39.001 INFO:tasks.rados.rados.2.smithi175.stdout:2476: writing smithi17557870-18 from 1857998 to 2386792 tid 2 2024-09-11T03:47:39.001 INFO:tasks.rados.rados.2.smithi175.stdout:2477: copy_from oid 22 from oid 11 current snap is 269 2024-09-11T03:47:39.001 INFO:tasks.rados.rados.2.smithi175.stdout:2478: rollback oid 37 current snap is 269 2024-09-11T03:47:39.001 INFO:tasks.rados.rados.2.smithi175.stdout:2478: done (2 left) 2024-09-11T03:47:39.001 INFO:tasks.rados.rados.2.smithi175.stdout:2479: read oid 25 snap -1 2024-09-11T03:47:39.001 INFO:tasks.rados.rados.2.smithi175.stdout:2479: expect (ObjNum 803 snap 255 seq_num 803) 2024-09-11T03:47:39.001 INFO:tasks.rados.rados.2.smithi175.stdout:2480: write oid 6 current snap is 269 2024-09-11T03:47:39.001 INFO:tasks.rados.rados.2.smithi175.stdout:2480: seq_num 857 ranges {403599=719929,1916480=441635} 2024-09-11T03:47:39.007 INFO:tasks.rados.rados.2.smithi175.stdout:2480: writing smithi17557870-6 from 403599 to 1123528 tid 1 2024-09-11T03:47:39.014 INFO:tasks.rados.rados.2.smithi175.stdout:2480: writing smithi17557870-6 from 1916480 to 2358115 tid 2 2024-09-11T03:47:39.015 INFO:tasks.rados.rados.2.smithi175.stdout:2481: write (excl) oid 7 current snap is 269 2024-09-11T03:47:39.015 INFO:tasks.rados.rados.2.smithi175.stdout:2481: seq_num 858 ranges {420406=795178,1974962=445444} 2024-09-11T03:47:39.021 INFO:tasks.rados.rados.2.smithi175.stdout:2481: writing smithi17557870-7 from 420406 to 1215584 tid 1 2024-09-11T03:47:39.028 INFO:tasks.rados.rados.2.smithi175.stdout:2481: writing smithi17557870-7 from 1974962 to 2420406 tid 2 2024-09-11T03:47:39.028 INFO:tasks.rados.rados.2.smithi175.stdout:2476: finishing write tid 1 to smithi17557870-18 2024-09-11T03:47:39.028 INFO:tasks.rados.rados.2.smithi175.stdout:2482: read oid 43 snap -1 2024-09-11T03:47:39.028 INFO:tasks.rados.rados.2.smithi175.stdout:2482: expect (ObjNum 848 snap 267 seq_num 848) 2024-09-11T03:47:39.028 INFO:tasks.rados.rados.2.smithi175.stdout:2483: snap_create 2024-09-11T03:47:39.028 INFO:tasks.rados.rados.2.smithi175.stdout:2477: finishing copy_from racing read to smithi17557870-22 2024-09-11T03:47:39.029 INFO:tasks.rados.rados.2.smithi175.stdout:2479: expect (ObjNum 803 snap 255 seq_num 803) 2024-09-11T03:47:39.047 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:39.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:39.068 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:39.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:39.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:39.190 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:39.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:39.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:39.228 INFO:tasks.rados.rados.2.smithi175.stdout:2476: finishing write tid 2 to smithi17557870-18 2024-09-11T03:47:39.228 INFO:tasks.rados.rados.2.smithi175.stdout:2476: finishing write tid 3 to smithi17557870-18 2024-09-11T03:47:39.228 INFO:tasks.rados.rados.2.smithi175.stdout:2476: finishing write tid 4 to smithi17557870-18 2024-09-11T03:47:39.228 INFO:tasks.rados.rados.2.smithi175.stdout:2476: oid 18 updating version 0 to 478 2024-09-11T03:47:39.228 INFO:tasks.rados.rados.2.smithi175.stdout:2476: oid 18 version 478 is already newer than 477 2024-09-11T03:47:39.228 INFO:tasks.rados.rados.2.smithi175.stdout:2476: oid 18 version 478 is already newer than 476 2024-09-11T03:47:39.228 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 478 (ObjNum 856 snap 269 seq_num 856) dirty exists 2024-09-11T03:47:39.228 INFO:tasks.rados.rados.2.smithi175.stdout:2476: left oid 18 (ObjNum 856 snap 269 seq_num 856) 2024-09-11T03:47:39.228 INFO:tasks.rados.rados.2.smithi175.stdout:2477: finishing copy_from to smithi17557870-22 2024-09-11T03:47:39.228 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 603 (ObjNum 845 snap 266 seq_num 845) dirty exists 2024-09-11T03:47:39.228 INFO:tasks.rados.rados.2.smithi175.stdout:2480: finishing write tid 1 to smithi17557870-6 2024-09-11T03:47:39.228 INFO:tasks.rados.rados.2.smithi175.stdout:2480: finishing write tid 2 to smithi17557870-6 2024-09-11T03:47:39.228 INFO:tasks.rados.rados.2.smithi175.stdout:2480: finishing write tid 3 to smithi17557870-6 2024-09-11T03:47:39.229 INFO:tasks.rados.rados.2.smithi175.stdout:2480: finishing write tid 4 to smithi17557870-6 2024-09-11T03:47:39.229 INFO:tasks.rados.rados.2.smithi175.stdout:2480: oid 6 updating version 0 to 884 2024-09-11T03:47:39.229 INFO:tasks.rados.rados.2.smithi175.stdout:2480: oid 6 updating version 884 to 885 2024-09-11T03:47:39.229 INFO:tasks.rados.rados.2.smithi175.stdout:2480: oid 6 version 885 is already newer than 883 2024-09-11T03:47:39.229 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 885 (ObjNum 857 snap 269 seq_num 857) dirty exists 2024-09-11T03:47:39.229 INFO:tasks.rados.rados.2.smithi175.stdout:2480: left oid 6 (ObjNum 857 snap 269 seq_num 857) 2024-09-11T03:47:39.229 INFO:tasks.rados.rados.2.smithi175.stdout:2481: finishing write tid 1 to smithi17557870-7 2024-09-11T03:47:39.229 INFO:tasks.rados.rados.2.smithi175.stdout:2481: finishing write tid 2 to smithi17557870-7 2024-09-11T03:47:39.229 INFO:tasks.rados.rados.2.smithi175.stdout:2481: finishing write tid 3 to smithi17557870-7 2024-09-11T03:47:39.229 INFO:tasks.rados.rados.2.smithi175.stdout:2481: finishing write tid 4 to smithi17557870-7 2024-09-11T03:47:39.229 INFO:tasks.rados.rados.2.smithi175.stdout:2481: oid 7 updating version 0 to 540 2024-09-11T03:47:39.229 INFO:tasks.rados.rados.2.smithi175.stdout:2481: oid 7 version 540 is already newer than 539 2024-09-11T03:47:39.229 INFO:tasks.rados.rados.2.smithi175.stdout:2481: oid 7 updating version 540 to 541 2024-09-11T03:47:39.229 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 541 (ObjNum 858 snap 269 seq_num 858) dirty exists 2024-09-11T03:47:39.230 INFO:tasks.rados.rados.2.smithi175.stdout:2481: left oid 7 (ObjNum 858 snap 269 seq_num 858) 2024-09-11T03:47:39.230 INFO:tasks.rados.rados.2.smithi175.stdout:2482: expect (ObjNum 848 snap 267 seq_num 848) 2024-09-11T03:47:39.292 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:39.314 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:39.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:39.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:39.372 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:47:39.372 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:47:39.416 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:39.437 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:39.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:39.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:39.539 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:39.561 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:39.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:39.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:39.663 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:39.686 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:39.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:39.787 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:39.810 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:39.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:39.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:39.912 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:39.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:39.935 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:39.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:40.031 INFO:tasks.rados.rados.2.smithi175.stdout:2476: done (6 left) 2024-09-11T03:47:40.031 INFO:tasks.rados.rados.2.smithi175.stdout:2477: done (5 left) 2024-09-11T03:47:40.031 INFO:tasks.rados.rados.2.smithi175.stdout:2479: done (4 left) 2024-09-11T03:47:40.031 INFO:tasks.rados.rados.2.smithi175.stdout:2480: done (3 left) 2024-09-11T03:47:40.031 INFO:tasks.rados.rados.2.smithi175.stdout:2481: done (2 left) 2024-09-11T03:47:40.031 INFO:tasks.rados.rados.2.smithi175.stdout:2482: done (1 left) 2024-09-11T03:47:40.031 INFO:tasks.rados.rados.2.smithi175.stdout:2483: done (0 left) 2024-09-11T03:47:40.032 INFO:tasks.rados.rados.2.smithi175.stdout:2484: copy_from oid 48 from oid 11 current snap is 270 2024-09-11T03:47:40.032 INFO:tasks.rados.rados.2.smithi175.stdout:2485: snap_create 2024-09-11T03:47:40.036 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:40.042 INFO:tasks.rados.rados.2.smithi175.stdout:2484: finishing copy_from racing read to smithi17557870-48 2024-09-11T03:47:40.045 INFO:tasks.rados.rados.2.smithi175.stdout:2484: finishing copy_from to smithi17557870-48 2024-09-11T03:47:40.045 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 603 (ObjNum 845 snap 266 seq_num 845) dirty exists 2024-09-11T03:47:40.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T03:47:40.059 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:40.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:40.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:40.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:40.184 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:40.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:40.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:40.280 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:39 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:47:40.280 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:39 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:47:40.280 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:39 smithi175 ceph-mon[30894]: pgmap v1116: 67 pgs: 67 active+clean; 119 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 3.4 MiB/s wr, 27 op/s 2024-09-11T03:47:40.280 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:39 smithi175 ceph-mon[30894]: osdmap e641: 12 total, 12 up, 12 in 2024-09-11T03:47:40.286 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:40.303 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:47:40.310 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:40.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:40.311 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:40.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:39 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:47:40.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:39 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:47:40.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:39 smithi084 ceph-mon[27254]: pgmap v1116: 67 pgs: 67 active+clean; 119 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 3.4 MiB/s wr, 27 op/s 2024-09-11T03:47:40.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:39 smithi084 ceph-mon[27254]: osdmap e641: 12 total, 12 up, 12 in 2024-09-11T03:47:40.412 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:40.413 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:47:40.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:39 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:47:40.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:39 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:47:40.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:39 smithi121 ceph-mon[30729]: pgmap v1116: 67 pgs: 67 active+clean; 119 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 3.4 MiB/s wr, 27 op/s 2024-09-11T03:47:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:39 smithi121 ceph-mon[30729]: osdmap e641: 12 total, 12 up, 12 in 2024-09-11T03:47:40.439 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:40.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:40.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:40.541 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:40.563 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:40.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:40.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:40.665 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:40.682 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:40.688 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:40.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:40.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:40.790 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:40.801 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:47:40.801 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:47:40.811 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:40.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:40.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:40.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:40.936 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:40.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:40.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:40.982 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:47:41.037 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:41.039 INFO:tasks.rados.rados.2.smithi175.stdout:2484: done (1 left) 2024-09-11T03:47:41.039 INFO:tasks.rados.rados.2.smithi175.stdout:2485: done (0 left) 2024-09-11T03:47:41.039 INFO:tasks.rados.rados.2.smithi175.stdout:2486: snap_remove snap 270 2024-09-11T03:47:41.059 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:41.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:41.161 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:41.181 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:47:41.181 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:47:41.182 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:41.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:41.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:41.200 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:47:41.200 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:47:41.218 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:47:41.219 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:47:41.219 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:41 smithi084 ceph-mon[27254]: osdmap e642: 12 total, 12 up, 12 in 2024-09-11T03:47:41.236 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:47:41.237 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:47:41.254 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:47:41.255 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:47:41.264 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:41.272 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:47:41.272 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:47:41.284 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:41.290 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:47:41.290 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:47:41.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:41.306 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:41.306 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:41.307 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:47:41.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:47:41.323 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:47:41.324 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:47:41.340 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:47:41.340 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:47:41.356 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:47:41.357 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:47:41.374 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:47:41.374 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:47:41.402 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:41 smithi121 ceph-mon[30729]: osdmap e642: 12 total, 12 up, 12 in 2024-09-11T03:47:41.407 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:41.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:41.428 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:41.428 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:41.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T03:47:41.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:41 smithi175 ceph-mon[30894]: osdmap e642: 12 total, 12 up, 12 in 2024-09-11T03:47:41.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:41.550 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:41.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:41.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:41.652 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:41.673 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:47:41.674 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:41.676 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:41.777 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:41.800 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:41.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:41.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:41.901 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:47:41.901 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:47:41.901 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T03:47:41.901 INFO:tasks.thrashosds.thrasher:Setting osd 11 primary_affinity to 1.000000 2024-09-11T03:47:41.901 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 11 1 2024-09-11T03:47:41.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:41.924 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:41.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:41.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:41.987 INFO:tasks.rados.rados.2.smithi175.stdout:2486: done (0 left) 2024-09-11T03:47:41.987 INFO:tasks.rados.rados.2.smithi175.stdout:2487: read oid 13 snap 269 2024-09-11T03:47:41.987 INFO:tasks.rados.rados.2.smithi175.stdout:2487: expect (ObjNum 791 snap 252 seq_num 791) 2024-09-11T03:47:41.987 INFO:tasks.rados.rados.2.smithi175.stdout:2488: snap_create 2024-09-11T03:47:41.991 INFO:tasks.rados.rados.2.smithi175.stdout:2487: expect (ObjNum 791 snap 252 seq_num 791) 2024-09-11T03:47:42.026 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:42.047 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:42.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:42.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:42.135 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:47:42.135 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:47:42.135 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:47:42.135 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:47:42.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:42 smithi121 ceph-mon[30729]: pgmap v1119: 67 pgs: 67 active+clean; 119 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 2.1 MiB/s wr, 9 op/s 2024-09-11T03:47:42.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:42 smithi121 ceph-mon[30729]: osdmap e643: 12 total, 12 up, 12 in 2024-09-11T03:47:42.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:42 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3514718677' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:42.136 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:42 smithi121 ceph-mon[30729]: osdmap e644: 12 total, 12 up, 12 in 2024-09-11T03:47:42.150 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:42.171 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:42.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:42.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:42.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:42.294 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:42.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:42.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:42.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:42 smithi084 ceph-mon[27254]: pgmap v1119: 67 pgs: 67 active+clean; 119 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 2.1 MiB/s wr, 9 op/s 2024-09-11T03:47:42.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:42 smithi084 ceph-mon[27254]: osdmap e643: 12 total, 12 up, 12 in 2024-09-11T03:47:42.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:42 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3514718677' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:42.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:42 smithi084 ceph-mon[27254]: osdmap e644: 12 total, 12 up, 12 in 2024-09-11T03:47:42.395 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:42.417 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:42.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:42.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:42 smithi175 ceph-mon[30894]: pgmap v1119: 67 pgs: 67 active+clean; 119 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 2.1 MiB/s wr, 9 op/s 2024-09-11T03:47:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:42 smithi175 ceph-mon[30894]: osdmap e643: 12 total, 12 up, 12 in 2024-09-11T03:47:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:42 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3514718677' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:42.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:42 smithi175 ceph-mon[30894]: osdmap e644: 12 total, 12 up, 12 in 2024-09-11T03:47:42.486 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:42.519 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:42.540 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:42.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:42.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:42.642 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:42.664 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:42.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:42.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:42.739 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T03:47:42.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:42.804 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:42.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:42.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:42.906 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:42.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:42.930 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:42.930 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:42.981 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:47:42.981 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:47:42.986 INFO:tasks.rados.rados.2.smithi175.stdout:2487: done (1 left) 2024-09-11T03:47:42.986 INFO:tasks.rados.rados.2.smithi175.stdout:2488: done (0 left) 2024-09-11T03:47:42.986 INFO:tasks.rados.rados.2.smithi175.stdout:2489: rollback oid 48 current snap is 272 2024-09-11T03:47:42.987 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 48 to 271 2024-09-11T03:47:42.987 INFO:tasks.rados.rados.2.smithi175.stdout:2490: delete oid 45 current snap is 272 2024-09-11T03:47:42.990 INFO:tasks.rados.rados.2.smithi175.stdout:2490: done (1 left) 2024-09-11T03:47:42.990 INFO:tasks.rados.rados.2.smithi175.stdout:2491: write oid 35 current snap is 272 2024-09-11T03:47:42.990 INFO:tasks.rados.rados.2.smithi175.stdout:2491: seq_num 860 ranges {454020=662029,1691926=588962} 2024-09-11T03:47:42.996 INFO:tasks.rados.rados.2.smithi175.stdout:2491: writing smithi17557870-35 from 454020 to 1116049 tid 1 2024-09-11T03:47:43.003 INFO:tasks.rados.rados.2.smithi175.stdout:2491: writing smithi17557870-35 from 1691926 to 2280888 tid 2 2024-09-11T03:47:43.003 INFO:tasks.rados.rados.2.smithi175.stdout:2492: rollback oid 47 current snap is 272 2024-09-11T03:47:43.003 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 47 to 269 2024-09-11T03:47:43.003 INFO:tasks.rados.rados.2.smithi175.stdout:2489: finishing rollback tid 0 to smithi17557870-48 2024-09-11T03:47:43.003 INFO:tasks.rados.rados.2.smithi175.stdout:2489: finishing rollback tid 1 to smithi17557870-48 2024-09-11T03:47:43.003 INFO:tasks.rados.rados.2.smithi175.stdout:2493: rollback oid 40 current snap is 272 2024-09-11T03:47:43.003 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 40 to 271 2024-09-11T03:47:43.003 INFO:tasks.rados.rados.2.smithi175.stdout:2489: finishing rollback tid 2 to smithi17557870-48 2024-09-11T03:47:43.003 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 849 (ObjNum 845 snap 266 seq_num 845) dirty exists 2024-09-11T03:47:43.003 INFO:tasks.rados.rados.2.smithi175.stdout:2491: finishing write tid 1 to smithi17557870-35 2024-09-11T03:47:43.003 INFO:tasks.rados.rados.2.smithi175.stdout:2489: done (3 left) 2024-09-11T03:47:43.003 INFO:tasks.rados.rados.2.smithi175.stdout:2494: snap_create 2024-09-11T03:47:43.005 INFO:tasks.rados.rados.2.smithi175.stdout:2492: finishing rollback tid 1 to smithi17557870-47 2024-09-11T03:47:43.005 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T03:47:43.006 INFO:tasks.rados.rados.2.smithi175.stdout:2493: finishing rollback tid 0 to smithi17557870-40 2024-09-11T03:47:43.006 INFO:tasks.rados.rados.2.smithi175.stdout:2491: finishing write tid 2 to smithi17557870-35 2024-09-11T03:47:43.006 INFO:tasks.rados.rados.2.smithi175.stdout:2493: finishing rollback tid 1 to smithi17557870-40 2024-09-11T03:47:43.006 INFO:tasks.rados.rados.2.smithi175.stdout:2491: finishing write tid 3 to smithi17557870-35 2024-09-11T03:47:43.006 INFO:tasks.rados.rados.2.smithi175.stdout:2493: finishing rollback tid 2 to smithi17557870-40 2024-09-11T03:47:43.006 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 454 (ObjNum 831 snap 263 seq_num 831) dirty exists 2024-09-11T03:47:43.006 INFO:tasks.rados.rados.2.smithi175.stdout:2491: finishing write tid 4 to smithi17557870-35 2024-09-11T03:47:43.006 INFO:tasks.rados.rados.2.smithi175.stdout:2491: oid 35 updating version 0 to 692 2024-09-11T03:47:43.006 INFO:tasks.rados.rados.2.smithi175.stdout:2491: oid 35 version 692 is already newer than 691 2024-09-11T03:47:43.006 INFO:tasks.rados.rados.2.smithi175.stdout:2491: oid 35 updating version 692 to 693 2024-09-11T03:47:43.007 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 693 (ObjNum 860 snap 272 seq_num 860) dirty exists 2024-09-11T03:47:43.007 INFO:tasks.rados.rados.2.smithi175.stdout:2491: left oid 35 (ObjNum 860 snap 272 seq_num 860) 2024-09-11T03:47:43.031 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:43.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:43.053 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:43.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:43.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-09-11T03:47:43.112+0000 7f9935cc1640 0 --2- 172.21.15.84:0/2011531556 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f99301076c0 0x7f9930197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-11T03:47:43.154 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:43.178 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:43.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:43.280 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:43.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:43.302 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:43.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:43.303 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:43 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3281917961' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:47:43.304 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:43 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:47:43.304 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:43 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:47:43.304 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:43 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:43.304 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:43 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3281917961' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:47:43.304 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:43 smithi121 ceph-mon[30729]: osdmap e645: 12 total, 12 up, 12 in 2024-09-11T03:47:43.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:43 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3281917961' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:47:43.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:43 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:47:43.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:43 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:47:43.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:43 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:43.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:43 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3281917961' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:47:43.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:43 smithi084 ceph-mon[27254]: osdmap e645: 12 total, 12 up, 12 in 2024-09-11T03:47:43.403 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:43.425 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:43.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:43.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:43 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3281917961' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:47:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:43 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:47:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:43 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:47:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:43 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:43 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3281917961' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:47:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:43 smithi175 ceph-mon[30894]: osdmap e645: 12 total, 12 up, 12 in 2024-09-11T03:47:43.484 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:47:43.484 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:47:43.484 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:47:43.484 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:47:43.484 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:47:43.527 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:43.548 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:43.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:43.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:43.650 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:43.671 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:43.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:43.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:43.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:43.795 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:43.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:43.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:43.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:43.918 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:43.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:43.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:43.982 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-11T03:47:43.986 INFO:tasks.rados.rados.2.smithi175.stdout:2491: done (3 left) 2024-09-11T03:47:43.986 INFO:tasks.rados.rados.2.smithi175.stdout:2492: done (2 left) 2024-09-11T03:47:43.986 INFO:tasks.rados.rados.2.smithi175.stdout:2493: done (1 left) 2024-09-11T03:47:43.986 INFO:tasks.rados.rados.2.smithi175.stdout:2494: done (0 left) 2024-09-11T03:47:43.986 INFO:tasks.rados.rados.2.smithi175.stdout:2495: read oid 40 snap -1 2024-09-11T03:47:43.986 INFO:tasks.rados.rados.2.smithi175.stdout:2495: expect (ObjNum 831 snap 263 seq_num 831) 2024-09-11T03:47:43.986 INFO:tasks.rados.rados.2.smithi175.stdout:2496: copy_from oid 7 from oid 30 current snap is 273 2024-09-11T03:47:43.987 INFO:tasks.rados.rados.2.smithi175.stdout:2497: snap_remove snap 269 2024-09-11T03:47:44.020 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:44.041 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:44.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:44.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T03:47:44.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:44.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:44 smithi084 ceph-mon[27254]: pgmap v1122: 67 pgs: 67 active+clean; 119 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 8 op/s 2024-09-11T03:47:44.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:44 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2011531556' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:47:44.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:44 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2011531556' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:47:44.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:44 smithi084 ceph-mon[27254]: osdmap e646: 12 total, 12 up, 12 in 2024-09-11T03:47:44.162 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:44.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:44.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:44.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:44.286 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:44.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:44.369 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:47:44.373 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:47:44.388 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:44.411 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:44.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:44.412 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:44.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:44 smithi121 ceph-mon[30729]: pgmap v1122: 67 pgs: 67 active+clean; 119 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 8 op/s 2024-09-11T03:47:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:44 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2011531556' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:47:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:44 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2011531556' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:47:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:44 smithi121 ceph-mon[30729]: osdmap e646: 12 total, 12 up, 12 in 2024-09-11T03:47:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:44 smithi175 ceph-mon[30894]: pgmap v1122: 67 pgs: 67 active+clean; 119 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 8 op/s 2024-09-11T03:47:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:44 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2011531556' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:47:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:44 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2011531556' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:47:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:44 smithi175 ceph-mon[30894]: osdmap e646: 12 total, 12 up, 12 in 2024-09-11T03:47:44.513 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:44.534 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:44.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:44.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:44.636 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:44.661 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:44.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:44.662 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:44.692 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:44.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:44.784 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:44.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:44.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:44.858 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:47:44.858 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:47:44.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:47:44.858 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:47:44.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.859 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:221 2.3 2:c6c49f2c:::smithi17557870-44:head [copy-from ver 9] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:47:44.859 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941456+0000", 2024-09-11T03:47:44.859 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 544.908057945, 2024-09-11T03:47:44.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015312882, 2024-09-11T03:47:44.859 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.859 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.859 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:44.859 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.859 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.860 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.860 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 221 2024-09-11T03:47:44.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.860 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941456+0000", 2024-09-11T03:47:44.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941456+0000", 2024-09-11T03:47:44.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941459+0000", 2024-09-11T03:47:44.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9189999999999999e-06 2024-09-11T03:47:44.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941469+0000", 2024-09-11T03:47:44.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0069e-05 2024-09-11T03:47:44.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941490+0000", 2024-09-11T03:47:44.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0947000000000001e-05 2024-09-11T03:47:44.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941501+0000", 2024-09-11T03:47:44.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0169e-05 2024-09-11T03:47:44.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941585+0000", 2024-09-11T03:47:44.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4022000000000003e-05 2024-09-11T03:47:44.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:47:44.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941590+0000", 2024-09-11T03:47:44.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4929999999999998e-06 2024-09-11T03:47:44.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.942889+0000", 2024-09-11T03:47:44.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001298539 2024-09-11T03:47:44.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.942984+0000", 2024-09-11T03:47:44.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5513999999999994e-05 2024-09-11T03:47:44.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:47:44.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.950816+0000", 2024-09-11T03:47:44.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078315850000000003 2024-09-11T03:47:44.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:44.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956638+0000", 2024-09-11T03:47:44.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058225769999999998 2024-09-11T03:47:44.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:44.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956649+0000", 2024-09-11T03:47:44.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0492000000000001e-05 2024-09-11T03:47:44.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:44.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956690+0000", 2024-09-11T03:47:44.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.142e-05 2024-09-11T03:47:44.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:44.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956728+0000", 2024-09-11T03:47:44.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8195000000000003e-05 2024-09-11T03:47:44.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956769+0000", 2024-09-11T03:47:44.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0941e-05 2024-09-11T03:47:44.867 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.867 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.867 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.868 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:222 2.3 2:c6c49f2c:::smithi17557870-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:47:44.868 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:47:44.868 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 544.90797986600001, 2024-09-11T03:47:44.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015032419, 2024-09-11T03:47:44.868 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.868 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.868 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:47:44.868 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.869 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.869 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.869 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 222 2024-09-11T03:47:44.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.869 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:47:44.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:47:44.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941535+0000", 2024-09-11T03:47:44.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.51e-07 2024-09-11T03:47:44.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941539+0000", 2024-09-11T03:47:44.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0060000000000003e-06 2024-09-11T03:47:44.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941547+0000", 2024-09-11T03:47:44.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9009999999999994e-06 2024-09-11T03:47:44.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941553+0000", 2024-09-11T03:47:44.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1519999999999997e-06 2024-09-11T03:47:44.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941691+0000", 2024-09-11T03:47:44.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013752600000000001 2024-09-11T03:47:44.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:44.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941697+0000", 2024-09-11T03:47:44.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3770000000000001e-06 2024-09-11T03:47:44.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943170+0000", 2024-09-11T03:47:44.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001473504 2024-09-11T03:47:44.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:44.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943225+0000", 2024-09-11T03:47:44.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4101000000000003e-05 2024-09-11T03:47:44.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956388+0000", 2024-09-11T03:47:44.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013163223999999999 2024-09-11T03:47:44.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956456+0000", 2024-09-11T03:47:44.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8563000000000001e-05 2024-09-11T03:47:44.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.956567+0000", 2024-09-11T03:47:44.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011061399999999999 2024-09-11T03:47:44.875 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.875 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.876 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.876 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:223 2.3 2:c3768452:::smithi17557870-11:head [copy-from ver 3] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-11T03:47:44.876 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:47:44.876 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 544.90793982100001, 2024-09-11T03:47:44.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016727301999999999, 2024-09-11T03:47:44.876 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.876 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.876 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:44.877 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.877 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.877 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.877 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 223 2024-09-11T03:47:44.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.877 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:47:44.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:47:44.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941575+0000", 2024-09-11T03:47:44.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0999999999999999e-07 2024-09-11T03:47:44.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941578+0000", 2024-09-11T03:47:44.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.021e-06 2024-09-11T03:47:44.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941583+0000", 2024-09-11T03:47:44.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0520000000000001e-06 2024-09-11T03:47:44.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941586+0000", 2024-09-11T03:47:44.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-09-11T03:47:44.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941750+0000", 2024-09-11T03:47:44.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016353200000000001 2024-09-11T03:47:44.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:44.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.941756+0000", 2024-09-11T03:47:44.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.75e-06 2024-09-11T03:47:44.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943289+0000", 2024-09-11T03:47:44.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001533747 2024-09-11T03:47:44.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.943383+0000", 2024-09-11T03:47:44.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3655999999999999e-05 2024-09-11T03:47:44.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:47:44.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.954899+0000", 2024-09-11T03:47:44.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011515463 2024-09-11T03:47:44.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:44.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.957952+0000", 2024-09-11T03:47:44.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030533560000000001 2024-09-11T03:47:44.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:44.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958200+0000", 2024-09-11T03:47:44.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024759100000000002 2024-09-11T03:47:44.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:44.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958229+0000", 2024-09-11T03:47:44.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9862e-05 2024-09-11T03:47:44.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:44.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958262+0000", 2024-09-11T03:47:44.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2267000000000001e-05 2024-09-11T03:47:44.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:39.958302+0000", 2024-09-11T03:47:44.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0204000000000003e-05 2024-09-11T03:47:44.884 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.884 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.884 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.884 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:343 2.2 2:494ba932:::smithi17557870-43:head [copy-from ver 6] snapc 8=[8] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-11T03:47:44.884 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:49.820776+0000", 2024-09-11T03:47:44.884 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 535.02873867899996, 2024-09-11T03:47:44.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017461323000000001, 2024-09-11T03:47:44.885 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.885 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.885 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:44.885 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.885 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.885 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.885 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 343 2024-09-11T03:47:44.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.885 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820776+0000", 2024-09-11T03:47:44.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820776+0000", 2024-09-11T03:47:44.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820779+0000", 2024-09-11T03:47:44.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3000000000000002e-06 2024-09-11T03:47:44.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820797+0000", 2024-09-11T03:47:44.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7730000000000001e-05 2024-09-11T03:47:44.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820853+0000", 2024-09-11T03:47:44.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5909000000000003e-05 2024-09-11T03:47:44.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820872+0000", 2024-09-11T03:47:44.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9709999999999999e-05 2024-09-11T03:47:44.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820976+0000", 2024-09-11T03:47:44.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010374099999999999 2024-09-11T03:47:44.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:47:44.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.820982+0000", 2024-09-11T03:47:44.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4389999999999999e-06 2024-09-11T03:47:44.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825547+0000", 2024-09-11T03:47:44.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045658260000000003 2024-09-11T03:47:44.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.825626+0000", 2024-09-11T03:47:44.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8289999999999998e-05 2024-09-11T03:47:44.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,6", 2024-09-11T03:47:44.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.830895+0000", 2024-09-11T03:47:44.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052695060000000002 2024-09-11T03:47:44.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:44.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.835040+0000", 2024-09-11T03:47:44.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041446549999999997 2024-09-11T03:47:44.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:44.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.835364+0000", 2024-09-11T03:47:44.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032384200000000002 2024-09-11T03:47:44.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:44.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838162+0000", 2024-09-11T03:47:44.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027987509999999999 2024-09-11T03:47:44.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:44.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838203+0000", 2024-09-11T03:47:44.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0209000000000001e-05 2024-09-11T03:47:44.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:49.838237+0000", 2024-09-11T03:47:44.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4415000000000002e-05 2024-09-11T03:47:44.892 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.892 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.892 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.893 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1068 2.3 2:c3768452:::smithi17557870-11:head [write 507611~777843,stat] snapc 28=[28,27,26] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:47:44.893 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.871196+0000", 2024-09-11T03:47:44.893 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 472.97831863099998, 2024-09-11T03:47:44.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.036362196999999999, 2024-09-11T03:47:44.893 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.893 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.893 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:44.893 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.893 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.893 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.893 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1068 2024-09-11T03:47:44.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.894 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.871196+0000", 2024-09-11T03:47:44.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.871196+0000", 2024-09-11T03:47:44.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.871198+0000", 2024-09-11T03:47:44.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5000000000000002e-06 2024-09-11T03:47:44.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873403+0000", 2024-09-11T03:47:44.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022048739999999999 2024-09-11T03:47:44.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873438+0000", 2024-09-11T03:47:44.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4518999999999999e-05 2024-09-11T03:47:44.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873453+0000", 2024-09-11T03:47:44.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5733999999999999e-05 2024-09-11T03:47:44.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873597+0000", 2024-09-11T03:47:44.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143702 2024-09-11T03:47:44.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873668+0000", 2024-09-11T03:47:44.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0996999999999998e-05 2024-09-11T03:47:44.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:47:44.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.873819+0000", 2024-09-11T03:47:44.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015105300000000001 2024-09-11T03:47:44.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:44.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.875607+0000", 2024-09-11T03:47:44.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017877419999999999 2024-09-11T03:47:44.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:44.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.875920+0000", 2024-09-11T03:47:44.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031312799999999997 2024-09-11T03:47:44.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:44.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907471+0000", 2024-09-11T03:47:44.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.031551348 2024-09-11T03:47:44.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:44.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907514+0000", 2024-09-11T03:47:44.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2873000000000001e-05 2024-09-11T03:47:44.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907558+0000", 2024-09-11T03:47:44.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3727000000000003e-05 2024-09-11T03:47:44.900 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.900 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.900 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.900 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1069 2.3 2:c3768452:::smithi17557870-11:head [write 2073973~444721] snapc 28=[28,27,26] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:47:44.900 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.877745+0000", 2024-09-11T03:47:44.901 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 472.97176915400001, 2024-09-11T03:47:44.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029936307999999998, 2024-09-11T03:47:44.901 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.901 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.901 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:44.901 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.901 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.901 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.901 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1069 2024-09-11T03:47:44.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.901 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.877745+0000", 2024-09-11T03:47:44.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.877745+0000", 2024-09-11T03:47:44.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.877747+0000", 2024-09-11T03:47:44.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.61e-06 2024-09-11T03:47:44.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878169+0000", 2024-09-11T03:47:44.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00042167499999999999 2024-09-11T03:47:44.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878181+0000", 2024-09-11T03:47:44.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2714999999999999e-05 2024-09-11T03:47:44.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878187+0000", 2024-09-11T03:47:44.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2389999999999999e-06 2024-09-11T03:47:44.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878284+0000", 2024-09-11T03:47:44.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6632000000000006e-05 2024-09-11T03:47:44.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878363+0000", 2024-09-11T03:47:44.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9369000000000005e-05 2024-09-11T03:47:44.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:47:44.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878472+0000", 2024-09-11T03:47:44.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108995 2024-09-11T03:47:44.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:44.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879469+0000", 2024-09-11T03:47:44.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00099629100000000006 2024-09-11T03:47:44.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:44.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879835+0000", 2024-09-11T03:47:44.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036609700000000002 2024-09-11T03:47:44.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:44.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907609+0000", 2024-09-11T03:47:44.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027773923999999998 2024-09-11T03:47:44.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:44.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907647+0000", 2024-09-11T03:47:44.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8124000000000003e-05 2024-09-11T03:47:44.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907682+0000", 2024-09-11T03:47:44.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4637000000000001e-05 2024-09-11T03:47:44.907 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.908 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.908 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.908 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1070 2.3 2:c3768452:::smithi17557870-11:head [setxattr _header (57) in=7b,truncate 2907611] snapc 28=[28,27,26] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:47:44.908 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.878201+0000", 2024-09-11T03:47:44.908 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 472.97131335799997, 2024-09-11T03:47:44.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029581774000000002, 2024-09-11T03:47:44.908 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.908 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.908 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:44.908 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.908 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.909 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.909 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1070 2024-09-11T03:47:44.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.909 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878201+0000", 2024-09-11T03:47:44.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878201+0000", 2024-09-11T03:47:44.910 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:44.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878201+0000", 2024-09-11T03:47:44.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4299999999999998e-07 2024-09-11T03:47:44.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878222+0000", 2024-09-11T03:47:44.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0591999999999999e-05 2024-09-11T03:47:44.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878228+0000", 2024-09-11T03:47:44.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.857e-06 2024-09-11T03:47:44.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878232+0000", 2024-09-11T03:47:44.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6459999999999999e-06 2024-09-11T03:47:44.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879123+0000", 2024-09-11T03:47:44.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089174700000000003 2024-09-11T03:47:44.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879183+0000", 2024-09-11T03:47:44.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9502000000000001e-05 2024-09-11T03:47:44.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:47:44.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879287+0000", 2024-09-11T03:47:44.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103952 2024-09-11T03:47:44.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:44.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879606+0000", 2024-09-11T03:47:44.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031872600000000002 2024-09-11T03:47:44.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:44.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879933+0000", 2024-09-11T03:47:44.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032754700000000003 2024-09-11T03:47:44.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:44.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907711+0000", 2024-09-11T03:47:44.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027778265999999999 2024-09-11T03:47:44.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:44.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907746+0000", 2024-09-11T03:47:44.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4403999999999998e-05 2024-09-11T03:47:44.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907783+0000", 2024-09-11T03:47:44.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7092000000000003e-05 2024-09-11T03:47:44.918 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.918 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.918 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.918 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1071 2.3 2:c3768452:::smithi17557870-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e175)", 2024-09-11T03:47:44.919 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:51.878273+0000", 2024-09-11T03:47:44.919 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 472.97124170299998, 2024-09-11T03:47:44.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.02966705, 2024-09-11T03:47:44.919 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.919 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.919 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:47:44.919 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.919 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.919 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.919 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1071 2024-09-11T03:47:44.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.920 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878273+0000", 2024-09-11T03:47:44.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878273+0000", 2024-09-11T03:47:44.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878273+0000", 2024-09-11T03:47:44.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1600000000000002e-07 2024-09-11T03:47:44.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878275+0000", 2024-09-11T03:47:44.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0949999999999998e-06 2024-09-11T03:47:44.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878281+0000", 2024-09-11T03:47:44.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6550000000000001e-06 2024-09-11T03:47:44.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.878284+0000", 2024-09-11T03:47:44.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5370000000000002e-06 2024-09-11T03:47:44.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879510+0000", 2024-09-11T03:47:44.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001225638 2024-09-11T03:47:44.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:47:44.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.879586+0000", 2024-09-11T03:47:44.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6058999999999998e-05 2024-09-11T03:47:44.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907811+0000", 2024-09-11T03:47:44.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028224775000000001 2024-09-11T03:47:44.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907873+0000", 2024-09-11T03:47:44.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2310000000000005e-05 2024-09-11T03:47:44.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:51.907940+0000", 2024-09-11T03:47:44.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6564999999999997e-05 2024-09-11T03:47:44.925 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.925 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.925 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.925 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1555 2.3 2:c6c49f2c:::smithi17557870-44:head [copy-from ver 82] snapc 3c=[3c,3a,38,37] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-11T03:47:44.925 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:30.880293+0000", 2024-09-11T03:47:44.925 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 433.96922167999998, 2024-09-11T03:47:44.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020483438999999999, 2024-09-11T03:47:44.925 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.926 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.926 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:44.926 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.926 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.926 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.926 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1555 2024-09-11T03:47:44.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.926 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880293+0000", 2024-09-11T03:47:44.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880293+0000", 2024-09-11T03:47:44.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880295+0000", 2024-09-11T03:47:44.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2409999999999998e-06 2024-09-11T03:47:44.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880305+0000", 2024-09-11T03:47:44.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5999999999999996e-06 2024-09-11T03:47:44.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880326+0000", 2024-09-11T03:47:44.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1248999999999999e-05 2024-09-11T03:47:44.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880335+0000", 2024-09-11T03:47:44.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4369999999999996e-06 2024-09-11T03:47:44.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880414+0000", 2024-09-11T03:47:44.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8560999999999995e-05 2024-09-11T03:47:44.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:47:44.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880422+0000", 2024-09-11T03:47:44.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.712e-06 2024-09-11T03:47:44.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887559+0000", 2024-09-11T03:47:44.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071376920000000002 2024-09-11T03:47:44.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887625+0000", 2024-09-11T03:47:44.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6256999999999999e-05 2024-09-11T03:47:44.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:47:44.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.893371+0000", 2024-09-11T03:47:44.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0057455579999999996 2024-09-11T03:47:44.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:44.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.899577+0000", 2024-09-11T03:47:44.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062060600000000002 2024-09-11T03:47:44.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:44.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900444+0000", 2024-09-11T03:47:44.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000867307 2024-09-11T03:47:44.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:44.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900703+0000", 2024-09-11T03:47:44.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00025901299999999999 2024-09-11T03:47:44.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:44.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900743+0000", 2024-09-11T03:47:44.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9471000000000002e-05 2024-09-11T03:47:44.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900776+0000", 2024-09-11T03:47:44.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3281000000000002e-05 2024-09-11T03:47:44.934 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.934 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.934 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.934 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1556 2.3 2:c6c49f2c:::smithi17557870-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e214)", 2024-09-11T03:47:44.934 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:30.880359+0000", 2024-09-11T03:47:44.934 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 433.96915498300001, 2024-09-11T03:47:44.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017105065999999999, 2024-09-11T03:47:44.935 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.935 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.935 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:47:44.935 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.935 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.935 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.935 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1556 2024-09-11T03:47:44.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.935 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880359+0000", 2024-09-11T03:47:44.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880359+0000", 2024-09-11T03:47:44.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880360+0000", 2024-09-11T03:47:44.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.75e-07 2024-09-11T03:47:44.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880362+0000", 2024-09-11T03:47:44.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4959999999999999e-06 2024-09-11T03:47:44.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880370+0000", 2024-09-11T03:47:44.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.153e-06 2024-09-11T03:47:44.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880376+0000", 2024-09-11T03:47:44.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0900000000000001e-06 2024-09-11T03:47:44.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880478+0000", 2024-09-11T03:47:44.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102482 2024-09-11T03:47:44.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:44.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880487+0000", 2024-09-11T03:47:44.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8109999999999992e-06 2024-09-11T03:47:44.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887725+0000", 2024-09-11T03:47:44.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072383930000000001 2024-09-11T03:47:44.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:44.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887757+0000", 2024-09-11T03:47:44.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1692000000000001e-05 2024-09-11T03:47:44.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.897328+0000", 2024-09-11T03:47:44.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095712149999999992 2024-09-11T03:47:44.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.897393+0000", 2024-09-11T03:47:44.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5005000000000005e-05 2024-09-11T03:47:44.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.897464+0000", 2024-09-11T03:47:44.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1254000000000006e-05 2024-09-11T03:47:44.941 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.941 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.942 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.942 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3040 2.d 2:b6a25a79:::smithi17557870-47:head [read 0~2201783,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:47:44.942 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:47:44.942 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 292.95253475599998, 2024-09-11T03:47:44.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020060432, 2024-09-11T03:47:44.942 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.942 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.943 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:47:44.943 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.943 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.943 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.943 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3040 2024-09-11T03:47:44.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.943 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:47:44.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:47:44.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896981+0000", 2024-09-11T03:47:44.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.747e-06 2024-09-11T03:47:44.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896989+0000", 2024-09-11T03:47:44.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9559999999999997e-06 2024-09-11T03:47:44.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897005+0000", 2024-09-11T03:47:44.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5704000000000001e-05 2024-09-11T03:47:44.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897013+0000", 2024-09-11T03:47:44.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.028e-06 2024-09-11T03:47:44.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897078+0000", 2024-09-11T03:47:44.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4813000000000004e-05 2024-09-11T03:47:44.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:47:44.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897083+0000", 2024-09-11T03:47:44.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1109999999999999e-06 2024-09-11T03:47:44.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.914637+0000", 2024-09-11T03:47:44.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017553701000000001 2024-09-11T03:47:44.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.914744+0000", 2024-09-11T03:47:44.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107131 2024-09-11T03:47:44.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917040+0000", 2024-09-11T03:47:44.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022962410000000001 2024-09-11T03:47:44.948 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.948 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.949 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.949 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3041 2.d 2:b6a25a79:::smithi17557870-47:head [sparse-read 0~1480887] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:47:44.949 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:47:44.949 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 292.9524692, 2024-09-11T03:47:44.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020163026000000001, 2024-09-11T03:47:44.949 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.949 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.949 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:47:44.949 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.950 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.950 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.950 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3041 2024-09-11T03:47:44.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.950 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:47:44.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:47:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897046+0000", 2024-09-11T03:47:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7e-07 2024-09-11T03:47:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897048+0000", 2024-09-11T03:47:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0700000000000001e-06 2024-09-11T03:47:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897058+0000", 2024-09-11T03:47:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0151e-05 2024-09-11T03:47:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897068+0000", 2024-09-11T03:47:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0329000000000001e-05 2024-09-11T03:47:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897128+0000", 2024-09-11T03:47:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0208999999999999e-05 2024-09-11T03:47:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:44.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897132+0000", 2024-09-11T03:47:44.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7290000000000002e-06 2024-09-11T03:47:44.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917070+0000", 2024-09-11T03:47:44.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019937951999999998 2024-09-11T03:47:44.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917123+0000", 2024-09-11T03:47:44.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3149000000000002e-05 2024-09-11T03:47:44.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917208+0000", 2024-09-11T03:47:44.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5067000000000004e-05 2024-09-11T03:47:44.955 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.955 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.955 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.955 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3042 2.d 2:b6a25a79:::smithi17557870-47:head [sparse-read 0~1480887] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:47:44.956 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:47:44.956 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 292.95243502099999, 2024-09-11T03:47:44.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020302042999999999, 2024-09-11T03:47:44.956 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.956 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.956 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:47:44.956 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.956 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.956 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.956 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3042 2024-09-11T03:47:44.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.957 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:47:44.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:47:44.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897080+0000", 2024-09-11T03:47:44.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.82e-07 2024-09-11T03:47:44.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897082+0000", 2024-09-11T03:47:44.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.418e-06 2024-09-11T03:47:44.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897089+0000", 2024-09-11T03:47:44.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8090000000000004e-06 2024-09-11T03:47:44.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897094+0000", 2024-09-11T03:47:44.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2399999999999998e-06 2024-09-11T03:47:44.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:47:44.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0928999999999999e-05 2024-09-11T03:47:44.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:44.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897158+0000", 2024-09-11T03:47:44.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9890000000000001e-06 2024-09-11T03:47:44.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917227+0000", 2024-09-11T03:47:44.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020069019 2024-09-11T03:47:44.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917290+0000", 2024-09-11T03:47:44.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2848999999999994e-05 2024-09-11T03:47:44.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917381+0000", 2024-09-11T03:47:44.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1308000000000001e-05 2024-09-11T03:47:44.962 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.962 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.962 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.962 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3046 2.d 2:b6874adf:::smithi17557870-17:head [sparse-read 0~1769472,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:47:44.962 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:47:44.962 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 292.95241306000003, 2024-09-11T03:47:44.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021672665000000001, 2024-09-11T03:47:44.962 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.963 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.963 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:47:44.963 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.963 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.963 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.963 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3046 2024-09-11T03:47:44.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.963 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:47:44.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:47:44.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897102+0000", 2024-09-11T03:47:44.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.65e-07 2024-09-11T03:47:44.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897105+0000", 2024-09-11T03:47:44.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1820000000000002e-06 2024-09-11T03:47:44.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897111+0000", 2024-09-11T03:47:44.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4330000000000001e-06 2024-09-11T03:47:44.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897117+0000", 2024-09-11T03:47:44.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2979999999999999e-06 2024-09-11T03:47:44.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897179+0000", 2024-09-11T03:47:44.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2825999999999998e-05 2024-09-11T03:47:44.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:44.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897182+0000", 2024-09-11T03:47:44.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7769999999999998e-06 2024-09-11T03:47:44.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917399+0000", 2024-09-11T03:47:44.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020216622 2024-09-11T03:47:44.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917444+0000", 2024-09-11T03:47:44.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5558e-05 2024-09-11T03:47:44.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918774+0000", 2024-09-11T03:47:44.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013296040000000001 2024-09-11T03:47:44.968 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.968 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.968 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.969 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3047 2.d 2:b6874adf:::smithi17557870-17:head [read 0~2453888,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:47:44.969 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:47:44.969 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 292.95238388799999, 2024-09-11T03:47:44.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022691901, 2024-09-11T03:47:44.969 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.969 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.969 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:47:44.969 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.969 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.969 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.970 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3047 2024-09-11T03:47:44.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.970 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:47:44.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:47:44.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:47:44.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8599999999999998e-07 2024-09-11T03:47:44.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897134+0000", 2024-09-11T03:47:44.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1470000000000001e-06 2024-09-11T03:47:44.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897141+0000", 2024-09-11T03:47:44.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8029999999999999e-06 2024-09-11T03:47:44.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897147+0000", 2024-09-11T03:47:44.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4520000000000002e-06 2024-09-11T03:47:44.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897202+0000", 2024-09-11T03:47:44.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4255999999999997e-05 2024-09-11T03:47:44.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:44.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897204+0000", 2024-09-11T03:47:44.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6649999999999999e-06 2024-09-11T03:47:44.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918807+0000", 2024-09-11T03:47:44.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021602962999999999 2024-09-11T03:47:44.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918887+0000", 2024-09-11T03:47:44.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9550999999999998e-05 2024-09-11T03:47:44.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919822+0000", 2024-09-11T03:47:44.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093557799999999995 2024-09-11T03:47:44.975 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.975 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.976 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.976 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3048 2.d 2:b6874adf:::smithi17557870-17:head [read 0~2453888,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:47:44.976 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:47:44.976 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 292.95235906400001, 2024-09-11T03:47:44.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.023274353000000001, 2024-09-11T03:47:44.976 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.976 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.977 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:47:44.977 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.977 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.977 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.977 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3048 2024-09-11T03:47:44.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.977 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:47:44.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:47:44.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897156+0000", 2024-09-11T03:47:44.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1199999999999998e-07 2024-09-11T03:47:44.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897159+0000", 2024-09-11T03:47:44.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7439999999999999e-06 2024-09-11T03:47:44.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897165+0000", 2024-09-11T03:47:44.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6749999999999996e-06 2024-09-11T03:47:44.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897170+0000", 2024-09-11T03:47:44.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9420000000000003e-06 2024-09-11T03:47:44.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897226+0000", 2024-09-11T03:47:44.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6026999999999997e-05 2024-09-11T03:47:44.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:44.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897232+0000", 2024-09-11T03:47:44.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2609999999999999e-06 2024-09-11T03:47:44.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919841+0000", 2024-09-11T03:47:44.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022609054999999999 2024-09-11T03:47:44.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919885+0000", 2024-09-11T03:47:44.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3383999999999997e-05 2024-09-11T03:47:44.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.920430+0000", 2024-09-11T03:47:44.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054485299999999998 2024-09-11T03:47:44.983 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.983 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.983 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.983 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4176 2.3 2:c6c49f2c:::smithi17557870-44:head [copy-from ver 412] snapc b5=[b5,b4,b2,b1,af,ae,ac,ab,a9,a8,9d] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-11T03:47:44.983 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:28.933487+0000", 2024-09-11T03:47:44.983 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 195.91602785699999, 2024-09-11T03:47:44.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018627511999999999, 2024-09-11T03:47:44.983 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.983 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.984 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:44.984 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.984 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.984 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.984 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4176 2024-09-11T03:47:44.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.984 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933487+0000", 2024-09-11T03:47:44.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933487+0000", 2024-09-11T03:47:44.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933489+0000", 2024-09-11T03:47:44.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2510000000000001e-06 2024-09-11T03:47:44.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933498+0000", 2024-09-11T03:47:44.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3219999999999999e-06 2024-09-11T03:47:44.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933516+0000", 2024-09-11T03:47:44.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7971000000000001e-05 2024-09-11T03:47:44.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933529+0000", 2024-09-11T03:47:44.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2476e-05 2024-09-11T03:47:44.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933657+0000", 2024-09-11T03:47:44.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000128842 2024-09-11T03:47:44.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:47:44.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933663+0000", 2024-09-11T03:47:44.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9580000000000004e-06 2024-09-11T03:47:44.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935429+0000", 2024-09-11T03:47:44.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001765146 2024-09-11T03:47:44.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935549+0000", 2024-09-11T03:47:44.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120504 2024-09-11T03:47:44.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:47:44.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.944235+0000", 2024-09-11T03:47:44.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086856499999999996 2024-09-11T03:47:44.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:44.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948823+0000", 2024-09-11T03:47:44.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045879570000000001 2024-09-11T03:47:44.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:44.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.951466+0000", 2024-09-11T03:47:44.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026434739999999998 2024-09-11T03:47:44.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:44.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.952026+0000", 2024-09-11T03:47:44.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055985599999999996 2024-09-11T03:47:44.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:44.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.952068+0000", 2024-09-11T03:47:44.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1918000000000001e-05 2024-09-11T03:47:44.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.952114+0000", 2024-09-11T03:47:44.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6186999999999997e-05 2024-09-11T03:47:44.991 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.991 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.992 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.992 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4177 2.3 2:c6c49f2c:::smithi17557870-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e451)", 2024-09-11T03:47:44.992 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:28.933561+0000", 2024-09-11T03:47:44.992 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 195.91595304399999, 2024-09-11T03:47:44.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014798686, 2024-09-11T03:47:44.992 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:44.992 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:44.992 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:47:44.992 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:44.993 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:44.993 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:44.993 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4177 2024-09-11T03:47:44.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.993 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:44.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:44.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933561+0000", 2024-09-11T03:47:44.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:44.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933561+0000", 2024-09-11T03:47:44.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:44.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:44.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933562+0000", 2024-09-11T03:47:44.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4600000000000005e-07 2024-09-11T03:47:44.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:44.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933565+0000", 2024-09-11T03:47:44.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8729999999999999e-06 2024-09-11T03:47:44.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:44.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933572+0000", 2024-09-11T03:47:44.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6090000000000003e-06 2024-09-11T03:47:44.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:44.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933579+0000", 2024-09-11T03:47:44.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4339999999999997e-06 2024-09-11T03:47:44.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933717+0000", 2024-09-11T03:47:44.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000137995 2024-09-11T03:47:44.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:44.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933721+0000", 2024-09-11T03:47:44.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6940000000000001e-06 2024-09-11T03:47:44.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935773+0000", 2024-09-11T03:47:44.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002052398 2024-09-11T03:47:44.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:44.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935834+0000", 2024-09-11T03:47:44.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1146e-05 2024-09-11T03:47:44.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:44.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948174+0000", 2024-09-11T03:47:44.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012339786 2024-09-11T03:47:44.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:44.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948263+0000", 2024-09-11T03:47:44.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8838999999999994e-05 2024-09-11T03:47:44.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:44.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948360+0000", 2024-09-11T03:47:44.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7366000000000003e-05 2024-09-11T03:47:44.999 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.999 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:44.999 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:44.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:44.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:44.999 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:6215 2.16 2:6dbd1e81:::smithi17557870-7:head [copy-from ver 268] snapc 112=[112,111,10f] ondisk+write+known_if_redirected+supports_pool_eio e646)", 2024-09-11T03:47:44.999 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:43.987602+0000", 2024-09-11T03:47:44.999 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.86191265399999994, 2024-09-11T03:47:45.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021363759, 2024-09-11T03:47:45.000 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:45.000 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:45.000 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:45.000 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:45.000 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:45.000 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:45.000 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 6215 2024-09-11T03:47:45.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.000 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:45.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:45.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987602+0000", 2024-09-11T03:47:45.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:45.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:45.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987602+0000", 2024-09-11T03:47:45.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:45.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:45.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987603+0000", 2024-09-11T03:47:45.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6339999999999999e-06 2024-09-11T03:47:45.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:45.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987612+0000", 2024-09-11T03:47:45.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1170000000000004e-06 2024-09-11T03:47:45.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:45.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987631+0000", 2024-09-11T03:47:45.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9245e-05 2024-09-11T03:47:45.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:45.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987641+0000", 2024-09-11T03:47:45.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8570000000000007e-06 2024-09-11T03:47:45.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:45.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987718+0000", 2024-09-11T03:47:45.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7812999999999995e-05 2024-09-11T03:47:45.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:47:45.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987726+0000", 2024-09-11T03:47:45.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1960000000000002e-06 2024-09-11T03:47:45.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:45.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.989804+0000", 2024-09-11T03:47:45.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002078004 2024-09-11T03:47:45.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:45.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.989909+0000", 2024-09-11T03:47:45.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105518 2024-09-11T03:47:45.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6,11", 2024-09-11T03:47:45.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.999305+0000", 2024-09-11T03:47:45.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.009396349 2024-09-11T03:47:45.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:45.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.007386+0000", 2024-09-11T03:47:45.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080807069999999995 2024-09-11T03:47:45.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:45.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.007696+0000", 2024-09-11T03:47:45.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030955000000000002 2024-09-11T03:47:45.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:45.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.008885+0000", 2024-09-11T03:47:45.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011892700000000001 2024-09-11T03:47:45.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:45.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.008929+0000", 2024-09-11T03:47:45.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4322e-05 2024-09-11T03:47:45.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:45.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.008966+0000", 2024-09-11T03:47:45.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6177000000000003e-05 2024-09-11T03:47:45.008 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:45.008 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:45.008 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:45.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.008 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:6216 2.16 2:6dbd1e81:::smithi17557870-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e646)", 2024-09-11T03:47:45.008 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:43.987704+0000", 2024-09-11T03:47:45.008 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.86181063199999997, 2024-09-11T03:47:45.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016255282999999999, 2024-09-11T03:47:45.009 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:45.009 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:45.009 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:47:45.009 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:45.009 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:45.009 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:45.009 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 6216 2024-09-11T03:47:45.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.009 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:45.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:45.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987704+0000", 2024-09-11T03:47:45.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:45.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:45.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987704+0000", 2024-09-11T03:47:45.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:45.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:45.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987704+0000", 2024-09-11T03:47:45.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5000000000000002e-07 2024-09-11T03:47:45.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:45.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987708+0000", 2024-09-11T03:47:45.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6610000000000002e-06 2024-09-11T03:47:45.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:45.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987718+0000", 2024-09-11T03:47:45.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7690000000000008e-06 2024-09-11T03:47:45.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:45.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987725+0000", 2024-09-11T03:47:45.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7279999999999998e-06 2024-09-11T03:47:45.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:45.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987785+0000", 2024-09-11T03:47:45.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0163e-05 2024-09-11T03:47:45.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:45.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987790+0000", 2024-09-11T03:47:45.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2410000000000003e-06 2024-09-11T03:47:45.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:45.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.990032+0000", 2024-09-11T03:47:45.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022420880000000002 2024-09-11T03:47:45.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:45.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.990058+0000", 2024-09-11T03:47:45.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6044999999999999e-05 2024-09-11T03:47:45.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:45.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.003794+0000", 2024-09-11T03:47:45.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01373564 2024-09-11T03:47:45.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:45.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.003878+0000", 2024-09-11T03:47:45.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3954000000000003e-05 2024-09-11T03:47:45.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:45.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:45.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:45.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.003959+0000", 2024-09-11T03:47:45.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1343999999999999e-05 2024-09-11T03:47:45.016 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:45.016 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:45.016 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:45.016 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:45.016 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:45.016 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:47:45.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:45.018 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:45.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:45.019 INFO:tasks.rados.rados.2.smithi175.stdout:2497: done (2 left) 2024-09-11T03:47:45.019 INFO:tasks.rados.rados.2.smithi175.stdout:2498: write oid 13 current snap is 273 2024-09-11T03:47:45.019 INFO:tasks.rados.rados.2.smithi175.stdout:2498: seq_num 861 ranges {470827=737278,1750408=720419} 2024-09-11T03:47:45.019 INFO:tasks.rados.rados.2.smithi175.stdout:2498: writing smithi17557870-13 from 470827 to 1208105 tid 1 2024-09-11T03:47:45.022 INFO:tasks.rados.rados.2.smithi175.stdout:2498: writing smithi17557870-13 from 1750408 to 2470827 tid 2 2024-09-11T03:47:45.023 INFO:tasks.rados.rados.2.smithi175.stdout:2499: write oid 28 current snap is 273 2024-09-11T03:47:45.023 INFO:tasks.rados.rados.2.smithi175.stdout:2499: seq_num 862 ranges {487634=412527,1408890=592631} 2024-09-11T03:47:45.027 INFO:tasks.rados.rados.2.smithi175.stdout:2499: writing smithi17557870-28 from 487634 to 900161 tid 1 2024-09-11T03:47:45.033 INFO:tasks.rados.rados.2.smithi175.stdout:2499: writing smithi17557870-28 from 1408890 to 2001521 tid 2 2024-09-11T03:47:45.034 INFO:tasks.rados.rados.2.smithi175.stdout:2500: delete oid 50 current snap is 273 2024-09-11T03:47:45.034 INFO:tasks.rados.rados.2.smithi175.stdout:2495: expect (ObjNum 831 snap 263 seq_num 831) 2024-09-11T03:47:45.119 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:45.141 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:45.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:45.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:45.148 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:47:45.149 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:47:45.165 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:47:45.165 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:47:45.167 INFO:tasks.rados.rados.2.smithi175.stdout:2496: finishing copy_from racing read to smithi17557870-7 2024-09-11T03:47:45.167 INFO:tasks.rados.rados.2.smithi175.stdout:2496: finishing copy_from to smithi17557870-7 2024-09-11T03:47:45.167 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 268 (ObjNum 800 snap 255 seq_num 800) dirty exists 2024-09-11T03:47:45.167 INFO:tasks.rados.rados.2.smithi175.stdout:2495: done (4 left) 2024-09-11T03:47:45.167 INFO:tasks.rados.rados.2.smithi175.stdout:2496: done (3 left) 2024-09-11T03:47:45.167 INFO:tasks.rados.rados.2.smithi175.stdout:2500: done (2 left) 2024-09-11T03:47:45.167 INFO:tasks.rados.rados.2.smithi175.stdout:2501: snap_remove snap 271 2024-09-11T03:47:45.181 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:47:45.181 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:47:45.197 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:47:45.197 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:47:45.197 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:45 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:47:45.197 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:45 smithi084 ceph-mon[27254]: 2.d scrub ok 2024-09-11T03:47:45.198 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:45 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:47:45.198 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:45 smithi084 ceph-mon[27254]: 2.8 scrub ok 2024-09-11T03:47:45.198 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:45 smithi084 ceph-mon[27254]: 1.19 scrub starts 2024-09-11T03:47:45.198 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:45 smithi084 ceph-mon[27254]: 1.19 scrub ok 2024-09-11T03:47:45.198 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:45 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:47:45.198 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:45 smithi084 ceph-mon[27254]: 2.14 scrub ok 2024-09-11T03:47:45.198 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:45 smithi084 ceph-mon[27254]: osdmap e647: 12 total, 12 up, 12 in 2024-09-11T03:47:45.214 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:47:45.214 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:47:45.231 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:47:45.231 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:47:45.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:45.249 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:47:45.249 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:47:45.264 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:47:45.264 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:47:45.269 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:45.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:45.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:45.281 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:47:45.281 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:47:45.298 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:47:45.298 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:47:45.314 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:47:45.314 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:47:45.331 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:47:45.331 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:47:45.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:45.392 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:45.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:45.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:45.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:45 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:47:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:45 smithi121 ceph-mon[30729]: 2.d scrub ok 2024-09-11T03:47:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:45 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:47:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:45 smithi121 ceph-mon[30729]: 2.8 scrub ok 2024-09-11T03:47:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:45 smithi121 ceph-mon[30729]: 1.19 scrub starts 2024-09-11T03:47:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:45 smithi121 ceph-mon[30729]: 1.19 scrub ok 2024-09-11T03:47:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:45 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:47:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:45 smithi121 ceph-mon[30729]: 2.14 scrub ok 2024-09-11T03:47:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:45 smithi121 ceph-mon[30729]: osdmap e647: 12 total, 12 up, 12 in 2024-09-11T03:47:45.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:45 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:47:45.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:45 smithi175 ceph-mon[30894]: 2.d scrub ok 2024-09-11T03:47:45.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:45 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:47:45.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:45 smithi175 ceph-mon[30894]: 2.8 scrub ok 2024-09-11T03:47:45.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:45 smithi175 ceph-mon[30894]: 1.19 scrub starts 2024-09-11T03:47:45.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:45 smithi175 ceph-mon[30894]: 1.19 scrub ok 2024-09-11T03:47:45.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:45 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:47:45.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:45 smithi175 ceph-mon[30894]: 2.14 scrub ok 2024-09-11T03:47:45.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:45 smithi175 ceph-mon[30894]: osdmap e647: 12 total, 12 up, 12 in 2024-09-11T03:47:45.495 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:45.516 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:45.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:45.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:45.618 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:45.640 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:45.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:45.641 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:45.742 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:45.758 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T03:47:45.763 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:45.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:45.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:45.843 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:47:45.865 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:45.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:45.908 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:45.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:45.996 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:47:46.009 INFO:tasks.rados.rados.2.smithi175.stdout:2501: done (2 left) 2024-09-11T03:47:46.009 INFO:tasks.rados.rados.2.smithi175.stdout:2502: write oid 35 current snap is 273 2024-09-11T03:47:46.009 INFO:tasks.rados.rados.2.smithi175.stdout:2498: finishing write tid 1 to smithi17557870-13 2024-09-11T03:47:46.009 INFO:tasks.rados.rados.2.smithi175.stdout:2498: finishing write tid 2 to smithi17557870-13 2024-09-11T03:47:46.009 INFO:tasks.rados.rados.2.smithi175.stdout:2502: seq_num 864 ranges {521248=563025,1809501=596300} 2024-09-11T03:47:46.009 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:46.014 INFO:tasks.rados.rados.2.smithi175.stdout:2502: writing smithi17557870-35 from 521248 to 1084273 tid 1 2024-09-11T03:47:46.023 INFO:tasks.rados.rados.2.smithi175.stdout:2502: writing smithi17557870-35 from 1809501 to 2405801 tid 2 2024-09-11T03:47:46.023 INFO:tasks.rados.rados.2.smithi175.stdout:2503: write oid 34 current snap is 273 2024-09-11T03:47:46.023 INFO:tasks.rados.rados.2.smithi175.stdout:2498: finishing write tid 3 to smithi17557870-13 2024-09-11T03:47:46.023 INFO:tasks.rados.rados.2.smithi175.stdout:2498: finishing write tid 4 to smithi17557870-13 2024-09-11T03:47:46.023 INFO:tasks.rados.rados.2.smithi175.stdout:2498: oid 13 updating version 0 to 899 2024-09-11T03:47:46.023 INFO:tasks.rados.rados.2.smithi175.stdout:2498: oid 13 updating version 899 to 900 2024-09-11T03:47:46.023 INFO:tasks.rados.rados.2.smithi175.stdout:2498: oid 13 updating version 900 to 901 2024-09-11T03:47:46.023 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 901 (ObjNum 861 snap 273 seq_num 861) dirty exists 2024-09-11T03:47:46.023 INFO:tasks.rados.rados.2.smithi175.stdout:2498: left oid 13 (ObjNum 861 snap 273 seq_num 861) 2024-09-11T03:47:46.023 INFO:tasks.rados.rados.2.smithi175.stdout:2499: finishing write tid 1 to smithi17557870-28 2024-09-11T03:47:46.024 INFO:tasks.rados.rados.2.smithi175.stdout:2499: finishing write tid 2 to smithi17557870-28 2024-09-11T03:47:46.024 INFO:tasks.rados.rados.2.smithi175.stdout:2499: finishing write tid 3 to smithi17557870-28 2024-09-11T03:47:46.024 INFO:tasks.rados.rados.2.smithi175.stdout:2499: finishing write tid 4 to smithi17557870-28 2024-09-11T03:47:46.024 INFO:tasks.rados.rados.2.smithi175.stdout:2499: oid 28 updating version 0 to 938 2024-09-11T03:47:46.024 INFO:tasks.rados.rados.2.smithi175.stdout:2499: oid 28 version 938 is already newer than 937 2024-09-11T03:47:46.024 INFO:tasks.rados.rados.2.smithi175.stdout:2499: oid 28 version 938 is already newer than 936 2024-09-11T03:47:46.024 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 938 (ObjNum 862 snap 273 seq_num 862) dirty exists 2024-09-11T03:47:46.024 INFO:tasks.rados.rados.2.smithi175.stdout:2499: left oid 28 (ObjNum 862 snap 273 seq_num 862) 2024-09-11T03:47:46.024 INFO:tasks.rados.rados.2.smithi175.stdout:2503: seq_num 865 ranges {538055=638274,1867983=670072} 2024-09-11T03:47:46.029 INFO:tasks.rados.rados.2.smithi175.stdout:2503: writing smithi17557870-34 from 538055 to 1176329 tid 1 2024-09-11T03:47:46.033 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:46.034 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:46.037 INFO:tasks.rados.rados.2.smithi175.stdout:2503: writing smithi17557870-34 from 1867983 to 2538055 tid 2 2024-09-11T03:47:46.037 INFO:tasks.rados.rados.2.smithi175.stdout:2498: done (3 left) 2024-09-11T03:47:46.037 INFO:tasks.rados.rados.2.smithi175.stdout:2499: done (2 left) 2024-09-11T03:47:46.037 INFO:tasks.rados.rados.2.smithi175.stdout:2504: delete oid 49 current snap is 273 2024-09-11T03:47:46.037 INFO:tasks.rados.rados.2.smithi175.stdout:2502: finishing write tid 1 to smithi17557870-35 2024-09-11T03:47:46.037 INFO:tasks.rados.rados.2.smithi175.stdout:2502: finishing write tid 2 to smithi17557870-35 2024-09-11T03:47:46.038 INFO:tasks.rados.rados.2.smithi175.stdout:2502: finishing write tid 3 to smithi17557870-35 2024-09-11T03:47:46.038 INFO:tasks.rados.rados.2.smithi175.stdout:2502: finishing write tid 4 to smithi17557870-35 2024-09-11T03:47:46.038 INFO:tasks.rados.rados.2.smithi175.stdout:2502: oid 35 updating version 0 to 701 2024-09-11T03:47:46.038 INFO:tasks.rados.rados.2.smithi175.stdout:2502: oid 35 version 701 is already newer than 700 2024-09-11T03:47:46.038 INFO:tasks.rados.rados.2.smithi175.stdout:2502: oid 35 version 701 is already newer than 697 2024-09-11T03:47:46.038 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 701 (ObjNum 864 snap 273 seq_num 864) dirty exists 2024-09-11T03:47:46.038 INFO:tasks.rados.rados.2.smithi175.stdout:2502: left oid 35 (ObjNum 864 snap 273 seq_num 864) 2024-09-11T03:47:46.038 INFO:tasks.rados.rados.2.smithi175.stdout:2503: finishing write tid 1 to smithi17557870-34 2024-09-11T03:47:46.041 INFO:tasks.rados.rados.2.smithi175.stdout:2502: done (2 left) 2024-09-11T03:47:46.041 INFO:tasks.rados.rados.2.smithi175.stdout:2504: done (1 left) 2024-09-11T03:47:46.041 INFO:tasks.rados.rados.2.smithi175.stdout:2505: rollback oid 38 current snap is 273 2024-09-11T03:47:46.041 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 38 to 272 2024-09-11T03:47:46.041 INFO:tasks.rados.rados.2.smithi175.stdout:2503: finishing write tid 2 to smithi17557870-34 2024-09-11T03:47:46.041 INFO:tasks.rados.rados.2.smithi175.stdout:2506: rollback oid 11 current snap is 273 2024-09-11T03:47:46.041 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 11 to 272 2024-09-11T03:47:46.041 INFO:tasks.rados.rados.2.smithi175.stdout:2507: rollback oid 39 current snap is 273 2024-09-11T03:47:46.041 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 39 to 272 2024-09-11T03:47:46.041 INFO:tasks.rados.rados.2.smithi175.stdout:2508: write (excl) oid 47 current snap is 273 2024-09-11T03:47:46.042 INFO:tasks.rados.rados.2.smithi175.stdout:2508: seq_num 867 ranges {571669=505125,1701300=743627} 2024-09-11T03:47:46.046 INFO:tasks.rados.rados.2.smithi175.stdout:2508: writing smithi17557870-47 from 571669 to 1076794 tid 1 2024-09-11T03:47:46.055 INFO:tasks.rados.rados.2.smithi175.stdout:2508: writing smithi17557870-47 from 1701300 to 2444927 tid 2 2024-09-11T03:47:46.055 INFO:tasks.rados.rados.2.smithi175.stdout:2509: read oid 33 snap 272 2024-09-11T03:47:46.055 INFO:tasks.rados.rados.2.smithi175.stdout:2509: expect deleted 2024-09-11T03:47:46.055 INFO:tasks.rados.rados.2.smithi175.stdout:2510: snap_create 2024-09-11T03:47:46.055 INFO:tasks.rados.rados.2.smithi175.stdout:2503: finishing write tid 3 to smithi17557870-34 2024-09-11T03:47:46.055 INFO:tasks.rados.rados.2.smithi175.stdout:2503: finishing write tid 4 to smithi17557870-34 2024-09-11T03:47:46.055 INFO:tasks.rados.rados.2.smithi175.stdout:2503: oid 34 updating version 0 to 876 2024-09-11T03:47:46.055 INFO:tasks.rados.rados.2.smithi175.stdout:2503: oid 34 updating version 876 to 878 2024-09-11T03:47:46.055 INFO:tasks.rados.rados.2.smithi175.stdout:2503: oid 34 version 878 is already newer than 877 2024-09-11T03:47:46.055 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 878 (ObjNum 865 snap 273 seq_num 865) dirty exists 2024-09-11T03:47:46.055 INFO:tasks.rados.rados.2.smithi175.stdout:2503: left oid 34 (ObjNum 865 snap 273 seq_num 865) 2024-09-11T03:47:46.055 INFO:tasks.rados.rados.2.smithi175.stdout:2507: finishing rollback tid 1 to smithi17557870-39 2024-09-11T03:47:46.056 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T03:47:46.056 INFO:tasks.rados.rados.2.smithi175.stdout:2505: finishing rollback tid 0 to smithi17557870-38 2024-09-11T03:47:46.056 INFO:tasks.rados.rados.2.smithi175.stdout:2505: finishing rollback tid 1 to smithi17557870-38 2024-09-11T03:47:46.056 INFO:tasks.rados.rados.2.smithi175.stdout:2505: finishing rollback tid 2 to smithi17557870-38 2024-09-11T03:47:46.056 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 904 (ObjNum 795 snap 254 seq_num 795) dirty exists 2024-09-11T03:47:46.056 INFO:tasks.rados.rados.2.smithi175.stdout:2506: finishing rollback tid 0 to smithi17557870-11 2024-09-11T03:47:46.056 INFO:tasks.rados.rados.2.smithi175.stdout:2506: finishing rollback tid 1 to smithi17557870-11 2024-09-11T03:47:46.056 INFO:tasks.rados.rados.2.smithi175.stdout:2506: finishing rollback tid 2 to smithi17557870-11 2024-09-11T03:47:46.056 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 606 (ObjNum 845 snap 266 seq_num 845) dirty exists 2024-09-11T03:47:46.056 INFO:tasks.rados.rados.2.smithi175.stdout:2508: finishing write tid 1 to smithi17557870-47 2024-09-11T03:47:46.058 INFO:tasks.rados.rados.2.smithi175.stdout:2508: finishing write tid 2 to smithi17557870-47 2024-09-11T03:47:46.058 INFO:tasks.rados.rados.2.smithi175.stdout:2508: finishing write tid 3 to smithi17557870-47 2024-09-11T03:47:46.058 INFO:tasks.rados.rados.2.smithi175.stdout:2508: finishing write tid 4 to smithi17557870-47 2024-09-11T03:47:46.058 INFO:tasks.rados.rados.2.smithi175.stdout:2508: oid 47 updating version 0 to 888 2024-09-11T03:47:46.058 INFO:tasks.rados.rados.2.smithi175.stdout:2508: oid 47 version 888 is already newer than 887 2024-09-11T03:47:46.058 INFO:tasks.rados.rados.2.smithi175.stdout:2508: oid 47 version 888 is already newer than 886 2024-09-11T03:47:46.058 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 888 (ObjNum 867 snap 273 seq_num 867) dirty exists 2024-09-11T03:47:46.058 INFO:tasks.rados.rados.2.smithi175.stdout:2508: left oid 47 (ObjNum 867 snap 273 seq_num 867) 2024-09-11T03:47:46.135 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:46.149 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:46.157 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:46.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:46.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:46.258 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:46.281 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:46.282 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:46.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:46 smithi084 ceph-mon[27254]: 1.10 scrub starts 2024-09-11T03:47:46.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:46 smithi084 ceph-mon[27254]: 1.10 scrub ok 2024-09-11T03:47:46.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:46 smithi084 ceph-mon[27254]: pgmap v1125: 67 pgs: 67 active+clean; 116 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 8 op/s 2024-09-11T03:47:46.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:46 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:47:46.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:46 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:47:46.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/177180297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:46.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:46 smithi084 ceph-mon[27254]: osdmap e648: 12 total, 12 up, 12 in 2024-09-11T03:47:46.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:46 smithi121 ceph-mon[30729]: 1.10 scrub starts 2024-09-11T03:47:46.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:46 smithi121 ceph-mon[30729]: 1.10 scrub ok 2024-09-11T03:47:46.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:46 smithi121 ceph-mon[30729]: pgmap v1125: 67 pgs: 67 active+clean; 116 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 8 op/s 2024-09-11T03:47:46.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:46 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:47:46.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:46 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:47:46.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/177180297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:46.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:46 smithi121 ceph-mon[30729]: osdmap e648: 12 total, 12 up, 12 in 2024-09-11T03:47:46.383 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:46.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:46.407 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:46.407 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:46.468 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:47:46.468 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:47:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:46 smithi175 ceph-mon[30894]: 1.10 scrub starts 2024-09-11T03:47:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:46 smithi175 ceph-mon[30894]: 1.10 scrub ok 2024-09-11T03:47:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:46 smithi175 ceph-mon[30894]: pgmap v1125: 67 pgs: 67 active+clean; 116 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.9 MiB/s wr, 8 op/s 2024-09-11T03:47:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:46 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:47:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:46 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:47:46.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/177180297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:46.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:46 smithi175 ceph-mon[30894]: osdmap e648: 12 total, 12 up, 12 in 2024-09-11T03:47:46.508 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:46.531 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:46.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:46.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:46.633 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:46.655 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:46.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:46.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:46.757 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:46.780 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:46.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:46.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:46.882 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:46.905 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:46.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:46.906 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:46.981 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:47:46.985 INFO:tasks.rados.rados.2.smithi175.stdout:2503: done (6 left) 2024-09-11T03:47:46.985 INFO:tasks.rados.rados.2.smithi175.stdout:2505: done (5 left) 2024-09-11T03:47:46.985 INFO:tasks.rados.rados.2.smithi175.stdout:2506: done (4 left) 2024-09-11T03:47:46.985 INFO:tasks.rados.rados.2.smithi175.stdout:2507: done (3 left) 2024-09-11T03:47:46.985 INFO:tasks.rados.rados.2.smithi175.stdout:2508: done (2 left) 2024-09-11T03:47:46.985 INFO:tasks.rados.rados.2.smithi175.stdout:2509: done (1 left) 2024-09-11T03:47:46.985 INFO:tasks.rados.rados.2.smithi175.stdout:2510: done (0 left) 2024-09-11T03:47:46.985 INFO:tasks.rados.rados.2.smithi175.stdout:2511: snap_remove snap 272 2024-09-11T03:47:47.007 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:47.028 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:47.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:47.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:47.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:47.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T03:47:47.153 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:47.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:47.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:47.255 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:47.276 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:47.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:47.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:47.344 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:47 smithi084 ceph-mon[27254]: 2.f scrub starts 2024-09-11T03:47:47.344 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:47 smithi084 ceph-mon[27254]: 2.f scrub ok 2024-09-11T03:47:47.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:47 smithi084 ceph-mon[27254]: 2.1e scrub starts 2024-09-11T03:47:47.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:47 smithi084 ceph-mon[27254]: 2.1e scrub ok 2024-09-11T03:47:47.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:47 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:47:47.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:47 smithi084 ceph-mon[27254]: 2.c scrub ok 2024-09-11T03:47:47.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:47 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1139628995' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:47:47.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:47 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:47:47.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:47 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:47:47.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:47 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1139628995' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:47:47.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:47 smithi084 ceph-mon[27254]: osdmap e649: 12 total, 12 up, 12 in 2024-09-11T03:47:47.376 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:47:47.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:47.401 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:47.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:47.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:47.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:47 smithi121 ceph-mon[30729]: 2.f scrub starts 2024-09-11T03:47:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:47 smithi121 ceph-mon[30729]: 2.f scrub ok 2024-09-11T03:47:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:47 smithi121 ceph-mon[30729]: 2.1e scrub starts 2024-09-11T03:47:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:47 smithi121 ceph-mon[30729]: 2.1e scrub ok 2024-09-11T03:47:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:47 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:47:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:47 smithi121 ceph-mon[30729]: 2.c scrub ok 2024-09-11T03:47:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:47 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1139628995' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:47:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:47 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:47:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:47 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:47:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:47 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1139628995' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:47:47.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:47 smithi121 ceph-mon[30729]: osdmap e649: 12 total, 12 up, 12 in 2024-09-11T03:47:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:47 smithi175 ceph-mon[30894]: 2.f scrub starts 2024-09-11T03:47:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:47 smithi175 ceph-mon[30894]: 2.f scrub ok 2024-09-11T03:47:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:47 smithi175 ceph-mon[30894]: 2.1e scrub starts 2024-09-11T03:47:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:47 smithi175 ceph-mon[30894]: 2.1e scrub ok 2024-09-11T03:47:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:47 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:47:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:47 smithi175 ceph-mon[30894]: 2.c scrub ok 2024-09-11T03:47:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:47 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1139628995' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:47:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:47 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:47:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:47 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:47:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:47 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1139628995' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:47:47.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:47 smithi175 ceph-mon[30894]: osdmap e649: 12 total, 12 up, 12 in 2024-09-11T03:47:47.503 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:47.524 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:47.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:47.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:47.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:47.648 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:47.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:47.750 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:47.770 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:47.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:47.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:47.842 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:47:47.842 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:47:47.872 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:47.898 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:47.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:47.900 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:47.988 INFO:tasks.rados.rados.2.smithi175.stdout:2511: done (0 left) 2024-09-11T03:47:47.989 INFO:tasks.rados.rados.2.smithi175.stdout:2512: copy_from oid 1 from oid 31 current snap is 274 2024-09-11T03:47:47.989 INFO:tasks.rados.rados.2.smithi175.stdout:2513: rollback oid 44 current snap is 274 2024-09-11T03:47:47.989 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 44 to 273 2024-09-11T03:47:47.989 INFO:tasks.rados.rados.2.smithi175.stdout:2514: write oid 32 current snap is 274 2024-09-11T03:47:47.989 INFO:tasks.rados.rados.2.smithi175.stdout:2514: seq_num 868 ranges {588476=580374,1643429=603638} 2024-09-11T03:47:47.995 INFO:tasks.rados.rados.2.smithi175.stdout:2514: writing smithi17557870-32 from 588476 to 1168850 tid 1 2024-09-11T03:47:48.001 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:48.004 INFO:tasks.rados.rados.2.smithi175.stdout:2514: writing smithi17557870-32 from 1643429 to 2247067 tid 2 2024-09-11T03:47:48.004 INFO:tasks.rados.rados.2.smithi175.stdout:2515: delete oid 42 current snap is 274 2024-09-11T03:47:48.004 INFO:tasks.rados.rados.2.smithi175.stdout:2513: finishing rollback tid 0 to smithi17557870-44 2024-09-11T03:47:48.004 INFO:tasks.rados.rados.2.smithi175.stdout:2513: finishing rollback tid 1 to smithi17557870-44 2024-09-11T03:47:48.004 INFO:tasks.rados.rados.2.smithi175.stdout:2513: finishing rollback tid 2 to smithi17557870-44 2024-09-11T03:47:48.004 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 813 (ObjNum 787 snap 249 seq_num 787) dirty exists 2024-09-11T03:47:48.004 INFO:tasks.rados.rados.2.smithi175.stdout:2512: finishing copy_from racing read to smithi17557870-1 2024-09-11T03:47:48.005 INFO:tasks.rados.rados.2.smithi175.stdout:2512: finishing copy_from to smithi17557870-1 2024-09-11T03:47:48.005 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 924 (ObjNum 758 snap 246 seq_num 758) dirty exists 2024-09-11T03:47:48.006 INFO:tasks.rados.rados.2.smithi175.stdout:2514: finishing write tid 1 to smithi17557870-32 2024-09-11T03:47:48.007 INFO:tasks.rados.rados.2.smithi175.stdout:2512: done (3 left) 2024-09-11T03:47:48.007 INFO:tasks.rados.rados.2.smithi175.stdout:2513: done (2 left) 2024-09-11T03:47:48.007 INFO:tasks.rados.rados.2.smithi175.stdout:2515: done (1 left) 2024-09-11T03:47:48.007 INFO:tasks.rados.rados.2.smithi175.stdout:2516: rollback oid 23 current snap is 274 2024-09-11T03:47:48.007 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 23 to 273 2024-09-11T03:47:48.007 INFO:tasks.rados.rados.2.smithi175.stdout:2517: write oid 33 current snap is 274 2024-09-11T03:47:48.007 INFO:tasks.rados.rados.2.smithi175.stdout:2517: seq_num 870 ranges {622090=730872,1760393=607307} 2024-09-11T03:47:48.014 INFO:tasks.rados.rados.2.smithi175.stdout:2517: writing smithi17557870-33 from 622090 to 1352962 tid 1 2024-09-11T03:47:48.020 INFO:tasks.rados.rados.2.smithi175.stdout:2517: writing smithi17557870-33 from 1760393 to 2367700 tid 2 2024-09-11T03:47:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:2518: rollback oid 45 current snap is 274 2024-09-11T03:47:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 45 to 273 2024-09-11T03:47:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:2514: finishing write tid 2 to smithi17557870-32 2024-09-11T03:47:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:2519: read oid 26 snap 273 2024-09-11T03:47:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:2519: expect (ObjNum 793 snap 252 seq_num 793) 2024-09-11T03:47:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:2520: rollback oid 2 current snap is 274 2024-09-11T03:47:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 2 to 273 2024-09-11T03:47:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:2514: finishing write tid 3 to smithi17557870-32 2024-09-11T03:47:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:2514: finishing write tid 4 to smithi17557870-32 2024-09-11T03:47:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:2514: oid 32 updating version 0 to 858 2024-09-11T03:47:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:2514: oid 32 version 858 is already newer than 857 2024-09-11T03:47:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:2514: oid 32 version 858 is already newer than 856 2024-09-11T03:47:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 858 (ObjNum 868 snap 274 seq_num 868) dirty exists 2024-09-11T03:47:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2514: left oid 32 (ObjNum 868 snap 274 seq_num 868) 2024-09-11T03:47:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2516: finishing rollback tid 0 to smithi17557870-23 2024-09-11T03:47:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2516: finishing rollback tid 1 to smithi17557870-23 2024-09-11T03:47:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2516: finishing rollback tid 2 to smithi17557870-23 2024-09-11T03:47:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 862 (ObjNum 855 snap 268 seq_num 855) dirty exists 2024-09-11T03:47:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2517: finishing write tid 1 to smithi17557870-33 2024-09-11T03:47:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2514: done (5 left) 2024-09-11T03:47:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2516: done (4 left) 2024-09-11T03:47:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2521: write oid 27 current snap is 274 2024-09-11T03:47:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2521: seq_num 871 ranges {638897=406121,1818875=750965} 2024-09-11T03:47:48.023 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:48.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:48.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:48.025 INFO:tasks.rados.rados.2.smithi175.stdout:2521: writing smithi17557870-27 from 638897 to 1045018 tid 1 2024-09-11T03:47:48.035 INFO:tasks.rados.rados.2.smithi175.stdout:2521: writing smithi17557870-27 from 1818875 to 2569840 tid 2 2024-09-11T03:47:48.035 INFO:tasks.rados.rados.2.smithi175.stdout:2522: write (excl) oid 28 current snap is 274 2024-09-11T03:47:48.035 INFO:tasks.rados.rados.2.smithi175.stdout:2518: finishing rollback tid 1 to smithi17557870-45 2024-09-11T03:47:48.035 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 0 (ObjNum 945 snap 0 seq_num 2956159312) dirty dne 2024-09-11T03:47:48.035 INFO:tasks.rados.rados.2.smithi175.stdout:2520: finishing rollback tid 0 to smithi17557870-2 2024-09-11T03:47:48.035 INFO:tasks.rados.rados.2.smithi175.stdout:2517: finishing write tid 2 to smithi17557870-33 2024-09-11T03:47:48.035 INFO:tasks.rados.rados.2.smithi175.stdout:2520: finishing rollback tid 1 to smithi17557870-2 2024-09-11T03:47:48.035 INFO:tasks.rados.rados.2.smithi175.stdout:2520: finishing rollback tid 2 to smithi17557870-2 2024-09-11T03:47:48.035 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 950 (ObjNum 765 snap 247 seq_num 765) dirty exists 2024-09-11T03:47:48.035 INFO:tasks.rados.rados.2.smithi175.stdout:2517: finishing write tid 3 to smithi17557870-33 2024-09-11T03:47:48.035 INFO:tasks.rados.rados.2.smithi175.stdout:2517: finishing write tid 4 to smithi17557870-33 2024-09-11T03:47:48.036 INFO:tasks.rados.rados.2.smithi175.stdout:2517: oid 33 updating version 0 to 945 2024-09-11T03:47:48.036 INFO:tasks.rados.rados.2.smithi175.stdout:2517: oid 33 updating version 945 to 946 2024-09-11T03:47:48.036 INFO:tasks.rados.rados.2.smithi175.stdout:2517: oid 33 updating version 946 to 947 2024-09-11T03:47:48.036 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 947 (ObjNum 870 snap 274 seq_num 870) dirty exists 2024-09-11T03:47:48.036 INFO:tasks.rados.rados.2.smithi175.stdout:2517: left oid 33 (ObjNum 870 snap 274 seq_num 870) 2024-09-11T03:47:48.036 INFO:tasks.rados.rados.2.smithi175.stdout:2521: finishing write tid 1 to smithi17557870-27 2024-09-11T03:47:48.036 INFO:tasks.rados.rados.2.smithi175.stdout:2522: seq_num 872 ranges {655704=481370,1761004=494623} 2024-09-11T03:47:48.040 INFO:tasks.rados.rados.2.smithi175.stdout:2522: writing smithi17557870-28 from 655704 to 1137074 tid 1 2024-09-11T03:47:48.046 INFO:tasks.rados.rados.2.smithi175.stdout:2522: writing smithi17557870-28 from 1761004 to 2255627 tid 2 2024-09-11T03:47:48.047 INFO:tasks.rados.rados.2.smithi175.stdout:2517: done (5 left) 2024-09-11T03:47:48.047 INFO:tasks.rados.rados.2.smithi175.stdout:2518: done (4 left) 2024-09-11T03:47:48.047 INFO:tasks.rados.rados.2.smithi175.stdout:2520: done (3 left) 2024-09-11T03:47:48.047 INFO:tasks.rados.rados.2.smithi175.stdout:2523: write (excl) oid 12 current snap is 274 2024-09-11T03:47:48.047 INFO:tasks.rados.rados.2.smithi175.stdout:2523: seq_num 873 ranges {672511=556619,1819486=754634} 2024-09-11T03:47:48.053 INFO:tasks.rados.rados.2.smithi175.stdout:2523: writing smithi17557870-12 from 672511 to 1229130 tid 1 2024-09-11T03:47:48.061 INFO:tasks.rados.rados.2.smithi175.stdout:2523: writing smithi17557870-12 from 1819486 to 2574120 tid 2 2024-09-11T03:47:48.061 INFO:tasks.rados.rados.2.smithi175.stdout:2524: read oid 13 snap -1 2024-09-11T03:47:48.061 INFO:tasks.rados.rados.2.smithi175.stdout:2524: expect (ObjNum 861 snap 273 seq_num 861) 2024-09-11T03:47:48.061 INFO:tasks.rados.rados.2.smithi175.stdout:2525: copy_from oid 44 from oid 16 current snap is 274 2024-09-11T03:47:48.061 INFO:tasks.rados.rados.2.smithi175.stdout:2519: expect (ObjNum 793 snap 252 seq_num 793) 2024-09-11T03:47:48.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:47:48.126 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:48.148 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:48.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:48.149 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:48.169 INFO:tasks.rados.rados.2.smithi175.stdout:2521: finishing write tid 2 to smithi17557870-27 2024-09-11T03:47:48.169 INFO:tasks.rados.rados.2.smithi175.stdout:2521: finishing write tid 3 to smithi17557870-27 2024-09-11T03:47:48.169 INFO:tasks.rados.rados.2.smithi175.stdout:2521: finishing write tid 4 to smithi17557870-27 2024-09-11T03:47:48.169 INFO:tasks.rados.rados.2.smithi175.stdout:2521: oid 27 updating version 0 to 954 2024-09-11T03:47:48.169 INFO:tasks.rados.rados.2.smithi175.stdout:2521: oid 27 version 954 is already newer than 952 2024-09-11T03:47:48.169 INFO:tasks.rados.rados.2.smithi175.stdout:2521: oid 27 version 954 is already newer than 953 2024-09-11T03:47:48.169 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 954 (ObjNum 871 snap 274 seq_num 871) dirty exists 2024-09-11T03:47:48.169 INFO:tasks.rados.rados.2.smithi175.stdout:2521: left oid 27 (ObjNum 871 snap 274 seq_num 871) 2024-09-11T03:47:48.170 INFO:tasks.rados.rados.2.smithi175.stdout:2522: finishing write tid 1 to smithi17557870-28 2024-09-11T03:47:48.170 INFO:tasks.rados.rados.2.smithi175.stdout:2522: finishing write tid 2 to smithi17557870-28 2024-09-11T03:47:48.170 INFO:tasks.rados.rados.2.smithi175.stdout:2519: done (5 left) 2024-09-11T03:47:48.170 INFO:tasks.rados.rados.2.smithi175.stdout:2521: done (4 left) 2024-09-11T03:47:48.170 INFO:tasks.rados.rados.2.smithi175.stdout:2526: read oid 26 snap -1 2024-09-11T03:47:48.170 INFO:tasks.rados.rados.2.smithi175.stdout:2526: expect (ObjNum 793 snap 252 seq_num 793) 2024-09-11T03:47:48.170 INFO:tasks.rados.rados.2.smithi175.stdout:2527: rollback oid 45 current snap is 274 2024-09-11T03:47:48.170 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 45 to 273 2024-09-11T03:47:48.170 INFO:tasks.rados.rados.2.smithi175.stdout:2522: finishing write tid 3 to smithi17557870-28 2024-09-11T03:47:48.170 INFO:tasks.rados.rados.2.smithi175.stdout:2522: finishing write tid 4 to smithi17557870-28 2024-09-11T03:47:48.170 INFO:tasks.rados.rados.2.smithi175.stdout:2522: oid 28 updating version 0 to 950 2024-09-11T03:47:48.170 INFO:tasks.rados.rados.2.smithi175.stdout:2522: oid 28 updating version 950 to 951 2024-09-11T03:47:48.170 INFO:tasks.rados.rados.2.smithi175.stdout:2522: oid 28 version 951 is already newer than 949 2024-09-11T03:47:48.171 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 951 (ObjNum 872 snap 274 seq_num 872) dirty exists 2024-09-11T03:47:48.171 INFO:tasks.rados.rados.2.smithi175.stdout:2522: left oid 28 (ObjNum 872 snap 274 seq_num 872) 2024-09-11T03:47:48.171 INFO:tasks.rados.rados.2.smithi175.stdout:2523: finishing write tid 1 to smithi17557870-12 2024-09-11T03:47:48.171 INFO:tasks.rados.rados.2.smithi175.stdout:2525: finishing copy_from to smithi17557870-44 2024-09-11T03:47:48.171 INFO:tasks.rados.rados.2.smithi175.stdout:2525: got expected ENOENT (src dne) 2024-09-11T03:47:48.171 INFO:tasks.rados.rados.2.smithi175.stdout:2525: finishing copy_from racing read to smithi17557870-44 2024-09-11T03:47:48.171 INFO:tasks.rados.rados.2.smithi175.stdout:2523: finishing write tid 2 to smithi17557870-12 2024-09-11T03:47:48.171 INFO:tasks.rados.rados.2.smithi175.stdout:2523: finishing write tid 3 to smithi17557870-12 2024-09-11T03:47:48.171 INFO:tasks.rados.rados.2.smithi175.stdout:2522: done (5 left) 2024-09-11T03:47:48.171 INFO:tasks.rados.rados.2.smithi175.stdout:2525: done (4 left) 2024-09-11T03:47:48.171 INFO:tasks.rados.rados.2.smithi175.stdout:2528: snap_create 2024-09-11T03:47:48.172 INFO:tasks.rados.rados.2.smithi175.stdout:2523: finishing write tid 4 to smithi17557870-12 2024-09-11T03:47:48.172 INFO:tasks.rados.rados.2.smithi175.stdout:2523: oid 12 updating version 0 to 927 2024-09-11T03:47:48.172 INFO:tasks.rados.rados.2.smithi175.stdout:2523: oid 12 version 927 is already newer than 926 2024-09-11T03:47:48.172 INFO:tasks.rados.rados.2.smithi175.stdout:2523: oid 12 version 927 is already newer than 925 2024-09-11T03:47:48.172 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 927 (ObjNum 873 snap 274 seq_num 873) dirty exists 2024-09-11T03:47:48.172 INFO:tasks.rados.rados.2.smithi175.stdout:2523: left oid 12 (ObjNum 873 snap 274 seq_num 873) 2024-09-11T03:47:48.172 INFO:tasks.rados.rados.2.smithi175.stdout:2524: expect (ObjNum 861 snap 273 seq_num 861) 2024-09-11T03:47:48.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:48 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:47:48.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:48 smithi121 ceph-mon[30729]: 2.0 scrub ok 2024-09-11T03:47:48.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:48 smithi121 ceph-mon[30729]: pgmap v1128: 67 pgs: 67 active+clean; 124 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.0 MiB/s wr, 9 op/s 2024-09-11T03:47:48.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:48 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:47:48.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:48 smithi121 ceph-mon[30729]: 2.b scrub ok 2024-09-11T03:47:48.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:48 smithi121 ceph-mon[30729]: 2.23 scrub starts 2024-09-11T03:47:48.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:48 smithi121 ceph-mon[30729]: 2.23 scrub ok 2024-09-11T03:47:48.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:48 smithi121 ceph-mon[30729]: 2.18 scrub starts 2024-09-11T03:47:48.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:48 smithi121 ceph-mon[30729]: 2.18 scrub ok 2024-09-11T03:47:48.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:48 smithi121 ceph-mon[30729]: osdmap e650: 12 total, 12 up, 12 in 2024-09-11T03:47:48.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:48.267 INFO:tasks.rados.rados.2.smithi175.stdout:2527: finishing rollback tid 1 to smithi17557870-45 2024-09-11T03:47:48.268 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 0 (ObjNum 945 snap 0 seq_num 1745293248) dirty dne 2024-09-11T03:47:48.268 INFO:tasks.rados.rados.2.smithi175.stdout:2526: expect (ObjNum 793 snap 252 seq_num 793) 2024-09-11T03:47:48.271 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:48.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:48.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:48.343 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:48.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:48 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:47:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:48 smithi084 ceph-mon[27254]: 2.0 scrub ok 2024-09-11T03:47:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:48 smithi084 ceph-mon[27254]: pgmap v1128: 67 pgs: 67 active+clean; 124 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.0 MiB/s wr, 9 op/s 2024-09-11T03:47:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:48 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:47:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:48 smithi084 ceph-mon[27254]: 2.b scrub ok 2024-09-11T03:47:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:48 smithi084 ceph-mon[27254]: 2.23 scrub starts 2024-09-11T03:47:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:48 smithi084 ceph-mon[27254]: 2.23 scrub ok 2024-09-11T03:47:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:48 smithi084 ceph-mon[27254]: 2.18 scrub starts 2024-09-11T03:47:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:48 smithi084 ceph-mon[27254]: 2.18 scrub ok 2024-09-11T03:47:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:48 smithi084 ceph-mon[27254]: osdmap e650: 12 total, 12 up, 12 in 2024-09-11T03:47:48.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:48.395 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:48.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:48 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:47:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:48 smithi175 ceph-mon[30894]: 2.0 scrub ok 2024-09-11T03:47:48.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:48 smithi175 ceph-mon[30894]: pgmap v1128: 67 pgs: 67 active+clean; 124 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.0 MiB/s wr, 9 op/s 2024-09-11T03:47:48.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:48 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:47:48.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:48 smithi175 ceph-mon[30894]: 2.b scrub ok 2024-09-11T03:47:48.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:48 smithi175 ceph-mon[30894]: 2.23 scrub starts 2024-09-11T03:47:48.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:48 smithi175 ceph-mon[30894]: 2.23 scrub ok 2024-09-11T03:47:48.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:48 smithi175 ceph-mon[30894]: 2.18 scrub starts 2024-09-11T03:47:48.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:48 smithi175 ceph-mon[30894]: 2.18 scrub ok 2024-09-11T03:47:48.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:48 smithi175 ceph-mon[30894]: osdmap e650: 12 total, 12 up, 12 in 2024-09-11T03:47:48.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:48.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T03:47:48.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:48.520 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:48.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:48.621 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:48.643 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:48.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:48.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:48.745 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:48.758 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:47:48.766 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:48.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:48.767 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:48.867 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:48.888 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:48.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:48.894 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:47:48.894 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:47:48.911 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:47:48.911 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:47:48.926 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:47:48.926 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:47:48.942 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:47:48.943 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:47:48.959 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:47:48.959 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:47:48.976 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:47:48.976 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:47:48.987 INFO:tasks.rados.rados.2.smithi175.stdout:2523: done (4 left) 2024-09-11T03:47:48.988 INFO:tasks.rados.rados.2.smithi175.stdout:2524: done (3 left) 2024-09-11T03:47:48.988 INFO:tasks.rados.rados.2.smithi175.stdout:2526: done (2 left) 2024-09-11T03:47:48.988 INFO:tasks.rados.rados.2.smithi175.stdout:2527: done (1 left) 2024-09-11T03:47:48.988 INFO:tasks.rados.rados.2.smithi175.stdout:2528: done (0 left) 2024-09-11T03:47:48.988 INFO:tasks.rados.rados.2.smithi175.stdout:2529: copy_from oid 37 from oid 47 current snap is 275 2024-09-11T03:47:48.988 INFO:tasks.rados.rados.2.smithi175.stdout:2530: snap_remove snap 273 2024-09-11T03:47:48.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:48.993 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:47:48.994 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:47:49.011 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:47:49.011 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:47:49.015 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:49.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:49.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:49.027 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:47:49.027 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:47:49.044 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:47:49.044 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:47:49.061 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:47:49.061 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:47:49.077 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:47:49.078 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:47:49.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:49.138 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:49.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:49.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:49 smithi121 ceph-mon[30729]: 2.1d scrub starts 2024-09-11T03:47:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:49 smithi121 ceph-mon[30729]: 2.1d scrub ok 2024-09-11T03:47:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:49 smithi121 ceph-mon[30729]: 1.15 scrub starts 2024-09-11T03:47:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:49 smithi121 ceph-mon[30729]: 1.15 scrub ok 2024-09-11T03:47:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:49 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:47:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:49 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:47:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:49 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:47:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:49 smithi121 ceph-mon[30729]: 2.1 scrub ok 2024-09-11T03:47:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:49 smithi121 ceph-mon[30729]: 2.21 scrub starts 2024-09-11T03:47:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:49 smithi121 ceph-mon[30729]: 2.21 scrub ok 2024-09-11T03:47:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4108259543' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:49.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:49 smithi121 ceph-mon[30729]: osdmap e651: 12 total, 12 up, 12 in 2024-09-11T03:47:49.233 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:47:49.233 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:47:49.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:47:49.233 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:47:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:402 2.7 2:e23d1762:::smithi17557870-21:head [copy-from ver 9] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:47:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:51.085659+0000", 2024-09-11T03:47:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 538.13799099699997, 2024-09-11T03:47:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012118869000000001, 2024-09-11T03:47:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 402 2024-09-11T03:47:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085659+0000", 2024-09-11T03:47:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085659+0000", 2024-09-11T03:47:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085661+0000", 2024-09-11T03:47:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1220000000000002e-06 2024-09-11T03:47:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085669+0000", 2024-09-11T03:47:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2779999999999993e-06 2024-09-11T03:47:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085689+0000", 2024-09-11T03:47:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9639999999999998e-05 2024-09-11T03:47:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085699+0000", 2024-09-11T03:47:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0524e-05 2024-09-11T03:47:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085789+0000", 2024-09-11T03:47:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9215000000000006e-05 2024-09-11T03:47:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085880+0000", 2024-09-11T03:47:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1187999999999995e-05 2024-09-11T03:47:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:47:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.090839+0000", 2024-09-11T03:47:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004958863 2024-09-11T03:47:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.093676+0000", 2024-09-11T03:47:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028368930000000001 2024-09-11T03:47:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.097457+0000", 2024-09-11T03:47:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003781197 2024-09-11T03:47:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.097707+0000", 2024-09-11T03:47:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024975900000000001 2024-09-11T03:47:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.097745+0000", 2024-09-11T03:47:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8596000000000001e-05 2024-09-11T03:47:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.097778+0000", 2024-09-11T03:47:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2594e-05 2024-09-11T03:47:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:403 2.7 2:e23d1762:::smithi17557870-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e114)", 2024-09-11T03:47:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:38:51.085720+0000", 2024-09-11T03:47:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 538.13793017700004, 2024-09-11T03:47:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076054490000000002, 2024-09-11T03:47:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:47:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 403 2024-09-11T03:47:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085720+0000", 2024-09-11T03:47:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085720+0000", 2024-09-11T03:47:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085720+0000", 2024-09-11T03:47:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5599999999999995e-07 2024-09-11T03:47:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085742+0000", 2024-09-11T03:47:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1968e-05 2024-09-11T03:47:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085750+0000", 2024-09-11T03:47:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8699999999999992e-06 2024-09-11T03:47:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.085756+0000", 2024-09-11T03:47:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-09-11T03:47:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.086016+0000", 2024-09-11T03:47:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00026035099999999997 2024-09-11T03:47:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.086071+0000", 2024-09-11T03:47:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4614999999999998e-05 2024-09-11T03:47:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.093154+0000", 2024-09-11T03:47:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070835050000000004 2024-09-11T03:47:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.093239+0000", 2024-09-11T03:47:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5309999999999995e-05 2024-09-11T03:47:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:38:51.093325+0000", 2024-09-11T03:47:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5654000000000004e-05 2024-09-11T03:47:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5414 2.7 2:e0ec5763:::smithi17557870-2:head [stat,delete] snapc f2=[f2,f1,f0,ee,ea] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-09-11T03:47:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:46:38.033566+0000", 2024-09-11T03:47:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 71.190083755000003, 2024-09-11T03:47:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053945740000000001, 2024-09-11T03:47:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:47:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5414 2024-09-11T03:47:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.033566+0000", 2024-09-11T03:47:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.033566+0000", 2024-09-11T03:47:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.033568+0000", 2024-09-11T03:47:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.717e-06 2024-09-11T03:47:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.033575+0000", 2024-09-11T03:47:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4039999999999998e-06 2024-09-11T03:47:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.033590+0000", 2024-09-11T03:47:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4654000000000001e-05 2024-09-11T03:47:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.033598+0000", 2024-09-11T03:47:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9719999999999992e-06 2024-09-11T03:47:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.033686+0000", 2024-09-11T03:47:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7996000000000002e-05 2024-09-11T03:47:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:47:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.033695+0000", 2024-09-11T03:47:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0890000000000006e-06 2024-09-11T03:47:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.037409+0000", 2024-09-11T03:47:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037139650000000001 2024-09-11T03:47:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.037466+0000", 2024-09-11T03:47:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7504999999999999e-05 2024-09-11T03:47:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.038961+0000", 2024-09-11T03:47:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001494272 2024-09-11T03:47:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:5415 2.23 e580/574 2:c6c49f2c:::smithi17557870-44:head v 580'385, mlcod=0'0)", 2024-09-11T03:47:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:46:38.046264+0000", 2024-09-11T03:47:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 71.177386136999999, 2024-09-11T03:47:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006751356, 2024-09-11T03:47:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.046264+0000", 2024-09-11T03:47:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.046264+0000", 2024-09-11T03:47:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.046264+0000", 2024-09-11T03:47:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.79e-07 2024-09-11T03:47:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.051016+0000", 2024-09-11T03:47:49.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047516349999999997 2024-09-11T03:47:49.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.051026+0000", 2024-09-11T03:47:49.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0661e-05 2024-09-11T03:47:49.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.051032+0000", 2024-09-11T03:47:49.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4e-06 2024-09-11T03:47:49.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.051083+0000", 2024-09-11T03:47:49.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1674e-05 2024-09-11T03:47:49.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.051099+0000", 2024-09-11T03:47:49.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5923999999999999e-05 2024-09-11T03:47:49.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:49.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.052980+0000", 2024-09-11T03:47:49.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018806269999999999 2024-09-11T03:47:49.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:38.053015+0000", 2024-09-11T03:47:49.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5256000000000003e-05 2024-09-11T03:47:49.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.256 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.256 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5510 2.7 2:e23d1762:::smithi17557870-21:head [write 538022~738348,stat] snapc f5=[f4,f0,ee] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-09-11T03:47:49.257 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:46:46.097904+0000", 2024-09-11T03:47:49.257 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 63.125745688999999, 2024-09-11T03:47:49.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0093603060000000005, 2024-09-11T03:47:49.257 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.257 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.257 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:49.257 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.257 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.257 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.257 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5510 2024-09-11T03:47:49.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.257 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.097904+0000", 2024-09-11T03:47:49.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.097904+0000", 2024-09-11T03:47:49.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.097906+0000", 2024-09-11T03:47:49.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.012e-06 2024-09-11T03:47:49.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.098914+0000", 2024-09-11T03:47:49.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010074820000000001 2024-09-11T03:47:49.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.098930+0000", 2024-09-11T03:47:49.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5968e-05 2024-09-11T03:47:49.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.098939+0000", 2024-09-11T03:47:49.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6390000000000004e-06 2024-09-11T03:47:49.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.099043+0000", 2024-09-11T03:47:49.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103501 2024-09-11T03:47:49.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:47:49.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.099049+0000", 2024-09-11T03:47:49.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-09-11T03:47:49.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.100984+0000", 2024-09-11T03:47:49.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019355360000000001 2024-09-11T03:47:49.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.101086+0000", 2024-09-11T03:47:49.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010211400000000001 2024-09-11T03:47:49.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:47:49.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.101272+0000", 2024-09-11T03:47:49.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018546200000000001 2024-09-11T03:47:49.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:49.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.104082+0000", 2024-09-11T03:47:49.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028102560000000001 2024-09-11T03:47:49.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.104465+0000", 2024-09-11T03:47:49.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000383018 2024-09-11T03:47:49.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.107181+0000", 2024-09-11T03:47:49.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027157430000000001 2024-09-11T03:47:49.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:49.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.107225+0000", 2024-09-11T03:47:49.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4048999999999998e-05 2024-09-11T03:47:49.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.107264+0000", 2024-09-11T03:47:49.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9452000000000001e-05 2024-09-11T03:47:49.263 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.263 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.263 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.264 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5521 2.7 2:e0eabcfb:::smithi17557870-9:head [stat,delete] snapc f5=[f0] ondisk+write+known_if_redirected+supports_pool_eio e590)", 2024-09-11T03:47:49.264 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:46:48.098690+0000", 2024-09-11T03:47:49.264 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.124960309000002, 2024-09-11T03:47:49.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.008942574, 2024-09-11T03:47:49.264 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.264 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.264 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:47:49.264 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.264 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.264 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.264 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5521 2024-09-11T03:47:49.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.265 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.098690+0000", 2024-09-11T03:47:49.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.098690+0000", 2024-09-11T03:47:49.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.098693+0000", 2024-09-11T03:47:49.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.977e-06 2024-09-11T03:47:49.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.098702+0000", 2024-09-11T03:47:49.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4199999999999996e-06 2024-09-11T03:47:49.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.098727+0000", 2024-09-11T03:47:49.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4856999999999999e-05 2024-09-11T03:47:49.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.098739+0000", 2024-09-11T03:47:49.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2520999999999999e-05 2024-09-11T03:47:49.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.098852+0000", 2024-09-11T03:47:49.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011317 2024-09-11T03:47:49.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:47:49.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.098862+0000", 2024-09-11T03:47:49.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7629999999999995e-06 2024-09-11T03:47:49.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.103253+0000", 2024-09-11T03:47:49.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043909420000000001 2024-09-11T03:47:49.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.103339+0000", 2024-09-11T03:47:49.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6062000000000004e-05 2024-09-11T03:47:49.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.107632+0000", 2024-09-11T03:47:49.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042928619999999997 2024-09-11T03:47:49.268 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.269 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.269 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.269 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5524 2.7 2:e77ce595:::smithi17557870-31:head [delete] snapc f6=[f0] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e592)", 2024-09-11T03:47:49.269 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:46:50.113743+0000", 2024-09-11T03:47:49.269 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 59.109906791999997, 2024-09-11T03:47:49.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070158130000000001, 2024-09-11T03:47:49.269 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.269 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.269 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:49.269 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.269 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.270 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.270 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5524 2024-09-11T03:47:49.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.270 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:50.113743+0000", 2024-09-11T03:47:49.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:50.113743+0000", 2024-09-11T03:47:49.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:50.113745+0000", 2024-09-11T03:47:49.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1600000000000001e-06 2024-09-11T03:47:49.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:50.113752+0000", 2024-09-11T03:47:49.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2570000000000001e-06 2024-09-11T03:47:49.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:50.113770+0000", 2024-09-11T03:47:49.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7997999999999999e-05 2024-09-11T03:47:49.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:50.113784+0000", 2024-09-11T03:47:49.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3920000000000001e-05 2024-09-11T03:47:49.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:50.113861+0000", 2024-09-11T03:47:49.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6123000000000003e-05 2024-09-11T03:47:49.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:47:49.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:50.113866+0000", 2024-09-11T03:47:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4280000000000004e-06 2024-09-11T03:47:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:50.118685+0000", 2024-09-11T03:47:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048191939999999997 2024-09-11T03:47:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:50.118843+0000", 2024-09-11T03:47:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015752299999999999 2024-09-11T03:47:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:47:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:50.119000+0000", 2024-09-11T03:47:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015757600000000001 2024-09-11T03:47:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:50.120018+0000", 2024-09-11T03:47:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010178920000000001 2024-09-11T03:47:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:50.120622+0000", 2024-09-11T03:47:49.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060402399999999999 2024-09-11T03:47:49.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:50.120688+0000", 2024-09-11T03:47:49.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6039999999999998e-05 2024-09-11T03:47:49.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:49.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:50.120728+0000", 2024-09-11T03:47:49.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9888000000000002e-05 2024-09-11T03:47:49.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:50.120759+0000", 2024-09-11T03:47:49.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0790000000000002e-05 2024-09-11T03:47:49.276 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.276 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.276 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.276 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5652 2.7 2:e0eabcfb:::smithi17557870-9:head [write 642232~661230] snapc f9=[] ondisk+write+known_if_redirected+supports_pool_eio e599)", 2024-09-11T03:47:49.276 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:46:56.982525+0000", 2024-09-11T03:47:49.276 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 52.241124956, 2024-09-11T03:47:49.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0088314529999999995, 2024-09-11T03:47:49.277 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.277 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.277 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:49.277 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.277 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.277 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.277 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5652 2024-09-11T03:47:49.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.277 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.277 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:49.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.982525+0000", 2024-09-11T03:47:49.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.982525+0000", 2024-09-11T03:47:49.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.982527+0000", 2024-09-11T03:47:49.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6390000000000001e-06 2024-09-11T03:47:49.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.983324+0000", 2024-09-11T03:47:49.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079795899999999999 2024-09-11T03:47:49.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.983341+0000", 2024-09-11T03:47:49.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033999999999999e-05 2024-09-11T03:47:49.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.983349+0000", 2024-09-11T03:47:49.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.619e-06 2024-09-11T03:47:49.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.983417+0000", 2024-09-11T03:47:49.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7756999999999995e-05 2024-09-11T03:47:49.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:47:49.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.983426+0000", 2024-09-11T03:47:49.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3030000000000005e-06 2024-09-11T03:47:49.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.986355+0000", 2024-09-11T03:47:49.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029289910000000001 2024-09-11T03:47:49.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.986479+0000", 2024-09-11T03:47:49.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123356 2024-09-11T03:47:49.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:47:49.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.986635+0000", 2024-09-11T03:47:49.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000156374 2024-09-11T03:47:49.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:49.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.989107+0000", 2024-09-11T03:47:49.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024717559999999999 2024-09-11T03:47:49.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.990053+0000", 2024-09-11T03:47:49.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00094608099999999998 2024-09-11T03:47:49.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.991248+0000", 2024-09-11T03:47:49.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001194961 2024-09-11T03:47:49.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:49.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.991300+0000", 2024-09-11T03:47:49.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2500999999999998e-05 2024-09-11T03:47:49.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.991356+0000", 2024-09-11T03:47:49.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6122000000000002e-05 2024-09-11T03:47:49.286 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.286 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.286 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.287 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5653 2.7 2:e0eabcfb:::smithi17557870-9:head [write 1942968~783923] snapc f9=[] ondisk+write+known_if_redirected+supports_pool_eio e599)", 2024-09-11T03:47:49.287 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:46:56.990885+0000", 2024-09-11T03:47:49.287 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 52.232764819000003, 2024-09-11T03:47:49.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0067564330000000001, 2024-09-11T03:47:49.287 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.287 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.287 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:49.287 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.287 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.287 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.287 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5653 2024-09-11T03:47:49.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.288 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.990885+0000", 2024-09-11T03:47:49.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.990885+0000", 2024-09-11T03:47:49.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.990887+0000", 2024-09-11T03:47:49.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.592e-06 2024-09-11T03:47:49.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992161+0000", 2024-09-11T03:47:49.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012748309999999999 2024-09-11T03:47:49.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992178+0000", 2024-09-11T03:47:49.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6169999999999999e-05 2024-09-11T03:47:49.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992187+0000", 2024-09-11T03:47:49.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5969999999999998e-06 2024-09-11T03:47:49.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992239+0000", 2024-09-11T03:47:49.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2181000000000001e-05 2024-09-11T03:47:49.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992299+0000", 2024-09-11T03:47:49.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9171e-05 2024-09-11T03:47:49.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:47:49.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992411+0000", 2024-09-11T03:47:49.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011273999999999999 2024-09-11T03:47:49.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:49.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.994692+0000", 2024-09-11T03:47:49.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022806929999999999 2024-09-11T03:47:49.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.997496+0000", 2024-09-11T03:47:49.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028044820000000001 2024-09-11T03:47:49.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.997567+0000", 2024-09-11T03:47:49.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0338000000000003e-05 2024-09-11T03:47:49.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:49.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.997605+0000", 2024-09-11T03:47:49.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8022000000000003e-05 2024-09-11T03:47:49.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.997641+0000", 2024-09-11T03:47:49.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6616000000000003e-05 2024-09-11T03:47:49.293 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.293 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.293 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.293 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5654 2.7 2:e0eabcfb:::smithi17557870-9:head [setxattr _header (57) in=7b,truncate 3042232] snapc f9=[] ondisk+write+known_if_redirected+supports_pool_eio e599)", 2024-09-11T03:47:49.293 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:46:56.992208+0000", 2024-09-11T03:47:49.293 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 52.231441736999997, 2024-09-11T03:47:49.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055383999999999997, 2024-09-11T03:47:49.293 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.293 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.294 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:49.294 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.294 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.294 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.294 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5654 2024-09-11T03:47:49.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.294 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992208+0000", 2024-09-11T03:47:49.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992208+0000", 2024-09-11T03:47:49.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992209+0000", 2024-09-11T03:47:49.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.195e-06 2024-09-11T03:47:49.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992216+0000", 2024-09-11T03:47:49.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7240000000000003e-06 2024-09-11T03:47:49.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992224+0000", 2024-09-11T03:47:49.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3219999999999993e-06 2024-09-11T03:47:49.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992230+0000", 2024-09-11T03:47:49.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.66e-06 2024-09-11T03:47:49.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.994028+0000", 2024-09-11T03:47:49.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001797636 2024-09-11T03:47:49.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.994095+0000", 2024-09-11T03:47:49.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6977999999999995e-05 2024-09-11T03:47:49.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:47:49.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.994205+0000", 2024-09-11T03:47:49.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010991 2024-09-11T03:47:49.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:49.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.994843+0000", 2024-09-11T03:47:49.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00063839099999999998 2024-09-11T03:47:49.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.997533+0000", 2024-09-11T03:47:49.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002690193 2024-09-11T03:47:49.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.997673+0000", 2024-09-11T03:47:49.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000139426 2024-09-11T03:47:49.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:49.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.997708+0000", 2024-09-11T03:47:49.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5689000000000003e-05 2024-09-11T03:47:49.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.997747+0000", 2024-09-11T03:47:49.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8275999999999997e-05 2024-09-11T03:47:49.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.299 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.299 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5655 2.7 2:e0eabcfb:::smithi17557870-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e599)", 2024-09-11T03:47:49.299 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:46:56.992246+0000", 2024-09-11T03:47:49.299 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 52.231403884000002, 2024-09-11T03:47:49.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056482590000000001, 2024-09-11T03:47:49.300 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.300 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.300 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:47:49.300 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.300 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.300 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.300 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5655 2024-09-11T03:47:49.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.300 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992246+0000", 2024-09-11T03:47:49.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992246+0000", 2024-09-11T03:47:49.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992247+0000", 2024-09-11T03:47:49.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.99e-07 2024-09-11T03:47:49.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992250+0000", 2024-09-11T03:47:49.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0819999999999999e-06 2024-09-11T03:47:49.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992258+0000", 2024-09-11T03:47:49.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8779999999999998e-06 2024-09-11T03:47:49.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.992263+0000", 2024-09-11T03:47:49.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7699999999999998e-06 2024-09-11T03:47:49.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.994721+0000", 2024-09-11T03:47:49.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024576070000000001 2024-09-11T03:47:49.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:47:49.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.994783+0000", 2024-09-11T03:47:49.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2230999999999994e-05 2024-09-11T03:47:49.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.997776+0000", 2024-09-11T03:47:49.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029928340000000002 2024-09-11T03:47:49.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.997834+0000", 2024-09-11T03:47:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7967999999999998e-05 2024-09-11T03:47:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.997894+0000", 2024-09-11T03:47:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.029e-05 2024-09-11T03:47:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.304 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5669 2.7 2:e6da0655:::smithi17557870-27:head [write 692653~486977,stat] snapc fa=[fa] ondisk+write+known_if_redirected+supports_pool_eio e600)", 2024-09-11T03:47:49.305 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:46:57.980193+0000", 2024-09-11T03:47:49.305 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.243456817000002, 2024-09-11T03:47:49.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055860550000000004, 2024-09-11T03:47:49.305 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.305 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.305 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:49.305 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.305 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.305 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.305 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5669 2024-09-11T03:47:49.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.305 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:57.980193+0000", 2024-09-11T03:47:49.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:57.980193+0000", 2024-09-11T03:47:49.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:57.980195+0000", 2024-09-11T03:47:49.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.951e-06 2024-09-11T03:47:49.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:57.980959+0000", 2024-09-11T03:47:49.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00076394099999999997 2024-09-11T03:47:49.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:57.980979+0000", 2024-09-11T03:47:49.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0159000000000001e-05 2024-09-11T03:47:49.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:57.980989+0000", 2024-09-11T03:47:49.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.005e-05 2024-09-11T03:47:49.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:57.981068+0000", 2024-09-11T03:47:49.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8607000000000001e-05 2024-09-11T03:47:49.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:47:49.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:57.981074+0000", 2024-09-11T03:47:49.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.455e-06 2024-09-11T03:47:49.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:57.982854+0000", 2024-09-11T03:47:49.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017796579999999999 2024-09-11T03:47:49.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:57.982926+0000", 2024-09-11T03:47:49.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1909000000000006e-05 2024-09-11T03:47:49.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:47:49.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:57.983054+0000", 2024-09-11T03:47:49.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127964 2024-09-11T03:47:49.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:49.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:57.984807+0000", 2024-09-11T03:47:49.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001753192 2024-09-11T03:47:49.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:57.985648+0000", 2024-09-11T03:47:49.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00084109199999999999 2024-09-11T03:47:49.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:57.985696+0000", 2024-09-11T03:47:49.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8322999999999998e-05 2024-09-11T03:47:49.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:49.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:57.985740+0000", 2024-09-11T03:47:49.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3198999999999998e-05 2024-09-11T03:47:49.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:57.985779+0000", 2024-09-11T03:47:49.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9555000000000003e-05 2024-09-11T03:47:49.311 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.311 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.311 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.311 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5683 2.14 2:2ef19c88:::smithi17557870-15:head [write 743074~712724] snapc fa=[] ondisk+write+known_if_redirected+supports_pool_eio e601)", 2024-09-11T03:47:49.311 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:46:58.984552+0000", 2024-09-11T03:47:49.311 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.239098276, 2024-09-11T03:47:49.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072381820000000001, 2024-09-11T03:47:49.312 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.312 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.312 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:49.312 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.312 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.312 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.312 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5683 2024-09-11T03:47:49.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.312 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:58.984552+0000", 2024-09-11T03:47:49.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:58.984552+0000", 2024-09-11T03:47:49.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:58.984553+0000", 2024-09-11T03:47:49.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.787e-06 2024-09-11T03:47:49.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:58.985340+0000", 2024-09-11T03:47:49.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078664800000000003 2024-09-11T03:47:49.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:58.985357+0000", 2024-09-11T03:47:49.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6685e-05 2024-09-11T03:47:49.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:58.985366+0000", 2024-09-11T03:47:49.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0909999999999999e-06 2024-09-11T03:47:49.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:58.985471+0000", 2024-09-11T03:47:49.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104886 2024-09-11T03:47:49.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:47:49.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:58.985476+0000", 2024-09-11T03:47:49.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3070000000000002e-06 2024-09-11T03:47:49.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:58.987482+0000", 2024-09-11T03:47:49.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020056990000000001 2024-09-11T03:47:49.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:58.987593+0000", 2024-09-11T03:47:49.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011110100000000001 2024-09-11T03:47:49.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5,9", 2024-09-11T03:47:49.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:58.987911+0000", 2024-09-11T03:47:49.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031775999999999999 2024-09-11T03:47:49.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:49.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:58.990070+0000", 2024-09-11T03:47:49.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021596739999999999 2024-09-11T03:47:49.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:58.990462+0000", 2024-09-11T03:47:49.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039207499999999998 2024-09-11T03:47:49.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:58.991699+0000", 2024-09-11T03:47:49.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001236525 2024-09-11T03:47:49.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:49.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:58.991750+0000", 2024-09-11T03:47:49.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1004999999999997e-05 2024-09-11T03:47:49.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:58.991790+0000", 2024-09-11T03:47:49.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9938999999999998e-05 2024-09-11T03:47:49.318 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.318 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.318 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.318 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5799 2.14 2:2ef19c88:::smithi17557870-15:head [write 645600~416265,stat] snapc 100=[100,ff,fe,fd] ondisk+write+known_if_redirected+supports_pool_eio e609)", 2024-09-11T03:47:49.319 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:06.983196+0000", 2024-09-11T03:47:49.319 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 42.240454038999999, 2024-09-11T03:47:49.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054531279999999998, 2024-09-11T03:47:49.319 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.319 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.319 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:49.319 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.319 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.319 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.319 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5799 2024-09-11T03:47:49.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.319 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:06.983196+0000", 2024-09-11T03:47:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:06.983196+0000", 2024-09-11T03:47:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:06.983198+0000", 2024-09-11T03:47:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.855e-06 2024-09-11T03:47:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:06.983724+0000", 2024-09-11T03:47:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052627000000000001 2024-09-11T03:47:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:06.983738+0000", 2024-09-11T03:47:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3767999999999999e-05 2024-09-11T03:47:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:06.983745+0000", 2024-09-11T03:47:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3520000000000001e-06 2024-09-11T03:47:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:06.983848+0000", 2024-09-11T03:47:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103385 2024-09-11T03:47:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:47:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:06.983855+0000", 2024-09-11T03:47:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0500000000000003e-06 2024-09-11T03:47:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:06.985700+0000", 2024-09-11T03:47:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018441289999999999 2024-09-11T03:47:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:06.985795+0000", 2024-09-11T03:47:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5111999999999993e-05 2024-09-11T03:47:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5,9", 2024-09-11T03:47:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:06.985970+0000", 2024-09-11T03:47:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017518199999999999 2024-09-11T03:47:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:06.988048+0000", 2024-09-11T03:47:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002078487 2024-09-11T03:47:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:06.988220+0000", 2024-09-11T03:47:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000171235 2024-09-11T03:47:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:06.988575+0000", 2024-09-11T03:47:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035574400000000001 2024-09-11T03:47:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:06.988615+0000", 2024-09-11T03:47:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9483000000000001e-05 2024-09-11T03:47:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:06.988649+0000", 2024-09-11T03:47:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4075999999999997e-05 2024-09-11T03:47:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5844 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 881] snapc 100=[100,ff] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-09-11T03:47:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:09.028558+0000", 2024-09-11T03:47:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.19509188, 2024-09-11T03:47:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020566675, 2024-09-11T03:47:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5844 2024-09-11T03:47:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028558+0000", 2024-09-11T03:47:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028558+0000", 2024-09-11T03:47:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028560+0000", 2024-09-11T03:47:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.001e-06 2024-09-11T03:47:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028570+0000", 2024-09-11T03:47:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7799999999999995e-06 2024-09-11T03:47:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028586+0000", 2024-09-11T03:47:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6047999999999999e-05 2024-09-11T03:47:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028596+0000", 2024-09-11T03:47:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.046e-05 2024-09-11T03:47:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028665+0000", 2024-09-11T03:47:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9152000000000005e-05 2024-09-11T03:47:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:47:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028670+0000", 2024-09-11T03:47:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6770000000000003e-06 2024-09-11T03:47:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.033863+0000", 2024-09-11T03:47:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051932799999999998 2024-09-11T03:47:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.033967+0000", 2024-09-11T03:47:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010344399999999999 2024-09-11T03:47:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,3", 2024-09-11T03:47:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.039468+0000", 2024-09-11T03:47:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055009330000000004 2024-09-11T03:47:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.043252+0000", 2024-09-11T03:47:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037844150000000002 2024-09-11T03:47:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.044699+0000", 2024-09-11T03:47:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0014469000000000001 2024-09-11T03:47:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.049058+0000", 2024-09-11T03:47:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043590399999999998 2024-09-11T03:47:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.049095+0000", 2024-09-11T03:47:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6745000000000001e-05 2024-09-11T03:47:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.049125+0000", 2024-09-11T03:47:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9799999999999999e-05 2024-09-11T03:47:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5845 2.19 2:98bb65f5:::smithi17557870-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e611)", 2024-09-11T03:47:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:09.028622+0000", 2024-09-11T03:47:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.195027474, 2024-09-11T03:47:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014191941, 2024-09-11T03:47:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:47:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5845 2024-09-11T03:47:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028622+0000", 2024-09-11T03:47:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028622+0000", 2024-09-11T03:47:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028623+0000", 2024-09-11T03:47:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6300000000000004e-07 2024-09-11T03:47:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028626+0000", 2024-09-11T03:47:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3270000000000001e-06 2024-09-11T03:47:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028643+0000", 2024-09-11T03:47:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.615e-05 2024-09-11T03:47:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028650+0000", 2024-09-11T03:47:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4970000000000004e-06 2024-09-11T03:47:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028730+0000", 2024-09-11T03:47:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9618999999999998e-05 2024-09-11T03:47:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028738+0000", 2024-09-11T03:47:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3569999999999998e-06 2024-09-11T03:47:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.034146+0000", 2024-09-11T03:47:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054079439999999996 2024-09-11T03:47:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.034193+0000", 2024-09-11T03:47:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7045000000000001e-05 2024-09-11T03:47:49.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.042662+0000", 2024-09-11T03:47:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084692180000000006 2024-09-11T03:47:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.042741+0000", 2024-09-11T03:47:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8749000000000001e-05 2024-09-11T03:47:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.042814+0000", 2024-09-11T03:47:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3271999999999999e-05 2024-09-11T03:47:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5976 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 600] snapc 106=[] ondisk+write+known_if_redirected+supports_pool_eio e625)", 2024-09-11T03:47:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:23.155338+0000", 2024-09-11T03:47:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 26.068311381000001, 2024-09-11T03:47:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017109479, 2024-09-11T03:47:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5976 2024-09-11T03:47:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155338+0000", 2024-09-11T03:47:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155338+0000", 2024-09-11T03:47:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155342+0000", 2024-09-11T03:47:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7230000000000002e-06 2024-09-11T03:47:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155357+0000", 2024-09-11T03:47:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4401e-05 2024-09-11T03:47:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155396+0000", 2024-09-11T03:47:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9301000000000002e-05 2024-09-11T03:47:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155408+0000", 2024-09-11T03:47:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2499e-05 2024-09-11T03:47:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155516+0000", 2024-09-11T03:47:49.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107948 2024-09-11T03:47:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155615+0000", 2024-09-11T03:47:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8627000000000003e-05 2024-09-11T03:47:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,3", 2024-09-11T03:47:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.159866+0000", 2024-09-11T03:47:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004250635 2024-09-11T03:47:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:49.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.163122+0000", 2024-09-11T03:47:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032564859999999998 2024-09-11T03:47:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.169373+0000", 2024-09-11T03:47:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062509230000000002 2024-09-11T03:47:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.172372+0000", 2024-09-11T03:47:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029991810000000001 2024-09-11T03:47:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.172414+0000", 2024-09-11T03:47:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1539999999999999e-05 2024-09-11T03:47:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.172448+0000", 2024-09-11T03:47:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4214999999999997e-05 2024-09-11T03:47:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5977 2.19 2:98bb65f5:::smithi17557870-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e625)", 2024-09-11T03:47:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:23.155440+0000", 2024-09-11T03:47:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 26.068209728999999, 2024-09-11T03:47:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0073194699999999998, 2024-09-11T03:47:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:47:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5977 2024-09-11T03:47:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.346 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155440+0000", 2024-09-11T03:47:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155440+0000", 2024-09-11T03:47:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155441+0000", 2024-09-11T03:47:49.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7999999999999995e-07 2024-09-11T03:47:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155446+0000", 2024-09-11T03:47:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6169999999999998e-06 2024-09-11T03:47:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155459+0000", 2024-09-11T03:47:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2882e-05 2024-09-11T03:47:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155464+0000", 2024-09-11T03:47:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3680000000000004e-06 2024-09-11T03:47:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155796+0000", 2024-09-11T03:47:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00033210800000000001 2024-09-11T03:47:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155867+0000", 2024-09-11T03:47:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0819000000000001e-05 2024-09-11T03:47:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.162598+0000", 2024-09-11T03:47:49.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0067311690000000004 2024-09-11T03:47:49.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.162678+0000", 2024-09-11T03:47:49.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0461000000000001e-05 2024-09-11T03:47:49.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.162760+0000", 2024-09-11T03:47:49.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1366000000000007e-05 2024-09-11T03:47:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:6102 2.7 2:e6da0655:::smithi17557870-27:head [write 618722~575837] snapc 10c=[10c] ondisk+write+known_if_redirected+supports_pool_eio e636)", 2024-09-11T03:47:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:34.013084+0000", 2024-09-11T03:47:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.210565728000001, 2024-09-11T03:47:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087560710000000007, 2024-09-11T03:47:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:47:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:47:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 6102 2024-09-11T03:47:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.013084+0000", 2024-09-11T03:47:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.013084+0000", 2024-09-11T03:47:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.013086+0000", 2024-09-11T03:47:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2989999999999999e-06 2024-09-11T03:47:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.013746+0000", 2024-09-11T03:47:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00065981199999999996 2024-09-11T03:47:49.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.013769+0000", 2024-09-11T03:47:49.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2322000000000001e-05 2024-09-11T03:47:49.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.013797+0000", 2024-09-11T03:47:49.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8935e-05 2024-09-11T03:47:49.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.013880+0000", 2024-09-11T03:47:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2131000000000004e-05 2024-09-11T03:47:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:47:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.013887+0000", 2024-09-11T03:47:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3239999999999997e-06 2024-09-11T03:47:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.017957+0000", 2024-09-11T03:47:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040699669999999999 2024-09-11T03:47:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.018042+0000", 2024-09-11T03:47:49.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4756e-05 2024-09-11T03:47:49.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:47:49.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.018165+0000", 2024-09-11T03:47:49.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012344100000000001 2024-09-11T03:47:49.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:47:49.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.021608+0000", 2024-09-11T03:47:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034432880000000001 2024-09-11T03:47:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.021674+0000", 2024-09-11T03:47:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5254999999999998e-05 2024-09-11T03:47:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.021716+0000", 2024-09-11T03:47:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2175000000000002e-05 2024-09-11T03:47:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.021780+0000", 2024-09-11T03:47:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4068999999999999e-05 2024-09-11T03:47:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.021840+0000", 2024-09-11T03:47:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0297000000000002e-05 2024-09-11T03:47:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:6215 2.16 e646/560 2:6dbd1e81:::smithi17557870-7:head v 646'431, mlcod=641'428)", 2024-09-11T03:47:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:43.999441+0000", 2024-09-11T03:47:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.2242089619999996, 2024-09-11T03:47:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0081217670000000002, 2024-09-11T03:47:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:47:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:47:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:47:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:47:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.999441+0000", 2024-09-11T03:47:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:47:49.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.999441+0000", 2024-09-11T03:47:49.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:47:49.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:47:49.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.999441+0000", 2024-09-11T03:47:49.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6899999999999999e-07 2024-09-11T03:47:49.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:47:49.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.002211+0000", 2024-09-11T03:47:49.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027698340000000001 2024-09-11T03:47:49.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:47:49.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.002224+0000", 2024-09-11T03:47:49.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3108000000000001e-05 2024-09-11T03:47:49.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:47:49.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.002231+0000", 2024-09-11T03:47:49.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.401e-06 2024-09-11T03:47:49.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:47:49.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.002287+0000", 2024-09-11T03:47:49.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5717000000000002e-05 2024-09-11T03:47:49.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:47:49.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.002311+0000", 2024-09-11T03:47:49.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3913e-05 2024-09-11T03:47:49.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:47:49.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.007518+0000", 2024-09-11T03:47:49.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052071749999999997 2024-09-11T03:47:49.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:47:49.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:47:49.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:47:49.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.007563+0000", 2024-09-11T03:47:49.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4450000000000003e-05 2024-09-11T03:47:49.363 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.363 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.363 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.363 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:47:49.363 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:47:49.363 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:47:49.385 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:49.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:49.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:49 smithi084 ceph-mon[27254]: 2.1d scrub starts 2024-09-11T03:47:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:49 smithi084 ceph-mon[27254]: 2.1d scrub ok 2024-09-11T03:47:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:49 smithi084 ceph-mon[27254]: 1.15 scrub starts 2024-09-11T03:47:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:49 smithi084 ceph-mon[27254]: 1.15 scrub ok 2024-09-11T03:47:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:49 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:47:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:49 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:47:49.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:49 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:47:49.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:49 smithi084 ceph-mon[27254]: 2.1 scrub ok 2024-09-11T03:47:49.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:49 smithi084 ceph-mon[27254]: 2.21 scrub starts 2024-09-11T03:47:49.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:49 smithi084 ceph-mon[27254]: 2.21 scrub ok 2024-09-11T03:47:49.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4108259543' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:49.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:49 smithi084 ceph-mon[27254]: osdmap e651: 12 total, 12 up, 12 in 2024-09-11T03:47:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:49 smithi175 ceph-mon[30894]: 2.1d scrub starts 2024-09-11T03:47:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:49 smithi175 ceph-mon[30894]: 2.1d scrub ok 2024-09-11T03:47:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:49 smithi175 ceph-mon[30894]: 1.15 scrub starts 2024-09-11T03:47:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:49 smithi175 ceph-mon[30894]: 1.15 scrub ok 2024-09-11T03:47:49.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:49 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:47:49.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:49 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:47:49.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:49 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:47:49.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:49 smithi175 ceph-mon[30894]: 2.1 scrub ok 2024-09-11T03:47:49.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:49 smithi175 ceph-mon[30894]: 2.21 scrub starts 2024-09-11T03:47:49.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:49 smithi175 ceph-mon[30894]: 2.21 scrub ok 2024-09-11T03:47:49.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4108259543' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:49.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:49 smithi175 ceph-mon[30894]: osdmap e651: 12 total, 12 up, 12 in 2024-09-11T03:47:49.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:49.509 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:49.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:49.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:49.611 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:49.633 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:49.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:49.634 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:49.648 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:47:49.648 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:47:49.648 INFO:tasks.thrashosds.thrasher:inject_pause on osd.11 2024-09-11T03:47:49.648 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T03:47:49.648 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T03:47:49.649 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 config set filestore_inject_stall 3 2024-09-11T03:47:49.722 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:47:49.735 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:49.756 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:49.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:49.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:49.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:49.879 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:49.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:49.916 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:47:49.966 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:49.980 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:49.989 INFO:tasks.rados.rados.2.smithi175.stdout:2530: done (1 left) 2024-09-11T03:47:49.989 INFO:tasks.rados.rados.2.smithi175.stdout:2531: read oid 24 snap -1 2024-09-11T03:47:49.989 INFO:tasks.rados.rados.2.smithi175.stdout:2531: expect (ObjNum 811 snap 256 seq_num 811) 2024-09-11T03:47:49.989 INFO:tasks.rados.rados.2.smithi175.stdout:2532: write (excl) oid 28 current snap is 275 2024-09-11T03:47:49.989 INFO:tasks.rados.rados.2.smithi175.stdout:2529: finishing copy_from racing read to smithi17557870-37 2024-09-11T03:47:49.989 INFO:tasks.rados.rados.2.smithi175.stdout:2532: seq_num 874 ranges {689318=631868,1877968=498292} 2024-09-11T03:47:49.996 INFO:tasks.rados.rados.2.smithi175.stdout:2532: writing smithi17557870-28 from 689318 to 1321186 tid 1 2024-09-11T03:47:50.002 INFO:tasks.rados.rados.2.smithi175.stdout:2532: writing smithi17557870-28 from 1877968 to 2376260 tid 2 2024-09-11T03:47:50.003 INFO:tasks.rados.rados.2.smithi175.stdout:2533: write oid 27 current snap is 275 2024-09-11T03:47:50.003 INFO:tasks.rados.rados.2.smithi175.stdout:2529: finishing copy_from to smithi17557870-37 2024-09-11T03:47:50.003 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 888 (ObjNum 867 snap 273 seq_num 867) dirty exists 2024-09-11T03:47:50.003 INFO:tasks.rados.rados.2.smithi175.stdout:2532: finishing write tid 1 to smithi17557870-28 2024-09-11T03:47:50.003 INFO:tasks.rados.rados.2.smithi175.stdout:2533: seq_num 875 ranges {706125=423470,1652803=758303} 2024-09-11T03:47:50.006 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:50.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:50.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:50.008 INFO:tasks.rados.rados.2.smithi175.stdout:2533: writing smithi17557870-27 from 706125 to 1129595 tid 1 2024-09-11T03:47:50.016 INFO:tasks.rados.rados.2.smithi175.stdout:2533: writing smithi17557870-27 from 1652803 to 2411106 tid 2 2024-09-11T03:47:50.016 INFO:tasks.rados.rados.2.smithi175.stdout:2529: done (3 left) 2024-09-11T03:47:50.016 INFO:tasks.rados.rados.2.smithi175.stdout:2534: rollback oid 29 current snap is 275 2024-09-11T03:47:50.016 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 29 to 274 2024-09-11T03:47:50.017 INFO:tasks.rados.rados.2.smithi175.stdout:2532: finishing write tid 2 to smithi17557870-28 2024-09-11T03:47:50.017 INFO:tasks.rados.rados.2.smithi175.stdout:2532: finishing write tid 3 to smithi17557870-28 2024-09-11T03:47:50.017 INFO:tasks.rados.rados.2.smithi175.stdout:2532: finishing write tid 4 to smithi17557870-28 2024-09-11T03:47:50.017 INFO:tasks.rados.rados.2.smithi175.stdout:2532: oid 28 updating version 0 to 957 2024-09-11T03:47:50.017 INFO:tasks.rados.rados.2.smithi175.stdout:2532: oid 28 version 957 is already newer than 955 2024-09-11T03:47:50.017 INFO:tasks.rados.rados.2.smithi175.stdout:2532: oid 28 version 957 is already newer than 956 2024-09-11T03:47:50.017 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 957 (ObjNum 874 snap 275 seq_num 874) dirty exists 2024-09-11T03:47:50.017 INFO:tasks.rados.rados.2.smithi175.stdout:2532: left oid 28 (ObjNum 874 snap 275 seq_num 874) 2024-09-11T03:47:50.017 INFO:tasks.rados.rados.2.smithi175.stdout:2531: expect (ObjNum 811 snap 256 seq_num 811) 2024-09-11T03:47:50.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:50.129 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:50.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:50.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:50.219 INFO:tasks.rados.rados.2.smithi175.stdout:2533: finishing write tid 1 to smithi17557870-27 2024-09-11T03:47:50.219 INFO:tasks.rados.rados.2.smithi175.stdout:2534: finishing rollback tid 0 to smithi17557870-29 2024-09-11T03:47:50.219 INFO:tasks.rados.rados.2.smithi175.stdout:2534: finishing rollback tid 1 to smithi17557870-29 2024-09-11T03:47:50.219 INFO:tasks.rados.rados.2.smithi175.stdout:2534: finishing rollback tid 2 to smithi17557870-29 2024-09-11T03:47:50.219 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 930 (ObjNum 829 snap 263 seq_num 829) dirty exists 2024-09-11T03:47:50.219 INFO:tasks.rados.rados.2.smithi175.stdout:2533: finishing write tid 2 to smithi17557870-27 2024-09-11T03:47:50.219 INFO:tasks.rados.rados.2.smithi175.stdout:2533: finishing write tid 3 to smithi17557870-27 2024-09-11T03:47:50.219 INFO:tasks.rados.rados.2.smithi175.stdout:2533: finishing write tid 4 to smithi17557870-27 2024-09-11T03:47:50.219 INFO:tasks.rados.rados.2.smithi175.stdout:2533: oid 27 updating version 0 to 960 2024-09-11T03:47:50.219 INFO:tasks.rados.rados.2.smithi175.stdout:2533: oid 27 updating version 960 to 961 2024-09-11T03:47:50.220 INFO:tasks.rados.rados.2.smithi175.stdout:2533: oid 27 updating version 961 to 962 2024-09-11T03:47:50.220 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 962 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:47:50.220 INFO:tasks.rados.rados.2.smithi175.stdout:2533: left oid 27 (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:47:50.220 INFO:tasks.rados.rados.2.smithi175.stdout:2531: done (3 left) 2024-09-11T03:47:50.220 INFO:tasks.rados.rados.2.smithi175.stdout:2532: done (2 left) 2024-09-11T03:47:50.220 INFO:tasks.rados.rados.2.smithi175.stdout:2533: done (1 left) 2024-09-11T03:47:50.220 INFO:tasks.rados.rados.2.smithi175.stdout:2534: done (0 left) 2024-09-11T03:47:50.220 INFO:tasks.rados.rados.2.smithi175.stdout:2535: snap_create 2024-09-11T03:47:50.231 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:50.253 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:50.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:50.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:50.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T03:47:50.339 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:50 smithi175 ceph-mon[30894]: pgmap v1131: 67 pgs: 67 active+clean; 122 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2024-09-11T03:47:50.339 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:50 smithi175 ceph-mon[30894]: 1.18 scrub starts 2024-09-11T03:47:50.340 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:50 smithi175 ceph-mon[30894]: 1.18 scrub ok 2024-09-11T03:47:50.340 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:50 smithi175 ceph-mon[30894]: 2.22 scrub starts 2024-09-11T03:47:50.340 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:50 smithi175 ceph-mon[30894]: 2.22 scrub ok 2024-09-11T03:47:50.340 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:50 smithi175 ceph-mon[30894]: osdmap e652: 12 total, 12 up, 12 in 2024-09-11T03:47:50.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:50.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:50 smithi084 ceph-mon[27254]: pgmap v1131: 67 pgs: 67 active+clean; 122 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2024-09-11T03:47:50.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:50 smithi084 ceph-mon[27254]: 1.18 scrub starts 2024-09-11T03:47:50.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:50 smithi084 ceph-mon[27254]: 1.18 scrub ok 2024-09-11T03:47:50.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:50 smithi084 ceph-mon[27254]: 2.22 scrub starts 2024-09-11T03:47:50.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:50 smithi084 ceph-mon[27254]: 2.22 scrub ok 2024-09-11T03:47:50.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:50 smithi084 ceph-mon[27254]: osdmap e652: 12 total, 12 up, 12 in 2024-09-11T03:47:50.376 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:50.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:50.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:50 smithi121 ceph-mon[30729]: pgmap v1131: 67 pgs: 67 active+clean; 122 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2024-09-11T03:47:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:50 smithi121 ceph-mon[30729]: 1.18 scrub starts 2024-09-11T03:47:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:50 smithi121 ceph-mon[30729]: 1.18 scrub ok 2024-09-11T03:47:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:50 smithi121 ceph-mon[30729]: 2.22 scrub starts 2024-09-11T03:47:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:50 smithi121 ceph-mon[30729]: 2.22 scrub ok 2024-09-11T03:47:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:50 smithi121 ceph-mon[30729]: osdmap e652: 12 total, 12 up, 12 in 2024-09-11T03:47:50.440 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:47:50.440 INFO:teuthology.orchestra.run.smithi175.stdout: "success": "" 2024-09-11T03:47:50.440 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:47:50.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:50.500 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:50.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:50.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:50.519 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:47:50.601 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:50.622 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:50.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:50.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:50.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:50.747 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:50.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:50.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:50.848 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:50.872 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:50.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:50.873 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:50.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:50.984 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:47:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:47:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:47:50.985 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:47:50.995 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:50.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:50.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:51.079 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:47:51.083 INFO:tasks.rados.rados.2.smithi175.stdout:2535: done (0 left) 2024-09-11T03:47:51.084 INFO:tasks.rados.rados.2.smithi175.stdout:2536: read oid 40 snap 274 2024-09-11T03:47:51.084 INFO:tasks.rados.rados.2.smithi175.stdout:2536: expect (ObjNum 831 snap 263 seq_num 831) 2024-09-11T03:47:51.084 INFO:tasks.rados.rados.2.smithi175.stdout:2537: rollback oid 30 current snap is 276 2024-09-11T03:47:51.084 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 30 to 275 2024-09-11T03:47:51.084 INFO:tasks.rados.rados.2.smithi175.stdout:2538: rollback oid 22 current snap is 276 2024-09-11T03:47:51.084 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 22 to 274 2024-09-11T03:47:51.084 INFO:tasks.rados.rados.2.smithi175.stdout:2539: write oid 46 current snap is 276 2024-09-11T03:47:51.084 INFO:tasks.rados.rados.2.smithi175.stdout:2539: seq_num 876 ranges {722932=498719,1994932=501961} 2024-09-11T03:47:51.090 INFO:tasks.rados.rados.2.smithi175.stdout:2539: writing smithi17557870-46 from 722932 to 1221651 tid 1 2024-09-11T03:47:51.097 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:51.098 INFO:tasks.rados.rados.2.smithi175.stdout:2539: writing smithi17557870-46 from 1994932 to 2496893 tid 2 2024-09-11T03:47:51.099 INFO:tasks.rados.rados.2.smithi175.stdout:2540: snap_create 2024-09-11T03:47:51.099 INFO:tasks.rados.rados.2.smithi175.stdout:2538: finishing rollback tid 0 to smithi17557870-22 2024-09-11T03:47:51.099 INFO:tasks.rados.rados.2.smithi175.stdout:2537: finishing rollback tid 0 to smithi17557870-30 2024-09-11T03:47:51.099 INFO:tasks.rados.rados.2.smithi175.stdout:2537: finishing rollback tid 1 to smithi17557870-30 2024-09-11T03:47:51.099 INFO:tasks.rados.rados.2.smithi175.stdout:2538: finishing rollback tid 1 to smithi17557870-22 2024-09-11T03:47:51.099 INFO:tasks.rados.rados.2.smithi175.stdout:2538: finishing rollback tid 2 to smithi17557870-22 2024-09-11T03:47:51.099 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 961 (ObjNum 845 snap 266 seq_num 845) dirty exists 2024-09-11T03:47:51.099 INFO:tasks.rados.rados.2.smithi175.stdout:2537: finishing rollback tid 2 to smithi17557870-30 2024-09-11T03:47:51.099 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 895 (ObjNum 800 snap 255 seq_num 800) dirty exists 2024-09-11T03:47:51.099 INFO:tasks.rados.rados.2.smithi175.stdout:2536: expect (ObjNum 831 snap 263 seq_num 831) 2024-09-11T03:47:51.119 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:51.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:51.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:51.221 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:51 smithi175 ceph-mon[30894]: 2.20 scrub starts 2024-09-11T03:47:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:51 smithi175 ceph-mon[30894]: 2.20 scrub ok 2024-09-11T03:47:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:51 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:47:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:51 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:47:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:51 smithi175 ceph-mon[30894]: 1.16 scrub starts 2024-09-11T03:47:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:51 smithi175 ceph-mon[30894]: 1.16 scrub ok 2024-09-11T03:47:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:51 smithi175 ceph-mon[30894]: 2.13 scrub starts 2024-09-11T03:47:51.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:51 smithi175 ceph-mon[30894]: 2.13 scrub ok 2024-09-11T03:47:51.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2918029683' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:47:51.231 INFO:tasks.rados.rados.2.smithi175.stdout:2539: finishing write tid 1 to smithi17557870-46 2024-09-11T03:47:51.231 INFO:tasks.rados.rados.2.smithi175.stdout:2539: finishing write tid 2 to smithi17557870-46 2024-09-11T03:47:51.231 INFO:tasks.rados.rados.2.smithi175.stdout:2539: finishing write tid 3 to smithi17557870-46 2024-09-11T03:47:51.231 INFO:tasks.rados.rados.2.smithi175.stdout:2539: finishing write tid 4 to smithi17557870-46 2024-09-11T03:47:51.231 INFO:tasks.rados.rados.2.smithi175.stdout:2539: oid 46 updating version 0 to 884 2024-09-11T03:47:51.231 INFO:tasks.rados.rados.2.smithi175.stdout:2539: oid 46 version 884 is already newer than 883 2024-09-11T03:47:51.231 INFO:tasks.rados.rados.2.smithi175.stdout:2539: oid 46 updating version 884 to 885 2024-09-11T03:47:51.232 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 885 (ObjNum 876 snap 276 seq_num 876) dirty exists 2024-09-11T03:47:51.232 INFO:tasks.rados.rados.2.smithi175.stdout:2539: left oid 46 (ObjNum 876 snap 276 seq_num 876) 2024-09-11T03:47:51.241 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:51.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:51.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:51.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:51 smithi121 ceph-mon[30729]: 2.20 scrub starts 2024-09-11T03:47:51.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:51 smithi121 ceph-mon[30729]: 2.20 scrub ok 2024-09-11T03:47:51.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:51 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:47:51.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:51 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:47:51.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:51 smithi121 ceph-mon[30729]: 1.16 scrub starts 2024-09-11T03:47:51.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:51 smithi121 ceph-mon[30729]: 1.16 scrub ok 2024-09-11T03:47:51.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:51 smithi121 ceph-mon[30729]: 2.13 scrub starts 2024-09-11T03:47:51.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:51 smithi121 ceph-mon[30729]: 2.13 scrub ok 2024-09-11T03:47:51.288 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2918029683' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:47:51.343 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:51.364 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:51.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:51.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:51 smithi084 ceph-mon[27254]: 2.20 scrub starts 2024-09-11T03:47:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:51 smithi084 ceph-mon[27254]: 2.20 scrub ok 2024-09-11T03:47:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:51 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:47:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:51 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:47:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:51 smithi084 ceph-mon[27254]: 1.16 scrub starts 2024-09-11T03:47:51.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:51 smithi084 ceph-mon[27254]: 1.16 scrub ok 2024-09-11T03:47:51.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:51 smithi084 ceph-mon[27254]: 2.13 scrub starts 2024-09-11T03:47:51.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:51 smithi084 ceph-mon[27254]: 2.13 scrub ok 2024-09-11T03:47:51.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2918029683' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:47:51.466 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:51.487 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:51.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:51.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:51.611 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:51.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:51.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:51.726 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:47:51.735 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:51.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:51.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T03:47:51.836 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:51.858 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:51.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:51.859 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:51.960 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:51.976 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:51.981 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:51.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:51.982 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:51.985 INFO:tasks.rados.rados.2.smithi175.stdout:2536: done (4 left) 2024-09-11T03:47:51.986 INFO:tasks.rados.rados.2.smithi175.stdout:2537: done (3 left) 2024-09-11T03:47:51.986 INFO:tasks.rados.rados.2.smithi175.stdout:2538: done (2 left) 2024-09-11T03:47:51.986 INFO:tasks.rados.rados.2.smithi175.stdout:2539: done (1 left) 2024-09-11T03:47:51.986 INFO:tasks.rados.rados.2.smithi175.stdout:2540: done (0 left) 2024-09-11T03:47:51.986 INFO:tasks.rados.rados.2.smithi175.stdout:2541: write oid 19 current snap is 277 2024-09-11T03:47:51.986 INFO:tasks.rados.rados.2.smithi175.stdout:2541: seq_num 877 ranges {739739=573968,2053414=686325} 2024-09-11T03:47:51.992 INFO:tasks.rados.rados.2.smithi175.stdout:2541: writing smithi17557870-19 from 739739 to 1313707 tid 1 2024-09-11T03:47:52.000 INFO:tasks.rados.rados.2.smithi175.stdout:2541: writing smithi17557870-19 from 2053414 to 2739739 tid 2 2024-09-11T03:47:52.001 INFO:tasks.rados.rados.2.smithi175.stdout:2542: snap_create 2024-09-11T03:47:52.001 INFO:tasks.rados.rados.2.smithi175.stdout:2541: finishing write tid 1 to smithi17557870-19 2024-09-11T03:47:52.004 INFO:tasks.rados.rados.2.smithi175.stdout:2541: finishing write tid 2 to smithi17557870-19 2024-09-11T03:47:52.004 INFO:tasks.rados.rados.2.smithi175.stdout:2541: finishing write tid 3 to smithi17557870-19 2024-09-11T03:47:52.004 INFO:tasks.rados.rados.2.smithi175.stdout:2541: finishing write tid 4 to smithi17557870-19 2024-09-11T03:47:52.004 INFO:tasks.rados.rados.2.smithi175.stdout:2541: oid 19 updating version 0 to 543 2024-09-11T03:47:52.004 INFO:tasks.rados.rados.2.smithi175.stdout:2541: oid 19 updating version 543 to 544 2024-09-11T03:47:52.004 INFO:tasks.rados.rados.2.smithi175.stdout:2541: oid 19 version 544 is already newer than 542 2024-09-11T03:47:52.004 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 544 (ObjNum 877 snap 277 seq_num 877) dirty exists 2024-09-11T03:47:52.004 INFO:tasks.rados.rados.2.smithi175.stdout:2541: left oid 19 (ObjNum 877 snap 277 seq_num 877) 2024-09-11T03:47:52.027 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:47:52.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:52.104 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:52.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:52.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:52.206 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:52.228 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:52.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:52.229 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:52.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:52.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:52 smithi121 ceph-mon[30729]: pgmap v1134: 67 pgs: 67 active+clean; 122 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 589 KiB/s wr, 6 op/s 2024-09-11T03:47:52.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:52 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:52.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:52 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2918029683' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:47:52.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:52 smithi121 ceph-mon[30729]: osdmap e653: 12 total, 12 up, 12 in 2024-09-11T03:47:52.331 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:52 smithi121 ceph-mon[30729]: osdmap e654: 12 total, 12 up, 12 in 2024-09-11T03:47:52.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:52.352 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:52.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:52.381 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:52 smithi084 ceph-mon[27254]: pgmap v1134: 67 pgs: 67 active+clean; 122 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 589 KiB/s wr, 6 op/s 2024-09-11T03:47:52.381 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:52 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:52.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:52 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2918029683' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:47:52.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:52 smithi084 ceph-mon[27254]: osdmap e653: 12 total, 12 up, 12 in 2024-09-11T03:47:52.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:52 smithi084 ceph-mon[27254]: osdmap e654: 12 total, 12 up, 12 in 2024-09-11T03:47:52.454 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:47:52.477 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:52 smithi175 ceph-mon[30894]: pgmap v1134: 67 pgs: 67 active+clean; 122 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 589 KiB/s wr, 6 op/s 2024-09-11T03:47:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:52 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:52 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2918029683' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:47:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:52 smithi175 ceph-mon[30894]: osdmap e653: 12 total, 12 up, 12 in 2024-09-11T03:47:52.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:52 smithi175 ceph-mon[30894]: osdmap e654: 12 total, 12 up, 12 in 2024-09-11T03:47:52.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:52.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:47:52.481 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:47:52.481 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:47:52.495 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:47:52.495 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:47:52.511 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:47:52.511 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:47:52.527 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:47:52.527 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:47:52.543 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:47:52.543 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:47:52.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:47:52.560 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:47:52.560 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:47:52.560 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:47:52.561 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:47:52.561 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:47:52.561 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:47:52.576 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:47:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:47:52.579 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:52.593 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:47:52.593 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:47:52.600 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:52.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:52.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:52.609 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:47:52.609 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:47:52.625 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:47:52.626 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:47:52.642 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:47:52.642 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:47:52.658 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:47:52.659 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:47:52.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:52.723 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:52.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:52.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:52.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:52.849 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:52.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:52.950 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:52.971 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:52.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:52.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:52.987 INFO:tasks.rados.rados.2.smithi175.stdout:2541: done (1 left) 2024-09-11T03:47:52.987 INFO:tasks.rados.rados.2.smithi175.stdout:2542: done (0 left) 2024-09-11T03:47:52.987 INFO:tasks.rados.rados.2.smithi175.stdout:2543: rollback oid 43 current snap is 278 2024-09-11T03:47:52.987 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 43 to 275 2024-09-11T03:47:52.988 INFO:tasks.rados.rados.2.smithi175.stdout:2544: snap_create 2024-09-11T03:47:52.991 INFO:tasks.rados.rados.2.smithi175.stdout:2543: finishing rollback tid 0 to smithi17557870-43 2024-09-11T03:47:52.991 INFO:tasks.rados.rados.2.smithi175.stdout:2543: finishing rollback tid 1 to smithi17557870-43 2024-09-11T03:47:52.991 INFO:tasks.rados.rados.2.smithi175.stdout:2543: finishing rollback tid 2 to smithi17557870-43 2024-09-11T03:47:52.991 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 462 (ObjNum 848 snap 267 seq_num 848) dirty exists 2024-09-11T03:47:53.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:53.095 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:53.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:53.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:53.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:53.218 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:53.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:53.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:53.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T03:47:53.320 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:53.341 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:53.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:53.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:53 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1292493125' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:53.369 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:53 smithi084 ceph-mon[27254]: osdmap e655: 12 total, 12 up, 12 in 2024-09-11T03:47:53.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:53 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1292493125' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:53.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:53 smithi121 ceph-mon[30729]: osdmap e655: 12 total, 12 up, 12 in 2024-09-11T03:47:53.443 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:53.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:53.465 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:53.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:53.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:53 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1292493125' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:53.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:53 smithi175 ceph-mon[30894]: osdmap e655: 12 total, 12 up, 12 in 2024-09-11T03:47:53.518 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:47:53.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:53.588 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:53.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:53.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:53.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:53.711 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:53.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:53.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:53.813 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:53.839 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:53.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:53.839 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:53.866 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:47:53.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:53.961 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:53.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:53.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:53.990 INFO:tasks.rados.rados.2.smithi175.stdout:2543: done (1 left) 2024-09-11T03:47:53.990 INFO:tasks.rados.rados.2.smithi175.stdout:2544: done (0 left) 2024-09-11T03:47:53.990 INFO:tasks.rados.rados.2.smithi175.stdout:2545: snap_remove snap 277 2024-09-11T03:47:54.017 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:47:54.017 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:47:54.063 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:54.085 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:54.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:54.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:54.112 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:54 smithi084 ceph-mon[27254]: pgmap v1137: 67 pgs: 67 active+clean; 116 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-11T03:47:54.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:54 smithi084 ceph-mon[27254]: osdmap e656: 12 total, 12 up, 12 in 2024-09-11T03:47:54.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:54 smithi121 ceph-mon[30729]: pgmap v1137: 67 pgs: 67 active+clean; 116 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-11T03:47:54.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:54 smithi121 ceph-mon[30729]: osdmap e656: 12 total, 12 up, 12 in 2024-09-11T03:47:54.187 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:54.213 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:54.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:54.214 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:54.315 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:54.340 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:54.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:54.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:54.442 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:54.467 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:54.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:54.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:54.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:54 smithi175 ceph-mon[30894]: pgmap v1137: 67 pgs: 67 active+clean; 116 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-11T03:47:54.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:54 smithi175 ceph-mon[30894]: osdmap e656: 12 total, 12 up, 12 in 2024-09-11T03:47:54.570 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:54.592 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:54.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:54.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:54.694 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:54.704 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-11T03:47:54.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:54.721 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:54.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:54.821 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:54.843 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:54.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:54.844 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:54.945 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:54.967 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:54.967 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:54.975 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:47:54.987 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:47:54.991 INFO:tasks.rados.rados.2.smithi175.stdout:2545: done (0 left) 2024-09-11T03:47:54.991 INFO:tasks.rados.rados.2.smithi175.stdout:2546: read oid 13 snap -1 2024-09-11T03:47:54.991 INFO:tasks.rados.rados.2.smithi175.stdout:2546: expect (ObjNum 861 snap 273 seq_num 861) 2024-09-11T03:47:54.991 INFO:tasks.rados.rados.2.smithi175.stdout:2547: delete oid 31 current snap is 279 2024-09-11T03:47:54.994 INFO:tasks.rados.rados.2.smithi175.stdout:2547: done (1 left) 2024-09-11T03:47:54.994 INFO:tasks.rados.rados.2.smithi175.stdout:2548: snap_create 2024-09-11T03:47:54.995 INFO:tasks.rados.rados.2.smithi175.stdout:2546: expect (ObjNum 861 snap 273 seq_num 861) 2024-09-11T03:47:55.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:55.095 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:55.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:55.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:55.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:55.220 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:55.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:55.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:55.300 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:55 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4085788921' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:47:55.300 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:55 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4085788921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:47:55.300 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:55 smithi084 ceph-mon[27254]: osdmap e657: 12 total, 12 up, 12 in 2024-09-11T03:47:55.322 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:55.343 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:55.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:55.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:55.396 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:55 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4085788921' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:47:55.396 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:55 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4085788921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:47:55.396 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:55 smithi175 ceph-mon[30894]: osdmap e657: 12 total, 12 up, 12 in 2024-09-11T03:47:55.398 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:47:55.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:55 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4085788921' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:47:55.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:55 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4085788921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:47:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:55 smithi121 ceph-mon[30729]: osdmap e657: 12 total, 12 up, 12 in 2024-09-11T03:47:55.445 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:55.465 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:55.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:55.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:55.496 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:47:55.496 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:47:55.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:55.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:55.594 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:55.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:55.695 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:55.717 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:55.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:55.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:55.723 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:55.819 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:55.840 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:55.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:55.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:55.942 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:55.966 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:55.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:55.991 INFO:tasks.rados.rados.2.smithi175.stdout:2546: done (1 left) 2024-09-11T03:47:55.992 INFO:tasks.rados.rados.2.smithi175.stdout:2548: done (0 left) 2024-09-11T03:47:55.992 INFO:tasks.rados.rados.2.smithi175.stdout:2549: read oid 22 snap -1 2024-09-11T03:47:55.992 INFO:tasks.rados.rados.2.smithi175.stdout:2549: expect (ObjNum 845 snap 266 seq_num 845) 2024-09-11T03:47:55.992 INFO:tasks.rados.rados.2.smithi175.stdout:2550: delete oid 12 current snap is 280 2024-09-11T03:47:55.994 INFO:tasks.rados.rados.2.smithi175.stdout:2550: done (1 left) 2024-09-11T03:47:55.994 INFO:tasks.rados.rados.2.smithi175.stdout:2551: read oid 48 snap -1 2024-09-11T03:47:55.994 INFO:tasks.rados.rados.2.smithi175.stdout:2551: expect (ObjNum 845 snap 266 seq_num 845) 2024-09-11T03:47:55.994 INFO:tasks.rados.rados.2.smithi175.stdout:2552: copy_from oid 7 from oid 26 current snap is 280 2024-09-11T03:47:55.994 INFO:tasks.rados.rados.2.smithi175.stdout:2553: snap_remove snap 276 2024-09-11T03:47:56.067 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:47:56.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:56.088 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:56.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:47:56.159 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:56 smithi175 ceph-mon[30894]: pgmap v1140: 67 pgs: 67 active+clean; 116 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 3.5 MiB/s wr, 27 op/s 2024-09-11T03:47:56.159 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:56 smithi175 ceph-mon[30894]: osdmap e658: 12 total, 12 up, 12 in 2024-09-11T03:47:56.185 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-11T03:47:56.189 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:56.204 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:47:56.204 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:47:56.204 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-11T03:47:56.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:47:56.230 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:47:56.231 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:47:56.246 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:47:56.246 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:47:56.246 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:56 smithi084 ceph-mon[27254]: pgmap v1140: 67 pgs: 67 active+clean; 116 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 3.5 MiB/s wr, 27 op/s 2024-09-11T03:47:56.247 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:56 smithi084 ceph-mon[27254]: osdmap e658: 12 total, 12 up, 12 in 2024-09-11T03:47:56.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:56.247 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:56.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:56.262 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:47:56.263 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:47:56.278 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:47:56.279 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:47:56.296 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:47:56.296 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:47:56.313 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:47:56.313 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:47:56.330 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:47:56.330 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:47:56.347 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:47:56.347 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:47:56.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:56.365 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:47:56.365 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:47:56.369 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:56.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:56.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:56.381 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:47:56.382 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:47:56.400 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:47:56.400 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:47:56.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:47:56.417 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:47:56.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:56 smithi121 ceph-mon[30729]: pgmap v1140: 67 pgs: 67 active+clean; 116 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 3.5 MiB/s wr, 27 op/s 2024-09-11T03:47:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:56 smithi121 ceph-mon[30729]: osdmap e658: 12 total, 12 up, 12 in 2024-09-11T03:47:56.469 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:47:56.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:56.492 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:56.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:56.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:56.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:56.619 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:56.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:56.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:56.798 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:56.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:56.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:56.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:56.924 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:56.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:56.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:56.945 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:47:56.945 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:47:56.945 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:47:56.945 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:47:56.945 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:47:56.990 INFO:tasks.rados.rados.2.smithi175.stdout:2553: done (3 left) 2024-09-11T03:47:56.990 INFO:tasks.rados.rados.2.smithi175.stdout:2554: snap_remove snap 278 2024-09-11T03:47:57.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:57.047 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:57.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:57.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:57.149 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:47:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:57.172 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:57.172 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:47:57.185 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:57.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:57.299 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:57.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:57.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:57 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4234707072' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:57 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:57 smithi084 ceph-mon[27254]: osdmap e659: 12 total, 12 up, 12 in 2024-09-11T03:47:57.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:57.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:57 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4234707072' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:57.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:57 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:57.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:57 smithi121 ceph-mon[30729]: osdmap e659: 12 total, 12 up, 12 in 2024-09-11T03:47:57.427 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:57.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:57 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4234707072' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:47:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:57 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:47:57.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:57 smithi175 ceph-mon[30894]: osdmap e659: 12 total, 12 up, 12 in 2024-09-11T03:47:57.528 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:57.551 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:57.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:57.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:57.586 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-11T03:47:57.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:57.678 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:57.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:57.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:57.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:47:57.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:57.805 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:57.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:57.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:57.850 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:47:57.850 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:47:57.866 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:47:57.906 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:47:57.932 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:57.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:57.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:47:57.989 INFO:tasks.rados.rados.2.smithi175.stdout:2554: done (3 left) 2024-09-11T03:47:57.989 INFO:tasks.rados.rados.2.smithi175.stdout:2555: rollback oid 18 current snap is 280 2024-09-11T03:47:57.989 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 18 to 275 2024-09-11T03:47:57.990 INFO:tasks.rados.rados.2.smithi175.stdout:2549: expect (ObjNum 845 snap 266 seq_num 845) 2024-09-11T03:47:58.033 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:47:58.054 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:58.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:58.055 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:47:58.116 INFO:tasks.rados.rados.2.smithi175.stdout:2552: finishing copy_from racing read to smithi17557870-7 2024-09-11T03:47:58.116 INFO:tasks.rados.rados.2.smithi175.stdout:2552: finishing copy_from to smithi17557870-7 2024-09-11T03:47:58.116 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 837 (ObjNum 793 snap 252 seq_num 793) dirty exists 2024-09-11T03:47:58.116 INFO:tasks.rados.rados.2.smithi175.stdout:2551: expect (ObjNum 845 snap 266 seq_num 845) 2024-09-11T03:47:58.155 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:58.175 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:58.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:58.177 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:58.242 INFO:tasks.rados.rados.2.smithi175.stdout:2555: finishing rollback tid 0 to smithi17557870-18 2024-09-11T03:47:58.243 INFO:tasks.rados.rados.2.smithi175.stdout:2555: finishing rollback tid 1 to smithi17557870-18 2024-09-11T03:47:58.243 INFO:tasks.rados.rados.2.smithi175.stdout:2555: finishing rollback tid 2 to smithi17557870-18 2024-09-11T03:47:58.243 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 481 (ObjNum 856 snap 269 seq_num 856) dirty exists 2024-09-11T03:47:58.243 INFO:tasks.rados.rados.2.smithi175.stdout:2549: done (3 left) 2024-09-11T03:47:58.243 INFO:tasks.rados.rados.2.smithi175.stdout:2551: done (2 left) 2024-09-11T03:47:58.243 INFO:tasks.rados.rados.2.smithi175.stdout:2552: done (1 left) 2024-09-11T03:47:58.243 INFO:tasks.rados.rados.2.smithi175.stdout:2555: done (0 left) 2024-09-11T03:47:58.243 INFO:tasks.rados.rados.2.smithi175.stdout:2556: snap_create 2024-09-11T03:47:58.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:58.279 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:58 smithi175 ceph-mon[30894]: pgmap v1143: 67 pgs: 67 active+clean; 121 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-11T03:47:58.280 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:58 smithi175 ceph-mon[30894]: osdmap e660: 12 total, 12 up, 12 in 2024-09-11T03:47:58.304 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:58.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:58.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:58.380 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:47:58.380 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:47:58.380 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:47:58.380 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:47:58.380 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.380 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:588 2.f 2:fda69313:::smithi17557870-16:head [copy-from ver 31] snapc 11=[11,f,e] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-11T03:47:58.381 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:05.838050+0000", 2024-09-11T03:47:58.381 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 532.53994057900002, 2024-09-11T03:47:58.381 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017475905, 2024-09-11T03:47:58.381 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.381 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.381 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:58.381 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.382 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.382 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.382 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 588 2024-09-11T03:47:58.382 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.382 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.382 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.382 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.382 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838050+0000", 2024-09-11T03:47:58.382 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.382 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.382 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.382 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.382 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838050+0000", 2024-09-11T03:47:58.383 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.383 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.383 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.383 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.383 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838050+0000", 2024-09-11T03:47:58.383 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2399999999999998e-07 2024-09-11T03:47:58.383 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.383 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.383 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.384 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838054+0000", 2024-09-11T03:47:58.384 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8519999999999997e-06 2024-09-11T03:47:58.384 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.384 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.384 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.384 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838064+0000", 2024-09-11T03:47:58.384 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7559999999999994e-06 2024-09-11T03:47:58.384 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.384 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.384 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.385 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838069+0000", 2024-09-11T03:47:58.385 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4759999999999997e-06 2024-09-11T03:47:58.385 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.385 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.385 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.385 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838118+0000", 2024-09-11T03:47:58.385 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8881999999999997e-05 2024-09-11T03:47:58.385 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.385 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.385 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.385 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838180+0000", 2024-09-11T03:47:58.385 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1569000000000006e-05 2024-09-11T03:47:58.385 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.386 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.386 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,9", 2024-09-11T03:47:58.386 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.847578+0000", 2024-09-11T03:47:58.386 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0093983880000000006 2024-09-11T03:47:58.386 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.386 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.386 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:47:58.386 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.851581+0000", 2024-09-11T03:47:58.386 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0040029929999999998 2024-09-11T03:47:58.386 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.386 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.386 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.386 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.853070+0000", 2024-09-11T03:47:58.387 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014892130000000001 2024-09-11T03:47:58.387 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.387 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.387 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.387 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.855406+0000", 2024-09-11T03:47:58.387 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00233533 2024-09-11T03:47:58.387 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.387 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.387 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:47:58.387 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.855473+0000", 2024-09-11T03:47:58.387 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7726999999999997e-05 2024-09-11T03:47:58.387 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.387 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.387 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.388 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.855526+0000", 2024-09-11T03:47:58.388 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2095000000000002e-05 2024-09-11T03:47:58.388 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.388 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.388 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.388 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.388 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.388 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:589 2.f 2:fda69313:::smithi17557870-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e129)", 2024-09-11T03:47:58.388 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:05.838080+0000", 2024-09-11T03:47:58.388 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 532.53991025100004, 2024-09-11T03:47:58.388 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013050325, 2024-09-11T03:47:58.389 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.389 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.389 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:47:58.389 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.389 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.389 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.389 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 589 2024-09-11T03:47:58.389 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.389 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.389 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.389 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.389 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838080+0000", 2024-09-11T03:47:58.389 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.389 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.390 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.390 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.390 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838080+0000", 2024-09-11T03:47:58.390 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.390 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.390 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.391 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.391 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838081+0000", 2024-09-11T03:47:58.391 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8699999999999995e-07 2024-09-11T03:47:58.391 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.391 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.391 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.391 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838083+0000", 2024-09-11T03:47:58.391 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3690000000000001e-06 2024-09-11T03:47:58.391 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.391 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.391 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.391 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838090+0000", 2024-09-11T03:47:58.391 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.669e-06 2024-09-11T03:47:58.391 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.391 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.392 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.392 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838095+0000", 2024-09-11T03:47:58.392 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1850000000000003e-06 2024-09-11T03:47:58.392 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.392 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.392 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.392 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838299+0000", 2024-09-11T03:47:58.392 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00020415400000000001 2024-09-11T03:47:58.392 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.392 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.392 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:58.392 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.838327+0000", 2024-09-11T03:47:58.392 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8045e-05 2024-09-11T03:47:58.392 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.393 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.393 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.393 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.850913+0000", 2024-09-11T03:47:58.393 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012586409 2024-09-11T03:47:58.393 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.393 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.393 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.393 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.851013+0000", 2024-09-11T03:47:58.393 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9287e-05 2024-09-11T03:47:58.393 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.393 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.393 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.393 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:05.851130+0000", 2024-09-11T03:47:58.393 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011772 2024-09-11T03:47:58.394 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.394 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.394 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.394 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.394 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.394 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1152 2.b 2:de6b1316:::smithi17557870-32:head [copy-from ver 77] snapc 2d=[2d,2c,2b,28,26] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-11T03:47:58.394 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:59.869443+0000", 2024-09-11T03:47:58.394 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 478.508547593, 2024-09-11T03:47:58.394 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015544123999999999, 2024-09-11T03:47:58.394 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.394 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.394 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:58.394 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.394 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.394 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.394 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1152 2024-09-11T03:47:58.395 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.395 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.395 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.395 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.395 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869443+0000", 2024-09-11T03:47:58.395 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.395 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.395 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.395 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.395 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869443+0000", 2024-09-11T03:47:58.395 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.395 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.395 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.396 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.396 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869444+0000", 2024-09-11T03:47:58.396 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3829999999999999e-06 2024-09-11T03:47:58.396 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.396 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.396 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.396 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869454+0000", 2024-09-11T03:47:58.396 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0166999999999999e-05 2024-09-11T03:47:58.396 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.396 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.396 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.396 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869467+0000", 2024-09-11T03:47:58.396 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3243e-05 2024-09-11T03:47:58.397 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.397 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.397 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.397 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869475+0000", 2024-09-11T03:47:58.397 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7209999999999994e-06 2024-09-11T03:47:58.397 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.397 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.397 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.397 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869537+0000", 2024-09-11T03:47:58.397 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2075000000000005e-05 2024-09-11T03:47:58.397 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.397 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.397 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.398 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869625+0000", 2024-09-11T03:47:58.398 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7736000000000002e-05 2024-09-11T03:47:58.398 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.398 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.398 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,11", 2024-09-11T03:47:58.398 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.878093+0000", 2024-09-11T03:47:58.398 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0084675710000000001 2024-09-11T03:47:58.398 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.398 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.399 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:47:58.399 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.882577+0000", 2024-09-11T03:47:58.399 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0044839349999999997 2024-09-11T03:47:58.399 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.399 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.399 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.399 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.883819+0000", 2024-09-11T03:47:58.399 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012421680000000001 2024-09-11T03:47:58.399 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.399 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.399 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.399 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.884915+0000", 2024-09-11T03:47:58.400 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0010967559999999999 2024-09-11T03:47:58.400 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.400 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.400 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:47:58.400 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.884952+0000", 2024-09-11T03:47:58.400 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6835999999999998e-05 2024-09-11T03:47:58.400 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.401 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.401 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.401 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.884987+0000", 2024-09-11T03:47:58.401 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4533000000000003e-05 2024-09-11T03:47:58.401 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.401 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.401 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.401 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.401 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.401 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1513 2.5 2:a6e3bbd9:::smithi17557870-38:head [copy-from ver 208] snapc 3b=[3a,38,37,35] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-09-11T03:47:58.401 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:28.917633+0000", 2024-09-11T03:47:58.402 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 449.46035743800002, 2024-09-11T03:47:58.402 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021282327, 2024-09-11T03:47:58.402 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.402 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.402 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:58.402 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.402 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.402 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.402 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1513 2024-09-11T03:47:58.402 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.402 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.402 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.402 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.402 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917633+0000", 2024-09-11T03:47:58.403 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.403 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.403 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.403 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.403 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917633+0000", 2024-09-11T03:47:58.403 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.403 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.403 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.403 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.403 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917635+0000", 2024-09-11T03:47:58.403 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6390000000000001e-06 2024-09-11T03:47:58.403 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.403 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.403 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.404 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917649+0000", 2024-09-11T03:47:58.404 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3721e-05 2024-09-11T03:47:58.404 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.404 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.404 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.404 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917669+0000", 2024-09-11T03:47:58.404 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9423999999999999e-05 2024-09-11T03:47:58.404 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.404 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.404 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.404 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917676+0000", 2024-09-11T03:47:58.404 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7330000000000003e-06 2024-09-11T03:47:58.404 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.404 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.405 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.405 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917747+0000", 2024-09-11T03:47:58.405 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1081999999999995e-05 2024-09-11T03:47:58.405 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.405 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.405 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.405 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917816+0000", 2024-09-11T03:47:58.405 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8504999999999995e-05 2024-09-11T03:47:58.405 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.405 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.405 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:47:58.405 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.928464+0000", 2024-09-11T03:47:58.405 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.01064815 2024-09-11T03:47:58.406 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.406 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.406 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:47:58.406 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.937323+0000", 2024-09-11T03:47:58.406 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0088585160000000003 2024-09-11T03:47:58.406 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.406 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.406 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.406 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.938350+0000", 2024-09-11T03:47:58.406 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00102728 2024-09-11T03:47:58.406 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.406 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.407 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.407 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.938848+0000", 2024-09-11T03:47:58.407 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00049766199999999995 2024-09-11T03:47:58.407 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.407 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.407 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:47:58.407 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.938884+0000", 2024-09-11T03:47:58.407 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6916999999999999e-05 2024-09-11T03:47:58.407 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.407 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.407 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.407 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.938915+0000", 2024-09-11T03:47:58.407 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0698000000000003e-05 2024-09-11T03:47:58.408 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.408 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.408 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.408 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.408 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.408 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1514 2.5 2:a6e3bbd9:::smithi17557870-38:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2024-09-11T03:47:58.408 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:28.917699+0000", 2024-09-11T03:47:58.408 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 449.46029107800001, 2024-09-11T03:47:58.408 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017658610000000002, 2024-09-11T03:47:58.408 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.408 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.408 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:47:58.408 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.409 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.409 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.409 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1514 2024-09-11T03:47:58.409 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.409 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.409 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.409 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.409 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917699+0000", 2024-09-11T03:47:58.409 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.409 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.409 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.409 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.410 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917699+0000", 2024-09-11T03:47:58.410 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.410 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.410 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.410 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.410 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917700+0000", 2024-09-11T03:47:58.410 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.99e-07 2024-09-11T03:47:58.410 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.410 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.410 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.410 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917703+0000", 2024-09-11T03:47:58.410 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8260000000000001e-06 2024-09-11T03:47:58.410 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.410 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.411 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.411 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917713+0000", 2024-09-11T03:47:58.411 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0539e-05 2024-09-11T03:47:58.411 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.411 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.411 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.411 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917719+0000", 2024-09-11T03:47:58.411 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1500000000000004e-06 2024-09-11T03:47:58.411 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.411 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.411 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.411 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917971+0000", 2024-09-11T03:47:58.411 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00025184900000000001 2024-09-11T03:47:58.412 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:58.413 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:58 smithi084 ceph-mon[27254]: pgmap v1143: 67 pgs: 67 active+clean; 121 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-11T03:47:58.413 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:58 smithi084 ceph-mon[27254]: osdmap e660: 12 total, 12 up, 12 in 2024-09-11T03:47:58.414 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.414 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.414 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:58.414 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.918032+0000", 2024-09-11T03:47:58.414 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0449999999999999e-05 2024-09-11T03:47:58.414 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.414 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.414 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.414 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.935206+0000", 2024-09-11T03:47:58.415 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017174405 2024-09-11T03:47:58.415 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.415 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.415 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.415 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.935277+0000", 2024-09-11T03:47:58.415 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0862e-05 2024-09-11T03:47:58.415 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.415 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.415 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.416 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.935358+0000", 2024-09-11T03:47:58.416 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0929999999999999e-05 2024-09-11T03:47:58.416 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.416 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.416 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.416 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.416 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.416 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1583 2.b 2:de6b1316:::smithi17557870-32:head [copy-from ver 82] snapc 3f=[3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-09-11T03:47:58.416 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:34.845880+0000", 2024-09-11T03:47:58.416 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 443.53211065900001, 2024-09-11T03:47:58.416 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020941156999999998, 2024-09-11T03:47:58.416 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.417 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.417 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:58.417 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.417 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.417 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.417 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1583 2024-09-11T03:47:58.417 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.417 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.417 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.417 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.417 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845880+0000", 2024-09-11T03:47:58.417 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.417 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.418 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.418 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.418 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845880+0000", 2024-09-11T03:47:58.418 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.418 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.418 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.418 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.418 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845881+0000", 2024-09-11T03:47:58.418 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.871e-06 2024-09-11T03:47:58.418 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.418 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.418 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.418 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845891+0000", 2024-09-11T03:47:58.419 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2410000000000001e-06 2024-09-11T03:47:58.419 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.419 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.419 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.419 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845909+0000", 2024-09-11T03:47:58.419 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8162999999999999e-05 2024-09-11T03:47:58.419 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.419 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.419 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.419 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845919+0000", 2024-09-11T03:47:58.419 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9869999999999994e-06 2024-09-11T03:47:58.419 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.419 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.420 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.420 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845994+0000", 2024-09-11T03:47:58.420 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4757000000000003e-05 2024-09-11T03:47:58.420 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.420 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.420 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:47:58.420 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845999+0000", 2024-09-11T03:47:58.420 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.434e-06 2024-09-11T03:47:58.420 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.420 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.420 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.420 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.847498+0000", 2024-09-11T03:47:58.420 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014989370000000001 2024-09-11T03:47:58.421 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.421 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.421 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.421 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.847585+0000", 2024-09-11T03:47:58.421 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7058999999999994e-05 2024-09-11T03:47:58.421 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.421 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.421 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,11", 2024-09-11T03:47:58.421 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.860229+0000", 2024-09-11T03:47:58.421 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012643765 2024-09-11T03:47:58.421 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.421 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.421 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:47:58.422 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.864139+0000", 2024-09-11T03:47:58.422 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039105019999999997 2024-09-11T03:47:58.422 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.422 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.422 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.422 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.865334+0000", 2024-09-11T03:47:58.422 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001194938 2024-09-11T03:47:58.422 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.422 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.422 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.422 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.866748+0000", 2024-09-11T03:47:58.422 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001413873 2024-09-11T03:47:58.422 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.422 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.423 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:47:58.423 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.866787+0000", 2024-09-11T03:47:58.423 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8571000000000001e-05 2024-09-11T03:47:58.423 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.423 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.423 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.423 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.866821+0000", 2024-09-11T03:47:58.423 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4059e-05 2024-09-11T03:47:58.423 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.423 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.423 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.423 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.423 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.424 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1584 2.b 2:de6b1316:::smithi17557870-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e218)", 2024-09-11T03:47:58.424 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:34.845942+0000", 2024-09-11T03:47:58.424 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 443.53204871100002, 2024-09-11T03:47:58.424 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017600596999999999, 2024-09-11T03:47:58.424 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.424 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.424 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:47:58.424 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.424 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.424 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.424 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1584 2024-09-11T03:47:58.424 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.424 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.425 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.425 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.425 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845942+0000", 2024-09-11T03:47:58.425 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.425 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.425 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.425 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.425 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845942+0000", 2024-09-11T03:47:58.425 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.425 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.425 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.425 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.426 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845942+0000", 2024-09-11T03:47:58.426 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8100000000000003e-07 2024-09-11T03:47:58.426 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.426 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.426 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.426 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845944+0000", 2024-09-11T03:47:58.426 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0899999999999999e-06 2024-09-11T03:47:58.426 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.426 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.426 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.426 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845949+0000", 2024-09-11T03:47:58.426 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3129999999999998e-06 2024-09-11T03:47:58.426 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.426 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.427 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.427 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845953+0000", 2024-09-11T03:47:58.427 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8060000000000001e-06 2024-09-11T03:47:58.427 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.427 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.427 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.427 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.846055+0000", 2024-09-11T03:47:58.427 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010187999999999999 2024-09-11T03:47:58.427 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.427 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.427 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:58.427 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.846062+0000", 2024-09-11T03:47:58.427 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6239999999999996e-06 2024-09-11T03:47:58.427 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.428 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.428 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.428 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.847720+0000", 2024-09-11T03:47:58.428 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001658681 2024-09-11T03:47:58.428 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.428 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.428 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:58.428 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.847773+0000", 2024-09-11T03:47:58.428 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2710999999999997e-05 2024-09-11T03:47:58.428 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.428 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.428 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.428 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.863362+0000", 2024-09-11T03:47:58.429 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015588493 2024-09-11T03:47:58.429 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.429 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.429 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.429 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.863447+0000", 2024-09-11T03:47:58.429 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5682999999999999e-05 2024-09-11T03:47:58.429 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.429 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.429 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.429 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.863542+0000", 2024-09-11T03:47:58.429 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4834999999999996e-05 2024-09-11T03:47:58.429 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.430 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.430 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.430 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.430 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.430 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1607 2.f 2:fa239c32:::smithi17557870-4:head [copy-from ver 82] snapc 40=[40,3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:47:58.430 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:35.897501+0000", 2024-09-11T03:47:58.430 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 442.480489014, 2024-09-11T03:47:58.430 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018113787999999999, 2024-09-11T03:47:58.430 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.430 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.430 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:58.430 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.430 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.430 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.431 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1607 2024-09-11T03:47:58.431 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.431 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.431 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.431 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.431 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897501+0000", 2024-09-11T03:47:58.431 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.431 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.431 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.431 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.431 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897501+0000", 2024-09-11T03:47:58.431 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897503+0000", 2024-09-11T03:47:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8840000000000001e-06 2024-09-11T03:47:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897510+0000", 2024-09-11T03:47:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4740000000000002e-06 2024-09-11T03:47:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897525+0000", 2024-09-11T03:47:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5702e-05 2024-09-11T03:47:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897532+0000", 2024-09-11T03:47:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7329999999999997e-06 2024-09-11T03:47:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897610+0000", 2024-09-11T03:47:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8153999999999997e-05 2024-09-11T03:47:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:47:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897618+0000", 2024-09-11T03:47:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5630000000000003e-06 2024-09-11T03:47:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.898813+0000", 2024-09-11T03:47:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0011955819999999999 2024-09-11T03:47:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.898858+0000", 2024-09-11T03:47:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5092000000000001e-05 2024-09-11T03:47:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,9", 2024-09-11T03:47:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.908861+0000", 2024-09-11T03:47:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010003025 2024-09-11T03:47:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:47:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.912807+0000", 2024-09-11T03:47:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039458339999999996 2024-09-11T03:47:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.914051+0000", 2024-09-11T03:47:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012439040000000001 2024-09-11T03:47:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.436 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.436 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.915500+0000", 2024-09-11T03:47:58.436 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014488839999999999 2024-09-11T03:47:58.436 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.436 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.436 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:47:58.436 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.915561+0000", 2024-09-11T03:47:58.436 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1290000000000004e-05 2024-09-11T03:47:58.436 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.436 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.436 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.436 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.915615+0000", 2024-09-11T03:47:58.436 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3667e-05 2024-09-11T03:47:58.437 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.437 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.437 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.437 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.437 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.437 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1608 2.f 2:fa239c32:::smithi17557870-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:47:58.437 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:35.897552+0000", 2024-09-11T03:47:58.437 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 442.48043870599997, 2024-09-11T03:47:58.437 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014781580000000001, 2024-09-11T03:47:58.437 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.437 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.437 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:47:58.438 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.438 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.438 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.438 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1608 2024-09-11T03:47:58.438 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.438 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.438 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.438 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.438 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897552+0000", 2024-09-11T03:47:58.438 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.438 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.438 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.438 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897552+0000", 2024-09-11T03:47:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897552+0000", 2024-09-11T03:47:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3399999999999999e-07 2024-09-11T03:47:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897558+0000", 2024-09-11T03:47:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7769999999999999e-06 2024-09-11T03:47:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897565+0000", 2024-09-11T03:47:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7410000000000003e-06 2024-09-11T03:47:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897568+0000", 2024-09-11T03:47:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6799999999999999e-06 2024-09-11T03:47:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897671+0000", 2024-09-11T03:47:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000102291 2024-09-11T03:47:58.441 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.441 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.441 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:58.441 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897674+0000", 2024-09-11T03:47:58.441 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6890000000000002e-06 2024-09-11T03:47:58.441 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.441 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.441 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.441 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.898967+0000", 2024-09-11T03:47:58.441 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001292996 2024-09-11T03:47:58.441 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.441 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.441 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.899052+0000", 2024-09-11T03:47:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5112999999999996e-05 2024-09-11T03:47:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.912099+0000", 2024-09-11T03:47:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013046775 2024-09-11T03:47:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.912198+0000", 2024-09-11T03:47:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8659000000000005e-05 2024-09-11T03:47:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.443 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.443 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.912333+0000", 2024-09-11T03:47:58.443 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000135325 2024-09-11T03:47:58.443 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.443 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.443 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.443 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.443 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.443 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1877 2.b 2:de6b1316:::smithi17557870-32:head [copy-from ver 223] snapc 4d=[4d,4b,49,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-11T03:47:58.443 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:03.907218+0000", 2024-09-11T03:47:58.443 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 414.47077210899999, 2024-09-11T03:47:58.443 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019706267999999999, 2024-09-11T03:47:58.443 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.444 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.444 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:58.444 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.444 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.444 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.444 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1877 2024-09-11T03:47:58.444 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.444 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.444 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.444 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.444 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907218+0000", 2024-09-11T03:47:58.444 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.444 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.445 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.445 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.445 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907218+0000", 2024-09-11T03:47:58.445 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.445 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.445 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.445 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.445 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907219+0000", 2024-09-11T03:47:58.445 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.221e-06 2024-09-11T03:47:58.445 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.445 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.445 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.445 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907226+0000", 2024-09-11T03:47:58.445 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1580000000000002e-06 2024-09-11T03:47:58.446 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.446 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.446 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.446 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907239+0000", 2024-09-11T03:47:58.446 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3865e-05 2024-09-11T03:47:58.446 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.446 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.446 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.446 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907247+0000", 2024-09-11T03:47:58.446 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8520000000000004e-06 2024-09-11T03:47:58.446 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.446 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.446 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.446 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907312+0000", 2024-09-11T03:47:58.447 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4437000000000007e-05 2024-09-11T03:47:58.447 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.447 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.447 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.447 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907417+0000", 2024-09-11T03:47:58.447 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000105698 2024-09-11T03:47:58.447 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.447 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.447 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,11", 2024-09-11T03:47:58.447 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.915919+0000", 2024-09-11T03:47:58.447 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.008501794 2024-09-11T03:47:58.447 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.447 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.448 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:47:58.448 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.920523+0000", 2024-09-11T03:47:58.448 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0046041220000000004 2024-09-11T03:47:58.448 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.448 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.448 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.448 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.920865+0000", 2024-09-11T03:47:58.448 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00034196500000000001 2024-09-11T03:47:58.448 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.448 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.448 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.448 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.926838+0000", 2024-09-11T03:47:58.448 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0059723420000000003 2024-09-11T03:47:58.449 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.449 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.449 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:47:58.449 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.926882+0000", 2024-09-11T03:47:58.449 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4199999999999997e-05 2024-09-11T03:47:58.449 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.449 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.449 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.449 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.926924+0000", 2024-09-11T03:47:58.449 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2614000000000002e-05 2024-09-11T03:47:58.449 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.449 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.449 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.450 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.450 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.450 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2115 2.5 2:a6e3bbd9:::smithi17557870-38:head [copy-from ver 221] snapc 57=[57,56,51,4e] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-11T03:47:58.450 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:24.863058+0000", 2024-09-11T03:47:58.450 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 393.51493179800002, 2024-09-11T03:47:58.450 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.025181835999999999, 2024-09-11T03:47:58.450 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.450 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.450 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:58.450 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.450 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.450 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.450 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2115 2024-09-11T03:47:58.450 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.451 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.451 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.451 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.451 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863058+0000", 2024-09-11T03:47:58.451 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.451 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.451 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.451 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.451 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863058+0000", 2024-09-11T03:47:58.451 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.451 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.451 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.451 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.451 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863061+0000", 2024-09-11T03:47:58.452 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2649999999999999e-06 2024-09-11T03:47:58.452 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.452 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.452 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.452 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863072+0000", 2024-09-11T03:47:58.452 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1409999999999999e-05 2024-09-11T03:47:58.452 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.452 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.452 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.452 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863091+0000", 2024-09-11T03:47:58.452 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8825e-05 2024-09-11T03:47:58.452 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.452 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.453 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.453 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863099+0000", 2024-09-11T03:47:58.453 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7849999999999992e-06 2024-09-11T03:47:58.453 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.453 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.453 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.453 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863164+0000", 2024-09-11T03:47:58.453 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5078000000000003e-05 2024-09-11T03:47:58.453 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.453 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.453 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:47:58.453 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863169+0000", 2024-09-11T03:47:58.453 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1649999999999997e-06 2024-09-11T03:47:58.454 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.454 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.454 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.454 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.864677+0000", 2024-09-11T03:47:58.454 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001507928 2024-09-11T03:47:58.454 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.454 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.454 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.454 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.864744+0000", 2024-09-11T03:47:58.454 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6990999999999996e-05 2024-09-11T03:47:58.454 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.454 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.454 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:47:58.455 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.876551+0000", 2024-09-11T03:47:58.455 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011806891 2024-09-11T03:47:58.455 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.455 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.455 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:47:58.455 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.882152+0000", 2024-09-11T03:47:58.455 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0056009240000000002 2024-09-11T03:47:58.455 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.455 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.455 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.455 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.884525+0000", 2024-09-11T03:47:58.455 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002373436 2024-09-11T03:47:58.455 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.455 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.456 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.456 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.888133+0000", 2024-09-11T03:47:58.456 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003608207 2024-09-11T03:47:58.456 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.456 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.456 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:47:58.456 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.888194+0000", 2024-09-11T03:47:58.456 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1007000000000001e-05 2024-09-11T03:47:58.456 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.456 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.456 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.456 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.888240+0000", 2024-09-11T03:47:58.457 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5924000000000003e-05 2024-09-11T03:47:58.457 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.457 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.457 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.457 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.457 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.457 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2116 2.5 2:a6e3bbd9:::smithi17557870-38:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e268)", 2024-09-11T03:47:58.457 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:24.863121+0000", 2024-09-11T03:47:58.457 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 393.51486927899998, 2024-09-11T03:47:58.457 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018358349999999999, 2024-09-11T03:47:58.457 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.457 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.458 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:47:58.458 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.458 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.458 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.458 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2116 2024-09-11T03:47:58.458 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.458 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.458 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.458 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.458 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863121+0000", 2024-09-11T03:47:58.458 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.458 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.458 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.459 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.459 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863121+0000", 2024-09-11T03:47:58.459 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.459 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.459 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.459 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.459 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863121+0000", 2024-09-11T03:47:58.459 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5900000000000002e-07 2024-09-11T03:47:58.459 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.459 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.459 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.459 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863126+0000", 2024-09-11T03:47:58.459 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8910000000000003e-06 2024-09-11T03:47:58.460 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.460 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.460 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.460 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863133+0000", 2024-09-11T03:47:58.460 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4489999999999996e-06 2024-09-11T03:47:58.460 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.460 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.460 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.460 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863137+0000", 2024-09-11T03:47:58.460 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4549999999999997e-06 2024-09-11T03:47:58.460 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.460 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.460 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.461 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863232+0000", 2024-09-11T03:47:58.461 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4308999999999994e-05 2024-09-11T03:47:58.461 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.461 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.461 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:47:58.461 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863238+0000", 2024-09-11T03:47:58.461 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6279999999999999e-06 2024-09-11T03:47:58.461 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.461 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.461 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.461 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.864857+0000", 2024-09-11T03:47:58.461 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0016184330000000001 2024-09-11T03:47:58.461 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.462 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.462 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:58.462 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.864900+0000", 2024-09-11T03:47:58.462 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3745000000000002e-05 2024-09-11T03:47:58.462 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.462 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.462 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.462 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.881241+0000", 2024-09-11T03:47:58.462 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016340232999999999 2024-09-11T03:47:58.462 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.462 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.462 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.462 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.881352+0000", 2024-09-11T03:47:58.462 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000110945 2024-09-11T03:47:58.463 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.463 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.463 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.463 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.881479+0000", 2024-09-11T03:47:58.463 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012780300000000001 2024-09-11T03:47:58.463 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.463 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.463 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.463 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.463 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.463 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2165 2.b 2:dde92fc6:::smithi17557870-41:head [copy-from ver 101] snapc 58=[58,57,56,51] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-09-11T03:47:58.463 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:26.857685+0000", 2024-09-11T03:47:58.463 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 391.52030488700001, 2024-09-11T03:47:58.464 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022414478000000002, 2024-09-11T03:47:58.464 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.464 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.464 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:58.464 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.464 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.464 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.464 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2165 2024-09-11T03:47:58.464 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.464 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.464 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.464 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.464 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857685+0000", 2024-09-11T03:47:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857685+0000", 2024-09-11T03:47:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857687+0000", 2024-09-11T03:47:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.716e-06 2024-09-11T03:47:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857694+0000", 2024-09-11T03:47:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9909999999999996e-06 2024-09-11T03:47:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857731+0000", 2024-09-11T03:47:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7083999999999999e-05 2024-09-11T03:47:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857741+0000", 2024-09-11T03:47:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0054999999999999e-05 2024-09-11T03:47:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857802+0000", 2024-09-11T03:47:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0402000000000002e-05 2024-09-11T03:47:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857878+0000", 2024-09-11T03:47:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5926000000000005e-05 2024-09-11T03:47:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,11", 2024-09-11T03:47:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.871599+0000", 2024-09-11T03:47:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013721492 2024-09-11T03:47:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.876926+0000", 2024-09-11T03:47:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0053270949999999996 2024-09-11T03:47:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:47:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.877015+0000", 2024-09-11T03:47:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.9315999999999997e-05 2024-09-11T03:47:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.880031+0000", 2024-09-11T03:47:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030155830000000001 2024-09-11T03:47:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:47:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.880069+0000", 2024-09-11T03:47:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8349999999999997e-05 2024-09-11T03:47:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.880100+0000", 2024-09-11T03:47:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0468e-05 2024-09-11T03:47:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2166 2.b 2:dde92fc6:::smithi17557870-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e270)", 2024-09-11T03:47:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:26.857761+0000", 2024-09-11T03:47:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 391.52022889199998, 2024-09-11T03:47:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018486750999999999, 2024-09-11T03:47:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:47:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2166 2024-09-11T03:47:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857761+0000", 2024-09-11T03:47:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857761+0000", 2024-09-11T03:47:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857762+0000", 2024-09-11T03:47:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.51e-07 2024-09-11T03:47:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857765+0000", 2024-09-11T03:47:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0029999999999999e-06 2024-09-11T03:47:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.473 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.473 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.473 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857772+0000", 2024-09-11T03:47:58.473 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1060000000000002e-06 2024-09-11T03:47:58.473 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.473 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.473 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.473 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857777+0000", 2024-09-11T03:47:58.473 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6290000000000001e-06 2024-09-11T03:47:58.473 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.473 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.473 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.473 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857989+0000", 2024-09-11T03:47:58.473 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00021282300000000001 2024-09-11T03:47:58.474 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.474 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.474 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:47:58.474 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.858027+0000", 2024-09-11T03:47:58.474 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7326999999999997e-05 2024-09-11T03:47:58.474 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.474 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.474 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.474 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.876066+0000", 2024-09-11T03:47:58.474 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018039276999999999 2024-09-11T03:47:58.474 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.474 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.474 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.475 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.876155+0000", 2024-09-11T03:47:58.475 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.9062999999999998e-05 2024-09-11T03:47:58.475 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.475 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.475 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.475 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.876248+0000", 2024-09-11T03:47:58.475 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2972000000000004e-05 2024-09-11T03:47:58.475 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.475 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.475 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.475 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.475 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.475 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2492 2.5 2:a1a99782:::smithi17557870-1:head [copy-from ver 279] snapc 6b=[6b,6a,69,68,67,65,64,63,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-11T03:47:58.476 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:58.892917+0000", 2024-09-11T03:47:58.476 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 359.48507337900003, 2024-09-11T03:47:58.476 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018454061000000001, 2024-09-11T03:47:58.476 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.476 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.476 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:58.476 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.476 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.476 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.476 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2492 2024-09-11T03:47:58.476 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.476 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.476 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.477 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.477 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892917+0000", 2024-09-11T03:47:58.477 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.477 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.477 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.477 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.477 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892917+0000", 2024-09-11T03:47:58.477 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.477 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.477 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.477 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.477 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892918+0000", 2024-09-11T03:47:58.477 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0899999999999999e-06 2024-09-11T03:47:58.477 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.478 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.478 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.478 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892925+0000", 2024-09-11T03:47:58.478 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5470000000000002e-06 2024-09-11T03:47:58.478 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.478 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.478 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.478 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892937+0000", 2024-09-11T03:47:58.478 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.269e-05 2024-09-11T03:47:58.478 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.478 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.478 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.478 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892945+0000", 2024-09-11T03:47:58.479 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7270000000000007e-06 2024-09-11T03:47:58.479 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.479 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.479 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.479 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.893002+0000", 2024-09-11T03:47:58.479 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6650999999999997e-05 2024-09-11T03:47:58.479 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.479 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.479 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.479 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.893112+0000", 2024-09-11T03:47:58.479 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011042100000000001 2024-09-11T03:47:58.479 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.479 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.480 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:47:58.480 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.901793+0000", 2024-09-11T03:47:58.480 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0086810189999999999 2024-09-11T03:47:58.480 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.480 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.480 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:47:58.480 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.905968+0000", 2024-09-11T03:47:58.480 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0041751239999999997 2024-09-11T03:47:58.480 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.480 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.480 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.480 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.910080+0000", 2024-09-11T03:47:58.480 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0041120799999999997 2024-09-11T03:47:58.481 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.481 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.481 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.481 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.911159+0000", 2024-09-11T03:47:58.481 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0010783609999999999 2024-09-11T03:47:58.481 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.481 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.481 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:47:58.481 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.911257+0000", 2024-09-11T03:47:58.481 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8167999999999999e-05 2024-09-11T03:47:58.481 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.481 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.481 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.482 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.911371+0000", 2024-09-11T03:47:58.482 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114183 2024-09-11T03:47:58.482 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.482 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.482 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.482 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.482 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.482 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2931 2.5 2:a6d17fc0:::smithi17557870-13:head [copy-from ver 419] snapc 80=[80,7b,7a,79,77,75,6c,65] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-11T03:47:58.482 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:43.119642+0000", 2024-09-11T03:47:58.482 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 315.25834840200002, 2024-09-11T03:47:58.482 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013261583, 2024-09-11T03:47:58.482 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.482 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.482 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:58.483 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.483 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.483 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.483 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2931 2024-09-11T03:47:58.483 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.483 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.483 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.483 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.483 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.119642+0000", 2024-09-11T03:47:58.483 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.483 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.483 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.483 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.484 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.119642+0000", 2024-09-11T03:47:58.484 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.484 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.484 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.484 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.484 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.119644+0000", 2024-09-11T03:47:58.484 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0889999999999998e-06 2024-09-11T03:47:58.484 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.484 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.484 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.484 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.119652+0000", 2024-09-11T03:47:58.484 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1159999999999999e-06 2024-09-11T03:47:58.484 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.119670+0000", 2024-09-11T03:47:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7903000000000001e-05 2024-09-11T03:47:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.119677+0000", 2024-09-11T03:47:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2520000000000002e-06 2024-09-11T03:47:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.119752+0000", 2024-09-11T03:47:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5092000000000005e-05 2024-09-11T03:47:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:47:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.119759+0000", 2024-09-11T03:47:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.165e-06 2024-09-11T03:47:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.121986+0000", 2024-09-11T03:47:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022266159999999998 2024-09-11T03:47:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.122074+0000", 2024-09-11T03:47:58.487 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8010999999999995e-05 2024-09-11T03:47:58.487 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.487 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.487 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:47:58.487 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.125529+0000", 2024-09-11T03:47:58.487 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0034549709999999998 2024-09-11T03:47:58.487 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.487 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.487 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:47:58.487 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.128468+0000", 2024-09-11T03:47:58.487 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0029391209999999998 2024-09-11T03:47:58.487 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.487 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.488 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.488 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.131454+0000", 2024-09-11T03:47:58.488 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002986135 2024-09-11T03:47:58.488 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.488 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.488 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.488 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.132843+0000", 2024-09-11T03:47:58.488 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001388762 2024-09-11T03:47:58.488 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.488 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.488 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:47:58.488 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.132876+0000", 2024-09-11T03:47:58.488 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.2740999999999997e-05 2024-09-11T03:47:58.489 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.489 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.489 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.489 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:43.132903+0000", 2024-09-11T03:47:58.489 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7608999999999999e-05 2024-09-11T03:47:58.489 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.489 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.489 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.489 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.489 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.489 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4164 2.b 2:dde92fc6:::smithi17557870-41:head [copy-from ver 223] snapc b4=[b4,b2,b1,af,ae,ac,ab,a9,a8,9d] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-09-11T03:47:58.490 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:27.930005+0000", 2024-09-11T03:47:58.490 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 210.447985601, 2024-09-11T03:47:58.490 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014694021999999999, 2024-09-11T03:47:58.490 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.490 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.490 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:58.490 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.490 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.490 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.490 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4164 2024-09-11T03:47:58.490 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.490 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.490 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.491 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.491 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930005+0000", 2024-09-11T03:47:58.491 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.491 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.491 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.491 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.491 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930005+0000", 2024-09-11T03:47:58.491 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.491 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.491 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.491 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.491 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930006+0000", 2024-09-11T03:47:58.492 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.466e-06 2024-09-11T03:47:58.492 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.492 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.492 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.492 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930011+0000", 2024-09-11T03:47:58.492 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9930000000000003e-06 2024-09-11T03:47:58.492 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.492 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.492 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.492 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930022+0000", 2024-09-11T03:47:58.492 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0831e-05 2024-09-11T03:47:58.492 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.492 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.493 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.493 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930029+0000", 2024-09-11T03:47:58.493 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.776e-06 2024-09-11T03:47:58.493 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.493 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.493 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.493 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930083+0000", 2024-09-11T03:47:58.493 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4586000000000002e-05 2024-09-11T03:47:58.493 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.493 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.493 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.493 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930150+0000", 2024-09-11T03:47:58.493 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6595999999999997e-05 2024-09-11T03:47:58.494 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.494 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.494 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,11", 2024-09-11T03:47:58.494 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.936619+0000", 2024-09-11T03:47:58.494 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0064690360000000001 2024-09-11T03:47:58.494 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.494 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.494 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.494 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.942717+0000", 2024-09-11T03:47:58.494 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0060981550000000001 2024-09-11T03:47:58.494 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.494 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.494 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.494 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.943814+0000", 2024-09-11T03:47:58.495 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001097251 2024-09-11T03:47:58.495 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.495 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.495 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:47:58.495 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.944627+0000", 2024-09-11T03:47:58.495 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00081236800000000005 2024-09-11T03:47:58.495 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.495 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.495 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:47:58.495 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.944664+0000", 2024-09-11T03:47:58.495 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7282e-05 2024-09-11T03:47:58.495 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.495 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.495 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.496 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.944699+0000", 2024-09-11T03:47:58.496 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4681999999999998e-05 2024-09-11T03:47:58.496 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.496 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.496 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.496 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.496 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.496 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4700 2.5 2:a1a99782:::smithi17557870-1:head [copy-from ver 541] snapc cd=[cd,cc,ca,c5,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-09-11T03:47:58.496 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.960471+0000", 2024-09-11T03:47:58.496 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 156.41751952499999, 2024-09-11T03:47:58.496 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015308754000000001, 2024-09-11T03:47:58.496 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.496 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.497 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:58.497 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.497 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.497 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.497 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4700 2024-09-11T03:47:58.497 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.497 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.497 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.497 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.497 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960471+0000", 2024-09-11T03:47:58.497 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.497 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.497 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.498 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.498 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960471+0000", 2024-09-11T03:47:58.498 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.498 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.498 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.498 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.498 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960473+0000", 2024-09-11T03:47:58.498 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8050000000000001e-06 2024-09-11T03:47:58.498 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.498 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.498 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.498 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960480+0000", 2024-09-11T03:47:58.498 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3780000000000003e-06 2024-09-11T03:47:58.499 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.499 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.499 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.499 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960497+0000", 2024-09-11T03:47:58.499 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7224e-05 2024-09-11T03:47:58.499 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.499 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.499 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.499 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960504+0000", 2024-09-11T03:47:58.499 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2620000000000001e-06 2024-09-11T03:47:58.499 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.499 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.499 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.500 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960575+0000", 2024-09-11T03:47:58.500 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0507999999999996e-05 2024-09-11T03:47:58.500 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.500 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.500 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.500 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960757+0000", 2024-09-11T03:47:58.500 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000182325 2024-09-11T03:47:58.500 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.500 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.500 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:47:58.500 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.967152+0000", 2024-09-11T03:47:58.500 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0063946519999999998 2024-09-11T03:47:58.500 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.501 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.501 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:47:58.501 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.972224+0000", 2024-09-11T03:47:58.501 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050720349999999999 2024-09-11T03:47:58.501 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.501 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.501 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.501 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.974955+0000", 2024-09-11T03:47:58.501 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0027314829999999998 2024-09-11T03:47:58.501 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.501 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.501 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.501 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.975676+0000", 2024-09-11T03:47:58.502 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00072077700000000003 2024-09-11T03:47:58.502 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.502 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.502 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:47:58.502 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.975727+0000", 2024-09-11T03:47:58.502 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0315999999999997e-05 2024-09-11T03:47:58.502 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.502 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.502 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.502 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.975779+0000", 2024-09-11T03:47:58.502 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2988999999999997e-05 2024-09-11T03:47:58.502 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.502 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.502 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.503 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.503 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.503 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4741 2.5 2:a286c222:::smithi17557870-29:head [copy-from ver 481] snapc d1=[d1,d0,cf,cd,cc,ca,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e510)", 2024-09-11T03:47:58.503 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:27.950605+0000", 2024-09-11T03:47:58.503 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 150.427385074, 2024-09-11T03:47:58.503 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.01346957, 2024-09-11T03:47:58.503 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.503 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.503 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:58.503 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.503 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.503 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.503 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4741 2024-09-11T03:47:58.504 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.504 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.504 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.504 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.504 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:27.950605+0000", 2024-09-11T03:47:58.504 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.504 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.504 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.504 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.504 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:27.950605+0000", 2024-09-11T03:47:58.504 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.504 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.504 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.504 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.505 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:27.950607+0000", 2024-09-11T03:47:58.505 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3650000000000001e-06 2024-09-11T03:47:58.505 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.505 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.505 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.505 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:27.950613+0000", 2024-09-11T03:47:58.505 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6259999999999997e-06 2024-09-11T03:47:58.505 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.505 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.505 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.505 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:27.950632+0000", 2024-09-11T03:47:58.505 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9185e-05 2024-09-11T03:47:58.506 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.506 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.506 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.506 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:27.950643+0000", 2024-09-11T03:47:58.506 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0302999999999999e-05 2024-09-11T03:47:58.506 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.506 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.506 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.506 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:27.950704+0000", 2024-09-11T03:47:58.506 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1261999999999997e-05 2024-09-11T03:47:58.506 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.506 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.506 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.506 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:27.950810+0000", 2024-09-11T03:47:58.507 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000106561 2024-09-11T03:47:58.507 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.507 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.507 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:47:58.507 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:27.954414+0000", 2024-09-11T03:47:58.507 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0036037909999999999 2024-09-11T03:47:58.507 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.507 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.507 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:47:58.507 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:27.960463+0000", 2024-09-11T03:47:58.507 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0060488649999999996 2024-09-11T03:47:58.507 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.507 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.508 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.508 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:27.963026+0000", 2024-09-11T03:47:58.508 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025626920000000001 2024-09-11T03:47:58.508 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.508 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.508 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.508 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:27.964012+0000", 2024-09-11T03:47:58.508 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00098588499999999993 2024-09-11T03:47:58.508 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.508 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.508 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:47:58.508 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:27.964047+0000", 2024-09-11T03:47:58.508 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5064000000000002e-05 2024-09-11T03:47:58.509 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.509 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.509 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.509 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:27.964075+0000", 2024-09-11T03:47:58.509 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7971e-05 2024-09-11T03:47:58.509 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.509 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.509 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.509 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.509 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.509 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5613 2.5 2:a1a99782:::smithi17557870-1:head [copy-from ver 868] snapc f8=[] ondisk+write+known_if_redirected+supports_pool_eio e597)", 2024-09-11T03:47:58.509 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:55.192940+0000", 2024-09-11T03:47:58.509 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 63.185050185999998, 2024-09-11T03:47:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013081663, 2024-09-11T03:47:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:47:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:47:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:47:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:47:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:47:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:47:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5613 2024-09-11T03:47:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:47:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:47:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:55.192940+0000", 2024-09-11T03:47:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:47:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:55.192940+0000", 2024-09-11T03:47:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:47:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:47:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:55.192942+0000", 2024-09-11T03:47:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.871e-06 2024-09-11T03:47:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:47:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:55.192950+0000", 2024-09-11T03:47:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5830000000000001e-06 2024-09-11T03:47:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:47:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:55.192966+0000", 2024-09-11T03:47:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6765e-05 2024-09-11T03:47:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:47:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:55.192974+0000", 2024-09-11T03:47:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2309999999999999e-06 2024-09-11T03:47:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.513 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:47:58.513 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:55.193044+0000", 2024-09-11T03:47:58.513 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0582999999999998e-05 2024-09-11T03:47:58.513 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.513 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.513 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:47:58.513 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:55.193115+0000", 2024-09-11T03:47:58.513 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1185000000000004e-05 2024-09-11T03:47:58.513 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.513 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.513 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:47:58.513 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:55.197518+0000", 2024-09-11T03:47:58.513 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0044031160000000003 2024-09-11T03:47:58.513 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.514 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.514 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:47:58.514 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:55.201580+0000", 2024-09-11T03:47:58.514 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0040611850000000001 2024-09-11T03:47:58.514 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.514 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.514 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.514 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:55.205858+0000", 2024-09-11T03:47:58.514 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0042781800000000004 2024-09-11T03:47:58.514 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.514 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.514 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:47:58.514 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:55.205947+0000", 2024-09-11T03:47:58.515 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.9270000000000004e-05 2024-09-11T03:47:58.515 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.515 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.515 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:47:58.515 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:55.205988+0000", 2024-09-11T03:47:58.515 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.1063000000000003e-05 2024-09-11T03:47:58.515 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:47:58.515 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:47:58.515 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:47:58.515 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:55.206022+0000", 2024-09-11T03:47:58.516 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3630999999999997e-05 2024-09-11T03:47:58.516 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.516 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.516 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.516 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:47:58.516 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:47:58.516 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:47:58.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:58 smithi121 ceph-mon[30729]: pgmap v1143: 67 pgs: 67 active+clean; 121 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-11T03:47:58.517 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:58 smithi121 ceph-mon[30729]: osdmap e660: 12 total, 12 up, 12 in 2024-09-11T03:47:58.529 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1144,"stamp":"2024-09-11T03:47:56.985189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":126635020,"num_objects":52,"num_object_clones":10,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":1,"num_read":2023,"num_read_kb":2113471,"num_write":1359,"num_write_kb":509244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":244,"num_bytes_recovered":60090984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19509,"ondisk_log_size":19509,"up":201,"acting":201,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":776060,"kb_used_data":324076,"kb_used_omap":18,"kb_used_meta":451693,"kb_avail":1124165764,"statfs":{"total":1151940427776,"available":1151145742336,"internally_reserved":0,"allocated":331853824,"data_stored":323049042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":462533947},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4588244,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":29,"num_read_kb":28498,"num_write":8,"num_write_kb":4482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000198"},"pg_stats":[{"pgid":"2.23","version":"652'401","reported_seq":3266,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.994063+0000","last_change":"2024-09-11T03:47:55.199469+0000","last_active":"2024-09-11T03:47:55.994063+0000","last_peered":"2024-09-11T03:47:55.994063+0000","last_clean":"2024-09-11T03:47:55.994063+0000","last_became_active":"2024-09-11T03:46:32.970100+0000","last_became_peered":"2024-09-11T03:46:32.970100+0000","last_unstale":"2024-09-11T03:47:55.994063+0000","last_undegraded":"2024-09-11T03:47:55.994063+0000","last_fullsized":"2024-09-11T03:47:55.994063+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"601'395","last_scrub_stamp":"2024-09-11T03:47:47.212647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:47.212647+0000","objects_scrubbed":1,"log_size":401,"log_dups_size":0,"ondisk_log_size":401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:07.552766+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00039973800000000001,"stat_sum":{"num_bytes":3227109,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":14955,"num_write":12,"num_write_kb":6104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"2.22","version":"531'216","reported_seq":3061,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:51.997225+0000","last_change":"2024-09-11T03:47:49.990648+0000","last_active":"2024-09-11T03:47:51.997225+0000","last_peered":"2024-09-11T03:47:51.997225+0000","last_clean":"2024-09-11T03:47:51.997225+0000","last_became_active":"2024-09-11T03:46:32.970633+0000","last_became_peered":"2024-09-11T03:46:32.970633+0000","last_unstale":"2024-09-11T03:47:51.997225+0000","last_undegraded":"2024-09-11T03:47:51.997225+0000","last_fullsized":"2024-09-11T03:47:51.997225+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:47:49.340068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:49.340068+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:01.256795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078982499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"2.21","version":"636'265","reported_seq":3193,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:51.996699+0000","last_change":"2024-09-11T03:47:50.154453+0000","last_active":"2024-09-11T03:47:51.996699+0000","last_peered":"2024-09-11T03:47:51.996699+0000","last_clean":"2024-09-11T03:47:51.996699+0000","last_became_active":"2024-09-11T03:46:36.024928+0000","last_became_peered":"2024-09-11T03:46:36.024928+0000","last_unstale":"2024-09-11T03:47:51.996699+0000","last_undegraded":"2024-09-11T03:47:51.996699+0000","last_fullsized":"2024-09-11T03:47:51.996699+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":6,"last_scrub":"636'265","last_scrub_stamp":"2024-09-11T03:47:48.316550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:48.316550+0000","objects_scrubbed":1,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:53.290340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0039292540000000001,"stat_sum":{"num_bytes":2201915,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":13,"num_write_kb":4970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,8],"acting":[11,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"2.20","version":"630'213","reported_seq":3213,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-11T03:47:54.997086+0000","last_change":"2024-09-11T03:47:54.997543+0000","last_active":"2024-09-11T03:47:54.997086+0000","last_peered":"2024-09-11T03:47:54.997086+0000","last_clean":"2024-09-11T03:47:54.997086+0000","last_became_active":"2024-09-11T03:46:32.974375+0000","last_became_peered":"2024-09-11T03:46:32.974375+0000","last_unstale":"2024-09-11T03:47:54.997086+0000","last_undegraded":"2024-09-11T03:47:54.997086+0000","last_fullsized":"2024-09-11T03:47:54.997086+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"630'213","last_scrub_stamp":"2024-09-11T03:47:49.390772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:49.390772+0000","objects_scrubbed":1,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:18.281121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073109599999999998,"stat_sum":{"num_bytes":3496021,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":20210,"num_write":7,"num_write_kb":1493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"2.1c","version":"650'703","reported_seq":3532,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:52.157984+0000","last_change":"2024-09-11T03:47:49.992700+0000","last_active":"2024-09-11T03:47:52.157984+0000","last_peered":"2024-09-11T03:47:52.157984+0000","last_clean":"2024-09-11T03:47:52.157984+0000","last_became_active":"2024-09-11T03:46:32.971036+0000","last_became_peered":"2024-09-11T03:46:32.971036+0000","last_unstale":"2024-09-11T03:47:52.157984+0000","last_undegraded":"2024-09-11T03:47:52.157984+0000","last_fullsized":"2024-09-11T03:47:52.157984+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"568'667","last_scrub_stamp":"2024-09-11T03:46:33.141943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.141943+0000","objects_scrubbed":1,"log_size":703,"log_dups_size":0,"ondisk_log_size":703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073497899999999997,"stat_sum":{"num_bytes":2521248,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":10466,"num_write":18,"num_write_kb":5118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"1.c","version":"20'92","reported_seq":371,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.994044+0000","last_change":"2024-09-11T03:47:26.195120+0000","last_active":"2024-09-11T03:47:55.994044+0000","last_peered":"2024-09-11T03:47:55.994044+0000","last_clean":"2024-09-11T03:47:55.994044+0000","last_became_active":"2024-09-11T03:46:15.974786+0000","last_became_peered":"2024-09-11T03:46:15.974786+0000","last_unstale":"2024-09-11T03:47:55.994044+0000","last_undegraded":"2024-09-11T03:47:55.994044+0000","last_fullsized":"2024-09-11T03:47:55.994044+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:26.195070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:26.195070+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:44.774748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.f","version":"557'465","reported_seq":3349,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.991243+0000","last_change":"2024-09-11T03:47:54.990514+0000","last_active":"2024-09-11T03:47:55.991243+0000","last_peered":"2024-09-11T03:47:55.991243+0000","last_clean":"2024-09-11T03:47:55.991243+0000","last_became_active":"2024-09-11T03:46:27.971779+0000","last_became_peered":"2024-09-11T03:46:27.971779+0000","last_unstale":"2024-09-11T03:47:55.991243+0000","last_undegraded":"2024-09-11T03:47:55.991243+0000","last_fullsized":"2024-09-11T03:47:55.991243+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:47:45.907980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:45.907980+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:57.900079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00051358100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"2.e","version":"653'895","reported_seq":3710,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-11T03:47:52.986947+0000","last_change":"2024-09-11T03:47:50.154717+0000","last_active":"2024-09-11T03:47:52.986947+0000","last_peered":"2024-09-11T03:47:52.986947+0000","last_clean":"2024-09-11T03:47:52.986947+0000","last_became_active":"2024-09-11T03:46:36.024843+0000","last_became_peered":"2024-09-11T03:46:36.024843+0000","last_unstale":"2024-09-11T03:47:52.986947+0000","last_undegraded":"2024-09-11T03:47:52.986947+0000","last_fullsized":"2024-09-11T03:47:52.986947+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'850","last_scrub_stamp":"2024-09-11T03:46:36.567100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:36.567100+0000","objects_scrubbed":3,"log_size":895,"log_dups_size":0,"ondisk_log_size":895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0036727090000000001,"stat_sum":{"num_bytes":3445600,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":5,"num_write":31,"num_write_kb":16191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"1.d","version":"20'92","reported_seq":937,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-11T03:47:54.996863+0000","last_change":"2024-09-11T03:42:51.733829+0000","last_active":"2024-09-11T03:47:54.996863+0000","last_peered":"2024-09-11T03:47:54.996863+0000","last_clean":"2024-09-11T03:47:54.996863+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:47:54.996863+0000","last_undegraded":"2024-09-11T03:47:54.996863+0000","last_fullsized":"2024-09-11T03:47:54.996863+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:51.733762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:51.733762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:11.597940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"648'797","reported_seq":3689,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.996871+0000","last_change":"2024-09-11T03:47:55.116696+0000","last_active":"2024-09-11T03:47:55.996871+0000","last_peered":"2024-09-11T03:47:55.996871+0000","last_clean":"2024-09-11T03:47:55.996871+0000","last_became_active":"2024-09-11T03:46:27.969385+0000","last_became_peered":"2024-09-11T03:46:27.969385+0000","last_unstale":"2024-09-11T03:47:55.996871+0000","last_undegraded":"2024-09-11T03:47:55.996871+0000","last_fullsized":"2024-09-11T03:47:55.996871+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"636'794","last_scrub_stamp":"2024-09-11T03:47:43.189166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:43.189166+0000","objects_scrubbed":3,"log_size":797,"log_dups_size":0,"ondisk_log_size":797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:47.682251+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000464981,"stat_sum":{"num_bytes":11395839,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":109,"num_read_kb":122809,"num_write":24,"num_write_kb":8471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":652,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-11T03:47:54.996913+0000","last_change":"2024-09-11T03:45:53.637950+0000","last_active":"2024-09-11T03:47:54.996913+0000","last_peered":"2024-09-11T03:47:54.996913+0000","last_clean":"2024-09-11T03:47:54.996913+0000","last_became_active":"2024-09-11T03:44:43.940022+0000","last_became_peered":"2024-09-11T03:44:43.940022+0000","last_unstale":"2024-09-11T03:47:54.996913+0000","last_undegraded":"2024-09-11T03:47:54.996913+0000","last_fullsized":"2024-09-11T03:47:54.996913+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:53.637891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:53.637891+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:14.519768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"652'721","reported_seq":3580,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:51.997084+0000","last_change":"2024-09-11T03:47:50.154043+0000","last_active":"2024-09-11T03:47:51.997084+0000","last_peered":"2024-09-11T03:47:51.997084+0000","last_clean":"2024-09-11T03:47:51.997084+0000","last_became_active":"2024-09-11T03:46:27.975493+0000","last_became_peered":"2024-09-11T03:46:27.975493+0000","last_unstale":"2024-09-11T03:47:51.997084+0000","last_undegraded":"2024-09-11T03:47:51.997084+0000","last_fullsized":"2024-09-11T03:47:51.997084+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"648'714","last_scrub_stamp":"2024-09-11T03:47:46.413186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:46.413186+0000","objects_scrubbed":3,"log_size":721,"log_dups_size":0,"ondisk_log_size":721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:14.217586+0000","scrub_duration":28,"objects_trimmed":2,"snaptrim_duration":0.0036361259999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":8821,"num_write":29,"num_write_kb":7012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"1.f","version":"20'92","reported_seq":212,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:51.997619+0000","last_change":"2024-09-11T03:46:16.476244+0000","last_active":"2024-09-11T03:47:51.997619+0000","last_peered":"2024-09-11T03:47:51.997619+0000","last_clean":"2024-09-11T03:47:51.997619+0000","last_became_active":"2024-09-11T03:46:15.973796+0000","last_became_peered":"2024-09-11T03:46:15.973796+0000","last_unstale":"2024-09-11T03:47:51.997619+0000","last_undegraded":"2024-09-11T03:47:51.997619+0000","last_fullsized":"2024-09-11T03:47:51.997619+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:16.476181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:16.476181+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":1351,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-11T03:47:54.997264+0000","last_change":"2024-09-11T03:47:28.495899+0000","last_active":"2024-09-11T03:47:54.997264+0000","last_peered":"2024-09-11T03:47:54.997264+0000","last_clean":"2024-09-11T03:47:54.997264+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:47:54.997264+0000","last_undegraded":"2024-09-11T03:47:54.997264+0000","last_fullsized":"2024-09-11T03:47:54.997264+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:28.495839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:28.495839+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:37.846037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"650'426","reported_seq":3302,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.996996+0000","last_change":"2024-09-11T03:47:55.116643+0000","last_active":"2024-09-11T03:47:55.996996+0000","last_peered":"2024-09-11T03:47:55.996996+0000","last_clean":"2024-09-11T03:47:55.996996+0000","last_became_active":"2024-09-11T03:46:27.972329+0000","last_became_peered":"2024-09-11T03:46:27.972329+0000","last_unstale":"2024-09-11T03:47:55.996996+0000","last_undegraded":"2024-09-11T03:47:55.996996+0000","last_fullsized":"2024-09-11T03:47:55.996996+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'386","last_scrub_stamp":"2024-09-11T03:46:33.722283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.722283+0000","objects_scrubbed":3,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000477007,"stat_sum":{"num_bytes":2201915,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":3869,"num_write":10,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":955,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.994297+0000","last_change":"2024-09-11T03:47:25.344219+0000","last_active":"2024-09-11T03:47:55.994297+0000","last_peered":"2024-09-11T03:47:55.994297+0000","last_clean":"2024-09-11T03:47:55.994297+0000","last_became_active":"2024-09-11T03:46:15.968960+0000","last_became_peered":"2024-09-11T03:46:15.968960+0000","last_unstale":"2024-09-11T03:47:55.994297+0000","last_undegraded":"2024-09-11T03:47:55.994297+0000","last_fullsized":"2024-09-11T03:47:55.994297+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:25.344170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:25.344170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:39.221544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"563'192","reported_seq":3137,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-11T03:47:54.997592+0000","last_change":"2024-09-11T03:47:54.997838+0000","last_active":"2024-09-11T03:47:54.997592+0000","last_peered":"2024-09-11T03:47:54.997592+0000","last_clean":"2024-09-11T03:47:54.997592+0000","last_became_active":"2024-09-11T03:46:27.974405+0000","last_became_peered":"2024-09-11T03:46:27.974405+0000","last_unstale":"2024-09-11T03:47:54.997592+0000","last_undegraded":"2024-09-11T03:47:54.997592+0000","last_fullsized":"2024-09-11T03:47:54.997592+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:47:46.424122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:46.424122+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:04.785917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00064660399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":909,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.994144+0000","last_change":"2024-09-11T03:46:41.056685+0000","last_active":"2024-09-11T03:47:55.994144+0000","last_peered":"2024-09-11T03:47:55.994144+0000","last_clean":"2024-09-11T03:47:55.994144+0000","last_became_active":"2024-09-11T03:44:25.934580+0000","last_became_peered":"2024-09-11T03:44:25.934580+0000","last_unstale":"2024-09-11T03:47:55.994144+0000","last_undegraded":"2024-09-11T03:47:55.994144+0000","last_fullsized":"2024-09-11T03:47:55.994144+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:23.494418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:23.494418+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:10.279429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"567'235","reported_seq":3074,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.994816+0000","last_change":"2024-09-11T03:47:55.161805+0000","last_active":"2024-09-11T03:47:55.994816+0000","last_peered":"2024-09-11T03:47:55.994816+0000","last_clean":"2024-09-11T03:47:55.994816+0000","last_became_active":"2024-09-11T03:46:36.024925+0000","last_became_peered":"2024-09-11T03:46:36.024925+0000","last_unstale":"2024-09-11T03:47:55.994816+0000","last_undegraded":"2024-09-11T03:47:55.994816+0000","last_fullsized":"2024-09-11T03:47:55.994816+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:47:48.293541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:48.293541+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:50.431330+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00062391300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":506,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:52.157712+0000","last_change":"2024-09-11T03:45:26.373529+0000","last_active":"2024-09-11T03:47:52.157712+0000","last_peered":"2024-09-11T03:47:52.157712+0000","last_clean":"2024-09-11T03:47:52.157712+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:47:52.157712+0000","last_undegraded":"2024-09-11T03:47:52.157712+0000","last_fullsized":"2024-09-11T03:47:52.157712+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:26.373478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:26.373478+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:31.438609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"652'222","reported_seq":3171,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.996768+0000","last_change":"2024-09-11T03:47:55.116401+0000","last_active":"2024-09-11T03:47:55.996768+0000","last_peered":"2024-09-11T03:47:55.996768+0000","last_clean":"2024-09-11T03:47:55.996768+0000","last_became_active":"2024-09-11T03:46:27.971649+0000","last_became_peered":"2024-09-11T03:46:27.971649+0000","last_unstale":"2024-09-11T03:47:55.996768+0000","last_undegraded":"2024-09-11T03:47:55.996768+0000","last_fullsized":"2024-09-11T03:47:55.996768+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:46:32.681048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:32.681048+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047389300000000002,"stat_sum":{"num_bytes":4975235,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":5188,"num_write":3,"num_write_kb":8751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":204,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:51.997288+0000","last_change":"2024-09-11T03:46:15.972170+0000","last_active":"2024-09-11T03:47:51.997288+0000","last_peered":"2024-09-11T03:47:51.997288+0000","last_clean":"2024-09-11T03:47:51.997288+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:47:51.997288+0000","last_undegraded":"2024-09-11T03:47:51.997288+0000","last_fullsized":"2024-09-11T03:47:51.997288+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:12.360829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:12.360829+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:33.401096+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"650'496","reported_seq":3412,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.994652+0000","last_change":"2024-09-11T03:47:55.199617+0000","last_active":"2024-09-11T03:47:55.994652+0000","last_peered":"2024-09-11T03:47:55.994652+0000","last_clean":"2024-09-11T03:47:55.994652+0000","last_became_active":"2024-09-11T03:44:10.921135+0000","last_became_peered":"2024-09-11T03:44:10.921135+0000","last_unstale":"2024-09-11T03:47:55.994652+0000","last_undegraded":"2024-09-11T03:47:55.994652+0000","last_fullsized":"2024-09-11T03:47:55.994652+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"590'463","last_scrub_stamp":"2024-09-11T03:46:49.936885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:49.936885+0000","objects_scrubbed":3,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:02.094622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00039273699999999999,"stat_sum":{"num_bytes":5866006,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":170,"num_read_kb":169703,"num_write":105,"num_write_kb":37453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":4118225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":509,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:52.157874+0000","last_change":"2024-09-11T03:46:59.252925+0000","last_active":"2024-09-11T03:47:52.157874+0000","last_peered":"2024-09-11T03:47:52.157874+0000","last_clean":"2024-09-11T03:47:52.157874+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:47:52.157874+0000","last_undegraded":"2024-09-11T03:47:52.157874+0000","last_fullsized":"2024-09-11T03:47:52.157874+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:59.252877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:59.252877+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:07.389660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"658'918","reported_seq":3845,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.991419+0000","last_change":"2024-09-11T03:47:54.990753+0000","last_active":"2024-09-11T03:47:55.991419+0000","last_peered":"2024-09-11T03:47:55.991419+0000","last_clean":"2024-09-11T03:47:55.991419+0000","last_became_active":"2024-09-11T03:41:33.866316+0000","last_became_peered":"2024-09-11T03:41:33.866316+0000","last_unstale":"2024-09-11T03:47:55.991419+0000","last_undegraded":"2024-09-11T03:47:55.991419+0000","last_fullsized":"2024-09-11T03:47:55.991419+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'849","last_scrub_stamp":"2024-09-11T03:46:59.044884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:59.044884+0000","objects_scrubbed":3,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:16.060698+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.00071582099999999995,"stat_sum":{"num_bytes":15177612,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":272,"num_read_kb":292030,"num_write":184,"num_write_kb":69607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":516,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:52.158099+0000","last_change":"2024-09-11T03:46:17.255120+0000","last_active":"2024-09-11T03:47:52.158099+0000","last_peered":"2024-09-11T03:47:52.158099+0000","last_clean":"2024-09-11T03:47:52.158099+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:47:52.158099+0000","last_undegraded":"2024-09-11T03:47:52.158099+0000","last_fullsized":"2024-09-11T03:47:52.158099+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:17.255073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:17.255073+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:32.395589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":3011,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.994248+0000","last_change":"2024-09-11T03:47:55.199808+0000","last_active":"2024-09-11T03:47:55.994248+0000","last_peered":"2024-09-11T03:47:55.994248+0000","last_clean":"2024-09-11T03:47:55.994248+0000","last_became_active":"2024-09-11T03:41:33.865499+0000","last_became_peered":"2024-09-11T03:41:33.865499+0000","last_unstale":"2024-09-11T03:47:55.994248+0000","last_undegraded":"2024-09-11T03:47:55.994248+0000","last_fullsized":"2024-09-11T03:47:55.994248+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:47:15.201479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:15.201479+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:25.204593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000423397,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":1108,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:51.996869+0000","last_change":"2024-09-11T03:46:59.164595+0000","last_active":"2024-09-11T03:47:51.996869+0000","last_peered":"2024-09-11T03:47:51.996869+0000","last_clean":"2024-09-11T03:47:51.996869+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:47:51.996869+0000","last_undegraded":"2024-09-11T03:47:51.996869+0000","last_fullsized":"2024-09-11T03:47:51.996869+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:59.164529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:59.164529+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:01.974929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"652'962","reported_seq":3916,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:51.997425+0000","last_change":"2024-09-11T03:47:49.995391+0000","last_active":"2024-09-11T03:47:51.997425+0000","last_peered":"2024-09-11T03:47:51.997425+0000","last_clean":"2024-09-11T03:47:51.997425+0000","last_became_active":"2024-09-11T03:46:19.004997+0000","last_became_peered":"2024-09-11T03:46:19.004997+0000","last_unstale":"2024-09-11T03:47:51.997425+0000","last_undegraded":"2024-09-11T03:47:51.997425+0000","last_fullsized":"2024-09-11T03:47:51.997425+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'924","last_scrub_stamp":"2024-09-11T03:46:53.434384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:53.434384+0000","objects_scrubbed":5,"log_size":962,"log_dups_size":0,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:14.624412+0000","scrub_duration":2,"objects_trimmed":2,"snaptrim_duration":0.0055409020000000003,"stat_sum":{"num_bytes":9484959,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":348,"num_read_kb":409386,"num_write":204,"num_write_kb":73335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":25429065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"1.b","version":"20'92","reported_seq":919,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:51.996999+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:47:51.996999+0000","last_peered":"2024-09-11T03:47:51.996999+0000","last_clean":"2024-09-11T03:47:51.996999+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:47:51.996999+0000","last_undegraded":"2024-09-11T03:47:51.996999+0000","last_fullsized":"2024-09-11T03:47:51.996999+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:06.223147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"641'331","reported_seq":3206,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-11T03:47:54.997285+0000","last_change":"2024-09-11T03:47:54.997578+0000","last_active":"2024-09-11T03:47:54.997285+0000","last_peered":"2024-09-11T03:47:54.997285+0000","last_clean":"2024-09-11T03:47:54.997285+0000","last_became_active":"2024-09-11T03:41:33.863875+0000","last_became_peered":"2024-09-11T03:41:33.863875+0000","last_unstale":"2024-09-11T03:47:54.997285+0000","last_undegraded":"2024-09-11T03:47:54.997285+0000","last_fullsized":"2024-09-11T03:47:54.997285+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"641'331","last_scrub_stamp":"2024-09-11T03:47:43.387058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:43.387058+0000","objects_scrubbed":1,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:54.211490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00043595,"stat_sum":{"num_bytes":2386792,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":45191,"num_write":43,"num_write_kb":12315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":1263,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-11T03:47:52.986520+0000","last_change":"2024-09-11T03:47:28.372829+0000","last_active":"2024-09-11T03:47:52.986520+0000","last_peered":"2024-09-11T03:47:52.986520+0000","last_clean":"2024-09-11T03:47:52.986520+0000","last_became_active":"2024-09-11T03:46:15.973271+0000","last_became_peered":"2024-09-11T03:46:15.973271+0000","last_unstale":"2024-09-11T03:47:52.986520+0000","last_undegraded":"2024-09-11T03:47:52.986520+0000","last_fullsized":"2024-09-11T03:47:52.986520+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:28.372745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:28.372745+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:58.221220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"653'961","reported_seq":3935,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.996228+0000","last_change":"2024-09-11T03:47:55.199830+0000","last_active":"2024-09-11T03:47:55.996228+0000","last_peered":"2024-09-11T03:47:55.996228+0000","last_clean":"2024-09-11T03:47:55.996228+0000","last_became_active":"2024-09-11T03:41:33.865638+0000","last_became_peered":"2024-09-11T03:41:33.865638+0000","last_unstale":"2024-09-11T03:47:55.996228+0000","last_undegraded":"2024-09-11T03:47:55.996228+0000","last_fullsized":"2024-09-11T03:47:55.996228+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'865","last_scrub_stamp":"2024-09-11T03:46:34.843368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:34.843368+0000","objects_scrubbed":7,"log_size":961,"log_dups_size":0,"ondisk_log_size":961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00034997499999999998,"stat_sum":{"num_bytes":10660521,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":369,"num_read_kb":381752,"num_write":223,"num_write_kb":75596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":1338,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-11T03:47:54.996596+0000","last_change":"2024-09-11T03:47:22.529885+0000","last_active":"2024-09-11T03:47:54.996596+0000","last_peered":"2024-09-11T03:47:54.996596+0000","last_clean":"2024-09-11T03:47:54.996596+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:47:54.996596+0000","last_undegraded":"2024-09-11T03:47:54.996596+0000","last_fullsized":"2024-09-11T03:47:54.996596+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:22.529805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:22.529805+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:34.199617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"653'312","reported_seq":3172,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:51.996741+0000","last_change":"2024-09-11T03:47:50.151375+0000","last_active":"2024-09-11T03:47:51.996741+0000","last_peered":"2024-09-11T03:47:51.996741+0000","last_clean":"2024-09-11T03:47:51.996741+0000","last_became_active":"2024-09-11T03:46:27.968719+0000","last_became_peered":"2024-09-11T03:46:27.968719+0000","last_unstale":"2024-09-11T03:47:51.996741+0000","last_undegraded":"2024-09-11T03:47:51.996741+0000","last_fullsized":"2024-09-11T03:47:51.996741+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'279","last_scrub_stamp":"2024-09-11T03:46:31.085715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:31.085715+0000","objects_scrubbed":2,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001312845,"stat_sum":{"num_bytes":4530315,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":14,"num_write_kb":7034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"1.8","version":"20'92","reported_seq":1002,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.996849+0000","last_change":"2024-09-11T03:46:20.861683+0000","last_active":"2024-09-11T03:47:55.996849+0000","last_peered":"2024-09-11T03:47:55.996849+0000","last_clean":"2024-09-11T03:47:55.996849+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:47:55.996849+0000","last_undegraded":"2024-09-11T03:47:55.996849+0000","last_fullsized":"2024-09-11T03:47:55.996849+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:20.861635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:20.861635+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:31.352393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"568'646","reported_seq":3522,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.991180+0000","last_change":"2024-09-11T03:47:54.990492+0000","last_active":"2024-09-11T03:47:55.991180+0000","last_peered":"2024-09-11T03:47:55.991180+0000","last_clean":"2024-09-11T03:47:55.991180+0000","last_became_active":"2024-09-11T03:46:27.974031+0000","last_became_peered":"2024-09-11T03:46:27.974031+0000","last_unstale":"2024-09-11T03:47:55.991180+0000","last_undegraded":"2024-09-11T03:47:55.991180+0000","last_fullsized":"2024-09-11T03:47:55.991180+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:47:46.957311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:46.957311+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:51.339165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071237899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":952,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.996682+0000","last_change":"2024-09-11T03:47:44.163354+0000","last_active":"2024-09-11T03:47:55.996682+0000","last_peered":"2024-09-11T03:47:55.996682+0000","last_clean":"2024-09-11T03:47:55.996682+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:47:55.996682+0000","last_undegraded":"2024-09-11T03:47:55.996682+0000","last_fullsized":"2024-09-11T03:47:55.996682+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:44.163296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:44.163296+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:00.005327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":2932,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-11T03:47:52.986724+0000","last_change":"2024-09-11T03:47:50.277290+0000","last_active":"2024-09-11T03:47:52.986724+0000","last_peered":"2024-09-11T03:47:52.986724+0000","last_clean":"2024-09-11T03:47:52.986724+0000","last_became_active":"2024-09-11T03:41:38.886834+0000","last_became_peered":"2024-09-11T03:41:38.886834+0000","last_unstale":"2024-09-11T03:47:52.986724+0000","last_undegraded":"2024-09-11T03:47:52.986724+0000","last_fullsized":"2024-09-11T03:47:52.986724+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:47:50.277248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:50.277248+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:07.752596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012439510000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"1.11","version":"20'92","reported_seq":927,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.994588+0000","last_change":"2024-09-11T03:47:00.162760+0000","last_active":"2024-09-11T03:47:55.994588+0000","last_peered":"2024-09-11T03:47:55.994588+0000","last_clean":"2024-09-11T03:47:55.994588+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:47:55.994588+0000","last_undegraded":"2024-09-11T03:47:55.994588+0000","last_fullsized":"2024-09-11T03:47:55.994588+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:00.162709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:00.162709+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:20.560991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"648'458","reported_seq":3320,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:52.158755+0000","last_change":"2024-09-11T03:47:49.993494+0000","last_active":"2024-09-11T03:47:52.158755+0000","last_peered":"2024-09-11T03:47:52.158755+0000","last_clean":"2024-09-11T03:47:52.158755+0000","last_became_active":"2024-09-11T03:43:55.918817+0000","last_became_peered":"2024-09-11T03:43:55.918817+0000","last_unstale":"2024-09-11T03:47:52.158755+0000","last_undegraded":"2024-09-11T03:47:52.158755+0000","last_fullsized":"2024-09-11T03:47:52.158755+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"590'420","last_scrub_stamp":"2024-09-11T03:46:50.265647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:50.265647+0000","objects_scrubbed":2,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:06.539123+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0014044019999999999,"stat_sum":{"num_bytes":6218953,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":190,"num_read_kb":210614,"num_write":115,"num_write_kb":40160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":15216236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"1.12","version":"20'92","reported_seq":941,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.996713+0000","last_change":"2024-09-11T03:47:23.339646+0000","last_active":"2024-09-11T03:47:55.996713+0000","last_peered":"2024-09-11T03:47:55.996713+0000","last_clean":"2024-09-11T03:47:55.996713+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:47:55.996713+0000","last_undegraded":"2024-09-11T03:47:55.996713+0000","last_fullsized":"2024-09-11T03:47:55.996713+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:23.339595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:23.339595+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:46.448313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":1995,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:51.997146+0000","last_change":"2024-09-11T03:47:49.992689+0000","last_active":"2024-09-11T03:47:51.997146+0000","last_peered":"2024-09-11T03:47:51.997146+0000","last_clean":"2024-09-11T03:47:51.997146+0000","last_became_active":"2024-09-11T03:46:19.005117+0000","last_became_peered":"2024-09-11T03:46:19.005117+0000","last_unstale":"2024-09-11T03:47:51.997146+0000","last_undegraded":"2024-09-11T03:47:51.997146+0000","last_fullsized":"2024-09-11T03:47:51.997146+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:47:20.437167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:20.437167+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:34.184251+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0026403239999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"1.13","version":"20'92","reported_seq":1042,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.994239+0000","last_change":"2024-09-11T03:47:18.217600+0000","last_active":"2024-09-11T03:47:55.994239+0000","last_peered":"2024-09-11T03:47:55.994239+0000","last_clean":"2024-09-11T03:47:55.994239+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:47:55.994239+0000","last_undegraded":"2024-09-11T03:47:55.994239+0000","last_fullsized":"2024-09-11T03:47:55.994239+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:18.217555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:18.217555+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:40.131743+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":1951,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.991006+0000","last_change":"2024-09-11T03:47:54.990198+0000","last_active":"2024-09-11T03:47:55.991006+0000","last_peered":"2024-09-11T03:47:55.991006+0000","last_clean":"2024-09-11T03:47:55.991006+0000","last_became_active":"2024-09-11T03:41:35.874295+0000","last_became_peered":"2024-09-11T03:41:35.874295+0000","last_unstale":"2024-09-11T03:47:55.991006+0000","last_undegraded":"2024-09-11T03:47:55.991006+0000","last_fullsized":"2024-09-11T03:47:55.991006+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:47:18.969898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:18.969898+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:44.079672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054847200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":506,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-11T03:47:54.996779+0000","last_change":"2024-09-11T03:46:36.567028+0000","last_active":"2024-09-11T03:47:54.996779+0000","last_peered":"2024-09-11T03:47:54.996779+0000","last_clean":"2024-09-11T03:47:54.996779+0000","last_became_active":"2024-09-11T03:46:36.026459+0000","last_became_peered":"2024-09-11T03:46:36.026459+0000","last_unstale":"2024-09-11T03:47:54.996779+0000","last_undegraded":"2024-09-11T03:47:54.996779+0000","last_fullsized":"2024-09-11T03:47:54.996779+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:36.566974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:36.566974+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:54.333688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":2491,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:51.997517+0000","last_change":"2024-09-11T03:47:49.994077+0000","last_active":"2024-09-11T03:47:51.997517+0000","last_peered":"2024-09-11T03:47:51.997517+0000","last_clean":"2024-09-11T03:47:51.997517+0000","last_became_active":"2024-09-11T03:46:19.003959+0000","last_became_peered":"2024-09-11T03:46:19.003959+0000","last_unstale":"2024-09-11T03:47:51.997517+0000","last_undegraded":"2024-09-11T03:47:51.997517+0000","last_fullsized":"2024-09-11T03:47:51.997517+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:46:35.381114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:35.381114+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003704344,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"1.15","version":"20'92","reported_seq":909,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-11T03:47:54.996963+0000","last_change":"2024-09-11T03:47:47.459856+0000","last_active":"2024-09-11T03:47:54.996963+0000","last_peered":"2024-09-11T03:47:54.996963+0000","last_clean":"2024-09-11T03:47:54.996963+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:47:54.996963+0000","last_undegraded":"2024-09-11T03:47:54.996963+0000","last_fullsized":"2024-09-11T03:47:54.996963+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:47.459803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:47.459803+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:06.089977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"658'443","reported_seq":3359,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:56.014911+0000","last_change":"2024-09-11T03:47:55.116397+0000","last_active":"2024-09-11T03:47:56.014911+0000","last_peered":"2024-09-11T03:47:56.014911+0000","last_clean":"2024-09-11T03:47:56.014911+0000","last_became_active":"2024-09-11T03:46:19.007019+0000","last_became_peered":"2024-09-11T03:46:19.007019+0000","last_unstale":"2024-09-11T03:47:56.014911+0000","last_undegraded":"2024-09-11T03:47:56.014911+0000","last_fullsized":"2024-09-11T03:47:56.014911+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"597'416","last_scrub_stamp":"2024-09-11T03:47:00.394934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:00.394934+0000","objects_scrubbed":0,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:03.018708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00052826499999999998,"stat_sum":{"num_bytes":10202145,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":81,"num_read_kb":64221,"num_write":91,"num_write_kb":38878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":5263193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":375,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:51.996658+0000","last_change":"2024-09-11T03:47:50.266422+0000","last_active":"2024-09-11T03:47:51.996658+0000","last_peered":"2024-09-11T03:47:51.996658+0000","last_clean":"2024-09-11T03:47:51.996658+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:47:51.996658+0000","last_undegraded":"2024-09-11T03:47:51.996658+0000","last_fullsized":"2024-09-11T03:47:51.996658+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:50.266374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:50.266374+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:06.293028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":3145,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-11T03:47:52.986667+0000","last_change":"2024-09-11T03:47:50.153730+0000","last_active":"2024-09-11T03:47:52.986667+0000","last_peered":"2024-09-11T03:47:52.986667+0000","last_clean":"2024-09-11T03:47:52.986667+0000","last_became_active":"2024-09-11T03:41:38.888228+0000","last_became_peered":"2024-09-11T03:41:38.888228+0000","last_unstale":"2024-09-11T03:47:52.986667+0000","last_undegraded":"2024-09-11T03:47:52.986667+0000","last_fullsized":"2024-09-11T03:47:52.986667+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:46:37.552437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:37.552437+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0030352769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"1.17","version":"20'92","reported_seq":929,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.991241+0000","last_change":"2024-09-11T03:47:29.086383+0000","last_active":"2024-09-11T03:47:55.991241+0000","last_peered":"2024-09-11T03:47:55.991241+0000","last_clean":"2024-09-11T03:47:55.991241+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:47:55.991241+0000","last_undegraded":"2024-09-11T03:47:55.991241+0000","last_fullsized":"2024-09-11T03:47:55.991241+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:29.086335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:29.086335+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:40.330795+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"614'276","reported_seq":3039,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:51.997522+0000","last_change":"2024-09-11T03:47:49.993281+0000","last_active":"2024-09-11T03:47:51.997522+0000","last_peered":"2024-09-11T03:47:51.997522+0000","last_clean":"2024-09-11T03:47:51.997522+0000","last_became_active":"2024-09-11T03:46:19.004558+0000","last_became_peered":"2024-09-11T03:46:19.004558+0000","last_unstale":"2024-09-11T03:47:51.997522+0000","last_undegraded":"2024-09-11T03:47:51.997522+0000","last_fullsized":"2024-09-11T03:47:51.997522+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"614'276","last_scrub_stamp":"2024-09-11T03:47:44.317518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:44.317518+0000","objects_scrubbed":1,"log_size":276,"log_dups_size":0,"ondisk_log_size":276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:44.440136+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031959050000000002,"stat_sum":{"num_bytes":3445600,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":8234,"num_write":36,"num_write_kb":8917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"1.18","version":"20'92","reported_seq":1037,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.996746+0000","last_change":"2024-09-11T03:47:49.115228+0000","last_active":"2024-09-11T03:47:55.996746+0000","last_peered":"2024-09-11T03:47:55.996746+0000","last_clean":"2024-09-11T03:47:55.996746+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:47:55.996746+0000","last_undegraded":"2024-09-11T03:47:55.996746+0000","last_fullsized":"2024-09-11T03:47:55.996746+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:49.115171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:49.115171+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:50.337826+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"652'706","reported_seq":3676,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:51.996739+0000","last_change":"2024-09-11T03:47:50.154037+0000","last_active":"2024-09-11T03:47:51.996739+0000","last_peered":"2024-09-11T03:47:51.996739+0000","last_clean":"2024-09-11T03:47:51.996739+0000","last_became_active":"2024-09-11T03:46:32.975567+0000","last_became_peered":"2024-09-11T03:46:32.975567+0000","last_unstale":"2024-09-11T03:47:51.996739+0000","last_undegraded":"2024-09-11T03:47:51.996739+0000","last_fullsized":"2024-09-11T03:47:51.996739+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"575'648","last_scrub_stamp":"2024-09-11T03:46:33.071702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.071702+0000","objects_scrubbed":5,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.003713133,"stat_sum":{"num_bytes":6487832,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":58,"num_read_kb":69184,"num_write":15,"num_write_kb":11184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7],"acting":[11,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"1.19","version":"20'92","reported_seq":521,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:52.157592+0000","last_change":"2024-09-11T03:47:43.394354+0000","last_active":"2024-09-11T03:47:52.157592+0000","last_peered":"2024-09-11T03:47:52.157592+0000","last_clean":"2024-09-11T03:47:52.157592+0000","last_became_active":"2024-09-11T03:46:15.972841+0000","last_became_peered":"2024-09-11T03:46:15.972841+0000","last_unstale":"2024-09-11T03:47:52.157592+0000","last_undegraded":"2024-09-11T03:47:52.157592+0000","last_fullsized":"2024-09-11T03:47:52.157592+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:43.394305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:43.394305+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:54.956879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"557'273","reported_seq":3065,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:51.997285+0000","last_change":"2024-09-11T03:47:49.993717+0000","last_active":"2024-09-11T03:47:51.997285+0000","last_peered":"2024-09-11T03:47:51.997285+0000","last_clean":"2024-09-11T03:47:51.997285+0000","last_became_active":"2024-09-11T03:46:32.971003+0000","last_became_peered":"2024-09-11T03:46:32.971003+0000","last_unstale":"2024-09-11T03:47:51.997285+0000","last_undegraded":"2024-09-11T03:47:51.997285+0000","last_fullsized":"2024-09-11T03:47:51.997285+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:46:34.407744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:34.407744+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:58.211057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034822159999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,2],"acting":[6,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"1.1a","version":"20'92","reported_seq":654,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.994273+0000","last_change":"2024-09-11T03:45:52.914136+0000","last_active":"2024-09-11T03:47:55.994273+0000","last_peered":"2024-09-11T03:47:55.994273+0000","last_clean":"2024-09-11T03:47:55.994273+0000","last_became_active":"2024-09-11T03:44:43.938873+0000","last_became_peered":"2024-09-11T03:44:43.938873+0000","last_unstale":"2024-09-11T03:47:55.994273+0000","last_undegraded":"2024-09-11T03:47:55.994273+0000","last_fullsized":"2024-09-11T03:47:55.994273+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:52.914088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:52.914088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:11.184197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"641'492","reported_seq":3416,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:51.996798+0000","last_change":"2024-09-11T03:47:49.992266+0000","last_active":"2024-09-11T03:47:51.996798+0000","last_peered":"2024-09-11T03:47:51.996798+0000","last_clean":"2024-09-11T03:47:51.996798+0000","last_became_active":"2024-09-11T03:46:19.002002+0000","last_became_peered":"2024-09-11T03:46:19.002002+0000","last_unstale":"2024-09-11T03:47:51.996798+0000","last_undegraded":"2024-09-11T03:47:51.996798+0000","last_fullsized":"2024-09-11T03:47:51.996798+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"594'474","last_scrub_stamp":"2024-09-11T03:46:52.412086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:52.412086+0000","objects_scrubbed":0,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:01.466589+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0022557290000000002,"stat_sum":{"num_bytes":2403599,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":59020,"num_write":43,"num_write_kb":21295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4870845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"1.1b","version":"20'92","reported_seq":897,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.991029+0000","last_change":"2024-09-11T03:47:25.035842+0000","last_active":"2024-09-11T03:47:55.991029+0000","last_peered":"2024-09-11T03:47:55.991029+0000","last_clean":"2024-09-11T03:47:55.991029+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:47:55.991029+0000","last_undegraded":"2024-09-11T03:47:55.991029+0000","last_fullsized":"2024-09-11T03:47:55.991029+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:25.035793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:25.035793+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:30.302122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"652'404","reported_seq":3322,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-11T03:47:52.986584+0000","last_change":"2024-09-11T03:47:50.153770+0000","last_active":"2024-09-11T03:47:52.986584+0000","last_peered":"2024-09-11T03:47:52.986584+0000","last_clean":"2024-09-11T03:47:52.986584+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:47:52.986584+0000","last_undegraded":"2024-09-11T03:47:52.986584+0000","last_fullsized":"2024-09-11T03:47:52.986584+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"641'398","last_scrub_stamp":"2024-09-11T03:47:47.328334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:47.328334+0000","objects_scrubbed":2,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:07.020613+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0031311749999999999,"stat_sum":{"num_bytes":6319795,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":133383,"num_write":74,"num_write_kb":26882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4865740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"2.1f","version":"648'520","reported_seq":3347,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.997656+0000","last_change":"2024-09-11T03:47:55.161495+0000","last_active":"2024-09-11T03:47:55.997656+0000","last_peered":"2024-09-11T03:47:55.997656+0000","last_clean":"2024-09-11T03:47:55.997656+0000","last_became_active":"2024-09-11T03:46:32.968696+0000","last_became_peered":"2024-09-11T03:46:32.968696+0000","last_unstale":"2024-09-11T03:47:55.997656+0000","last_undegraded":"2024-09-11T03:47:55.997656+0000","last_fullsized":"2024-09-11T03:47:55.997656+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:46:33.227651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:46:33.227651+0000","objects_scrubbed":3,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:47:58.085009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000394053,"stat_sum":{"num_bytes":5765164,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":71065,"num_write":31,"num_write_kb":12120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"1.1c","version":"20'92","reported_seq":904,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.991169+0000","last_change":"2024-09-11T03:46:20.752592+0000","last_active":"2024-09-11T03:47:55.991169+0000","last_peered":"2024-09-11T03:47:55.991169+0000","last_clean":"2024-09-11T03:47:55.991169+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:47:55.991169+0000","last_undegraded":"2024-09-11T03:47:55.991169+0000","last_fullsized":"2024-09-11T03:47:55.991169+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:20.752526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:20.752526+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:38.077736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1e","version":"630'915","reported_seq":3750,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.994686+0000","last_change":"2024-09-11T03:47:55.161530+0000","last_active":"2024-09-11T03:47:55.994686+0000","last_peered":"2024-09-11T03:47:55.994686+0000","last_clean":"2024-09-11T03:47:55.994686+0000","last_became_active":"2024-09-11T03:46:32.970637+0000","last_became_peered":"2024-09-11T03:46:32.970637+0000","last_unstale":"2024-09-11T03:47:55.994686+0000","last_undegraded":"2024-09-11T03:47:55.994686+0000","last_fullsized":"2024-09-11T03:47:55.994686+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"630'915","last_scrub_stamp":"2024-09-11T03:47:46.307055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:46.307055+0000","objects_scrubbed":1,"log_size":915,"log_dups_size":0,"ondisk_log_size":915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:13.703617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00037642300000000002,"stat_sum":{"num_bytes":3630477,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":13360,"num_write":34,"num_write_kb":15522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"112","length":"2"},{"start":"117","length":"1"}]},{"pgid":"1.1d","version":"20'92","reported_seq":895,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-11T03:47:55.994098+0000","last_change":"2024-09-11T03:47:22.312507+0000","last_active":"2024-09-11T03:47:55.994098+0000","last_peered":"2024-09-11T03:47:55.994098+0000","last_clean":"2024-09-11T03:47:55.994098+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:47:55.994098+0000","last_undegraded":"2024-09-11T03:47:55.994098+0000","last_fullsized":"2024-09-11T03:47:55.994098+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:22.312456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:22.312456+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:44.701057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"567'752","reported_seq":3566,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:52.157883+0000","last_change":"2024-09-11T03:47:49.992774+0000","last_active":"2024-09-11T03:47:52.157883+0000","last_peered":"2024-09-11T03:47:52.157883+0000","last_clean":"2024-09-11T03:47:52.157883+0000","last_became_active":"2024-09-11T03:46:32.972371+0000","last_became_peered":"2024-09-11T03:46:32.972371+0000","last_unstale":"2024-09-11T03:47:52.157883+0000","last_undegraded":"2024-09-11T03:47:52.157883+0000","last_fullsized":"2024-09-11T03:47:52.157883+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:47:47.265243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:47.265243+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:56.905748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000634261,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,11],"acting":[10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"112","length":"2"}]},{"pgid":"1.1e","version":"20'92","reported_seq":899,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-09-11T03:47:51.997576+0000","last_change":"2024-09-11T03:47:00.206764+0000","last_active":"2024-09-11T03:47:51.997576+0000","last_peered":"2024-09-11T03:47:51.997576+0000","last_clean":"2024-09-11T03:47:51.997576+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:47:51.997576+0000","last_undegraded":"2024-09-11T03:47:51.997576+0000","last_fullsized":"2024-09-11T03:47:51.997576+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:00.206720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:00.206720+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:48:28.907706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":36,"stat_sum":{"num_bytes":126044652,"num_objects":50,"num_object_clones":10,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":1,"num_read":2023,"num_read_kb":2113471,"num_write":1359,"num_write_kb":509244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":59763304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237826048,"data_stored":265187704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16657,"ondisk_log_size":16657,"up":108,"acting":108,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285696,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64820,"kb_used_data":21152,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93680332,"statfs":{"total":95995035648,"available":95928659968,"internally_reserved":0,"allocated":21659648,"data_stored":19950226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":44693956},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514628,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60208,"kb_used_data":24852,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93684944,"statfs":{"total":95995035648,"available":95933382656,"internally_reserved":0,"allocated":25448448,"data_stored":22933854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743562,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57596,"kb_used_data":26404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93687556,"statfs":{"total":95995035648,"available":95936057344,"internally_reserved":0,"allocated":27037696,"data_stored":27974252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037906,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54224,"kb_used_data":23032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93690928,"statfs":{"total":95995035648,"available":95939510272,"internally_reserved":0,"allocated":23584768,"data_stored":22413454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299541,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89352,"kb_used_data":35952,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93655800,"statfs":{"total":95995035648,"available":95903539200,"internally_reserved":0,"allocated":36814848,"data_stored":37242679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528473,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64260,"kb_used_data":28912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93680892,"statfs":{"total":95995035648,"available":95929233408,"internally_reserved":0,"allocated":29605888,"data_stored":27138960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757408,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73156,"kb_used_data":29488,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93671996,"statfs":{"total":95995035648,"available":95920123904,"internally_reserved":0,"allocated":30195712,"data_stored":27829344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986341,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56760,"kb_used_data":25572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93688392,"statfs":{"total":95995035648,"available":95936913408,"internally_reserved":0,"allocated":26185728,"data_stored":25004258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94216,"kb_used_data":36660,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93650936,"statfs":{"total":95995035648,"available":95898558464,"internally_reserved":0,"allocated":37539840,"data_stored":40466105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411503,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":46236,"kb_used_data":19204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93698916,"statfs":{"total":95995035648,"available":95947689984,"internally_reserved":0,"allocated":19664896,"data_stored":19988157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54216,"kb_used_data":23020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93690936,"statfs":{"total":95995035648,"available":95939518464,"internally_reserved":0,"allocated":23572480,"data_stored":22759567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738554,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61016,"kb_used_data":29828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93684136,"statfs":{"total":95995035648,"available":95932555264,"internally_reserved":0,"allocated":30543872,"data_stored":29348186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":22294528,"data_stored":23899563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15626240,"data_stored":17622138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11563008,"data_stored":14682364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29315072,"data_stored":36449892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18276352,"data_stored":19891514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":22044672,"data_stored":22490667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":22659072,"data_stored":23562673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28676096,"data_stored":31912217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":15650816,"data_stored":17284240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":19128320,"data_stored":22861508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":17551360,"data_stored":17829309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":15040512,"data_stored":16701619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:47:58.532 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-11T03:47:58.533 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pg_num 61 2024-09-11T03:47:58.536 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:58.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:58.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:58.595 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:58.638 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:58.658 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:58.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:58.659 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:58.760 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:58.786 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:58.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:58.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:58.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:47:58.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:58.911 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:58.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:47:59.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:59.037 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:59.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:59.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:47:59.110 INFO:tasks.rados.rados.2.smithi175.stdout:2556: done (0 left) 2024-09-11T03:47:59.110 INFO:tasks.rados.rados.2.smithi175.stdout:2557: write oid 45 current snap is 281 2024-09-11T03:47:59.110 INFO:tasks.rados.rados.2.smithi175.stdout:2557: seq_num 880 ranges {790160=799715,2112507=509299} 2024-09-11T03:47:59.118 INFO:tasks.rados.rados.2.smithi175.stdout:2557: writing smithi17557870-45 from 790160 to 1589875 tid 1 2024-09-11T03:47:59.125 INFO:tasks.rados.rados.2.smithi175.stdout:2557: writing smithi17557870-45 from 2112507 to 2621806 tid 2 2024-09-11T03:47:59.125 INFO:tasks.rados.rados.2.smithi175.stdout:2558: copy_from oid 32 from oid 27 current snap is 281 2024-09-11T03:47:59.125 INFO:tasks.rados.rados.2.smithi175.stdout:2557: finishing write tid 1 to smithi17557870-45 2024-09-11T03:47:59.125 INFO:tasks.rados.rados.2.smithi175.stdout:2559: delete oid 26 current snap is 281 2024-09-11T03:47:59.127 INFO:tasks.rados.rados.2.smithi175.stdout:2559: done (2 left) 2024-09-11T03:47:59.127 INFO:tasks.rados.rados.2.smithi175.stdout:2560: rollback oid 36 current snap is 281 2024-09-11T03:47:59.128 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 36 to 279 2024-09-11T03:47:59.128 INFO:tasks.rados.rados.2.smithi175.stdout:2561: rollback oid 9 current snap is 281 2024-09-11T03:47:59.128 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 9 to 279 2024-09-11T03:47:59.128 INFO:tasks.rados.rados.2.smithi175.stdout:2562: snap_create 2024-09-11T03:47:59.128 INFO:tasks.rados.rados.2.smithi175.stdout:2557: finishing write tid 2 to smithi17557870-45 2024-09-11T03:47:59.128 INFO:tasks.rados.rados.2.smithi175.stdout:2557: finishing write tid 3 to smithi17557870-45 2024-09-11T03:47:59.128 INFO:tasks.rados.rados.2.smithi175.stdout:2557: finishing write tid 4 to smithi17557870-45 2024-09-11T03:47:59.129 INFO:tasks.rados.rados.2.smithi175.stdout:2557: oid 45 updating version 0 to 723 2024-09-11T03:47:59.129 INFO:tasks.rados.rados.2.smithi175.stdout:2557: oid 45 version 723 is already newer than 722 2024-09-11T03:47:59.129 INFO:tasks.rados.rados.2.smithi175.stdout:2557: oid 45 updating version 723 to 724 2024-09-11T03:47:59.129 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 724 (ObjNum 880 snap 281 seq_num 880) dirty exists 2024-09-11T03:47:59.129 INFO:tasks.rados.rados.2.smithi175.stdout:2557: left oid 45 (ObjNum 880 snap 281 seq_num 880) 2024-09-11T03:47:59.132 INFO:tasks.rados.rados.2.smithi175.stdout:2560: finishing rollback tid 0 to smithi17557870-36 2024-09-11T03:47:59.132 INFO:tasks.rados.rados.2.smithi175.stdout:2560: finishing rollback tid 1 to smithi17557870-36 2024-09-11T03:47:59.132 INFO:tasks.rados.rados.2.smithi175.stdout:2560: finishing rollback tid 2 to smithi17557870-36 2024-09-11T03:47:59.132 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 865 (ObjNum 830 snap 263 seq_num 830) dirty exists 2024-09-11T03:47:59.133 INFO:tasks.rados.rados.2.smithi175.stdout:2561: finishing rollback tid 1 to smithi17557870-9 2024-09-11T03:47:59.133 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 0 (ObjNum 33 snap 0 seq_num 1745346304) dirty dne 2024-09-11T03:47:59.138 INFO:tasks.rados.rados.2.smithi175.stdout:2558: finishing copy_from racing read to smithi17557870-32 2024-09-11T03:47:59.139 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:47:59.140 INFO:tasks.rados.rados.2.smithi175.stdout:2558: finishing copy_from to smithi17557870-32 2024-09-11T03:47:59.140 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 962 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:47:59.159 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:47:59.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:59.160 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:47:59.171 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:47:59.182 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:59.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:47:59 smithi084 ceph-mon[27254]: from='client.16665 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:47:59.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:47:59 smithi175 ceph-mon[30894]: from='client.16665 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:47:59.261 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:47:59.283 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:59.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:59.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:47:59.384 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:47:59.411 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:59.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:59.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:47:59.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:47:59 smithi121 ceph-mon[30729]: from='client.16665 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:47:59.425 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-11T03:47:59.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:59.538 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:59.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:59.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:59.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:47:59.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:59.664 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:47:59.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:47:59.664 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:47:59.674 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:47:59.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:47:59.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:59.792 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:59.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:47:59.828 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: specified pg_num 61 < pg_num_max 32 2024-09-11T03:47:59.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:47:59.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:47:59.919 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:47:59.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:00.020 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:00.042 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:00.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:00.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:00.113 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:48:00.118 INFO:tasks.rados.rados.2.smithi175.stdout:2557: done (4 left) 2024-09-11T03:48:00.118 INFO:tasks.rados.rados.2.smithi175.stdout:2558: done (3 left) 2024-09-11T03:48:00.118 INFO:tasks.rados.rados.2.smithi175.stdout:2560: done (2 left) 2024-09-11T03:48:00.118 INFO:tasks.rados.rados.2.smithi175.stdout:2561: done (1 left) 2024-09-11T03:48:00.119 INFO:tasks.rados.rados.2.smithi175.stdout:2562: done (0 left) 2024-09-11T03:48:00.119 INFO:tasks.rados.rados.2.smithi175.stdout:2563: write (excl) oid 49 current snap is 282 2024-09-11T03:48:00.119 INFO:tasks.rados.rados.2.smithi175.stdout:2563: seq_num 882 ranges {423774=666566,1545824=512968,2704609=119165} 2024-09-11T03:48:00.124 INFO:tasks.rados.rados.2.smithi175.stdout:2563: writing smithi17557870-49 from 423774 to 1090340 tid 1 2024-09-11T03:48:00.129 INFO:tasks.rados.rados.2.smithi175.stdout:2563: writing smithi17557870-49 from 1545824 to 2058792 tid 2 2024-09-11T03:48:00.134 INFO:tasks.rados.rados.2.smithi175.stdout:2563: writing smithi17557870-49 from 2704609 to 2823774 tid 3 2024-09-11T03:48:00.134 INFO:tasks.rados.rados.2.smithi175.stdout:2564: write oid 22 current snap is 282 2024-09-11T03:48:00.134 INFO:tasks.rados.rados.2.smithi175.stdout:2563: finishing write tid 1 to smithi17557870-49 2024-09-11T03:48:00.134 INFO:tasks.rados.rados.2.smithi175.stdout:2563: finishing write tid 2 to smithi17557870-49 2024-09-11T03:48:00.134 INFO:tasks.rados.rados.2.smithi175.stdout:2564: seq_num 883 ranges {440581=741815,1604306=656626,2732032=108549} 2024-09-11T03:48:00.140 INFO:tasks.rados.rados.2.smithi175.stdout:2564: writing smithi17557870-22 from 440581 to 1182396 tid 1 2024-09-11T03:48:00.144 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:00.146 INFO:tasks.rados.rados.2.smithi175.stdout:2564: writing smithi17557870-22 from 1604306 to 2260932 tid 2 2024-09-11T03:48:00.150 INFO:tasks.rados.rados.2.smithi175.stdout:2564: writing smithi17557870-22 from 2732032 to 2840581 tid 3 2024-09-11T03:48:00.150 INFO:tasks.rados.rados.2.smithi175.stdout:2565: snap_remove snap 281 2024-09-11T03:48:00.165 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:00.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:00.166 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:00.238 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:48:00.238 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:48:00.238 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:48:00.238 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:48:00.267 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:00.289 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:00.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:00.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:00.391 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:00.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:00 smithi084 ceph-mon[27254]: pgmap v1146: 67 pgs: 67 active+clean; 121 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-11T03:48:00.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:00 smithi084 ceph-mon[27254]: osdmap e661: 12 total, 12 up, 12 in 2024-09-11T03:48:00.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:00 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4026031825' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:48:00.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:00 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4055327098' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"} : dispatch 2024-09-11T03:48:00.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:00.415 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:00.415 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:00.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:00 smithi121 ceph-mon[30729]: pgmap v1146: 67 pgs: 67 active+clean; 121 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-11T03:48:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:00 smithi121 ceph-mon[30729]: osdmap e661: 12 total, 12 up, 12 in 2024-09-11T03:48:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:00 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4026031825' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:48:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:00 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4055327098' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"} : dispatch 2024-09-11T03:48:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:00 smithi175 ceph-mon[30894]: pgmap v1146: 67 pgs: 67 active+clean; 121 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-11T03:48:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:00 smithi175 ceph-mon[30894]: osdmap e661: 12 total, 12 up, 12 in 2024-09-11T03:48:00.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:00 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4026031825' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:48:00.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:00 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4055327098' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"} : dispatch 2024-09-11T03:48:00.516 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:00.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:00.538 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:00.538 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:00.592 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:48:00.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:00.666 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:00.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:00.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:00.768 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:00.789 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:00.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:00.790 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:00.798 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:48:00.798 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:48:00.815 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:48:00.815 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:48:00.831 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:48:00.831 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:48:00.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:48:00.848 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:48:00.864 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:48:00.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:48:00.882 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:48:00.882 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:48:00.890 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:00.899 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:48:00.899 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:48:00.912 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:00.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:00.914 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:00.916 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:48:00.917 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:48:00.927 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-11T03:48:00.933 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:48:00.933 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:48:00.949 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:48:00.949 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:48:00.965 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:48:00.966 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:48:00.982 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:48:00.983 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:48:01.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:01.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:01.038 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:01.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:01.126 INFO:tasks.rados.rados.2.smithi175.stdout:2565: done (2 left) 2024-09-11T03:48:01.127 INFO:tasks.rados.rados.2.smithi175.stdout:2566: read oid 35 snap -1 2024-09-11T03:48:01.127 INFO:tasks.rados.rados.2.smithi175.stdout:2566: expect (ObjNum 864 snap 273 seq_num 864) 2024-09-11T03:48:01.127 INFO:tasks.rados.rados.2.smithi175.stdout:2567: delete oid 40 current snap is 282 2024-09-11T03:48:01.127 INFO:tasks.rados.rados.2.smithi175.stdout:2563: finishing write tid 3 to smithi17557870-49 2024-09-11T03:48:01.127 INFO:tasks.rados.rados.2.smithi175.stdout:2563: finishing write tid 4 to smithi17557870-49 2024-09-11T03:48:01.127 INFO:tasks.rados.rados.2.smithi175.stdout:2563: finishing write tid 5 to smithi17557870-49 2024-09-11T03:48:01.127 INFO:tasks.rados.rados.2.smithi175.stdout:2563: oid 49 updating version 0 to 728 2024-09-11T03:48:01.127 INFO:tasks.rados.rados.2.smithi175.stdout:2563: oid 49 version 728 is already newer than 726 2024-09-11T03:48:01.127 INFO:tasks.rados.rados.2.smithi175.stdout:2563: oid 49 version 728 is already newer than 727 2024-09-11T03:48:01.127 INFO:tasks.rados.rados.2.smithi175.stdout:2563: oid 49 version 728 is already newer than 725 2024-09-11T03:48:01.127 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 728 (ObjNum 882 snap 282 seq_num 882) dirty exists 2024-09-11T03:48:01.127 INFO:tasks.rados.rados.2.smithi175.stdout:2563: left oid 49 (ObjNum 882 snap 282 seq_num 882) 2024-09-11T03:48:01.127 INFO:tasks.rados.rados.2.smithi175.stdout:2564: finishing write tid 1 to smithi17557870-22 2024-09-11T03:48:01.127 INFO:tasks.rados.rados.2.smithi175.stdout:2564: finishing write tid 2 to smithi17557870-22 2024-09-11T03:48:01.128 INFO:tasks.rados.rados.2.smithi175.stdout:2564: finishing write tid 3 to smithi17557870-22 2024-09-11T03:48:01.128 INFO:tasks.rados.rados.2.smithi175.stdout:2564: finishing write tid 4 to smithi17557870-22 2024-09-11T03:48:01.128 INFO:tasks.rados.rados.2.smithi175.stdout:2564: finishing write tid 5 to smithi17557870-22 2024-09-11T03:48:01.128 INFO:tasks.rados.rados.2.smithi175.stdout:2564: oid 22 updating version 0 to 966 2024-09-11T03:48:01.128 INFO:tasks.rados.rados.2.smithi175.stdout:2564: oid 22 version 966 is already newer than 963 2024-09-11T03:48:01.128 INFO:tasks.rados.rados.2.smithi175.stdout:2564: oid 22 version 966 is already newer than 965 2024-09-11T03:48:01.128 INFO:tasks.rados.rados.2.smithi175.stdout:2564: oid 22 version 966 is already newer than 964 2024-09-11T03:48:01.128 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 966 (ObjNum 883 snap 282 seq_num 883) dirty exists 2024-09-11T03:48:01.128 INFO:tasks.rados.rados.2.smithi175.stdout:2564: left oid 22 (ObjNum 883 snap 282 seq_num 883) 2024-09-11T03:48:01.129 INFO:tasks.rados.rados.2.smithi175.stdout:2563: done (3 left) 2024-09-11T03:48:01.129 INFO:tasks.rados.rados.2.smithi175.stdout:2564: done (2 left) 2024-09-11T03:48:01.129 INFO:tasks.rados.rados.2.smithi175.stdout:2567: done (1 left) 2024-09-11T03:48:01.129 INFO:tasks.rados.rados.2.smithi175.stdout:2568: read oid 19 snap -1 2024-09-11T03:48:01.129 INFO:tasks.rados.rados.2.smithi175.stdout:2568: expect (ObjNum 877 snap 277 seq_num 877) 2024-09-11T03:48:01.129 INFO:tasks.rados.rados.2.smithi175.stdout:2569: rollback oid 9 current snap is 282 2024-09-11T03:48:01.129 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 9 to 279 2024-09-11T03:48:01.129 INFO:tasks.rados.rados.2.smithi175.stdout:2570: snap_create 2024-09-11T03:48:01.131 INFO:tasks.rados.rados.2.smithi175.stdout:2566: expect (ObjNum 864 snap 273 seq_num 864) 2024-09-11T03:48:01.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:01.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:01.165 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:01.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:01.216 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:48:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:01 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:48:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:01 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:48:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4026031825' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:48:01.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:01 smithi175 ceph-mon[30894]: osdmap e662: 12 total, 12 up, 12 in 2024-09-11T03:48:01.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:01 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:48:01.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:01 smithi175 ceph-mon[30894]: 2.9 scrub ok 2024-09-11T03:48:01.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:01 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:48:01.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:01 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:48:01.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3578464768' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:01.266 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:01.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:01.289 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:01.290 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:01.318 INFO:tasks.rados.rados.2.smithi175.stdout:2569: finishing rollback tid 1 to smithi17557870-9 2024-09-11T03:48:01.318 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 0 (ObjNum 545 snap 0 seq_num 1746355648) dirty dne 2024-09-11T03:48:01.318 INFO:tasks.rados.rados.2.smithi175.stdout:2568: expect (ObjNum 877 snap 277 seq_num 877) 2024-09-11T03:48:01.390 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:01 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:48:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:01 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:48:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4026031825' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:48:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:01 smithi084 ceph-mon[27254]: osdmap e662: 12 total, 12 up, 12 in 2024-09-11T03:48:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:01 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:48:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:01 smithi084 ceph-mon[27254]: 2.9 scrub ok 2024-09-11T03:48:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:01 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:48:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:01 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:48:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3578464768' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:01.412 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:01.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:01.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:01.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:01 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:48:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:01 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:48:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4026031825' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:48:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:01 smithi121 ceph-mon[30729]: osdmap e662: 12 total, 12 up, 12 in 2024-09-11T03:48:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:01 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:48:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:01 smithi121 ceph-mon[30729]: 2.9 scrub ok 2024-09-11T03:48:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:01 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:48:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:01 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:48:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3578464768' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:01.514 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:01.535 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:01.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:01.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:01.637 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:01.659 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:01.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:01.700 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:48:01.701 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:48:01.760 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:01.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:01.783 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:01.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:01.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:01.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:01.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:01.992 INFO:tasks.rados.rados.2.smithi175.stdout:2566: done (3 left) 2024-09-11T03:48:01.993 INFO:tasks.rados.rados.2.smithi175.stdout:2568: done (2 left) 2024-09-11T03:48:01.993 INFO:tasks.rados.rados.2.smithi175.stdout:2569: done (1 left) 2024-09-11T03:48:01.993 INFO:tasks.rados.rados.2.smithi175.stdout:2570: done (0 left) 2024-09-11T03:48:01.993 INFO:tasks.rados.rados.2.smithi175.stdout:2571: delete oid 9 current snap is 283 2024-09-11T03:48:01.997 INFO:tasks.rados.rados.2.smithi175.stdout:2571: done (0 left) 2024-09-11T03:48:01.997 INFO:tasks.rados.rados.2.smithi175.stdout:2572: read oid 36 snap -1 2024-09-11T03:48:01.997 INFO:tasks.rados.rados.2.smithi175.stdout:2572: expect (ObjNum 830 snap 263 seq_num 830) 2024-09-11T03:48:01.997 INFO:tasks.rados.rados.2.smithi175.stdout:2573: write oid 13 current snap is 283 2024-09-11T03:48:01.997 INFO:tasks.rados.rados.2.smithi175.stdout:2573: seq_num 886 ranges {491002=567562,1663399=520306,2814301=76701} 2024-09-11T03:48:02.002 INFO:tasks.rados.rados.2.smithi175.stdout:2573: writing smithi17557870-13 from 491002 to 1058564 tid 1 2024-09-11T03:48:02.009 INFO:tasks.rados.rados.2.smithi175.stdout:2573: writing smithi17557870-13 from 1663399 to 2183705 tid 2 2024-09-11T03:48:02.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:02.014 INFO:tasks.rados.rados.2.smithi175.stdout:2573: writing smithi17557870-13 from 2814301 to 2891002 tid 3 2024-09-11T03:48:02.014 INFO:tasks.rados.rados.2.smithi175.stdout:2574: copy_from oid 33 from oid 6 current snap is 283 2024-09-11T03:48:02.014 INFO:tasks.rados.rados.2.smithi175.stdout:2572: expect (ObjNum 830 snap 263 seq_num 830) 2024-09-11T03:48:02.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:02.037 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:02.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:02.139 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:02.160 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:02.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:02.197 INFO:tasks.rados.rados.2.smithi175.stdout:2573: finishing write tid 1 to smithi17557870-13 2024-09-11T03:48:02.197 INFO:tasks.rados.rados.2.smithi175.stdout:2573: finishing write tid 2 to smithi17557870-13 2024-09-11T03:48:02.197 INFO:tasks.rados.rados.2.smithi175.stdout:2573: finishing write tid 3 to smithi17557870-13 2024-09-11T03:48:02.197 INFO:tasks.rados.rados.2.smithi175.stdout:2572: done (2 left) 2024-09-11T03:48:02.197 INFO:tasks.rados.rados.2.smithi175.stdout:2575: write (excl) oid 17 current snap is 283 2024-09-11T03:48:02.197 INFO:tasks.rados.rados.2.smithi175.stdout:2573: finishing write tid 4 to smithi17557870-13 2024-09-11T03:48:02.198 INFO:tasks.rados.rados.2.smithi175.stdout:2575: seq_num 887 ranges {507809=642811,1721881=663964} 2024-09-11T03:48:02.203 INFO:tasks.rados.rados.2.smithi175.stdout:2575: writing smithi17557870-17 from 507809 to 1150620 tid 1 2024-09-11T03:48:02.210 INFO:tasks.rados.rados.2.smithi175.stdout:2575: writing smithi17557870-17 from 1721881 to 2385845 tid 2 2024-09-11T03:48:02.210 INFO:tasks.rados.rados.2.smithi175.stdout:2576: write oid 14 current snap is 283 2024-09-11T03:48:02.210 INFO:tasks.rados.rados.2.smithi175.stdout:2576: seq_num 888 ranges {524616=718060,1664010=523975,2752794=171822} 2024-09-11T03:48:02.216 INFO:tasks.rados.rados.2.smithi175.stdout:2576: writing smithi17557870-14 from 524616 to 1242676 tid 1 2024-09-11T03:48:02.221 INFO:tasks.rados.rados.2.smithi175.stdout:2576: writing smithi17557870-14 from 1664010 to 2187985 tid 2 2024-09-11T03:48:02.226 INFO:tasks.rados.rados.2.smithi175.stdout:2576: writing smithi17557870-14 from 2752794 to 2924616 tid 3 2024-09-11T03:48:02.226 INFO:tasks.rados.rados.2.smithi175.stdout:2577: delete oid 37 current snap is 283 2024-09-11T03:48:02.226 INFO:tasks.rados.rados.2.smithi175.stdout:2573: finishing write tid 5 to smithi17557870-13 2024-09-11T03:48:02.226 INFO:tasks.rados.rados.2.smithi175.stdout:2573: oid 13 updating version 0 to 934 2024-09-11T03:48:02.226 INFO:tasks.rados.rados.2.smithi175.stdout:2573: oid 13 version 934 is already newer than 933 2024-09-11T03:48:02.226 INFO:tasks.rados.rados.2.smithi175.stdout:2573: oid 13 updating version 934 to 935 2024-09-11T03:48:02.226 INFO:tasks.rados.rados.2.smithi175.stdout:2573: oid 13 version 935 is already newer than 932 2024-09-11T03:48:02.227 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 935 (ObjNum 886 snap 283 seq_num 886) dirty exists 2024-09-11T03:48:02.227 INFO:tasks.rados.rados.2.smithi175.stdout:2573: left oid 13 (ObjNum 886 snap 283 seq_num 886) 2024-09-11T03:48:02.227 INFO:tasks.rados.rados.2.smithi175.stdout:2574: finishing copy_from racing read to smithi17557870-33 2024-09-11T03:48:02.227 INFO:tasks.rados.rados.2.smithi175.stdout:2574: finishing copy_from to smithi17557870-33 2024-09-11T03:48:02.227 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 885 (ObjNum 857 snap 269 seq_num 857) dirty exists 2024-09-11T03:48:02.227 INFO:tasks.rados.rados.2.smithi175.stdout:2575: finishing write tid 1 to smithi17557870-17 2024-09-11T03:48:02.227 INFO:tasks.rados.rados.2.smithi175.stdout:2575: finishing write tid 2 to smithi17557870-17 2024-09-11T03:48:02.227 INFO:tasks.rados.rados.2.smithi175.stdout:2575: finishing write tid 3 to smithi17557870-17 2024-09-11T03:48:02.227 INFO:tasks.rados.rados.2.smithi175.stdout:2575: finishing write tid 4 to smithi17557870-17 2024-09-11T03:48:02.227 INFO:tasks.rados.rados.2.smithi175.stdout:2575: oid 17 updating version 0 to 890 2024-09-11T03:48:02.227 INFO:tasks.rados.rados.2.smithi175.stdout:2575: oid 17 updating version 890 to 891 2024-09-11T03:48:02.227 INFO:tasks.rados.rados.2.smithi175.stdout:2575: oid 17 version 891 is already newer than 889 2024-09-11T03:48:02.227 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 891 (ObjNum 887 snap 283 seq_num 887) dirty exists 2024-09-11T03:48:02.228 INFO:tasks.rados.rados.2.smithi175.stdout:2575: left oid 17 (ObjNum 887 snap 283 seq_num 887) 2024-09-11T03:48:02.228 INFO:tasks.rados.rados.2.smithi175.stdout:2576: finishing write tid 1 to smithi17557870-14 2024-09-11T03:48:02.228 INFO:tasks.rados.rados.2.smithi175.stdout:2576: finishing write tid 2 to smithi17557870-14 2024-09-11T03:48:02.229 INFO:tasks.rados.rados.2.smithi175.stdout:2573: done (4 left) 2024-09-11T03:48:02.230 INFO:tasks.rados.rados.2.smithi175.stdout:2574: done (3 left) 2024-09-11T03:48:02.230 INFO:tasks.rados.rados.2.smithi175.stdout:2575: done (2 left) 2024-09-11T03:48:02.230 INFO:tasks.rados.rados.2.smithi175.stdout:2577: done (1 left) 2024-09-11T03:48:02.230 INFO:tasks.rados.rados.2.smithi175.stdout:2578: rollback oid 6 current snap is 283 2024-09-11T03:48:02.230 INFO:tasks.rados.rados.2.smithi175.stdout:2576: finishing write tid 3 to smithi17557870-14 2024-09-11T03:48:02.230 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 6 to 275 2024-09-11T03:48:02.230 INFO:tasks.rados.rados.2.smithi175.stdout:2579: delete oid 34 current snap is 283 2024-09-11T03:48:02.230 INFO:tasks.rados.rados.2.smithi175.stdout:2576: finishing write tid 4 to smithi17557870-14 2024-09-11T03:48:02.230 INFO:tasks.rados.rados.2.smithi175.stdout:2576: finishing write tid 5 to smithi17557870-14 2024-09-11T03:48:02.230 INFO:tasks.rados.rados.2.smithi175.stdout:2576: oid 14 updating version 0 to 919 2024-09-11T03:48:02.230 INFO:tasks.rados.rados.2.smithi175.stdout:2576: oid 14 updating version 919 to 920 2024-09-11T03:48:02.230 INFO:tasks.rados.rados.2.smithi175.stdout:2576: oid 14 updating version 920 to 922 2024-09-11T03:48:02.230 INFO:tasks.rados.rados.2.smithi175.stdout:2576: oid 14 version 922 is already newer than 921 2024-09-11T03:48:02.231 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 922 (ObjNum 888 snap 283 seq_num 888) dirty exists 2024-09-11T03:48:02.231 INFO:tasks.rados.rados.2.smithi175.stdout:2576: left oid 14 (ObjNum 888 snap 283 seq_num 888) 2024-09-11T03:48:02.231 INFO:tasks.rados.rados.2.smithi175.stdout:2576: done (2 left) 2024-09-11T03:48:02.231 INFO:tasks.rados.rados.2.smithi175.stdout:2579: done (1 left) 2024-09-11T03:48:02.231 INFO:tasks.rados.rados.2.smithi175.stdout:2580: snap_create 2024-09-11T03:48:02.231 INFO:tasks.rados.rados.2.smithi175.stdout:2578: finishing rollback tid 0 to smithi17557870-6 2024-09-11T03:48:02.231 INFO:tasks.rados.rados.2.smithi175.stdout:2578: finishing rollback tid 1 to smithi17557870-6 2024-09-11T03:48:02.231 INFO:tasks.rados.rados.2.smithi175.stdout:2578: finishing rollback tid 2 to smithi17557870-6 2024-09-11T03:48:02.231 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 888 (ObjNum 857 snap 269 seq_num 857) dirty exists 2024-09-11T03:48:02.237 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:02 smithi175 ceph-mon[30894]: pgmap v1149: 67 pgs: 67 active+clean; 121 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-11T03:48:02.238 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:02 smithi175 ceph-mon[30894]: osdmap e663: 12 total, 12 up, 12 in 2024-09-11T03:48:02.238 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:02 smithi175 ceph-mon[30894]: 2.1f scrub starts 2024-09-11T03:48:02.238 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:02 smithi175 ceph-mon[30894]: 2.1f scrub ok 2024-09-11T03:48:02.238 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:02 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:48:02.238 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:02 smithi175 ceph-mon[30894]: 2.15 scrub ok 2024-09-11T03:48:02.238 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:02 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:02.238 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:02 smithi175 ceph-mon[30894]: osdmap e664: 12 total, 12 up, 12 in 2024-09-11T03:48:02.263 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:02.264 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-11T03:48:02.285 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:02.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:02.387 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:02.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:02 smithi084 ceph-mon[27254]: pgmap v1149: 67 pgs: 67 active+clean; 121 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-11T03:48:02.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:02 smithi084 ceph-mon[27254]: osdmap e663: 12 total, 12 up, 12 in 2024-09-11T03:48:02.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:02 smithi084 ceph-mon[27254]: 2.1f scrub starts 2024-09-11T03:48:02.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:02 smithi084 ceph-mon[27254]: 2.1f scrub ok 2024-09-11T03:48:02.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:02 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:48:02.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:02 smithi084 ceph-mon[27254]: 2.15 scrub ok 2024-09-11T03:48:02.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:02 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:02.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:02 smithi084 ceph-mon[27254]: osdmap e664: 12 total, 12 up, 12 in 2024-09-11T03:48:02.408 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:02.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:02.409 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:02.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:02 smithi121 ceph-mon[30729]: pgmap v1149: 67 pgs: 67 active+clean; 121 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-11T03:48:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:02 smithi121 ceph-mon[30729]: osdmap e663: 12 total, 12 up, 12 in 2024-09-11T03:48:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:02 smithi121 ceph-mon[30729]: 2.1f scrub starts 2024-09-11T03:48:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:02 smithi121 ceph-mon[30729]: 2.1f scrub ok 2024-09-11T03:48:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:02 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:48:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:02 smithi121 ceph-mon[30729]: 2.15 scrub ok 2024-09-11T03:48:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:02 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:02 smithi121 ceph-mon[30729]: osdmap e664: 12 total, 12 up, 12 in 2024-09-11T03:48:02.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:02.531 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:02.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:02.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:02.546 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:48:02.633 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:02.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:02.655 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:02.655 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:02.756 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:02.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:02.779 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:02.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:02.880 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:02.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:02.902 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:02.902 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:03.003 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:03.025 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:03.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:03.026 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:03.026 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:48:03.027 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:48:03.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:03.131 INFO:tasks.rados.rados.2.smithi175.stdout:2578: done (1 left) 2024-09-11T03:48:03.131 INFO:tasks.rados.rados.2.smithi175.stdout:2580: done (0 left) 2024-09-11T03:48:03.131 INFO:tasks.rados.rados.2.smithi175.stdout:2581: read oid 49 snap 280 2024-09-11T03:48:03.131 INFO:tasks.rados.rados.2.smithi175.stdout:2581: expect deleted 2024-09-11T03:48:03.132 INFO:tasks.rados.rados.2.smithi175.stdout:2582: write oid 8 current snap is 284 2024-09-11T03:48:03.132 INFO:tasks.rados.rados.2.smithi175.stdout:2582: seq_num 891 ranges {575037=660160,1955809=671302} 2024-09-11T03:48:03.138 INFO:tasks.rados.rados.2.smithi175.stdout:2582: writing smithi17557870-8 from 575037 to 1235197 tid 1 2024-09-11T03:48:03.145 INFO:tasks.rados.rados.2.smithi175.stdout:2582: writing smithi17557870-8 from 1955809 to 2627111 tid 2 2024-09-11T03:48:03.146 INFO:tasks.rados.rados.2.smithi175.stdout:2583: write oid 14 current snap is 284 2024-09-11T03:48:03.146 INFO:tasks.rados.rados.2.smithi175.stdout:2583: seq_num 892 ranges {591844=735409,1897938=531313,2862486=129358} 2024-09-11T03:48:03.153 INFO:tasks.rados.rados.2.smithi175.stdout:2583: writing smithi17557870-14 from 591844 to 1327253 tid 1 2024-09-11T03:48:03.153 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:03.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:03.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:03.159 INFO:tasks.rados.rados.2.smithi175.stdout:2583: writing smithi17557870-14 from 1897938 to 2429251 tid 2 2024-09-11T03:48:03.162 INFO:tasks.rados.rados.2.smithi175.stdout:2583: writing smithi17557870-14 from 2862486 to 2991844 tid 3 2024-09-11T03:48:03.162 INFO:tasks.rados.rados.2.smithi175.stdout:2584: write oid 15 current snap is 284 2024-09-11T03:48:03.162 INFO:tasks.rados.rados.2.smithi175.stdout:2582: finishing write tid 1 to smithi17557870-8 2024-09-11T03:48:03.162 INFO:tasks.rados.rados.2.smithi175.stdout:2584: seq_num 893 ranges {608651=410658,1556420=674971,2773556=235095} 2024-09-11T03:48:03.167 INFO:tasks.rados.rados.2.smithi175.stdout:2584: writing smithi17557870-15 from 608651 to 1019309 tid 1 2024-09-11T03:48:03.174 INFO:tasks.rados.rados.2.smithi175.stdout:2584: writing smithi17557870-15 from 1556420 to 2231391 tid 2 2024-09-11T03:48:03.179 INFO:tasks.rados.rados.2.smithi175.stdout:2584: writing smithi17557870-15 from 2773556 to 3008651 tid 3 2024-09-11T03:48:03.179 INFO:tasks.rados.rados.2.smithi175.stdout:2581: done (3 left) 2024-09-11T03:48:03.179 INFO:tasks.rados.rados.2.smithi175.stdout:2585: read oid 36 snap -1 2024-09-11T03:48:03.179 INFO:tasks.rados.rados.2.smithi175.stdout:2585: expect (ObjNum 830 snap 263 seq_num 830) 2024-09-11T03:48:03.179 INFO:tasks.rados.rados.2.smithi175.stdout:2586: rollback oid 18 current snap is 284 2024-09-11T03:48:03.180 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 18 to 283 2024-09-11T03:48:03.180 INFO:tasks.rados.rados.2.smithi175.stdout:2582: finishing write tid 2 to smithi17557870-8 2024-09-11T03:48:03.180 INFO:tasks.rados.rados.2.smithi175.stdout:2582: finishing write tid 3 to smithi17557870-8 2024-09-11T03:48:03.180 INFO:tasks.rados.rados.2.smithi175.stdout:2582: finishing write tid 4 to smithi17557870-8 2024-09-11T03:48:03.180 INFO:tasks.rados.rados.2.smithi175.stdout:2582: oid 8 updating version 0 to 897 2024-09-11T03:48:03.180 INFO:tasks.rados.rados.2.smithi175.stdout:2582: oid 8 version 897 is already newer than 896 2024-09-11T03:48:03.180 INFO:tasks.rados.rados.2.smithi175.stdout:2582: oid 8 updating version 897 to 898 2024-09-11T03:48:03.180 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 898 (ObjNum 891 snap 284 seq_num 891) dirty exists 2024-09-11T03:48:03.180 INFO:tasks.rados.rados.2.smithi175.stdout:2582: left oid 8 (ObjNum 891 snap 284 seq_num 891) 2024-09-11T03:48:03.180 INFO:tasks.rados.rados.2.smithi175.stdout:2583: finishing write tid 1 to smithi17557870-14 2024-09-11T03:48:03.180 INFO:tasks.rados.rados.2.smithi175.stdout:2583: finishing write tid 2 to smithi17557870-14 2024-09-11T03:48:03.180 INFO:tasks.rados.rados.2.smithi175.stdout:2583: finishing write tid 3 to smithi17557870-14 2024-09-11T03:48:03.180 INFO:tasks.rados.rados.2.smithi175.stdout:2582: done (4 left) 2024-09-11T03:48:03.181 INFO:tasks.rados.rados.2.smithi175.stdout:2587: write oid 17 current snap is 284 2024-09-11T03:48:03.181 INFO:tasks.rados.rados.2.smithi175.stdout:2583: finishing write tid 4 to smithi17557870-14 2024-09-11T03:48:03.181 INFO:tasks.rados.rados.2.smithi175.stdout:2583: finishing write tid 5 to smithi17557870-14 2024-09-11T03:48:03.181 INFO:tasks.rados.rados.2.smithi175.stdout:2583: oid 14 updating version 0 to 924 2024-09-11T03:48:03.181 INFO:tasks.rados.rados.2.smithi175.stdout:2583: oid 14 updating version 924 to 926 2024-09-11T03:48:03.181 INFO:tasks.rados.rados.2.smithi175.stdout:2583: oid 14 version 926 is already newer than 925 2024-09-11T03:48:03.181 INFO:tasks.rados.rados.2.smithi175.stdout:2583: oid 14 version 926 is already newer than 923 2024-09-11T03:48:03.181 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 926 (ObjNum 892 snap 284 seq_num 892) dirty exists 2024-09-11T03:48:03.181 INFO:tasks.rados.rados.2.smithi175.stdout:2583: left oid 14 (ObjNum 892 snap 284 seq_num 892) 2024-09-11T03:48:03.181 INFO:tasks.rados.rados.2.smithi175.stdout:2584: finishing write tid 1 to smithi17557870-15 2024-09-11T03:48:03.181 INFO:tasks.rados.rados.2.smithi175.stdout:2584: finishing write tid 2 to smithi17557870-15 2024-09-11T03:48:03.181 INFO:tasks.rados.rados.2.smithi175.stdout:2587: seq_num 894 ranges {625458=485907,1614902=534982,2917332=108126} 2024-09-11T03:48:03.185 INFO:tasks.rados.rados.2.smithi175.stdout:2587: writing smithi17557870-17 from 625458 to 1111365 tid 1 2024-09-11T03:48:03.190 INFO:tasks.rados.rados.2.smithi175.stdout:2587: writing smithi17557870-17 from 1614902 to 2149884 tid 2 2024-09-11T03:48:03.195 INFO:tasks.rados.rados.2.smithi175.stdout:2587: writing smithi17557870-17 from 2917332 to 3025458 tid 3 2024-09-11T03:48:03.195 INFO:tasks.rados.rados.2.smithi175.stdout:2583: done (4 left) 2024-09-11T03:48:03.195 INFO:tasks.rados.rados.2.smithi175.stdout:2588: write (excl) oid 30 current snap is 284 2024-09-11T03:48:03.195 INFO:tasks.rados.rados.2.smithi175.stdout:2588: seq_num 895 ranges {642265=561156,1673384=678640,2828402=213863} 2024-09-11T03:48:03.201 INFO:tasks.rados.rados.2.smithi175.stdout:2588: writing smithi17557870-30 from 642265 to 1203421 tid 1 2024-09-11T03:48:03.207 INFO:tasks.rados.rados.2.smithi175.stdout:2588: writing smithi17557870-30 from 1673384 to 2352024 tid 2 2024-09-11T03:48:03.211 INFO:tasks.rados.rados.2.smithi175.stdout:2588: writing smithi17557870-30 from 2828402 to 3042265 tid 3 2024-09-11T03:48:03.212 INFO:tasks.rados.rados.2.smithi175.stdout:2589: write oid 48 current snap is 284 2024-09-11T03:48:03.212 INFO:tasks.rados.rados.2.smithi175.stdout:2589: seq_num 896 ranges {659072=636405,1731866=422298,2855825=203247} 2024-09-11T03:48:03.218 INFO:tasks.rados.rados.2.smithi175.stdout:2589: writing smithi17557870-48 from 659072 to 1295477 tid 1 2024-09-11T03:48:03.223 INFO:tasks.rados.rados.2.smithi175.stdout:2589: writing smithi17557870-48 from 1731866 to 2154164 tid 2 2024-09-11T03:48:03.229 INFO:tasks.rados.rados.2.smithi175.stdout:2589: writing smithi17557870-48 from 2855825 to 3059072 tid 3 2024-09-11T03:48:03.229 INFO:tasks.rados.rados.2.smithi175.stdout:2590: rollback oid 46 current snap is 284 2024-09-11T03:48:03.229 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 46 to 282 2024-09-11T03:48:03.229 INFO:tasks.rados.rados.2.smithi175.stdout:2585: expect (ObjNum 830 snap 263 seq_num 830) 2024-09-11T03:48:03.255 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:03.276 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:03.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:03.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:03.378 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:03.400 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:03.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:03.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:03.414 INFO:tasks.rados.rados.2.smithi175.stdout:2584: finishing write tid 3 to smithi17557870-15 2024-09-11T03:48:03.414 INFO:tasks.rados.rados.2.smithi175.stdout:2584: finishing write tid 4 to smithi17557870-15 2024-09-11T03:48:03.414 INFO:tasks.rados.rados.2.smithi175.stdout:2585: done (6 left) 2024-09-11T03:48:03.414 INFO:tasks.rados.rados.2.smithi175.stdout:2591: read oid 32 snap 275 2024-09-11T03:48:03.414 INFO:tasks.rados.rados.2.smithi175.stdout:2591: expect (ObjNum 868 snap 274 seq_num 868) 2024-09-11T03:48:03.414 INFO:tasks.rados.rados.2.smithi175.stdout:2592: snap_create 2024-09-11T03:48:03.414 INFO:tasks.rados.rados.2.smithi175.stdout:2584: finishing write tid 5 to smithi17557870-15 2024-09-11T03:48:03.415 INFO:tasks.rados.rados.2.smithi175.stdout:2584: oid 15 updating version 0 to 280 2024-09-11T03:48:03.415 INFO:tasks.rados.rados.2.smithi175.stdout:2584: oid 15 updating version 280 to 281 2024-09-11T03:48:03.415 INFO:tasks.rados.rados.2.smithi175.stdout:2584: oid 15 version 281 is already newer than 279 2024-09-11T03:48:03.415 INFO:tasks.rados.rados.2.smithi175.stdout:2584: oid 15 version 281 is already newer than 278 2024-09-11T03:48:03.415 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 281 (ObjNum 893 snap 284 seq_num 893) dirty exists 2024-09-11T03:48:03.415 INFO:tasks.rados.rados.2.smithi175.stdout:2584: left oid 15 (ObjNum 893 snap 284 seq_num 893) 2024-09-11T03:48:03.415 INFO:tasks.rados.rados.2.smithi175.stdout:2586: finishing rollback tid 0 to smithi17557870-18 2024-09-11T03:48:03.415 INFO:tasks.rados.rados.2.smithi175.stdout:2586: finishing rollback tid 1 to smithi17557870-18 2024-09-11T03:48:03.415 INFO:tasks.rados.rados.2.smithi175.stdout:2586: finishing rollback tid 2 to smithi17557870-18 2024-09-11T03:48:03.415 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 484 (ObjNum 856 snap 269 seq_num 856) dirty exists 2024-09-11T03:48:03.415 INFO:tasks.rados.rados.2.smithi175.stdout:2587: finishing write tid 1 to smithi17557870-17 2024-09-11T03:48:03.415 INFO:tasks.rados.rados.2.smithi175.stdout:2587: finishing write tid 2 to smithi17557870-17 2024-09-11T03:48:03.415 INFO:tasks.rados.rados.2.smithi175.stdout:2587: finishing write tid 3 to smithi17557870-17 2024-09-11T03:48:03.416 INFO:tasks.rados.rados.2.smithi175.stdout:2587: finishing write tid 4 to smithi17557870-17 2024-09-11T03:48:03.416 INFO:tasks.rados.rados.2.smithi175.stdout:2587: finishing write tid 5 to smithi17557870-17 2024-09-11T03:48:03.416 INFO:tasks.rados.rados.2.smithi175.stdout:2587: oid 17 updating version 0 to 893 2024-09-11T03:48:03.416 INFO:tasks.rados.rados.2.smithi175.stdout:2587: oid 17 updating version 893 to 895 2024-09-11T03:48:03.416 INFO:tasks.rados.rados.2.smithi175.stdout:2587: oid 17 version 895 is already newer than 894 2024-09-11T03:48:03.416 INFO:tasks.rados.rados.2.smithi175.stdout:2587: oid 17 version 895 is already newer than 892 2024-09-11T03:48:03.416 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 895 (ObjNum 894 snap 284 seq_num 894) dirty exists 2024-09-11T03:48:03.416 INFO:tasks.rados.rados.2.smithi175.stdout:2587: left oid 17 (ObjNum 894 snap 284 seq_num 894) 2024-09-11T03:48:03.416 INFO:tasks.rados.rados.2.smithi175.stdout:2588: finishing write tid 1 to smithi17557870-30 2024-09-11T03:48:03.416 INFO:tasks.rados.rados.2.smithi175.stdout:2588: finishing write tid 2 to smithi17557870-30 2024-09-11T03:48:03.416 INFO:tasks.rados.rados.2.smithi175.stdout:2588: finishing write tid 3 to smithi17557870-30 2024-09-11T03:48:03.416 INFO:tasks.rados.rados.2.smithi175.stdout:2588: finishing write tid 4 to smithi17557870-30 2024-09-11T03:48:03.417 INFO:tasks.rados.rados.2.smithi175.stdout:2588: finishing write tid 5 to smithi17557870-30 2024-09-11T03:48:03.417 INFO:tasks.rados.rados.2.smithi175.stdout:2588: oid 30 updating version 0 to 901 2024-09-11T03:48:03.417 INFO:tasks.rados.rados.2.smithi175.stdout:2588: oid 30 updating version 901 to 902 2024-09-11T03:48:03.417 INFO:tasks.rados.rados.2.smithi175.stdout:2588: oid 30 updating version 902 to 903 2024-09-11T03:48:03.417 INFO:tasks.rados.rados.2.smithi175.stdout:2588: oid 30 version 903 is already newer than 900 2024-09-11T03:48:03.417 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 903 (ObjNum 895 snap 284 seq_num 895) dirty exists 2024-09-11T03:48:03.417 INFO:tasks.rados.rados.2.smithi175.stdout:2588: left oid 30 (ObjNum 895 snap 284 seq_num 895) 2024-09-11T03:48:03.417 INFO:tasks.rados.rados.2.smithi175.stdout:2589: finishing write tid 1 to smithi17557870-48 2024-09-11T03:48:03.417 INFO:tasks.rados.rados.2.smithi175.stdout:2589: finishing write tid 2 to smithi17557870-48 2024-09-11T03:48:03.417 INFO:tasks.rados.rados.2.smithi175.stdout:2589: finishing write tid 3 to smithi17557870-48 2024-09-11T03:48:03.417 INFO:tasks.rados.rados.2.smithi175.stdout:2589: finishing write tid 4 to smithi17557870-48 2024-09-11T03:48:03.417 INFO:tasks.rados.rados.2.smithi175.stdout:2589: finishing write tid 5 to smithi17557870-48 2024-09-11T03:48:03.417 INFO:tasks.rados.rados.2.smithi175.stdout:2589: oid 48 updating version 0 to 852 2024-09-11T03:48:03.417 INFO:tasks.rados.rados.2.smithi175.stdout:2589: oid 48 version 852 is already newer than 850 2024-09-11T03:48:03.418 INFO:tasks.rados.rados.2.smithi175.stdout:2589: oid 48 version 852 is already newer than 851 2024-09-11T03:48:03.418 INFO:tasks.rados.rados.2.smithi175.stdout:2589: oid 48 updating version 852 to 853 2024-09-11T03:48:03.418 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 853 (ObjNum 896 snap 284 seq_num 896) dirty exists 2024-09-11T03:48:03.418 INFO:tasks.rados.rados.2.smithi175.stdout:2589: left oid 48 (ObjNum 896 snap 284 seq_num 896) 2024-09-11T03:48:03.418 INFO:tasks.rados.rados.2.smithi175.stdout:2590: finishing rollback tid 0 to smithi17557870-46 2024-09-11T03:48:03.418 INFO:tasks.rados.rados.2.smithi175.stdout:2590: finishing rollback tid 1 to smithi17557870-46 2024-09-11T03:48:03.418 INFO:tasks.rados.rados.2.smithi175.stdout:2590: finishing rollback tid 2 to smithi17557870-46 2024-09-11T03:48:03.418 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 888 (ObjNum 876 snap 276 seq_num 876) dirty exists 2024-09-11T03:48:03.418 INFO:tasks.rados.rados.2.smithi175.stdout:2591: expect (ObjNum 868 snap 274 seq_num 868) 2024-09-11T03:48:03.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:03.528 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:03.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:03.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:03.630 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:03.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:03.653 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:03.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:03.697 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-11T03:48:03.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:03.776 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:03.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:03.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:03.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:03.904 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:03.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:03.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:03.936 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:48:04.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:04.017 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:48:04.029 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:04.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:04.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:04.135 INFO:tasks.rados.rados.2.smithi175.stdout:2584: done (7 left) 2024-09-11T03:48:04.135 INFO:tasks.rados.rados.2.smithi175.stdout:2586: done (6 left) 2024-09-11T03:48:04.135 INFO:tasks.rados.rados.2.smithi175.stdout:2587: done (5 left) 2024-09-11T03:48:04.135 INFO:tasks.rados.rados.2.smithi175.stdout:2588: done (4 left) 2024-09-11T03:48:04.135 INFO:tasks.rados.rados.2.smithi175.stdout:2589: done (3 left) 2024-09-11T03:48:04.135 INFO:tasks.rados.rados.2.smithi175.stdout:2590: done (2 left) 2024-09-11T03:48:04.135 INFO:tasks.rados.rados.2.smithi175.stdout:2591: done (1 left) 2024-09-11T03:48:04.135 INFO:tasks.rados.rados.2.smithi175.stdout:2592: done (0 left) 2024-09-11T03:48:04.135 INFO:tasks.rados.rados.2.smithi175.stdout:2593: delete oid 43 current snap is 285 2024-09-11T03:48:04.140 INFO:tasks.rados.rados.2.smithi175.stdout:2593: done (0 left) 2024-09-11T03:48:04.140 INFO:tasks.rados.rados.2.smithi175.stdout:2594: rollback oid 7 current snap is 285 2024-09-11T03:48:04.140 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 7 to 275 2024-09-11T03:48:04.140 INFO:tasks.rados.rados.2.smithi175.stdout:2595: read oid 31 snap -1 2024-09-11T03:48:04.141 INFO:tasks.rados.rados.2.smithi175.stdout:2595: expect deleted 2024-09-11T03:48:04.141 INFO:tasks.rados.rados.2.smithi175.stdout:2596: read oid 28 snap -1 2024-09-11T03:48:04.141 INFO:tasks.rados.rados.2.smithi175.stdout:2596: expect (ObjNum 874 snap 275 seq_num 874) 2024-09-11T03:48:04.141 INFO:tasks.rados.rados.2.smithi175.stdout:2597: delete oid 9 current snap is 285 2024-09-11T03:48:04.146 INFO:tasks.rados.rados.2.smithi175.stdout:2594: finishing rollback tid 0 to smithi17557870-7 2024-09-11T03:48:04.146 INFO:tasks.rados.rados.2.smithi175.stdout:2594: finishing rollback tid 1 to smithi17557870-7 2024-09-11T03:48:04.146 INFO:tasks.rados.rados.2.smithi175.stdout:2595: done (3 left) 2024-09-11T03:48:04.147 INFO:tasks.rados.rados.2.smithi175.stdout:2597: done (2 left) 2024-09-11T03:48:04.147 INFO:tasks.rados.rados.2.smithi175.stdout:2598: delete oid 31 current snap is 285 2024-09-11T03:48:04.147 INFO:tasks.rados.rados.2.smithi175.stdout:2594: finishing rollback tid 2 to smithi17557870-7 2024-09-11T03:48:04.147 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 840 (ObjNum 800 snap 255 seq_num 800) dirty exists 2024-09-11T03:48:04.148 INFO:tasks.rados.rados.2.smithi175.stdout:2594: done (2 left) 2024-09-11T03:48:04.149 INFO:tasks.rados.rados.2.smithi175.stdout:2598: done (1 left) 2024-09-11T03:48:04.149 INFO:tasks.rados.rados.2.smithi175.stdout:2599: write (excl) oid 15 current snap is 285 2024-09-11T03:48:04.149 INFO:tasks.rados.rados.2.smithi175.stdout:2599: seq_num 900 ranges {726300=653754,1965794=429636,2965517=160783} 2024-09-11T03:48:04.152 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:04.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:04.156 INFO:tasks.rados.rados.2.smithi175.stdout:2599: writing smithi17557870-15 from 726300 to 1380054 tid 1 2024-09-11T03:48:04.162 INFO:tasks.rados.rados.2.smithi175.stdout:2599: writing smithi17557870-15 from 1965794 to 2395430 tid 2 2024-09-11T03:48:04.166 INFO:tasks.rados.rados.2.smithi175.stdout:2599: writing smithi17557870-15 from 2965517 to 3126300 tid 3 2024-09-11T03:48:04.166 INFO:tasks.rados.rados.2.smithi175.stdout:2600: read oid 14 snap -1 2024-09-11T03:48:04.167 INFO:tasks.rados.rados.2.smithi175.stdout:2600: expect (ObjNum 892 snap 284 seq_num 892) 2024-09-11T03:48:04.167 INFO:tasks.rados.rados.2.smithi175.stdout:2601: delete oid 35 current snap is 285 2024-09-11T03:48:04.167 INFO:tasks.rados.rados.2.smithi175.stdout:2596: expect (ObjNum 874 snap 275 seq_num 874) 2024-09-11T03:48:04.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:48:04.255 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:04.276 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:04.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:04.289 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:04.331 INFO:tasks.rados.rados.2.smithi175.stdout:2599: finishing write tid 1 to smithi17557870-15 2024-09-11T03:48:04.331 INFO:tasks.rados.rados.2.smithi175.stdout:2599: finishing write tid 2 to smithi17557870-15 2024-09-11T03:48:04.331 INFO:tasks.rados.rados.2.smithi175.stdout:2599: finishing write tid 3 to smithi17557870-15 2024-09-11T03:48:04.331 INFO:tasks.rados.rados.2.smithi175.stdout:2599: finishing write tid 4 to smithi17557870-15 2024-09-11T03:48:04.331 INFO:tasks.rados.rados.2.smithi175.stdout:2599: finishing write tid 5 to smithi17557870-15 2024-09-11T03:48:04.332 INFO:tasks.rados.rados.2.smithi175.stdout:2599: oid 15 updating version 0 to 285 2024-09-11T03:48:04.332 INFO:tasks.rados.rados.2.smithi175.stdout:2599: oid 15 updating version 285 to 286 2024-09-11T03:48:04.332 INFO:tasks.rados.rados.2.smithi175.stdout:2599: oid 15 version 286 is already newer than 283 2024-09-11T03:48:04.332 INFO:tasks.rados.rados.2.smithi175.stdout:2599: oid 15 version 286 is already newer than 284 2024-09-11T03:48:04.332 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 286 (ObjNum 900 snap 285 seq_num 900) dirty exists 2024-09-11T03:48:04.332 INFO:tasks.rados.rados.2.smithi175.stdout:2599: left oid 15 (ObjNum 900 snap 285 seq_num 900) 2024-09-11T03:48:04.332 INFO:tasks.rados.rados.2.smithi175.stdout:2600: expect (ObjNum 892 snap 284 seq_num 892) 2024-09-11T03:48:04.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:04.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:04 smithi084 ceph-mon[27254]: pgmap v1152: 67 pgs: 67 active+clean; 131 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.6 MiB/s wr, 10 op/s 2024-09-11T03:48:04.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:04 smithi084 ceph-mon[27254]: osdmap e665: 12 total, 12 up, 12 in 2024-09-11T03:48:04.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:04 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:48:04.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:04 smithi084 ceph-mon[27254]: 2.a scrub ok 2024-09-11T03:48:04.403 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:04.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:04.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:04.408 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:04 smithi175 ceph-mon[30894]: pgmap v1152: 67 pgs: 67 active+clean; 131 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.6 MiB/s wr, 10 op/s 2024-09-11T03:48:04.408 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:04 smithi175 ceph-mon[30894]: osdmap e665: 12 total, 12 up, 12 in 2024-09-11T03:48:04.408 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:04 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:48:04.408 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:04 smithi175 ceph-mon[30894]: 2.a scrub ok 2024-09-11T03:48:04.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:04 smithi121 ceph-mon[30729]: pgmap v1152: 67 pgs: 67 active+clean; 131 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.6 MiB/s wr, 10 op/s 2024-09-11T03:48:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:04 smithi121 ceph-mon[30729]: osdmap e665: 12 total, 12 up, 12 in 2024-09-11T03:48:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:04 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:48:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:04 smithi121 ceph-mon[30729]: 2.a scrub ok 2024-09-11T03:48:04.470 INFO:tasks.rados.rados.2.smithi175.stdout:2596: done (3 left) 2024-09-11T03:48:04.470 INFO:tasks.rados.rados.2.smithi175.stdout:2599: done (2 left) 2024-09-11T03:48:04.470 INFO:tasks.rados.rados.2.smithi175.stdout:2600: done (1 left) 2024-09-11T03:48:04.470 INFO:tasks.rados.rados.2.smithi175.stdout:2601: done (0 left) 2024-09-11T03:48:04.470 INFO:tasks.rados.rados.2.smithi175.stdout:2602: rollback oid 46 current snap is 285 2024-09-11T03:48:04.470 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 46 to 284 2024-09-11T03:48:04.470 INFO:tasks.rados.rados.2.smithi175.stdout:2603: read oid 40 snap -1 2024-09-11T03:48:04.470 INFO:tasks.rados.rados.2.smithi175.stdout:2603: expect deleted 2024-09-11T03:48:04.471 INFO:tasks.rados.rados.2.smithi175.stdout:2604: rollback oid 14 current snap is 285 2024-09-11T03:48:04.471 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 14 to 279 2024-09-11T03:48:04.471 INFO:tasks.rados.rados.2.smithi175.stdout:2605: copy_from oid 15 from oid 33 current snap is 285 2024-09-11T03:48:04.471 INFO:tasks.rados.rados.2.smithi175.stdout:2606: read oid 45 snap -1 2024-09-11T03:48:04.471 INFO:tasks.rados.rados.2.smithi175.stdout:2606: expect (ObjNum 880 snap 281 seq_num 880) 2024-09-11T03:48:04.471 INFO:tasks.rados.rados.2.smithi175.stdout:2607: copy_from oid 22 from oid 42 current snap is 285 2024-09-11T03:48:04.471 INFO:tasks.rados.rados.2.smithi175.stdout:2608: snap_remove snap 282 2024-09-11T03:48:04.506 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:04.510 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:48:04.510 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:48:04.527 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:04.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:04.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:04.629 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:04.650 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:04.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:04.651 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:04.752 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:04.759 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:48:04.759 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:48:04.774 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:04.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:04.775 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:04.776 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:48:04.777 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:48:04.799 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:48:04.799 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:48:04.816 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:48:04.817 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:48:04.818 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:04.834 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:48:04.835 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:48:04.853 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:48:04.853 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:48:04.870 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:48:04.870 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:48:04.876 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:04.887 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:48:04.887 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:48:04.897 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:04.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:04.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:04.904 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:48:04.904 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:48:04.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:05.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:05.027 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:05.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:05.123 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:48:05.124 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:48:05.128 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:05.142 INFO:tasks.rados.rados.2.smithi175.stdout:2608: done (6 left) 2024-09-11T03:48:05.142 INFO:tasks.rados.rados.2.smithi175.stdout:2609: delete oid 12 current snap is 285 2024-09-11T03:48:05.143 INFO:tasks.rados.rados.2.smithi175.stdout:2604: finishing rollback tid 0 to smithi17557870-14 2024-09-11T03:48:05.143 INFO:tasks.rados.rados.2.smithi175.stdout:2604: finishing rollback tid 1 to smithi17557870-14 2024-09-11T03:48:05.143 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 928 (ObjNum 33 snap 0 seq_num 198288930) dirty dne 2024-09-11T03:48:05.143 INFO:tasks.rados.rados.2.smithi175.stdout:2602: finishing rollback tid 0 to smithi17557870-46 2024-09-11T03:48:05.143 INFO:tasks.rados.rados.2.smithi175.stdout:2602: finishing rollback tid 1 to smithi17557870-46 2024-09-11T03:48:05.143 INFO:tasks.rados.rados.2.smithi175.stdout:2602: finishing rollback tid 2 to smithi17557870-46 2024-09-11T03:48:05.143 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 891 (ObjNum 876 snap 276 seq_num 876) dirty exists 2024-09-11T03:48:05.143 INFO:tasks.rados.rados.2.smithi175.stdout:2607: finishing copy_from to smithi17557870-22 2024-09-11T03:48:05.143 INFO:tasks.rados.rados.2.smithi175.stdout:2607: got expected ENOENT (src dne) 2024-09-11T03:48:05.143 INFO:tasks.rados.rados.2.smithi175.stdout:2607: finishing copy_from racing read to smithi17557870-22 2024-09-11T03:48:05.143 INFO:tasks.rados.rados.2.smithi175.stdout:2606: expect (ObjNum 880 snap 281 seq_num 880) 2024-09-11T03:48:05.145 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:48:05.145 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:48:05.153 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:05.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:05.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:05.163 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:48:05.163 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:48:05.235 INFO:tasks.rados.rados.2.smithi175.stdout:2605: finishing copy_from racing read to smithi17557870-15 2024-09-11T03:48:05.235 INFO:tasks.rados.rados.2.smithi175.stdout:2605: finishing copy_from to smithi17557870-15 2024-09-11T03:48:05.236 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 885 (ObjNum 857 snap 269 seq_num 857) dirty exists 2024-09-11T03:48:05.237 INFO:tasks.rados.rados.2.smithi175.stdout:2602: done (6 left) 2024-09-11T03:48:05.238 INFO:tasks.rados.rados.2.smithi175.stdout:2603: done (5 left) 2024-09-11T03:48:05.238 INFO:tasks.rados.rados.2.smithi175.stdout:2604: done (4 left) 2024-09-11T03:48:05.238 INFO:tasks.rados.rados.2.smithi175.stdout:2605: done (3 left) 2024-09-11T03:48:05.238 INFO:tasks.rados.rados.2.smithi175.stdout:2606: done (2 left) 2024-09-11T03:48:05.238 INFO:tasks.rados.rados.2.smithi175.stdout:2607: done (1 left) 2024-09-11T03:48:05.238 INFO:tasks.rados.rados.2.smithi175.stdout:2609: done (0 left) 2024-09-11T03:48:05.238 INFO:tasks.rados.rados.2.smithi175.stdout:2610: snap_remove snap 275 2024-09-11T03:48:05.254 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:05.275 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:05.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:05.276 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:05.372 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:05 smithi175 ceph-mon[30894]: 2.12 scrub starts 2024-09-11T03:48:05.372 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:05 smithi175 ceph-mon[30894]: 2.12 scrub ok 2024-09-11T03:48:05.372 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:05 smithi175 ceph-mon[30894]: osdmap e666: 12 total, 12 up, 12 in 2024-09-11T03:48:05.372 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:05 smithi175 ceph-mon[30894]: 2.1b scrub starts 2024-09-11T03:48:05.372 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:05 smithi175 ceph-mon[30894]: 2.1b scrub ok 2024-09-11T03:48:05.372 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:05 smithi175 ceph-mon[30894]: 2.1a scrub starts 2024-09-11T03:48:05.372 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:05 smithi175 ceph-mon[30894]: 2.1a scrub ok 2024-09-11T03:48:05.372 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:05 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3288489464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:05.373 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-11T03:48:05.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:05.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:05 smithi084 ceph-mon[27254]: 2.12 scrub starts 2024-09-11T03:48:05.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:05 smithi084 ceph-mon[27254]: 2.12 scrub ok 2024-09-11T03:48:05.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:05 smithi084 ceph-mon[27254]: osdmap e666: 12 total, 12 up, 12 in 2024-09-11T03:48:05.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:05 smithi084 ceph-mon[27254]: 2.1b scrub starts 2024-09-11T03:48:05.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:05 smithi084 ceph-mon[27254]: 2.1b scrub ok 2024-09-11T03:48:05.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:05 smithi084 ceph-mon[27254]: 2.1a scrub starts 2024-09-11T03:48:05.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:05 smithi084 ceph-mon[27254]: 2.1a scrub ok 2024-09-11T03:48:05.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:05 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3288489464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:05.402 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:05.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:05.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:05 smithi121 ceph-mon[30729]: 2.12 scrub starts 2024-09-11T03:48:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:05 smithi121 ceph-mon[30729]: 2.12 scrub ok 2024-09-11T03:48:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:05 smithi121 ceph-mon[30729]: osdmap e666: 12 total, 12 up, 12 in 2024-09-11T03:48:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:05 smithi121 ceph-mon[30729]: 2.1b scrub starts 2024-09-11T03:48:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:05 smithi121 ceph-mon[30729]: 2.1b scrub ok 2024-09-11T03:48:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:05 smithi121 ceph-mon[30729]: 2.1a scrub starts 2024-09-11T03:48:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:05 smithi121 ceph-mon[30729]: 2.1a scrub ok 2024-09-11T03:48:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:05 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3288489464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:05.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:05.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:05.526 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:05.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:05.594 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:48:05.594 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:48:05.594 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-11T03:48:05.594 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T03:48:05.594 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T03:48:05.595 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-11T03:48:05.615 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:48:05.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:05.653 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:05.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:05.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:05.755 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:05.776 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:05.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:05.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:05.878 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:05.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:05.900 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:05.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:06.000 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:06.021 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:06.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:06.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:06.052 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:48:06.052 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:48:06.052 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:48:06.052 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:48:06.052 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:48:06.123 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:06.129 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:06.144 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:48:06.147 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:06.148 INFO:tasks.rados.rados.2.smithi175.stdout:2610: done (0 left) 2024-09-11T03:48:06.148 INFO:tasks.rados.rados.2.smithi175.stdout:2611: rollback oid 23 current snap is 285 2024-09-11T03:48:06.148 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 23 to 274 2024-09-11T03:48:06.148 INFO:tasks.rados.rados.2.smithi175.stdout:2612: read oid 39 snap 279 2024-09-11T03:48:06.148 INFO:tasks.rados.rados.2.smithi175.stdout:2612: expect deleted 2024-09-11T03:48:06.148 INFO:tasks.rados.rados.2.smithi175.stdout:2613: write (excl) oid 41 current snap is 285 2024-09-11T03:48:06.148 INFO:tasks.rados.rados.2.smithi175.stdout:2613: seq_num 903 ranges {776721=479501,2024887=693316} 2024-09-11T03:48:06.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:06.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:06.155 INFO:tasks.rados.rados.2.smithi175.stdout:2613: writing smithi17557870-41 from 776721 to 1256222 tid 1 2024-09-11T03:48:06.164 INFO:tasks.rados.rados.2.smithi175.stdout:2613: writing smithi17557870-41 from 2024887 to 2718203 tid 2 2024-09-11T03:48:06.164 INFO:tasks.rados.rados.2.smithi175.stdout:2614: read oid 1 snap -1 2024-09-11T03:48:06.164 INFO:tasks.rados.rados.2.smithi175.stdout:2614: expect (ObjNum 758 snap 246 seq_num 758) 2024-09-11T03:48:06.164 INFO:tasks.rados.rados.2.smithi175.stdout:2615: snap_create 2024-09-11T03:48:06.164 INFO:tasks.rados.rados.2.smithi175.stdout:2611: finishing rollback tid 0 to smithi17557870-23 2024-09-11T03:48:06.164 INFO:tasks.rados.rados.2.smithi175.stdout:2611: finishing rollback tid 1 to smithi17557870-23 2024-09-11T03:48:06.165 INFO:tasks.rados.rados.2.smithi175.stdout:2611: finishing rollback tid 2 to smithi17557870-23 2024-09-11T03:48:06.165 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 868 (ObjNum 855 snap 268 seq_num 855) dirty exists 2024-09-11T03:48:06.165 INFO:tasks.rados.rados.2.smithi175.stdout:2613: finishing write tid 1 to smithi17557870-41 2024-09-11T03:48:06.168 INFO:tasks.rados.rados.2.smithi175.stdout:2614: expect (ObjNum 758 snap 246 seq_num 758) 2024-09-11T03:48:06.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:06.262 INFO:tasks.rados.rados.2.smithi175.stdout:2613: finishing write tid 2 to smithi17557870-41 2024-09-11T03:48:06.262 INFO:tasks.rados.rados.2.smithi175.stdout:2613: finishing write tid 3 to smithi17557870-41 2024-09-11T03:48:06.262 INFO:tasks.rados.rados.2.smithi175.stdout:2613: finishing write tid 4 to smithi17557870-41 2024-09-11T03:48:06.262 INFO:tasks.rados.rados.2.smithi175.stdout:2613: oid 41 updating version 0 to 963 2024-09-11T03:48:06.262 INFO:tasks.rados.rados.2.smithi175.stdout:2613: oid 41 updating version 963 to 964 2024-09-11T03:48:06.262 INFO:tasks.rados.rados.2.smithi175.stdout:2613: oid 41 updating version 964 to 965 2024-09-11T03:48:06.262 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 965 (ObjNum 903 snap 285 seq_num 903) dirty exists 2024-09-11T03:48:06.262 INFO:tasks.rados.rados.2.smithi175.stdout:2613: left oid 41 (ObjNum 903 snap 285 seq_num 903) 2024-09-11T03:48:06.272 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:06.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:06.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:06.374 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:06 smithi084 ceph-mon[27254]: osdmap e667: 12 total, 12 up, 12 in 2024-09-11T03:48:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:06 smithi084 ceph-mon[27254]: 1.4 scrub starts 2024-09-11T03:48:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:06 smithi084 ceph-mon[27254]: 1.4 scrub ok 2024-09-11T03:48:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:06 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:48:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:06 smithi084 ceph-mon[27254]: 2.2 scrub ok 2024-09-11T03:48:06.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:06 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1657603008' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:48:06.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:48:06.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:48:06.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:48:06.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:06 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:48:06.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:06.396 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:06.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:06.413 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:06 smithi175 ceph-mon[30894]: osdmap e667: 12 total, 12 up, 12 in 2024-09-11T03:48:06.413 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:06 smithi175 ceph-mon[30894]: 1.4 scrub starts 2024-09-11T03:48:06.413 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:06 smithi175 ceph-mon[30894]: 1.4 scrub ok 2024-09-11T03:48:06.413 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:06 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:48:06.413 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:06 smithi175 ceph-mon[30894]: 2.2 scrub ok 2024-09-11T03:48:06.413 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:06 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1657603008' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:48:06.413 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:48:06.413 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:48:06.413 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:48:06.413 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:06 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:48:06.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:06 smithi121 ceph-mon[30729]: osdmap e667: 12 total, 12 up, 12 in 2024-09-11T03:48:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:06 smithi121 ceph-mon[30729]: 1.4 scrub starts 2024-09-11T03:48:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:06 smithi121 ceph-mon[30729]: 1.4 scrub ok 2024-09-11T03:48:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:06 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:48:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:06 smithi121 ceph-mon[30729]: 2.2 scrub ok 2024-09-11T03:48:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:06 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1657603008' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:48:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:06 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:48:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:06 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:48:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:06 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:48:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:06 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:48:06.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:06.519 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:06.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:06.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:06.621 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:06.642 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:06.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:06.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:06.744 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:06.766 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:06.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:06.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:06.824 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-11T03:48:06.868 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:06.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:06.890 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:06.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:06.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:06.992 INFO:tasks.rados.rados.2.smithi175.stdout:2611: done (4 left) 2024-09-11T03:48:06.992 INFO:tasks.rados.rados.2.smithi175.stdout:2612: done (3 left) 2024-09-11T03:48:06.993 INFO:tasks.rados.rados.2.smithi175.stdout:2613: done (2 left) 2024-09-11T03:48:06.993 INFO:tasks.rados.rados.2.smithi175.stdout:2614: done (1 left) 2024-09-11T03:48:06.993 INFO:tasks.rados.rados.2.smithi175.stdout:2615: done (0 left) 2024-09-11T03:48:06.993 INFO:tasks.rados.rados.2.smithi175.stdout:2616: read oid 21 snap 285 2024-09-11T03:48:06.993 INFO:tasks.rados.rados.2.smithi175.stdout:2616: expect deleted 2024-09-11T03:48:06.993 INFO:tasks.rados.rados.2.smithi175.stdout:2617: rollback oid 12 current snap is 286 2024-09-11T03:48:06.993 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 12 to 279 2024-09-11T03:48:06.993 INFO:tasks.rados.rados.2.smithi175.stdout:2618: read oid 36 snap 279 2024-09-11T03:48:06.993 INFO:tasks.rados.rados.2.smithi175.stdout:2618: expect (ObjNum 830 snap 263 seq_num 830) 2024-09-11T03:48:06.993 INFO:tasks.rados.rados.2.smithi175.stdout:2619: write oid 5 current snap is 286 2024-09-11T03:48:06.993 INFO:tasks.rados.rados.2.smithi175.stdout:2619: seq_num 904 ranges {793528=554750,2083369=436974,2958856=234672} 2024-09-11T03:48:07.000 INFO:tasks.rados.rados.2.smithi175.stdout:2619: writing smithi17557870-5 from 793528 to 1348278 tid 1 2024-09-11T03:48:07.007 INFO:tasks.rados.rados.2.smithi175.stdout:2619: writing smithi17557870-5 from 2083369 to 2520343 tid 2 2024-09-11T03:48:07.011 INFO:tasks.rados.rados.2.smithi175.stdout:2619: writing smithi17557870-5 from 2958856 to 3193528 tid 3 2024-09-11T03:48:07.011 INFO:tasks.rados.rados.2.smithi175.stdout:2620: read oid 43 snap -1 2024-09-11T03:48:07.011 INFO:tasks.rados.rados.2.smithi175.stdout:2620: expect deleted 2024-09-11T03:48:07.011 INFO:tasks.rados.rados.2.smithi175.stdout:2621: read oid 14 snap -1 2024-09-11T03:48:07.012 INFO:tasks.rados.rados.2.smithi175.stdout:2621: expect deleted 2024-09-11T03:48:07.012 INFO:tasks.rados.rados.2.smithi175.stdout:2622: rollback oid 42 current snap is 286 2024-09-11T03:48:07.012 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 42 to 280 2024-09-11T03:48:07.012 INFO:tasks.rados.rados.2.smithi175.stdout:2617: finishing rollback tid 1 to smithi17557870-12 2024-09-11T03:48:07.012 INFO:tasks.rados.rados.2.smithi175.stdout:2617: finishing rollback tid 2 to smithi17557870-12 2024-09-11T03:48:07.012 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 937 (ObjNum 873 snap 274 seq_num 873) dirty exists 2024-09-11T03:48:07.012 INFO:tasks.rados.rados.2.smithi175.stdout:2618: expect (ObjNum 830 snap 263 seq_num 830) 2024-09-11T03:48:07.014 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:07.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:07.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:07.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:07.137 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:07.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:07.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:07.201 INFO:tasks.rados.rados.2.smithi175.stdout:2619: finishing write tid 1 to smithi17557870-5 2024-09-11T03:48:07.201 INFO:tasks.rados.rados.2.smithi175.stdout:2619: finishing write tid 2 to smithi17557870-5 2024-09-11T03:48:07.201 INFO:tasks.rados.rados.2.smithi175.stdout:2616: done (6 left) 2024-09-11T03:48:07.201 INFO:tasks.rados.rados.2.smithi175.stdout:2617: done (5 left) 2024-09-11T03:48:07.201 INFO:tasks.rados.rados.2.smithi175.stdout:2618: done (4 left) 2024-09-11T03:48:07.201 INFO:tasks.rados.rados.2.smithi175.stdout:2623: read oid 21 snap -1 2024-09-11T03:48:07.201 INFO:tasks.rados.rados.2.smithi175.stdout:2623: expect deleted 2024-09-11T03:48:07.201 INFO:tasks.rados.rados.2.smithi175.stdout:2624: write oid 34 current snap is 286 2024-09-11T03:48:07.201 INFO:tasks.rados.rados.2.smithi175.stdout:2622: finishing rollback tid 1 to smithi17557870-42 2024-09-11T03:48:07.201 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 0 (ObjNum 129 snap 0 seq_num 1745197232) dirty dne 2024-09-11T03:48:07.202 INFO:tasks.rados.rados.2.smithi175.stdout:2619: finishing write tid 3 to smithi17557870-5 2024-09-11T03:48:07.202 INFO:tasks.rados.rados.2.smithi175.stdout:2619: finishing write tid 4 to smithi17557870-5 2024-09-11T03:48:07.202 INFO:tasks.rados.rados.2.smithi175.stdout:2619: finishing write tid 5 to smithi17557870-5 2024-09-11T03:48:07.202 INFO:tasks.rados.rados.2.smithi175.stdout:2619: oid 5 updating version 0 to 716 2024-09-11T03:48:07.202 INFO:tasks.rados.rados.2.smithi175.stdout:2619: oid 5 updating version 716 to 718 2024-09-11T03:48:07.202 INFO:tasks.rados.rados.2.smithi175.stdout:2619: oid 5 version 718 is already newer than 717 2024-09-11T03:48:07.202 INFO:tasks.rados.rados.2.smithi175.stdout:2619: oid 5 updating version 718 to 719 2024-09-11T03:48:07.202 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 719 (ObjNum 904 snap 286 seq_num 904) dirty exists 2024-09-11T03:48:07.202 INFO:tasks.rados.rados.2.smithi175.stdout:2619: left oid 5 (ObjNum 904 snap 286 seq_num 904) 2024-09-11T03:48:07.202 INFO:tasks.rados.rados.2.smithi175.stdout:2624: seq_num 905 ranges {410335=746352,1741851=696985,3102632=107703} 2024-09-11T03:48:07.207 INFO:tasks.rados.rados.2.smithi175.stdout:2624: writing smithi17557870-34 from 410335 to 1156687 tid 1 2024-09-11T03:48:07.214 INFO:tasks.rados.rados.2.smithi175.stdout:2624: writing smithi17557870-34 from 1741851 to 2438836 tid 2 2024-09-11T03:48:07.219 INFO:tasks.rados.rados.2.smithi175.stdout:2624: writing smithi17557870-34 from 3102632 to 3210335 tid 3 2024-09-11T03:48:07.219 INFO:tasks.rados.rados.2.smithi175.stdout:2619: done (5 left) 2024-09-11T03:48:07.219 INFO:tasks.rados.rados.2.smithi175.stdout:2620: done (4 left) 2024-09-11T03:48:07.219 INFO:tasks.rados.rados.2.smithi175.stdout:2621: done (3 left) 2024-09-11T03:48:07.219 INFO:tasks.rados.rados.2.smithi175.stdout:2622: done (2 left) 2024-09-11T03:48:07.219 INFO:tasks.rados.rados.2.smithi175.stdout:2625: delete oid 44 current snap is 286 2024-09-11T03:48:07.219 INFO:tasks.rados.rados.2.smithi175.stdout:2624: finishing write tid 1 to smithi17557870-34 2024-09-11T03:48:07.219 INFO:tasks.rados.rados.2.smithi175.stdout:2624: finishing write tid 2 to smithi17557870-34 2024-09-11T03:48:07.221 INFO:tasks.rados.rados.2.smithi175.stdout:2623: done (2 left) 2024-09-11T03:48:07.221 INFO:tasks.rados.rados.2.smithi175.stdout:2625: done (1 left) 2024-09-11T03:48:07.221 INFO:tasks.rados.rados.2.smithi175.stdout:2626: copy_from oid 8 from oid 13 current snap is 286 2024-09-11T03:48:07.221 INFO:tasks.rados.rados.2.smithi175.stdout:2627: copy_from oid 3 from oid 2 current snap is 286 2024-09-11T03:48:07.221 INFO:tasks.rados.rados.2.smithi175.stdout:2628: read oid 36 snap -1 2024-09-11T03:48:07.221 INFO:tasks.rados.rados.2.smithi175.stdout:2628: expect (ObjNum 830 snap 263 seq_num 830) 2024-09-11T03:48:07.221 INFO:tasks.rados.rados.2.smithi175.stdout:2629: snap_create 2024-09-11T03:48:07.221 INFO:tasks.rados.rados.2.smithi175.stdout:2624: finishing write tid 3 to smithi17557870-34 2024-09-11T03:48:07.222 INFO:tasks.rados.rados.2.smithi175.stdout:2624: finishing write tid 4 to smithi17557870-34 2024-09-11T03:48:07.222 INFO:tasks.rados.rados.2.smithi175.stdout:2624: finishing write tid 5 to smithi17557870-34 2024-09-11T03:48:07.222 INFO:tasks.rados.rados.2.smithi175.stdout:2624: oid 34 updating version 0 to 882 2024-09-11T03:48:07.222 INFO:tasks.rados.rados.2.smithi175.stdout:2624: oid 34 updating version 882 to 883 2024-09-11T03:48:07.222 INFO:tasks.rados.rados.2.smithi175.stdout:2624: oid 34 updating version 883 to 884 2024-09-11T03:48:07.222 INFO:tasks.rados.rados.2.smithi175.stdout:2624: oid 34 version 884 is already newer than 881 2024-09-11T03:48:07.222 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 884 (ObjNum 905 snap 286 seq_num 905) dirty exists 2024-09-11T03:48:07.222 INFO:tasks.rados.rados.2.smithi175.stdout:2624: left oid 34 (ObjNum 905 snap 286 seq_num 905) 2024-09-11T03:48:07.225 INFO:tasks.rados.rados.2.smithi175.stdout:2628: expect (ObjNum 830 snap 263 seq_num 830) 2024-09-11T03:48:07.238 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:07.258 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:48:07.258 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T03:48:07.259 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:48:07.261 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:07.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:07.262 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:07.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:07.384 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:07.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:07.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:07.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:07 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:48:07.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:07 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:48:07.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1657603008' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:48:07.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:07 smithi084 ceph-mon[27254]: osdmap e668: 12 total, 12 up, 12 in 2024-09-11T03:48:07.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:07 smithi084 ceph-mon[27254]: 2.3 scrub starts 2024-09-11T03:48:07.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:07 smithi084 ceph-mon[27254]: 2.3 scrub ok 2024-09-11T03:48:07.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:07 smithi084 ceph-mon[27254]: 2.19 scrub starts 2024-09-11T03:48:07.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:07 smithi084 ceph-mon[27254]: 2.19 scrub ok 2024-09-11T03:48:07.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:07 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:48:07.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:07 smithi084 ceph-mon[27254]: 2.e scrub ok 2024-09-11T03:48:07.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:07 smithi084 ceph-mon[27254]: osdmap e669: 12 total, 12 up, 12 in 2024-09-11T03:48:07.413 INFO:tasks.rados.rados.2.smithi175.stdout:2626: finishing copy_from racing read to smithi17557870-8 2024-09-11T03:48:07.413 INFO:tasks.rados.rados.2.smithi175.stdout:2627: finishing copy_from racing read to smithi17557870-3 2024-09-11T03:48:07.413 INFO:tasks.rados.rados.2.smithi175.stdout:2627: finishing copy_from to smithi17557870-3 2024-09-11T03:48:07.413 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 950 (ObjNum 765 snap 247 seq_num 765) dirty exists 2024-09-11T03:48:07.413 INFO:tasks.rados.rados.2.smithi175.stdout:2626: finishing copy_from to smithi17557870-8 2024-09-11T03:48:07.413 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 935 (ObjNum 886 snap 283 seq_num 886) dirty exists 2024-09-11T03:48:07.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:07 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:48:07.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:07 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:48:07.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1657603008' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:48:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:07 smithi121 ceph-mon[30729]: osdmap e668: 12 total, 12 up, 12 in 2024-09-11T03:48:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:07 smithi121 ceph-mon[30729]: 2.3 scrub starts 2024-09-11T03:48:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:07 smithi121 ceph-mon[30729]: 2.3 scrub ok 2024-09-11T03:48:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:07 smithi121 ceph-mon[30729]: 2.19 scrub starts 2024-09-11T03:48:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:07 smithi121 ceph-mon[30729]: 2.19 scrub ok 2024-09-11T03:48:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:07 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:48:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:07 smithi121 ceph-mon[30729]: 2.e scrub ok 2024-09-11T03:48:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:07 smithi121 ceph-mon[30729]: osdmap e669: 12 total, 12 up, 12 in 2024-09-11T03:48:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:07 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:48:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:07 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:48:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1657603008' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:48:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:07 smithi175 ceph-mon[30894]: osdmap e668: 12 total, 12 up, 12 in 2024-09-11T03:48:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:07 smithi175 ceph-mon[30894]: 2.3 scrub starts 2024-09-11T03:48:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:07 smithi175 ceph-mon[30894]: 2.3 scrub ok 2024-09-11T03:48:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:07 smithi175 ceph-mon[30894]: 2.19 scrub starts 2024-09-11T03:48:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:07 smithi175 ceph-mon[30894]: 2.19 scrub ok 2024-09-11T03:48:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:07 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:48:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:07 smithi175 ceph-mon[30894]: 2.e scrub ok 2024-09-11T03:48:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:07 smithi175 ceph-mon[30894]: osdmap e669: 12 total, 12 up, 12 in 2024-09-11T03:48:07.485 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:07.508 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:07.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:07.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:07.609 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:07.630 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:48:07.634 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:07.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:07.635 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:07.736 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:07.757 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:07.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:07.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:07.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:07.882 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:07.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:07.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:07.901 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:48:07.984 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:08.005 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:08.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:08.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:08.107 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:08.129 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:08.130 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:08.150 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:48:08.150 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:48:08.150 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:48:08.150 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:48:08.151 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.151 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:3937 2.12 2:494ba932:::smithi17557870-43:head [read 0~3008552,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e426)", 2024-09-11T03:48:08.151 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:03.919427+0000", 2024-09-11T03:48:08.151 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 244.229435209, 2024-09-11T03:48:08.151 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0072392760000000002, 2024-09-11T03:48:08.151 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.151 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.151 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:08.151 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:08.151 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:08.151 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:08.151 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 3937 2024-09-11T03:48:08.151 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.151 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.151 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.152 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.152 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:03.919427+0000", 2024-09-11T03:48:08.152 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.152 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.152 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.152 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.152 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:03.919427+0000", 2024-09-11T03:48:08.152 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.152 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.152 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.152 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.152 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:03.919427+0000", 2024-09-11T03:48:08.152 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8400000000000005e-07 2024-09-11T03:48:08.152 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.152 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.153 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.153 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:03.919430+0000", 2024-09-11T03:48:08.153 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0469999999999998e-06 2024-09-11T03:48:08.153 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.153 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.153 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.153 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:03.919437+0000", 2024-09-11T03:48:08.153 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9079999999999998e-06 2024-09-11T03:48:08.153 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.153 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.153 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.153 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:03.919454+0000", 2024-09-11T03:48:08.153 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7041000000000002e-05 2024-09-11T03:48:08.153 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.154 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.154 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.154 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:03.919529+0000", 2024-09-11T03:48:08.154 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5030000000000005e-05 2024-09-11T03:48:08.154 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.154 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.154 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:48:08.154 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:03.919533+0000", 2024-09-11T03:48:08.154 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6839999999999998e-06 2024-09-11T03:48:08.154 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.154 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.154 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.154 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:03.925629+0000", 2024-09-11T03:48:08.154 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0060959569999999999 2024-09-11T03:48:08.154 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.155 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.155 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.155 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:03.925702+0000", 2024-09-11T03:48:08.155 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3172999999999999e-05 2024-09-11T03:48:08.155 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.155 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.155 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.155 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:03.926666+0000", 2024-09-11T03:48:08.155 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00096395200000000002 2024-09-11T03:48:08.155 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.155 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.155 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.155 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.155 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.155 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4021 2.7 2:e77ce595:::smithi17557870-31:head [copy-from ver 473] snapc ac=[ac,ab,a9,a8,a6,a3,a1,9e,9d] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-11T03:48:08.156 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:13.490109+0000", 2024-09-11T03:48:08.156 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 234.65875248399999, 2024-09-11T03:48:08.156 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.02074401, 2024-09-11T03:48:08.156 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.156 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.156 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:08.156 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:08.156 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:08.156 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:08.156 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4021 2024-09-11T03:48:08.156 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.156 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.156 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.156 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.157 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490109+0000", 2024-09-11T03:48:08.157 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.157 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.157 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.157 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.157 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490109+0000", 2024-09-11T03:48:08.157 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.157 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.157 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.157 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.157 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490111+0000", 2024-09-11T03:48:08.157 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.438e-06 2024-09-11T03:48:08.157 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.157 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.158 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.158 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490118+0000", 2024-09-11T03:48:08.158 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8950000000000001e-06 2024-09-11T03:48:08.158 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.158 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.158 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.158 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490131+0000", 2024-09-11T03:48:08.158 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2889e-05 2024-09-11T03:48:08.158 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.158 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.158 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.158 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490138+0000", 2024-09-11T03:48:08.158 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3849999999999996e-06 2024-09-11T03:48:08.158 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.158 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.159 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.159 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490199+0000", 2024-09-11T03:48:08.159 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0822000000000001e-05 2024-09-11T03:48:08.159 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.159 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.159 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.159 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490297+0000", 2024-09-11T03:48:08.159 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7726999999999995e-05 2024-09-11T03:48:08.159 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.159 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.159 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:48:08.159 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.499178+0000", 2024-09-11T03:48:08.159 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0088812119999999994 2024-09-11T03:48:08.159 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.159 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.160 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:48:08.160 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.502683+0000", 2024-09-11T03:48:08.160 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035055559999999999 2024-09-11T03:48:08.160 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.160 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.160 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:08.160 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.507713+0000", 2024-09-11T03:48:08.160 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050300299999999996 2024-09-11T03:48:08.160 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.160 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.160 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:08.160 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.510764+0000", 2024-09-11T03:48:08.160 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030502469999999999 2024-09-11T03:48:08.160 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.160 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.161 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:48:08.161 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.510803+0000", 2024-09-11T03:48:08.161 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9629999999999998e-05 2024-09-11T03:48:08.161 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.161 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.161 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.161 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.510853+0000", 2024-09-11T03:48:08.161 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0179000000000002e-05 2024-09-11T03:48:08.161 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.161 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.161 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.161 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.161 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.161 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4022 2.7 2:e77ce595:::smithi17557870-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e435)", 2024-09-11T03:48:08.161 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:13.490154+0000", 2024-09-11T03:48:08.162 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 234.65870806999999, 2024-09-11T03:48:08.162 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.01199774, 2024-09-11T03:48:08.162 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.162 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.162 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:08.162 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:08.162 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:08.162 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:08.162 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4022 2024-09-11T03:48:08.162 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.162 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.162 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.162 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.162 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490154+0000", 2024-09-11T03:48:08.162 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.163 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.163 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.163 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.163 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490154+0000", 2024-09-11T03:48:08.163 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.163 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.163 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.163 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.163 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490154+0000", 2024-09-11T03:48:08.163 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3000000000000001e-07 2024-09-11T03:48:08.163 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.163 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.164 INFO:tasks.rados.rados.2.smithi175.stdout:2624: done (4 left) 2024-09-11T03:48:08.164 INFO:tasks.rados.rados.2.smithi175.stdout:2626: done (3 left) 2024-09-11T03:48:08.164 INFO:tasks.rados.rados.2.smithi175.stdout:2627: done (2 left) 2024-09-11T03:48:08.164 INFO:tasks.rados.rados.2.smithi175.stdout:2628: done (1 left) 2024-09-11T03:48:08.164 INFO:tasks.rados.rados.2.smithi175.stdout:2629: done (0 left) 2024-09-11T03:48:08.164 INFO:tasks.rados.rados.2.smithi175.stdout:2630: read oid 10 snap -1 2024-09-11T03:48:08.164 INFO:tasks.rados.rados.2.smithi175.stdout:2630: expect (ObjNum 736 snap 243 seq_num 736) 2024-09-11T03:48:08.165 INFO:tasks.rados.rados.2.smithi175.stdout:2631: copy_from oid 35 from oid 46 current snap is 287 2024-09-11T03:48:08.165 INFO:tasks.rados.rados.2.smithi175.stdout:2632: read oid 14 snap -1 2024-09-11T03:48:08.165 INFO:tasks.rados.rados.2.smithi175.stdout:2632: expect deleted 2024-09-11T03:48:08.165 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.165 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490157+0000", 2024-09-11T03:48:08.165 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2460000000000002e-06 2024-09-11T03:48:08.165 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.165 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.165 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.165 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490163+0000", 2024-09-11T03:48:08.165 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6189999999999996e-06 2024-09-11T03:48:08.166 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.166 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.166 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.166 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490169+0000", 2024-09-11T03:48:08.166 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9139999999999996e-06 2024-09-11T03:48:08.166 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.166 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.166 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.166 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490486+0000", 2024-09-11T03:48:08.166 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00031683599999999997 2024-09-11T03:48:08.166 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.166 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.166 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:48:08.166 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490520+0000", 2024-09-11T03:48:08.166 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4548999999999997e-05 2024-09-11T03:48:08.167 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.167 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.167 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.167 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.502009+0000", 2024-09-11T03:48:08.167 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011488437000000001 2024-09-11T03:48:08.167 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.167 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.167 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.167 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.502077+0000", 2024-09-11T03:48:08.167 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7890000000000002e-05 2024-09-11T03:48:08.167 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.167 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.167 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.167 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.502152+0000", 2024-09-11T03:48:08.168 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4819000000000003e-05 2024-09-11T03:48:08.168 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.168 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.168 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.168 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.168 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.168 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:4288 2.0 e462/417 2:0762776a:::smithi17557870-25:head v 462'115, mlcod=393'111)", 2024-09-11T03:48:08.168 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:39.987982+0000", 2024-09-11T03:48:08.168 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 208.16088023099999, 2024-09-11T03:48:08.168 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013639949, 2024-09-11T03:48:08.168 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.168 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.168 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:08.168 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.168 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.169 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.169 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.987982+0000", 2024-09-11T03:48:08.169 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.169 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.169 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.169 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.169 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.987982+0000", 2024-09-11T03:48:08.169 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.169 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.169 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.169 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.169 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.987982+0000", 2024-09-11T03:48:08.169 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.65e-07 2024-09-11T03:48:08.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.170 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.170 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.170 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.997342+0000", 2024-09-11T03:48:08.170 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0093600669999999997 2024-09-11T03:48:08.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.170 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.170 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.170 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.997361+0000", 2024-09-11T03:48:08.170 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8719000000000002e-05 2024-09-11T03:48:08.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.170 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.170 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.170 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.997370+0000", 2024-09-11T03:48:08.170 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4259999999999992e-06 2024-09-11T03:48:08.171 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.171 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.171 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.171 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.997462+0000", 2024-09-11T03:48:08.171 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2015000000000006e-05 2024-09-11T03:48:08.171 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.171 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.171 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.171 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.997496+0000", 2024-09-11T03:48:08.171 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3837999999999997e-05 2024-09-11T03:48:08.171 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.171 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.171 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:48:08.171 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:40.001567+0000", 2024-09-11T03:48:08.171 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0040706969999999999 2024-09-11T03:48:08.172 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.172 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.172 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.172 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:40.001622+0000", 2024-09-11T03:48:08.172 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4922e-05 2024-09-11T03:48:08.172 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.172 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.172 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.172 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.172 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.172 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4386 2.7 2:e77ce595:::smithi17557870-31:head [read 0~2319465,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e475)", 2024-09-11T03:48:08.172 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:52.940312+0000", 2024-09-11T03:48:08.172 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 195.20854951800001, 2024-09-11T03:48:08.173 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0072042, 2024-09-11T03:48:08.173 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.173 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.173 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:08.173 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:08.173 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:08.173 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:08.173 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4386 2024-09-11T03:48:08.173 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.173 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.173 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.173 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.173 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.940312+0000", 2024-09-11T03:48:08.174 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.174 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.174 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.174 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.174 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.940312+0000", 2024-09-11T03:48:08.174 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.174 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.174 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.174 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.174 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.940313+0000", 2024-09-11T03:48:08.174 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8800000000000002e-07 2024-09-11T03:48:08.174 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.174 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.174 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.175 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.940317+0000", 2024-09-11T03:48:08.175 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5129999999999998e-06 2024-09-11T03:48:08.175 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.175 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.175 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.175 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.940327+0000", 2024-09-11T03:48:08.175 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3090000000000001e-06 2024-09-11T03:48:08.175 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.175 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.175 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.175 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.940333+0000", 2024-09-11T03:48:08.175 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9030000000000001e-06 2024-09-11T03:48:08.175 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.175 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.176 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.176 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.940461+0000", 2024-09-11T03:48:08.176 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000127988 2024-09-11T03:48:08.176 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.176 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.176 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:48:08.176 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.940469+0000", 2024-09-11T03:48:08.176 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8339999999999999e-06 2024-09-11T03:48:08.176 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.176 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.176 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.176 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.946533+0000", 2024-09-11T03:48:08.176 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0060646520000000002 2024-09-11T03:48:08.176 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.177 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.177 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.177 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.946607+0000", 2024-09-11T03:48:08.177 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3424999999999996e-05 2024-09-11T03:48:08.177 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.177 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.177 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.177 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.947517+0000", 2024-09-11T03:48:08.177 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00090998800000000003 2024-09-11T03:48:08.177 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.177 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.177 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.177 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.178 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.178 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4387 2.7 2:e77ce595:::smithi17557870-31:head [sparse-read 0~1270889] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e475)", 2024-09-11T03:48:08.178 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:52.940354+0000", 2024-09-11T03:48:08.178 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 195.20850751099999, 2024-09-11T03:48:08.178 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0073993180000000002, 2024-09-11T03:48:08.178 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.178 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.178 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:08.178 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:08.178 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:08.178 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:08.178 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4387 2024-09-11T03:48:08.178 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.178 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.178 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.179 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.179 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.940354+0000", 2024-09-11T03:48:08.179 INFO:tasks.rados.rados.2.smithi175.stdout:2633: read oid 4 snap 283 2024-09-11T03:48:08.179 INFO:tasks.rados.rados.2.smithi175.stdout:2633: expect (ObjNum 807 snap 255 seq_num 807) 2024-09-11T03:48:08.179 INFO:tasks.rados.rados.2.smithi175.stdout:2634: copy_from oid 12 from oid 29 current snap is 287 2024-09-11T03:48:08.179 INFO:tasks.rados.rados.2.smithi175.stdout:2635: write (excl) oid 13 current snap is 287 2024-09-11T03:48:08.179 INFO:tasks.rados.rados.2.smithi175.stdout:2635: seq_num 907 ranges {443949=496850,1458815=584301,2641125=602824} 2024-09-11T03:48:08.180 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.180 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.180 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.180 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.180 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.940354+0000", 2024-09-11T03:48:08.180 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.180 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.180 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.180 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.180 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.940355+0000", 2024-09-11T03:48:08.180 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1500000000000005e-07 2024-09-11T03:48:08.180 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.181 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.181 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.181 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.940358+0000", 2024-09-11T03:48:08.181 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4170000000000001e-06 2024-09-11T03:48:08.181 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.181 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.181 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.181 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.940365+0000", 2024-09-11T03:48:08.181 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6590000000000001e-06 2024-09-11T03:48:08.181 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.181 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.181 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.181 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.940370+0000", 2024-09-11T03:48:08.181 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1580000000000004e-06 2024-09-11T03:48:08.182 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.182 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.182 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.182 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.940501+0000", 2024-09-11T03:48:08.182 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013108999999999999 2024-09-11T03:48:08.182 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.182 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.182 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:48:08.182 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.940507+0000", 2024-09-11T03:48:08.182 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8200000000000002e-06 2024-09-11T03:48:08.182 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.182 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.182 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.182 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.947559+0000", 2024-09-11T03:48:08.182 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0070515229999999996 2024-09-11T03:48:08.183 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.183 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.183 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.183 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.947648+0000", 2024-09-11T03:48:08.183 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.9265000000000007e-05 2024-09-11T03:48:08.183 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.183 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.183 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.183 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:52.947754+0000", 2024-09-11T03:48:08.183 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000105871 2024-09-11T03:48:08.183 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.183 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.183 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.183 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.184 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.184 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4893 2.7 2:e23d1762:::smithi17557870-21:head [append 0~0] snapc d7=[d7,d5,d4,d3,d2,d1,d0,ca] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-09-11T03:48:08.184 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:40.069714+0000", 2024-09-11T03:48:08.184 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 148.07914813599999, 2024-09-11T03:48:08.184 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0085313709999999994, 2024-09-11T03:48:08.184 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.184 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.184 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:08.184 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:08.184 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:08.184 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:08.184 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4893 2024-09-11T03:48:08.184 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.184 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.184 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.185 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.185 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069714+0000", 2024-09-11T03:48:08.185 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.185 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.185 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.185 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.185 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069714+0000", 2024-09-11T03:48:08.185 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.185 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.185 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.185 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.185 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069714+0000", 2024-09-11T03:48:08.185 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2300000000000002e-07 2024-09-11T03:48:08.185 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.186 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.186 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.186 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069716+0000", 2024-09-11T03:48:08.186 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1689999999999999e-06 2024-09-11T03:48:08.186 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.186 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.186 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.186 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069722+0000", 2024-09-11T03:48:08.186 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8470000000000001e-06 2024-09-11T03:48:08.186 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.186 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.186 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.186 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069727+0000", 2024-09-11T03:48:08.186 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9729999999999997e-06 2024-09-11T03:48:08.186 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.186 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.187 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.187 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.073410+0000", 2024-09-11T03:48:08.187 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0036824520000000001 2024-09-11T03:48:08.187 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.187 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.187 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.187 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.073465+0000", 2024-09-11T03:48:08.187 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5741e-05 2024-09-11T03:48:08.187 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.187 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.187 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:48:08.187 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.073548+0000", 2024-09-11T03:48:08.187 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2820999999999999e-05 2024-09-11T03:48:08.187 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.188 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.188 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:08.188 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.078085+0000", 2024-09-11T03:48:08.188 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.004536539 2024-09-11T03:48:08.188 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.188 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.188 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:08.188 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.078135+0000", 2024-09-11T03:48:08.188 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0291999999999999e-05 2024-09-11T03:48:08.188 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.188 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.188 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:48:08.188 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.078150+0000", 2024-09-11T03:48:08.189 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5315000000000002e-05 2024-09-11T03:48:08.189 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.189 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.189 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:48:08.189 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.078199+0000", 2024-09-11T03:48:08.189 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8303999999999997e-05 2024-09-11T03:48:08.189 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.189 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.189 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.189 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.078245+0000", 2024-09-11T03:48:08.189 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6495000000000001e-05 2024-09-11T03:48:08.189 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.189 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.189 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.190 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.190 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.190 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4895 2.7 2:e0eabcfb:::smithi17557870-9:head [read 0~3159848,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e522)", 2024-09-11T03:48:08.190 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:40.069760+0000", 2024-09-11T03:48:08.190 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 148.079101974, 2024-09-11T03:48:08.190 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0082682050000000007, 2024-09-11T03:48:08.190 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.190 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.190 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:08.190 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:08.190 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:08.190 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:08.190 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4895 2024-09-11T03:48:08.190 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.191 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.191 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.191 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.191 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069760+0000", 2024-09-11T03:48:08.191 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.191 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.191 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.191 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.191 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069760+0000", 2024-09-11T03:48:08.191 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.191 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.191 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.191 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.191 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069760+0000", 2024-09-11T03:48:08.192 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2599999999999998e-07 2024-09-11T03:48:08.192 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.192 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.192 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.192 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069763+0000", 2024-09-11T03:48:08.192 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3640000000000002e-06 2024-09-11T03:48:08.192 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.192 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.192 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.192 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069769+0000", 2024-09-11T03:48:08.192 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8429999999999998e-06 2024-09-11T03:48:08.192 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.192 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.193 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.193 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069774+0000", 2024-09-11T03:48:08.193 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4999999999999999e-06 2024-09-11T03:48:08.193 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.193 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.193 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.193 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.075834+0000", 2024-09-11T03:48:08.193 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0060603410000000003 2024-09-11T03:48:08.193 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.193 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.193 INFO:tasks.rados.rados.2.smithi175.stdout:2635: writing smithi17557870-13 from 443949 to 940799 tid 1 2024-09-11T03:48:08.194 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.194 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.075925+0000", 2024-09-11T03:48:08.194 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.0241999999999994e-05 2024-09-11T03:48:08.194 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.194 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.194 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.194 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.078028+0000", 2024-09-11T03:48:08.194 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0021034890000000001 2024-09-11T03:48:08.194 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.194 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.194 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.194 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.195 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.195 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4896 2.7 2:e0eabcfb:::smithi17557870-9:head [sparse-read 0~1248271] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e522)", 2024-09-11T03:48:08.195 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:40.069780+0000", 2024-09-11T03:48:08.195 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 148.079082239, 2024-09-11T03:48:08.195 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0086472939999999998, 2024-09-11T03:48:08.195 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.195 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.195 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:08.195 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:08.195 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:08.195 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:08.195 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4896 2024-09-11T03:48:08.195 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.195 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.195 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.196 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.196 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069780+0000", 2024-09-11T03:48:08.196 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.196 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.196 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.196 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.196 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069780+0000", 2024-09-11T03:48:08.196 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.196 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.196 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.196 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.196 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069780+0000", 2024-09-11T03:48:08.196 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6800000000000001e-07 2024-09-11T03:48:08.196 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.197 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.197 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.197 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069782+0000", 2024-09-11T03:48:08.197 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.72e-06 2024-09-11T03:48:08.197 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.197 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.197 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.197 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069787+0000", 2024-09-11T03:48:08.197 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7209999999999999e-06 2024-09-11T03:48:08.197 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.197 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.197 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.197 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069793+0000", 2024-09-11T03:48:08.197 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4949999999999999e-06 2024-09-11T03:48:08.197 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.198 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.198 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.198 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.078273+0000", 2024-09-11T03:48:08.198 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0084799130000000004 2024-09-11T03:48:08.198 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.198 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.198 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.198 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.078336+0000", 2024-09-11T03:48:08.198 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2854000000000005e-05 2024-09-11T03:48:08.198 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.198 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.198 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.198 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.078427+0000", 2024-09-11T03:48:08.199 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1223000000000004e-05 2024-09-11T03:48:08.199 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.199 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.199 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.199 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.199 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.199 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4907 2.7 2:e0ec5763:::smithi17557870-2:head [copy-from ver 761] snapc d7=[d7,d5,d4,d3,d2,d1,d0,ca] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-09-11T03:48:08.199 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:40.106137+0000", 2024-09-11T03:48:08.199 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 148.04272507499999, 2024-09-11T03:48:08.199 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021335674999999998, 2024-09-11T03:48:08.199 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.199 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.199 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:08.199 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:08.199 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:08.200 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:08.200 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4907 2024-09-11T03:48:08.200 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.200 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.200 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.200 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.200 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106137+0000", 2024-09-11T03:48:08.200 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.200 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.200 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.200 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.200 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106137+0000", 2024-09-11T03:48:08.200 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.200 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.200 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.201 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.201 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106139+0000", 2024-09-11T03:48:08.201 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3250000000000002e-06 2024-09-11T03:48:08.201 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.201 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.201 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.201 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106149+0000", 2024-09-11T03:48:08.201 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8079999999999999e-06 2024-09-11T03:48:08.201 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.201 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.201 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.201 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106171+0000", 2024-09-11T03:48:08.201 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2507999999999999e-05 2024-09-11T03:48:08.201 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.201 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.202 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.202 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106184+0000", 2024-09-11T03:48:08.202 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2224e-05 2024-09-11T03:48:08.202 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.202 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.202 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.202 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106253+0000", 2024-09-11T03:48:08.202 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9615000000000003e-05 2024-09-11T03:48:08.202 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.202 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.202 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.202 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106357+0000", 2024-09-11T03:48:08.202 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010383900000000001 2024-09-11T03:48:08.202 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.203 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.203 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:48:08.203 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.116615+0000", 2024-09-11T03:48:08.203 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010257779999999999 2024-09-11T03:48:08.203 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.203 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.203 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:48:08.203 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.120710+0000", 2024-09-11T03:48:08.203 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0040951579999999998 2024-09-11T03:48:08.203 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.203 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.203 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:08.203 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.127226+0000", 2024-09-11T03:48:08.203 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0065160610000000001 2024-09-11T03:48:08.203 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.204 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.204 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:08.204 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.127380+0000", 2024-09-11T03:48:08.204 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015357599999999999 2024-09-11T03:48:08.204 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.204 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.204 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:48:08.204 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.127423+0000", 2024-09-11T03:48:08.204 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3680999999999997e-05 2024-09-11T03:48:08.204 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.204 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.204 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.204 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.127473+0000", 2024-09-11T03:48:08.204 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9100000000000001e-05 2024-09-11T03:48:08.204 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.205 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.205 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.205 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.205 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.205 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4908 2.7 2:e0ec5763:::smithi17557870-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e522)", 2024-09-11T03:48:08.205 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:40.106211+0000", 2024-09-11T03:48:08.205 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 148.04265133199999, 2024-09-11T03:48:08.205 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013824826, 2024-09-11T03:48:08.205 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.205 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.205 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:08.205 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:08.205 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:08.205 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:08.206 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4908 2024-09-11T03:48:08.206 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.206 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.206 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.206 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.206 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106211+0000", 2024-09-11T03:48:08.206 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.206 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.206 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.206 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.206 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106211+0000", 2024-09-11T03:48:08.206 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.206 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.207 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.207 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.207 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106211+0000", 2024-09-11T03:48:08.207 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4200000000000001e-07 2024-09-11T03:48:08.207 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.207 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.207 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.207 INFO:tasks.rados.rados.2.smithi175.stdout:2635: writing smithi17557870-13 from 1458815 to 2043116 tid 2 2024-09-11T03:48:08.208 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106214+0000", 2024-09-11T03:48:08.208 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9490000000000001e-06 2024-09-11T03:48:08.208 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.208 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.208 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.208 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106221+0000", 2024-09-11T03:48:08.208 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3300000000000001e-06 2024-09-11T03:48:08.208 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.208 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.208 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.208 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106225+0000", 2024-09-11T03:48:08.208 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5700000000000001e-06 2024-09-11T03:48:08.209 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.209 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.209 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.209 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106521+0000", 2024-09-11T03:48:08.209 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00029598599999999998 2024-09-11T03:48:08.209 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.209 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.209 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:48:08.209 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106551+0000", 2024-09-11T03:48:08.209 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0165e-05 2024-09-11T03:48:08.209 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.209 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.209 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.210 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.119851+0000", 2024-09-11T03:48:08.210 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013300354 2024-09-11T03:48:08.210 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.210 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.210 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.210 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.119919+0000", 2024-09-11T03:48:08.210 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7569000000000003e-05 2024-09-11T03:48:08.210 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.210 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.210 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.210 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.120035+0000", 2024-09-11T03:48:08.210 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000116461 2024-09-11T03:48:08.210 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.210 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.211 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.211 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.211 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.211 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4936 2.7 2:e23d1762:::smithi17557870-21:head [read 0~3815288,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e526)", 2024-09-11T03:48:08.211 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:43.953865+0000", 2024-09-11T03:48:08.211 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 144.19499712999999, 2024-09-11T03:48:08.211 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0084249630000000006, 2024-09-11T03:48:08.211 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.211 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.211 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:08.211 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:08.211 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:08.211 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:08.212 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4936 2024-09-11T03:48:08.212 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.212 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.212 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.212 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.212 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953865+0000", 2024-09-11T03:48:08.212 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.212 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.212 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.212 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.212 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953865+0000", 2024-09-11T03:48:08.212 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.212 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.213 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.213 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.213 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953865+0000", 2024-09-11T03:48:08.213 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7000000000000005e-07 2024-09-11T03:48:08.213 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.213 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.213 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.213 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953869+0000", 2024-09-11T03:48:08.213 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6169999999999998e-06 2024-09-11T03:48:08.213 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.213 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.213 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.213 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953877+0000", 2024-09-11T03:48:08.213 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9310000000000008e-06 2024-09-11T03:48:08.214 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.214 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.214 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.214 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953882+0000", 2024-09-11T03:48:08.214 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8720000000000001e-06 2024-09-11T03:48:08.214 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.216 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.216 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.216 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953967+0000", 2024-09-11T03:48:08.216 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4759999999999995e-05 2024-09-11T03:48:08.216 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.216 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.216 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:48:08.216 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953972+0000", 2024-09-11T03:48:08.216 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2889999999999997e-06 2024-09-11T03:48:08.216 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.216 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.216 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.216 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.960197+0000", 2024-09-11T03:48:08.217 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0062247240000000001 2024-09-11T03:48:08.217 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.217 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.217 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.217 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.960267+0000", 2024-09-11T03:48:08.217 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0531999999999995e-05 2024-09-11T03:48:08.217 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.217 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.217 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.217 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.962290+0000", 2024-09-11T03:48:08.217 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002022668 2024-09-11T03:48:08.217 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.217 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.217 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.218 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.218 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.218 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4937 2.7 2:e23d1762:::smithi17557870-21:head [read 0~3815288,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e526)", 2024-09-11T03:48:08.218 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:43.953926+0000", 2024-09-11T03:48:08.218 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 144.194935681, 2024-09-11T03:48:08.218 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010177080999999999, 2024-09-11T03:48:08.218 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.218 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.218 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:08.218 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:08.218 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:08.218 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:08.218 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4937 2024-09-11T03:48:08.218 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.219 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.219 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.219 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.219 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953926+0000", 2024-09-11T03:48:08.219 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.219 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.219 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.219 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.219 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953926+0000", 2024-09-11T03:48:08.219 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.219 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.219 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.219 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.219 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953927+0000", 2024-09-11T03:48:08.219 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1699999999999997e-07 2024-09-11T03:48:08.220 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.220 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.220 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.220 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953933+0000", 2024-09-11T03:48:08.220 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5050000000000004e-06 2024-09-11T03:48:08.220 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.220 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.220 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.220 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953945+0000", 2024-09-11T03:48:08.220 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1202e-05 2024-09-11T03:48:08.220 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.220 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.220 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.220 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953951+0000", 2024-09-11T03:48:08.221 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.708e-06 2024-09-11T03:48:08.221 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.221 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.221 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.221 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953988+0000", 2024-09-11T03:48:08.221 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6356999999999998e-05 2024-09-11T03:48:08.221 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.221 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.221 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:48:08.221 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953992+0000", 2024-09-11T03:48:08.221 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5469999999999998e-06 2024-09-11T03:48:08.221 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.221 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.221 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.222 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.962348+0000", 2024-09-11T03:48:08.222 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0083553180000000005 2024-09-11T03:48:08.222 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.222 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.222 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.222 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.962519+0000", 2024-09-11T03:48:08.222 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000171817 2024-09-11T03:48:08.222 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.222 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.222 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.222 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.964103+0000", 2024-09-11T03:48:08.222 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00158391 2024-09-11T03:48:08.222 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.223 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.223 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.223 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.223 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.223 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4941 2.7 2:e0ec5763:::smithi17557870-2:d5 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e526)", 2024-09-11T03:48:08.223 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:43.957778+0000", 2024-09-11T03:48:08.223 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 144.19108380899999, 2024-09-11T03:48:08.223 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0066820239999999999, 2024-09-11T03:48:08.223 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.223 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.223 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "reached pg", 2024-09-11T03:48:08.223 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:08.224 INFO:tasks.rados.rados.2.smithi175.stdout:2635: writing smithi17557870-13 from 2641125 to 3243949 tid 3 2024-09-11T03:48:08.224 INFO:tasks.rados.rados.2.smithi175.stdout:2636: snap_create 2024-09-11T03:48:08.224 INFO:tasks.rados.rados.2.smithi175.stdout:2634: finishing copy_from racing read to smithi17557870-12 2024-09-11T03:48:08.224 INFO:tasks.rados.rados.2.smithi175.stdout:2630: expect (ObjNum 736 snap 243 seq_num 736) 2024-09-11T03:48:08.225 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:08.225 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:08.225 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:08.225 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4941 2024-09-11T03:48:08.225 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.226 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.226 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.226 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.226 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.957778+0000", 2024-09-11T03:48:08.226 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.226 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.226 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.226 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.226 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.957778+0000", 2024-09-11T03:48:08.226 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.226 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.226 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.226 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.227 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.957778+0000", 2024-09-11T03:48:08.227 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3000000000000002e-07 2024-09-11T03:48:08.227 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.227 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.227 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.227 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.957780+0000", 2024-09-11T03:48:08.227 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0150000000000002e-06 2024-09-11T03:48:08.227 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.227 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.227 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.227 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.957786+0000", 2024-09-11T03:48:08.227 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3140000000000003e-06 2024-09-11T03:48:08.227 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.228 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.228 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.228 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.957789+0000", 2024-09-11T03:48:08.228 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0249999999999998e-06 2024-09-11T03:48:08.228 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.228 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.228 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.228 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.964371+0000", 2024-09-11T03:48:08.228 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0065823970000000002 2024-09-11T03:48:08.228 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.228 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.228 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.228 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.964460+0000", 2024-09-11T03:48:08.228 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8943000000000005e-05 2024-09-11T03:48:08.229 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.229 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.229 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.229 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.229 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.229 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:5359 2.4 e573/432 2:220bb244:::smithi17557870-34:head v 573'441, mlcod=571'437)", 2024-09-11T03:48:08.229 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.991255+0000", 2024-09-11T03:48:08.229 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 97.157606623000007, 2024-09-11T03:48:08.229 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0082473760000000007, 2024-09-11T03:48:08.229 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.229 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.229 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:08.229 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.230 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.230 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.230 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.991255+0000", 2024-09-11T03:48:08.230 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.230 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.230 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.230 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.230 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.991255+0000", 2024-09-11T03:48:08.230 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.230 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.230 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.230 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.230 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.991256+0000", 2024-09-11T03:48:08.230 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6700000000000002e-07 2024-09-11T03:48:08.231 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.231 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.231 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.231 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.996427+0000", 2024-09-11T03:48:08.231 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0051710619999999997 2024-09-11T03:48:08.231 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.231 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.231 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.231 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.996448+0000", 2024-09-11T03:48:08.231 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0443000000000001e-05 2024-09-11T03:48:08.231 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.231 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.231 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.232 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.996456+0000", 2024-09-11T03:48:08.232 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0029999999999994e-06 2024-09-11T03:48:08.232 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.232 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.232 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.232 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.996544+0000", 2024-09-11T03:48:08.232 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8695000000000004e-05 2024-09-11T03:48:08.232 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.232 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.232 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.232 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.996575+0000", 2024-09-11T03:48:08.232 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0238000000000001e-05 2024-09-11T03:48:08.232 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.233 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.233 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:48:08.233 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.999449+0000", 2024-09-11T03:48:08.233 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0028740630000000001 2024-09-11T03:48:08.233 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.233 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.233 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.233 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.999503+0000", 2024-09-11T03:48:08.233 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3905e-05 2024-09-11T03:48:08.233 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.233 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.233 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.233 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.233 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.234 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:5650 2.1e e599/574 2:7b0c3397:::smithi17557870-14:head v 599'866, mlcod=597'864)", 2024-09-11T03:48:08.234 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:56.992686+0000", 2024-09-11T03:48:08.234 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 71.156176302000006, 2024-09-11T03:48:08.234 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010088503, 2024-09-11T03:48:08.234 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.234 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.234 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:08.234 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.234 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.234 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.234 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:56.992686+0000", 2024-09-11T03:48:08.234 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.234 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.234 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.234 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.234 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:56.992686+0000", 2024-09-11T03:48:08.235 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.235 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.235 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.235 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.235 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:56.992686+0000", 2024-09-11T03:48:08.235 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.17e-07 2024-09-11T03:48:08.235 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.235 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.235 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.235 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:57.000330+0000", 2024-09-11T03:48:08.235 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0076439680000000001 2024-09-11T03:48:08.235 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.235 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.236 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.236 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:57.000346+0000", 2024-09-11T03:48:08.236 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6376e-05 2024-09-11T03:48:08.236 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.236 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.236 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.236 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:57.000355+0000", 2024-09-11T03:48:08.236 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.9770000000000006e-06 2024-09-11T03:48:08.236 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.236 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.236 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.236 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:57.000428+0000", 2024-09-11T03:48:08.236 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2843999999999996e-05 2024-09-11T03:48:08.236 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.236 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.237 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.237 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:57.000457+0000", 2024-09-11T03:48:08.237 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8745999999999999e-05 2024-09-11T03:48:08.237 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.237 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.237 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:48:08.237 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:57.002715+0000", 2024-09-11T03:48:08.237 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022582549999999998 2024-09-11T03:48:08.237 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.237 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.237 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.237 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:57.002774+0000", 2024-09-11T03:48:08.237 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9120000000000003e-05 2024-09-11T03:48:08.237 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.237 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.238 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.238 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.238 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.238 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5777 2.12 2:4e5019e5:::smithi17557870-40:head [read 0~3714479,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e608)", 2024-09-11T03:48:08.238 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:06.021644+0000", 2024-09-11T03:48:08.238 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 62.127218171999999, 2024-09-11T03:48:08.238 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0086002600000000002, 2024-09-11T03:48:08.238 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.238 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.238 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:08.238 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:08.238 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:08.238 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:08.238 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5777 2024-09-11T03:48:08.238 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.239 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.239 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.239 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.239 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021644+0000", 2024-09-11T03:48:08.239 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.239 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.239 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.239 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.239 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021644+0000", 2024-09-11T03:48:08.239 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.239 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.239 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.239 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.239 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021644+0000", 2024-09-11T03:48:08.240 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.44e-07 2024-09-11T03:48:08.240 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.240 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.240 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.240 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021647+0000", 2024-09-11T03:48:08.240 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0580000000000002e-06 2024-09-11T03:48:08.240 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.240 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.240 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.240 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021654+0000", 2024-09-11T03:48:08.240 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4010000000000002e-06 2024-09-11T03:48:08.240 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.240 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.240 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.241 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021659+0000", 2024-09-11T03:48:08.241 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7910000000000004e-06 2024-09-11T03:48:08.241 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.241 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.241 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.241 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021819+0000", 2024-09-11T03:48:08.241 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000160777 2024-09-11T03:48:08.241 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.241 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.241 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:48:08.241 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021826+0000", 2024-09-11T03:48:08.241 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9709999999999998e-06 2024-09-11T03:48:08.241 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.241 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.242 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.242 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.028005+0000", 2024-09-11T03:48:08.242 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0061789710000000001 2024-09-11T03:48:08.242 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.242 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.242 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.242 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.028102+0000", 2024-09-11T03:48:08.242 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6484000000000007e-05 2024-09-11T03:48:08.242 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.242 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.242 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.242 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.030244+0000", 2024-09-11T03:48:08.242 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0021422630000000002 2024-09-11T03:48:08.243 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.243 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.243 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.243 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.243 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.243 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:5783 2.1e e608/574 2:7b0c3397:::smithi17557870-14:head v 608'869, mlcod=599'866)", 2024-09-11T03:48:08.243 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:06.035596+0000", 2024-09-11T03:48:08.243 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 62.113266293000002, 2024-09-11T03:48:08.243 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0091205959999999999, 2024-09-11T03:48:08.243 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.243 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.243 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:08.243 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.244 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.244 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.244 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.035596+0000", 2024-09-11T03:48:08.244 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.244 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.244 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.244 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.244 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.035596+0000", 2024-09-11T03:48:08.244 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.244 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.244 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.244 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.244 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.035597+0000", 2024-09-11T03:48:08.244 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.449e-06 2024-09-11T03:48:08.245 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.245 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.245 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.245 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.036120+0000", 2024-09-11T03:48:08.245 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00052300299999999997 2024-09-11T03:48:08.245 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.245 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.245 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.245 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.036136+0000", 2024-09-11T03:48:08.245 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5820000000000001e-05 2024-09-11T03:48:08.245 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.245 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.245 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.246 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.036144+0000", 2024-09-11T03:48:08.246 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9920000000000007e-06 2024-09-11T03:48:08.246 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.246 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.246 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.246 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.036203+0000", 2024-09-11T03:48:08.246 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8937999999999997e-05 2024-09-11T03:48:08.246 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.246 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.246 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.246 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.036221+0000", 2024-09-11T03:48:08.246 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8136999999999999e-05 2024-09-11T03:48:08.246 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.246 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.247 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:48:08.247 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044680+0000", 2024-09-11T03:48:08.247 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0084586750000000006 2024-09-11T03:48:08.247 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.247 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.247 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.247 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044716+0000", 2024-09-11T03:48:08.247 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6582000000000003e-05 2024-09-11T03:48:08.247 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.247 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.247 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.247 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.247 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.248 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6115 2.12 2:494ba932:::smithi17557870-43:head [write 652336~726335,stat] snapc 10c=[10c] ondisk+write+known_if_redirected+supports_pool_eio e636)", 2024-09-11T03:48:08.248 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:34.031920+0000", 2024-09-11T03:48:08.248 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 34.116942336999998, 2024-09-11T03:48:08.248 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068061859999999997, 2024-09-11T03:48:08.248 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.248 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.248 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:08.248 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:08.248 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:08.248 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:08.248 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6115 2024-09-11T03:48:08.248 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.248 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.249 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.249 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.249 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.031920+0000", 2024-09-11T03:48:08.249 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.249 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.249 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.249 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.249 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.031920+0000", 2024-09-11T03:48:08.249 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.249 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.249 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.249 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.249 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.031923+0000", 2024-09-11T03:48:08.250 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.2619999999999999e-06 2024-09-11T03:48:08.250 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.250 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.250 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.250 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.032695+0000", 2024-09-11T03:48:08.250 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00077226499999999995 2024-09-11T03:48:08.250 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.250 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.250 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.250 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.032739+0000", 2024-09-11T03:48:08.250 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4404999999999999e-05 2024-09-11T03:48:08.250 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.250 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.251 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.251 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.032766+0000", 2024-09-11T03:48:08.251 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6533000000000002e-05 2024-09-11T03:48:08.251 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.251 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.251 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.251 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.032868+0000", 2024-09-11T03:48:08.251 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000101524 2024-09-11T03:48:08.251 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.251 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.251 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.251 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.032988+0000", 2024-09-11T03:48:08.251 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000120489 2024-09-11T03:48:08.251 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.251 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.252 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:48:08.252 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.033204+0000", 2024-09-11T03:48:08.252 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00021645599999999999 2024-09-11T03:48:08.252 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.252 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.252 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:48:08.252 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.035579+0000", 2024-09-11T03:48:08.252 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023748290000000002 2024-09-11T03:48:08.252 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.252 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.252 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:08.252 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.036477+0000", 2024-09-11T03:48:08.252 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00089806100000000002 2024-09-11T03:48:08.252 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.252 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.253 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:08.253 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.038607+0000", 2024-09-11T03:48:08.253 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0021293359999999999 2024-09-11T03:48:08.253 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.253 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.253 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:48:08.253 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.038667+0000", 2024-09-11T03:48:08.253 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0331999999999998e-05 2024-09-11T03:48:08.253 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.253 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.253 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.253 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.038726+0000", 2024-09-11T03:48:08.253 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8693999999999997e-05 2024-09-11T03:48:08.254 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.254 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.254 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.254 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.254 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.254 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6214 2.12 2:4e5019e5:::smithi17557870-40:head [read 0~3966617,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e646)", 2024-09-11T03:48:08.254 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:43.987667+0000", 2024-09-11T03:48:08.254 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 24.161194403, 2024-09-11T03:48:08.254 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0091709329999999992, 2024-09-11T03:48:08.254 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:08.254 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:08.254 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:08.254 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:08.254 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:08.254 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:08.255 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6214 2024-09-11T03:48:08.255 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.255 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:08.255 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.255 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:08.255 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.987667+0000", 2024-09-11T03:48:08.255 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.255 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.255 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.255 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:08.255 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.987667+0000", 2024-09-11T03:48:08.255 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:08.255 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.255 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.255 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:08.255 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.987668+0000", 2024-09-11T03:48:08.256 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.08e-07 2024-09-11T03:48:08.256 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.256 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.256 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:08.256 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.987671+0000", 2024-09-11T03:48:08.256 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1310000000000002e-06 2024-09-11T03:48:08.256 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.256 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.256 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:08.256 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.987682+0000", 2024-09-11T03:48:08.256 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0541e-05 2024-09-11T03:48:08.256 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.256 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.256 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:08.257 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.987687+0000", 2024-09-11T03:48:08.257 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.186e-06 2024-09-11T03:48:08.257 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.257 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.257 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.257 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.987821+0000", 2024-09-11T03:48:08.257 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000134012 2024-09-11T03:48:08.257 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.257 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.257 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:48:08.257 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.987827+0000", 2024-09-11T03:48:08.257 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.818e-06 2024-09-11T03:48:08.257 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.257 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.258 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:08.258 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.993522+0000", 2024-09-11T03:48:08.258 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0056953500000000001 2024-09-11T03:48:08.258 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.258 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.258 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:08.258 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.993643+0000", 2024-09-11T03:48:08.258 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012071800000000001 2024-09-11T03:48:08.258 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:08.258 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:08.258 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:08.258 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.996838+0000", 2024-09-11T03:48:08.258 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0031957690000000002 2024-09-11T03:48:08.258 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.258 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.259 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.259 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:08.259 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:08.259 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:48:08.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:08.263 INFO:tasks.rados.rados.2.smithi175.stdout:2631: finishing copy_from racing read to smithi17557870-35 2024-09-11T03:48:08.263 INFO:tasks.rados.rados.2.smithi175.stdout:2634: finishing copy_from to smithi17557870-12 2024-09-11T03:48:08.263 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 930 (ObjNum 829 snap 263 seq_num 829) dirty exists 2024-09-11T03:48:08.263 INFO:tasks.rados.rados.2.smithi175.stdout:2633: expect (ObjNum 807 snap 255 seq_num 807) 2024-09-11T03:48:08.280 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:08.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:08.382 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:08 smithi084 ceph-mon[27254]: 2.1c scrub starts 2024-09-11T03:48:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:08 smithi084 ceph-mon[27254]: 2.1c scrub ok 2024-09-11T03:48:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:08 smithi084 ceph-mon[27254]: pgmap v1158: 67 pgs: 67 active+clean; 150 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.1 MiB/s wr, 24 op/s 2024-09-11T03:48:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:08 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:48:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:08 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:48:08.403 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:08.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:08.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:08.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:08 smithi121 ceph-mon[30729]: 2.1c scrub starts 2024-09-11T03:48:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:08 smithi121 ceph-mon[30729]: 2.1c scrub ok 2024-09-11T03:48:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:08 smithi121 ceph-mon[30729]: pgmap v1158: 67 pgs: 67 active+clean; 150 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.1 MiB/s wr, 24 op/s 2024-09-11T03:48:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:08 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:48:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:08 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:48:08.460 INFO:tasks.rados.rados.2.smithi175.stdout:2635: finishing write tid 1 to smithi17557870-13 2024-09-11T03:48:08.460 INFO:tasks.rados.rados.2.smithi175.stdout:2631: finishing copy_from to smithi17557870-35 2024-09-11T03:48:08.461 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 891 (ObjNum 876 snap 276 seq_num 876) dirty exists 2024-09-11T03:48:08.461 INFO:tasks.rados.rados.2.smithi175.stdout:2635: finishing write tid 2 to smithi17557870-13 2024-09-11T03:48:08.461 INFO:tasks.rados.rados.2.smithi175.stdout:2635: finishing write tid 3 to smithi17557870-13 2024-09-11T03:48:08.461 INFO:tasks.rados.rados.2.smithi175.stdout:2635: finishing write tid 4 to smithi17557870-13 2024-09-11T03:48:08.461 INFO:tasks.rados.rados.2.smithi175.stdout:2635: finishing write tid 5 to smithi17557870-13 2024-09-11T03:48:08.461 INFO:tasks.rados.rados.2.smithi175.stdout:2635: oid 13 updating version 0 to 943 2024-09-11T03:48:08.461 INFO:tasks.rados.rados.2.smithi175.stdout:2635: oid 13 version 943 is already newer than 940 2024-09-11T03:48:08.461 INFO:tasks.rados.rados.2.smithi175.stdout:2635: oid 13 version 943 is already newer than 941 2024-09-11T03:48:08.461 INFO:tasks.rados.rados.2.smithi175.stdout:2635: oid 13 version 943 is already newer than 942 2024-09-11T03:48:08.461 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 943 (ObjNum 907 snap 287 seq_num 907) dirty exists 2024-09-11T03:48:08.461 INFO:tasks.rados.rados.2.smithi175.stdout:2635: left oid 13 (ObjNum 907 snap 287 seq_num 907) 2024-09-11T03:48:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:08 smithi175 ceph-mon[30894]: 2.1c scrub starts 2024-09-11T03:48:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:08 smithi175 ceph-mon[30894]: 2.1c scrub ok 2024-09-11T03:48:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:08 smithi175 ceph-mon[30894]: pgmap v1158: 67 pgs: 67 active+clean; 150 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.1 MiB/s wr, 24 op/s 2024-09-11T03:48:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:08 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:48:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:08 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:48:08.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:08.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:08.531 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:08.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:08.633 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:08.636 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:48:08.637 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:48:08.653 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:48:08.654 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:48:08.654 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:08.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:08.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:08.671 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:48:08.671 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:48:08.687 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:48:08.687 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:48:08.703 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:48:08.703 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:48:08.721 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:48:08.721 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:48:08.737 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:48:08.737 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:48:08.754 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:48:08.754 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:48:08.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:08.770 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:48:08.770 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:48:08.777 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:08.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:08.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:08.785 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:48:08.787 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:48:08.787 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:48:08.803 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:48:08.803 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:48:08.820 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:48:08.820 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:48:08.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:08.899 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:08.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:08.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:09.001 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:09.022 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:09.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:09.023 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:09.124 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:09.137 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-11T03:48:09.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:09.146 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:09.147 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:09.158 INFO:tasks.rados.rados.2.smithi175.stdout:2630: done (6 left) 2024-09-11T03:48:09.158 INFO:tasks.rados.rados.2.smithi175.stdout:2631: done (5 left) 2024-09-11T03:48:09.158 INFO:tasks.rados.rados.2.smithi175.stdout:2632: done (4 left) 2024-09-11T03:48:09.158 INFO:tasks.rados.rados.2.smithi175.stdout:2633: done (3 left) 2024-09-11T03:48:09.158 INFO:tasks.rados.rados.2.smithi175.stdout:2634: done (2 left) 2024-09-11T03:48:09.158 INFO:tasks.rados.rados.2.smithi175.stdout:2635: done (1 left) 2024-09-11T03:48:09.159 INFO:tasks.rados.rados.2.smithi175.stdout:2636: done (0 left) 2024-09-11T03:48:09.159 INFO:tasks.rados.rados.2.smithi175.stdout:2637: rollback oid 4 current snap is 288 2024-09-11T03:48:09.159 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 4 to 284 2024-09-11T03:48:09.159 INFO:tasks.rados.rados.2.smithi175.stdout:2638: snap_create 2024-09-11T03:48:09.162 INFO:tasks.rados.rados.2.smithi175.stdout:2637: finishing rollback tid 0 to smithi17557870-4 2024-09-11T03:48:09.163 INFO:tasks.rados.rados.2.smithi175.stdout:2637: finishing rollback tid 1 to smithi17557870-4 2024-09-11T03:48:09.163 INFO:tasks.rados.rados.2.smithi175.stdout:2637: finishing rollback tid 2 to smithi17557870-4 2024-09-11T03:48:09.163 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 856 (ObjNum 807 snap 255 seq_num 807) dirty exists 2024-09-11T03:48:09.248 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:09.269 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:09.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:09.270 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:09.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:09.387 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:09 smithi084 ceph-mon[27254]: osdmap e670: 12 total, 12 up, 12 in 2024-09-11T03:48:09.387 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:09 smithi084 ceph-mon[27254]: 2.16 scrub starts 2024-09-11T03:48:09.387 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:09 smithi084 ceph-mon[27254]: 2.16 scrub ok 2024-09-11T03:48:09.387 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:09 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:48:09.387 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:09 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:48:09.387 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:09 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:48:09.387 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:09 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2221391972' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:09.388 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:48:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:09.393 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:09.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:09 smithi121 ceph-mon[30729]: osdmap e670: 12 total, 12 up, 12 in 2024-09-11T03:48:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:09 smithi121 ceph-mon[30729]: 2.16 scrub starts 2024-09-11T03:48:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:09 smithi121 ceph-mon[30729]: 2.16 scrub ok 2024-09-11T03:48:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:09 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:48:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:09 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:48:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:09 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:48:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:09 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2221391972' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:09 smithi175 ceph-mon[30894]: osdmap e670: 12 total, 12 up, 12 in 2024-09-11T03:48:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:09 smithi175 ceph-mon[30894]: 2.16 scrub starts 2024-09-11T03:48:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:09 smithi175 ceph-mon[30894]: 2.16 scrub ok 2024-09-11T03:48:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:09 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:48:09.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:09 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:48:09.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:09 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:48:09.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:09 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2221391972' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:09.494 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:09.515 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:09.516 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:09.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:09.517 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:09.618 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:09.639 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:09.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:09.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:09.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:09.762 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:09.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:09.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:09.840 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:48:09.840 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:48:09.864 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:09.885 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:09.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:09.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:09.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:10.008 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:10.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:10.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:10.110 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:10.133 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:10.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:10.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:10.159 INFO:tasks.rados.rados.2.smithi175.stdout:2637: done (1 left) 2024-09-11T03:48:10.159 INFO:tasks.rados.rados.2.smithi175.stdout:2638: done (0 left) 2024-09-11T03:48:10.160 INFO:tasks.rados.rados.2.smithi175.stdout:2639: delete oid 17 current snap is 289 2024-09-11T03:48:10.163 INFO:tasks.rados.rados.2.smithi175.stdout:2639: done (0 left) 2024-09-11T03:48:10.163 INFO:tasks.rados.rados.2.smithi175.stdout:2640: rollback oid 1 current snap is 289 2024-09-11T03:48:10.163 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 1 to 283 2024-09-11T03:48:10.163 INFO:tasks.rados.rados.2.smithi175.stdout:2641: write (excl) oid 40 current snap is 289 2024-09-11T03:48:10.163 INFO:tasks.rados.rados.2.smithi175.stdout:2641: seq_num 909 ranges {477563=647348,1859426=587970,2979618=297945} 2024-09-11T03:48:10.169 INFO:tasks.rados.rados.2.smithi175.stdout:2641: writing smithi17557870-40 from 477563 to 1124911 tid 1 2024-09-11T03:48:10.176 INFO:tasks.rados.rados.2.smithi175.stdout:2641: writing smithi17557870-40 from 1859426 to 2447396 tid 2 2024-09-11T03:48:10.181 INFO:tasks.rados.rados.2.smithi175.stdout:2641: writing smithi17557870-40 from 2979618 to 3277563 tid 3 2024-09-11T03:48:10.181 INFO:tasks.rados.rados.2.smithi175.stdout:2642: write oid 46 current snap is 289 2024-09-11T03:48:10.181 INFO:tasks.rados.rados.2.smithi175.stdout:2642: seq_num 910 ranges {494370=722597,1917908=447981,3007041=287329} 2024-09-11T03:48:10.188 INFO:tasks.rados.rados.2.smithi175.stdout:2642: writing smithi17557870-46 from 494370 to 1216967 tid 1 2024-09-11T03:48:10.194 INFO:tasks.rados.rados.2.smithi175.stdout:2642: writing smithi17557870-46 from 1917908 to 2365889 tid 2 2024-09-11T03:48:10.200 INFO:tasks.rados.rados.2.smithi175.stdout:2642: writing smithi17557870-46 from 3007041 to 3294370 tid 3 2024-09-11T03:48:10.200 INFO:tasks.rados.rados.2.smithi175.stdout:2643: snap_create 2024-09-11T03:48:10.200 INFO:tasks.rados.rados.2.smithi175.stdout:2640: finishing rollback tid 0 to smithi17557870-1 2024-09-11T03:48:10.200 INFO:tasks.rados.rados.2.smithi175.stdout:2640: finishing rollback tid 1 to smithi17557870-1 2024-09-11T03:48:10.200 INFO:tasks.rados.rados.2.smithi175.stdout:2640: finishing rollback tid 2 to smithi17557870-1 2024-09-11T03:48:10.200 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 947 (ObjNum 758 snap 246 seq_num 758) dirty exists 2024-09-11T03:48:10.200 INFO:tasks.rados.rados.2.smithi175.stdout:2641: finishing write tid 1 to smithi17557870-40 2024-09-11T03:48:10.201 INFO:tasks.rados.rados.2.smithi175.stdout:2641: finishing write tid 2 to smithi17557870-40 2024-09-11T03:48:10.201 INFO:tasks.rados.rados.2.smithi175.stdout:2641: finishing write tid 3 to smithi17557870-40 2024-09-11T03:48:10.201 INFO:tasks.rados.rados.2.smithi175.stdout:2641: finishing write tid 4 to smithi17557870-40 2024-09-11T03:48:10.201 INFO:tasks.rados.rados.2.smithi175.stdout:2641: finishing write tid 5 to smithi17557870-40 2024-09-11T03:48:10.201 INFO:tasks.rados.rados.2.smithi175.stdout:2641: oid 40 updating version 0 to 478 2024-09-11T03:48:10.201 INFO:tasks.rados.rados.2.smithi175.stdout:2641: oid 40 updating version 478 to 479 2024-09-11T03:48:10.201 INFO:tasks.rados.rados.2.smithi175.stdout:2641: oid 40 updating version 479 to 480 2024-09-11T03:48:10.201 INFO:tasks.rados.rados.2.smithi175.stdout:2641: oid 40 version 480 is already newer than 477 2024-09-11T03:48:10.201 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 480 (ObjNum 909 snap 289 seq_num 909) dirty exists 2024-09-11T03:48:10.201 INFO:tasks.rados.rados.2.smithi175.stdout:2641: left oid 40 (ObjNum 909 snap 289 seq_num 909) 2024-09-11T03:48:10.201 INFO:tasks.rados.rados.2.smithi175.stdout:2642: finishing write tid 1 to smithi17557870-46 2024-09-11T03:48:10.201 INFO:tasks.rados.rados.2.smithi175.stdout:2642: finishing write tid 2 to smithi17557870-46 2024-09-11T03:48:10.202 INFO:tasks.rados.rados.2.smithi175.stdout:2642: finishing write tid 3 to smithi17557870-46 2024-09-11T03:48:10.202 INFO:tasks.rados.rados.2.smithi175.stdout:2642: finishing write tid 4 to smithi17557870-46 2024-09-11T03:48:10.202 INFO:tasks.rados.rados.2.smithi175.stdout:2642: finishing write tid 5 to smithi17557870-46 2024-09-11T03:48:10.202 INFO:tasks.rados.rados.2.smithi175.stdout:2642: oid 46 updating version 0 to 895 2024-09-11T03:48:10.202 INFO:tasks.rados.rados.2.smithi175.stdout:2642: oid 46 version 895 is already newer than 893 2024-09-11T03:48:10.202 INFO:tasks.rados.rados.2.smithi175.stdout:2642: oid 46 version 895 is already newer than 894 2024-09-11T03:48:10.202 INFO:tasks.rados.rados.2.smithi175.stdout:2642: oid 46 version 895 is already newer than 892 2024-09-11T03:48:10.202 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 895 (ObjNum 910 snap 289 seq_num 910) dirty exists 2024-09-11T03:48:10.203 INFO:tasks.rados.rados.2.smithi175.stdout:2642: left oid 46 (ObjNum 910 snap 289 seq_num 910) 2024-09-11T03:48:10.235 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:10.257 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:10.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:10.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:10.359 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:10.381 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:10.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:10.382 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:10.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:10 smithi084 ceph-mon[27254]: pgmap v1161: 67 pgs: 67 active+clean; 159 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 4.9 MiB/s wr, 35 op/s 2024-09-11T03:48:10.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:10 smithi084 ceph-mon[27254]: osdmap e671: 12 total, 12 up, 12 in 2024-09-11T03:48:10.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:10 smithi121 ceph-mon[30729]: pgmap v1161: 67 pgs: 67 active+clean; 159 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 4.9 MiB/s wr, 35 op/s 2024-09-11T03:48:10.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:10 smithi121 ceph-mon[30729]: osdmap e671: 12 total, 12 up, 12 in 2024-09-11T03:48:10.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:10 smithi175 ceph-mon[30894]: pgmap v1161: 67 pgs: 67 active+clean; 159 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 4.9 MiB/s wr, 35 op/s 2024-09-11T03:48:10.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:10 smithi175 ceph-mon[30894]: osdmap e671: 12 total, 12 up, 12 in 2024-09-11T03:48:10.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:10.504 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:10.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:10.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:10.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:10.629 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:10.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:10.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:10.722 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-11T03:48:10.730 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:10.752 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:10.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:10.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:10.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:10.877 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:10.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:10.963 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:48:10.978 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:11.001 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:11.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:11.102 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:11.125 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:11.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:11.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:11.162 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:48:11.165 INFO:tasks.rados.rados.2.smithi175.stdout:2640: done (3 left) 2024-09-11T03:48:11.165 INFO:tasks.rados.rados.2.smithi175.stdout:2641: done (2 left) 2024-09-11T03:48:11.165 INFO:tasks.rados.rados.2.smithi175.stdout:2642: done (1 left) 2024-09-11T03:48:11.166 INFO:tasks.rados.rados.2.smithi175.stdout:2643: done (0 left) 2024-09-11T03:48:11.166 INFO:tasks.rados.rados.2.smithi175.stdout:2644: read oid 8 snap -1 2024-09-11T03:48:11.166 INFO:tasks.rados.rados.2.smithi175.stdout:2644: expect (ObjNum 886 snap 283 seq_num 886) 2024-09-11T03:48:11.166 INFO:tasks.rados.rados.2.smithi175.stdout:2645: read oid 43 snap -1 2024-09-11T03:48:11.166 INFO:tasks.rados.rados.2.smithi175.stdout:2645: expect deleted 2024-09-11T03:48:11.166 INFO:tasks.rados.rados.2.smithi175.stdout:2646: delete oid 42 current snap is 290 2024-09-11T03:48:11.169 INFO:tasks.rados.rados.2.smithi175.stdout:2645: done (2 left) 2024-09-11T03:48:11.169 INFO:tasks.rados.rados.2.smithi175.stdout:2646: done (1 left) 2024-09-11T03:48:11.169 INFO:tasks.rados.rados.2.smithi175.stdout:2647: write (excl) oid 15 current snap is 290 2024-09-11T03:48:11.169 INFO:tasks.rados.rados.2.smithi175.stdout:2647: seq_num 912 ranges {527984=473095,1634872=451650,2661887=634311} 2024-09-11T03:48:11.174 INFO:tasks.rados.rados.2.smithi175.stdout:2647: writing smithi17557870-15 from 527984 to 1001079 tid 1 2024-09-11T03:48:11.180 INFO:tasks.rados.rados.2.smithi175.stdout:2647: writing smithi17557870-15 from 1634872 to 2086522 tid 2 2024-09-11T03:48:11.187 INFO:tasks.rados.rados.2.smithi175.stdout:2647: writing smithi17557870-15 from 2661887 to 3296198 tid 3 2024-09-11T03:48:11.187 INFO:tasks.rados.rados.2.smithi175.stdout:2648: snap_remove snap 288 2024-09-11T03:48:11.227 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:11.250 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:11.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:11.251 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:11.264 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:11 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3655105193' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:48:11.264 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:11 smithi175 ceph-mon[30894]: osdmap e672: 12 total, 12 up, 12 in 2024-09-11T03:48:11.264 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:11 smithi175 ceph-mon[30894]: 2.4 scrub starts 2024-09-11T03:48:11.264 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:11 smithi175 ceph-mon[30894]: 2.4 scrub ok 2024-09-11T03:48:11.264 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:11 smithi175 ceph-mon[30894]: 1.6 scrub starts 2024-09-11T03:48:11.264 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:11 smithi175 ceph-mon[30894]: 1.6 scrub ok 2024-09-11T03:48:11.352 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:11.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:11.375 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:11.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:11 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3655105193' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:48:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:11 smithi121 ceph-mon[30729]: osdmap e672: 12 total, 12 up, 12 in 2024-09-11T03:48:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:11 smithi121 ceph-mon[30729]: 2.4 scrub starts 2024-09-11T03:48:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:11 smithi121 ceph-mon[30729]: 2.4 scrub ok 2024-09-11T03:48:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:11 smithi121 ceph-mon[30729]: 1.6 scrub starts 2024-09-11T03:48:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:11 smithi121 ceph-mon[30729]: 1.6 scrub ok 2024-09-11T03:48:11.471 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:11 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3655105193' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:48:11.471 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:11 smithi084 ceph-mon[27254]: osdmap e672: 12 total, 12 up, 12 in 2024-09-11T03:48:11.472 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:11 smithi084 ceph-mon[27254]: 2.4 scrub starts 2024-09-11T03:48:11.472 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:11 smithi084 ceph-mon[27254]: 2.4 scrub ok 2024-09-11T03:48:11.472 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:11 smithi084 ceph-mon[27254]: 1.6 scrub starts 2024-09-11T03:48:11.472 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:11 smithi084 ceph-mon[27254]: 1.6 scrub ok 2024-09-11T03:48:11.476 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:11.486 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:48:11.486 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:48:11.497 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:11.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:11.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:11.521 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:48:11.599 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:11.619 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:11.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:11.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:11.721 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:11.742 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:11.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:11.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:11.844 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:11.866 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:11.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:11.867 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:11.949 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:11.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:11.992 INFO:tasks.rados.rados.2.smithi175.stdout:2648: done (2 left) 2024-09-11T03:48:11.992 INFO:tasks.rados.rados.2.smithi175.stdout:2649: write oid 34 current snap is 290 2024-09-11T03:48:11.992 INFO:tasks.rados.rados.2.smithi175.stdout:2649: seq_num 913 ranges {544791=664697,1693354=595308,2689310=445583} 2024-09-11T03:48:11.994 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:11.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:11.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:11.998 INFO:tasks.rados.rados.2.smithi175.stdout:2649: writing smithi17557870-34 from 544791 to 1209488 tid 1 2024-09-11T03:48:12.005 INFO:tasks.rados.rados.2.smithi175.stdout:2649: writing smithi17557870-34 from 1693354 to 2288662 tid 2 2024-09-11T03:48:12.010 INFO:tasks.rados.rados.2.smithi175.stdout:2649: writing smithi17557870-34 from 2689310 to 3134893 tid 3 2024-09-11T03:48:12.010 INFO:tasks.rados.rados.2.smithi175.stdout:2650: snap_create 2024-09-11T03:48:12.010 INFO:tasks.rados.rados.2.smithi175.stdout:2644: expect (ObjNum 886 snap 283 seq_num 886) 2024-09-11T03:48:12.096 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:12.118 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:12.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:12.119 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:12.141 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-11T03:48:12.168 INFO:tasks.rados.rados.2.smithi175.stdout:2647: finishing write tid 1 to smithi17557870-15 2024-09-11T03:48:12.168 INFO:tasks.rados.rados.2.smithi175.stdout:2647: finishing write tid 2 to smithi17557870-15 2024-09-11T03:48:12.168 INFO:tasks.rados.rados.2.smithi175.stdout:2647: finishing write tid 3 to smithi17557870-15 2024-09-11T03:48:12.168 INFO:tasks.rados.rados.2.smithi175.stdout:2647: finishing write tid 4 to smithi17557870-15 2024-09-11T03:48:12.168 INFO:tasks.rados.rados.2.smithi175.stdout:2647: finishing write tid 5 to smithi17557870-15 2024-09-11T03:48:12.168 INFO:tasks.rados.rados.2.smithi175.stdout:2647: oid 15 updating version 0 to 888 2024-09-11T03:48:12.168 INFO:tasks.rados.rados.2.smithi175.stdout:2647: oid 15 version 888 is already newer than 887 2024-09-11T03:48:12.168 INFO:tasks.rados.rados.2.smithi175.stdout:2647: oid 15 version 888 is already newer than 886 2024-09-11T03:48:12.169 INFO:tasks.rados.rados.2.smithi175.stdout:2647: oid 15 updating version 888 to 889 2024-09-11T03:48:12.169 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 889 (ObjNum 912 snap 290 seq_num 912) dirty exists 2024-09-11T03:48:12.169 INFO:tasks.rados.rados.2.smithi175.stdout:2647: left oid 15 (ObjNum 912 snap 290 seq_num 912) 2024-09-11T03:48:12.169 INFO:tasks.rados.rados.2.smithi175.stdout:2649: finishing write tid 1 to smithi17557870-34 2024-09-11T03:48:12.169 INFO:tasks.rados.rados.2.smithi175.stdout:2649: finishing write tid 2 to smithi17557870-34 2024-09-11T03:48:12.169 INFO:tasks.rados.rados.2.smithi175.stdout:2649: finishing write tid 3 to smithi17557870-34 2024-09-11T03:48:12.169 INFO:tasks.rados.rados.2.smithi175.stdout:2649: finishing write tid 4 to smithi17557870-34 2024-09-11T03:48:12.169 INFO:tasks.rados.rados.2.smithi175.stdout:2649: finishing write tid 5 to smithi17557870-34 2024-09-11T03:48:12.169 INFO:tasks.rados.rados.2.smithi175.stdout:2649: oid 34 updating version 0 to 886 2024-09-11T03:48:12.169 INFO:tasks.rados.rados.2.smithi175.stdout:2649: oid 34 version 886 is already newer than 885 2024-09-11T03:48:12.169 INFO:tasks.rados.rados.2.smithi175.stdout:2649: oid 34 updating version 886 to 887 2024-09-11T03:48:12.169 INFO:tasks.rados.rados.2.smithi175.stdout:2649: oid 34 updating version 887 to 888 2024-09-11T03:48:12.169 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 888 (ObjNum 913 snap 290 seq_num 913) dirty exists 2024-09-11T03:48:12.169 INFO:tasks.rados.rados.2.smithi175.stdout:2649: left oid 34 (ObjNum 913 snap 290 seq_num 913) 2024-09-11T03:48:12.220 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:12.242 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:12.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:12.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:12.344 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:12.365 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:12.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:12.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:12 smithi084 ceph-mon[27254]: pgmap v1164: 67 pgs: 67 active+clean; 159 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 18 op/s 2024-09-11T03:48:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:12 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:12 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3655105193' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:48:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:12 smithi084 ceph-mon[27254]: osdmap e673: 12 total, 12 up, 12 in 2024-09-11T03:48:12.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:12 smithi084 ceph-mon[27254]: osdmap e674: 12 total, 12 up, 12 in 2024-09-11T03:48:12.416 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:48:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:12 smithi121 ceph-mon[30729]: pgmap v1164: 67 pgs: 67 active+clean; 159 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 18 op/s 2024-09-11T03:48:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:12 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:12 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3655105193' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:48:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:12 smithi121 ceph-mon[30729]: osdmap e673: 12 total, 12 up, 12 in 2024-09-11T03:48:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:12 smithi121 ceph-mon[30729]: osdmap e674: 12 total, 12 up, 12 in 2024-09-11T03:48:12.452 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:48:12.452 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:48:12.467 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:12.468 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:48:12.469 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:48:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:12 smithi175 ceph-mon[30894]: pgmap v1164: 67 pgs: 67 active+clean; 159 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 18 op/s 2024-09-11T03:48:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:12 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:12 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3655105193' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:48:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:12 smithi175 ceph-mon[30894]: osdmap e673: 12 total, 12 up, 12 in 2024-09-11T03:48:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:12 smithi175 ceph-mon[30894]: osdmap e674: 12 total, 12 up, 12 in 2024-09-11T03:48:12.485 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:48:12.485 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:48:12.489 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:12.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:12.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:12.502 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:48:12.502 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:48:12.519 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:48:12.519 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:48:12.535 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:48:12.536 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:48:12.552 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:48:12.552 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:48:12.568 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:48:12.569 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:48:12.585 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:48:12.585 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:48:12.592 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:12.602 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:48:12.602 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:48:12.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:12.614 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:12.614 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:12.618 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:48:12.618 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:48:12.634 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:48:12.634 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:48:12.715 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:12.741 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:12.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:12.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:12.843 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:12.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:12.866 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:12.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:12.915 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:48:12.916 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:48:12.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:48:12.916 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:48:12.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.916 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:640 2.a 2:5014e305:::smithi17557870-46:head [copy-from ver 23] snapc 11=[e] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-11T03:48:12.916 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:48:12.916 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 544.80399585299995, 2024-09-11T03:48:12.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019221736, 2024-09-11T03:48:12.916 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:12.916 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:12.917 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:12.917 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:12.917 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:12.917 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:12.917 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 640 2024-09-11T03:48:12.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.917 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:12.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:12.918 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:48:12.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:12.918 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110293+0000", 2024-09-11T03:48:12.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:12.918 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110295+0000", 2024-09-11T03:48:12.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0609999999999998e-06 2024-09-11T03:48:12.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.919 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:12.919 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110305+0000", 2024-09-11T03:48:12.919 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4169999999999998e-06 2024-09-11T03:48:12.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.919 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:12.919 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110321+0000", 2024-09-11T03:48:12.919 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6288000000000001e-05 2024-09-11T03:48:12.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.920 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:12.920 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110328+0000", 2024-09-11T03:48:12.920 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3950000000000003e-06 2024-09-11T03:48:12.920 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.920 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.920 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.920 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110396+0000", 2024-09-11T03:48:12.920 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7164999999999998e-05 2024-09-11T03:48:12.920 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:12.921 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.110507+0000", 2024-09-11T03:48:12.921 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000111217 2024-09-11T03:48:12.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:48:12.921 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.118213+0000", 2024-09-11T03:48:12.921 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0077066249999999999 2024-09-11T03:48:12.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:48:12.921 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125103+0000", 2024-09-11T03:48:12.921 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068894359999999997 2024-09-11T03:48:12.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:12.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.125459+0000", 2024-09-11T03:48:12.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000356146 2024-09-11T03:48:12.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:12.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129369+0000", 2024-09-11T03:48:12.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039099249999999999 2024-09-11T03:48:12.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:48:12.923 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129428+0000", 2024-09-11T03:48:12.923 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9113000000000001e-05 2024-09-11T03:48:12.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.923 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.923 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:12.923 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:08.129515+0000", 2024-09-11T03:48:12.924 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6947999999999995e-05 2024-09-11T03:48:12.924 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.924 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:12.924 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.924 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.924 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.924 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:752 2.c 2:3c007163:::smithi17557870-35:head [copy-from ver 39] snapc 14=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-11T03:48:12.924 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:48:12.924 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 537.90527317999999, 2024-09-11T03:48:12.924 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017596783000000001, 2024-09-11T03:48:12.924 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:12.925 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:12.925 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:12.925 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:12.925 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:12.925 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:12.925 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 752 2024-09-11T03:48:12.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.925 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:12.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.926 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:12.926 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:48:12.926 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.926 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.926 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:12.926 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009016+0000", 2024-09-11T03:48:12.926 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.926 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.926 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:12.926 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009018+0000", 2024-09-11T03:48:12.927 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.48e-06 2024-09-11T03:48:12.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.927 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:12.927 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009029+0000", 2024-09-11T03:48:12.927 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0377000000000001e-05 2024-09-11T03:48:12.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.927 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:12.927 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009050+0000", 2024-09-11T03:48:12.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0849e-05 2024-09-11T03:48:12.928 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.928 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.928 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:12.928 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009058+0000", 2024-09-11T03:48:12.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8019999999999998e-06 2024-09-11T03:48:12.928 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.928 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.928 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.928 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009125+0000", 2024-09-11T03:48:12.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6388000000000002e-05 2024-09-11T03:48:12.928 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.929 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.929 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:48:12.929 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.009130+0000", 2024-09-11T03:48:12.929 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0900000000000004e-06 2024-09-11T03:48:12.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.929 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.929 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.929 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010518+0000", 2024-09-11T03:48:12.929 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013880030000000001 2024-09-11T03:48:12.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.929 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.929 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:12.930 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.010582+0000", 2024-09-11T03:48:12.930 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.368e-05 2024-09-11T03:48:12.930 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.930 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.930 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:48:12.930 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.015350+0000", 2024-09-11T03:48:12.930 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0047682870000000004 2024-09-11T03:48:12.930 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.931 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:48:12.931 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.019005+0000", 2024-09-11T03:48:12.931 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0036549040000000001 2024-09-11T03:48:12.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.931 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:12.931 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.024096+0000", 2024-09-11T03:48:12.931 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050917439999999996 2024-09-11T03:48:12.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.931 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:12.931 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026521+0000", 2024-09-11T03:48:12.931 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024247399999999999 2024-09-11T03:48:12.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.932 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:48:12.932 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026559+0000", 2024-09-11T03:48:12.932 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8232999999999998e-05 2024-09-11T03:48:12.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.933 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:12.933 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:15.026613+0000", 2024-09-11T03:48:12.933 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3205999999999998e-05 2024-09-11T03:48:12.933 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.933 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:12.933 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.934 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.934 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1175 2.c 2:32ade24e:::smithi17557870-45:head [copy-from ver 130] snapc 2e=[2e,2c,2b,28,26] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:48:12.934 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:48:12.934 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:48:12.935 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-11T03:48:12.935 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T03:48:12.935 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T03:48:12.935 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-11T03:48:12.937 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:48:12.937 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 490.81214940400002, 2024-09-11T03:48:12.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015807878000000001, 2024-09-11T03:48:12.938 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:12.938 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:12.938 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:12.938 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:12.938 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:12.938 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:12.938 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1175 2024-09-11T03:48:12.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.938 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:12.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.939 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:12.939 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:48:12.939 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.939 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.939 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:12.939 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102140+0000", 2024-09-11T03:48:12.939 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.939 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.939 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:12.940 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102142+0000", 2024-09-11T03:48:12.940 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3369999999999998e-06 2024-09-11T03:48:12.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.940 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:12.940 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102152+0000", 2024-09-11T03:48:12.940 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8400000000000007e-06 2024-09-11T03:48:12.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.940 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:12.940 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102172+0000", 2024-09-11T03:48:12.940 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0537e-05 2024-09-11T03:48:12.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:12.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102183+0000", 2024-09-11T03:48:12.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0423e-05 2024-09-11T03:48:12.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102253+0000", 2024-09-11T03:48:12.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0528e-05 2024-09-11T03:48:12.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:48:12.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.102259+0000", 2024-09-11T03:48:12.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1589999999999998e-06 2024-09-11T03:48:12.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.943 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.943 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103477+0000", 2024-09-11T03:48:12.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012174550000000001 2024-09-11T03:48:12.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.943 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:12.943 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.103545+0000", 2024-09-11T03:48:12.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7595999999999995e-05 2024-09-11T03:48:12.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:48:12.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.110323+0000", 2024-09-11T03:48:12.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0067785730000000004 2024-09-11T03:48:12.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:48:12.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.113411+0000", 2024-09-11T03:48:12.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030882769999999999 2024-09-11T03:48:12.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:12.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117742+0000", 2024-09-11T03:48:12.945 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0043301920000000001 2024-09-11T03:48:12.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:12.945 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117835+0000", 2024-09-11T03:48:12.945 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3848e-05 2024-09-11T03:48:12.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:48:12.945 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117916+0000", 2024-09-11T03:48:12.945 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0505000000000003e-05 2024-09-11T03:48:12.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.946 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:12.946 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:02.117948+0000", 2024-09-11T03:48:12.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1608e-05 2024-09-11T03:48:12.946 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.946 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:12.946 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.946 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2972 2.c 2:32ade24e:::smithi17557870-45:head [copy-from ver 171] snapc 80=[80,7b,7a,79,77,75] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-11T03:48:12.947 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:48:12.947 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 327.84019510899998, 2024-09-11T03:48:12.947 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021036193000000002, 2024-09-11T03:48:12.947 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:12.947 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:12.947 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:12.947 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:12.947 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:12.947 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:12.947 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2972 2024-09-11T03:48:12.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.947 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:12.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:12.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:48:12.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:12.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:48:12.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:12.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074096+0000", 2024-09-11T03:48:12.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.685e-06 2024-09-11T03:48:12.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:12.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074104+0000", 2024-09-11T03:48:12.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9950000000000005e-06 2024-09-11T03:48:12.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:12.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074123+0000", 2024-09-11T03:48:12.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9587999999999999e-05 2024-09-11T03:48:12.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.950 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:12.950 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074132+0000", 2024-09-11T03:48:12.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1579999999999994e-06 2024-09-11T03:48:12.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.950 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.950 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074204+0000", 2024-09-11T03:48:12.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2072999999999999e-05 2024-09-11T03:48:12.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:12.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074302+0000", 2024-09-11T03:48:12.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7706000000000003e-05 2024-09-11T03:48:12.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:48:12.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.087397+0000", 2024-09-11T03:48:12.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013095292999999999 2024-09-11T03:48:12.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:48:12.952 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090766+0000", 2024-09-11T03:48:12.952 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0033682109999999999 2024-09-11T03:48:12.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.952 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:12.952 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.094977+0000", 2024-09-11T03:48:12.952 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0042108759999999997 2024-09-11T03:48:12.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.952 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:12.952 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095061+0000", 2024-09-11T03:48:12.952 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4827999999999995e-05 2024-09-11T03:48:12.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.953 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:48:12.953 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095099+0000", 2024-09-11T03:48:12.953 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7546999999999998e-05 2024-09-11T03:48:12.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.953 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.953 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:12.953 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095130+0000", 2024-09-11T03:48:12.953 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1232999999999997e-05 2024-09-11T03:48:12.953 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.953 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:12.953 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.953 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.953 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2973 2.c 2:32ade24e:::smithi17557870-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e348)", 2024-09-11T03:48:12.954 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:48:12.954 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 327.84012999100003, 2024-09-11T03:48:12.954 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016062098, 2024-09-11T03:48:12.954 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:12.954 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:12.954 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:12.954 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:12.954 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:12.954 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:12.954 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2973 2024-09-11T03:48:12.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.954 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:12.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.955 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:12.955 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:48:12.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.955 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:12.955 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:48:12.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.955 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:12.955 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074160+0000", 2024-09-11T03:48:12.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.46e-07 2024-09-11T03:48:12.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.956 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:12.956 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074162+0000", 2024-09-11T03:48:12.956 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5220000000000002e-06 2024-09-11T03:48:12.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.956 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:12.956 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074178+0000", 2024-09-11T03:48:12.956 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.609e-05 2024-09-11T03:48:12.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:12.957 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074183+0000", 2024-09-11T03:48:12.957 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8280000000000002e-06 2024-09-11T03:48:12.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.957 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074428+0000", 2024-09-11T03:48:12.957 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00024529599999999999 2024-09-11T03:48:12.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:48:12.957 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074469+0000", 2024-09-11T03:48:12.957 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0463000000000002e-05 2024-09-11T03:48:12.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.958 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.958 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090093+0000", 2024-09-11T03:48:12.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015624449 2024-09-11T03:48:12.958 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.958 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.958 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:12.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090155+0000", 2024-09-11T03:48:12.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1618999999999994e-05 2024-09-11T03:48:12.958 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.958 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.958 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:12.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090221+0000", 2024-09-11T03:48:12.959 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6384999999999995e-05 2024-09-11T03:48:12.959 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.959 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:12.959 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.959 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5348 2.1b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e571)", 2024-09-11T03:48:12.959 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:28.996464+0000", 2024-09-11T03:48:12.959 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 103.917824575, 2024-09-11T03:48:12.959 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.973168225, 2024-09-11T03:48:12.959 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:12.960 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:12.960 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "reached pg", 2024-09-11T03:48:12.960 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:12.960 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:12.960 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:12.960 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5348 2024-09-11T03:48:12.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.960 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:12.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.960 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:12.960 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996464+0000", 2024-09-11T03:48:12.960 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.961 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:12.961 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996464+0000", 2024-09-11T03:48:12.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.961 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:12.961 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996465+0000", 2024-09-11T03:48:12.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3500000000000002e-07 2024-09-11T03:48:12.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.962 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:12.962 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996467+0000", 2024-09-11T03:48:12.962 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.576e-06 2024-09-11T03:48:12.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.962 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:12.962 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996475+0000", 2024-09-11T03:48:12.962 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1350000000000003e-06 2024-09-11T03:48:12.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.963 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:12.963 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996478+0000", 2024-09-11T03:48:12.963 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5240000000000001e-06 2024-09-11T03:48:12.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.963 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.963 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996545+0000", 2024-09-11T03:48:12.963 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7379000000000007e-05 2024-09-11T03:48:12.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.963 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:48:12.963 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996557+0000", 2024-09-11T03:48:12.963 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2019e-05 2024-09-11T03:48:12.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.964 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.964 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.964 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969589+0000", 2024-09-11T03:48:12.964 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97303155699999999 2024-09-11T03:48:12.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.964 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.964 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:12.964 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969633+0000", 2024-09-11T03:48:12.964 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3699999999999998e-05 2024-09-11T03:48:12.964 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.964 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:12.964 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.965 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.965 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5349 2.1b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e571)", 2024-09-11T03:48:12.965 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:48:12.965 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 103.917801447, 2024-09-11T03:48:12.965 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97318775899999999, 2024-09-11T03:48:12.965 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:12.965 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:12.965 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "reached pg", 2024-09-11T03:48:12.965 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:12.965 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:12.965 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:12.965 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5349 2024-09-11T03:48:12.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.966 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:12.966 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.966 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:12.966 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:48:12.966 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.966 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.966 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.966 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:12.966 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:48:12.966 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.966 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.967 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:12.967 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:48:12.967 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7300000000000002e-07 2024-09-11T03:48:12.967 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.967 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:12.967 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996490+0000", 2024-09-11T03:48:12.967 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8470000000000001e-06 2024-09-11T03:48:12.967 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.967 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:12.968 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996494+0000", 2024-09-11T03:48:12.968 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8419999999999998e-06 2024-09-11T03:48:12.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.968 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.968 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:12.968 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996497+0000", 2024-09-11T03:48:12.968 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8820000000000001e-06 2024-09-11T03:48:12.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.968 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.968 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.968 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996602+0000", 2024-09-11T03:48:12.968 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010565699999999999 2024-09-11T03:48:12.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.968 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.969 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:48:12.969 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996613+0000", 2024-09-11T03:48:12.969 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0359e-05 2024-09-11T03:48:12.969 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.969 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.969 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.969 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969667+0000", 2024-09-11T03:48:12.969 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97305483599999998 2024-09-11T03:48:12.970 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.970 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.970 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:12.970 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969675+0000", 2024-09-11T03:48:12.970 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9629999999999998e-06 2024-09-11T03:48:12.970 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.970 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:12.970 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.970 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.970 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.970 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5350 2.1b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e571)", 2024-09-11T03:48:12.971 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:48:12.971 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 103.91776629100001, 2024-09-11T03:48:12.971 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.973193639, 2024-09-11T03:48:12.971 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:12.971 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:12.971 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "reached pg", 2024-09-11T03:48:12.971 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:12.971 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:12.971 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:12.971 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5350 2024-09-11T03:48:12.971 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.971 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:12.972 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.972 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:12.972 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:48:12.972 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.972 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.972 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.973 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:12.973 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:48:12.973 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.973 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.973 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.973 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:12.973 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:48:12.973 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.27e-07 2024-09-11T03:48:12.973 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.973 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.973 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:12.973 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996525+0000", 2024-09-11T03:48:12.974 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.719e-06 2024-09-11T03:48:12.974 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.974 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.974 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:12.974 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996529+0000", 2024-09-11T03:48:12.974 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2690000000000001e-06 2024-09-11T03:48:12.974 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.974 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.974 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:12.974 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996532+0000", 2024-09-11T03:48:12.974 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4139999999999998e-06 2024-09-11T03:48:12.974 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.974 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.974 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.975 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996643+0000", 2024-09-11T03:48:12.975 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011041599999999999 2024-09-11T03:48:12.975 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.975 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.975 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:48:12.975 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996651+0000", 2024-09-11T03:48:12.975 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4309999999999994e-06 2024-09-11T03:48:12.975 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.975 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.975 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.975 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969706+0000", 2024-09-11T03:48:12.975 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97305510699999997 2024-09-11T03:48:12.975 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.976 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.976 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:12.976 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969716+0000", 2024-09-11T03:48:12.976 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9559999999999992e-06 2024-09-11T03:48:12.976 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.976 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:12.976 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.976 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.976 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.976 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5348 2.1b 2:dde92fc6:::smithi17557870-41:head [sparse-read 0~1317959,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:48:12.976 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973173+0000", 2024-09-11T03:48:12.977 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 101.94111611300001, 2024-09-11T03:48:12.977 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019341553000000001, 2024-09-11T03:48:12.977 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:12.977 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:12.977 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:12.977 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:12.977 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:12.977 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:12.977 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5348 2024-09-11T03:48:12.977 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.977 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:12.977 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.978 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:12.978 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973173+0000", 2024-09-11T03:48:12.978 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.978 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:12.978 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973173+0000", 2024-09-11T03:48:12.978 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.978 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:12.978 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973175+0000", 2024-09-11T03:48:12.978 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.074e-06 2024-09-11T03:48:12.979 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.979 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.979 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:12.979 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973183+0000", 2024-09-11T03:48:12.979 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3010000000000007e-06 2024-09-11T03:48:12.979 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.979 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.979 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:12.979 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973199+0000", 2024-09-11T03:48:12.979 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5279000000000001e-05 2024-09-11T03:48:12.979 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.979 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.980 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:12.980 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973214+0000", 2024-09-11T03:48:12.980 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5891000000000001e-05 2024-09-11T03:48:12.980 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.980 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.980 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.980 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973747+0000", 2024-09-11T03:48:12.980 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00053236600000000005 2024-09-11T03:48:12.980 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.980 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.981 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:48:12.981 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973760+0000", 2024-09-11T03:48:12.981 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2976000000000001e-05 2024-09-11T03:48:12.981 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.981 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.981 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.981 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.990782+0000", 2024-09-11T03:48:12.981 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017022371000000001 2024-09-11T03:48:12.981 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.981 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.981 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:12.981 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.990945+0000", 2024-09-11T03:48:12.981 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016245699999999999 2024-09-11T03:48:12.982 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.982 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.982 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:12.982 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.992514+0000", 2024-09-11T03:48:12.982 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001569838 2024-09-11T03:48:12.982 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.982 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:12.982 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.982 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.982 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.982 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5349 2.1b 2:dde92fc6:::smithi17557870-41:head [read 0~2235463,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:48:12.982 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:48:12.982 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 101.941051401, 2024-09-11T03:48:12.982 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020826846, 2024-09-11T03:48:12.983 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:12.983 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:12.983 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:12.983 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:12.983 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:12.983 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:12.983 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5349 2024-09-11T03:48:12.983 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.983 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:12.983 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.983 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:12.983 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:48:12.983 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.984 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.984 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.984 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:12.984 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:48:12.984 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.984 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.984 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.984 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:12.984 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:48:12.984 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2099999999999996e-07 2024-09-11T03:48:12.984 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.984 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.984 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:12.985 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973242+0000", 2024-09-11T03:48:12.985 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0489999999999997e-06 2024-09-11T03:48:12.985 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.985 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.985 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:12.985 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973250+0000", 2024-09-11T03:48:12.985 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0700000000000007e-06 2024-09-11T03:48:12.985 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.985 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.985 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:12.985 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973256+0000", 2024-09-11T03:48:12.985 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3979999999999998e-06 2024-09-11T03:48:12.985 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.986 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.986 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.986 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973794+0000", 2024-09-11T03:48:12.986 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00053782700000000005 2024-09-11T03:48:12.986 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.986 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.986 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:48:12.986 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973801+0000", 2024-09-11T03:48:12.986 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5299999999999999e-06 2024-09-11T03:48:12.986 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.986 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.986 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.986 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.992813+0000", 2024-09-11T03:48:12.987 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019011917999999999 2024-09-11T03:48:12.987 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.987 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.987 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:12.987 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.992923+0000", 2024-09-11T03:48:12.987 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000109874 2024-09-11T03:48:12.987 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.987 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.987 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:12.987 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994064+0000", 2024-09-11T03:48:12.988 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001141459 2024-09-11T03:48:12.988 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.988 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:12.988 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.988 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.988 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.988 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5350 2.1b 2:dde92fc6:::smithi17557870-41:head [sparse-read 0~1317959] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:48:12.988 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973264+0000", 2024-09-11T03:48:12.988 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 101.94102467899999, 2024-09-11T03:48:12.988 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021144998000000002, 2024-09-11T03:48:12.988 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:12.988 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:12.989 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:12.989 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:12.989 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:12.989 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:12.989 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5350 2024-09-11T03:48:12.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.989 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:12.989 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.989 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:12.989 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973264+0000", 2024-09-11T03:48:12.989 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.990 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.990 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.990 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:12.990 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973264+0000", 2024-09-11T03:48:12.990 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.990 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.990 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.990 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:12.991 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973265+0000", 2024-09-11T03:48:12.991 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5899999999999996e-07 2024-09-11T03:48:12.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.991 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:12.991 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973268+0000", 2024-09-11T03:48:12.991 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6970000000000002e-06 2024-09-11T03:48:12.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.991 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:12.991 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973274+0000", 2024-09-11T03:48:12.991 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6390000000000003e-06 2024-09-11T03:48:12.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.992 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:12.992 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973280+0000", 2024-09-11T03:48:12.992 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.451e-06 2024-09-11T03:48:12.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.992 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.992 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.992 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973888+0000", 2024-09-11T03:48:12.992 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00060798999999999998 2024-09-11T03:48:12.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.992 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.992 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:48:12.992 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973895+0000", 2024-09-11T03:48:12.993 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6199999999999999e-06 2024-09-11T03:48:12.993 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.993 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.993 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.993 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994115+0000", 2024-09-11T03:48:12.993 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020219487000000001 2024-09-11T03:48:12.993 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.993 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.993 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:12.993 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994236+0000", 2024-09-11T03:48:12.993 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000120715 2024-09-11T03:48:12.994 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.994 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.994 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:12.994 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994409+0000", 2024-09-11T03:48:12.994 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017384 2024-09-11T03:48:12.994 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.994 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:12.994 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:12.994 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.994 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.994 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5362 2.1b 2:de6b1316:::smithi17557870-32:head [read 0~2117847,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:48:12.994 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973757+0000", 2024-09-11T03:48:12.994 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 101.94053177000001, 2024-09-11T03:48:12.995 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.023415378000000001, 2024-09-11T03:48:12.995 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:12.995 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:12.995 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:12.995 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:12.995 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:12.995 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:12.995 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5362 2024-09-11T03:48:12.995 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.995 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:12.995 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.995 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:12.996 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973757+0000", 2024-09-11T03:48:12.996 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.996 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.996 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.996 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:12.996 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973757+0000", 2024-09-11T03:48:12.996 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:12.996 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.996 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.996 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:12.996 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973758+0000", 2024-09-11T03:48:12.996 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.088e-06 2024-09-11T03:48:12.997 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.997 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.997 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:12.997 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973765+0000", 2024-09-11T03:48:12.997 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1310000000000003e-06 2024-09-11T03:48:12.997 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.997 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.997 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:12.997 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973776+0000", 2024-09-11T03:48:12.997 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1762e-05 2024-09-11T03:48:12.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.998 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:12.998 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973785+0000", 2024-09-11T03:48:12.998 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8629999999999997e-06 2024-09-11T03:48:12.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.998 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.998 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973985+0000", 2024-09-11T03:48:12.998 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000200051 2024-09-11T03:48:12.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.998 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:48:12.998 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973995+0000", 2024-09-11T03:48:12.999 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0054000000000001e-05 2024-09-11T03:48:12.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.999 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.999 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:12.999 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994538+0000", 2024-09-11T03:48:12.999 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020542878000000001 2024-09-11T03:48:12.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.999 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:12.999 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:12.999 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994729+0000", 2024-09-11T03:48:12.999 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00019097100000000001 2024-09-11T03:48:12.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:12.999 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.000 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:13.000 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.997173+0000", 2024-09-11T03:48:13.000 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024435799999999999 2024-09-11T03:48:13.000 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.000 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:13.000 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.000 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.000 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.000 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5363 2.1b 2:de6b1316:::smithi17557870-32:head [read 0~2117847,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:48:13.000 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973799+0000", 2024-09-11T03:48:13.000 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 101.94048976099999, 2024-09-11T03:48:13.000 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.024619513999999999, 2024-09-11T03:48:13.001 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:13.001 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:13.001 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:13.001 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:13.001 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:13.001 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:13.001 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5363 2024-09-11T03:48:13.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.001 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:13.002 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.002 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:13.002 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973799+0000", 2024-09-11T03:48:13.002 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:13.002 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.002 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.002 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:13.002 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973799+0000", 2024-09-11T03:48:13.002 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:13.002 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.002 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.002 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:13.002 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973800+0000", 2024-09-11T03:48:13.002 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2500000000000005e-07 2024-09-11T03:48:13.003 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.003 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.003 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:13.003 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973804+0000", 2024-09-11T03:48:13.003 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6049999999999998e-06 2024-09-11T03:48:13.003 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.003 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.003 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:13.003 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973811+0000", 2024-09-11T03:48:13.003 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0350000000000004e-06 2024-09-11T03:48:13.003 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.003 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.003 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:13.004 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973816+0000", 2024-09-11T03:48:13.004 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.536e-06 2024-09-11T03:48:13.004 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.004 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.004 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:13.004 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974024+0000", 2024-09-11T03:48:13.004 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00020766099999999999 2024-09-11T03:48:13.004 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.004 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.004 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:48:13.005 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974032+0000", 2024-09-11T03:48:13.005 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3340000000000002e-06 2024-09-11T03:48:13.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.005 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.005 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:13.005 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.997304+0000", 2024-09-11T03:48:13.005 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.02327183 2024-09-11T03:48:13.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.005 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.005 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:13.006 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.997455+0000", 2024-09-11T03:48:13.006 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015147000000000001 2024-09-11T03:48:13.006 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.006 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.006 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:13.006 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.998419+0000", 2024-09-11T03:48:13.006 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00096341799999999996 2024-09-11T03:48:13.006 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.006 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:13.006 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.006 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.006 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.006 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5364 2.1b 2:de6b1316:::smithi17557870-32:head [read 0~2117847,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:48:13.007 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:48:13.007 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 101.94046604099999, 2024-09-11T03:48:13.007 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.025348697, 2024-09-11T03:48:13.007 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:13.007 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:13.007 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:13.007 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:13.007 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:13.007 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:13.007 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5364 2024-09-11T03:48:13.007 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.007 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:13.008 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.008 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:13.008 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:48:13.008 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:13.008 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.008 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.008 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:13.008 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:48:13.008 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:13.008 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.009 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.009 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:13.009 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:48:13.009 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4099999999999999e-07 2024-09-11T03:48:13.009 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.009 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.009 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:13.009 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973826+0000", 2024-09-11T03:48:13.009 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7690000000000001e-06 2024-09-11T03:48:13.009 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.009 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.009 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:13.010 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973833+0000", 2024-09-11T03:48:13.010 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4679999999999997e-06 2024-09-11T03:48:13.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.010 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.010 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:13.010 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973837+0000", 2024-09-11T03:48:13.010 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6070000000000001e-06 2024-09-11T03:48:13.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.010 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.010 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:13.011 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974062+0000", 2024-09-11T03:48:13.011 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00022431400000000001 2024-09-11T03:48:13.011 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.011 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.011 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:48:13.011 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974069+0000", 2024-09-11T03:48:13.011 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8599999999999993e-06 2024-09-11T03:48:13.012 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.012 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.012 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:13.012 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.998467+0000", 2024-09-11T03:48:13.012 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.024397723999999999 2024-09-11T03:48:13.012 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.012 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.012 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:13.012 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.998581+0000", 2024-09-11T03:48:13.012 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000113769 2024-09-11T03:48:13.012 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.013 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.013 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:13.013 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.999172+0000", 2024-09-11T03:48:13.013 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00059074500000000005 2024-09-11T03:48:13.013 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.013 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:13.013 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.013 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.013 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.014 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5778 2.a 2:5014e305:::smithi17557870-46:head [copy-from ver 878] snapc ff=[ff,fe,fd] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-09-11T03:48:13.014 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:06.021580+0000", 2024-09-11T03:48:13.014 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 66.892709480999997, 2024-09-11T03:48:13.014 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.023197538, 2024-09-11T03:48:13.014 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:13.014 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:13.015 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:13.015 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:13.015 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:13.015 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:13.015 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5778 2024-09-11T03:48:13.015 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.015 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:13.015 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.015 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:13.015 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021580+0000", 2024-09-11T03:48:13.015 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:13.015 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.015 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.016 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:13.016 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021580+0000", 2024-09-11T03:48:13.016 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:13.016 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.016 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.016 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:13.016 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021581+0000", 2024-09-11T03:48:13.016 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6099999999999999e-07 2024-09-11T03:48:13.016 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.016 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.016 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:13.016 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021585+0000", 2024-09-11T03:48:13.017 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.279e-06 2024-09-11T03:48:13.017 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.017 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.017 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:13.017 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021593+0000", 2024-09-11T03:48:13.017 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1999999999999994e-06 2024-09-11T03:48:13.017 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.017 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.017 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:13.017 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021599+0000", 2024-09-11T03:48:13.017 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8200000000000002e-06 2024-09-11T03:48:13.017 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.017 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.017 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:13.018 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021691+0000", 2024-09-11T03:48:13.018 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2142e-05 2024-09-11T03:48:13.018 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.018 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.018 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:13.018 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021868+0000", 2024-09-11T03:48:13.018 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017677200000000001 2024-09-11T03:48:13.018 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.018 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.018 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:48:13.018 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.030689+0000", 2024-09-11T03:48:13.019 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0088215310000000005 2024-09-11T03:48:13.019 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.019 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.019 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:13.019 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.038924+0000", 2024-09-11T03:48:13.019 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.008234412 2024-09-11T03:48:13.019 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.019 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.019 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:13.019 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044669+0000", 2024-09-11T03:48:13.019 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0057455529999999996 2024-09-11T03:48:13.019 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.019 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.020 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:48:13.020 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044696+0000", 2024-09-11T03:48:13.020 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6843e-05 2024-09-11T03:48:13.020 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.020 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.020 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:48:13.020 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044748+0000", 2024-09-11T03:48:13.020 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1711e-05 2024-09-11T03:48:13.020 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.020 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.020 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:13.021 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:13.022 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044777+0000", 2024-09-11T03:48:13.022 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9314000000000001e-05 2024-09-11T03:48:13.023 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.023 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:13.023 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.023 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.023 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.023 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5779 2.a 2:5014e305:::smithi17557870-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e608)", 2024-09-11T03:48:13.023 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:06.021627+0000", 2024-09-11T03:48:13.023 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 66.89266164, 2024-09-11T03:48:13.023 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014262804, 2024-09-11T03:48:13.023 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:13.023 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:13.023 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:13.023 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:13.024 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:13.024 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:13.024 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5779 2024-09-11T03:48:13.024 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.024 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:13.024 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.024 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:13.024 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021627+0000", 2024-09-11T03:48:13.024 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:13.024 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.024 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.024 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:13.024 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021627+0000", 2024-09-11T03:48:13.025 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:13.025 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.025 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.025 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:13.025 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021628+0000", 2024-09-11T03:48:13.025 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6199999999999998e-07 2024-09-11T03:48:13.025 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.025 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.025 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:13.025 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021631+0000", 2024-09-11T03:48:13.025 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5339999999999998e-06 2024-09-11T03:48:13.025 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.026 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.026 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:13.026 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021641+0000", 2024-09-11T03:48:13.026 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0696999999999999e-05 2024-09-11T03:48:13.026 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.026 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.026 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:13.026 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021647+0000", 2024-09-11T03:48:13.026 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5879999999999997e-06 2024-09-11T03:48:13.026 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.026 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.026 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:13.026 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021980+0000", 2024-09-11T03:48:13.026 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00033303 2024-09-11T03:48:13.027 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.027 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.027 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:48:13.027 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.022028+0000", 2024-09-11T03:48:13.027 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8285000000000003e-05 2024-09-11T03:48:13.027 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.027 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.027 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:13.027 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.035501+0000", 2024-09-11T03:48:13.027 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013473076 2024-09-11T03:48:13.027 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.027 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.027 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:13.028 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.035711+0000", 2024-09-11T03:48:13.028 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000209932 2024-09-11T03:48:13.028 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.028 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.028 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:13.028 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.035890+0000", 2024-09-11T03:48:13.028 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017909999999999999 2024-09-11T03:48:13.028 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.028 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:13.028 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.028 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.029 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.029 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5983 2.1b 2:de6b1316:::smithi17557870-32:head [copy-from ver 692] snapc 108=[108,107] ondisk+write+known_if_redirected+supports_pool_eio e627)", 2024-09-11T03:48:13.029 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:25.010709+0000", 2024-09-11T03:48:13.029 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 47.903580380999998, 2024-09-11T03:48:13.029 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022123384999999999, 2024-09-11T03:48:13.029 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:13.029 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:13.029 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:13.029 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:13.029 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:13.029 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:13.029 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5983 2024-09-11T03:48:13.030 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.030 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:13.030 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.030 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:13.030 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010709+0000", 2024-09-11T03:48:13.030 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:13.030 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.030 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.031 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:13.031 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010709+0000", 2024-09-11T03:48:13.031 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:13.031 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.031 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.031 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:13.031 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010711+0000", 2024-09-11T03:48:13.032 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.367e-06 2024-09-11T03:48:13.032 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.032 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.032 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:13.032 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010723+0000", 2024-09-11T03:48:13.032 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2098e-05 2024-09-11T03:48:13.032 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.032 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.033 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:13.033 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010748+0000", 2024-09-11T03:48:13.033 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4641e-05 2024-09-11T03:48:13.033 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.033 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.033 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:13.034 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010761+0000", 2024-09-11T03:48:13.034 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.329e-05 2024-09-11T03:48:13.034 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.034 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.034 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:13.034 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010856+0000", 2024-09-11T03:48:13.035 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5278000000000005e-05 2024-09-11T03:48:13.035 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.035 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.035 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:48:13.035 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010862+0000", 2024-09-11T03:48:13.035 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.088e-06 2024-09-11T03:48:13.036 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.036 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.036 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:13.036 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.012360+0000", 2024-09-11T03:48:13.036 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014979940000000001 2024-09-11T03:48:13.036 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.036 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.036 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:13.037 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.012437+0000", 2024-09-11T03:48:13.037 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6959000000000006e-05 2024-09-11T03:48:13.037 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.037 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.037 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,9", 2024-09-11T03:48:13.037 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.020310+0000", 2024-09-11T03:48:13.037 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0078724099999999998 2024-09-11T03:48:13.037 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.037 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.037 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:48:13.037 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.028691+0000", 2024-09-11T03:48:13.038 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.008381299 2024-09-11T03:48:13.038 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.038 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.038 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:13.038 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.028800+0000", 2024-09-11T03:48:13.038 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000109082 2024-09-11T03:48:13.039 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.039 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.039 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:13.039 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.032710+0000", 2024-09-11T03:48:13.039 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039093660000000001 2024-09-11T03:48:13.039 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.039 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.039 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:48:13.039 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.032764+0000", 2024-09-11T03:48:13.039 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4415e-05 2024-09-11T03:48:13.039 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.040 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.040 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:13.040 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.032832+0000", 2024-09-11T03:48:13.040 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8097999999999998e-05 2024-09-11T03:48:13.040 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.040 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:13.040 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.040 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.040 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.040 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5984 2.1b 2:de6b1316:::smithi17557870-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e627)", 2024-09-11T03:48:13.040 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:25.010792+0000", 2024-09-11T03:48:13.040 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 47.903496943999997, 2024-09-11T03:48:13.040 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014851705, 2024-09-11T03:48:13.041 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:13.041 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:13.041 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:48:13.041 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:13.041 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:13.041 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:13.041 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5984 2024-09-11T03:48:13.041 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.041 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:13.041 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.041 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:13.041 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010792+0000", 2024-09-11T03:48:13.041 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:13.041 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.042 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.042 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:13.042 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010792+0000", 2024-09-11T03:48:13.042 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:13.042 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.042 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.042 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:13.042 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010793+0000", 2024-09-11T03:48:13.042 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8999999999999996e-07 2024-09-11T03:48:13.042 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.042 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.042 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:13.043 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010796+0000", 2024-09-11T03:48:13.043 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6200000000000001e-06 2024-09-11T03:48:13.043 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.043 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.043 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:13.043 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010811+0000", 2024-09-11T03:48:13.043 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4620999999999999e-05 2024-09-11T03:48:13.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.044 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.044 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:13.044 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010818+0000", 2024-09-11T03:48:13.044 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5490000000000003e-06 2024-09-11T03:48:13.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.044 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.044 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:13.044 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010926+0000", 2024-09-11T03:48:13.044 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108318 2024-09-11T03:48:13.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.044 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.044 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:48:13.045 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010931+0000", 2024-09-11T03:48:13.045 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1669999999999998e-06 2024-09-11T03:48:13.045 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.045 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.045 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:13.045 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.012597+0000", 2024-09-11T03:48:13.045 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001665487 2024-09-11T03:48:13.045 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.045 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.045 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:48:13.046 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.012646+0000", 2024-09-11T03:48:13.046 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9073000000000003e-05 2024-09-11T03:48:13.046 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.046 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.046 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:13.046 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.025501+0000", 2024-09-11T03:48:13.047 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012855263 2024-09-11T03:48:13.047 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.047 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.047 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:13.047 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.025569+0000", 2024-09-11T03:48:13.047 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8099e-05 2024-09-11T03:48:13.048 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.048 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.048 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:13.048 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.025644+0000", 2024-09-11T03:48:13.048 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4818000000000001e-05 2024-09-11T03:48:13.048 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.048 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:13.048 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.048 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.048 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.048 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6106 2.21 2:8638b65e:::smithi17557870-5:head [copy-from ver 603] snapc 10c=[10c] ondisk+write+known_if_redirected+supports_pool_eio e636)", 2024-09-11T03:48:13.048 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:34.018719+0000", 2024-09-11T03:48:13.048 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 38.895569829000003, 2024-09-11T03:48:13.049 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016460907, 2024-09-11T03:48:13.049 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:13.049 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:13.049 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:13.049 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:13.049 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:13.049 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:13.049 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6106 2024-09-11T03:48:13.049 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.050 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:13.050 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.050 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:13.050 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018719+0000", 2024-09-11T03:48:13.050 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:13.050 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.050 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.050 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:13.050 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018719+0000", 2024-09-11T03:48:13.050 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:13.050 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.050 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.050 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:13.051 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018722+0000", 2024-09-11T03:48:13.051 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6730000000000001e-06 2024-09-11T03:48:13.051 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.051 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.051 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:13.051 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018732+0000", 2024-09-11T03:48:13.051 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0349e-05 2024-09-11T03:48:13.051 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.051 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.051 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:13.051 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018756+0000", 2024-09-11T03:48:13.051 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3493999999999999e-05 2024-09-11T03:48:13.051 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.052 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.052 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:13.052 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018767+0000", 2024-09-11T03:48:13.052 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.079e-05 2024-09-11T03:48:13.052 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.052 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.052 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:13.052 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018875+0000", 2024-09-11T03:48:13.052 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108789 2024-09-11T03:48:13.052 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.052 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.052 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:48:13.053 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018884+0000", 2024-09-11T03:48:13.053 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4800000000000001e-06 2024-09-11T03:48:13.053 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.053 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.053 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:13.053 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.020397+0000", 2024-09-11T03:48:13.053 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015135770000000001 2024-09-11T03:48:13.053 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.053 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.053 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:13.053 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.020508+0000", 2024-09-11T03:48:13.054 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000110351 2024-09-11T03:48:13.054 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.054 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.054 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,8", 2024-09-11T03:48:13.054 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.028802+0000", 2024-09-11T03:48:13.054 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0082943189999999997 2024-09-11T03:48:13.054 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.054 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.054 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:48:13.054 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.033207+0000", 2024-09-11T03:48:13.054 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0044050820000000003 2024-09-11T03:48:13.054 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.054 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.055 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:13.055 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.033684+0000", 2024-09-11T03:48:13.055 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00047677899999999998 2024-09-11T03:48:13.055 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.055 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.055 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:13.055 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.035101+0000", 2024-09-11T03:48:13.055 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014171539999999999 2024-09-11T03:48:13.055 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.055 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.055 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:48:13.055 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.035147+0000", 2024-09-11T03:48:13.055 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5731e-05 2024-09-11T03:48:13.055 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.056 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.056 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:13.056 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.035180+0000", 2024-09-11T03:48:13.056 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3339e-05 2024-09-11T03:48:13.056 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.056 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:13.056 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.056 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.056 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.056 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6365 2.1b 2:de6b1316:::smithi17557870-32:head [copy-from ver 962] snapc 11a=[11a,119,115,114] ondisk+write+known_if_redirected+supports_pool_eio e661)", 2024-09-11T03:48:13.056 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:59.126859+0000", 2024-09-11T03:48:13.056 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 13.787430235, 2024-09-11T03:48:13.056 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014696558, 2024-09-11T03:48:13.057 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:48:13.057 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:48:13.057 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:13.057 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:48:13.057 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:48:13.057 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:13.057 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6365 2024-09-11T03:48:13.057 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.057 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:48:13.057 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.057 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:48:13.057 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:59.126859+0000", 2024-09-11T03:48:13.057 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:13.058 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.058 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.058 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:48:13.058 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:59.126859+0000", 2024-09-11T03:48:13.058 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:48:13.058 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.058 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.058 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:48:13.058 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:59.126859+0000", 2024-09-11T03:48:13.058 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3799999999999998e-07 2024-09-11T03:48:13.058 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.058 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.058 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:48:13.058 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:59.126861+0000", 2024-09-11T03:48:13.058 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1019999999999999e-06 2024-09-11T03:48:13.059 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.059 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.059 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:48:13.059 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:59.126865+0000", 2024-09-11T03:48:13.059 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6899999999999998e-06 2024-09-11T03:48:13.059 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.059 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.059 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:48:13.059 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:59.126868+0000", 2024-09-11T03:48:13.059 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7470000000000001e-06 2024-09-11T03:48:13.059 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.059 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.060 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:48:13.060 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:59.126905+0000", 2024-09-11T03:48:13.060 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7564000000000002e-05 2024-09-11T03:48:13.060 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.060 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.060 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:48:13.060 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:59.126954+0000", 2024-09-11T03:48:13.060 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8365000000000002e-05 2024-09-11T03:48:13.060 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.060 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.060 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,9", 2024-09-11T03:48:13.060 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:59.135872+0000", 2024-09-11T03:48:13.060 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0089188489999999995 2024-09-11T03:48:13.060 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.061 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.061 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:48:13.061 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:59.140006+0000", 2024-09-11T03:48:13.061 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0041331010000000001 2024-09-11T03:48:13.061 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.061 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.061 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:13.061 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:59.140293+0000", 2024-09-11T03:48:13.061 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00028749400000000001 2024-09-11T03:48:13.061 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.061 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.061 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:13.061 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:59.141479+0000", 2024-09-11T03:48:13.062 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001185923 2024-09-11T03:48:13.062 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.062 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.062 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:48:13.062 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:59.141524+0000", 2024-09-11T03:48:13.062 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5059999999999999e-05 2024-09-11T03:48:13.062 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:48:13.062 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:48:13.062 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:48:13.062 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:59.141555+0000", 2024-09-11T03:48:13.063 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1325000000000002e-05 2024-09-11T03:48:13.063 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.063 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:13.063 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.063 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:48:13.063 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:48:13.063 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:48:13.065 INFO:tasks.rados.rados.2.smithi175.stdout:2644: done (3 left) 2024-09-11T03:48:13.065 INFO:tasks.rados.rados.2.smithi175.stdout:2647: done (2 left) 2024-09-11T03:48:13.065 INFO:tasks.rados.rados.2.smithi175.stdout:2649: done (1 left) 2024-09-11T03:48:13.065 INFO:tasks.rados.rados.2.smithi175.stdout:2650: done (0 left) 2024-09-11T03:48:13.065 INFO:tasks.rados.rados.2.smithi175.stdout:2651: copy_from oid 23 from oid 19 current snap is 291 2024-09-11T03:48:13.065 INFO:tasks.rados.rados.2.smithi175.stdout:2652: copy_from oid 18 from oid 17 current snap is 291 2024-09-11T03:48:13.065 INFO:tasks.rados.rados.2.smithi175.stdout:2653: rollback oid 39 current snap is 291 2024-09-11T03:48:13.065 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 39 to 289 2024-09-11T03:48:13.065 INFO:tasks.rados.rados.2.smithi175.stdout:2654: delete oid 46 current snap is 291 2024-09-11T03:48:13.065 INFO:tasks.rados.rados.2.smithi175.stdout:2653: finishing rollback tid 1 to smithi17557870-39 2024-09-11T03:48:13.065 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 0 (ObjNum 81 snap 0 seq_num 1) dirty dne 2024-09-11T03:48:13.066 INFO:tasks.rados.rados.2.smithi175.stdout:2653: done (3 left) 2024-09-11T03:48:13.066 INFO:tasks.rados.rados.2.smithi175.stdout:2654: done (2 left) 2024-09-11T03:48:13.066 INFO:tasks.rados.rados.2.smithi175.stdout:2655: read oid 48 snap 290 2024-09-11T03:48:13.066 INFO:tasks.rados.rados.2.smithi175.stdout:2655: expect (ObjNum 896 snap 284 seq_num 896) 2024-09-11T03:48:13.066 INFO:tasks.rados.rados.2.smithi175.stdout:2656: snap_create 2024-09-11T03:48:13.066 INFO:tasks.rados.rados.2.smithi175.stdout:2652: finishing copy_from to smithi17557870-18 2024-09-11T03:48:13.066 INFO:tasks.rados.rados.2.smithi175.stdout:2652: got expected ENOENT (src dne) 2024-09-11T03:48:13.066 INFO:tasks.rados.rados.2.smithi175.stdout:2652: finishing copy_from racing read to smithi17557870-18 2024-09-11T03:48:13.066 INFO:tasks.rados.rados.2.smithi175.stdout:2655: expect (ObjNum 896 snap 284 seq_num 896) 2024-09-11T03:48:13.088 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:13.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:13.089 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:13.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:13.195 INFO:tasks.rados.rados.2.smithi175.stdout:2651: finishing copy_from racing read to smithi17557870-23 2024-09-11T03:48:13.195 INFO:tasks.rados.rados.2.smithi175.stdout:2651: finishing copy_from to smithi17557870-23 2024-09-11T03:48:13.195 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 544 (ObjNum 877 snap 277 seq_num 877) dirty exists 2024-09-11T03:48:13.216 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:13.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:13.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:13.318 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:13.340 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:13.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:13.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:13.346 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:13.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:13 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1545046826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:13 smithi121 ceph-mon[30729]: osdmap e675: 12 total, 12 up, 12 in 2024-09-11T03:48:13.445 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:13.467 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:13 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1545046826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:13.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:13 smithi084 ceph-mon[27254]: osdmap e675: 12 total, 12 up, 12 in 2024-09-11T03:48:13.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:13.468 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:13.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:13 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1545046826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:13 smithi175 ceph-mon[30894]: osdmap e675: 12 total, 12 up, 12 in 2024-09-11T03:48:13.570 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:13.591 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:13.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:13.592 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:13.692 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:13.713 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:13.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:13.714 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:13.815 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:13.834 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:48:13.834 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T03:48:13.834 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:48:13.835 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:13.837 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:13.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:48:13.939 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:13.960 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:13.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:13.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:13.961 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T03:48:13.994 INFO:tasks.rados.rados.2.smithi175.stdout:2651: done (3 left) 2024-09-11T03:48:13.994 INFO:tasks.rados.rados.2.smithi175.stdout:2652: done (2 left) 2024-09-11T03:48:13.994 INFO:tasks.rados.rados.2.smithi175.stdout:2655: done (1 left) 2024-09-11T03:48:13.994 INFO:tasks.rados.rados.2.smithi175.stdout:2656: done (0 left) 2024-09-11T03:48:13.995 INFO:tasks.rados.rados.2.smithi175.stdout:2657: write (excl) oid 3 current snap is 292 2024-09-11T03:48:13.995 INFO:tasks.rados.rados.2.smithi175.stdout:2657: seq_num 915 ranges {578405=415195,1410318=598977,2744156=584480} 2024-09-11T03:48:13.999 INFO:tasks.rados.rados.2.smithi175.stdout:2657: writing smithi17557870-3 from 578405 to 993600 tid 1 2024-09-11T03:48:14.005 INFO:tasks.rados.rados.2.smithi175.stdout:2657: writing smithi17557870-3 from 1410318 to 2009295 tid 2 2024-09-11T03:48:14.012 INFO:tasks.rados.rados.2.smithi175.stdout:2657: writing smithi17557870-3 from 2744156 to 3328636 tid 3 2024-09-11T03:48:14.012 INFO:tasks.rados.rados.2.smithi175.stdout:2658: read oid 23 snap -1 2024-09-11T03:48:14.012 INFO:tasks.rados.rados.2.smithi175.stdout:2658: expect (ObjNum 877 snap 277 seq_num 877) 2024-09-11T03:48:14.012 INFO:tasks.rados.rados.2.smithi175.stdout:2659: snap_remove snap 290 2024-09-11T03:48:14.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:14.084 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:14.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:14.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:14.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:14.207 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:14.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:14.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:14 smithi175 ceph-mon[30894]: pgmap v1167: 67 pgs: 67 active+clean; 182 MiB data, 846 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.1 MiB/s wr, 16 op/s 2024-09-11T03:48:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:14 smithi175 ceph-mon[30894]: osdmap e676: 12 total, 12 up, 12 in 2024-09-11T03:48:14.309 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:14.335 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:14.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:14.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:14.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:14 smithi084 ceph-mon[27254]: pgmap v1167: 67 pgs: 67 active+clean; 182 MiB data, 846 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.1 MiB/s wr, 16 op/s 2024-09-11T03:48:14.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:14 smithi084 ceph-mon[27254]: osdmap e676: 12 total, 12 up, 12 in 2024-09-11T03:48:14.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:14 smithi121 ceph-mon[30729]: pgmap v1167: 67 pgs: 67 active+clean; 182 MiB data, 846 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.1 MiB/s wr, 16 op/s 2024-09-11T03:48:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:14 smithi121 ceph-mon[30729]: osdmap e676: 12 total, 12 up, 12 in 2024-09-11T03:48:14.436 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:14.459 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:14.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:14.460 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:14.561 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:14.568 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:14.582 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:14.587 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:14.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:14.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:14.689 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:14.714 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:14.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:14.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:14.816 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:14.840 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:14.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:14.841 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:14.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:14.963 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:14.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:14.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:14.999 INFO:tasks.rados.rados.2.smithi175.stdout:2659: done (2 left) 2024-09-11T03:48:14.999 INFO:tasks.rados.rados.2.smithi175.stdout:2660: snap_create 2024-09-11T03:48:14.999 INFO:tasks.rados.rados.2.smithi175.stdout:2657: finishing write tid 1 to smithi17557870-3 2024-09-11T03:48:14.999 INFO:tasks.rados.rados.2.smithi175.stdout:2657: finishing write tid 2 to smithi17557870-3 2024-09-11T03:48:14.999 INFO:tasks.rados.rados.2.smithi175.stdout:2657: finishing write tid 3 to smithi17557870-3 2024-09-11T03:48:14.999 INFO:tasks.rados.rados.2.smithi175.stdout:2657: finishing write tid 4 to smithi17557870-3 2024-09-11T03:48:14.999 INFO:tasks.rados.rados.2.smithi175.stdout:2657: finishing write tid 5 to smithi17557870-3 2024-09-11T03:48:14.999 INFO:tasks.rados.rados.2.smithi175.stdout:2657: oid 3 updating version 0 to 981 2024-09-11T03:48:14.999 INFO:tasks.rados.rados.2.smithi175.stdout:2657: oid 3 version 981 is already newer than 980 2024-09-11T03:48:14.999 INFO:tasks.rados.rados.2.smithi175.stdout:2657: oid 3 updating version 981 to 982 2024-09-11T03:48:14.999 INFO:tasks.rados.rados.2.smithi175.stdout:2657: oid 3 updating version 982 to 983 2024-09-11T03:48:14.999 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 983 (ObjNum 915 snap 292 seq_num 915) dirty exists 2024-09-11T03:48:15.000 INFO:tasks.rados.rados.2.smithi175.stdout:2657: left oid 3 (ObjNum 915 snap 292 seq_num 915) 2024-09-11T03:48:15.000 INFO:tasks.rados.rados.2.smithi175.stdout:2658: expect (ObjNum 877 snap 277 seq_num 877) 2024-09-11T03:48:15.064 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:15.087 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:15.088 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:15.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:15.215 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:15.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:15.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:15.305 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:48:15.316 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:15.369 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:15.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:15.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:15.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:15.492 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:15.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:15.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:15.533 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:48:15.534 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:48:15.534 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:48:15.534 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:48:15.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:15.620 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:15.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:15.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:15.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:15.747 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:15.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:15.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:15.870 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:15.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:15.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:15.972 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:15.994 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:15.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:15.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:15.995 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:48:15.998 INFO:tasks.rados.rados.2.smithi175.stdout:2657: done (2 left) 2024-09-11T03:48:15.998 INFO:tasks.rados.rados.2.smithi175.stdout:2658: done (1 left) 2024-09-11T03:48:15.998 INFO:tasks.rados.rados.2.smithi175.stdout:2660: done (0 left) 2024-09-11T03:48:15.998 INFO:tasks.rados.rados.2.smithi175.stdout:2661: read oid 8 snap -1 2024-09-11T03:48:15.998 INFO:tasks.rados.rados.2.smithi175.stdout:2661: expect (ObjNum 886 snap 283 seq_num 886) 2024-09-11T03:48:15.998 INFO:tasks.rados.rados.2.smithi175.stdout:2662: write (excl) oid 45 current snap is 293 2024-09-11T03:48:15.998 INFO:tasks.rados.rados.2.smithi175.stdout:2662: seq_num 916 ranges {595212=490444,1868800=458988,2887932=507280} 2024-09-11T03:48:16.004 INFO:tasks.rados.rados.2.smithi175.stdout:2662: writing smithi17557870-45 from 595212 to 1085656 tid 1 2024-09-11T03:48:16.011 INFO:tasks.rados.rados.2.smithi175.stdout:2662: writing smithi17557870-45 from 1868800 to 2327788 tid 2 2024-09-11T03:48:16.017 INFO:tasks.rados.rados.2.smithi175.stdout:2662: writing smithi17557870-45 from 2887932 to 3395212 tid 3 2024-09-11T03:48:16.017 INFO:tasks.rados.rados.2.smithi175.stdout:2663: snap_remove snap 280 2024-09-11T03:48:16.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:16.121 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:16.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:16.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:16.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:15 smithi084 ceph-mon[27254]: pgmap v1170: 67 pgs: 67 active+clean; 185 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 4.8 MiB/s wr, 19 op/s 2024-09-11T03:48:16.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:15 smithi084 ceph-mon[27254]: osdmap e677: 12 total, 12 up, 12 in 2024-09-11T03:48:16.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:15 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/483807821' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:48:16.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:16.248 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:16.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:16.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:16.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:16.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:16.375 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:16.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:16.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:15 smithi121 ceph-mon[30729]: pgmap v1170: 67 pgs: 67 active+clean; 185 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 4.8 MiB/s wr, 19 op/s 2024-09-11T03:48:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:15 smithi121 ceph-mon[30729]: osdmap e677: 12 total, 12 up, 12 in 2024-09-11T03:48:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:15 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/483807821' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:48:16.476 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:15 smithi175 ceph-mon[30894]: pgmap v1170: 67 pgs: 67 active+clean; 185 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 4.8 MiB/s wr, 19 op/s 2024-09-11T03:48:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:16 smithi175 ceph-mon[30894]: osdmap e677: 12 total, 12 up, 12 in 2024-09-11T03:48:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:16 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/483807821' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:48:16.490 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T03:48:16.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:16.499 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:16.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:16.601 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:16.624 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:16.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:16.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:16.726 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:16.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:16.748 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:16.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:16.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:16.875 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:16.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:16.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:16.977 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:16.995 INFO:tasks.rados.rados.2.smithi175.stdout:2663: done (2 left) 2024-09-11T03:48:16.995 INFO:tasks.rados.rados.2.smithi175.stdout:2664: rollback oid 7 current snap is 293 2024-09-11T03:48:16.995 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 7 to 285 2024-09-11T03:48:16.995 INFO:tasks.rados.rados.2.smithi175.stdout:2661: expect (ObjNum 886 snap 283 seq_num 886) 2024-09-11T03:48:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:17.003 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:17.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:17.029 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:17.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:17.125 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:17.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:17.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:17.149 INFO:tasks.rados.rados.2.smithi175.stdout:2662: finishing write tid 1 to smithi17557870-45 2024-09-11T03:48:17.150 INFO:tasks.rados.rados.2.smithi175.stdout:2662: finishing write tid 2 to smithi17557870-45 2024-09-11T03:48:17.150 INFO:tasks.rados.rados.2.smithi175.stdout:2662: finishing write tid 3 to smithi17557870-45 2024-09-11T03:48:17.150 INFO:tasks.rados.rados.2.smithi175.stdout:2662: finishing write tid 4 to smithi17557870-45 2024-09-11T03:48:17.150 INFO:tasks.rados.rados.2.smithi175.stdout:2662: finishing write tid 5 to smithi17557870-45 2024-09-11T03:48:17.150 INFO:tasks.rados.rados.2.smithi175.stdout:2662: oid 45 updating version 0 to 734 2024-09-11T03:48:17.150 INFO:tasks.rados.rados.2.smithi175.stdout:2662: oid 45 version 734 is already newer than 732 2024-09-11T03:48:17.150 INFO:tasks.rados.rados.2.smithi175.stdout:2662: oid 45 version 734 is already newer than 733 2024-09-11T03:48:17.150 INFO:tasks.rados.rados.2.smithi175.stdout:2662: oid 45 version 734 is already newer than 731 2024-09-11T03:48:17.150 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 734 (ObjNum 916 snap 293 seq_num 916) dirty exists 2024-09-11T03:48:17.150 INFO:tasks.rados.rados.2.smithi175.stdout:2662: left oid 45 (ObjNum 916 snap 293 seq_num 916) 2024-09-11T03:48:17.150 INFO:tasks.rados.rados.2.smithi175.stdout:2664: finishing rollback tid 0 to smithi17557870-7 2024-09-11T03:48:17.151 INFO:tasks.rados.rados.2.smithi175.stdout:2664: finishing rollback tid 1 to smithi17557870-7 2024-09-11T03:48:17.151 INFO:tasks.rados.rados.2.smithi175.stdout:2664: finishing rollback tid 2 to smithi17557870-7 2024-09-11T03:48:17.151 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 843 (ObjNum 800 snap 255 seq_num 800) dirty exists 2024-09-11T03:48:17.151 INFO:tasks.rados.rados.2.smithi175.stdout:2661: done (2 left) 2024-09-11T03:48:17.151 INFO:tasks.rados.rados.2.smithi175.stdout:2662: done (1 left) 2024-09-11T03:48:17.151 INFO:tasks.rados.rados.2.smithi175.stdout:2664: done (0 left) 2024-09-11T03:48:17.151 INFO:tasks.rados.rados.2.smithi175.stdout:2665: snap_create 2024-09-11T03:48:17.227 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:17.249 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:17.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:17.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:17.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/483807821' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:48:17.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:17 smithi084 ceph-mon[27254]: osdmap e678: 12 total, 12 up, 12 in 2024-09-11T03:48:17.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:17 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:17.296 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:17 smithi084 ceph-mon[27254]: osdmap e679: 12 total, 12 up, 12 in 2024-09-11T03:48:17.351 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:17.371 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:17.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:17.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:17.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/483807821' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:48:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:17 smithi121 ceph-mon[30729]: osdmap e678: 12 total, 12 up, 12 in 2024-09-11T03:48:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:17 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:17 smithi121 ceph-mon[30729]: osdmap e679: 12 total, 12 up, 12 in 2024-09-11T03:48:17.472 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/483807821' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:48:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:17 smithi175 ceph-mon[30894]: osdmap e678: 12 total, 12 up, 12 in 2024-09-11T03:48:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:17 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:17 smithi175 ceph-mon[30894]: osdmap e679: 12 total, 12 up, 12 in 2024-09-11T03:48:17.494 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:17.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:17.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:17.552 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:48:17.552 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:48:17.567 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:48:17.567 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:48:17.584 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:48:17.584 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:48:17.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:17.602 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:48:17.602 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:48:17.602 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:17.621 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:48:17.621 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:48:17.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:17.622 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:17.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:17.639 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:48:17.639 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:48:17.656 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:48:17.656 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:48:17.672 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:48:17.673 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:48:17.689 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:48:17.689 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:48:17.706 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:48:17.706 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:48:17.722 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:48:17.722 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:48:17.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:17.739 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:48:17.739 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:48:17.744 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:17.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:17.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:17.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:17.871 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:17.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:17.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:17.973 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:17.994 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:17.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:17.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:18.010 INFO:tasks.rados.rados.2.smithi175.stdout:2665: done (0 left) 2024-09-11T03:48:18.010 INFO:tasks.rados.rados.2.smithi175.stdout:2666: copy_from oid 34 from oid 20 current snap is 294 2024-09-11T03:48:18.010 INFO:tasks.rados.rados.2.smithi175.stdout:2667: read oid 42 snap -1 2024-09-11T03:48:18.010 INFO:tasks.rados.rados.2.smithi175.stdout:2667: expect deleted 2024-09-11T03:48:18.010 INFO:tasks.rados.rados.2.smithi175.stdout:2668: copy_from oid 36 from oid 27 current snap is 294 2024-09-11T03:48:18.010 INFO:tasks.rados.rados.2.smithi175.stdout:2669: write oid 18 current snap is 294 2024-09-11T03:48:18.010 INFO:tasks.rados.rados.2.smithi175.stdout:2669: seq_num 917 ranges {612019=565693,1810929=602646,3082649=329370} 2024-09-11T03:48:18.017 INFO:tasks.rados.rados.2.smithi175.stdout:2669: writing smithi17557870-18 from 612019 to 1177712 tid 1 2024-09-11T03:48:18.024 INFO:tasks.rados.rados.2.smithi175.stdout:2669: writing smithi17557870-18 from 1810929 to 2413575 tid 2 2024-09-11T03:48:18.032 INFO:tasks.rados.rados.2.smithi175.stdout:2669: writing smithi17557870-18 from 3082649 to 3412019 tid 3 2024-09-11T03:48:18.033 INFO:tasks.rados.rados.2.smithi175.stdout:2670: rollback oid 28 current snap is 294 2024-09-11T03:48:18.033 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 28 to 274 2024-09-11T03:48:18.033 INFO:tasks.rados.rados.2.smithi175.stdout:2669: finishing write tid 1 to smithi17557870-18 2024-09-11T03:48:18.033 INFO:tasks.rados.rados.2.smithi175.stdout:2666: finishing copy_from racing read to smithi17557870-34 2024-09-11T03:48:18.033 INFO:tasks.rados.rados.2.smithi175.stdout:2668: finishing copy_from racing read to smithi17557870-36 2024-09-11T03:48:18.034 INFO:tasks.rados.rados.2.smithi175.stdout:2667: done (4 left) 2024-09-11T03:48:18.034 INFO:tasks.rados.rados.2.smithi175.stdout:2671: write (excl) oid 14 current snap is 294 2024-09-11T03:48:18.034 INFO:tasks.rados.rados.2.smithi175.stdout:2671: seq_num 918 ranges {628826=640942,1869411=462657,2826425=534649} 2024-09-11T03:48:18.038 INFO:tasks.rados.rados.2.smithi175.stdout:2671: writing smithi17557870-14 from 628826 to 1269768 tid 1 2024-09-11T03:48:18.044 INFO:tasks.rados.rados.2.smithi175.stdout:2671: writing smithi17557870-14 from 1869411 to 2332068 tid 2 2024-09-11T03:48:18.050 INFO:tasks.rados.rados.2.smithi175.stdout:2671: writing smithi17557870-14 from 2826425 to 3361074 tid 3 2024-09-11T03:48:18.050 INFO:tasks.rados.rados.2.smithi175.stdout:2672: delete oid 19 current snap is 294 2024-09-11T03:48:18.050 INFO:tasks.rados.rados.2.smithi175.stdout:2666: finishing copy_from to smithi17557870-34 2024-09-11T03:48:18.050 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 773 (ObjNum 775 snap 248 seq_num 775) dirty exists 2024-09-11T03:48:18.050 INFO:tasks.rados.rados.2.smithi175.stdout:2669: finishing write tid 2 to smithi17557870-18 2024-09-11T03:48:18.051 INFO:tasks.rados.rados.2.smithi175.stdout:2669: finishing write tid 3 to smithi17557870-18 2024-09-11T03:48:18.051 INFO:tasks.rados.rados.2.smithi175.stdout:2670: finishing rollback tid 0 to smithi17557870-28 2024-09-11T03:48:18.051 INFO:tasks.rados.rados.2.smithi175.stdout:2669: finishing write tid 4 to smithi17557870-18 2024-09-11T03:48:18.051 INFO:tasks.rados.rados.2.smithi175.stdout:2670: finishing rollback tid 1 to smithi17557870-28 2024-09-11T03:48:18.051 INFO:tasks.rados.rados.2.smithi175.stdout:2670: finishing rollback tid 2 to smithi17557870-28 2024-09-11T03:48:18.051 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 995 (ObjNum 872 snap 274 seq_num 872) dirty exists 2024-09-11T03:48:18.051 INFO:tasks.rados.rados.2.smithi175.stdout:2669: finishing write tid 5 to smithi17557870-18 2024-09-11T03:48:18.051 INFO:tasks.rados.rados.2.smithi175.stdout:2669: oid 18 updating version 0 to 487 2024-09-11T03:48:18.051 INFO:tasks.rados.rados.2.smithi175.stdout:2669: oid 18 updating version 487 to 488 2024-09-11T03:48:18.051 INFO:tasks.rados.rados.2.smithi175.stdout:2669: oid 18 version 488 is already newer than 486 2024-09-11T03:48:18.051 INFO:tasks.rados.rados.2.smithi175.stdout:2669: oid 18 version 488 is already newer than 485 2024-09-11T03:48:18.051 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 488 (ObjNum 917 snap 294 seq_num 917) dirty exists 2024-09-11T03:48:18.051 INFO:tasks.rados.rados.2.smithi175.stdout:2669: left oid 18 (ObjNum 917 snap 294 seq_num 917) 2024-09-11T03:48:18.052 INFO:tasks.rados.rados.2.smithi175.stdout:2668: finishing copy_from to smithi17557870-36 2024-09-11T03:48:18.052 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 962 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:48:18.052 INFO:tasks.rados.rados.2.smithi175.stdout:2671: finishing write tid 1 to smithi17557870-14 2024-09-11T03:48:18.052 INFO:tasks.rados.rados.2.smithi175.stdout:2671: finishing write tid 2 to smithi17557870-14 2024-09-11T03:48:18.053 INFO:tasks.rados.rados.2.smithi175.stdout:2666: done (5 left) 2024-09-11T03:48:18.053 INFO:tasks.rados.rados.2.smithi175.stdout:2668: done (4 left) 2024-09-11T03:48:18.053 INFO:tasks.rados.rados.2.smithi175.stdout:2669: done (3 left) 2024-09-11T03:48:18.053 INFO:tasks.rados.rados.2.smithi175.stdout:2670: done (2 left) 2024-09-11T03:48:18.053 INFO:tasks.rados.rados.2.smithi175.stdout:2672: done (1 left) 2024-09-11T03:48:18.053 INFO:tasks.rados.rados.2.smithi175.stdout:2673: copy_from oid 24 from oid 35 current snap is 294 2024-09-11T03:48:18.053 INFO:tasks.rados.rados.2.smithi175.stdout:2671: finishing write tid 3 to smithi17557870-14 2024-09-11T03:48:18.053 INFO:tasks.rados.rados.2.smithi175.stdout:2674: read oid 18 snap -1 2024-09-11T03:48:18.053 INFO:tasks.rados.rados.2.smithi175.stdout:2674: expect (ObjNum 917 snap 294 seq_num 917) 2024-09-11T03:48:18.053 INFO:tasks.rados.rados.2.smithi175.stdout:2675: write (excl) oid 22 current snap is 294 2024-09-11T03:48:18.054 INFO:tasks.rados.rados.2.smithi175.stdout:2675: seq_num 920 ranges {662440=507793,1702728=749973,2881271=557193} 2024-09-11T03:48:18.059 INFO:tasks.rados.rados.2.smithi175.stdout:2675: writing smithi17557870-22 from 662440 to 1170233 tid 1 2024-09-11T03:48:18.066 INFO:tasks.rados.rados.2.smithi175.stdout:2675: writing smithi17557870-22 from 1702728 to 2452701 tid 2 2024-09-11T03:48:18.071 INFO:tasks.rados.rados.2.smithi175.stdout:2675: writing smithi17557870-22 from 2881271 to 3438464 tid 3 2024-09-11T03:48:18.071 INFO:tasks.rados.rados.2.smithi175.stdout:2676: write oid 11 current snap is 294 2024-09-11T03:48:18.071 INFO:tasks.rados.rados.2.smithi175.stdout:2676: seq_num 921 ranges {679247=583042,2044857=609984,3192341=286906} 2024-09-11T03:48:18.078 INFO:tasks.rados.rados.2.smithi175.stdout:2676: writing smithi17557870-11 from 679247 to 1262289 tid 1 2024-09-11T03:48:18.085 INFO:tasks.rados.rados.2.smithi175.stdout:2676: writing smithi17557870-11 from 2044857 to 2654841 tid 2 2024-09-11T03:48:18.089 INFO:tasks.rados.rados.2.smithi175.stdout:2676: writing smithi17557870-11 from 3192341 to 3479247 tid 3 2024-09-11T03:48:18.089 INFO:tasks.rados.rados.2.smithi175.stdout:2677: copy_from oid 23 from oid 5 current snap is 294 2024-09-11T03:48:18.089 INFO:tasks.rados.rados.2.smithi175.stdout:2671: finishing write tid 4 to smithi17557870-14 2024-09-11T03:48:18.090 INFO:tasks.rados.rados.2.smithi175.stdout:2671: finishing write tid 5 to smithi17557870-14 2024-09-11T03:48:18.090 INFO:tasks.rados.rados.2.smithi175.stdout:2671: oid 14 updating version 0 to 930 2024-09-11T03:48:18.090 INFO:tasks.rados.rados.2.smithi175.stdout:2671: oid 14 updating version 930 to 931 2024-09-11T03:48:18.090 INFO:tasks.rados.rados.2.smithi175.stdout:2671: oid 14 updating version 931 to 932 2024-09-11T03:48:18.090 INFO:tasks.rados.rados.2.smithi175.stdout:2671: oid 14 version 932 is already newer than 929 2024-09-11T03:48:18.090 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 932 (ObjNum 918 snap 294 seq_num 918) dirty exists 2024-09-11T03:48:18.090 INFO:tasks.rados.rados.2.smithi175.stdout:2671: left oid 14 (ObjNum 918 snap 294 seq_num 918) 2024-09-11T03:48:18.090 INFO:tasks.rados.rados.2.smithi175.stdout:2674: expect (ObjNum 917 snap 294 seq_num 917) 2024-09-11T03:48:18.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:18.118 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:18.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:18.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:18 smithi084 ceph-mon[27254]: pgmap v1173: 67 pgs: 67 active+clean; 188 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-11T03:48:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:18 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:48:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:18 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:48:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:18 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2766971558' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:18.220 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:18.241 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:18.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:18.255 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:48:18.255 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:48:18.261 INFO:tasks.rados.rados.2.smithi175.stdout:2673: finishing copy_from racing read to smithi17557870-24 2024-09-11T03:48:18.261 INFO:tasks.rados.rados.2.smithi175.stdout:2675: finishing write tid 1 to smithi17557870-22 2024-09-11T03:48:18.262 INFO:tasks.rados.rados.2.smithi175.stdout:2673: finishing copy_from to smithi17557870-24 2024-09-11T03:48:18.262 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 891 (ObjNum 876 snap 276 seq_num 876) dirty exists 2024-09-11T03:48:18.262 INFO:tasks.rados.rados.2.smithi175.stdout:2675: finishing write tid 2 to smithi17557870-22 2024-09-11T03:48:18.262 INFO:tasks.rados.rados.2.smithi175.stdout:2671: done (5 left) 2024-09-11T03:48:18.262 INFO:tasks.rados.rados.2.smithi175.stdout:2673: done (4 left) 2024-09-11T03:48:18.262 INFO:tasks.rados.rados.2.smithi175.stdout:2674: done (3 left) 2024-09-11T03:48:18.262 INFO:tasks.rados.rados.2.smithi175.stdout:2678: delete oid 16 current snap is 294 2024-09-11T03:48:18.262 INFO:tasks.rados.rados.2.smithi175.stdout:2675: finishing write tid 3 to smithi17557870-22 2024-09-11T03:48:18.262 INFO:tasks.rados.rados.2.smithi175.stdout:2675: finishing write tid 4 to smithi17557870-22 2024-09-11T03:48:18.262 INFO:tasks.rados.rados.2.smithi175.stdout:2675: finishing write tid 5 to smithi17557870-22 2024-09-11T03:48:18.262 INFO:tasks.rados.rados.2.smithi175.stdout:2675: oid 22 updating version 0 to 998 2024-09-11T03:48:18.262 INFO:tasks.rados.rados.2.smithi175.stdout:2675: oid 22 updating version 998 to 1000 2024-09-11T03:48:18.262 INFO:tasks.rados.rados.2.smithi175.stdout:2675: oid 22 version 1000 is already newer than 997 2024-09-11T03:48:18.262 INFO:tasks.rados.rados.2.smithi175.stdout:2675: oid 22 version 1000 is already newer than 999 2024-09-11T03:48:18.262 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 1000 (ObjNum 920 snap 294 seq_num 920) dirty exists 2024-09-11T03:48:18.263 INFO:tasks.rados.rados.2.smithi175.stdout:2675: left oid 22 (ObjNum 920 snap 294 seq_num 920) 2024-09-11T03:48:18.263 INFO:tasks.rados.rados.2.smithi175.stdout:2676: finishing write tid 1 to smithi17557870-11 2024-09-11T03:48:18.263 INFO:tasks.rados.rados.2.smithi175.stdout:2676: finishing write tid 2 to smithi17557870-11 2024-09-11T03:48:18.263 INFO:tasks.rados.rados.2.smithi175.stdout:2676: finishing write tid 3 to smithi17557870-11 2024-09-11T03:48:18.263 INFO:tasks.rados.rados.2.smithi175.stdout:2676: finishing write tid 4 to smithi17557870-11 2024-09-11T03:48:18.263 INFO:tasks.rados.rados.2.smithi175.stdout:2676: finishing write tid 5 to smithi17557870-11 2024-09-11T03:48:18.263 INFO:tasks.rados.rados.2.smithi175.stdout:2676: oid 11 updating version 0 to 609 2024-09-11T03:48:18.263 INFO:tasks.rados.rados.2.smithi175.stdout:2676: oid 11 version 609 is already newer than 607 2024-09-11T03:48:18.263 INFO:tasks.rados.rados.2.smithi175.stdout:2676: oid 11 version 609 is already newer than 608 2024-09-11T03:48:18.263 INFO:tasks.rados.rados.2.smithi175.stdout:2676: oid 11 updating version 609 to 610 2024-09-11T03:48:18.263 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 610 (ObjNum 921 snap 294 seq_num 921) dirty exists 2024-09-11T03:48:18.263 INFO:tasks.rados.rados.2.smithi175.stdout:2676: left oid 11 (ObjNum 921 snap 294 seq_num 921) 2024-09-11T03:48:18.263 INFO:tasks.rados.rados.2.smithi175.stdout:2677: finishing copy_from racing read to smithi17557870-23 2024-09-11T03:48:18.263 INFO:tasks.rados.rados.2.smithi175.stdout:2677: finishing copy_from to smithi17557870-23 2024-09-11T03:48:18.264 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 719 (ObjNum 904 snap 286 seq_num 904) dirty exists 2024-09-11T03:48:18.264 INFO:tasks.rados.rados.2.smithi175.stdout:2675: done (3 left) 2024-09-11T03:48:18.264 INFO:tasks.rados.rados.2.smithi175.stdout:2676: done (2 left) 2024-09-11T03:48:18.264 INFO:tasks.rados.rados.2.smithi175.stdout:2677: done (1 left) 2024-09-11T03:48:18.264 INFO:tasks.rados.rados.2.smithi175.stdout:2678: done (0 left) 2024-09-11T03:48:18.265 INFO:tasks.rados.rados.2.smithi175.stdout:2679: read oid 22 snap -1 2024-09-11T03:48:18.265 INFO:tasks.rados.rados.2.smithi175.stdout:2679: expect (ObjNum 920 snap 294 seq_num 920) 2024-09-11T03:48:18.265 INFO:tasks.rados.rados.2.smithi175.stdout:2680: write oid 48 current snap is 294 2024-09-11T03:48:18.265 INFO:tasks.rados.rados.2.smithi175.stdout:2680: seq_num 923 ranges {712861=733540,2161821=613653,3247187=265674} 2024-09-11T03:48:18.271 INFO:tasks.rados.rados.2.smithi175.stdout:2680: writing smithi17557870-48 from 712861 to 1446401 tid 1 2024-09-11T03:48:18.279 INFO:tasks.rados.rados.2.smithi175.stdout:2680: writing smithi17557870-48 from 2161821 to 2775474 tid 2 2024-09-11T03:48:18.283 INFO:tasks.rados.rados.2.smithi175.stdout:2680: writing smithi17557870-48 from 3247187 to 3512861 tid 3 2024-09-11T03:48:18.283 INFO:tasks.rados.rados.2.smithi175.stdout:2681: write oid 14 current snap is 294 2024-09-11T03:48:18.284 INFO:tasks.rados.rados.2.smithi175.stdout:2681: seq_num 924 ranges {729668=408789,1820303=757311,3274610=255058} 2024-09-11T03:48:18.289 INFO:tasks.rados.rados.2.smithi175.stdout:2681: writing smithi17557870-14 from 729668 to 1138457 tid 1 2024-09-11T03:48:18.297 INFO:tasks.rados.rados.2.smithi175.stdout:2681: writing smithi17557870-14 from 1820303 to 2577614 tid 2 2024-09-11T03:48:18.302 INFO:tasks.rados.rados.2.smithi175.stdout:2681: writing smithi17557870-14 from 3274610 to 3529668 tid 3 2024-09-11T03:48:18.302 INFO:tasks.rados.rados.2.smithi175.stdout:2682: write oid 31 current snap is 294 2024-09-11T03:48:18.302 INFO:tasks.rados.rados.2.smithi175.stdout:2682: seq_num 925 ranges {746475=484038,1762432=617322,2785680=529906} 2024-09-11T03:48:18.309 INFO:tasks.rados.rados.2.smithi175.stdout:2682: writing smithi17557870-31 from 746475 to 1230513 tid 1 2024-09-11T03:48:18.315 INFO:tasks.rados.rados.2.smithi175.stdout:2682: writing smithi17557870-31 from 1762432 to 2379754 tid 2 2024-09-11T03:48:18.320 INFO:tasks.rados.rados.2.smithi175.stdout:2682: writing smithi17557870-31 from 2785680 to 3315586 tid 3 2024-09-11T03:48:18.321 INFO:tasks.rados.rados.2.smithi175.stdout:2683: rollback oid 42 current snap is 294 2024-09-11T03:48:18.321 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 42 to 292 2024-09-11T03:48:18.321 INFO:tasks.rados.rados.2.smithi175.stdout:2680: finishing write tid 1 to smithi17557870-48 2024-09-11T03:48:18.321 INFO:tasks.rados.rados.2.smithi175.stdout:2679: expect (ObjNum 920 snap 294 seq_num 920) 2024-09-11T03:48:18.343 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:18.364 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:18.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:18.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:18 smithi121 ceph-mon[30729]: pgmap v1173: 67 pgs: 67 active+clean; 188 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-11T03:48:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:18 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:48:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:18 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:48:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:18 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2766971558' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:18.466 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:18 smithi175 ceph-mon[30894]: pgmap v1173: 67 pgs: 67 active+clean; 188 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-11T03:48:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:18 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:48:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:18 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:48:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:18 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2766971558' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:18.487 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:18.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:18.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:18.555 INFO:tasks.rados.rados.2.smithi175.stdout:2680: finishing write tid 2 to smithi17557870-48 2024-09-11T03:48:18.556 INFO:tasks.rados.rados.2.smithi175.stdout:2680: finishing write tid 3 to smithi17557870-48 2024-09-11T03:48:18.556 INFO:tasks.rados.rados.2.smithi175.stdout:2680: finishing write tid 4 to smithi17557870-48 2024-09-11T03:48:18.556 INFO:tasks.rados.rados.2.smithi175.stdout:2680: finishing write tid 5 to smithi17557870-48 2024-09-11T03:48:18.556 INFO:tasks.rados.rados.2.smithi175.stdout:2680: oid 48 updating version 0 to 857 2024-09-11T03:48:18.556 INFO:tasks.rados.rados.2.smithi175.stdout:2680: oid 48 updating version 857 to 858 2024-09-11T03:48:18.556 INFO:tasks.rados.rados.2.smithi175.stdout:2680: oid 48 updating version 858 to 860 2024-09-11T03:48:18.556 INFO:tasks.rados.rados.2.smithi175.stdout:2680: oid 48 version 860 is already newer than 859 2024-09-11T03:48:18.556 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 860 (ObjNum 923 snap 294 seq_num 923) dirty exists 2024-09-11T03:48:18.556 INFO:tasks.rados.rados.2.smithi175.stdout:2680: left oid 48 (ObjNum 923 snap 294 seq_num 923) 2024-09-11T03:48:18.556 INFO:tasks.rados.rados.2.smithi175.stdout:2681: finishing write tid 1 to smithi17557870-14 2024-09-11T03:48:18.556 INFO:tasks.rados.rados.2.smithi175.stdout:2681: finishing write tid 2 to smithi17557870-14 2024-09-11T03:48:18.557 INFO:tasks.rados.rados.2.smithi175.stdout:2681: finishing write tid 3 to smithi17557870-14 2024-09-11T03:48:18.557 INFO:tasks.rados.rados.2.smithi175.stdout:2679: done (4 left) 2024-09-11T03:48:18.557 INFO:tasks.rados.rados.2.smithi175.stdout:2680: done (3 left) 2024-09-11T03:48:18.557 INFO:tasks.rados.rados.2.smithi175.stdout:2684: snap_remove snap 287 2024-09-11T03:48:18.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:18.615 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:18.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:18.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:18.717 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:18.738 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:18.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:18.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:18.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:18.865 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:18.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:18.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:18.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:18.991 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:18.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:18.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:19.023 INFO:tasks.rados.rados.2.smithi175.stdout:2684: done (3 left) 2024-09-11T03:48:19.023 INFO:tasks.rados.rados.2.smithi175.stdout:2685: copy_from oid 27 from oid 37 current snap is 294 2024-09-11T03:48:19.023 INFO:tasks.rados.rados.2.smithi175.stdout:2681: finishing write tid 4 to smithi17557870-14 2024-09-11T03:48:19.023 INFO:tasks.rados.rados.2.smithi175.stdout:2681: finishing write tid 5 to smithi17557870-14 2024-09-11T03:48:19.023 INFO:tasks.rados.rados.2.smithi175.stdout:2681: oid 14 updating version 0 to 936 2024-09-11T03:48:19.024 INFO:tasks.rados.rados.2.smithi175.stdout:2681: oid 14 updating version 936 to 937 2024-09-11T03:48:19.024 INFO:tasks.rados.rados.2.smithi175.stdout:2681: oid 14 version 937 is already newer than 934 2024-09-11T03:48:19.024 INFO:tasks.rados.rados.2.smithi175.stdout:2681: oid 14 version 937 is already newer than 935 2024-09-11T03:48:19.024 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 937 (ObjNum 924 snap 294 seq_num 924) dirty exists 2024-09-11T03:48:19.024 INFO:tasks.rados.rados.2.smithi175.stdout:2681: left oid 14 (ObjNum 924 snap 294 seq_num 924) 2024-09-11T03:48:19.024 INFO:tasks.rados.rados.2.smithi175.stdout:2682: finishing write tid 1 to smithi17557870-31 2024-09-11T03:48:19.024 INFO:tasks.rados.rados.2.smithi175.stdout:2682: finishing write tid 2 to smithi17557870-31 2024-09-11T03:48:19.024 INFO:tasks.rados.rados.2.smithi175.stdout:2683: finishing rollback tid 1 to smithi17557870-42 2024-09-11T03:48:19.024 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-11T03:48:19.024 INFO:tasks.rados.rados.2.smithi175.stdout:2682: finishing write tid 3 to smithi17557870-31 2024-09-11T03:48:19.024 INFO:tasks.rados.rados.2.smithi175.stdout:2682: finishing write tid 4 to smithi17557870-31 2024-09-11T03:48:19.024 INFO:tasks.rados.rados.2.smithi175.stdout:2682: finishing write tid 5 to smithi17557870-31 2024-09-11T03:48:19.024 INFO:tasks.rados.rados.2.smithi175.stdout:2682: oid 31 updating version 0 to 978 2024-09-11T03:48:19.025 INFO:tasks.rados.rados.2.smithi175.stdout:2682: oid 31 updating version 978 to 979 2024-09-11T03:48:19.025 INFO:tasks.rados.rados.2.smithi175.stdout:2682: oid 31 version 979 is already newer than 977 2024-09-11T03:48:19.025 INFO:tasks.rados.rados.2.smithi175.stdout:2682: oid 31 version 979 is already newer than 976 2024-09-11T03:48:19.025 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 979 (ObjNum 925 snap 294 seq_num 925) dirty exists 2024-09-11T03:48:19.025 INFO:tasks.rados.rados.2.smithi175.stdout:2682: left oid 31 (ObjNum 925 snap 294 seq_num 925) 2024-09-11T03:48:19.025 INFO:tasks.rados.rados.2.smithi175.stdout:2681: done (3 left) 2024-09-11T03:48:19.025 INFO:tasks.rados.rados.2.smithi175.stdout:2682: done (2 left) 2024-09-11T03:48:19.025 INFO:tasks.rados.rados.2.smithi175.stdout:2683: done (1 left) 2024-09-11T03:48:19.025 INFO:tasks.rados.rados.2.smithi175.stdout:2686: snap_create 2024-09-11T03:48:19.026 INFO:tasks.rados.rados.2.smithi175.stdout:2685: finishing copy_from to smithi17557870-27 2024-09-11T03:48:19.026 INFO:tasks.rados.rados.2.smithi175.stdout:2685: got expected ENOENT (src dne) 2024-09-11T03:48:19.027 INFO:tasks.rados.rados.2.smithi175.stdout:2685: finishing copy_from racing read to smithi17557870-27 2024-09-11T03:48:19.056 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:48:19.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:19.117 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:19.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:19.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:19.219 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:19.243 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:19.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:19.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:19.345 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:19.369 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:19.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:19.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:19.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:19 smithi084 ceph-mon[27254]: osdmap e680: 12 total, 12 up, 12 in 2024-09-11T03:48:19.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:19 smithi121 ceph-mon[30729]: osdmap e680: 12 total, 12 up, 12 in 2024-09-11T03:48:19.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:19 smithi175 ceph-mon[30894]: osdmap e680: 12 total, 12 up, 12 in 2024-09-11T03:48:19.493 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:48:19.493 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:48:19.494 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-11T03:48:19.494 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T03:48:19.494 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T03:48:19.494 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-11T03:48:19.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:19.495 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:19.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:19.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:19.618 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:19.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:19.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:19.622 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T03:48:19.720 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:19.743 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:19.744 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:19.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:19.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:19.846 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:19.872 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:19.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:19.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:19.977 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:20.001 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:20.002 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:20.023 INFO:tasks.rados.rados.2.smithi175.stdout:2685: done (1 left) 2024-09-11T03:48:20.023 INFO:tasks.rados.rados.2.smithi175.stdout:2686: done (0 left) 2024-09-11T03:48:20.023 INFO:tasks.rados.rados.2.smithi175.stdout:2687: write (excl) oid 17 current snap is 295 2024-09-11T03:48:20.023 INFO:tasks.rados.rados.2.smithi175.stdout:2687: seq_num 926 ranges {763282=559287,1820914=760980,3213103=350179} 2024-09-11T03:48:20.030 INFO:tasks.rados.rados.2.smithi175.stdout:2687: writing smithi17557870-17 from 763282 to 1322569 tid 1 2024-09-11T03:48:20.037 INFO:tasks.rados.rados.2.smithi175.stdout:2687: writing smithi17557870-17 from 1820914 to 2581894 tid 2 2024-09-11T03:48:20.043 INFO:tasks.rados.rados.2.smithi175.stdout:2687: writing smithi17557870-17 from 3213103 to 3563282 tid 3 2024-09-11T03:48:20.043 INFO:tasks.rados.rados.2.smithi175.stdout:2688: copy_from oid 7 from oid 31 current snap is 295 2024-09-11T03:48:20.043 INFO:tasks.rados.rados.2.smithi175.stdout:2687: finishing write tid 1 to smithi17557870-17 2024-09-11T03:48:20.044 INFO:tasks.rados.rados.2.smithi175.stdout:2687: finishing write tid 2 to smithi17557870-17 2024-09-11T03:48:20.044 INFO:tasks.rados.rados.2.smithi175.stdout:2689: read oid 48 snap 286 2024-09-11T03:48:20.044 INFO:tasks.rados.rados.2.smithi175.stdout:2689: expect (ObjNum 896 snap 284 seq_num 896) 2024-09-11T03:48:20.044 INFO:tasks.rados.rados.2.smithi175.stdout:2690: write oid 35 current snap is 295 2024-09-11T03:48:20.044 INFO:tasks.rados.rados.2.smithi175.stdout:2690: seq_num 927 ranges {780089=634536,1879396=620991,3240526=339563} 2024-09-11T03:48:20.051 INFO:tasks.rados.rados.2.smithi175.stdout:2690: writing smithi17557870-35 from 780089 to 1414625 tid 1 2024-09-11T03:48:20.057 INFO:tasks.rados.rados.2.smithi175.stdout:2690: writing smithi17557870-35 from 1879396 to 2500387 tid 2 2024-09-11T03:48:20.063 INFO:tasks.rados.rados.2.smithi175.stdout:2690: writing smithi17557870-35 from 3240526 to 3580089 tid 3 2024-09-11T03:48:20.064 INFO:tasks.rados.rados.2.smithi175.stdout:2691: delete oid 45 current snap is 295 2024-09-11T03:48:20.064 INFO:tasks.rados.rados.2.smithi175.stdout:2687: finishing write tid 3 to smithi17557870-17 2024-09-11T03:48:20.064 INFO:tasks.rados.rados.2.smithi175.stdout:2687: finishing write tid 4 to smithi17557870-17 2024-09-11T03:48:20.064 INFO:tasks.rados.rados.2.smithi175.stdout:2687: finishing write tid 5 to smithi17557870-17 2024-09-11T03:48:20.064 INFO:tasks.rados.rados.2.smithi175.stdout:2687: oid 17 updating version 0 to 898 2024-09-11T03:48:20.064 INFO:tasks.rados.rados.2.smithi175.stdout:2687: oid 17 version 898 is already newer than 897 2024-09-11T03:48:20.064 INFO:tasks.rados.rados.2.smithi175.stdout:2687: oid 17 updating version 898 to 899 2024-09-11T03:48:20.064 INFO:tasks.rados.rados.2.smithi175.stdout:2687: oid 17 updating version 899 to 900 2024-09-11T03:48:20.064 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 900 (ObjNum 926 snap 295 seq_num 926) dirty exists 2024-09-11T03:48:20.064 INFO:tasks.rados.rados.2.smithi175.stdout:2687: left oid 17 (ObjNum 926 snap 295 seq_num 926) 2024-09-11T03:48:20.064 INFO:tasks.rados.rados.2.smithi175.stdout:2688: finishing copy_from racing read to smithi17557870-7 2024-09-11T03:48:20.064 INFO:tasks.rados.rados.2.smithi175.stdout:2689: expect (ObjNum 896 snap 284 seq_num 896) 2024-09-11T03:48:20.104 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:20.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:20 smithi084 ceph-mon[27254]: pgmap v1176: 67 pgs: 67 active+clean; 195 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 2.5 MiB/s wr, 12 op/s 2024-09-11T03:48:20.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:20 smithi084 ceph-mon[27254]: osdmap e681: 12 total, 12 up, 12 in 2024-09-11T03:48:20.126 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:20.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:20.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:20.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:20 smithi121 ceph-mon[30729]: pgmap v1176: 67 pgs: 67 active+clean; 195 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 2.5 MiB/s wr, 12 op/s 2024-09-11T03:48:20.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:20 smithi121 ceph-mon[30729]: osdmap e681: 12 total, 12 up, 12 in 2024-09-11T03:48:20.228 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:20.233 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:20.247 INFO:tasks.rados.rados.2.smithi175.stdout:2690: finishing write tid 1 to smithi17557870-35 2024-09-11T03:48:20.247 INFO:tasks.rados.rados.2.smithi175.stdout:2690: finishing write tid 2 to smithi17557870-35 2024-09-11T03:48:20.247 INFO:tasks.rados.rados.2.smithi175.stdout:2688: finishing copy_from to smithi17557870-7 2024-09-11T03:48:20.247 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 979 (ObjNum 925 snap 294 seq_num 925) dirty exists 2024-09-11T03:48:20.247 INFO:tasks.rados.rados.2.smithi175.stdout:2690: finishing write tid 3 to smithi17557870-35 2024-09-11T03:48:20.247 INFO:tasks.rados.rados.2.smithi175.stdout:2690: finishing write tid 4 to smithi17557870-35 2024-09-11T03:48:20.247 INFO:tasks.rados.rados.2.smithi175.stdout:2690: finishing write tid 5 to smithi17557870-35 2024-09-11T03:48:20.247 INFO:tasks.rados.rados.2.smithi175.stdout:2690: oid 35 updating version 0 to 893 2024-09-11T03:48:20.248 INFO:tasks.rados.rados.2.smithi175.stdout:2690: oid 35 updating version 893 to 894 2024-09-11T03:48:20.248 INFO:tasks.rados.rados.2.smithi175.stdout:2690: oid 35 updating version 894 to 895 2024-09-11T03:48:20.248 INFO:tasks.rados.rados.2.smithi175.stdout:2690: oid 35 version 895 is already newer than 892 2024-09-11T03:48:20.248 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 895 (ObjNum 927 snap 295 seq_num 927) dirty exists 2024-09-11T03:48:20.248 INFO:tasks.rados.rados.2.smithi175.stdout:2690: left oid 35 (ObjNum 927 snap 295 seq_num 927) 2024-09-11T03:48:20.248 INFO:tasks.rados.rados.2.smithi175.stdout:2687: done (4 left) 2024-09-11T03:48:20.248 INFO:tasks.rados.rados.2.smithi175.stdout:2688: done (3 left) 2024-09-11T03:48:20.248 INFO:tasks.rados.rados.2.smithi175.stdout:2689: done (2 left) 2024-09-11T03:48:20.248 INFO:tasks.rados.rados.2.smithi175.stdout:2690: done (1 left) 2024-09-11T03:48:20.248 INFO:tasks.rados.rados.2.smithi175.stdout:2691: done (0 left) 2024-09-11T03:48:20.248 INFO:tasks.rados.rados.2.smithi175.stdout:2692: rollback oid 34 current snap is 295 2024-09-11T03:48:20.248 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 34 to 283 2024-09-11T03:48:20.249 INFO:tasks.rados.rados.2.smithi175.stdout:2693: delete oid 44 current snap is 295 2024-09-11T03:48:20.249 INFO:tasks.rados.rados.2.smithi175.stdout:2693: done (1 left) 2024-09-11T03:48:20.249 INFO:tasks.rados.rados.2.smithi175.stdout:2694: rollback oid 6 current snap is 295 2024-09-11T03:48:20.249 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 6 to 294 2024-09-11T03:48:20.249 INFO:tasks.rados.rados.2.smithi175.stdout:2695: rollback oid 23 current snap is 295 2024-09-11T03:48:20.249 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 23 to 274 2024-09-11T03:48:20.250 INFO:tasks.rados.rados.2.smithi175.stdout:2696: snap_create 2024-09-11T03:48:20.251 INFO:tasks.rados.rados.2.smithi175.stdout:2692: finishing rollback tid 0 to smithi17557870-34 2024-09-11T03:48:20.251 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:20.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:20.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:20.253 INFO:tasks.rados.rados.2.smithi175.stdout:2694: finishing rollback tid 0 to smithi17557870-6 2024-09-11T03:48:20.253 INFO:tasks.rados.rados.2.smithi175.stdout:2694: finishing rollback tid 1 to smithi17557870-6 2024-09-11T03:48:20.253 INFO:tasks.rados.rados.2.smithi175.stdout:2694: finishing rollback tid 2 to smithi17557870-6 2024-09-11T03:48:20.253 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 891 (ObjNum 857 snap 269 seq_num 857) dirty exists 2024-09-11T03:48:20.253 INFO:tasks.rados.rados.2.smithi175.stdout:2695: finishing rollback tid 0 to smithi17557870-23 2024-09-11T03:48:20.253 INFO:tasks.rados.rados.2.smithi175.stdout:2692: finishing rollback tid 1 to smithi17557870-34 2024-09-11T03:48:20.253 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 890 (ObjNum 0 snap 0 seq_num 304) dirty dne 2024-09-11T03:48:20.254 INFO:tasks.rados.rados.2.smithi175.stdout:2695: finishing rollback tid 1 to smithi17557870-23 2024-09-11T03:48:20.254 INFO:tasks.rados.rados.2.smithi175.stdout:2695: finishing rollback tid 2 to smithi17557870-23 2024-09-11T03:48:20.254 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 965 (ObjNum 855 snap 268 seq_num 855) dirty exists 2024-09-11T03:48:20.265 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:20.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:20.375 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:20.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:20.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:20.477 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:20 smithi175 ceph-mon[30894]: pgmap v1176: 67 pgs: 67 active+clean; 195 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 2.5 MiB/s wr, 12 op/s 2024-09-11T03:48:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:20 smithi175 ceph-mon[30894]: osdmap e681: 12 total, 12 up, 12 in 2024-09-11T03:48:20.499 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:20.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:20.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:20.601 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:20.623 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:20.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:20.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:20.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:20.747 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:20.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:20.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:48:20.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:20.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:20.897 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:20.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:20.967 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:48:20.967 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "" 2024-09-11T03:48:20.967 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:48:20.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:21.020 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:21.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:21.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:21.027 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:48:21.033 INFO:tasks.rados.rados.2.smithi175.stdout:2692: done (3 left) 2024-09-11T03:48:21.033 INFO:tasks.rados.rados.2.smithi175.stdout:2694: done (2 left) 2024-09-11T03:48:21.033 INFO:tasks.rados.rados.2.smithi175.stdout:2695: done (1 left) 2024-09-11T03:48:21.033 INFO:tasks.rados.rados.2.smithi175.stdout:2696: done (0 left) 2024-09-11T03:48:21.033 INFO:tasks.rados.rados.2.smithi175.stdout:2697: snap_remove snap 294 2024-09-11T03:48:21.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:21.144 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:21.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:21.247 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:21.269 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:21.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:21.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:21.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:21.393 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:21.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:21 smithi084 ceph-mon[27254]: osdmap e682: 12 total, 12 up, 12 in 2024-09-11T03:48:21.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1565251660' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:48:21.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:21.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:21.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:21 smithi121 ceph-mon[30729]: osdmap e682: 12 total, 12 up, 12 in 2024-09-11T03:48:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:21 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1565251660' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:48:21.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:21 smithi175 ceph-mon[30894]: osdmap e682: 12 total, 12 up, 12 in 2024-09-11T03:48:21.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:21 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1565251660' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:48:21.496 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:21.518 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:21.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:21.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:21.620 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:21.641 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:21.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:21.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:21.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:21.766 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:21.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:21.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:21.868 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:21.894 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:21.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:21.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:21.996 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:21.996 INFO:tasks.rados.rados.2.smithi175.stdout:2697: done (0 left) 2024-09-11T03:48:21.997 INFO:tasks.rados.rados.2.smithi175.stdout:2698: snap_create 2024-09-11T03:48:22.023 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:22.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:22.024 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:22.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:22 smithi084 ceph-mon[27254]: pgmap v1179: 67 pgs: 67 active+clean; 195 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2024-09-11T03:48:22.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:22 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1565251660' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:48:22.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:22 smithi084 ceph-mon[27254]: osdmap e683: 12 total, 12 up, 12 in 2024-09-11T03:48:22.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:22 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:22.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:22 smithi084 ceph-mon[27254]: osdmap e684: 12 total, 12 up, 12 in 2024-09-11T03:48:22.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:22.138 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:48:22.138 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:48:22.148 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:22.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:22.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:22.250 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:22.271 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:22.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:22.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:22.322 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:22 smithi175 ceph-mon[30894]: pgmap v1179: 67 pgs: 67 active+clean; 195 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2024-09-11T03:48:22.322 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:22 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1565251660' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:48:22.322 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:22 smithi175 ceph-mon[30894]: osdmap e683: 12 total, 12 up, 12 in 2024-09-11T03:48:22.322 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:22 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:22.322 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:22 smithi175 ceph-mon[30894]: osdmap e684: 12 total, 12 up, 12 in 2024-09-11T03:48:22.373 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:22.395 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:22.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:22.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:22.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:22 smithi121 ceph-mon[30729]: pgmap v1179: 67 pgs: 67 active+clean; 195 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2024-09-11T03:48:22.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:22 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1565251660' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:48:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:22 smithi121 ceph-mon[30729]: osdmap e683: 12 total, 12 up, 12 in 2024-09-11T03:48:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:22 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:22 smithi121 ceph-mon[30729]: osdmap e684: 12 total, 12 up, 12 in 2024-09-11T03:48:22.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:22.519 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:22.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:22.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:22.621 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:22.644 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:22.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:22.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:22.746 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:22.768 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:22.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:22.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:22.882 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:22.896 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:22.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:22.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:22.998 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:22.999 INFO:tasks.rados.rados.2.smithi175.stdout:2698: done (0 left) 2024-09-11T03:48:22.999 INFO:tasks.rados.rados.2.smithi175.stdout:2699: delete oid 45 current snap is 297 2024-09-11T03:48:23.002 INFO:tasks.rados.rados.2.smithi175.stdout:2699: done (0 left) 2024-09-11T03:48:23.002 INFO:tasks.rados.rados.2.smithi175.stdout:2700: read oid 14 snap -1 2024-09-11T03:48:23.002 INFO:tasks.rados.rados.2.smithi175.stdout:2700: expect (ObjNum 924 snap 294 seq_num 924) 2024-09-11T03:48:23.002 INFO:tasks.rados.rados.2.smithi175.stdout:2701: read oid 47 snap -1 2024-09-11T03:48:23.002 INFO:tasks.rados.rados.2.smithi175.stdout:2701: expect (ObjNum 867 snap 273 seq_num 867) 2024-09-11T03:48:23.002 INFO:tasks.rados.rados.2.smithi175.stdout:2702: write (excl) oid 26 current snap is 297 2024-09-11T03:48:23.002 INFO:tasks.rados.rados.2.smithi175.stdout:2702: seq_num 931 ranges {447317=651885,1713324=628329,3066571=580746} 2024-09-11T03:48:23.007 INFO:tasks.rados.rados.2.smithi175.stdout:2702: writing smithi17557870-26 from 447317 to 1099202 tid 1 2024-09-11T03:48:23.014 INFO:tasks.rados.rados.2.smithi175.stdout:2702: writing smithi17557870-26 from 1713324 to 2341653 tid 2 2024-09-11T03:48:23.022 INFO:tasks.rados.rados.2.smithi175.stdout:2702: writing smithi17557870-26 from 3066571 to 3647317 tid 3 2024-09-11T03:48:23.022 INFO:tasks.rados.rados.2.smithi175.stdout:2703: write (excl) oid 37 current snap is 297 2024-09-11T03:48:23.022 INFO:tasks.rados.rados.2.smithi175.stdout:2703: seq_num 932 ranges {464124=727134,1771806=771987,2977641=641516} 2024-09-11T03:48:23.027 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:23.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:23.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:23.028 INFO:tasks.rados.rados.2.smithi175.stdout:2703: writing smithi17557870-37 from 464124 to 1191258 tid 1 2024-09-11T03:48:23.036 INFO:tasks.rados.rados.2.smithi175.stdout:2703: writing smithi17557870-37 from 1771806 to 2543793 tid 2 2024-09-11T03:48:23.042 INFO:tasks.rados.rados.2.smithi175.stdout:2703: writing smithi17557870-37 from 2977641 to 3619157 tid 3 2024-09-11T03:48:23.042 INFO:tasks.rados.rados.2.smithi175.stdout:2704: delete oid 5 current snap is 297 2024-09-11T03:48:23.042 INFO:tasks.rados.rados.2.smithi175.stdout:2701: expect (ObjNum 867 snap 273 seq_num 867) 2024-09-11T03:48:23.124 INFO:tasks.rados.rados.2.smithi175.stdout:2702: finishing write tid 1 to smithi17557870-26 2024-09-11T03:48:23.124 INFO:tasks.rados.rados.2.smithi175.stdout:2700: expect (ObjNum 924 snap 294 seq_num 924) 2024-09-11T03:48:23.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:23.153 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:23.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:23.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:23.255 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:23.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:23.278 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:23.278 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:23.283 INFO:tasks.rados.rados.2.smithi175.stdout:2702: finishing write tid 2 to smithi17557870-26 2024-09-11T03:48:23.284 INFO:tasks.rados.rados.2.smithi175.stdout:2702: finishing write tid 3 to smithi17557870-26 2024-09-11T03:48:23.284 INFO:tasks.rados.rados.2.smithi175.stdout:2700: done (4 left) 2024-09-11T03:48:23.284 INFO:tasks.rados.rados.2.smithi175.stdout:2701: done (3 left) 2024-09-11T03:48:23.284 INFO:tasks.rados.rados.2.smithi175.stdout:2704: done (2 left) 2024-09-11T03:48:23.284 INFO:tasks.rados.rados.2.smithi175.stdout:2705: copy_from oid 47 from oid 9 current snap is 297 2024-09-11T03:48:23.284 INFO:tasks.rados.rados.2.smithi175.stdout:2702: finishing write tid 4 to smithi17557870-26 2024-09-11T03:48:23.284 INFO:tasks.rados.rados.2.smithi175.stdout:2702: finishing write tid 5 to smithi17557870-26 2024-09-11T03:48:23.284 INFO:tasks.rados.rados.2.smithi175.stdout:2702: oid 26 updating version 0 to 891 2024-09-11T03:48:23.284 INFO:tasks.rados.rados.2.smithi175.stdout:2702: oid 26 updating version 891 to 892 2024-09-11T03:48:23.284 INFO:tasks.rados.rados.2.smithi175.stdout:2702: oid 26 updating version 892 to 893 2024-09-11T03:48:23.284 INFO:tasks.rados.rados.2.smithi175.stdout:2702: oid 26 updating version 893 to 894 2024-09-11T03:48:23.285 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 894 (ObjNum 931 snap 297 seq_num 931) dirty exists 2024-09-11T03:48:23.285 INFO:tasks.rados.rados.2.smithi175.stdout:2702: left oid 26 (ObjNum 931 snap 297 seq_num 931) 2024-09-11T03:48:23.285 INFO:tasks.rados.rados.2.smithi175.stdout:2703: finishing write tid 1 to smithi17557870-37 2024-09-11T03:48:23.285 INFO:tasks.rados.rados.2.smithi175.stdout:2703: finishing write tid 2 to smithi17557870-37 2024-09-11T03:48:23.285 INFO:tasks.rados.rados.2.smithi175.stdout:2703: finishing write tid 3 to smithi17557870-37 2024-09-11T03:48:23.285 INFO:tasks.rados.rados.2.smithi175.stdout:2703: finishing write tid 4 to smithi17557870-37 2024-09-11T03:48:23.285 INFO:tasks.rados.rados.2.smithi175.stdout:2703: finishing write tid 5 to smithi17557870-37 2024-09-11T03:48:23.285 INFO:tasks.rados.rados.2.smithi175.stdout:2703: oid 37 updating version 0 to 892 2024-09-11T03:48:23.285 INFO:tasks.rados.rados.2.smithi175.stdout:2703: oid 37 updating version 892 to 893 2024-09-11T03:48:23.285 INFO:tasks.rados.rados.2.smithi175.stdout:2703: oid 37 version 893 is already newer than 891 2024-09-11T03:48:23.285 INFO:tasks.rados.rados.2.smithi175.stdout:2703: oid 37 version 893 is already newer than 890 2024-09-11T03:48:23.285 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 893 (ObjNum 932 snap 297 seq_num 932) dirty exists 2024-09-11T03:48:23.285 INFO:tasks.rados.rados.2.smithi175.stdout:2703: left oid 37 (ObjNum 932 snap 297 seq_num 932) 2024-09-11T03:48:23.286 INFO:tasks.rados.rados.2.smithi175.stdout:2702: done (2 left) 2024-09-11T03:48:23.286 INFO:tasks.rados.rados.2.smithi175.stdout:2703: done (1 left) 2024-09-11T03:48:23.286 INFO:tasks.rados.rados.2.smithi175.stdout:2706: delete oid 41 current snap is 297 2024-09-11T03:48:23.286 INFO:tasks.rados.rados.2.smithi175.stdout:2705: finishing copy_from to smithi17557870-47 2024-09-11T03:48:23.286 INFO:tasks.rados.rados.2.smithi175.stdout:2705: got expected ENOENT (src dne) 2024-09-11T03:48:23.286 INFO:tasks.rados.rados.2.smithi175.stdout:2705: finishing copy_from racing read to smithi17557870-47 2024-09-11T03:48:23.287 INFO:tasks.rados.rados.2.smithi175.stdout:2705: done (1 left) 2024-09-11T03:48:23.287 INFO:tasks.rados.rados.2.smithi175.stdout:2706: done (0 left) 2024-09-11T03:48:23.287 INFO:tasks.rados.rados.2.smithi175.stdout:2707: write (excl) oid 34 current snap is 297 2024-09-11T03:48:23.287 INFO:tasks.rados.rados.2.smithi175.stdout:2707: seq_num 935 ranges {514545=552881,1830899=519314,2943557=475332} 2024-09-11T03:48:23.289 INFO:tasks.rados.rados.2.smithi175.stdout:2707: writing smithi17557870-34 from 514545 to 1067426 tid 1 2024-09-11T03:48:23.297 INFO:tasks.rados.rados.2.smithi175.stdout:2707: writing smithi17557870-34 from 1830899 to 2350213 tid 2 2024-09-11T03:48:23.303 INFO:tasks.rados.rados.2.smithi175.stdout:2707: writing smithi17557870-34 from 2943557 to 3418889 tid 3 2024-09-11T03:48:23.303 INFO:tasks.rados.rados.2.smithi175.stdout:2708: delete oid 21 current snap is 297 2024-09-11T03:48:23.303 INFO:tasks.rados.rados.2.smithi175.stdout:2707: finishing write tid 1 to smithi17557870-34 2024-09-11T03:48:23.303 INFO:tasks.rados.rados.2.smithi175.stdout:2707: finishing write tid 2 to smithi17557870-34 2024-09-11T03:48:23.305 INFO:tasks.rados.rados.2.smithi175.stdout:2708: done (1 left) 2024-09-11T03:48:23.305 INFO:tasks.rados.rados.2.smithi175.stdout:2709: read oid 15 snap -1 2024-09-11T03:48:23.305 INFO:tasks.rados.rados.2.smithi175.stdout:2709: expect (ObjNum 912 snap 290 seq_num 912) 2024-09-11T03:48:23.305 INFO:tasks.rados.rados.2.smithi175.stdout:2710: write oid 14 current snap is 297 2024-09-11T03:48:23.305 INFO:tasks.rados.rados.2.smithi175.stdout:2710: seq_num 937 ranges {548159=419732,1664216=522983,2714756=614229} 2024-09-11T03:48:23.310 INFO:tasks.rados.rados.2.smithi175.stdout:2710: writing smithi17557870-14 from 548159 to 967891 tid 1 2024-09-11T03:48:23.317 INFO:tasks.rados.rados.2.smithi175.stdout:2710: writing smithi17557870-14 from 1664216 to 2187199 tid 2 2024-09-11T03:48:23.323 INFO:tasks.rados.rados.2.smithi175.stdout:2710: writing smithi17557870-14 from 2714756 to 3328985 tid 3 2024-09-11T03:48:23.323 INFO:tasks.rados.rados.2.smithi175.stdout:2711: delete oid 43 current snap is 297 2024-09-11T03:48:23.323 INFO:tasks.rados.rados.2.smithi175.stdout:2707: finishing write tid 3 to smithi17557870-34 2024-09-11T03:48:23.323 INFO:tasks.rados.rados.2.smithi175.stdout:2707: finishing write tid 4 to smithi17557870-34 2024-09-11T03:48:23.323 INFO:tasks.rados.rados.2.smithi175.stdout:2707: finishing write tid 5 to smithi17557870-34 2024-09-11T03:48:23.324 INFO:tasks.rados.rados.2.smithi175.stdout:2707: oid 34 updating version 0 to 895 2024-09-11T03:48:23.324 INFO:tasks.rados.rados.2.smithi175.stdout:2707: oid 34 updating version 895 to 896 2024-09-11T03:48:23.324 INFO:tasks.rados.rados.2.smithi175.stdout:2707: oid 34 updating version 896 to 898 2024-09-11T03:48:23.324 INFO:tasks.rados.rados.2.smithi175.stdout:2707: oid 34 version 898 is already newer than 897 2024-09-11T03:48:23.324 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 898 (ObjNum 935 snap 297 seq_num 935) dirty exists 2024-09-11T03:48:23.324 INFO:tasks.rados.rados.2.smithi175.stdout:2707: left oid 34 (ObjNum 935 snap 297 seq_num 935) 2024-09-11T03:48:23.324 INFO:tasks.rados.rados.2.smithi175.stdout:2710: finishing write tid 1 to smithi17557870-14 2024-09-11T03:48:23.324 INFO:tasks.rados.rados.2.smithi175.stdout:2710: finishing write tid 2 to smithi17557870-14 2024-09-11T03:48:23.324 INFO:tasks.rados.rados.2.smithi175.stdout:2709: expect (ObjNum 912 snap 290 seq_num 912) 2024-09-11T03:48:23.376 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T03:48:23.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:23.403 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:23.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:23.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:23.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:23.528 INFO:tasks.rados.rados.2.smithi175.stdout:2710: finishing write tid 3 to smithi17557870-14 2024-09-11T03:48:23.528 INFO:tasks.rados.rados.2.smithi175.stdout:2710: finishing write tid 4 to smithi17557870-14 2024-09-11T03:48:23.528 INFO:tasks.rados.rados.2.smithi175.stdout:2710: finishing write tid 5 to smithi17557870-14 2024-09-11T03:48:23.528 INFO:tasks.rados.rados.2.smithi175.stdout:2710: oid 14 updating version 0 to 941 2024-09-11T03:48:23.528 INFO:tasks.rados.rados.2.smithi175.stdout:2710: oid 14 updating version 941 to 943 2024-09-11T03:48:23.528 INFO:tasks.rados.rados.2.smithi175.stdout:2710: oid 14 updating version 943 to 944 2024-09-11T03:48:23.528 INFO:tasks.rados.rados.2.smithi175.stdout:2710: oid 14 version 944 is already newer than 942 2024-09-11T03:48:23.529 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 944 (ObjNum 937 snap 297 seq_num 937) dirty exists 2024-09-11T03:48:23.529 INFO:tasks.rados.rados.2.smithi175.stdout:2710: left oid 14 (ObjNum 937 snap 297 seq_num 937) 2024-09-11T03:48:23.529 INFO:tasks.rados.rados.2.smithi175.stdout:2707: done (3 left) 2024-09-11T03:48:23.529 INFO:tasks.rados.rados.2.smithi175.stdout:2709: done (2 left) 2024-09-11T03:48:23.529 INFO:tasks.rados.rados.2.smithi175.stdout:2710: done (1 left) 2024-09-11T03:48:23.529 INFO:tasks.rados.rados.2.smithi175.stdout:2711: done (0 left) 2024-09-11T03:48:23.529 INFO:tasks.rados.rados.2.smithi175.stdout:2712: write (excl) oid 44 current snap is 297 2024-09-11T03:48:23.529 INFO:tasks.rados.rados.2.smithi175.stdout:2712: seq_num 939 ranges {581773=570230,1664827=526652,2653249=636773} 2024-09-11T03:48:23.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:23.530 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:23.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:23.533 INFO:tasks.rados.rados.2.smithi175.stdout:2712: writing smithi17557870-44 from 581773 to 1152003 tid 1 2024-09-11T03:48:23.539 INFO:tasks.rados.rados.2.smithi175.stdout:2712: writing smithi17557870-44 from 1664827 to 2191479 tid 2 2024-09-11T03:48:23.545 INFO:tasks.rados.rados.2.smithi175.stdout:2712: writing smithi17557870-44 from 2653249 to 3290022 tid 3 2024-09-11T03:48:23.545 INFO:tasks.rados.rados.2.smithi175.stdout:2713: write (excl) oid 16 current snap is 297 2024-09-11T03:48:23.545 INFO:tasks.rados.rados.2.smithi175.stdout:2713: seq_num 940 ranges {598580=645479,1723309=786663,3080672=448045} 2024-09-11T03:48:23.552 INFO:tasks.rados.rados.2.smithi175.stdout:2713: writing smithi17557870-16 from 598580 to 1244059 tid 1 2024-09-11T03:48:23.558 INFO:tasks.rados.rados.2.smithi175.stdout:2713: writing smithi17557870-16 from 1723309 to 2509972 tid 2 2024-09-11T03:48:23.564 INFO:tasks.rados.rados.2.smithi175.stdout:2713: writing smithi17557870-16 from 3080672 to 3528717 tid 3 2024-09-11T03:48:23.564 INFO:tasks.rados.rados.2.smithi175.stdout:2714: write (excl) oid 38 current snap is 297 2024-09-11T03:48:23.564 INFO:tasks.rados.rados.2.smithi175.stdout:2714: seq_num 941 ranges {615387=720728,1781791=530321,3108095=707292} 2024-09-11T03:48:23.571 INFO:tasks.rados.rados.2.smithi175.stdout:2714: writing smithi17557870-38 from 615387 to 1336115 tid 1 2024-09-11T03:48:23.576 INFO:tasks.rados.rados.2.smithi175.stdout:2714: writing smithi17557870-38 from 1781791 to 2312112 tid 2 2024-09-11T03:48:23.585 INFO:tasks.rados.rados.2.smithi175.stdout:2714: writing smithi17557870-38 from 3108095 to 3815387 tid 3 2024-09-11T03:48:23.585 INFO:tasks.rados.rados.2.smithi175.stdout:2715: snap_remove snap 291 2024-09-11T03:48:23.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:23.652 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:23.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:23.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:23.754 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:23.776 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:23.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:23.777 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:23.808 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:48:23.808 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:48:23.829 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:23.829 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:48:23.830 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:48:23.846 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:48:23.847 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:48:23.863 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:48:23.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:48:23.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:23.880 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:48:23.881 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:48:23.897 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:48:23.897 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:48:23.902 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:48:23.902 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:23.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:23.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:23.915 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:48:23.915 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:48:23.938 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:48:23.938 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:48:23.955 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:48:23.955 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:48:23.973 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:48:23.973 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:48:23.990 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:48:23.990 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:48:24.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:24.008 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:48:24.008 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:48:24.026 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:24.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:24.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:24.035 INFO:tasks.rados.rados.2.smithi175.stdout:2715: done (3 left) 2024-09-11T03:48:24.035 INFO:tasks.rados.rados.2.smithi175.stdout:2716: read oid 15 snap 286 2024-09-11T03:48:24.035 INFO:tasks.rados.rados.2.smithi175.stdout:2716: expect (ObjNum 857 snap 269 seq_num 857) 2024-09-11T03:48:24.035 INFO:tasks.rados.rados.2.smithi175.stdout:2717: write oid 28 current snap is 297 2024-09-11T03:48:24.035 INFO:tasks.rados.rados.2.smithi175.stdout:2712: finishing write tid 1 to smithi17557870-44 2024-09-11T03:48:24.035 INFO:tasks.rados.rados.2.smithi175.stdout:2712: finishing write tid 2 to smithi17557870-44 2024-09-11T03:48:24.035 INFO:tasks.rados.rados.2.smithi175.stdout:2712: finishing write tid 3 to smithi17557870-44 2024-09-11T03:48:24.035 INFO:tasks.rados.rados.2.smithi175.stdout:2712: finishing write tid 4 to smithi17557870-44 2024-09-11T03:48:24.035 INFO:tasks.rados.rados.2.smithi175.stdout:2717: seq_num 942 ranges {632194=795977,2123920=790332,3419165=413029} 2024-09-11T03:48:24.043 INFO:tasks.rados.rados.2.smithi175.stdout:2717: writing smithi17557870-28 from 632194 to 1428171 tid 1 2024-09-11T03:48:24.051 INFO:tasks.rados.rados.2.smithi175.stdout:2717: writing smithi17557870-28 from 2123920 to 2914252 tid 2 2024-09-11T03:48:24.057 INFO:tasks.rados.rados.2.smithi175.stdout:2717: writing smithi17557870-28 from 3419165 to 3832194 tid 3 2024-09-11T03:48:24.057 INFO:tasks.rados.rados.2.smithi175.stdout:2718: read oid 42 snap -1 2024-09-11T03:48:24.057 INFO:tasks.rados.rados.2.smithi175.stdout:2718: expect deleted 2024-09-11T03:48:24.057 INFO:tasks.rados.rados.2.smithi175.stdout:2719: read oid 27 snap -1 2024-09-11T03:48:24.057 INFO:tasks.rados.rados.2.smithi175.stdout:2719: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:48:24.057 INFO:tasks.rados.rados.2.smithi175.stdout:2720: read oid 20 snap -1 2024-09-11T03:48:24.057 INFO:tasks.rados.rados.2.smithi175.stdout:2720: expect (ObjNum 775 snap 248 seq_num 775) 2024-09-11T03:48:24.057 INFO:tasks.rados.rados.2.smithi175.stdout:2721: write oid 29 current snap is 297 2024-09-11T03:48:24.058 INFO:tasks.rados.rados.2.smithi175.stdout:2712: finishing write tid 5 to smithi17557870-44 2024-09-11T03:48:24.058 INFO:tasks.rados.rados.2.smithi175.stdout:2712: oid 44 updating version 0 to 815 2024-09-11T03:48:24.058 INFO:tasks.rados.rados.2.smithi175.stdout:2712: oid 44 updating version 815 to 818 2024-09-11T03:48:24.058 INFO:tasks.rados.rados.2.smithi175.stdout:2712: oid 44 version 818 is already newer than 816 2024-09-11T03:48:24.058 INFO:tasks.rados.rados.2.smithi175.stdout:2712: oid 44 version 818 is already newer than 817 2024-09-11T03:48:24.058 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 818 (ObjNum 939 snap 297 seq_num 939) dirty exists 2024-09-11T03:48:24.058 INFO:tasks.rados.rados.2.smithi175.stdout:2712: left oid 44 (ObjNum 939 snap 297 seq_num 939) 2024-09-11T03:48:24.058 INFO:tasks.rados.rados.2.smithi175.stdout:2713: finishing write tid 1 to smithi17557870-16 2024-09-11T03:48:24.058 INFO:tasks.rados.rados.2.smithi175.stdout:2713: finishing write tid 2 to smithi17557870-16 2024-09-11T03:48:24.058 INFO:tasks.rados.rados.2.smithi175.stdout:2721: seq_num 943 ranges {649001=587579,1898755=533990,3162941=686060} 2024-09-11T03:48:24.063 INFO:tasks.rados.rados.2.smithi175.stdout:2721: writing smithi17557870-29 from 649001 to 1236580 tid 1 2024-09-11T03:48:24.070 INFO:tasks.rados.rados.2.smithi175.stdout:2721: writing smithi17557870-29 from 1898755 to 2432745 tid 2 2024-09-11T03:48:24.079 INFO:tasks.rados.rados.2.smithi175.stdout:2721: writing smithi17557870-29 from 3162941 to 3849001 tid 3 2024-09-11T03:48:24.079 INFO:tasks.rados.rados.2.smithi175.stdout:2712: done (8 left) 2024-09-11T03:48:24.079 INFO:tasks.rados.rados.2.smithi175.stdout:2722: rollback oid 22 current snap is 297 2024-09-11T03:48:24.079 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 22 to 279 2024-09-11T03:48:24.079 INFO:tasks.rados.rados.2.smithi175.stdout:2713: finishing write tid 3 to smithi17557870-16 2024-09-11T03:48:24.079 INFO:tasks.rados.rados.2.smithi175.stdout:2723: snap_create 2024-09-11T03:48:24.079 INFO:tasks.rados.rados.2.smithi175.stdout:2713: finishing write tid 4 to smithi17557870-16 2024-09-11T03:48:24.079 INFO:tasks.rados.rados.2.smithi175.stdout:2713: finishing write tid 5 to smithi17557870-16 2024-09-11T03:48:24.080 INFO:tasks.rados.rados.2.smithi175.stdout:2713: oid 16 updating version 0 to 864 2024-09-11T03:48:24.080 INFO:tasks.rados.rados.2.smithi175.stdout:2713: oid 16 version 864 is already newer than 861 2024-09-11T03:48:24.080 INFO:tasks.rados.rados.2.smithi175.stdout:2713: oid 16 version 864 is already newer than 863 2024-09-11T03:48:24.080 INFO:tasks.rados.rados.2.smithi175.stdout:2713: oid 16 version 864 is already newer than 862 2024-09-11T03:48:24.080 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 864 (ObjNum 940 snap 297 seq_num 940) dirty exists 2024-09-11T03:48:24.080 INFO:tasks.rados.rados.2.smithi175.stdout:2713: left oid 16 (ObjNum 940 snap 297 seq_num 940) 2024-09-11T03:48:24.080 INFO:tasks.rados.rados.2.smithi175.stdout:2714: finishing write tid 1 to smithi17557870-38 2024-09-11T03:48:24.080 INFO:tasks.rados.rados.2.smithi175.stdout:2714: finishing write tid 2 to smithi17557870-38 2024-09-11T03:48:24.080 INFO:tasks.rados.rados.2.smithi175.stdout:2714: finishing write tid 3 to smithi17557870-38 2024-09-11T03:48:24.080 INFO:tasks.rados.rados.2.smithi175.stdout:2714: finishing write tid 4 to smithi17557870-38 2024-09-11T03:48:24.080 INFO:tasks.rados.rados.2.smithi175.stdout:2714: finishing write tid 5 to smithi17557870-38 2024-09-11T03:48:24.080 INFO:tasks.rados.rados.2.smithi175.stdout:2714: oid 38 updating version 0 to 958 2024-09-11T03:48:24.080 INFO:tasks.rados.rados.2.smithi175.stdout:2714: oid 38 updating version 958 to 960 2024-09-11T03:48:24.081 INFO:tasks.rados.rados.2.smithi175.stdout:2714: oid 38 version 960 is already newer than 959 2024-09-11T03:48:24.081 INFO:tasks.rados.rados.2.smithi175.stdout:2714: oid 38 version 960 is already newer than 957 2024-09-11T03:48:24.081 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 960 (ObjNum 941 snap 297 seq_num 941) dirty exists 2024-09-11T03:48:24.081 INFO:tasks.rados.rados.2.smithi175.stdout:2714: left oid 38 (ObjNum 941 snap 297 seq_num 941) 2024-09-11T03:48:24.081 INFO:tasks.rados.rados.2.smithi175.stdout:2717: finishing write tid 1 to smithi17557870-28 2024-09-11T03:48:24.081 INFO:tasks.rados.rados.2.smithi175.stdout:2716: expect (ObjNum 857 snap 269 seq_num 857) 2024-09-11T03:48:24.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:24.151 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:24.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:24.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:24.210 INFO:tasks.rados.rados.2.smithi175.stdout:2717: finishing write tid 2 to smithi17557870-28 2024-09-11T03:48:24.210 INFO:tasks.rados.rados.2.smithi175.stdout:2717: finishing write tid 3 to smithi17557870-28 2024-09-11T03:48:24.210 INFO:tasks.rados.rados.2.smithi175.stdout:2717: finishing write tid 4 to smithi17557870-28 2024-09-11T03:48:24.211 INFO:tasks.rados.rados.2.smithi175.stdout:2717: finishing write tid 5 to smithi17557870-28 2024-09-11T03:48:24.211 INFO:tasks.rados.rados.2.smithi175.stdout:2717: oid 28 updating version 0 to 1014 2024-09-11T03:48:24.211 INFO:tasks.rados.rados.2.smithi175.stdout:2717: oid 28 updating version 1014 to 1015 2024-09-11T03:48:24.211 INFO:tasks.rados.rados.2.smithi175.stdout:2717: oid 28 updating version 1015 to 1016 2024-09-11T03:48:24.211 INFO:tasks.rados.rados.2.smithi175.stdout:2717: oid 28 updating version 1016 to 1017 2024-09-11T03:48:24.211 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 1017 (ObjNum 942 snap 297 seq_num 942) dirty exists 2024-09-11T03:48:24.211 INFO:tasks.rados.rados.2.smithi175.stdout:2717: left oid 28 (ObjNum 942 snap 297 seq_num 942) 2024-09-11T03:48:24.212 INFO:tasks.rados.rados.2.smithi175.stdout:2721: finishing write tid 1 to smithi17557870-29 2024-09-11T03:48:24.212 INFO:tasks.rados.rados.2.smithi175.stdout:2719: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:48:24.252 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:24.274 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:24.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:24.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:24.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:24.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:23 smithi084 ceph-mon[27254]: pgmap v1182: 67 pgs: 67 active+clean; 225 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.6 MiB/s wr, 24 op/s 2024-09-11T03:48:24.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:23 smithi084 ceph-mon[27254]: osdmap e685: 12 total, 12 up, 12 in 2024-09-11T03:48:24.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:23 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/631147494' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:24.397 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:24.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:24.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:23 smithi121 ceph-mon[30729]: pgmap v1182: 67 pgs: 67 active+clean; 225 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.6 MiB/s wr, 24 op/s 2024-09-11T03:48:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:23 smithi121 ceph-mon[30729]: osdmap e685: 12 total, 12 up, 12 in 2024-09-11T03:48:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:23 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/631147494' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:24.442 INFO:tasks.rados.rados.2.smithi175.stdout:2721: finishing write tid 2 to smithi17557870-29 2024-09-11T03:48:24.442 INFO:tasks.rados.rados.2.smithi175.stdout:2720: expect (ObjNum 775 snap 248 seq_num 775) 2024-09-11T03:48:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:23 smithi175 ceph-mon[30894]: pgmap v1182: 67 pgs: 67 active+clean; 225 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.6 MiB/s wr, 24 op/s 2024-09-11T03:48:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:23 smithi175 ceph-mon[30894]: osdmap e685: 12 total, 12 up, 12 in 2024-09-11T03:48:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:23 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/631147494' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:24.499 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:24.520 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:24.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:24.521 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:24.621 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:24.642 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:24.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:24.643 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:24.668 INFO:tasks.rados.rados.2.smithi175.stdout:2722: finishing rollback tid 0 to smithi17557870-22 2024-09-11T03:48:24.668 INFO:tasks.rados.rados.2.smithi175.stdout:2722: finishing rollback tid 1 to smithi17557870-22 2024-09-11T03:48:24.668 INFO:tasks.rados.rados.2.smithi175.stdout:2721: finishing write tid 3 to smithi17557870-29 2024-09-11T03:48:24.668 INFO:tasks.rados.rados.2.smithi175.stdout:2721: finishing write tid 4 to smithi17557870-29 2024-09-11T03:48:24.668 INFO:tasks.rados.rados.2.smithi175.stdout:2721: finishing write tid 5 to smithi17557870-29 2024-09-11T03:48:24.668 INFO:tasks.rados.rados.2.smithi175.stdout:2721: oid 29 updating version 0 to 965 2024-09-11T03:48:24.669 INFO:tasks.rados.rados.2.smithi175.stdout:2721: oid 29 version 965 is already newer than 964 2024-09-11T03:48:24.669 INFO:tasks.rados.rados.2.smithi175.stdout:2721: oid 29 updating version 965 to 966 2024-09-11T03:48:24.669 INFO:tasks.rados.rados.2.smithi175.stdout:2721: oid 29 updating version 966 to 967 2024-09-11T03:48:24.669 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 967 (ObjNum 943 snap 297 seq_num 943) dirty exists 2024-09-11T03:48:24.669 INFO:tasks.rados.rados.2.smithi175.stdout:2721: left oid 29 (ObjNum 943 snap 297 seq_num 943) 2024-09-11T03:48:24.669 INFO:tasks.rados.rados.2.smithi175.stdout:2722: finishing rollback tid 2 to smithi17557870-22 2024-09-11T03:48:24.669 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 1021 (ObjNum 845 snap 266 seq_num 845) dirty exists 2024-09-11T03:48:24.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:24.766 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:24.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:24.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:24.867 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:24.890 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:24.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:24.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:24.992 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:25.014 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:25.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:25.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:25.038 INFO:tasks.rados.rados.2.smithi175.stdout:2713: done (9 left) 2024-09-11T03:48:25.038 INFO:tasks.rados.rados.2.smithi175.stdout:2714: done (8 left) 2024-09-11T03:48:25.038 INFO:tasks.rados.rados.2.smithi175.stdout:2716: done (7 left) 2024-09-11T03:48:25.038 INFO:tasks.rados.rados.2.smithi175.stdout:2717: done (6 left) 2024-09-11T03:48:25.039 INFO:tasks.rados.rados.2.smithi175.stdout:2718: done (5 left) 2024-09-11T03:48:25.039 INFO:tasks.rados.rados.2.smithi175.stdout:2719: done (4 left) 2024-09-11T03:48:25.039 INFO:tasks.rados.rados.2.smithi175.stdout:2720: done (3 left) 2024-09-11T03:48:25.039 INFO:tasks.rados.rados.2.smithi175.stdout:2721: done (2 left) 2024-09-11T03:48:25.039 INFO:tasks.rados.rados.2.smithi175.stdout:2722: done (1 left) 2024-09-11T03:48:25.039 INFO:tasks.rados.rados.2.smithi175.stdout:2723: done (0 left) 2024-09-11T03:48:25.039 INFO:tasks.rados.rados.2.smithi175.stdout:2724: snap_remove snap 295 2024-09-11T03:48:25.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:25 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:48:25.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:25 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:48:25.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:25 smithi084 ceph-mon[27254]: osdmap e686: 12 total, 12 up, 12 in 2024-09-11T03:48:25.053 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:48:25.053 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:48:25.053 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:48:25.053 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:48:25.116 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:25.138 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:25.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:25.139 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:25.240 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:25.261 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:25.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:25.317 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:25.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:25.387 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:25.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:25.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:25.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:25 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:48:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:25 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:48:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:25 smithi121 ceph-mon[30729]: osdmap e686: 12 total, 12 up, 12 in 2024-09-11T03:48:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:25 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:48:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:25 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:48:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:25 smithi175 ceph-mon[30894]: osdmap e686: 12 total, 12 up, 12 in 2024-09-11T03:48:25.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:25.511 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:25.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:25.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:25.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:25.635 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:25.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:25.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:25.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:25.760 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:25.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:25.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:25.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T03:48:25.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:25.884 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:25.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:25.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:25.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:26.008 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:26.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:26.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:26.041 INFO:tasks.rados.rados.2.smithi175.stdout:2724: done (0 left) 2024-09-11T03:48:26.041 INFO:tasks.rados.rados.2.smithi175.stdout:2725: copy_from oid 22 from oid 12 current snap is 298 2024-09-11T03:48:26.041 INFO:tasks.rados.rados.2.smithi175.stdout:2726: read oid 1 snap -1 2024-09-11T03:48:26.041 INFO:tasks.rados.rados.2.smithi175.stdout:2726: expect (ObjNum 758 snap 246 seq_num 758) 2024-09-11T03:48:26.041 INFO:tasks.rados.rados.2.smithi175.stdout:2727: read oid 39 snap 284 2024-09-11T03:48:26.041 INFO:tasks.rados.rados.2.smithi175.stdout:2727: expect deleted 2024-09-11T03:48:26.041 INFO:tasks.rados.rados.2.smithi175.stdout:2728: snap_create 2024-09-11T03:48:26.049 INFO:tasks.rados.rados.2.smithi175.stdout:2726: expect (ObjNum 758 snap 246 seq_num 758) 2024-09-11T03:48:26.110 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:26.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:26.132 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:26.132 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:26.143 INFO:tasks.rados.rados.2.smithi175.stdout:2725: finishing copy_from racing read to smithi17557870-22 2024-09-11T03:48:26.143 INFO:tasks.rados.rados.2.smithi175.stdout:2725: finishing copy_from to smithi17557870-22 2024-09-11T03:48:26.143 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 930 (ObjNum 829 snap 263 seq_num 829) dirty exists 2024-09-11T03:48:26.233 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:26.255 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:26.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:26.256 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:26.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:26 smithi084 ceph-mon[27254]: pgmap v1185: 67 pgs: 67 active+clean; 224 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.6 MiB/s wr, 25 op/s 2024-09-11T03:48:26.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:26 smithi084 ceph-mon[27254]: osdmap e687: 12 total, 12 up, 12 in 2024-09-11T03:48:26.334 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:26.356 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:26.378 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:26.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:26.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:26 smithi121 ceph-mon[30729]: pgmap v1185: 67 pgs: 67 active+clean; 224 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.6 MiB/s wr, 25 op/s 2024-09-11T03:48:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:26 smithi121 ceph-mon[30729]: osdmap e687: 12 total, 12 up, 12 in 2024-09-11T03:48:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:26 smithi175 ceph-mon[30894]: pgmap v1185: 67 pgs: 67 active+clean; 224 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.6 MiB/s wr, 25 op/s 2024-09-11T03:48:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:26 smithi175 ceph-mon[30894]: osdmap e687: 12 total, 12 up, 12 in 2024-09-11T03:48:26.479 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:26.501 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:26.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:26.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:26.603 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:26.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:26.629 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:26.629 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:26.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:26.752 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:26.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:26.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:26.767 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:48:26.767 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:48:26.768 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:48:26.768 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:48:26.768 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:48:26.832 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:48:26.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:26.876 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:26.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:26.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:26.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:26.992 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:48:26.997 INFO:tasks.rados.rados.2.smithi175.stdout:2725: done (3 left) 2024-09-11T03:48:26.997 INFO:tasks.rados.rados.2.smithi175.stdout:2726: done (2 left) 2024-09-11T03:48:26.997 INFO:tasks.rados.rados.2.smithi175.stdout:2727: done (1 left) 2024-09-11T03:48:26.997 INFO:tasks.rados.rados.2.smithi175.stdout:2728: done (0 left) 2024-09-11T03:48:26.997 INFO:tasks.rados.rados.2.smithi175.stdout:2729: snap_remove snap 274 2024-09-11T03:48:27.001 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:27.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:27.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:27.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:27.125 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:27.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:27.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:27 smithi084 ceph-mon[27254]: osdmap e688: 12 total, 12 up, 12 in 2024-09-11T03:48:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:27 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3086453186' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:48:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:27 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:48:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:27 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:48:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:27 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3086453186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:48:27.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:27 smithi084 ceph-mon[27254]: osdmap e689: 12 total, 12 up, 12 in 2024-09-11T03:48:27.227 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:27.249 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:27.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:27.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:27.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:27.396 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:27.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:27.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:27.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:27 smithi121 ceph-mon[30729]: osdmap e688: 12 total, 12 up, 12 in 2024-09-11T03:48:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:27 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3086453186' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:48:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:27 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:48:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:27 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:48:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:27 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3086453186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:48:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:27 smithi121 ceph-mon[30729]: osdmap e689: 12 total, 12 up, 12 in 2024-09-11T03:48:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:27 smithi175 ceph-mon[30894]: osdmap e688: 12 total, 12 up, 12 in 2024-09-11T03:48:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:27 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3086453186' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:48:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:27 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:48:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:27 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:48:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:27 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3086453186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:48:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:27 smithi175 ceph-mon[30894]: osdmap e689: 12 total, 12 up, 12 in 2024-09-11T03:48:27.498 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:27.524 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:27.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:27.525 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:27.626 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:27.652 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:27.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:27.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:27.680 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:48:27.681 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:48:27.681 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T03:48:27.681 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-11T03:48:27.734 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T03:48:27.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:27.779 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:27.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:27.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:27.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:27.907 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:27.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:27.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:27.999 INFO:tasks.rados.rados.2.smithi175.stdout:2729: done (0 left) 2024-09-11T03:48:27.999 INFO:tasks.rados.rados.2.smithi175.stdout:2730: delete oid 19 current snap is 299 2024-09-11T03:48:28.006 INFO:tasks.rados.rados.2.smithi175.stdout:2730: done (0 left) 2024-09-11T03:48:28.006 INFO:tasks.rados.rados.2.smithi175.stdout:2731: read oid 39 snap -1 2024-09-11T03:48:28.006 INFO:tasks.rados.rados.2.smithi175.stdout:2731: expect deleted 2024-09-11T03:48:28.006 INFO:tasks.rados.rados.2.smithi175.stdout:2732: snap_create 2024-09-11T03:48:28.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:28.032 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:28.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:28.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:28.134 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:28.159 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:28.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:28.161 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:28.222 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:28.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:28.282 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:28.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:28.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:28.344 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:28 smithi084 ceph-mon[27254]: pgmap v1188: 67 pgs: 67 active+clean; 251 MiB data, 969 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.9 MiB/s wr, 25 op/s 2024-09-11T03:48:28.344 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:28 smithi084 ceph-mon[27254]: osdmap e690: 12 total, 12 up, 12 in 2024-09-11T03:48:28.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:28.411 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:28.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:28.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:28.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:28 smithi121 ceph-mon[30729]: pgmap v1188: 67 pgs: 67 active+clean; 251 MiB data, 969 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.9 MiB/s wr, 25 op/s 2024-09-11T03:48:28.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:28 smithi121 ceph-mon[30729]: osdmap e690: 12 total, 12 up, 12 in 2024-09-11T03:48:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:28 smithi175 ceph-mon[30894]: pgmap v1188: 67 pgs: 67 active+clean; 251 MiB data, 969 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.9 MiB/s wr, 25 op/s 2024-09-11T03:48:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:28 smithi175 ceph-mon[30894]: osdmap e690: 12 total, 12 up, 12 in 2024-09-11T03:48:28.513 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:28.533 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:28.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:28.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:28.635 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:28.661 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:28.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:28.662 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:28.733 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:48:28.734 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:48:28.738 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:28.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:48:28.749 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:48:28.763 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:28.768 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:48:28.768 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:48:28.786 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:48:28.786 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:48:28.788 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:28.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:28.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:28.803 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:48:28.804 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:48:28.821 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:48:28.821 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:48:28.837 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:48:28.837 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:48:28.853 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:48:28.854 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:48:28.871 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:48:28.871 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:48:28.892 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:28.893 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:48:28.893 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:48:28.906 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:48:28.906 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:48:28.918 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:28.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:28.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:28.923 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:48:28.923 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:48:29.001 INFO:tasks.rados.rados.2.smithi175.stdout:2731: done (1 left) 2024-09-11T03:48:29.001 INFO:tasks.rados.rados.2.smithi175.stdout:2732: done (0 left) 2024-09-11T03:48:29.001 INFO:tasks.rados.rados.2.smithi175.stdout:2733: write oid 26 current snap is 300 2024-09-11T03:48:29.002 INFO:tasks.rados.rados.2.smithi175.stdout:2733: seq_num 945 ranges {682615=738077,2015719=537659,3217787=420758} 2024-09-11T03:48:29.008 INFO:tasks.rados.rados.2.smithi175.stdout:2733: writing smithi17557870-26 from 682615 to 1420692 tid 1 2024-09-11T03:48:29.015 INFO:tasks.rados.rados.2.smithi175.stdout:2733: writing smithi17557870-26 from 2015719 to 2553378 tid 2 2024-09-11T03:48:29.020 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:29.021 INFO:tasks.rados.rados.2.smithi175.stdout:2733: writing smithi17557870-26 from 3217787 to 3638545 tid 3 2024-09-11T03:48:29.021 INFO:tasks.rados.rados.2.smithi175.stdout:2734: snap_create 2024-09-11T03:48:29.022 INFO:tasks.rados.rados.2.smithi175.stdout:2733: finishing write tid 1 to smithi17557870-26 2024-09-11T03:48:29.022 INFO:tasks.rados.rados.2.smithi175.stdout:2733: finishing write tid 2 to smithi17557870-26 2024-09-11T03:48:29.024 INFO:tasks.rados.rados.2.smithi175.stdout:2733: finishing write tid 3 to smithi17557870-26 2024-09-11T03:48:29.024 INFO:tasks.rados.rados.2.smithi175.stdout:2733: finishing write tid 4 to smithi17557870-26 2024-09-11T03:48:29.024 INFO:tasks.rados.rados.2.smithi175.stdout:2733: finishing write tid 5 to smithi17557870-26 2024-09-11T03:48:29.024 INFO:tasks.rados.rados.2.smithi175.stdout:2733: oid 26 updating version 0 to 902 2024-09-11T03:48:29.024 INFO:tasks.rados.rados.2.smithi175.stdout:2733: oid 26 version 902 is already newer than 901 2024-09-11T03:48:29.024 INFO:tasks.rados.rados.2.smithi175.stdout:2733: oid 26 version 902 is already newer than 900 2024-09-11T03:48:29.024 INFO:tasks.rados.rados.2.smithi175.stdout:2733: oid 26 version 902 is already newer than 899 2024-09-11T03:48:29.024 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 902 (ObjNum 945 snap 300 seq_num 945) dirty exists 2024-09-11T03:48:29.024 INFO:tasks.rados.rados.2.smithi175.stdout:2733: left oid 26 (ObjNum 945 snap 300 seq_num 945) 2024-09-11T03:48:29.042 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:29.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:29.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:29.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:29.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:29 smithi084 ceph-mon[27254]: 2.17 scrub starts 2024-09-11T03:48:29.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:29 smithi084 ceph-mon[27254]: 2.17 scrub ok 2024-09-11T03:48:29.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:29 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/421709707' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:29.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:29 smithi084 ceph-mon[27254]: osdmap e691: 12 total, 12 up, 12 in 2024-09-11T03:48:29.170 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:29.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:29.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:29.272 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:29.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:29.301 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:29.301 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:29.403 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:29.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:29 smithi121 ceph-mon[30729]: 2.17 scrub starts 2024-09-11T03:48:29.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:29 smithi121 ceph-mon[30729]: 2.17 scrub ok 2024-09-11T03:48:29.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:29 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/421709707' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:29.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:29 smithi121 ceph-mon[30729]: osdmap e691: 12 total, 12 up, 12 in 2024-09-11T03:48:29.428 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:29.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:29.429 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:29 smithi175 ceph-mon[30894]: 2.17 scrub starts 2024-09-11T03:48:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:29 smithi175 ceph-mon[30894]: 2.17 scrub ok 2024-09-11T03:48:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:29 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/421709707' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:29 smithi175 ceph-mon[30894]: osdmap e691: 12 total, 12 up, 12 in 2024-09-11T03:48:29.530 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:29.551 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:29.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:29.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:29.653 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:29.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:29.675 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:29.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:29.776 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:29.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:29.799 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:29.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:29.899 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:29.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:29.921 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:29.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:30.003 INFO:tasks.rados.rados.2.smithi175.stdout:2733: done (1 left) 2024-09-11T03:48:30.004 INFO:tasks.rados.rados.2.smithi175.stdout:2734: done (0 left) 2024-09-11T03:48:30.004 INFO:tasks.rados.rados.2.smithi175.stdout:2735: read oid 27 snap -1 2024-09-11T03:48:30.004 INFO:tasks.rados.rados.2.smithi175.stdout:2735: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:48:30.004 INFO:tasks.rados.rados.2.smithi175.stdout:2736: rollback oid 14 current snap is 301 2024-09-11T03:48:30.004 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 14 to 285 2024-09-11T03:48:30.004 INFO:tasks.rados.rados.2.smithi175.stdout:2737: rollback oid 40 current snap is 301 2024-09-11T03:48:30.004 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 40 to 286 2024-09-11T03:48:30.004 INFO:tasks.rados.rados.2.smithi175.stdout:2738: delete oid 45 current snap is 301 2024-09-11T03:48:30.007 INFO:tasks.rados.rados.2.smithi175.stdout:2736: finishing rollback tid 0 to smithi17557870-14 2024-09-11T03:48:30.008 INFO:tasks.rados.rados.2.smithi175.stdout:2736: finishing rollback tid 1 to smithi17557870-14 2024-09-11T03:48:30.008 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 953 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-11T03:48:30.008 INFO:tasks.rados.rados.2.smithi175.stdout:2737: finishing rollback tid 0 to smithi17557870-40 2024-09-11T03:48:30.009 INFO:tasks.rados.rados.2.smithi175.stdout:2736: done (3 left) 2024-09-11T03:48:30.009 INFO:tasks.rados.rados.2.smithi175.stdout:2738: done (2 left) 2024-09-11T03:48:30.009 INFO:tasks.rados.rados.2.smithi175.stdout:2739: read oid 6 snap -1 2024-09-11T03:48:30.009 INFO:tasks.rados.rados.2.smithi175.stdout:2739: expect (ObjNum 857 snap 269 seq_num 857) 2024-09-11T03:48:30.009 INFO:tasks.rados.rados.2.smithi175.stdout:2737: finishing rollback tid 1 to smithi17557870-40 2024-09-11T03:48:30.009 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 492 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T03:48:30.009 INFO:tasks.rados.rados.2.smithi175.stdout:2737: done (2 left) 2024-09-11T03:48:30.009 INFO:tasks.rados.rados.2.smithi175.stdout:2740: write (excl) oid 3 current snap is 301 2024-09-11T03:48:30.009 INFO:tasks.rados.rados.2.smithi175.stdout:2740: seq_num 947 ranges {716229=488575,1616330=541328,2756280=559655} 2024-09-11T03:48:30.015 INFO:tasks.rados.rados.2.smithi175.stdout:2740: writing smithi17557870-3 from 716229 to 1204804 tid 1 2024-09-11T03:48:30.020 INFO:tasks.rados.rados.2.smithi175.stdout:2740: writing smithi17557870-3 from 1616330 to 2157658 tid 2 2024-09-11T03:48:30.022 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:30.027 INFO:tasks.rados.rados.2.smithi175.stdout:2740: writing smithi17557870-3 from 2756280 to 3315935 tid 3 2024-09-11T03:48:30.027 INFO:tasks.rados.rados.2.smithi175.stdout:2741: read oid 50 snap -1 2024-09-11T03:48:30.027 INFO:tasks.rados.rados.2.smithi175.stdout:2741: expect deleted 2024-09-11T03:48:30.027 INFO:tasks.rados.rados.2.smithi175.stdout:2742: delete oid 49 current snap is 301 2024-09-11T03:48:30.028 INFO:tasks.rados.rados.2.smithi175.stdout:2735: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:48:30.045 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:30.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:30.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:30.073 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:30.098 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-11T03:48:30.098 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-11T03:48:30.098 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-11T03:48:30.098 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-11T03:48:30.098 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-11T03:48:30.098 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-11T03:48:30.098 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-11T03:48:30.098 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-11T03:48:30.098 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-11T03:48:30.098 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-11T03:48:30.099 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-11T03:48:30.099 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-11T03:48:30.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:30 smithi084 ceph-mon[27254]: pgmap v1191: 67 pgs: 67 active+clean; 251 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 4.9 MiB/s wr, 31 op/s 2024-09-11T03:48:30.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:30 smithi084 ceph-mon[27254]: 2.5 scrub starts 2024-09-11T03:48:30.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:30 smithi084 ceph-mon[27254]: 2.5 scrub ok 2024-09-11T03:48:30.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:30 smithi084 ceph-mon[27254]: osdmap e692: 12 total, 12 up, 12 in 2024-09-11T03:48:30.147 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:30.168 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:30.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:30.169 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:30.219 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:48:30.269 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:30.271 INFO:tasks.rados.rados.2.smithi175.stdout:2740: finishing write tid 1 to smithi17557870-3 2024-09-11T03:48:30.271 INFO:tasks.rados.rados.2.smithi175.stdout:2740: finishing write tid 2 to smithi17557870-3 2024-09-11T03:48:30.271 INFO:tasks.rados.rados.2.smithi175.stdout:2739: expect (ObjNum 857 snap 269 seq_num 857) 2024-09-11T03:48:30.289 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:30.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:30.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:30.391 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:30.404 INFO:tasks.rados.rados.2.smithi175.stdout:2740: finishing write tid 3 to smithi17557870-3 2024-09-11T03:48:30.404 INFO:tasks.rados.rados.2.smithi175.stdout:2740: finishing write tid 4 to smithi17557870-3 2024-09-11T03:48:30.404 INFO:tasks.rados.rados.2.smithi175.stdout:2740: finishing write tid 5 to smithi17557870-3 2024-09-11T03:48:30.404 INFO:tasks.rados.rados.2.smithi175.stdout:2740: oid 3 updating version 0 to 1038 2024-09-11T03:48:30.404 INFO:tasks.rados.rados.2.smithi175.stdout:2740: oid 3 updating version 1038 to 1040 2024-09-11T03:48:30.404 INFO:tasks.rados.rados.2.smithi175.stdout:2740: oid 3 version 1040 is already newer than 1037 2024-09-11T03:48:30.405 INFO:tasks.rados.rados.2.smithi175.stdout:2740: oid 3 version 1040 is already newer than 1039 2024-09-11T03:48:30.405 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 1040 (ObjNum 947 snap 301 seq_num 947) dirty exists 2024-09-11T03:48:30.405 INFO:tasks.rados.rados.2.smithi175.stdout:2740: left oid 3 (ObjNum 947 snap 301 seq_num 947) 2024-09-11T03:48:30.405 INFO:tasks.rados.rados.2.smithi175.stdout:2735: done (4 left) 2024-09-11T03:48:30.405 INFO:tasks.rados.rados.2.smithi175.stdout:2739: done (3 left) 2024-09-11T03:48:30.405 INFO:tasks.rados.rados.2.smithi175.stdout:2740: done (2 left) 2024-09-11T03:48:30.405 INFO:tasks.rados.rados.2.smithi175.stdout:2741: done (1 left) 2024-09-11T03:48:30.405 INFO:tasks.rados.rados.2.smithi175.stdout:2742: done (0 left) 2024-09-11T03:48:30.405 INFO:tasks.rados.rados.2.smithi175.stdout:2743: delete oid 48 current snap is 301 2024-09-11T03:48:30.406 INFO:tasks.rados.rados.2.smithi175.stdout:2743: done (0 left) 2024-09-11T03:48:30.406 INFO:tasks.rados.rados.2.smithi175.stdout:2744: snap_create 2024-09-11T03:48:30.413 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:30.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:30.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:30 smithi121 ceph-mon[30729]: pgmap v1191: 67 pgs: 67 active+clean; 251 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 4.9 MiB/s wr, 31 op/s 2024-09-11T03:48:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:30 smithi121 ceph-mon[30729]: 2.5 scrub starts 2024-09-11T03:48:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:30 smithi121 ceph-mon[30729]: 2.5 scrub ok 2024-09-11T03:48:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:30 smithi121 ceph-mon[30729]: osdmap e692: 12 total, 12 up, 12 in 2024-09-11T03:48:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:30 smithi175 ceph-mon[30894]: pgmap v1191: 67 pgs: 67 active+clean; 251 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 4.9 MiB/s wr, 31 op/s 2024-09-11T03:48:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:30 smithi175 ceph-mon[30894]: 2.5 scrub starts 2024-09-11T03:48:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:30 smithi175 ceph-mon[30894]: 2.5 scrub ok 2024-09-11T03:48:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:30 smithi175 ceph-mon[30894]: osdmap e692: 12 total, 12 up, 12 in 2024-09-11T03:48:30.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:30.540 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:30.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:30.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:30.642 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:30.662 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:30.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:30.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:30.764 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:30.785 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:30.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:30.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:30.840 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:48:30.840 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:48:30.887 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:30.908 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:30.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:31.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:31.031 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:31.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:31.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:31.072 INFO:tasks.rados.rados.2.smithi175.stdout:2744: done (0 left) 2024-09-11T03:48:31.072 INFO:tasks.rados.rados.2.smithi175.stdout:2745: snap_create 2024-09-11T03:48:31.133 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:31.159 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:31.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:31.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:31.261 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:31.282 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:31.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:31.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:31.384 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:31.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:31 smithi084 ceph-mon[27254]: 2.6 scrub starts 2024-09-11T03:48:31.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:31 smithi084 ceph-mon[27254]: 2.6 scrub ok 2024-09-11T03:48:31.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:31 smithi084 ceph-mon[27254]: 1.11 scrub starts 2024-09-11T03:48:31.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:31 smithi084 ceph-mon[27254]: 1.11 scrub ok 2024-09-11T03:48:31.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:31 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:48:31.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:31 smithi084 ceph-mon[27254]: 2.7 scrub ok 2024-09-11T03:48:31.405 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:31.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:31.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:31 smithi121 ceph-mon[30729]: 2.6 scrub starts 2024-09-11T03:48:31.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:31 smithi121 ceph-mon[30729]: 2.6 scrub ok 2024-09-11T03:48:31.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:31 smithi121 ceph-mon[30729]: 1.11 scrub starts 2024-09-11T03:48:31.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:31 smithi121 ceph-mon[30729]: 1.11 scrub ok 2024-09-11T03:48:31.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:31 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:48:31.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:31 smithi121 ceph-mon[30729]: 2.7 scrub ok 2024-09-11T03:48:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:31 smithi175 ceph-mon[30894]: 2.6 scrub starts 2024-09-11T03:48:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:31 smithi175 ceph-mon[30894]: 2.6 scrub ok 2024-09-11T03:48:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:31 smithi175 ceph-mon[30894]: 1.11 scrub starts 2024-09-11T03:48:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:31 smithi175 ceph-mon[30894]: 1.11 scrub ok 2024-09-11T03:48:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:31 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:48:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:31 smithi175 ceph-mon[30894]: 2.7 scrub ok 2024-09-11T03:48:31.507 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:31.533 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:31.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:31.534 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:31.593 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:48:31.594 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T03:48:31.594 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-11T03:48:31.634 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:31.635 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:31.661 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:31.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:31.662 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:31.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:31.789 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:31.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:31.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:31.836 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:48:31.892 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:31.918 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:31.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:31.998 INFO:tasks.rados.rados.2.smithi175.stdout:2745: done (0 left) 2024-09-11T03:48:31.998 INFO:tasks.rados.rados.2.smithi175.stdout:2746: write (excl) oid 15 current snap is 303 2024-09-11T03:48:31.999 INFO:tasks.rados.rados.2.smithi175.stdout:2746: seq_num 950 ranges {766650=714322,2075423=688655,3522196=444454} 2024-09-11T03:48:32.006 INFO:tasks.rados.rados.2.smithi175.stdout:2746: writing smithi17557870-15 from 766650 to 1480972 tid 1 2024-09-11T03:48:32.013 INFO:tasks.rados.rados.2.smithi175.stdout:2746: writing smithi17557870-15 from 2075423 to 2764078 tid 2 2024-09-11T03:48:32.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:32.021 INFO:tasks.rados.rados.2.smithi175.stdout:2746: writing smithi17557870-15 from 3522196 to 3966650 tid 3 2024-09-11T03:48:32.021 INFO:tasks.rados.rados.2.smithi175.stdout:2747: read oid 30 snap -1 2024-09-11T03:48:32.021 INFO:tasks.rados.rados.2.smithi175.stdout:2747: expect (ObjNum 895 snap 284 seq_num 895) 2024-09-11T03:48:32.021 INFO:tasks.rados.rados.2.smithi175.stdout:2748: snap_create 2024-09-11T03:48:32.021 INFO:tasks.rados.rados.2.smithi175.stdout:2746: finishing write tid 1 to smithi17557870-15 2024-09-11T03:48:32.021 INFO:tasks.rados.rados.2.smithi175.stdout:2746: finishing write tid 2 to smithi17557870-15 2024-09-11T03:48:32.024 INFO:tasks.rados.rados.2.smithi175.stdout:2746: finishing write tid 3 to smithi17557870-15 2024-09-11T03:48:32.024 INFO:tasks.rados.rados.2.smithi175.stdout:2746: finishing write tid 4 to smithi17557870-15 2024-09-11T03:48:32.024 INFO:tasks.rados.rados.2.smithi175.stdout:2746: finishing write tid 5 to smithi17557870-15 2024-09-11T03:48:32.024 INFO:tasks.rados.rados.2.smithi175.stdout:2746: oid 15 updating version 0 to 891 2024-09-11T03:48:32.024 INFO:tasks.rados.rados.2.smithi175.stdout:2746: oid 15 updating version 891 to 892 2024-09-11T03:48:32.024 INFO:tasks.rados.rados.2.smithi175.stdout:2746: oid 15 version 892 is already newer than 890 2024-09-11T03:48:32.024 INFO:tasks.rados.rados.2.smithi175.stdout:2746: oid 15 updating version 892 to 893 2024-09-11T03:48:32.024 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 893 (ObjNum 950 snap 303 seq_num 950) dirty exists 2024-09-11T03:48:32.025 INFO:tasks.rados.rados.2.smithi175.stdout:2746: left oid 15 (ObjNum 950 snap 303 seq_num 950) 2024-09-11T03:48:32.027 INFO:tasks.rados.rados.2.smithi175.stdout:2747: expect (ObjNum 895 snap 284 seq_num 895) 2024-09-11T03:48:32.044 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:32.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:32.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:32.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T03:48:32.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:32.170 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:32.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:32.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:32.272 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:32.294 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:32.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:32.295 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:32.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:32 smithi084 ceph-mon[27254]: pgmap v1194: 67 pgs: 67 active+clean; 251 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 0 B/s wr, 6 op/s 2024-09-11T03:48:32.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:32 smithi084 ceph-mon[27254]: 1.1b scrub starts 2024-09-11T03:48:32.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:32 smithi084 ceph-mon[27254]: 1.1b scrub ok 2024-09-11T03:48:32.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:32 smithi084 ceph-mon[27254]: osdmap e693: 12 total, 12 up, 12 in 2024-09-11T03:48:32.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:32 smithi084 ceph-mon[27254]: 1.1e scrub starts 2024-09-11T03:48:32.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:32 smithi084 ceph-mon[27254]: 1.1e scrub ok 2024-09-11T03:48:32.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:32 smithi084 ceph-mon[27254]: osdmap e694: 12 total, 12 up, 12 in 2024-09-11T03:48:32.395 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:32.416 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:32.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:32.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:32 smithi121 ceph-mon[30729]: pgmap v1194: 67 pgs: 67 active+clean; 251 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 0 B/s wr, 6 op/s 2024-09-11T03:48:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:32 smithi121 ceph-mon[30729]: 1.1b scrub starts 2024-09-11T03:48:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:32 smithi121 ceph-mon[30729]: 1.1b scrub ok 2024-09-11T03:48:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:32 smithi121 ceph-mon[30729]: osdmap e693: 12 total, 12 up, 12 in 2024-09-11T03:48:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:32 smithi121 ceph-mon[30729]: 1.1e scrub starts 2024-09-11T03:48:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:32 smithi121 ceph-mon[30729]: 1.1e scrub ok 2024-09-11T03:48:32.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:32 smithi121 ceph-mon[30729]: osdmap e694: 12 total, 12 up, 12 in 2024-09-11T03:48:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:32 smithi175 ceph-mon[30894]: pgmap v1194: 67 pgs: 67 active+clean; 251 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 0 B/s wr, 6 op/s 2024-09-11T03:48:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:32 smithi175 ceph-mon[30894]: 1.1b scrub starts 2024-09-11T03:48:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:32 smithi175 ceph-mon[30894]: 1.1b scrub ok 2024-09-11T03:48:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:32 smithi175 ceph-mon[30894]: osdmap e693: 12 total, 12 up, 12 in 2024-09-11T03:48:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:32 smithi175 ceph-mon[30894]: 1.1e scrub starts 2024-09-11T03:48:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:32 smithi175 ceph-mon[30894]: 1.1e scrub ok 2024-09-11T03:48:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:32 smithi175 ceph-mon[30894]: osdmap e694: 12 total, 12 up, 12 in 2024-09-11T03:48:32.518 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:32.544 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:32.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:32.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:32.602 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:32.603 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:32.604 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:32.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:32.671 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:32.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:32.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:32.773 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:32.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:32.795 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:32.795 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:32.896 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:32.918 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:32.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:32.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:32.995 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:48:33.001 INFO:tasks.rados.rados.2.smithi175.stdout:2746: done (2 left) 2024-09-11T03:48:33.001 INFO:tasks.rados.rados.2.smithi175.stdout:2747: done (1 left) 2024-09-11T03:48:33.001 INFO:tasks.rados.rados.2.smithi175.stdout:2748: done (0 left) 2024-09-11T03:48:33.001 INFO:tasks.rados.rados.2.smithi175.stdout:2749: write (excl) oid 17 current snap is 304 2024-09-11T03:48:33.001 INFO:tasks.rados.rados.2.smithi175.stdout:2749: seq_num 951 ranges {783457=505924,1850258=548666,2865972=721096} 2024-09-11T03:48:33.008 INFO:tasks.rados.rados.2.smithi175.stdout:2749: writing smithi17557870-17 from 783457 to 1289381 tid 1 2024-09-11T03:48:33.014 INFO:tasks.rados.rados.2.smithi175.stdout:2749: writing smithi17557870-17 from 1850258 to 2398924 tid 2 2024-09-11T03:48:33.020 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:33.021 INFO:tasks.rados.rados.2.smithi175.stdout:2749: writing smithi17557870-17 from 2865972 to 3587068 tid 3 2024-09-11T03:48:33.021 INFO:tasks.rados.rados.2.smithi175.stdout:2750: write oid 12 current snap is 304 2024-09-11T03:48:33.021 INFO:tasks.rados.rados.2.smithi175.stdout:2749: finishing write tid 1 to smithi17557870-17 2024-09-11T03:48:33.021 INFO:tasks.rados.rados.2.smithi175.stdout:2749: finishing write tid 2 to smithi17557870-17 2024-09-11T03:48:33.021 INFO:tasks.rados.rados.2.smithi175.stdout:2750: seq_num 952 ranges {400264=581173,1508740=491524} 2024-09-11T03:48:33.025 INFO:tasks.rados.rados.2.smithi175.stdout:2750: writing smithi17557870-12 from 400264 to 981437 tid 1 2024-09-11T03:48:33.032 INFO:tasks.rados.rados.2.smithi175.stdout:2750: writing smithi17557870-12 from 1508740 to 2000264 tid 2 2024-09-11T03:48:33.032 INFO:tasks.rados.rados.2.smithi175.stdout:2751: copy_from oid 23 from oid 32 current snap is 304 2024-09-11T03:48:33.032 INFO:tasks.rados.rados.2.smithi175.stdout:2749: finishing write tid 3 to smithi17557870-17 2024-09-11T03:48:33.032 INFO:tasks.rados.rados.2.smithi175.stdout:2749: finishing write tid 4 to smithi17557870-17 2024-09-11T03:48:33.032 INFO:tasks.rados.rados.2.smithi175.stdout:2749: finishing write tid 5 to smithi17557870-17 2024-09-11T03:48:33.033 INFO:tasks.rados.rados.2.smithi175.stdout:2749: oid 17 updating version 0 to 902 2024-09-11T03:48:33.033 INFO:tasks.rados.rados.2.smithi175.stdout:2749: oid 17 updating version 902 to 904 2024-09-11T03:48:33.033 INFO:tasks.rados.rados.2.smithi175.stdout:2749: oid 17 version 904 is already newer than 903 2024-09-11T03:48:33.033 INFO:tasks.rados.rados.2.smithi175.stdout:2749: oid 17 version 904 is already newer than 901 2024-09-11T03:48:33.033 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 904 (ObjNum 951 snap 304 seq_num 951) dirty exists 2024-09-11T03:48:33.033 INFO:tasks.rados.rados.2.smithi175.stdout:2749: left oid 17 (ObjNum 951 snap 304 seq_num 951) 2024-09-11T03:48:33.033 INFO:tasks.rados.rados.2.smithi175.stdout:2750: finishing write tid 1 to smithi17557870-12 2024-09-11T03:48:33.033 INFO:tasks.rados.rados.2.smithi175.stdout:2749: done (2 left) 2024-09-11T03:48:33.033 INFO:tasks.rados.rados.2.smithi175.stdout:2752: write oid 3 current snap is 304 2024-09-11T03:48:33.033 INFO:tasks.rados.rados.2.smithi175.stdout:2752: seq_num 953 ranges {417071=656422,1567222=449849} 2024-09-11T03:48:33.038 INFO:tasks.rados.rados.2.smithi175.stdout:2752: writing smithi17557870-3 from 417071 to 1073493 tid 1 2024-09-11T03:48:33.044 INFO:tasks.rados.rados.2.smithi175.stdout:2752: writing smithi17557870-3 from 1567222 to 2017071 tid 2 2024-09-11T03:48:33.044 INFO:tasks.rados.rados.2.smithi175.stdout:2753: snap_create 2024-09-11T03:48:33.044 INFO:tasks.rados.rados.2.smithi175.stdout:2750: finishing write tid 2 to smithi17557870-12 2024-09-11T03:48:33.044 INFO:tasks.rados.rados.2.smithi175.stdout:2750: finishing write tid 3 to smithi17557870-12 2024-09-11T03:48:33.044 INFO:tasks.rados.rados.2.smithi175.stdout:2750: finishing write tid 4 to smithi17557870-12 2024-09-11T03:48:33.044 INFO:tasks.rados.rados.2.smithi175.stdout:2750: oid 12 updating version 0 to 984 2024-09-11T03:48:33.044 INFO:tasks.rados.rados.2.smithi175.stdout:2750: oid 12 version 984 is already newer than 983 2024-09-11T03:48:33.044 INFO:tasks.rados.rados.2.smithi175.stdout:2750: oid 12 updating version 984 to 985 2024-09-11T03:48:33.044 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 985 (ObjNum 952 snap 304 seq_num 952) dirty exists 2024-09-11T03:48:33.044 INFO:tasks.rados.rados.2.smithi175.stdout:2750: left oid 12 (ObjNum 952 snap 304 seq_num 952) 2024-09-11T03:48:33.044 INFO:tasks.rados.rados.2.smithi175.stdout:2751: finishing copy_from racing read to smithi17557870-23 2024-09-11T03:48:33.045 INFO:tasks.rados.rados.2.smithi175.stdout:2752: finishing write tid 1 to smithi17557870-3 2024-09-11T03:48:33.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:33.045 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:33.046 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:33.048 INFO:tasks.rados.rados.2.smithi175.stdout:2752: finishing write tid 2 to smithi17557870-3 2024-09-11T03:48:33.048 INFO:tasks.rados.rados.2.smithi175.stdout:2752: finishing write tid 3 to smithi17557870-3 2024-09-11T03:48:33.048 INFO:tasks.rados.rados.2.smithi175.stdout:2752: finishing write tid 4 to smithi17557870-3 2024-09-11T03:48:33.048 INFO:tasks.rados.rados.2.smithi175.stdout:2752: oid 3 updating version 0 to 1043 2024-09-11T03:48:33.048 INFO:tasks.rados.rados.2.smithi175.stdout:2752: oid 3 updating version 1043 to 1044 2024-09-11T03:48:33.048 INFO:tasks.rados.rados.2.smithi175.stdout:2752: oid 3 version 1044 is already newer than 1042 2024-09-11T03:48:33.048 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 1044 (ObjNum 953 snap 304 seq_num 953) dirty exists 2024-09-11T03:48:33.048 INFO:tasks.rados.rados.2.smithi175.stdout:2752: left oid 3 (ObjNum 953 snap 304 seq_num 953) 2024-09-11T03:48:33.050 INFO:tasks.rados.rados.2.smithi175.stdout:2751: finishing copy_from to smithi17557870-23 2024-09-11T03:48:33.050 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 962 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:48:33.148 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:33.170 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:33.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:33.171 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:33.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:33.296 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:33.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:33.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:33.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:33 smithi084 ceph-mon[27254]: 2.10 scrub starts 2024-09-11T03:48:33.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:33 smithi084 ceph-mon[27254]: 2.10 scrub ok 2024-09-11T03:48:33.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:33 smithi084 ceph-mon[27254]: 2.11 scrub starts 2024-09-11T03:48:33.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:33 smithi084 ceph-mon[27254]: 2.11 scrub ok 2024-09-11T03:48:33.339 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:33 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1440106911' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:48:33.339 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:33 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:33.339 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:33 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1440106911' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:48:33.339 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:33 smithi084 ceph-mon[27254]: osdmap e695: 12 total, 12 up, 12 in 2024-09-11T03:48:33.347 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-11T03:48:33.347 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-11T03:48:33.348 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-11T03:48:33.348 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-11T03:48:33.348 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-11T03:48:33.348 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-11T03:48:33.348 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-11T03:48:33.348 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-11T03:48:33.348 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-11T03:48:33.348 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-11T03:48:33.348 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-11T03:48:33.348 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-11T03:48:33.398 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:33.420 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:33.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:33.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:33.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:33 smithi121 ceph-mon[30729]: 2.10 scrub starts 2024-09-11T03:48:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:33 smithi121 ceph-mon[30729]: 2.10 scrub ok 2024-09-11T03:48:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:33 smithi121 ceph-mon[30729]: 2.11 scrub starts 2024-09-11T03:48:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:33 smithi121 ceph-mon[30729]: 2.11 scrub ok 2024-09-11T03:48:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:33 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1440106911' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:48:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:33 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:33 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1440106911' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:48:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:33 smithi121 ceph-mon[30729]: osdmap e695: 12 total, 12 up, 12 in 2024-09-11T03:48:33.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:33 smithi175 ceph-mon[30894]: 2.10 scrub starts 2024-09-11T03:48:33.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:33 smithi175 ceph-mon[30894]: 2.10 scrub ok 2024-09-11T03:48:33.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:33 smithi175 ceph-mon[30894]: 2.11 scrub starts 2024-09-11T03:48:33.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:33 smithi175 ceph-mon[30894]: 2.11 scrub ok 2024-09-11T03:48:33.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:33 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1440106911' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:48:33.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:33 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:33.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:33 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1440106911' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:48:33.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:33 smithi175 ceph-mon[30894]: osdmap e695: 12 total, 12 up, 12 in 2024-09-11T03:48:33.521 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:33.543 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:33.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:33.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:33.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:33.671 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:33.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:33.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:33.773 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:33.795 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:33.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:33.796 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:33.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:33.923 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:33.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:33.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:34.001 INFO:tasks.rados.rados.2.smithi175.stdout:2750: done (3 left) 2024-09-11T03:48:34.001 INFO:tasks.rados.rados.2.smithi175.stdout:2751: done (2 left) 2024-09-11T03:48:34.001 INFO:tasks.rados.rados.2.smithi175.stdout:2752: done (1 left) 2024-09-11T03:48:34.001 INFO:tasks.rados.rados.2.smithi175.stdout:2753: done (0 left) 2024-09-11T03:48:34.001 INFO:tasks.rados.rados.2.smithi175.stdout:2754: copy_from oid 27 from oid 2 current snap is 305 2024-09-11T03:48:34.001 INFO:tasks.rados.rados.2.smithi175.stdout:2755: delete oid 4 current snap is 305 2024-09-11T03:48:34.005 INFO:tasks.rados.rados.2.smithi175.stdout:2755: done (1 left) 2024-09-11T03:48:34.005 INFO:tasks.rados.rados.2.smithi175.stdout:2756: rollback oid 35 current snap is 305 2024-09-11T03:48:34.005 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 35 to 300 2024-09-11T03:48:34.005 INFO:tasks.rados.rados.2.smithi175.stdout:2757: read oid 36 snap -1 2024-09-11T03:48:34.005 INFO:tasks.rados.rados.2.smithi175.stdout:2757: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:48:34.005 INFO:tasks.rados.rados.2.smithi175.stdout:2758: snap_remove snap 303 2024-09-11T03:48:34.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:34.046 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:34.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:34.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:34.148 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:34.175 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:34.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:34.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:34.179 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:48:34.179 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T03:48:34.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-11T03:48:34.276 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:34.301 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:34.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:34.302 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:34.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:34 smithi084 ceph-mon[27254]: 1.1 scrub starts 2024-09-11T03:48:34.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:34 smithi084 ceph-mon[27254]: 1.1 scrub ok 2024-09-11T03:48:34.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:34 smithi084 ceph-mon[27254]: pgmap v1197: 67 pgs: 67 active+clean; 248 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-11T03:48:34.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:34 smithi084 ceph-mon[27254]: osdmap e696: 12 total, 12 up, 12 in 2024-09-11T03:48:34.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:34.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:34 smithi121 ceph-mon[30729]: 1.1 scrub starts 2024-09-11T03:48:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:34 smithi121 ceph-mon[30729]: 1.1 scrub ok 2024-09-11T03:48:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:34 smithi121 ceph-mon[30729]: pgmap v1197: 67 pgs: 67 active+clean; 248 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-11T03:48:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:34 smithi121 ceph-mon[30729]: osdmap e696: 12 total, 12 up, 12 in 2024-09-11T03:48:34.428 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:34.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:34.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:34 smithi175 ceph-mon[30894]: 1.1 scrub starts 2024-09-11T03:48:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:34 smithi175 ceph-mon[30894]: 1.1 scrub ok 2024-09-11T03:48:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:34 smithi175 ceph-mon[30894]: pgmap v1197: 67 pgs: 67 active+clean; 248 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-11T03:48:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:34 smithi175 ceph-mon[30894]: osdmap e696: 12 total, 12 up, 12 in 2024-09-11T03:48:34.529 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:34.551 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:34.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:34.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:34.652 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:48:34.652 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:48:34.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:34.669 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:48:34.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:48:34.678 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:34.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:34.685 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:48:34.686 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:48:34.701 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:48:34.701 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:48:34.718 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:48:34.718 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:48:34.735 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:48:34.735 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:48:34.752 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:48:34.752 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:48:34.769 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:48:34.769 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:48:34.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:34.788 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:48:34.789 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:48:34.805 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:34.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:34.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:34.806 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:48:34.806 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:48:34.823 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:48:34.824 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:48:34.841 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:48:34.841 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:48:34.878 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:48:34.878 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:48:34.879 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:48:34.879 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:48:34.906 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:34.927 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:34.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:34.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:35.008 INFO:tasks.rados.rados.2.smithi175.stdout:2758: done (3 left) 2024-09-11T03:48:35.008 INFO:tasks.rados.rados.2.smithi175.stdout:2759: read oid 42 snap 304 2024-09-11T03:48:35.008 INFO:tasks.rados.rados.2.smithi175.stdout:2759: expect deleted 2024-09-11T03:48:35.008 INFO:tasks.rados.rados.2.smithi175.stdout:2756: finishing rollback tid 0 to smithi17557870-35 2024-09-11T03:48:35.008 INFO:tasks.rados.rados.2.smithi175.stdout:2760: read oid 23 snap -1 2024-09-11T03:48:35.008 INFO:tasks.rados.rados.2.smithi175.stdout:2760: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:48:35.009 INFO:tasks.rados.rados.2.smithi175.stdout:2761: read oid 46 snap -1 2024-09-11T03:48:35.009 INFO:tasks.rados.rados.2.smithi175.stdout:2761: expect deleted 2024-09-11T03:48:35.009 INFO:tasks.rados.rados.2.smithi175.stdout:2762: write oid 47 current snap is 305 2024-09-11T03:48:35.009 INFO:tasks.rados.rados.2.smithi175.stdout:2756: finishing rollback tid 1 to smithi17557870-35 2024-09-11T03:48:35.009 INFO:tasks.rados.rados.2.smithi175.stdout:2756: finishing rollback tid 2 to smithi17557870-35 2024-09-11T03:48:35.009 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 898 (ObjNum 927 snap 295 seq_num 927) dirty exists 2024-09-11T03:48:35.009 INFO:tasks.rados.rados.2.smithi175.stdout:2762: seq_num 955 ranges {450685=406920,1567833=482852} 2024-09-11T03:48:35.012 INFO:tasks.rados.rados.2.smithi175.stdout:2762: writing smithi17557870-47 from 450685 to 857605 tid 1 2024-09-11T03:48:35.019 INFO:tasks.rados.rados.2.smithi175.stdout:2762: writing smithi17557870-47 from 1567833 to 2050685 tid 2 2024-09-11T03:48:35.019 INFO:tasks.rados.rados.2.smithi175.stdout:2756: done (6 left) 2024-09-11T03:48:35.020 INFO:tasks.rados.rados.2.smithi175.stdout:2763: read oid 49 snap -1 2024-09-11T03:48:35.020 INFO:tasks.rados.rados.2.smithi175.stdout:2763: expect deleted 2024-09-11T03:48:35.020 INFO:tasks.rados.rados.2.smithi175.stdout:2764: read oid 9 snap 302 2024-09-11T03:48:35.020 INFO:tasks.rados.rados.2.smithi175.stdout:2764: expect deleted 2024-09-11T03:48:35.020 INFO:tasks.rados.rados.2.smithi175.stdout:2765: read oid 3 snap -1 2024-09-11T03:48:35.020 INFO:tasks.rados.rados.2.smithi175.stdout:2765: expect (ObjNum 953 snap 304 seq_num 953) 2024-09-11T03:48:35.020 INFO:tasks.rados.rados.2.smithi175.stdout:2766: copy_from oid 34 from oid 35 current snap is 305 2024-09-11T03:48:35.020 INFO:tasks.rados.rados.2.smithi175.stdout:2757: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:48:35.029 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:35.052 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:35.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:35.053 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:35.154 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:35.175 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:35.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:35.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:35.259 INFO:tasks.rados.rados.2.smithi175.stdout:2754: finishing copy_from racing read to smithi17557870-27 2024-09-11T03:48:35.259 INFO:tasks.rados.rados.2.smithi175.stdout:2754: finishing copy_from to smithi17557870-27 2024-09-11T03:48:35.259 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 950 (ObjNum 765 snap 247 seq_num 765) dirty exists 2024-09-11T03:48:35.260 INFO:tasks.rados.rados.2.smithi175.stdout:2754: done (9 left) 2024-09-11T03:48:35.260 INFO:tasks.rados.rados.2.smithi175.stdout:2757: done (8 left) 2024-09-11T03:48:35.260 INFO:tasks.rados.rados.2.smithi175.stdout:2759: done (7 left) 2024-09-11T03:48:35.260 INFO:tasks.rados.rados.2.smithi175.stdout:2761: done (6 left) 2024-09-11T03:48:35.260 INFO:tasks.rados.rados.2.smithi175.stdout:2767: write oid 37 current snap is 305 2024-09-11T03:48:35.260 INFO:tasks.rados.rados.2.smithi175.stdout:2762: finishing write tid 1 to smithi17557870-47 2024-09-11T03:48:35.260 INFO:tasks.rados.rados.2.smithi175.stdout:2760: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:48:35.277 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:35.298 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:35.299 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:35 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/165871058' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:35 smithi084 ceph-mon[27254]: osdmap e697: 12 total, 12 up, 12 in 2024-09-11T03:48:35.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:35.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:35 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/165871058' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:35 smithi121 ceph-mon[30729]: osdmap e697: 12 total, 12 up, 12 in 2024-09-11T03:48:35.426 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:35.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:35.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:35.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:35 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/165871058' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:35.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:35 smithi175 ceph-mon[30894]: osdmap e697: 12 total, 12 up, 12 in 2024-09-11T03:48:35.493 INFO:tasks.rados.rados.2.smithi175.stdout:2762: finishing write tid 2 to smithi17557870-47 2024-09-11T03:48:35.493 INFO:tasks.rados.rados.2.smithi175.stdout:2762: finishing write tid 3 to smithi17557870-47 2024-09-11T03:48:35.493 INFO:tasks.rados.rados.2.smithi175.stdout:2762: finishing write tid 4 to smithi17557870-47 2024-09-11T03:48:35.493 INFO:tasks.rados.rados.2.smithi175.stdout:2762: oid 47 updating version 0 to 907 2024-09-11T03:48:35.493 INFO:tasks.rados.rados.2.smithi175.stdout:2762: oid 47 version 907 is already newer than 906 2024-09-11T03:48:35.494 INFO:tasks.rados.rados.2.smithi175.stdout:2762: oid 47 version 907 is already newer than 905 2024-09-11T03:48:35.494 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 907 (ObjNum 955 snap 305 seq_num 955) dirty exists 2024-09-11T03:48:35.494 INFO:tasks.rados.rados.2.smithi175.stdout:2762: left oid 47 (ObjNum 955 snap 305 seq_num 955) 2024-09-11T03:48:35.494 INFO:tasks.rados.rados.2.smithi175.stdout:2765: expect (ObjNum 953 snap 304 seq_num 953) 2024-09-11T03:48:35.527 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:35.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:35.549 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:35.550 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:35.616 INFO:tasks.rados.rados.2.smithi175.stdout:2766: finishing copy_from racing read to smithi17557870-34 2024-09-11T03:48:35.616 INFO:tasks.rados.rados.2.smithi175.stdout:2766: finishing copy_from to smithi17557870-34 2024-09-11T03:48:35.616 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 898 (ObjNum 927 snap 295 seq_num 927) dirty exists 2024-09-11T03:48:35.616 INFO:tasks.rados.rados.2.smithi175.stdout:2767: seq_num 956 ranges {467492=482169,1626315=441177} 2024-09-11T03:48:35.621 INFO:tasks.rados.rados.2.smithi175.stdout:2767: writing smithi17557870-37 from 467492 to 949661 tid 1 2024-09-11T03:48:35.628 INFO:tasks.rados.rados.2.smithi175.stdout:2767: writing smithi17557870-37 from 1626315 to 2067492 tid 2 2024-09-11T03:48:35.628 INFO:tasks.rados.rados.2.smithi175.stdout:2760: done (6 left) 2024-09-11T03:48:35.628 INFO:tasks.rados.rados.2.smithi175.stdout:2762: done (5 left) 2024-09-11T03:48:35.628 INFO:tasks.rados.rados.2.smithi175.stdout:2763: done (4 left) 2024-09-11T03:48:35.628 INFO:tasks.rados.rados.2.smithi175.stdout:2764: done (3 left) 2024-09-11T03:48:35.628 INFO:tasks.rados.rados.2.smithi175.stdout:2765: done (2 left) 2024-09-11T03:48:35.628 INFO:tasks.rados.rados.2.smithi175.stdout:2766: done (1 left) 2024-09-11T03:48:35.628 INFO:tasks.rados.rados.2.smithi175.stdout:2768: rollback oid 13 current snap is 305 2024-09-11T03:48:35.629 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 13 to 296 2024-09-11T03:48:35.629 INFO:tasks.rados.rados.2.smithi175.stdout:2767: finishing write tid 1 to smithi17557870-37 2024-09-11T03:48:35.629 INFO:tasks.rados.rados.2.smithi175.stdout:2769: read oid 20 snap 292 2024-09-11T03:48:35.629 INFO:tasks.rados.rados.2.smithi175.stdout:2769: expect (ObjNum 775 snap 248 seq_num 775) 2024-09-11T03:48:35.629 INFO:tasks.rados.rados.2.smithi175.stdout:2770: write oid 9 current snap is 305 2024-09-11T03:48:35.629 INFO:tasks.rados.rados.2.smithi175.stdout:2770: seq_num 957 ranges {484299=557418,1684797=399502} 2024-09-11T03:48:35.633 INFO:tasks.rados.rados.2.smithi175.stdout:2770: writing smithi17557870-9 from 484299 to 1041717 tid 1 2024-09-11T03:48:35.639 INFO:tasks.rados.rados.2.smithi175.stdout:2770: writing smithi17557870-9 from 1684797 to 2084299 tid 2 2024-09-11T03:48:35.640 INFO:tasks.rados.rados.2.smithi175.stdout:2771: copy_from oid 15 from oid 46 current snap is 305 2024-09-11T03:48:35.640 INFO:tasks.rados.rados.2.smithi175.stdout:2768: finishing rollback tid 0 to smithi17557870-13 2024-09-11T03:48:35.640 INFO:tasks.rados.rados.2.smithi175.stdout:2768: finishing rollback tid 1 to smithi17557870-13 2024-09-11T03:48:35.640 INFO:tasks.rados.rados.2.smithi175.stdout:2768: finishing rollback tid 2 to smithi17557870-13 2024-09-11T03:48:35.640 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 991 (ObjNum 907 snap 287 seq_num 907) dirty exists 2024-09-11T03:48:35.640 INFO:tasks.rados.rados.2.smithi175.stdout:2767: finishing write tid 2 to smithi17557870-37 2024-09-11T03:48:35.640 INFO:tasks.rados.rados.2.smithi175.stdout:2767: finishing write tid 3 to smithi17557870-37 2024-09-11T03:48:35.640 INFO:tasks.rados.rados.2.smithi175.stdout:2767: finishing write tid 4 to smithi17557870-37 2024-09-11T03:48:35.640 INFO:tasks.rados.rados.2.smithi175.stdout:2767: oid 37 updating version 0 to 895 2024-09-11T03:48:35.641 INFO:tasks.rados.rados.2.smithi175.stdout:2767: oid 37 version 895 is already newer than 894 2024-09-11T03:48:35.641 INFO:tasks.rados.rados.2.smithi175.stdout:2767: oid 37 updating version 895 to 896 2024-09-11T03:48:35.641 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 896 (ObjNum 956 snap 305 seq_num 956) dirty exists 2024-09-11T03:48:35.641 INFO:tasks.rados.rados.2.smithi175.stdout:2767: left oid 37 (ObjNum 956 snap 305 seq_num 956) 2024-09-11T03:48:35.641 INFO:tasks.rados.rados.2.smithi175.stdout:2767: done (4 left) 2024-09-11T03:48:35.641 INFO:tasks.rados.rados.2.smithi175.stdout:2768: done (3 left) 2024-09-11T03:48:35.641 INFO:tasks.rados.rados.2.smithi175.stdout:2772: rollback oid 19 current snap is 305 2024-09-11T03:48:35.641 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 19 to 285 2024-09-11T03:48:35.641 INFO:tasks.rados.rados.2.smithi175.stdout:2773: snap_create 2024-09-11T03:48:35.641 INFO:tasks.rados.rados.2.smithi175.stdout:2770: finishing write tid 1 to smithi17557870-9 2024-09-11T03:48:35.642 INFO:tasks.rados.rados.2.smithi175.stdout:2771: finishing copy_from to smithi17557870-15 2024-09-11T03:48:35.642 INFO:tasks.rados.rados.2.smithi175.stdout:2771: got expected ENOENT (src dne) 2024-09-11T03:48:35.642 INFO:tasks.rados.rados.2.smithi175.stdout:2771: finishing copy_from racing read to smithi17557870-15 2024-09-11T03:48:35.642 INFO:tasks.rados.rados.2.smithi175.stdout:2769: expect (ObjNum 775 snap 248 seq_num 775) 2024-09-11T03:48:35.651 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:35.673 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:35.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:35.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:35.775 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:35.797 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:35.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:35.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:35.864 INFO:tasks.rados.rados.2.smithi175.stdout:2770: finishing write tid 2 to smithi17557870-9 2024-09-11T03:48:35.864 INFO:tasks.rados.rados.2.smithi175.stdout:2772: finishing rollback tid 1 to smithi17557870-19 2024-09-11T03:48:35.864 INFO:tasks.rados.rados.2.smithi175.stdout:2772: finishing rollback tid 2 to smithi17557870-19 2024-09-11T03:48:35.864 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 981 (ObjNum 877 snap 277 seq_num 877) dirty exists 2024-09-11T03:48:35.864 INFO:tasks.rados.rados.2.smithi175.stdout:2770: finishing write tid 3 to smithi17557870-9 2024-09-11T03:48:35.864 INFO:tasks.rados.rados.2.smithi175.stdout:2770: finishing write tid 4 to smithi17557870-9 2024-09-11T03:48:35.864 INFO:tasks.rados.rados.2.smithi175.stdout:2770: oid 9 updating version 0 to 989 2024-09-11T03:48:35.864 INFO:tasks.rados.rados.2.smithi175.stdout:2770: oid 9 updating version 989 to 990 2024-09-11T03:48:35.864 INFO:tasks.rados.rados.2.smithi175.stdout:2770: oid 9 updating version 990 to 991 2024-09-11T03:48:35.864 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 991 (ObjNum 957 snap 305 seq_num 957) dirty exists 2024-09-11T03:48:35.865 INFO:tasks.rados.rados.2.smithi175.stdout:2770: left oid 9 (ObjNum 957 snap 305 seq_num 957) 2024-09-11T03:48:35.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:35.924 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:35.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:35.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:36.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:36.047 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:36.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:36.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:36.087 INFO:tasks.rados.rados.2.smithi175.stdout:2769: done (4 left) 2024-09-11T03:48:36.088 INFO:tasks.rados.rados.2.smithi175.stdout:2770: done (3 left) 2024-09-11T03:48:36.088 INFO:tasks.rados.rados.2.smithi175.stdout:2771: done (2 left) 2024-09-11T03:48:36.088 INFO:tasks.rados.rados.2.smithi175.stdout:2772: done (1 left) 2024-09-11T03:48:36.088 INFO:tasks.rados.rados.2.smithi175.stdout:2773: done (0 left) 2024-09-11T03:48:36.088 INFO:tasks.rados.rados.2.smithi175.stdout:2774: read oid 6 snap -1 2024-09-11T03:48:36.088 INFO:tasks.rados.rados.2.smithi175.stdout:2774: expect (ObjNum 857 snap 269 seq_num 857) 2024-09-11T03:48:36.088 INFO:tasks.rados.rados.2.smithi175.stdout:2775: delete oid 37 current snap is 306 2024-09-11T03:48:36.096 INFO:tasks.rados.rados.2.smithi175.stdout:2775: done (1 left) 2024-09-11T03:48:36.096 INFO:tasks.rados.rados.2.smithi175.stdout:2776: rollback oid 34 current snap is 306 2024-09-11T03:48:36.096 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 34 to 289 2024-09-11T03:48:36.096 INFO:tasks.rados.rados.2.smithi175.stdout:2777: snap_remove snap 304 2024-09-11T03:48:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:36 smithi084 ceph-mon[27254]: pgmap v1200: 67 pgs: 67 active+clean; 250 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2024-09-11T03:48:36.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:36.175 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:36.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:36.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:36.277 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:36.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:36.298 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:36.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:36.399 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:36.420 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:36.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:36.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:36 smithi121 ceph-mon[30729]: pgmap v1200: 67 pgs: 67 active+clean; 250 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2024-09-11T03:48:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:36 smithi175 ceph-mon[30894]: pgmap v1200: 67 pgs: 67 active+clean; 250 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2024-09-11T03:48:36.522 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:36.543 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:36.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:36.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:36.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:36.671 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:36.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:36.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:36.773 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:36.794 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:36.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:36.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:36.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:36.921 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:36.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:36.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:36.999 INFO:tasks.rados.rados.2.smithi175.stdout:2777: done (2 left) 2024-09-11T03:48:36.999 INFO:tasks.rados.rados.2.smithi175.stdout:2778: read oid 24 snap 289 2024-09-11T03:48:36.999 INFO:tasks.rados.rados.2.smithi175.stdout:2778: expect (ObjNum 811 snap 256 seq_num 811) 2024-09-11T03:48:36.999 INFO:tasks.rados.rados.2.smithi175.stdout:2779: delete oid 40 current snap is 306 2024-09-11T03:48:36.999 INFO:tasks.rados.rados.2.smithi175.stdout:2776: finishing rollback tid 0 to smithi17557870-34 2024-09-11T03:48:36.999 INFO:tasks.rados.rados.2.smithi175.stdout:2776: finishing rollback tid 1 to smithi17557870-34 2024-09-11T03:48:36.999 INFO:tasks.rados.rados.2.smithi175.stdout:2776: finishing rollback tid 2 to smithi17557870-34 2024-09-11T03:48:37.000 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 905 (ObjNum 905 snap 286 seq_num 905) dirty exists 2024-09-11T03:48:37.000 INFO:tasks.rados.rados.2.smithi175.stdout:2774: expect (ObjNum 857 snap 269 seq_num 857) 2024-09-11T03:48:37.023 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:37.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:37.045 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:37.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:37.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T03:48:37.121 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:37.134 INFO:tasks.rados.rados.2.smithi175.stdout:2778: expect (ObjNum 811 snap 256 seq_num 811) 2024-09-11T03:48:37.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:37.173 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:37.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:37.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:37.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:37.301 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:37.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:37.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:37.325 INFO:tasks.rados.rados.2.smithi175.stdout:2774: done (3 left) 2024-09-11T03:48:37.325 INFO:tasks.rados.rados.2.smithi175.stdout:2776: done (2 left) 2024-09-11T03:48:37.325 INFO:tasks.rados.rados.2.smithi175.stdout:2778: done (1 left) 2024-09-11T03:48:37.325 INFO:tasks.rados.rados.2.smithi175.stdout:2779: done (0 left) 2024-09-11T03:48:37.325 INFO:tasks.rados.rados.2.smithi175.stdout:2780: copy_from oid 42 from oid 1 current snap is 306 2024-09-11T03:48:37.325 INFO:tasks.rados.rados.2.smithi175.stdout:2781: read oid 38 snap -1 2024-09-11T03:48:37.325 INFO:tasks.rados.rados.2.smithi175.stdout:2781: expect (ObjNum 941 snap 297 seq_num 941) 2024-09-11T03:48:37.326 INFO:tasks.rados.rados.2.smithi175.stdout:2782: snap_create 2024-09-11T03:48:37.330 INFO:tasks.rados.rados.2.smithi175.stdout:2781: expect (ObjNum 941 snap 297 seq_num 941) 2024-09-11T03:48:37.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:37 smithi084 ceph-mon[27254]: osdmap e698: 12 total, 12 up, 12 in 2024-09-11T03:48:37.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:37 smithi084 ceph-mon[27254]: osdmap e699: 12 total, 12 up, 12 in 2024-09-11T03:48:37.402 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:37 smithi121 ceph-mon[30729]: osdmap e698: 12 total, 12 up, 12 in 2024-09-11T03:48:37.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:37 smithi121 ceph-mon[30729]: osdmap e699: 12 total, 12 up, 12 in 2024-09-11T03:48:37.424 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:37.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:37.425 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:37 smithi175 ceph-mon[30894]: osdmap e698: 12 total, 12 up, 12 in 2024-09-11T03:48:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:37 smithi175 ceph-mon[30894]: osdmap e699: 12 total, 12 up, 12 in 2024-09-11T03:48:37.500 INFO:tasks.rados.rados.2.smithi175.stdout:2780: finishing copy_from racing read to smithi17557870-42 2024-09-11T03:48:37.500 INFO:tasks.rados.rados.2.smithi175.stdout:2780: finishing copy_from to smithi17557870-42 2024-09-11T03:48:37.500 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 947 (ObjNum 758 snap 246 seq_num 758) dirty exists 2024-09-11T03:48:37.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:37.553 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:37.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:37.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:37.655 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:37.677 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:37.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:37.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:37.734 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-11T03:48:37.734 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-11T03:48:37.734 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-11T03:48:37.734 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-11T03:48:37.735 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-11T03:48:37.735 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-11T03:48:37.735 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-11T03:48:37.735 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-11T03:48:37.735 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-11T03:48:37.735 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-11T03:48:37.735 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-11T03:48:37.735 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-11T03:48:37.779 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:37.799 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:37.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:37.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:37.889 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:48:37.901 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:37.922 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:37.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:37.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:38.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:38.046 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:38.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:38.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:38.103 INFO:tasks.rados.rados.2.smithi175.stdout:2780: done (2 left) 2024-09-11T03:48:38.103 INFO:tasks.rados.rados.2.smithi175.stdout:2781: done (1 left) 2024-09-11T03:48:38.103 INFO:tasks.rados.rados.2.smithi175.stdout:2782: done (0 left) 2024-09-11T03:48:38.103 INFO:tasks.rados.rados.2.smithi175.stdout:2783: rollback oid 38 current snap is 307 2024-09-11T03:48:38.103 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 38 to 293 2024-09-11T03:48:38.103 INFO:tasks.rados.rados.2.smithi175.stdout:2784: delete oid 9 current snap is 307 2024-09-11T03:48:38.105 INFO:tasks.rados.rados.2.smithi175.stdout:2783: finishing rollback tid 0 to smithi17557870-38 2024-09-11T03:48:38.106 INFO:tasks.rados.rados.2.smithi175.stdout:2783: finishing rollback tid 1 to smithi17557870-38 2024-09-11T03:48:38.106 INFO:tasks.rados.rados.2.smithi175.stdout:2783: finishing rollback tid 2 to smithi17557870-38 2024-09-11T03:48:38.106 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 997 (ObjNum 795 snap 254 seq_num 795) dirty exists 2024-09-11T03:48:38.108 INFO:tasks.rados.rados.2.smithi175.stdout:2783: done (1 left) 2024-09-11T03:48:38.108 INFO:tasks.rados.rados.2.smithi175.stdout:2784: done (0 left) 2024-09-11T03:48:38.108 INFO:tasks.rados.rados.2.smithi175.stdout:2785: delete oid 3 current snap is 307 2024-09-11T03:48:38.111 INFO:tasks.rados.rados.2.smithi175.stdout:2785: done (0 left) 2024-09-11T03:48:38.112 INFO:tasks.rados.rados.2.smithi175.stdout:2786: read oid 33 snap 279 2024-09-11T03:48:38.112 INFO:tasks.rados.rados.2.smithi175.stdout:2786: expect (ObjNum 870 snap 274 seq_num 870) 2024-09-11T03:48:38.112 INFO:tasks.rados.rados.2.smithi175.stdout:2787: rollback oid 37 current snap is 307 2024-09-11T03:48:38.112 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 37 to 279 2024-09-11T03:48:38.112 INFO:tasks.rados.rados.2.smithi175.stdout:2788: read oid 32 snap 286 2024-09-11T03:48:38.112 INFO:tasks.rados.rados.2.smithi175.stdout:2788: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:48:38.112 INFO:tasks.rados.rados.2.smithi175.stdout:2789: read oid 25 snap -1 2024-09-11T03:48:38.112 INFO:tasks.rados.rados.2.smithi175.stdout:2789: expect (ObjNum 803 snap 255 seq_num 803) 2024-09-11T03:48:38.112 INFO:tasks.rados.rados.2.smithi175.stdout:2790: snap_create 2024-09-11T03:48:38.116 INFO:tasks.rados.rados.2.smithi175.stdout:2787: finishing rollback tid 1 to smithi17557870-37 2024-09-11T03:48:38.116 INFO:tasks.rados.rados.2.smithi175.stdout:2788: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:48:38.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:38.172 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:38.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:38.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:38.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:38.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:38.298 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:38.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:38.350 INFO:tasks.rados.rados.2.smithi175.stdout:2787: finishing rollback tid 2 to smithi17557870-37 2024-09-11T03:48:38.350 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 899 (ObjNum 867 snap 273 seq_num 867) dirty exists 2024-09-11T03:48:38.350 INFO:tasks.rados.rados.2.smithi175.stdout:2789: expect (ObjNum 803 snap 255 seq_num 803) 2024-09-11T03:48:38.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:38 smithi084 ceph-mon[27254]: pgmap v1203: 67 pgs: 67 active+clean; 265 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2024-09-11T03:48:38.399 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:38.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:38.422 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:38.422 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:38 smithi121 ceph-mon[30729]: pgmap v1203: 67 pgs: 67 active+clean; 265 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2024-09-11T03:48:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:38 smithi175 ceph-mon[30894]: pgmap v1203: 67 pgs: 67 active+clean; 265 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2024-09-11T03:48:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:48:38.524 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T03:48:38.525 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-11T03:48:38.526 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:38.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:38.550 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:38.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:38.550 INFO:tasks.rados.rados.2.smithi175.stdout:2786: expect (ObjNum 870 snap 274 seq_num 870) 2024-09-11T03:48:38.567 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:38.650 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:38.665 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:48:38.671 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:38.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:38.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:38.773 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:38.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:38.795 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:38.795 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:38.896 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:38.918 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:38.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:38.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:39.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:39.042 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:39.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:39.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:39.112 INFO:tasks.rados.rados.2.smithi175.stdout:2786: done (4 left) 2024-09-11T03:48:39.112 INFO:tasks.rados.rados.2.smithi175.stdout:2787: done (3 left) 2024-09-11T03:48:39.112 INFO:tasks.rados.rados.2.smithi175.stdout:2788: done (2 left) 2024-09-11T03:48:39.112 INFO:tasks.rados.rados.2.smithi175.stdout:2789: done (1 left) 2024-09-11T03:48:39.112 INFO:tasks.rados.rados.2.smithi175.stdout:2790: done (0 left) 2024-09-11T03:48:39.113 INFO:tasks.rados.rados.2.smithi175.stdout:2791: write (excl) oid 45 current snap is 308 2024-09-11T03:48:39.113 INFO:tasks.rados.rados.2.smithi175.stdout:2791: seq_num 962 ranges {568334=650016,1860854=307480} 2024-09-11T03:48:39.118 INFO:tasks.rados.rados.2.smithi175.stdout:2791: writing smithi17557870-45 from 568334 to 1218350 tid 1 2024-09-11T03:48:39.123 INFO:tasks.rados.rados.2.smithi175.stdout:2791: writing smithi17557870-45 from 1860854 to 2168334 tid 2 2024-09-11T03:48:39.123 INFO:tasks.rados.rados.2.smithi175.stdout:2791: finishing write tid 1 to smithi17557870-45 2024-09-11T03:48:39.124 INFO:tasks.rados.rados.2.smithi175.stdout:2792: snap_remove snap 306 2024-09-11T03:48:39.144 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:39.165 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:39.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:39.166 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:39.172 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:48:39.173 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:48:39.196 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:39.201 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:48:39.201 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:48:39.201 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:39.220 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:48:39.220 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:48:39.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:48:39.238 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:48:39.254 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:48:39.255 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:48:39.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:39.271 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:48:39.271 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:48:39.287 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:48:39.287 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:48:39.288 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:39.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:39.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:39.303 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:48:39.303 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:48:39.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:39 smithi084 ceph-mon[27254]: osdmap e700: 12 total, 12 up, 12 in 2024-09-11T03:48:39.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:39 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:48:39.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:39 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:48:39.320 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:48:39.320 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:48:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:48:39.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:48:39.353 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:48:39.353 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:48:39.371 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:48:39.371 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:48:39.389 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:39.411 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:39.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:39.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:39.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:39 smithi121 ceph-mon[30729]: osdmap e700: 12 total, 12 up, 12 in 2024-09-11T03:48:39.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:39 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:48:39.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:39 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:48:39.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:39 smithi175 ceph-mon[30894]: osdmap e700: 12 total, 12 up, 12 in 2024-09-11T03:48:39.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:39 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:48:39.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:39 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:48:39.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:39.534 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:39.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:39.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:39.636 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:39.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:39.658 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:39.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:39.758 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:39.779 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:39.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:39.881 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:39.902 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:39.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:39.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:40.004 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:40.031 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:40.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:40.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:40.114 INFO:tasks.rados.rados.2.smithi175.stdout:2792: done (1 left) 2024-09-11T03:48:40.114 INFO:tasks.rados.rados.2.smithi175.stdout:2793: snap_create 2024-09-11T03:48:40.114 INFO:tasks.rados.rados.2.smithi175.stdout:2791: finishing write tid 2 to smithi17557870-45 2024-09-11T03:48:40.114 INFO:tasks.rados.rados.2.smithi175.stdout:2791: finishing write tid 3 to smithi17557870-45 2024-09-11T03:48:40.115 INFO:tasks.rados.rados.2.smithi175.stdout:2791: finishing write tid 4 to smithi17557870-45 2024-09-11T03:48:40.115 INFO:tasks.rados.rados.2.smithi175.stdout:2791: oid 45 updating version 0 to 948 2024-09-11T03:48:40.115 INFO:tasks.rados.rados.2.smithi175.stdout:2791: oid 45 updating version 948 to 949 2024-09-11T03:48:40.115 INFO:tasks.rados.rados.2.smithi175.stdout:2791: oid 45 updating version 949 to 950 2024-09-11T03:48:40.115 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 950 (ObjNum 962 snap 308 seq_num 962) dirty exists 2024-09-11T03:48:40.115 INFO:tasks.rados.rados.2.smithi175.stdout:2791: left oid 45 (ObjNum 962 snap 308 seq_num 962) 2024-09-11T03:48:40.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:40.156 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:40.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:40.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:40.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:40.280 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:40.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:40.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:40 smithi084 ceph-mon[27254]: pgmap v1206: 67 pgs: 67 active+clean; 270 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 3.1 MiB/s wr, 22 op/s 2024-09-11T03:48:40.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:40 smithi084 ceph-mon[27254]: osdmap e701: 12 total, 12 up, 12 in 2024-09-11T03:48:40.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:40 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/934301341' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:40.335 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-11T03:48:40.336 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-11T03:48:40.336 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-11T03:48:40.336 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-11T03:48:40.336 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-11T03:48:40.336 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-11T03:48:40.336 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-11T03:48:40.336 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-11T03:48:40.336 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-11T03:48:40.336 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-11T03:48:40.336 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-11T03:48:40.336 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-11T03:48:40.378 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:40 smithi175 ceph-mon[30894]: pgmap v1206: 67 pgs: 67 active+clean; 270 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 3.1 MiB/s wr, 22 op/s 2024-09-11T03:48:40.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:40 smithi175 ceph-mon[30894]: osdmap e701: 12 total, 12 up, 12 in 2024-09-11T03:48:40.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:40 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/934301341' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:40.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:40.402 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:40.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:40.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:40.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:40 smithi121 ceph-mon[30729]: pgmap v1206: 67 pgs: 67 active+clean; 270 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 3.1 MiB/s wr, 22 op/s 2024-09-11T03:48:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:40 smithi121 ceph-mon[30729]: osdmap e701: 12 total, 12 up, 12 in 2024-09-11T03:48:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:40 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/934301341' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:40.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:40.525 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:40.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:40.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:40.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:40.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:40.649 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:40.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:40.750 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:40.771 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:40.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:40.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:40.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:40.895 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:40.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:40.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:40.998 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:41.020 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:41.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:41.021 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:41.080 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:41.115 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:48:41.119 INFO:tasks.rados.rados.2.smithi175.stdout:2791: done (1 left) 2024-09-11T03:48:41.119 INFO:tasks.rados.rados.2.smithi175.stdout:2793: done (0 left) 2024-09-11T03:48:41.119 INFO:tasks.rados.rados.2.smithi175.stdout:2794: read oid 6 snap -1 2024-09-11T03:48:41.119 INFO:tasks.rados.rados.2.smithi175.stdout:2794: expect (ObjNum 857 snap 269 seq_num 857) 2024-09-11T03:48:41.119 INFO:tasks.rados.rados.2.smithi175.stdout:2795: snap_create 2024-09-11T03:48:41.122 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:41.131 INFO:tasks.rados.rados.2.smithi175.stdout:2794: expect (ObjNum 857 snap 269 seq_num 857) 2024-09-11T03:48:41.143 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:41.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:41.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:41.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:41.269 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:41.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:41.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:41.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:41 smithi084 ceph-mon[27254]: osdmap e702: 12 total, 12 up, 12 in 2024-09-11T03:48:41.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:41 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2217930628' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:48:41.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:41 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:48:41.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:41 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:48:41.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:41.395 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:41.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:41.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:41 smithi121 ceph-mon[30729]: osdmap e702: 12 total, 12 up, 12 in 2024-09-11T03:48:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:41 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2217930628' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:48:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:41 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:48:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:41 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:48:41.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:41 smithi175 ceph-mon[30894]: osdmap e702: 12 total, 12 up, 12 in 2024-09-11T03:48:41.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:41 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2217930628' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:48:41.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:41 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:48:41.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:41 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:48:41.496 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:41.519 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:41.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:41.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:41.546 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:48:41.547 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T03:48:41.547 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-11T03:48:41.620 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:41.650 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:41.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:41.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:41.752 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:41.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:41.776 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:41.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:41.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:41.898 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:41.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:41.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:41.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:42.001 INFO:tasks.rados.rados.2.smithi175.stdout:2794: done (1 left) 2024-09-11T03:48:42.001 INFO:tasks.rados.rados.2.smithi175.stdout:2795: done (0 left) 2024-09-11T03:48:42.001 INFO:tasks.rados.rados.2.smithi175.stdout:2796: copy_from oid 43 from oid 14 current snap is 310 2024-09-11T03:48:42.001 INFO:tasks.rados.rados.2.smithi175.stdout:2797: read oid 9 snap -1 2024-09-11T03:48:42.001 INFO:tasks.rados.rados.2.smithi175.stdout:2797: expect deleted 2024-09-11T03:48:42.001 INFO:tasks.rados.rados.2.smithi175.stdout:2798: copy_from oid 8 from oid 28 current snap is 310 2024-09-11T03:48:42.001 INFO:tasks.rados.rados.2.smithi175.stdout:2799: write oid 19 current snap is 310 2024-09-11T03:48:42.001 INFO:tasks.rados.rados.2.smithi175.stdout:2799: seq_num 963 ranges {585141=725265,1919336=265805} 2024-09-11T03:48:42.007 INFO:tasks.rados.rados.2.smithi175.stdout:2799: writing smithi17557870-19 from 585141 to 1310406 tid 1 2024-09-11T03:48:42.012 INFO:tasks.rados.rados.2.smithi175.stdout:2799: writing smithi17557870-19 from 1919336 to 2185141 tid 2 2024-09-11T03:48:42.012 INFO:tasks.rados.rados.2.smithi175.stdout:2800: copy_from oid 42 from oid 45 current snap is 310 2024-09-11T03:48:42.013 INFO:tasks.rados.rados.2.smithi175.stdout:2796: finishing copy_from to smithi17557870-43 2024-09-11T03:48:42.013 INFO:tasks.rados.rados.2.smithi175.stdout:2796: got expected ENOENT (src dne) 2024-09-11T03:48:42.013 INFO:tasks.rados.rados.2.smithi175.stdout:2796: finishing copy_from racing read to smithi17557870-43 2024-09-11T03:48:42.013 INFO:tasks.rados.rados.2.smithi175.stdout:2796: done (4 left) 2024-09-11T03:48:42.013 INFO:tasks.rados.rados.2.smithi175.stdout:2797: done (3 left) 2024-09-11T03:48:42.013 INFO:tasks.rados.rados.2.smithi175.stdout:2801: copy_from oid 7 from oid 39 current snap is 310 2024-09-11T03:48:42.013 INFO:tasks.rados.rados.2.smithi175.stdout:2802: write (excl) oid 33 current snap is 310 2024-09-11T03:48:42.013 INFO:tasks.rados.rados.2.smithi175.stdout:2802: seq_num 964 ranges {601948=400514,1577818=624130} 2024-09-11T03:48:42.019 INFO:tasks.rados.rados.2.smithi175.stdout:2802: writing smithi17557870-33 from 601948 to 1002462 tid 1 2024-09-11T03:48:42.023 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:48:42.023 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:48:42.026 INFO:tasks.rados.rados.2.smithi175.stdout:2802: writing smithi17557870-33 from 1577818 to 2201948 tid 2 2024-09-11T03:48:42.026 INFO:tasks.rados.rados.2.smithi175.stdout:2803: write oid 10 current snap is 310 2024-09-11T03:48:42.026 INFO:tasks.rados.rados.2.smithi175.stdout:2801: finishing copy_from to smithi17557870-7 2024-09-11T03:48:42.026 INFO:tasks.rados.rados.2.smithi175.stdout:2801: got expected ENOENT (src dne) 2024-09-11T03:48:42.026 INFO:tasks.rados.rados.2.smithi175.stdout:2803: seq_num 965 ranges {618755=475763,1636300=457996} 2024-09-11T03:48:42.028 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:42.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:42.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:42.031 INFO:tasks.rados.rados.2.smithi175.stdout:2803: writing smithi17557870-10 from 618755 to 1094518 tid 1 2024-09-11T03:48:42.037 INFO:tasks.rados.rados.2.smithi175.stdout:2803: writing smithi17557870-10 from 1636300 to 2094296 tid 2 2024-09-11T03:48:42.037 INFO:tasks.rados.rados.2.smithi175.stdout:2804: snap_create 2024-09-11T03:48:42.038 INFO:tasks.rados.rados.2.smithi175.stdout:2801: finishing copy_from racing read to smithi17557870-7 2024-09-11T03:48:42.038 INFO:tasks.rados.rados.2.smithi175.stdout:2799: finishing write tid 1 to smithi17557870-19 2024-09-11T03:48:42.038 INFO:tasks.rados.rados.2.smithi175.stdout:2799: finishing write tid 2 to smithi17557870-19 2024-09-11T03:48:42.038 INFO:tasks.rados.rados.2.smithi175.stdout:2799: finishing write tid 3 to smithi17557870-19 2024-09-11T03:48:42.038 INFO:tasks.rados.rados.2.smithi175.stdout:2799: finishing write tid 4 to smithi17557870-19 2024-09-11T03:48:42.038 INFO:tasks.rados.rados.2.smithi175.stdout:2799: oid 19 updating version 0 to 984 2024-09-11T03:48:42.038 INFO:tasks.rados.rados.2.smithi175.stdout:2799: oid 19 version 984 is already newer than 982 2024-09-11T03:48:42.038 INFO:tasks.rados.rados.2.smithi175.stdout:2799: oid 19 version 984 is already newer than 983 2024-09-11T03:48:42.038 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 984 (ObjNum 963 snap 310 seq_num 963) dirty exists 2024-09-11T03:48:42.038 INFO:tasks.rados.rados.2.smithi175.stdout:2799: left oid 19 (ObjNum 963 snap 310 seq_num 963) 2024-09-11T03:48:42.038 INFO:tasks.rados.rados.2.smithi175.stdout:2800: finishing copy_from racing read to smithi17557870-42 2024-09-11T03:48:42.038 INFO:tasks.rados.rados.2.smithi175.stdout:2798: finishing copy_from racing read to smithi17557870-8 2024-09-11T03:48:42.038 INFO:tasks.rados.rados.2.smithi175.stdout:2802: finishing write tid 1 to smithi17557870-33 2024-09-11T03:48:42.038 INFO:tasks.rados.rados.2.smithi175.stdout:2800: finishing copy_from to smithi17557870-42 2024-09-11T03:48:42.039 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 950 (ObjNum 962 snap 308 seq_num 962) dirty exists 2024-09-11T03:48:42.039 INFO:tasks.rados.rados.2.smithi175.stdout:2802: finishing write tid 2 to smithi17557870-33 2024-09-11T03:48:42.039 INFO:tasks.rados.rados.2.smithi175.stdout:2802: finishing write tid 3 to smithi17557870-33 2024-09-11T03:48:42.039 INFO:tasks.rados.rados.2.smithi175.stdout:2802: finishing write tid 4 to smithi17557870-33 2024-09-11T03:48:42.039 INFO:tasks.rados.rados.2.smithi175.stdout:2802: oid 33 updating version 0 to 1052 2024-09-11T03:48:42.039 INFO:tasks.rados.rados.2.smithi175.stdout:2802: oid 33 updating version 1052 to 1054 2024-09-11T03:48:42.039 INFO:tasks.rados.rados.2.smithi175.stdout:2802: oid 33 version 1054 is already newer than 1053 2024-09-11T03:48:42.039 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 1054 (ObjNum 964 snap 310 seq_num 964) dirty exists 2024-09-11T03:48:42.039 INFO:tasks.rados.rados.2.smithi175.stdout:2802: left oid 33 (ObjNum 964 snap 310 seq_num 964) 2024-09-11T03:48:42.039 INFO:tasks.rados.rados.2.smithi175.stdout:2798: finishing copy_from to smithi17557870-8 2024-09-11T03:48:42.039 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 1017 (ObjNum 942 snap 297 seq_num 942) dirty exists 2024-09-11T03:48:42.039 INFO:tasks.rados.rados.2.smithi175.stdout:2803: finishing write tid 1 to smithi17557870-10 2024-09-11T03:48:42.040 INFO:tasks.rados.rados.2.smithi175.stdout:2803: finishing write tid 2 to smithi17557870-10 2024-09-11T03:48:42.040 INFO:tasks.rados.rados.2.smithi175.stdout:2803: finishing write tid 3 to smithi17557870-10 2024-09-11T03:48:42.040 INFO:tasks.rados.rados.2.smithi175.stdout:2803: finishing write tid 4 to smithi17557870-10 2024-09-11T03:48:42.040 INFO:tasks.rados.rados.2.smithi175.stdout:2803: oid 10 updating version 0 to 908 2024-09-11T03:48:42.041 INFO:tasks.rados.rados.2.smithi175.stdout:2803: oid 10 updating version 908 to 909 2024-09-11T03:48:42.041 INFO:tasks.rados.rados.2.smithi175.stdout:2803: oid 10 updating version 909 to 910 2024-09-11T03:48:42.041 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 910 (ObjNum 965 snap 310 seq_num 965) dirty exists 2024-09-11T03:48:42.041 INFO:tasks.rados.rados.2.smithi175.stdout:2803: left oid 10 (ObjNum 965 snap 310 seq_num 965) 2024-09-11T03:48:42.129 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:42.151 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:42.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:42.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:42.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:42.277 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:42.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:42.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:42.379 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:42 smithi084 ceph-mon[27254]: pgmap v1209: 67 pgs: 67 active+clean; 270 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 932 KiB/s wr, 9 op/s 2024-09-11T03:48:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:42 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:42 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2217930628' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:48:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:42 smithi084 ceph-mon[27254]: osdmap e703: 12 total, 12 up, 12 in 2024-09-11T03:48:42.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:42 smithi084 ceph-mon[27254]: osdmap e704: 12 total, 12 up, 12 in 2024-09-11T03:48:42.401 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:42.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:42.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:42 smithi121 ceph-mon[30729]: pgmap v1209: 67 pgs: 67 active+clean; 270 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 932 KiB/s wr, 9 op/s 2024-09-11T03:48:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:42 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:42 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2217930628' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:48:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:42 smithi121 ceph-mon[30729]: osdmap e703: 12 total, 12 up, 12 in 2024-09-11T03:48:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:42 smithi121 ceph-mon[30729]: osdmap e704: 12 total, 12 up, 12 in 2024-09-11T03:48:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:42 smithi175 ceph-mon[30894]: pgmap v1209: 67 pgs: 67 active+clean; 270 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 932 KiB/s wr, 9 op/s 2024-09-11T03:48:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:42 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:42.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:42 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2217930628' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:48:42.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:42 smithi175 ceph-mon[30894]: osdmap e703: 12 total, 12 up, 12 in 2024-09-11T03:48:42.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:42 smithi175 ceph-mon[30894]: osdmap e704: 12 total, 12 up, 12 in 2024-09-11T03:48:42.502 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:42.523 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:42.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:42.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:42.625 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:42.647 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:42.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:42.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:42.749 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:42.775 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:42.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:42.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:42.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:48:42.876 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:42.903 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:42.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:42.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:43.001 INFO:tasks.rados.rados.2.smithi175.stdout:2798: done (6 left) 2024-09-11T03:48:43.001 INFO:tasks.rados.rados.2.smithi175.stdout:2799: done (5 left) 2024-09-11T03:48:43.001 INFO:tasks.rados.rados.2.smithi175.stdout:2800: done (4 left) 2024-09-11T03:48:43.001 INFO:tasks.rados.rados.2.smithi175.stdout:2801: done (3 left) 2024-09-11T03:48:43.001 INFO:tasks.rados.rados.2.smithi175.stdout:2802: done (2 left) 2024-09-11T03:48:43.002 INFO:tasks.rados.rados.2.smithi175.stdout:2803: done (1 left) 2024-09-11T03:48:43.002 INFO:tasks.rados.rados.2.smithi175.stdout:2804: done (0 left) 2024-09-11T03:48:43.002 INFO:tasks.rados.rados.2.smithi175.stdout:2805: delete oid 14 current snap is 311 2024-09-11T03:48:43.004 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:43.005 INFO:tasks.rados.rados.2.smithi175.stdout:2805: done (0 left) 2024-09-11T03:48:43.005 INFO:tasks.rados.rados.2.smithi175.stdout:2806: snap_remove snap 297 2024-09-11T03:48:43.030 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:43.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:43.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:43.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:43.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:43.159 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:43.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:43.260 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:43.287 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:43.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:43.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:43.389 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:43.400 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T03:48:43.415 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:43.416 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:43.433 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:43.474 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:43.516 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:43.542 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:43.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:43.543 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:43.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:43.670 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:43.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:43.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:43.772 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:43.797 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:43.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:43.798 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:43.899 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:43.924 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:43.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:43.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:44.007 INFO:tasks.rados.rados.2.smithi175.stdout:2806: done (0 left) 2024-09-11T03:48:44.008 INFO:tasks.rados.rados.2.smithi175.stdout:2807: rollback oid 23 current snap is 311 2024-09-11T03:48:44.008 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 23 to 300 2024-09-11T03:48:44.009 INFO:tasks.rados.rados.2.smithi175.stdout:2808: write oid 48 current snap is 311 2024-09-11T03:48:44.009 INFO:tasks.rados.rados.2.smithi175.stdout:2808: seq_num 967 ranges {652369=742614,2153264=99105} 2024-09-11T03:48:44.015 INFO:tasks.rados.rados.2.smithi175.stdout:2808: writing smithi17557870-48 from 652369 to 1394983 tid 1 2024-09-11T03:48:44.021 INFO:tasks.rados.rados.2.smithi175.stdout:2808: writing smithi17557870-48 from 2153264 to 2252369 tid 2 2024-09-11T03:48:44.021 INFO:tasks.rados.rados.2.smithi175.stdout:2809: delete oid 16 current snap is 311 2024-09-11T03:48:44.021 INFO:tasks.rados.rados.2.smithi175.stdout:2807: finishing rollback tid 0 to smithi17557870-23 2024-09-11T03:48:44.021 INFO:tasks.rados.rados.2.smithi175.stdout:2807: finishing rollback tid 1 to smithi17557870-23 2024-09-11T03:48:44.021 INFO:tasks.rados.rados.2.smithi175.stdout:2807: finishing rollback tid 2 to smithi17557870-23 2024-09-11T03:48:44.021 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 968 (ObjNum 855 snap 268 seq_num 855) dirty exists 2024-09-11T03:48:44.021 INFO:tasks.rados.rados.2.smithi175.stdout:2808: finishing write tid 1 to smithi17557870-48 2024-09-11T03:48:44.024 INFO:tasks.rados.rados.2.smithi175.stdout:2808: finishing write tid 2 to smithi17557870-48 2024-09-11T03:48:44.024 INFO:tasks.rados.rados.2.smithi175.stdout:2808: finishing write tid 3 to smithi17557870-48 2024-09-11T03:48:44.024 INFO:tasks.rados.rados.2.smithi175.stdout:2808: finishing write tid 4 to smithi17557870-48 2024-09-11T03:48:44.024 INFO:tasks.rados.rados.2.smithi175.stdout:2808: oid 48 updating version 0 to 869 2024-09-11T03:48:44.024 INFO:tasks.rados.rados.2.smithi175.stdout:2808: oid 48 version 869 is already newer than 867 2024-09-11T03:48:44.024 INFO:tasks.rados.rados.2.smithi175.stdout:2808: oid 48 version 869 is already newer than 868 2024-09-11T03:48:44.024 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 869 (ObjNum 967 snap 311 seq_num 967) dirty exists 2024-09-11T03:48:44.024 INFO:tasks.rados.rados.2.smithi175.stdout:2808: left oid 48 (ObjNum 967 snap 311 seq_num 967) 2024-09-11T03:48:44.024 INFO:tasks.rados.rados.2.smithi175.stdout:2807: done (2 left) 2024-09-11T03:48:44.024 INFO:tasks.rados.rados.2.smithi175.stdout:2808: done (1 left) 2024-09-11T03:48:44.025 INFO:tasks.rados.rados.2.smithi175.stdout:2809: done (0 left) 2024-09-11T03:48:44.025 INFO:tasks.rados.rados.2.smithi175.stdout:2810: write oid 29 current snap is 311 2024-09-11T03:48:44.025 INFO:tasks.rados.rados.2.smithi175.stdout:2810: seq_num 969 ranges {685983=493112,1870228=415755} 2024-09-11T03:48:44.026 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:44.030 INFO:tasks.rados.rados.2.smithi175.stdout:2810: writing smithi17557870-29 from 685983 to 1179095 tid 1 2024-09-11T03:48:44.037 INFO:tasks.rados.rados.2.smithi175.stdout:2810: writing smithi17557870-29 from 1870228 to 2285983 tid 2 2024-09-11T03:48:44.037 INFO:tasks.rados.rados.2.smithi175.stdout:2811: rollback oid 30 current snap is 311 2024-09-11T03:48:44.037 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 30 to 301 2024-09-11T03:48:44.037 INFO:tasks.rados.rados.2.smithi175.stdout:2810: finishing write tid 1 to smithi17557870-29 2024-09-11T03:48:44.037 INFO:tasks.rados.rados.2.smithi175.stdout:2812: write (excl) oid 38 current snap is 311 2024-09-11T03:48:44.038 INFO:tasks.rados.rados.2.smithi175.stdout:2812: seq_num 970 ranges {702790=568361,1812357=490433} 2024-09-11T03:48:44.043 INFO:tasks.rados.rados.2.smithi175.stdout:2812: writing smithi17557870-38 from 702790 to 1271151 tid 1 2024-09-11T03:48:44.052 INFO:tasks.rados.rados.2.smithi175.stdout:2812: writing smithi17557870-38 from 1812357 to 2302790 tid 2 2024-09-11T03:48:44.052 INFO:tasks.rados.rados.2.smithi175.stdout:2813: rollback oid 20 current snap is 311 2024-09-11T03:48:44.052 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 20 to 305 2024-09-11T03:48:44.052 INFO:tasks.rados.rados.2.smithi175.stdout:2810: finishing write tid 2 to smithi17557870-29 2024-09-11T03:48:44.053 INFO:tasks.rados.rados.2.smithi175.stdout:2810: finishing write tid 3 to smithi17557870-29 2024-09-11T03:48:44.053 INFO:tasks.rados.rados.2.smithi175.stdout:2811: finishing rollback tid 0 to smithi17557870-30 2024-09-11T03:48:44.053 INFO:tasks.rados.rados.2.smithi175.stdout:2810: finishing write tid 4 to smithi17557870-29 2024-09-11T03:48:44.053 INFO:tasks.rados.rados.2.smithi175.stdout:2810: oid 29 updating version 0 to 1009 2024-09-11T03:48:44.053 INFO:tasks.rados.rados.2.smithi175.stdout:2810: oid 29 version 1009 is already newer than 1008 2024-09-11T03:48:44.053 INFO:tasks.rados.rados.2.smithi175.stdout:2810: oid 29 version 1009 is already newer than 1007 2024-09-11T03:48:44.054 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 1009 (ObjNum 969 snap 311 seq_num 969) dirty exists 2024-09-11T03:48:44.054 INFO:tasks.rados.rados.2.smithi175.stdout:2810: left oid 29 (ObjNum 969 snap 311 seq_num 969) 2024-09-11T03:48:44.054 INFO:tasks.rados.rados.2.smithi175.stdout:2811: finishing rollback tid 1 to smithi17557870-30 2024-09-11T03:48:44.054 INFO:tasks.rados.rados.2.smithi175.stdout:2811: finishing rollback tid 2 to smithi17557870-30 2024-09-11T03:48:44.054 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 1020 (ObjNum 895 snap 284 seq_num 895) dirty exists 2024-09-11T03:48:44.054 INFO:tasks.rados.rados.2.smithi175.stdout:2812: finishing write tid 1 to smithi17557870-38 2024-09-11T03:48:44.055 INFO:tasks.rados.rados.2.smithi175.stdout:2810: done (3 left) 2024-09-11T03:48:44.055 INFO:tasks.rados.rados.2.smithi175.stdout:2811: done (2 left) 2024-09-11T03:48:44.055 INFO:tasks.rados.rados.2.smithi175.stdout:2814: read oid 17 snap 309 2024-09-11T03:48:44.055 INFO:tasks.rados.rados.2.smithi175.stdout:2814: expect (ObjNum 951 snap 304 seq_num 951) 2024-09-11T03:48:44.055 INFO:tasks.rados.rados.2.smithi175.stdout:2815: read oid 43 snap -1 2024-09-11T03:48:44.055 INFO:tasks.rados.rados.2.smithi175.stdout:2815: expect deleted 2024-09-11T03:48:44.056 INFO:tasks.rados.rados.2.smithi175.stdout:2816: snap_create 2024-09-11T03:48:44.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:44.057 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:44.058 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:44.059 INFO:tasks.rados.rados.2.smithi175.stdout:2812: finishing write tid 2 to smithi17557870-38 2024-09-11T03:48:44.059 INFO:tasks.rados.rados.2.smithi175.stdout:2812: finishing write tid 3 to smithi17557870-38 2024-09-11T03:48:44.059 INFO:tasks.rados.rados.2.smithi175.stdout:2812: finishing write tid 4 to smithi17557870-38 2024-09-11T03:48:44.059 INFO:tasks.rados.rados.2.smithi175.stdout:2812: oid 38 updating version 0 to 1013 2024-09-11T03:48:44.059 INFO:tasks.rados.rados.2.smithi175.stdout:2812: oid 38 version 1013 is already newer than 1012 2024-09-11T03:48:44.060 INFO:tasks.rados.rados.2.smithi175.stdout:2812: oid 38 version 1013 is already newer than 1011 2024-09-11T03:48:44.060 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 1013 (ObjNum 970 snap 311 seq_num 970) dirty exists 2024-09-11T03:48:44.060 INFO:tasks.rados.rados.2.smithi175.stdout:2812: left oid 38 (ObjNum 970 snap 311 seq_num 970) 2024-09-11T03:48:44.068 INFO:tasks.rados.rados.2.smithi175.stdout:2813: finishing rollback tid 0 to smithi17557870-20 2024-09-11T03:48:44.069 INFO:tasks.rados.rados.2.smithi175.stdout:2813: finishing rollback tid 1 to smithi17557870-20 2024-09-11T03:48:44.069 INFO:tasks.rados.rados.2.smithi175.stdout:2813: finishing rollback tid 2 to smithi17557870-20 2024-09-11T03:48:44.069 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 913 (ObjNum 775 snap 248 seq_num 775) dirty exists 2024-09-11T03:48:44.077 INFO:tasks.rados.rados.2.smithi175.stdout:2814: expect (ObjNum 951 snap 304 seq_num 951) 2024-09-11T03:48:44.141 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-11T03:48:44.142 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-11T03:48:44.142 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-11T03:48:44.142 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-11T03:48:44.142 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-11T03:48:44.142 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-11T03:48:44.142 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-11T03:48:44.142 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-11T03:48:44.142 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-11T03:48:44.142 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-11T03:48:44.142 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-11T03:48:44.142 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-11T03:48:44.159 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:44.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:44.182 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:44.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:44.283 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:44.303 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:44.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:44.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:44.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:44 smithi084 ceph-mon[27254]: pgmap v1212: 67 pgs: 67 active+clean; 281 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2024-09-11T03:48:44.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:44 smithi084 ceph-mon[27254]: osdmap e705: 12 total, 12 up, 12 in 2024-09-11T03:48:44.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:44 smithi121 ceph-mon[30729]: pgmap v1212: 67 pgs: 67 active+clean; 281 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2024-09-11T03:48:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:44 smithi121 ceph-mon[30729]: osdmap e705: 12 total, 12 up, 12 in 2024-09-11T03:48:44.431 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:44.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:44.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:44 smithi175 ceph-mon[30894]: pgmap v1212: 67 pgs: 67 active+clean; 281 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2024-09-11T03:48:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:44 smithi175 ceph-mon[30894]: osdmap e705: 12 total, 12 up, 12 in 2024-09-11T03:48:44.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:44.558 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:44.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:44.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:44.661 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:44.682 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:44.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:44.683 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:44.784 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:44.806 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:44.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:44.807 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:44.875 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:48:44.875 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T03:48:44.876 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-11T03:48:44.908 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:44.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:44.930 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:44.930 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:44.938 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:48:45.012 INFO:tasks.rados.rados.2.smithi175.stdout:2812: done (4 left) 2024-09-11T03:48:45.012 INFO:tasks.rados.rados.2.smithi175.stdout:2813: done (3 left) 2024-09-11T03:48:45.012 INFO:tasks.rados.rados.2.smithi175.stdout:2814: done (2 left) 2024-09-11T03:48:45.013 INFO:tasks.rados.rados.2.smithi175.stdout:2815: done (1 left) 2024-09-11T03:48:45.013 INFO:tasks.rados.rados.2.smithi175.stdout:2816: done (0 left) 2024-09-11T03:48:45.013 INFO:tasks.rados.rados.2.smithi175.stdout:2817: write (excl) oid 14 current snap is 312 2024-09-11T03:48:45.013 INFO:tasks.rados.rados.2.smithi175.stdout:2817: seq_num 971 ranges {719597=643610,1870839=448758} 2024-09-11T03:48:45.019 INFO:tasks.rados.rados.2.smithi175.stdout:2817: writing smithi17557870-14 from 719597 to 1363207 tid 1 2024-09-11T03:48:45.025 INFO:tasks.rados.rados.2.smithi175.stdout:2817: writing smithi17557870-14 from 1870839 to 2319597 tid 2 2024-09-11T03:48:45.025 INFO:tasks.rados.rados.2.smithi175.stdout:2818: snap_create 2024-09-11T03:48:45.025 INFO:tasks.rados.rados.2.smithi175.stdout:2817: finishing write tid 1 to smithi17557870-14 2024-09-11T03:48:45.027 INFO:tasks.rados.rados.2.smithi175.stdout:2817: finishing write tid 2 to smithi17557870-14 2024-09-11T03:48:45.028 INFO:tasks.rados.rados.2.smithi175.stdout:2817: finishing write tid 3 to smithi17557870-14 2024-09-11T03:48:45.028 INFO:tasks.rados.rados.2.smithi175.stdout:2817: finishing write tid 4 to smithi17557870-14 2024-09-11T03:48:45.028 INFO:tasks.rados.rados.2.smithi175.stdout:2817: oid 14 updating version 0 to 958 2024-09-11T03:48:45.028 INFO:tasks.rados.rados.2.smithi175.stdout:2817: oid 14 updating version 958 to 959 2024-09-11T03:48:45.028 INFO:tasks.rados.rados.2.smithi175.stdout:2817: oid 14 version 959 is already newer than 957 2024-09-11T03:48:45.028 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 959 (ObjNum 971 snap 312 seq_num 971) dirty exists 2024-09-11T03:48:45.029 INFO:tasks.rados.rados.2.smithi175.stdout:2817: left oid 14 (ObjNum 971 snap 312 seq_num 971) 2024-09-11T03:48:45.031 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:45.052 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:45.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:45.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:45.076 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:48:45.076 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:48:45.096 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:45.104 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:48:45.104 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:48:45.130 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:48:45.130 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:48:45.153 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:45.154 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:48:45.154 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:48:45.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:48:45.171 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:48:45.175 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:45.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:45.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:45.188 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:48:45.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:48:45.205 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:48:45.205 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:48:45.218 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:45 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:48:45.219 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:45 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:48:45.219 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:45 smithi084 ceph-mon[27254]: osdmap e706: 12 total, 12 up, 12 in 2024-09-11T03:48:45.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:48:45.222 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:48:45.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:48:45.238 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:48:45.255 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:48:45.255 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:48:45.271 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:48:45.272 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:48:45.276 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:45.288 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:48:45.289 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:48:45.297 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:45.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:45.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:45.399 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:45.421 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:45.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:45 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:48:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:45 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:48:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:45 smithi121 ceph-mon[30729]: osdmap e706: 12 total, 12 up, 12 in 2024-09-11T03:48:45.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:45.423 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:45.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:45 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:48:45.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:45 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:48:45.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:45 smithi175 ceph-mon[30894]: osdmap e706: 12 total, 12 up, 12 in 2024-09-11T03:48:45.523 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:45.545 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:45.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:45.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:45.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:45.672 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:45.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:45.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:45.774 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:45.776 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:45.796 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:45.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:45.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:45.804 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:48:45.804 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:48:45.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:48:45.805 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:48:45.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.805 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1068 2.3 e175/95 2:c3768452:::smithi17557870-11:head v 175'55, mlcod=165'51)", 2024-09-11T03:48:45.805 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:39:51.874076+0000", 2024-09-11T03:48:45.805 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 533.92942115599999, 2024-09-11T03:48:45.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033166769999999998, 2024-09-11T03:48:45.805 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.805 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.805 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:45.805 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.874076+0000", 2024-09-11T03:48:45.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.874076+0000", 2024-09-11T03:48:45.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.874076+0000", 2024-09-11T03:48:45.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.66e-07 2024-09-11T03:48:45.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.906033+0000", 2024-09-11T03:48:45.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.031956971000000001 2024-09-11T03:48:45.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.906049+0000", 2024-09-11T03:48:45.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6503000000000001e-05 2024-09-11T03:48:45.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.906058+0000", 2024-09-11T03:48:45.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3860000000000007e-06 2024-09-11T03:48:45.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.906111+0000", 2024-09-11T03:48:45.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3053999999999997e-05 2024-09-11T03:48:45.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.906129+0000", 2024-09-11T03:48:45.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8071999999999999e-05 2024-09-11T03:48:45.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:48:45.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.907203+0000", 2024-09-11T03:48:45.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010736960000000001 2024-09-11T03:48:45.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:39:51.907243+0000", 2024-09-11T03:48:45.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9922000000000002e-05 2024-09-11T03:48:45.809 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.809 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.809 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.810 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1159 2.e 2:750de3b7:::smithi17557870-8:head [copy-from ver 77] snapc 2e=[2e,2c,2b,28,26] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:48:45.810 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:01.831728+0000", 2024-09-11T03:48:45.810 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 523.97176900500006, 2024-09-11T03:48:45.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016271035999999999, 2024-09-11T03:48:45.810 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.810 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.810 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:45.810 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:48:45.810 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:48:45.810 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:45.810 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1159 2024-09-11T03:48:45.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.811 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831728+0000", 2024-09-11T03:48:45.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831728+0000", 2024-09-11T03:48:45.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831730+0000", 2024-09-11T03:48:45.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.283e-06 2024-09-11T03:48:45.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831741+0000", 2024-09-11T03:48:45.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0475000000000001e-05 2024-09-11T03:48:45.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831760+0000", 2024-09-11T03:48:45.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9034000000000001e-05 2024-09-11T03:48:45.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831776+0000", 2024-09-11T03:48:45.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6106000000000001e-05 2024-09-11T03:48:45.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831836+0000", 2024-09-11T03:48:45.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0420000000000001e-05 2024-09-11T03:48:45.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:48:45.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831841+0000", 2024-09-11T03:48:45.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8770000000000001e-06 2024-09-11T03:48:45.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.833455+0000", 2024-09-11T03:48:45.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016140639999999999 2024-09-11T03:48:45.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.833568+0000", 2024-09-11T03:48:45.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011259700000000001 2024-09-11T03:48:45.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:48:45.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.840230+0000", 2024-09-11T03:48:45.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066619189999999997 2024-09-11T03:48:45.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:48:45.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.845836+0000", 2024-09-11T03:48:45.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056066459999999999 2024-09-11T03:48:45.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.846094+0000", 2024-09-11T03:48:45.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025752300000000001 2024-09-11T03:48:45.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.847897+0000", 2024-09-11T03:48:45.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018033529999999999 2024-09-11T03:48:45.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:48:45.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.847954+0000", 2024-09-11T03:48:45.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6436e-05 2024-09-11T03:48:45.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.847999+0000", 2024-09-11T03:48:45.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5303000000000003e-05 2024-09-11T03:48:45.817 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.817 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.817 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.817 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1160 2.e 2:750de3b7:::smithi17557870-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:48:45.817 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:01.831798+0000", 2024-09-11T03:48:45.817 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 523.971699285, 2024-09-11T03:48:45.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012910331000000001, 2024-09-11T03:48:45.817 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.817 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.817 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:48:45.817 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:48:45.818 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:48:45.818 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:45.818 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1160 2024-09-11T03:48:45.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.818 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831798+0000", 2024-09-11T03:48:45.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831798+0000", 2024-09-11T03:48:45.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831798+0000", 2024-09-11T03:48:45.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0699999999999998e-07 2024-09-11T03:48:45.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831800+0000", 2024-09-11T03:48:45.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8679999999999999e-06 2024-09-11T03:48:45.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831806+0000", 2024-09-11T03:48:45.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6339999999999998e-06 2024-09-11T03:48:45.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831809+0000", 2024-09-11T03:48:45.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8650000000000003e-06 2024-09-11T03:48:45.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831896+0000", 2024-09-11T03:48:45.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6700000000000007e-05 2024-09-11T03:48:45.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:48:45.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831904+0000", 2024-09-11T03:48:45.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8099999999999998e-06 2024-09-11T03:48:45.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.833745+0000", 2024-09-11T03:48:45.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018409979999999999 2024-09-11T03:48:45.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:48:45.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.833795+0000", 2024-09-11T03:48:45.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0413e-05 2024-09-11T03:48:45.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.844574+0000", 2024-09-11T03:48:45.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010778993000000001 2024-09-11T03:48:45.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.844637+0000", 2024-09-11T03:48:45.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3002000000000004e-05 2024-09-11T03:48:45.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.844708+0000", 2024-09-11T03:48:45.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0641000000000003e-05 2024-09-11T03:48:45.823 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.823 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.823 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.824 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1285 2.1 2:8638b65e:::smithi17557870-5:head [copy-from ver 153] snapc 33=[32,30,2c,2b] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:48:45.824 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:12.836052+0000", 2024-09-11T03:48:45.824 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 512.967444688, 2024-09-11T03:48:45.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020441596999999999, 2024-09-11T03:48:45.824 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.824 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.824 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:45.824 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:48:45.824 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:48:45.824 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:45.824 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1285 2024-09-11T03:48:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836052+0000", 2024-09-11T03:48:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836052+0000", 2024-09-11T03:48:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836053+0000", 2024-09-11T03:48:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.34e-07 2024-09-11T03:48:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836055+0000", 2024-09-11T03:48:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7549999999999999e-06 2024-09-11T03:48:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836061+0000", 2024-09-11T03:48:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6579999999999999e-06 2024-09-11T03:48:45.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836071+0000", 2024-09-11T03:48:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0141e-05 2024-09-11T03:48:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836118+0000", 2024-09-11T03:48:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7200999999999998e-05 2024-09-11T03:48:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:48:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836124+0000", 2024-09-11T03:48:45.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8150000000000002e-06 2024-09-11T03:48:45.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.837592+0000", 2024-09-11T03:48:45.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014672509999999999 2024-09-11T03:48:45.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.837642+0000", 2024-09-11T03:48:45.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0145999999999997e-05 2024-09-11T03:48:45.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,3", 2024-09-11T03:48:45.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.849946+0000", 2024-09-11T03:48:45.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012303869 2024-09-11T03:48:45.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:48:45.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.852055+0000", 2024-09-11T03:48:45.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002109835 2024-09-11T03:48:45.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856314+0000", 2024-09-11T03:48:45.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042587700000000003 2024-09-11T03:48:45.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856368+0000", 2024-09-11T03:48:45.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3538999999999997e-05 2024-09-11T03:48:45.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:48:45.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856435+0000", 2024-09-11T03:48:45.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7470000000000003e-05 2024-09-11T03:48:45.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856494+0000", 2024-09-11T03:48:45.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8712999999999998e-05 2024-09-11T03:48:45.830 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.831 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.831 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.831 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1286 2.1 2:8638b65e:::smithi17557870-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:48:45.831 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:12.836083+0000", 2024-09-11T03:48:45.831 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 512.96741381799995, 2024-09-11T03:48:45.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015739692, 2024-09-11T03:48:45.831 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.831 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.831 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:48:45.831 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:48:45.831 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:48:45.831 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:45.832 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1286 2024-09-11T03:48:45.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.832 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836083+0000", 2024-09-11T03:48:45.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836083+0000", 2024-09-11T03:48:45.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836084+0000", 2024-09-11T03:48:45.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2500000000000001e-07 2024-09-11T03:48:45.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836085+0000", 2024-09-11T03:48:45.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8020000000000001e-06 2024-09-11T03:48:45.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836092+0000", 2024-09-11T03:48:45.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3169999999999998e-06 2024-09-11T03:48:45.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836096+0000", 2024-09-11T03:48:45.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.634e-06 2024-09-11T03:48:45.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836147+0000", 2024-09-11T03:48:45.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0374999999999998e-05 2024-09-11T03:48:45.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:48:45.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836150+0000", 2024-09-11T03:48:45.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.072e-06 2024-09-11T03:48:45.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.837748+0000", 2024-09-11T03:48:45.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001597867 2024-09-11T03:48:45.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:48:45.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.837783+0000", 2024-09-11T03:48:45.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5395000000000002e-05 2024-09-11T03:48:45.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.851559+0000", 2024-09-11T03:48:45.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013776343999999999 2024-09-11T03:48:45.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.851665+0000", 2024-09-11T03:48:45.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010521199999999999 2024-09-11T03:48:45.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.851823+0000", 2024-09-11T03:48:45.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015824900000000001 2024-09-11T03:48:45.837 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.837 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.837 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.837 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1434 2.e 2:7efbbea8:::smithi17557870-24:head [copy-from ver 75] snapc 36=[35,2b] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-11T03:48:45.837 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:20.864420+0000", 2024-09-11T03:48:45.837 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 504.939077467, 2024-09-11T03:48:45.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012389887, 2024-09-11T03:48:45.837 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.837 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.837 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:45.837 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:48:45.838 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:48:45.838 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:45.838 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1434 2024-09-11T03:48:45.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.838 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864420+0000", 2024-09-11T03:48:45.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864420+0000", 2024-09-11T03:48:45.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864421+0000", 2024-09-11T03:48:45.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8369999999999999e-06 2024-09-11T03:48:45.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864430+0000", 2024-09-11T03:48:45.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.132e-06 2024-09-11T03:48:45.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864447+0000", 2024-09-11T03:48:45.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6305999999999999e-05 2024-09-11T03:48:45.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864454+0000", 2024-09-11T03:48:45.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1230000000000002e-06 2024-09-11T03:48:45.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864527+0000", 2024-09-11T03:48:45.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2606000000000002e-05 2024-09-11T03:48:45.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:48:45.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864536+0000", 2024-09-11T03:48:45.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7790000000000007e-06 2024-09-11T03:48:45.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.865844+0000", 2024-09-11T03:48:45.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307473 2024-09-11T03:48:45.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.865934+0000", 2024-09-11T03:48:45.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9785999999999997e-05 2024-09-11T03:48:45.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:48:45.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.869567+0000", 2024-09-11T03:48:45.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0036332859999999999 2024-09-11T03:48:45.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:48:45.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.874154+0000", 2024-09-11T03:48:45.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.004586957 2024-09-11T03:48:45.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.875683+0000", 2024-09-11T03:48:45.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015289920000000001 2024-09-11T03:48:45.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.876715+0000", 2024-09-11T03:48:45.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010321620000000001 2024-09-11T03:48:45.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:48:45.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.876752+0000", 2024-09-11T03:48:45.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6820000000000003e-05 2024-09-11T03:48:45.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.876809+0000", 2024-09-11T03:48:45.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7627999999999998e-05 2024-09-11T03:48:45.844 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.844 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.844 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.844 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1589 2.7 2:e77ce595:::smithi17557870-31:head [copy-from ver 148] snapc 3f=[3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-09-11T03:48:45.844 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:35.104207+0000", 2024-09-11T03:48:45.844 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 490.699289814, 2024-09-11T03:48:45.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021309244000000001, 2024-09-11T03:48:45.844 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.844 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.844 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:45.845 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:48:45.845 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:48:45.845 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:45.845 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1589 2024-09-11T03:48:45.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.845 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104207+0000", 2024-09-11T03:48:45.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104207+0000", 2024-09-11T03:48:45.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104208+0000", 2024-09-11T03:48:45.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2780000000000001e-06 2024-09-11T03:48:45.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104213+0000", 2024-09-11T03:48:45.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5979999999999999e-06 2024-09-11T03:48:45.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104225+0000", 2024-09-11T03:48:45.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1819e-05 2024-09-11T03:48:45.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104249+0000", 2024-09-11T03:48:45.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4117999999999998e-05 2024-09-11T03:48:45.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104305+0000", 2024-09-11T03:48:45.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6069999999999997e-05 2024-09-11T03:48:45.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104375+0000", 2024-09-11T03:48:45.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0156999999999999e-05 2024-09-11T03:48:45.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:48:45.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116294+0000", 2024-09-11T03:48:45.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011918676 2024-09-11T03:48:45.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:48:45.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.123116+0000", 2024-09-11T03:48:45.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068219259999999999 2024-09-11T03:48:45.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.124470+0000", 2024-09-11T03:48:45.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001354079 2024-09-11T03:48:45.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125455+0000", 2024-09-11T03:48:45.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00098535300000000001 2024-09-11T03:48:45.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:48:45.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125487+0000", 2024-09-11T03:48:45.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2200999999999999e-05 2024-09-11T03:48:45.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125516+0000", 2024-09-11T03:48:45.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8969e-05 2024-09-11T03:48:45.850 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.850 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.850 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.851 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1590 2.7 2:e77ce595:::smithi17557870-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e218)", 2024-09-11T03:48:45.851 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:35.104267+0000", 2024-09-11T03:48:45.851 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 490.69923023400003, 2024-09-11T03:48:45.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017183714999999999, 2024-09-11T03:48:45.851 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.851 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.851 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:48:45.851 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:48:45.851 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:48:45.851 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:45.851 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1590 2024-09-11T03:48:45.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.851 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104267+0000", 2024-09-11T03:48:45.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104267+0000", 2024-09-11T03:48:45.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104267+0000", 2024-09-11T03:48:45.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.77e-07 2024-09-11T03:48:45.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104269+0000", 2024-09-11T03:48:45.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8199999999999999e-06 2024-09-11T03:48:45.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104274+0000", 2024-09-11T03:48:45.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2839999999999997e-06 2024-09-11T03:48:45.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104277+0000", 2024-09-11T03:48:45.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0149999999999999e-06 2024-09-11T03:48:45.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104478+0000", 2024-09-11T03:48:45.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200256 2024-09-11T03:48:45.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:48:45.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104511+0000", 2024-09-11T03:48:45.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3155999999999999e-05 2024-09-11T03:48:45.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.121250+0000", 2024-09-11T03:48:45.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016738843999999999 2024-09-11T03:48:45.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.121352+0000", 2024-09-11T03:48:45.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102384 2024-09-11T03:48:45.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.121450+0000", 2024-09-11T03:48:45.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8579000000000006e-05 2024-09-11T03:48:45.856 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.856 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.856 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.856 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1634 2.7 2:e23d1762:::smithi17557870-21:head [copy-from ver 214] snapc 40=[40,3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:48:45.856 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:36.206869+0000", 2024-09-11T03:48:45.856 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 489.59662752200001, 2024-09-11T03:48:45.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017521808999999999, 2024-09-11T03:48:45.856 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.856 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.856 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:45.857 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:48:45.857 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:48:45.857 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:45.857 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1634 2024-09-11T03:48:45.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.857 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206869+0000", 2024-09-11T03:48:45.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206869+0000", 2024-09-11T03:48:45.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206872+0000", 2024-09-11T03:48:45.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.926e-06 2024-09-11T03:48:45.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206898+0000", 2024-09-11T03:48:45.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5987e-05 2024-09-11T03:48:45.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206986+0000", 2024-09-11T03:48:45.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7714999999999996e-05 2024-09-11T03:48:45.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207030+0000", 2024-09-11T03:48:45.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3455999999999999e-05 2024-09-11T03:48:45.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207171+0000", 2024-09-11T03:48:45.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014184700000000001 2024-09-11T03:48:45.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207283+0000", 2024-09-11T03:48:45.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000111976 2024-09-11T03:48:45.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:48:45.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.216671+0000", 2024-09-11T03:48:45.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0093877200000000004 2024-09-11T03:48:45.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.223379+0000", 2024-09-11T03:48:45.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067082289999999996 2024-09-11T03:48:45.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:48:45.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224223+0000", 2024-09-11T03:48:45.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000843771 2024-09-11T03:48:45.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224293+0000", 2024-09-11T03:48:45.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0339000000000005e-05 2024-09-11T03:48:45.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:48:45.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224347+0000", 2024-09-11T03:48:45.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3288999999999998e-05 2024-09-11T03:48:45.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224391+0000", 2024-09-11T03:48:45.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4554000000000001e-05 2024-09-11T03:48:45.862 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.862 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.862 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.863 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1635 2.7 2:e23d1762:::smithi17557870-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:48:45.863 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:36.207063+0000", 2024-09-11T03:48:45.863 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 489.59643374500001, 2024-09-11T03:48:45.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013462717000000001, 2024-09-11T03:48:45.863 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.863 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.863 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:48:45.863 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:48:45.863 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:48:45.863 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:45.863 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1635 2024-09-11T03:48:45.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.864 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207063+0000", 2024-09-11T03:48:45.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207063+0000", 2024-09-11T03:48:45.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207064+0000", 2024-09-11T03:48:45.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.043e-06 2024-09-11T03:48:45.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207071+0000", 2024-09-11T03:48:45.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8929999999999999e-06 2024-09-11T03:48:45.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207086+0000", 2024-09-11T03:48:45.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4596999999999999e-05 2024-09-11T03:48:45.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207095+0000", 2024-09-11T03:48:45.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9930000000000001e-06 2024-09-11T03:48:45.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207386+0000", 2024-09-11T03:48:45.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029111399999999998 2024-09-11T03:48:45.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:48:45.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207423+0000", 2024-09-11T03:48:45.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6825000000000001e-05 2024-09-11T03:48:45.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.220376+0000", 2024-09-11T03:48:45.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012953549 2024-09-11T03:48:45.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.220446+0000", 2024-09-11T03:48:45.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9578000000000004e-05 2024-09-11T03:48:45.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.220526+0000", 2024-09-11T03:48:45.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0124999999999996e-05 2024-09-11T03:48:45.868 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.868 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.868 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.868 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1866 2.e 2:7b0c3397:::smithi17557870-14:head [copy-from ver 238] snapc 4c=[4c,4b,49,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:48:45.868 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:48:45.868 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 463.95336390800003, 2024-09-11T03:48:45.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012778199000000001, 2024-09-11T03:48:45.868 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.868 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.869 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:45.869 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:48:45.869 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:48:45.869 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:45.869 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1866 2024-09-11T03:48:45.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.869 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:48:45.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:48:45.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850135+0000", 2024-09-11T03:48:45.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5850000000000001e-06 2024-09-11T03:48:45.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850142+0000", 2024-09-11T03:48:45.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2010000000000002e-06 2024-09-11T03:48:45.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850160+0000", 2024-09-11T03:48:45.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7858000000000001e-05 2024-09-11T03:48:45.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850170+0000", 2024-09-11T03:48:45.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9219999999999992e-06 2024-09-11T03:48:45.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850235+0000", 2024-09-11T03:48:45.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4849000000000002e-05 2024-09-11T03:48:45.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:48:45.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850246+0000", 2024-09-11T03:48:45.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1938e-05 2024-09-11T03:48:45.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.851390+0000", 2024-09-11T03:48:45.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011435219999999999 2024-09-11T03:48:45.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.851491+0000", 2024-09-11T03:48:45.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000100868 2024-09-11T03:48:45.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:48:45.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.857380+0000", 2024-09-11T03:48:45.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058892839999999998 2024-09-11T03:48:45.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:48:45.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.861564+0000", 2024-09-11T03:48:45.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041843369999999998 2024-09-11T03:48:45.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.861883+0000", 2024-09-11T03:48:45.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031904399999999999 2024-09-11T03:48:45.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.862846+0000", 2024-09-11T03:48:45.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096234000000000001 2024-09-11T03:48:45.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:48:45.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.862883+0000", 2024-09-11T03:48:45.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7132000000000003e-05 2024-09-11T03:48:45.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.862911+0000", 2024-09-11T03:48:45.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8319000000000001e-05 2024-09-11T03:48:45.875 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.875 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.875 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.875 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1895 2.e 2:7efbbea8:::smithi17557870-24:head [copy-from ver 160] snapc 4e=[4e,4d,49,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-09-11T03:48:45.875 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:05.924255+0000", 2024-09-11T03:48:45.875 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 459.879241912, 2024-09-11T03:48:45.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016112593000000001, 2024-09-11T03:48:45.875 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.875 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.876 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:45.876 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:48:45.876 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:48:45.876 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:45.876 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1895 2024-09-11T03:48:45.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.876 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924255+0000", 2024-09-11T03:48:45.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924255+0000", 2024-09-11T03:48:45.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.877 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:45.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924257+0000", 2024-09-11T03:48:45.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.742e-06 2024-09-11T03:48:45.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924264+0000", 2024-09-11T03:48:45.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6090000000000003e-06 2024-09-11T03:48:45.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924281+0000", 2024-09-11T03:48:45.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6157000000000001e-05 2024-09-11T03:48:45.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924290+0000", 2024-09-11T03:48:45.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0499999999999997e-06 2024-09-11T03:48:45.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924352+0000", 2024-09-11T03:48:45.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1968e-05 2024-09-11T03:48:45.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:48:45.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924358+0000", 2024-09-11T03:48:45.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1380000000000004e-06 2024-09-11T03:48:45.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.925553+0000", 2024-09-11T03:48:45.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001194802 2024-09-11T03:48:45.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.925622+0000", 2024-09-11T03:48:45.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9598999999999996e-05 2024-09-11T03:48:45.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:48:45.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.931889+0000", 2024-09-11T03:48:45.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062668419999999999 2024-09-11T03:48:45.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:48:45.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.937406+0000", 2024-09-11T03:48:45.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055165659999999997 2024-09-11T03:48:45.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.937537+0000", 2024-09-11T03:48:45.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000131941 2024-09-11T03:48:45.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.940300+0000", 2024-09-11T03:48:45.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027624720000000002 2024-09-11T03:48:45.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:48:45.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.940337+0000", 2024-09-11T03:48:45.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6869999999999998e-05 2024-09-11T03:48:45.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.940368+0000", 2024-09-11T03:48:45.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0837000000000003e-05 2024-09-11T03:48:45.883 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.883 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.883 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.883 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2565 2.7 2:e0eabcfb:::smithi17557870-9:head [copy-from ver 223] snapc 70=[70,6f,6e,6c,6a,68,65,63,5d] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-11T03:48:45.883 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:09.876844+0000", 2024-09-11T03:48:45.883 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 395.92665337599999, 2024-09-11T03:48:45.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015411238000000001, 2024-09-11T03:48:45.883 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.883 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.883 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:45.884 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:48:45.884 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:48:45.884 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:45.884 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2565 2024-09-11T03:48:45.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.884 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876844+0000", 2024-09-11T03:48:45.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876844+0000", 2024-09-11T03:48:45.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876845+0000", 2024-09-11T03:48:45.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.78e-07 2024-09-11T03:48:45.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876850+0000", 2024-09-11T03:48:45.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9540000000000003e-06 2024-09-11T03:48:45.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876863+0000", 2024-09-11T03:48:45.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3349e-05 2024-09-11T03:48:45.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876873+0000", 2024-09-11T03:48:45.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9590000000000007e-06 2024-09-11T03:48:45.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876919+0000", 2024-09-11T03:48:45.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5720000000000003e-05 2024-09-11T03:48:45.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876984+0000", 2024-09-11T03:48:45.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5437000000000004e-05 2024-09-11T03:48:45.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:48:45.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.887314+0000", 2024-09-11T03:48:45.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0103298 2024-09-11T03:48:45.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:48:45.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.891590+0000", 2024-09-11T03:48:45.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042757380000000003 2024-09-11T03:48:45.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.891885+0000", 2024-09-11T03:48:45.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029539200000000001 2024-09-11T03:48:45.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.892177+0000", 2024-09-11T03:48:45.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029223699999999999 2024-09-11T03:48:45.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:48:45.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.892215+0000", 2024-09-11T03:48:45.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7740000000000001e-05 2024-09-11T03:48:45.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.892255+0000", 2024-09-11T03:48:45.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9934000000000001e-05 2024-09-11T03:48:45.891 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.891 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.891 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.891 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2566 2.7 2:e0eabcfb:::smithi17557870-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e313)", 2024-09-11T03:48:45.891 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:09.876886+0000", 2024-09-11T03:48:45.891 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 395.92661118799998, 2024-09-11T03:48:45.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014077338, 2024-09-11T03:48:45.891 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.891 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.892 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:48:45.892 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:48:45.892 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:48:45.892 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:45.892 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2566 2024-09-11T03:48:45.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.892 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876886+0000", 2024-09-11T03:48:45.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876886+0000", 2024-09-11T03:48:45.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876886+0000", 2024-09-11T03:48:45.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7000000000000005e-07 2024-09-11T03:48:45.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876889+0000", 2024-09-11T03:48:45.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5730000000000002e-06 2024-09-11T03:48:45.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876895+0000", 2024-09-11T03:48:45.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4160000000000001e-06 2024-09-11T03:48:45.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876899+0000", 2024-09-11T03:48:45.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3060000000000002e-06 2024-09-11T03:48:45.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.877088+0000", 2024-09-11T03:48:45.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018935499999999999 2024-09-11T03:48:45.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:48:45.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.877121+0000", 2024-09-11T03:48:45.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2566e-05 2024-09-11T03:48:45.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.890821+0000", 2024-09-11T03:48:45.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013699932999999999 2024-09-11T03:48:45.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.897 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.890886+0000", 2024-09-11T03:48:45.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5129999999999995e-05 2024-09-11T03:48:45.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.897 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.890963+0000", 2024-09-11T03:48:45.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7489000000000003e-05 2024-09-11T03:48:45.897 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.897 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.897 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.898 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:4498 2.e 2:7b0c3397:::smithi17557870-14:head [copy-from ver 491] snapc c5=[c5,c4,c3,c2,c1,c0,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-09-11T03:48:45.898 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:45:03.958685+0000", 2024-09-11T03:48:45.898 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 221.844811537, 2024-09-11T03:48:45.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014061295999999999, 2024-09-11T03:48:45.898 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.898 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.898 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:45.898 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:48:45.898 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:48:45.898 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:45.898 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4498 2024-09-11T03:48:45.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.898 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958685+0000", 2024-09-11T03:48:45.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958685+0000", 2024-09-11T03:48:45.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958687+0000", 2024-09-11T03:48:45.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.609e-06 2024-09-11T03:48:45.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958695+0000", 2024-09-11T03:48:45.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5410000000000003e-06 2024-09-11T03:48:45.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958711+0000", 2024-09-11T03:48:45.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6719999999999999e-05 2024-09-11T03:48:45.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958718+0000", 2024-09-11T03:48:45.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9720000000000003e-06 2024-09-11T03:48:45.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958804+0000", 2024-09-11T03:48:45.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5483999999999997e-05 2024-09-11T03:48:45.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:48:45.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958811+0000", 2024-09-11T03:48:45.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4059999999999999e-06 2024-09-11T03:48:45.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.960161+0000", 2024-09-11T03:48:45.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001349772 2024-09-11T03:48:45.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.960244+0000", 2024-09-11T03:48:45.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3461999999999995e-05 2024-09-11T03:48:45.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:48:45.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.967582+0000", 2024-09-11T03:48:45.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073378480000000001 2024-09-11T03:48:45.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:48:45.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.970934+0000", 2024-09-11T03:48:45.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003351812 2024-09-11T03:48:45.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.971224+0000", 2024-09-11T03:48:45.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029005399999999998 2024-09-11T03:48:45.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.972661+0000", 2024-09-11T03:48:45.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001436866 2024-09-11T03:48:45.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:48:45.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.972700+0000", 2024-09-11T03:48:45.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9341000000000002e-05 2024-09-11T03:48:45.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.972747+0000", 2024-09-11T03:48:45.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6409000000000002e-05 2024-09-11T03:48:45.905 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.905 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.905 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.905 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5201 2.e 2:750de3b7:::smithi17557870-8:head [copy-from ver 512] snapc e6=[e5,e1,e0,da,d1] ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-09-11T03:48:45.905 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:12.961903+0000", 2024-09-11T03:48:45.905 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 152.84159381200001, 2024-09-11T03:48:45.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022459643000000001, 2024-09-11T03:48:45.906 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.906 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.906 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:45.906 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:48:45.906 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:48:45.906 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:45.906 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5201 2024-09-11T03:48:45.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.906 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961903+0000", 2024-09-11T03:48:45.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961903+0000", 2024-09-11T03:48:45.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961903+0000", 2024-09-11T03:48:45.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0199999999999998e-07 2024-09-11T03:48:45.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961906+0000", 2024-09-11T03:48:45.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.119e-06 2024-09-11T03:48:45.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961911+0000", 2024-09-11T03:48:45.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7529999999999998e-06 2024-09-11T03:48:45.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961915+0000", 2024-09-11T03:48:45.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6320000000000001e-06 2024-09-11T03:48:45.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961955+0000", 2024-09-11T03:48:45.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0497999999999997e-05 2024-09-11T03:48:45.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:48:45.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961961+0000", 2024-09-11T03:48:45.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3800000000000002e-06 2024-09-11T03:48:45.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.963330+0000", 2024-09-11T03:48:45.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001368773 2024-09-11T03:48:45.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.963415+0000", 2024-09-11T03:48:45.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5579000000000002e-05 2024-09-11T03:48:45.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:48:45.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.970784+0000", 2024-09-11T03:48:45.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073691750000000004 2024-09-11T03:48:45.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:48:45.911 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.975393+0000", 2024-09-11T03:48:45.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046089010000000003 2024-09-11T03:48:45.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.911 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.976463+0000", 2024-09-11T03:48:45.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010695850000000001 2024-09-11T03:48:45.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:48:45.911 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.984271+0000", 2024-09-11T03:48:45.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0078078970000000003 2024-09-11T03:48:45.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.912 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:48:45.912 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.984316+0000", 2024-09-11T03:48:45.912 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5209e-05 2024-09-11T03:48:45.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.912 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.912 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.984363+0000", 2024-09-11T03:48:45.912 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6839999999999999e-05 2024-09-11T03:48:45.912 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.912 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.912 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.913 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5202 2.e 2:750de3b7:::smithi17557870-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e555)", 2024-09-11T03:48:45.913 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:12.961921+0000", 2024-09-11T03:48:45.913 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 152.84157561699999, 2024-09-11T03:48:45.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013412446999999999, 2024-09-11T03:48:45.913 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.913 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.913 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:48:45.913 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:48:45.913 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:48:45.913 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:48:45.913 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5202 2024-09-11T03:48:45.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.913 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961921+0000", 2024-09-11T03:48:45.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961921+0000", 2024-09-11T03:48:45.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961922+0000", 2024-09-11T03:48:45.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2499999999999999e-07 2024-09-11T03:48:45.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961923+0000", 2024-09-11T03:48:45.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.42e-06 2024-09-11T03:48:45.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961927+0000", 2024-09-11T03:48:45.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9960000000000004e-06 2024-09-11T03:48:45.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961930+0000", 2024-09-11T03:48:45.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9610000000000001e-06 2024-09-11T03:48:45.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961983+0000", 2024-09-11T03:48:45.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2967000000000003e-05 2024-09-11T03:48:45.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:48:45.917 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961986+0000", 2024-09-11T03:48:45.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0960000000000001e-06 2024-09-11T03:48:45.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.917 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.963568+0000", 2024-09-11T03:48:45.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015823530000000001 2024-09-11T03:48:45.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:48:45.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.963621+0000", 2024-09-11T03:48:45.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2358000000000003e-05 2024-09-11T03:48:45.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.974667+0000", 2024-09-11T03:48:45.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011046517 2024-09-11T03:48:45.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.974875+0000", 2024-09-11T03:48:45.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000207853 2024-09-11T03:48:45.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.975334+0000", 2024-09-11T03:48:45.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000458701 2024-09-11T03:48:45.919 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.919 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.919 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.919 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:5691 2.e e601/577 2:77dadb97:::smithi17557870-30:head v 601'877, mlcod=598'875)", 2024-09-11T03:48:45.919 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:59.009702+0000", 2024-09-11T03:48:45.919 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 106.79379502, 2024-09-11T03:48:45.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051323566000000001, 2024-09-11T03:48:45.919 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.919 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.920 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:45.920 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.009702+0000", 2024-09-11T03:48:45.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.009702+0000", 2024-09-11T03:48:45.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.009703+0000", 2024-09-11T03:48:45.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4750000000000001e-06 2024-09-11T03:48:45.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.059761+0000", 2024-09-11T03:48:45.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.050057579999999997 2024-09-11T03:48:45.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.059778+0000", 2024-09-11T03:48:45.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7051999999999999e-05 2024-09-11T03:48:45.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.059784+0000", 2024-09-11T03:48:45.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.648e-06 2024-09-11T03:48:45.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.059832+0000", 2024-09-11T03:48:45.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7828999999999999e-05 2024-09-11T03:48:45.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.059847+0000", 2024-09-11T03:48:45.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5778999999999999e-05 2024-09-11T03:48:45.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:48:45.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.060993+0000", 2024-09-11T03:48:45.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001145566 2024-09-11T03:48:45.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.061026+0000", 2024-09-11T03:48:45.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2636999999999999e-05 2024-09-11T03:48:45.923 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.923 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.924 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.924 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:6152 2.e e637/577 2:77dadb97:::smithi17557870-30:head v 637'891, mlcod=634'888)", 2024-09-11T03:48:45.924 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:47:35.714649+0000", 2024-09-11T03:48:45.924 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 70.088847517000005, 2024-09-11T03:48:45.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011875179, 2024-09-11T03:48:45.924 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.924 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.924 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:45.924 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.925 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714649+0000", 2024-09-11T03:48:45.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.925 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714649+0000", 2024-09-11T03:48:45.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714650+0000", 2024-09-11T03:48:45.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.44e-07 2024-09-11T03:48:45.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721021+0000", 2024-09-11T03:48:45.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0063714569999999996 2024-09-11T03:48:45.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721040+0000", 2024-09-11T03:48:45.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8590999999999999e-05 2024-09-11T03:48:45.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721048+0000", 2024-09-11T03:48:45.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9489999999999996e-06 2024-09-11T03:48:45.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721107+0000", 2024-09-11T03:48:45.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8736999999999997e-05 2024-09-11T03:48:45.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721123+0000", 2024-09-11T03:48:45.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6735000000000002e-05 2024-09-11T03:48:45.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:48:45.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.726469+0000", 2024-09-11T03:48:45.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053456140000000003 2024-09-11T03:48:45.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.726525+0000", 2024-09-11T03:48:45.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5551999999999999e-05 2024-09-11T03:48:45.929 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.929 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.929 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.929 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:6546 2.e e669/577 2:750de3b7:::smithi17557870-8:head v 669'909, mlcod=667'905)", 2024-09-11T03:48:45.929 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:07.227022+0000", 2024-09-11T03:48:45.929 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 38.576474556999997, 2024-09-11T03:48:45.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013091959, 2024-09-11T03:48:45.929 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:48:45.930 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:48:45.930 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:48:45.930 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:48:45.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:48:45.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.227022+0000", 2024-09-11T03:48:45.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:48:45.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.227022+0000", 2024-09-11T03:48:45.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:48:45.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:48:45.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.227023+0000", 2024-09-11T03:48:45.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7300000000000001e-07 2024-09-11T03:48:45.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:48:45.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.234175+0000", 2024-09-11T03:48:45.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071522260000000002 2024-09-11T03:48:45.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:48:45.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.234198+0000", 2024-09-11T03:48:45.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2492000000000001e-05 2024-09-11T03:48:45.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:48:45.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.234211+0000", 2024-09-11T03:48:45.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3534e-05 2024-09-11T03:48:45.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:48:45.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.234289+0000", 2024-09-11T03:48:45.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8078999999999996e-05 2024-09-11T03:48:45.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:48:45.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.234307+0000", 2024-09-11T03:48:45.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8206000000000002e-05 2024-09-11T03:48:45.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:48:45.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.240060+0000", 2024-09-11T03:48:45.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0057523069999999999 2024-09-11T03:48:45.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:48:45.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:48:45.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:48:45.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.240114+0000", 2024-09-11T03:48:45.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4642000000000003e-05 2024-09-11T03:48:45.934 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.935 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.935 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.935 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:48:45.935 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:48:45.935 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:48:45.935 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:45.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:45.958 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:45.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:46.021 INFO:tasks.rados.rados.2.smithi175.stdout:2817: done (1 left) 2024-09-11T03:48:46.021 INFO:tasks.rados.rados.2.smithi175.stdout:2818: done (0 left) 2024-09-11T03:48:46.021 INFO:tasks.rados.rados.2.smithi175.stdout:2819: snap_remove snap 301 2024-09-11T03:48:46.059 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:46.080 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:46.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:46.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:46.182 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:46.203 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:46.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:46.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:46.305 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:46.326 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:46.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:46.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:46.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:46 smithi084 ceph-mon[27254]: pgmap v1215: 67 pgs: 67 active+clean; 284 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-11T03:48:46.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:46 smithi084 ceph-mon[27254]: osdmap e707: 12 total, 12 up, 12 in 2024-09-11T03:48:46.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1040206322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:46 smithi121 ceph-mon[30729]: pgmap v1215: 67 pgs: 67 active+clean; 284 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-11T03:48:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:46 smithi121 ceph-mon[30729]: osdmap e707: 12 total, 12 up, 12 in 2024-09-11T03:48:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1040206322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:46.427 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:46.453 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:46.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:46.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:46 smithi175 ceph-mon[30894]: pgmap v1215: 67 pgs: 67 active+clean; 284 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-11T03:48:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:46 smithi175 ceph-mon[30894]: osdmap e707: 12 total, 12 up, 12 in 2024-09-11T03:48:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1040206322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:46.555 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:46.580 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:46.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:46.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:46.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:46.708 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:46.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:46.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:46.810 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:46.836 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:46.837 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:46.938 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:46.964 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:46.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:46.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:47.005 INFO:tasks.rados.rados.2.smithi175.stdout:2819: done (0 left) 2024-09-11T03:48:47.005 INFO:tasks.rados.rados.2.smithi175.stdout:2820: copy_from oid 48 from oid 32 current snap is 313 2024-09-11T03:48:47.006 INFO:tasks.rados.rados.2.smithi175.stdout:2821: copy_from oid 43 from oid 7 current snap is 313 2024-09-11T03:48:47.006 INFO:tasks.rados.rados.2.smithi175.stdout:2822: copy_from oid 26 from oid 27 current snap is 313 2024-09-11T03:48:47.006 INFO:tasks.rados.rados.2.smithi175.stdout:2823: copy_from oid 23 from oid 49 current snap is 313 2024-09-11T03:48:47.006 INFO:tasks.rados.rados.2.smithi175.stdout:2824: copy_from oid 35 from oid 1 current snap is 313 2024-09-11T03:48:47.006 INFO:tasks.rados.rados.2.smithi175.stdout:2825: snap_remove snap 302 2024-09-11T03:48:47.066 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:47.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:47.088 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:47.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:47.190 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:47.211 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:47.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:47.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:47.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:47.342 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:47.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:47.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:47.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:47 smithi084 ceph-mon[27254]: osdmap e708: 12 total, 12 up, 12 in 2024-09-11T03:48:47.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:47 smithi084 ceph-mon[27254]: osdmap e709: 12 total, 12 up, 12 in 2024-09-11T03:48:47.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:47 smithi121 ceph-mon[30729]: osdmap e708: 12 total, 12 up, 12 in 2024-09-11T03:48:47.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:47 smithi121 ceph-mon[30729]: osdmap e709: 12 total, 12 up, 12 in 2024-09-11T03:48:47.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:47.469 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T03:48:47.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:47.470 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:47.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:47 smithi175 ceph-mon[30894]: osdmap e708: 12 total, 12 up, 12 in 2024-09-11T03:48:47.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:47 smithi175 ceph-mon[30894]: osdmap e709: 12 total, 12 up, 12 in 2024-09-11T03:48:47.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:47.593 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:47.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:47.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:47.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:47.720 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:47.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:47.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:47.822 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:47.848 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:47.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:47.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:47.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:47.975 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:47.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:47.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:48.005 INFO:tasks.rados.rados.2.smithi175.stdout:2825: done (5 left) 2024-09-11T03:48:48.005 INFO:tasks.rados.rados.2.smithi175.stdout:2826: write (excl) oid 6 current snap is 313 2024-09-11T03:48:48.005 INFO:tasks.rados.rados.2.smithi175.stdout:2823: finishing copy_from to smithi17557870-23 2024-09-11T03:48:48.005 INFO:tasks.rados.rados.2.smithi175.stdout:2823: got expected ENOENT (src dne) 2024-09-11T03:48:48.005 INFO:tasks.rados.rados.2.smithi175.stdout:2826: seq_num 972 ranges {736404=718859,1929321=407083} 2024-09-11T03:48:48.013 INFO:tasks.rados.rados.2.smithi175.stdout:2826: writing smithi17557870-6 from 736404 to 1455263 tid 1 2024-09-11T03:48:48.020 INFO:tasks.rados.rados.2.smithi175.stdout:2826: writing smithi17557870-6 from 1929321 to 2336404 tid 2 2024-09-11T03:48:48.020 INFO:tasks.rados.rados.2.smithi175.stdout:2827: copy_from oid 25 from oid 15 current snap is 313 2024-09-11T03:48:48.020 INFO:tasks.rados.rados.2.smithi175.stdout:2823: finishing copy_from racing read to smithi17557870-23 2024-09-11T03:48:48.020 INFO:tasks.rados.rados.2.smithi175.stdout:2820: finishing copy_from racing read to smithi17557870-48 2024-09-11T03:48:48.020 INFO:tasks.rados.rados.2.smithi175.stdout:2824: finishing copy_from racing read to smithi17557870-35 2024-09-11T03:48:48.020 INFO:tasks.rados.rados.2.smithi175.stdout:2823: done (6 left) 2024-09-11T03:48:48.020 INFO:tasks.rados.rados.2.smithi175.stdout:2828: read oid 37 snap -1 2024-09-11T03:48:48.020 INFO:tasks.rados.rados.2.smithi175.stdout:2828: expect (ObjNum 867 snap 273 seq_num 867) 2024-09-11T03:48:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:2829: rollback oid 10 current snap is 313 2024-09-11T03:48:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 10 to 296 2024-09-11T03:48:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:2824: finishing copy_from to smithi17557870-35 2024-09-11T03:48:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 947 (ObjNum 758 snap 246 seq_num 758) dirty exists 2024-09-11T03:48:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:2820: finishing copy_from to smithi17557870-48 2024-09-11T03:48:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 962 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:48:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:2822: finishing copy_from racing read to smithi17557870-26 2024-09-11T03:48:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:2821: finishing copy_from racing read to smithi17557870-43 2024-09-11T03:48:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:2822: finishing copy_from to smithi17557870-26 2024-09-11T03:48:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 950 (ObjNum 765 snap 247 seq_num 765) dirty exists 2024-09-11T03:48:48.021 INFO:tasks.rados.rados.2.smithi175.stdout:2821: finishing copy_from to smithi17557870-43 2024-09-11T03:48:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 979 (ObjNum 925 snap 294 seq_num 925) dirty exists 2024-09-11T03:48:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2826: finishing write tid 1 to smithi17557870-6 2024-09-11T03:48:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2820: done (7 left) 2024-09-11T03:48:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2821: done (6 left) 2024-09-11T03:48:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2822: done (5 left) 2024-09-11T03:48:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2824: done (4 left) 2024-09-11T03:48:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2830: read oid 8 snap -1 2024-09-11T03:48:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2830: expect (ObjNum 942 snap 297 seq_num 942) 2024-09-11T03:48:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2831: write (excl) oid 9 current snap is 313 2024-09-11T03:48:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:2831: seq_num 973 ranges {753211=794108,1987803=365408} 2024-09-11T03:48:48.027 INFO:tasks.rados.rados.2.smithi175.stdout:2831: writing smithi17557870-9 from 753211 to 1547319 tid 1 2024-09-11T03:48:48.032 INFO:tasks.rados.rados.2.smithi175.stdout:2831: writing smithi17557870-9 from 1987803 to 2353211 tid 2 2024-09-11T03:48:48.033 INFO:tasks.rados.rados.2.smithi175.stdout:2832: copy_from oid 21 from oid 22 current snap is 313 2024-09-11T03:48:48.033 INFO:tasks.rados.rados.2.smithi175.stdout:2826: finishing write tid 2 to smithi17557870-6 2024-09-11T03:48:48.033 INFO:tasks.rados.rados.2.smithi175.stdout:2826: finishing write tid 3 to smithi17557870-6 2024-09-11T03:48:48.033 INFO:tasks.rados.rados.2.smithi175.stdout:2833: read oid 44 snap 289 2024-09-11T03:48:48.033 INFO:tasks.rados.rados.2.smithi175.stdout:2833: expect deleted 2024-09-11T03:48:48.033 INFO:tasks.rados.rados.2.smithi175.stdout:2834: write oid 34 current snap is 313 2024-09-11T03:48:48.033 INFO:tasks.rados.rados.2.smithi175.stdout:2830: expect (ObjNum 942 snap 297 seq_num 942) 2024-09-11T03:48:48.078 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:48.100 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:48.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:48.101 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:48.202 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:48.219 INFO:tasks.rados.rados.2.smithi175.stdout:2826: finishing write tid 4 to smithi17557870-6 2024-09-11T03:48:48.219 INFO:tasks.rados.rados.2.smithi175.stdout:2826: oid 6 updating version 0 to 892 2024-09-11T03:48:48.219 INFO:tasks.rados.rados.2.smithi175.stdout:2826: oid 6 updating version 892 to 893 2024-09-11T03:48:48.219 INFO:tasks.rados.rados.2.smithi175.stdout:2826: oid 6 updating version 893 to 894 2024-09-11T03:48:48.219 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 894 (ObjNum 972 snap 313 seq_num 972) dirty exists 2024-09-11T03:48:48.220 INFO:tasks.rados.rados.2.smithi175.stdout:2826: left oid 6 (ObjNum 972 snap 313 seq_num 972) 2024-09-11T03:48:48.220 INFO:tasks.rados.rados.2.smithi175.stdout:2829: finishing rollback tid 0 to smithi17557870-10 2024-09-11T03:48:48.220 INFO:tasks.rados.rados.2.smithi175.stdout:2828: expect (ObjNum 867 snap 273 seq_num 867) 2024-09-11T03:48:48.224 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:48.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:48.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:48.238 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-11T03:48:48.238 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-11T03:48:48.238 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-11T03:48:48.238 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-11T03:48:48.238 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-11T03:48:48.238 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-11T03:48:48.238 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-11T03:48:48.238 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-11T03:48:48.238 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-11T03:48:48.238 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-11T03:48:48.238 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-11T03:48:48.239 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-11T03:48:48.242 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:48.301 INFO:tasks.rados.rados.2.smithi175.stdout:2829: finishing rollback tid 1 to smithi17557870-10 2024-09-11T03:48:48.301 INFO:tasks.rados.rados.2.smithi175.stdout:2829: finishing rollback tid 2 to smithi17557870-10 2024-09-11T03:48:48.301 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 916 (ObjNum 736 snap 243 seq_num 736) dirty exists 2024-09-11T03:48:48.301 INFO:tasks.rados.rados.2.smithi175.stdout:2834: seq_num 974 ranges {770018=585710,2046285=323733} 2024-09-11T03:48:48.308 INFO:tasks.rados.rados.2.smithi175.stdout:2834: writing smithi17557870-34 from 770018 to 1355728 tid 1 2024-09-11T03:48:48.314 INFO:tasks.rados.rados.2.smithi175.stdout:2834: writing smithi17557870-34 from 2046285 to 2370018 tid 2 2024-09-11T03:48:48.314 INFO:tasks.rados.rados.2.smithi175.stdout:2826: done (8 left) 2024-09-11T03:48:48.314 INFO:tasks.rados.rados.2.smithi175.stdout:2828: done (7 left) 2024-09-11T03:48:48.314 INFO:tasks.rados.rados.2.smithi175.stdout:2829: done (6 left) 2024-09-11T03:48:48.314 INFO:tasks.rados.rados.2.smithi175.stdout:2830: done (5 left) 2024-09-11T03:48:48.314 INFO:tasks.rados.rados.2.smithi175.stdout:2833: done (4 left) 2024-09-11T03:48:48.315 INFO:tasks.rados.rados.2.smithi175.stdout:2835: write oid 24 current snap is 313 2024-09-11T03:48:48.315 INFO:tasks.rados.rados.2.smithi175.stdout:2827: finishing copy_from racing read to smithi17557870-25 2024-09-11T03:48:48.315 INFO:tasks.rados.rados.2.smithi175.stdout:2831: finishing write tid 1 to smithi17557870-9 2024-09-11T03:48:48.315 INFO:tasks.rados.rados.2.smithi175.stdout:2827: finishing copy_from to smithi17557870-25 2024-09-11T03:48:48.315 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 893 (ObjNum 950 snap 303 seq_num 950) dirty exists 2024-09-11T03:48:48.315 INFO:tasks.rados.rados.2.smithi175.stdout:2831: finishing write tid 2 to smithi17557870-9 2024-09-11T03:48:48.315 INFO:tasks.rados.rados.2.smithi175.stdout:2831: finishing write tid 3 to smithi17557870-9 2024-09-11T03:48:48.315 INFO:tasks.rados.rados.2.smithi175.stdout:2831: finishing write tid 4 to smithi17557870-9 2024-09-11T03:48:48.315 INFO:tasks.rados.rados.2.smithi175.stdout:2831: oid 9 updating version 0 to 1005 2024-09-11T03:48:48.315 INFO:tasks.rados.rados.2.smithi175.stdout:2831: oid 9 updating version 1005 to 1006 2024-09-11T03:48:48.315 INFO:tasks.rados.rados.2.smithi175.stdout:2831: oid 9 version 1006 is already newer than 1004 2024-09-11T03:48:48.315 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 1006 (ObjNum 973 snap 313 seq_num 973) dirty exists 2024-09-11T03:48:48.315 INFO:tasks.rados.rados.2.smithi175.stdout:2831: left oid 9 (ObjNum 973 snap 313 seq_num 973) 2024-09-11T03:48:48.316 INFO:tasks.rados.rados.2.smithi175.stdout:2832: finishing copy_from racing read to smithi17557870-21 2024-09-11T03:48:48.316 INFO:tasks.rados.rados.2.smithi175.stdout:2832: finishing copy_from to smithi17557870-21 2024-09-11T03:48:48.316 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 930 (ObjNum 829 snap 263 seq_num 829) dirty exists 2024-09-11T03:48:48.316 INFO:tasks.rados.rados.2.smithi175.stdout:2835: seq_num 975 ranges {786825=660959,2104767=282058} 2024-09-11T03:48:48.321 INFO:tasks.rados.rados.2.smithi175.stdout:2835: writing smithi17557870-24 from 786825 to 1447784 tid 1 2024-09-11T03:48:48.326 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:48.327 INFO:tasks.rados.rados.2.smithi175.stdout:2835: writing smithi17557870-24 from 2104767 to 2386825 tid 2 2024-09-11T03:48:48.327 INFO:tasks.rados.rados.2.smithi175.stdout:2827: done (4 left) 2024-09-11T03:48:48.327 INFO:tasks.rados.rados.2.smithi175.stdout:2831: done (3 left) 2024-09-11T03:48:48.327 INFO:tasks.rados.rados.2.smithi175.stdout:2832: done (2 left) 2024-09-11T03:48:48.327 INFO:tasks.rados.rados.2.smithi175.stdout:2836: delete oid 42 current snap is 313 2024-09-11T03:48:48.327 INFO:tasks.rados.rados.2.smithi175.stdout:2834: finishing write tid 1 to smithi17557870-34 2024-09-11T03:48:48.327 INFO:tasks.rados.rados.2.smithi175.stdout:2834: finishing write tid 2 to smithi17557870-34 2024-09-11T03:48:48.327 INFO:tasks.rados.rados.2.smithi175.stdout:2834: finishing write tid 3 to smithi17557870-34 2024-09-11T03:48:48.327 INFO:tasks.rados.rados.2.smithi175.stdout:2834: finishing write tid 4 to smithi17557870-34 2024-09-11T03:48:48.328 INFO:tasks.rados.rados.2.smithi175.stdout:2834: oid 34 updating version 0 to 952 2024-09-11T03:48:48.328 INFO:tasks.rados.rados.2.smithi175.stdout:2834: oid 34 updating version 952 to 953 2024-09-11T03:48:48.328 INFO:tasks.rados.rados.2.smithi175.stdout:2834: oid 34 version 953 is already newer than 951 2024-09-11T03:48:48.328 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 953 (ObjNum 974 snap 313 seq_num 974) dirty exists 2024-09-11T03:48:48.328 INFO:tasks.rados.rados.2.smithi175.stdout:2834: left oid 34 (ObjNum 974 snap 313 seq_num 974) 2024-09-11T03:48:48.328 INFO:tasks.rados.rados.2.smithi175.stdout:2835: finishing write tid 1 to smithi17557870-24 2024-09-11T03:48:48.330 INFO:tasks.rados.rados.2.smithi175.stdout:2834: done (2 left) 2024-09-11T03:48:48.330 INFO:tasks.rados.rados.2.smithi175.stdout:2836: done (1 left) 2024-09-11T03:48:48.330 INFO:tasks.rados.rados.2.smithi175.stdout:2837: delete oid 20 current snap is 313 2024-09-11T03:48:48.330 INFO:tasks.rados.rados.2.smithi175.stdout:2835: finishing write tid 2 to smithi17557870-24 2024-09-11T03:48:48.330 INFO:tasks.rados.rados.2.smithi175.stdout:2835: finishing write tid 3 to smithi17557870-24 2024-09-11T03:48:48.331 INFO:tasks.rados.rados.2.smithi175.stdout:2835: finishing write tid 4 to smithi17557870-24 2024-09-11T03:48:48.332 INFO:tasks.rados.rados.2.smithi175.stdout:2835: oid 24 updating version 0 to 962 2024-09-11T03:48:48.332 INFO:tasks.rados.rados.2.smithi175.stdout:2835: oid 24 version 962 is already newer than 961 2024-09-11T03:48:48.332 INFO:tasks.rados.rados.2.smithi175.stdout:2835: oid 24 updating version 962 to 963 2024-09-11T03:48:48.332 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 963 (ObjNum 975 snap 313 seq_num 975) dirty exists 2024-09-11T03:48:48.332 INFO:tasks.rados.rados.2.smithi175.stdout:2835: left oid 24 (ObjNum 975 snap 313 seq_num 975) 2024-09-11T03:48:48.332 INFO:tasks.rados.rados.2.smithi175.stdout:2835: done (1 left) 2024-09-11T03:48:48.332 INFO:tasks.rados.rados.2.smithi175.stdout:2837: done (0 left) 2024-09-11T03:48:48.332 INFO:tasks.rados.rados.2.smithi175.stdout:2838: snap_remove snap 285 2024-09-11T03:48:48.348 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:48.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:48.349 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:48 smithi084 ceph-mon[27254]: pgmap v1218: 67 pgs: 67 active+clean; 292 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-11T03:48:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:48 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:48:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:48 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:48:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:48 smithi084 ceph-mon[27254]: osdmap e710: 12 total, 12 up, 12 in 2024-09-11T03:48:48.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:48 smithi121 ceph-mon[30729]: pgmap v1218: 67 pgs: 67 active+clean; 292 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-11T03:48:48.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:48 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:48:48.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:48 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:48:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:48 smithi121 ceph-mon[30729]: osdmap e710: 12 total, 12 up, 12 in 2024-09-11T03:48:48.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:48.472 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:48.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:48.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:48 smithi175 ceph-mon[30894]: pgmap v1218: 67 pgs: 67 active+clean; 292 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-11T03:48:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:48 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:48:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:48 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:48:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:48 smithi175 ceph-mon[30894]: osdmap e710: 12 total, 12 up, 12 in 2024-09-11T03:48:48.574 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:48.596 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:48.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:48.597 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:48.698 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:48.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:48.720 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:48.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:48.821 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:48.828 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:48:48.828 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T03:48:48.828 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-11T03:48:48.842 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:48.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:48.843 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:48.859 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:48:48.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:48.968 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:48.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:48.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:49.031 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:48:49.035 INFO:tasks.rados.rados.2.smithi175.stdout:2838: done (0 left) 2024-09-11T03:48:49.035 INFO:tasks.rados.rados.2.smithi175.stdout:2839: read oid 41 snap -1 2024-09-11T03:48:49.035 INFO:tasks.rados.rados.2.smithi175.stdout:2839: expect deleted 2024-09-11T03:48:49.035 INFO:tasks.rados.rados.2.smithi175.stdout:2840: delete oid 14 current snap is 313 2024-09-11T03:48:49.041 INFO:tasks.rados.rados.2.smithi175.stdout:2839: done (1 left) 2024-09-11T03:48:49.041 INFO:tasks.rados.rados.2.smithi175.stdout:2840: done (0 left) 2024-09-11T03:48:49.041 INFO:tasks.rados.rados.2.smithi175.stdout:2841: rollback oid 35 current snap is 313 2024-09-11T03:48:49.041 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 35 to 286 2024-09-11T03:48:49.042 INFO:tasks.rados.rados.2.smithi175.stdout:2842: write (excl) oid 43 current snap is 313 2024-09-11T03:48:49.042 INFO:tasks.rados.rados.2.smithi175.stdout:2842: seq_num 979 ranges {454053=561955,1422342=483679,2368404=85649} 2024-09-11T03:48:49.047 INFO:tasks.rados.rados.2.smithi175.stdout:2842: writing smithi17557870-43 from 454053 to 1016008 tid 1 2024-09-11T03:48:49.051 INFO:tasks.rados.rados.2.smithi175.stdout:2842: writing smithi17557870-43 from 1422342 to 1906021 tid 2 2024-09-11T03:48:49.052 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2977534728' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:48:49.055 INFO:tasks.rados.rados.2.smithi175.stdout:2842: writing smithi17557870-43 from 2368404 to 2454053 tid 3 2024-09-11T03:48:49.055 INFO:tasks.rados.rados.2.smithi175.stdout:2843: copy_from oid 46 from oid 32 current snap is 313 2024-09-11T03:48:49.055 INFO:tasks.rados.rados.2.smithi175.stdout:2841: finishing rollback tid 0 to smithi17557870-35 2024-09-11T03:48:49.055 INFO:tasks.rados.rados.2.smithi175.stdout:2841: finishing rollback tid 1 to smithi17557870-35 2024-09-11T03:48:49.055 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 949 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T03:48:49.056 INFO:tasks.rados.rados.2.smithi175.stdout:2842: finishing write tid 1 to smithi17557870-43 2024-09-11T03:48:49.056 INFO:tasks.rados.rados.2.smithi175.stdout:2841: done (2 left) 2024-09-11T03:48:49.056 INFO:tasks.rados.rados.2.smithi175.stdout:2844: snap_create 2024-09-11T03:48:49.056 INFO:tasks.rados.rados.2.smithi175.stdout:2842: finishing write tid 2 to smithi17557870-43 2024-09-11T03:48:49.057 INFO:tasks.rados.rados.2.smithi175.stdout:2842: finishing write tid 3 to smithi17557870-43 2024-09-11T03:48:49.057 INFO:tasks.rados.rados.2.smithi175.stdout:2842: finishing write tid 4 to smithi17557870-43 2024-09-11T03:48:49.057 INFO:tasks.rados.rados.2.smithi175.stdout:2842: finishing write tid 5 to smithi17557870-43 2024-09-11T03:48:49.057 INFO:tasks.rados.rados.2.smithi175.stdout:2842: oid 43 updating version 0 to 981 2024-09-11T03:48:49.057 INFO:tasks.rados.rados.2.smithi175.stdout:2842: oid 43 version 981 is already newer than 980 2024-09-11T03:48:49.057 INFO:tasks.rados.rados.2.smithi175.stdout:2842: oid 43 updating version 981 to 982 2024-09-11T03:48:49.057 INFO:tasks.rados.rados.2.smithi175.stdout:2842: oid 43 updating version 982 to 983 2024-09-11T03:48:49.057 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 983 (ObjNum 979 snap 313 seq_num 979) dirty exists 2024-09-11T03:48:49.057 INFO:tasks.rados.rados.2.smithi175.stdout:2842: left oid 43 (ObjNum 979 snap 313 seq_num 979) 2024-09-11T03:48:49.061 INFO:tasks.rados.rados.2.smithi175.stdout:2843: finishing copy_from racing read to smithi17557870-46 2024-09-11T03:48:49.063 INFO:tasks.rados.rados.2.smithi175.stdout:2843: finishing copy_from to smithi17557870-46 2024-09-11T03:48:49.063 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 962 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:48:49.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:49.091 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:49.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:49.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:49.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:49.215 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:49.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:49.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:49.317 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:49.340 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:49.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:49.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2977534728' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:48:49.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:49.465 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:49.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:49.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2977534728' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:48:49.480 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:48:49.480 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:48:49.480 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:48:49.480 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:48:49.568 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:49.590 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:49.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:49.591 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:49.692 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:49.713 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:49.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:49.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:49.815 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:49.836 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:49.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:49.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:49.938 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:49.961 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:49.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:49.962 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:50.035 INFO:tasks.rados.rados.2.smithi175.stdout:2842: done (2 left) 2024-09-11T03:48:50.035 INFO:tasks.rados.rados.2.smithi175.stdout:2843: done (1 left) 2024-09-11T03:48:50.035 INFO:tasks.rados.rados.2.smithi175.stdout:2844: done (0 left) 2024-09-11T03:48:50.035 INFO:tasks.rados.rados.2.smithi175.stdout:2845: read oid 10 snap -1 2024-09-11T03:48:50.036 INFO:tasks.rados.rados.2.smithi175.stdout:2845: expect (ObjNum 736 snap 243 seq_num 736) 2024-09-11T03:48:50.036 INFO:tasks.rados.rados.2.smithi175.stdout:2846: snap_remove snap 289 2024-09-11T03:48:50.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:50.085 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:50.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:50.126 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:50 smithi084 ceph-mon[27254]: pgmap v1221: 67 pgs: 67 active+clean; 300 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 3.1 MiB/s wr, 17 op/s 2024-09-11T03:48:50.126 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:50 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:50.126 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:50 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2977534728' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:48:50.126 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:50 smithi084 ceph-mon[27254]: osdmap e711: 12 total, 12 up, 12 in 2024-09-11T03:48:50.126 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:50 smithi084 ceph-mon[27254]: 2.18 scrub starts 2024-09-11T03:48:50.126 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:50 smithi084 ceph-mon[27254]: 2.18 scrub ok 2024-09-11T03:48:50.153 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T03:48:50.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:50.214 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:50.219 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:50.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:50.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:50.320 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:50.343 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:50.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:50.344 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:50 smithi121 ceph-mon[30729]: pgmap v1221: 67 pgs: 67 active+clean; 300 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 3.1 MiB/s wr, 17 op/s 2024-09-11T03:48:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:50 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:50 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2977534728' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:48:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:50 smithi121 ceph-mon[30729]: osdmap e711: 12 total, 12 up, 12 in 2024-09-11T03:48:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:50 smithi121 ceph-mon[30729]: 2.18 scrub starts 2024-09-11T03:48:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:50 smithi121 ceph-mon[30729]: 2.18 scrub ok 2024-09-11T03:48:50.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:50.470 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:50.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:50.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:50.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:50 smithi175 ceph-mon[30894]: pgmap v1221: 67 pgs: 67 active+clean; 300 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 3.1 MiB/s wr, 17 op/s 2024-09-11T03:48:50.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:50 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:48:50.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:50 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2977534728' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:48:50.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:50 smithi175 ceph-mon[30894]: osdmap e711: 12 total, 12 up, 12 in 2024-09-11T03:48:50.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:50 smithi175 ceph-mon[30894]: 2.18 scrub starts 2024-09-11T03:48:50.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:50 smithi175 ceph-mon[30894]: 2.18 scrub ok 2024-09-11T03:48:50.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:50.594 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:50.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:50.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:50.696 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:50.721 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:50.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:50.722 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:50.784 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:50.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:50.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:50.849 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:50.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:50.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:50.976 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:50.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:51.077 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:51.091 INFO:tasks.rados.rados.2.smithi175.stdout:2846: done (1 left) 2024-09-11T03:48:51.091 INFO:tasks.rados.rados.2.smithi175.stdout:2847: write (excl) oid 49 current snap is 314 2024-09-11T03:48:51.091 INFO:tasks.rados.rados.2.smithi175.stdout:2847: seq_num 980 ranges {470860=637204,1880824=590036} 2024-09-11T03:48:51.098 INFO:tasks.rados.rados.2.smithi175.stdout:2847: writing smithi17557870-49 from 470860 to 1108064 tid 1 2024-09-11T03:48:51.107 INFO:tasks.rados.rados.2.smithi175.stdout:2847: writing smithi17557870-49 from 1880824 to 2470860 tid 2 2024-09-11T03:48:51.107 INFO:tasks.rados.rados.2.smithi175.stdout:2845: expect (ObjNum 736 snap 243 seq_num 736) 2024-09-11T03:48:51.112 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:51.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:51.113 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:51.190 INFO:tasks.rados.rados.2.smithi175.stdout:2847: finishing write tid 1 to smithi17557870-49 2024-09-11T03:48:51.190 INFO:tasks.rados.rados.2.smithi175.stdout:2847: finishing write tid 2 to smithi17557870-49 2024-09-11T03:48:51.190 INFO:tasks.rados.rados.2.smithi175.stdout:2847: finishing write tid 3 to smithi17557870-49 2024-09-11T03:48:51.190 INFO:tasks.rados.rados.2.smithi175.stdout:2847: finishing write tid 4 to smithi17557870-49 2024-09-11T03:48:51.191 INFO:tasks.rados.rados.2.smithi175.stdout:2847: oid 49 updating version 0 to 953 2024-09-11T03:48:51.191 INFO:tasks.rados.rados.2.smithi175.stdout:2847: oid 49 updating version 953 to 954 2024-09-11T03:48:51.191 INFO:tasks.rados.rados.2.smithi175.stdout:2847: oid 49 version 954 is already newer than 952 2024-09-11T03:48:51.191 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 954 (ObjNum 980 snap 314 seq_num 980) dirty exists 2024-09-11T03:48:51.191 INFO:tasks.rados.rados.2.smithi175.stdout:2847: left oid 49 (ObjNum 980 snap 314 seq_num 980) 2024-09-11T03:48:51.191 INFO:tasks.rados.rados.2.smithi175.stdout:2845: done (1 left) 2024-09-11T03:48:51.191 INFO:tasks.rados.rados.2.smithi175.stdout:2847: done (0 left) 2024-09-11T03:48:51.191 INFO:tasks.rados.rados.2.smithi175.stdout:2848: read oid 27 snap -1 2024-09-11T03:48:51.191 INFO:tasks.rados.rados.2.smithi175.stdout:2848: expect (ObjNum 765 snap 247 seq_num 765) 2024-09-11T03:48:51.191 INFO:tasks.rados.rados.2.smithi175.stdout:2849: delete oid 50 current snap is 314 2024-09-11T03:48:51.193 INFO:tasks.rados.rados.2.smithi175.stdout:2849: done (1 left) 2024-09-11T03:48:51.193 INFO:tasks.rados.rados.2.smithi175.stdout:2850: copy_from oid 5 from oid 1 current snap is 314 2024-09-11T03:48:51.193 INFO:tasks.rados.rados.2.smithi175.stdout:2851: copy_from oid 42 from oid 21 current snap is 314 2024-09-11T03:48:51.193 INFO:tasks.rados.rados.2.smithi175.stdout:2852: copy_from oid 8 from oid 19 current snap is 314 2024-09-11T03:48:51.193 INFO:tasks.rados.rados.2.smithi175.stdout:2853: rollback oid 40 current snap is 314 2024-09-11T03:48:51.193 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 40 to 307 2024-09-11T03:48:51.193 INFO:tasks.rados.rados.2.smithi175.stdout:2854: rollback oid 11 current snap is 314 2024-09-11T03:48:51.193 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 11 to 292 2024-09-11T03:48:51.193 INFO:tasks.rados.rados.2.smithi175.stdout:2855: snap_create 2024-09-11T03:48:51.195 INFO:tasks.rados.rados.2.smithi175.stdout:2853: finishing rollback tid 1 to smithi17557870-40 2024-09-11T03:48:51.195 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 492 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T03:48:51.197 INFO:tasks.rados.rados.2.smithi175.stdout:2854: finishing rollback tid 0 to smithi17557870-11 2024-09-11T03:48:51.197 INFO:tasks.rados.rados.2.smithi175.stdout:2854: finishing rollback tid 1 to smithi17557870-11 2024-09-11T03:48:51.197 INFO:tasks.rados.rados.2.smithi175.stdout:2854: finishing rollback tid 2 to smithi17557870-11 2024-09-11T03:48:51.197 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 613 (ObjNum 845 snap 266 seq_num 845) dirty exists 2024-09-11T03:48:51.199 INFO:tasks.rados.rados.2.smithi175.stdout:2850: finishing copy_from racing read to smithi17557870-5 2024-09-11T03:48:51.205 INFO:tasks.rados.rados.2.smithi175.stdout:2850: finishing copy_from to smithi17557870-5 2024-09-11T03:48:51.205 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 947 (ObjNum 758 snap 246 seq_num 758) dirty exists 2024-09-11T03:48:51.208 INFO:tasks.rados.rados.2.smithi175.stdout:2848: expect (ObjNum 765 snap 247 seq_num 765) 2024-09-11T03:48:51.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:51.238 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:51.239 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:51.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:51.275 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-11T03:48:51.275 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-11T03:48:51.275 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-11T03:48:51.275 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-11T03:48:51.275 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-11T03:48:51.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:51 smithi084 ceph-mon[27254]: osdmap e712: 12 total, 12 up, 12 in 2024-09-11T03:48:51.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:51 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:48:51.276 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:51 smithi084 ceph-mon[27254]: 2.8 scrub ok 2024-09-11T03:48:51.276 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-11T03:48:51.276 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-11T03:48:51.276 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-11T03:48:51.276 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-11T03:48:51.276 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-11T03:48:51.276 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-11T03:48:51.276 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-11T03:48:51.332 INFO:tasks.rados.rados.2.smithi175.stdout:2851: finishing copy_from racing read to smithi17557870-42 2024-09-11T03:48:51.332 INFO:tasks.rados.rados.2.smithi175.stdout:2852: finishing copy_from racing read to smithi17557870-8 2024-09-11T03:48:51.333 INFO:tasks.rados.rados.2.smithi175.stdout:2851: finishing copy_from to smithi17557870-42 2024-09-11T03:48:51.333 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 930 (ObjNum 829 snap 263 seq_num 829) dirty exists 2024-09-11T03:48:51.333 INFO:tasks.rados.rados.2.smithi175.stdout:2852: finishing copy_from to smithi17557870-8 2024-09-11T03:48:51.333 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 984 (ObjNum 963 snap 310 seq_num 963) dirty exists 2024-09-11T03:48:51.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:51.363 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:51.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:51.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:51.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:51 smithi121 ceph-mon[30729]: osdmap e712: 12 total, 12 up, 12 in 2024-09-11T03:48:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:51 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:48:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:51 smithi121 ceph-mon[30729]: 2.8 scrub ok 2024-09-11T03:48:51.465 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:51.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:51 smithi175 ceph-mon[30894]: osdmap e712: 12 total, 12 up, 12 in 2024-09-11T03:48:51.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:51 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:48:51.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:51 smithi175 ceph-mon[30894]: 2.8 scrub ok 2024-09-11T03:48:51.488 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:51.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:51.590 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:51.611 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:51.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:51.713 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:51.734 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:51.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:51.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:51.836 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:51.857 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:51.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:51.858 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:51.959 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:51.983 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:51.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:51.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:52.008 INFO:tasks.rados.rados.2.smithi175.stdout:2848: done (6 left) 2024-09-11T03:48:52.008 INFO:tasks.rados.rados.2.smithi175.stdout:2850: done (5 left) 2024-09-11T03:48:52.008 INFO:tasks.rados.rados.2.smithi175.stdout:2851: done (4 left) 2024-09-11T03:48:52.008 INFO:tasks.rados.rados.2.smithi175.stdout:2852: done (3 left) 2024-09-11T03:48:52.008 INFO:tasks.rados.rados.2.smithi175.stdout:2853: done (2 left) 2024-09-11T03:48:52.008 INFO:tasks.rados.rados.2.smithi175.stdout:2854: done (1 left) 2024-09-11T03:48:52.008 INFO:tasks.rados.rados.2.smithi175.stdout:2855: done (0 left) 2024-09-11T03:48:52.008 INFO:tasks.rados.rados.2.smithi175.stdout:2856: read oid 4 snap 312 2024-09-11T03:48:52.008 INFO:tasks.rados.rados.2.smithi175.stdout:2856: expect deleted 2024-09-11T03:48:52.008 INFO:tasks.rados.rados.2.smithi175.stdout:2857: write oid 29 current snap is 315 2024-09-11T03:48:52.009 INFO:tasks.rados.rados.2.smithi175.stdout:2857: seq_num 982 ranges {504474=504055,1714141=631006} 2024-09-11T03:48:52.013 INFO:tasks.rados.rados.2.smithi175.stdout:2857: writing smithi17557870-29 from 504474 to 1008529 tid 1 2024-09-11T03:48:52.020 INFO:tasks.rados.rados.2.smithi175.stdout:2857: writing smithi17557870-29 from 1714141 to 2345147 tid 2 2024-09-11T03:48:52.021 INFO:tasks.rados.rados.2.smithi175.stdout:2858: write (excl) oid 34 current snap is 315 2024-09-11T03:48:52.021 INFO:tasks.rados.rados.2.smithi175.stdout:2858: seq_num 983 ranges {521281=579304,1656270=774664} 2024-09-11T03:48:52.026 INFO:tasks.rados.rados.2.smithi175.stdout:2858: writing smithi17557870-34 from 521281 to 1100585 tid 1 2024-09-11T03:48:52.034 INFO:tasks.rados.rados.2.smithi175.stdout:2858: writing smithi17557870-34 from 1656270 to 2430934 tid 2 2024-09-11T03:48:52.034 INFO:tasks.rados.rados.2.smithi175.stdout:2859: read oid 47 snap -1 2024-09-11T03:48:52.034 INFO:tasks.rados.rados.2.smithi175.stdout:2859: expect (ObjNum 955 snap 305 seq_num 955) 2024-09-11T03:48:52.035 INFO:tasks.rados.rados.2.smithi175.stdout:2860: copy_from oid 17 from oid 12 current snap is 315 2024-09-11T03:48:52.035 INFO:tasks.rados.rados.2.smithi175.stdout:2857: finishing write tid 1 to smithi17557870-29 2024-09-11T03:48:52.035 INFO:tasks.rados.rados.2.smithi175.stdout:2856: done (4 left) 2024-09-11T03:48:52.035 INFO:tasks.rados.rados.2.smithi175.stdout:2861: write (excl) oid 32 current snap is 315 2024-09-11T03:48:52.035 INFO:tasks.rados.rados.2.smithi175.stdout:2857: finishing write tid 2 to smithi17557870-29 2024-09-11T03:48:52.035 INFO:tasks.rados.rados.2.smithi175.stdout:2857: finishing write tid 3 to smithi17557870-29 2024-09-11T03:48:52.035 INFO:tasks.rados.rados.2.smithi175.stdout:2857: finishing write tid 4 to smithi17557870-29 2024-09-11T03:48:52.035 INFO:tasks.rados.rados.2.smithi175.stdout:2857: oid 29 updating version 0 to 1047 2024-09-11T03:48:52.035 INFO:tasks.rados.rados.2.smithi175.stdout:2857: oid 29 updating version 1047 to 1049 2024-09-11T03:48:52.035 INFO:tasks.rados.rados.2.smithi175.stdout:2857: oid 29 version 1049 is already newer than 1048 2024-09-11T03:48:52.035 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 1049 (ObjNum 982 snap 315 seq_num 982) dirty exists 2024-09-11T03:48:52.035 INFO:tasks.rados.rados.2.smithi175.stdout:2857: left oid 29 (ObjNum 982 snap 315 seq_num 982) 2024-09-11T03:48:52.035 INFO:tasks.rados.rados.2.smithi175.stdout:2861: seq_num 984 ranges {538088=654553,1714752=634675} 2024-09-11T03:48:52.041 INFO:tasks.rados.rados.2.smithi175.stdout:2861: writing smithi17557870-32 from 538088 to 1192641 tid 1 2024-09-11T03:48:52.048 INFO:tasks.rados.rados.2.smithi175.stdout:2861: writing smithi17557870-32 from 1714752 to 2349427 tid 2 2024-09-11T03:48:52.048 INFO:tasks.rados.rados.2.smithi175.stdout:2857: done (4 left) 2024-09-11T03:48:52.048 INFO:tasks.rados.rados.2.smithi175.stdout:2862: snap_create 2024-09-11T03:48:52.048 INFO:tasks.rados.rados.2.smithi175.stdout:2858: finishing write tid 1 to smithi17557870-34 2024-09-11T03:48:52.048 INFO:tasks.rados.rados.2.smithi175.stdout:2858: finishing write tid 2 to smithi17557870-34 2024-09-11T03:48:52.048 INFO:tasks.rados.rados.2.smithi175.stdout:2858: finishing write tid 3 to smithi17557870-34 2024-09-11T03:48:52.048 INFO:tasks.rados.rados.2.smithi175.stdout:2858: finishing write tid 4 to smithi17557870-34 2024-09-11T03:48:52.048 INFO:tasks.rados.rados.2.smithi175.stdout:2858: oid 34 updating version 0 to 956 2024-09-11T03:48:52.048 INFO:tasks.rados.rados.2.smithi175.stdout:2858: oid 34 version 956 is already newer than 954 2024-09-11T03:48:52.048 INFO:tasks.rados.rados.2.smithi175.stdout:2858: oid 34 version 956 is already newer than 955 2024-09-11T03:48:52.049 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 956 (ObjNum 983 snap 315 seq_num 983) dirty exists 2024-09-11T03:48:52.049 INFO:tasks.rados.rados.2.smithi175.stdout:2858: left oid 34 (ObjNum 983 snap 315 seq_num 983) 2024-09-11T03:48:52.049 INFO:tasks.rados.rados.2.smithi175.stdout:2861: finishing write tid 1 to smithi17557870-32 2024-09-11T03:48:52.051 INFO:tasks.rados.rados.2.smithi175.stdout:2861: finishing write tid 2 to smithi17557870-32 2024-09-11T03:48:52.051 INFO:tasks.rados.rados.2.smithi175.stdout:2861: finishing write tid 3 to smithi17557870-32 2024-09-11T03:48:52.051 INFO:tasks.rados.rados.2.smithi175.stdout:2861: finishing write tid 4 to smithi17557870-32 2024-09-11T03:48:52.052 INFO:tasks.rados.rados.2.smithi175.stdout:2861: oid 32 updating version 0 to 969 2024-09-11T03:48:52.052 INFO:tasks.rados.rados.2.smithi175.stdout:2861: oid 32 version 969 is already newer than 968 2024-09-11T03:48:52.052 INFO:tasks.rados.rados.2.smithi175.stdout:2861: oid 32 version 969 is already newer than 967 2024-09-11T03:48:52.052 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 969 (ObjNum 984 snap 315 seq_num 984) dirty exists 2024-09-11T03:48:52.052 INFO:tasks.rados.rados.2.smithi175.stdout:2861: left oid 32 (ObjNum 984 snap 315 seq_num 984) 2024-09-11T03:48:52.054 INFO:tasks.rados.rados.2.smithi175.stdout:2859: expect (ObjNum 955 snap 305 seq_num 955) 2024-09-11T03:48:52.085 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:52.106 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:52.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:52.106 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:52.137 INFO:tasks.rados.rados.2.smithi175.stdout:2860: finishing copy_from racing read to smithi17557870-17 2024-09-11T03:48:52.137 INFO:tasks.rados.rados.2.smithi175.stdout:2860: finishing copy_from to smithi17557870-17 2024-09-11T03:48:52.137 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 985 (ObjNum 952 snap 304 seq_num 952) dirty exists 2024-09-11T03:48:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:52 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:48:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:52 smithi084 ceph-mon[27254]: 2.14 scrub ok 2024-09-11T03:48:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:52 smithi084 ceph-mon[27254]: pgmap v1224: 67 pgs: 67 active+clean; 300 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.5 MiB/s wr, 4 op/s 2024-09-11T03:48:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:52 smithi084 ceph-mon[27254]: osdmap e713: 12 total, 12 up, 12 in 2024-09-11T03:48:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:52 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1380412414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:52.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:52 smithi084 ceph-mon[27254]: osdmap e714: 12 total, 12 up, 12 in 2024-09-11T03:48:52.208 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:52.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:52.230 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:52.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:52.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:52.355 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:52.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:52.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:52.417 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:48:52.418 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T03:48:52.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-09-11T03:48:52.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:52 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:48:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:52 smithi121 ceph-mon[30729]: 2.14 scrub ok 2024-09-11T03:48:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:52 smithi121 ceph-mon[30729]: pgmap v1224: 67 pgs: 67 active+clean; 300 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.5 MiB/s wr, 4 op/s 2024-09-11T03:48:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:52 smithi121 ceph-mon[30729]: osdmap e713: 12 total, 12 up, 12 in 2024-09-11T03:48:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:52 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1380412414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:52 smithi121 ceph-mon[30729]: osdmap e714: 12 total, 12 up, 12 in 2024-09-11T03:48:52.457 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:52 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:48:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:52 smithi175 ceph-mon[30894]: 2.14 scrub ok 2024-09-11T03:48:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:52 smithi175 ceph-mon[30894]: pgmap v1224: 67 pgs: 67 active+clean; 300 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.5 MiB/s wr, 4 op/s 2024-09-11T03:48:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:52 smithi175 ceph-mon[30894]: osdmap e713: 12 total, 12 up, 12 in 2024-09-11T03:48:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:52 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1380412414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:52 smithi175 ceph-mon[30894]: osdmap e714: 12 total, 12 up, 12 in 2024-09-11T03:48:52.479 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:52.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:52.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:52.533 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:48:52.533 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:48:52.533 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:48:52.533 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:48:52.533 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:48:52.533 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:48:52.534 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:48:52.534 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:48:52.534 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:48:52.534 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:48:52.534 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:48:52.534 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:48:52.534 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:48:52.534 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:48:52.534 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:48:52.535 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:48:52.535 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:48:52.535 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:48:52.535 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:48:52.535 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:48:52.535 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:48:52.535 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:48:52.535 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:48:52.535 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:48:52.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:52.604 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:52.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:52.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:52.705 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:48:52.707 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:52.729 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:52.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:52.730 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:52.831 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:52.852 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:52.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:52.853 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:52.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:52.979 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:52.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:52.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:53.013 INFO:tasks.rados.rados.2.smithi175.stdout:2858: done (4 left) 2024-09-11T03:48:53.013 INFO:tasks.rados.rados.2.smithi175.stdout:2859: done (3 left) 2024-09-11T03:48:53.013 INFO:tasks.rados.rados.2.smithi175.stdout:2860: done (2 left) 2024-09-11T03:48:53.013 INFO:tasks.rados.rados.2.smithi175.stdout:2861: done (1 left) 2024-09-11T03:48:53.014 INFO:tasks.rados.rados.2.smithi175.stdout:2862: done (0 left) 2024-09-11T03:48:53.014 INFO:tasks.rados.rados.2.smithi175.stdout:2863: write (excl) oid 6 current snap is 316 2024-09-11T03:48:53.014 INFO:tasks.rados.rados.2.smithi175.stdout:2863: seq_num 985 ranges {554895=729802,1773234=778333} 2024-09-11T03:48:53.020 INFO:tasks.rados.rados.2.smithi175.stdout:2863: writing smithi17557870-6 from 554895 to 1284697 tid 1 2024-09-11T03:48:53.027 INFO:tasks.rados.rados.2.smithi175.stdout:2863: writing smithi17557870-6 from 1773234 to 2551567 tid 2 2024-09-11T03:48:53.027 INFO:tasks.rados.rados.2.smithi175.stdout:2864: read oid 31 snap 313 2024-09-11T03:48:53.027 INFO:tasks.rados.rados.2.smithi175.stdout:2864: expect (ObjNum 925 snap 294 seq_num 925) 2024-09-11T03:48:53.027 INFO:tasks.rados.rados.2.smithi175.stdout:2865: copy_from oid 40 from oid 9 current snap is 316 2024-09-11T03:48:53.027 INFO:tasks.rados.rados.2.smithi175.stdout:2866: read oid 25 snap -1 2024-09-11T03:48:53.028 INFO:tasks.rados.rados.2.smithi175.stdout:2866: expect (ObjNum 950 snap 303 seq_num 950) 2024-09-11T03:48:53.028 INFO:tasks.rados.rados.2.smithi175.stdout:2867: read oid 18 snap -1 2024-09-11T03:48:53.028 INFO:tasks.rados.rados.2.smithi175.stdout:2867: expect (ObjNum 917 snap 294 seq_num 917) 2024-09-11T03:48:53.028 INFO:tasks.rados.rados.2.smithi175.stdout:2868: rollback oid 24 current snap is 316 2024-09-11T03:48:53.028 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 296 2024-09-11T03:48:53.028 INFO:tasks.rados.rados.2.smithi175.stdout:2869: rollback oid 14 current snap is 316 2024-09-11T03:48:53.028 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 14 to 279 2024-09-11T03:48:53.028 INFO:tasks.rados.rados.2.smithi175.stdout:2870: delete oid 2 current snap is 316 2024-09-11T03:48:53.028 INFO:tasks.rados.rados.2.smithi175.stdout:2863: finishing write tid 1 to smithi17557870-6 2024-09-11T03:48:53.029 INFO:tasks.rados.rados.2.smithi175.stdout:2868: finishing rollback tid 0 to smithi17557870-24 2024-09-11T03:48:53.030 INFO:tasks.rados.rados.2.smithi175.stdout:2868: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:48:53.030 INFO:tasks.rados.rados.2.smithi175.stdout:2868: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:48:53.030 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 973 (ObjNum 876 snap 276 seq_num 876) dirty exists 2024-09-11T03:48:53.031 INFO:tasks.rados.rados.2.smithi175.stdout:2869: finishing rollback tid 1 to smithi17557870-14 2024-09-11T03:48:53.031 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 965 (ObjNum 33 snap 0 seq_num 1746033776) dirty dne 2024-09-11T03:48:53.041 INFO:tasks.rados.rados.2.smithi175.stdout:2864: expect (ObjNum 925 snap 294 seq_num 925) 2024-09-11T03:48:53.081 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:53.085 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:48:53.085 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:48:53.103 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:53.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:53.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:53.155 INFO:tasks.rados.rados.2.smithi175.stdout:2863: finishing write tid 2 to smithi17557870-6 2024-09-11T03:48:53.155 INFO:tasks.rados.rados.2.smithi175.stdout:2863: finishing write tid 3 to smithi17557870-6 2024-09-11T03:48:53.155 INFO:tasks.rados.rados.2.smithi175.stdout:2863: finishing write tid 4 to smithi17557870-6 2024-09-11T03:48:53.155 INFO:tasks.rados.rados.2.smithi175.stdout:2863: oid 6 updating version 0 to 896 2024-09-11T03:48:53.155 INFO:tasks.rados.rados.2.smithi175.stdout:2863: oid 6 updating version 896 to 897 2024-09-11T03:48:53.155 INFO:tasks.rados.rados.2.smithi175.stdout:2863: oid 6 version 897 is already newer than 895 2024-09-11T03:48:53.155 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 897 (ObjNum 985 snap 316 seq_num 985) dirty exists 2024-09-11T03:48:53.155 INFO:tasks.rados.rados.2.smithi175.stdout:2863: left oid 6 (ObjNum 985 snap 316 seq_num 985) 2024-09-11T03:48:53.156 INFO:tasks.rados.rados.2.smithi175.stdout:2865: finishing copy_from racing read to smithi17557870-40 2024-09-11T03:48:53.156 INFO:tasks.rados.rados.2.smithi175.stdout:2865: finishing copy_from to smithi17557870-40 2024-09-11T03:48:53.156 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 1006 (ObjNum 973 snap 313 seq_num 973) dirty exists 2024-09-11T03:48:53.156 INFO:tasks.rados.rados.2.smithi175.stdout:2866: expect (ObjNum 950 snap 303 seq_num 950) 2024-09-11T03:48:53.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:53.228 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:53.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:53.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:53.330 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:53.351 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:53.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:53.352 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:53.394 INFO:tasks.rados.rados.2.smithi175.stdout:2867: expect (ObjNum 917 snap 294 seq_num 917) 2024-09-11T03:48:53.453 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:53.475 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:53.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:53.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:53.566 INFO:tasks.rados.rados.2.smithi175.stdout:2863: done (7 left) 2024-09-11T03:48:53.566 INFO:tasks.rados.rados.2.smithi175.stdout:2864: done (6 left) 2024-09-11T03:48:53.566 INFO:tasks.rados.rados.2.smithi175.stdout:2865: done (5 left) 2024-09-11T03:48:53.566 INFO:tasks.rados.rados.2.smithi175.stdout:2866: done (4 left) 2024-09-11T03:48:53.566 INFO:tasks.rados.rados.2.smithi175.stdout:2867: done (3 left) 2024-09-11T03:48:53.566 INFO:tasks.rados.rados.2.smithi175.stdout:2868: done (2 left) 2024-09-11T03:48:53.567 INFO:tasks.rados.rados.2.smithi175.stdout:2869: done (1 left) 2024-09-11T03:48:53.567 INFO:tasks.rados.rados.2.smithi175.stdout:2870: done (0 left) 2024-09-11T03:48:53.567 INFO:tasks.rados.rados.2.smithi175.stdout:2871: write (excl) oid 9 current snap is 316 2024-09-11T03:48:53.567 INFO:tasks.rados.rados.2.smithi175.stdout:2871: seq_num 987 ranges {588509=480300,1773845=782002} 2024-09-11T03:48:53.571 INFO:tasks.rados.rados.2.smithi175.stdout:2871: writing smithi17557870-9 from 588509 to 1068809 tid 1 2024-09-11T03:48:53.577 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:53.580 INFO:tasks.rados.rados.2.smithi175.stdout:2871: writing smithi17557870-9 from 1773845 to 2555847 tid 2 2024-09-11T03:48:53.580 INFO:tasks.rados.rados.2.smithi175.stdout:2872: write (excl) oid 42 current snap is 316 2024-09-11T03:48:53.580 INFO:tasks.rados.rados.2.smithi175.stdout:2872: seq_num 988 ranges {605316=555549,1832327=642013} 2024-09-11T03:48:53.585 INFO:tasks.rados.rados.2.smithi175.stdout:2872: writing smithi17557870-42 from 605316 to 1160865 tid 1 2024-09-11T03:48:53.593 INFO:tasks.rados.rados.2.smithi175.stdout:2872: writing smithi17557870-42 from 1832327 to 2474340 tid 2 2024-09-11T03:48:53.593 INFO:tasks.rados.rados.2.smithi175.stdout:2873: write (excl) oid 20 current snap is 316 2024-09-11T03:48:53.593 INFO:tasks.rados.rados.2.smithi175.stdout:2871: finishing write tid 1 to smithi17557870-9 2024-09-11T03:48:53.593 INFO:tasks.rados.rados.2.smithi175.stdout:2871: finishing write tid 2 to smithi17557870-9 2024-09-11T03:48:53.593 INFO:tasks.rados.rados.2.smithi175.stdout:2871: finishing write tid 3 to smithi17557870-9 2024-09-11T03:48:53.593 INFO:tasks.rados.rados.2.smithi175.stdout:2873: seq_num 989 ranges {622123=747151,2007162=614961} 2024-09-11T03:48:53.600 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:53.600 INFO:tasks.rados.rados.2.smithi175.stdout:2873: writing smithi17557870-20 from 622123 to 1369274 tid 1 2024-09-11T03:48:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:53.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:53.607 INFO:tasks.rados.rados.2.smithi175.stdout:2873: writing smithi17557870-20 from 2007162 to 2622123 tid 2 2024-09-11T03:48:53.607 INFO:tasks.rados.rados.2.smithi175.stdout:2874: rollback oid 35 current snap is 316 2024-09-11T03:48:53.608 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 35 to 314 2024-09-11T03:48:53.608 INFO:tasks.rados.rados.2.smithi175.stdout:2871: finishing write tid 4 to smithi17557870-9 2024-09-11T03:48:53.608 INFO:tasks.rados.rados.2.smithi175.stdout:2871: oid 9 updating version 0 to 1017 2024-09-11T03:48:53.608 INFO:tasks.rados.rados.2.smithi175.stdout:2871: oid 9 version 1017 is already newer than 1016 2024-09-11T03:48:53.608 INFO:tasks.rados.rados.2.smithi175.stdout:2871: oid 9 version 1017 is already newer than 1015 2024-09-11T03:48:53.608 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 1017 (ObjNum 987 snap 316 seq_num 987) dirty exists 2024-09-11T03:48:53.608 INFO:tasks.rados.rados.2.smithi175.stdout:2871: left oid 9 (ObjNum 987 snap 316 seq_num 987) 2024-09-11T03:48:53.608 INFO:tasks.rados.rados.2.smithi175.stdout:2872: finishing write tid 1 to smithi17557870-42 2024-09-11T03:48:53.608 INFO:tasks.rados.rados.2.smithi175.stdout:2871: done (3 left) 2024-09-11T03:48:53.608 INFO:tasks.rados.rados.2.smithi175.stdout:2875: snap_create 2024-09-11T03:48:53.608 INFO:tasks.rados.rados.2.smithi175.stdout:2872: finishing write tid 2 to smithi17557870-42 2024-09-11T03:48:53.608 INFO:tasks.rados.rados.2.smithi175.stdout:2872: finishing write tid 3 to smithi17557870-42 2024-09-11T03:48:53.608 INFO:tasks.rados.rados.2.smithi175.stdout:2872: finishing write tid 4 to smithi17557870-42 2024-09-11T03:48:53.609 INFO:tasks.rados.rados.2.smithi175.stdout:2872: oid 42 updating version 0 to 955 2024-09-11T03:48:53.609 INFO:tasks.rados.rados.2.smithi175.stdout:2872: oid 42 updating version 955 to 956 2024-09-11T03:48:53.609 INFO:tasks.rados.rados.2.smithi175.stdout:2872: oid 42 updating version 956 to 957 2024-09-11T03:48:53.609 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 957 (ObjNum 988 snap 316 seq_num 988) dirty exists 2024-09-11T03:48:53.609 INFO:tasks.rados.rados.2.smithi175.stdout:2872: left oid 42 (ObjNum 988 snap 316 seq_num 988) 2024-09-11T03:48:53.609 INFO:tasks.rados.rados.2.smithi175.stdout:2873: finishing write tid 1 to smithi17557870-20 2024-09-11T03:48:53.609 INFO:tasks.rados.rados.2.smithi175.stdout:2874: finishing rollback tid 1 to smithi17557870-35 2024-09-11T03:48:53.609 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 949 (ObjNum 49 snap 0 seq_num 2931793152) dirty dne 2024-09-11T03:48:53.610 INFO:tasks.rados.rados.2.smithi175.stdout:2873: finishing write tid 2 to smithi17557870-20 2024-09-11T03:48:53.610 INFO:tasks.rados.rados.2.smithi175.stdout:2873: finishing write tid 3 to smithi17557870-20 2024-09-11T03:48:53.610 INFO:tasks.rados.rados.2.smithi175.stdout:2873: finishing write tid 4 to smithi17557870-20 2024-09-11T03:48:53.610 INFO:tasks.rados.rados.2.smithi175.stdout:2873: oid 20 updating version 0 to 988 2024-09-11T03:48:53.610 INFO:tasks.rados.rados.2.smithi175.stdout:2873: oid 20 version 988 is already newer than 987 2024-09-11T03:48:53.610 INFO:tasks.rados.rados.2.smithi175.stdout:2873: oid 20 version 988 is already newer than 986 2024-09-11T03:48:53.610 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 988 (ObjNum 989 snap 316 seq_num 989) dirty exists 2024-09-11T03:48:53.611 INFO:tasks.rados.rados.2.smithi175.stdout:2873: left oid 20 (ObjNum 989 snap 316 seq_num 989) 2024-09-11T03:48:53.701 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:53.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:53.723 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:53.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:53.820 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:53.824 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:53.845 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:53.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:53.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:53.947 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:53.969 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:53.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:53.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:54.070 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:54.089 INFO:tasks.rados.rados.2.smithi175.stdout:2872: done (3 left) 2024-09-11T03:48:54.089 INFO:tasks.rados.rados.2.smithi175.stdout:2873: done (2 left) 2024-09-11T03:48:54.089 INFO:tasks.rados.rados.2.smithi175.stdout:2874: done (1 left) 2024-09-11T03:48:54.089 INFO:tasks.rados.rados.2.smithi175.stdout:2875: done (0 left) 2024-09-11T03:48:54.089 INFO:tasks.rados.rados.2.smithi175.stdout:2876: read oid 28 snap -1 2024-09-11T03:48:54.089 INFO:tasks.rados.rados.2.smithi175.stdout:2876: expect (ObjNum 942 snap 297 seq_num 942) 2024-09-11T03:48:54.089 INFO:tasks.rados.rados.2.smithi175.stdout:2877: snap_remove snap 312 2024-09-11T03:48:54.092 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:54.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:54.093 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:54.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:54.220 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:54.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:54.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:54.322 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:54.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:54.345 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:54.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:54.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:54 smithi084 ceph-mon[27254]: pgmap v1227: 67 pgs: 67 active+clean; 324 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 7.0 MiB/s wr, 21 op/s 2024-09-11T03:48:54.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:54 smithi084 ceph-mon[27254]: osdmap e715: 12 total, 12 up, 12 in 2024-09-11T03:48:54.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:54 smithi121 ceph-mon[30729]: pgmap v1227: 67 pgs: 67 active+clean; 324 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 7.0 MiB/s wr, 21 op/s 2024-09-11T03:48:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:54 smithi121 ceph-mon[30729]: osdmap e715: 12 total, 12 up, 12 in 2024-09-11T03:48:54.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:54.467 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:54.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:54.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:54.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:54 smithi175 ceph-mon[30894]: pgmap v1227: 67 pgs: 67 active+clean; 324 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 7.0 MiB/s wr, 21 op/s 2024-09-11T03:48:54.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:54 smithi175 ceph-mon[30894]: osdmap e715: 12 total, 12 up, 12 in 2024-09-11T03:48:54.532 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T03:48:54.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:54.595 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:54.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:54.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:54.697 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:54.724 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:54.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:54.725 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:54.748 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-11T03:48:54.749 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-11T03:48:54.749 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-11T03:48:54.749 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-11T03:48:54.749 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-11T03:48:54.749 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-11T03:48:54.749 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-11T03:48:54.749 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-11T03:48:54.749 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-11T03:48:54.750 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-11T03:48:54.750 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-11T03:48:54.750 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-11T03:48:54.826 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:54.852 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:54.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:54.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:54.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:54.975 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:54.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:54.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:55.077 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:55.099 INFO:tasks.rados.rados.2.smithi175.stdout:2877: done (1 left) 2024-09-11T03:48:55.100 INFO:tasks.rados.rados.2.smithi175.stdout:2878: read oid 4 snap -1 2024-09-11T03:48:55.100 INFO:tasks.rados.rados.2.smithi175.stdout:2878: expect deleted 2024-09-11T03:48:55.100 INFO:tasks.rados.rados.2.smithi175.stdout:2879: snap_remove snap 298 2024-09-11T03:48:55.106 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:55.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:55.107 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:55.208 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:55.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:55.230 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:55.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:55.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:55.357 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:55.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:55.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:55.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:55 smithi084 ceph-mon[27254]: 1.18 scrub starts 2024-09-11T03:48:55.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:55 smithi084 ceph-mon[27254]: 1.18 scrub ok 2024-09-11T03:48:55.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:55 smithi084 ceph-mon[27254]: osdmap e716: 12 total, 12 up, 12 in 2024-09-11T03:48:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:55 smithi121 ceph-mon[30729]: 1.18 scrub starts 2024-09-11T03:48:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:55 smithi121 ceph-mon[30729]: 1.18 scrub ok 2024-09-11T03:48:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:55 smithi121 ceph-mon[30729]: osdmap e716: 12 total, 12 up, 12 in 2024-09-11T03:48:55.459 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:48:55.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:55 smithi175 ceph-mon[30894]: 1.18 scrub starts 2024-09-11T03:48:55.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:55 smithi175 ceph-mon[30894]: 1.18 scrub ok 2024-09-11T03:48:55.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:55 smithi175 ceph-mon[30894]: osdmap e716: 12 total, 12 up, 12 in 2024-09-11T03:48:55.485 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:55.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:55.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:48:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:48:55.522 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T03:48:55.522 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-09-11T03:48:55.546 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:55.555 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:55.587 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:55.598 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:48:55.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:55.609 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:55.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:55.711 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:55.734 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:55.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:55.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:55.836 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:55.859 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:55.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:55.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:55.962 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:55.984 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:55.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:55.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:56.076 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:48:56.076 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:48:56.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:56.101 INFO:tasks.rados.rados.2.smithi175.stdout:2879: done (2 left) 2024-09-11T03:48:56.101 INFO:tasks.rados.rados.2.smithi175.stdout:2880: write oid 17 current snap is 317 2024-09-11T03:48:56.101 INFO:tasks.rados.rados.2.smithi175.stdout:2880: seq_num 990 ranges {638930=422400,1665644=645682} 2024-09-11T03:48:56.107 INFO:tasks.rados.rados.2.smithi175.stdout:2880: writing smithi17557870-17 from 638930 to 1061330 tid 1 2024-09-11T03:48:56.115 INFO:tasks.rados.rados.2.smithi175.stdout:2880: writing smithi17557870-17 from 1665644 to 2311326 tid 2 2024-09-11T03:48:56.115 INFO:tasks.rados.rados.2.smithi175.stdout:2881: write (excl) oid 39 current snap is 317 2024-09-11T03:48:56.115 INFO:tasks.rados.rados.2.smithi175.stdout:2881: seq_num 991 ranges {655737=497649,1607773=789340} 2024-09-11T03:48:56.118 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:56.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:56.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:56.121 INFO:tasks.rados.rados.2.smithi175.stdout:2881: writing smithi17557870-39 from 655737 to 1153386 tid 1 2024-09-11T03:48:56.129 INFO:tasks.rados.rados.2.smithi175.stdout:2881: writing smithi17557870-39 from 1607773 to 2397113 tid 2 2024-09-11T03:48:56.129 INFO:tasks.rados.rados.2.smithi175.stdout:2882: write (excl) oid 47 current snap is 317 2024-09-11T03:48:56.129 INFO:tasks.rados.rados.2.smithi175.stdout:2882: seq_num 992 ranges {672544=572898,1666255=532998} 2024-09-11T03:48:56.136 INFO:tasks.rados.rados.2.smithi175.stdout:2882: writing smithi17557870-47 from 672544 to 1245442 tid 1 2024-09-11T03:48:56.141 INFO:tasks.rados.rados.2.smithi175.stdout:2882: writing smithi17557870-47 from 1666255 to 2199253 tid 2 2024-09-11T03:48:56.141 INFO:tasks.rados.rados.2.smithi175.stdout:2883: snap_remove snap 316 2024-09-11T03:48:56.220 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:56.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:56.245 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:56.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:56.280 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:56.281 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:48:56.347 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:56.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:56 smithi084 ceph-mon[27254]: pgmap v1230: 67 pgs: 67 active+clean; 326 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 7.5 MiB/s wr, 21 op/s 2024-09-11T03:48:56.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:56 smithi084 ceph-mon[27254]: osdmap e717: 12 total, 12 up, 12 in 2024-09-11T03:48:56.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:56 smithi084 ceph-mon[27254]: 2.1d scrub starts 2024-09-11T03:48:56.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:56 smithi084 ceph-mon[27254]: 2.1d scrub ok 2024-09-11T03:48:56.373 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:56.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:56.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:56.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:56 smithi121 ceph-mon[30729]: pgmap v1230: 67 pgs: 67 active+clean; 326 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 7.5 MiB/s wr, 21 op/s 2024-09-11T03:48:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:56 smithi121 ceph-mon[30729]: osdmap e717: 12 total, 12 up, 12 in 2024-09-11T03:48:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:56 smithi121 ceph-mon[30729]: 2.1d scrub starts 2024-09-11T03:48:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:56 smithi121 ceph-mon[30729]: 2.1d scrub ok 2024-09-11T03:48:56.474 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:56.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:56 smithi175 ceph-mon[30894]: pgmap v1230: 67 pgs: 67 active+clean; 326 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 7.5 MiB/s wr, 21 op/s 2024-09-11T03:48:56.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:56 smithi175 ceph-mon[30894]: osdmap e717: 12 total, 12 up, 12 in 2024-09-11T03:48:56.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:56 smithi175 ceph-mon[30894]: 2.1d scrub starts 2024-09-11T03:48:56.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:56 smithi175 ceph-mon[30894]: 2.1d scrub ok 2024-09-11T03:48:56.496 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:56.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:56.598 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:56.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:56.624 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:56.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:56.687 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T03:48:56.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:56.752 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:56.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:56.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:56.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:56.875 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:56.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:56.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:56.941 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:48:56.977 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:56.998 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:56.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:56.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:57.008 INFO:tasks.rados.rados.2.smithi175.stdout:2883: done (5 left) 2024-09-11T03:48:57.008 INFO:tasks.rados.rados.2.smithi175.stdout:2884: read oid 34 snap -1 2024-09-11T03:48:57.008 INFO:tasks.rados.rados.2.smithi175.stdout:2884: expect (ObjNum 983 snap 315 seq_num 983) 2024-09-11T03:48:57.008 INFO:tasks.rados.rados.2.smithi175.stdout:2885: snap_create 2024-09-11T03:48:57.008 INFO:tasks.rados.rados.2.smithi175.stdout:2876: expect (ObjNum 942 snap 297 seq_num 942) 2024-09-11T03:48:57.100 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:57.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:57.123 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:57.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:57.188 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:57 smithi084 ceph-mon[27254]: osdmap e718: 12 total, 12 up, 12 in 2024-09-11T03:48:57.189 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:57 smithi084 ceph-mon[27254]: 1.19 scrub starts 2024-09-11T03:48:57.189 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:57 smithi084 ceph-mon[27254]: 1.19 scrub ok 2024-09-11T03:48:57.190 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:57 smithi084 ceph-mon[27254]: 2.1e scrub starts 2024-09-11T03:48:57.190 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:57 smithi084 ceph-mon[27254]: 2.1e scrub ok 2024-09-11T03:48:57.190 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:57 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:48:57.190 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:57 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:48:57.190 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:57 smithi084 ceph-mon[27254]: osdmap e719: 12 total, 12 up, 12 in 2024-09-11T03:48:57.197 INFO:tasks.rados.rados.2.smithi175.stdout:2880: finishing write tid 1 to smithi17557870-17 2024-09-11T03:48:57.197 INFO:tasks.rados.rados.2.smithi175.stdout:2880: finishing write tid 2 to smithi17557870-17 2024-09-11T03:48:57.197 INFO:tasks.rados.rados.2.smithi175.stdout:2880: finishing write tid 3 to smithi17557870-17 2024-09-11T03:48:57.197 INFO:tasks.rados.rados.2.smithi175.stdout:2880: finishing write tid 4 to smithi17557870-17 2024-09-11T03:48:57.197 INFO:tasks.rados.rados.2.smithi175.stdout:2880: oid 17 updating version 0 to 990 2024-09-11T03:48:57.197 INFO:tasks.rados.rados.2.smithi175.stdout:2880: oid 17 version 990 is already newer than 989 2024-09-11T03:48:57.197 INFO:tasks.rados.rados.2.smithi175.stdout:2880: oid 17 updating version 990 to 991 2024-09-11T03:48:57.197 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 991 (ObjNum 990 snap 317 seq_num 990) dirty exists 2024-09-11T03:48:57.197 INFO:tasks.rados.rados.2.smithi175.stdout:2880: left oid 17 (ObjNum 990 snap 317 seq_num 990) 2024-09-11T03:48:57.197 INFO:tasks.rados.rados.2.smithi175.stdout:2881: finishing write tid 1 to smithi17557870-39 2024-09-11T03:48:57.198 INFO:tasks.rados.rados.2.smithi175.stdout:2881: finishing write tid 2 to smithi17557870-39 2024-09-11T03:48:57.198 INFO:tasks.rados.rados.2.smithi175.stdout:2881: finishing write tid 3 to smithi17557870-39 2024-09-11T03:48:57.198 INFO:tasks.rados.rados.2.smithi175.stdout:2881: finishing write tid 4 to smithi17557870-39 2024-09-11T03:48:57.198 INFO:tasks.rados.rados.2.smithi175.stdout:2881: oid 39 updating version 0 to 970 2024-09-11T03:48:57.198 INFO:tasks.rados.rados.2.smithi175.stdout:2881: oid 39 updating version 970 to 971 2024-09-11T03:48:57.198 INFO:tasks.rados.rados.2.smithi175.stdout:2881: oid 39 updating version 971 to 972 2024-09-11T03:48:57.198 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 972 (ObjNum 991 snap 317 seq_num 991) dirty exists 2024-09-11T03:48:57.198 INFO:tasks.rados.rados.2.smithi175.stdout:2881: left oid 39 (ObjNum 991 snap 317 seq_num 991) 2024-09-11T03:48:57.198 INFO:tasks.rados.rados.2.smithi175.stdout:2882: finishing write tid 1 to smithi17557870-47 2024-09-11T03:48:57.198 INFO:tasks.rados.rados.2.smithi175.stdout:2882: finishing write tid 2 to smithi17557870-47 2024-09-11T03:48:57.198 INFO:tasks.rados.rados.2.smithi175.stdout:2882: finishing write tid 3 to smithi17557870-47 2024-09-11T03:48:57.198 INFO:tasks.rados.rados.2.smithi175.stdout:2882: finishing write tid 4 to smithi17557870-47 2024-09-11T03:48:57.199 INFO:tasks.rados.rados.2.smithi175.stdout:2882: oid 47 updating version 0 to 992 2024-09-11T03:48:57.199 INFO:tasks.rados.rados.2.smithi175.stdout:2882: oid 47 updating version 992 to 993 2024-09-11T03:48:57.199 INFO:tasks.rados.rados.2.smithi175.stdout:2882: oid 47 updating version 993 to 994 2024-09-11T03:48:57.199 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 994 (ObjNum 992 snap 317 seq_num 992) dirty exists 2024-09-11T03:48:57.199 INFO:tasks.rados.rados.2.smithi175.stdout:2882: left oid 47 (ObjNum 992 snap 317 seq_num 992) 2024-09-11T03:48:57.199 INFO:tasks.rados.rados.2.smithi175.stdout:2884: expect (ObjNum 983 snap 315 seq_num 983) 2024-09-11T03:48:57.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:48:57.246 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:57.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:57.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:48:57.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:57.371 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:57.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:57.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:57.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:57 smithi121 ceph-mon[30729]: osdmap e718: 12 total, 12 up, 12 in 2024-09-11T03:48:57.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:57 smithi121 ceph-mon[30729]: 1.19 scrub starts 2024-09-11T03:48:57.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:57 smithi121 ceph-mon[30729]: 1.19 scrub ok 2024-09-11T03:48:57.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:57 smithi121 ceph-mon[30729]: 2.1e scrub starts 2024-09-11T03:48:57.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:57 smithi121 ceph-mon[30729]: 2.1e scrub ok 2024-09-11T03:48:57.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:57 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:48:57.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:57 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:48:57.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:57 smithi121 ceph-mon[30729]: osdmap e719: 12 total, 12 up, 12 in 2024-09-11T03:48:57.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:48:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:57 smithi175 ceph-mon[30894]: osdmap e718: 12 total, 12 up, 12 in 2024-09-11T03:48:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:57 smithi175 ceph-mon[30894]: 1.19 scrub starts 2024-09-11T03:48:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:57 smithi175 ceph-mon[30894]: 1.19 scrub ok 2024-09-11T03:48:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:57 smithi175 ceph-mon[30894]: 2.1e scrub starts 2024-09-11T03:48:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:57 smithi175 ceph-mon[30894]: 2.1e scrub ok 2024-09-11T03:48:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:57 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:48:57.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:57 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:48:57.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:57 smithi175 ceph-mon[30894]: osdmap e719: 12 total, 12 up, 12 in 2024-09-11T03:48:57.494 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:57.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:57.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:48:57.497 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:48:57.498 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:48:57.589 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:48:57.590 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:48:57.595 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:57.606 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:48:57.607 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:48:57.616 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:57.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:57.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:57.623 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:48:57.623 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:48:57.639 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:48:57.639 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:48:57.657 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:48:57.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:48:57.676 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:48:57.676 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:48:57.693 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:48:57.693 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:48:57.709 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:48:57.709 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:48:57.717 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:48:57.726 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:48:57.726 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:48:57.738 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:57.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:57.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:48:57.743 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:48:57.743 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:48:57.760 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:48:57.761 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:48:57.777 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:48:57.777 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:48:57.840 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:57.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:57.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:57.963 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:48:57.985 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:57.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:57.986 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:48:58.005 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:48:58.008 INFO:tasks.rados.rados.2.smithi175.stdout:2876: done (6 left) 2024-09-11T03:48:58.008 INFO:tasks.rados.rados.2.smithi175.stdout:2878: done (5 left) 2024-09-11T03:48:58.008 INFO:tasks.rados.rados.2.smithi175.stdout:2880: done (4 left) 2024-09-11T03:48:58.008 INFO:tasks.rados.rados.2.smithi175.stdout:2881: done (3 left) 2024-09-11T03:48:58.008 INFO:tasks.rados.rados.2.smithi175.stdout:2882: done (2 left) 2024-09-11T03:48:58.008 INFO:tasks.rados.rados.2.smithi175.stdout:2884: done (1 left) 2024-09-11T03:48:58.009 INFO:tasks.rados.rados.2.smithi175.stdout:2885: done (0 left) 2024-09-11T03:48:58.009 INFO:tasks.rados.rados.2.smithi175.stdout:2886: snap_create 2024-09-11T03:48:58.087 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:58.108 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:58.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:58.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:58.210 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:48:58.232 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:58.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:58.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:48:58.334 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:58.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:58 smithi084 ceph-mon[27254]: pgmap v1233: 67 pgs: 67 active+clean; 335 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 2.4 MiB/s wr, 21 op/s 2024-09-11T03:48:58.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:58 smithi084 ceph-mon[27254]: 2.23 scrub starts 2024-09-11T03:48:58.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:58 smithi084 ceph-mon[27254]: 2.23 scrub ok 2024-09-11T03:48:58.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:58 smithi084 ceph-mon[27254]: 1.14 scrub starts 2024-09-11T03:48:58.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:58 smithi084 ceph-mon[27254]: 1.14 scrub ok 2024-09-11T03:48:58.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:58 smithi084 ceph-mon[27254]: 1.f scrub starts 2024-09-11T03:48:58.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:58 smithi084 ceph-mon[27254]: 1.f scrub ok 2024-09-11T03:48:58.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:58 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/97529713' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:58.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:58 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3742535424' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:48:58.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:58 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:48:58.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:58 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:48:58.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:58 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3742535424' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:48:58.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:48:58 smithi084 ceph-mon[27254]: osdmap e720: 12 total, 12 up, 12 in 2024-09-11T03:48:58.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:58.397 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:58.397 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:58 smithi121 ceph-mon[30729]: pgmap v1233: 67 pgs: 67 active+clean; 335 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 2.4 MiB/s wr, 21 op/s 2024-09-11T03:48:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:58 smithi121 ceph-mon[30729]: 2.23 scrub starts 2024-09-11T03:48:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:58 smithi121 ceph-mon[30729]: 2.23 scrub ok 2024-09-11T03:48:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:58 smithi121 ceph-mon[30729]: 1.14 scrub starts 2024-09-11T03:48:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:58 smithi121 ceph-mon[30729]: 1.14 scrub ok 2024-09-11T03:48:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:58 smithi121 ceph-mon[30729]: 1.f scrub starts 2024-09-11T03:48:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:58 smithi121 ceph-mon[30729]: 1.f scrub ok 2024-09-11T03:48:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:58 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/97529713' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:58 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3742535424' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:48:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:58 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:48:58.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:58 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:48:58.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:58 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3742535424' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:48:58.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:48:58 smithi121 ceph-mon[30729]: osdmap e720: 12 total, 12 up, 12 in 2024-09-11T03:48:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:58 smithi175 ceph-mon[30894]: pgmap v1233: 67 pgs: 67 active+clean; 335 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 2.4 MiB/s wr, 21 op/s 2024-09-11T03:48:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:58 smithi175 ceph-mon[30894]: 2.23 scrub starts 2024-09-11T03:48:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:58 smithi175 ceph-mon[30894]: 2.23 scrub ok 2024-09-11T03:48:58.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:58 smithi175 ceph-mon[30894]: 1.14 scrub starts 2024-09-11T03:48:58.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:58 smithi175 ceph-mon[30894]: 1.14 scrub ok 2024-09-11T03:48:58.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:58 smithi175 ceph-mon[30894]: 1.f scrub starts 2024-09-11T03:48:58.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:58 smithi175 ceph-mon[30894]: 1.f scrub ok 2024-09-11T03:48:58.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:58 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/97529713' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:48:58.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:58 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3742535424' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:48:58.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:58 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:48:58.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:58 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:48:58.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:58 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3742535424' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:48:58.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:48:58 smithi175 ceph-mon[30894]: osdmap e720: 12 total, 12 up, 12 in 2024-09-11T03:48:58.498 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:48:58.499 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T03:48:58.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:58.521 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:58.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:48:58.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:48:58.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:58.644 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:58.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:48:58.746 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:58.771 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:58.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:58.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:58.784 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:48:58.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:58.895 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:58.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:58.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:58.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:59.010 INFO:tasks.rados.rados.2.smithi175.stdout:2886: done (0 left) 2024-09-11T03:48:59.010 INFO:tasks.rados.rados.2.smithi175.stdout:2887: snap_remove snap 313 2024-09-11T03:48:59.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:59.023 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:59.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:59.063 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-11T03:48:59.063 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-11T03:48:59.063 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-11T03:48:59.063 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-11T03:48:59.063 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-11T03:48:59.063 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-11T03:48:59.064 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-11T03:48:59.064 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-11T03:48:59.064 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-11T03:48:59.064 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-11T03:48:59.064 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-11T03:48:59.064 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-11T03:48:59.124 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:59.150 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:59.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:59.151 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:59.231 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:48:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:48:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:48:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:48:59.231 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:48:59.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:48:59.273 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:48:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:59.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:48:59.375 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:59.401 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:59.402 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:59.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:48:59.525 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:48:59.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:59.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:48:59.627 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:48:59.653 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:59.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:59.654 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:48:59.755 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:59.781 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:59.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:59.782 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:48:59.883 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:48:59.909 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:48:59.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:48:59.912 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T03:48:59.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:00.011 INFO:tasks.rados.rados.2.smithi175.stdout:2887: done (0 left) 2024-09-11T03:49:00.011 INFO:tasks.rados.rados.2.smithi175.stdout:2888: read oid 3 snap -1 2024-09-11T03:49:00.011 INFO:tasks.rados.rados.2.smithi175.stdout:2888: expect deleted 2024-09-11T03:49:00.012 INFO:tasks.rados.rados.2.smithi175.stdout:2889: copy_from oid 30 from oid 48 current snap is 319 2024-09-11T03:49:00.012 INFO:tasks.rados.rados.2.smithi175.stdout:2890: snap_create 2024-09-11T03:49:00.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:00.027 INFO:tasks.rados.rados.2.smithi175.stdout:2889: finishing copy_from racing read to smithi17557870-30 2024-09-11T03:49:00.031 INFO:tasks.rados.rados.2.smithi175.stdout:2889: finishing copy_from to smithi17557870-30 2024-09-11T03:49:00.031 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 962 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:49:00.037 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:00.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:00.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:00.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:00.160 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:49:00.162 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:00.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:00.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:00.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:00 smithi121 ceph-mon[30729]: pgmap v1236: 67 pgs: 67 active+clean; 345 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 4.0 MiB/s wr, 30 op/s 2024-09-11T03:49:00.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:00 smithi121 ceph-mon[30729]: osdmap e721: 12 total, 12 up, 12 in 2024-09-11T03:49:00.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:00 smithi121 ceph-mon[30729]: 1.a scrub starts 2024-09-11T03:49:00.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:00 smithi121 ceph-mon[30729]: 1.a scrub ok 2024-09-11T03:49:00.264 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:00.286 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:00.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:00.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:00.388 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:00.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:00 smithi084 ceph-mon[27254]: pgmap v1236: 67 pgs: 67 active+clean; 345 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 4.0 MiB/s wr, 30 op/s 2024-09-11T03:49:00.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:00 smithi084 ceph-mon[27254]: osdmap e721: 12 total, 12 up, 12 in 2024-09-11T03:49:00.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:00 smithi084 ceph-mon[27254]: 1.a scrub starts 2024-09-11T03:49:00.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:00 smithi084 ceph-mon[27254]: 1.a scrub ok 2024-09-11T03:49:00.410 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:00.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:00.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:00 smithi175 ceph-mon[30894]: pgmap v1236: 67 pgs: 67 active+clean; 345 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 4.0 MiB/s wr, 30 op/s 2024-09-11T03:49:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:00 smithi175 ceph-mon[30894]: osdmap e721: 12 total, 12 up, 12 in 2024-09-11T03:49:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:00 smithi175 ceph-mon[30894]: 1.a scrub starts 2024-09-11T03:49:00.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:00 smithi175 ceph-mon[30894]: 1.a scrub ok 2024-09-11T03:49:00.486 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:49:00.486 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T03:49:00.486 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-09-11T03:49:00.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:00.550 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:00.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:00.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:00.603 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:49:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:49:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:49:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:49:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:588 2.f e129/95 2:fda69313:::smithi17557870-16:head v 129'32, mlcod=124'28)", 2024-09-11T03:49:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:39:05.847680+0000", 2024-09-11T03:49:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 594.75441885299995, 2024-09-11T03:49:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075524320000000004, 2024-09-11T03:49:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.847680+0000", 2024-09-11T03:49:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.847680+0000", 2024-09-11T03:49:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.847681+0000", 2024-09-11T03:49:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.184e-06 2024-09-11T03:49:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.853139+0000", 2024-09-11T03:49:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054575960000000003 2024-09-11T03:49:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.853154+0000", 2024-09-11T03:49:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5285999999999999e-05 2024-09-11T03:49:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.853179+0000", 2024-09-11T03:49:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4380999999999999e-05 2024-09-11T03:49:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.853234+0000", 2024-09-11T03:49:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5164999999999998e-05 2024-09-11T03:49:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.853250+0000", 2024-09-11T03:49:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6541999999999998e-05 2024-09-11T03:49:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.855165+0000", 2024-09-11T03:49:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019147560000000001 2024-09-11T03:49:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:39:05.855233+0000", 2024-09-11T03:49:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7521999999999995e-05 2024-09-11T03:49:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1513 2.5 e212/95 2:a6e3bbd9:::smithi17557870-38:head v 212'178, mlcod=203'174)", 2024-09-11T03:49:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:28.928761+0000", 2024-09-11T03:49:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 511.67333825200001, 2024-09-11T03:49:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0094340099999999996, 2024-09-11T03:49:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.928761+0000", 2024-09-11T03:49:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.928761+0000", 2024-09-11T03:49:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.928761+0000", 2024-09-11T03:49:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.68e-07 2024-09-11T03:49:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.935532+0000", 2024-09-11T03:49:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0067704449999999999 2024-09-11T03:49:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.935544+0000", 2024-09-11T03:49:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2427e-05 2024-09-11T03:49:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.935569+0000", 2024-09-11T03:49:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5397000000000001e-05 2024-09-11T03:49:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.935616+0000", 2024-09-11T03:49:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6248999999999997e-05 2024-09-11T03:49:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.935633+0000", 2024-09-11T03:49:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7405e-05 2024-09-11T03:49:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.938161+0000", 2024-09-11T03:49:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002527928 2024-09-11T03:49:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:28.938195+0000", 2024-09-11T03:49:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3991e-05 2024-09-11T03:49:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1589 2.7 e218/191 2:e77ce595:::smithi17557870-31:head v 218'174, mlcod=218'168)", 2024-09-11T03:49:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:35.117324+0000", 2024-09-11T03:49:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 505.48477513, 2024-09-11T03:49:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0068407770000000001, 2024-09-11T03:49:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.117324+0000", 2024-09-11T03:49:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.117324+0000", 2024-09-11T03:49:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.117328+0000", 2024-09-11T03:49:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3669999999999999e-06 2024-09-11T03:49:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.120926+0000", 2024-09-11T03:49:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035978490000000002 2024-09-11T03:49:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.120944+0000", 2024-09-11T03:49:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.766e-05 2024-09-11T03:49:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.120952+0000", 2024-09-11T03:49:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7819999999999994e-06 2024-09-11T03:49:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.121018+0000", 2024-09-11T03:49:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6452999999999995e-05 2024-09-11T03:49:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.121041+0000", 2024-09-11T03:49:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3102999999999999e-05 2024-09-11T03:49:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.124136+0000", 2024-09-11T03:49:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030948220000000001 2024-09-11T03:49:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:35.124165+0000", 2024-09-11T03:49:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8741000000000001e-05 2024-09-11T03:49:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:1816 2.d e243/95 2:b7bdfe52:::smithi17557870-10:head v 243'237, mlcod=243'233)", 2024-09-11T03:49:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:59.922654+0000", 2024-09-11T03:49:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 480.67944522900001, 2024-09-11T03:49:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092334370000000006, 2024-09-11T03:49:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.922654+0000", 2024-09-11T03:49:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.922654+0000", 2024-09-11T03:49:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.922656+0000", 2024-09-11T03:49:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8500000000000001e-06 2024-09-11T03:49:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.929356+0000", 2024-09-11T03:49:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0066998589999999998 2024-09-11T03:49:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.929367+0000", 2024-09-11T03:49:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.182e-05 2024-09-11T03:49:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.929374+0000", 2024-09-11T03:49:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4649999999999999e-06 2024-09-11T03:49:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.929468+0000", 2024-09-11T03:49:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4549000000000006e-05 2024-09-11T03:49:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.929487+0000", 2024-09-11T03:49:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8870000000000001e-05 2024-09-11T03:49:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.931844+0000", 2024-09-11T03:49:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023565840000000001 2024-09-11T03:49:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:59.931887+0000", 2024-09-11T03:49:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3439999999999997e-05 2024-09-11T03:49:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:2115 2.5 e268/95 2:a6e3bbd9:::smithi17557870-38:head v 268'256, mlcod=260'252)", 2024-09-11T03:49:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:24.876823+0000", 2024-09-11T03:49:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 455.72527611499999, 2024-09-11T03:49:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075190609999999996, 2024-09-11T03:49:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.876823+0000", 2024-09-11T03:49:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.876823+0000", 2024-09-11T03:49:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.876835+0000", 2024-09-11T03:49:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1647e-05 2024-09-11T03:49:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.881215+0000", 2024-09-11T03:49:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043805720000000001 2024-09-11T03:49:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.881229+0000", 2024-09-11T03:49:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4088e-05 2024-09-11T03:49:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.881257+0000", 2024-09-11T03:49:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7937e-05 2024-09-11T03:49:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.881355+0000", 2024-09-11T03:49:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8159999999999995e-05 2024-09-11T03:49:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.881377+0000", 2024-09-11T03:49:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.19e-05 2024-09-11T03:49:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.884304+0000", 2024-09-11T03:49:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002926452 2024-09-11T03:49:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:24.884342+0000", 2024-09-11T03:49:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8305e-05 2024-09-11T03:49:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:2492 2.5 e302/276 2:a1a99782:::smithi17557870-1:head v 302'331, mlcod=297'328)", 2024-09-11T03:49:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:41:58.902222+0000", 2024-09-11T03:49:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 421.69987714799998, 2024-09-11T03:49:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.007615704, 2024-09-11T03:49:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.902222+0000", 2024-09-11T03:49:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.902222+0000", 2024-09-11T03:49:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.902223+0000", 2024-09-11T03:49:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0999999999999999e-07 2024-09-11T03:49:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.907105+0000", 2024-09-11T03:49:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048825120000000003 2024-09-11T03:49:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.907120+0000", 2024-09-11T03:49:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5362999999999998e-05 2024-09-11T03:49:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.907145+0000", 2024-09-11T03:49:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4941999999999999e-05 2024-09-11T03:49:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.907217+0000", 2024-09-11T03:49:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1585e-05 2024-09-11T03:49:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.907233+0000", 2024-09-11T03:49:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5684000000000001e-05 2024-09-11T03:49:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.909765+0000", 2024-09-11T03:49:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025323889999999999 2024-09-11T03:49:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:41:58.909838+0000", 2024-09-11T03:49:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2719000000000006e-05 2024-09-11T03:49:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:2511 2.d e305/95 2:b6a25a79:::smithi17557870-47:head v 305'368, mlcod=304'363)", 2024-09-11T03:49:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:01.878380+0000", 2024-09-11T03:49:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 418.72371873600002, 2024-09-11T03:49:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065527210000000001, 2024-09-11T03:49:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.878380+0000", 2024-09-11T03:49:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.878380+0000", 2024-09-11T03:49:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.878381+0000", 2024-09-11T03:49:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0599999999999999e-07 2024-09-11T03:49:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.883199+0000", 2024-09-11T03:49:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048174150000000002 2024-09-11T03:49:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.883211+0000", 2024-09-11T03:49:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2095e-05 2024-09-11T03:49:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.883218+0000", 2024-09-11T03:49:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5820000000000004e-06 2024-09-11T03:49:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.883261+0000", 2024-09-11T03:49:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3163e-05 2024-09-11T03:49:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.883275+0000", 2024-09-11T03:49:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.344e-05 2024-09-11T03:49:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.884892+0000", 2024-09-11T03:49:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016175580000000001 2024-09-11T03:49:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:01.884933+0000", 2024-09-11T03:49:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0661999999999998e-05 2024-09-11T03:49:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:2568 2.d e314/95 2:b6a25a79:::smithi17557870-47:head v 314'392, mlcod=314'388)", 2024-09-11T03:49:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:10.885931+0000", 2024-09-11T03:49:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 409.71616842600002, 2024-09-11T03:49:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0089540990000000001, 2024-09-11T03:49:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.885931+0000", 2024-09-11T03:49:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.885931+0000", 2024-09-11T03:49:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.885932+0000", 2024-09-11T03:49:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3900000000000003e-07 2024-09-11T03:49:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.892118+0000", 2024-09-11T03:49:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061860530000000004 2024-09-11T03:49:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.892130+0000", 2024-09-11T03:49:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2381999999999999e-05 2024-09-11T03:49:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.892136+0000", 2024-09-11T03:49:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2820000000000002e-06 2024-09-11T03:49:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.892219+0000", 2024-09-11T03:49:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3115000000000006e-05 2024-09-11T03:49:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.892236+0000", 2024-09-11T03:49:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6864e-05 2024-09-11T03:49:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.894845+0000", 2024-09-11T03:49:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002608806 2024-09-11T03:49:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:10.894885+0000", 2024-09-11T03:49:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9657999999999999e-05 2024-09-11T03:49:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:2931 2.5 e346/276 2:a6d17fc0:::smithi17557870-13:head v 346'401, mlcod=340'397)", 2024-09-11T03:49:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:43.125695+0000", 2024-09-11T03:49:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 377.47640420900001, 2024-09-11T03:49:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070637080000000001, 2024-09-11T03:49:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.125695+0000", 2024-09-11T03:49:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.125695+0000", 2024-09-11T03:49:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.125696+0000", 2024-09-11T03:49:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1800000000000004e-07 2024-09-11T03:49:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.130541+0000", 2024-09-11T03:49:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048448349999999996 2024-09-11T03:49:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.130555+0000", 2024-09-11T03:49:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4122e-05 2024-09-11T03:49:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.130562+0000", 2024-09-11T03:49:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1489999999999997e-06 2024-09-11T03:49:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.130646+0000", 2024-09-11T03:49:00.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3823e-05 2024-09-11T03:49:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.130663+0000", 2024-09-11T03:49:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7385e-05 2024-09-11T03:49:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.132716+0000", 2024-09-11T03:49:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020528759999999999 2024-09-11T03:49:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:43.132759+0000", 2024-09-11T03:49:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2599999999999999e-05 2024-09-11T03:49:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:2970 2.d e348/333 2:b1c0f730:::smithi17557870-20:head v 348'452, mlcod=348'448)", 2024-09-11T03:49:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:44.989767+0000", 2024-09-11T03:49:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 375.61233203799998, 2024-09-11T03:49:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006454502, 2024-09-11T03:49:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.648 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.989767+0000", 2024-09-11T03:49:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.989767+0000", 2024-09-11T03:49:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.989768+0000", 2024-09-11T03:49:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5099999999999999e-07 2024-09-11T03:49:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.991761+0000", 2024-09-11T03:49:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019935859999999999 2024-09-11T03:49:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.991795+0000", 2024-09-11T03:49:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3954000000000001e-05 2024-09-11T03:49:00.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.991810+0000", 2024-09-11T03:49:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.502e-05 2024-09-11T03:49:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.991908+0000", 2024-09-11T03:49:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7943999999999996e-05 2024-09-11T03:49:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.991931+0000", 2024-09-11T03:49:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2572999999999999e-05 2024-09-11T03:49:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.996168+0000", 2024-09-11T03:49:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042373039999999999 2024-09-11T03:49:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:44.996222+0000", 2024-09-11T03:49:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3369999999999999e-05 2024-09-11T03:49:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.652 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(osd.8.0:622 2.5 e353/276 2:a6e3bbd9:::smithi17557870-38:7a v 353'425, mlcod=352'421)", 2024-09-11T03:49:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:50.005197+0000", 2024-09-11T03:49:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 370.59690212100003, 2024-09-11T03:49:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078228199999999994, 2024-09-11T03:49:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005197+0000", 2024-09-11T03:49:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005197+0000", 2024-09-11T03:49:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005197+0000", 2024-09-11T03:49:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3300000000000001e-07 2024-09-11T03:49:00.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005208+0000", 2024-09-11T03:49:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0366e-05 2024-09-11T03:49:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005227+0000", 2024-09-11T03:49:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8953999999999998e-05 2024-09-11T03:49:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005235+0000", 2024-09-11T03:49:00.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7369999999999995e-06 2024-09-11T03:49:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005320+0000", 2024-09-11T03:49:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4956000000000004e-05 2024-09-11T03:49:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005357+0000", 2024-09-11T03:49:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6640999999999997e-05 2024-09-11T03:49:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.012805+0000", 2024-09-11T03:49:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0074477349999999996 2024-09-11T03:49:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.013020+0000", 2024-09-11T03:49:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000215198 2024-09-11T03:49:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(osd.8.0:624 2.5 e353/276 2:a6d17fc0:::smithi17557870-13:80 v 353'427, mlcod=352'421)", 2024-09-11T03:49:00.657 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:50.005913+0000", 2024-09-11T03:49:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 370.596185827, 2024-09-11T03:49:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076474830000000001, 2024-09-11T03:49:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005913+0000", 2024-09-11T03:49:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005913+0000", 2024-09-11T03:49:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005914+0000", 2024-09-11T03:49:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0999999999999998e-07 2024-09-11T03:49:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005921+0000", 2024-09-11T03:49:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5849999999999996e-06 2024-09-11T03:49:00.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005929+0000", 2024-09-11T03:49:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3210000000000005e-06 2024-09-11T03:49:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005935+0000", 2024-09-11T03:49:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9560000000000002e-06 2024-09-11T03:49:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005981+0000", 2024-09-11T03:49:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6323999999999999e-05 2024-09-11T03:49:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.005993+0000", 2024-09-11T03:49:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2211e-05 2024-09-11T03:49:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.013516+0000", 2024-09-11T03:49:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075221560000000003 2024-09-11T03:49:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:50.013561+0000", 2024-09-11T03:49:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5219999999999997e-05 2024-09-11T03:49:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:3067 2.f e356/95 2:fda69313:::smithi17557870-16:head v 356'243, mlcod=354'240)", 2024-09-11T03:49:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:53.502578+0000", 2024-09-11T03:49:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 367.09952110799998, 2024-09-11T03:49:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059635189999999996, 2024-09-11T03:49:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.662 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:53.502578+0000", 2024-09-11T03:49:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:53.502578+0000", 2024-09-11T03:49:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:53.502579+0000", 2024-09-11T03:49:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0999999999999999e-07 2024-09-11T03:49:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:53.507218+0000", 2024-09-11T03:49:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046390160000000001 2024-09-11T03:49:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:53.507225+0000", 2024-09-11T03:49:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8080000000000005e-06 2024-09-11T03:49:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:53.507230+0000", 2024-09-11T03:49:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5419999999999999e-06 2024-09-11T03:49:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:53.507267+0000", 2024-09-11T03:49:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6990000000000003e-05 2024-09-11T03:49:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:53.507281+0000", 2024-09-11T03:49:00.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4374e-05 2024-09-11T03:49:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:53.508515+0000", 2024-09-11T03:49:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012336249999999999 2024-09-11T03:49:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:53.508542+0000", 2024-09-11T03:49:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6653999999999999e-05 2024-09-11T03:49:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.666 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:4687 2.18 e503/312 2:1984a3a9:::smithi17557870-23:head v 503'348, mlcod=498'344)", 2024-09-11T03:49:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:45:21.005901+0000", 2024-09-11T03:49:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 219.59619813500001, 2024-09-11T03:49:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.026100851000000001, 2024-09-11T03:49:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.005901+0000", 2024-09-11T03:49:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.005901+0000", 2024-09-11T03:49:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.005901+0000", 2024-09-11T03:49:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0699999999999999e-07 2024-09-11T03:49:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030699+0000", 2024-09-11T03:49:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.024797744 2024-09-11T03:49:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030709+0000", 2024-09-11T03:49:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7669999999999998e-06 2024-09-11T03:49:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030715+0000", 2024-09-11T03:49:00.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3010000000000003e-06 2024-09-11T03:49:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030768+0000", 2024-09-11T03:49:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3219e-05 2024-09-11T03:49:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030783+0000", 2024-09-11T03:49:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.521e-05 2024-09-11T03:49:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.031969+0000", 2024-09-11T03:49:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011853740000000001 2024-09-11T03:49:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.032002+0000", 2024-09-11T03:49:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3028999999999998e-05 2024-09-11T03:49:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:4700 2.5 e504/276 2:a1a99782:::smithi17557870-1:head v 504'697, mlcod=504'693)", 2024-09-11T03:49:00.671 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:45:21.967523+0000", 2024-09-11T03:49:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 218.63457645899999, 2024-09-11T03:49:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0073000740000000001, 2024-09-11T03:49:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.967523+0000", 2024-09-11T03:49:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.967523+0000", 2024-09-11T03:49:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.967524+0000", 2024-09-11T03:49:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3400000000000001e-06 2024-09-11T03:49:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.972450+0000", 2024-09-11T03:49:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004926204 2024-09-11T03:49:00.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.972470+0000", 2024-09-11T03:49:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.001e-05 2024-09-11T03:49:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.972480+0000", 2024-09-11T03:49:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0111e-05 2024-09-11T03:49:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.972544+0000", 2024-09-11T03:49:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-09-11T03:49:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.972564+0000", 2024-09-11T03:49:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0038999999999999e-05 2024-09-11T03:49:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.974656+0000", 2024-09-11T03:49:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020926080000000001 2024-09-11T03:49:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.974823+0000", 2024-09-11T03:49:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016638800000000001 2024-09-11T03:49:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:4741 2.5 e510/276 2:a286c222:::smithi17557870-29:head v 510'708, mlcod=506'704)", 2024-09-11T03:49:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:45:27.954793+0000", 2024-09-11T03:49:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 212.647306125, 2024-09-11T03:49:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0090818500000000007, 2024-09-11T03:49:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.676 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:27.954793+0000", 2024-09-11T03:49:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:27.954793+0000", 2024-09-11T03:49:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:27.954794+0000", 2024-09-11T03:49:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6100000000000001e-07 2024-09-11T03:49:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:27.960783+0000", 2024-09-11T03:49:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059890250000000003 2024-09-11T03:49:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:27.960803+0000", 2024-09-11T03:49:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0046000000000001e-05 2024-09-11T03:49:00.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:27.960811+0000", 2024-09-11T03:49:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1419999999999993e-06 2024-09-11T03:49:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:27.960883+0000", 2024-09-11T03:49:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1967999999999999e-05 2024-09-11T03:49:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:27.960900+0000", 2024-09-11T03:49:00.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7091e-05 2024-09-11T03:49:00.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.680 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:00.680 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:49:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:27.963811+0000", 2024-09-11T03:49:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029111530000000001 2024-09-11T03:49:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:27.963875+0000", 2024-09-11T03:49:00.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3863999999999997e-05 2024-09-11T03:49:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:5613 2.5 e597/276 2:a1a99782:::smithi17557870-1:head v 597'842, mlcod=594'840)", 2024-09-11T03:49:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:46:55.197693+0000", 2024-09-11T03:49:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 125.404406034, 2024-09-11T03:49:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076326980000000003, 2024-09-11T03:49:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.683 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:55.197693+0000", 2024-09-11T03:49:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:55.197693+0000", 2024-09-11T03:49:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:55.197694+0000", 2024-09-11T03:49:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6899999999999996e-07 2024-09-11T03:49:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:55.202881+0000", 2024-09-11T03:49:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051867709999999997 2024-09-11T03:49:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:55.202903+0000", 2024-09-11T03:49:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2175999999999999e-05 2024-09-11T03:49:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:55.202912+0000", 2024-09-11T03:49:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3389999999999999e-06 2024-09-11T03:49:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:55.203017+0000", 2024-09-11T03:49:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104299 2024-09-11T03:49:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:55.203038+0000", 2024-09-11T03:49:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0922999999999999e-05 2024-09-11T03:49:00.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:55.205272+0000", 2024-09-11T03:49:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022346850000000001 2024-09-11T03:49:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:55.205326+0000", 2024-09-11T03:49:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3535999999999997e-05 2024-09-11T03:49:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:5841 2.5 e610/276 2:a51832fe:::smithi17557870-12:head v 610'865, mlcod=608'862)", 2024-09-11T03:49:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:08.049071+0000", 2024-09-11T03:49:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 112.553028084, 2024-09-11T03:49:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060912529999999996, 2024-09-11T03:49:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:08.049071+0000", 2024-09-11T03:49:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:08.049071+0000", 2024-09-11T03:49:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:08.049072+0000", 2024-09-11T03:49:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9800000000000002e-07 2024-09-11T03:49:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:08.053325+0000", 2024-09-11T03:49:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042533279999999998 2024-09-11T03:49:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:08.053340+0000", 2024-09-11T03:49:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4538999999999999e-05 2024-09-11T03:49:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:08.053366+0000", 2024-09-11T03:49:00.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.618e-05 2024-09-11T03:49:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:08.053421+0000", 2024-09-11T03:49:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4728000000000002e-05 2024-09-11T03:49:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:08.053439+0000", 2024-09-11T03:49:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8521000000000001e-05 2024-09-11T03:49:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:08.055128+0000", 2024-09-11T03:49:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016889629999999999 2024-09-11T03:49:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:08.055162+0000", 2024-09-11T03:49:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3995999999999998e-05 2024-09-11T03:49:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:6565 2.5 e670/276 2:a51832fe:::smithi17557870-12:head v 670'938, mlcod=669'935)", 2024-09-11T03:49:00.692 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:08.162109+0000", 2024-09-11T03:49:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 52.439990332000001, 2024-09-11T03:49:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011558758000000001, 2024-09-11T03:49:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:08.162109+0000", 2024-09-11T03:49:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:08.162109+0000", 2024-09-11T03:49:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:08.162109+0000", 2024-09-11T03:49:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-07 2024-09-11T03:49:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:08.169908+0000", 2024-09-11T03:49:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077987150000000003 2024-09-11T03:49:00.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:08.169926+0000", 2024-09-11T03:49:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8241999999999999e-05 2024-09-11T03:49:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:08.169935+0000", 2024-09-11T03:49:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9069999999999996e-06 2024-09-11T03:49:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:08.169995+0000", 2024-09-11T03:49:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9988999999999998e-05 2024-09-11T03:49:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:08.170021+0000", 2024-09-11T03:49:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5381999999999999e-05 2024-09-11T03:49:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:08.173628+0000", 2024-09-11T03:49:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036079240000000002 2024-09-11T03:49:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:08.173668+0000", 2024-09-11T03:49:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9069000000000002e-05 2024-09-11T03:49:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:6611 2.18 e675/312 2:1984a3a9:::smithi17557870-23:head v 675'416, mlcod=668'412)", 2024-09-11T03:49:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:13.005618+0000", 2024-09-11T03:49:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.596480829999997, 2024-09-11T03:49:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0082407399999999999, 2024-09-11T03:49:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:00.697 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:13.005618+0000", 2024-09-11T03:49:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:13.005618+0000", 2024-09-11T03:49:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:13.005619+0000", 2024-09-11T03:49:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.04e-07 2024-09-11T03:49:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:13.010461+0000", 2024-09-11T03:49:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048423980000000004 2024-09-11T03:49:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:13.010482+0000", 2024-09-11T03:49:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1393e-05 2024-09-11T03:49:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:13.010511+0000", 2024-09-11T03:49:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8456999999999999e-05 2024-09-11T03:49:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:13.010624+0000", 2024-09-11T03:49:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113239 2024-09-11T03:49:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:13.010647+0000", 2024-09-11T03:49:00.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3439e-05 2024-09-11T03:49:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:13.013801+0000", 2024-09-11T03:49:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003153861 2024-09-11T03:49:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:13.013859+0000", 2024-09-11T03:49:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7748999999999999e-05 2024-09-11T03:49:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.701 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:00.702 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:00.702 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:49:00.723 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:00.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:00.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:00.730 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:00.825 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:00.850 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:00.851 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:00.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:00.973 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:00.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:00.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:01.022 INFO:tasks.rados.rados.2.smithi175.stdout:2888: done (2 left) 2024-09-11T03:49:01.022 INFO:tasks.rados.rados.2.smithi175.stdout:2889: done (1 left) 2024-09-11T03:49:01.022 INFO:tasks.rados.rados.2.smithi175.stdout:2890: done (0 left) 2024-09-11T03:49:01.022 INFO:tasks.rados.rados.2.smithi175.stdout:2891: write (excl) oid 17 current snap is 320 2024-09-11T03:49:01.022 INFO:tasks.rados.rados.2.smithi175.stdout:2891: seq_num 993 ranges {689351=648147,2124737=564614} 2024-09-11T03:49:01.029 INFO:tasks.rados.rados.2.smithi175.stdout:2891: writing smithi17557870-17 from 689351 to 1337498 tid 1 2024-09-11T03:49:01.037 INFO:tasks.rados.rados.2.smithi175.stdout:2891: writing smithi17557870-17 from 2124737 to 2689351 tid 2 2024-09-11T03:49:01.037 INFO:tasks.rados.rados.2.smithi175.stdout:2892: write (excl) oid 47 current snap is 320 2024-09-11T03:49:01.037 INFO:tasks.rados.rados.2.smithi175.stdout:2892: seq_num 994 ranges {706158=723396,2183219=522939} 2024-09-11T03:49:01.044 INFO:tasks.rados.rados.2.smithi175.stdout:2892: writing smithi17557870-47 from 706158 to 1429554 tid 1 2024-09-11T03:49:01.051 INFO:tasks.rados.rados.2.smithi175.stdout:2892: writing smithi17557870-47 from 2183219 to 2706158 tid 2 2024-09-11T03:49:01.051 INFO:tasks.rados.rados.2.smithi175.stdout:2893: write oid 3 current snap is 320 2024-09-11T03:49:01.051 INFO:tasks.rados.rados.2.smithi175.stdout:2893: seq_num 995 ranges {722965=798645,2125348=597617} 2024-09-11T03:49:01.059 INFO:tasks.rados.rados.2.smithi175.stdout:2893: writing smithi17557870-3 from 722965 to 1521610 tid 1 2024-09-11T03:49:01.066 INFO:tasks.rados.rados.2.smithi175.stdout:2893: writing smithi17557870-3 from 2125348 to 2722965 tid 2 2024-09-11T03:49:01.067 INFO:tasks.rados.rados.2.smithi175.stdout:2894: read oid 7 snap 299 2024-09-11T03:49:01.067 INFO:tasks.rados.rados.2.smithi175.stdout:2894: expect (ObjNum 925 snap 294 seq_num 925) 2024-09-11T03:49:01.067 INFO:tasks.rados.rados.2.smithi175.stdout:2891: finishing write tid 1 to smithi17557870-17 2024-09-11T03:49:01.067 INFO:tasks.rados.rados.2.smithi175.stdout:2891: finishing write tid 2 to smithi17557870-17 2024-09-11T03:49:01.067 INFO:tasks.rados.rados.2.smithi175.stdout:2891: finishing write tid 3 to smithi17557870-17 2024-09-11T03:49:01.067 INFO:tasks.rados.rados.2.smithi175.stdout:2891: finishing write tid 4 to smithi17557870-17 2024-09-11T03:49:01.067 INFO:tasks.rados.rados.2.smithi175.stdout:2891: oid 17 updating version 0 to 996 2024-09-11T03:49:01.067 INFO:tasks.rados.rados.2.smithi175.stdout:2891: oid 17 updating version 996 to 997 2024-09-11T03:49:01.067 INFO:tasks.rados.rados.2.smithi175.stdout:2891: oid 17 version 997 is already newer than 995 2024-09-11T03:49:01.067 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 997 (ObjNum 993 snap 320 seq_num 993) dirty exists 2024-09-11T03:49:01.067 INFO:tasks.rados.rados.2.smithi175.stdout:2891: left oid 17 (ObjNum 993 snap 320 seq_num 993) 2024-09-11T03:49:01.067 INFO:tasks.rados.rados.2.smithi175.stdout:2892: finishing write tid 1 to smithi17557870-47 2024-09-11T03:49:01.067 INFO:tasks.rados.rados.2.smithi175.stdout:2891: done (3 left) 2024-09-11T03:49:01.068 INFO:tasks.rados.rados.2.smithi175.stdout:2895: copy_from oid 6 from oid 39 current snap is 320 2024-09-11T03:49:01.068 INFO:tasks.rados.rados.2.smithi175.stdout:2892: finishing write tid 2 to smithi17557870-47 2024-09-11T03:49:01.068 INFO:tasks.rados.rados.2.smithi175.stdout:2892: finishing write tid 3 to smithi17557870-47 2024-09-11T03:49:01.068 INFO:tasks.rados.rados.2.smithi175.stdout:2892: finishing write tid 4 to smithi17557870-47 2024-09-11T03:49:01.068 INFO:tasks.rados.rados.2.smithi175.stdout:2892: oid 47 updating version 0 to 999 2024-09-11T03:49:01.068 INFO:tasks.rados.rados.2.smithi175.stdout:2892: oid 47 version 999 is already newer than 998 2024-09-11T03:49:01.068 INFO:tasks.rados.rados.2.smithi175.stdout:2892: oid 47 updating version 999 to 1000 2024-09-11T03:49:01.068 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 1000 (ObjNum 994 snap 320 seq_num 994) dirty exists 2024-09-11T03:49:01.068 INFO:tasks.rados.rados.2.smithi175.stdout:2892: left oid 47 (ObjNum 994 snap 320 seq_num 994) 2024-09-11T03:49:01.068 INFO:tasks.rados.rados.2.smithi175.stdout:2893: finishing write tid 1 to smithi17557870-3 2024-09-11T03:49:01.068 INFO:tasks.rados.rados.2.smithi175.stdout:2892: done (3 left) 2024-09-11T03:49:01.068 INFO:tasks.rados.rados.2.smithi175.stdout:2896: delete oid 29 current snap is 320 2024-09-11T03:49:01.069 INFO:tasks.rados.rados.2.smithi175.stdout:2896: done (3 left) 2024-09-11T03:49:01.069 INFO:tasks.rados.rados.2.smithi175.stdout:2897: copy_from oid 20 from oid 34 current snap is 320 2024-09-11T03:49:01.069 INFO:tasks.rados.rados.2.smithi175.stdout:2898: read oid 33 snap -1 2024-09-11T03:49:01.069 INFO:tasks.rados.rados.2.smithi175.stdout:2898: expect (ObjNum 964 snap 310 seq_num 964) 2024-09-11T03:49:01.069 INFO:tasks.rados.rados.2.smithi175.stdout:2899: copy_from oid 12 from oid 45 current snap is 320 2024-09-11T03:49:01.070 INFO:tasks.rados.rados.2.smithi175.stdout:2900: delete oid 24 current snap is 320 2024-09-11T03:49:01.071 INFO:tasks.rados.rados.2.smithi175.stdout:2900: done (6 left) 2024-09-11T03:49:01.071 INFO:tasks.rados.rados.2.smithi175.stdout:2901: delete oid 9 current snap is 320 2024-09-11T03:49:01.074 INFO:tasks.rados.rados.2.smithi175.stdout:2901: done (6 left) 2024-09-11T03:49:01.074 INFO:tasks.rados.rados.2.smithi175.stdout:2902: read oid 27 snap -1 2024-09-11T03:49:01.075 INFO:tasks.rados.rados.2.smithi175.stdout:2902: expect (ObjNum 765 snap 247 seq_num 765) 2024-09-11T03:49:01.075 INFO:tasks.rados.rados.2.smithi175.stdout:2903: snap_create 2024-09-11T03:49:01.075 INFO:tasks.rados.rados.2.smithi175.stdout:2894: expect (ObjNum 925 snap 294 seq_num 925) 2024-09-11T03:49:01.075 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:01.100 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:01.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:01.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:01.112 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:01 smithi084 ceph-mon[27254]: 2.f scrub starts 2024-09-11T03:49:01.112 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:01 smithi084 ceph-mon[27254]: 2.f scrub ok 2024-09-11T03:49:01.112 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:01 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:49:01.112 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:01 smithi084 ceph-mon[27254]: 2.d scrub ok 2024-09-11T03:49:01.112 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:01 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:49:01.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:01 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:49:01.113 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:01 smithi084 ceph-mon[27254]: osdmap e722: 12 total, 12 up, 12 in 2024-09-11T03:49:01.188 INFO:tasks.rados.rados.2.smithi175.stdout:2899: finishing copy_from racing read to smithi17557870-12 2024-09-11T03:49:01.189 INFO:tasks.rados.rados.2.smithi175.stdout:2893: finishing write tid 2 to smithi17557870-3 2024-09-11T03:49:01.189 INFO:tasks.rados.rados.2.smithi175.stdout:2893: finishing write tid 3 to smithi17557870-3 2024-09-11T03:49:01.189 INFO:tasks.rados.rados.2.smithi175.stdout:2893: finishing write tid 4 to smithi17557870-3 2024-09-11T03:49:01.189 INFO:tasks.rados.rados.2.smithi175.stdout:2893: oid 3 updating version 0 to 1091 2024-09-11T03:49:01.189 INFO:tasks.rados.rados.2.smithi175.stdout:2893: oid 3 version 1091 is already newer than 1090 2024-09-11T03:49:01.189 INFO:tasks.rados.rados.2.smithi175.stdout:2893: oid 3 updating version 1091 to 1092 2024-09-11T03:49:01.190 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 1092 (ObjNum 995 snap 320 seq_num 995) dirty exists 2024-09-11T03:49:01.190 INFO:tasks.rados.rados.2.smithi175.stdout:2893: left oid 3 (ObjNum 995 snap 320 seq_num 995) 2024-09-11T03:49:01.190 INFO:tasks.rados.rados.2.smithi175.stdout:2897: finishing copy_from racing read to smithi17557870-20 2024-09-11T03:49:01.190 INFO:tasks.rados.rados.2.smithi175.stdout:2898: expect (ObjNum 964 snap 310 seq_num 964) 2024-09-11T03:49:01.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:01.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:01.225 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-11T03:49:01.225 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-11T03:49:01.225 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-11T03:49:01.226 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-11T03:49:01.226 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-11T03:49:01.226 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-11T03:49:01.226 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-11T03:49:01.226 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-11T03:49:01.226 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-11T03:49:01.226 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-11T03:49:01.226 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-11T03:49:01.226 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-11T03:49:01.313 INFO:tasks.rados.rados.2.smithi175.stdout:2897: finishing copy_from to smithi17557870-20 2024-09-11T03:49:01.313 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 956 (ObjNum 983 snap 315 seq_num 983) dirty exists 2024-09-11T03:49:01.313 INFO:tasks.rados.rados.2.smithi175.stdout:2899: finishing copy_from to smithi17557870-12 2024-09-11T03:49:01.313 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 950 (ObjNum 962 snap 308 seq_num 962) dirty exists 2024-09-11T03:49:01.313 INFO:tasks.rados.rados.2.smithi175.stdout:2902: expect (ObjNum 765 snap 247 seq_num 765) 2024-09-11T03:49:01.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:01.345 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:01.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:01.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:01.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:01 smithi121 ceph-mon[30729]: 2.f scrub starts 2024-09-11T03:49:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:01 smithi121 ceph-mon[30729]: 2.f scrub ok 2024-09-11T03:49:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:01 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:49:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:01 smithi121 ceph-mon[30729]: 2.d scrub ok 2024-09-11T03:49:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:01 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:49:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:01 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:49:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:01 smithi121 ceph-mon[30729]: osdmap e722: 12 total, 12 up, 12 in 2024-09-11T03:49:01.435 INFO:tasks.rados.rados.2.smithi175.stdout:2895: finishing copy_from racing read to smithi17557870-6 2024-09-11T03:49:01.436 INFO:tasks.rados.rados.2.smithi175.stdout:2895: finishing copy_from to smithi17557870-6 2024-09-11T03:49:01.436 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 972 (ObjNum 991 snap 317 seq_num 991) dirty exists 2024-09-11T03:49:01.447 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:01.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:01.470 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:01.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:01.477 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:01 smithi175 ceph-mon[30894]: 2.f scrub starts 2024-09-11T03:49:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:01 smithi175 ceph-mon[30894]: 2.f scrub ok 2024-09-11T03:49:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:01 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:49:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:01 smithi175 ceph-mon[30894]: 2.d scrub ok 2024-09-11T03:49:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:01 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:49:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:01 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:49:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:01 smithi175 ceph-mon[30894]: osdmap e722: 12 total, 12 up, 12 in 2024-09-11T03:49:01.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:01.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:01.594 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:01.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:01.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T03:49:01.694 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:01.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:01.717 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:01.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:01.774 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:49:01.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:01.839 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:01.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:01.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:01.870 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:49:01.870 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T03:49:01.870 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.10 'trim stale osdmaps' 2024-09-11T03:49:01.909 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:49:01.940 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:01.962 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:01.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:01.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:02.010 INFO:tasks.rados.rados.2.smithi175.stdout:2893: done (7 left) 2024-09-11T03:49:02.010 INFO:tasks.rados.rados.2.smithi175.stdout:2894: done (6 left) 2024-09-11T03:49:02.010 INFO:tasks.rados.rados.2.smithi175.stdout:2895: done (5 left) 2024-09-11T03:49:02.011 INFO:tasks.rados.rados.2.smithi175.stdout:2897: done (4 left) 2024-09-11T03:49:02.011 INFO:tasks.rados.rados.2.smithi175.stdout:2898: done (3 left) 2024-09-11T03:49:02.011 INFO:tasks.rados.rados.2.smithi175.stdout:2899: done (2 left) 2024-09-11T03:49:02.011 INFO:tasks.rados.rados.2.smithi175.stdout:2902: done (1 left) 2024-09-11T03:49:02.011 INFO:tasks.rados.rados.2.smithi175.stdout:2903: done (0 left) 2024-09-11T03:49:02.011 INFO:tasks.rados.rados.2.smithi175.stdout:2904: snap_create 2024-09-11T03:49:02.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:02 smithi084 ceph-mon[27254]: 2.20 scrub starts 2024-09-11T03:49:02.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:02 smithi084 ceph-mon[27254]: 2.20 scrub ok 2024-09-11T03:49:02.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:02 smithi084 ceph-mon[27254]: 1.10 scrub starts 2024-09-11T03:49:02.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:02 smithi084 ceph-mon[27254]: 1.10 scrub ok 2024-09-11T03:49:02.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:02 smithi084 ceph-mon[27254]: pgmap v1239: 67 pgs: 67 active+clean; 345 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 10 op/s 2024-09-11T03:49:02.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:02 smithi084 ceph-mon[27254]: osdmap e723: 12 total, 12 up, 12 in 2024-09-11T03:49:02.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:02 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:49:02.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:02 smithi084 ceph-mon[27254]: 2.1 scrub ok 2024-09-11T03:49:02.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:02 smithi084 ceph-mon[27254]: osdmap e724: 12 total, 12 up, 12 in 2024-09-11T03:49:02.064 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:02.086 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:02.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:02.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:02.188 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:02.210 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:02.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:02.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:02.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:02.333 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:02.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:02.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:02.338 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:02 smithi121 ceph-mon[30729]: 2.20 scrub starts 2024-09-11T03:49:02.338 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:02 smithi121 ceph-mon[30729]: 2.20 scrub ok 2024-09-11T03:49:02.338 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:02 smithi121 ceph-mon[30729]: 1.10 scrub starts 2024-09-11T03:49:02.338 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:02 smithi121 ceph-mon[30729]: 1.10 scrub ok 2024-09-11T03:49:02.339 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:02 smithi121 ceph-mon[30729]: pgmap v1239: 67 pgs: 67 active+clean; 345 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 10 op/s 2024-09-11T03:49:02.339 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:02 smithi121 ceph-mon[30729]: osdmap e723: 12 total, 12 up, 12 in 2024-09-11T03:49:02.339 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:02 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:49:02.339 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:02 smithi121 ceph-mon[30729]: 2.1 scrub ok 2024-09-11T03:49:02.339 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:02 smithi121 ceph-mon[30729]: osdmap e724: 12 total, 12 up, 12 in 2024-09-11T03:49:02.356 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:49:02.356 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:49:02.356 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:49:02.357 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:49:02.403 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:49:02.403 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:49:02.418 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:49:02.419 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:49:02.434 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:49:02.434 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:49:02.436 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:02.452 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:49:02.452 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:49:02.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:02.458 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:02.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:02.470 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:49:02.471 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:49:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:02 smithi175 ceph-mon[30894]: 2.20 scrub starts 2024-09-11T03:49:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:02 smithi175 ceph-mon[30894]: 2.20 scrub ok 2024-09-11T03:49:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:02 smithi175 ceph-mon[30894]: 1.10 scrub starts 2024-09-11T03:49:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:02 smithi175 ceph-mon[30894]: 1.10 scrub ok 2024-09-11T03:49:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:02 smithi175 ceph-mon[30894]: pgmap v1239: 67 pgs: 67 active+clean; 345 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 10 op/s 2024-09-11T03:49:02.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:02 smithi175 ceph-mon[30894]: osdmap e723: 12 total, 12 up, 12 in 2024-09-11T03:49:02.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:02 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:49:02.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:02 smithi175 ceph-mon[30894]: 2.1 scrub ok 2024-09-11T03:49:02.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:02 smithi175 ceph-mon[30894]: osdmap e724: 12 total, 12 up, 12 in 2024-09-11T03:49:02.479 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:02.484 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:02.489 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:49:02.489 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:49:02.513 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:49:02.514 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:49:02.530 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:49:02.530 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:49:02.547 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:49:02.547 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:49:02.559 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:02.563 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:49:02.564 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:49:02.580 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:49:02.580 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:49:02.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:02.581 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:02.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:02.597 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:49:02.597 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:49:02.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:02.703 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:02.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:02.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:02.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:02.827 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:02.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:02.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:02.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:02.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:02.951 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:02.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:03.010 INFO:tasks.rados.rados.2.smithi175.stdout:2904: done (0 left) 2024-09-11T03:49:03.010 INFO:tasks.rados.rados.2.smithi175.stdout:2905: copy_from oid 43 from oid 36 current snap is 322 2024-09-11T03:49:03.010 INFO:tasks.rados.rados.2.smithi175.stdout:2906: read oid 9 snap -1 2024-09-11T03:49:03.010 INFO:tasks.rados.rados.2.smithi175.stdout:2906: expect deleted 2024-09-11T03:49:03.010 INFO:tasks.rados.rados.2.smithi175.stdout:2907: read oid 16 snap 309 2024-09-11T03:49:03.010 INFO:tasks.rados.rados.2.smithi175.stdout:2907: expect (ObjNum 940 snap 297 seq_num 940) 2024-09-11T03:49:03.011 INFO:tasks.rados.rados.2.smithi175.stdout:2908: snap_remove snap 319 2024-09-11T03:49:03.012 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T03:49:03.051 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:03.077 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:03.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:03.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:03.114 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-11T03:49:03.114 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-11T03:49:03.114 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-11T03:49:03.114 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-11T03:49:03.114 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-11T03:49:03.114 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-11T03:49:03.114 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-11T03:49:03.114 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-11T03:49:03.114 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-11T03:49:03.115 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-11T03:49:03.115 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-11T03:49:03.115 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-11T03:49:03.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:03.201 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:03.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:03.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:03.257 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:49:03.303 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:03.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:03.330 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:03.330 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:03.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:03 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4242473579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:03.377 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:03 smithi121 ceph-mon[30729]: osdmap e725: 12 total, 12 up, 12 in 2024-09-11T03:49:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:03 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4242473579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:03 smithi084 ceph-mon[27254]: osdmap e725: 12 total, 12 up, 12 in 2024-09-11T03:49:03.432 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:03.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:03.454 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:03.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:03.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:03 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4242473579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:03.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:03 smithi175 ceph-mon[30894]: osdmap e725: 12 total, 12 up, 12 in 2024-09-11T03:49:03.555 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:03.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:03.577 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:03.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:03.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:03.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:03.702 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:03.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:03.747 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:49:03.747 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:49:03.804 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:03.808 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:49:03.809 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T03:49:03.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-11T03:49:03.825 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:03.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:03.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:03.927 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:03.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:03.949 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:03.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:04.011 INFO:tasks.rados.rados.2.smithi175.stdout:2908: done (3 left) 2024-09-11T03:49:04.011 INFO:tasks.rados.rados.2.smithi175.stdout:2909: read oid 35 snap 296 2024-09-11T03:49:04.011 INFO:tasks.rados.rados.2.smithi175.stdout:2909: expect (ObjNum 927 snap 295 seq_num 927) 2024-09-11T03:49:04.012 INFO:tasks.rados.rados.2.smithi175.stdout:2910: snap_remove snap 283 2024-09-11T03:49:04.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:04.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:04.072 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:04.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:04.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:04 smithi084 ceph-mon[27254]: pgmap v1242: 67 pgs: 67 active+clean; 357 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2024-09-11T03:49:04.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:04 smithi084 ceph-mon[27254]: osdmap e726: 12 total, 12 up, 12 in 2024-09-11T03:49:04.174 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:04.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:04.196 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:04.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:04.297 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:04.319 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:04.321 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:04.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:04.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:04.405 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:04 smithi121 ceph-mon[30729]: pgmap v1242: 67 pgs: 67 active+clean; 357 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2024-09-11T03:49:04.405 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:04 smithi121 ceph-mon[30729]: osdmap e726: 12 total, 12 up, 12 in 2024-09-11T03:49:04.423 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:04.448 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:04.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:04 smithi175 ceph-mon[30894]: pgmap v1242: 67 pgs: 67 active+clean; 357 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2024-09-11T03:49:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:04 smithi175 ceph-mon[30894]: osdmap e726: 12 total, 12 up, 12 in 2024-09-11T03:49:04.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:04.572 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:04.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:04.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:04.630 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T03:49:04.675 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:04.700 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:04.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:04.777 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-11T03:49:04.777 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-11T03:49:04.777 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-11T03:49:04.777 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-11T03:49:04.778 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-11T03:49:04.778 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-11T03:49:04.778 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-11T03:49:04.778 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-11T03:49:04.778 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-11T03:49:04.778 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-11T03:49:04.778 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-11T03:49:04.778 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-11T03:49:04.801 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:04.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:04.828 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:04.828 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:04.871 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:49:04.929 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:04.954 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:04.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:04.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:05.009 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:49:05.013 INFO:tasks.rados.rados.2.smithi175.stdout:2910: done (4 left) 2024-09-11T03:49:05.013 INFO:tasks.rados.rados.2.smithi175.stdout:2911: snap_create 2024-09-11T03:49:05.013 INFO:tasks.rados.rados.2.smithi175.stdout:2905: finishing copy_from racing read to smithi17557870-43 2024-09-11T03:49:05.013 INFO:tasks.rados.rados.2.smithi175.stdout:2905: finishing copy_from to smithi17557870-43 2024-09-11T03:49:05.013 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 962 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:49:05.013 INFO:tasks.rados.rados.2.smithi175.stdout:2907: expect (ObjNum 940 snap 297 seq_num 940) 2024-09-11T03:49:05.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:05.078 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:05.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:05.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:05.139 INFO:tasks.rados.rados.2.smithi175.stdout:2909: expect (ObjNum 927 snap 295 seq_num 927) 2024-09-11T03:49:05.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:05.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:05.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:05.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:05.247 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:05 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3716991097' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:49:05.247 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:05 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:05.247 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:05 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3716991097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:49:05.248 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:05 smithi121 ceph-mon[30729]: osdmap e727: 12 total, 12 up, 12 in 2024-09-11T03:49:05.273 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:05 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3716991097' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:49:05.273 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:05 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:05.273 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:05 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3716991097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:49:05.273 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:05 smithi084 ceph-mon[27254]: osdmap e727: 12 total, 12 up, 12 in 2024-09-11T03:49:05.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:05.325 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:05.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:05.348 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:49:05.348 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:49:05.427 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:05.448 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:05.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:05.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:05 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3716991097' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:49:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:05 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:05 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3716991097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:49:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:05 smithi175 ceph-mon[30894]: osdmap e727: 12 total, 12 up, 12 in 2024-09-11T03:49:05.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:05.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:05.572 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:05.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:05.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:05.695 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:05.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:05.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:05.797 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:05.820 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:05.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:05.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:05.899 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:49:05.900 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T03:49:05.921 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:05.944 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:05.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:06.004 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T03:49:06.014 INFO:tasks.rados.rados.2.smithi175.stdout:2905: done (4 left) 2024-09-11T03:49:06.014 INFO:tasks.rados.rados.2.smithi175.stdout:2906: done (3 left) 2024-09-11T03:49:06.014 INFO:tasks.rados.rados.2.smithi175.stdout:2907: done (2 left) 2024-09-11T03:49:06.014 INFO:tasks.rados.rados.2.smithi175.stdout:2909: done (1 left) 2024-09-11T03:49:06.014 INFO:tasks.rados.rados.2.smithi175.stdout:2911: done (0 left) 2024-09-11T03:49:06.014 INFO:tasks.rados.rados.2.smithi175.stdout:2912: snap_create 2024-09-11T03:49:06.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:06.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:06.068 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:06.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:06.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:06 smithi084 ceph-mon[27254]: pgmap v1245: 67 pgs: 67 active+clean; 359 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.3 MiB/s wr, 15 op/s 2024-09-11T03:49:06.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:06 smithi084 ceph-mon[27254]: osdmap e728: 12 total, 12 up, 12 in 2024-09-11T03:49:06.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:06.192 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:06.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:06.247 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:49:06.293 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:06.317 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:06.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:06.318 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:06.367 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:06 smithi121 ceph-mon[30729]: pgmap v1245: 67 pgs: 67 active+clean; 359 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.3 MiB/s wr, 15 op/s 2024-09-11T03:49:06.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:06 smithi121 ceph-mon[30729]: osdmap e728: 12 total, 12 up, 12 in 2024-09-11T03:49:06.419 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:06.442 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:06.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:06.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:06.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:06 smithi175 ceph-mon[30894]: pgmap v1245: 67 pgs: 67 active+clean; 359 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.3 MiB/s wr, 15 op/s 2024-09-11T03:49:06.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:06 smithi175 ceph-mon[30894]: osdmap e728: 12 total, 12 up, 12 in 2024-09-11T03:49:06.543 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:06.567 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:06.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:06.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:06.669 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:06.692 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:06.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:06.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:06.719 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:49:06.719 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:49:06.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:06.816 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:06.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:06.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:06.943 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:06.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:06.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:07.013 INFO:tasks.rados.rados.2.smithi175.stdout:2912: done (0 left) 2024-09-11T03:49:07.013 INFO:tasks.rados.rados.2.smithi175.stdout:2913: copy_from oid 22 from oid 34 current snap is 324 2024-09-11T03:49:07.013 INFO:tasks.rados.rados.2.smithi175.stdout:2914: delete oid 47 current snap is 324 2024-09-11T03:49:07.018 INFO:tasks.rados.rados.2.smithi175.stdout:2914: done (1 left) 2024-09-11T03:49:07.018 INFO:tasks.rados.rados.2.smithi175.stdout:2915: delete oid 6 current snap is 324 2024-09-11T03:49:07.021 INFO:tasks.rados.rados.2.smithi175.stdout:2913: finishing copy_from racing read to smithi17557870-22 2024-09-11T03:49:07.023 INFO:tasks.rados.rados.2.smithi175.stdout:2913: finishing copy_from to smithi17557870-22 2024-09-11T03:49:07.023 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 956 (ObjNum 983 snap 315 seq_num 983) dirty exists 2024-09-11T03:49:07.024 INFO:tasks.rados.rados.2.smithi175.stdout:2913: done (1 left) 2024-09-11T03:49:07.024 INFO:tasks.rados.rados.2.smithi175.stdout:2915: done (0 left) 2024-09-11T03:49:07.024 INFO:tasks.rados.rados.2.smithi175.stdout:2916: write (excl) oid 50 current snap is 324 2024-09-11T03:49:07.024 INFO:tasks.rados.rados.2.smithi175.stdout:2916: seq_num 1001 ranges {423807=566492,1676240=407685,2739004=84803} 2024-09-11T03:49:07.029 INFO:tasks.rados.rados.2.smithi175.stdout:2916: writing smithi17557870-50 from 423807 to 990299 tid 1 2024-09-11T03:49:07.035 INFO:tasks.rados.rados.2.smithi175.stdout:2916: writing smithi17557870-50 from 1676240 to 2083925 tid 2 2024-09-11T03:49:07.040 INFO:tasks.rados.rados.2.smithi175.stdout:2916: writing smithi17557870-50 from 2739004 to 2823807 tid 3 2024-09-11T03:49:07.040 INFO:tasks.rados.rados.2.smithi175.stdout:2917: delete oid 24 current snap is 324 2024-09-11T03:49:07.040 INFO:tasks.rados.rados.2.smithi175.stdout:2916: finishing write tid 1 to smithi17557870-50 2024-09-11T03:49:07.040 INFO:tasks.rados.rados.2.smithi175.stdout:2916: finishing write tid 2 to smithi17557870-50 2024-09-11T03:49:07.041 INFO:tasks.rados.rados.2.smithi175.stdout:2917: done (1 left) 2024-09-11T03:49:07.041 INFO:tasks.rados.rados.2.smithi175.stdout:2918: write (excl) oid 39 current snap is 324 2024-09-11T03:49:07.041 INFO:tasks.rados.rados.2.smithi175.stdout:2918: seq_num 1003 ranges {457421=716990,1676851=411354,2677497=179924} 2024-09-11T03:49:07.046 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:07.050 INFO:tasks.rados.rados.2.smithi175.stdout:2918: writing smithi17557870-39 from 457421 to 1174411 tid 1 2024-09-11T03:49:07.053 INFO:tasks.rados.rados.2.smithi175.stdout:2918: writing smithi17557870-39 from 1676851 to 2088205 tid 2 2024-09-11T03:49:07.058 INFO:tasks.rados.rados.2.smithi175.stdout:2918: writing smithi17557870-39 from 2677497 to 2857421 tid 3 2024-09-11T03:49:07.058 INFO:tasks.rados.rados.2.smithi175.stdout:2919: read oid 30 snap -1 2024-09-11T03:49:07.058 INFO:tasks.rados.rados.2.smithi175.stdout:2919: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:49:07.058 INFO:tasks.rados.rados.2.smithi175.stdout:2920: write (excl) oid 26 current snap is 324 2024-09-11T03:49:07.058 INFO:tasks.rados.rados.2.smithi175.stdout:2916: finishing write tid 3 to smithi17557870-50 2024-09-11T03:49:07.058 INFO:tasks.rados.rados.2.smithi175.stdout:2916: finishing write tid 4 to smithi17557870-50 2024-09-11T03:49:07.058 INFO:tasks.rados.rados.2.smithi175.stdout:2920: seq_num 1004 ranges {474228=508592,1451686=555012,2704920=169308} 2024-09-11T03:49:07.063 INFO:tasks.rados.rados.2.smithi175.stdout:2920: writing smithi17557870-26 from 474228 to 982820 tid 1 2024-09-11T03:49:07.069 INFO:tasks.rados.rados.2.smithi175.stdout:2920: writing smithi17557870-26 from 1451686 to 2006698 tid 2 2024-09-11T03:49:07.073 INFO:tasks.rados.rados.2.smithi175.stdout:2920: writing smithi17557870-26 from 2704920 to 2874228 tid 3 2024-09-11T03:49:07.074 INFO:tasks.rados.rados.2.smithi175.stdout:2921: snap_remove snap 309 2024-09-11T03:49:07.074 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:07.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:07.075 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:07.176 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:07.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:07.201 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:07.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:07.300 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:49:07.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:07.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:07.324 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:07.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:07.426 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:07.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T03:49:07.447 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:07.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:07.547 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:07.549 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:07.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:07.577 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:07.577 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:07.678 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:07.686 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:49:07.704 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:07.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:07.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:07.806 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:07.828 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:07.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:07.929 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:07.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:07.951 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:07.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:07.996 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:49:07.996 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:49:08.011 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:49:08.011 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:49:08.027 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:49:08.027 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:49:08.040 INFO:tasks.rados.rados.2.smithi175.stdout:2921: done (4 left) 2024-09-11T03:49:08.040 INFO:tasks.rados.rados.2.smithi175.stdout:2922: read oid 13 snap -1 2024-09-11T03:49:08.040 INFO:tasks.rados.rados.2.smithi175.stdout:2922: expect (ObjNum 907 snap 287 seq_num 907) 2024-09-11T03:49:08.040 INFO:tasks.rados.rados.2.smithi175.stdout:2923: write (excl) oid 28 current snap is 324 2024-09-11T03:49:08.040 INFO:tasks.rados.rados.2.smithi175.stdout:2923: seq_num 1005 ranges {491035=583841,1510168=698670,2732343=158692} 2024-09-11T03:49:08.047 INFO:tasks.rados.rados.2.smithi175.stdout:2923: writing smithi17557870-28 from 491035 to 1074876 tid 1 2024-09-11T03:49:08.049 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:49:08.049 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:49:08.052 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:08.054 INFO:tasks.rados.rados.2.smithi175.stdout:2923: writing smithi17557870-28 from 1510168 to 2208838 tid 2 2024-09-11T03:49:08.059 INFO:tasks.rados.rados.2.smithi175.stdout:2923: writing smithi17557870-28 from 2732343 to 2891035 tid 3 2024-09-11T03:49:08.059 INFO:tasks.rados.rados.2.smithi175.stdout:2924: read oid 5 snap -1 2024-09-11T03:49:08.059 INFO:tasks.rados.rados.2.smithi175.stdout:2924: expect (ObjNum 758 snap 246 seq_num 758) 2024-09-11T03:49:08.059 INFO:tasks.rados.rados.2.smithi175.stdout:2925: write (excl) oid 49 current snap is 324 2024-09-11T03:49:08.059 INFO:tasks.rados.rados.2.smithi175.stdout:2916: finishing write tid 5 to smithi17557870-50 2024-09-11T03:49:08.059 INFO:tasks.rados.rados.2.smithi175.stdout:2916: oid 50 updating version 0 to 1106 2024-09-11T03:49:08.059 INFO:tasks.rados.rados.2.smithi175.stdout:2916: oid 50 version 1106 is already newer than 1103 2024-09-11T03:49:08.060 INFO:tasks.rados.rados.2.smithi175.stdout:2916: oid 50 version 1106 is already newer than 1104 2024-09-11T03:49:08.060 INFO:tasks.rados.rados.2.smithi175.stdout:2916: oid 50 version 1106 is already newer than 1105 2024-09-11T03:49:08.060 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 1106 (ObjNum 1001 snap 324 seq_num 1001) dirty exists 2024-09-11T03:49:08.060 INFO:tasks.rados.rados.2.smithi175.stdout:2916: left oid 50 (ObjNum 1001 snap 324 seq_num 1001) 2024-09-11T03:49:08.060 INFO:tasks.rados.rados.2.smithi175.stdout:2918: finishing write tid 1 to smithi17557870-39 2024-09-11T03:49:08.060 INFO:tasks.rados.rados.2.smithi175.stdout:2918: finishing write tid 2 to smithi17557870-39 2024-09-11T03:49:08.060 INFO:tasks.rados.rados.2.smithi175.stdout:2918: finishing write tid 3 to smithi17557870-39 2024-09-11T03:49:08.060 INFO:tasks.rados.rados.2.smithi175.stdout:2918: finishing write tid 4 to smithi17557870-39 2024-09-11T03:49:08.060 INFO:tasks.rados.rados.2.smithi175.stdout:2925: seq_num 1006 ranges {507842=659090,1568650=558681,2759766=148076} 2024-09-11T03:49:08.065 INFO:tasks.rados.rados.2.smithi175.stdout:2925: writing smithi17557870-49 from 507842 to 1166932 tid 1 2024-09-11T03:49:08.069 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:49:08.069 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:49:08.070 INFO:tasks.rados.rados.2.smithi175.stdout:2925: writing smithi17557870-49 from 1568650 to 2127331 tid 2 2024-09-11T03:49:08.075 INFO:tasks.rados.rados.2.smithi175.stdout:2925: writing smithi17557870-49 from 2759766 to 2907842 tid 3 2024-09-11T03:49:08.075 INFO:tasks.rados.rados.2.smithi175.stdout:2916: done (7 left) 2024-09-11T03:49:08.075 INFO:tasks.rados.rados.2.smithi175.stdout:2926: read oid 25 snap -1 2024-09-11T03:49:08.075 INFO:tasks.rados.rados.2.smithi175.stdout:2926: expect (ObjNum 950 snap 303 seq_num 950) 2024-09-11T03:49:08.075 INFO:tasks.rados.rados.2.smithi175.stdout:2927: write (excl) oid 11 current snap is 324 2024-09-11T03:49:08.076 INFO:tasks.rados.rados.2.smithi175.stdout:2918: finishing write tid 5 to smithi17557870-39 2024-09-11T03:49:08.076 INFO:tasks.rados.rados.2.smithi175.stdout:2918: oid 39 updating version 0 to 976 2024-09-11T03:49:08.076 INFO:tasks.rados.rados.2.smithi175.stdout:2918: oid 39 version 976 is already newer than 974 2024-09-11T03:49:08.076 INFO:tasks.rados.rados.2.smithi175.stdout:2918: oid 39 version 976 is already newer than 975 2024-09-11T03:49:08.076 INFO:tasks.rados.rados.2.smithi175.stdout:2918: oid 39 version 976 is already newer than 973 2024-09-11T03:49:08.076 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 976 (ObjNum 1003 snap 324 seq_num 1003) dirty exists 2024-09-11T03:49:08.076 INFO:tasks.rados.rados.2.smithi175.stdout:2918: left oid 39 (ObjNum 1003 snap 324 seq_num 1003) 2024-09-11T03:49:08.076 INFO:tasks.rados.rados.2.smithi175.stdout:2919: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:49:08.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:08.077 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:08.077 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:08.088 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:49:08.089 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:49:08.106 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:49:08.106 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:49:08.122 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:49:08.122 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:49:08.139 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:49:08.140 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:49:08.152 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:08 smithi121 ceph-mon[30729]: pgmap v1248: 67 pgs: 67 active+clean; 357 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 661 KiB/s wr, 4 op/s 2024-09-11T03:49:08.152 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:08 smithi121 ceph-mon[30729]: osdmap e729: 12 total, 12 up, 12 in 2024-09-11T03:49:08.152 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:08 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2173621904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:08.155 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:49:08.155 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:49:08.155 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:49:08.155 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:49:08.156 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:49:08.156 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:49:08.156 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:49:08.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:49:08.172 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:49:08.178 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:08.187 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:49:08.188 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:49:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:08.201 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:08.201 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:08.301 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:08.315 INFO:tasks.rados.rados.2.smithi175.stdout:2920: finishing write tid 1 to smithi17557870-26 2024-09-11T03:49:08.315 INFO:tasks.rados.rados.2.smithi175.stdout:2920: finishing write tid 2 to smithi17557870-26 2024-09-11T03:49:08.315 INFO:tasks.rados.rados.2.smithi175.stdout:2920: finishing write tid 3 to smithi17557870-26 2024-09-11T03:49:08.315 INFO:tasks.rados.rados.2.smithi175.stdout:2920: finishing write tid 4 to smithi17557870-26 2024-09-11T03:49:08.315 INFO:tasks.rados.rados.2.smithi175.stdout:2927: seq_num 1007 ranges {524649=734339,1910779=702339} 2024-09-11T03:49:08.321 INFO:tasks.rados.rados.2.smithi175.stdout:2927: writing smithi17557870-11 from 524649 to 1258988 tid 1 2024-09-11T03:49:08.322 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:08.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:08.323 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:08.329 INFO:tasks.rados.rados.2.smithi175.stdout:2927: writing smithi17557870-11 from 1910779 to 2613118 tid 2 2024-09-11T03:49:08.329 INFO:tasks.rados.rados.2.smithi175.stdout:2918: done (8 left) 2024-09-11T03:49:08.329 INFO:tasks.rados.rados.2.smithi175.stdout:2919: done (7 left) 2024-09-11T03:49:08.330 INFO:tasks.rados.rados.2.smithi175.stdout:2928: snap_create 2024-09-11T03:49:08.330 INFO:tasks.rados.rados.2.smithi175.stdout:2920: finishing write tid 5 to smithi17557870-26 2024-09-11T03:49:08.330 INFO:tasks.rados.rados.2.smithi175.stdout:2920: oid 26 updating version 0 to 957 2024-09-11T03:49:08.330 INFO:tasks.rados.rados.2.smithi175.stdout:2920: oid 26 updating version 957 to 960 2024-09-11T03:49:08.330 INFO:tasks.rados.rados.2.smithi175.stdout:2920: oid 26 version 960 is already newer than 958 2024-09-11T03:49:08.330 INFO:tasks.rados.rados.2.smithi175.stdout:2920: oid 26 version 960 is already newer than 959 2024-09-11T03:49:08.330 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 960 (ObjNum 1004 snap 324 seq_num 1004) dirty exists 2024-09-11T03:49:08.330 INFO:tasks.rados.rados.2.smithi175.stdout:2920: left oid 26 (ObjNum 1004 snap 324 seq_num 1004) 2024-09-11T03:49:08.330 INFO:tasks.rados.rados.2.smithi175.stdout:2922: expect (ObjNum 907 snap 287 seq_num 907) 2024-09-11T03:49:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:08 smithi084 ceph-mon[27254]: pgmap v1248: 67 pgs: 67 active+clean; 357 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 661 KiB/s wr, 4 op/s 2024-09-11T03:49:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:08 smithi084 ceph-mon[27254]: osdmap e729: 12 total, 12 up, 12 in 2024-09-11T03:49:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:08 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2173621904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:08.424 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:08.445 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:08.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:08.446 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:08 smithi175 ceph-mon[30894]: pgmap v1248: 67 pgs: 67 active+clean; 357 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 661 KiB/s wr, 4 op/s 2024-09-11T03:49:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:08 smithi175 ceph-mon[30894]: osdmap e729: 12 total, 12 up, 12 in 2024-09-11T03:49:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:08 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2173621904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:08.519 INFO:tasks.rados.rados.2.smithi175.stdout:2923: finishing write tid 1 to smithi17557870-28 2024-09-11T03:49:08.519 INFO:tasks.rados.rados.2.smithi175.stdout:2923: finishing write tid 2 to smithi17557870-28 2024-09-11T03:49:08.520 INFO:tasks.rados.rados.2.smithi175.stdout:2923: finishing write tid 3 to smithi17557870-28 2024-09-11T03:49:08.520 INFO:tasks.rados.rados.2.smithi175.stdout:2923: finishing write tid 4 to smithi17557870-28 2024-09-11T03:49:08.520 INFO:tasks.rados.rados.2.smithi175.stdout:2923: finishing write tid 5 to smithi17557870-28 2024-09-11T03:49:08.520 INFO:tasks.rados.rados.2.smithi175.stdout:2923: oid 28 updating version 0 to 1115 2024-09-11T03:49:08.520 INFO:tasks.rados.rados.2.smithi175.stdout:2923: oid 28 version 1115 is already newer than 1114 2024-09-11T03:49:08.520 INFO:tasks.rados.rados.2.smithi175.stdout:2923: oid 28 version 1115 is already newer than 1112 2024-09-11T03:49:08.520 INFO:tasks.rados.rados.2.smithi175.stdout:2923: oid 28 version 1115 is already newer than 1113 2024-09-11T03:49:08.520 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 1115 (ObjNum 1005 snap 324 seq_num 1005) dirty exists 2024-09-11T03:49:08.520 INFO:tasks.rados.rados.2.smithi175.stdout:2923: left oid 28 (ObjNum 1005 snap 324 seq_num 1005) 2024-09-11T03:49:08.520 INFO:tasks.rados.rados.2.smithi175.stdout:2924: expect (ObjNum 758 snap 246 seq_num 758) 2024-09-11T03:49:08.547 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:08.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:49:08.568 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:08.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:08.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:08.608 INFO:tasks.rados.rados.2.smithi175.stdout:2925: finishing write tid 1 to smithi17557870-49 2024-09-11T03:49:08.609 INFO:tasks.rados.rados.2.smithi175.stdout:2925: finishing write tid 2 to smithi17557870-49 2024-09-11T03:49:08.609 INFO:tasks.rados.rados.2.smithi175.stdout:2925: finishing write tid 3 to smithi17557870-49 2024-09-11T03:49:08.609 INFO:tasks.rados.rados.2.smithi175.stdout:2925: finishing write tid 4 to smithi17557870-49 2024-09-11T03:49:08.609 INFO:tasks.rados.rados.2.smithi175.stdout:2925: finishing write tid 5 to smithi17557870-49 2024-09-11T03:49:08.609 INFO:tasks.rados.rados.2.smithi175.stdout:2925: oid 49 updating version 0 to 959 2024-09-11T03:49:08.609 INFO:tasks.rados.rados.2.smithi175.stdout:2925: oid 49 updating version 959 to 960 2024-09-11T03:49:08.609 INFO:tasks.rados.rados.2.smithi175.stdout:2925: oid 49 version 960 is already newer than 958 2024-09-11T03:49:08.609 INFO:tasks.rados.rados.2.smithi175.stdout:2925: oid 49 updating version 960 to 961 2024-09-11T03:49:08.609 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 961 (ObjNum 1006 snap 324 seq_num 1006) dirty exists 2024-09-11T03:49:08.609 INFO:tasks.rados.rados.2.smithi175.stdout:2925: left oid 49 (ObjNum 1006 snap 324 seq_num 1006) 2024-09-11T03:49:08.609 INFO:tasks.rados.rados.2.smithi175.stdout:2926: expect (ObjNum 950 snap 303 seq_num 950) 2024-09-11T03:49:08.670 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:08.691 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:08.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:08.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:08.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:08.817 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:08.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:08.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:08.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T03:49:08.856 INFO:tasks.rados.rados.2.smithi175.stdout:2927: finishing write tid 1 to smithi17557870-11 2024-09-11T03:49:08.856 INFO:tasks.rados.rados.2.smithi175.stdout:2927: finishing write tid 2 to smithi17557870-11 2024-09-11T03:49:08.857 INFO:tasks.rados.rados.2.smithi175.stdout:2927: finishing write tid 3 to smithi17557870-11 2024-09-11T03:49:08.857 INFO:tasks.rados.rados.2.smithi175.stdout:2927: finishing write tid 4 to smithi17557870-11 2024-09-11T03:49:08.857 INFO:tasks.rados.rados.2.smithi175.stdout:2927: oid 11 updating version 0 to 616 2024-09-11T03:49:08.857 INFO:tasks.rados.rados.2.smithi175.stdout:2927: oid 11 version 616 is already newer than 615 2024-09-11T03:49:08.857 INFO:tasks.rados.rados.2.smithi175.stdout:2927: oid 11 version 616 is already newer than 614 2024-09-11T03:49:08.857 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 616 (ObjNum 1007 snap 324 seq_num 1007) dirty exists 2024-09-11T03:49:08.857 INFO:tasks.rados.rados.2.smithi175.stdout:2927: left oid 11 (ObjNum 1007 snap 324 seq_num 1007) 2024-09-11T03:49:08.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:08.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:08.941 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:08.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:09.042 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:09.045 INFO:tasks.rados.rados.2.smithi175.stdout:2920: done (7 left) 2024-09-11T03:49:09.046 INFO:tasks.rados.rados.2.smithi175.stdout:2922: done (6 left) 2024-09-11T03:49:09.046 INFO:tasks.rados.rados.2.smithi175.stdout:2923: done (5 left) 2024-09-11T03:49:09.046 INFO:tasks.rados.rados.2.smithi175.stdout:2924: done (4 left) 2024-09-11T03:49:09.046 INFO:tasks.rados.rados.2.smithi175.stdout:2925: done (3 left) 2024-09-11T03:49:09.046 INFO:tasks.rados.rados.2.smithi175.stdout:2926: done (2 left) 2024-09-11T03:49:09.046 INFO:tasks.rados.rados.2.smithi175.stdout:2927: done (1 left) 2024-09-11T03:49:09.046 INFO:tasks.rados.rados.2.smithi175.stdout:2928: done (0 left) 2024-09-11T03:49:09.047 INFO:tasks.rados.rados.2.smithi175.stdout:2929: snap_create 2024-09-11T03:49:09.052 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:09.064 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:09.064 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:09.111 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:49:09.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:09.185 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:09.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:09.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:09.201 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:09 smithi084 ceph-mon[27254]: osdmap e730: 12 total, 12 up, 12 in 2024-09-11T03:49:09.201 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:09 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:49:09.286 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:09.307 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:09.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:09.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:09.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:09.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:09 smithi121 ceph-mon[30729]: osdmap e730: 12 total, 12 up, 12 in 2024-09-11T03:49:09.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:09 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:49:09.429 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:09.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:09 smithi175 ceph-mon[30894]: osdmap e730: 12 total, 12 up, 12 in 2024-09-11T03:49:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:09 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:49:09.531 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:09.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:09.554 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:09.554 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:09.655 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:09.677 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:09.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:09.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:09.772 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:49:09.772 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:49:09.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:49:09.772 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:49:09.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.772 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5510 2.7 2:e23d1762:::smithi17557870-21:head [write 538022~738348,stat] snapc f5=[f4,f0,ee] ondisk+write+known_if_redirected+supports_pool_eio e588)", 2024-09-11T03:49:09.772 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:46:46.097904+0000", 2024-09-11T03:49:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 143.667205556, 2024-09-11T03:49:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0093603060000000005, 2024-09-11T03:49:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:49:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:49:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5510 2024-09-11T03:49:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.097904+0000", 2024-09-11T03:49:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.097904+0000", 2024-09-11T03:49:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.097906+0000", 2024-09-11T03:49:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.012e-06 2024-09-11T03:49:09.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.098914+0000", 2024-09-11T03:49:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010074820000000001 2024-09-11T03:49:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.098930+0000", 2024-09-11T03:49:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5968e-05 2024-09-11T03:49:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.098939+0000", 2024-09-11T03:49:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6390000000000004e-06 2024-09-11T03:49:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.099043+0000", 2024-09-11T03:49:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103501 2024-09-11T03:49:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:49:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.099049+0000", 2024-09-11T03:49:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-09-11T03:49:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.100984+0000", 2024-09-11T03:49:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019355360000000001 2024-09-11T03:49:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.101086+0000", 2024-09-11T03:49:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010211400000000001 2024-09-11T03:49:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:49:09.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.101272+0000", 2024-09-11T03:49:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018546200000000001 2024-09-11T03:49:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:49:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.104082+0000", 2024-09-11T03:49:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028102560000000001 2024-09-11T03:49:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.104465+0000", 2024-09-11T03:49:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000383018 2024-09-11T03:49:09.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.107181+0000", 2024-09-11T03:49:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027157430000000001 2024-09-11T03:49:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.107225+0000", 2024-09-11T03:49:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4048999999999998e-05 2024-09-11T03:49:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:46.107264+0000", 2024-09-11T03:49:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9452000000000001e-05 2024-09-11T03:49:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5521 2.7 2:e0eabcfb:::smithi17557870-9:head [stat,delete] snapc f5=[f0] ondisk+write+known_if_redirected+supports_pool_eio e590)", 2024-09-11T03:49:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:46:48.098690+0000", 2024-09-11T03:49:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 141.666420176, 2024-09-11T03:49:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.008942574, 2024-09-11T03:49:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.780 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:49:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:49:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:49:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5521 2024-09-11T03:49:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.098690+0000", 2024-09-11T03:49:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.098690+0000", 2024-09-11T03:49:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.098693+0000", 2024-09-11T03:49:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.977e-06 2024-09-11T03:49:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.098702+0000", 2024-09-11T03:49:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4199999999999996e-06 2024-09-11T03:49:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.098727+0000", 2024-09-11T03:49:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4856999999999999e-05 2024-09-11T03:49:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.098739+0000", 2024-09-11T03:49:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2520999999999999e-05 2024-09-11T03:49:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.098852+0000", 2024-09-11T03:49:09.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011317 2024-09-11T03:49:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:49:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.098862+0000", 2024-09-11T03:49:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7629999999999995e-06 2024-09-11T03:49:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.103253+0000", 2024-09-11T03:49:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043909420000000001 2024-09-11T03:49:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.103339+0000", 2024-09-11T03:49:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6062000000000004e-05 2024-09-11T03:49:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:48.107632+0000", 2024-09-11T03:49:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042928619999999997 2024-09-11T03:49:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5652 2.7 2:e0eabcfb:::smithi17557870-9:head [write 642232~661230] snapc f9=[] ondisk+write+known_if_redirected+supports_pool_eio e599)", 2024-09-11T03:49:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:46:56.982525+0000", 2024-09-11T03:49:09.786 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 132.78258482300001, 2024-09-11T03:49:09.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0088314529999999995, 2024-09-11T03:49:09.786 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.786 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.786 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:09.786 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:49:09.786 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:49:09.786 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:09.786 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5652 2024-09-11T03:49:09.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.982525+0000", 2024-09-11T03:49:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.982525+0000", 2024-09-11T03:49:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.982527+0000", 2024-09-11T03:49:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6390000000000001e-06 2024-09-11T03:49:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.983324+0000", 2024-09-11T03:49:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079795899999999999 2024-09-11T03:49:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.983341+0000", 2024-09-11T03:49:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033999999999999e-05 2024-09-11T03:49:09.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.983349+0000", 2024-09-11T03:49:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.619e-06 2024-09-11T03:49:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.983417+0000", 2024-09-11T03:49:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7756999999999995e-05 2024-09-11T03:49:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:49:09.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.983426+0000", 2024-09-11T03:49:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3030000000000005e-06 2024-09-11T03:49:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.986355+0000", 2024-09-11T03:49:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029289910000000001 2024-09-11T03:49:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.986479+0000", 2024-09-11T03:49:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123356 2024-09-11T03:49:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:49:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.986635+0000", 2024-09-11T03:49:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000156374 2024-09-11T03:49:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:49:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.989107+0000", 2024-09-11T03:49:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024717559999999999 2024-09-11T03:49:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.990053+0000", 2024-09-11T03:49:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00094608099999999998 2024-09-11T03:49:09.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.991248+0000", 2024-09-11T03:49:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001194961 2024-09-11T03:49:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.991300+0000", 2024-09-11T03:49:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2500999999999998e-05 2024-09-11T03:49:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:46:56.991356+0000", 2024-09-11T03:49:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6122000000000002e-05 2024-09-11T03:49:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5844 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 881] snapc 100=[100,ff] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-09-11T03:49:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:09.028558+0000", 2024-09-11T03:49:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 120.73655174699999, 2024-09-11T03:49:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020566675, 2024-09-11T03:49:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.793 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:49:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:49:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5844 2024-09-11T03:49:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028558+0000", 2024-09-11T03:49:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028558+0000", 2024-09-11T03:49:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028560+0000", 2024-09-11T03:49:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.001e-06 2024-09-11T03:49:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028570+0000", 2024-09-11T03:49:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7799999999999995e-06 2024-09-11T03:49:09.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028586+0000", 2024-09-11T03:49:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6047999999999999e-05 2024-09-11T03:49:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028596+0000", 2024-09-11T03:49:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.046e-05 2024-09-11T03:49:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028665+0000", 2024-09-11T03:49:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9152000000000005e-05 2024-09-11T03:49:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:49:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028670+0000", 2024-09-11T03:49:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6770000000000003e-06 2024-09-11T03:49:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.033863+0000", 2024-09-11T03:49:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051932799999999998 2024-09-11T03:49:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.033967+0000", 2024-09-11T03:49:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010344399999999999 2024-09-11T03:49:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,3", 2024-09-11T03:49:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.039468+0000", 2024-09-11T03:49:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055009330000000004 2024-09-11T03:49:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:49:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.043252+0000", 2024-09-11T03:49:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037844150000000002 2024-09-11T03:49:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.044699+0000", 2024-09-11T03:49:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0014469000000000001 2024-09-11T03:49:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.049058+0000", 2024-09-11T03:49:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043590399999999998 2024-09-11T03:49:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.049095+0000", 2024-09-11T03:49:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6745000000000001e-05 2024-09-11T03:49:09.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.049125+0000", 2024-09-11T03:49:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9799999999999999e-05 2024-09-11T03:49:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.800 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5845 2.19 2:98bb65f5:::smithi17557870-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e611)", 2024-09-11T03:49:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:09.028622+0000", 2024-09-11T03:49:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 120.736487341, 2024-09-11T03:49:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014191941, 2024-09-11T03:49:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:49:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:49:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:49:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5845 2024-09-11T03:49:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028622+0000", 2024-09-11T03:49:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028622+0000", 2024-09-11T03:49:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028623+0000", 2024-09-11T03:49:09.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6300000000000004e-07 2024-09-11T03:49:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028626+0000", 2024-09-11T03:49:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3270000000000001e-06 2024-09-11T03:49:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028643+0000", 2024-09-11T03:49:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.615e-05 2024-09-11T03:49:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028650+0000", 2024-09-11T03:49:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4970000000000004e-06 2024-09-11T03:49:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028730+0000", 2024-09-11T03:49:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9618999999999998e-05 2024-09-11T03:49:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:49:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028738+0000", 2024-09-11T03:49:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3569999999999998e-06 2024-09-11T03:49:09.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.034146+0000", 2024-09-11T03:49:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054079439999999996 2024-09-11T03:49:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:49:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.034193+0000", 2024-09-11T03:49:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7045000000000001e-05 2024-09-11T03:49:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.042662+0000", 2024-09-11T03:49:09.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084692180000000006 2024-09-11T03:49:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.042741+0000", 2024-09-11T03:49:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8749000000000001e-05 2024-09-11T03:49:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.042814+0000", 2024-09-11T03:49:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3271999999999999e-05 2024-09-11T03:49:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5976 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 600] snapc 106=[] ondisk+write+known_if_redirected+supports_pool_eio e625)", 2024-09-11T03:49:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:23.155338+0000", 2024-09-11T03:49:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 106.609771248, 2024-09-11T03:49:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017109479, 2024-09-11T03:49:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:49:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:49:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5976 2024-09-11T03:49:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155338+0000", 2024-09-11T03:49:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155338+0000", 2024-09-11T03:49:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155342+0000", 2024-09-11T03:49:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7230000000000002e-06 2024-09-11T03:49:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155357+0000", 2024-09-11T03:49:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4401e-05 2024-09-11T03:49:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155396+0000", 2024-09-11T03:49:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9301000000000002e-05 2024-09-11T03:49:09.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155408+0000", 2024-09-11T03:49:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2499e-05 2024-09-11T03:49:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155516+0000", 2024-09-11T03:49:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107948 2024-09-11T03:49:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155615+0000", 2024-09-11T03:49:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8627000000000003e-05 2024-09-11T03:49:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,3", 2024-09-11T03:49:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.159866+0000", 2024-09-11T03:49:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004250635 2024-09-11T03:49:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:49:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.163122+0000", 2024-09-11T03:49:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032564859999999998 2024-09-11T03:49:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.169373+0000", 2024-09-11T03:49:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062509230000000002 2024-09-11T03:49:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.172372+0000", 2024-09-11T03:49:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029991810000000001 2024-09-11T03:49:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.172414+0000", 2024-09-11T03:49:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1539999999999999e-05 2024-09-11T03:49:09.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.172448+0000", 2024-09-11T03:49:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4214999999999997e-05 2024-09-11T03:49:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:6102 2.7 2:e6da0655:::smithi17557870-27:head [write 618722~575837] snapc 10c=[10c] ondisk+write+known_if_redirected+supports_pool_eio e636)", 2024-09-11T03:49:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:34.013084+0000", 2024-09-11T03:49:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 95.752025595000006, 2024-09-11T03:49:09.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087560710000000007, 2024-09-11T03:49:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:49:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:49:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 6102 2024-09-11T03:49:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.013084+0000", 2024-09-11T03:49:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.013084+0000", 2024-09-11T03:49:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.013086+0000", 2024-09-11T03:49:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2989999999999999e-06 2024-09-11T03:49:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.013746+0000", 2024-09-11T03:49:09.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00065981199999999996 2024-09-11T03:49:09.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.013769+0000", 2024-09-11T03:49:09.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2322000000000001e-05 2024-09-11T03:49:09.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.013797+0000", 2024-09-11T03:49:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8935e-05 2024-09-11T03:49:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.013880+0000", 2024-09-11T03:49:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2131000000000004e-05 2024-09-11T03:49:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:49:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.013887+0000", 2024-09-11T03:49:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3239999999999997e-06 2024-09-11T03:49:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.017957+0000", 2024-09-11T03:49:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040699669999999999 2024-09-11T03:49:09.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.018042+0000", 2024-09-11T03:49:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4756e-05 2024-09-11T03:49:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:49:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.018165+0000", 2024-09-11T03:49:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012344100000000001 2024-09-11T03:49:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:49:09.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.021608+0000", 2024-09-11T03:49:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034432880000000001 2024-09-11T03:49:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.021674+0000", 2024-09-11T03:49:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5254999999999998e-05 2024-09-11T03:49:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.021716+0000", 2024-09-11T03:49:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2175000000000002e-05 2024-09-11T03:49:09.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.021780+0000", 2024-09-11T03:49:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4068999999999999e-05 2024-09-11T03:49:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:34.021840+0000", 2024-09-11T03:49:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0297000000000002e-05 2024-09-11T03:49:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:6215 2.16 e646/560 2:6dbd1e81:::smithi17557870-7:head v 646'431, mlcod=641'428)", 2024-09-11T03:49:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:43.999441+0000", 2024-09-11T03:49:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 85.765668829000006, 2024-09-11T03:49:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0081217670000000002, 2024-09-11T03:49:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.999441+0000", 2024-09-11T03:49:09.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.999441+0000", 2024-09-11T03:49:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.999441+0000", 2024-09-11T03:49:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6899999999999999e-07 2024-09-11T03:49:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.002211+0000", 2024-09-11T03:49:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027698340000000001 2024-09-11T03:49:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.002224+0000", 2024-09-11T03:49:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3108000000000001e-05 2024-09-11T03:49:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.002231+0000", 2024-09-11T03:49:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.401e-06 2024-09-11T03:49:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.002287+0000", 2024-09-11T03:49:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5717000000000002e-05 2024-09-11T03:49:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.002311+0000", 2024-09-11T03:49:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3913e-05 2024-09-11T03:49:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.007518+0000", 2024-09-11T03:49:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052071749999999997 2024-09-11T03:49:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.007563+0000", 2024-09-11T03:49:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4450000000000003e-05 2024-09-11T03:49:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.827 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:6935 2.19 2:98bb65f5:::smithi17557870-6:head [read 0~2403599,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e698)", 2024-09-11T03:49:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:36.088969+0000", 2024-09-11T03:49:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.676140771999997, 2024-09-11T03:49:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076953029999999997, 2024-09-11T03:49:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:49:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:49:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:49:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 6935 2024-09-11T03:49:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.088969+0000", 2024-09-11T03:49:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.088969+0000", 2024-09-11T03:49:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.088971+0000", 2024-09-11T03:49:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0990000000000001e-06 2024-09-11T03:49:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.088980+0000", 2024-09-11T03:49:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8140000000000007e-06 2024-09-11T03:49:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089000+0000", 2024-09-11T03:49:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9868999999999999e-05 2024-09-11T03:49:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089010+0000", 2024-09-11T03:49:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0628999999999999e-05 2024-09-11T03:49:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089087+0000", 2024-09-11T03:49:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6645999999999998e-05 2024-09-11T03:49:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:49:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089094+0000", 2024-09-11T03:49:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6889999999999998e-06 2024-09-11T03:49:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.095293+0000", 2024-09-11T03:49:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061992920000000003 2024-09-11T03:49:09.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.095388+0000", 2024-09-11T03:49:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5229000000000006e-05 2024-09-11T03:49:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.096664+0000", 2024-09-11T03:49:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012760359999999999 2024-09-11T03:49:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:6936 2.19 2:98bb65f5:::smithi17557870-6:head [sparse-read 0~1610900] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e698)", 2024-09-11T03:49:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:36.089039+0000", 2024-09-11T03:49:09.837 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.676071159000003, 2024-09-11T03:49:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078948560000000004, 2024-09-11T03:49:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:49:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:49:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:49:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 6936 2024-09-11T03:49:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089039+0000", 2024-09-11T03:49:09.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089039+0000", 2024-09-11T03:49:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089039+0000", 2024-09-11T03:49:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0999999999999998e-07 2024-09-11T03:49:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089045+0000", 2024-09-11T03:49:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6760000000000004e-06 2024-09-11T03:49:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089053+0000", 2024-09-11T03:49:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2889999999999998e-06 2024-09-11T03:49:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089060+0000", 2024-09-11T03:49:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9979999999999997e-06 2024-09-11T03:49:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089162+0000", 2024-09-11T03:49:09.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010210299999999999 2024-09-11T03:49:09.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:49:09.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089167+0000", 2024-09-11T03:49:09.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5809999999999998e-06 2024-09-11T03:49:09.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.096696+0000", 2024-09-11T03:49:09.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075288910000000002 2024-09-11T03:49:09.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.096785+0000", 2024-09-11T03:49:09.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9140000000000004e-05 2024-09-11T03:49:09.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.096933+0000", 2024-09-11T03:49:09.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014846799999999999 2024-09-11T03:49:09.843 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.843 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.843 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.843 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:6937 2.19 2:98bb65f5:::smithi17557870-6:head [read 0~2403599,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e698)", 2024-09-11T03:49:09.843 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:36.089071+0000", 2024-09-11T03:49:09.843 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.676039113999998, 2024-09-11T03:49:09.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086434530000000006, 2024-09-11T03:49:09.844 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.844 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.844 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:49:09.844 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:49:09.844 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:49:09.844 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:09.844 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 6937 2024-09-11T03:49:09.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.844 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089071+0000", 2024-09-11T03:49:09.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089071+0000", 2024-09-11T03:49:09.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089071+0000", 2024-09-11T03:49:09.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4300000000000003e-07 2024-09-11T03:49:09.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089075+0000", 2024-09-11T03:49:09.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7459999999999998e-06 2024-09-11T03:49:09.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089083+0000", 2024-09-11T03:49:09.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6939999999999995e-06 2024-09-11T03:49:09.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089088+0000", 2024-09-11T03:49:09.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4110000000000004e-06 2024-09-11T03:49:09.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089186+0000", 2024-09-11T03:49:09.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8213000000000003e-05 2024-09-11T03:49:09.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:49:09.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.089189+0000", 2024-09-11T03:49:09.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8660000000000002e-06 2024-09-11T03:49:09.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.096964+0000", 2024-09-11T03:49:09.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0077750529999999997 2024-09-11T03:49:09.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.097050+0000", 2024-09-11T03:49:09.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5457000000000005e-05 2024-09-11T03:49:09.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:36.097714+0000", 2024-09-11T03:49:09.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00066447000000000001 2024-09-11T03:49:09.849 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.849 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.849 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.849 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:49 2.7 2:e6da0655:::smithi17557870-27:head [assert-version v950,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e709)", 2024-09-11T03:49:09.849 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:47.009467+0000", 2024-09-11T03:49:09.849 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.755642258999998, 2024-09-11T03:49:09.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.009719732, 2024-09-11T03:49:09.849 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.849 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.849 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:49:09.849 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:47.009467+0000", 2024-09-11T03:49:09.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:47.009467+0000", 2024-09-11T03:49:09.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:47.009470+0000", 2024-09-11T03:49:09.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.345e-06 2024-09-11T03:49:09.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:47.009478+0000", 2024-09-11T03:49:09.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2789999999999998e-06 2024-09-11T03:49:09.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:47.009498+0000", 2024-09-11T03:49:09.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0230000000000001e-05 2024-09-11T03:49:09.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:47.009509+0000", 2024-09-11T03:49:09.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1037e-05 2024-09-11T03:49:09.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:47.009612+0000", 2024-09-11T03:49:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102981 2024-09-11T03:49:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:49:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:47.009620+0000", 2024-09-11T03:49:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6389999999999992e-06 2024-09-11T03:49:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:47.014725+0000", 2024-09-11T03:49:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051051910000000002 2024-09-11T03:49:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:47.014842+0000", 2024-09-11T03:49:09.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011638599999999999 2024-09-11T03:49:09.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:47.019187+0000", 2024-09-11T03:49:09.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043456440000000001 2024-09-11T03:49:09.854 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.854 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.854 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.855 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:7056 2.20 e710/574 2:0762776a:::smithi17557870-25:head v 710'215, mlcod=628'211)", 2024-09-11T03:49:09.855 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:48.029433+0000", 2024-09-11T03:49:09.855 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.735676721000001, 2024-09-11T03:49:09.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014420986, 2024-09-11T03:49:09.855 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.855 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.855 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:09.855 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.029433+0000", 2024-09-11T03:49:09.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.029433+0000", 2024-09-11T03:49:09.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.029433+0000", 2024-09-11T03:49:09.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2899999999999999e-07 2024-09-11T03:49:09.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032027+0000", 2024-09-11T03:49:09.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025937949999999999 2024-09-11T03:49:09.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032097+0000", 2024-09-11T03:49:09.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0153000000000004e-05 2024-09-11T03:49:09.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032132+0000", 2024-09-11T03:49:09.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4808000000000003e-05 2024-09-11T03:49:09.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032238+0000", 2024-09-11T03:49:09.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010623799999999999 2024-09-11T03:49:09.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032264+0000", 2024-09-11T03:49:09.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6026000000000001e-05 2024-09-11T03:49:09.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:09.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.043786+0000", 2024-09-11T03:49:09.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011521306 2024-09-11T03:49:09.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.043854+0000", 2024-09-11T03:49:09.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8330999999999993e-05 2024-09-11T03:49:09.859 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.859 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.859 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.859 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7068 2.7 2:e0eabcfb:::smithi17557870-9:head [write 1987803~365408] snapc 13a=[13a,139,138,137,136,135,133,12e,12d,12c,12a,127,126,123,120,11f,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e710)", 2024-09-11T03:49:09.859 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:48.033655+0000", 2024-09-11T03:49:09.859 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.731455167, 2024-09-11T03:49:09.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012180658, 2024-09-11T03:49:09.859 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.859 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.860 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:09.860 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:49:09.860 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:49:09.860 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:09.860 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7068 2024-09-11T03:49:09.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.860 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.033655+0000", 2024-09-11T03:49:09.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.033655+0000", 2024-09-11T03:49:09.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.033657+0000", 2024-09-11T03:49:09.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1619999999999998e-06 2024-09-11T03:49:09.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034007+0000", 2024-09-11T03:49:09.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035045600000000002 2024-09-11T03:49:09.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034023+0000", 2024-09-11T03:49:09.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5954999999999999e-05 2024-09-11T03:49:09.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034032+0000", 2024-09-11T03:49:09.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3729999999999999e-06 2024-09-11T03:49:09.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034090+0000", 2024-09-11T03:49:09.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7467999999999999e-05 2024-09-11T03:49:09.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034214+0000", 2024-09-11T03:49:09.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012389599999999999 2024-09-11T03:49:09.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:49:09.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034402+0000", 2024-09-11T03:49:09.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018850099999999999 2024-09-11T03:49:09.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.036271+0000", 2024-09-11T03:49:09.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00186853 2024-09-11T03:49:09.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.036536+0000", 2024-09-11T03:49:09.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00026485099999999998 2024-09-11T03:49:09.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:49:09.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.045741+0000", 2024-09-11T03:49:09.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092049769999999996 2024-09-11T03:49:09.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:09.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.045787+0000", 2024-09-11T03:49:09.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6594000000000001e-05 2024-09-11T03:49:09.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.045835+0000", 2024-09-11T03:49:09.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7895000000000001e-05 2024-09-11T03:49:09.865 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.865 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.865 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.866 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7069 2.7 2:e0eabcfb:::smithi17557870-9:head [setxattr _header (57) in=7b,truncate 2353211] snapc 13a=[13a,139,138,137,136,135,133,12e,12d,12c,12a,127,126,123,120,11f,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e710)", 2024-09-11T03:49:09.866 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:48.034054+0000", 2024-09-11T03:49:09.866 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.731055220999998, 2024-09-11T03:49:09.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011857098999999999, 2024-09-11T03:49:09.866 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.866 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.866 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:09.866 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:49:09.866 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:49:09.866 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:09.866 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7069 2024-09-11T03:49:09.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.866 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034054+0000", 2024-09-11T03:49:09.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034054+0000", 2024-09-11T03:49:09.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034055+0000", 2024-09-11T03:49:09.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2900000000000002e-07 2024-09-11T03:49:09.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034064+0000", 2024-09-11T03:49:09.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3070000000000003e-06 2024-09-11T03:49:09.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034078+0000", 2024-09-11T03:49:09.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4592e-05 2024-09-11T03:49:09.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034085+0000", 2024-09-11T03:49:09.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7710000000000001e-06 2024-09-11T03:49:09.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.035524+0000", 2024-09-11T03:49:09.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001439294 2024-09-11T03:49:09.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.035627+0000", 2024-09-11T03:49:09.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103065 2024-09-11T03:49:09.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:49:09.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.035763+0000", 2024-09-11T03:49:09.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000135838 2024-09-11T03:49:09.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.036570+0000", 2024-09-11T03:49:09.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00080645300000000005 2024-09-11T03:49:09.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.036603+0000", 2024-09-11T03:49:09.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2840999999999999e-05 2024-09-11T03:49:09.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:49:09.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.045845+0000", 2024-09-11T03:49:09.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092421840000000005 2024-09-11T03:49:09.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:09.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.045877+0000", 2024-09-11T03:49:09.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1789999999999999e-05 2024-09-11T03:49:09.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.045912+0000", 2024-09-11T03:49:09.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5034999999999999e-05 2024-09-11T03:49:09.872 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.872 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.872 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.872 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7070 2.7 2:e0eabcfb:::smithi17557870-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e710)", 2024-09-11T03:49:09.872 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:48.034097+0000", 2024-09-11T03:49:09.872 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.731012833000001, 2024-09-11T03:49:09.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012010824, 2024-09-11T03:49:09.872 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.872 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.873 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:49:09.873 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:49:09.873 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:49:09.873 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:09.873 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7070 2024-09-11T03:49:09.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.873 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034097+0000", 2024-09-11T03:49:09.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034097+0000", 2024-09-11T03:49:09.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034097+0000", 2024-09-11T03:49:09.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9800000000000003e-07 2024-09-11T03:49:09.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034100+0000", 2024-09-11T03:49:09.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.723e-06 2024-09-11T03:49:09.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034108+0000", 2024-09-11T03:49:09.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4370000000000001e-06 2024-09-11T03:49:09.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034113+0000", 2024-09-11T03:49:09.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6999999999999996e-06 2024-09-11T03:49:09.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.036309+0000", 2024-09-11T03:49:09.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002195584 2024-09-11T03:49:09.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:49:09.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.036353+0000", 2024-09-11T03:49:09.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4512999999999999e-05 2024-09-11T03:49:09.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.045950+0000", 2024-09-11T03:49:09.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095970680000000003 2024-09-11T03:49:09.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.046027+0000", 2024-09-11T03:49:09.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6831999999999999e-05 2024-09-11T03:49:09.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.046108+0000", 2024-09-11T03:49:09.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0369000000000003e-05 2024-09-11T03:49:09.877 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.877 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.877 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.877 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7071 2.7 2:e23d1762:::smithi17557870-21:head [copy-from ver 930] snapc 13a=[13a,139,138,137,136,135,133,12e,12d,12c,12a,127,126,123,120,11f,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e710)", 2024-09-11T03:49:09.877 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:48.034123+0000", 2024-09-11T03:49:09.878 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.730986711, 2024-09-11T03:49:09.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020390357000000001, 2024-09-11T03:49:09.878 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.878 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.878 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:09.878 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:49:09.878 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:49:09.878 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:09.878 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7071 2024-09-11T03:49:09.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.878 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034123+0000", 2024-09-11T03:49:09.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034123+0000", 2024-09-11T03:49:09.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034124+0000", 2024-09-11T03:49:09.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4600000000000004e-07 2024-09-11T03:49:09.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034127+0000", 2024-09-11T03:49:09.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0510000000000001e-06 2024-09-11T03:49:09.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034134+0000", 2024-09-11T03:49:09.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7749999999999993e-06 2024-09-11T03:49:09.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034140+0000", 2024-09-11T03:49:09.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1050000000000003e-06 2024-09-11T03:49:09.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.036380+0000", 2024-09-11T03:49:09.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022401639999999998 2024-09-11T03:49:09.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.036429+0000", 2024-09-11T03:49:09.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8918000000000002e-05 2024-09-11T03:49:09.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:49:09.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.043986+0000", 2024-09-11T03:49:09.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075575140000000004 2024-09-11T03:49:09.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:49:09.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.046488+0000", 2024-09-11T03:49:09.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025022640000000001 2024-09-11T03:49:09.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.052144+0000", 2024-09-11T03:49:09.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056556080000000003 2024-09-11T03:49:09.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.054409+0000", 2024-09-11T03:49:09.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002265254 2024-09-11T03:49:09.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:09.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.054471+0000", 2024-09-11T03:49:09.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1290000000000004e-05 2024-09-11T03:49:09.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.054513+0000", 2024-09-11T03:49:09.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2768000000000001e-05 2024-09-11T03:49:09.883 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.883 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.883 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.884 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7072 2.7 2:e23d1762:::smithi17557870-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e710)", 2024-09-11T03:49:09.884 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:48.034148+0000", 2024-09-11T03:49:09.884 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.730962084000002, 2024-09-11T03:49:09.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012072318, 2024-09-11T03:49:09.884 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.884 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.884 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:49:09.884 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:49:09.884 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:49:09.884 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:09.884 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7072 2024-09-11T03:49:09.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.885 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034148+0000", 2024-09-11T03:49:09.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034148+0000", 2024-09-11T03:49:09.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034148+0000", 2024-09-11T03:49:09.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.75e-07 2024-09-11T03:49:09.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034150+0000", 2024-09-11T03:49:09.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2060000000000001e-06 2024-09-11T03:49:09.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034158+0000", 2024-09-11T03:49:09.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7109999999999995e-06 2024-09-11T03:49:09.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034166+0000", 2024-09-11T03:49:09.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6290000000000001e-06 2024-09-11T03:49:09.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.036621+0000", 2024-09-11T03:49:09.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024548920000000002 2024-09-11T03:49:09.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:49:09.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.036646+0000", 2024-09-11T03:49:09.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5870000000000001e-05 2024-09-11T03:49:09.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.046126+0000", 2024-09-11T03:49:09.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0094794969999999999 2024-09-11T03:49:09.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.046172+0000", 2024-09-11T03:49:09.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5887999999999998e-05 2024-09-11T03:49:09.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.046220+0000", 2024-09-11T03:49:09.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8149999999999998e-05 2024-09-11T03:49:09.889 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.889 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.889 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.889 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7227 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 972] snapc 141=[141,140,13f,13d,13c,139,138,137,136,135,133,12e,12d,12a,127,126,120,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-09-11T03:49:09.889 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:01.067580+0000", 2024-09-11T03:49:09.889 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.6975293360000006, 2024-09-11T03:49:09.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028948451, 2024-09-11T03:49:09.889 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.889 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.889 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:09.889 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:49:09.890 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:49:09.890 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:09.890 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7227 2024-09-11T03:49:09.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.890 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067580+0000", 2024-09-11T03:49:09.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067580+0000", 2024-09-11T03:49:09.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067582+0000", 2024-09-11T03:49:09.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.951e-06 2024-09-11T03:49:09.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067594+0000", 2024-09-11T03:49:09.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1272e-05 2024-09-11T03:49:09.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067610+0000", 2024-09-11T03:49:09.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6889999999999999e-05 2024-09-11T03:49:09.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067623+0000", 2024-09-11T03:49:09.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2680999999999999e-05 2024-09-11T03:49:09.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067684+0000", 2024-09-11T03:49:09.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1005999999999998e-05 2024-09-11T03:49:09.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:49:09.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067692+0000", 2024-09-11T03:49:09.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4189999999999996e-06 2024-09-11T03:49:09.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.069575+0000", 2024-09-11T03:49:09.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001882959 2024-09-11T03:49:09.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.069682+0000", 2024-09-11T03:49:09.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107416 2024-09-11T03:49:09.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,3", 2024-09-11T03:49:09.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.089088+0000", 2024-09-11T03:49:09.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019406102000000001 2024-09-11T03:49:09.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:49:09.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.092329+0000", 2024-09-11T03:49:09.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032404589999999998 2024-09-11T03:49:09.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.096291+0000", 2024-09-11T03:49:09.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039621470000000001 2024-09-11T03:49:09.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:09.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.096455+0000", 2024-09-11T03:49:09.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016462099999999999 2024-09-11T03:49:09.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:49:09.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.096494+0000", 2024-09-11T03:49:09.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8705000000000003e-05 2024-09-11T03:49:09.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.096529+0000", 2024-09-11T03:49:09.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4823000000000002e-05 2024-09-11T03:49:09.897 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.897 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.897 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.897 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7228 2.19 2:98bb65f5:::smithi17557870-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e723)", 2024-09-11T03:49:09.897 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:01.067641+0000", 2024-09-11T03:49:09.897 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.6974685540000003, 2024-09-11T03:49:09.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.024383425, 2024-09-11T03:49:09.898 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:49:09.898 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:49:09.898 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:49:09.898 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:49:09.898 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:49:09.898 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:09.898 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7228 2024-09-11T03:49:09.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.898 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:49:09.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:49:09.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067641+0000", 2024-09-11T03:49:09.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:49:09.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067641+0000", 2024-09-11T03:49:09.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:49:09.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:49:09.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067641+0000", 2024-09-11T03:49:09.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3700000000000001e-07 2024-09-11T03:49:09.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:49:09.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067643+0000", 2024-09-11T03:49:09.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7579999999999999e-06 2024-09-11T03:49:09.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:49:09.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067651+0000", 2024-09-11T03:49:09.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2830000000000004e-06 2024-09-11T03:49:09.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:49:09.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067656+0000", 2024-09-11T03:49:09.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2009999999999998e-06 2024-09-11T03:49:09.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067757+0000", 2024-09-11T03:49:09.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000100888 2024-09-11T03:49:09.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:49:09.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067762+0000", 2024-09-11T03:49:09.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3859999999999998e-06 2024-09-11T03:49:09.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.069849+0000", 2024-09-11T03:49:09.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020867500000000001 2024-09-11T03:49:09.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:49:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.069896+0000", 2024-09-11T03:49:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859e-05 2024-09-11T03:49:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:49:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.091800+0000", 2024-09-11T03:49:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021904534999999999 2024-09-11T03:49:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:49:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.091916+0000", 2024-09-11T03:49:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116092 2024-09-11T03:49:09.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:49:09.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:49:09.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:49:09.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.092025+0000", 2024-09-11T03:49:09.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108336 2024-09-11T03:49:09.903 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.903 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.903 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.903 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:49:09.903 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:49:09.903 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:49:09.924 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:09.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:09.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:10.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:10.047 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:49:10.051 INFO:tasks.rados.rados.2.smithi175.stdout:2929: done (0 left) 2024-09-11T03:49:10.051 INFO:tasks.rados.rados.2.smithi175.stdout:2930: snap_remove snap 292 2024-09-11T03:49:10.051 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:10.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:10.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:10.153 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:10.175 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:10.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:10.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:10.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:10.299 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:10.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:10.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:10.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:10 smithi084 ceph-mon[27254]: pgmap v1251: 67 pgs: 67 active+clean; 360 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 661 KiB/s wr, 9 op/s 2024-09-11T03:49:10.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:10 smithi084 ceph-mon[27254]: osdmap e731: 12 total, 12 up, 12 in 2024-09-11T03:49:10.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:10 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:49:10.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:10 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:49:10.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:10 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:49:10.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:10 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1434631005' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:49:10.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:10 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:49:10.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:10 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:49:10.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:10 smithi175 ceph-mon[30894]: pgmap v1251: 67 pgs: 67 active+clean; 360 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 661 KiB/s wr, 9 op/s 2024-09-11T03:49:10.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:10 smithi175 ceph-mon[30894]: osdmap e731: 12 total, 12 up, 12 in 2024-09-11T03:49:10.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:10 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:49:10.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:10 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:49:10.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:10 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:49:10.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:10 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1434631005' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:49:10.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:10 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:49:10.379 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:10 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:49:10.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:10 smithi121 ceph-mon[30729]: pgmap v1251: 67 pgs: 67 active+clean; 360 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 661 KiB/s wr, 9 op/s 2024-09-11T03:49:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:10 smithi121 ceph-mon[30729]: osdmap e731: 12 total, 12 up, 12 in 2024-09-11T03:49:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:49:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:49:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:10 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:49:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1434631005' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:49:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:10 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:49:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:10 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:49:10.423 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:10.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:10.525 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:10.552 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:10.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:10.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:10.654 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:10.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T03:49:10.676 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:10.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:10.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:10.777 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:10.799 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:10.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:10.800 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:10.901 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:10.901 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:49:10.901 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:49:10.902 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-11T03:49:10.902 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T03:49:10.902 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T03:49:10.902 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-11T03:49:10.911 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:49:10.924 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:10.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:10.972 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:49:11.026 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:11.048 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:11.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:11.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:11.061 INFO:tasks.rados.rados.2.smithi175.stdout:2930: done (0 left) 2024-09-11T03:49:11.061 INFO:tasks.rados.rados.2.smithi175.stdout:2931: write (excl) oid 43 current snap is 326 2024-09-11T03:49:11.061 INFO:tasks.rados.rados.2.smithi175.stdout:2931: seq_num 1008 ranges {541456=409588,1569261=562350,2814612=126844} 2024-09-11T03:49:11.063 INFO:tasks.rados.rados.2.smithi175.stdout:2931: writing smithi17557870-43 from 541456 to 951044 tid 1 2024-09-11T03:49:11.071 INFO:tasks.rados.rados.2.smithi175.stdout:2931: writing smithi17557870-43 from 1569261 to 2131611 tid 2 2024-09-11T03:49:11.076 INFO:tasks.rados.rados.2.smithi175.stdout:2931: writing smithi17557870-43 from 2814612 to 2941456 tid 3 2024-09-11T03:49:11.076 INFO:tasks.rados.rados.2.smithi175.stdout:2932: write (excl) oid 4 current snap is 326 2024-09-11T03:49:11.076 INFO:tasks.rados.rados.2.smithi175.stdout:2932: seq_num 1009 ranges {558263=484837,1627743=706008} 2024-09-11T03:49:11.081 INFO:tasks.rados.rados.2.smithi175.stdout:2932: writing smithi17557870-4 from 558263 to 1043100 tid 1 2024-09-11T03:49:11.089 INFO:tasks.rados.rados.2.smithi175.stdout:2932: writing smithi17557870-4 from 1627743 to 2333751 tid 2 2024-09-11T03:49:11.090 INFO:tasks.rados.rados.2.smithi175.stdout:2933: read oid 44 snap -1 2024-09-11T03:49:11.090 INFO:tasks.rados.rados.2.smithi175.stdout:2933: expect (ObjNum 939 snap 297 seq_num 939) 2024-09-11T03:49:11.090 INFO:tasks.rados.rados.2.smithi175.stdout:2934: snap_remove snap 318 2024-09-11T03:49:11.151 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:11.163 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:11.172 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:11.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:11.173 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:11.274 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:11.287 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:11 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:11.287 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:11 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1434631005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:49:11.287 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:11 smithi084 ceph-mon[27254]: osdmap e732: 12 total, 12 up, 12 in 2024-09-11T03:49:11.295 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:11.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:11.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:11.337 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:11 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:11.337 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:11 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1434631005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:49:11.338 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:11 smithi121 ceph-mon[30729]: osdmap e732: 12 total, 12 up, 12 in 2024-09-11T03:49:11.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:11.419 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:11.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:11.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:11.457 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:49:11.457 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:49:11.457 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:49:11.457 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:49:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:11 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:11 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1434631005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:49:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:11 smithi175 ceph-mon[30894]: osdmap e732: 12 total, 12 up, 12 in 2024-09-11T03:49:11.521 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:11.543 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:11.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:11.544 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:11.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:11.652 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:49:11.652 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "" 2024-09-11T03:49:11.652 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:49:11.659 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:11.667 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:11.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:11.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:11.769 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:11.791 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:11.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:11.792 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:11.893 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:11.914 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:11.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:11.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:12.012 INFO:tasks.rados.rados.2.smithi175.stdout:2934: done (3 left) 2024-09-11T03:49:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:2935: snap_create 2024-09-11T03:49:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:2931: finishing write tid 1 to smithi17557870-43 2024-09-11T03:49:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:2931: finishing write tid 2 to smithi17557870-43 2024-09-11T03:49:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:2931: finishing write tid 3 to smithi17557870-43 2024-09-11T03:49:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:2931: finishing write tid 4 to smithi17557870-43 2024-09-11T03:49:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:2931: finishing write tid 5 to smithi17557870-43 2024-09-11T03:49:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:2931: oid 43 updating version 0 to 1009 2024-09-11T03:49:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:2931: oid 43 version 1009 is already newer than 1008 2024-09-11T03:49:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:2931: oid 43 version 1009 is already newer than 1007 2024-09-11T03:49:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:2931: oid 43 updating version 1009 to 1010 2024-09-11T03:49:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 1010 (ObjNum 1008 snap 326 seq_num 1008) dirty exists 2024-09-11T03:49:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:2931: left oid 43 (ObjNum 1008 snap 326 seq_num 1008) 2024-09-11T03:49:12.013 INFO:tasks.rados.rados.2.smithi175.stdout:2932: finishing write tid 1 to smithi17557870-4 2024-09-11T03:49:12.014 INFO:tasks.rados.rados.2.smithi175.stdout:2932: finishing write tid 2 to smithi17557870-4 2024-09-11T03:49:12.014 INFO:tasks.rados.rados.2.smithi175.stdout:2932: finishing write tid 3 to smithi17557870-4 2024-09-11T03:49:12.014 INFO:tasks.rados.rados.2.smithi175.stdout:2932: finishing write tid 4 to smithi17557870-4 2024-09-11T03:49:12.014 INFO:tasks.rados.rados.2.smithi175.stdout:2932: oid 4 updating version 0 to 963 2024-09-11T03:49:12.014 INFO:tasks.rados.rados.2.smithi175.stdout:2932: oid 4 updating version 963 to 964 2024-09-11T03:49:12.014 INFO:tasks.rados.rados.2.smithi175.stdout:2932: oid 4 updating version 964 to 965 2024-09-11T03:49:12.014 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 965 (ObjNum 1009 snap 326 seq_num 1009) dirty exists 2024-09-11T03:49:12.014 INFO:tasks.rados.rados.2.smithi175.stdout:2932: left oid 4 (ObjNum 1009 snap 326 seq_num 1009) 2024-09-11T03:49:12.014 INFO:tasks.rados.rados.2.smithi175.stdout:2933: expect (ObjNum 939 snap 297 seq_num 939) 2024-09-11T03:49:12.016 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:12.038 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:12.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:12.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:12.113 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T03:49:12.140 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:12.161 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:12.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:12.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:12.263 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:12.284 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:12.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:12.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:12.299 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:12 smithi084 ceph-mon[27254]: pgmap v1254: 67 pgs: 67 active+clean; 360 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 661 KiB/s wr, 5 op/s 2024-09-11T03:49:12.299 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:12 smithi084 ceph-mon[27254]: osdmap e733: 12 total, 12 up, 12 in 2024-09-11T03:49:12.299 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:12 smithi084 ceph-mon[27254]: osdmap e734: 12 total, 12 up, 12 in 2024-09-11T03:49:12.351 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:49:12.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:12.406 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:12.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:12.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:12.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:12 smithi121 ceph-mon[30729]: pgmap v1254: 67 pgs: 67 active+clean; 360 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 661 KiB/s wr, 5 op/s 2024-09-11T03:49:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:12 smithi121 ceph-mon[30729]: osdmap e733: 12 total, 12 up, 12 in 2024-09-11T03:49:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:12 smithi121 ceph-mon[30729]: osdmap e734: 12 total, 12 up, 12 in 2024-09-11T03:49:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:12 smithi175 ceph-mon[30894]: pgmap v1254: 67 pgs: 67 active+clean; 360 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 661 KiB/s wr, 5 op/s 2024-09-11T03:49:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:12 smithi175 ceph-mon[30894]: osdmap e733: 12 total, 12 up, 12 in 2024-09-11T03:49:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:12 smithi175 ceph-mon[30894]: osdmap e734: 12 total, 12 up, 12 in 2024-09-11T03:49:12.508 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:12.530 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:12.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:12.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:12.632 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:12.654 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:12.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:12.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:12.720 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:49:12.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:12.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:12.779 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:12.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:12.810 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:49:12.811 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:49:12.827 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:49:12.827 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:49:12.840 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:49:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:49:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:49:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:49:12.840 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:49:12.843 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:49:12.843 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:49:12.859 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:49:12.859 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:49:12.875 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:49:12.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:49:12.880 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:12.893 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:49:12.893 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:49:12.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:12.905 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:12.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:12.910 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:49:12.910 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:49:12.934 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:49:12.935 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:49:12.955 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:49:12.956 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:49:12.973 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:49:12.974 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:49:12.978 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:12.993 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:49:12.993 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:49:13.006 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:13.015 INFO:tasks.rados.rados.2.smithi175.stdout:2931: done (3 left) 2024-09-11T03:49:13.015 INFO:tasks.rados.rados.2.smithi175.stdout:2932: done (2 left) 2024-09-11T03:49:13.016 INFO:tasks.rados.rados.2.smithi175.stdout:2933: done (1 left) 2024-09-11T03:49:13.016 INFO:tasks.rados.rados.2.smithi175.stdout:2935: done (0 left) 2024-09-11T03:49:13.016 INFO:tasks.rados.rados.2.smithi175.stdout:2936: read oid 41 snap -1 2024-09-11T03:49:13.016 INFO:tasks.rados.rados.2.smithi175.stdout:2936: expect deleted 2024-09-11T03:49:13.016 INFO:tasks.rados.rados.2.smithi175.stdout:2937: snap_remove snap 317 2024-09-11T03:49:13.016 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:49:13.016 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:49:13.030 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:13.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:13.031 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:13.099 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:13 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4179991657' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:13.100 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:13 smithi084 ceph-mon[27254]: osdmap e735: 12 total, 12 up, 12 in 2024-09-11T03:49:13.131 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:13.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:13.154 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:13.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:13.256 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:13.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:13.280 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:13.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:13.381 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:13.407 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:13.408 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:13.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:13 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4179991657' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:13 smithi121 ceph-mon[30729]: osdmap e735: 12 total, 12 up, 12 in 2024-09-11T03:49:13.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T03:49:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:13 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4179991657' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:13 smithi175 ceph-mon[30894]: osdmap e735: 12 total, 12 up, 12 in 2024-09-11T03:49:13.509 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:13.534 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:13.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:13.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:13.637 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:13.662 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:13.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:13.663 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:13.695 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:49:13.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:13.784 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:13.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:13.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:13.886 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:13.912 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:13.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:13.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:14.014 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:14.017 INFO:tasks.rados.rados.2.smithi175.stdout:2937: done (1 left) 2024-09-11T03:49:14.017 INFO:tasks.rados.rados.2.smithi175.stdout:2938: snap_create 2024-09-11T03:49:14.040 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:14.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:14.041 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:14.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:14.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:14 smithi084 ceph-mon[27254]: pgmap v1257: 67 pgs: 67 active+clean; 373 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-11T03:49:14.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:14 smithi084 ceph-mon[27254]: osdmap e736: 12 total, 12 up, 12 in 2024-09-11T03:49:14.163 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:14.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:14.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:14.165 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:49:14.165 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:49:14.265 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:14.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:14.287 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:14.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:14.388 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:14.409 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:14.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:14.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:14.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:14 smithi121 ceph-mon[30729]: pgmap v1257: 67 pgs: 67 active+clean; 373 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-11T03:49:14.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:14 smithi121 ceph-mon[30729]: osdmap e736: 12 total, 12 up, 12 in 2024-09-11T03:49:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:14 smithi175 ceph-mon[30894]: pgmap v1257: 67 pgs: 67 active+clean; 373 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-11T03:49:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:14 smithi175 ceph-mon[30894]: osdmap e736: 12 total, 12 up, 12 in 2024-09-11T03:49:14.511 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:14.533 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:14.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:14.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:14.635 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:14.656 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:14.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:14.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:14.758 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:14.779 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:14.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:14.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:14.878 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-11T03:49:14.881 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:14.905 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:14.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:14.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:15.007 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:15.012 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:49:15.016 INFO:tasks.rados.rados.2.smithi175.stdout:2936: done (1 left) 2024-09-11T03:49:15.016 INFO:tasks.rados.rados.2.smithi175.stdout:2938: done (0 left) 2024-09-11T03:49:15.016 INFO:tasks.rados.rados.2.smithi175.stdout:2939: snap_create 2024-09-11T03:49:15.032 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:15.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:15.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:15.134 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:15.135 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:49:15.156 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:15.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:15.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:15.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:15 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2907111823' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:49:15.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:15 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2907111823' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:49:15.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:15 smithi121 ceph-mon[30729]: osdmap e737: 12 total, 12 up, 12 in 2024-09-11T03:49:15.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:15 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2907111823' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:49:15.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:15 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2907111823' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:49:15.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:15 smithi175 ceph-mon[30894]: osdmap e737: 12 total, 12 up, 12 in 2024-09-11T03:49:15.258 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:15.280 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:15.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:15.281 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:15.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:15 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2907111823' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:49:15.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:15 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2907111823' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:49:15.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:15 smithi084 ceph-mon[27254]: osdmap e737: 12 total, 12 up, 12 in 2024-09-11T03:49:15.381 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:15.403 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:15.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:15.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:15.528 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:15.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:15.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:15.630 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:15.652 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:15.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:15.653 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:15.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:15.778 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:15.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:15.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:15.792 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:49:15.792 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:49:15.880 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:15.903 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:15.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:15.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:15.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:49:16.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:16.020 INFO:tasks.rados.rados.2.smithi175.stdout:2939: done (0 left) 2024-09-11T03:49:16.020 INFO:tasks.rados.rados.2.smithi175.stdout:2940: copy_from oid 47 from oid 18 current snap is 329 2024-09-11T03:49:16.020 INFO:tasks.rados.rados.2.smithi175.stdout:2941: snap_create 2024-09-11T03:49:16.026 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:16.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:16.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:16.033 INFO:tasks.rados.rados.2.smithi175.stdout:2940: finishing copy_from racing read to smithi17557870-47 2024-09-11T03:49:16.034 INFO:tasks.rados.rados.2.smithi175.stdout:2940: finishing copy_from to smithi17557870-47 2024-09-11T03:49:16.035 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 488 (ObjNum 917 snap 294 seq_num 917) dirty exists 2024-09-11T03:49:16.128 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:16.151 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:16.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:16.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:16.193 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:16.253 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:16.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:16.275 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:16.276 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:16.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:16 smithi084 ceph-mon[27254]: pgmap v1260: 67 pgs: 67 active+clean; 373 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-11T03:49:16.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:16 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:49:16.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:16 smithi084 ceph-mon[27254]: 2.0 scrub ok 2024-09-11T03:49:16.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:16 smithi084 ceph-mon[27254]: 2.13 scrub starts 2024-09-11T03:49:16.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:16 smithi084 ceph-mon[27254]: 2.13 scrub ok 2024-09-11T03:49:16.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:16 smithi084 ceph-mon[27254]: osdmap e738: 12 total, 12 up, 12 in 2024-09-11T03:49:16.376 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:16.398 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:16.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:16.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:16 smithi121 ceph-mon[30729]: pgmap v1260: 67 pgs: 67 active+clean; 373 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-11T03:49:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:16 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:49:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:16 smithi121 ceph-mon[30729]: 2.0 scrub ok 2024-09-11T03:49:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:16 smithi121 ceph-mon[30729]: 2.13 scrub starts 2024-09-11T03:49:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:16 smithi121 ceph-mon[30729]: 2.13 scrub ok 2024-09-11T03:49:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:16 smithi121 ceph-mon[30729]: osdmap e738: 12 total, 12 up, 12 in 2024-09-11T03:49:16.431 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:16 smithi175 ceph-mon[30894]: pgmap v1260: 67 pgs: 67 active+clean; 373 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-09-11T03:49:16.431 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:16 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:49:16.431 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:16 smithi175 ceph-mon[30894]: 2.0 scrub ok 2024-09-11T03:49:16.431 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:16 smithi175 ceph-mon[30894]: 2.13 scrub starts 2024-09-11T03:49:16.431 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:16 smithi175 ceph-mon[30894]: 2.13 scrub ok 2024-09-11T03:49:16.431 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:16 smithi175 ceph-mon[30894]: osdmap e738: 12 total, 12 up, 12 in 2024-09-11T03:49:16.457 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-11T03:49:16.499 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:16.526 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:16.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:16.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:16.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:16.654 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:16.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:16.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:16.671 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:49:16.672 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:49:16.688 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:49:16.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:49:16.704 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:49:16.704 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:49:16.720 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:49:16.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:49:16.737 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:49:16.737 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:49:16.739 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:49:16.754 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:49:16.754 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:49:16.756 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:16.772 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:49:16.772 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:49:16.777 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:16.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:16.778 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:16.788 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:49:16.789 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:49:16.805 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:49:16.806 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:49:16.822 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:49:16.823 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:49:16.839 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:49:16.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:49:16.857 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:49:16.857 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:49:16.879 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:16.905 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:16.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:16.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:17.007 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:17.013 INFO:tasks.rados.rados.2.smithi175.stdout:2940: done (1 left) 2024-09-11T03:49:17.013 INFO:tasks.rados.rados.2.smithi175.stdout:2941: done (0 left) 2024-09-11T03:49:17.013 INFO:tasks.rados.rados.2.smithi175.stdout:2942: delete oid 37 current snap is 330 2024-09-11T03:49:17.017 INFO:tasks.rados.rados.2.smithi175.stdout:2942: done (0 left) 2024-09-11T03:49:17.017 INFO:tasks.rados.rados.2.smithi175.stdout:2943: delete oid 2 current snap is 330 2024-09-11T03:49:17.020 INFO:tasks.rados.rados.2.smithi175.stdout:2943: done (0 left) 2024-09-11T03:49:17.020 INFO:tasks.rados.rados.2.smithi175.stdout:2944: write (excl) oid 18 current snap is 330 2024-09-11T03:49:17.020 INFO:tasks.rados.rados.2.smithi175.stdout:2944: seq_num 1012 ranges {608684=426937,1803189=569688,2924304=84380} 2024-09-11T03:49:17.025 INFO:tasks.rados.rados.2.smithi175.stdout:2944: writing smithi17557870-18 from 608684 to 1035621 tid 1 2024-09-11T03:49:17.030 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:17.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:17.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:17.034 INFO:tasks.rados.rados.2.smithi175.stdout:2944: writing smithi17557870-18 from 1803189 to 2372877 tid 2 2024-09-11T03:49:17.038 INFO:tasks.rados.rados.2.smithi175.stdout:2944: writing smithi17557870-18 from 2924304 to 3008684 tid 3 2024-09-11T03:49:17.038 INFO:tasks.rados.rados.2.smithi175.stdout:2945: read oid 7 snap -1 2024-09-11T03:49:17.038 INFO:tasks.rados.rados.2.smithi175.stdout:2945: expect (ObjNum 925 snap 294 seq_num 925) 2024-09-11T03:49:17.038 INFO:tasks.rados.rados.2.smithi175.stdout:2946: snap_create 2024-09-11T03:49:17.038 INFO:tasks.rados.rados.2.smithi175.stdout:2944: finishing write tid 1 to smithi17557870-18 2024-09-11T03:49:17.038 INFO:tasks.rados.rados.2.smithi175.stdout:2944: finishing write tid 2 to smithi17557870-18 2024-09-11T03:49:17.040 INFO:tasks.rados.rados.2.smithi175.stdout:2944: finishing write tid 3 to smithi17557870-18 2024-09-11T03:49:17.040 INFO:tasks.rados.rados.2.smithi175.stdout:2944: finishing write tid 4 to smithi17557870-18 2024-09-11T03:49:17.040 INFO:tasks.rados.rados.2.smithi175.stdout:2944: finishing write tid 5 to smithi17557870-18 2024-09-11T03:49:17.040 INFO:tasks.rados.rados.2.smithi175.stdout:2944: oid 18 updating version 0 to 491 2024-09-11T03:49:17.040 INFO:tasks.rados.rados.2.smithi175.stdout:2944: oid 18 updating version 491 to 492 2024-09-11T03:49:17.040 INFO:tasks.rados.rados.2.smithi175.stdout:2944: oid 18 version 492 is already newer than 489 2024-09-11T03:49:17.040 INFO:tasks.rados.rados.2.smithi175.stdout:2944: oid 18 version 492 is already newer than 490 2024-09-11T03:49:17.040 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 492 (ObjNum 1012 snap 330 seq_num 1012) dirty exists 2024-09-11T03:49:17.040 INFO:tasks.rados.rados.2.smithi175.stdout:2944: left oid 18 (ObjNum 1012 snap 330 seq_num 1012) 2024-09-11T03:49:17.051 INFO:tasks.rados.rados.2.smithi175.stdout:2945: expect (ObjNum 925 snap 294 seq_num 925) 2024-09-11T03:49:17.132 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:17.154 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:17.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:17.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:17.184 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:17 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:49:17.184 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:17 smithi175 ceph-mon[30894]: 2.2 scrub ok 2024-09-11T03:49:17.184 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1267233618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:17.185 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:17 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:17.185 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:17 smithi175 ceph-mon[30894]: osdmap e739: 12 total, 12 up, 12 in 2024-09-11T03:49:17.256 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:17.281 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:17.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:17.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:17.284 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:49:17.284 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:49:17.284 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:49:17.284 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:49:17.284 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:49:17.325 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:49:17.325 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:49:17.326 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-11T03:49:17.326 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:49:17.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:17 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:49:17.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:17 smithi084 ceph-mon[27254]: 2.2 scrub ok 2024-09-11T03:49:17.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1267233618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:17.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:17 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:17.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:17 smithi084 ceph-mon[27254]: osdmap e739: 12 total, 12 up, 12 in 2024-09-11T03:49:17.383 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:17.404 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:17.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:17.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:17.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:17 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:49:17.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:17 smithi121 ceph-mon[30729]: 2.2 scrub ok 2024-09-11T03:49:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1267233618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:17 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:17 smithi121 ceph-mon[30729]: osdmap e739: 12 total, 12 up, 12 in 2024-09-11T03:49:17.509 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:17.536 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:17.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:17.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:17.638 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:17.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:17.662 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:17.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:17.701 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:17.763 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:17.781 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:49:17.784 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:17.786 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:17.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:17.912 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:17.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:18.014 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:18.017 INFO:tasks.rados.rados.2.smithi175.stdout:2944: done (2 left) 2024-09-11T03:49:18.017 INFO:tasks.rados.rados.2.smithi175.stdout:2945: done (1 left) 2024-09-11T03:49:18.017 INFO:tasks.rados.rados.2.smithi175.stdout:2946: done (0 left) 2024-09-11T03:49:18.017 INFO:tasks.rados.rados.2.smithi175.stdout:2947: read oid 46 snap 311 2024-09-11T03:49:18.017 INFO:tasks.rados.rados.2.smithi175.stdout:2947: expect deleted 2024-09-11T03:49:18.017 INFO:tasks.rados.rados.2.smithi175.stdout:2948: read oid 47 snap -1 2024-09-11T03:49:18.017 INFO:tasks.rados.rados.2.smithi175.stdout:2948: expect (ObjNum 917 snap 294 seq_num 917) 2024-09-11T03:49:18.017 INFO:tasks.rados.rados.2.smithi175.stdout:2949: read oid 17 snap -1 2024-09-11T03:49:18.018 INFO:tasks.rados.rados.2.smithi175.stdout:2949: expect (ObjNum 993 snap 320 seq_num 993) 2024-09-11T03:49:18.018 INFO:tasks.rados.rados.2.smithi175.stdout:2950: snap_remove snap 330 2024-09-11T03:49:18.036 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:18.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:18.037 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:18.138 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:18.160 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:18.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:18.161 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:18.168 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-11T03:49:18.220 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:18.262 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:18.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:18.284 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:18.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:18.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:18 smithi084 ceph-mon[27254]: 2.4 scrub starts 2024-09-11T03:49:18.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:18 smithi084 ceph-mon[27254]: 2.4 scrub ok 2024-09-11T03:49:18.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:18 smithi084 ceph-mon[27254]: pgmap v1263: 67 pgs: 67 active+clean; 374 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.1 MiB/s wr, 1 op/s 2024-09-11T03:49:18.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:18 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:49:18.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:18 smithi084 ceph-mon[27254]: 2.15 scrub ok 2024-09-11T03:49:18.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:18 smithi084 ceph-mon[27254]: 1.2 scrub starts 2024-09-11T03:49:18.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:18 smithi084 ceph-mon[27254]: 1.2 scrub ok 2024-09-11T03:49:18.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:18 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:49:18.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:18 smithi084 ceph-mon[27254]: 2.c scrub ok 2024-09-11T03:49:18.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:18 smithi084 ceph-mon[27254]: osdmap e740: 12 total, 12 up, 12 in 2024-09-11T03:49:18.364 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:49:18.365 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:49:18.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:18.411 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:18.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:18.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:18 smithi121 ceph-mon[30729]: 2.4 scrub starts 2024-09-11T03:49:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:18 smithi121 ceph-mon[30729]: 2.4 scrub ok 2024-09-11T03:49:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:18 smithi121 ceph-mon[30729]: pgmap v1263: 67 pgs: 67 active+clean; 374 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.1 MiB/s wr, 1 op/s 2024-09-11T03:49:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:18 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:49:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:18 smithi121 ceph-mon[30729]: 2.15 scrub ok 2024-09-11T03:49:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:18 smithi121 ceph-mon[30729]: 1.2 scrub starts 2024-09-11T03:49:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:18 smithi121 ceph-mon[30729]: 1.2 scrub ok 2024-09-11T03:49:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:18 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:49:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:18 smithi121 ceph-mon[30729]: 2.c scrub ok 2024-09-11T03:49:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:18 smithi121 ceph-mon[30729]: osdmap e740: 12 total, 12 up, 12 in 2024-09-11T03:49:18.451 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:49:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:18 smithi175 ceph-mon[30894]: 2.4 scrub starts 2024-09-11T03:49:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:18 smithi175 ceph-mon[30894]: 2.4 scrub ok 2024-09-11T03:49:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:18 smithi175 ceph-mon[30894]: pgmap v1263: 67 pgs: 67 active+clean; 374 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.1 MiB/s wr, 1 op/s 2024-09-11T03:49:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:18 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:49:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:18 smithi175 ceph-mon[30894]: 2.15 scrub ok 2024-09-11T03:49:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:18 smithi175 ceph-mon[30894]: 1.2 scrub starts 2024-09-11T03:49:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:18 smithi175 ceph-mon[30894]: 1.2 scrub ok 2024-09-11T03:49:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:18 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:49:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:18 smithi175 ceph-mon[30894]: 2.c scrub ok 2024-09-11T03:49:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:18 smithi175 ceph-mon[30894]: osdmap e740: 12 total, 12 up, 12 in 2024-09-11T03:49:18.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:18.539 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:18.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:18.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:18.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:18.666 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:18.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:18.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:18.768 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:18.790 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:18.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:18.892 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:18.917 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:18.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:18.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:18.932 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:49:18.933 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:49:18.933 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:49:18.933 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:49:18.933 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.933 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1152 2.b 2:de6b1316:::smithi17557870-32:head [copy-from ver 77] snapc 2d=[2d,2c,2b,28,26] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-11T03:49:18.933 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:39:59.869443+0000", 2024-09-11T03:49:18.933 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 559.06237312799999, 2024-09-11T03:49:18.933 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015544123999999999, 2024-09-11T03:49:18.933 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:18.933 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:18.933 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1152 2024-09-11T03:49:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869443+0000", 2024-09-11T03:49:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869443+0000", 2024-09-11T03:49:18.935 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.935 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:18.935 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869444+0000", 2024-09-11T03:49:18.935 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3829999999999999e-06 2024-09-11T03:49:18.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.935 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:18.935 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869454+0000", 2024-09-11T03:49:18.935 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0166999999999999e-05 2024-09-11T03:49:18.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.935 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:18.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869467+0000", 2024-09-11T03:49:18.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3243e-05 2024-09-11T03:49:18.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.936 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:18.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869475+0000", 2024-09-11T03:49:18.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7209999999999994e-06 2024-09-11T03:49:18.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.936 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869537+0000", 2024-09-11T03:49:18.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2075000000000005e-05 2024-09-11T03:49:18.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.869625+0000", 2024-09-11T03:49:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7736000000000002e-05 2024-09-11T03:49:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,11", 2024-09-11T03:49:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.878093+0000", 2024-09-11T03:49:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0084675710000000001 2024-09-11T03:49:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.882577+0000", 2024-09-11T03:49:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0044839349999999997 2024-09-11T03:49:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.883819+0000", 2024-09-11T03:49:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012421680000000001 2024-09-11T03:49:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.884915+0000", 2024-09-11T03:49:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0010967559999999999 2024-09-11T03:49:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.884952+0000", 2024-09-11T03:49:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6835999999999998e-05 2024-09-11T03:49:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:39:59.884987+0000", 2024-09-11T03:49:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4533000000000003e-05 2024-09-11T03:49:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1513 2.5 2:a6e3bbd9:::smithi17557870-38:head [copy-from ver 208] snapc 3b=[3a,38,37,35] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-09-11T03:49:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:28.917633+0000", 2024-09-11T03:49:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 530.01418297299995, 2024-09-11T03:49:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021282327, 2024-09-11T03:49:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1513 2024-09-11T03:49:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917633+0000", 2024-09-11T03:49:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917633+0000", 2024-09-11T03:49:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917635+0000", 2024-09-11T03:49:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6390000000000001e-06 2024-09-11T03:49:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917649+0000", 2024-09-11T03:49:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3721e-05 2024-09-11T03:49:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917669+0000", 2024-09-11T03:49:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9423999999999999e-05 2024-09-11T03:49:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917676+0000", 2024-09-11T03:49:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7330000000000003e-06 2024-09-11T03:49:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917747+0000", 2024-09-11T03:49:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1081999999999995e-05 2024-09-11T03:49:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917816+0000", 2024-09-11T03:49:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8504999999999995e-05 2024-09-11T03:49:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:49:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.928464+0000", 2024-09-11T03:49:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.01064815 2024-09-11T03:49:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.937323+0000", 2024-09-11T03:49:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0088585160000000003 2024-09-11T03:49:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.938350+0000", 2024-09-11T03:49:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00102728 2024-09-11T03:49:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.938848+0000", 2024-09-11T03:49:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00049766199999999995 2024-09-11T03:49:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.938884+0000", 2024-09-11T03:49:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6916999999999999e-05 2024-09-11T03:49:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.938915+0000", 2024-09-11T03:49:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0698000000000003e-05 2024-09-11T03:49:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1514 2.5 2:a6e3bbd9:::smithi17557870-38:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2024-09-11T03:49:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:28.917699+0000", 2024-09-11T03:49:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 530.01411661300006, 2024-09-11T03:49:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017658610000000002, 2024-09-11T03:49:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:49:18.947 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:18.947 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:18.947 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:18.947 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1514 2024-09-11T03:49:18.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917699+0000", 2024-09-11T03:49:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917699+0000", 2024-09-11T03:49:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917700+0000", 2024-09-11T03:49:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.99e-07 2024-09-11T03:49:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917703+0000", 2024-09-11T03:49:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8260000000000001e-06 2024-09-11T03:49:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917713+0000", 2024-09-11T03:49:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0539e-05 2024-09-11T03:49:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917719+0000", 2024-09-11T03:49:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1500000000000004e-06 2024-09-11T03:49:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.917971+0000", 2024-09-11T03:49:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00025184900000000001 2024-09-11T03:49:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:49:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.918032+0000", 2024-09-11T03:49:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0449999999999999e-05 2024-09-11T03:49:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.935206+0000", 2024-09-11T03:49:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017174405 2024-09-11T03:49:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.935277+0000", 2024-09-11T03:49:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0862e-05 2024-09-11T03:49:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:28.935358+0000", 2024-09-11T03:49:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0929999999999999e-05 2024-09-11T03:49:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1583 2.b 2:de6b1316:::smithi17557870-32:head [copy-from ver 82] snapc 3f=[3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-09-11T03:49:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:34.845880+0000", 2024-09-11T03:49:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 524.08593619400006, 2024-09-11T03:49:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020941156999999998, 2024-09-11T03:49:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1583 2024-09-11T03:49:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845880+0000", 2024-09-11T03:49:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845880+0000", 2024-09-11T03:49:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845881+0000", 2024-09-11T03:49:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.871e-06 2024-09-11T03:49:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845891+0000", 2024-09-11T03:49:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2410000000000001e-06 2024-09-11T03:49:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845909+0000", 2024-09-11T03:49:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8162999999999999e-05 2024-09-11T03:49:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845919+0000", 2024-09-11T03:49:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9869999999999994e-06 2024-09-11T03:49:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845994+0000", 2024-09-11T03:49:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4757000000000003e-05 2024-09-11T03:49:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:49:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845999+0000", 2024-09-11T03:49:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.434e-06 2024-09-11T03:49:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.847498+0000", 2024-09-11T03:49:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014989370000000001 2024-09-11T03:49:18.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:18.957 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.847585+0000", 2024-09-11T03:49:18.957 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7058999999999994e-05 2024-09-11T03:49:18.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,11", 2024-09-11T03:49:18.957 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.860229+0000", 2024-09-11T03:49:18.957 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012643765 2024-09-11T03:49:18.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.864139+0000", 2024-09-11T03:49:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039105019999999997 2024-09-11T03:49:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.865334+0000", 2024-09-11T03:49:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001194938 2024-09-11T03:49:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.866748+0000", 2024-09-11T03:49:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001413873 2024-09-11T03:49:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.866787+0000", 2024-09-11T03:49:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8571000000000001e-05 2024-09-11T03:49:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.866821+0000", 2024-09-11T03:49:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4059e-05 2024-09-11T03:49:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1584 2.b 2:de6b1316:::smithi17557870-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e218)", 2024-09-11T03:49:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:34.845942+0000", 2024-09-11T03:49:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 524.085874246, 2024-09-11T03:49:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017600596999999999, 2024-09-11T03:49:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:49:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1584 2024-09-11T03:49:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845942+0000", 2024-09-11T03:49:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845942+0000", 2024-09-11T03:49:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845942+0000", 2024-09-11T03:49:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8100000000000003e-07 2024-09-11T03:49:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845944+0000", 2024-09-11T03:49:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0899999999999999e-06 2024-09-11T03:49:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845949+0000", 2024-09-11T03:49:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3129999999999998e-06 2024-09-11T03:49:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.845953+0000", 2024-09-11T03:49:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8060000000000001e-06 2024-09-11T03:49:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.846055+0000", 2024-09-11T03:49:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010187999999999999 2024-09-11T03:49:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:49:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.846062+0000", 2024-09-11T03:49:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6239999999999996e-06 2024-09-11T03:49:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.847720+0000", 2024-09-11T03:49:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001658681 2024-09-11T03:49:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:49:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.847773+0000", 2024-09-11T03:49:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2710999999999997e-05 2024-09-11T03:49:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.863362+0000", 2024-09-11T03:49:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015588493 2024-09-11T03:49:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.965 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.965 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:18.965 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.863447+0000", 2024-09-11T03:49:18.965 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5682999999999999e-05 2024-09-11T03:49:18.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.965 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.965 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:18.965 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:34.863542+0000", 2024-09-11T03:49:18.965 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4834999999999996e-05 2024-09-11T03:49:18.965 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.965 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:18.965 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.966 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.966 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1607 2.f 2:fa239c32:::smithi17557870-4:head [copy-from ver 82] snapc 40=[40,3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:49:18.966 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:35.897501+0000", 2024-09-11T03:49:18.966 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 523.03431454899999, 2024-09-11T03:49:18.966 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018113787999999999, 2024-09-11T03:49:18.966 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:18.966 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:18.966 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:18.966 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:18.966 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:18.966 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:18.966 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1607 2024-09-11T03:49:18.966 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.967 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:18.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.967 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:18.967 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897501+0000", 2024-09-11T03:49:18.967 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.967 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.967 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:18.967 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897501+0000", 2024-09-11T03:49:18.967 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.967 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.967 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:18.967 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897503+0000", 2024-09-11T03:49:18.968 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8840000000000001e-06 2024-09-11T03:49:18.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.968 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.968 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:18.968 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897510+0000", 2024-09-11T03:49:18.968 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4740000000000002e-06 2024-09-11T03:49:18.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.968 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.968 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:18.968 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897525+0000", 2024-09-11T03:49:18.968 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5702e-05 2024-09-11T03:49:18.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.968 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.968 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:18.969 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897532+0000", 2024-09-11T03:49:18.969 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7329999999999997e-06 2024-09-11T03:49:18.969 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.969 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.969 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.969 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897610+0000", 2024-09-11T03:49:18.969 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8153999999999997e-05 2024-09-11T03:49:18.969 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.969 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.969 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:49:18.969 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897618+0000", 2024-09-11T03:49:18.969 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5630000000000003e-06 2024-09-11T03:49:18.969 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.969 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.970 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.970 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.898813+0000", 2024-09-11T03:49:18.970 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0011955819999999999 2024-09-11T03:49:18.970 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.970 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.970 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:18.970 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.898858+0000", 2024-09-11T03:49:18.970 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5092000000000001e-05 2024-09-11T03:49:18.970 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.970 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.970 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,9", 2024-09-11T03:49:18.970 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.908861+0000", 2024-09-11T03:49:18.971 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010003025 2024-09-11T03:49:18.971 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.971 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.971 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:18.971 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.912807+0000", 2024-09-11T03:49:18.971 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039458339999999996 2024-09-11T03:49:18.971 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.971 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.971 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:18.971 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.914051+0000", 2024-09-11T03:49:18.971 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012439040000000001 2024-09-11T03:49:18.971 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.971 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.971 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:18.971 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.915500+0000", 2024-09-11T03:49:18.972 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014488839999999999 2024-09-11T03:49:18.972 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.972 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.972 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:18.972 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.915561+0000", 2024-09-11T03:49:18.972 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1290000000000004e-05 2024-09-11T03:49:18.972 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.972 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.972 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:18.972 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.915615+0000", 2024-09-11T03:49:18.973 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3667e-05 2024-09-11T03:49:18.973 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.973 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:18.973 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.973 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.973 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.973 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1608 2.f 2:fa239c32:::smithi17557870-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:49:18.973 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:40:35.897552+0000", 2024-09-11T03:49:18.973 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 523.03426424099996, 2024-09-11T03:49:18.973 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014781580000000001, 2024-09-11T03:49:18.973 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:18.973 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:18.974 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:49:18.974 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:18.974 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:18.974 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:18.974 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1608 2024-09-11T03:49:18.974 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.974 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:18.974 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.974 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:18.974 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897552+0000", 2024-09-11T03:49:18.974 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.974 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.974 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.975 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:18.975 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897552+0000", 2024-09-11T03:49:18.975 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.975 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.975 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.975 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:18.975 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897552+0000", 2024-09-11T03:49:18.975 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3399999999999999e-07 2024-09-11T03:49:18.975 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.975 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.975 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:18.975 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897558+0000", 2024-09-11T03:49:18.975 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7769999999999999e-06 2024-09-11T03:49:18.975 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.976 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.976 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:18.977 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897565+0000", 2024-09-11T03:49:18.977 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7410000000000003e-06 2024-09-11T03:49:18.977 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.977 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.977 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:18.977 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897568+0000", 2024-09-11T03:49:18.977 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6799999999999999e-06 2024-09-11T03:49:18.977 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.978 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.978 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897671+0000", 2024-09-11T03:49:18.978 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000102291 2024-09-11T03:49:18.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.978 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:49:18.978 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.897674+0000", 2024-09-11T03:49:18.978 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6890000000000002e-06 2024-09-11T03:49:18.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.978 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.978 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.898967+0000", 2024-09-11T03:49:18.979 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001292996 2024-09-11T03:49:18.979 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.979 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.979 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:49:18.979 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.899052+0000", 2024-09-11T03:49:18.979 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5112999999999996e-05 2024-09-11T03:49:18.979 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.979 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.979 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.979 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.912099+0000", 2024-09-11T03:49:18.979 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013046775 2024-09-11T03:49:18.979 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.979 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.980 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:18.980 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.912198+0000", 2024-09-11T03:49:18.980 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8659000000000005e-05 2024-09-11T03:49:18.980 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.980 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.980 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:18.980 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:40:35.912333+0000", 2024-09-11T03:49:18.980 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000135325 2024-09-11T03:49:18.980 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.980 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:18.980 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.980 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.980 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.981 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:1877 2.b 2:de6b1316:::smithi17557870-32:head [copy-from ver 223] snapc 4d=[4d,4b,49,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-11T03:49:18.981 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:03.907218+0000", 2024-09-11T03:49:18.981 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 495.02459764399998, 2024-09-11T03:49:18.981 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019706267999999999, 2024-09-11T03:49:18.981 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:18.981 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:18.981 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:18.981 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:18.981 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:18.981 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:18.981 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 1877 2024-09-11T03:49:18.981 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.981 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:18.982 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.982 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:18.982 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907218+0000", 2024-09-11T03:49:18.982 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.982 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.982 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.982 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:18.982 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907218+0000", 2024-09-11T03:49:18.982 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.982 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.982 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.982 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:18.982 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907219+0000", 2024-09-11T03:49:18.983 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.221e-06 2024-09-11T03:49:18.983 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.983 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.983 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:18.983 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907226+0000", 2024-09-11T03:49:18.983 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1580000000000002e-06 2024-09-11T03:49:18.983 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.983 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.983 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:18.983 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907239+0000", 2024-09-11T03:49:18.983 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3865e-05 2024-09-11T03:49:18.983 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.984 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.984 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:18.984 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907247+0000", 2024-09-11T03:49:18.984 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8520000000000004e-06 2024-09-11T03:49:18.984 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.984 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.984 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.984 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907312+0000", 2024-09-11T03:49:18.984 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4437000000000007e-05 2024-09-11T03:49:18.984 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.984 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.984 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:18.984 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.907417+0000", 2024-09-11T03:49:18.984 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000105698 2024-09-11T03:49:18.985 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.985 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.985 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,11", 2024-09-11T03:49:18.985 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.915919+0000", 2024-09-11T03:49:18.985 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.008501794 2024-09-11T03:49:18.985 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.985 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.985 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:18.985 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.920523+0000", 2024-09-11T03:49:18.985 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0046041220000000004 2024-09-11T03:49:18.985 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.985 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.985 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:18.986 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.920865+0000", 2024-09-11T03:49:18.986 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00034196500000000001 2024-09-11T03:49:18.986 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.986 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.986 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:18.986 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.926838+0000", 2024-09-11T03:49:18.986 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0059723420000000003 2024-09-11T03:49:18.986 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.986 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.986 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:18.986 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.926882+0000", 2024-09-11T03:49:18.986 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4199999999999997e-05 2024-09-11T03:49:18.986 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.986 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.987 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:18.987 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:03.926924+0000", 2024-09-11T03:49:18.987 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2614000000000002e-05 2024-09-11T03:49:18.987 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.987 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:18.987 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.987 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.987 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.987 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2115 2.5 2:a6e3bbd9:::smithi17557870-38:head [copy-from ver 221] snapc 57=[57,56,51,4e] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-11T03:49:18.987 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:24.863058+0000", 2024-09-11T03:49:18.987 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 474.06875733300001, 2024-09-11T03:49:18.987 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.025181835999999999, 2024-09-11T03:49:18.987 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:18.988 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:18.988 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:18.988 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:18.988 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:18.988 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:18.988 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2115 2024-09-11T03:49:18.988 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.988 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:18.988 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.988 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:18.988 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863058+0000", 2024-09-11T03:49:18.988 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.989 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.989 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:18.989 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863058+0000", 2024-09-11T03:49:18.989 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.989 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.989 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:18.989 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863061+0000", 2024-09-11T03:49:18.989 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2649999999999999e-06 2024-09-11T03:49:18.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.989 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.989 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:18.989 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863072+0000", 2024-09-11T03:49:18.990 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1409999999999999e-05 2024-09-11T03:49:18.990 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.990 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.990 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:18.990 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863091+0000", 2024-09-11T03:49:18.990 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8825e-05 2024-09-11T03:49:18.990 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.990 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.990 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:18.990 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863099+0000", 2024-09-11T03:49:18.990 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7849999999999992e-06 2024-09-11T03:49:18.990 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.990 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.990 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.991 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863164+0000", 2024-09-11T03:49:18.991 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5078000000000003e-05 2024-09-11T03:49:18.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.991 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:49:18.991 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863169+0000", 2024-09-11T03:49:18.991 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1649999999999997e-06 2024-09-11T03:49:18.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.991 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.991 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.864677+0000", 2024-09-11T03:49:18.991 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001507928 2024-09-11T03:49:18.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.864744+0000", 2024-09-11T03:49:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6990999999999996e-05 2024-09-11T03:49:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:49:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.876551+0000", 2024-09-11T03:49:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011806891 2024-09-11T03:49:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:18.993 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.882152+0000", 2024-09-11T03:49:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0056009240000000002 2024-09-11T03:49:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.884525+0000", 2024-09-11T03:49:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002373436 2024-09-11T03:49:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.888133+0000", 2024-09-11T03:49:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003608207 2024-09-11T03:49:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.888194+0000", 2024-09-11T03:49:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1007000000000001e-05 2024-09-11T03:49:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.888240+0000", 2024-09-11T03:49:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5924000000000003e-05 2024-09-11T03:49:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2116 2.5 2:a6e3bbd9:::smithi17557870-38:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e268)", 2024-09-11T03:49:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:24.863121+0000", 2024-09-11T03:49:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 474.06869481400003, 2024-09-11T03:49:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018358349999999999, 2024-09-11T03:49:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:49:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2116 2024-09-11T03:49:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863121+0000", 2024-09-11T03:49:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863121+0000", 2024-09-11T03:49:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863121+0000", 2024-09-11T03:49:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5900000000000002e-07 2024-09-11T03:49:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863126+0000", 2024-09-11T03:49:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8910000000000003e-06 2024-09-11T03:49:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863133+0000", 2024-09-11T03:49:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4489999999999996e-06 2024-09-11T03:49:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863137+0000", 2024-09-11T03:49:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4549999999999997e-06 2024-09-11T03:49:18.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.999 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.999 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:18.999 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863232+0000", 2024-09-11T03:49:18.999 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4308999999999994e-05 2024-09-11T03:49:18.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:18.999 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:18.999 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:49:18.999 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.863238+0000", 2024-09-11T03:49:19.000 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6279999999999999e-06 2024-09-11T03:49:19.000 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.000 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.000 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:19.000 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.864857+0000", 2024-09-11T03:49:19.000 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0016184330000000001 2024-09-11T03:49:19.000 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.000 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.000 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:49:19.000 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.864900+0000", 2024-09-11T03:49:19.000 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3745000000000002e-05 2024-09-11T03:49:19.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.001 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.001 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:19.001 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.881241+0000", 2024-09-11T03:49:19.001 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016340232999999999 2024-09-11T03:49:19.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.001 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.001 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:19.001 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.881352+0000", 2024-09-11T03:49:19.001 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000110945 2024-09-11T03:49:19.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.001 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.001 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:19.001 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:24.881479+0000", 2024-09-11T03:49:19.002 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012780300000000001 2024-09-11T03:49:19.002 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.002 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:19.002 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.002 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.002 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.002 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2165 2.b 2:dde92fc6:::smithi17557870-41:head [copy-from ver 101] snapc 58=[58,57,56,51] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-09-11T03:49:19.002 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:26.857685+0000", 2024-09-11T03:49:19.002 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 472.074130422, 2024-09-11T03:49:19.002 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022414478000000002, 2024-09-11T03:49:19.002 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:19.002 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:19.002 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:19.003 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:19.003 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:19.003 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:19.003 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2165 2024-09-11T03:49:19.003 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.003 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:19.003 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.003 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:19.003 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857685+0000", 2024-09-11T03:49:19.003 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.003 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.003 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.003 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:19.003 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857685+0000", 2024-09-11T03:49:19.004 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.004 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.004 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.004 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:19.004 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857687+0000", 2024-09-11T03:49:19.004 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.716e-06 2024-09-11T03:49:19.004 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.004 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.004 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:19.004 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857694+0000", 2024-09-11T03:49:19.004 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9909999999999996e-06 2024-09-11T03:49:19.004 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.004 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.004 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:19.005 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857731+0000", 2024-09-11T03:49:19.005 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7083999999999999e-05 2024-09-11T03:49:19.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.005 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.005 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:19.005 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857741+0000", 2024-09-11T03:49:19.005 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0054999999999999e-05 2024-09-11T03:49:19.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.005 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.005 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:19.005 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857802+0000", 2024-09-11T03:49:19.005 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0402000000000002e-05 2024-09-11T03:49:19.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.006 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.006 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:19.006 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857878+0000", 2024-09-11T03:49:19.006 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5926000000000005e-05 2024-09-11T03:49:19.006 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.006 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.006 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,11", 2024-09-11T03:49:19.006 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.871599+0000", 2024-09-11T03:49:19.006 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013721492 2024-09-11T03:49:19.006 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.006 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.006 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:19.006 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.876926+0000", 2024-09-11T03:49:19.007 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0053270949999999996 2024-09-11T03:49:19.007 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.007 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.007 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:19.007 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.877015+0000", 2024-09-11T03:49:19.007 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.9315999999999997e-05 2024-09-11T03:49:19.007 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.007 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.007 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:19.007 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.880031+0000", 2024-09-11T03:49:19.007 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030155830000000001 2024-09-11T03:49:19.007 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.007 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.007 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:19.008 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.880069+0000", 2024-09-11T03:49:19.008 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8349999999999997e-05 2024-09-11T03:49:19.008 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.008 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.008 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:19.008 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.880100+0000", 2024-09-11T03:49:19.008 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0468e-05 2024-09-11T03:49:19.008 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.009 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:19.009 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.009 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.009 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.009 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2166 2.b 2:dde92fc6:::smithi17557870-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e270)", 2024-09-11T03:49:19.009 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:26.857761+0000", 2024-09-11T03:49:19.009 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 472.07405442700002, 2024-09-11T03:49:19.009 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018486750999999999, 2024-09-11T03:49:19.009 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:19.009 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:19.009 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:49:19.009 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:19.009 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:19.009 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:19.010 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2166 2024-09-11T03:49:19.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.010 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:19.010 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.010 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:19.010 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857761+0000", 2024-09-11T03:49:19.010 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.010 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.010 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:19.010 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857761+0000", 2024-09-11T03:49:19.010 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.011 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.011 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.011 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:19.011 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857762+0000", 2024-09-11T03:49:19.011 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.51e-07 2024-09-11T03:49:19.011 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.011 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.011 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:19.011 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857765+0000", 2024-09-11T03:49:19.011 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0029999999999999e-06 2024-09-11T03:49:19.012 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.012 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.012 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:19.012 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857772+0000", 2024-09-11T03:49:19.013 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1060000000000002e-06 2024-09-11T03:49:19.013 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.013 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.013 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:19.013 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857777+0000", 2024-09-11T03:49:19.013 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6290000000000001e-06 2024-09-11T03:49:19.013 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.013 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.013 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:19.013 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.857989+0000", 2024-09-11T03:49:19.013 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00021282300000000001 2024-09-11T03:49:19.013 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.013 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.014 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:49:19.014 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.858027+0000", 2024-09-11T03:49:19.014 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7326999999999997e-05 2024-09-11T03:49:19.014 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.014 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.014 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:19.014 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.876066+0000", 2024-09-11T03:49:19.014 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018039276999999999 2024-09-11T03:49:19.014 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.014 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.014 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:19.015 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.876155+0000", 2024-09-11T03:49:19.015 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.9062999999999998e-05 2024-09-11T03:49:19.015 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.015 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.015 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:19.015 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:26.876248+0000", 2024-09-11T03:49:19.015 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2972000000000004e-05 2024-09-11T03:49:19.015 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.015 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:19.015 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.015 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.016 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.016 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2492 2.5 2:a1a99782:::smithi17557870-1:head [copy-from ver 279] snapc 6b=[6b,6a,69,68,67,65,64,63,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-11T03:49:19.016 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:58.892917+0000", 2024-09-11T03:49:19.016 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 440.03889891400001, 2024-09-11T03:49:19.016 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018454061000000001, 2024-09-11T03:49:19.016 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:19.016 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:19.016 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:19.016 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:19.016 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:19.016 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:19.016 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2492 2024-09-11T03:49:19.016 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.016 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:19.017 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.017 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:19.017 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892917+0000", 2024-09-11T03:49:19.017 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.017 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.017 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.017 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:19.017 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892917+0000", 2024-09-11T03:49:19.017 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.017 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.017 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.017 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:19.017 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892918+0000", 2024-09-11T03:49:19.017 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0899999999999999e-06 2024-09-11T03:49:19.018 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.018 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.018 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:19.018 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892925+0000", 2024-09-11T03:49:19.018 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5470000000000002e-06 2024-09-11T03:49:19.018 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.018 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.018 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:19.018 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892937+0000", 2024-09-11T03:49:19.018 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.269e-05 2024-09-11T03:49:19.018 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.018 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.018 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:19.018 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.892945+0000", 2024-09-11T03:49:19.019 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7270000000000007e-06 2024-09-11T03:49:19.019 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.019 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.019 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:19.019 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.893002+0000", 2024-09-11T03:49:19.019 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6650999999999997e-05 2024-09-11T03:49:19.019 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.019 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.019 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:19.019 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.893112+0000", 2024-09-11T03:49:19.019 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011042100000000001 2024-09-11T03:49:19.020 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.020 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.020 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:49:19.020 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.901793+0000", 2024-09-11T03:49:19.020 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0086810189999999999 2024-09-11T03:49:19.020 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.020 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.020 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:19.020 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.905968+0000", 2024-09-11T03:49:19.020 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0041751239999999997 2024-09-11T03:49:19.020 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.020 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.020 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:19.021 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.910080+0000", 2024-09-11T03:49:19.021 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0041120799999999997 2024-09-11T03:49:19.021 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.021 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.021 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:19.021 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.911159+0000", 2024-09-11T03:49:19.021 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0010783609999999999 2024-09-11T03:49:19.021 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.021 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.021 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:19.021 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.911257+0000", 2024-09-11T03:49:19.021 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8167999999999999e-05 2024-09-11T03:49:19.021 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.021 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.021 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:19.022 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:58.911371+0000", 2024-09-11T03:49:19.022 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114183 2024-09-11T03:49:19.022 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.022 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:19.022 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.022 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.022 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.022 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4164 2.b 2:dde92fc6:::smithi17557870-41:head [copy-from ver 223] snapc b4=[b4,b2,b1,af,ae,ac,ab,a9,a8,9d] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-09-11T03:49:19.022 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:27.930005+0000", 2024-09-11T03:49:19.022 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 291.00181113600001, 2024-09-11T03:49:19.022 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014694021999999999, 2024-09-11T03:49:19.022 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:19.022 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:19.022 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:19.023 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:19.023 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:19.023 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:19.023 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4164 2024-09-11T03:49:19.023 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.023 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:19.023 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.023 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:19.023 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930005+0000", 2024-09-11T03:49:19.023 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.023 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.023 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.023 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:19.023 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930005+0000", 2024-09-11T03:49:19.024 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.024 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.024 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.024 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:19.024 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930006+0000", 2024-09-11T03:49:19.024 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.466e-06 2024-09-11T03:49:19.024 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.024 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.024 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:19.025 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930011+0000", 2024-09-11T03:49:19.025 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9930000000000003e-06 2024-09-11T03:49:19.025 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.025 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.025 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:19.025 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930022+0000", 2024-09-11T03:49:19.025 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0831e-05 2024-09-11T03:49:19.025 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.025 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.025 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:19.025 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930029+0000", 2024-09-11T03:49:19.025 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.776e-06 2024-09-11T03:49:19.025 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.025 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.026 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:19.026 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930083+0000", 2024-09-11T03:49:19.026 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4586000000000002e-05 2024-09-11T03:49:19.026 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.026 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.026 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:19.026 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930150+0000", 2024-09-11T03:49:19.026 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6595999999999997e-05 2024-09-11T03:49:19.026 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.026 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.026 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,11", 2024-09-11T03:49:19.026 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.936619+0000", 2024-09-11T03:49:19.026 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0064690360000000001 2024-09-11T03:49:19.027 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.027 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.027 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:19.027 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.942717+0000", 2024-09-11T03:49:19.027 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0060981550000000001 2024-09-11T03:49:19.027 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.027 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.027 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:19.027 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.943814+0000", 2024-09-11T03:49:19.027 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001097251 2024-09-11T03:49:19.027 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.027 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.027 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:19.027 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.944627+0000", 2024-09-11T03:49:19.028 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00081236800000000005 2024-09-11T03:49:19.028 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.028 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.028 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:19.028 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.944664+0000", 2024-09-11T03:49:19.028 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7282e-05 2024-09-11T03:49:19.028 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.028 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.028 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:19.028 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.944699+0000", 2024-09-11T03:49:19.028 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4681999999999998e-05 2024-09-11T03:49:19.028 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.028 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:19.029 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.029 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.029 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.029 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4700 2.5 2:a1a99782:::smithi17557870-1:head [copy-from ver 541] snapc cd=[cd,cc,ca,c5,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-09-11T03:49:19.029 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.960471+0000", 2024-09-11T03:49:19.029 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 236.97134506, 2024-09-11T03:49:19.029 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015308754000000001, 2024-09-11T03:49:19.029 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:19.029 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:19.029 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:19.029 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:19.030 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:19.030 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:19.030 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4700 2024-09-11T03:49:19.030 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.030 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:19.030 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.030 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:19.030 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960471+0000", 2024-09-11T03:49:19.030 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.030 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.030 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.030 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:19.030 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960471+0000", 2024-09-11T03:49:19.031 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.031 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.031 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.031 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:19.031 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960473+0000", 2024-09-11T03:49:19.031 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8050000000000001e-06 2024-09-11T03:49:19.031 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.031 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.031 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:19.031 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960480+0000", 2024-09-11T03:49:19.031 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3780000000000003e-06 2024-09-11T03:49:19.031 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.031 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.031 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:19.031 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960497+0000", 2024-09-11T03:49:19.032 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7224e-05 2024-09-11T03:49:19.032 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.032 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.032 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:19.032 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960504+0000", 2024-09-11T03:49:19.032 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2620000000000001e-06 2024-09-11T03:49:19.032 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.032 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.032 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:19.032 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960575+0000", 2024-09-11T03:49:19.032 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0507999999999996e-05 2024-09-11T03:49:19.032 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.032 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.033 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:19.033 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960757+0000", 2024-09-11T03:49:19.033 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000182325 2024-09-11T03:49:19.033 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.033 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.033 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:49:19.033 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.967152+0000", 2024-09-11T03:49:19.033 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0063946519999999998 2024-09-11T03:49:19.033 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.033 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.033 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:19.033 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.972224+0000", 2024-09-11T03:49:19.033 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050720349999999999 2024-09-11T03:49:19.033 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.034 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.034 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:19.034 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.974955+0000", 2024-09-11T03:49:19.034 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0027314829999999998 2024-09-11T03:49:19.034 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.034 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.034 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:19.034 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.975676+0000", 2024-09-11T03:49:19.034 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00072077700000000003 2024-09-11T03:49:19.034 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.034 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.034 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:19.034 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.975727+0000", 2024-09-11T03:49:19.034 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0315999999999997e-05 2024-09-11T03:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.975779+0000", 2024-09-11T03:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2988999999999997e-05 2024-09-11T03:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6565 2.5 2:a51832fe:::smithi17557870-12:head [copy-from ver 930] snapc 120=[120,11f,11e,11d,11a,119,114] ondisk+write+known_if_redirected+supports_pool_eio e670)", 2024-09-11T03:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:08.152253+0000", 2024-09-11T03:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 70.779562626000001, 2024-09-11T03:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021627759, 2024-09-11T03:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6565 2024-09-11T03:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152253+0000", 2024-09-11T03:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152253+0000", 2024-09-11T03:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152255+0000", 2024-09-11T03:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9860000000000001e-06 2024-09-11T03:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152264+0000", 2024-09-11T03:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6070000000000008e-06 2024-09-11T03:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152283+0000", 2024-09-11T03:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9578000000000001e-05 2024-09-11T03:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152292+0000", 2024-09-11T03:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.0469999999999999e-06 2024-09-11T03:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152369+0000", 2024-09-11T03:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6310999999999995e-05 2024-09-11T03:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152491+0000", 2024-09-11T03:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000122712 2024-09-11T03:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.161123+0000", 2024-09-11T03:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.008631906 2024-09-11T03:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.171985+0000", 2024-09-11T03:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010861572 2024-09-11T03:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.173347+0000", 2024-09-11T03:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00136213 2024-09-11T03:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.173764+0000", 2024-09-11T03:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00041722999999999999 2024-09-11T03:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.173827+0000", 2024-09-11T03:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2780999999999994e-05 2024-09-11T03:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.173881+0000", 2024-09-11T03:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3899e-05 2024-09-11T03:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6566 2.5 2:a51832fe:::smithi17557870-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e670)", 2024-09-11T03:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:08.152319+0000", 2024-09-11T03:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 70.779496416000001, 2024-09-11T03:49:19.042 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015878518000000001, 2024-09-11T03:49:19.042 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:19.042 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:19.042 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:49:19.042 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:19.042 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:19.042 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:19.042 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6566 2024-09-11T03:49:19.042 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.042 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:19.042 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.044 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:19.044 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152319+0000", 2024-09-11T03:49:19.044 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.044 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.044 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:19.044 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152319+0000", 2024-09-11T03:49:19.045 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.045 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.045 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.045 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:19.045 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152320+0000", 2024-09-11T03:49:19.045 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6000000000000004e-07 2024-09-11T03:49:19.045 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.045 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.045 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:19.045 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152323+0000", 2024-09-11T03:49:19.045 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8930000000000001e-06 2024-09-11T03:49:19.045 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.045 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.045 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:19.046 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152338+0000", 2024-09-11T03:49:19.046 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5197e-05 2024-09-11T03:49:19.046 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.046 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.046 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:19.046 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152345+0000", 2024-09-11T03:49:19.046 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4790000000000001e-06 2024-09-11T03:49:19.046 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.046 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.046 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:19.046 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152651+0000", 2024-09-11T03:49:19.046 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000306206 2024-09-11T03:49:19.046 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.046 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.047 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:49:19.047 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152707+0000", 2024-09-11T03:49:19.047 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6200999999999999e-05 2024-09-11T03:49:19.047 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.047 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.047 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:19.047 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.167971+0000", 2024-09-11T03:49:19.047 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015263914999999999 2024-09-11T03:49:19.047 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.047 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.047 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:19.047 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.168075+0000", 2024-09-11T03:49:19.047 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000104199 2024-09-11T03:49:19.047 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.048 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.048 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:19.048 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.168198+0000", 2024-09-11T03:49:19.048 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012286799999999999 2024-09-11T03:49:19.048 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.048 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:19.048 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.048 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.048 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.048 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6567 2.5 2:a6d17fc0:::smithi17557870-13:head [write 443949~496850,stat] snapc 120=[120,11f,11e,11d,11a,119,114] ondisk+write+known_if_redirected+supports_pool_eio e670)", 2024-09-11T03:49:19.048 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:08.161541+0000", 2024-09-11T03:49:19.048 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 70.770274658999995, 2024-09-11T03:49:19.048 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014579708, 2024-09-11T03:49:19.049 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:19.049 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:19.049 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:19.049 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:19.049 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:19.049 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:19.049 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6567 2024-09-11T03:49:19.049 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.049 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:19.049 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.049 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:19.049 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.161541+0000", 2024-09-11T03:49:19.049 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.049 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.049 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.050 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:19.050 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.161541+0000", 2024-09-11T03:49:19.050 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.050 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.050 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.050 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:19.050 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.161544+0000", 2024-09-11T03:49:19.050 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7480000000000002e-06 2024-09-11T03:49:19.050 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.050 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.050 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:19.050 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.161782+0000", 2024-09-11T03:49:19.050 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00023799699999999999 2024-09-11T03:49:19.050 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.051 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.051 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:19.051 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.161807+0000", 2024-09-11T03:49:19.051 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5567000000000001e-05 2024-09-11T03:49:19.051 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.051 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.051 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:19.051 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.161824+0000", 2024-09-11T03:49:19.051 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6348999999999999e-05 2024-09-11T03:49:19.051 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.051 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.051 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:19.051 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.168347+0000", 2024-09-11T03:49:19.051 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0065227920000000003 2024-09-11T03:49:19.052 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.052 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.052 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:19.052 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.168469+0000", 2024-09-11T03:49:19.052 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012229 2024-09-11T03:49:19.052 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.052 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.052 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:49:19.052 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.168692+0000", 2024-09-11T03:49:19.052 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00022280800000000001 2024-09-11T03:49:19.052 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.052 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.052 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:19.052 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.173400+0000", 2024-09-11T03:49:19.053 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0047080619999999998 2024-09-11T03:49:19.053 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.053 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.053 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:19.053 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.175710+0000", 2024-09-11T03:49:19.053 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023102019999999999 2024-09-11T03:49:19.053 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.053 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.053 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:19.053 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.175985+0000", 2024-09-11T03:49:19.053 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000275495 2024-09-11T03:49:19.053 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.053 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.053 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:19.054 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.176058+0000", 2024-09-11T03:49:19.054 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2063000000000004e-05 2024-09-11T03:49:19.054 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.054 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.054 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:19.054 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.176121+0000", 2024-09-11T03:49:19.054 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3335000000000003e-05 2024-09-11T03:49:19.054 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.054 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:19.054 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.054 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.054 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.054 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7235 2.5 2:a51832fe:::smithi17557870-12:head [copy-from ver 950] snapc 141=[141,140,13f,13d,13c,139,138,137,136,135,133,12e,12d,12a,127,126,120,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-09-11T03:49:19.055 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:49:01.070004+0000", 2024-09-11T03:49:19.055 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 17.861811449000001, 2024-09-11T03:49:19.055 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.01550867, 2024-09-11T03:49:19.055 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:19.055 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:19.055 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:19.055 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:19.055 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:19.055 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:19.055 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7235 2024-09-11T03:49:19.055 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.055 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:19.055 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.055 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:19.056 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.070004+0000", 2024-09-11T03:49:19.056 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.056 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.056 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.056 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:19.056 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.070004+0000", 2024-09-11T03:49:19.056 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.056 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.056 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.056 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:19.056 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.070005+0000", 2024-09-11T03:49:19.056 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0809999999999999e-06 2024-09-11T03:49:19.056 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.056 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.057 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:19.057 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.070010+0000", 2024-09-11T03:49:19.057 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9370000000000003e-06 2024-09-11T03:49:19.057 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.057 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.057 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:19.057 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.070020+0000", 2024-09-11T03:49:19.057 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7999999999999993e-06 2024-09-11T03:49:19.057 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.057 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.057 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:19.057 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.070026+0000", 2024-09-11T03:49:19.057 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5949999999999998e-06 2024-09-11T03:49:19.058 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.058 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.058 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:19.058 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.070094+0000", 2024-09-11T03:49:19.058 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8169000000000004e-05 2024-09-11T03:49:19.058 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.058 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.058 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:19.058 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.070194+0000", 2024-09-11T03:49:19.058 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9807000000000002e-05 2024-09-11T03:49:19.058 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.058 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.058 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:49:19.058 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.074863+0000", 2024-09-11T03:49:19.058 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0046691559999999998 2024-09-11T03:49:19.059 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.059 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.059 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:19.059 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.078953+0000", 2024-09-11T03:49:19.059 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0040905489999999997 2024-09-11T03:49:19.059 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.059 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.059 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:19.059 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.084272+0000", 2024-09-11T03:49:19.059 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0053186539999999999 2024-09-11T03:49:19.059 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.059 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.059 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:19.060 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.085295+0000", 2024-09-11T03:49:19.060 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0010226339999999999 2024-09-11T03:49:19.060 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.060 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.060 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:19.060 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.085409+0000", 2024-09-11T03:49:19.060 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000113828 2024-09-11T03:49:19.060 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.060 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.060 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:19.060 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.085513+0000", 2024-09-11T03:49:19.060 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010446 2024-09-11T03:49:19.060 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.060 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:19.061 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.061 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.061 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.061 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:7315 2.1f e733/574 2:fa239c32:::smithi17557870-4:head v 733'612, mlcod=733'610)", 2024-09-11T03:49:19.061 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:49:11.091486+0000", 2024-09-11T03:49:19.061 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 7.8403301289999998, 2024-09-11T03:49:19.061 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0021988810000000002, 2024-09-11T03:49:19.061 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:19.061 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:19.061 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:19.061 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:19.061 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.061 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:19.061 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:11.091486+0000", 2024-09-11T03:49:19.061 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.062 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.062 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.062 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:19.062 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:11.091486+0000", 2024-09-11T03:49:19.062 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:19.062 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.062 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.062 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:19.062 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:11.091487+0000", 2024-09-11T03:49:19.062 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5099999999999998e-07 2024-09-11T03:49:19.062 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.062 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.062 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:19.062 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:11.092644+0000", 2024-09-11T03:49:19.063 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0011576329999999999 2024-09-11T03:49:19.063 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.063 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.063 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:19.063 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:11.092656+0000", 2024-09-11T03:49:19.063 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2092999999999999e-05 2024-09-11T03:49:19.063 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.063 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.063 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:19.063 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:11.092665+0000", 2024-09-11T03:49:19.063 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7779999999999996e-06 2024-09-11T03:49:19.063 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.063 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.063 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:11.092717+0000", 2024-09-11T03:49:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1842999999999998e-05 2024-09-11T03:49:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:11.092735+0000", 2024-09-11T03:49:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7778000000000002e-05 2024-09-11T03:49:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:11.093639+0000", 2024-09-11T03:49:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00090444199999999998 2024-09-11T03:49:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:19.064 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:19.065 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:19.065 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:11.093685+0000", 2024-09-11T03:49:19.065 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5362999999999999e-05 2024-09-11T03:49:19.065 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.065 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:19.065 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.065 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:19.065 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:19.065 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:49:19.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:19.067 INFO:tasks.rados.rados.2.smithi175.stdout:2950: done (3 left) 2024-09-11T03:49:19.067 INFO:tasks.rados.rados.2.smithi175.stdout:2951: copy_from oid 18 from oid 21 current snap is 331 2024-09-11T03:49:19.067 INFO:tasks.rados.rados.2.smithi175.stdout:2948: expect (ObjNum 917 snap 294 seq_num 917) 2024-09-11T03:49:19.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:19.089 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:19.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:19 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:49:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:19 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:49:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:19 smithi121 ceph-mon[30729]: 1.6 scrub starts 2024-09-11T03:49:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:19 smithi121 ceph-mon[30729]: 1.6 scrub ok 2024-09-11T03:49:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:19 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:49:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:19 smithi121 ceph-mon[30729]: 2.e scrub ok 2024-09-11T03:49:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:19 smithi121 ceph-mon[30729]: from='client.16920 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:49:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:19 smithi121 ceph-mon[30729]: 2.22 scrub starts 2024-09-11T03:49:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:19 smithi121 ceph-mon[30729]: 2.22 scrub ok 2024-09-11T03:49:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:19 smithi121 ceph-mon[30729]: 2.21 scrub starts 2024-09-11T03:49:19.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:19 smithi121 ceph-mon[30729]: 2.21 scrub ok 2024-09-11T03:49:19.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:19 smithi121 ceph-mon[30729]: osdmap e741: 12 total, 12 up, 12 in 2024-09-11T03:49:19.191 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:19.194 INFO:tasks.rados.rados.2.smithi175.stdout:2949: expect (ObjNum 993 snap 320 seq_num 993) 2024-09-11T03:49:19.212 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:19.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:19.213 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:19.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:19 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:49:19.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:19 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:49:19.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:19 smithi175 ceph-mon[30894]: 1.6 scrub starts 2024-09-11T03:49:19.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:19 smithi175 ceph-mon[30894]: 1.6 scrub ok 2024-09-11T03:49:19.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:19 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:49:19.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:19 smithi175 ceph-mon[30894]: 2.e scrub ok 2024-09-11T03:49:19.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:19 smithi175 ceph-mon[30894]: from='client.16920 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:49:19.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:19 smithi175 ceph-mon[30894]: 2.22 scrub starts 2024-09-11T03:49:19.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:19 smithi175 ceph-mon[30894]: 2.22 scrub ok 2024-09-11T03:49:19.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:19 smithi175 ceph-mon[30894]: 2.21 scrub starts 2024-09-11T03:49:19.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:19 smithi175 ceph-mon[30894]: 2.21 scrub ok 2024-09-11T03:49:19.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:19 smithi175 ceph-mon[30894]: osdmap e741: 12 total, 12 up, 12 in 2024-09-11T03:49:19.314 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:19.335 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:19.335 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:19.354 INFO:tasks.rados.rados.2.smithi175.stdout:2951: finishing copy_from racing read to smithi17557870-18 2024-09-11T03:49:19.354 INFO:tasks.rados.rados.2.smithi175.stdout:2951: finishing copy_from to smithi17557870-18 2024-09-11T03:49:19.354 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 930 (ObjNum 829 snap 263 seq_num 829) dirty exists 2024-09-11T03:49:19.355 INFO:tasks.rados.rados.2.smithi175.stdout:2947: done (3 left) 2024-09-11T03:49:19.355 INFO:tasks.rados.rados.2.smithi175.stdout:2948: done (2 left) 2024-09-11T03:49:19.355 INFO:tasks.rados.rados.2.smithi175.stdout:2949: done (1 left) 2024-09-11T03:49:19.355 INFO:tasks.rados.rados.2.smithi175.stdout:2951: done (0 left) 2024-09-11T03:49:19.355 INFO:tasks.rados.rados.2.smithi175.stdout:2952: rollback oid 28 current snap is 331 2024-09-11T03:49:19.355 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 28 to 315 2024-09-11T03:49:19.355 INFO:tasks.rados.rados.2.smithi175.stdout:2953: copy_from oid 11 from oid 16 current snap is 331 2024-09-11T03:49:19.355 INFO:tasks.rados.rados.2.smithi175.stdout:2954: write (excl) oid 50 current snap is 331 2024-09-11T03:49:19.355 INFO:tasks.rados.rados.2.smithi175.stdout:2954: seq_num 1013 ranges {625491=502186,1861671=713346} 2024-09-11T03:49:19.360 INFO:tasks.rados.rados.2.smithi175.stdout:2954: writing smithi17557870-50 from 625491 to 1127677 tid 1 2024-09-11T03:49:19.368 INFO:tasks.rados.rados.2.smithi175.stdout:2954: writing smithi17557870-50 from 1861671 to 2575017 tid 2 2024-09-11T03:49:19.368 INFO:tasks.rados.rados.2.smithi175.stdout:2955: rollback oid 25 current snap is 331 2024-09-11T03:49:19.369 INFO:tasks.rados.rados.2.smithi175.stdout:2952: finishing rollback tid 0 to smithi17557870-28 2024-09-11T03:49:19.369 INFO:tasks.rados.rados.2.smithi175.stdout:2952: finishing rollback tid 1 to smithi17557870-28 2024-09-11T03:49:19.369 INFO:tasks.rados.rados.2.smithi175.stdout:2952: finishing rollback tid 2 to smithi17557870-28 2024-09-11T03:49:19.369 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 1135 (ObjNum 942 snap 297 seq_num 942) dirty exists 2024-09-11T03:49:19.369 INFO:tasks.rados.rados.2.smithi175.stdout:2953: finishing copy_from to smithi17557870-11 2024-09-11T03:49:19.369 INFO:tasks.rados.rados.2.smithi175.stdout:2953: got expected ENOENT (src dne) 2024-09-11T03:49:19.369 INFO:tasks.rados.rados.2.smithi175.stdout:2953: finishing copy_from racing read to smithi17557870-11 2024-09-11T03:49:19.369 INFO:tasks.rados.rados.2.smithi175.stdout:2954: finishing write tid 1 to smithi17557870-50 2024-09-11T03:49:19.369 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 25 to 321 2024-09-11T03:49:19.369 INFO:tasks.rados.rados.2.smithi175.stdout:2952: done (3 left) 2024-09-11T03:49:19.369 INFO:tasks.rados.rados.2.smithi175.stdout:2953: done (2 left) 2024-09-11T03:49:19.369 INFO:tasks.rados.rados.2.smithi175.stdout:2956: snap_create 2024-09-11T03:49:19.372 INFO:tasks.rados.rados.2.smithi175.stdout:2955: finishing rollback tid 0 to smithi17557870-25 2024-09-11T03:49:19.372 INFO:tasks.rados.rados.2.smithi175.stdout:2955: finishing rollback tid 1 to smithi17557870-25 2024-09-11T03:49:19.372 INFO:tasks.rados.rados.2.smithi175.stdout:2954: finishing write tid 2 to smithi17557870-50 2024-09-11T03:49:19.372 INFO:tasks.rados.rados.2.smithi175.stdout:2955: finishing rollback tid 2 to smithi17557870-25 2024-09-11T03:49:19.372 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 896 (ObjNum 950 snap 303 seq_num 950) dirty exists 2024-09-11T03:49:19.372 INFO:tasks.rados.rados.2.smithi175.stdout:2954: finishing write tid 3 to smithi17557870-50 2024-09-11T03:49:19.373 INFO:tasks.rados.rados.2.smithi175.stdout:2954: finishing write tid 4 to smithi17557870-50 2024-09-11T03:49:19.373 INFO:tasks.rados.rados.2.smithi175.stdout:2954: oid 50 updating version 0 to 1139 2024-09-11T03:49:19.373 INFO:tasks.rados.rados.2.smithi175.stdout:2954: oid 50 version 1139 is already newer than 1138 2024-09-11T03:49:19.373 INFO:tasks.rados.rados.2.smithi175.stdout:2954: oid 50 version 1139 is already newer than 1137 2024-09-11T03:49:19.373 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 1139 (ObjNum 1013 snap 331 seq_num 1013) dirty exists 2024-09-11T03:49:19.373 INFO:tasks.rados.rados.2.smithi175.stdout:2954: left oid 50 (ObjNum 1013 snap 331 seq_num 1013) 2024-09-11T03:49:19.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:19 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:49:19.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:19 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:49:19.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:19 smithi084 ceph-mon[27254]: 1.6 scrub starts 2024-09-11T03:49:19.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:19 smithi084 ceph-mon[27254]: 1.6 scrub ok 2024-09-11T03:49:19.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:19 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:49:19.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:19 smithi084 ceph-mon[27254]: 2.e scrub ok 2024-09-11T03:49:19.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:19 smithi084 ceph-mon[27254]: from='client.16920 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:49:19.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:19 smithi084 ceph-mon[27254]: 2.22 scrub starts 2024-09-11T03:49:19.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:19 smithi084 ceph-mon[27254]: 2.22 scrub ok 2024-09-11T03:49:19.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:19 smithi084 ceph-mon[27254]: 2.21 scrub starts 2024-09-11T03:49:19.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:19 smithi084 ceph-mon[27254]: 2.21 scrub ok 2024-09-11T03:49:19.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:19 smithi084 ceph-mon[27254]: osdmap e741: 12 total, 12 up, 12 in 2024-09-11T03:49:19.436 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:19.462 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:19.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:19.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:19.498 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1265,"stamp":"2024-09-11T03:49:18.011971+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":388970907,"num_objects":184,"num_object_clones":132,"num_object_copies":552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":9,"num_read":2772,"num_read_kb":2893827,"num_write":1881,"num_write_kb":733669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":244,"num_bytes_recovered":60090984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21160,"ondisk_log_size":21160,"up":201,"acting":201,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":1276620,"kb_used_data":819060,"kb_used_omap":18,"kb_used_meta":457261,"kb_avail":1123665204,"statfs":{"total":1151940427776,"available":1150633168896,"internally_reserved":0,"allocated":838717440,"data_stored":1031665472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":468235579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2252173,"num_objects":-6,"num_object_clones":-6,"num_object_copies":-18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":-2,"num_read":3,"num_read_kb":1,"num_write":5,"num_write_kb":4497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.999387"},"pg_stats":[{"pgid":"2.23","version":"727'416","reported_seq":3653,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.299040+0000","last_change":"2024-09-11T03:49:14.175238+0000","last_active":"2024-09-11T03:49:16.299040+0000","last_peered":"2024-09-11T03:49:16.299040+0000","last_clean":"2024-09-11T03:49:16.299040+0000","last_became_active":"2024-09-11T03:46:32.970100+0000","last_became_peered":"2024-09-11T03:46:32.970100+0000","last_unstale":"2024-09-11T03:49:16.299040+0000","last_undegraded":"2024-09-11T03:49:16.299040+0000","last_fullsized":"2024-09-11T03:49:16.299040+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"711'414","last_scrub_stamp":"2024-09-11T03:48:57.105274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:57.105274+0000","objects_scrubbed":2,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:26.337033+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016141490000000001,"stat_sum":{"num_bytes":7008882,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":31430,"num_write":18,"num_write_kb":7798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"2.22","version":"531'216","reported_seq":3439,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-09-11T03:49:17.018483+0000","last_change":"2024-09-11T03:49:14.175095+0000","last_active":"2024-09-11T03:49:17.018483+0000","last_peered":"2024-09-11T03:49:17.018483+0000","last_clean":"2024-09-11T03:49:17.018483+0000","last_became_active":"2024-09-11T03:46:32.970633+0000","last_became_peered":"2024-09-11T03:46:32.970633+0000","last_unstale":"2024-09-11T03:49:17.018483+0000","last_undegraded":"2024-09-11T03:49:17.018483+0000","last_fullsized":"2024-09-11T03:49:17.018483+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:47:49.340068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:49.340068+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001222236,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"126","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"2.21","version":"733'289","reported_seq":3598,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.881295+0000","last_change":"2024-09-11T03:49:14.175880+0000","last_active":"2024-09-11T03:49:16.881295+0000","last_peered":"2024-09-11T03:49:16.881295+0000","last_clean":"2024-09-11T03:49:16.881295+0000","last_became_active":"2024-09-11T03:46:36.024928+0000","last_became_peered":"2024-09-11T03:46:36.024928+0000","last_unstale":"2024-09-11T03:49:16.881295+0000","last_undegraded":"2024-09-11T03:49:16.881295+0000","last_fullsized":"2024-09-11T03:49:16.881295+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":6,"last_scrub":"636'265","last_scrub_stamp":"2024-09-11T03:47:48.316550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:48.316550+0000","objects_scrubbed":1,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015904859999999999,"stat_sum":{"num_bytes":6606530,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":11799,"num_write":20,"num_write_kb":8845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,8],"acting":[11,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"2.20","version":"733'229","reported_seq":3611,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.024727+0000","last_change":"2024-09-11T03:49:14.022482+0000","last_active":"2024-09-11T03:49:16.024727+0000","last_peered":"2024-09-11T03:49:16.024727+0000","last_clean":"2024-09-11T03:49:16.024727+0000","last_became_active":"2024-09-11T03:46:32.974375+0000","last_became_peered":"2024-09-11T03:46:32.974375+0000","last_unstale":"2024-09-11T03:49:16.024727+0000","last_undegraded":"2024-09-11T03:49:16.024727+0000","last_fullsized":"2024-09-11T03:49:16.024727+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"718'223","last_scrub_stamp":"2024-09-11T03:49:00.376119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.376119+0000","objects_scrubbed":2,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:04.171277+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00072309499999999999,"stat_sum":{"num_bytes":7462671,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":71530,"num_write":8,"num_write_kb":5367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"2.1c","version":"733'754","reported_seq":3976,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-09-11T03:49:17.020926+0000","last_change":"2024-09-11T03:49:14.020081+0000","last_active":"2024-09-11T03:49:17.020926+0000","last_peered":"2024-09-11T03:49:17.020926+0000","last_clean":"2024-09-11T03:49:17.020926+0000","last_became_active":"2024-09-11T03:46:32.971036+0000","last_became_peered":"2024-09-11T03:46:32.971036+0000","last_unstale":"2024-09-11T03:49:17.020926+0000","last_undegraded":"2024-09-11T03:49:17.020926+0000","last_fullsized":"2024-09-11T03:49:17.020926+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"668'709","last_scrub_stamp":"2024-09-11T03:48:06.283714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:06.283714+0000","objects_scrubbed":2,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00088264500000000004,"stat_sum":{"num_bytes":7356864,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":34,"num_read_kb":35078,"num_write":30,"num_write_kb":12013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"126","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.c","version":"20'92","reported_seq":526,"reported_epoch":737,"state":"active+clean","last_fresh":"2024-09-11T03:49:15.168961+0000","last_change":"2024-09-11T03:47:26.195120+0000","last_active":"2024-09-11T03:49:15.168961+0000","last_peered":"2024-09-11T03:49:15.168961+0000","last_clean":"2024-09-11T03:49:15.168961+0000","last_became_active":"2024-09-11T03:46:15.974786+0000","last_became_peered":"2024-09-11T03:46:15.974786+0000","last_unstale":"2024-09-11T03:49:15.168961+0000","last_undegraded":"2024-09-11T03:49:15.168961+0000","last_fullsized":"2024-09-11T03:49:15.168961+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:26.195070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:26.195070+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:45.026219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.f","version":"557'465","reported_seq":3718,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.017553+0000","last_change":"2024-09-11T03:49:14.015817+0000","last_active":"2024-09-11T03:49:16.017553+0000","last_peered":"2024-09-11T03:49:16.017553+0000","last_clean":"2024-09-11T03:49:16.017553+0000","last_became_active":"2024-09-11T03:46:27.971779+0000","last_became_peered":"2024-09-11T03:46:27.971779+0000","last_unstale":"2024-09-11T03:49:16.017553+0000","last_undegraded":"2024-09-11T03:49:16.017553+0000","last_fullsized":"2024-09-11T03:49:16.017553+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:48:59.211781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:59.211781+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:02.942040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00085012900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"2.e","version":"734'961","reported_seq":4153,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-11T03:49:13.401564+0000","last_change":"2024-09-11T03:49:12.034683+0000","last_active":"2024-09-11T03:49:13.401564+0000","last_peered":"2024-09-11T03:49:13.401564+0000","last_clean":"2024-09-11T03:49:13.401564+0000","last_became_active":"2024-09-11T03:46:36.024843+0000","last_became_peered":"2024-09-11T03:46:36.024843+0000","last_unstale":"2024-09-11T03:49:13.401564+0000","last_undegraded":"2024-09-11T03:49:13.401564+0000","last_fullsized":"2024-09-11T03:49:13.401564+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"668'907","last_scrub_stamp":"2024-09-11T03:48:06.417394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:06.417394+0000","objects_scrubbed":4,"log_size":961,"log_dups_size":0,"ondisk_log_size":961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:28.628468+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0028918500000000001,"stat_sum":{"num_bytes":19488758,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":65,"num_read_kb":66223,"num_write":46,"num_write_kb":30257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"140","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":1097,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.025223+0000","last_change":"2024-09-11T03:42:51.733829+0000","last_active":"2024-09-11T03:49:16.025223+0000","last_peered":"2024-09-11T03:49:16.025223+0000","last_clean":"2024-09-11T03:49:16.025223+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:49:16.025223+0000","last_undegraded":"2024-09-11T03:49:16.025223+0000","last_fullsized":"2024-09-11T03:49:16.025223+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:42:51.733762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:42:51.733762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:18.343088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"738'975","reported_seq":4229,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.035849+0000","last_change":"2024-09-11T03:49:14.467438+0000","last_active":"2024-09-11T03:49:16.035849+0000","last_peered":"2024-09-11T03:49:16.035849+0000","last_clean":"2024-09-11T03:49:16.035849+0000","last_became_active":"2024-09-11T03:46:27.969385+0000","last_became_peered":"2024-09-11T03:46:27.969385+0000","last_unstale":"2024-09-11T03:49:16.035849+0000","last_undegraded":"2024-09-11T03:49:16.035849+0000","last_fullsized":"2024-09-11T03:49:16.035849+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"719'918","last_scrub_stamp":"2024-09-11T03:48:59.720034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:59.720034+0000","objects_scrubbed":16,"log_size":975,"log_dups_size":0,"ondisk_log_size":975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:00.795181+0000","scrub_duration":6,"objects_trimmed":3,"snaptrim_duration":0.0039779689999999996,"stat_sum":{"num_bytes":33023838,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":195,"num_read_kb":204969,"num_write":81,"num_write_kb":29613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.e","version":"20'92","reported_seq":813,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.025239+0000","last_change":"2024-09-11T03:45:53.637950+0000","last_active":"2024-09-11T03:49:16.025239+0000","last_peered":"2024-09-11T03:49:16.025239+0000","last_clean":"2024-09-11T03:49:16.025239+0000","last_became_active":"2024-09-11T03:44:43.940022+0000","last_became_peered":"2024-09-11T03:44:43.940022+0000","last_unstale":"2024-09-11T03:49:16.025239+0000","last_undegraded":"2024-09-11T03:49:16.025239+0000","last_fullsized":"2024-09-11T03:49:16.025239+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:53.637891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:53.637891+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:20.299413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"736'796","reported_seq":4036,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.881775+0000","last_change":"2024-09-11T03:49:14.178999+0000","last_active":"2024-09-11T03:49:16.881775+0000","last_peered":"2024-09-11T03:49:16.881775+0000","last_clean":"2024-09-11T03:49:16.881775+0000","last_became_active":"2024-09-11T03:46:27.975493+0000","last_became_peered":"2024-09-11T03:46:27.975493+0000","last_unstale":"2024-09-11T03:49:16.881775+0000","last_undegraded":"2024-09-11T03:49:16.881775+0000","last_fullsized":"2024-09-11T03:49:16.881775+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"648'714","last_scrub_stamp":"2024-09-11T03:47:46.413186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:46.413186+0000","objects_scrubbed":3,"log_size":796,"log_dups_size":0,"ondisk_log_size":796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0.0043116430000000004,"stat_sum":{"num_bytes":23500970,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":46,"num_read_kb":21011,"num_write":66,"num_write_kb":24260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.f","version":"20'92","reported_seq":388,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-09-11T03:49:17.019340+0000","last_change":"2024-09-11T03:48:57.363099+0000","last_active":"2024-09-11T03:49:17.019340+0000","last_peered":"2024-09-11T03:49:17.019340+0000","last_clean":"2024-09-11T03:49:17.019340+0000","last_became_active":"2024-09-11T03:46:15.973796+0000","last_became_peered":"2024-09-11T03:46:15.973796+0000","last_unstale":"2024-09-11T03:49:17.019340+0000","last_undegraded":"2024-09-11T03:49:17.019340+0000","last_fullsized":"2024-09-11T03:49:17.019340+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:57.363027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:57.363027+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:08.181547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":1511,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.025306+0000","last_change":"2024-09-11T03:47:28.495899+0000","last_active":"2024-09-11T03:49:16.025306+0000","last_peered":"2024-09-11T03:49:16.025306+0000","last_clean":"2024-09-11T03:49:16.025306+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:49:16.025306+0000","last_undegraded":"2024-09-11T03:49:16.025306+0000","last_fullsized":"2024-09-11T03:49:16.025306+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:28.495839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:28.495839+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:45.331949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"736'465","reported_seq":3702,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.023100+0000","last_change":"2024-09-11T03:49:14.465763+0000","last_active":"2024-09-11T03:49:16.023100+0000","last_peered":"2024-09-11T03:49:16.023100+0000","last_clean":"2024-09-11T03:49:16.023100+0000","last_became_active":"2024-09-11T03:46:27.972329+0000","last_became_peered":"2024-09-11T03:46:27.972329+0000","last_unstale":"2024-09-11T03:49:16.023100+0000","last_undegraded":"2024-09-11T03:49:16.023100+0000","last_fullsized":"2024-09-11T03:49:16.023100+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"650'426","last_scrub_stamp":"2024-09-11T03:48:06.234263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:06.234263+0000","objects_scrubbed":1,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:19.016953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002296251,"stat_sum":{"num_bytes":6707656,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":3871,"num_write":21,"num_write_kb":3687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.3","version":"20'92","reported_seq":1114,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.299209+0000","last_change":"2024-09-11T03:47:25.344219+0000","last_active":"2024-09-11T03:49:16.299209+0000","last_peered":"2024-09-11T03:49:16.299209+0000","last_clean":"2024-09-11T03:49:16.299209+0000","last_became_active":"2024-09-11T03:46:15.968960+0000","last_became_peered":"2024-09-11T03:46:15.968960+0000","last_unstale":"2024-09-11T03:49:16.299209+0000","last_undegraded":"2024-09-11T03:49:16.299209+0000","last_fullsized":"2024-09-11T03:49:16.299209+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:25.344170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:25.344170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:41.126045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"563'192","reported_seq":3509,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.025090+0000","last_change":"2024-09-11T03:49:15.233227+0000","last_active":"2024-09-11T03:49:16.025090+0000","last_peered":"2024-09-11T03:49:16.025090+0000","last_clean":"2024-09-11T03:49:16.025090+0000","last_became_active":"2024-09-11T03:46:27.974405+0000","last_became_peered":"2024-09-11T03:46:27.974405+0000","last_unstale":"2024-09-11T03:49:16.025090+0000","last_undegraded":"2024-09-11T03:49:16.025090+0000","last_fullsized":"2024-09-11T03:49:16.025090+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:49:15.233165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:15.233165+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:32.425805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043589300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.2","version":"20'92","reported_seq":1065,"reported_epoch":737,"state":"active+clean","last_fresh":"2024-09-11T03:49:15.168698+0000","last_change":"2024-09-11T03:46:41.056685+0000","last_active":"2024-09-11T03:49:15.168698+0000","last_peered":"2024-09-11T03:49:15.168698+0000","last_clean":"2024-09-11T03:49:15.168698+0000","last_became_active":"2024-09-11T03:44:25.934580+0000","last_became_peered":"2024-09-11T03:44:25.934580+0000","last_unstale":"2024-09-11T03:49:15.168698+0000","last_undegraded":"2024-09-11T03:49:15.168698+0000","last_fullsized":"2024-09-11T03:49:15.168698+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:43:23.494418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:43:23.494418+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"567'235","reported_seq":3438,"reported_epoch":737,"state":"active+clean","last_fresh":"2024-09-11T03:49:15.169191+0000","last_change":"2024-09-11T03:49:14.017352+0000","last_active":"2024-09-11T03:49:15.169191+0000","last_peered":"2024-09-11T03:49:15.169191+0000","last_clean":"2024-09-11T03:49:15.169191+0000","last_became_active":"2024-09-11T03:46:36.024925+0000","last_became_peered":"2024-09-11T03:46:36.024925+0000","last_unstale":"2024-09-11T03:49:15.169191+0000","last_undegraded":"2024-09-11T03:49:15.169191+0000","last_fullsized":"2024-09-11T03:49:15.169191+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:49:01.138426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:01.138426+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:21.314691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000532508,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.1","version":"20'92","reported_seq":682,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-09-11T03:49:17.020843+0000","last_change":"2024-09-11T03:48:32.199588+0000","last_active":"2024-09-11T03:49:17.020843+0000","last_peered":"2024-09-11T03:49:17.020843+0000","last_clean":"2024-09-11T03:49:17.020843+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:49:17.020843+0000","last_undegraded":"2024-09-11T03:49:17.020843+0000","last_fullsized":"2024-09-11T03:49:17.020843+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:32.199524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:32.199524+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:00.036523+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"718'254","reported_seq":3575,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.023036+0000","last_change":"2024-09-11T03:49:15.510423+0000","last_active":"2024-09-11T03:49:16.023036+0000","last_peered":"2024-09-11T03:49:16.023036+0000","last_clean":"2024-09-11T03:49:16.023036+0000","last_became_active":"2024-09-11T03:46:27.971649+0000","last_became_peered":"2024-09-11T03:46:27.971649+0000","last_unstale":"2024-09-11T03:49:16.023036+0000","last_undegraded":"2024-09-11T03:49:16.023036+0000","last_fullsized":"2024-09-11T03:49:16.023036+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"718'254","last_scrub_stamp":"2024-09-11T03:49:15.510323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:15.510323+0000","objects_scrubbed":4,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:39.276913+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003151802,"stat_sum":{"num_bytes":9730808,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":13034,"num_write":15,"num_write_kb":11745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.7","version":"20'92","reported_seq":372,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-09-11T03:49:17.018655+0000","last_change":"2024-09-11T03:46:15.972170+0000","last_active":"2024-09-11T03:49:17.018655+0000","last_peered":"2024-09-11T03:49:17.018655+0000","last_clean":"2024-09-11T03:49:17.018655+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:49:17.018655+0000","last_undegraded":"2024-09-11T03:49:17.018655+0000","last_fullsized":"2024-09-11T03:49:17.018655+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:12.360829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:12.360829+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:35.334156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"736'602","reported_seq":3889,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.302707+0000","last_change":"2024-09-11T03:49:16.302707+0000","last_active":"2024-09-11T03:49:16.302707+0000","last_peered":"2024-09-11T03:49:16.302707+0000","last_clean":"2024-09-11T03:49:16.302707+0000","last_became_active":"2024-09-11T03:44:10.921135+0000","last_became_peered":"2024-09-11T03:44:10.921135+0000","last_unstale":"2024-09-11T03:49:16.302707+0000","last_undegraded":"2024-09-11T03:49:16.302707+0000","last_fullsized":"2024-09-11T03:49:16.302707+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"736'602","last_scrub_stamp":"2024-09-11T03:49:16.302642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:16.302642+0000","objects_scrubbed":13,"log_size":602,"log_dups_size":0,"ondisk_log_size":602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:20.290434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0044198129999999999,"stat_sum":{"num_bytes":33444488,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":194,"num_read_kb":181247,"num_write":152,"num_write_kb":58288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":4118225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.6","version":"20'92","reported_seq":685,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-09-11T03:49:17.020941+0000","last_change":"2024-09-11T03:48:10.301559+0000","last_active":"2024-09-11T03:49:17.020941+0000","last_peered":"2024-09-11T03:49:17.020941+0000","last_clean":"2024-09-11T03:49:17.020941+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:49:17.020941+0000","last_undegraded":"2024-09-11T03:49:17.020941+0000","last_fullsized":"2024-09-11T03:49:17.020941+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:10.301516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:10.301516+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"736'1099","reported_seq":4388,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.017641+0000","last_change":"2024-09-11T03:49:14.018170+0000","last_active":"2024-09-11T03:49:16.017641+0000","last_peered":"2024-09-11T03:49:16.017641+0000","last_clean":"2024-09-11T03:49:16.017641+0000","last_became_active":"2024-09-11T03:41:33.866316+0000","last_became_peered":"2024-09-11T03:41:33.866316+0000","last_unstale":"2024-09-11T03:49:16.017641+0000","last_undegraded":"2024-09-11T03:49:16.017641+0000","last_fullsized":"2024-09-11T03:49:16.017641+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"690'981","last_scrub_stamp":"2024-09-11T03:48:29.034446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:29.034446+0000","objects_scrubbed":12,"log_size":1099,"log_dups_size":0,"ondisk_log_size":1099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:30.173892+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0034054229999999999,"stat_sum":{"num_bytes":36748713,"num_objects":20,"num_object_clones":15,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":319,"num_read_kb":333909,"num_write":230,"num_write_kb":86113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.5","version":"20'92","reported_seq":684,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-09-11T03:49:17.020894+0000","last_change":"2024-09-11T03:46:17.255120+0000","last_active":"2024-09-11T03:49:17.020894+0000","last_peered":"2024-09-11T03:49:17.020894+0000","last_clean":"2024-09-11T03:49:17.020894+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:49:17.020894+0000","last_undegraded":"2024-09-11T03:49:17.020894+0000","last_fullsized":"2024-09-11T03:49:17.020894+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:17.255073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:17.255073+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:35.234526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":3379,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.299072+0000","last_change":"2024-09-11T03:49:14.174154+0000","last_active":"2024-09-11T03:49:16.299072+0000","last_peered":"2024-09-11T03:49:16.299072+0000","last_clean":"2024-09-11T03:49:16.299072+0000","last_became_active":"2024-09-11T03:41:33.865499+0000","last_became_peered":"2024-09-11T03:41:33.865499+0000","last_unstale":"2024-09-11T03:49:16.299072+0000","last_undegraded":"2024-09-11T03:49:16.299072+0000","last_fullsized":"2024-09-11T03:49:16.299072+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:48:30.091010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:30.091010+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:58.359610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00056147300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.4","version":"20'92","reported_seq":1283,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.881690+0000","last_change":"2024-09-11T03:48:05.173060+0000","last_active":"2024-09-11T03:49:16.881690+0000","last_peered":"2024-09-11T03:49:16.881690+0000","last_clean":"2024-09-11T03:49:16.881690+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:49:16.881690+0000","last_undegraded":"2024-09-11T03:49:16.881690+0000","last_fullsized":"2024-09-11T03:49:16.881690+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:05.172997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:05.172997+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"739'1046","reported_seq":4405,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-09-11T03:49:17.019140+0000","last_change":"2024-09-11T03:49:14.178358+0000","last_active":"2024-09-11T03:49:17.019140+0000","last_peered":"2024-09-11T03:49:17.019140+0000","last_clean":"2024-09-11T03:49:17.019140+0000","last_became_active":"2024-09-11T03:46:19.004997+0000","last_became_peered":"2024-09-11T03:46:19.004997+0000","last_unstale":"2024-09-11T03:49:17.019140+0000","last_undegraded":"2024-09-11T03:49:17.019140+0000","last_fullsized":"2024-09-11T03:49:17.019140+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"690'984","last_scrub_stamp":"2024-09-11T03:48:30.353823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:30.353823+0000","objects_scrubbed":3,"log_size":1046,"log_dups_size":0,"ondisk_log_size":1046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:35.144635+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0044676919999999997,"stat_sum":{"num_bytes":20337823,"num_objects":9,"num_object_clones":4,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":400,"num_read_kb":467112,"num_write":227,"num_write_kb":84864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":25429065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"126","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.b","version":"20'92","reported_seq":1085,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.881679+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:49:16.881679+0000","last_peered":"2024-09-11T03:49:16.881679+0000","last_clean":"2024-09-11T03:49:16.881679+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:49:16.881679+0000","last_undegraded":"2024-09-11T03:49:16.881679+0000","last_fullsized":"2024-09-11T03:49:16.881679+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:13.961107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"733'364","reported_seq":3618,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.025245+0000","last_change":"2024-09-11T03:49:14.022719+0000","last_active":"2024-09-11T03:49:16.025245+0000","last_peered":"2024-09-11T03:49:16.025245+0000","last_clean":"2024-09-11T03:49:16.025245+0000","last_became_active":"2024-09-11T03:41:33.863875+0000","last_became_peered":"2024-09-11T03:41:33.863875+0000","last_unstale":"2024-09-11T03:49:16.025245+0000","last_undegraded":"2024-09-11T03:49:16.025245+0000","last_fullsized":"2024-09-11T03:49:16.025245+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'358","last_scrub_stamp":"2024-09-11T03:48:50.332058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:50.332058+0000","objects_scrubbed":4,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:10.037024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00059281799999999995,"stat_sum":{"num_bytes":4553575,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":95410,"num_write":52,"num_write_kb":13779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.a","version":"20'92","reported_seq":1430,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-11T03:49:13.401042+0000","last_change":"2024-09-11T03:48:59.762212+0000","last_active":"2024-09-11T03:49:13.401042+0000","last_peered":"2024-09-11T03:49:13.401042+0000","last_clean":"2024-09-11T03:49:13.401042+0000","last_became_active":"2024-09-11T03:46:15.973271+0000","last_became_peered":"2024-09-11T03:46:15.973271+0000","last_unstale":"2024-09-11T03:49:13.401042+0000","last_undegraded":"2024-09-11T03:49:13.401042+0000","last_fullsized":"2024-09-11T03:49:13.401042+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:59.762145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:59.762145+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:02.580429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"736'1131","reported_seq":4472,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.298956+0000","last_change":"2024-09-11T03:49:14.177501+0000","last_active":"2024-09-11T03:49:16.298956+0000","last_peered":"2024-09-11T03:49:16.298956+0000","last_clean":"2024-09-11T03:49:16.298956+0000","last_became_active":"2024-09-11T03:41:33.865638+0000","last_became_peered":"2024-09-11T03:41:33.865638+0000","last_unstale":"2024-09-11T03:49:16.298956+0000","last_undegraded":"2024-09-11T03:49:16.298956+0000","last_fullsized":"2024-09-11T03:49:16.298956+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"662'966","last_scrub_stamp":"2024-09-11T03:48:00.178735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:00.178735+0000","objects_scrubbed":7,"log_size":1131,"log_dups_size":0,"ondisk_log_size":1131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.004036703,"stat_sum":{"num_bytes":39636347,"num_objects":18,"num_object_clones":13,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":442,"num_read_kb":447273,"num_write":279,"num_write_kb":101250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.9","version":"20'92","reported_seq":1499,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.024766+0000","last_change":"2024-09-11T03:47:22.529885+0000","last_active":"2024-09-11T03:49:16.024766+0000","last_peered":"2024-09-11T03:49:16.024766+0000","last_clean":"2024-09-11T03:49:16.024766+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:49:16.024766+0000","last_undegraded":"2024-09-11T03:49:16.024766+0000","last_fullsized":"2024-09-11T03:49:16.024766+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:22.529805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:22.529805+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:34.334076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"727'348","reported_seq":3591,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.881339+0000","last_change":"2024-09-11T03:49:14.175420+0000","last_active":"2024-09-11T03:49:16.881339+0000","last_peered":"2024-09-11T03:49:16.881339+0000","last_clean":"2024-09-11T03:49:16.881339+0000","last_became_active":"2024-09-11T03:46:27.968719+0000","last_became_peered":"2024-09-11T03:46:27.968719+0000","last_unstale":"2024-09-11T03:49:16.881339+0000","last_undegraded":"2024-09-11T03:49:16.881339+0000","last_fullsized":"2024-09-11T03:49:16.881339+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"653'312","last_scrub_stamp":"2024-09-11T03:48:03.157506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:03.157506+0000","objects_scrubbed":2,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011792549999999999,"stat_sum":{"num_bytes":5429057,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":25,"num_write_kb":11102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.8","version":"20'92","reported_seq":1158,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.023106+0000","last_change":"2024-09-11T03:46:20.861683+0000","last_active":"2024-09-11T03:49:16.023106+0000","last_peered":"2024-09-11T03:49:16.023106+0000","last_clean":"2024-09-11T03:49:16.023106+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:49:16.023106+0000","last_undegraded":"2024-09-11T03:49:16.023106+0000","last_fullsized":"2024-09-11T03:49:16.023106+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:20.861635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:20.861635+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:33.947969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"568'646","reported_seq":3884,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.017459+0000","last_change":"2024-09-11T03:49:14.015534+0000","last_active":"2024-09-11T03:49:16.017459+0000","last_peered":"2024-09-11T03:49:16.017459+0000","last_clean":"2024-09-11T03:49:16.017459+0000","last_became_active":"2024-09-11T03:46:27.974031+0000","last_became_peered":"2024-09-11T03:46:27.974031+0000","last_unstale":"2024-09-11T03:49:16.017459+0000","last_undegraded":"2024-09-11T03:49:16.017459+0000","last_fullsized":"2024-09-11T03:49:16.017459+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:47:46.957311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:46.957311+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092814099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.10","version":"20'92","reported_seq":1115,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.023052+0000","last_change":"2024-09-11T03:49:00.710015+0000","last_active":"2024-09-11T03:49:16.023052+0000","last_peered":"2024-09-11T03:49:16.023052+0000","last_clean":"2024-09-11T03:49:16.023052+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:49:16.023052+0000","last_undegraded":"2024-09-11T03:49:16.023052+0000","last_fullsized":"2024-09-11T03:49:16.023052+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:00.709966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.709966+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:30.175623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":3293,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-11T03:49:13.401285+0000","last_change":"2024-09-11T03:49:12.032986+0000","last_active":"2024-09-11T03:49:13.401285+0000","last_peered":"2024-09-11T03:49:13.401285+0000","last_clean":"2024-09-11T03:49:13.401285+0000","last_became_active":"2024-09-11T03:41:38.886834+0000","last_became_peered":"2024-09-11T03:41:38.886834+0000","last_unstale":"2024-09-11T03:49:13.401285+0000","last_undegraded":"2024-09-11T03:49:13.401285+0000","last_fullsized":"2024-09-11T03:49:13.401285+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:47:50.277248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:47:50.277248+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001258924,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"140","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":1090,"reported_epoch":737,"state":"active+clean","last_fresh":"2024-09-11T03:49:15.169071+0000","last_change":"2024-09-11T03:48:30.121620+0000","last_active":"2024-09-11T03:49:15.169071+0000","last_peered":"2024-09-11T03:49:15.169071+0000","last_clean":"2024-09-11T03:49:15.169071+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:49:15.169071+0000","last_undegraded":"2024-09-11T03:49:15.169071+0000","last_fullsized":"2024-09-11T03:49:15.169071+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:30.121571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:30.121571+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:39.567385+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"736'523","reported_seq":3768,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-09-11T03:49:17.021141+0000","last_change":"2024-09-11T03:49:14.022179+0000","last_active":"2024-09-11T03:49:17.021141+0000","last_peered":"2024-09-11T03:49:17.021141+0000","last_clean":"2024-09-11T03:49:17.021141+0000","last_became_active":"2024-09-11T03:43:55.918817+0000","last_became_peered":"2024-09-11T03:43:55.918817+0000","last_unstale":"2024-09-11T03:49:17.021141+0000","last_undegraded":"2024-09-11T03:49:17.021141+0000","last_fullsized":"2024-09-11T03:49:17.021141+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"663'468","last_scrub_stamp":"2024-09-11T03:48:03.279952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:03.279952+0000","objects_scrubbed":4,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:18.120608+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0025897730000000001,"stat_sum":{"num_bytes":18217174,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":200,"num_read_kb":210617,"num_write":139,"num_write_kb":52245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":15216236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"126","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.12","version":"20'92","reported_seq":1096,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.022882+0000","last_change":"2024-09-11T03:47:23.339646+0000","last_active":"2024-09-11T03:49:16.022882+0000","last_peered":"2024-09-11T03:49:16.022882+0000","last_clean":"2024-09-11T03:49:16.022882+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:49:16.022882+0000","last_undegraded":"2024-09-11T03:49:16.022882+0000","last_fullsized":"2024-09-11T03:49:16.022882+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:23.339595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:23.339595+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:48.807974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":2381,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-09-11T03:49:17.018861+0000","last_change":"2024-09-11T03:49:14.174425+0000","last_active":"2024-09-11T03:49:17.018861+0000","last_peered":"2024-09-11T03:49:17.018861+0000","last_clean":"2024-09-11T03:49:17.018861+0000","last_became_active":"2024-09-11T03:46:19.005117+0000","last_became_peered":"2024-09-11T03:46:19.005117+0000","last_unstale":"2024-09-11T03:49:17.018861+0000","last_undegraded":"2024-09-11T03:49:17.018861+0000","last_fullsized":"2024-09-11T03:49:17.018861+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:48:32.341279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:32.341279+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:38.330003+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00068696299999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"126","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.13","version":"20'92","reported_seq":1200,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.299104+0000","last_change":"2024-09-11T03:47:18.217600+0000","last_active":"2024-09-11T03:49:16.299104+0000","last_peered":"2024-09-11T03:49:16.299104+0000","last_clean":"2024-09-11T03:49:16.299104+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:49:16.299104+0000","last_undegraded":"2024-09-11T03:49:16.299104+0000","last_fullsized":"2024-09-11T03:49:16.299104+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:18.217555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:18.217555+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:43.129073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":2322,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.017342+0000","last_change":"2024-09-11T03:49:14.015227+0000","last_active":"2024-09-11T03:49:16.017342+0000","last_peered":"2024-09-11T03:49:16.017342+0000","last_clean":"2024-09-11T03:49:16.017342+0000","last_became_active":"2024-09-11T03:41:35.874295+0000","last_became_peered":"2024-09-11T03:41:35.874295+0000","last_unstale":"2024-09-11T03:49:16.017342+0000","last_undegraded":"2024-09-11T03:49:16.017342+0000","last_fullsized":"2024-09-11T03:49:16.017342+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:48:32.029810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:32.029810+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:47.951353+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00070804500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.14","version":"20'92","reported_seq":675,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.025093+0000","last_change":"2024-09-11T03:48:57.320199+0000","last_active":"2024-09-11T03:49:16.025093+0000","last_peered":"2024-09-11T03:49:16.025093+0000","last_clean":"2024-09-11T03:49:16.025093+0000","last_became_active":"2024-09-11T03:46:36.026459+0000","last_became_peered":"2024-09-11T03:46:36.026459+0000","last_unstale":"2024-09-11T03:49:16.025093+0000","last_undegraded":"2024-09-11T03:49:16.025093+0000","last_fullsized":"2024-09-11T03:49:16.025093+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:57.320159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:57.320159+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:16.384062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":2877,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-09-11T03:49:17.018973+0000","last_change":"2024-09-11T03:49:14.177316+0000","last_active":"2024-09-11T03:49:17.018973+0000","last_peered":"2024-09-11T03:49:17.018973+0000","last_clean":"2024-09-11T03:49:17.018973+0000","last_became_active":"2024-09-11T03:46:19.003959+0000","last_became_peered":"2024-09-11T03:46:19.003959+0000","last_unstale":"2024-09-11T03:49:17.018973+0000","last_undegraded":"2024-09-11T03:49:17.018973+0000","last_fullsized":"2024-09-11T03:49:17.018973+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:48:28.306771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:28.306771+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:34.871765+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031285190000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"126","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.15","version":"20'92","reported_seq":1069,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.025186+0000","last_change":"2024-09-11T03:47:47.459856+0000","last_active":"2024-09-11T03:49:16.025186+0000","last_peered":"2024-09-11T03:49:16.025186+0000","last_clean":"2024-09-11T03:49:16.025186+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:49:16.025186+0000","last_undegraded":"2024-09-11T03:49:16.025186+0000","last_fullsized":"2024-09-11T03:49:16.025186+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:47.459803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:47.459803+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:08.322414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"733'508","reported_seq":3785,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.023113+0000","last_change":"2024-09-11T03:49:14.466210+0000","last_active":"2024-09-11T03:49:16.023113+0000","last_peered":"2024-09-11T03:49:16.023113+0000","last_clean":"2024-09-11T03:49:16.023113+0000","last_became_active":"2024-09-11T03:46:19.007019+0000","last_became_peered":"2024-09-11T03:46:19.007019+0000","last_unstale":"2024-09-11T03:49:16.023113+0000","last_undegraded":"2024-09-11T03:49:16.023113+0000","last_fullsized":"2024-09-11T03:49:16.023113+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"668'459","last_scrub_stamp":"2024-09-11T03:48:08.217400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:08.217400+0000","objects_scrubbed":5,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:31.375843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027581820000000001,"stat_sum":{"num_bytes":20236174,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":99,"num_read_kb":80157,"num_write":102,"num_write_kb":43311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":5263193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.16","version":"20'92","reported_seq":541,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.881518+0000","last_change":"2024-09-11T03:47:50.266422+0000","last_active":"2024-09-11T03:49:16.881518+0000","last_peered":"2024-09-11T03:49:16.881518+0000","last_clean":"2024-09-11T03:49:16.881518+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:49:16.881518+0000","last_undegraded":"2024-09-11T03:49:16.881518+0000","last_fullsized":"2024-09-11T03:49:16.881518+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:50.266374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:50.266374+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":3513,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-11T03:49:13.401286+0000","last_change":"2024-09-11T03:49:12.033008+0000","last_active":"2024-09-11T03:49:13.401286+0000","last_peered":"2024-09-11T03:49:13.401286+0000","last_clean":"2024-09-11T03:49:13.401286+0000","last_became_active":"2024-09-11T03:41:38.888228+0000","last_became_peered":"2024-09-11T03:41:38.888228+0000","last_unstale":"2024-09-11T03:49:13.401286+0000","last_undegraded":"2024-09-11T03:49:13.401286+0000","last_fullsized":"2024-09-11T03:49:13.401286+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:48:01.476240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:01.476240+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012805519999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"140","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":1088,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.017430+0000","last_change":"2024-09-11T03:47:29.086383+0000","last_active":"2024-09-11T03:49:16.017430+0000","last_peered":"2024-09-11T03:49:16.017430+0000","last_clean":"2024-09-11T03:49:16.017430+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:49:16.017430+0000","last_undegraded":"2024-09-11T03:49:16.017430+0000","last_fullsized":"2024-09-11T03:49:16.017430+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:29.086335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:29.086335+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:42.049323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"733'325","reported_seq":3483,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-09-11T03:49:17.018731+0000","last_change":"2024-09-11T03:49:14.177023+0000","last_active":"2024-09-11T03:49:17.018731+0000","last_peered":"2024-09-11T03:49:17.018731+0000","last_clean":"2024-09-11T03:49:17.018731+0000","last_became_active":"2024-09-11T03:46:19.004558+0000","last_became_peered":"2024-09-11T03:46:19.004558+0000","last_unstale":"2024-09-11T03:49:17.018731+0000","last_undegraded":"2024-09-11T03:49:17.018731+0000","last_fullsized":"2024-09-11T03:49:17.018731+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"711'317","last_scrub_stamp":"2024-09-11T03:48:50.287601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:50.287601+0000","objects_scrubbed":5,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:03.994212+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0028442049999999998,"stat_sum":{"num_bytes":11060696,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":27600,"num_write":57,"num_write_kb":17105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"126","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.18","version":"20'92","reported_seq":1200,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.022837+0000","last_change":"2024-09-11T03:48:53.702605+0000","last_active":"2024-09-11T03:49:16.022837+0000","last_peered":"2024-09-11T03:49:16.022837+0000","last_clean":"2024-09-11T03:49:16.022837+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:49:16.022837+0000","last_undegraded":"2024-09-11T03:49:16.022837+0000","last_fullsized":"2024-09-11T03:49:16.022837+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:53.702523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:53.702523+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:55.160792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"736'765","reported_seq":4126,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.881460+0000","last_change":"2024-09-11T03:49:14.177930+0000","last_active":"2024-09-11T03:49:16.881460+0000","last_peered":"2024-09-11T03:49:16.881460+0000","last_clean":"2024-09-11T03:49:16.881460+0000","last_became_active":"2024-09-11T03:46:32.975567+0000","last_became_peered":"2024-09-11T03:46:32.975567+0000","last_unstale":"2024-09-11T03:49:16.881460+0000","last_undegraded":"2024-09-11T03:49:16.881460+0000","last_fullsized":"2024-09-11T03:49:16.881460+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"661'708","last_scrub_stamp":"2024-09-11T03:48:04.159386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:04.159386+0000","objects_scrubbed":3,"log_size":765,"log_dups_size":0,"ondisk_log_size":765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:32.674661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0037353780000000001,"stat_sum":{"num_bytes":15641767,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":84,"num_read_kb":90520,"num_write":34,"num_write_kb":18770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7],"acting":[11,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.19","version":"20'92","reported_seq":697,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-09-11T03:49:17.020791+0000","last_change":"2024-09-11T03:48:56.129575+0000","last_active":"2024-09-11T03:49:17.020791+0000","last_peered":"2024-09-11T03:49:17.020791+0000","last_clean":"2024-09-11T03:49:17.020791+0000","last_became_active":"2024-09-11T03:46:15.972841+0000","last_became_peered":"2024-09-11T03:46:15.972841+0000","last_unstale":"2024-09-11T03:49:17.020791+0000","last_undegraded":"2024-09-11T03:49:17.020791+0000","last_fullsized":"2024-09-11T03:49:17.020791+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:56.129516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:56.129516+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:24.133914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"557'273","reported_seq":3451,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-09-11T03:49:17.019172+0000","last_change":"2024-09-11T03:49:14.176730+0000","last_active":"2024-09-11T03:49:17.019172+0000","last_peered":"2024-09-11T03:49:17.019172+0000","last_clean":"2024-09-11T03:49:17.019172+0000","last_became_active":"2024-09-11T03:46:32.971003+0000","last_became_peered":"2024-09-11T03:46:32.971003+0000","last_unstale":"2024-09-11T03:49:17.019172+0000","last_undegraded":"2024-09-11T03:49:17.019172+0000","last_fullsized":"2024-09-11T03:49:17.019172+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:48:04.318702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:04.318702+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:32.340183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026632140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,2],"acting":[6,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"126","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.1a","version":"20'92","reported_seq":812,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.298880+0000","last_change":"2024-09-11T03:45:52.914136+0000","last_active":"2024-09-11T03:49:16.298880+0000","last_peered":"2024-09-11T03:49:16.298880+0000","last_clean":"2024-09-11T03:49:16.298880+0000","last_became_active":"2024-09-11T03:44:43.938873+0000","last_became_peered":"2024-09-11T03:44:43.938873+0000","last_unstale":"2024-09-11T03:49:16.298880+0000","last_undegraded":"2024-09-11T03:49:16.298880+0000","last_fullsized":"2024-09-11T03:49:16.298880+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:52.914088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:52.914088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:11.333208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"736'546","reported_seq":3863,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-09-11T03:49:17.018182+0000","last_change":"2024-09-11T03:49:14.176331+0000","last_active":"2024-09-11T03:49:17.018182+0000","last_peered":"2024-09-11T03:49:17.018182+0000","last_clean":"2024-09-11T03:49:17.018182+0000","last_became_active":"2024-09-11T03:46:19.002002+0000","last_became_peered":"2024-09-11T03:46:19.002002+0000","last_unstale":"2024-09-11T03:49:17.018182+0000","last_undegraded":"2024-09-11T03:49:17.018182+0000","last_fullsized":"2024-09-11T03:49:17.018182+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"668'500","last_scrub_stamp":"2024-09-11T03:48:06.342113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:06.342113+0000","objects_scrubbed":2,"log_size":546,"log_dups_size":0,"ondisk_log_size":546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0026416930000000001,"stat_sum":{"num_bytes":6185278,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":92,"num_read_kb":91932,"num_write":57,"num_write_kb":26464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4870845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"126","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.1b","version":"20'92","reported_seq":1065,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.017278+0000","last_change":"2024-09-11T03:48:31.033896+0000","last_active":"2024-09-11T03:49:16.017278+0000","last_peered":"2024-09-11T03:49:16.017278+0000","last_clean":"2024-09-11T03:49:16.017278+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:49:16.017278+0000","last_undegraded":"2024-09-11T03:49:16.017278+0000","last_fullsized":"2024-09-11T03:49:16.017278+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:31.033837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:31.033837+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:39.213887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"733'478","reported_seq":3776,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-11T03:49:13.401174+0000","last_change":"2024-09-11T03:49:12.033422+0000","last_active":"2024-09-11T03:49:13.401174+0000","last_peered":"2024-09-11T03:49:13.401174+0000","last_clean":"2024-09-11T03:49:13.401174+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:49:13.401174+0000","last_undegraded":"2024-09-11T03:49:13.401174+0000","last_fullsized":"2024-09-11T03:49:13.401174+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"711'462","last_scrub_stamp":"2024-09-11T03:48:49.842909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:49.842909+0000","objects_scrubbed":9,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:53.645887+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001337714,"stat_sum":{"num_bytes":19211754,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":182,"num_read_kb":220025,"num_write":86,"num_write_kb":37963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4865740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"140","length":"1"}]},{"pgid":"2.1f","version":"733'613","reported_seq":3807,"reported_epoch":737,"state":"active+clean","last_fresh":"2024-09-11T03:49:15.169244+0000","last_change":"2024-09-11T03:49:14.018047+0000","last_active":"2024-09-11T03:49:15.169244+0000","last_peered":"2024-09-11T03:49:15.169244+0000","last_clean":"2024-09-11T03:49:15.169244+0000","last_became_active":"2024-09-11T03:46:32.968696+0000","last_became_peered":"2024-09-11T03:46:32.968696+0000","last_unstale":"2024-09-11T03:49:15.169244+0000","last_undegraded":"2024-09-11T03:49:15.169244+0000","last_fullsized":"2024-09-11T03:49:15.169244+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"648'520","last_scrub_stamp":"2024-09-11T03:48:01.219639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:01.219639+0000","objects_scrubbed":2,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:19.353188+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001051209,"stat_sum":{"num_bytes":21197343,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":114,"num_read_kb":137132,"num_write":60,"num_write_kb":21399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.1c","version":"20'92","reported_seq":1064,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.017386+0000","last_change":"2024-09-11T03:46:20.752592+0000","last_active":"2024-09-11T03:49:16.017386+0000","last_peered":"2024-09-11T03:49:16.017386+0000","last_clean":"2024-09-11T03:49:16.017386+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:49:16.017386+0000","last_undegraded":"2024-09-11T03:49:16.017386+0000","last_fullsized":"2024-09-11T03:49:16.017386+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:46:20.752526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:46:20.752526+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:39.076533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1e","version":"736'1001","reported_seq":4205,"reported_epoch":737,"state":"active+clean","last_fresh":"2024-09-11T03:49:15.168914+0000","last_change":"2024-09-11T03:49:14.019748+0000","last_active":"2024-09-11T03:49:15.168914+0000","last_peered":"2024-09-11T03:49:15.168914+0000","last_clean":"2024-09-11T03:49:15.168914+0000","last_became_active":"2024-09-11T03:46:32.970637+0000","last_became_peered":"2024-09-11T03:46:32.970637+0000","last_unstale":"2024-09-11T03:49:15.168914+0000","last_undegraded":"2024-09-11T03:49:15.168914+0000","last_fullsized":"2024-09-11T03:49:15.168914+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"718'982","last_scrub_stamp":"2024-09-11T03:48:56.189316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:56.189316+0000","objects_scrubbed":9,"log_size":1001,"log_dups_size":0,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:56.978430+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0028591020000000001,"stat_sum":{"num_bytes":15593373,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":54,"num_read_kb":51944,"num_write":76,"num_write_kb":27431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"126","length":"1"},{"start":"137","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.1d","version":"20'92","reported_seq":1054,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.298998+0000","last_change":"2024-09-11T03:47:22.312507+0000","last_active":"2024-09-11T03:49:16.298998+0000","last_peered":"2024-09-11T03:49:16.298998+0000","last_clean":"2024-09-11T03:49:16.298998+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:49:16.298998+0000","last_undegraded":"2024-09-11T03:49:16.298998+0000","last_fullsized":"2024-09-11T03:49:16.298998+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:22.312456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:22.312456+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:45.114530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"567'752","reported_seq":3952,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-09-11T03:49:17.020997+0000","last_change":"2024-09-11T03:49:14.019999+0000","last_active":"2024-09-11T03:49:17.020997+0000","last_peered":"2024-09-11T03:49:17.020997+0000","last_clean":"2024-09-11T03:49:17.020997+0000","last_became_active":"2024-09-11T03:46:32.972371+0000","last_became_peered":"2024-09-11T03:46:32.972371+0000","last_unstale":"2024-09-11T03:49:17.020997+0000","last_undegraded":"2024-09-11T03:49:17.020997+0000","last_fullsized":"2024-09-11T03:49:17.020997+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:48:55.139430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:55.139430+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:17.188944+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000682951,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,11],"acting":[10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"126","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"1.1e","version":"20'92","reported_seq":1074,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-09-11T03:49:16.881613+0000","last_change":"2024-09-11T03:48:31.346487+0000","last_active":"2024-09-11T03:49:16.881613+0000","last_peered":"2024-09-11T03:49:16.881613+0000","last_clean":"2024-09-11T03:49:16.881613+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:49:16.881613+0000","last_undegraded":"2024-09-11T03:49:16.881613+0000","last_fullsized":"2024-09-11T03:49:16.881613+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:31.346410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:31.346410+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:34.007357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":36,"stat_sum":{"num_bytes":388380539,"num_objects":182,"num_object_clones":132,"num_object_copies":546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":9,"num_read":2772,"num_read_kb":2893827,"num_write":1881,"num_write_kb":733669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":59763304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":730165248,"data_stored":960621652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18308,"ondisk_log_size":18308,"up":108,"acting":108,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285713,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76332,"kb_used_data":49300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93668820,"statfs":{"total":95995035648,"available":95916871680,"internally_reserved":0,"allocated":50483200,"data_stored":63135799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514645,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":106820,"kb_used_data":67304,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93638332,"statfs":{"total":95995035648,"available":95885651968,"internally_reserved":0,"allocated":68919296,"data_stored":76025875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743578,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111220,"kb_used_data":57816,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93633932,"statfs":{"total":95995035648,"available":95881146368,"internally_reserved":0,"allocated":59203584,"data_stored":77890131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037922,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83048,"kb_used_data":51856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93662104,"statfs":{"total":95995035648,"available":95909994496,"internally_reserved":0,"allocated":53100544,"data_stored":68504124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299557,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":153836,"kb_used_data":96280,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93591316,"statfs":{"total":95995035648,"available":95837507584,"internally_reserved":0,"allocated":98590720,"data_stored":122743679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528490,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95560,"kb_used_data":56052,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93649592,"statfs":{"total":95995035648,"available":95897182208,"internally_reserved":0,"allocated":57397248,"data_stored":70412427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757424,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88892,"kb_used_data":61852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93656260,"statfs":{"total":95995035648,"available":95904010240,"internally_reserved":0,"allocated":63336448,"data_stored":80402553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986357,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90048,"kb_used_data":58864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93655104,"statfs":{"total":95995035648,"available":95902826496,"internally_reserved":0,"allocated":60276736,"data_stored":80050385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":163628,"kb_used_data":106064,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93581524,"statfs":{"total":95995035648,"available":95827480576,"internally_reserved":0,"allocated":108609536,"data_stored":135010771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411519,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74152,"kb_used_data":42964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93671000,"statfs":{"total":95995035648,"available":95919104000,"internally_reserved":0,"allocated":43995136,"data_stored":47837220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575045,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107740,"kb_used_data":76552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93637412,"statfs":{"total":95995035648,"available":95884709888,"internally_reserved":0,"allocated":78389248,"data_stored":88860352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738570,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":125344,"kb_used_data":94156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93619808,"statfs":{"total":95995035648,"available":95866683392,"internally_reserved":0,"allocated":96415744,"data_stored":120792156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":87171072,"data_stored":114673632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69472256,"data_stored":83069508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34910208,"data_stored":41869877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":99401728,"data_stored":128916996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":51372032,"data_stored":74267772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":54202368,"data_stored":74402230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":48140288,"data_stored":64285672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":89456640,"data_stored":116743356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":44171264,"data_stored":62705049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":50323456,"data_stored":72124036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":59990016,"data_stored":70226800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":41553920,"data_stored":57336724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:49:19.501 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-11T03:49:19.502 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 46 2024-09-11T03:49:19.563 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:19.585 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:19.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:19.586 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:19.648 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:49:19.686 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:19.714 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:19.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:19.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:19.747 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:19.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:19.840 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:19.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:19.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:19.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:19.968 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:19.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:19.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:19.985 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-11T03:49:20.069 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:20.083 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:49:20.086 INFO:tasks.rados.rados.2.smithi175.stdout:2954: done (2 left) 2024-09-11T03:49:20.086 INFO:tasks.rados.rados.2.smithi175.stdout:2955: done (1 left) 2024-09-11T03:49:20.086 INFO:tasks.rados.rados.2.smithi175.stdout:2956: done (0 left) 2024-09-11T03:49:20.086 INFO:tasks.rados.rados.2.smithi175.stdout:2957: delete oid 14 current snap is 332 2024-09-11T03:49:20.091 INFO:tasks.rados.rados.2.smithi175.stdout:2957: done (0 left) 2024-09-11T03:49:20.091 INFO:tasks.rados.rados.2.smithi175.stdout:2958: read oid 46 snap -1 2024-09-11T03:49:20.091 INFO:tasks.rados.rados.2.smithi175.stdout:2958: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:49:20.091 INFO:tasks.rados.rados.2.smithi175.stdout:2959: delete oid 3 current snap is 332 2024-09-11T03:49:20.093 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:20.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:20.094 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:20.097 INFO:tasks.rados.rados.2.smithi175.stdout:2959: done (1 left) 2024-09-11T03:49:20.097 INFO:tasks.rados.rados.2.smithi175.stdout:2960: delete oid 29 current snap is 332 2024-09-11T03:49:20.099 INFO:tasks.rados.rados.2.smithi175.stdout:2958: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:49:20.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:20.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:20.221 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:20.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:20 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:49:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:20 smithi175 ceph-mon[30894]: 2.b scrub ok 2024-09-11T03:49:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:20 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:49:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:20 smithi175 ceph-mon[30894]: 2.9 scrub ok 2024-09-11T03:49:20.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:20 smithi175 ceph-mon[30894]: pgmap v1266: 67 pgs: 67 active+clean; 371 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.1 MiB/s wr, 1 op/s 2024-09-11T03:49:20.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:20 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3358045025' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:49:20.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:20 smithi175 ceph-mon[30894]: 2.19 scrub starts 2024-09-11T03:49:20.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:20 smithi175 ceph-mon[30894]: 2.19 scrub ok 2024-09-11T03:49:20.251 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:49:20.323 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:20.344 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:20.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:20.345 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:20.356 INFO:tasks.rados.rados.2.smithi175.stdout:2958: done (1 left) 2024-09-11T03:49:20.356 INFO:tasks.rados.rados.2.smithi175.stdout:2960: done (0 left) 2024-09-11T03:49:20.356 INFO:tasks.rados.rados.2.smithi175.stdout:2961: snap_remove snap 299 2024-09-11T03:49:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:20 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:49:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:20 smithi084 ceph-mon[27254]: 2.b scrub ok 2024-09-11T03:49:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:20 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:49:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:20 smithi084 ceph-mon[27254]: 2.9 scrub ok 2024-09-11T03:49:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:20 smithi084 ceph-mon[27254]: pgmap v1266: 67 pgs: 67 active+clean; 371 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.1 MiB/s wr, 1 op/s 2024-09-11T03:49:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:20 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3358045025' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:49:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:20 smithi084 ceph-mon[27254]: 2.19 scrub starts 2024-09-11T03:49:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:20 smithi084 ceph-mon[27254]: 2.19 scrub ok 2024-09-11T03:49:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:20 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:49:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:20 smithi121 ceph-mon[30729]: 2.b scrub ok 2024-09-11T03:49:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:20 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:49:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:20 smithi121 ceph-mon[30729]: 2.9 scrub ok 2024-09-11T03:49:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:20 smithi121 ceph-mon[30729]: pgmap v1266: 67 pgs: 67 active+clean; 371 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.1 MiB/s wr, 1 op/s 2024-09-11T03:49:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:20 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3358045025' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:49:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:20 smithi121 ceph-mon[30729]: 2.19 scrub starts 2024-09-11T03:49:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:20 smithi121 ceph-mon[30729]: 2.19 scrub ok 2024-09-11T03:49:20.446 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:20.467 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:20.468 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:20.568 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:20.596 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:20.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:20.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:20.698 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:20.719 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:20.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:20.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:20.742 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:49:20.742 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:49:20.742 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:49:20.743 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:49:20.821 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:20.843 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:20.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:20.844 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:20.944 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:20.947 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:20.966 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:20.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:20.967 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:21.069 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:21.090 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:21.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:21.091 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:21.096 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 2 pg_num to 46 2024-09-11T03:49:21.109 INFO:tasks.rados.rados.2.smithi175.stdout:2961: done (0 left) 2024-09-11T03:49:21.109 INFO:tasks.rados.rados.2.smithi175.stdout:2962: read oid 2 snap 296 2024-09-11T03:49:21.109 INFO:tasks.rados.rados.2.smithi175.stdout:2962: expect (ObjNum 765 snap 247 seq_num 765) 2024-09-11T03:49:21.109 INFO:tasks.rados.rados.2.smithi175.stdout:2963: rollback oid 4 current snap is 332 2024-09-11T03:49:21.109 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 4 to 300 2024-09-11T03:49:21.109 INFO:tasks.rados.rados.2.smithi175.stdout:2964: read oid 12 snap -1 2024-09-11T03:49:21.109 INFO:tasks.rados.rados.2.smithi175.stdout:2964: expect (ObjNum 962 snap 308 seq_num 962) 2024-09-11T03:49:21.109 INFO:tasks.rados.rados.2.smithi175.stdout:2965: copy_from oid 9 from oid 27 current snap is 332 2024-09-11T03:49:21.109 INFO:tasks.rados.rados.2.smithi175.stdout:2966: snap_remove snap 325 2024-09-11T03:49:21.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:21 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:49:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:21 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:49:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3358045025' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:49:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:21 smithi084 ceph-mon[27254]: osdmap e742: 12 total, 12 up, 12 in 2024-09-11T03:49:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:21 smithi084 ceph-mon[27254]: 1.d scrub starts 2024-09-11T03:49:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:21 smithi084 ceph-mon[27254]: 1.d scrub ok 2024-09-11T03:49:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:21 smithi084 ceph-mon[27254]: 2.1f scrub starts 2024-09-11T03:49:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:21 smithi084 ceph-mon[27254]: 2.1f scrub ok 2024-09-11T03:49:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4090981107' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"} : dispatch 2024-09-11T03:49:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:21 smithi084 ceph-mon[27254]: 2.1a scrub starts 2024-09-11T03:49:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:21 smithi084 ceph-mon[27254]: 2.1a scrub ok 2024-09-11T03:49:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:21 smithi084 ceph-mon[27254]: 1.16 scrub starts 2024-09-11T03:49:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:21 smithi084 ceph-mon[27254]: 1.16 scrub ok 2024-09-11T03:49:21.192 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:21.214 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:21.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:21.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:21.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:21.341 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:21.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:21.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:21 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:49:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:21 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:49:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:21 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3358045025' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:49:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:21 smithi121 ceph-mon[30729]: osdmap e742: 12 total, 12 up, 12 in 2024-09-11T03:49:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:21 smithi121 ceph-mon[30729]: 1.d scrub starts 2024-09-11T03:49:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:21 smithi121 ceph-mon[30729]: 1.d scrub ok 2024-09-11T03:49:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:21 smithi121 ceph-mon[30729]: 2.1f scrub starts 2024-09-11T03:49:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:21 smithi121 ceph-mon[30729]: 2.1f scrub ok 2024-09-11T03:49:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:21 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4090981107' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"} : dispatch 2024-09-11T03:49:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:21 smithi121 ceph-mon[30729]: 2.1a scrub starts 2024-09-11T03:49:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:21 smithi121 ceph-mon[30729]: 2.1a scrub ok 2024-09-11T03:49:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:21 smithi121 ceph-mon[30729]: 1.16 scrub starts 2024-09-11T03:49:21.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:21 smithi121 ceph-mon[30729]: 1.16 scrub ok 2024-09-11T03:49:21.439 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:21 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:49:21.440 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:21 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:49:21.440 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:21 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3358045025' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:49:21.440 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:21 smithi175 ceph-mon[30894]: osdmap e742: 12 total, 12 up, 12 in 2024-09-11T03:49:21.440 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:21 smithi175 ceph-mon[30894]: 1.d scrub starts 2024-09-11T03:49:21.440 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:21 smithi175 ceph-mon[30894]: 1.d scrub ok 2024-09-11T03:49:21.440 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:21 smithi175 ceph-mon[30894]: 2.1f scrub starts 2024-09-11T03:49:21.440 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:21 smithi175 ceph-mon[30894]: 2.1f scrub ok 2024-09-11T03:49:21.440 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:21 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4090981107' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"} : dispatch 2024-09-11T03:49:21.440 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:21 smithi175 ceph-mon[30894]: 2.1a scrub starts 2024-09-11T03:49:21.440 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:21 smithi175 ceph-mon[30894]: 2.1a scrub ok 2024-09-11T03:49:21.440 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:21 smithi175 ceph-mon[30894]: 1.16 scrub starts 2024-09-11T03:49:21.440 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:21 smithi175 ceph-mon[30894]: 1.16 scrub ok 2024-09-11T03:49:21.443 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:21.464 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:21.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:21.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:21.466 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-11T03:49:21.566 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:21.587 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:21.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:21.588 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:21.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:21.708 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:49:21.711 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:21.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:21.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:21.839 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:21.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:21.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:21.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:21.965 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:21.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:21.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:22.017 INFO:tasks.rados.rados.2.smithi175.stdout:2966: done (4 left) 2024-09-11T03:49:22.017 INFO:tasks.rados.rados.2.smithi175.stdout:2967: copy_from oid 48 from oid 10 current snap is 332 2024-09-11T03:49:22.017 INFO:tasks.rados.rados.2.smithi175.stdout:2963: finishing rollback tid 0 to smithi17557870-4 2024-09-11T03:49:22.017 INFO:tasks.rados.rados.2.smithi175.stdout:2963: finishing rollback tid 1 to smithi17557870-4 2024-09-11T03:49:22.017 INFO:tasks.rados.rados.2.smithi175.stdout:2963: finishing rollback tid 2 to smithi17557870-4 2024-09-11T03:49:22.017 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 968 (ObjNum 807 snap 255 seq_num 807) dirty exists 2024-09-11T03:49:22.017 INFO:tasks.rados.rados.2.smithi175.stdout:2964: expect (ObjNum 962 snap 308 seq_num 962) 2024-09-11T03:49:22.067 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:22.073 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:49:22.073 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:49:22.088 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:22.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:22.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:22.090 INFO:tasks.rados.rados.2.smithi175.stdout:2962: expect (ObjNum 765 snap 247 seq_num 765) 2024-09-11T03:49:22.090 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:49:22.090 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:49:22.106 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:49:22.107 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:49:22.123 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:49:22.123 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:49:22.139 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:49:22.140 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:49:22.156 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:49:22.157 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:49:22.173 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:49:22.173 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:49:22.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:22.190 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:49:22.190 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:49:22.206 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:49:22.206 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:49:22.213 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:22.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:22.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:22.220 INFO:tasks.rados.rados.2.smithi175.stdout:2965: finishing copy_from racing read to smithi17557870-9 2024-09-11T03:49:22.220 INFO:tasks.rados.rados.2.smithi175.stdout:2965: finishing copy_from to smithi17557870-9 2024-09-11T03:49:22.220 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 950 (ObjNum 765 snap 247 seq_num 765) dirty exists 2024-09-11T03:49:22.220 INFO:tasks.rados.rados.2.smithi175.stdout:2962: done (4 left) 2024-09-11T03:49:22.220 INFO:tasks.rados.rados.2.smithi175.stdout:2963: done (3 left) 2024-09-11T03:49:22.220 INFO:tasks.rados.rados.2.smithi175.stdout:2964: done (2 left) 2024-09-11T03:49:22.220 INFO:tasks.rados.rados.2.smithi175.stdout:2965: done (1 left) 2024-09-11T03:49:22.220 INFO:tasks.rados.rados.2.smithi175.stdout:2968: copy_from oid 44 from oid 38 current snap is 332 2024-09-11T03:49:22.220 INFO:tasks.rados.rados.2.smithi175.stdout:2967: finishing copy_from racing read to smithi17557870-48 2024-09-11T03:49:22.220 INFO:tasks.rados.rados.2.smithi175.stdout:2967: finishing copy_from to smithi17557870-48 2024-09-11T03:49:22.221 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 916 (ObjNum 736 snap 243 seq_num 736) dirty exists 2024-09-11T03:49:22.221 INFO:tasks.rados.rados.2.smithi175.stdout:2967: done (1 left) 2024-09-11T03:49:22.221 INFO:tasks.rados.rados.2.smithi175.stdout:2969: snap_create 2024-09-11T03:49:22.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:49:22.223 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:49:22.226 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:49:22.226 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:49:22.226 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:49:22.226 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:49:22.226 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:49:22.231 INFO:tasks.rados.rados.2.smithi175.stdout:2968: finishing copy_from racing read to smithi17557870-44 2024-09-11T03:49:22.233 INFO:tasks.rados.rados.2.smithi175.stdout:2968: finishing copy_from to smithi17557870-44 2024-09-11T03:49:22.233 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 1013 (ObjNum 970 snap 311 seq_num 970) dirty exists 2024-09-11T03:49:22.240 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:49:22.240 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:49:22.256 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:49:22.256 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:49:22.315 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:22.337 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:22.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:22.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:22 smithi084 ceph-mon[27254]: 2.12 scrub starts 2024-09-11T03:49:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:22 smithi084 ceph-mon[27254]: 2.12 scrub ok 2024-09-11T03:49:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:22 smithi084 ceph-mon[27254]: pgmap v1269: 67 pgs: 67 active+clean; 371 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:49:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:22 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:49:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:22 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:49:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:22 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4090981107' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]': finished 2024-09-11T03:49:22.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:22 smithi084 ceph-mon[27254]: osdmap e743: 12 total, 12 up, 12 in 2024-09-11T03:49:22.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:22 smithi084 ceph-mon[27254]: 1.4 scrub starts 2024-09-11T03:49:22.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:22 smithi084 ceph-mon[27254]: 1.4 scrub ok 2024-09-11T03:49:22.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:22 smithi084 ceph-mon[27254]: osdmap e744: 12 total, 12 up, 12 in 2024-09-11T03:49:22.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:22 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2614080734' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:22 smithi121 ceph-mon[30729]: 2.12 scrub starts 2024-09-11T03:49:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:22 smithi121 ceph-mon[30729]: 2.12 scrub ok 2024-09-11T03:49:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:22 smithi121 ceph-mon[30729]: pgmap v1269: 67 pgs: 67 active+clean; 371 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:49:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:22 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:49:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:22 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:49:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:22 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4090981107' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]': finished 2024-09-11T03:49:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:22 smithi121 ceph-mon[30729]: osdmap e743: 12 total, 12 up, 12 in 2024-09-11T03:49:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:22 smithi121 ceph-mon[30729]: 1.4 scrub starts 2024-09-11T03:49:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:22 smithi121 ceph-mon[30729]: 1.4 scrub ok 2024-09-11T03:49:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:22 smithi121 ceph-mon[30729]: osdmap e744: 12 total, 12 up, 12 in 2024-09-11T03:49:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:22 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2614080734' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:22.439 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:22.465 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:22.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:22.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:22 smithi175 ceph-mon[30894]: 2.12 scrub starts 2024-09-11T03:49:22.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:22 smithi175 ceph-mon[30894]: 2.12 scrub ok 2024-09-11T03:49:22.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:22 smithi175 ceph-mon[30894]: pgmap v1269: 67 pgs: 67 active+clean; 371 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:49:22.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:22 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:49:22.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:22 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:49:22.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:22 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4090981107' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]': finished 2024-09-11T03:49:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:22 smithi175 ceph-mon[30894]: osdmap e743: 12 total, 12 up, 12 in 2024-09-11T03:49:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:22 smithi175 ceph-mon[30894]: 1.4 scrub starts 2024-09-11T03:49:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:22 smithi175 ceph-mon[30894]: 1.4 scrub ok 2024-09-11T03:49:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:22 smithi175 ceph-mon[30894]: osdmap e744: 12 total, 12 up, 12 in 2024-09-11T03:49:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:22 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2614080734' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:22.566 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:22.588 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:22.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:22.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:22.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:22.711 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:22.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:22.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:22.812 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:22.838 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:22.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:22.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:22.887 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:49:22.940 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:22.962 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:22.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:22.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:22.990 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-11T03:49:23.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:23.088 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:23.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:23.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:23.112 INFO:tasks.rados.rados.2.smithi175.stdout:2968: done (1 left) 2024-09-11T03:49:23.112 INFO:tasks.rados.rados.2.smithi175.stdout:2969: done (0 left) 2024-09-11T03:49:23.112 INFO:tasks.rados.rados.2.smithi175.stdout:2970: read oid 24 snap 327 2024-09-11T03:49:23.112 INFO:tasks.rados.rados.2.smithi175.stdout:2970: expect deleted 2024-09-11T03:49:23.112 INFO:tasks.rados.rados.2.smithi175.stdout:2971: snap_create 2024-09-11T03:49:23.147 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:23.189 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:23.211 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:23.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:23.212 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:23.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:23 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:49:23.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:23 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:49:23.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:23 smithi175 ceph-mon[30894]: 1.e scrub starts 2024-09-11T03:49:23.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:23 smithi175 ceph-mon[30894]: 1.e scrub ok 2024-09-11T03:49:23.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:23 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"} : dispatch 2024-09-11T03:49:23.267 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:23 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:49:23.267 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:23 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:49:23.267 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:23 smithi084 ceph-mon[27254]: 1.e scrub starts 2024-09-11T03:49:23.268 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:23 smithi084 ceph-mon[27254]: 1.e scrub ok 2024-09-11T03:49:23.268 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:23 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"} : dispatch 2024-09-11T03:49:23.281 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:49:23.313 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:23.334 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:23.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:23.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:23 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:49:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:23 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:49:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:23 smithi121 ceph-mon[30729]: 1.e scrub starts 2024-09-11T03:49:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:23 smithi121 ceph-mon[30729]: 1.e scrub ok 2024-09-11T03:49:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:23 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"} : dispatch 2024-09-11T03:49:23.436 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:23.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:23.457 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:23.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:23.559 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:23.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:23.581 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:23.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:23.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:23.704 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:23.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:23.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:23.767 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:49:23.767 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:49:23.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:23.826 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:23.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:23.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:23.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:23.955 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:23.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:24.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:24.078 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:24.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:24.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:24.105 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:49:24.111 INFO:tasks.rados.rados.2.smithi175.stdout:2970: done (1 left) 2024-09-11T03:49:24.111 INFO:tasks.rados.rados.2.smithi175.stdout:2971: done (0 left) 2024-09-11T03:49:24.111 INFO:tasks.rados.rados.2.smithi175.stdout:2972: copy_from oid 17 from oid 14 current snap is 334 2024-09-11T03:49:24.111 INFO:tasks.rados.rados.2.smithi175.stdout:2973: delete oid 31 current snap is 334 2024-09-11T03:49:24.124 INFO:tasks.rados.rados.2.smithi175.stdout:2973: done (1 left) 2024-09-11T03:49:24.125 INFO:tasks.rados.rados.2.smithi175.stdout:2974: read oid 19 snap -1 2024-09-11T03:49:24.125 INFO:tasks.rados.rados.2.smithi175.stdout:2974: expect (ObjNum 963 snap 310 seq_num 963) 2024-09-11T03:49:24.125 INFO:tasks.rados.rados.2.smithi175.stdout:2975: delete oid 49 current snap is 334 2024-09-11T03:49:24.126 INFO:tasks.rados.rados.2.smithi175.stdout:2975: done (2 left) 2024-09-11T03:49:24.127 INFO:tasks.rados.rados.2.smithi175.stdout:2976: write (excl) oid 34 current snap is 334 2024-09-11T03:49:24.127 INFO:tasks.rados.rados.2.smithi175.stdout:2976: seq_num 1019 ranges {726333=670033,2096210=724353} 2024-09-11T03:49:24.134 INFO:tasks.rados.rados.2.smithi175.stdout:2976: writing smithi17557870-34 from 726333 to 1396366 tid 1 2024-09-11T03:49:24.142 INFO:tasks.rados.rados.2.smithi175.stdout:2976: writing smithi17557870-34 from 2096210 to 2820563 tid 2 2024-09-11T03:49:24.142 INFO:tasks.rados.rados.2.smithi175.stdout:2977: write oid 12 current snap is 334 2024-09-11T03:49:24.142 INFO:tasks.rados.rados.2.smithi175.stdout:2977: seq_num 1020 ranges {743140=745282,2154692=468011} 2024-09-11T03:49:24.150 INFO:tasks.rados.rados.2.smithi175.stdout:2977: writing smithi17557870-12 from 743140 to 1488422 tid 1 2024-09-11T03:49:24.157 INFO:tasks.rados.rados.2.smithi175.stdout:2977: writing smithi17557870-12 from 2154692 to 2622703 tid 2 2024-09-11T03:49:24.157 INFO:tasks.rados.rados.2.smithi175.stdout:2978: read oid 27 snap -1 2024-09-11T03:49:24.157 INFO:tasks.rados.rados.2.smithi175.stdout:2978: expect (ObjNum 765 snap 247 seq_num 765) 2024-09-11T03:49:24.157 INFO:tasks.rados.rados.2.smithi175.stdout:2979: copy_from oid 2 from oid 41 current snap is 334 2024-09-11T03:49:24.157 INFO:tasks.rados.rados.2.smithi175.stdout:2974: expect (ObjNum 963 snap 310 seq_num 963) 2024-09-11T03:49:24.180 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:24.202 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:24.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:24.203 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:24.278 INFO:tasks.rados.rados.2.smithi175.stdout:2976: finishing write tid 1 to smithi17557870-34 2024-09-11T03:49:24.279 INFO:tasks.rados.rados.2.smithi175.stdout:2976: finishing write tid 2 to smithi17557870-34 2024-09-11T03:49:24.279 INFO:tasks.rados.rados.2.smithi175.stdout:2976: finishing write tid 3 to smithi17557870-34 2024-09-11T03:49:24.279 INFO:tasks.rados.rados.2.smithi175.stdout:2976: finishing write tid 4 to smithi17557870-34 2024-09-11T03:49:24.279 INFO:tasks.rados.rados.2.smithi175.stdout:2976: oid 34 updating version 0 to 963 2024-09-11T03:49:24.279 INFO:tasks.rados.rados.2.smithi175.stdout:2976: oid 34 version 963 is already newer than 961 2024-09-11T03:49:24.279 INFO:tasks.rados.rados.2.smithi175.stdout:2976: oid 34 version 963 is already newer than 962 2024-09-11T03:49:24.279 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 963 (ObjNum 1019 snap 334 seq_num 1019) dirty exists 2024-09-11T03:49:24.279 INFO:tasks.rados.rados.2.smithi175.stdout:2976: left oid 34 (ObjNum 1019 snap 334 seq_num 1019) 2024-09-11T03:49:24.279 INFO:tasks.rados.rados.2.smithi175.stdout:2977: finishing write tid 1 to smithi17557870-12 2024-09-11T03:49:24.279 INFO:tasks.rados.rados.2.smithi175.stdout:2974: done (5 left) 2024-09-11T03:49:24.279 INFO:tasks.rados.rados.2.smithi175.stdout:2976: done (4 left) 2024-09-11T03:49:24.279 INFO:tasks.rados.rados.2.smithi175.stdout:2980: delete oid 7 current snap is 334 2024-09-11T03:49:24.280 INFO:tasks.rados.rados.2.smithi175.stdout:2977: finishing write tid 2 to smithi17557870-12 2024-09-11T03:49:24.280 INFO:tasks.rados.rados.2.smithi175.stdout:2977: finishing write tid 3 to smithi17557870-12 2024-09-11T03:49:24.280 INFO:tasks.rados.rados.2.smithi175.stdout:2977: finishing write tid 4 to smithi17557870-12 2024-09-11T03:49:24.280 INFO:tasks.rados.rados.2.smithi175.stdout:2977: oid 12 updating version 0 to 1110 2024-09-11T03:49:24.280 INFO:tasks.rados.rados.2.smithi175.stdout:2977: oid 12 updating version 1110 to 1111 2024-09-11T03:49:24.280 INFO:tasks.rados.rados.2.smithi175.stdout:2977: oid 12 updating version 1111 to 1112 2024-09-11T03:49:24.280 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 1112 (ObjNum 1020 snap 334 seq_num 1020) dirty exists 2024-09-11T03:49:24.280 INFO:tasks.rados.rados.2.smithi175.stdout:2977: left oid 12 (ObjNum 1020 snap 334 seq_num 1020) 2024-09-11T03:49:24.280 INFO:tasks.rados.rados.2.smithi175.stdout:2979: finishing copy_from to smithi17557870-2 2024-09-11T03:49:24.280 INFO:tasks.rados.rados.2.smithi175.stdout:2979: got expected ENOENT (src dne) 2024-09-11T03:49:24.280 INFO:tasks.rados.rados.2.smithi175.stdout:2978: expect (ObjNum 765 snap 247 seq_num 765) 2024-09-11T03:49:24.304 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:24.328 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:24.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:24.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:24.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:24 smithi084 ceph-mon[27254]: pgmap v1272: 67 pgs: 67 active+clean; 379 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.2 MiB/s wr, 18 op/s 2024-09-11T03:49:24.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:24 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]': finished 2024-09-11T03:49:24.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:24 smithi084 ceph-mon[27254]: osdmap e745: 12 total, 12 up, 12 in 2024-09-11T03:49:24.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:24 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/734224345' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:49:24.404 INFO:tasks.rados.rados.2.smithi175.stdout:2979: finishing copy_from racing read to smithi17557870-2 2024-09-11T03:49:24.404 INFO:tasks.rados.rados.2.smithi175.stdout:2977: done (4 left) 2024-09-11T03:49:24.404 INFO:tasks.rados.rados.2.smithi175.stdout:2978: done (3 left) 2024-09-11T03:49:24.404 INFO:tasks.rados.rados.2.smithi175.stdout:2979: done (2 left) 2024-09-11T03:49:24.404 INFO:tasks.rados.rados.2.smithi175.stdout:2980: done (1 left) 2024-09-11T03:49:24.404 INFO:tasks.rados.rados.2.smithi175.stdout:2981: snap_remove snap 328 2024-09-11T03:49:24.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:24 smithi121 ceph-mon[30729]: pgmap v1272: 67 pgs: 67 active+clean; 379 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.2 MiB/s wr, 18 op/s 2024-09-11T03:49:24.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:24 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]': finished 2024-09-11T03:49:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:24 smithi121 ceph-mon[30729]: osdmap e745: 12 total, 12 up, 12 in 2024-09-11T03:49:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:24 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/734224345' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:49:24.429 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:24.434 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:24 smithi175 ceph-mon[30894]: pgmap v1272: 67 pgs: 67 active+clean; 379 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.2 MiB/s wr, 18 op/s 2024-09-11T03:49:24.434 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:24 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]': finished 2024-09-11T03:49:24.434 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:24 smithi175 ceph-mon[30894]: osdmap e745: 12 total, 12 up, 12 in 2024-09-11T03:49:24.434 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:24 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/734224345' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:49:24.451 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:24.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:24.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:24.530 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-11T03:49:24.552 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:24.577 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:24.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:24.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:24.679 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:24.701 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:24.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:24.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:24.771 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:49:24.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:24.828 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:24.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:24.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:24.930 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:24.956 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:24.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:24.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:25.058 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:25.078 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:25.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:25.079 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:25.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:49:25.132 INFO:tasks.rados.rados.2.smithi175.stdout:2981: done (1 left) 2024-09-11T03:49:25.132 INFO:tasks.rados.rados.2.smithi175.stdout:2982: write (excl) oid 5 current snap is 334 2024-09-11T03:49:25.132 INFO:tasks.rados.rados.2.smithi175.stdout:2982: seq_num 1022 ranges {776754=495780,1871656=471680,2965828=210926} 2024-09-11T03:49:25.139 INFO:tasks.rados.rados.2.smithi175.stdout:2982: writing smithi17557870-5 from 776754 to 1272534 tid 1 2024-09-11T03:49:25.141 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:25 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:25.142 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:25 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/734224345' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:49:25.142 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:25 smithi175 ceph-mon[30894]: osdmap e746: 12 total, 12 up, 12 in 2024-09-11T03:49:25.142 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:25 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:49:25.142 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:25 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:49:25.142 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:25 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:49:25.142 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:25 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:49:25.142 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:25 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"} : dispatch 2024-09-11T03:49:25.142 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:25 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:49:25.145 INFO:tasks.rados.rados.2.smithi175.stdout:2982: writing smithi17557870-5 from 1871656 to 2343336 tid 2 2024-09-11T03:49:25.149 INFO:tasks.rados.rados.2.smithi175.stdout:2982: writing smithi17557870-5 from 2965828 to 3176754 tid 3 2024-09-11T03:49:25.150 INFO:tasks.rados.rados.2.smithi175.stdout:2983: snap_create 2024-09-11T03:49:25.150 INFO:tasks.rados.rados.2.smithi175.stdout:2982: finishing write tid 1 to smithi17557870-5 2024-09-11T03:49:25.150 INFO:tasks.rados.rados.2.smithi175.stdout:2982: finishing write tid 2 to smithi17557870-5 2024-09-11T03:49:25.152 INFO:tasks.rados.rados.2.smithi175.stdout:2982: finishing write tid 3 to smithi17557870-5 2024-09-11T03:49:25.152 INFO:tasks.rados.rados.2.smithi175.stdout:2982: finishing write tid 4 to smithi17557870-5 2024-09-11T03:49:25.153 INFO:tasks.rados.rados.2.smithi175.stdout:2982: finishing write tid 5 to smithi17557870-5 2024-09-11T03:49:25.153 INFO:tasks.rados.rados.2.smithi175.stdout:2982: oid 5 updating version 0 to 951 2024-09-11T03:49:25.153 INFO:tasks.rados.rados.2.smithi175.stdout:2982: oid 5 version 951 is already newer than 950 2024-09-11T03:49:25.153 INFO:tasks.rados.rados.2.smithi175.stdout:2982: oid 5 version 951 is already newer than 949 2024-09-11T03:49:25.153 INFO:tasks.rados.rados.2.smithi175.stdout:2982: oid 5 version 951 is already newer than 948 2024-09-11T03:49:25.153 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 951 (ObjNum 1022 snap 334 seq_num 1022) dirty exists 2024-09-11T03:49:25.153 INFO:tasks.rados.rados.2.smithi175.stdout:2982: left oid 5 (ObjNum 1022 snap 334 seq_num 1022) 2024-09-11T03:49:25.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:25.207 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:25.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:25.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:25.258 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:49:25.258 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:49:25.309 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:25.331 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:25.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:25.355 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:25.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:25 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:25.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:25 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/734224345' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:49:25.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:25 smithi084 ceph-mon[27254]: osdmap e746: 12 total, 12 up, 12 in 2024-09-11T03:49:25.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:25 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:49:25.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:25 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:49:25.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:25 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:49:25.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:25 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:49:25.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:25 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"} : dispatch 2024-09-11T03:49:25.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:25 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:49:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:25 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:25 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/734224345' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:49:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:25 smithi121 ceph-mon[30729]: osdmap e746: 12 total, 12 up, 12 in 2024-09-11T03:49:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:25 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:49:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:25 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:49:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:25 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:49:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:25 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:49:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:25 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"} : dispatch 2024-09-11T03:49:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:25 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:49:25.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:25.458 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:25.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:25.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:25.560 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:25.581 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:25.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:25.582 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:25.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:25.709 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:25.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:25.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:25.799 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:49:25.799 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:49:25.810 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:25.815 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:49:25.815 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:49:25.831 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:49:25.831 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:49:25.836 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:25.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:25.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:25.846 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:49:25.847 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:49:25.863 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:49:25.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:49:25.880 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:49:25.880 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:49:25.896 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:49:25.897 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:49:25.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:49:25.913 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:49:25.930 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:49:25.930 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:49:25.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:25.946 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:49:25.946 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:49:25.962 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:25.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:25.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:25.963 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:49:25.963 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:49:25.979 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:49:25.979 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:49:26.022 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-11T03:49:26.064 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:26.086 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:26.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:26.087 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:26.131 INFO:tasks.rados.rados.2.smithi175.stdout:2982: done (2 left) 2024-09-11T03:49:26.131 INFO:tasks.rados.rados.2.smithi175.stdout:2983: done (1 left) 2024-09-11T03:49:26.131 INFO:tasks.rados.rados.2.smithi175.stdout:2984: write (excl) oid 23 current snap is 335 2024-09-11T03:49:26.131 INFO:tasks.rados.rados.2.smithi175.stdout:2984: seq_num 1023 ranges {793561=571029,1813785=731691,2993251=200310} 2024-09-11T03:49:26.137 INFO:tasks.rados.rados.2.smithi175.stdout:2984: writing smithi17557870-23 from 793561 to 1364590 tid 1 2024-09-11T03:49:26.144 INFO:tasks.rados.rados.2.smithi175.stdout:2984: writing smithi17557870-23 from 1813785 to 2545476 tid 2 2024-09-11T03:49:26.147 INFO:tasks.rados.rados.2.smithi175.stdout:2984: writing smithi17557870-23 from 2993251 to 3193561 tid 3 2024-09-11T03:49:26.148 INFO:tasks.rados.rados.2.smithi175.stdout:2985: snap_create 2024-09-11T03:49:26.148 INFO:tasks.rados.rados.2.smithi175.stdout:2984: finishing write tid 1 to smithi17557870-23 2024-09-11T03:49:26.148 INFO:tasks.rados.rados.2.smithi175.stdout:2984: finishing write tid 2 to smithi17557870-23 2024-09-11T03:49:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:2984: finishing write tid 3 to smithi17557870-23 2024-09-11T03:49:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:2984: finishing write tid 4 to smithi17557870-23 2024-09-11T03:49:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:2984: finishing write tid 5 to smithi17557870-23 2024-09-11T03:49:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:2984: oid 23 updating version 0 to 969 2024-09-11T03:49:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:2984: oid 23 updating version 969 to 971 2024-09-11T03:49:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:2984: oid 23 version 971 is already newer than 970 2024-09-11T03:49:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:2984: oid 23 updating version 971 to 972 2024-09-11T03:49:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 972 (ObjNum 1023 snap 335 seq_num 1023) dirty exists 2024-09-11T03:49:26.150 INFO:tasks.rados.rados.2.smithi175.stdout:2984: left oid 23 (ObjNum 1023 snap 335 seq_num 1023) 2024-09-11T03:49:26.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:26.213 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:26.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:26.270 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:49:26.314 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:26.335 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:26.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:26.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:26.391 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:26 smithi175 ceph-mon[30894]: 2.6 scrub starts 2024-09-11T03:49:26.391 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:26 smithi175 ceph-mon[30894]: pgmap v1275: 77 pgs: 10 unknown, 67 active+clean; 379 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.2 MiB/s wr, 18 op/s 2024-09-11T03:49:26.391 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:26 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]': finished 2024-09-11T03:49:26.391 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:26 smithi175 ceph-mon[30894]: osdmap e747: 12 total, 12 up, 12 in 2024-09-11T03:49:26.391 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:26 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:49:26.391 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:26 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:49:26.392 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:26 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:49:26.392 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:26 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:49:26.392 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1630150976' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:26.392 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:26 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:49:26.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:26 smithi084 ceph-mon[27254]: 2.6 scrub starts 2024-09-11T03:49:26.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:26 smithi084 ceph-mon[27254]: pgmap v1275: 77 pgs: 10 unknown, 67 active+clean; 379 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.2 MiB/s wr, 18 op/s 2024-09-11T03:49:26.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:26 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]': finished 2024-09-11T03:49:26.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:26 smithi084 ceph-mon[27254]: osdmap e747: 12 total, 12 up, 12 in 2024-09-11T03:49:26.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:26 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:49:26.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:26 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:49:26.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:26 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:49:26.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:26 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:49:26.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1630150976' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:26.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:26 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:49:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:26 smithi121 ceph-mon[30729]: 2.6 scrub starts 2024-09-11T03:49:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:26 smithi121 ceph-mon[30729]: pgmap v1275: 77 pgs: 10 unknown, 67 active+clean; 379 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.2 MiB/s wr, 18 op/s 2024-09-11T03:49:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:26 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]': finished 2024-09-11T03:49:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:26 smithi121 ceph-mon[30729]: osdmap e747: 12 total, 12 up, 12 in 2024-09-11T03:49:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:26 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:49:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:26 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:49:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:26 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:49:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:26 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:49:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1630150976' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:26 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:49:26.437 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:26.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:26.459 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:26.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:26.560 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:26.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:26.582 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:26.583 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:26.684 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:26.705 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:26.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:26.706 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:26.765 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:49:26.765 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:49:26.765 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:49:26.765 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:49:26.765 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:49:26.795 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:49:26.807 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:26.828 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:26.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:26.829 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:26.860 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:49:26.861 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:49:26.861 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T03:49:26.861 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.982715 2024-09-11T03:49:26.861 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 7 0.9827149362422528 2024-09-11T03:49:26.930 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:26.951 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:26.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:26.952 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:27.048 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:27.052 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:27.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:27.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:27.142 INFO:tasks.rados.rados.2.smithi175.stdout:2984: done (2 left) 2024-09-11T03:49:27.142 INFO:tasks.rados.rados.2.smithi175.stdout:2985: done (1 left) 2024-09-11T03:49:27.143 INFO:tasks.rados.rados.2.smithi175.stdout:2986: rollback oid 9 current snap is 336 2024-09-11T03:49:27.143 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 9 to 333 2024-09-11T03:49:27.143 INFO:tasks.rados.rados.2.smithi175.stdout:2987: rollback oid 37 current snap is 336 2024-09-11T03:49:27.143 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 37 to 322 2024-09-11T03:49:27.143 INFO:tasks.rados.rados.2.smithi175.stdout:2988: read oid 26 snap -1 2024-09-11T03:49:27.143 INFO:tasks.rados.rados.2.smithi175.stdout:2988: expect (ObjNum 1004 snap 324 seq_num 1004) 2024-09-11T03:49:27.143 INFO:tasks.rados.rados.2.smithi175.stdout:2989: snap_create 2024-09-11T03:49:27.155 INFO:tasks.rados.rados.2.smithi175.stdout:2987: finishing rollback tid 1 to smithi17557870-37 2024-09-11T03:49:27.155 INFO:tasks.rados.rados.2.smithi175.stdout:2987: finishing rollback tid 2 to smithi17557870-37 2024-09-11T03:49:27.155 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 902 (ObjNum 867 snap 273 seq_num 867) dirty exists 2024-09-11T03:49:27.156 INFO:tasks.rados.rados.2.smithi175.stdout:2986: finishing rollback tid 0 to smithi17557870-9 2024-09-11T03:49:27.156 INFO:tasks.rados.rados.2.smithi175.stdout:2986: finishing rollback tid 1 to smithi17557870-9 2024-09-11T03:49:27.156 INFO:tasks.rados.rados.2.smithi175.stdout:2986: finishing rollback tid 2 to smithi17557870-9 2024-09-11T03:49:27.156 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 1055 (ObjNum 765 snap 247 seq_num 765) dirty exists 2024-09-11T03:49:27.168 INFO:tasks.rados.rados.2.smithi175.stdout:2972: finishing copy_from to smithi17557870-17 2024-09-11T03:49:27.168 INFO:tasks.rados.rados.2.smithi175.stdout:2972: got expected ENOENT (src dne) 2024-09-11T03:49:27.168 INFO:tasks.rados.rados.2.smithi175.stdout:2972: finishing copy_from racing read to smithi17557870-17 2024-09-11T03:49:27.172 INFO:tasks.rados.rados.2.smithi175.stdout:2988: expect (ObjNum 1004 snap 324 seq_num 1004) 2024-09-11T03:49:27.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:27.209 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:27.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:27.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:27.311 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:27.355 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:27.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:27.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:27 smithi084 ceph-mon[27254]: 2.6 scrub starts 2024-09-11T03:49:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:27 smithi084 ceph-mon[27254]: osdmap e748: 12 total, 12 up, 12 in 2024-09-11T03:49:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:27 smithi084 ceph-mon[27254]: 2.5 scrub starts 2024-09-11T03:49:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:27 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:49:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:27 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:49:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:27 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:49:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:27 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:49:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:27 smithi121 ceph-mon[30729]: 2.6 scrub starts 2024-09-11T03:49:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:27 smithi121 ceph-mon[30729]: osdmap e748: 12 total, 12 up, 12 in 2024-09-11T03:49:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:27 smithi121 ceph-mon[30729]: 2.5 scrub starts 2024-09-11T03:49:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:27 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:49:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:27 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:49:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:27 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:49:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:27 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:49:27.458 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:27.461 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:27 smithi175 ceph-mon[30894]: 2.6 scrub starts 2024-09-11T03:49:27.462 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:27 smithi175 ceph-mon[30894]: osdmap e748: 12 total, 12 up, 12 in 2024-09-11T03:49:27.462 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:27 smithi175 ceph-mon[30894]: 2.5 scrub starts 2024-09-11T03:49:27.462 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:27 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:49:27.462 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:27 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:49:27.462 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:27 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:49:27.462 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:27 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:49:27.480 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:27.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:27.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:27.488 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-11T03:49:27.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:27.592 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:27.603 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:27.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:27.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:27.705 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:27.726 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:49:27.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:27.728 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:27.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:27.749 INFO:teuthology.orchestra.run.smithi084.stderr:2024-09-11T03:49:27.749+0000 7f2a8cba1640 0 --2- 172.21.15.84:0/1074624284 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a88103b90 0x7f2a88103f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-11T03:49:27.829 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:27.851 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:27.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:27.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:27.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:27.975 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:27.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:28.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:28.098 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:28.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:28.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:28.137 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:49:28.144 INFO:tasks.rados.rados.2.smithi175.stdout:2972: done (4 left) 2024-09-11T03:49:28.144 INFO:tasks.rados.rados.2.smithi175.stdout:2986: done (3 left) 2024-09-11T03:49:28.144 INFO:tasks.rados.rados.2.smithi175.stdout:2987: done (2 left) 2024-09-11T03:49:28.144 INFO:tasks.rados.rados.2.smithi175.stdout:2988: done (1 left) 2024-09-11T03:49:28.144 INFO:tasks.rados.rados.2.smithi175.stdout:2989: done (0 left) 2024-09-11T03:49:28.144 INFO:tasks.rados.rados.2.smithi175.stdout:2990: snap_remove snap 324 2024-09-11T03:49:28.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:28.222 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:28.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:28.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:28.258 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:49:28.258 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:49:28.258 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:49:28.258 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:49:28.258 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.258 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4021 2.7 2:e77ce595:::smithi17557870-31:head [copy-from ver 473] snapc ac=[ac,ab,a9,a8,a6,a3,a1,9e,9d] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-11T03:49:28.258 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:13.490109+0000", 2024-09-11T03:49:28.258 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 314.76713925199999, 2024-09-11T03:49:28.258 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.02074401, 2024-09-11T03:49:28.259 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.259 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.259 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:28.259 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:28.259 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:28.259 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:28.259 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4021 2024-09-11T03:49:28.259 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.259 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.259 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.259 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.259 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490109+0000", 2024-09-11T03:49:28.259 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.260 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.260 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.260 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.260 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490109+0000", 2024-09-11T03:49:28.260 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.260 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.260 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.260 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.260 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490111+0000", 2024-09-11T03:49:28.260 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.438e-06 2024-09-11T03:49:28.260 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.260 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.260 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.261 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490118+0000", 2024-09-11T03:49:28.261 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8950000000000001e-06 2024-09-11T03:49:28.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.261 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.261 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490131+0000", 2024-09-11T03:49:28.261 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2889e-05 2024-09-11T03:49:28.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.261 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.261 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490138+0000", 2024-09-11T03:49:28.261 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3849999999999996e-06 2024-09-11T03:49:28.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.262 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.262 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490199+0000", 2024-09-11T03:49:28.262 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0822000000000001e-05 2024-09-11T03:49:28.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.262 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.262 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490297+0000", 2024-09-11T03:49:28.262 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7726999999999995e-05 2024-09-11T03:49:28.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.262 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:49:28.262 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.499178+0000", 2024-09-11T03:49:28.263 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0088812119999999994 2024-09-11T03:49:28.263 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.263 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.263 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:28.263 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.502683+0000", 2024-09-11T03:49:28.263 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035055559999999999 2024-09-11T03:49:28.263 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.263 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.263 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:28.263 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.507713+0000", 2024-09-11T03:49:28.263 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050300299999999996 2024-09-11T03:49:28.263 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.263 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.264 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:28.264 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.510764+0000", 2024-09-11T03:49:28.264 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030502469999999999 2024-09-11T03:49:28.264 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.264 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.264 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:28.264 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.510803+0000", 2024-09-11T03:49:28.264 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9629999999999998e-05 2024-09-11T03:49:28.264 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.264 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.264 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.264 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.510853+0000", 2024-09-11T03:49:28.264 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0179000000000002e-05 2024-09-11T03:49:28.265 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.265 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.265 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.265 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.265 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.265 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4022 2.7 2:e77ce595:::smithi17557870-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e435)", 2024-09-11T03:49:28.265 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:13.490154+0000", 2024-09-11T03:49:28.265 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 314.76709483799999, 2024-09-11T03:49:28.265 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.01199774, 2024-09-11T03:49:28.265 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.265 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.265 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:49:28.265 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:28.266 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:28.266 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:28.266 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4022 2024-09-11T03:49:28.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.266 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.266 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.266 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.266 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490154+0000", 2024-09-11T03:49:28.266 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.266 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.266 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.266 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490154+0000", 2024-09-11T03:49:28.266 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.267 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.267 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.267 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.267 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490154+0000", 2024-09-11T03:49:28.267 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3000000000000001e-07 2024-09-11T03:49:28.267 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.267 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.267 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.267 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490157+0000", 2024-09-11T03:49:28.267 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2460000000000002e-06 2024-09-11T03:49:28.267 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.267 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.267 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.267 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490163+0000", 2024-09-11T03:49:28.268 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6189999999999996e-06 2024-09-11T03:49:28.268 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.268 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.268 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.268 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490169+0000", 2024-09-11T03:49:28.268 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9139999999999996e-06 2024-09-11T03:49:28.268 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.268 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.268 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.268 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490486+0000", 2024-09-11T03:49:28.268 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00031683599999999997 2024-09-11T03:49:28.268 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.268 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.269 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:49:28.269 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490520+0000", 2024-09-11T03:49:28.269 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4548999999999997e-05 2024-09-11T03:49:28.269 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.269 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.269 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.269 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.502009+0000", 2024-09-11T03:49:28.269 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011488437000000001 2024-09-11T03:49:28.269 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.269 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.269 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.269 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.502077+0000", 2024-09-11T03:49:28.269 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7890000000000002e-05 2024-09-11T03:49:28.269 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.270 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.270 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.270 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.502152+0000", 2024-09-11T03:49:28.270 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4819000000000003e-05 2024-09-11T03:49:28.270 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.270 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.270 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.270 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.270 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.270 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:4288 2.0 e462/417 2:0762776a:::smithi17557870-25:head v 462'115, mlcod=393'111)", 2024-09-11T03:49:28.270 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:39.987982+0000", 2024-09-11T03:49:28.270 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 288.26926699900002, 2024-09-11T03:49:28.270 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013639949, 2024-09-11T03:49:28.270 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.271 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.271 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:28.271 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.271 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.271 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.271 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.987982+0000", 2024-09-11T03:49:28.271 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.271 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.271 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.271 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.271 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.987982+0000", 2024-09-11T03:49:28.271 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.271 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.271 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.272 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.272 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.987982+0000", 2024-09-11T03:49:28.272 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.65e-07 2024-09-11T03:49:28.272 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.272 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.272 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.272 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.997342+0000", 2024-09-11T03:49:28.272 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0093600669999999997 2024-09-11T03:49:28.272 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.272 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.272 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.273 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.997361+0000", 2024-09-11T03:49:28.273 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8719000000000002e-05 2024-09-11T03:49:28.273 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.274 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.274 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.274 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.997370+0000", 2024-09-11T03:49:28.274 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4259999999999992e-06 2024-09-11T03:49:28.274 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.274 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.274 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.274 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.997462+0000", 2024-09-11T03:49:28.274 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2015000000000006e-05 2024-09-11T03:49:28.274 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.274 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.274 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.274 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.997496+0000", 2024-09-11T03:49:28.275 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3837999999999997e-05 2024-09-11T03:49:28.275 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.275 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.275 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:28.275 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:40.001567+0000", 2024-09-11T03:49:28.275 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0040706969999999999 2024-09-11T03:49:28.275 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.275 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.275 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.275 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:40.001622+0000", 2024-09-11T03:49:28.275 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4922e-05 2024-09-11T03:49:28.275 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.275 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.275 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.276 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.276 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.276 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4896 2.7 2:e0eabcfb:::smithi17557870-9:head [sparse-read 0~1248271] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e522)", 2024-09-11T03:49:28.276 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:40.069780+0000", 2024-09-11T03:49:28.276 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 228.187469007, 2024-09-11T03:49:28.276 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0086472939999999998, 2024-09-11T03:49:28.276 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.276 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.276 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:49:28.276 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:28.276 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:28.276 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:28.276 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4896 2024-09-11T03:49:28.277 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.277 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.277 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.277 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.277 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069780+0000", 2024-09-11T03:49:28.277 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.277 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.277 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.277 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.277 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069780+0000", 2024-09-11T03:49:28.277 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.277 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.277 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.277 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.278 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069780+0000", 2024-09-11T03:49:28.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6800000000000001e-07 2024-09-11T03:49:28.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.278 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.278 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.278 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069782+0000", 2024-09-11T03:49:28.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.72e-06 2024-09-11T03:49:28.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.278 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.278 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.278 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069787+0000", 2024-09-11T03:49:28.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7209999999999999e-06 2024-09-11T03:49:28.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.279 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.279 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.279 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.069793+0000", 2024-09-11T03:49:28.279 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4949999999999999e-06 2024-09-11T03:49:28.279 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.279 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.279 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.279 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.078273+0000", 2024-09-11T03:49:28.279 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0084799130000000004 2024-09-11T03:49:28.280 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.280 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.280 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.280 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.078336+0000", 2024-09-11T03:49:28.280 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2854000000000005e-05 2024-09-11T03:49:28.280 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.280 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.280 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.280 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.078427+0000", 2024-09-11T03:49:28.280 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1223000000000004e-05 2024-09-11T03:49:28.280 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.280 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.280 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.281 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.281 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.281 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4907 2.7 2:e0ec5763:::smithi17557870-2:head [copy-from ver 761] snapc d7=[d7,d5,d4,d3,d2,d1,d0,ca] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-09-11T03:49:28.281 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:40.106137+0000", 2024-09-11T03:49:28.281 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 228.151111843, 2024-09-11T03:49:28.281 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021335674999999998, 2024-09-11T03:49:28.281 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.281 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.281 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:28.281 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:28.281 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:28.281 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:28.281 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4907 2024-09-11T03:49:28.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.282 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.282 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.282 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106137+0000", 2024-09-11T03:49:28.282 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.282 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.282 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106137+0000", 2024-09-11T03:49:28.282 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.283 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.283 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106139+0000", 2024-09-11T03:49:28.283 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3250000000000002e-06 2024-09-11T03:49:28.283 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.283 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.283 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106149+0000", 2024-09-11T03:49:28.283 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8079999999999999e-06 2024-09-11T03:49:28.283 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.283 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.283 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106171+0000", 2024-09-11T03:49:28.283 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2507999999999999e-05 2024-09-11T03:49:28.283 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.284 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.284 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106184+0000", 2024-09-11T03:49:28.284 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2224e-05 2024-09-11T03:49:28.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.284 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.284 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106253+0000", 2024-09-11T03:49:28.284 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9615000000000003e-05 2024-09-11T03:49:28.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.284 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.284 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106357+0000", 2024-09-11T03:49:28.285 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010383900000000001 2024-09-11T03:49:28.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.285 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:49:28.285 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.116615+0000", 2024-09-11T03:49:28.285 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010257779999999999 2024-09-11T03:49:28.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.285 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:28.285 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.120710+0000", 2024-09-11T03:49:28.285 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0040951579999999998 2024-09-11T03:49:28.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.285 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:28.286 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.127226+0000", 2024-09-11T03:49:28.286 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0065160610000000001 2024-09-11T03:49:28.286 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.286 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:28.286 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.127380+0000", 2024-09-11T03:49:28.286 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015357599999999999 2024-09-11T03:49:28.286 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.286 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:28.286 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.127423+0000", 2024-09-11T03:49:28.286 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3680999999999997e-05 2024-09-11T03:49:28.286 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.287 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.287 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.287 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.127473+0000", 2024-09-11T03:49:28.287 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9100000000000001e-05 2024-09-11T03:49:28.287 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.287 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.287 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.287 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.287 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.287 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4908 2.7 2:e0ec5763:::smithi17557870-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e522)", 2024-09-11T03:49:28.287 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:40.106211+0000", 2024-09-11T03:49:28.287 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 228.15103809999999, 2024-09-11T03:49:28.287 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013824826, 2024-09-11T03:49:28.287 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.288 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.288 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:49:28.288 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:28.288 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:28.288 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:28.288 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4908 2024-09-11T03:49:28.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.288 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.288 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.288 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.288 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106211+0000", 2024-09-11T03:49:28.288 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.289 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.289 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106211+0000", 2024-09-11T03:49:28.289 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.289 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.289 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106211+0000", 2024-09-11T03:49:28.289 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4200000000000001e-07 2024-09-11T03:49:28.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.289 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.289 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106214+0000", 2024-09-11T03:49:28.290 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9490000000000001e-06 2024-09-11T03:49:28.290 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.290 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.290 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.290 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106221+0000", 2024-09-11T03:49:28.290 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3300000000000001e-06 2024-09-11T03:49:28.290 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.290 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.290 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.290 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106225+0000", 2024-09-11T03:49:28.290 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5700000000000001e-06 2024-09-11T03:49:28.290 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.290 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.291 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.291 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106521+0000", 2024-09-11T03:49:28.291 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00029598599999999998 2024-09-11T03:49:28.291 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.291 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.291 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:49:28.291 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106551+0000", 2024-09-11T03:49:28.291 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0165e-05 2024-09-11T03:49:28.291 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.291 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.291 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.291 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.119851+0000", 2024-09-11T03:49:28.291 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013300354 2024-09-11T03:49:28.291 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.292 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.292 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.292 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.119919+0000", 2024-09-11T03:49:28.292 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7569000000000003e-05 2024-09-11T03:49:28.292 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.292 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.292 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.292 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.120035+0000", 2024-09-11T03:49:28.292 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000116461 2024-09-11T03:49:28.292 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.292 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.292 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.292 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.293 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.293 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4937 2.7 2:e23d1762:::smithi17557870-21:head [read 0~3815288,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e526)", 2024-09-11T03:49:28.293 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:43.953926+0000", 2024-09-11T03:49:28.293 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 224.30332244900001, 2024-09-11T03:49:28.293 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010177080999999999, 2024-09-11T03:49:28.293 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.293 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.293 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:49:28.293 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:28.293 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:28.293 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:28.293 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4937 2024-09-11T03:49:28.293 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.293 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.294 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.294 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.294 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953926+0000", 2024-09-11T03:49:28.294 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.294 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.294 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.294 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.294 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953926+0000", 2024-09-11T03:49:28.294 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.294 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.294 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.294 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.294 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953927+0000", 2024-09-11T03:49:28.295 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1699999999999997e-07 2024-09-11T03:49:28.295 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.295 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.295 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.295 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953933+0000", 2024-09-11T03:49:28.295 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5050000000000004e-06 2024-09-11T03:49:28.295 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.295 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.295 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.295 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953945+0000", 2024-09-11T03:49:28.295 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1202e-05 2024-09-11T03:49:28.295 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.295 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.295 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.296 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953951+0000", 2024-09-11T03:49:28.296 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.708e-06 2024-09-11T03:49:28.296 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.296 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.296 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.296 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953988+0000", 2024-09-11T03:49:28.296 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6356999999999998e-05 2024-09-11T03:49:28.296 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.296 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.296 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:49:28.296 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953992+0000", 2024-09-11T03:49:28.296 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5469999999999998e-06 2024-09-11T03:49:28.296 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.297 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.297 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.297 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.962348+0000", 2024-09-11T03:49:28.297 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0083553180000000005 2024-09-11T03:49:28.297 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.297 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.297 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.297 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.962519+0000", 2024-09-11T03:49:28.297 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000171817 2024-09-11T03:49:28.297 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.297 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.297 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.297 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.964103+0000", 2024-09-11T03:49:28.297 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00158391 2024-09-11T03:49:28.298 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.298 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.298 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.298 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.298 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.298 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:5650 2.1e e599/574 2:7b0c3397:::smithi17557870-14:head v 599'866, mlcod=597'864)", 2024-09-11T03:49:28.298 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:56.992686+0000", 2024-09-11T03:49:28.298 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 151.26456307000001, 2024-09-11T03:49:28.298 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010088503, 2024-09-11T03:49:28.298 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.298 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.298 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:28.298 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.298 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.299 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.299 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:56.992686+0000", 2024-09-11T03:49:28.299 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.299 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.299 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.299 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.299 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:56.992686+0000", 2024-09-11T03:49:28.299 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.299 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.299 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.299 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.299 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:56.992686+0000", 2024-09-11T03:49:28.299 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.17e-07 2024-09-11T03:49:28.300 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.300 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.300 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.300 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:57.000330+0000", 2024-09-11T03:49:28.300 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0076439680000000001 2024-09-11T03:49:28.300 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.300 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.300 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.300 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:57.000346+0000", 2024-09-11T03:49:28.300 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6376e-05 2024-09-11T03:49:28.300 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.300 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.300 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.301 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:57.000355+0000", 2024-09-11T03:49:28.301 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.9770000000000006e-06 2024-09-11T03:49:28.301 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.301 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.301 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.301 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:57.000428+0000", 2024-09-11T03:49:28.301 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2843999999999996e-05 2024-09-11T03:49:28.301 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.301 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.301 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.301 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:57.000457+0000", 2024-09-11T03:49:28.301 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8745999999999999e-05 2024-09-11T03:49:28.301 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.302 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.302 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:28.302 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:57.002715+0000", 2024-09-11T03:49:28.302 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022582549999999998 2024-09-11T03:49:28.302 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.302 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.302 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.302 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:57.002774+0000", 2024-09-11T03:49:28.302 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9120000000000003e-05 2024-09-11T03:49:28.302 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.302 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.302 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.302 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.302 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.303 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:5783 2.1e e608/574 2:7b0c3397:::smithi17557870-14:head v 608'869, mlcod=599'866)", 2024-09-11T03:49:28.303 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:06.035596+0000", 2024-09-11T03:49:28.303 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 142.22165306100001, 2024-09-11T03:49:28.303 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0091205959999999999, 2024-09-11T03:49:28.303 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.303 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.303 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:28.303 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.303 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.303 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.303 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.035596+0000", 2024-09-11T03:49:28.303 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.303 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.303 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.304 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.304 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.035596+0000", 2024-09-11T03:49:28.304 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.304 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.304 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.304 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.304 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.035597+0000", 2024-09-11T03:49:28.304 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.449e-06 2024-09-11T03:49:28.304 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.304 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.304 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.304 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.036120+0000", 2024-09-11T03:49:28.304 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00052300299999999997 2024-09-11T03:49:28.305 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.305 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.305 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.305 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.036136+0000", 2024-09-11T03:49:28.305 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5820000000000001e-05 2024-09-11T03:49:28.305 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.305 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.305 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.305 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.036144+0000", 2024-09-11T03:49:28.305 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9920000000000007e-06 2024-09-11T03:49:28.305 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.305 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.305 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.306 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.036203+0000", 2024-09-11T03:49:28.306 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8937999999999997e-05 2024-09-11T03:49:28.306 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.306 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.306 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.306 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.036221+0000", 2024-09-11T03:49:28.306 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8136999999999999e-05 2024-09-11T03:49:28.306 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.306 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.306 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:28.306 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044680+0000", 2024-09-11T03:49:28.306 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0084586750000000006 2024-09-11T03:49:28.306 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.307 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.307 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.307 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044716+0000", 2024-09-11T03:49:28.307 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6582000000000003e-05 2024-09-11T03:49:28.307 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.307 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.307 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.307 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.307 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.307 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6214 2.12 2:4e5019e5:::smithi17557870-40:head [read 0~3966617,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e646)", 2024-09-11T03:49:28.307 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:43.987667+0000", 2024-09-11T03:49:28.307 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 104.269581171, 2024-09-11T03:49:28.307 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0091709329999999992, 2024-09-11T03:49:28.307 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.308 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.308 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:49:28.308 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:28.308 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:28.308 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:28.308 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6214 2024-09-11T03:49:28.308 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.308 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.308 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.308 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.308 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.987667+0000", 2024-09-11T03:49:28.308 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.308 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.308 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.309 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.309 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.987667+0000", 2024-09-11T03:49:28.309 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.309 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.309 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.309 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.309 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.987668+0000", 2024-09-11T03:49:28.309 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.08e-07 2024-09-11T03:49:28.309 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.309 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.309 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.309 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.987671+0000", 2024-09-11T03:49:28.309 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1310000000000002e-06 2024-09-11T03:49:28.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.310 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.310 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.987682+0000", 2024-09-11T03:49:28.310 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0541e-05 2024-09-11T03:49:28.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.310 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.310 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.987687+0000", 2024-09-11T03:49:28.310 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.186e-06 2024-09-11T03:49:28.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.310 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.310 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.987821+0000", 2024-09-11T03:49:28.311 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000134012 2024-09-11T03:49:28.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.311 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:49:28.311 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.987827+0000", 2024-09-11T03:49:28.311 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.818e-06 2024-09-11T03:49:28.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.311 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.311 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.993522+0000", 2024-09-11T03:49:28.311 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0056953500000000001 2024-09-11T03:49:28.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.311 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.312 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.993643+0000", 2024-09-11T03:49:28.312 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012071800000000001 2024-09-11T03:49:28.312 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.312 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.312 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.312 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:43.996838+0000", 2024-09-11T03:49:28.312 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0031957690000000002 2024-09-11T03:49:28.312 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.312 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.312 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.312 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.312 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.312 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6557 2.1c 2:3c007163:::smithi17557870-35:head [copy-from ver 891] snapc 120=[120,11f,11e,11d,11a,119,114] ondisk+write+known_if_redirected+supports_pool_eio e670)", 2024-09-11T03:49:28.313 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:08.152129+0000", 2024-09-11T03:49:28.313 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 80.105119985000002, 2024-09-11T03:49:28.313 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.024595260000000001, 2024-09-11T03:49:28.313 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.313 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.313 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:28.313 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:28.313 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:28.313 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:28.313 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6557 2024-09-11T03:49:28.313 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.314 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.314 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.314 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.314 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152129+0000", 2024-09-11T03:49:28.314 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.314 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.314 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.314 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.314 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152129+0000", 2024-09-11T03:49:28.315 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.315 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.315 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.315 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.315 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152132+0000", 2024-09-11T03:49:28.315 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8650000000000001e-06 2024-09-11T03:49:28.315 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.315 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.315 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.315 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152146+0000", 2024-09-11T03:49:28.316 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4438e-05 2024-09-11T03:49:28.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.316 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.316 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152177+0000", 2024-09-11T03:49:28.316 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1149999999999998e-05 2024-09-11T03:49:28.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.316 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.316 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152191+0000", 2024-09-11T03:49:28.316 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4022000000000001e-05 2024-09-11T03:49:28.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.317 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.317 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152324+0000", 2024-09-11T03:49:28.317 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013306799999999999 2024-09-11T03:49:28.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.317 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.317 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:49:28.317 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152332+0000", 2024-09-11T03:49:28.317 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0490000000000004e-06 2024-09-11T03:49:28.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.317 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.317 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.317 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.154966+0000", 2024-09-11T03:49:28.317 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0026336900000000002 2024-09-11T03:49:28.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.318 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.318 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.318 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.155098+0000", 2024-09-11T03:49:28.318 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013232700000000001 2024-09-11T03:49:28.318 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.318 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.318 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,8", 2024-09-11T03:49:28.318 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.166673+0000", 2024-09-11T03:49:28.318 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011575062000000001 2024-09-11T03:49:28.318 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.318 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.318 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:28.318 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.172194+0000", 2024-09-11T03:49:28.319 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.005520619 2024-09-11T03:49:28.319 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.319 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.319 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:28.319 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.175894+0000", 2024-09-11T03:49:28.319 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003700451 2024-09-11T03:49:28.319 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.319 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.319 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:28.319 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.176612+0000", 2024-09-11T03:49:28.319 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00071733300000000001 2024-09-11T03:49:28.319 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.319 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.320 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:28.320 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.176669+0000", 2024-09-11T03:49:28.320 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6949e-05 2024-09-11T03:49:28.320 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.320 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.320 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.320 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.176724+0000", 2024-09-11T03:49:28.320 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5237e-05 2024-09-11T03:49:28.320 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.320 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.320 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.320 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.320 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.320 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6558 2.1c 2:3c007163:::smithi17557870-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e670)", 2024-09-11T03:49:28.321 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:08.152229+0000", 2024-09-11T03:49:28.321 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 80.105019229999996, 2024-09-11T03:49:28.321 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019338949000000001, 2024-09-11T03:49:28.321 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.321 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.321 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:49:28.321 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:28.321 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:28.321 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:28.321 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6558 2024-09-11T03:49:28.321 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.321 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.321 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.321 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.322 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152229+0000", 2024-09-11T03:49:28.322 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.322 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.322 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.322 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.322 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152229+0000", 2024-09-11T03:49:28.322 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.322 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.322 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.322 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.322 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152230+0000", 2024-09-11T03:49:28.322 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4600000000000004e-07 2024-09-11T03:49:28.322 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.323 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.323 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.323 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152234+0000", 2024-09-11T03:49:28.323 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5719999999999999e-06 2024-09-11T03:49:28.323 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.323 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.323 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.323 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152247+0000", 2024-09-11T03:49:28.323 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2852e-05 2024-09-11T03:49:28.323 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.323 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.323 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.323 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152254+0000", 2024-09-11T03:49:28.323 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2559999999999996e-06 2024-09-11T03:49:28.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.324 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.324 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.324 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152405+0000", 2024-09-11T03:49:28.324 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015162500000000001 2024-09-11T03:49:28.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.324 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.324 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:49:28.324 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152411+0000", 2024-09-11T03:49:28.324 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1880000000000001e-06 2024-09-11T03:49:28.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.324 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.324 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.325 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.155324+0000", 2024-09-11T03:49:28.325 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0029129749999999999 2024-09-11T03:49:28.325 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.325 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.325 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:49:28.325 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.155374+0000", 2024-09-11T03:49:28.325 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0581999999999998e-05 2024-09-11T03:49:28.325 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.325 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.325 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.325 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.171261+0000", 2024-09-11T03:49:28.325 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.01588674 2024-09-11T03:49:28.325 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.325 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.326 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.326 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.171402+0000", 2024-09-11T03:49:28.326 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000140684 2024-09-11T03:49:28.326 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.326 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.326 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.326 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.171568+0000", 2024-09-11T03:49:28.326 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016682900000000001 2024-09-11T03:49:28.326 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.326 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.326 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.326 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.326 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.327 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:6914 2.4 e697/432 2:220bb244:::smithi17557870-34:head v 697'549, mlcod=691'546)", 2024-09-11T03:49:28.327 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:35.033956+0000", 2024-09-11T03:49:28.327 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 53.223292198000003, 2024-09-11T03:49:28.327 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0094575240000000001, 2024-09-11T03:49:28.327 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.327 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.327 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:28.327 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.327 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.327 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.327 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:35.033956+0000", 2024-09-11T03:49:28.327 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.327 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.327 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.328 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.328 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:35.033956+0000", 2024-09-11T03:49:28.328 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.328 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.328 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.328 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.328 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:35.033957+0000", 2024-09-11T03:49:28.328 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9399999999999999e-07 2024-09-11T03:49:28.328 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.328 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.328 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.329 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:35.040150+0000", 2024-09-11T03:49:28.329 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0061933129999999998 2024-09-11T03:49:28.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.329 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.329 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.329 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:35.040175+0000", 2024-09-11T03:49:28.329 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5018999999999999e-05 2024-09-11T03:49:28.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.329 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.329 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.329 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:35.040186+0000", 2024-09-11T03:49:28.329 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0849999999999999e-05 2024-09-11T03:49:28.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.330 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.330 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.330 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:35.040262+0000", 2024-09-11T03:49:28.330 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5834999999999995e-05 2024-09-11T03:49:28.330 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.330 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.330 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.330 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:35.040282+0000", 2024-09-11T03:49:28.330 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9862999999999999e-05 2024-09-11T03:49:28.330 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.330 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.330 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:28.330 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:35.043364+0000", 2024-09-11T03:49:28.331 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030825129999999998 2024-09-11T03:49:28.331 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.331 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.331 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.331 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:35.043414+0000", 2024-09-11T03:49:28.331 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9937e-05 2024-09-11T03:49:28.331 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.331 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.331 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.331 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.331 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.331 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7043 2.12 2:494ba932:::smithi17557870-43:head [copy-from ver 979] snapc 13a=[13a,139,138,137,136,135,133,130,12e,12d,12c,12a,127,126,123,120,11f,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e709)", 2024-09-11T03:49:28.331 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:47.006408+0000", 2024-09-11T03:49:28.331 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 41.25084056, 2024-09-11T03:49:28.332 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.033513329000000001, 2024-09-11T03:49:28.332 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.332 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.332 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:28.332 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:28.332 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:28.332 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:28.333 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:28.333 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7043 2024-09-11T03:49:28.334 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.334 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.334 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.334 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.334 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006408+0000", 2024-09-11T03:49:28.334 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.334 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.334 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.334 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.334 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006408+0000", 2024-09-11T03:49:28.334 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.334 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.335 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.335 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.335 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006411+0000", 2024-09-11T03:49:28.335 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6869999999999999e-06 2024-09-11T03:49:28.335 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.335 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.335 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.335 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006423+0000", 2024-09-11T03:49:28.335 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1762e-05 2024-09-11T03:49:28.335 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.335 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.335 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.335 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006462+0000", 2024-09-11T03:49:28.335 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9755999999999997e-05 2024-09-11T03:49:28.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.336 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.336 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.336 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006479+0000", 2024-09-11T03:49:28.336 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7031e-05 2024-09-11T03:49:28.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.336 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.336 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.336 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006583+0000", 2024-09-11T03:49:28.336 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010370999999999999 2024-09-11T03:49:28.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.336 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.336 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:49:28.337 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006592+0000", 2024-09-11T03:49:28.337 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2199999999999998e-06 2024-09-11T03:49:28.337 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.337 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.337 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.337 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.009741+0000", 2024-09-11T03:49:28.337 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0031483639999999999 2024-09-11T03:49:28.337 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.337 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.337 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.337 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.009831+0000", 2024-09-11T03:49:28.337 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.0351000000000003e-05 2024-09-11T03:49:28.337 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.338 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.338 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:49:28.338 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.030947+0000", 2024-09-11T03:49:28.338 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.02111586 2024-09-11T03:49:28.338 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.338 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.338 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:28.338 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.034462+0000", 2024-09-11T03:49:28.338 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035156279999999998 2024-09-11T03:49:28.338 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.338 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.338 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:28.338 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.039094+0000", 2024-09-11T03:49:28.339 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0046314779999999996 2024-09-11T03:49:28.339 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.339 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.339 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:28.339 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.039783+0000", 2024-09-11T03:49:28.339 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00068866399999999996 2024-09-11T03:49:28.339 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.339 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.339 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:28.339 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.039883+0000", 2024-09-11T03:49:28.339 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000100641 2024-09-11T03:49:28.339 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.339 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.340 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.340 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.039921+0000", 2024-09-11T03:49:28.340 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8176999999999997e-05 2024-09-11T03:49:28.340 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.340 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.340 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.340 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.340 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.340 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7044 2.12 2:494ba932:::smithi17557870-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e709)", 2024-09-11T03:49:28.340 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:47.006515+0000", 2024-09-11T03:49:28.340 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 41.250733513999997, 2024-09-11T03:49:28.340 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.027479966000000002, 2024-09-11T03:49:28.340 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.340 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.341 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:49:28.341 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:28.341 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:28.341 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:28.341 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7044 2024-09-11T03:49:28.341 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.341 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.341 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.341 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.341 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006515+0000", 2024-09-11T03:49:28.341 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.341 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.341 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.342 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.342 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006515+0000", 2024-09-11T03:49:28.342 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.342 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.342 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.342 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.342 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006516+0000", 2024-09-11T03:49:28.342 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1999999999999999e-07 2024-09-11T03:49:28.342 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.342 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.342 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.342 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006520+0000", 2024-09-11T03:49:28.342 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5210000000000004e-06 2024-09-11T03:49:28.342 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.343 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.343 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.343 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006545+0000", 2024-09-11T03:49:28.343 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4623000000000001e-05 2024-09-11T03:49:28.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.343 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.343 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.343 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006556+0000", 2024-09-11T03:49:28.343 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0844e-05 2024-09-11T03:49:28.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.343 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.343 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.343 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006713+0000", 2024-09-11T03:49:28.344 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015744899999999999 2024-09-11T03:49:28.344 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.344 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.344 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:49:28.344 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006718+0000", 2024-09-11T03:49:28.344 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9119999999999997e-06 2024-09-11T03:49:28.344 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.344 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.344 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.344 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.010037+0000", 2024-09-11T03:49:28.344 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003318283 2024-09-11T03:49:28.344 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.344 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.344 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:49:28.345 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.010071+0000", 2024-09-11T03:49:28.345 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4344999999999997e-05 2024-09-11T03:49:28.345 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.345 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.345 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.345 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.033776+0000", 2024-09-11T03:49:28.345 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.023705086 2024-09-11T03:49:28.345 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.345 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.345 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.345 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.033886+0000", 2024-09-11T03:49:28.345 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000110491 2024-09-11T03:49:28.345 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.345 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.346 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.346 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.033995+0000", 2024-09-11T03:49:28.346 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108692 2024-09-11T03:49:28.346 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.346 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.346 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.346 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.346 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7049 2.1c 2:3c007163:::smithi17557870-35:head [copy-from ver 947] snapc 13a=[13a,139,138,137,136,135,133,130,12e,12d,12c,12a,127,126,123,120,11f,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e709)", 2024-09-11T03:49:28.346 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:47.006590+0000", 2024-09-11T03:49:28.346 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 41.250659122000002, 2024-09-11T03:49:28.346 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021051984999999999, 2024-09-11T03:49:28.346 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.347 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.347 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:28.347 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:28.347 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:28.347 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:28.347 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7049 2024-09-11T03:49:28.347 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.347 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.347 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.347 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.347 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006590+0000", 2024-09-11T03:49:28.347 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.347 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.348 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.348 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.348 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006590+0000", 2024-09-11T03:49:28.348 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.348 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.348 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.348 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.348 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006590+0000", 2024-09-11T03:49:28.348 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7100000000000001e-07 2024-09-11T03:49:28.348 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.348 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.348 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.348 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006596+0000", 2024-09-11T03:49:28.349 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2870000000000004e-06 2024-09-11T03:49:28.349 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.349 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.349 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.349 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006610+0000", 2024-09-11T03:49:28.349 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4781e-05 2024-09-11T03:49:28.349 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.349 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.349 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.349 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006617+0000", 2024-09-11T03:49:28.349 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0099999999999998e-06 2024-09-11T03:49:28.349 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.349 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.350 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.350 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006678+0000", 2024-09-11T03:49:28.350 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0269999999999997e-05 2024-09-11T03:49:28.350 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.350 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.350 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:49:28.350 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006685+0000", 2024-09-11T03:49:28.350 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.311e-06 2024-09-11T03:49:28.350 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.350 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.350 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.350 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.008762+0000", 2024-09-11T03:49:28.350 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0020774719999999999 2024-09-11T03:49:28.350 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.351 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.351 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.351 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.008877+0000", 2024-09-11T03:49:28.351 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114695 2024-09-11T03:49:28.351 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.351 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.351 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,8", 2024-09-11T03:49:28.351 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.018702+0000", 2024-09-11T03:49:28.351 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0098247839999999996 2024-09-11T03:49:28.351 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.351 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.351 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:28.351 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.023780+0000", 2024-09-11T03:49:28.352 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050777699999999997 2024-09-11T03:49:28.352 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.352 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.352 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:49:28.352 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.024140+0000", 2024-09-11T03:49:28.352 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000360219 2024-09-11T03:49:28.352 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.352 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.352 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:49:28.352 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.027423+0000", 2024-09-11T03:49:28.352 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032829309999999998 2024-09-11T03:49:28.352 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.352 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.353 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:28.353 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.027580+0000", 2024-09-11T03:49:28.353 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015743199999999999 2024-09-11T03:49:28.353 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.353 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.353 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.353 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.027642+0000", 2024-09-11T03:49:28.353 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1252000000000002e-05 2024-09-11T03:49:28.353 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.353 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.353 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.353 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.353 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.353 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7050 2.1c 2:3c007163:::smithi17557870-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e709)", 2024-09-11T03:49:28.354 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:47.006639+0000", 2024-09-11T03:49:28.354 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 41.250609888, 2024-09-11T03:49:28.354 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016888205999999999, 2024-09-11T03:49:28.354 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.354 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.354 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:49:28.354 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:49:28.354 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:49:28.354 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:49:28.354 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7050 2024-09-11T03:49:28.354 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.354 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.354 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.355 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.355 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006639+0000", 2024-09-11T03:49:28.355 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.355 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.355 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.355 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.355 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006639+0000", 2024-09-11T03:49:28.355 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.355 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.355 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.355 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.355 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006639+0000", 2024-09-11T03:49:28.355 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9299999999999998e-07 2024-09-11T03:49:28.356 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.356 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.356 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.356 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006643+0000", 2024-09-11T03:49:28.356 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3129999999999999e-06 2024-09-11T03:49:28.356 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.356 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.356 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.356 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006650+0000", 2024-09-11T03:49:28.356 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8040000000000002e-06 2024-09-11T03:49:28.356 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.356 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.356 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.357 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006656+0000", 2024-09-11T03:49:28.357 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9229999999999999e-06 2024-09-11T03:49:28.357 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.357 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.357 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.357 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006711+0000", 2024-09-11T03:49:28.357 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4710999999999998e-05 2024-09-11T03:49:28.357 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.357 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.357 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:49:28.357 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006717+0000", 2024-09-11T03:49:28.357 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4e-06 2024-09-11T03:49:28.357 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.358 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.358 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.009041+0000", 2024-09-11T03:49:28.358 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002324497 2024-09-11T03:49:28.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.358 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:49:28.358 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.009084+0000", 2024-09-11T03:49:28.358 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2896999999999999e-05 2024-09-11T03:49:28.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.358 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.358 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.023331+0000", 2024-09-11T03:49:28.359 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014247344 2024-09-11T03:49:28.359 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.359 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.359 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.359 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.023430+0000", 2024-09-11T03:49:28.359 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8884999999999999e-05 2024-09-11T03:49:28.359 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.359 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.359 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.359 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.023527+0000", 2024-09-11T03:49:28.359 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6838999999999999e-05 2024-09-11T03:49:28.359 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.359 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.359 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.360 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.360 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.360 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:7045 2.4 e709/432 2:25f5f8b9:::smithi17557870-26:head v 709'563, mlcod=706'559)", 2024-09-11T03:49:28.360 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:47.025349+0000", 2024-09-11T03:49:28.360 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 41.231899370000001, 2024-09-11T03:49:28.360 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011744064, 2024-09-11T03:49:28.360 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.360 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.360 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:28.360 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.360 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.360 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.360 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.025349+0000", 2024-09-11T03:49:28.360 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.361 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.361 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.361 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.361 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.025349+0000", 2024-09-11T03:49:28.361 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.361 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.361 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.361 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.361 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.025351+0000", 2024-09-11T03:49:28.361 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4470000000000001e-06 2024-09-11T03:49:28.361 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.361 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.361 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.361 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.033663+0000", 2024-09-11T03:49:28.362 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0083124940000000001 2024-09-11T03:49:28.362 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.362 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.362 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.362 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.033679+0000", 2024-09-11T03:49:28.362 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5690999999999999e-05 2024-09-11T03:49:28.362 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.362 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.362 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.362 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.033690+0000", 2024-09-11T03:49:28.362 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1372000000000001e-05 2024-09-11T03:49:28.362 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.362 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.363 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.363 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.033798+0000", 2024-09-11T03:49:28.363 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010728799999999999 2024-09-11T03:49:28.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.363 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.363 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.363 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.033824+0000", 2024-09-11T03:49:28.363 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6092999999999999e-05 2024-09-11T03:49:28.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.363 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.363 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:28.363 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.037044+0000", 2024-09-11T03:49:28.363 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032200919999999999 2024-09-11T03:49:28.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.364 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.364 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.364 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.037093+0000", 2024-09-11T03:49:28.364 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9586999999999998e-05 2024-09-11T03:49:28.364 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.364 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.364 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.364 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.364 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.364 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:7071 2.7 e710/560 2:e23d1762:::smithi17557870-21:head v 710'1007, mlcod=710'1003)", 2024-09-11T03:49:28.364 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:48.044105+0000", 2024-09-11T03:49:28.364 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 40.213143723000002, 2024-09-11T03:49:28.364 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010103607000000001, 2024-09-11T03:49:28.365 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.365 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.365 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:28.365 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.365 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.365 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.365 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:48.044105+0000", 2024-09-11T03:49:28.365 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.365 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.365 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.365 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.365 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:48.044105+0000", 2024-09-11T03:49:28.365 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.366 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.366 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.366 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.366 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:48.044105+0000", 2024-09-11T03:49:28.366 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7899999999999999e-07 2024-09-11T03:49:28.366 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.366 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.366 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.366 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:48.051966+0000", 2024-09-11T03:49:28.366 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0078602050000000003 2024-09-11T03:49:28.366 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.366 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.366 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.367 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:48.051985+0000", 2024-09-11T03:49:28.367 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8955000000000001e-05 2024-09-11T03:49:28.367 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.367 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.367 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.367 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:48.052011+0000", 2024-09-11T03:49:28.367 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6381999999999999e-05 2024-09-11T03:49:28.367 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.367 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.367 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.367 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:48.052073+0000", 2024-09-11T03:49:28.367 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2187000000000006e-05 2024-09-11T03:49:28.367 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.367 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.368 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.368 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:48.052099+0000", 2024-09-11T03:49:28.368 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5541000000000002e-05 2024-09-11T03:49:28.368 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.368 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.368 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:28.368 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:48.054168+0000", 2024-09-11T03:49:28.368 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002069226 2024-09-11T03:49:28.368 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.368 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.368 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.368 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:48.054209+0000", 2024-09-11T03:49:28.368 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0632e-05 2024-09-11T03:49:28.369 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.369 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.369 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.369 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.369 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.369 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:7384 2.7 e743/560 2:e0eabcfb:::smithi17557870-9:head v 743'1051, mlcod=743'1048)", 2024-09-11T03:49:28.369 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:49:21.121268+0000", 2024-09-11T03:49:28.369 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 7.1359802730000004, 2024-09-11T03:49:28.369 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0099859479999999997, 2024-09-11T03:49:28.369 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:49:28.369 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:49:28.369 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:49:28.369 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:49:28.370 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.370 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:49:28.370 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:21.121268+0000", 2024-09-11T03:49:28.370 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.370 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.370 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.370 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:49:28.370 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:21.121268+0000", 2024-09-11T03:49:28.370 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:49:28.370 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.370 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.370 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:49:28.370 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:21.121270+0000", 2024-09-11T03:49:28.370 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.64e-06 2024-09-11T03:49:28.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.371 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.371 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:49:28.371 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:21.128685+0000", 2024-09-11T03:49:28.371 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0074152319999999999 2024-09-11T03:49:28.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.371 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.371 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:49:28.371 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:21.128703+0000", 2024-09-11T03:49:28.371 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7884e-05 2024-09-11T03:49:28.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.371 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.371 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:49:28.371 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:21.128729+0000", 2024-09-11T03:49:28.372 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5701999999999999e-05 2024-09-11T03:49:28.372 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.372 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.372 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:49:28.372 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:21.128788+0000", 2024-09-11T03:49:28.372 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9169000000000002e-05 2024-09-11T03:49:28.372 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.372 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.372 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:49:28.372 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:21.128805+0000", 2024-09-11T03:49:28.372 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.734e-05 2024-09-11T03:49:28.372 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.372 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.372 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:49:28.373 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:21.131215+0000", 2024-09-11T03:49:28.373 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024097760000000002 2024-09-11T03:49:28.373 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:49:28.373 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:49:28.373 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:49:28.373 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:21.131254+0000", 2024-09-11T03:49:28.373 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9205000000000002e-05 2024-09-11T03:49:28.373 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.373 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.373 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.373 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:49:28.373 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:49:28.373 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:49:28.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:28 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:49:28.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:28 smithi084 ceph-mon[27254]: pgmap v1278: 77 pgs: 4 active+clean+scrubbing, 4 remapped+peering, 6 unknown, 63 active+clean; 358 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:49:28.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:28 smithi084 ceph-mon[27254]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-11T03:49:28.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:28 smithi084 ceph-mon[27254]: osdmap e749: 12 total, 12 up, 12 in 2024-09-11T03:49:28.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:28 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:49:28.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:28 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:49:28.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:28 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:49:28.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:28 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:49:28.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:28 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:49:28.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:28 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3949686818' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:49:28.396 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:28.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:28.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:28 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:49:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:28 smithi121 ceph-mon[30729]: pgmap v1278: 77 pgs: 4 active+clean+scrubbing, 4 remapped+peering, 6 unknown, 63 active+clean; 358 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:49:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:28 smithi121 ceph-mon[30729]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-11T03:49:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:28 smithi121 ceph-mon[30729]: osdmap e749: 12 total, 12 up, 12 in 2024-09-11T03:49:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:28 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:49:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:28 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:49:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:28 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:49:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:28 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:49:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:28 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:49:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:28 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3949686818' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:49:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:28 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:49:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:28 smithi175 ceph-mon[30894]: pgmap v1278: 77 pgs: 4 active+clean+scrubbing, 4 remapped+peering, 6 unknown, 63 active+clean; 358 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:49:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:28 smithi175 ceph-mon[30894]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-11T03:49:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:28 smithi175 ceph-mon[30894]: osdmap e749: 12 total, 12 up, 12 in 2024-09-11T03:49:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:28 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:49:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:28 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:49:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:28 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:49:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:28 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:49:28.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:28 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:49:28.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:28 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3949686818' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:49:28.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:28.519 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:28.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:28.620 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:28.641 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:28.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:28.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:28.698 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:49:28.743 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:28.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:28.765 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:28.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:28.866 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:28.889 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:28.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:28.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:28.948 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:28.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:29.017 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:29.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:29.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:29.085 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-11T03:49:29.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:29.144 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.7 primary-affinity to 0.982715 (8644032) 2024-09-11T03:49:29.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:29.145 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:29.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:29.152 INFO:tasks.rados.rados.2.smithi175.stdout:2990: done (0 left) 2024-09-11T03:49:29.152 INFO:tasks.rados.rados.2.smithi175.stdout:2991: write oid 16 current snap is 337 2024-09-11T03:49:29.152 INFO:tasks.rados.rados.2.smithi175.stdout:2991: seq_num 1024 ranges {410368=646278,1472267=475349,2504321=405600} 2024-09-11T03:49:29.158 INFO:tasks.rados.rados.2.smithi175.stdout:2991: writing smithi17557870-16 from 410368 to 1056646 tid 1 2024-09-11T03:49:29.164 INFO:tasks.rados.rados.2.smithi175.stdout:2991: writing smithi17557870-16 from 1472267 to 1947616 tid 2 2024-09-11T03:49:29.170 INFO:tasks.rados.rados.2.smithi175.stdout:2991: writing smithi17557870-16 from 2504321 to 2909921 tid 3 2024-09-11T03:49:29.170 INFO:tasks.rados.rados.2.smithi175.stdout:2992: delete oid 38 current snap is 337 2024-09-11T03:49:29.170 INFO:tasks.rados.rados.2.smithi175.stdout:2991: finishing write tid 1 to smithi17557870-16 2024-09-11T03:49:29.170 INFO:tasks.rados.rados.2.smithi175.stdout:2991: finishing write tid 2 to smithi17557870-16 2024-09-11T03:49:29.172 INFO:tasks.rados.rados.2.smithi175.stdout:2992: done (1 left) 2024-09-11T03:49:29.172 INFO:tasks.rados.rados.2.smithi175.stdout:2993: read oid 10 snap -1 2024-09-11T03:49:29.172 INFO:tasks.rados.rados.2.smithi175.stdout:2993: expect (ObjNum 736 snap 243 seq_num 736) 2024-09-11T03:49:29.172 INFO:tasks.rados.rados.2.smithi175.stdout:2994: write (excl) oid 2 current snap is 337 2024-09-11T03:49:29.172 INFO:tasks.rados.rados.2.smithi175.stdout:2994: seq_num 1026 ranges {443982=796776,1989231=479018,2959167=284815} 2024-09-11T03:49:29.178 INFO:tasks.rados.rados.2.smithi175.stdout:2994: writing smithi17557870-2 from 443982 to 1240758 tid 1 2024-09-11T03:49:29.185 INFO:tasks.rados.rados.2.smithi175.stdout:2994: writing smithi17557870-2 from 1989231 to 2468249 tid 2 2024-09-11T03:49:29.190 INFO:tasks.rados.rados.2.smithi175.stdout:2994: writing smithi17557870-2 from 2959167 to 3243982 tid 3 2024-09-11T03:49:29.190 INFO:tasks.rados.rados.2.smithi175.stdout:2995: read oid 41 snap -1 2024-09-11T03:49:29.190 INFO:tasks.rados.rados.2.smithi175.stdout:2995: expect deleted 2024-09-11T03:49:29.190 INFO:tasks.rados.rados.2.smithi175.stdout:2996: snap_remove snap 323 2024-09-11T03:49:29.246 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:29.267 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:29.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:29.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:29.341 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:49:29.368 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:29.389 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:29.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:29.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:29.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:29 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:49:29.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:29 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3949686818' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:49:29.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:29 smithi084 ceph-mon[27254]: osdmap e750: 12 total, 12 up, 12 in 2024-09-11T03:49:29.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:29 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:49:29.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:29 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/379000383' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.9827149362422528} : dispatch 2024-09-11T03:49:29.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:29 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:49:29.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:29 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:49:29.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:29 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:49:29.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:29 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:49:29.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:29 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:49:29.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:29 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3949686818' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:49:29.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:29 smithi121 ceph-mon[30729]: osdmap e750: 12 total, 12 up, 12 in 2024-09-11T03:49:29.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:29 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:49:29.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:29 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/379000383' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.9827149362422528} : dispatch 2024-09-11T03:49:29.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:29 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:49:29.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:29 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:49:29.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:29 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:49:29.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:29 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:49:29.462 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:29 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:49:29.462 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:29 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3949686818' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:49:29.463 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:29 smithi175 ceph-mon[30894]: osdmap e750: 12 total, 12 up, 12 in 2024-09-11T03:49:29.463 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:29 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:49:29.463 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:29 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/379000383' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.9827149362422528} : dispatch 2024-09-11T03:49:29.463 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:29 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:49:29.463 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:29 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:49:29.463 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:29 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:49:29.463 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:29 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:49:29.491 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:29.512 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:29.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:29.570 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:49:29.570 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:49:29.587 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:49:29.587 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:49:29.602 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:49:29.602 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:49:29.615 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:29.618 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:49:29.618 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:49:29.635 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:29.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:29.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:29.639 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:49:29.639 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:49:29.656 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:49:29.656 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:49:29.673 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:49:29.673 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:49:29.690 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:49:29.691 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:49:29.707 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:49:29.707 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:49:29.724 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:49:29.725 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:49:29.737 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:29.741 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:49:29.741 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:49:29.758 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:49:29.758 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:49:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:29.760 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:29.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:29.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:29.887 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:29.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:29.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:29.909 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:49:29.909 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:49:29.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:30.014 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:30.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:30.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:30.115 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:30.136 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:30.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:30.152 INFO:tasks.rados.rados.2.smithi175.stdout:2996: done (4 left) 2024-09-11T03:49:30.152 INFO:tasks.rados.rados.2.smithi175.stdout:2997: snap_remove snap 310 2024-09-11T03:49:30.238 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:30.259 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:30.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:30.260 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:30.361 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:30.382 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:30.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:30.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:30 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:49:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:30 smithi084 ceph-mon[27254]: pgmap v1281: 77 pgs: 1 peering, 1 active+undersized+remapped, 5 active+clean+scrubbing, 3 remapped+peering, 5 unknown, 62 active+clean; 347 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s, 8 objects/s recovering 2024-09-11T03:49:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:30 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/379000383' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.9827149362422528}]': finished 2024-09-11T03:49:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:30 smithi084 ceph-mon[27254]: osdmap e751: 12 total, 12 up, 12 in 2024-09-11T03:49:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:30 smithi084 ceph-mon[27254]: 2.5 scrub starts 2024-09-11T03:49:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:30 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:49:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:30 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:49:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:30 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/598438519' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:30 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:49:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:30 smithi084 ceph-mon[27254]: 2.2b scrub starts 2024-09-11T03:49:30.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:30 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:49:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:30 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:49:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:30 smithi121 ceph-mon[30729]: pgmap v1281: 77 pgs: 1 peering, 1 active+undersized+remapped, 5 active+clean+scrubbing, 3 remapped+peering, 5 unknown, 62 active+clean; 347 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s, 8 objects/s recovering 2024-09-11T03:49:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:30 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/379000383' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.9827149362422528}]': finished 2024-09-11T03:49:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:30 smithi121 ceph-mon[30729]: osdmap e751: 12 total, 12 up, 12 in 2024-09-11T03:49:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:30 smithi121 ceph-mon[30729]: 2.5 scrub starts 2024-09-11T03:49:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:30 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:49:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:30 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:49:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:30 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/598438519' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:30 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:49:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:30 smithi121 ceph-mon[30729]: 2.2b scrub starts 2024-09-11T03:49:30.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:30 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:49:30.447 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:30 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:49:30.447 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:30 smithi175 ceph-mon[30894]: pgmap v1281: 77 pgs: 1 peering, 1 active+undersized+remapped, 5 active+clean+scrubbing, 3 remapped+peering, 5 unknown, 62 active+clean; 347 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s, 8 objects/s recovering 2024-09-11T03:49:30.447 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:30 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/379000383' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.9827149362422528}]': finished 2024-09-11T03:49:30.447 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:30 smithi175 ceph-mon[30894]: osdmap e751: 12 total, 12 up, 12 in 2024-09-11T03:49:30.447 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:30 smithi175 ceph-mon[30894]: 2.5 scrub starts 2024-09-11T03:49:30.447 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:30 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:49:30.447 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:30 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:49:30.447 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:30 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/598438519' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:30.447 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:30 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:49:30.447 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:30 smithi175 ceph-mon[30894]: 2.2b scrub starts 2024-09-11T03:49:30.447 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:30 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:49:30.474 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-11T03:49:30.484 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:30.504 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:30.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:30.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:30.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:30.632 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:30.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:30.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:30.712 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:49:30.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:30.760 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:30.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:30.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:30.861 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:30.883 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:30.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:30.910 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:49:30.985 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:31.006 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:31.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:31.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:31.108 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:31.130 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:31.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:31.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:31.155 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:31.160 INFO:tasks.rados.rados.2.smithi175.stdout:2997: done (4 left) 2024-09-11T03:49:31.160 INFO:tasks.rados.rados.2.smithi175.stdout:2998: rollback oid 12 current snap is 337 2024-09-11T03:49:31.160 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 12 to 311 2024-09-11T03:49:31.160 INFO:tasks.rados.rados.2.smithi175.stdout:2999: rollback oid 44 current snap is 337 2024-09-11T03:49:31.161 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 44 to 333 2024-09-11T03:49:31.161 INFO:tasks.rados.rados.2.smithi175.stdout:3000: snap_remove snap 300 2024-09-11T03:49:31.232 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:31.234 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:49:31.234 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:49:31.255 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:31.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:31.256 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:31.357 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:31.378 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:31.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:31.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:31.406 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:31 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:49:31.406 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:31 smithi084 ceph-mon[27254]: osdmap e752: 12 total, 12 up, 12 in 2024-09-11T03:49:31.406 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:31 smithi084 ceph-mon[27254]: 2.2c scrub starts 2024-09-11T03:49:31.406 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:31 smithi084 ceph-mon[27254]: 2.26 scrub starts 2024-09-11T03:49:31.406 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:31 smithi084 ceph-mon[27254]: 2.29 scrub starts 2024-09-11T03:49:31.406 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:31 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:49:31.406 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:31 smithi084 ceph-mon[27254]: 2.2b scrub starts 2024-09-11T03:49:31.406 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:31 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:49:31.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:31 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:49:31.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:31 smithi121 ceph-mon[30729]: osdmap e752: 12 total, 12 up, 12 in 2024-09-11T03:49:31.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:31 smithi121 ceph-mon[30729]: 2.2c scrub starts 2024-09-11T03:49:31.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:31 smithi121 ceph-mon[30729]: 2.26 scrub starts 2024-09-11T03:49:31.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:31 smithi121 ceph-mon[30729]: 2.29 scrub starts 2024-09-11T03:49:31.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:31 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:49:31.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:31 smithi121 ceph-mon[30729]: 2.2b scrub starts 2024-09-11T03:49:31.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:31 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:49:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:31 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:49:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:31 smithi175 ceph-mon[30894]: osdmap e752: 12 total, 12 up, 12 in 2024-09-11T03:49:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:31 smithi175 ceph-mon[30894]: 2.2c scrub starts 2024-09-11T03:49:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:31 smithi175 ceph-mon[30894]: 2.26 scrub starts 2024-09-11T03:49:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:31 smithi175 ceph-mon[30894]: 2.29 scrub starts 2024-09-11T03:49:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:31 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:49:31.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:31 smithi175 ceph-mon[30894]: 2.2b scrub starts 2024-09-11T03:49:31.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:31 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:49:31.480 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:31.501 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:31.502 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:31.603 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:31.624 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:31.625 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:31.726 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:31.747 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:31.748 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:31.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:31.870 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:31.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:31.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:31.883 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-11T03:49:31.972 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:31.997 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:31.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:31.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:32.099 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:32.122 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:32.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:32.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:32.123 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:49:32.162 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:49:32.167 INFO:tasks.rados.rados.2.smithi175.stdout:3000: done (6 left) 2024-09-11T03:49:32.167 INFO:tasks.rados.rados.2.smithi175.stdout:3001: snap_remove snap 336 2024-09-11T03:49:32.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:32.248 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:32.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:32.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:32.350 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:32.372 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:32.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:32.373 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:32.407 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:32 smithi175 ceph-mon[30894]: 2.28 scrub starts 2024-09-11T03:49:32.408 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:32 smithi175 ceph-mon[30894]: pgmap v1284: 77 pgs: 1 peering, 1 active+undersized+remapped, 5 active+clean+scrubbing, 3 remapped+peering, 5 unknown, 62 active+clean; 347 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s, 11 objects/s recovering 2024-09-11T03:49:32.408 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:32 smithi175 ceph-mon[30894]: osdmap e753: 12 total, 12 up, 12 in 2024-09-11T03:49:32.408 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:32 smithi175 ceph-mon[30894]: 1.1a scrub starts 2024-09-11T03:49:32.408 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:32 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:49:32.408 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:32 smithi175 ceph-mon[30894]: 2.24 scrub starts 2024-09-11T03:49:32.408 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:32 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:49:32.408 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:32 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1676016831' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:49:32.408 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:32 smithi175 ceph-mon[30894]: 2.2b scrub starts 2024-09-11T03:49:32.408 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:32 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:49:32.408 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:32 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:32 smithi121 ceph-mon[30729]: 2.28 scrub starts 2024-09-11T03:49:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:32 smithi121 ceph-mon[30729]: pgmap v1284: 77 pgs: 1 peering, 1 active+undersized+remapped, 5 active+clean+scrubbing, 3 remapped+peering, 5 unknown, 62 active+clean; 347 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s, 11 objects/s recovering 2024-09-11T03:49:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:32 smithi121 ceph-mon[30729]: osdmap e753: 12 total, 12 up, 12 in 2024-09-11T03:49:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:32 smithi121 ceph-mon[30729]: 1.1a scrub starts 2024-09-11T03:49:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:32 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:49:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:32 smithi121 ceph-mon[30729]: 2.24 scrub starts 2024-09-11T03:49:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:32 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:49:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:32 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1676016831' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:49:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:32 smithi121 ceph-mon[30729]: 2.2b scrub starts 2024-09-11T03:49:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:32 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:49:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:32 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:32.474 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:32.496 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:32.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:32.497 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:32.517 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:32 smithi084 ceph-mon[27254]: 2.28 scrub starts 2024-09-11T03:49:32.517 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:32 smithi084 ceph-mon[27254]: pgmap v1284: 77 pgs: 1 peering, 1 active+undersized+remapped, 5 active+clean+scrubbing, 3 remapped+peering, 5 unknown, 62 active+clean; 347 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s, 11 objects/s recovering 2024-09-11T03:49:32.518 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:32 smithi084 ceph-mon[27254]: osdmap e753: 12 total, 12 up, 12 in 2024-09-11T03:49:32.518 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:32 smithi084 ceph-mon[27254]: 1.1a scrub starts 2024-09-11T03:49:32.518 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:32 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:49:32.518 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:32 smithi084 ceph-mon[27254]: 2.24 scrub starts 2024-09-11T03:49:32.518 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:32 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:49:32.518 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:32 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1676016831' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:49:32.518 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:32 smithi084 ceph-mon[27254]: 2.2b scrub starts 2024-09-11T03:49:32.518 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:32 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:49:32.518 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:32 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:32.591 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:49:32.591 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:49:32.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:32.623 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:32.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:32.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:32.725 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:32.747 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:32.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:32.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:32.780 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:49:32.849 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:32.872 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:32.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:32.972 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:32.994 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:32.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:32.995 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:33.027 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:33.096 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:33.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:33.119 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:33.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:33.171 INFO:tasks.rados.rados.2.smithi175.stdout:3001: done (6 left) 2024-09-11T03:49:33.171 INFO:tasks.rados.rados.2.smithi175.stdout:3002: rollback oid 37 current snap is 337 2024-09-11T03:49:33.171 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 37 to 333 2024-09-11T03:49:33.171 INFO:tasks.rados.rados.2.smithi175.stdout:2991: finishing write tid 3 to smithi17557870-16 2024-09-11T03:49:33.171 INFO:tasks.rados.rados.2.smithi175.stdout:2991: finishing write tid 4 to smithi17557870-16 2024-09-11T03:49:33.171 INFO:tasks.rados.rados.2.smithi175.stdout:2991: finishing write tid 5 to smithi17557870-16 2024-09-11T03:49:33.171 INFO:tasks.rados.rados.2.smithi175.stdout:2991: oid 16 updating version 0 to 970 2024-09-11T03:49:33.171 INFO:tasks.rados.rados.2.smithi175.stdout:2991: oid 16 updating version 970 to 972 2024-09-11T03:49:33.171 INFO:tasks.rados.rados.2.smithi175.stdout:2991: oid 16 version 972 is already newer than 969 2024-09-11T03:49:33.171 INFO:tasks.rados.rados.2.smithi175.stdout:2991: oid 16 version 972 is already newer than 971 2024-09-11T03:49:33.171 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 972 (ObjNum 1024 snap 337 seq_num 1024) dirty exists 2024-09-11T03:49:33.171 INFO:tasks.rados.rados.2.smithi175.stdout:2991: left oid 16 (ObjNum 1024 snap 337 seq_num 1024) 2024-09-11T03:49:33.172 INFO:tasks.rados.rados.2.smithi175.stdout:2993: expect (ObjNum 736 snap 243 seq_num 736) 2024-09-11T03:49:33.220 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:33 smithi175 ceph-mon[30894]: 2.28 scrub starts 2024-09-11T03:49:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:33 smithi175 ceph-mon[30894]: 2.27 scrub starts 2024-09-11T03:49:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:33 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1676016831' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:49:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:33 smithi175 ceph-mon[30894]: osdmap e754: 12 total, 12 up, 12 in 2024-09-11T03:49:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:33 smithi175 ceph-mon[30894]: 2.2c scrub starts 2024-09-11T03:49:33.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:33 smithi175 ceph-mon[30894]: 2.2c scrub ok 2024-09-11T03:49:33.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:33 smithi175 ceph-mon[30894]: 2.2d scrub starts 2024-09-11T03:49:33.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:33 smithi175 ceph-mon[30894]: 2.2d scrub ok 2024-09-11T03:49:33.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:33 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:49:33.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:33 smithi175 ceph-mon[30894]: 2.d scrub ok 2024-09-11T03:49:33.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:33 smithi175 ceph-mon[30894]: 2.2b scrub starts 2024-09-11T03:49:33.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:33 smithi175 ceph-mon[30894]: 2.2b scrub ok 2024-09-11T03:49:33.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:33 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:49:33.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:33 smithi175 ceph-mon[30894]: 2.a scrub ok 2024-09-11T03:49:33.242 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:33.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:33.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:33.255 INFO:tasks.rados.rados.2.smithi175.stdout:2994: finishing write tid 1 to smithi17557870-2 2024-09-11T03:49:33.255 INFO:tasks.rados.rados.2.smithi175.stdout:2994: finishing write tid 2 to smithi17557870-2 2024-09-11T03:49:33.255 INFO:tasks.rados.rados.2.smithi175.stdout:2991: done (6 left) 2024-09-11T03:49:33.255 INFO:tasks.rados.rados.2.smithi175.stdout:2993: done (5 left) 2024-09-11T03:49:33.255 INFO:tasks.rados.rados.2.smithi175.stdout:3003: rollback oid 48 current snap is 337 2024-09-11T03:49:33.255 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 48 to 320 2024-09-11T03:49:33.255 INFO:tasks.rados.rados.2.smithi175.stdout:2994: finishing write tid 3 to smithi17557870-2 2024-09-11T03:49:33.256 INFO:tasks.rados.rados.2.smithi175.stdout:2994: finishing write tid 4 to smithi17557870-2 2024-09-11T03:49:33.256 INFO:tasks.rados.rados.2.smithi175.stdout:2994: finishing write tid 5 to smithi17557870-2 2024-09-11T03:49:33.256 INFO:tasks.rados.rados.2.smithi175.stdout:2994: oid 2 updating version 0 to 1058 2024-09-11T03:49:33.256 INFO:tasks.rados.rados.2.smithi175.stdout:2994: oid 2 version 1058 is already newer than 1057 2024-09-11T03:49:33.256 INFO:tasks.rados.rados.2.smithi175.stdout:2994: oid 2 updating version 1058 to 1059 2024-09-11T03:49:33.256 INFO:tasks.rados.rados.2.smithi175.stdout:2994: oid 2 version 1059 is already newer than 1056 2024-09-11T03:49:33.256 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 1059 (ObjNum 1026 snap 337 seq_num 1026) dirty exists 2024-09-11T03:49:33.256 INFO:tasks.rados.rados.2.smithi175.stdout:2994: left oid 2 (ObjNum 1026 snap 337 seq_num 1026) 2024-09-11T03:49:33.256 INFO:tasks.rados.rados.2.smithi175.stdout:2999: finishing rollback tid 0 to smithi17557870-44 2024-09-11T03:49:33.256 INFO:tasks.rados.rados.2.smithi175.stdout:2999: finishing rollback tid 1 to smithi17557870-44 2024-09-11T03:49:33.256 INFO:tasks.rados.rados.2.smithi175.stdout:2999: finishing rollback tid 2 to smithi17557870-44 2024-09-11T03:49:33.256 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 1016 (ObjNum 970 snap 311 seq_num 970) dirty exists 2024-09-11T03:49:33.256 INFO:tasks.rados.rados.2.smithi175.stdout:2998: finishing rollback tid 0 to smithi17557870-12 2024-09-11T03:49:33.256 INFO:tasks.rados.rados.2.smithi175.stdout:2998: finishing rollback tid 1 to smithi17557870-12 2024-09-11T03:49:33.257 INFO:tasks.rados.rados.2.smithi175.stdout:2998: finishing rollback tid 2 to smithi17557870-12 2024-09-11T03:49:33.257 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 1128 (ObjNum 952 snap 304 seq_num 952) dirty exists 2024-09-11T03:49:33.257 INFO:tasks.rados.rados.2.smithi175.stdout:2994: done (5 left) 2024-09-11T03:49:33.257 INFO:tasks.rados.rados.2.smithi175.stdout:2995: done (4 left) 2024-09-11T03:49:33.257 INFO:tasks.rados.rados.2.smithi175.stdout:2998: done (3 left) 2024-09-11T03:49:33.257 INFO:tasks.rados.rados.2.smithi175.stdout:2999: done (2 left) 2024-09-11T03:49:33.257 INFO:tasks.rados.rados.2.smithi175.stdout:3004: read oid 35 snap -1 2024-09-11T03:49:33.257 INFO:tasks.rados.rados.2.smithi175.stdout:3004: expect deleted 2024-09-11T03:49:33.257 INFO:tasks.rados.rados.2.smithi175.stdout:3005: delete oid 7 current snap is 337 2024-09-11T03:49:33.257 INFO:tasks.rados.rados.2.smithi175.stdout:3002: finishing rollback tid 0 to smithi17557870-37 2024-09-11T03:49:33.257 INFO:tasks.rados.rados.2.smithi175.stdout:3002: finishing rollback tid 1 to smithi17557870-37 2024-09-11T03:49:33.257 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 904 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-11T03:49:33.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T03:49:33.259 INFO:tasks.rados.rados.2.smithi175.stdout:3002: done (3 left) 2024-09-11T03:49:33.259 INFO:tasks.rados.rados.2.smithi175.stdout:3004: done (2 left) 2024-09-11T03:49:33.259 INFO:tasks.rados.rados.2.smithi175.stdout:3005: done (1 left) 2024-09-11T03:49:33.259 INFO:tasks.rados.rados.2.smithi175.stdout:3006: delete oid 9 current snap is 337 2024-09-11T03:49:33.260 INFO:tasks.rados.rados.2.smithi175.stdout:3003: finishing rollback tid 0 to smithi17557870-48 2024-09-11T03:49:33.260 INFO:tasks.rados.rados.2.smithi175.stdout:3003: finishing rollback tid 1 to smithi17557870-48 2024-09-11T03:49:33.260 INFO:tasks.rados.rados.2.smithi175.stdout:3003: finishing rollback tid 2 to smithi17557870-48 2024-09-11T03:49:33.260 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 975 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:49:33.260 INFO:tasks.rados.rados.2.smithi175.stdout:3003: done (1 left) 2024-09-11T03:49:33.260 INFO:tasks.rados.rados.2.smithi175.stdout:3006: done (0 left) 2024-09-11T03:49:33.260 INFO:tasks.rados.rados.2.smithi175.stdout:3007: snap_create 2024-09-11T03:49:33.344 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:33.365 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:33.366 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:33.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:33 smithi084 ceph-mon[27254]: 2.28 scrub starts 2024-09-11T03:49:33.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:33 smithi084 ceph-mon[27254]: 2.27 scrub starts 2024-09-11T03:49:33.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:33 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1676016831' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:49:33.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:33 smithi084 ceph-mon[27254]: osdmap e754: 12 total, 12 up, 12 in 2024-09-11T03:49:33.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:33 smithi084 ceph-mon[27254]: 2.2c scrub starts 2024-09-11T03:49:33.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:33 smithi084 ceph-mon[27254]: 2.2c scrub ok 2024-09-11T03:49:33.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:33 smithi084 ceph-mon[27254]: 2.2d scrub starts 2024-09-11T03:49:33.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:33 smithi084 ceph-mon[27254]: 2.2d scrub ok 2024-09-11T03:49:33.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:33 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:49:33.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:33 smithi084 ceph-mon[27254]: 2.d scrub ok 2024-09-11T03:49:33.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:33 smithi084 ceph-mon[27254]: 2.2b scrub starts 2024-09-11T03:49:33.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:33 smithi084 ceph-mon[27254]: 2.2b scrub ok 2024-09-11T03:49:33.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:33 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:49:33.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:33 smithi084 ceph-mon[27254]: 2.a scrub ok 2024-09-11T03:49:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:33 smithi121 ceph-mon[30729]: 2.28 scrub starts 2024-09-11T03:49:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:33 smithi121 ceph-mon[30729]: 2.27 scrub starts 2024-09-11T03:49:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:33 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1676016831' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:49:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:33 smithi121 ceph-mon[30729]: osdmap e754: 12 total, 12 up, 12 in 2024-09-11T03:49:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:33 smithi121 ceph-mon[30729]: 2.2c scrub starts 2024-09-11T03:49:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:33 smithi121 ceph-mon[30729]: 2.2c scrub ok 2024-09-11T03:49:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:33 smithi121 ceph-mon[30729]: 2.2d scrub starts 2024-09-11T03:49:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:33 smithi121 ceph-mon[30729]: 2.2d scrub ok 2024-09-11T03:49:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:33 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:49:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:33 smithi121 ceph-mon[30729]: 2.d scrub ok 2024-09-11T03:49:33.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:33 smithi121 ceph-mon[30729]: 2.2b scrub starts 2024-09-11T03:49:33.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:33 smithi121 ceph-mon[30729]: 2.2b scrub ok 2024-09-11T03:49:33.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:33 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:49:33.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:33 smithi121 ceph-mon[30729]: 2.a scrub ok 2024-09-11T03:49:33.467 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:33.488 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:33.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:33.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:33.521 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:49:33.522 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:49:33.522 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:49:33.522 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:49:33.536 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:49:33.536 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:49:33.554 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:49:33.554 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:49:33.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:49:33.572 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:49:33.588 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:33.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:33.590 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:49:33.591 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:49:33.610 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:49:33.611 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:49:33.615 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:33.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:33.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:33.628 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:49:33.628 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:49:33.645 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:49:33.645 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:49:33.661 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:49:33.662 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:49:33.678 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:49:33.678 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:49:33.695 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:49:33.695 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:49:33.717 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:33.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:33.739 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:33.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:33.841 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:33.863 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:33.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:33.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:33.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:33.993 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:33.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:34.094 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:34.116 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:34.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:34.181 INFO:tasks.rados.rados.2.smithi175.stdout:3007: done (0 left) 2024-09-11T03:49:34.181 INFO:tasks.rados.rados.2.smithi175.stdout:3008: read oid 3 snap 314 2024-09-11T03:49:34.181 INFO:tasks.rados.rados.2.smithi175.stdout:3008: expect deleted 2024-09-11T03:49:34.182 INFO:tasks.rados.rados.2.smithi175.stdout:3009: write (excl) oid 11 current snap is 338 2024-09-11T03:49:34.182 INFO:tasks.rados.rados.2.smithi175.stdout:3009: seq_num 1029 ranges {494403=738876,1764677=626345,3041436=252967} 2024-09-11T03:49:34.187 INFO:tasks.rados.rados.2.smithi175.stdout:3009: writing smithi17557870-11 from 494403 to 1233279 tid 1 2024-09-11T03:49:34.194 INFO:tasks.rados.rados.2.smithi175.stdout:3009: writing smithi17557870-11 from 1764677 to 2391022 tid 2 2024-09-11T03:49:34.199 INFO:tasks.rados.rados.2.smithi175.stdout:3009: writing smithi17557870-11 from 3041436 to 3294403 tid 3 2024-09-11T03:49:34.199 INFO:tasks.rados.rados.2.smithi175.stdout:3010: snap_remove snap 284 2024-09-11T03:49:34.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:34.241 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:34.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:34.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:34.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:34.367 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:34.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: 2.28 scrub starts 2024-09-11T03:49:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: 2.28 scrub ok 2024-09-11T03:49:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: pgmap v1287: 77 pgs: 1 peering, 3 active+clean+scrubbing, 73 active+clean; 406 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s rd, 18 MiB/s wr, 124 op/s; 16 MiB/s, 51 objects/s recovering 2024-09-11T03:49:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: osdmap e755: 12 total, 12 up, 12 in 2024-09-11T03:49:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: 2.26 scrub starts 2024-09-11T03:49:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: 2.26 scrub ok 2024-09-11T03:49:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:49:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: 2.b scrub ok 2024-09-11T03:49:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:49:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: 2.9 scrub ok 2024-09-11T03:49:34.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2025757422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:34.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: 2.29 scrub starts 2024-09-11T03:49:34.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: 2.29 scrub ok 2024-09-11T03:49:34.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: 2.1c scrub starts 2024-09-11T03:49:34.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: 2.1c scrub ok 2024-09-11T03:49:34.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: 2.16 scrub starts 2024-09-11T03:49:34.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: 2.16 scrub ok 2024-09-11T03:49:34.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:49:34.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:34 smithi121 ceph-mon[30729]: 2.c scrub ok 2024-09-11T03:49:34.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: 2.28 scrub starts 2024-09-11T03:49:34.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: 2.28 scrub ok 2024-09-11T03:49:34.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: pgmap v1287: 77 pgs: 1 peering, 3 active+clean+scrubbing, 73 active+clean; 406 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s rd, 18 MiB/s wr, 124 op/s; 16 MiB/s, 51 objects/s recovering 2024-09-11T03:49:34.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: osdmap e755: 12 total, 12 up, 12 in 2024-09-11T03:49:34.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: 2.26 scrub starts 2024-09-11T03:49:34.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: 2.26 scrub ok 2024-09-11T03:49:34.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:49:34.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: 2.b scrub ok 2024-09-11T03:49:34.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:49:34.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: 2.9 scrub ok 2024-09-11T03:49:34.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2025757422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:34.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: 2.29 scrub starts 2024-09-11T03:49:34.436 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: 2.29 scrub ok 2024-09-11T03:49:34.436 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: 2.1c scrub starts 2024-09-11T03:49:34.436 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: 2.1c scrub ok 2024-09-11T03:49:34.436 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: 2.16 scrub starts 2024-09-11T03:49:34.436 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: 2.16 scrub ok 2024-09-11T03:49:34.436 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:49:34.436 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:34 smithi084 ceph-mon[27254]: 2.c scrub ok 2024-09-11T03:49:34.468 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: 2.28 scrub starts 2024-09-11T03:49:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: 2.28 scrub ok 2024-09-11T03:49:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: pgmap v1287: 77 pgs: 1 peering, 3 active+clean+scrubbing, 73 active+clean; 406 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s rd, 18 MiB/s wr, 124 op/s; 16 MiB/s, 51 objects/s recovering 2024-09-11T03:49:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: osdmap e755: 12 total, 12 up, 12 in 2024-09-11T03:49:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: 2.26 scrub starts 2024-09-11T03:49:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: 2.26 scrub ok 2024-09-11T03:49:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:49:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: 2.b scrub ok 2024-09-11T03:49:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:49:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: 2.9 scrub ok 2024-09-11T03:49:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2025757422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: 2.29 scrub starts 2024-09-11T03:49:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: 2.29 scrub ok 2024-09-11T03:49:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: 2.1c scrub starts 2024-09-11T03:49:34.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: 2.1c scrub ok 2024-09-11T03:49:34.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: 2.16 scrub starts 2024-09-11T03:49:34.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: 2.16 scrub ok 2024-09-11T03:49:34.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:49:34.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:34 smithi175 ceph-mon[30894]: 2.c scrub ok 2024-09-11T03:49:34.491 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:34.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:34.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:34.593 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:34.618 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:34.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:34.619 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:34.721 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:34.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:34.742 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:34.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:34.797 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:49:34.797 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:49:34.797 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:49:34.797 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:49:34.844 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:34.869 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:34.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:34.870 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:34.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:49:34.971 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:34.996 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:34.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:34.997 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:35.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:35.110 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:49:35.110 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:49:35.110 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-11T03:49:35.110 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T03:49:35.110 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T03:49:35.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-11T03:49:35.123 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:35.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:35.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:35.193 INFO:tasks.rados.rados.2.smithi175.stdout:3010: done (2 left) 2024-09-11T03:49:35.193 INFO:tasks.rados.rados.2.smithi175.stdout:3011: copy_from oid 9 from oid 35 current snap is 338 2024-09-11T03:49:35.193 INFO:tasks.rados.rados.2.smithi175.stdout:3012: write (excl) oid 39 current snap is 338 2024-09-11T03:49:35.193 INFO:tasks.rados.rados.2.smithi175.stdout:3012: seq_num 1030 ranges {511210=414125,1423159=486356,2668859=589585} 2024-09-11T03:49:35.198 INFO:tasks.rados.rados.2.smithi175.stdout:3012: writing smithi17557870-39 from 511210 to 925335 tid 1 2024-09-11T03:49:35.204 INFO:tasks.rados.rados.2.smithi175.stdout:3012: writing smithi17557870-39 from 1423159 to 1909515 tid 2 2024-09-11T03:49:35.213 INFO:tasks.rados.rados.2.smithi175.stdout:3012: writing smithi17557870-39 from 2668859 to 3258444 tid 3 2024-09-11T03:49:35.213 INFO:tasks.rados.rados.2.smithi175.stdout:3013: read oid 27 snap -1 2024-09-11T03:49:35.213 INFO:tasks.rados.rados.2.smithi175.stdout:3013: expect (ObjNum 765 snap 247 seq_num 765) 2024-09-11T03:49:35.213 INFO:tasks.rados.rados.2.smithi175.stdout:3014: snap_create 2024-09-11T03:49:35.213 INFO:tasks.rados.rados.2.smithi175.stdout:3009: finishing write tid 1 to smithi17557870-11 2024-09-11T03:49:35.213 INFO:tasks.rados.rados.2.smithi175.stdout:3009: finishing write tid 2 to smithi17557870-11 2024-09-11T03:49:35.213 INFO:tasks.rados.rados.2.smithi175.stdout:3009: finishing write tid 3 to smithi17557870-11 2024-09-11T03:49:35.213 INFO:tasks.rados.rados.2.smithi175.stdout:3009: finishing write tid 4 to smithi17557870-11 2024-09-11T03:49:35.213 INFO:tasks.rados.rados.2.smithi175.stdout:3009: finishing write tid 5 to smithi17557870-11 2024-09-11T03:49:35.213 INFO:tasks.rados.rados.2.smithi175.stdout:3009: oid 11 updating version 0 to 617 2024-09-11T03:49:35.213 INFO:tasks.rados.rados.2.smithi175.stdout:3009: oid 11 updating version 617 to 618 2024-09-11T03:49:35.213 INFO:tasks.rados.rados.2.smithi175.stdout:3009: oid 11 updating version 618 to 620 2024-09-11T03:49:35.214 INFO:tasks.rados.rados.2.smithi175.stdout:3009: oid 11 version 620 is already newer than 619 2024-09-11T03:49:35.214 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 620 (ObjNum 1029 snap 338 seq_num 1029) dirty exists 2024-09-11T03:49:35.214 INFO:tasks.rados.rados.2.smithi175.stdout:3009: left oid 11 (ObjNum 1029 snap 338 seq_num 1029) 2024-09-11T03:49:35.214 INFO:tasks.rados.rados.2.smithi175.stdout:3011: finishing copy_from to smithi17557870-9 2024-09-11T03:49:35.214 INFO:tasks.rados.rados.2.smithi175.stdout:3011: got expected ENOENT (src dne) 2024-09-11T03:49:35.214 INFO:tasks.rados.rados.2.smithi175.stdout:3011: finishing copy_from racing read to smithi17557870-9 2024-09-11T03:49:35.214 INFO:tasks.rados.rados.2.smithi175.stdout:3012: finishing write tid 1 to smithi17557870-39 2024-09-11T03:49:35.214 INFO:tasks.rados.rados.2.smithi175.stdout:3012: finishing write tid 2 to smithi17557870-39 2024-09-11T03:49:35.215 INFO:tasks.rados.rados.2.smithi175.stdout:3012: finishing write tid 3 to smithi17557870-39 2024-09-11T03:49:35.215 INFO:tasks.rados.rados.2.smithi175.stdout:3012: finishing write tid 4 to smithi17557870-39 2024-09-11T03:49:35.215 INFO:tasks.rados.rados.2.smithi175.stdout:3012: finishing write tid 5 to smithi17557870-39 2024-09-11T03:49:35.216 INFO:tasks.rados.rados.2.smithi175.stdout:3012: oid 39 updating version 0 to 980 2024-09-11T03:49:35.216 INFO:tasks.rados.rados.2.smithi175.stdout:3012: oid 39 version 980 is already newer than 977 2024-09-11T03:49:35.216 INFO:tasks.rados.rados.2.smithi175.stdout:3012: oid 39 version 980 is already newer than 978 2024-09-11T03:49:35.216 INFO:tasks.rados.rados.2.smithi175.stdout:3012: oid 39 version 980 is already newer than 979 2024-09-11T03:49:35.216 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 980 (ObjNum 1030 snap 338 seq_num 1030) dirty exists 2024-09-11T03:49:35.216 INFO:tasks.rados.rados.2.smithi175.stdout:3012: left oid 39 (ObjNum 1030 snap 338 seq_num 1030) 2024-09-11T03:49:35.225 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:35.228 INFO:tasks.rados.rados.2.smithi175.stdout:3013: expect (ObjNum 765 snap 247 seq_num 765) 2024-09-11T03:49:35.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:35.249 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:35.249 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:35.350 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:35.371 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:35.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:35.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:35.381 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:49:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:35 smithi084 ceph-mon[27254]: osdmap e756: 12 total, 12 up, 12 in 2024-09-11T03:49:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:35 smithi084 ceph-mon[27254]: 2.25 scrub starts 2024-09-11T03:49:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:35 smithi084 ceph-mon[27254]: 2.25 scrub ok 2024-09-11T03:49:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:35 smithi084 ceph-mon[27254]: 2.2a scrub starts 2024-09-11T03:49:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:35 smithi084 ceph-mon[27254]: 2.2a scrub ok 2024-09-11T03:49:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:35 smithi084 ceph-mon[27254]: 2.4 scrub starts 2024-09-11T03:49:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:35 smithi084 ceph-mon[27254]: 2.4 scrub ok 2024-09-11T03:49:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:35 smithi084 ceph-mon[27254]: 2.24 scrub starts 2024-09-11T03:49:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:35 smithi084 ceph-mon[27254]: 2.24 scrub ok 2024-09-11T03:49:35.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:35 smithi084 ceph-mon[27254]: 1.8 scrub starts 2024-09-11T03:49:35.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:35 smithi084 ceph-mon[27254]: 1.8 scrub ok 2024-09-11T03:49:35.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:35 smithi084 ceph-mon[27254]: 2.1b scrub starts 2024-09-11T03:49:35.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:35 smithi084 ceph-mon[27254]: 2.1b scrub ok 2024-09-11T03:49:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:35 smithi121 ceph-mon[30729]: osdmap e756: 12 total, 12 up, 12 in 2024-09-11T03:49:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:35 smithi121 ceph-mon[30729]: 2.25 scrub starts 2024-09-11T03:49:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:35 smithi121 ceph-mon[30729]: 2.25 scrub ok 2024-09-11T03:49:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:35 smithi121 ceph-mon[30729]: 2.2a scrub starts 2024-09-11T03:49:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:35 smithi121 ceph-mon[30729]: 2.2a scrub ok 2024-09-11T03:49:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:35 smithi121 ceph-mon[30729]: 2.4 scrub starts 2024-09-11T03:49:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:35 smithi121 ceph-mon[30729]: 2.4 scrub ok 2024-09-11T03:49:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:35 smithi121 ceph-mon[30729]: 2.24 scrub starts 2024-09-11T03:49:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:35 smithi121 ceph-mon[30729]: 2.24 scrub ok 2024-09-11T03:49:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:35 smithi121 ceph-mon[30729]: 1.8 scrub starts 2024-09-11T03:49:35.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:35 smithi121 ceph-mon[30729]: 1.8 scrub ok 2024-09-11T03:49:35.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:35 smithi121 ceph-mon[30729]: 2.1b scrub starts 2024-09-11T03:49:35.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:35 smithi121 ceph-mon[30729]: 2.1b scrub ok 2024-09-11T03:49:35.473 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:35.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:35 smithi175 ceph-mon[30894]: osdmap e756: 12 total, 12 up, 12 in 2024-09-11T03:49:35.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:35 smithi175 ceph-mon[30894]: 2.25 scrub starts 2024-09-11T03:49:35.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:35 smithi175 ceph-mon[30894]: 2.25 scrub ok 2024-09-11T03:49:35.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:35 smithi175 ceph-mon[30894]: 2.2a scrub starts 2024-09-11T03:49:35.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:35 smithi175 ceph-mon[30894]: 2.2a scrub ok 2024-09-11T03:49:35.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:35 smithi175 ceph-mon[30894]: 2.4 scrub starts 2024-09-11T03:49:35.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:35 smithi175 ceph-mon[30894]: 2.4 scrub ok 2024-09-11T03:49:35.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:35 smithi175 ceph-mon[30894]: 2.24 scrub starts 2024-09-11T03:49:35.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:35 smithi175 ceph-mon[30894]: 2.24 scrub ok 2024-09-11T03:49:35.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:35 smithi175 ceph-mon[30894]: 1.8 scrub starts 2024-09-11T03:49:35.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:35 smithi175 ceph-mon[30894]: 1.8 scrub ok 2024-09-11T03:49:35.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:35 smithi175 ceph-mon[30894]: 2.1b scrub starts 2024-09-11T03:49:35.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:35 smithi175 ceph-mon[30894]: 2.1b scrub ok 2024-09-11T03:49:35.495 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:35.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:35.496 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:35.531 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T03:49:35.596 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:35.601 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:35.618 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:35.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:35.721 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:35.742 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:35.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:35.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:35.844 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:35.857 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:49:35.857 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T03:49:35.857 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:49:35.867 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:35.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:35.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:35.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:35.990 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:35.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:35.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:36.092 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:36.102 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:36.117 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:36.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:36.118 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:36.200 INFO:tasks.rados.rados.2.smithi175.stdout:3008: done (5 left) 2024-09-11T03:49:36.200 INFO:tasks.rados.rados.2.smithi175.stdout:3009: done (4 left) 2024-09-11T03:49:36.200 INFO:tasks.rados.rados.2.smithi175.stdout:3011: done (3 left) 2024-09-11T03:49:36.200 INFO:tasks.rados.rados.2.smithi175.stdout:3012: done (2 left) 2024-09-11T03:49:36.200 INFO:tasks.rados.rados.2.smithi175.stdout:3013: done (1 left) 2024-09-11T03:49:36.200 INFO:tasks.rados.rados.2.smithi175.stdout:3014: done (0 left) 2024-09-11T03:49:36.201 INFO:tasks.rados.rados.2.smithi175.stdout:3015: read oid 2 snap -1 2024-09-11T03:49:36.201 INFO:tasks.rados.rados.2.smithi175.stdout:3015: expect (ObjNum 1026 snap 337 seq_num 1026) 2024-09-11T03:49:36.201 INFO:tasks.rados.rados.2.smithi175.stdout:3016: read oid 25 snap -1 2024-09-11T03:49:36.201 INFO:tasks.rados.rados.2.smithi175.stdout:3016: expect (ObjNum 950 snap 303 seq_num 950) 2024-09-11T03:49:36.201 INFO:tasks.rados.rados.2.smithi175.stdout:3017: rollback oid 1 current snap is 339 2024-09-11T03:49:36.201 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 1 to 326 2024-09-11T03:49:36.201 INFO:tasks.rados.rados.2.smithi175.stdout:3018: snap_create 2024-09-11T03:49:36.203 INFO:tasks.rados.rados.2.smithi175.stdout:3017: finishing rollback tid 0 to smithi17557870-1 2024-09-11T03:49:36.203 INFO:tasks.rados.rados.2.smithi175.stdout:3017: finishing rollback tid 1 to smithi17557870-1 2024-09-11T03:49:36.204 INFO:tasks.rados.rados.2.smithi175.stdout:3017: finishing rollback tid 2 to smithi17557870-1 2024-09-11T03:49:36.204 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 1120 (ObjNum 758 snap 246 seq_num 758) dirty exists 2024-09-11T03:49:36.219 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:36.219 INFO:tasks.rados.rados.2.smithi175.stdout:3015: expect (ObjNum 1026 snap 337 seq_num 1026) 2024-09-11T03:49:36.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:36.241 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:36.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:36.321 INFO:tasks.rados.rados.2.smithi175.stdout:3016: expect (ObjNum 950 snap 303 seq_num 950) 2024-09-11T03:49:36.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:36.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:36.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:36.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:36 smithi121 ceph-mon[30729]: pgmap v1290: 77 pgs: 3 active+clean+scrubbing, 74 active+clean; 404 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s rd, 13 MiB/s wr, 98 op/s; 16 MiB/s, 51 objects/s recovering 2024-09-11T03:49:36.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:36 smithi121 ceph-mon[30729]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-11T03:49:36.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:36 smithi121 ceph-mon[30729]: 2.5 scrub starts 2024-09-11T03:49:36.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:36 smithi121 ceph-mon[30729]: osdmap e757: 12 total, 12 up, 12 in 2024-09-11T03:49:36.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:36 smithi121 ceph-mon[30729]: 2.5 scrub ok 2024-09-11T03:49:36.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:36 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:49:36.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:36 smithi121 ceph-mon[30729]: 2.8 scrub ok 2024-09-11T03:49:36.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:36 smithi121 ceph-mon[30729]: 2.6 scrub starts 2024-09-11T03:49:36.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:36 smithi121 ceph-mon[30729]: 2.6 scrub ok 2024-09-11T03:49:36.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:36 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:49:36.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:36 smithi121 ceph-mon[30729]: 2.7 scrub ok 2024-09-11T03:49:36.465 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:36.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:36 smithi084 ceph-mon[27254]: pgmap v1290: 77 pgs: 3 active+clean+scrubbing, 74 active+clean; 404 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s rd, 13 MiB/s wr, 98 op/s; 16 MiB/s, 51 objects/s recovering 2024-09-11T03:49:36.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:36 smithi084 ceph-mon[27254]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-11T03:49:36.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:36 smithi084 ceph-mon[27254]: 2.5 scrub starts 2024-09-11T03:49:36.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:36 smithi084 ceph-mon[27254]: osdmap e757: 12 total, 12 up, 12 in 2024-09-11T03:49:36.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:36 smithi084 ceph-mon[27254]: 2.5 scrub ok 2024-09-11T03:49:36.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:36 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:49:36.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:36 smithi084 ceph-mon[27254]: 2.8 scrub ok 2024-09-11T03:49:36.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:36 smithi084 ceph-mon[27254]: 2.6 scrub starts 2024-09-11T03:49:36.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:36 smithi084 ceph-mon[27254]: 2.6 scrub ok 2024-09-11T03:49:36.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:36 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:49:36.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:36 smithi084 ceph-mon[27254]: 2.7 scrub ok 2024-09-11T03:49:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:36 smithi175 ceph-mon[30894]: pgmap v1290: 77 pgs: 3 active+clean+scrubbing, 74 active+clean; 404 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s rd, 13 MiB/s wr, 98 op/s; 16 MiB/s, 51 objects/s recovering 2024-09-11T03:49:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:36 smithi175 ceph-mon[30894]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-11T03:49:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:36 smithi175 ceph-mon[30894]: 2.5 scrub starts 2024-09-11T03:49:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:36 smithi175 ceph-mon[30894]: osdmap e757: 12 total, 12 up, 12 in 2024-09-11T03:49:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:36 smithi175 ceph-mon[30894]: 2.5 scrub ok 2024-09-11T03:49:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:36 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:49:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:36 smithi175 ceph-mon[30894]: 2.8 scrub ok 2024-09-11T03:49:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:36 smithi175 ceph-mon[30894]: 2.6 scrub starts 2024-09-11T03:49:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:36 smithi175 ceph-mon[30894]: 2.6 scrub ok 2024-09-11T03:49:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:36 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:49:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:36 smithi175 ceph-mon[30894]: 2.7 scrub ok 2024-09-11T03:49:36.486 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:36.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:36.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:36.568 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:49:36.568 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:49:36.568 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:49:36.569 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:49:36.569 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:49:36.588 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:36.595 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:49:36.609 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:36.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:36.711 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:36.731 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:36.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:36.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:36.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:36.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:36.856 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:36.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:36.958 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:36.979 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:36.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:36.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:37.014 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:49:37.020 INFO:tasks.rados.rados.2.smithi175.stdout:3015: done (3 left) 2024-09-11T03:49:37.020 INFO:tasks.rados.rados.2.smithi175.stdout:3016: done (2 left) 2024-09-11T03:49:37.020 INFO:tasks.rados.rados.2.smithi175.stdout:3017: done (1 left) 2024-09-11T03:49:37.020 INFO:tasks.rados.rados.2.smithi175.stdout:3018: done (0 left) 2024-09-11T03:49:37.020 INFO:tasks.rados.rados.2.smithi175.stdout:3019: copy_from oid 24 from oid 31 current snap is 340 2024-09-11T03:49:37.020 INFO:tasks.rados.rados.2.smithi175.stdout:3020: copy_from oid 38 from oid 48 current snap is 340 2024-09-11T03:49:37.020 INFO:tasks.rados.rados.2.smithi175.stdout:3021: delete oid 30 current snap is 340 2024-09-11T03:49:37.023 INFO:tasks.rados.rados.2.smithi175.stdout:3021: done (2 left) 2024-09-11T03:49:37.024 INFO:tasks.rados.rados.2.smithi175.stdout:3022: read oid 22 snap -1 2024-09-11T03:49:37.024 INFO:tasks.rados.rados.2.smithi175.stdout:3022: expect (ObjNum 983 snap 315 seq_num 983) 2024-09-11T03:49:37.024 INFO:tasks.rados.rados.2.smithi175.stdout:3023: write (excl) oid 15 current snap is 340 2024-09-11T03:49:37.024 INFO:tasks.rados.rados.2.smithi175.stdout:3023: seq_num 1032 ranges {544824=564623,1823770=490025,3007352=337472} 2024-09-11T03:49:37.030 INFO:tasks.rados.rados.2.smithi175.stdout:3023: writing smithi17557870-15 from 544824 to 1109447 tid 1 2024-09-11T03:49:37.037 INFO:tasks.rados.rados.2.smithi175.stdout:3023: writing smithi17557870-15 from 1823770 to 2313795 tid 2 2024-09-11T03:49:37.043 INFO:tasks.rados.rados.2.smithi175.stdout:3023: writing smithi17557870-15 from 3007352 to 3344824 tid 3 2024-09-11T03:49:37.043 INFO:tasks.rados.rados.2.smithi175.stdout:3024: read oid 2 snap -1 2024-09-11T03:49:37.043 INFO:tasks.rados.rados.2.smithi175.stdout:3024: expect (ObjNum 1026 snap 337 seq_num 1026) 2024-09-11T03:49:37.044 INFO:tasks.rados.rados.2.smithi175.stdout:3025: rollback oid 30 current snap is 340 2024-09-11T03:49:37.044 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 30 to 329 2024-09-11T03:49:37.044 INFO:tasks.rados.rados.2.smithi175.stdout:3019: finishing copy_from to smithi17557870-24 2024-09-11T03:49:37.044 INFO:tasks.rados.rados.2.smithi175.stdout:3019: got expected ENOENT (src dne) 2024-09-11T03:49:37.044 INFO:tasks.rados.rados.2.smithi175.stdout:3019: finishing copy_from racing read to smithi17557870-24 2024-09-11T03:49:37.044 INFO:tasks.rados.rados.2.smithi175.stdout:3020: finishing copy_from racing read to smithi17557870-38 2024-09-11T03:49:37.044 INFO:tasks.rados.rados.2.smithi175.stdout:3019: done (5 left) 2024-09-11T03:49:37.044 INFO:tasks.rados.rados.2.smithi175.stdout:3026: read oid 7 snap 308 2024-09-11T03:49:37.044 INFO:tasks.rados.rados.2.smithi175.stdout:3026: expect (ObjNum 925 snap 294 seq_num 925) 2024-09-11T03:49:37.044 INFO:tasks.rados.rados.2.smithi175.stdout:3027: write (excl) oid 9 current snap is 340 2024-09-11T03:49:37.044 INFO:tasks.rados.rados.2.smithi175.stdout:3020: finishing copy_from to smithi17557870-38 2024-09-11T03:49:37.044 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 975 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:49:37.044 INFO:tasks.rados.rados.2.smithi175.stdout:3022: expect (ObjNum 983 snap 315 seq_num 983) 2024-09-11T03:49:37.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:37.102 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:37.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:37.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:37.164 INFO:tasks.rados.rados.2.smithi175.stdout:3023: finishing write tid 1 to smithi17557870-15 2024-09-11T03:49:37.164 INFO:tasks.rados.rados.2.smithi175.stdout:3023: finishing write tid 2 to smithi17557870-15 2024-09-11T03:49:37.164 INFO:tasks.rados.rados.2.smithi175.stdout:3025: finishing rollback tid 1 to smithi17557870-30 2024-09-11T03:49:37.165 INFO:tasks.rados.rados.2.smithi175.stdout:3025: finishing rollback tid 2 to smithi17557870-30 2024-09-11T03:49:37.165 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 1023 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:49:37.165 INFO:tasks.rados.rados.2.smithi175.stdout:3024: expect (ObjNum 1026 snap 337 seq_num 1026) 2024-09-11T03:49:37.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:37.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:37.227 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:37.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:37.268 INFO:tasks.rados.rados.2.smithi175.stdout:3023: finishing write tid 3 to smithi17557870-15 2024-09-11T03:49:37.268 INFO:tasks.rados.rados.2.smithi175.stdout:3023: finishing write tid 4 to smithi17557870-15 2024-09-11T03:49:37.268 INFO:tasks.rados.rados.2.smithi175.stdout:3023: finishing write tid 5 to smithi17557870-15 2024-09-11T03:49:37.268 INFO:tasks.rados.rados.2.smithi175.stdout:3023: oid 15 updating version 0 to 894 2024-09-11T03:49:37.268 INFO:tasks.rados.rados.2.smithi175.stdout:3023: oid 15 updating version 894 to 896 2024-09-11T03:49:37.268 INFO:tasks.rados.rados.2.smithi175.stdout:3023: oid 15 updating version 896 to 897 2024-09-11T03:49:37.268 INFO:tasks.rados.rados.2.smithi175.stdout:3023: oid 15 version 897 is already newer than 895 2024-09-11T03:49:37.269 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 897 (ObjNum 1032 snap 340 seq_num 1032) dirty exists 2024-09-11T03:49:37.269 INFO:tasks.rados.rados.2.smithi175.stdout:3023: left oid 15 (ObjNum 1032 snap 340 seq_num 1032) 2024-09-11T03:49:37.269 INFO:tasks.rados.rados.2.smithi175.stdout:3026: expect (ObjNum 925 snap 294 seq_num 925) 2024-09-11T03:49:37.274 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T03:49:37.328 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:37.348 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:37.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:37.349 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:37.383 INFO:tasks.rados.rados.2.smithi175.stdout:3027: seq_num 1033 ranges {561631=639872,1882252=633683,3034775=326856} 2024-09-11T03:49:37.390 INFO:tasks.rados.rados.2.smithi175.stdout:3027: writing smithi17557870-9 from 561631 to 1201503 tid 1 2024-09-11T03:49:37.397 INFO:tasks.rados.rados.2.smithi175.stdout:3027: writing smithi17557870-9 from 1882252 to 2515935 tid 2 2024-09-11T03:49:37.402 INFO:tasks.rados.rados.2.smithi175.stdout:3027: writing smithi17557870-9 from 3034775 to 3361631 tid 3 2024-09-11T03:49:37.402 INFO:tasks.rados.rados.2.smithi175.stdout:3020: done (6 left) 2024-09-11T03:49:37.402 INFO:tasks.rados.rados.2.smithi175.stdout:3022: done (5 left) 2024-09-11T03:49:37.402 INFO:tasks.rados.rados.2.smithi175.stdout:3023: done (4 left) 2024-09-11T03:49:37.403 INFO:tasks.rados.rados.2.smithi175.stdout:3024: done (3 left) 2024-09-11T03:49:37.403 INFO:tasks.rados.rados.2.smithi175.stdout:3025: done (2 left) 2024-09-11T03:49:37.403 INFO:tasks.rados.rados.2.smithi175.stdout:3026: done (1 left) 2024-09-11T03:49:37.403 INFO:tasks.rados.rados.2.smithi175.stdout:3028: rollback oid 26 current snap is 340 2024-09-11T03:49:37.403 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 26 to 327 2024-09-11T03:49:37.403 INFO:tasks.rados.rados.2.smithi175.stdout:3027: finishing write tid 1 to smithi17557870-9 2024-09-11T03:49:37.403 INFO:tasks.rados.rados.2.smithi175.stdout:3027: finishing write tid 2 to smithi17557870-9 2024-09-11T03:49:37.403 INFO:tasks.rados.rados.2.smithi175.stdout:3029: copy_from oid 17 from oid 32 current snap is 340 2024-09-11T03:49:37.403 INFO:tasks.rados.rados.2.smithi175.stdout:3030: write (excl) oid 50 current snap is 340 2024-09-11T03:49:37.403 INFO:tasks.rados.rados.2.smithi175.stdout:3030: seq_num 1034 ranges {578438=431474,1657087=493694,2778551=599887} 2024-09-11T03:49:37.407 INFO:tasks.rados.rados.2.smithi175.stdout:3030: writing smithi17557870-50 from 578438 to 1009912 tid 1 2024-09-11T03:49:37.414 INFO:tasks.rados.rados.2.smithi175.stdout:3030: writing smithi17557870-50 from 1657087 to 2150781 tid 2 2024-09-11T03:49:37.421 INFO:tasks.rados.rados.2.smithi175.stdout:3030: writing smithi17557870-50 from 2778551 to 3378438 tid 3 2024-09-11T03:49:37.422 INFO:tasks.rados.rados.2.smithi175.stdout:3031: rollback oid 24 current snap is 340 2024-09-11T03:49:37.422 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 314 2024-09-11T03:49:37.422 INFO:tasks.rados.rados.2.smithi175.stdout:3027: finishing write tid 3 to smithi17557870-9 2024-09-11T03:49:37.422 INFO:tasks.rados.rados.2.smithi175.stdout:3028: finishing rollback tid 0 to smithi17557870-26 2024-09-11T03:49:37.422 INFO:tasks.rados.rados.2.smithi175.stdout:3027: finishing write tid 4 to smithi17557870-9 2024-09-11T03:49:37.422 INFO:tasks.rados.rados.2.smithi175.stdout:3028: finishing rollback tid 1 to smithi17557870-26 2024-09-11T03:49:37.422 INFO:tasks.rados.rados.2.smithi175.stdout:3032: copy_from oid 21 from oid 5 current snap is 340 2024-09-11T03:49:37.422 INFO:tasks.rados.rados.2.smithi175.stdout:3027: finishing write tid 5 to smithi17557870-9 2024-09-11T03:49:37.422 INFO:tasks.rados.rados.2.smithi175.stdout:3027: oid 9 updating version 0 to 1068 2024-09-11T03:49:37.422 INFO:tasks.rados.rados.2.smithi175.stdout:3027: oid 9 version 1068 is already newer than 1067 2024-09-11T03:49:37.422 INFO:tasks.rados.rados.2.smithi175.stdout:3027: oid 9 updating version 1068 to 1070 2024-09-11T03:49:37.422 INFO:tasks.rados.rados.2.smithi175.stdout:3027: oid 9 version 1070 is already newer than 1069 2024-09-11T03:49:37.422 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 1070 (ObjNum 1033 snap 340 seq_num 1033) dirty exists 2024-09-11T03:49:37.423 INFO:tasks.rados.rados.2.smithi175.stdout:3027: left oid 9 (ObjNum 1033 snap 340 seq_num 1033) 2024-09-11T03:49:37.423 INFO:tasks.rados.rados.2.smithi175.stdout:3028: finishing rollback tid 2 to smithi17557870-26 2024-09-11T03:49:37.423 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 963 (ObjNum 1004 snap 324 seq_num 1004) dirty exists 2024-09-11T03:49:37.423 INFO:tasks.rados.rados.2.smithi175.stdout:3030: finishing write tid 1 to smithi17557870-50 2024-09-11T03:49:37.423 INFO:tasks.rados.rados.2.smithi175.stdout:3029: finishing copy_from racing read to smithi17557870-17 2024-09-11T03:49:37.423 INFO:tasks.rados.rados.2.smithi175.stdout:3029: finishing copy_from to smithi17557870-17 2024-09-11T03:49:37.423 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 969 (ObjNum 984 snap 315 seq_num 984) dirty exists 2024-09-11T03:49:37.423 INFO:tasks.rados.rados.2.smithi175.stdout:3030: finishing write tid 2 to smithi17557870-50 2024-09-11T03:49:37.423 INFO:tasks.rados.rados.2.smithi175.stdout:3027: done (5 left) 2024-09-11T03:49:37.423 INFO:tasks.rados.rados.2.smithi175.stdout:3028: done (4 left) 2024-09-11T03:49:37.423 INFO:tasks.rados.rados.2.smithi175.stdout:3029: done (3 left) 2024-09-11T03:49:37.423 INFO:tasks.rados.rados.2.smithi175.stdout:3033: delete oid 13 current snap is 340 2024-09-11T03:49:37.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:37 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:49:37.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:37 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:49:37.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:37 smithi121 ceph-mon[30729]: 2.27 scrub starts 2024-09-11T03:49:37.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:37 smithi121 ceph-mon[30729]: 2.27 scrub ok 2024-09-11T03:49:37.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:37 smithi121 ceph-mon[30729]: osdmap e758: 12 total, 12 up, 12 in 2024-09-11T03:49:37.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:37 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4198222925' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:49:37.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:37 smithi121 ceph-mon[30729]: 1.5 scrub starts 2024-09-11T03:49:37.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:37 smithi121 ceph-mon[30729]: 1.5 scrub ok 2024-09-11T03:49:37.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:37 smithi121 ceph-mon[30729]: 1.1e scrub starts 2024-09-11T03:49:37.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:37 smithi121 ceph-mon[30729]: 1.1e scrub ok 2024-09-11T03:49:37.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:37 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:37.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:37 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:49:37.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:37 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:49:37.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:37 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4198222925' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:49:37.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:37 smithi121 ceph-mon[30729]: osdmap e759: 12 total, 12 up, 12 in 2024-09-11T03:49:37.425 INFO:tasks.rados.rados.2.smithi175.stdout:3033: done (3 left) 2024-09-11T03:49:37.426 INFO:tasks.rados.rados.2.smithi175.stdout:3034: read oid 47 snap -1 2024-09-11T03:49:37.426 INFO:tasks.rados.rados.2.smithi175.stdout:3034: expect (ObjNum 917 snap 294 seq_num 917) 2024-09-11T03:49:37.426 INFO:tasks.rados.rados.2.smithi175.stdout:3035: snap_remove snap 333 2024-09-11T03:49:37.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:37.472 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:37.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:37.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:37 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:49:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:37 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:49:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:37 smithi175 ceph-mon[30894]: 2.27 scrub starts 2024-09-11T03:49:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:37 smithi175 ceph-mon[30894]: 2.27 scrub ok 2024-09-11T03:49:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:37 smithi175 ceph-mon[30894]: osdmap e758: 12 total, 12 up, 12 in 2024-09-11T03:49:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:37 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4198222925' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:49:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:37 smithi175 ceph-mon[30894]: 1.5 scrub starts 2024-09-11T03:49:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:37 smithi175 ceph-mon[30894]: 1.5 scrub ok 2024-09-11T03:49:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:37 smithi175 ceph-mon[30894]: 1.1e scrub starts 2024-09-11T03:49:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:37 smithi175 ceph-mon[30894]: 1.1e scrub ok 2024-09-11T03:49:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:37 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:37 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:49:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:37 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:49:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:37 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4198222925' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:49:37.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:37 smithi175 ceph-mon[30894]: osdmap e759: 12 total, 12 up, 12 in 2024-09-11T03:49:37.574 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:37.596 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:37.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:37.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:37.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:37 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:49:37.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:37 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:49:37.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:37 smithi084 ceph-mon[27254]: 2.27 scrub starts 2024-09-11T03:49:37.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:37 smithi084 ceph-mon[27254]: 2.27 scrub ok 2024-09-11T03:49:37.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:37 smithi084 ceph-mon[27254]: osdmap e758: 12 total, 12 up, 12 in 2024-09-11T03:49:37.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4198222925' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:49:37.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:37 smithi084 ceph-mon[27254]: 1.5 scrub starts 2024-09-11T03:49:37.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:37 smithi084 ceph-mon[27254]: 1.5 scrub ok 2024-09-11T03:49:37.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:37 smithi084 ceph-mon[27254]: 1.1e scrub starts 2024-09-11T03:49:37.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:37 smithi084 ceph-mon[27254]: 1.1e scrub ok 2024-09-11T03:49:37.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:37 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:37.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:37 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:49:37.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:37 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:49:37.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4198222925' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:49:37.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:37 smithi084 ceph-mon[27254]: osdmap e759: 12 total, 12 up, 12 in 2024-09-11T03:49:37.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:37.719 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:37.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:37.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:37.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:37.843 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:37.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:37.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:37.945 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:37.967 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:37.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:37.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:38.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:38.071 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:38.077 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:38.090 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:38.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:38.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:38.192 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:38.208 INFO:tasks.rados.rados.2.smithi175.stdout:3031: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:49:38.208 INFO:tasks.rados.rados.2.smithi175.stdout:3031: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:49:38.208 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 1018 (ObjNum 975 snap 313 seq_num 975) dirty exists 2024-09-11T03:49:38.209 INFO:tasks.rados.rados.2.smithi175.stdout:3030: finishing write tid 3 to smithi17557870-50 2024-09-11T03:49:38.209 INFO:tasks.rados.rados.2.smithi175.stdout:3030: finishing write tid 4 to smithi17557870-50 2024-09-11T03:49:38.209 INFO:tasks.rados.rados.2.smithi175.stdout:3030: finishing write tid 5 to smithi17557870-50 2024-09-11T03:49:38.209 INFO:tasks.rados.rados.2.smithi175.stdout:3030: oid 50 updating version 0 to 1178 2024-09-11T03:49:38.209 INFO:tasks.rados.rados.2.smithi175.stdout:3030: oid 50 version 1178 is already newer than 1177 2024-09-11T03:49:38.209 INFO:tasks.rados.rados.2.smithi175.stdout:3030: oid 50 updating version 1178 to 1179 2024-09-11T03:49:38.209 INFO:tasks.rados.rados.2.smithi175.stdout:3030: oid 50 updating version 1179 to 1180 2024-09-11T03:49:38.209 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 1180 (ObjNum 1034 snap 340 seq_num 1034) dirty exists 2024-09-11T03:49:38.209 INFO:tasks.rados.rados.2.smithi175.stdout:3030: left oid 50 (ObjNum 1034 snap 340 seq_num 1034) 2024-09-11T03:49:38.209 INFO:tasks.rados.rados.2.smithi175.stdout:3034: expect (ObjNum 917 snap 294 seq_num 917) 2024-09-11T03:49:38.221 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:38.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:38.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:38.235 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:38 smithi084 ceph-mon[27254]: pgmap v1293: 77 pgs: 1 active+clean+scrubbing, 76 active+clean; 379 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:49:38.236 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:38 smithi084 ceph-mon[27254]: 1.9 scrub starts 2024-09-11T03:49:38.236 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:38 smithi084 ceph-mon[27254]: 1.9 scrub ok 2024-09-11T03:49:38.236 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:38 smithi084 ceph-mon[27254]: 1.7 scrub starts 2024-09-11T03:49:38.236 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:38 smithi084 ceph-mon[27254]: 1.7 scrub ok 2024-09-11T03:49:38.236 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:38 smithi084 ceph-mon[27254]: 2.3 scrub starts 2024-09-11T03:49:38.236 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:38 smithi084 ceph-mon[27254]: 2.3 scrub ok 2024-09-11T03:49:38.323 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:38.344 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:38.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:38.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:38.387 INFO:tasks.rados.rados.2.smithi175.stdout:3032: finishing copy_from racing read to smithi17557870-21 2024-09-11T03:49:38.387 INFO:tasks.rados.rados.2.smithi175.stdout:3032: finishing copy_from to smithi17557870-21 2024-09-11T03:49:38.387 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 951 (ObjNum 1022 snap 334 seq_num 1022) dirty exists 2024-09-11T03:49:38.387 INFO:tasks.rados.rados.2.smithi175.stdout:3030: done (4 left) 2024-09-11T03:49:38.387 INFO:tasks.rados.rados.2.smithi175.stdout:3031: done (3 left) 2024-09-11T03:49:38.387 INFO:tasks.rados.rados.2.smithi175.stdout:3032: done (2 left) 2024-09-11T03:49:38.388 INFO:tasks.rados.rados.2.smithi175.stdout:3034: done (1 left) 2024-09-11T03:49:38.388 INFO:tasks.rados.rados.2.smithi175.stdout:3035: done (0 left) 2024-09-11T03:49:38.388 INFO:tasks.rados.rados.2.smithi175.stdout:3036: write (excl) oid 21 current snap is 340 2024-09-11T03:49:38.388 INFO:tasks.rados.rados.2.smithi175.stdout:3036: seq_num 1036 ranges {612052=581972,1657698=497363,2717044=695008} 2024-09-11T03:49:38.393 INFO:tasks.rados.rados.2.smithi175.stdout:3036: writing smithi17557870-21 from 612052 to 1194024 tid 1 2024-09-11T03:49:38.398 INFO:tasks.rados.rados.2.smithi175.stdout:3036: writing smithi17557870-21 from 1657698 to 2155061 tid 2 2024-09-11T03:49:38.405 INFO:tasks.rados.rados.2.smithi175.stdout:3036: writing smithi17557870-21 from 2717044 to 3412052 tid 3 2024-09-11T03:49:38.405 INFO:tasks.rados.rados.2.smithi175.stdout:3037: write (excl) oid 33 current snap is 340 2024-09-11T03:49:38.406 INFO:tasks.rados.rados.2.smithi175.stdout:3036: finishing write tid 1 to smithi17557870-21 2024-09-11T03:49:38.406 INFO:tasks.rados.rados.2.smithi175.stdout:3036: finishing write tid 2 to smithi17557870-21 2024-09-11T03:49:38.406 INFO:tasks.rados.rados.2.smithi175.stdout:3037: seq_num 1037 ranges {628859=657221,1716180=641021,3144467=284392} 2024-09-11T03:49:38.412 INFO:tasks.rados.rados.2.smithi175.stdout:3037: writing smithi17557870-33 from 628859 to 1286080 tid 1 2024-09-11T03:49:38.418 INFO:tasks.rados.rados.2.smithi175.stdout:3037: writing smithi17557870-33 from 1716180 to 2357201 tid 2 2024-09-11T03:49:38.424 INFO:tasks.rados.rados.2.smithi175.stdout:3037: writing smithi17557870-33 from 3144467 to 3428859 tid 3 2024-09-11T03:49:38.424 INFO:tasks.rados.rados.2.smithi175.stdout:3038: delete oid 29 current snap is 340 2024-09-11T03:49:38.424 INFO:tasks.rados.rados.2.smithi175.stdout:3036: finishing write tid 3 to smithi17557870-21 2024-09-11T03:49:38.424 INFO:tasks.rados.rados.2.smithi175.stdout:3036: finishing write tid 4 to smithi17557870-21 2024-09-11T03:49:38.424 INFO:tasks.rados.rados.2.smithi175.stdout:3036: finishing write tid 5 to smithi17557870-21 2024-09-11T03:49:38.424 INFO:tasks.rados.rados.2.smithi175.stdout:3036: oid 21 updating version 0 to 1079 2024-09-11T03:49:38.424 INFO:tasks.rados.rados.2.smithi175.stdout:3036: oid 21 version 1079 is already newer than 1078 2024-09-11T03:49:38.425 INFO:tasks.rados.rados.2.smithi175.stdout:3036: oid 21 updating version 1079 to 1080 2024-09-11T03:49:38.425 INFO:tasks.rados.rados.2.smithi175.stdout:3036: oid 21 version 1080 is already newer than 1077 2024-09-11T03:49:38.425 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 1080 (ObjNum 1036 snap 340 seq_num 1036) dirty exists 2024-09-11T03:49:38.425 INFO:tasks.rados.rados.2.smithi175.stdout:3036: left oid 21 (ObjNum 1036 snap 340 seq_num 1036) 2024-09-11T03:49:38.425 INFO:tasks.rados.rados.2.smithi175.stdout:3037: finishing write tid 1 to smithi17557870-33 2024-09-11T03:49:38.425 INFO:tasks.rados.rados.2.smithi175.stdout:3037: finishing write tid 2 to smithi17557870-33 2024-09-11T03:49:38.425 INFO:tasks.rados.rados.2.smithi175.stdout:3036: done (2 left) 2024-09-11T03:49:38.426 INFO:tasks.rados.rados.2.smithi175.stdout:3038: done (1 left) 2024-09-11T03:49:38.426 INFO:tasks.rados.rados.2.smithi175.stdout:3039: delete oid 16 current snap is 340 2024-09-11T03:49:38.427 INFO:tasks.rados.rados.2.smithi175.stdout:3037: finishing write tid 3 to smithi17557870-33 2024-09-11T03:49:38.427 INFO:tasks.rados.rados.2.smithi175.stdout:3037: finishing write tid 4 to smithi17557870-33 2024-09-11T03:49:38.427 INFO:tasks.rados.rados.2.smithi175.stdout:3037: finishing write tid 5 to smithi17557870-33 2024-09-11T03:49:38.427 INFO:tasks.rados.rados.2.smithi175.stdout:3037: oid 33 updating version 0 to 1186 2024-09-11T03:49:38.427 INFO:tasks.rados.rados.2.smithi175.stdout:3037: oid 33 updating version 1186 to 1187 2024-09-11T03:49:38.427 INFO:tasks.rados.rados.2.smithi175.stdout:3037: oid 33 version 1187 is already newer than 1185 2024-09-11T03:49:38.427 INFO:tasks.rados.rados.2.smithi175.stdout:3037: oid 33 version 1187 is already newer than 1184 2024-09-11T03:49:38.428 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 1187 (ObjNum 1037 snap 340 seq_num 1037) dirty exists 2024-09-11T03:49:38.428 INFO:tasks.rados.rados.2.smithi175.stdout:3037: left oid 33 (ObjNum 1037 snap 340 seq_num 1037) 2024-09-11T03:49:38.428 INFO:tasks.rados.rados.2.smithi175.stdout:3037: done (1 left) 2024-09-11T03:49:38.428 INFO:tasks.rados.rados.2.smithi175.stdout:3039: done (0 left) 2024-09-11T03:49:38.428 INFO:tasks.rados.rados.2.smithi175.stdout:3040: read oid 6 snap -1 2024-09-11T03:49:38.428 INFO:tasks.rados.rados.2.smithi175.stdout:3040: expect deleted 2024-09-11T03:49:38.428 INFO:tasks.rados.rados.2.smithi175.stdout:3041: rollback oid 44 current snap is 340 2024-09-11T03:49:38.428 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 44 to 322 2024-09-11T03:49:38.428 INFO:tasks.rados.rados.2.smithi175.stdout:3042: read oid 19 snap -1 2024-09-11T03:49:38.428 INFO:tasks.rados.rados.2.smithi175.stdout:3042: expect (ObjNum 963 snap 310 seq_num 963) 2024-09-11T03:49:38.428 INFO:tasks.rados.rados.2.smithi175.stdout:3043: copy_from oid 23 from oid 5 current snap is 340 2024-09-11T03:49:38.428 INFO:tasks.rados.rados.2.smithi175.stdout:3044: delete oid 14 current snap is 340 2024-09-11T03:49:38.429 INFO:tasks.rados.rados.2.smithi175.stdout:3040: done (4 left) 2024-09-11T03:49:38.429 INFO:tasks.rados.rados.2.smithi175.stdout:3044: done (3 left) 2024-09-11T03:49:38.429 INFO:tasks.rados.rados.2.smithi175.stdout:3045: snap_remove snap 320 2024-09-11T03:49:38.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:38.467 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:38.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:38.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:38 smithi175 ceph-mon[30894]: pgmap v1293: 77 pgs: 1 active+clean+scrubbing, 76 active+clean; 379 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:49:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:38 smithi175 ceph-mon[30894]: 1.9 scrub starts 2024-09-11T03:49:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:38 smithi175 ceph-mon[30894]: 1.9 scrub ok 2024-09-11T03:49:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:38 smithi175 ceph-mon[30894]: 1.7 scrub starts 2024-09-11T03:49:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:38 smithi175 ceph-mon[30894]: 1.7 scrub ok 2024-09-11T03:49:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:38 smithi175 ceph-mon[30894]: 2.3 scrub starts 2024-09-11T03:49:38.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:38 smithi175 ceph-mon[30894]: 2.3 scrub ok 2024-09-11T03:49:38.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:38.591 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:38.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:38.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:38 smithi121 ceph-mon[30729]: pgmap v1293: 77 pgs: 1 active+clean+scrubbing, 76 active+clean; 379 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:49:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:38 smithi121 ceph-mon[30729]: 1.9 scrub starts 2024-09-11T03:49:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:38 smithi121 ceph-mon[30729]: 1.9 scrub ok 2024-09-11T03:49:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:38 smithi121 ceph-mon[30729]: 1.7 scrub starts 2024-09-11T03:49:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:38 smithi121 ceph-mon[30729]: 1.7 scrub ok 2024-09-11T03:49:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:38 smithi121 ceph-mon[30729]: 2.3 scrub starts 2024-09-11T03:49:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:38 smithi121 ceph-mon[30729]: 2.3 scrub ok 2024-09-11T03:49:38.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:38.715 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:38.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:38.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:38.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:38.829 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:49:38.829 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:49:38.838 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:38.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:38.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:38.845 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:49:38.845 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:49:38.861 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:49:38.861 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:49:38.877 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:49:38.877 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:49:38.896 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:49:38.896 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:49:38.912 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:49:38.912 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:49:38.928 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:49:38.929 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:49:38.940 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:38.945 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:49:38.945 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:49:38.962 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:49:38.963 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:49:38.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:38.964 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:38.964 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:38.979 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:49:38.980 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:49:38.997 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:49:38.997 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:49:39.008 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:49:39.008 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:49:39.014 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:49:39.014 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:49:39.064 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:39.086 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:39.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:39.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:39.188 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:39.212 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:39.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:39.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:39.217 INFO:tasks.rados.rados.2.smithi175.stdout:3045: done (3 left) 2024-09-11T03:49:39.217 INFO:tasks.rados.rados.2.smithi175.stdout:3046: read oid 21 snap -1 2024-09-11T03:49:39.217 INFO:tasks.rados.rados.2.smithi175.stdout:3046: expect (ObjNum 1036 snap 340 seq_num 1036) 2024-09-11T03:49:39.218 INFO:tasks.rados.rados.2.smithi175.stdout:3047: copy_from oid 12 from oid 16 current snap is 340 2024-09-11T03:49:39.218 INFO:tasks.rados.rados.2.smithi175.stdout:3048: write (excl) oid 32 current snap is 340 2024-09-11T03:49:39.218 INFO:tasks.rados.rados.2.smithi175.stdout:3048: seq_num 1041 ranges {696087=558217,1833755=648359,3137806=358281} 2024-09-11T03:49:39.225 INFO:tasks.rados.rados.2.smithi175.stdout:3048: writing smithi17557870-32 from 696087 to 1254304 tid 1 2024-09-11T03:49:39.232 INFO:tasks.rados.rados.2.smithi175.stdout:3048: writing smithi17557870-32 from 1833755 to 2482114 tid 2 2024-09-11T03:49:39.238 INFO:tasks.rados.rados.2.smithi175.stdout:3048: writing smithi17557870-32 from 3137806 to 3496087 tid 3 2024-09-11T03:49:39.238 INFO:tasks.rados.rados.2.smithi175.stdout:3049: read oid 36 snap 339 2024-09-11T03:49:39.238 INFO:tasks.rados.rados.2.smithi175.stdout:3049: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:49:39.238 INFO:tasks.rados.rados.2.smithi175.stdout:3050: write oid 42 current snap is 340 2024-09-11T03:49:39.238 INFO:tasks.rados.rados.2.smithi175.stdout:3041: finishing rollback tid 0 to smithi17557870-44 2024-09-11T03:49:39.239 INFO:tasks.rados.rados.2.smithi175.stdout:3041: finishing rollback tid 1 to smithi17557870-44 2024-09-11T03:49:39.239 INFO:tasks.rados.rados.2.smithi175.stdout:3041: finishing rollback tid 2 to smithi17557870-44 2024-09-11T03:49:39.239 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 1019 (ObjNum 939 snap 297 seq_num 939) dirty exists 2024-09-11T03:49:39.239 INFO:tasks.rados.rados.2.smithi175.stdout:3050: seq_num 1042 ranges {712894=749819,2008590=792017,3281582=231312} 2024-09-11T03:49:39.246 INFO:tasks.rados.rados.2.smithi175.stdout:3050: writing smithi17557870-42 from 712894 to 1462713 tid 1 2024-09-11T03:49:39.253 INFO:tasks.rados.rados.2.smithi175.stdout:3050: writing smithi17557870-42 from 2008590 to 2800607 tid 2 2024-09-11T03:49:39.257 INFO:tasks.rados.rados.2.smithi175.stdout:3050: writing smithi17557870-42 from 3281582 to 3512894 tid 3 2024-09-11T03:49:39.257 INFO:tasks.rados.rados.2.smithi175.stdout:3041: done (7 left) 2024-09-11T03:49:39.257 INFO:tasks.rados.rados.2.smithi175.stdout:3051: snap_remove snap 279 2024-09-11T03:49:39.314 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:39.336 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:39.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:39.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:39.438 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:39.460 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:39.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:39.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:39.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:39 smithi175 ceph-mon[30894]: osdmap e760: 12 total, 12 up, 12 in 2024-09-11T03:49:39.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:39 smithi175 ceph-mon[30894]: 2.17 scrub starts 2024-09-11T03:49:39.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:39 smithi175 ceph-mon[30894]: 2.17 scrub ok 2024-09-11T03:49:39.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:39 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4176152638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:39.562 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:39.584 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:39.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:39.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:39.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:39 smithi084 ceph-mon[27254]: osdmap e760: 12 total, 12 up, 12 in 2024-09-11T03:49:39.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:39 smithi084 ceph-mon[27254]: 2.17 scrub starts 2024-09-11T03:49:39.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:39 smithi084 ceph-mon[27254]: 2.17 scrub ok 2024-09-11T03:49:39.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:39 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4176152638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:39 smithi121 ceph-mon[30729]: osdmap e760: 12 total, 12 up, 12 in 2024-09-11T03:49:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:39 smithi121 ceph-mon[30729]: 2.17 scrub starts 2024-09-11T03:49:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:39 smithi121 ceph-mon[30729]: 2.17 scrub ok 2024-09-11T03:49:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:39 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4176152638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:39.686 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:39.708 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:39.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:39.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:39.810 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:39.836 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:39.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:39.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:39.937 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:39.961 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:39.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:39.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:40.004 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:49:40.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:40.085 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:40.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:40.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:40.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T03:49:40.187 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:40.209 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:40.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:40.223 INFO:tasks.rados.rados.2.smithi175.stdout:3051: done (7 left) 2024-09-11T03:49:40.224 INFO:tasks.rados.rados.2.smithi175.stdout:3052: snap_remove snap 332 2024-09-11T03:49:40.311 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:40.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:40 smithi084 ceph-mon[27254]: pgmap v1296: 77 pgs: 77 active+clean; 397 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:49:40.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:40 smithi084 ceph-mon[27254]: osdmap e761: 12 total, 12 up, 12 in 2024-09-11T03:49:40.337 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:40.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:40.338 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:40.439 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:40.464 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:40.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:40.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:40.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:40 smithi175 ceph-mon[30894]: pgmap v1296: 77 pgs: 77 active+clean; 397 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:49:40.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:40 smithi175 ceph-mon[30894]: osdmap e761: 12 total, 12 up, 12 in 2024-09-11T03:49:40.566 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:40.588 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:40.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:40.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:40.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:40 smithi121 ceph-mon[30729]: pgmap v1296: 77 pgs: 77 active+clean; 397 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:49:40.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:40 smithi121 ceph-mon[30729]: osdmap e761: 12 total, 12 up, 12 in 2024-09-11T03:49:40.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:40.713 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:40.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:40.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:40.773 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:40.797 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:40.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:40.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:40.842 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:40.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:40.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:40.969 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:40.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:40.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:41.071 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:41.097 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:41.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:41.098 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:41.198 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:41.227 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:41.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:41.233 INFO:tasks.rados.rados.2.smithi175.stdout:3052: done (7 left) 2024-09-11T03:49:41.233 INFO:tasks.rados.rados.2.smithi175.stdout:3053: read oid 25 snap -1 2024-09-11T03:49:41.233 INFO:tasks.rados.rados.2.smithi175.stdout:3053: expect (ObjNum 950 snap 303 seq_num 950) 2024-09-11T03:49:41.233 INFO:tasks.rados.rados.2.smithi175.stdout:3054: write oid 8 current snap is 340 2024-09-11T03:49:41.233 INFO:tasks.rados.rados.2.smithi175.stdout:3054: seq_num 1043 ranges {729701=425068,1667072=652028,2909005=485180} 2024-09-11T03:49:41.240 INFO:tasks.rados.rados.2.smithi175.stdout:3054: writing smithi17557870-8 from 729701 to 1154769 tid 1 2024-09-11T03:49:41.246 INFO:tasks.rados.rados.2.smithi175.stdout:3054: writing smithi17557870-8 from 1667072 to 2319100 tid 2 2024-09-11T03:49:41.253 INFO:tasks.rados.rados.2.smithi175.stdout:3054: writing smithi17557870-8 from 2909005 to 3394185 tid 3 2024-09-11T03:49:41.253 INFO:tasks.rados.rados.2.smithi175.stdout:3055: rollback oid 30 current snap is 340 2024-09-11T03:49:41.253 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 30 to 296 2024-09-11T03:49:41.253 INFO:tasks.rados.rados.2.smithi175.stdout:3043: finishing copy_from racing read to smithi17557870-23 2024-09-11T03:49:41.253 INFO:tasks.rados.rados.2.smithi175.stdout:3042: expect (ObjNum 963 snap 310 seq_num 963) 2024-09-11T03:49:41.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:41 smithi084 ceph-mon[27254]: 1.1c scrub starts 2024-09-11T03:49:41.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:41 smithi084 ceph-mon[27254]: 1.1c scrub ok 2024-09-11T03:49:41.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:41 smithi084 ceph-mon[27254]: osdmap e762: 12 total, 12 up, 12 in 2024-09-11T03:49:41.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:41 smithi084 ceph-mon[27254]: 2.11 scrub starts 2024-09-11T03:49:41.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:41 smithi084 ceph-mon[27254]: 2.11 scrub ok 2024-09-11T03:49:41.329 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:41.350 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:41.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:41.351 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:41.373 INFO:tasks.rados.rados.2.smithi175.stdout:3043: finishing copy_from to smithi17557870-23 2024-09-11T03:49:41.373 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 951 (ObjNum 1022 snap 334 seq_num 1022) dirty exists 2024-09-11T03:49:41.373 INFO:tasks.rados.rados.2.smithi175.stdout:3047: finishing copy_from to smithi17557870-12 2024-09-11T03:49:41.373 INFO:tasks.rados.rados.2.smithi175.stdout:3047: got expected ENOENT (src dne) 2024-09-11T03:49:41.373 INFO:tasks.rados.rados.2.smithi175.stdout:3047: finishing copy_from racing read to smithi17557870-12 2024-09-11T03:49:41.373 INFO:tasks.rados.rados.2.smithi175.stdout:3048: finishing write tid 1 to smithi17557870-32 2024-09-11T03:49:41.373 INFO:tasks.rados.rados.2.smithi175.stdout:3046: expect (ObjNum 1036 snap 340 seq_num 1036) 2024-09-11T03:49:41.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:41.477 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:41.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:41.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:41.515 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:49:41.515 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:49:41.531 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:49:41.531 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:49:41.531 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-09-11T03:49:41.532 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:49:41.532 INFO:tasks.rados.rados.2.smithi175.stdout:3048: finishing write tid 2 to smithi17557870-32 2024-09-11T03:49:41.533 INFO:tasks.rados.rados.2.smithi175.stdout:3048: finishing write tid 3 to smithi17557870-32 2024-09-11T03:49:41.533 INFO:tasks.rados.rados.2.smithi175.stdout:3049: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:49:41.578 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:41.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:41.600 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:41.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:41 smithi121 ceph-mon[30729]: 1.1c scrub starts 2024-09-11T03:49:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:41 smithi121 ceph-mon[30729]: 1.1c scrub ok 2024-09-11T03:49:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:41 smithi121 ceph-mon[30729]: osdmap e762: 12 total, 12 up, 12 in 2024-09-11T03:49:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:41 smithi121 ceph-mon[30729]: 2.11 scrub starts 2024-09-11T03:49:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:41 smithi121 ceph-mon[30729]: 2.11 scrub ok 2024-09-11T03:49:41.702 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:41.722 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:41.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:41.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:41.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:41 smithi175 ceph-mon[30894]: 1.1c scrub starts 2024-09-11T03:49:41.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:41 smithi175 ceph-mon[30894]: 1.1c scrub ok 2024-09-11T03:49:41.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:41 smithi175 ceph-mon[30894]: osdmap e762: 12 total, 12 up, 12 in 2024-09-11T03:49:41.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:41 smithi175 ceph-mon[30894]: 2.11 scrub starts 2024-09-11T03:49:41.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:41 smithi175 ceph-mon[30894]: 2.11 scrub ok 2024-09-11T03:49:41.768 INFO:tasks.rados.rados.2.smithi175.stdout:3048: finishing write tid 4 to smithi17557870-32 2024-09-11T03:49:41.768 INFO:tasks.rados.rados.2.smithi175.stdout:3048: finishing write tid 5 to smithi17557870-32 2024-09-11T03:49:41.768 INFO:tasks.rados.rados.2.smithi175.stdout:3048: oid 32 updating version 0 to 983 2024-09-11T03:49:41.768 INFO:tasks.rados.rados.2.smithi175.stdout:3048: oid 32 updating version 983 to 984 2024-09-11T03:49:41.768 INFO:tasks.rados.rados.2.smithi175.stdout:3048: oid 32 version 984 is already newer than 982 2024-09-11T03:49:41.768 INFO:tasks.rados.rados.2.smithi175.stdout:3048: oid 32 version 984 is already newer than 981 2024-09-11T03:49:41.768 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 984 (ObjNum 1041 snap 340 seq_num 1041) dirty exists 2024-09-11T03:49:41.768 INFO:tasks.rados.rados.2.smithi175.stdout:3048: left oid 32 (ObjNum 1041 snap 340 seq_num 1041) 2024-09-11T03:49:41.768 INFO:tasks.rados.rados.2.smithi175.stdout:3050: finishing write tid 1 to smithi17557870-42 2024-09-11T03:49:41.769 INFO:tasks.rados.rados.2.smithi175.stdout:3050: finishing write tid 2 to smithi17557870-42 2024-09-11T03:49:41.769 INFO:tasks.rados.rados.2.smithi175.stdout:3050: finishing write tid 3 to smithi17557870-42 2024-09-11T03:49:41.769 INFO:tasks.rados.rados.2.smithi175.stdout:3050: finishing write tid 4 to smithi17557870-42 2024-09-11T03:49:41.769 INFO:tasks.rados.rados.2.smithi175.stdout:3050: finishing write tid 5 to smithi17557870-42 2024-09-11T03:49:41.769 INFO:tasks.rados.rados.2.smithi175.stdout:3050: oid 42 updating version 0 to 963 2024-09-11T03:49:41.769 INFO:tasks.rados.rados.2.smithi175.stdout:3050: oid 42 updating version 963 to 965 2024-09-11T03:49:41.769 INFO:tasks.rados.rados.2.smithi175.stdout:3050: oid 42 version 965 is already newer than 964 2024-09-11T03:49:41.769 INFO:tasks.rados.rados.2.smithi175.stdout:3050: oid 42 version 965 is already newer than 962 2024-09-11T03:49:41.769 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 965 (ObjNum 1042 snap 340 seq_num 1042) dirty exists 2024-09-11T03:49:41.769 INFO:tasks.rados.rados.2.smithi175.stdout:3050: left oid 42 (ObjNum 1042 snap 340 seq_num 1042) 2024-09-11T03:49:41.769 INFO:tasks.rados.rados.2.smithi175.stdout:3054: finishing write tid 1 to smithi17557870-8 2024-09-11T03:49:41.770 INFO:tasks.rados.rados.2.smithi175.stdout:3053: expect (ObjNum 950 snap 303 seq_num 950) 2024-09-11T03:49:41.824 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:41.845 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:41.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:41.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:41.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:41.973 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:41.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:42.003 INFO:tasks.rados.rados.2.smithi175.stdout:3054: finishing write tid 2 to smithi17557870-8 2024-09-11T03:49:42.004 INFO:tasks.rados.rados.2.smithi175.stdout:3054: finishing write tid 3 to smithi17557870-8 2024-09-11T03:49:42.004 INFO:tasks.rados.rados.2.smithi175.stdout:3054: finishing write tid 4 to smithi17557870-8 2024-09-11T03:49:42.004 INFO:tasks.rados.rados.2.smithi175.stdout:3054: finishing write tid 5 to smithi17557870-8 2024-09-11T03:49:42.004 INFO:tasks.rados.rados.2.smithi175.stdout:3054: oid 8 updating version 0 to 1024 2024-09-11T03:49:42.004 INFO:tasks.rados.rados.2.smithi175.stdout:3054: oid 8 updating version 1024 to 1027 2024-09-11T03:49:42.004 INFO:tasks.rados.rados.2.smithi175.stdout:3054: oid 8 version 1027 is already newer than 1025 2024-09-11T03:49:42.004 INFO:tasks.rados.rados.2.smithi175.stdout:3054: oid 8 version 1027 is already newer than 1026 2024-09-11T03:49:42.004 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 1027 (ObjNum 1043 snap 340 seq_num 1043) dirty exists 2024-09-11T03:49:42.005 INFO:tasks.rados.rados.2.smithi175.stdout:3054: left oid 8 (ObjNum 1043 snap 340 seq_num 1043) 2024-09-11T03:49:42.005 INFO:tasks.rados.rados.2.smithi175.stdout:3055: finishing rollback tid 0 to smithi17557870-30 2024-09-11T03:49:42.005 INFO:tasks.rados.rados.2.smithi175.stdout:3055: finishing rollback tid 1 to smithi17557870-30 2024-09-11T03:49:42.005 INFO:tasks.rados.rados.2.smithi175.stdout:3042: done (9 left) 2024-09-11T03:49:42.005 INFO:tasks.rados.rados.2.smithi175.stdout:3043: done (8 left) 2024-09-11T03:49:42.005 INFO:tasks.rados.rados.2.smithi175.stdout:3046: done (7 left) 2024-09-11T03:49:42.005 INFO:tasks.rados.rados.2.smithi175.stdout:3047: done (6 left) 2024-09-11T03:49:42.005 INFO:tasks.rados.rados.2.smithi175.stdout:3048: done (5 left) 2024-09-11T03:49:42.005 INFO:tasks.rados.rados.2.smithi175.stdout:3049: done (4 left) 2024-09-11T03:49:42.005 INFO:tasks.rados.rados.2.smithi175.stdout:3050: done (3 left) 2024-09-11T03:49:42.006 INFO:tasks.rados.rados.2.smithi175.stdout:3053: done (2 left) 2024-09-11T03:49:42.006 INFO:tasks.rados.rados.2.smithi175.stdout:3054: done (1 left) 2024-09-11T03:49:42.006 INFO:tasks.rados.rados.2.smithi175.stdout:3056: read oid 37 snap -1 2024-09-11T03:49:42.006 INFO:tasks.rados.rados.2.smithi175.stdout:3056: expect deleted 2024-09-11T03:49:42.006 INFO:tasks.rados.rados.2.smithi175.stdout:3057: read oid 41 snap -1 2024-09-11T03:49:42.006 INFO:tasks.rados.rados.2.smithi175.stdout:3057: expect deleted 2024-09-11T03:49:42.006 INFO:tasks.rados.rados.2.smithi175.stdout:3058: copy_from oid 12 from oid 26 current snap is 340 2024-09-11T03:49:42.006 INFO:tasks.rados.rados.2.smithi175.stdout:3055: finishing rollback tid 2 to smithi17557870-30 2024-09-11T03:49:42.006 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 1030 (ObjNum 895 snap 284 seq_num 895) dirty exists 2024-09-11T03:49:42.006 INFO:tasks.rados.rados.2.smithi175.stdout:3055: done (3 left) 2024-09-11T03:49:42.006 INFO:tasks.rados.rados.2.smithi175.stdout:3059: snap_remove snap 327 2024-09-11T03:49:42.023 INFO:tasks.rados.rados.2.smithi175.stdout:3059: done (3 left) 2024-09-11T03:49:42.023 INFO:tasks.rados.rados.2.smithi175.stdout:3060: snap_remove snap 308 2024-09-11T03:49:42.025 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:49:42.030 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:42.075 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:42.096 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:42.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:42.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:42.198 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:42.224 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:42.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:42.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:42.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:42.351 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:42.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:42.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:42.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:42 smithi084 ceph-mon[27254]: pgmap v1299: 77 pgs: 77 active+clean; 397 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:49:42.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:42 smithi084 ceph-mon[27254]: 2.10 scrub starts 2024-09-11T03:49:42.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:42 smithi084 ceph-mon[27254]: 2.10 scrub ok 2024-09-11T03:49:42.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:42 smithi084 ceph-mon[27254]: 1.11 scrub starts 2024-09-11T03:49:42.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:42 smithi084 ceph-mon[27254]: 1.11 scrub ok 2024-09-11T03:49:42.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:42 smithi084 ceph-mon[27254]: osdmap e763: 12 total, 12 up, 12 in 2024-09-11T03:49:42.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:42 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/623744291' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:49:42.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:42 smithi084 ceph-mon[27254]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:49:42.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:42 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:42.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:42 smithi084 ceph-mon[27254]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:49:42.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:42 smithi084 ceph-mon[27254]: osdmap e764: 12 total, 12 up, 12 in 2024-09-11T03:49:42.453 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:42.475 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:42.476 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:42 smithi175 ceph-mon[30894]: pgmap v1299: 77 pgs: 77 active+clean; 397 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:49:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:42 smithi175 ceph-mon[30894]: 2.10 scrub starts 2024-09-11T03:49:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:42 smithi175 ceph-mon[30894]: 2.10 scrub ok 2024-09-11T03:49:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:42 smithi175 ceph-mon[30894]: 1.11 scrub starts 2024-09-11T03:49:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:42 smithi175 ceph-mon[30894]: 1.11 scrub ok 2024-09-11T03:49:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:42 smithi175 ceph-mon[30894]: osdmap e763: 12 total, 12 up, 12 in 2024-09-11T03:49:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:42 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/623744291' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:49:42.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:42 smithi175 ceph-mon[30894]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:49:42.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:42 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:42.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:42 smithi175 ceph-mon[30894]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:49:42.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:42 smithi175 ceph-mon[30894]: osdmap e764: 12 total, 12 up, 12 in 2024-09-11T03:49:42.577 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:42.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:42.599 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:42.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:42.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:42 smithi121 ceph-mon[30729]: pgmap v1299: 77 pgs: 77 active+clean; 397 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:49:42.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:42 smithi121 ceph-mon[30729]: 2.10 scrub starts 2024-09-11T03:49:42.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:42 smithi121 ceph-mon[30729]: 2.10 scrub ok 2024-09-11T03:49:42.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:42 smithi121 ceph-mon[30729]: 1.11 scrub starts 2024-09-11T03:49:42.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:42 smithi121 ceph-mon[30729]: 1.11 scrub ok 2024-09-11T03:49:42.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:42 smithi121 ceph-mon[30729]: osdmap e763: 12 total, 12 up, 12 in 2024-09-11T03:49:42.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:42 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/623744291' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:49:42.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:42 smithi121 ceph-mon[30729]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:49:42.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:42 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:42.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:42 smithi121 ceph-mon[30729]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:49:42.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:42 smithi121 ceph-mon[30729]: osdmap e764: 12 total, 12 up, 12 in 2024-09-11T03:49:42.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:42.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:49:42.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:42.726 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:42.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:42.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T03:49:42.827 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:42.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:42.849 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:42.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:42.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:42.976 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:42.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:42.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:43.022 INFO:tasks.rados.rados.2.smithi175.stdout:3060: done (3 left) 2024-09-11T03:49:43.022 INFO:tasks.rados.rados.2.smithi175.stdout:3061: copy_from oid 10 from oid 32 current snap is 340 2024-09-11T03:49:43.022 INFO:tasks.rados.rados.2.smithi175.stdout:3058: finishing copy_from racing read to smithi17557870-12 2024-09-11T03:49:43.023 INFO:tasks.rados.rados.2.smithi175.stdout:3058: finishing copy_from to smithi17557870-12 2024-09-11T03:49:43.023 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 963 (ObjNum 1004 snap 324 seq_num 1004) dirty exists 2024-09-11T03:49:43.023 INFO:tasks.rados.rados.2.smithi175.stdout:3056: done (3 left) 2024-09-11T03:49:43.023 INFO:tasks.rados.rados.2.smithi175.stdout:3057: done (2 left) 2024-09-11T03:49:43.023 INFO:tasks.rados.rados.2.smithi175.stdout:3058: done (1 left) 2024-09-11T03:49:43.023 INFO:tasks.rados.rados.2.smithi175.stdout:3062: write oid 25 current snap is 340 2024-09-11T03:49:43.023 INFO:tasks.rados.rados.2.smithi175.stdout:3062: seq_num 1044 ranges {746508=500317,2009201=795686,3220075=326433} 2024-09-11T03:49:43.030 INFO:tasks.rados.rados.2.smithi175.stdout:3062: writing smithi17557870-25 from 746508 to 1246825 tid 1 2024-09-11T03:49:43.038 INFO:tasks.rados.rados.2.smithi175.stdout:3062: writing smithi17557870-25 from 2009201 to 2804887 tid 2 2024-09-11T03:49:43.043 INFO:tasks.rados.rados.2.smithi175.stdout:3062: writing smithi17557870-25 from 3220075 to 3546508 tid 3 2024-09-11T03:49:43.044 INFO:tasks.rados.rados.2.smithi175.stdout:3063: read oid 6 snap -1 2024-09-11T03:49:43.044 INFO:tasks.rados.rados.2.smithi175.stdout:3063: expect deleted 2024-09-11T03:49:43.044 INFO:tasks.rados.rados.2.smithi175.stdout:3064: copy_from oid 1 from oid 40 current snap is 340 2024-09-11T03:49:43.044 INFO:tasks.rados.rados.2.smithi175.stdout:3062: finishing write tid 1 to smithi17557870-25 2024-09-11T03:49:43.044 INFO:tasks.rados.rados.2.smithi175.stdout:3061: finishing copy_from racing read to smithi17557870-10 2024-09-11T03:49:43.044 INFO:tasks.rados.rados.2.smithi175.stdout:3065: copy_from oid 22 from oid 3 current snap is 340 2024-09-11T03:49:43.044 INFO:tasks.rados.rados.2.smithi175.stdout:3066: read oid 46 snap 334 2024-09-11T03:49:43.044 INFO:tasks.rados.rados.2.smithi175.stdout:3066: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:49:43.044 INFO:tasks.rados.rados.2.smithi175.stdout:3067: rollback oid 12 current snap is 340 2024-09-11T03:49:43.044 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 12 to 337 2024-09-11T03:49:43.044 INFO:tasks.rados.rados.2.smithi175.stdout:3068: read oid 39 snap -1 2024-09-11T03:49:43.044 INFO:tasks.rados.rados.2.smithi175.stdout:3068: expect (ObjNum 1030 snap 338 seq_num 1030) 2024-09-11T03:49:43.045 INFO:tasks.rados.rados.2.smithi175.stdout:3069: delete oid 31 current snap is 340 2024-09-11T03:49:43.045 INFO:tasks.rados.rados.2.smithi175.stdout:3065: finishing copy_from to smithi17557870-22 2024-09-11T03:49:43.045 INFO:tasks.rados.rados.2.smithi175.stdout:3065: got expected ENOENT (src dne) 2024-09-11T03:49:43.045 INFO:tasks.rados.rados.2.smithi175.stdout:3065: finishing copy_from racing read to smithi17557870-22 2024-09-11T03:49:43.046 INFO:tasks.rados.rados.2.smithi175.stdout:3066: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:49:43.078 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:43.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:43.101 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:43.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:43.203 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:43.225 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:43.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:43.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:43.283 INFO:tasks.rados.rados.2.smithi175.stdout:3062: finishing write tid 2 to smithi17557870-25 2024-09-11T03:49:43.283 INFO:tasks.rados.rados.2.smithi175.stdout:3062: finishing write tid 3 to smithi17557870-25 2024-09-11T03:49:43.283 INFO:tasks.rados.rados.2.smithi175.stdout:3062: finishing write tid 4 to smithi17557870-25 2024-09-11T03:49:43.283 INFO:tasks.rados.rados.2.smithi175.stdout:3067: finishing rollback tid 0 to smithi17557870-12 2024-09-11T03:49:43.283 INFO:tasks.rados.rados.2.smithi175.stdout:3067: finishing rollback tid 1 to smithi17557870-12 2024-09-11T03:49:43.283 INFO:tasks.rados.rados.2.smithi175.stdout:3067: finishing rollback tid 2 to smithi17557870-12 2024-09-11T03:49:43.283 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 1186 (ObjNum 952 snap 304 seq_num 952) dirty exists 2024-09-11T03:49:43.283 INFO:tasks.rados.rados.2.smithi175.stdout:3062: finishing write tid 5 to smithi17557870-25 2024-09-11T03:49:43.283 INFO:tasks.rados.rados.2.smithi175.stdout:3062: oid 25 updating version 0 to 898 2024-09-11T03:49:43.283 INFO:tasks.rados.rados.2.smithi175.stdout:3062: oid 25 version 898 is already newer than 897 2024-09-11T03:49:43.283 INFO:tasks.rados.rados.2.smithi175.stdout:3062: oid 25 updating version 898 to 899 2024-09-11T03:49:43.284 INFO:tasks.rados.rados.2.smithi175.stdout:3062: oid 25 updating version 899 to 900 2024-09-11T03:49:43.284 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 900 (ObjNum 1044 snap 340 seq_num 1044) dirty exists 2024-09-11T03:49:43.284 INFO:tasks.rados.rados.2.smithi175.stdout:3062: left oid 25 (ObjNum 1044 snap 340 seq_num 1044) 2024-09-11T03:49:43.284 INFO:tasks.rados.rados.2.smithi175.stdout:3068: expect (ObjNum 1030 snap 338 seq_num 1030) 2024-09-11T03:49:43.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:43.349 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:43.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:43.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:43.451 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:43.469 INFO:tasks.rados.rados.2.smithi175.stdout:3061: finishing copy_from to smithi17557870-10 2024-09-11T03:49:43.470 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 984 (ObjNum 1041 snap 340 seq_num 1041) dirty exists 2024-09-11T03:49:43.470 INFO:tasks.rados.rados.2.smithi175.stdout:3064: finishing copy_from racing read to smithi17557870-1 2024-09-11T03:49:43.470 INFO:tasks.rados.rados.2.smithi175.stdout:3064: finishing copy_from to smithi17557870-1 2024-09-11T03:49:43.470 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 1006 (ObjNum 973 snap 313 seq_num 973) dirty exists 2024-09-11T03:49:43.470 INFO:tasks.rados.rados.2.smithi175.stdout:3061: done (8 left) 2024-09-11T03:49:43.470 INFO:tasks.rados.rados.2.smithi175.stdout:3062: done (7 left) 2024-09-11T03:49:43.470 INFO:tasks.rados.rados.2.smithi175.stdout:3063: done (6 left) 2024-09-11T03:49:43.471 INFO:tasks.rados.rados.2.smithi175.stdout:3064: done (5 left) 2024-09-11T03:49:43.471 INFO:tasks.rados.rados.2.smithi175.stdout:3065: done (4 left) 2024-09-11T03:49:43.471 INFO:tasks.rados.rados.2.smithi175.stdout:3066: done (3 left) 2024-09-11T03:49:43.471 INFO:tasks.rados.rados.2.smithi175.stdout:3067: done (2 left) 2024-09-11T03:49:43.471 INFO:tasks.rados.rados.2.smithi175.stdout:3068: done (1 left) 2024-09-11T03:49:43.471 INFO:tasks.rados.rados.2.smithi175.stdout:3069: done (0 left) 2024-09-11T03:49:43.471 INFO:tasks.rados.rados.2.smithi175.stdout:3070: copy_from oid 21 from oid 38 current snap is 340 2024-09-11T03:49:43.471 INFO:tasks.rados.rados.2.smithi175.stdout:3071: copy_from oid 1 from oid 37 current snap is 340 2024-09-11T03:49:43.471 INFO:tasks.rados.rados.2.smithi175.stdout:3072: copy_from oid 5 from oid 8 current snap is 340 2024-09-11T03:49:43.471 INFO:tasks.rados.rados.2.smithi175.stdout:3073: delete oid 16 current snap is 340 2024-09-11T03:49:43.472 INFO:tasks.rados.rados.2.smithi175.stdout:3071: finishing copy_from to smithi17557870-1 2024-09-11T03:49:43.472 INFO:tasks.rados.rados.2.smithi175.stdout:3071: got expected ENOENT (src dne) 2024-09-11T03:49:43.472 INFO:tasks.rados.rados.2.smithi175.stdout:3071: finishing copy_from racing read to smithi17557870-1 2024-09-11T03:49:43.472 INFO:tasks.rados.rados.2.smithi175.stdout:3071: done (3 left) 2024-09-11T03:49:43.472 INFO:tasks.rados.rados.2.smithi175.stdout:3073: done (2 left) 2024-09-11T03:49:43.472 INFO:tasks.rados.rados.2.smithi175.stdout:3074: write oid 26 current snap is 340 2024-09-11T03:49:43.473 INFO:tasks.rados.rados.2.smithi175.stdout:3074: seq_num 1047 ranges {796929=726064,2184647=659366,3302344=294585} 2024-09-11T03:49:43.473 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:43.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:43.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:43.480 INFO:tasks.rados.rados.2.smithi175.stdout:3074: writing smithi17557870-26 from 796929 to 1522993 tid 1 2024-09-11T03:49:43.487 INFO:tasks.rados.rados.2.smithi175.stdout:3074: writing smithi17557870-26 from 2184647 to 2844013 tid 2 2024-09-11T03:49:43.492 INFO:tasks.rados.rados.2.smithi175.stdout:3074: writing smithi17557870-26 from 3302344 to 3596929 tid 3 2024-09-11T03:49:43.493 INFO:tasks.rados.rados.2.smithi175.stdout:3075: rollback oid 16 current snap is 340 2024-09-11T03:49:43.493 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 16 to 339 2024-09-11T03:49:43.493 INFO:tasks.rados.rados.2.smithi175.stdout:3072: finishing copy_from racing read to smithi17557870-5 2024-09-11T03:49:43.493 INFO:tasks.rados.rados.2.smithi175.stdout:3070: finishing copy_from racing read to smithi17557870-21 2024-09-11T03:49:43.493 INFO:tasks.rados.rados.2.smithi175.stdout:3072: finishing copy_from to smithi17557870-5 2024-09-11T03:49:43.493 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 1027 (ObjNum 1043 snap 340 seq_num 1043) dirty exists 2024-09-11T03:49:43.493 INFO:tasks.rados.rados.2.smithi175.stdout:3074: finishing write tid 1 to smithi17557870-26 2024-09-11T03:49:43.493 INFO:tasks.rados.rados.2.smithi175.stdout:3074: finishing write tid 2 to smithi17557870-26 2024-09-11T03:49:43.493 INFO:tasks.rados.rados.2.smithi175.stdout:3072: done (3 left) 2024-09-11T03:49:43.493 INFO:tasks.rados.rados.2.smithi175.stdout:3076: rollback oid 13 current snap is 340 2024-09-11T03:49:43.493 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 13 to 322 2024-09-11T03:49:43.493 INFO:tasks.rados.rados.2.smithi175.stdout:3077: delete oid 37 current snap is 340 2024-09-11T03:49:43.494 INFO:tasks.rados.rados.2.smithi175.stdout:3070: finishing copy_from to smithi17557870-21 2024-09-11T03:49:43.494 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 975 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:49:43.494 INFO:tasks.rados.rados.2.smithi175.stdout:3075: finishing rollback tid 1 to smithi17557870-16 2024-09-11T03:49:43.495 INFO:tasks.rados.rados.2.smithi175.stdout:3075: finishing rollback tid 2 to smithi17557870-16 2024-09-11T03:49:43.495 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 978 (ObjNum 1024 snap 337 seq_num 1024) dirty exists 2024-09-11T03:49:43.495 INFO:tasks.rados.rados.2.smithi175.stdout:3076: finishing rollback tid 1 to smithi17557870-13 2024-09-11T03:49:43.495 INFO:tasks.rados.rados.2.smithi175.stdout:3076: finishing rollback tid 2 to smithi17557870-13 2024-09-11T03:49:43.495 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 1188 (ObjNum 907 snap 287 seq_num 907) dirty exists 2024-09-11T03:49:43.497 INFO:tasks.rados.rados.2.smithi175.stdout:3070: done (4 left) 2024-09-11T03:49:43.497 INFO:tasks.rados.rados.2.smithi175.stdout:3075: done (3 left) 2024-09-11T03:49:43.497 INFO:tasks.rados.rados.2.smithi175.stdout:3076: done (2 left) 2024-09-11T03:49:43.497 INFO:tasks.rados.rados.2.smithi175.stdout:3077: done (1 left) 2024-09-11T03:49:43.497 INFO:tasks.rados.rados.2.smithi175.stdout:3078: rollback oid 12 current snap is 340 2024-09-11T03:49:43.497 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 12 to 321 2024-09-11T03:49:43.497 INFO:tasks.rados.rados.2.smithi175.stdout:3079: rollback oid 38 current snap is 340 2024-09-11T03:49:43.498 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 38 to 296 2024-09-11T03:49:43.498 INFO:tasks.rados.rados.2.smithi175.stdout:3080: read oid 33 snap -1 2024-09-11T03:49:43.498 INFO:tasks.rados.rados.2.smithi175.stdout:3080: expect (ObjNum 1037 snap 340 seq_num 1037) 2024-09-11T03:49:43.498 INFO:tasks.rados.rados.2.smithi175.stdout:3081: read oid 43 snap -1 2024-09-11T03:49:43.498 INFO:tasks.rados.rados.2.smithi175.stdout:3081: expect (ObjNum 1008 snap 326 seq_num 1008) 2024-09-11T03:49:43.498 INFO:tasks.rados.rados.2.smithi175.stdout:3082: rollback oid 50 current snap is 340 2024-09-11T03:49:43.498 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 50 to 334 2024-09-11T03:49:43.498 INFO:tasks.rados.rados.2.smithi175.stdout:3074: finishing write tid 3 to smithi17557870-26 2024-09-11T03:49:43.498 INFO:tasks.rados.rados.2.smithi175.stdout:3083: copy_from oid 34 from oid 21 current snap is 340 2024-09-11T03:49:43.498 INFO:tasks.rados.rados.2.smithi175.stdout:3084: read oid 25 snap -1 2024-09-11T03:49:43.499 INFO:tasks.rados.rados.2.smithi175.stdout:3084: expect (ObjNum 1044 snap 340 seq_num 1044) 2024-09-11T03:49:43.499 INFO:tasks.rados.rados.2.smithi175.stdout:3085: read oid 6 snap 305 2024-09-11T03:49:43.499 INFO:tasks.rados.rados.2.smithi175.stdout:3085: expect (ObjNum 857 snap 269 seq_num 857) 2024-09-11T03:49:43.499 INFO:tasks.rados.rados.2.smithi175.stdout:3086: rollback oid 13 current snap is 340 2024-09-11T03:49:43.499 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 13 to 305 2024-09-11T03:49:43.499 INFO:tasks.rados.rados.2.smithi175.stdout:3074: finishing write tid 4 to smithi17557870-26 2024-09-11T03:49:43.499 INFO:tasks.rados.rados.2.smithi175.stdout:3087: copy_from oid 20 from oid 16 current snap is 340 2024-09-11T03:49:43.499 INFO:tasks.rados.rados.2.smithi175.stdout:3088: delete oid 15 current snap is 340 2024-09-11T03:49:43.499 INFO:tasks.rados.rados.2.smithi175.stdout:3074: finishing write tid 5 to smithi17557870-26 2024-09-11T03:49:43.499 INFO:tasks.rados.rados.2.smithi175.stdout:3074: oid 26 updating version 0 to 964 2024-09-11T03:49:43.500 INFO:tasks.rados.rados.2.smithi175.stdout:3074: oid 26 updating version 964 to 967 2024-09-11T03:49:43.500 INFO:tasks.rados.rados.2.smithi175.stdout:3074: oid 26 version 967 is already newer than 965 2024-09-11T03:49:43.500 INFO:tasks.rados.rados.2.smithi175.stdout:3074: oid 26 version 967 is already newer than 966 2024-09-11T03:49:43.500 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 967 (ObjNum 1047 snap 340 seq_num 1047) dirty exists 2024-09-11T03:49:43.500 INFO:tasks.rados.rados.2.smithi175.stdout:3074: left oid 26 (ObjNum 1047 snap 340 seq_num 1047) 2024-09-11T03:49:43.501 INFO:tasks.rados.rados.2.smithi175.stdout:3074: done (11 left) 2024-09-11T03:49:43.501 INFO:tasks.rados.rados.2.smithi175.stdout:3088: done (10 left) 2024-09-11T03:49:43.501 INFO:tasks.rados.rados.2.smithi175.stdout:3089: read oid 3 snap -1 2024-09-11T03:49:43.501 INFO:tasks.rados.rados.2.smithi175.stdout:3089: expect deleted 2024-09-11T03:49:43.502 INFO:tasks.rados.rados.2.smithi175.stdout:3090: snap_create 2024-09-11T03:49:43.502 INFO:tasks.rados.rados.2.smithi175.stdout:3078: finishing rollback tid 0 to smithi17557870-12 2024-09-11T03:49:43.502 INFO:tasks.rados.rados.2.smithi175.stdout:3078: finishing rollback tid 1 to smithi17557870-12 2024-09-11T03:49:43.502 INFO:tasks.rados.rados.2.smithi175.stdout:3078: finishing rollback tid 2 to smithi17557870-12 2024-09-11T03:49:43.502 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 1191 (ObjNum 962 snap 308 seq_num 962) dirty exists 2024-09-11T03:49:43.502 INFO:tasks.rados.rados.2.smithi175.stdout:3079: finishing rollback tid 0 to smithi17557870-38 2024-09-11T03:49:43.502 INFO:tasks.rados.rados.2.smithi175.stdout:3079: finishing rollback tid 1 to smithi17557870-38 2024-09-11T03:49:43.502 INFO:tasks.rados.rados.2.smithi175.stdout:3081: expect (ObjNum 1008 snap 326 seq_num 1008) 2024-09-11T03:49:43.575 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:43.597 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:43.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:43.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:43.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:43.721 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:43.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:43.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:43.734 INFO:tasks.rados.rados.2.smithi175.stdout:3079: finishing rollback tid 2 to smithi17557870-38 2024-09-11T03:49:43.734 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 1194 (ObjNum 795 snap 254 seq_num 795) dirty exists 2024-09-11T03:49:43.734 INFO:tasks.rados.rados.2.smithi175.stdout:3086: finishing rollback tid 0 to smithi17557870-13 2024-09-11T03:49:43.734 INFO:tasks.rados.rados.2.smithi175.stdout:3086: finishing rollback tid 1 to smithi17557870-13 2024-09-11T03:49:43.734 INFO:tasks.rados.rados.2.smithi175.stdout:3086: finishing rollback tid 2 to smithi17557870-13 2024-09-11T03:49:43.734 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 1197 (ObjNum 907 snap 287 seq_num 907) dirty exists 2024-09-11T03:49:43.734 INFO:tasks.rados.rados.2.smithi175.stdout:3087: finishing copy_from racing read to smithi17557870-20 2024-09-11T03:49:43.734 INFO:tasks.rados.rados.2.smithi175.stdout:3087: finishing copy_from to smithi17557870-20 2024-09-11T03:49:43.734 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 978 (ObjNum 1024 snap 337 seq_num 1024) dirty exists 2024-09-11T03:49:43.734 INFO:tasks.rados.rados.2.smithi175.stdout:3080: expect (ObjNum 1037 snap 340 seq_num 1037) 2024-09-11T03:49:43.798 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:49:43.799 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:49:43.822 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:43.843 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:43.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:43.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:43.926 INFO:tasks.rados.rados.2.smithi175.stdout:3082: finishing rollback tid 0 to smithi17557870-50 2024-09-11T03:49:43.926 INFO:tasks.rados.rados.2.smithi175.stdout:3082: finishing rollback tid 1 to smithi17557870-50 2024-09-11T03:49:43.926 INFO:tasks.rados.rados.2.smithi175.stdout:3082: finishing rollback tid 2 to smithi17557870-50 2024-09-11T03:49:43.926 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 1212 (ObjNum 1013 snap 331 seq_num 1013) dirty exists 2024-09-11T03:49:43.926 INFO:tasks.rados.rados.2.smithi175.stdout:3083: finishing copy_from racing read to smithi17557870-34 2024-09-11T03:49:43.926 INFO:tasks.rados.rados.2.smithi175.stdout:3083: finishing copy_from to smithi17557870-34 2024-09-11T03:49:43.926 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 975 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:49:43.926 INFO:tasks.rados.rados.2.smithi175.stdout:3085: expect (ObjNum 857 snap 269 seq_num 857) 2024-09-11T03:49:43.945 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:43.966 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:43.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:43.967 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:44.059 INFO:tasks.rados.rados.2.smithi175.stdout:3084: expect (ObjNum 1044 snap 340 seq_num 1044) 2024-09-11T03:49:44.062 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:44.067 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:44.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:44.090 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:44.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:44.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:44.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:44 smithi084 ceph-mon[27254]: pgmap v1302: 77 pgs: 77 active+clean; 379 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.4 MiB/s wr, 24 op/s 2024-09-11T03:49:44.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:44 smithi084 ceph-mon[27254]: osdmap e765: 12 total, 12 up, 12 in 2024-09-11T03:49:44.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:44 smithi084 ceph-mon[27254]: from='client.17004 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:49:44.192 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:44.213 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:44.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:44.263 INFO:tasks.rados.rados.2.smithi175.stdout:3078: done (11 left) 2024-09-11T03:49:44.263 INFO:tasks.rados.rados.2.smithi175.stdout:3079: done (10 left) 2024-09-11T03:49:44.264 INFO:tasks.rados.rados.2.smithi175.stdout:3080: done (9 left) 2024-09-11T03:49:44.264 INFO:tasks.rados.rados.2.smithi175.stdout:3081: done (8 left) 2024-09-11T03:49:44.264 INFO:tasks.rados.rados.2.smithi175.stdout:3082: done (7 left) 2024-09-11T03:49:44.264 INFO:tasks.rados.rados.2.smithi175.stdout:3083: done (6 left) 2024-09-11T03:49:44.264 INFO:tasks.rados.rados.2.smithi175.stdout:3084: done (5 left) 2024-09-11T03:49:44.264 INFO:tasks.rados.rados.2.smithi175.stdout:3085: done (4 left) 2024-09-11T03:49:44.264 INFO:tasks.rados.rados.2.smithi175.stdout:3086: done (3 left) 2024-09-11T03:49:44.264 INFO:tasks.rados.rados.2.smithi175.stdout:3087: done (2 left) 2024-09-11T03:49:44.264 INFO:tasks.rados.rados.2.smithi175.stdout:3089: done (1 left) 2024-09-11T03:49:44.264 INFO:tasks.rados.rados.2.smithi175.stdout:3090: done (0 left) 2024-09-11T03:49:44.264 INFO:tasks.rados.rados.2.smithi175.stdout:3091: read oid 21 snap -1 2024-09-11T03:49:44.264 INFO:tasks.rados.rados.2.smithi175.stdout:3091: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:49:44.264 INFO:tasks.rados.rados.2.smithi175.stdout:3092: snap_create 2024-09-11T03:49:44.278 INFO:tasks.rados.rados.2.smithi175.stdout:3091: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:49:44.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:44.339 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:44.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:44 smithi121 ceph-mon[30729]: pgmap v1302: 77 pgs: 77 active+clean; 379 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.4 MiB/s wr, 24 op/s 2024-09-11T03:49:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:44 smithi121 ceph-mon[30729]: osdmap e765: 12 total, 12 up, 12 in 2024-09-11T03:49:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:44 smithi121 ceph-mon[30729]: from='client.17004 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:49:44.440 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:44.461 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:44.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:44.462 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:44 smithi175 ceph-mon[30894]: pgmap v1302: 77 pgs: 77 active+clean; 379 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.4 MiB/s wr, 24 op/s 2024-09-11T03:49:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:44 smithi175 ceph-mon[30894]: osdmap e765: 12 total, 12 up, 12 in 2024-09-11T03:49:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:44 smithi175 ceph-mon[30894]: from='client.17004 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:49:44.493 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1303,"stamp":"2024-09-11T03:49:43.018744+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":397318771,"num_objects":190,"num_object_clones":138,"num_object_copies":570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":12,"num_read":1231,"num_read_kb":1285366,"num_write":865,"num_write_kb":358502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":30615821,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29183,"ondisk_log_size":29183,"up":231,"acting":231,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":231,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":1351696,"kb_used_data":831688,"kb_used_omap":18,"kb_used_meta":519725,"kb_avail":1123590128,"statfs":{"total":1151940427776,"available":1150556291072,"internally_reserved":0,"allocated":851648512,"data_stored":1068742869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":532198715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18685634,"num_objects":-12,"num_object_clones":-12,"num_object_copies":-36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12,"num_whiteouts":-1,"num_read":59,"num_read_kb":61116,"num_write":40,"num_write_kb":17823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.806583"},"pg_stats":[{"pgid":"2.2d","version":"764'1021","reported_seq":4411,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.037861+0000","last_change":"2024-09-11T03:49:42.038043+0000","last_active":"2024-09-11T03:49:42.037861+0000","last_peered":"2024-09-11T03:49:42.037861+0000","last_clean":"2024-09-11T03:49:42.037861+0000","last_became_active":"2024-09-11T03:49:29.163201+0000","last_became_peered":"2024-09-11T03:49:29.163201+0000","last_unstale":"2024-09-11T03:49:42.037861+0000","last_undegraded":"2024-09-11T03:49:42.037861+0000","last_fullsized":"2024-09-11T03:49:42.037861+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'997","last_scrub_stamp":"2024-09-11T03:49:32.462332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.462332+0000","objects_scrubbed":13,"log_size":1021,"log_dups_size":0,"ondisk_log_size":1021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.051281+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0099888119999999997,"stat_sum":{"num_bytes":28080482,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":9,"num_read_kb":10392,"num_write":1,"num_write_kb":2479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"119","length":"1"},{"start":"142","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"2"}]},{"pgid":"2.2c","version":"763'807","reported_seq":4181,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.235088+0000","last_change":"2024-09-11T03:49:41.235663+0000","last_active":"2024-09-11T03:49:41.235088+0000","last_peered":"2024-09-11T03:49:41.235088+0000","last_clean":"2024-09-11T03:49:41.235088+0000","last_became_active":"2024-09-11T03:49:29.154872+0000","last_became_peered":"2024-09-11T03:49:29.154872+0000","last_unstale":"2024-09-11T03:49:41.235088+0000","last_undegraded":"2024-09-11T03:49:41.235088+0000","last_fullsized":"2024-09-11T03:49:41.235088+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"753'800","last_scrub_stamp":"2024-09-11T03:49:32.185401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.185401+0000","objects_scrubbed":4,"log_size":807,"log_dups_size":0,"ondisk_log_size":807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:52.303047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055529129999999996,"stat_sum":{"num_bytes":12292728,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3],"acting":[8,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"2.2b","version":"568'646","reported_seq":3138,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-09-11T03:49:38.217400+0000","last_change":"2024-09-11T03:49:38.218102+0000","last_active":"2024-09-11T03:49:38.217400+0000","last_peered":"2024-09-11T03:49:38.217400+0000","last_clean":"2024-09-11T03:49:38.217400+0000","last_became_active":"2024-09-11T03:49:29.157735+0000","last_became_peered":"2024-09-11T03:49:29.157735+0000","last_unstale":"2024-09-11T03:49:38.217400+0000","last_undegraded":"2024-09-11T03:49:38.217400+0000","last_fullsized":"2024-09-11T03:49:38.217400+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:49:32.876969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.876969+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:34.820084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065993249999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"14f","length":"1"},{"start":"152","length":"1"}]},{"pgid":"2.2a","version":"727'348","reported_seq":3642,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.227369+0000","last_change":"2024-09-11T03:49:41.227757+0000","last_active":"2024-09-11T03:49:41.227369+0000","last_peered":"2024-09-11T03:49:41.227369+0000","last_clean":"2024-09-11T03:49:41.227369+0000","last_became_active":"2024-09-11T03:49:29.160595+0000","last_became_peered":"2024-09-11T03:49:29.160595+0000","last_unstale":"2024-09-11T03:49:41.227369+0000","last_undegraded":"2024-09-11T03:49:41.227369+0000","last_fullsized":"2024-09-11T03:49:41.227369+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:49:34.219244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.219244+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:00.483958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000663471,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,5],"acting":[7,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"2.29","version":"764'1181","reported_seq":4679,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.033951+0000","last_change":"2024-09-11T03:49:42.034653+0000","last_active":"2024-09-11T03:49:42.033951+0000","last_peered":"2024-09-11T03:49:42.033951+0000","last_clean":"2024-09-11T03:49:42.033951+0000","last_became_active":"2024-09-11T03:49:29.157194+0000","last_became_peered":"2024-09-11T03:49:29.157194+0000","last_unstale":"2024-09-11T03:49:42.033951+0000","last_undegraded":"2024-09-11T03:49:42.033951+0000","last_fullsized":"2024-09-11T03:49:42.033951+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'1167","last_scrub_stamp":"2024-09-11T03:49:33.506786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.506786+0000","objects_scrubbed":11,"log_size":1181,"log_dups_size":0,"ondisk_log_size":1181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:01.087470+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0091973980000000007,"stat_sum":{"num_bytes":19640813,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":8,"num_read_kb":8514,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8],"acting":[6,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"119","length":"1"},{"start":"142","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"2"}]},{"pgid":"2.28","version":"762'388","reported_seq":3780,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.230926+0000","last_change":"2024-09-11T03:49:41.231194+0000","last_active":"2024-09-11T03:49:41.230926+0000","last_peered":"2024-09-11T03:49:41.230926+0000","last_clean":"2024-09-11T03:49:41.230926+0000","last_became_active":"2024-09-11T03:49:29.154884+0000","last_became_peered":"2024-09-11T03:49:29.154884+0000","last_unstale":"2024-09-11T03:49:41.230926+0000","last_undegraded":"2024-09-11T03:49:41.230926+0000","last_fullsized":"2024-09-11T03:49:41.230926+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'382","last_scrub_stamp":"2024-09-11T03:49:32.335285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.335285+0000","objects_scrubbed":4,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:53.032116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000760879,"stat_sum":{"num_bytes":8486578,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"2.27","version":"763'1073","reported_seq":4577,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.231647+0000","last_change":"2024-09-11T03:49:41.231928+0000","last_active":"2024-09-11T03:49:41.231647+0000","last_peered":"2024-09-11T03:49:41.231647+0000","last_clean":"2024-09-11T03:49:41.231647+0000","last_became_active":"2024-09-11T03:49:29.159799+0000","last_became_peered":"2024-09-11T03:49:29.159799+0000","last_unstale":"2024-09-11T03:49:41.231647+0000","last_undegraded":"2024-09-11T03:49:41.231647+0000","last_fullsized":"2024-09-11T03:49:41.231647+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"757'1065","last_scrub_stamp":"2024-09-11T03:49:36.196385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.196385+0000","objects_scrubbed":4,"log_size":1073,"log_dups_size":0,"ondisk_log_size":1073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:38.815485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0041107849999999996,"stat_sum":{"num_bytes":9109955,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":9,"num_read_kb":13447,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,9],"acting":[7,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"2.26","version":"269'171","reported_seq":3527,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.228405+0000","last_change":"2024-09-11T03:49:41.229066+0000","last_active":"2024-09-11T03:49:41.228405+0000","last_peered":"2024-09-11T03:49:41.228405+0000","last_clean":"2024-09-11T03:49:41.228405+0000","last_became_active":"2024-09-11T03:49:29.158577+0000","last_became_peered":"2024-09-11T03:49:29.158577+0000","last_unstale":"2024-09-11T03:49:41.228405+0000","last_undegraded":"2024-09-11T03:49:41.228405+0000","last_fullsized":"2024-09-11T03:49:41.228405+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:49:33.199779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.199779+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:52.894681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001245276,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,11],"acting":[7,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"2.25","version":"763'1169","reported_seq":4767,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.239529+0000","last_change":"2024-09-11T03:49:41.239796+0000","last_active":"2024-09-11T03:49:41.239529+0000","last_peered":"2024-09-11T03:49:41.239529+0000","last_clean":"2024-09-11T03:49:41.239529+0000","last_became_active":"2024-09-11T03:49:29.159733+0000","last_became_peered":"2024-09-11T03:49:29.159733+0000","last_unstale":"2024-09-11T03:49:41.239529+0000","last_undegraded":"2024-09-11T03:49:41.239529+0000","last_fullsized":"2024-09-11T03:49:41.239529+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"755'1138","last_scrub_stamp":"2024-09-11T03:49:34.198067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.198067+0000","objects_scrubbed":16,"log_size":1169,"log_dups_size":0,"ondisk_log_size":1169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:36.928621+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0092298039999999994,"stat_sum":{"num_bytes":28744860,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":3,"num_read_kb":0,"num_write":2,"num_write_kb":2643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,6],"acting":[8,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"2.24","version":"764'626","reported_seq":4043,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.036006+0000","last_change":"2024-09-11T03:49:42.036318+0000","last_active":"2024-09-11T03:49:42.036006+0000","last_peered":"2024-09-11T03:49:42.036006+0000","last_clean":"2024-09-11T03:49:42.036006+0000","last_became_active":"2024-09-11T03:49:29.155740+0000","last_became_peered":"2024-09-11T03:49:29.155740+0000","last_unstale":"2024-09-11T03:49:42.036006+0000","last_undegraded":"2024-09-11T03:49:42.036006+0000","last_fullsized":"2024-09-11T03:49:42.036006+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'612","last_scrub_stamp":"2024-09-11T03:49:34.554335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.554335+0000","objects_scrubbed":4,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:45.008935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0098967559999999996,"stat_sum":{"num_bytes":7972882,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,7],"acting":[6,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"119","length":"1"},{"start":"142","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"2"}]},{"pgid":"2.23","version":"763'448","reported_seq":3814,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.235027+0000","last_change":"2024-09-11T03:49:41.235291+0000","last_active":"2024-09-11T03:49:41.235027+0000","last_peered":"2024-09-11T03:49:41.235027+0000","last_clean":"2024-09-11T03:49:41.235027+0000","last_became_active":"2024-09-11T03:46:32.970100+0000","last_became_peered":"2024-09-11T03:46:32.970100+0000","last_unstale":"2024-09-11T03:49:41.235027+0000","last_undegraded":"2024-09-11T03:49:41.235027+0000","last_fullsized":"2024-09-11T03:49:41.235027+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"711'414","last_scrub_stamp":"2024-09-11T03:48:57.105274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:57.105274+0000","objects_scrubbed":2,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:16.897444+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0046841440000000003,"stat_sum":{"num_bytes":9866336,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":31430,"num_write":23,"num_write_kb":10047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"2.22","version":"531'216","reported_seq":3582,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.028537+0000","last_change":"2024-09-11T03:49:42.028836+0000","last_active":"2024-09-11T03:49:42.028537+0000","last_peered":"2024-09-11T03:49:42.028537+0000","last_clean":"2024-09-11T03:49:42.028537+0000","last_became_active":"2024-09-11T03:46:32.970633+0000","last_became_peered":"2024-09-11T03:46:32.970633+0000","last_unstale":"2024-09-11T03:49:42.028537+0000","last_undegraded":"2024-09-11T03:49:42.028537+0000","last_fullsized":"2024-09-11T03:49:42.028537+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:49:18.406097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:18.406097+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:18.597049+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0041958689999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"119","length":"1"},{"start":"142","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"2"}]},{"pgid":"2.21","version":"764'310","reported_seq":3767,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.033648+0000","last_change":"2024-09-11T03:49:42.033828+0000","last_active":"2024-09-11T03:49:42.033648+0000","last_peered":"2024-09-11T03:49:42.033648+0000","last_clean":"2024-09-11T03:49:42.033648+0000","last_became_active":"2024-09-11T03:46:36.024928+0000","last_became_peered":"2024-09-11T03:46:36.024928+0000","last_unstale":"2024-09-11T03:49:42.033648+0000","last_undegraded":"2024-09-11T03:49:42.033648+0000","last_fullsized":"2024-09-11T03:49:42.033648+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":6,"last_scrub":"733'289","last_scrub_stamp":"2024-09-11T03:49:18.917801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:18.917801+0000","objects_scrubbed":3,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:38.402484+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002617182,"stat_sum":{"num_bytes":7337742,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":11800,"num_write":25,"num_write_kb":9997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,8],"acting":[11,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"119","length":"1"},{"start":"142","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"2"}]},{"pgid":"2.20","version":"762'253","reported_seq":3771,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.236593+0000","last_change":"2024-09-11T03:49:41.228080+0000","last_active":"2024-09-11T03:49:41.236593+0000","last_peered":"2024-09-11T03:49:41.236593+0000","last_clean":"2024-09-11T03:49:41.236593+0000","last_became_active":"2024-09-11T03:46:32.974375+0000","last_became_peered":"2024-09-11T03:46:32.974375+0000","last_unstale":"2024-09-11T03:49:41.236593+0000","last_undegraded":"2024-09-11T03:49:41.236593+0000","last_fullsized":"2024-09-11T03:49:41.236593+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"718'223","last_scrub_stamp":"2024-09-11T03:49:00.376119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.376119+0000","objects_scrubbed":2,"log_size":253,"log_dups_size":0,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:03.376069+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011804529999999999,"stat_sum":{"num_bytes":7462671,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":67,"num_read_kb":109254,"num_write":10,"num_write_kb":5367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"2.1c","version":"762'764","reported_seq":4129,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.027333+0000","last_change":"2024-09-11T03:49:42.027748+0000","last_active":"2024-09-11T03:49:42.027333+0000","last_peered":"2024-09-11T03:49:42.027333+0000","last_clean":"2024-09-11T03:49:42.027333+0000","last_became_active":"2024-09-11T03:46:32.971036+0000","last_became_peered":"2024-09-11T03:46:32.971036+0000","last_unstale":"2024-09-11T03:49:42.027333+0000","last_undegraded":"2024-09-11T03:49:42.027333+0000","last_fullsized":"2024-09-11T03:49:42.027333+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"754'760","last_scrub_stamp":"2024-09-11T03:49:33.579794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.579794+0000","objects_scrubbed":5,"log_size":764,"log_dups_size":0,"ondisk_log_size":764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:47.885319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00068615400000000002,"stat_sum":{"num_bytes":4835616,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":34,"num_read_kb":35078,"num_write":30,"num_write_kb":12013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"119","length":"1"},{"start":"142","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"2"}]},{"pgid":"1.c","version":"20'92","reported_seq":577,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-09-11T03:49:40.227269+0000","last_change":"2024-09-11T03:47:26.195120+0000","last_active":"2024-09-11T03:49:40.227269+0000","last_peered":"2024-09-11T03:49:40.227269+0000","last_clean":"2024-09-11T03:49:40.227269+0000","last_became_active":"2024-09-11T03:46:15.974786+0000","last_became_peered":"2024-09-11T03:46:15.974786+0000","last_unstale":"2024-09-11T03:49:40.227269+0000","last_undegraded":"2024-09-11T03:49:40.227269+0000","last_fullsized":"2024-09-11T03:49:40.227269+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:26.195070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:26.195070+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:45.026219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.f","version":"557'465","reported_seq":3849,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.230536+0000","last_change":"2024-09-11T03:49:41.230924+0000","last_active":"2024-09-11T03:49:41.230536+0000","last_peered":"2024-09-11T03:49:41.230536+0000","last_clean":"2024-09-11T03:49:41.230536+0000","last_became_active":"2024-09-11T03:46:27.971779+0000","last_became_peered":"2024-09-11T03:46:27.971779+0000","last_unstale":"2024-09-11T03:49:41.230536+0000","last_undegraded":"2024-09-11T03:49:41.230536+0000","last_fullsized":"2024-09-11T03:49:41.230536+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:48:59.211781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:59.211781+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:18.489926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00115665,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"2.e","version":"760'985","reported_seq":4303,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-09-11T03:49:38.216329+0000","last_change":"2024-09-11T03:49:38.217041+0000","last_active":"2024-09-11T03:49:38.216329+0000","last_peered":"2024-09-11T03:49:38.216329+0000","last_clean":"2024-09-11T03:49:38.216329+0000","last_became_active":"2024-09-11T03:46:36.024843+0000","last_became_peered":"2024-09-11T03:46:36.024843+0000","last_unstale":"2024-09-11T03:49:38.216329+0000","last_undegraded":"2024-09-11T03:49:38.216329+0000","last_fullsized":"2024-09-11T03:49:38.216329+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"736'963","last_scrub_stamp":"2024-09-11T03:49:17.918033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:17.918033+0000","objects_scrubbed":8,"log_size":985,"log_dups_size":0,"ondisk_log_size":985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:36.697324+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0059298579999999997,"stat_sum":{"num_bytes":19219846,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":66,"num_read_kb":66223,"num_write":48,"num_write_kb":30257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"14f","length":"1"},{"start":"152","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":1156,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.228007+0000","last_change":"2024-09-11T03:49:20.176154+0000","last_active":"2024-09-11T03:49:41.228007+0000","last_peered":"2024-09-11T03:49:41.228007+0000","last_clean":"2024-09-11T03:49:41.228007+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:49:41.228007+0000","last_undegraded":"2024-09-11T03:49:41.228007+0000","last_fullsized":"2024-09-11T03:49:41.228007+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:20.176098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.176098+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:41.602355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"762'991","reported_seq":4419,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.238878+0000","last_change":"2024-09-11T03:49:41.239219+0000","last_active":"2024-09-11T03:49:41.238878+0000","last_peered":"2024-09-11T03:49:41.238878+0000","last_clean":"2024-09-11T03:49:41.238878+0000","last_became_active":"2024-09-11T03:49:24.116744+0000","last_became_peered":"2024-09-11T03:49:24.116744+0000","last_unstale":"2024-09-11T03:49:41.238878+0000","last_undegraded":"2024-09-11T03:49:41.238878+0000","last_fullsized":"2024-09-11T03:49:41.238878+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'987","last_scrub_stamp":"2024-09-11T03:49:32.653263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.653263+0000","objects_scrubbed":3,"log_size":991,"log_dups_size":0,"ondisk_log_size":991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:59.232687+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00070436399999999997,"stat_sum":{"num_bytes":5546706,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":871,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.227717+0000","last_change":"2024-09-11T03:49:22.192021+0000","last_active":"2024-09-11T03:49:41.227717+0000","last_peered":"2024-09-11T03:49:41.227717+0000","last_clean":"2024-09-11T03:49:41.227717+0000","last_became_active":"2024-09-11T03:44:43.940022+0000","last_became_peered":"2024-09-11T03:44:43.940022+0000","last_unstale":"2024-09-11T03:49:41.227717+0000","last_undegraded":"2024-09-11T03:49:41.227717+0000","last_fullsized":"2024-09-11T03:49:41.227717+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:22.191965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:22.191965+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:33.940974+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"764'820","reported_seq":4218,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.035456+0000","last_change":"2024-09-11T03:49:42.035728+0000","last_active":"2024-09-11T03:49:42.035456+0000","last_peered":"2024-09-11T03:49:42.035456+0000","last_clean":"2024-09-11T03:49:42.035456+0000","last_became_active":"2024-09-11T03:49:24.115405+0000","last_became_peered":"2024-09-11T03:49:24.115405+0000","last_unstale":"2024-09-11T03:49:42.035456+0000","last_undegraded":"2024-09-11T03:49:42.035456+0000","last_fullsized":"2024-09-11T03:49:42.035456+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'808","last_scrub_stamp":"2024-09-11T03:49:33.953477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.953477+0000","objects_scrubbed":7,"log_size":820,"log_dups_size":0,"ondisk_log_size":820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:55.750477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0038961899999999999,"stat_sum":{"num_bytes":13462365,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"119","length":"1"},{"start":"142","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"2"}]},{"pgid":"1.f","version":"20'92","reported_seq":438,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.026660+0000","last_change":"2024-09-11T03:48:57.363099+0000","last_active":"2024-09-11T03:49:42.026660+0000","last_peered":"2024-09-11T03:49:42.026660+0000","last_clean":"2024-09-11T03:49:42.026660+0000","last_became_active":"2024-09-11T03:46:15.973796+0000","last_became_peered":"2024-09-11T03:46:15.973796+0000","last_unstale":"2024-09-11T03:49:42.026660+0000","last_undegraded":"2024-09-11T03:49:42.026660+0000","last_fullsized":"2024-09-11T03:49:42.026660+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:57.363027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:57.363027+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:08.181547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":1561,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.227621+0000","last_change":"2024-09-11T03:47:28.495899+0000","last_active":"2024-09-11T03:49:41.227621+0000","last_peered":"2024-09-11T03:49:41.227621+0000","last_clean":"2024-09-11T03:49:41.227621+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:49:41.227621+0000","last_undegraded":"2024-09-11T03:49:41.227621+0000","last_fullsized":"2024-09-11T03:49:41.227621+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:28.495839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:28.495839+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:45.331949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"763'488","reported_seq":3865,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.242070+0000","last_change":"2024-09-11T03:49:41.242313+0000","last_active":"2024-09-11T03:49:41.242070+0000","last_peered":"2024-09-11T03:49:41.242070+0000","last_clean":"2024-09-11T03:49:41.242070+0000","last_became_active":"2024-09-11T03:46:27.972329+0000","last_became_peered":"2024-09-11T03:46:27.972329+0000","last_unstale":"2024-09-11T03:49:41.242070+0000","last_undegraded":"2024-09-11T03:49:41.242070+0000","last_fullsized":"2024-09-11T03:49:41.242070+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'480","last_scrub_stamp":"2024-09-11T03:49:37.694768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.694768+0000","objects_scrubbed":5,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:54.922924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0037662609999999999,"stat_sum":{"num_bytes":8442631,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":3872,"num_write":26,"num_write_kb":5269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":1164,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.230576+0000","last_change":"2024-09-11T03:47:25.344219+0000","last_active":"2024-09-11T03:49:41.230576+0000","last_peered":"2024-09-11T03:49:41.230576+0000","last_clean":"2024-09-11T03:49:41.230576+0000","last_became_active":"2024-09-11T03:46:15.968960+0000","last_became_peered":"2024-09-11T03:46:15.968960+0000","last_unstale":"2024-09-11T03:49:41.230576+0000","last_undegraded":"2024-09-11T03:49:41.230576+0000","last_fullsized":"2024-09-11T03:49:41.230576+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:25.344170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:25.344170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"563'192","reported_seq":3640,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.227823+0000","last_change":"2024-09-11T03:49:41.229020+0000","last_active":"2024-09-11T03:49:41.227823+0000","last_peered":"2024-09-11T03:49:41.227823+0000","last_clean":"2024-09-11T03:49:41.227823+0000","last_became_active":"2024-09-11T03:46:27.974405+0000","last_became_peered":"2024-09-11T03:46:27.974405+0000","last_unstale":"2024-09-11T03:49:41.227823+0000","last_undegraded":"2024-09-11T03:49:41.227823+0000","last_fullsized":"2024-09-11T03:49:41.227823+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:49:15.233165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:15.233165+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:28.279174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010065,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":1123,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-09-11T03:49:40.227258+0000","last_change":"2024-09-11T03:49:17.252987+0000","last_active":"2024-09-11T03:49:40.227258+0000","last_peered":"2024-09-11T03:49:40.227258+0000","last_clean":"2024-09-11T03:49:40.227258+0000","last_became_active":"2024-09-11T03:44:25.934580+0000","last_became_peered":"2024-09-11T03:44:25.934580+0000","last_unstale":"2024-09-11T03:49:40.227258+0000","last_undegraded":"2024-09-11T03:49:40.227258+0000","last_fullsized":"2024-09-11T03:49:40.227258+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:17.252916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:17.252916+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:39.008724+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"567'235","reported_seq":3565,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-09-11T03:49:40.227388+0000","last_change":"2024-09-11T03:49:40.227675+0000","last_active":"2024-09-11T03:49:40.227388+0000","last_peered":"2024-09-11T03:49:40.227388+0000","last_clean":"2024-09-11T03:49:40.227388+0000","last_became_active":"2024-09-11T03:46:36.024925+0000","last_became_peered":"2024-09-11T03:46:36.024925+0000","last_unstale":"2024-09-11T03:49:40.227388+0000","last_undegraded":"2024-09-11T03:49:40.227388+0000","last_fullsized":"2024-09-11T03:49:40.227388+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:49:01.138426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:01.138426+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:10.996367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00049794499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14f","length":"1"},{"start":"152","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":732,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.027047+0000","last_change":"2024-09-11T03:48:32.199588+0000","last_active":"2024-09-11T03:49:42.027047+0000","last_peered":"2024-09-11T03:49:42.027047+0000","last_clean":"2024-09-11T03:49:42.027047+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:49:42.027047+0000","last_undegraded":"2024-09-11T03:49:42.027047+0000","last_fullsized":"2024-09-11T03:49:42.027047+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:32.199524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:32.199524+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:00.036523+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"762'278","reported_seq":3729,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.239294+0000","last_change":"2024-09-11T03:49:41.239579+0000","last_active":"2024-09-11T03:49:41.239294+0000","last_peered":"2024-09-11T03:49:41.239294+0000","last_clean":"2024-09-11T03:49:41.239294+0000","last_became_active":"2024-09-11T03:46:27.971649+0000","last_became_peered":"2024-09-11T03:46:27.971649+0000","last_unstale":"2024-09-11T03:49:41.239294+0000","last_undegraded":"2024-09-11T03:49:41.239294+0000","last_fullsized":"2024-09-11T03:49:41.239294+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"718'254","last_scrub_stamp":"2024-09-11T03:49:15.510323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:15.510323+0000","objects_scrubbed":4,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:43.220320+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001011348,"stat_sum":{"num_bytes":4639161,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":22,"num_read_kb":13034,"num_write":19,"num_write_kb":11745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":430,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.026148+0000","last_change":"2024-09-11T03:49:37.637065+0000","last_active":"2024-09-11T03:49:42.026148+0000","last_peered":"2024-09-11T03:49:42.026148+0000","last_clean":"2024-09-11T03:49:42.026148+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:49:42.026148+0000","last_undegraded":"2024-09-11T03:49:42.026148+0000","last_fullsized":"2024-09-11T03:49:42.026148+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:37.637018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.637018+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:05.221389+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"763'628","reported_seq":4059,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.265985+0000","last_change":"2024-09-11T03:49:41.266189+0000","last_active":"2024-09-11T03:49:41.265985+0000","last_peered":"2024-09-11T03:49:41.265985+0000","last_clean":"2024-09-11T03:49:41.265985+0000","last_became_active":"2024-09-11T03:49:24.114330+0000","last_became_peered":"2024-09-11T03:49:24.114330+0000","last_unstale":"2024-09-11T03:49:41.265985+0000","last_undegraded":"2024-09-11T03:49:41.265985+0000","last_fullsized":"2024-09-11T03:49:41.265985+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'620","last_scrub_stamp":"2024-09-11T03:49:34.288083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.288083+0000","objects_scrubbed":8,"log_size":628,"log_dups_size":0,"ondisk_log_size":628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:38.100295+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.035272568999999997,"stat_sum":{"num_bytes":15894717,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":743,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.027016+0000","last_change":"2024-09-11T03:49:17.858224+0000","last_active":"2024-09-11T03:49:42.027016+0000","last_peered":"2024-09-11T03:49:42.027016+0000","last_clean":"2024-09-11T03:49:42.027016+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:49:42.027016+0000","last_undegraded":"2024-09-11T03:49:42.027016+0000","last_fullsized":"2024-09-11T03:49:42.027016+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:17.858165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:17.858165+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:40.674663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"763'1131","reported_seq":4570,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.265971+0000","last_change":"2024-09-11T03:49:41.266268+0000","last_active":"2024-09-11T03:49:41.265971+0000","last_peered":"2024-09-11T03:49:41.265971+0000","last_clean":"2024-09-11T03:49:41.265971+0000","last_became_active":"2024-09-11T03:49:24.115095+0000","last_became_peered":"2024-09-11T03:49:24.115095+0000","last_unstale":"2024-09-11T03:49:41.265971+0000","last_undegraded":"2024-09-11T03:49:41.265971+0000","last_fullsized":"2024-09-11T03:49:41.265971+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'1112","last_scrub_stamp":"2024-09-11T03:49:35.188307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.188307+0000","objects_scrubbed":7,"log_size":1131,"log_dups_size":0,"ondisk_log_size":1131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:00.724225+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.035287266999999997,"stat_sum":{"num_bytes":10683593,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":742,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.027055+0000","last_change":"2024-09-11T03:49:36.654074+0000","last_active":"2024-09-11T03:49:42.027055+0000","last_peered":"2024-09-11T03:49:42.027055+0000","last_clean":"2024-09-11T03:49:42.027055+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:49:42.027055+0000","last_undegraded":"2024-09-11T03:49:42.027055+0000","last_fullsized":"2024-09-11T03:49:42.027055+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:36.654025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.654025+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:06.216209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":3530,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.235362+0000","last_change":"2024-09-11T03:49:41.235668+0000","last_active":"2024-09-11T03:49:41.235362+0000","last_peered":"2024-09-11T03:49:41.235362+0000","last_clean":"2024-09-11T03:49:41.235362+0000","last_became_active":"2024-09-11T03:49:24.113518+0000","last_became_peered":"2024-09-11T03:49:24.113518+0000","last_unstale":"2024-09-11T03:49:41.235362+0000","last_undegraded":"2024-09-11T03:49:41.235362+0000","last_fullsized":"2024-09-11T03:49:41.235362+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:49:35.267227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.267227+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:57.480786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049656600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":1343,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.031092+0000","last_change":"2024-09-11T03:49:21.909643+0000","last_active":"2024-09-11T03:49:42.031092+0000","last_peered":"2024-09-11T03:49:42.031092+0000","last_clean":"2024-09-11T03:49:42.031092+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:49:42.031092+0000","last_undegraded":"2024-09-11T03:49:42.031092+0000","last_fullsized":"2024-09-11T03:49:42.031092+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:21.909594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:21.909594+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:22.293071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"764'1090","reported_seq":4630,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.032200+0000","last_change":"2024-09-11T03:49:42.033148+0000","last_active":"2024-09-11T03:49:42.032200+0000","last_peered":"2024-09-11T03:49:42.032200+0000","last_clean":"2024-09-11T03:49:42.032200+0000","last_became_active":"2024-09-11T03:49:24.121333+0000","last_became_peered":"2024-09-11T03:49:24.121333+0000","last_unstale":"2024-09-11T03:49:42.032200+0000","last_undegraded":"2024-09-11T03:49:42.032200+0000","last_fullsized":"2024-09-11T03:49:42.032200+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'1066","last_scrub_stamp":"2024-09-11T03:49:35.600201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.600201+0000","objects_scrubbed":7,"log_size":1090,"log_dups_size":0,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:43.306393+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0073603829999999999,"stat_sum":{"num_bytes":24102888,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":29546,"num_write":11,"num_write_kb":6401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"119","length":"1"},{"start":"142","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"2"}]},{"pgid":"1.b","version":"20'92","reported_seq":1138,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.031672+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:49:42.031672+0000","last_peered":"2024-09-11T03:49:42.031672+0000","last_clean":"2024-09-11T03:49:42.031672+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:49:42.031672+0000","last_undegraded":"2024-09-11T03:49:42.031672+0000","last_fullsized":"2024-09-11T03:49:42.031672+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:13.961107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"744'374","reported_seq":3792,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.228060+0000","last_change":"2024-09-11T03:49:41.228435+0000","last_active":"2024-09-11T03:49:41.228060+0000","last_peered":"2024-09-11T03:49:41.228060+0000","last_clean":"2024-09-11T03:49:41.228060+0000","last_became_active":"2024-09-11T03:49:24.115150+0000","last_became_peered":"2024-09-11T03:49:24.115150+0000","last_unstale":"2024-09-11T03:49:41.228060+0000","last_undegraded":"2024-09-11T03:49:41.228060+0000","last_fullsized":"2024-09-11T03:49:41.228060+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'374","last_scrub_stamp":"2024-09-11T03:49:35.190806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.190806+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:40.689172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088419300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":1480,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-09-11T03:49:38.210447+0000","last_change":"2024-09-11T03:48:59.762212+0000","last_active":"2024-09-11T03:49:38.210447+0000","last_peered":"2024-09-11T03:49:38.210447+0000","last_clean":"2024-09-11T03:49:38.210447+0000","last_became_active":"2024-09-11T03:46:15.973271+0000","last_became_peered":"2024-09-11T03:46:15.973271+0000","last_unstale":"2024-09-11T03:49:38.210447+0000","last_undegraded":"2024-09-11T03:49:38.210447+0000","last_fullsized":"2024-09-11T03:49:38.210447+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:59.762145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:59.762145+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:02.580429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"763'1199","reported_seq":4694,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.235582+0000","last_change":"2024-09-11T03:49:41.235878+0000","last_active":"2024-09-11T03:49:41.235582+0000","last_peered":"2024-09-11T03:49:41.235582+0000","last_clean":"2024-09-11T03:49:41.235582+0000","last_became_active":"2024-09-11T03:49:24.114865+0000","last_became_peered":"2024-09-11T03:49:24.114865+0000","last_unstale":"2024-09-11T03:49:41.235582+0000","last_undegraded":"2024-09-11T03:49:41.235582+0000","last_fullsized":"2024-09-11T03:49:41.235582+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'1171","last_scrub_stamp":"2024-09-11T03:49:33.292043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.292043+0000","objects_scrubbed":10,"log_size":1199,"log_dups_size":0,"ondisk_log_size":1199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:54.147322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0054514300000000002,"stat_sum":{"num_bytes":21711515,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":10,"num_write_kb":3037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":1557,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.226740+0000","last_change":"2024-09-11T03:49:37.206392+0000","last_active":"2024-09-11T03:49:41.226740+0000","last_peered":"2024-09-11T03:49:41.226740+0000","last_clean":"2024-09-11T03:49:41.226740+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:49:41.226740+0000","last_undegraded":"2024-09-11T03:49:41.226740+0000","last_fullsized":"2024-09-11T03:49:41.226740+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:37.206321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.206321+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:53.641968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"762'352","reported_seq":3775,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.031553+0000","last_change":"2024-09-11T03:49:42.032062+0000","last_active":"2024-09-11T03:49:42.031553+0000","last_peered":"2024-09-11T03:49:42.031553+0000","last_clean":"2024-09-11T03:49:42.031553+0000","last_became_active":"2024-09-11T03:49:24.114551+0000","last_became_peered":"2024-09-11T03:49:24.114551+0000","last_unstale":"2024-09-11T03:49:42.031553+0000","last_undegraded":"2024-09-11T03:49:42.031553+0000","last_fullsized":"2024-09-11T03:49:42.031553+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:49:32.911631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.911631+0000","objects_scrubbed":4,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:02.283595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078678400000000001,"stat_sum":{"num_bytes":5429057,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"119","length":"1"},{"start":"142","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"2"}]},{"pgid":"1.8","version":"20'92","reported_seq":1216,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.238521+0000","last_change":"2024-09-11T03:49:34.621588+0000","last_active":"2024-09-11T03:49:41.238521+0000","last_peered":"2024-09-11T03:49:41.238521+0000","last_clean":"2024-09-11T03:49:41.238521+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:49:41.238521+0000","last_undegraded":"2024-09-11T03:49:41.238521+0000","last_fullsized":"2024-09-11T03:49:41.238521+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:34.621527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.621527+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.563733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"568'646","reported_seq":4050,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.229955+0000","last_change":"2024-09-11T03:49:41.230272+0000","last_active":"2024-09-11T03:49:41.229955+0000","last_peered":"2024-09-11T03:49:41.229955+0000","last_clean":"2024-09-11T03:49:41.229955+0000","last_became_active":"2024-09-11T03:49:24.113147+0000","last_became_peered":"2024-09-11T03:49:24.113147+0000","last_unstale":"2024-09-11T03:49:41.229955+0000","last_undegraded":"2024-09-11T03:49:41.229955+0000","last_fullsized":"2024-09-11T03:49:41.229955+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:49:33.204497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.204497+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:56.659223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00065256099999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":1165,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.238397+0000","last_change":"2024-09-11T03:49:00.710015+0000","last_active":"2024-09-11T03:49:41.238397+0000","last_peered":"2024-09-11T03:49:41.238397+0000","last_clean":"2024-09-11T03:49:41.238397+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:49:41.238397+0000","last_undegraded":"2024-09-11T03:49:41.238397+0000","last_fullsized":"2024-09-11T03:49:41.238397+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:00.709966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.709966+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:30.175623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":3424,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-09-11T03:49:38.217492+0000","last_change":"2024-09-11T03:49:38.218081+0000","last_active":"2024-09-11T03:49:38.217492+0000","last_peered":"2024-09-11T03:49:38.217492+0000","last_clean":"2024-09-11T03:49:38.217492+0000","last_became_active":"2024-09-11T03:41:38.886834+0000","last_became_peered":"2024-09-11T03:41:38.886834+0000","last_unstale":"2024-09-11T03:49:38.217492+0000","last_undegraded":"2024-09-11T03:49:38.217492+0000","last_fullsized":"2024-09-11T03:49:38.217492+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:49:15.904230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:15.904230+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:37.102094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0064060469999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"14f","length":"1"},{"start":"152","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":1140,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-09-11T03:49:40.226977+0000","last_change":"2024-09-11T03:48:30.121620+0000","last_active":"2024-09-11T03:49:40.226977+0000","last_peered":"2024-09-11T03:49:40.226977+0000","last_clean":"2024-09-11T03:49:40.226977+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:49:40.226977+0000","last_undegraded":"2024-09-11T03:49:40.226977+0000","last_fullsized":"2024-09-11T03:49:40.226977+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:30.121571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:30.121571+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"762'541","reported_seq":3928,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.028002+0000","last_change":"2024-09-11T03:49:42.028406+0000","last_active":"2024-09-11T03:49:42.028002+0000","last_peered":"2024-09-11T03:49:42.028002+0000","last_clean":"2024-09-11T03:49:42.028002+0000","last_became_active":"2024-09-11T03:43:55.918817+0000","last_became_peered":"2024-09-11T03:43:55.918817+0000","last_unstale":"2024-09-11T03:49:42.028002+0000","last_undegraded":"2024-09-11T03:49:42.028002+0000","last_fullsized":"2024-09-11T03:49:42.028002+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"736'523","last_scrub_stamp":"2024-09-11T03:49:20.735392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.735392+0000","objects_scrubbed":7,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:40.849232+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00094337299999999995,"stat_sum":{"num_bytes":11998221,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":200,"num_read_kb":210617,"num_write":139,"num_write_kb":52245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":15216236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"119","length":"1"},{"start":"142","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"2"}]},{"pgid":"1.12","version":"20'92","reported_seq":1146,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.238675+0000","last_change":"2024-09-11T03:47:23.339646+0000","last_active":"2024-09-11T03:49:41.238675+0000","last_peered":"2024-09-11T03:49:41.238675+0000","last_clean":"2024-09-11T03:49:41.238675+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:49:41.238675+0000","last_undegraded":"2024-09-11T03:49:41.238675+0000","last_fullsized":"2024-09-11T03:49:41.238675+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:23.339595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:23.339595+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:48.807974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":2524,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.033583+0000","last_change":"2024-09-11T03:49:42.033966+0000","last_active":"2024-09-11T03:49:42.033583+0000","last_peered":"2024-09-11T03:49:42.033583+0000","last_clean":"2024-09-11T03:49:42.033583+0000","last_became_active":"2024-09-11T03:46:19.005117+0000","last_became_peered":"2024-09-11T03:46:19.005117+0000","last_unstale":"2024-09-11T03:49:42.033583+0000","last_undegraded":"2024-09-11T03:49:42.033583+0000","last_fullsized":"2024-09-11T03:49:42.033583+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:49:40.618009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:40.618009+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:00.650949+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0075434020000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"119","length":"1"},{"start":"142","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"2"}]},{"pgid":"1.13","version":"20'92","reported_seq":1250,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.230476+0000","last_change":"2024-09-11T03:47:18.217600+0000","last_active":"2024-09-11T03:49:41.230476+0000","last_peered":"2024-09-11T03:49:41.230476+0000","last_clean":"2024-09-11T03:49:41.230476+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:49:41.230476+0000","last_undegraded":"2024-09-11T03:49:41.230476+0000","last_fullsized":"2024-09-11T03:49:41.230476+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:18.217555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:18.217555+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:43.129073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":2461,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.230618+0000","last_change":"2024-09-11T03:49:41.230977+0000","last_active":"2024-09-11T03:49:41.230618+0000","last_peered":"2024-09-11T03:49:41.230618+0000","last_clean":"2024-09-11T03:49:41.230618+0000","last_became_active":"2024-09-11T03:41:35.874295+0000","last_became_peered":"2024-09-11T03:41:35.874295+0000","last_unstale":"2024-09-11T03:49:41.230618+0000","last_undegraded":"2024-09-11T03:49:41.230618+0000","last_fullsized":"2024-09-11T03:49:41.230618+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:49:41.179684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:41.179684+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:51.810663+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011631160000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":725,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.227072+0000","last_change":"2024-09-11T03:48:57.320199+0000","last_active":"2024-09-11T03:49:41.227072+0000","last_peered":"2024-09-11T03:49:41.227072+0000","last_clean":"2024-09-11T03:49:41.227072+0000","last_became_active":"2024-09-11T03:46:36.026459+0000","last_became_peered":"2024-09-11T03:46:36.026459+0000","last_unstale":"2024-09-11T03:49:41.227072+0000","last_undegraded":"2024-09-11T03:49:41.227072+0000","last_fullsized":"2024-09-11T03:49:41.227072+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:57.320159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:57.320159+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:16.384062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":3020,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.034725+0000","last_change":"2024-09-11T03:49:42.035261+0000","last_active":"2024-09-11T03:49:42.034725+0000","last_peered":"2024-09-11T03:49:42.034725+0000","last_clean":"2024-09-11T03:49:42.034725+0000","last_became_active":"2024-09-11T03:46:19.003959+0000","last_became_peered":"2024-09-11T03:46:19.003959+0000","last_unstale":"2024-09-11T03:49:42.034725+0000","last_undegraded":"2024-09-11T03:49:42.034725+0000","last_fullsized":"2024-09-11T03:49:42.034725+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:49:38.613856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:38.613856+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:00.537228+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0088165410000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"119","length":"1"},{"start":"142","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"2"}]},{"pgid":"1.15","version":"20'92","reported_seq":1119,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.227882+0000","last_change":"2024-09-11T03:47:47.459856+0000","last_active":"2024-09-11T03:49:41.227882+0000","last_peered":"2024-09-11T03:49:41.227882+0000","last_clean":"2024-09-11T03:49:41.227882+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:49:41.227882+0000","last_undegraded":"2024-09-11T03:49:41.227882+0000","last_fullsized":"2024-09-11T03:49:41.227882+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:47.459803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:47.459803+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:08.322414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"763'535","reported_seq":3960,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.243133+0000","last_change":"2024-09-11T03:49:41.243602+0000","last_active":"2024-09-11T03:49:41.243133+0000","last_peered":"2024-09-11T03:49:41.243133+0000","last_clean":"2024-09-11T03:49:41.243133+0000","last_became_active":"2024-09-11T03:46:19.007019+0000","last_became_peered":"2024-09-11T03:46:19.007019+0000","last_unstale":"2024-09-11T03:49:41.243133+0000","last_undegraded":"2024-09-11T03:49:41.243133+0000","last_fullsized":"2024-09-11T03:49:41.243133+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"755'521","last_scrub_stamp":"2024-09-11T03:49:33.626967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.626967+0000","objects_scrubbed":9,"log_size":535,"log_dups_size":0,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:53.371757+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.004974175,"stat_sum":{"num_bytes":13462623,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":133,"num_read_kb":122995,"num_write":103,"num_write_kb":43311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":5263193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14e","length":"2"},{"start":"152","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":601,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.032098+0000","last_change":"2024-09-11T03:49:20.949194+0000","last_active":"2024-09-11T03:49:42.032098+0000","last_peered":"2024-09-11T03:49:42.032098+0000","last_clean":"2024-09-11T03:49:42.032098+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:49:42.032098+0000","last_undegraded":"2024-09-11T03:49:42.032098+0000","last_fullsized":"2024-09-11T03:49:42.032098+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:20.949118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.949118+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:36.681631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":3644,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-09-11T03:49:38.211402+0000","last_change":"2024-09-11T03:49:38.212792+0000","last_active":"2024-09-11T03:49:38.211402+0000","last_peered":"2024-09-11T03:49:38.211402+0000","last_clean":"2024-09-11T03:49:38.211402+0000","last_became_active":"2024-09-11T03:41:38.888228+0000","last_became_peered":"2024-09-11T03:41:38.888228+0000","last_unstale":"2024-09-11T03:49:38.211402+0000","last_undegraded":"2024-09-11T03:49:38.211402+0000","last_fullsized":"2024-09-11T03:49:38.211402+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:49:16.919210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:16.919210+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:34.268029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001947021,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"14f","length":"1"},{"start":"152","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":1138,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.229636+0000","last_change":"2024-09-11T03:47:29.086383+0000","last_active":"2024-09-11T03:49:41.229636+0000","last_peered":"2024-09-11T03:49:41.229636+0000","last_clean":"2024-09-11T03:49:41.229636+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:49:41.229636+0000","last_undegraded":"2024-09-11T03:49:41.229636+0000","last_fullsized":"2024-09-11T03:49:41.229636+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:29.086335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:29.086335+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:42.049323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"764'346","reported_seq":3639,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.036747+0000","last_change":"2024-09-11T03:49:42.037068+0000","last_active":"2024-09-11T03:49:42.036747+0000","last_peered":"2024-09-11T03:49:42.036747+0000","last_clean":"2024-09-11T03:49:42.036747+0000","last_became_active":"2024-09-11T03:46:19.004558+0000","last_became_peered":"2024-09-11T03:46:19.004558+0000","last_unstale":"2024-09-11T03:49:42.036747+0000","last_undegraded":"2024-09-11T03:49:42.036747+0000","last_fullsized":"2024-09-11T03:49:42.036747+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"711'317","last_scrub_stamp":"2024-09-11T03:48:50.287601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:50.287601+0000","objects_scrubbed":5,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:11.226958+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0098077479999999998,"stat_sum":{"num_bytes":7686492,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":47,"num_read_kb":27601,"num_write":62,"num_write_kb":18466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"119","length":"1"},{"start":"142","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"2"}]},{"pgid":"1.18","version":"20'92","reported_seq":1250,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.238282+0000","last_change":"2024-09-11T03:48:53.702605+0000","last_active":"2024-09-11T03:49:41.238282+0000","last_peered":"2024-09-11T03:49:41.238282+0000","last_clean":"2024-09-11T03:49:41.238282+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:49:41.238282+0000","last_undegraded":"2024-09-11T03:49:41.238282+0000","last_fullsized":"2024-09-11T03:49:41.238282+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:53.702523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:53.702523+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:55.160792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"764'803","reported_seq":4307,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.035932+0000","last_change":"2024-09-11T03:49:42.036295+0000","last_active":"2024-09-11T03:49:42.035932+0000","last_peered":"2024-09-11T03:49:42.035932+0000","last_clean":"2024-09-11T03:49:42.035932+0000","last_became_active":"2024-09-11T03:46:32.975567+0000","last_became_peered":"2024-09-11T03:46:32.975567+0000","last_unstale":"2024-09-11T03:49:42.035932+0000","last_undegraded":"2024-09-11T03:49:42.035932+0000","last_fullsized":"2024-09-11T03:49:42.035932+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"754'773","last_scrub_stamp":"2024-09-11T03:49:34.919322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.919322+0000","objects_scrubbed":8,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:47.150403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.004791577,"stat_sum":{"num_bytes":14865246,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":88,"num_read_kb":90522,"num_write":44,"num_write_kb":21756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7],"acting":[11,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"119","length":"1"},{"start":"142","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"2"}]},{"pgid":"1.19","version":"20'92","reported_seq":747,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.026916+0000","last_change":"2024-09-11T03:48:56.129575+0000","last_active":"2024-09-11T03:49:42.026916+0000","last_peered":"2024-09-11T03:49:42.026916+0000","last_clean":"2024-09-11T03:49:42.026916+0000","last_became_active":"2024-09-11T03:46:15.972841+0000","last_became_peered":"2024-09-11T03:46:15.972841+0000","last_unstale":"2024-09-11T03:49:42.026916+0000","last_undegraded":"2024-09-11T03:49:42.026916+0000","last_fullsized":"2024-09-11T03:49:42.026916+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:56.129516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:56.129516+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:24.133914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"557'273","reported_seq":3594,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.037136+0000","last_change":"2024-09-11T03:49:42.037679+0000","last_active":"2024-09-11T03:49:42.037136+0000","last_peered":"2024-09-11T03:49:42.037136+0000","last_clean":"2024-09-11T03:49:42.037136+0000","last_became_active":"2024-09-11T03:46:32.971003+0000","last_became_peered":"2024-09-11T03:46:32.971003+0000","last_unstale":"2024-09-11T03:49:42.037136+0000","last_undegraded":"2024-09-11T03:49:42.037136+0000","last_fullsized":"2024-09-11T03:49:42.037136+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:49:20.466140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.466140+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:31.465358+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0094710869999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,2],"acting":[6,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"119","length":"1"},{"start":"142","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"2"}]},{"pgid":"1.1a","version":"20'92","reported_seq":282,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.227197+0000","last_change":"2024-09-11T03:49:31.427740+0000","last_active":"2024-09-11T03:49:41.227197+0000","last_peered":"2024-09-11T03:49:41.227197+0000","last_clean":"2024-09-11T03:49:41.227197+0000","last_became_active":"2024-09-11T03:49:30.162694+0000","last_became_peered":"2024-09-11T03:49:30.162694+0000","last_unstale":"2024-09-11T03:49:41.227197+0000","last_undegraded":"2024-09-11T03:49:41.227197+0000","last_fullsized":"2024-09-11T03:49:41.227197+0000","mapping_epoch":751,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":752,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:52.914088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:52.914088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:50:01.193085+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":72127,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.19","version":"762'560","reported_seq":4020,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.033126+0000","last_change":"2024-09-11T03:49:42.033467+0000","last_active":"2024-09-11T03:49:42.033126+0000","last_peered":"2024-09-11T03:49:42.033126+0000","last_clean":"2024-09-11T03:49:42.033126+0000","last_became_active":"2024-09-11T03:46:19.002002+0000","last_became_peered":"2024-09-11T03:46:19.002002+0000","last_unstale":"2024-09-11T03:49:42.033126+0000","last_undegraded":"2024-09-11T03:49:42.033126+0000","last_fullsized":"2024-09-11T03:49:42.033126+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"736'546","last_scrub_stamp":"2024-09-11T03:49:19.458345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:19.458345+0000","objects_scrubbed":6,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:37.279007+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0074931219999999996,"stat_sum":{"num_bytes":6185278,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":92,"num_read_kb":91932,"num_write":57,"num_write_kb":26464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4870845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"119","length":"1"},{"start":"142","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"2"}]},{"pgid":"1.1b","version":"20'92","reported_seq":1115,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.229492+0000","last_change":"2024-09-11T03:48:31.033896+0000","last_active":"2024-09-11T03:49:41.229492+0000","last_peered":"2024-09-11T03:49:41.229492+0000","last_clean":"2024-09-11T03:49:41.229492+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:49:41.229492+0000","last_undegraded":"2024-09-11T03:49:41.229492+0000","last_fullsized":"2024-09-11T03:49:41.229492+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:31.033837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:31.033837+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"760'501","reported_seq":3921,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-09-11T03:49:38.435963+0000","last_change":"2024-09-11T03:49:38.217110+0000","last_active":"2024-09-11T03:49:38.435963+0000","last_peered":"2024-09-11T03:49:38.435963+0000","last_clean":"2024-09-11T03:49:38.435963+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:49:38.435963+0000","last_undegraded":"2024-09-11T03:49:38.435963+0000","last_fullsized":"2024-09-11T03:49:38.435963+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"711'462","last_scrub_stamp":"2024-09-11T03:48:49.842909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:49.842909+0000","objects_scrubbed":9,"log_size":501,"log_dups_size":0,"ondisk_log_size":501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:04.358868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0061873409999999998,"stat_sum":{"num_bytes":24339313,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":185,"num_read_kb":220026,"num_write":92,"num_write_kb":42535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4865740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"14f","length":"1"},{"start":"152","length":"1"}]},{"pgid":"2.1f","version":"762'663","reported_seq":3986,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-09-11T03:49:40.232956+0000","last_change":"2024-09-11T03:49:40.233615+0000","last_active":"2024-09-11T03:49:40.232956+0000","last_peered":"2024-09-11T03:49:40.232956+0000","last_clean":"2024-09-11T03:49:40.232956+0000","last_became_active":"2024-09-11T03:46:32.968696+0000","last_became_peered":"2024-09-11T03:46:32.968696+0000","last_unstale":"2024-09-11T03:49:40.232956+0000","last_undegraded":"2024-09-11T03:49:40.232956+0000","last_fullsized":"2024-09-11T03:49:40.232956+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"733'613","last_scrub_stamp":"2024-09-11T03:49:20.295589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.295589+0000","objects_scrubbed":10,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.544182+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0057899240000000001,"stat_sum":{"num_bytes":31988334,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":117,"num_read_kb":137133,"num_write":71,"num_write_kb":25208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14f","length":"1"},{"start":"152","length":"1"}]},{"pgid":"1.1c","version":"20'92","reported_seq":1122,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.230253+0000","last_change":"2024-09-11T03:49:40.156559+0000","last_active":"2024-09-11T03:49:41.230253+0000","last_peered":"2024-09-11T03:49:41.230253+0000","last_clean":"2024-09-11T03:49:41.230253+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:49:41.230253+0000","last_undegraded":"2024-09-11T03:49:41.230253+0000","last_fullsized":"2024-09-11T03:49:41.230253+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:40.156495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:40.156495+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:04.047069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1e","version":"762'1021","reported_seq":4349,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-09-11T03:49:40.230778+0000","last_change":"2024-09-11T03:49:40.232370+0000","last_active":"2024-09-11T03:49:40.230778+0000","last_peered":"2024-09-11T03:49:40.230778+0000","last_clean":"2024-09-11T03:49:40.230778+0000","last_became_active":"2024-09-11T03:46:32.970637+0000","last_became_peered":"2024-09-11T03:46:32.970637+0000","last_unstale":"2024-09-11T03:49:40.230778+0000","last_undegraded":"2024-09-11T03:49:40.230778+0000","last_fullsized":"2024-09-11T03:49:40.230778+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"718'982","last_scrub_stamp":"2024-09-11T03:48:56.189316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:56.189316+0000","objects_scrubbed":9,"log_size":1021,"log_dups_size":0,"ondisk_log_size":1021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:06.337071+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0044645079999999998,"stat_sum":{"num_bytes":13212919,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":54,"num_read_kb":51944,"num_write":77,"num_write_kb":27431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"119","length":"1"},{"start":"11e","length":"1"},{"start":"12e","length":"1"},{"start":"138","length":"1"},{"start":"142","length":"1"},{"start":"14f","length":"1"},{"start":"152","length":"1"}]},{"pgid":"1.1d","version":"20'92","reported_seq":1104,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-11T03:49:41.230602+0000","last_change":"2024-09-11T03:47:22.312507+0000","last_active":"2024-09-11T03:49:41.230602+0000","last_peered":"2024-09-11T03:49:41.230602+0000","last_clean":"2024-09-11T03:49:41.230602+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:49:41.230602+0000","last_undegraded":"2024-09-11T03:49:41.230602+0000","last_fullsized":"2024-09-11T03:49:41.230602+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:22.312456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:22.312456+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:45.114530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"567'752","reported_seq":4087,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.027469+0000","last_change":"2024-09-11T03:49:42.027910+0000","last_active":"2024-09-11T03:49:42.027469+0000","last_peered":"2024-09-11T03:49:42.027469+0000","last_clean":"2024-09-11T03:49:42.027469+0000","last_became_active":"2024-09-11T03:46:32.972371+0000","last_became_peered":"2024-09-11T03:46:32.972371+0000","last_unstale":"2024-09-11T03:49:42.027469+0000","last_undegraded":"2024-09-11T03:49:42.027469+0000","last_fullsized":"2024-09-11T03:49:42.027469+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:48:55.139430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:55.139430+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:21.112534+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00069368999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,11],"acting":[10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"119","length":"1"},{"start":"142","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"2"}]},{"pgid":"1.1e","version":"20'92","reported_seq":1134,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-09-11T03:49:42.032186+0000","last_change":"2024-09-11T03:49:36.949199+0000","last_active":"2024-09-11T03:49:42.032186+0000","last_peered":"2024-09-11T03:49:42.032186+0000","last_clean":"2024-09-11T03:49:42.032186+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:49:42.032186+0000","last_undegraded":"2024-09-11T03:49:42.032186+0000","last_fullsized":"2024-09-11T03:49:42.032186+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:36.949149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.949149+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:45.388809+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":46,"stat_sum":{"num_bytes":396701568,"num_objects":188,"num_object_clones":138,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":12,"num_read":1227,"num_read_kb":1285363,"num_write":859,"num_write_kb":358404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":30216014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":747483136,"data_stored":999701826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26331,"ondisk_log_size":26331,"up":138,"acting":138,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":399807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91720,"kb_used_data":60524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93653432,"statfs":{"total":95995035648,"available":95901114368,"internally_reserved":0,"allocated":61976576,"data_stored":73100573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514650,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90448,"kb_used_data":50940,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93654704,"statfs":{"total":95995035648,"available":95902416896,"internally_reserved":0,"allocated":52162560,"data_stored":59201737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743583,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":123572,"kb_used_data":70176,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93621580,"statfs":{"total":95995035648,"available":95868497920,"internally_reserved":0,"allocated":71860224,"data_stored":93758424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037927,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":128468,"kb_used_data":75068,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93616684,"statfs":{"total":95995035648,"available":95863484416,"internally_reserved":0,"allocated":76869632,"data_stored":99243784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299562,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":173904,"kb_used_data":116348,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93571248,"statfs":{"total":95995035648,"available":95816957952,"internally_reserved":0,"allocated":119140352,"data_stored":154320452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528495,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":146844,"kb_used_data":107336,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93598308,"statfs":{"total":95995035648,"available":95844667392,"internally_reserved":0,"allocated":109912064,"data_stored":137993009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757429,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85100,"kb_used_data":58064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93660052,"statfs":{"total":95995035648,"available":95907893248,"internally_reserved":0,"allocated":59457536,"data_stored":75905427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986362,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90256,"kb_used_data":36860,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93654896,"statfs":{"total":95995035648,"available":95902613504,"internally_reserved":0,"allocated":37744640,"data_stored":47753984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149886,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":141092,"kb_used_data":96020,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93604060,"statfs":{"total":95995035648,"available":95850557440,"internally_reserved":0,"allocated":98324480,"data_stored":124428337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411524,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72952,"kb_used_data":41760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93672200,"statfs":{"total":95995035648,"available":95920332800,"internally_reserved":0,"allocated":42762240,"data_stored":48509635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575050,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85268,"kb_used_data":49916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93659884,"statfs":{"total":95995035648,"available":95907721216,"internally_reserved":0,"allocated":51113984,"data_stored":63830257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738575,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":122072,"kb_used_data":68676,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93623080,"statfs":{"total":95995035648,"available":95870033920,"internally_reserved":0,"allocated":70324224,"data_stored":90697250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":58077184,"data_stored":79618543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41889792,"data_stored":57830484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36392960,"data_stored":45341312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90406912,"data_stored":122102967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28532736,"data_stored":37906587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":50028544,"data_stored":69704406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":102699008,"data_stored":134007145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":109711360,"data_stored":148119431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":67645440,"data_stored":93244011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":64081920,"data_stored":89190938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":42926080,"data_stored":53193613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":55091200,"data_stored":69442389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:49:44.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set unique_pool_0 pgp_num 46 2024-09-11T03:49:44.563 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:44.584 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:44.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:44.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:44.686 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:44.707 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:44.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:44.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:44.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:44.829 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:44.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:44.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:44.932 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:44.952 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:44.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:44.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:45.054 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:45.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:45.078 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:45.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:45.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:45.203 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:45.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:45.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:45.237 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:49:45.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:49:45.238 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:49:45.238 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:49:45.240 INFO:tasks.rados.rados.2.smithi175.stdout:3091: done (1 left) 2024-09-11T03:49:45.240 INFO:tasks.rados.rados.2.smithi175.stdout:3092: done (0 left) 2024-09-11T03:49:45.240 INFO:tasks.rados.rados.2.smithi175.stdout:3093: read oid 40 snap 311 2024-09-11T03:49:45.240 INFO:tasks.rados.rados.2.smithi175.stdout:3093: expect deleted 2024-09-11T03:49:45.241 INFO:tasks.rados.rados.2.smithi175.stdout:3094: snap_create 2024-09-11T03:49:45.253 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:49:45.254 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:49:45.271 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:49:45.271 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:49:45.288 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:49:45.288 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:49:45.304 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:49:45.305 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:49:45.305 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:45.321 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:49:45.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:49:45.328 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:45.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:45.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:45.337 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:49:45.338 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:49:45.354 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:49:45.354 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:49:45.370 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:49:45.371 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:49:45.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:45 smithi084 ceph-mon[27254]: osdmap e766: 12 total, 12 up, 12 in 2024-09-11T03:49:45.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:45 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1912057244' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:45.387 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:49:45.388 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:49:45.404 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:49:45.404 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:49:45.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:45.453 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:45.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:45.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:45.555 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:45.578 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:45.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:45.656 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:49:45.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:45 smithi121 ceph-mon[30729]: osdmap e766: 12 total, 12 up, 12 in 2024-09-11T03:49:45.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:45 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1912057244' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:45.680 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:45.703 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:45.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:45.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:45.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:45 smithi175 ceph-mon[30894]: osdmap e766: 12 total, 12 up, 12 in 2024-09-11T03:49:45.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:45 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1912057244' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:45.805 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:45.830 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:45.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:45.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:45.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:45.955 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:45.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:45.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:46.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:46.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:46.079 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:46.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:46.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:46.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:46.208 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:46.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:46.245 INFO:tasks.rados.rados.2.smithi175.stdout:3093: done (1 left) 2024-09-11T03:49:46.245 INFO:tasks.rados.rados.2.smithi175.stdout:3094: done (0 left) 2024-09-11T03:49:46.245 INFO:tasks.rados.rados.2.smithi175.stdout:3095: delete oid 49 current snap is 343 2024-09-11T03:49:46.249 INFO:tasks.rados.rados.2.smithi175.stdout:3095: done (0 left) 2024-09-11T03:49:46.249 INFO:tasks.rados.rados.2.smithi175.stdout:3096: read oid 35 snap -1 2024-09-11T03:49:46.249 INFO:tasks.rados.rados.2.smithi175.stdout:3096: expect deleted 2024-09-11T03:49:46.249 INFO:tasks.rados.rados.2.smithi175.stdout:3097: read oid 42 snap -1 2024-09-11T03:49:46.249 INFO:tasks.rados.rados.2.smithi175.stdout:3097: expect (ObjNum 1042 snap 340 seq_num 1042) 2024-09-11T03:49:46.249 INFO:tasks.rados.rados.2.smithi175.stdout:3098: delete oid 32 current snap is 343 2024-09-11T03:49:46.252 INFO:tasks.rados.rados.2.smithi175.stdout:3096: done (2 left) 2024-09-11T03:49:46.252 INFO:tasks.rados.rados.2.smithi175.stdout:3098: done (1 left) 2024-09-11T03:49:46.252 INFO:tasks.rados.rados.2.smithi175.stdout:3099: delete oid 20 current snap is 343 2024-09-11T03:49:46.256 INFO:tasks.rados.rados.2.smithi175.stdout:3099: done (1 left) 2024-09-11T03:49:46.256 INFO:tasks.rados.rados.2.smithi175.stdout:3100: delete oid 43 current snap is 343 2024-09-11T03:49:46.257 INFO:tasks.rados.rados.2.smithi175.stdout:3097: expect (ObjNum 1042 snap 340 seq_num 1042) 2024-09-11T03:49:46.259 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:49:46.259 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:49:46.259 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:49:46.259 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:49:46.260 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:49:46.309 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:46.330 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:46.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:46.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:46.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:46 smithi084 ceph-mon[27254]: pgmap v1305: 77 pgs: 77 active+clean; 380 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4.7 MiB/s wr, 28 op/s 2024-09-11T03:49:46.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:46 smithi084 ceph-mon[27254]: osdmap e767: 12 total, 12 up, 12 in 2024-09-11T03:49:46.407 INFO:tasks.rados.rados.2.smithi175.stdout:3097: done (1 left) 2024-09-11T03:49:46.407 INFO:tasks.rados.rados.2.smithi175.stdout:3100: done (0 left) 2024-09-11T03:49:46.407 INFO:tasks.rados.rados.2.smithi175.stdout:3101: snap_remove snap 329 2024-09-11T03:49:46.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:46.458 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:46.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:46.465 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:46.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:46.583 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:46.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:46.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:46.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:46 smithi121 ceph-mon[30729]: pgmap v1305: 77 pgs: 77 active+clean; 380 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4.7 MiB/s wr, 28 op/s 2024-09-11T03:49:46.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:46 smithi121 ceph-mon[30729]: osdmap e767: 12 total, 12 up, 12 in 2024-09-11T03:49:46.685 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:46.707 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:46.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:46.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:46.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:46 smithi175 ceph-mon[30894]: pgmap v1305: 77 pgs: 77 active+clean; 380 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4.7 MiB/s wr, 28 op/s 2024-09-11T03:49:46.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:46 smithi175 ceph-mon[30894]: osdmap e767: 12 total, 12 up, 12 in 2024-09-11T03:49:46.808 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:46.830 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:46.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:46.831 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:46.932 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:46.953 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:46.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:46.954 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:47.000 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T03:49:47.022 INFO:tasks.rados.rados.2.smithi175.stdout:3101: done (0 left) 2024-09-11T03:49:47.022 INFO:tasks.rados.rados.2.smithi175.stdout:3102: write (excl) oid 22 current snap is 343 2024-09-11T03:49:47.022 INFO:tasks.rados.rados.2.smithi175.stdout:3102: seq_num 1054 ranges {514578=569160,1677668=414031,2577952=641878} 2024-09-11T03:49:47.028 INFO:tasks.rados.rados.2.smithi175.stdout:3102: writing smithi17557870-22 from 514578 to 1083738 tid 1 2024-09-11T03:49:47.034 INFO:tasks.rados.rados.2.smithi175.stdout:3102: writing smithi17557870-22 from 1677668 to 2091699 tid 2 2024-09-11T03:49:47.037 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:47.040 INFO:tasks.rados.rados.2.smithi175.stdout:3102: writing smithi17557870-22 from 2577952 to 3219830 tid 3 2024-09-11T03:49:47.041 INFO:tasks.rados.rados.2.smithi175.stdout:3103: delete oid 48 current snap is 343 2024-09-11T03:49:47.041 INFO:tasks.rados.rados.2.smithi175.stdout:3102: finishing write tid 1 to smithi17557870-22 2024-09-11T03:49:47.041 INFO:tasks.rados.rados.2.smithi175.stdout:3102: finishing write tid 2 to smithi17557870-22 2024-09-11T03:49:47.042 INFO:tasks.rados.rados.2.smithi175.stdout:3103: done (1 left) 2024-09-11T03:49:47.042 INFO:tasks.rados.rados.2.smithi175.stdout:3104: snap_create 2024-09-11T03:49:47.045 INFO:tasks.rados.rados.2.smithi175.stdout:3102: finishing write tid 3 to smithi17557870-22 2024-09-11T03:49:47.045 INFO:tasks.rados.rados.2.smithi175.stdout:3102: finishing write tid 4 to smithi17557870-22 2024-09-11T03:49:47.045 INFO:tasks.rados.rados.2.smithi175.stdout:3102: finishing write tid 5 to smithi17557870-22 2024-09-11T03:49:47.046 INFO:tasks.rados.rados.2.smithi175.stdout:3102: oid 22 updating version 0 to 1187 2024-09-11T03:49:47.046 INFO:tasks.rados.rados.2.smithi175.stdout:3102: oid 22 updating version 1187 to 1188 2024-09-11T03:49:47.046 INFO:tasks.rados.rados.2.smithi175.stdout:3102: oid 22 updating version 1188 to 1189 2024-09-11T03:49:47.046 INFO:tasks.rados.rados.2.smithi175.stdout:3102: oid 22 updating version 1189 to 1190 2024-09-11T03:49:47.046 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 1190 (ObjNum 1054 snap 343 seq_num 1054) dirty exists 2024-09-11T03:49:47.046 INFO:tasks.rados.rados.2.smithi175.stdout:3102: left oid 22 (ObjNum 1054 snap 343 seq_num 1054) 2024-09-11T03:49:47.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:47.076 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:47.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:47.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:47.204 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:47.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:47.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:47.306 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:47.327 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:47.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:47.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:47.429 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:47.450 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:47.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:47.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:47.552 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:47.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:47 smithi084 ceph-mon[27254]: osdmap e768: 12 total, 12 up, 12 in 2024-09-11T03:49:47.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:47 smithi084 ceph-mon[27254]: osdmap e769: 12 total, 12 up, 12 in 2024-09-11T03:49:47.574 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:47.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:47.575 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:47.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:47 smithi121 ceph-mon[30729]: osdmap e768: 12 total, 12 up, 12 in 2024-09-11T03:49:47.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:47 smithi121 ceph-mon[30729]: osdmap e769: 12 total, 12 up, 12 in 2024-09-11T03:49:47.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:47.702 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:47.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:47.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:47.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:47 smithi175 ceph-mon[30894]: osdmap e768: 12 total, 12 up, 12 in 2024-09-11T03:49:47.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:47 smithi175 ceph-mon[30894]: osdmap e769: 12 total, 12 up, 12 in 2024-09-11T03:49:47.804 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:47.825 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:47.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:47.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:47.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:47.952 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:47.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:48.019 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 2 pgp_num to 46 2024-09-11T03:49:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:3102: done (1 left) 2024-09-11T03:49:48.022 INFO:tasks.rados.rados.2.smithi175.stdout:3104: done (0 left) 2024-09-11T03:49:48.023 INFO:tasks.rados.rados.2.smithi175.stdout:3105: write (excl) oid 6 current snap is 344 2024-09-11T03:49:48.023 INFO:tasks.rados.rados.2.smithi175.stdout:3105: seq_num 1056 ranges {548192=719658,1678279=417700,2516445=780775} 2024-09-11T03:49:48.029 INFO:tasks.rados.rados.2.smithi175.stdout:3105: writing smithi17557870-6 from 548192 to 1267850 tid 1 2024-09-11T03:49:48.034 INFO:tasks.rados.rados.2.smithi175.stdout:3105: writing smithi17557870-6 from 1678279 to 2095979 tid 2 2024-09-11T03:49:48.040 INFO:tasks.rados.rados.2.smithi175.stdout:3105: writing smithi17557870-6 from 2516445 to 3297220 tid 3 2024-09-11T03:49:48.040 INFO:tasks.rados.rados.2.smithi175.stdout:3106: write (excl) oid 42 current snap is 344 2024-09-11T03:49:48.041 INFO:tasks.rados.rados.2.smithi175.stdout:3105: finishing write tid 1 to smithi17557870-6 2024-09-11T03:49:48.041 INFO:tasks.rados.rados.2.smithi175.stdout:3105: finishing write tid 2 to smithi17557870-6 2024-09-11T03:49:48.041 INFO:tasks.rados.rados.2.smithi175.stdout:3106: seq_num 1057 ranges {564999=511260,1853114=561358,3060221=592047} 2024-09-11T03:49:48.046 INFO:tasks.rados.rados.2.smithi175.stdout:3106: writing smithi17557870-42 from 564999 to 1076259 tid 1 2024-09-11T03:49:48.053 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:48.054 INFO:tasks.rados.rados.2.smithi175.stdout:3106: writing smithi17557870-42 from 1853114 to 2414472 tid 2 2024-09-11T03:49:48.060 INFO:tasks.rados.rados.2.smithi175.stdout:3106: writing smithi17557870-42 from 3060221 to 3652268 tid 3 2024-09-11T03:49:48.061 INFO:tasks.rados.rados.2.smithi175.stdout:3107: snap_remove snap 342 2024-09-11T03:49:48.078 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:48.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:48.079 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:48.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:49:48.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:48.200 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:48.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:48.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:48.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:48.328 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:48.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:48.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:48.430 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:48.451 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:48.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:48.553 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:48.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:48.575 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:48.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:48.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:48 smithi084 ceph-mon[27254]: pgmap v1308: 77 pgs: 77 active+clean; 398 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 17 op/s 2024-09-11T03:49:48.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:48 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:49:48.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:48 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:49:48.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:48 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2835283051' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "46"} : dispatch 2024-09-11T03:49:48.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:48 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2835283051' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "46"}]': finished 2024-09-11T03:49:48.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:48 smithi084 ceph-mon[27254]: osdmap e770: 12 total, 12 up, 12 in 2024-09-11T03:49:48.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:48 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2500042571' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:49:48.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:48 smithi121 ceph-mon[30729]: pgmap v1308: 77 pgs: 77 active+clean; 398 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 17 op/s 2024-09-11T03:49:48.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:48 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:49:48.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:48 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:49:48.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:48 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2835283051' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "46"} : dispatch 2024-09-11T03:49:48.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:48 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2835283051' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "46"}]': finished 2024-09-11T03:49:48.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:48 smithi121 ceph-mon[30729]: osdmap e770: 12 total, 12 up, 12 in 2024-09-11T03:49:48.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:48 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2500042571' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:49:48.676 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:48.697 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:48.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:48.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:48.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:48 smithi175 ceph-mon[30894]: pgmap v1308: 77 pgs: 77 active+clean; 398 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 17 op/s 2024-09-11T03:49:48.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:48 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:49:48.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:48 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:49:48.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:48 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2835283051' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "46"} : dispatch 2024-09-11T03:49:48.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:48 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2835283051' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "46"}]': finished 2024-09-11T03:49:48.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:48 smithi175 ceph-mon[30894]: osdmap e770: 12 total, 12 up, 12 in 2024-09-11T03:49:48.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:48 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2500042571' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:49:48.799 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:48.820 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:48.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:48.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:48.904 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:48.908 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:48.923 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:48.949 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:48.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:48.950 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:49.020 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:49:49.027 INFO:tasks.rados.rados.2.smithi175.stdout:3107: done (2 left) 2024-09-11T03:49:49.027 INFO:tasks.rados.rados.2.smithi175.stdout:3108: copy_from oid 25 from oid 34 current snap is 344 2024-09-11T03:49:49.027 INFO:tasks.rados.rados.2.smithi175.stdout:3105: finishing write tid 3 to smithi17557870-6 2024-09-11T03:49:49.027 INFO:tasks.rados.rados.2.smithi175.stdout:3105: finishing write tid 4 to smithi17557870-6 2024-09-11T03:49:49.027 INFO:tasks.rados.rados.2.smithi175.stdout:3105: finishing write tid 5 to smithi17557870-6 2024-09-11T03:49:49.027 INFO:tasks.rados.rados.2.smithi175.stdout:3105: oid 6 updating version 0 to 976 2024-09-11T03:49:49.027 INFO:tasks.rados.rados.2.smithi175.stdout:3105: oid 6 version 976 is already newer than 975 2024-09-11T03:49:49.027 INFO:tasks.rados.rados.2.smithi175.stdout:3105: oid 6 updating version 976 to 977 2024-09-11T03:49:49.028 INFO:tasks.rados.rados.2.smithi175.stdout:3105: oid 6 version 977 is already newer than 974 2024-09-11T03:49:49.028 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 977 (ObjNum 1056 snap 344 seq_num 1056) dirty exists 2024-09-11T03:49:49.028 INFO:tasks.rados.rados.2.smithi175.stdout:3105: left oid 6 (ObjNum 1056 snap 344 seq_num 1056) 2024-09-11T03:49:49.028 INFO:tasks.rados.rados.2.smithi175.stdout:3106: finishing write tid 1 to smithi17557870-42 2024-09-11T03:49:49.028 INFO:tasks.rados.rados.2.smithi175.stdout:3106: finishing write tid 2 to smithi17557870-42 2024-09-11T03:49:49.028 INFO:tasks.rados.rados.2.smithi175.stdout:3106: finishing write tid 3 to smithi17557870-42 2024-09-11T03:49:49.028 INFO:tasks.rados.rados.2.smithi175.stdout:3106: finishing write tid 4 to smithi17557870-42 2024-09-11T03:49:49.028 INFO:tasks.rados.rados.2.smithi175.stdout:3106: finishing write tid 5 to smithi17557870-42 2024-09-11T03:49:49.028 INFO:tasks.rados.rados.2.smithi175.stdout:3106: oid 42 updating version 0 to 967 2024-09-11T03:49:49.028 INFO:tasks.rados.rados.2.smithi175.stdout:3106: oid 42 updating version 967 to 968 2024-09-11T03:49:49.028 INFO:tasks.rados.rados.2.smithi175.stdout:3106: oid 42 updating version 968 to 969 2024-09-11T03:49:49.028 INFO:tasks.rados.rados.2.smithi175.stdout:3106: oid 42 version 969 is already newer than 966 2024-09-11T03:49:49.028 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 969 (ObjNum 1057 snap 344 seq_num 1057) dirty exists 2024-09-11T03:49:49.028 INFO:tasks.rados.rados.2.smithi175.stdout:3106: left oid 42 (ObjNum 1057 snap 344 seq_num 1057) 2024-09-11T03:49:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:3105: done (2 left) 2024-09-11T03:49:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:3106: done (1 left) 2024-09-11T03:49:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:3109: read oid 15 snap -1 2024-09-11T03:49:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:3109: expect deleted 2024-09-11T03:49:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:3110: rollback oid 14 current snap is 344 2024-09-11T03:49:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 14 to 337 2024-09-11T03:49:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:3111: write (excl) oid 4 current snap is 344 2024-09-11T03:49:49.029 INFO:tasks.rados.rados.2.smithi175.stdout:3111: seq_num 1058 ranges {581806=586509,1911596=421369,3087644=403319} 2024-09-11T03:49:49.033 INFO:tasks.rados.rados.2.smithi175.stdout:3111: writing smithi17557870-4 from 581806 to 1168315 tid 1 2024-09-11T03:49:49.039 INFO:tasks.rados.rados.2.smithi175.stdout:3111: writing smithi17557870-4 from 1911596 to 2332965 tid 2 2024-09-11T03:49:49.046 INFO:tasks.rados.rados.2.smithi175.stdout:3111: writing smithi17557870-4 from 3087644 to 3490963 tid 3 2024-09-11T03:49:49.046 INFO:tasks.rados.rados.2.smithi175.stdout:3112: rollback oid 22 current snap is 344 2024-09-11T03:49:49.046 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 22 to 311 2024-09-11T03:49:49.046 INFO:tasks.rados.rados.2.smithi175.stdout:3110: finishing rollback tid 1 to smithi17557870-14 2024-09-11T03:49:49.047 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 965 (ObjNum 1745486336 snap 22076 seq_num 304) dirty dne 2024-09-11T03:49:49.047 INFO:tasks.rados.rados.2.smithi175.stdout:3108: finishing copy_from racing read to smithi17557870-25 2024-09-11T03:49:49.047 INFO:tasks.rados.rados.2.smithi175.stdout:3111: finishing write tid 1 to smithi17557870-4 2024-09-11T03:49:49.047 INFO:tasks.rados.rados.2.smithi175.stdout:3108: finishing copy_from to smithi17557870-25 2024-09-11T03:49:49.047 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 975 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:49:49.047 INFO:tasks.rados.rados.2.smithi175.stdout:3111: finishing write tid 2 to smithi17557870-4 2024-09-11T03:49:49.047 INFO:tasks.rados.rados.2.smithi175.stdout:3108: done (4 left) 2024-09-11T03:49:49.047 INFO:tasks.rados.rados.2.smithi175.stdout:3109: done (3 left) 2024-09-11T03:49:49.047 INFO:tasks.rados.rados.2.smithi175.stdout:3110: done (2 left) 2024-09-11T03:49:49.047 INFO:tasks.rados.rados.2.smithi175.stdout:3113: snap_create 2024-09-11T03:49:49.049 INFO:tasks.rados.rados.2.smithi175.stdout:3112: finishing rollback tid 0 to smithi17557870-22 2024-09-11T03:49:49.049 INFO:tasks.rados.rados.2.smithi175.stdout:3112: finishing rollback tid 1 to smithi17557870-22 2024-09-11T03:49:49.049 INFO:tasks.rados.rados.2.smithi175.stdout:3111: finishing write tid 3 to smithi17557870-4 2024-09-11T03:49:49.049 INFO:tasks.rados.rados.2.smithi175.stdout:3112: finishing rollback tid 2 to smithi17557870-22 2024-09-11T03:49:49.049 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 1196 (ObjNum 829 snap 263 seq_num 829) dirty exists 2024-09-11T03:49:49.049 INFO:tasks.rados.rados.2.smithi175.stdout:3111: finishing write tid 4 to smithi17557870-4 2024-09-11T03:49:49.050 INFO:tasks.rados.rados.2.smithi175.stdout:3111: finishing write tid 5 to smithi17557870-4 2024-09-11T03:49:49.050 INFO:tasks.rados.rados.2.smithi175.stdout:3111: oid 4 updating version 0 to 981 2024-09-11T03:49:49.050 INFO:tasks.rados.rados.2.smithi175.stdout:3111: oid 4 updating version 981 to 982 2024-09-11T03:49:49.050 INFO:tasks.rados.rados.2.smithi175.stdout:3111: oid 4 version 982 is already newer than 980 2024-09-11T03:49:49.050 INFO:tasks.rados.rados.2.smithi175.stdout:3111: oid 4 updating version 982 to 983 2024-09-11T03:49:49.050 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 983 (ObjNum 1058 snap 344 seq_num 1058) dirty exists 2024-09-11T03:49:49.050 INFO:tasks.rados.rados.2.smithi175.stdout:3111: left oid 4 (ObjNum 1058 snap 344 seq_num 1058) 2024-09-11T03:49:49.051 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:49.076 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:49.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:49.077 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:49.178 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:49.198 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:49.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:49.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:49.300 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:49.324 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:49.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:49 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2500042571' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:49:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:49 smithi084 ceph-mon[27254]: osdmap e771: 12 total, 12 up, 12 in 2024-09-11T03:49:49.424 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:49.448 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:49.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:49.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:49.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:49.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:49.573 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:49.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:49.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:49 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:49.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2500042571' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:49:49.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:49 smithi121 ceph-mon[30729]: osdmap e771: 12 total, 12 up, 12 in 2024-09-11T03:49:49.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:49.696 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:49.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:49.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:49.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:49 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:49.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2500042571' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:49:49.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:49 smithi175 ceph-mon[30894]: osdmap e771: 12 total, 12 up, 12 in 2024-09-11T03:49:49.797 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:49.824 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:49.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:49.825 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:49.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:49.952 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:49.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:50.027 INFO:tasks.rados.rados.2.smithi175.stdout:3111: done (2 left) 2024-09-11T03:49:50.027 INFO:tasks.rados.rados.2.smithi175.stdout:3112: done (1 left) 2024-09-11T03:49:50.027 INFO:tasks.rados.rados.2.smithi175.stdout:3113: done (0 left) 2024-09-11T03:49:50.027 INFO:tasks.rados.rados.2.smithi175.stdout:3114: rollback oid 37 current snap is 345 2024-09-11T03:49:50.027 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 37 to 339 2024-09-11T03:49:50.027 INFO:tasks.rados.rados.2.smithi175.stdout:3115: read oid 31 snap 337 2024-09-11T03:49:50.027 INFO:tasks.rados.rados.2.smithi175.stdout:3115: expect deleted 2024-09-11T03:49:50.028 INFO:tasks.rados.rados.2.smithi175.stdout:3116: read oid 24 snap 321 2024-09-11T03:49:50.028 INFO:tasks.rados.rados.2.smithi175.stdout:3116: expect deleted 2024-09-11T03:49:50.028 INFO:tasks.rados.rados.2.smithi175.stdout:3117: write (excl) oid 32 current snap is 345 2024-09-11T03:49:50.028 INFO:tasks.rados.rados.2.smithi175.stdout:3117: seq_num 1059 ranges {598613=661758,1970078=565027,3115067=614591} 2024-09-11T03:49:50.033 INFO:tasks.rados.rados.2.smithi175.stdout:3117: writing smithi17557870-32 from 598613 to 1260371 tid 1 2024-09-11T03:49:50.041 INFO:tasks.rados.rados.2.smithi175.stdout:3117: writing smithi17557870-32 from 1970078 to 2535105 tid 2 2024-09-11T03:49:50.047 INFO:tasks.rados.rados.2.smithi175.stdout:3117: writing smithi17557870-32 from 3115067 to 3729658 tid 3 2024-09-11T03:49:50.048 INFO:tasks.rados.rados.2.smithi175.stdout:3118: read oid 7 snap 335 2024-09-11T03:49:50.048 INFO:tasks.rados.rados.2.smithi175.stdout:3118: expect deleted 2024-09-11T03:49:50.048 INFO:tasks.rados.rados.2.smithi175.stdout:3119: read oid 5 snap -1 2024-09-11T03:49:50.048 INFO:tasks.rados.rados.2.smithi175.stdout:3119: expect (ObjNum 1043 snap 340 seq_num 1043) 2024-09-11T03:49:50.048 INFO:tasks.rados.rados.2.smithi175.stdout:3120: write oid 6 current snap is 345 2024-09-11T03:49:50.048 INFO:tasks.rados.rados.2.smithi175.stdout:3120: seq_num 1060 ranges {615420=737007,1912207=425038,3026137=542216} 2024-09-11T03:49:50.053 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:50.055 INFO:tasks.rados.rados.2.smithi175.stdout:3120: writing smithi17557870-6 from 615420 to 1352427 tid 1 2024-09-11T03:49:50.060 INFO:tasks.rados.rados.2.smithi175.stdout:3120: writing smithi17557870-6 from 1912207 to 2337245 tid 2 2024-09-11T03:49:50.067 INFO:tasks.rados.rados.2.smithi175.stdout:3120: writing smithi17557870-6 from 3026137 to 3568353 tid 3 2024-09-11T03:49:50.067 INFO:tasks.rados.rados.2.smithi175.stdout:3121: read oid 19 snap -1 2024-09-11T03:49:50.068 INFO:tasks.rados.rados.2.smithi175.stdout:3121: expect (ObjNum 963 snap 310 seq_num 963) 2024-09-11T03:49:50.068 INFO:tasks.rados.rados.2.smithi175.stdout:3122: rollback oid 46 current snap is 345 2024-09-11T03:49:50.068 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 46 to 331 2024-09-11T03:49:50.068 INFO:tasks.rados.rados.2.smithi175.stdout:3114: finishing rollback tid 1 to smithi17557870-37 2024-09-11T03:49:50.068 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 904 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-11T03:49:50.068 INFO:tasks.rados.rados.2.smithi175.stdout:3117: finishing write tid 1 to smithi17557870-32 2024-09-11T03:49:50.068 INFO:tasks.rados.rados.2.smithi175.stdout:3117: finishing write tid 2 to smithi17557870-32 2024-09-11T03:49:50.068 INFO:tasks.rados.rados.2.smithi175.stdout:3117: finishing write tid 3 to smithi17557870-32 2024-09-11T03:49:50.068 INFO:tasks.rados.rados.2.smithi175.stdout:3117: finishing write tid 4 to smithi17557870-32 2024-09-11T03:49:50.068 INFO:tasks.rados.rados.2.smithi175.stdout:3117: finishing write tid 5 to smithi17557870-32 2024-09-11T03:49:50.068 INFO:tasks.rados.rados.2.smithi175.stdout:3117: oid 32 updating version 0 to 987 2024-09-11T03:49:50.068 INFO:tasks.rados.rados.2.smithi175.stdout:3117: oid 32 updating version 987 to 988 2024-09-11T03:49:50.068 INFO:tasks.rados.rados.2.smithi175.stdout:3117: oid 32 version 988 is already newer than 986 2024-09-11T03:49:50.069 INFO:tasks.rados.rados.2.smithi175.stdout:3117: oid 32 updating version 988 to 989 2024-09-11T03:49:50.069 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 989 (ObjNum 1059 snap 345 seq_num 1059) dirty exists 2024-09-11T03:49:50.069 INFO:tasks.rados.rados.2.smithi175.stdout:3117: left oid 32 (ObjNum 1059 snap 345 seq_num 1059) 2024-09-11T03:49:50.069 INFO:tasks.rados.rados.2.smithi175.stdout:3119: expect (ObjNum 1043 snap 340 seq_num 1043) 2024-09-11T03:49:50.076 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:50.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:50.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:50.178 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:50.199 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:50.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:50.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:50.280 INFO:tasks.rados.rados.2.smithi175.stdout:3120: finishing write tid 1 to smithi17557870-6 2024-09-11T03:49:50.280 INFO:tasks.rados.rados.2.smithi175.stdout:3120: finishing write tid 2 to smithi17557870-6 2024-09-11T03:49:50.281 INFO:tasks.rados.rados.2.smithi175.stdout:3122: finishing rollback tid 0 to smithi17557870-46 2024-09-11T03:49:50.281 INFO:tasks.rados.rados.2.smithi175.stdout:3122: finishing rollback tid 1 to smithi17557870-46 2024-09-11T03:49:50.281 INFO:tasks.rados.rados.2.smithi175.stdout:3122: finishing rollback tid 2 to smithi17557870-46 2024-09-11T03:49:50.281 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 965 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:49:50.281 INFO:tasks.rados.rados.2.smithi175.stdout:3120: finishing write tid 3 to smithi17557870-6 2024-09-11T03:49:50.282 INFO:tasks.rados.rados.2.smithi175.stdout:3120: finishing write tid 4 to smithi17557870-6 2024-09-11T03:49:50.282 INFO:tasks.rados.rados.2.smithi175.stdout:3120: finishing write tid 5 to smithi17557870-6 2024-09-11T03:49:50.282 INFO:tasks.rados.rados.2.smithi175.stdout:3120: oid 6 updating version 0 to 981 2024-09-11T03:49:50.282 INFO:tasks.rados.rados.2.smithi175.stdout:3120: oid 6 version 981 is already newer than 979 2024-09-11T03:49:50.282 INFO:tasks.rados.rados.2.smithi175.stdout:3120: oid 6 version 981 is already newer than 980 2024-09-11T03:49:50.282 INFO:tasks.rados.rados.2.smithi175.stdout:3120: oid 6 version 981 is already newer than 978 2024-09-11T03:49:50.283 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 981 (ObjNum 1060 snap 345 seq_num 1060) dirty exists 2024-09-11T03:49:50.283 INFO:tasks.rados.rados.2.smithi175.stdout:3120: left oid 6 (ObjNum 1060 snap 345 seq_num 1060) 2024-09-11T03:49:50.283 INFO:tasks.rados.rados.2.smithi175.stdout:3121: expect (ObjNum 963 snap 310 seq_num 963) 2024-09-11T03:49:50.302 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:50.323 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:50.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:50.384 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:49:50.385 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:49:50.395 INFO:tasks.rados.rados.2.smithi175.stdout:3114: done (8 left) 2024-09-11T03:49:50.395 INFO:tasks.rados.rados.2.smithi175.stdout:3115: done (7 left) 2024-09-11T03:49:50.395 INFO:tasks.rados.rados.2.smithi175.stdout:3116: done (6 left) 2024-09-11T03:49:50.396 INFO:tasks.rados.rados.2.smithi175.stdout:3117: done (5 left) 2024-09-11T03:49:50.396 INFO:tasks.rados.rados.2.smithi175.stdout:3118: done (4 left) 2024-09-11T03:49:50.396 INFO:tasks.rados.rados.2.smithi175.stdout:3119: done (3 left) 2024-09-11T03:49:50.396 INFO:tasks.rados.rados.2.smithi175.stdout:3120: done (2 left) 2024-09-11T03:49:50.396 INFO:tasks.rados.rados.2.smithi175.stdout:3121: done (1 left) 2024-09-11T03:49:50.396 INFO:tasks.rados.rados.2.smithi175.stdout:3122: done (0 left) 2024-09-11T03:49:50.396 INFO:tasks.rados.rados.2.smithi175.stdout:3123: snap_create 2024-09-11T03:49:50.400 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:49:50.400 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:49:50.416 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:49:50.416 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:49:50.425 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:50.432 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:49:50.432 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:49:50.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:50 smithi084 ceph-mon[27254]: pgmap v1311: 77 pgs: 77 active+clean; 406 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 6.2 MiB/s wr, 30 op/s 2024-09-11T03:49:50.433 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:50 smithi084 ceph-mon[27254]: osdmap e772: 12 total, 12 up, 12 in 2024-09-11T03:49:50.446 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:50.447 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:50.449 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:49:50.450 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:49:50.467 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:49:50.467 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:49:50.484 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:49:50.484 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:49:50.500 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:49:50.501 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:49:50.517 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:49:50.517 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:49:50.535 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:49:50.535 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:49:50.548 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:50.552 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:49:50.552 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:49:50.569 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:49:50.569 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:49:50.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:50.570 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:50.570 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:50.620 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:49:50.621 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:49:50.670 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:50.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:50 smithi121 ceph-mon[30729]: pgmap v1311: 77 pgs: 77 active+clean; 406 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 6.2 MiB/s wr, 30 op/s 2024-09-11T03:49:50.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:50 smithi121 ceph-mon[30729]: osdmap e772: 12 total, 12 up, 12 in 2024-09-11T03:49:50.692 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:50.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:50.693 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:50.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:50 smithi175 ceph-mon[30894]: pgmap v1311: 77 pgs: 77 active+clean; 406 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 6.2 MiB/s wr, 30 op/s 2024-09-11T03:49:50.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:50 smithi175 ceph-mon[30894]: osdmap e772: 12 total, 12 up, 12 in 2024-09-11T03:49:50.794 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:50.815 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:50.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:50.816 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:50.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:50.943 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:50.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:50.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:51.045 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:51.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:51.068 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:51.068 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:51.168 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:51.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:51.191 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:51.192 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:51.265 INFO:tasks.rados.rados.2.smithi175.stdout:3123: done (0 left) 2024-09-11T03:49:51.265 INFO:tasks.rados.rados.2.smithi175.stdout:3124: write oid 7 current snap is 346 2024-09-11T03:49:51.265 INFO:tasks.rados.rados.2.smithi175.stdout:3124: seq_num 1061 ranges {632227=412256,1570689=568696,2653560=753488} 2024-09-11T03:49:51.270 INFO:tasks.rados.rados.2.smithi175.stdout:3124: writing smithi17557870-7 from 632227 to 1044483 tid 1 2024-09-11T03:49:51.276 INFO:tasks.rados.rados.2.smithi175.stdout:3124: writing smithi17557870-7 from 1570689 to 2139385 tid 2 2024-09-11T03:49:51.284 INFO:tasks.rados.rados.2.smithi175.stdout:3124: writing smithi17557870-7 from 2653560 to 3407048 tid 3 2024-09-11T03:49:51.284 INFO:tasks.rados.rados.2.smithi175.stdout:3125: read oid 41 snap 345 2024-09-11T03:49:51.284 INFO:tasks.rados.rados.2.smithi175.stdout:3125: expect deleted 2024-09-11T03:49:51.284 INFO:tasks.rados.rados.2.smithi175.stdout:3124: finishing write tid 1 to smithi17557870-7 2024-09-11T03:49:51.284 INFO:tasks.rados.rados.2.smithi175.stdout:3124: finishing write tid 2 to smithi17557870-7 2024-09-11T03:49:51.284 INFO:tasks.rados.rados.2.smithi175.stdout:3126: read oid 23 snap -1 2024-09-11T03:49:51.284 INFO:tasks.rados.rados.2.smithi175.stdout:3126: expect (ObjNum 1022 snap 334 seq_num 1022) 2024-09-11T03:49:51.284 INFO:tasks.rados.rados.2.smithi175.stdout:3127: snap_create 2024-09-11T03:49:51.288 INFO:tasks.rados.rados.2.smithi175.stdout:3124: finishing write tid 3 to smithi17557870-7 2024-09-11T03:49:51.288 INFO:tasks.rados.rados.2.smithi175.stdout:3124: finishing write tid 4 to smithi17557870-7 2024-09-11T03:49:51.288 INFO:tasks.rados.rados.2.smithi175.stdout:3124: finishing write tid 5 to smithi17557870-7 2024-09-11T03:49:51.288 INFO:tasks.rados.rados.2.smithi175.stdout:3124: oid 7 updating version 0 to 988 2024-09-11T03:49:51.288 INFO:tasks.rados.rados.2.smithi175.stdout:3124: oid 7 updating version 988 to 989 2024-09-11T03:49:51.288 INFO:tasks.rados.rados.2.smithi175.stdout:3124: oid 7 version 989 is already newer than 986 2024-09-11T03:49:51.288 INFO:tasks.rados.rados.2.smithi175.stdout:3124: oid 7 version 989 is already newer than 987 2024-09-11T03:49:51.288 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 989 (ObjNum 1061 snap 346 seq_num 1061) dirty exists 2024-09-11T03:49:51.289 INFO:tasks.rados.rados.2.smithi175.stdout:3124: left oid 7 (ObjNum 1061 snap 346 seq_num 1061) 2024-09-11T03:49:51.291 INFO:tasks.rados.rados.2.smithi175.stdout:3126: expect (ObjNum 1022 snap 334 seq_num 1022) 2024-09-11T03:49:51.293 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:51.316 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:51.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:51.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:51.418 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:51.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:51.440 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:51.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:51.541 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:51.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:51.563 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:51.563 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:51.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1743844618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:51.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:51.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1743844618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:51.691 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:51.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:51.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:51.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1743844618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:51.793 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:51.817 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:51.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:51.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:51.919 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:51.940 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:51.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:51.941 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:51.948 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:49:52.008 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T03:49:52.025 INFO:tasks.rados.rados.2.smithi175.stdout:3124: done (3 left) 2024-09-11T03:49:52.025 INFO:tasks.rados.rados.2.smithi175.stdout:3125: done (2 left) 2024-09-11T03:49:52.025 INFO:tasks.rados.rados.2.smithi175.stdout:3126: done (1 left) 2024-09-11T03:49:52.025 INFO:tasks.rados.rados.2.smithi175.stdout:3127: done (0 left) 2024-09-11T03:49:52.025 INFO:tasks.rados.rados.2.smithi175.stdout:3128: write (excl) oid 42 current snap is 347 2024-09-11T03:49:52.025 INFO:tasks.rados.rados.2.smithi175.stdout:3128: seq_num 1062 ranges {649034=487505,1629171=428707,2680983=564760,3798768=50266} 2024-09-11T03:49:52.030 INFO:tasks.rados.rados.2.smithi175.stdout:3128: writing smithi17557870-42 from 649034 to 1136539 tid 1 2024-09-11T03:49:52.035 INFO:tasks.rados.rados.2.smithi175.stdout:3128: writing smithi17557870-42 from 1629171 to 2057878 tid 2 2024-09-11T03:49:52.042 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:52.042 INFO:tasks.rados.rados.2.smithi175.stdout:3128: writing smithi17557870-42 from 2680983 to 3245743 tid 3 2024-09-11T03:49:52.046 INFO:tasks.rados.rados.2.smithi175.stdout:3128: writing smithi17557870-42 from 3798768 to 3849034 tid 4 2024-09-11T03:49:52.046 INFO:tasks.rados.rados.2.smithi175.stdout:3129: read oid 34 snap -1 2024-09-11T03:49:52.046 INFO:tasks.rados.rados.2.smithi175.stdout:3129: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:49:52.046 INFO:tasks.rados.rados.2.smithi175.stdout:3130: write oid 5 current snap is 347 2024-09-11T03:49:52.046 INFO:tasks.rados.rados.2.smithi175.stdout:3128: finishing write tid 1 to smithi17557870-42 2024-09-11T03:49:52.046 INFO:tasks.rados.rados.2.smithi175.stdout:3128: finishing write tid 2 to smithi17557870-42 2024-09-11T03:49:52.046 INFO:tasks.rados.rados.2.smithi175.stdout:3128: finishing write tid 3 to smithi17557870-42 2024-09-11T03:49:52.046 INFO:tasks.rados.rados.2.smithi175.stdout:3130: seq_num 1063 ranges {665841=562754,1687653=572365,2708406=776032} 2024-09-11T03:49:52.053 INFO:tasks.rados.rados.2.smithi175.stdout:3130: writing smithi17557870-5 from 665841 to 1228595 tid 1 2024-09-11T03:49:52.058 INFO:tasks.rados.rados.2.smithi175.stdout:3130: writing smithi17557870-5 from 1687653 to 2260018 tid 2 2024-09-11T03:49:52.066 INFO:tasks.rados.rados.2.smithi175.stdout:3130: writing smithi17557870-5 from 2708406 to 3484438 tid 3 2024-09-11T03:49:52.066 INFO:tasks.rados.rados.2.smithi175.stdout:3131: read oid 32 snap -1 2024-09-11T03:49:52.066 INFO:tasks.rados.rados.2.smithi175.stdout:3131: expect (ObjNum 1059 snap 345 seq_num 1059) 2024-09-11T03:49:52.066 INFO:tasks.rados.rados.2.smithi175.stdout:3132: rollback oid 16 current snap is 347 2024-09-11T03:49:52.066 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 16 to 340 2024-09-11T03:49:52.066 INFO:tasks.rados.rados.2.smithi175.stdout:3128: finishing write tid 4 to smithi17557870-42 2024-09-11T03:49:52.066 INFO:tasks.rados.rados.2.smithi175.stdout:3128: finishing write tid 5 to smithi17557870-42 2024-09-11T03:49:52.066 INFO:tasks.rados.rados.2.smithi175.stdout:3128: finishing write tid 6 to smithi17557870-42 2024-09-11T03:49:52.066 INFO:tasks.rados.rados.2.smithi175.stdout:3128: oid 42 updating version 0 to 971 2024-09-11T03:49:52.066 INFO:tasks.rados.rados.2.smithi175.stdout:3128: oid 42 updating version 971 to 972 2024-09-11T03:49:52.067 INFO:tasks.rados.rados.2.smithi175.stdout:3128: oid 42 updating version 972 to 974 2024-09-11T03:49:52.067 INFO:tasks.rados.rados.2.smithi175.stdout:3128: oid 42 version 974 is already newer than 973 2024-09-11T03:49:52.067 INFO:tasks.rados.rados.2.smithi175.stdout:3128: oid 42 version 974 is already newer than 970 2024-09-11T03:49:52.067 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 974 (ObjNum 1062 snap 347 seq_num 1062) dirty exists 2024-09-11T03:49:52.067 INFO:tasks.rados.rados.2.smithi175.stdout:3128: left oid 42 (ObjNum 1062 snap 347 seq_num 1062) 2024-09-11T03:49:52.067 INFO:tasks.rados.rados.2.smithi175.stdout:3130: finishing write tid 1 to smithi17557870-5 2024-09-11T03:49:52.067 INFO:tasks.rados.rados.2.smithi175.stdout:3129: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:49:52.075 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:52.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:52.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:52.177 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:52.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:52.200 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:52.200 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:52.219 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:52.302 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:52.304 INFO:tasks.rados.rados.2.smithi175.stdout:3130: finishing write tid 2 to smithi17557870-5 2024-09-11T03:49:52.304 INFO:tasks.rados.rados.2.smithi175.stdout:3130: finishing write tid 3 to smithi17557870-5 2024-09-11T03:49:52.304 INFO:tasks.rados.rados.2.smithi175.stdout:3130: finishing write tid 4 to smithi17557870-5 2024-09-11T03:49:52.304 INFO:tasks.rados.rados.2.smithi175.stdout:3130: finishing write tid 5 to smithi17557870-5 2024-09-11T03:49:52.304 INFO:tasks.rados.rados.2.smithi175.stdout:3130: oid 5 updating version 0 to 1028 2024-09-11T03:49:52.304 INFO:tasks.rados.rados.2.smithi175.stdout:3130: oid 5 updating version 1028 to 1029 2024-09-11T03:49:52.305 INFO:tasks.rados.rados.2.smithi175.stdout:3130: oid 5 updating version 1029 to 1030 2024-09-11T03:49:52.305 INFO:tasks.rados.rados.2.smithi175.stdout:3130: oid 5 updating version 1030 to 1031 2024-09-11T03:49:52.305 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 1031 (ObjNum 1063 snap 347 seq_num 1063) dirty exists 2024-09-11T03:49:52.305 INFO:tasks.rados.rados.2.smithi175.stdout:3130: left oid 5 (ObjNum 1063 snap 347 seq_num 1063) 2024-09-11T03:49:52.305 INFO:tasks.rados.rados.2.smithi175.stdout:3132: finishing rollback tid 0 to smithi17557870-16 2024-09-11T03:49:52.305 INFO:tasks.rados.rados.2.smithi175.stdout:3132: finishing rollback tid 1 to smithi17557870-16 2024-09-11T03:49:52.305 INFO:tasks.rados.rados.2.smithi175.stdout:3128: done (4 left) 2024-09-11T03:49:52.305 INFO:tasks.rados.rados.2.smithi175.stdout:3129: done (3 left) 2024-09-11T03:49:52.305 INFO:tasks.rados.rados.2.smithi175.stdout:3130: done (2 left) 2024-09-11T03:49:52.305 INFO:tasks.rados.rados.2.smithi175.stdout:3133: delete oid 38 current snap is 347 2024-09-11T03:49:52.306 INFO:tasks.rados.rados.2.smithi175.stdout:3132: finishing rollback tid 2 to smithi17557870-16 2024-09-11T03:49:52.306 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 986 (ObjNum 1024 snap 337 seq_num 1024) dirty exists 2024-09-11T03:49:52.306 INFO:tasks.rados.rados.2.smithi175.stdout:3131: expect (ObjNum 1059 snap 345 seq_num 1059) 2024-09-11T03:49:52.326 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:52.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:52.327 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:52.427 INFO:tasks.rados.rados.2.smithi175.stdout:3131: done (2 left) 2024-09-11T03:49:52.427 INFO:tasks.rados.rados.2.smithi175.stdout:3132: done (1 left) 2024-09-11T03:49:52.427 INFO:tasks.rados.rados.2.smithi175.stdout:3133: done (0 left) 2024-09-11T03:49:52.427 INFO:tasks.rados.rados.2.smithi175.stdout:3134: rollback oid 24 current snap is 347 2024-09-11T03:49:52.427 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 341 2024-09-11T03:49:52.427 INFO:tasks.rados.rados.2.smithi175.stdout:3135: rollback oid 8 current snap is 347 2024-09-11T03:49:52.427 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 8 to 321 2024-09-11T03:49:52.428 INFO:tasks.rados.rados.2.smithi175.stdout:3136: snap_create 2024-09-11T03:49:52.428 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:52.429 INFO:tasks.rados.rados.2.smithi175.stdout:3135: finishing rollback tid 0 to smithi17557870-8 2024-09-11T03:49:52.429 INFO:tasks.rados.rados.2.smithi175.stdout:3135: finishing rollback tid 1 to smithi17557870-8 2024-09-11T03:49:52.429 INFO:tasks.rados.rados.2.smithi175.stdout:3135: finishing rollback tid 2 to smithi17557870-8 2024-09-11T03:49:52.430 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 1033 (ObjNum 963 snap 310 seq_num 963) dirty exists 2024-09-11T03:49:52.431 INFO:tasks.rados.rados.2.smithi175.stdout:3134: finishing rollback tid 0 to smithi17557870-24 2024-09-11T03:49:52.431 INFO:tasks.rados.rados.2.smithi175.stdout:3134: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:49:52.431 INFO:tasks.rados.rados.2.smithi175.stdout:3134: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:49:52.431 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 1028 (ObjNum 975 snap 313 seq_num 975) dirty exists 2024-09-11T03:49:52.449 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:52.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:52.509 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:52 smithi084 ceph-mon[27254]: pgmap v1314: 77 pgs: 77 active+clean; 406 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 3.1 MiB/s wr, 16 op/s 2024-09-11T03:49:52.510 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:52 smithi084 ceph-mon[27254]: osdmap e773: 12 total, 12 up, 12 in 2024-09-11T03:49:52.510 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:52 smithi084 ceph-mon[27254]: osdmap e774: 12 total, 12 up, 12 in 2024-09-11T03:49:52.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:52.573 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:52.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:52.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:52.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:52 smithi121 ceph-mon[30729]: pgmap v1314: 77 pgs: 77 active+clean; 406 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 3.1 MiB/s wr, 16 op/s 2024-09-11T03:49:52.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:52 smithi121 ceph-mon[30729]: osdmap e773: 12 total, 12 up, 12 in 2024-09-11T03:49:52.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:52 smithi121 ceph-mon[30729]: osdmap e774: 12 total, 12 up, 12 in 2024-09-11T03:49:52.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:52.695 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:52.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:52.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:52.726 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:52.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:52 smithi175 ceph-mon[30894]: pgmap v1314: 77 pgs: 77 active+clean; 406 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 3.1 MiB/s wr, 16 op/s 2024-09-11T03:49:52.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:52 smithi175 ceph-mon[30894]: osdmap e773: 12 total, 12 up, 12 in 2024-09-11T03:49:52.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:52 smithi175 ceph-mon[30894]: osdmap e774: 12 total, 12 up, 12 in 2024-09-11T03:49:52.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:52.826 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:52.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:52.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:52.953 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:52.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:52.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:53.055 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:53.077 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:53.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:53.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:53.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:53.191 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:49:53.191 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:49:53.201 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:53.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:53.265 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:49:53.272 INFO:tasks.rados.rados.2.smithi175.stdout:3134: done (2 left) 2024-09-11T03:49:53.272 INFO:tasks.rados.rados.2.smithi175.stdout:3135: done (1 left) 2024-09-11T03:49:53.272 INFO:tasks.rados.rados.2.smithi175.stdout:3136: done (0 left) 2024-09-11T03:49:53.272 INFO:tasks.rados.rados.2.smithi175.stdout:3137: copy_from oid 36 from oid 37 current snap is 348 2024-09-11T03:49:53.272 INFO:tasks.rados.rados.2.smithi175.stdout:3138: write (excl) oid 35 current snap is 348 2024-09-11T03:49:53.272 INFO:tasks.rados.rados.2.smithi175.stdout:3138: seq_num 1065 ranges {699455=429605,1804617=576034,3163252=514929} 2024-09-11T03:49:53.277 INFO:tasks.rados.rados.2.smithi175.stdout:3138: writing smithi17557870-35 from 699455 to 1129060 tid 1 2024-09-11T03:49:53.284 INFO:tasks.rados.rados.2.smithi175.stdout:3138: writing smithi17557870-35 from 1804617 to 2380651 tid 2 2024-09-11T03:49:53.292 INFO:tasks.rados.rados.2.smithi175.stdout:3138: writing smithi17557870-35 from 3163252 to 3678181 tid 3 2024-09-11T03:49:53.292 INFO:tasks.rados.rados.2.smithi175.stdout:3139: copy_from oid 16 from oid 4 current snap is 348 2024-09-11T03:49:53.293 INFO:tasks.rados.rados.2.smithi175.stdout:3137: finishing copy_from to smithi17557870-36 2024-09-11T03:49:53.293 INFO:tasks.rados.rados.2.smithi175.stdout:3137: got expected ENOENT (src dne) 2024-09-11T03:49:53.293 INFO:tasks.rados.rados.2.smithi175.stdout:3137: finishing copy_from racing read to smithi17557870-36 2024-09-11T03:49:53.293 INFO:tasks.rados.rados.2.smithi175.stdout:3138: finishing write tid 1 to smithi17557870-35 2024-09-11T03:49:53.293 INFO:tasks.rados.rados.2.smithi175.stdout:3138: finishing write tid 2 to smithi17557870-35 2024-09-11T03:49:53.293 INFO:tasks.rados.rados.2.smithi175.stdout:3137: done (2 left) 2024-09-11T03:49:53.293 INFO:tasks.rados.rados.2.smithi175.stdout:3140: read oid 22 snap -1 2024-09-11T03:49:53.293 INFO:tasks.rados.rados.2.smithi175.stdout:3140: expect (ObjNum 829 snap 263 seq_num 829) 2024-09-11T03:49:53.293 INFO:tasks.rados.rados.2.smithi175.stdout:3141: copy_from oid 24 from oid 11 current snap is 348 2024-09-11T03:49:53.293 INFO:tasks.rados.rados.2.smithi175.stdout:3142: copy_from oid 42 from oid 20 current snap is 348 2024-09-11T03:49:53.293 INFO:tasks.rados.rados.2.smithi175.stdout:3143: read oid 13 snap 337 2024-09-11T03:49:53.293 INFO:tasks.rados.rados.2.smithi175.stdout:3143: expect (ObjNum 907 snap 287 seq_num 907) 2024-09-11T03:49:53.294 INFO:tasks.rados.rados.2.smithi175.stdout:3144: write oid 9 current snap is 348 2024-09-11T03:49:53.294 INFO:tasks.rados.rados.2.smithi175.stdout:3144: seq_num 1066 ranges {716262=504854,1863099=719692,3074322=726201} 2024-09-11T03:49:53.299 INFO:tasks.rados.rados.2.smithi175.stdout:3144: writing smithi17557870-9 from 716262 to 1221116 tid 1 2024-09-11T03:49:53.304 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:53.307 INFO:tasks.rados.rados.2.smithi175.stdout:3144: writing smithi17557870-9 from 1863099 to 2582791 tid 2 2024-09-11T03:49:53.314 INFO:tasks.rados.rados.2.smithi175.stdout:3144: writing smithi17557870-9 from 3074322 to 3800523 tid 3 2024-09-11T03:49:53.314 INFO:tasks.rados.rados.2.smithi175.stdout:3145: delete oid 46 current snap is 348 2024-09-11T03:49:53.314 INFO:tasks.rados.rados.2.smithi175.stdout:3142: finishing copy_from to smithi17557870-42 2024-09-11T03:49:53.314 INFO:tasks.rados.rados.2.smithi175.stdout:3142: got expected ENOENT (src dne) 2024-09-11T03:49:53.314 INFO:tasks.rados.rados.2.smithi175.stdout:3142: finishing copy_from racing read to smithi17557870-42 2024-09-11T03:49:53.315 INFO:tasks.rados.rados.2.smithi175.stdout:3138: finishing write tid 3 to smithi17557870-35 2024-09-11T03:49:53.315 INFO:tasks.rados.rados.2.smithi175.stdout:3138: finishing write tid 4 to smithi17557870-35 2024-09-11T03:49:53.315 INFO:tasks.rados.rados.2.smithi175.stdout:3138: finishing write tid 5 to smithi17557870-35 2024-09-11T03:49:53.315 INFO:tasks.rados.rados.2.smithi175.stdout:3138: oid 35 updating version 0 to 953 2024-09-11T03:49:53.315 INFO:tasks.rados.rados.2.smithi175.stdout:3138: oid 35 version 953 is already newer than 951 2024-09-11T03:49:53.315 INFO:tasks.rados.rados.2.smithi175.stdout:3138: oid 35 version 953 is already newer than 950 2024-09-11T03:49:53.315 INFO:tasks.rados.rados.2.smithi175.stdout:3138: oid 35 version 953 is already newer than 952 2024-09-11T03:49:53.315 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 953 (ObjNum 1065 snap 348 seq_num 1065) dirty exists 2024-09-11T03:49:53.315 INFO:tasks.rados.rados.2.smithi175.stdout:3138: left oid 35 (ObjNum 1065 snap 348 seq_num 1065) 2024-09-11T03:49:53.315 INFO:tasks.rados.rados.2.smithi175.stdout:3143: expect (ObjNum 907 snap 287 seq_num 907) 2024-09-11T03:49:53.327 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:53.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:53.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:53.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:49:53.428 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:53.450 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:53.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:53.502 INFO:tasks.rados.rados.2.smithi175.stdout:3139: finishing copy_from racing read to smithi17557870-16 2024-09-11T03:49:53.502 INFO:tasks.rados.rados.2.smithi175.stdout:3141: finishing copy_from racing read to smithi17557870-24 2024-09-11T03:49:53.502 INFO:tasks.rados.rados.2.smithi175.stdout:3139: finishing copy_from to smithi17557870-16 2024-09-11T03:49:53.502 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 983 (ObjNum 1058 snap 344 seq_num 1058) dirty exists 2024-09-11T03:49:53.503 INFO:tasks.rados.rados.2.smithi175.stdout:3141: finishing copy_from to smithi17557870-24 2024-09-11T03:49:53.503 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 620 (ObjNum 1029 snap 338 seq_num 1029) dirty exists 2024-09-11T03:49:53.503 INFO:tasks.rados.rados.2.smithi175.stdout:3140: expect (ObjNum 829 snap 263 seq_num 829) 2024-09-11T03:49:53.552 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:53.573 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:53.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:53.574 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:53.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:53 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/734401421' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:49:53.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:53 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/734401421' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:49:53.675 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:53.685 INFO:tasks.rados.rados.2.smithi175.stdout:3144: finishing write tid 1 to smithi17557870-9 2024-09-11T03:49:53.686 INFO:tasks.rados.rados.2.smithi175.stdout:3144: finishing write tid 2 to smithi17557870-9 2024-09-11T03:49:53.686 INFO:tasks.rados.rados.2.smithi175.stdout:3138: done (7 left) 2024-09-11T03:49:53.686 INFO:tasks.rados.rados.2.smithi175.stdout:3139: done (6 left) 2024-09-11T03:49:53.686 INFO:tasks.rados.rados.2.smithi175.stdout:3140: done (5 left) 2024-09-11T03:49:53.686 INFO:tasks.rados.rados.2.smithi175.stdout:3141: done (4 left) 2024-09-11T03:49:53.686 INFO:tasks.rados.rados.2.smithi175.stdout:3142: done (3 left) 2024-09-11T03:49:53.686 INFO:tasks.rados.rados.2.smithi175.stdout:3143: done (2 left) 2024-09-11T03:49:53.686 INFO:tasks.rados.rados.2.smithi175.stdout:3145: done (1 left) 2024-09-11T03:49:53.686 INFO:tasks.rados.rados.2.smithi175.stdout:3146: read oid 20 snap -1 2024-09-11T03:49:53.686 INFO:tasks.rados.rados.2.smithi175.stdout:3146: expect deleted 2024-09-11T03:49:53.687 INFO:tasks.rados.rados.2.smithi175.stdout:3147: read oid 25 snap -1 2024-09-11T03:49:53.687 INFO:tasks.rados.rados.2.smithi175.stdout:3147: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:49:53.687 INFO:tasks.rados.rados.2.smithi175.stdout:3148: snap_create 2024-09-11T03:49:53.687 INFO:tasks.rados.rados.2.smithi175.stdout:3144: finishing write tid 3 to smithi17557870-9 2024-09-11T03:49:53.687 INFO:tasks.rados.rados.2.smithi175.stdout:3144: finishing write tid 4 to smithi17557870-9 2024-09-11T03:49:53.687 INFO:tasks.rados.rados.2.smithi175.stdout:3144: finishing write tid 5 to smithi17557870-9 2024-09-11T03:49:53.687 INFO:tasks.rados.rados.2.smithi175.stdout:3144: oid 9 updating version 0 to 1100 2024-09-11T03:49:53.687 INFO:tasks.rados.rados.2.smithi175.stdout:3144: oid 9 version 1100 is already newer than 1098 2024-09-11T03:49:53.687 INFO:tasks.rados.rados.2.smithi175.stdout:3144: oid 9 version 1100 is already newer than 1099 2024-09-11T03:49:53.687 INFO:tasks.rados.rados.2.smithi175.stdout:3144: oid 9 version 1100 is already newer than 1097 2024-09-11T03:49:53.687 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 1100 (ObjNum 1066 snap 348 seq_num 1066) dirty exists 2024-09-11T03:49:53.687 INFO:tasks.rados.rados.2.smithi175.stdout:3144: left oid 9 (ObjNum 1066 snap 348 seq_num 1066) 2024-09-11T03:49:53.696 INFO:tasks.rados.rados.2.smithi175.stdout:3147: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:49:53.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:53.697 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:53.697 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:53.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:53 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/734401421' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:49:53.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:53.821 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:49:53.822 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:49:53.822 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-11T03:49:53.822 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:49:53.824 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:53.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:53.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:53.925 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:53.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:53.947 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:53.947 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:54.048 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:54.069 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:54.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:54.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:54.171 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:54.193 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:54.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:54.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:54.275 INFO:tasks.rados.rados.2.smithi175.stdout:3144: done (3 left) 2024-09-11T03:49:54.275 INFO:tasks.rados.rados.2.smithi175.stdout:3146: done (2 left) 2024-09-11T03:49:54.275 INFO:tasks.rados.rados.2.smithi175.stdout:3147: done (1 left) 2024-09-11T03:49:54.275 INFO:tasks.rados.rados.2.smithi175.stdout:3148: done (0 left) 2024-09-11T03:49:54.275 INFO:tasks.rados.rados.2.smithi175.stdout:3149: snap_create 2024-09-11T03:49:54.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:54.321 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:54.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:54.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:54.423 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:54.445 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:54.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:54.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:54.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:54.573 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:54.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:54.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:54.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:54 smithi084 ceph-mon[27254]: pgmap v1317: 77 pgs: 77 active+clean; 428 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 3.1 MiB/s wr, 16 op/s 2024-09-11T03:49:54.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:54 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/734401421' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:49:54.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:54 smithi084 ceph-mon[27254]: osdmap e775: 12 total, 12 up, 12 in 2024-09-11T03:49:54.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:54 smithi121 ceph-mon[30729]: pgmap v1317: 77 pgs: 77 active+clean; 428 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 3.1 MiB/s wr, 16 op/s 2024-09-11T03:49:54.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:54 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/734401421' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:49:54.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:54 smithi121 ceph-mon[30729]: osdmap e775: 12 total, 12 up, 12 in 2024-09-11T03:49:54.675 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:54.698 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:54.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:54.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:54.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:54 smithi175 ceph-mon[30894]: pgmap v1317: 77 pgs: 77 active+clean; 428 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 3.1 MiB/s wr, 16 op/s 2024-09-11T03:49:54.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:54 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/734401421' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:49:54.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:54 smithi175 ceph-mon[30894]: osdmap e775: 12 total, 12 up, 12 in 2024-09-11T03:49:54.729 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T03:49:54.770 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:54.801 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:54.824 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:54.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:54.825 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:54.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:54.952 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:54.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:55.054 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:55.075 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:55.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:55.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:55.176 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:55.202 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:55.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:55.203 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:55.281 INFO:tasks.rados.rados.2.smithi175.stdout:3149: done (0 left) 2024-09-11T03:49:55.282 INFO:tasks.rados.rados.2.smithi175.stdout:3150: delete oid 24 current snap is 350 2024-09-11T03:49:55.286 INFO:tasks.rados.rados.2.smithi175.stdout:3150: done (0 left) 2024-09-11T03:49:55.286 INFO:tasks.rados.rados.2.smithi175.stdout:3151: rollback oid 19 current snap is 350 2024-09-11T03:49:55.286 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 19 to 321 2024-09-11T03:49:55.286 INFO:tasks.rados.rados.2.smithi175.stdout:3152: copy_from oid 28 from oid 33 current snap is 350 2024-09-11T03:49:55.286 INFO:tasks.rados.rados.2.smithi175.stdout:3153: copy_from oid 42 from oid 11 current snap is 350 2024-09-11T03:49:55.286 INFO:tasks.rados.rados.2.smithi175.stdout:3154: snap_remove snap 334 2024-09-11T03:49:55.303 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:55.327 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:55.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:55.328 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:55.353 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:55.356 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:55.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:55.455 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:55.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:55.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:55.506 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:49:55.507 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:49:55.557 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:55.583 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:55.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:55.584 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:55.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:55 smithi084 ceph-mon[27254]: osdmap e776: 12 total, 12 up, 12 in 2024-09-11T03:49:55.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:55 smithi121 ceph-mon[30729]: osdmap e776: 12 total, 12 up, 12 in 2024-09-11T03:49:55.685 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:55.709 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:55.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:55.710 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:55.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:55 smithi175 ceph-mon[30894]: osdmap e776: 12 total, 12 up, 12 in 2024-09-11T03:49:55.811 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:49:55.835 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:55.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:55.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:49:55.938 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:49:55.963 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:55.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:55.964 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:49:56.065 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:56.090 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:56.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:56.091 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:56.192 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:56.194 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:49:56.217 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:56.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:56.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:56.284 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:49:56.285 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:49:56.293 INFO:tasks.rados.rados.2.smithi175.stdout:3154: done (3 left) 2024-09-11T03:49:56.293 INFO:tasks.rados.rados.2.smithi175.stdout:3155: delete oid 38 current snap is 350 2024-09-11T03:49:56.293 INFO:tasks.rados.rados.2.smithi175.stdout:3151: finishing rollback tid 0 to smithi17557870-19 2024-09-11T03:49:56.293 INFO:tasks.rados.rados.2.smithi175.stdout:3151: finishing rollback tid 1 to smithi17557870-19 2024-09-11T03:49:56.293 INFO:tasks.rados.rados.2.smithi175.stdout:3151: finishing rollback tid 2 to smithi17557870-19 2024-09-11T03:49:56.293 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 992 (ObjNum 963 snap 310 seq_num 963) dirty exists 2024-09-11T03:49:56.293 INFO:tasks.rados.rados.2.smithi175.stdout:3152: finishing copy_from racing read to smithi17557870-28 2024-09-11T03:49:56.293 INFO:tasks.rados.rados.2.smithi175.stdout:3152: finishing copy_from to smithi17557870-28 2024-09-11T03:49:56.293 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 1187 (ObjNum 1037 snap 340 seq_num 1037) dirty exists 2024-09-11T03:49:56.293 INFO:tasks.rados.rados.2.smithi175.stdout:3153: finishing copy_from racing read to smithi17557870-42 2024-09-11T03:49:56.293 INFO:tasks.rados.rados.2.smithi175.stdout:3153: finishing copy_from to smithi17557870-42 2024-09-11T03:49:56.294 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 620 (ObjNum 1029 snap 338 seq_num 1029) dirty exists 2024-09-11T03:49:56.296 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:49:56.297 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:49:56.298 INFO:tasks.rados.rados.2.smithi175.stdout:3151: done (3 left) 2024-09-11T03:49:56.298 INFO:tasks.rados.rados.2.smithi175.stdout:3152: done (2 left) 2024-09-11T03:49:56.298 INFO:tasks.rados.rados.2.smithi175.stdout:3153: done (1 left) 2024-09-11T03:49:56.298 INFO:tasks.rados.rados.2.smithi175.stdout:3155: done (0 left) 2024-09-11T03:49:56.298 INFO:tasks.rados.rados.2.smithi175.stdout:3156: copy_from oid 47 from oid 7 current snap is 350 2024-09-11T03:49:56.298 INFO:tasks.rados.rados.2.smithi175.stdout:3157: read oid 16 snap 339 2024-09-11T03:49:56.298 INFO:tasks.rados.rados.2.smithi175.stdout:3157: expect (ObjNum 1024 snap 337 seq_num 1024) 2024-09-11T03:49:56.298 INFO:tasks.rados.rados.2.smithi175.stdout:3158: snap_remove snap 343 2024-09-11T03:49:56.316 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:49:56.316 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:49:56.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:56 smithi084 ceph-mon[27254]: pgmap v1320: 77 pgs: 77 active+clean; 429 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 3.1 MiB/s wr, 19 op/s 2024-09-11T03:49:56.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:56 smithi084 ceph-mon[27254]: osdmap e777: 12 total, 12 up, 12 in 2024-09-11T03:49:56.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:56 smithi084 ceph-mon[27254]: from='client.17040 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:49:56.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/223981261' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:56.319 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:56.333 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:49:56.333 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:49:56.342 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:56.343 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:56.352 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:49:56.352 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:49:56.368 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:49:56.369 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:49:56.385 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:49:56.385 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:49:56.403 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:49:56.403 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:49:56.420 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:49:56.420 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:49:56.437 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:49:56.437 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:49:56.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:49:56.454 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:49:56.454 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:49:56.466 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:56.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:56.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:49:56.471 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:49:56.471 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:49:56.544 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:49:56.544 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:49:56.544 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:49:56.544 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:49:56.544 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:49:56.567 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:56.591 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:56.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:56.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:56.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:56 smithi121 ceph-mon[30729]: pgmap v1320: 77 pgs: 77 active+clean; 429 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 3.1 MiB/s wr, 19 op/s 2024-09-11T03:49:56.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:56 smithi121 ceph-mon[30729]: osdmap e777: 12 total, 12 up, 12 in 2024-09-11T03:49:56.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:56 smithi121 ceph-mon[30729]: from='client.17040 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:49:56.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:56 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/223981261' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:56.693 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:56.717 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:56.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:56.718 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:56.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:56 smithi175 ceph-mon[30894]: pgmap v1320: 77 pgs: 77 active+clean; 429 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 3.1 MiB/s wr, 19 op/s 2024-09-11T03:49:56.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:56 smithi175 ceph-mon[30894]: osdmap e777: 12 total, 12 up, 12 in 2024-09-11T03:49:56.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:56 smithi175 ceph-mon[30894]: from='client.17040 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:49:56.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:56 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/223981261' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:49:56.819 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:56.840 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:56.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:56.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:56.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:56.964 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:56.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:56.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:57.026 INFO:tasks.rados.rados.2.smithi175.stdout:3158: done (2 left) 2024-09-11T03:49:57.026 INFO:tasks.rados.rados.2.smithi175.stdout:3159: delete oid 35 current snap is 350 2024-09-11T03:49:57.026 INFO:tasks.rados.rados.2.smithi175.stdout:3157: expect (ObjNum 1024 snap 337 seq_num 1024) 2024-09-11T03:49:57.066 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:57.088 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:57.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:57.089 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:57.135 INFO:tasks.rados.rados.2.smithi175.stdout:3156: finishing copy_from racing read to smithi17557870-47 2024-09-11T03:49:57.135 INFO:tasks.rados.rados.2.smithi175.stdout:3156: finishing copy_from to smithi17557870-47 2024-09-11T03:49:57.136 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 989 (ObjNum 1061 snap 346 seq_num 1061) dirty exists 2024-09-11T03:49:57.136 INFO:tasks.rados.rados.2.smithi175.stdout:3156: done (2 left) 2024-09-11T03:49:57.136 INFO:tasks.rados.rados.2.smithi175.stdout:3157: done (1 left) 2024-09-11T03:49:57.136 INFO:tasks.rados.rados.2.smithi175.stdout:3159: done (0 left) 2024-09-11T03:49:57.136 INFO:tasks.rados.rados.2.smithi175.stdout:3160: copy_from oid 7 from oid 36 current snap is 350 2024-09-11T03:49:57.136 INFO:tasks.rados.rados.2.smithi175.stdout:3161: write oid 39 current snap is 350 2024-09-11T03:49:57.136 INFO:tasks.rados.rados.2.smithi175.stdout:3161: seq_num 1071 ranges {400297=481099,1639156=361141} 2024-09-11T03:49:57.140 INFO:tasks.rados.rados.2.smithi175.stdout:3161: writing smithi17557870-39 from 400297 to 881396 tid 1 2024-09-11T03:49:57.147 INFO:tasks.rados.rados.2.smithi175.stdout:3161: writing smithi17557870-39 from 1639156 to 2000297 tid 2 2024-09-11T03:49:57.147 INFO:tasks.rados.rados.2.smithi175.stdout:3162: delete oid 37 current snap is 350 2024-09-11T03:49:57.147 INFO:tasks.rados.rados.2.smithi175.stdout:3160: finishing copy_from racing read to smithi17557870-7 2024-09-11T03:49:57.148 INFO:tasks.rados.rados.2.smithi175.stdout:3161: finishing write tid 1 to smithi17557870-39 2024-09-11T03:49:57.148 INFO:tasks.rados.rados.2.smithi175.stdout:3160: finishing copy_from to smithi17557870-7 2024-09-11T03:49:57.148 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 962 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:49:57.149 INFO:tasks.rados.rados.2.smithi175.stdout:3160: done (2 left) 2024-09-11T03:49:57.149 INFO:tasks.rados.rados.2.smithi175.stdout:3162: done (1 left) 2024-09-11T03:49:57.149 INFO:tasks.rados.rados.2.smithi175.stdout:3163: copy_from oid 22 from oid 17 current snap is 350 2024-09-11T03:49:57.149 INFO:tasks.rados.rados.2.smithi175.stdout:3164: write (excl) oid 41 current snap is 350 2024-09-11T03:49:57.149 INFO:tasks.rados.rados.2.smithi175.stdout:3164: seq_num 1073 ranges {433911=747950,1756120=277791} 2024-09-11T03:49:57.155 INFO:tasks.rados.rados.2.smithi175.stdout:3164: writing smithi17557870-41 from 433911 to 1181861 tid 1 2024-09-11T03:49:57.161 INFO:tasks.rados.rados.2.smithi175.stdout:3164: writing smithi17557870-41 from 1756120 to 2033911 tid 2 2024-09-11T03:49:57.161 INFO:tasks.rados.rados.2.smithi175.stdout:3165: delete oid 14 current snap is 350 2024-09-11T03:49:57.161 INFO:tasks.rados.rados.2.smithi175.stdout:3161: finishing write tid 2 to smithi17557870-39 2024-09-11T03:49:57.161 INFO:tasks.rados.rados.2.smithi175.stdout:3161: finishing write tid 3 to smithi17557870-39 2024-09-11T03:49:57.161 INFO:tasks.rados.rados.2.smithi175.stdout:3161: finishing write tid 4 to smithi17557870-39 2024-09-11T03:49:57.161 INFO:tasks.rados.rados.2.smithi175.stdout:3161: oid 39 updating version 0 to 992 2024-09-11T03:49:57.161 INFO:tasks.rados.rados.2.smithi175.stdout:3161: oid 39 version 992 is already newer than 991 2024-09-11T03:49:57.161 INFO:tasks.rados.rados.2.smithi175.stdout:3161: oid 39 version 992 is already newer than 990 2024-09-11T03:49:57.161 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 992 (ObjNum 1071 snap 350 seq_num 1071) dirty exists 2024-09-11T03:49:57.161 INFO:tasks.rados.rados.2.smithi175.stdout:3161: left oid 39 (ObjNum 1071 snap 350 seq_num 1071) 2024-09-11T03:49:57.161 INFO:tasks.rados.rados.2.smithi175.stdout:3163: finishing copy_from racing read to smithi17557870-22 2024-09-11T03:49:57.161 INFO:tasks.rados.rados.2.smithi175.stdout:3164: finishing write tid 1 to smithi17557870-41 2024-09-11T03:49:57.162 INFO:tasks.rados.rados.2.smithi175.stdout:3161: done (3 left) 2024-09-11T03:49:57.162 INFO:tasks.rados.rados.2.smithi175.stdout:3165: done (2 left) 2024-09-11T03:49:57.163 INFO:tasks.rados.rados.2.smithi175.stdout:3166: copy_from oid 2 from oid 18 current snap is 350 2024-09-11T03:49:57.163 INFO:tasks.rados.rados.2.smithi175.stdout:3167: read oid 44 snap 338 2024-09-11T03:49:57.163 INFO:tasks.rados.rados.2.smithi175.stdout:3167: expect (ObjNum 970 snap 311 seq_num 970) 2024-09-11T03:49:57.163 INFO:tasks.rados.rados.2.smithi175.stdout:3168: copy_from oid 1 from oid 40 current snap is 350 2024-09-11T03:49:57.163 INFO:tasks.rados.rados.2.smithi175.stdout:3169: write oid 10 current snap is 350 2024-09-11T03:49:57.163 INFO:tasks.rados.rados.2.smithi175.stdout:3169: seq_num 1075 ranges {467525=498448,1473084=594379} 2024-09-11T03:49:57.167 INFO:tasks.rados.rados.2.smithi175.stdout:3169: writing smithi17557870-10 from 467525 to 965973 tid 1 2024-09-11T03:49:57.174 INFO:tasks.rados.rados.2.smithi175.stdout:3169: writing smithi17557870-10 from 1473084 to 2067463 tid 2 2024-09-11T03:49:57.174 INFO:tasks.rados.rados.2.smithi175.stdout:3170: write (excl) oid 4 current snap is 350 2024-09-11T03:49:57.174 INFO:tasks.rados.rados.2.smithi175.stdout:3164: finishing write tid 2 to smithi17557870-41 2024-09-11T03:49:57.174 INFO:tasks.rados.rados.2.smithi175.stdout:3164: finishing write tid 3 to smithi17557870-41 2024-09-11T03:49:57.174 INFO:tasks.rados.rados.2.smithi175.stdout:3164: finishing write tid 4 to smithi17557870-41 2024-09-11T03:49:57.175 INFO:tasks.rados.rados.2.smithi175.stdout:3164: oid 41 updating version 0 to 993 2024-09-11T03:49:57.175 INFO:tasks.rados.rados.2.smithi175.stdout:3164: oid 41 updating version 993 to 995 2024-09-11T03:49:57.175 INFO:tasks.rados.rados.2.smithi175.stdout:3164: oid 41 version 995 is already newer than 994 2024-09-11T03:49:57.175 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 995 (ObjNum 1073 snap 350 seq_num 1073) dirty exists 2024-09-11T03:49:57.175 INFO:tasks.rados.rados.2.smithi175.stdout:3164: left oid 41 (ObjNum 1073 snap 350 seq_num 1073) 2024-09-11T03:49:57.175 INFO:tasks.rados.rados.2.smithi175.stdout:3163: finishing copy_from to smithi17557870-22 2024-09-11T03:49:57.175 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 969 (ObjNum 984 snap 315 seq_num 984) dirty exists 2024-09-11T03:49:57.175 INFO:tasks.rados.rados.2.smithi175.stdout:3168: finishing copy_from racing read to smithi17557870-1 2024-09-11T03:49:57.175 INFO:tasks.rados.rados.2.smithi175.stdout:3170: seq_num 1076 ranges {484332=573697,1531566=552766} 2024-09-11T03:49:57.179 INFO:tasks.rados.rados.2.smithi175.stdout:3170: writing smithi17557870-4 from 484332 to 1058029 tid 1 2024-09-11T03:49:57.186 INFO:tasks.rados.rados.2.smithi175.stdout:3170: writing smithi17557870-4 from 1531566 to 2084332 tid 2 2024-09-11T03:49:57.187 INFO:tasks.rados.rados.2.smithi175.stdout:3163: done (6 left) 2024-09-11T03:49:57.187 INFO:tasks.rados.rados.2.smithi175.stdout:3164: done (5 left) 2024-09-11T03:49:57.187 INFO:tasks.rados.rados.2.smithi175.stdout:3171: copy_from oid 27 from oid 9 current snap is 350 2024-09-11T03:49:57.187 INFO:tasks.rados.rados.2.smithi175.stdout:3168: finishing copy_from to smithi17557870-1 2024-09-11T03:49:57.187 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 1006 (ObjNum 973 snap 313 seq_num 973) dirty exists 2024-09-11T03:49:57.187 INFO:tasks.rados.rados.2.smithi175.stdout:3169: finishing write tid 1 to smithi17557870-10 2024-09-11T03:49:57.187 INFO:tasks.rados.rados.2.smithi175.stdout:3166: finishing copy_from racing read to smithi17557870-2 2024-09-11T03:49:57.187 INFO:tasks.rados.rados.2.smithi175.stdout:3169: finishing write tid 2 to smithi17557870-10 2024-09-11T03:49:57.187 INFO:tasks.rados.rados.2.smithi175.stdout:3169: finishing write tid 3 to smithi17557870-10 2024-09-11T03:49:57.187 INFO:tasks.rados.rados.2.smithi175.stdout:3169: finishing write tid 4 to smithi17557870-10 2024-09-11T03:49:57.187 INFO:tasks.rados.rados.2.smithi175.stdout:3169: oid 10 updating version 0 to 1045 2024-09-11T03:49:57.187 INFO:tasks.rados.rados.2.smithi175.stdout:3169: oid 10 version 1045 is already newer than 1043 2024-09-11T03:49:57.187 INFO:tasks.rados.rados.2.smithi175.stdout:3169: oid 10 version 1045 is already newer than 1044 2024-09-11T03:49:57.188 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 1045 (ObjNum 1075 snap 350 seq_num 1075) dirty exists 2024-09-11T03:49:57.188 INFO:tasks.rados.rados.2.smithi175.stdout:3169: left oid 10 (ObjNum 1075 snap 350 seq_num 1075) 2024-09-11T03:49:57.188 INFO:tasks.rados.rados.2.smithi175.stdout:3167: expect (ObjNum 970 snap 311 seq_num 970) 2024-09-11T03:49:57.190 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:49:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:57.211 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:57.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:49:57.279 INFO:tasks.rados.rados.2.smithi175.stdout:3170: finishing write tid 1 to smithi17557870-4 2024-09-11T03:49:57.279 INFO:tasks.rados.rados.2.smithi175.stdout:3166: finishing copy_from to smithi17557870-2 2024-09-11T03:49:57.279 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 930 (ObjNum 829 snap 263 seq_num 829) dirty exists 2024-09-11T03:49:57.279 INFO:tasks.rados.rados.2.smithi175.stdout:3170: finishing write tid 2 to smithi17557870-4 2024-09-11T03:49:57.279 INFO:tasks.rados.rados.2.smithi175.stdout:3170: finishing write tid 3 to smithi17557870-4 2024-09-11T03:49:57.279 INFO:tasks.rados.rados.2.smithi175.stdout:3170: finishing write tid 4 to smithi17557870-4 2024-09-11T03:49:57.279 INFO:tasks.rados.rados.2.smithi175.stdout:3170: oid 4 updating version 0 to 987 2024-09-11T03:49:57.280 INFO:tasks.rados.rados.2.smithi175.stdout:3170: oid 4 updating version 987 to 988 2024-09-11T03:49:57.280 INFO:tasks.rados.rados.2.smithi175.stdout:3170: oid 4 updating version 988 to 989 2024-09-11T03:49:57.280 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 989 (ObjNum 1076 snap 350 seq_num 1076) dirty exists 2024-09-11T03:49:57.280 INFO:tasks.rados.rados.2.smithi175.stdout:3170: left oid 4 (ObjNum 1076 snap 350 seq_num 1076) 2024-09-11T03:49:57.280 INFO:tasks.rados.rados.2.smithi175.stdout:3171: finishing copy_from racing read to smithi17557870-27 2024-09-11T03:49:57.280 INFO:tasks.rados.rados.2.smithi175.stdout:3171: finishing copy_from to smithi17557870-27 2024-09-11T03:49:57.280 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 1100 (ObjNum 1066 snap 348 seq_num 1066) dirty exists 2024-09-11T03:49:57.280 INFO:tasks.rados.rados.2.smithi175.stdout:3166: done (5 left) 2024-09-11T03:49:57.280 INFO:tasks.rados.rados.2.smithi175.stdout:3167: done (4 left) 2024-09-11T03:49:57.280 INFO:tasks.rados.rados.2.smithi175.stdout:3168: done (3 left) 2024-09-11T03:49:57.280 INFO:tasks.rados.rados.2.smithi175.stdout:3169: done (2 left) 2024-09-11T03:49:57.280 INFO:tasks.rados.rados.2.smithi175.stdout:3170: done (1 left) 2024-09-11T03:49:57.280 INFO:tasks.rados.rados.2.smithi175.stdout:3171: done (0 left) 2024-09-11T03:49:57.280 INFO:tasks.rados.rados.2.smithi175.stdout:3172: snap_create 2024-09-11T03:49:57.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:57.332 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1321,"stamp":"2024-09-11T03:49:55.278872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":450255194,"num_objects":207,"num_object_clones":155,"num_object_copies":621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":207,"num_whiteouts":12,"num_read":1364,"num_read_kb":1433521,"num_write":948,"num_write_kb":397920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":30615821,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29495,"ondisk_log_size":29495,"up":231,"acting":231,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":231,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":1418528,"kb_used_data":886036,"kb_used_omap":18,"kb_used_meta":532205,"kb_avail":1123523296,"statfs":{"total":1151940427776,"available":1150487855104,"internally_reserved":0,"allocated":907300864,"data_stored":1156693068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":544978235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22510182,"num_objects":6,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":-3,"num_read":37,"num_read_kb":39526,"num_write":32,"num_write_kb":11306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019622"},"pg_stats":[{"pgid":"2.2d","version":"769'1033","reported_seq":4463,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.077617+0000","last_change":"2024-09-11T03:49:49.033652+0000","last_active":"2024-09-11T03:49:52.077617+0000","last_peered":"2024-09-11T03:49:52.077617+0000","last_clean":"2024-09-11T03:49:52.077617+0000","last_became_active":"2024-09-11T03:49:29.163201+0000","last_became_peered":"2024-09-11T03:49:29.163201+0000","last_unstale":"2024-09-11T03:49:52.077617+0000","last_undegraded":"2024-09-11T03:49:52.077617+0000","last_fullsized":"2024-09-11T03:49:52.077617+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'997","last_scrub_stamp":"2024-09-11T03:49:32.462332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.462332+0000","objects_scrubbed":13,"log_size":1033,"log_dups_size":0,"ondisk_log_size":1033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:46.080849+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0026518449999999999,"stat_sum":{"num_bytes":31576569,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":10,"num_read_kb":10392,"num_write":2,"num_write_kb":5894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"2.2c","version":"771'820","reported_seq":4243,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.265003+0000","last_change":"2024-09-11T03:49:49.027801+0000","last_active":"2024-09-11T03:49:51.265003+0000","last_peered":"2024-09-11T03:49:51.265003+0000","last_clean":"2024-09-11T03:49:51.265003+0000","last_became_active":"2024-09-11T03:49:29.154872+0000","last_became_peered":"2024-09-11T03:49:29.154872+0000","last_unstale":"2024-09-11T03:49:51.265003+0000","last_undegraded":"2024-09-11T03:49:51.265003+0000","last_fullsized":"2024-09-11T03:49:51.265003+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"753'800","last_scrub_stamp":"2024-09-11T03:49:32.185401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.185401+0000","objects_scrubbed":4,"log_size":820,"log_dups_size":0,"ondisk_log_size":820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:45.609050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040774790000000002,"stat_sum":{"num_bytes":14070124,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":10886,"num_write":10,"num_write_kb":3361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3],"acting":[8,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"2.2b","version":"568'646","reported_seq":3211,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-09-11T03:49:53.274278+0000","last_change":"2024-09-11T03:49:49.127046+0000","last_active":"2024-09-11T03:49:53.274278+0000","last_peered":"2024-09-11T03:49:53.274278+0000","last_clean":"2024-09-11T03:49:53.274278+0000","last_became_active":"2024-09-11T03:49:29.157735+0000","last_became_peered":"2024-09-11T03:49:29.157735+0000","last_unstale":"2024-09-11T03:49:53.274278+0000","last_undegraded":"2024-09-11T03:49:53.274278+0000","last_fullsized":"2024-09-11T03:49:53.274278+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:49:32.876969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.876969+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:02.019082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071689500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"2.2a","version":"727'348","reported_seq":3685,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.030146+0000","last_change":"2024-09-11T03:49:49.030494+0000","last_active":"2024-09-11T03:49:50.030146+0000","last_peered":"2024-09-11T03:49:50.030146+0000","last_clean":"2024-09-11T03:49:50.030146+0000","last_became_active":"2024-09-11T03:49:29.160595+0000","last_became_peered":"2024-09-11T03:49:29.160595+0000","last_unstale":"2024-09-11T03:49:50.030146+0000","last_undegraded":"2024-09-11T03:49:50.030146+0000","last_fullsized":"2024-09-11T03:49:50.030146+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:49:34.219244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.219244+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:55.247967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084572300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,5],"acting":[7,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"2.29","version":"771'1196","reported_seq":4736,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.076745+0000","last_change":"2024-09-11T03:49:49.033292+0000","last_active":"2024-09-11T03:49:52.076745+0000","last_peered":"2024-09-11T03:49:52.076745+0000","last_clean":"2024-09-11T03:49:52.076745+0000","last_became_active":"2024-09-11T03:49:29.157194+0000","last_became_peered":"2024-09-11T03:49:29.157194+0000","last_unstale":"2024-09-11T03:49:52.076745+0000","last_undegraded":"2024-09-11T03:49:52.076745+0000","last_fullsized":"2024-09-11T03:49:52.076745+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'1167","last_scrub_stamp":"2024-09-11T03:49:33.506786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.506786+0000","objects_scrubbed":11,"log_size":1196,"log_dups_size":0,"ondisk_log_size":1196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:34.243180+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0035154069999999999,"stat_sum":{"num_bytes":25750304,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":11,"num_read_kb":8515,"num_write":7,"num_write_kb":1588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8],"acting":[6,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"2.28","version":"769'394","reported_seq":3832,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.260043+0000","last_change":"2024-09-11T03:49:49.025321+0000","last_active":"2024-09-11T03:49:51.260043+0000","last_peered":"2024-09-11T03:49:51.260043+0000","last_clean":"2024-09-11T03:49:51.260043+0000","last_became_active":"2024-09-11T03:49:29.154884+0000","last_became_peered":"2024-09-11T03:49:29.154884+0000","last_unstale":"2024-09-11T03:49:51.260043+0000","last_undegraded":"2024-09-11T03:49:51.260043+0000","last_fullsized":"2024-09-11T03:49:51.260043+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'382","last_scrub_stamp":"2024-09-11T03:49:32.335285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.335285+0000","objects_scrubbed":4,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:43.876896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000477858,"stat_sum":{"num_bytes":8486578,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"2.27","version":"769'1080","reported_seq":4627,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.030370+0000","last_change":"2024-09-11T03:49:49.032395+0000","last_active":"2024-09-11T03:49:50.030370+0000","last_peered":"2024-09-11T03:49:50.030370+0000","last_clean":"2024-09-11T03:49:50.030370+0000","last_became_active":"2024-09-11T03:49:29.159799+0000","last_became_peered":"2024-09-11T03:49:29.159799+0000","last_unstale":"2024-09-11T03:49:50.030370+0000","last_undegraded":"2024-09-11T03:49:50.030370+0000","last_fullsized":"2024-09-11T03:49:50.030370+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"757'1065","last_scrub_stamp":"2024-09-11T03:49:36.196385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.196385+0000","objects_scrubbed":4,"log_size":1080,"log_dups_size":0,"ondisk_log_size":1080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:46.932148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002046104,"stat_sum":{"num_bytes":9109955,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":9,"num_read_kb":13447,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,9],"acting":[7,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"2.26","version":"269'171","reported_seq":3570,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.030201+0000","last_change":"2024-09-11T03:49:49.031936+0000","last_active":"2024-09-11T03:49:50.030201+0000","last_peered":"2024-09-11T03:49:50.030201+0000","last_clean":"2024-09-11T03:49:50.030201+0000","last_became_active":"2024-09-11T03:49:29.158577+0000","last_became_peered":"2024-09-11T03:49:29.158577+0000","last_unstale":"2024-09-11T03:49:50.030201+0000","last_undegraded":"2024-09-11T03:49:50.030201+0000","last_fullsized":"2024-09-11T03:49:50.030201+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:49:33.199779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.199779+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:41.774456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017887560000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,11],"acting":[7,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"2.25","version":"769'1203","reported_seq":4845,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.265368+0000","last_change":"2024-09-11T03:49:49.025486+0000","last_active":"2024-09-11T03:49:51.265368+0000","last_peered":"2024-09-11T03:49:51.265368+0000","last_clean":"2024-09-11T03:49:51.265368+0000","last_became_active":"2024-09-11T03:49:29.159733+0000","last_became_peered":"2024-09-11T03:49:29.159733+0000","last_unstale":"2024-09-11T03:49:51.265368+0000","last_undegraded":"2024-09-11T03:49:51.265368+0000","last_fullsized":"2024-09-11T03:49:51.265368+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"755'1138","last_scrub_stamp":"2024-09-11T03:49:34.198067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.198067+0000","objects_scrubbed":16,"log_size":1203,"log_dups_size":0,"ondisk_log_size":1203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:40.902982+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0013169810000000001,"stat_sum":{"num_bytes":34812583,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":12,"num_write_kb":5450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,6],"acting":[8,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"2.24","version":"769'634","reported_seq":4094,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.077208+0000","last_change":"2024-09-11T03:49:49.032883+0000","last_active":"2024-09-11T03:49:52.077208+0000","last_peered":"2024-09-11T03:49:52.077208+0000","last_clean":"2024-09-11T03:49:52.077208+0000","last_became_active":"2024-09-11T03:49:29.155740+0000","last_became_peered":"2024-09-11T03:49:29.155740+0000","last_unstale":"2024-09-11T03:49:52.077208+0000","last_undegraded":"2024-09-11T03:49:52.077208+0000","last_fullsized":"2024-09-11T03:49:52.077208+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'612","last_scrub_stamp":"2024-09-11T03:49:34.554335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.554335+0000","objects_scrubbed":4,"log_size":634,"log_dups_size":0,"ondisk_log_size":634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:38.777295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023166419999999998,"stat_sum":{"num_bytes":10081013,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,7],"acting":[6,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"2.23","version":"769'454","reported_seq":3866,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.260188+0000","last_change":"2024-09-11T03:49:49.025974+0000","last_active":"2024-09-11T03:49:51.260188+0000","last_peered":"2024-09-11T03:49:51.260188+0000","last_clean":"2024-09-11T03:49:51.260188+0000","last_became_active":"2024-09-11T03:46:32.970100+0000","last_became_peered":"2024-09-11T03:46:32.970100+0000","last_unstale":"2024-09-11T03:49:51.260188+0000","last_undegraded":"2024-09-11T03:49:51.260188+0000","last_fullsized":"2024-09-11T03:49:51.260188+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"711'414","last_scrub_stamp":"2024-09-11T03:48:57.105274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:57.105274+0000","objects_scrubbed":2,"log_size":454,"log_dups_size":0,"ondisk_log_size":454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:00.044058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00091780799999999999,"stat_sum":{"num_bytes":9866336,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":31430,"num_write":23,"num_write_kb":10047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"2.22","version":"531'216","reported_seq":3624,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.076839+0000","last_change":"2024-09-11T03:49:49.031572+0000","last_active":"2024-09-11T03:49:52.076839+0000","last_peered":"2024-09-11T03:49:52.076839+0000","last_clean":"2024-09-11T03:49:52.076839+0000","last_became_active":"2024-09-11T03:46:32.970633+0000","last_became_peered":"2024-09-11T03:46:32.970633+0000","last_unstale":"2024-09-11T03:49:52.076839+0000","last_undegraded":"2024-09-11T03:49:52.076839+0000","last_fullsized":"2024-09-11T03:49:52.076839+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:49:18.406097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:18.406097+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:21.988285+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0016267790000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"2.21","version":"769'314","reported_seq":3816,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.024628+0000","last_change":"2024-09-11T03:49:49.124502+0000","last_active":"2024-09-11T03:49:52.024628+0000","last_peered":"2024-09-11T03:49:52.024628+0000","last_clean":"2024-09-11T03:49:52.024628+0000","last_became_active":"2024-09-11T03:46:36.024928+0000","last_became_peered":"2024-09-11T03:46:36.024928+0000","last_unstale":"2024-09-11T03:49:52.024628+0000","last_undegraded":"2024-09-11T03:49:52.024628+0000","last_fullsized":"2024-09-11T03:49:52.024628+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":6,"last_scrub":"733'289","last_scrub_stamp":"2024-09-11T03:49:18.917801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:18.917801+0000","objects_scrubbed":3,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:45.367194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00065857899999999996,"stat_sum":{"num_bytes":10867443,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":27704,"num_write":26,"num_write_kb":13444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,8],"acting":[11,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"2.20","version":"771'266","reported_seq":3830,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.030466+0000","last_change":"2024-09-11T03:49:49.031952+0000","last_active":"2024-09-11T03:49:50.030466+0000","last_peered":"2024-09-11T03:49:50.030466+0000","last_clean":"2024-09-11T03:49:50.030466+0000","last_became_active":"2024-09-11T03:46:32.974375+0000","last_became_peered":"2024-09-11T03:46:32.974375+0000","last_unstale":"2024-09-11T03:49:50.030466+0000","last_undegraded":"2024-09-11T03:49:50.030466+0000","last_fullsized":"2024-09-11T03:49:50.030466+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"718'223","last_scrub_stamp":"2024-09-11T03:49:00.376119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.376119+0000","objects_scrubbed":2,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:10.643726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0021024379999999999,"stat_sum":{"num_bytes":11791232,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":78,"num_read_kb":126195,"num_write":16,"num_write_kb":9596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"2.1c","version":"765'766","reported_seq":4173,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.179126+0000","last_change":"2024-09-11T03:49:49.031286+0000","last_active":"2024-09-11T03:49:52.179126+0000","last_peered":"2024-09-11T03:49:52.179126+0000","last_clean":"2024-09-11T03:49:52.179126+0000","last_became_active":"2024-09-11T03:46:32.971036+0000","last_became_peered":"2024-09-11T03:46:32.971036+0000","last_unstale":"2024-09-11T03:49:52.179126+0000","last_undegraded":"2024-09-11T03:49:52.179126+0000","last_fullsized":"2024-09-11T03:49:52.179126+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"754'760","last_scrub_stamp":"2024-09-11T03:49:33.579794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.579794+0000","objects_scrubbed":5,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:40.202439+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00061417000000000004,"stat_sum":{"num_bytes":4835616,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":34,"num_read_kb":35078,"num_write":30,"num_write_kb":12013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":597,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.029438+0000","last_change":"2024-09-11T03:47:26.195120+0000","last_active":"2024-09-11T03:49:50.029438+0000","last_peered":"2024-09-11T03:49:50.029438+0000","last_clean":"2024-09-11T03:49:50.029438+0000","last_became_active":"2024-09-11T03:46:15.974786+0000","last_became_peered":"2024-09-11T03:46:15.974786+0000","last_unstale":"2024-09-11T03:49:50.029438+0000","last_undegraded":"2024-09-11T03:49:50.029438+0000","last_fullsized":"2024-09-11T03:49:50.029438+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:26.195070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:26.195070+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.f","version":"557'465","reported_seq":3896,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.265483+0000","last_change":"2024-09-11T03:49:49.025161+0000","last_active":"2024-09-11T03:49:51.265483+0000","last_peered":"2024-09-11T03:49:51.265483+0000","last_clean":"2024-09-11T03:49:51.265483+0000","last_became_active":"2024-09-11T03:46:27.971779+0000","last_became_peered":"2024-09-11T03:46:27.971779+0000","last_unstale":"2024-09-11T03:49:51.265483+0000","last_undegraded":"2024-09-11T03:49:51.265483+0000","last_fullsized":"2024-09-11T03:49:51.265483+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:48:59.211781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:59.211781+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:59.724167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001088505,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"2.e","version":"774'1015","reported_seq":4403,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-09-11T03:49:53.274557+0000","last_change":"2024-09-11T03:49:49.127611+0000","last_active":"2024-09-11T03:49:53.274557+0000","last_peered":"2024-09-11T03:49:53.274557+0000","last_clean":"2024-09-11T03:49:53.274557+0000","last_became_active":"2024-09-11T03:46:36.024843+0000","last_became_peered":"2024-09-11T03:46:36.024843+0000","last_unstale":"2024-09-11T03:49:53.274557+0000","last_undegraded":"2024-09-11T03:49:53.274557+0000","last_fullsized":"2024-09-11T03:49:53.274557+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"736'963","last_scrub_stamp":"2024-09-11T03:49:17.918033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:17.918033+0000","objects_scrubbed":8,"log_size":1015,"log_dups_size":0,"ondisk_log_size":1015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:40.505254+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001042196,"stat_sum":{"num_bytes":20929826,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":67,"num_read_kb":66224,"num_write":57,"num_write_kb":31784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":1174,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.030669+0000","last_change":"2024-09-11T03:49:20.176154+0000","last_active":"2024-09-11T03:49:50.030669+0000","last_peered":"2024-09-11T03:49:50.030669+0000","last_clean":"2024-09-11T03:49:50.030669+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:49:50.030669+0000","last_undegraded":"2024-09-11T03:49:50.030669+0000","last_fullsized":"2024-09-11T03:49:50.030669+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:20.176098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.176098+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:41.602355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"771'1001","reported_seq":4474,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.261109+0000","last_change":"2024-09-11T03:49:49.044528+0000","last_active":"2024-09-11T03:49:51.261109+0000","last_peered":"2024-09-11T03:49:51.261109+0000","last_clean":"2024-09-11T03:49:51.261109+0000","last_became_active":"2024-09-11T03:49:24.116744+0000","last_became_peered":"2024-09-11T03:49:24.116744+0000","last_unstale":"2024-09-11T03:49:51.261109+0000","last_undegraded":"2024-09-11T03:49:51.261109+0000","last_fullsized":"2024-09-11T03:49:51.261109+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'987","last_scrub_stamp":"2024-09-11T03:49:32.653263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.653263+0000","objects_scrubbed":3,"log_size":1001,"log_dups_size":0,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:02.142664+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0047001600000000001,"stat_sum":{"num_bytes":8757074,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":889,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.030175+0000","last_change":"2024-09-11T03:49:22.192021+0000","last_active":"2024-09-11T03:49:50.030175+0000","last_peered":"2024-09-11T03:49:50.030175+0000","last_clean":"2024-09-11T03:49:50.030175+0000","last_became_active":"2024-09-11T03:44:43.940022+0000","last_became_peered":"2024-09-11T03:44:43.940022+0000","last_unstale":"2024-09-11T03:49:50.030175+0000","last_undegraded":"2024-09-11T03:49:50.030175+0000","last_fullsized":"2024-09-11T03:49:50.030175+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:22.191965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:22.191965+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:33.940974+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"769'823","reported_seq":4264,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.024363+0000","last_change":"2024-09-11T03:49:49.124742+0000","last_active":"2024-09-11T03:49:52.024363+0000","last_peered":"2024-09-11T03:49:52.024363+0000","last_clean":"2024-09-11T03:49:52.024363+0000","last_became_active":"2024-09-11T03:49:24.115405+0000","last_became_peered":"2024-09-11T03:49:24.115405+0000","last_unstale":"2024-09-11T03:49:52.024363+0000","last_undegraded":"2024-09-11T03:49:52.024363+0000","last_fullsized":"2024-09-11T03:49:52.024363+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'808","last_scrub_stamp":"2024-09-11T03:49:33.953477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.953477+0000","objects_scrubbed":7,"log_size":823,"log_dups_size":0,"ondisk_log_size":823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:54.144247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00064249800000000003,"stat_sum":{"num_bytes":13462365,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.f","version":"20'92","reported_seq":458,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.077663+0000","last_change":"2024-09-11T03:48:57.363099+0000","last_active":"2024-09-11T03:49:52.077663+0000","last_peered":"2024-09-11T03:49:52.077663+0000","last_clean":"2024-09-11T03:49:52.077663+0000","last_became_active":"2024-09-11T03:46:15.973796+0000","last_became_peered":"2024-09-11T03:46:15.973796+0000","last_unstale":"2024-09-11T03:49:52.077663+0000","last_undegraded":"2024-09-11T03:49:52.077663+0000","last_fullsized":"2024-09-11T03:49:52.077663+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:57.363027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:57.363027+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:08.181547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":1580,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.031032+0000","last_change":"2024-09-11T03:47:28.495899+0000","last_active":"2024-09-11T03:49:50.031032+0000","last_peered":"2024-09-11T03:49:50.031032+0000","last_clean":"2024-09-11T03:49:50.031032+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:49:50.031032+0000","last_undegraded":"2024-09-11T03:49:50.031032+0000","last_fullsized":"2024-09-11T03:49:50.031032+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:28.495839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:28.495839+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.015311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"769'494","reported_seq":3917,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.260976+0000","last_change":"2024-09-11T03:49:49.040017+0000","last_active":"2024-09-11T03:49:51.260976+0000","last_peered":"2024-09-11T03:49:51.260976+0000","last_clean":"2024-09-11T03:49:51.260976+0000","last_became_active":"2024-09-11T03:46:27.972329+0000","last_became_peered":"2024-09-11T03:46:27.972329+0000","last_unstale":"2024-09-11T03:49:51.260976+0000","last_undegraded":"2024-09-11T03:49:51.260976+0000","last_fullsized":"2024-09-11T03:49:51.260976+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'480","last_scrub_stamp":"2024-09-11T03:49:37.694768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.694768+0000","objects_scrubbed":5,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:54.967635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00063778800000000003,"stat_sum":{"num_bytes":8442631,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":3872,"num_write":26,"num_write_kb":5269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":1184,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.260403+0000","last_change":"2024-09-11T03:47:25.344219+0000","last_active":"2024-09-11T03:49:51.260403+0000","last_peered":"2024-09-11T03:49:51.260403+0000","last_clean":"2024-09-11T03:49:51.260403+0000","last_became_active":"2024-09-11T03:46:15.968960+0000","last_became_peered":"2024-09-11T03:46:15.968960+0000","last_unstale":"2024-09-11T03:49:51.260403+0000","last_undegraded":"2024-09-11T03:49:51.260403+0000","last_fullsized":"2024-09-11T03:49:51.260403+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:25.344170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:25.344170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:42.179046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"563'192","reported_seq":3683,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.030805+0000","last_change":"2024-09-11T03:49:49.032623+0000","last_active":"2024-09-11T03:49:50.030805+0000","last_peered":"2024-09-11T03:49:50.030805+0000","last_clean":"2024-09-11T03:49:50.030805+0000","last_became_active":"2024-09-11T03:46:27.974405+0000","last_became_peered":"2024-09-11T03:46:27.974405+0000","last_unstale":"2024-09-11T03:49:50.030805+0000","last_undegraded":"2024-09-11T03:49:50.030805+0000","last_fullsized":"2024-09-11T03:49:50.030805+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:49:15.233165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:15.233165+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:36.698598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025488960000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":1143,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.029231+0000","last_change":"2024-09-11T03:49:17.252987+0000","last_active":"2024-09-11T03:49:50.029231+0000","last_peered":"2024-09-11T03:49:50.029231+0000","last_clean":"2024-09-11T03:49:50.029231+0000","last_became_active":"2024-09-11T03:44:25.934580+0000","last_became_peered":"2024-09-11T03:44:25.934580+0000","last_unstale":"2024-09-11T03:49:50.029231+0000","last_undegraded":"2024-09-11T03:49:50.029231+0000","last_fullsized":"2024-09-11T03:49:50.029231+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:17.252916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:17.252916+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:39.008724+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"567'235","reported_seq":3615,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.029422+0000","last_change":"2024-09-11T03:49:49.027881+0000","last_active":"2024-09-11T03:49:50.029422+0000","last_peered":"2024-09-11T03:49:50.029422+0000","last_clean":"2024-09-11T03:49:50.029422+0000","last_became_active":"2024-09-11T03:46:36.024925+0000","last_became_peered":"2024-09-11T03:46:36.024925+0000","last_unstale":"2024-09-11T03:49:50.029422+0000","last_undegraded":"2024-09-11T03:49:50.029422+0000","last_fullsized":"2024-09-11T03:49:50.029422+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:49:01.138426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:01.138426+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:29.825702+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00093503200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":752,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.178575+0000","last_change":"2024-09-11T03:48:32.199588+0000","last_active":"2024-09-11T03:49:52.178575+0000","last_peered":"2024-09-11T03:49:52.178575+0000","last_clean":"2024-09-11T03:49:52.178575+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:49:52.178575+0000","last_undegraded":"2024-09-11T03:49:52.178575+0000","last_fullsized":"2024-09-11T03:49:52.178575+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:32.199524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:32.199524+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:00.036523+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"772'286","reported_seq":3783,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.260886+0000","last_change":"2024-09-11T03:49:49.039957+0000","last_active":"2024-09-11T03:49:51.260886+0000","last_peered":"2024-09-11T03:49:51.260886+0000","last_clean":"2024-09-11T03:49:51.260886+0000","last_became_active":"2024-09-11T03:46:27.971649+0000","last_became_peered":"2024-09-11T03:46:27.971649+0000","last_unstale":"2024-09-11T03:49:51.260886+0000","last_undegraded":"2024-09-11T03:49:51.260886+0000","last_fullsized":"2024-09-11T03:49:51.260886+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"718'254","last_scrub_stamp":"2024-09-11T03:49:15.510323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:15.510323+0000","objects_scrubbed":4,"log_size":286,"log_dups_size":0,"ondisk_log_size":286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:41.457628+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00079791599999999997,"stat_sum":{"num_bytes":4639161,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":22,"num_read_kb":13034,"num_write":19,"num_write_kb":11745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":450,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.077519+0000","last_change":"2024-09-11T03:49:37.637065+0000","last_active":"2024-09-11T03:49:52.077519+0000","last_peered":"2024-09-11T03:49:52.077519+0000","last_clean":"2024-09-11T03:49:52.077519+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:49:52.077519+0000","last_undegraded":"2024-09-11T03:49:52.077519+0000","last_fullsized":"2024-09-11T03:49:52.077519+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:37.637018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.637018+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:05.221389+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"769'636","reported_seq":4114,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.260381+0000","last_change":"2024-09-11T03:49:49.025823+0000","last_active":"2024-09-11T03:49:51.260381+0000","last_peered":"2024-09-11T03:49:51.260381+0000","last_clean":"2024-09-11T03:49:51.260381+0000","last_became_active":"2024-09-11T03:49:24.114330+0000","last_became_peered":"2024-09-11T03:49:24.114330+0000","last_unstale":"2024-09-11T03:49:51.260381+0000","last_undegraded":"2024-09-11T03:49:51.260381+0000","last_fullsized":"2024-09-11T03:49:51.260381+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'620","last_scrub_stamp":"2024-09-11T03:49:34.288083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.288083+0000","objects_scrubbed":8,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:39.755351+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00045728700000000001,"stat_sum":{"num_bytes":18600842,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":2643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":763,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.178749+0000","last_change":"2024-09-11T03:49:17.858224+0000","last_active":"2024-09-11T03:49:52.178749+0000","last_peered":"2024-09-11T03:49:52.178749+0000","last_clean":"2024-09-11T03:49:52.178749+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:49:52.178749+0000","last_undegraded":"2024-09-11T03:49:52.178749+0000","last_fullsized":"2024-09-11T03:49:52.178749+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:17.858165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:17.858165+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:40.674663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"769'1141","reported_seq":4626,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.265431+0000","last_change":"2024-09-11T03:49:49.025792+0000","last_active":"2024-09-11T03:49:51.265431+0000","last_peered":"2024-09-11T03:49:51.265431+0000","last_clean":"2024-09-11T03:49:51.265431+0000","last_became_active":"2024-09-11T03:49:24.115095+0000","last_became_peered":"2024-09-11T03:49:24.115095+0000","last_unstale":"2024-09-11T03:49:51.265431+0000","last_undegraded":"2024-09-11T03:49:51.265431+0000","last_fullsized":"2024-09-11T03:49:51.265431+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'1112","last_scrub_stamp":"2024-09-11T03:49:35.188307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.188307+0000","objects_scrubbed":7,"log_size":1141,"log_dups_size":0,"ondisk_log_size":1141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:40.334328+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001593236,"stat_sum":{"num_bytes":13036804,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":2299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":762,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.178300+0000","last_change":"2024-09-11T03:49:36.654074+0000","last_active":"2024-09-11T03:49:52.178300+0000","last_peered":"2024-09-11T03:49:52.178300+0000","last_clean":"2024-09-11T03:49:52.178300+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:49:52.178300+0000","last_undegraded":"2024-09-11T03:49:52.178300+0000","last_fullsized":"2024-09-11T03:49:52.178300+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:36.654025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.654025+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:06.216209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":3576,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.260246+0000","last_change":"2024-09-11T03:49:49.026432+0000","last_active":"2024-09-11T03:49:51.260246+0000","last_peered":"2024-09-11T03:49:51.260246+0000","last_clean":"2024-09-11T03:49:51.260246+0000","last_became_active":"2024-09-11T03:49:24.113518+0000","last_became_peered":"2024-09-11T03:49:24.113518+0000","last_unstale":"2024-09-11T03:49:51.260246+0000","last_undegraded":"2024-09-11T03:49:51.260246+0000","last_fullsized":"2024-09-11T03:49:51.260246+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:49:35.267227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.267227+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.830151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000457699,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":1363,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.024284+0000","last_change":"2024-09-11T03:49:21.909643+0000","last_active":"2024-09-11T03:49:52.024284+0000","last_peered":"2024-09-11T03:49:52.024284+0000","last_clean":"2024-09-11T03:49:52.024284+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:49:52.024284+0000","last_undegraded":"2024-09-11T03:49:52.024284+0000","last_fullsized":"2024-09-11T03:49:52.024284+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:21.909594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:21.909594+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:22.293071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"769'1095","reported_seq":4682,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.077416+0000","last_change":"2024-09-11T03:49:49.033378+0000","last_active":"2024-09-11T03:49:52.077416+0000","last_peered":"2024-09-11T03:49:52.077416+0000","last_clean":"2024-09-11T03:49:52.077416+0000","last_became_active":"2024-09-11T03:49:24.121333+0000","last_became_peered":"2024-09-11T03:49:24.121333+0000","last_unstale":"2024-09-11T03:49:52.077416+0000","last_undegraded":"2024-09-11T03:49:52.077416+0000","last_fullsized":"2024-09-11T03:49:52.077416+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'1066","last_scrub_stamp":"2024-09-11T03:49:35.600201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.600201+0000","objects_scrubbed":7,"log_size":1095,"log_dups_size":0,"ondisk_log_size":1095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.650664+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0025209690000000001,"stat_sum":{"num_bytes":23396961,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":41786,"num_write":12,"num_write_kb":9044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":1158,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.024294+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:49:52.024294+0000","last_peered":"2024-09-11T03:49:52.024294+0000","last_clean":"2024-09-11T03:49:52.024294+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:49:52.024294+0000","last_undegraded":"2024-09-11T03:49:52.024294+0000","last_fullsized":"2024-09-11T03:49:52.024294+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:13.961107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"744'374","reported_seq":3835,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.030780+0000","last_change":"2024-09-11T03:49:49.032178+0000","last_active":"2024-09-11T03:49:50.030780+0000","last_peered":"2024-09-11T03:49:50.030780+0000","last_clean":"2024-09-11T03:49:50.030780+0000","last_became_active":"2024-09-11T03:49:24.115150+0000","last_became_peered":"2024-09-11T03:49:24.115150+0000","last_unstale":"2024-09-11T03:49:50.030780+0000","last_undegraded":"2024-09-11T03:49:50.030780+0000","last_fullsized":"2024-09-11T03:49:50.030780+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'374","last_scrub_stamp":"2024-09-11T03:49:35.190806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.190806+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:36.487486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00160663,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":1510,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-09-11T03:49:53.274174+0000","last_change":"2024-09-11T03:48:59.762212+0000","last_active":"2024-09-11T03:49:53.274174+0000","last_peered":"2024-09-11T03:49:53.274174+0000","last_clean":"2024-09-11T03:49:53.274174+0000","last_became_active":"2024-09-11T03:46:15.973271+0000","last_became_peered":"2024-09-11T03:46:15.973271+0000","last_unstale":"2024-09-11T03:49:53.274174+0000","last_undegraded":"2024-09-11T03:49:53.274174+0000","last_fullsized":"2024-09-11T03:49:53.274174+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:59.762145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:59.762145+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:02.580429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"769'1218","reported_seq":4759,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.260096+0000","last_change":"2024-09-11T03:49:49.025511+0000","last_active":"2024-09-11T03:49:51.260096+0000","last_peered":"2024-09-11T03:49:51.260096+0000","last_clean":"2024-09-11T03:49:51.260096+0000","last_became_active":"2024-09-11T03:49:24.114865+0000","last_became_peered":"2024-09-11T03:49:24.114865+0000","last_unstale":"2024-09-11T03:49:51.260096+0000","last_undegraded":"2024-09-11T03:49:51.260096+0000","last_fullsized":"2024-09-11T03:49:51.260096+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'1171","last_scrub_stamp":"2024-09-11T03:49:33.292043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.292043+0000","objects_scrubbed":10,"log_size":1218,"log_dups_size":0,"ondisk_log_size":1218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:43.083379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00051702400000000004,"stat_sum":{"num_bytes":21358568,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":15593,"num_write":12,"num_write_kb":3037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":1575,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.029959+0000","last_change":"2024-09-11T03:49:37.206392+0000","last_active":"2024-09-11T03:49:50.029959+0000","last_peered":"2024-09-11T03:49:50.029959+0000","last_clean":"2024-09-11T03:49:50.029959+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:49:50.029959+0000","last_undegraded":"2024-09-11T03:49:50.029959+0000","last_fullsized":"2024-09-11T03:49:50.029959+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:37.206321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.206321+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:53.641968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"772'356","reported_seq":3823,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.024322+0000","last_change":"2024-09-11T03:49:49.124437+0000","last_active":"2024-09-11T03:49:52.024322+0000","last_peered":"2024-09-11T03:49:52.024322+0000","last_clean":"2024-09-11T03:49:52.024322+0000","last_became_active":"2024-09-11T03:49:24.114551+0000","last_became_peered":"2024-09-11T03:49:24.114551+0000","last_unstale":"2024-09-11T03:49:52.024322+0000","last_undegraded":"2024-09-11T03:49:52.024322+0000","last_fullsized":"2024-09-11T03:49:52.024322+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:49:32.911631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.911631+0000","objects_scrubbed":4,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:01.079455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056214199999999996,"stat_sum":{"num_bytes":5429057,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":12240,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":1236,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.260862+0000","last_change":"2024-09-11T03:49:34.621588+0000","last_active":"2024-09-11T03:49:51.260862+0000","last_peered":"2024-09-11T03:49:51.260862+0000","last_clean":"2024-09-11T03:49:51.260862+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:49:51.260862+0000","last_undegraded":"2024-09-11T03:49:51.260862+0000","last_fullsized":"2024-09-11T03:49:51.260862+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:34.621527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.621527+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.563733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"568'646","reported_seq":4097,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.265066+0000","last_change":"2024-09-11T03:49:49.024722+0000","last_active":"2024-09-11T03:49:51.265066+0000","last_peered":"2024-09-11T03:49:51.265066+0000","last_clean":"2024-09-11T03:49:51.265066+0000","last_became_active":"2024-09-11T03:49:24.113147+0000","last_became_peered":"2024-09-11T03:49:24.113147+0000","last_unstale":"2024-09-11T03:49:51.265066+0000","last_undegraded":"2024-09-11T03:49:51.265066+0000","last_fullsized":"2024-09-11T03:49:51.265066+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:49:33.204497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.204497+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:48.004691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00090716099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":1186,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.260783+0000","last_change":"2024-09-11T03:49:00.710015+0000","last_active":"2024-09-11T03:49:51.260783+0000","last_peered":"2024-09-11T03:49:51.260783+0000","last_clean":"2024-09-11T03:49:51.260783+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:49:51.260783+0000","last_undegraded":"2024-09-11T03:49:51.260783+0000","last_fullsized":"2024-09-11T03:49:51.260783+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:00.709966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.709966+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:30.175623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":3497,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-09-11T03:49:53.274555+0000","last_change":"2024-09-11T03:49:49.127354+0000","last_active":"2024-09-11T03:49:53.274555+0000","last_peered":"2024-09-11T03:49:53.274555+0000","last_clean":"2024-09-11T03:49:53.274555+0000","last_became_active":"2024-09-11T03:41:38.886834+0000","last_became_peered":"2024-09-11T03:41:38.886834+0000","last_unstale":"2024-09-11T03:49:53.274555+0000","last_undegraded":"2024-09-11T03:49:53.274555+0000","last_fullsized":"2024-09-11T03:49:53.274555+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:49:15.904230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:15.904230+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:27.948295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087295000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":1169,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.029321+0000","last_change":"2024-09-11T03:49:41.224105+0000","last_active":"2024-09-11T03:49:50.029321+0000","last_peered":"2024-09-11T03:49:50.029321+0000","last_clean":"2024-09-11T03:49:50.029321+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:49:50.029321+0000","last_undegraded":"2024-09-11T03:49:50.029321+0000","last_fullsized":"2024-09-11T03:49:50.029321+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:41.224059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:41.224059+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:58.932206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"771'547","reported_seq":3979,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.178382+0000","last_change":"2024-09-11T03:49:49.033581+0000","last_active":"2024-09-11T03:49:52.178382+0000","last_peered":"2024-09-11T03:49:52.178382+0000","last_clean":"2024-09-11T03:49:52.178382+0000","last_became_active":"2024-09-11T03:43:55.918817+0000","last_became_peered":"2024-09-11T03:43:55.918817+0000","last_unstale":"2024-09-11T03:49:52.178382+0000","last_undegraded":"2024-09-11T03:49:52.178382+0000","last_fullsized":"2024-09-11T03:49:52.178382+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"736'523","last_scrub_stamp":"2024-09-11T03:49:20.735392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.735392+0000","objects_scrubbed":7,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:42.754228+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025624860000000001,"stat_sum":{"num_bytes":11998221,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":209,"num_read_kb":223952,"num_write":140,"num_write_kb":52245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":15216236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":1167,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.260707+0000","last_change":"2024-09-11T03:47:23.339646+0000","last_active":"2024-09-11T03:49:51.260707+0000","last_peered":"2024-09-11T03:49:51.260707+0000","last_clean":"2024-09-11T03:49:51.260707+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:49:51.260707+0000","last_undegraded":"2024-09-11T03:49:51.260707+0000","last_fullsized":"2024-09-11T03:49:51.260707+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:23.339595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:23.339595+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:50.634356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":2566,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.077427+0000","last_change":"2024-09-11T03:49:49.031983+0000","last_active":"2024-09-11T03:49:52.077427+0000","last_peered":"2024-09-11T03:49:52.077427+0000","last_clean":"2024-09-11T03:49:52.077427+0000","last_became_active":"2024-09-11T03:46:19.005117+0000","last_became_peered":"2024-09-11T03:46:19.005117+0000","last_unstale":"2024-09-11T03:49:52.077427+0000","last_undegraded":"2024-09-11T03:49:52.077427+0000","last_fullsized":"2024-09-11T03:49:52.077427+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:49:40.618009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:40.618009+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:53.462274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001702498,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.13","version":"20'92","reported_seq":1271,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.260343+0000","last_change":"2024-09-11T03:47:18.217600+0000","last_active":"2024-09-11T03:49:51.260343+0000","last_peered":"2024-09-11T03:49:51.260343+0000","last_clean":"2024-09-11T03:49:51.260343+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:49:51.260343+0000","last_undegraded":"2024-09-11T03:49:51.260343+0000","last_fullsized":"2024-09-11T03:49:51.260343+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:18.217555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:18.217555+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:43.139528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":2507,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.264864+0000","last_change":"2024-09-11T03:49:49.024366+0000","last_active":"2024-09-11T03:49:51.264864+0000","last_peered":"2024-09-11T03:49:51.264864+0000","last_clean":"2024-09-11T03:49:51.264864+0000","last_became_active":"2024-09-11T03:41:35.874295+0000","last_became_peered":"2024-09-11T03:41:35.874295+0000","last_unstale":"2024-09-11T03:49:51.264864+0000","last_undegraded":"2024-09-11T03:49:51.264864+0000","last_fullsized":"2024-09-11T03:49:51.264864+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:49:41.179684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:41.179684+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:43.687208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00080587499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":743,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.030048+0000","last_change":"2024-09-11T03:48:57.320199+0000","last_active":"2024-09-11T03:49:50.030048+0000","last_peered":"2024-09-11T03:49:50.030048+0000","last_clean":"2024-09-11T03:49:50.030048+0000","last_became_active":"2024-09-11T03:46:36.026459+0000","last_became_peered":"2024-09-11T03:46:36.026459+0000","last_unstale":"2024-09-11T03:49:50.030048+0000","last_undegraded":"2024-09-11T03:49:50.030048+0000","last_fullsized":"2024-09-11T03:49:50.030048+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:57.320159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:57.320159+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:16.384062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":3062,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.077797+0000","last_change":"2024-09-11T03:49:49.033935+0000","last_active":"2024-09-11T03:49:52.077797+0000","last_peered":"2024-09-11T03:49:52.077797+0000","last_clean":"2024-09-11T03:49:52.077797+0000","last_became_active":"2024-09-11T03:46:19.003959+0000","last_became_peered":"2024-09-11T03:46:19.003959+0000","last_unstale":"2024-09-11T03:49:52.077797+0000","last_undegraded":"2024-09-11T03:49:52.077797+0000","last_fullsized":"2024-09-11T03:49:52.077797+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:49:38.613856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:38.613856+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:56.602276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027500210000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":1137,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.029923+0000","last_change":"2024-09-11T03:47:47.459856+0000","last_active":"2024-09-11T03:49:50.029923+0000","last_peered":"2024-09-11T03:49:50.029923+0000","last_clean":"2024-09-11T03:49:50.029923+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:49:50.029923+0000","last_undegraded":"2024-09-11T03:49:50.029923+0000","last_fullsized":"2024-09-11T03:49:50.029923+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:47.459803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:47.459803+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:08.322414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"773'547","reported_seq":4021,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.289090+0000","last_change":"2024-09-11T03:49:49.040403+0000","last_active":"2024-09-11T03:49:51.289090+0000","last_peered":"2024-09-11T03:49:51.289090+0000","last_clean":"2024-09-11T03:49:51.289090+0000","last_became_active":"2024-09-11T03:46:19.007019+0000","last_became_peered":"2024-09-11T03:46:19.007019+0000","last_unstale":"2024-09-11T03:49:51.289090+0000","last_undegraded":"2024-09-11T03:49:51.289090+0000","last_fullsized":"2024-09-11T03:49:51.289090+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"755'521","last_scrub_stamp":"2024-09-11T03:49:33.626967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.626967+0000","objects_scrubbed":9,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:38.809291+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001243505,"stat_sum":{"num_bytes":17294850,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":143,"num_read_kb":132822,"num_write":108,"num_write_kb":45006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":5263193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":622,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.024471+0000","last_change":"2024-09-11T03:49:20.949194+0000","last_active":"2024-09-11T03:49:52.024471+0000","last_peered":"2024-09-11T03:49:52.024471+0000","last_clean":"2024-09-11T03:49:52.024471+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:49:52.024471+0000","last_undegraded":"2024-09-11T03:49:52.024471+0000","last_fullsized":"2024-09-11T03:49:52.024471+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:20.949118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.949118+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:36.681631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":3717,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-09-11T03:49:53.274461+0000","last_change":"2024-09-11T03:49:49.127367+0000","last_active":"2024-09-11T03:49:53.274461+0000","last_peered":"2024-09-11T03:49:53.274461+0000","last_clean":"2024-09-11T03:49:53.274461+0000","last_became_active":"2024-09-11T03:41:38.888228+0000","last_became_peered":"2024-09-11T03:41:38.888228+0000","last_unstale":"2024-09-11T03:49:53.274461+0000","last_undegraded":"2024-09-11T03:49:53.274461+0000","last_fullsized":"2024-09-11T03:49:53.274461+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:49:16.919210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:16.919210+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:24.781276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00090339000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":1158,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.265318+0000","last_change":"2024-09-11T03:47:29.086383+0000","last_active":"2024-09-11T03:49:51.265318+0000","last_peered":"2024-09-11T03:49:51.265318+0000","last_clean":"2024-09-11T03:49:51.265318+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:49:51.265318+0000","last_undegraded":"2024-09-11T03:49:51.265318+0000","last_fullsized":"2024-09-11T03:49:51.265318+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:29.086335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:29.086335+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:43.171032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"769'351","reported_seq":3686,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.077236+0000","last_change":"2024-09-11T03:49:49.033611+0000","last_active":"2024-09-11T03:49:52.077236+0000","last_peered":"2024-09-11T03:49:52.077236+0000","last_clean":"2024-09-11T03:49:52.077236+0000","last_became_active":"2024-09-11T03:46:19.004558+0000","last_became_peered":"2024-09-11T03:46:19.004558+0000","last_unstale":"2024-09-11T03:49:52.077236+0000","last_undegraded":"2024-09-11T03:49:52.077236+0000","last_fullsized":"2024-09-11T03:49:52.077236+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"711'317","last_scrub_stamp":"2024-09-11T03:48:50.287601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:50.287601+0000","objects_scrubbed":5,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:01.224290+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0027103959999999999,"stat_sum":{"num_bytes":6294372,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":48,"num_read_kb":27601,"num_write":63,"num_write_kb":18466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.18","version":"20'92","reported_seq":1270,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.260607+0000","last_change":"2024-09-11T03:48:53.702605+0000","last_active":"2024-09-11T03:49:51.260607+0000","last_peered":"2024-09-11T03:49:51.260607+0000","last_clean":"2024-09-11T03:49:51.260607+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:49:51.260607+0000","last_undegraded":"2024-09-11T03:49:51.260607+0000","last_fullsized":"2024-09-11T03:49:51.260607+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:53.702523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:53.702523+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:49:55.160792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"772'817","reported_seq":4366,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.024847+0000","last_change":"2024-09-11T03:49:49.126894+0000","last_active":"2024-09-11T03:49:52.024847+0000","last_peered":"2024-09-11T03:49:52.024847+0000","last_clean":"2024-09-11T03:49:52.024847+0000","last_became_active":"2024-09-11T03:46:32.975567+0000","last_became_peered":"2024-09-11T03:46:32.975567+0000","last_unstale":"2024-09-11T03:49:52.024847+0000","last_undegraded":"2024-09-11T03:49:52.024847+0000","last_fullsized":"2024-09-11T03:49:52.024847+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"754'773","last_scrub_stamp":"2024-09-11T03:49:34.919322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.919322+0000","objects_scrubbed":8,"log_size":817,"log_dups_size":0,"ondisk_log_size":817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:48.487796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.003001339,"stat_sum":{"num_bytes":18663859,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":98,"num_read_kb":101451,"num_write":50,"num_write_kb":23556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7],"acting":[11,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.19","version":"20'92","reported_seq":767,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.178170+0000","last_change":"2024-09-11T03:48:56.129575+0000","last_active":"2024-09-11T03:49:52.178170+0000","last_peered":"2024-09-11T03:49:52.178170+0000","last_clean":"2024-09-11T03:49:52.178170+0000","last_became_active":"2024-09-11T03:46:15.972841+0000","last_became_peered":"2024-09-11T03:46:15.972841+0000","last_unstale":"2024-09-11T03:49:52.178170+0000","last_undegraded":"2024-09-11T03:49:52.178170+0000","last_fullsized":"2024-09-11T03:49:52.178170+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:56.129516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:56.129516+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:24.133914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"557'273","reported_seq":3636,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.077165+0000","last_change":"2024-09-11T03:49:49.032267+0000","last_active":"2024-09-11T03:49:52.077165+0000","last_peered":"2024-09-11T03:49:52.077165+0000","last_clean":"2024-09-11T03:49:52.077165+0000","last_became_active":"2024-09-11T03:46:32.971003+0000","last_became_peered":"2024-09-11T03:46:32.971003+0000","last_unstale":"2024-09-11T03:49:52.077165+0000","last_undegraded":"2024-09-11T03:49:52.077165+0000","last_fullsized":"2024-09-11T03:49:52.077165+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:49:20.466140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.466140+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:22.905207+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0019850190000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,2],"acting":[6,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.1a","version":"20'92","reported_seq":300,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.030103+0000","last_change":"2024-09-11T03:49:31.427740+0000","last_active":"2024-09-11T03:49:50.030103+0000","last_peered":"2024-09-11T03:49:50.030103+0000","last_clean":"2024-09-11T03:49:50.030103+0000","last_became_active":"2024-09-11T03:49:30.162694+0000","last_became_peered":"2024-09-11T03:49:30.162694+0000","last_unstale":"2024-09-11T03:49:50.030103+0000","last_undegraded":"2024-09-11T03:49:50.030103+0000","last_fullsized":"2024-09-11T03:49:50.030103+0000","mapping_epoch":751,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":752,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:45:52.914088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:45:52.914088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:50:01.193085+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":72127,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.19","version":"772'571","reported_seq":4077,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.077149+0000","last_change":"2024-09-11T03:49:49.031554+0000","last_active":"2024-09-11T03:49:52.077149+0000","last_peered":"2024-09-11T03:49:52.077149+0000","last_clean":"2024-09-11T03:49:52.077149+0000","last_became_active":"2024-09-11T03:46:19.002002+0000","last_became_peered":"2024-09-11T03:46:19.002002+0000","last_unstale":"2024-09-11T03:49:52.077149+0000","last_undegraded":"2024-09-11T03:49:52.077149+0000","last_fullsized":"2024-09-11T03:49:52.077149+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"736'546","last_scrub_stamp":"2024-09-11T03:49:19.458345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:19.458345+0000","objects_scrubbed":6,"log_size":571,"log_dups_size":0,"ondisk_log_size":571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:45.772932+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0012912030000000001,"stat_sum":{"num_bytes":11704959,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":103,"num_read_kb":99782,"num_write":67,"num_write_kb":30004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4870845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.1b","version":"20'92","reported_seq":1135,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.264897+0000","last_change":"2024-09-11T03:48:31.033896+0000","last_active":"2024-09-11T03:49:51.264897+0000","last_peered":"2024-09-11T03:49:51.264897+0000","last_clean":"2024-09-11T03:49:51.264897+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:49:51.264897+0000","last_undegraded":"2024-09-11T03:49:51.264897+0000","last_fullsized":"2024-09-11T03:49:51.264897+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:31.033837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:31.033837+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:46.181076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"769'515","reported_seq":4014,"reported_epoch":775,"state":"active+clean","last_fresh":"2024-09-11T03:49:53.274516+0000","last_change":"2024-09-11T03:49:49.127091+0000","last_active":"2024-09-11T03:49:53.274516+0000","last_peered":"2024-09-11T03:49:53.274516+0000","last_clean":"2024-09-11T03:49:53.274516+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:49:53.274516+0000","last_undegraded":"2024-09-11T03:49:53.274516+0000","last_fullsized":"2024-09-11T03:49:53.274516+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"711'462","last_scrub_stamp":"2024-09-11T03:48:49.842909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:49.842909+0000","objects_scrubbed":9,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:00.715146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00063917100000000001,"stat_sum":{"num_bytes":24339313,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":203,"num_read_kb":242435,"num_write":92,"num_write_kb":42535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4865740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"2.1f","version":"771'687","reported_seq":4058,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.029426+0000","last_change":"2024-09-11T03:49:49.032001+0000","last_active":"2024-09-11T03:49:50.029426+0000","last_peered":"2024-09-11T03:49:50.029426+0000","last_clean":"2024-09-11T03:49:50.029426+0000","last_became_active":"2024-09-11T03:46:32.968696+0000","last_became_peered":"2024-09-11T03:46:32.968696+0000","last_unstale":"2024-09-11T03:49:50.029426+0000","last_undegraded":"2024-09-11T03:49:50.029426+0000","last_fullsized":"2024-09-11T03:49:50.029426+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"733'613","last_scrub_stamp":"2024-09-11T03:49:20.295589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.295589+0000","objects_scrubbed":10,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:31.076306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0052368559999999998,"stat_sum":{"num_bytes":36828456,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":120,"num_read_kb":137134,"num_write":78,"num_write_kb":26587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.1c","version":"20'92","reported_seq":1142,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.265174+0000","last_change":"2024-09-11T03:49:40.156559+0000","last_active":"2024-09-11T03:49:51.265174+0000","last_peered":"2024-09-11T03:49:51.265174+0000","last_clean":"2024-09-11T03:49:51.265174+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:49:51.265174+0000","last_undegraded":"2024-09-11T03:49:51.265174+0000","last_fullsized":"2024-09-11T03:49:51.265174+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:40.156495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:40.156495+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:04.047069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1e","version":"771'1024","reported_seq":4402,"reported_epoch":772,"state":"active+clean","last_fresh":"2024-09-11T03:49:50.029437+0000","last_change":"2024-09-11T03:49:49.028545+0000","last_active":"2024-09-11T03:49:50.029437+0000","last_peered":"2024-09-11T03:49:50.029437+0000","last_clean":"2024-09-11T03:49:50.029437+0000","last_became_active":"2024-09-11T03:46:32.970637+0000","last_became_peered":"2024-09-11T03:46:32.970637+0000","last_unstale":"2024-09-11T03:49:50.029437+0000","last_undegraded":"2024-09-11T03:49:50.029437+0000","last_fullsized":"2024-09-11T03:49:50.029437+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"718'982","last_scrub_stamp":"2024-09-11T03:48:56.189316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:56.189316+0000","objects_scrubbed":9,"log_size":1024,"log_dups_size":0,"ondisk_log_size":1024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:19.947123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001585153,"stat_sum":{"num_bytes":13212919,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":54,"num_read_kb":51944,"num_write":77,"num_write_kb":27431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"136","length":"1"},{"start":"149","length":"1"},{"start":"14b","length":"1"},{"start":"14e","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.1d","version":"20'92","reported_seq":1124,"reported_epoch":773,"state":"active+clean","last_fresh":"2024-09-11T03:49:51.260256+0000","last_change":"2024-09-11T03:47:22.312507+0000","last_active":"2024-09-11T03:49:51.260256+0000","last_peered":"2024-09-11T03:49:51.260256+0000","last_clean":"2024-09-11T03:49:51.260256+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:49:51.260256+0000","last_undegraded":"2024-09-11T03:49:51.260256+0000","last_fullsized":"2024-09-11T03:49:51.260256+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:22.312456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:22.312456+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.239645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"567'752","reported_seq":4129,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.178359+0000","last_change":"2024-09-11T03:49:49.031210+0000","last_active":"2024-09-11T03:49:52.178359+0000","last_peered":"2024-09-11T03:49:52.178359+0000","last_clean":"2024-09-11T03:49:52.178359+0000","last_became_active":"2024-09-11T03:46:32.972371+0000","last_became_peered":"2024-09-11T03:46:32.972371+0000","last_unstale":"2024-09-11T03:49:52.178359+0000","last_undegraded":"2024-09-11T03:49:52.178359+0000","last_fullsized":"2024-09-11T03:49:52.178359+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:48:55.139430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:55.139430+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:06.817175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00061934200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,11],"acting":[10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"14b","length":"1"},{"start":"158","length":"1"}]},{"pgid":"1.1e","version":"20'92","reported_seq":1154,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-09-11T03:49:52.024503+0000","last_change":"2024-09-11T03:49:36.949199+0000","last_active":"2024-09-11T03:49:52.024503+0000","last_peered":"2024-09-11T03:49:52.024503+0000","last_clean":"2024-09-11T03:49:52.024503+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:49:52.024503+0000","last_undegraded":"2024-09-11T03:49:52.024503+0000","last_fullsized":"2024-09-11T03:49:52.024503+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:36.949149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.949149+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:45.388809+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":46,"stat_sum":{"num_bytes":449637991,"num_objects":205,"num_object_clones":155,"num_object_copies":615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":12,"num_read":1360,"num_read_kb":1433518,"num_write":942,"num_write_kb":397822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":30216014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":797503488,"data_stored":1087554144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26643,"ondisk_log_size":26643,"up":138,"acting":138,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":399807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94084,"kb_used_data":62888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93651068,"statfs":{"total":95995035648,"available":95898693632,"internally_reserved":0,"allocated":64397312,"data_stored":82247535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514652,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91828,"kb_used_data":52320,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93653324,"statfs":{"total":95995035648,"available":95901003776,"internally_reserved":0,"allocated":53575680,"data_stored":60589200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743586,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":124828,"kb_used_data":67272,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93620324,"statfs":{"total":95995035648,"available":95867211776,"internally_reserved":0,"allocated":68886528,"data_stored":84773642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037929,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":134876,"kb_used_data":81476,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93610276,"statfs":{"total":95995035648,"available":95856922624,"internally_reserved":0,"allocated":83431424,"data_stored":117375726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299564,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":182772,"kb_used_data":125212,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93562380,"statfs":{"total":95995035648,"available":95807877120,"internally_reserved":0,"allocated":128217088,"data_stored":167521347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528497,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":161620,"kb_used_data":117952,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93583532,"statfs":{"total":95995035648,"available":95829536768,"internally_reserved":0,"allocated":120782848,"data_stored":156238196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757431,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96788,"kb_used_data":65596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93648364,"statfs":{"total":95995035648,"available":95895924736,"internally_reserved":0,"allocated":67170304,"data_stored":83588277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986364,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":93256,"kb_used_data":39856,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93651896,"statfs":{"total":95995035648,"available":95899541504,"internally_reserved":0,"allocated":40812544,"data_stored":46924348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149889,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":144332,"kb_used_data":99260,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93600820,"statfs":{"total":95995035648,"available":95847239680,"internally_reserved":0,"allocated":101642240,"data_stored":136929921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411527,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79552,"kb_used_data":48360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93665600,"statfs":{"total":95995035648,"available":95913574400,"internally_reserved":0,"allocated":49520640,"data_stored":55207623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575052,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85560,"kb_used_data":50212,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93659592,"statfs":{"total":95995035648,"available":95907422208,"internally_reserved":0,"allocated":51417088,"data_stored":65051888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738577,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":129032,"kb_used_data":75632,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93616120,"statfs":{"total":95995035648,"available":95862906880,"internally_reserved":0,"allocated":77447168,"data_stored":100245365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67772416,"data_stored":93834696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42057728,"data_stored":58960636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":39944192,"data_stored":50371033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94257152,"data_stored":132814017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":31477760,"data_stored":40849622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57606144,"data_stored":77295777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":111095808,"data_stored":149819264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":118665216,"data_stored":161237110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":74084352,"data_stored":111292737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":59514880,"data_stored":78674055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":44228608,"data_stored":54506211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":56799232,"data_stored":77898986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:49:57.336 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-11T03:49:57.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 56 2024-09-11T03:49:57.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:57.337 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:57.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:57.438 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:57.459 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:57.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:57.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:57.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:57.582 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:57.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:57.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:57.635 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:57 smithi084 ceph-mon[27254]: osdmap e778: 12 total, 12 up, 12 in 2024-09-11T03:49:57.635 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:57 smithi084 ceph-mon[27254]: 1.18 scrub starts 2024-09-11T03:49:57.635 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:57 smithi084 ceph-mon[27254]: 1.18 scrub ok 2024-09-11T03:49:57.635 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:57 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:57.635 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:57 smithi084 ceph-mon[27254]: osdmap e779: 12 total, 12 up, 12 in 2024-09-11T03:49:57.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:57 smithi121 ceph-mon[30729]: osdmap e778: 12 total, 12 up, 12 in 2024-09-11T03:49:57.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:57 smithi121 ceph-mon[30729]: 1.18 scrub starts 2024-09-11T03:49:57.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:57 smithi121 ceph-mon[30729]: 1.18 scrub ok 2024-09-11T03:49:57.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:57 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:57.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:57 smithi121 ceph-mon[30729]: osdmap e779: 12 total, 12 up, 12 in 2024-09-11T03:49:57.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:57.706 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:57.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:57.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:57.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:57 smithi175 ceph-mon[30894]: osdmap e778: 12 total, 12 up, 12 in 2024-09-11T03:49:57.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:57 smithi175 ceph-mon[30894]: 1.18 scrub starts 2024-09-11T03:49:57.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:57 smithi175 ceph-mon[30894]: 1.18 scrub ok 2024-09-11T03:49:57.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:57 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:49:57.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:57 smithi175 ceph-mon[30894]: osdmap e779: 12 total, 12 up, 12 in 2024-09-11T03:49:57.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:57.829 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:57.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:57.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:57.932 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:57.957 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:57.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:57.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:58.027 INFO:tasks.rados.rados.2.smithi175.stdout:3172: done (0 left) 2024-09-11T03:49:58.027 INFO:tasks.rados.rados.2.smithi175.stdout:3173: copy_from oid 27 from oid 23 current snap is 351 2024-09-11T03:49:58.027 INFO:tasks.rados.rados.2.smithi175.stdout:3174: snap_create 2024-09-11T03:49:58.041 INFO:tasks.rados.rados.2.smithi175.stdout:3173: finishing copy_from racing read to smithi17557870-27 2024-09-11T03:49:58.044 INFO:tasks.rados.rados.2.smithi175.stdout:3173: finishing copy_from to smithi17557870-27 2024-09-11T03:49:58.044 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 951 (ObjNum 1022 snap 334 seq_num 1022) dirty exists 2024-09-11T03:49:58.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:49:58.081 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:58.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:58.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:49:58.183 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:58.205 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:58.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:58.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:58.307 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:58.332 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:58.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:58.333 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:58.434 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:58.455 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:58.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:58.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:58.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:58.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:58.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:58 smithi084 ceph-mon[27254]: pgmap v1323: 77 pgs: 77 active+clean; 444 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.7 MiB/s wr, 15 op/s 2024-09-11T03:49:58.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:49:58 smithi084 ceph-mon[27254]: osdmap e780: 12 total, 12 up, 12 in 2024-09-11T03:49:58.649 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T03:49:58.658 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:58.663 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:58.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:58 smithi121 ceph-mon[30729]: pgmap v1323: 77 pgs: 77 active+clean; 444 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.7 MiB/s wr, 15 op/s 2024-09-11T03:49:58.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:49:58 smithi121 ceph-mon[30729]: osdmap e780: 12 total, 12 up, 12 in 2024-09-11T03:49:58.686 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:58.707 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:58.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:58.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:58.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:58 smithi175 ceph-mon[30894]: pgmap v1323: 77 pgs: 77 active+clean; 444 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.7 MiB/s wr, 15 op/s 2024-09-11T03:49:58.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:49:58 smithi175 ceph-mon[30894]: osdmap e780: 12 total, 12 up, 12 in 2024-09-11T03:49:58.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:58.830 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:58.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:58.932 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:58.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:58.954 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:58.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:49:59.027 INFO:tasks.rados.rados.2.smithi175.stdout:3173: done (1 left) 2024-09-11T03:49:59.027 INFO:tasks.rados.rados.2.smithi175.stdout:3174: done (0 left) 2024-09-11T03:49:59.027 INFO:tasks.rados.rados.2.smithi175.stdout:3175: snap_remove snap 346 2024-09-11T03:49:59.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:49:59.081 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:59.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:59.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:49:59.184 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:49:59.207 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:59.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:59.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:49:59.309 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:59.330 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:59.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:59.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:59.433 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:49:59.459 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:59.459 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:49:59.561 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:49:59.586 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:49:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:59.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:49:59.616 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:49:59.688 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:49:59.713 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:49:59.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:59.714 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:49:59.815 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:49:59.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:59.838 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:59.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:49:59.939 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:49:59.961 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:49:59.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:49:59.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:00.025 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:50:00.025 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 2 pg_num to 56 2024-09-11T03:50:00.030 INFO:tasks.rados.rados.2.smithi175.stdout:3175: done (0 left) 2024-09-11T03:50:00.030 INFO:tasks.rados.rados.2.smithi175.stdout:3176: read oid 3 snap -1 2024-09-11T03:50:00.030 INFO:tasks.rados.rados.2.smithi175.stdout:3176: expect deleted 2024-09-11T03:50:00.030 INFO:tasks.rados.rados.2.smithi175.stdout:3177: write (excl) oid 11 current snap is 352 2024-09-11T03:50:00.031 INFO:tasks.rados.rados.2.smithi175.stdout:3177: seq_num 1077 ranges {501139=648946,1873695=227444} 2024-09-11T03:50:00.036 INFO:tasks.rados.rados.2.smithi175.stdout:3177: writing smithi17557870-11 from 501139 to 1150085 tid 1 2024-09-11T03:50:00.040 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:50:00.043 INFO:tasks.rados.rados.2.smithi175.stdout:3177: writing smithi17557870-11 from 1873695 to 2101139 tid 2 2024-09-11T03:50:00.043 INFO:tasks.rados.rados.2.smithi175.stdout:3178: read oid 12 snap -1 2024-09-11T03:50:00.043 INFO:tasks.rados.rados.2.smithi175.stdout:3178: expect (ObjNum 962 snap 308 seq_num 962) 2024-09-11T03:50:00.043 INFO:tasks.rados.rados.2.smithi175.stdout:3179: write oid 42 current snap is 352 2024-09-11T03:50:00.043 INFO:tasks.rados.rados.2.smithi175.stdout:3179: seq_num 1078 ranges {517946=724195,1932177=185769} 2024-09-11T03:50:00.049 INFO:tasks.rados.rados.2.smithi175.stdout:3179: writing smithi17557870-42 from 517946 to 1242141 tid 1 2024-09-11T03:50:00.054 INFO:tasks.rados.rados.2.smithi175.stdout:3179: writing smithi17557870-42 from 1932177 to 2117946 tid 2 2024-09-11T03:50:00.054 INFO:tasks.rados.rados.2.smithi175.stdout:3180: copy_from oid 17 from oid 29 current snap is 352 2024-09-11T03:50:00.055 INFO:tasks.rados.rados.2.smithi175.stdout:3177: finishing write tid 1 to smithi17557870-11 2024-09-11T03:50:00.055 INFO:tasks.rados.rados.2.smithi175.stdout:3177: finishing write tid 2 to smithi17557870-11 2024-09-11T03:50:00.055 INFO:tasks.rados.rados.2.smithi175.stdout:3177: finishing write tid 3 to smithi17557870-11 2024-09-11T03:50:00.055 INFO:tasks.rados.rados.2.smithi175.stdout:3178: expect (ObjNum 962 snap 308 seq_num 962) 2024-09-11T03:50:00.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:00.086 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:00.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:00.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:00.102 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:00 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:50:00.102 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:00 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:50:00.102 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:00 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:50:00.102 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:00 smithi084 ceph-mon[27254]: 2.14 scrub ok 2024-09-11T03:50:00.102 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:00 smithi084 ceph-mon[27254]: pgmap v1326: 77 pgs: 77 active+clean; 461 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 7.7 MiB/s wr, 25 op/s 2024-09-11T03:50:00.102 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:00 smithi084 ceph-mon[27254]: osdmap e781: 12 total, 12 up, 12 in 2024-09-11T03:50:00.103 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:00 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4012925461' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:00.103 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:00 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/174023279' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "56"} : dispatch 2024-09-11T03:50:00.103 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:00 smithi084 ceph-mon[27254]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-11T03:50:00.103 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:00 smithi084 ceph-mon[27254]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-11T03:50:00.103 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:50:00.103 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:50:00.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:50:00.103 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:50:00.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.104 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1159 2.e 2:750de3b7:::smithi17557870-8:head [copy-from ver 77] snapc 2e=[2e,2c,2b,28,26] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:50:00.104 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:01.831728+0000", 2024-09-11T03:50:00.104 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 598.26954032100002, 2024-09-11T03:50:00.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016271035999999999, 2024-09-11T03:50:00.104 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.104 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.104 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:00.104 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:00.104 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:00.104 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:00.104 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1159 2024-09-11T03:50:00.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.105 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831728+0000", 2024-09-11T03:50:00.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831728+0000", 2024-09-11T03:50:00.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.106 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831730+0000", 2024-09-11T03:50:00.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.283e-06 2024-09-11T03:50:00.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.106 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.106 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831741+0000", 2024-09-11T03:50:00.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0475000000000001e-05 2024-09-11T03:50:00.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.106 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.106 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831760+0000", 2024-09-11T03:50:00.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9034000000000001e-05 2024-09-11T03:50:00.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.107 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.107 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831776+0000", 2024-09-11T03:50:00.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6106000000000001e-05 2024-09-11T03:50:00.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.107 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.107 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831836+0000", 2024-09-11T03:50:00.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0420000000000001e-05 2024-09-11T03:50:00.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.108 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:50:00.108 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831841+0000", 2024-09-11T03:50:00.108 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8770000000000001e-06 2024-09-11T03:50:00.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.108 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.108 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.108 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.833455+0000", 2024-09-11T03:50:00.108 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016140639999999999 2024-09-11T03:50:00.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.108 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.108 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.108 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.833568+0000", 2024-09-11T03:50:00.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011259700000000001 2024-09-11T03:50:00.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.109 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:50:00.109 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.840230+0000", 2024-09-11T03:50:00.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066619189999999997 2024-09-11T03:50:00.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.109 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:00.109 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.845836+0000", 2024-09-11T03:50:00.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056066459999999999 2024-09-11T03:50:00.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.846094+0000", 2024-09-11T03:50:00.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025752300000000001 2024-09-11T03:50:00.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.847897+0000", 2024-09-11T03:50:00.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018033529999999999 2024-09-11T03:50:00.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:00.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.847954+0000", 2024-09-11T03:50:00.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6436e-05 2024-09-11T03:50:00.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.847999+0000", 2024-09-11T03:50:00.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5303000000000003e-05 2024-09-11T03:50:00.111 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.111 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.111 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.112 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1160 2.e 2:750de3b7:::smithi17557870-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e185)", 2024-09-11T03:50:00.112 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:01.831798+0000", 2024-09-11T03:50:00.112 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 598.26947060099997, 2024-09-11T03:50:00.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012910331000000001, 2024-09-11T03:50:00.112 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.112 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.112 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:00.112 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:00.112 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:00.112 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:00.113 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1160 2024-09-11T03:50:00.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.113 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.113 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.113 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831798+0000", 2024-09-11T03:50:00.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.113 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831798+0000", 2024-09-11T03:50:00.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831798+0000", 2024-09-11T03:50:00.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0699999999999998e-07 2024-09-11T03:50:00.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831800+0000", 2024-09-11T03:50:00.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8679999999999999e-06 2024-09-11T03:50:00.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.115 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831806+0000", 2024-09-11T03:50:00.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6339999999999998e-06 2024-09-11T03:50:00.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.115 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831809+0000", 2024-09-11T03:50:00.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8650000000000003e-06 2024-09-11T03:50:00.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831896+0000", 2024-09-11T03:50:00.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6700000000000007e-05 2024-09-11T03:50:00.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:50:00.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.831904+0000", 2024-09-11T03:50:00.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8099999999999998e-06 2024-09-11T03:50:00.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.833745+0000", 2024-09-11T03:50:00.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018409979999999999 2024-09-11T03:50:00.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.117 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:00.117 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.833795+0000", 2024-09-11T03:50:00.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0413e-05 2024-09-11T03:50:00.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.117 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.117 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.844574+0000", 2024-09-11T03:50:00.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010778993000000001 2024-09-11T03:50:00.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.118 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.118 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.844637+0000", 2024-09-11T03:50:00.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3002000000000004e-05 2024-09-11T03:50:00.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.118 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.118 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:01.844708+0000", 2024-09-11T03:50:00.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0641000000000003e-05 2024-09-11T03:50:00.118 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.118 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.118 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.119 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1285 2.1 2:8638b65e:::smithi17557870-5:head [copy-from ver 153] snapc 33=[32,30,2c,2b] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:50:00.119 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:12.836052+0000", 2024-09-11T03:50:00.119 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 587.26521600399997, 2024-09-11T03:50:00.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020441596999999999, 2024-09-11T03:50:00.119 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.119 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.119 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:00.119 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:00.119 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:00.119 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:00.119 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1285 2024-09-11T03:50:00.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.119 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836052+0000", 2024-09-11T03:50:00.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836052+0000", 2024-09-11T03:50:00.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836053+0000", 2024-09-11T03:50:00.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.34e-07 2024-09-11T03:50:00.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836055+0000", 2024-09-11T03:50:00.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7549999999999999e-06 2024-09-11T03:50:00.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836061+0000", 2024-09-11T03:50:00.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6579999999999999e-06 2024-09-11T03:50:00.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836071+0000", 2024-09-11T03:50:00.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0141e-05 2024-09-11T03:50:00.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836118+0000", 2024-09-11T03:50:00.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7200999999999998e-05 2024-09-11T03:50:00.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:50:00.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836124+0000", 2024-09-11T03:50:00.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8150000000000002e-06 2024-09-11T03:50:00.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.124 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.124 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.837592+0000", 2024-09-11T03:50:00.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014672509999999999 2024-09-11T03:50:00.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.124 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.124 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.837642+0000", 2024-09-11T03:50:00.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0145999999999997e-05 2024-09-11T03:50:00.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.124 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,3", 2024-09-11T03:50:00.124 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.849946+0000", 2024-09-11T03:50:00.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012303869 2024-09-11T03:50:00.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:00.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.852055+0000", 2024-09-11T03:50:00.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002109835 2024-09-11T03:50:00.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856314+0000", 2024-09-11T03:50:00.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042587700000000003 2024-09-11T03:50:00.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856368+0000", 2024-09-11T03:50:00.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3538999999999997e-05 2024-09-11T03:50:00.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:00.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856435+0000", 2024-09-11T03:50:00.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7470000000000003e-05 2024-09-11T03:50:00.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.856494+0000", 2024-09-11T03:50:00.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8712999999999998e-05 2024-09-11T03:50:00.127 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.127 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.127 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.127 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1286 2.1 2:8638b65e:::smithi17557870-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e196)", 2024-09-11T03:50:00.127 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:12.836083+0000", 2024-09-11T03:50:00.128 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 587.26518513400003, 2024-09-11T03:50:00.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015739692, 2024-09-11T03:50:00.128 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.128 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.128 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:00.128 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:00.128 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:00.128 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:00.128 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1286 2024-09-11T03:50:00.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.128 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836083+0000", 2024-09-11T03:50:00.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836083+0000", 2024-09-11T03:50:00.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836084+0000", 2024-09-11T03:50:00.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2500000000000001e-07 2024-09-11T03:50:00.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.130 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.130 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836085+0000", 2024-09-11T03:50:00.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8020000000000001e-06 2024-09-11T03:50:00.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.130 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.130 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.130 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836092+0000", 2024-09-11T03:50:00.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3169999999999998e-06 2024-09-11T03:50:00.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.130 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.130 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.130 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836096+0000", 2024-09-11T03:50:00.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.634e-06 2024-09-11T03:50:00.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.130 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.131 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836147+0000", 2024-09-11T03:50:00.131 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0374999999999998e-05 2024-09-11T03:50:00.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:50:00.131 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.836150+0000", 2024-09-11T03:50:00.131 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.072e-06 2024-09-11T03:50:00.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.837748+0000", 2024-09-11T03:50:00.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001597867 2024-09-11T03:50:00.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:00.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.837783+0000", 2024-09-11T03:50:00.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5395000000000002e-05 2024-09-11T03:50:00.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.851559+0000", 2024-09-11T03:50:00.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013776343999999999 2024-09-11T03:50:00.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.851665+0000", 2024-09-11T03:50:00.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010521199999999999 2024-09-11T03:50:00.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:12.851823+0000", 2024-09-11T03:50:00.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015824900000000001 2024-09-11T03:50:00.135 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.136 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.136 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.136 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1434 2.e 2:7efbbea8:::smithi17557870-24:head [copy-from ver 75] snapc 36=[35,2b] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-11T03:50:00.137 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:20.864420+0000", 2024-09-11T03:50:00.137 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 579.23684878300003, 2024-09-11T03:50:00.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012389887, 2024-09-11T03:50:00.137 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.137 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.137 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:00.137 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:00.137 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:00.137 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:00.137 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1434 2024-09-11T03:50:00.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.138 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864420+0000", 2024-09-11T03:50:00.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864420+0000", 2024-09-11T03:50:00.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864421+0000", 2024-09-11T03:50:00.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8369999999999999e-06 2024-09-11T03:50:00.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864430+0000", 2024-09-11T03:50:00.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.132e-06 2024-09-11T03:50:00.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864447+0000", 2024-09-11T03:50:00.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6305999999999999e-05 2024-09-11T03:50:00.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864454+0000", 2024-09-11T03:50:00.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1230000000000002e-06 2024-09-11T03:50:00.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864527+0000", 2024-09-11T03:50:00.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2606000000000002e-05 2024-09-11T03:50:00.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:50:00.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.864536+0000", 2024-09-11T03:50:00.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7790000000000007e-06 2024-09-11T03:50:00.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.865844+0000", 2024-09-11T03:50:00.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307473 2024-09-11T03:50:00.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.865934+0000", 2024-09-11T03:50:00.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9785999999999997e-05 2024-09-11T03:50:00.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:50:00.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.869567+0000", 2024-09-11T03:50:00.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0036332859999999999 2024-09-11T03:50:00.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:00.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.874154+0000", 2024-09-11T03:50:00.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.004586957 2024-09-11T03:50:00.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.875683+0000", 2024-09-11T03:50:00.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015289920000000001 2024-09-11T03:50:00.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.876715+0000", 2024-09-11T03:50:00.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010321620000000001 2024-09-11T03:50:00.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:00.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.876752+0000", 2024-09-11T03:50:00.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6820000000000003e-05 2024-09-11T03:50:00.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:20.876809+0000", 2024-09-11T03:50:00.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7627999999999998e-05 2024-09-11T03:50:00.147 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.147 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.147 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.147 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1589 2.7 2:e77ce595:::smithi17557870-31:head [copy-from ver 148] snapc 3f=[3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-09-11T03:50:00.148 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:35.104207+0000", 2024-09-11T03:50:00.148 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 564.99706113000002, 2024-09-11T03:50:00.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021309244000000001, 2024-09-11T03:50:00.148 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.148 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.148 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:00.148 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:00.148 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:00.148 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:00.148 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1589 2024-09-11T03:50:00.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.148 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104207+0000", 2024-09-11T03:50:00.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104207+0000", 2024-09-11T03:50:00.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104208+0000", 2024-09-11T03:50:00.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2780000000000001e-06 2024-09-11T03:50:00.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104213+0000", 2024-09-11T03:50:00.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5979999999999999e-06 2024-09-11T03:50:00.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104225+0000", 2024-09-11T03:50:00.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1819e-05 2024-09-11T03:50:00.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104249+0000", 2024-09-11T03:50:00.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4117999999999998e-05 2024-09-11T03:50:00.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.151 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.151 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104305+0000", 2024-09-11T03:50:00.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6069999999999997e-05 2024-09-11T03:50:00.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.151 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.151 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104375+0000", 2024-09-11T03:50:00.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0156999999999999e-05 2024-09-11T03:50:00.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.151 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:50:00.151 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116294+0000", 2024-09-11T03:50:00.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011918676 2024-09-11T03:50:00.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:00.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.123116+0000", 2024-09-11T03:50:00.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068219259999999999 2024-09-11T03:50:00.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.124470+0000", 2024-09-11T03:50:00.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001354079 2024-09-11T03:50:00.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125455+0000", 2024-09-11T03:50:00.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00098535300000000001 2024-09-11T03:50:00.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:00.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125487+0000", 2024-09-11T03:50:00.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2200999999999999e-05 2024-09-11T03:50:00.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125516+0000", 2024-09-11T03:50:00.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8969e-05 2024-09-11T03:50:00.153 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.153 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.154 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.154 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1590 2.7 2:e77ce595:::smithi17557870-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e218)", 2024-09-11T03:50:00.154 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:35.104267+0000", 2024-09-11T03:50:00.154 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 564.99700155000005, 2024-09-11T03:50:00.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017183714999999999, 2024-09-11T03:50:00.154 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.154 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.154 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:00.154 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:00.154 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:00.154 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:00.154 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1590 2024-09-11T03:50:00.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.155 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104267+0000", 2024-09-11T03:50:00.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104267+0000", 2024-09-11T03:50:00.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104267+0000", 2024-09-11T03:50:00.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.77e-07 2024-09-11T03:50:00.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104269+0000", 2024-09-11T03:50:00.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8199999999999999e-06 2024-09-11T03:50:00.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104274+0000", 2024-09-11T03:50:00.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2839999999999997e-06 2024-09-11T03:50:00.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104277+0000", 2024-09-11T03:50:00.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0149999999999999e-06 2024-09-11T03:50:00.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104478+0000", 2024-09-11T03:50:00.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000200256 2024-09-11T03:50:00.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:00.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.104511+0000", 2024-09-11T03:50:00.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3155999999999999e-05 2024-09-11T03:50:00.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.121250+0000", 2024-09-11T03:50:00.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016738843999999999 2024-09-11T03:50:00.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.121352+0000", 2024-09-11T03:50:00.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102384 2024-09-11T03:50:00.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.121450+0000", 2024-09-11T03:50:00.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8579000000000006e-05 2024-09-11T03:50:00.159 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.159 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.159 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.159 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1634 2.7 2:e23d1762:::smithi17557870-21:head [copy-from ver 214] snapc 40=[40,3f,3e,3d,3c,3a,38] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:50:00.159 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:36.206869+0000", 2024-09-11T03:50:00.159 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 563.89439883800003, 2024-09-11T03:50:00.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017521808999999999, 2024-09-11T03:50:00.160 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.160 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.160 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:00.160 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:00.160 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:00.160 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:00.160 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1634 2024-09-11T03:50:00.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.160 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206869+0000", 2024-09-11T03:50:00.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206869+0000", 2024-09-11T03:50:00.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206872+0000", 2024-09-11T03:50:00.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.926e-06 2024-09-11T03:50:00.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206898+0000", 2024-09-11T03:50:00.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5987e-05 2024-09-11T03:50:00.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.206986+0000", 2024-09-11T03:50:00.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7714999999999996e-05 2024-09-11T03:50:00.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207030+0000", 2024-09-11T03:50:00.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3455999999999999e-05 2024-09-11T03:50:00.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207171+0000", 2024-09-11T03:50:00.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014184700000000001 2024-09-11T03:50:00.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207283+0000", 2024-09-11T03:50:00.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000111976 2024-09-11T03:50:00.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:50:00.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.216671+0000", 2024-09-11T03:50:00.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0093877200000000004 2024-09-11T03:50:00.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.223379+0000", 2024-09-11T03:50:00.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067082289999999996 2024-09-11T03:50:00.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:00.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224223+0000", 2024-09-11T03:50:00.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000843771 2024-09-11T03:50:00.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224293+0000", 2024-09-11T03:50:00.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0339000000000005e-05 2024-09-11T03:50:00.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:00.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224347+0000", 2024-09-11T03:50:00.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3288999999999998e-05 2024-09-11T03:50:00.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.224391+0000", 2024-09-11T03:50:00.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4554000000000001e-05 2024-09-11T03:50:00.165 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.165 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.166 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.166 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1635 2.7 2:e23d1762:::smithi17557870-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e219)", 2024-09-11T03:50:00.166 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:36.207063+0000", 2024-09-11T03:50:00.166 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 563.89420506099998, 2024-09-11T03:50:00.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013462717000000001, 2024-09-11T03:50:00.166 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.166 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.166 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:00.166 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:00.166 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:00.166 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:00.167 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1635 2024-09-11T03:50:00.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.167 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207063+0000", 2024-09-11T03:50:00.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207063+0000", 2024-09-11T03:50:00.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207064+0000", 2024-09-11T03:50:00.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.043e-06 2024-09-11T03:50:00.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207071+0000", 2024-09-11T03:50:00.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8929999999999999e-06 2024-09-11T03:50:00.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207086+0000", 2024-09-11T03:50:00.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4596999999999999e-05 2024-09-11T03:50:00.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207095+0000", 2024-09-11T03:50:00.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9930000000000001e-06 2024-09-11T03:50:00.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207386+0000", 2024-09-11T03:50:00.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029111399999999998 2024-09-11T03:50:00.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:00.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.207423+0000", 2024-09-11T03:50:00.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6825000000000001e-05 2024-09-11T03:50:00.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.220376+0000", 2024-09-11T03:50:00.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012953549 2024-09-11T03:50:00.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.220446+0000", 2024-09-11T03:50:00.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9578000000000004e-05 2024-09-11T03:50:00.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:36.220526+0000", 2024-09-11T03:50:00.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0124999999999996e-05 2024-09-11T03:50:00.171 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.171 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.171 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.171 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1866 2.e 2:7b0c3397:::smithi17557870-14:head [copy-from ver 238] snapc 4c=[4c,4b,49,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-09-11T03:50:00.171 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:50:00.172 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 538.251135224, 2024-09-11T03:50:00.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012778199000000001, 2024-09-11T03:50:00.172 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.172 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.172 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:00.172 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:00.172 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:00.172 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:00.172 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1866 2024-09-11T03:50:00.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.172 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:50:00.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850133+0000", 2024-09-11T03:50:00.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850135+0000", 2024-09-11T03:50:00.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5850000000000001e-06 2024-09-11T03:50:00.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850142+0000", 2024-09-11T03:50:00.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2010000000000002e-06 2024-09-11T03:50:00.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850160+0000", 2024-09-11T03:50:00.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7858000000000001e-05 2024-09-11T03:50:00.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850170+0000", 2024-09-11T03:50:00.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9219999999999992e-06 2024-09-11T03:50:00.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850235+0000", 2024-09-11T03:50:00.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4849000000000002e-05 2024-09-11T03:50:00.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:50:00.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.850246+0000", 2024-09-11T03:50:00.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1938e-05 2024-09-11T03:50:00.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.851390+0000", 2024-09-11T03:50:00.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011435219999999999 2024-09-11T03:50:00.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.851491+0000", 2024-09-11T03:50:00.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000100868 2024-09-11T03:50:00.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:50:00.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.857380+0000", 2024-09-11T03:50:00.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058892839999999998 2024-09-11T03:50:00.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:00.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.861564+0000", 2024-09-11T03:50:00.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041843369999999998 2024-09-11T03:50:00.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.861883+0000", 2024-09-11T03:50:00.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031904399999999999 2024-09-11T03:50:00.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.862846+0000", 2024-09-11T03:50:00.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00096234000000000001 2024-09-11T03:50:00.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:00.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.862883+0000", 2024-09-11T03:50:00.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7132000000000003e-05 2024-09-11T03:50:00.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:01.862911+0000", 2024-09-11T03:50:00.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8319000000000001e-05 2024-09-11T03:50:00.180 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.180 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.180 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.181 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:1895 2.e 2:7efbbea8:::smithi17557870-24:head [copy-from ver 160] snapc 4e=[4e,4d,49,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-09-11T03:50:00.181 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:05.924255+0000", 2024-09-11T03:50:00.181 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 534.17701322799996, 2024-09-11T03:50:00.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016112593000000001, 2024-09-11T03:50:00.181 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.181 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.181 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:00.181 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:00.181 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:00.182 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:00.182 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1895 2024-09-11T03:50:00.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.182 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924255+0000", 2024-09-11T03:50:00.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924255+0000", 2024-09-11T03:50:00.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924257+0000", 2024-09-11T03:50:00.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.742e-06 2024-09-11T03:50:00.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924264+0000", 2024-09-11T03:50:00.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6090000000000003e-06 2024-09-11T03:50:00.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924281+0000", 2024-09-11T03:50:00.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6157000000000001e-05 2024-09-11T03:50:00.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924290+0000", 2024-09-11T03:50:00.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0499999999999997e-06 2024-09-11T03:50:00.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924352+0000", 2024-09-11T03:50:00.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1968e-05 2024-09-11T03:50:00.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:50:00.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.924358+0000", 2024-09-11T03:50:00.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1380000000000004e-06 2024-09-11T03:50:00.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.925553+0000", 2024-09-11T03:50:00.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001194802 2024-09-11T03:50:00.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.925622+0000", 2024-09-11T03:50:00.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9598999999999996e-05 2024-09-11T03:50:00.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:50:00.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.931889+0000", 2024-09-11T03:50:00.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062668419999999999 2024-09-11T03:50:00.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:00.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.937406+0000", 2024-09-11T03:50:00.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055165659999999997 2024-09-11T03:50:00.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.937537+0000", 2024-09-11T03:50:00.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000131941 2024-09-11T03:50:00.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.940300+0000", 2024-09-11T03:50:00.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027624720000000002 2024-09-11T03:50:00.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:00.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.940337+0000", 2024-09-11T03:50:00.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6869999999999998e-05 2024-09-11T03:50:00.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:05.940368+0000", 2024-09-11T03:50:00.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0837000000000003e-05 2024-09-11T03:50:00.188 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.188 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.188 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.189 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2565 2.7 2:e0eabcfb:::smithi17557870-9:head [copy-from ver 223] snapc 70=[70,6f,6e,6c,6a,68,65,63,5d] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-11T03:50:00.189 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:09.876844+0000", 2024-09-11T03:50:00.189 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 470.22442469200001, 2024-09-11T03:50:00.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015411238000000001, 2024-09-11T03:50:00.189 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.189 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.189 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:00.189 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:00.189 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:00.189 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:00.189 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2565 2024-09-11T03:50:00.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.189 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876844+0000", 2024-09-11T03:50:00.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876844+0000", 2024-09-11T03:50:00.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876845+0000", 2024-09-11T03:50:00.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.78e-07 2024-09-11T03:50:00.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876850+0000", 2024-09-11T03:50:00.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9540000000000003e-06 2024-09-11T03:50:00.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876863+0000", 2024-09-11T03:50:00.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3349e-05 2024-09-11T03:50:00.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876873+0000", 2024-09-11T03:50:00.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9590000000000007e-06 2024-09-11T03:50:00.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876919+0000", 2024-09-11T03:50:00.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5720000000000003e-05 2024-09-11T03:50:00.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876984+0000", 2024-09-11T03:50:00.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5437000000000004e-05 2024-09-11T03:50:00.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:50:00.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.887314+0000", 2024-09-11T03:50:00.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0103298 2024-09-11T03:50:00.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:00.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.891590+0000", 2024-09-11T03:50:00.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042757380000000003 2024-09-11T03:50:00.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.891885+0000", 2024-09-11T03:50:00.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029539200000000001 2024-09-11T03:50:00.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.892177+0000", 2024-09-11T03:50:00.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029223699999999999 2024-09-11T03:50:00.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:00.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.892215+0000", 2024-09-11T03:50:00.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7740000000000001e-05 2024-09-11T03:50:00.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.892255+0000", 2024-09-11T03:50:00.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9934000000000001e-05 2024-09-11T03:50:00.195 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.195 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.195 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.195 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2566 2.7 2:e0eabcfb:::smithi17557870-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e313)", 2024-09-11T03:50:00.195 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:09.876886+0000", 2024-09-11T03:50:00.195 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 470.224382504, 2024-09-11T03:50:00.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014077338, 2024-09-11T03:50:00.196 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.196 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.196 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:00.196 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:00.196 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:00.196 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:00.196 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2566 2024-09-11T03:50:00.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.196 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876886+0000", 2024-09-11T03:50:00.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876886+0000", 2024-09-11T03:50:00.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876886+0000", 2024-09-11T03:50:00.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7000000000000005e-07 2024-09-11T03:50:00.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876889+0000", 2024-09-11T03:50:00.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5730000000000002e-06 2024-09-11T03:50:00.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876895+0000", 2024-09-11T03:50:00.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4160000000000001e-06 2024-09-11T03:50:00.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.876899+0000", 2024-09-11T03:50:00.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3060000000000002e-06 2024-09-11T03:50:00.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.877088+0000", 2024-09-11T03:50:00.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018935499999999999 2024-09-11T03:50:00.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:00.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.877121+0000", 2024-09-11T03:50:00.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2566e-05 2024-09-11T03:50:00.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.890821+0000", 2024-09-11T03:50:00.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013699932999999999 2024-09-11T03:50:00.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.890886+0000", 2024-09-11T03:50:00.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5129999999999995e-05 2024-09-11T03:50:00.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:09.890963+0000", 2024-09-11T03:50:00.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7489000000000003e-05 2024-09-11T03:50:00.201 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.201 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.201 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.201 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:4498 2.e 2:7b0c3397:::smithi17557870-14:head [copy-from ver 491] snapc c5=[c5,c4,c3,c2,c1,c0,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-09-11T03:50:00.201 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:45:03.958685+0000", 2024-09-11T03:50:00.201 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 296.14258285300002, 2024-09-11T03:50:00.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014061295999999999, 2024-09-11T03:50:00.201 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.201 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.201 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:00.202 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:00.202 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:00.202 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:00.202 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4498 2024-09-11T03:50:00.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.202 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958685+0000", 2024-09-11T03:50:00.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958685+0000", 2024-09-11T03:50:00.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958687+0000", 2024-09-11T03:50:00.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.609e-06 2024-09-11T03:50:00.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958695+0000", 2024-09-11T03:50:00.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5410000000000003e-06 2024-09-11T03:50:00.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958711+0000", 2024-09-11T03:50:00.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6719999999999999e-05 2024-09-11T03:50:00.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958718+0000", 2024-09-11T03:50:00.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9720000000000003e-06 2024-09-11T03:50:00.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958804+0000", 2024-09-11T03:50:00.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5483999999999997e-05 2024-09-11T03:50:00.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:50:00.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958811+0000", 2024-09-11T03:50:00.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4059999999999999e-06 2024-09-11T03:50:00.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.960161+0000", 2024-09-11T03:50:00.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001349772 2024-09-11T03:50:00.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.960244+0000", 2024-09-11T03:50:00.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3461999999999995e-05 2024-09-11T03:50:00.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:50:00.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.967582+0000", 2024-09-11T03:50:00.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073378480000000001 2024-09-11T03:50:00.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:00.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.970934+0000", 2024-09-11T03:50:00.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003351812 2024-09-11T03:50:00.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.971224+0000", 2024-09-11T03:50:00.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029005399999999998 2024-09-11T03:50:00.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.972661+0000", 2024-09-11T03:50:00.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001436866 2024-09-11T03:50:00.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:00.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.972700+0000", 2024-09-11T03:50:00.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9341000000000002e-05 2024-09-11T03:50:00.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.972747+0000", 2024-09-11T03:50:00.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6409000000000002e-05 2024-09-11T03:50:00.208 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.208 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.208 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.208 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5201 2.e 2:750de3b7:::smithi17557870-8:head [copy-from ver 512] snapc e6=[e5,e1,e0,da,d1] ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-09-11T03:50:00.208 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:12.961903+0000", 2024-09-11T03:50:00.209 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 227.13936512800001, 2024-09-11T03:50:00.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022459643000000001, 2024-09-11T03:50:00.209 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.209 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.209 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:00.209 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:00.209 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:00.209 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:00.209 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5201 2024-09-11T03:50:00.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.209 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961903+0000", 2024-09-11T03:50:00.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961903+0000", 2024-09-11T03:50:00.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961903+0000", 2024-09-11T03:50:00.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0199999999999998e-07 2024-09-11T03:50:00.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961906+0000", 2024-09-11T03:50:00.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.119e-06 2024-09-11T03:50:00.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961911+0000", 2024-09-11T03:50:00.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7529999999999998e-06 2024-09-11T03:50:00.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961915+0000", 2024-09-11T03:50:00.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6320000000000001e-06 2024-09-11T03:50:00.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961955+0000", 2024-09-11T03:50:00.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0497999999999997e-05 2024-09-11T03:50:00.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:50:00.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961961+0000", 2024-09-11T03:50:00.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3800000000000002e-06 2024-09-11T03:50:00.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.963330+0000", 2024-09-11T03:50:00.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001368773 2024-09-11T03:50:00.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.963415+0000", 2024-09-11T03:50:00.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5579000000000002e-05 2024-09-11T03:50:00.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:50:00.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.970784+0000", 2024-09-11T03:50:00.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073691750000000004 2024-09-11T03:50:00.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:00.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.975393+0000", 2024-09-11T03:50:00.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046089010000000003 2024-09-11T03:50:00.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.976463+0000", 2024-09-11T03:50:00.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010695850000000001 2024-09-11T03:50:00.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:00.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.984271+0000", 2024-09-11T03:50:00.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0078078970000000003 2024-09-11T03:50:00.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:00.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.984316+0000", 2024-09-11T03:50:00.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5209e-05 2024-09-11T03:50:00.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.984363+0000", 2024-09-11T03:50:00.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6839999999999999e-05 2024-09-11T03:50:00.216 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.216 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.216 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.216 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5202 2.e 2:750de3b7:::smithi17557870-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e555)", 2024-09-11T03:50:00.216 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:12.961921+0000", 2024-09-11T03:50:00.216 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 227.13934693300001, 2024-09-11T03:50:00.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013412446999999999, 2024-09-11T03:50:00.217 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.217 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.217 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:00.217 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:00.217 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:00.217 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:00.217 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5202 2024-09-11T03:50:00.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.217 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961921+0000", 2024-09-11T03:50:00.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961921+0000", 2024-09-11T03:50:00.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961922+0000", 2024-09-11T03:50:00.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2499999999999999e-07 2024-09-11T03:50:00.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961923+0000", 2024-09-11T03:50:00.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.42e-06 2024-09-11T03:50:00.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961927+0000", 2024-09-11T03:50:00.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9960000000000004e-06 2024-09-11T03:50:00.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961930+0000", 2024-09-11T03:50:00.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9610000000000001e-06 2024-09-11T03:50:00.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961983+0000", 2024-09-11T03:50:00.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2967000000000003e-05 2024-09-11T03:50:00.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:50:00.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961986+0000", 2024-09-11T03:50:00.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0960000000000001e-06 2024-09-11T03:50:00.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.963568+0000", 2024-09-11T03:50:00.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015823530000000001 2024-09-11T03:50:00.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:00.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.963621+0000", 2024-09-11T03:50:00.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2358000000000003e-05 2024-09-11T03:50:00.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.974667+0000", 2024-09-11T03:50:00.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011046517 2024-09-11T03:50:00.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.974875+0000", 2024-09-11T03:50:00.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000207853 2024-09-11T03:50:00.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.975334+0000", 2024-09-11T03:50:00.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000458701 2024-09-11T03:50:00.222 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.222 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.222 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.222 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:5691 2.e e601/577 2:77dadb97:::smithi17557870-30:head v 601'877, mlcod=598'875)", 2024-09-11T03:50:00.223 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:59.009702+0000", 2024-09-11T03:50:00.223 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 181.091566336, 2024-09-11T03:50:00.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051323566000000001, 2024-09-11T03:50:00.223 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.223 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.223 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:00.223 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.009702+0000", 2024-09-11T03:50:00.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.009702+0000", 2024-09-11T03:50:00.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.009703+0000", 2024-09-11T03:50:00.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4750000000000001e-06 2024-09-11T03:50:00.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.059761+0000", 2024-09-11T03:50:00.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.050057579999999997 2024-09-11T03:50:00.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.059778+0000", 2024-09-11T03:50:00.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7051999999999999e-05 2024-09-11T03:50:00.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.059784+0000", 2024-09-11T03:50:00.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.648e-06 2024-09-11T03:50:00.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.059832+0000", 2024-09-11T03:50:00.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7828999999999999e-05 2024-09-11T03:50:00.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.059847+0000", 2024-09-11T03:50:00.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5778999999999999e-05 2024-09-11T03:50:00.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:00.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.060993+0000", 2024-09-11T03:50:00.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001145566 2024-09-11T03:50:00.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.061026+0000", 2024-09-11T03:50:00.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2636999999999999e-05 2024-09-11T03:50:00.227 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.227 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.227 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.227 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:6152 2.e e637/577 2:77dadb97:::smithi17557870-30:head v 637'891, mlcod=634'888)", 2024-09-11T03:50:00.227 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:47:35.714649+0000", 2024-09-11T03:50:00.227 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 144.386618833, 2024-09-11T03:50:00.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011875179, 2024-09-11T03:50:00.227 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.227 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.227 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:00.228 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714649+0000", 2024-09-11T03:50:00.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714649+0000", 2024-09-11T03:50:00.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714650+0000", 2024-09-11T03:50:00.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.44e-07 2024-09-11T03:50:00.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721021+0000", 2024-09-11T03:50:00.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0063714569999999996 2024-09-11T03:50:00.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721040+0000", 2024-09-11T03:50:00.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8590999999999999e-05 2024-09-11T03:50:00.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721048+0000", 2024-09-11T03:50:00.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9489999999999996e-06 2024-09-11T03:50:00.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721107+0000", 2024-09-11T03:50:00.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8736999999999997e-05 2024-09-11T03:50:00.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721123+0000", 2024-09-11T03:50:00.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6735000000000002e-05 2024-09-11T03:50:00.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:00.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.726469+0000", 2024-09-11T03:50:00.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053456140000000003 2024-09-11T03:50:00.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.726525+0000", 2024-09-11T03:50:00.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5551999999999999e-05 2024-09-11T03:50:00.231 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.231 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.231 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.232 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:6546 2.e e669/577 2:750de3b7:::smithi17557870-8:head v 669'909, mlcod=667'905)", 2024-09-11T03:50:00.232 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:07.227022+0000", 2024-09-11T03:50:00.232 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 112.87424587300001, 2024-09-11T03:50:00.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013091959, 2024-09-11T03:50:00.232 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.232 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.233 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:00.233 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.227022+0000", 2024-09-11T03:50:00.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.227022+0000", 2024-09-11T03:50:00.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.227023+0000", 2024-09-11T03:50:00.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7300000000000001e-07 2024-09-11T03:50:00.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.234175+0000", 2024-09-11T03:50:00.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071522260000000002 2024-09-11T03:50:00.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.234198+0000", 2024-09-11T03:50:00.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2492000000000001e-05 2024-09-11T03:50:00.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.234211+0000", 2024-09-11T03:50:00.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3534e-05 2024-09-11T03:50:00.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.234289+0000", 2024-09-11T03:50:00.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8078999999999996e-05 2024-09-11T03:50:00.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.234307+0000", 2024-09-11T03:50:00.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8206000000000002e-05 2024-09-11T03:50:00.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:00.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.240060+0000", 2024-09-11T03:50:00.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0057523069999999999 2024-09-11T03:50:00.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.240114+0000", 2024-09-11T03:50:00.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4642000000000003e-05 2024-09-11T03:50:00.237 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.237 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.237 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.237 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:7879 2.7 e779/745 2:e0ec5763:::smithi17557870-2:head v 779'1108, mlcod=778'1104)", 2024-09-11T03:50:00.237 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:49:57.174324+0000", 2024-09-11T03:50:00.237 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.926944314, 2024-09-11T03:50:00.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081549459999999997, 2024-09-11T03:50:00.237 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:00.237 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:00.237 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:00.238 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:00.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:00.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.174324+0000", 2024-09-11T03:50:00.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:00.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.174324+0000", 2024-09-11T03:50:00.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:00.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:00.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.174325+0000", 2024-09-11T03:50:00.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3990000000000001e-06 2024-09-11T03:50:00.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:00.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.180038+0000", 2024-09-11T03:50:00.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0057125869999999999 2024-09-11T03:50:00.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:00.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.180058+0000", 2024-09-11T03:50:00.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0271e-05 2024-09-11T03:50:00.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:00.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.180067+0000", 2024-09-11T03:50:00.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6470000000000004e-06 2024-09-11T03:50:00.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:00.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.180129+0000", 2024-09-11T03:50:00.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2193999999999994e-05 2024-09-11T03:50:00.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:00.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.180149+0000", 2024-09-11T03:50:00.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9553999999999999e-05 2024-09-11T03:50:00.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:00.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.182435+0000", 2024-09-11T03:50:00.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022862289999999999 2024-09-11T03:50:00.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:00.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:00.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:00.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.182479+0000", 2024-09-11T03:50:00.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4064999999999999e-05 2024-09-11T03:50:00.241 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.242 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.242 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.242 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:00.242 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:00.242 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:50:00.243 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:00.245 INFO:tasks.rados.rados.2.smithi175.stdout:3177: finishing write tid 4 to smithi17557870-11 2024-09-11T03:50:00.245 INFO:tasks.rados.rados.2.smithi175.stdout:3177: oid 11 updating version 0 to 622 2024-09-11T03:50:00.245 INFO:tasks.rados.rados.2.smithi175.stdout:3177: oid 11 version 622 is already newer than 621 2024-09-11T03:50:00.245 INFO:tasks.rados.rados.2.smithi175.stdout:3177: oid 11 updating version 622 to 623 2024-09-11T03:50:00.245 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 623 (ObjNum 1077 snap 352 seq_num 1077) dirty exists 2024-09-11T03:50:00.245 INFO:tasks.rados.rados.2.smithi175.stdout:3177: left oid 11 (ObjNum 1077 snap 352 seq_num 1077) 2024-09-11T03:50:00.245 INFO:tasks.rados.rados.2.smithi175.stdout:3179: finishing write tid 1 to smithi17557870-42 2024-09-11T03:50:00.245 INFO:tasks.rados.rados.2.smithi175.stdout:3180: finishing copy_from to smithi17557870-17 2024-09-11T03:50:00.246 INFO:tasks.rados.rados.2.smithi175.stdout:3180: got expected ENOENT (src dne) 2024-09-11T03:50:00.246 INFO:tasks.rados.rados.2.smithi175.stdout:3180: finishing copy_from racing read to smithi17557870-17 2024-09-11T03:50:00.246 INFO:tasks.rados.rados.2.smithi175.stdout:3179: finishing write tid 2 to smithi17557870-42 2024-09-11T03:50:00.246 INFO:tasks.rados.rados.2.smithi175.stdout:3179: finishing write tid 3 to smithi17557870-42 2024-09-11T03:50:00.246 INFO:tasks.rados.rados.2.smithi175.stdout:3179: finishing write tid 4 to smithi17557870-42 2024-09-11T03:50:00.246 INFO:tasks.rados.rados.2.smithi175.stdout:3179: oid 42 updating version 0 to 975 2024-09-11T03:50:00.246 INFO:tasks.rados.rados.2.smithi175.stdout:3179: oid 42 updating version 975 to 976 2024-09-11T03:50:00.246 INFO:tasks.rados.rados.2.smithi175.stdout:3179: oid 42 updating version 976 to 977 2024-09-11T03:50:00.246 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 977 (ObjNum 1078 snap 352 seq_num 1078) dirty exists 2024-09-11T03:50:00.246 INFO:tasks.rados.rados.2.smithi175.stdout:3179: left oid 42 (ObjNum 1078 snap 352 seq_num 1078) 2024-09-11T03:50:00.246 INFO:tasks.rados.rados.2.smithi175.stdout:3176: done (4 left) 2024-09-11T03:50:00.246 INFO:tasks.rados.rados.2.smithi175.stdout:3177: done (3 left) 2024-09-11T03:50:00.246 INFO:tasks.rados.rados.2.smithi175.stdout:3178: done (2 left) 2024-09-11T03:50:00.247 INFO:tasks.rados.rados.2.smithi175.stdout:3179: done (1 left) 2024-09-11T03:50:00.247 INFO:tasks.rados.rados.2.smithi175.stdout:3180: done (0 left) 2024-09-11T03:50:00.247 INFO:tasks.rados.rados.2.smithi175.stdout:3181: copy_from oid 3 from oid 15 current snap is 352 2024-09-11T03:50:00.247 INFO:tasks.rados.rados.2.smithi175.stdout:3182: read oid 48 snap -1 2024-09-11T03:50:00.247 INFO:tasks.rados.rados.2.smithi175.stdout:3182: expect deleted 2024-09-11T03:50:00.247 INFO:tasks.rados.rados.2.smithi175.stdout:3183: write oid 23 current snap is 352 2024-09-11T03:50:00.247 INFO:tasks.rados.rados.2.smithi175.stdout:3183: seq_num 1079 ranges {534753=799444,1990659=144094} 2024-09-11T03:50:00.247 INFO:tasks.rados.rados.2.smithi175.stdout:3183: writing smithi17557870-23 from 534753 to 1334197 tid 1 2024-09-11T03:50:00.247 INFO:tasks.rados.rados.2.smithi175.stdout:3183: writing smithi17557870-23 from 1990659 to 2134753 tid 2 2024-09-11T03:50:00.247 INFO:tasks.rados.rados.2.smithi175.stdout:3184: copy_from oid 39 from oid 24 current snap is 352 2024-09-11T03:50:00.247 INFO:tasks.rados.rados.2.smithi175.stdout:3181: finishing copy_from to smithi17557870-3 2024-09-11T03:50:00.247 INFO:tasks.rados.rados.2.smithi175.stdout:3181: got expected ENOENT (src dne) 2024-09-11T03:50:00.247 INFO:tasks.rados.rados.2.smithi175.stdout:3181: finishing copy_from racing read to smithi17557870-3 2024-09-11T03:50:00.247 INFO:tasks.rados.rados.2.smithi175.stdout:3181: done (3 left) 2024-09-11T03:50:00.248 INFO:tasks.rados.rados.2.smithi175.stdout:3182: done (2 left) 2024-09-11T03:50:00.248 INFO:tasks.rados.rados.2.smithi175.stdout:3185: copy_from oid 26 from oid 1 current snap is 352 2024-09-11T03:50:00.248 INFO:tasks.rados.rados.2.smithi175.stdout:3186: write (excl) oid 37 current snap is 352 2024-09-11T03:50:00.248 INFO:tasks.rados.rados.2.smithi175.stdout:3186: seq_num 1080 ranges {551560=591046,1765494=386066} 2024-09-11T03:50:00.248 INFO:tasks.rados.rados.2.smithi175.stdout:3186: writing smithi17557870-37 from 551560 to 1142606 tid 1 2024-09-11T03:50:00.248 INFO:tasks.rados.rados.2.smithi175.stdout:3186: writing smithi17557870-37 from 1765494 to 2151560 tid 2 2024-09-11T03:50:00.248 INFO:tasks.rados.rados.2.smithi175.stdout:3187: write oid 3 current snap is 352 2024-09-11T03:50:00.248 INFO:tasks.rados.rados.2.smithi175.stdout:3184: finishing copy_from to smithi17557870-39 2024-09-11T03:50:00.248 INFO:tasks.rados.rados.2.smithi175.stdout:3184: got expected ENOENT (src dne) 2024-09-11T03:50:00.248 INFO:tasks.rados.rados.2.smithi175.stdout:3184: finishing copy_from racing read to smithi17557870-39 2024-09-11T03:50:00.248 INFO:tasks.rados.rados.2.smithi175.stdout:3183: finishing write tid 1 to smithi17557870-23 2024-09-11T03:50:00.248 INFO:tasks.rados.rados.2.smithi175.stdout:3187: seq_num 1081 ranges {568367=666295,1707623=460744} 2024-09-11T03:50:00.248 INFO:tasks.rados.rados.2.smithi175.stdout:3187: writing smithi17557870-3 from 568367 to 1234662 tid 1 2024-09-11T03:50:00.248 INFO:tasks.rados.rados.2.smithi175.stdout:3187: writing smithi17557870-3 from 1707623 to 2168367 tid 2 2024-09-11T03:50:00.249 INFO:tasks.rados.rados.2.smithi175.stdout:3184: done (4 left) 2024-09-11T03:50:00.249 INFO:tasks.rados.rados.2.smithi175.stdout:3188: delete oid 8 current snap is 352 2024-09-11T03:50:00.249 INFO:tasks.rados.rados.2.smithi175.stdout:3183: finishing write tid 2 to smithi17557870-23 2024-09-11T03:50:00.249 INFO:tasks.rados.rados.2.smithi175.stdout:3183: finishing write tid 3 to smithi17557870-23 2024-09-11T03:50:00.249 INFO:tasks.rados.rados.2.smithi175.stdout:3183: finishing write tid 4 to smithi17557870-23 2024-09-11T03:50:00.249 INFO:tasks.rados.rados.2.smithi175.stdout:3183: oid 23 updating version 0 to 974 2024-09-11T03:50:00.249 INFO:tasks.rados.rados.2.smithi175.stdout:3183: oid 23 updating version 974 to 975 2024-09-11T03:50:00.249 INFO:tasks.rados.rados.2.smithi175.stdout:3183: oid 23 version 975 is already newer than 973 2024-09-11T03:50:00.249 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 975 (ObjNum 1079 snap 352 seq_num 1079) dirty exists 2024-09-11T03:50:00.249 INFO:tasks.rados.rados.2.smithi175.stdout:3183: left oid 23 (ObjNum 1079 snap 352 seq_num 1079) 2024-09-11T03:50:00.249 INFO:tasks.rados.rados.2.smithi175.stdout:3185: finishing copy_from racing read to smithi17557870-26 2024-09-11T03:50:00.249 INFO:tasks.rados.rados.2.smithi175.stdout:3186: finishing write tid 1 to smithi17557870-37 2024-09-11T03:50:00.249 INFO:tasks.rados.rados.2.smithi175.stdout:3185: finishing copy_from to smithi17557870-26 2024-09-11T03:50:00.250 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 1006 (ObjNum 973 snap 313 seq_num 973) dirty exists 2024-09-11T03:50:00.250 INFO:tasks.rados.rados.2.smithi175.stdout:3186: finishing write tid 2 to smithi17557870-37 2024-09-11T03:50:00.250 INFO:tasks.rados.rados.2.smithi175.stdout:3186: finishing write tid 3 to smithi17557870-37 2024-09-11T03:50:00.250 INFO:tasks.rados.rados.2.smithi175.stdout:3186: finishing write tid 4 to smithi17557870-37 2024-09-11T03:50:00.250 INFO:tasks.rados.rados.2.smithi175.stdout:3186: oid 37 updating version 0 to 907 2024-09-11T03:50:00.250 INFO:tasks.rados.rados.2.smithi175.stdout:3186: oid 37 version 907 is already newer than 906 2024-09-11T03:50:00.250 INFO:tasks.rados.rados.2.smithi175.stdout:3186: oid 37 version 907 is already newer than 905 2024-09-11T03:50:00.250 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 907 (ObjNum 1080 snap 352 seq_num 1080) dirty exists 2024-09-11T03:50:00.250 INFO:tasks.rados.rados.2.smithi175.stdout:3186: left oid 37 (ObjNum 1080 snap 352 seq_num 1080) 2024-09-11T03:50:00.250 INFO:tasks.rados.rados.2.smithi175.stdout:3187: finishing write tid 1 to smithi17557870-3 2024-09-11T03:50:00.250 INFO:tasks.rados.rados.2.smithi175.stdout:3183: done (4 left) 2024-09-11T03:50:00.250 INFO:tasks.rados.rados.2.smithi175.stdout:3185: done (3 left) 2024-09-11T03:50:00.250 INFO:tasks.rados.rados.2.smithi175.stdout:3186: done (2 left) 2024-09-11T03:50:00.250 INFO:tasks.rados.rados.2.smithi175.stdout:3188: done (1 left) 2024-09-11T03:50:00.250 INFO:tasks.rados.rados.2.smithi175.stdout:3189: snap_remove snap 315 2024-09-11T03:50:00.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:00.272 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:00.272 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:00.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:00.399 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:00.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:00.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:00 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:50:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:00 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:50:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:00 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:50:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:00 smithi121 ceph-mon[30729]: 2.14 scrub ok 2024-09-11T03:50:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:00 smithi121 ceph-mon[30729]: pgmap v1326: 77 pgs: 77 active+clean; 461 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 7.7 MiB/s wr, 25 op/s 2024-09-11T03:50:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:00 smithi121 ceph-mon[30729]: osdmap e781: 12 total, 12 up, 12 in 2024-09-11T03:50:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:00 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4012925461' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:00 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/174023279' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "56"} : dispatch 2024-09-11T03:50:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:00 smithi121 ceph-mon[30729]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-11T03:50:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:00 smithi121 ceph-mon[30729]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-11T03:50:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:00 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:50:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:00 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:50:00.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:00 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:50:00.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:00 smithi175 ceph-mon[30894]: 2.14 scrub ok 2024-09-11T03:50:00.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:00 smithi175 ceph-mon[30894]: pgmap v1326: 77 pgs: 77 active+clean; 461 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 7.7 MiB/s wr, 25 op/s 2024-09-11T03:50:00.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:00 smithi175 ceph-mon[30894]: osdmap e781: 12 total, 12 up, 12 in 2024-09-11T03:50:00.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:00 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4012925461' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:00.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:00 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/174023279' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "56"} : dispatch 2024-09-11T03:50:00.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:00 smithi175 ceph-mon[30894]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-11T03:50:00.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:00 smithi175 ceph-mon[30894]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-11T03:50:00.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:00.526 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:00.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:00.628 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:00.651 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:00.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:00.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:00.753 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:00.775 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:00.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:00.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:00.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:00.899 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:00.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:00.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:01.001 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:01.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:01.027 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:01.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:01.047 INFO:tasks.rados.rados.2.smithi175.stdout:3189: done (1 left) 2024-09-11T03:50:01.048 INFO:tasks.rados.rados.2.smithi175.stdout:3190: delete oid 29 current snap is 352 2024-09-11T03:50:01.048 INFO:tasks.rados.rados.2.smithi175.stdout:3187: finishing write tid 2 to smithi17557870-3 2024-09-11T03:50:01.048 INFO:tasks.rados.rados.2.smithi175.stdout:3187: finishing write tid 3 to smithi17557870-3 2024-09-11T03:50:01.048 INFO:tasks.rados.rados.2.smithi175.stdout:3187: finishing write tid 4 to smithi17557870-3 2024-09-11T03:50:01.048 INFO:tasks.rados.rados.2.smithi175.stdout:3187: oid 3 updating version 0 to 1206 2024-09-11T03:50:01.048 INFO:tasks.rados.rados.2.smithi175.stdout:3187: oid 3 version 1206 is already newer than 1205 2024-09-11T03:50:01.049 INFO:tasks.rados.rados.2.smithi175.stdout:3187: oid 3 updating version 1206 to 1207 2024-09-11T03:50:01.049 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 1207 (ObjNum 1081 snap 352 seq_num 1081) dirty exists 2024-09-11T03:50:01.049 INFO:tasks.rados.rados.2.smithi175.stdout:3187: left oid 3 (ObjNum 1081 snap 352 seq_num 1081) 2024-09-11T03:50:01.057 INFO:tasks.rados.rados.2.smithi175.stdout:3187: done (1 left) 2024-09-11T03:50:01.057 INFO:tasks.rados.rados.2.smithi175.stdout:3190: done (0 left) 2024-09-11T03:50:01.057 INFO:tasks.rados.rados.2.smithi175.stdout:3191: rollback oid 47 current snap is 352 2024-09-11T03:50:01.057 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 47 to 345 2024-09-11T03:50:01.057 INFO:tasks.rados.rados.2.smithi175.stdout:3192: rollback oid 38 current snap is 352 2024-09-11T03:50:01.058 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 38 to 339 2024-09-11T03:50:01.058 INFO:tasks.rados.rados.2.smithi175.stdout:3193: snap_create 2024-09-11T03:50:01.059 INFO:tasks.rados.rados.2.smithi175.stdout:3192: finishing rollback tid 1 to smithi17557870-38 2024-09-11T03:50:01.060 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 1205 (ObjNum 81 snap 0 seq_num 0) dirty dne 2024-09-11T03:50:01.062 INFO:tasks.rados.rados.2.smithi175.stdout:3191: finishing rollback tid 0 to smithi17557870-47 2024-09-11T03:50:01.062 INFO:tasks.rados.rados.2.smithi175.stdout:3191: finishing rollback tid 1 to smithi17557870-47 2024-09-11T03:50:01.063 INFO:tasks.rados.rados.2.smithi175.stdout:3191: finishing rollback tid 2 to smithi17557870-47 2024-09-11T03:50:01.063 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 1053 (ObjNum 917 snap 294 seq_num 917) dirty exists 2024-09-11T03:50:01.128 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:01.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:01.152 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:01.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:01.253 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:01.274 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:01.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:01.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:01.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:01 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:50:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:01 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:50:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:01 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:50:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:01 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:50:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4012925461' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/174023279' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "56"}]': finished 2024-09-11T03:50:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:01 smithi084 ceph-mon[27254]: osdmap e782: 12 total, 12 up, 12 in 2024-09-11T03:50:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:01 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"} : dispatch 2024-09-11T03:50:01.402 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:01.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:01.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:01.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:01 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:50:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:01 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:50:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:01 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:50:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:01 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:50:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4012925461' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/174023279' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "56"}]': finished 2024-09-11T03:50:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:01 smithi121 ceph-mon[30729]: osdmap e782: 12 total, 12 up, 12 in 2024-09-11T03:50:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:01 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"} : dispatch 2024-09-11T03:50:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:01 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:50:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:01 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:50:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:01 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:50:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:01 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:50:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4012925461' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/174023279' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "56"}]': finished 2024-09-11T03:50:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:01 smithi175 ceph-mon[30894]: osdmap e782: 12 total, 12 up, 12 in 2024-09-11T03:50:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:01 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"} : dispatch 2024-09-11T03:50:01.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:01.528 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:01.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:01.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:01.630 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:01.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:01.657 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:01.657 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:01.758 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:01.784 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:01.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:01.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:01.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:01.912 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:01.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:01.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:02.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:02.040 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:02.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:02.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:02.050 INFO:tasks.rados.rados.2.smithi175.stdout:3191: done (2 left) 2024-09-11T03:50:02.050 INFO:tasks.rados.rados.2.smithi175.stdout:3192: done (1 left) 2024-09-11T03:50:02.051 INFO:tasks.rados.rados.2.smithi175.stdout:3193: done (0 left) 2024-09-11T03:50:02.051 INFO:tasks.rados.rados.2.smithi175.stdout:3194: read oid 8 snap -1 2024-09-11T03:50:02.051 INFO:tasks.rados.rados.2.smithi175.stdout:3194: expect deleted 2024-09-11T03:50:02.051 INFO:tasks.rados.rados.2.smithi175.stdout:3195: read oid 18 snap 352 2024-09-11T03:50:02.051 INFO:tasks.rados.rados.2.smithi175.stdout:3195: expect (ObjNum 829 snap 263 seq_num 829) 2024-09-11T03:50:02.051 INFO:tasks.rados.rados.2.smithi175.stdout:3196: read oid 42 snap 348 2024-09-11T03:50:02.051 INFO:tasks.rados.rados.2.smithi175.stdout:3196: expect (ObjNum 1062 snap 347 seq_num 1062) 2024-09-11T03:50:02.051 INFO:tasks.rados.rados.2.smithi175.stdout:3197: delete oid 36 current snap is 353 2024-09-11T03:50:02.062 INFO:tasks.rados.rados.2.smithi175.stdout:3196: expect (ObjNum 1062 snap 347 seq_num 1062) 2024-09-11T03:50:02.142 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:02.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:02.167 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:02.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:02.267 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:02.287 INFO:tasks.rados.rados.2.smithi175.stdout:3195: expect (ObjNum 829 snap 263 seq_num 829) 2024-09-11T03:50:02.289 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:02.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:02.290 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:02.373 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:02.390 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:02.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:02 smithi084 ceph-mon[27254]: pgmap v1329: 77 pgs: 77 active+clean; 461 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 13 op/s 2024-09-11T03:50:02.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:02 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"}]': finished 2024-09-11T03:50:02.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:02 smithi084 ceph-mon[27254]: osdmap e783: 12 total, 12 up, 12 in 2024-09-11T03:50:02.414 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:02.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:02.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:02 smithi121 ceph-mon[30729]: pgmap v1329: 77 pgs: 77 active+clean; 461 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 13 op/s 2024-09-11T03:50:02.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:02 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"}]': finished 2024-09-11T03:50:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:02 smithi121 ceph-mon[30729]: osdmap e783: 12 total, 12 up, 12 in 2024-09-11T03:50:02.469 INFO:tasks.rados.rados.2.smithi175.stdout:3194: done (3 left) 2024-09-11T03:50:02.469 INFO:tasks.rados.rados.2.smithi175.stdout:3195: done (2 left) 2024-09-11T03:50:02.469 INFO:tasks.rados.rados.2.smithi175.stdout:3196: done (1 left) 2024-09-11T03:50:02.470 INFO:tasks.rados.rados.2.smithi175.stdout:3197: done (0 left) 2024-09-11T03:50:02.470 INFO:tasks.rados.rados.2.smithi175.stdout:3198: read oid 9 snap -1 2024-09-11T03:50:02.470 INFO:tasks.rados.rados.2.smithi175.stdout:3198: expect (ObjNum 1066 snap 348 seq_num 1066) 2024-09-11T03:50:02.470 INFO:tasks.rados.rados.2.smithi175.stdout:3199: snap_remove snap 305 2024-09-11T03:50:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:02 smithi175 ceph-mon[30894]: pgmap v1329: 77 pgs: 77 active+clean; 461 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 13 op/s 2024-09-11T03:50:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:02 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"}]': finished 2024-09-11T03:50:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:02 smithi175 ceph-mon[30894]: osdmap e783: 12 total, 12 up, 12 in 2024-09-11T03:50:02.515 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:02.540 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:02.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:02.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:02.642 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:02.664 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:02.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:02.665 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:02.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:02.792 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:02.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:02.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:02.837 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T03:50:02.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:02.919 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:02.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:02.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:03.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:03.045 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:03.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:03.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:03.060 INFO:tasks.rados.rados.2.smithi175.stdout:3199: done (1 left) 2024-09-11T03:50:03.060 INFO:tasks.rados.rados.2.smithi175.stdout:3200: read oid 11 snap 341 2024-09-11T03:50:03.060 INFO:tasks.rados.rados.2.smithi175.stdout:3200: expect (ObjNum 1029 snap 338 seq_num 1029) 2024-09-11T03:50:03.061 INFO:tasks.rados.rados.2.smithi175.stdout:3201: write oid 29 current snap is 353 2024-09-11T03:50:03.061 INFO:tasks.rados.rados.2.smithi175.stdout:3201: seq_num 1085 ranges {635595=567291,1825198=410397} 2024-09-11T03:50:03.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:03 smithi084 ceph-mon[27254]: 1.1a scrub starts 2024-09-11T03:50:03.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:03 smithi084 ceph-mon[27254]: 1.1a scrub ok 2024-09-11T03:50:03.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:03 smithi084 ceph-mon[27254]: 2.18 scrub starts 2024-09-11T03:50:03.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:03 smithi084 ceph-mon[27254]: 2.18 scrub ok 2024-09-11T03:50:03.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:03 smithi084 ceph-mon[27254]: osdmap e784: 12 total, 12 up, 12 in 2024-09-11T03:50:03.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:03 smithi084 ceph-mon[27254]: 2.11 scrub starts 2024-09-11T03:50:03.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:03 smithi084 ceph-mon[27254]: 2.11 scrub ok 2024-09-11T03:50:03.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "56"} : dispatch 2024-09-11T03:50:03.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:50:03.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:50:03.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:50:03.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:50:03.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:50:03.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:50:03.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:03 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:50:03.069 INFO:tasks.rados.rados.2.smithi175.stdout:3201: writing smithi17557870-29 from 635595 to 1202886 tid 1 2024-09-11T03:50:03.075 INFO:tasks.rados.rados.2.smithi175.stdout:3201: writing smithi17557870-29 from 1825198 to 2235595 tid 2 2024-09-11T03:50:03.075 INFO:tasks.rados.rados.2.smithi175.stdout:3202: write oid 36 current snap is 353 2024-09-11T03:50:03.075 INFO:tasks.rados.rados.2.smithi175.stdout:3202: seq_num 1086 ranges {652402=642540,1883680=368722} 2024-09-11T03:50:03.082 INFO:tasks.rados.rados.2.smithi175.stdout:3202: writing smithi17557870-36 from 652402 to 1294942 tid 1 2024-09-11T03:50:03.088 INFO:tasks.rados.rados.2.smithi175.stdout:3202: writing smithi17557870-36 from 1883680 to 2252402 tid 2 2024-09-11T03:50:03.088 INFO:tasks.rados.rados.2.smithi175.stdout:3203: write oid 14 current snap is 353 2024-09-11T03:50:03.088 INFO:tasks.rados.rados.2.smithi175.stdout:3203: seq_num 1087 ranges {669209=717789,1942162=327047} 2024-09-11T03:50:03.095 INFO:tasks.rados.rados.2.smithi175.stdout:3203: writing smithi17557870-14 from 669209 to 1386998 tid 1 2024-09-11T03:50:03.100 INFO:tasks.rados.rados.2.smithi175.stdout:3203: writing smithi17557870-14 from 1942162 to 2269209 tid 2 2024-09-11T03:50:03.100 INFO:tasks.rados.rados.2.smithi175.stdout:3204: snap_remove snap 338 2024-09-11T03:50:03.147 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:03.168 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:03.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:03.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:03.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:03.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:03.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:03.301 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:50:03.301 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:50:03.317 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:50:03.318 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:50:03.336 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:03.336 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:50:03.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:50:03.353 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:50:03.354 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:50:03.372 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:50:03.373 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:50:03.389 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:50:03.389 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:50:03.394 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:03.408 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:50:03.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:50:03.418 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:03.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:03.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:03.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:03 smithi121 ceph-mon[30729]: 1.1a scrub starts 2024-09-11T03:50:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:03 smithi121 ceph-mon[30729]: 1.1a scrub ok 2024-09-11T03:50:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:03 smithi121 ceph-mon[30729]: 2.18 scrub starts 2024-09-11T03:50:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:03 smithi121 ceph-mon[30729]: 2.18 scrub ok 2024-09-11T03:50:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:03 smithi121 ceph-mon[30729]: osdmap e784: 12 total, 12 up, 12 in 2024-09-11T03:50:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:03 smithi121 ceph-mon[30729]: 2.11 scrub starts 2024-09-11T03:50:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:03 smithi121 ceph-mon[30729]: 2.11 scrub ok 2024-09-11T03:50:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "56"} : dispatch 2024-09-11T03:50:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:50:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:50:03.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:50:03.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:50:03.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:50:03.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:50:03.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:03 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:50:03.429 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:50:03.429 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:50:03.446 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:50:03.446 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:50:03.463 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:50:03.463 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:50:03.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:03 smithi175 ceph-mon[30894]: 1.1a scrub starts 2024-09-11T03:50:03.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:03 smithi175 ceph-mon[30894]: 1.1a scrub ok 2024-09-11T03:50:03.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:03 smithi175 ceph-mon[30894]: 2.18 scrub starts 2024-09-11T03:50:03.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:03 smithi175 ceph-mon[30894]: 2.18 scrub ok 2024-09-11T03:50:03.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:03 smithi175 ceph-mon[30894]: osdmap e784: 12 total, 12 up, 12 in 2024-09-11T03:50:03.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:03 smithi175 ceph-mon[30894]: 2.11 scrub starts 2024-09-11T03:50:03.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:03 smithi175 ceph-mon[30894]: 2.11 scrub ok 2024-09-11T03:50:03.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "56"} : dispatch 2024-09-11T03:50:03.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:50:03.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:50:03.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:50:03.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:50:03.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:50:03.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:50:03.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:03 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:50:03.480 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:50:03.480 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:50:03.497 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:50:03.497 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:50:03.520 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:03.542 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:03.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:03.543 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:03.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:03.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:03.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:03.769 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:03.791 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:03.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:03.893 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:03.915 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:03.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:04.017 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:04.039 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:04.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:04.040 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:04.060 INFO:tasks.rados.rados.2.smithi175.stdout:3204: done (5 left) 2024-09-11T03:50:04.061 INFO:tasks.rados.rados.2.smithi175.stdout:3205: snap_remove snap 337 2024-09-11T03:50:04.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:04.163 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:04.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:04.169 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:50:04.265 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:04.287 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:04.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:04.288 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:04.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:04 smithi084 ceph-mon[27254]: 2.f scrub starts 2024-09-11T03:50:04.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:04 smithi084 ceph-mon[27254]: 2.f scrub ok 2024-09-11T03:50:04.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:04 smithi084 ceph-mon[27254]: 2.12 scrub starts 2024-09-11T03:50:04.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:04 smithi084 ceph-mon[27254]: 2.12 scrub ok 2024-09-11T03:50:04.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:04 smithi084 ceph-mon[27254]: 2.16 scrub starts 2024-09-11T03:50:04.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:04 smithi084 ceph-mon[27254]: 2.16 scrub ok 2024-09-11T03:50:04.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:04 smithi084 ceph-mon[27254]: 2.13 scrub starts 2024-09-11T03:50:04.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:04 smithi084 ceph-mon[27254]: 2.13 scrub ok 2024-09-11T03:50:04.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:04 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "56"}]': finished 2024-09-11T03:50:04.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:04 smithi084 ceph-mon[27254]: osdmap e785: 12 total, 12 up, 12 in 2024-09-11T03:50:04.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:04 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2248834607' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:04.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:04 smithi084 ceph-mon[27254]: 2.17 scrub starts 2024-09-11T03:50:04.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:04 smithi084 ceph-mon[27254]: 2.17 scrub ok 2024-09-11T03:50:04.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:04 smithi084 ceph-mon[27254]: osdmap e786: 12 total, 12 up, 12 in 2024-09-11T03:50:04.389 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:04.410 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:04.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:04.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:04.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:04 smithi121 ceph-mon[30729]: 2.f scrub starts 2024-09-11T03:50:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:04 smithi121 ceph-mon[30729]: 2.f scrub ok 2024-09-11T03:50:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:04 smithi121 ceph-mon[30729]: 2.12 scrub starts 2024-09-11T03:50:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:04 smithi121 ceph-mon[30729]: 2.12 scrub ok 2024-09-11T03:50:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:04 smithi121 ceph-mon[30729]: 2.16 scrub starts 2024-09-11T03:50:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:04 smithi121 ceph-mon[30729]: 2.16 scrub ok 2024-09-11T03:50:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:04 smithi121 ceph-mon[30729]: 2.13 scrub starts 2024-09-11T03:50:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:04 smithi121 ceph-mon[30729]: 2.13 scrub ok 2024-09-11T03:50:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:04 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "56"}]': finished 2024-09-11T03:50:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:04 smithi121 ceph-mon[30729]: osdmap e785: 12 total, 12 up, 12 in 2024-09-11T03:50:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:04 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2248834607' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:04 smithi121 ceph-mon[30729]: 2.17 scrub starts 2024-09-11T03:50:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:04 smithi121 ceph-mon[30729]: 2.17 scrub ok 2024-09-11T03:50:04.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:04 smithi121 ceph-mon[30729]: osdmap e786: 12 total, 12 up, 12 in 2024-09-11T03:50:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:04 smithi175 ceph-mon[30894]: 2.f scrub starts 2024-09-11T03:50:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:04 smithi175 ceph-mon[30894]: 2.f scrub ok 2024-09-11T03:50:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:04 smithi175 ceph-mon[30894]: 2.12 scrub starts 2024-09-11T03:50:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:04 smithi175 ceph-mon[30894]: 2.12 scrub ok 2024-09-11T03:50:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:04 smithi175 ceph-mon[30894]: 2.16 scrub starts 2024-09-11T03:50:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:04 smithi175 ceph-mon[30894]: 2.16 scrub ok 2024-09-11T03:50:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:04 smithi175 ceph-mon[30894]: 2.13 scrub starts 2024-09-11T03:50:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:04 smithi175 ceph-mon[30894]: 2.13 scrub ok 2024-09-11T03:50:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:04 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "56"}]': finished 2024-09-11T03:50:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:04 smithi175 ceph-mon[30894]: osdmap e785: 12 total, 12 up, 12 in 2024-09-11T03:50:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:04 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2248834607' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:04 smithi175 ceph-mon[30894]: 2.17 scrub starts 2024-09-11T03:50:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:04 smithi175 ceph-mon[30894]: 2.17 scrub ok 2024-09-11T03:50:04.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:04 smithi175 ceph-mon[30894]: osdmap e786: 12 total, 12 up, 12 in 2024-09-11T03:50:04.492 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:50:04.492 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:50:04.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:04.533 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:04.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:04.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:04.635 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:04.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:04.660 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:04.660 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:04.761 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:04.786 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:04.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:04.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:04.888 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:04.914 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:04.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:04.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:05.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:05.041 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:05.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:05.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:05.063 INFO:tasks.rados.rados.2.smithi175.stdout:3205: done (5 left) 2024-09-11T03:50:05.063 INFO:tasks.rados.rados.2.smithi175.stdout:3206: delete oid 27 current snap is 353 2024-09-11T03:50:05.063 INFO:tasks.rados.rados.2.smithi175.stdout:3198: expect (ObjNum 1066 snap 348 seq_num 1066) 2024-09-11T03:50:05.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:05.167 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:05.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:05.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:05.204 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T03:50:05.269 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:05.270 INFO:tasks.rados.rados.2.smithi175.stdout:3201: finishing write tid 1 to smithi17557870-29 2024-09-11T03:50:05.270 INFO:tasks.rados.rados.2.smithi175.stdout:3201: finishing write tid 2 to smithi17557870-29 2024-09-11T03:50:05.270 INFO:tasks.rados.rados.2.smithi175.stdout:3201: finishing write tid 3 to smithi17557870-29 2024-09-11T03:50:05.270 INFO:tasks.rados.rados.2.smithi175.stdout:3198: done (5 left) 2024-09-11T03:50:05.270 INFO:tasks.rados.rados.2.smithi175.stdout:3206: done (4 left) 2024-09-11T03:50:05.271 INFO:tasks.rados.rados.2.smithi175.stdout:3207: rollback oid 3 current snap is 353 2024-09-11T03:50:05.271 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 3 to 286 2024-09-11T03:50:05.271 INFO:tasks.rados.rados.2.smithi175.stdout:3201: finishing write tid 4 to smithi17557870-29 2024-09-11T03:50:05.271 INFO:tasks.rados.rados.2.smithi175.stdout:3201: oid 29 updating version 0 to 1159 2024-09-11T03:50:05.271 INFO:tasks.rados.rados.2.smithi175.stdout:3201: oid 29 version 1159 is already newer than 1157 2024-09-11T03:50:05.271 INFO:tasks.rados.rados.2.smithi175.stdout:3201: oid 29 version 1159 is already newer than 1158 2024-09-11T03:50:05.271 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 1159 (ObjNum 1085 snap 353 seq_num 1085) dirty exists 2024-09-11T03:50:05.271 INFO:tasks.rados.rados.2.smithi175.stdout:3201: left oid 29 (ObjNum 1085 snap 353 seq_num 1085) 2024-09-11T03:50:05.271 INFO:tasks.rados.rados.2.smithi175.stdout:3200: expect (ObjNum 1029 snap 338 seq_num 1029) 2024-09-11T03:50:05.290 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:05.291 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:05.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:05.292 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:05.393 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:05.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:05 smithi084 ceph-mon[27254]: 2.10 scrub starts 2024-09-11T03:50:05.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:05 smithi084 ceph-mon[27254]: 2.10 scrub ok 2024-09-11T03:50:05.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:05 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:50:05.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:05 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:50:05.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:05 smithi084 ceph-mon[27254]: 2.1e scrub starts 2024-09-11T03:50:05.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:05 smithi084 ceph-mon[27254]: 2.1e scrub ok 2024-09-11T03:50:05.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:05 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:50:05.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:05 smithi084 ceph-mon[27254]: 2.15 scrub ok 2024-09-11T03:50:05.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:05 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:50:05.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:05 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:50:05.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:05 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:50:05.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:05 smithi084 ceph-mon[27254]: 2.14 scrub ok 2024-09-11T03:50:05.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:05 smithi084 ceph-mon[27254]: osdmap e787: 12 total, 12 up, 12 in 2024-09-11T03:50:05.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:05.417 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:05.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:05 smithi121 ceph-mon[30729]: 2.10 scrub starts 2024-09-11T03:50:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:05 smithi121 ceph-mon[30729]: 2.10 scrub ok 2024-09-11T03:50:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:05 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:50:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:05 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:50:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:05 smithi121 ceph-mon[30729]: 2.1e scrub starts 2024-09-11T03:50:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:05 smithi121 ceph-mon[30729]: 2.1e scrub ok 2024-09-11T03:50:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:05 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:50:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:05 smithi121 ceph-mon[30729]: 2.15 scrub ok 2024-09-11T03:50:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:05 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:50:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:05 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:50:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:05 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:50:05.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:05 smithi121 ceph-mon[30729]: 2.14 scrub ok 2024-09-11T03:50:05.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:05 smithi121 ceph-mon[30729]: osdmap e787: 12 total, 12 up, 12 in 2024-09-11T03:50:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:05 smithi175 ceph-mon[30894]: 2.10 scrub starts 2024-09-11T03:50:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:05 smithi175 ceph-mon[30894]: 2.10 scrub ok 2024-09-11T03:50:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:05 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:50:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:05 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:50:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:05 smithi175 ceph-mon[30894]: 2.1e scrub starts 2024-09-11T03:50:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:05 smithi175 ceph-mon[30894]: 2.1e scrub ok 2024-09-11T03:50:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:05 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:50:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:05 smithi175 ceph-mon[30894]: 2.15 scrub ok 2024-09-11T03:50:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:05 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:50:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:05 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:50:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:05 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:50:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:05 smithi175 ceph-mon[30894]: 2.14 scrub ok 2024-09-11T03:50:05.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:05 smithi175 ceph-mon[30894]: osdmap e787: 12 total, 12 up, 12 in 2024-09-11T03:50:05.501 INFO:tasks.rados.rados.2.smithi175.stdout:3202: finishing write tid 1 to smithi17557870-36 2024-09-11T03:50:05.501 INFO:tasks.rados.rados.2.smithi175.stdout:3202: finishing write tid 2 to smithi17557870-36 2024-09-11T03:50:05.502 INFO:tasks.rados.rados.2.smithi175.stdout:3202: finishing write tid 3 to smithi17557870-36 2024-09-11T03:50:05.502 INFO:tasks.rados.rados.2.smithi175.stdout:3202: finishing write tid 4 to smithi17557870-36 2024-09-11T03:50:05.502 INFO:tasks.rados.rados.2.smithi175.stdout:3202: oid 36 updating version 0 to 979 2024-09-11T03:50:05.502 INFO:tasks.rados.rados.2.smithi175.stdout:3202: oid 36 version 979 is already newer than 977 2024-09-11T03:50:05.502 INFO:tasks.rados.rados.2.smithi175.stdout:3202: oid 36 version 979 is already newer than 978 2024-09-11T03:50:05.502 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 979 (ObjNum 1086 snap 353 seq_num 1086) dirty exists 2024-09-11T03:50:05.502 INFO:tasks.rados.rados.2.smithi175.stdout:3202: left oid 36 (ObjNum 1086 snap 353 seq_num 1086) 2024-09-11T03:50:05.502 INFO:tasks.rados.rados.2.smithi175.stdout:3203: finishing write tid 1 to smithi17557870-14 2024-09-11T03:50:05.502 INFO:tasks.rados.rados.2.smithi175.stdout:3200: done (4 left) 2024-09-11T03:50:05.502 INFO:tasks.rados.rados.2.smithi175.stdout:3201: done (3 left) 2024-09-11T03:50:05.502 INFO:tasks.rados.rados.2.smithi175.stdout:3202: done (2 left) 2024-09-11T03:50:05.503 INFO:tasks.rados.rados.2.smithi175.stdout:3208: delete oid 18 current snap is 353 2024-09-11T03:50:05.503 INFO:tasks.rados.rados.2.smithi175.stdout:3203: finishing write tid 2 to smithi17557870-14 2024-09-11T03:50:05.503 INFO:tasks.rados.rados.2.smithi175.stdout:3203: finishing write tid 3 to smithi17557870-14 2024-09-11T03:50:05.503 INFO:tasks.rados.rados.2.smithi175.stdout:3203: finishing write tid 4 to smithi17557870-14 2024-09-11T03:50:05.503 INFO:tasks.rados.rados.2.smithi175.stdout:3203: oid 14 updating version 0 to 1042 2024-09-11T03:50:05.503 INFO:tasks.rados.rados.2.smithi175.stdout:3203: oid 14 updating version 1042 to 1044 2024-09-11T03:50:05.503 INFO:tasks.rados.rados.2.smithi175.stdout:3203: oid 14 version 1044 is already newer than 1043 2024-09-11T03:50:05.503 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 1044 (ObjNum 1087 snap 353 seq_num 1087) dirty exists 2024-09-11T03:50:05.503 INFO:tasks.rados.rados.2.smithi175.stdout:3203: left oid 14 (ObjNum 1087 snap 353 seq_num 1087) 2024-09-11T03:50:05.503 INFO:tasks.rados.rados.2.smithi175.stdout:3207: finishing rollback tid 0 to smithi17557870-3 2024-09-11T03:50:05.503 INFO:tasks.rados.rados.2.smithi175.stdout:3207: finishing rollback tid 1 to smithi17557870-3 2024-09-11T03:50:05.504 INFO:tasks.rados.rados.2.smithi175.stdout:3207: finishing rollback tid 2 to smithi17557870-3 2024-09-11T03:50:05.504 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 1221 (ObjNum 765 snap 247 seq_num 765) dirty exists 2024-09-11T03:50:05.504 INFO:tasks.rados.rados.2.smithi175.stdout:3203: done (2 left) 2024-09-11T03:50:05.505 INFO:tasks.rados.rados.2.smithi175.stdout:3207: done (1 left) 2024-09-11T03:50:05.505 INFO:tasks.rados.rados.2.smithi175.stdout:3208: done (0 left) 2024-09-11T03:50:05.505 INFO:tasks.rados.rados.2.smithi175.stdout:3209: read oid 23 snap -1 2024-09-11T03:50:05.505 INFO:tasks.rados.rados.2.smithi175.stdout:3209: expect (ObjNum 1079 snap 352 seq_num 1079) 2024-09-11T03:50:05.505 INFO:tasks.rados.rados.2.smithi175.stdout:3210: write (excl) oid 27 current snap is 353 2024-09-11T03:50:05.505 INFO:tasks.rados.rados.2.smithi175.stdout:3210: seq_num 1090 ranges {719630=659889,2117608=202022} 2024-09-11T03:50:05.510 INFO:tasks.rados.rados.2.smithi175.stdout:3210: writing smithi17557870-27 from 719630 to 1379519 tid 1 2024-09-11T03:50:05.516 INFO:tasks.rados.rados.2.smithi175.stdout:3210: writing smithi17557870-27 from 2117608 to 2319630 tid 2 2024-09-11T03:50:05.516 INFO:tasks.rados.rados.2.smithi175.stdout:3211: copy_from oid 45 from oid 32 current snap is 353 2024-09-11T03:50:05.516 INFO:tasks.rados.rados.2.smithi175.stdout:3209: expect (ObjNum 1079 snap 352 seq_num 1079) 2024-09-11T03:50:05.517 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:05.539 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:05.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:05.540 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:05.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:05.645 INFO:tasks.rados.rados.2.smithi175.stdout:3210: finishing write tid 1 to smithi17557870-27 2024-09-11T03:50:05.645 INFO:tasks.rados.rados.2.smithi175.stdout:3210: finishing write tid 2 to smithi17557870-27 2024-09-11T03:50:05.645 INFO:tasks.rados.rados.2.smithi175.stdout:3210: finishing write tid 3 to smithi17557870-27 2024-09-11T03:50:05.645 INFO:tasks.rados.rados.2.smithi175.stdout:3210: finishing write tid 4 to smithi17557870-27 2024-09-11T03:50:05.645 INFO:tasks.rados.rados.2.smithi175.stdout:3210: oid 27 updating version 0 to 1102 2024-09-11T03:50:05.645 INFO:tasks.rados.rados.2.smithi175.stdout:3210: oid 27 updating version 1102 to 1103 2024-09-11T03:50:05.646 INFO:tasks.rados.rados.2.smithi175.stdout:3210: oid 27 updating version 1103 to 1104 2024-09-11T03:50:05.646 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 1104 (ObjNum 1090 snap 353 seq_num 1090) dirty exists 2024-09-11T03:50:05.646 INFO:tasks.rados.rados.2.smithi175.stdout:3210: left oid 27 (ObjNum 1090 snap 353 seq_num 1090) 2024-09-11T03:50:05.646 INFO:tasks.rados.rados.2.smithi175.stdout:3211: finishing copy_from racing read to smithi17557870-45 2024-09-11T03:50:05.646 INFO:tasks.rados.rados.2.smithi175.stdout:3209: done (2 left) 2024-09-11T03:50:05.646 INFO:tasks.rados.rados.2.smithi175.stdout:3210: done (1 left) 2024-09-11T03:50:05.646 INFO:tasks.rados.rados.2.smithi175.stdout:3212: snap_create 2024-09-11T03:50:05.646 INFO:tasks.rados.rados.2.smithi175.stdout:3211: finishing copy_from to smithi17557870-45 2024-09-11T03:50:05.646 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 989 (ObjNum 1059 snap 345 seq_num 1059) dirty exists 2024-09-11T03:50:05.662 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:05.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:05.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:05.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:05.785 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:05.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:05.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:05.807 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:05.886 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:05.907 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:05.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:06.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:06.032 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:06.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:06.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:06.069 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:50:06.075 INFO:tasks.rados.rados.2.smithi175.stdout:3211: done (1 left) 2024-09-11T03:50:06.075 INFO:tasks.rados.rados.2.smithi175.stdout:3212: done (0 left) 2024-09-11T03:50:06.075 INFO:tasks.rados.rados.2.smithi175.stdout:3213: delete oid 30 current snap is 354 2024-09-11T03:50:06.134 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:06.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:06.156 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:06.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:06.223 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:50:06.223 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:50:06.224 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-11T03:50:06.224 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T03:50:06.224 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T03:50:06.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-11T03:50:06.257 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:06.271 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:50:06.289 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:06.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:06.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:06.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:06 smithi084 ceph-mon[27254]: 2.30 scrub starts 2024-09-11T03:50:06.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:06 smithi084 ceph-mon[27254]: 2.30 scrub ok 2024-09-11T03:50:06.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:06 smithi084 ceph-mon[27254]: pgmap v1335: 87 pgs: 1 active+clean+snaptrim, 3 peering, 6 unknown, 77 active+clean; 481 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 5.2 MiB/s wr, 28 op/s 2024-09-11T03:50:06.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:06 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:50:06.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:06 smithi084 ceph-mon[27254]: 2.e scrub ok 2024-09-11T03:50:06.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:06 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/781436240' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:50:06.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:06 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:06.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:06 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/781436240' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:50:06.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:06 smithi084 ceph-mon[27254]: osdmap e788: 12 total, 12 up, 12 in 2024-09-11T03:50:06.390 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:06.412 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:06.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:06.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:06 smithi121 ceph-mon[30729]: 2.30 scrub starts 2024-09-11T03:50:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:06 smithi121 ceph-mon[30729]: 2.30 scrub ok 2024-09-11T03:50:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:06 smithi121 ceph-mon[30729]: pgmap v1335: 87 pgs: 1 active+clean+snaptrim, 3 peering, 6 unknown, 77 active+clean; 481 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 5.2 MiB/s wr, 28 op/s 2024-09-11T03:50:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:06 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:50:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:06 smithi121 ceph-mon[30729]: 2.e scrub ok 2024-09-11T03:50:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:06 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/781436240' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:50:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:06 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:06 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/781436240' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:50:06.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:06 smithi121 ceph-mon[30729]: osdmap e788: 12 total, 12 up, 12 in 2024-09-11T03:50:06.469 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:50:06.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:06 smithi175 ceph-mon[30894]: 2.30 scrub starts 2024-09-11T03:50:06.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:06 smithi175 ceph-mon[30894]: 2.30 scrub ok 2024-09-11T03:50:06.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:06 smithi175 ceph-mon[30894]: pgmap v1335: 87 pgs: 1 active+clean+snaptrim, 3 peering, 6 unknown, 77 active+clean; 481 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 5.2 MiB/s wr, 28 op/s 2024-09-11T03:50:06.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:06 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:50:06.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:06 smithi175 ceph-mon[30894]: 2.e scrub ok 2024-09-11T03:50:06.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:06 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/781436240' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:50:06.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:06 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:06.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:06 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/781436240' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:50:06.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:06 smithi175 ceph-mon[30894]: osdmap e788: 12 total, 12 up, 12 in 2024-09-11T03:50:06.490 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:50:06.490 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:50:06.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:06.536 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:06.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:06.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:06.659 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:06.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:06.760 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:06.783 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:06.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:06.784 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:06.885 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:06.907 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:06.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:06.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:06.973 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:50:06.974 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "" 2024-09-11T03:50:06.974 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:50:07.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:07.030 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:07.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:07.082 INFO:tasks.rados.rados.2.smithi175.stdout:3213: done (0 left) 2024-09-11T03:50:07.083 INFO:tasks.rados.rados.2.smithi175.stdout:3214: rollback oid 33 current snap is 354 2024-09-11T03:50:07.083 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 33 to 349 2024-09-11T03:50:07.083 INFO:tasks.rados.rados.2.smithi175.stdout:3215: delete oid 25 current snap is 354 2024-09-11T03:50:07.085 INFO:tasks.rados.rados.2.smithi175.stdout:3215: done (1 left) 2024-09-11T03:50:07.085 INFO:tasks.rados.rados.2.smithi175.stdout:3216: read oid 13 snap -1 2024-09-11T03:50:07.085 INFO:tasks.rados.rados.2.smithi175.stdout:3216: expect (ObjNum 907 snap 287 seq_num 907) 2024-09-11T03:50:07.085 INFO:tasks.rados.rados.2.smithi175.stdout:3217: snap_create 2024-09-11T03:50:07.086 INFO:tasks.rados.rados.2.smithi175.stdout:3214: finishing rollback tid 0 to smithi17557870-33 2024-09-11T03:50:07.086 INFO:tasks.rados.rados.2.smithi175.stdout:3214: finishing rollback tid 1 to smithi17557870-33 2024-09-11T03:50:07.086 INFO:tasks.rados.rados.2.smithi175.stdout:3214: finishing rollback tid 2 to smithi17557870-33 2024-09-11T03:50:07.086 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 1254 (ObjNum 1037 snap 340 seq_num 1037) dirty exists 2024-09-11T03:50:07.092 INFO:tasks.rados.rados.2.smithi175.stdout:3216: expect (ObjNum 907 snap 287 seq_num 907) 2024-09-11T03:50:07.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:07 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:50:07.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:07 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:50:07.128 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:07 smithi121 ceph-mon[30729]: osdmap e789: 12 total, 12 up, 12 in 2024-09-11T03:50:07.132 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:07.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:07 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:50:07.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:07 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:50:07.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:07 smithi084 ceph-mon[27254]: osdmap e789: 12 total, 12 up, 12 in 2024-09-11T03:50:07.155 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:07.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:07.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:07.257 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:07.279 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:07.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:07.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:07.296 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T03:50:07.371 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:07.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:07.403 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:07.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:07.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:07 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:50:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:07 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:50:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:07 smithi175 ceph-mon[30894]: osdmap e789: 12 total, 12 up, 12 in 2024-09-11T03:50:07.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:07.527 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:07.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:07.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:07.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:07.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:07.653 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:07.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:07.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:07.775 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:07.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:07.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:07.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:07.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:07.899 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:07.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:07.915 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:07.999 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:08.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:08.022 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:08.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:08.075 INFO:tasks.rados.rados.2.smithi175.stdout:3214: done (2 left) 2024-09-11T03:50:08.076 INFO:tasks.rados.rados.2.smithi175.stdout:3216: done (1 left) 2024-09-11T03:50:08.076 INFO:tasks.rados.rados.2.smithi175.stdout:3217: done (0 left) 2024-09-11T03:50:08.076 INFO:tasks.rados.rados.2.smithi175.stdout:3218: rollback oid 39 current snap is 355 2024-09-11T03:50:08.076 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 39 to 344 2024-09-11T03:50:08.076 INFO:tasks.rados.rados.2.smithi175.stdout:3219: write oid 50 current snap is 355 2024-09-11T03:50:08.076 INFO:tasks.rados.rados.2.smithi175.stdout:3219: seq_num 1093 ranges {770051=485636,1776701=511047} 2024-09-11T03:50:08.082 INFO:tasks.rados.rados.2.smithi175.stdout:3219: writing smithi17557870-50 from 770051 to 1255687 tid 1 2024-09-11T03:50:08.089 INFO:tasks.rados.rados.2.smithi175.stdout:3219: writing smithi17557870-50 from 1776701 to 2287748 tid 2 2024-09-11T03:50:08.089 INFO:tasks.rados.rados.2.smithi175.stdout:3220: copy_from oid 34 from oid 14 current snap is 355 2024-09-11T03:50:08.089 INFO:tasks.rados.rados.2.smithi175.stdout:3218: finishing rollback tid 0 to smithi17557870-39 2024-09-11T03:50:08.089 INFO:tasks.rados.rados.2.smithi175.stdout:3218: finishing rollback tid 1 to smithi17557870-39 2024-09-11T03:50:08.089 INFO:tasks.rados.rados.2.smithi175.stdout:3218: finishing rollback tid 2 to smithi17557870-39 2024-09-11T03:50:08.089 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 998 (ObjNum 1030 snap 338 seq_num 1030) dirty exists 2024-09-11T03:50:08.089 INFO:tasks.rados.rados.2.smithi175.stdout:3219: finishing write tid 1 to smithi17557870-50 2024-09-11T03:50:08.089 INFO:tasks.rados.rados.2.smithi175.stdout:3218: done (2 left) 2024-09-11T03:50:08.089 INFO:tasks.rados.rados.2.smithi175.stdout:3221: write (excl) oid 45 current snap is 355 2024-09-11T03:50:08.089 INFO:tasks.rados.rados.2.smithi175.stdout:3221: seq_num 1094 ranges {786858=560885,1835183=551675} 2024-09-11T03:50:08.095 INFO:tasks.rados.rados.2.smithi175.stdout:3221: writing smithi17557870-45 from 786858 to 1347743 tid 1 2024-09-11T03:50:08.103 INFO:tasks.rados.rados.2.smithi175.stdout:3221: writing smithi17557870-45 from 1835183 to 2386858 tid 2 2024-09-11T03:50:08.103 INFO:tasks.rados.rados.2.smithi175.stdout:3222: write (excl) oid 22 current snap is 355 2024-09-11T03:50:08.103 INFO:tasks.rados.rados.2.smithi175.stdout:3219: finishing write tid 2 to smithi17557870-50 2024-09-11T03:50:08.103 INFO:tasks.rados.rados.2.smithi175.stdout:3219: finishing write tid 3 to smithi17557870-50 2024-09-11T03:50:08.103 INFO:tasks.rados.rados.2.smithi175.stdout:3219: finishing write tid 4 to smithi17557870-50 2024-09-11T03:50:08.103 INFO:tasks.rados.rados.2.smithi175.stdout:3219: oid 50 updating version 0 to 1256 2024-09-11T03:50:08.103 INFO:tasks.rados.rados.2.smithi175.stdout:3219: oid 50 updating version 1256 to 1258 2024-09-11T03:50:08.103 INFO:tasks.rados.rados.2.smithi175.stdout:3219: oid 50 version 1258 is already newer than 1257 2024-09-11T03:50:08.103 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 1258 (ObjNum 1093 snap 355 seq_num 1093) dirty exists 2024-09-11T03:50:08.103 INFO:tasks.rados.rados.2.smithi175.stdout:3219: left oid 50 (ObjNum 1093 snap 355 seq_num 1093) 2024-09-11T03:50:08.103 INFO:tasks.rados.rados.2.smithi175.stdout:3220: finishing copy_from racing read to smithi17557870-34 2024-09-11T03:50:08.104 INFO:tasks.rados.rados.2.smithi175.stdout:3221: finishing write tid 1 to smithi17557870-45 2024-09-11T03:50:08.104 INFO:tasks.rados.rados.2.smithi175.stdout:3220: finishing copy_from to smithi17557870-34 2024-09-11T03:50:08.104 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 1044 (ObjNum 1087 snap 353 seq_num 1087) dirty exists 2024-09-11T03:50:08.104 INFO:tasks.rados.rados.2.smithi175.stdout:3222: seq_num 1095 ranges {403665=752487,1610018=514716} 2024-09-11T03:50:08.108 INFO:tasks.rados.rados.2.smithi175.stdout:3222: writing smithi17557870-22 from 403665 to 1156152 tid 1 2024-09-11T03:50:08.114 INFO:tasks.rados.rados.2.smithi175.stdout:3222: writing smithi17557870-22 from 1610018 to 2124734 tid 2 2024-09-11T03:50:08.114 INFO:tasks.rados.rados.2.smithi175.stdout:3219: done (3 left) 2024-09-11T03:50:08.114 INFO:tasks.rados.rados.2.smithi175.stdout:3220: done (2 left) 2024-09-11T03:50:08.114 INFO:tasks.rados.rados.2.smithi175.stdout:3223: rollback oid 30 current snap is 355 2024-09-11T03:50:08.115 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 30 to 307 2024-09-11T03:50:08.115 INFO:tasks.rados.rados.2.smithi175.stdout:3221: finishing write tid 2 to smithi17557870-45 2024-09-11T03:50:08.115 INFO:tasks.rados.rados.2.smithi175.stdout:3221: finishing write tid 3 to smithi17557870-45 2024-09-11T03:50:08.115 INFO:tasks.rados.rados.2.smithi175.stdout:3221: finishing write tid 4 to smithi17557870-45 2024-09-11T03:50:08.115 INFO:tasks.rados.rados.2.smithi175.stdout:3221: oid 45 updating version 0 to 992 2024-09-11T03:50:08.115 INFO:tasks.rados.rados.2.smithi175.stdout:3221: oid 45 version 992 is already newer than 990 2024-09-11T03:50:08.115 INFO:tasks.rados.rados.2.smithi175.stdout:3221: oid 45 version 992 is already newer than 991 2024-09-11T03:50:08.115 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 992 (ObjNum 1094 snap 355 seq_num 1094) dirty exists 2024-09-11T03:50:08.115 INFO:tasks.rados.rados.2.smithi175.stdout:3221: left oid 45 (ObjNum 1094 snap 355 seq_num 1094) 2024-09-11T03:50:08.116 INFO:tasks.rados.rados.2.smithi175.stdout:3221: done (2 left) 2024-09-11T03:50:08.116 INFO:tasks.rados.rados.2.smithi175.stdout:3224: rollback oid 37 current snap is 355 2024-09-11T03:50:08.116 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 37 to 296 2024-09-11T03:50:08.116 INFO:tasks.rados.rados.2.smithi175.stdout:3225: copy_from oid 10 from oid 46 current snap is 355 2024-09-11T03:50:08.116 INFO:tasks.rados.rados.2.smithi175.stdout:3226: snap_create 2024-09-11T03:50:08.118 INFO:tasks.rados.rados.2.smithi175.stdout:3225: finishing copy_from to smithi17557870-10 2024-09-11T03:50:08.118 INFO:tasks.rados.rados.2.smithi175.stdout:3225: got expected ENOENT (src dne) 2024-09-11T03:50:08.118 INFO:tasks.rados.rados.2.smithi175.stdout:3225: finishing copy_from racing read to smithi17557870-10 2024-09-11T03:50:08.118 INFO:tasks.rados.rados.2.smithi175.stdout:3222: finishing write tid 1 to smithi17557870-22 2024-09-11T03:50:08.118 INFO:tasks.rados.rados.2.smithi175.stdout:3222: finishing write tid 2 to smithi17557870-22 2024-09-11T03:50:08.118 INFO:tasks.rados.rados.2.smithi175.stdout:3223: finishing rollback tid 1 to smithi17557870-30 2024-09-11T03:50:08.118 INFO:tasks.rados.rados.2.smithi175.stdout:3223: finishing rollback tid 2 to smithi17557870-30 2024-09-11T03:50:08.118 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 1041 (ObjNum 895 snap 284 seq_num 895) dirty exists 2024-09-11T03:50:08.119 INFO:tasks.rados.rados.2.smithi175.stdout:3224: finishing rollback tid 0 to smithi17557870-37 2024-09-11T03:50:08.119 INFO:tasks.rados.rados.2.smithi175.stdout:3222: finishing write tid 3 to smithi17557870-22 2024-09-11T03:50:08.119 INFO:tasks.rados.rados.2.smithi175.stdout:3224: finishing rollback tid 1 to smithi17557870-37 2024-09-11T03:50:08.119 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 909 (ObjNum 129 snap 0 seq_num 1745219792) dirty dne 2024-09-11T03:50:08.119 INFO:tasks.rados.rados.2.smithi175.stdout:3222: finishing write tid 4 to smithi17557870-22 2024-09-11T03:50:08.119 INFO:tasks.rados.rados.2.smithi175.stdout:3222: oid 22 updating version 0 to 1225 2024-09-11T03:50:08.119 INFO:tasks.rados.rados.2.smithi175.stdout:3222: oid 22 version 1225 is already newer than 1224 2024-09-11T03:50:08.119 INFO:tasks.rados.rados.2.smithi175.stdout:3222: oid 22 version 1225 is already newer than 1223 2024-09-11T03:50:08.119 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 1225 (ObjNum 1095 snap 355 seq_num 1095) dirty exists 2024-09-11T03:50:08.119 INFO:tasks.rados.rados.2.smithi175.stdout:3222: left oid 22 (ObjNum 1095 snap 355 seq_num 1095) 2024-09-11T03:50:08.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:08.146 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:08.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:08.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:08.248 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:08.270 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:08.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:08.271 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:08.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:08 smithi084 ceph-mon[27254]: pgmap v1338: 87 pgs: 5 peering, 2 unknown, 80 active+clean; 486 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 2.6 MiB/s wr, 28 op/s; 7.3 MiB/s, 23 objects/s recovering 2024-09-11T03:50:08.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:08 smithi084 ceph-mon[27254]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-11T03:50:08.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:08 smithi084 ceph-mon[27254]: 2.32 scrub starts 2024-09-11T03:50:08.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:08 smithi084 ceph-mon[27254]: 2.36 scrub starts 2024-09-11T03:50:08.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:08 smithi084 ceph-mon[27254]: 2.34 scrub starts 2024-09-11T03:50:08.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:08 smithi084 ceph-mon[27254]: 2.2f scrub starts 2024-09-11T03:50:08.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:08 smithi084 ceph-mon[27254]: osdmap e790: 12 total, 12 up, 12 in 2024-09-11T03:50:08.372 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:08.394 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:08.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:08.395 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:08.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:08 smithi121 ceph-mon[30729]: pgmap v1338: 87 pgs: 5 peering, 2 unknown, 80 active+clean; 486 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 2.6 MiB/s wr, 28 op/s; 7.3 MiB/s, 23 objects/s recovering 2024-09-11T03:50:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:08 smithi121 ceph-mon[30729]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-11T03:50:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:08 smithi121 ceph-mon[30729]: 2.32 scrub starts 2024-09-11T03:50:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:08 smithi121 ceph-mon[30729]: 2.36 scrub starts 2024-09-11T03:50:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:08 smithi121 ceph-mon[30729]: 2.34 scrub starts 2024-09-11T03:50:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:08 smithi121 ceph-mon[30729]: 2.2f scrub starts 2024-09-11T03:50:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:08 smithi121 ceph-mon[30729]: osdmap e790: 12 total, 12 up, 12 in 2024-09-11T03:50:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:08 smithi175 ceph-mon[30894]: pgmap v1338: 87 pgs: 5 peering, 2 unknown, 80 active+clean; 486 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 2.6 MiB/s wr, 28 op/s; 7.3 MiB/s, 23 objects/s recovering 2024-09-11T03:50:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:08 smithi175 ceph-mon[30894]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-11T03:50:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:08 smithi175 ceph-mon[30894]: 2.32 scrub starts 2024-09-11T03:50:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:08 smithi175 ceph-mon[30894]: 2.36 scrub starts 2024-09-11T03:50:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:08 smithi175 ceph-mon[30894]: 2.34 scrub starts 2024-09-11T03:50:08.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:08 smithi175 ceph-mon[30894]: 2.2f scrub starts 2024-09-11T03:50:08.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:08 smithi175 ceph-mon[30894]: osdmap e790: 12 total, 12 up, 12 in 2024-09-11T03:50:08.496 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:08.518 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:08.519 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:08.620 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:08.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:08.641 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:08.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:08.718 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:50:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:50:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:50:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:50:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1589 2.7 e218/191 2:e77ce595:::smithi17557870-31:head v 218'174, mlcod=218'168)", 2024-09-11T03:50:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:35.116398+0000", 2024-09-11T03:50:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 573.58534435299998, 2024-09-11T03:50:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0089646050000000005, 2024-09-11T03:50:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116398+0000", 2024-09-11T03:50:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116398+0000", 2024-09-11T03:50:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.116400+0000", 2024-09-11T03:50:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6109999999999999e-06 2024-09-11T03:50:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119261+0000", 2024-09-11T03:50:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028605309999999999 2024-09-11T03:50:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119281+0000", 2024-09-11T03:50:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0491000000000001e-05 2024-09-11T03:50:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119292+0000", 2024-09-11T03:50:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0759e-05 2024-09-11T03:50:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119350+0000", 2024-09-11T03:50:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8588999999999998e-05 2024-09-11T03:50:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.119369+0000", 2024-09-11T03:50:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8227e-05 2024-09-11T03:50:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125320+0000", 2024-09-11T03:50:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059510409999999998 2024-09-11T03:50:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:35.125363+0000", 2024-09-11T03:50:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3356000000000003e-05 2024-09-11T03:50:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:1816 2.d e243/95 2:b7bdfe52:::smithi17557870-10:head v 243'237, mlcod=243'233)", 2024-09-11T03:50:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:40:59.922567+0000", 2024-09-11T03:50:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 548.77917600199999, 2024-09-11T03:50:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094601100000000007, 2024-09-11T03:50:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:08.724 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.922567+0000", 2024-09-11T03:50:08.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.922567+0000", 2024-09-11T03:50:08.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.922568+0000", 2024-09-11T03:50:08.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.345e-06 2024-09-11T03:50:08.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929634+0000", 2024-09-11T03:50:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0070657130000000004 2024-09-11T03:50:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929654+0000", 2024-09-11T03:50:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9972000000000001e-05 2024-09-11T03:50:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929663+0000", 2024-09-11T03:50:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3770000000000002e-06 2024-09-11T03:50:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929731+0000", 2024-09-11T03:50:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8095000000000004e-05 2024-09-11T03:50:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.929759+0000", 2024-09-11T03:50:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7248999999999999e-05 2024-09-11T03:50:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.931991+0000", 2024-09-11T03:50:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022320679999999998 2024-09-11T03:50:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:40:59.932027+0000", 2024-09-11T03:50:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6291000000000002e-05 2024-09-11T03:50:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2431 2.12 2:4e5019e5:::smithi17557870-40:head [copy-from ver 279] snapc 66=[66,65,64,63,61,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-09-11T03:50:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:51.877192+0000", 2024-09-11T03:50:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 496.82455070999998, 2024-09-11T03:50:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013330089, 2024-09-11T03:50:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2431 2024-09-11T03:50:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877192+0000", 2024-09-11T03:50:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877192+0000", 2024-09-11T03:50:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877194+0000", 2024-09-11T03:50:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.767e-06 2024-09-11T03:50:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877202+0000", 2024-09-11T03:50:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0770000000000008e-06 2024-09-11T03:50:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877220+0000", 2024-09-11T03:50:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8561000000000001e-05 2024-09-11T03:50:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877230+0000", 2024-09-11T03:50:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2820000000000002e-06 2024-09-11T03:50:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877290+0000", 2024-09-11T03:50:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0025999999999998e-05 2024-09-11T03:50:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877404+0000", 2024-09-11T03:50:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011425999999999999 2024-09-11T03:50:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,11", 2024-09-11T03:50:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.884630+0000", 2024-09-11T03:50:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072256769999999998 2024-09-11T03:50:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.888559+0000", 2024-09-11T03:50:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039288750000000001 2024-09-11T03:50:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.888615+0000", 2024-09-11T03:50:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6609e-05 2024-09-11T03:50:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.890459+0000", 2024-09-11T03:50:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001844116 2024-09-11T03:50:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.890494+0000", 2024-09-11T03:50:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4921e-05 2024-09-11T03:50:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.890522+0000", 2024-09-11T03:50:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7917999999999999e-05 2024-09-11T03:50:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2432 2.12 2:4e5019e5:::smithi17557870-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e295)", 2024-09-11T03:50:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:51.877251+0000", 2024-09-11T03:50:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 496.82449205799998, 2024-09-11T03:50:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010600994000000001, 2024-09-11T03:50:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2432 2024-09-11T03:50:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877251+0000", 2024-09-11T03:50:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877251+0000", 2024-09-11T03:50:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877251+0000", 2024-09-11T03:50:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3399999999999999e-07 2024-09-11T03:50:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877254+0000", 2024-09-11T03:50:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.379e-06 2024-09-11T03:50:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877262+0000", 2024-09-11T03:50:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0779999999999996e-06 2024-09-11T03:50:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877265+0000", 2024-09-11T03:50:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.641e-06 2024-09-11T03:50:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877539+0000", 2024-09-11T03:50:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027397000000000001 2024-09-11T03:50:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877571+0000", 2024-09-11T03:50:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2122999999999997e-05 2024-09-11T03:50:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.887709+0000", 2024-09-11T03:50:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010137922000000001 2024-09-11T03:50:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.887781+0000", 2024-09-11T03:50:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1384999999999995e-05 2024-09-11T03:50:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.887852+0000", 2024-09-11T03:50:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0962000000000003e-05 2024-09-11T03:50:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2443 2.12 2:494ba932:::smithi17557870-43:head [copy-from ver 342] snapc 66=[65,64,63,61,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-11T03:50:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:52.884558+0000", 2024-09-11T03:50:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 495.81718434099997, 2024-09-11T03:50:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019937844999999999, 2024-09-11T03:50:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2443 2024-09-11T03:50:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884558+0000", 2024-09-11T03:50:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884558+0000", 2024-09-11T03:50:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884560+0000", 2024-09-11T03:50:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7e-06 2024-09-11T03:50:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884568+0000", 2024-09-11T03:50:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0940000000000008e-06 2024-09-11T03:50:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884584+0000", 2024-09-11T03:50:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6277e-05 2024-09-11T03:50:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884593+0000", 2024-09-11T03:50:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1810000000000001e-06 2024-09-11T03:50:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884660+0000", 2024-09-11T03:50:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7362999999999999e-05 2024-09-11T03:50:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884731+0000", 2024-09-11T03:50:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1316999999999995e-05 2024-09-11T03:50:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,11", 2024-09-11T03:50:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.895614+0000", 2024-09-11T03:50:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010882911 2024-09-11T03:50:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.900723+0000", 2024-09-11T03:50:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005109143 2024-09-11T03:50:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.902758+0000", 2024-09-11T03:50:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020342870000000001 2024-09-11T03:50:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:08.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.904417+0000", 2024-09-11T03:50:08.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001658875 2024-09-11T03:50:08.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:08.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.904454+0000", 2024-09-11T03:50:08.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7883999999999998e-05 2024-09-11T03:50:08.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.904496+0000", 2024-09-11T03:50:08.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1813000000000001e-05 2024-09-11T03:50:08.747 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.747 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.747 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.747 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2444 2.12 2:494ba932:::smithi17557870-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e296)", 2024-09-11T03:50:08.747 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:52.884615+0000", 2024-09-11T03:50:08.747 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 495.81712778000002, 2024-09-11T03:50:08.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015042823, 2024-09-11T03:50:08.747 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.747 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.747 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:08.747 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:08.747 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:08.748 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:08.748 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2444 2024-09-11T03:50:08.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.748 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884615+0000", 2024-09-11T03:50:08.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884615+0000", 2024-09-11T03:50:08.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884616+0000", 2024-09-11T03:50:08.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5499999999999998e-07 2024-09-11T03:50:08.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884618+0000", 2024-09-11T03:50:08.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.818e-06 2024-09-11T03:50:08.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884632+0000", 2024-09-11T03:50:08.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3219e-05 2024-09-11T03:50:08.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884639+0000", 2024-09-11T03:50:08.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9990000000000002e-06 2024-09-11T03:50:08.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884918+0000", 2024-09-11T03:50:08.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027906699999999998 2024-09-11T03:50:08.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:08.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884959+0000", 2024-09-11T03:50:08.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1844999999999997e-05 2024-09-11T03:50:08.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.899489+0000", 2024-09-11T03:50:08.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014529034999999999 2024-09-11T03:50:08.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.899577+0000", 2024-09-11T03:50:08.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8512999999999998e-05 2024-09-11T03:50:08.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.899658+0000", 2024-09-11T03:50:08.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0772000000000005e-05 2024-09-11T03:50:08.752 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.752 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.752 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.752 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2641 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 350] snapc 75=[75,74,73,71,6f,6c,68,65,5d] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-11T03:50:08.752 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:19.887918+0000", 2024-09-11T03:50:08.752 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 468.81382448599999, 2024-09-11T03:50:08.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011895905, 2024-09-11T03:50:08.753 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.753 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.753 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:08.753 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:08.753 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:08.753 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:08.753 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2641 2024-09-11T03:50:08.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.753 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887918+0000", 2024-09-11T03:50:08.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887918+0000", 2024-09-11T03:50:08.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887920+0000", 2024-09-11T03:50:08.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.762e-06 2024-09-11T03:50:08.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887928+0000", 2024-09-11T03:50:08.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8900000000000007e-06 2024-09-11T03:50:08.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887945+0000", 2024-09-11T03:50:08.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6707000000000001e-05 2024-09-11T03:50:08.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887953+0000", 2024-09-11T03:50:08.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5620000000000004e-06 2024-09-11T03:50:08.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.888014+0000", 2024-09-11T03:50:08.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0798000000000003e-05 2024-09-11T03:50:08.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:50:08.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.888019+0000", 2024-09-11T03:50:08.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5990000000000004e-06 2024-09-11T03:50:08.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.889290+0000", 2024-09-11T03:50:08.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012718009999999999 2024-09-11T03:50:08.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.889393+0000", 2024-09-11T03:50:08.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102298 2024-09-11T03:50:08.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,8", 2024-09-11T03:50:08.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.894120+0000", 2024-09-11T03:50:08.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047277259999999998 2024-09-11T03:50:08.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:08.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.897232+0000", 2024-09-11T03:50:08.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031110859999999999 2024-09-11T03:50:08.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:08.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.897530+0000", 2024-09-11T03:50:08.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000298342 2024-09-11T03:50:08.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:08.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.899728+0000", 2024-09-11T03:50:08.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021984819999999999 2024-09-11T03:50:08.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:08.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.899782+0000", 2024-09-11T03:50:08.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3922999999999999e-05 2024-09-11T03:50:08.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.899814+0000", 2024-09-11T03:50:08.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1928999999999999e-05 2024-09-11T03:50:08.759 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.759 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.759 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.759 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2944 2.12 2:494ba932:::smithi17557870-43:head [copy-from ver 311] snapc 80=[80,7b,7a,79,77,75,6c] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-11T03:50:08.760 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:43.962947+0000", 2024-09-11T03:50:08.760 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 444.73879528600003, 2024-09-11T03:50:08.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014367049, 2024-09-11T03:50:08.760 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.760 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.760 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:08.760 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:08.760 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:08.760 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:08.760 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2944 2024-09-11T03:50:08.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.760 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962947+0000", 2024-09-11T03:50:08.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962947+0000", 2024-09-11T03:50:08.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962949+0000", 2024-09-11T03:50:08.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.689e-06 2024-09-11T03:50:08.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962955+0000", 2024-09-11T03:50:08.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.297e-06 2024-09-11T03:50:08.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962967+0000", 2024-09-11T03:50:08.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1517e-05 2024-09-11T03:50:08.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962975+0000", 2024-09-11T03:50:08.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2889999999999998e-06 2024-09-11T03:50:08.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963875+0000", 2024-09-11T03:50:08.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00089981200000000005 2024-09-11T03:50:08.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.964025+0000", 2024-09-11T03:50:08.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000149624 2024-09-11T03:50:08.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,7", 2024-09-11T03:50:08.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.971400+0000", 2024-09-11T03:50:08.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073748349999999997 2024-09-11T03:50:08.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:08.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.975907+0000", 2024-09-11T03:50:08.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045076159999999999 2024-09-11T03:50:08.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:08.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.976808+0000", 2024-09-11T03:50:08.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00090078700000000001 2024-09-11T03:50:08.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:08.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.977244+0000", 2024-09-11T03:50:08.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00043565800000000001 2024-09-11T03:50:08.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:08.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.977281+0000", 2024-09-11T03:50:08.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7150999999999997e-05 2024-09-11T03:50:08.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.977315+0000", 2024-09-11T03:50:08.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3773999999999999e-05 2024-09-11T03:50:08.766 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.766 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.766 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.767 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2945 2.12 2:494ba932:::smithi17557870-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e347)", 2024-09-11T03:50:08.767 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:43.963009+0000", 2024-09-11T03:50:08.767 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 444.73873368599999, 2024-09-11T03:50:08.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012192595000000001, 2024-09-11T03:50:08.767 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.767 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.767 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:08.767 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:08.767 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:08.767 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:08.767 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2945 2024-09-11T03:50:08.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.768 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963009+0000", 2024-09-11T03:50:08.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963009+0000", 2024-09-11T03:50:08.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963010+0000", 2024-09-11T03:50:08.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0699999999999997e-07 2024-09-11T03:50:08.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963012+0000", 2024-09-11T03:50:08.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6599999999999999e-06 2024-09-11T03:50:08.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963020+0000", 2024-09-11T03:50:08.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2649999999999999e-06 2024-09-11T03:50:08.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963025+0000", 2024-09-11T03:50:08.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6860000000000003e-06 2024-09-11T03:50:08.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.964185+0000", 2024-09-11T03:50:08.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011599329999999999 2024-09-11T03:50:08.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:08.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.964230+0000", 2024-09-11T03:50:08.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4336999999999999e-05 2024-09-11T03:50:08.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.974986+0000", 2024-09-11T03:50:08.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010755928999999999 2024-09-11T03:50:08.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.975091+0000", 2024-09-11T03:50:08.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105701 2024-09-11T03:50:08.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.975202+0000", 2024-09-11T03:50:08.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011047700000000001 2024-09-11T03:50:08.772 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.772 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.772 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.772 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:3536 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 408] snapc 90=[90,8f,8d,8c,8b,8a] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-11T03:50:08.772 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:43:18.903548+0000", 2024-09-11T03:50:08.772 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 409.79819435399997, 2024-09-11T03:50:08.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014015196000000001, 2024-09-11T03:50:08.773 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.773 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.773 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:08.773 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:08.773 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:08.773 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:08.773 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3536 2024-09-11T03:50:08.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.773 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903548+0000", 2024-09-11T03:50:08.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903548+0000", 2024-09-11T03:50:08.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903550+0000", 2024-09-11T03:50:08.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.499e-06 2024-09-11T03:50:08.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903555+0000", 2024-09-11T03:50:08.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6890000000000003e-06 2024-09-11T03:50:08.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903567+0000", 2024-09-11T03:50:08.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1912999999999999e-05 2024-09-11T03:50:08.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903574+0000", 2024-09-11T03:50:08.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7260000000000002e-06 2024-09-11T03:50:08.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904043+0000", 2024-09-11T03:50:08.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00046877799999999999 2024-09-11T03:50:08.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904136+0000", 2024-09-11T03:50:08.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2745999999999996e-05 2024-09-11T03:50:08.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,8", 2024-09-11T03:50:08.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.912436+0000", 2024-09-11T03:50:08.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083000049999999992 2024-09-11T03:50:08.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:08.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.915813+0000", 2024-09-11T03:50:08.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033772839999999999 2024-09-11T03:50:08.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:08.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.916094+0000", 2024-09-11T03:50:08.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028135600000000002 2024-09-11T03:50:08.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:08.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.917498+0000", 2024-09-11T03:50:08.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014033680000000001 2024-09-11T03:50:08.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:08.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.917534+0000", 2024-09-11T03:50:08.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6180000000000003e-05 2024-09-11T03:50:08.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.917564+0000", 2024-09-11T03:50:08.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9652e-05 2024-09-11T03:50:08.778 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.778 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.779 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.779 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:3537 2.19 2:98bb65f5:::smithi17557870-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e381)", 2024-09-11T03:50:08.779 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:50:08.779 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 409.79814688300002, 2024-09-11T03:50:08.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011559349999999999, 2024-09-11T03:50:08.779 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.779 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.779 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:08.779 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:08.779 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:08.779 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:08.779 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3537 2024-09-11T03:50:08.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.780 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:50:08.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:50:08.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:50:08.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8899999999999999e-07 2024-09-11T03:50:08.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903599+0000", 2024-09-11T03:50:08.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3750000000000001e-06 2024-09-11T03:50:08.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903606+0000", 2024-09-11T03:50:08.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1969999999999999e-06 2024-09-11T03:50:08.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903611+0000", 2024-09-11T03:50:08.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3680000000000001e-06 2024-09-11T03:50:08.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904244+0000", 2024-09-11T03:50:08.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00063279999999999999 2024-09-11T03:50:08.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:08.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904291+0000", 2024-09-11T03:50:08.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6755999999999998e-05 2024-09-11T03:50:08.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.914973+0000", 2024-09-11T03:50:08.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010682199999999999 2024-09-11T03:50:08.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.915043+0000", 2024-09-11T03:50:08.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0297999999999997e-05 2024-09-11T03:50:08.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.915155+0000", 2024-09-11T03:50:08.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000111767 2024-09-11T03:50:08.784 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.784 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.784 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.785 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:4907 2.7 e522/417 2:e0ec5763:::smithi17557870-2:head v 522'753, mlcod=522'751)", 2024-09-11T03:50:08.785 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:45:40.117077+0000", 2024-09-11T03:50:08.785 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 268.58466537800001, 2024-09-11T03:50:08.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010151360999999999, 2024-09-11T03:50:08.785 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.785 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.785 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:08.785 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.117077+0000", 2024-09-11T03:50:08.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.117077+0000", 2024-09-11T03:50:08.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.117079+0000", 2024-09-11T03:50:08.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7799999999999999e-06 2024-09-11T03:50:08.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.122431+0000", 2024-09-11T03:50:08.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053519090000000002 2024-09-11T03:50:08.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.122448+0000", 2024-09-11T03:50:08.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6521e-05 2024-09-11T03:50:08.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.122455+0000", 2024-09-11T03:50:08.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3629999999999996e-06 2024-09-11T03:50:08.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.122511+0000", 2024-09-11T03:50:08.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5967000000000001e-05 2024-09-11T03:50:08.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.122528+0000", 2024-09-11T03:50:08.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6996000000000001e-05 2024-09-11T03:50:08.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:08.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.127188+0000", 2024-09-11T03:50:08.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046599509999999999 2024-09-11T03:50:08.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:40.127229+0000", 2024-09-11T03:50:08.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0874000000000002e-05 2024-09-11T03:50:08.789 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.789 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.789 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.789 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:5885 2.e e613/577 2:77dadb97:::smithi17557870-30:head v 613'880, mlcod=601'877)", 2024-09-11T03:50:08.789 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:47:11.245847+0000", 2024-09-11T03:50:08.790 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 177.45589603400001, 2024-09-11T03:50:08.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012027239, 2024-09-11T03:50:08.790 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.790 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.790 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:08.790 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.245847+0000", 2024-09-11T03:50:08.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.245847+0000", 2024-09-11T03:50:08.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.245847+0000", 2024-09-11T03:50:08.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7300000000000002e-07 2024-09-11T03:50:08.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.252621+0000", 2024-09-11T03:50:08.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067743339999999999 2024-09-11T03:50:08.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.252641+0000", 2024-09-11T03:50:08.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9734000000000001e-05 2024-09-11T03:50:08.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.252650+0000", 2024-09-11T03:50:08.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3400000000000004e-06 2024-09-11T03:50:08.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.792 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:08.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.252721+0000", 2024-09-11T03:50:08.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0767999999999997e-05 2024-09-11T03:50:08.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.252745+0000", 2024-09-11T03:50:08.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4278999999999999e-05 2024-09-11T03:50:08.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:08.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.257835+0000", 2024-09-11T03:50:08.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0050897060000000003 2024-09-11T03:50:08.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.257874+0000", 2024-09-11T03:50:08.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8804999999999999e-05 2024-09-11T03:50:08.795 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.795 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.795 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.795 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:6152 2.e e637/577 2:77dadb97:::smithi17557870-30:head v 637'891, mlcod=634'888)", 2024-09-11T03:50:08.795 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:47:35.714675+0000", 2024-09-11T03:50:08.795 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 152.98706735499999, 2024-09-11T03:50:08.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011964542, 2024-09-11T03:50:08.795 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.795 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.795 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:08.795 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714675+0000", 2024-09-11T03:50:08.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714675+0000", 2024-09-11T03:50:08.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714676+0000", 2024-09-11T03:50:08.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0100000000000001e-07 2024-09-11T03:50:08.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.720928+0000", 2024-09-11T03:50:08.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062523839999999997 2024-09-11T03:50:08.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.720951+0000", 2024-09-11T03:50:08.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2751e-05 2024-09-11T03:50:08.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.720964+0000", 2024-09-11T03:50:08.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2673999999999999e-05 2024-09-11T03:50:08.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721056+0000", 2024-09-11T03:50:08.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2620000000000004e-05 2024-09-11T03:50:08.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721084+0000", 2024-09-11T03:50:08.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7974999999999999e-05 2024-09-11T03:50:08.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:08.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.726587+0000", 2024-09-11T03:50:08.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055026420000000003 2024-09-11T03:50:08.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.726640+0000", 2024-09-11T03:50:08.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3195000000000001e-05 2024-09-11T03:50:08.799 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.799 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.799 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.799 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:6546 2.e e669/577 2:750de3b7:::smithi17557870-8:head v 669'909, mlcod=667'905)", 2024-09-11T03:50:08.799 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:07.227071+0000", 2024-09-11T03:50:08.799 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 121.474671329, 2024-09-11T03:50:08.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013198415999999999, 2024-09-11T03:50:08.799 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.799 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.800 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:08.800 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.227071+0000", 2024-09-11T03:50:08.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.227071+0000", 2024-09-11T03:50:08.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.227072+0000", 2024-09-11T03:50:08.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2e-07 2024-09-11T03:50:08.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.235454+0000", 2024-09-11T03:50:08.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083820730000000003 2024-09-11T03:50:08.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.235470+0000", 2024-09-11T03:50:08.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5951e-05 2024-09-11T03:50:08.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.235478+0000", 2024-09-11T03:50:08.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4249999999999998e-06 2024-09-11T03:50:08.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.235539+0000", 2024-09-11T03:50:08.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0473000000000002e-05 2024-09-11T03:50:08.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.235556+0000", 2024-09-11T03:50:08.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7110000000000001e-05 2024-09-11T03:50:08.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:08.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.240234+0000", 2024-09-11T03:50:08.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046781720000000004 2024-09-11T03:50:08.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.240270+0000", 2024-09-11T03:50:08.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5791999999999999e-05 2024-09-11T03:50:08.804 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.804 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.804 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.804 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:6548 2.9 e669/276 2:9718db5a:::smithi17557870-3:head v 669'978, mlcod=668'974)", 2024-09-11T03:50:08.804 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:07.229009+0000", 2024-09-11T03:50:08.804 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 121.472734253, 2024-09-11T03:50:08.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011091311, 2024-09-11T03:50:08.804 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.804 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.804 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:08.804 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.229009+0000", 2024-09-11T03:50:08.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.229009+0000", 2024-09-11T03:50:08.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.229010+0000", 2024-09-11T03:50:08.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9000000000000005e-07 2024-09-11T03:50:08.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.231542+0000", 2024-09-11T03:50:08.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025329570000000002 2024-09-11T03:50:08.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.231569+0000", 2024-09-11T03:50:08.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6387e-05 2024-09-11T03:50:08.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.231586+0000", 2024-09-11T03:50:08.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7320999999999999e-05 2024-09-11T03:50:08.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.231672+0000", 2024-09-11T03:50:08.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6117000000000002e-05 2024-09-11T03:50:08.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.231712+0000", 2024-09-11T03:50:08.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0114000000000003e-05 2024-09-11T03:50:08.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:08.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.240043+0000", 2024-09-11T03:50:08.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083307539999999992 2024-09-11T03:50:08.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.240100+0000", 2024-09-11T03:50:08.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6671e-05 2024-09-11T03:50:08.809 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.809 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.809 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.809 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:7117 2.e e713/577 2:750de3b7:::smithi17557870-8:head v 713'939, mlcod=713'935)", 2024-09-11T03:50:08.809 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:51.211659+0000", 2024-09-11T03:50:08.809 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 77.490084143000004, 2024-09-11T03:50:08.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011072423, 2024-09-11T03:50:08.809 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.809 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.810 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:08.810 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.211659+0000", 2024-09-11T03:50:08.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.211659+0000", 2024-09-11T03:50:08.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.211660+0000", 2024-09-11T03:50:08.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.22e-06 2024-09-11T03:50:08.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.217534+0000", 2024-09-11T03:50:08.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058737700000000004 2024-09-11T03:50:08.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.217555+0000", 2024-09-11T03:50:08.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0968e-05 2024-09-11T03:50:08.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.217568+0000", 2024-09-11T03:50:08.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3108999999999999e-05 2024-09-11T03:50:08.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.217675+0000", 2024-09-11T03:50:08.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000106969 2024-09-11T03:50:08.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.217694+0000", 2024-09-11T03:50:08.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9817999999999999e-05 2024-09-11T03:50:08.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:08.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.222679+0000", 2024-09-11T03:50:08.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049843140000000001 2024-09-11T03:50:08.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.222731+0000", 2024-09-11T03:50:08.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2255e-05 2024-09-11T03:50:08.814 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.814 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.814 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.814 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:7213 2.d e723/569 2:b6874adf:::smithi17557870-17:head v 723'925, mlcod=722'922)", 2024-09-11T03:50:08.814 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:49:01.039129+0000", 2024-09-11T03:50:08.815 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 67.662614251999997, 2024-09-11T03:50:08.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013254739999999999, 2024-09-11T03:50:08.815 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.815 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.815 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:08.815 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.039129+0000", 2024-09-11T03:50:08.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.039129+0000", 2024-09-11T03:50:08.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.039129+0000", 2024-09-11T03:50:08.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1900000000000002e-07 2024-09-11T03:50:08.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.051043+0000", 2024-09-11T03:50:08.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011913604 2024-09-11T03:50:08.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.051052+0000", 2024-09-11T03:50:08.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4930000000000004e-06 2024-09-11T03:50:08.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.051060+0000", 2024-09-11T03:50:08.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8199999999999997e-06 2024-09-11T03:50:08.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.051100+0000", 2024-09-11T03:50:08.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0394e-05 2024-09-11T03:50:08.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.051114+0000", 2024-09-11T03:50:08.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3083e-05 2024-09-11T03:50:08.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:08.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.052353+0000", 2024-09-11T03:50:08.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012397899999999999 2024-09-11T03:50:08.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.052383+0000", 2024-09-11T03:50:08.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9937000000000002e-05 2024-09-11T03:50:08.819 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.819 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.819 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.819 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7396 2.2d 2:b6874adf:::smithi17557870-17:head [copy-from ver 0] snapc 14f=[14f,14e,14d,14b,14a,149,148,146,145,144,143,142,13d,13c,139,138,136,135,133,12e,12a,127,120,11e,119] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-09-11T03:50:08.819 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:49:27.145662+0000", 2024-09-11T03:50:08.820 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 41.556081194000001, 2024-09-11T03:50:08.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022605074999999999, 2024-09-11T03:50:08.820 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.820 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.820 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:08.820 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:08.820 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:08.820 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:08.820 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7396 2024-09-11T03:50:08.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.820 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145662+0000", 2024-09-11T03:50:08.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145662+0000", 2024-09-11T03:50:08.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145663+0000", 2024-09-11T03:50:08.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.449e-06 2024-09-11T03:50:08.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145669+0000", 2024-09-11T03:50:08.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.558e-06 2024-09-11T03:50:08.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145683+0000", 2024-09-11T03:50:08.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4708999999999999e-05 2024-09-11T03:50:08.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145693+0000", 2024-09-11T03:50:08.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5140000000000007e-06 2024-09-11T03:50:08.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145751+0000", 2024-09-11T03:50:08.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8065000000000001e-05 2024-09-11T03:50:08.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:50:08.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145762+0000", 2024-09-11T03:50:08.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1625e-05 2024-09-11T03:50:08.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.166297+0000", 2024-09-11T03:50:08.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020534759999999999 2024-09-11T03:50:08.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.166499+0000", 2024-09-11T03:50:08.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000201454 2024-09-11T03:50:08.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.168267+0000", 2024-09-11T03:50:08.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017679410000000001 2024-09-11T03:50:08.825 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.825 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.825 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.825 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7397 2.2d 2:b6874adf:::smithi17557870-17:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e749)", 2024-09-11T03:50:08.825 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:49:27.145715+0000", 2024-09-11T03:50:08.825 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 41.556028112999996, 2024-09-11T03:50:08.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022944445000000001, 2024-09-11T03:50:08.825 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:08.825 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:08.826 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:08.826 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:08.826 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:08.826 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:08.826 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7397 2024-09-11T03:50:08.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.826 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:08.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:08.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145715+0000", 2024-09-11T03:50:08.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:08.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145715+0000", 2024-09-11T03:50:08.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:08.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:08.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145715+0000", 2024-09-11T03:50:08.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5899999999999996e-07 2024-09-11T03:50:08.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:08.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145718+0000", 2024-09-11T03:50:08.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5969999999999999e-06 2024-09-11T03:50:08.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:08.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145725+0000", 2024-09-11T03:50:08.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3180000000000001e-06 2024-09-11T03:50:08.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:08.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145731+0000", 2024-09-11T03:50:08.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6829999999999996e-06 2024-09-11T03:50:08.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145835+0000", 2024-09-11T03:50:08.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000103977 2024-09-11T03:50:08.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:50:08.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145845+0000", 2024-09-11T03:50:08.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0599e-05 2024-09-11T03:50:08.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.166949+0000", 2024-09-11T03:50:08.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021103706 2024-09-11T03:50:08.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:08.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.167035+0000", 2024-09-11T03:50:08.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6244999999999998e-05 2024-09-11T03:50:08.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:08.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.168341+0000", 2024-09-11T03:50:08.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001305949 2024-09-11T03:50:08.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:08.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.168440+0000", 2024-09-11T03:50:08.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8346999999999999e-05 2024-09-11T03:50:08.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:08.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:08.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:08.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.168659+0000", 2024-09-11T03:50:08.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021946499999999999 2024-09-11T03:50:08.831 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.831 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.831 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.831 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:08.831 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:08.831 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:50:08.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:08.852 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:08.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:08.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:08.974 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:08.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:08.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:09.076 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:09.096 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:50:09.096 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:50:09.096 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:50:09.096 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:50:09.096 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:50:09.096 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:50:09.096 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:50:09.096 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:50:09.097 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:50:09.097 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:50:09.097 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:50:09.097 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:50:09.097 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:50:09.097 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:50:09.097 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:50:09.097 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:50:09.097 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:50:09.098 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:50:09.098 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:50:09.098 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:50:09.098 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:50:09.098 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:50:09.098 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:50:09.098 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:50:09.102 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:09.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:09.103 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:09.110 INFO:tasks.rados.rados.2.smithi175.stdout:3222: done (4 left) 2024-09-11T03:50:09.110 INFO:tasks.rados.rados.2.smithi175.stdout:3223: done (3 left) 2024-09-11T03:50:09.110 INFO:tasks.rados.rados.2.smithi175.stdout:3224: done (2 left) 2024-09-11T03:50:09.110 INFO:tasks.rados.rados.2.smithi175.stdout:3225: done (1 left) 2024-09-11T03:50:09.110 INFO:tasks.rados.rados.2.smithi175.stdout:3226: done (0 left) 2024-09-11T03:50:09.110 INFO:tasks.rados.rados.2.smithi175.stdout:3227: snap_remove snap 355 2024-09-11T03:50:09.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:09.231 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:09.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:09.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:09.333 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:09.359 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:09.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:09.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:09.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:09 smithi084 ceph-mon[27254]: 2.37 scrub starts 2024-09-11T03:50:09.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:09 smithi084 ceph-mon[27254]: 2.33 scrub starts 2024-09-11T03:50:09.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:09 smithi084 ceph-mon[27254]: 2.31 scrub starts 2024-09-11T03:50:09.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:09 smithi084 ceph-mon[27254]: 2.32 scrub starts 2024-09-11T03:50:09.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:09 smithi084 ceph-mon[27254]: 2.35 scrub starts 2024-09-11T03:50:09.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:09 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:50:09.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:09 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:50:09.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:09 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1663376638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:09.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:09 smithi084 ceph-mon[27254]: 2.34 scrub starts 2024-09-11T03:50:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:09 smithi121 ceph-mon[30729]: 2.37 scrub starts 2024-09-11T03:50:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:09 smithi121 ceph-mon[30729]: 2.33 scrub starts 2024-09-11T03:50:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:09 smithi121 ceph-mon[30729]: 2.31 scrub starts 2024-09-11T03:50:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:09 smithi121 ceph-mon[30729]: 2.32 scrub starts 2024-09-11T03:50:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:09 smithi121 ceph-mon[30729]: 2.35 scrub starts 2024-09-11T03:50:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:09 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:50:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:09 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:50:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:09 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1663376638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:09 smithi121 ceph-mon[30729]: 2.34 scrub starts 2024-09-11T03:50:09.462 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:09 smithi175 ceph-mon[30894]: 2.37 scrub starts 2024-09-11T03:50:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:09 smithi175 ceph-mon[30894]: 2.33 scrub starts 2024-09-11T03:50:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:09 smithi175 ceph-mon[30894]: 2.31 scrub starts 2024-09-11T03:50:09.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:09 smithi175 ceph-mon[30894]: 2.32 scrub starts 2024-09-11T03:50:09.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:09 smithi175 ceph-mon[30894]: 2.35 scrub starts 2024-09-11T03:50:09.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:09 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:50:09.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:09 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:50:09.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:09 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1663376638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:09.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:09 smithi175 ceph-mon[30894]: 2.34 scrub starts 2024-09-11T03:50:09.483 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:09.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:09.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:09.585 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:09.611 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:09.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:09.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:09.713 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:09.738 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:09.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:09.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:09.840 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:09.841 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:50:09.862 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:09.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:09.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:09.990 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:09.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:09.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:10.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:10.112 INFO:tasks.rados.rados.2.smithi175.stdout:3227: done (0 left) 2024-09-11T03:50:10.112 INFO:tasks.rados.rados.2.smithi175.stdout:3228: write oid 11 current snap is 356 2024-09-11T03:50:10.112 INFO:tasks.rados.rados.2.smithi175.stdout:3228: seq_num 1096 ranges {420472=427736,1268500=658374,2347953=72519} 2024-09-11T03:50:10.117 INFO:tasks.rados.rados.2.smithi175.stdout:3228: writing smithi17557870-11 from 420472 to 848208 tid 1 2024-09-11T03:50:10.120 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:10.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:10.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:10.123 INFO:tasks.rados.rados.2.smithi175.stdout:3228: writing smithi17557870-11 from 1268500 to 1926874 tid 2 2024-09-11T03:50:10.126 INFO:tasks.rados.rados.2.smithi175.stdout:3228: writing smithi17557870-11 from 2347953 to 2420472 tid 3 2024-09-11T03:50:10.126 INFO:tasks.rados.rados.2.smithi175.stdout:3229: read oid 46 snap -1 2024-09-11T03:50:10.126 INFO:tasks.rados.rados.2.smithi175.stdout:3229: expect deleted 2024-09-11T03:50:10.126 INFO:tasks.rados.rados.2.smithi175.stdout:3230: snap_create 2024-09-11T03:50:10.126 INFO:tasks.rados.rados.2.smithi175.stdout:3228: finishing write tid 1 to smithi17557870-11 2024-09-11T03:50:10.126 INFO:tasks.rados.rados.2.smithi175.stdout:3228: finishing write tid 2 to smithi17557870-11 2024-09-11T03:50:10.128 INFO:tasks.rados.rados.2.smithi175.stdout:3228: finishing write tid 3 to smithi17557870-11 2024-09-11T03:50:10.128 INFO:tasks.rados.rados.2.smithi175.stdout:3228: finishing write tid 4 to smithi17557870-11 2024-09-11T03:50:10.128 INFO:tasks.rados.rados.2.smithi175.stdout:3228: finishing write tid 5 to smithi17557870-11 2024-09-11T03:50:10.128 INFO:tasks.rados.rados.2.smithi175.stdout:3228: oid 11 updating version 0 to 627 2024-09-11T03:50:10.128 INFO:tasks.rados.rados.2.smithi175.stdout:3228: oid 11 version 627 is already newer than 626 2024-09-11T03:50:10.128 INFO:tasks.rados.rados.2.smithi175.stdout:3228: oid 11 version 627 is already newer than 625 2024-09-11T03:50:10.128 INFO:tasks.rados.rados.2.smithi175.stdout:3228: oid 11 version 627 is already newer than 624 2024-09-11T03:50:10.129 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 627 (ObjNum 1096 snap 356 seq_num 1096) dirty exists 2024-09-11T03:50:10.129 INFO:tasks.rados.rados.2.smithi175.stdout:3228: left oid 11 (ObjNum 1096 snap 356 seq_num 1096) 2024-09-11T03:50:10.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:10 smithi084 ceph-mon[27254]: 2.37 scrub starts 2024-09-11T03:50:10.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:10 smithi084 ceph-mon[27254]: 2.33 scrub starts 2024-09-11T03:50:10.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:10 smithi084 ceph-mon[27254]: pgmap v1341: 87 pgs: 6 peering, 81 active+clean; 477 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s, 22 objects/s recovering 2024-09-11T03:50:10.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:10 smithi084 ceph-mon[27254]: 2.31 scrub starts 2024-09-11T03:50:10.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:10 smithi084 ceph-mon[27254]: 2.2f scrub starts 2024-09-11T03:50:10.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:10 smithi084 ceph-mon[27254]: osdmap e791: 12 total, 12 up, 12 in 2024-09-11T03:50:10.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:10 smithi084 ceph-mon[27254]: 2.34 scrub starts 2024-09-11T03:50:10.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:10.245 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:10.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:10.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:10.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:10.369 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:10.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:10.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:10 smithi121 ceph-mon[30729]: 2.37 scrub starts 2024-09-11T03:50:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:10 smithi121 ceph-mon[30729]: 2.33 scrub starts 2024-09-11T03:50:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:10 smithi121 ceph-mon[30729]: pgmap v1341: 87 pgs: 6 peering, 81 active+clean; 477 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s, 22 objects/s recovering 2024-09-11T03:50:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:10 smithi121 ceph-mon[30729]: 2.31 scrub starts 2024-09-11T03:50:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:10 smithi121 ceph-mon[30729]: 2.2f scrub starts 2024-09-11T03:50:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:10 smithi121 ceph-mon[30729]: osdmap e791: 12 total, 12 up, 12 in 2024-09-11T03:50:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:10 smithi121 ceph-mon[30729]: 2.34 scrub starts 2024-09-11T03:50:10.471 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:10.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:10 smithi175 ceph-mon[30894]: 2.37 scrub starts 2024-09-11T03:50:10.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:10 smithi175 ceph-mon[30894]: 2.33 scrub starts 2024-09-11T03:50:10.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:10 smithi175 ceph-mon[30894]: pgmap v1341: 87 pgs: 6 peering, 81 active+clean; 477 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s, 22 objects/s recovering 2024-09-11T03:50:10.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:10 smithi175 ceph-mon[30894]: 2.31 scrub starts 2024-09-11T03:50:10.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:10 smithi175 ceph-mon[30894]: 2.2f scrub starts 2024-09-11T03:50:10.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:10 smithi175 ceph-mon[30894]: osdmap e791: 12 total, 12 up, 12 in 2024-09-11T03:50:10.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:10 smithi175 ceph-mon[30894]: 2.34 scrub starts 2024-09-11T03:50:10.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:10.498 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:10.498 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:10.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T03:50:10.599 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:10.624 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:10.625 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:10.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:10.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:10.727 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:10.749 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:10.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:10.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:10.803 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:50:10.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:10.873 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:10.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:10.975 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:11.002 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:11.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:11.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:11.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:11 smithi084 ceph-mon[27254]: 2.35 scrub starts 2024-09-11T03:50:11.126 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:11.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:11.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:11.129 INFO:tasks.rados.rados.2.smithi175.stdout:3228: done (2 left) 2024-09-11T03:50:11.129 INFO:tasks.rados.rados.2.smithi175.stdout:3229: done (1 left) 2024-09-11T03:50:11.129 INFO:tasks.rados.rados.2.smithi175.stdout:3230: done (0 left) 2024-09-11T03:50:11.129 INFO:tasks.rados.rados.2.smithi175.stdout:3231: read oid 31 snap -1 2024-09-11T03:50:11.129 INFO:tasks.rados.rados.2.smithi175.stdout:3231: expect deleted 2024-09-11T03:50:11.130 INFO:tasks.rados.rados.2.smithi175.stdout:3232: write (excl) oid 12 current snap is 357 2024-09-11T03:50:11.130 INFO:tasks.rados.rados.2.smithi175.stdout:3232: seq_num 1097 ranges {437279=502985,1610629=518385} 2024-09-11T03:50:11.132 INFO:tasks.rados.rados.2.smithi175.stdout:3232: writing smithi17557870-12 from 437279 to 940264 tid 1 2024-09-11T03:50:11.139 INFO:tasks.rados.rados.2.smithi175.stdout:3232: writing smithi17557870-12 from 1610629 to 2129014 tid 2 2024-09-11T03:50:11.140 INFO:tasks.rados.rados.2.smithi175.stdout:3233: write oid 7 current snap is 357 2024-09-11T03:50:11.140 INFO:tasks.rados.rados.2.smithi175.stdout:3233: seq_num 1098 ranges {454086=578234,1669111=662043} 2024-09-11T03:50:11.145 INFO:tasks.rados.rados.2.smithi175.stdout:3233: writing smithi17557870-7 from 454086 to 1032320 tid 1 2024-09-11T03:50:11.153 INFO:tasks.rados.rados.2.smithi175.stdout:3233: writing smithi17557870-7 from 1669111 to 2331154 tid 2 2024-09-11T03:50:11.153 INFO:tasks.rados.rados.2.smithi175.stdout:3234: rollback oid 14 current snap is 357 2024-09-11T03:50:11.153 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 14 to 322 2024-09-11T03:50:11.153 INFO:tasks.rados.rados.2.smithi175.stdout:3232: finishing write tid 1 to smithi17557870-12 2024-09-11T03:50:11.153 INFO:tasks.rados.rados.2.smithi175.stdout:3232: finishing write tid 2 to smithi17557870-12 2024-09-11T03:50:11.153 INFO:tasks.rados.rados.2.smithi175.stdout:3232: finishing write tid 3 to smithi17557870-12 2024-09-11T03:50:11.153 INFO:tasks.rados.rados.2.smithi175.stdout:3232: finishing write tid 4 to smithi17557870-12 2024-09-11T03:50:11.153 INFO:tasks.rados.rados.2.smithi175.stdout:3232: oid 12 updating version 0 to 1241 2024-09-11T03:50:11.153 INFO:tasks.rados.rados.2.smithi175.stdout:3232: oid 12 version 1241 is already newer than 1239 2024-09-11T03:50:11.153 INFO:tasks.rados.rados.2.smithi175.stdout:3232: oid 12 version 1241 is already newer than 1240 2024-09-11T03:50:11.154 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 1241 (ObjNum 1097 snap 357 seq_num 1097) dirty exists 2024-09-11T03:50:11.154 INFO:tasks.rados.rados.2.smithi175.stdout:3232: left oid 12 (ObjNum 1097 snap 357 seq_num 1097) 2024-09-11T03:50:11.154 INFO:tasks.rados.rados.2.smithi175.stdout:3233: finishing write tid 1 to smithi17557870-7 2024-09-11T03:50:11.154 INFO:tasks.rados.rados.2.smithi175.stdout:3231: done (3 left) 2024-09-11T03:50:11.154 INFO:tasks.rados.rados.2.smithi175.stdout:3232: done (2 left) 2024-09-11T03:50:11.154 INFO:tasks.rados.rados.2.smithi175.stdout:3235: delete oid 4 current snap is 357 2024-09-11T03:50:11.156 INFO:tasks.rados.rados.2.smithi175.stdout:3233: finishing write tid 2 to smithi17557870-7 2024-09-11T03:50:11.156 INFO:tasks.rados.rados.2.smithi175.stdout:3233: finishing write tid 3 to smithi17557870-7 2024-09-11T03:50:11.157 INFO:tasks.rados.rados.2.smithi175.stdout:3233: finishing write tid 4 to smithi17557870-7 2024-09-11T03:50:11.157 INFO:tasks.rados.rados.2.smithi175.stdout:3233: oid 7 updating version 0 to 995 2024-09-11T03:50:11.157 INFO:tasks.rados.rados.2.smithi175.stdout:3233: oid 7 version 995 is already newer than 994 2024-09-11T03:50:11.157 INFO:tasks.rados.rados.2.smithi175.stdout:3233: oid 7 version 995 is already newer than 993 2024-09-11T03:50:11.157 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 995 (ObjNum 1098 snap 357 seq_num 1098) dirty exists 2024-09-11T03:50:11.157 INFO:tasks.rados.rados.2.smithi175.stdout:3233: left oid 7 (ObjNum 1098 snap 357 seq_num 1098) 2024-09-11T03:50:11.159 INFO:tasks.rados.rados.2.smithi175.stdout:3233: done (2 left) 2024-09-11T03:50:11.159 INFO:tasks.rados.rados.2.smithi175.stdout:3235: done (1 left) 2024-09-11T03:50:11.159 INFO:tasks.rados.rados.2.smithi175.stdout:3236: delete oid 39 current snap is 357 2024-09-11T03:50:11.159 INFO:tasks.rados.rados.2.smithi175.stdout:3234: finishing rollback tid 0 to smithi17557870-14 2024-09-11T03:50:11.160 INFO:tasks.rados.rados.2.smithi175.stdout:3234: finishing rollback tid 1 to smithi17557870-14 2024-09-11T03:50:11.160 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 1047 (ObjNum 129 snap 0 seq_num 1745219792) dirty dne 2024-09-11T03:50:11.162 INFO:tasks.rados.rados.2.smithi175.stdout:3234: done (1 left) 2024-09-11T03:50:11.162 INFO:tasks.rados.rados.2.smithi175.stdout:3236: done (0 left) 2024-09-11T03:50:11.162 INFO:tasks.rados.rados.2.smithi175.stdout:3237: copy_from oid 3 from oid 1 current snap is 357 2024-09-11T03:50:11.162 INFO:tasks.rados.rados.2.smithi175.stdout:3238: delete oid 2 current snap is 357 2024-09-11T03:50:11.164 INFO:tasks.rados.rados.2.smithi175.stdout:3238: done (1 left) 2024-09-11T03:50:11.164 INFO:tasks.rados.rados.2.smithi175.stdout:3239: delete oid 41 current snap is 357 2024-09-11T03:50:11.165 INFO:tasks.rados.rados.2.smithi175.stdout:3239: done (1 left) 2024-09-11T03:50:11.166 INFO:tasks.rados.rados.2.smithi175.stdout:3240: read oid 37 snap -1 2024-09-11T03:50:11.166 INFO:tasks.rados.rados.2.smithi175.stdout:3240: expect deleted 2024-09-11T03:50:11.166 INFO:tasks.rados.rados.2.smithi175.stdout:3241: snap_create 2024-09-11T03:50:11.168 INFO:tasks.rados.rados.2.smithi175.stdout:3237: finishing copy_from racing read to smithi17557870-3 2024-09-11T03:50:11.172 INFO:tasks.rados.rados.2.smithi175.stdout:3237: finishing copy_from to smithi17557870-3 2024-09-11T03:50:11.172 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 1006 (ObjNum 973 snap 313 seq_num 973) dirty exists 2024-09-11T03:50:11.198 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:11 smithi121 ceph-mon[30729]: 2.35 scrub starts 2024-09-11T03:50:11.199 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:11 smithi121 ceph-mon[30729]: 2.37 scrub starts 2024-09-11T03:50:11.199 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:11 smithi121 ceph-mon[30729]: 2.33 scrub starts 2024-09-11T03:50:11.199 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:11 smithi121 ceph-mon[30729]: 2.31 scrub starts 2024-09-11T03:50:11.199 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:11 smithi121 ceph-mon[30729]: 2.2f scrub starts 2024-09-11T03:50:11.199 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:11 smithi121 ceph-mon[30729]: osdmap e792: 12 total, 12 up, 12 in 2024-09-11T03:50:11.199 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:11 smithi121 ceph-mon[30729]: 2.32 scrub starts 2024-09-11T03:50:11.199 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:11 smithi121 ceph-mon[30729]: 2.34 scrub starts 2024-09-11T03:50:11.228 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:11.254 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:11.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:11.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:11.307 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:50:11.307 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:50:11.307 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:50:11.307 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:50:11.355 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:11.377 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:11.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:11.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:11.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:11 smithi084 ceph-mon[27254]: 2.37 scrub starts 2024-09-11T03:50:11.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:11 smithi084 ceph-mon[27254]: 2.33 scrub starts 2024-09-11T03:50:11.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:11 smithi084 ceph-mon[27254]: 2.31 scrub starts 2024-09-11T03:50:11.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:11 smithi084 ceph-mon[27254]: 2.2f scrub starts 2024-09-11T03:50:11.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:11 smithi084 ceph-mon[27254]: osdmap e792: 12 total, 12 up, 12 in 2024-09-11T03:50:11.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:11 smithi084 ceph-mon[27254]: 2.32 scrub starts 2024-09-11T03:50:11.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:11 smithi084 ceph-mon[27254]: 2.34 scrub starts 2024-09-11T03:50:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:11 smithi175 ceph-mon[30894]: 2.35 scrub starts 2024-09-11T03:50:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:11 smithi175 ceph-mon[30894]: 2.37 scrub starts 2024-09-11T03:50:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:11 smithi175 ceph-mon[30894]: 2.33 scrub starts 2024-09-11T03:50:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:11 smithi175 ceph-mon[30894]: 2.31 scrub starts 2024-09-11T03:50:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:11 smithi175 ceph-mon[30894]: 2.2f scrub starts 2024-09-11T03:50:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:11 smithi175 ceph-mon[30894]: osdmap e792: 12 total, 12 up, 12 in 2024-09-11T03:50:11.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:11 smithi175 ceph-mon[30894]: 2.32 scrub starts 2024-09-11T03:50:11.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:11 smithi175 ceph-mon[30894]: 2.34 scrub starts 2024-09-11T03:50:11.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:11.500 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:11.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:11.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:11.602 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:11.623 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:11.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:11.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:11.725 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:11.751 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:11.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:11.752 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:11.853 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:11.879 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:11.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:11.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:11.980 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:12.003 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:12.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:12.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:12.023 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:50:12.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T03:50:12.030 INFO:tasks.rados.rados.2.smithi175.stdout:3237: done (2 left) 2024-09-11T03:50:12.030 INFO:tasks.rados.rados.2.smithi175.stdout:3240: done (1 left) 2024-09-11T03:50:12.031 INFO:tasks.rados.rados.2.smithi175.stdout:3241: done (0 left) 2024-09-11T03:50:12.031 INFO:tasks.rados.rados.2.smithi175.stdout:3242: write (excl) oid 39 current snap is 358 2024-09-11T03:50:12.031 INFO:tasks.rados.rados.2.smithi175.stdout:3242: seq_num 1103 ranges {538121=670832,1961521=413039} 2024-09-11T03:50:12.036 INFO:tasks.rados.rados.2.smithi175.stdout:3242: writing smithi17557870-39 from 538121 to 1208953 tid 1 2024-09-11T03:50:12.043 INFO:tasks.rados.rados.2.smithi175.stdout:3242: writing smithi17557870-39 from 1961521 to 2374560 tid 2 2024-09-11T03:50:12.043 INFO:tasks.rados.rados.2.smithi175.stdout:3243: write (excl) oid 21 current snap is 358 2024-09-11T03:50:12.043 INFO:tasks.rados.rados.2.smithi175.stdout:3242: finishing write tid 1 to smithi17557870-39 2024-09-11T03:50:12.043 INFO:tasks.rados.rados.2.smithi175.stdout:3243: seq_num 1104 ranges {554928=746081,2020003=534925} 2024-09-11T03:50:12.052 INFO:tasks.rados.rados.2.smithi175.stdout:3243: writing smithi17557870-21 from 554928 to 1301009 tid 1 2024-09-11T03:50:12.054 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:50:12.058 INFO:tasks.rados.rados.2.smithi175.stdout:3243: writing smithi17557870-21 from 2020003 to 2554928 tid 2 2024-09-11T03:50:12.058 INFO:tasks.rados.rados.2.smithi175.stdout:3244: write (excl) oid 42 current snap is 358 2024-09-11T03:50:12.058 INFO:tasks.rados.rados.2.smithi175.stdout:3242: finishing write tid 2 to smithi17557870-39 2024-09-11T03:50:12.058 INFO:tasks.rados.rados.2.smithi175.stdout:3242: finishing write tid 3 to smithi17557870-39 2024-09-11T03:50:12.058 INFO:tasks.rados.rados.2.smithi175.stdout:3242: finishing write tid 4 to smithi17557870-39 2024-09-11T03:50:12.058 INFO:tasks.rados.rados.2.smithi175.stdout:3242: oid 39 updating version 0 to 1003 2024-09-11T03:50:12.058 INFO:tasks.rados.rados.2.smithi175.stdout:3242: oid 39 version 1003 is already newer than 1002 2024-09-11T03:50:12.059 INFO:tasks.rados.rados.2.smithi175.stdout:3242: oid 39 version 1003 is already newer than 1001 2024-09-11T03:50:12.059 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 1003 (ObjNum 1103 snap 358 seq_num 1103) dirty exists 2024-09-11T03:50:12.059 INFO:tasks.rados.rados.2.smithi175.stdout:3242: left oid 39 (ObjNum 1103 snap 358 seq_num 1103) 2024-09-11T03:50:12.059 INFO:tasks.rados.rados.2.smithi175.stdout:3243: finishing write tid 1 to smithi17557870-21 2024-09-11T03:50:12.059 INFO:tasks.rados.rados.2.smithi175.stdout:3244: seq_num 1105 ranges {571735=421330,1562132=416708} 2024-09-11T03:50:12.063 INFO:tasks.rados.rados.2.smithi175.stdout:3244: writing smithi17557870-42 from 571735 to 993065 tid 1 2024-09-11T03:50:12.069 INFO:tasks.rados.rados.2.smithi175.stdout:3244: writing smithi17557870-42 from 1562132 to 1978840 tid 2 2024-09-11T03:50:12.069 INFO:tasks.rados.rados.2.smithi175.stdout:3242: done (2 left) 2024-09-11T03:50:12.069 INFO:tasks.rados.rados.2.smithi175.stdout:3245: write (excl) oid 4 current snap is 358 2024-09-11T03:50:12.069 INFO:tasks.rados.rados.2.smithi175.stdout:3243: finishing write tid 2 to smithi17557870-21 2024-09-11T03:50:12.069 INFO:tasks.rados.rados.2.smithi175.stdout:3243: finishing write tid 3 to smithi17557870-21 2024-09-11T03:50:12.069 INFO:tasks.rados.rados.2.smithi175.stdout:3243: finishing write tid 4 to smithi17557870-21 2024-09-11T03:50:12.069 INFO:tasks.rados.rados.2.smithi175.stdout:3243: oid 21 updating version 0 to 1135 2024-09-11T03:50:12.069 INFO:tasks.rados.rados.2.smithi175.stdout:3243: oid 21 updating version 1135 to 1136 2024-09-11T03:50:12.069 INFO:tasks.rados.rados.2.smithi175.stdout:3243: oid 21 version 1136 is already newer than 1134 2024-09-11T03:50:12.069 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 1136 (ObjNum 1104 snap 358 seq_num 1104) dirty exists 2024-09-11T03:50:12.070 INFO:tasks.rados.rados.2.smithi175.stdout:3243: left oid 21 (ObjNum 1104 snap 358 seq_num 1104) 2024-09-11T03:50:12.070 INFO:tasks.rados.rados.2.smithi175.stdout:3244: finishing write tid 1 to smithi17557870-42 2024-09-11T03:50:12.070 INFO:tasks.rados.rados.2.smithi175.stdout:3245: seq_num 1106 ranges {588542=496579,1620614=676719} 2024-09-11T03:50:12.074 INFO:tasks.rados.rados.2.smithi175.stdout:3245: writing smithi17557870-4 from 588542 to 1085121 tid 1 2024-09-11T03:50:12.081 INFO:tasks.rados.rados.2.smithi175.stdout:3245: writing smithi17557870-4 from 1620614 to 2297333 tid 2 2024-09-11T03:50:12.082 INFO:tasks.rados.rados.2.smithi175.stdout:3243: done (2 left) 2024-09-11T03:50:12.082 INFO:tasks.rados.rados.2.smithi175.stdout:3246: write (excl) oid 9 current snap is 358 2024-09-11T03:50:12.082 INFO:tasks.rados.rados.2.smithi175.stdout:3244: finishing write tid 2 to smithi17557870-42 2024-09-11T03:50:12.082 INFO:tasks.rados.rados.2.smithi175.stdout:3244: finishing write tid 3 to smithi17557870-42 2024-09-11T03:50:12.082 INFO:tasks.rados.rados.2.smithi175.stdout:3244: finishing write tid 4 to smithi17557870-42 2024-09-11T03:50:12.082 INFO:tasks.rados.rados.2.smithi175.stdout:3244: oid 42 updating version 0 to 979 2024-09-11T03:50:12.082 INFO:tasks.rados.rados.2.smithi175.stdout:3244: oid 42 version 979 is already newer than 978 2024-09-11T03:50:12.082 INFO:tasks.rados.rados.2.smithi175.stdout:3244: oid 42 updating version 979 to 980 2024-09-11T03:50:12.082 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 980 (ObjNum 1105 snap 358 seq_num 1105) dirty exists 2024-09-11T03:50:12.082 INFO:tasks.rados.rados.2.smithi175.stdout:3244: left oid 42 (ObjNum 1105 snap 358 seq_num 1105) 2024-09-11T03:50:12.082 INFO:tasks.rados.rados.2.smithi175.stdout:3245: finishing write tid 1 to smithi17557870-4 2024-09-11T03:50:12.082 INFO:tasks.rados.rados.2.smithi175.stdout:3246: seq_num 1107 ranges {605349=571828,1679096=420377} 2024-09-11T03:50:12.087 INFO:tasks.rados.rados.2.smithi175.stdout:3246: writing smithi17557870-9 from 605349 to 1177177 tid 1 2024-09-11T03:50:12.093 INFO:tasks.rados.rados.2.smithi175.stdout:3246: writing smithi17557870-9 from 1679096 to 2099473 tid 2 2024-09-11T03:50:12.093 INFO:tasks.rados.rados.2.smithi175.stdout:3244: done (2 left) 2024-09-11T03:50:12.093 INFO:tasks.rados.rados.2.smithi175.stdout:3247: delete oid 27 current snap is 358 2024-09-11T03:50:12.093 INFO:tasks.rados.rados.2.smithi175.stdout:3245: finishing write tid 2 to smithi17557870-4 2024-09-11T03:50:12.093 INFO:tasks.rados.rados.2.smithi175.stdout:3245: finishing write tid 3 to smithi17557870-4 2024-09-11T03:50:12.093 INFO:tasks.rados.rados.2.smithi175.stdout:3245: finishing write tid 4 to smithi17557870-4 2024-09-11T03:50:12.094 INFO:tasks.rados.rados.2.smithi175.stdout:3245: oid 4 updating version 0 to 991 2024-09-11T03:50:12.094 INFO:tasks.rados.rados.2.smithi175.stdout:3245: oid 4 updating version 991 to 993 2024-09-11T03:50:12.094 INFO:tasks.rados.rados.2.smithi175.stdout:3245: oid 4 version 993 is already newer than 992 2024-09-11T03:50:12.094 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 993 (ObjNum 1106 snap 358 seq_num 1106) dirty exists 2024-09-11T03:50:12.094 INFO:tasks.rados.rados.2.smithi175.stdout:3245: left oid 4 (ObjNum 1106 snap 358 seq_num 1106) 2024-09-11T03:50:12.094 INFO:tasks.rados.rados.2.smithi175.stdout:3246: finishing write tid 1 to smithi17557870-9 2024-09-11T03:50:12.095 INFO:tasks.rados.rados.2.smithi175.stdout:3245: done (2 left) 2024-09-11T03:50:12.095 INFO:tasks.rados.rados.2.smithi175.stdout:3247: done (1 left) 2024-09-11T03:50:12.095 INFO:tasks.rados.rados.2.smithi175.stdout:3248: read oid 10 snap -1 2024-09-11T03:50:12.095 INFO:tasks.rados.rados.2.smithi175.stdout:3248: expect (ObjNum 1075 snap 350 seq_num 1075) 2024-09-11T03:50:12.096 INFO:tasks.rados.rados.2.smithi175.stdout:3249: rollback oid 28 current snap is 358 2024-09-11T03:50:12.096 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 28 to 344 2024-09-11T03:50:12.096 INFO:tasks.rados.rados.2.smithi175.stdout:3250: rollback oid 44 current snap is 358 2024-09-11T03:50:12.096 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 44 to 353 2024-09-11T03:50:12.096 INFO:tasks.rados.rados.2.smithi175.stdout:3251: write (excl) oid 21 current snap is 358 2024-09-11T03:50:12.096 INFO:tasks.rados.rados.2.smithi175.stdout:3251: seq_num 1109 ranges {638963=722326,2079707=424046} 2024-09-11T03:50:12.102 INFO:tasks.rados.rados.2.smithi175.stdout:3251: writing smithi17557870-21 from 638963 to 1361289 tid 1 2024-09-11T03:50:12.105 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:12.109 INFO:tasks.rados.rados.2.smithi175.stdout:3251: writing smithi17557870-21 from 2079707 to 2503753 tid 2 2024-09-11T03:50:12.109 INFO:tasks.rados.rados.2.smithi175.stdout:3252: read oid 49 snap 339 2024-09-11T03:50:12.109 INFO:tasks.rados.rados.2.smithi175.stdout:3252: expect deleted 2024-09-11T03:50:12.109 INFO:tasks.rados.rados.2.smithi175.stdout:3253: read oid 50 snap -1 2024-09-11T03:50:12.109 INFO:tasks.rados.rados.2.smithi175.stdout:3253: expect (ObjNum 1093 snap 355 seq_num 1093) 2024-09-11T03:50:12.109 INFO:tasks.rados.rados.2.smithi175.stdout:3254: snap_remove snap 351 2024-09-11T03:50:12.127 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:12.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:12.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:12.229 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:12.251 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:12.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:12.252 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:12.317 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:50:12.352 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:12.374 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:12.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:12.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:12.387 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:12 smithi084 ceph-mon[27254]: 2.35 scrub starts 2024-09-11T03:50:12.387 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:12 smithi084 ceph-mon[27254]: 2.33 scrub starts 2024-09-11T03:50:12.387 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:12 smithi084 ceph-mon[27254]: pgmap v1344: 87 pgs: 6 peering, 81 active+clean; 477 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:12.387 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:12 smithi084 ceph-mon[27254]: osdmap e793: 12 total, 12 up, 12 in 2024-09-11T03:50:12.387 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:12 smithi084 ceph-mon[27254]: 2.32 scrub starts 2024-09-11T03:50:12.387 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:12 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1968512660' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:12.387 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:12 smithi084 ceph-mon[27254]: 2.34 scrub starts 2024-09-11T03:50:12.387 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:12 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:12.387 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:12 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1968512660' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:12.388 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:12 smithi084 ceph-mon[27254]: osdmap e794: 12 total, 12 up, 12 in 2024-09-11T03:50:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:12 smithi121 ceph-mon[30729]: 2.35 scrub starts 2024-09-11T03:50:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:12 smithi121 ceph-mon[30729]: 2.33 scrub starts 2024-09-11T03:50:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:12 smithi121 ceph-mon[30729]: pgmap v1344: 87 pgs: 6 peering, 81 active+clean; 477 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:12 smithi121 ceph-mon[30729]: osdmap e793: 12 total, 12 up, 12 in 2024-09-11T03:50:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:12 smithi121 ceph-mon[30729]: 2.32 scrub starts 2024-09-11T03:50:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:12 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1968512660' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:12 smithi121 ceph-mon[30729]: 2.34 scrub starts 2024-09-11T03:50:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:12 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:12 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1968512660' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:12 smithi121 ceph-mon[30729]: osdmap e794: 12 total, 12 up, 12 in 2024-09-11T03:50:12.475 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:12 smithi175 ceph-mon[30894]: 2.35 scrub starts 2024-09-11T03:50:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:12 smithi175 ceph-mon[30894]: 2.33 scrub starts 2024-09-11T03:50:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:12 smithi175 ceph-mon[30894]: pgmap v1344: 87 pgs: 6 peering, 81 active+clean; 477 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:12 smithi175 ceph-mon[30894]: osdmap e793: 12 total, 12 up, 12 in 2024-09-11T03:50:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:12 smithi175 ceph-mon[30894]: 2.32 scrub starts 2024-09-11T03:50:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:12 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1968512660' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:12 smithi175 ceph-mon[30894]: 2.34 scrub starts 2024-09-11T03:50:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:12 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:12 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1968512660' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:12.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:12 smithi175 ceph-mon[30894]: osdmap e794: 12 total, 12 up, 12 in 2024-09-11T03:50:12.496 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:12.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:12.497 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:12.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:12.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:12.640 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:12.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:12.689 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:50:12.689 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:50:12.689 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-11T03:50:12.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:50:12.742 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:12.763 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:12.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:12.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:12.816 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:12.844 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:50:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:50:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:50:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:50:12.844 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:50:12.865 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:12.886 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:12.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:12.887 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:12.988 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:13.009 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:13.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:13.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:13.029 INFO:tasks.rados.rados.2.smithi175.stdout:3254: done (7 left) 2024-09-11T03:50:13.029 INFO:tasks.rados.rados.2.smithi175.stdout:3255: delete oid 6 current snap is 358 2024-09-11T03:50:13.029 INFO:tasks.rados.rados.2.smithi175.stdout:3246: finishing write tid 2 to smithi17557870-9 2024-09-11T03:50:13.029 INFO:tasks.rados.rados.2.smithi175.stdout:3246: finishing write tid 3 to smithi17557870-9 2024-09-11T03:50:13.029 INFO:tasks.rados.rados.2.smithi175.stdout:3246: finishing write tid 4 to smithi17557870-9 2024-09-11T03:50:13.030 INFO:tasks.rados.rados.2.smithi175.stdout:3246: oid 9 updating version 0 to 1140 2024-09-11T03:50:13.030 INFO:tasks.rados.rados.2.smithi175.stdout:3246: oid 9 version 1140 is already newer than 1138 2024-09-11T03:50:13.030 INFO:tasks.rados.rados.2.smithi175.stdout:3246: oid 9 version 1140 is already newer than 1139 2024-09-11T03:50:13.030 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 1140 (ObjNum 1107 snap 358 seq_num 1107) dirty exists 2024-09-11T03:50:13.030 INFO:tasks.rados.rados.2.smithi175.stdout:3246: left oid 9 (ObjNum 1107 snap 358 seq_num 1107) 2024-09-11T03:50:13.030 INFO:tasks.rados.rados.2.smithi175.stdout:3250: finishing rollback tid 0 to smithi17557870-44 2024-09-11T03:50:13.030 INFO:tasks.rados.rados.2.smithi175.stdout:3250: finishing rollback tid 1 to smithi17557870-44 2024-09-11T03:50:13.030 INFO:tasks.rados.rados.2.smithi175.stdout:3250: finishing rollback tid 2 to smithi17557870-44 2024-09-11T03:50:13.030 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 1022 (ObjNum 939 snap 297 seq_num 939) dirty exists 2024-09-11T03:50:13.030 INFO:tasks.rados.rados.2.smithi175.stdout:3249: finishing rollback tid 0 to smithi17557870-28 2024-09-11T03:50:13.030 INFO:tasks.rados.rados.2.smithi175.stdout:3249: finishing rollback tid 1 to smithi17557870-28 2024-09-11T03:50:13.030 INFO:tasks.rados.rados.2.smithi175.stdout:3249: finishing rollback tid 2 to smithi17557870-28 2024-09-11T03:50:13.030 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 1262 (ObjNum 942 snap 297 seq_num 942) dirty exists 2024-09-11T03:50:13.030 INFO:tasks.rados.rados.2.smithi175.stdout:3248: expect (ObjNum 1075 snap 350 seq_num 1075) 2024-09-11T03:50:13.111 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:13.126 INFO:tasks.rados.rados.2.smithi175.stdout:3251: finishing write tid 1 to smithi17557870-21 2024-09-11T03:50:13.126 INFO:tasks.rados.rados.2.smithi175.stdout:3251: finishing write tid 2 to smithi17557870-21 2024-09-11T03:50:13.126 INFO:tasks.rados.rados.2.smithi175.stdout:3251: finishing write tid 3 to smithi17557870-21 2024-09-11T03:50:13.126 INFO:tasks.rados.rados.2.smithi175.stdout:3251: finishing write tid 4 to smithi17557870-21 2024-09-11T03:50:13.126 INFO:tasks.rados.rados.2.smithi175.stdout:3251: oid 21 updating version 0 to 1143 2024-09-11T03:50:13.126 INFO:tasks.rados.rados.2.smithi175.stdout:3251: oid 21 version 1143 is already newer than 1142 2024-09-11T03:50:13.126 INFO:tasks.rados.rados.2.smithi175.stdout:3251: oid 21 version 1143 is already newer than 1141 2024-09-11T03:50:13.126 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 1143 (ObjNum 1109 snap 358 seq_num 1109) dirty exists 2024-09-11T03:50:13.126 INFO:tasks.rados.rados.2.smithi175.stdout:3251: left oid 21 (ObjNum 1109 snap 358 seq_num 1109) 2024-09-11T03:50:13.126 INFO:tasks.rados.rados.2.smithi175.stdout:3246: done (7 left) 2024-09-11T03:50:13.127 INFO:tasks.rados.rados.2.smithi175.stdout:3248: done (6 left) 2024-09-11T03:50:13.127 INFO:tasks.rados.rados.2.smithi175.stdout:3249: done (5 left) 2024-09-11T03:50:13.127 INFO:tasks.rados.rados.2.smithi175.stdout:3250: done (4 left) 2024-09-11T03:50:13.127 INFO:tasks.rados.rados.2.smithi175.stdout:3251: done (3 left) 2024-09-11T03:50:13.127 INFO:tasks.rados.rados.2.smithi175.stdout:3252: done (2 left) 2024-09-11T03:50:13.127 INFO:tasks.rados.rados.2.smithi175.stdout:3255: done (1 left) 2024-09-11T03:50:13.127 INFO:tasks.rados.rados.2.smithi175.stdout:3256: read oid 5 snap -1 2024-09-11T03:50:13.127 INFO:tasks.rados.rados.2.smithi175.stdout:3256: expect (ObjNum 1063 snap 347 seq_num 1063) 2024-09-11T03:50:13.127 INFO:tasks.rados.rados.2.smithi175.stdout:3257: copy_from oid 21 from oid 23 current snap is 358 2024-09-11T03:50:13.127 INFO:tasks.rados.rados.2.smithi175.stdout:3253: expect (ObjNum 1093 snap 355 seq_num 1093) 2024-09-11T03:50:13.135 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:13.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:13.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:13.237 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:13.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:13 smithi084 ceph-mon[27254]: 2.36 scrub starts 2024-09-11T03:50:13.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:13 smithi084 ceph-mon[27254]: 2.37 scrub starts 2024-09-11T03:50:13.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:13 smithi084 ceph-mon[27254]: 2.33 scrub starts 2024-09-11T03:50:13.242 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:13 smithi084 ceph-mon[27254]: 2.31 scrub starts 2024-09-11T03:50:13.242 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:13 smithi084 ceph-mon[27254]: 2.2f scrub starts 2024-09-11T03:50:13.242 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:13 smithi084 ceph-mon[27254]: 2.32 scrub starts 2024-09-11T03:50:13.242 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:13 smithi084 ceph-mon[27254]: 2.35 scrub starts 2024-09-11T03:50:13.242 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:13 smithi084 ceph-mon[27254]: 2.34 scrub starts 2024-09-11T03:50:13.242 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:13 smithi084 ceph-mon[27254]: osdmap e795: 12 total, 12 up, 12 in 2024-09-11T03:50:13.258 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:13.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:13.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:13.261 INFO:tasks.rados.rados.2.smithi175.stdout:3256: expect (ObjNum 1063 snap 347 seq_num 1063) 2024-09-11T03:50:13.349 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:50:13.350 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:50:13.357 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:50:13.357 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:50:13.360 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:13.372 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:50:13.373 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:50:13.380 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:13.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:13.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:13.389 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:50:13.389 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:50:13.406 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:50:13.407 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:50:13.419 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:13 smithi121 ceph-mon[30729]: 2.36 scrub starts 2024-09-11T03:50:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:13 smithi121 ceph-mon[30729]: 2.37 scrub starts 2024-09-11T03:50:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:13 smithi121 ceph-mon[30729]: 2.33 scrub starts 2024-09-11T03:50:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:13 smithi121 ceph-mon[30729]: 2.31 scrub starts 2024-09-11T03:50:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:13 smithi121 ceph-mon[30729]: 2.2f scrub starts 2024-09-11T03:50:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:13 smithi121 ceph-mon[30729]: 2.32 scrub starts 2024-09-11T03:50:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:13 smithi121 ceph-mon[30729]: 2.35 scrub starts 2024-09-11T03:50:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:13 smithi121 ceph-mon[30729]: 2.34 scrub starts 2024-09-11T03:50:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:13 smithi121 ceph-mon[30729]: osdmap e795: 12 total, 12 up, 12 in 2024-09-11T03:50:13.424 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:50:13.424 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:50:13.447 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:50:13.448 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:50:13.464 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:50:13.464 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:50:13.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:13 smithi175 ceph-mon[30894]: 2.36 scrub starts 2024-09-11T03:50:13.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:13 smithi175 ceph-mon[30894]: 2.37 scrub starts 2024-09-11T03:50:13.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:13 smithi175 ceph-mon[30894]: 2.33 scrub starts 2024-09-11T03:50:13.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:13 smithi175 ceph-mon[30894]: 2.31 scrub starts 2024-09-11T03:50:13.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:13 smithi175 ceph-mon[30894]: 2.2f scrub starts 2024-09-11T03:50:13.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:13 smithi175 ceph-mon[30894]: 2.32 scrub starts 2024-09-11T03:50:13.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:13 smithi175 ceph-mon[30894]: 2.35 scrub starts 2024-09-11T03:50:13.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:13 smithi175 ceph-mon[30894]: 2.34 scrub starts 2024-09-11T03:50:13.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:13 smithi175 ceph-mon[30894]: osdmap e795: 12 total, 12 up, 12 in 2024-09-11T03:50:13.482 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:50:13.482 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:50:13.482 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:13.499 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:50:13.499 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:50:13.503 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:13.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:13.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:13.515 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:50:13.515 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:50:13.532 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:50:13.532 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:50:13.538 INFO:tasks.rados.rados.2.smithi175.stdout:3257: finishing copy_from racing read to smithi17557870-21 2024-09-11T03:50:13.538 INFO:tasks.rados.rados.2.smithi175.stdout:3257: finishing copy_from to smithi17557870-21 2024-09-11T03:50:13.538 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 975 (ObjNum 1079 snap 352 seq_num 1079) dirty exists 2024-09-11T03:50:13.538 INFO:tasks.rados.rados.2.smithi175.stdout:3253: done (2 left) 2024-09-11T03:50:13.538 INFO:tasks.rados.rados.2.smithi175.stdout:3256: done (1 left) 2024-09-11T03:50:13.538 INFO:tasks.rados.rados.2.smithi175.stdout:3257: done (0 left) 2024-09-11T03:50:13.538 INFO:tasks.rados.rados.2.smithi175.stdout:3258: write (excl) oid 16 current snap is 358 2024-09-11T03:50:13.538 INFO:tasks.rados.rados.2.smithi175.stdout:3258: seq_num 1111 ranges {672577=589177,1913024=427715} 2024-09-11T03:50:13.544 INFO:tasks.rados.rados.2.smithi175.stdout:3258: writing smithi17557870-16 from 672577 to 1261754 tid 1 2024-09-11T03:50:13.551 INFO:tasks.rados.rados.2.smithi175.stdout:3258: writing smithi17557870-16 from 1913024 to 2340739 tid 2 2024-09-11T03:50:13.551 INFO:tasks.rados.rados.2.smithi175.stdout:3259: copy_from oid 9 from oid 22 current snap is 358 2024-09-11T03:50:13.551 INFO:tasks.rados.rados.2.smithi175.stdout:3258: finishing write tid 1 to smithi17557870-16 2024-09-11T03:50:13.551 INFO:tasks.rados.rados.2.smithi175.stdout:3260: write (excl) oid 36 current snap is 358 2024-09-11T03:50:13.551 INFO:tasks.rados.rados.2.smithi175.stdout:3260: seq_num 1112 ranges {689384=664426,1971506=687726} 2024-09-11T03:50:13.558 INFO:tasks.rados.rados.2.smithi175.stdout:3260: writing smithi17557870-36 from 689384 to 1353810 tid 1 2024-09-11T03:50:13.566 INFO:tasks.rados.rados.2.smithi175.stdout:3260: writing smithi17557870-36 from 1971506 to 2659232 tid 2 2024-09-11T03:50:13.566 INFO:tasks.rados.rados.2.smithi175.stdout:3261: rollback oid 38 current snap is 358 2024-09-11T03:50:13.566 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 38 to 331 2024-09-11T03:50:13.567 INFO:tasks.rados.rados.2.smithi175.stdout:3258: finishing write tid 2 to smithi17557870-16 2024-09-11T03:50:13.567 INFO:tasks.rados.rados.2.smithi175.stdout:3258: finishing write tid 3 to smithi17557870-16 2024-09-11T03:50:13.567 INFO:tasks.rados.rados.2.smithi175.stdout:3258: finishing write tid 4 to smithi17557870-16 2024-09-11T03:50:13.567 INFO:tasks.rados.rados.2.smithi175.stdout:3258: oid 16 updating version 0 to 995 2024-09-11T03:50:13.567 INFO:tasks.rados.rados.2.smithi175.stdout:3258: oid 16 version 995 is already newer than 994 2024-09-11T03:50:13.567 INFO:tasks.rados.rados.2.smithi175.stdout:3258: oid 16 updating version 995 to 996 2024-09-11T03:50:13.567 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 996 (ObjNum 1111 snap 358 seq_num 1111) dirty exists 2024-09-11T03:50:13.567 INFO:tasks.rados.rados.2.smithi175.stdout:3258: left oid 16 (ObjNum 1111 snap 358 seq_num 1111) 2024-09-11T03:50:13.567 INFO:tasks.rados.rados.2.smithi175.stdout:3259: finishing copy_from racing read to smithi17557870-9 2024-09-11T03:50:13.567 INFO:tasks.rados.rados.2.smithi175.stdout:3258: done (3 left) 2024-09-11T03:50:13.567 INFO:tasks.rados.rados.2.smithi175.stdout:3262: read oid 17 snap 331 2024-09-11T03:50:13.567 INFO:tasks.rados.rados.2.smithi175.stdout:3262: expect (ObjNum 993 snap 320 seq_num 993) 2024-09-11T03:50:13.567 INFO:tasks.rados.rados.2.smithi175.stdout:3263: read oid 15 snap -1 2024-09-11T03:50:13.567 INFO:tasks.rados.rados.2.smithi175.stdout:3263: expect deleted 2024-09-11T03:50:13.567 INFO:tasks.rados.rados.2.smithi175.stdout:3264: rollback oid 39 current snap is 358 2024-09-11T03:50:13.568 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 39 to 314 2024-09-11T03:50:13.568 INFO:tasks.rados.rados.2.smithi175.stdout:3259: finishing copy_from to smithi17557870-9 2024-09-11T03:50:13.568 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 1225 (ObjNum 1095 snap 355 seq_num 1095) dirty exists 2024-09-11T03:50:13.568 INFO:tasks.rados.rados.2.smithi175.stdout:3260: finishing write tid 1 to smithi17557870-36 2024-09-11T03:50:13.568 INFO:tasks.rados.rados.2.smithi175.stdout:3259: done (5 left) 2024-09-11T03:50:13.568 INFO:tasks.rados.rados.2.smithi175.stdout:3265: delete oid 43 current snap is 358 2024-09-11T03:50:13.568 INFO:tasks.rados.rados.2.smithi175.stdout:3263: done (5 left) 2024-09-11T03:50:13.569 INFO:tasks.rados.rados.2.smithi175.stdout:3265: done (4 left) 2024-09-11T03:50:13.569 INFO:tasks.rados.rados.2.smithi175.stdout:3266: delete oid 1 current snap is 358 2024-09-11T03:50:13.569 INFO:tasks.rados.rados.2.smithi175.stdout:3264: finishing rollback tid 0 to smithi17557870-39 2024-09-11T03:50:13.569 INFO:tasks.rados.rados.2.smithi175.stdout:3264: finishing rollback tid 1 to smithi17557870-39 2024-09-11T03:50:13.569 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 1005 (ObjNum 81 snap 0 seq_num 1) dirty dne 2024-09-11T03:50:13.570 INFO:tasks.rados.rados.2.smithi175.stdout:3260: finishing write tid 2 to smithi17557870-36 2024-09-11T03:50:13.570 INFO:tasks.rados.rados.2.smithi175.stdout:3264: done (4 left) 2024-09-11T03:50:13.570 INFO:tasks.rados.rados.2.smithi175.stdout:3266: done (3 left) 2024-09-11T03:50:13.570 INFO:tasks.rados.rados.2.smithi175.stdout:3267: rollback oid 7 current snap is 358 2024-09-11T03:50:13.570 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 7 to 347 2024-09-11T03:50:13.570 INFO:tasks.rados.rados.2.smithi175.stdout:3268: read oid 12 snap -1 2024-09-11T03:50:13.570 INFO:tasks.rados.rados.2.smithi175.stdout:3268: expect (ObjNum 1097 snap 357 seq_num 1097) 2024-09-11T03:50:13.570 INFO:tasks.rados.rados.2.smithi175.stdout:3260: finishing write tid 3 to smithi17557870-36 2024-09-11T03:50:13.571 INFO:tasks.rados.rados.2.smithi175.stdout:3269: delete oid 34 current snap is 358 2024-09-11T03:50:13.571 INFO:tasks.rados.rados.2.smithi175.stdout:3260: finishing write tid 4 to smithi17557870-36 2024-09-11T03:50:13.571 INFO:tasks.rados.rados.2.smithi175.stdout:3260: oid 36 updating version 0 to 982 2024-09-11T03:50:13.571 INFO:tasks.rados.rados.2.smithi175.stdout:3260: oid 36 version 982 is already newer than 980 2024-09-11T03:50:13.571 INFO:tasks.rados.rados.2.smithi175.stdout:3260: oid 36 version 982 is already newer than 981 2024-09-11T03:50:13.571 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 982 (ObjNum 1112 snap 358 seq_num 1112) dirty exists 2024-09-11T03:50:13.571 INFO:tasks.rados.rados.2.smithi175.stdout:3260: left oid 36 (ObjNum 1112 snap 358 seq_num 1112) 2024-09-11T03:50:13.572 INFO:tasks.rados.rados.2.smithi175.stdout:3260: done (5 left) 2024-09-11T03:50:13.572 INFO:tasks.rados.rados.2.smithi175.stdout:3269: done (4 left) 2024-09-11T03:50:13.572 INFO:tasks.rados.rados.2.smithi175.stdout:3270: delete oid 2 current snap is 358 2024-09-11T03:50:13.572 INFO:tasks.rados.rados.2.smithi175.stdout:3267: finishing rollback tid 0 to smithi17557870-7 2024-09-11T03:50:13.573 INFO:tasks.rados.rados.2.smithi175.stdout:3267: finishing rollback tid 1 to smithi17557870-7 2024-09-11T03:50:13.575 INFO:tasks.rados.rados.2.smithi175.stdout:3261: finishing rollback tid 1 to smithi17557870-38 2024-09-11T03:50:13.575 INFO:tasks.rados.rados.2.smithi175.stdout:3261: finishing rollback tid 2 to smithi17557870-38 2024-09-11T03:50:13.575 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 1245 (ObjNum 970 snap 311 seq_num 970) dirty exists 2024-09-11T03:50:13.575 INFO:tasks.rados.rados.2.smithi175.stdout:3267: finishing rollback tid 2 to smithi17557870-7 2024-09-11T03:50:13.575 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 998 (ObjNum 1061 snap 346 seq_num 1061) dirty exists 2024-09-11T03:50:13.576 INFO:tasks.rados.rados.2.smithi175.stdout:3268: expect (ObjNum 1097 snap 357 seq_num 1097) 2024-09-11T03:50:13.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T03:50:13.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:13.664 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:13.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:13.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:13.684 INFO:tasks.rados.rados.2.smithi175.stdout:3262: expect (ObjNum 993 snap 320 seq_num 993) 2024-09-11T03:50:13.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:13.787 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:13.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:13.839 INFO:tasks.rados.rados.2.smithi175.stdout:3261: done (4 left) 2024-09-11T03:50:13.839 INFO:tasks.rados.rados.2.smithi175.stdout:3262: done (3 left) 2024-09-11T03:50:13.839 INFO:tasks.rados.rados.2.smithi175.stdout:3267: done (2 left) 2024-09-11T03:50:13.839 INFO:tasks.rados.rados.2.smithi175.stdout:3268: done (1 left) 2024-09-11T03:50:13.839 INFO:tasks.rados.rados.2.smithi175.stdout:3270: done (0 left) 2024-09-11T03:50:13.839 INFO:tasks.rados.rados.2.smithi175.stdout:3271: copy_from oid 16 from oid 31 current snap is 358 2024-09-11T03:50:13.839 INFO:tasks.rados.rados.2.smithi175.stdout:3272: delete oid 17 current snap is 358 2024-09-11T03:50:13.840 INFO:tasks.rados.rados.2.smithi175.stdout:3271: finishing copy_from to smithi17557870-16 2024-09-11T03:50:13.840 INFO:tasks.rados.rados.2.smithi175.stdout:3271: got expected ENOENT (src dne) 2024-09-11T03:50:13.840 INFO:tasks.rados.rados.2.smithi175.stdout:3271: finishing copy_from racing read to smithi17557870-16 2024-09-11T03:50:13.841 INFO:tasks.rados.rados.2.smithi175.stdout:3271: done (1 left) 2024-09-11T03:50:13.841 INFO:tasks.rados.rados.2.smithi175.stdout:3272: done (0 left) 2024-09-11T03:50:13.841 INFO:tasks.rados.rados.2.smithi175.stdout:3273: snap_create 2024-09-11T03:50:13.889 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:50:13.889 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:13.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:13.911 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:13.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:14.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:14.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:14.035 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:14.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:14.076 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:50:14.077 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:50:14.136 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:14.138 INFO:tasks.rados.rados.2.smithi175.stdout:3273: done (0 left) 2024-09-11T03:50:14.138 INFO:tasks.rados.rados.2.smithi175.stdout:3274: copy_from oid 10 from oid 15 current snap is 359 2024-09-11T03:50:14.139 INFO:tasks.rados.rados.2.smithi175.stdout:3275: write (excl) oid 34 current snap is 359 2024-09-11T03:50:14.139 INFO:tasks.rados.rados.2.smithi175.stdout:3275: seq_num 1118 ranges {790226=432273,1806045=582380} 2024-09-11T03:50:14.144 INFO:tasks.rados.rados.2.smithi175.stdout:3275: writing smithi17557870-34 from 790226 to 1222499 tid 1 2024-09-11T03:50:14.150 INFO:tasks.rados.rados.2.smithi175.stdout:3275: writing smithi17557870-34 from 1806045 to 2388425 tid 2 2024-09-11T03:50:14.150 INFO:tasks.rados.rados.2.smithi175.stdout:3276: snap_remove snap 356 2024-09-11T03:50:14.160 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:14.161 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:14.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:14 smithi121 ceph-mon[30729]: 2.37 scrub starts 2024-09-11T03:50:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:14 smithi121 ceph-mon[30729]: 2.33 scrub starts 2024-09-11T03:50:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:14 smithi121 ceph-mon[30729]: pgmap v1347: 87 pgs: 2 active+clean+scrubbing, 85 active+clean; 496 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.7 MiB/s wr, 23 op/s 2024-09-11T03:50:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:14 smithi121 ceph-mon[30729]: 2.31 scrub starts 2024-09-11T03:50:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:14 smithi121 ceph-mon[30729]: 2.2f scrub starts 2024-09-11T03:50:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:14 smithi121 ceph-mon[30729]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-11T03:50:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:14 smithi121 ceph-mon[30729]: 2.32 scrub starts 2024-09-11T03:50:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:14 smithi121 ceph-mon[30729]: 2.35 scrub starts 2024-09-11T03:50:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:14 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3444580257' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:14.264 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:14 smithi121 ceph-mon[30729]: 2.34 scrub starts 2024-09-11T03:50:14.284 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:14.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:14.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:14.371 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:50:14.372 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:50:14.385 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:14.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:14 smithi084 ceph-mon[27254]: 2.37 scrub starts 2024-09-11T03:50:14.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:14 smithi084 ceph-mon[27254]: 2.33 scrub starts 2024-09-11T03:50:14.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:14 smithi084 ceph-mon[27254]: pgmap v1347: 87 pgs: 2 active+clean+scrubbing, 85 active+clean; 496 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.7 MiB/s wr, 23 op/s 2024-09-11T03:50:14.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:14 smithi084 ceph-mon[27254]: 2.31 scrub starts 2024-09-11T03:50:14.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:14 smithi084 ceph-mon[27254]: 2.2f scrub starts 2024-09-11T03:50:14.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:14 smithi084 ceph-mon[27254]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-11T03:50:14.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:14 smithi084 ceph-mon[27254]: 2.32 scrub starts 2024-09-11T03:50:14.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:14 smithi084 ceph-mon[27254]: 2.35 scrub starts 2024-09-11T03:50:14.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:14 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3444580257' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:14.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:14 smithi084 ceph-mon[27254]: 2.34 scrub starts 2024-09-11T03:50:14.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:14.407 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:14.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:14 smithi175 ceph-mon[30894]: 2.37 scrub starts 2024-09-11T03:50:14.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:14 smithi175 ceph-mon[30894]: 2.33 scrub starts 2024-09-11T03:50:14.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:14 smithi175 ceph-mon[30894]: pgmap v1347: 87 pgs: 2 active+clean+scrubbing, 85 active+clean; 496 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.7 MiB/s wr, 23 op/s 2024-09-11T03:50:14.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:14 smithi175 ceph-mon[30894]: 2.31 scrub starts 2024-09-11T03:50:14.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:14 smithi175 ceph-mon[30894]: 2.2f scrub starts 2024-09-11T03:50:14.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:14 smithi175 ceph-mon[30894]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-11T03:50:14.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:14 smithi175 ceph-mon[30894]: 2.32 scrub starts 2024-09-11T03:50:14.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:14 smithi175 ceph-mon[30894]: 2.35 scrub starts 2024-09-11T03:50:14.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:14 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3444580257' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:14.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:14 smithi175 ceph-mon[30894]: 2.34 scrub starts 2024-09-11T03:50:14.509 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:14.530 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:14.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:14.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:14.632 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:14.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:14.654 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:14.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:14.756 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:14.779 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:14.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:14.780 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:14.816 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:50:14.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:14.902 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:14.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:14.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:15.003 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:15.025 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:15.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:15.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:15.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T03:50:15.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:15.147 INFO:tasks.rados.rados.2.smithi175.stdout:3274: finishing copy_from to smithi17557870-10 2024-09-11T03:50:15.147 INFO:tasks.rados.rados.2.smithi175.stdout:3274: got expected ENOENT (src dne) 2024-09-11T03:50:15.147 INFO:tasks.rados.rados.2.smithi175.stdout:3274: finishing copy_from racing read to smithi17557870-10 2024-09-11T03:50:15.147 INFO:tasks.rados.rados.2.smithi175.stdout:3275: finishing write tid 1 to smithi17557870-34 2024-09-11T03:50:15.147 INFO:tasks.rados.rados.2.smithi175.stdout:3275: finishing write tid 2 to smithi17557870-34 2024-09-11T03:50:15.147 INFO:tasks.rados.rados.2.smithi175.stdout:3275: finishing write tid 3 to smithi17557870-34 2024-09-11T03:50:15.147 INFO:tasks.rados.rados.2.smithi175.stdout:3275: finishing write tid 4 to smithi17557870-34 2024-09-11T03:50:15.147 INFO:tasks.rados.rados.2.smithi175.stdout:3275: oid 34 updating version 0 to 1046 2024-09-11T03:50:15.147 INFO:tasks.rados.rados.2.smithi175.stdout:3275: oid 34 updating version 1046 to 1048 2024-09-11T03:50:15.147 INFO:tasks.rados.rados.2.smithi175.stdout:3275: oid 34 version 1048 is already newer than 1047 2024-09-11T03:50:15.147 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 1048 (ObjNum 1118 snap 359 seq_num 1118) dirty exists 2024-09-11T03:50:15.147 INFO:tasks.rados.rados.2.smithi175.stdout:3275: left oid 34 (ObjNum 1118 snap 359 seq_num 1118) 2024-09-11T03:50:15.147 INFO:tasks.rados.rados.2.smithi175.stdout:3274: done (2 left) 2024-09-11T03:50:15.148 INFO:tasks.rados.rados.2.smithi175.stdout:3275: done (1 left) 2024-09-11T03:50:15.148 INFO:tasks.rados.rados.2.smithi175.stdout:3276: done (0 left) 2024-09-11T03:50:15.148 INFO:tasks.rados.rados.2.smithi175.stdout:3277: read oid 10 snap -1 2024-09-11T03:50:15.148 INFO:tasks.rados.rados.2.smithi175.stdout:3277: expect (ObjNum 1075 snap 350 seq_num 1075) 2024-09-11T03:50:15.148 INFO:tasks.rados.rados.2.smithi175.stdout:3278: rollback oid 45 current snap is 359 2024-09-11T03:50:15.148 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 45 to 358 2024-09-11T03:50:15.148 INFO:tasks.rados.rados.2.smithi175.stdout:3279: delete oid 22 current snap is 359 2024-09-11T03:50:15.149 INFO:tasks.rados.rados.2.smithi175.stdout:3278: finishing rollback tid 0 to smithi17557870-45 2024-09-11T03:50:15.150 INFO:tasks.rados.rados.2.smithi175.stdout:3278: finishing rollback tid 1 to smithi17557870-45 2024-09-11T03:50:15.151 INFO:tasks.rados.rados.2.smithi175.stdout:3278: finishing rollback tid 2 to smithi17557870-45 2024-09-11T03:50:15.151 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 995 (ObjNum 1094 snap 355 seq_num 1094) dirty exists 2024-09-11T03:50:15.157 INFO:tasks.rados.rados.2.smithi175.stdout:3277: expect (ObjNum 1075 snap 350 seq_num 1075) 2024-09-11T03:50:15.158 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:15.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:15.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:15.251 INFO:tasks.rados.rados.2.smithi175.stdout:3277: done (2 left) 2024-09-11T03:50:15.251 INFO:tasks.rados.rados.2.smithi175.stdout:3278: done (1 left) 2024-09-11T03:50:15.252 INFO:tasks.rados.rados.2.smithi175.stdout:3279: done (0 left) 2024-09-11T03:50:15.252 INFO:tasks.rados.rados.2.smithi175.stdout:3280: write oid 24 current snap is 359 2024-09-11T03:50:15.252 INFO:tasks.rados.rados.2.smithi175.stdout:3280: seq_num 1120 ranges {423840=582771,1523009=586049,2657046=166794} 2024-09-11T03:50:15.256 INFO:tasks.rados.rados.2.smithi175.stdout:3280: writing smithi17557870-24 from 423840 to 1006611 tid 1 2024-09-11T03:50:15.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:15.262 INFO:tasks.rados.rados.2.smithi175.stdout:3280: writing smithi17557870-24 from 1523009 to 2109058 tid 2 2024-09-11T03:50:15.266 INFO:tasks.rados.rados.2.smithi175.stdout:3280: writing smithi17557870-24 from 2657046 to 2823840 tid 3 2024-09-11T03:50:15.266 INFO:tasks.rados.rados.2.smithi175.stdout:3281: snap_create 2024-09-11T03:50:15.266 INFO:tasks.rados.rados.2.smithi175.stdout:3280: finishing write tid 1 to smithi17557870-24 2024-09-11T03:50:15.267 INFO:tasks.rados.rados.2.smithi175.stdout:3280: finishing write tid 2 to smithi17557870-24 2024-09-11T03:50:15.268 INFO:tasks.rados.rados.2.smithi175.stdout:3280: finishing write tid 3 to smithi17557870-24 2024-09-11T03:50:15.269 INFO:tasks.rados.rados.2.smithi175.stdout:3280: finishing write tid 4 to smithi17557870-24 2024-09-11T03:50:15.269 INFO:tasks.rados.rados.2.smithi175.stdout:3280: finishing write tid 5 to smithi17557870-24 2024-09-11T03:50:15.269 INFO:tasks.rados.rados.2.smithi175.stdout:3280: oid 24 updating version 0 to 1051 2024-09-11T03:50:15.269 INFO:tasks.rados.rados.2.smithi175.stdout:3280: oid 24 updating version 1051 to 1053 2024-09-11T03:50:15.269 INFO:tasks.rados.rados.2.smithi175.stdout:3280: oid 24 version 1053 is already newer than 1052 2024-09-11T03:50:15.269 INFO:tasks.rados.rados.2.smithi175.stdout:3280: oid 24 version 1053 is already newer than 1050 2024-09-11T03:50:15.269 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 1053 (ObjNum 1120 snap 359 seq_num 1120) dirty exists 2024-09-11T03:50:15.269 INFO:tasks.rados.rados.2.smithi175.stdout:3280: left oid 24 (ObjNum 1120 snap 359 seq_num 1120) 2024-09-11T03:50:15.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:15.282 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:15.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:15.333 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:50:15.383 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:15.389 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1348,"stamp":"2024-09-11T03:50:13.026277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":520555766,"num_objects":246,"num_object_clones":195,"num_object_copies":738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":246,"num_whiteouts":17,"num_read":1436,"num_read_kb":1499093,"num_write":995,"num_write_kb":428502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":56337953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33916,"ondisk_log_size":33916,"up":261,"acting":261,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":268,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":1545752,"kb_used_data":1020160,"kb_used_omap":18,"kb_used_meta":525293,"kb_avail":1123396072,"statfs":{"total":1151940427776,"available":1150357577728,"internally_reserved":0,"allocated":1044643840,"data_stored":1339262171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":537900347},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20756798,"num_objects":18,"num_object_clones":18,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":4,"num_read":41,"num_read_kb":24566,"num_write":52,"num_write_kb":15158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.918489"},"pg_stats":[{"pgid":"2.37","version":"274'312","reported_seq":2118,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.056659+0000","last_change":"2024-09-11T03:50:12.056595+0000","last_active":"2024-09-11T03:50:12.056659+0000","last_peered":"2024-09-11T03:50:12.056659+0000","last_clean":"2024-09-11T03:50:12.056659+0000","last_became_active":"2024-09-11T03:50:07.077607+0000","last_became_peered":"2024-09-11T03:50:07.077607+0000","last_unstale":"2024-09-11T03:50:12.056659+0000","last_undegraded":"2024-09-11T03:50:12.056659+0000","last_fullsized":"2024-09-11T03:50:12.056659+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:49:38.613856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:38.613856+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00089849000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,9],"acting":[10,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"2.36","version":"793'575","reported_seq":4150,"reported_epoch":793,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T03:50:11.325043+0000","last_change":"2024-09-11T03:50:11.324884+0000","last_active":"2024-09-11T03:50:11.325043+0000","last_peered":"2024-09-11T03:50:11.325043+0000","last_clean":"2024-09-11T03:50:11.325043+0000","last_became_active":"2024-09-11T03:50:07.078667+0000","last_became_peered":"2024-09-11T03:50:07.078667+0000","last_unstale":"2024-09-11T03:50:11.325043+0000","last_undegraded":"2024-09-11T03:50:11.325043+0000","last_fullsized":"2024-09-11T03:50:11.325043+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"755'521","last_scrub_stamp":"2024-09-11T03:49:33.626967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.626967+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00062598799999999996,"stat_sum":{"num_bytes":11240765,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":73,"num_read_kb":66412,"num_write":60,"num_write_kb":25037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":15316121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,8],"acting":[1,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"2.35","version":"274'273","reported_seq":1284,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.120231+0000","last_change":"2024-09-11T03:50:11.120155+0000","last_active":"2024-09-11T03:50:11.120231+0000","last_peered":"2024-09-11T03:50:11.120231+0000","last_clean":"2024-09-11T03:50:11.120231+0000","last_became_active":"2024-09-11T03:50:07.079331+0000","last_became_peered":"2024-09-11T03:50:07.079331+0000","last_unstale":"2024-09-11T03:50:11.120231+0000","last_undegraded":"2024-09-11T03:50:11.120231+0000","last_fullsized":"2024-09-11T03:50:11.120231+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:49:16.919210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:16.919210+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00077651499999999997,"stat_sum":{"num_bytes":4574079,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":48602,"num_write":32,"num_write_kb":11190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"2.34","version":"789'357","reported_seq":3861,"reported_epoch":793,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T03:50:11.633877+0000","last_change":"2024-09-11T03:50:11.633634+0000","last_active":"2024-09-11T03:50:11.633877+0000","last_peered":"2024-09-11T03:50:11.633877+0000","last_clean":"2024-09-11T03:50:11.633877+0000","last_became_active":"2024-09-11T03:50:07.080419+0000","last_became_peered":"2024-09-11T03:50:07.080419+0000","last_unstale":"2024-09-11T03:50:11.633877+0000","last_undegraded":"2024-09-11T03:50:11.633877+0000","last_fullsized":"2024-09-11T03:50:11.633877+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"778'353","last_scrub_stamp":"2024-09-11T03:49:58.635588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:58.635588+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021685110000000001,"stat_sum":{"num_bytes":3147186,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":24,"num_read_kb":13801,"num_write":32,"num_write_kb":9233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":5044887,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"2.33","version":"274'111","reported_seq":1065,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.130039+0000","last_change":"2024-09-11T03:50:11.129960+0000","last_active":"2024-09-11T03:50:11.130039+0000","last_peered":"2024-09-11T03:50:11.130039+0000","last_clean":"2024-09-11T03:50:11.130039+0000","last_became_active":"2024-09-11T03:50:07.077684+0000","last_became_peered":"2024-09-11T03:50:07.077684+0000","last_unstale":"2024-09-11T03:50:11.130039+0000","last_undegraded":"2024-09-11T03:50:11.130039+0000","last_fullsized":"2024-09-11T03:50:11.130039+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:49:15.904230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:15.904230+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00081941499999999999,"stat_sum":{"num_bytes":1352972,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":10616,"num_write":11,"num_write_kb":4125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"2.32","version":"778'549","reported_seq":4052,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.040395+0000","last_change":"2024-09-11T03:50:12.040312+0000","last_active":"2024-09-11T03:50:12.040395+0000","last_peered":"2024-09-11T03:50:12.040395+0000","last_clean":"2024-09-11T03:50:12.040395+0000","last_became_active":"2024-09-11T03:50:07.077594+0000","last_became_peered":"2024-09-11T03:50:07.077594+0000","last_unstale":"2024-09-11T03:50:12.040395+0000","last_undegraded":"2024-09-11T03:50:12.040395+0000","last_fullsized":"2024-09-11T03:50:12.040395+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"736'523","last_scrub_stamp":"2024-09-11T03:49:20.735392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.735392+0000","objects_scrubbed":0,"log_size":549,"log_dups_size":0,"ondisk_log_size":549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011558919999999999,"stat_sum":{"num_bytes":5999111,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":105,"num_read_kb":111976,"num_write":70,"num_write_kb":26123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":10380480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9],"acting":[11,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"2.31","version":"274'107","reported_seq":1555,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-09-11T03:50:10.937103+0000","last_change":"2024-09-11T03:50:10.937103+0000","last_active":"2024-09-11T03:50:10.937103+0000","last_peered":"2024-09-11T03:50:10.937103+0000","last_clean":"2024-09-11T03:50:10.937103+0000","last_became_active":"2024-09-11T03:50:07.075956+0000","last_became_peered":"2024-09-11T03:50:07.075956+0000","last_unstale":"2024-09-11T03:50:10.937103+0000","last_undegraded":"2024-09-11T03:50:10.937103+0000","last_fullsized":"2024-09-11T03:50:10.937103+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:49:40.618009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:40.618009+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00048876300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,4],"acting":[0,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"2.30","version":"274'67","reported_seq":59,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.056422+0000","last_change":"2024-09-11T03:50:10.614912+0000","last_active":"2024-09-11T03:50:12.056422+0000","last_peered":"2024-09-11T03:50:12.056422+0000","last_clean":"2024-09-11T03:50:12.056422+0000","last_became_active":"2024-09-11T03:50:04.069393+0000","last_became_peered":"2024-09-11T03:50:04.069393+0000","last_unstale":"2024-09-11T03:50:12.056422+0000","last_undegraded":"2024-09-11T03:50:12.056422+0000","last_fullsized":"2024-09-11T03:50:12.056422+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:50:04.608333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.608333+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:32.428864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076297599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,9],"acting":[10,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"2.2f","version":"557'465","reported_seq":2859,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-09-11T03:50:08.080857+0000","last_change":"2024-09-11T03:50:08.080780+0000","last_active":"2024-09-11T03:50:08.080857+0000","last_peered":"2024-09-11T03:50:08.080857+0000","last_clean":"2024-09-11T03:50:08.080857+0000","last_became_active":"2024-09-11T03:50:07.077840+0000","last_became_peered":"2024-09-11T03:50:07.077840+0000","last_unstale":"2024-09-11T03:50:08.080857+0000","last_undegraded":"2024-09-11T03:50:08.080857+0000","last_fullsized":"2024-09-11T03:50:08.080857+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:48:59.211781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:59.211781+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0033554219999999998,"stat_sum":{"num_bytes":2643000,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":39368,"num_write":39,"num_write_kb":14570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"}]},{"pgid":"2.2e","version":"790'1041","reported_seq":4509,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.130342+0000","last_change":"2024-09-11T03:50:10.113259+0000","last_active":"2024-09-11T03:50:11.130342+0000","last_peered":"2024-09-11T03:50:11.130342+0000","last_clean":"2024-09-11T03:50:11.130342+0000","last_became_active":"2024-09-11T03:50:07.078910+0000","last_became_peered":"2024-09-11T03:50:07.078910+0000","last_unstale":"2024-09-11T03:50:11.130342+0000","last_undegraded":"2024-09-11T03:50:11.130342+0000","last_fullsized":"2024-09-11T03:50:11.130342+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"736'963","last_scrub_stamp":"2024-09-11T03:49:17.918033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:17.918033+0000","objects_scrubbed":8,"log_size":1041,"log_dups_size":0,"ondisk_log_size":1041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001376677,"stat_sum":{"num_bytes":13507178,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":35,"num_read_kb":33112,"num_write":31,"num_write_kb":15892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":15532200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"2.2d","version":"787'1077","reported_seq":4597,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.098997+0000","last_change":"2024-09-11T03:50:10.646868+0000","last_active":"2024-09-11T03:50:12.098997+0000","last_peered":"2024-09-11T03:50:12.098997+0000","last_clean":"2024-09-11T03:50:12.098997+0000","last_became_active":"2024-09-11T03:49:29.163201+0000","last_became_peered":"2024-09-11T03:49:29.163201+0000","last_unstale":"2024-09-11T03:50:12.098997+0000","last_undegraded":"2024-09-11T03:50:12.098997+0000","last_fullsized":"2024-09-11T03:50:12.098997+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'997","last_scrub_stamp":"2024-09-11T03:49:32.462332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.462332+0000","objects_scrubbed":13,"log_size":1077,"log_dups_size":0,"ondisk_log_size":1077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:39.459985+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00082447,"stat_sum":{"num_bytes":38880454,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":23,"num_read_kb":19881,"num_write":9,"num_write_kb":10705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"2.2c","version":"787'844","reported_seq":4362,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.136254+0000","last_change":"2024-09-11T03:50:10.112913+0000","last_active":"2024-09-11T03:50:11.136254+0000","last_peered":"2024-09-11T03:50:11.136254+0000","last_clean":"2024-09-11T03:50:11.136254+0000","last_became_active":"2024-09-11T03:49:29.154872+0000","last_became_peered":"2024-09-11T03:49:29.154872+0000","last_unstale":"2024-09-11T03:50:11.136254+0000","last_undegraded":"2024-09-11T03:50:11.136254+0000","last_fullsized":"2024-09-11T03:50:11.136254+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"753'800","last_scrub_stamp":"2024-09-11T03:49:32.185401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.185401+0000","objects_scrubbed":4,"log_size":844,"log_dups_size":0,"ondisk_log_size":844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:00.649873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010961129999999999,"stat_sum":{"num_bytes":19839498,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":23,"num_read_kb":19598,"num_write":21,"num_write_kb":8967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3],"acting":[8,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"2.2b","version":"568'646","reported_seq":3283,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-09-11T03:50:08.080401+0000","last_change":"2024-09-11T03:50:05.066384+0000","last_active":"2024-09-11T03:50:08.080401+0000","last_peered":"2024-09-11T03:50:08.080401+0000","last_clean":"2024-09-11T03:50:08.080401+0000","last_became_active":"2024-09-11T03:49:29.157735+0000","last_became_peered":"2024-09-11T03:49:29.157735+0000","last_unstale":"2024-09-11T03:50:08.080401+0000","last_undegraded":"2024-09-11T03:50:08.080401+0000","last_fullsized":"2024-09-11T03:50:08.080401+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:49:32.876969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.876969+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:34.748477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003257982,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"}]},{"pgid":"2.2a","version":"727'348","reported_seq":3779,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.130063+0000","last_change":"2024-09-11T03:50:10.112343+0000","last_active":"2024-09-11T03:50:11.130063+0000","last_peered":"2024-09-11T03:50:11.130063+0000","last_clean":"2024-09-11T03:50:11.130063+0000","last_became_active":"2024-09-11T03:49:29.160595+0000","last_became_peered":"2024-09-11T03:49:29.160595+0000","last_unstale":"2024-09-11T03:50:11.130063+0000","last_undegraded":"2024-09-11T03:50:11.130063+0000","last_fullsized":"2024-09-11T03:50:11.130063+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:49:34.219244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.219244+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:39.181938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001259989,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,5],"acting":[7,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"2.29","version":"793'1227","reported_seq":4860,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.054078+0000","last_change":"2024-09-11T03:50:10.647673+0000","last_active":"2024-09-11T03:50:12.054078+0000","last_peered":"2024-09-11T03:50:12.054078+0000","last_clean":"2024-09-11T03:50:12.054078+0000","last_became_active":"2024-09-11T03:49:29.157194+0000","last_became_peered":"2024-09-11T03:49:29.157194+0000","last_unstale":"2024-09-11T03:50:12.054078+0000","last_undegraded":"2024-09-11T03:50:12.054078+0000","last_fullsized":"2024-09-11T03:50:12.054078+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'1167","last_scrub_stamp":"2024-09-11T03:49:33.506786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.506786+0000","objects_scrubbed":11,"log_size":1227,"log_dups_size":0,"ondisk_log_size":1227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:48.277633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0013991170000000001,"stat_sum":{"num_bytes":33520458,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":31567,"num_write":19,"num_write_kb":8705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8],"acting":[6,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"2.28","version":"787'400","reported_seq":3932,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.119925+0000","last_change":"2024-09-11T03:50:10.110514+0000","last_active":"2024-09-11T03:50:11.119925+0000","last_peered":"2024-09-11T03:50:11.119925+0000","last_clean":"2024-09-11T03:50:11.119925+0000","last_became_active":"2024-09-11T03:49:29.154884+0000","last_became_peered":"2024-09-11T03:49:29.154884+0000","last_unstale":"2024-09-11T03:50:11.119925+0000","last_undegraded":"2024-09-11T03:50:11.119925+0000","last_fullsized":"2024-09-11T03:50:11.119925+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'382","last_scrub_stamp":"2024-09-11T03:49:32.335285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.335285+0000","objects_scrubbed":4,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:44.314918+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051936099999999998,"stat_sum":{"num_bytes":8486578,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":9,"num_read_kb":11526,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"2.27","version":"787'1103","reported_seq":4742,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.130198+0000","last_change":"2024-09-11T03:50:10.113405+0000","last_active":"2024-09-11T03:50:11.130198+0000","last_peered":"2024-09-11T03:50:11.130198+0000","last_clean":"2024-09-11T03:50:11.130198+0000","last_became_active":"2024-09-11T03:49:29.159799+0000","last_became_peered":"2024-09-11T03:49:29.159799+0000","last_unstale":"2024-09-11T03:50:11.130198+0000","last_undegraded":"2024-09-11T03:50:11.130198+0000","last_fullsized":"2024-09-11T03:50:11.130198+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"757'1065","last_scrub_stamp":"2024-09-11T03:49:36.196385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.196385+0000","objects_scrubbed":4,"log_size":1103,"log_dups_size":0,"ondisk_log_size":1103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:44.594068+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001495688,"stat_sum":{"num_bytes":18522601,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":14,"num_read_kb":13448,"num_write":7,"num_write_kb":7771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,9],"acting":[7,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"2.26","version":"269'171","reported_seq":3664,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.130235+0000","last_change":"2024-09-11T03:50:10.113030+0000","last_active":"2024-09-11T03:50:11.130235+0000","last_peered":"2024-09-11T03:50:11.130235+0000","last_clean":"2024-09-11T03:50:11.130235+0000","last_became_active":"2024-09-11T03:49:29.158577+0000","last_became_peered":"2024-09-11T03:49:29.158577+0000","last_unstale":"2024-09-11T03:50:11.130235+0000","last_undegraded":"2024-09-11T03:50:11.130235+0000","last_fullsized":"2024-09-11T03:50:11.130235+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:49:33.199779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.199779+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:42.655289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00140386,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,11],"acting":[7,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"2.25","version":"793'1241","reported_seq":4977,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.143820+0000","last_change":"2024-09-11T03:50:10.111787+0000","last_active":"2024-09-11T03:50:11.143820+0000","last_peered":"2024-09-11T03:50:11.143820+0000","last_clean":"2024-09-11T03:50:11.143820+0000","last_became_active":"2024-09-11T03:49:29.159733+0000","last_became_peered":"2024-09-11T03:49:29.159733+0000","last_unstale":"2024-09-11T03:50:11.143820+0000","last_undegraded":"2024-09-11T03:50:11.143820+0000","last_fullsized":"2024-09-11T03:50:11.143820+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"755'1138","last_scrub_stamp":"2024-09-11T03:49:34.198067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.198067+0000","objects_scrubbed":16,"log_size":1241,"log_dups_size":0,"ondisk_log_size":1241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:50.164552+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00076126199999999996,"stat_sum":{"num_bytes":34144557,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":30,"num_read_kb":29290,"num_write":17,"num_write_kb":6449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,6],"acting":[8,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"2.24","version":"787'646","reported_seq":4197,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.054242+0000","last_change":"2024-09-11T03:50:10.647201+0000","last_active":"2024-09-11T03:50:12.054242+0000","last_peered":"2024-09-11T03:50:12.054242+0000","last_clean":"2024-09-11T03:50:12.054242+0000","last_became_active":"2024-09-11T03:49:29.155740+0000","last_became_peered":"2024-09-11T03:49:29.155740+0000","last_unstale":"2024-09-11T03:50:12.054242+0000","last_undegraded":"2024-09-11T03:50:12.054242+0000","last_fullsized":"2024-09-11T03:50:12.054242+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'612","last_scrub_stamp":"2024-09-11T03:49:34.554335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.554335+0000","objects_scrubbed":4,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:55.418115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001042664,"stat_sum":{"num_bytes":12434224,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":3941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,7],"acting":[6,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"2.23","version":"787'464","reported_seq":3970,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.119909+0000","last_change":"2024-09-11T03:50:10.110786+0000","last_active":"2024-09-11T03:50:11.119909+0000","last_peered":"2024-09-11T03:50:11.119909+0000","last_clean":"2024-09-11T03:50:11.119909+0000","last_became_active":"2024-09-11T03:46:32.970100+0000","last_became_peered":"2024-09-11T03:46:32.970100+0000","last_unstale":"2024-09-11T03:50:11.119909+0000","last_undegraded":"2024-09-11T03:50:11.119909+0000","last_fullsized":"2024-09-11T03:50:11.119909+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"711'414","last_scrub_stamp":"2024-09-11T03:48:57.105274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:57.105274+0000","objects_scrubbed":2,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:16.257517+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000668938,"stat_sum":{"num_bytes":9866336,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":44928,"num_write":23,"num_write_kb":10047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"2.22","version":"531'216","reported_seq":3715,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.054002+0000","last_change":"2024-09-11T03:50:10.646815+0000","last_active":"2024-09-11T03:50:12.054002+0000","last_peered":"2024-09-11T03:50:12.054002+0000","last_clean":"2024-09-11T03:50:12.054002+0000","last_became_active":"2024-09-11T03:46:32.970633+0000","last_became_peered":"2024-09-11T03:46:32.970633+0000","last_unstale":"2024-09-11T03:50:12.054002+0000","last_undegraded":"2024-09-11T03:50:12.054002+0000","last_fullsized":"2024-09-11T03:50:12.054002+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:49:18.406097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:18.406097+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:24.100197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00086494600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"2.21","version":"787'331","reported_seq":3924,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.039607+0000","last_change":"2024-09-11T03:50:10.128954+0000","last_active":"2024-09-11T03:50:12.039607+0000","last_peered":"2024-09-11T03:50:12.039607+0000","last_clean":"2024-09-11T03:50:12.039607+0000","last_became_active":"2024-09-11T03:46:36.024928+0000","last_became_peered":"2024-09-11T03:46:36.024928+0000","last_unstale":"2024-09-11T03:50:12.039607+0000","last_undegraded":"2024-09-11T03:50:12.039607+0000","last_fullsized":"2024-09-11T03:50:12.039607+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":6,"last_scrub":"733'289","last_scrub_stamp":"2024-09-11T03:49:18.917801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:18.917801+0000","objects_scrubbed":3,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:35.721209+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051374299999999999,"stat_sum":{"num_bytes":13114734,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":27705,"num_write":31,"num_write_kb":15311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,8],"acting":[11,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"2.20","version":"789'280","reported_seq":3940,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.130406+0000","last_change":"2024-09-11T03:50:10.112274+0000","last_active":"2024-09-11T03:50:11.130406+0000","last_peered":"2024-09-11T03:50:11.130406+0000","last_clean":"2024-09-11T03:50:11.130406+0000","last_became_active":"2024-09-11T03:46:32.974375+0000","last_became_peered":"2024-09-11T03:46:32.974375+0000","last_unstale":"2024-09-11T03:50:11.130406+0000","last_undegraded":"2024-09-11T03:50:11.130406+0000","last_fullsized":"2024-09-11T03:50:11.130406+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"718'223","last_scrub_stamp":"2024-09-11T03:49:00.376119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.376119+0000","objects_scrubbed":2,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:22.669413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0013782009999999999,"stat_sum":{"num_bytes":11791232,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":87,"num_read_kb":134813,"num_write":17,"num_write_kb":9596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"2.1c","version":"785'774","reported_seq":4272,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.056758+0000","last_change":"2024-09-11T03:50:10.615281+0000","last_active":"2024-09-11T03:50:12.056758+0000","last_peered":"2024-09-11T03:50:12.056758+0000","last_clean":"2024-09-11T03:50:12.056758+0000","last_became_active":"2024-09-11T03:46:32.971036+0000","last_became_peered":"2024-09-11T03:46:32.971036+0000","last_unstale":"2024-09-11T03:50:12.056758+0000","last_undegraded":"2024-09-11T03:50:12.056758+0000","last_fullsized":"2024-09-11T03:50:12.056758+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"754'760","last_scrub_stamp":"2024-09-11T03:49:33.579794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.579794+0000","objects_scrubbed":5,"log_size":774,"log_dups_size":0,"ondisk_log_size":774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:58.835203+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00095236599999999995,"stat_sum":{"num_bytes":8735071,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":36,"num_read_kb":35079,"num_write":36,"num_write_kb":13499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":637,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-09-11T03:50:10.936658+0000","last_change":"2024-09-11T03:47:26.195120+0000","last_active":"2024-09-11T03:50:10.936658+0000","last_peered":"2024-09-11T03:50:10.936658+0000","last_clean":"2024-09-11T03:50:10.936658+0000","last_became_active":"2024-09-11T03:46:15.974786+0000","last_became_peered":"2024-09-11T03:46:15.974786+0000","last_unstale":"2024-09-11T03:50:10.936658+0000","last_undegraded":"2024-09-11T03:50:10.936658+0000","last_fullsized":"2024-09-11T03:50:10.936658+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:26.195070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:26.195070+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:53.091756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.f","version":"557'465","reported_seq":4000,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.136870+0000","last_change":"2024-09-11T03:50:10.112533+0000","last_active":"2024-09-11T03:50:11.136870+0000","last_peered":"2024-09-11T03:50:11.136870+0000","last_clean":"2024-09-11T03:50:11.136870+0000","last_became_active":"2024-09-11T03:50:02.055146+0000","last_became_peered":"2024-09-11T03:50:02.055146+0000","last_unstale":"2024-09-11T03:50:11.136870+0000","last_undegraded":"2024-09-11T03:50:11.136870+0000","last_fullsized":"2024-09-11T03:50:11.136870+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:50:02.368307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.368307+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:11.646789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001205539,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"2.e","version":"782'1025","reported_seq":4496,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-09-11T03:50:08.080810+0000","last_change":"2024-09-11T03:50:05.067364+0000","last_active":"2024-09-11T03:50:08.080810+0000","last_peered":"2024-09-11T03:50:08.080810+0000","last_clean":"2024-09-11T03:50:08.080810+0000","last_became_active":"2024-09-11T03:50:02.051717+0000","last_became_peered":"2024-09-11T03:50:02.051717+0000","last_unstale":"2024-09-11T03:50:08.080810+0000","last_undegraded":"2024-09-11T03:50:08.080810+0000","last_fullsized":"2024-09-11T03:50:08.080810+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":0,"last_scrub":"782'1025","last_scrub_stamp":"2024-09-11T03:50:04.968160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.968160+0000","objects_scrubbed":0,"log_size":1025,"log_dups_size":0,"ondisk_log_size":1025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:23.605994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037207339999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":1216,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.130209+0000","last_change":"2024-09-11T03:49:20.176154+0000","last_active":"2024-09-11T03:50:11.130209+0000","last_peered":"2024-09-11T03:50:11.130209+0000","last_clean":"2024-09-11T03:50:11.130209+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:50:11.130209+0000","last_undegraded":"2024-09-11T03:50:11.130209+0000","last_fullsized":"2024-09-11T03:50:11.130209+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:20.176098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.176098+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:41.602355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"787'1009","reported_seq":4573,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.169147+0000","last_change":"2024-09-11T03:50:10.122771+0000","last_active":"2024-09-11T03:50:11.169147+0000","last_peered":"2024-09-11T03:50:11.169147+0000","last_clean":"2024-09-11T03:50:11.169147+0000","last_became_active":"2024-09-11T03:49:24.116744+0000","last_became_peered":"2024-09-11T03:49:24.116744+0000","last_unstale":"2024-09-11T03:50:11.169147+0000","last_undegraded":"2024-09-11T03:50:11.169147+0000","last_fullsized":"2024-09-11T03:50:11.169147+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'987","last_scrub_stamp":"2024-09-11T03:49:32.653263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.653263+0000","objects_scrubbed":3,"log_size":1009,"log_dups_size":0,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:01.963732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0021470640000000002,"stat_sum":{"num_bytes":8757074,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":931,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.130435+0000","last_change":"2024-09-11T03:49:22.192021+0000","last_active":"2024-09-11T03:50:11.130435+0000","last_peered":"2024-09-11T03:50:11.130435+0000","last_clean":"2024-09-11T03:50:11.130435+0000","last_became_active":"2024-09-11T03:44:43.940022+0000","last_became_peered":"2024-09-11T03:44:43.940022+0000","last_unstale":"2024-09-11T03:50:11.130435+0000","last_undegraded":"2024-09-11T03:50:11.130435+0000","last_fullsized":"2024-09-11T03:50:11.130435+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:22.191965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:22.191965+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:33.940974+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"790'831","reported_seq":4363,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.039882+0000","last_change":"2024-09-11T03:50:10.129883+0000","last_active":"2024-09-11T03:50:12.039882+0000","last_peered":"2024-09-11T03:50:12.039882+0000","last_clean":"2024-09-11T03:50:12.039882+0000","last_became_active":"2024-09-11T03:49:24.115405+0000","last_became_peered":"2024-09-11T03:49:24.115405+0000","last_unstale":"2024-09-11T03:50:12.039882+0000","last_undegraded":"2024-09-11T03:50:12.039882+0000","last_fullsized":"2024-09-11T03:50:12.039882+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'808","last_scrub_stamp":"2024-09-11T03:49:33.953477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.953477+0000","objects_scrubbed":7,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:33.965722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00093345400000000001,"stat_sum":{"num_bytes":18373538,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":3,"num_read_kb":1,"num_write":5,"num_write_kb":4797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"1.f","version":"20'92","reported_seq":498,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.054656+0000","last_change":"2024-09-11T03:48:57.363099+0000","last_active":"2024-09-11T03:50:12.054656+0000","last_peered":"2024-09-11T03:50:12.054656+0000","last_clean":"2024-09-11T03:50:12.054656+0000","last_became_active":"2024-09-11T03:46:15.973796+0000","last_became_peered":"2024-09-11T03:46:15.973796+0000","last_unstale":"2024-09-11T03:50:12.054656+0000","last_undegraded":"2024-09-11T03:50:12.054656+0000","last_fullsized":"2024-09-11T03:50:12.054656+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:57.363027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:57.363027+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:10.644654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":1622,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.130561+0000","last_change":"2024-09-11T03:47:28.495899+0000","last_active":"2024-09-11T03:50:11.130561+0000","last_peered":"2024-09-11T03:50:11.130561+0000","last_clean":"2024-09-11T03:50:11.130561+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:50:11.130561+0000","last_undegraded":"2024-09-11T03:50:11.130561+0000","last_fullsized":"2024-09-11T03:50:11.130561+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:28.495839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:28.495839+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.015311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"792'513","reported_seq":4032,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.168966+0000","last_change":"2024-09-11T03:50:10.122408+0000","last_active":"2024-09-11T03:50:11.168966+0000","last_peered":"2024-09-11T03:50:11.168966+0000","last_clean":"2024-09-11T03:50:11.168966+0000","last_became_active":"2024-09-11T03:46:27.972329+0000","last_became_peered":"2024-09-11T03:46:27.972329+0000","last_unstale":"2024-09-11T03:50:11.168966+0000","last_undegraded":"2024-09-11T03:50:11.168966+0000","last_fullsized":"2024-09-11T03:50:11.168966+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'480","last_scrub_stamp":"2024-09-11T03:49:37.694768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.694768+0000","objects_scrubbed":5,"log_size":513,"log_dups_size":0,"ondisk_log_size":513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:05.230263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0021806429999999999,"stat_sum":{"num_bytes":10724464,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":27,"num_read_kb":23186,"num_write":35,"num_write_kb":7258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":1224,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.120019+0000","last_change":"2024-09-11T03:47:25.344219+0000","last_active":"2024-09-11T03:50:11.120019+0000","last_peered":"2024-09-11T03:50:11.120019+0000","last_clean":"2024-09-11T03:50:11.120019+0000","last_became_active":"2024-09-11T03:46:15.968960+0000","last_became_peered":"2024-09-11T03:46:15.968960+0000","last_unstale":"2024-09-11T03:50:11.120019+0000","last_undegraded":"2024-09-11T03:50:11.120019+0000","last_fullsized":"2024-09-11T03:50:11.120019+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:25.344170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:25.344170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:42.179046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"563'192","reported_seq":3777,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.130383+0000","last_change":"2024-09-11T03:50:10.113211+0000","last_active":"2024-09-11T03:50:11.130383+0000","last_peered":"2024-09-11T03:50:11.130383+0000","last_clean":"2024-09-11T03:50:11.130383+0000","last_became_active":"2024-09-11T03:46:27.974405+0000","last_became_peered":"2024-09-11T03:46:27.974405+0000","last_unstale":"2024-09-11T03:50:11.130383+0000","last_undegraded":"2024-09-11T03:50:11.130383+0000","last_fullsized":"2024-09-11T03:50:11.130383+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:49:15.233165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:15.233165+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:24.894590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001335904,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":1183,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-09-11T03:50:10.936434+0000","last_change":"2024-09-11T03:49:17.252987+0000","last_active":"2024-09-11T03:50:10.936434+0000","last_peered":"2024-09-11T03:50:10.936434+0000","last_clean":"2024-09-11T03:50:10.936434+0000","last_became_active":"2024-09-11T03:44:25.934580+0000","last_became_peered":"2024-09-11T03:44:25.934580+0000","last_unstale":"2024-09-11T03:50:10.936434+0000","last_undegraded":"2024-09-11T03:50:10.936434+0000","last_fullsized":"2024-09-11T03:50:10.936434+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:17.252916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:17.252916+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:39.008724+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"567'235","reported_seq":3706,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-09-11T03:50:10.937228+0000","last_change":"2024-09-11T03:50:10.937431+0000","last_active":"2024-09-11T03:50:10.937228+0000","last_peered":"2024-09-11T03:50:10.937228+0000","last_clean":"2024-09-11T03:50:10.937228+0000","last_became_active":"2024-09-11T03:46:36.024925+0000","last_became_peered":"2024-09-11T03:46:36.024925+0000","last_unstale":"2024-09-11T03:50:10.937228+0000","last_undegraded":"2024-09-11T03:50:10.937228+0000","last_fullsized":"2024-09-11T03:50:10.937228+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:49:01.138426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:01.138426+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:28.509829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00057616199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":793,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.056577+0000","last_change":"2024-09-11T03:48:32.199588+0000","last_active":"2024-09-11T03:50:12.056577+0000","last_peered":"2024-09-11T03:50:12.056577+0000","last_clean":"2024-09-11T03:50:12.056577+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:50:12.056577+0000","last_undegraded":"2024-09-11T03:50:12.056577+0000","last_fullsized":"2024-09-11T03:50:12.056577+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:32.199524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:32.199524+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"790'297","reported_seq":3885,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.168763+0000","last_change":"2024-09-11T03:50:10.122703+0000","last_active":"2024-09-11T03:50:11.168763+0000","last_peered":"2024-09-11T03:50:11.168763+0000","last_clean":"2024-09-11T03:50:11.168763+0000","last_became_active":"2024-09-11T03:46:27.971649+0000","last_became_peered":"2024-09-11T03:46:27.971649+0000","last_unstale":"2024-09-11T03:50:11.168763+0000","last_undegraded":"2024-09-11T03:50:11.168763+0000","last_fullsized":"2024-09-11T03:50:11.168763+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"718'254","last_scrub_stamp":"2024-09-11T03:49:15.510323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:15.510323+0000","objects_scrubbed":4,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:44.831436+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002289742,"stat_sum":{"num_bytes":4723229,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":24,"num_read_kb":13035,"num_write":25,"num_write_kb":12701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":490,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.054330+0000","last_change":"2024-09-11T03:49:37.637065+0000","last_active":"2024-09-11T03:50:12.054330+0000","last_peered":"2024-09-11T03:50:12.054330+0000","last_clean":"2024-09-11T03:50:12.054330+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:50:12.054330+0000","last_undegraded":"2024-09-11T03:50:12.054330+0000","last_fullsized":"2024-09-11T03:50:12.054330+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:37.637018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.637018+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:05.221389+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"790'648","reported_seq":4220,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.120215+0000","last_change":"2024-09-11T03:50:10.110833+0000","last_active":"2024-09-11T03:50:11.120215+0000","last_peered":"2024-09-11T03:50:11.120215+0000","last_clean":"2024-09-11T03:50:11.120215+0000","last_became_active":"2024-09-11T03:49:24.114330+0000","last_became_peered":"2024-09-11T03:49:24.114330+0000","last_unstale":"2024-09-11T03:50:11.120215+0000","last_undegraded":"2024-09-11T03:50:11.120215+0000","last_fullsized":"2024-09-11T03:50:11.120215+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'620","last_scrub_stamp":"2024-09-11T03:49:34.288083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.288083+0000","objects_scrubbed":8,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:04.051432+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00062406099999999997,"stat_sum":{"num_bytes":17289962,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":10,"num_read_kb":8618,"num_write":2,"num_write_kb":4860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":803,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.056406+0000","last_change":"2024-09-11T03:49:17.858224+0000","last_active":"2024-09-11T03:50:12.056406+0000","last_peered":"2024-09-11T03:50:12.056406+0000","last_clean":"2024-09-11T03:50:12.056406+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:50:12.056406+0000","last_undegraded":"2024-09-11T03:50:12.056406+0000","last_fullsized":"2024-09-11T03:50:12.056406+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:17.858165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:17.858165+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:40.674663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"787'1163","reported_seq":4740,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.136576+0000","last_change":"2024-09-11T03:50:10.112536+0000","last_active":"2024-09-11T03:50:11.136576+0000","last_peered":"2024-09-11T03:50:11.136576+0000","last_clean":"2024-09-11T03:50:11.136576+0000","last_became_active":"2024-09-11T03:49:24.115095+0000","last_became_peered":"2024-09-11T03:49:24.115095+0000","last_unstale":"2024-09-11T03:50:11.136576+0000","last_undegraded":"2024-09-11T03:50:11.136576+0000","last_fullsized":"2024-09-11T03:50:11.136576+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'1112","last_scrub_stamp":"2024-09-11T03:49:35.188307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.188307+0000","objects_scrubbed":7,"log_size":1163,"log_dups_size":0,"ondisk_log_size":1163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:35.621935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001110301,"stat_sum":{"num_bytes":16331222,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":8,"num_write_kb":5553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":803,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.057146+0000","last_change":"2024-09-11T03:49:36.654074+0000","last_active":"2024-09-11T03:50:12.057146+0000","last_peered":"2024-09-11T03:50:12.057146+0000","last_clean":"2024-09-11T03:50:12.057146+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:50:12.057146+0000","last_undegraded":"2024-09-11T03:50:12.057146+0000","last_fullsized":"2024-09-11T03:50:12.057146+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:36.654025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.654025+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:06.216209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":3667,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.120010+0000","last_change":"2024-09-11T03:50:10.111336+0000","last_active":"2024-09-11T03:50:11.120010+0000","last_peered":"2024-09-11T03:50:11.120010+0000","last_clean":"2024-09-11T03:50:11.120010+0000","last_became_active":"2024-09-11T03:49:24.113518+0000","last_became_peered":"2024-09-11T03:49:24.113518+0000","last_unstale":"2024-09-11T03:50:11.120010+0000","last_undegraded":"2024-09-11T03:50:11.120010+0000","last_fullsized":"2024-09-11T03:50:11.120010+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:49:35.267227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.267227+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:53.961658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068758899999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":1403,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.039630+0000","last_change":"2024-09-11T03:49:21.909643+0000","last_active":"2024-09-11T03:50:12.039630+0000","last_peered":"2024-09-11T03:50:12.039630+0000","last_clean":"2024-09-11T03:50:12.039630+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:50:12.039630+0000","last_undegraded":"2024-09-11T03:50:12.039630+0000","last_fullsized":"2024-09-11T03:50:12.039630+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:21.909594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:21.909594+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:22.293071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"794'1143","reported_seq":4819,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.114693+0000","last_change":"2024-09-11T03:50:10.647548+0000","last_active":"2024-09-11T03:50:12.114693+0000","last_peered":"2024-09-11T03:50:12.114693+0000","last_clean":"2024-09-11T03:50:12.114693+0000","last_became_active":"2024-09-11T03:49:24.121333+0000","last_became_peered":"2024-09-11T03:49:24.121333+0000","last_unstale":"2024-09-11T03:50:12.114693+0000","last_undegraded":"2024-09-11T03:50:12.114693+0000","last_fullsized":"2024-09-11T03:50:12.114693+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'1066","last_scrub_stamp":"2024-09-11T03:49:35.600201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.600201+0000","objects_scrubbed":7,"log_size":1143,"log_dups_size":0,"ondisk_log_size":1143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:35.662020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012916049999999999,"stat_sum":{"num_bytes":29729677,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":57,"num_read_kb":59189,"num_write":31,"num_write_kb":18135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":1198,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.039704+0000","last_change":"2024-09-11T03:40:39.282700+0000","last_active":"2024-09-11T03:50:12.039704+0000","last_peered":"2024-09-11T03:50:12.039704+0000","last_clean":"2024-09-11T03:50:12.039704+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:50:12.039704+0000","last_undegraded":"2024-09-11T03:50:12.039704+0000","last_fullsized":"2024-09-11T03:50:12.039704+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:40:39.282648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:40:39.282648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:13.961107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"744'374","reported_seq":3929,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.130202+0000","last_change":"2024-09-11T03:50:10.112929+0000","last_active":"2024-09-11T03:50:11.130202+0000","last_peered":"2024-09-11T03:50:11.130202+0000","last_clean":"2024-09-11T03:50:11.130202+0000","last_became_active":"2024-09-11T03:49:24.115150+0000","last_became_peered":"2024-09-11T03:49:24.115150+0000","last_unstale":"2024-09-11T03:50:11.130202+0000","last_undegraded":"2024-09-11T03:50:11.130202+0000","last_fullsized":"2024-09-11T03:50:11.130202+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'374","last_scrub_stamp":"2024-09-11T03:49:35.190806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.190806+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:51.267806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001280996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":1540,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-09-11T03:50:08.080235+0000","last_change":"2024-09-11T03:48:59.762212+0000","last_active":"2024-09-11T03:50:08.080235+0000","last_peered":"2024-09-11T03:50:08.080235+0000","last_clean":"2024-09-11T03:50:08.080235+0000","last_became_active":"2024-09-11T03:46:15.973271+0000","last_became_peered":"2024-09-11T03:46:15.973271+0000","last_unstale":"2024-09-11T03:50:08.080235+0000","last_undegraded":"2024-09-11T03:50:08.080235+0000","last_fullsized":"2024-09-11T03:50:08.080235+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:59.762145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:59.762145+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"790'1258","reported_seq":4885,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.120115+0000","last_change":"2024-09-11T03:50:10.110520+0000","last_active":"2024-09-11T03:50:11.120115+0000","last_peered":"2024-09-11T03:50:11.120115+0000","last_clean":"2024-09-11T03:50:11.120115+0000","last_became_active":"2024-09-11T03:49:24.114865+0000","last_became_peered":"2024-09-11T03:49:24.114865+0000","last_unstale":"2024-09-11T03:50:11.120115+0000","last_undegraded":"2024-09-11T03:50:11.120115+0000","last_fullsized":"2024-09-11T03:50:11.120115+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'1171","last_scrub_stamp":"2024-09-11T03:49:33.292043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.292043+0000","objects_scrubbed":10,"log_size":1258,"log_dups_size":0,"ondisk_log_size":1258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:53.124733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00045085500000000001,"stat_sum":{"num_bytes":25784110,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":15,"num_read_kb":15594,"num_write":19,"num_write_kb":7361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":1617,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.129881+0000","last_change":"2024-09-11T03:49:37.206392+0000","last_active":"2024-09-11T03:50:11.129881+0000","last_peered":"2024-09-11T03:50:11.129881+0000","last_clean":"2024-09-11T03:50:11.129881+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:50:11.129881+0000","last_undegraded":"2024-09-11T03:50:11.129881+0000","last_fullsized":"2024-09-11T03:50:11.129881+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:37.206321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.206321+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:53.641968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"787'370","reported_seq":3927,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.040365+0000","last_change":"2024-09-11T03:50:10.129728+0000","last_active":"2024-09-11T03:50:12.040365+0000","last_peered":"2024-09-11T03:50:12.040365+0000","last_clean":"2024-09-11T03:50:12.040365+0000","last_became_active":"2024-09-11T03:49:24.114551+0000","last_became_peered":"2024-09-11T03:49:24.114551+0000","last_unstale":"2024-09-11T03:50:12.040365+0000","last_undegraded":"2024-09-11T03:50:12.040365+0000","last_fullsized":"2024-09-11T03:50:12.040365+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:49:32.911631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.911631+0000","objects_scrubbed":4,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:54.661712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011661169999999999,"stat_sum":{"num_bytes":5429057,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":10,"num_read_kb":12240,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":1276,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.168701+0000","last_change":"2024-09-11T03:49:34.621588+0000","last_active":"2024-09-11T03:50:11.168701+0000","last_peered":"2024-09-11T03:50:11.168701+0000","last_clean":"2024-09-11T03:50:11.168701+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:50:11.168701+0000","last_undegraded":"2024-09-11T03:50:11.168701+0000","last_fullsized":"2024-09-11T03:50:11.168701+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:34.621527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.621527+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.563733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"568'646","reported_seq":4188,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.136032+0000","last_change":"2024-09-11T03:50:10.112912+0000","last_active":"2024-09-11T03:50:11.136032+0000","last_peered":"2024-09-11T03:50:11.136032+0000","last_clean":"2024-09-11T03:50:11.136032+0000","last_became_active":"2024-09-11T03:49:24.113147+0000","last_became_peered":"2024-09-11T03:49:24.113147+0000","last_unstale":"2024-09-11T03:50:11.136032+0000","last_undegraded":"2024-09-11T03:50:11.136032+0000","last_fullsized":"2024-09-11T03:50:11.136032+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:49:33.204497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.204497+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:36.793494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014637949999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":1226,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.168646+0000","last_change":"2024-09-11T03:49:00.710015+0000","last_active":"2024-09-11T03:50:11.168646+0000","last_peered":"2024-09-11T03:50:11.168646+0000","last_clean":"2024-09-11T03:50:11.168646+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:50:11.168646+0000","last_undegraded":"2024-09-11T03:50:11.168646+0000","last_fullsized":"2024-09-11T03:50:11.168646+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:00.709966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.709966+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:30.175623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":3582,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-09-11T03:50:08.080418+0000","last_change":"2024-09-11T03:50:05.067346+0000","last_active":"2024-09-11T03:50:08.080418+0000","last_peered":"2024-09-11T03:50:08.080418+0000","last_clean":"2024-09-11T03:50:08.080418+0000","last_became_active":"2024-09-11T03:50:02.046906+0000","last_became_peered":"2024-09-11T03:50:02.046906+0000","last_unstale":"2024-09-11T03:50:08.080418+0000","last_undegraded":"2024-09-11T03:50:08.080418+0000","last_fullsized":"2024-09-11T03:50:08.080418+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:50:02.987179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.987179+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:31.616973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310743,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":1209,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-09-11T03:50:10.936655+0000","last_change":"2024-09-11T03:49:41.224105+0000","last_active":"2024-09-11T03:50:10.936655+0000","last_peered":"2024-09-11T03:50:10.936655+0000","last_clean":"2024-09-11T03:50:10.936655+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:50:10.936655+0000","last_undegraded":"2024-09-11T03:50:10.936655+0000","last_fullsized":"2024-09-11T03:50:10.936655+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:41.224059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:41.224059+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:58.932206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"787'555","reported_seq":4093,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.056573+0000","last_change":"2024-09-11T03:50:10.615222+0000","last_active":"2024-09-11T03:50:12.056573+0000","last_peered":"2024-09-11T03:50:12.056573+0000","last_clean":"2024-09-11T03:50:12.056573+0000","last_became_active":"2024-09-11T03:50:02.053513+0000","last_became_peered":"2024-09-11T03:50:02.053513+0000","last_unstale":"2024-09-11T03:50:12.056573+0000","last_undegraded":"2024-09-11T03:50:12.056573+0000","last_fullsized":"2024-09-11T03:50:12.056573+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"784'551","last_scrub_stamp":"2024-09-11T03:50:02.647690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.647690+0000","objects_scrubbed":4,"log_size":555,"log_dups_size":0,"ondisk_log_size":555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:28.183008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00084574900000000002,"stat_sum":{"num_bytes":6367447,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":1207,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.168622+0000","last_change":"2024-09-11T03:47:23.339646+0000","last_active":"2024-09-11T03:50:11.168622+0000","last_peered":"2024-09-11T03:50:11.168622+0000","last_clean":"2024-09-11T03:50:11.168622+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:50:11.168622+0000","last_undegraded":"2024-09-11T03:50:11.168622+0000","last_fullsized":"2024-09-11T03:50:11.168622+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:23.339595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:23.339595+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:50.634356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":2670,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.054356+0000","last_change":"2024-09-11T03:50:10.647934+0000","last_active":"2024-09-11T03:50:12.054356+0000","last_peered":"2024-09-11T03:50:12.054356+0000","last_clean":"2024-09-11T03:50:12.054356+0000","last_became_active":"2024-09-11T03:50:02.048777+0000","last_became_peered":"2024-09-11T03:50:02.048777+0000","last_unstale":"2024-09-11T03:50:12.054356+0000","last_undegraded":"2024-09-11T03:50:12.054356+0000","last_fullsized":"2024-09-11T03:50:12.054356+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:50:02.696833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.696833+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:30.083227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013219799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"1.13","version":"20'92","reported_seq":1311,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.119918+0000","last_change":"2024-09-11T03:47:18.217600+0000","last_active":"2024-09-11T03:50:11.119918+0000","last_peered":"2024-09-11T03:50:11.119918+0000","last_clean":"2024-09-11T03:50:11.119918+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:50:11.119918+0000","last_undegraded":"2024-09-11T03:50:11.119918+0000","last_fullsized":"2024-09-11T03:50:11.119918+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:18.217555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:18.217555+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:43.139528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":2611,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.135919+0000","last_change":"2024-09-11T03:50:10.111816+0000","last_active":"2024-09-11T03:50:11.135919+0000","last_peered":"2024-09-11T03:50:11.135919+0000","last_clean":"2024-09-11T03:50:11.135919+0000","last_became_active":"2024-09-11T03:50:02.057196+0000","last_became_peered":"2024-09-11T03:50:02.057196+0000","last_unstale":"2024-09-11T03:50:11.135919+0000","last_undegraded":"2024-09-11T03:50:11.135919+0000","last_fullsized":"2024-09-11T03:50:11.135919+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:50:03.345563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:03.345563+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:30.506542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093920099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":785,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.130147+0000","last_change":"2024-09-11T03:48:57.320199+0000","last_active":"2024-09-11T03:50:11.130147+0000","last_peered":"2024-09-11T03:50:11.130147+0000","last_clean":"2024-09-11T03:50:11.130147+0000","last_became_active":"2024-09-11T03:46:36.026459+0000","last_became_peered":"2024-09-11T03:46:36.026459+0000","last_unstale":"2024-09-11T03:50:11.130147+0000","last_undegraded":"2024-09-11T03:50:11.130147+0000","last_fullsized":"2024-09-11T03:50:11.130147+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:57.320159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:57.320159+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:16.384062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":3166,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.054659+0000","last_change":"2024-09-11T03:50:10.647819+0000","last_active":"2024-09-11T03:50:12.054659+0000","last_peered":"2024-09-11T03:50:12.054659+0000","last_clean":"2024-09-11T03:50:12.054659+0000","last_became_active":"2024-09-11T03:50:02.052278+0000","last_became_peered":"2024-09-11T03:50:02.052278+0000","last_unstale":"2024-09-11T03:50:12.054659+0000","last_undegraded":"2024-09-11T03:50:12.054659+0000","last_fullsized":"2024-09-11T03:50:12.054659+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:50:03.690374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:03.690374+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:18.850321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001345696,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":1179,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.130177+0000","last_change":"2024-09-11T03:47:47.459856+0000","last_active":"2024-09-11T03:50:11.130177+0000","last_peered":"2024-09-11T03:50:11.130177+0000","last_clean":"2024-09-11T03:50:11.130177+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:50:11.130177+0000","last_undegraded":"2024-09-11T03:50:11.130177+0000","last_fullsized":"2024-09-11T03:50:11.130177+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:47.459803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:47.459803+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"782'561","reported_seq":4138,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.169015+0000","last_change":"2024-09-11T03:50:10.123199+0000","last_active":"2024-09-11T03:50:11.169015+0000","last_peered":"2024-09-11T03:50:11.169015+0000","last_clean":"2024-09-11T03:50:11.169015+0000","last_became_active":"2024-09-11T03:50:02.054667+0000","last_became_peered":"2024-09-11T03:50:02.054667+0000","last_unstale":"2024-09-11T03:50:11.169015+0000","last_undegraded":"2024-09-11T03:50:11.169015+0000","last_fullsized":"2024-09-11T03:50:11.169015+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"782'561","last_scrub_stamp":"2024-09-11T03:50:02.696931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.696931+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:09.049132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024404040000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":662,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.040351+0000","last_change":"2024-09-11T03:49:20.949194+0000","last_active":"2024-09-11T03:50:12.040351+0000","last_peered":"2024-09-11T03:50:12.040351+0000","last_clean":"2024-09-11T03:50:12.040351+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:50:12.040351+0000","last_undegraded":"2024-09-11T03:50:12.040351+0000","last_fullsized":"2024-09-11T03:50:12.040351+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:20.949118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.949118+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:36.681631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":3802,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-09-11T03:50:08.080643+0000","last_change":"2024-09-11T03:50:05.064450+0000","last_active":"2024-09-11T03:50:08.080643+0000","last_peered":"2024-09-11T03:50:08.080643+0000","last_clean":"2024-09-11T03:50:08.080643+0000","last_became_active":"2024-09-11T03:50:02.051850+0000","last_became_peered":"2024-09-11T03:50:02.051850+0000","last_unstale":"2024-09-11T03:50:08.080643+0000","last_undegraded":"2024-09-11T03:50:08.080643+0000","last_fullsized":"2024-09-11T03:50:08.080643+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:50:04.004006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.004006+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:19.961856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012676409999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":1198,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.136736+0000","last_change":"2024-09-11T03:47:29.086383+0000","last_active":"2024-09-11T03:50:11.136736+0000","last_peered":"2024-09-11T03:50:11.136736+0000","last_clean":"2024-09-11T03:50:11.136736+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:50:11.136736+0000","last_undegraded":"2024-09-11T03:50:11.136736+0000","last_fullsized":"2024-09-11T03:50:11.136736+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:29.086335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:29.086335+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:43.171032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"778'353","reported_seq":3800,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.054341+0000","last_change":"2024-09-11T03:50:10.648092+0000","last_active":"2024-09-11T03:50:12.054341+0000","last_peered":"2024-09-11T03:50:12.054341+0000","last_clean":"2024-09-11T03:50:12.054341+0000","last_became_active":"2024-09-11T03:50:02.054211+0000","last_became_peered":"2024-09-11T03:50:02.054211+0000","last_unstale":"2024-09-11T03:50:12.054341+0000","last_undegraded":"2024-09-11T03:50:12.054341+0000","last_fullsized":"2024-09-11T03:50:12.054341+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"778'353","last_scrub_stamp":"2024-09-11T03:50:04.646331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.646331+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:08.472636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014285019999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"1.18","version":"20'92","reported_seq":1318,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.168572+0000","last_change":"2024-09-11T03:49:56.630710+0000","last_active":"2024-09-11T03:50:11.168572+0000","last_peered":"2024-09-11T03:50:11.168572+0000","last_clean":"2024-09-11T03:50:11.168572+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:50:11.168572+0000","last_undegraded":"2024-09-11T03:50:11.168572+0000","last_fullsized":"2024-09-11T03:50:11.168572+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:56.630651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:56.630651+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:18.735201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"794'851","reported_seq":4491,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.039889+0000","last_change":"2024-09-11T03:50:10.129991+0000","last_active":"2024-09-11T03:50:12.039889+0000","last_peered":"2024-09-11T03:50:12.039889+0000","last_clean":"2024-09-11T03:50:12.039889+0000","last_became_active":"2024-09-11T03:46:32.975567+0000","last_became_peered":"2024-09-11T03:46:32.975567+0000","last_unstale":"2024-09-11T03:50:12.039889+0000","last_undegraded":"2024-09-11T03:50:12.039889+0000","last_fullsized":"2024-09-11T03:50:12.039889+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"754'773","last_scrub_stamp":"2024-09-11T03:49:34.919322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.919322+0000","objects_scrubbed":8,"log_size":851,"log_dups_size":0,"ondisk_log_size":851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:51.427918+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011630169999999999,"stat_sum":{"num_bytes":24902116,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":114,"num_read_kb":123717,"num_write":63,"num_write_kb":26038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7],"acting":[11,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"1.19","version":"20'92","reported_seq":807,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.056465+0000","last_change":"2024-09-11T03:48:56.129575+0000","last_active":"2024-09-11T03:50:12.056465+0000","last_peered":"2024-09-11T03:50:12.056465+0000","last_clean":"2024-09-11T03:50:12.056465+0000","last_became_active":"2024-09-11T03:46:15.972841+0000","last_became_peered":"2024-09-11T03:46:15.972841+0000","last_unstale":"2024-09-11T03:50:12.056465+0000","last_undegraded":"2024-09-11T03:50:12.056465+0000","last_fullsized":"2024-09-11T03:50:12.056465+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:56.129516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:56.129516+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:24.133914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"557'273","reported_seq":3727,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.054163+0000","last_change":"2024-09-11T03:50:10.647125+0000","last_active":"2024-09-11T03:50:12.054163+0000","last_peered":"2024-09-11T03:50:12.054163+0000","last_clean":"2024-09-11T03:50:12.054163+0000","last_became_active":"2024-09-11T03:46:32.971003+0000","last_became_peered":"2024-09-11T03:46:32.971003+0000","last_unstale":"2024-09-11T03:50:12.054163+0000","last_undegraded":"2024-09-11T03:50:12.054163+0000","last_fullsized":"2024-09-11T03:50:12.054163+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:49:20.466140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.466140+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:41.916709+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00109161,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,2],"acting":[6,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"1.1a","version":"20'92","reported_seq":350,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.130017+0000","last_change":"2024-09-11T03:50:01.843384+0000","last_active":"2024-09-11T03:50:11.130017+0000","last_peered":"2024-09-11T03:50:11.130017+0000","last_clean":"2024-09-11T03:50:11.130017+0000","last_became_active":"2024-09-11T03:49:30.162694+0000","last_became_peered":"2024-09-11T03:49:30.162694+0000","last_unstale":"2024-09-11T03:50:11.130017+0000","last_undegraded":"2024-09-11T03:50:11.130017+0000","last_fullsized":"2024-09-11T03:50:11.130017+0000","mapping_epoch":751,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":752,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:01.843302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:01.843302+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:12.888176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.19","version":"785'575","reported_seq":4172,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.053998+0000","last_change":"2024-09-11T03:50:10.647431+0000","last_active":"2024-09-11T03:50:12.053998+0000","last_peered":"2024-09-11T03:50:12.053998+0000","last_clean":"2024-09-11T03:50:12.053998+0000","last_became_active":"2024-09-11T03:46:19.002002+0000","last_became_peered":"2024-09-11T03:46:19.002002+0000","last_unstale":"2024-09-11T03:50:12.053998+0000","last_undegraded":"2024-09-11T03:50:12.053998+0000","last_fullsized":"2024-09-11T03:50:12.053998+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"736'546","last_scrub_stamp":"2024-09-11T03:49:19.458345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:19.458345+0000","objects_scrubbed":6,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:47.802932+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001214653,"stat_sum":{"num_bytes":11704959,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":103,"num_read_kb":99782,"num_write":67,"num_write_kb":30004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4870845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"1.1b","version":"20'92","reported_seq":1175,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.135883+0000","last_change":"2024-09-11T03:48:31.033896+0000","last_active":"2024-09-11T03:50:11.135883+0000","last_peered":"2024-09-11T03:50:11.135883+0000","last_clean":"2024-09-11T03:50:11.135883+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:50:11.135883+0000","last_undegraded":"2024-09-11T03:50:11.135883+0000","last_fullsized":"2024-09-11T03:50:11.135883+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:31.033837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:31.033837+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:46.181076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"787'540","reported_seq":4121,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-09-11T03:50:08.080026+0000","last_change":"2024-09-11T03:50:05.081453+0000","last_active":"2024-09-11T03:50:08.080026+0000","last_peered":"2024-09-11T03:50:08.080026+0000","last_clean":"2024-09-11T03:50:08.080026+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:50:08.080026+0000","last_undegraded":"2024-09-11T03:50:08.080026+0000","last_fullsized":"2024-09-11T03:50:08.080026+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"783'523","last_scrub_stamp":"2024-09-11T03:50:01.960691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:01.960691+0000","objects_scrubbed":10,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:11.285415+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.017260238000000001,"stat_sum":{"num_bytes":27535253,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":213,"num_read_kb":248887,"num_write":101,"num_write_kb":44446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4865740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"}]},{"pgid":"2.1f","version":"787'725","reported_seq":4185,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-09-11T03:50:10.937172+0000","last_change":"2024-09-11T03:50:10.937409+0000","last_active":"2024-09-11T03:50:10.937172+0000","last_peered":"2024-09-11T03:50:10.937172+0000","last_clean":"2024-09-11T03:50:10.937172+0000","last_became_active":"2024-09-11T03:46:32.968696+0000","last_became_peered":"2024-09-11T03:46:32.968696+0000","last_unstale":"2024-09-11T03:50:10.937172+0000","last_undegraded":"2024-09-11T03:50:10.937172+0000","last_fullsized":"2024-09-11T03:50:10.937172+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"733'613","last_scrub_stamp":"2024-09-11T03:49:20.295589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.295589+0000","objects_scrubbed":10,"log_size":725,"log_dups_size":0,"ondisk_log_size":725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:33.756079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00074054999999999998,"stat_sum":{"num_bytes":41736725,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":132,"num_read_kb":151175,"num_write":85,"num_write_kb":31382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"1.1c","version":"20'92","reported_seq":1182,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.136372+0000","last_change":"2024-09-11T03:49:40.156559+0000","last_active":"2024-09-11T03:50:11.136372+0000","last_peered":"2024-09-11T03:50:11.136372+0000","last_clean":"2024-09-11T03:50:11.136372+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:50:11.136372+0000","last_undegraded":"2024-09-11T03:50:11.136372+0000","last_fullsized":"2024-09-11T03:50:11.136372+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:40.156495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:40.156495+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:04.047069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1e","version":"785'1044","reported_seq":4521,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-09-11T03:50:10.936971+0000","last_change":"2024-09-11T03:50:10.937132+0000","last_active":"2024-09-11T03:50:10.936971+0000","last_peered":"2024-09-11T03:50:10.936971+0000","last_clean":"2024-09-11T03:50:10.936971+0000","last_became_active":"2024-09-11T03:46:32.970637+0000","last_became_peered":"2024-09-11T03:46:32.970637+0000","last_unstale":"2024-09-11T03:50:10.936971+0000","last_undegraded":"2024-09-11T03:50:10.936971+0000","last_fullsized":"2024-09-11T03:50:10.936971+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"785'1044","last_scrub_stamp":"2024-09-11T03:50:03.954442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:03.954442+0000","objects_scrubbed":9,"log_size":1044,"log_dups_size":0,"ondisk_log_size":1044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:17.997608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00050155999999999996,"stat_sum":{"num_bytes":18776531,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":57,"num_read_kb":51945,"num_write":85,"num_write_kb":31670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"133","length":"1"},{"start":"13d","length":"1"},{"start":"150","length":"1"},{"start":"153","length":"2"},{"start":"159","length":"1"},{"start":"15c","length":"1"},{"start":"165","length":"1"}]},{"pgid":"1.1d","version":"20'92","reported_seq":1164,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-09-11T03:50:11.120369+0000","last_change":"2024-09-11T03:47:22.312507+0000","last_active":"2024-09-11T03:50:11.120369+0000","last_peered":"2024-09-11T03:50:11.120369+0000","last_clean":"2024-09-11T03:50:11.120369+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:50:11.120369+0000","last_undegraded":"2024-09-11T03:50:11.120369+0000","last_fullsized":"2024-09-11T03:50:11.120369+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:22.312456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:22.312456+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.239645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"567'752","reported_seq":4220,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.056931+0000","last_change":"2024-09-11T03:50:10.615587+0000","last_active":"2024-09-11T03:50:12.056931+0000","last_peered":"2024-09-11T03:50:12.056931+0000","last_clean":"2024-09-11T03:50:12.056931+0000","last_became_active":"2024-09-11T03:46:32.972371+0000","last_became_peered":"2024-09-11T03:46:32.972371+0000","last_unstale":"2024-09-11T03:50:12.056931+0000","last_undegraded":"2024-09-11T03:50:12.056931+0000","last_fullsized":"2024-09-11T03:50:12.056931+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:48:55.139430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:55.139430+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00090608599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,11],"acting":[10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"133","length":"1"},{"start":"153","length":"2"},{"start":"165","length":"1"}]},{"pgid":"1.1e","version":"20'92","reported_seq":1194,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-09-11T03:50:12.040601+0000","last_change":"2024-09-11T03:49:36.949199+0000","last_active":"2024-09-11T03:50:12.040601+0000","last_peered":"2024-09-11T03:50:12.040601+0000","last_clean":"2024-09-11T03:50:12.040601+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:50:12.040601+0000","last_undegraded":"2024-09-11T03:50:12.040601+0000","last_fullsized":"2024-09-11T03:50:12.040601+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:36.949149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.949149+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:45.388809+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":56,"stat_sum":{"num_bytes":519965398,"num_objects":244,"num_object_clones":195,"num_object_copies":732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":244,"num_whiteouts":17,"num_read":1436,"num_read_kb":1499093,"num_write":995,"num_write_kb":428502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":56010273,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":933777408,"data_stored":1274512714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31064,"ondisk_log_size":31064,"up":168,"acting":168,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285724,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107464,"kb_used_data":76268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93637688,"statfs":{"total":95995035648,"available":95884992512,"internally_reserved":0,"allocated":78098432,"data_stored":101143911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514656,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99856,"kb_used_data":56188,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93645296,"statfs":{"total":95995035648,"available":95892783104,"internally_reserved":0,"allocated":57536512,"data_stored":66443370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":44693952},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743589,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":120344,"kb_used_data":62788,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93624808,"statfs":{"total":95995035648,"available":95871803392,"internally_reserved":0,"allocated":64294912,"data_stored":71537211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037933,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":183492,"kb_used_data":130088,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93561660,"statfs":{"total":95995035648,"available":95807139840,"internally_reserved":0,"allocated":133210112,"data_stored":185773162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299568,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":205088,"kb_used_data":160012,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93540064,"statfs":{"total":95995035648,"available":95785025536,"internally_reserved":0,"allocated":163852288,"data_stored":219495541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528501,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":138648,"kb_used_data":111612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93606504,"statfs":{"total":95995035648,"available":95853060096,"internally_reserved":0,"allocated":114290688,"data_stored":145942991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757435,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91800,"kb_used_data":60612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93653352,"statfs":{"total":95995035648,"available":95901032448,"internally_reserved":0,"allocated":62066688,"data_stored":76402694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986368,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":103664,"kb_used_data":50268,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93641488,"statfs":{"total":95995035648,"available":95888883712,"internally_reserved":0,"allocated":51474432,"data_stored":57295716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149892,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":157628,"kb_used_data":112552,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93587524,"statfs":{"total":95995035648,"available":95833624576,"internally_reserved":0,"allocated":115253248,"data_stored":160203083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411530,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71604,"kb_used_data":40416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93673548,"statfs":{"total":95995035648,"available":95921713152,"internally_reserved":0,"allocated":41385984,"data_stored":42339262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575056,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":122228,"kb_used_data":68824,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93622924,"statfs":{"total":95995035648,"available":95869874176,"internally_reserved":0,"allocated":70475776,"data_stored":91734042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738581,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":143936,"kb_used_data":90532,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93601216,"statfs":{"total":95995035648,"available":95847645184,"internally_reserved":0,"allocated":92704768,"data_stored":120951188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":82796544,"data_stored":114382070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":60870656,"data_stored":85475958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":31625216,"data_stored":35913184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":105357312,"data_stored":153642294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":41893888,"data_stored":51054278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":52256768,"data_stored":69943362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":106708992,"data_stored":144442487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":154054656,"data_stored":213044504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":123604992,"data_stored":179515078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":54738944,"data_stored":67592910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":50282496,"data_stored":63536891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":69586944,"data_stored":95969698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:50:15.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pgp_num 61 2024-09-11T03:50:15.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:15 smithi084 ceph-mon[27254]: 2.37 scrub starts 2024-09-11T03:50:15.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:15 smithi084 ceph-mon[27254]: 2.33 scrub starts 2024-09-11T03:50:15.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:15 smithi084 ceph-mon[27254]: 2.31 scrub starts 2024-09-11T03:50:15.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:15 smithi084 ceph-mon[27254]: 2.2f scrub starts 2024-09-11T03:50:15.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:15 smithi084 ceph-mon[27254]: from='client.17094 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:50:15.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:15 smithi084 ceph-mon[27254]: osdmap e796: 12 total, 12 up, 12 in 2024-09-11T03:50:15.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:15 smithi084 ceph-mon[27254]: 2.36 scrub starts 2024-09-11T03:50:15.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:15 smithi084 ceph-mon[27254]: 2.34 scrub starts 2024-09-11T03:50:15.405 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:15.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:15.406 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:15 smithi121 ceph-mon[30729]: 2.37 scrub starts 2024-09-11T03:50:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:15 smithi121 ceph-mon[30729]: 2.33 scrub starts 2024-09-11T03:50:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:15 smithi121 ceph-mon[30729]: 2.31 scrub starts 2024-09-11T03:50:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:15 smithi121 ceph-mon[30729]: 2.2f scrub starts 2024-09-11T03:50:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:15 smithi121 ceph-mon[30729]: from='client.17094 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:50:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:15 smithi121 ceph-mon[30729]: osdmap e796: 12 total, 12 up, 12 in 2024-09-11T03:50:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:15 smithi121 ceph-mon[30729]: 2.36 scrub starts 2024-09-11T03:50:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:15 smithi121 ceph-mon[30729]: 2.34 scrub starts 2024-09-11T03:50:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:15 smithi175 ceph-mon[30894]: 2.37 scrub starts 2024-09-11T03:50:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:15 smithi175 ceph-mon[30894]: 2.33 scrub starts 2024-09-11T03:50:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:15 smithi175 ceph-mon[30894]: 2.31 scrub starts 2024-09-11T03:50:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:15 smithi175 ceph-mon[30894]: 2.2f scrub starts 2024-09-11T03:50:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:15 smithi175 ceph-mon[30894]: from='client.17094 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:50:15.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:15 smithi175 ceph-mon[30894]: osdmap e796: 12 total, 12 up, 12 in 2024-09-11T03:50:15.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:15 smithi175 ceph-mon[30894]: 2.36 scrub starts 2024-09-11T03:50:15.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:15 smithi175 ceph-mon[30894]: 2.34 scrub starts 2024-09-11T03:50:15.497 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:15.507 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:15.528 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:15.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:15.529 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:15.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:15.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:15.653 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:15.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:15.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:15.775 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:15.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:15.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:15.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:15.892 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:50:15.892 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:50:15.898 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:15.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:15.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:16.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:16.006 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:16.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:16.023 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:16.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:16.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:16.145 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:16.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:16.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:16.152 INFO:tasks.rados.rados.2.smithi175.stdout:3280: done (1 left) 2024-09-11T03:50:16.152 INFO:tasks.rados.rados.2.smithi175.stdout:3281: done (0 left) 2024-09-11T03:50:16.152 INFO:tasks.rados.rados.2.smithi175.stdout:3282: read oid 21 snap -1 2024-09-11T03:50:16.153 INFO:tasks.rados.rados.2.smithi175.stdout:3282: expect (ObjNum 1079 snap 352 seq_num 1079) 2024-09-11T03:50:16.153 INFO:tasks.rados.rados.2.smithi175.stdout:3283: delete oid 3 current snap is 360 2024-09-11T03:50:16.157 INFO:tasks.rados.rados.2.smithi175.stdout:3283: done (1 left) 2024-09-11T03:50:16.157 INFO:tasks.rados.rados.2.smithi175.stdout:3284: read oid 2 snap -1 2024-09-11T03:50:16.157 INFO:tasks.rados.rados.2.smithi175.stdout:3284: expect deleted 2024-09-11T03:50:16.157 INFO:tasks.rados.rados.2.smithi175.stdout:3285: snap_remove snap 353 2024-09-11T03:50:16.247 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:16.269 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:16.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:16.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:16.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:16.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:16 smithi084 ceph-mon[27254]: 2.37 scrub starts 2024-09-11T03:50:16.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:16 smithi084 ceph-mon[27254]: 2.33 scrub starts 2024-09-11T03:50:16.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:16 smithi084 ceph-mon[27254]: pgmap v1350: 87 pgs: 2 active+clean+scrubbing, 85 active+clean; 496 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.7 MiB/s wr, 23 op/s 2024-09-11T03:50:16.385 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:16 smithi084 ceph-mon[27254]: 2.31 scrub starts 2024-09-11T03:50:16.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:16 smithi084 ceph-mon[27254]: 2.2f scrub starts 2024-09-11T03:50:16.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:16 smithi084 ceph-mon[27254]: osdmap e797: 12 total, 12 up, 12 in 2024-09-11T03:50:16.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:16 smithi084 ceph-mon[27254]: 2.32 scrub starts 2024-09-11T03:50:16.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:16 smithi084 ceph-mon[27254]: 2.34 scrub starts 2024-09-11T03:50:16.392 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:16.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:16.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:16 smithi121 ceph-mon[30729]: 2.37 scrub starts 2024-09-11T03:50:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:16 smithi121 ceph-mon[30729]: 2.33 scrub starts 2024-09-11T03:50:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:16 smithi121 ceph-mon[30729]: pgmap v1350: 87 pgs: 2 active+clean+scrubbing, 85 active+clean; 496 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.7 MiB/s wr, 23 op/s 2024-09-11T03:50:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:16 smithi121 ceph-mon[30729]: 2.31 scrub starts 2024-09-11T03:50:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:16 smithi121 ceph-mon[30729]: 2.2f scrub starts 2024-09-11T03:50:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:16 smithi121 ceph-mon[30729]: osdmap e797: 12 total, 12 up, 12 in 2024-09-11T03:50:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:16 smithi121 ceph-mon[30729]: 2.32 scrub starts 2024-09-11T03:50:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:16 smithi121 ceph-mon[30729]: 2.34 scrub starts 2024-09-11T03:50:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:16 smithi175 ceph-mon[30894]: 2.37 scrub starts 2024-09-11T03:50:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:16 smithi175 ceph-mon[30894]: 2.33 scrub starts 2024-09-11T03:50:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:16 smithi175 ceph-mon[30894]: pgmap v1350: 87 pgs: 2 active+clean+scrubbing, 85 active+clean; 496 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.7 MiB/s wr, 23 op/s 2024-09-11T03:50:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:16 smithi175 ceph-mon[30894]: 2.31 scrub starts 2024-09-11T03:50:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:16 smithi175 ceph-mon[30894]: 2.2f scrub starts 2024-09-11T03:50:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:16 smithi175 ceph-mon[30894]: osdmap e797: 12 total, 12 up, 12 in 2024-09-11T03:50:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:16 smithi175 ceph-mon[30894]: 2.32 scrub starts 2024-09-11T03:50:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:16 smithi175 ceph-mon[30894]: 2.34 scrub starts 2024-09-11T03:50:16.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:16.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:16.516 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:16.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:16.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T03:50:16.593 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:50:16.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:16.658 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: specified pgp_num 61 > pg_num 31 2024-09-11T03:50:16.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:16.659 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:16.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:16.760 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:16.782 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:16.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:16.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:16.840 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:50:16.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:16.905 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:16.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:16.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:17.007 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:17.027 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:50:17.033 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:17.033 INFO:tasks.rados.rados.2.smithi175.stdout:3285: done (2 left) 2024-09-11T03:50:17.033 INFO:tasks.rados.rados.2.smithi175.stdout:3286: read oid 22 snap -1 2024-09-11T03:50:17.033 INFO:tasks.rados.rados.2.smithi175.stdout:3286: expect deleted 2024-09-11T03:50:17.033 INFO:tasks.rados.rados.2.smithi175.stdout:3287: write oid 7 current snap is 360 2024-09-11T03:50:17.034 INFO:tasks.rados.rados.2.smithi175.stdout:3287: seq_num 1122 ranges {457454=733269,1923620=589718} 2024-09-11T03:50:17.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:17.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:17.040 INFO:tasks.rados.rados.2.smithi175.stdout:3287: writing smithi17557870-7 from 457454 to 1190723 tid 1 2024-09-11T03:50:17.049 INFO:tasks.rados.rados.2.smithi175.stdout:3287: writing smithi17557870-7 from 1923620 to 2513338 tid 2 2024-09-11T03:50:17.049 INFO:tasks.rados.rados.2.smithi175.stdout:3288: read oid 5 snap 307 2024-09-11T03:50:17.049 INFO:tasks.rados.rados.2.smithi175.stdout:3288: expect deleted 2024-09-11T03:50:17.049 INFO:tasks.rados.rados.2.smithi175.stdout:3289: read oid 45 snap -1 2024-09-11T03:50:17.050 INFO:tasks.rados.rados.2.smithi175.stdout:3289: expect (ObjNum 1094 snap 355 seq_num 1094) 2024-09-11T03:50:17.050 INFO:tasks.rados.rados.2.smithi175.stdout:3290: copy_from oid 28 from oid 13 current snap is 360 2024-09-11T03:50:17.050 INFO:tasks.rados.rados.2.smithi175.stdout:3282: expect (ObjNum 1079 snap 352 seq_num 1079) 2024-09-11T03:50:17.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:17.158 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:17.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:17.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:17.180 INFO:tasks.rados.rados.2.smithi175.stdout:3287: finishing write tid 1 to smithi17557870-7 2024-09-11T03:50:17.180 INFO:tasks.rados.rados.2.smithi175.stdout:3287: finishing write tid 2 to smithi17557870-7 2024-09-11T03:50:17.180 INFO:tasks.rados.rados.2.smithi175.stdout:3287: finishing write tid 3 to smithi17557870-7 2024-09-11T03:50:17.180 INFO:tasks.rados.rados.2.smithi175.stdout:3287: finishing write tid 4 to smithi17557870-7 2024-09-11T03:50:17.181 INFO:tasks.rados.rados.2.smithi175.stdout:3287: oid 7 updating version 0 to 1000 2024-09-11T03:50:17.181 INFO:tasks.rados.rados.2.smithi175.stdout:3287: oid 7 version 1000 is already newer than 999 2024-09-11T03:50:17.181 INFO:tasks.rados.rados.2.smithi175.stdout:3287: oid 7 updating version 1000 to 1001 2024-09-11T03:50:17.181 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 1001 (ObjNum 1122 snap 360 seq_num 1122) dirty exists 2024-09-11T03:50:17.181 INFO:tasks.rados.rados.2.smithi175.stdout:3287: left oid 7 (ObjNum 1122 snap 360 seq_num 1122) 2024-09-11T03:50:17.181 INFO:tasks.rados.rados.2.smithi175.stdout:3289: expect (ObjNum 1094 snap 355 seq_num 1094) 2024-09-11T03:50:17.245 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:17 smithi121 ceph-mon[30729]: 2.35 scrub starts 2024-09-11T03:50:17.245 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:17 smithi121 ceph-mon[30729]: 2.37 scrub starts 2024-09-11T03:50:17.245 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:17 smithi121 ceph-mon[30729]: 2.33 scrub starts 2024-09-11T03:50:17.245 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:17 smithi121 ceph-mon[30729]: 2.31 scrub starts 2024-09-11T03:50:17.245 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:17 smithi121 ceph-mon[30729]: 2.2f scrub starts 2024-09-11T03:50:17.245 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:17 smithi121 ceph-mon[30729]: osdmap e798: 12 total, 12 up, 12 in 2024-09-11T03:50:17.245 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/316931420' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:50:17.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:17 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:50:17.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:17 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:50:17.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3416891280' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-11T03:50:17.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:17 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:17.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/316931420' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:50:17.246 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:17 smithi121 ceph-mon[30729]: osdmap e799: 12 total, 12 up, 12 in 2024-09-11T03:50:17.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:17.267 INFO:tasks.rados.rados.2.smithi175.stdout:3290: finishing copy_from racing read to smithi17557870-28 2024-09-11T03:50:17.267 INFO:tasks.rados.rados.2.smithi175.stdout:3290: finishing copy_from to smithi17557870-28 2024-09-11T03:50:17.267 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 1197 (ObjNum 907 snap 287 seq_num 907) dirty exists 2024-09-11T03:50:17.267 INFO:tasks.rados.rados.2.smithi175.stdout:3282: done (6 left) 2024-09-11T03:50:17.267 INFO:tasks.rados.rados.2.smithi175.stdout:3284: done (5 left) 2024-09-11T03:50:17.267 INFO:tasks.rados.rados.2.smithi175.stdout:3286: done (4 left) 2024-09-11T03:50:17.267 INFO:tasks.rados.rados.2.smithi175.stdout:3287: done (3 left) 2024-09-11T03:50:17.267 INFO:tasks.rados.rados.2.smithi175.stdout:3288: done (2 left) 2024-09-11T03:50:17.267 INFO:tasks.rados.rados.2.smithi175.stdout:3289: done (1 left) 2024-09-11T03:50:17.267 INFO:tasks.rados.rados.2.smithi175.stdout:3290: done (0 left) 2024-09-11T03:50:17.267 INFO:tasks.rados.rados.2.smithi175.stdout:3291: copy_from oid 15 from oid 24 current snap is 360 2024-09-11T03:50:17.267 INFO:tasks.rados.rados.2.smithi175.stdout:3292: read oid 50 snap 335 2024-09-11T03:50:17.268 INFO:tasks.rados.rados.2.smithi175.stdout:3292: expect (ObjNum 1013 snap 331 seq_num 1013) 2024-09-11T03:50:17.268 INFO:tasks.rados.rados.2.smithi175.stdout:3293: snap_create 2024-09-11T03:50:17.278 INFO:tasks.rados.rados.2.smithi175.stdout:3292: expect (ObjNum 1013 snap 331 seq_num 1013) 2024-09-11T03:50:17.283 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:17.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:17.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:17.344 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:50:17.344 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:50:17.385 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:17.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:17 smithi084 ceph-mon[27254]: 2.35 scrub starts 2024-09-11T03:50:17.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:17 smithi084 ceph-mon[27254]: 2.37 scrub starts 2024-09-11T03:50:17.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:17 smithi084 ceph-mon[27254]: 2.33 scrub starts 2024-09-11T03:50:17.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:17 smithi084 ceph-mon[27254]: 2.31 scrub starts 2024-09-11T03:50:17.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:17 smithi084 ceph-mon[27254]: 2.2f scrub starts 2024-09-11T03:50:17.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:17 smithi084 ceph-mon[27254]: osdmap e798: 12 total, 12 up, 12 in 2024-09-11T03:50:17.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/316931420' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:50:17.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:17 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:50:17.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:17 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:50:17.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3416891280' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-11T03:50:17.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:17 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:17.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/316931420' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:50:17.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:17 smithi084 ceph-mon[27254]: osdmap e799: 12 total, 12 up, 12 in 2024-09-11T03:50:17.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:17.408 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:17.408 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:17.416 INFO:tasks.rados.rados.2.smithi175.stdout:3291: finishing copy_from racing read to smithi17557870-15 2024-09-11T03:50:17.416 INFO:tasks.rados.rados.2.smithi175.stdout:3291: finishing copy_from to smithi17557870-15 2024-09-11T03:50:17.416 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 1053 (ObjNum 1120 snap 359 seq_num 1120) dirty exists 2024-09-11T03:50:17.417 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:50:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:17 smithi175 ceph-mon[30894]: 2.35 scrub starts 2024-09-11T03:50:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:17 smithi175 ceph-mon[30894]: 2.37 scrub starts 2024-09-11T03:50:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:17 smithi175 ceph-mon[30894]: 2.33 scrub starts 2024-09-11T03:50:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:17 smithi175 ceph-mon[30894]: 2.31 scrub starts 2024-09-11T03:50:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:17 smithi175 ceph-mon[30894]: 2.2f scrub starts 2024-09-11T03:50:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:17 smithi175 ceph-mon[30894]: osdmap e798: 12 total, 12 up, 12 in 2024-09-11T03:50:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/316931420' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:50:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:17 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:50:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:17 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:50:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3416891280' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-11T03:50:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:17 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/316931420' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:50:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:17 smithi175 ceph-mon[30894]: osdmap e799: 12 total, 12 up, 12 in 2024-09-11T03:50:17.509 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:17.532 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:17.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:17.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:17.634 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:17.656 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:17.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:17.657 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:17.714 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:17.757 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:17.779 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:17.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:17.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:17.881 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:17.903 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:17.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:18.004 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:18.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:18.028 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:18.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:18.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T03:50:18.129 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:18.151 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:18.153 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:18.155 INFO:tasks.rados.rados.2.smithi175.stdout:3291: done (2 left) 2024-09-11T03:50:18.156 INFO:tasks.rados.rados.2.smithi175.stdout:3292: done (1 left) 2024-09-11T03:50:18.156 INFO:tasks.rados.rados.2.smithi175.stdout:3293: done (0 left) 2024-09-11T03:50:18.156 INFO:tasks.rados.rados.2.smithi175.stdout:3294: copy_from oid 29 from oid 22 current snap is 361 2024-09-11T03:50:18.156 INFO:tasks.rados.rados.2.smithi175.stdout:3295: rollback oid 5 current snap is 361 2024-09-11T03:50:18.156 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 5 to 344 2024-09-11T03:50:18.156 INFO:tasks.rados.rados.2.smithi175.stdout:3296: snap_remove snap 360 2024-09-11T03:50:18.253 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:18.277 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:18.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:18.278 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:18.292 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:50:18.379 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:18.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:18 smithi084 ceph-mon[27254]: 2.36 scrub starts 2024-09-11T03:50:18.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:18 smithi084 ceph-mon[27254]: 2.33 scrub starts 2024-09-11T03:50:18.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:18 smithi084 ceph-mon[27254]: pgmap v1353: 87 pgs: 4 active+clean+scrubbing, 83 active+clean; 516 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-09-11T03:50:18.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:18 smithi084 ceph-mon[27254]: 2.32 scrub starts 2024-09-11T03:50:18.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:18 smithi084 ceph-mon[27254]: 2.32 scrub ok 2024-09-11T03:50:18.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:18 smithi084 ceph-mon[27254]: 2.34 scrub starts 2024-09-11T03:50:18.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:18 smithi084 ceph-mon[27254]: 2.34 scrub ok 2024-09-11T03:50:18.400 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:18.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:18.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:18.408 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:18 smithi121 ceph-mon[30729]: 2.36 scrub starts 2024-09-11T03:50:18.408 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:18 smithi121 ceph-mon[30729]: 2.33 scrub starts 2024-09-11T03:50:18.408 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:18 smithi121 ceph-mon[30729]: pgmap v1353: 87 pgs: 4 active+clean+scrubbing, 83 active+clean; 516 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-09-11T03:50:18.408 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:18 smithi121 ceph-mon[30729]: 2.32 scrub starts 2024-09-11T03:50:18.408 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:18 smithi121 ceph-mon[30729]: 2.32 scrub ok 2024-09-11T03:50:18.408 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:18 smithi121 ceph-mon[30729]: 2.34 scrub starts 2024-09-11T03:50:18.408 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:18 smithi121 ceph-mon[30729]: 2.34 scrub ok 2024-09-11T03:50:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:18 smithi175 ceph-mon[30894]: 2.36 scrub starts 2024-09-11T03:50:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:18 smithi175 ceph-mon[30894]: 2.33 scrub starts 2024-09-11T03:50:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:18 smithi175 ceph-mon[30894]: pgmap v1353: 87 pgs: 4 active+clean+scrubbing, 83 active+clean; 516 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-09-11T03:50:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:18 smithi175 ceph-mon[30894]: 2.32 scrub starts 2024-09-11T03:50:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:18 smithi175 ceph-mon[30894]: 2.32 scrub ok 2024-09-11T03:50:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:18 smithi175 ceph-mon[30894]: 2.34 scrub starts 2024-09-11T03:50:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:18 smithi175 ceph-mon[30894]: 2.34 scrub ok 2024-09-11T03:50:18.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:18.524 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:18.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:18.626 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:18.644 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:50:18.644 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:50:18.650 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:18.651 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:18.661 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:50:18.661 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:50:18.677 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:50:18.677 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:50:18.693 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:50:18.693 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:50:18.710 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:50:18.710 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:50:18.727 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:50:18.727 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:50:18.743 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:50:18.743 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:50:18.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:18.760 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:50:18.761 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:50:18.770 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:50:18.770 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:50:18.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:50:18.770 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:50:18.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.770 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1555 2.3 2:c6c49f2c:::smithi17557870-44:head [copy-from ver 82] snapc 3c=[3c,3a,38,37] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-11T03:50:18.770 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:30.880293+0000", 2024-09-11T03:50:18.770 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 587.87316864100001, 2024-09-11T03:50:18.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020483438999999999, 2024-09-11T03:50:18.770 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.771 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.771 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:18.771 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.771 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.771 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.771 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1555 2024-09-11T03:50:18.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.771 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880293+0000", 2024-09-11T03:50:18.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880293+0000", 2024-09-11T03:50:18.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880295+0000", 2024-09-11T03:50:18.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2409999999999998e-06 2024-09-11T03:50:18.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880305+0000", 2024-09-11T03:50:18.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5999999999999996e-06 2024-09-11T03:50:18.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880326+0000", 2024-09-11T03:50:18.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1248999999999999e-05 2024-09-11T03:50:18.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880335+0000", 2024-09-11T03:50:18.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4369999999999996e-06 2024-09-11T03:50:18.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880414+0000", 2024-09-11T03:50:18.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8560999999999995e-05 2024-09-11T03:50:18.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:50:18.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880422+0000", 2024-09-11T03:50:18.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.712e-06 2024-09-11T03:50:18.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887559+0000", 2024-09-11T03:50:18.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071376920000000002 2024-09-11T03:50:18.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887625+0000", 2024-09-11T03:50:18.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6256999999999999e-05 2024-09-11T03:50:18.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:50:18.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.893371+0000", 2024-09-11T03:50:18.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0057455579999999996 2024-09-11T03:50:18.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:18.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.899577+0000", 2024-09-11T03:50:18.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062060600000000002 2024-09-11T03:50:18.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:50:18.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900444+0000", 2024-09-11T03:50:18.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000867307 2024-09-11T03:50:18.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:18.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900703+0000", 2024-09-11T03:50:18.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00025901299999999999 2024-09-11T03:50:18.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:50:18.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900743+0000", 2024-09-11T03:50:18.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9471000000000002e-05 2024-09-11T03:50:18.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.900776+0000", 2024-09-11T03:50:18.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3281000000000002e-05 2024-09-11T03:50:18.777 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.777 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.777 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.777 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:1556 2.3 2:c6c49f2c:::smithi17557870-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e214)", 2024-09-11T03:50:18.778 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:40:30.880359+0000", 2024-09-11T03:50:18.778 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 587.87310194400004, 2024-09-11T03:50:18.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017105065999999999, 2024-09-11T03:50:18.778 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.778 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.778 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:50:18.778 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.778 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.778 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.778 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1556 2024-09-11T03:50:18.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.778 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880359+0000", 2024-09-11T03:50:18.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880359+0000", 2024-09-11T03:50:18.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880360+0000", 2024-09-11T03:50:18.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.75e-07 2024-09-11T03:50:18.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880362+0000", 2024-09-11T03:50:18.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4959999999999999e-06 2024-09-11T03:50:18.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880370+0000", 2024-09-11T03:50:18.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.153e-06 2024-09-11T03:50:18.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880376+0000", 2024-09-11T03:50:18.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0900000000000001e-06 2024-09-11T03:50:18.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880478+0000", 2024-09-11T03:50:18.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102482 2024-09-11T03:50:18.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:50:18.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.880487+0000", 2024-09-11T03:50:18.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8109999999999992e-06 2024-09-11T03:50:18.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887725+0000", 2024-09-11T03:50:18.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072383930000000001 2024-09-11T03:50:18.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:18.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.887757+0000", 2024-09-11T03:50:18.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1692000000000001e-05 2024-09-11T03:50:18.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.897328+0000", 2024-09-11T03:50:18.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095712149999999992 2024-09-11T03:50:18.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.897393+0000", 2024-09-11T03:50:18.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5005000000000005e-05 2024-09-11T03:50:18.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:40:30.897464+0000", 2024-09-11T03:50:18.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1254000000000006e-05 2024-09-11T03:50:18.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.783 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.784 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3040 2.d 2:b6a25a79:::smithi17557870-47:head [read 0~2201783,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:50:18.784 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:50:18.784 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 446.85648171700001, 2024-09-11T03:50:18.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020060432, 2024-09-11T03:50:18.784 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.784 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.784 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:50:18.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.785 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3040 2024-09-11T03:50:18.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.785 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:50:18.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:50:18.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896981+0000", 2024-09-11T03:50:18.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.747e-06 2024-09-11T03:50:18.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896989+0000", 2024-09-11T03:50:18.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9559999999999997e-06 2024-09-11T03:50:18.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897005+0000", 2024-09-11T03:50:18.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5704000000000001e-05 2024-09-11T03:50:18.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897013+0000", 2024-09-11T03:50:18.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.028e-06 2024-09-11T03:50:18.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897078+0000", 2024-09-11T03:50:18.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4813000000000004e-05 2024-09-11T03:50:18.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:50:18.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897083+0000", 2024-09-11T03:50:18.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1109999999999999e-06 2024-09-11T03:50:18.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.914637+0000", 2024-09-11T03:50:18.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017553701000000001 2024-09-11T03:50:18.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.914744+0000", 2024-09-11T03:50:18.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107131 2024-09-11T03:50:18.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917040+0000", 2024-09-11T03:50:18.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022962410000000001 2024-09-11T03:50:18.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.789 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.789 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3041 2.d 2:b6a25a79:::smithi17557870-47:head [sparse-read 0~1480887] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:50:18.789 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:50:18.789 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 446.85641616100003, 2024-09-11T03:50:18.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020163026000000001, 2024-09-11T03:50:18.789 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.790 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.790 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:50:18.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.790 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3041 2024-09-11T03:50:18.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.790 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:50:18.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:50:18.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897046+0000", 2024-09-11T03:50:18.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7e-07 2024-09-11T03:50:18.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897048+0000", 2024-09-11T03:50:18.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0700000000000001e-06 2024-09-11T03:50:18.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897058+0000", 2024-09-11T03:50:18.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0151e-05 2024-09-11T03:50:18.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897068+0000", 2024-09-11T03:50:18.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0329000000000001e-05 2024-09-11T03:50:18.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897128+0000", 2024-09-11T03:50:18.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0208999999999999e-05 2024-09-11T03:50:18.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:50:18.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897132+0000", 2024-09-11T03:50:18.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7290000000000002e-06 2024-09-11T03:50:18.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917070+0000", 2024-09-11T03:50:18.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019937951999999998 2024-09-11T03:50:18.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917123+0000", 2024-09-11T03:50:18.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3149000000000002e-05 2024-09-11T03:50:18.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917208+0000", 2024-09-11T03:50:18.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5067000000000004e-05 2024-09-11T03:50:18.797 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.797 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.797 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.797 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3042 2.d 2:b6a25a79:::smithi17557870-47:head [sparse-read 0~1480887] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:50:18.797 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:50:18.797 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 446.85638198200002, 2024-09-11T03:50:18.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020302042999999999, 2024-09-11T03:50:18.798 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.798 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.798 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:50:18.798 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.798 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.798 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.798 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3042 2024-09-11T03:50:18.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.798 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:50:18.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:50:18.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897080+0000", 2024-09-11T03:50:18.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.82e-07 2024-09-11T03:50:18.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897082+0000", 2024-09-11T03:50:18.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.418e-06 2024-09-11T03:50:18.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897089+0000", 2024-09-11T03:50:18.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8090000000000004e-06 2024-09-11T03:50:18.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897094+0000", 2024-09-11T03:50:18.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2399999999999998e-06 2024-09-11T03:50:18.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:50:18.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0928999999999999e-05 2024-09-11T03:50:18.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:50:18.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897158+0000", 2024-09-11T03:50:18.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9890000000000001e-06 2024-09-11T03:50:18.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917227+0000", 2024-09-11T03:50:18.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020069019 2024-09-11T03:50:18.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917290+0000", 2024-09-11T03:50:18.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2848999999999994e-05 2024-09-11T03:50:18.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917381+0000", 2024-09-11T03:50:18.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1308000000000001e-05 2024-09-11T03:50:18.802 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.802 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.802 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.803 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3046 2.d 2:b6874adf:::smithi17557870-17:head [sparse-read 0~1769472,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:50:18.803 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:50:18.803 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 446.856360021, 2024-09-11T03:50:18.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021672665000000001, 2024-09-11T03:50:18.803 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.803 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.803 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:50:18.803 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.803 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.803 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.803 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3046 2024-09-11T03:50:18.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.804 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:50:18.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:50:18.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897102+0000", 2024-09-11T03:50:18.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.65e-07 2024-09-11T03:50:18.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897105+0000", 2024-09-11T03:50:18.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1820000000000002e-06 2024-09-11T03:50:18.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897111+0000", 2024-09-11T03:50:18.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4330000000000001e-06 2024-09-11T03:50:18.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897117+0000", 2024-09-11T03:50:18.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2979999999999999e-06 2024-09-11T03:50:18.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897179+0000", 2024-09-11T03:50:18.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2825999999999998e-05 2024-09-11T03:50:18.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:50:18.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897182+0000", 2024-09-11T03:50:18.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7769999999999998e-06 2024-09-11T03:50:18.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917399+0000", 2024-09-11T03:50:18.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020216622 2024-09-11T03:50:18.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917444+0000", 2024-09-11T03:50:18.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5558e-05 2024-09-11T03:50:18.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918774+0000", 2024-09-11T03:50:18.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013296040000000001 2024-09-11T03:50:18.809 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.809 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.809 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.809 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3047 2.d 2:b6874adf:::smithi17557870-17:head [read 0~2453888,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:50:18.809 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:50:18.809 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 446.85633084900002, 2024-09-11T03:50:18.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022691901, 2024-09-11T03:50:18.810 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.810 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.810 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:50:18.810 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.810 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.810 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.810 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3047 2024-09-11T03:50:18.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.810 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:50:18.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:50:18.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:50:18.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8599999999999998e-07 2024-09-11T03:50:18.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897134+0000", 2024-09-11T03:50:18.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1470000000000001e-06 2024-09-11T03:50:18.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897141+0000", 2024-09-11T03:50:18.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8029999999999999e-06 2024-09-11T03:50:18.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897147+0000", 2024-09-11T03:50:18.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4520000000000002e-06 2024-09-11T03:50:18.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897202+0000", 2024-09-11T03:50:18.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4255999999999997e-05 2024-09-11T03:50:18.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:50:18.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897204+0000", 2024-09-11T03:50:18.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6649999999999999e-06 2024-09-11T03:50:18.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918807+0000", 2024-09-11T03:50:18.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021602962999999999 2024-09-11T03:50:18.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918887+0000", 2024-09-11T03:50:18.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9550999999999998e-05 2024-09-11T03:50:18.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919822+0000", 2024-09-11T03:50:18.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093557799999999995 2024-09-11T03:50:18.814 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.814 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.815 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.815 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3048 2.d 2:b6874adf:::smithi17557870-17:head [read 0~2453888,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:50:18.815 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:50:18.815 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 446.85630602499998, 2024-09-11T03:50:18.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.023274353000000001, 2024-09-11T03:50:18.815 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.815 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.815 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:50:18.815 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.815 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.815 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.815 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3048 2024-09-11T03:50:18.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.816 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:50:18.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:50:18.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897156+0000", 2024-09-11T03:50:18.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1199999999999998e-07 2024-09-11T03:50:18.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897159+0000", 2024-09-11T03:50:18.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7439999999999999e-06 2024-09-11T03:50:18.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897165+0000", 2024-09-11T03:50:18.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6749999999999996e-06 2024-09-11T03:50:18.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897170+0000", 2024-09-11T03:50:18.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9420000000000003e-06 2024-09-11T03:50:18.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897226+0000", 2024-09-11T03:50:18.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6026999999999997e-05 2024-09-11T03:50:18.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:50:18.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897232+0000", 2024-09-11T03:50:18.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2609999999999999e-06 2024-09-11T03:50:18.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919841+0000", 2024-09-11T03:50:18.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022609054999999999 2024-09-11T03:50:18.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919885+0000", 2024-09-11T03:50:18.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3383999999999997e-05 2024-09-11T03:50:18.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.920430+0000", 2024-09-11T03:50:18.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054485299999999998 2024-09-11T03:50:18.820 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.820 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.820 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.820 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4176 2.3 2:c6c49f2c:::smithi17557870-44:head [copy-from ver 412] snapc b5=[b5,b4,b2,b1,af,ae,ac,ab,a9,a8,9d] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-11T03:50:18.820 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:28.933487+0000", 2024-09-11T03:50:18.820 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 349.81997481799999, 2024-09-11T03:50:18.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018627511999999999, 2024-09-11T03:50:18.821 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.821 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.821 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:18.821 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.821 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.821 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.821 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4176 2024-09-11T03:50:18.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.821 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933487+0000", 2024-09-11T03:50:18.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933487+0000", 2024-09-11T03:50:18.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933489+0000", 2024-09-11T03:50:18.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2510000000000001e-06 2024-09-11T03:50:18.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933498+0000", 2024-09-11T03:50:18.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3219999999999999e-06 2024-09-11T03:50:18.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933516+0000", 2024-09-11T03:50:18.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7971000000000001e-05 2024-09-11T03:50:18.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933529+0000", 2024-09-11T03:50:18.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2476e-05 2024-09-11T03:50:18.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933657+0000", 2024-09-11T03:50:18.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000128842 2024-09-11T03:50:18.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:50:18.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933663+0000", 2024-09-11T03:50:18.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9580000000000004e-06 2024-09-11T03:50:18.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935429+0000", 2024-09-11T03:50:18.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001765146 2024-09-11T03:50:18.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935549+0000", 2024-09-11T03:50:18.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120504 2024-09-11T03:50:18.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:50:18.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.944235+0000", 2024-09-11T03:50:18.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086856499999999996 2024-09-11T03:50:18.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:50:18.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948823+0000", 2024-09-11T03:50:18.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045879570000000001 2024-09-11T03:50:18.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:18.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.951466+0000", 2024-09-11T03:50:18.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026434739999999998 2024-09-11T03:50:18.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:18.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.952026+0000", 2024-09-11T03:50:18.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055985599999999996 2024-09-11T03:50:18.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:50:18.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.952068+0000", 2024-09-11T03:50:18.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1918000000000001e-05 2024-09-11T03:50:18.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.952114+0000", 2024-09-11T03:50:18.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6186999999999997e-05 2024-09-11T03:50:18.827 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.827 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.827 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.828 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4177 2.3 2:c6c49f2c:::smithi17557870-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e451)", 2024-09-11T03:50:18.828 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:28.933561+0000", 2024-09-11T03:50:18.828 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 349.81990000500002, 2024-09-11T03:50:18.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014798686, 2024-09-11T03:50:18.828 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.828 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.828 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:50:18.828 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.828 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.828 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.828 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4177 2024-09-11T03:50:18.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.829 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933561+0000", 2024-09-11T03:50:18.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933561+0000", 2024-09-11T03:50:18.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933562+0000", 2024-09-11T03:50:18.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4600000000000005e-07 2024-09-11T03:50:18.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933565+0000", 2024-09-11T03:50:18.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8729999999999999e-06 2024-09-11T03:50:18.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933572+0000", 2024-09-11T03:50:18.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6090000000000003e-06 2024-09-11T03:50:18.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933579+0000", 2024-09-11T03:50:18.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4339999999999997e-06 2024-09-11T03:50:18.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933717+0000", 2024-09-11T03:50:18.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000137995 2024-09-11T03:50:18.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:50:18.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933721+0000", 2024-09-11T03:50:18.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6940000000000001e-06 2024-09-11T03:50:18.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935773+0000", 2024-09-11T03:50:18.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002052398 2024-09-11T03:50:18.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:18.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935834+0000", 2024-09-11T03:50:18.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1146e-05 2024-09-11T03:50:18.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948174+0000", 2024-09-11T03:50:18.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012339786 2024-09-11T03:50:18.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948263+0000", 2024-09-11T03:50:18.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8838999999999994e-05 2024-09-11T03:50:18.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948360+0000", 2024-09-11T03:50:18.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7366000000000003e-05 2024-09-11T03:50:18.833 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.833 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.833 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.834 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:6215 2.16 2:6dbd1e81:::smithi17557870-7:head [copy-from ver 268] snapc 112=[112,111,10f] ondisk+write+known_if_redirected+supports_pool_eio e646)", 2024-09-11T03:50:18.834 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:43.987602+0000", 2024-09-11T03:50:18.834 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 154.76585961500001, 2024-09-11T03:50:18.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021363759, 2024-09-11T03:50:18.834 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.834 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.834 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:18.834 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.834 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.834 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.834 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 6215 2024-09-11T03:50:18.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.835 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987602+0000", 2024-09-11T03:50:18.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987602+0000", 2024-09-11T03:50:18.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987603+0000", 2024-09-11T03:50:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6339999999999999e-06 2024-09-11T03:50:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987612+0000", 2024-09-11T03:50:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1170000000000004e-06 2024-09-11T03:50:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987631+0000", 2024-09-11T03:50:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9245e-05 2024-09-11T03:50:18.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987641+0000", 2024-09-11T03:50:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8570000000000007e-06 2024-09-11T03:50:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987718+0000", 2024-09-11T03:50:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7812999999999995e-05 2024-09-11T03:50:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:50:18.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987726+0000", 2024-09-11T03:50:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1960000000000002e-06 2024-09-11T03:50:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.989804+0000", 2024-09-11T03:50:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002078004 2024-09-11T03:50:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.989909+0000", 2024-09-11T03:50:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105518 2024-09-11T03:50:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6,11", 2024-09-11T03:50:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.999305+0000", 2024-09-11T03:50:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.009396349 2024-09-11T03:50:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:50:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.007386+0000", 2024-09-11T03:50:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080807069999999995 2024-09-11T03:50:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.007696+0000", 2024-09-11T03:50:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030955000000000002 2024-09-11T03:50:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.008885+0000", 2024-09-11T03:50:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011892700000000001 2024-09-11T03:50:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:50:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.008929+0000", 2024-09-11T03:50:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4322e-05 2024-09-11T03:50:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.008966+0000", 2024-09-11T03:50:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6177000000000003e-05 2024-09-11T03:50:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.840 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:6216 2.16 2:6dbd1e81:::smithi17557870-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e646)", 2024-09-11T03:50:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:43.987704+0000", 2024-09-11T03:50:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 154.76575759299999, 2024-09-11T03:50:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016255282999999999, 2024-09-11T03:50:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:50:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.841 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 6216 2024-09-11T03:50:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987704+0000", 2024-09-11T03:50:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987704+0000", 2024-09-11T03:50:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987704+0000", 2024-09-11T03:50:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5000000000000002e-07 2024-09-11T03:50:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987708+0000", 2024-09-11T03:50:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6610000000000002e-06 2024-09-11T03:50:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987718+0000", 2024-09-11T03:50:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7690000000000008e-06 2024-09-11T03:50:18.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987725+0000", 2024-09-11T03:50:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7279999999999998e-06 2024-09-11T03:50:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987785+0000", 2024-09-11T03:50:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0163e-05 2024-09-11T03:50:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:50:18.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987790+0000", 2024-09-11T03:50:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2410000000000003e-06 2024-09-11T03:50:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.990032+0000", 2024-09-11T03:50:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022420880000000002 2024-09-11T03:50:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.990058+0000", 2024-09-11T03:50:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6044999999999999e-05 2024-09-11T03:50:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.003794+0000", 2024-09-11T03:50:18.846 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:50:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01373564 2024-09-11T03:50:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.003878+0000", 2024-09-11T03:50:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3954000000000003e-05 2024-09-11T03:50:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.003959+0000", 2024-09-11T03:50:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1343999999999999e-05 2024-09-11T03:50:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.847 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.848 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.848 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:7056 2.20 e710/574 2:0762776a:::smithi17557870-25:head v 710'215, mlcod=628'211)", 2024-09-11T03:50:18.848 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:48.028737+0000", 2024-09-11T03:50:18.848 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 90.724724348999999, 2024-09-11T03:50:18.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015591568, 2024-09-11T03:50:18.848 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.848 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.848 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:18.848 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.028737+0000", 2024-09-11T03:50:18.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.028737+0000", 2024-09-11T03:50:18.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.028737+0000", 2024-09-11T03:50:18.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0600000000000001e-07 2024-09-11T03:50:18.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032034+0000", 2024-09-11T03:50:18.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032964890000000001 2024-09-11T03:50:18.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032066+0000", 2024-09-11T03:50:18.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2011999999999998e-05 2024-09-11T03:50:18.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032079+0000", 2024-09-11T03:50:18.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2702e-05 2024-09-11T03:50:18.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032193+0000", 2024-09-11T03:50:18.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114737 2024-09-11T03:50:18.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032231+0000", 2024-09-11T03:50:18.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7867000000000002e-05 2024-09-11T03:50:18.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:50:18.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.044229+0000", 2024-09-11T03:50:18.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011997499999999999 2024-09-11T03:50:18.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.044329+0000", 2024-09-11T03:50:18.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9854999999999999e-05 2024-09-11T03:50:18.852 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.852 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.852 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.852 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7213 2.d 2:b6874adf:::smithi17557870-17:head [write 2124737~564614] snapc 141=[141,140,13f,13d,13c,139,138,137,136,135,133,12e,12d,12a,127,126,120,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-09-11T03:50:18.853 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:01.038236+0000", 2024-09-11T03:50:18.853 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 77.715225618999995, 2024-09-11T03:50:18.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014496043, 2024-09-11T03:50:18.853 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.853 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.853 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:18.853 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.853 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.853 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.853 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7213 2024-09-11T03:50:18.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.853 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038236+0000", 2024-09-11T03:50:18.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038236+0000", 2024-09-11T03:50:18.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038237+0000", 2024-09-11T03:50:18.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.153e-06 2024-09-11T03:50:18.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038729+0000", 2024-09-11T03:50:18.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049247500000000003 2024-09-11T03:50:18.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038742+0000", 2024-09-11T03:50:18.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2169e-05 2024-09-11T03:50:18.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038747+0000", 2024-09-11T03:50:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6099999999999997e-06 2024-09-11T03:50:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038800+0000", 2024-09-11T03:50:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2920999999999997e-05 2024-09-11T03:50:18.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038911+0000", 2024-09-11T03:50:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011138599999999999 2024-09-11T03:50:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:50:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.039079+0000", 2024-09-11T03:50:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000167928 2024-09-11T03:50:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:50:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.040472+0000", 2024-09-11T03:50:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001392826 2024-09-11T03:50:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:18.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.040615+0000", 2024-09-11T03:50:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014295700000000001 2024-09-11T03:50:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:18.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.052594+0000", 2024-09-11T03:50:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011979001 2024-09-11T03:50:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:50:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.052675+0000", 2024-09-11T03:50:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0611999999999993e-05 2024-09-11T03:50:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.052732+0000", 2024-09-11T03:50:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7005000000000001e-05 2024-09-11T03:50:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7215 2.d 2:b6874adf:::smithi17557870-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e723)", 2024-09-11T03:50:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:01.038790+0000", 2024-09-11T03:50:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 77.714671799000001, 2024-09-11T03:50:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014304828, 2024-09-11T03:50:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:50:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.865 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7215 2024-09-11T03:50:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038790+0000", 2024-09-11T03:50:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038790+0000", 2024-09-11T03:50:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038790+0000", 2024-09-11T03:50:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9499999999999998e-07 2024-09-11T03:50:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.869 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:50:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:18.870 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:50:18.870 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:50:18.870 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:18.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:18.871 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:50:18.871 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:50:18.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038791+0000", 2024-09-11T03:50:18.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.544e-06 2024-09-11T03:50:18.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038800+0000", 2024-09-11T03:50:18.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4309999999999994e-06 2024-09-11T03:50:18.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038803+0000", 2024-09-11T03:50:18.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3739999999999998e-06 2024-09-11T03:50:18.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.040394+0000", 2024-09-11T03:50:18.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001591189 2024-09-11T03:50:18.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:50:18.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.040462+0000", 2024-09-11T03:50:18.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8053999999999996e-05 2024-09-11T03:50:18.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.052896+0000", 2024-09-11T03:50:18.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012434031 2024-09-11T03:50:18.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.052994+0000", 2024-09-11T03:50:18.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7693000000000001e-05 2024-09-11T03:50:18.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.053094+0000", 2024-09-11T03:50:18.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010021700000000001 2024-09-11T03:50:18.875 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.875 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.875 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.875 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7230 2.d 2:b1c0f730:::smithi17557870-20:head [copy-from ver 956] snapc 141=[141,140,13f,13d,13c,139,138,137,136,135,133,12e,12d,12a,127,126,120,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-09-11T03:50:18.875 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:01.070035+0000", 2024-09-11T03:50:18.875 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 77.683426796999996, 2024-09-11T03:50:18.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014365849999999999, 2024-09-11T03:50:18.875 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.875 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.876 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:18.876 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.876 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.876 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.876 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7230 2024-09-11T03:50:18.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.876 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070035+0000", 2024-09-11T03:50:18.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070035+0000", 2024-09-11T03:50:18.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070036+0000", 2024-09-11T03:50:18.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6559999999999999e-06 2024-09-11T03:50:18.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070047+0000", 2024-09-11T03:50:18.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.047e-05 2024-09-11T03:50:18.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070063+0000", 2024-09-11T03:50:18.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6310000000000001e-05 2024-09-11T03:50:18.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070072+0000", 2024-09-11T03:50:18.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9819999999999997e-06 2024-09-11T03:50:18.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070145+0000", 2024-09-11T03:50:18.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3323000000000003e-05 2024-09-11T03:50:18.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070237+0000", 2024-09-11T03:50:18.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1266000000000003e-05 2024-09-11T03:50:18.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:50:18.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.076752+0000", 2024-09-11T03:50:18.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065155760000000004 2024-09-11T03:50:18.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:18.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.083798+0000", 2024-09-11T03:50:18.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070462249999999997 2024-09-11T03:50:18.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:50:18.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.083901+0000", 2024-09-11T03:50:18.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103115 2024-09-11T03:50:18.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:18.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.084281+0000", 2024-09-11T03:50:18.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037937799999999999 2024-09-11T03:50:18.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:50:18.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.084346+0000", 2024-09-11T03:50:18.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4817e-05 2024-09-11T03:50:18.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.084400+0000", 2024-09-11T03:50:18.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4731999999999997e-05 2024-09-11T03:50:18.882 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.882 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.882 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.883 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7865 2.16 2:6dbd1e81:::smithi17557870-7:head [copy-from ver 962] snapc 15f=[15f,15e,15d,15c,15b,15a,157,156,155,154,153,151,14d,148,144,143,13d,13c,139,135,133,12a,127,120] ondisk+write+known_if_redirected+supports_pool_eio e779)", 2024-09-11T03:50:18.883 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:57.136704+0000", 2024-09-11T03:50:18.883 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.616757602, 2024-09-11T03:50:18.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012761855000000001, 2024-09-11T03:50:18.883 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.883 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.883 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:18.883 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.883 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.883 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.883 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7865 2024-09-11T03:50:18.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.884 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.136704+0000", 2024-09-11T03:50:18.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.136704+0000", 2024-09-11T03:50:18.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.136706+0000", 2024-09-11T03:50:18.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7829999999999999e-06 2024-09-11T03:50:18.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.136712+0000", 2024-09-11T03:50:18.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2199999999999997e-06 2024-09-11T03:50:18.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.136726+0000", 2024-09-11T03:50:18.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4192999999999999e-05 2024-09-11T03:50:18.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.136732+0000", 2024-09-11T03:50:18.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4570000000000002e-06 2024-09-11T03:50:18.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.136821+0000", 2024-09-11T03:50:18.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8348000000000002e-05 2024-09-11T03:50:18.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.136920+0000", 2024-09-11T03:50:18.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8750000000000002e-05 2024-09-11T03:50:18.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6,11", 2024-09-11T03:50:18.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.141151+0000", 2024-09-11T03:50:18.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042313760000000002 2024-09-11T03:50:18.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:50:18.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.146774+0000", 2024-09-11T03:50:18.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056231220000000004 2024-09-11T03:50:18.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:18.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.147153+0000", 2024-09-11T03:50:18.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000378608 2024-09-11T03:50:18.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:18.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.149299+0000", 2024-09-11T03:50:18.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021462199999999999 2024-09-11T03:50:18.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:50:18.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.149378+0000", 2024-09-11T03:50:18.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9099999999999998e-05 2024-09-11T03:50:18.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.149466+0000", 2024-09-11T03:50:18.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7677999999999996e-05 2024-09-11T03:50:18.890 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.890 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.890 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.890 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7866 2.16 2:6dbd1e81:::smithi17557870-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e779)", 2024-09-11T03:50:18.890 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:57.136748+0000", 2024-09-11T03:50:18.890 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.616713351000001, 2024-09-11T03:50:18.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080337619999999998, 2024-09-11T03:50:18.890 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.891 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.891 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:50:18.891 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.891 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.891 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.891 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7866 2024-09-11T03:50:18.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.891 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.136748+0000", 2024-09-11T03:50:18.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.136748+0000", 2024-09-11T03:50:18.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.136748+0000", 2024-09-11T03:50:18.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9099999999999999e-07 2024-09-11T03:50:18.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.136750+0000", 2024-09-11T03:50:18.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.658e-06 2024-09-11T03:50:18.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.136776+0000", 2024-09-11T03:50:18.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5802000000000001e-05 2024-09-11T03:50:18.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.136779+0000", 2024-09-11T03:50:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4999999999999999e-06 2024-09-11T03:50:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.137049+0000", 2024-09-11T03:50:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00026971299999999998 2024-09-11T03:50:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.137084+0000", 2024-09-11T03:50:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5054e-05 2024-09-11T03:50:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.144582+0000", 2024-09-11T03:50:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0074977669999999998 2024-09-11T03:50:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.144695+0000", 2024-09-11T03:50:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113445 2024-09-11T03:50:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.144782+0000", 2024-09-11T03:50:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6432000000000002e-05 2024-09-11T03:50:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7961 2.3 2:c3768452:::smithi17557870-11:157 [read 0~3294403,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e785)", 2024-09-11T03:50:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:50:03.063230+0000", 2024-09-11T03:50:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.690231385000001, 2024-09-11T03:50:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0088286930000000003, 2024-09-11T03:50:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:50:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7961 2024-09-11T03:50:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.063230+0000", 2024-09-11T03:50:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.063230+0000", 2024-09-11T03:50:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.063231+0000", 2024-09-11T03:50:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0020000000000001e-06 2024-09-11T03:50:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.063241+0000", 2024-09-11T03:50:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0084e-05 2024-09-11T03:50:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.063255+0000", 2024-09-11T03:50:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3987e-05 2024-09-11T03:50:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.063267+0000", 2024-09-11T03:50:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1599e-05 2024-09-11T03:50:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.068878+0000", 2024-09-11T03:50:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056114379999999998 2024-09-11T03:50:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.069005+0000", 2024-09-11T03:50:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000126709 2024-09-11T03:50:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.072059+0000", 2024-09-11T03:50:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030538739999999998 2024-09-11T03:50:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7962 2.3 2:c3768452:::smithi17557870-11:157 [read 0~3294403,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e785)", 2024-09-11T03:50:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:50:03.064615+0000", 2024-09-11T03:50:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.688846855, 2024-09-11T03:50:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069939939999999999, 2024-09-11T03:50:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:50:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:50:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:50:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:50:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:50:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7962 2024-09-11T03:50:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:50:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:50:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.064615+0000", 2024-09-11T03:50:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:50:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.064615+0000", 2024-09-11T03:50:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:50:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:50:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.064615+0000", 2024-09-11T03:50:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.09e-07 2024-09-11T03:50:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:50:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.064621+0000", 2024-09-11T03:50:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2140000000000004e-06 2024-09-11T03:50:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:50:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.064630+0000", 2024-09-11T03:50:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7399999999999999e-06 2024-09-11T03:50:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:50:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.064651+0000", 2024-09-11T03:50:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0251999999999999e-05 2024-09-11T03:50:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:50:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.070535+0000", 2024-09-11T03:50:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058844530000000004 2024-09-11T03:50:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:50:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.070663+0000", 2024-09-11T03:50:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012777900000000001 2024-09-11T03:50:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:50:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:50:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:50:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:03.071609+0000", 2024-09-11T03:50:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00094564699999999998 2024-09-11T03:50:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.906 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.906 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.906 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:50:18.906 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:50:18.906 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:50:18.906 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:50:18.906 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:50:18.972 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:18.996 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:18.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:19.000 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:19.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:19.122 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:19.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:19.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:19.171 INFO:tasks.rados.rados.2.smithi175.stdout:3296: done (2 left) 2024-09-11T03:50:19.172 INFO:tasks.rados.rados.2.smithi175.stdout:3297: read oid 38 snap -1 2024-09-11T03:50:19.172 INFO:tasks.rados.rados.2.smithi175.stdout:3297: expect (ObjNum 970 snap 311 seq_num 970) 2024-09-11T03:50:19.172 INFO:tasks.rados.rados.2.smithi175.stdout:3298: snap_create 2024-09-11T03:50:19.172 INFO:tasks.rados.rados.2.smithi175.stdout:3295: finishing rollback tid 0 to smithi17557870-5 2024-09-11T03:50:19.172 INFO:tasks.rados.rados.2.smithi175.stdout:3295: finishing rollback tid 1 to smithi17557870-5 2024-09-11T03:50:19.172 INFO:tasks.rados.rados.2.smithi175.stdout:3295: finishing rollback tid 2 to smithi17557870-5 2024-09-11T03:50:19.172 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 1034 (ObjNum 1043 snap 340 seq_num 1043) dirty exists 2024-09-11T03:50:19.172 INFO:tasks.rados.rados.2.smithi175.stdout:3294: finishing copy_from to smithi17557870-29 2024-09-11T03:50:19.172 INFO:tasks.rados.rados.2.smithi175.stdout:3294: got expected ENOENT (src dne) 2024-09-11T03:50:19.172 INFO:tasks.rados.rados.2.smithi175.stdout:3294: finishing copy_from racing read to smithi17557870-29 2024-09-11T03:50:19.178 INFO:tasks.rados.rados.2.smithi175.stdout:3297: expect (ObjNum 970 snap 311 seq_num 970) 2024-09-11T03:50:19.224 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:19.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:19.247 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:19.247 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:19.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:19.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:19.370 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:19.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:19.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:19 smithi084 ceph-mon[27254]: 2.35 scrub starts 2024-09-11T03:50:19.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:19 smithi084 ceph-mon[27254]: 2.35 scrub ok 2024-09-11T03:50:19.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:19 smithi084 ceph-mon[27254]: 2.37 scrub starts 2024-09-11T03:50:19.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:19 smithi084 ceph-mon[27254]: 2.37 scrub ok 2024-09-11T03:50:19.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:19 smithi084 ceph-mon[27254]: 2.33 scrub starts 2024-09-11T03:50:19.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:19 smithi084 ceph-mon[27254]: 2.33 scrub ok 2024-09-11T03:50:19.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:19 smithi084 ceph-mon[27254]: 2.31 scrub starts 2024-09-11T03:50:19.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:19 smithi084 ceph-mon[27254]: 2.31 scrub ok 2024-09-11T03:50:19.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:19 smithi084 ceph-mon[27254]: 2.2f scrub starts 2024-09-11T03:50:19.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:19 smithi084 ceph-mon[27254]: 2.2f scrub ok 2024-09-11T03:50:19.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:19 smithi084 ceph-mon[27254]: osdmap e800: 12 total, 12 up, 12 in 2024-09-11T03:50:19.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:19 smithi084 ceph-mon[27254]: 1.b scrub starts 2024-09-11T03:50:19.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:19 smithi084 ceph-mon[27254]: 1.b scrub ok 2024-09-11T03:50:19.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:19 smithi084 ceph-mon[27254]: 2.36 scrub starts 2024-09-11T03:50:19.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:19 smithi084 ceph-mon[27254]: 2.36 scrub ok 2024-09-11T03:50:19.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:19 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2499851111' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:19 smithi121 ceph-mon[30729]: 2.35 scrub starts 2024-09-11T03:50:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:19 smithi121 ceph-mon[30729]: 2.35 scrub ok 2024-09-11T03:50:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:19 smithi121 ceph-mon[30729]: 2.37 scrub starts 2024-09-11T03:50:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:19 smithi121 ceph-mon[30729]: 2.37 scrub ok 2024-09-11T03:50:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:19 smithi121 ceph-mon[30729]: 2.33 scrub starts 2024-09-11T03:50:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:19 smithi121 ceph-mon[30729]: 2.33 scrub ok 2024-09-11T03:50:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:19 smithi121 ceph-mon[30729]: 2.31 scrub starts 2024-09-11T03:50:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:19 smithi121 ceph-mon[30729]: 2.31 scrub ok 2024-09-11T03:50:19.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:19 smithi121 ceph-mon[30729]: 2.2f scrub starts 2024-09-11T03:50:19.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:19 smithi121 ceph-mon[30729]: 2.2f scrub ok 2024-09-11T03:50:19.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:19 smithi121 ceph-mon[30729]: osdmap e800: 12 total, 12 up, 12 in 2024-09-11T03:50:19.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:19 smithi121 ceph-mon[30729]: 1.b scrub starts 2024-09-11T03:50:19.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:19 smithi121 ceph-mon[30729]: 1.b scrub ok 2024-09-11T03:50:19.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:19 smithi121 ceph-mon[30729]: 2.36 scrub starts 2024-09-11T03:50:19.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:19 smithi121 ceph-mon[30729]: 2.36 scrub ok 2024-09-11T03:50:19.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:19 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2499851111' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:19.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:19 smithi175 ceph-mon[30894]: 2.35 scrub starts 2024-09-11T03:50:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:19 smithi175 ceph-mon[30894]: 2.35 scrub ok 2024-09-11T03:50:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:19 smithi175 ceph-mon[30894]: 2.37 scrub starts 2024-09-11T03:50:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:19 smithi175 ceph-mon[30894]: 2.37 scrub ok 2024-09-11T03:50:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:19 smithi175 ceph-mon[30894]: 2.33 scrub starts 2024-09-11T03:50:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:19 smithi175 ceph-mon[30894]: 2.33 scrub ok 2024-09-11T03:50:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:19 smithi175 ceph-mon[30894]: 2.31 scrub starts 2024-09-11T03:50:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:19 smithi175 ceph-mon[30894]: 2.31 scrub ok 2024-09-11T03:50:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:19 smithi175 ceph-mon[30894]: 2.2f scrub starts 2024-09-11T03:50:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:19 smithi175 ceph-mon[30894]: 2.2f scrub ok 2024-09-11T03:50:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:19 smithi175 ceph-mon[30894]: osdmap e800: 12 total, 12 up, 12 in 2024-09-11T03:50:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:19 smithi175 ceph-mon[30894]: 1.b scrub starts 2024-09-11T03:50:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:19 smithi175 ceph-mon[30894]: 1.b scrub ok 2024-09-11T03:50:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:19 smithi175 ceph-mon[30894]: 2.36 scrub starts 2024-09-11T03:50:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:19 smithi175 ceph-mon[30894]: 2.36 scrub ok 2024-09-11T03:50:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:19 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2499851111' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:19.493 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:19.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:19.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:19.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:19.615 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:19.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:19.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:19.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T03:50:19.718 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:19.740 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:19.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:19.741 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:19.842 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:19.866 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:19.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:19.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:19.920 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:50:19.967 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:19.990 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:19.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:19.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:20.092 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:20.113 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:50:20.115 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:20.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:20.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:20.168 INFO:tasks.rados.rados.2.smithi175.stdout:3294: done (3 left) 2024-09-11T03:50:20.168 INFO:tasks.rados.rados.2.smithi175.stdout:3295: done (2 left) 2024-09-11T03:50:20.168 INFO:tasks.rados.rados.2.smithi175.stdout:3297: done (1 left) 2024-09-11T03:50:20.168 INFO:tasks.rados.rados.2.smithi175.stdout:3298: done (0 left) 2024-09-11T03:50:20.169 INFO:tasks.rados.rados.2.smithi175.stdout:3299: copy_from oid 14 from oid 10 current snap is 362 2024-09-11T03:50:20.169 INFO:tasks.rados.rados.2.smithi175.stdout:3300: snap_create 2024-09-11T03:50:20.179 INFO:tasks.rados.rados.2.smithi175.stdout:3299: finishing copy_from racing read to smithi17557870-14 2024-09-11T03:50:20.183 INFO:tasks.rados.rados.2.smithi175.stdout:3299: finishing copy_from to smithi17557870-14 2024-09-11T03:50:20.183 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 1045 (ObjNum 1075 snap 350 seq_num 1075) dirty exists 2024-09-11T03:50:20.218 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:20.239 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:20.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:20.313 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:20 smithi121 ceph-mon[30729]: 1.1 scrub starts 2024-09-11T03:50:20.313 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:20 smithi121 ceph-mon[30729]: 1.1 scrub ok 2024-09-11T03:50:20.313 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:20 smithi121 ceph-mon[30729]: 2.2e scrub starts 2024-09-11T03:50:20.314 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:20 smithi121 ceph-mon[30729]: 2.2e scrub ok 2024-09-11T03:50:20.314 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:20 smithi121 ceph-mon[30729]: pgmap v1356: 87 pgs: 1 active+clean+snaptrim, 4 active+clean+scrubbing, 82 active+clean; 514 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 0 B/s wr, 8 op/s 2024-09-11T03:50:20.314 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:20 smithi121 ceph-mon[30729]: 1.a scrub starts 2024-09-11T03:50:20.314 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:20 smithi121 ceph-mon[30729]: 1.a scrub ok 2024-09-11T03:50:20.314 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:20 smithi121 ceph-mon[30729]: osdmap e801: 12 total, 12 up, 12 in 2024-09-11T03:50:20.340 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:20.363 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:20.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:20.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:20.385 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:20 smithi084 ceph-mon[27254]: 1.1 scrub starts 2024-09-11T03:50:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:20 smithi084 ceph-mon[27254]: 1.1 scrub ok 2024-09-11T03:50:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:20 smithi084 ceph-mon[27254]: 2.2e scrub starts 2024-09-11T03:50:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:20 smithi084 ceph-mon[27254]: 2.2e scrub ok 2024-09-11T03:50:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:20 smithi084 ceph-mon[27254]: pgmap v1356: 87 pgs: 1 active+clean+snaptrim, 4 active+clean+scrubbing, 82 active+clean; 514 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 0 B/s wr, 8 op/s 2024-09-11T03:50:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:20 smithi084 ceph-mon[27254]: 1.a scrub starts 2024-09-11T03:50:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:20 smithi084 ceph-mon[27254]: 1.a scrub ok 2024-09-11T03:50:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:20 smithi084 ceph-mon[27254]: osdmap e801: 12 total, 12 up, 12 in 2024-09-11T03:50:20.430 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:50:20.430 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:50:20.430 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:50:20.430 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:50:20.465 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:20 smithi175 ceph-mon[30894]: 1.1 scrub starts 2024-09-11T03:50:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:20 smithi175 ceph-mon[30894]: 1.1 scrub ok 2024-09-11T03:50:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:20 smithi175 ceph-mon[30894]: 2.2e scrub starts 2024-09-11T03:50:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:20 smithi175 ceph-mon[30894]: 2.2e scrub ok 2024-09-11T03:50:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:20 smithi175 ceph-mon[30894]: pgmap v1356: 87 pgs: 1 active+clean+snaptrim, 4 active+clean+scrubbing, 82 active+clean; 514 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 0 B/s wr, 8 op/s 2024-09-11T03:50:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:20 smithi175 ceph-mon[30894]: 1.a scrub starts 2024-09-11T03:50:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:20 smithi175 ceph-mon[30894]: 1.a scrub ok 2024-09-11T03:50:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:20 smithi175 ceph-mon[30894]: osdmap e801: 12 total, 12 up, 12 in 2024-09-11T03:50:20.486 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:20.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:20.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:20.588 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:20.609 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:20.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:20.611 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:20.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:20.734 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:20.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:20.837 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:20.858 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:20.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:20.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:20.960 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:20.982 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:20.983 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:21.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T03:50:21.084 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:21.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:21.108 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:21.109 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:21.170 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:50:21.174 INFO:tasks.rados.rados.2.smithi175.stdout:3299: done (1 left) 2024-09-11T03:50:21.174 INFO:tasks.rados.rados.2.smithi175.stdout:3300: done (0 left) 2024-09-11T03:50:21.174 INFO:tasks.rados.rados.2.smithi175.stdout:3301: delete oid 41 current snap is 363 2024-09-11T03:50:21.181 INFO:tasks.rados.rados.2.smithi175.stdout:3301: done (0 left) 2024-09-11T03:50:21.181 INFO:tasks.rados.rados.2.smithi175.stdout:3302: rollback oid 11 current snap is 363 2024-09-11T03:50:21.181 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 11 to 335 2024-09-11T03:50:21.181 INFO:tasks.rados.rados.2.smithi175.stdout:3303: snap_create 2024-09-11T03:50:21.186 INFO:tasks.rados.rados.2.smithi175.stdout:3302: finishing rollback tid 0 to smithi17557870-11 2024-09-11T03:50:21.186 INFO:tasks.rados.rados.2.smithi175.stdout:3302: finishing rollback tid 1 to smithi17557870-11 2024-09-11T03:50:21.186 INFO:tasks.rados.rados.2.smithi175.stdout:3302: finishing rollback tid 2 to smithi17557870-11 2024-09-11T03:50:21.186 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 630 (ObjNum 1007 snap 324 seq_num 1007) dirty exists 2024-09-11T03:50:21.210 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:21.231 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:21.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:21.289 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:50:21.333 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:21.355 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:21.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:21.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:21.409 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:21 smithi121 ceph-mon[30729]: 2.23 scrub starts 2024-09-11T03:50:21.409 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:21 smithi121 ceph-mon[30729]: 2.23 scrub ok 2024-09-11T03:50:21.409 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:21 smithi121 ceph-mon[30729]: 1.15 scrub starts 2024-09-11T03:50:21.409 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:21 smithi121 ceph-mon[30729]: 1.15 scrub ok 2024-09-11T03:50:21.409 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:21 smithi121 ceph-mon[30729]: osdmap e802: 12 total, 12 up, 12 in 2024-09-11T03:50:21.409 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:50:21.409 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:50:21.410 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:50:21.410 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:50:21.410 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:21 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:50:21.410 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:21 smithi121 ceph-mon[30729]: 2.2 scrub ok 2024-09-11T03:50:21.410 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:50:21.410 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:50:21.410 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:50:21.410 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:21 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2183696085' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:21.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:21.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:21 smithi084 ceph-mon[27254]: 2.23 scrub starts 2024-09-11T03:50:21.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:21 smithi084 ceph-mon[27254]: 2.23 scrub ok 2024-09-11T03:50:21.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:21 smithi084 ceph-mon[27254]: 1.15 scrub starts 2024-09-11T03:50:21.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:21 smithi084 ceph-mon[27254]: 1.15 scrub ok 2024-09-11T03:50:21.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:21 smithi084 ceph-mon[27254]: osdmap e802: 12 total, 12 up, 12 in 2024-09-11T03:50:21.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:50:21.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:50:21.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:50:21.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:50:21.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:21 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:50:21.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:21 smithi084 ceph-mon[27254]: 2.2 scrub ok 2024-09-11T03:50:21.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:50:21.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:50:21.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:50:21.476 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2183696085' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:21.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:21 smithi175 ceph-mon[30894]: 2.23 scrub starts 2024-09-11T03:50:21.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:21 smithi175 ceph-mon[30894]: 2.23 scrub ok 2024-09-11T03:50:21.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:21 smithi175 ceph-mon[30894]: 1.15 scrub starts 2024-09-11T03:50:21.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:21 smithi175 ceph-mon[30894]: 1.15 scrub ok 2024-09-11T03:50:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:21 smithi175 ceph-mon[30894]: osdmap e802: 12 total, 12 up, 12 in 2024-09-11T03:50:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:50:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:50:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:50:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:50:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:21 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:50:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:21 smithi175 ceph-mon[30894]: 2.2 scrub ok 2024-09-11T03:50:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:50:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:50:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:50:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:21 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2183696085' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:21.479 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:21.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:21.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:50:21.581 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:21.604 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:21.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:21.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:21.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:21.729 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:21.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:21.744 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:50:21.744 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:50:21.744 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:50:21.744 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:50:21.744 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:50:21.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:21.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:21.854 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:21.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:21.954 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:21.981 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:21.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:21.982 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:22.033 INFO:tasks.rados.rados.2.smithi175.stdout:3302: done (1 left) 2024-09-11T03:50:22.034 INFO:tasks.rados.rados.2.smithi175.stdout:3303: done (0 left) 2024-09-11T03:50:22.034 INFO:tasks.rados.rados.2.smithi175.stdout:3304: write oid 18 current snap is 364 2024-09-11T03:50:22.034 INFO:tasks.rados.rados.2.smithi175.stdout:3304: seq_num 1124 ranges {491068=483767,1640584=593387,2650385=240683} 2024-09-11T03:50:22.038 INFO:tasks.rados.rados.2.smithi175.stdout:3304: writing smithi17557870-18 from 491068 to 974835 tid 1 2024-09-11T03:50:22.046 INFO:tasks.rados.rados.2.smithi175.stdout:3304: writing smithi17557870-18 from 1640584 to 2233971 tid 2 2024-09-11T03:50:22.050 INFO:tasks.rados.rados.2.smithi175.stdout:3304: writing smithi17557870-18 from 2650385 to 2891068 tid 3 2024-09-11T03:50:22.050 INFO:tasks.rados.rados.2.smithi175.stdout:3305: snap_remove snap 350 2024-09-11T03:50:22.050 INFO:tasks.rados.rados.2.smithi175.stdout:3304: finishing write tid 1 to smithi17557870-18 2024-09-11T03:50:22.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:22.104 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:22.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:22.116 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:22.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:22.226 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:22.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:22.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:22.329 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:22.355 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:22.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:22.356 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:22.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:22 smithi084 ceph-mon[27254]: 1.14 scrub starts 2024-09-11T03:50:22.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:22 smithi084 ceph-mon[27254]: 1.14 scrub ok 2024-09-11T03:50:22.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:22 smithi084 ceph-mon[27254]: pgmap v1359: 87 pgs: 1 active+clean+snaptrim, 4 active+clean+scrubbing, 82 active+clean; 514 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:22.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:22 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:50:22.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:22 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:50:22.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:22 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2183696085' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:22.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:22 smithi084 ceph-mon[27254]: osdmap e803: 12 total, 12 up, 12 in 2024-09-11T03:50:22.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:22 smithi084 ceph-mon[27254]: osdmap e804: 12 total, 12 up, 12 in 2024-09-11T03:50:22.418 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:50:22.418 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:50:22.418 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-11T03:50:22.419 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:50:22.427 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T03:50:22.449 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:22 smithi121 ceph-mon[30729]: 1.14 scrub starts 2024-09-11T03:50:22.449 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:22 smithi121 ceph-mon[30729]: 1.14 scrub ok 2024-09-11T03:50:22.449 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:22 smithi121 ceph-mon[30729]: pgmap v1359: 87 pgs: 1 active+clean+snaptrim, 4 active+clean+scrubbing, 82 active+clean; 514 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:22.450 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:22 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:50:22.450 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:22 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:50:22.450 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:22 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2183696085' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:22.450 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:22 smithi121 ceph-mon[30729]: osdmap e803: 12 total, 12 up, 12 in 2024-09-11T03:50:22.450 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:22 smithi121 ceph-mon[30729]: osdmap e804: 12 total, 12 up, 12 in 2024-09-11T03:50:22.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:22.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:22 smithi175 ceph-mon[30894]: 1.14 scrub starts 2024-09-11T03:50:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:22 smithi175 ceph-mon[30894]: 1.14 scrub ok 2024-09-11T03:50:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:22 smithi175 ceph-mon[30894]: pgmap v1359: 87 pgs: 1 active+clean+snaptrim, 4 active+clean+scrubbing, 82 active+clean; 514 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:22 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:50:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:22 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:50:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:22 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2183696085' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:22 smithi175 ceph-mon[30894]: osdmap e803: 12 total, 12 up, 12 in 2024-09-11T03:50:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:22 smithi175 ceph-mon[30894]: osdmap e804: 12 total, 12 up, 12 in 2024-09-11T03:50:22.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:22.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:22.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:22.601 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:22.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:22.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:22.623 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:50:22.623 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:50:22.644 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:50:22.645 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:50:22.666 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:50:22.666 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:50:22.672 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:50:22.691 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:22.703 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:22.726 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:22.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:22.827 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:22.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:22.849 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:22.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:22.885 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:50:22.885 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:50:22.901 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:50:22.902 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:50:22.918 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:50:22.919 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:50:22.935 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:50:22.935 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:50:22.950 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:22.952 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:50:22.953 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:50:22.972 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:22.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:22.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:22.974 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:50:22.974 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:50:22.991 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:50:22.991 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:50:23.040 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:50:23.040 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:50:23.044 INFO:tasks.rados.rados.2.smithi175.stdout:3305: done (1 left) 2024-09-11T03:50:23.044 INFO:tasks.rados.rados.2.smithi175.stdout:3306: copy_from oid 42 from oid 24 current snap is 364 2024-09-11T03:50:23.044 INFO:tasks.rados.rados.2.smithi175.stdout:3307: snap_remove snap 296 2024-09-11T03:50:23.066 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:50:23.066 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:50:23.075 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:23.098 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:23.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:23.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:23.172 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:50:23.172 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:50:23.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:23.220 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:23.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:23.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:23.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:23.343 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:23.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:23.359 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:50:23.360 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:50:23.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:23 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4123455973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:23 smithi121 ceph-mon[30729]: osdmap e805: 12 total, 12 up, 12 in 2024-09-11T03:50:23.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:23.466 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:23.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:23.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:23.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:23 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4123455973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:23.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:23 smithi175 ceph-mon[30894]: osdmap e805: 12 total, 12 up, 12 in 2024-09-11T03:50:23.568 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:23.590 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:23.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:23.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:23.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:23 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4123455973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:23.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:23 smithi084 ceph-mon[27254]: osdmap e805: 12 total, 12 up, 12 in 2024-09-11T03:50:23.692 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:23.714 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:23.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:23.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:23.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:23.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T03:50:23.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:23.838 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:23.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:23.939 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:23.960 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:23.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:24.048 INFO:tasks.rados.rados.2.smithi175.stdout:3307: done (2 left) 2024-09-11T03:50:24.048 INFO:tasks.rados.rados.2.smithi175.stdout:3308: write oid 49 current snap is 364 2024-09-11T03:50:24.048 INFO:tasks.rados.rados.2.smithi175.stdout:3308: seq_num 1125 ranges {507875=559016,1699066=737045} 2024-09-11T03:50:24.054 INFO:tasks.rados.rados.2.smithi175.stdout:3308: writing smithi17557870-49 from 507875 to 1066891 tid 1 2024-09-11T03:50:24.061 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:24.063 INFO:tasks.rados.rados.2.smithi175.stdout:3308: writing smithi17557870-49 from 1699066 to 2436111 tid 2 2024-09-11T03:50:24.063 INFO:tasks.rados.rados.2.smithi175.stdout:3309: snap_create 2024-09-11T03:50:24.063 INFO:tasks.rados.rados.2.smithi175.stdout:3304: finishing write tid 2 to smithi17557870-18 2024-09-11T03:50:24.063 INFO:tasks.rados.rados.2.smithi175.stdout:3304: finishing write tid 3 to smithi17557870-18 2024-09-11T03:50:24.063 INFO:tasks.rados.rados.2.smithi175.stdout:3304: finishing write tid 4 to smithi17557870-18 2024-09-11T03:50:24.064 INFO:tasks.rados.rados.2.smithi175.stdout:3304: finishing write tid 5 to smithi17557870-18 2024-09-11T03:50:24.064 INFO:tasks.rados.rados.2.smithi175.stdout:3304: oid 18 updating version 0 to 934 2024-09-11T03:50:24.064 INFO:tasks.rados.rados.2.smithi175.stdout:3304: oid 18 updating version 934 to 935 2024-09-11T03:50:24.064 INFO:tasks.rados.rados.2.smithi175.stdout:3304: oid 18 version 935 is already newer than 932 2024-09-11T03:50:24.064 INFO:tasks.rados.rados.2.smithi175.stdout:3304: oid 18 version 935 is already newer than 933 2024-09-11T03:50:24.064 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 935 (ObjNum 1124 snap 364 seq_num 1124) dirty exists 2024-09-11T03:50:24.064 INFO:tasks.rados.rados.2.smithi175.stdout:3304: left oid 18 (ObjNum 1124 snap 364 seq_num 1124) 2024-09-11T03:50:24.064 INFO:tasks.rados.rados.2.smithi175.stdout:3306: finishing copy_from racing read to smithi17557870-42 2024-09-11T03:50:24.065 INFO:tasks.rados.rados.2.smithi175.stdout:3306: finishing copy_from to smithi17557870-42 2024-09-11T03:50:24.065 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 1053 (ObjNum 1120 snap 359 seq_num 1120) dirty exists 2024-09-11T03:50:24.065 INFO:tasks.rados.rados.2.smithi175.stdout:3308: finishing write tid 1 to smithi17557870-49 2024-09-11T03:50:24.066 INFO:tasks.rados.rados.2.smithi175.stdout:3308: finishing write tid 2 to smithi17557870-49 2024-09-11T03:50:24.066 INFO:tasks.rados.rados.2.smithi175.stdout:3308: finishing write tid 3 to smithi17557870-49 2024-09-11T03:50:24.066 INFO:tasks.rados.rados.2.smithi175.stdout:3308: finishing write tid 4 to smithi17557870-49 2024-09-11T03:50:24.067 INFO:tasks.rados.rados.2.smithi175.stdout:3308: oid 49 updating version 0 to 998 2024-09-11T03:50:24.067 INFO:tasks.rados.rados.2.smithi175.stdout:3308: oid 49 version 998 is already newer than 997 2024-09-11T03:50:24.067 INFO:tasks.rados.rados.2.smithi175.stdout:3308: oid 49 version 998 is already newer than 996 2024-09-11T03:50:24.067 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 998 (ObjNum 1125 snap 364 seq_num 1125) dirty exists 2024-09-11T03:50:24.067 INFO:tasks.rados.rados.2.smithi175.stdout:3308: left oid 49 (ObjNum 1125 snap 364 seq_num 1125) 2024-09-11T03:50:24.078 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:50:24.085 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:24.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:24.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:24.087 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:50:24.162 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1363,"stamp":"2024-09-11T03:50:23.040774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":569688000,"num_objects":271,"num_object_clones":219,"num_object_copies":813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":271,"num_whiteouts":20,"num_read":1187,"num_read_kb":1273784,"num_write":775,"num_write_kb":338935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":10064265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34150,"ondisk_log_size":34150,"up":261,"acting":261,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":261,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":1588416,"kb_used_data":1071148,"kb_used_omap":18,"kb_used_meta":516973,"kb_avail":1123353408,"statfs":{"total":1151940427776,"available":1150313889792,"internally_reserved":0,"allocated":1096855552,"data_stored":1426946552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":529380667},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31155606,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":-3,"num_read":-272,"num_read_kb":-251133,"num_write":-232,"num_write_kb":-86412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-167,"num_bytes_recovered":-46273688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.875841"},"pg_stats":[{"pgid":"2.37","version":"274'312","reported_seq":2179,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287461+0000","last_change":"2024-09-11T03:50:19.417544+0000","last_active":"2024-09-11T03:50:21.287461+0000","last_peered":"2024-09-11T03:50:21.287461+0000","last_clean":"2024-09-11T03:50:21.287461+0000","last_became_active":"2024-09-11T03:50:07.077607+0000","last_became_peered":"2024-09-11T03:50:07.077607+0000","last_unstale":"2024-09-11T03:50:21.287461+0000","last_undegraded":"2024-09-11T03:50:21.287461+0000","last_fullsized":"2024-09-11T03:50:21.287461+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:50:17.684567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.684567+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:18.728668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097237799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,9],"acting":[10,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.36","version":"799'589","reported_seq":4223,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.173269+0000","last_change":"2024-09-11T03:50:19.309303+0000","last_active":"2024-09-11T03:50:21.173269+0000","last_peered":"2024-09-11T03:50:21.173269+0000","last_clean":"2024-09-11T03:50:21.173269+0000","last_became_active":"2024-09-11T03:50:07.078667+0000","last_became_peered":"2024-09-11T03:50:07.078667+0000","last_unstale":"2024-09-11T03:50:21.173269+0000","last_undegraded":"2024-09-11T03:50:21.173269+0000","last_fullsized":"2024-09-11T03:50:21.173269+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'589","last_scrub_stamp":"2024-09-11T03:50:18.280936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.280936+0000","objects_scrubbed":12,"log_size":589,"log_dups_size":0,"ondisk_log_size":589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:41.309150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00076371000000000002,"stat_sum":{"num_bytes":26396466,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,8],"acting":[1,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.35","version":"274'273","reported_seq":1347,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.173003+0000","last_change":"2024-09-11T03:50:19.309028+0000","last_active":"2024-09-11T03:50:21.173003+0000","last_peered":"2024-09-11T03:50:21.173003+0000","last_clean":"2024-09-11T03:50:21.173003+0000","last_became_active":"2024-09-11T03:50:07.079331+0000","last_became_peered":"2024-09-11T03:50:07.079331+0000","last_unstale":"2024-09-11T03:50:21.173003+0000","last_undegraded":"2024-09-11T03:50:21.173003+0000","last_fullsized":"2024-09-11T03:50:21.173003+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:50:17.314165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.314165+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:29.465470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054737799999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.34","version":"799'358","reported_seq":3927,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.184496+0000","last_change":"2024-09-11T03:50:19.166686+0000","last_active":"2024-09-11T03:50:21.184496+0000","last_peered":"2024-09-11T03:50:21.184496+0000","last_clean":"2024-09-11T03:50:21.184496+0000","last_became_active":"2024-09-11T03:50:07.080419+0000","last_became_peered":"2024-09-11T03:50:07.080419+0000","last_unstale":"2024-09-11T03:50:21.184496+0000","last_undegraded":"2024-09-11T03:50:21.184496+0000","last_fullsized":"2024-09-11T03:50:21.184496+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'358","last_scrub_stamp":"2024-09-11T03:50:17.613025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.613025+0000","objects_scrubbed":4,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:27.088723+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00063675199999999998,"stat_sum":{"num_bytes":9118212,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.33","version":"274'111","reported_seq":1135,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.182768+0000","last_change":"2024-09-11T03:50:19.416923+0000","last_active":"2024-09-11T03:50:21.182768+0000","last_peered":"2024-09-11T03:50:21.182768+0000","last_clean":"2024-09-11T03:50:21.182768+0000","last_became_active":"2024-09-11T03:50:07.077684+0000","last_became_peered":"2024-09-11T03:50:07.077684+0000","last_unstale":"2024-09-11T03:50:21.182768+0000","last_undegraded":"2024-09-11T03:50:21.182768+0000","last_fullsized":"2024-09-11T03:50:21.182768+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:50:17.796171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.796171+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:19.981309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045266000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.32","version":"778'549","reported_seq":4108,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.180052+0000","last_change":"2024-09-11T03:50:19.417396+0000","last_active":"2024-09-11T03:50:21.180052+0000","last_peered":"2024-09-11T03:50:21.180052+0000","last_clean":"2024-09-11T03:50:21.180052+0000","last_became_active":"2024-09-11T03:50:07.077594+0000","last_became_peered":"2024-09-11T03:50:07.077594+0000","last_unstale":"2024-09-11T03:50:21.180052+0000","last_undegraded":"2024-09-11T03:50:21.180052+0000","last_fullsized":"2024-09-11T03:50:21.180052+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"778'549","last_scrub_stamp":"2024-09-11T03:50:17.199232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.199232+0000","objects_scrubbed":2,"log_size":549,"log_dups_size":0,"ondisk_log_size":549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:40.829017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013741440000000001,"stat_sum":{"num_bytes":5630774,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9],"acting":[11,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.31","version":"274'107","reported_seq":1622,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-09-11T03:50:20.171030+0000","last_change":"2024-09-11T03:50:19.416119+0000","last_active":"2024-09-11T03:50:20.171030+0000","last_peered":"2024-09-11T03:50:20.171030+0000","last_clean":"2024-09-11T03:50:20.171030+0000","last_became_active":"2024-09-11T03:50:07.075956+0000","last_became_peered":"2024-09-11T03:50:07.075956+0000","last_unstale":"2024-09-11T03:50:20.171030+0000","last_undegraded":"2024-09-11T03:50:20.171030+0000","last_fullsized":"2024-09-11T03:50:20.171030+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:50:17.857515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.857515+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:27.784194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047599900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,4],"acting":[0,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.30","version":"274'67","reported_seq":100,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287083+0000","last_change":"2024-09-11T03:50:19.417685+0000","last_active":"2024-09-11T03:50:21.287083+0000","last_peered":"2024-09-11T03:50:21.287083+0000","last_clean":"2024-09-11T03:50:21.287083+0000","last_became_active":"2024-09-11T03:50:04.069393+0000","last_became_peered":"2024-09-11T03:50:04.069393+0000","last_unstale":"2024-09-11T03:50:21.287083+0000","last_undegraded":"2024-09-11T03:50:21.287083+0000","last_fullsized":"2024-09-11T03:50:21.287083+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:50:04.608333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.608333+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:33.603451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095874999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,9],"acting":[10,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.2f","version":"557'465","reported_seq":2932,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-11T03:50:18.276045+0000","last_change":"2024-09-11T03:50:17.948611+0000","last_active":"2024-09-11T03:50:18.276045+0000","last_peered":"2024-09-11T03:50:18.276045+0000","last_clean":"2024-09-11T03:50:18.276045+0000","last_became_active":"2024-09-11T03:50:07.077840+0000","last_became_peered":"2024-09-11T03:50:07.077840+0000","last_unstale":"2024-09-11T03:50:18.276045+0000","last_undegraded":"2024-09-11T03:50:18.276045+0000","last_fullsized":"2024-09-11T03:50:18.276045+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:50:17.948546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.948546+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:31.315764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0021351059999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"}]},{"pgid":"2.2e","version":"799'1047","reported_seq":4566,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.183309+0000","last_change":"2024-09-11T03:50:19.417453+0000","last_active":"2024-09-11T03:50:21.183309+0000","last_peered":"2024-09-11T03:50:21.183309+0000","last_clean":"2024-09-11T03:50:21.183309+0000","last_became_active":"2024-09-11T03:50:07.078910+0000","last_became_peered":"2024-09-11T03:50:07.078910+0000","last_unstale":"2024-09-11T03:50:21.183309+0000","last_undegraded":"2024-09-11T03:50:21.183309+0000","last_fullsized":"2024-09-11T03:50:21.183309+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'1047","last_scrub_stamp":"2024-09-11T03:50:18.779326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.779326+0000","objects_scrubbed":11,"log_size":1047,"log_dups_size":0,"ondisk_log_size":1047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:19.120742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000623822,"stat_sum":{"num_bytes":23972091,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.2d","version":"799'1085","reported_seq":4655,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287638+0000","last_change":"2024-09-11T03:50:19.311426+0000","last_active":"2024-09-11T03:50:21.287638+0000","last_peered":"2024-09-11T03:50:21.287638+0000","last_clean":"2024-09-11T03:50:21.287638+0000","last_became_active":"2024-09-11T03:49:29.163201+0000","last_became_peered":"2024-09-11T03:49:29.163201+0000","last_unstale":"2024-09-11T03:50:21.287638+0000","last_undegraded":"2024-09-11T03:50:21.287638+0000","last_fullsized":"2024-09-11T03:50:21.287638+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'997","last_scrub_stamp":"2024-09-11T03:49:32.462332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.462332+0000","objects_scrubbed":13,"log_size":1085,"log_dups_size":0,"ondisk_log_size":1085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:00.645476+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001090843,"stat_sum":{"num_bytes":38880454,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":41,"num_read_kb":38985,"num_write":10,"num_write_kb":10705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.2c","version":"799'854","reported_seq":4418,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.173756+0000","last_change":"2024-09-11T03:50:19.175448+0000","last_active":"2024-09-11T03:50:21.173756+0000","last_peered":"2024-09-11T03:50:21.173756+0000","last_clean":"2024-09-11T03:50:21.173756+0000","last_became_active":"2024-09-11T03:49:29.154872+0000","last_became_peered":"2024-09-11T03:49:29.154872+0000","last_unstale":"2024-09-11T03:50:21.173756+0000","last_undegraded":"2024-09-11T03:50:21.173756+0000","last_fullsized":"2024-09-11T03:50:21.173756+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"753'800","last_scrub_stamp":"2024-09-11T03:49:32.185401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.185401+0000","objects_scrubbed":4,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:01.433375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001012527,"stat_sum":{"num_bytes":21131325,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":25,"num_read_kb":19599,"num_write":25,"num_write_kb":9786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3],"acting":[8,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.2b","version":"568'646","reported_seq":3327,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-11T03:50:18.275932+0000","last_change":"2024-09-11T03:50:17.045223+0000","last_active":"2024-09-11T03:50:18.275932+0000","last_peered":"2024-09-11T03:50:18.275932+0000","last_clean":"2024-09-11T03:50:18.275932+0000","last_became_active":"2024-09-11T03:49:29.157735+0000","last_became_peered":"2024-09-11T03:49:29.157735+0000","last_unstale":"2024-09-11T03:50:18.275932+0000","last_undegraded":"2024-09-11T03:50:18.275932+0000","last_fullsized":"2024-09-11T03:50:18.275932+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:49:32.876969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.876969+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:40.253841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001226417,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"}]},{"pgid":"2.2a","version":"727'348","reported_seq":3823,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.182757+0000","last_change":"2024-09-11T03:50:19.416439+0000","last_active":"2024-09-11T03:50:21.182757+0000","last_peered":"2024-09-11T03:50:21.182757+0000","last_clean":"2024-09-11T03:50:21.182757+0000","last_became_active":"2024-09-11T03:49:29.160595+0000","last_became_peered":"2024-09-11T03:49:29.160595+0000","last_unstale":"2024-09-11T03:50:21.182757+0000","last_undegraded":"2024-09-11T03:50:21.182757+0000","last_fullsized":"2024-09-11T03:50:21.182757+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:49:34.219244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.219244+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:40.157227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045680999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,5],"acting":[7,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.29","version":"799'1239","reported_seq":4913,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287072+0000","last_change":"2024-09-11T03:50:19.310777+0000","last_active":"2024-09-11T03:50:21.287072+0000","last_peered":"2024-09-11T03:50:21.287072+0000","last_clean":"2024-09-11T03:50:21.287072+0000","last_became_active":"2024-09-11T03:49:29.157194+0000","last_became_peered":"2024-09-11T03:49:29.157194+0000","last_unstale":"2024-09-11T03:50:21.287072+0000","last_undegraded":"2024-09-11T03:50:21.287072+0000","last_fullsized":"2024-09-11T03:50:21.287072+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'1167","last_scrub_stamp":"2024-09-11T03:49:33.506786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.506786+0000","objects_scrubbed":11,"log_size":1239,"log_dups_size":0,"ondisk_log_size":1239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:37.068001+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00083357899999999998,"stat_sum":{"num_bytes":33520458,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":26,"num_read_kb":31567,"num_write":21,"num_write_kb":8705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8],"acting":[6,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.28","version":"795'402","reported_seq":3980,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.172648+0000","last_change":"2024-09-11T03:50:19.308632+0000","last_active":"2024-09-11T03:50:21.172648+0000","last_peered":"2024-09-11T03:50:21.172648+0000","last_clean":"2024-09-11T03:50:21.172648+0000","last_became_active":"2024-09-11T03:49:29.154884+0000","last_became_peered":"2024-09-11T03:49:29.154884+0000","last_unstale":"2024-09-11T03:50:21.172648+0000","last_undegraded":"2024-09-11T03:50:21.172648+0000","last_fullsized":"2024-09-11T03:50:21.172648+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'382","last_scrub_stamp":"2024-09-11T03:49:32.335285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.335285+0000","objects_scrubbed":4,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:52.938232+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00049171100000000004,"stat_sum":{"num_bytes":8486578,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":9,"num_read_kb":11526,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.27","version":"799'1111","reported_seq":4793,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.183207+0000","last_change":"2024-09-11T03:50:19.416946+0000","last_active":"2024-09-11T03:50:21.183207+0000","last_peered":"2024-09-11T03:50:21.183207+0000","last_clean":"2024-09-11T03:50:21.183207+0000","last_became_active":"2024-09-11T03:49:29.159799+0000","last_became_peered":"2024-09-11T03:49:29.159799+0000","last_unstale":"2024-09-11T03:50:21.183207+0000","last_undegraded":"2024-09-11T03:50:21.183207+0000","last_fullsized":"2024-09-11T03:50:21.183207+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"757'1065","last_scrub_stamp":"2024-09-11T03:49:36.196385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.196385+0000","objects_scrubbed":4,"log_size":1111,"log_dups_size":0,"ondisk_log_size":1111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:59.011808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00042269999999999997,"stat_sum":{"num_bytes":18522601,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":14,"num_read_kb":13448,"num_write":8,"num_write_kb":7771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,9],"acting":[7,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.26","version":"269'171","reported_seq":3708,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.183103+0000","last_change":"2024-09-11T03:50:19.417148+0000","last_active":"2024-09-11T03:50:21.183103+0000","last_peered":"2024-09-11T03:50:21.183103+0000","last_clean":"2024-09-11T03:50:21.183103+0000","last_became_active":"2024-09-11T03:49:29.158577+0000","last_became_peered":"2024-09-11T03:49:29.158577+0000","last_unstale":"2024-09-11T03:50:21.183103+0000","last_undegraded":"2024-09-11T03:50:21.183103+0000","last_fullsized":"2024-09-11T03:50:21.183103+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:49:33.199779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.199779+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:35.684686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057408299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,11],"acting":[7,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.25","version":"799'1249","reported_seq":5038,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.173879+0000","last_change":"2024-09-11T03:50:19.179280+0000","last_active":"2024-09-11T03:50:21.173879+0000","last_peered":"2024-09-11T03:50:21.173879+0000","last_clean":"2024-09-11T03:50:21.173879+0000","last_became_active":"2024-09-11T03:49:29.159733+0000","last_became_peered":"2024-09-11T03:49:29.159733+0000","last_unstale":"2024-09-11T03:50:21.173879+0000","last_undegraded":"2024-09-11T03:50:21.173879+0000","last_fullsized":"2024-09-11T03:50:21.173879+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"755'1138","last_scrub_stamp":"2024-09-11T03:49:34.198067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.198067+0000","objects_scrubbed":16,"log_size":1249,"log_dups_size":0,"ondisk_log_size":1249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:02.103866+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0047820019999999996,"stat_sum":{"num_bytes":36447347,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":49,"num_read_kb":53766,"num_write":18,"num_write_kb":6449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,6],"acting":[8,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.24","version":"795'648","reported_seq":4242,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287438+0000","last_change":"2024-09-11T03:50:19.310987+0000","last_active":"2024-09-11T03:50:21.287438+0000","last_peered":"2024-09-11T03:50:21.287438+0000","last_clean":"2024-09-11T03:50:21.287438+0000","last_became_active":"2024-09-11T03:49:29.155740+0000","last_became_peered":"2024-09-11T03:49:29.155740+0000","last_unstale":"2024-09-11T03:50:21.287438+0000","last_undegraded":"2024-09-11T03:50:21.287438+0000","last_fullsized":"2024-09-11T03:50:21.287438+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'612","last_scrub_stamp":"2024-09-11T03:49:34.554335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.554335+0000","objects_scrubbed":4,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:44.008787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00083026699999999998,"stat_sum":{"num_bytes":12434224,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":3941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,7],"acting":[6,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.23","version":"799'474","reported_seq":4032,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.172990+0000","last_change":"2024-09-11T03:50:19.312103+0000","last_active":"2024-09-11T03:50:21.172990+0000","last_peered":"2024-09-11T03:50:21.172990+0000","last_clean":"2024-09-11T03:50:21.172990+0000","last_became_active":"2024-09-11T03:46:32.970100+0000","last_became_peered":"2024-09-11T03:46:32.970100+0000","last_unstale":"2024-09-11T03:50:21.172990+0000","last_undegraded":"2024-09-11T03:50:21.172990+0000","last_fullsized":"2024-09-11T03:50:21.172990+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'474","last_scrub_stamp":"2024-09-11T03:50:19.311509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:19.311509+0000","objects_scrubbed":5,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:28.507275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0037852990000000002,"stat_sum":{"num_bytes":9866336,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":37,"num_read_kb":44928,"num_write":25,"num_write_kb":10047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.22","version":"531'216","reported_seq":3758,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287122+0000","last_change":"2024-09-11T03:50:19.310537+0000","last_active":"2024-09-11T03:50:21.287122+0000","last_peered":"2024-09-11T03:50:21.287122+0000","last_clean":"2024-09-11T03:50:21.287122+0000","last_became_active":"2024-09-11T03:46:32.970633+0000","last_became_peered":"2024-09-11T03:46:32.970633+0000","last_unstale":"2024-09-11T03:50:21.287122+0000","last_undegraded":"2024-09-11T03:50:21.287122+0000","last_fullsized":"2024-09-11T03:50:21.287122+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:49:18.406097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:18.406097+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:38.067889+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00048125199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.21","version":"801'337","reported_seq":3971,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.180084+0000","last_change":"2024-09-11T03:50:19.419097+0000","last_active":"2024-09-11T03:50:21.180084+0000","last_peered":"2024-09-11T03:50:21.180084+0000","last_clean":"2024-09-11T03:50:21.180084+0000","last_became_active":"2024-09-11T03:46:36.024928+0000","last_became_peered":"2024-09-11T03:46:36.024928+0000","last_unstale":"2024-09-11T03:50:21.180084+0000","last_undegraded":"2024-09-11T03:50:21.180084+0000","last_fullsized":"2024-09-11T03:50:21.180084+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":6,"last_scrub":"733'289","last_scrub_stamp":"2024-09-11T03:49:18.917801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:18.917801+0000","objects_scrubbed":3,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:22.890400+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003183539,"stat_sum":{"num_bytes":15025885,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":45171,"num_write":33,"num_write_kb":15311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,8],"acting":[11,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.20","version":"799'284","reported_seq":3988,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.182552+0000","last_change":"2024-09-11T03:50:19.417463+0000","last_active":"2024-09-11T03:50:21.182552+0000","last_peered":"2024-09-11T03:50:21.182552+0000","last_clean":"2024-09-11T03:50:21.182552+0000","last_became_active":"2024-09-11T03:46:32.974375+0000","last_became_peered":"2024-09-11T03:46:32.974375+0000","last_unstale":"2024-09-11T03:50:21.182552+0000","last_undegraded":"2024-09-11T03:50:21.182552+0000","last_fullsized":"2024-09-11T03:50:21.182552+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"718'223","last_scrub_stamp":"2024-09-11T03:49:00.376119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.376119+0000","objects_scrubbed":2,"log_size":284,"log_dups_size":0,"ondisk_log_size":284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:24.250345+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00053215800000000002,"stat_sum":{"num_bytes":11791232,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":87,"num_read_kb":134813,"num_write":17,"num_write_kb":9596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.1c","version":"785'774","reported_seq":4313,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287327+0000","last_change":"2024-09-11T03:50:19.417611+0000","last_active":"2024-09-11T03:50:21.287327+0000","last_peered":"2024-09-11T03:50:21.287327+0000","last_clean":"2024-09-11T03:50:21.287327+0000","last_became_active":"2024-09-11T03:46:32.971036+0000","last_became_peered":"2024-09-11T03:46:32.971036+0000","last_unstale":"2024-09-11T03:50:21.287327+0000","last_undegraded":"2024-09-11T03:50:21.287327+0000","last_fullsized":"2024-09-11T03:50:21.287327+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"754'760","last_scrub_stamp":"2024-09-11T03:49:33.579794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.579794+0000","objects_scrubbed":5,"log_size":774,"log_dups_size":0,"ondisk_log_size":774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:52.905582+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0013711299999999999,"stat_sum":{"num_bytes":8735071,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":36,"num_read_kb":35079,"num_write":36,"num_write_kb":13499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":656,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-09-11T03:50:20.171185+0000","last_change":"2024-09-11T03:47:26.195120+0000","last_active":"2024-09-11T03:50:20.171185+0000","last_peered":"2024-09-11T03:50:20.171185+0000","last_clean":"2024-09-11T03:50:20.171185+0000","last_became_active":"2024-09-11T03:46:15.974786+0000","last_became_peered":"2024-09-11T03:46:15.974786+0000","last_unstale":"2024-09-11T03:50:20.171185+0000","last_undegraded":"2024-09-11T03:50:20.171185+0000","last_fullsized":"2024-09-11T03:50:20.171185+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:26.195070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:26.195070+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:53.091756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.f","version":"557'465","reported_seq":4046,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.174084+0000","last_change":"2024-09-11T03:50:19.179710+0000","last_active":"2024-09-11T03:50:21.174084+0000","last_peered":"2024-09-11T03:50:21.174084+0000","last_clean":"2024-09-11T03:50:21.174084+0000","last_became_active":"2024-09-11T03:50:02.055146+0000","last_became_peered":"2024-09-11T03:50:02.055146+0000","last_unstale":"2024-09-11T03:50:21.174084+0000","last_undegraded":"2024-09-11T03:50:21.174084+0000","last_fullsized":"2024-09-11T03:50:21.174084+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:50:02.368307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.368307+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:17.871848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049233030000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.e","version":"782'1025","reported_seq":4540,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-11T03:50:18.275995+0000","last_change":"2024-09-11T03:50:17.047342+0000","last_active":"2024-09-11T03:50:18.275995+0000","last_peered":"2024-09-11T03:50:18.275995+0000","last_clean":"2024-09-11T03:50:18.275995+0000","last_became_active":"2024-09-11T03:50:02.051717+0000","last_became_peered":"2024-09-11T03:50:02.051717+0000","last_unstale":"2024-09-11T03:50:18.275995+0000","last_undegraded":"2024-09-11T03:50:18.275995+0000","last_fullsized":"2024-09-11T03:50:18.275995+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":0,"last_scrub":"782'1025","last_scrub_stamp":"2024-09-11T03:50:04.968160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.968160+0000","objects_scrubbed":0,"log_size":1025,"log_dups_size":0,"ondisk_log_size":1025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:29.590429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027614290000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":1235,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.183027+0000","last_change":"2024-09-11T03:49:20.176154+0000","last_active":"2024-09-11T03:50:21.183027+0000","last_peered":"2024-09-11T03:50:21.183027+0000","last_clean":"2024-09-11T03:50:21.183027+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:50:21.183027+0000","last_undegraded":"2024-09-11T03:50:21.183027+0000","last_fullsized":"2024-09-11T03:50:21.183027+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:20.176098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.176098+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:41.602355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"787'1009","reported_seq":4617,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.184731+0000","last_change":"2024-09-11T03:50:19.166959+0000","last_active":"2024-09-11T03:50:21.184731+0000","last_peered":"2024-09-11T03:50:21.184731+0000","last_clean":"2024-09-11T03:50:21.184731+0000","last_became_active":"2024-09-11T03:49:24.116744+0000","last_became_peered":"2024-09-11T03:49:24.116744+0000","last_unstale":"2024-09-11T03:50:21.184731+0000","last_undegraded":"2024-09-11T03:50:21.184731+0000","last_fullsized":"2024-09-11T03:50:21.184731+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'987","last_scrub_stamp":"2024-09-11T03:49:32.653263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.653263+0000","objects_scrubbed":3,"log_size":1009,"log_dups_size":0,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:55.077934+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00059440800000000005,"stat_sum":{"num_bytes":8757074,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":950,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.183404+0000","last_change":"2024-09-11T03:49:22.192021+0000","last_active":"2024-09-11T03:50:21.183404+0000","last_peered":"2024-09-11T03:50:21.183404+0000","last_clean":"2024-09-11T03:50:21.183404+0000","last_became_active":"2024-09-11T03:44:43.940022+0000","last_became_peered":"2024-09-11T03:44:43.940022+0000","last_unstale":"2024-09-11T03:50:21.183404+0000","last_undegraded":"2024-09-11T03:50:21.183404+0000","last_fullsized":"2024-09-11T03:50:21.183404+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:22.191965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:22.191965+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:33.940974+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"799'839","reported_seq":4412,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.180145+0000","last_change":"2024-09-11T03:50:19.418609+0000","last_active":"2024-09-11T03:50:21.180145+0000","last_peered":"2024-09-11T03:50:21.180145+0000","last_clean":"2024-09-11T03:50:21.180145+0000","last_became_active":"2024-09-11T03:49:24.115405+0000","last_became_peered":"2024-09-11T03:49:24.115405+0000","last_unstale":"2024-09-11T03:50:21.180145+0000","last_undegraded":"2024-09-11T03:50:21.180145+0000","last_fullsized":"2024-09-11T03:50:21.180145+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'808","last_scrub_stamp":"2024-09-11T03:49:33.953477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.953477+0000","objects_scrubbed":7,"log_size":839,"log_dups_size":0,"ondisk_log_size":839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.503774+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025141920000000002,"stat_sum":{"num_bytes":18373538,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":13,"num_read_kb":13991,"num_write":7,"num_write_kb":4797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.f","version":"20'92","reported_seq":516,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287885+0000","last_change":"2024-09-11T03:48:57.363099+0000","last_active":"2024-09-11T03:50:21.287885+0000","last_peered":"2024-09-11T03:50:21.287885+0000","last_clean":"2024-09-11T03:50:21.287885+0000","last_became_active":"2024-09-11T03:46:15.973796+0000","last_became_peered":"2024-09-11T03:46:15.973796+0000","last_unstale":"2024-09-11T03:50:21.287885+0000","last_undegraded":"2024-09-11T03:50:21.287885+0000","last_fullsized":"2024-09-11T03:50:21.287885+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:57.363027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:57.363027+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:10.644654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":1641,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.183570+0000","last_change":"2024-09-11T03:47:28.495899+0000","last_active":"2024-09-11T03:50:21.183570+0000","last_peered":"2024-09-11T03:50:21.183570+0000","last_clean":"2024-09-11T03:50:21.183570+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:50:21.183570+0000","last_undegraded":"2024-09-11T03:50:21.183570+0000","last_fullsized":"2024-09-11T03:50:21.183570+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:28.495839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:28.495839+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.015311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"803'521","reported_seq":4083,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.187737+0000","last_change":"2024-09-11T03:50:19.166459+0000","last_active":"2024-09-11T03:50:21.187737+0000","last_peered":"2024-09-11T03:50:21.187737+0000","last_clean":"2024-09-11T03:50:21.187737+0000","last_became_active":"2024-09-11T03:46:27.972329+0000","last_became_peered":"2024-09-11T03:46:27.972329+0000","last_unstale":"2024-09-11T03:50:21.187737+0000","last_undegraded":"2024-09-11T03:50:21.187737+0000","last_fullsized":"2024-09-11T03:50:21.187737+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'480","last_scrub_stamp":"2024-09-11T03:49:37.694768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.694768+0000","objects_scrubbed":5,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:43.550624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000451767,"stat_sum":{"num_bytes":13193420,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":27,"num_read_kb":23186,"num_write":37,"num_write_kb":7258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":1244,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.172906+0000","last_change":"2024-09-11T03:47:25.344219+0000","last_active":"2024-09-11T03:50:21.172906+0000","last_peered":"2024-09-11T03:50:21.172906+0000","last_clean":"2024-09-11T03:50:21.172906+0000","last_became_active":"2024-09-11T03:46:15.968960+0000","last_became_peered":"2024-09-11T03:46:15.968960+0000","last_unstale":"2024-09-11T03:50:21.172906+0000","last_undegraded":"2024-09-11T03:50:21.172906+0000","last_fullsized":"2024-09-11T03:50:21.172906+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:25.344170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:25.344170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:42.179046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"563'192","reported_seq":3821,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.183451+0000","last_change":"2024-09-11T03:50:19.417844+0000","last_active":"2024-09-11T03:50:21.183451+0000","last_peered":"2024-09-11T03:50:21.183451+0000","last_clean":"2024-09-11T03:50:21.183451+0000","last_became_active":"2024-09-11T03:46:27.974405+0000","last_became_peered":"2024-09-11T03:46:27.974405+0000","last_unstale":"2024-09-11T03:50:21.183451+0000","last_undegraded":"2024-09-11T03:50:21.183451+0000","last_fullsized":"2024-09-11T03:50:21.183451+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:49:15.233165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:15.233165+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:29.809974+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058246700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":1202,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-09-11T03:50:20.171027+0000","last_change":"2024-09-11T03:49:17.252987+0000","last_active":"2024-09-11T03:50:20.171027+0000","last_peered":"2024-09-11T03:50:20.171027+0000","last_clean":"2024-09-11T03:50:20.171027+0000","last_became_active":"2024-09-11T03:44:25.934580+0000","last_became_peered":"2024-09-11T03:44:25.934580+0000","last_unstale":"2024-09-11T03:50:20.171027+0000","last_undegraded":"2024-09-11T03:50:20.171027+0000","last_fullsized":"2024-09-11T03:50:20.171027+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:17.252916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:17.252916+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:39.008724+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"567'235","reported_seq":3750,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-09-11T03:50:20.171462+0000","last_change":"2024-09-11T03:50:19.416466+0000","last_active":"2024-09-11T03:50:20.171462+0000","last_peered":"2024-09-11T03:50:20.171462+0000","last_clean":"2024-09-11T03:50:20.171462+0000","last_became_active":"2024-09-11T03:46:36.024925+0000","last_became_peered":"2024-09-11T03:46:36.024925+0000","last_unstale":"2024-09-11T03:50:20.171462+0000","last_undegraded":"2024-09-11T03:50:20.171462+0000","last_fullsized":"2024-09-11T03:50:20.171462+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:49:01.138426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:01.138426+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059223399999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":818,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287153+0000","last_change":"2024-09-11T03:50:18.699606+0000","last_active":"2024-09-11T03:50:21.287153+0000","last_peered":"2024-09-11T03:50:21.287153+0000","last_clean":"2024-09-11T03:50:21.287153+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:50:21.287153+0000","last_undegraded":"2024-09-11T03:50:21.287153+0000","last_fullsized":"2024-09-11T03:50:21.287153+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:18.699532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.699532+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:27.930456+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"799'299","reported_seq":3939,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.184581+0000","last_change":"2024-09-11T03:50:20.621293+0000","last_active":"2024-09-11T03:50:21.184581+0000","last_peered":"2024-09-11T03:50:21.184581+0000","last_clean":"2024-09-11T03:50:21.184581+0000","last_became_active":"2024-09-11T03:46:27.971649+0000","last_became_peered":"2024-09-11T03:46:27.971649+0000","last_unstale":"2024-09-11T03:50:21.184581+0000","last_undegraded":"2024-09-11T03:50:21.184581+0000","last_fullsized":"2024-09-11T03:50:21.184581+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"799'299","last_scrub_stamp":"2024-09-11T03:50:20.621227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:20.621227+0000","objects_scrubbed":3,"log_size":299,"log_dups_size":0,"ondisk_log_size":299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:36.946930+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00047804800000000001,"stat_sum":{"num_bytes":4723229,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":24,"num_read_kb":13035,"num_write":25,"num_write_kb":12701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":508,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287717+0000","last_change":"2024-09-11T03:49:37.637065+0000","last_active":"2024-09-11T03:50:21.287717+0000","last_peered":"2024-09-11T03:50:21.287717+0000","last_clean":"2024-09-11T03:50:21.287717+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:50:21.287717+0000","last_undegraded":"2024-09-11T03:50:21.287717+0000","last_fullsized":"2024-09-11T03:50:21.287717+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:37.637018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.637018+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:05.221389+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"799'659","reported_seq":4277,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.172994+0000","last_change":"2024-09-11T03:50:19.308923+0000","last_active":"2024-09-11T03:50:21.172994+0000","last_peered":"2024-09-11T03:50:21.172994+0000","last_clean":"2024-09-11T03:50:21.172994+0000","last_became_active":"2024-09-11T03:49:24.114330+0000","last_became_peered":"2024-09-11T03:49:24.114330+0000","last_unstale":"2024-09-11T03:50:21.172994+0000","last_undegraded":"2024-09-11T03:50:21.172994+0000","last_fullsized":"2024-09-11T03:50:21.172994+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'620","last_scrub_stamp":"2024-09-11T03:49:34.288083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.288083+0000","objects_scrubbed":8,"log_size":659,"log_dups_size":0,"ondisk_log_size":659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:42.352241+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00046181999999999999,"stat_sum":{"num_bytes":20080188,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":12,"num_read_kb":8619,"num_write":7,"num_write_kb":5852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":821,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287308+0000","last_change":"2024-09-11T03:49:17.858224+0000","last_active":"2024-09-11T03:50:21.287308+0000","last_peered":"2024-09-11T03:50:21.287308+0000","last_clean":"2024-09-11T03:50:21.287308+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:50:21.287308+0000","last_undegraded":"2024-09-11T03:50:21.287308+0000","last_fullsized":"2024-09-11T03:50:21.287308+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:17.858165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:17.858165+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:40.674663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"799'1169","reported_seq":4792,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.174093+0000","last_change":"2024-09-11T03:50:19.179305+0000","last_active":"2024-09-11T03:50:21.174093+0000","last_peered":"2024-09-11T03:50:21.174093+0000","last_clean":"2024-09-11T03:50:21.174093+0000","last_became_active":"2024-09-11T03:49:24.115095+0000","last_became_peered":"2024-09-11T03:49:24.115095+0000","last_unstale":"2024-09-11T03:50:21.174093+0000","last_undegraded":"2024-09-11T03:50:21.174093+0000","last_fullsized":"2024-09-11T03:50:21.174093+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'1112","last_scrub_stamp":"2024-09-11T03:49:35.188307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.188307+0000","objects_scrubbed":7,"log_size":1169,"log_dups_size":0,"ondisk_log_size":1169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:43.462413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0045577259999999998,"stat_sum":{"num_bytes":16331222,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":5,"num_read_kb":1,"num_write":9,"num_write_kb":5553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":820,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287306+0000","last_change":"2024-09-11T03:49:36.654074+0000","last_active":"2024-09-11T03:50:21.287306+0000","last_peered":"2024-09-11T03:50:21.287306+0000","last_clean":"2024-09-11T03:50:21.287306+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:50:21.287306+0000","last_undegraded":"2024-09-11T03:50:21.287306+0000","last_fullsized":"2024-09-11T03:50:21.287306+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:36.654025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.654025+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:06.216209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":3713,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.173491+0000","last_change":"2024-09-11T03:50:19.309413+0000","last_active":"2024-09-11T03:50:21.173491+0000","last_peered":"2024-09-11T03:50:21.173491+0000","last_clean":"2024-09-11T03:50:21.173491+0000","last_became_active":"2024-09-11T03:49:24.113518+0000","last_became_peered":"2024-09-11T03:49:24.113518+0000","last_unstale":"2024-09-11T03:50:21.173491+0000","last_undegraded":"2024-09-11T03:50:21.173491+0000","last_fullsized":"2024-09-11T03:50:21.173491+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:49:35.267227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.267227+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.848934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083273899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":1419,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.180015+0000","last_change":"2024-09-11T03:49:21.909643+0000","last_active":"2024-09-11T03:50:21.180015+0000","last_peered":"2024-09-11T03:50:21.180015+0000","last_clean":"2024-09-11T03:50:21.180015+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:50:21.180015+0000","last_undegraded":"2024-09-11T03:50:21.180015+0000","last_fullsized":"2024-09-11T03:50:21.180015+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:21.909594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:21.909594+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:22.293071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"799'1164","reported_seq":4885,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287596+0000","last_change":"2024-09-11T03:50:19.311000+0000","last_active":"2024-09-11T03:50:21.287596+0000","last_peered":"2024-09-11T03:50:21.287596+0000","last_clean":"2024-09-11T03:50:21.287596+0000","last_became_active":"2024-09-11T03:49:24.121333+0000","last_became_peered":"2024-09-11T03:49:24.121333+0000","last_unstale":"2024-09-11T03:50:21.287596+0000","last_undegraded":"2024-09-11T03:50:21.287596+0000","last_fullsized":"2024-09-11T03:50:21.287596+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'1066","last_scrub_stamp":"2024-09-11T03:49:35.600201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.600201+0000","objects_scrubbed":7,"log_size":1164,"log_dups_size":0,"ondisk_log_size":1164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:05.565915+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00067884300000000005,"stat_sum":{"num_bytes":31850569,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":66,"num_read_kb":62760,"num_write":33,"num_write_kb":22568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":1222,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.180091+0000","last_change":"2024-09-11T03:50:18.176206+0000","last_active":"2024-09-11T03:50:21.180091+0000","last_peered":"2024-09-11T03:50:21.180091+0000","last_clean":"2024-09-11T03:50:21.180091+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:50:21.180091+0000","last_undegraded":"2024-09-11T03:50:21.180091+0000","last_fullsized":"2024-09-11T03:50:21.180091+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:18.176156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.176156+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:43.809898+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"744'374","reported_seq":3973,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.183423+0000","last_change":"2024-09-11T03:50:19.417660+0000","last_active":"2024-09-11T03:50:21.183423+0000","last_peered":"2024-09-11T03:50:21.183423+0000","last_clean":"2024-09-11T03:50:21.183423+0000","last_became_active":"2024-09-11T03:49:24.115150+0000","last_became_peered":"2024-09-11T03:49:24.115150+0000","last_unstale":"2024-09-11T03:50:21.183423+0000","last_undegraded":"2024-09-11T03:50:21.183423+0000","last_fullsized":"2024-09-11T03:50:21.183423+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'374","last_scrub_stamp":"2024-09-11T03:49:35.190806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.190806+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:55.134834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052494200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":1559,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-11T03:50:18.275624+0000","last_change":"2024-09-11T03:48:59.762212+0000","last_active":"2024-09-11T03:50:18.275624+0000","last_peered":"2024-09-11T03:50:18.275624+0000","last_clean":"2024-09-11T03:50:18.275624+0000","last_became_active":"2024-09-11T03:46:15.973271+0000","last_became_peered":"2024-09-11T03:46:15.973271+0000","last_unstale":"2024-09-11T03:50:18.275624+0000","last_undegraded":"2024-09-11T03:50:18.275624+0000","last_fullsized":"2024-09-11T03:50:18.275624+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:59.762145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:59.762145+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"799'1278","reported_seq":4954,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.172648+0000","last_change":"2024-09-11T03:50:19.309163+0000","last_active":"2024-09-11T03:50:21.172648+0000","last_peered":"2024-09-11T03:50:21.172648+0000","last_clean":"2024-09-11T03:50:21.172648+0000","last_became_active":"2024-09-11T03:49:24.114865+0000","last_became_peered":"2024-09-11T03:49:24.114865+0000","last_unstale":"2024-09-11T03:50:21.172648+0000","last_undegraded":"2024-09-11T03:50:21.172648+0000","last_fullsized":"2024-09-11T03:50:21.172648+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'1171","last_scrub_stamp":"2024-09-11T03:49:33.292043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.292043+0000","objects_scrubbed":10,"log_size":1278,"log_dups_size":0,"ondisk_log_size":1278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:01.827500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00064786300000000004,"stat_sum":{"num_bytes":32860253,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":33,"num_read_kb":35558,"num_write":22,"num_write_kb":10529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":1636,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.182531+0000","last_change":"2024-09-11T03:49:37.206392+0000","last_active":"2024-09-11T03:50:21.182531+0000","last_peered":"2024-09-11T03:50:21.182531+0000","last_clean":"2024-09-11T03:50:21.182531+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:50:21.182531+0000","last_undegraded":"2024-09-11T03:50:21.182531+0000","last_fullsized":"2024-09-11T03:50:21.182531+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:37.206321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.206321+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:53.641968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"787'370","reported_seq":3966,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.180431+0000","last_change":"2024-09-11T03:50:19.418848+0000","last_active":"2024-09-11T03:50:21.180431+0000","last_peered":"2024-09-11T03:50:21.180431+0000","last_clean":"2024-09-11T03:50:21.180431+0000","last_became_active":"2024-09-11T03:49:24.114551+0000","last_became_peered":"2024-09-11T03:49:24.114551+0000","last_unstale":"2024-09-11T03:50:21.180431+0000","last_undegraded":"2024-09-11T03:50:21.180431+0000","last_fullsized":"2024-09-11T03:50:21.180431+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:49:32.911631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.911631+0000","objects_scrubbed":4,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:52.268693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026279979999999999,"stat_sum":{"num_bytes":5429057,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":10,"num_read_kb":12240,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":1295,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.184400+0000","last_change":"2024-09-11T03:49:34.621588+0000","last_active":"2024-09-11T03:50:21.184400+0000","last_peered":"2024-09-11T03:50:21.184400+0000","last_clean":"2024-09-11T03:50:21.184400+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:50:21.184400+0000","last_undegraded":"2024-09-11T03:50:21.184400+0000","last_fullsized":"2024-09-11T03:50:21.184400+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:34.621527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.621527+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.563733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"568'646","reported_seq":4234,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.173761+0000","last_change":"2024-09-11T03:50:19.175142+0000","last_active":"2024-09-11T03:50:21.173761+0000","last_peered":"2024-09-11T03:50:21.173761+0000","last_clean":"2024-09-11T03:50:21.173761+0000","last_became_active":"2024-09-11T03:49:24.113147+0000","last_became_peered":"2024-09-11T03:49:24.113147+0000","last_unstale":"2024-09-11T03:50:21.173761+0000","last_undegraded":"2024-09-11T03:50:21.173761+0000","last_fullsized":"2024-09-11T03:50:21.173761+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:49:33.204497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.204497+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:51.429036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072388499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":1245,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.184397+0000","last_change":"2024-09-11T03:49:00.710015+0000","last_active":"2024-09-11T03:50:21.184397+0000","last_peered":"2024-09-11T03:50:21.184397+0000","last_clean":"2024-09-11T03:50:21.184397+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:50:21.184397+0000","last_undegraded":"2024-09-11T03:50:21.184397+0000","last_fullsized":"2024-09-11T03:50:21.184397+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:00.709966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.709966+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:30.175623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":3626,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-11T03:50:18.276161+0000","last_change":"2024-09-11T03:50:17.046834+0000","last_active":"2024-09-11T03:50:18.276161+0000","last_peered":"2024-09-11T03:50:18.276161+0000","last_clean":"2024-09-11T03:50:18.276161+0000","last_became_active":"2024-09-11T03:50:02.046906+0000","last_became_peered":"2024-09-11T03:50:02.046906+0000","last_unstale":"2024-09-11T03:50:18.276161+0000","last_undegraded":"2024-09-11T03:50:18.276161+0000","last_fullsized":"2024-09-11T03:50:18.276161+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:50:02.987179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.987179+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:21.446356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019638569999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":1228,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-09-11T03:50:20.171360+0000","last_change":"2024-09-11T03:49:41.224105+0000","last_active":"2024-09-11T03:50:20.171360+0000","last_peered":"2024-09-11T03:50:20.171360+0000","last_clean":"2024-09-11T03:50:20.171360+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:50:20.171360+0000","last_undegraded":"2024-09-11T03:50:20.171360+0000","last_fullsized":"2024-09-11T03:50:20.171360+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:41.224059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:41.224059+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:58.932206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"795'556","reported_seq":4135,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287258+0000","last_change":"2024-09-11T03:50:19.416857+0000","last_active":"2024-09-11T03:50:21.287258+0000","last_peered":"2024-09-11T03:50:21.287258+0000","last_clean":"2024-09-11T03:50:21.287258+0000","last_became_active":"2024-09-11T03:50:02.053513+0000","last_became_peered":"2024-09-11T03:50:02.053513+0000","last_unstale":"2024-09-11T03:50:21.287258+0000","last_undegraded":"2024-09-11T03:50:21.287258+0000","last_fullsized":"2024-09-11T03:50:21.287258+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"784'551","last_scrub_stamp":"2024-09-11T03:50:02.647690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.647690+0000","objects_scrubbed":4,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:15.152709+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073850599999999999,"stat_sum":{"num_bytes":6367447,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":1226,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.184441+0000","last_change":"2024-09-11T03:47:23.339646+0000","last_active":"2024-09-11T03:50:21.184441+0000","last_peered":"2024-09-11T03:50:21.184441+0000","last_clean":"2024-09-11T03:50:21.184441+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:50:21.184441+0000","last_undegraded":"2024-09-11T03:50:21.184441+0000","last_fullsized":"2024-09-11T03:50:21.184441+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:23.339595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:23.339595+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:50.634356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":2713,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287320+0000","last_change":"2024-09-11T03:50:19.311484+0000","last_active":"2024-09-11T03:50:21.287320+0000","last_peered":"2024-09-11T03:50:21.287320+0000","last_clean":"2024-09-11T03:50:21.287320+0000","last_became_active":"2024-09-11T03:50:02.048777+0000","last_became_peered":"2024-09-11T03:50:02.048777+0000","last_unstale":"2024-09-11T03:50:21.287320+0000","last_undegraded":"2024-09-11T03:50:21.287320+0000","last_fullsized":"2024-09-11T03:50:21.287320+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:50:02.696833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.696833+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:26.797941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001024561,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.13","version":"20'92","reported_seq":1331,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.172809+0000","last_change":"2024-09-11T03:47:18.217600+0000","last_active":"2024-09-11T03:50:21.172809+0000","last_peered":"2024-09-11T03:50:21.172809+0000","last_clean":"2024-09-11T03:50:21.172809+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:50:21.172809+0000","last_undegraded":"2024-09-11T03:50:21.172809+0000","last_fullsized":"2024-09-11T03:50:21.172809+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:18.217555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:18.217555+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:43.139528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":2657,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.173582+0000","last_change":"2024-09-11T03:50:19.175201+0000","last_active":"2024-09-11T03:50:21.173582+0000","last_peered":"2024-09-11T03:50:21.173582+0000","last_clean":"2024-09-11T03:50:21.173582+0000","last_became_active":"2024-09-11T03:50:02.057196+0000","last_became_peered":"2024-09-11T03:50:02.057196+0000","last_unstale":"2024-09-11T03:50:21.173582+0000","last_undegraded":"2024-09-11T03:50:21.173582+0000","last_fullsized":"2024-09-11T03:50:21.173582+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:50:03.345563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:03.345563+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:08.696449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098105900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":812,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.183196+0000","last_change":"2024-09-11T03:50:20.772575+0000","last_active":"2024-09-11T03:50:21.183196+0000","last_peered":"2024-09-11T03:50:21.183196+0000","last_clean":"2024-09-11T03:50:21.183196+0000","last_became_active":"2024-09-11T03:46:36.026459+0000","last_became_peered":"2024-09-11T03:46:36.026459+0000","last_unstale":"2024-09-11T03:50:21.183196+0000","last_undegraded":"2024-09-11T03:50:21.183196+0000","last_fullsized":"2024-09-11T03:50:21.183196+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:20.772524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:20.772524+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:26.554678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":3209,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287842+0000","last_change":"2024-09-11T03:50:19.311661+0000","last_active":"2024-09-11T03:50:21.287842+0000","last_peered":"2024-09-11T03:50:21.287842+0000","last_clean":"2024-09-11T03:50:21.287842+0000","last_became_active":"2024-09-11T03:50:02.052278+0000","last_became_peered":"2024-09-11T03:50:02.052278+0000","last_unstale":"2024-09-11T03:50:21.287842+0000","last_undegraded":"2024-09-11T03:50:21.287842+0000","last_fullsized":"2024-09-11T03:50:21.287842+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:50:03.690374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:03.690374+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:12.901918+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00100592,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":1207,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.182962+0000","last_change":"2024-09-11T03:50:19.741484+0000","last_active":"2024-09-11T03:50:21.182962+0000","last_peered":"2024-09-11T03:50:21.182962+0000","last_clean":"2024-09-11T03:50:21.182962+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:50:21.182962+0000","last_undegraded":"2024-09-11T03:50:21.182962+0000","last_fullsized":"2024-09-11T03:50:21.182962+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:19.741428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:19.741428+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:36.778923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"782'561","reported_seq":4182,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.184598+0000","last_change":"2024-09-11T03:50:19.166721+0000","last_active":"2024-09-11T03:50:21.184598+0000","last_peered":"2024-09-11T03:50:21.184598+0000","last_clean":"2024-09-11T03:50:21.184598+0000","last_became_active":"2024-09-11T03:50:02.054667+0000","last_became_peered":"2024-09-11T03:50:02.054667+0000","last_unstale":"2024-09-11T03:50:21.184598+0000","last_undegraded":"2024-09-11T03:50:21.184598+0000","last_fullsized":"2024-09-11T03:50:21.184598+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"782'561","last_scrub_stamp":"2024-09-11T03:50:02.696931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.696931+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:27.280066+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00037953800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":678,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.180428+0000","last_change":"2024-09-11T03:49:20.949194+0000","last_active":"2024-09-11T03:50:21.180428+0000","last_peered":"2024-09-11T03:50:21.180428+0000","last_clean":"2024-09-11T03:50:21.180428+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:50:21.180428+0000","last_undegraded":"2024-09-11T03:50:21.180428+0000","last_fullsized":"2024-09-11T03:50:21.180428+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:20.949118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.949118+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:36.681631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":3846,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-11T03:50:18.275856+0000","last_change":"2024-09-11T03:50:17.044089+0000","last_active":"2024-09-11T03:50:18.275856+0000","last_peered":"2024-09-11T03:50:18.275856+0000","last_clean":"2024-09-11T03:50:18.275856+0000","last_became_active":"2024-09-11T03:50:02.051850+0000","last_became_peered":"2024-09-11T03:50:02.051850+0000","last_unstale":"2024-09-11T03:50:18.275856+0000","last_undegraded":"2024-09-11T03:50:18.275856+0000","last_fullsized":"2024-09-11T03:50:18.275856+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:50:04.004006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.004006+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:29.258381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070946500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":1219,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.173948+0000","last_change":"2024-09-11T03:47:29.086383+0000","last_active":"2024-09-11T03:50:21.173948+0000","last_peered":"2024-09-11T03:50:21.173948+0000","last_clean":"2024-09-11T03:50:21.173948+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:50:21.173948+0000","last_undegraded":"2024-09-11T03:50:21.173948+0000","last_fullsized":"2024-09-11T03:50:21.173948+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:29.086335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:29.086335+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:43.171032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"778'353","reported_seq":3843,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287671+0000","last_change":"2024-09-11T03:50:19.311215+0000","last_active":"2024-09-11T03:50:21.287671+0000","last_peered":"2024-09-11T03:50:21.287671+0000","last_clean":"2024-09-11T03:50:21.287671+0000","last_became_active":"2024-09-11T03:50:02.054211+0000","last_became_peered":"2024-09-11T03:50:02.054211+0000","last_unstale":"2024-09-11T03:50:21.287671+0000","last_undegraded":"2024-09-11T03:50:21.287671+0000","last_fullsized":"2024-09-11T03:50:21.287671+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"778'353","last_scrub_stamp":"2024-09-11T03:50:04.646331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.646331+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:33.694456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090133300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.18","version":"20'92","reported_seq":1338,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.184345+0000","last_change":"2024-09-11T03:49:56.630710+0000","last_active":"2024-09-11T03:50:21.184345+0000","last_peered":"2024-09-11T03:50:21.184345+0000","last_clean":"2024-09-11T03:50:21.184345+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:50:21.184345+0000","last_undegraded":"2024-09-11T03:50:21.184345+0000","last_fullsized":"2024-09-11T03:50:21.184345+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:56.630651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:56.630651+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:18.735201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"803'868","reported_seq":4545,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.180228+0000","last_change":"2024-09-11T03:50:19.419134+0000","last_active":"2024-09-11T03:50:21.180228+0000","last_peered":"2024-09-11T03:50:21.180228+0000","last_clean":"2024-09-11T03:50:21.180228+0000","last_became_active":"2024-09-11T03:46:32.975567+0000","last_became_peered":"2024-09-11T03:46:32.975567+0000","last_unstale":"2024-09-11T03:50:21.180228+0000","last_undegraded":"2024-09-11T03:50:21.180228+0000","last_fullsized":"2024-09-11T03:50:21.180228+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"754'773","last_scrub_stamp":"2024-09-11T03:49:34.919322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.919322+0000","objects_scrubbed":8,"log_size":868,"log_dups_size":0,"ondisk_log_size":868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:54.969694+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024172429999999999,"stat_sum":{"num_bytes":21540010,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":115,"num_read_kb":123718,"num_write":68,"num_write_kb":26442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7],"acting":[11,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.19","version":"20'92","reported_seq":824,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287083+0000","last_change":"2024-09-11T03:48:56.129575+0000","last_active":"2024-09-11T03:50:21.287083+0000","last_peered":"2024-09-11T03:50:21.287083+0000","last_clean":"2024-09-11T03:50:21.287083+0000","last_became_active":"2024-09-11T03:46:15.972841+0000","last_became_peered":"2024-09-11T03:46:15.972841+0000","last_unstale":"2024-09-11T03:50:21.287083+0000","last_undegraded":"2024-09-11T03:50:21.287083+0000","last_fullsized":"2024-09-11T03:50:21.287083+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:56.129516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:56.129516+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:24.133914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"557'273","reported_seq":3770,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287425+0000","last_change":"2024-09-11T03:50:19.310769+0000","last_active":"2024-09-11T03:50:21.287425+0000","last_peered":"2024-09-11T03:50:21.287425+0000","last_clean":"2024-09-11T03:50:21.287425+0000","last_became_active":"2024-09-11T03:46:32.971003+0000","last_became_peered":"2024-09-11T03:46:32.971003+0000","last_unstale":"2024-09-11T03:50:21.287425+0000","last_undegraded":"2024-09-11T03:50:21.287425+0000","last_fullsized":"2024-09-11T03:50:21.287425+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:49:20.466140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.466140+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:42.601783+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000640458,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,2],"acting":[6,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.1a","version":"20'92","reported_seq":369,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.182745+0000","last_change":"2024-09-11T03:50:01.843384+0000","last_active":"2024-09-11T03:50:21.182745+0000","last_peered":"2024-09-11T03:50:21.182745+0000","last_clean":"2024-09-11T03:50:21.182745+0000","last_became_active":"2024-09-11T03:49:30.162694+0000","last_became_peered":"2024-09-11T03:49:30.162694+0000","last_unstale":"2024-09-11T03:50:21.182745+0000","last_undegraded":"2024-09-11T03:50:21.182745+0000","last_fullsized":"2024-09-11T03:50:21.182745+0000","mapping_epoch":751,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":752,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:01.843302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:01.843302+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:12.888176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.19","version":"799'581","reported_seq":4220,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287157+0000","last_change":"2024-09-11T03:50:19.311264+0000","last_active":"2024-09-11T03:50:21.287157+0000","last_peered":"2024-09-11T03:50:21.287157+0000","last_clean":"2024-09-11T03:50:21.287157+0000","last_became_active":"2024-09-11T03:46:19.002002+0000","last_became_peered":"2024-09-11T03:46:19.002002+0000","last_unstale":"2024-09-11T03:50:21.287157+0000","last_undegraded":"2024-09-11T03:50:21.287157+0000","last_fullsized":"2024-09-11T03:50:21.287157+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":5,"last_scrub":"736'546","last_scrub_stamp":"2024-09-11T03:49:19.458345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:19.458345+0000","objects_scrubbed":6,"log_size":581,"log_dups_size":0,"ondisk_log_size":581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:23.751919+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00093114800000000002,"stat_sum":{"num_bytes":11704959,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":104,"num_read_kb":99782,"num_write":68,"num_write_kb":30004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4870845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.1b","version":"20'92","reported_seq":1195,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.173619+0000","last_change":"2024-09-11T03:48:31.033896+0000","last_active":"2024-09-11T03:50:21.173619+0000","last_peered":"2024-09-11T03:50:21.173619+0000","last_clean":"2024-09-11T03:50:21.173619+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:50:21.173619+0000","last_undegraded":"2024-09-11T03:50:21.173619+0000","last_fullsized":"2024-09-11T03:50:21.173619+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:31.033837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:31.033837+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:46.181076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"799'552","reported_seq":4177,"reported_epoch":800,"state":"active+clean","last_fresh":"2024-09-11T03:50:18.275647+0000","last_change":"2024-09-11T03:50:17.051902+0000","last_active":"2024-09-11T03:50:18.275647+0000","last_peered":"2024-09-11T03:50:18.275647+0000","last_clean":"2024-09-11T03:50:18.275647+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:50:18.275647+0000","last_undegraded":"2024-09-11T03:50:18.275647+0000","last_fullsized":"2024-09-11T03:50:18.275647+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"783'523","last_scrub_stamp":"2024-09-11T03:50:01.960691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:01.960691+0000","objects_scrubbed":10,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:24.911794+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.006174003,"stat_sum":{"num_bytes":28917557,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":215,"num_read_kb":248888,"num_write":105,"num_write_kb":45767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4865740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"}]},{"pgid":"2.1f","version":"799'744","reported_seq":4247,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-09-11T03:50:20.171717+0000","last_change":"2024-09-11T03:50:19.416403+0000","last_active":"2024-09-11T03:50:20.171717+0000","last_peered":"2024-09-11T03:50:20.171717+0000","last_clean":"2024-09-11T03:50:20.171717+0000","last_became_active":"2024-09-11T03:46:32.968696+0000","last_became_peered":"2024-09-11T03:46:32.968696+0000","last_unstale":"2024-09-11T03:50:20.171717+0000","last_undegraded":"2024-09-11T03:50:20.171717+0000","last_fullsized":"2024-09-11T03:50:20.171717+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"733'613","last_scrub_stamp":"2024-09-11T03:49:20.295589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.295589+0000","objects_scrubbed":10,"log_size":744,"log_dups_size":0,"ondisk_log_size":744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:38.049499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00057476800000000002,"stat_sum":{"num_bytes":45342159,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":137,"num_read_kb":151177,"num_write":94,"num_write_kb":33522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.1c","version":"20'92","reported_seq":1202,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.173832+0000","last_change":"2024-09-11T03:49:40.156559+0000","last_active":"2024-09-11T03:50:21.173832+0000","last_peered":"2024-09-11T03:50:21.173832+0000","last_clean":"2024-09-11T03:50:21.173832+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:50:21.173832+0000","last_undegraded":"2024-09-11T03:50:21.173832+0000","last_fullsized":"2024-09-11T03:50:21.173832+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:40.156495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:40.156495+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:04.047069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1e","version":"802'1056","reported_seq":4579,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-09-11T03:50:20.180865+0000","last_change":"2024-09-11T03:50:19.416208+0000","last_active":"2024-09-11T03:50:20.180865+0000","last_peered":"2024-09-11T03:50:20.180865+0000","last_clean":"2024-09-11T03:50:20.180865+0000","last_became_active":"2024-09-11T03:46:32.970637+0000","last_became_peered":"2024-09-11T03:46:32.970637+0000","last_unstale":"2024-09-11T03:50:20.180865+0000","last_undegraded":"2024-09-11T03:50:20.180865+0000","last_fullsized":"2024-09-11T03:50:20.180865+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"785'1044","last_scrub_stamp":"2024-09-11T03:50:03.954442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:03.954442+0000","objects_scrubbed":9,"log_size":1056,"log_dups_size":0,"ondisk_log_size":1056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:15.070932+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00041738300000000001,"stat_sum":{"num_bytes":23667896,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":51946,"num_write":93,"num_write_kb":34996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.1d","version":"20'92","reported_seq":1184,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.173095+0000","last_change":"2024-09-11T03:47:22.312507+0000","last_active":"2024-09-11T03:50:21.173095+0000","last_peered":"2024-09-11T03:50:21.173095+0000","last_clean":"2024-09-11T03:50:21.173095+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:50:21.173095+0000","last_undegraded":"2024-09-11T03:50:21.173095+0000","last_fullsized":"2024-09-11T03:50:21.173095+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:22.312456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:22.312456+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.239645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"567'752","reported_seq":4261,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.287385+0000","last_change":"2024-09-11T03:50:19.418254+0000","last_active":"2024-09-11T03:50:21.287385+0000","last_peered":"2024-09-11T03:50:21.287385+0000","last_clean":"2024-09-11T03:50:21.287385+0000","last_became_active":"2024-09-11T03:46:32.972371+0000","last_became_peered":"2024-09-11T03:46:32.972371+0000","last_unstale":"2024-09-11T03:50:21.287385+0000","last_undegraded":"2024-09-11T03:50:21.287385+0000","last_fullsized":"2024-09-11T03:50:21.287385+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":5,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:48:55.139430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:48:55.139430+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001333447,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,11],"acting":[10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"166","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"1.1e","version":"20'92","reported_seq":1210,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-09-11T03:50:21.180541+0000","last_change":"2024-09-11T03:49:36.949199+0000","last_active":"2024-09-11T03:50:21.180541+0000","last_peered":"2024-09-11T03:50:21.180541+0000","last_clean":"2024-09-11T03:50:21.180541+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:50:21.180541+0000","last_undegraded":"2024-09-11T03:50:21.180541+0000","last_fullsized":"2024-09-11T03:50:21.180541+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:36.949149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.949149+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:45.388809+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":56,"stat_sum":{"num_bytes":569097632,"num_objects":269,"num_object_clones":219,"num_object_copies":807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":20,"num_read":1187,"num_read_kb":1273784,"num_write":775,"num_write_kb":338935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":9736585,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":978731008,"data_stored":1351221771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31298,"ondisk_log_size":31298,"up":168,"acting":168,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285726,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108876,"kb_used_data":77688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93636276,"statfs":{"total":95995035648,"available":95883546624,"internally_reserved":0,"allocated":79552512,"data_stored":107246984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514658,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89940,"kb_used_data":62908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93655212,"statfs":{"total":95995035648,"available":95902937088,"internally_reserved":0,"allocated":64417792,"data_stored":73254324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743591,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":121792,"kb_used_data":64236,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93623360,"statfs":{"total":95995035648,"available":95870320640,"internally_reserved":0,"allocated":65777664,"data_stored":73097365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037935,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":193520,"kb_used_data":135956,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93551632,"statfs":{"total":95995035648,"available":95796871168,"internally_reserved":0,"allocated":139218944,"data_stored":199201752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299570,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":211204,"kb_used_data":166128,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93533948,"statfs":{"total":95995035648,"available":95778762752,"internally_reserved":0,"allocated":170115072,"data_stored":227385492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528503,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":141596,"kb_used_data":114568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93603556,"statfs":{"total":95995035648,"available":95850041344,"internally_reserved":0,"allocated":117317632,"data_stored":151873732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757437,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":93404,"kb_used_data":62212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93651748,"statfs":{"total":95995035648,"available":95899389952,"internally_reserved":0,"allocated":63705088,"data_stored":79813074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986370,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105116,"kb_used_data":51716,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93640036,"statfs":{"total":95995035648,"available":95887396864,"internally_reserved":0,"allocated":52957184,"data_stored":58855796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149894,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":169276,"kb_used_data":120036,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93575876,"statfs":{"total":95995035648,"available":95821697024,"internally_reserved":0,"allocated":122916864,"data_stored":173195147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411532,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75712,"kb_used_data":44516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93669440,"statfs":{"total":95995035648,"available":95917506560,"internally_reserved":0,"allocated":45584384,"data_stored":47075660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575058,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":127172,"kb_used_data":73776,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93617980,"statfs":{"total":95995035648,"available":95864811520,"internally_reserved":0,"allocated":75546624,"data_stored":104917400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738583,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":150808,"kb_used_data":97408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93594344,"statfs":{"total":95995035648,"available":95840608256,"internally_reserved":0,"allocated":99745792,"data_stored":131029826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":89702400,"data_stored":124369341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65818624,"data_stored":98576348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35700736,"data_stored":40566388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":112898048,"data_stored":166551212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":43253760,"data_stored":52531238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53772288,"data_stored":73270774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":107261952,"data_stored":145205000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":160206848,"data_stored":222673978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":129503232,"data_stored":192868947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":56098816,"data_stored":66789341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":54689792,"data_stored":66913272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":69824512,"data_stored":100905932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:50:24.167 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-11T03:50:24.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 66 2024-09-11T03:50:24.187 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:24.225 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:24.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:24.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:24.327 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:24.349 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:24.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:24.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:24.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:24 smithi084 ceph-mon[27254]: pgmap v1362: 87 pgs: 87 active+clean; 543 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:24.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:24 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:50:24.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:24 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:50:24.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:24 smithi084 ceph-mon[27254]: from='client.17130 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:50:24.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:24 smithi084 ceph-mon[27254]: osdmap e806: 12 total, 12 up, 12 in 2024-09-11T03:50:24.451 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:24.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:24 smithi121 ceph-mon[30729]: pgmap v1362: 87 pgs: 87 active+clean; 543 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:24.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:24 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:50:24.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:24 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:50:24.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:24 smithi121 ceph-mon[30729]: from='client.17130 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:50:24.468 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:24 smithi121 ceph-mon[30729]: osdmap e806: 12 total, 12 up, 12 in 2024-09-11T03:50:24.477 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:24 smithi175 ceph-mon[30894]: pgmap v1362: 87 pgs: 87 active+clean; 543 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:24 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:50:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:24 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:50:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:24 smithi175 ceph-mon[30894]: from='client.17130 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:50:24.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:24 smithi175 ceph-mon[30894]: osdmap e806: 12 total, 12 up, 12 in 2024-09-11T03:50:24.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:24.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:24.567 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:50:24.568 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:50:24.579 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:24.601 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:24.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:24.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:24.703 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:24.728 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:24.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:24.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:24.830 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:24.853 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:24.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:24.868 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:24.883 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:24.955 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:24.981 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:24.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:24.982 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:25.048 INFO:tasks.rados.rados.2.smithi175.stdout:3304: done (3 left) 2024-09-11T03:50:25.048 INFO:tasks.rados.rados.2.smithi175.stdout:3306: done (2 left) 2024-09-11T03:50:25.048 INFO:tasks.rados.rados.2.smithi175.stdout:3308: done (1 left) 2024-09-11T03:50:25.048 INFO:tasks.rados.rados.2.smithi175.stdout:3309: done (0 left) 2024-09-11T03:50:25.048 INFO:tasks.rados.rados.2.smithi175.stdout:3310: rollback oid 38 current snap is 365 2024-09-11T03:50:25.049 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 38 to 344 2024-09-11T03:50:25.049 INFO:tasks.rados.rados.2.smithi175.stdout:3311: read oid 46 snap 364 2024-09-11T03:50:25.049 INFO:tasks.rados.rados.2.smithi175.stdout:3311: expect deleted 2024-09-11T03:50:25.049 INFO:tasks.rados.rados.2.smithi175.stdout:3312: read oid 30 snap 311 2024-09-11T03:50:25.049 INFO:tasks.rados.rados.2.smithi175.stdout:3312: expect (ObjNum 895 snap 284 seq_num 895) 2024-09-11T03:50:25.049 INFO:tasks.rados.rados.2.smithi175.stdout:3313: snap_remove snap 335 2024-09-11T03:50:25.083 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:25.108 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:25.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:25.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:25.211 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:25.231 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:25.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:25.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:25.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:25 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:50:25.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:25 smithi084 ceph-mon[27254]: 2.1 scrub ok 2024-09-11T03:50:25.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:25 smithi084 ceph-mon[27254]: osdmap e807: 12 total, 12 up, 12 in 2024-09-11T03:50:25.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T03:50:25.334 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:25.358 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:25.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:25.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:25.460 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:25 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:50:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:25 smithi175 ceph-mon[30894]: 2.1 scrub ok 2024-09-11T03:50:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:25 smithi175 ceph-mon[30894]: osdmap e807: 12 total, 12 up, 12 in 2024-09-11T03:50:25.482 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:25.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:25.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:25.514 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:50:25.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:25.604 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:25.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:25.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:25 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:50:25.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:25 smithi121 ceph-mon[30729]: 2.1 scrub ok 2024-09-11T03:50:25.635 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:25 smithi121 ceph-mon[30729]: osdmap e807: 12 total, 12 up, 12 in 2024-09-11T03:50:25.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:25.727 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:25.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:25.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:25.829 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:25.854 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:25.855 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:25.956 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:25.977 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:25.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:25.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:26.046 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 2 pg_num to 66 2024-09-11T03:50:26.046 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:50:26.052 INFO:tasks.rados.rados.2.smithi175.stdout:3313: done (3 left) 2024-09-11T03:50:26.052 INFO:tasks.rados.rados.2.smithi175.stdout:3314: snap_create 2024-09-11T03:50:26.052 INFO:tasks.rados.rados.2.smithi175.stdout:3310: finishing rollback tid 0 to smithi17557870-38 2024-09-11T03:50:26.053 INFO:tasks.rados.rados.2.smithi175.stdout:3310: finishing rollback tid 1 to smithi17557870-38 2024-09-11T03:50:26.053 INFO:tasks.rados.rados.2.smithi175.stdout:3310: finishing rollback tid 2 to smithi17557870-38 2024-09-11T03:50:26.053 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 1261 (ObjNum 795 snap 254 seq_num 795) dirty exists 2024-09-11T03:50:26.053 INFO:tasks.rados.rados.2.smithi175.stdout:3312: expect (ObjNum 895 snap 284 seq_num 895) 2024-09-11T03:50:26.079 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:26.103 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:26.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:26.104 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:26.175 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:50:26.176 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:50:26.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:26.226 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:26.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:26.328 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:26.349 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:26.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:26.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:26 smithi084 ceph-mon[27254]: 2.1f scrub starts 2024-09-11T03:50:26.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:26 smithi084 ceph-mon[27254]: 2.1f scrub ok 2024-09-11T03:50:26.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:26 smithi084 ceph-mon[27254]: pgmap v1365: 87 pgs: 87 active+clean; 543 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:26.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:26 smithi084 ceph-mon[27254]: 2.1d scrub starts 2024-09-11T03:50:26.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:26 smithi084 ceph-mon[27254]: 2.1d scrub ok 2024-09-11T03:50:26.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3522813930' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:50:26.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/69498479' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "66"} : dispatch 2024-09-11T03:50:26.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:26 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:26.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3522813930' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:50:26.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/69498479' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "66"}]': finished 2024-09-11T03:50:26.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:26 smithi084 ceph-mon[27254]: osdmap e808: 12 total, 12 up, 12 in 2024-09-11T03:50:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:26 smithi121 ceph-mon[30729]: 2.1f scrub starts 2024-09-11T03:50:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:26 smithi121 ceph-mon[30729]: 2.1f scrub ok 2024-09-11T03:50:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:26 smithi121 ceph-mon[30729]: pgmap v1365: 87 pgs: 87 active+clean; 543 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:26 smithi121 ceph-mon[30729]: 2.1d scrub starts 2024-09-11T03:50:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:26 smithi121 ceph-mon[30729]: 2.1d scrub ok 2024-09-11T03:50:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3522813930' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:50:26.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/69498479' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "66"} : dispatch 2024-09-11T03:50:26.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:26 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:26.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3522813930' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:50:26.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/69498479' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "66"}]': finished 2024-09-11T03:50:26.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:26 smithi121 ceph-mon[30729]: osdmap e808: 12 total, 12 up, 12 in 2024-09-11T03:50:26.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:26.473 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:26.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:26.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:26 smithi175 ceph-mon[30894]: 2.1f scrub starts 2024-09-11T03:50:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:26 smithi175 ceph-mon[30894]: 2.1f scrub ok 2024-09-11T03:50:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:26 smithi175 ceph-mon[30894]: pgmap v1365: 87 pgs: 87 active+clean; 543 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:26 smithi175 ceph-mon[30894]: 2.1d scrub starts 2024-09-11T03:50:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:26 smithi175 ceph-mon[30894]: 2.1d scrub ok 2024-09-11T03:50:26.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3522813930' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:50:26.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/69498479' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "66"} : dispatch 2024-09-11T03:50:26.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:26 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:26.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3522813930' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:50:26.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/69498479' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "66"}]': finished 2024-09-11T03:50:26.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:26 smithi175 ceph-mon[30894]: osdmap e808: 12 total, 12 up, 12 in 2024-09-11T03:50:26.575 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:26.597 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:26.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:26.598 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:26.699 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:26.721 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:26.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:26.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:26.756 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:50:26.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T03:50:26.823 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:26.843 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:26.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:26.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:26.945 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:26.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:26.970 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:26.970 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:27.027 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:50:27.035 INFO:tasks.rados.rados.2.smithi175.stdout:3310: done (3 left) 2024-09-11T03:50:27.035 INFO:tasks.rados.rados.2.smithi175.stdout:3311: done (2 left) 2024-09-11T03:50:27.035 INFO:tasks.rados.rados.2.smithi175.stdout:3312: done (1 left) 2024-09-11T03:50:27.035 INFO:tasks.rados.rados.2.smithi175.stdout:3314: done (0 left) 2024-09-11T03:50:27.035 INFO:tasks.rados.rados.2.smithi175.stdout:3315: delete oid 48 current snap is 366 2024-09-11T03:50:27.072 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:27.095 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:27.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:27.095 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:27.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:27.218 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:27.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:27.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:27.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:27 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:50:27.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:27 smithi084 ceph-mon[27254]: 2.0 scrub ok 2024-09-11T03:50:27.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "66"} : dispatch 2024-09-11T03:50:27.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]} : dispatch 2024-09-11T03:50:27.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 0]} : dispatch 2024-09-11T03:50:27.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]} : dispatch 2024-09-11T03:50:27.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "66"}]': finished 2024-09-11T03:50:27.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]': finished 2024-09-11T03:50:27.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 0]}]': finished 2024-09-11T03:50:27.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]': finished 2024-09-11T03:50:27.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:27 smithi084 ceph-mon[27254]: osdmap e809: 12 total, 12 up, 12 in 2024-09-11T03:50:27.320 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:27.341 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:27.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:27.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:27 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:50:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:27 smithi121 ceph-mon[30729]: 2.0 scrub ok 2024-09-11T03:50:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "66"} : dispatch 2024-09-11T03:50:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]} : dispatch 2024-09-11T03:50:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 0]} : dispatch 2024-09-11T03:50:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]} : dispatch 2024-09-11T03:50:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "66"}]': finished 2024-09-11T03:50:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]': finished 2024-09-11T03:50:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 0]}]': finished 2024-09-11T03:50:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]': finished 2024-09-11T03:50:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:27 smithi121 ceph-mon[30729]: osdmap e809: 12 total, 12 up, 12 in 2024-09-11T03:50:27.438 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:27.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:27.466 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:27.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:27.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:27 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:50:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:27 smithi175 ceph-mon[30894]: 2.0 scrub ok 2024-09-11T03:50:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "66"} : dispatch 2024-09-11T03:50:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]} : dispatch 2024-09-11T03:50:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 0]} : dispatch 2024-09-11T03:50:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]} : dispatch 2024-09-11T03:50:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "66"}]': finished 2024-09-11T03:50:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]': finished 2024-09-11T03:50:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 0]}]': finished 2024-09-11T03:50:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]': finished 2024-09-11T03:50:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:27 smithi175 ceph-mon[30894]: osdmap e809: 12 total, 12 up, 12 in 2024-09-11T03:50:27.532 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:50:27.532 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:50:27.568 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:27.589 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:27.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:27.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:27.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:27.713 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:27.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:27.814 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:27.840 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:27.841 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:27.941 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:27.967 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:27.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:27.968 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:28.013 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:50:28.013 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:50:28.019 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:50:28.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:50:28.039 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:50:28.039 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:50:28.045 INFO:tasks.rados.rados.2.smithi175.stdout:3315: done (0 left) 2024-09-11T03:50:28.045 INFO:tasks.rados.rados.2.smithi175.stdout:3316: read oid 25 snap -1 2024-09-11T03:50:28.045 INFO:tasks.rados.rados.2.smithi175.stdout:3316: expect deleted 2024-09-11T03:50:28.045 INFO:tasks.rados.rados.2.smithi175.stdout:3317: read oid 23 snap -1 2024-09-11T03:50:28.045 INFO:tasks.rados.rados.2.smithi175.stdout:3317: expect (ObjNum 1079 snap 352 seq_num 1079) 2024-09-11T03:50:28.045 INFO:tasks.rados.rados.2.smithi175.stdout:3318: write oid 13 current snap is 366 2024-09-11T03:50:28.045 INFO:tasks.rados.rados.2.smithi175.stdout:3318: seq_num 1127 ranges {541489=425867,1416030=740714,2732654=208835} 2024-09-11T03:50:28.049 INFO:tasks.rados.rados.2.smithi175.stdout:3318: writing smithi17557870-13 from 541489 to 967356 tid 1 2024-09-11T03:50:28.057 INFO:tasks.rados.rados.2.smithi175.stdout:3318: writing smithi17557870-13 from 1416030 to 2156744 tid 2 2024-09-11T03:50:28.057 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:50:28.058 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:50:28.061 INFO:tasks.rados.rados.2.smithi175.stdout:3318: writing smithi17557870-13 from 2732654 to 2941489 tid 3 2024-09-11T03:50:28.061 INFO:tasks.rados.rados.2.smithi175.stdout:3319: delete oid 11 current snap is 366 2024-09-11T03:50:28.062 INFO:tasks.rados.rados.2.smithi175.stdout:3317: expect (ObjNum 1079 snap 352 seq_num 1079) 2024-09-11T03:50:28.069 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:28.074 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:50:28.074 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:50:28.090 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:28.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:28.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:28.091 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:50:28.091 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:50:28.108 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:50:28.108 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:50:28.125 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:50:28.125 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:50:28.141 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:50:28.141 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:50:28.157 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:50:28.158 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:50:28.174 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:50:28.175 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:50:28.192 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:50:28.193 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:50:28.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:28.196 INFO:tasks.rados.rados.2.smithi175.stdout:3318: finishing write tid 1 to smithi17557870-13 2024-09-11T03:50:28.196 INFO:tasks.rados.rados.2.smithi175.stdout:3318: finishing write tid 2 to smithi17557870-13 2024-09-11T03:50:28.196 INFO:tasks.rados.rados.2.smithi175.stdout:3316: done (3 left) 2024-09-11T03:50:28.196 INFO:tasks.rados.rados.2.smithi175.stdout:3317: done (2 left) 2024-09-11T03:50:28.196 INFO:tasks.rados.rados.2.smithi175.stdout:3319: done (1 left) 2024-09-11T03:50:28.196 INFO:tasks.rados.rados.2.smithi175.stdout:3320: delete oid 24 current snap is 366 2024-09-11T03:50:28.197 INFO:tasks.rados.rados.2.smithi175.stdout:3318: finishing write tid 3 to smithi17557870-13 2024-09-11T03:50:28.197 INFO:tasks.rados.rados.2.smithi175.stdout:3318: finishing write tid 4 to smithi17557870-13 2024-09-11T03:50:28.197 INFO:tasks.rados.rados.2.smithi175.stdout:3318: finishing write tid 5 to smithi17557870-13 2024-09-11T03:50:28.197 INFO:tasks.rados.rados.2.smithi175.stdout:3318: oid 13 updating version 0 to 1271 2024-09-11T03:50:28.197 INFO:tasks.rados.rados.2.smithi175.stdout:3318: oid 13 version 1271 is already newer than 1270 2024-09-11T03:50:28.197 INFO:tasks.rados.rados.2.smithi175.stdout:3318: oid 13 updating version 1271 to 1272 2024-09-11T03:50:28.197 INFO:tasks.rados.rados.2.smithi175.stdout:3318: oid 13 version 1272 is already newer than 1269 2024-09-11T03:50:28.197 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 1272 (ObjNum 1127 snap 366 seq_num 1127) dirty exists 2024-09-11T03:50:28.198 INFO:tasks.rados.rados.2.smithi175.stdout:3318: left oid 13 (ObjNum 1127 snap 366 seq_num 1127) 2024-09-11T03:50:28.198 INFO:tasks.rados.rados.2.smithi175.stdout:3318: done (1 left) 2024-09-11T03:50:28.198 INFO:tasks.rados.rados.2.smithi175.stdout:3320: done (0 left) 2024-09-11T03:50:28.198 INFO:tasks.rados.rados.2.smithi175.stdout:3321: read oid 15 snap -1 2024-09-11T03:50:28.198 INFO:tasks.rados.rados.2.smithi175.stdout:3321: expect (ObjNum 1120 snap 359 seq_num 1120) 2024-09-11T03:50:28.198 INFO:tasks.rados.rados.2.smithi175.stdout:3322: delete oid 5 current snap is 366 2024-09-11T03:50:28.198 INFO:tasks.rados.rados.2.smithi175.stdout:3322: done (1 left) 2024-09-11T03:50:28.198 INFO:tasks.rados.rados.2.smithi175.stdout:3323: rollback oid 18 current snap is 366 2024-09-11T03:50:28.199 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 18 to 354 2024-09-11T03:50:28.199 INFO:tasks.rados.rados.2.smithi175.stdout:3324: snap_create 2024-09-11T03:50:28.200 INFO:tasks.rados.rados.2.smithi175.stdout:3323: finishing rollback tid 0 to smithi17557870-18 2024-09-11T03:50:28.200 INFO:tasks.rados.rados.2.smithi175.stdout:3323: finishing rollback tid 1 to smithi17557870-18 2024-09-11T03:50:28.200 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 937 (ObjNum 129 snap 0 seq_num 197922775) dirty dne 2024-09-11T03:50:28.203 INFO:tasks.rados.rados.2.smithi175.stdout:3321: expect (ObjNum 1120 snap 359 seq_num 1120) 2024-09-11T03:50:28.220 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:28.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:28.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:28.322 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:28.343 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:28.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:28.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:28.370 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:28 smithi121 ceph-mon[30729]: pgmap v1368: 87 pgs: 87 active+clean; 536 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1012 KiB/s wr, 4 op/s 2024-09-11T03:50:28.370 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:28 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3029253826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:28.370 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:28 smithi121 ceph-mon[30729]: osdmap e810: 12 total, 12 up, 12 in 2024-09-11T03:50:28.396 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-11T03:50:28.444 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:28.466 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:28.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:28 smithi175 ceph-mon[30894]: pgmap v1368: 87 pgs: 87 active+clean; 536 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1012 KiB/s wr, 4 op/s 2024-09-11T03:50:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:28 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3029253826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:28 smithi175 ceph-mon[30894]: osdmap e810: 12 total, 12 up, 12 in 2024-09-11T03:50:28.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:28.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:28 smithi084 ceph-mon[27254]: pgmap v1368: 87 pgs: 87 active+clean; 536 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1012 KiB/s wr, 4 op/s 2024-09-11T03:50:28.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:28 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3029253826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:28.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:28 smithi084 ceph-mon[27254]: osdmap e810: 12 total, 12 up, 12 in 2024-09-11T03:50:28.589 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:28.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:28.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:28.636 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:50:28.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:28.709 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:50:28.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:28.713 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:28.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:28.813 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:28.838 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:28.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:28.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:28.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:28.963 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:28.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:28.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:29.001 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:29.040 INFO:tasks.rados.rados.2.smithi175.stdout:3321: done (2 left) 2024-09-11T03:50:29.040 INFO:tasks.rados.rados.2.smithi175.stdout:3323: done (1 left) 2024-09-11T03:50:29.040 INFO:tasks.rados.rados.2.smithi175.stdout:3324: done (0 left) 2024-09-11T03:50:29.040 INFO:tasks.rados.rados.2.smithi175.stdout:3325: write oid 26 current snap is 367 2024-09-11T03:50:29.040 INFO:tasks.rados.rados.2.smithi175.stdout:3325: seq_num 1131 ranges {608717=726863,1933605=748052} 2024-09-11T03:50:29.047 INFO:tasks.rados.rados.2.smithi175.stdout:3325: writing smithi17557870-26 from 608717 to 1335580 tid 1 2024-09-11T03:50:29.055 INFO:tasks.rados.rados.2.smithi175.stdout:3325: writing smithi17557870-26 from 1933605 to 2681657 tid 2 2024-09-11T03:50:29.055 INFO:tasks.rados.rados.2.smithi175.stdout:3326: rollback oid 42 current snap is 367 2024-09-11T03:50:29.055 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 42 to 349 2024-09-11T03:50:29.055 INFO:tasks.rados.rados.2.smithi175.stdout:3325: finishing write tid 1 to smithi17557870-26 2024-09-11T03:50:29.056 INFO:tasks.rados.rados.2.smithi175.stdout:3327: write oid 47 current snap is 367 2024-09-11T03:50:29.056 INFO:tasks.rados.rados.2.smithi175.stdout:3327: seq_num 1132 ranges {625524=402112,1592087=608063,2869769=155755} 2024-09-11T03:50:29.061 INFO:tasks.rados.rados.2.smithi175.stdout:3327: writing smithi17557870-47 from 625524 to 1027636 tid 1 2024-09-11T03:50:29.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:29.067 INFO:tasks.rados.rados.2.smithi175.stdout:3327: writing smithi17557870-47 from 1592087 to 2200150 tid 2 2024-09-11T03:50:29.072 INFO:tasks.rados.rados.2.smithi175.stdout:3327: writing smithi17557870-47 from 2869769 to 3025524 tid 3 2024-09-11T03:50:29.072 INFO:tasks.rados.rados.2.smithi175.stdout:3328: write oid 30 current snap is 367 2024-09-11T03:50:29.072 INFO:tasks.rados.rados.2.smithi175.stdout:3326: finishing rollback tid 0 to smithi17557870-42 2024-09-11T03:50:29.072 INFO:tasks.rados.rados.2.smithi175.stdout:3328: seq_num 1133 ranges {642331=593714,1766922=751721} 2024-09-11T03:50:29.079 INFO:tasks.rados.rados.2.smithi175.stdout:3328: writing smithi17557870-30 from 642331 to 1236045 tid 1 2024-09-11T03:50:29.087 INFO:tasks.rados.rados.2.smithi175.stdout:3328: writing smithi17557870-30 from 1766922 to 2518643 tid 2 2024-09-11T03:50:29.087 INFO:tasks.rados.rados.2.smithi175.stdout:3329: write (excl) oid 12 current snap is 367 2024-09-11T03:50:29.087 INFO:tasks.rados.rados.2.smithi175.stdout:3326: finishing rollback tid 1 to smithi17557870-42 2024-09-11T03:50:29.087 INFO:tasks.rados.rados.2.smithi175.stdout:3326: finishing rollback tid 2 to smithi17557870-42 2024-09-11T03:50:29.087 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 1056 (ObjNum 1062 snap 347 seq_num 1062) dirty exists 2024-09-11T03:50:29.087 INFO:tasks.rados.rados.2.smithi175.stdout:3325: finishing write tid 2 to smithi17557870-26 2024-09-11T03:50:29.087 INFO:tasks.rados.rados.2.smithi175.stdout:3325: finishing write tid 3 to smithi17557870-26 2024-09-11T03:50:29.087 INFO:tasks.rados.rados.2.smithi175.stdout:3325: finishing write tid 4 to smithi17557870-26 2024-09-11T03:50:29.087 INFO:tasks.rados.rados.2.smithi175.stdout:3325: oid 26 updating version 0 to 1007 2024-09-11T03:50:29.088 INFO:tasks.rados.rados.2.smithi175.stdout:3325: oid 26 updating version 1007 to 1009 2024-09-11T03:50:29.088 INFO:tasks.rados.rados.2.smithi175.stdout:3325: oid 26 version 1009 is already newer than 1008 2024-09-11T03:50:29.088 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 1009 (ObjNum 1131 snap 367 seq_num 1131) dirty exists 2024-09-11T03:50:29.088 INFO:tasks.rados.rados.2.smithi175.stdout:3325: left oid 26 (ObjNum 1131 snap 367 seq_num 1131) 2024-09-11T03:50:29.088 INFO:tasks.rados.rados.2.smithi175.stdout:3327: finishing write tid 1 to smithi17557870-47 2024-09-11T03:50:29.088 INFO:tasks.rados.rados.2.smithi175.stdout:3327: finishing write tid 2 to smithi17557870-47 2024-09-11T03:50:29.088 INFO:tasks.rados.rados.2.smithi175.stdout:3329: seq_num 1134 ranges {659138=668963,2109051=611732} 2024-09-11T03:50:29.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:29.089 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:29.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:29.093 INFO:tasks.rados.rados.2.smithi175.stdout:3329: writing smithi17557870-12 from 659138 to 1328101 tid 1 2024-09-11T03:50:29.102 INFO:tasks.rados.rados.2.smithi175.stdout:3329: writing smithi17557870-12 from 2109051 to 2720783 tid 2 2024-09-11T03:50:29.102 INFO:tasks.rados.rados.2.smithi175.stdout:3325: done (4 left) 2024-09-11T03:50:29.102 INFO:tasks.rados.rados.2.smithi175.stdout:3326: done (3 left) 2024-09-11T03:50:29.102 INFO:tasks.rados.rados.2.smithi175.stdout:3330: read oid 18 snap -1 2024-09-11T03:50:29.102 INFO:tasks.rados.rados.2.smithi175.stdout:3330: expect deleted 2024-09-11T03:50:29.103 INFO:tasks.rados.rados.2.smithi175.stdout:3331: rollback oid 45 current snap is 367 2024-09-11T03:50:29.103 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 45 to 321 2024-09-11T03:50:29.103 INFO:tasks.rados.rados.2.smithi175.stdout:3327: finishing write tid 3 to smithi17557870-47 2024-09-11T03:50:29.103 INFO:tasks.rados.rados.2.smithi175.stdout:3327: finishing write tid 4 to smithi17557870-47 2024-09-11T03:50:29.103 INFO:tasks.rados.rados.2.smithi175.stdout:3327: finishing write tid 5 to smithi17557870-47 2024-09-11T03:50:29.103 INFO:tasks.rados.rados.2.smithi175.stdout:3327: oid 47 updating version 0 to 1103 2024-09-11T03:50:29.103 INFO:tasks.rados.rados.2.smithi175.stdout:3327: oid 47 updating version 1103 to 1105 2024-09-11T03:50:29.103 INFO:tasks.rados.rados.2.smithi175.stdout:3327: oid 47 updating version 1105 to 1106 2024-09-11T03:50:29.103 INFO:tasks.rados.rados.2.smithi175.stdout:3327: oid 47 version 1106 is already newer than 1104 2024-09-11T03:50:29.103 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 1106 (ObjNum 1132 snap 367 seq_num 1132) dirty exists 2024-09-11T03:50:29.103 INFO:tasks.rados.rados.2.smithi175.stdout:3327: left oid 47 (ObjNum 1132 snap 367 seq_num 1132) 2024-09-11T03:50:29.103 INFO:tasks.rados.rados.2.smithi175.stdout:3328: finishing write tid 1 to smithi17557870-30 2024-09-11T03:50:29.103 INFO:tasks.rados.rados.2.smithi175.stdout:3327: done (4 left) 2024-09-11T03:50:29.104 INFO:tasks.rados.rados.2.smithi175.stdout:3332: delete oid 4 current snap is 367 2024-09-11T03:50:29.104 INFO:tasks.rados.rados.2.smithi175.stdout:3328: finishing write tid 2 to smithi17557870-30 2024-09-11T03:50:29.104 INFO:tasks.rados.rados.2.smithi175.stdout:3328: finishing write tid 3 to smithi17557870-30 2024-09-11T03:50:29.104 INFO:tasks.rados.rados.2.smithi175.stdout:3328: finishing write tid 4 to smithi17557870-30 2024-09-11T03:50:29.104 INFO:tasks.rados.rados.2.smithi175.stdout:3328: oid 30 updating version 0 to 1061 2024-09-11T03:50:29.104 INFO:tasks.rados.rados.2.smithi175.stdout:3328: oid 30 updating version 1061 to 1063 2024-09-11T03:50:29.104 INFO:tasks.rados.rados.2.smithi175.stdout:3328: oid 30 version 1063 is already newer than 1062 2024-09-11T03:50:29.104 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 1063 (ObjNum 1133 snap 367 seq_num 1133) dirty exists 2024-09-11T03:50:29.104 INFO:tasks.rados.rados.2.smithi175.stdout:3328: left oid 30 (ObjNum 1133 snap 367 seq_num 1133) 2024-09-11T03:50:29.104 INFO:tasks.rados.rados.2.smithi175.stdout:3329: finishing write tid 1 to smithi17557870-12 2024-09-11T03:50:29.105 INFO:tasks.rados.rados.2.smithi175.stdout:3328: done (4 left) 2024-09-11T03:50:29.105 INFO:tasks.rados.rados.2.smithi175.stdout:3330: done (3 left) 2024-09-11T03:50:29.105 INFO:tasks.rados.rados.2.smithi175.stdout:3332: done (2 left) 2024-09-11T03:50:29.105 INFO:tasks.rados.rados.2.smithi175.stdout:3333: write (excl) oid 42 current snap is 367 2024-09-11T03:50:29.105 INFO:tasks.rados.rados.2.smithi175.stdout:3333: seq_num 1136 ranges {692752=419461,1826015=615401,2979461=113291} 2024-09-11T03:50:29.109 INFO:tasks.rados.rados.2.smithi175.stdout:3333: writing smithi17557870-42 from 692752 to 1112213 tid 1 2024-09-11T03:50:29.117 INFO:tasks.rados.rados.2.smithi175.stdout:3333: writing smithi17557870-42 from 1826015 to 2441416 tid 2 2024-09-11T03:50:29.122 INFO:tasks.rados.rados.2.smithi175.stdout:3333: writing smithi17557870-42 from 2979461 to 3092752 tid 3 2024-09-11T03:50:29.122 INFO:tasks.rados.rados.2.smithi175.stdout:3334: rollback oid 13 current snap is 367 2024-09-11T03:50:29.122 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 13 to 358 2024-09-11T03:50:29.122 INFO:tasks.rados.rados.2.smithi175.stdout:3331: finishing rollback tid 0 to smithi17557870-45 2024-09-11T03:50:29.122 INFO:tasks.rados.rados.2.smithi175.stdout:3331: finishing rollback tid 1 to smithi17557870-45 2024-09-11T03:50:29.122 INFO:tasks.rados.rados.2.smithi175.stdout:3329: finishing write tid 2 to smithi17557870-12 2024-09-11T03:50:29.122 INFO:tasks.rados.rados.2.smithi175.stdout:3329: finishing write tid 3 to smithi17557870-12 2024-09-11T03:50:29.122 INFO:tasks.rados.rados.2.smithi175.stdout:3331: finishing rollback tid 2 to smithi17557870-45 2024-09-11T03:50:29.123 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 1001 (ObjNum 962 snap 308 seq_num 962) dirty exists 2024-09-11T03:50:29.123 INFO:tasks.rados.rados.2.smithi175.stdout:3329: finishing write tid 4 to smithi17557870-12 2024-09-11T03:50:29.123 INFO:tasks.rados.rados.2.smithi175.stdout:3329: oid 12 updating version 0 to 1276 2024-09-11T03:50:29.123 INFO:tasks.rados.rados.2.smithi175.stdout:3329: oid 12 version 1276 is already newer than 1275 2024-09-11T03:50:29.123 INFO:tasks.rados.rados.2.smithi175.stdout:3329: oid 12 version 1276 is already newer than 1274 2024-09-11T03:50:29.123 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 1276 (ObjNum 1134 snap 367 seq_num 1134) dirty exists 2024-09-11T03:50:29.123 INFO:tasks.rados.rados.2.smithi175.stdout:3329: left oid 12 (ObjNum 1134 snap 367 seq_num 1134) 2024-09-11T03:50:29.123 INFO:tasks.rados.rados.2.smithi175.stdout:3333: finishing write tid 1 to smithi17557870-42 2024-09-11T03:50:29.123 INFO:tasks.rados.rados.2.smithi175.stdout:3333: finishing write tid 2 to smithi17557870-42 2024-09-11T03:50:29.123 INFO:tasks.rados.rados.2.smithi175.stdout:3329: done (3 left) 2024-09-11T03:50:29.123 INFO:tasks.rados.rados.2.smithi175.stdout:3331: done (2 left) 2024-09-11T03:50:29.123 INFO:tasks.rados.rados.2.smithi175.stdout:3335: read oid 22 snap 321 2024-09-11T03:50:29.123 INFO:tasks.rados.rados.2.smithi175.stdout:3335: expect (ObjNum 829 snap 263 seq_num 829) 2024-09-11T03:50:29.124 INFO:tasks.rados.rados.2.smithi175.stdout:3336: read oid 41 snap 348 2024-09-11T03:50:29.124 INFO:tasks.rados.rados.2.smithi175.stdout:3336: expect deleted 2024-09-11T03:50:29.124 INFO:tasks.rados.rados.2.smithi175.stdout:3337: write oid 21 current snap is 367 2024-09-11T03:50:29.124 INFO:tasks.rados.rados.2.smithi175.stdout:3337: seq_num 1137 ranges {709559=494710,1884497=759059} 2024-09-11T03:50:29.127 INFO:tasks.rados.rados.2.smithi175.stdout:3337: writing smithi17557870-21 from 709559 to 1204269 tid 1 2024-09-11T03:50:29.137 INFO:tasks.rados.rados.2.smithi175.stdout:3337: writing smithi17557870-21 from 1884497 to 2643556 tid 2 2024-09-11T03:50:29.137 INFO:tasks.rados.rados.2.smithi175.stdout:3338: rollback oid 20 current snap is 367 2024-09-11T03:50:29.137 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 20 to 344 2024-09-11T03:50:29.137 INFO:tasks.rados.rados.2.smithi175.stdout:3333: finishing write tid 3 to smithi17557870-42 2024-09-11T03:50:29.137 INFO:tasks.rados.rados.2.smithi175.stdout:3333: finishing write tid 4 to smithi17557870-42 2024-09-11T03:50:29.137 INFO:tasks.rados.rados.2.smithi175.stdout:3333: finishing write tid 5 to smithi17557870-42 2024-09-11T03:50:29.137 INFO:tasks.rados.rados.2.smithi175.stdout:3333: oid 42 updating version 0 to 1057 2024-09-11T03:50:29.137 INFO:tasks.rados.rados.2.smithi175.stdout:3333: oid 42 updating version 1057 to 1060 2024-09-11T03:50:29.137 INFO:tasks.rados.rados.2.smithi175.stdout:3333: oid 42 version 1060 is already newer than 1058 2024-09-11T03:50:29.137 INFO:tasks.rados.rados.2.smithi175.stdout:3333: oid 42 version 1060 is already newer than 1059 2024-09-11T03:50:29.138 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 1060 (ObjNum 1136 snap 367 seq_num 1136) dirty exists 2024-09-11T03:50:29.138 INFO:tasks.rados.rados.2.smithi175.stdout:3333: left oid 42 (ObjNum 1136 snap 367 seq_num 1136) 2024-09-11T03:50:29.138 INFO:tasks.rados.rados.2.smithi175.stdout:3334: finishing rollback tid 0 to smithi17557870-13 2024-09-11T03:50:29.138 INFO:tasks.rados.rados.2.smithi175.stdout:3334: finishing rollback tid 1 to smithi17557870-13 2024-09-11T03:50:29.138 INFO:tasks.rados.rados.2.smithi175.stdout:3334: finishing rollback tid 2 to smithi17557870-13 2024-09-11T03:50:29.138 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 1280 (ObjNum 907 snap 287 seq_num 907) dirty exists 2024-09-11T03:50:29.138 INFO:tasks.rados.rados.2.smithi175.stdout:3333: done (5 left) 2024-09-11T03:50:29.138 INFO:tasks.rados.rados.2.smithi175.stdout:3334: done (4 left) 2024-09-11T03:50:29.138 INFO:tasks.rados.rados.2.smithi175.stdout:3339: read oid 5 snap -1 2024-09-11T03:50:29.138 INFO:tasks.rados.rados.2.smithi175.stdout:3339: expect deleted 2024-09-11T03:50:29.138 INFO:tasks.rados.rados.2.smithi175.stdout:3340: read oid 36 snap 341 2024-09-11T03:50:29.138 INFO:tasks.rados.rados.2.smithi175.stdout:3340: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:50:29.138 INFO:tasks.rados.rados.2.smithi175.stdout:3341: rollback oid 9 current snap is 367 2024-09-11T03:50:29.138 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 9 to 340 2024-09-11T03:50:29.138 INFO:tasks.rados.rados.2.smithi175.stdout:3342: snap_remove snap 363 2024-09-11T03:50:29.185 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:50:29.186 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:50:29.186 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:50:29.186 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:50:29.190 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:29.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:29.214 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:29.214 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:29.315 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:29.336 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:29.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:29.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:29.438 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:29.460 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:29.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:29 smithi175 ceph-mon[30894]: 2.3b scrub starts 2024-09-11T03:50:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:29 smithi175 ceph-mon[30894]: 2.18 scrub starts 2024-09-11T03:50:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:29 smithi175 ceph-mon[30894]: 2.1a scrub starts 2024-09-11T03:50:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:29 smithi175 ceph-mon[30894]: 2.1d scrub starts 2024-09-11T03:50:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:29 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "66"} : dispatch 2024-09-11T03:50:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:29 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "66"}]': finished 2024-09-11T03:50:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:29 smithi175 ceph-mon[30894]: osdmap e811: 12 total, 12 up, 12 in 2024-09-11T03:50:29.505 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:29 smithi084 ceph-mon[27254]: 2.3b scrub starts 2024-09-11T03:50:29.505 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:29 smithi084 ceph-mon[27254]: 2.18 scrub starts 2024-09-11T03:50:29.505 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:29 smithi084 ceph-mon[27254]: 2.1a scrub starts 2024-09-11T03:50:29.505 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:29 smithi084 ceph-mon[27254]: 2.1d scrub starts 2024-09-11T03:50:29.505 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:29 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "66"} : dispatch 2024-09-11T03:50:29.505 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:29 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "66"}]': finished 2024-09-11T03:50:29.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:29 smithi084 ceph-mon[27254]: osdmap e811: 12 total, 12 up, 12 in 2024-09-11T03:50:29.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:29.584 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:29.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:29.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:29.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:29 smithi121 ceph-mon[30729]: 2.3b scrub starts 2024-09-11T03:50:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:29 smithi121 ceph-mon[30729]: 2.18 scrub starts 2024-09-11T03:50:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:29 smithi121 ceph-mon[30729]: 2.1a scrub starts 2024-09-11T03:50:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:29 smithi121 ceph-mon[30729]: 2.1d scrub starts 2024-09-11T03:50:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:29 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "66"} : dispatch 2024-09-11T03:50:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:29 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "66"}]': finished 2024-09-11T03:50:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:29 smithi121 ceph-mon[30729]: osdmap e811: 12 total, 12 up, 12 in 2024-09-11T03:50:29.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:29.706 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:29.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:29.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:29.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:29.829 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:29.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:29.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:29.874 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-11T03:50:29.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:29.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:29.953 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:29.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:30.035 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:50:30.045 INFO:tasks.rados.rados.2.smithi175.stdout:3342: done (7 left) 2024-09-11T03:50:30.045 INFO:tasks.rados.rados.2.smithi175.stdout:3343: delete oid 49 current snap is 367 2024-09-11T03:50:30.045 INFO:tasks.rados.rados.2.smithi175.stdout:3338: finishing rollback tid 1 to smithi17557870-20 2024-09-11T03:50:30.045 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 1002 (ObjNum 33 snap 0 seq_num 1713067256) dirty dne 2024-09-11T03:50:30.045 INFO:tasks.rados.rados.2.smithi175.stdout:3335: expect (ObjNum 829 snap 263 seq_num 829) 2024-09-11T03:50:30.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:30.076 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:30.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:30.128 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:50:30.177 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:30.198 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:30.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:30.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:30.238 INFO:tasks.rados.rados.2.smithi175.stdout:3337: finishing write tid 1 to smithi17557870-21 2024-09-11T03:50:30.238 INFO:tasks.rados.rados.2.smithi175.stdout:3340: expect (ObjNum 875 snap 275 seq_num 875) 2024-09-11T03:50:30.300 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:30.322 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:30.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:30.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:30 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:50:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:30 smithi084 ceph-mon[27254]: pgmap v1371: 97 pgs: 1 active+clean+scrubbing, 5 peering, 10 unknown, 81 active+clean; 518 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s, 0 objects/s recovering 2024-09-11T03:50:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:30 smithi084 ceph-mon[27254]: 2.18 scrub starts 2024-09-11T03:50:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:30 smithi084 ceph-mon[27254]: 2.1a scrub starts 2024-09-11T03:50:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:30 smithi084 ceph-mon[27254]: 2.1d scrub starts 2024-09-11T03:50:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:30 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4050574954' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:30 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4050574954' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:30 smithi084 ceph-mon[27254]: osdmap e812: 12 total, 12 up, 12 in 2024-09-11T03:50:30.424 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:30.445 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:30.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:30.446 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:30 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:50:30.523 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:30 smithi175 ceph-mon[30894]: pgmap v1371: 97 pgs: 1 active+clean+scrubbing, 5 peering, 10 unknown, 81 active+clean; 518 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s, 0 objects/s recovering 2024-09-11T03:50:30.524 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:30 smithi175 ceph-mon[30894]: 2.18 scrub starts 2024-09-11T03:50:30.524 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:30 smithi175 ceph-mon[30894]: 2.1a scrub starts 2024-09-11T03:50:30.524 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:30 smithi175 ceph-mon[30894]: 2.1d scrub starts 2024-09-11T03:50:30.524 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:30 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4050574954' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:30.524 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:30 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4050574954' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:30.524 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:30 smithi175 ceph-mon[30894]: osdmap e812: 12 total, 12 up, 12 in 2024-09-11T03:50:30.526 INFO:tasks.rados.rados.2.smithi175.stdout:3337: finishing write tid 2 to smithi17557870-21 2024-09-11T03:50:30.526 INFO:tasks.rados.rados.2.smithi175.stdout:3337: finishing write tid 3 to smithi17557870-21 2024-09-11T03:50:30.526 INFO:tasks.rados.rados.2.smithi175.stdout:3337: finishing write tid 4 to smithi17557870-21 2024-09-11T03:50:30.526 INFO:tasks.rados.rados.2.smithi175.stdout:3337: oid 21 updating version 0 to 1226 2024-09-11T03:50:30.526 INFO:tasks.rados.rados.2.smithi175.stdout:3337: oid 21 updating version 1226 to 1228 2024-09-11T03:50:30.526 INFO:tasks.rados.rados.2.smithi175.stdout:3337: oid 21 version 1228 is already newer than 1227 2024-09-11T03:50:30.526 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 1228 (ObjNum 1137 snap 367 seq_num 1137) dirty exists 2024-09-11T03:50:30.526 INFO:tasks.rados.rados.2.smithi175.stdout:3337: left oid 21 (ObjNum 1137 snap 367 seq_num 1137) 2024-09-11T03:50:30.526 INFO:tasks.rados.rados.2.smithi175.stdout:3341: finishing rollback tid 0 to smithi17557870-9 2024-09-11T03:50:30.527 INFO:tasks.rados.rados.2.smithi175.stdout:3341: finishing rollback tid 1 to smithi17557870-9 2024-09-11T03:50:30.527 INFO:tasks.rados.rados.2.smithi175.stdout:3341: finishing rollback tid 2 to smithi17557870-9 2024-09-11T03:50:30.527 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 1231 (ObjNum 1033 snap 340 seq_num 1033) dirty exists 2024-09-11T03:50:30.527 INFO:tasks.rados.rados.2.smithi175.stdout:3335: done (7 left) 2024-09-11T03:50:30.527 INFO:tasks.rados.rados.2.smithi175.stdout:3337: done (6 left) 2024-09-11T03:50:30.527 INFO:tasks.rados.rados.2.smithi175.stdout:3338: done (5 left) 2024-09-11T03:50:30.527 INFO:tasks.rados.rados.2.smithi175.stdout:3339: done (4 left) 2024-09-11T03:50:30.527 INFO:tasks.rados.rados.2.smithi175.stdout:3340: done (3 left) 2024-09-11T03:50:30.527 INFO:tasks.rados.rados.2.smithi175.stdout:3341: done (2 left) 2024-09-11T03:50:30.527 INFO:tasks.rados.rados.2.smithi175.stdout:3343: done (1 left) 2024-09-11T03:50:30.527 INFO:tasks.rados.rados.2.smithi175.stdout:3344: delete oid 9 current snap is 367 2024-09-11T03:50:30.527 INFO:tasks.rados.rados.2.smithi175.stdout:3344: done (1 left) 2024-09-11T03:50:30.527 INFO:tasks.rados.rados.2.smithi175.stdout:3345: delete oid 43 current snap is 367 2024-09-11T03:50:30.527 INFO:tasks.rados.rados.2.smithi175.stdout:3345: done (1 left) 2024-09-11T03:50:30.528 INFO:tasks.rados.rados.2.smithi175.stdout:3346: read oid 37 snap -1 2024-09-11T03:50:30.528 INFO:tasks.rados.rados.2.smithi175.stdout:3346: expect deleted 2024-09-11T03:50:30.528 INFO:tasks.rados.rados.2.smithi175.stdout:3347: delete oid 1 current snap is 367 2024-09-11T03:50:30.528 INFO:tasks.rados.rados.2.smithi175.stdout:3346: done (2 left) 2024-09-11T03:50:30.528 INFO:tasks.rados.rados.2.smithi175.stdout:3347: done (1 left) 2024-09-11T03:50:30.528 INFO:tasks.rados.rados.2.smithi175.stdout:3348: rollback oid 34 current snap is 367 2024-09-11T03:50:30.528 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 34 to 347 2024-09-11T03:50:30.528 INFO:tasks.rados.rados.2.smithi175.stdout:3349: snap_create 2024-09-11T03:50:30.528 INFO:tasks.rados.rados.2.smithi175.stdout:3348: finishing rollback tid 0 to smithi17557870-34 2024-09-11T03:50:30.528 INFO:tasks.rados.rados.2.smithi175.stdout:3348: finishing rollback tid 1 to smithi17557870-34 2024-09-11T03:50:30.528 INFO:tasks.rados.rados.2.smithi175.stdout:3348: finishing rollback tid 2 to smithi17557870-34 2024-09-11T03:50:30.528 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 1051 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:50:30.547 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:30.568 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:30.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:30.569 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:30.630 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:50:30.630 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:50:30.631 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:50:30.631 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:50:30.631 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:50:30.670 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:30 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:50:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:30 smithi121 ceph-mon[30729]: pgmap v1371: 97 pgs: 1 active+clean+scrubbing, 5 peering, 10 unknown, 81 active+clean; 518 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s, 0 objects/s recovering 2024-09-11T03:50:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:30 smithi121 ceph-mon[30729]: 2.18 scrub starts 2024-09-11T03:50:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:30 smithi121 ceph-mon[30729]: 2.1a scrub starts 2024-09-11T03:50:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:30 smithi121 ceph-mon[30729]: 2.1d scrub starts 2024-09-11T03:50:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:30 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4050574954' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:30 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4050574954' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:30 smithi121 ceph-mon[30729]: osdmap e812: 12 total, 12 up, 12 in 2024-09-11T03:50:30.692 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:30.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:30.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:30.794 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:30.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:30.818 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:30.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:30.919 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:30.941 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:30.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:30.974 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:50:31.042 INFO:tasks.rados.rados.2.smithi175.stdout:3348: done (2 left) 2024-09-11T03:50:31.042 INFO:tasks.rados.rados.2.smithi175.stdout:3349: done (1 left) 2024-09-11T03:50:31.043 INFO:tasks.rados.rados.2.smithi175.stdout:3350: rollback oid 39 current snap is 368 2024-09-11T03:50:31.043 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 39 to 314 2024-09-11T03:50:31.043 INFO:tasks.rados.rados.2.smithi175.stdout:3351: snap_remove snap 362 2024-09-11T03:50:31.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:31.069 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:31.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:31.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:31.172 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:31.192 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:31.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:31.240 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:31.294 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:31.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:31.317 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:31.317 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:31.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:31.440 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:31.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:31.485 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:31 smithi175 ceph-mon[30894]: 2.1f scrub starts 2024-09-11T03:50:31.485 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:31 smithi175 ceph-mon[30894]: 2.18 scrub starts 2024-09-11T03:50:31.485 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:31 smithi175 ceph-mon[30894]: 2.19 scrub starts 2024-09-11T03:50:31.485 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:31 smithi175 ceph-mon[30894]: 2.1d scrub starts 2024-09-11T03:50:31.485 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:31 smithi175 ceph-mon[30894]: osdmap e813: 12 total, 12 up, 12 in 2024-09-11T03:50:31.486 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-11T03:50:31.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:31 smithi084 ceph-mon[27254]: 2.1f scrub starts 2024-09-11T03:50:31.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:31 smithi084 ceph-mon[27254]: 2.18 scrub starts 2024-09-11T03:50:31.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:31 smithi084 ceph-mon[27254]: 2.19 scrub starts 2024-09-11T03:50:31.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:31 smithi084 ceph-mon[27254]: 2.1d scrub starts 2024-09-11T03:50:31.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:31 smithi084 ceph-mon[27254]: osdmap e813: 12 total, 12 up, 12 in 2024-09-11T03:50:31.542 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:31.571 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:31.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:31.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:31.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:31 smithi121 ceph-mon[30729]: 2.1f scrub starts 2024-09-11T03:50:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:31 smithi121 ceph-mon[30729]: 2.18 scrub starts 2024-09-11T03:50:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:31 smithi121 ceph-mon[30729]: 2.19 scrub starts 2024-09-11T03:50:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:31 smithi121 ceph-mon[30729]: 2.1d scrub starts 2024-09-11T03:50:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:31 smithi121 ceph-mon[30729]: osdmap e813: 12 total, 12 up, 12 in 2024-09-11T03:50:31.672 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:31.694 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:50:31.694 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:50:31.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:31.695 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:31.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:31.711 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:50:31.711 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:50:31.727 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:50:31.728 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:50:31.739 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:50:31.743 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:50:31.743 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:50:31.759 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:50:31.760 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:50:31.776 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:50:31.776 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:50:31.792 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:50:31.793 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:50:31.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:31.809 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:50:31.810 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:50:31.817 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:31.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:31.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:31.826 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:50:31.827 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:50:31.843 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:50:31.844 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:50:31.860 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:50:31.861 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:50:31.877 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:50:31.877 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:50:31.919 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:31.940 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:31.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:31.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:32.038 INFO:tasks.rados.rados.2.smithi175.stdout:3351: done (2 left) 2024-09-11T03:50:32.038 INFO:tasks.rados.rados.2.smithi175.stdout:3352: copy_from oid 19 from oid 18 current snap is 368 2024-09-11T03:50:32.038 INFO:tasks.rados.rados.2.smithi175.stdout:3353: delete oid 31 current snap is 368 2024-09-11T03:50:32.038 INFO:tasks.rados.rados.2.smithi175.stdout:3350: finishing rollback tid 1 to smithi17557870-39 2024-09-11T03:50:32.038 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 1005 (ObjNum 1746358592 snap 22076 seq_num 304) dirty dne 2024-09-11T03:50:32.039 INFO:tasks.rados.rados.2.smithi175.stdout:3352: finishing copy_from to smithi17557870-19 2024-09-11T03:50:32.039 INFO:tasks.rados.rados.2.smithi175.stdout:3352: got expected ENOENT (src dne) 2024-09-11T03:50:32.039 INFO:tasks.rados.rados.2.smithi175.stdout:3352: finishing copy_from racing read to smithi17557870-19 2024-09-11T03:50:32.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:32.050 INFO:tasks.rados.rados.2.smithi175.stdout:3336: done (3 left) 2024-09-11T03:50:32.050 INFO:tasks.rados.rados.2.smithi175.stdout:3350: done (2 left) 2024-09-11T03:50:32.051 INFO:tasks.rados.rados.2.smithi175.stdout:3352: done (1 left) 2024-09-11T03:50:32.051 INFO:tasks.rados.rados.2.smithi175.stdout:3353: done (0 left) 2024-09-11T03:50:32.051 INFO:tasks.rados.rados.2.smithi175.stdout:3354: read oid 43 snap -1 2024-09-11T03:50:32.051 INFO:tasks.rados.rados.2.smithi175.stdout:3354: expect deleted 2024-09-11T03:50:32.051 INFO:tasks.rados.rados.2.smithi175.stdout:3355: rollback oid 42 current snap is 368 2024-09-11T03:50:32.051 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 42 to 341 2024-09-11T03:50:32.051 INFO:tasks.rados.rados.2.smithi175.stdout:3356: write oid 27 current snap is 368 2024-09-11T03:50:32.051 INFO:tasks.rados.rados.2.smithi175.stdout:3356: seq_num 1143 ranges {410401=662557,1719036=770066,2938716=271685} 2024-09-11T03:50:32.055 INFO:tasks.rados.rados.2.smithi175.stdout:3356: writing smithi17557870-27 from 410401 to 1072958 tid 1 2024-09-11T03:50:32.063 INFO:tasks.rados.rados.2.smithi175.stdout:3356: writing smithi17557870-27 from 1719036 to 2489102 tid 2 2024-09-11T03:50:32.066 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:32.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:32.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:32.068 INFO:tasks.rados.rados.2.smithi175.stdout:3356: writing smithi17557870-27 from 2938716 to 3210401 tid 3 2024-09-11T03:50:32.068 INFO:tasks.rados.rados.2.smithi175.stdout:3357: read oid 39 snap -1 2024-09-11T03:50:32.068 INFO:tasks.rados.rados.2.smithi175.stdout:3357: expect deleted 2024-09-11T03:50:32.068 INFO:tasks.rados.rados.2.smithi175.stdout:3358: snap_create 2024-09-11T03:50:32.068 INFO:tasks.rados.rados.2.smithi175.stdout:3355: finishing rollback tid 0 to smithi17557870-42 2024-09-11T03:50:32.068 INFO:tasks.rados.rados.2.smithi175.stdout:3355: finishing rollback tid 1 to smithi17557870-42 2024-09-11T03:50:32.068 INFO:tasks.rados.rados.2.smithi175.stdout:3355: finishing rollback tid 2 to smithi17557870-42 2024-09-11T03:50:32.068 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 1063 (ObjNum 1042 snap 340 seq_num 1042) dirty exists 2024-09-11T03:50:32.068 INFO:tasks.rados.rados.2.smithi175.stdout:3356: finishing write tid 1 to smithi17557870-27 2024-09-11T03:50:32.068 INFO:tasks.rados.rados.2.smithi175.stdout:3356: finishing write tid 2 to smithi17557870-27 2024-09-11T03:50:32.069 INFO:tasks.rados.rados.2.smithi175.stdout:3356: finishing write tid 3 to smithi17557870-27 2024-09-11T03:50:32.069 INFO:tasks.rados.rados.2.smithi175.stdout:3356: finishing write tid 4 to smithi17557870-27 2024-09-11T03:50:32.069 INFO:tasks.rados.rados.2.smithi175.stdout:3356: finishing write tid 5 to smithi17557870-27 2024-09-11T03:50:32.069 INFO:tasks.rados.rados.2.smithi175.stdout:3356: oid 27 updating version 0 to 1121 2024-09-11T03:50:32.069 INFO:tasks.rados.rados.2.smithi175.stdout:3356: oid 27 updating version 1121 to 1124 2024-09-11T03:50:32.070 INFO:tasks.rados.rados.2.smithi175.stdout:3356: oid 27 version 1124 is already newer than 1123 2024-09-11T03:50:32.070 INFO:tasks.rados.rados.2.smithi175.stdout:3356: oid 27 version 1124 is already newer than 1122 2024-09-11T03:50:32.070 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 1124 (ObjNum 1143 snap 368 seq_num 1143) dirty exists 2024-09-11T03:50:32.070 INFO:tasks.rados.rados.2.smithi175.stdout:3356: left oid 27 (ObjNum 1143 snap 368 seq_num 1143) 2024-09-11T03:50:32.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:32.190 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:32.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:32.213 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:50:32.214 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:50:32.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:32.312 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:32.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:32.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:32.348 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:50:32.348 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:50:32.348 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-09-11T03:50:32.349 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:50:32.414 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:32.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:32.437 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:32.437 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:32 smithi175 ceph-mon[30894]: 2.1e scrub starts 2024-09-11T03:50:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:32 smithi175 ceph-mon[30894]: pgmap v1374: 97 pgs: 1 active+clean+scrubbing, 5 peering, 1 remapped+peering, 10 unknown, 80 active+clean; 518 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s, 0 objects/s recovering 2024-09-11T03:50:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:32 smithi175 ceph-mon[30894]: 2.1b scrub starts 2024-09-11T03:50:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:32 smithi175 ceph-mon[30894]: 2.18 scrub starts 2024-09-11T03:50:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:32 smithi175 ceph-mon[30894]: 2.1a scrub starts 2024-09-11T03:50:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:32 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2276784814' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:32 smithi175 ceph-mon[30894]: 2.1c scrub starts 2024-09-11T03:50:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:32 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:32 smithi175 ceph-mon[30894]: osdmap e814: 12 total, 12 up, 12 in 2024-09-11T03:50:32.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:32.560 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:32.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:32.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:32 smithi084 ceph-mon[27254]: 2.1e scrub starts 2024-09-11T03:50:32.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:32 smithi084 ceph-mon[27254]: pgmap v1374: 97 pgs: 1 active+clean+scrubbing, 5 peering, 1 remapped+peering, 10 unknown, 80 active+clean; 518 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s, 0 objects/s recovering 2024-09-11T03:50:32.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:32 smithi084 ceph-mon[27254]: 2.1b scrub starts 2024-09-11T03:50:32.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:32 smithi084 ceph-mon[27254]: 2.18 scrub starts 2024-09-11T03:50:32.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:32 smithi084 ceph-mon[27254]: 2.1a scrub starts 2024-09-11T03:50:32.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:32 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2276784814' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:32.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:32 smithi084 ceph-mon[27254]: 2.1c scrub starts 2024-09-11T03:50:32.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:32 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:32.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:32 smithi084 ceph-mon[27254]: osdmap e814: 12 total, 12 up, 12 in 2024-09-11T03:50:32.662 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:32.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:32 smithi121 ceph-mon[30729]: 2.1e scrub starts 2024-09-11T03:50:32.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:32 smithi121 ceph-mon[30729]: pgmap v1374: 97 pgs: 1 active+clean+scrubbing, 5 peering, 1 remapped+peering, 10 unknown, 80 active+clean; 518 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s, 0 objects/s recovering 2024-09-11T03:50:32.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:32 smithi121 ceph-mon[30729]: 2.1b scrub starts 2024-09-11T03:50:32.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:32 smithi121 ceph-mon[30729]: 2.18 scrub starts 2024-09-11T03:50:32.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:32 smithi121 ceph-mon[30729]: 2.1a scrub starts 2024-09-11T03:50:32.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:32 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2276784814' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:32.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:32 smithi121 ceph-mon[30729]: 2.1c scrub starts 2024-09-11T03:50:32.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:32 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:32.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:32 smithi121 ceph-mon[30729]: osdmap e814: 12 total, 12 up, 12 in 2024-09-11T03:50:32.684 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:32.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:32.786 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:32.808 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:32.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:32.838 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:50:32.910 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:32.922 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:32.933 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:32.934 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:33.035 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:33.038 INFO:tasks.rados.rados.2.smithi175.stdout:3354: done (4 left) 2024-09-11T03:50:33.038 INFO:tasks.rados.rados.2.smithi175.stdout:3355: done (3 left) 2024-09-11T03:50:33.039 INFO:tasks.rados.rados.2.smithi175.stdout:3356: done (2 left) 2024-09-11T03:50:33.039 INFO:tasks.rados.rados.2.smithi175.stdout:3357: done (1 left) 2024-09-11T03:50:33.039 INFO:tasks.rados.rados.2.smithi175.stdout:3358: done (0 left) 2024-09-11T03:50:33.039 INFO:tasks.rados.rados.2.smithi175.stdout:3359: read oid 19 snap -1 2024-09-11T03:50:33.039 INFO:tasks.rados.rados.2.smithi175.stdout:3359: expect (ObjNum 963 snap 310 seq_num 963) 2024-09-11T03:50:33.039 INFO:tasks.rados.rados.2.smithi175.stdout:3360: snap_remove snap 352 2024-09-11T03:50:33.062 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:33.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:33.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:33.095 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-11T03:50:33.164 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:33.185 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:33.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:33.186 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:33.287 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:33.310 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:33.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:33.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:33.340 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:50:33.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:33 smithi084 ceph-mon[27254]: 2.1f scrub starts 2024-09-11T03:50:33.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:33 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:50:33.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:33 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:50:33.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:33 smithi084 ceph-mon[27254]: 2.1b scrub starts 2024-09-11T03:50:33.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:33 smithi084 ceph-mon[27254]: 2.18 scrub starts 2024-09-11T03:50:33.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:33 smithi084 ceph-mon[27254]: 2.1a scrub starts 2024-09-11T03:50:33.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:33 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:50:33.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:33 smithi084 ceph-mon[27254]: osdmap e815: 12 total, 12 up, 12 in 2024-09-11T03:50:33.380 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:33.412 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:33.433 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:33.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:33.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:33.535 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:33.549 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:50:33.551 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:50:33.558 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:33.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:33.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:33.589 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:33 smithi175 ceph-mon[30894]: 2.1f scrub starts 2024-09-11T03:50:33.589 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:33 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:50:33.590 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:33 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:50:33.590 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:33 smithi175 ceph-mon[30894]: 2.1b scrub starts 2024-09-11T03:50:33.590 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:33 smithi175 ceph-mon[30894]: 2.18 scrub starts 2024-09-11T03:50:33.590 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:33 smithi175 ceph-mon[30894]: 2.1a scrub starts 2024-09-11T03:50:33.590 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:33 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:50:33.590 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:33 smithi175 ceph-mon[30894]: osdmap e815: 12 total, 12 up, 12 in 2024-09-11T03:50:33.660 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:33.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:33 smithi121 ceph-mon[30729]: 2.1f scrub starts 2024-09-11T03:50:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:33 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:50:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:33 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:50:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:33 smithi121 ceph-mon[30729]: 2.1b scrub starts 2024-09-11T03:50:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:33 smithi121 ceph-mon[30729]: 2.18 scrub starts 2024-09-11T03:50:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:33 smithi121 ceph-mon[30729]: 2.1a scrub starts 2024-09-11T03:50:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:33 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:50:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:33 smithi121 ceph-mon[30729]: osdmap e815: 12 total, 12 up, 12 in 2024-09-11T03:50:33.681 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:33.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:33.682 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:33.783 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:33.805 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:33.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:33.829 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:50:33.829 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:50:33.906 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:33.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:33.928 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:33.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:34.029 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:34.047 INFO:tasks.rados.rados.2.smithi175.stdout:3360: done (1 left) 2024-09-11T03:50:34.047 INFO:tasks.rados.rados.2.smithi175.stdout:3361: snap_create 2024-09-11T03:50:34.047 INFO:tasks.rados.rados.2.smithi175.stdout:3359: expect (ObjNum 963 snap 310 seq_num 963) 2024-09-11T03:50:34.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:34.061 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:34.061 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:34.162 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:34.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:34.184 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:34.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:34.285 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:34.306 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:34.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:34.307 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:34.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:34 smithi084 ceph-mon[27254]: 2.1f scrub starts 2024-09-11T03:50:34.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:34 smithi084 ceph-mon[27254]: pgmap v1377: 97 pgs: 3 peering, 2 active+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 activating+undersized+remapped, 1 active+clean+scrubbing, 1 remapped+peering, 1 unknown, 86 active+clean; 548 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s rd, 10 MiB/s wr, 76 op/s; 142/831 objects degraded (17.088%); 52/831 objects misplaced (6.258%); 6.3 MiB/s, 24 objects/s recovering 2024-09-11T03:50:34.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:34 smithi084 ceph-mon[27254]: 2.1b scrub starts 2024-09-11T03:50:34.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:34 smithi084 ceph-mon[27254]: Health check failed: Degraded data redundancy: 142/831 objects degraded (17.088%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T03:50:34.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:34 smithi084 ceph-mon[27254]: 2.18 scrub starts 2024-09-11T03:50:34.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:34 smithi084 ceph-mon[27254]: 2.38 scrub starts 2024-09-11T03:50:34.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:34 smithi084 ceph-mon[27254]: 2.3b scrub starts 2024-09-11T03:50:34.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:34 smithi084 ceph-mon[27254]: 2.1a scrub starts 2024-09-11T03:50:34.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:34 smithi084 ceph-mon[27254]: from='client.17166 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:50:34.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:34 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:50:34.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:34 smithi084 ceph-mon[27254]: osdmap e816: 12 total, 12 up, 12 in 2024-09-11T03:50:34.408 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:34.429 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:34.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:34 smithi175 ceph-mon[30894]: 2.1f scrub starts 2024-09-11T03:50:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:34 smithi175 ceph-mon[30894]: pgmap v1377: 97 pgs: 3 peering, 2 active+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 activating+undersized+remapped, 1 active+clean+scrubbing, 1 remapped+peering, 1 unknown, 86 active+clean; 548 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s rd, 10 MiB/s wr, 76 op/s; 142/831 objects degraded (17.088%); 52/831 objects misplaced (6.258%); 6.3 MiB/s, 24 objects/s recovering 2024-09-11T03:50:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:34 smithi175 ceph-mon[30894]: 2.1b scrub starts 2024-09-11T03:50:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:34 smithi175 ceph-mon[30894]: Health check failed: Degraded data redundancy: 142/831 objects degraded (17.088%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T03:50:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:34 smithi175 ceph-mon[30894]: 2.18 scrub starts 2024-09-11T03:50:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:34 smithi175 ceph-mon[30894]: 2.38 scrub starts 2024-09-11T03:50:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:34 smithi175 ceph-mon[30894]: 2.3b scrub starts 2024-09-11T03:50:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:34 smithi175 ceph-mon[30894]: 2.1a scrub starts 2024-09-11T03:50:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:34 smithi175 ceph-mon[30894]: from='client.17166 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:50:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:34 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:50:34.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:34 smithi175 ceph-mon[30894]: osdmap e816: 12 total, 12 up, 12 in 2024-09-11T03:50:34.531 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:34.551 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:34.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:34.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:34.629 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-11T03:50:34.653 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:34 smithi121 ceph-mon[30729]: 2.1f scrub starts 2024-09-11T03:50:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:34 smithi121 ceph-mon[30729]: pgmap v1377: 97 pgs: 3 peering, 2 active+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 activating+undersized+remapped, 1 active+clean+scrubbing, 1 remapped+peering, 1 unknown, 86 active+clean; 548 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s rd, 10 MiB/s wr, 76 op/s; 142/831 objects degraded (17.088%); 52/831 objects misplaced (6.258%); 6.3 MiB/s, 24 objects/s recovering 2024-09-11T03:50:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:34 smithi121 ceph-mon[30729]: 2.1b scrub starts 2024-09-11T03:50:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:34 smithi121 ceph-mon[30729]: Health check failed: Degraded data redundancy: 142/831 objects degraded (17.088%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T03:50:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:34 smithi121 ceph-mon[30729]: 2.18 scrub starts 2024-09-11T03:50:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:34 smithi121 ceph-mon[30729]: 2.38 scrub starts 2024-09-11T03:50:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:34 smithi121 ceph-mon[30729]: 2.3b scrub starts 2024-09-11T03:50:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:34 smithi121 ceph-mon[30729]: 2.1a scrub starts 2024-09-11T03:50:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:34 smithi121 ceph-mon[30729]: from='client.17166 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:50:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:34 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:50:34.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:34 smithi121 ceph-mon[30729]: osdmap e816: 12 total, 12 up, 12 in 2024-09-11T03:50:34.674 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:34.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:34.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:34.729 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1378,"stamp":"2024-09-11T03:50:33.035304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":574413609,"num_objects":277,"num_object_clones":225,"num_object_copies":831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":142,"num_objects_misplaced":52,"num_objects_unfound":0,"num_objects_dirty":277,"num_whiteouts":26,"num_read":1337,"num_read_kb":1423013,"num_write":900,"num_write_kb":376007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":37426308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40549,"ondisk_log_size":40549,"up":291,"acting":285,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":288,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":1598544,"kb_used_data":1081264,"kb_used_omap":18,"kb_used_meta":516973,"kb_avail":1123343280,"statfs":{"total":1151940427776,"available":1150303518720,"internally_reserved":0,"allocated":1107214336,"data_stored":1450309424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":529380667},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31662190,"num_objects":24,"num_object_clones":23,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":142,"num_objects_misplaced":52,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":9,"num_read":168,"num_read_kb":167914,"num_write":140,"num_write_kb":42721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":26265285,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.998216"},"pg_stats":[{"pgid":"2.41","version":"567'235","reported_seq":3803,"reported_epoch":814,"state":"peering","last_fresh":"2024-09-11T03:50:31.583809+0000","last_change":"2024-09-11T03:50:32.401371+0000","last_active":"2024-09-11T03:50:32.401371+0000","last_peered":"2024-09-11T03:50:31.583809+0000","last_clean":"2024-09-11T03:50:28.041364+0000","last_became_active":"2024-09-11T03:50:31.583592+0000","last_became_peered":"2024-09-11T03:50:31.583592+0000","last_unstale":"2024-09-11T03:50:31.583809+0000","last_undegraded":"2024-09-11T03:50:31.583809+0000","last_fullsized":"2024-09-11T03:50:31.040689+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":7,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:50:23.836407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:23.836407+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"2.40","version":"563'192","reported_seq":3862,"reported_epoch":814,"state":"peering","last_fresh":"2024-09-11T03:50:31.043319+0000","last_change":"2024-09-11T03:50:32.276544+0000","last_active":"2024-09-11T03:50:28.037669+0000","last_peered":"2024-09-11T03:50:26.059346+0000","last_clean":"2024-09-11T03:50:26.059346+0000","last_became_active":"2024-09-11T03:46:27.974405+0000","last_became_peered":"2024-09-11T03:46:27.974405+0000","last_unstale":"2024-09-11T03:50:31.043319+0000","last_undegraded":"2024-09-11T03:50:31.043319+0000","last_fullsized":"2024-09-11T03:50:31.043319+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":7,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:50:25.721409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:25.721409+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"2.3f","version":"808'756","reported_seq":4296,"reported_epoch":813,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-09-11T03:50:31.050935+0000","last_change":"2024-09-11T03:50:31.050935+0000","last_active":"2024-09-11T03:50:29.038630+0000","last_peered":"2024-09-11T03:50:26.055720+0000","last_clean":"2024-09-11T03:50:26.055720+0000","last_became_active":"2024-09-11T03:46:32.968696+0000","last_became_peered":"2024-09-11T03:46:32.968696+0000","last_unstale":"2024-09-11T03:50:31.050935+0000","last_undegraded":"2024-09-11T03:50:31.044754+0000","last_fullsized":"2024-09-11T03:50:31.040862+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":6,"last_scrub":"806'752","last_scrub_stamp":"2024-09-11T03:50:24.822925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:24.822925+0000","objects_scrubbed":19,"log_size":756,"log_dups_size":0,"ondisk_log_size":756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20585613,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":69,"num_read_kb":75589,"num_write":47,"num_write_kb":16761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,3],"avail_no_missing":["7","0","8"],"object_location_counts":[{"shards":"0,7,8","objects":6},{"shards":"3,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"2.3e","version":"810'1062","reported_seq":4630,"reported_epoch":814,"state":"peering","last_fresh":"2024-09-11T03:50:31.144995+0000","last_change":"2024-09-11T03:50:32.051679+0000","last_active":"2024-09-11T03:50:32.051679+0000","last_peered":"2024-09-11T03:50:31.144995+0000","last_clean":"2024-09-11T03:50:28.195322+0000","last_became_active":"2024-09-11T03:50:31.053835+0000","last_became_peered":"2024-09-11T03:50:31.053835+0000","last_unstale":"2024-09-11T03:50:31.144995+0000","last_undegraded":"2024-09-11T03:50:31.144995+0000","last_fullsized":"2024-09-11T03:50:31.041139+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":6,"last_scrub":"785'1044","last_scrub_stamp":"2024-09-11T03:50:03.954442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:03.954442+0000","objects_scrubbed":9,"log_size":1062,"log_dups_size":0,"ondisk_log_size":1062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10069114,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":31,"num_read_kb":25973,"num_write":48,"num_write_kb":17498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":11723750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"2.3d","version":"567'752","reported_seq":3138,"reported_epoch":813,"state":"active+undersized+remapped","last_fresh":"2024-09-11T03:50:31.048377+0000","last_change":"2024-09-11T03:50:31.048037+0000","last_active":"2024-09-11T03:50:31.048377+0000","last_peered":"2024-09-11T03:50:31.048377+0000","last_clean":"2024-09-11T03:46:24.984087+0000","last_became_active":"2024-09-11T03:50:31.048037+0000","last_became_peered":"2024-09-11T03:50:31.048037+0000","last_unstale":"2024-09-11T03:50:31.048377+0000","last_undegraded":"2024-09-11T03:50:31.048377+0000","last_fullsized":"2024-09-11T03:50:31.040840+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":6,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:50:25.778626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:25.778626+0000","objects_scrubbed":4,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2219454,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":78,"num_read_kb":70484,"num_write":57,"num_write_kb":20660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4365206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[7,1],"avail_no_missing":["7","1","4","8","11"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":7,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"2.3c","version":"806'776","reported_seq":4391,"reported_epoch":813,"state":"active+undersized+remapped","last_fresh":"2024-09-11T03:50:31.096160+0000","last_change":"2024-09-11T03:50:31.096160+0000","last_active":"2024-09-11T03:50:31.096160+0000","last_peered":"2024-09-11T03:50:31.096160+0000","last_clean":"2024-09-11T03:50:24.051702+0000","last_became_active":"2024-09-11T03:50:31.050568+0000","last_became_peered":"2024-09-11T03:50:31.050568+0000","last_unstale":"2024-09-11T03:50:31.096160+0000","last_undegraded":"2024-09-11T03:50:31.096160+0000","last_fullsized":"2024-09-11T03:50:31.040847+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'760","last_scrub_stamp":"2024-09-11T03:49:33.579794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.579794+0000","objects_scrubbed":5,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4367536,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":18,"num_read_kb":17540,"num_write":18,"num_write_kb":6750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":5248766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,10],"acting":[7,10],"avail_no_missing":["7","1","8","10"],"object_location_counts":[{"shards":"7,10","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"2.3b","version":"808'880","reported_seq":4583,"reported_epoch":813,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-09-11T03:50:31.059304+0000","last_change":"2024-09-11T03:50:31.059010+0000","last_active":"2024-09-11T03:50:30.044141+0000","last_peered":"2024-09-11T03:50:26.063772+0000","last_clean":"2024-09-11T03:50:26.063772+0000","last_became_active":"2024-09-11T03:46:32.975567+0000","last_became_peered":"2024-09-11T03:46:32.975567+0000","last_unstale":"2024-09-11T03:50:31.059304+0000","last_undegraded":"2024-09-11T03:50:31.049814+0000","last_fullsized":"2024-09-11T03:50:31.042172+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'773","last_scrub_stamp":"2024-09-11T03:49:34.919322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.919322+0000","objects_scrubbed":8,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10770005,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":104,"num_objects_misplaced":52,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":58,"num_read_kb":61859,"num_write":34,"num_write_kb":13221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,10],"acting":[7,4],"avail_no_missing":["7","9","11"],"object_location_counts":[{"shards":"4,7","objects":-1},{"shards":"7,9,11","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":7,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"2.3a","version":"557'273","reported_seq":3798,"reported_epoch":808,"state":"unknown","last_fresh":"2024-09-11T03:50:26.054742+0000","last_change":"2024-09-11T03:50:27.275919+0000","last_active":"2024-09-11T03:50:27.275919+0000","last_peered":"2024-09-11T03:50:26.054742+0000","last_clean":"2024-09-11T03:50:26.054742+0000","last_became_active":"2024-09-11T03:46:32.971003+0000","last_became_peered":"2024-09-11T03:46:32.971003+0000","last_unstale":"2024-09-11T03:50:26.054742+0000","last_undegraded":"2024-09-11T03:50:26.054742+0000","last_fullsized":"2024-09-11T03:50:26.054742+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:49:20.466140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.466140+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,2],"acting":[6,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"163","length":"1"},{"start":"16a","length":"1"}]},{"pgid":"2.39","version":"806'585","reported_seq":4244,"reported_epoch":813,"state":"activating+undersized+remapped","last_fresh":"2024-09-11T03:50:31.048378+0000","last_change":"2024-09-11T03:50:31.040874+0000","last_active":"2024-09-11T03:50:30.041961+0000","last_peered":"2024-09-11T03:50:24.045497+0000","last_clean":"2024-09-11T03:50:24.045497+0000","last_became_active":"2024-09-11T03:46:19.002002+0000","last_became_peered":"2024-09-11T03:46:19.002002+0000","last_unstale":"2024-09-11T03:50:31.048378+0000","last_undegraded":"2024-09-11T03:50:31.048378+0000","last_fullsized":"2024-09-11T03:50:31.040558+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":6,"last_scrub":"736'546","last_scrub_stamp":"2024-09-11T03:49:19.458345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:19.458345+0000","objects_scrubbed":6,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5852480,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":52,"num_read_kb":49891,"num_write":34,"num_write_kb":15002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2435423,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,2],"acting":[0,2],"avail_no_missing":["0","2","3","5","6","9"],"object_location_counts":[{"shards":"0,2","objects":4}],"blocked_by":[],"up_primary":9,"acting_primary":0,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"2.38","version":"808'564","reported_seq":4224,"reported_epoch":812,"state":"remapped+peering","last_fresh":"2024-09-11T03:50:30.042599+0000","last_change":"2024-09-11T03:50:30.042599+0000","last_active":"2024-09-11T03:50:30.042599+0000","last_peered":"2024-09-11T03:50:26.049706+0000","last_clean":"2024-09-11T03:50:26.049706+0000","last_became_active":"2024-09-11T03:42:09.873420+0000","last_became_peered":"2024-09-11T03:42:09.873420+0000","last_unstale":"2024-09-11T03:50:30.042599+0000","last_undegraded":"2024-09-11T03:50:30.042599+0000","last_fullsized":"2024-09-11T03:50:30.042599+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":6,"last_scrub":"783'523","last_scrub_stamp":"2024-09-11T03:50:01.960691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:01.960691+0000","objects_scrubbed":10,"log_size":564,"log_dups_size":0,"ondisk_log_size":564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13273786,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":108,"num_read_kb":124444,"num_write":53,"num_write_kb":22884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2432870,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":4,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"2.37","version":"274'312","reported_seq":2231,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.050176+0000","last_change":"2024-09-11T03:50:32.051288+0000","last_active":"2024-09-11T03:50:32.050176+0000","last_peered":"2024-09-11T03:50:32.050176+0000","last_clean":"2024-09-11T03:50:32.050176+0000","last_became_active":"2024-09-11T03:50:07.077607+0000","last_became_peered":"2024-09-11T03:50:07.077607+0000","last_unstale":"2024-09-11T03:50:32.050176+0000","last_undegraded":"2024-09-11T03:50:32.050176+0000","last_fullsized":"2024-09-11T03:50:32.050176+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:50:17.684567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.684567+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:22.566642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039299720000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,9],"acting":[10,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"2.36","version":"808'595","reported_seq":4274,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.039691+0000","last_change":"2024-09-11T03:50:30.045859+0000","last_active":"2024-09-11T03:50:31.039691+0000","last_peered":"2024-09-11T03:50:31.039691+0000","last_clean":"2024-09-11T03:50:31.039691+0000","last_became_active":"2024-09-11T03:50:07.078667+0000","last_became_peered":"2024-09-11T03:50:07.078667+0000","last_unstale":"2024-09-11T03:50:31.039691+0000","last_undegraded":"2024-09-11T03:50:31.039691+0000","last_fullsized":"2024-09-11T03:50:31.039691+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'589","last_scrub_stamp":"2024-09-11T03:50:18.280936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.280936+0000","objects_scrubbed":12,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:45.307331+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00092877499999999996,"stat_sum":{"num_bytes":26396466,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,8],"acting":[1,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"2.35","version":"274'273","reported_seq":1392,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.039050+0000","last_change":"2024-09-11T03:50:30.043340+0000","last_active":"2024-09-11T03:50:31.039050+0000","last_peered":"2024-09-11T03:50:31.039050+0000","last_clean":"2024-09-11T03:50:31.039050+0000","last_became_active":"2024-09-11T03:50:07.079331+0000","last_became_peered":"2024-09-11T03:50:07.079331+0000","last_unstale":"2024-09-11T03:50:31.039050+0000","last_undegraded":"2024-09-11T03:50:31.039050+0000","last_fullsized":"2024-09-11T03:50:31.039050+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:50:17.314165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.314165+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:32.814840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087528699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"2.34","version":"808'362","reported_seq":3979,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.518276+0000","last_change":"2024-09-11T03:50:30.047805+0000","last_active":"2024-09-11T03:50:31.518276+0000","last_peered":"2024-09-11T03:50:31.518276+0000","last_clean":"2024-09-11T03:50:31.518276+0000","last_became_active":"2024-09-11T03:50:07.080419+0000","last_became_peered":"2024-09-11T03:50:07.080419+0000","last_unstale":"2024-09-11T03:50:31.518276+0000","last_undegraded":"2024-09-11T03:50:31.518276+0000","last_fullsized":"2024-09-11T03:50:31.518276+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'358","last_scrub_stamp":"2024-09-11T03:50:17.613025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.613025+0000","objects_scrubbed":4,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:19.044857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011517700000000001,"stat_sum":{"num_bytes":9118212,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":12514,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"2.33","version":"274'111","reported_seq":1180,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.041604+0000","last_change":"2024-09-11T03:50:30.042359+0000","last_active":"2024-09-11T03:50:31.041604+0000","last_peered":"2024-09-11T03:50:31.041604+0000","last_clean":"2024-09-11T03:50:31.041604+0000","last_became_active":"2024-09-11T03:50:07.077684+0000","last_became_peered":"2024-09-11T03:50:07.077684+0000","last_unstale":"2024-09-11T03:50:31.041604+0000","last_undegraded":"2024-09-11T03:50:31.041604+0000","last_fullsized":"2024-09-11T03:50:31.041604+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:50:17.796171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.796171+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:28.852006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013012659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"2.32","version":"806'551","reported_seq":4162,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.043349+0000","last_change":"2024-09-11T03:50:32.043668+0000","last_active":"2024-09-11T03:50:32.043349+0000","last_peered":"2024-09-11T03:50:32.043349+0000","last_clean":"2024-09-11T03:50:32.043349+0000","last_became_active":"2024-09-11T03:50:07.077594+0000","last_became_peered":"2024-09-11T03:50:07.077594+0000","last_unstale":"2024-09-11T03:50:32.043349+0000","last_undegraded":"2024-09-11T03:50:32.043349+0000","last_fullsized":"2024-09-11T03:50:32.043349+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"778'549","last_scrub_stamp":"2024-09-11T03:50:17.199232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.199232+0000","objects_scrubbed":2,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:18.865807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001195565,"stat_sum":{"num_bytes":5630774,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9],"acting":[11,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"2.31","version":"274'107","reported_seq":1670,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.041191+0000","last_change":"2024-09-11T03:50:30.045096+0000","last_active":"2024-09-11T03:50:31.041191+0000","last_peered":"2024-09-11T03:50:31.041191+0000","last_clean":"2024-09-11T03:50:31.041191+0000","last_became_active":"2024-09-11T03:50:07.075956+0000","last_became_peered":"2024-09-11T03:50:07.075956+0000","last_unstale":"2024-09-11T03:50:31.041191+0000","last_undegraded":"2024-09-11T03:50:31.041191+0000","last_fullsized":"2024-09-11T03:50:31.041191+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:50:17.857515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.857515+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:23.245644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00121886,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,4],"acting":[0,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"2.30","version":"274'67","reported_seq":152,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.047627+0000","last_change":"2024-09-11T03:50:32.049944+0000","last_active":"2024-09-11T03:50:32.047627+0000","last_peered":"2024-09-11T03:50:32.047627+0000","last_clean":"2024-09-11T03:50:32.047627+0000","last_became_active":"2024-09-11T03:50:04.069393+0000","last_became_peered":"2024-09-11T03:50:04.069393+0000","last_unstale":"2024-09-11T03:50:32.047627+0000","last_undegraded":"2024-09-11T03:50:32.047627+0000","last_fullsized":"2024-09-11T03:50:32.047627+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:50:04.608333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.608333+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:29.461642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031143159999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,9],"acting":[10,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"2.2f","version":"557'465","reported_seq":2977,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-11T03:50:28.042240+0000","last_change":"2024-09-11T03:50:26.052640+0000","last_active":"2024-09-11T03:50:28.042240+0000","last_peered":"2024-09-11T03:50:28.042240+0000","last_clean":"2024-09-11T03:50:28.042240+0000","last_became_active":"2024-09-11T03:50:07.077840+0000","last_became_peered":"2024-09-11T03:50:07.077840+0000","last_unstale":"2024-09-11T03:50:28.042240+0000","last_undegraded":"2024-09-11T03:50:28.042240+0000","last_fullsized":"2024-09-11T03:50:28.042240+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:50:17.948546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.948546+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:18.960336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024904459999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"2.2e","version":"812'1065","reported_seq":4629,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.041319+0000","last_change":"2024-09-11T03:50:30.052281+0000","last_active":"2024-09-11T03:50:31.041319+0000","last_peered":"2024-09-11T03:50:31.041319+0000","last_clean":"2024-09-11T03:50:31.041319+0000","last_became_active":"2024-09-11T03:50:07.078910+0000","last_became_peered":"2024-09-11T03:50:07.078910+0000","last_unstale":"2024-09-11T03:50:31.041319+0000","last_undegraded":"2024-09-11T03:50:31.041319+0000","last_fullsized":"2024-09-11T03:50:31.041319+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'1047","last_scrub_stamp":"2024-09-11T03:50:18.779326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.779326+0000","objects_scrubbed":11,"log_size":1065,"log_dups_size":0,"ondisk_log_size":1065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:33.650605+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.010698911,"stat_sum":{"num_bytes":25317592,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":9,"num_read_kb":10737,"num_write":4,"num_write_kb":1315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"2.2d","version":"814'1110","reported_seq":4729,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.049491+0000","last_change":"2024-09-11T03:50:32.049736+0000","last_active":"2024-09-11T03:50:32.049491+0000","last_peered":"2024-09-11T03:50:32.049491+0000","last_clean":"2024-09-11T03:50:32.049491+0000","last_became_active":"2024-09-11T03:49:29.163201+0000","last_became_peered":"2024-09-11T03:49:29.163201+0000","last_unstale":"2024-09-11T03:50:32.049491+0000","last_undegraded":"2024-09-11T03:50:32.049491+0000","last_fullsized":"2024-09-11T03:50:32.049491+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'997","last_scrub_stamp":"2024-09-11T03:49:32.462332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.462332+0000","objects_scrubbed":13,"log_size":1110,"log_dups_size":0,"ondisk_log_size":1110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:40.026374+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0039376309999999996,"stat_sum":{"num_bytes":34462257,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":42,"num_read_kb":38986,"num_write":15,"num_write_kb":11845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"2.2c","version":"812'870","reported_seq":4479,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.039122+0000","last_change":"2024-09-11T03:50:30.052042+0000","last_active":"2024-09-11T03:50:31.039122+0000","last_peered":"2024-09-11T03:50:31.039122+0000","last_clean":"2024-09-11T03:50:31.039122+0000","last_became_active":"2024-09-11T03:49:29.154872+0000","last_became_peered":"2024-09-11T03:49:29.154872+0000","last_unstale":"2024-09-11T03:50:31.039122+0000","last_undegraded":"2024-09-11T03:50:31.039122+0000","last_fullsized":"2024-09-11T03:50:31.039122+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"753'800","last_scrub_stamp":"2024-09-11T03:49:32.185401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.185401+0000","objects_scrubbed":4,"log_size":870,"log_dups_size":0,"ondisk_log_size":870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:54.637839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012546767,"stat_sum":{"num_bytes":24961496,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":28,"num_read_kb":19600,"num_write":33,"num_write_kb":13666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3],"acting":[8,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"2.2b","version":"568'646","reported_seq":3372,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-11T03:50:28.041186+0000","last_change":"2024-09-11T03:50:26.051300+0000","last_active":"2024-09-11T03:50:28.041186+0000","last_peered":"2024-09-11T03:50:28.041186+0000","last_clean":"2024-09-11T03:50:28.041186+0000","last_became_active":"2024-09-11T03:49:29.157735+0000","last_became_peered":"2024-09-11T03:49:29.157735+0000","last_unstale":"2024-09-11T03:50:28.041186+0000","last_undegraded":"2024-09-11T03:50:28.041186+0000","last_fullsized":"2024-09-11T03:50:28.041186+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:49:32.876969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.876969+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:38.794846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017473689999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"2.2a","version":"727'348","reported_seq":3868,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.041551+0000","last_change":"2024-09-11T03:50:30.042152+0000","last_active":"2024-09-11T03:50:31.041551+0000","last_peered":"2024-09-11T03:50:31.041551+0000","last_clean":"2024-09-11T03:50:31.041551+0000","last_became_active":"2024-09-11T03:49:29.160595+0000","last_became_peered":"2024-09-11T03:49:29.160595+0000","last_unstale":"2024-09-11T03:50:31.041551+0000","last_undegraded":"2024-09-11T03:50:31.041551+0000","last_fullsized":"2024-09-11T03:50:31.041551+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:49:34.219244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.219244+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:46.747953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015149530000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,5],"acting":[7,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"2.29","version":"808'1247","reported_seq":4975,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.043610+0000","last_change":"2024-09-11T03:50:32.044326+0000","last_active":"2024-09-11T03:50:32.043610+0000","last_peered":"2024-09-11T03:50:32.043610+0000","last_clean":"2024-09-11T03:50:32.043610+0000","last_became_active":"2024-09-11T03:49:29.157194+0000","last_became_peered":"2024-09-11T03:49:29.157194+0000","last_unstale":"2024-09-11T03:50:32.043610+0000","last_undegraded":"2024-09-11T03:50:32.043610+0000","last_fullsized":"2024-09-11T03:50:32.043610+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'1167","last_scrub_stamp":"2024-09-11T03:49:33.506786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.506786+0000","objects_scrubbed":11,"log_size":1247,"log_dups_size":0,"ondisk_log_size":1247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:38.035514+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00095044900000000002,"stat_sum":{"num_bytes":31640833,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":36,"num_read_kb":54617,"num_write":21,"num_write_kb":8705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8],"acting":[6,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"2.28","version":"810'415","reported_seq":4038,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.038797+0000","last_change":"2024-09-11T03:50:30.041465+0000","last_active":"2024-09-11T03:50:31.038797+0000","last_peered":"2024-09-11T03:50:31.038797+0000","last_clean":"2024-09-11T03:50:31.038797+0000","last_became_active":"2024-09-11T03:49:29.154884+0000","last_became_peered":"2024-09-11T03:49:29.154884+0000","last_unstale":"2024-09-11T03:50:31.038797+0000","last_undegraded":"2024-09-11T03:50:31.038797+0000","last_fullsized":"2024-09-11T03:50:31.038797+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'382","last_scrub_stamp":"2024-09-11T03:49:32.335285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.335285+0000","objects_scrubbed":4,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:32.793485+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001206768,"stat_sum":{"num_bytes":11377646,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":10,"num_read_kb":11527,"num_write":8,"num_write_kb":1289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"2.27","version":"808'1119","reported_seq":4845,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.045256+0000","last_change":"2024-09-11T03:50:30.043450+0000","last_active":"2024-09-11T03:50:31.045256+0000","last_peered":"2024-09-11T03:50:31.045256+0000","last_clean":"2024-09-11T03:50:31.045256+0000","last_became_active":"2024-09-11T03:49:29.159799+0000","last_became_peered":"2024-09-11T03:49:29.159799+0000","last_unstale":"2024-09-11T03:50:31.045256+0000","last_undegraded":"2024-09-11T03:50:31.045256+0000","last_fullsized":"2024-09-11T03:50:31.045256+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"757'1065","last_scrub_stamp":"2024-09-11T03:49:36.196385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.196385+0000","objects_scrubbed":4,"log_size":1119,"log_dups_size":0,"ondisk_log_size":1119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:38.991535+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0017753599999999999,"stat_sum":{"num_bytes":14606339,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":14,"num_read_kb":13448,"num_write":8,"num_write_kb":7771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,9],"acting":[7,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"2.26","version":"269'171","reported_seq":3753,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.041172+0000","last_change":"2024-09-11T03:50:30.042572+0000","last_active":"2024-09-11T03:50:31.041172+0000","last_peered":"2024-09-11T03:50:31.041172+0000","last_clean":"2024-09-11T03:50:31.041172+0000","last_became_active":"2024-09-11T03:49:29.158577+0000","last_became_peered":"2024-09-11T03:49:29.158577+0000","last_unstale":"2024-09-11T03:50:31.041172+0000","last_undegraded":"2024-09-11T03:50:31.041172+0000","last_fullsized":"2024-09-11T03:50:31.041172+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:49:33.199779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.199779+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:53.912894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001121967,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,11],"acting":[7,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"2.25","version":"812'1286","reported_seq":5115,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.039423+0000","last_change":"2024-09-11T03:50:30.053413+0000","last_active":"2024-09-11T03:50:31.039423+0000","last_peered":"2024-09-11T03:50:31.039423+0000","last_clean":"2024-09-11T03:50:31.039423+0000","last_became_active":"2024-09-11T03:49:29.159733+0000","last_became_peered":"2024-09-11T03:49:29.159733+0000","last_unstale":"2024-09-11T03:50:31.039423+0000","last_undegraded":"2024-09-11T03:50:31.039423+0000","last_fullsized":"2024-09-11T03:50:31.039423+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"755'1138","last_scrub_stamp":"2024-09-11T03:49:34.198067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.198067+0000","objects_scrubbed":16,"log_size":1286,"log_dups_size":0,"ondisk_log_size":1286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:56.759912+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.014076451,"stat_sum":{"num_bytes":42747524,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":52,"num_read_kb":53768,"num_write":31,"num_write_kb":9045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,6],"acting":[8,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"2.24","version":"814'660","reported_seq":4306,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.051954+0000","last_change":"2024-09-11T03:50:32.052158+0000","last_active":"2024-09-11T03:50:32.051954+0000","last_peered":"2024-09-11T03:50:32.051954+0000","last_clean":"2024-09-11T03:50:32.051954+0000","last_became_active":"2024-09-11T03:49:29.155740+0000","last_became_peered":"2024-09-11T03:49:29.155740+0000","last_unstale":"2024-09-11T03:50:32.051954+0000","last_undegraded":"2024-09-11T03:50:32.051954+0000","last_fullsized":"2024-09-11T03:50:32.051954+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'612","last_scrub_stamp":"2024-09-11T03:49:34.554335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.554335+0000","objects_scrubbed":4,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:42.085247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0071852110000000004,"stat_sum":{"num_bytes":14236199,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":12,"num_write_kb":5382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,7],"acting":[6,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"2.23","version":"808'478","reported_seq":4081,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.039364+0000","last_change":"2024-09-11T03:50:30.041780+0000","last_active":"2024-09-11T03:50:31.039364+0000","last_peered":"2024-09-11T03:50:31.039364+0000","last_clean":"2024-09-11T03:50:31.039364+0000","last_became_active":"2024-09-11T03:46:32.970100+0000","last_became_peered":"2024-09-11T03:46:32.970100+0000","last_unstale":"2024-09-11T03:50:31.039364+0000","last_undegraded":"2024-09-11T03:50:31.039364+0000","last_fullsized":"2024-09-11T03:50:31.039364+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'474","last_scrub_stamp":"2024-09-11T03:50:19.311509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:19.311509+0000","objects_scrubbed":5,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:44.076689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010344359999999999,"stat_sum":{"num_bytes":9866336,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":44928,"num_write":25,"num_write_kb":10047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"2.22","version":"531'216","reported_seq":3810,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.044264+0000","last_change":"2024-09-11T03:50:32.046039+0000","last_active":"2024-09-11T03:50:32.044264+0000","last_peered":"2024-09-11T03:50:32.044264+0000","last_clean":"2024-09-11T03:50:32.044264+0000","last_became_active":"2024-09-11T03:46:32.970633+0000","last_became_peered":"2024-09-11T03:46:32.970633+0000","last_unstale":"2024-09-11T03:50:32.044264+0000","last_undegraded":"2024-09-11T03:50:32.044264+0000","last_fullsized":"2024-09-11T03:50:32.044264+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:49:18.406097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:18.406097+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002473509,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"2.21","version":"814'349","reported_seq":4034,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.047818+0000","last_change":"2024-09-11T03:50:32.048096+0000","last_active":"2024-09-11T03:50:32.047818+0000","last_peered":"2024-09-11T03:50:32.047818+0000","last_clean":"2024-09-11T03:50:32.047818+0000","last_became_active":"2024-09-11T03:46:36.024928+0000","last_became_peered":"2024-09-11T03:46:36.024928+0000","last_unstale":"2024-09-11T03:50:32.047818+0000","last_undegraded":"2024-09-11T03:50:32.047818+0000","last_fullsized":"2024-09-11T03:50:32.047818+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":6,"last_scrub":"733'289","last_scrub_stamp":"2024-09-11T03:49:18.917801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:18.917801+0000","objects_scrubbed":3,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:43.854959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0029493029999999999,"stat_sum":{"num_bytes":15025885,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":39,"num_read_kb":45171,"num_write":34,"num_write_kb":15311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,8],"acting":[11,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"2.20","version":"808'290","reported_seq":4039,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.041491+0000","last_change":"2024-09-11T03:50:30.043231+0000","last_active":"2024-09-11T03:50:31.041491+0000","last_peered":"2024-09-11T03:50:31.041491+0000","last_clean":"2024-09-11T03:50:31.041491+0000","last_became_active":"2024-09-11T03:46:32.974375+0000","last_became_peered":"2024-09-11T03:46:32.974375+0000","last_unstale":"2024-09-11T03:50:31.041491+0000","last_undegraded":"2024-09-11T03:50:31.041491+0000","last_fullsized":"2024-09-11T03:50:31.041491+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"718'223","last_scrub_stamp":"2024-09-11T03:49:00.376119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.376119+0000","objects_scrubbed":2,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0019118410000000001,"stat_sum":{"num_bytes":11791232,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":87,"num_read_kb":134813,"num_write":17,"num_write_kb":9596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"2.1c","version":"806'776","reported_seq":4376,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.051401+0000","last_change":"2024-09-11T03:50:32.051888+0000","last_active":"2024-09-11T03:50:32.051401+0000","last_peered":"2024-09-11T03:50:32.051401+0000","last_clean":"2024-09-11T03:50:32.051401+0000","last_became_active":"2024-09-11T03:50:28.047548+0000","last_became_peered":"2024-09-11T03:50:28.047548+0000","last_unstale":"2024-09-11T03:50:32.051401+0000","last_undegraded":"2024-09-11T03:50:32.051401+0000","last_fullsized":"2024-09-11T03:50:32.051401+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"754'760","last_scrub_stamp":"2024-09-11T03:49:33.579794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.579794+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0034507700000000001,"stat_sum":{"num_bytes":4367535,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":17539,"num_write":18,"num_write_kb":6749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"1.c","version":"20'92","reported_seq":678,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.040919+0000","last_change":"2024-09-11T03:47:26.195120+0000","last_active":"2024-09-11T03:50:31.040919+0000","last_peered":"2024-09-11T03:50:31.040919+0000","last_clean":"2024-09-11T03:50:31.040919+0000","last_became_active":"2024-09-11T03:46:15.974786+0000","last_became_peered":"2024-09-11T03:46:15.974786+0000","last_unstale":"2024-09-11T03:50:31.040919+0000","last_undegraded":"2024-09-11T03:50:31.040919+0000","last_fullsized":"2024-09-11T03:50:31.040919+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:26.195070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:26.195070+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:53.091756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.f","version":"557'465","reported_seq":4091,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.039325+0000","last_change":"2024-09-11T03:50:30.040050+0000","last_active":"2024-09-11T03:50:31.039325+0000","last_peered":"2024-09-11T03:50:31.039325+0000","last_clean":"2024-09-11T03:50:31.039325+0000","last_became_active":"2024-09-11T03:50:02.055146+0000","last_became_peered":"2024-09-11T03:50:02.055146+0000","last_unstale":"2024-09-11T03:50:31.039325+0000","last_undegraded":"2024-09-11T03:50:31.039325+0000","last_fullsized":"2024-09-11T03:50:31.039325+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:50:02.368307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.368307+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:18.444513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080314899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"2.e","version":"782'1025","reported_seq":4585,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-11T03:50:28.040372+0000","last_change":"2024-09-11T03:50:26.051913+0000","last_active":"2024-09-11T03:50:28.040372+0000","last_peered":"2024-09-11T03:50:28.040372+0000","last_clean":"2024-09-11T03:50:28.040372+0000","last_became_active":"2024-09-11T03:50:02.051717+0000","last_became_peered":"2024-09-11T03:50:02.051717+0000","last_unstale":"2024-09-11T03:50:28.040372+0000","last_undegraded":"2024-09-11T03:50:28.040372+0000","last_fullsized":"2024-09-11T03:50:28.040372+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":0,"last_scrub":"782'1025","last_scrub_stamp":"2024-09-11T03:50:04.968160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.968160+0000","objects_scrubbed":0,"log_size":1025,"log_dups_size":0,"ondisk_log_size":1025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:34.246264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022693280000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":1255,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.041921+0000","last_change":"2024-09-11T03:49:20.176154+0000","last_active":"2024-09-11T03:50:31.041921+0000","last_peered":"2024-09-11T03:50:31.041921+0000","last_clean":"2024-09-11T03:50:31.041921+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:50:31.041921+0000","last_undegraded":"2024-09-11T03:50:31.041921+0000","last_fullsized":"2024-09-11T03:50:31.041921+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:20.176098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.176098+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:41.602355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"811'1014","reported_seq":4667,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.519232+0000","last_change":"2024-09-11T03:50:30.048205+0000","last_active":"2024-09-11T03:50:31.519232+0000","last_peered":"2024-09-11T03:50:31.519232+0000","last_clean":"2024-09-11T03:50:31.519232+0000","last_became_active":"2024-09-11T03:49:24.116744+0000","last_became_peered":"2024-09-11T03:49:24.116744+0000","last_unstale":"2024-09-11T03:50:31.519232+0000","last_undegraded":"2024-09-11T03:50:31.519232+0000","last_fullsized":"2024-09-11T03:50:31.519232+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'987","last_scrub_stamp":"2024-09-11T03:49:32.653263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.653263+0000","objects_scrubbed":3,"log_size":1014,"log_dups_size":0,"ondisk_log_size":1014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:48.897042+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00127289,"stat_sum":{"num_bytes":8757074,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":970,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.041857+0000","last_change":"2024-09-11T03:49:22.192021+0000","last_active":"2024-09-11T03:50:31.041857+0000","last_peered":"2024-09-11T03:50:31.041857+0000","last_clean":"2024-09-11T03:50:31.041857+0000","last_became_active":"2024-09-11T03:44:43.940022+0000","last_became_peered":"2024-09-11T03:44:43.940022+0000","last_unstale":"2024-09-11T03:50:31.041857+0000","last_undegraded":"2024-09-11T03:50:31.041857+0000","last_fullsized":"2024-09-11T03:50:31.041857+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:22.191965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:22.191965+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:33.940974+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"814'858","reported_seq":4482,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.049207+0000","last_change":"2024-09-11T03:50:32.049779+0000","last_active":"2024-09-11T03:50:32.049207+0000","last_peered":"2024-09-11T03:50:32.049207+0000","last_clean":"2024-09-11T03:50:32.049207+0000","last_became_active":"2024-09-11T03:49:24.115405+0000","last_became_peered":"2024-09-11T03:49:24.115405+0000","last_unstale":"2024-09-11T03:50:32.049207+0000","last_undegraded":"2024-09-11T03:50:32.049207+0000","last_fullsized":"2024-09-11T03:50:32.049207+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'808","last_scrub_stamp":"2024-09-11T03:49:33.953477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.953477+0000","objects_scrubbed":7,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:45.557713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0066881989999999997,"stat_sum":{"num_bytes":23449747,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":14,"num_read_kb":13992,"num_write":14,"num_write_kb":6063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"1.f","version":"20'92","reported_seq":538,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.047341+0000","last_change":"2024-09-11T03:48:57.363099+0000","last_active":"2024-09-11T03:50:32.047341+0000","last_peered":"2024-09-11T03:50:32.047341+0000","last_clean":"2024-09-11T03:50:32.047341+0000","last_became_active":"2024-09-11T03:46:15.973796+0000","last_became_peered":"2024-09-11T03:46:15.973796+0000","last_unstale":"2024-09-11T03:50:32.047341+0000","last_undegraded":"2024-09-11T03:50:32.047341+0000","last_fullsized":"2024-09-11T03:50:32.047341+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:57.363027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:57.363027+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:10.644654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":1661,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.043455+0000","last_change":"2024-09-11T03:47:28.495899+0000","last_active":"2024-09-11T03:50:31.043455+0000","last_peered":"2024-09-11T03:50:31.043455+0000","last_clean":"2024-09-11T03:50:31.043455+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:50:31.043455+0000","last_undegraded":"2024-09-11T03:50:31.043455+0000","last_fullsized":"2024-09-11T03:50:31.043455+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:28.495839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:28.495839+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.015311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"812'531","reported_seq":4137,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.527604+0000","last_change":"2024-09-11T03:50:30.050256+0000","last_active":"2024-09-11T03:50:31.527604+0000","last_peered":"2024-09-11T03:50:31.527604+0000","last_clean":"2024-09-11T03:50:31.527604+0000","last_became_active":"2024-09-11T03:46:27.972329+0000","last_became_peered":"2024-09-11T03:46:27.972329+0000","last_unstale":"2024-09-11T03:50:31.527604+0000","last_undegraded":"2024-09-11T03:50:31.527604+0000","last_fullsized":"2024-09-11T03:50:31.527604+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'480","last_scrub_stamp":"2024-09-11T03:49:37.694768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.694768+0000","objects_scrubbed":5,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:48.007291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0038808940000000002,"stat_sum":{"num_bytes":13193420,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":27,"num_read_kb":23186,"num_write":38,"num_write_kb":7258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":1265,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.039517+0000","last_change":"2024-09-11T03:47:25.344219+0000","last_active":"2024-09-11T03:50:31.039517+0000","last_peered":"2024-09-11T03:50:31.039517+0000","last_clean":"2024-09-11T03:50:31.039517+0000","last_became_active":"2024-09-11T03:46:15.968960+0000","last_became_peered":"2024-09-11T03:46:15.968960+0000","last_unstale":"2024-09-11T03:50:31.039517+0000","last_undegraded":"2024-09-11T03:50:31.039517+0000","last_fullsized":"2024-09-11T03:50:31.039517+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:25.344170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:25.344170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:42.179046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"563'192","reported_seq":2667,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.050105+0000","last_change":"2024-09-11T03:50:32.050387+0000","last_active":"2024-09-11T03:50:32.050105+0000","last_peered":"2024-09-11T03:50:32.050105+0000","last_clean":"2024-09-11T03:50:32.050105+0000","last_became_active":"2024-09-11T03:50:31.044399+0000","last_became_peered":"2024-09-11T03:50:31.044399+0000","last_unstale":"2024-09-11T03:50:32.050105+0000","last_undegraded":"2024-09-11T03:50:32.050105+0000","last_fullsized":"2024-09-11T03:50:32.050105+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:50:25.721409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:25.721409+0000","objects_scrubbed":4,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0031983379999999998,"stat_sum":{"num_bytes":861391,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":19922,"num_write":7,"num_write_kb":2259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1096758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"1.2","version":"20'92","reported_seq":1224,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.040363+0000","last_change":"2024-09-11T03:49:17.252987+0000","last_active":"2024-09-11T03:50:31.040363+0000","last_peered":"2024-09-11T03:50:31.040363+0000","last_clean":"2024-09-11T03:50:31.040363+0000","last_became_active":"2024-09-11T03:44:25.934580+0000","last_became_peered":"2024-09-11T03:44:25.934580+0000","last_unstale":"2024-09-11T03:50:31.040363+0000","last_undegraded":"2024-09-11T03:50:31.040363+0000","last_fullsized":"2024-09-11T03:50:31.040363+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:17.252916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:17.252916+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:39.008724+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"567'235","reported_seq":3815,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.041737+0000","last_change":"2024-09-11T03:50:30.045639+0000","last_active":"2024-09-11T03:50:31.041737+0000","last_peered":"2024-09-11T03:50:31.041737+0000","last_clean":"2024-09-11T03:50:31.041737+0000","last_became_active":"2024-09-11T03:50:28.039300+0000","last_became_peered":"2024-09-11T03:50:28.039300+0000","last_unstale":"2024-09-11T03:50:31.041737+0000","last_undegraded":"2024-09-11T03:50:31.041737+0000","last_fullsized":"2024-09-11T03:50:31.041737+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:50:23.836407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:23.836407+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012549,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":840,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.048239+0000","last_change":"2024-09-11T03:50:18.699606+0000","last_active":"2024-09-11T03:50:32.048239+0000","last_peered":"2024-09-11T03:50:32.048239+0000","last_clean":"2024-09-11T03:50:32.048239+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:50:32.048239+0000","last_undegraded":"2024-09-11T03:50:32.048239+0000","last_fullsized":"2024-09-11T03:50:32.048239+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:18.699532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.699532+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:27.930456+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"799'299","reported_seq":3984,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.518917+0000","last_change":"2024-09-11T03:50:30.047066+0000","last_active":"2024-09-11T03:50:31.518917+0000","last_peered":"2024-09-11T03:50:31.518917+0000","last_clean":"2024-09-11T03:50:31.518917+0000","last_became_active":"2024-09-11T03:46:27.971649+0000","last_became_peered":"2024-09-11T03:46:27.971649+0000","last_unstale":"2024-09-11T03:50:31.518917+0000","last_undegraded":"2024-09-11T03:50:31.518917+0000","last_fullsized":"2024-09-11T03:50:31.518917+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"799'299","last_scrub_stamp":"2024-09-11T03:50:20.621227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:20.621227+0000","objects_scrubbed":3,"log_size":299,"log_dups_size":0,"ondisk_log_size":299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:40.373065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000731662,"stat_sum":{"num_bytes":4723229,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":24,"num_read_kb":13035,"num_write":25,"num_write_kb":12701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":530,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.047121+0000","last_change":"2024-09-11T03:49:37.637065+0000","last_active":"2024-09-11T03:50:32.047121+0000","last_peered":"2024-09-11T03:50:32.047121+0000","last_clean":"2024-09-11T03:50:32.047121+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:50:32.047121+0000","last_undegraded":"2024-09-11T03:50:32.047121+0000","last_fullsized":"2024-09-11T03:50:32.047121+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:37.637018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.637018+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:05.221389+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"812'667","reported_seq":4329,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.039570+0000","last_change":"2024-09-11T03:50:30.044444+0000","last_active":"2024-09-11T03:50:31.039570+0000","last_peered":"2024-09-11T03:50:31.039570+0000","last_clean":"2024-09-11T03:50:31.039570+0000","last_became_active":"2024-09-11T03:49:24.114330+0000","last_became_peered":"2024-09-11T03:49:24.114330+0000","last_unstale":"2024-09-11T03:50:31.039570+0000","last_undegraded":"2024-09-11T03:50:31.039570+0000","last_fullsized":"2024-09-11T03:50:31.039570+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'620","last_scrub_stamp":"2024-09-11T03:49:34.288083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.288083+0000","objects_scrubbed":8,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:51.777332+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00092167799999999999,"stat_sum":{"num_bytes":22786313,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":12,"num_read_kb":8619,"num_write":9,"num_write_kb":5852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":843,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.047050+0000","last_change":"2024-09-11T03:49:17.858224+0000","last_active":"2024-09-11T03:50:32.047050+0000","last_peered":"2024-09-11T03:50:32.047050+0000","last_clean":"2024-09-11T03:50:32.047050+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:50:32.047050+0000","last_undegraded":"2024-09-11T03:50:32.047050+0000","last_fullsized":"2024-09-11T03:50:32.047050+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:17.858165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:17.858165+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:40.674663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"812'1178","reported_seq":4845,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.039558+0000","last_change":"2024-09-11T03:50:30.052392+0000","last_active":"2024-09-11T03:50:31.039558+0000","last_peered":"2024-09-11T03:50:31.039558+0000","last_clean":"2024-09-11T03:50:31.039558+0000","last_became_active":"2024-09-11T03:49:24.115095+0000","last_became_peered":"2024-09-11T03:49:24.115095+0000","last_unstale":"2024-09-11T03:50:31.039558+0000","last_undegraded":"2024-09-11T03:50:31.039558+0000","last_fullsized":"2024-09-11T03:50:31.039558+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'1112","last_scrub_stamp":"2024-09-11T03:49:35.188307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.188307+0000","objects_scrubbed":7,"log_size":1178,"log_dups_size":0,"ondisk_log_size":1178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:44.167600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.012830001000000001,"stat_sum":{"num_bytes":16331222,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":5,"num_read_kb":1,"num_write":9,"num_write_kb":5553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":842,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.048980+0000","last_change":"2024-09-11T03:49:36.654074+0000","last_active":"2024-09-11T03:50:32.048980+0000","last_peered":"2024-09-11T03:50:32.048980+0000","last_clean":"2024-09-11T03:50:32.048980+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:50:32.048980+0000","last_undegraded":"2024-09-11T03:50:32.048980+0000","last_fullsized":"2024-09-11T03:50:32.048980+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:36.654025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.654025+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:06.216209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":3758,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.039815+0000","last_change":"2024-09-11T03:50:30.046121+0000","last_active":"2024-09-11T03:50:31.039815+0000","last_peered":"2024-09-11T03:50:31.039815+0000","last_clean":"2024-09-11T03:50:31.039815+0000","last_became_active":"2024-09-11T03:49:24.113518+0000","last_became_peered":"2024-09-11T03:49:24.113518+0000","last_unstale":"2024-09-11T03:50:31.039815+0000","last_undegraded":"2024-09-11T03:50:31.039815+0000","last_fullsized":"2024-09-11T03:50:31.039815+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:49:35.267227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.267227+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:47.317337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083350899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":1442,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.042810+0000","last_change":"2024-09-11T03:49:21.909643+0000","last_active":"2024-09-11T03:50:32.042810+0000","last_peered":"2024-09-11T03:50:32.042810+0000","last_clean":"2024-09-11T03:50:32.042810+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:50:32.042810+0000","last_undegraded":"2024-09-11T03:50:32.042810+0000","last_fullsized":"2024-09-11T03:50:32.042810+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:21.909594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:21.909594+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:26.077940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"814'1193","reported_seq":4961,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.052845+0000","last_change":"2024-09-11T03:50:32.053019+0000","last_active":"2024-09-11T03:50:32.052845+0000","last_peered":"2024-09-11T03:50:32.052845+0000","last_clean":"2024-09-11T03:50:32.052845+0000","last_became_active":"2024-09-11T03:49:24.121333+0000","last_became_peered":"2024-09-11T03:49:24.121333+0000","last_unstale":"2024-09-11T03:50:32.052845+0000","last_undegraded":"2024-09-11T03:50:32.052845+0000","last_fullsized":"2024-09-11T03:50:32.052845+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'1066","last_scrub_stamp":"2024-09-11T03:49:35.600201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.600201+0000","objects_scrubbed":7,"log_size":1193,"log_dups_size":0,"ondisk_log_size":1193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:46.829921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0059712799999999998,"stat_sum":{"num_bytes":30712986,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":68,"num_read_kb":62761,"num_write":40,"num_write_kb":23794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"1.b","version":"20'92","reported_seq":1244,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.045963+0000","last_change":"2024-09-11T03:50:18.176206+0000","last_active":"2024-09-11T03:50:32.045963+0000","last_peered":"2024-09-11T03:50:32.045963+0000","last_clean":"2024-09-11T03:50:32.045963+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:50:32.045963+0000","last_undegraded":"2024-09-11T03:50:32.045963+0000","last_fullsized":"2024-09-11T03:50:32.045963+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:18.176156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.176156+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:43.809898+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"744'374","reported_seq":4018,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.042850+0000","last_change":"2024-09-11T03:50:30.043731+0000","last_active":"2024-09-11T03:50:31.042850+0000","last_peered":"2024-09-11T03:50:31.042850+0000","last_clean":"2024-09-11T03:50:31.042850+0000","last_became_active":"2024-09-11T03:49:24.115150+0000","last_became_peered":"2024-09-11T03:49:24.115150+0000","last_unstale":"2024-09-11T03:50:31.042850+0000","last_undegraded":"2024-09-11T03:50:31.042850+0000","last_fullsized":"2024-09-11T03:50:31.042850+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'374","last_scrub_stamp":"2024-09-11T03:49:35.190806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.190806+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:59.529317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015094640000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":1587,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-11T03:50:28.039902+0000","last_change":"2024-09-11T03:50:18.934008+0000","last_active":"2024-09-11T03:50:28.039902+0000","last_peered":"2024-09-11T03:50:28.039902+0000","last_clean":"2024-09-11T03:50:28.039902+0000","last_became_active":"2024-09-11T03:46:15.973271+0000","last_became_peered":"2024-09-11T03:46:15.973271+0000","last_unstale":"2024-09-11T03:50:28.039902+0000","last_undegraded":"2024-09-11T03:50:28.039902+0000","last_fullsized":"2024-09-11T03:50:28.039902+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:18.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.933952+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:39.183063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"808'1294","reported_seq":5012,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.038959+0000","last_change":"2024-09-11T03:50:30.044754+0000","last_active":"2024-09-11T03:50:31.038959+0000","last_peered":"2024-09-11T03:50:31.038959+0000","last_clean":"2024-09-11T03:50:31.038959+0000","last_became_active":"2024-09-11T03:49:24.114865+0000","last_became_peered":"2024-09-11T03:49:24.114865+0000","last_unstale":"2024-09-11T03:50:31.038959+0000","last_undegraded":"2024-09-11T03:50:31.038959+0000","last_fullsized":"2024-09-11T03:50:31.038959+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'1171","last_scrub_stamp":"2024-09-11T03:49:33.292043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.292043+0000","objects_scrubbed":10,"log_size":1294,"log_dups_size":0,"ondisk_log_size":1294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:48.954447+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019592580000000002,"stat_sum":{"num_bytes":31976045,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":33,"num_read_kb":35558,"num_write":22,"num_write_kb":10529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":1656,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.041031+0000","last_change":"2024-09-11T03:49:37.206392+0000","last_active":"2024-09-11T03:50:31.041031+0000","last_peered":"2024-09-11T03:50:31.041031+0000","last_clean":"2024-09-11T03:50:31.041031+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:50:31.041031+0000","last_undegraded":"2024-09-11T03:50:31.041031+0000","last_fullsized":"2024-09-11T03:50:31.041031+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:37.206321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.206321+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:53.641968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"808'372","reported_seq":4020,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.043400+0000","last_change":"2024-09-11T03:50:32.043892+0000","last_active":"2024-09-11T03:50:32.043400+0000","last_peered":"2024-09-11T03:50:32.043400+0000","last_clean":"2024-09-11T03:50:32.043400+0000","last_became_active":"2024-09-11T03:49:24.114551+0000","last_became_peered":"2024-09-11T03:49:24.114551+0000","last_unstale":"2024-09-11T03:50:32.043400+0000","last_undegraded":"2024-09-11T03:50:32.043400+0000","last_fullsized":"2024-09-11T03:50:32.043400+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:49:32.911631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.911631+0000","objects_scrubbed":4,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:02.339739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001250637,"stat_sum":{"num_bytes":5429057,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":10,"num_read_kb":12240,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"1.8","version":"20'92","reported_seq":1315,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.519102+0000","last_change":"2024-09-11T03:49:34.621588+0000","last_active":"2024-09-11T03:50:31.519102+0000","last_peered":"2024-09-11T03:50:31.519102+0000","last_clean":"2024-09-11T03:50:31.519102+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:50:31.519102+0000","last_undegraded":"2024-09-11T03:50:31.519102+0000","last_fullsized":"2024-09-11T03:50:31.519102+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:34.621527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.621527+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.563733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"568'646","reported_seq":4285,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.043269+0000","last_change":"2024-09-11T03:50:31.043269+0000","last_active":"2024-09-11T03:50:31.043269+0000","last_peered":"2024-09-11T03:50:31.043269+0000","last_clean":"2024-09-11T03:50:31.043269+0000","last_became_active":"2024-09-11T03:50:31.042191+0000","last_became_peered":"2024-09-11T03:50:31.042191+0000","last_unstale":"2024-09-11T03:50:31.043269+0000","last_undegraded":"2024-09-11T03:50:31.043269+0000","last_fullsized":"2024-09-11T03:50:31.043269+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:49:33.204497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.204497+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:41.704568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00060428599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,0],"acting":[8,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":1265,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.518998+0000","last_change":"2024-09-11T03:49:00.710015+0000","last_active":"2024-09-11T03:50:31.518998+0000","last_peered":"2024-09-11T03:50:31.518998+0000","last_clean":"2024-09-11T03:50:31.518998+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:50:31.518998+0000","last_undegraded":"2024-09-11T03:50:31.518998+0000","last_fullsized":"2024-09-11T03:50:31.518998+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:00.709966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.709966+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":3671,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-11T03:50:28.041869+0000","last_change":"2024-09-11T03:50:26.053348+0000","last_active":"2024-09-11T03:50:28.041869+0000","last_peered":"2024-09-11T03:50:28.041869+0000","last_clean":"2024-09-11T03:50:28.041869+0000","last_became_active":"2024-09-11T03:50:02.046906+0000","last_became_peered":"2024-09-11T03:50:02.046906+0000","last_unstale":"2024-09-11T03:50:28.041869+0000","last_undegraded":"2024-09-11T03:50:28.041869+0000","last_fullsized":"2024-09-11T03:50:28.041869+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:50:02.987179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.987179+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:17.412133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029465250000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":1250,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.041556+0000","last_change":"2024-09-11T03:49:41.224105+0000","last_active":"2024-09-11T03:50:31.041556+0000","last_peered":"2024-09-11T03:50:31.041556+0000","last_clean":"2024-09-11T03:50:31.041556+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:50:31.041556+0000","last_undegraded":"2024-09-11T03:50:31.041556+0000","last_fullsized":"2024-09-11T03:50:31.041556+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:41.224059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:41.224059+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:58.932206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"812'559","reported_seq":4190,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.050490+0000","last_change":"2024-09-11T03:50:32.050940+0000","last_active":"2024-09-11T03:50:32.050490+0000","last_peered":"2024-09-11T03:50:32.050490+0000","last_clean":"2024-09-11T03:50:32.050490+0000","last_became_active":"2024-09-11T03:50:02.053513+0000","last_became_peered":"2024-09-11T03:50:02.053513+0000","last_unstale":"2024-09-11T03:50:32.050490+0000","last_undegraded":"2024-09-11T03:50:32.050490+0000","last_fullsized":"2024-09-11T03:50:32.050490+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"784'551","last_scrub_stamp":"2024-09-11T03:50:02.647690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.647690+0000","objects_scrubbed":4,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:11.422888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029262400000000001,"stat_sum":{"num_bytes":6367447,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"1.12","version":"20'92","reported_seq":1246,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.518709+0000","last_change":"2024-09-11T03:47:23.339646+0000","last_active":"2024-09-11T03:50:31.518709+0000","last_peered":"2024-09-11T03:50:31.518709+0000","last_clean":"2024-09-11T03:50:31.518709+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:50:31.518709+0000","last_undegraded":"2024-09-11T03:50:31.518709+0000","last_fullsized":"2024-09-11T03:50:31.518709+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:23.339595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:23.339595+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:50.634356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":2765,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.046129+0000","last_change":"2024-09-11T03:50:32.046472+0000","last_active":"2024-09-11T03:50:32.046129+0000","last_peered":"2024-09-11T03:50:32.046129+0000","last_clean":"2024-09-11T03:50:32.046129+0000","last_became_active":"2024-09-11T03:50:02.048777+0000","last_became_peered":"2024-09-11T03:50:02.048777+0000","last_unstale":"2024-09-11T03:50:32.046129+0000","last_undegraded":"2024-09-11T03:50:32.046129+0000","last_fullsized":"2024-09-11T03:50:32.046129+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:50:02.696833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.696833+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:18.706256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024096009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"1.13","version":"20'92","reported_seq":1351,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.039491+0000","last_change":"2024-09-11T03:47:18.217600+0000","last_active":"2024-09-11T03:50:31.039491+0000","last_peered":"2024-09-11T03:50:31.039491+0000","last_clean":"2024-09-11T03:50:31.039491+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:50:31.039491+0000","last_undegraded":"2024-09-11T03:50:31.039491+0000","last_fullsized":"2024-09-11T03:50:31.039491+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:18.217555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:18.217555+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:43.139528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":2702,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.038760+0000","last_change":"2024-09-11T03:50:30.038839+0000","last_active":"2024-09-11T03:50:31.038760+0000","last_peered":"2024-09-11T03:50:31.038760+0000","last_clean":"2024-09-11T03:50:31.038760+0000","last_became_active":"2024-09-11T03:50:02.057196+0000","last_became_peered":"2024-09-11T03:50:02.057196+0000","last_unstale":"2024-09-11T03:50:31.038760+0000","last_undegraded":"2024-09-11T03:50:31.038760+0000","last_fullsized":"2024-09-11T03:50:31.038760+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:50:03.345563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:03.345563+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:12.914364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052121700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":832,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.044875+0000","last_change":"2024-09-11T03:50:20.772575+0000","last_active":"2024-09-11T03:50:31.044875+0000","last_peered":"2024-09-11T03:50:31.044875+0000","last_clean":"2024-09-11T03:50:31.044875+0000","last_became_active":"2024-09-11T03:46:36.026459+0000","last_became_peered":"2024-09-11T03:46:36.026459+0000","last_unstale":"2024-09-11T03:50:31.044875+0000","last_undegraded":"2024-09-11T03:50:31.044875+0000","last_fullsized":"2024-09-11T03:50:31.044875+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:20.772524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:20.772524+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:26.554678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":3261,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.052224+0000","last_change":"2024-09-11T03:50:32.052501+0000","last_active":"2024-09-11T03:50:32.052224+0000","last_peered":"2024-09-11T03:50:32.052224+0000","last_clean":"2024-09-11T03:50:32.052224+0000","last_became_active":"2024-09-11T03:50:02.052278+0000","last_became_peered":"2024-09-11T03:50:02.052278+0000","last_unstale":"2024-09-11T03:50:32.052224+0000","last_undegraded":"2024-09-11T03:50:32.052224+0000","last_fullsized":"2024-09-11T03:50:32.052224+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:50:03.690374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:03.690374+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:13.213586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0051599100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"1.15","version":"20'92","reported_seq":1227,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.041793+0000","last_change":"2024-09-11T03:50:19.741484+0000","last_active":"2024-09-11T03:50:31.041793+0000","last_peered":"2024-09-11T03:50:31.041793+0000","last_clean":"2024-09-11T03:50:31.041793+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:50:31.041793+0000","last_undegraded":"2024-09-11T03:50:31.041793+0000","last_fullsized":"2024-09-11T03:50:31.041793+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:19.741428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:19.741428+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:36.778923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"782'561","reported_seq":4227,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.519145+0000","last_change":"2024-09-11T03:50:30.047392+0000","last_active":"2024-09-11T03:50:31.519145+0000","last_peered":"2024-09-11T03:50:31.519145+0000","last_clean":"2024-09-11T03:50:31.519145+0000","last_became_active":"2024-09-11T03:50:02.054667+0000","last_became_peered":"2024-09-11T03:50:02.054667+0000","last_unstale":"2024-09-11T03:50:31.519145+0000","last_undegraded":"2024-09-11T03:50:31.519145+0000","last_fullsized":"2024-09-11T03:50:31.519145+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"782'561","last_scrub_stamp":"2024-09-11T03:50:02.696931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.696931+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:07.724988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010092230000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":700,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.045160+0000","last_change":"2024-09-11T03:49:20.949194+0000","last_active":"2024-09-11T03:50:32.045160+0000","last_peered":"2024-09-11T03:50:32.045160+0000","last_clean":"2024-09-11T03:50:32.045160+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:50:32.045160+0000","last_undegraded":"2024-09-11T03:50:32.045160+0000","last_fullsized":"2024-09-11T03:50:32.045160+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:20.949118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.949118+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:36.681631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":3891,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-09-11T03:50:28.039792+0000","last_change":"2024-09-11T03:50:26.050281+0000","last_active":"2024-09-11T03:50:28.039792+0000","last_peered":"2024-09-11T03:50:28.039792+0000","last_clean":"2024-09-11T03:50:28.039792+0000","last_became_active":"2024-09-11T03:50:02.051850+0000","last_became_peered":"2024-09-11T03:50:02.051850+0000","last_unstale":"2024-09-11T03:50:28.039792+0000","last_undegraded":"2024-09-11T03:50:28.039792+0000","last_fullsized":"2024-09-11T03:50:28.039792+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:50:04.004006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.004006+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:11.007385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075771900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":1239,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.039160+0000","last_change":"2024-09-11T03:47:29.086383+0000","last_active":"2024-09-11T03:50:31.039160+0000","last_peered":"2024-09-11T03:50:31.039160+0000","last_clean":"2024-09-11T03:50:31.039160+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:50:31.039160+0000","last_undegraded":"2024-09-11T03:50:31.039160+0000","last_fullsized":"2024-09-11T03:50:31.039160+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:29.086335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:29.086335+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:43.171032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"778'353","reported_seq":3895,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.046454+0000","last_change":"2024-09-11T03:50:32.047323+0000","last_active":"2024-09-11T03:50:32.046454+0000","last_peered":"2024-09-11T03:50:32.046454+0000","last_clean":"2024-09-11T03:50:32.046454+0000","last_became_active":"2024-09-11T03:50:02.054211+0000","last_became_peered":"2024-09-11T03:50:02.054211+0000","last_unstale":"2024-09-11T03:50:32.046454+0000","last_undegraded":"2024-09-11T03:50:32.046454+0000","last_fullsized":"2024-09-11T03:50:32.046454+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"778'353","last_scrub_stamp":"2024-09-11T03:50:04.646331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.646331+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:34.210751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020462029999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"1.18","version":"20'92","reported_seq":1358,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.518865+0000","last_change":"2024-09-11T03:49:56.630710+0000","last_active":"2024-09-11T03:50:31.518865+0000","last_peered":"2024-09-11T03:50:31.518865+0000","last_clean":"2024-09-11T03:50:31.518865+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:50:31.518865+0000","last_undegraded":"2024-09-11T03:50:31.518865+0000","last_fullsized":"2024-09-11T03:50:31.518865+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:56.630651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:56.630651+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:18.735201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"813'881","reported_seq":4663,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.048225+0000","last_change":"2024-09-11T03:50:32.048519+0000","last_active":"2024-09-11T03:50:32.048225+0000","last_peered":"2024-09-11T03:50:32.048225+0000","last_clean":"2024-09-11T03:50:32.048225+0000","last_became_active":"2024-09-11T03:50:31.047148+0000","last_became_peered":"2024-09-11T03:50:31.047148+0000","last_unstale":"2024-09-11T03:50:32.048225+0000","last_undegraded":"2024-09-11T03:50:32.048225+0000","last_fullsized":"2024-09-11T03:50:32.048225+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":5,"last_scrub":"754'773","last_scrub_stamp":"2024-09-11T03:49:34.919322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.919322+0000","objects_scrubbed":0,"log_size":881,"log_dups_size":0,"ondisk_log_size":881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0032117500000000002,"stat_sum":{"num_bytes":10770005,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":57,"num_read_kb":61859,"num_write":34,"num_write_kb":13221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":4927563,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"1.19","version":"20'92","reported_seq":846,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.048209+0000","last_change":"2024-09-11T03:48:56.129575+0000","last_active":"2024-09-11T03:50:32.048209+0000","last_peered":"2024-09-11T03:50:32.048209+0000","last_clean":"2024-09-11T03:50:32.048209+0000","last_became_active":"2024-09-11T03:46:15.972841+0000","last_became_peered":"2024-09-11T03:46:15.972841+0000","last_unstale":"2024-09-11T03:50:32.048209+0000","last_undegraded":"2024-09-11T03:50:32.048209+0000","last_fullsized":"2024-09-11T03:50:32.048209+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:56.129516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:56.129516+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:26.739103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"557'273","reported_seq":3837,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.043849+0000","last_change":"2024-09-11T03:50:32.044117+0000","last_active":"2024-09-11T03:50:32.043849+0000","last_peered":"2024-09-11T03:50:32.043849+0000","last_clean":"2024-09-11T03:50:32.043849+0000","last_became_active":"2024-09-11T03:50:28.038587+0000","last_became_peered":"2024-09-11T03:50:28.038587+0000","last_unstale":"2024-09-11T03:50:32.043849+0000","last_undegraded":"2024-09-11T03:50:32.043849+0000","last_fullsized":"2024-09-11T03:50:32.043849+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:49:20.466140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.466140+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00069077099999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,2],"acting":[6,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"1.1a","version":"20'92","reported_seq":389,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.041649+0000","last_change":"2024-09-11T03:50:01.843384+0000","last_active":"2024-09-11T03:50:31.041649+0000","last_peered":"2024-09-11T03:50:31.041649+0000","last_clean":"2024-09-11T03:50:31.041649+0000","last_became_active":"2024-09-11T03:49:30.162694+0000","last_became_peered":"2024-09-11T03:49:30.162694+0000","last_unstale":"2024-09-11T03:50:31.041649+0000","last_undegraded":"2024-09-11T03:50:31.041649+0000","last_fullsized":"2024-09-11T03:50:31.041649+0000","mapping_epoch":751,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":752,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:01.843302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:01.843302+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:12.888176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.19","version":"806'585","reported_seq":4286,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.044730+0000","last_change":"2024-09-11T03:50:32.046048+0000","last_active":"2024-09-11T03:50:32.044730+0000","last_peered":"2024-09-11T03:50:32.044730+0000","last_clean":"2024-09-11T03:50:32.044730+0000","last_became_active":"2024-09-11T03:50:28.039694+0000","last_became_peered":"2024-09-11T03:50:28.039694+0000","last_unstale":"2024-09-11T03:50:32.044730+0000","last_undegraded":"2024-09-11T03:50:32.044730+0000","last_fullsized":"2024-09-11T03:50:32.044730+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"736'546","last_scrub_stamp":"2024-09-11T03:49:19.458345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:19.458345+0000","objects_scrubbed":0,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0023382749999999999,"stat_sum":{"num_bytes":5852479,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":49891,"num_write":34,"num_write_kb":15002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2435422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"1.1b","version":"20'92","reported_seq":1215,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.038879+0000","last_change":"2024-09-11T03:48:31.033896+0000","last_active":"2024-09-11T03:50:31.038879+0000","last_peered":"2024-09-11T03:50:31.038879+0000","last_clean":"2024-09-11T03:50:31.038879+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:50:31.038879+0000","last_undegraded":"2024-09-11T03:50:31.038879+0000","last_fullsized":"2024-09-11T03:50:31.038879+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:31.033837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:31.033837+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:46.181076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"808'564","reported_seq":4244,"reported_epoch":810,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T03:50:28.135065+0000","last_change":"2024-09-11T03:50:28.134926+0000","last_active":"2024-09-11T03:50:28.135065+0000","last_peered":"2024-09-11T03:50:28.135065+0000","last_clean":"2024-09-11T03:50:28.135065+0000","last_became_active":"2024-09-11T03:50:28.044833+0000","last_became_peered":"2024-09-11T03:50:28.044833+0000","last_unstale":"2024-09-11T03:50:28.135065+0000","last_undegraded":"2024-09-11T03:50:28.135065+0000","last_fullsized":"2024-09-11T03:50:28.135065+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"783'523","last_scrub_stamp":"2024-09-11T03:50:01.960691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:01.960691+0000","objects_scrubbed":0,"log_size":564,"log_dups_size":0,"ondisk_log_size":564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.011635688999999999,"stat_sum":{"num_bytes":13273786,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":117,"num_read_kb":136958,"num_write":52,"num_write_kb":22883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2432870,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"2.1f","version":"812'761","reported_seq":4326,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.041022+0000","last_change":"2024-09-11T03:50:30.052928+0000","last_active":"2024-09-11T03:50:31.041022+0000","last_peered":"2024-09-11T03:50:31.041022+0000","last_clean":"2024-09-11T03:50:31.041022+0000","last_became_active":"2024-09-11T03:50:28.041110+0000","last_became_peered":"2024-09-11T03:50:28.041110+0000","last_unstale":"2024-09-11T03:50:31.041022+0000","last_undegraded":"2024-09-11T03:50:31.041022+0000","last_fullsized":"2024-09-11T03:50:31.041022+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"806'752","last_scrub_stamp":"2024-09-11T03:50:24.822925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:24.822925+0000","objects_scrubbed":0,"log_size":761,"log_dups_size":0,"ondisk_log_size":761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.010799738,"stat_sum":{"num_bytes":20585612,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":69,"num_read_kb":75588,"num_write":48,"num_write_kb":16761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"1.1c","version":"20'92","reported_seq":1222,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.039191+0000","last_change":"2024-09-11T03:49:40.156559+0000","last_active":"2024-09-11T03:50:31.039191+0000","last_peered":"2024-09-11T03:50:31.039191+0000","last_clean":"2024-09-11T03:50:31.039191+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:50:31.039191+0000","last_undegraded":"2024-09-11T03:50:31.039191+0000","last_fullsized":"2024-09-11T03:50:31.039191+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:40.156495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:40.156495+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:04.047069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1e","version":"806'1060","reported_seq":4641,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.041639+0000","last_change":"2024-09-11T03:50:31.041120+0000","last_active":"2024-09-11T03:50:31.041639+0000","last_peered":"2024-09-11T03:50:31.041639+0000","last_clean":"2024-09-11T03:50:31.041639+0000","last_became_active":"2024-09-11T03:50:28.044451+0000","last_became_peered":"2024-09-11T03:50:28.044451+0000","last_unstale":"2024-09-11T03:50:31.041639+0000","last_undegraded":"2024-09-11T03:50:31.041639+0000","last_fullsized":"2024-09-11T03:50:31.041639+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"785'1044","last_scrub_stamp":"2024-09-11T03:50:03.954442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:03.954442+0000","objects_scrubbed":0,"log_size":1060,"log_dups_size":0,"ondisk_log_size":1060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000915736,"stat_sum":{"num_bytes":10069114,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":25973,"num_write":46,"num_write_kb":17498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16d","length":"1"}]},{"pgid":"1.1d","version":"20'92","reported_seq":1204,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-09-11T03:50:31.039211+0000","last_change":"2024-09-11T03:47:22.312507+0000","last_active":"2024-09-11T03:50:31.039211+0000","last_peered":"2024-09-11T03:50:31.039211+0000","last_clean":"2024-09-11T03:50:31.039211+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:50:31.039211+0000","last_undegraded":"2024-09-11T03:50:31.039211+0000","last_fullsized":"2024-09-11T03:50:31.039211+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:22.312456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:22.312456+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.239645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"567'752","reported_seq":4336,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.051553+0000","last_change":"2024-09-11T03:50:32.051898+0000","last_active":"2024-09-11T03:50:32.051553+0000","last_peered":"2024-09-11T03:50:32.051553+0000","last_clean":"2024-09-11T03:50:32.051553+0000","last_became_active":"2024-09-11T03:50:28.045687+0000","last_became_peered":"2024-09-11T03:50:28.045687+0000","last_unstale":"2024-09-11T03:50:32.051553+0000","last_undegraded":"2024-09-11T03:50:32.051553+0000","last_fullsized":"2024-09-11T03:50:32.051553+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:50:25.778626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:25.778626+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0028596189999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,11],"acting":[10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12a","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"},{"start":"16c","length":"2"}]},{"pgid":"1.1e","version":"20'92","reported_seq":1232,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-09-11T03:50:32.045303+0000","last_change":"2024-09-11T03:49:36.949199+0000","last_active":"2024-09-11T03:50:32.045303+0000","last_peered":"2024-09-11T03:50:32.045303+0000","last_clean":"2024-09-11T03:50:32.045303+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:50:32.045303+0000","last_undegraded":"2024-09-11T03:50:32.045303+0000","last_fullsized":"2024-09-11T03:50:32.045303+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:36.949149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.949149+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:45.388809+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":66,"stat_sum":{"num_bytes":573823241,"num_objects":275,"num_object_clones":225,"num_object_copies":825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":142,"num_objects_misplaced":52,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":26,"num_read":1337,"num_read_kb":1423013,"num_write":900,"num_write_kb":376007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":37098628,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1001345024,"data_stored":1387161666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37697,"ondisk_log_size":37697,"up":198,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285728,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":123180,"kb_used_data":91988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93621972,"statfs":{"total":95995035648,"available":95868899328,"internally_reserved":0,"allocated":94195712,"data_stored":130370513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514660,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96484,"kb_used_data":69452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93648668,"statfs":{"total":95995035648,"available":95896236032,"internally_reserved":0,"allocated":71118848,"data_stored":80435040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743593,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":119684,"kb_used_data":62128,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93625468,"statfs":{"total":95995035648,"available":95872479232,"internally_reserved":0,"allocated":63619072,"data_stored":69125574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037937,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":193020,"kb_used_data":135456,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93552132,"statfs":{"total":95995035648,"available":95797383168,"internally_reserved":0,"allocated":138706944,"data_stored":201038157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299572,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":207960,"kb_used_data":162876,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93537192,"statfs":{"total":95995035648,"available":95782084608,"internally_reserved":0,"allocated":166785024,"data_stored":221457299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528505,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":139676,"kb_used_data":112648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93605476,"statfs":{"total":95995035648,"available":95852007424,"internally_reserved":0,"allocated":115351552,"data_stored":153768781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757439,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91444,"kb_used_data":60252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93653708,"statfs":{"total":95995035648,"available":95901396992,"internally_reserved":0,"allocated":61698048,"data_stored":78158296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986372,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":103000,"kb_used_data":49596,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93642152,"statfs":{"total":95995035648,"available":95889563648,"internally_reserved":0,"allocated":50786304,"data_stored":54875908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149896,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":167556,"kb_used_data":118312,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93577596,"statfs":{"total":95995035648,"available":95823458304,"internally_reserved":0,"allocated":121151488,"data_stored":167711606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411534,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73552,"kb_used_data":42356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93671600,"statfs":{"total":95995035648,"available":95919718400,"internally_reserved":0,"allocated":43372544,"data_stored":48802611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575060,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":134452,"kb_used_data":81056,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93610700,"statfs":{"total":95995035648,"available":95857356800,"internally_reserved":0,"allocated":83001344,"data_stored":113587574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738585,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148536,"kb_used_data":95144,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93596616,"statfs":{"total":95995035648,"available":95842934784,"internally_reserved":0,"allocated":97427456,"data_stored":130978065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":87261184,"data_stored":124233589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73150464,"data_stored":107162098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":33366016,"data_stored":42210301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":111009792,"data_stored":160984664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960000,"data_stored":48468209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61419520,"data_stored":82113790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":114118656,"data_stored":158452510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":151773184,"data_stored":208234231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":128868352,"data_stored":194621361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":53805056,"data_stored":62726312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":61267968,"data_stored":74009564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":84344832,"data_stored":123945037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:50:34.734 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set unique_pool_0 pgp_num 66 2024-09-11T03:50:34.775 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:34.795 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:50:34.801 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:34.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:34.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:34.867 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:50:34.903 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:34.924 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:34.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:34.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:34.983 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:35.025 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:35.037 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:50:35.040 INFO:tasks.rados.rados.2.smithi175.stdout:3359: done (1 left) 2024-09-11T03:50:35.040 INFO:tasks.rados.rados.2.smithi175.stdout:3361: done (0 left) 2024-09-11T03:50:35.040 INFO:tasks.rados.rados.2.smithi175.stdout:3362: read oid 14 snap -1 2024-09-11T03:50:35.040 INFO:tasks.rados.rados.2.smithi175.stdout:3362: expect (ObjNum 1075 snap 350 seq_num 1075) 2024-09-11T03:50:35.040 INFO:tasks.rados.rados.2.smithi175.stdout:3363: write oid 11 current snap is 370 2024-09-11T03:50:35.040 INFO:tasks.rados.rados.2.smithi175.stdout:3363: seq_num 1144 ranges {427208=737806,1777518=513724,2966139=261069} 2024-09-11T03:50:35.047 INFO:tasks.rados.rados.2.smithi175.stdout:3363: writing smithi17557870-11 from 427208 to 1165014 tid 1 2024-09-11T03:50:35.047 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:35.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:35.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:35.053 INFO:tasks.rados.rados.2.smithi175.stdout:3363: writing smithi17557870-11 from 1777518 to 2291242 tid 2 2024-09-11T03:50:35.058 INFO:tasks.rados.rados.2.smithi175.stdout:3363: writing smithi17557870-11 from 2966139 to 3227208 tid 3 2024-09-11T03:50:35.058 INFO:tasks.rados.rados.2.smithi175.stdout:3364: rollback oid 9 current snap is 370 2024-09-11T03:50:35.058 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 9 to 345 2024-09-11T03:50:35.058 INFO:tasks.rados.rados.2.smithi175.stdout:3362: expect (ObjNum 1075 snap 350 seq_num 1075) 2024-09-11T03:50:35.149 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:35.156 INFO:tasks.rados.rados.2.smithi175.stdout:3363: finishing write tid 1 to smithi17557870-11 2024-09-11T03:50:35.156 INFO:tasks.rados.rados.2.smithi175.stdout:3362: done (2 left) 2024-09-11T03:50:35.156 INFO:tasks.rados.rados.2.smithi175.stdout:3365: delete oid 46 current snap is 370 2024-09-11T03:50:35.156 INFO:tasks.rados.rados.2.smithi175.stdout:3363: finishing write tid 2 to smithi17557870-11 2024-09-11T03:50:35.156 INFO:tasks.rados.rados.2.smithi175.stdout:3363: finishing write tid 3 to smithi17557870-11 2024-09-11T03:50:35.156 INFO:tasks.rados.rados.2.smithi175.stdout:3364: finishing rollback tid 1 to smithi17557870-9 2024-09-11T03:50:35.156 INFO:tasks.rados.rados.2.smithi175.stdout:3363: finishing write tid 4 to smithi17557870-11 2024-09-11T03:50:35.156 INFO:tasks.rados.rados.2.smithi175.stdout:3363: finishing write tid 5 to smithi17557870-11 2024-09-11T03:50:35.156 INFO:tasks.rados.rados.2.smithi175.stdout:3363: oid 11 updating version 0 to 635 2024-09-11T03:50:35.156 INFO:tasks.rados.rados.2.smithi175.stdout:3363: oid 11 version 635 is already newer than 633 2024-09-11T03:50:35.157 INFO:tasks.rados.rados.2.smithi175.stdout:3363: oid 11 version 635 is already newer than 634 2024-09-11T03:50:35.157 INFO:tasks.rados.rados.2.smithi175.stdout:3363: oid 11 version 635 is already newer than 632 2024-09-11T03:50:35.157 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 635 (ObjNum 1144 snap 370 seq_num 1144) dirty exists 2024-09-11T03:50:35.157 INFO:tasks.rados.rados.2.smithi175.stdout:3363: left oid 11 (ObjNum 1144 snap 370 seq_num 1144) 2024-09-11T03:50:35.157 INFO:tasks.rados.rados.2.smithi175.stdout:3364: finishing rollback tid 2 to smithi17557870-9 2024-09-11T03:50:35.157 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 1234 (ObjNum 1033 snap 340 seq_num 1033) dirty exists 2024-09-11T03:50:35.161 INFO:tasks.rados.rados.2.smithi175.stdout:3363: done (2 left) 2024-09-11T03:50:35.161 INFO:tasks.rados.rados.2.smithi175.stdout:3364: done (1 left) 2024-09-11T03:50:35.161 INFO:tasks.rados.rados.2.smithi175.stdout:3365: done (0 left) 2024-09-11T03:50:35.162 INFO:tasks.rados.rados.2.smithi175.stdout:3366: copy_from oid 46 from oid 3 current snap is 370 2024-09-11T03:50:35.162 INFO:tasks.rados.rados.2.smithi175.stdout:3367: read oid 14 snap -1 2024-09-11T03:50:35.162 INFO:tasks.rados.rados.2.smithi175.stdout:3367: expect (ObjNum 1075 snap 350 seq_num 1075) 2024-09-11T03:50:35.162 INFO:tasks.rados.rados.2.smithi175.stdout:3368: copy_from oid 27 from oid 9 current snap is 370 2024-09-11T03:50:35.162 INFO:tasks.rados.rados.2.smithi175.stdout:3369: rollback oid 32 current snap is 370 2024-09-11T03:50:35.162 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 32 to 349 2024-09-11T03:50:35.162 INFO:tasks.rados.rados.2.smithi175.stdout:3370: copy_from oid 19 from oid 23 current snap is 370 2024-09-11T03:50:35.162 INFO:tasks.rados.rados.2.smithi175.stdout:3371: read oid 21 snap 358 2024-09-11T03:50:35.162 INFO:tasks.rados.rados.2.smithi175.stdout:3371: expect (ObjNum 1079 snap 352 seq_num 1079) 2024-09-11T03:50:35.162 INFO:tasks.rados.rados.2.smithi175.stdout:3372: write (excl) oid 37 current snap is 370 2024-09-11T03:50:35.162 INFO:tasks.rados.rados.2.smithi175.stdout:3372: seq_num 1146 ranges {460822=488304,1378129=517393,2504632=405962} 2024-09-11T03:50:35.165 INFO:tasks.rados.rados.2.smithi175.stdout:3372: writing smithi17557870-37 from 460822 to 949126 tid 1 2024-09-11T03:50:35.170 INFO:tasks.rados.rados.2.smithi175.stdout:3372: writing smithi17557870-37 from 1378129 to 1895522 tid 2 2024-09-11T03:50:35.172 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:35.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:35.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:35.177 INFO:tasks.rados.rados.2.smithi175.stdout:3372: writing smithi17557870-37 from 2504632 to 2910594 tid 3 2024-09-11T03:50:35.177 INFO:tasks.rados.rados.2.smithi175.stdout:3373: write oid 8 current snap is 370 2024-09-11T03:50:35.177 INFO:tasks.rados.rados.2.smithi175.stdout:3366: finishing copy_from to smithi17557870-46 2024-09-11T03:50:35.177 INFO:tasks.rados.rados.2.smithi175.stdout:3366: got expected ENOENT (src dne) 2024-09-11T03:50:35.177 INFO:tasks.rados.rados.2.smithi175.stdout:3366: finishing copy_from racing read to smithi17557870-46 2024-09-11T03:50:35.177 INFO:tasks.rados.rados.2.smithi175.stdout:3367: expect (ObjNum 1075 snap 350 seq_num 1075) 2024-09-11T03:50:35.274 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:35.274 INFO:tasks.rados.rados.2.smithi175.stdout:3369: finishing rollback tid 0 to smithi17557870-32 2024-09-11T03:50:35.275 INFO:tasks.rados.rados.2.smithi175.stdout:3369: finishing rollback tid 1 to smithi17557870-32 2024-09-11T03:50:35.275 INFO:tasks.rados.rados.2.smithi175.stdout:3373: seq_num 1147 ranges {477629=563553,1836611=777404,3048408=229221} 2024-09-11T03:50:35.279 INFO:tasks.rados.rados.2.smithi175.stdout:3373: writing smithi17557870-8 from 477629 to 1041182 tid 1 2024-09-11T03:50:35.288 INFO:tasks.rados.rados.2.smithi175.stdout:3373: writing smithi17557870-8 from 1836611 to 2614015 tid 2 2024-09-11T03:50:35.293 INFO:tasks.rados.rados.2.smithi175.stdout:3373: writing smithi17557870-8 from 3048408 to 3277629 tid 3 2024-09-11T03:50:35.293 INFO:tasks.rados.rados.2.smithi175.stdout:3366: done (7 left) 2024-09-11T03:50:35.293 INFO:tasks.rados.rados.2.smithi175.stdout:3367: done (6 left) 2024-09-11T03:50:35.293 INFO:tasks.rados.rados.2.smithi175.stdout:3374: rollback oid 22 current snap is 370 2024-09-11T03:50:35.293 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 22 to 347 2024-09-11T03:50:35.294 INFO:tasks.rados.rados.2.smithi175.stdout:3369: finishing rollback tid 2 to smithi17557870-32 2024-09-11T03:50:35.294 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 1008 (ObjNum 1059 snap 345 seq_num 1059) dirty exists 2024-09-11T03:50:35.294 INFO:tasks.rados.rados.2.smithi175.stdout:3368: finishing copy_from racing read to smithi17557870-27 2024-09-11T03:50:35.294 INFO:tasks.rados.rados.2.smithi175.stdout:3368: finishing copy_from to smithi17557870-27 2024-09-11T03:50:35.294 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 1234 (ObjNum 1033 snap 340 seq_num 1033) dirty exists 2024-09-11T03:50:35.294 INFO:tasks.rados.rados.2.smithi175.stdout:3372: finishing write tid 1 to smithi17557870-37 2024-09-11T03:50:35.294 INFO:tasks.rados.rados.2.smithi175.stdout:3372: finishing write tid 2 to smithi17557870-37 2024-09-11T03:50:35.294 INFO:tasks.rados.rados.2.smithi175.stdout:3372: finishing write tid 3 to smithi17557870-37 2024-09-11T03:50:35.294 INFO:tasks.rados.rados.2.smithi175.stdout:3372: finishing write tid 4 to smithi17557870-37 2024-09-11T03:50:35.294 INFO:tasks.rados.rados.2.smithi175.stdout:3372: finishing write tid 5 to smithi17557870-37 2024-09-11T03:50:35.294 INFO:tasks.rados.rados.2.smithi175.stdout:3372: oid 37 updating version 0 to 911 2024-09-11T03:50:35.294 INFO:tasks.rados.rados.2.smithi175.stdout:3372: oid 37 updating version 911 to 913 2024-09-11T03:50:35.294 INFO:tasks.rados.rados.2.smithi175.stdout:3372: oid 37 version 913 is already newer than 910 2024-09-11T03:50:35.294 INFO:tasks.rados.rados.2.smithi175.stdout:3372: oid 37 version 913 is already newer than 912 2024-09-11T03:50:35.295 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 913 (ObjNum 1146 snap 370 seq_num 1146) dirty exists 2024-09-11T03:50:35.295 INFO:tasks.rados.rados.2.smithi175.stdout:3372: left oid 37 (ObjNum 1146 snap 370 seq_num 1146) 2024-09-11T03:50:35.295 INFO:tasks.rados.rados.2.smithi175.stdout:3370: finishing copy_from racing read to smithi17557870-19 2024-09-11T03:50:35.295 INFO:tasks.rados.rados.2.smithi175.stdout:3368: done (6 left) 2024-09-11T03:50:35.295 INFO:tasks.rados.rados.2.smithi175.stdout:3369: done (5 left) 2024-09-11T03:50:35.295 INFO:tasks.rados.rados.2.smithi175.stdout:3372: done (4 left) 2024-09-11T03:50:35.295 INFO:tasks.rados.rados.2.smithi175.stdout:3375: rollback oid 16 current snap is 370 2024-09-11T03:50:35.295 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 16 to 307 2024-09-11T03:50:35.295 INFO:tasks.rados.rados.2.smithi175.stdout:3371: expect (ObjNum 1079 snap 352 seq_num 1079) 2024-09-11T03:50:35.297 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:35.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:35.299 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:35.391 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:50:35.392 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:50:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:35 smithi084 ceph-mon[27254]: 2.3d scrub starts 2024-09-11T03:50:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:35 smithi084 ceph-mon[27254]: 2.3f scrub starts 2024-09-11T03:50:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:35 smithi084 ceph-mon[27254]: 2.1f scrub starts 2024-09-11T03:50:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:35 smithi084 ceph-mon[27254]: 2.1b scrub starts 2024-09-11T03:50:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:35 smithi084 ceph-mon[27254]: 2.39 scrub starts 2024-09-11T03:50:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:35 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3183394095' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:50:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:35 smithi084 ceph-mon[27254]: 2.38 scrub starts 2024-09-11T03:50:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:35 smithi084 ceph-mon[27254]: 2.40 scrub starts 2024-09-11T03:50:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:35 smithi084 ceph-mon[27254]: 2.3b scrub starts 2024-09-11T03:50:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:35 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:50:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:35 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3183394095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:50:35.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:35 smithi084 ceph-mon[27254]: osdmap e817: 12 total, 12 up, 12 in 2024-09-11T03:50:35.400 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:35.422 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:35.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:35.423 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:35.427 INFO:tasks.rados.rados.2.smithi175.stdout:3370: finishing copy_from to smithi17557870-19 2024-09-11T03:50:35.427 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 975 (ObjNum 1079 snap 352 seq_num 1079) dirty exists 2024-09-11T03:50:35.427 INFO:tasks.rados.rados.2.smithi175.stdout:3373: finishing write tid 1 to smithi17557870-8 2024-09-11T03:50:35.427 INFO:tasks.rados.rados.2.smithi175.stdout:3370: done (4 left) 2024-09-11T03:50:35.427 INFO:tasks.rados.rados.2.smithi175.stdout:3371: done (3 left) 2024-09-11T03:50:35.427 INFO:tasks.rados.rados.2.smithi175.stdout:3376: snap_create 2024-09-11T03:50:35.427 INFO:tasks.rados.rados.2.smithi175.stdout:3374: finishing rollback tid 1 to smithi17557870-22 2024-09-11T03:50:35.427 INFO:tasks.rados.rados.2.smithi175.stdout:3374: finishing rollback tid 2 to smithi17557870-22 2024-09-11T03:50:35.427 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 1253 (ObjNum 829 snap 263 seq_num 829) dirty exists 2024-09-11T03:50:35.427 INFO:tasks.rados.rados.2.smithi175.stdout:3375: finishing rollback tid 0 to smithi17557870-16 2024-09-11T03:50:35.427 INFO:tasks.rados.rados.2.smithi175.stdout:3373: finishing write tid 2 to smithi17557870-8 2024-09-11T03:50:35.427 INFO:tasks.rados.rados.2.smithi175.stdout:3375: finishing rollback tid 1 to smithi17557870-16 2024-09-11T03:50:35.427 INFO:tasks.rados.rados.2.smithi175.stdout:3373: finishing write tid 3 to smithi17557870-8 2024-09-11T03:50:35.428 INFO:tasks.rados.rados.2.smithi175.stdout:3375: finishing rollback tid 2 to smithi17557870-16 2024-09-11T03:50:35.428 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 999 (ObjNum 940 snap 297 seq_num 940) dirty exists 2024-09-11T03:50:35.428 INFO:tasks.rados.rados.2.smithi175.stdout:3373: finishing write tid 4 to smithi17557870-8 2024-09-11T03:50:35.428 INFO:tasks.rados.rados.2.smithi175.stdout:3373: finishing write tid 5 to smithi17557870-8 2024-09-11T03:50:35.428 INFO:tasks.rados.rados.2.smithi175.stdout:3373: oid 8 updating version 0 to 1071 2024-09-11T03:50:35.428 INFO:tasks.rados.rados.2.smithi175.stdout:3373: oid 8 updating version 1071 to 1073 2024-09-11T03:50:35.428 INFO:tasks.rados.rados.2.smithi175.stdout:3373: oid 8 version 1073 is already newer than 1072 2024-09-11T03:50:35.429 INFO:tasks.rados.rados.2.smithi175.stdout:3373: oid 8 version 1073 is already newer than 1070 2024-09-11T03:50:35.429 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 1073 (ObjNum 1147 snap 370 seq_num 1147) dirty exists 2024-09-11T03:50:35.429 INFO:tasks.rados.rados.2.smithi175.stdout:3373: left oid 8 (ObjNum 1147 snap 370 seq_num 1147) 2024-09-11T03:50:35.524 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:35.546 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:35.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:35.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:35.648 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:35.669 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:35.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:35.670 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:35 smithi121 ceph-mon[30729]: 2.3d scrub starts 2024-09-11T03:50:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:35 smithi121 ceph-mon[30729]: 2.3f scrub starts 2024-09-11T03:50:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:35 smithi121 ceph-mon[30729]: 2.1f scrub starts 2024-09-11T03:50:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:35 smithi121 ceph-mon[30729]: 2.1b scrub starts 2024-09-11T03:50:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:35 smithi121 ceph-mon[30729]: 2.39 scrub starts 2024-09-11T03:50:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:35 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3183394095' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:50:35.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:35 smithi121 ceph-mon[30729]: 2.38 scrub starts 2024-09-11T03:50:35.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:35 smithi121 ceph-mon[30729]: 2.40 scrub starts 2024-09-11T03:50:35.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:35 smithi121 ceph-mon[30729]: 2.3b scrub starts 2024-09-11T03:50:35.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:35 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:50:35.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:35 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3183394095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:50:35.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:35 smithi121 ceph-mon[30729]: osdmap e817: 12 total, 12 up, 12 in 2024-09-11T03:50:35.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:35 smithi175 ceph-mon[30894]: 2.3d scrub starts 2024-09-11T03:50:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:35 smithi175 ceph-mon[30894]: 2.3f scrub starts 2024-09-11T03:50:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:35 smithi175 ceph-mon[30894]: 2.1f scrub starts 2024-09-11T03:50:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:35 smithi175 ceph-mon[30894]: 2.1b scrub starts 2024-09-11T03:50:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:35 smithi175 ceph-mon[30894]: 2.39 scrub starts 2024-09-11T03:50:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:35 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3183394095' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:50:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:35 smithi175 ceph-mon[30894]: 2.38 scrub starts 2024-09-11T03:50:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:35 smithi175 ceph-mon[30894]: 2.40 scrub starts 2024-09-11T03:50:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:35 smithi175 ceph-mon[30894]: 2.3b scrub starts 2024-09-11T03:50:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:35 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:50:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:35 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3183394095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:50:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:35 smithi175 ceph-mon[30894]: osdmap e817: 12 total, 12 up, 12 in 2024-09-11T03:50:35.771 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:35.793 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:35.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:35.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:35.895 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:35.916 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:35.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:35.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:36.018 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:36.040 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:36.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:36.041 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:36.143 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:36.150 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:36.165 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:36.166 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:36.206 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-11T03:50:36.249 INFO:tasks.rados.rados.2.smithi175.stdout:3373: done (3 left) 2024-09-11T03:50:36.249 INFO:tasks.rados.rados.2.smithi175.stdout:3374: done (2 left) 2024-09-11T03:50:36.249 INFO:tasks.rados.rados.2.smithi175.stdout:3375: done (1 left) 2024-09-11T03:50:36.249 INFO:tasks.rados.rados.2.smithi175.stdout:3376: done (0 left) 2024-09-11T03:50:36.249 INFO:tasks.rados.rados.2.smithi175.stdout:3377: write (excl) oid 25 current snap is 371 2024-09-11T03:50:36.250 INFO:tasks.rados.rados.2.smithi175.stdout:3377: seq_num 1148 ranges {494436=755155,2011446=521062,3075831=218605} 2024-09-11T03:50:36.255 INFO:tasks.rados.rados.2.smithi175.stdout:3377: writing smithi17557870-25 from 494436 to 1249591 tid 1 2024-09-11T03:50:36.262 INFO:tasks.rados.rados.2.smithi175.stdout:3377: writing smithi17557870-25 from 2011446 to 2532508 tid 2 2024-09-11T03:50:36.266 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:36.268 INFO:tasks.rados.rados.2.smithi175.stdout:3377: writing smithi17557870-25 from 3075831 to 3294436 tid 3 2024-09-11T03:50:36.268 INFO:tasks.rados.rados.2.smithi175.stdout:3378: copy_from oid 10 from oid 15 current snap is 371 2024-09-11T03:50:36.268 INFO:tasks.rados.rados.2.smithi175.stdout:3377: finishing write tid 1 to smithi17557870-25 2024-09-11T03:50:36.268 INFO:tasks.rados.rados.2.smithi175.stdout:3377: finishing write tid 2 to smithi17557870-25 2024-09-11T03:50:36.268 INFO:tasks.rados.rados.2.smithi175.stdout:3379: write (excl) oid 48 current snap is 371 2024-09-11T03:50:36.268 INFO:tasks.rados.rados.2.smithi175.stdout:3379: seq_num 1149 ranges {511243=430404,1669928=664720,3103254=207989} 2024-09-11T03:50:36.272 INFO:tasks.rados.rados.2.smithi175.stdout:3379: writing smithi17557870-48 from 511243 to 941647 tid 1 2024-09-11T03:50:36.280 INFO:tasks.rados.rados.2.smithi175.stdout:3379: writing smithi17557870-48 from 1669928 to 2334648 tid 2 2024-09-11T03:50:36.286 INFO:tasks.rados.rados.2.smithi175.stdout:3379: writing smithi17557870-48 from 3103254 to 3311243 tid 3 2024-09-11T03:50:36.287 INFO:tasks.rados.rados.2.smithi175.stdout:3380: snap_create 2024-09-11T03:50:36.287 INFO:tasks.rados.rados.2.smithi175.stdout:3377: finishing write tid 3 to smithi17557870-25 2024-09-11T03:50:36.287 INFO:tasks.rados.rados.2.smithi175.stdout:3377: finishing write tid 4 to smithi17557870-25 2024-09-11T03:50:36.287 INFO:tasks.rados.rados.2.smithi175.stdout:3377: finishing write tid 5 to smithi17557870-25 2024-09-11T03:50:36.287 INFO:tasks.rados.rados.2.smithi175.stdout:3377: oid 25 updating version 0 to 980 2024-09-11T03:50:36.287 INFO:tasks.rados.rados.2.smithi175.stdout:3377: oid 25 version 980 is already newer than 977 2024-09-11T03:50:36.287 INFO:tasks.rados.rados.2.smithi175.stdout:3377: oid 25 version 980 is already newer than 978 2024-09-11T03:50:36.287 INFO:tasks.rados.rados.2.smithi175.stdout:3377: oid 25 version 980 is already newer than 979 2024-09-11T03:50:36.287 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 980 (ObjNum 1148 snap 371 seq_num 1148) dirty exists 2024-09-11T03:50:36.287 INFO:tasks.rados.rados.2.smithi175.stdout:3377: left oid 25 (ObjNum 1148 snap 371 seq_num 1148) 2024-09-11T03:50:36.287 INFO:tasks.rados.rados.2.smithi175.stdout:3378: finishing copy_from racing read to smithi17557870-10 2024-09-11T03:50:36.287 INFO:tasks.rados.rados.2.smithi175.stdout:3378: finishing copy_from to smithi17557870-10 2024-09-11T03:50:36.288 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 1053 (ObjNum 1120 snap 359 seq_num 1120) dirty exists 2024-09-11T03:50:36.288 INFO:tasks.rados.rados.2.smithi175.stdout:3379: finishing write tid 1 to smithi17557870-48 2024-09-11T03:50:36.288 INFO:tasks.rados.rados.2.smithi175.stdout:3379: finishing write tid 2 to smithi17557870-48 2024-09-11T03:50:36.289 INFO:tasks.rados.rados.2.smithi175.stdout:3379: finishing write tid 3 to smithi17557870-48 2024-09-11T03:50:36.289 INFO:tasks.rados.rados.2.smithi175.stdout:3379: finishing write tid 4 to smithi17557870-48 2024-09-11T03:50:36.289 INFO:tasks.rados.rados.2.smithi175.stdout:3379: finishing write tid 5 to smithi17557870-48 2024-09-11T03:50:36.289 INFO:tasks.rados.rados.2.smithi175.stdout:3379: oid 48 updating version 0 to 1001 2024-09-11T03:50:36.289 INFO:tasks.rados.rados.2.smithi175.stdout:3379: oid 48 version 1001 is already newer than 999 2024-09-11T03:50:36.289 INFO:tasks.rados.rados.2.smithi175.stdout:3379: oid 48 version 1001 is already newer than 1000 2024-09-11T03:50:36.289 INFO:tasks.rados.rados.2.smithi175.stdout:3379: oid 48 version 1001 is already newer than 998 2024-09-11T03:50:36.289 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 1001 (ObjNum 1149 snap 371 seq_num 1149) dirty exists 2024-09-11T03:50:36.289 INFO:tasks.rados.rados.2.smithi175.stdout:3379: left oid 48 (ObjNum 1149 snap 371 seq_num 1149) 2024-09-11T03:50:36.292 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:36.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:36.394 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:36.415 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:36.416 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:36.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:36 smithi084 ceph-mon[27254]: 2.3d scrub starts 2024-09-11T03:50:36.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:36 smithi084 ceph-mon[27254]: 2.3c scrub starts 2024-09-11T03:50:36.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:36 smithi084 ceph-mon[27254]: 2.1f scrub starts 2024-09-11T03:50:36.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:36 smithi084 ceph-mon[27254]: pgmap v1380: 97 pgs: 1 active+clean+scrubbing, 3 peering, 2 active+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 remapped+peering, 88 active+clean; 549 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s rd, 12 MiB/s wr, 93 op/s; 142/834 objects degraded (17.026%); 52/834 objects misplaced (6.235%); 6.3 MiB/s, 24 objects/s recovering 2024-09-11T03:50:36.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:36 smithi084 ceph-mon[27254]: 2.1b scrub starts 2024-09-11T03:50:36.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:36 smithi084 ceph-mon[27254]: 2.1b scrub ok 2024-09-11T03:50:36.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:36 smithi084 ceph-mon[27254]: 2.18 scrub starts 2024-09-11T03:50:36.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:36 smithi084 ceph-mon[27254]: 2.18 scrub ok 2024-09-11T03:50:36.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:36 smithi084 ceph-mon[27254]: 2.40 scrub starts 2024-09-11T03:50:36.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:36 smithi084 ceph-mon[27254]: 2.40 scrub ok 2024-09-11T03:50:36.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:36 smithi084 ceph-mon[27254]: 2.3b scrub starts 2024-09-11T03:50:36.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:36 smithi084 ceph-mon[27254]: 2.3b scrub ok 2024-09-11T03:50:36.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:36 smithi084 ceph-mon[27254]: 2.38 scrub starts 2024-09-11T03:50:36.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:36 smithi084 ceph-mon[27254]: 2.38 scrub ok 2024-09-11T03:50:36.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:36 smithi084 ceph-mon[27254]: 2.41 scrub starts 2024-09-11T03:50:36.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:36 smithi084 ceph-mon[27254]: 2.41 scrub ok 2024-09-11T03:50:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:36 smithi175 ceph-mon[30894]: 2.3d scrub starts 2024-09-11T03:50:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:36 smithi175 ceph-mon[30894]: 2.3c scrub starts 2024-09-11T03:50:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:36 smithi175 ceph-mon[30894]: 2.1f scrub starts 2024-09-11T03:50:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:36 smithi175 ceph-mon[30894]: pgmap v1380: 97 pgs: 1 active+clean+scrubbing, 3 peering, 2 active+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 remapped+peering, 88 active+clean; 549 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s rd, 12 MiB/s wr, 93 op/s; 142/834 objects degraded (17.026%); 52/834 objects misplaced (6.235%); 6.3 MiB/s, 24 objects/s recovering 2024-09-11T03:50:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:36 smithi175 ceph-mon[30894]: 2.1b scrub starts 2024-09-11T03:50:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:36 smithi175 ceph-mon[30894]: 2.1b scrub ok 2024-09-11T03:50:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:36 smithi175 ceph-mon[30894]: 2.18 scrub starts 2024-09-11T03:50:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:36 smithi175 ceph-mon[30894]: 2.18 scrub ok 2024-09-11T03:50:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:36 smithi175 ceph-mon[30894]: 2.40 scrub starts 2024-09-11T03:50:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:36 smithi175 ceph-mon[30894]: 2.40 scrub ok 2024-09-11T03:50:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:36 smithi175 ceph-mon[30894]: 2.3b scrub starts 2024-09-11T03:50:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:36 smithi175 ceph-mon[30894]: 2.3b scrub ok 2024-09-11T03:50:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:36 smithi175 ceph-mon[30894]: 2.38 scrub starts 2024-09-11T03:50:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:36 smithi175 ceph-mon[30894]: 2.38 scrub ok 2024-09-11T03:50:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:36 smithi175 ceph-mon[30894]: 2.41 scrub starts 2024-09-11T03:50:36.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:36 smithi175 ceph-mon[30894]: 2.41 scrub ok 2024-09-11T03:50:36.493 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:50:36.516 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:36.539 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:36.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:36.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:36.641 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:36.662 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:50:36.662 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:50:36.662 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:36.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:36.663 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:36 smithi121 ceph-mon[30729]: 2.3d scrub starts 2024-09-11T03:50:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:36 smithi121 ceph-mon[30729]: 2.3c scrub starts 2024-09-11T03:50:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:36 smithi121 ceph-mon[30729]: 2.1f scrub starts 2024-09-11T03:50:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:36 smithi121 ceph-mon[30729]: pgmap v1380: 97 pgs: 1 active+clean+scrubbing, 3 peering, 2 active+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 remapped+peering, 88 active+clean; 549 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s rd, 12 MiB/s wr, 93 op/s; 142/834 objects degraded (17.026%); 52/834 objects misplaced (6.235%); 6.3 MiB/s, 24 objects/s recovering 2024-09-11T03:50:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:36 smithi121 ceph-mon[30729]: 2.1b scrub starts 2024-09-11T03:50:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:36 smithi121 ceph-mon[30729]: 2.1b scrub ok 2024-09-11T03:50:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:36 smithi121 ceph-mon[30729]: 2.18 scrub starts 2024-09-11T03:50:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:36 smithi121 ceph-mon[30729]: 2.18 scrub ok 2024-09-11T03:50:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:36 smithi121 ceph-mon[30729]: 2.40 scrub starts 2024-09-11T03:50:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:36 smithi121 ceph-mon[30729]: 2.40 scrub ok 2024-09-11T03:50:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:36 smithi121 ceph-mon[30729]: 2.3b scrub starts 2024-09-11T03:50:36.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:36 smithi121 ceph-mon[30729]: 2.3b scrub ok 2024-09-11T03:50:36.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:36 smithi121 ceph-mon[30729]: 2.38 scrub starts 2024-09-11T03:50:36.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:36 smithi121 ceph-mon[30729]: 2.38 scrub ok 2024-09-11T03:50:36.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:36 smithi121 ceph-mon[30729]: 2.41 scrub starts 2024-09-11T03:50:36.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:36 smithi121 ceph-mon[30729]: 2.41 scrub ok 2024-09-11T03:50:36.679 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:50:36.679 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:50:36.695 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:50:36.695 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:50:36.711 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:50:36.711 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:50:36.727 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:50:36.727 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:50:36.744 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:50:36.744 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:50:36.761 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:50:36.761 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:50:36.764 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:36.777 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:50:36.778 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:50:36.786 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:36.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:36.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:36.794 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:50:36.794 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:50:36.810 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:50:36.810 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:50:36.826 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:50:36.826 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:50:36.843 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:50:36.843 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:50:36.888 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:36.909 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:36.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:36.910 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:36.975 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:50:36.976 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:50:36.976 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:50:36.976 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:50:36.976 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.976 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2254 2.18 2:1984a3a9:::smithi17557870-23:head [write 746310~635349,stat] snapc 5d=[5d,5a,59,58,56] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:50:36.976 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.883765+0000", 2024-09-11T03:50:36.976 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 540.08384711300005, 2024-09-11T03:50:36.976 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022154396, 2024-09-11T03:50:36.976 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:36.976 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:36.977 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:36.977 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:36.977 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:36.977 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:36.977 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2254 2024-09-11T03:50:36.977 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.977 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:36.977 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.977 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:36.977 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.883765+0000", 2024-09-11T03:50:36.977 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:36.977 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.977 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.978 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:36.978 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.883765+0000", 2024-09-11T03:50:36.978 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:36.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.978 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:36.978 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.883768+0000", 2024-09-11T03:50:36.978 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4080000000000001e-06 2024-09-11T03:50:36.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.978 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:36.978 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884378+0000", 2024-09-11T03:50:36.978 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00061034200000000005 2024-09-11T03:50:36.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.979 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.979 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:36.979 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884399+0000", 2024-09-11T03:50:36.979 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1175e-05 2024-09-11T03:50:36.979 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.979 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.979 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:36.979 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884412+0000", 2024-09-11T03:50:36.979 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3015e-05 2024-09-11T03:50:36.979 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.979 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.979 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:36.979 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884520+0000", 2024-09-11T03:50:36.979 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000107706 2024-09-11T03:50:36.980 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.980 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.980 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:50:36.980 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884534+0000", 2024-09-11T03:50:36.980 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3923e-05 2024-09-11T03:50:36.980 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.980 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.980 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:36.980 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.895937+0000", 2024-09-11T03:50:36.980 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011403207 2024-09-11T03:50:36.980 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.980 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.981 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:50:36.981 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896424+0000", 2024-09-11T03:50:36.981 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00048641199999999998 2024-09-11T03:50:36.981 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.981 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.981 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:36.981 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.898138+0000", 2024-09-11T03:50:36.981 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001714272 2024-09-11T03:50:36.981 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.981 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.981 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:36.981 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.898309+0000", 2024-09-11T03:50:36.981 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017075200000000001 2024-09-11T03:50:36.981 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.982 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.982 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:50:36.982 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.898561+0000", 2024-09-11T03:50:36.982 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00025223500000000001 2024-09-11T03:50:36.982 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.982 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.982 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:36.982 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903048+0000", 2024-09-11T03:50:36.982 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0044876320000000001 2024-09-11T03:50:36.982 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.982 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.982 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:36.982 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903118+0000", 2024-09-11T03:50:36.982 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9608000000000002e-05 2024-09-11T03:50:36.983 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.983 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.983 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:36.983 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905694+0000", 2024-09-11T03:50:36.983 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025761650000000001 2024-09-11T03:50:36.983 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.983 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.983 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:36.983 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905749+0000", 2024-09-11T03:50:36.983 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5037999999999997e-05 2024-09-11T03:50:36.983 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.983 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.983 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:36.984 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905920+0000", 2024-09-11T03:50:36.984 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017050600000000001 2024-09-11T03:50:36.984 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:36.984 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:36.984 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:36.984 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.984 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.984 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2255 2.18 2:1984a3a9:::smithi17557870-23:head [write 2077646~576443] snapc 5d=[5d,5a,59,58,56] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:50:36.984 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.884463+0000", 2024-09-11T03:50:36.984 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 540.08314938199999, 2024-09-11T03:50:36.984 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022426167, 2024-09-11T03:50:36.984 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:36.985 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:36.985 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:36.985 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:36.985 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:36.985 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:36.985 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2255 2024-09-11T03:50:36.985 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.985 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:36.985 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.985 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:36.985 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884463+0000", 2024-09-11T03:50:36.985 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:36.985 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.986 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.986 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:36.986 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884463+0000", 2024-09-11T03:50:36.986 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:36.986 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.986 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.986 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:36.986 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.884464+0000", 2024-09-11T03:50:36.986 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.64e-07 2024-09-11T03:50:36.986 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.986 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.986 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:36.986 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885204+0000", 2024-09-11T03:50:36.987 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00074007200000000004 2024-09-11T03:50:36.987 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.987 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.987 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:36.987 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885217+0000", 2024-09-11T03:50:36.987 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2547e-05 2024-09-11T03:50:36.987 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.987 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.987 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:36.987 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885226+0000", 2024-09-11T03:50:36.987 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1849999999999993e-06 2024-09-11T03:50:36.987 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.987 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.987 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:36.988 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885295+0000", 2024-09-11T03:50:36.988 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8657000000000004e-05 2024-09-11T03:50:36.988 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.988 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.988 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:50:36.988 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885307+0000", 2024-09-11T03:50:36.988 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2529e-05 2024-09-11T03:50:36.988 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.988 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.988 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:36.988 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896570+0000", 2024-09-11T03:50:36.988 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011262529 2024-09-11T03:50:36.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.989 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.989 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:50:36.989 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896624+0000", 2024-09-11T03:50:36.989 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4614000000000003e-05 2024-09-11T03:50:36.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.989 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.989 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:36.989 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.900547+0000", 2024-09-11T03:50:36.989 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039228850000000001 2024-09-11T03:50:36.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.989 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.990 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:36.990 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.900706+0000", 2024-09-11T03:50:36.990 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015907000000000001 2024-09-11T03:50:36.990 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.990 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.990 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:50:36.990 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.900901+0000", 2024-09-11T03:50:36.990 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00019514399999999999 2024-09-11T03:50:36.990 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.990 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.990 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:36.990 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903192+0000", 2024-09-11T03:50:36.991 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022906519999999998 2024-09-11T03:50:36.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.991 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:36.991 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.904724+0000", 2024-09-11T03:50:36.991 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015319520000000001 2024-09-11T03:50:36.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.991 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:36.991 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906254+0000", 2024-09-11T03:50:36.991 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015302759999999999 2024-09-11T03:50:36.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.992 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.992 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:36.992 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906358+0000", 2024-09-11T03:50:36.992 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010404300000000001 2024-09-11T03:50:36.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.992 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.992 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:36.992 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906889+0000", 2024-09-11T03:50:36.992 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000531048 2024-09-11T03:50:36.992 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:36.992 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:36.993 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:36.993 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.993 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.993 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2256 2.18 2:1984a3a9:::smithi17557870-23:head [write 3362764~183546] snapc 5d=[5d,5a,59,58,56] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:50:36.993 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.885408+0000", 2024-09-11T03:50:36.993 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 540.082204047, 2024-09-11T03:50:36.993 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021926139000000001, 2024-09-11T03:50:36.993 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:36.993 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:36.993 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:36.993 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:36.993 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:36.994 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:36.994 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2256 2024-09-11T03:50:36.994 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.994 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:36.994 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.994 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:36.994 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885408+0000", 2024-09-11T03:50:36.994 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:36.994 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.994 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.994 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:36.994 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885408+0000", 2024-09-11T03:50:36.994 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:36.994 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.995 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.995 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:36.995 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885410+0000", 2024-09-11T03:50:36.995 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4270000000000001e-06 2024-09-11T03:50:36.995 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.995 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.995 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:36.995 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885561+0000", 2024-09-11T03:50:36.995 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000151254 2024-09-11T03:50:36.995 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.995 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.995 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:36.995 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885573+0000", 2024-09-11T03:50:36.996 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1897e-05 2024-09-11T03:50:36.996 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.996 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.996 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:36.996 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885581+0000", 2024-09-11T03:50:36.996 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6290000000000001e-06 2024-09-11T03:50:36.996 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.996 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.996 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:36.997 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885643+0000", 2024-09-11T03:50:36.997 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2215e-05 2024-09-11T03:50:36.997 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.997 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.997 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:50:36.997 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885652+0000", 2024-09-11T03:50:36.997 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5580000000000006e-06 2024-09-11T03:50:36.997 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.997 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.997 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:36.997 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896678+0000", 2024-09-11T03:50:36.997 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011025956 2024-09-11T03:50:36.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.998 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:50:36.998 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896729+0000", 2024-09-11T03:50:36.998 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1019000000000001e-05 2024-09-11T03:50:36.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.998 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:36.998 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903250+0000", 2024-09-11T03:50:36.999 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0065205109999999997 2024-09-11T03:50:36.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.999 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.999 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:36.999 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903415+0000", 2024-09-11T03:50:36.999 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016545399999999999 2024-09-11T03:50:36.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.999 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:36.999 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:50:36.999 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.903701+0000", 2024-09-11T03:50:36.999 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00028586399999999999 2024-09-11T03:50:36.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:36.999 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.000 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.000 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905681+0000", 2024-09-11T03:50:37.000 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0019798950000000002 2024-09-11T03:50:37.000 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.000 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.000 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.000 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905999+0000", 2024-09-11T03:50:37.000 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00031789999999999998 2024-09-11T03:50:37.000 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.000 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.000 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:37.000 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906904+0000", 2024-09-11T03:50:37.001 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00090535599999999996 2024-09-11T03:50:37.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.001 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.001 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:37.001 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907002+0000", 2024-09-11T03:50:37.001 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7653999999999997e-05 2024-09-11T03:50:37.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.001 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.001 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.001 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907335+0000", 2024-09-11T03:50:37.001 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00033254999999999998 2024-09-11T03:50:37.001 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.001 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.002 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.002 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.002 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.002 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2257 2.18 2:1984a3a9:::smithi17557870-23:head [setxattr _header (57) in=7b,truncate 3546310] snapc 5d=[5d,5a,59,58,56] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:50:37.002 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.885667+0000", 2024-09-11T03:50:37.002 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 540.08194502599997, 2024-09-11T03:50:37.002 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022144878999999999, 2024-09-11T03:50:37.002 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:37.002 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:37.002 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:37.002 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:37.002 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:37.003 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:37.003 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2257 2024-09-11T03:50:37.003 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.003 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:37.003 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.003 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:37.003 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885667+0000", 2024-09-11T03:50:37.003 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.003 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.003 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.003 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:37.003 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885667+0000", 2024-09-11T03:50:37.003 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.004 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.004 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.004 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:37.004 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885668+0000", 2024-09-11T03:50:37.004 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5600000000000005e-07 2024-09-11T03:50:37.004 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.004 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.004 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:37.004 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885673+0000", 2024-09-11T03:50:37.004 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4970000000000003e-06 2024-09-11T03:50:37.004 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.004 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.005 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:37.005 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885680+0000", 2024-09-11T03:50:37.005 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3780000000000003e-06 2024-09-11T03:50:37.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.005 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.005 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:37.005 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.885686+0000", 2024-09-11T03:50:37.005 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0789999999999997e-06 2024-09-11T03:50:37.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.005 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.005 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.005 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886080+0000", 2024-09-11T03:50:37.006 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00039392399999999998 2024-09-11T03:50:37.006 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.006 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.006 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:50:37.006 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886091+0000", 2024-09-11T03:50:37.006 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0472000000000001e-05 2024-09-11T03:50:37.006 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.006 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.006 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.006 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896776+0000", 2024-09-11T03:50:37.006 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010684953000000001 2024-09-11T03:50:37.006 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.006 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.007 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:50:37.007 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896839+0000", 2024-09-11T03:50:37.007 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3293999999999994e-05 2024-09-11T03:50:37.007 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.007 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.007 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.008 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.904784+0000", 2024-09-11T03:50:37.008 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.007944843 2024-09-11T03:50:37.008 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.008 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.008 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:37.008 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.904909+0000", 2024-09-11T03:50:37.008 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012491600000000001 2024-09-11T03:50:37.008 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.008 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.008 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:50:37.009 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905105+0000", 2024-09-11T03:50:37.009 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000196336 2024-09-11T03:50:37.009 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.009 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.009 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.009 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.906057+0000", 2024-09-11T03:50:37.009 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00095239800000000002 2024-09-11T03:50:37.009 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.009 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.009 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:37.009 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907348+0000", 2024-09-11T03:50:37.009 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012904489999999999 2024-09-11T03:50:37.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.010 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.010 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.010 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907457+0000", 2024-09-11T03:50:37.010 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108869 2024-09-11T03:50:37.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.010 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.010 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:37.010 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907543+0000", 2024-09-11T03:50:37.010 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6212999999999996e-05 2024-09-11T03:50:37.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.010 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.010 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.011 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907812+0000", 2024-09-11T03:50:37.011 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00026950200000000002 2024-09-11T03:50:37.011 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.011 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.011 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.011 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.011 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.011 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2258 2.18 2:1984a3a9:::smithi17557870-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:50:37.011 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:36.886095+0000", 2024-09-11T03:50:37.011 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 540.08151726100004, 2024-09-11T03:50:37.011 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022147336, 2024-09-11T03:50:37.011 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:37.011 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:37.012 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:50:37.012 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:37.012 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:37.012 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:37.012 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2258 2024-09-11T03:50:37.012 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.012 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:37.012 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.012 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:37.012 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886095+0000", 2024-09-11T03:50:37.012 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.013 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.013 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.013 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:37.013 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886095+0000", 2024-09-11T03:50:37.013 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.013 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.013 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.013 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:37.013 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886096+0000", 2024-09-11T03:50:37.013 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0200000000000001e-07 2024-09-11T03:50:37.013 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.013 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.013 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:37.014 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886100+0000", 2024-09-11T03:50:37.014 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9859999999999996e-06 2024-09-11T03:50:37.014 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.014 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.014 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:37.014 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886109+0000", 2024-09-11T03:50:37.014 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6349999999999995e-06 2024-09-11T03:50:37.014 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.014 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.014 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:37.014 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886116+0000", 2024-09-11T03:50:37.014 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0890000000000002e-06 2024-09-11T03:50:37.014 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.014 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.015 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.015 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886389+0000", 2024-09-11T03:50:37.015 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00027359200000000002 2024-09-11T03:50:37.015 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.015 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.015 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:50:37.015 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.886399+0000", 2024-09-11T03:50:37.015 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9599999999999995e-06 2024-09-11T03:50:37.015 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.015 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.015 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.015 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896922+0000", 2024-09-11T03:50:37.015 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010522567 2024-09-11T03:50:37.016 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.016 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.016 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for missing object", 2024-09-11T03:50:37.016 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.896977+0000", 2024-09-11T03:50:37.016 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5003999999999997e-05 2024-09-11T03:50:37.016 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.016 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.016 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.016 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905540+0000", 2024-09-11T03:50:37.016 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0085636329999999993 2024-09-11T03:50:37.016 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.016 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.016 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:50:37.017 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.905625+0000", 2024-09-11T03:50:37.017 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4468999999999994e-05 2024-09-11T03:50:37.017 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.017 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.017 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.017 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.907869+0000", 2024-09-11T03:50:37.017 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022438290000000001 2024-09-11T03:50:37.017 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.017 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.017 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:37.017 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.908014+0000", 2024-09-11T03:50:37.017 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00014511799999999999 2024-09-11T03:50:37.017 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.018 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.018 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.018 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:36.908243+0000", 2024-09-11T03:50:37.018 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00022865200000000001 2024-09-11T03:50:37.018 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.018 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.018 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.018 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.018 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.018 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2421 2.18 2:1bdf1a36:::smithi17557870-36:head [copy-from ver 157] snapc 65=[65,64,63,61,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-11T03:50:37.018 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:41:50.954849+0000", 2024-09-11T03:50:37.018 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 526.01276323100001, 2024-09-11T03:50:37.018 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022711387999999999, 2024-09-11T03:50:37.019 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:37.019 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:37.019 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:37.019 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:37.019 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:37.019 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:37.019 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2421 2024-09-11T03:50:37.019 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.019 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:37.019 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.019 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:37.019 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954849+0000", 2024-09-11T03:50:37.019 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.019 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.020 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.020 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:37.020 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954849+0000", 2024-09-11T03:50:37.020 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.020 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.020 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.020 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:37.020 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954851+0000", 2024-09-11T03:50:37.020 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3090000000000001e-06 2024-09-11T03:50:37.020 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.020 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.020 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:37.020 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954856+0000", 2024-09-11T03:50:37.021 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9059999999999999e-06 2024-09-11T03:50:37.021 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.021 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.021 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:37.021 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954867+0000", 2024-09-11T03:50:37.021 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0879999999999999e-05 2024-09-11T03:50:37.021 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.021 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.021 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:37.021 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954873+0000", 2024-09-11T03:50:37.021 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.203e-06 2024-09-11T03:50:37.021 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.022 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.022 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.022 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.954921+0000", 2024-09-11T03:50:37.022 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8180000000000003e-05 2024-09-11T03:50:37.022 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.022 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.022 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:37.022 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.955004+0000", 2024-09-11T03:50:37.022 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2824000000000005e-05 2024-09-11T03:50:37.022 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.022 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.022 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,11", 2024-09-11T03:50:37.022 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.964099+0000", 2024-09-11T03:50:37.023 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0090952499999999992 2024-09-11T03:50:37.023 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.023 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.023 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.023 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.970182+0000", 2024-09-11T03:50:37.023 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0060831510000000002 2024-09-11T03:50:37.023 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.023 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.023 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:37.023 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977039+0000", 2024-09-11T03:50:37.023 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0068571980000000001 2024-09-11T03:50:37.023 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.023 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.024 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.024 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977432+0000", 2024-09-11T03:50:37.024 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00039304900000000002 2024-09-11T03:50:37.024 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.024 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.024 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:37.024 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977504+0000", 2024-09-11T03:50:37.024 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1863e-05 2024-09-11T03:50:37.024 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.024 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.024 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.024 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:41:50.977561+0000", 2024-09-11T03:50:37.024 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6575e-05 2024-09-11T03:50:37.025 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.025 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.025 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.025 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.025 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.025 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4687 2.18 2:1984a3a9:::smithi17557870-23:head [write 739673~657763,stat] snapc cd=[cd,cc,ca,c5,c2,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-11T03:50:37.025 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.003513+0000", 2024-09-11T03:50:37.025 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 315.96409992299999, 2024-09-11T03:50:37.025 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.028701775999999998, 2024-09-11T03:50:37.025 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:37.025 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:37.025 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:37.025 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:37.026 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:37.026 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:37.026 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4687 2024-09-11T03:50:37.026 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.026 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:37.026 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.026 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:37.026 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003513+0000", 2024-09-11T03:50:37.026 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.026 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.026 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.027 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:37.027 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003513+0000", 2024-09-11T03:50:37.027 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.027 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.027 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.027 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:37.027 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003515+0000", 2024-09-11T03:50:37.027 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5600000000000001e-06 2024-09-11T03:50:37.027 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.027 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.027 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:37.027 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003862+0000", 2024-09-11T03:50:37.027 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00034641300000000001 2024-09-11T03:50:37.028 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.028 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.028 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:37.028 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003880+0000", 2024-09-11T03:50:37.028 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8669e-05 2024-09-11T03:50:37.028 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.028 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.028 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:37.028 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003906+0000", 2024-09-11T03:50:37.028 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5432999999999999e-05 2024-09-11T03:50:37.028 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.028 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.028 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.029 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003972+0000", 2024-09-11T03:50:37.029 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6729000000000004e-05 2024-09-11T03:50:37.029 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.029 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.029 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:50:37.029 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003982+0000", 2024-09-11T03:50:37.029 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2979999999999997e-06 2024-09-11T03:50:37.029 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.029 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.029 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.029 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.005521+0000", 2024-09-11T03:50:37.029 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015392959999999999 2024-09-11T03:50:37.029 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.029 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.030 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:37.030 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.005611+0000", 2024-09-11T03:50:37.030 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.0373999999999999e-05 2024-09-11T03:50:37.030 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.030 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.030 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,7", 2024-09-11T03:50:37.030 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.005729+0000", 2024-09-11T03:50:37.030 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000117834 2024-09-11T03:50:37.030 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.030 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.030 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:37.030 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.007175+0000", 2024-09-11T03:50:37.030 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014461140000000001 2024-09-11T03:50:37.031 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.031 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.031 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.031 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.008417+0000", 2024-09-11T03:50:37.031 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012421120000000001 2024-09-11T03:50:37.031 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.031 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.031 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.031 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032091+0000", 2024-09-11T03:50:37.031 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.023674066000000001 2024-09-11T03:50:37.031 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.031 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.031 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:37.032 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032146+0000", 2024-09-11T03:50:37.032 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4786e-05 2024-09-11T03:50:37.032 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.032 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.032 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.032 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032214+0000", 2024-09-11T03:50:37.032 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8091999999999998e-05 2024-09-11T03:50:37.032 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.032 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.032 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.032 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.032 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.032 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4688 2.18 2:1984a3a9:::smithi17557870-23:head [write 2076229~663444] snapc cd=[cd,cc,ca,c5,c2,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-11T03:50:37.032 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.010935+0000", 2024-09-11T03:50:37.033 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 315.95667786299998, 2024-09-11T03:50:37.033 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021404982999999999, 2024-09-11T03:50:37.033 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:37.033 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:37.033 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:37.033 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:37.033 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:37.033 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:37.033 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4688 2024-09-11T03:50:37.033 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.033 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:37.033 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.033 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:37.034 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.010935+0000", 2024-09-11T03:50:37.034 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.034 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.034 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.034 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:37.034 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.010935+0000", 2024-09-11T03:50:37.034 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.034 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.034 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.034 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:37.034 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.010936+0000", 2024-09-11T03:50:37.034 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1689999999999999e-06 2024-09-11T03:50:37.034 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.034 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.035 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:37.035 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011065+0000", 2024-09-11T03:50:37.035 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000128997 2024-09-11T03:50:37.035 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.035 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.035 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:37.035 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011076+0000", 2024-09-11T03:50:37.035 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1561e-05 2024-09-11T03:50:37.035 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.035 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.035 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:37.035 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011082+0000", 2024-09-11T03:50:37.035 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2599999999999996e-06 2024-09-11T03:50:37.036 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.036 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.036 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.036 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011127+0000", 2024-09-11T03:50:37.036 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5509000000000001e-05 2024-09-11T03:50:37.036 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.036 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.036 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:37.036 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011212+0000", 2024-09-11T03:50:37.036 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:37.037 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 2 pgp_num to 66 2024-09-11T03:50:37.038 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5173999999999995e-05 2024-09-11T03:50:37.039 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.039 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.039 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,7", 2024-09-11T03:50:37.039 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011309+0000", 2024-09-11T03:50:37.039 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6980999999999999e-05 2024-09-11T03:50:37.039 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.039 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.039 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:37.039 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012804+0000", 2024-09-11T03:50:37.039 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001495019 2024-09-11T03:50:37.039 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.039 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.040 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.040 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.031586+0000", 2024-09-11T03:50:37.040 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018782099 2024-09-11T03:50:37.040 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.040 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.040 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.040 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032261+0000", 2024-09-11T03:50:37.040 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000674846 2024-09-11T03:50:37.040 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.040 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.040 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:37.041 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032297+0000", 2024-09-11T03:50:37.041 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6010000000000003e-05 2024-09-11T03:50:37.041 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.041 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.041 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.041 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032340+0000", 2024-09-11T03:50:37.041 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2357999999999997e-05 2024-09-11T03:50:37.041 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.041 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.041 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.041 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.041 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.042 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4689 2.18 2:1984a3a9:::smithi17557870-23:head [setxattr _header (58) in=7b,truncate 2739673] snapc cd=[cd,cc,ca,c5,c2,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-11T03:50:37.042 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.011093+0000", 2024-09-11T03:50:37.042 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 315.95651938700001, 2024-09-11T03:50:37.042 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.02140688, 2024-09-11T03:50:37.042 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:37.042 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:37.042 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:37.042 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:37.042 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:37.042 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:37.042 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4689 2024-09-11T03:50:37.042 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.043 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:37.043 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.043 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:37.043 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011093+0000", 2024-09-11T03:50:37.043 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.043 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.043 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.043 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:37.043 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011093+0000", 2024-09-11T03:50:37.043 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.043 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.043 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.043 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:37.044 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011094+0000", 2024-09-11T03:50:37.044 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9999999999999997e-07 2024-09-11T03:50:37.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.044 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.044 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:37.044 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011100+0000", 2024-09-11T03:50:37.044 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1369999999999999e-06 2024-09-11T03:50:37.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.044 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.044 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:37.044 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011106+0000", 2024-09-11T03:50:37.044 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.421e-06 2024-09-11T03:50:37.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.045 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.045 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:37.045 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011117+0000", 2024-09-11T03:50:37.045 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1012999999999999e-05 2024-09-11T03:50:37.045 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.045 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.045 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.045 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012151+0000", 2024-09-11T03:50:37.045 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0010333880000000001 2024-09-11T03:50:37.045 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.045 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.045 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:37.045 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012265+0000", 2024-09-11T03:50:37.046 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114141 2024-09-11T03:50:37.046 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.046 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.046 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,7", 2024-09-11T03:50:37.046 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012418+0000", 2024-09-11T03:50:37.046 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015286499999999999 2024-09-11T03:50:37.046 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.046 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.046 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:37.046 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012807+0000", 2024-09-11T03:50:37.046 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00038979800000000001 2024-09-11T03:50:37.046 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.046 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.047 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.047 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.031717+0000", 2024-09-11T03:50:37.047 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018909888 2024-09-11T03:50:37.047 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.047 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.047 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.047 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032383+0000", 2024-09-11T03:50:37.047 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00066515900000000002 2024-09-11T03:50:37.047 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.047 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.047 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:37.047 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032447+0000", 2024-09-11T03:50:37.047 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4958999999999999e-05 2024-09-11T03:50:37.048 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.048 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.048 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.048 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032500+0000", 2024-09-11T03:50:37.048 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2510999999999999e-05 2024-09-11T03:50:37.048 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.048 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.048 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.048 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.048 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.048 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4690 2.18 2:1984a3a9:::smithi17557870-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e503)", 2024-09-11T03:50:37.048 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.011126+0000", 2024-09-11T03:50:37.048 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 315.95648615900001, 2024-09-11T03:50:37.049 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021513529, 2024-09-11T03:50:37.049 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:37.049 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:37.049 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:50:37.049 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:37.049 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:37.049 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:37.049 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4690 2024-09-11T03:50:37.049 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.049 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:37.049 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.049 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:37.049 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011126+0000", 2024-09-11T03:50:37.050 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.050 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.050 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.050 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:37.050 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011126+0000", 2024-09-11T03:50:37.050 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.050 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.050 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.050 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:37.050 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011127+0000", 2024-09-11T03:50:37.050 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8800000000000003e-07 2024-09-11T03:50:37.050 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.051 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.051 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:37.051 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011129+0000", 2024-09-11T03:50:37.051 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0090000000000002e-06 2024-09-11T03:50:37.051 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.051 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.051 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:37.051 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011146+0000", 2024-09-11T03:50:37.051 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7209000000000001e-05 2024-09-11T03:50:37.052 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.052 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.052 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:37.052 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011152+0000", 2024-09-11T03:50:37.052 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5160000000000002e-06 2024-09-11T03:50:37.052 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.052 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.052 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.052 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012718+0000", 2024-09-11T03:50:37.052 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001566413 2024-09-11T03:50:37.052 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.052 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.052 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:50:37.053 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012793+0000", 2024-09-11T03:50:37.053 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5197999999999994e-05 2024-09-11T03:50:37.053 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.053 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.053 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.053 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032530+0000", 2024-09-11T03:50:37.053 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019736848000000001 2024-09-11T03:50:37.053 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.053 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.053 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:37.053 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032577+0000", 2024-09-11T03:50:37.053 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6878000000000001e-05 2024-09-11T03:50:37.053 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.054 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.054 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.054 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032640+0000", 2024-09-11T03:50:37.054 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2970000000000002e-05 2024-09-11T03:50:37.054 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.054 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.054 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.054 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.054 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.054 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5691 2.e 2:77dadb97:::smithi17557870-30:head [write 2119636~654941] snapc fa=[] ondisk+write+known_if_redirected+supports_pool_eio e601)", 2024-09-11T03:50:37.054 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:59.009165+0000", 2024-09-11T03:50:37.055 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 217.95844719300001, 2024-09-11T03:50:37.055 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.052209984000000001, 2024-09-11T03:50:37.055 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:37.055 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:37.055 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:37.055 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:37.055 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:37.055 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:37.055 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5691 2024-09-11T03:50:37.055 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.055 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:37.055 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.055 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:37.055 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009165+0000", 2024-09-11T03:50:37.056 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.056 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.056 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.056 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:37.056 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009165+0000", 2024-09-11T03:50:37.056 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.056 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.056 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.056 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:37.056 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009166+0000", 2024-09-11T03:50:37.056 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.204e-06 2024-09-11T03:50:37.056 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.057 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.057 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:37.057 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009306+0000", 2024-09-11T03:50:37.057 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000139426 2024-09-11T03:50:37.057 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.057 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.057 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:37.057 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009318+0000", 2024-09-11T03:50:37.057 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2329e-05 2024-09-11T03:50:37.057 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.057 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.057 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:37.057 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009324+0000", 2024-09-11T03:50:37.058 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9000000000000003e-06 2024-09-11T03:50:37.058 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.058 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.058 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.058 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009379+0000", 2024-09-11T03:50:37.058 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5285999999999999e-05 2024-09-11T03:50:37.058 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.058 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.058 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:37.058 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009484+0000", 2024-09-11T03:50:37.058 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010491400000000001 2024-09-11T03:50:37.058 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.058 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.059 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:50:37.059 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009651+0000", 2024-09-11T03:50:37.059 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016665500000000001 2024-09-11T03:50:37.059 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.059 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.059 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:37.059 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.011223+0000", 2024-09-11T03:50:37.059 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001572452 2024-09-11T03:50:37.059 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.059 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.060 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.060 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.021352+0000", 2024-09-11T03:50:37.060 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010128587999999999 2024-09-11T03:50:37.060 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.060 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.060 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.060 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061260+0000", 2024-09-11T03:50:37.060 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.039907610000000003 2024-09-11T03:50:37.060 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.060 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.060 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:37.060 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061325+0000", 2024-09-11T03:50:37.060 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.58e-05 2024-09-11T03:50:37.061 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.061 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.061 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.061 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061375+0000", 2024-09-11T03:50:37.061 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9820000000000001e-05 2024-09-11T03:50:37.061 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.061 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.061 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.061 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.061 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.061 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5692 2.e 2:77dadb97:::smithi17557870-30:head [setxattr _header (58) in=7b,truncate 3159881] snapc fa=[] ondisk+write+known_if_redirected+supports_pool_eio e601)", 2024-09-11T03:50:37.061 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:59.009337+0000", 2024-09-11T03:50:37.061 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 217.95827502500001, 2024-09-11T03:50:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.052168173999999998, 2024-09-11T03:50:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5692 2024-09-11T03:50:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009337+0000", 2024-09-11T03:50:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009337+0000", 2024-09-11T03:50:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009338+0000", 2024-09-11T03:50:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0900000000000002e-07 2024-09-11T03:50:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009340+0000", 2024-09-11T03:50:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2120000000000002e-06 2024-09-11T03:50:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009345+0000", 2024-09-11T03:50:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5079999999999999e-06 2024-09-11T03:50:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009360+0000", 2024-09-11T03:50:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5191000000000001e-05 2024-09-11T03:50:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.010683+0000", 2024-09-11T03:50:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013236929999999999 2024-09-11T03:50:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.010767+0000", 2024-09-11T03:50:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3662e-05 2024-09-11T03:50:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:50:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.010905+0000", 2024-09-11T03:50:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013769100000000001 2024-09-11T03:50:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.011384+0000", 2024-09-11T03:50:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00047889999999999999 2024-09-11T03:50:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.021405+0000", 2024-09-11T03:50:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010021486 2024-09-11T03:50:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061414+0000", 2024-09-11T03:50:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.040009265000000002 2024-09-11T03:50:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061467+0000", 2024-09-11T03:50:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2589000000000001e-05 2024-09-11T03:50:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061506+0000", 2024-09-11T03:50:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8568000000000001e-05 2024-09-11T03:50:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5693 2.e 2:77dadb97:::smithi17557870-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e601)", 2024-09-11T03:50:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:59.009365+0000", 2024-09-11T03:50:37.069 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 217.958247269, 2024-09-11T03:50:37.069 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.052354291999999997, 2024-09-11T03:50:37.069 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:37.069 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:37.069 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:50:37.069 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:37.069 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:37.069 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:37.069 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5693 2024-09-11T03:50:37.069 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.069 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:37.069 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.070 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:37.070 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009365+0000", 2024-09-11T03:50:37.070 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.070 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.070 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.070 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:37.070 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009365+0000", 2024-09-11T03:50:37.070 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.070 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.070 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.070 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:37.070 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009366+0000", 2024-09-11T03:50:37.070 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0600000000000001e-07 2024-09-11T03:50:37.070 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.071 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.071 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:37.071 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009367+0000", 2024-09-11T03:50:37.071 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5990000000000001e-06 2024-09-11T03:50:37.071 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.071 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.071 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:37.071 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009371+0000", 2024-09-11T03:50:37.071 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.1200000000000004e-06 2024-09-11T03:50:37.071 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.071 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.071 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:37.072 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009374+0000", 2024-09-11T03:50:37.072 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0400000000000001e-06 2024-09-11T03:50:37.072 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.072 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.072 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.072 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.011251+0000", 2024-09-11T03:50:37.072 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001877058 2024-09-11T03:50:37.072 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.072 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.072 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:50:37.072 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.011330+0000", 2024-09-11T03:50:37.072 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8325000000000006e-05 2024-09-11T03:50:37.072 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.073 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.073 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.073 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061544+0000", 2024-09-11T03:50:37.073 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.050214591000000003 2024-09-11T03:50:37.073 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.073 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.073 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:37.073 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061626+0000", 2024-09-11T03:50:37.073 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1976000000000003e-05 2024-09-11T03:50:37.073 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.073 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.073 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.073 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061720+0000", 2024-09-11T03:50:37.074 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3276999999999995e-05 2024-09-11T03:50:37.074 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.074 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.074 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.074 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.074 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.074 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5885 2.e 2:77dadb97:::smithi17557870-30:head [copy-from ver 882] snapc 101=[101,100] ondisk+write+known_if_redirected+supports_pool_eio e613)", 2024-09-11T03:50:37.074 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:11.238029+0000", 2024-09-11T03:50:37.074 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 205.72958332100001, 2024-09-11T03:50:37.074 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.02012736, 2024-09-11T03:50:37.074 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:37.074 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:37.074 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:37.075 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:37.075 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:37.075 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:37.075 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5885 2024-09-11T03:50:37.075 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.075 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:37.075 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.075 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:37.075 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.238029+0000", 2024-09-11T03:50:37.075 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.075 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.075 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.076 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:37.076 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.238029+0000", 2024-09-11T03:50:37.076 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.076 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.076 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.076 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:37.076 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.238032+0000", 2024-09-11T03:50:37.076 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.401e-06 2024-09-11T03:50:37.076 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.076 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.076 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:37.076 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.238042+0000", 2024-09-11T03:50:37.077 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0108e-05 2024-09-11T03:50:37.077 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.077 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.077 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:37.077 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.238064+0000", 2024-09-11T03:50:37.077 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2651999999999999e-05 2024-09-11T03:50:37.077 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.077 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.077 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:37.077 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.238074+0000", 2024-09-11T03:50:37.077 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3079999999999997e-06 2024-09-11T03:50:37.077 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.078 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.078 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.078 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.238165+0000", 2024-09-11T03:50:37.078 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1645999999999996e-05 2024-09-11T03:50:37.078 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.078 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.078 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:50:37.078 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.238173+0000", 2024-09-11T03:50:37.078 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9400000000000002e-06 2024-09-11T03:50:37.078 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.078 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.078 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.078 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.239797+0000", 2024-09-11T03:50:37.079 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0016233650000000001 2024-09-11T03:50:37.079 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.079 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.079 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:37.079 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.239860+0000", 2024-09-11T03:50:37.079 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3040999999999995e-05 2024-09-11T03:50:37.079 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.079 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.079 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:50:37.079 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.245678+0000", 2024-09-11T03:50:37.079 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0058182770000000002 2024-09-11T03:50:37.079 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.079 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.080 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:37.080 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.250568+0000", 2024-09-11T03:50:37.080 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0048903560000000002 2024-09-11T03:50:37.080 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.080 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.080 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.080 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.256230+0000", 2024-09-11T03:50:37.080 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0056613940000000001 2024-09-11T03:50:37.080 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.080 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.080 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.080 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.258076+0000", 2024-09-11T03:50:37.080 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001846618 2024-09-11T03:50:37.081 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.081 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.081 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:37.081 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.258118+0000", 2024-09-11T03:50:37.081 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.1913999999999999e-05 2024-09-11T03:50:37.081 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.081 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.081 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.081 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.258157+0000", 2024-09-11T03:50:37.081 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8340000000000002e-05 2024-09-11T03:50:37.081 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.081 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.081 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.082 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.082 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.082 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6152 2.e 2:77dadb97:::smithi17557870-30:head [copy-from ver 268] snapc 10c=[] ondisk+write+known_if_redirected+supports_pool_eio e637)", 2024-09-11T03:50:37.082 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:35.707196+0000", 2024-09-11T03:50:37.082 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 181.26041680200001, 2024-09-11T03:50:37.082 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019734676, 2024-09-11T03:50:37.082 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:37.082 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:37.082 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:37.082 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:37.082 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:37.082 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:37.082 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6152 2024-09-11T03:50:37.083 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.083 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:37.083 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.083 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:37.083 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.707196+0000", 2024-09-11T03:50:37.083 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.083 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.083 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.083 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:37.083 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.707196+0000", 2024-09-11T03:50:37.083 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.083 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.083 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.083 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:37.084 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.707198+0000", 2024-09-11T03:50:37.084 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.125e-06 2024-09-11T03:50:37.084 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.084 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.084 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:37.084 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.707207+0000", 2024-09-11T03:50:37.084 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.0839999999999998e-06 2024-09-11T03:50:37.084 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.084 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.084 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:37.084 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.707225+0000", 2024-09-11T03:50:37.084 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8060999999999999e-05 2024-09-11T03:50:37.085 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.085 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.085 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:37.085 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.707233+0000", 2024-09-11T03:50:37.085 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2409999999999995e-06 2024-09-11T03:50:37.085 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.085 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.085 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.085 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.707297+0000", 2024-09-11T03:50:37.085 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3503999999999994e-05 2024-09-11T03:50:37.085 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.085 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.086 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:37.086 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.707366+0000", 2024-09-11T03:50:37.086 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9663e-05 2024-09-11T03:50:37.086 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.086 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.086 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:50:37.086 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.714472+0000", 2024-09-11T03:50:37.086 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.007106022 2024-09-11T03:50:37.086 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.086 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.086 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:37.086 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.718742+0000", 2024-09-11T03:50:37.086 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.004269424 2024-09-11T03:50:37.087 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.087 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.087 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.087 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.726773+0000", 2024-09-11T03:50:37.087 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0080315779999999993 2024-09-11T03:50:37.087 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.087 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.087 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.087 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.726818+0000", 2024-09-11T03:50:37.087 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4274000000000003e-05 2024-09-11T03:50:37.087 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.087 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.087 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:37.088 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.726881+0000", 2024-09-11T03:50:37.088 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2889e-05 2024-09-11T03:50:37.088 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.088 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.088 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.088 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.726930+0000", 2024-09-11T03:50:37.088 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9811000000000001e-05 2024-09-11T03:50:37.088 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.088 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.088 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.088 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.088 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.089 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6611 2.18 2:1984a3a9:::smithi17557870-23:head [copy-from ver 544] snapc 124=[124,123,121,120,11f,11e,11d,11a,119,114] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-09-11T03:50:37.089 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:12.994229+0000", 2024-09-11T03:50:37.089 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 143.97338304300001, 2024-09-11T03:50:37.089 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021949901000000001, 2024-09-11T03:50:37.089 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:37.089 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:37.089 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:37.089 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:37.089 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:37.089 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:37.089 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6611 2024-09-11T03:50:37.089 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.089 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:37.090 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.090 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:37.090 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.994229+0000", 2024-09-11T03:50:37.090 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.090 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.090 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.090 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:37.090 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.994229+0000", 2024-09-11T03:50:37.090 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.090 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.090 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.090 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:37.090 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.994231+0000", 2024-09-11T03:50:37.091 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9269999999999999e-06 2024-09-11T03:50:37.091 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.091 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.091 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:37.091 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.994240+0000", 2024-09-11T03:50:37.091 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.9309999999999997e-06 2024-09-11T03:50:37.091 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.091 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.091 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:37.091 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.994262+0000", 2024-09-11T03:50:37.091 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1341000000000001e-05 2024-09-11T03:50:37.091 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.091 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.092 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:37.092 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.994277+0000", 2024-09-11T03:50:37.092 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5407e-05 2024-09-11T03:50:37.092 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.092 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.092 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.092 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.994347+0000", 2024-09-11T03:50:37.092 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9555999999999996e-05 2024-09-11T03:50:37.092 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.092 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.092 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:50:37.092 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.994355+0000", 2024-09-11T03:50:37.092 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9889999999999992e-06 2024-09-11T03:50:37.092 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.093 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.093 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.996025+0000", 2024-09-11T03:50:37.093 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0016699320000000001 2024-09-11T03:50:37.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.093 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:37.093 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.996110+0000", 2024-09-11T03:50:37.093 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5711999999999995e-05 2024-09-11T03:50:37.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.094 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,7", 2024-09-11T03:50:37.094 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:13.005397+0000", 2024-09-11T03:50:37.094 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.009286908 2024-09-11T03:50:37.094 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.094 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.094 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:37.094 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:13.009436+0000", 2024-09-11T03:50:37.094 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0040392450000000003 2024-09-11T03:50:37.094 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.094 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.094 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.094 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:13.014053+0000", 2024-09-11T03:50:37.094 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00461678 2024-09-11T03:50:37.095 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.095 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.095 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.095 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:13.016093+0000", 2024-09-11T03:50:37.095 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0020393799999999999 2024-09-11T03:50:37.095 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.095 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.095 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:37.095 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:13.016133+0000", 2024-09-11T03:50:37.095 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0123999999999997e-05 2024-09-11T03:50:37.095 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.095 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.095 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.095 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:13.016179+0000", 2024-09-11T03:50:37.096 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6669000000000003e-05 2024-09-11T03:50:37.096 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.096 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.096 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.096 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.096 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.096 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7117 2.e 2:750de3b7:::smithi17557870-8:head [copy-from ver 984] snapc 13b=[13b,13a,139,138,137,136,135,133,12e,12d,12c,12a,127,126,120,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e713)", 2024-09-11T03:50:37.096 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:51.194048+0000", 2024-09-11T03:50:37.096 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 105.77356415, 2024-09-11T03:50:37.096 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.029233823999999999, 2024-09-11T03:50:37.096 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:37.096 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:37.097 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:37.097 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:37.097 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:37.097 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:37.097 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7117 2024-09-11T03:50:37.097 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.097 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:37.097 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.097 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:37.097 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194048+0000", 2024-09-11T03:50:37.098 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.098 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.098 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.098 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:37.098 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194048+0000", 2024-09-11T03:50:37.098 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.098 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.098 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.098 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:37.098 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194051+0000", 2024-09-11T03:50:37.098 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2390000000000001e-06 2024-09-11T03:50:37.098 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.098 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.098 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:37.099 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194060+0000", 2024-09-11T03:50:37.099 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9280000000000005e-06 2024-09-11T03:50:37.099 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.099 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.099 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:37.099 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194092+0000", 2024-09-11T03:50:37.099 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1603999999999998e-05 2024-09-11T03:50:37.099 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.099 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.099 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:37.100 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194102+0000", 2024-09-11T03:50:37.100 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0285999999999999e-05 2024-09-11T03:50:37.100 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.100 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.100 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.100 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194180+0000", 2024-09-11T03:50:37.100 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7231999999999995e-05 2024-09-11T03:50:37.100 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.100 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.100 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:37.101 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194274+0000", 2024-09-11T03:50:37.101 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4704000000000007e-05 2024-09-11T03:50:37.101 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.101 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.101 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:50:37.101 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.211121+0000", 2024-09-11T03:50:37.101 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016847141 2024-09-11T03:50:37.101 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.101 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.101 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:37.101 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.214395+0000", 2024-09-11T03:50:37.101 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032732429999999999 2024-09-11T03:50:37.102 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.102 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.102 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.102 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223020+0000", 2024-09-11T03:50:37.102 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0086250600000000004 2024-09-11T03:50:37.102 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.102 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.102 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.102 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223211+0000", 2024-09-11T03:50:37.102 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000191349 2024-09-11T03:50:37.102 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.102 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.102 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:37.103 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223250+0000", 2024-09-11T03:50:37.103 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9366999999999998e-05 2024-09-11T03:50:37.103 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.103 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.103 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.103 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223282+0000", 2024-09-11T03:50:37.103 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1671000000000002e-05 2024-09-11T03:50:37.103 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.103 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.103 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.103 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.103 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.103 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7118 2.e 2:750de3b7:::smithi17557870-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e713)", 2024-09-11T03:50:37.104 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:51.194365+0000", 2024-09-11T03:50:37.104 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 105.773247746, 2024-09-11T03:50:37.104 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019680618, 2024-09-11T03:50:37.104 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:37.104 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:37.104 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:50:37.104 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:37.104 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:37.104 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:37.104 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7118 2024-09-11T03:50:37.104 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.104 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:37.104 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194365+0000", 2024-09-11T03:50:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194365+0000", 2024-09-11T03:50:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194365+0000", 2024-09-11T03:50:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3799999999999997e-07 2024-09-11T03:50:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194369+0000", 2024-09-11T03:50:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0430000000000001e-06 2024-09-11T03:50:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194379+0000", 2024-09-11T03:50:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.268e-06 2024-09-11T03:50:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194386+0000", 2024-09-11T03:50:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3420000000000002e-06 2024-09-11T03:50:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194423+0000", 2024-09-11T03:50:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7125000000000001e-05 2024-09-11T03:50:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194486+0000", 2024-09-11T03:50:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2824999999999996e-05 2024-09-11T03:50:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.213801+0000", 2024-09-11T03:50:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019315300000000001 2024-09-11T03:50:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.213904+0000", 2024-09-11T03:50:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000102876 2024-09-11T03:50:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.214045+0000", 2024-09-11T03:50:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00014120100000000001 2024-09-11T03:50:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7209 2.e 2:77dadb97:::smithi17557870-30:head [copy-from ver 962] snapc 140=[140,13f,13d,13c,139,138,137,136,135,133,12e,12d,12a,127,126,120,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-09-11T03:50:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:49:00.012732+0000", 2024-09-11T03:50:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 96.954880883000001, 2024-09-11T03:50:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019941586000000001, 2024-09-11T03:50:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7209 2024-09-11T03:50:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.012732+0000", 2024-09-11T03:50:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.012732+0000", 2024-09-11T03:50:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.012734+0000", 2024-09-11T03:50:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2630000000000002e-06 2024-09-11T03:50:37.112 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.112 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.112 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:37.112 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.012741+0000", 2024-09-11T03:50:37.112 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7209999999999997e-06 2024-09-11T03:50:37.112 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.112 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.112 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:37.112 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.012756+0000", 2024-09-11T03:50:37.112 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5099000000000001e-05 2024-09-11T03:50:37.112 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.112 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.112 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:37.113 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.012764+0000", 2024-09-11T03:50:37.113 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4600000000000003e-06 2024-09-11T03:50:37.113 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.113 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.113 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.113 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.012826+0000", 2024-09-11T03:50:37.113 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1569999999999995e-05 2024-09-11T03:50:37.113 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.113 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.113 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:50:37.113 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.012833+0000", 2024-09-11T03:50:37.114 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1840000000000002e-06 2024-09-11T03:50:37.114 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.114 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.114 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.114 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.015099+0000", 2024-09-11T03:50:37.114 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022662469999999999 2024-09-11T03:50:37.114 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.114 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.114 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:37.114 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.015212+0000", 2024-09-11T03:50:37.114 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000113232 2024-09-11T03:50:37.114 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.114 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.115 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:50:37.115 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.024487+0000", 2024-09-11T03:50:37.115 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0092743140000000005 2024-09-11T03:50:37.115 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.115 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.115 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:37.115 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.028401+0000", 2024-09-11T03:50:37.115 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039138669999999997 2024-09-11T03:50:37.115 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.115 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.115 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.115 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.032461+0000", 2024-09-11T03:50:37.116 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0040599980000000004 2024-09-11T03:50:37.116 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.116 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.116 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:37.116 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.032598+0000", 2024-09-11T03:50:37.116 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000137904 2024-09-11T03:50:37.116 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.116 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.116 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:37.116 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.032639+0000", 2024-09-11T03:50:37.116 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.1026999999999999e-05 2024-09-11T03:50:37.116 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.117 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.117 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.117 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.032673+0000", 2024-09-11T03:50:37.117 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3699999999999999e-05 2024-09-11T03:50:37.117 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.117 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.117 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.117 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.117 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.117 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "MOSDPGPush(2.3e 813/812 [PushOp(2:7efbbea8:::smithi17557870-24:127, version: 762'1020, data_included: [393216~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7efbbea8:::smithi17557870-24:127@762'1020, size: 3630477, copy_subset: [(0, 3630477)], clone_subset: {}, snapset: 16f=[]:{127: [127, 120], 13a: [139, 135], 13d: [13c], 15c: [15b, 15a, 157, 156], 15d: [15d], 15f: [15f, 15e], 16f: [16f, 16e, 16d, 16c, 16b, 169]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1441792, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T03:50:37.117 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:50:31.090513+0000", 2024-09-11T03:50:37.118 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 5.8770997019999998, 2024-09-11T03:50:37.118 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002463135, 2024-09-11T03:50:37.118 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:37.118 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:37.118 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:50:37.118 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:37.118 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.118 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:37.118 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:31.090513+0000", 2024-09-11T03:50:37.118 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.118 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.118 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.118 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:37.119 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:31.090513+0000", 2024-09-11T03:50:37.119 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:37.119 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.119 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.119 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:37.119 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:31.090513+0000", 2024-09-11T03:50:37.119 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9200000000000002e-07 2024-09-11T03:50:37.119 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.119 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.119 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:37.119 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:31.092483+0000", 2024-09-11T03:50:37.119 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0019694719999999999 2024-09-11T03:50:37.119 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.120 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.120 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:37.120 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:31.092505+0000", 2024-09-11T03:50:37.120 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1973000000000001e-05 2024-09-11T03:50:37.120 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.120 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.120 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:37.120 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:31.092516+0000", 2024-09-11T03:50:37.120 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1732999999999999e-05 2024-09-11T03:50:37.120 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.120 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.121 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:37.121 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:31.092582+0000", 2024-09-11T03:50:37.121 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5451999999999996e-05 2024-09-11T03:50:37.121 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.121 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.121 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:37.121 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:31.092590+0000", 2024-09-11T03:50:37.122 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.568e-06 2024-09-11T03:50:37.122 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:37.122 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:37.122 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:37.122 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:31.092976+0000", 2024-09-11T03:50:37.122 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000385545 2024-09-11T03:50:37.122 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.122 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.122 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.122 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:37.122 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:37.122 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:50:37.128 INFO:tasks.rados.rados.2.smithi175.stdout:3377: done (3 left) 2024-09-11T03:50:37.128 INFO:tasks.rados.rados.2.smithi175.stdout:3378: done (2 left) 2024-09-11T03:50:37.128 INFO:tasks.rados.rados.2.smithi175.stdout:3379: done (1 left) 2024-09-11T03:50:37.128 INFO:tasks.rados.rados.2.smithi175.stdout:3380: done (0 left) 2024-09-11T03:50:37.128 INFO:tasks.rados.rados.2.smithi175.stdout:3381: write oid 1 current snap is 372 2024-09-11T03:50:37.128 INFO:tasks.rados.rados.2.smithi175.stdout:3381: seq_num 1150 ranges {528050=505653,1612057=524731,2614324=567403} 2024-09-11T03:50:37.128 INFO:tasks.rados.rados.2.smithi175.stdout:3381: writing smithi17557870-1 from 528050 to 1033703 tid 1 2024-09-11T03:50:37.128 INFO:tasks.rados.rados.2.smithi175.stdout:3381: writing smithi17557870-1 from 1612057 to 2136788 tid 2 2024-09-11T03:50:37.128 INFO:tasks.rados.rados.2.smithi175.stdout:3381: writing smithi17557870-1 from 2614324 to 3181727 tid 3 2024-09-11T03:50:37.128 INFO:tasks.rados.rados.2.smithi175.stdout:3382: write oid 15 current snap is 372 2024-09-11T03:50:37.129 INFO:tasks.rados.rados.2.smithi175.stdout:3381: finishing write tid 1 to smithi17557870-1 2024-09-11T03:50:37.129 INFO:tasks.rados.rados.2.smithi175.stdout:3381: finishing write tid 2 to smithi17557870-1 2024-09-11T03:50:37.129 INFO:tasks.rados.rados.2.smithi175.stdout:3382: seq_num 1151 ranges {544857=580902,1670539=668389,3041747=303110} 2024-09-11T03:50:37.129 INFO:tasks.rados.rados.2.smithi175.stdout:3382: writing smithi17557870-15 from 544857 to 1125759 tid 1 2024-09-11T03:50:37.129 INFO:tasks.rados.rados.2.smithi175.stdout:3382: writing smithi17557870-15 from 1670539 to 2338928 tid 2 2024-09-11T03:50:37.129 INFO:tasks.rados.rados.2.smithi175.stdout:3382: writing smithi17557870-15 from 3041747 to 3344857 tid 3 2024-09-11T03:50:37.129 INFO:tasks.rados.rados.2.smithi175.stdout:3383: rollback oid 49 current snap is 372 2024-09-11T03:50:37.129 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 49 to 349 2024-09-11T03:50:37.129 INFO:tasks.rados.rados.2.smithi175.stdout:3381: finishing write tid 3 to smithi17557870-1 2024-09-11T03:50:37.129 INFO:tasks.rados.rados.2.smithi175.stdout:3381: finishing write tid 4 to smithi17557870-1 2024-09-11T03:50:37.129 INFO:tasks.rados.rados.2.smithi175.stdout:3381: finishing write tid 5 to smithi17557870-1 2024-09-11T03:50:37.129 INFO:tasks.rados.rados.2.smithi175.stdout:3381: oid 1 updating version 0 to 1182 2024-09-11T03:50:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:3381: oid 1 updating version 1182 to 1183 2024-09-11T03:50:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:3381: oid 1 updating version 1183 to 1184 2024-09-11T03:50:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:3381: oid 1 version 1184 is already newer than 1181 2024-09-11T03:50:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 1184 (ObjNum 1150 snap 372 seq_num 1150) dirty exists 2024-09-11T03:50:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:3381: left oid 1 (ObjNum 1150 snap 372 seq_num 1150) 2024-09-11T03:50:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:3382: finishing write tid 1 to smithi17557870-15 2024-09-11T03:50:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:3381: done (2 left) 2024-09-11T03:50:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:3384: rollback oid 29 current snap is 372 2024-09-11T03:50:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 29 to 331 2024-09-11T03:50:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:3382: finishing write tid 2 to smithi17557870-15 2024-09-11T03:50:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:3385: read oid 39 snap 370 2024-09-11T03:50:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:3385: expect deleted 2024-09-11T03:50:37.130 INFO:tasks.rados.rados.2.smithi175.stdout:3386: write (excl) oid 14 current snap is 372 2024-09-11T03:50:37.131 INFO:tasks.rados.rados.2.smithi175.stdout:3386: seq_num 1152 ranges {561664=656151,1729021=528400,2669170=692494} 2024-09-11T03:50:37.131 INFO:tasks.rados.rados.2.smithi175.stdout:3386: writing smithi17557870-14 from 561664 to 1217815 tid 1 2024-09-11T03:50:37.131 INFO:tasks.rados.rados.2.smithi175.stdout:3386: writing smithi17557870-14 from 1729021 to 2257421 tid 2 2024-09-11T03:50:37.131 INFO:tasks.rados.rados.2.smithi175.stdout:3386: writing smithi17557870-14 from 2669170 to 3361664 tid 3 2024-09-11T03:50:37.131 INFO:tasks.rados.rados.2.smithi175.stdout:3387: read oid 5 snap -1 2024-09-11T03:50:37.131 INFO:tasks.rados.rados.2.smithi175.stdout:3387: expect deleted 2024-09-11T03:50:37.131 INFO:tasks.rados.rados.2.smithi175.stdout:3388: write oid 50 current snap is 372 2024-09-11T03:50:37.131 INFO:tasks.rados.rados.2.smithi175.stdout:3382: finishing write tid 3 to smithi17557870-15 2024-09-11T03:50:37.131 INFO:tasks.rados.rados.2.smithi175.stdout:3388: seq_num 1153 ranges {578471=731400,1787503=672058,3096593=281878} 2024-09-11T03:50:37.131 INFO:tasks.rados.rados.2.smithi175.stdout:3388: writing smithi17557870-50 from 578471 to 1309871 tid 1 2024-09-11T03:50:37.132 INFO:tasks.rados.rados.2.smithi175.stdout:3388: writing smithi17557870-50 from 1787503 to 2459561 tid 2 2024-09-11T03:50:37.132 INFO:tasks.rados.rados.2.smithi175.stdout:3388: writing smithi17557870-50 from 3096593 to 3378471 tid 3 2024-09-11T03:50:37.132 INFO:tasks.rados.rados.2.smithi175.stdout:3389: snap_create 2024-09-11T03:50:37.132 INFO:tasks.rados.rados.2.smithi175.stdout:3384: finishing rollback tid 0 to smithi17557870-29 2024-09-11T03:50:37.132 INFO:tasks.rados.rados.2.smithi175.stdout:3382: finishing write tid 4 to smithi17557870-15 2024-09-11T03:50:37.132 INFO:tasks.rados.rados.2.smithi175.stdout:3382: finishing write tid 5 to smithi17557870-15 2024-09-11T03:50:37.132 INFO:tasks.rados.rados.2.smithi175.stdout:3382: oid 15 updating version 0 to 1057 2024-09-11T03:50:37.132 INFO:tasks.rados.rados.2.smithi175.stdout:3382: oid 15 version 1057 is already newer than 1056 2024-09-11T03:50:37.132 INFO:tasks.rados.rados.2.smithi175.stdout:3382: oid 15 version 1057 is already newer than 1054 2024-09-11T03:50:37.132 INFO:tasks.rados.rados.2.smithi175.stdout:3382: oid 15 version 1057 is already newer than 1055 2024-09-11T03:50:37.132 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 1057 (ObjNum 1151 snap 372 seq_num 1151) dirty exists 2024-09-11T03:50:37.132 INFO:tasks.rados.rados.2.smithi175.stdout:3382: left oid 15 (ObjNum 1151 snap 372 seq_num 1151) 2024-09-11T03:50:37.133 INFO:tasks.rados.rados.2.smithi175.stdout:3384: finishing rollback tid 1 to smithi17557870-29 2024-09-11T03:50:37.133 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 1187 (ObjNum 273 snap 0 seq_num 1746082976) dirty dne 2024-09-11T03:50:37.133 INFO:tasks.rados.rados.2.smithi175.stdout:3383: finishing rollback tid 1 to smithi17557870-49 2024-09-11T03:50:37.133 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 1002 (ObjNum 33 snap 0 seq_num 1713067256) dirty dne 2024-09-11T03:50:37.133 INFO:tasks.rados.rados.2.smithi175.stdout:3386: finishing write tid 1 to smithi17557870-14 2024-09-11T03:50:37.133 INFO:tasks.rados.rados.2.smithi175.stdout:3386: finishing write tid 2 to smithi17557870-14 2024-09-11T03:50:37.133 INFO:tasks.rados.rados.2.smithi175.stdout:3386: finishing write tid 3 to smithi17557870-14 2024-09-11T03:50:37.133 INFO:tasks.rados.rados.2.smithi175.stdout:3386: finishing write tid 4 to smithi17557870-14 2024-09-11T03:50:37.133 INFO:tasks.rados.rados.2.smithi175.stdout:3386: finishing write tid 5 to smithi17557870-14 2024-09-11T03:50:37.133 INFO:tasks.rados.rados.2.smithi175.stdout:3386: oid 14 updating version 0 to 1064 2024-09-11T03:50:37.133 INFO:tasks.rados.rados.2.smithi175.stdout:3386: oid 14 updating version 1064 to 1065 2024-09-11T03:50:37.133 INFO:tasks.rados.rados.2.smithi175.stdout:3386: oid 14 version 1065 is already newer than 1062 2024-09-11T03:50:37.133 INFO:tasks.rados.rados.2.smithi175.stdout:3386: oid 14 version 1065 is already newer than 1063 2024-09-11T03:50:37.134 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 1065 (ObjNum 1152 snap 372 seq_num 1152) dirty exists 2024-09-11T03:50:37.134 INFO:tasks.rados.rados.2.smithi175.stdout:3386: left oid 14 (ObjNum 1152 snap 372 seq_num 1152) 2024-09-11T03:50:37.134 INFO:tasks.rados.rados.2.smithi175.stdout:3388: finishing write tid 1 to smithi17557870-50 2024-09-11T03:50:37.134 INFO:tasks.rados.rados.2.smithi175.stdout:3388: finishing write tid 2 to smithi17557870-50 2024-09-11T03:50:37.134 INFO:tasks.rados.rados.2.smithi175.stdout:3388: finishing write tid 3 to smithi17557870-50 2024-09-11T03:50:37.134 INFO:tasks.rados.rados.2.smithi175.stdout:3388: finishing write tid 4 to smithi17557870-50 2024-09-11T03:50:37.134 INFO:tasks.rados.rados.2.smithi175.stdout:3388: finishing write tid 5 to smithi17557870-50 2024-09-11T03:50:37.134 INFO:tasks.rados.rados.2.smithi175.stdout:3388: oid 50 updating version 0 to 1305 2024-09-11T03:50:37.134 INFO:tasks.rados.rados.2.smithi175.stdout:3388: oid 50 version 1305 is already newer than 1304 2024-09-11T03:50:37.134 INFO:tasks.rados.rados.2.smithi175.stdout:3388: oid 50 version 1305 is already newer than 1302 2024-09-11T03:50:37.134 INFO:tasks.rados.rados.2.smithi175.stdout:3388: oid 50 version 1305 is already newer than 1303 2024-09-11T03:50:37.134 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 1305 (ObjNum 1153 snap 372 seq_num 1153) dirty exists 2024-09-11T03:50:37.134 INFO:tasks.rados.rados.2.smithi175.stdout:3388: left oid 50 (ObjNum 1153 snap 372 seq_num 1153) 2024-09-11T03:50:37.143 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:37.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:37.144 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:37.245 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:37.266 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:37.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:37.267 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:37.368 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:37.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: 2.3d scrub starts 2024-09-11T03:50:37.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: 2.3d scrub ok 2024-09-11T03:50:37.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: 2.3c scrub starts 2024-09-11T03:50:37.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: 2.3c scrub ok 2024-09-11T03:50:37.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: 2.1e scrub starts 2024-09-11T03:50:37.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: 2.1e scrub ok 2024-09-11T03:50:37.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: 2.3e scrub starts 2024-09-11T03:50:37.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: 2.3e scrub ok 2024-09-11T03:50:37.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: osdmap e818: 12 total, 12 up, 12 in 2024-09-11T03:50:37.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: 2.39 scrub starts 2024-09-11T03:50:37.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: 2.39 scrub ok 2024-09-11T03:50:37.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3526717639' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "66"} : dispatch 2024-09-11T03:50:37.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: 2.19 scrub starts 2024-09-11T03:50:37.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: 2.19 scrub ok 2024-09-11T03:50:37.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1599208362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:37.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:50:37.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: 2.0 scrub ok 2024-09-11T03:50:37.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:37.381 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3526717639' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "66"}]': finished 2024-09-11T03:50:37.381 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:37 smithi084 ceph-mon[27254]: osdmap e819: 12 total, 12 up, 12 in 2024-09-11T03:50:37.389 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:37.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:37.390 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:37.491 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:37.512 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:37.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:37.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:37.615 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:37.625 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: 2.3d scrub starts 2024-09-11T03:50:37.625 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: 2.3d scrub ok 2024-09-11T03:50:37.625 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: 2.3c scrub starts 2024-09-11T03:50:37.625 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: 2.3c scrub ok 2024-09-11T03:50:37.625 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: 2.1e scrub starts 2024-09-11T03:50:37.625 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: 2.1e scrub ok 2024-09-11T03:50:37.625 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: 2.3e scrub starts 2024-09-11T03:50:37.625 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: 2.3e scrub ok 2024-09-11T03:50:37.625 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: osdmap e818: 12 total, 12 up, 12 in 2024-09-11T03:50:37.625 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: 2.39 scrub starts 2024-09-11T03:50:37.626 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: 2.39 scrub ok 2024-09-11T03:50:37.626 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3526717639' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "66"} : dispatch 2024-09-11T03:50:37.626 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: 2.19 scrub starts 2024-09-11T03:50:37.626 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: 2.19 scrub ok 2024-09-11T03:50:37.626 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1599208362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:37.626 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:50:37.626 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: 2.0 scrub ok 2024-09-11T03:50:37.626 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:37.626 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3526717639' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "66"}]': finished 2024-09-11T03:50:37.626 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:37 smithi175 ceph-mon[30894]: osdmap e819: 12 total, 12 up, 12 in 2024-09-11T03:50:37.636 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:37.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:37.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: 2.3d scrub starts 2024-09-11T03:50:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: 2.3d scrub ok 2024-09-11T03:50:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: 2.3c scrub starts 2024-09-11T03:50:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: 2.3c scrub ok 2024-09-11T03:50:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: 2.1e scrub starts 2024-09-11T03:50:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: 2.1e scrub ok 2024-09-11T03:50:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: 2.3e scrub starts 2024-09-11T03:50:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: 2.3e scrub ok 2024-09-11T03:50:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: osdmap e818: 12 total, 12 up, 12 in 2024-09-11T03:50:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: 2.39 scrub starts 2024-09-11T03:50:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: 2.39 scrub ok 2024-09-11T03:50:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3526717639' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "66"} : dispatch 2024-09-11T03:50:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: 2.19 scrub starts 2024-09-11T03:50:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: 2.19 scrub ok 2024-09-11T03:50:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1599208362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:50:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: 2.0 scrub ok 2024-09-11T03:50:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3526717639' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "66"}]': finished 2024-09-11T03:50:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:37 smithi121 ceph-mon[30729]: osdmap e819: 12 total, 12 up, 12 in 2024-09-11T03:50:37.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:37.761 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:37.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:37.862 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:37.885 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:37.886 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:37.915 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:50:37.987 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:38.009 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:38.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:38.010 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:38.038 INFO:tasks.rados.rados.2.smithi175.stdout:3382: done (7 left) 2024-09-11T03:50:38.039 INFO:tasks.rados.rados.2.smithi175.stdout:3383: done (6 left) 2024-09-11T03:50:38.039 INFO:tasks.rados.rados.2.smithi175.stdout:3384: done (5 left) 2024-09-11T03:50:38.039 INFO:tasks.rados.rados.2.smithi175.stdout:3385: done (4 left) 2024-09-11T03:50:38.039 INFO:tasks.rados.rados.2.smithi175.stdout:3386: done (3 left) 2024-09-11T03:50:38.039 INFO:tasks.rados.rados.2.smithi175.stdout:3387: done (2 left) 2024-09-11T03:50:38.039 INFO:tasks.rados.rados.2.smithi175.stdout:3388: done (1 left) 2024-09-11T03:50:38.039 INFO:tasks.rados.rados.2.smithi175.stdout:3389: done (0 left) 2024-09-11T03:50:38.039 INFO:tasks.rados.rados.2.smithi175.stdout:3390: read oid 25 snap -1 2024-09-11T03:50:38.040 INFO:tasks.rados.rados.2.smithi175.stdout:3390: expect (ObjNum 1148 snap 371 seq_num 1148) 2024-09-11T03:50:38.040 INFO:tasks.rados.rados.2.smithi175.stdout:3391: rollback oid 30 current snap is 373 2024-09-11T03:50:38.040 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 30 to 345 2024-09-11T03:50:38.040 INFO:tasks.rados.rados.2.smithi175.stdout:3392: snap_create 2024-09-11T03:50:38.045 INFO:tasks.rados.rados.2.smithi175.stdout:3391: finishing rollback tid 0 to smithi17557870-30 2024-09-11T03:50:38.045 INFO:tasks.rados.rados.2.smithi175.stdout:3391: finishing rollback tid 1 to smithi17557870-30 2024-09-11T03:50:38.045 INFO:tasks.rados.rados.2.smithi175.stdout:3391: finishing rollback tid 2 to smithi17557870-30 2024-09-11T03:50:38.045 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 1077 (ObjNum 895 snap 284 seq_num 895) dirty exists 2024-09-11T03:50:38.052 INFO:tasks.rados.rados.2.smithi175.stdout:3390: expect (ObjNum 1148 snap 371 seq_num 1148) 2024-09-11T03:50:38.072 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-11T03:50:38.111 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:38.134 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:38.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:38.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:38.262 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:38.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:38.353 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:50:38.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:38.388 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:38.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:38.474 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: 2.3f scrub starts 2024-09-11T03:50:38.474 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: 2.3f scrub ok 2024-09-11T03:50:38.475 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:50:38.475 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: 2.1 scrub ok 2024-09-11T03:50:38.475 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: pgmap v1383: 97 pgs: 1 active+clean+scrubbing, 3 peering, 93 active+clean; 561 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:38.475 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: 1.16 scrub starts 2024-09-11T03:50:38.475 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: 1.16 scrub ok 2024-09-11T03:50:38.475 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 142/834 objects degraded (17.026%), 2 pgs degraded) 2024-09-11T03:50:38.475 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: 2.3a scrub starts 2024-09-11T03:50:38.475 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: 2.3a scrub ok 2024-09-11T03:50:38.475 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:50:38.475 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: 2.d scrub ok 2024-09-11T03:50:38.475 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: 2.2c scrub starts 2024-09-11T03:50:38.475 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: 2.2c scrub ok 2024-09-11T03:50:38.475 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: 2.1a scrub starts 2024-09-11T03:50:38.476 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: 2.1a scrub ok 2024-09-11T03:50:38.476 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: 2.1c scrub starts 2024-09-11T03:50:38.476 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: 2.1c scrub ok 2024-09-11T03:50:38.476 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:38 smithi175 ceph-mon[30894]: osdmap e820: 12 total, 12 up, 12 in 2024-09-11T03:50:38.488 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:38.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:38.515 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:38.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:38.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:38.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: 2.3f scrub starts 2024-09-11T03:50:38.573 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: 2.3f scrub ok 2024-09-11T03:50:38.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:50:38.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: 2.1 scrub ok 2024-09-11T03:50:38.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: pgmap v1383: 97 pgs: 1 active+clean+scrubbing, 3 peering, 93 active+clean; 561 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:38.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: 1.16 scrub starts 2024-09-11T03:50:38.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: 1.16 scrub ok 2024-09-11T03:50:38.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 142/834 objects degraded (17.026%), 2 pgs degraded) 2024-09-11T03:50:38.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: 2.3a scrub starts 2024-09-11T03:50:38.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: 2.3a scrub ok 2024-09-11T03:50:38.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:50:38.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: 2.d scrub ok 2024-09-11T03:50:38.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: 2.2c scrub starts 2024-09-11T03:50:38.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: 2.2c scrub ok 2024-09-11T03:50:38.574 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: 2.1a scrub starts 2024-09-11T03:50:38.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: 2.1a scrub ok 2024-09-11T03:50:38.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: 2.1c scrub starts 2024-09-11T03:50:38.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: 2.1c scrub ok 2024-09-11T03:50:38.575 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:38 smithi121 ceph-mon[30729]: osdmap e820: 12 total, 12 up, 12 in 2024-09-11T03:50:38.608 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: 2.3f scrub starts 2024-09-11T03:50:38.609 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: 2.3f scrub ok 2024-09-11T03:50:38.609 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:50:38.609 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: 2.1 scrub ok 2024-09-11T03:50:38.609 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: pgmap v1383: 97 pgs: 1 active+clean+scrubbing, 3 peering, 93 active+clean; 561 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:38.609 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: 1.16 scrub starts 2024-09-11T03:50:38.609 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: 1.16 scrub ok 2024-09-11T03:50:38.609 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 142/834 objects degraded (17.026%), 2 pgs degraded) 2024-09-11T03:50:38.609 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: 2.3a scrub starts 2024-09-11T03:50:38.609 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: 2.3a scrub ok 2024-09-11T03:50:38.609 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:50:38.609 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: 2.d scrub ok 2024-09-11T03:50:38.609 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: 2.2c scrub starts 2024-09-11T03:50:38.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: 2.2c scrub ok 2024-09-11T03:50:38.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: 2.1a scrub starts 2024-09-11T03:50:38.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: 2.1a scrub ok 2024-09-11T03:50:38.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: 2.1c scrub starts 2024-09-11T03:50:38.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: 2.1c scrub ok 2024-09-11T03:50:38.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:38 smithi084 ceph-mon[27254]: osdmap e820: 12 total, 12 up, 12 in 2024-09-11T03:50:38.617 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:38.638 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:38.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:38.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:38.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:38.762 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:38.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:38.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:38.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:38.887 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:50:38.888 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:50:38.888 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:50:38.888 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:50:38.889 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:38.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:38.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:38.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:39.014 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:39.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:39.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:39.040 INFO:tasks.rados.rados.2.smithi175.stdout:3390: done (2 left) 2024-09-11T03:50:39.040 INFO:tasks.rados.rados.2.smithi175.stdout:3391: done (1 left) 2024-09-11T03:50:39.040 INFO:tasks.rados.rados.2.smithi175.stdout:3392: done (0 left) 2024-09-11T03:50:39.040 INFO:tasks.rados.rados.2.smithi175.stdout:3393: rollback oid 37 current snap is 374 2024-09-11T03:50:39.040 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 37 to 359 2024-09-11T03:50:39.040 INFO:tasks.rados.rados.2.smithi175.stdout:3394: snap_create 2024-09-11T03:50:39.044 INFO:tasks.rados.rados.2.smithi175.stdout:3393: finishing rollback tid 0 to smithi17557870-37 2024-09-11T03:50:39.044 INFO:tasks.rados.rados.2.smithi175.stdout:3393: finishing rollback tid 1 to smithi17557870-37 2024-09-11T03:50:39.044 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 915 (ObjNum 257 snap 0 seq_num 1743420432) dirty dne 2024-09-11T03:50:39.115 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:39.137 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:39.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:39.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:39.239 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:39.260 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:39.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:39.362 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:39.383 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:39.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:39.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:39.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:39.507 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:39.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:39.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:39.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:39.629 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:39.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:39.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:39.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:39 smithi084 ceph-mon[27254]: 2.6 scrub starts 2024-09-11T03:50:39.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:39 smithi084 ceph-mon[27254]: 2.6 scrub ok 2024-09-11T03:50:39.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:39 smithi084 ceph-mon[27254]: 2.20 scrub starts 2024-09-11T03:50:39.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:39 smithi084 ceph-mon[27254]: 2.20 scrub ok 2024-09-11T03:50:39.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:39 smithi084 ceph-mon[27254]: 2.1f scrub starts 2024-09-11T03:50:39.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:39 smithi084 ceph-mon[27254]: 2.1f scrub ok 2024-09-11T03:50:39.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:39 smithi084 ceph-mon[27254]: 2.22 scrub starts 2024-09-11T03:50:39.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:39 smithi084 ceph-mon[27254]: 2.22 scrub ok 2024-09-11T03:50:39.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:39 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:50:39.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:39 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:50:39.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:39 smithi084 ceph-mon[27254]: 2.1d scrub starts 2024-09-11T03:50:39.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:39 smithi084 ceph-mon[27254]: 2.1d scrub ok 2024-09-11T03:50:39.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:39 smithi084 ceph-mon[27254]: osdmap e821: 12 total, 12 up, 12 in 2024-09-11T03:50:39.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:39 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/241027678' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:39.658 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:39 smithi175 ceph-mon[30894]: 2.6 scrub starts 2024-09-11T03:50:39.658 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:39 smithi175 ceph-mon[30894]: 2.6 scrub ok 2024-09-11T03:50:39.658 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:39 smithi175 ceph-mon[30894]: 2.20 scrub starts 2024-09-11T03:50:39.659 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:39 smithi175 ceph-mon[30894]: 2.20 scrub ok 2024-09-11T03:50:39.659 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:39 smithi175 ceph-mon[30894]: 2.1f scrub starts 2024-09-11T03:50:39.659 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:39 smithi175 ceph-mon[30894]: 2.1f scrub ok 2024-09-11T03:50:39.659 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:39 smithi175 ceph-mon[30894]: 2.22 scrub starts 2024-09-11T03:50:39.659 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:39 smithi175 ceph-mon[30894]: 2.22 scrub ok 2024-09-11T03:50:39.659 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:39 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:50:39.659 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:39 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:50:39.659 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:39 smithi175 ceph-mon[30894]: 2.1d scrub starts 2024-09-11T03:50:39.659 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:39 smithi175 ceph-mon[30894]: 2.1d scrub ok 2024-09-11T03:50:39.659 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:39 smithi175 ceph-mon[30894]: osdmap e821: 12 total, 12 up, 12 in 2024-09-11T03:50:39.659 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:39 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/241027678' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:39 smithi121 ceph-mon[30729]: 2.6 scrub starts 2024-09-11T03:50:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:39 smithi121 ceph-mon[30729]: 2.6 scrub ok 2024-09-11T03:50:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:39 smithi121 ceph-mon[30729]: 2.20 scrub starts 2024-09-11T03:50:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:39 smithi121 ceph-mon[30729]: 2.20 scrub ok 2024-09-11T03:50:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:39 smithi121 ceph-mon[30729]: 2.1f scrub starts 2024-09-11T03:50:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:39 smithi121 ceph-mon[30729]: 2.1f scrub ok 2024-09-11T03:50:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:39 smithi121 ceph-mon[30729]: 2.22 scrub starts 2024-09-11T03:50:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:39 smithi121 ceph-mon[30729]: 2.22 scrub ok 2024-09-11T03:50:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:39 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:50:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:39 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:50:39.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:39 smithi121 ceph-mon[30729]: 2.1d scrub starts 2024-09-11T03:50:39.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:39 smithi121 ceph-mon[30729]: 2.1d scrub ok 2024-09-11T03:50:39.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:39 smithi121 ceph-mon[30729]: osdmap e821: 12 total, 12 up, 12 in 2024-09-11T03:50:39.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:39 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/241027678' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:39.687 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-11T03:50:39.731 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:39.752 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:39.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:39.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:39.854 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:39.876 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:39.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:39.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:39.929 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:50:39.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:39.998 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:39.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:39.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:40.038 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:50:40.041 INFO:tasks.rados.rados.2.smithi175.stdout:3393: done (1 left) 2024-09-11T03:50:40.042 INFO:tasks.rados.rados.2.smithi175.stdout:3394: done (0 left) 2024-09-11T03:50:40.042 INFO:tasks.rados.rados.2.smithi175.stdout:3395: read oid 37 snap -1 2024-09-11T03:50:40.042 INFO:tasks.rados.rados.2.smithi175.stdout:3395: expect deleted 2024-09-11T03:50:40.042 INFO:tasks.rados.rados.2.smithi175.stdout:3396: snap_remove snap 326 2024-09-11T03:50:40.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:40.121 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:40.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:40.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:40.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:40.244 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:40.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:40.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:40.337 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:40 smithi175 ceph-mon[30894]: 1.e scrub starts 2024-09-11T03:50:40.337 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:40 smithi175 ceph-mon[30894]: 1.e scrub ok 2024-09-11T03:50:40.337 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:40 smithi175 ceph-mon[30894]: pgmap v1386: 97 pgs: 97 active+clean; 581 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:40.337 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:40 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:50:40.338 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:40 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:50:40.338 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:40 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/241027678' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:40.338 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:40 smithi175 ceph-mon[30894]: osdmap e822: 12 total, 12 up, 12 in 2024-09-11T03:50:40.346 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:40.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:40 smithi084 ceph-mon[27254]: 1.e scrub starts 2024-09-11T03:50:40.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:40 smithi084 ceph-mon[27254]: 1.e scrub ok 2024-09-11T03:50:40.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:40 smithi084 ceph-mon[27254]: pgmap v1386: 97 pgs: 97 active+clean; 581 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:40.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:40 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:50:40.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:40 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:50:40.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:40 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/241027678' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:40.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:40 smithi084 ceph-mon[27254]: osdmap e822: 12 total, 12 up, 12 in 2024-09-11T03:50:40.368 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:40.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:40.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:40.369 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:50:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:40 smithi121 ceph-mon[30729]: 1.e scrub starts 2024-09-11T03:50:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:40 smithi121 ceph-mon[30729]: 1.e scrub ok 2024-09-11T03:50:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:40 smithi121 ceph-mon[30729]: pgmap v1386: 97 pgs: 97 active+clean; 581 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:40 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:50:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:40 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:50:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:40 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/241027678' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:40.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:40 smithi121 ceph-mon[30729]: osdmap e822: 12 total, 12 up, 12 in 2024-09-11T03:50:40.438 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:50:40.438 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:50:40.438 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:50:40.439 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:50:40.439 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:50:40.470 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:40.491 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:40.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:40.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:40.593 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:40.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:40.615 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:40.616 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:40.716 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:40.738 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:40.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:40.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:40.742 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:40.840 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:40.862 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:40.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:40.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:40.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:40.988 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:40.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:40.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:41.062 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-11T03:50:41.067 INFO:tasks.rados.rados.2.smithi175.stdout:3396: done (1 left) 2024-09-11T03:50:41.068 INFO:tasks.rados.rados.2.smithi175.stdout:3397: write (excl) oid 41 current snap is 375 2024-09-11T03:50:41.068 INFO:tasks.rados.rados.2.smithi175.stdout:3397: seq_num 1154 ranges {595278=406649,1729632=532069,3007663=387615} 2024-09-11T03:50:41.075 INFO:tasks.rados.rados.2.smithi175.stdout:3397: writing smithi17557870-41 from 595278 to 1001927 tid 1 2024-09-11T03:50:41.085 INFO:tasks.rados.rados.2.smithi175.stdout:3397: writing smithi17557870-41 from 1729632 to 2261701 tid 2 2024-09-11T03:50:41.089 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:41.092 INFO:tasks.rados.rados.2.smithi175.stdout:3397: writing smithi17557870-41 from 3007663 to 3395278 tid 3 2024-09-11T03:50:41.092 INFO:tasks.rados.rados.2.smithi175.stdout:3398: rollback oid 15 current snap is 375 2024-09-11T03:50:41.092 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 15 to 341 2024-09-11T03:50:41.092 INFO:tasks.rados.rados.2.smithi175.stdout:3397: finishing write tid 1 to smithi17557870-41 2024-09-11T03:50:41.092 INFO:tasks.rados.rados.2.smithi175.stdout:3397: finishing write tid 2 to smithi17557870-41 2024-09-11T03:50:41.092 INFO:tasks.rados.rados.2.smithi175.stdout:3395: done (2 left) 2024-09-11T03:50:41.092 INFO:tasks.rados.rados.2.smithi175.stdout:3399: snap_create 2024-09-11T03:50:41.094 INFO:tasks.rados.rados.2.smithi175.stdout:3397: finishing write tid 3 to smithi17557870-41 2024-09-11T03:50:41.096 INFO:tasks.rados.rados.2.smithi175.stdout:3397: finishing write tid 4 to smithi17557870-41 2024-09-11T03:50:41.096 INFO:tasks.rados.rados.2.smithi175.stdout:3398: finishing rollback tid 0 to smithi17557870-15 2024-09-11T03:50:41.096 INFO:tasks.rados.rados.2.smithi175.stdout:3397: finishing write tid 5 to smithi17557870-41 2024-09-11T03:50:41.096 INFO:tasks.rados.rados.2.smithi175.stdout:3397: oid 41 updating version 0 to 1012 2024-09-11T03:50:41.096 INFO:tasks.rados.rados.2.smithi175.stdout:3397: oid 41 version 1012 is already newer than 1009 2024-09-11T03:50:41.096 INFO:tasks.rados.rados.2.smithi175.stdout:3397: oid 41 version 1012 is already newer than 1010 2024-09-11T03:50:41.096 INFO:tasks.rados.rados.2.smithi175.stdout:3397: oid 41 version 1012 is already newer than 1011 2024-09-11T03:50:41.096 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 1012 (ObjNum 1154 snap 375 seq_num 1154) dirty exists 2024-09-11T03:50:41.096 INFO:tasks.rados.rados.2.smithi175.stdout:3397: left oid 41 (ObjNum 1154 snap 375 seq_num 1154) 2024-09-11T03:50:41.096 INFO:tasks.rados.rados.2.smithi175.stdout:3398: finishing rollback tid 1 to smithi17557870-15 2024-09-11T03:50:41.096 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 1059 (ObjNum 129 snap 0 seq_num 1745458968) dirty dne 2024-09-11T03:50:41.110 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:41.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:41.111 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:41.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:41.225 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:50:41.226 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:50:41.233 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:41.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:41.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:41.243 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:50:41.243 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:50:41.260 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:50:41.260 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:50:41.276 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:50:41.277 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:50:41.293 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:50:41.293 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:50:41.310 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:50:41.310 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:50:41.317 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:50:41.326 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:50:41.327 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:50:41.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:41.343 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:50:41.344 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:50:41.356 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:41.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:41.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:41.361 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:50:41.361 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:50:41.378 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:50:41.378 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:50:41.395 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:50:41.395 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:50:41.411 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:50:41.411 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:50:41.437 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:41 smithi175 ceph-mon[30894]: 2.21 scrub starts 2024-09-11T03:50:41.437 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:41 smithi175 ceph-mon[30894]: 2.21 scrub ok 2024-09-11T03:50:41.437 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:41 smithi175 ceph-mon[30894]: 2.2b scrub starts 2024-09-11T03:50:41.437 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:41 smithi175 ceph-mon[30894]: 2.2b scrub ok 2024-09-11T03:50:41.437 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:41 smithi175 ceph-mon[30894]: 2.5 scrub starts 2024-09-11T03:50:41.437 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:41 smithi175 ceph-mon[30894]: 2.5 scrub ok 2024-09-11T03:50:41.437 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:41 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:50:41.437 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:41 smithi175 ceph-mon[30894]: 2.7 scrub ok 2024-09-11T03:50:41.438 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:41 smithi175 ceph-mon[30894]: 1.6 scrub starts 2024-09-11T03:50:41.438 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:41 smithi175 ceph-mon[30894]: 1.6 scrub ok 2024-09-11T03:50:41.438 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:41 smithi175 ceph-mon[30894]: osdmap e823: 12 total, 12 up, 12 in 2024-09-11T03:50:41.438 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:41 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/220249397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:41.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:41.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:41.479 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:41.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:41.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:41.606 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:41.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:41.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:41.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:41 smithi084 ceph-mon[27254]: 2.21 scrub starts 2024-09-11T03:50:41.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:41 smithi084 ceph-mon[27254]: 2.21 scrub ok 2024-09-11T03:50:41.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:41 smithi084 ceph-mon[27254]: 2.2b scrub starts 2024-09-11T03:50:41.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:41 smithi084 ceph-mon[27254]: 2.2b scrub ok 2024-09-11T03:50:41.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:41 smithi084 ceph-mon[27254]: 2.5 scrub starts 2024-09-11T03:50:41.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:41 smithi084 ceph-mon[27254]: 2.5 scrub ok 2024-09-11T03:50:41.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:41 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:50:41.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:41 smithi084 ceph-mon[27254]: 2.7 scrub ok 2024-09-11T03:50:41.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:41 smithi084 ceph-mon[27254]: 1.6 scrub starts 2024-09-11T03:50:41.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:41 smithi084 ceph-mon[27254]: 1.6 scrub ok 2024-09-11T03:50:41.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:41 smithi084 ceph-mon[27254]: osdmap e823: 12 total, 12 up, 12 in 2024-09-11T03:50:41.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:41 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/220249397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:41 smithi121 ceph-mon[30729]: 2.21 scrub starts 2024-09-11T03:50:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:41 smithi121 ceph-mon[30729]: 2.21 scrub ok 2024-09-11T03:50:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:41 smithi121 ceph-mon[30729]: 2.2b scrub starts 2024-09-11T03:50:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:41 smithi121 ceph-mon[30729]: 2.2b scrub ok 2024-09-11T03:50:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:41 smithi121 ceph-mon[30729]: 2.5 scrub starts 2024-09-11T03:50:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:41 smithi121 ceph-mon[30729]: 2.5 scrub ok 2024-09-11T03:50:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:41 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:50:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:41 smithi121 ceph-mon[30729]: 2.7 scrub ok 2024-09-11T03:50:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:41 smithi121 ceph-mon[30729]: 1.6 scrub starts 2024-09-11T03:50:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:41 smithi121 ceph-mon[30729]: 1.6 scrub ok 2024-09-11T03:50:41.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:41 smithi121 ceph-mon[30729]: osdmap e823: 12 total, 12 up, 12 in 2024-09-11T03:50:41.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:41 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/220249397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:41.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:41.734 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:41.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:41.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:41.806 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:50:41.806 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:50:41.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:41.861 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:41.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:41.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:41.963 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:41.989 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:41.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:42.041 INFO:tasks.rados.rados.2.smithi175.stdout:3397: done (2 left) 2024-09-11T03:50:42.041 INFO:tasks.rados.rados.2.smithi175.stdout:3398: done (1 left) 2024-09-11T03:50:42.042 INFO:tasks.rados.rados.2.smithi175.stdout:3399: done (0 left) 2024-09-11T03:50:42.042 INFO:tasks.rados.rados.2.smithi175.stdout:3400: snap_create 2024-09-11T03:50:42.090 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:42.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:42.117 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:42.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:42.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:42.243 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:42.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:42.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:42.346 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:42.368 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:42.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:42.370 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:42.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:42 smithi175 ceph-mon[30894]: pgmap v1389: 97 pgs: 97 active+clean; 581 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:42 smithi175 ceph-mon[30894]: osdmap e824: 12 total, 12 up, 12 in 2024-09-11T03:50:42.497 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:42.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:42.537 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-11T03:50:42.598 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:42.622 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:42.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:42.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:42 smithi084 ceph-mon[27254]: pgmap v1389: 97 pgs: 97 active+clean; 581 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:42.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:42 smithi084 ceph-mon[27254]: osdmap e824: 12 total, 12 up, 12 in 2024-09-11T03:50:42.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:42 smithi121 ceph-mon[30729]: pgmap v1389: 97 pgs: 97 active+clean; 581 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:42.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:42 smithi121 ceph-mon[30729]: osdmap e824: 12 total, 12 up, 12 in 2024-09-11T03:50:42.696 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:50:42.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:42.761 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:42.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:42.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:42.775 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:50:42.807 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:50:42.807 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:50:42.807 INFO:tasks.thrashosds.thrasher:inject_pause on osd.9 2024-09-11T03:50:42.808 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T03:50:42.808 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T03:50:42.808 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 config set filestore_inject_stall 3 2024-09-11T03:50:42.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:42.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:42.886 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:42.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:42.943 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:42.987 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:43.008 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:43.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:43.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:43.042 INFO:tasks.rados.rados.2.smithi175.stdout:3400: done (0 left) 2024-09-11T03:50:43.042 INFO:tasks.rados.rados.2.smithi175.stdout:3401: copy_from oid 48 from oid 20 current snap is 377 2024-09-11T03:50:43.043 INFO:tasks.rados.rados.2.smithi175.stdout:3402: copy_from oid 30 from oid 8 current snap is 377 2024-09-11T03:50:43.043 INFO:tasks.rados.rados.2.smithi175.stdout:3403: snap_create 2024-09-11T03:50:43.046 INFO:tasks.rados.rados.2.smithi175.stdout:3401: finishing copy_from to smithi17557870-48 2024-09-11T03:50:43.046 INFO:tasks.rados.rados.2.smithi175.stdout:3401: got expected ENOENT (src dne) 2024-09-11T03:50:43.057 INFO:tasks.rados.rados.2.smithi175.stdout:3401: finishing copy_from racing read to smithi17557870-48 2024-09-11T03:50:43.058 INFO:tasks.rados.rados.2.smithi175.stdout:3402: finishing copy_from racing read to smithi17557870-30 2024-09-11T03:50:43.060 INFO:tasks.rados.rados.2.smithi175.stdout:3402: finishing copy_from to smithi17557870-30 2024-09-11T03:50:43.061 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 1073 (ObjNum 1147 snap 370 seq_num 1147) dirty exists 2024-09-11T03:50:43.109 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:43.130 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:43.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:43.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:43.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:43.253 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:43.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:43.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:43.329 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:50:43.329 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:50:43.355 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:43.378 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:43 smithi084 ceph-mon[27254]: 1.2 scrub starts 2024-09-11T03:50:43.378 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:43 smithi084 ceph-mon[27254]: 1.2 scrub ok 2024-09-11T03:50:43.378 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:43 smithi084 ceph-mon[27254]: osdmap e825: 12 total, 12 up, 12 in 2024-09-11T03:50:43.381 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:43.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:43 smithi175 ceph-mon[30894]: 1.2 scrub starts 2024-09-11T03:50:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:43 smithi175 ceph-mon[30894]: 1.2 scrub ok 2024-09-11T03:50:43.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:43 smithi175 ceph-mon[30894]: osdmap e825: 12 total, 12 up, 12 in 2024-09-11T03:50:43.482 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:43.506 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:43.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:43.507 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:43.609 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:43.631 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:43.632 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:43.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:43 smithi121 ceph-mon[30729]: 1.2 scrub starts 2024-09-11T03:50:43.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:43 smithi121 ceph-mon[30729]: 1.2 scrub ok 2024-09-11T03:50:43.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:43 smithi121 ceph-mon[30729]: osdmap e825: 12 total, 12 up, 12 in 2024-09-11T03:50:43.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:43.758 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:43.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:43.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:43.860 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:43.885 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:43.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:43.970 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-11T03:50:43.974 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:50:43.987 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:44.009 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:44.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:44.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:44.038 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:50:44.042 INFO:tasks.rados.rados.2.smithi175.stdout:3401: done (2 left) 2024-09-11T03:50:44.042 INFO:tasks.rados.rados.2.smithi175.stdout:3402: done (1 left) 2024-09-11T03:50:44.042 INFO:tasks.rados.rados.2.smithi175.stdout:3403: done (0 left) 2024-09-11T03:50:44.042 INFO:tasks.rados.rados.2.smithi175.stdout:3404: write oid 21 current snap is 378 2024-09-11T03:50:44.042 INFO:tasks.rados.rados.2.smithi175.stdout:3404: seq_num 1155 ranges {612085=481898,1788114=675727,3035086=376999} 2024-09-11T03:50:44.047 INFO:tasks.rados.rados.2.smithi175.stdout:3404: writing smithi17557870-21 from 612085 to 1093983 tid 1 2024-09-11T03:50:44.055 INFO:tasks.rados.rados.2.smithi175.stdout:3404: writing smithi17557870-21 from 1788114 to 2463841 tid 2 2024-09-11T03:50:44.061 INFO:tasks.rados.rados.2.smithi175.stdout:3404: writing smithi17557870-21 from 3035086 to 3412085 tid 3 2024-09-11T03:50:44.061 INFO:tasks.rados.rados.2.smithi175.stdout:3405: rollback oid 10 current snap is 378 2024-09-11T03:50:44.061 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 10 to 365 2024-09-11T03:50:44.061 INFO:tasks.rados.rados.2.smithi175.stdout:3404: finishing write tid 1 to smithi17557870-21 2024-09-11T03:50:44.061 INFO:tasks.rados.rados.2.smithi175.stdout:3404: finishing write tid 2 to smithi17557870-21 2024-09-11T03:50:44.061 INFO:tasks.rados.rados.2.smithi175.stdout:3406: read oid 3 snap -1 2024-09-11T03:50:44.061 INFO:tasks.rados.rados.2.smithi175.stdout:3406: expect deleted 2024-09-11T03:50:44.062 INFO:tasks.rados.rados.2.smithi175.stdout:3407: snap_remove snap 307 2024-09-11T03:50:44.110 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:44.132 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:44.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:44.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:44.234 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:44.259 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:44.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:44.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:44.339 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:44 smithi084 ceph-mon[27254]: pgmap v1392: 97 pgs: 97 active+clean; 593 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:44.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:44 smithi084 ceph-mon[27254]: 1.17 scrub starts 2024-09-11T03:50:44.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:44 smithi084 ceph-mon[27254]: 1.17 scrub ok 2024-09-11T03:50:44.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:44 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/224888400' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:50:44.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:44 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:44.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:44 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/224888400' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:50:44.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:44 smithi084 ceph-mon[27254]: osdmap e826: 12 total, 12 up, 12 in 2024-09-11T03:50:44.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:44.367 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:50:44.387 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:44.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:44.484 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:50:44.484 INFO:teuthology.orchestra.run.smithi175.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T03:50:44.484 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:50:44.488 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:44.514 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:44.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:44.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:44.616 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:44.637 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:44.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:44.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:44 smithi121 ceph-mon[30729]: pgmap v1392: 97 pgs: 97 active+clean; 593 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:44 smithi121 ceph-mon[30729]: 1.17 scrub starts 2024-09-11T03:50:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:44 smithi121 ceph-mon[30729]: 1.17 scrub ok 2024-09-11T03:50:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:44 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/224888400' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:50:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:44 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:44 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/224888400' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:50:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:44 smithi121 ceph-mon[30729]: osdmap e826: 12 total, 12 up, 12 in 2024-09-11T03:50:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:44 smithi175 ceph-mon[30894]: pgmap v1392: 97 pgs: 97 active+clean; 593 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:44 smithi175 ceph-mon[30894]: 1.17 scrub starts 2024-09-11T03:50:44.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:44 smithi175 ceph-mon[30894]: 1.17 scrub ok 2024-09-11T03:50:44.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:44 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/224888400' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:50:44.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:44 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:44.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:44 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/224888400' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:50:44.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:44 smithi175 ceph-mon[30894]: osdmap e826: 12 total, 12 up, 12 in 2024-09-11T03:50:44.739 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:44.765 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:44.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:44.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:44.789 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:44.866 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:44.889 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:44.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:44.890 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:44.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:45.017 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:45.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:45.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:45.044 INFO:tasks.rados.rados.2.smithi175.stdout:3407: done (3 left) 2024-09-11T03:50:45.045 INFO:tasks.rados.rados.2.smithi175.stdout:3408: snap_create 2024-09-11T03:50:45.045 INFO:tasks.rados.rados.2.smithi175.stdout:3404: finishing write tid 3 to smithi17557870-21 2024-09-11T03:50:45.045 INFO:tasks.rados.rados.2.smithi175.stdout:3404: finishing write tid 4 to smithi17557870-21 2024-09-11T03:50:45.045 INFO:tasks.rados.rados.2.smithi175.stdout:3405: finishing rollback tid 0 to smithi17557870-10 2024-09-11T03:50:45.045 INFO:tasks.rados.rados.2.smithi175.stdout:3405: finishing rollback tid 1 to smithi17557870-10 2024-09-11T03:50:45.045 INFO:tasks.rados.rados.2.smithi175.stdout:3404: finishing write tid 5 to smithi17557870-21 2024-09-11T03:50:45.045 INFO:tasks.rados.rados.2.smithi175.stdout:3404: oid 21 updating version 0 to 1236 2024-09-11T03:50:45.045 INFO:tasks.rados.rados.2.smithi175.stdout:3404: oid 21 updating version 1236 to 1238 2024-09-11T03:50:45.045 INFO:tasks.rados.rados.2.smithi175.stdout:3404: oid 21 version 1238 is already newer than 1235 2024-09-11T03:50:45.045 INFO:tasks.rados.rados.2.smithi175.stdout:3404: oid 21 version 1238 is already newer than 1237 2024-09-11T03:50:45.045 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 1238 (ObjNum 1155 snap 378 seq_num 1155) dirty exists 2024-09-11T03:50:45.045 INFO:tasks.rados.rados.2.smithi175.stdout:3404: left oid 21 (ObjNum 1155 snap 378 seq_num 1155) 2024-09-11T03:50:45.045 INFO:tasks.rados.rados.2.smithi175.stdout:3405: finishing rollback tid 2 to smithi17557870-10 2024-09-11T03:50:45.046 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 1124 (ObjNum 1075 snap 350 seq_num 1075) dirty exists 2024-09-11T03:50:45.118 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:45.134 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:50:45.140 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:45.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:45.141 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:45.242 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:45.265 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:45.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:45.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:45.277 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:50:45.277 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:50:45.294 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:50:45.294 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:50:45.310 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:50:45.311 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:50:45.327 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:50:45.327 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:50:45.343 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:50:45.344 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:50:45.361 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:50:45.361 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:50:45.367 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:45.377 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:50:45.378 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:50:45.389 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:45.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:45.390 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:45.396 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:50:45.396 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:50:45.410 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:50:45.411 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:50:45.427 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:50:45.427 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:50:45.443 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:50:45.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:50:45.459 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:50:45.460 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:50:45.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:45.507 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:45 smithi175 ceph-mon[30894]: 2.26 scrub starts 2024-09-11T03:50:45.507 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:45 smithi175 ceph-mon[30894]: 2.26 scrub ok 2024-09-11T03:50:45.507 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:45 smithi175 ceph-mon[30894]: osdmap e827: 12 total, 12 up, 12 in 2024-09-11T03:50:45.507 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:45 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2075323701' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:45.517 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:45.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:45.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:45.614 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:50:45.614 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:50:45.614 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:50:45.614 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:50:45.614 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:50:45.619 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:45.640 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:45.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:45.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:45.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:45 smithi084 ceph-mon[27254]: 2.26 scrub starts 2024-09-11T03:50:45.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:45 smithi084 ceph-mon[27254]: 2.26 scrub ok 2024-09-11T03:50:45.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:45 smithi084 ceph-mon[27254]: osdmap e827: 12 total, 12 up, 12 in 2024-09-11T03:50:45.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:45 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2075323701' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:45.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:45 smithi121 ceph-mon[30729]: 2.26 scrub starts 2024-09-11T03:50:45.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:45 smithi121 ceph-mon[30729]: 2.26 scrub ok 2024-09-11T03:50:45.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:45 smithi121 ceph-mon[30729]: osdmap e827: 12 total, 12 up, 12 in 2024-09-11T03:50:45.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:45 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2075323701' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:45.742 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:45.764 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:45.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:45.765 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:45.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:45.892 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:45.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:45.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:45.994 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:46.017 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:46.017 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:46.045 INFO:tasks.rados.rados.2.smithi175.stdout:3404: done (3 left) 2024-09-11T03:50:46.045 INFO:tasks.rados.rados.2.smithi175.stdout:3405: done (2 left) 2024-09-11T03:50:46.045 INFO:tasks.rados.rados.2.smithi175.stdout:3406: done (1 left) 2024-09-11T03:50:46.045 INFO:tasks.rados.rados.2.smithi175.stdout:3408: done (0 left) 2024-09-11T03:50:46.045 INFO:tasks.rados.rados.2.smithi175.stdout:3409: delete oid 16 current snap is 379 2024-09-11T03:50:46.050 INFO:tasks.rados.rados.2.smithi175.stdout:3409: done (0 left) 2024-09-11T03:50:46.050 INFO:tasks.rados.rados.2.smithi175.stdout:3410: copy_from oid 37 from oid 22 current snap is 379 2024-09-11T03:50:46.050 INFO:tasks.rados.rados.2.smithi175.stdout:3411: copy_from oid 38 from oid 19 current snap is 379 2024-09-11T03:50:46.050 INFO:tasks.rados.rados.2.smithi175.stdout:3412: write (excl) oid 33 current snap is 379 2024-09-11T03:50:46.050 INFO:tasks.rados.rados.2.smithi175.stdout:3412: seq_num 1157 ranges {645699=748749,2021431=679396,3206285=239414} 2024-09-11T03:50:46.058 INFO:tasks.rados.rados.2.smithi175.stdout:3412: writing smithi17557870-33 from 645699 to 1394448 tid 1 2024-09-11T03:50:46.065 INFO:tasks.rados.rados.2.smithi175.stdout:3412: writing smithi17557870-33 from 2021431 to 2700827 tid 2 2024-09-11T03:50:46.069 INFO:tasks.rados.rados.2.smithi175.stdout:3412: writing smithi17557870-33 from 3206285 to 3445699 tid 3 2024-09-11T03:50:46.069 INFO:tasks.rados.rados.2.smithi175.stdout:3413: copy_from oid 21 from oid 40 current snap is 379 2024-09-11T03:50:46.070 INFO:tasks.rados.rados.2.smithi175.stdout:3411: finishing copy_from racing read to smithi17557870-38 2024-09-11T03:50:46.070 INFO:tasks.rados.rados.2.smithi175.stdout:3411: finishing copy_from to smithi17557870-38 2024-09-11T03:50:46.070 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 975 (ObjNum 1079 snap 352 seq_num 1079) dirty exists 2024-09-11T03:50:46.070 INFO:tasks.rados.rados.2.smithi175.stdout:3412: finishing write tid 1 to smithi17557870-33 2024-09-11T03:50:46.070 INFO:tasks.rados.rados.2.smithi175.stdout:3410: finishing copy_from racing read to smithi17557870-37 2024-09-11T03:50:46.071 INFO:tasks.rados.rados.2.smithi175.stdout:3410: finishing copy_from to smithi17557870-37 2024-09-11T03:50:46.071 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 1253 (ObjNum 829 snap 263 seq_num 829) dirty exists 2024-09-11T03:50:46.071 INFO:tasks.rados.rados.2.smithi175.stdout:3410: done (3 left) 2024-09-11T03:50:46.071 INFO:tasks.rados.rados.2.smithi175.stdout:3411: done (2 left) 2024-09-11T03:50:46.071 INFO:tasks.rados.rados.2.smithi175.stdout:3414: copy_from oid 30 from oid 46 current snap is 379 2024-09-11T03:50:46.071 INFO:tasks.rados.rados.2.smithi175.stdout:3412: finishing write tid 2 to smithi17557870-33 2024-09-11T03:50:46.071 INFO:tasks.rados.rados.2.smithi175.stdout:3415: read oid 49 snap -1 2024-09-11T03:50:46.071 INFO:tasks.rados.rados.2.smithi175.stdout:3415: expect deleted 2024-09-11T03:50:46.071 INFO:tasks.rados.rados.2.smithi175.stdout:3416: rollback oid 47 current snap is 379 2024-09-11T03:50:46.072 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 47 to 373 2024-09-11T03:50:46.072 INFO:tasks.rados.rados.2.smithi175.stdout:3417: copy_from oid 31 from oid 48 current snap is 379 2024-09-11T03:50:46.072 INFO:tasks.rados.rados.2.smithi175.stdout:3418: rollback oid 15 current snap is 379 2024-09-11T03:50:46.072 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 15 to 321 2024-09-11T03:50:46.072 INFO:tasks.rados.rados.2.smithi175.stdout:3419: delete oid 20 current snap is 379 2024-09-11T03:50:46.073 INFO:tasks.rados.rados.2.smithi175.stdout:3414: finishing copy_from to smithi17557870-30 2024-09-11T03:50:46.073 INFO:tasks.rados.rados.2.smithi175.stdout:3414: got expected ENOENT (src dne) 2024-09-11T03:50:46.073 INFO:tasks.rados.rados.2.smithi175.stdout:3414: finishing copy_from racing read to smithi17557870-30 2024-09-11T03:50:46.073 INFO:tasks.rados.rados.2.smithi175.stdout:3418: finishing rollback tid 1 to smithi17557870-15 2024-09-11T03:50:46.073 INFO:tasks.rados.rados.2.smithi175.stdout:3414: done (7 left) 2024-09-11T03:50:46.073 INFO:tasks.rados.rados.2.smithi175.stdout:3415: done (6 left) 2024-09-11T03:50:46.074 INFO:tasks.rados.rados.2.smithi175.stdout:3419: done (5 left) 2024-09-11T03:50:46.074 INFO:tasks.rados.rados.2.smithi175.stdout:3420: rollback oid 10 current snap is 379 2024-09-11T03:50:46.074 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 10 to 293 2024-09-11T03:50:46.074 INFO:tasks.rados.rados.2.smithi175.stdout:3421: read oid 35 snap -1 2024-09-11T03:50:46.074 INFO:tasks.rados.rados.2.smithi175.stdout:3421: expect deleted 2024-09-11T03:50:46.074 INFO:tasks.rados.rados.2.smithi175.stdout:3422: write oid 19 current snap is 379 2024-09-11T03:50:46.074 INFO:tasks.rados.rados.2.smithi175.stdout:3422: seq_num 1159 ranges {679313=499247,1622042=683065,2744778=701557} 2024-09-11T03:50:46.078 INFO:tasks.rados.rados.2.smithi175.stdout:3422: writing smithi17557870-19 from 679313 to 1178560 tid 1 2024-09-11T03:50:46.085 INFO:tasks.rados.rados.2.smithi175.stdout:3422: writing smithi17557870-19 from 1622042 to 2305107 tid 2 2024-09-11T03:50:46.092 INFO:tasks.rados.rados.2.smithi175.stdout:3422: writing smithi17557870-19 from 2744778 to 3446335 tid 3 2024-09-11T03:50:46.092 INFO:tasks.rados.rados.2.smithi175.stdout:3423: delete oid 5 current snap is 379 2024-09-11T03:50:46.092 INFO:tasks.rados.rados.2.smithi175.stdout:3412: finishing write tid 3 to smithi17557870-33 2024-09-11T03:50:46.092 INFO:tasks.rados.rados.2.smithi175.stdout:3418: finishing rollback tid 2 to smithi17557870-15 2024-09-11T03:50:46.093 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 1061 (ObjNum 950 snap 303 seq_num 950) dirty exists 2024-09-11T03:50:46.093 INFO:tasks.rados.rados.2.smithi175.stdout:3412: finishing write tid 4 to smithi17557870-33 2024-09-11T03:50:46.093 INFO:tasks.rados.rados.2.smithi175.stdout:3412: finishing write tid 5 to smithi17557870-33 2024-09-11T03:50:46.093 INFO:tasks.rados.rados.2.smithi175.stdout:3412: oid 33 updating version 0 to 1319 2024-09-11T03:50:46.093 INFO:tasks.rados.rados.2.smithi175.stdout:3412: oid 33 version 1319 is already newer than 1318 2024-09-11T03:50:46.093 INFO:tasks.rados.rados.2.smithi175.stdout:3412: oid 33 version 1319 is already newer than 1317 2024-09-11T03:50:46.093 INFO:tasks.rados.rados.2.smithi175.stdout:3412: oid 33 updating version 1319 to 1320 2024-09-11T03:50:46.093 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 1320 (ObjNum 1157 snap 379 seq_num 1157) dirty exists 2024-09-11T03:50:46.093 INFO:tasks.rados.rados.2.smithi175.stdout:3412: left oid 33 (ObjNum 1157 snap 379 seq_num 1157) 2024-09-11T03:50:46.093 INFO:tasks.rados.rados.2.smithi175.stdout:3416: finishing rollback tid 0 to smithi17557870-47 2024-09-11T03:50:46.093 INFO:tasks.rados.rados.2.smithi175.stdout:3416: finishing rollback tid 1 to smithi17557870-47 2024-09-11T03:50:46.094 INFO:tasks.rados.rados.2.smithi175.stdout:3416: finishing rollback tid 2 to smithi17557870-47 2024-09-11T03:50:46.094 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 1134 (ObjNum 1132 snap 367 seq_num 1132) dirty exists 2024-09-11T03:50:46.094 INFO:tasks.rados.rados.2.smithi175.stdout:3420: finishing rollback tid 0 to smithi17557870-10 2024-09-11T03:50:46.094 INFO:tasks.rados.rados.2.smithi175.stdout:3420: finishing rollback tid 1 to smithi17557870-10 2024-09-11T03:50:46.094 INFO:tasks.rados.rados.2.smithi175.stdout:3420: finishing rollback tid 2 to smithi17557870-10 2024-09-11T03:50:46.094 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 1138 (ObjNum 736 snap 243 seq_num 736) dirty exists 2024-09-11T03:50:46.094 INFO:tasks.rados.rados.2.smithi175.stdout:3413: finishing copy_from racing read to smithi17557870-21 2024-09-11T03:50:46.094 INFO:tasks.rados.rados.2.smithi175.stdout:3413: finishing copy_from to smithi17557870-21 2024-09-11T03:50:46.094 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 1006 (ObjNum 973 snap 313 seq_num 973) dirty exists 2024-09-11T03:50:46.094 INFO:tasks.rados.rados.2.smithi175.stdout:3422: finishing write tid 1 to smithi17557870-19 2024-09-11T03:50:46.094 INFO:tasks.rados.rados.2.smithi175.stdout:3417: finishing copy_from racing read to smithi17557870-31 2024-09-11T03:50:46.094 INFO:tasks.rados.rados.2.smithi175.stdout:3422: finishing write tid 2 to smithi17557870-19 2024-09-11T03:50:46.094 INFO:tasks.rados.rados.2.smithi175.stdout:3417: finishing copy_from to smithi17557870-31 2024-09-11T03:50:46.095 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 1001 (ObjNum 1149 snap 371 seq_num 1149) dirty exists 2024-09-11T03:50:46.098 INFO:tasks.rados.rados.2.smithi175.stdout:3412: done (8 left) 2024-09-11T03:50:46.098 INFO:tasks.rados.rados.2.smithi175.stdout:3413: done (7 left) 2024-09-11T03:50:46.098 INFO:tasks.rados.rados.2.smithi175.stdout:3416: done (6 left) 2024-09-11T03:50:46.098 INFO:tasks.rados.rados.2.smithi175.stdout:3417: done (5 left) 2024-09-11T03:50:46.098 INFO:tasks.rados.rados.2.smithi175.stdout:3418: done (4 left) 2024-09-11T03:50:46.098 INFO:tasks.rados.rados.2.smithi175.stdout:3420: done (3 left) 2024-09-11T03:50:46.098 INFO:tasks.rados.rados.2.smithi175.stdout:3421: done (2 left) 2024-09-11T03:50:46.098 INFO:tasks.rados.rados.2.smithi175.stdout:3423: done (1 left) 2024-09-11T03:50:46.098 INFO:tasks.rados.rados.2.smithi175.stdout:3424: rollback oid 10 current snap is 379 2024-09-11T03:50:46.098 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 10 to 357 2024-09-11T03:50:46.099 INFO:tasks.rados.rados.2.smithi175.stdout:3425: rollback oid 41 current snap is 379 2024-09-11T03:50:46.099 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 41 to 370 2024-09-11T03:50:46.099 INFO:tasks.rados.rados.2.smithi175.stdout:3426: snap_create 2024-09-11T03:50:46.099 INFO:tasks.rados.rados.2.smithi175.stdout:3425: finishing rollback tid 0 to smithi17557870-41 2024-09-11T03:50:46.099 INFO:tasks.rados.rados.2.smithi175.stdout:3422: finishing write tid 3 to smithi17557870-19 2024-09-11T03:50:46.099 INFO:tasks.rados.rados.2.smithi175.stdout:3422: finishing write tid 4 to smithi17557870-19 2024-09-11T03:50:46.099 INFO:tasks.rados.rados.2.smithi175.stdout:3425: finishing rollback tid 1 to smithi17557870-41 2024-09-11T03:50:46.099 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 1014 (ObjNum 65 snap 0 seq_num 1746396496) dirty dne 2024-09-11T03:50:46.099 INFO:tasks.rados.rados.2.smithi175.stdout:3422: finishing write tid 5 to smithi17557870-19 2024-09-11T03:50:46.099 INFO:tasks.rados.rados.2.smithi175.stdout:3422: oid 19 updating version 0 to 1004 2024-09-11T03:50:46.100 INFO:tasks.rados.rados.2.smithi175.stdout:3422: oid 19 updating version 1004 to 1005 2024-09-11T03:50:46.100 INFO:tasks.rados.rados.2.smithi175.stdout:3422: oid 19 version 1005 is already newer than 1002 2024-09-11T03:50:46.100 INFO:tasks.rados.rados.2.smithi175.stdout:3422: oid 19 version 1005 is already newer than 1003 2024-09-11T03:50:46.100 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 1005 (ObjNum 1159 snap 379 seq_num 1159) dirty exists 2024-09-11T03:50:46.100 INFO:tasks.rados.rados.2.smithi175.stdout:3422: left oid 19 (ObjNum 1159 snap 379 seq_num 1159) 2024-09-11T03:50:46.100 INFO:tasks.rados.rados.2.smithi175.stdout:3424: finishing rollback tid 0 to smithi17557870-10 2024-09-11T03:50:46.100 INFO:tasks.rados.rados.2.smithi175.stdout:3424: finishing rollback tid 1 to smithi17557870-10 2024-09-11T03:50:46.100 INFO:tasks.rados.rados.2.smithi175.stdout:3424: finishing rollback tid 2 to smithi17557870-10 2024-09-11T03:50:46.100 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 1141 (ObjNum 1075 snap 350 seq_num 1075) dirty exists 2024-09-11T03:50:46.118 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:46.140 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:46.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:46.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:46.242 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:46.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:46.264 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:46.265 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:46.319 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-11T03:50:46.365 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:46.387 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:46.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:46.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:46.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:46.513 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:46.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:46.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:46.559 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:50:46.615 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:46.638 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:46.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:46.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:46.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:46 smithi084 ceph-mon[27254]: 1.3 scrub starts 2024-09-11T03:50:46.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:46 smithi084 ceph-mon[27254]: pgmap v1395: 97 pgs: 97 active+clean; 593 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:46.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:46 smithi084 ceph-mon[27254]: osdmap e828: 12 total, 12 up, 12 in 2024-09-11T03:50:46.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:46 smithi121 ceph-mon[30729]: 1.3 scrub starts 2024-09-11T03:50:46.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:46 smithi121 ceph-mon[30729]: pgmap v1395: 97 pgs: 97 active+clean; 593 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:46.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:46 smithi121 ceph-mon[30729]: osdmap e828: 12 total, 12 up, 12 in 2024-09-11T03:50:46.679 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:46 smithi175 ceph-mon[30894]: 1.3 scrub starts 2024-09-11T03:50:46.679 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:46 smithi175 ceph-mon[30894]: pgmap v1395: 97 pgs: 97 active+clean; 593 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:50:46.679 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:46 smithi175 ceph-mon[30894]: osdmap e828: 12 total, 12 up, 12 in 2024-09-11T03:50:46.741 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:46.756 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:50:46.763 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:46.764 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:46.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:46.890 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:46.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:46.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:47.004 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:50:47.004 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:50:47.004 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:50:47.004 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:50:47.004 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.004 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2972 2.c 2:32ade24e:::smithi17557870-45:head [copy-from ver 171] snapc 80=[80,7b,7a,79,77,75] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-11T03:50:47.004 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:50:47.004 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 481.92903417000002, 2024-09-11T03:50:47.004 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021036193000000002, 2024-09-11T03:50:47.005 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.005 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.005 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:47.005 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.005 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.005 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.005 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2972 2024-09-11T03:50:47.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.005 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.005 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.005 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.006 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:50:47.006 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.006 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.006 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.006 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.006 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:50:47.006 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.006 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.006 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.006 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.006 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074096+0000", 2024-09-11T03:50:47.006 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.685e-06 2024-09-11T03:50:47.007 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.007 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.007 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.007 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074104+0000", 2024-09-11T03:50:47.007 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9950000000000005e-06 2024-09-11T03:50:47.007 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.007 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.007 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.007 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074123+0000", 2024-09-11T03:50:47.007 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9587999999999999e-05 2024-09-11T03:50:47.007 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.007 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.007 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.008 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074132+0000", 2024-09-11T03:50:47.008 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1579999999999994e-06 2024-09-11T03:50:47.008 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.008 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.008 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.008 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074204+0000", 2024-09-11T03:50:47.008 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2072999999999999e-05 2024-09-11T03:50:47.008 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.008 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.008 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:47.008 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074302+0000", 2024-09-11T03:50:47.008 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7706000000000003e-05 2024-09-11T03:50:47.008 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.009 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.009 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:50:47.009 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.087397+0000", 2024-09-11T03:50:47.009 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013095292999999999 2024-09-11T03:50:47.009 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.009 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.009 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:47.009 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090766+0000", 2024-09-11T03:50:47.009 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0033682109999999999 2024-09-11T03:50:47.009 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.009 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.009 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:47.010 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.094977+0000", 2024-09-11T03:50:47.010 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0042108759999999997 2024-09-11T03:50:47.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.010 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.010 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:47.010 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095061+0000", 2024-09-11T03:50:47.010 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4827999999999995e-05 2024-09-11T03:50:47.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.010 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.010 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:47.010 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095099+0000", 2024-09-11T03:50:47.010 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7546999999999998e-05 2024-09-11T03:50:47.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.011 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.011 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.011 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095130+0000", 2024-09-11T03:50:47.011 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1232999999999997e-05 2024-09-11T03:50:47.011 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.011 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.011 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.011 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.011 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.011 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2973 2.c 2:32ade24e:::smithi17557870-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e348)", 2024-09-11T03:50:47.011 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:50:47.011 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 481.92896905200001, 2024-09-11T03:50:47.012 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016062098, 2024-09-11T03:50:47.012 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.012 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.012 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:50:47.012 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.012 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.012 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.012 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2973 2024-09-11T03:50:47.012 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.012 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.012 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.012 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.012 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:50:47.013 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.013 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.013 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.013 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.013 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:50:47.013 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.013 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.013 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.013 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.013 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074160+0000", 2024-09-11T03:50:47.013 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.46e-07 2024-09-11T03:50:47.013 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.014 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.014 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.014 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074162+0000", 2024-09-11T03:50:47.014 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5220000000000002e-06 2024-09-11T03:50:47.014 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.014 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.014 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.014 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074178+0000", 2024-09-11T03:50:47.014 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.609e-05 2024-09-11T03:50:47.014 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.014 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.014 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.014 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074183+0000", 2024-09-11T03:50:47.015 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8280000000000002e-06 2024-09-11T03:50:47.015 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.015 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.015 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.015 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074428+0000", 2024-09-11T03:50:47.015 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00024529599999999999 2024-09-11T03:50:47.015 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.015 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.015 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:47.015 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074469+0000", 2024-09-11T03:50:47.015 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0463000000000002e-05 2024-09-11T03:50:47.015 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.015 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.016 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.016 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090093+0000", 2024-09-11T03:50:47.016 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015624449 2024-09-11T03:50:47.016 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.016 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.016 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:47.016 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090155+0000", 2024-09-11T03:50:47.016 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1618999999999994e-05 2024-09-11T03:50:47.016 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.016 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.016 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.016 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090221+0000", 2024-09-11T03:50:47.016 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6384999999999995e-05 2024-09-11T03:50:47.017 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.017 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.017 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.017 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.017 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.017 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5348 2.1b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e571)", 2024-09-11T03:50:47.017 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:28.996464+0000", 2024-09-11T03:50:47.017 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 258.00666363599998, 2024-09-11T03:50:47.017 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.973168225, 2024-09-11T03:50:47.017 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.017 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.017 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "reached pg", 2024-09-11T03:50:47.017 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.017 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.018 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.018 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5348 2024-09-11T03:50:47.018 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.018 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.018 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.018 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.018 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996464+0000", 2024-09-11T03:50:47.018 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.018 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.018 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.018 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.018 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996464+0000", 2024-09-11T03:50:47.018 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.018 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.019 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.019 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.019 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996465+0000", 2024-09-11T03:50:47.019 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3500000000000002e-07 2024-09-11T03:50:47.019 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.019 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.019 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.020 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:47.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:47.021 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996467+0000", 2024-09-11T03:50:47.021 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.576e-06 2024-09-11T03:50:47.021 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.021 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.021 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.021 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996475+0000", 2024-09-11T03:50:47.021 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1350000000000003e-06 2024-09-11T03:50:47.021 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.021 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.021 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.022 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996478+0000", 2024-09-11T03:50:47.022 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5240000000000001e-06 2024-09-11T03:50:47.022 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.022 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.022 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.022 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996545+0000", 2024-09-11T03:50:47.022 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7379000000000007e-05 2024-09-11T03:50:47.022 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.022 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.022 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:50:47.022 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996557+0000", 2024-09-11T03:50:47.022 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2019e-05 2024-09-11T03:50:47.022 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.023 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.023 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.023 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969589+0000", 2024-09-11T03:50:47.023 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97303155699999999 2024-09-11T03:50:47.023 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.023 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.023 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.023 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969633+0000", 2024-09-11T03:50:47.023 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3699999999999998e-05 2024-09-11T03:50:47.023 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.023 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.023 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.023 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.024 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.024 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5349 2.1b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e571)", 2024-09-11T03:50:47.024 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:50:47.024 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 258.00664050799998, 2024-09-11T03:50:47.024 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97318775899999999, 2024-09-11T03:50:47.024 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.024 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.024 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "reached pg", 2024-09-11T03:50:47.024 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.024 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.024 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.024 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5349 2024-09-11T03:50:47.024 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.024 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.025 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.025 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.025 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:50:47.025 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.025 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.025 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.025 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.025 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:50:47.025 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.025 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.025 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.025 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.025 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:50:47.026 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7300000000000002e-07 2024-09-11T03:50:47.026 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.026 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.026 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.026 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996490+0000", 2024-09-11T03:50:47.026 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8470000000000001e-06 2024-09-11T03:50:47.026 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.026 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.026 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.026 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996494+0000", 2024-09-11T03:50:47.026 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8419999999999998e-06 2024-09-11T03:50:47.026 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.026 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.026 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.027 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996497+0000", 2024-09-11T03:50:47.027 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8820000000000001e-06 2024-09-11T03:50:47.027 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.027 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.027 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.027 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996602+0000", 2024-09-11T03:50:47.027 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010565699999999999 2024-09-11T03:50:47.027 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.027 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.027 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:50:47.027 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996613+0000", 2024-09-11T03:50:47.027 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0359e-05 2024-09-11T03:50:47.028 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.028 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.028 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.028 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969667+0000", 2024-09-11T03:50:47.028 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97305483599999998 2024-09-11T03:50:47.028 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.028 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.028 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.028 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969675+0000", 2024-09-11T03:50:47.028 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9629999999999998e-06 2024-09-11T03:50:47.028 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.028 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.028 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.029 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.029 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.029 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5350 2.1b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e571)", 2024-09-11T03:50:47.029 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:50:47.029 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 258.00660535200001, 2024-09-11T03:50:47.029 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.973193639, 2024-09-11T03:50:47.029 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.029 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.029 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "reached pg", 2024-09-11T03:50:47.032 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.032 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.032 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.032 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5350 2024-09-11T03:50:47.032 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.032 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.032 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.032 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.032 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:50:47.032 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.032 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.033 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.033 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.033 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:50:47.033 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.033 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.033 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.033 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.033 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:50:47.033 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.27e-07 2024-09-11T03:50:47.033 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.033 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.033 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.033 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996525+0000", 2024-09-11T03:50:47.033 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.719e-06 2024-09-11T03:50:47.034 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.034 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.034 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.034 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996529+0000", 2024-09-11T03:50:47.034 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2690000000000001e-06 2024-09-11T03:50:47.034 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.034 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.034 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.034 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996532+0000", 2024-09-11T03:50:47.034 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4139999999999998e-06 2024-09-11T03:50:47.034 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.034 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.034 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.035 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996643+0000", 2024-09-11T03:50:47.035 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011041599999999999 2024-09-11T03:50:47.035 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.035 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.035 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:50:47.035 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996651+0000", 2024-09-11T03:50:47.035 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4309999999999994e-06 2024-09-11T03:50:47.035 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.035 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.035 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.035 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969706+0000", 2024-09-11T03:50:47.035 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97305510699999997 2024-09-11T03:50:47.035 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.035 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.036 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.036 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969716+0000", 2024-09-11T03:50:47.036 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9559999999999992e-06 2024-09-11T03:50:47.036 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.036 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.036 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.036 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.036 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.036 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5348 2.1b 2:dde92fc6:::smithi17557870-41:head [sparse-read 0~1317959,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:50:47.036 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973173+0000", 2024-09-11T03:50:47.036 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 256.02995517400001, 2024-09-11T03:50:47.036 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019341553000000001, 2024-09-11T03:50:47.036 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.036 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.037 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:50:47.037 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.037 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.037 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.037 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5348 2024-09-11T03:50:47.037 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.037 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.037 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.037 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.037 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973173+0000", 2024-09-11T03:50:47.037 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.037 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.037 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.038 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.038 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973173+0000", 2024-09-11T03:50:47.038 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.038 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.038 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.038 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.038 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973175+0000", 2024-09-11T03:50:47.038 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.074e-06 2024-09-11T03:50:47.038 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.038 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.038 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.038 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973183+0000", 2024-09-11T03:50:47.038 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3010000000000007e-06 2024-09-11T03:50:47.039 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.039 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.039 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.039 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973199+0000", 2024-09-11T03:50:47.039 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5279000000000001e-05 2024-09-11T03:50:47.039 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.039 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.039 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.039 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973214+0000", 2024-09-11T03:50:47.039 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5891000000000001e-05 2024-09-11T03:50:47.039 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.039 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.039 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.039 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973747+0000", 2024-09-11T03:50:47.040 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00053236600000000005 2024-09-11T03:50:47.040 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.040 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.040 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:50:47.040 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973760+0000", 2024-09-11T03:50:47.040 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2976000000000001e-05 2024-09-11T03:50:47.040 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.040 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.040 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.040 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.990782+0000", 2024-09-11T03:50:47.040 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017022371000000001 2024-09-11T03:50:47.040 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.040 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.041 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:47.041 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.990945+0000", 2024-09-11T03:50:47.041 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016245699999999999 2024-09-11T03:50:47.041 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.041 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.041 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.041 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.992514+0000", 2024-09-11T03:50:47.041 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001569838 2024-09-11T03:50:47.041 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.041 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.041 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.041 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.041 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.042 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5349 2.1b 2:dde92fc6:::smithi17557870-41:head [read 0~2235463,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:50:47.042 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:50:47.042 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 256.02989046200003, 2024-09-11T03:50:47.042 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020826846, 2024-09-11T03:50:47.042 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.042 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.042 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:50:47.042 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.042 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.042 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.042 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5349 2024-09-11T03:50:47.042 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.042 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.043 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.043 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.043 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:50:47.043 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.043 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.043 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.043 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.043 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:50:47.043 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.043 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.043 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.043 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.043 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:50:47.043 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2099999999999996e-07 2024-09-11T03:50:47.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.044 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.044 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.044 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973242+0000", 2024-09-11T03:50:47.044 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0489999999999997e-06 2024-09-11T03:50:47.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.044 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.044 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.044 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973250+0000", 2024-09-11T03:50:47.044 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0700000000000007e-06 2024-09-11T03:50:47.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.044 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.044 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.045 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973256+0000", 2024-09-11T03:50:47.045 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3979999999999998e-06 2024-09-11T03:50:47.045 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.045 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.045 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.045 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973794+0000", 2024-09-11T03:50:47.045 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00053782700000000005 2024-09-11T03:50:47.045 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.045 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.045 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:50:47.045 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973801+0000", 2024-09-11T03:50:47.045 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5299999999999999e-06 2024-09-11T03:50:47.045 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.046 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.046 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.046 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.992813+0000", 2024-09-11T03:50:47.046 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019011917999999999 2024-09-11T03:50:47.046 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.046 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.046 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:47.046 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.992923+0000", 2024-09-11T03:50:47.046 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000109874 2024-09-11T03:50:47.046 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.046 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.046 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.046 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994064+0000", 2024-09-11T03:50:47.047 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001141459 2024-09-11T03:50:47.047 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.047 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.047 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.047 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.047 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.047 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5350 2.1b 2:dde92fc6:::smithi17557870-41:head [sparse-read 0~1317959] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:50:47.047 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973264+0000", 2024-09-11T03:50:47.047 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 256.02986374, 2024-09-11T03:50:47.047 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021144998000000002, 2024-09-11T03:50:47.047 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.047 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.047 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:50:47.048 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.048 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.048 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.048 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5350 2024-09-11T03:50:47.048 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.048 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.048 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.048 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.048 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973264+0000", 2024-09-11T03:50:47.048 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.048 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.049 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.049 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.049 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973264+0000", 2024-09-11T03:50:47.049 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.049 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.049 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.049 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.049 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973265+0000", 2024-09-11T03:50:47.049 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5899999999999996e-07 2024-09-11T03:50:47.049 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.049 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.049 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.049 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973268+0000", 2024-09-11T03:50:47.049 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6970000000000002e-06 2024-09-11T03:50:47.049 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.050 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.050 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.050 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973274+0000", 2024-09-11T03:50:47.050 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6390000000000003e-06 2024-09-11T03:50:47.050 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.050 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.050 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.050 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973280+0000", 2024-09-11T03:50:47.050 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.451e-06 2024-09-11T03:50:47.050 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.050 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.050 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.050 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973888+0000", 2024-09-11T03:50:47.050 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00060798999999999998 2024-09-11T03:50:47.050 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.051 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.051 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:50:47.051 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973895+0000", 2024-09-11T03:50:47.051 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6199999999999999e-06 2024-09-11T03:50:47.051 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.051 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.051 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.051 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994115+0000", 2024-09-11T03:50:47.051 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020219487000000001 2024-09-11T03:50:47.051 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.051 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.051 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:47.051 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994236+0000", 2024-09-11T03:50:47.051 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000120715 2024-09-11T03:50:47.051 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.052 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.052 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.052 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994409+0000", 2024-09-11T03:50:47.052 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017384 2024-09-11T03:50:47.052 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.052 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.052 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.052 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.052 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.052 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5362 2.1b 2:de6b1316:::smithi17557870-32:head [read 0~2117847,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:50:47.052 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973757+0000", 2024-09-11T03:50:47.052 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 256.02937083099999, 2024-09-11T03:50:47.052 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.023415378000000001, 2024-09-11T03:50:47.052 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.053 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.053 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:50:47.053 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.053 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.053 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.053 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5362 2024-09-11T03:50:47.053 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.053 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.053 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.053 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.053 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973757+0000", 2024-09-11T03:50:47.053 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.053 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.053 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.054 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.054 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973757+0000", 2024-09-11T03:50:47.054 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.054 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.054 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.054 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.054 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973758+0000", 2024-09-11T03:50:47.054 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.088e-06 2024-09-11T03:50:47.054 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.054 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.054 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.054 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973765+0000", 2024-09-11T03:50:47.054 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1310000000000003e-06 2024-09-11T03:50:47.054 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.054 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.055 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.055 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973776+0000", 2024-09-11T03:50:47.055 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1762e-05 2024-09-11T03:50:47.055 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.055 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.055 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.055 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973785+0000", 2024-09-11T03:50:47.055 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8629999999999997e-06 2024-09-11T03:50:47.055 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.055 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.055 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.055 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973985+0000", 2024-09-11T03:50:47.055 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000200051 2024-09-11T03:50:47.056 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.056 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.056 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:50:47.056 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973995+0000", 2024-09-11T03:50:47.056 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0054000000000001e-05 2024-09-11T03:50:47.056 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.056 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.056 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.056 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994538+0000", 2024-09-11T03:50:47.056 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020542878000000001 2024-09-11T03:50:47.056 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.056 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.056 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:47.057 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994729+0000", 2024-09-11T03:50:47.057 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00019097100000000001 2024-09-11T03:50:47.057 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.057 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.057 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.057 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.997173+0000", 2024-09-11T03:50:47.057 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024435799999999999 2024-09-11T03:50:47.057 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.057 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.057 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.057 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.057 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.057 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5363 2.1b 2:de6b1316:::smithi17557870-32:head [read 0~2117847,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:50:47.058 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973799+0000", 2024-09-11T03:50:47.058 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 256.02932882200002, 2024-09-11T03:50:47.058 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.024619513999999999, 2024-09-11T03:50:47.058 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.058 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.058 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:50:47.058 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.058 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.058 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.058 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5363 2024-09-11T03:50:47.058 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.058 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.058 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.059 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.059 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973799+0000", 2024-09-11T03:50:47.059 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.059 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.059 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.059 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.059 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973799+0000", 2024-09-11T03:50:47.059 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.059 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.059 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.059 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.059 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973800+0000", 2024-09-11T03:50:47.059 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2500000000000005e-07 2024-09-11T03:50:47.060 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.060 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.060 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.060 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973804+0000", 2024-09-11T03:50:47.060 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6049999999999998e-06 2024-09-11T03:50:47.060 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.060 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.060 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.060 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973811+0000", 2024-09-11T03:50:47.060 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0350000000000004e-06 2024-09-11T03:50:47.060 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.060 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.060 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.061 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973816+0000", 2024-09-11T03:50:47.061 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.536e-06 2024-09-11T03:50:47.061 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.061 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.061 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.061 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974024+0000", 2024-09-11T03:50:47.061 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00020766099999999999 2024-09-11T03:50:47.061 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.061 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.061 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:50:47.061 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974032+0000", 2024-09-11T03:50:47.061 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3340000000000002e-06 2024-09-11T03:50:47.061 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.061 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.062 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.062 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.997304+0000", 2024-09-11T03:50:47.062 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.02327183 2024-09-11T03:50:47.062 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.062 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.062 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:47.062 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.997455+0000", 2024-09-11T03:50:47.062 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015147000000000001 2024-09-11T03:50:47.062 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.062 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.062 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.062 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.998419+0000", 2024-09-11T03:50:47.062 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00096341799999999996 2024-09-11T03:50:47.062 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.063 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.063 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.063 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.063 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.063 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5364 2.1b 2:de6b1316:::smithi17557870-32:head [read 0~2117847,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:50:47.063 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:50:47.063 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 256.02930510200002, 2024-09-11T03:50:47.063 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.025348697, 2024-09-11T03:50:47.063 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.063 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.063 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:50:47.063 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.063 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.064 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.064 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5364 2024-09-11T03:50:47.064 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.064 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.064 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.064 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.064 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:50:47.064 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.064 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.065 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.065 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.065 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:50:47.065 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.065 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.065 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.065 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.065 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:50:47.065 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4099999999999999e-07 2024-09-11T03:50:47.065 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.065 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.065 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.065 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973826+0000", 2024-09-11T03:50:47.066 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7690000000000001e-06 2024-09-11T03:50:47.066 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.066 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.066 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.066 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973833+0000", 2024-09-11T03:50:47.066 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4679999999999997e-06 2024-09-11T03:50:47.066 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.066 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.066 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.066 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973837+0000", 2024-09-11T03:50:47.066 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6070000000000001e-06 2024-09-11T03:50:47.066 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.066 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.067 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.067 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974062+0000", 2024-09-11T03:50:47.067 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00022431400000000001 2024-09-11T03:50:47.067 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.067 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.067 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:50:47.067 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974069+0000", 2024-09-11T03:50:47.067 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8599999999999993e-06 2024-09-11T03:50:47.067 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.067 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.067 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.067 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.998467+0000", 2024-09-11T03:50:47.067 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.024397723999999999 2024-09-11T03:50:47.068 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.068 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.068 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:47.068 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.998581+0000", 2024-09-11T03:50:47.068 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000113769 2024-09-11T03:50:47.068 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.068 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.068 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.068 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.999172+0000", 2024-09-11T03:50:47.068 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00059074500000000005 2024-09-11T03:50:47.068 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.068 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.069 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.069 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.069 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.069 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5778 2.a 2:5014e305:::smithi17557870-46:head [copy-from ver 878] snapc ff=[ff,fe,fd] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-09-11T03:50:47.069 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:06.021580+0000", 2024-09-11T03:50:47.069 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 220.98154854200001, 2024-09-11T03:50:47.069 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.023197538, 2024-09-11T03:50:47.069 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.069 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.069 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:47.069 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.069 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.070 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.070 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5778 2024-09-11T03:50:47.070 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.070 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.070 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.070 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.070 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021580+0000", 2024-09-11T03:50:47.070 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.070 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.070 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.070 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.070 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021580+0000", 2024-09-11T03:50:47.070 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.071 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.071 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.071 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.071 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021581+0000", 2024-09-11T03:50:47.071 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6099999999999999e-07 2024-09-11T03:50:47.071 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.071 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.071 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.071 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021585+0000", 2024-09-11T03:50:47.071 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.279e-06 2024-09-11T03:50:47.071 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.071 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.071 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.071 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021593+0000", 2024-09-11T03:50:47.071 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1999999999999994e-06 2024-09-11T03:50:47.072 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.072 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.072 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.072 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021599+0000", 2024-09-11T03:50:47.072 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8200000000000002e-06 2024-09-11T03:50:47.072 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.072 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.072 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.072 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021691+0000", 2024-09-11T03:50:47.072 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2142e-05 2024-09-11T03:50:47.072 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.072 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.073 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:47.073 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021868+0000", 2024-09-11T03:50:47.073 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017677200000000001 2024-09-11T03:50:47.073 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.073 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.073 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:50:47.073 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.030689+0000", 2024-09-11T03:50:47.073 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0088215310000000005 2024-09-11T03:50:47.073 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.073 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.073 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:47.073 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.038924+0000", 2024-09-11T03:50:47.073 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.008234412 2024-09-11T03:50:47.073 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.074 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.074 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:47.074 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044669+0000", 2024-09-11T03:50:47.074 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0057455529999999996 2024-09-11T03:50:47.074 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.074 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.074 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:47.074 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044696+0000", 2024-09-11T03:50:47.074 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6843e-05 2024-09-11T03:50:47.074 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.074 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.074 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:47.075 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044748+0000", 2024-09-11T03:50:47.075 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1711e-05 2024-09-11T03:50:47.075 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.075 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.075 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.075 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044777+0000", 2024-09-11T03:50:47.075 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9314000000000001e-05 2024-09-11T03:50:47.075 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.075 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.075 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.075 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.075 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.075 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5983 2.1b 2:de6b1316:::smithi17557870-32:head [copy-from ver 692] snapc 108=[108,107] ondisk+write+known_if_redirected+supports_pool_eio e627)", 2024-09-11T03:50:47.076 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:25.010709+0000", 2024-09-11T03:50:47.076 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 201.992419442, 2024-09-11T03:50:47.076 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022123384999999999, 2024-09-11T03:50:47.076 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.076 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.076 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:47.076 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.076 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.076 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.076 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5983 2024-09-11T03:50:47.076 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.076 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.076 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.077 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.077 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010709+0000", 2024-09-11T03:50:47.077 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.077 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.077 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.077 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.077 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010709+0000", 2024-09-11T03:50:47.077 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.077 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.077 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.077 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.077 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010711+0000", 2024-09-11T03:50:47.077 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.367e-06 2024-09-11T03:50:47.078 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.078 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.078 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.078 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010723+0000", 2024-09-11T03:50:47.078 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2098e-05 2024-09-11T03:50:47.078 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.078 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.078 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.078 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010748+0000", 2024-09-11T03:50:47.078 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4641e-05 2024-09-11T03:50:47.078 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.078 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.078 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.079 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010761+0000", 2024-09-11T03:50:47.079 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.329e-05 2024-09-11T03:50:47.079 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.079 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.079 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.079 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010856+0000", 2024-09-11T03:50:47.079 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5278000000000005e-05 2024-09-11T03:50:47.079 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.079 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.079 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:50:47.079 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010862+0000", 2024-09-11T03:50:47.079 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.088e-06 2024-09-11T03:50:47.079 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.080 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.080 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.080 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.012360+0000", 2024-09-11T03:50:47.080 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014979940000000001 2024-09-11T03:50:47.080 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.080 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.080 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:47.080 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.012437+0000", 2024-09-11T03:50:47.080 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6959000000000006e-05 2024-09-11T03:50:47.080 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.080 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.080 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,9", 2024-09-11T03:50:47.080 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.020310+0000", 2024-09-11T03:50:47.081 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0078724099999999998 2024-09-11T03:50:47.081 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.081 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.081 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:47.081 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.028691+0000", 2024-09-11T03:50:47.081 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.008381299 2024-09-11T03:50:47.081 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.081 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.081 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:47.081 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.028800+0000", 2024-09-11T03:50:47.081 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000109082 2024-09-11T03:50:47.081 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.082 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.082 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:47.082 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.032710+0000", 2024-09-11T03:50:47.082 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039093660000000001 2024-09-11T03:50:47.082 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.082 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:47.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:47.083 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.083 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:47.083 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.032764+0000", 2024-09-11T03:50:47.083 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4415e-05 2024-09-11T03:50:47.084 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.084 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.084 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.084 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.032832+0000", 2024-09-11T03:50:47.084 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8097999999999998e-05 2024-09-11T03:50:47.084 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.084 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.084 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.084 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.084 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.084 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5984 2.1b 2:de6b1316:::smithi17557870-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e627)", 2024-09-11T03:50:47.084 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:25.010792+0000", 2024-09-11T03:50:47.084 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 201.992336005, 2024-09-11T03:50:47.085 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014851705, 2024-09-11T03:50:47.085 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.085 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.085 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:50:47.085 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.085 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.085 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.085 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5984 2024-09-11T03:50:47.085 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.085 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.085 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.085 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.085 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010792+0000", 2024-09-11T03:50:47.086 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.086 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.086 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.086 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.086 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010792+0000", 2024-09-11T03:50:47.086 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.086 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.086 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.086 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.086 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010793+0000", 2024-09-11T03:50:47.086 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8999999999999996e-07 2024-09-11T03:50:47.086 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.086 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.087 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.087 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010796+0000", 2024-09-11T03:50:47.087 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6200000000000001e-06 2024-09-11T03:50:47.087 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.087 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.087 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.087 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010811+0000", 2024-09-11T03:50:47.087 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4620999999999999e-05 2024-09-11T03:50:47.087 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.087 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.087 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.087 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010818+0000", 2024-09-11T03:50:47.088 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5490000000000003e-06 2024-09-11T03:50:47.088 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.088 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.088 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.088 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010926+0000", 2024-09-11T03:50:47.088 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108318 2024-09-11T03:50:47.088 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.088 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.088 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:50:47.088 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010931+0000", 2024-09-11T03:50:47.088 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1669999999999998e-06 2024-09-11T03:50:47.088 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.088 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.089 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.089 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.012597+0000", 2024-09-11T03:50:47.089 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001665487 2024-09-11T03:50:47.089 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.089 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.089 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:47.089 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.012646+0000", 2024-09-11T03:50:47.089 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9073000000000003e-05 2024-09-11T03:50:47.089 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.089 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.089 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.089 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.025501+0000", 2024-09-11T03:50:47.089 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012855263 2024-09-11T03:50:47.090 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.090 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.090 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:47.090 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.025569+0000", 2024-09-11T03:50:47.090 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8099e-05 2024-09-11T03:50:47.090 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.090 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.090 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.090 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.025644+0000", 2024-09-11T03:50:47.090 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4818000000000001e-05 2024-09-11T03:50:47.090 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.090 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.090 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.091 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.091 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.091 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6106 2.21 2:8638b65e:::smithi17557870-5:head [copy-from ver 603] snapc 10c=[10c] ondisk+write+known_if_redirected+supports_pool_eio e636)", 2024-09-11T03:50:47.091 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:34.018719+0000", 2024-09-11T03:50:47.091 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 192.98440889, 2024-09-11T03:50:47.091 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016460907, 2024-09-11T03:50:47.091 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.091 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.091 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:47.091 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.091 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.091 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.091 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6106 2024-09-11T03:50:47.092 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.092 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.092 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.092 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.092 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018719+0000", 2024-09-11T03:50:47.092 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.092 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.092 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.092 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.092 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018719+0000", 2024-09-11T03:50:47.092 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.092 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.092 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.092 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.093 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018722+0000", 2024-09-11T03:50:47.093 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6730000000000001e-06 2024-09-11T03:50:47.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.093 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.093 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018732+0000", 2024-09-11T03:50:47.093 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0349e-05 2024-09-11T03:50:47.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.093 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.093 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018756+0000", 2024-09-11T03:50:47.093 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3493999999999999e-05 2024-09-11T03:50:47.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.094 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.094 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018767+0000", 2024-09-11T03:50:47.094 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.079e-05 2024-09-11T03:50:47.094 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.094 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.094 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.094 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018875+0000", 2024-09-11T03:50:47.094 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108789 2024-09-11T03:50:47.094 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.094 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.094 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:50:47.094 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018884+0000", 2024-09-11T03:50:47.094 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4800000000000001e-06 2024-09-11T03:50:47.094 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.095 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.095 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.095 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.020397+0000", 2024-09-11T03:50:47.095 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015135770000000001 2024-09-11T03:50:47.095 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.095 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.095 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:47.095 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.020508+0000", 2024-09-11T03:50:47.095 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000110351 2024-09-11T03:50:47.095 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.095 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.095 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,8", 2024-09-11T03:50:47.095 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.028802+0000", 2024-09-11T03:50:47.095 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0082943189999999997 2024-09-11T03:50:47.096 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.096 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.096 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:47.096 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.033207+0000", 2024-09-11T03:50:47.096 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0044050820000000003 2024-09-11T03:50:47.096 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.096 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.096 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:47.096 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.033684+0000", 2024-09-11T03:50:47.096 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00047677899999999998 2024-09-11T03:50:47.096 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.096 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.096 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:47.096 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.035101+0000", 2024-09-11T03:50:47.097 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014171539999999999 2024-09-11T03:50:47.097 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.097 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.097 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:47.097 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.035147+0000", 2024-09-11T03:50:47.097 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5731e-05 2024-09-11T03:50:47.097 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.097 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.097 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.097 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.035180+0000", 2024-09-11T03:50:47.097 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3339e-05 2024-09-11T03:50:47.097 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.097 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.098 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.098 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.098 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.098 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7115 2.c 2:3603e322:::smithi17557870-42:head [copy-from ver 930] snapc 13b=[13b,13a,139,138,137,136,135,133,12e,12d,12c,12a,127,126,120,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e713)", 2024-09-11T03:50:47.098 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:51.194110+0000", 2024-09-11T03:50:47.098 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 115.80901780400001, 2024-09-11T03:50:47.098 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.029033742000000001, 2024-09-11T03:50:47.098 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.098 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.098 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:47.098 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.098 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.099 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.099 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7115 2024-09-11T03:50:47.099 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.099 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.099 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.099 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.099 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194110+0000", 2024-09-11T03:50:47.099 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.099 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.099 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.099 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.099 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194110+0000", 2024-09-11T03:50:47.099 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.099 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.100 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.100 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.100 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194111+0000", 2024-09-11T03:50:47.100 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3200000000000005e-07 2024-09-11T03:50:47.100 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.100 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.100 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.100 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194115+0000", 2024-09-11T03:50:47.100 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5980000000000001e-06 2024-09-11T03:50:47.100 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.100 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.100 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.100 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194121+0000", 2024-09-11T03:50:47.101 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3289999999999999e-06 2024-09-11T03:50:47.101 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.101 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.101 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.101 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194126+0000", 2024-09-11T03:50:47.101 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4e-06 2024-09-11T03:50:47.101 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.101 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.101 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.101 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194207+0000", 2024-09-11T03:50:47.101 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0721000000000002e-05 2024-09-11T03:50:47.101 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.101 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.102 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:47.102 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194303+0000", 2024-09-11T03:50:47.102 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5546999999999998e-05 2024-09-11T03:50:47.102 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.102 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.102 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:50:47.102 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.207212+0000", 2024-09-11T03:50:47.102 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012909132 2024-09-11T03:50:47.102 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.102 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.102 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:47.102 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.209890+0000", 2024-09-11T03:50:47.102 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002677876 2024-09-11T03:50:47.103 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.103 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.103 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:47.103 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223023+0000", 2024-09-11T03:50:47.103 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013133137 2024-09-11T03:50:47.103 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.103 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.103 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:47.103 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223063+0000", 2024-09-11T03:50:47.103 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0413000000000001e-05 2024-09-11T03:50:47.103 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.103 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.103 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:47.103 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223107+0000", 2024-09-11T03:50:47.104 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3569999999999998e-05 2024-09-11T03:50:47.104 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.104 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.104 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.104 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223144+0000", 2024-09-11T03:50:47.104 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7387e-05 2024-09-11T03:50:47.104 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.104 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.104 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.104 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.104 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.104 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7116 2.c 2:3603e322:::smithi17557870-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e713)", 2024-09-11T03:50:47.104 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:51.194143+0000", 2024-09-11T03:50:47.105 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 115.808985236, 2024-09-11T03:50:47.105 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015378424, 2024-09-11T03:50:47.105 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.105 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.105 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:50:47.105 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.105 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.105 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.105 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7116 2024-09-11T03:50:47.105 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.105 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.105 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.105 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.105 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194143+0000", 2024-09-11T03:50:47.106 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.106 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.106 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.106 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.106 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194143+0000", 2024-09-11T03:50:47.106 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.106 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.106 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.106 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.106 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194143+0000", 2024-09-11T03:50:47.106 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9299999999999998e-07 2024-09-11T03:50:47.106 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.106 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.107 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.107 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194146+0000", 2024-09-11T03:50:47.107 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2390000000000001e-06 2024-09-11T03:50:47.107 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.107 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.107 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.107 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194152+0000", 2024-09-11T03:50:47.107 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5080000000000002e-06 2024-09-11T03:50:47.107 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.107 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.107 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.107 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194158+0000", 2024-09-11T03:50:47.107 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.942e-06 2024-09-11T03:50:47.108 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.108 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.108 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.108 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194461+0000", 2024-09-11T03:50:47.108 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00030244899999999999 2024-09-11T03:50:47.108 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.108 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.108 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:47.108 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194502+0000", 2024-09-11T03:50:47.108 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.1736000000000002e-05 2024-09-11T03:50:47.108 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.108 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.108 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.108 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.209298+0000", 2024-09-11T03:50:47.109 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014795338999999999 2024-09-11T03:50:47.109 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.109 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.109 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:47.109 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.209398+0000", 2024-09-11T03:50:47.109 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010077300000000001 2024-09-11T03:50:47.109 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.109 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.109 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.109 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.209521+0000", 2024-09-11T03:50:47.109 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012294500000000001 2024-09-11T03:50:47.109 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.109 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.110 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.110 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.110 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.110 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7489 2.1b 2:d8055656:::smithi17557870-39:head [write 511210~414125,stat] snapc 153=[153,151,150,14f,14e,14d,14b,149,148,144,143,142,13d,13c,139,136,135,133,12a,127,120,119] ondisk+write+known_if_redirected+supports_pool_eio e757)", 2024-09-11T03:50:47.110 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:49:35.200276+0000", 2024-09-11T03:50:47.110 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 71.80285164, 2024-09-11T03:50:47.110 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.006934618, 2024-09-11T03:50:47.110 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.110 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.110 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:47.110 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.110 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.110 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.110 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7489 2024-09-11T03:50:47.111 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.111 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.111 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.111 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.111 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:35.200276+0000", 2024-09-11T03:50:47.111 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.111 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.111 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.111 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.111 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:35.200276+0000", 2024-09-11T03:50:47.111 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.111 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.111 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.112 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.112 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:35.200278+0000", 2024-09-11T03:50:47.112 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.978e-06 2024-09-11T03:50:47.112 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.112 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.112 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.112 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:35.200577+0000", 2024-09-11T03:50:47.112 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000298728 2024-09-11T03:50:47.112 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.112 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.112 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.112 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:35.200589+0000", 2024-09-11T03:50:47.112 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1885000000000001e-05 2024-09-11T03:50:47.113 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.113 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.113 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.113 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:35.200597+0000", 2024-09-11T03:50:47.113 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1210000000000007e-06 2024-09-11T03:50:47.113 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.113 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.113 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.113 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:35.202622+0000", 2024-09-11T03:50:47.113 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0020243409999999998 2024-09-11T03:50:47.113 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.113 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.113 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:47.114 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:35.202752+0000", 2024-09-11T03:50:47.114 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013025299999999999 2024-09-11T03:50:47.114 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.114 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.114 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,9", 2024-09-11T03:50:47.114 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:35.202976+0000", 2024-09-11T03:50:47.114 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00022468299999999999 2024-09-11T03:50:47.114 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.114 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.114 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:50:47.114 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:35.206144+0000", 2024-09-11T03:50:47.114 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0031678729999999999 2024-09-11T03:50:47.114 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.115 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.115 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:47.115 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:35.207084+0000", 2024-09-11T03:50:47.115 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00093932 2024-09-11T03:50:47.115 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.115 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.115 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:47.115 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:35.207110+0000", 2024-09-11T03:50:47.115 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6224000000000002e-05 2024-09-11T03:50:47.115 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.115 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.115 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:47.115 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:35.207163+0000", 2024-09-11T03:50:47.116 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2725000000000001e-05 2024-09-11T03:50:47.116 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.116 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.116 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.116 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:35.207211+0000", 2024-09-11T03:50:47.116 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8486999999999998e-05 2024-09-11T03:50:47.116 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.116 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.116 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.116 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.116 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.116 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:7865 2.16 e779/560 2:6dbd1e81:::smithi17557870-7:head v 779'557, mlcod=778'553)", 2024-09-11T03:50:47.117 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:49:57.142873+0000", 2024-09-11T03:50:47.117 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 49.860255537, 2024-09-11T03:50:47.117 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0062625620000000002, 2024-09-11T03:50:47.117 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.117 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.117 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:47.117 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.117 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.117 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.117 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.142873+0000", 2024-09-11T03:50:47.117 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.117 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.118 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.118 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.118 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.142873+0000", 2024-09-11T03:50:47.118 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.118 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.118 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.118 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.118 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.142873+0000", 2024-09-11T03:50:47.118 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8500000000000001e-07 2024-09-11T03:50:47.118 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.118 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.118 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.118 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.147210+0000", 2024-09-11T03:50:47.119 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.004337095 2024-09-11T03:50:47.119 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.119 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.119 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.119 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.147220+0000", 2024-09-11T03:50:47.119 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4530000000000008e-06 2024-09-11T03:50:47.119 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.119 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.119 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.119 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.147224+0000", 2024-09-11T03:50:47.119 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4769999999999997e-06 2024-09-11T03:50:47.119 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.119 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.119 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.120 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.147267+0000", 2024-09-11T03:50:47.120 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3127999999999998e-05 2024-09-11T03:50:47.120 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.120 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.120 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:47.120 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.147291+0000", 2024-09-11T03:50:47.120 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3230999999999998e-05 2024-09-11T03:50:47.120 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.120 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.120 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:50:47.120 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.149102+0000", 2024-09-11T03:50:47.120 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001811113 2024-09-11T03:50:47.120 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.120 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.121 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.121 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.149135+0000", 2024-09-11T03:50:47.121 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3380000000000002e-05 2024-09-11T03:50:47.121 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.121 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.121 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.121 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.121 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.121 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:8201 2.1b 2:dde92fc6:::smithi17557870-41:head [delete] snapc 16c=[16c,16b,169,168,167,164,162,160,15f,15e,15d,15b,15a,157,156,155,151,14d,148,144,143,13c,139,135,12a,127,120] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e803)", 2024-09-11T03:50:47.121 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:50:21.175869+0000", 2024-09-11T03:50:47.121 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 25.827259569999999, 2024-09-11T03:50:47.121 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.006831223, 2024-09-11T03:50:47.122 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:50:47.122 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:50:47.122 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:50:47.122 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:50:47.122 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:50:47.122 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:47.122 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 8201 2024-09-11T03:50:47.122 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.122 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:50:47.122 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.122 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:50:47.122 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:21.175869+0000", 2024-09-11T03:50:47.122 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.122 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.123 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.123 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:50:47.123 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:21.175869+0000", 2024-09-11T03:50:47.123 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:50:47.123 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.123 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.123 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:50:47.123 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:21.175871+0000", 2024-09-11T03:50:47.123 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4399999999999999e-06 2024-09-11T03:50:47.123 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.123 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.123 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:50:47.123 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:21.175880+0000", 2024-09-11T03:50:47.124 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1929999999999999e-06 2024-09-11T03:50:47.124 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.124 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.124 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:50:47.124 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:21.175900+0000", 2024-09-11T03:50:47.124 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9446e-05 2024-09-11T03:50:47.124 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.124 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.124 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:50:47.124 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:21.175910+0000", 2024-09-11T03:50:47.124 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9520000000000006e-06 2024-09-11T03:50:47.124 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.124 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.124 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.125 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:21.175972+0000", 2024-09-11T03:50:47.125 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2044000000000004e-05 2024-09-11T03:50:47.125 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.125 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.125 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:50:47.125 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:21.175976+0000", 2024-09-11T03:50:47.125 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.746e-06 2024-09-11T03:50:47.125 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.125 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.125 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:50:47.125 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:21.180678+0000", 2024-09-11T03:50:47.125 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0047017409999999997 2024-09-11T03:50:47.125 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.125 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.125 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:50:47.126 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:21.180787+0000", 2024-09-11T03:50:47.126 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108732 2024-09-11T03:50:47.126 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:50:47.126 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:50:47.126 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:50:47.126 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:21.182700+0000", 2024-09-11T03:50:47.126 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0019129290000000001 2024-09-11T03:50:47.126 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.126 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.126 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.126 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:50:47.126 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:50:47.126 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:50:47.127 INFO:tasks.rados.rados.2.smithi175.stdout:3422: done (3 left) 2024-09-11T03:50:47.127 INFO:tasks.rados.rados.2.smithi175.stdout:3424: done (2 left) 2024-09-11T03:50:47.127 INFO:tasks.rados.rados.2.smithi175.stdout:3425: done (1 left) 2024-09-11T03:50:47.127 INFO:tasks.rados.rados.2.smithi175.stdout:3426: done (0 left) 2024-09-11T03:50:47.127 INFO:tasks.rados.rados.2.smithi175.stdout:3427: delete oid 49 current snap is 380 2024-09-11T03:50:47.127 INFO:tasks.rados.rados.2.smithi175.stdout:3427: done (0 left) 2024-09-11T03:50:47.127 INFO:tasks.rados.rados.2.smithi175.stdout:3428: snap_remove snap 369 2024-09-11T03:50:47.183 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:47.204 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:47.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:47.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:47.306 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:47.327 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:47.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:47.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:47.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:47.456 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:47.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:47.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:47.558 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:47.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:47.581 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:47.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:47.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:47.708 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:47.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:47.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:47.801 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T03:50:47.810 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:47.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:47.831 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:47.831 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:47.932 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:47.953 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:47.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:48.044 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:50:48.045 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:48.048 INFO:tasks.rados.rados.2.smithi175.stdout:3428: done (0 left) 2024-09-11T03:50:48.048 INFO:tasks.rados.rados.2.smithi175.stdout:3429: read oid 7 snap -1 2024-09-11T03:50:48.049 INFO:tasks.rados.rados.2.smithi175.stdout:3429: expect (ObjNum 1122 snap 360 seq_num 1122) 2024-09-11T03:50:48.049 INFO:tasks.rados.rados.2.smithi175.stdout:3430: delete oid 49 current snap is 380 2024-09-11T03:50:48.055 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:48.056 INFO:tasks.rados.rados.2.smithi175.stdout:3430: done (1 left) 2024-09-11T03:50:48.056 INFO:tasks.rados.rados.2.smithi175.stdout:3431: rollback oid 13 current snap is 380 2024-09-11T03:50:48.056 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 13 to 311 2024-09-11T03:50:48.056 INFO:tasks.rados.rados.2.smithi175.stdout:3432: snap_create 2024-09-11T03:50:48.063 INFO:tasks.rados.rados.2.smithi175.stdout:3431: finishing rollback tid 0 to smithi17557870-13 2024-09-11T03:50:48.063 INFO:tasks.rados.rados.2.smithi175.stdout:3431: finishing rollback tid 1 to smithi17557870-13 2024-09-11T03:50:48.063 INFO:tasks.rados.rados.2.smithi175.stdout:3431: finishing rollback tid 2 to smithi17557870-13 2024-09-11T03:50:48.063 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 1316 (ObjNum 907 snap 287 seq_num 907) dirty exists 2024-09-11T03:50:48.066 INFO:tasks.rados.rados.2.smithi175.stdout:3429: expect (ObjNum 1122 snap 360 seq_num 1122) 2024-09-11T03:50:48.079 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:48.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:48.080 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:48.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:48 smithi084 ceph-mon[27254]: pgmap v1398: 97 pgs: 97 active+clean; 602 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 3.9 MiB/s wr, 6 op/s 2024-09-11T03:50:48.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:48 smithi084 ceph-mon[27254]: osdmap e829: 12 total, 12 up, 12 in 2024-09-11T03:50:48.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:48 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2441689076' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:48.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:48.207 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:48.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:48.226 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:50:48.309 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:48.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:48.331 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:48.331 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:48.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:48 smithi121 ceph-mon[30729]: pgmap v1398: 97 pgs: 97 active+clean; 602 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 3.9 MiB/s wr, 6 op/s 2024-09-11T03:50:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:48 smithi121 ceph-mon[30729]: osdmap e829: 12 total, 12 up, 12 in 2024-09-11T03:50:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:48 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2441689076' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:48.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:48.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:48.461 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:48.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:48 smithi175 ceph-mon[30894]: pgmap v1398: 97 pgs: 97 active+clean; 602 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 3.9 MiB/s wr, 6 op/s 2024-09-11T03:50:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:48 smithi175 ceph-mon[30894]: osdmap e829: 12 total, 12 up, 12 in 2024-09-11T03:50:48.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:48 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2441689076' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:50:48.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:48.588 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:48.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:48.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:48.689 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:48.714 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:48.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:48.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:48.816 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:48.839 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:48.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:48.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:48.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:48.967 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:48.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:48.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:49.061 INFO:tasks.rados.rados.2.smithi175.stdout:3429: done (2 left) 2024-09-11T03:50:49.061 INFO:tasks.rados.rados.2.smithi175.stdout:3431: done (1 left) 2024-09-11T03:50:49.061 INFO:tasks.rados.rados.2.smithi175.stdout:3432: done (0 left) 2024-09-11T03:50:49.061 INFO:tasks.rados.rados.2.smithi175.stdout:3433: rollback oid 32 current snap is 381 2024-09-11T03:50:49.062 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 32 to 321 2024-09-11T03:50:49.062 INFO:tasks.rados.rados.2.smithi175.stdout:3434: snap_create 2024-09-11T03:50:49.065 INFO:tasks.rados.rados.2.smithi175.stdout:3433: finishing rollback tid 0 to smithi17557870-32 2024-09-11T03:50:49.065 INFO:tasks.rados.rados.2.smithi175.stdout:3433: finishing rollback tid 1 to smithi17557870-32 2024-09-11T03:50:49.065 INFO:tasks.rados.rados.2.smithi175.stdout:3433: finishing rollback tid 2 to smithi17557870-32 2024-09-11T03:50:49.065 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 1017 (ObjNum 984 snap 315 seq_num 984) dirty exists 2024-09-11T03:50:49.072 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:49.094 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:49.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:49.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:49.196 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:49.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:49.218 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:49.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:49.319 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:49.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2441689076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:49.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:49 smithi084 ceph-mon[27254]: osdmap e830: 12 total, 12 up, 12 in 2024-09-11T03:50:49.341 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:49.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:49.353 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:50:49.353 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:50:49.353 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:50:49.353 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:50:49.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2441689076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:49.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:49 smithi121 ceph-mon[30729]: osdmap e830: 12 total, 12 up, 12 in 2024-09-11T03:50:49.442 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:49.463 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:49.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2441689076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:50:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:49 smithi175 ceph-mon[30894]: osdmap e830: 12 total, 12 up, 12 in 2024-09-11T03:50:49.565 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:49.592 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:49.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:49.594 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:49.694 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:49.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:49.721 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:49.721 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:49.822 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:49.848 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:49.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:49.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:49.951 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:49.972 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:49.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:49.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:50.063 INFO:tasks.rados.rados.2.smithi175.stdout:3433: done (1 left) 2024-09-11T03:50:50.063 INFO:tasks.rados.rados.2.smithi175.stdout:3434: done (0 left) 2024-09-11T03:50:50.063 INFO:tasks.rados.rados.2.smithi175.stdout:3435: rollback oid 43 current snap is 382 2024-09-11T03:50:50.063 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 43 to 322 2024-09-11T03:50:50.063 INFO:tasks.rados.rados.2.smithi175.stdout:3436: read oid 47 snap -1 2024-09-11T03:50:50.064 INFO:tasks.rados.rados.2.smithi175.stdout:3436: expect (ObjNum 1132 snap 367 seq_num 1132) 2024-09-11T03:50:50.064 INFO:tasks.rados.rados.2.smithi175.stdout:3437: delete oid 20 current snap is 382 2024-09-11T03:50:50.070 INFO:tasks.rados.rados.2.smithi175.stdout:3437: done (2 left) 2024-09-11T03:50:50.070 INFO:tasks.rados.rados.2.smithi175.stdout:3438: delete oid 37 current snap is 382 2024-09-11T03:50:50.070 INFO:tasks.rados.rados.2.smithi175.stdout:3435: finishing rollback tid 1 to smithi17557870-43 2024-09-11T03:50:50.070 INFO:tasks.rados.rados.2.smithi175.stdout:3435: finishing rollback tid 2 to smithi17557870-43 2024-09-11T03:50:50.070 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 1013 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:50:50.074 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:50.075 INFO:tasks.rados.rados.2.smithi175.stdout:3435: done (2 left) 2024-09-11T03:50:50.075 INFO:tasks.rados.rados.2.smithi175.stdout:3438: done (1 left) 2024-09-11T03:50:50.075 INFO:tasks.rados.rados.2.smithi175.stdout:3439: write oid 14 current snap is 382 2024-09-11T03:50:50.075 INFO:tasks.rados.rados.2.smithi175.stdout:3439: seq_num 1165 ranges {780155=667094,1972934=694072,3309316=270839} 2024-09-11T03:50:50.082 INFO:tasks.rados.rados.2.smithi175.stdout:3439: writing smithi17557870-14 from 780155 to 1447249 tid 1 2024-09-11T03:50:50.089 INFO:tasks.rados.rados.2.smithi175.stdout:3439: writing smithi17557870-14 from 1972934 to 2667006 tid 2 2024-09-11T03:50:50.089 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T03:50:50.094 INFO:tasks.rados.rados.2.smithi175.stdout:3439: writing smithi17557870-14 from 3309316 to 3580155 tid 3 2024-09-11T03:50:50.094 INFO:tasks.rados.rados.2.smithi175.stdout:3440: read oid 44 snap -1 2024-09-11T03:50:50.095 INFO:tasks.rados.rados.2.smithi175.stdout:3440: expect (ObjNum 939 snap 297 seq_num 939) 2024-09-11T03:50:50.095 INFO:tasks.rados.rados.2.smithi175.stdout:3441: read oid 19 snap 364 2024-09-11T03:50:50.095 INFO:tasks.rados.rados.2.smithi175.stdout:3441: expect (ObjNum 963 snap 310 seq_num 963) 2024-09-11T03:50:50.095 INFO:tasks.rados.rados.2.smithi175.stdout:3442: read oid 16 snap -1 2024-09-11T03:50:50.095 INFO:tasks.rados.rados.2.smithi175.stdout:3442: expect deleted 2024-09-11T03:50:50.095 INFO:tasks.rados.rados.2.smithi175.stdout:3443: delete oid 49 current snap is 382 2024-09-11T03:50:50.095 INFO:tasks.rados.rados.2.smithi175.stdout:3436: expect (ObjNum 1132 snap 367 seq_num 1132) 2024-09-11T03:50:50.099 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:50.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:50.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:50.102 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:50.103 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:50:50.103 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:50:50.104 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-11T03:50:50.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:50:50.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:50.224 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:50.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:50.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:50.263 INFO:tasks.rados.rados.2.smithi175.stdout:3439: finishing write tid 1 to smithi17557870-14 2024-09-11T03:50:50.263 INFO:tasks.rados.rados.2.smithi175.stdout:3439: finishing write tid 2 to smithi17557870-14 2024-09-11T03:50:50.263 INFO:tasks.rados.rados.2.smithi175.stdout:3439: finishing write tid 3 to smithi17557870-14 2024-09-11T03:50:50.263 INFO:tasks.rados.rados.2.smithi175.stdout:3439: finishing write tid 4 to smithi17557870-14 2024-09-11T03:50:50.263 INFO:tasks.rados.rados.2.smithi175.stdout:3439: finishing write tid 5 to smithi17557870-14 2024-09-11T03:50:50.263 INFO:tasks.rados.rados.2.smithi175.stdout:3439: oid 14 updating version 0 to 1069 2024-09-11T03:50:50.264 INFO:tasks.rados.rados.2.smithi175.stdout:3439: oid 14 updating version 1069 to 1072 2024-09-11T03:50:50.264 INFO:tasks.rados.rados.2.smithi175.stdout:3439: oid 14 version 1072 is already newer than 1070 2024-09-11T03:50:50.264 INFO:tasks.rados.rados.2.smithi175.stdout:3439: oid 14 version 1072 is already newer than 1071 2024-09-11T03:50:50.264 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 1072 (ObjNum 1165 snap 382 seq_num 1165) dirty exists 2024-09-11T03:50:50.264 INFO:tasks.rados.rados.2.smithi175.stdout:3439: left oid 14 (ObjNum 1165 snap 382 seq_num 1165) 2024-09-11T03:50:50.264 INFO:tasks.rados.rados.2.smithi175.stdout:3440: expect (ObjNum 939 snap 297 seq_num 939) 2024-09-11T03:50:50.325 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:50.346 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:50.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:50.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:50.383 INFO:tasks.rados.rados.2.smithi175.stdout:3441: expect (ObjNum 963 snap 310 seq_num 963) 2024-09-11T03:50:50.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:50 smithi084 ceph-mon[27254]: pgmap v1401: 97 pgs: 97 active+clean; 607 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4.8 MiB/s wr, 10 op/s 2024-09-11T03:50:50.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:50 smithi084 ceph-mon[27254]: osdmap e831: 12 total, 12 up, 12 in 2024-09-11T03:50:50.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:50 smithi121 ceph-mon[30729]: pgmap v1401: 97 pgs: 97 active+clean; 607 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4.8 MiB/s wr, 10 op/s 2024-09-11T03:50:50.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:50 smithi121 ceph-mon[30729]: osdmap e831: 12 total, 12 up, 12 in 2024-09-11T03:50:50.448 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:50.469 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:50.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:50.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:50.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:50 smithi175 ceph-mon[30894]: pgmap v1401: 97 pgs: 97 active+clean; 607 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4.8 MiB/s wr, 10 op/s 2024-09-11T03:50:50.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:50 smithi175 ceph-mon[30894]: osdmap e831: 12 total, 12 up, 12 in 2024-09-11T03:50:50.505 INFO:tasks.rados.rados.2.smithi175.stdout:3436: done (5 left) 2024-09-11T03:50:50.505 INFO:tasks.rados.rados.2.smithi175.stdout:3439: done (4 left) 2024-09-11T03:50:50.505 INFO:tasks.rados.rados.2.smithi175.stdout:3440: done (3 left) 2024-09-11T03:50:50.505 INFO:tasks.rados.rados.2.smithi175.stdout:3441: done (2 left) 2024-09-11T03:50:50.505 INFO:tasks.rados.rados.2.smithi175.stdout:3442: done (1 left) 2024-09-11T03:50:50.505 INFO:tasks.rados.rados.2.smithi175.stdout:3443: done (0 left) 2024-09-11T03:50:50.505 INFO:tasks.rados.rados.2.smithi175.stdout:3444: snap_remove snap 377 2024-09-11T03:50:50.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:50.592 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:50:50.592 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:50:50.597 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:50.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:50.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:50.607 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:50:50.607 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:50:50.626 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:50:50.627 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:50:50.637 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:50.645 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:50:50.645 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:50:50.646 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:50.665 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:50:50.665 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:50:50.683 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:50:50.683 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:50:50.698 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:50.700 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:50:50.700 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:50:50.717 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:50:50.717 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:50:50.719 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:50.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:50.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:50.734 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:50:50.734 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:50:50.750 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:50:50.751 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:50:50.769 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:50:50.769 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:50:50.785 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:50:50.786 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:50:50.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:50.842 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:50.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:50.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:50.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:50.965 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:50.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:50.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:51.067 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:51.074 INFO:tasks.rados.rados.2.smithi175.stdout:3444: done (0 left) 2024-09-11T03:50:51.074 INFO:tasks.rados.rados.2.smithi175.stdout:3445: copy_from oid 26 from oid 30 current snap is 382 2024-09-11T03:50:51.074 INFO:tasks.rados.rados.2.smithi175.stdout:3446: write (excl) oid 18 current snap is 382 2024-09-11T03:50:51.074 INFO:tasks.rados.rados.2.smithi175.stdout:3446: seq_num 1167 ranges {413769=417592,1573545=697741,2847809=624439} 2024-09-11T03:50:51.079 INFO:tasks.rados.rados.2.smithi175.stdout:3446: writing smithi17557870-18 from 413769 to 831361 tid 1 2024-09-11T03:50:51.087 INFO:tasks.rados.rados.2.smithi175.stdout:3446: writing smithi17557870-18 from 1573545 to 2271286 tid 2 2024-09-11T03:50:51.093 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:51.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:51.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:51.095 INFO:tasks.rados.rados.2.smithi175.stdout:3446: writing smithi17557870-18 from 2847809 to 3472248 tid 3 2024-09-11T03:50:51.095 INFO:tasks.rados.rados.2.smithi175.stdout:3447: snap_create 2024-09-11T03:50:51.095 INFO:tasks.rados.rados.2.smithi175.stdout:3446: finishing write tid 1 to smithi17557870-18 2024-09-11T03:50:51.095 INFO:tasks.rados.rados.2.smithi175.stdout:3446: finishing write tid 2 to smithi17557870-18 2024-09-11T03:50:51.095 INFO:tasks.rados.rados.2.smithi175.stdout:3445: finishing copy_from racing read to smithi17557870-26 2024-09-11T03:50:51.098 INFO:tasks.rados.rados.2.smithi175.stdout:3446: finishing write tid 3 to smithi17557870-18 2024-09-11T03:50:51.098 INFO:tasks.rados.rados.2.smithi175.stdout:3445: finishing copy_from to smithi17557870-26 2024-09-11T03:50:51.098 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 1073 (ObjNum 1147 snap 370 seq_num 1147) dirty exists 2024-09-11T03:50:51.098 INFO:tasks.rados.rados.2.smithi175.stdout:3446: finishing write tid 4 to smithi17557870-18 2024-09-11T03:50:51.098 INFO:tasks.rados.rados.2.smithi175.stdout:3446: finishing write tid 5 to smithi17557870-18 2024-09-11T03:50:51.098 INFO:tasks.rados.rados.2.smithi175.stdout:3446: oid 18 updating version 0 to 938 2024-09-11T03:50:51.098 INFO:tasks.rados.rados.2.smithi175.stdout:3446: oid 18 updating version 938 to 939 2024-09-11T03:50:51.098 INFO:tasks.rados.rados.2.smithi175.stdout:3446: oid 18 updating version 939 to 940 2024-09-11T03:50:51.099 INFO:tasks.rados.rados.2.smithi175.stdout:3446: oid 18 updating version 940 to 941 2024-09-11T03:50:51.099 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 941 (ObjNum 1167 snap 382 seq_num 1167) dirty exists 2024-09-11T03:50:51.099 INFO:tasks.rados.rados.2.smithi175.stdout:3446: left oid 18 (ObjNum 1167 snap 382 seq_num 1167) 2024-09-11T03:50:51.119 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:50:51.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:51.219 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:51.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:51.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:51.321 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:51.343 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:51.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:51.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:51 smithi084 ceph-mon[27254]: osdmap e832: 12 total, 12 up, 12 in 2024-09-11T03:50:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1466695909' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:51 smithi121 ceph-mon[30729]: osdmap e832: 12 total, 12 up, 12 in 2024-09-11T03:50:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1466695909' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:51.445 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:51.465 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:50:51.466 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:50:51.469 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:51.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:51.470 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:51.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:51 smithi175 ceph-mon[30894]: osdmap e832: 12 total, 12 up, 12 in 2024-09-11T03:50:51.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1466695909' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:51.571 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:51.593 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:51.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:51.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:51.695 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:51.721 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:51.722 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:51.823 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:51.847 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:51.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:51.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:51.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:51.975 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:51.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:51.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:52.041 INFO:tasks.rados.rados.2.smithi175.stdout:3445: done (2 left) 2024-09-11T03:50:52.041 INFO:tasks.rados.rados.2.smithi175.stdout:3446: done (1 left) 2024-09-11T03:50:52.041 INFO:tasks.rados.rados.2.smithi175.stdout:3447: done (0 left) 2024-09-11T03:50:52.042 INFO:tasks.rados.rados.2.smithi175.stdout:3448: write (excl) oid 18 current snap is 383 2024-09-11T03:50:52.042 INFO:tasks.rados.rados.2.smithi175.stdout:3448: seq_num 1168 ranges {430576=492841,1632027=441399,2475232=435711,3365397=265179} 2024-09-11T03:50:52.046 INFO:tasks.rados.rados.2.smithi175.stdout:3448: writing smithi17557870-18 from 430576 to 923417 tid 1 2024-09-11T03:50:52.052 INFO:tasks.rados.rados.2.smithi175.stdout:3448: writing smithi17557870-18 from 1632027 to 2073426 tid 2 2024-09-11T03:50:52.057 INFO:tasks.rados.rados.2.smithi175.stdout:3448: writing smithi17557870-18 from 2475232 to 2910943 tid 3 2024-09-11T03:50:52.061 INFO:tasks.rados.rados.2.smithi175.stdout:3448: writing smithi17557870-18 from 3365397 to 3630576 tid 4 2024-09-11T03:50:52.061 INFO:tasks.rados.rados.2.smithi175.stdout:3449: snap_remove snap 358 2024-09-11T03:50:52.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:52.099 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:52.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:52.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:52.202 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:52.228 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:52.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:52.229 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:52.331 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:52.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:52.353 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:52.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:52 smithi084 ceph-mon[27254]: pgmap v1404: 97 pgs: 97 active+clean; 607 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 950 KiB/s wr, 3 op/s 2024-09-11T03:50:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:52 smithi084 ceph-mon[27254]: osdmap e833: 12 total, 12 up, 12 in 2024-09-11T03:50:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:52 smithi084 ceph-mon[27254]: from='client.17226 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:50:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:52 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:52 smithi084 ceph-mon[27254]: osdmap e834: 12 total, 12 up, 12 in 2024-09-11T03:50:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:52 smithi121 ceph-mon[30729]: pgmap v1404: 97 pgs: 97 active+clean; 607 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 950 KiB/s wr, 3 op/s 2024-09-11T03:50:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:52 smithi121 ceph-mon[30729]: osdmap e833: 12 total, 12 up, 12 in 2024-09-11T03:50:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:52 smithi121 ceph-mon[30729]: from='client.17226 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:50:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:52 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:52 smithi121 ceph-mon[30729]: osdmap e834: 12 total, 12 up, 12 in 2024-09-11T03:50:52.455 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:52 smithi175 ceph-mon[30894]: pgmap v1404: 97 pgs: 97 active+clean; 607 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 950 KiB/s wr, 3 op/s 2024-09-11T03:50:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:52 smithi175 ceph-mon[30894]: osdmap e833: 12 total, 12 up, 12 in 2024-09-11T03:50:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:52 smithi175 ceph-mon[30894]: from='client.17226 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:50:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:52 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:52 smithi175 ceph-mon[30894]: osdmap e834: 12 total, 12 up, 12 in 2024-09-11T03:50:52.481 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:52.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:52.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:52.583 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:52.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:52.605 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:52.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:52.642 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1405,"stamp":"2024-09-11T03:50:51.070415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":638463460,"num_objects":291,"num_object_clones":239,"num_object_copies":873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":291,"num_whiteouts":17,"num_read":641,"num_read_kb":653075,"num_write":472,"num_write_kb":213592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40919,"ondisk_log_size":40919,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":1654676,"kb_used_data":1133224,"kb_used_omap":18,"kb_used_meta":521133,"kb_avail":1123287148,"statfs":{"total":1151940427776,"available":1150246039552,"internally_reserved":0,"allocated":1160421376,"data_stored":1542052949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":533640507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6948765,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":19,"num_write_kb":5394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.025669"},"pg_stats":[{"pgid":"2.41","version":"567'235","reported_seq":3883,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.328672+0000","last_change":"2024-09-11T03:50:45.055329+0000","last_active":"2024-09-11T03:50:47.328672+0000","last_peered":"2024-09-11T03:50:47.328672+0000","last_clean":"2024-09-11T03:50:47.328672+0000","last_became_active":"2024-09-11T03:50:33.046488+0000","last_became_peered":"2024-09-11T03:50:33.046488+0000","last_unstale":"2024-09-11T03:50:47.328672+0000","last_undegraded":"2024-09-11T03:50:47.328672+0000","last_fullsized":"2024-09-11T03:50:47.328672+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":7,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:50:35.786143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.786143+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:54.966535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054524499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.40","version":"563'192","reported_seq":3942,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.050269+0000","last_change":"2024-09-11T03:50:45.045493+0000","last_active":"2024-09-11T03:50:46.050269+0000","last_peered":"2024-09-11T03:50:46.050269+0000","last_clean":"2024-09-11T03:50:46.050269+0000","last_became_active":"2024-09-11T03:50:33.047623+0000","last_became_peered":"2024-09-11T03:50:33.047623+0000","last_unstale":"2024-09-11T03:50:46.050269+0000","last_undegraded":"2024-09-11T03:50:46.050269+0000","last_fullsized":"2024-09-11T03:50:46.050269+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":7,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:50:35.472175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.472175+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:46.266531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071398000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.3f","version":"828'766","reported_seq":4448,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.048591+0000","last_change":"2024-09-11T03:50:45.068289+0000","last_active":"2024-09-11T03:50:46.048591+0000","last_peered":"2024-09-11T03:50:46.048591+0000","last_clean":"2024-09-11T03:50:46.048591+0000","last_became_active":"2024-09-11T03:50:33.046798+0000","last_became_peered":"2024-09-11T03:50:33.046798+0000","last_unstale":"2024-09-11T03:50:46.048591+0000","last_undegraded":"2024-09-11T03:50:46.048591+0000","last_fullsized":"2024-09-11T03:50:46.048591+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"817'762","last_scrub_stamp":"2024-09-11T03:50:36.829688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.829688+0000","objects_scrubbed":7,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:04.245180+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.016674799000000001,"stat_sum":{"num_bytes":15018014,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.3e","version":"827'1066","reported_seq":4714,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.046467+0000","last_change":"2024-09-11T03:50:45.068012+0000","last_active":"2024-09-11T03:50:47.046467+0000","last_peered":"2024-09-11T03:50:47.046467+0000","last_clean":"2024-09-11T03:50:47.046467+0000","last_became_active":"2024-09-11T03:50:33.045235+0000","last_became_peered":"2024-09-11T03:50:33.045235+0000","last_unstale":"2024-09-11T03:50:47.046467+0000","last_undegraded":"2024-09-11T03:50:47.046467+0000","last_fullsized":"2024-09-11T03:50:47.046467+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"815'1064","last_scrub_stamp":"2024-09-11T03:50:36.212789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.212789+0000","objects_scrubbed":8,"log_size":1066,"log_dups_size":0,"ondisk_log_size":1066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:03.235961+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.022530297000000001,"stat_sum":{"num_bytes":15801494,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.3d","version":"567'752","reported_seq":3215,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.044421+0000","last_change":"2024-09-11T03:50:45.068971+0000","last_active":"2024-09-11T03:50:46.044421+0000","last_peered":"2024-09-11T03:50:46.044421+0000","last_clean":"2024-09-11T03:50:46.044421+0000","last_became_active":"2024-09-11T03:50:33.039143+0000","last_became_peered":"2024-09-11T03:50:33.039143+0000","last_unstale":"2024-09-11T03:50:46.044421+0000","last_undegraded":"2024-09-11T03:50:46.044421+0000","last_fullsized":"2024-09-11T03:50:46.044421+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:50:35.414581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.414581+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:49.104717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015450033,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.3c","version":"827'778","reported_seq":4474,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.048186+0000","last_change":"2024-09-11T03:50:45.056113+0000","last_active":"2024-09-11T03:50:46.048186+0000","last_peered":"2024-09-11T03:50:46.048186+0000","last_clean":"2024-09-11T03:50:46.048186+0000","last_became_active":"2024-09-11T03:50:33.048487+0000","last_became_peered":"2024-09-11T03:50:33.048487+0000","last_unstale":"2024-09-11T03:50:46.048186+0000","last_undegraded":"2024-09-11T03:50:46.048186+0000","last_fullsized":"2024-09-11T03:50:46.048186+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"806'776","last_scrub_stamp":"2024-09-11T03:50:35.839653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.839653+0000","objects_scrubbed":4,"log_size":778,"log_dups_size":0,"ondisk_log_size":778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:43.203685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058174680000000001,"stat_sum":{"num_bytes":8735071,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,10],"acting":[7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.3b","version":"828'897","reported_seq":4681,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.096548+0000","last_change":"2024-09-11T03:50:45.071227+0000","last_active":"2024-09-11T03:50:46.096548+0000","last_peered":"2024-09-11T03:50:46.096548+0000","last_clean":"2024-09-11T03:50:46.096548+0000","last_became_active":"2024-09-11T03:50:33.046784+0000","last_became_peered":"2024-09-11T03:50:33.046784+0000","last_unstale":"2024-09-11T03:50:46.096548+0000","last_undegraded":"2024-09-11T03:50:46.096548+0000","last_fullsized":"2024-09-11T03:50:46.096548+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"817'886","last_scrub_stamp":"2024-09-11T03:50:35.496439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.496439+0000","objects_scrubbed":8,"log_size":897,"log_dups_size":0,"ondisk_log_size":897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:01.502038+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.022016457,"stat_sum":{"num_bytes":18564451,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":7,"num_write_kb":1297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,10],"acting":[5,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.3a","version":"557'273","reported_seq":2722,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-09-11T03:50:48.067022+0000","last_change":"2024-09-11T03:50:48.067250+0000","last_active":"2024-09-11T03:50:48.067022+0000","last_peered":"2024-09-11T03:50:48.067022+0000","last_clean":"2024-09-11T03:50:48.067022+0000","last_became_active":"2024-09-11T03:50:33.043178+0000","last_became_peered":"2024-09-11T03:50:33.043178+0000","last_unstale":"2024-09-11T03:50:48.067022+0000","last_undegraded":"2024-09-11T03:50:48.067022+0000","last_fullsized":"2024-09-11T03:50:48.067022+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:50:37.286055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.286055+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:48.834547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001120825,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"},{"start":"173","length":"1"}]},{"pgid":"2.39","version":"806'585","reported_seq":4332,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-09-11T03:50:48.066370+0000","last_change":"2024-09-11T03:50:48.066972+0000","last_active":"2024-09-11T03:50:48.066370+0000","last_peered":"2024-09-11T03:50:48.066370+0000","last_clean":"2024-09-11T03:50:48.066370+0000","last_became_active":"2024-09-11T03:50:33.043651+0000","last_became_peered":"2024-09-11T03:50:33.043651+0000","last_unstale":"2024-09-11T03:50:48.066370+0000","last_undegraded":"2024-09-11T03:50:48.066370+0000","last_fullsized":"2024-09-11T03:50:48.066370+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"806'585","last_scrub_stamp":"2024-09-11T03:50:36.264989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.264989+0000","objects_scrubbed":0,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:39.217562+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00091424400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,2],"acting":[9,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"},{"start":"173","length":"1"}]},{"pgid":"2.38","version":"808'564","reported_seq":4311,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.044963+0000","last_change":"2024-09-11T03:50:45.042942+0000","last_active":"2024-09-11T03:50:46.044963+0000","last_peered":"2024-09-11T03:50:46.044963+0000","last_clean":"2024-09-11T03:50:46.044963+0000","last_became_active":"2024-09-11T03:50:33.042740+0000","last_became_peered":"2024-09-11T03:50:33.042740+0000","last_unstale":"2024-09-11T03:50:46.044963+0000","last_undegraded":"2024-09-11T03:50:46.044963+0000","last_fullsized":"2024-09-11T03:50:46.044963+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"808'564","last_scrub_stamp":"2024-09-11T03:50:35.529587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.529587+0000","objects_scrubbed":0,"log_size":564,"log_dups_size":0,"ondisk_log_size":564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:00.686816+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00048016500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,1],"acting":[8,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.37","version":"274'312","reported_seq":2288,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.328965+0000","last_change":"2024-09-11T03:50:45.056729+0000","last_active":"2024-09-11T03:50:47.328965+0000","last_peered":"2024-09-11T03:50:47.328965+0000","last_clean":"2024-09-11T03:50:47.328965+0000","last_became_active":"2024-09-11T03:50:07.077607+0000","last_became_peered":"2024-09-11T03:50:07.077607+0000","last_unstale":"2024-09-11T03:50:47.328965+0000","last_undegraded":"2024-09-11T03:50:47.328965+0000","last_fullsized":"2024-09-11T03:50:47.328965+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:50:17.684567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.684567+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:46.982699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014148089999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,9],"acting":[10,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.36","version":"828'612","reported_seq":4355,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.096836+0000","last_change":"2024-09-11T03:50:45.072625+0000","last_active":"2024-09-11T03:50:46.096836+0000","last_peered":"2024-09-11T03:50:46.096836+0000","last_clean":"2024-09-11T03:50:46.096836+0000","last_became_active":"2024-09-11T03:50:07.078667+0000","last_became_peered":"2024-09-11T03:50:07.078667+0000","last_unstale":"2024-09-11T03:50:46.096836+0000","last_undegraded":"2024-09-11T03:50:46.096836+0000","last_fullsized":"2024-09-11T03:50:46.096836+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'589","last_scrub_stamp":"2024-09-11T03:50:18.280936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.280936+0000","objects_scrubbed":12,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:34.923795+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.018624037,"stat_sum":{"num_bytes":29342069,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":12,"num_read_kb":9827,"num_write":6,"num_write_kb":3927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,8],"acting":[1,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.35","version":"274'273","reported_seq":1453,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.044450+0000","last_change":"2024-09-11T03:50:45.069276+0000","last_active":"2024-09-11T03:50:46.044450+0000","last_peered":"2024-09-11T03:50:46.044450+0000","last_clean":"2024-09-11T03:50:46.044450+0000","last_became_active":"2024-09-11T03:50:07.079331+0000","last_became_peered":"2024-09-11T03:50:07.079331+0000","last_unstale":"2024-09-11T03:50:46.044450+0000","last_undegraded":"2024-09-11T03:50:46.044450+0000","last_fullsized":"2024-09-11T03:50:46.044450+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:50:17.314165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.314165+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:45.539691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015671653000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.34","version":"828'376","reported_seq":4054,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.072305+0000","last_change":"2024-09-11T03:50:45.067353+0000","last_active":"2024-09-11T03:50:46.072305+0000","last_peered":"2024-09-11T03:50:46.072305+0000","last_clean":"2024-09-11T03:50:46.072305+0000","last_became_active":"2024-09-11T03:50:07.080419+0000","last_became_peered":"2024-09-11T03:50:07.080419+0000","last_unstale":"2024-09-11T03:50:46.072305+0000","last_undegraded":"2024-09-11T03:50:46.072305+0000","last_fullsized":"2024-09-11T03:50:46.072305+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'358","last_scrub_stamp":"2024-09-11T03:50:17.613025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.613025+0000","objects_scrubbed":4,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:20.524683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.018564912999999999,"stat_sum":{"num_bytes":14855170,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":12515,"num_write":8,"num_write_kb":1518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.33","version":"274'111","reported_seq":1241,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.048181+0000","last_change":"2024-09-11T03:50:45.050467+0000","last_active":"2024-09-11T03:50:46.048181+0000","last_peered":"2024-09-11T03:50:46.048181+0000","last_clean":"2024-09-11T03:50:46.048181+0000","last_became_active":"2024-09-11T03:50:07.077684+0000","last_became_peered":"2024-09-11T03:50:07.077684+0000","last_unstale":"2024-09-11T03:50:46.048181+0000","last_undegraded":"2024-09-11T03:50:46.048181+0000","last_fullsized":"2024-09-11T03:50:46.048181+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:50:17.796171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.796171+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:21.698920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073264000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.32","version":"806'551","reported_seq":4220,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.046045+0000","last_change":"2024-09-11T03:50:45.045352+0000","last_active":"2024-09-11T03:50:47.046045+0000","last_peered":"2024-09-11T03:50:47.046045+0000","last_clean":"2024-09-11T03:50:47.046045+0000","last_became_active":"2024-09-11T03:50:07.077594+0000","last_became_peered":"2024-09-11T03:50:07.077594+0000","last_unstale":"2024-09-11T03:50:47.046045+0000","last_undegraded":"2024-09-11T03:50:47.046045+0000","last_fullsized":"2024-09-11T03:50:47.046045+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"778'549","last_scrub_stamp":"2024-09-11T03:50:17.199232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.199232+0000","objects_scrubbed":2,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:44.177140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058474700000000004,"stat_sum":{"num_bytes":5630774,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9],"acting":[11,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.31","version":"274'107","reported_seq":1747,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-09-11T03:50:50.086405+0000","last_change":"2024-09-11T03:50:48.051355+0000","last_active":"2024-09-11T03:50:50.086405+0000","last_peered":"2024-09-11T03:50:50.086405+0000","last_clean":"2024-09-11T03:50:50.086405+0000","last_became_active":"2024-09-11T03:50:07.075956+0000","last_became_peered":"2024-09-11T03:50:07.075956+0000","last_unstale":"2024-09-11T03:50:50.086405+0000","last_undegraded":"2024-09-11T03:50:50.086405+0000","last_fullsized":"2024-09-11T03:50:50.086405+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:50:17.857515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.857515+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:45.827024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092758900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,4],"acting":[0,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"},{"start":"173","length":"1"}]},{"pgid":"2.30","version":"274'67","reported_seq":209,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.328606+0000","last_change":"2024-09-11T03:50:45.055585+0000","last_active":"2024-09-11T03:50:47.328606+0000","last_peered":"2024-09-11T03:50:47.328606+0000","last_clean":"2024-09-11T03:50:47.328606+0000","last_became_active":"2024-09-11T03:50:04.069393+0000","last_became_peered":"2024-09-11T03:50:04.069393+0000","last_unstale":"2024-09-11T03:50:47.328606+0000","last_undegraded":"2024-09-11T03:50:47.328606+0000","last_fullsized":"2024-09-11T03:50:47.328606+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:50:04.608333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.608333+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:24.535258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088447199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,9],"acting":[10,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.2f","version":"557'465","reported_seq":3061,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-09-11T03:50:48.067675+0000","last_change":"2024-09-11T03:50:48.067900+0000","last_active":"2024-09-11T03:50:48.067675+0000","last_peered":"2024-09-11T03:50:48.067675+0000","last_clean":"2024-09-11T03:50:48.067675+0000","last_became_active":"2024-09-11T03:50:07.077840+0000","last_became_peered":"2024-09-11T03:50:07.077840+0000","last_unstale":"2024-09-11T03:50:48.067675+0000","last_undegraded":"2024-09-11T03:50:48.067675+0000","last_fullsized":"2024-09-11T03:50:48.067675+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:50:17.948546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.948546+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:39.091663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00091326199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"},{"start":"173","length":"1"}]},{"pgid":"2.2e","version":"827'1087","reported_seq":4712,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.048691+0000","last_change":"2024-09-11T03:50:45.068471+0000","last_active":"2024-09-11T03:50:46.048691+0000","last_peered":"2024-09-11T03:50:46.048691+0000","last_clean":"2024-09-11T03:50:46.048691+0000","last_became_active":"2024-09-11T03:50:07.078910+0000","last_became_peered":"2024-09-11T03:50:07.078910+0000","last_unstale":"2024-09-11T03:50:46.048691+0000","last_undegraded":"2024-09-11T03:50:46.048691+0000","last_fullsized":"2024-09-11T03:50:46.048691+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'1047","last_scrub_stamp":"2024-09-11T03:50:18.779326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.779326+0000","objects_scrubbed":11,"log_size":1087,"log_dups_size":0,"ondisk_log_size":1087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:27.232080+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.01776136,"stat_sum":{"num_bytes":34915115,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":12,"num_read_kb":10738,"num_write":12,"num_write_kb":6051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.2d","version":"828'1141","reported_seq":4808,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.097499+0000","last_change":"2024-09-11T03:50:45.074495+0000","last_active":"2024-09-11T03:50:46.097499+0000","last_peered":"2024-09-11T03:50:46.097499+0000","last_clean":"2024-09-11T03:50:46.097499+0000","last_became_active":"2024-09-11T03:49:29.163201+0000","last_became_peered":"2024-09-11T03:49:29.163201+0000","last_unstale":"2024-09-11T03:50:46.097499+0000","last_undegraded":"2024-09-11T03:50:46.097499+0000","last_fullsized":"2024-09-11T03:50:46.097499+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'997","last_scrub_stamp":"2024-09-11T03:49:32.462332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.462332+0000","objects_scrubbed":13,"log_size":1141,"log_dups_size":0,"ondisk_log_size":1141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.470625+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.029146760000000001,"stat_sum":{"num_bytes":41421147,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":43,"num_read_kb":38986,"num_write":24,"num_write_kb":14603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.2c","version":"827'882","reported_seq":4559,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.045273+0000","last_change":"2024-09-11T03:50:45.045614+0000","last_active":"2024-09-11T03:50:46.045273+0000","last_peered":"2024-09-11T03:50:46.045273+0000","last_clean":"2024-09-11T03:50:46.045273+0000","last_became_active":"2024-09-11T03:49:29.154872+0000","last_became_peered":"2024-09-11T03:49:29.154872+0000","last_unstale":"2024-09-11T03:50:46.045273+0000","last_undegraded":"2024-09-11T03:50:46.045273+0000","last_fullsized":"2024-09-11T03:50:46.045273+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"816'878","last_scrub_stamp":"2024-09-11T03:50:37.544645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.544645+0000","objects_scrubbed":12,"log_size":882,"log_dups_size":0,"ondisk_log_size":882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:48.044407+0000","scrub_duration":4,"objects_trimmed":1,"snaptrim_duration":0.0030400420000000002,"stat_sum":{"num_bytes":25734684,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":28,"num_read_kb":19600,"num_write":35,"num_write_kb":13666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3],"acting":[8,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.2b","version":"568'646","reported_seq":3463,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-09-11T03:50:48.067126+0000","last_change":"2024-09-11T03:50:48.067367+0000","last_active":"2024-09-11T03:50:48.067126+0000","last_peered":"2024-09-11T03:50:48.067126+0000","last_clean":"2024-09-11T03:50:48.067126+0000","last_became_active":"2024-09-11T03:49:29.157735+0000","last_became_peered":"2024-09-11T03:49:29.157735+0000","last_unstale":"2024-09-11T03:50:48.067126+0000","last_undegraded":"2024-09-11T03:50:48.067126+0000","last_fullsized":"2024-09-11T03:50:48.067126+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:50:40.311643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:40.311643+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:02.762010+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00077579899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"},{"start":"173","length":"1"}]},{"pgid":"2.2a","version":"727'348","reported_seq":3929,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.048345+0000","last_change":"2024-09-11T03:50:45.050833+0000","last_active":"2024-09-11T03:50:46.048345+0000","last_peered":"2024-09-11T03:50:46.048345+0000","last_clean":"2024-09-11T03:50:46.048345+0000","last_became_active":"2024-09-11T03:49:29.160595+0000","last_became_peered":"2024-09-11T03:49:29.160595+0000","last_unstale":"2024-09-11T03:50:46.048345+0000","last_undegraded":"2024-09-11T03:50:46.048345+0000","last_fullsized":"2024-09-11T03:50:46.048345+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:49:34.219244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.219244+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096057599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,5],"acting":[7,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.29","version":"827'1259","reported_seq":5040,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.051141+0000","last_change":"2024-09-11T03:50:45.069336+0000","last_active":"2024-09-11T03:50:46.051141+0000","last_peered":"2024-09-11T03:50:46.051141+0000","last_clean":"2024-09-11T03:50:46.051141+0000","last_became_active":"2024-09-11T03:49:29.157194+0000","last_became_peered":"2024-09-11T03:49:29.157194+0000","last_unstale":"2024-09-11T03:50:46.051141+0000","last_undegraded":"2024-09-11T03:50:46.051141+0000","last_fullsized":"2024-09-11T03:50:46.051141+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'1167","last_scrub_stamp":"2024-09-11T03:49:33.506786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.506786+0000","objects_scrubbed":11,"log_size":1259,"log_dups_size":0,"ondisk_log_size":1259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:58.325883+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.02418706,"stat_sum":{"num_bytes":33405469,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":36,"num_read_kb":54617,"num_write":22,"num_write_kb":8705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8],"acting":[6,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.28","version":"827'421","reported_seq":4105,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.044116+0000","last_change":"2024-09-11T03:50:45.067083+0000","last_active":"2024-09-11T03:50:46.044116+0000","last_peered":"2024-09-11T03:50:46.044116+0000","last_clean":"2024-09-11T03:50:46.044116+0000","last_became_active":"2024-09-11T03:49:29.154884+0000","last_became_peered":"2024-09-11T03:49:29.154884+0000","last_unstale":"2024-09-11T03:50:46.044116+0000","last_undegraded":"2024-09-11T03:50:46.044116+0000","last_fullsized":"2024-09-11T03:50:46.044116+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'382","last_scrub_stamp":"2024-09-11T03:49:32.335285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.335285+0000","objects_scrubbed":4,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.013818771000000001,"stat_sum":{"num_bytes":11377646,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":10,"num_read_kb":11527,"num_write":8,"num_write_kb":1289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.27","version":"828'1137","reported_seq":4924,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.085669+0000","last_change":"2024-09-11T03:50:45.071561+0000","last_active":"2024-09-11T03:50:46.085669+0000","last_peered":"2024-09-11T03:50:46.085669+0000","last_clean":"2024-09-11T03:50:46.085669+0000","last_became_active":"2024-09-11T03:49:29.159799+0000","last_became_peered":"2024-09-11T03:49:29.159799+0000","last_unstale":"2024-09-11T03:50:46.085669+0000","last_undegraded":"2024-09-11T03:50:46.085669+0000","last_fullsized":"2024-09-11T03:50:46.085669+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"757'1065","last_scrub_stamp":"2024-09-11T03:49:36.196385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.196385+0000","objects_scrubbed":4,"log_size":1137,"log_dups_size":0,"ondisk_log_size":1137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.019615700999999999,"stat_sum":{"num_bytes":21312860,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":17,"num_read_kb":13449,"num_write":15,"num_write_kb":15955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,9],"acting":[7,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.26","version":"269'171","reported_seq":3822,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.048515+0000","last_change":"2024-09-11T03:50:45.055664+0000","last_active":"2024-09-11T03:50:46.048515+0000","last_peered":"2024-09-11T03:50:46.048515+0000","last_clean":"2024-09-11T03:50:46.048515+0000","last_became_active":"2024-09-11T03:49:29.158577+0000","last_became_peered":"2024-09-11T03:49:29.158577+0000","last_unstale":"2024-09-11T03:50:46.048515+0000","last_undegraded":"2024-09-11T03:50:46.048515+0000","last_fullsized":"2024-09-11T03:50:46.048515+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:50:43.910786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:43.910786+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:57.542229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0049171129999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,11],"acting":[7,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.25","version":"828'1310","reported_seq":5196,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.060357+0000","last_change":"2024-09-11T03:50:45.050097+0000","last_active":"2024-09-11T03:50:46.060357+0000","last_peered":"2024-09-11T03:50:46.060357+0000","last_clean":"2024-09-11T03:50:46.060357+0000","last_became_active":"2024-09-11T03:49:29.159733+0000","last_became_peered":"2024-09-11T03:49:29.159733+0000","last_unstale":"2024-09-11T03:50:46.060357+0000","last_undegraded":"2024-09-11T03:50:46.060357+0000","last_fullsized":"2024-09-11T03:50:46.060357+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"755'1138","last_scrub_stamp":"2024-09-11T03:49:34.198067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.198067+0000","objects_scrubbed":16,"log_size":1310,"log_dups_size":0,"ondisk_log_size":1310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":1,"snaptrim_duration":0.007268928,"stat_sum":{"num_bytes":43560691,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":53,"num_read_kb":53768,"num_write":32,"num_write_kb":11130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,6],"acting":[8,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.24","version":"827'666","reported_seq":4366,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.050441+0000","last_change":"2024-09-11T03:50:45.051556+0000","last_active":"2024-09-11T03:50:46.050441+0000","last_peered":"2024-09-11T03:50:46.050441+0000","last_clean":"2024-09-11T03:50:46.050441+0000","last_became_active":"2024-09-11T03:49:29.155740+0000","last_became_peered":"2024-09-11T03:49:29.155740+0000","last_unstale":"2024-09-11T03:50:46.050441+0000","last_undegraded":"2024-09-11T03:50:46.050441+0000","last_fullsized":"2024-09-11T03:50:46.050441+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'612","last_scrub_stamp":"2024-09-11T03:49:34.554335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.554335+0000","objects_scrubbed":4,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:00.220182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0066349240000000004,"stat_sum":{"num_bytes":14236199,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":12,"num_write_kb":5382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,7],"acting":[6,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.23","version":"827'484","reported_seq":4149,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.044459+0000","last_change":"2024-09-11T03:50:45.085281+0000","last_active":"2024-09-11T03:50:46.044459+0000","last_peered":"2024-09-11T03:50:46.044459+0000","last_clean":"2024-09-11T03:50:46.044459+0000","last_became_active":"2024-09-11T03:46:32.970100+0000","last_became_peered":"2024-09-11T03:46:32.970100+0000","last_unstale":"2024-09-11T03:50:46.044459+0000","last_undegraded":"2024-09-11T03:50:46.044459+0000","last_fullsized":"2024-09-11T03:50:46.044459+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'474","last_scrub_stamp":"2024-09-11T03:50:19.311509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:19.311509+0000","objects_scrubbed":5,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:40.333214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.032016564999999997,"stat_sum":{"num_bytes":9866336,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":44928,"num_write":25,"num_write_kb":10047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.22","version":"531'216","reported_seq":3873,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.050485+0000","last_change":"2024-09-11T03:50:45.045981+0000","last_active":"2024-09-11T03:50:46.050485+0000","last_peered":"2024-09-11T03:50:46.050485+0000","last_clean":"2024-09-11T03:50:46.050485+0000","last_became_active":"2024-09-11T03:46:32.970633+0000","last_became_peered":"2024-09-11T03:46:32.970633+0000","last_unstale":"2024-09-11T03:50:46.050485+0000","last_undegraded":"2024-09-11T03:50:46.050485+0000","last_fullsized":"2024-09-11T03:50:46.050485+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:50:38.617374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:38.617374+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:59.696885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00098563100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.21","version":"828'354","reported_seq":4104,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.046159+0000","last_change":"2024-09-11T03:50:45.045333+0000","last_active":"2024-09-11T03:50:47.046159+0000","last_peered":"2024-09-11T03:50:47.046159+0000","last_clean":"2024-09-11T03:50:47.046159+0000","last_became_active":"2024-09-11T03:46:36.024928+0000","last_became_peered":"2024-09-11T03:46:36.024928+0000","last_unstale":"2024-09-11T03:50:47.046159+0000","last_undegraded":"2024-09-11T03:50:47.046159+0000","last_fullsized":"2024-09-11T03:50:47.046159+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":6,"last_scrub":"816'351","last_scrub_stamp":"2024-09-11T03:50:40.225380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:40.225380+0000","objects_scrubbed":7,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:59.228284+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000603453,"stat_sum":{"num_bytes":15025885,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":39,"num_read_kb":45171,"num_write":34,"num_write_kb":15311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,8],"acting":[11,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.20","version":"827'300","reported_seq":4123,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.048200+0000","last_change":"2024-09-11T03:50:45.054545+0000","last_active":"2024-09-11T03:50:46.048200+0000","last_peered":"2024-09-11T03:50:46.048200+0000","last_clean":"2024-09-11T03:50:46.048200+0000","last_became_active":"2024-09-11T03:46:32.974375+0000","last_became_peered":"2024-09-11T03:46:32.974375+0000","last_unstale":"2024-09-11T03:50:46.048200+0000","last_undegraded":"2024-09-11T03:50:46.048200+0000","last_fullsized":"2024-09-11T03:50:46.048200+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"818'296","last_scrub_stamp":"2024-09-11T03:50:37.825854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.825854+0000","objects_scrubbed":6,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:47.869264+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0048700610000000002,"stat_sum":{"num_bytes":15085668,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":97,"num_read_kb":144174,"num_write":22,"num_write_kb":11057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.1c","version":"806'776","reported_seq":4441,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.328899+0000","last_change":"2024-09-11T03:50:45.056511+0000","last_active":"2024-09-11T03:50:47.328899+0000","last_peered":"2024-09-11T03:50:47.328899+0000","last_clean":"2024-09-11T03:50:47.328899+0000","last_became_active":"2024-09-11T03:50:28.047548+0000","last_became_peered":"2024-09-11T03:50:28.047548+0000","last_unstale":"2024-09-11T03:50:47.328899+0000","last_undegraded":"2024-09-11T03:50:47.328899+0000","last_fullsized":"2024-09-11T03:50:47.328899+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"806'776","last_scrub_stamp":"2024-09-11T03:50:37.794065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.794065+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:52.895108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012302840000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":716,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-09-11T03:50:50.086427+0000","last_change":"2024-09-11T03:47:26.195120+0000","last_active":"2024-09-11T03:50:50.086427+0000","last_peered":"2024-09-11T03:50:50.086427+0000","last_clean":"2024-09-11T03:50:50.086427+0000","last_became_active":"2024-09-11T03:46:15.974786+0000","last_became_peered":"2024-09-11T03:46:15.974786+0000","last_unstale":"2024-09-11T03:50:50.086427+0000","last_undegraded":"2024-09-11T03:50:50.086427+0000","last_fullsized":"2024-09-11T03:50:50.086427+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:26.195070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:26.195070+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:53.091756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.f","version":"557'465","reported_seq":4152,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.045405+0000","last_change":"2024-09-11T03:50:45.050490+0000","last_active":"2024-09-11T03:50:46.045405+0000","last_peered":"2024-09-11T03:50:46.045405+0000","last_clean":"2024-09-11T03:50:46.045405+0000","last_became_active":"2024-09-11T03:50:02.055146+0000","last_became_peered":"2024-09-11T03:50:02.055146+0000","last_unstale":"2024-09-11T03:50:46.045405+0000","last_undegraded":"2024-09-11T03:50:46.045405+0000","last_fullsized":"2024-09-11T03:50:46.045405+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:50:02.368307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.368307+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:22.700436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073495330000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.e","version":"782'1025","reported_seq":4669,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-09-11T03:50:48.067518+0000","last_change":"2024-09-11T03:50:48.067711+0000","last_active":"2024-09-11T03:50:48.067518+0000","last_peered":"2024-09-11T03:50:48.067518+0000","last_clean":"2024-09-11T03:50:48.067518+0000","last_became_active":"2024-09-11T03:50:02.051717+0000","last_became_peered":"2024-09-11T03:50:02.051717+0000","last_unstale":"2024-09-11T03:50:48.067518+0000","last_undegraded":"2024-09-11T03:50:48.067518+0000","last_fullsized":"2024-09-11T03:50:48.067518+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":0,"last_scrub":"782'1025","last_scrub_stamp":"2024-09-11T03:50:04.968160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.968160+0000","objects_scrubbed":0,"log_size":1025,"log_dups_size":0,"ondisk_log_size":1025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:13.049310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074290000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"},{"start":"173","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":1285,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.048339+0000","last_change":"2024-09-11T03:49:20.176154+0000","last_active":"2024-09-11T03:50:46.048339+0000","last_peered":"2024-09-11T03:50:46.048339+0000","last_clean":"2024-09-11T03:50:46.048339+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:50:46.048339+0000","last_undegraded":"2024-09-11T03:50:46.048339+0000","last_fullsized":"2024-09-11T03:50:46.048339+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:20.176098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:20.176098+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"828'1019","reported_seq":4741,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.053198+0000","last_change":"2024-09-11T03:50:45.067821+0000","last_active":"2024-09-11T03:50:46.053198+0000","last_peered":"2024-09-11T03:50:46.053198+0000","last_clean":"2024-09-11T03:50:46.053198+0000","last_became_active":"2024-09-11T03:49:24.116744+0000","last_became_peered":"2024-09-11T03:49:24.116744+0000","last_unstale":"2024-09-11T03:50:46.053198+0000","last_undegraded":"2024-09-11T03:50:46.053198+0000","last_fullsized":"2024-09-11T03:50:46.053198+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"811'1014","last_scrub_stamp":"2024-09-11T03:50:37.438736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.438736+0000","objects_scrubbed":5,"log_size":1019,"log_dups_size":0,"ondisk_log_size":1019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:55.045449+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.018592865,"stat_sum":{"num_bytes":8757074,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":1008,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.048784+0000","last_change":"2024-09-11T03:50:38.843495+0000","last_active":"2024-09-11T03:50:46.048784+0000","last_peered":"2024-09-11T03:50:46.048784+0000","last_clean":"2024-09-11T03:50:46.048784+0000","last_became_active":"2024-09-11T03:44:43.940022+0000","last_became_peered":"2024-09-11T03:44:43.940022+0000","last_unstale":"2024-09-11T03:50:46.048784+0000","last_undegraded":"2024-09-11T03:50:46.048784+0000","last_fullsized":"2024-09-11T03:50:46.048784+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:38.843438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:38.843438+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:45.743283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"829'866","reported_seq":4546,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.046396+0000","last_change":"2024-09-11T03:50:45.045737+0000","last_active":"2024-09-11T03:50:47.046396+0000","last_peered":"2024-09-11T03:50:47.046396+0000","last_clean":"2024-09-11T03:50:47.046396+0000","last_became_active":"2024-09-11T03:49:24.115405+0000","last_became_peered":"2024-09-11T03:49:24.115405+0000","last_unstale":"2024-09-11T03:50:47.046396+0000","last_undegraded":"2024-09-11T03:50:47.046396+0000","last_fullsized":"2024-09-11T03:50:47.046396+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'808","last_scrub_stamp":"2024-09-11T03:49:33.953477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.953477+0000","objects_scrubbed":7,"log_size":866,"log_dups_size":0,"ondisk_log_size":866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:01.906058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00095474799999999995,"stat_sum":{"num_bytes":23449747,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":14,"num_read_kb":13992,"num_write":14,"num_write_kb":6063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.f","version":"20'92","reported_seq":566,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.051253+0000","last_change":"2024-09-11T03:48:57.363099+0000","last_active":"2024-09-11T03:50:46.051253+0000","last_peered":"2024-09-11T03:50:46.051253+0000","last_clean":"2024-09-11T03:50:46.051253+0000","last_became_active":"2024-09-11T03:46:15.973796+0000","last_became_peered":"2024-09-11T03:46:15.973796+0000","last_unstale":"2024-09-11T03:50:46.051253+0000","last_undegraded":"2024-09-11T03:50:46.051253+0000","last_fullsized":"2024-09-11T03:50:46.051253+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:57.363027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:57.363027+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:10.644654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":1691,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.048743+0000","last_change":"2024-09-11T03:47:28.495899+0000","last_active":"2024-09-11T03:50:46.048743+0000","last_peered":"2024-09-11T03:50:46.048743+0000","last_clean":"2024-09-11T03:50:46.048743+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:50:46.048743+0000","last_undegraded":"2024-09-11T03:50:46.048743+0000","last_fullsized":"2024-09-11T03:50:46.048743+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:28.495839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:28.495839+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.015311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"827'543","reported_seq":4211,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.053233+0000","last_change":"2024-09-11T03:50:45.071667+0000","last_active":"2024-09-11T03:50:46.053233+0000","last_peered":"2024-09-11T03:50:46.053233+0000","last_clean":"2024-09-11T03:50:46.053233+0000","last_became_active":"2024-09-11T03:46:27.972329+0000","last_became_peered":"2024-09-11T03:46:27.972329+0000","last_unstale":"2024-09-11T03:50:46.053233+0000","last_undegraded":"2024-09-11T03:50:46.053233+0000","last_fullsized":"2024-09-11T03:50:46.053233+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'480","last_scrub_stamp":"2024-09-11T03:49:37.694768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.694768+0000","objects_scrubbed":5,"log_size":543,"log_dups_size":0,"ondisk_log_size":543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.022136904999999998,"stat_sum":{"num_bytes":16420628,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":28,"num_read_kb":23187,"num_write":43,"num_write_kb":8736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":1298,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.044458+0000","last_change":"2024-09-11T03:50:45.053323+0000","last_active":"2024-09-11T03:50:46.044458+0000","last_peered":"2024-09-11T03:50:46.044458+0000","last_clean":"2024-09-11T03:50:46.044458+0000","last_became_active":"2024-09-11T03:46:15.968960+0000","last_became_peered":"2024-09-11T03:46:15.968960+0000","last_unstale":"2024-09-11T03:50:46.044458+0000","last_undegraded":"2024-09-11T03:50:46.044458+0000","last_fullsized":"2024-09-11T03:50:46.044458+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:25.344170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:25.344170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:14.363606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"563'192","reported_seq":2741,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.328726+0000","last_change":"2024-09-11T03:50:45.056117+0000","last_active":"2024-09-11T03:50:47.328726+0000","last_peered":"2024-09-11T03:50:47.328726+0000","last_clean":"2024-09-11T03:50:47.328726+0000","last_became_active":"2024-09-11T03:50:31.044399+0000","last_became_peered":"2024-09-11T03:50:31.044399+0000","last_unstale":"2024-09-11T03:50:47.328726+0000","last_undegraded":"2024-09-11T03:50:47.328726+0000","last_fullsized":"2024-09-11T03:50:47.328726+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:50:36.771360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.771360+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:45.211739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001079134,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":1271,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-09-11T03:50:50.086251+0000","last_change":"2024-09-11T03:50:41.975709+0000","last_active":"2024-09-11T03:50:50.086251+0000","last_peered":"2024-09-11T03:50:50.086251+0000","last_clean":"2024-09-11T03:50:50.086251+0000","last_became_active":"2024-09-11T03:44:25.934580+0000","last_became_peered":"2024-09-11T03:44:25.934580+0000","last_unstale":"2024-09-11T03:50:50.086251+0000","last_undegraded":"2024-09-11T03:50:50.086251+0000","last_fullsized":"2024-09-11T03:50:50.086251+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:41.975660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:41.975660+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:00.591013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"567'235","reported_seq":3901,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-09-11T03:50:50.086545+0000","last_change":"2024-09-11T03:50:48.051798+0000","last_active":"2024-09-11T03:50:50.086545+0000","last_peered":"2024-09-11T03:50:50.086545+0000","last_clean":"2024-09-11T03:50:50.086545+0000","last_became_active":"2024-09-11T03:50:28.039300+0000","last_became_peered":"2024-09-11T03:50:28.039300+0000","last_unstale":"2024-09-11T03:50:50.086545+0000","last_undegraded":"2024-09-11T03:50:50.086545+0000","last_fullsized":"2024-09-11T03:50:50.086545+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:50:36.844578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.844578+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:53.985961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099457199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"},{"start":"173","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":870,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.328737+0000","last_change":"2024-09-11T03:50:18.699606+0000","last_active":"2024-09-11T03:50:47.328737+0000","last_peered":"2024-09-11T03:50:47.328737+0000","last_clean":"2024-09-11T03:50:47.328737+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:50:47.328737+0000","last_undegraded":"2024-09-11T03:50:47.328737+0000","last_fullsized":"2024-09-11T03:50:47.328737+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:18.699532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.699532+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:27.930456+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"828'313","reported_seq":4060,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.065270+0000","last_change":"2024-09-11T03:50:45.054220+0000","last_active":"2024-09-11T03:50:46.065270+0000","last_peered":"2024-09-11T03:50:46.065270+0000","last_clean":"2024-09-11T03:50:46.065270+0000","last_became_active":"2024-09-11T03:46:27.971649+0000","last_became_peered":"2024-09-11T03:46:27.971649+0000","last_unstale":"2024-09-11T03:50:46.065270+0000","last_undegraded":"2024-09-11T03:50:46.065270+0000","last_fullsized":"2024-09-11T03:50:46.065270+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"799'299","last_scrub_stamp":"2024-09-11T03:50:20.621227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:20.621227+0000","objects_scrubbed":3,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:38.376687+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0049684990000000004,"stat_sum":{"num_bytes":11917054,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":13036,"num_write":33,"num_write_kb":17922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":559,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.050819+0000","last_change":"2024-09-11T03:49:37.637065+0000","last_active":"2024-09-11T03:50:46.050819+0000","last_peered":"2024-09-11T03:50:46.050819+0000","last_clean":"2024-09-11T03:50:46.050819+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:50:46.050819+0000","last_undegraded":"2024-09-11T03:50:46.050819+0000","last_fullsized":"2024-09-11T03:50:46.050819+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:37.637018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.637018+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:05.221389+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"827'675","reported_seq":4398,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.044522+0000","last_change":"2024-09-11T03:50:45.070653+0000","last_active":"2024-09-11T03:50:46.044522+0000","last_peered":"2024-09-11T03:50:46.044522+0000","last_clean":"2024-09-11T03:50:46.044522+0000","last_became_active":"2024-09-11T03:49:24.114330+0000","last_became_peered":"2024-09-11T03:49:24.114330+0000","last_unstale":"2024-09-11T03:50:46.044522+0000","last_undegraded":"2024-09-11T03:50:46.044522+0000","last_fullsized":"2024-09-11T03:50:46.044522+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'620","last_scrub_stamp":"2024-09-11T03:49:34.288083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.288083+0000","objects_scrubbed":8,"log_size":675,"log_dups_size":0,"ondisk_log_size":675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:00.387566+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.016971122000000002,"stat_sum":{"num_bytes":22786313,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":12,"num_read_kb":8619,"num_write":9,"num_write_kb":5852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":881,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.328841+0000","last_change":"2024-09-11T03:50:40.811000+0000","last_active":"2024-09-11T03:50:47.328841+0000","last_peered":"2024-09-11T03:50:47.328841+0000","last_clean":"2024-09-11T03:50:47.328841+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:50:47.328841+0000","last_undegraded":"2024-09-11T03:50:47.328841+0000","last_fullsized":"2024-09-11T03:50:47.328841+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:40.810941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:40.810941+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:46.731652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"827'1193","reported_seq":4928,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.045698+0000","last_change":"2024-09-11T03:50:45.067072+0000","last_active":"2024-09-11T03:50:46.045698+0000","last_peered":"2024-09-11T03:50:46.045698+0000","last_clean":"2024-09-11T03:50:46.045698+0000","last_became_active":"2024-09-11T03:49:24.115095+0000","last_became_peered":"2024-09-11T03:49:24.115095+0000","last_unstale":"2024-09-11T03:50:46.045698+0000","last_undegraded":"2024-09-11T03:50:46.045698+0000","last_fullsized":"2024-09-11T03:50:46.045698+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"819'1187","last_scrub_stamp":"2024-09-11T03:50:40.500149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:40.500149+0000","objects_scrubbed":11,"log_size":1193,"log_dups_size":0,"ondisk_log_size":1193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:57.673789+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.023981987999999999,"stat_sum":{"num_bytes":17946110,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":7115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":872,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.328993+0000","last_change":"2024-09-11T03:49:36.654074+0000","last_active":"2024-09-11T03:50:47.328993+0000","last_peered":"2024-09-11T03:50:47.328993+0000","last_clean":"2024-09-11T03:50:47.328993+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:50:47.328993+0000","last_undegraded":"2024-09-11T03:50:47.328993+0000","last_fullsized":"2024-09-11T03:50:47.328993+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:36.654025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.654025+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:06.216209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":3827,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.044673+0000","last_change":"2024-09-11T03:50:45.072888+0000","last_active":"2024-09-11T03:50:46.044673+0000","last_peered":"2024-09-11T03:50:46.044673+0000","last_clean":"2024-09-11T03:50:46.044673+0000","last_became_active":"2024-09-11T03:49:24.113518+0000","last_became_peered":"2024-09-11T03:49:24.113518+0000","last_unstale":"2024-09-11T03:50:46.044673+0000","last_undegraded":"2024-09-11T03:50:46.044673+0000","last_fullsized":"2024-09-11T03:50:46.044673+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:50:37.453595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.453595+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:54.390063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.018723021999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":1472,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.045968+0000","last_change":"2024-09-11T03:49:21.909643+0000","last_active":"2024-09-11T03:50:47.045968+0000","last_peered":"2024-09-11T03:50:47.045968+0000","last_clean":"2024-09-11T03:50:47.045968+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:50:47.045968+0000","last_undegraded":"2024-09-11T03:50:47.045968+0000","last_fullsized":"2024-09-11T03:50:47.045968+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:21.909594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:21.909594+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:26.077940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"828'1212","reported_seq":5049,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.078324+0000","last_change":"2024-09-11T03:50:45.072468+0000","last_active":"2024-09-11T03:50:46.078324+0000","last_peered":"2024-09-11T03:50:46.078324+0000","last_clean":"2024-09-11T03:50:46.078324+0000","last_became_active":"2024-09-11T03:49:24.121333+0000","last_became_peered":"2024-09-11T03:49:24.121333+0000","last_unstale":"2024-09-11T03:50:46.078324+0000","last_undegraded":"2024-09-11T03:50:46.078324+0000","last_fullsized":"2024-09-11T03:50:46.078324+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"817'1201","last_scrub_stamp":"2024-09-11T03:50:40.600544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:40.600544+0000","objects_scrubbed":13,"log_size":1212,"log_dups_size":0,"ondisk_log_size":1212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:07.067397+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.026612203000000001,"stat_sum":{"num_bytes":37962452,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":86,"num_read_kb":75866,"num_write":47,"num_write_kb":27593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":1274,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.046458+0000","last_change":"2024-09-11T03:50:18.176206+0000","last_active":"2024-09-11T03:50:47.046458+0000","last_peered":"2024-09-11T03:50:47.046458+0000","last_clean":"2024-09-11T03:50:47.046458+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:50:47.046458+0000","last_undegraded":"2024-09-11T03:50:47.046458+0000","last_fullsized":"2024-09-11T03:50:47.046458+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:18.176156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.176156+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:43.809898+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"744'374","reported_seq":4079,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.048610+0000","last_change":"2024-09-11T03:50:45.053331+0000","last_active":"2024-09-11T03:50:46.048610+0000","last_peered":"2024-09-11T03:50:46.048610+0000","last_clean":"2024-09-11T03:50:46.048610+0000","last_became_active":"2024-09-11T03:49:24.115150+0000","last_became_peered":"2024-09-11T03:49:24.115150+0000","last_unstale":"2024-09-11T03:50:46.048610+0000","last_undegraded":"2024-09-11T03:50:46.048610+0000","last_fullsized":"2024-09-11T03:50:46.048610+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'374","last_scrub_stamp":"2024-09-11T03:49:35.190806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:35.190806+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:59.920680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002965564,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":1627,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-09-11T03:50:48.066393+0000","last_change":"2024-09-11T03:50:18.934008+0000","last_active":"2024-09-11T03:50:48.066393+0000","last_peered":"2024-09-11T03:50:48.066393+0000","last_clean":"2024-09-11T03:50:48.066393+0000","last_became_active":"2024-09-11T03:46:15.973271+0000","last_became_peered":"2024-09-11T03:46:15.973271+0000","last_unstale":"2024-09-11T03:50:48.066393+0000","last_undegraded":"2024-09-11T03:50:48.066393+0000","last_fullsized":"2024-09-11T03:50:48.066393+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:18.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.933952+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:39.183063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"828'1320","reported_seq":5096,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.073616+0000","last_change":"2024-09-11T03:50:45.068642+0000","last_active":"2024-09-11T03:50:46.073616+0000","last_peered":"2024-09-11T03:50:46.073616+0000","last_clean":"2024-09-11T03:50:46.073616+0000","last_became_active":"2024-09-11T03:49:24.114865+0000","last_became_peered":"2024-09-11T03:49:24.114865+0000","last_unstale":"2024-09-11T03:50:46.073616+0000","last_undegraded":"2024-09-11T03:50:46.073616+0000","last_fullsized":"2024-09-11T03:50:46.073616+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'1171","last_scrub_stamp":"2024-09-11T03:49:33.292043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.292043+0000","objects_scrubbed":10,"log_size":1320,"log_dups_size":0,"ondisk_log_size":1320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015474182,"stat_sum":{"num_bytes":33309237,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":36,"num_read_kb":35560,"num_write":32,"num_write_kb":13807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":1686,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.047915+0000","last_change":"2024-09-11T03:49:37.206392+0000","last_active":"2024-09-11T03:50:46.047915+0000","last_peered":"2024-09-11T03:50:46.047915+0000","last_clean":"2024-09-11T03:50:46.047915+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:50:46.047915+0000","last_undegraded":"2024-09-11T03:50:46.047915+0000","last_fullsized":"2024-09-11T03:50:46.047915+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:37.206321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.206321+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:53.641968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"823'375","reported_seq":4080,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.046084+0000","last_change":"2024-09-11T03:50:45.046175+0000","last_active":"2024-09-11T03:50:47.046084+0000","last_peered":"2024-09-11T03:50:47.046084+0000","last_clean":"2024-09-11T03:50:47.046084+0000","last_became_active":"2024-09-11T03:49:24.114551+0000","last_became_peered":"2024-09-11T03:49:24.114551+0000","last_unstale":"2024-09-11T03:50:47.046084+0000","last_undegraded":"2024-09-11T03:50:47.046084+0000","last_fullsized":"2024-09-11T03:50:47.046084+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:49:32.911631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.911631+0000","objects_scrubbed":4,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001362362,"stat_sum":{"num_bytes":5429057,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":10,"num_read_kb":12240,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":1345,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.052749+0000","last_change":"2024-09-11T03:49:34.621588+0000","last_active":"2024-09-11T03:50:46.052749+0000","last_peered":"2024-09-11T03:50:46.052749+0000","last_clean":"2024-09-11T03:50:46.052749+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:50:46.052749+0000","last_undegraded":"2024-09-11T03:50:46.052749+0000","last_fullsized":"2024-09-11T03:50:46.052749+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:34.621527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.621527+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.563733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"568'646","reported_seq":4346,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.045123+0000","last_change":"2024-09-11T03:50:45.043234+0000","last_active":"2024-09-11T03:50:46.045123+0000","last_peered":"2024-09-11T03:50:46.045123+0000","last_clean":"2024-09-11T03:50:46.045123+0000","last_became_active":"2024-09-11T03:50:31.042191+0000","last_became_peered":"2024-09-11T03:50:31.042191+0000","last_unstale":"2024-09-11T03:50:46.045123+0000","last_undegraded":"2024-09-11T03:50:46.045123+0000","last_fullsized":"2024-09-11T03:50:46.045123+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:49:33.204497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.204497+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00075609199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,0],"acting":[8,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":1295,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.052638+0000","last_change":"2024-09-11T03:49:00.710015+0000","last_active":"2024-09-11T03:50:46.052638+0000","last_peered":"2024-09-11T03:50:46.052638+0000","last_clean":"2024-09-11T03:50:46.052638+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:50:46.052638+0000","last_undegraded":"2024-09-11T03:50:46.052638+0000","last_fullsized":"2024-09-11T03:50:46.052638+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:00.709966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.709966+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:32.521638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":3755,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-09-11T03:50:48.067416+0000","last_change":"2024-09-11T03:50:48.067621+0000","last_active":"2024-09-11T03:50:48.067416+0000","last_peered":"2024-09-11T03:50:48.067416+0000","last_clean":"2024-09-11T03:50:48.067416+0000","last_became_active":"2024-09-11T03:50:02.046906+0000","last_became_peered":"2024-09-11T03:50:02.046906+0000","last_unstale":"2024-09-11T03:50:48.067416+0000","last_undegraded":"2024-09-11T03:50:48.067416+0000","last_fullsized":"2024-09-11T03:50:48.067416+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:50:02.987179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.987179+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:18.040118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089301699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"},{"start":"173","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":1288,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-09-11T03:50:50.086384+0000","last_change":"2024-09-11T03:49:41.224105+0000","last_active":"2024-09-11T03:50:50.086384+0000","last_peered":"2024-09-11T03:50:50.086384+0000","last_clean":"2024-09-11T03:50:50.086384+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:50:50.086384+0000","last_undegraded":"2024-09-11T03:50:50.086384+0000","last_fullsized":"2024-09-11T03:50:50.086384+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:41.224059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:41.224059+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:58.932206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"823'561","reported_seq":4249,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.328693+0000","last_change":"2024-09-11T03:50:45.055813+0000","last_active":"2024-09-11T03:50:47.328693+0000","last_peered":"2024-09-11T03:50:47.328693+0000","last_clean":"2024-09-11T03:50:47.328693+0000","last_became_active":"2024-09-11T03:50:02.053513+0000","last_became_peered":"2024-09-11T03:50:02.053513+0000","last_unstale":"2024-09-11T03:50:47.328693+0000","last_undegraded":"2024-09-11T03:50:47.328693+0000","last_fullsized":"2024-09-11T03:50:47.328693+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"784'551","last_scrub_stamp":"2024-09-11T03:50:02.647690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.647690+0000","objects_scrubbed":4,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:27.565529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010128489999999999,"stat_sum":{"num_bytes":6367447,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":1276,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.052595+0000","last_change":"2024-09-11T03:47:23.339646+0000","last_active":"2024-09-11T03:50:46.052595+0000","last_peered":"2024-09-11T03:50:46.052595+0000","last_clean":"2024-09-11T03:50:46.052595+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:50:46.052595+0000","last_undegraded":"2024-09-11T03:50:46.052595+0000","last_fullsized":"2024-09-11T03:50:46.052595+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:23.339595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:23.339595+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:50.634356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":2819,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.051115+0000","last_change":"2024-09-11T03:50:45.073145+0000","last_active":"2024-09-11T03:50:46.051115+0000","last_peered":"2024-09-11T03:50:46.051115+0000","last_clean":"2024-09-11T03:50:46.051115+0000","last_became_active":"2024-09-11T03:50:02.048777+0000","last_became_peered":"2024-09-11T03:50:02.048777+0000","last_unstale":"2024-09-11T03:50:46.051115+0000","last_undegraded":"2024-09-11T03:50:46.051115+0000","last_fullsized":"2024-09-11T03:50:46.051115+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:50:02.696833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.696833+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:18.266684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026725655000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.13","version":"20'92","reported_seq":1381,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.044333+0000","last_change":"2024-09-11T03:47:18.217600+0000","last_active":"2024-09-11T03:50:46.044333+0000","last_peered":"2024-09-11T03:50:46.044333+0000","last_clean":"2024-09-11T03:50:46.044333+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:50:46.044333+0000","last_undegraded":"2024-09-11T03:50:46.044333+0000","last_fullsized":"2024-09-11T03:50:46.044333+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:18.217555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:18.217555+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":2763,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.044956+0000","last_change":"2024-09-11T03:50:45.042903+0000","last_active":"2024-09-11T03:50:46.044956+0000","last_peered":"2024-09-11T03:50:46.044956+0000","last_clean":"2024-09-11T03:50:46.044956+0000","last_became_active":"2024-09-11T03:50:02.057196+0000","last_became_peered":"2024-09-11T03:50:02.057196+0000","last_unstale":"2024-09-11T03:50:46.044956+0000","last_undegraded":"2024-09-11T03:50:46.044956+0000","last_fullsized":"2024-09-11T03:50:46.044956+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:50:03.345563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:03.345563+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:09.790296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056405500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":862,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.048483+0000","last_change":"2024-09-11T03:50:20.772575+0000","last_active":"2024-09-11T03:50:46.048483+0000","last_peered":"2024-09-11T03:50:46.048483+0000","last_clean":"2024-09-11T03:50:46.048483+0000","last_became_active":"2024-09-11T03:46:36.026459+0000","last_became_peered":"2024-09-11T03:46:36.026459+0000","last_unstale":"2024-09-11T03:50:46.048483+0000","last_undegraded":"2024-09-11T03:50:46.048483+0000","last_fullsized":"2024-09-11T03:50:46.048483+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:20.772524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:20.772524+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:26.554678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":3315,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.051056+0000","last_change":"2024-09-11T03:50:45.072771+0000","last_active":"2024-09-11T03:50:46.051056+0000","last_peered":"2024-09-11T03:50:46.051056+0000","last_clean":"2024-09-11T03:50:46.051056+0000","last_became_active":"2024-09-11T03:50:02.052278+0000","last_became_peered":"2024-09-11T03:50:02.052278+0000","last_unstale":"2024-09-11T03:50:46.051056+0000","last_undegraded":"2024-09-11T03:50:46.051056+0000","last_fullsized":"2024-09-11T03:50:46.051056+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:50:03.690374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:03.690374+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:17.003362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.026541608000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":1257,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.048348+0000","last_change":"2024-09-11T03:50:19.741484+0000","last_active":"2024-09-11T03:50:46.048348+0000","last_peered":"2024-09-11T03:50:46.048348+0000","last_clean":"2024-09-11T03:50:46.048348+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:50:46.048348+0000","last_undegraded":"2024-09-11T03:50:46.048348+0000","last_fullsized":"2024-09-11T03:50:46.048348+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:19.741428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:19.741428+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:36.778923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"782'561","reported_seq":4288,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.053050+0000","last_change":"2024-09-11T03:50:45.048131+0000","last_active":"2024-09-11T03:50:46.053050+0000","last_peered":"2024-09-11T03:50:46.053050+0000","last_clean":"2024-09-11T03:50:46.053050+0000","last_became_active":"2024-09-11T03:50:02.054667+0000","last_became_peered":"2024-09-11T03:50:02.054667+0000","last_unstale":"2024-09-11T03:50:46.053050+0000","last_undegraded":"2024-09-11T03:50:46.053050+0000","last_fullsized":"2024-09-11T03:50:46.053050+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"782'561","last_scrub_stamp":"2024-09-11T03:50:02.696931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.696931+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:03.112751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00077139600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":738,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.046666+0000","last_change":"2024-09-11T03:50:37.230454+0000","last_active":"2024-09-11T03:50:47.046666+0000","last_peered":"2024-09-11T03:50:47.046666+0000","last_clean":"2024-09-11T03:50:47.046666+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:50:47.046666+0000","last_undegraded":"2024-09-11T03:50:47.046666+0000","last_fullsized":"2024-09-11T03:50:47.046666+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:37.230388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.230388+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:04.405605+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":3975,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-09-11T03:50:48.067298+0000","last_change":"2024-09-11T03:50:48.067480+0000","last_active":"2024-09-11T03:50:48.067298+0000","last_peered":"2024-09-11T03:50:48.067298+0000","last_clean":"2024-09-11T03:50:48.067298+0000","last_became_active":"2024-09-11T03:50:02.051850+0000","last_became_peered":"2024-09-11T03:50:02.051850+0000","last_unstale":"2024-09-11T03:50:48.067298+0000","last_undegraded":"2024-09-11T03:50:48.067298+0000","last_fullsized":"2024-09-11T03:50:48.067298+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:50:04.004006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.004006+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:26.496641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088338700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"},{"start":"173","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":1277,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.045256+0000","last_change":"2024-09-11T03:50:43.547521+0000","last_active":"2024-09-11T03:50:46.045256+0000","last_peered":"2024-09-11T03:50:46.045256+0000","last_clean":"2024-09-11T03:50:46.045256+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:50:46.045256+0000","last_undegraded":"2024-09-11T03:50:46.045256+0000","last_fullsized":"2024-09-11T03:50:46.045256+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:43.547475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:43.547475+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:57.034349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"778'353","reported_seq":3949,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.050776+0000","last_change":"2024-09-11T03:50:45.070119+0000","last_active":"2024-09-11T03:50:46.050776+0000","last_peered":"2024-09-11T03:50:46.050776+0000","last_clean":"2024-09-11T03:50:46.050776+0000","last_became_active":"2024-09-11T03:50:02.054211+0000","last_became_peered":"2024-09-11T03:50:02.054211+0000","last_unstale":"2024-09-11T03:50:46.050776+0000","last_undegraded":"2024-09-11T03:50:46.050776+0000","last_fullsized":"2024-09-11T03:50:46.050776+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"778'353","last_scrub_stamp":"2024-09-11T03:50:04.646331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.646331+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:07.749156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.024452181999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.18","version":"20'92","reported_seq":1388,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.052603+0000","last_change":"2024-09-11T03:49:56.630710+0000","last_active":"2024-09-11T03:50:46.052603+0000","last_peered":"2024-09-11T03:50:46.052603+0000","last_clean":"2024-09-11T03:50:46.052603+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:50:46.052603+0000","last_undegraded":"2024-09-11T03:50:46.052603+0000","last_fullsized":"2024-09-11T03:50:46.052603+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:56.630651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:56.630651+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:18.735201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"823'885","reported_seq":4741,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.046333+0000","last_change":"2024-09-11T03:50:45.046861+0000","last_active":"2024-09-11T03:50:47.046333+0000","last_peered":"2024-09-11T03:50:47.046333+0000","last_clean":"2024-09-11T03:50:47.046333+0000","last_became_active":"2024-09-11T03:50:31.047148+0000","last_became_peered":"2024-09-11T03:50:31.047148+0000","last_unstale":"2024-09-11T03:50:47.046333+0000","last_undegraded":"2024-09-11T03:50:47.046333+0000","last_fullsized":"2024-09-11T03:50:47.046333+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":5,"last_scrub":"816'883","last_scrub_stamp":"2024-09-11T03:50:35.180324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.180324+0000","objects_scrubbed":5,"log_size":885,"log_dups_size":0,"ondisk_log_size":885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:51.145887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001145053,"stat_sum":{"num_bytes":6370837,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.19","version":"20'92","reported_seq":876,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.328639+0000","last_change":"2024-09-11T03:48:56.129575+0000","last_active":"2024-09-11T03:50:47.328639+0000","last_peered":"2024-09-11T03:50:47.328639+0000","last_clean":"2024-09-11T03:50:47.328639+0000","last_became_active":"2024-09-11T03:46:15.972841+0000","last_became_peered":"2024-09-11T03:46:15.972841+0000","last_unstale":"2024-09-11T03:50:47.328639+0000","last_undegraded":"2024-09-11T03:50:47.328639+0000","last_fullsized":"2024-09-11T03:50:47.328639+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:56.129516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:56.129516+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:26.739103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"557'273","reported_seq":3905,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.050703+0000","last_change":"2024-09-11T03:50:45.069710+0000","last_active":"2024-09-11T03:50:46.050703+0000","last_peered":"2024-09-11T03:50:46.050703+0000","last_clean":"2024-09-11T03:50:46.050703+0000","last_became_active":"2024-09-11T03:50:28.038587+0000","last_became_peered":"2024-09-11T03:50:28.038587+0000","last_unstale":"2024-09-11T03:50:46.050703+0000","last_undegraded":"2024-09-11T03:50:46.050703+0000","last_fullsized":"2024-09-11T03:50:46.050703+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:50:37.565876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.565876+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:58.686607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.024308924999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,2],"acting":[6,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.1a","version":"20'92","reported_seq":419,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.048092+0000","last_change":"2024-09-11T03:50:01.843384+0000","last_active":"2024-09-11T03:50:46.048092+0000","last_peered":"2024-09-11T03:50:46.048092+0000","last_clean":"2024-09-11T03:50:46.048092+0000","last_became_active":"2024-09-11T03:49:30.162694+0000","last_became_peered":"2024-09-11T03:49:30.162694+0000","last_unstale":"2024-09-11T03:50:46.048092+0000","last_undegraded":"2024-09-11T03:50:46.048092+0000","last_fullsized":"2024-09-11T03:50:46.048092+0000","mapping_epoch":751,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":752,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:01.843302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:01.843302+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:12.888176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.19","version":"827'589","reported_seq":4352,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.050794+0000","last_change":"2024-09-11T03:50:45.069351+0000","last_active":"2024-09-11T03:50:46.050794+0000","last_peered":"2024-09-11T03:50:46.050794+0000","last_clean":"2024-09-11T03:50:46.050794+0000","last_became_active":"2024-09-11T03:50:28.039694+0000","last_became_peered":"2024-09-11T03:50:28.039694+0000","last_unstale":"2024-09-11T03:50:46.050794+0000","last_undegraded":"2024-09-11T03:50:46.050794+0000","last_fullsized":"2024-09-11T03:50:46.050794+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"816'587","last_scrub_stamp":"2024-09-11T03:50:36.522195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.522195+0000","objects_scrubbed":7,"log_size":589,"log_dups_size":0,"ondisk_log_size":589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:47.425807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023949062,"stat_sum":{"num_bytes":11704959,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.1b","version":"20'92","reported_seq":1246,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.045058+0000","last_change":"2024-09-11T03:48:31.033896+0000","last_active":"2024-09-11T03:50:46.045058+0000","last_peered":"2024-09-11T03:50:46.045058+0000","last_clean":"2024-09-11T03:50:46.045058+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:50:46.045058+0000","last_undegraded":"2024-09-11T03:50:46.045058+0000","last_fullsized":"2024-09-11T03:50:46.045058+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:31.033837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:31.033837+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:46.181076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"827'574","reported_seq":4363,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-09-11T03:50:48.066457+0000","last_change":"2024-09-11T03:50:48.067241+0000","last_active":"2024-09-11T03:50:48.066457+0000","last_peered":"2024-09-11T03:50:48.066457+0000","last_clean":"2024-09-11T03:50:48.066457+0000","last_became_active":"2024-09-11T03:50:28.044833+0000","last_became_peered":"2024-09-11T03:50:28.044833+0000","last_unstale":"2024-09-11T03:50:48.066457+0000","last_undegraded":"2024-09-11T03:50:48.066457+0000","last_fullsized":"2024-09-11T03:50:48.066457+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"816'566","last_scrub_stamp":"2024-09-11T03:50:35.256345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.256345+0000","objects_scrubbed":11,"log_size":574,"log_dups_size":0,"ondisk_log_size":574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:55.750111+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00098061400000000009,"stat_sum":{"num_bytes":23841447,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"},{"start":"173","length":"1"}]},{"pgid":"2.1f","version":"823'773","reported_seq":4438,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-09-11T03:50:50.086485+0000","last_change":"2024-09-11T03:50:48.052411+0000","last_active":"2024-09-11T03:50:50.086485+0000","last_peered":"2024-09-11T03:50:50.086485+0000","last_clean":"2024-09-11T03:50:50.086485+0000","last_became_active":"2024-09-11T03:50:28.041110+0000","last_became_peered":"2024-09-11T03:50:28.041110+0000","last_unstale":"2024-09-11T03:50:50.086485+0000","last_undegraded":"2024-09-11T03:50:50.086485+0000","last_fullsized":"2024-09-11T03:50:50.086485+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"818'769","last_scrub_stamp":"2024-09-11T03:50:37.881096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.881096+0000","objects_scrubbed":13,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:52.183637+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001126555,"stat_sum":{"num_bytes":29464454,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"},{"start":"173","length":"1"}]},{"pgid":"1.1c","version":"20'92","reported_seq":1253,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.045374+0000","last_change":"2024-09-11T03:49:40.156559+0000","last_active":"2024-09-11T03:50:46.045374+0000","last_peered":"2024-09-11T03:50:46.045374+0000","last_clean":"2024-09-11T03:50:46.045374+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:50:46.045374+0000","last_undegraded":"2024-09-11T03:50:46.045374+0000","last_fullsized":"2024-09-11T03:50:46.045374+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:40.156495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:40.156495+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:04.047069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1e","version":"832'1072","reported_seq":4745,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-09-11T03:50:50.099035+0000","last_change":"2024-09-11T03:50:48.055887+0000","last_active":"2024-09-11T03:50:50.099035+0000","last_peered":"2024-09-11T03:50:50.099035+0000","last_clean":"2024-09-11T03:50:50.099035+0000","last_became_active":"2024-09-11T03:50:28.044451+0000","last_became_peered":"2024-09-11T03:50:28.044451+0000","last_unstale":"2024-09-11T03:50:50.099035+0000","last_undegraded":"2024-09-11T03:50:50.099035+0000","last_fullsized":"2024-09-11T03:50:50.099035+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"806'1060","last_scrub_stamp":"2024-09-11T03:50:35.880642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.880642+0000","objects_scrubbed":2,"log_size":1072,"log_dups_size":0,"ondisk_log_size":1072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:04.798097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049103870000000004,"stat_sum":{"num_bytes":8257533,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":11270,"num_write":10,"num_write_kb":3430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"},{"start":"173","length":"1"}]},{"pgid":"1.1d","version":"20'92","reported_seq":1235,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-11T03:50:46.044637+0000","last_change":"2024-09-11T03:47:22.312507+0000","last_active":"2024-09-11T03:50:46.044637+0000","last_peered":"2024-09-11T03:50:46.044637+0000","last_clean":"2024-09-11T03:50:46.044637+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:50:46.044637+0000","last_undegraded":"2024-09-11T03:50:46.044637+0000","last_fullsized":"2024-09-11T03:50:46.044637+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:22.312456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:22.312456+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:50:49.239645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"567'752","reported_seq":4401,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.328980+0000","last_change":"2024-09-11T03:50:45.056049+0000","last_active":"2024-09-11T03:50:47.328980+0000","last_peered":"2024-09-11T03:50:47.328980+0000","last_clean":"2024-09-11T03:50:47.328980+0000","last_became_active":"2024-09-11T03:50:28.045687+0000","last_became_peered":"2024-09-11T03:50:28.045687+0000","last_unstale":"2024-09-11T03:50:47.328980+0000","last_undegraded":"2024-09-11T03:50:47.328980+0000","last_fullsized":"2024-09-11T03:50:47.328980+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:50:38.812209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:38.812209+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:45.184370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00115055,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,11],"acting":[10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"135","length":"1"},{"start":"148","length":"1"}]},{"pgid":"1.1e","version":"20'92","reported_seq":1263,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-11T03:50:47.046754+0000","last_change":"2024-09-11T03:49:36.949199+0000","last_active":"2024-09-11T03:50:47.046754+0000","last_peered":"2024-09-11T03:50:47.046754+0000","last_clean":"2024-09-11T03:50:47.046754+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:50:47.046754+0000","last_undegraded":"2024-09-11T03:50:47.046754+0000","last_fullsized":"2024-09-11T03:50:47.046754+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:36.949149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.949149+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":66,"stat_sum":{"num_bytes":637873092,"num_objects":289,"num_object_clones":239,"num_object_copies":867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":17,"num_read":641,"num_read_kb":653075,"num_write":472,"num_write_kb":213592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1044553728,"data_stored":1475868064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38067,"ondisk_log_size":38067,"up":198,"acting":198,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285731,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":116852,"kb_used_data":85660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93628300,"statfs":{"total":95995035648,"available":95875379200,"internally_reserved":0,"allocated":87715840,"data_stored":121182428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514663,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92788,"kb_used_data":65756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93652364,"statfs":{"total":95995035648,"available":95900020736,"internally_reserved":0,"allocated":67334144,"data_stored":76656210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743597,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":125136,"kb_used_data":67572,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93620016,"statfs":{"total":95995035648,"available":95866896384,"internally_reserved":0,"allocated":69193728,"data_stored":76665496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037940,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":184196,"kb_used_data":126632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93560956,"statfs":{"total":95995035648,"available":95806418944,"internally_reserved":0,"allocated":129671168,"data_stored":191546413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299575,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":202664,"kb_used_data":157580,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93542488,"statfs":{"total":95995035648,"available":95787507712,"internally_reserved":0,"allocated":161361920,"data_stored":216198536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528508,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":156472,"kb_used_data":129440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93588680,"statfs":{"total":95995035648,"available":95834808320,"internally_reserved":0,"allocated":132546560,"data_stored":181867986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757442,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111760,"kb_used_data":80564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93633392,"statfs":{"total":95995035648,"available":95880593408,"internally_reserved":0,"allocated":82497536,"data_stored":110002868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986376,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":115800,"kb_used_data":62400,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93629352,"statfs":{"total":95995035648,"available":95876456448,"internally_reserved":0,"allocated":63897600,"data_stored":71630166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149900,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":184600,"kb_used_data":135360,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93560552,"statfs":{"total":95995035648,"available":95806005248,"internally_reserved":0,"allocated":138608640,"data_stored":200373690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411538,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84088,"kb_used_data":52896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93661064,"statfs":{"total":95995035648,"available":95908929536,"internally_reserved":0,"allocated":54165504,"data_stored":63757107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575063,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":144960,"kb_used_data":87404,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93600192,"statfs":{"total":95995035648,"available":95846596608,"internally_reserved":0,"allocated":89501696,"data_stored":119789784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738589,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":135360,"kb_used_data":81960,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93609792,"statfs":{"total":95995035648,"available":95856427008,"internally_reserved":0,"allocated":83927040,"data_stored":112382265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73515008,"data_stored":105470253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79466496,"data_stored":113238764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":43913216,"data_stored":56996537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":128221184,"data_stored":193478314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53825536,"data_stored":65054152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":72257536,"data_stored":103250600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":122183680,"data_stored":174989286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":154296320,"data_stored":216812798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":122761216,"data_stored":191742962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":59133952,"data_stored":70097800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":57298944,"data_stored":70105190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":77680640,"data_stored":114631408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:50:52.647 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pgp_num 61 2024-09-11T03:50:52.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:52.733 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:52.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:52.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:52.835 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:52.857 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:52.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:52.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:52.960 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:52.982 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:52.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:52.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:53.043 INFO:tasks.rados.rados.2.smithi175.stdout:3449: done (1 left) 2024-09-11T03:50:53.043 INFO:tasks.rados.rados.2.smithi175.stdout:3450: write (excl) oid 4 current snap is 383 2024-09-11T03:50:53.043 INFO:tasks.rados.rados.2.smithi175.stdout:3448: finishing write tid 1 to smithi17557870-18 2024-09-11T03:50:53.043 INFO:tasks.rados.rados.2.smithi175.stdout:3450: seq_num 1169 ranges {447383=568090,1690509=701410,2902655=646983} 2024-09-11T03:50:53.049 INFO:tasks.rados.rados.2.smithi175.stdout:3450: writing smithi17557870-4 from 447383 to 1015473 tid 1 2024-09-11T03:50:53.057 INFO:tasks.rados.rados.2.smithi175.stdout:3450: writing smithi17557870-4 from 1690509 to 2391919 tid 2 2024-09-11T03:50:53.064 INFO:tasks.rados.rados.2.smithi175.stdout:3450: writing smithi17557870-4 from 2902655 to 3549638 tid 3 2024-09-11T03:50:53.064 INFO:tasks.rados.rados.2.smithi175.stdout:3451: write (excl) oid 43 current snap is 383 2024-09-11T03:50:53.064 INFO:tasks.rados.rados.2.smithi175.stdout:3448: finishing write tid 2 to smithi17557870-18 2024-09-11T03:50:53.064 INFO:tasks.rados.rados.2.smithi175.stdout:3448: finishing write tid 3 to smithi17557870-18 2024-09-11T03:50:53.065 INFO:tasks.rados.rados.2.smithi175.stdout:3451: seq_num 1170 ranges {464190=643339,1748991=445068,2930078=574608} 2024-09-11T03:50:53.070 INFO:tasks.rados.rados.2.smithi175.stdout:3451: writing smithi17557870-43 from 464190 to 1107529 tid 1 2024-09-11T03:50:53.074 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:53.076 INFO:tasks.rados.rados.2.smithi175.stdout:3451: writing smithi17557870-43 from 1748991 to 2194059 tid 2 2024-09-11T03:50:53.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:53.085 INFO:tasks.rados.rados.2.smithi175.stdout:3451: writing smithi17557870-43 from 2930078 to 3504686 tid 3 2024-09-11T03:50:53.085 INFO:tasks.rados.rados.2.smithi175.stdout:3452: read oid 37 snap -1 2024-09-11T03:50:53.085 INFO:tasks.rados.rados.2.smithi175.stdout:3452: expect deleted 2024-09-11T03:50:53.085 INFO:tasks.rados.rados.2.smithi175.stdout:3453: snap_create 2024-09-11T03:50:53.085 INFO:tasks.rados.rados.2.smithi175.stdout:3448: finishing write tid 4 to smithi17557870-18 2024-09-11T03:50:53.085 INFO:tasks.rados.rados.2.smithi175.stdout:3448: finishing write tid 5 to smithi17557870-18 2024-09-11T03:50:53.085 INFO:tasks.rados.rados.2.smithi175.stdout:3448: finishing write tid 6 to smithi17557870-18 2024-09-11T03:50:53.085 INFO:tasks.rados.rados.2.smithi175.stdout:3448: oid 18 updating version 0 to 942 2024-09-11T03:50:53.086 INFO:tasks.rados.rados.2.smithi175.stdout:3448: oid 18 updating version 942 to 944 2024-09-11T03:50:53.086 INFO:tasks.rados.rados.2.smithi175.stdout:3448: oid 18 version 944 is already newer than 943 2024-09-11T03:50:53.086 INFO:tasks.rados.rados.2.smithi175.stdout:3448: oid 18 updating version 944 to 946 2024-09-11T03:50:53.086 INFO:tasks.rados.rados.2.smithi175.stdout:3448: oid 18 version 946 is already newer than 945 2024-09-11T03:50:53.086 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 946 (ObjNum 1168 snap 383 seq_num 1168) dirty exists 2024-09-11T03:50:53.086 INFO:tasks.rados.rados.2.smithi175.stdout:3448: left oid 18 (ObjNum 1168 snap 383 seq_num 1168) 2024-09-11T03:50:53.086 INFO:tasks.rados.rados.2.smithi175.stdout:3450: finishing write tid 1 to smithi17557870-4 2024-09-11T03:50:53.086 INFO:tasks.rados.rados.2.smithi175.stdout:3450: finishing write tid 2 to smithi17557870-4 2024-09-11T03:50:53.086 INFO:tasks.rados.rados.2.smithi175.stdout:3450: finishing write tid 3 to smithi17557870-4 2024-09-11T03:50:53.086 INFO:tasks.rados.rados.2.smithi175.stdout:3450: finishing write tid 4 to smithi17557870-4 2024-09-11T03:50:53.086 INFO:tasks.rados.rados.2.smithi175.stdout:3450: finishing write tid 5 to smithi17557870-4 2024-09-11T03:50:53.086 INFO:tasks.rados.rados.2.smithi175.stdout:3450: oid 4 updating version 0 to 1004 2024-09-11T03:50:53.086 INFO:tasks.rados.rados.2.smithi175.stdout:3450: oid 4 version 1004 is already newer than 1002 2024-09-11T03:50:53.086 INFO:tasks.rados.rados.2.smithi175.stdout:3450: oid 4 version 1004 is already newer than 1003 2024-09-11T03:50:53.086 INFO:tasks.rados.rados.2.smithi175.stdout:3450: oid 4 updating version 1004 to 1005 2024-09-11T03:50:53.087 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 1005 (ObjNum 1169 snap 383 seq_num 1169) dirty exists 2024-09-11T03:50:53.087 INFO:tasks.rados.rados.2.smithi175.stdout:3450: left oid 4 (ObjNum 1169 snap 383 seq_num 1169) 2024-09-11T03:50:53.087 INFO:tasks.rados.rados.2.smithi175.stdout:3451: finishing write tid 1 to smithi17557870-43 2024-09-11T03:50:53.087 INFO:tasks.rados.rados.2.smithi175.stdout:3451: finishing write tid 2 to smithi17557870-43 2024-09-11T03:50:53.088 INFO:tasks.rados.rados.2.smithi175.stdout:3451: finishing write tid 3 to smithi17557870-43 2024-09-11T03:50:53.088 INFO:tasks.rados.rados.2.smithi175.stdout:3451: finishing write tid 4 to smithi17557870-43 2024-09-11T03:50:53.088 INFO:tasks.rados.rados.2.smithi175.stdout:3451: finishing write tid 5 to smithi17557870-43 2024-09-11T03:50:53.088 INFO:tasks.rados.rados.2.smithi175.stdout:3451: oid 43 updating version 0 to 1016 2024-09-11T03:50:53.088 INFO:tasks.rados.rados.2.smithi175.stdout:3451: oid 43 version 1016 is already newer than 1015 2024-09-11T03:50:53.088 INFO:tasks.rados.rados.2.smithi175.stdout:3451: oid 43 updating version 1016 to 1017 2024-09-11T03:50:53.088 INFO:tasks.rados.rados.2.smithi175.stdout:3451: oid 43 version 1017 is already newer than 1014 2024-09-11T03:50:53.088 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 1017 (ObjNum 1170 snap 383 seq_num 1170) dirty exists 2024-09-11T03:50:53.089 INFO:tasks.rados.rados.2.smithi175.stdout:3451: left oid 43 (ObjNum 1170 snap 383 seq_num 1170) 2024-09-11T03:50:53.110 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:53.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:53.112 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:50:53.112 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:50:53.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:53.235 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:53.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:53.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:53.337 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:53.359 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:53.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:53.461 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:50:53.487 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:53.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:53.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:50:53.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:53.615 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:53.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:53.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:53.717 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:53.742 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:53.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:53.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:53.810 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:53.837 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:50:53.844 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:53.900 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:53.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:53.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:54.002 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:54.027 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:54.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:54.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:54.072 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:50:54.075 INFO:tasks.rados.rados.2.smithi175.stdout:3448: done (4 left) 2024-09-11T03:50:54.076 INFO:tasks.rados.rados.2.smithi175.stdout:3450: done (3 left) 2024-09-11T03:50:54.076 INFO:tasks.rados.rados.2.smithi175.stdout:3451: done (2 left) 2024-09-11T03:50:54.076 INFO:tasks.rados.rados.2.smithi175.stdout:3452: done (1 left) 2024-09-11T03:50:54.076 INFO:tasks.rados.rados.2.smithi175.stdout:3453: done (0 left) 2024-09-11T03:50:54.076 INFO:tasks.rados.rados.2.smithi175.stdout:3454: read oid 15 snap 366 2024-09-11T03:50:54.076 INFO:tasks.rados.rados.2.smithi175.stdout:3454: expect (ObjNum 1120 snap 359 seq_num 1120) 2024-09-11T03:50:54.076 INFO:tasks.rados.rados.2.smithi175.stdout:3455: delete oid 50 current snap is 384 2024-09-11T03:50:54.088 INFO:tasks.rados.rados.2.smithi175.stdout:3454: expect (ObjNum 1120 snap 359 seq_num 1120) 2024-09-11T03:50:54.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:54.155 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:54.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:54.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:54.177 INFO:tasks.rados.rados.2.smithi175.stdout:3454: done (1 left) 2024-09-11T03:50:54.177 INFO:tasks.rados.rados.2.smithi175.stdout:3455: done (0 left) 2024-09-11T03:50:54.177 INFO:tasks.rados.rados.2.smithi175.stdout:3456: read oid 42 snap -1 2024-09-11T03:50:54.177 INFO:tasks.rados.rados.2.smithi175.stdout:3456: expect (ObjNum 1042 snap 340 seq_num 1042) 2024-09-11T03:50:54.177 INFO:tasks.rados.rados.2.smithi175.stdout:3457: copy_from oid 28 from oid 19 current snap is 384 2024-09-11T03:50:54.177 INFO:tasks.rados.rados.2.smithi175.stdout:3458: write (excl) oid 13 current snap is 384 2024-09-11T03:50:54.177 INFO:tasks.rados.rados.2.smithi175.stdout:3458: seq_num 1172 ranges {497804=510190,1465955=448737,2584924=597152} 2024-09-11T03:50:54.182 INFO:tasks.rados.rados.2.smithi175.stdout:3458: writing smithi17557870-13 from 497804 to 1007994 tid 1 2024-09-11T03:50:54.188 INFO:tasks.rados.rados.2.smithi175.stdout:3458: writing smithi17557870-13 from 1465955 to 1914692 tid 2 2024-09-11T03:50:54.194 INFO:tasks.rados.rados.2.smithi175.stdout:3458: writing smithi17557870-13 from 2584924 to 3182076 tid 3 2024-09-11T03:50:54.194 INFO:tasks.rados.rados.2.smithi175.stdout:3459: snap_remove snap 340 2024-09-11T03:50:54.257 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:54.281 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:54.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:54.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:54.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:54.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:54 smithi084 ceph-mon[27254]: pgmap v1407: 97 pgs: 97 active+clean; 624 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2024-09-11T03:50:54.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:54 smithi084 ceph-mon[27254]: osdmap e835: 12 total, 12 up, 12 in 2024-09-11T03:50:54.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:54 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/574371397' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:50:54.407 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:54.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:54.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:54.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:54 smithi121 ceph-mon[30729]: pgmap v1407: 97 pgs: 97 active+clean; 624 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2024-09-11T03:50:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:54 smithi121 ceph-mon[30729]: osdmap e835: 12 total, 12 up, 12 in 2024-09-11T03:50:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:54 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/574371397' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:50:54.455 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: specified pgp_num 61 > pg_num 31 2024-09-11T03:50:54.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:54 smithi175 ceph-mon[30894]: pgmap v1407: 97 pgs: 97 active+clean; 624 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2024-09-11T03:50:54.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:54 smithi175 ceph-mon[30894]: osdmap e835: 12 total, 12 up, 12 in 2024-09-11T03:50:54.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:54 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/574371397' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:50:54.509 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:54.532 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:54.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:54.533 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:54.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:54.656 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:54.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:54.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:54.758 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:54.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:54.780 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:54.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:54.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:54.907 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:54.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:54.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:55.008 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:55.033 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:55.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:55.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:55.083 INFO:tasks.rados.rados.2.smithi175.stdout:3459: done (3 left) 2024-09-11T03:50:55.083 INFO:tasks.rados.rados.2.smithi175.stdout:3460: write oid 20 current snap is 384 2024-09-11T03:50:55.083 INFO:tasks.rados.rados.2.smithi175.stdout:3460: seq_num 1173 ranges {514611=585439,1524437=592395,2895994=408424} 2024-09-11T03:50:55.089 INFO:tasks.rados.rados.2.smithi175.stdout:3460: writing smithi17557870-20 from 514611 to 1100050 tid 1 2024-09-11T03:50:55.096 INFO:tasks.rados.rados.2.smithi175.stdout:3460: writing smithi17557870-20 from 1524437 to 2116832 tid 2 2024-09-11T03:50:55.103 INFO:tasks.rados.rados.2.smithi175.stdout:3460: writing smithi17557870-20 from 2895994 to 3304418 tid 3 2024-09-11T03:50:55.103 INFO:tasks.rados.rados.2.smithi175.stdout:3461: read oid 17 snap 321 2024-09-11T03:50:55.103 INFO:tasks.rados.rados.2.smithi175.stdout:3461: expect (ObjNum 993 snap 320 seq_num 993) 2024-09-11T03:50:55.103 INFO:tasks.rados.rados.2.smithi175.stdout:3462: read oid 23 snap -1 2024-09-11T03:50:55.103 INFO:tasks.rados.rados.2.smithi175.stdout:3462: expect (ObjNum 1079 snap 352 seq_num 1079) 2024-09-11T03:50:55.103 INFO:tasks.rados.rados.2.smithi175.stdout:3463: copy_from oid 2 from oid 8 current snap is 384 2024-09-11T03:50:55.103 INFO:tasks.rados.rados.2.smithi175.stdout:3456: expect (ObjNum 1042 snap 340 seq_num 1042) 2024-09-11T03:50:55.135 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:55.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:55.158 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:55.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:55.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:55.261 INFO:tasks.rados.rados.2.smithi175.stdout:3458: finishing write tid 1 to smithi17557870-13 2024-09-11T03:50:55.261 INFO:tasks.rados.rados.2.smithi175.stdout:3457: finishing copy_from racing read to smithi17557870-28 2024-09-11T03:50:55.261 INFO:tasks.rados.rados.2.smithi175.stdout:3458: finishing write tid 2 to smithi17557870-13 2024-09-11T03:50:55.261 INFO:tasks.rados.rados.2.smithi175.stdout:3457: finishing copy_from to smithi17557870-28 2024-09-11T03:50:55.262 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 1005 (ObjNum 1159 snap 379 seq_num 1159) dirty exists 2024-09-11T03:50:55.262 INFO:tasks.rados.rados.2.smithi175.stdout:3458: finishing write tid 3 to smithi17557870-13 2024-09-11T03:50:55.262 INFO:tasks.rados.rados.2.smithi175.stdout:3456: done (6 left) 2024-09-11T03:50:55.262 INFO:tasks.rados.rados.2.smithi175.stdout:3457: done (5 left) 2024-09-11T03:50:55.262 INFO:tasks.rados.rados.2.smithi175.stdout:3464: rollback oid 18 current snap is 384 2024-09-11T03:50:55.262 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 18 to 347 2024-09-11T03:50:55.263 INFO:tasks.rados.rados.2.smithi175.stdout:3458: finishing write tid 4 to smithi17557870-13 2024-09-11T03:50:55.263 INFO:tasks.rados.rados.2.smithi175.stdout:3458: finishing write tid 5 to smithi17557870-13 2024-09-11T03:50:55.263 INFO:tasks.rados.rados.2.smithi175.stdout:3458: oid 13 updating version 0 to 1330 2024-09-11T03:50:55.263 INFO:tasks.rados.rados.2.smithi175.stdout:3458: oid 13 version 1330 is already newer than 1328 2024-09-11T03:50:55.263 INFO:tasks.rados.rados.2.smithi175.stdout:3458: oid 13 version 1330 is already newer than 1329 2024-09-11T03:50:55.263 INFO:tasks.rados.rados.2.smithi175.stdout:3458: oid 13 updating version 1330 to 1331 2024-09-11T03:50:55.263 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 1331 (ObjNum 1172 snap 384 seq_num 1172) dirty exists 2024-09-11T03:50:55.263 INFO:tasks.rados.rados.2.smithi175.stdout:3458: left oid 13 (ObjNum 1172 snap 384 seq_num 1172) 2024-09-11T03:50:55.263 INFO:tasks.rados.rados.2.smithi175.stdout:3460: finishing write tid 1 to smithi17557870-20 2024-09-11T03:50:55.263 INFO:tasks.rados.rados.2.smithi175.stdout:3460: finishing write tid 2 to smithi17557870-20 2024-09-11T03:50:55.263 INFO:tasks.rados.rados.2.smithi175.stdout:3460: finishing write tid 3 to smithi17557870-20 2024-09-11T03:50:55.263 INFO:tasks.rados.rados.2.smithi175.stdout:3460: finishing write tid 4 to smithi17557870-20 2024-09-11T03:50:55.263 INFO:tasks.rados.rados.2.smithi175.stdout:3460: finishing write tid 5 to smithi17557870-20 2024-09-11T03:50:55.263 INFO:tasks.rados.rados.2.smithi175.stdout:3460: oid 20 updating version 0 to 1024 2024-09-11T03:50:55.264 INFO:tasks.rados.rados.2.smithi175.stdout:3460: oid 20 updating version 1024 to 1025 2024-09-11T03:50:55.264 INFO:tasks.rados.rados.2.smithi175.stdout:3460: oid 20 updating version 1025 to 1026 2024-09-11T03:50:55.264 INFO:tasks.rados.rados.2.smithi175.stdout:3460: oid 20 version 1026 is already newer than 1023 2024-09-11T03:50:55.264 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 1026 (ObjNum 1173 snap 384 seq_num 1173) dirty exists 2024-09-11T03:50:55.264 INFO:tasks.rados.rados.2.smithi175.stdout:3460: left oid 20 (ObjNum 1173 snap 384 seq_num 1173) 2024-09-11T03:50:55.264 INFO:tasks.rados.rados.2.smithi175.stdout:3462: expect (ObjNum 1079 snap 352 seq_num 1079) 2024-09-11T03:50:55.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:55.281 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:55.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:55.382 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:55.385 INFO:tasks.rados.rados.2.smithi175.stdout:3461: expect (ObjNum 993 snap 320 seq_num 993) 2024-09-11T03:50:55.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:55 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/574371397' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:50:55.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:55 smithi084 ceph-mon[27254]: osdmap e836: 12 total, 12 up, 12 in 2024-09-11T03:50:55.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:55 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1496115347' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-11T03:50:55.404 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:55.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:55 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/574371397' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:50:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:55 smithi121 ceph-mon[30729]: osdmap e836: 12 total, 12 up, 12 in 2024-09-11T03:50:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:55 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1496115347' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-11T03:50:55.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:55 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/574371397' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:50:55.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:55 smithi175 ceph-mon[30894]: osdmap e836: 12 total, 12 up, 12 in 2024-09-11T03:50:55.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:55 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1496115347' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-11T03:50:55.506 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:55.527 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:55.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:55.528 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:55.544 INFO:tasks.rados.rados.2.smithi175.stdout:3463: finishing copy_from racing read to smithi17557870-2 2024-09-11T03:50:55.544 INFO:tasks.rados.rados.2.smithi175.stdout:3463: finishing copy_from to smithi17557870-2 2024-09-11T03:50:55.545 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 1073 (ObjNum 1147 snap 370 seq_num 1147) dirty exists 2024-09-11T03:50:55.545 INFO:tasks.rados.rados.2.smithi175.stdout:3464: finishing rollback tid 0 to smithi17557870-18 2024-09-11T03:50:55.545 INFO:tasks.rados.rados.2.smithi175.stdout:3464: finishing rollback tid 1 to smithi17557870-18 2024-09-11T03:50:55.545 INFO:tasks.rados.rados.2.smithi175.stdout:3464: finishing rollback tid 2 to smithi17557870-18 2024-09-11T03:50:55.545 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 949 (ObjNum 829 snap 263 seq_num 829) dirty exists 2024-09-11T03:50:55.545 INFO:tasks.rados.rados.2.smithi175.stdout:3458: done (5 left) 2024-09-11T03:50:55.545 INFO:tasks.rados.rados.2.smithi175.stdout:3460: done (4 left) 2024-09-11T03:50:55.545 INFO:tasks.rados.rados.2.smithi175.stdout:3461: done (3 left) 2024-09-11T03:50:55.545 INFO:tasks.rados.rados.2.smithi175.stdout:3462: done (2 left) 2024-09-11T03:50:55.545 INFO:tasks.rados.rados.2.smithi175.stdout:3463: done (1 left) 2024-09-11T03:50:55.545 INFO:tasks.rados.rados.2.smithi175.stdout:3464: done (0 left) 2024-09-11T03:50:55.545 INFO:tasks.rados.rados.2.smithi175.stdout:3465: rollback oid 43 current snap is 384 2024-09-11T03:50:55.545 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 43 to 380 2024-09-11T03:50:55.545 INFO:tasks.rados.rados.2.smithi175.stdout:3466: rollback oid 24 current snap is 384 2024-09-11T03:50:55.546 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 349 2024-09-11T03:50:55.546 INFO:tasks.rados.rados.2.smithi175.stdout:3467: write (excl) oid 22 current snap is 384 2024-09-11T03:50:55.546 INFO:tasks.rados.rados.2.smithi175.stdout:3467: seq_num 1174 ranges {531418=660688,1982919=452406,3039770=691648} 2024-09-11T03:50:55.550 INFO:tasks.rados.rados.2.smithi175.stdout:3467: writing smithi17557870-22 from 531418 to 1192106 tid 1 2024-09-11T03:50:55.557 INFO:tasks.rados.rados.2.smithi175.stdout:3467: writing smithi17557870-22 from 1982919 to 2435325 tid 2 2024-09-11T03:50:55.564 INFO:tasks.rados.rados.2.smithi175.stdout:3467: writing smithi17557870-22 from 3039770 to 3731418 tid 3 2024-09-11T03:50:55.564 INFO:tasks.rados.rados.2.smithi175.stdout:3468: rollback oid 32 current snap is 384 2024-09-11T03:50:55.564 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 32 to 357 2024-09-11T03:50:55.565 INFO:tasks.rados.rados.2.smithi175.stdout:3466: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:50:55.565 INFO:tasks.rados.rados.2.smithi175.stdout:3465: finishing rollback tid 0 to smithi17557870-43 2024-09-11T03:50:55.565 INFO:tasks.rados.rados.2.smithi175.stdout:3466: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:50:55.565 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 1070 (ObjNum 1029 snap 338 seq_num 1029) dirty exists 2024-09-11T03:50:55.565 INFO:tasks.rados.rados.2.smithi175.stdout:3465: finishing rollback tid 1 to smithi17557870-43 2024-09-11T03:50:55.565 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 1019 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-11T03:50:55.565 INFO:tasks.rados.rados.2.smithi175.stdout:3467: finishing write tid 1 to smithi17557870-22 2024-09-11T03:50:55.565 INFO:tasks.rados.rados.2.smithi175.stdout:3467: finishing write tid 2 to smithi17557870-22 2024-09-11T03:50:55.565 INFO:tasks.rados.rados.2.smithi175.stdout:3465: done (3 left) 2024-09-11T03:50:55.565 INFO:tasks.rados.rados.2.smithi175.stdout:3466: done (2 left) 2024-09-11T03:50:55.565 INFO:tasks.rados.rados.2.smithi175.stdout:3469: delete oid 34 current snap is 384 2024-09-11T03:50:55.566 INFO:tasks.rados.rados.2.smithi175.stdout:3469: done (2 left) 2024-09-11T03:50:55.566 INFO:tasks.rados.rados.2.smithi175.stdout:3470: snap_remove snap 339 2024-09-11T03:50:55.611 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T03:50:55.629 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:55.650 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:55.651 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:55.752 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:55.773 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:55.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:55.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:55.875 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:55.897 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:55.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:55.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:55.999 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:50:56.019 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:56.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:56.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:50:56.106 INFO:tasks.rados.rados.2.smithi175.stdout:3470: done (2 left) 2024-09-11T03:50:56.106 INFO:tasks.rados.rados.2.smithi175.stdout:3471: write oid 6 current snap is 384 2024-09-11T03:50:56.106 INFO:tasks.rados.rados.2.smithi175.stdout:3468: finishing rollback tid 0 to smithi17557870-32 2024-09-11T03:50:56.106 INFO:tasks.rados.rados.2.smithi175.stdout:3468: finishing rollback tid 1 to smithi17557870-32 2024-09-11T03:50:56.106 INFO:tasks.rados.rados.2.smithi175.stdout:3467: finishing write tid 3 to smithi17557870-22 2024-09-11T03:50:56.106 INFO:tasks.rados.rados.2.smithi175.stdout:3468: finishing rollback tid 2 to smithi17557870-32 2024-09-11T03:50:56.107 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 1020 (ObjNum 1059 snap 345 seq_num 1059) dirty exists 2024-09-11T03:50:56.107 INFO:tasks.rados.rados.2.smithi175.stdout:3467: finishing write tid 4 to smithi17557870-22 2024-09-11T03:50:56.107 INFO:tasks.rados.rados.2.smithi175.stdout:3467: finishing write tid 5 to smithi17557870-22 2024-09-11T03:50:56.107 INFO:tasks.rados.rados.2.smithi175.stdout:3467: oid 22 updating version 0 to 1267 2024-09-11T03:50:56.107 INFO:tasks.rados.rados.2.smithi175.stdout:3467: oid 22 updating version 1267 to 1268 2024-09-11T03:50:56.107 INFO:tasks.rados.rados.2.smithi175.stdout:3467: oid 22 updating version 1268 to 1269 2024-09-11T03:50:56.107 INFO:tasks.rados.rados.2.smithi175.stdout:3467: oid 22 updating version 1269 to 1270 2024-09-11T03:50:56.107 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 1270 (ObjNum 1174 snap 384 seq_num 1174) dirty exists 2024-09-11T03:50:56.107 INFO:tasks.rados.rados.2.smithi175.stdout:3467: left oid 22 (ObjNum 1174 snap 384 seq_num 1174) 2024-09-11T03:50:56.107 INFO:tasks.rados.rados.2.smithi175.stdout:3471: seq_num 1176 ranges {565032=411186,1583530=456075,2578263=758593} 2024-09-11T03:50:56.111 INFO:tasks.rados.rados.2.smithi175.stdout:3471: writing smithi17557870-6 from 565032 to 976218 tid 1 2024-09-11T03:50:56.119 INFO:tasks.rados.rados.2.smithi175.stdout:3471: writing smithi17557870-6 from 1583530 to 2039605 tid 2 2024-09-11T03:50:56.121 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:56.127 INFO:tasks.rados.rados.2.smithi175.stdout:3471: writing smithi17557870-6 from 2578263 to 3336856 tid 3 2024-09-11T03:50:56.127 INFO:tasks.rados.rados.2.smithi175.stdout:3467: done (2 left) 2024-09-11T03:50:56.127 INFO:tasks.rados.rados.2.smithi175.stdout:3468: done (1 left) 2024-09-11T03:50:56.127 INFO:tasks.rados.rados.2.smithi175.stdout:3472: rollback oid 27 current snap is 384 2024-09-11T03:50:56.127 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 27 to 359 2024-09-11T03:50:56.127 INFO:tasks.rados.rados.2.smithi175.stdout:3471: finishing write tid 1 to smithi17557870-6 2024-09-11T03:50:56.127 INFO:tasks.rados.rados.2.smithi175.stdout:3471: finishing write tid 2 to smithi17557870-6 2024-09-11T03:50:56.127 INFO:tasks.rados.rados.2.smithi175.stdout:3473: delete oid 39 current snap is 384 2024-09-11T03:50:56.128 INFO:tasks.rados.rados.2.smithi175.stdout:3473: done (2 left) 2024-09-11T03:50:56.128 INFO:tasks.rados.rados.2.smithi175.stdout:3474: write (excl) oid 11 current snap is 384 2024-09-11T03:50:56.128 INFO:tasks.rados.rados.2.smithi175.stdout:3474: seq_num 1178 ranges {598646=561684,1700494=459744,2633109=781137} 2024-09-11T03:50:56.134 INFO:tasks.rados.rados.2.smithi175.stdout:3474: writing smithi17557870-11 from 598646 to 1160330 tid 1 2024-09-11T03:50:56.139 INFO:tasks.rados.rados.2.smithi175.stdout:3474: writing smithi17557870-11 from 1700494 to 2160238 tid 2 2024-09-11T03:50:56.146 INFO:tasks.rados.rados.2.smithi175.stdout:3474: writing smithi17557870-11 from 2633109 to 3414246 tid 3 2024-09-11T03:50:56.147 INFO:tasks.rados.rados.2.smithi175.stdout:3475: read oid 7 snap -1 2024-09-11T03:50:56.147 INFO:tasks.rados.rados.2.smithi175.stdout:3475: expect (ObjNum 1122 snap 360 seq_num 1122) 2024-09-11T03:50:56.147 INFO:tasks.rados.rados.2.smithi175.stdout:3476: snap_create 2024-09-11T03:50:56.147 INFO:tasks.rados.rados.2.smithi175.stdout:3472: finishing rollback tid 0 to smithi17557870-27 2024-09-11T03:50:56.147 INFO:tasks.rados.rados.2.smithi175.stdout:3472: finishing rollback tid 1 to smithi17557870-27 2024-09-11T03:50:56.147 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 1236 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T03:50:56.147 INFO:tasks.rados.rados.2.smithi175.stdout:3471: finishing write tid 3 to smithi17557870-6 2024-09-11T03:50:56.147 INFO:tasks.rados.rados.2.smithi175.stdout:3471: finishing write tid 4 to smithi17557870-6 2024-09-11T03:50:56.147 INFO:tasks.rados.rados.2.smithi175.stdout:3471: finishing write tid 5 to smithi17557870-6 2024-09-11T03:50:56.147 INFO:tasks.rados.rados.2.smithi175.stdout:3471: oid 6 updating version 0 to 983 2024-09-11T03:50:56.147 INFO:tasks.rados.rados.2.smithi175.stdout:3471: oid 6 updating version 983 to 986 2024-09-11T03:50:56.147 INFO:tasks.rados.rados.2.smithi175.stdout:3471: oid 6 version 986 is already newer than 985 2024-09-11T03:50:56.147 INFO:tasks.rados.rados.2.smithi175.stdout:3471: oid 6 version 986 is already newer than 984 2024-09-11T03:50:56.148 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 986 (ObjNum 1176 snap 384 seq_num 1176) dirty exists 2024-09-11T03:50:56.148 INFO:tasks.rados.rados.2.smithi175.stdout:3471: left oid 6 (ObjNum 1176 snap 384 seq_num 1176) 2024-09-11T03:50:56.148 INFO:tasks.rados.rados.2.smithi175.stdout:3474: finishing write tid 1 to smithi17557870-11 2024-09-11T03:50:56.148 INFO:tasks.rados.rados.2.smithi175.stdout:3474: finishing write tid 2 to smithi17557870-11 2024-09-11T03:50:56.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:56.149 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:56.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:56.150 INFO:tasks.rados.rados.2.smithi175.stdout:3474: finishing write tid 3 to smithi17557870-11 2024-09-11T03:50:56.150 INFO:tasks.rados.rados.2.smithi175.stdout:3474: finishing write tid 4 to smithi17557870-11 2024-09-11T03:50:56.150 INFO:tasks.rados.rados.2.smithi175.stdout:3474: finishing write tid 5 to smithi17557870-11 2024-09-11T03:50:56.150 INFO:tasks.rados.rados.2.smithi175.stdout:3474: oid 11 updating version 0 to 637 2024-09-11T03:50:56.150 INFO:tasks.rados.rados.2.smithi175.stdout:3474: oid 11 updating version 637 to 639 2024-09-11T03:50:56.150 INFO:tasks.rados.rados.2.smithi175.stdout:3474: oid 11 version 639 is already newer than 638 2024-09-11T03:50:56.150 INFO:tasks.rados.rados.2.smithi175.stdout:3474: oid 11 version 639 is already newer than 636 2024-09-11T03:50:56.151 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 639 (ObjNum 1178 snap 384 seq_num 1178) dirty exists 2024-09-11T03:50:56.151 INFO:tasks.rados.rados.2.smithi175.stdout:3474: left oid 11 (ObjNum 1178 snap 384 seq_num 1178) 2024-09-11T03:50:56.152 INFO:tasks.rados.rados.2.smithi175.stdout:3475: expect (ObjNum 1122 snap 360 seq_num 1122) 2024-09-11T03:50:56.219 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:56.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:56.252 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:50:56.273 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:56.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:56.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:56.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:56 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:50:56.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:56 smithi084 ceph-mon[27254]: 2.b scrub ok 2024-09-11T03:50:56.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:56 smithi084 ceph-mon[27254]: 1.c scrub starts 2024-09-11T03:50:56.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:56 smithi084 ceph-mon[27254]: 1.c scrub ok 2024-09-11T03:50:56.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:56 smithi084 ceph-mon[27254]: osdmap e837: 12 total, 12 up, 12 in 2024-09-11T03:50:56.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:50:56.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:50:56.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:50:56.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:50:56.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:50:56.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:50:56.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:56 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:50:56.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:56 smithi084 ceph-mon[27254]: 2.2d scrub starts 2024-09-11T03:50:56.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:56 smithi084 ceph-mon[27254]: 2.2d scrub ok 2024-09-11T03:50:56.374 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:56.396 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:56.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:56 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:50:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:56 smithi121 ceph-mon[30729]: 2.b scrub ok 2024-09-11T03:50:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:56 smithi121 ceph-mon[30729]: 1.c scrub starts 2024-09-11T03:50:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:56 smithi121 ceph-mon[30729]: 1.c scrub ok 2024-09-11T03:50:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:56 smithi121 ceph-mon[30729]: osdmap e837: 12 total, 12 up, 12 in 2024-09-11T03:50:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:56 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:50:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:56 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:50:56.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:56 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:50:56.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:56 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:50:56.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:56 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:50:56.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:56 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:50:56.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:56 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:50:56.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:56 smithi121 ceph-mon[30729]: 2.2d scrub starts 2024-09-11T03:50:56.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:56 smithi121 ceph-mon[30729]: 2.2d scrub ok 2024-09-11T03:50:56.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:56 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:50:56.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:56 smithi175 ceph-mon[30894]: 2.b scrub ok 2024-09-11T03:50:56.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:56 smithi175 ceph-mon[30894]: 1.c scrub starts 2024-09-11T03:50:56.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:56 smithi175 ceph-mon[30894]: 1.c scrub ok 2024-09-11T03:50:56.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:56 smithi175 ceph-mon[30894]: osdmap e837: 12 total, 12 up, 12 in 2024-09-11T03:50:56.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:56 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:50:56.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:56 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:50:56.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:56 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:50:56.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:56 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:50:56.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:56 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:50:56.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:56 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:50:56.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:56 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:50:56.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:56 smithi175 ceph-mon[30894]: 2.2d scrub starts 2024-09-11T03:50:56.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:56 smithi175 ceph-mon[30894]: 2.2d scrub ok 2024-09-11T03:50:56.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:56.525 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:56.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:56.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:56.653 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:56.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:56.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:56.665 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:56.693 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:50:56.694 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:50:56.711 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:50:56.711 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:50:56.726 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:50:56.727 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:50:56.742 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:50:56.743 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:50:56.755 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:56.759 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:50:56.760 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:50:56.776 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:56.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:56.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:56.778 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:50:56.778 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:50:56.795 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:50:56.796 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:50:56.812 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:50:56.812 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:50:56.829 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:50:56.829 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:50:56.845 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:50:56.845 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:50:56.861 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:50:56.862 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:50:56.878 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:50:56.878 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:50:56.878 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:56.899 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:56.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:56.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:57.001 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:57.022 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:57.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:57.023 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:57.043 INFO:tasks.rados.rados.2.smithi175.stdout:3471: done (4 left) 2024-09-11T03:50:57.043 INFO:tasks.rados.rados.2.smithi175.stdout:3472: done (3 left) 2024-09-11T03:50:57.043 INFO:tasks.rados.rados.2.smithi175.stdout:3474: done (2 left) 2024-09-11T03:50:57.043 INFO:tasks.rados.rados.2.smithi175.stdout:3475: done (1 left) 2024-09-11T03:50:57.043 INFO:tasks.rados.rados.2.smithi175.stdout:3476: done (0 left) 2024-09-11T03:50:57.043 INFO:tasks.rados.rados.2.smithi175.stdout:3477: copy_from oid 42 from oid 49 current snap is 385 2024-09-11T03:50:57.043 INFO:tasks.rados.rados.2.smithi175.stdout:3478: write (excl) oid 29 current snap is 385 2024-09-11T03:50:57.043 INFO:tasks.rados.rados.2.smithi175.stdout:3478: seq_num 1179 ranges {615453=753286,2158976=603402,3460532=354921} 2024-09-11T03:50:57.050 INFO:tasks.rados.rados.2.smithi175.stdout:3478: writing smithi17557870-29 from 615453 to 1368739 tid 1 2024-09-11T03:50:57.057 INFO:tasks.rados.rados.2.smithi175.stdout:3478: writing smithi17557870-29 from 2158976 to 2762378 tid 2 2024-09-11T03:50:57.065 INFO:tasks.rados.rados.2.smithi175.stdout:3478: writing smithi17557870-29 from 3460532 to 3815453 tid 3 2024-09-11T03:50:57.065 INFO:tasks.rados.rados.2.smithi175.stdout:3479: rollback oid 20 current snap is 385 2024-09-11T03:50:57.065 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 20 to 381 2024-09-11T03:50:57.065 INFO:tasks.rados.rados.2.smithi175.stdout:3477: finishing copy_from to smithi17557870-42 2024-09-11T03:50:57.065 INFO:tasks.rados.rados.2.smithi175.stdout:3477: got expected ENOENT (src dne) 2024-09-11T03:50:57.065 INFO:tasks.rados.rados.2.smithi175.stdout:3477: finishing copy_from racing read to smithi17557870-42 2024-09-11T03:50:57.065 INFO:tasks.rados.rados.2.smithi175.stdout:3478: finishing write tid 1 to smithi17557870-29 2024-09-11T03:50:57.065 INFO:tasks.rados.rados.2.smithi175.stdout:3478: finishing write tid 2 to smithi17557870-29 2024-09-11T03:50:57.065 INFO:tasks.rados.rados.2.smithi175.stdout:3477: done (2 left) 2024-09-11T03:50:57.065 INFO:tasks.rados.rados.2.smithi175.stdout:3480: snap_remove snap 365 2024-09-11T03:50:57.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:57.148 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:57.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:57.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:57.251 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:57.272 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:57.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:57.273 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:57.374 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:57 smithi084 ceph-mon[27254]: osdmap e838: 12 total, 12 up, 12 in 2024-09-11T03:50:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:57 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/900621779' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:57 smithi084 ceph-mon[27254]: 2.24 scrub starts 2024-09-11T03:50:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:57 smithi084 ceph-mon[27254]: 2.24 scrub ok 2024-09-11T03:50:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:57 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:57 smithi084 ceph-mon[27254]: osdmap e839: 12 total, 12 up, 12 in 2024-09-11T03:50:57.397 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:57.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:57.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:57 smithi121 ceph-mon[30729]: osdmap e838: 12 total, 12 up, 12 in 2024-09-11T03:50:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:57 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/900621779' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:57 smithi121 ceph-mon[30729]: 2.24 scrub starts 2024-09-11T03:50:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:57 smithi121 ceph-mon[30729]: 2.24 scrub ok 2024-09-11T03:50:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:57 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:57 smithi121 ceph-mon[30729]: osdmap e839: 12 total, 12 up, 12 in 2024-09-11T03:50:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:57 smithi175 ceph-mon[30894]: osdmap e838: 12 total, 12 up, 12 in 2024-09-11T03:50:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:57 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/900621779' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:50:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:57 smithi175 ceph-mon[30894]: 2.24 scrub starts 2024-09-11T03:50:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:57 smithi175 ceph-mon[30894]: 2.24 scrub ok 2024-09-11T03:50:57.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:57 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:50:57.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:57 smithi175 ceph-mon[30894]: osdmap e839: 12 total, 12 up, 12 in 2024-09-11T03:50:57.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:57.520 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:57.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:57.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:57.622 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:50:57.644 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:57.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:57.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:50:57.746 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:57.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:57.770 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:57.770 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:57.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:50:57.897 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:57.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:50:57.918 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:50:57.919 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:50:57.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:50:57.919 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:50:57.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.919 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2242 2.14 2:2ef19c88:::smithi17557870-15:58 [sparse-read 0~2125189] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:50:57.919 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:36.886426+0000", 2024-09-11T03:50:57.920 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 561.03127128400001, 2024-09-11T03:50:57.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012666422, 2024-09-11T03:50:57.920 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:57.920 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:57.920 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:57.920 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:57.920 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:57.920 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:57.920 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2242 2024-09-11T03:50:57.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.920 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:57.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:57.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886426+0000", 2024-09-11T03:50:57.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:57.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886426+0000", 2024-09-11T03:50:57.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:57.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886427+0000", 2024-09-11T03:50:57.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2799999999999996e-07 2024-09-11T03:50:57.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:57.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886431+0000", 2024-09-11T03:50:57.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7910000000000002e-06 2024-09-11T03:50:57.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:57.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886439+0000", 2024-09-11T03:50:57.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.038e-06 2024-09-11T03:50:57.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:57.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886445+0000", 2024-09-11T03:50:57.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3799999999999999e-06 2024-09-11T03:50:57.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886527+0000", 2024-09-11T03:50:57.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1797000000000003e-05 2024-09-11T03:50:57.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:50:57.925 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886534+0000", 2024-09-11T03:50:57.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2599999999999999e-06 2024-09-11T03:50:57.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.898657+0000", 2024-09-11T03:50:57.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012123073 2024-09-11T03:50:57.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:57.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.898831+0000", 2024-09-11T03:50:57.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000173931 2024-09-11T03:50:57.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.899093+0000", 2024-09-11T03:50:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026152400000000001 2024-09-11T03:50:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2243 2.14 2:2ef19c88:::smithi17557870-15:58 [read 0~3327819,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e280)", 2024-09-11T03:50:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:36.886461+0000", 2024-09-11T03:50:57.927 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 561.03123651600004, 2024-09-11T03:50:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014874943999999999, 2024-09-11T03:50:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:57.928 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2243 2024-09-11T03:50:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886461+0000", 2024-09-11T03:50:57.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886461+0000", 2024-09-11T03:50:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886462+0000", 2024-09-11T03:50:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1899999999999998e-07 2024-09-11T03:50:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886464+0000", 2024-09-11T03:50:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.632e-06 2024-09-11T03:50:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886470+0000", 2024-09-11T03:50:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.772e-06 2024-09-11T03:50:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886474+0000", 2024-09-11T03:50:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2549999999999999e-06 2024-09-11T03:50:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886554+0000", 2024-09-11T03:50:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9851999999999994e-05 2024-09-11T03:50:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:50:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.886560+0000", 2024-09-11T03:50:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.117e-06 2024-09-11T03:50:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.899326+0000", 2024-09-11T03:50:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012765841999999999 2024-09-11T03:50:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.899468+0000", 2024-09-11T03:50:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014217099999999999 2024-09-11T03:50:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:36.901336+0000", 2024-09-11T03:50:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001867884 2024-09-11T03:50:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.934 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:4913 2.19 2:98bb65f5:::smithi17557870-6:head [sparse-read 0~2219734,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e523)", 2024-09-11T03:50:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:45:41.041241+0000", 2024-09-11T03:50:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 316.87645656500001, 2024-09-11T03:50:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013112360999999999, 2024-09-11T03:50:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:57.935 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4913 2024-09-11T03:50:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041241+0000", 2024-09-11T03:50:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041241+0000", 2024-09-11T03:50:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041243+0000", 2024-09-11T03:50:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7010000000000001e-06 2024-09-11T03:50:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041250+0000", 2024-09-11T03:50:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.674e-06 2024-09-11T03:50:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041263+0000", 2024-09-11T03:50:57.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2942000000000001e-05 2024-09-11T03:50:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041306+0000", 2024-09-11T03:50:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3622999999999999e-05 2024-09-11T03:50:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.042352+0000", 2024-09-11T03:50:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001045891 2024-09-11T03:50:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:50:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.042358+0000", 2024-09-11T03:50:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1460000000000001e-06 2024-09-11T03:50:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.051787+0000", 2024-09-11T03:50:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094287030000000001 2024-09-11T03:50:57.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.051892+0000", 2024-09-11T03:50:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000104706 2024-09-11T03:50:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.054354+0000", 2024-09-11T03:50:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024619749999999999 2024-09-11T03:50:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:4914 2.19 2:98bb65f5:::smithi17557870-6:head [read 0~3075813,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e523)", 2024-09-11T03:50:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:45:41.041333+0000", 2024-09-11T03:50:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 316.87636459100003, 2024-09-11T03:50:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014730996999999999, 2024-09-11T03:50:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4914 2024-09-11T03:50:57.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041333+0000", 2024-09-11T03:50:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041333+0000", 2024-09-11T03:50:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041334+0000", 2024-09-11T03:50:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8199999999999999e-07 2024-09-11T03:50:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041338+0000", 2024-09-11T03:50:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2590000000000002e-06 2024-09-11T03:50:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041347+0000", 2024-09-11T03:50:57.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1689999999999998e-06 2024-09-11T03:50:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041354+0000", 2024-09-11T03:50:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.365e-06 2024-09-11T03:50:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.042433+0000", 2024-09-11T03:50:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010788709999999999 2024-09-11T03:50:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:50:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.042439+0000", 2024-09-11T03:50:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2179999999999996e-06 2024-09-11T03:50:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.054401+0000", 2024-09-11T03:50:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011962494000000001 2024-09-11T03:50:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.054509+0000", 2024-09-11T03:50:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010773900000000001 2024-09-11T03:50:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.056064+0000", 2024-09-11T03:50:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015552000000000001 2024-09-11T03:50:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:57.947 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:4915 2.19 2:98bb65f5:::smithi17557870-6:head [sparse-read 0~2219734] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e523)", 2024-09-11T03:50:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:45:41.041363+0000", 2024-09-11T03:50:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 316.87633486700003, 2024-09-11T03:50:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015358287999999999, 2024-09-11T03:50:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:57.948 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4915 2024-09-11T03:50:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041363+0000", 2024-09-11T03:50:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041363+0000", 2024-09-11T03:50:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041363+0000", 2024-09-11T03:50:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8400000000000004e-07 2024-09-11T03:50:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041368+0000", 2024-09-11T03:50:57.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1060000000000002e-06 2024-09-11T03:50:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041374+0000", 2024-09-11T03:50:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.866e-06 2024-09-11T03:50:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:57.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041380+0000", 2024-09-11T03:50:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3720000000000004e-06 2024-09-11T03:50:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.042462+0000", 2024-09-11T03:50:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001082258 2024-09-11T03:50:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:50:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.042467+0000", 2024-09-11T03:50:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0130000000000001e-06 2024-09-11T03:50:57.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.056146+0000", 2024-09-11T03:50:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013679314999999999 2024-09-11T03:50:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.056249+0000", 2024-09-11T03:50:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102586 2024-09-11T03:50:57.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.056721+0000", 2024-09-11T03:50:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00047218800000000001 2024-09-11T03:50:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5648 2.1f 2:fda69313:::smithi17557870-16:head [copy-from ver 826] snapc f9=[] ondisk+write+known_if_redirected+supports_pool_eio e599)", 2024-09-11T03:50:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:56.974414+0000", 2024-09-11T03:50:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 240.94328343800001, 2024-09-11T03:50:57.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021325499000000001, 2024-09-11T03:50:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5648 2024-09-11T03:50:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974414+0000", 2024-09-11T03:50:57.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974414+0000", 2024-09-11T03:50:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974416+0000", 2024-09-11T03:50:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.666e-06 2024-09-11T03:50:57.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974422+0000", 2024-09-11T03:50:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2400000000000004e-06 2024-09-11T03:50:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974434+0000", 2024-09-11T03:50:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2111e-05 2024-09-11T03:50:57.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974442+0000", 2024-09-11T03:50:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9149999999999996e-06 2024-09-11T03:50:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974501+0000", 2024-09-11T03:50:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8369000000000003e-05 2024-09-11T03:50:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:50:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974509+0000", 2024-09-11T03:50:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2260000000000005e-06 2024-09-11T03:50:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976102+0000", 2024-09-11T03:50:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001593187 2024-09-11T03:50:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976234+0000", 2024-09-11T03:50:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000131903 2024-09-11T03:50:57.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7,8", 2024-09-11T03:50:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.986796+0000", 2024-09-11T03:50:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010562344 2024-09-11T03:50:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.960 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.989855+0000", 2024-09-11T03:50:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030587660000000001 2024-09-11T03:50:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.992580+0000", 2024-09-11T03:50:57.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027246950000000001 2024-09-11T03:50:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.995661+0000", 2024-09-11T03:50:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030807500000000002 2024-09-11T03:50:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:57.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.995708+0000", 2024-09-11T03:50:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7945999999999998e-05 2024-09-11T03:50:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.995740+0000", 2024-09-11T03:50:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1381000000000003e-05 2024-09-11T03:50:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5649 2.1f 2:fda69313:::smithi17557870-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e599)", 2024-09-11T03:50:57.963 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:56.974454+0000", 2024-09-11T03:50:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 240.94324391999999, 2024-09-11T03:50:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015178341999999999, 2024-09-11T03:50:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5649 2024-09-11T03:50:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:57.964 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974454+0000", 2024-09-11T03:50:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974454+0000", 2024-09-11T03:50:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974454+0000", 2024-09-11T03:50:57.965 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8700000000000001e-07 2024-09-11T03:50:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974457+0000", 2024-09-11T03:50:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.61e-06 2024-09-11T03:50:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974462+0000", 2024-09-11T03:50:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6700000000000002e-06 2024-09-11T03:50:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:57.966 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974465+0000", 2024-09-11T03:50:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4829999999999999e-06 2024-09-11T03:50:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974546+0000", 2024-09-11T03:50:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0938000000000003e-05 2024-09-11T03:50:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:50:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974549+0000", 2024-09-11T03:50:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3830000000000001e-06 2024-09-11T03:50:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976401+0000", 2024-09-11T03:50:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018517480000000001 2024-09-11T03:50:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976445+0000", 2024-09-11T03:50:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3439000000000002e-05 2024-09-11T03:50:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.989435+0000", 2024-09-11T03:50:57.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012990714 2024-09-11T03:50:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.989531+0000", 2024-09-11T03:50:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5304999999999996e-05 2024-09-11T03:50:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.989632+0000", 2024-09-11T03:50:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000101665 2024-09-11T03:50:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:57.969 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5650 2.1e 2:7b0c3397:::smithi17557870-14:head [copy-from ver 875] snapc f9=[] ondisk+write+known_if_redirected+supports_pool_eio e599)", 2024-09-11T03:50:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:56.974491+0000", 2024-09-11T03:50:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 240.94320640999999, 2024-09-11T03:50:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.029056842999999999, 2024-09-11T03:50:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:57.970 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5650 2024-09-11T03:50:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974491+0000", 2024-09-11T03:50:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974491+0000", 2024-09-11T03:50:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974492+0000", 2024-09-11T03:50:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.72e-07 2024-09-11T03:50:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974494+0000", 2024-09-11T03:50:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0480000000000001e-06 2024-09-11T03:50:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:57.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974498+0000", 2024-09-11T03:50:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1699999999999999e-06 2024-09-11T03:50:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974501+0000", 2024-09-11T03:50:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0479999999999999e-06 2024-09-11T03:50:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974537+0000", 2024-09-11T03:50:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6041000000000003e-05 2024-09-11T03:50:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:50:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974541+0000", 2024-09-11T03:50:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2010000000000001e-06 2024-09-11T03:50:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.975950+0000", 2024-09-11T03:50:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014090489999999999 2024-09-11T03:50:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976045+0000", 2024-09-11T03:50:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4704999999999995e-05 2024-09-11T03:50:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,10", 2024-09-11T03:50:57.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.992128+0000", 2024-09-11T03:50:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016083318999999999 2024-09-11T03:50:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.994884+0000", 2024-09-11T03:50:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027554860000000001 2024-09-11T03:50:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.997365+0000", 2024-09-11T03:50:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024811630000000002 2024-09-11T03:50:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:57.003332+0000", 2024-09-11T03:50:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059666720000000001 2024-09-11T03:50:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:57.003487+0000", 2024-09-11T03:50:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015519399999999999 2024-09-11T03:50:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:57.003548+0000", 2024-09-11T03:50:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1375000000000001e-05 2024-09-11T03:50:57.977 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5651 2.1e 2:7b0c3397:::smithi17557870-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e599)", 2024-09-11T03:50:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:56.974509+0000", 2024-09-11T03:50:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 240.943188518, 2024-09-11T03:50:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.02013765, 2024-09-11T03:50:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:57.978 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5651 2024-09-11T03:50:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974509+0000", 2024-09-11T03:50:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974509+0000", 2024-09-11T03:50:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974510+0000", 2024-09-11T03:50:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5999999999999999e-07 2024-09-11T03:50:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974511+0000", 2024-09-11T03:50:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.72e-06 2024-09-11T03:50:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974517+0000", 2024-09-11T03:50:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0499999999999997e-06 2024-09-11T03:50:57.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974522+0000", 2024-09-11T03:50:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7770000000000002e-06 2024-09-11T03:50:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974570+0000", 2024-09-11T03:50:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7729000000000003e-05 2024-09-11T03:50:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:50:57.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974574+0000", 2024-09-11T03:50:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7929999999999999e-06 2024-09-11T03:50:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976323+0000", 2024-09-11T03:50:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017491760000000001 2024-09-11T03:50:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976367+0000", 2024-09-11T03:50:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3912000000000002e-05 2024-09-11T03:50:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.994395+0000", 2024-09-11T03:50:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018028512 2024-09-11T03:50:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.994492+0000", 2024-09-11T03:50:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6679000000000001e-05 2024-09-11T03:50:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.994647+0000", 2024-09-11T03:50:57.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015484200000000001 2024-09-11T03:50:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7041 2.1f 2:fba9fb28:::smithi17557870-48:head [copy-from ver 962] snapc 13a=[13a,139,138,137,136,135,133,130,12e,12d,12c,12a,127,126,123,120,11f,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e709)", 2024-09-11T03:50:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:47.006333+0000", 2024-09-11T03:50:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 130.91136448500001, 2024-09-11T03:50:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023502866000000001, 2024-09-11T03:50:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:57.984 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7041 2024-09-11T03:50:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006333+0000", 2024-09-11T03:50:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.985 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006333+0000", 2024-09-11T03:50:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006335+0000", 2024-09-11T03:50:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.528e-06 2024-09-11T03:50:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006341+0000", 2024-09-11T03:50:57.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2700000000000001e-06 2024-09-11T03:50:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006355+0000", 2024-09-11T03:50:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3593e-05 2024-09-11T03:50:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006361+0000", 2024-09-11T03:50:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7290000000000003e-06 2024-09-11T03:50:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006495+0000", 2024-09-11T03:50:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133158 2024-09-11T03:50:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:50:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006501+0000", 2024-09-11T03:50:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4289999999999998e-06 2024-09-11T03:50:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.007153+0000", 2024-09-11T03:50:57.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00065224399999999996 2024-09-11T03:50:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.007271+0000", 2024-09-11T03:50:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000117743 2024-09-11T03:50:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7,8", 2024-09-11T03:50:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.019731+0000", 2024-09-11T03:50:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012460024 2024-09-11T03:50:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.023726+0000", 2024-09-11T03:50:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003995139 2024-09-11T03:50:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.027785+0000", 2024-09-11T03:50:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040589809999999997 2024-09-11T03:50:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.029747+0000", 2024-09-11T03:50:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019621819999999998 2024-09-11T03:50:57.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.029801+0000", 2024-09-11T03:50:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3715999999999999e-05 2024-09-11T03:50:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.029836+0000", 2024-09-11T03:50:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5129999999999997e-05 2024-09-11T03:50:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7042 2.1f 2:fba9fb28:::smithi17557870-48:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e709)", 2024-09-11T03:50:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:47.006384+0000", 2024-09-11T03:50:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 130.911314116, 2024-09-11T03:50:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016792033000000001, 2024-09-11T03:50:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7042 2024-09-11T03:50:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.992 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006384+0000", 2024-09-11T03:50:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006384+0000", 2024-09-11T03:50:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006384+0000", 2024-09-11T03:50:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8800000000000002e-07 2024-09-11T03:50:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006387+0000", 2024-09-11T03:50:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7140000000000002e-06 2024-09-11T03:50:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006395+0000", 2024-09-11T03:50:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3110000000000006e-06 2024-09-11T03:50:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006405+0000", 2024-09-11T03:50:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6679999999999996e-06 2024-09-11T03:50:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.007550+0000", 2024-09-11T03:50:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011449489999999999 2024-09-11T03:50:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:50:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.007587+0000", 2024-09-11T03:50:57.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7372000000000001e-05 2024-09-11T03:50:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.022980+0000", 2024-09-11T03:50:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015392641 2024-09-11T03:50:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.023063+0000", 2024-09-11T03:50:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3288999999999995e-05 2024-09-11T03:50:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.023176+0000", 2024-09-11T03:50:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112501 2024-09-11T03:50:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:7115 2.c e713/569 2:3603e322:::smithi17557870-42:head v 713'769, mlcod=713'767)", 2024-09-11T03:50:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:51.207248+0000", 2024-09-11T03:50:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 126.71044968699999, 2024-09-11T03:50:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015546002999999999, 2024-09-11T03:50:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:57.997 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.207248+0000", 2024-09-11T03:50:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.207248+0000", 2024-09-11T03:50:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:57.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.207250+0000", 2024-09-11T03:50:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6810000000000001e-06 2024-09-11T03:50:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.216300+0000", 2024-09-11T03:50:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0090505670000000007 2024-09-11T03:50:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.216317+0000", 2024-09-11T03:50:57.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6546999999999999e-05 2024-09-11T03:50:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.216325+0000", 2024-09-11T03:50:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3550000000000005e-06 2024-09-11T03:50:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.216394+0000", 2024-09-11T03:50:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8627000000000005e-05 2024-09-11T03:50:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.216419+0000", 2024-09-11T03:50:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5003000000000001e-05 2024-09-11T03:50:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.222743+0000", 2024-09-11T03:50:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0063239129999999996 2024-09-11T03:50:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.222794+0000", 2024-09-11T03:50:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1310000000000002e-05 2024-09-11T03:50:58.001 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7387 2.1f 2:fba9fb28:::smithi17557870-48:head [copy-from ver 916] snapc 14d=[14d,14b,14a,149,148,146,145,144,143,142,13d,13c,139,138,136,135,133,12e,12a,127,120,11e,119] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-09-11T03:50:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:49:22.015590+0000", 2024-09-11T03:50:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 95.902107473000001, 2024-09-11T03:50:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016730015000000001, 2024-09-11T03:50:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:58.002 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7387 2024-09-11T03:50:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.015590+0000", 2024-09-11T03:50:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:58.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.015590+0000", 2024-09-11T03:50:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.015592+0000", 2024-09-11T03:50:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9290000000000001e-06 2024-09-11T03:50:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.015600+0000", 2024-09-11T03:50:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6890000000000004e-06 2024-09-11T03:50:58.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.015617+0000", 2024-09-11T03:50:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.751e-05 2024-09-11T03:50:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.015626+0000", 2024-09-11T03:50:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7970000000000006e-06 2024-09-11T03:50:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:58.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.015694+0000", 2024-09-11T03:50:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7941999999999994e-05 2024-09-11T03:50:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:50:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.015703+0000", 2024-09-11T03:50:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6160000000000002e-06 2024-09-11T03:50:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.019591+0000", 2024-09-11T03:50:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003888318 2024-09-11T03:50:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.006 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.019689+0000", 2024-09-11T03:50:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8111999999999998e-05 2024-09-11T03:50:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7,8", 2024-09-11T03:50:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.025025+0000", 2024-09-11T03:50:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053357129999999997 2024-09-11T03:50:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:58.007 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.028804+0000", 2024-09-11T03:50:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037790950000000001 2024-09-11T03:50:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.031565+0000", 2024-09-11T03:50:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027610209999999998 2024-09-11T03:50:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.032229+0000", 2024-09-11T03:50:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066412299999999997 2024-09-11T03:50:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.032272+0000", 2024-09-11T03:50:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2400000000000001e-05 2024-09-11T03:50:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.032320+0000", 2024-09-11T03:50:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8749999999999999e-05 2024-09-11T03:50:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.009 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7435 2.24 2:25f5f8b9:::smithi17557870-26:head [read 0~2874228,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e749)", 2024-09-11T03:50:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:49:27.143759+0000", 2024-09-11T03:50:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 90.773938513999994, 2024-09-11T03:50:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016534205, 2024-09-11T03:50:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7435 2024-09-11T03:50:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:58.010 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143759+0000", 2024-09-11T03:50:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143759+0000", 2024-09-11T03:50:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143761+0000", 2024-09-11T03:50:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5880000000000001e-06 2024-09-11T03:50:58.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143773+0000", 2024-09-11T03:50:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2599000000000001e-05 2024-09-11T03:50:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143786+0000", 2024-09-11T03:50:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.29e-05 2024-09-11T03:50:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:58.012 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143796+0000", 2024-09-11T03:50:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5289999999999997e-06 2024-09-11T03:50:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143949+0000", 2024-09-11T03:50:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000153499 2024-09-11T03:50:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:50:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143961+0000", 2024-09-11T03:50:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1382e-05 2024-09-11T03:50:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.156142+0000", 2024-09-11T03:50:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012180972 2024-09-11T03:50:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.156294+0000", 2024-09-11T03:50:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015199300000000001 2024-09-11T03:50:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.160293+0000", 2024-09-11T03:50:58.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003999743 2024-09-11T03:50:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7436 2.24 2:25f5f8b9:::smithi17557870-26:head [read 0~2874228,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e749)", 2024-09-11T03:50:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:49:27.143810+0000", 2024-09-11T03:50:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 90.773887950000002, 2024-09-11T03:50:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017603882000000001, 2024-09-11T03:50:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:58.015 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7436 2024-09-11T03:50:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143810+0000", 2024-09-11T03:50:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143810+0000", 2024-09-11T03:50:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143810+0000", 2024-09-11T03:50:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3e-07 2024-09-11T03:50:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.018 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:50:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143814+0000", 2024-09-11T03:50:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3890000000000001e-06 2024-09-11T03:50:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143821+0000", 2024-09-11T03:50:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5909999999999998e-06 2024-09-11T03:50:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143827+0000", 2024-09-11T03:50:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4929999999999998e-06 2024-09-11T03:50:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143994+0000", 2024-09-11T03:50:58.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016706300000000001 2024-09-11T03:50:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:50:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.144015+0000", 2024-09-11T03:50:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0859000000000001e-05 2024-09-11T03:50:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.160488+0000", 2024-09-11T03:50:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016472958999999999 2024-09-11T03:50:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:58.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.160605+0000", 2024-09-11T03:50:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011704799999999999 2024-09-11T03:50:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.161414+0000", 2024-09-11T03:50:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00080884999999999996 2024-09-11T03:50:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7437 2.24 2:25f5f8b9:::smithi17557870-26:head [sparse-read 0~2349940] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e749)", 2024-09-11T03:50:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:49:27.143834+0000", 2024-09-11T03:50:58.022 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 90.773863289999994, 2024-09-11T03:50:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018037481000000001, 2024-09-11T03:50:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7437 2024-09-11T03:50:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:58.023 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143834+0000", 2024-09-11T03:50:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143834+0000", 2024-09-11T03:50:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143835+0000", 2024-09-11T03:50:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9100000000000004e-07 2024-09-11T03:50:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143837+0000", 2024-09-11T03:50:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3889999999999999e-06 2024-09-11T03:50:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143843+0000", 2024-09-11T03:50:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9279999999999998e-06 2024-09-11T03:50:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143848+0000", 2024-09-11T03:50:58.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7720000000000002e-06 2024-09-11T03:50:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.144062+0000", 2024-09-11T03:50:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021335000000000001 2024-09-11T03:50:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:50:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.144071+0000", 2024-09-11T03:50:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6099999999999995e-06 2024-09-11T03:50:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.161477+0000", 2024-09-11T03:50:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017405522999999999 2024-09-11T03:50:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.161607+0000", 2024-09-11T03:50:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013022500000000001 2024-09-11T03:50:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.161872+0000", 2024-09-11T03:50:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026509299999999999 2024-09-11T03:50:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.027 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7972 2.1e 2:7b0c3397:::smithi17557870-14:head [write 1942162~327047] snapc 162=[162,161,160,15f,15e,15d,15b,15a,157,156,155,154,153,151,14d,148,144,143,13c,139,135,12a,127,120] ondisk+write+known_if_redirected+supports_pool_eio e785)", 2024-09-11T03:50:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:50:03.101803+0000", 2024-09-11T03:50:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 54.815895085999998, 2024-09-11T03:50:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037451759000000001, 2024-09-11T03:50:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7972 2024-09-11T03:50:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.101803+0000", 2024-09-11T03:50:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.101803+0000", 2024-09-11T03:50:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.029 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.101804+0000", 2024-09-11T03:50:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1200000000000001e-06 2024-09-11T03:50:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102107+0000", 2024-09-11T03:50:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00030352599999999998 2024-09-11T03:50:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102118+0000", 2024-09-11T03:50:58.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0655999999999999e-05 2024-09-11T03:50:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102123+0000", 2024-09-11T03:50:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2630000000000003e-06 2024-09-11T03:50:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102162+0000", 2024-09-11T03:50:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9054000000000003e-05 2024-09-11T03:50:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:58.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102256+0000", 2024-09-11T03:50:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3591999999999994e-05 2024-09-11T03:50:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,10", 2024-09-11T03:50:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102374+0000", 2024-09-11T03:50:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000118497 2024-09-11T03:50:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103648+0000", 2024-09-11T03:50:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012732780000000001 2024-09-11T03:50:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:58.033 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103825+0000", 2024-09-11T03:50:58.033 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017712000000000001 2024-09-11T03:50:58.033 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.033 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.033 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:58.033 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139177+0000", 2024-09-11T03:50:58.033 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.035352078000000002 2024-09-11T03:50:58.033 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.033 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.033 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:58.033 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139218+0000", 2024-09-11T03:50:58.033 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1387999999999997e-05 2024-09-11T03:50:58.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.034 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:58.034 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139254+0000", 2024-09-11T03:50:58.034 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6186999999999998e-05 2024-09-11T03:50:58.034 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.034 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:58.034 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.034 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7973 2.1e 2:7b0c3397:::smithi17557870-14:head [setxattr _header (58) in=7b,truncate 2269209] snapc 162=[162,161,160,15f,15e,15d,15b,15a,157,156,155,154,153,151,14d,148,144,143,13c,139,135,12a,127,120] ondisk+write+known_if_redirected+supports_pool_eio e785)", 2024-09-11T03:50:58.034 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:50:03.102135+0000", 2024-09-11T03:50:58.034 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 54.815563071, 2024-09-11T03:50:58.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037234972999999998, 2024-09-11T03:50:58.035 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:58.035 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:58.035 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:58.035 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:58.035 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:58.035 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:58.035 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7973 2024-09-11T03:50:58.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.035 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:58.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:58.035 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102135+0000", 2024-09-11T03:50:58.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:58.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.036 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:58.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102135+0000", 2024-09-11T03:50:58.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:58.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.036 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:58.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102135+0000", 2024-09-11T03:50:58.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8299999999999997e-07 2024-09-11T03:50:58.037 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.037 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:58.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102139+0000", 2024-09-11T03:50:58.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4970000000000001e-06 2024-09-11T03:50:58.037 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.037 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:58.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102144+0000", 2024-09-11T03:50:58.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0499999999999999e-06 2024-09-11T03:50:58.037 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:58.038 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102148+0000", 2024-09-11T03:50:58.038 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9940000000000002e-06 2024-09-11T03:50:58.038 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:58.038 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103021+0000", 2024-09-11T03:50:58.038 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00087274900000000003 2024-09-11T03:50:58.038 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:58.038 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103119+0000", 2024-09-11T03:50:58.039 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8740000000000007e-05 2024-09-11T03:50:58.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.039 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.039 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,10", 2024-09-11T03:50:58.039 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103254+0000", 2024-09-11T03:50:58.039 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013457599999999999 2024-09-11T03:50:58.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.039 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.039 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:58.039 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103651+0000", 2024-09-11T03:50:58.039 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00039681999999999999 2024-09-11T03:50:58.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.039 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.039 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:58.040 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103971+0000", 2024-09-11T03:50:58.040 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032051399999999999 2024-09-11T03:50:58.040 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.040 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.040 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:58.040 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139293+0000", 2024-09-11T03:50:58.040 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.035321726999999997 2024-09-11T03:50:58.040 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.040 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.040 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:58.040 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139331+0000", 2024-09-11T03:50:58.040 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7934e-05 2024-09-11T03:50:58.041 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.041 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:58.041 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139370+0000", 2024-09-11T03:50:58.041 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8788999999999997e-05 2024-09-11T03:50:58.041 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.041 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:58.041 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.041 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.041 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7974 2.1e 2:7b0c3397:::smithi17557870-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e785)", 2024-09-11T03:50:58.041 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:50:03.102155+0000", 2024-09-11T03:50:58.041 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 54.815542803, 2024-09-11T03:50:58.042 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037356064000000001, 2024-09-11T03:50:58.042 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:58.042 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:58.042 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:50:58.042 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:58.042 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:58.042 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:58.042 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7974 2024-09-11T03:50:58.042 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.042 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:58.042 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.042 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:58.043 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102155+0000", 2024-09-11T03:50:58.043 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:58.043 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.043 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.043 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:58.043 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102155+0000", 2024-09-11T03:50:58.043 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:58.043 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.043 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.043 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:58.043 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102155+0000", 2024-09-11T03:50:58.043 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1500000000000005e-07 2024-09-11T03:50:58.044 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.044 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.044 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:58.044 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102157+0000", 2024-09-11T03:50:58.044 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.012e-06 2024-09-11T03:50:58.044 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.044 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.044 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:58.044 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102164+0000", 2024-09-11T03:50:58.044 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1689999999999997e-06 2024-09-11T03:50:58.044 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.044 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.044 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:58.044 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102168+0000", 2024-09-11T03:50:58.044 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4229999999999998e-06 2024-09-11T03:50:58.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:58.045 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103583+0000", 2024-09-11T03:50:58.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014153950000000001 2024-09-11T03:50:58.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-11T03:50:58.045 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103639+0000", 2024-09-11T03:50:58.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5517999999999999e-05 2024-09-11T03:50:58.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:58.046 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139402+0000", 2024-09-11T03:50:58.046 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.035763164 2024-09-11T03:50:58.046 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.046 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.046 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:58.046 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139451+0000", 2024-09-11T03:50:58.046 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8374999999999997e-05 2024-09-11T03:50:58.046 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.046 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.046 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:58.046 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139511+0000", 2024-09-11T03:50:58.046 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0492999999999998e-05 2024-09-11T03:50:58.046 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.046 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:58.047 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.047 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.047 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.047 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:8198 2.1e 2:7b0c3397:::smithi17557870-14:head [copy-from ver 1045] snapc 16b=[16b,169,168,167,164,162,160,15f,15e,15d,15b,15a,157,156,155,151,14d,148,144,143,13c,139,135,12a,127,120] ondisk+write+known_if_redirected+supports_pool_eio e802)", 2024-09-11T03:50:58.047 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:50:20.169494+0000", 2024-09-11T03:50:58.047 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 37.748204252999997, 2024-09-11T03:50:58.047 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014646783, 2024-09-11T03:50:58.047 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:50:58.047 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:50:58.047 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:50:58.047 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:50:58.047 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:50:58.047 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:50:58.048 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8198 2024-09-11T03:50:58.048 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.048 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:50:58.048 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.048 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:50:58.048 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:20.169494+0000", 2024-09-11T03:50:58.048 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:58.048 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.048 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.048 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:50:58.048 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:20.169494+0000", 2024-09-11T03:50:58.048 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:50:58.049 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.049 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.049 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:50:58.049 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:20.169496+0000", 2024-09-11T03:50:58.049 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2960000000000001e-06 2024-09-11T03:50:58.049 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.049 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.049 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:50:58.049 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:20.169506+0000", 2024-09-11T03:50:58.049 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9499999999999996e-06 2024-09-11T03:50:58.049 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.049 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.049 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:50:58.049 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:20.169526+0000", 2024-09-11T03:50:58.050 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0693999999999999e-05 2024-09-11T03:50:58.050 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.050 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.050 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:50:58.050 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:20.169538+0000", 2024-09-11T03:50:58.050 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.166e-05 2024-09-11T03:50:58.050 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.050 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.050 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:58.050 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:20.169614+0000", 2024-09-11T03:50:58.050 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5918000000000001e-05 2024-09-11T03:50:58.050 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.050 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:50:58.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:20.169621+0000", 2024-09-11T03:50:58.051 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3719999999999999e-06 2024-09-11T03:50:58.051 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.051 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:50:58.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:20.171427+0000", 2024-09-11T03:50:58.051 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018059040000000001 2024-09-11T03:50:58.051 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.051 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:50:58.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:20.171513+0000", 2024-09-11T03:50:58.051 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5337e-05 2024-09-11T03:50:58.051 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.052 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.052 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,10", 2024-09-11T03:50:58.052 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:20.177143+0000", 2024-09-11T03:50:58.052 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056301909999999997 2024-09-11T03:50:58.052 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.052 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.052 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:58.052 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:20.181378+0000", 2024-09-11T03:50:58.052 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042351200000000002 2024-09-11T03:50:58.052 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.052 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.052 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:50:58.052 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:20.181572+0000", 2024-09-11T03:50:58.052 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000194007 2024-09-11T03:50:58.053 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.053 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.053 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:50:58.053 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:20.184056+0000", 2024-09-11T03:50:58.053 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002484394 2024-09-11T03:50:58.053 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.053 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.053 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:50:58.053 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:20.184097+0000", 2024-09-11T03:50:58.053 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0414999999999998e-05 2024-09-11T03:50:58.053 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:50:58.053 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:50:58.054 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:50:58.054 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:20.184140+0000", 2024-09-11T03:50:58.054 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3525000000000001e-05 2024-09-11T03:50:58.054 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.054 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:58.054 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.054 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:50:58.054 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:50:58.054 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:50:58.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:58.055 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:58.056 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:50:58.056 INFO:tasks.rados.rados.2.smithi175.stdout:3480: done (2 left) 2024-09-11T03:50:58.056 INFO:tasks.rados.rados.2.smithi175.stdout:3481: delete oid 27 current snap is 385 2024-09-11T03:50:58.057 INFO:tasks.rados.rados.2.smithi175.stdout:3478: finishing write tid 3 to smithi17557870-29 2024-09-11T03:50:58.057 INFO:tasks.rados.rados.2.smithi175.stdout:3478: finishing write tid 4 to smithi17557870-29 2024-09-11T03:50:58.057 INFO:tasks.rados.rados.2.smithi175.stdout:3478: finishing write tid 5 to smithi17557870-29 2024-09-11T03:50:58.057 INFO:tasks.rados.rados.2.smithi175.stdout:3478: oid 29 updating version 0 to 1204 2024-09-11T03:50:58.057 INFO:tasks.rados.rados.2.smithi175.stdout:3478: oid 29 updating version 1204 to 1205 2024-09-11T03:50:58.057 INFO:tasks.rados.rados.2.smithi175.stdout:3478: oid 29 version 1205 is already newer than 1202 2024-09-11T03:50:58.057 INFO:tasks.rados.rados.2.smithi175.stdout:3478: oid 29 version 1205 is already newer than 1203 2024-09-11T03:50:58.057 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 1205 (ObjNum 1179 snap 385 seq_num 1179) dirty exists 2024-09-11T03:50:58.057 INFO:tasks.rados.rados.2.smithi175.stdout:3478: left oid 29 (ObjNum 1179 snap 385 seq_num 1179) 2024-09-11T03:50:58.057 INFO:tasks.rados.rados.2.smithi175.stdout:3479: finishing rollback tid 0 to smithi17557870-20 2024-09-11T03:50:58.057 INFO:tasks.rados.rados.2.smithi175.stdout:3479: finishing rollback tid 1 to smithi17557870-20 2024-09-11T03:50:58.057 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 1031 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-11T03:50:58.058 INFO:tasks.rados.rados.2.smithi175.stdout:3478: done (2 left) 2024-09-11T03:50:58.058 INFO:tasks.rados.rados.2.smithi175.stdout:3479: done (1 left) 2024-09-11T03:50:58.058 INFO:tasks.rados.rados.2.smithi175.stdout:3481: done (0 left) 2024-09-11T03:50:58.058 INFO:tasks.rados.rados.2.smithi175.stdout:3482: write oid 17 current snap is 385 2024-09-11T03:50:58.058 INFO:tasks.rados.rados.2.smithi175.stdout:3482: seq_num 1181 ranges {649067=503784,1875940=607071,3115378=731306} 2024-09-11T03:50:58.064 INFO:tasks.rados.rados.2.smithi175.stdout:3482: writing smithi17557870-17 from 649067 to 1152851 tid 1 2024-09-11T03:50:58.071 INFO:tasks.rados.rados.2.smithi175.stdout:3482: writing smithi17557870-17 from 1875940 to 2483011 tid 2 2024-09-11T03:50:58.079 INFO:tasks.rados.rados.2.smithi175.stdout:3482: writing smithi17557870-17 from 3115378 to 3846684 tid 3 2024-09-11T03:50:58.079 INFO:tasks.rados.rados.2.smithi175.stdout:3483: write oid 42 current snap is 385 2024-09-11T03:50:58.079 INFO:tasks.rados.rados.2.smithi175.stdout:3483: seq_num 1182 ranges {665874=579033,1934422=467082,3142801=542578} 2024-09-11T03:50:58.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:50:58.085 INFO:tasks.rados.rados.2.smithi175.stdout:3483: writing smithi17557870-42 from 665874 to 1244907 tid 1 2024-09-11T03:50:58.092 INFO:tasks.rados.rados.2.smithi175.stdout:3483: writing smithi17557870-42 from 1934422 to 2401504 tid 2 2024-09-11T03:50:58.100 INFO:tasks.rados.rados.2.smithi175.stdout:3483: writing smithi17557870-42 from 3142801 to 3685379 tid 3 2024-09-11T03:50:58.100 INFO:tasks.rados.rados.2.smithi175.stdout:3484: read oid 16 snap -1 2024-09-11T03:50:58.100 INFO:tasks.rados.rados.2.smithi175.stdout:3484: expect deleted 2024-09-11T03:50:58.100 INFO:tasks.rados.rados.2.smithi175.stdout:3485: copy_from oid 11 from oid 9 current snap is 385 2024-09-11T03:50:58.100 INFO:tasks.rados.rados.2.smithi175.stdout:3482: finishing write tid 1 to smithi17557870-17 2024-09-11T03:50:58.100 INFO:tasks.rados.rados.2.smithi175.stdout:3482: finishing write tid 2 to smithi17557870-17 2024-09-11T03:50:58.100 INFO:tasks.rados.rados.2.smithi175.stdout:3482: finishing write tid 3 to smithi17557870-17 2024-09-11T03:50:58.100 INFO:tasks.rados.rados.2.smithi175.stdout:3482: finishing write tid 4 to smithi17557870-17 2024-09-11T03:50:58.100 INFO:tasks.rados.rados.2.smithi175.stdout:3482: finishing write tid 5 to smithi17557870-17 2024-09-11T03:50:58.100 INFO:tasks.rados.rados.2.smithi175.stdout:3482: oid 17 updating version 0 to 1171 2024-09-11T03:50:58.100 INFO:tasks.rados.rados.2.smithi175.stdout:3482: oid 17 updating version 1171 to 1172 2024-09-11T03:50:58.101 INFO:tasks.rados.rados.2.smithi175.stdout:3482: oid 17 version 1172 is already newer than 1170 2024-09-11T03:50:58.101 INFO:tasks.rados.rados.2.smithi175.stdout:3482: oid 17 updating version 1172 to 1173 2024-09-11T03:50:58.101 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 1173 (ObjNum 1181 snap 385 seq_num 1181) dirty exists 2024-09-11T03:50:58.101 INFO:tasks.rados.rados.2.smithi175.stdout:3482: left oid 17 (ObjNum 1181 snap 385 seq_num 1181) 2024-09-11T03:50:58.101 INFO:tasks.rados.rados.2.smithi175.stdout:3483: finishing write tid 1 to smithi17557870-42 2024-09-11T03:50:58.101 INFO:tasks.rados.rados.2.smithi175.stdout:3483: finishing write tid 2 to smithi17557870-42 2024-09-11T03:50:58.101 INFO:tasks.rados.rados.2.smithi175.stdout:3482: done (3 left) 2024-09-11T03:50:58.101 INFO:tasks.rados.rados.2.smithi175.stdout:3486: write oid 44 current snap is 385 2024-09-11T03:50:58.101 INFO:tasks.rados.rados.2.smithi175.stdout:3486: seq_num 1183 ranges {682681=654282,1876551=610740,3053871=470203} 2024-09-11T03:50:58.106 INFO:tasks.rados.rados.2.smithi175.stdout:3486: writing smithi17557870-44 from 682681 to 1336963 tid 1 2024-09-11T03:50:58.113 INFO:tasks.rados.rados.2.smithi175.stdout:3486: writing smithi17557870-44 from 1876551 to 2487291 tid 2 2024-09-11T03:50:58.119 INFO:tasks.rados.rados.2.smithi175.stdout:3486: writing smithi17557870-44 from 3053871 to 3524074 tid 3 2024-09-11T03:50:58.119 INFO:tasks.rados.rados.2.smithi175.stdout:3487: snap_create 2024-09-11T03:50:58.119 INFO:tasks.rados.rados.2.smithi175.stdout:3483: finishing write tid 3 to smithi17557870-42 2024-09-11T03:50:58.119 INFO:tasks.rados.rados.2.smithi175.stdout:3483: finishing write tid 4 to smithi17557870-42 2024-09-11T03:50:58.119 INFO:tasks.rados.rados.2.smithi175.stdout:3483: finishing write tid 5 to smithi17557870-42 2024-09-11T03:50:58.120 INFO:tasks.rados.rados.2.smithi175.stdout:3483: oid 42 updating version 0 to 1066 2024-09-11T03:50:58.120 INFO:tasks.rados.rados.2.smithi175.stdout:3483: oid 42 version 1066 is already newer than 1065 2024-09-11T03:50:58.120 INFO:tasks.rados.rados.2.smithi175.stdout:3483: oid 42 updating version 1066 to 1067 2024-09-11T03:50:58.120 INFO:tasks.rados.rados.2.smithi175.stdout:3483: oid 42 version 1067 is already newer than 1064 2024-09-11T03:50:58.120 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 1067 (ObjNum 1182 snap 385 seq_num 1182) dirty exists 2024-09-11T03:50:58.120 INFO:tasks.rados.rados.2.smithi175.stdout:3483: left oid 42 (ObjNum 1182 snap 385 seq_num 1182) 2024-09-11T03:50:58.120 INFO:tasks.rados.rados.2.smithi175.stdout:3485: finishing copy_from racing read to smithi17557870-11 2024-09-11T03:50:58.120 INFO:tasks.rados.rados.2.smithi175.stdout:3485: finishing copy_from to smithi17557870-11 2024-09-11T03:50:58.120 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 1234 (ObjNum 1033 snap 340 seq_num 1033) dirty exists 2024-09-11T03:50:58.120 INFO:tasks.rados.rados.2.smithi175.stdout:3486: finishing write tid 1 to smithi17557870-44 2024-09-11T03:50:58.120 INFO:tasks.rados.rados.2.smithi175.stdout:3486: finishing write tid 2 to smithi17557870-44 2024-09-11T03:50:58.122 INFO:tasks.rados.rados.2.smithi175.stdout:3486: finishing write tid 3 to smithi17557870-44 2024-09-11T03:50:58.122 INFO:tasks.rados.rados.2.smithi175.stdout:3486: finishing write tid 4 to smithi17557870-44 2024-09-11T03:50:58.122 INFO:tasks.rados.rados.2.smithi175.stdout:3486: finishing write tid 5 to smithi17557870-44 2024-09-11T03:50:58.122 INFO:tasks.rados.rados.2.smithi175.stdout:3486: oid 44 updating version 0 to 1026 2024-09-11T03:50:58.122 INFO:tasks.rados.rados.2.smithi175.stdout:3486: oid 44 version 1026 is already newer than 1023 2024-09-11T03:50:58.122 INFO:tasks.rados.rados.2.smithi175.stdout:3486: oid 44 version 1026 is already newer than 1024 2024-09-11T03:50:58.122 INFO:tasks.rados.rados.2.smithi175.stdout:3486: oid 44 version 1026 is already newer than 1025 2024-09-11T03:50:58.122 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 1026 (ObjNum 1183 snap 385 seq_num 1183) dirty exists 2024-09-11T03:50:58.122 INFO:tasks.rados.rados.2.smithi175.stdout:3486: left oid 44 (ObjNum 1183 snap 385 seq_num 1183) 2024-09-11T03:50:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:58 smithi084 ceph-mon[27254]: 1.1d scrub starts 2024-09-11T03:50:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:58 smithi084 ceph-mon[27254]: 1.1d scrub ok 2024-09-11T03:50:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:58 smithi084 ceph-mon[27254]: pgmap v1413: 97 pgs: 97 active+clean; 638 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 2.9 MiB/s wr, 17 op/s 2024-09-11T03:50:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:58 smithi084 ceph-mon[27254]: 2.29 scrub starts 2024-09-11T03:50:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:58 smithi084 ceph-mon[27254]: 2.29 scrub ok 2024-09-11T03:50:58.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:58 smithi084 ceph-mon[27254]: osdmap e840: 12 total, 12 up, 12 in 2024-09-11T03:50:58.156 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:58.177 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:58.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:58.178 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:58.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:58.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:58.301 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:58.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:58.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:58 smithi121 ceph-mon[30729]: 1.1d scrub starts 2024-09-11T03:50:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:58 smithi121 ceph-mon[30729]: 1.1d scrub ok 2024-09-11T03:50:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:58 smithi121 ceph-mon[30729]: pgmap v1413: 97 pgs: 97 active+clean; 638 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 2.9 MiB/s wr, 17 op/s 2024-09-11T03:50:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:58 smithi121 ceph-mon[30729]: 2.29 scrub starts 2024-09-11T03:50:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:58 smithi121 ceph-mon[30729]: 2.29 scrub ok 2024-09-11T03:50:58.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:58 smithi121 ceph-mon[30729]: osdmap e840: 12 total, 12 up, 12 in 2024-09-11T03:50:58.423 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:58.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:58 smithi175 ceph-mon[30894]: 1.1d scrub starts 2024-09-11T03:50:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:58 smithi175 ceph-mon[30894]: 1.1d scrub ok 2024-09-11T03:50:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:58 smithi175 ceph-mon[30894]: pgmap v1413: 97 pgs: 97 active+clean; 638 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 2.9 MiB/s wr, 17 op/s 2024-09-11T03:50:58.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:58 smithi175 ceph-mon[30894]: 2.29 scrub starts 2024-09-11T03:50:58.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:58 smithi175 ceph-mon[30894]: 2.29 scrub ok 2024-09-11T03:50:58.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:58 smithi175 ceph-mon[30894]: osdmap e840: 12 total, 12 up, 12 in 2024-09-11T03:50:58.524 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:50:58.545 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:58.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:58.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:50:58.647 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:58.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:58.669 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:58.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:50:58.770 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:58.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:58.794 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:58.794 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:58.801 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T03:50:58.847 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:58.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:58.921 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:58.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:58.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:59.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:59.049 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:59.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:59.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:59.130 INFO:tasks.rados.rados.2.smithi175.stdout:3483: done (4 left) 2024-09-11T03:50:59.130 INFO:tasks.rados.rados.2.smithi175.stdout:3484: done (3 left) 2024-09-11T03:50:59.130 INFO:tasks.rados.rados.2.smithi175.stdout:3485: done (2 left) 2024-09-11T03:50:59.130 INFO:tasks.rados.rados.2.smithi175.stdout:3486: done (1 left) 2024-09-11T03:50:59.130 INFO:tasks.rados.rados.2.smithi175.stdout:3487: done (0 left) 2024-09-11T03:50:59.130 INFO:tasks.rados.rados.2.smithi175.stdout:3488: snap_create 2024-09-11T03:50:59.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:50:59.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:59.178 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:59.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:50:59.279 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:50:59.306 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:59.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:59.307 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:50:59.337 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:50:59.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:59 smithi084 ceph-mon[27254]: 2.3 scrub starts 2024-09-11T03:50:59.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:50:59 smithi084 ceph-mon[27254]: 2.3 scrub ok 2024-09-11T03:50:59.407 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:59.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:59 smithi121 ceph-mon[30729]: 2.3 scrub starts 2024-09-11T03:50:59.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:50:59 smithi121 ceph-mon[30729]: 2.3 scrub ok 2024-09-11T03:50:59.434 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:59.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:59.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:59.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:59 smithi175 ceph-mon[30894]: 2.3 scrub starts 2024-09-11T03:50:59.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:50:59 smithi175 ceph-mon[30894]: 2.3 scrub ok 2024-09-11T03:50:59.536 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:50:59.562 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:50:59.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:59.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:50:59.663 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:50:59.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:50:59.689 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:59.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:59.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:50:59.787 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:50:59.788 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:50:59.788 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:50:59.788 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:50:59.791 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:50:59.817 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:50:59.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:59.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:50:59.919 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:50:59.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:50:59.941 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:50:59.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:00.043 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:00.064 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:00.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:00.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:00.128 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:51:00.132 INFO:tasks.rados.rados.2.smithi175.stdout:3488: done (0 left) 2024-09-11T03:51:00.132 INFO:tasks.rados.rados.2.smithi175.stdout:3489: read oid 32 snap 381 2024-09-11T03:51:00.132 INFO:tasks.rados.rados.2.smithi175.stdout:3489: expect (ObjNum 984 snap 315 seq_num 984) 2024-09-11T03:51:00.132 INFO:tasks.rados.rados.2.smithi175.stdout:3490: delete oid 19 current snap is 387 2024-09-11T03:51:00.134 INFO:tasks.rados.rados.2.smithi175.stdout:3490: done (1 left) 2024-09-11T03:51:00.134 INFO:tasks.rados.rados.2.smithi175.stdout:3491: write (excl) oid 38 current snap is 387 2024-09-11T03:51:00.134 INFO:tasks.rados.rados.2.smithi175.stdout:3491: seq_num 1185 ranges {716295=404780,1593515=614409,2708717=492747,3640519=275776} 2024-09-11T03:51:00.141 INFO:tasks.rados.rados.2.smithi175.stdout:3491: writing smithi17557870-38 from 716295 to 1121075 tid 1 2024-09-11T03:51:00.148 INFO:tasks.rados.rados.2.smithi175.stdout:3491: writing smithi17557870-38 from 1593515 to 2207924 tid 2 2024-09-11T03:51:00.153 INFO:tasks.rados.rados.2.smithi175.stdout:3491: writing smithi17557870-38 from 2708717 to 3201464 tid 3 2024-09-11T03:51:00.158 INFO:tasks.rados.rados.2.smithi175.stdout:3491: writing smithi17557870-38 from 3640519 to 3916295 tid 4 2024-09-11T03:51:00.158 INFO:tasks.rados.rados.2.smithi175.stdout:3492: read oid 4 snap -1 2024-09-11T03:51:00.158 INFO:tasks.rados.rados.2.smithi175.stdout:3492: expect (ObjNum 1169 snap 383 seq_num 1169) 2024-09-11T03:51:00.158 INFO:tasks.rados.rados.2.smithi175.stdout:3493: read oid 46 snap -1 2024-09-11T03:51:00.158 INFO:tasks.rados.rados.2.smithi175.stdout:3493: expect deleted 2024-09-11T03:51:00.158 INFO:tasks.rados.rados.2.smithi175.stdout:3494: snap_create 2024-09-11T03:51:00.158 INFO:tasks.rados.rados.2.smithi175.stdout:3489: expect (ObjNum 984 snap 315 seq_num 984) 2024-09-11T03:51:00.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:00.189 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:00.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:00.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:00.292 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:00.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:00.317 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:00.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:00.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:00 smithi084 ceph-mon[27254]: pgmap v1416: 97 pgs: 97 active+clean; 648 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.9 MiB/s wr, 28 op/s 2024-09-11T03:51:00.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:00 smithi084 ceph-mon[27254]: 1.d scrub starts 2024-09-11T03:51:00.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:00 smithi084 ceph-mon[27254]: 1.d scrub ok 2024-09-11T03:51:00.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:00 smithi084 ceph-mon[27254]: osdmap e841: 12 total, 12 up, 12 in 2024-09-11T03:51:00.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:00 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:51:00.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:00 smithi084 ceph-mon[27254]: 2.9 scrub ok 2024-09-11T03:51:00.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:00 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3723568311' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:51:00.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:00 smithi084 ceph-mon[27254]: 2.25 scrub starts 2024-09-11T03:51:00.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:00 smithi084 ceph-mon[27254]: 2.25 scrub ok 2024-09-11T03:51:00.400 INFO:tasks.rados.rados.2.smithi175.stdout:3491: finishing write tid 1 to smithi17557870-38 2024-09-11T03:51:00.400 INFO:tasks.rados.rados.2.smithi175.stdout:3491: finishing write tid 2 to smithi17557870-38 2024-09-11T03:51:00.400 INFO:tasks.rados.rados.2.smithi175.stdout:3491: finishing write tid 3 to smithi17557870-38 2024-09-11T03:51:00.401 INFO:tasks.rados.rados.2.smithi175.stdout:3491: finishing write tid 4 to smithi17557870-38 2024-09-11T03:51:00.401 INFO:tasks.rados.rados.2.smithi175.stdout:3491: finishing write tid 5 to smithi17557870-38 2024-09-11T03:51:00.401 INFO:tasks.rados.rados.2.smithi175.stdout:3491: finishing write tid 6 to smithi17557870-38 2024-09-11T03:51:00.401 INFO:tasks.rados.rados.2.smithi175.stdout:3491: oid 38 updating version 0 to 1351 2024-09-11T03:51:00.401 INFO:tasks.rados.rados.2.smithi175.stdout:3491: oid 38 updating version 1351 to 1352 2024-09-11T03:51:00.401 INFO:tasks.rados.rados.2.smithi175.stdout:3491: oid 38 version 1352 is already newer than 1350 2024-09-11T03:51:00.401 INFO:tasks.rados.rados.2.smithi175.stdout:3491: oid 38 version 1352 is already newer than 1349 2024-09-11T03:51:00.401 INFO:tasks.rados.rados.2.smithi175.stdout:3491: oid 38 updating version 1352 to 1353 2024-09-11T03:51:00.401 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 1353 (ObjNum 1185 snap 387 seq_num 1185) dirty exists 2024-09-11T03:51:00.401 INFO:tasks.rados.rados.2.smithi175.stdout:3491: left oid 38 (ObjNum 1185 snap 387 seq_num 1185) 2024-09-11T03:51:00.401 INFO:tasks.rados.rados.2.smithi175.stdout:3492: expect (ObjNum 1169 snap 383 seq_num 1169) 2024-09-11T03:51:00.419 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:00 smithi121 ceph-mon[30729]: pgmap v1416: 97 pgs: 97 active+clean; 648 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.9 MiB/s wr, 28 op/s 2024-09-11T03:51:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:00 smithi121 ceph-mon[30729]: 1.d scrub starts 2024-09-11T03:51:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:00 smithi121 ceph-mon[30729]: 1.d scrub ok 2024-09-11T03:51:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:00 smithi121 ceph-mon[30729]: osdmap e841: 12 total, 12 up, 12 in 2024-09-11T03:51:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:00 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:51:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:00 smithi121 ceph-mon[30729]: 2.9 scrub ok 2024-09-11T03:51:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:00 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3723568311' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:51:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:00 smithi121 ceph-mon[30729]: 2.25 scrub starts 2024-09-11T03:51:00.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:00 smithi121 ceph-mon[30729]: 2.25 scrub ok 2024-09-11T03:51:00.440 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:00.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:00 smithi175 ceph-mon[30894]: pgmap v1416: 97 pgs: 97 active+clean; 648 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.9 MiB/s wr, 28 op/s 2024-09-11T03:51:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:00 smithi175 ceph-mon[30894]: 1.d scrub starts 2024-09-11T03:51:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:00 smithi175 ceph-mon[30894]: 1.d scrub ok 2024-09-11T03:51:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:00 smithi175 ceph-mon[30894]: osdmap e841: 12 total, 12 up, 12 in 2024-09-11T03:51:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:00 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:51:00.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:00 smithi175 ceph-mon[30894]: 2.9 scrub ok 2024-09-11T03:51:00.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:00 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3723568311' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:51:00.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:00 smithi175 ceph-mon[30894]: 2.25 scrub starts 2024-09-11T03:51:00.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:00 smithi175 ceph-mon[30894]: 2.25 scrub ok 2024-09-11T03:51:00.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T03:51:00.542 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:00.568 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:00.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:00.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:00.670 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:00.696 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:00.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:00.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:00.798 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:00.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:00.821 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:00.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:00.923 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:00.944 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:00.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:00.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:01.046 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:01.068 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:01.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:01.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:01.143 INFO:tasks.rados.rados.2.smithi175.stdout:3489: done (4 left) 2024-09-11T03:51:01.143 INFO:tasks.rados.rados.2.smithi175.stdout:3491: done (3 left) 2024-09-11T03:51:01.143 INFO:tasks.rados.rados.2.smithi175.stdout:3492: done (2 left) 2024-09-11T03:51:01.143 INFO:tasks.rados.rados.2.smithi175.stdout:3493: done (1 left) 2024-09-11T03:51:01.143 INFO:tasks.rados.rados.2.smithi175.stdout:3494: done (0 left) 2024-09-11T03:51:01.143 INFO:tasks.rados.rados.2.smithi175.stdout:3495: rollback oid 9 current snap is 388 2024-09-11T03:51:01.143 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 9 to 345 2024-09-11T03:51:01.143 INFO:tasks.rados.rados.2.smithi175.stdout:3496: write oid 2 current snap is 388 2024-09-11T03:51:01.143 INFO:tasks.rados.rados.2.smithi175.stdout:3496: seq_num 1186 ranges {733102=596382,1768350=758067,3136140=704019} 2024-09-11T03:51:01.149 INFO:tasks.rados.rados.2.smithi175.stdout:3496: writing smithi17557870-2 from 733102 to 1329484 tid 1 2024-09-11T03:51:01.156 INFO:tasks.rados.rados.2.smithi175.stdout:3496: writing smithi17557870-2 from 1768350 to 2526417 tid 2 2024-09-11T03:51:01.164 INFO:tasks.rados.rados.2.smithi175.stdout:3496: writing smithi17557870-2 from 3136140 to 3840159 tid 3 2024-09-11T03:51:01.164 INFO:tasks.rados.rados.2.smithi175.stdout:3497: rollback oid 20 current snap is 388 2024-09-11T03:51:01.164 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 20 to 331 2024-09-11T03:51:01.164 INFO:tasks.rados.rados.2.smithi175.stdout:3495: finishing rollback tid 0 to smithi17557870-9 2024-09-11T03:51:01.164 INFO:tasks.rados.rados.2.smithi175.stdout:3495: finishing rollback tid 1 to smithi17557870-9 2024-09-11T03:51:01.164 INFO:tasks.rados.rados.2.smithi175.stdout:3495: finishing rollback tid 2 to smithi17557870-9 2024-09-11T03:51:01.164 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 1241 (ObjNum 1033 snap 340 seq_num 1033) dirty exists 2024-09-11T03:51:01.164 INFO:tasks.rados.rados.2.smithi175.stdout:3496: finishing write tid 1 to smithi17557870-2 2024-09-11T03:51:01.164 INFO:tasks.rados.rados.2.smithi175.stdout:3496: finishing write tid 2 to smithi17557870-2 2024-09-11T03:51:01.164 INFO:tasks.rados.rados.2.smithi175.stdout:3495: done (2 left) 2024-09-11T03:51:01.164 INFO:tasks.rados.rados.2.smithi175.stdout:3498: read oid 15 snap 286 2024-09-11T03:51:01.164 INFO:tasks.rados.rados.2.smithi175.stdout:3498: expect (ObjNum 857 snap 269 seq_num 857) 2024-09-11T03:51:01.165 INFO:tasks.rados.rados.2.smithi175.stdout:3499: delete oid 45 current snap is 388 2024-09-11T03:51:01.167 INFO:tasks.rados.rados.2.smithi175.stdout:3499: done (3 left) 2024-09-11T03:51:01.168 INFO:tasks.rados.rados.2.smithi175.stdout:3500: write oid 37 current snap is 388 2024-09-11T03:51:01.168 INFO:tasks.rados.rados.2.smithi175.stdout:3500: seq_num 1188 ranges {766716=746880,2168961=761736,3474633=442916} 2024-09-11T03:51:01.170 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:01.176 INFO:tasks.rados.rados.2.smithi175.stdout:3500: writing smithi17557870-37 from 766716 to 1513596 tid 1 2024-09-11T03:51:01.184 INFO:tasks.rados.rados.2.smithi175.stdout:3500: writing smithi17557870-37 from 2168961 to 2930697 tid 2 2024-09-11T03:51:01.242 INFO:tasks.rados.rados.2.smithi175.stdout:3500: writing smithi17557870-37 from 3474633 to 3917549 tid 3 2024-09-11T03:51:01.242 INFO:tasks.rados.rados.2.smithi175.stdout:3501: snap_remove snap 344 2024-09-11T03:51:01.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:01.243 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:01.243 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:01.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:01.253 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:51:01.253 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:51:01.253 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-11T03:51:01.253 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:51:01.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:01 smithi084 ceph-mon[27254]: 2.27 scrub starts 2024-09-11T03:51:01.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:01 smithi084 ceph-mon[27254]: 2.27 scrub ok 2024-09-11T03:51:01.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:01 smithi084 ceph-mon[27254]: 1.11 scrub starts 2024-09-11T03:51:01.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:01 smithi084 ceph-mon[27254]: 1.11 scrub ok 2024-09-11T03:51:01.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:01 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:51:01.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:01 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:51:01.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3723568311' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:51:01.275 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:01 smithi084 ceph-mon[27254]: osdmap e842: 12 total, 12 up, 12 in 2024-09-11T03:51:01.276 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:01 smithi084 ceph-mon[27254]: 2.4 scrub starts 2024-09-11T03:51:01.276 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:01 smithi084 ceph-mon[27254]: 2.4 scrub ok 2024-09-11T03:51:01.276 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:01 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:51:01.276 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:01 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:51:01.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:01.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:01.367 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:01.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:01 smithi121 ceph-mon[30729]: 2.27 scrub starts 2024-09-11T03:51:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:01 smithi121 ceph-mon[30729]: 2.27 scrub ok 2024-09-11T03:51:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:01 smithi121 ceph-mon[30729]: 1.11 scrub starts 2024-09-11T03:51:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:01 smithi121 ceph-mon[30729]: 1.11 scrub ok 2024-09-11T03:51:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:01 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:51:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:01 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:51:01.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3723568311' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:51:01.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:01 smithi121 ceph-mon[30729]: osdmap e842: 12 total, 12 up, 12 in 2024-09-11T03:51:01.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:01 smithi121 ceph-mon[30729]: 2.4 scrub starts 2024-09-11T03:51:01.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:01 smithi121 ceph-mon[30729]: 2.4 scrub ok 2024-09-11T03:51:01.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:01 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:51:01.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:01 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:51:01.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:01 smithi175 ceph-mon[30894]: 2.27 scrub starts 2024-09-11T03:51:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:01 smithi175 ceph-mon[30894]: 2.27 scrub ok 2024-09-11T03:51:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:01 smithi175 ceph-mon[30894]: 1.11 scrub starts 2024-09-11T03:51:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:01 smithi175 ceph-mon[30894]: 1.11 scrub ok 2024-09-11T03:51:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:01 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:51:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:01 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:51:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3723568311' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:51:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:01 smithi175 ceph-mon[30894]: osdmap e842: 12 total, 12 up, 12 in 2024-09-11T03:51:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:01 smithi175 ceph-mon[30894]: 2.4 scrub starts 2024-09-11T03:51:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:01 smithi175 ceph-mon[30894]: 2.4 scrub ok 2024-09-11T03:51:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:01 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:51:01.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:01 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:51:01.494 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:01.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:01.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:01.596 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:01.618 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:01.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:01.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:01.720 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:01.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:01.742 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:01.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:01.751 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:51:01.751 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:51:01.757 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:01.767 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:51:01.767 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:51:01.785 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:51:01.785 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:51:01.791 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:01.802 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:51:01.803 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:51:01.820 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:51:01.821 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:51:01.842 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:01.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:01.864 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:01.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:01.965 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:01.986 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:01.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:01.987 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:02.042 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:51:02.043 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:51:02.060 INFO:tasks.rados.rados.2.smithi175.stdout:3501: done (4 left) 2024-09-11T03:51:02.060 INFO:tasks.rados.rados.2.smithi175.stdout:3502: write (excl) oid 23 current snap is 388 2024-09-11T03:51:02.061 INFO:tasks.rados.rados.2.smithi175.stdout:3502: seq_num 1189 ranges {783523=422129,1827443=621747,3218409=654188} 2024-09-11T03:51:02.067 INFO:tasks.rados.rados.2.smithi175.stdout:3502: writing smithi17557870-23 from 783523 to 1205652 tid 1 2024-09-11T03:51:02.075 INFO:tasks.rados.rados.2.smithi175.stdout:3502: writing smithi17557870-23 from 1827443 to 2449190 tid 2 2024-09-11T03:51:02.084 INFO:tasks.rados.rados.2.smithi175.stdout:3502: writing smithi17557870-23 from 3218409 to 3872597 tid 3 2024-09-11T03:51:02.084 INFO:tasks.rados.rados.2.smithi175.stdout:3503: write oid 50 current snap is 388 2024-09-11T03:51:02.084 INFO:tasks.rados.rados.2.smithi175.stdout:3503: seq_num 1190 ranges {400330=497378,1485925=514405} 2024-09-11T03:51:02.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:02.089 INFO:tasks.rados.rados.2.smithi175.stdout:3503: writing smithi17557870-50 from 400330 to 897708 tid 1 2024-09-11T03:51:02.094 INFO:tasks.rados.rados.2.smithi175.stdout:3503: writing smithi17557870-50 from 1485925 to 2000330 tid 2 2024-09-11T03:51:02.094 INFO:tasks.rados.rados.2.smithi175.stdout:3504: snap_remove snap 347 2024-09-11T03:51:02.095 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:51:02.095 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:51:02.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:02.111 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:02.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:02.113 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:51:02.113 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:51:02.130 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:51:02.130 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:51:02.148 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:51:02.148 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:51:02.164 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:51:02.165 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:51:02.184 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:51:02.184 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:51:02.212 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:02.233 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:02.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:02.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:02.269 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:02 smithi084 ceph-mon[27254]: pgmap v1419: 97 pgs: 97 active+clean; 648 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2024-09-11T03:51:02.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:02 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:51:02.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:02 smithi084 ceph-mon[27254]: 2.8 scrub ok 2024-09-11T03:51:02.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:02 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:51:02.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:02 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:51:02.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:02 smithi084 ceph-mon[27254]: osdmap e843: 12 total, 12 up, 12 in 2024-09-11T03:51:02.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:02 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/36760139' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:02.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:02 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:51:02.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:02 smithi084 ceph-mon[27254]: 2.c scrub ok 2024-09-11T03:51:02.270 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:02 smithi084 ceph-mon[27254]: osdmap e844: 12 total, 12 up, 12 in 2024-09-11T03:51:02.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:02.356 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:02.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:02.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:02 smithi121 ceph-mon[30729]: pgmap v1419: 97 pgs: 97 active+clean; 648 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2024-09-11T03:51:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:02 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:51:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:02 smithi121 ceph-mon[30729]: 2.8 scrub ok 2024-09-11T03:51:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:02 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:51:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:02 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:51:02.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:02 smithi121 ceph-mon[30729]: osdmap e843: 12 total, 12 up, 12 in 2024-09-11T03:51:02.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:02 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/36760139' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:02.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:02 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:51:02.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:02 smithi121 ceph-mon[30729]: 2.c scrub ok 2024-09-11T03:51:02.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:02 smithi121 ceph-mon[30729]: osdmap e844: 12 total, 12 up, 12 in 2024-09-11T03:51:02.458 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:02 smithi175 ceph-mon[30894]: pgmap v1419: 97 pgs: 97 active+clean; 648 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2024-09-11T03:51:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:02 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:51:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:02 smithi175 ceph-mon[30894]: 2.8 scrub ok 2024-09-11T03:51:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:02 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:51:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:02 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:51:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:02 smithi175 ceph-mon[30894]: osdmap e843: 12 total, 12 up, 12 in 2024-09-11T03:51:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:02 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/36760139' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:02 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:51:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:02 smithi175 ceph-mon[30894]: 2.c scrub ok 2024-09-11T03:51:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:02 smithi175 ceph-mon[30894]: osdmap e844: 12 total, 12 up, 12 in 2024-09-11T03:51:02.484 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:02.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:02.586 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:02.607 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:02.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:02.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:02.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:02.735 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:02.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:02.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:02.812 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:51:02.813 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:51:02.837 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:02.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:02.859 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:02.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:02.960 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:02.981 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:02.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:02.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:02.990 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:51:02.990 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:51:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3504: done (6 left) 2024-09-11T03:51:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3505: delete oid 10 current snap is 388 2024-09-11T03:51:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3496: finishing write tid 3 to smithi17557870-2 2024-09-11T03:51:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3496: finishing write tid 4 to smithi17557870-2 2024-09-11T03:51:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3496: finishing write tid 5 to smithi17557870-2 2024-09-11T03:51:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3496: oid 2 updating version 0 to 1244 2024-09-11T03:51:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3496: oid 2 updating version 1244 to 1245 2024-09-11T03:51:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3496: oid 2 version 1245 is already newer than 1242 2024-09-11T03:51:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3496: oid 2 version 1245 is already newer than 1243 2024-09-11T03:51:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 1245 (ObjNum 1186 snap 388 seq_num 1186) dirty exists 2024-09-11T03:51:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3496: left oid 2 (ObjNum 1186 snap 388 seq_num 1186) 2024-09-11T03:51:03.064 INFO:tasks.rados.rados.2.smithi175.stdout:3497: finishing rollback tid 1 to smithi17557870-20 2024-09-11T03:51:03.064 INFO:tasks.rados.rados.2.smithi175.stdout:3498: expect (ObjNum 857 snap 269 seq_num 857) 2024-09-11T03:51:03.083 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:03.103 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:03.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:03.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:03.202 INFO:tasks.rados.rados.2.smithi175.stdout:3497: finishing rollback tid 2 to smithi17557870-20 2024-09-11T03:51:03.202 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 1033 (ObjNum 983 snap 315 seq_num 983) dirty exists 2024-09-11T03:51:03.202 INFO:tasks.rados.rados.2.smithi175.stdout:3500: finishing write tid 1 to smithi17557870-37 2024-09-11T03:51:03.202 INFO:tasks.rados.rados.2.smithi175.stdout:3500: finishing write tid 2 to smithi17557870-37 2024-09-11T03:51:03.202 INFO:tasks.rados.rados.2.smithi175.stdout:3500: finishing write tid 3 to smithi17557870-37 2024-09-11T03:51:03.202 INFO:tasks.rados.rados.2.smithi175.stdout:3496: done (6 left) 2024-09-11T03:51:03.202 INFO:tasks.rados.rados.2.smithi175.stdout:3497: done (5 left) 2024-09-11T03:51:03.202 INFO:tasks.rados.rados.2.smithi175.stdout:3498: done (4 left) 2024-09-11T03:51:03.202 INFO:tasks.rados.rados.2.smithi175.stdout:3505: done (3 left) 2024-09-11T03:51:03.202 INFO:tasks.rados.rados.2.smithi175.stdout:3506: snap_create 2024-09-11T03:51:03.202 INFO:tasks.rados.rados.2.smithi175.stdout:3500: finishing write tid 4 to smithi17557870-37 2024-09-11T03:51:03.202 INFO:tasks.rados.rados.2.smithi175.stdout:3500: finishing write tid 5 to smithi17557870-37 2024-09-11T03:51:03.203 INFO:tasks.rados.rados.2.smithi175.stdout:3500: oid 37 updating version 0 to 1255 2024-09-11T03:51:03.203 INFO:tasks.rados.rados.2.smithi175.stdout:3500: oid 37 updating version 1255 to 1257 2024-09-11T03:51:03.203 INFO:tasks.rados.rados.2.smithi175.stdout:3500: oid 37 updating version 1257 to 1258 2024-09-11T03:51:03.203 INFO:tasks.rados.rados.2.smithi175.stdout:3500: oid 37 version 1258 is already newer than 1256 2024-09-11T03:51:03.203 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 1258 (ObjNum 1188 snap 388 seq_num 1188) dirty exists 2024-09-11T03:51:03.204 INFO:tasks.rados.rados.2.smithi175.stdout:3500: left oid 37 (ObjNum 1188 snap 388 seq_num 1188) 2024-09-11T03:51:03.204 INFO:tasks.rados.rados.2.smithi175.stdout:3502: finishing write tid 1 to smithi17557870-23 2024-09-11T03:51:03.204 INFO:tasks.rados.rados.2.smithi175.stdout:3502: finishing write tid 2 to smithi17557870-23 2024-09-11T03:51:03.204 INFO:tasks.rados.rados.2.smithi175.stdout:3502: finishing write tid 3 to smithi17557870-23 2024-09-11T03:51:03.204 INFO:tasks.rados.rados.2.smithi175.stdout:3502: finishing write tid 4 to smithi17557870-23 2024-09-11T03:51:03.204 INFO:tasks.rados.rados.2.smithi175.stdout:3502: finishing write tid 5 to smithi17557870-23 2024-09-11T03:51:03.204 INFO:tasks.rados.rados.2.smithi175.stdout:3502: oid 23 updating version 0 to 984 2024-09-11T03:51:03.204 INFO:tasks.rados.rados.2.smithi175.stdout:3502: oid 23 version 984 is already newer than 983 2024-09-11T03:51:03.204 INFO:tasks.rados.rados.2.smithi175.stdout:3502: oid 23 updating version 984 to 985 2024-09-11T03:51:03.204 INFO:tasks.rados.rados.2.smithi175.stdout:3502: oid 23 updating version 985 to 986 2024-09-11T03:51:03.204 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 986 (ObjNum 1189 snap 388 seq_num 1189) dirty exists 2024-09-11T03:51:03.204 INFO:tasks.rados.rados.2.smithi175.stdout:3502: left oid 23 (ObjNum 1189 snap 388 seq_num 1189) 2024-09-11T03:51:03.204 INFO:tasks.rados.rados.2.smithi175.stdout:3503: finishing write tid 1 to smithi17557870-50 2024-09-11T03:51:03.205 INFO:tasks.rados.rados.2.smithi175.stdout:3503: finishing write tid 2 to smithi17557870-50 2024-09-11T03:51:03.205 INFO:tasks.rados.rados.2.smithi175.stdout:3503: finishing write tid 3 to smithi17557870-50 2024-09-11T03:51:03.205 INFO:tasks.rados.rados.2.smithi175.stdout:3503: finishing write tid 4 to smithi17557870-50 2024-09-11T03:51:03.205 INFO:tasks.rados.rados.2.smithi175.stdout:3503: oid 50 updating version 0 to 1362 2024-09-11T03:51:03.205 INFO:tasks.rados.rados.2.smithi175.stdout:3503: oid 50 updating version 1362 to 1363 2024-09-11T03:51:03.205 INFO:tasks.rados.rados.2.smithi175.stdout:3503: oid 50 version 1363 is already newer than 1361 2024-09-11T03:51:03.205 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 1363 (ObjNum 1190 snap 388 seq_num 1190) dirty exists 2024-09-11T03:51:03.205 INFO:tasks.rados.rados.2.smithi175.stdout:3503: left oid 50 (ObjNum 1190 snap 388 seq_num 1190) 2024-09-11T03:51:03.205 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:03.230 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:03.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:03.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:03.243 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:51:03.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:03.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:03 smithi084 ceph-mon[27254]: 1.1e scrub starts 2024-09-11T03:51:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:03 smithi084 ceph-mon[27254]: 1.1e scrub ok 2024-09-11T03:51:03.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:03 smithi084 ceph-mon[27254]: osdmap e845: 12 total, 12 up, 12 in 2024-09-11T03:51:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:03 smithi121 ceph-mon[30729]: 1.1e scrub starts 2024-09-11T03:51:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:03 smithi121 ceph-mon[30729]: 1.1e scrub ok 2024-09-11T03:51:03.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:03 smithi121 ceph-mon[30729]: osdmap e845: 12 total, 12 up, 12 in 2024-09-11T03:51:03.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:03.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:03 smithi175 ceph-mon[30894]: 1.1e scrub starts 2024-09-11T03:51:03.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:03 smithi175 ceph-mon[30894]: 1.1e scrub ok 2024-09-11T03:51:03.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:03 smithi175 ceph-mon[30894]: osdmap e845: 12 total, 12 up, 12 in 2024-09-11T03:51:03.478 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:03.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:03.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:03.580 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:03.602 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:03.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:03.603 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:03.704 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:03.730 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:03.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:03.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:03.831 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:03.853 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:03.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:03.854 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:03.952 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1421,"stamp":"2024-09-11T03:51:02.058211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":699865321,"num_objects":306,"num_object_clones":254,"num_object_copies":918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":306,"num_whiteouts":14,"num_read":774,"num_read_kb":777364,"num_write":569,"num_write_kb":250236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41316,"ondisk_log_size":41316,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":1788472,"kb_used_data":1214276,"kb_used_omap":18,"kb_used_meta":573869,"kb_avail":1123153352,"statfs":{"total":1151940427776,"available":1150109032448,"internally_reserved":0,"allocated":1243418624,"data_stored":1672606448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":587642171},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18047157,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":-1,"num_read":43,"num_read_kb":34269,"num_write":31,"num_write_kb":11861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.013001"},"pg_stats":[{"pgid":"2.41","version":"567'235","reported_seq":3933,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.386512+0000","last_change":"2024-09-11T03:50:56.120692+0000","last_active":"2024-09-11T03:50:57.386512+0000","last_peered":"2024-09-11T03:50:57.386512+0000","last_clean":"2024-09-11T03:50:57.386512+0000","last_became_active":"2024-09-11T03:50:33.046488+0000","last_became_peered":"2024-09-11T03:50:33.046488+0000","last_unstale":"2024-09-11T03:50:57.386512+0000","last_undegraded":"2024-09-11T03:50:57.386512+0000","last_fullsized":"2024-09-11T03:50:57.386512+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":7,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:50:35.786143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.786143+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:38.876464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014607330000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.40","version":"563'192","reported_seq":3995,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.042364+0000","last_change":"2024-09-11T03:50:56.113422+0000","last_active":"2024-09-11T03:50:57.042364+0000","last_peered":"2024-09-11T03:50:57.042364+0000","last_clean":"2024-09-11T03:50:57.042364+0000","last_became_active":"2024-09-11T03:50:33.047623+0000","last_became_peered":"2024-09-11T03:50:33.047623+0000","last_unstale":"2024-09-11T03:50:57.042364+0000","last_undegraded":"2024-09-11T03:50:57.042364+0000","last_fullsized":"2024-09-11T03:50:57.042364+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":7,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:50:35.472175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.472175+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:02.207945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096096500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.3f","version":"840'778","reported_seq":4529,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.186303+0000","last_change":"2024-09-11T03:50:58.060543+0000","last_active":"2024-09-11T03:51:01.186303+0000","last_peered":"2024-09-11T03:51:01.186303+0000","last_clean":"2024-09-11T03:51:01.186303+0000","last_became_active":"2024-09-11T03:50:33.046798+0000","last_became_peered":"2024-09-11T03:50:33.046798+0000","last_unstale":"2024-09-11T03:51:01.186303+0000","last_undegraded":"2024-09-11T03:51:01.186303+0000","last_fullsized":"2024-09-11T03:51:01.186303+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"817'762","last_scrub_stamp":"2024-09-11T03:50:36.829688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.829688+0000","objects_scrubbed":7,"log_size":778,"log_dups_size":0,"ondisk_log_size":778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:53.787671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0087700420000000005,"stat_sum":{"num_bytes":11807646,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.3e","version":"837'1070","reported_seq":4768,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.046274+0000","last_change":"2024-09-11T03:50:56.119639+0000","last_active":"2024-09-11T03:50:57.046274+0000","last_peered":"2024-09-11T03:50:57.046274+0000","last_clean":"2024-09-11T03:50:57.046274+0000","last_became_active":"2024-09-11T03:50:33.045235+0000","last_became_peered":"2024-09-11T03:50:33.045235+0000","last_unstale":"2024-09-11T03:50:57.046274+0000","last_undegraded":"2024-09-11T03:50:57.046274+0000","last_fullsized":"2024-09-11T03:50:57.046274+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"815'1064","last_scrub_stamp":"2024-09-11T03:50:36.212789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.212789+0000","objects_scrubbed":8,"log_size":1070,"log_dups_size":0,"ondisk_log_size":1070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:02.276216+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0065339980000000001,"stat_sum":{"num_bytes":19095897,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.3d","version":"567'752","reported_seq":3284,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.141715+0000","last_change":"2024-09-11T03:50:58.056737+0000","last_active":"2024-09-11T03:51:01.141715+0000","last_peered":"2024-09-11T03:51:01.141715+0000","last_clean":"2024-09-11T03:51:01.141715+0000","last_became_active":"2024-09-11T03:50:33.039143+0000","last_became_peered":"2024-09-11T03:50:33.039143+0000","last_unstale":"2024-09-11T03:51:01.141715+0000","last_undegraded":"2024-09-11T03:51:01.141715+0000","last_fullsized":"2024-09-11T03:51:01.141715+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:50:35.414581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.414581+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:59.637284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043248410000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.3c","version":"827'778","reported_seq":4543,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.185793+0000","last_change":"2024-09-11T03:50:58.050987+0000","last_active":"2024-09-11T03:51:01.185793+0000","last_peered":"2024-09-11T03:51:01.185793+0000","last_clean":"2024-09-11T03:51:01.185793+0000","last_became_active":"2024-09-11T03:50:33.048487+0000","last_became_peered":"2024-09-11T03:50:33.048487+0000","last_unstale":"2024-09-11T03:51:01.185793+0000","last_undegraded":"2024-09-11T03:51:01.185793+0000","last_fullsized":"2024-09-11T03:51:01.185793+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"806'776","last_scrub_stamp":"2024-09-11T03:50:35.839653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.839653+0000","objects_scrubbed":4,"log_size":778,"log_dups_size":0,"ondisk_log_size":778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:43.877026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066374800000000003,"stat_sum":{"num_bytes":8735071,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,10],"acting":[7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.3b","version":"840'919","reported_seq":4772,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.143801+0000","last_change":"2024-09-11T03:50:58.061187+0000","last_active":"2024-09-11T03:51:01.143801+0000","last_peered":"2024-09-11T03:51:01.143801+0000","last_clean":"2024-09-11T03:51:01.143801+0000","last_became_active":"2024-09-11T03:50:33.046784+0000","last_became_peered":"2024-09-11T03:50:33.046784+0000","last_unstale":"2024-09-11T03:51:01.143801+0000","last_undegraded":"2024-09-11T03:51:01.143801+0000","last_fullsized":"2024-09-11T03:51:01.143801+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"817'886","last_scrub_stamp":"2024-09-11T03:50:35.496439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.496439+0000","objects_scrubbed":8,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:49.366473+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0057387109999999996,"stat_sum":{"num_bytes":24901152,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":10,"num_read_kb":8231,"num_write":11,"num_write_kb":1297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,10],"acting":[5,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.3a","version":"557'273","reported_seq":2772,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-09-11T03:50:58.057586+0000","last_change":"2024-09-11T03:50:58.057992+0000","last_active":"2024-09-11T03:50:58.057586+0000","last_peered":"2024-09-11T03:50:58.057586+0000","last_clean":"2024-09-11T03:50:58.057586+0000","last_became_active":"2024-09-11T03:50:33.043178+0000","last_became_peered":"2024-09-11T03:50:33.043178+0000","last_unstale":"2024-09-11T03:50:58.057586+0000","last_undegraded":"2024-09-11T03:50:58.057586+0000","last_fullsized":"2024-09-11T03:50:58.057586+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:50:37.286055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.286055+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:45.681746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001554817,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.39","version":"806'585","reported_seq":4382,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-09-11T03:50:58.058072+0000","last_change":"2024-09-11T03:50:58.058478+0000","last_active":"2024-09-11T03:50:58.058072+0000","last_peered":"2024-09-11T03:50:58.058072+0000","last_clean":"2024-09-11T03:50:58.058072+0000","last_became_active":"2024-09-11T03:50:33.043651+0000","last_became_peered":"2024-09-11T03:50:33.043651+0000","last_unstale":"2024-09-11T03:50:58.058072+0000","last_undegraded":"2024-09-11T03:50:58.058072+0000","last_fullsized":"2024-09-11T03:50:58.058072+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"806'585","last_scrub_stamp":"2024-09-11T03:50:36.264989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.264989+0000","objects_scrubbed":0,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:43.187678+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0019846249999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,2],"acting":[9,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.38","version":"808'564","reported_seq":4380,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.140081+0000","last_change":"2024-09-11T03:50:58.047048+0000","last_active":"2024-09-11T03:51:01.140081+0000","last_peered":"2024-09-11T03:51:01.140081+0000","last_clean":"2024-09-11T03:51:01.140081+0000","last_became_active":"2024-09-11T03:50:33.042740+0000","last_became_peered":"2024-09-11T03:50:33.042740+0000","last_unstale":"2024-09-11T03:51:01.140081+0000","last_undegraded":"2024-09-11T03:51:01.140081+0000","last_fullsized":"2024-09-11T03:51:01.140081+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"808'564","last_scrub_stamp":"2024-09-11T03:50:35.529587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.529587+0000","objects_scrubbed":0,"log_size":564,"log_dups_size":0,"ondisk_log_size":564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:58.448406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00074919299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,1],"acting":[8,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.37","version":"274'312","reported_seq":2338,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.387702+0000","last_change":"2024-09-11T03:50:56.125331+0000","last_active":"2024-09-11T03:50:57.387702+0000","last_peered":"2024-09-11T03:50:57.387702+0000","last_clean":"2024-09-11T03:50:57.387702+0000","last_became_active":"2024-09-11T03:50:07.077607+0000","last_became_peered":"2024-09-11T03:50:07.077607+0000","last_unstale":"2024-09-11T03:50:57.387702+0000","last_undegraded":"2024-09-11T03:50:57.387702+0000","last_fullsized":"2024-09-11T03:50:57.387702+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:50:17.684567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.684567+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:21.208110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041154629999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,9],"acting":[10,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.36","version":"842'628","reported_seq":4449,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.141913+0000","last_change":"2024-09-11T03:50:58.060521+0000","last_active":"2024-09-11T03:51:01.141913+0000","last_peered":"2024-09-11T03:51:01.141913+0000","last_clean":"2024-09-11T03:51:01.141913+0000","last_became_active":"2024-09-11T03:50:07.078667+0000","last_became_peered":"2024-09-11T03:50:07.078667+0000","last_unstale":"2024-09-11T03:51:01.141913+0000","last_undegraded":"2024-09-11T03:51:01.141913+0000","last_fullsized":"2024-09-11T03:51:01.141913+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'589","last_scrub_stamp":"2024-09-11T03:50:18.280936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.280936+0000","objects_scrubbed":12,"log_size":628,"log_dups_size":0,"ondisk_log_size":628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:38.814347+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0073604509999999996,"stat_sum":{"num_bytes":29342069,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":39,"num_read_kb":41151,"num_write":7,"num_write_kb":3927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,8],"acting":[1,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.35","version":"274'273","reported_seq":1522,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.141778+0000","last_change":"2024-09-11T03:50:58.057064+0000","last_active":"2024-09-11T03:51:01.141778+0000","last_peered":"2024-09-11T03:51:01.141778+0000","last_clean":"2024-09-11T03:51:01.141778+0000","last_became_active":"2024-09-11T03:50:07.079331+0000","last_became_peered":"2024-09-11T03:50:07.079331+0000","last_unstale":"2024-09-11T03:51:01.141778+0000","last_undegraded":"2024-09-11T03:51:01.141778+0000","last_fullsized":"2024-09-11T03:51:01.141778+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:50:17.314165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.314165+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:35.658879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043731719999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.34","version":"840'382","reported_seq":4135,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.167078+0000","last_change":"2024-09-11T03:50:58.058801+0000","last_active":"2024-09-11T03:51:01.167078+0000","last_peered":"2024-09-11T03:51:01.167078+0000","last_clean":"2024-09-11T03:51:01.167078+0000","last_became_active":"2024-09-11T03:50:07.080419+0000","last_became_peered":"2024-09-11T03:50:07.080419+0000","last_unstale":"2024-09-11T03:51:01.167078+0000","last_undegraded":"2024-09-11T03:51:01.167078+0000","last_fullsized":"2024-09-11T03:51:01.167078+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'358","last_scrub_stamp":"2024-09-11T03:50:17.613025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.613025+0000","objects_scrubbed":4,"log_size":382,"log_dups_size":0,"ondisk_log_size":382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:18.717669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0045920570000000001,"stat_sum":{"num_bytes":14855170,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":28891,"num_write":8,"num_write_kb":1518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.33","version":"274'111","reported_seq":1310,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.185820+0000","last_change":"2024-09-11T03:50:58.051107+0000","last_active":"2024-09-11T03:51:01.185820+0000","last_peered":"2024-09-11T03:51:01.185820+0000","last_clean":"2024-09-11T03:51:01.185820+0000","last_became_active":"2024-09-11T03:50:07.077684+0000","last_became_peered":"2024-09-11T03:50:07.077684+0000","last_unstale":"2024-09-11T03:51:01.185820+0000","last_undegraded":"2024-09-11T03:51:01.185820+0000","last_fullsized":"2024-09-11T03:51:01.185820+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:50:17.796171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.796171+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:23.305888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081167499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.32","version":"806'551","reported_seq":4270,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.045904+0000","last_change":"2024-09-11T03:50:56.112983+0000","last_active":"2024-09-11T03:50:57.045904+0000","last_peered":"2024-09-11T03:50:57.045904+0000","last_clean":"2024-09-11T03:50:57.045904+0000","last_became_active":"2024-09-11T03:50:07.077594+0000","last_became_peered":"2024-09-11T03:50:07.077594+0000","last_unstale":"2024-09-11T03:50:57.045904+0000","last_undegraded":"2024-09-11T03:50:57.045904+0000","last_fullsized":"2024-09-11T03:50:57.045904+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"778'549","last_scrub_stamp":"2024-09-11T03:50:17.199232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.199232+0000","objects_scrubbed":2,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:28.810078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00081053199999999998,"stat_sum":{"num_bytes":5630774,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9],"acting":[11,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.31","version":"274'107","reported_seq":1797,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-11T03:51:00.160289+0000","last_change":"2024-09-11T03:50:58.049562+0000","last_active":"2024-09-11T03:51:00.160289+0000","last_peered":"2024-09-11T03:51:00.160289+0000","last_clean":"2024-09-11T03:51:00.160289+0000","last_became_active":"2024-09-11T03:50:07.075956+0000","last_became_peered":"2024-09-11T03:50:07.075956+0000","last_unstale":"2024-09-11T03:51:00.160289+0000","last_undegraded":"2024-09-11T03:51:00.160289+0000","last_fullsized":"2024-09-11T03:51:00.160289+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:50:17.857515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.857515+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:18.896598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000787906,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,4],"acting":[0,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.30","version":"274'67","reported_seq":259,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.386565+0000","last_change":"2024-09-11T03:50:56.124549+0000","last_active":"2024-09-11T03:50:57.386565+0000","last_peered":"2024-09-11T03:50:57.386565+0000","last_clean":"2024-09-11T03:50:57.386565+0000","last_became_active":"2024-09-11T03:50:04.069393+0000","last_became_peered":"2024-09-11T03:50:04.069393+0000","last_unstale":"2024-09-11T03:50:57.386565+0000","last_undegraded":"2024-09-11T03:50:57.386565+0000","last_fullsized":"2024-09-11T03:50:57.386565+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:50:04.608333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.608333+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:26.267211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036460580000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,9],"acting":[10,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.2f","version":"557'465","reported_seq":3111,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-09-11T03:50:58.058674+0000","last_change":"2024-09-11T03:50:58.059184+0000","last_active":"2024-09-11T03:50:58.058674+0000","last_peered":"2024-09-11T03:50:58.058674+0000","last_clean":"2024-09-11T03:50:58.058674+0000","last_became_active":"2024-09-11T03:50:07.077840+0000","last_became_peered":"2024-09-11T03:50:07.077840+0000","last_unstale":"2024-09-11T03:50:58.058674+0000","last_undegraded":"2024-09-11T03:50:58.058674+0000","last_fullsized":"2024-09-11T03:50:58.058674+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:50:17.948546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:17.948546+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:21.159341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019123759999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.2e","version":"840'1101","reported_seq":4795,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.186200+0000","last_change":"2024-09-11T03:50:58.056113+0000","last_active":"2024-09-11T03:51:01.186200+0000","last_peered":"2024-09-11T03:51:01.186200+0000","last_clean":"2024-09-11T03:51:01.186200+0000","last_became_active":"2024-09-11T03:50:07.078910+0000","last_became_peered":"2024-09-11T03:50:07.078910+0000","last_unstale":"2024-09-11T03:51:01.186200+0000","last_undegraded":"2024-09-11T03:51:01.186200+0000","last_fullsized":"2024-09-11T03:51:01.186200+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'1047","last_scrub_stamp":"2024-09-11T03:50:18.779326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.779326+0000","objects_scrubbed":11,"log_size":1101,"log_dups_size":0,"ondisk_log_size":1101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:37.578901+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0045066780000000001,"stat_sum":{"num_bytes":34915115,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":12,"num_read_kb":10738,"num_write":12,"num_write_kb":6051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.2d","version":"838'1165","reported_seq":4894,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.043617+0000","last_change":"2024-09-11T03:50:56.129781+0000","last_active":"2024-09-11T03:50:57.043617+0000","last_peered":"2024-09-11T03:50:57.043617+0000","last_clean":"2024-09-11T03:50:57.043617+0000","last_became_active":"2024-09-11T03:49:29.163201+0000","last_became_peered":"2024-09-11T03:49:29.163201+0000","last_unstale":"2024-09-11T03:50:57.043617+0000","last_undegraded":"2024-09-11T03:50:57.043617+0000","last_fullsized":"2024-09-11T03:50:57.043617+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"837'1159","last_scrub_stamp":"2024-09-11T03:50:55.762457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:55.762457+0000","objects_scrubbed":20,"log_size":1165,"log_dups_size":0,"ondisk_log_size":1165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:20.224724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.015170879999999999,"stat_sum":{"num_bytes":41421147,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":62,"num_read_kb":68374,"num_write":24,"num_write_kb":14603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.2c","version":"840'895","reported_seq":4645,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.140190+0000","last_change":"2024-09-11T03:50:58.049690+0000","last_active":"2024-09-11T03:51:01.140190+0000","last_peered":"2024-09-11T03:51:01.140190+0000","last_clean":"2024-09-11T03:51:01.140190+0000","last_became_active":"2024-09-11T03:49:29.154872+0000","last_became_peered":"2024-09-11T03:49:29.154872+0000","last_unstale":"2024-09-11T03:51:01.140190+0000","last_undegraded":"2024-09-11T03:51:01.140190+0000","last_fullsized":"2024-09-11T03:51:01.140190+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"816'878","last_scrub_stamp":"2024-09-11T03:50:37.544645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.544645+0000","objects_scrubbed":12,"log_size":895,"log_dups_size":0,"ondisk_log_size":895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:55.742674+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0028526139999999998,"stat_sum":{"num_bytes":27503872,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":35338,"num_write":40,"num_write_kb":15219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3],"acting":[8,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.2b","version":"568'646","reported_seq":3513,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-09-11T03:50:58.059101+0000","last_change":"2024-09-11T03:50:58.059446+0000","last_active":"2024-09-11T03:50:58.059101+0000","last_peered":"2024-09-11T03:50:58.059101+0000","last_clean":"2024-09-11T03:50:58.059101+0000","last_became_active":"2024-09-11T03:49:29.157735+0000","last_became_peered":"2024-09-11T03:49:29.157735+0000","last_unstale":"2024-09-11T03:50:58.059101+0000","last_undegraded":"2024-09-11T03:50:58.059101+0000","last_fullsized":"2024-09-11T03:50:58.059101+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:50:40.311643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:40.311643+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:59.059954+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002025149,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.2a","version":"727'348","reported_seq":3998,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.185794+0000","last_change":"2024-09-11T03:50:58.052144+0000","last_active":"2024-09-11T03:51:01.185794+0000","last_peered":"2024-09-11T03:51:01.185794+0000","last_clean":"2024-09-11T03:51:01.185794+0000","last_became_active":"2024-09-11T03:49:29.160595+0000","last_became_peered":"2024-09-11T03:49:29.160595+0000","last_unstale":"2024-09-11T03:51:01.185794+0000","last_undegraded":"2024-09-11T03:51:01.185794+0000","last_fullsized":"2024-09-11T03:51:01.185794+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:49:34.219244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.219244+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001305283,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,5],"acting":[7,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.29","version":"838'1272","reported_seq":5105,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.042758+0000","last_change":"2024-09-11T03:50:56.122221+0000","last_active":"2024-09-11T03:50:57.042758+0000","last_peered":"2024-09-11T03:50:57.042758+0000","last_clean":"2024-09-11T03:50:57.042758+0000","last_became_active":"2024-09-11T03:49:29.157194+0000","last_became_peered":"2024-09-11T03:49:29.157194+0000","last_unstale":"2024-09-11T03:50:57.042758+0000","last_undegraded":"2024-09-11T03:50:57.042758+0000","last_fullsized":"2024-09-11T03:50:57.042758+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'1167","last_scrub_stamp":"2024-09-11T03:49:33.506786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.506786+0000","objects_scrubbed":11,"log_size":1272,"log_dups_size":0,"ondisk_log_size":1272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0092953470000000007,"stat_sum":{"num_bytes":35210211,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":38,"num_read_kb":54618,"num_write":27,"num_write_kb":10469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8],"acting":[6,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.28","version":"840'441","reported_seq":4194,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.141475+0000","last_change":"2024-09-11T03:50:58.056306+0000","last_active":"2024-09-11T03:51:01.141475+0000","last_peered":"2024-09-11T03:51:01.141475+0000","last_clean":"2024-09-11T03:51:01.141475+0000","last_became_active":"2024-09-11T03:49:29.154884+0000","last_became_peered":"2024-09-11T03:49:29.154884+0000","last_unstale":"2024-09-11T03:51:01.141475+0000","last_undegraded":"2024-09-11T03:51:01.141475+0000","last_fullsized":"2024-09-11T03:51:01.141475+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'382","last_scrub_stamp":"2024-09-11T03:49:32.335285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.335285+0000","objects_scrubbed":4,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0045355990000000004,"stat_sum":{"num_bytes":20559548,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":14,"num_read_kb":11529,"num_write":21,"num_write_kb":4588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.27","version":"840'1147","reported_seq":5010,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.186325+0000","last_change":"2024-09-11T03:50:59.981244+0000","last_active":"2024-09-11T03:51:01.186325+0000","last_peered":"2024-09-11T03:51:01.186325+0000","last_clean":"2024-09-11T03:51:01.186325+0000","last_became_active":"2024-09-11T03:49:29.159799+0000","last_became_peered":"2024-09-11T03:49:29.159799+0000","last_unstale":"2024-09-11T03:51:01.186325+0000","last_undegraded":"2024-09-11T03:51:01.186325+0000","last_fullsized":"2024-09-11T03:51:01.186325+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"840'1147","last_scrub_stamp":"2024-09-11T03:50:59.981154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:59.981154+0000","objects_scrubbed":8,"log_size":1147,"log_dups_size":0,"ondisk_log_size":1147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:07.916446+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0053605320000000003,"stat_sum":{"num_bytes":21312860,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":17,"num_read_kb":13449,"num_write":17,"num_write_kb":15955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,9],"acting":[7,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.26","version":"269'171","reported_seq":3891,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.186199+0000","last_change":"2024-09-11T03:50:58.055207+0000","last_active":"2024-09-11T03:51:01.186199+0000","last_peered":"2024-09-11T03:51:01.186199+0000","last_clean":"2024-09-11T03:51:01.186199+0000","last_became_active":"2024-09-11T03:49:29.158577+0000","last_became_peered":"2024-09-11T03:49:29.158577+0000","last_unstale":"2024-09-11T03:51:01.186199+0000","last_undegraded":"2024-09-11T03:51:01.186199+0000","last_fullsized":"2024-09-11T03:51:01.186199+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:50:43.910786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:43.910786+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:46.024520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00354639,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,11],"acting":[7,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.25","version":"842'1353","reported_seq":5310,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.140491+0000","last_change":"2024-09-11T03:50:59.564543+0000","last_active":"2024-09-11T03:51:01.140491+0000","last_peered":"2024-09-11T03:51:01.140491+0000","last_clean":"2024-09-11T03:51:01.140491+0000","last_became_active":"2024-09-11T03:49:29.159733+0000","last_became_peered":"2024-09-11T03:49:29.159733+0000","last_unstale":"2024-09-11T03:51:01.140491+0000","last_undegraded":"2024-09-11T03:51:01.140491+0000","last_fullsized":"2024-09-11T03:51:01.140491+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"840'1347","last_scrub_stamp":"2024-09-11T03:50:59.564459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:59.564459+0000","objects_scrubbed":21,"log_size":1353,"log_dups_size":0,"ondisk_log_size":1353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:06.246332+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0089287280000000004,"stat_sum":{"num_bytes":49541870,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":57,"num_read_kb":53770,"num_write":45,"num_write_kb":14401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,6],"acting":[8,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.24","version":"838'674","reported_seq":4435,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.042905+0000","last_change":"2024-09-11T03:50:56.770279+0000","last_active":"2024-09-11T03:50:57.042905+0000","last_peered":"2024-09-11T03:50:57.042905+0000","last_clean":"2024-09-11T03:50:57.042905+0000","last_became_active":"2024-09-11T03:49:29.155740+0000","last_became_peered":"2024-09-11T03:49:29.155740+0000","last_unstale":"2024-09-11T03:50:57.042905+0000","last_undegraded":"2024-09-11T03:50:57.042905+0000","last_fullsized":"2024-09-11T03:50:57.042905+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"838'674","last_scrub_stamp":"2024-09-11T03:50:56.770184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:56.770184+0000","objects_scrubbed":7,"log_size":674,"log_dups_size":0,"ondisk_log_size":674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:25.472934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.007135415,"stat_sum":{"num_bytes":17513828,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":13,"num_write_kb":8583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,7],"acting":[6,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.23","version":"840'493","reported_seq":4230,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.141632+0000","last_change":"2024-09-11T03:50:58.056609+0000","last_active":"2024-09-11T03:51:01.141632+0000","last_peered":"2024-09-11T03:51:01.141632+0000","last_clean":"2024-09-11T03:51:01.141632+0000","last_became_active":"2024-09-11T03:46:32.970100+0000","last_became_peered":"2024-09-11T03:46:32.970100+0000","last_unstale":"2024-09-11T03:51:01.141632+0000","last_undegraded":"2024-09-11T03:51:01.141632+0000","last_fullsized":"2024-09-11T03:51:01.141632+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'474","last_scrub_stamp":"2024-09-11T03:50:19.311509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:19.311509+0000","objects_scrubbed":5,"log_size":493,"log_dups_size":0,"ondisk_log_size":493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:29.668660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0047513340000000003,"stat_sum":{"num_bytes":9399679,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":45,"num_read_kb":50015,"num_write":30,"num_write_kb":11743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.22","version":"531'216","reported_seq":3926,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.042512+0000","last_change":"2024-09-11T03:50:56.115255+0000","last_active":"2024-09-11T03:50:57.042512+0000","last_peered":"2024-09-11T03:50:57.042512+0000","last_clean":"2024-09-11T03:50:57.042512+0000","last_became_active":"2024-09-11T03:46:32.970633+0000","last_became_peered":"2024-09-11T03:46:32.970633+0000","last_unstale":"2024-09-11T03:50:57.042512+0000","last_undegraded":"2024-09-11T03:50:57.042512+0000","last_fullsized":"2024-09-11T03:50:57.042512+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:50:38.617374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:38.617374+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:46.547242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027694939999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.21","version":"838'360","reported_seq":4160,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.045904+0000","last_change":"2024-09-11T03:50:56.116042+0000","last_active":"2024-09-11T03:50:57.045904+0000","last_peered":"2024-09-11T03:50:57.045904+0000","last_clean":"2024-09-11T03:50:57.045904+0000","last_became_active":"2024-09-11T03:46:36.024928+0000","last_became_peered":"2024-09-11T03:46:36.024928+0000","last_unstale":"2024-09-11T03:50:57.045904+0000","last_undegraded":"2024-09-11T03:50:57.045904+0000","last_fullsized":"2024-09-11T03:50:57.045904+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":6,"last_scrub":"816'351","last_scrub_stamp":"2024-09-11T03:50:40.225380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:40.225380+0000","objects_scrubbed":7,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:51.273369+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0039377889999999997,"stat_sum":{"num_bytes":13621377,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":39,"num_read_kb":45171,"num_write":34,"num_write_kb":15311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,8],"acting":[11,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.20","version":"838'304","reported_seq":4196,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.185605+0000","last_change":"2024-09-11T03:50:58.054149+0000","last_active":"2024-09-11T03:51:01.185605+0000","last_peered":"2024-09-11T03:51:01.185605+0000","last_clean":"2024-09-11T03:51:01.185605+0000","last_became_active":"2024-09-11T03:46:32.974375+0000","last_became_peered":"2024-09-11T03:46:32.974375+0000","last_unstale":"2024-09-11T03:51:01.185605+0000","last_undegraded":"2024-09-11T03:51:01.185605+0000","last_fullsized":"2024-09-11T03:51:01.185605+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"818'296","last_scrub_stamp":"2024-09-11T03:50:37.825854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.825854+0000","objects_scrubbed":6,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:55.757308+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003171007,"stat_sum":{"num_bytes":15085668,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":97,"num_read_kb":144174,"num_write":22,"num_write_kb":11057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.1c","version":"806'776","reported_seq":4491,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.387114+0000","last_change":"2024-09-11T03:50:56.123852+0000","last_active":"2024-09-11T03:50:57.387114+0000","last_peered":"2024-09-11T03:50:57.387114+0000","last_clean":"2024-09-11T03:50:57.387114+0000","last_became_active":"2024-09-11T03:50:28.047548+0000","last_became_peered":"2024-09-11T03:50:28.047548+0000","last_unstale":"2024-09-11T03:50:57.387114+0000","last_undegraded":"2024-09-11T03:50:57.387114+0000","last_fullsized":"2024-09-11T03:50:57.387114+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"806'776","last_scrub_stamp":"2024-09-11T03:50:37.794065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.794065+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:57.962762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031268200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":744,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-11T03:51:00.160450+0000","last_change":"2024-09-11T03:50:55.035249+0000","last_active":"2024-09-11T03:51:00.160450+0000","last_peered":"2024-09-11T03:51:00.160450+0000","last_clean":"2024-09-11T03:51:00.160450+0000","last_became_active":"2024-09-11T03:46:15.974786+0000","last_became_peered":"2024-09-11T03:46:15.974786+0000","last_unstale":"2024-09-11T03:51:00.160450+0000","last_undegraded":"2024-09-11T03:51:00.160450+0000","last_fullsized":"2024-09-11T03:51:00.160450+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:55.035202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:55.035202+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:09.185590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.f","version":"557'465","reported_seq":4221,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.140318+0000","last_change":"2024-09-11T03:50:58.055528+0000","last_active":"2024-09-11T03:51:01.140318+0000","last_peered":"2024-09-11T03:51:01.140318+0000","last_clean":"2024-09-11T03:51:01.140318+0000","last_became_active":"2024-09-11T03:50:02.055146+0000","last_became_peered":"2024-09-11T03:50:02.055146+0000","last_unstale":"2024-09-11T03:51:01.140318+0000","last_undegraded":"2024-09-11T03:51:01.140318+0000","last_fullsized":"2024-09-11T03:51:01.140318+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:50:02.368307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.368307+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:05.602776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084588549999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.e","version":"782'1025","reported_seq":4719,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-09-11T03:50:58.058610+0000","last_change":"2024-09-11T03:50:58.059027+0000","last_active":"2024-09-11T03:50:58.058610+0000","last_peered":"2024-09-11T03:50:58.058610+0000","last_clean":"2024-09-11T03:50:58.058610+0000","last_became_active":"2024-09-11T03:50:02.051717+0000","last_became_peered":"2024-09-11T03:50:02.051717+0000","last_unstale":"2024-09-11T03:50:58.058610+0000","last_undegraded":"2024-09-11T03:50:58.058610+0000","last_fullsized":"2024-09-11T03:50:58.058610+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":0,"last_scrub":"782'1025","last_scrub_stamp":"2024-09-11T03:50:04.968160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.968160+0000","objects_scrubbed":0,"log_size":1025,"log_dups_size":0,"ondisk_log_size":1025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:34.328051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023897979999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":1323,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.186058+0000","last_change":"2024-09-11T03:50:58.932445+0000","last_active":"2024-09-11T03:51:01.186058+0000","last_peered":"2024-09-11T03:51:01.186058+0000","last_clean":"2024-09-11T03:51:01.186058+0000","last_became_active":"2024-09-11T03:40:37.852200+0000","last_became_peered":"2024-09-11T03:40:37.852200+0000","last_unstale":"2024-09-11T03:51:01.186058+0000","last_undegraded":"2024-09-11T03:51:01.186058+0000","last_fullsized":"2024-09-11T03:51:01.186058+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:58.932376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:58.932376+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:04.051978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"843'1033","reported_seq":4824,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.168366+0000","last_change":"2024-09-11T03:50:58.061534+0000","last_active":"2024-09-11T03:51:01.168366+0000","last_peered":"2024-09-11T03:51:01.168366+0000","last_clean":"2024-09-11T03:51:01.168366+0000","last_became_active":"2024-09-11T03:49:24.116744+0000","last_became_peered":"2024-09-11T03:49:24.116744+0000","last_unstale":"2024-09-11T03:51:01.168366+0000","last_undegraded":"2024-09-11T03:51:01.168366+0000","last_fullsized":"2024-09-11T03:51:01.168366+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"811'1014","last_scrub_stamp":"2024-09-11T03:50:37.438736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.438736+0000","objects_scrubbed":5,"log_size":1033,"log_dups_size":0,"ondisk_log_size":1033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:39.387697+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0062675070000000003,"stat_sum":{"num_bytes":14992966,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":10,"num_write_kb":4686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":1038,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.186398+0000","last_change":"2024-09-11T03:50:38.843495+0000","last_active":"2024-09-11T03:51:01.186398+0000","last_peered":"2024-09-11T03:51:01.186398+0000","last_clean":"2024-09-11T03:51:01.186398+0000","last_became_active":"2024-09-11T03:44:43.940022+0000","last_became_peered":"2024-09-11T03:44:43.940022+0000","last_unstale":"2024-09-11T03:51:01.186398+0000","last_undegraded":"2024-09-11T03:51:01.186398+0000","last_fullsized":"2024-09-11T03:51:01.186398+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:38.843438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:38.843438+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:45.743283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"838'874","reported_seq":4604,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.046379+0000","last_change":"2024-09-11T03:50:56.119193+0000","last_active":"2024-09-11T03:50:57.046379+0000","last_peered":"2024-09-11T03:50:57.046379+0000","last_clean":"2024-09-11T03:50:57.046379+0000","last_became_active":"2024-09-11T03:49:24.115405+0000","last_became_peered":"2024-09-11T03:49:24.115405+0000","last_unstale":"2024-09-11T03:50:57.046379+0000","last_undegraded":"2024-09-11T03:50:57.046379+0000","last_fullsized":"2024-09-11T03:50:57.046379+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'808","last_scrub_stamp":"2024-09-11T03:49:33.953477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:33.953477+0000","objects_scrubbed":7,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0064066820000000003,"stat_sum":{"num_bytes":23449747,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":14,"num_read_kb":13992,"num_write":14,"num_write_kb":6063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.f","version":"20'92","reported_seq":589,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.044313+0000","last_change":"2024-09-11T03:48:57.363099+0000","last_active":"2024-09-11T03:50:57.044313+0000","last_peered":"2024-09-11T03:50:57.044313+0000","last_clean":"2024-09-11T03:50:57.044313+0000","last_became_active":"2024-09-11T03:46:15.973796+0000","last_became_peered":"2024-09-11T03:46:15.973796+0000","last_unstale":"2024-09-11T03:50:57.044313+0000","last_undegraded":"2024-09-11T03:50:57.044313+0000","last_fullsized":"2024-09-11T03:50:57.044313+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:57.363027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:57.363027+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:10.644654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":1721,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.186216+0000","last_change":"2024-09-11T03:47:28.495899+0000","last_active":"2024-09-11T03:51:01.186216+0000","last_peered":"2024-09-11T03:51:01.186216+0000","last_clean":"2024-09-11T03:51:01.186216+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:51:01.186216+0000","last_undegraded":"2024-09-11T03:51:01.186216+0000","last_fullsized":"2024-09-11T03:51:01.186216+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:28.495839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:28.495839+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:50.881162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"840'558","reported_seq":4304,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.144092+0000","last_change":"2024-09-11T03:50:58.400971+0000","last_active":"2024-09-11T03:51:01.144092+0000","last_peered":"2024-09-11T03:51:01.144092+0000","last_clean":"2024-09-11T03:51:01.144092+0000","last_became_active":"2024-09-11T03:46:27.972329+0000","last_became_peered":"2024-09-11T03:46:27.972329+0000","last_unstale":"2024-09-11T03:51:01.144092+0000","last_undegraded":"2024-09-11T03:51:01.144092+0000","last_fullsized":"2024-09-11T03:51:01.144092+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"840'558","last_scrub_stamp":"2024-09-11T03:50:58.400895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:58.400895+0000","objects_scrubbed":11,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:12.315191+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0068950189999999996,"stat_sum":{"num_bytes":21969224,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":31,"num_read_kb":23188,"num_write":49,"num_write_kb":13780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":1328,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.141732+0000","last_change":"2024-09-11T03:50:45.053323+0000","last_active":"2024-09-11T03:51:01.141732+0000","last_peered":"2024-09-11T03:51:01.141732+0000","last_clean":"2024-09-11T03:51:01.141732+0000","last_became_active":"2024-09-11T03:46:15.968960+0000","last_became_peered":"2024-09-11T03:46:15.968960+0000","last_unstale":"2024-09-11T03:51:01.141732+0000","last_undegraded":"2024-09-11T03:51:01.141732+0000","last_fullsized":"2024-09-11T03:51:01.141732+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:25.344170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:25.344170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:14.363606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"563'192","reported_seq":2791,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.386728+0000","last_change":"2024-09-11T03:50:56.125108+0000","last_active":"2024-09-11T03:50:57.386728+0000","last_peered":"2024-09-11T03:50:57.386728+0000","last_clean":"2024-09-11T03:50:57.386728+0000","last_became_active":"2024-09-11T03:50:31.044399+0000","last_became_peered":"2024-09-11T03:50:31.044399+0000","last_unstale":"2024-09-11T03:50:57.386728+0000","last_undegraded":"2024-09-11T03:50:57.386728+0000","last_fullsized":"2024-09-11T03:50:57.386728+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:50:36.771360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.771360+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:52.347489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041009030000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":1291,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-11T03:51:00.160284+0000","last_change":"2024-09-11T03:50:41.975709+0000","last_active":"2024-09-11T03:51:00.160284+0000","last_peered":"2024-09-11T03:51:00.160284+0000","last_clean":"2024-09-11T03:51:00.160284+0000","last_became_active":"2024-09-11T03:44:25.934580+0000","last_became_peered":"2024-09-11T03:44:25.934580+0000","last_unstale":"2024-09-11T03:51:00.160284+0000","last_undegraded":"2024-09-11T03:51:00.160284+0000","last_fullsized":"2024-09-11T03:51:00.160284+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:41.975660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:41.975660+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:00.591013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"567'235","reported_seq":3951,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-11T03:51:00.160569+0000","last_change":"2024-09-11T03:50:58.049927+0000","last_active":"2024-09-11T03:51:00.160569+0000","last_peered":"2024-09-11T03:51:00.160569+0000","last_clean":"2024-09-11T03:51:00.160569+0000","last_became_active":"2024-09-11T03:50:28.039300+0000","last_became_peered":"2024-09-11T03:50:28.039300+0000","last_unstale":"2024-09-11T03:51:00.160569+0000","last_undegraded":"2024-09-11T03:51:00.160569+0000","last_fullsized":"2024-09-11T03:51:00.160569+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:50:36.844578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.844578+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:38.625814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087759700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":890,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.387078+0000","last_change":"2024-09-11T03:50:18.699606+0000","last_active":"2024-09-11T03:50:57.387078+0000","last_peered":"2024-09-11T03:50:57.387078+0000","last_clean":"2024-09-11T03:50:57.387078+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:50:57.387078+0000","last_undegraded":"2024-09-11T03:50:57.387078+0000","last_fullsized":"2024-09-11T03:50:57.387078+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:18.699532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.699532+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:27.930456+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"843'319","reported_seq":4136,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.193756+0000","last_change":"2024-09-11T03:50:58.056194+0000","last_active":"2024-09-11T03:51:01.193756+0000","last_peered":"2024-09-11T03:51:01.193756+0000","last_clean":"2024-09-11T03:51:01.193756+0000","last_became_active":"2024-09-11T03:46:27.971649+0000","last_became_peered":"2024-09-11T03:46:27.971649+0000","last_unstale":"2024-09-11T03:51:01.193756+0000","last_undegraded":"2024-09-11T03:51:01.193756+0000","last_fullsized":"2024-09-11T03:51:01.193756+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"799'299","last_scrub_stamp":"2024-09-11T03:50:20.621227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:20.621227+0000","objects_scrubbed":3,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:36.215235+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0023013339999999999,"stat_sum":{"num_bytes":15883770,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":13037,"num_write":39,"num_write_kb":19829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":581,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.043315+0000","last_change":"2024-09-11T03:49:37.637065+0000","last_active":"2024-09-11T03:50:57.043315+0000","last_peered":"2024-09-11T03:50:57.043315+0000","last_clean":"2024-09-11T03:50:57.043315+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:50:57.043315+0000","last_undegraded":"2024-09-11T03:50:57.043315+0000","last_fullsized":"2024-09-11T03:50:57.043315+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:37.637018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.637018+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:05.221389+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"840'683","reported_seq":4483,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.141943+0000","last_change":"2024-09-11T03:51:00.279675+0000","last_active":"2024-09-11T03:51:01.141943+0000","last_peered":"2024-09-11T03:51:01.141943+0000","last_clean":"2024-09-11T03:51:01.141943+0000","last_became_active":"2024-09-11T03:49:24.114330+0000","last_became_peered":"2024-09-11T03:49:24.114330+0000","last_unstale":"2024-09-11T03:51:01.141943+0000","last_undegraded":"2024-09-11T03:51:01.141943+0000","last_fullsized":"2024-09-11T03:51:01.141943+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"840'683","last_scrub_stamp":"2024-09-11T03:51:00.279620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:00.279620+0000","objects_scrubbed":10,"log_size":683,"log_dups_size":0,"ondisk_log_size":683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:07.972242+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0042853600000000002,"stat_sum":{"num_bytes":21086157,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":13,"num_read_kb":8619,"num_write":10,"num_write_kb":5852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":901,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.386989+0000","last_change":"2024-09-11T03:50:40.811000+0000","last_active":"2024-09-11T03:50:57.386989+0000","last_peered":"2024-09-11T03:50:57.386989+0000","last_clean":"2024-09-11T03:50:57.386989+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:50:57.386989+0000","last_undegraded":"2024-09-11T03:50:57.386989+0000","last_fullsized":"2024-09-11T03:50:57.386989+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:40.810941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:40.810941+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:46.731652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"840'1207","reported_seq":5013,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.140702+0000","last_change":"2024-09-11T03:50:58.055660+0000","last_active":"2024-09-11T03:51:01.140702+0000","last_peered":"2024-09-11T03:51:01.140702+0000","last_clean":"2024-09-11T03:51:01.140702+0000","last_became_active":"2024-09-11T03:49:24.115095+0000","last_became_peered":"2024-09-11T03:49:24.115095+0000","last_unstale":"2024-09-11T03:51:01.140702+0000","last_undegraded":"2024-09-11T03:51:01.140702+0000","last_fullsized":"2024-09-11T03:51:01.140702+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"819'1187","last_scrub_stamp":"2024-09-11T03:50:40.500149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:40.500149+0000","objects_scrubbed":11,"log_size":1207,"log_dups_size":0,"ondisk_log_size":1207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:45.008006+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0077775259999999999,"stat_sum":{"num_bytes":21761563,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":8,"num_read_kb":3,"num_write":21,"num_write_kb":8788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":892,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.386846+0000","last_change":"2024-09-11T03:49:36.654074+0000","last_active":"2024-09-11T03:50:57.386846+0000","last_peered":"2024-09-11T03:50:57.386846+0000","last_clean":"2024-09-11T03:50:57.386846+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:50:57.386846+0000","last_undegraded":"2024-09-11T03:50:57.386846+0000","last_fullsized":"2024-09-11T03:50:57.386846+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:36.654025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.654025+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:06.216209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":3896,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.142114+0000","last_change":"2024-09-11T03:50:58.060864+0000","last_active":"2024-09-11T03:51:01.142114+0000","last_peered":"2024-09-11T03:51:01.142114+0000","last_clean":"2024-09-11T03:51:01.142114+0000","last_became_active":"2024-09-11T03:49:24.113518+0000","last_became_peered":"2024-09-11T03:49:24.113518+0000","last_unstale":"2024-09-11T03:51:01.142114+0000","last_undegraded":"2024-09-11T03:51:01.142114+0000","last_fullsized":"2024-09-11T03:51:01.142114+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:50:37.453595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.453595+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:48.220636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00767785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":1492,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.046199+0000","last_change":"2024-09-11T03:49:21.909643+0000","last_active":"2024-09-11T03:50:57.046199+0000","last_peered":"2024-09-11T03:50:57.046199+0000","last_clean":"2024-09-11T03:50:57.046199+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:50:57.046199+0000","last_undegraded":"2024-09-11T03:50:57.046199+0000","last_fullsized":"2024-09-11T03:50:57.046199+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:21.909594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:21.909594+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:26.077940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"838'1231","reported_seq":5119,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.043061+0000","last_change":"2024-09-11T03:50:56.126461+0000","last_active":"2024-09-11T03:50:57.043061+0000","last_peered":"2024-09-11T03:50:57.043061+0000","last_clean":"2024-09-11T03:50:57.043061+0000","last_became_active":"2024-09-11T03:49:24.121333+0000","last_became_peered":"2024-09-11T03:49:24.121333+0000","last_unstale":"2024-09-11T03:50:57.043061+0000","last_undegraded":"2024-09-11T03:50:57.043061+0000","last_fullsized":"2024-09-11T03:50:57.043061+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"817'1201","last_scrub_stamp":"2024-09-11T03:50:40.600544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:40.600544+0000","objects_scrubbed":13,"log_size":1231,"log_dups_size":0,"ondisk_log_size":1231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:44.090357+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.012570356,"stat_sum":{"num_bytes":41240081,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":87,"num_read_kb":75866,"num_write":48,"num_write_kb":30794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":1294,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.046046+0000","last_change":"2024-09-11T03:50:18.176206+0000","last_active":"2024-09-11T03:50:57.046046+0000","last_peered":"2024-09-11T03:50:57.046046+0000","last_clean":"2024-09-11T03:50:57.046046+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:50:57.046046+0000","last_undegraded":"2024-09-11T03:50:57.046046+0000","last_fullsized":"2024-09-11T03:50:57.046046+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:18.176156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.176156+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:43.809898+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"744'374","reported_seq":4156,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.186114+0000","last_change":"2024-09-11T03:51:00.973238+0000","last_active":"2024-09-11T03:51:01.186114+0000","last_peered":"2024-09-11T03:51:01.186114+0000","last_clean":"2024-09-11T03:51:01.186114+0000","last_became_active":"2024-09-11T03:49:24.115150+0000","last_became_peered":"2024-09-11T03:49:24.115150+0000","last_unstale":"2024-09-11T03:51:01.186114+0000","last_undegraded":"2024-09-11T03:51:01.186114+0000","last_fullsized":"2024-09-11T03:51:01.186114+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'374","last_scrub_stamp":"2024-09-11T03:51:00.973170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:00.973170+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:05.898752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0047127549999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":1648,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-09-11T03:50:58.056671+0000","last_change":"2024-09-11T03:50:18.934008+0000","last_active":"2024-09-11T03:50:58.056671+0000","last_peered":"2024-09-11T03:50:58.056671+0000","last_clean":"2024-09-11T03:50:58.056671+0000","last_became_active":"2024-09-11T03:46:15.973271+0000","last_became_peered":"2024-09-11T03:46:15.973271+0000","last_unstale":"2024-09-11T03:50:58.056671+0000","last_undegraded":"2024-09-11T03:50:58.056671+0000","last_fullsized":"2024-09-11T03:50:58.056671+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:18.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.933952+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:39.183063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"840'1354","reported_seq":5201,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.141496+0000","last_change":"2024-09-11T03:50:59.237422+0000","last_active":"2024-09-11T03:51:01.141496+0000","last_peered":"2024-09-11T03:51:01.141496+0000","last_clean":"2024-09-11T03:51:01.141496+0000","last_became_active":"2024-09-11T03:49:24.114865+0000","last_became_peered":"2024-09-11T03:49:24.114865+0000","last_unstale":"2024-09-11T03:51:01.141496+0000","last_undegraded":"2024-09-11T03:51:01.141496+0000","last_fullsized":"2024-09-11T03:51:01.141496+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"840'1354","last_scrub_stamp":"2024-09-11T03:50:59.237345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:59.237345+0000","objects_scrubbed":17,"log_size":1354,"log_dups_size":0,"ondisk_log_size":1354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:02.874416+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0093108159999999995,"stat_sum":{"num_bytes":36788550,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":37,"num_read_kb":35560,"num_write":34,"num_write_kb":17205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":1717,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.185625+0000","last_change":"2024-09-11T03:49:37.206392+0000","last_active":"2024-09-11T03:51:01.185625+0000","last_peered":"2024-09-11T03:51:01.185625+0000","last_clean":"2024-09-11T03:51:01.185625+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:51:01.185625+0000","last_undegraded":"2024-09-11T03:51:01.185625+0000","last_fullsized":"2024-09-11T03:51:01.185625+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:37.206321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.206321+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"838'379","reported_seq":4134,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.045948+0000","last_change":"2024-09-11T03:50:56.117876+0000","last_active":"2024-09-11T03:50:57.045948+0000","last_peered":"2024-09-11T03:50:57.045948+0000","last_clean":"2024-09-11T03:50:57.045948+0000","last_became_active":"2024-09-11T03:49:24.114551+0000","last_became_peered":"2024-09-11T03:49:24.114551+0000","last_unstale":"2024-09-11T03:50:57.045948+0000","last_undegraded":"2024-09-11T03:50:57.045948+0000","last_fullsized":"2024-09-11T03:50:57.045948+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:49:32.911631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:49:32.911631+0000","objects_scrubbed":4,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0055336370000000001,"stat_sum":{"num_bytes":5429057,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":10,"num_read_kb":12240,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":1376,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.143628+0000","last_change":"2024-09-11T03:49:34.621588+0000","last_active":"2024-09-11T03:51:01.143628+0000","last_peered":"2024-09-11T03:51:01.143628+0000","last_clean":"2024-09-11T03:51:01.143628+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:51:01.143628+0000","last_undegraded":"2024-09-11T03:51:01.143628+0000","last_fullsized":"2024-09-11T03:51:01.143628+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:34.621527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:34.621527+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:53.465869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"568'646","reported_seq":4423,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.140167+0000","last_change":"2024-09-11T03:50:58.047562+0000","last_active":"2024-09-11T03:51:01.140167+0000","last_peered":"2024-09-11T03:51:01.140167+0000","last_clean":"2024-09-11T03:51:01.140167+0000","last_became_active":"2024-09-11T03:50:31.042191+0000","last_became_peered":"2024-09-11T03:50:31.042191+0000","last_unstale":"2024-09-11T03:51:01.140167+0000","last_undegraded":"2024-09-11T03:51:01.140167+0000","last_fullsized":"2024-09-11T03:51:01.140167+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:50:54.564800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:54.564800+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:08.327633+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00098698899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,0],"acting":[8,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":1325,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.143389+0000","last_change":"2024-09-11T03:49:00.710015+0000","last_active":"2024-09-11T03:51:01.143389+0000","last_peered":"2024-09-11T03:51:01.143389+0000","last_clean":"2024-09-11T03:51:01.143389+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:51:01.143389+0000","last_undegraded":"2024-09-11T03:51:01.143389+0000","last_fullsized":"2024-09-11T03:51:01.143389+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:00.709966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.709966+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:32.521638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":3806,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-09-11T03:50:58.058051+0000","last_change":"2024-09-11T03:50:58.058051+0000","last_active":"2024-09-11T03:50:58.058051+0000","last_peered":"2024-09-11T03:50:58.058051+0000","last_clean":"2024-09-11T03:50:58.058051+0000","last_became_active":"2024-09-11T03:50:02.046906+0000","last_became_peered":"2024-09-11T03:50:02.046906+0000","last_unstale":"2024-09-11T03:50:58.058051+0000","last_undegraded":"2024-09-11T03:50:58.058051+0000","last_fullsized":"2024-09-11T03:50:58.058051+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:50:02.987179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.987179+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:12.254603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020437839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":1316,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-11T03:51:00.160483+0000","last_change":"2024-09-11T03:51:00.033938+0000","last_active":"2024-09-11T03:51:00.160483+0000","last_peered":"2024-09-11T03:51:00.160483+0000","last_clean":"2024-09-11T03:51:00.160483+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:51:00.160483+0000","last_undegraded":"2024-09-11T03:51:00.160483+0000","last_fullsized":"2024-09-11T03:51:00.160483+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:00.033889+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:05.022524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"838'575","reported_seq":4313,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.387202+0000","last_change":"2024-09-11T03:50:56.124531+0000","last_active":"2024-09-11T03:50:57.387202+0000","last_peered":"2024-09-11T03:50:57.387202+0000","last_clean":"2024-09-11T03:50:57.387202+0000","last_became_active":"2024-09-11T03:50:02.053513+0000","last_became_peered":"2024-09-11T03:50:02.053513+0000","last_unstale":"2024-09-11T03:50:57.387202+0000","last_undegraded":"2024-09-11T03:50:57.387202+0000","last_fullsized":"2024-09-11T03:50:57.387202+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"784'551","last_scrub_stamp":"2024-09-11T03:50:02.647690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.647690+0000","objects_scrubbed":4,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:04.224079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0038818839999999999,"stat_sum":{"num_bytes":11120044,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":1626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":1306,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.143369+0000","last_change":"2024-09-11T03:47:23.339646+0000","last_active":"2024-09-11T03:51:01.143369+0000","last_peered":"2024-09-11T03:51:01.143369+0000","last_clean":"2024-09-11T03:51:01.143369+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:51:01.143369+0000","last_undegraded":"2024-09-11T03:51:01.143369+0000","last_fullsized":"2024-09-11T03:51:01.143369+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:23.339595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:23.339595+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:51.441797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":2872,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.043221+0000","last_change":"2024-09-11T03:50:56.124287+0000","last_active":"2024-09-11T03:50:57.043221+0000","last_peered":"2024-09-11T03:50:57.043221+0000","last_clean":"2024-09-11T03:50:57.043221+0000","last_became_active":"2024-09-11T03:50:02.048777+0000","last_became_peered":"2024-09-11T03:50:02.048777+0000","last_unstale":"2024-09-11T03:50:57.043221+0000","last_undegraded":"2024-09-11T03:50:57.043221+0000","last_fullsized":"2024-09-11T03:50:57.043221+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:50:02.696833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.696833+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:25.427999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096688799999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.13","version":"20'92","reported_seq":1411,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.141626+0000","last_change":"2024-09-11T03:47:18.217600+0000","last_active":"2024-09-11T03:51:01.141626+0000","last_peered":"2024-09-11T03:51:01.141626+0000","last_clean":"2024-09-11T03:51:01.141626+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:51:01.141626+0000","last_undegraded":"2024-09-11T03:51:01.141626+0000","last_fullsized":"2024-09-11T03:51:01.141626+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:47:18.217555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:47:18.217555+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:51.265971+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":2833,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.140044+0000","last_change":"2024-09-11T03:50:58.046975+0000","last_active":"2024-09-11T03:51:01.140044+0000","last_peered":"2024-09-11T03:51:01.140044+0000","last_clean":"2024-09-11T03:51:01.140044+0000","last_became_active":"2024-09-11T03:50:02.057196+0000","last_became_peered":"2024-09-11T03:50:02.057196+0000","last_unstale":"2024-09-11T03:51:01.140044+0000","last_undegraded":"2024-09-11T03:51:01.140044+0000","last_fullsized":"2024-09-11T03:51:01.140044+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:50:03.345563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:03.345563+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:21.028556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065680299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":892,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.186019+0000","last_change":"2024-09-11T03:50:20.772575+0000","last_active":"2024-09-11T03:51:01.186019+0000","last_peered":"2024-09-11T03:51:01.186019+0000","last_clean":"2024-09-11T03:51:01.186019+0000","last_became_active":"2024-09-11T03:46:36.026459+0000","last_became_peered":"2024-09-11T03:46:36.026459+0000","last_unstale":"2024-09-11T03:51:01.186019+0000","last_undegraded":"2024-09-11T03:51:01.186019+0000","last_fullsized":"2024-09-11T03:51:01.186019+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:20.772524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:20.772524+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:26.554678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":3368,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.044027+0000","last_change":"2024-09-11T03:50:56.123899+0000","last_active":"2024-09-11T03:50:57.044027+0000","last_peered":"2024-09-11T03:50:57.044027+0000","last_clean":"2024-09-11T03:50:57.044027+0000","last_became_active":"2024-09-11T03:50:02.052278+0000","last_became_peered":"2024-09-11T03:50:02.052278+0000","last_unstale":"2024-09-11T03:50:57.044027+0000","last_undegraded":"2024-09-11T03:50:57.044027+0000","last_fullsized":"2024-09-11T03:50:57.044027+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:50:03.690374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:03.690374+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:07.913666+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0094961060000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":1287,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.186088+0000","last_change":"2024-09-11T03:50:19.741484+0000","last_active":"2024-09-11T03:51:01.186088+0000","last_peered":"2024-09-11T03:51:01.186088+0000","last_clean":"2024-09-11T03:51:01.186088+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:51:01.186088+0000","last_undegraded":"2024-09-11T03:51:01.186088+0000","last_fullsized":"2024-09-11T03:51:01.186088+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:19.741428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:19.741428+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:36.778923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"782'561","reported_seq":4357,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.143795+0000","last_change":"2024-09-11T03:50:58.054787+0000","last_active":"2024-09-11T03:51:01.143795+0000","last_peered":"2024-09-11T03:51:01.143795+0000","last_clean":"2024-09-11T03:51:01.143795+0000","last_became_active":"2024-09-11T03:50:02.054667+0000","last_became_peered":"2024-09-11T03:50:02.054667+0000","last_unstale":"2024-09-11T03:51:01.143795+0000","last_undegraded":"2024-09-11T03:51:01.143795+0000","last_fullsized":"2024-09-11T03:51:01.143795+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"782'561","last_scrub_stamp":"2024-09-11T03:50:02.696931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:02.696931+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:06.365650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010715169999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":758,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.046489+0000","last_change":"2024-09-11T03:50:37.230454+0000","last_active":"2024-09-11T03:50:57.046489+0000","last_peered":"2024-09-11T03:50:57.046489+0000","last_clean":"2024-09-11T03:50:57.046489+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:50:57.046489+0000","last_undegraded":"2024-09-11T03:50:57.046489+0000","last_fullsized":"2024-09-11T03:50:57.046489+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:37.230388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.230388+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:04.405605+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":4025,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-09-11T03:50:58.055507+0000","last_change":"2024-09-11T03:50:58.055890+0000","last_active":"2024-09-11T03:50:58.055507+0000","last_peered":"2024-09-11T03:50:58.055507+0000","last_clean":"2024-09-11T03:50:58.055507+0000","last_became_active":"2024-09-11T03:50:02.051850+0000","last_became_peered":"2024-09-11T03:50:02.051850+0000","last_unstale":"2024-09-11T03:50:58.055507+0000","last_undegraded":"2024-09-11T03:50:58.055507+0000","last_fullsized":"2024-09-11T03:50:58.055507+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:50:04.004006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.004006+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:28.369798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065095799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":1307,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.140242+0000","last_change":"2024-09-11T03:50:43.547521+0000","last_active":"2024-09-11T03:51:01.140242+0000","last_peered":"2024-09-11T03:51:01.140242+0000","last_clean":"2024-09-11T03:51:01.140242+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:51:01.140242+0000","last_undegraded":"2024-09-11T03:51:01.140242+0000","last_fullsized":"2024-09-11T03:51:01.140242+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:43.547475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:43.547475+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:57.034349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"778'353","reported_seq":4002,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.042900+0000","last_change":"2024-09-11T03:50:56.117244+0000","last_active":"2024-09-11T03:50:57.042900+0000","last_peered":"2024-09-11T03:50:57.042900+0000","last_clean":"2024-09-11T03:50:57.042900+0000","last_became_active":"2024-09-11T03:50:02.054211+0000","last_became_peered":"2024-09-11T03:50:02.054211+0000","last_unstale":"2024-09-11T03:50:57.042900+0000","last_undegraded":"2024-09-11T03:50:57.042900+0000","last_fullsized":"2024-09-11T03:50:57.042900+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"778'353","last_scrub_stamp":"2024-09-11T03:50:04.646331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:04.646331+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:13.145197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045636499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.18","version":"20'92","reported_seq":1418,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.143372+0000","last_change":"2024-09-11T03:49:56.630710+0000","last_active":"2024-09-11T03:51:01.143372+0000","last_peered":"2024-09-11T03:51:01.143372+0000","last_clean":"2024-09-11T03:51:01.143372+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:51:01.143372+0000","last_undegraded":"2024-09-11T03:51:01.143372+0000","last_fullsized":"2024-09-11T03:51:01.143372+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:56.630651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:56.630651+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:18.735201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"838'890","reported_seq":4796,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.046004+0000","last_change":"2024-09-11T03:50:56.124146+0000","last_active":"2024-09-11T03:50:57.046004+0000","last_peered":"2024-09-11T03:50:57.046004+0000","last_clean":"2024-09-11T03:50:57.046004+0000","last_became_active":"2024-09-11T03:50:31.047148+0000","last_became_peered":"2024-09-11T03:50:31.047148+0000","last_unstale":"2024-09-11T03:50:57.046004+0000","last_undegraded":"2024-09-11T03:50:57.046004+0000","last_fullsized":"2024-09-11T03:50:57.046004+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":5,"last_scrub":"816'883","last_scrub_stamp":"2024-09-11T03:50:35.180324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.180324+0000","objects_scrubbed":5,"log_size":890,"log_dups_size":0,"ondisk_log_size":890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:36.699051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.011053189,"stat_sum":{"num_bytes":6370837,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.19","version":"20'92","reported_seq":896,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.386952+0000","last_change":"2024-09-11T03:48:56.129575+0000","last_active":"2024-09-11T03:50:57.386952+0000","last_peered":"2024-09-11T03:50:57.386952+0000","last_clean":"2024-09-11T03:50:57.386952+0000","last_became_active":"2024-09-11T03:46:15.972841+0000","last_became_peered":"2024-09-11T03:46:15.972841+0000","last_unstale":"2024-09-11T03:50:57.386952+0000","last_undegraded":"2024-09-11T03:50:57.386952+0000","last_fullsized":"2024-09-11T03:50:57.386952+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:56.129516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:56.129516+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:26.739103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"557'273","reported_seq":3958,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.042619+0000","last_change":"2024-09-11T03:50:56.118019+0000","last_active":"2024-09-11T03:50:57.042619+0000","last_peered":"2024-09-11T03:50:57.042619+0000","last_clean":"2024-09-11T03:50:57.042619+0000","last_became_active":"2024-09-11T03:50:28.038587+0000","last_became_peered":"2024-09-11T03:50:28.038587+0000","last_unstale":"2024-09-11T03:50:57.042619+0000","last_undegraded":"2024-09-11T03:50:57.042619+0000","last_fullsized":"2024-09-11T03:50:57.042619+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:50:37.565876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.565876+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:45.365473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052818850000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,2],"acting":[6,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.1a","version":"20'92","reported_seq":449,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.185745+0000","last_change":"2024-09-11T03:50:01.843384+0000","last_active":"2024-09-11T03:51:01.185745+0000","last_peered":"2024-09-11T03:51:01.185745+0000","last_clean":"2024-09-11T03:51:01.185745+0000","last_became_active":"2024-09-11T03:49:30.162694+0000","last_became_peered":"2024-09-11T03:49:30.162694+0000","last_unstale":"2024-09-11T03:51:01.185745+0000","last_undegraded":"2024-09-11T03:51:01.185745+0000","last_fullsized":"2024-09-11T03:51:01.185745+0000","mapping_epoch":751,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":752,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:01.843302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:01.843302+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:12.888176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.19","version":"838'593","reported_seq":4410,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.043109+0000","last_change":"2024-09-11T03:50:56.123265+0000","last_active":"2024-09-11T03:50:57.043109+0000","last_peered":"2024-09-11T03:50:57.043109+0000","last_clean":"2024-09-11T03:50:57.043109+0000","last_became_active":"2024-09-11T03:50:28.039694+0000","last_became_peered":"2024-09-11T03:50:28.039694+0000","last_unstale":"2024-09-11T03:50:57.043109+0000","last_undegraded":"2024-09-11T03:50:57.043109+0000","last_fullsized":"2024-09-11T03:50:57.043109+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"816'587","last_scrub_stamp":"2024-09-11T03:50:36.522195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.522195+0000","objects_scrubbed":7,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:57.975776+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0092621549999999993,"stat_sum":{"num_bytes":15469991,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.1b","version":"20'92","reported_seq":1276,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.139971+0000","last_change":"2024-09-11T03:48:31.033896+0000","last_active":"2024-09-11T03:51:01.139971+0000","last_peered":"2024-09-11T03:51:01.139971+0000","last_clean":"2024-09-11T03:51:01.139971+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:51:01.139971+0000","last_undegraded":"2024-09-11T03:51:01.139971+0000","last_fullsized":"2024-09-11T03:51:01.139971+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:48:31.033837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:48:31.033837+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:46.629650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"838'582","reported_seq":4422,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-09-11T03:50:58.058148+0000","last_change":"2024-09-11T03:50:58.058524+0000","last_active":"2024-09-11T03:50:58.058148+0000","last_peered":"2024-09-11T03:50:58.058148+0000","last_clean":"2024-09-11T03:50:58.058148+0000","last_became_active":"2024-09-11T03:50:28.044833+0000","last_became_peered":"2024-09-11T03:50:28.044833+0000","last_unstale":"2024-09-11T03:50:58.058148+0000","last_undegraded":"2024-09-11T03:50:58.058148+0000","last_fullsized":"2024-09-11T03:50:58.058148+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"816'566","last_scrub_stamp":"2024-09-11T03:50:35.256345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.256345+0000","objects_scrubbed":11,"log_size":582,"log_dups_size":0,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:58.525090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00199737,"stat_sum":{"num_bytes":21890642,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":10,"num_read_kb":12514,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"2.1f","version":"840'789","reported_seq":4506,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-11T03:51:00.162458+0000","last_change":"2024-09-11T03:50:58.054943+0000","last_active":"2024-09-11T03:51:00.162458+0000","last_peered":"2024-09-11T03:51:00.162458+0000","last_clean":"2024-09-11T03:51:00.162458+0000","last_became_active":"2024-09-11T03:50:28.041110+0000","last_became_peered":"2024-09-11T03:50:28.041110+0000","last_unstale":"2024-09-11T03:51:00.162458+0000","last_undegraded":"2024-09-11T03:51:00.162458+0000","last_fullsized":"2024-09-11T03:51:00.162458+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"818'769","last_scrub_stamp":"2024-09-11T03:50:37.881096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.881096+0000","objects_scrubbed":13,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:49.193360+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0058668959999999999,"stat_sum":{"num_bytes":33111837,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":10,"num_read_kb":5621,"num_write":5,"num_write_kb":1872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.1c","version":"20'92","reported_seq":1283,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.140303+0000","last_change":"2024-09-11T03:49:40.156559+0000","last_active":"2024-09-11T03:51:01.140303+0000","last_peered":"2024-09-11T03:51:01.140303+0000","last_clean":"2024-09-11T03:51:01.140303+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:51:01.140303+0000","last_undegraded":"2024-09-11T03:51:01.140303+0000","last_fullsized":"2024-09-11T03:51:01.140303+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:40.156495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:40.156495+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:04.047069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1e","version":"840'1076","reported_seq":4799,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-11T03:51:00.160669+0000","last_change":"2024-09-11T03:50:58.053215+0000","last_active":"2024-09-11T03:51:00.160669+0000","last_peered":"2024-09-11T03:51:00.160669+0000","last_clean":"2024-09-11T03:51:00.160669+0000","last_became_active":"2024-09-11T03:50:28.044451+0000","last_became_peered":"2024-09-11T03:50:28.044451+0000","last_unstale":"2024-09-11T03:51:00.160669+0000","last_undegraded":"2024-09-11T03:51:00.160669+0000","last_fullsized":"2024-09-11T03:51:00.160669+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"806'1060","last_scrub_stamp":"2024-09-11T03:50:35.880642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.880642+0000","objects_scrubbed":2,"log_size":1076,"log_dups_size":0,"ondisk_log_size":1076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:39.738207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042531579999999999,"stat_sum":{"num_bytes":8257533,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":11270,"num_write":10,"num_write_kb":3430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"16f","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.1d","version":"20'92","reported_seq":1273,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-09-11T03:51:01.141937+0000","last_change":"2024-09-11T03:50:56.316951+0000","last_active":"2024-09-11T03:51:01.141937+0000","last_peered":"2024-09-11T03:51:01.141937+0000","last_clean":"2024-09-11T03:51:01.141937+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:51:01.141937+0000","last_undegraded":"2024-09-11T03:51:01.141937+0000","last_fullsized":"2024-09-11T03:51:01.141937+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:56.316896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:56.316896+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:07.998967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"567'752","reported_seq":4451,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.386901+0000","last_change":"2024-09-11T03:50:56.121117+0000","last_active":"2024-09-11T03:50:57.386901+0000","last_peered":"2024-09-11T03:50:57.386901+0000","last_clean":"2024-09-11T03:50:57.386901+0000","last_became_active":"2024-09-11T03:50:28.045687+0000","last_became_peered":"2024-09-11T03:50:28.045687+0000","last_unstale":"2024-09-11T03:50:57.386901+0000","last_undegraded":"2024-09-11T03:50:57.386901+0000","last_fullsized":"2024-09-11T03:50:57.386901+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:50:38.812209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:38.812209+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:51:46.145228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010633909999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,11],"acting":[10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"155","length":"2"},{"start":"168","length":"1"},{"start":"17b","length":"1"}]},{"pgid":"1.1e","version":"20'92","reported_seq":1283,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-09-11T03:50:57.046579+0000","last_change":"2024-09-11T03:49:36.949199+0000","last_active":"2024-09-11T03:50:57.046579+0000","last_peered":"2024-09-11T03:50:57.046579+0000","last_clean":"2024-09-11T03:50:57.046579+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:50:57.046579+0000","last_undegraded":"2024-09-11T03:50:57.046579+0000","last_fullsized":"2024-09-11T03:50:57.046579+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:36.949149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.949149+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":66,"stat_sum":{"num_bytes":699274953,"num_objects":304,"num_object_clones":254,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":14,"num_read":774,"num_read_kb":777364,"num_write":569,"num_write_kb":250236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1120165888,"data_stored":1590043119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38464,"ondisk_log_size":38464,"up":198,"acting":198,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285733,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":118492,"kb_used_data":87300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93626660,"statfs":{"total":95995035648,"available":95873699840,"internally_reserved":0,"allocated":89395200,"data_stored":128525796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514665,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96384,"kb_used_data":69360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93648768,"statfs":{"total":95995035648,"available":95896338432,"internally_reserved":0,"allocated":71024640,"data_stored":86969728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743599,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":125380,"kb_used_data":67812,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93619772,"statfs":{"total":95995035648,"available":95866646528,"internally_reserved":0,"allocated":69439488,"data_stored":78554670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037943,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":196444,"kb_used_data":138880,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93548708,"statfs":{"total":95995035648,"available":95793876992,"internally_reserved":0,"allocated":142213120,"data_stored":209990846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299578,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":224748,"kb_used_data":175504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93520404,"statfs":{"total":95995035648,"available":95764893696,"internally_reserved":0,"allocated":179716096,"data_stored":241907627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528510,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":166076,"kb_used_data":134880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93579076,"statfs":{"total":95995035648,"available":95824973824,"internally_reserved":0,"allocated":138117120,"data_stored":189846232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757445,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":138796,"kb_used_data":89552,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93606356,"statfs":{"total":95995035648,"available":95852908544,"internally_reserved":0,"allocated":91701248,"data_stored":124755411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986378,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":117680,"kb_used_data":64276,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93627472,"statfs":{"total":95995035648,"available":95874531328,"internally_reserved":0,"allocated":65818624,"data_stored":73593895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149902,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":199060,"kb_used_data":149816,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93546092,"statfs":{"total":95995035648,"available":95791198208,"internally_reserved":0,"allocated":153411584,"data_stored":226084912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411540,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108624,"kb_used_data":55228,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93636528,"statfs":{"total":95995035648,"available":95883804672,"internally_reserved":0,"allocated":56553472,"data_stored":66215049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575066,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":151788,"kb_used_data":94232,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93593364,"statfs":{"total":95995035648,"available":95839604736,"internally_reserved":0,"allocated":96493568,"data_stored":129338568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738591,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":145000,"kb_used_data":87436,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93600152,"statfs":{"total":95995035648,"available":95846555648,"internally_reserved":0,"allocated":89534464,"data_stored":116823714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":78999552,"data_stored":109828018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86274048,"data_stored":122662118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":47968256,"data_stored":61159610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":142901248,"data_stored":219105884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":55623680,"data_stored":66934213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":81276928,"data_stored":117877713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":127631360,"data_stored":182883968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":169304064,"data_stored":235038363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":132005888,"data_stored":203322830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":59256832,"data_stored":71903322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":60866560,"data_stored":80335144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":78057472,"data_stored":118991936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:51:03.957 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-11T03:51:03.957 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 76 2024-09-11T03:51:03.958 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:03.979 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:03.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:03.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:04.080 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:04.102 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:04.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:04.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:04.148 INFO:tasks.rados.rados.2.smithi175.stdout:3500: done (3 left) 2024-09-11T03:51:04.148 INFO:tasks.rados.rados.2.smithi175.stdout:3502: done (2 left) 2024-09-11T03:51:04.148 INFO:tasks.rados.rados.2.smithi175.stdout:3503: done (1 left) 2024-09-11T03:51:04.149 INFO:tasks.rados.rados.2.smithi175.stdout:3506: done (0 left) 2024-09-11T03:51:04.149 INFO:tasks.rados.rados.2.smithi175.stdout:3507: copy_from oid 38 from oid 5 current snap is 389 2024-09-11T03:51:04.149 INFO:tasks.rados.rados.2.smithi175.stdout:3508: rollback oid 12 current snap is 389 2024-09-11T03:51:04.149 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 12 to 379 2024-09-11T03:51:04.149 INFO:tasks.rados.rados.2.smithi175.stdout:3509: rollback oid 27 current snap is 389 2024-09-11T03:51:04.149 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 27 to 357 2024-09-11T03:51:04.149 INFO:tasks.rados.rados.2.smithi175.stdout:3510: rollback oid 28 current snap is 389 2024-09-11T03:51:04.149 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 28 to 322 2024-09-11T03:51:04.149 INFO:tasks.rados.rados.2.smithi175.stdout:3511: read oid 39 snap 349 2024-09-11T03:51:04.149 INFO:tasks.rados.rados.2.smithi175.stdout:3511: expect (ObjNum 1030 snap 338 seq_num 1030) 2024-09-11T03:51:04.149 INFO:tasks.rados.rados.2.smithi175.stdout:3512: snap_create 2024-09-11T03:51:04.152 INFO:tasks.rados.rados.2.smithi175.stdout:3507: finishing copy_from to smithi17557870-38 2024-09-11T03:51:04.152 INFO:tasks.rados.rados.2.smithi175.stdout:3507: got expected ENOENT (src dne) 2024-09-11T03:51:04.152 INFO:tasks.rados.rados.2.smithi175.stdout:3507: finishing copy_from racing read to smithi17557870-38 2024-09-11T03:51:04.152 INFO:tasks.rados.rados.2.smithi175.stdout:3510: finishing rollback tid 0 to smithi17557870-28 2024-09-11T03:51:04.153 INFO:tasks.rados.rados.2.smithi175.stdout:3510: finishing rollback tid 1 to smithi17557870-28 2024-09-11T03:51:04.153 INFO:tasks.rados.rados.2.smithi175.stdout:3510: finishing rollback tid 2 to smithi17557870-28 2024-09-11T03:51:04.153 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 1373 (ObjNum 942 snap 297 seq_num 942) dirty exists 2024-09-11T03:51:04.153 INFO:tasks.rados.rados.2.smithi175.stdout:3508: finishing rollback tid 0 to smithi17557870-12 2024-09-11T03:51:04.153 INFO:tasks.rados.rados.2.smithi175.stdout:3508: finishing rollback tid 1 to smithi17557870-12 2024-09-11T03:51:04.154 INFO:tasks.rados.rados.2.smithi175.stdout:3509: finishing rollback tid 1 to smithi17557870-27 2024-09-11T03:51:04.154 INFO:tasks.rados.rados.2.smithi175.stdout:3508: finishing rollback tid 2 to smithi17557870-12 2024-09-11T03:51:04.154 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 1367 (ObjNum 1134 snap 367 seq_num 1134) dirty exists 2024-09-11T03:51:04.154 INFO:tasks.rados.rados.2.smithi175.stdout:3509: finishing rollback tid 2 to smithi17557870-27 2024-09-11T03:51:04.154 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 1238 (ObjNum 1090 snap 353 seq_num 1090) dirty exists 2024-09-11T03:51:04.155 INFO:tasks.rados.rados.2.smithi175.stdout:3511: expect (ObjNum 1030 snap 338 seq_num 1030) 2024-09-11T03:51:04.204 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:04.225 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:04.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:04.229 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:04.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:04.356 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:04.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:04.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:04 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:51:04.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:04 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:51:04.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:04 smithi084 ceph-mon[27254]: from='client.17262 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:51:04.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:04 smithi084 ceph-mon[27254]: pgmap v1422: 97 pgs: 97 active+clean; 675 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.8 MiB/s wr, 19 op/s 2024-09-11T03:51:04.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:04 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:51:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:04 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:51:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:04 smithi121 ceph-mon[30729]: from='client.17262 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:51:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:04 smithi121 ceph-mon[30729]: pgmap v1422: 97 pgs: 97 active+clean; 675 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.8 MiB/s wr, 19 op/s 2024-09-11T03:51:04.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:04 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:51:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:04 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:51:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:04 smithi175 ceph-mon[30894]: from='client.17262 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:51:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:04 smithi175 ceph-mon[30894]: pgmap v1422: 97 pgs: 97 active+clean; 675 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.8 MiB/s wr, 19 op/s 2024-09-11T03:51:04.480 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:04.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:04.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:04.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:04.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:04.605 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:04.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:04.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:04.732 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:04.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:04.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:04.834 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:04.856 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:04.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:04.857 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:04.917 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T03:51:04.946 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:04.957 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:04.969 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:04.979 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:04.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:04.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:05.081 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:05.102 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:05.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:05.149 INFO:tasks.rados.rados.2.smithi175.stdout:3507: done (5 left) 2024-09-11T03:51:05.149 INFO:tasks.rados.rados.2.smithi175.stdout:3508: done (4 left) 2024-09-11T03:51:05.150 INFO:tasks.rados.rados.2.smithi175.stdout:3509: done (3 left) 2024-09-11T03:51:05.150 INFO:tasks.rados.rados.2.smithi175.stdout:3510: done (2 left) 2024-09-11T03:51:05.150 INFO:tasks.rados.rados.2.smithi175.stdout:3511: done (1 left) 2024-09-11T03:51:05.150 INFO:tasks.rados.rados.2.smithi175.stdout:3512: done (0 left) 2024-09-11T03:51:05.150 INFO:tasks.rados.rados.2.smithi175.stdout:3513: delete oid 22 current snap is 390 2024-09-11T03:51:05.156 INFO:tasks.rados.rados.2.smithi175.stdout:3513: done (0 left) 2024-09-11T03:51:05.156 INFO:tasks.rados.rados.2.smithi175.stdout:3514: read oid 29 snap 389 2024-09-11T03:51:05.156 INFO:tasks.rados.rados.2.smithi175.stdout:3514: expect (ObjNum 1179 snap 385 seq_num 1179) 2024-09-11T03:51:05.156 INFO:tasks.rados.rados.2.smithi175.stdout:3515: write oid 20 current snap is 390 2024-09-11T03:51:05.156 INFO:tasks.rados.rados.2.smithi175.stdout:3515: seq_num 1193 ranges {450751=723125,1945018=105733} 2024-09-11T03:51:05.162 INFO:tasks.rados.rados.2.smithi175.stdout:3515: writing smithi17557870-20 from 450751 to 1173876 tid 1 2024-09-11T03:51:05.167 INFO:tasks.rados.rados.2.smithi175.stdout:3515: writing smithi17557870-20 from 1945018 to 2050751 tid 2 2024-09-11T03:51:05.167 INFO:tasks.rados.rados.2.smithi175.stdout:3516: write oid 34 current snap is 390 2024-09-11T03:51:05.167 INFO:tasks.rados.rados.2.smithi175.stdout:3516: seq_num 1194 ranges {467558=514727,1719853=347705} 2024-09-11T03:51:05.172 INFO:tasks.rados.rados.2.smithi175.stdout:3516: writing smithi17557870-34 from 467558 to 982285 tid 1 2024-09-11T03:51:05.178 INFO:tasks.rados.rados.2.smithi175.stdout:3516: writing smithi17557870-34 from 1719853 to 2067558 tid 2 2024-09-11T03:51:05.179 INFO:tasks.rados.rados.2.smithi175.stdout:3517: read oid 5 snap -1 2024-09-11T03:51:05.179 INFO:tasks.rados.rados.2.smithi175.stdout:3517: expect deleted 2024-09-11T03:51:05.179 INFO:tasks.rados.rados.2.smithi175.stdout:3518: rollback oid 18 current snap is 390 2024-09-11T03:51:05.179 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 18 to 361 2024-09-11T03:51:05.179 INFO:tasks.rados.rados.2.smithi175.stdout:3514: expect (ObjNum 1179 snap 385 seq_num 1179) 2024-09-11T03:51:05.203 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:05.226 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:05.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:05.302 INFO:tasks.rados.rados.2.smithi175.stdout:3515: finishing write tid 1 to smithi17557870-20 2024-09-11T03:51:05.302 INFO:tasks.rados.rados.2.smithi175.stdout:3515: finishing write tid 2 to smithi17557870-20 2024-09-11T03:51:05.302 INFO:tasks.rados.rados.2.smithi175.stdout:3515: finishing write tid 3 to smithi17557870-20 2024-09-11T03:51:05.302 INFO:tasks.rados.rados.2.smithi175.stdout:3515: finishing write tid 4 to smithi17557870-20 2024-09-11T03:51:05.302 INFO:tasks.rados.rados.2.smithi175.stdout:3515: oid 20 updating version 0 to 1036 2024-09-11T03:51:05.302 INFO:tasks.rados.rados.2.smithi175.stdout:3515: oid 20 updating version 1036 to 1037 2024-09-11T03:51:05.302 INFO:tasks.rados.rados.2.smithi175.stdout:3515: oid 20 version 1037 is already newer than 1035 2024-09-11T03:51:05.302 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 1037 (ObjNum 1193 snap 390 seq_num 1193) dirty exists 2024-09-11T03:51:05.303 INFO:tasks.rados.rados.2.smithi175.stdout:3515: left oid 20 (ObjNum 1193 snap 390 seq_num 1193) 2024-09-11T03:51:05.303 INFO:tasks.rados.rados.2.smithi175.stdout:3516: finishing write tid 1 to smithi17557870-34 2024-09-11T03:51:05.303 INFO:tasks.rados.rados.2.smithi175.stdout:3514: done (4 left) 2024-09-11T03:51:05.303 INFO:tasks.rados.rados.2.smithi175.stdout:3515: done (3 left) 2024-09-11T03:51:05.303 INFO:tasks.rados.rados.2.smithi175.stdout:3519: snap_create 2024-09-11T03:51:05.303 INFO:tasks.rados.rados.2.smithi175.stdout:3518: finishing rollback tid 0 to smithi17557870-18 2024-09-11T03:51:05.303 INFO:tasks.rados.rados.2.smithi175.stdout:3516: finishing write tid 2 to smithi17557870-34 2024-09-11T03:51:05.303 INFO:tasks.rados.rados.2.smithi175.stdout:3516: finishing write tid 3 to smithi17557870-34 2024-09-11T03:51:05.303 INFO:tasks.rados.rados.2.smithi175.stdout:3516: finishing write tid 4 to smithi17557870-34 2024-09-11T03:51:05.303 INFO:tasks.rados.rados.2.smithi175.stdout:3516: oid 34 updating version 0 to 1054 2024-09-11T03:51:05.303 INFO:tasks.rados.rados.2.smithi175.stdout:3516: oid 34 version 1054 is already newer than 1053 2024-09-11T03:51:05.303 INFO:tasks.rados.rados.2.smithi175.stdout:3516: oid 34 updating version 1054 to 1055 2024-09-11T03:51:05.303 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 1055 (ObjNum 1194 snap 390 seq_num 1194) dirty exists 2024-09-11T03:51:05.303 INFO:tasks.rados.rados.2.smithi175.stdout:3516: left oid 34 (ObjNum 1194 snap 390 seq_num 1194) 2024-09-11T03:51:05.304 INFO:tasks.rados.rados.2.smithi175.stdout:3518: finishing rollback tid 1 to smithi17557870-18 2024-09-11T03:51:05.304 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 951 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T03:51:05.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:05.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:05.349 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:05.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:05 smithi121 ceph-mon[30729]: osdmap e846: 12 total, 12 up, 12 in 2024-09-11T03:51:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:05 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:51:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:05 smithi121 ceph-mon[30729]: 2.a scrub ok 2024-09-11T03:51:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:05 smithi121 ceph-mon[30729]: osdmap e847: 12 total, 12 up, 12 in 2024-09-11T03:51:05.449 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:05.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:05.472 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:05.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:05.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:05 smithi084 ceph-mon[27254]: osdmap e846: 12 total, 12 up, 12 in 2024-09-11T03:51:05.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:05 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:51:05.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:05 smithi084 ceph-mon[27254]: 2.a scrub ok 2024-09-11T03:51:05.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:05 smithi084 ceph-mon[27254]: osdmap e847: 12 total, 12 up, 12 in 2024-09-11T03:51:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:05 smithi175 ceph-mon[30894]: osdmap e846: 12 total, 12 up, 12 in 2024-09-11T03:51:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:05 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:51:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:05 smithi175 ceph-mon[30894]: 2.a scrub ok 2024-09-11T03:51:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:05 smithi175 ceph-mon[30894]: osdmap e847: 12 total, 12 up, 12 in 2024-09-11T03:51:05.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:05.599 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:05.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:05.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:05.701 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:05.723 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:05.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:05.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:05.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:05.850 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:05.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:05.929 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:05.952 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:05.974 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:05.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:05.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:06.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:06.101 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:06.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:06.166 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 2 pg_num to 76 2024-09-11T03:51:06.167 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:51:06.170 INFO:tasks.rados.rados.2.smithi175.stdout:3516: done (3 left) 2024-09-11T03:51:06.170 INFO:tasks.rados.rados.2.smithi175.stdout:3517: done (2 left) 2024-09-11T03:51:06.170 INFO:tasks.rados.rados.2.smithi175.stdout:3518: done (1 left) 2024-09-11T03:51:06.170 INFO:tasks.rados.rados.2.smithi175.stdout:3519: done (0 left) 2024-09-11T03:51:06.170 INFO:tasks.rados.rados.2.smithi175.stdout:3520: write (excl) oid 49 current snap is 391 2024-09-11T03:51:06.170 INFO:tasks.rados.rados.2.smithi175.stdout:3520: seq_num 1195 ranges {484365=589976,1661982=422383} 2024-09-11T03:51:06.176 INFO:tasks.rados.rados.2.smithi175.stdout:3520: writing smithi17557870-49 from 484365 to 1074341 tid 1 2024-09-11T03:51:06.181 INFO:tasks.rados.rados.2.smithi175.stdout:3520: writing smithi17557870-49 from 1661982 to 2084365 tid 2 2024-09-11T03:51:06.181 INFO:tasks.rados.rados.2.smithi175.stdout:3521: write oid 2 current snap is 391 2024-09-11T03:51:06.181 INFO:tasks.rados.rados.2.smithi175.stdout:3520: finishing write tid 1 to smithi17557870-49 2024-09-11T03:51:06.182 INFO:tasks.rados.rados.2.smithi175.stdout:3521: seq_num 1196 ranges {501172=665225,1720464=380708} 2024-09-11T03:51:06.187 INFO:tasks.rados.rados.2.smithi175.stdout:3521: writing smithi17557870-2 from 501172 to 1166397 tid 1 2024-09-11T03:51:06.193 INFO:tasks.rados.rados.2.smithi175.stdout:3521: writing smithi17557870-2 from 1720464 to 2101172 tid 2 2024-09-11T03:51:06.193 INFO:tasks.rados.rados.2.smithi175.stdout:3522: write (excl) oid 46 current snap is 391 2024-09-11T03:51:06.193 INFO:tasks.rados.rados.2.smithi175.stdout:3520: finishing write tid 2 to smithi17557870-49 2024-09-11T03:51:06.193 INFO:tasks.rados.rados.2.smithi175.stdout:3520: finishing write tid 3 to smithi17557870-49 2024-09-11T03:51:06.193 INFO:tasks.rados.rados.2.smithi175.stdout:3520: finishing write tid 4 to smithi17557870-49 2024-09-11T03:51:06.193 INFO:tasks.rados.rados.2.smithi175.stdout:3520: oid 49 updating version 0 to 1006 2024-09-11T03:51:06.194 INFO:tasks.rados.rados.2.smithi175.stdout:3520: oid 49 version 1006 is already newer than 1005 2024-09-11T03:51:06.194 INFO:tasks.rados.rados.2.smithi175.stdout:3520: oid 49 version 1006 is already newer than 1004 2024-09-11T03:51:06.194 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 1006 (ObjNum 1195 snap 391 seq_num 1195) dirty exists 2024-09-11T03:51:06.194 INFO:tasks.rados.rados.2.smithi175.stdout:3520: left oid 49 (ObjNum 1195 snap 391 seq_num 1195) 2024-09-11T03:51:06.194 INFO:tasks.rados.rados.2.smithi175.stdout:3522: seq_num 1197 ranges {517979=740474,1778946=339033} 2024-09-11T03:51:06.199 INFO:tasks.rados.rados.2.smithi175.stdout:3522: writing smithi17557870-46 from 517979 to 1258453 tid 1 2024-09-11T03:51:06.203 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:06.205 INFO:tasks.rados.rados.2.smithi175.stdout:3522: writing smithi17557870-46 from 1778946 to 2117979 tid 2 2024-09-11T03:51:06.205 INFO:tasks.rados.rados.2.smithi175.stdout:3520: done (2 left) 2024-09-11T03:51:06.205 INFO:tasks.rados.rados.2.smithi175.stdout:3523: rollback oid 47 current snap is 391 2024-09-11T03:51:06.205 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 47 to 378 2024-09-11T03:51:06.205 INFO:tasks.rados.rados.2.smithi175.stdout:3521: finishing write tid 1 to smithi17557870-2 2024-09-11T03:51:06.205 INFO:tasks.rados.rados.2.smithi175.stdout:3521: finishing write tid 2 to smithi17557870-2 2024-09-11T03:51:06.205 INFO:tasks.rados.rados.2.smithi175.stdout:3521: finishing write tid 3 to smithi17557870-2 2024-09-11T03:51:06.205 INFO:tasks.rados.rados.2.smithi175.stdout:3521: finishing write tid 4 to smithi17557870-2 2024-09-11T03:51:06.206 INFO:tasks.rados.rados.2.smithi175.stdout:3521: oid 2 updating version 0 to 1260 2024-09-11T03:51:06.206 INFO:tasks.rados.rados.2.smithi175.stdout:3521: oid 2 version 1260 is already newer than 1259 2024-09-11T03:51:06.206 INFO:tasks.rados.rados.2.smithi175.stdout:3521: oid 2 version 1260 is already newer than 1258 2024-09-11T03:51:06.206 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 1260 (ObjNum 1196 snap 391 seq_num 1196) dirty exists 2024-09-11T03:51:06.206 INFO:tasks.rados.rados.2.smithi175.stdout:3521: left oid 2 (ObjNum 1196 snap 391 seq_num 1196) 2024-09-11T03:51:06.206 INFO:tasks.rados.rados.2.smithi175.stdout:3522: finishing write tid 1 to smithi17557870-46 2024-09-11T03:51:06.206 INFO:tasks.rados.rados.2.smithi175.stdout:3521: done (2 left) 2024-09-11T03:51:06.206 INFO:tasks.rados.rados.2.smithi175.stdout:3524: snap_create 2024-09-11T03:51:06.207 INFO:tasks.rados.rados.2.smithi175.stdout:3522: finishing write tid 2 to smithi17557870-46 2024-09-11T03:51:06.208 INFO:tasks.rados.rados.2.smithi175.stdout:3522: finishing write tid 3 to smithi17557870-46 2024-09-11T03:51:06.208 INFO:tasks.rados.rados.2.smithi175.stdout:3522: finishing write tid 4 to smithi17557870-46 2024-09-11T03:51:06.208 INFO:tasks.rados.rados.2.smithi175.stdout:3522: oid 46 updating version 0 to 967 2024-09-11T03:51:06.208 INFO:tasks.rados.rados.2.smithi175.stdout:3522: oid 46 updating version 967 to 969 2024-09-11T03:51:06.208 INFO:tasks.rados.rados.2.smithi175.stdout:3522: oid 46 version 969 is already newer than 968 2024-09-11T03:51:06.208 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 969 (ObjNum 1197 snap 391 seq_num 1197) dirty exists 2024-09-11T03:51:06.208 INFO:tasks.rados.rados.2.smithi175.stdout:3522: left oid 46 (ObjNum 1197 snap 391 seq_num 1197) 2024-09-11T03:51:06.210 INFO:tasks.rados.rados.2.smithi175.stdout:3523: finishing rollback tid 0 to smithi17557870-47 2024-09-11T03:51:06.210 INFO:tasks.rados.rados.2.smithi175.stdout:3523: finishing rollback tid 1 to smithi17557870-47 2024-09-11T03:51:06.210 INFO:tasks.rados.rados.2.smithi175.stdout:3523: finishing rollback tid 2 to smithi17557870-47 2024-09-11T03:51:06.210 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 1191 (ObjNum 1132 snap 367 seq_num 1132) dirty exists 2024-09-11T03:51:06.228 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:06.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:06.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:06.330 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:06.353 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:06.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:06.354 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:06 smithi084 ceph-mon[27254]: 2.12 scrub starts 2024-09-11T03:51:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:06 smithi084 ceph-mon[27254]: 2.12 scrub ok 2024-09-11T03:51:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:06 smithi084 ceph-mon[27254]: pgmap v1425: 97 pgs: 97 active+clean; 677 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.2 MiB/s wr, 21 op/s 2024-09-11T03:51:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:06 smithi084 ceph-mon[27254]: 1.9 scrub starts 2024-09-11T03:51:06.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:06 smithi084 ceph-mon[27254]: 1.9 scrub ok 2024-09-11T03:51:06.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:06 smithi084 ceph-mon[27254]: 1.1c scrub starts 2024-09-11T03:51:06.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:06 smithi084 ceph-mon[27254]: 1.1c scrub ok 2024-09-11T03:51:06.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:06 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/348958519' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:51:06.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:06 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/536484538' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "76"} : dispatch 2024-09-11T03:51:06.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:06 smithi121 ceph-mon[30729]: 2.12 scrub starts 2024-09-11T03:51:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:06 smithi121 ceph-mon[30729]: 2.12 scrub ok 2024-09-11T03:51:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:06 smithi121 ceph-mon[30729]: pgmap v1425: 97 pgs: 97 active+clean; 677 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.2 MiB/s wr, 21 op/s 2024-09-11T03:51:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:06 smithi121 ceph-mon[30729]: 1.9 scrub starts 2024-09-11T03:51:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:06 smithi121 ceph-mon[30729]: 1.9 scrub ok 2024-09-11T03:51:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:06 smithi121 ceph-mon[30729]: 1.1c scrub starts 2024-09-11T03:51:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:06 smithi121 ceph-mon[30729]: 1.1c scrub ok 2024-09-11T03:51:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:06 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/348958519' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:51:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:06 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/536484538' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "76"} : dispatch 2024-09-11T03:51:06.454 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:06.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:06 smithi175 ceph-mon[30894]: 2.12 scrub starts 2024-09-11T03:51:06.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:06 smithi175 ceph-mon[30894]: 2.12 scrub ok 2024-09-11T03:51:06.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:06 smithi175 ceph-mon[30894]: pgmap v1425: 97 pgs: 97 active+clean; 677 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.2 MiB/s wr, 21 op/s 2024-09-11T03:51:06.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:06 smithi175 ceph-mon[30894]: 1.9 scrub starts 2024-09-11T03:51:06.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:06 smithi175 ceph-mon[30894]: 1.9 scrub ok 2024-09-11T03:51:06.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:06 smithi175 ceph-mon[30894]: 1.1c scrub starts 2024-09-11T03:51:06.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:06 smithi175 ceph-mon[30894]: 1.1c scrub ok 2024-09-11T03:51:06.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:06 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/348958519' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:51:06.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:06 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/536484538' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "76"} : dispatch 2024-09-11T03:51:06.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:06.481 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:06.482 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:06.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:51:06.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:06.608 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:06.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:06.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:06.676 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:51:06.676 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:51:06.710 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:06.733 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:06.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:06.734 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:06.835 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:06.858 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:06.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:06.859 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:06.960 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:06.982 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:06.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:06.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:07.045 INFO:tasks.rados.rados.2.smithi175.stdout:3522: done (2 left) 2024-09-11T03:51:07.045 INFO:tasks.rados.rados.2.smithi175.stdout:3523: done (1 left) 2024-09-11T03:51:07.045 INFO:tasks.rados.rados.2.smithi175.stdout:3524: done (0 left) 2024-09-11T03:51:07.045 INFO:tasks.rados.rados.2.smithi175.stdout:3525: snap_create 2024-09-11T03:51:07.084 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:07.107 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:07.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:07.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:07.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:07.232 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:07.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:07.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:07.334 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:07.358 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:07.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:07.359 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:07.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:07 smithi084 ceph-mon[27254]: 2.28 scrub starts 2024-09-11T03:51:07.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:07 smithi084 ceph-mon[27254]: 2.28 scrub ok 2024-09-11T03:51:07.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:07 smithi084 ceph-mon[27254]: 2.2a scrub starts 2024-09-11T03:51:07.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:07 smithi084 ceph-mon[27254]: 2.2a scrub ok 2024-09-11T03:51:07.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:07 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:07.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/348958519' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:51:07.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/536484538' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "76"}]': finished 2024-09-11T03:51:07.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:07 smithi084 ceph-mon[27254]: osdmap e848: 12 total, 12 up, 12 in 2024-09-11T03:51:07.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:07 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:51:07.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:07 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:51:07.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:07 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "76"} : dispatch 2024-09-11T03:51:07.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:07 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "76"} : dispatch 2024-09-11T03:51:07.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:07 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "76"}]': finished 2024-09-11T03:51:07.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:07 smithi084 ceph-mon[27254]: osdmap e849: 12 total, 12 up, 12 in 2024-09-11T03:51:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:07 smithi121 ceph-mon[30729]: 2.28 scrub starts 2024-09-11T03:51:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:07 smithi121 ceph-mon[30729]: 2.28 scrub ok 2024-09-11T03:51:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:07 smithi121 ceph-mon[30729]: 2.2a scrub starts 2024-09-11T03:51:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:07 smithi121 ceph-mon[30729]: 2.2a scrub ok 2024-09-11T03:51:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:07 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/348958519' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:51:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/536484538' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "76"}]': finished 2024-09-11T03:51:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:07 smithi121 ceph-mon[30729]: osdmap e848: 12 total, 12 up, 12 in 2024-09-11T03:51:07.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:07 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:51:07.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:07 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:51:07.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:07 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "76"} : dispatch 2024-09-11T03:51:07.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:07 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "76"} : dispatch 2024-09-11T03:51:07.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:07 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "76"}]': finished 2024-09-11T03:51:07.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:07 smithi121 ceph-mon[30729]: osdmap e849: 12 total, 12 up, 12 in 2024-09-11T03:51:07.424 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T03:51:07.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:07 smithi175 ceph-mon[30894]: 2.28 scrub starts 2024-09-11T03:51:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:07 smithi175 ceph-mon[30894]: 2.28 scrub ok 2024-09-11T03:51:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:07 smithi175 ceph-mon[30894]: 2.2a scrub starts 2024-09-11T03:51:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:07 smithi175 ceph-mon[30894]: 2.2a scrub ok 2024-09-11T03:51:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:07 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/348958519' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:51:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/536484538' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "76"}]': finished 2024-09-11T03:51:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:07 smithi175 ceph-mon[30894]: osdmap e848: 12 total, 12 up, 12 in 2024-09-11T03:51:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:07 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:51:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:07 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:51:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:07 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "76"} : dispatch 2024-09-11T03:51:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:07 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "76"} : dispatch 2024-09-11T03:51:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:07 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "76"}]': finished 2024-09-11T03:51:07.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:07 smithi175 ceph-mon[30894]: osdmap e849: 12 total, 12 up, 12 in 2024-09-11T03:51:07.484 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:07.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:07.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:07.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:07.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:07.609 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:07.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:07.710 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:07.736 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:07.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:07.737 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:07.838 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:07.862 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:07.863 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:07.963 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:07.988 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:07.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:07.989 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:08.049 INFO:tasks.rados.rados.2.smithi175.stdout:3525: done (0 left) 2024-09-11T03:51:08.049 INFO:tasks.rados.rados.2.smithi175.stdout:3526: write oid 27 current snap is 393 2024-09-11T03:51:08.049 INFO:tasks.rados.rados.2.smithi175.stdout:3526: seq_num 1198 ranges {534786=415723,1437428=697358} 2024-09-11T03:51:08.055 INFO:tasks.rados.rados.2.smithi175.stdout:3526: writing smithi17557870-27 from 534786 to 950509 tid 1 2024-09-11T03:51:08.062 INFO:tasks.rados.rados.2.smithi175.stdout:3526: writing smithi17557870-27 from 1437428 to 2134786 tid 2 2024-09-11T03:51:08.062 INFO:tasks.rados.rados.2.smithi175.stdout:3527: write (excl) oid 6 current snap is 393 2024-09-11T03:51:08.063 INFO:tasks.rados.rados.2.smithi175.stdout:3526: finishing write tid 1 to smithi17557870-27 2024-09-11T03:51:08.063 INFO:tasks.rados.rados.2.smithi175.stdout:3527: seq_num 1199 ranges {551593=490972,1779557=372036} 2024-09-11T03:51:08.068 INFO:tasks.rados.rados.2.smithi175.stdout:3527: writing smithi17557870-6 from 551593 to 1042565 tid 1 2024-09-11T03:51:08.075 INFO:tasks.rados.rados.2.smithi175.stdout:3527: writing smithi17557870-6 from 1779557 to 2151593 tid 2 2024-09-11T03:51:08.075 INFO:tasks.rados.rados.2.smithi175.stdout:3528: snap_remove snap 375 2024-09-11T03:51:08.089 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:08.112 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:08.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:08.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:08.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:08.236 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:08.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:08.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:08.338 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:08.363 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:08.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:08.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:08 smithi084 ceph-mon[27254]: pgmap v1428: 97 pgs: 97 active+clean; 688 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-11T03:51:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:08 smithi084 ceph-mon[27254]: osdmap e850: 12 total, 12 up, 12 in 2024-09-11T03:51:08.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:08 smithi121 ceph-mon[30729]: pgmap v1428: 97 pgs: 97 active+clean; 688 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-11T03:51:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:08 smithi121 ceph-mon[30729]: osdmap e850: 12 total, 12 up, 12 in 2024-09-11T03:51:08.465 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:08 smithi175 ceph-mon[30894]: pgmap v1428: 97 pgs: 97 active+clean; 688 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-11T03:51:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:08 smithi175 ceph-mon[30894]: osdmap e850: 12 total, 12 up, 12 in 2024-09-11T03:51:08.491 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:08.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:08.492 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:08.592 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:08.618 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:08.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:08.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:08.682 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:08.711 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:08.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:08.741 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:08.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:08.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:08.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:08.865 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:08.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:08.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:08.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:08.988 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:08.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:08.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:09.052 INFO:tasks.rados.rados.2.smithi175.stdout:3528: done (2 left) 2024-09-11T03:51:09.053 INFO:tasks.rados.rados.2.smithi175.stdout:3529: snap_create 2024-09-11T03:51:09.053 INFO:tasks.rados.rados.2.smithi175.stdout:3526: finishing write tid 2 to smithi17557870-27 2024-09-11T03:51:09.053 INFO:tasks.rados.rados.2.smithi175.stdout:3526: finishing write tid 3 to smithi17557870-27 2024-09-11T03:51:09.053 INFO:tasks.rados.rados.2.smithi175.stdout:3526: finishing write tid 4 to smithi17557870-27 2024-09-11T03:51:09.053 INFO:tasks.rados.rados.2.smithi175.stdout:3526: oid 27 updating version 0 to 1240 2024-09-11T03:51:09.053 INFO:tasks.rados.rados.2.smithi175.stdout:3526: oid 27 updating version 1240 to 1241 2024-09-11T03:51:09.053 INFO:tasks.rados.rados.2.smithi175.stdout:3526: oid 27 version 1241 is already newer than 1239 2024-09-11T03:51:09.053 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 1241 (ObjNum 1198 snap 393 seq_num 1198) dirty exists 2024-09-11T03:51:09.053 INFO:tasks.rados.rados.2.smithi175.stdout:3526: left oid 27 (ObjNum 1198 snap 393 seq_num 1198) 2024-09-11T03:51:09.053 INFO:tasks.rados.rados.2.smithi175.stdout:3527: finishing write tid 1 to smithi17557870-6 2024-09-11T03:51:09.053 INFO:tasks.rados.rados.2.smithi175.stdout:3527: finishing write tid 2 to smithi17557870-6 2024-09-11T03:51:09.053 INFO:tasks.rados.rados.2.smithi175.stdout:3527: finishing write tid 3 to smithi17557870-6 2024-09-11T03:51:09.053 INFO:tasks.rados.rados.2.smithi175.stdout:3527: finishing write tid 4 to smithi17557870-6 2024-09-11T03:51:09.054 INFO:tasks.rados.rados.2.smithi175.stdout:3527: oid 6 updating version 0 to 987 2024-09-11T03:51:09.054 INFO:tasks.rados.rados.2.smithi175.stdout:3527: oid 6 updating version 987 to 988 2024-09-11T03:51:09.054 INFO:tasks.rados.rados.2.smithi175.stdout:3527: oid 6 updating version 988 to 989 2024-09-11T03:51:09.054 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 989 (ObjNum 1199 snap 393 seq_num 1199) dirty exists 2024-09-11T03:51:09.054 INFO:tasks.rados.rados.2.smithi175.stdout:3527: left oid 6 (ObjNum 1199 snap 393 seq_num 1199) 2024-09-11T03:51:09.090 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:09.116 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:09.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:09.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:09.204 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:09 smithi084 ceph-mon[27254]: 2.6 scrub starts 2024-09-11T03:51:09.204 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:09 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:51:09.204 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:09 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:51:09.204 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:09 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:51:09.204 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:09 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "76"} : dispatch 2024-09-11T03:51:09.204 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:09 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "76"} : dispatch 2024-09-11T03:51:09.204 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:09 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:51:09.204 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:09 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "76"}]': finished 2024-09-11T03:51:09.205 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:09 smithi084 ceph-mon[27254]: osdmap e851: 12 total, 12 up, 12 in 2024-09-11T03:51:09.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:09.238 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:09.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:09.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:09.340 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:09.361 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:09.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:09.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:09.437 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:51:09.437 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:51:09.452 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:51:09.452 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:51:09.463 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:09.468 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:51:09.468 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:51:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:09 smithi175 ceph-mon[30894]: 2.6 scrub starts 2024-09-11T03:51:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:09 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:51:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:09 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:51:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:09 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:51:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:09 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "76"} : dispatch 2024-09-11T03:51:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:09 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "76"} : dispatch 2024-09-11T03:51:09.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:09 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:51:09.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:09 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "76"}]': finished 2024-09-11T03:51:09.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:09 smithi175 ceph-mon[30894]: osdmap e851: 12 total, 12 up, 12 in 2024-09-11T03:51:09.483 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:09.484 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:51:09.484 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:51:09.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:09.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:09.500 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:51:09.500 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:51:09.517 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:51:09.517 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:51:09.527 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:09 smithi121 ceph-mon[30729]: 2.6 scrub starts 2024-09-11T03:51:09.528 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:09 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:51:09.528 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:09 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:51:09.528 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:09 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:51:09.528 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:09 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "76"} : dispatch 2024-09-11T03:51:09.528 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:09 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "76"} : dispatch 2024-09-11T03:51:09.528 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:09 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:51:09.528 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:09 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "76"}]': finished 2024-09-11T03:51:09.528 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:09 smithi121 ceph-mon[30729]: osdmap e851: 12 total, 12 up, 12 in 2024-09-11T03:51:09.533 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:51:09.533 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:51:09.552 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:51:09.553 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:51:09.567 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:51:09.567 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:51:09.584 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:51:09.584 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:51:09.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:09.601 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:51:09.601 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:51:09.607 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:09.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:09.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:09.618 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:51:09.618 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:51:09.672 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:51:09.672 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:51:09.672 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:51:09.672 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:51:09.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:09.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:09.731 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:09.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:09.832 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:09.858 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:09.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:09.859 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:09.960 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:09.985 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:09.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:09.986 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:10.053 INFO:tasks.rados.rados.2.smithi175.stdout:3526: done (2 left) 2024-09-11T03:51:10.053 INFO:tasks.rados.rados.2.smithi175.stdout:3527: done (1 left) 2024-09-11T03:51:10.053 INFO:tasks.rados.rados.2.smithi175.stdout:3529: done (0 left) 2024-09-11T03:51:10.053 INFO:tasks.rados.rados.2.smithi175.stdout:3530: snap_remove snap 384 2024-09-11T03:51:10.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:10.107 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:10.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:10.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:10.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:10.230 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:10.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:10.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:10.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:10.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:10.354 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:10.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:10.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:10.476 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:10.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:10.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:10.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:10 smithi175 ceph-mon[30894]: pgmap v1431: 107 pgs: 2 peering, 10 unknown, 95 active+clean; 693 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2024-09-11T03:51:10.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:10 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:51:10.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:10 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:51:10.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:10 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:51:10.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:10 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1992429383' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:10.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:10 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:51:10.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:10 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:51:10.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:10 smithi175 ceph-mon[30894]: osdmap e852: 12 total, 12 up, 12 in 2024-09-11T03:51:10.578 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:10.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:10.602 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:10.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:10.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:10 smithi084 ceph-mon[27254]: pgmap v1431: 107 pgs: 2 peering, 10 unknown, 95 active+clean; 693 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2024-09-11T03:51:10.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:10 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:51:10.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:10 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:51:10.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:10 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:51:10.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:10 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1992429383' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:10.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:10 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:51:10.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:10 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:51:10.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:10 smithi084 ceph-mon[27254]: osdmap e852: 12 total, 12 up, 12 in 2024-09-11T03:51:10.656 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:51:10.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:10 smithi121 ceph-mon[30729]: pgmap v1431: 107 pgs: 2 peering, 10 unknown, 95 active+clean; 693 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2024-09-11T03:51:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:10 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:51:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:10 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:51:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:10 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:51:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1992429383' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:10 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:51:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:10 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:51:10.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:10 smithi121 ceph-mon[30729]: osdmap e852: 12 total, 12 up, 12 in 2024-09-11T03:51:10.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:10.724 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:10.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:10.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:10.826 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:10.848 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:10.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:10.950 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:10.976 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:10.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:11.052 INFO:tasks.rados.rados.2.smithi175.stdout:3530: done (0 left) 2024-09-11T03:51:11.053 INFO:tasks.rados.rados.2.smithi175.stdout:3531: read oid 23 snap -1 2024-09-11T03:51:11.053 INFO:tasks.rados.rados.2.smithi175.stdout:3531: expect (ObjNum 1189 snap 388 seq_num 1189) 2024-09-11T03:51:11.053 INFO:tasks.rados.rados.2.smithi175.stdout:3532: copy_from oid 20 from oid 21 current snap is 394 2024-09-11T03:51:11.053 INFO:tasks.rados.rados.2.smithi175.stdout:3533: delete oid 35 current snap is 394 2024-09-11T03:51:11.067 INFO:tasks.rados.rados.2.smithi175.stdout:3533: done (2 left) 2024-09-11T03:51:11.067 INFO:tasks.rados.rados.2.smithi175.stdout:3534: read oid 17 snap -1 2024-09-11T03:51:11.067 INFO:tasks.rados.rados.2.smithi175.stdout:3534: expect (ObjNum 1181 snap 385 seq_num 1181) 2024-09-11T03:51:11.068 INFO:tasks.rados.rados.2.smithi175.stdout:3535: read oid 27 snap -1 2024-09-11T03:51:11.068 INFO:tasks.rados.rados.2.smithi175.stdout:3535: expect (ObjNum 1198 snap 393 seq_num 1198) 2024-09-11T03:51:11.068 INFO:tasks.rados.rados.2.smithi175.stdout:3536: delete oid 30 current snap is 394 2024-09-11T03:51:11.068 INFO:tasks.rados.rados.2.smithi175.stdout:3531: expect (ObjNum 1189 snap 388 seq_num 1189) 2024-09-11T03:51:11.078 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:11.103 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T03:51:11.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:11.127 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:11.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:11.169 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:11.228 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:11.253 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:11.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:11.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:11.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:11 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:51:11.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:11 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:51:11.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:11 smithi084 ceph-mon[27254]: 2.4 scrub starts 2024-09-11T03:51:11.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:11 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:51:11.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:11 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:51:11.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:11 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:51:11.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:11 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:51:11.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:11 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:51:11.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:11 smithi084 ceph-mon[27254]: osdmap e853: 12 total, 12 up, 12 in 2024-09-11T03:51:11.304 INFO:tasks.rados.rados.2.smithi175.stdout:3535: expect (ObjNum 1198 snap 393 seq_num 1198) 2024-09-11T03:51:11.355 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:11.379 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:11.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:11.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:11.386 INFO:tasks.rados.rados.2.smithi175.stdout:3532: finishing copy_from racing read to smithi17557870-20 2024-09-11T03:51:11.386 INFO:tasks.rados.rados.2.smithi175.stdout:3534: expect (ObjNum 1181 snap 385 seq_num 1181) 2024-09-11T03:51:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:11 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:51:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:11 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:51:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:11 smithi175 ceph-mon[30894]: 2.4 scrub starts 2024-09-11T03:51:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:11 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:51:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:11 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:51:11.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:11 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:51:11.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:11 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:51:11.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:11 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:51:11.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:11 smithi175 ceph-mon[30894]: osdmap e853: 12 total, 12 up, 12 in 2024-09-11T03:51:11.481 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:11.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:11.503 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:11.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:11.508 INFO:tasks.rados.rados.2.smithi175.stdout:3532: finishing copy_from to smithi17557870-20 2024-09-11T03:51:11.508 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 1006 (ObjNum 973 snap 313 seq_num 973) dirty exists 2024-09-11T03:51:11.508 INFO:tasks.rados.rados.2.smithi175.stdout:3531: done (4 left) 2024-09-11T03:51:11.508 INFO:tasks.rados.rados.2.smithi175.stdout:3532: done (3 left) 2024-09-11T03:51:11.508 INFO:tasks.rados.rados.2.smithi175.stdout:3534: done (2 left) 2024-09-11T03:51:11.508 INFO:tasks.rados.rados.2.smithi175.stdout:3535: done (1 left) 2024-09-11T03:51:11.508 INFO:tasks.rados.rados.2.smithi175.stdout:3536: done (0 left) 2024-09-11T03:51:11.508 INFO:tasks.rados.rados.2.smithi175.stdout:3537: read oid 43 snap -1 2024-09-11T03:51:11.509 INFO:tasks.rados.rados.2.smithi175.stdout:3537: expect deleted 2024-09-11T03:51:11.509 INFO:tasks.rados.rados.2.smithi175.stdout:3538: snap_create 2024-09-11T03:51:11.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:11.625 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:11.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:11.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:11.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:11 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:51:11.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:11 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:51:11.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:11 smithi121 ceph-mon[30729]: 2.4 scrub starts 2024-09-11T03:51:11.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:11 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:51:11.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:11 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:51:11.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:11 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:51:11.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:11 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:51:11.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:11 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:51:11.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:11 smithi121 ceph-mon[30729]: osdmap e853: 12 total, 12 up, 12 in 2024-09-11T03:51:11.727 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:11.744 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:11.748 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:11.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:11.749 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:11.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:11.872 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:11.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:11.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:11.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:11.996 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:11.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:11.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:12.069 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:51:12.072 INFO:tasks.rados.rados.2.smithi175.stdout:3537: done (1 left) 2024-09-11T03:51:12.072 INFO:tasks.rados.rados.2.smithi175.stdout:3538: done (0 left) 2024-09-11T03:51:12.072 INFO:tasks.rados.rados.2.smithi175.stdout:3539: delete oid 16 current snap is 395 2024-09-11T03:51:12.077 INFO:tasks.rados.rados.2.smithi175.stdout:3539: done (0 left) 2024-09-11T03:51:12.077 INFO:tasks.rados.rados.2.smithi175.stdout:3540: write oid 38 current snap is 395 2024-09-11T03:51:12.077 INFO:tasks.rados.rados.2.smithi175.stdout:3540: seq_num 1203 ranges {618821=508321,1613485=531077} 2024-09-11T03:51:12.083 INFO:tasks.rados.rados.2.smithi175.stdout:3540: writing smithi17557870-38 from 618821 to 1127142 tid 1 2024-09-11T03:51:12.089 INFO:tasks.rados.rados.2.smithi175.stdout:3540: writing smithi17557870-38 from 1613485 to 2144562 tid 2 2024-09-11T03:51:12.089 INFO:tasks.rados.rados.2.smithi175.stdout:3541: read oid 43 snap -1 2024-09-11T03:51:12.089 INFO:tasks.rados.rados.2.smithi175.stdout:3541: expect deleted 2024-09-11T03:51:12.089 INFO:tasks.rados.rados.2.smithi175.stdout:3542: write (excl) oid 5 current snap is 395 2024-09-11T03:51:12.090 INFO:tasks.rados.rados.2.smithi175.stdout:3540: finishing write tid 1 to smithi17557870-38 2024-09-11T03:51:12.090 INFO:tasks.rados.rados.2.smithi175.stdout:3542: seq_num 1204 ranges {635628=583570,1671967=563661} 2024-09-11T03:51:12.096 INFO:tasks.rados.rados.2.smithi175.stdout:3542: writing smithi17557870-5 from 635628 to 1219198 tid 1 2024-09-11T03:51:12.098 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:12.102 INFO:tasks.rados.rados.2.smithi175.stdout:3542: writing smithi17557870-5 from 1671967 to 2235628 tid 2 2024-09-11T03:51:12.102 INFO:tasks.rados.rados.2.smithi175.stdout:3543: read oid 14 snap -1 2024-09-11T03:51:12.102 INFO:tasks.rados.rados.2.smithi175.stdout:3543: expect (ObjNum 1165 snap 382 seq_num 1165) 2024-09-11T03:51:12.102 INFO:tasks.rados.rados.2.smithi175.stdout:3544: read oid 8 snap 322 2024-09-11T03:51:12.103 INFO:tasks.rados.rados.2.smithi175.stdout:3544: expect (ObjNum 963 snap 310 seq_num 963) 2024-09-11T03:51:12.103 INFO:tasks.rados.rados.2.smithi175.stdout:3545: rollback oid 46 current snap is 395 2024-09-11T03:51:12.103 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 46 to 331 2024-09-11T03:51:12.103 INFO:tasks.rados.rados.2.smithi175.stdout:3540: finishing write tid 2 to smithi17557870-38 2024-09-11T03:51:12.103 INFO:tasks.rados.rados.2.smithi175.stdout:3541: done (5 left) 2024-09-11T03:51:12.103 INFO:tasks.rados.rados.2.smithi175.stdout:3546: write oid 13 current snap is 395 2024-09-11T03:51:12.103 INFO:tasks.rados.rados.2.smithi175.stdout:3540: finishing write tid 3 to smithi17557870-38 2024-09-11T03:51:12.103 INFO:tasks.rados.rados.2.smithi175.stdout:3540: finishing write tid 4 to smithi17557870-38 2024-09-11T03:51:12.103 INFO:tasks.rados.rados.2.smithi175.stdout:3540: oid 38 updating version 0 to 1381 2024-09-11T03:51:12.103 INFO:tasks.rados.rados.2.smithi175.stdout:3540: oid 38 version 1381 is already newer than 1379 2024-09-11T03:51:12.103 INFO:tasks.rados.rados.2.smithi175.stdout:3540: oid 38 version 1381 is already newer than 1380 2024-09-11T03:51:12.103 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 1381 (ObjNum 1203 snap 395 seq_num 1203) dirty exists 2024-09-11T03:51:12.103 INFO:tasks.rados.rados.2.smithi175.stdout:3540: left oid 38 (ObjNum 1203 snap 395 seq_num 1203) 2024-09-11T03:51:12.103 INFO:tasks.rados.rados.2.smithi175.stdout:3542: finishing write tid 1 to smithi17557870-5 2024-09-11T03:51:12.104 INFO:tasks.rados.rados.2.smithi175.stdout:3546: seq_num 1205 ranges {652435=658819,1730449=521986} 2024-09-11T03:51:12.109 INFO:tasks.rados.rados.2.smithi175.stdout:3546: writing smithi17557870-13 from 652435 to 1311254 tid 1 2024-09-11T03:51:12.115 INFO:tasks.rados.rados.2.smithi175.stdout:3546: writing smithi17557870-13 from 1730449 to 2252435 tid 2 2024-09-11T03:51:12.115 INFO:tasks.rados.rados.2.smithi175.stdout:3540: done (5 left) 2024-09-11T03:51:12.115 INFO:tasks.rados.rados.2.smithi175.stdout:3547: snap_create 2024-09-11T03:51:12.115 INFO:tasks.rados.rados.2.smithi175.stdout:3542: finishing write tid 2 to smithi17557870-5 2024-09-11T03:51:12.115 INFO:tasks.rados.rados.2.smithi175.stdout:3542: finishing write tid 3 to smithi17557870-5 2024-09-11T03:51:12.115 INFO:tasks.rados.rados.2.smithi175.stdout:3545: finishing rollback tid 0 to smithi17557870-46 2024-09-11T03:51:12.115 INFO:tasks.rados.rados.2.smithi175.stdout:3542: finishing write tid 4 to smithi17557870-5 2024-09-11T03:51:12.115 INFO:tasks.rados.rados.2.smithi175.stdout:3542: oid 5 updating version 0 to 1038 2024-09-11T03:51:12.116 INFO:tasks.rados.rados.2.smithi175.stdout:3542: oid 5 version 1038 is already newer than 1036 2024-09-11T03:51:12.116 INFO:tasks.rados.rados.2.smithi175.stdout:3542: oid 5 version 1038 is already newer than 1037 2024-09-11T03:51:12.116 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 1038 (ObjNum 1204 snap 395 seq_num 1204) dirty exists 2024-09-11T03:51:12.116 INFO:tasks.rados.rados.2.smithi175.stdout:3542: left oid 5 (ObjNum 1204 snap 395 seq_num 1204) 2024-09-11T03:51:12.116 INFO:tasks.rados.rados.2.smithi175.stdout:3545: finishing rollback tid 1 to smithi17557870-46 2024-09-11T03:51:12.116 INFO:tasks.rados.rados.2.smithi175.stdout:3545: finishing rollback tid 2 to smithi17557870-46 2024-09-11T03:51:12.116 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 972 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:51:12.116 INFO:tasks.rados.rados.2.smithi175.stdout:3546: finishing write tid 1 to smithi17557870-13 2024-09-11T03:51:12.118 INFO:tasks.rados.rados.2.smithi175.stdout:3546: finishing write tid 2 to smithi17557870-13 2024-09-11T03:51:12.118 INFO:tasks.rados.rados.2.smithi175.stdout:3546: finishing write tid 3 to smithi17557870-13 2024-09-11T03:51:12.118 INFO:tasks.rados.rados.2.smithi175.stdout:3544: expect (ObjNum 963 snap 310 seq_num 963) 2024-09-11T03:51:12.123 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:12.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:12.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:12.225 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:12.238 INFO:tasks.rados.rados.2.smithi175.stdout:3546: finishing write tid 4 to smithi17557870-13 2024-09-11T03:51:12.238 INFO:tasks.rados.rados.2.smithi175.stdout:3546: oid 13 updating version 0 to 1384 2024-09-11T03:51:12.239 INFO:tasks.rados.rados.2.smithi175.stdout:3546: oid 13 updating version 1384 to 1385 2024-09-11T03:51:12.239 INFO:tasks.rados.rados.2.smithi175.stdout:3546: oid 13 version 1385 is already newer than 1383 2024-09-11T03:51:12.239 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 1385 (ObjNum 1205 snap 395 seq_num 1205) dirty exists 2024-09-11T03:51:12.239 INFO:tasks.rados.rados.2.smithi175.stdout:3546: left oid 13 (ObjNum 1205 snap 395 seq_num 1205) 2024-09-11T03:51:12.239 INFO:tasks.rados.rados.2.smithi175.stdout:3543: expect (ObjNum 1165 snap 382 seq_num 1165) 2024-09-11T03:51:12.246 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:12.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:12.247 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:12.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:12.370 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:12.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:12.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:12 smithi084 ceph-mon[27254]: pgmap v1434: 107 pgs: 2 peering, 10 unknown, 95 active+clean; 693 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 767 KiB/s wr, 6 op/s 2024-09-11T03:51:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:12 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:51:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:12 smithi084 ceph-mon[27254]: 2.4 scrub starts 2024-09-11T03:51:12.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:12 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:51:12.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:12 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:51:12.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:12 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:51:12.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:12 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1445441475' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:51:12.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:12 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:51:12.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:12 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:12.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:12 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1445441475' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:51:12.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:12 smithi084 ceph-mon[27254]: osdmap e854: 12 total, 12 up, 12 in 2024-09-11T03:51:12.398 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:51:12.472 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:12 smithi175 ceph-mon[30894]: pgmap v1434: 107 pgs: 2 peering, 10 unknown, 95 active+clean; 693 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 767 KiB/s wr, 6 op/s 2024-09-11T03:51:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:12 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:51:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:12 smithi175 ceph-mon[30894]: 2.4 scrub starts 2024-09-11T03:51:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:12 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:51:12.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:12 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:51:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:12 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:51:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:12 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1445441475' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:51:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:12 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:51:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:12 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:12 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1445441475' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:51:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:12 smithi175 ceph-mon[30894]: osdmap e854: 12 total, 12 up, 12 in 2024-09-11T03:51:12.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:12.493 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:12.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:12.594 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:12.615 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:12.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:12.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:12.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:12 smithi121 ceph-mon[30729]: pgmap v1434: 107 pgs: 2 peering, 10 unknown, 95 active+clean; 693 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 767 KiB/s wr, 6 op/s 2024-09-11T03:51:12.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:12 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:51:12.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:12 smithi121 ceph-mon[30729]: 2.4 scrub starts 2024-09-11T03:51:12.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:12 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:51:12.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:12 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:51:12.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:12 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:51:12.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:12 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1445441475' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:51:12.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:12 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:51:12.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:12 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:12.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:12 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1445441475' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:51:12.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:12 smithi121 ceph-mon[30729]: osdmap e854: 12 total, 12 up, 12 in 2024-09-11T03:51:12.716 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:12.740 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:12.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:12.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:12.842 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:12.863 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:12.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:12.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:12.990 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:12.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:12.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:13.058 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:51:13.058 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:51:13.058 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:51:13.059 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:51:13.059 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:51:13.074 INFO:tasks.rados.rados.2.smithi175.stdout:3542: done (5 left) 2024-09-11T03:51:13.074 INFO:tasks.rados.rados.2.smithi175.stdout:3543: done (4 left) 2024-09-11T03:51:13.074 INFO:tasks.rados.rados.2.smithi175.stdout:3544: done (3 left) 2024-09-11T03:51:13.074 INFO:tasks.rados.rados.2.smithi175.stdout:3545: done (2 left) 2024-09-11T03:51:13.075 INFO:tasks.rados.rados.2.smithi175.stdout:3546: done (1 left) 2024-09-11T03:51:13.075 INFO:tasks.rados.rados.2.smithi175.stdout:3547: done (0 left) 2024-09-11T03:51:13.075 INFO:tasks.rados.rados.2.smithi175.stdout:3548: snap_create 2024-09-11T03:51:13.090 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:51:13.090 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:51:13.090 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 10, 6] 2024-09-11T03:51:13.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd out 4 2024-09-11T03:51:13.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:13.112 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:13.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:13.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:13.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:13.234 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:13.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:13.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:13.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:13.357 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:13.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:13.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:13.459 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:13 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:51:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:13 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:51:13.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:13 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:51:13.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:13 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:51:13.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:13 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:51:13.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:13 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:51:13.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:13 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:51:13.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:13 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:51:13.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:13 smithi175 ceph-mon[30894]: osdmap e855: 12 total, 12 up, 12 in 2024-09-11T03:51:13.485 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:13.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:13.486 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:13.587 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:13.613 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:13.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:13.614 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:13.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:13 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:51:13.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:13 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:51:13.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:13 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:51:13.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:13 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:51:13.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:13 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:51:13.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:13 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:51:13.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:13 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:51:13.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:13 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:51:13.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:13 smithi084 ceph-mon[27254]: osdmap e855: 12 total, 12 up, 12 in 2024-09-11T03:51:13.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:13 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:51:13.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:13 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:51:13.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:13 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:51:13.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:13 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:51:13.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:13 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:51:13.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:13 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:51:13.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:13 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:51:13.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:13 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:51:13.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:13 smithi121 ceph-mon[30729]: osdmap e855: 12 total, 12 up, 12 in 2024-09-11T03:51:13.704 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T03:51:13.714 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:13.725 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:13.740 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:13.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:13.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:13.842 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:13.868 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:13.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:13.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:13.970 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:13.991 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:13.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:13.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:14.075 INFO:tasks.rados.rados.2.smithi175.stdout:3548: done (0 left) 2024-09-11T03:51:14.075 INFO:tasks.rados.rados.2.smithi175.stdout:3549: rollback oid 16 current snap is 397 2024-09-11T03:51:14.075 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 16 to 381 2024-09-11T03:51:14.075 INFO:tasks.rados.rados.2.smithi175.stdout:3550: write (excl) oid 48 current snap is 397 2024-09-11T03:51:14.076 INFO:tasks.rados.rados.2.smithi175.stdout:3550: seq_num 1206 ranges {669242=734068,2188931=80311} 2024-09-11T03:51:14.082 INFO:tasks.rados.rados.2.smithi175.stdout:3550: writing smithi17557870-48 from 669242 to 1403310 tid 1 2024-09-11T03:51:14.088 INFO:tasks.rados.rados.2.smithi175.stdout:3550: writing smithi17557870-48 from 2188931 to 2269242 tid 2 2024-09-11T03:51:14.088 INFO:tasks.rados.rados.2.smithi175.stdout:3549: finishing rollback tid 1 to smithi17557870-16 2024-09-11T03:51:14.088 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 1000 (ObjNum 65 snap 0 seq_num 0) dirty dne 2024-09-11T03:51:14.088 INFO:tasks.rados.rados.2.smithi175.stdout:3551: read oid 39 snap -1 2024-09-11T03:51:14.088 INFO:tasks.rados.rados.2.smithi175.stdout:3551: expect deleted 2024-09-11T03:51:14.089 INFO:tasks.rados.rados.2.smithi175.stdout:3549: done (2 left) 2024-09-11T03:51:14.089 INFO:tasks.rados.rados.2.smithi175.stdout:3552: delete oid 45 current snap is 397 2024-09-11T03:51:14.089 INFO:tasks.rados.rados.2.smithi175.stdout:3550: finishing write tid 1 to smithi17557870-48 2024-09-11T03:51:14.089 INFO:tasks.rados.rados.2.smithi175.stdout:3550: finishing write tid 2 to smithi17557870-48 2024-09-11T03:51:14.090 INFO:tasks.rados.rados.2.smithi175.stdout:3550: finishing write tid 3 to smithi17557870-48 2024-09-11T03:51:14.090 INFO:tasks.rados.rados.2.smithi175.stdout:3550: finishing write tid 4 to smithi17557870-48 2024-09-11T03:51:14.090 INFO:tasks.rados.rados.2.smithi175.stdout:3550: oid 48 updating version 0 to 1007 2024-09-11T03:51:14.090 INFO:tasks.rados.rados.2.smithi175.stdout:3550: oid 48 version 1007 is already newer than 1006 2024-09-11T03:51:14.091 INFO:tasks.rados.rados.2.smithi175.stdout:3550: oid 48 updating version 1007 to 1008 2024-09-11T03:51:14.091 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 1008 (ObjNum 1206 snap 397 seq_num 1206) dirty exists 2024-09-11T03:51:14.091 INFO:tasks.rados.rados.2.smithi175.stdout:3550: left oid 48 (ObjNum 1206 snap 397 seq_num 1206) 2024-09-11T03:51:14.091 INFO:tasks.rados.rados.2.smithi175.stdout:3550: done (2 left) 2024-09-11T03:51:14.091 INFO:tasks.rados.rados.2.smithi175.stdout:3551: done (1 left) 2024-09-11T03:51:14.091 INFO:tasks.rados.rados.2.smithi175.stdout:3552: done (0 left) 2024-09-11T03:51:14.092 INFO:tasks.rados.rados.2.smithi175.stdout:3553: write (excl) oid 34 current snap is 397 2024-09-11T03:51:14.092 INFO:tasks.rados.rados.2.smithi175.stdout:3553: seq_num 1208 ranges {702856=484566,1789542=513314} 2024-09-11T03:51:14.093 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:14.098 INFO:tasks.rados.rados.2.smithi175.stdout:3553: writing smithi17557870-34 from 702856 to 1187422 tid 1 2024-09-11T03:51:14.105 INFO:tasks.rados.rados.2.smithi175.stdout:3553: writing smithi17557870-34 from 1789542 to 2302856 tid 2 2024-09-11T03:51:14.105 INFO:tasks.rados.rados.2.smithi175.stdout:3554: write (excl) oid 10 current snap is 397 2024-09-11T03:51:14.105 INFO:tasks.rados.rados.2.smithi175.stdout:3553: finishing write tid 1 to smithi17557870-34 2024-09-11T03:51:14.105 INFO:tasks.rados.rados.2.smithi175.stdout:3554: seq_num 1209 ranges {719663=676168,1964377=355286} 2024-09-11T03:51:14.112 INFO:tasks.rados.rados.2.smithi175.stdout:3554: writing smithi17557870-10 from 719663 to 1395831 tid 1 2024-09-11T03:51:14.118 INFO:tasks.rados.rados.2.smithi175.stdout:3554: writing smithi17557870-10 from 1964377 to 2319663 tid 2 2024-09-11T03:51:14.118 INFO:tasks.rados.rados.2.smithi175.stdout:3555: write (excl) oid 5 current snap is 397 2024-09-11T03:51:14.118 INFO:tasks.rados.rados.2.smithi175.stdout:3553: finishing write tid 2 to smithi17557870-34 2024-09-11T03:51:14.118 INFO:tasks.rados.rados.2.smithi175.stdout:3553: finishing write tid 3 to smithi17557870-34 2024-09-11T03:51:14.118 INFO:tasks.rados.rados.2.smithi175.stdout:3553: finishing write tid 4 to smithi17557870-34 2024-09-11T03:51:14.118 INFO:tasks.rados.rados.2.smithi175.stdout:3553: oid 34 updating version 0 to 1057 2024-09-11T03:51:14.118 INFO:tasks.rados.rados.2.smithi175.stdout:3553: oid 34 updating version 1057 to 1058 2024-09-11T03:51:14.118 INFO:tasks.rados.rados.2.smithi175.stdout:3553: oid 34 version 1058 is already newer than 1056 2024-09-11T03:51:14.118 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 1058 (ObjNum 1208 snap 397 seq_num 1208) dirty exists 2024-09-11T03:51:14.119 INFO:tasks.rados.rados.2.smithi175.stdout:3553: left oid 34 (ObjNum 1208 snap 397 seq_num 1208) 2024-09-11T03:51:14.119 INFO:tasks.rados.rados.2.smithi175.stdout:3554: finishing write tid 1 to smithi17557870-10 2024-09-11T03:51:14.119 INFO:tasks.rados.rados.2.smithi175.stdout:3555: seq_num 1210 ranges {736470=751417,2022859=313611} 2024-09-11T03:51:14.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:14.120 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:14.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:14.126 INFO:tasks.rados.rados.2.smithi175.stdout:3555: writing smithi17557870-5 from 736470 to 1487887 tid 1 2024-09-11T03:51:14.131 INFO:tasks.rados.rados.2.smithi175.stdout:3555: writing smithi17557870-5 from 2022859 to 2336470 tid 2 2024-09-11T03:51:14.131 INFO:tasks.rados.rados.2.smithi175.stdout:3553: done (2 left) 2024-09-11T03:51:14.131 INFO:tasks.rados.rados.2.smithi175.stdout:3556: rollback oid 31 current snap is 397 2024-09-11T03:51:14.131 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 31 to 348 2024-09-11T03:51:14.131 INFO:tasks.rados.rados.2.smithi175.stdout:3554: finishing write tid 2 to smithi17557870-10 2024-09-11T03:51:14.131 INFO:tasks.rados.rados.2.smithi175.stdout:3554: finishing write tid 3 to smithi17557870-10 2024-09-11T03:51:14.131 INFO:tasks.rados.rados.2.smithi175.stdout:3554: finishing write tid 4 to smithi17557870-10 2024-09-11T03:51:14.132 INFO:tasks.rados.rados.2.smithi175.stdout:3554: oid 10 updating version 0 to 1202 2024-09-11T03:51:14.132 INFO:tasks.rados.rados.2.smithi175.stdout:3554: oid 10 version 1202 is already newer than 1200 2024-09-11T03:51:14.132 INFO:tasks.rados.rados.2.smithi175.stdout:3554: oid 10 version 1202 is already newer than 1201 2024-09-11T03:51:14.132 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 1202 (ObjNum 1209 snap 397 seq_num 1209) dirty exists 2024-09-11T03:51:14.132 INFO:tasks.rados.rados.2.smithi175.stdout:3554: left oid 10 (ObjNum 1209 snap 397 seq_num 1209) 2024-09-11T03:51:14.132 INFO:tasks.rados.rados.2.smithi175.stdout:3555: finishing write tid 1 to smithi17557870-5 2024-09-11T03:51:14.132 INFO:tasks.rados.rados.2.smithi175.stdout:3554: done (2 left) 2024-09-11T03:51:14.132 INFO:tasks.rados.rados.2.smithi175.stdout:3557: read oid 32 snap -1 2024-09-11T03:51:14.132 INFO:tasks.rados.rados.2.smithi175.stdout:3557: expect (ObjNum 1059 snap 345 seq_num 1059) 2024-09-11T03:51:14.132 INFO:tasks.rados.rados.2.smithi175.stdout:3558: write (excl) oid 21 current snap is 397 2024-09-11T03:51:14.132 INFO:tasks.rados.rados.2.smithi175.stdout:3558: seq_num 1211 ranges {753277=426666,1681341=545753} 2024-09-11T03:51:14.137 INFO:tasks.rados.rados.2.smithi175.stdout:3558: writing smithi17557870-21 from 753277 to 1179943 tid 1 2024-09-11T03:51:14.143 INFO:tasks.rados.rados.2.smithi175.stdout:3558: writing smithi17557870-21 from 1681341 to 2227094 tid 2 2024-09-11T03:51:14.143 INFO:tasks.rados.rados.2.smithi175.stdout:3559: write oid 22 current snap is 397 2024-09-11T03:51:14.143 INFO:tasks.rados.rados.2.smithi175.stdout:3555: finishing write tid 2 to smithi17557870-5 2024-09-11T03:51:14.143 INFO:tasks.rados.rados.2.smithi175.stdout:3556: finishing rollback tid 0 to smithi17557870-31 2024-09-11T03:51:14.143 INFO:tasks.rados.rados.2.smithi175.stdout:3559: seq_num 1212 ranges {770084=501915,2023470=346614} 2024-09-11T03:51:14.150 INFO:tasks.rados.rados.2.smithi175.stdout:3559: writing smithi17557870-22 from 770084 to 1271999 tid 1 2024-09-11T03:51:14.156 INFO:tasks.rados.rados.2.smithi175.stdout:3559: writing smithi17557870-22 from 2023470 to 2370084 tid 2 2024-09-11T03:51:14.156 INFO:tasks.rados.rados.2.smithi175.stdout:3560: write (excl) oid 50 current snap is 397 2024-09-11T03:51:14.156 INFO:tasks.rados.rados.2.smithi175.stdout:3560: seq_num 1213 ranges {786891=577164,2081952=304939} 2024-09-11T03:51:14.163 INFO:tasks.rados.rados.2.smithi175.stdout:3560: writing smithi17557870-50 from 786891 to 1364055 tid 1 2024-09-11T03:51:14.169 INFO:tasks.rados.rados.2.smithi175.stdout:3560: writing smithi17557870-50 from 2081952 to 2386891 tid 2 2024-09-11T03:51:14.169 INFO:tasks.rados.rados.2.smithi175.stdout:3561: read oid 2 snap 374 2024-09-11T03:51:14.170 INFO:tasks.rados.rados.2.smithi175.stdout:3561: expect deleted 2024-09-11T03:51:14.170 INFO:tasks.rados.rados.2.smithi175.stdout:3562: read oid 34 snap -1 2024-09-11T03:51:14.170 INFO:tasks.rados.rados.2.smithi175.stdout:3562: expect (ObjNum 1208 snap 397 seq_num 1208) 2024-09-11T03:51:14.170 INFO:tasks.rados.rados.2.smithi175.stdout:3563: write (excl) oid 27 current snap is 397 2024-09-11T03:51:14.170 INFO:tasks.rados.rados.2.smithi175.stdout:3563: seq_num 1214 ranges {403698=652413,1740434=663264} 2024-09-11T03:51:14.175 INFO:tasks.rados.rados.2.smithi175.stdout:3563: writing smithi17557870-27 from 403698 to 1056111 tid 1 2024-09-11T03:51:14.183 INFO:tasks.rados.rados.2.smithi175.stdout:3563: writing smithi17557870-27 from 1740434 to 2403698 tid 2 2024-09-11T03:51:14.184 INFO:tasks.rados.rados.2.smithi175.stdout:3564: rollback oid 8 current snap is 397 2024-09-11T03:51:14.184 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 8 to 379 2024-09-11T03:51:14.184 INFO:tasks.rados.rados.2.smithi175.stdout:3555: finishing write tid 3 to smithi17557870-5 2024-09-11T03:51:14.184 INFO:tasks.rados.rados.2.smithi175.stdout:3556: finishing rollback tid 1 to smithi17557870-31 2024-09-11T03:51:14.184 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 1243 (ObjNum 0 snap 0 seq_num 304) dirty dne 2024-09-11T03:51:14.184 INFO:tasks.rados.rados.2.smithi175.stdout:3555: finishing write tid 4 to smithi17557870-5 2024-09-11T03:51:14.184 INFO:tasks.rados.rados.2.smithi175.stdout:3555: oid 5 updating version 0 to 1040 2024-09-11T03:51:14.184 INFO:tasks.rados.rados.2.smithi175.stdout:3555: oid 5 updating version 1040 to 1041 2024-09-11T03:51:14.184 INFO:tasks.rados.rados.2.smithi175.stdout:3555: oid 5 version 1041 is already newer than 1039 2024-09-11T03:51:14.184 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 1041 (ObjNum 1210 snap 397 seq_num 1210) dirty exists 2024-09-11T03:51:14.184 INFO:tasks.rados.rados.2.smithi175.stdout:3555: left oid 5 (ObjNum 1210 snap 397 seq_num 1210) 2024-09-11T03:51:14.184 INFO:tasks.rados.rados.2.smithi175.stdout:3557: expect (ObjNum 1059 snap 345 seq_num 1059) 2024-09-11T03:51:14.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:14.244 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:14.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:14.304 INFO:tasks.rados.rados.2.smithi175.stdout:3555: done (9 left) 2024-09-11T03:51:14.304 INFO:tasks.rados.rados.2.smithi175.stdout:3556: done (8 left) 2024-09-11T03:51:14.304 INFO:tasks.rados.rados.2.smithi175.stdout:3557: done (7 left) 2024-09-11T03:51:14.304 INFO:tasks.rados.rados.2.smithi175.stdout:3565: snap_remove snap 372 2024-09-11T03:51:14.347 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:14.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:51:14.367 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:51:14.368 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:14.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:14.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:14.383 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:51:14.383 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:51:14.391 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:14.405 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:51:14.405 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:51:14.424 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:51:14.425 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:51:14.427 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:14.442 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:51:14.442 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:51:14.459 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:51:14.460 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:51:14.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:14 smithi084 ceph-mon[27254]: pgmap v1437: 107 pgs: 1 peering, 6 active+undersized+remapped, 3 active+clean+scrubbing, 1 remapped+peering, 2 unknown, 94 active+clean; 693 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.0 MiB/s wr, 7 op/s; 16 MiB/s, 34 objects/s recovering 2024-09-11T03:51:14.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:14 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:51:14.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:14 smithi084 ceph-mon[27254]: 2.47 scrub starts 2024-09-11T03:51:14.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:14 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:51:14.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:14 smithi084 ceph-mon[27254]: 2.48 scrub starts 2024-09-11T03:51:14.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:14 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:51:14.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:14 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:51:14.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:14 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:51:14.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:14 smithi084 ceph-mon[27254]: 2.4a scrub starts 2024-09-11T03:51:14.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:14 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:51:14.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:14 smithi084 ceph-mon[27254]: osdmap e856: 12 total, 12 up, 12 in 2024-09-11T03:51:14.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:14.476 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:51:14.476 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:51:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:14 smithi175 ceph-mon[30894]: pgmap v1437: 107 pgs: 1 peering, 6 active+undersized+remapped, 3 active+clean+scrubbing, 1 remapped+peering, 2 unknown, 94 active+clean; 693 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.0 MiB/s wr, 7 op/s; 16 MiB/s, 34 objects/s recovering 2024-09-11T03:51:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:14 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:51:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:14 smithi175 ceph-mon[30894]: 2.47 scrub starts 2024-09-11T03:51:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:14 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:51:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:14 smithi175 ceph-mon[30894]: 2.48 scrub starts 2024-09-11T03:51:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:14 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:51:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:14 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:51:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:14 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:51:14.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:14 smithi175 ceph-mon[30894]: 2.4a scrub starts 2024-09-11T03:51:14.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:14 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:51:14.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:14 smithi175 ceph-mon[30894]: osdmap e856: 12 total, 12 up, 12 in 2024-09-11T03:51:14.493 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:51:14.493 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:51:14.495 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:14.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:14.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:14.509 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:51:14.509 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:51:14.525 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:51:14.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:51:14.542 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:51:14.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:51:14.559 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:51:14.559 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:51:14.597 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:14.618 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:14.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:14.619 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:14.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:14 smithi121 ceph-mon[30729]: pgmap v1437: 107 pgs: 1 peering, 6 active+undersized+remapped, 3 active+clean+scrubbing, 1 remapped+peering, 2 unknown, 94 active+clean; 693 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.0 MiB/s wr, 7 op/s; 16 MiB/s, 34 objects/s recovering 2024-09-11T03:51:14.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:14 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:51:14.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:14 smithi121 ceph-mon[30729]: 2.47 scrub starts 2024-09-11T03:51:14.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:14 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:51:14.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:14 smithi121 ceph-mon[30729]: 2.48 scrub starts 2024-09-11T03:51:14.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:14 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:51:14.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:14 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:51:14.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:14 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:51:14.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:14 smithi121 ceph-mon[30729]: 2.4a scrub starts 2024-09-11T03:51:14.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:14 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:51:14.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:14 smithi121 ceph-mon[30729]: osdmap e856: 12 total, 12 up, 12 in 2024-09-11T03:51:14.719 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:14.740 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:14.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:14.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:14.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:14.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:14.868 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:14.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:14.902 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:51:14.969 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:14.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:14.991 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:14.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:15.093 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:15.114 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:15.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:15.206 INFO:tasks.rados.rados.2.smithi175.stdout:3565: done (7 left) 2024-09-11T03:51:15.206 INFO:tasks.rados.rados.2.smithi175.stdout:3566: write (excl) oid 38 current snap is 397 2024-09-11T03:51:15.206 INFO:tasks.rados.rados.2.smithi175.stdout:3566: seq_num 1215 ranges {420505=727662,1798916=553091} 2024-09-11T03:51:15.213 INFO:tasks.rados.rados.2.smithi175.stdout:3566: writing smithi17557870-38 from 420505 to 1148167 tid 1 2024-09-11T03:51:15.215 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:15.222 INFO:tasks.rados.rados.2.smithi175.stdout:3566: writing smithi17557870-38 from 1798916 to 2352007 tid 2 2024-09-11T03:51:15.222 INFO:tasks.rados.rados.2.smithi175.stdout:3567: read oid 19 snap -1 2024-09-11T03:51:15.222 INFO:tasks.rados.rados.2.smithi175.stdout:3567: expect deleted 2024-09-11T03:51:15.222 INFO:tasks.rados.rados.2.smithi175.stdout:3568: rollback oid 14 current snap is 397 2024-09-11T03:51:15.223 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 14 to 381 2024-09-11T03:51:15.223 INFO:tasks.rados.rados.2.smithi175.stdout:3560: finishing write tid 1 to smithi17557870-50 2024-09-11T03:51:15.223 INFO:tasks.rados.rados.2.smithi175.stdout:3560: finishing write tid 2 to smithi17557870-50 2024-09-11T03:51:15.223 INFO:tasks.rados.rados.2.smithi175.stdout:3560: finishing write tid 3 to smithi17557870-50 2024-09-11T03:51:15.223 INFO:tasks.rados.rados.2.smithi175.stdout:3560: finishing write tid 4 to smithi17557870-50 2024-09-11T03:51:15.223 INFO:tasks.rados.rados.2.smithi175.stdout:3560: oid 50 updating version 0 to 1384 2024-09-11T03:51:15.224 INFO:tasks.rados.rados.2.smithi175.stdout:3560: oid 50 updating version 1384 to 1385 2024-09-11T03:51:15.224 INFO:tasks.rados.rados.2.smithi175.stdout:3560: oid 50 version 1385 is already newer than 1383 2024-09-11T03:51:15.224 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 1385 (ObjNum 1213 snap 397 seq_num 1213) dirty exists 2024-09-11T03:51:15.224 INFO:tasks.rados.rados.2.smithi175.stdout:3560: left oid 50 (ObjNum 1213 snap 397 seq_num 1213) 2024-09-11T03:51:15.224 INFO:tasks.rados.rados.2.smithi175.stdout:3558: finishing write tid 1 to smithi17557870-21 2024-09-11T03:51:15.224 INFO:tasks.rados.rados.2.smithi175.stdout:3559: finishing write tid 1 to smithi17557870-22 2024-09-11T03:51:15.225 INFO:tasks.rados.rados.2.smithi175.stdout:3559: finishing write tid 2 to smithi17557870-22 2024-09-11T03:51:15.225 INFO:tasks.rados.rados.2.smithi175.stdout:3559: finishing write tid 3 to smithi17557870-22 2024-09-11T03:51:15.225 INFO:tasks.rados.rados.2.smithi175.stdout:3559: finishing write tid 4 to smithi17557870-22 2024-09-11T03:51:15.226 INFO:tasks.rados.rados.2.smithi175.stdout:3559: oid 22 updating version 0 to 1284 2024-09-11T03:51:15.226 INFO:tasks.rados.rados.2.smithi175.stdout:3559: oid 22 version 1284 is already newer than 1283 2024-09-11T03:51:15.226 INFO:tasks.rados.rados.2.smithi175.stdout:3559: oid 22 updating version 1284 to 1285 2024-09-11T03:51:15.226 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 1285 (ObjNum 1212 snap 397 seq_num 1212) dirty exists 2024-09-11T03:51:15.226 INFO:tasks.rados.rados.2.smithi175.stdout:3559: left oid 22 (ObjNum 1212 snap 397 seq_num 1212) 2024-09-11T03:51:15.226 INFO:tasks.rados.rados.2.smithi175.stdout:3558: finishing write tid 2 to smithi17557870-21 2024-09-11T03:51:15.227 INFO:tasks.rados.rados.2.smithi175.stdout:3558: finishing write tid 3 to smithi17557870-21 2024-09-11T03:51:15.227 INFO:tasks.rados.rados.2.smithi175.stdout:3558: finishing write tid 4 to smithi17557870-21 2024-09-11T03:51:15.227 INFO:tasks.rados.rados.2.smithi175.stdout:3558: oid 21 updating version 0 to 1265 2024-09-11T03:51:15.227 INFO:tasks.rados.rados.2.smithi175.stdout:3558: oid 21 updating version 1265 to 1266 2024-09-11T03:51:15.227 INFO:tasks.rados.rados.2.smithi175.stdout:3558: oid 21 version 1266 is already newer than 1264 2024-09-11T03:51:15.227 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 1266 (ObjNum 1211 snap 397 seq_num 1211) dirty exists 2024-09-11T03:51:15.228 INFO:tasks.rados.rados.2.smithi175.stdout:3558: left oid 21 (ObjNum 1211 snap 397 seq_num 1211) 2024-09-11T03:51:15.228 INFO:tasks.rados.rados.2.smithi175.stdout:3562: expect (ObjNum 1208 snap 397 seq_num 1208) 2024-09-11T03:51:15.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:15.240 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:15.240 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:15.323 INFO:tasks.rados.rados.2.smithi175.stdout:3563: finishing write tid 1 to smithi17557870-27 2024-09-11T03:51:15.323 INFO:tasks.rados.rados.2.smithi175.stdout:3563: finishing write tid 2 to smithi17557870-27 2024-09-11T03:51:15.324 INFO:tasks.rados.rados.2.smithi175.stdout:3558: done (9 left) 2024-09-11T03:51:15.324 INFO:tasks.rados.rados.2.smithi175.stdout:3559: done (8 left) 2024-09-11T03:51:15.324 INFO:tasks.rados.rados.2.smithi175.stdout:3560: done (7 left) 2024-09-11T03:51:15.324 INFO:tasks.rados.rados.2.smithi175.stdout:3561: done (6 left) 2024-09-11T03:51:15.324 INFO:tasks.rados.rados.2.smithi175.stdout:3562: done (5 left) 2024-09-11T03:51:15.324 INFO:tasks.rados.rados.2.smithi175.stdout:3569: copy_from oid 4 from oid 25 current snap is 397 2024-09-11T03:51:15.324 INFO:tasks.rados.rados.2.smithi175.stdout:3563: finishing write tid 3 to smithi17557870-27 2024-09-11T03:51:15.324 INFO:tasks.rados.rados.2.smithi175.stdout:3563: finishing write tid 4 to smithi17557870-27 2024-09-11T03:51:15.324 INFO:tasks.rados.rados.2.smithi175.stdout:3563: oid 27 updating version 0 to 1244 2024-09-11T03:51:15.324 INFO:tasks.rados.rados.2.smithi175.stdout:3563: oid 27 updating version 1244 to 1246 2024-09-11T03:51:15.324 INFO:tasks.rados.rados.2.smithi175.stdout:3563: oid 27 version 1246 is already newer than 1245 2024-09-11T03:51:15.324 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 1246 (ObjNum 1214 snap 397 seq_num 1214) dirty exists 2024-09-11T03:51:15.324 INFO:tasks.rados.rados.2.smithi175.stdout:3563: left oid 27 (ObjNum 1214 snap 397 seq_num 1214) 2024-09-11T03:51:15.325 INFO:tasks.rados.rados.2.smithi175.stdout:3564: finishing rollback tid 0 to smithi17557870-8 2024-09-11T03:51:15.325 INFO:tasks.rados.rados.2.smithi175.stdout:3564: finishing rollback tid 1 to smithi17557870-8 2024-09-11T03:51:15.325 INFO:tasks.rados.rados.2.smithi175.stdout:3564: finishing rollback tid 2 to smithi17557870-8 2024-09-11T03:51:15.325 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 1117 (ObjNum 1147 snap 370 seq_num 1147) dirty exists 2024-09-11T03:51:15.325 INFO:tasks.rados.rados.2.smithi175.stdout:3568: finishing rollback tid 0 to smithi17557870-14 2024-09-11T03:51:15.325 INFO:tasks.rados.rados.2.smithi175.stdout:3568: finishing rollback tid 1 to smithi17557870-14 2024-09-11T03:51:15.325 INFO:tasks.rados.rados.2.smithi175.stdout:3566: finishing write tid 1 to smithi17557870-38 2024-09-11T03:51:15.325 INFO:tasks.rados.rados.2.smithi175.stdout:3568: finishing rollback tid 2 to smithi17557870-14 2024-09-11T03:51:15.325 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 1084 (ObjNum 1152 snap 372 seq_num 1152) dirty exists 2024-09-11T03:51:15.325 INFO:tasks.rados.rados.2.smithi175.stdout:3566: finishing write tid 2 to smithi17557870-38 2024-09-11T03:51:15.325 INFO:tasks.rados.rados.2.smithi175.stdout:3566: finishing write tid 3 to smithi17557870-38 2024-09-11T03:51:15.325 INFO:tasks.rados.rados.2.smithi175.stdout:3566: finishing write tid 4 to smithi17557870-38 2024-09-11T03:51:15.325 INFO:tasks.rados.rados.2.smithi175.stdout:3566: oid 38 updating version 0 to 1394 2024-09-11T03:51:15.325 INFO:tasks.rados.rados.2.smithi175.stdout:3566: oid 38 updating version 1394 to 1395 2024-09-11T03:51:15.326 INFO:tasks.rados.rados.2.smithi175.stdout:3566: oid 38 version 1395 is already newer than 1391 2024-09-11T03:51:15.326 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 1395 (ObjNum 1215 snap 397 seq_num 1215) dirty exists 2024-09-11T03:51:15.326 INFO:tasks.rados.rados.2.smithi175.stdout:3566: left oid 38 (ObjNum 1215 snap 397 seq_num 1215) 2024-09-11T03:51:15.326 INFO:tasks.rados.rados.2.smithi175.stdout:3563: done (5 left) 2024-09-11T03:51:15.326 INFO:tasks.rados.rados.2.smithi175.stdout:3564: done (4 left) 2024-09-11T03:51:15.326 INFO:tasks.rados.rados.2.smithi175.stdout:3566: done (3 left) 2024-09-11T03:51:15.326 INFO:tasks.rados.rados.2.smithi175.stdout:3567: done (2 left) 2024-09-11T03:51:15.326 INFO:tasks.rados.rados.2.smithi175.stdout:3568: done (1 left) 2024-09-11T03:51:15.326 INFO:tasks.rados.rados.2.smithi175.stdout:3570: snap_create 2024-09-11T03:51:15.332 INFO:tasks.rados.rados.2.smithi175.stdout:3569: finishing copy_from racing read to smithi17557870-4 2024-09-11T03:51:15.338 INFO:tasks.rados.rados.2.smithi175.stdout:3569: finishing copy_from to smithi17557870-4 2024-09-11T03:51:15.338 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 980 (ObjNum 1148 snap 371 seq_num 1148) dirty exists 2024-09-11T03:51:15.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:15.365 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:15.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:15.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:15.467 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:15 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:51:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:15 smithi175 ceph-mon[30894]: 2.47 scrub starts 2024-09-11T03:51:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:15 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/875214882' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:15 smithi175 ceph-mon[30894]: 2.48 scrub starts 2024-09-11T03:51:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:15 smithi175 ceph-mon[30894]: 2.49 scrub starts 2024-09-11T03:51:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:15 smithi175 ceph-mon[30894]: 2.42 scrub starts 2024-09-11T03:51:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:15 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:51:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:15 smithi175 ceph-mon[30894]: 2.4a scrub starts 2024-09-11T03:51:15.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:15 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:51:15.488 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:15.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:15.489 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:15.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:15 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:51:15.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:15 smithi084 ceph-mon[27254]: 2.47 scrub starts 2024-09-11T03:51:15.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:15 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/875214882' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:15.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:15 smithi084 ceph-mon[27254]: 2.48 scrub starts 2024-09-11T03:51:15.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:15 smithi084 ceph-mon[27254]: 2.49 scrub starts 2024-09-11T03:51:15.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:15 smithi084 ceph-mon[27254]: 2.42 scrub starts 2024-09-11T03:51:15.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:15 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:51:15.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:15 smithi084 ceph-mon[27254]: 2.4a scrub starts 2024-09-11T03:51:15.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:15 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:51:15.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:15.615 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:15.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:15 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:51:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:15 smithi121 ceph-mon[30729]: 2.47 scrub starts 2024-09-11T03:51:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:15 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/875214882' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:15 smithi121 ceph-mon[30729]: 2.48 scrub starts 2024-09-11T03:51:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:15 smithi121 ceph-mon[30729]: 2.49 scrub starts 2024-09-11T03:51:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:15 smithi121 ceph-mon[30729]: 2.42 scrub starts 2024-09-11T03:51:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:15 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:51:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:15 smithi121 ceph-mon[30729]: 2.4a scrub starts 2024-09-11T03:51:15.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:15 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:51:15.717 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:15.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:15.740 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:15.740 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:15.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:15.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:15.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:15.969 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:15.990 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:15.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:15.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:16.092 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:16.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:16.114 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:16.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:16.138 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:51:16.139 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:51:16.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:15 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a[27250]: 2024-09-11T03:51:15.812+0000 7fae0fb7d640 -1 mon.a@0(leader).osd e857 definitely_dead 0 2024-09-11T03:51:16.143 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:16.202 INFO:teuthology.orchestra.run.smithi084.stderr:marked out osd.4. 2024-09-11T03:51:16.205 INFO:tasks.rados.rados.2.smithi175.stdout:3569: done (1 left) 2024-09-11T03:51:16.205 INFO:tasks.rados.rados.2.smithi175.stdout:3570: done (0 left) 2024-09-11T03:51:16.205 INFO:tasks.rados.rados.2.smithi175.stdout:3571: read oid 9 snap 367 2024-09-11T03:51:16.205 INFO:tasks.rados.rados.2.smithi175.stdout:3571: expect deleted 2024-09-11T03:51:16.205 INFO:tasks.rados.rados.2.smithi175.stdout:3572: snap_remove snap 381 2024-09-11T03:51:16.215 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:16.236 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:16.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:16.237 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:16.338 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:16.359 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:16.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:16.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:16.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:16 smithi084 ceph-mon[27254]: 2.45 scrub starts 2024-09-11T03:51:16.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:16 smithi084 ceph-mon[27254]: pgmap v1440: 107 pgs: 1 peering, 6 active+undersized+remapped, 3 active+clean+scrubbing, 1 remapped+peering, 1 unknown, 95 active+clean; 695 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 2.2 MiB/s wr, 17 op/s; 16 MiB/s, 34 objects/s recovering 2024-09-11T03:51:16.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:16 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:51:16.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:16 smithi084 ceph-mon[27254]: osdmap e857: 12 total, 12 up, 12 in 2024-09-11T03:51:16.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:16 smithi084 ceph-mon[27254]: 2.4b scrub starts 2024-09-11T03:51:16.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:16 smithi084 ceph-mon[27254]: 2.48 scrub starts 2024-09-11T03:51:16.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:16 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:51:16.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:16 smithi084 ceph-mon[27254]: 2.42 scrub starts 2024-09-11T03:51:16.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:16 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:51:16.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:16 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3634184663' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-11T03:51:16.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:16 smithi084 ceph-mon[27254]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-11T03:51:16.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:16 smithi084 ceph-mon[27254]: 2.4a scrub starts 2024-09-11T03:51:16.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:16 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:51:16.461 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:16 smithi175 ceph-mon[30894]: 2.45 scrub starts 2024-09-11T03:51:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:16 smithi175 ceph-mon[30894]: pgmap v1440: 107 pgs: 1 peering, 6 active+undersized+remapped, 3 active+clean+scrubbing, 1 remapped+peering, 1 unknown, 95 active+clean; 695 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 2.2 MiB/s wr, 17 op/s; 16 MiB/s, 34 objects/s recovering 2024-09-11T03:51:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:16 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:51:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:16 smithi175 ceph-mon[30894]: osdmap e857: 12 total, 12 up, 12 in 2024-09-11T03:51:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:16 smithi175 ceph-mon[30894]: 2.4b scrub starts 2024-09-11T03:51:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:16 smithi175 ceph-mon[30894]: 2.48 scrub starts 2024-09-11T03:51:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:16 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:51:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:16 smithi175 ceph-mon[30894]: 2.42 scrub starts 2024-09-11T03:51:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:16 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:51:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:16 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3634184663' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-11T03:51:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:16 smithi175 ceph-mon[30894]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-11T03:51:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:16 smithi175 ceph-mon[30894]: 2.4a scrub starts 2024-09-11T03:51:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:16 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:51:16.483 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:16.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:16.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:16.585 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:16.606 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:16.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:16.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:16.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:16 smithi121 ceph-mon[30729]: 2.45 scrub starts 2024-09-11T03:51:16.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:16 smithi121 ceph-mon[30729]: pgmap v1440: 107 pgs: 1 peering, 6 active+undersized+remapped, 3 active+clean+scrubbing, 1 remapped+peering, 1 unknown, 95 active+clean; 695 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 2.2 MiB/s wr, 17 op/s; 16 MiB/s, 34 objects/s recovering 2024-09-11T03:51:16.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:16 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:51:16.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:16 smithi121 ceph-mon[30729]: osdmap e857: 12 total, 12 up, 12 in 2024-09-11T03:51:16.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:16 smithi121 ceph-mon[30729]: 2.4b scrub starts 2024-09-11T03:51:16.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:16 smithi121 ceph-mon[30729]: 2.48 scrub starts 2024-09-11T03:51:16.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:16 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:51:16.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:16 smithi121 ceph-mon[30729]: 2.42 scrub starts 2024-09-11T03:51:16.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:16 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:51:16.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:16 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3634184663' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-11T03:51:16.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:16 smithi121 ceph-mon[30729]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-11T03:51:16.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:16 smithi121 ceph-mon[30729]: 2.4a scrub starts 2024-09-11T03:51:16.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:16 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:51:16.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:16.734 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:16.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:16.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:16.836 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:16.859 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:16.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:16.860 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:16.961 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:16.983 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:16.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:16.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:17.050 INFO:tasks.rados.rados.2.smithi175.stdout:3572: done (1 left) 2024-09-11T03:51:17.050 INFO:tasks.rados.rados.2.smithi175.stdout:3573: snap_remove snap 341 2024-09-11T03:51:17.085 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:17.107 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:17.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:17.109 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:17.210 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:17.230 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:17.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:17.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:17.361 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:17.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:17.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:17.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:17 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:51:17.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:17 smithi084 ceph-mon[27254]: 2.47 scrub starts 2024-09-11T03:51:17.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3634184663' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-11T03:51:17.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:17 smithi084 ceph-mon[27254]: osdmap e858: 12 total, 12 up, 11 in 2024-09-11T03:51:17.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:17 smithi084 ceph-mon[27254]: 2.48 scrub starts 2024-09-11T03:51:17.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:17 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:51:17.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:17 smithi084 ceph-mon[27254]: 2.42 scrub starts 2024-09-11T03:51:17.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:17 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:51:17.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:17 smithi084 ceph-mon[27254]: 2.4a scrub starts 2024-09-11T03:51:17.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:17 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:51:17.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:17 smithi084 ceph-mon[27254]: osdmap e859: 12 total, 12 up, 11 in 2024-09-11T03:51:17.443 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:51:17.463 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:17 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:51:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:17 smithi175 ceph-mon[30894]: 2.47 scrub starts 2024-09-11T03:51:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3634184663' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-11T03:51:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:17 smithi175 ceph-mon[30894]: osdmap e858: 12 total, 12 up, 11 in 2024-09-11T03:51:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:17 smithi175 ceph-mon[30894]: 2.48 scrub starts 2024-09-11T03:51:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:17 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:51:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:17 smithi175 ceph-mon[30894]: 2.42 scrub starts 2024-09-11T03:51:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:17 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:51:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:17 smithi175 ceph-mon[30894]: 2.4a scrub starts 2024-09-11T03:51:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:17 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:51:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:17 smithi175 ceph-mon[30894]: osdmap e859: 12 total, 12 up, 11 in 2024-09-11T03:51:17.508 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:17.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:17.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:17.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T03:51:17.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:17.635 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:17.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:17.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:17.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:17 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:51:17.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:17 smithi121 ceph-mon[30729]: 2.47 scrub starts 2024-09-11T03:51:17.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3634184663' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-11T03:51:17.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:17 smithi121 ceph-mon[30729]: osdmap e858: 12 total, 12 up, 11 in 2024-09-11T03:51:17.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:17 smithi121 ceph-mon[30729]: 2.48 scrub starts 2024-09-11T03:51:17.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:17 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:51:17.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:17 smithi121 ceph-mon[30729]: 2.42 scrub starts 2024-09-11T03:51:17.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:17 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:51:17.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:17 smithi121 ceph-mon[30729]: 2.4a scrub starts 2024-09-11T03:51:17.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:17 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:51:17.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:17 smithi121 ceph-mon[30729]: osdmap e859: 12 total, 12 up, 11 in 2024-09-11T03:51:17.738 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:17.764 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:17.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:17.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:17.866 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:17.890 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:17.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:17.891 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:17.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:18.014 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:18.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:18.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:18.025 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:18.052 INFO:tasks.rados.rados.2.smithi175.stdout:3573: done (1 left) 2024-09-11T03:51:18.052 INFO:tasks.rados.rados.2.smithi175.stdout:3574: read oid 5 snap 331 2024-09-11T03:51:18.052 INFO:tasks.rados.rados.2.smithi175.stdout:3574: expect (ObjNum 758 snap 246 seq_num 758) 2024-09-11T03:51:18.052 INFO:tasks.rados.rados.2.smithi175.stdout:3575: snap_create 2024-09-11T03:51:18.070 INFO:tasks.rados.rados.2.smithi175.stdout:3574: expect (ObjNum 758 snap 246 seq_num 758) 2024-09-11T03:51:18.116 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:18.141 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:18.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:18.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:18.243 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:18.264 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:18.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:18.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:18.366 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:18.387 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:18.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:18.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:18.436 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:18 smithi084 ceph-mon[27254]: 2.4b scrub starts 2024-09-11T03:51:18.436 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:18 smithi084 ceph-mon[27254]: pgmap v1443: 107 pgs: 1 unknown, 12 peering, 2 active+clean+scrubbing, 92 active+clean; 706 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 18 MiB/s rd, 3.9 MiB/s wr, 32 op/s 2024-09-11T03:51:18.436 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:18 smithi084 ceph-mon[27254]: 2.43 scrub starts 2024-09-11T03:51:18.436 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:18 smithi084 ceph-mon[27254]: 2.48 scrub starts 2024-09-11T03:51:18.437 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:18 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:51:18.437 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:18 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:51:18.437 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:18 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:51:18.437 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:18 smithi084 ceph-mon[27254]: 2.4a scrub starts 2024-09-11T03:51:18.437 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:18 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:51:18.437 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:18 smithi084 ceph-mon[27254]: osdmap e860: 12 total, 12 up, 11 in 2024-09-11T03:51:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:18 smithi175 ceph-mon[30894]: 2.4b scrub starts 2024-09-11T03:51:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:18 smithi175 ceph-mon[30894]: pgmap v1443: 107 pgs: 1 unknown, 12 peering, 2 active+clean+scrubbing, 92 active+clean; 706 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 18 MiB/s rd, 3.9 MiB/s wr, 32 op/s 2024-09-11T03:51:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:18 smithi175 ceph-mon[30894]: 2.43 scrub starts 2024-09-11T03:51:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:18 smithi175 ceph-mon[30894]: 2.48 scrub starts 2024-09-11T03:51:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:18 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:51:18.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:18 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:51:18.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:18 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:51:18.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:18 smithi175 ceph-mon[30894]: 2.4a scrub starts 2024-09-11T03:51:18.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:18 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:51:18.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:18 smithi175 ceph-mon[30894]: osdmap e860: 12 total, 12 up, 11 in 2024-09-11T03:51:18.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:18.510 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:18.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:18.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:18.544 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:51:18.545 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:51:18.551 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:51:18.551 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:51:18.567 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:51:18.567 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:51:18.582 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:51:18.583 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:51:18.600 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:51:18.600 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:51:18.611 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:18.616 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:51:18.616 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:51:18.627 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:18.633 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:18.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:18.634 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:18.635 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:51:18.635 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:51:18.651 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:51:18.651 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:51:18.669 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:51:18.669 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:51:18.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:18 smithi121 ceph-mon[30729]: 2.4b scrub starts 2024-09-11T03:51:18.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:18 smithi121 ceph-mon[30729]: pgmap v1443: 107 pgs: 1 unknown, 12 peering, 2 active+clean+scrubbing, 92 active+clean; 706 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 18 MiB/s rd, 3.9 MiB/s wr, 32 op/s 2024-09-11T03:51:18.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:18 smithi121 ceph-mon[30729]: 2.43 scrub starts 2024-09-11T03:51:18.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:18 smithi121 ceph-mon[30729]: 2.48 scrub starts 2024-09-11T03:51:18.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:18 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:51:18.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:18 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:51:18.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:18 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:51:18.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:18 smithi121 ceph-mon[30729]: 2.4a scrub starts 2024-09-11T03:51:18.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:18 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:51:18.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:18 smithi121 ceph-mon[30729]: osdmap e860: 12 total, 12 up, 11 in 2024-09-11T03:51:18.685 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:51:18.685 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:51:18.702 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:51:18.702 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:51:18.718 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:51:18.719 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:51:18.735 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:18.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:18.758 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:18.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:18.859 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:18.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:18.882 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:18.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:18.984 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:19.004 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:19.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:19.005 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:19.049 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:51:19.052 INFO:tasks.rados.rados.2.smithi175.stdout:3571: done (2 left) 2024-09-11T03:51:19.052 INFO:tasks.rados.rados.2.smithi175.stdout:3574: done (1 left) 2024-09-11T03:51:19.052 INFO:tasks.rados.rados.2.smithi175.stdout:3575: done (0 left) 2024-09-11T03:51:19.052 INFO:tasks.rados.rados.2.smithi175.stdout:3576: delete oid 34 current snap is 399 2024-09-11T03:51:19.059 INFO:tasks.rados.rados.2.smithi175.stdout:3576: done (0 left) 2024-09-11T03:51:19.059 INFO:tasks.rados.rados.2.smithi175.stdout:3577: write (excl) oid 38 current snap is 399 2024-09-11T03:51:19.059 INFO:tasks.rados.rados.2.smithi175.stdout:3577: seq_num 1217 ranges {454119=594513,1515880=556760} 2024-09-11T03:51:19.064 INFO:tasks.rados.rados.2.smithi175.stdout:3577: writing smithi17557870-38 from 454119 to 1048632 tid 1 2024-09-11T03:51:19.071 INFO:tasks.rados.rados.2.smithi175.stdout:3577: writing smithi17557870-38 from 1515880 to 2072640 tid 2 2024-09-11T03:51:19.071 INFO:tasks.rados.rados.2.smithi175.stdout:3578: write (excl) oid 16 current snap is 399 2024-09-11T03:51:19.071 INFO:tasks.rados.rados.2.smithi175.stdout:3577: finishing write tid 1 to smithi17557870-38 2024-09-11T03:51:19.071 INFO:tasks.rados.rados.2.smithi175.stdout:3578: seq_num 1218 ranges {470926=669762,1574362=700418} 2024-09-11T03:51:19.076 INFO:tasks.rados.rados.2.smithi175.stdout:3578: writing smithi17557870-16 from 470926 to 1140688 tid 1 2024-09-11T03:51:19.082 INFO:tasks.rados.rados.2.smithi175.stdout:3578: writing smithi17557870-16 from 1574362 to 2274780 tid 2 2024-09-11T03:51:19.082 INFO:tasks.rados.rados.2.smithi175.stdout:3579: snap_remove snap 331 2024-09-11T03:51:19.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:19.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:19.130 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:19.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:19.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:19.254 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:19.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:19.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:19.356 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:19.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:19.381 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:19.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:19.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:19 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:51:19.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:19 smithi084 ceph-mon[27254]: 2.46 scrub starts 2024-09-11T03:51:19.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:19 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1036565661' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:51:19.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:19 smithi084 ceph-mon[27254]: 2.4b scrub starts 2024-09-11T03:51:19.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:19 smithi084 ceph-mon[27254]: 2.48 scrub starts 2024-09-11T03:51:19.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:19 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:51:19.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:19 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2827052679' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:19.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:19 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:51:19.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:19 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:51:19.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:19 smithi084 ceph-mon[27254]: 2.4a scrub starts 2024-09-11T03:51:19.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:19 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:51:19.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:19 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:19.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:19 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1036565661' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:51:19.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:19 smithi084 ceph-mon[27254]: osdmap e861: 12 total, 12 up, 11 in 2024-09-11T03:51:19.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:19 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:51:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:19 smithi121 ceph-mon[30729]: 2.46 scrub starts 2024-09-11T03:51:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:19 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1036565661' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:51:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:19 smithi121 ceph-mon[30729]: 2.4b scrub starts 2024-09-11T03:51:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:19 smithi121 ceph-mon[30729]: 2.48 scrub starts 2024-09-11T03:51:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:19 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:51:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:19 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2827052679' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:19 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:51:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:19 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:51:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:19 smithi121 ceph-mon[30729]: 2.4a scrub starts 2024-09-11T03:51:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:19 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:51:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:19 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:19 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1036565661' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:51:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:19 smithi121 ceph-mon[30729]: osdmap e861: 12 total, 12 up, 11 in 2024-09-11T03:51:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:19 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:51:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:19 smithi175 ceph-mon[30894]: 2.46 scrub starts 2024-09-11T03:51:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:19 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1036565661' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:51:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:19 smithi175 ceph-mon[30894]: 2.4b scrub starts 2024-09-11T03:51:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:19 smithi175 ceph-mon[30894]: 2.48 scrub starts 2024-09-11T03:51:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:19 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:51:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:19 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2827052679' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:19 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:51:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:19 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:51:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:19 smithi175 ceph-mon[30894]: 2.4a scrub starts 2024-09-11T03:51:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:19 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:51:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:19 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:19 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1036565661' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:51:19.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:19 smithi175 ceph-mon[30894]: osdmap e861: 12 total, 12 up, 11 in 2024-09-11T03:51:19.482 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:19.502 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:19.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:19.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:19.567 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:51:19.567 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:51:19.568 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:51:19.568 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:51:19.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:19.626 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:19.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:19.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:19.728 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:19.751 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:19.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:19.752 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:19.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:19.874 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:19.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:19.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:19.976 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:19.999 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:20.000 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:20.057 INFO:tasks.rados.rados.2.smithi175.stdout:3579: done (2 left) 2024-09-11T03:51:20.057 INFO:tasks.rados.rados.2.smithi175.stdout:3580: read oid 4 snap -1 2024-09-11T03:51:20.057 INFO:tasks.rados.rados.2.smithi175.stdout:3580: expect (ObjNum 1148 snap 371 seq_num 1148) 2024-09-11T03:51:20.057 INFO:tasks.rados.rados.2.smithi175.stdout:3581: rollback oid 5 current snap is 399 2024-09-11T03:51:20.057 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 5 to 314 2024-09-11T03:51:20.057 INFO:tasks.rados.rados.2.smithi175.stdout:3577: finishing write tid 2 to smithi17557870-38 2024-09-11T03:51:20.057 INFO:tasks.rados.rados.2.smithi175.stdout:3577: finishing write tid 3 to smithi17557870-38 2024-09-11T03:51:20.057 INFO:tasks.rados.rados.2.smithi175.stdout:3577: finishing write tid 4 to smithi17557870-38 2024-09-11T03:51:20.057 INFO:tasks.rados.rados.2.smithi175.stdout:3577: oid 38 updating version 0 to 1411 2024-09-11T03:51:20.057 INFO:tasks.rados.rados.2.smithi175.stdout:3577: oid 38 version 1411 is already newer than 1410 2024-09-11T03:51:20.057 INFO:tasks.rados.rados.2.smithi175.stdout:3577: oid 38 version 1411 is already newer than 1409 2024-09-11T03:51:20.058 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 1411 (ObjNum 1217 snap 399 seq_num 1217) dirty exists 2024-09-11T03:51:20.058 INFO:tasks.rados.rados.2.smithi175.stdout:3577: left oid 38 (ObjNum 1217 snap 399 seq_num 1217) 2024-09-11T03:51:20.058 INFO:tasks.rados.rados.2.smithi175.stdout:3578: finishing write tid 1 to smithi17557870-16 2024-09-11T03:51:20.058 INFO:tasks.rados.rados.2.smithi175.stdout:3578: finishing write tid 2 to smithi17557870-16 2024-09-11T03:51:20.058 INFO:tasks.rados.rados.2.smithi175.stdout:3578: finishing write tid 3 to smithi17557870-16 2024-09-11T03:51:20.058 INFO:tasks.rados.rados.2.smithi175.stdout:3578: finishing write tid 4 to smithi17557870-16 2024-09-11T03:51:20.058 INFO:tasks.rados.rados.2.smithi175.stdout:3578: oid 16 updating version 0 to 1003 2024-09-11T03:51:20.058 INFO:tasks.rados.rados.2.smithi175.stdout:3578: oid 16 version 1003 is already newer than 1001 2024-09-11T03:51:20.058 INFO:tasks.rados.rados.2.smithi175.stdout:3578: oid 16 version 1003 is already newer than 1002 2024-09-11T03:51:20.058 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 1003 (ObjNum 1218 snap 399 seq_num 1218) dirty exists 2024-09-11T03:51:20.058 INFO:tasks.rados.rados.2.smithi175.stdout:3578: left oid 16 (ObjNum 1218 snap 399 seq_num 1218) 2024-09-11T03:51:20.058 INFO:tasks.rados.rados.2.smithi175.stdout:3577: done (3 left) 2024-09-11T03:51:20.058 INFO:tasks.rados.rados.2.smithi175.stdout:3578: done (2 left) 2024-09-11T03:51:20.059 INFO:tasks.rados.rados.2.smithi175.stdout:3582: snap_remove snap 367 2024-09-11T03:51:20.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:20.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:20.124 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:20.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:20.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:20.246 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:20.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:20.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:20.301 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T03:51:20.348 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:20.370 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:20.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:20.371 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:20.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:20 smithi175 ceph-mon[30894]: pgmap v1446: 107 pgs: 4 active+undersized+remapped, 3 remapped+peering, 1 unknown, 11 peering, 2 active+clean+scrubbing, 86 active+clean; 716 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 9.3 MiB/s rd, 3.8 MiB/s wr, 27 op/s; 1.9 MiB/s, 6 objects/s recovering 2024-09-11T03:51:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:20 smithi175 ceph-mon[30894]: 2.43 scrub starts 2024-09-11T03:51:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:20 smithi175 ceph-mon[30894]: 2.48 scrub starts 2024-09-11T03:51:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:20 smithi175 ceph-mon[30894]: 2.48 scrub ok 2024-09-11T03:51:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:20 smithi175 ceph-mon[30894]: 2.2 scrub starts 2024-09-11T03:51:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:20 smithi175 ceph-mon[30894]: 2.2 scrub ok 2024-09-11T03:51:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:20 smithi175 ceph-mon[30894]: 2.42 scrub starts 2024-09-11T03:51:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:20 smithi175 ceph-mon[30894]: 2.42 scrub ok 2024-09-11T03:51:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:20 smithi175 ceph-mon[30894]: 2.7 scrub starts 2024-09-11T03:51:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:20 smithi175 ceph-mon[30894]: 2.7 scrub ok 2024-09-11T03:51:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:20 smithi175 ceph-mon[30894]: 2.4a scrub starts 2024-09-11T03:51:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:20 smithi175 ceph-mon[30894]: 2.4a scrub ok 2024-09-11T03:51:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:20 smithi175 ceph-mon[30894]: 2.a scrub starts 2024-09-11T03:51:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:20 smithi175 ceph-mon[30894]: 2.a scrub ok 2024-09-11T03:51:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:20 smithi175 ceph-mon[30894]: osdmap e862: 12 total, 12 up, 11 in 2024-09-11T03:51:20.493 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:20.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:20.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:20.595 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:20.617 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:20.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:20.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:20 smithi084 ceph-mon[27254]: pgmap v1446: 107 pgs: 4 active+undersized+remapped, 3 remapped+peering, 1 unknown, 11 peering, 2 active+clean+scrubbing, 86 active+clean; 716 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 9.3 MiB/s rd, 3.8 MiB/s wr, 27 op/s; 1.9 MiB/s, 6 objects/s recovering 2024-09-11T03:51:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:20 smithi084 ceph-mon[27254]: 2.43 scrub starts 2024-09-11T03:51:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:20 smithi084 ceph-mon[27254]: 2.48 scrub starts 2024-09-11T03:51:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:20 smithi084 ceph-mon[27254]: 2.48 scrub ok 2024-09-11T03:51:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:20 smithi084 ceph-mon[27254]: 2.2 scrub starts 2024-09-11T03:51:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:20 smithi084 ceph-mon[27254]: 2.2 scrub ok 2024-09-11T03:51:20.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:20 smithi084 ceph-mon[27254]: 2.42 scrub starts 2024-09-11T03:51:20.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:20 smithi084 ceph-mon[27254]: 2.42 scrub ok 2024-09-11T03:51:20.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:20 smithi084 ceph-mon[27254]: 2.7 scrub starts 2024-09-11T03:51:20.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:20 smithi084 ceph-mon[27254]: 2.7 scrub ok 2024-09-11T03:51:20.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:20 smithi084 ceph-mon[27254]: 2.4a scrub starts 2024-09-11T03:51:20.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:20 smithi084 ceph-mon[27254]: 2.4a scrub ok 2024-09-11T03:51:20.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:20 smithi084 ceph-mon[27254]: 2.a scrub starts 2024-09-11T03:51:20.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:20 smithi084 ceph-mon[27254]: 2.a scrub ok 2024-09-11T03:51:20.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:20 smithi084 ceph-mon[27254]: osdmap e862: 12 total, 12 up, 11 in 2024-09-11T03:51:20.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:20 smithi121 ceph-mon[30729]: pgmap v1446: 107 pgs: 4 active+undersized+remapped, 3 remapped+peering, 1 unknown, 11 peering, 2 active+clean+scrubbing, 86 active+clean; 716 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 9.3 MiB/s rd, 3.8 MiB/s wr, 27 op/s; 1.9 MiB/s, 6 objects/s recovering 2024-09-11T03:51:20.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:20 smithi121 ceph-mon[30729]: 2.43 scrub starts 2024-09-11T03:51:20.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:20 smithi121 ceph-mon[30729]: 2.48 scrub starts 2024-09-11T03:51:20.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:20 smithi121 ceph-mon[30729]: 2.48 scrub ok 2024-09-11T03:51:20.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:20 smithi121 ceph-mon[30729]: 2.2 scrub starts 2024-09-11T03:51:20.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:20 smithi121 ceph-mon[30729]: 2.2 scrub ok 2024-09-11T03:51:20.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:20 smithi121 ceph-mon[30729]: 2.42 scrub starts 2024-09-11T03:51:20.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:20 smithi121 ceph-mon[30729]: 2.42 scrub ok 2024-09-11T03:51:20.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:20 smithi121 ceph-mon[30729]: 2.7 scrub starts 2024-09-11T03:51:20.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:20 smithi121 ceph-mon[30729]: 2.7 scrub ok 2024-09-11T03:51:20.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:20 smithi121 ceph-mon[30729]: 2.4a scrub starts 2024-09-11T03:51:20.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:20 smithi121 ceph-mon[30729]: 2.4a scrub ok 2024-09-11T03:51:20.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:20 smithi121 ceph-mon[30729]: 2.a scrub starts 2024-09-11T03:51:20.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:20 smithi121 ceph-mon[30729]: 2.a scrub ok 2024-09-11T03:51:20.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:20 smithi121 ceph-mon[30729]: osdmap e862: 12 total, 12 up, 11 in 2024-09-11T03:51:20.719 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:20.741 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:20.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:20.742 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:20.843 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:20.865 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:20.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:20.866 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:20.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:20.990 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:20.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:20.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:21.057 INFO:tasks.rados.rados.2.smithi175.stdout:3582: done (2 left) 2024-09-11T03:51:21.057 INFO:tasks.rados.rados.2.smithi175.stdout:3583: read oid 40 snap -1 2024-09-11T03:51:21.057 INFO:tasks.rados.rados.2.smithi175.stdout:3583: expect (ObjNum 973 snap 313 seq_num 973) 2024-09-11T03:51:21.057 INFO:tasks.rados.rados.2.smithi175.stdout:3584: rollback oid 23 current snap is 399 2024-09-11T03:51:21.057 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 23 to 364 2024-09-11T03:51:21.057 INFO:tasks.rados.rados.2.smithi175.stdout:3585: read oid 1 snap -1 2024-09-11T03:51:21.057 INFO:tasks.rados.rados.2.smithi175.stdout:3585: expect (ObjNum 1150 snap 372 seq_num 1150) 2024-09-11T03:51:21.057 INFO:tasks.rados.rados.2.smithi175.stdout:3586: read oid 48 snap 388 2024-09-11T03:51:21.057 INFO:tasks.rados.rados.2.smithi175.stdout:3586: expect (ObjNum 1149 snap 371 seq_num 1149) 2024-09-11T03:51:21.058 INFO:tasks.rados.rados.2.smithi175.stdout:3587: read oid 45 snap 394 2024-09-11T03:51:21.058 INFO:tasks.rados.rados.2.smithi175.stdout:3587: expect deleted 2024-09-11T03:51:21.058 INFO:tasks.rados.rados.2.smithi175.stdout:3588: read oid 32 snap -1 2024-09-11T03:51:21.058 INFO:tasks.rados.rados.2.smithi175.stdout:3588: expect (ObjNum 1059 snap 345 seq_num 1059) 2024-09-11T03:51:21.058 INFO:tasks.rados.rados.2.smithi175.stdout:3589: delete oid 9 current snap is 399 2024-09-11T03:51:21.058 INFO:tasks.rados.rados.2.smithi175.stdout:3581: finishing rollback tid 0 to smithi17557870-5 2024-09-11T03:51:21.058 INFO:tasks.rados.rados.2.smithi175.stdout:3581: finishing rollback tid 1 to smithi17557870-5 2024-09-11T03:51:21.058 INFO:tasks.rados.rados.2.smithi175.stdout:3581: finishing rollback tid 2 to smithi17557870-5 2024-09-11T03:51:21.058 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 1044 (ObjNum 758 snap 246 seq_num 758) dirty exists 2024-09-11T03:51:21.058 INFO:tasks.rados.rados.2.smithi175.stdout:3580: expect (ObjNum 1148 snap 371 seq_num 1148) 2024-09-11T03:51:21.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:21.113 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:21.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:21.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:21.172 INFO:tasks.rados.rados.2.smithi175.stdout:3583: expect (ObjNum 973 snap 313 seq_num 973) 2024-09-11T03:51:21.211 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:21.215 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:21.235 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:21.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:21.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:21.248 INFO:tasks.rados.rados.2.smithi175.stdout:3585: expect (ObjNum 1150 snap 372 seq_num 1150) 2024-09-11T03:51:21.338 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:21.352 INFO:tasks.rados.rados.2.smithi175.stdout:3584: finishing rollback tid 0 to smithi17557870-23 2024-09-11T03:51:21.352 INFO:tasks.rados.rados.2.smithi175.stdout:3584: finishing rollback tid 1 to smithi17557870-23 2024-09-11T03:51:21.352 INFO:tasks.rados.rados.2.smithi175.stdout:3584: finishing rollback tid 2 to smithi17557870-23 2024-09-11T03:51:21.352 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 989 (ObjNum 1079 snap 352 seq_num 1079) dirty exists 2024-09-11T03:51:21.353 INFO:tasks.rados.rados.2.smithi175.stdout:3586: expect (ObjNum 1149 snap 371 seq_num 1149) 2024-09-11T03:51:21.358 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:21.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:21.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:21.460 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:21.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:21 smithi084 ceph-mon[27254]: 2.9 scrub starts 2024-09-11T03:51:21.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:21 smithi084 ceph-mon[27254]: 2.9 scrub ok 2024-09-11T03:51:21.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:21 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:51:21.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:21 smithi084 ceph-mon[27254]: 2.8 scrub ok 2024-09-11T03:51:21.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:21 smithi084 ceph-mon[27254]: 2.47 scrub starts 2024-09-11T03:51:21.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:21 smithi084 ceph-mon[27254]: 2.47 scrub ok 2024-09-11T03:51:21.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:21 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:51:21.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:21 smithi084 ceph-mon[27254]: 2.15 scrub ok 2024-09-11T03:51:21.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:21 smithi084 ceph-mon[27254]: 2.44 scrub starts 2024-09-11T03:51:21.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:21 smithi084 ceph-mon[27254]: 2.3 scrub starts 2024-09-11T03:51:21.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:21 smithi084 ceph-mon[27254]: 2.3 scrub ok 2024-09-11T03:51:21.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:21 smithi084 ceph-mon[27254]: 2.b scrub starts 2024-09-11T03:51:21.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:21 smithi084 ceph-mon[27254]: 2.b scrub ok 2024-09-11T03:51:21.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:21 smithi084 ceph-mon[27254]: osdmap e863: 12 total, 12 up, 11 in 2024-09-11T03:51:21.462 INFO:tasks.rados.rados.2.smithi175.stdout:3588: expect (ObjNum 1059 snap 345 seq_num 1059) 2024-09-11T03:51:21.482 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:21.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:21.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:21.578 INFO:tasks.rados.rados.2.smithi175.stdout:3580: done (8 left) 2024-09-11T03:51:21.578 INFO:tasks.rados.rados.2.smithi175.stdout:3581: done (7 left) 2024-09-11T03:51:21.578 INFO:tasks.rados.rados.2.smithi175.stdout:3583: done (6 left) 2024-09-11T03:51:21.578 INFO:tasks.rados.rados.2.smithi175.stdout:3584: done (5 left) 2024-09-11T03:51:21.579 INFO:tasks.rados.rados.2.smithi175.stdout:3585: done (4 left) 2024-09-11T03:51:21.579 INFO:tasks.rados.rados.2.smithi175.stdout:3586: done (3 left) 2024-09-11T03:51:21.579 INFO:tasks.rados.rados.2.smithi175.stdout:3587: done (2 left) 2024-09-11T03:51:21.579 INFO:tasks.rados.rados.2.smithi175.stdout:3588: done (1 left) 2024-09-11T03:51:21.579 INFO:tasks.rados.rados.2.smithi175.stdout:3589: done (0 left) 2024-09-11T03:51:21.579 INFO:tasks.rados.rados.2.smithi175.stdout:3590: snap_remove snap 397 2024-09-11T03:51:21.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:21.605 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:21.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:21.656 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:51:21.656 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:51:21.656 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:51:21.656 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:51:21.656 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:51:21.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:21 smithi121 ceph-mon[30729]: 2.9 scrub starts 2024-09-11T03:51:21.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:21 smithi121 ceph-mon[30729]: 2.9 scrub ok 2024-09-11T03:51:21.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:21 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:51:21.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:21 smithi121 ceph-mon[30729]: 2.8 scrub ok 2024-09-11T03:51:21.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:21 smithi121 ceph-mon[30729]: 2.47 scrub starts 2024-09-11T03:51:21.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:21 smithi121 ceph-mon[30729]: 2.47 scrub ok 2024-09-11T03:51:21.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:21 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:51:21.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:21 smithi121 ceph-mon[30729]: 2.15 scrub ok 2024-09-11T03:51:21.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:21 smithi121 ceph-mon[30729]: 2.44 scrub starts 2024-09-11T03:51:21.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:21 smithi121 ceph-mon[30729]: 2.3 scrub starts 2024-09-11T03:51:21.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:21 smithi121 ceph-mon[30729]: 2.3 scrub ok 2024-09-11T03:51:21.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:21 smithi121 ceph-mon[30729]: 2.b scrub starts 2024-09-11T03:51:21.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:21 smithi121 ceph-mon[30729]: 2.b scrub ok 2024-09-11T03:51:21.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:21 smithi121 ceph-mon[30729]: osdmap e863: 12 total, 12 up, 11 in 2024-09-11T03:51:21.706 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:21.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:21 smithi175 ceph-mon[30894]: 2.9 scrub starts 2024-09-11T03:51:21.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:21 smithi175 ceph-mon[30894]: 2.9 scrub ok 2024-09-11T03:51:21.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:21 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:51:21.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:21 smithi175 ceph-mon[30894]: 2.8 scrub ok 2024-09-11T03:51:21.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:21 smithi175 ceph-mon[30894]: 2.47 scrub starts 2024-09-11T03:51:21.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:21 smithi175 ceph-mon[30894]: 2.47 scrub ok 2024-09-11T03:51:21.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:21 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:51:21.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:21 smithi175 ceph-mon[30894]: 2.15 scrub ok 2024-09-11T03:51:21.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:21 smithi175 ceph-mon[30894]: 2.44 scrub starts 2024-09-11T03:51:21.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:21 smithi175 ceph-mon[30894]: 2.3 scrub starts 2024-09-11T03:51:21.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:21 smithi175 ceph-mon[30894]: 2.3 scrub ok 2024-09-11T03:51:21.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:21 smithi175 ceph-mon[30894]: 2.b scrub starts 2024-09-11T03:51:21.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:21 smithi175 ceph-mon[30894]: 2.b scrub ok 2024-09-11T03:51:21.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:21 smithi175 ceph-mon[30894]: osdmap e863: 12 total, 12 up, 11 in 2024-09-11T03:51:21.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:21.729 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:21.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:21.830 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:21.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:21.853 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:21.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:21.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:21.978 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:21.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:22.079 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:22.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:22.101 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:22.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:22.121 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 8, 9, 11, 10, 6] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:51:22.121 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:51:22.121 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-11T03:51:22.121 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T03:51:22.121 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T03:51:22.122 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-11T03:51:22.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:22.224 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:22.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:22.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:22.235 INFO:tasks.rados.rados.2.smithi175.stdout:3590: done (0 left) 2024-09-11T03:51:22.235 INFO:tasks.rados.rados.2.smithi175.stdout:3591: copy_from oid 25 from oid 48 current snap is 399 2024-09-11T03:51:22.235 INFO:tasks.rados.rados.2.smithi175.stdout:3592: write oid 4 current snap is 399 2024-09-11T03:51:22.235 INFO:tasks.rados.rados.2.smithi175.stdout:3592: seq_num 1220 ranges {504540=420260,1574973=704087} 2024-09-11T03:51:22.240 INFO:tasks.rados.rados.2.smithi175.stdout:3592: writing smithi17557870-4 from 504540 to 924800 tid 1 2024-09-11T03:51:22.248 INFO:tasks.rados.rados.2.smithi175.stdout:3592: writing smithi17557870-4 from 1574973 to 2279060 tid 2 2024-09-11T03:51:22.249 INFO:tasks.rados.rados.2.smithi175.stdout:3593: delete oid 10 current snap is 399 2024-09-11T03:51:22.249 INFO:tasks.rados.rados.2.smithi175.stdout:3592: finishing write tid 1 to smithi17557870-4 2024-09-11T03:51:22.249 INFO:tasks.rados.rados.2.smithi175.stdout:3591: finishing copy_from racing read to smithi17557870-25 2024-09-11T03:51:22.253 INFO:tasks.rados.rados.2.smithi175.stdout:3592: finishing write tid 2 to smithi17557870-4 2024-09-11T03:51:22.253 INFO:tasks.rados.rados.2.smithi175.stdout:3591: finishing copy_from to smithi17557870-25 2024-09-11T03:51:22.253 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 1008 (ObjNum 1206 snap 397 seq_num 1206) dirty exists 2024-09-11T03:51:22.253 INFO:tasks.rados.rados.2.smithi175.stdout:3592: finishing write tid 3 to smithi17557870-4 2024-09-11T03:51:22.253 INFO:tasks.rados.rados.2.smithi175.stdout:3592: finishing write tid 4 to smithi17557870-4 2024-09-11T03:51:22.253 INFO:tasks.rados.rados.2.smithi175.stdout:3592: oid 4 updating version 0 to 1009 2024-09-11T03:51:22.253 INFO:tasks.rados.rados.2.smithi175.stdout:3592: oid 4 updating version 1009 to 1011 2024-09-11T03:51:22.254 INFO:tasks.rados.rados.2.smithi175.stdout:3592: oid 4 version 1011 is already newer than 1010 2024-09-11T03:51:22.254 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 1011 (ObjNum 1220 snap 399 seq_num 1220) dirty exists 2024-09-11T03:51:22.254 INFO:tasks.rados.rados.2.smithi175.stdout:3592: left oid 4 (ObjNum 1220 snap 399 seq_num 1220) 2024-09-11T03:51:22.254 INFO:tasks.rados.rados.2.smithi175.stdout:3591: done (2 left) 2024-09-11T03:51:22.254 INFO:tasks.rados.rados.2.smithi175.stdout:3592: done (1 left) 2024-09-11T03:51:22.254 INFO:tasks.rados.rados.2.smithi175.stdout:3593: done (0 left) 2024-09-11T03:51:22.254 INFO:tasks.rados.rados.2.smithi175.stdout:3594: write oid 27 current snap is 399 2024-09-11T03:51:22.254 INFO:tasks.rados.rados.2.smithi175.stdout:3594: seq_num 1222 ranges {538154=570758,1691937=707756} 2024-09-11T03:51:22.258 INFO:tasks.rados.rados.2.smithi175.stdout:3594: writing smithi17557870-27 from 538154 to 1108912 tid 1 2024-09-11T03:51:22.266 INFO:tasks.rados.rados.2.smithi175.stdout:3594: writing smithi17557870-27 from 1691937 to 2399693 tid 2 2024-09-11T03:51:22.266 INFO:tasks.rados.rados.2.smithi175.stdout:3595: write oid 39 current snap is 399 2024-09-11T03:51:22.266 INFO:tasks.rados.rados.2.smithi175.stdout:3594: finishing write tid 1 to smithi17557870-27 2024-09-11T03:51:22.267 INFO:tasks.rados.rados.2.smithi175.stdout:3595: seq_num 1223 ranges {554961=646007,1750419=451414} 2024-09-11T03:51:22.273 INFO:tasks.rados.rados.2.smithi175.stdout:3595: writing smithi17557870-39 from 554961 to 1200968 tid 1 2024-09-11T03:51:22.279 INFO:tasks.rados.rados.2.smithi175.stdout:3595: writing smithi17557870-39 from 1750419 to 2201833 tid 2 2024-09-11T03:51:22.279 INFO:tasks.rados.rados.2.smithi175.stdout:3596: copy_from oid 5 from oid 30 current snap is 399 2024-09-11T03:51:22.279 INFO:tasks.rados.rados.2.smithi175.stdout:3594: finishing write tid 2 to smithi17557870-27 2024-09-11T03:51:22.279 INFO:tasks.rados.rados.2.smithi175.stdout:3594: finishing write tid 3 to smithi17557870-27 2024-09-11T03:51:22.279 INFO:tasks.rados.rados.2.smithi175.stdout:3594: finishing write tid 4 to smithi17557870-27 2024-09-11T03:51:22.279 INFO:tasks.rados.rados.2.smithi175.stdout:3594: oid 27 updating version 0 to 1247 2024-09-11T03:51:22.279 INFO:tasks.rados.rados.2.smithi175.stdout:3594: oid 27 updating version 1247 to 1248 2024-09-11T03:51:22.280 INFO:tasks.rados.rados.2.smithi175.stdout:3594: oid 27 updating version 1248 to 1249 2024-09-11T03:51:22.280 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 1249 (ObjNum 1222 snap 399 seq_num 1222) dirty exists 2024-09-11T03:51:22.280 INFO:tasks.rados.rados.2.smithi175.stdout:3594: left oid 27 (ObjNum 1222 snap 399 seq_num 1222) 2024-09-11T03:51:22.280 INFO:tasks.rados.rados.2.smithi175.stdout:3595: finishing write tid 1 to smithi17557870-39 2024-09-11T03:51:22.280 INFO:tasks.rados.rados.2.smithi175.stdout:3594: done (2 left) 2024-09-11T03:51:22.280 INFO:tasks.rados.rados.2.smithi175.stdout:3597: read oid 28 snap -1 2024-09-11T03:51:22.280 INFO:tasks.rados.rados.2.smithi175.stdout:3597: expect (ObjNum 942 snap 297 seq_num 942) 2024-09-11T03:51:22.280 INFO:tasks.rados.rados.2.smithi175.stdout:3598: snap_remove snap 383 2024-09-11T03:51:22.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:22.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:22.348 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:22.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:22 smithi084 ceph-mon[27254]: 2.44 scrub ok 2024-09-11T03:51:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:22 smithi084 ceph-mon[27254]: pgmap v1449: 107 pgs: 4 active+undersized+remapped, 3 remapped+peering, 1 unknown, 11 peering, 2 active+clean+scrubbing, 86 active+clean; 716 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 1023 B/s rd, 1000 KiB/s wr, 6 op/s; 3.8 MiB/s, 11 objects/s recovering 2024-09-11T03:51:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:22 smithi084 ceph-mon[27254]: 2.46 scrub starts 2024-09-11T03:51:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:22 smithi084 ceph-mon[27254]: 2.46 scrub ok 2024-09-11T03:51:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:22 smithi084 ceph-mon[27254]: 2.49 scrub starts 2024-09-11T03:51:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:22 smithi084 ceph-mon[27254]: 2.49 scrub ok 2024-09-11T03:51:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:22 smithi084 ceph-mon[27254]: 2.5 scrub starts 2024-09-11T03:51:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:22 smithi084 ceph-mon[27254]: 2.5 scrub ok 2024-09-11T03:51:22.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:22 smithi084 ceph-mon[27254]: 1.f scrub starts 2024-09-11T03:51:22.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:22 smithi084 ceph-mon[27254]: 1.f scrub ok 2024-09-11T03:51:22.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:22 smithi084 ceph-mon[27254]: 2.30 scrub starts 2024-09-11T03:51:22.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:22 smithi084 ceph-mon[27254]: 2.30 scrub ok 2024-09-11T03:51:22.449 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:22.471 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:22.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:22.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:22.475 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T03:51:22.525 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:22.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:22.595 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:22.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:22.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:22 smithi121 ceph-mon[30729]: 2.44 scrub ok 2024-09-11T03:51:22.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:22 smithi121 ceph-mon[30729]: pgmap v1449: 107 pgs: 4 active+undersized+remapped, 3 remapped+peering, 1 unknown, 11 peering, 2 active+clean+scrubbing, 86 active+clean; 716 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 1023 B/s rd, 1000 KiB/s wr, 6 op/s; 3.8 MiB/s, 11 objects/s recovering 2024-09-11T03:51:22.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:22 smithi121 ceph-mon[30729]: 2.46 scrub starts 2024-09-11T03:51:22.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:22 smithi121 ceph-mon[30729]: 2.46 scrub ok 2024-09-11T03:51:22.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:22 smithi121 ceph-mon[30729]: 2.49 scrub starts 2024-09-11T03:51:22.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:22 smithi121 ceph-mon[30729]: 2.49 scrub ok 2024-09-11T03:51:22.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:22 smithi121 ceph-mon[30729]: 2.5 scrub starts 2024-09-11T03:51:22.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:22 smithi121 ceph-mon[30729]: 2.5 scrub ok 2024-09-11T03:51:22.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:22 smithi121 ceph-mon[30729]: 1.f scrub starts 2024-09-11T03:51:22.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:22 smithi121 ceph-mon[30729]: 1.f scrub ok 2024-09-11T03:51:22.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:22 smithi121 ceph-mon[30729]: 2.30 scrub starts 2024-09-11T03:51:22.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:22 smithi121 ceph-mon[30729]: 2.30 scrub ok 2024-09-11T03:51:22.697 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:22.719 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:22.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:22.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:22.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:22 smithi175 ceph-mon[30894]: 2.44 scrub ok 2024-09-11T03:51:22.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:22 smithi175 ceph-mon[30894]: pgmap v1449: 107 pgs: 4 active+undersized+remapped, 3 remapped+peering, 1 unknown, 11 peering, 2 active+clean+scrubbing, 86 active+clean; 716 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 1023 B/s rd, 1000 KiB/s wr, 6 op/s; 3.8 MiB/s, 11 objects/s recovering 2024-09-11T03:51:22.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:22 smithi175 ceph-mon[30894]: 2.46 scrub starts 2024-09-11T03:51:22.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:22 smithi175 ceph-mon[30894]: 2.46 scrub ok 2024-09-11T03:51:22.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:22 smithi175 ceph-mon[30894]: 2.49 scrub starts 2024-09-11T03:51:22.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:22 smithi175 ceph-mon[30894]: 2.49 scrub ok 2024-09-11T03:51:22.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:22 smithi175 ceph-mon[30894]: 2.5 scrub starts 2024-09-11T03:51:22.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:22 smithi175 ceph-mon[30894]: 2.5 scrub ok 2024-09-11T03:51:22.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:22 smithi175 ceph-mon[30894]: 1.f scrub starts 2024-09-11T03:51:22.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:22 smithi175 ceph-mon[30894]: 1.f scrub ok 2024-09-11T03:51:22.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:22 smithi175 ceph-mon[30894]: 2.30 scrub starts 2024-09-11T03:51:22.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:22 smithi175 ceph-mon[30894]: 2.30 scrub ok 2024-09-11T03:51:22.731 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:51:22.821 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:22.845 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:22.846 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:22.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:22.968 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:22.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:22.991 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:51:22.991 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "" 2024-09-11T03:51:22.991 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:51:22.999 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:23.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:23.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:23.093 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:23.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:23.195 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:23.197 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:51:23.220 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:23.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:23.221 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:23.247 INFO:tasks.rados.rados.2.smithi175.stdout:3598: done (3 left) 2024-09-11T03:51:23.248 INFO:tasks.rados.rados.2.smithi175.stdout:3599: copy_from oid 36 from oid 2 current snap is 399 2024-09-11T03:51:23.248 INFO:tasks.rados.rados.2.smithi175.stdout:3596: finishing copy_from to smithi17557870-5 2024-09-11T03:51:23.248 INFO:tasks.rados.rados.2.smithi175.stdout:3596: got expected ENOENT (src dne) 2024-09-11T03:51:23.248 INFO:tasks.rados.rados.2.smithi175.stdout:3596: finishing copy_from racing read to smithi17557870-5 2024-09-11T03:51:23.248 INFO:tasks.rados.rados.2.smithi175.stdout:3595: finishing write tid 2 to smithi17557870-39 2024-09-11T03:51:23.248 INFO:tasks.rados.rados.2.smithi175.stdout:3595: finishing write tid 3 to smithi17557870-39 2024-09-11T03:51:23.248 INFO:tasks.rados.rados.2.smithi175.stdout:3595: finishing write tid 4 to smithi17557870-39 2024-09-11T03:51:23.248 INFO:tasks.rados.rados.2.smithi175.stdout:3595: oid 39 updating version 0 to 1008 2024-09-11T03:51:23.248 INFO:tasks.rados.rados.2.smithi175.stdout:3595: oid 39 version 1008 is already newer than 1007 2024-09-11T03:51:23.248 INFO:tasks.rados.rados.2.smithi175.stdout:3595: oid 39 version 1008 is already newer than 1006 2024-09-11T03:51:23.248 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 1008 (ObjNum 1223 snap 399 seq_num 1223) dirty exists 2024-09-11T03:51:23.248 INFO:tasks.rados.rados.2.smithi175.stdout:3595: left oid 39 (ObjNum 1223 snap 399 seq_num 1223) 2024-09-11T03:51:23.248 INFO:tasks.rados.rados.2.smithi175.stdout:3597: expect (ObjNum 942 snap 297 seq_num 942) 2024-09-11T03:51:23.321 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:23.342 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:23.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:23.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:23.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:23 smithi084 ceph-mon[27254]: 2.4b scrub starts 2024-09-11T03:51:23.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:23 smithi084 ceph-mon[27254]: 2.4b scrub ok 2024-09-11T03:51:23.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:23 smithi084 ceph-mon[27254]: 2.43 scrub starts 2024-09-11T03:51:23.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:23 smithi084 ceph-mon[27254]: 2.43 scrub ok 2024-09-11T03:51:23.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:23 smithi084 ceph-mon[27254]: osdmap e864: 12 total, 12 up, 11 in 2024-09-11T03:51:23.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:23 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:51:23.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:23 smithi084 ceph-mon[27254]: 2.e scrub ok 2024-09-11T03:51:23.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:23 smithi084 ceph-mon[27254]: 2.34 scrub starts 2024-09-11T03:51:23.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:23 smithi084 ceph-mon[27254]: 2.34 scrub ok 2024-09-11T03:51:23.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:23 smithi084 ceph-mon[27254]: 2.37 scrub starts 2024-09-11T03:51:23.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:23 smithi084 ceph-mon[27254]: 2.37 scrub ok 2024-09-11T03:51:23.434 INFO:tasks.rados.rados.2.smithi175.stdout:3599: finishing copy_from racing read to smithi17557870-36 2024-09-11T03:51:23.434 INFO:tasks.rados.rados.2.smithi175.stdout:3599: finishing copy_from to smithi17557870-36 2024-09-11T03:51:23.435 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 1260 (ObjNum 1196 snap 391 seq_num 1196) dirty exists 2024-09-11T03:51:23.435 INFO:tasks.rados.rados.2.smithi175.stdout:3595: done (3 left) 2024-09-11T03:51:23.435 INFO:tasks.rados.rados.2.smithi175.stdout:3596: done (2 left) 2024-09-11T03:51:23.435 INFO:tasks.rados.rados.2.smithi175.stdout:3597: done (1 left) 2024-09-11T03:51:23.435 INFO:tasks.rados.rados.2.smithi175.stdout:3599: done (0 left) 2024-09-11T03:51:23.435 INFO:tasks.rados.rados.2.smithi175.stdout:3600: delete oid 40 current snap is 399 2024-09-11T03:51:23.436 INFO:tasks.rados.rados.2.smithi175.stdout:3600: done (0 left) 2024-09-11T03:51:23.436 INFO:tasks.rados.rados.2.smithi175.stdout:3601: snap_remove snap 366 2024-09-11T03:51:23.444 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:23.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:23.466 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:23.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:23.567 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:23.589 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:23.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:23.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:23.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:23 smithi121 ceph-mon[30729]: 2.4b scrub starts 2024-09-11T03:51:23.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:23 smithi121 ceph-mon[30729]: 2.4b scrub ok 2024-09-11T03:51:23.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:23 smithi121 ceph-mon[30729]: 2.43 scrub starts 2024-09-11T03:51:23.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:23 smithi121 ceph-mon[30729]: 2.43 scrub ok 2024-09-11T03:51:23.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:23 smithi121 ceph-mon[30729]: osdmap e864: 12 total, 12 up, 11 in 2024-09-11T03:51:23.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:23 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:51:23.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:23 smithi121 ceph-mon[30729]: 2.e scrub ok 2024-09-11T03:51:23.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:23 smithi121 ceph-mon[30729]: 2.34 scrub starts 2024-09-11T03:51:23.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:23 smithi121 ceph-mon[30729]: 2.34 scrub ok 2024-09-11T03:51:23.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:23 smithi121 ceph-mon[30729]: 2.37 scrub starts 2024-09-11T03:51:23.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:23 smithi121 ceph-mon[30729]: 2.37 scrub ok 2024-09-11T03:51:23.691 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:23.713 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:23.713 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:23.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:23 smithi175 ceph-mon[30894]: 2.4b scrub starts 2024-09-11T03:51:23.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:23 smithi175 ceph-mon[30894]: 2.4b scrub ok 2024-09-11T03:51:23.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:23 smithi175 ceph-mon[30894]: 2.43 scrub starts 2024-09-11T03:51:23.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:23 smithi175 ceph-mon[30894]: 2.43 scrub ok 2024-09-11T03:51:23.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:23 smithi175 ceph-mon[30894]: osdmap e864: 12 total, 12 up, 11 in 2024-09-11T03:51:23.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:23 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:51:23.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:23 smithi175 ceph-mon[30894]: 2.e scrub ok 2024-09-11T03:51:23.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:23 smithi175 ceph-mon[30894]: 2.34 scrub starts 2024-09-11T03:51:23.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:23 smithi175 ceph-mon[30894]: 2.34 scrub ok 2024-09-11T03:51:23.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:23 smithi175 ceph-mon[30894]: 2.37 scrub starts 2024-09-11T03:51:23.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:23 smithi175 ceph-mon[30894]: 2.37 scrub ok 2024-09-11T03:51:23.814 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:23.835 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:23.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:23.836 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:23.937 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:23.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:23.959 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:23.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:24.060 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:24.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:24.083 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:24.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:24.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:24.207 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:24.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:24.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:24.231 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:24.231 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:24.254 INFO:tasks.rados.rados.2.smithi175.stdout:3601: done (0 left) 2024-09-11T03:51:24.254 INFO:tasks.rados.rados.2.smithi175.stdout:3602: snap_create 2024-09-11T03:51:24.308 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:24.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:24.330 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:24.331 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:24.432 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:24.454 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:24.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:24.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:24.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:24 smithi084 ceph-mon[27254]: 2.6 scrub starts 2024-09-11T03:51:24.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:24 smithi084 ceph-mon[27254]: 2.6 scrub ok 2024-09-11T03:51:24.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:24 smithi084 ceph-mon[27254]: pgmap v1452: 107 pgs: 4 active+undersized+remapped, 1 active+clean+scrubbing, 102 active+clean; 697 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 490 KiB/s, 23 objects/s recovering 2024-09-11T03:51:24.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:24 smithi084 ceph-mon[27254]: 1.1a scrub starts 2024-09-11T03:51:24.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:24 smithi084 ceph-mon[27254]: 1.1a scrub ok 2024-09-11T03:51:24.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:24 smithi084 ceph-mon[27254]: osdmap e865: 12 total, 12 up, 11 in 2024-09-11T03:51:24.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:24 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:51:24.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:24 smithi084 ceph-mon[27254]: 2.14 scrub ok 2024-09-11T03:51:24.474 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:51:24.474 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:51:24.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:51:24.475 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:51:24.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.475 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2431 2.12 2:4e5019e5:::smithi17557870-40:head [copy-from ver 279] snapc 66=[66,65,64,63,61,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-09-11T03:51:24.475 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:51.877192+0000", 2024-09-11T03:51:24.475 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 572.59560290100001, 2024-09-11T03:51:24.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013330089, 2024-09-11T03:51:24.475 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.475 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.475 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:24.475 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:51:24.475 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:51:24.475 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:24.476 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2431 2024-09-11T03:51:24.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.476 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877192+0000", 2024-09-11T03:51:24.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877192+0000", 2024-09-11T03:51:24.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877194+0000", 2024-09-11T03:51:24.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.767e-06 2024-09-11T03:51:24.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877202+0000", 2024-09-11T03:51:24.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0770000000000008e-06 2024-09-11T03:51:24.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877220+0000", 2024-09-11T03:51:24.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8561000000000001e-05 2024-09-11T03:51:24.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877230+0000", 2024-09-11T03:51:24.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2820000000000002e-06 2024-09-11T03:51:24.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877290+0000", 2024-09-11T03:51:24.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0025999999999998e-05 2024-09-11T03:51:24.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.877404+0000", 2024-09-11T03:51:24.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011425999999999999 2024-09-11T03:51:24.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,11", 2024-09-11T03:51:24.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.884630+0000", 2024-09-11T03:51:24.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072256769999999998 2024-09-11T03:51:24.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:51:24.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.888559+0000", 2024-09-11T03:51:24.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039288750000000001 2024-09-11T03:51:24.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:24.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.888615+0000", 2024-09-11T03:51:24.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6609e-05 2024-09-11T03:51:24.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:24.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.890459+0000", 2024-09-11T03:51:24.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001844116 2024-09-11T03:51:24.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:51:24.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.890494+0000", 2024-09-11T03:51:24.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4921e-05 2024-09-11T03:51:24.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:51.890522+0000", 2024-09-11T03:51:24.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7917999999999999e-05 2024-09-11T03:51:24.481 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.481 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.481 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.481 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2443 2.12 2:494ba932:::smithi17557870-43:head [copy-from ver 342] snapc 66=[65,64,63,61,5d,58] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-11T03:51:24.482 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:52.884558+0000", 2024-09-11T03:51:24.482 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 571.588236532, 2024-09-11T03:51:24.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019937844999999999, 2024-09-11T03:51:24.482 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.482 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.482 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:24.482 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:51:24.482 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:51:24.482 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:24.482 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2443 2024-09-11T03:51:24.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.482 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884558+0000", 2024-09-11T03:51:24.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884558+0000", 2024-09-11T03:51:24.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884560+0000", 2024-09-11T03:51:24.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7e-06 2024-09-11T03:51:24.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884568+0000", 2024-09-11T03:51:24.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0940000000000008e-06 2024-09-11T03:51:24.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884584+0000", 2024-09-11T03:51:24.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6277e-05 2024-09-11T03:51:24.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884593+0000", 2024-09-11T03:51:24.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1810000000000001e-06 2024-09-11T03:51:24.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884660+0000", 2024-09-11T03:51:24.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7362999999999999e-05 2024-09-11T03:51:24.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884731+0000", 2024-09-11T03:51:24.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1316999999999995e-05 2024-09-11T03:51:24.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,11", 2024-09-11T03:51:24.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.895614+0000", 2024-09-11T03:51:24.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010882911 2024-09-11T03:51:24.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:51:24.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.900723+0000", 2024-09-11T03:51:24.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005109143 2024-09-11T03:51:24.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:24.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.902758+0000", 2024-09-11T03:51:24.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020342870000000001 2024-09-11T03:51:24.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:24.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.904417+0000", 2024-09-11T03:51:24.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001658875 2024-09-11T03:51:24.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:51:24.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.904454+0000", 2024-09-11T03:51:24.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7883999999999998e-05 2024-09-11T03:51:24.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.904496+0000", 2024-09-11T03:51:24.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1813000000000001e-05 2024-09-11T03:51:24.487 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.488 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.488 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.488 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2444 2.12 2:494ba932:::smithi17557870-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e296)", 2024-09-11T03:51:24.488 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:41:52.884615+0000", 2024-09-11T03:51:24.488 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 571.58817997100005, 2024-09-11T03:51:24.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015042823, 2024-09-11T03:51:24.488 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.488 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.488 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:51:24.488 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:51:24.488 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:51:24.488 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:24.489 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2444 2024-09-11T03:51:24.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.489 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884615+0000", 2024-09-11T03:51:24.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884615+0000", 2024-09-11T03:51:24.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884616+0000", 2024-09-11T03:51:24.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5499999999999998e-07 2024-09-11T03:51:24.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884618+0000", 2024-09-11T03:51:24.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.818e-06 2024-09-11T03:51:24.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884632+0000", 2024-09-11T03:51:24.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3219e-05 2024-09-11T03:51:24.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884639+0000", 2024-09-11T03:51:24.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9990000000000002e-06 2024-09-11T03:51:24.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884918+0000", 2024-09-11T03:51:24.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027906699999999998 2024-09-11T03:51:24.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:51:24.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.884959+0000", 2024-09-11T03:51:24.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1844999999999997e-05 2024-09-11T03:51:24.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.899489+0000", 2024-09-11T03:51:24.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014529034999999999 2024-09-11T03:51:24.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.899577+0000", 2024-09-11T03:51:24.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8512999999999998e-05 2024-09-11T03:51:24.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:41:52.899658+0000", 2024-09-11T03:51:24.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0772000000000005e-05 2024-09-11T03:51:24.493 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.493 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.493 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.494 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2641 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 350] snapc 75=[75,74,73,71,6f,6c,68,65,5d] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-11T03:51:24.494 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:19.887918+0000", 2024-09-11T03:51:24.494 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 544.58487667700001, 2024-09-11T03:51:24.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011895905, 2024-09-11T03:51:24.494 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.494 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.494 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:24.494 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:51:24.494 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:51:24.494 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:24.494 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2641 2024-09-11T03:51:24.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.495 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887918+0000", 2024-09-11T03:51:24.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887918+0000", 2024-09-11T03:51:24.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887920+0000", 2024-09-11T03:51:24.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.762e-06 2024-09-11T03:51:24.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887928+0000", 2024-09-11T03:51:24.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8900000000000007e-06 2024-09-11T03:51:24.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887945+0000", 2024-09-11T03:51:24.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6707000000000001e-05 2024-09-11T03:51:24.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.887953+0000", 2024-09-11T03:51:24.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5620000000000004e-06 2024-09-11T03:51:24.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.888014+0000", 2024-09-11T03:51:24.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0798000000000003e-05 2024-09-11T03:51:24.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:51:24.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.888019+0000", 2024-09-11T03:51:24.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5990000000000004e-06 2024-09-11T03:51:24.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.889290+0000", 2024-09-11T03:51:24.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012718009999999999 2024-09-11T03:51:24.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.889393+0000", 2024-09-11T03:51:24.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102298 2024-09-11T03:51:24.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,8", 2024-09-11T03:51:24.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.894120+0000", 2024-09-11T03:51:24.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047277259999999998 2024-09-11T03:51:24.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:51:24.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.897232+0000", 2024-09-11T03:51:24.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031110859999999999 2024-09-11T03:51:24.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:24.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.897530+0000", 2024-09-11T03:51:24.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000298342 2024-09-11T03:51:24.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:24.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.899728+0000", 2024-09-11T03:51:24.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021984819999999999 2024-09-11T03:51:24.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:51:24.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.899782+0000", 2024-09-11T03:51:24.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3922999999999999e-05 2024-09-11T03:51:24.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:19.899814+0000", 2024-09-11T03:51:24.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1928999999999999e-05 2024-09-11T03:51:24.501 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.501 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.501 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.501 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2944 2.12 2:494ba932:::smithi17557870-43:head [copy-from ver 311] snapc 80=[80,7b,7a,79,77,75,6c] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-11T03:51:24.501 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:43.962947+0000", 2024-09-11T03:51:24.501 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 520.50984747699999, 2024-09-11T03:51:24.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014367049, 2024-09-11T03:51:24.501 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.501 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.502 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:24.502 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:51:24.502 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:51:24.502 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:24.502 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2944 2024-09-11T03:51:24.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.502 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962947+0000", 2024-09-11T03:51:24.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962947+0000", 2024-09-11T03:51:24.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962949+0000", 2024-09-11T03:51:24.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.689e-06 2024-09-11T03:51:24.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962955+0000", 2024-09-11T03:51:24.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.297e-06 2024-09-11T03:51:24.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962967+0000", 2024-09-11T03:51:24.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1517e-05 2024-09-11T03:51:24.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.962975+0000", 2024-09-11T03:51:24.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2889999999999998e-06 2024-09-11T03:51:24.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963875+0000", 2024-09-11T03:51:24.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00089981200000000005 2024-09-11T03:51:24.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.964025+0000", 2024-09-11T03:51:24.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000149624 2024-09-11T03:51:24.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,7", 2024-09-11T03:51:24.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.971400+0000", 2024-09-11T03:51:24.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073748349999999997 2024-09-11T03:51:24.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:51:24.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.975907+0000", 2024-09-11T03:51:24.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045076159999999999 2024-09-11T03:51:24.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:24.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.976808+0000", 2024-09-11T03:51:24.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00090078700000000001 2024-09-11T03:51:24.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:24.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.977244+0000", 2024-09-11T03:51:24.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00043565800000000001 2024-09-11T03:51:24.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:51:24.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.977281+0000", 2024-09-11T03:51:24.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7150999999999997e-05 2024-09-11T03:51:24.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.977315+0000", 2024-09-11T03:51:24.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3773999999999999e-05 2024-09-11T03:51:24.508 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.509 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.509 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.509 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:2945 2.12 2:494ba932:::smithi17557870-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e347)", 2024-09-11T03:51:24.509 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:42:43.963009+0000", 2024-09-11T03:51:24.509 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 520.50978587700001, 2024-09-11T03:51:24.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012192595000000001, 2024-09-11T03:51:24.509 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.509 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.509 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:51:24.509 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:51:24.509 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:51:24.509 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:24.510 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2945 2024-09-11T03:51:24.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.510 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963009+0000", 2024-09-11T03:51:24.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963009+0000", 2024-09-11T03:51:24.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963010+0000", 2024-09-11T03:51:24.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0699999999999997e-07 2024-09-11T03:51:24.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963012+0000", 2024-09-11T03:51:24.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6599999999999999e-06 2024-09-11T03:51:24.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963020+0000", 2024-09-11T03:51:24.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2649999999999999e-06 2024-09-11T03:51:24.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.963025+0000", 2024-09-11T03:51:24.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6860000000000003e-06 2024-09-11T03:51:24.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.964185+0000", 2024-09-11T03:51:24.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011599329999999999 2024-09-11T03:51:24.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:51:24.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.964230+0000", 2024-09-11T03:51:24.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4336999999999999e-05 2024-09-11T03:51:24.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.974986+0000", 2024-09-11T03:51:24.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010755928999999999 2024-09-11T03:51:24.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.975091+0000", 2024-09-11T03:51:24.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105701 2024-09-11T03:51:24.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:42:43.975202+0000", 2024-09-11T03:51:24.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011047700000000001 2024-09-11T03:51:24.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.514 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.514 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:3536 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 408] snapc 90=[90,8f,8d,8c,8b,8a] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-09-11T03:51:24.514 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:43:18.903548+0000", 2024-09-11T03:51:24.514 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 485.569246545, 2024-09-11T03:51:24.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014015196000000001, 2024-09-11T03:51:24.515 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.515 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.515 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:24.515 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:51:24.515 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:51:24.515 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:24.515 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3536 2024-09-11T03:51:24.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.515 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903548+0000", 2024-09-11T03:51:24.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903548+0000", 2024-09-11T03:51:24.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903550+0000", 2024-09-11T03:51:24.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.499e-06 2024-09-11T03:51:24.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903555+0000", 2024-09-11T03:51:24.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6890000000000003e-06 2024-09-11T03:51:24.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903567+0000", 2024-09-11T03:51:24.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1912999999999999e-05 2024-09-11T03:51:24.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903574+0000", 2024-09-11T03:51:24.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7260000000000002e-06 2024-09-11T03:51:24.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904043+0000", 2024-09-11T03:51:24.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00046877799999999999 2024-09-11T03:51:24.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904136+0000", 2024-09-11T03:51:24.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2745999999999996e-05 2024-09-11T03:51:24.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,8", 2024-09-11T03:51:24.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.912436+0000", 2024-09-11T03:51:24.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083000049999999992 2024-09-11T03:51:24.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:51:24.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.915813+0000", 2024-09-11T03:51:24.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033772839999999999 2024-09-11T03:51:24.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:24.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.916094+0000", 2024-09-11T03:51:24.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028135600000000002 2024-09-11T03:51:24.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:24.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.917498+0000", 2024-09-11T03:51:24.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014033680000000001 2024-09-11T03:51:24.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:51:24.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.917534+0000", 2024-09-11T03:51:24.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6180000000000003e-05 2024-09-11T03:51:24.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.917564+0000", 2024-09-11T03:51:24.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9652e-05 2024-09-11T03:51:24.521 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.521 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.521 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.521 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:3537 2.19 2:98bb65f5:::smithi17557870-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e381)", 2024-09-11T03:51:24.521 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:51:24.521 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 485.56919907399998, 2024-09-11T03:51:24.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011559349999999999, 2024-09-11T03:51:24.521 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.521 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.522 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:51:24.522 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:51:24.522 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:51:24.522 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:24.522 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3537 2024-09-11T03:51:24.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.522 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:51:24.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:51:24.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903596+0000", 2024-09-11T03:51:24.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8899999999999999e-07 2024-09-11T03:51:24.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903599+0000", 2024-09-11T03:51:24.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3750000000000001e-06 2024-09-11T03:51:24.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903606+0000", 2024-09-11T03:51:24.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1969999999999999e-06 2024-09-11T03:51:24.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.903611+0000", 2024-09-11T03:51:24.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3680000000000001e-06 2024-09-11T03:51:24.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904244+0000", 2024-09-11T03:51:24.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00063279999999999999 2024-09-11T03:51:24.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:51:24.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.904291+0000", 2024-09-11T03:51:24.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6755999999999998e-05 2024-09-11T03:51:24.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.914973+0000", 2024-09-11T03:51:24.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010682199999999999 2024-09-11T03:51:24.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.915043+0000", 2024-09-11T03:51:24.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0297999999999997e-05 2024-09-11T03:51:24.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:18.915155+0000", 2024-09-11T03:51:24.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000111767 2024-09-11T03:51:24.526 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.526 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.526 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.527 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:5885 2.e e613/577 2:77dadb97:::smithi17557870-30:head v 613'880, mlcod=601'877)", 2024-09-11T03:51:24.527 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:47:11.245847+0000", 2024-09-11T03:51:24.527 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 253.226948225, 2024-09-11T03:51:24.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012027239, 2024-09-11T03:51:24.527 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.527 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.527 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:24.527 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.245847+0000", 2024-09-11T03:51:24.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.245847+0000", 2024-09-11T03:51:24.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.245847+0000", 2024-09-11T03:51:24.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7300000000000002e-07 2024-09-11T03:51:24.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.252621+0000", 2024-09-11T03:51:24.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067743339999999999 2024-09-11T03:51:24.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.252641+0000", 2024-09-11T03:51:24.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9734000000000001e-05 2024-09-11T03:51:24.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.252650+0000", 2024-09-11T03:51:24.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3400000000000004e-06 2024-09-11T03:51:24.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.252721+0000", 2024-09-11T03:51:24.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0767999999999997e-05 2024-09-11T03:51:24.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.252745+0000", 2024-09-11T03:51:24.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4278999999999999e-05 2024-09-11T03:51:24.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:51:24.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.257835+0000", 2024-09-11T03:51:24.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0050897060000000003 2024-09-11T03:51:24.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:11.257874+0000", 2024-09-11T03:51:24.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8804999999999999e-05 2024-09-11T03:51:24.531 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.531 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.531 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.531 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:6152 2.e e637/577 2:77dadb97:::smithi17557870-30:head v 637'891, mlcod=634'888)", 2024-09-11T03:51:24.531 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:47:35.714675+0000", 2024-09-11T03:51:24.531 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 228.75811954599999, 2024-09-11T03:51:24.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011964542, 2024-09-11T03:51:24.531 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.531 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.532 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:24.532 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714675+0000", 2024-09-11T03:51:24.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714675+0000", 2024-09-11T03:51:24.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714676+0000", 2024-09-11T03:51:24.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0100000000000001e-07 2024-09-11T03:51:24.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.720928+0000", 2024-09-11T03:51:24.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062523839999999997 2024-09-11T03:51:24.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.720951+0000", 2024-09-11T03:51:24.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2751e-05 2024-09-11T03:51:24.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.720964+0000", 2024-09-11T03:51:24.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2673999999999999e-05 2024-09-11T03:51:24.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721056+0000", 2024-09-11T03:51:24.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2620000000000004e-05 2024-09-11T03:51:24.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721084+0000", 2024-09-11T03:51:24.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7974999999999999e-05 2024-09-11T03:51:24.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:51:24.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.726587+0000", 2024-09-11T03:51:24.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055026420000000003 2024-09-11T03:51:24.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.726640+0000", 2024-09-11T03:51:24.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3195000000000001e-05 2024-09-11T03:51:24.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.535 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.536 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.536 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:6546 2.e e669/577 2:750de3b7:::smithi17557870-8:head v 669'909, mlcod=667'905)", 2024-09-11T03:51:24.536 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:07.227071+0000", 2024-09-11T03:51:24.536 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 197.24572352000001, 2024-09-11T03:51:24.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013198415999999999, 2024-09-11T03:51:24.536 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.536 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.536 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:24.536 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.227071+0000", 2024-09-11T03:51:24.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.227071+0000", 2024-09-11T03:51:24.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.227072+0000", 2024-09-11T03:51:24.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2e-07 2024-09-11T03:51:24.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.235454+0000", 2024-09-11T03:51:24.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083820730000000003 2024-09-11T03:51:24.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.235470+0000", 2024-09-11T03:51:24.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5951e-05 2024-09-11T03:51:24.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.235478+0000", 2024-09-11T03:51:24.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4249999999999998e-06 2024-09-11T03:51:24.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.235539+0000", 2024-09-11T03:51:24.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0473000000000002e-05 2024-09-11T03:51:24.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.235556+0000", 2024-09-11T03:51:24.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7110000000000001e-05 2024-09-11T03:51:24.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:51:24.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.240234+0000", 2024-09-11T03:51:24.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046781720000000004 2024-09-11T03:51:24.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.240270+0000", 2024-09-11T03:51:24.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5791999999999999e-05 2024-09-11T03:51:24.540 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.540 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.540 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.541 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:6548 2.9 e669/276 2:9718db5a:::smithi17557870-3:head v 669'978, mlcod=668'974)", 2024-09-11T03:51:24.541 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:07.229009+0000", 2024-09-11T03:51:24.541 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 197.24378644399999, 2024-09-11T03:51:24.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011091311, 2024-09-11T03:51:24.541 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.541 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.541 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:24.541 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.229009+0000", 2024-09-11T03:51:24.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.229009+0000", 2024-09-11T03:51:24.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.229010+0000", 2024-09-11T03:51:24.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9000000000000005e-07 2024-09-11T03:51:24.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.231542+0000", 2024-09-11T03:51:24.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025329570000000002 2024-09-11T03:51:24.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.231569+0000", 2024-09-11T03:51:24.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6387e-05 2024-09-11T03:51:24.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.231586+0000", 2024-09-11T03:51:24.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7320999999999999e-05 2024-09-11T03:51:24.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.231672+0000", 2024-09-11T03:51:24.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6117000000000002e-05 2024-09-11T03:51:24.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.231712+0000", 2024-09-11T03:51:24.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0114000000000003e-05 2024-09-11T03:51:24.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:51:24.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.240043+0000", 2024-09-11T03:51:24.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083307539999999992 2024-09-11T03:51:24.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.240100+0000", 2024-09-11T03:51:24.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6671e-05 2024-09-11T03:51:24.545 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.545 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.545 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.545 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:7117 2.e e713/577 2:750de3b7:::smithi17557870-8:head v 713'939, mlcod=713'935)", 2024-09-11T03:51:24.545 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:51.211659+0000", 2024-09-11T03:51:24.545 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 153.26113633400001, 2024-09-11T03:51:24.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011072423, 2024-09-11T03:51:24.545 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.546 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.546 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:24.546 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.211659+0000", 2024-09-11T03:51:24.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.211659+0000", 2024-09-11T03:51:24.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.211660+0000", 2024-09-11T03:51:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.22e-06 2024-09-11T03:51:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.217534+0000", 2024-09-11T03:51:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058737700000000004 2024-09-11T03:51:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.217555+0000", 2024-09-11T03:51:24.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0968e-05 2024-09-11T03:51:24.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.217568+0000", 2024-09-11T03:51:24.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3108999999999999e-05 2024-09-11T03:51:24.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.217675+0000", 2024-09-11T03:51:24.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000106969 2024-09-11T03:51:24.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.217694+0000", 2024-09-11T03:51:24.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9817999999999999e-05 2024-09-11T03:51:24.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:51:24.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.222679+0000", 2024-09-11T03:51:24.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049843140000000001 2024-09-11T03:51:24.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.222731+0000", 2024-09-11T03:51:24.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2255e-05 2024-09-11T03:51:24.549 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.550 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.550 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.550 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:7213 2.d e723/569 2:b6874adf:::smithi17557870-17:head v 723'925, mlcod=722'922)", 2024-09-11T03:51:24.550 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:49:01.039129+0000", 2024-09-11T03:51:24.550 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 143.43366644299999, 2024-09-11T03:51:24.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013254739999999999, 2024-09-11T03:51:24.550 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.550 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.550 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:24.550 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.039129+0000", 2024-09-11T03:51:24.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.039129+0000", 2024-09-11T03:51:24.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.039129+0000", 2024-09-11T03:51:24.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1900000000000002e-07 2024-09-11T03:51:24.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.051043+0000", 2024-09-11T03:51:24.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011913604 2024-09-11T03:51:24.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.051052+0000", 2024-09-11T03:51:24.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4930000000000004e-06 2024-09-11T03:51:24.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.051060+0000", 2024-09-11T03:51:24.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8199999999999997e-06 2024-09-11T03:51:24.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.051100+0000", 2024-09-11T03:51:24.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0394e-05 2024-09-11T03:51:24.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.051114+0000", 2024-09-11T03:51:24.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3083e-05 2024-09-11T03:51:24.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:51:24.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.052353+0000", 2024-09-11T03:51:24.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012397899999999999 2024-09-11T03:51:24.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:01.052383+0000", 2024-09-11T03:51:24.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9937000000000002e-05 2024-09-11T03:51:24.554 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.554 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.555 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.555 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7396 2.2d 2:b6874adf:::smithi17557870-17:head [copy-from ver 0] snapc 14f=[14f,14e,14d,14b,14a,149,148,146,145,144,143,142,13d,13c,139,138,136,135,133,12e,12a,127,120,11e,119] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-09-11T03:51:24.555 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:49:27.145662+0000", 2024-09-11T03:51:24.555 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 117.327133385, 2024-09-11T03:51:24.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022605074999999999, 2024-09-11T03:51:24.555 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.555 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.555 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:51:24.555 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:51:24.555 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:51:24.555 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:24.555 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7396 2024-09-11T03:51:24.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.556 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145662+0000", 2024-09-11T03:51:24.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145662+0000", 2024-09-11T03:51:24.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145663+0000", 2024-09-11T03:51:24.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.449e-06 2024-09-11T03:51:24.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145669+0000", 2024-09-11T03:51:24.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.558e-06 2024-09-11T03:51:24.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145683+0000", 2024-09-11T03:51:24.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4708999999999999e-05 2024-09-11T03:51:24.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145693+0000", 2024-09-11T03:51:24.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5140000000000007e-06 2024-09-11T03:51:24.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145751+0000", 2024-09-11T03:51:24.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8065000000000001e-05 2024-09-11T03:51:24.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:51:24.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145762+0000", 2024-09-11T03:51:24.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1625e-05 2024-09-11T03:51:24.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.166297+0000", 2024-09-11T03:51:24.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020534759999999999 2024-09-11T03:51:24.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.166499+0000", 2024-09-11T03:51:24.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000201454 2024-09-11T03:51:24.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.168267+0000", 2024-09-11T03:51:24.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017679410000000001 2024-09-11T03:51:24.560 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.560 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.560 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.560 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7397 2.2d 2:b6874adf:::smithi17557870-17:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e749)", 2024-09-11T03:51:24.560 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:49:27.145715+0000", 2024-09-11T03:51:24.560 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 117.32708030400001, 2024-09-11T03:51:24.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022944445000000001, 2024-09-11T03:51:24.560 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.560 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.561 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:51:24.561 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:51:24.561 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:51:24.561 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:24.561 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7397 2024-09-11T03:51:24.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.561 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145715+0000", 2024-09-11T03:51:24.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145715+0000", 2024-09-11T03:51:24.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145715+0000", 2024-09-11T03:51:24.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5899999999999996e-07 2024-09-11T03:51:24.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145718+0000", 2024-09-11T03:51:24.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5969999999999999e-06 2024-09-11T03:51:24.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145725+0000", 2024-09-11T03:51:24.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3180000000000001e-06 2024-09-11T03:51:24.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145731+0000", 2024-09-11T03:51:24.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6829999999999996e-06 2024-09-11T03:51:24.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145835+0000", 2024-09-11T03:51:24.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000103977 2024-09-11T03:51:24.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:51:24.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.145845+0000", 2024-09-11T03:51:24.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0599e-05 2024-09-11T03:51:24.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.166949+0000", 2024-09-11T03:51:24.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021103706 2024-09-11T03:51:24.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:51:24.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.167035+0000", 2024-09-11T03:51:24.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6244999999999998e-05 2024-09-11T03:51:24.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.168341+0000", 2024-09-11T03:51:24.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001305949 2024-09-11T03:51:24.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.168440+0000", 2024-09-11T03:51:24.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8346999999999999e-05 2024-09-11T03:51:24.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.168659+0000", 2024-09-11T03:51:24.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021946499999999999 2024-09-11T03:51:24.566 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.566 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.566 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.566 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.8.0:1632 2.2c e853/750 2:3603e322:::smithi17557870-42:182 v 853'903, mlcod=845'899)", 2024-09-11T03:51:24.566 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:11.053445+0000", 2024-09-11T03:51:24.566 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 13.419349496000001, 2024-09-11T03:51:24.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011808134, 2024-09-11T03:51:24.567 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.567 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.567 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:24.567 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.053445+0000", 2024-09-11T03:51:24.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.053445+0000", 2024-09-11T03:51:24.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.053446+0000", 2024-09-11T03:51:24.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2899999999999999e-07 2024-09-11T03:51:24.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.053456+0000", 2024-09-11T03:51:24.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8220000000000002e-06 2024-09-11T03:51:24.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.053463+0000", 2024-09-11T03:51:24.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1459999999999999e-06 2024-09-11T03:51:24.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.053470+0000", 2024-09-11T03:51:24.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8900000000000001e-06 2024-09-11T03:51:24.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.053528+0000", 2024-09-11T03:51:24.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8663999999999999e-05 2024-09-11T03:51:24.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.053549+0000", 2024-09-11T03:51:24.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0928e-05 2024-09-11T03:51:24.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:51:24.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.065199+0000", 2024-09-11T03:51:24.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011650199999999999 2024-09-11T03:51:24.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.065254+0000", 2024-09-11T03:51:24.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4154999999999999e-05 2024-09-11T03:51:24.570 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.571 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.571 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.571 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:700 2.19 e853/809 2:98bb65f5:::smithi17557870-6:18a v 853'603, mlcod=850'600)", 2024-09-11T03:51:24.571 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:11.054329+0000", 2024-09-11T03:51:24.571 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 13.418465692, 2024-09-11T03:51:24.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011251738000000001, 2024-09-11T03:51:24.571 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.571 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.571 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:24.571 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.054329+0000", 2024-09-11T03:51:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.054329+0000", 2024-09-11T03:51:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.054330+0000", 2024-09-11T03:51:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3200000000000001e-07 2024-09-11T03:51:24.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.054334+0000", 2024-09-11T03:51:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6820000000000002e-06 2024-09-11T03:51:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.054341+0000", 2024-09-11T03:51:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8979999999999999e-06 2024-09-11T03:51:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.054347+0000", 2024-09-11T03:51:24.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2299999999999996e-06 2024-09-11T03:51:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.055663+0000", 2024-09-11T03:51:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0013152159999999999 2024-09-11T03:51:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.055683+0000", 2024-09-11T03:51:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0780000000000001e-05 2024-09-11T03:51:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:51:24.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.065544+0000", 2024-09-11T03:51:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0098603449999999995 2024-09-11T03:51:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.065581+0000", 2024-09-11T03:51:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7255000000000002e-05 2024-09-11T03:51:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:8786 2.d e853/745 2:b1c0f730:::smithi17557870-20:head v 853'1041, mlcod=847'1037)", 2024-09-11T03:51:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:11.085247+0000", 2024-09-11T03:51:24.575 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 13.387548318, 2024-09-11T03:51:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019809757000000001, 2024-09-11T03:51:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.085247+0000", 2024-09-11T03:51:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.085247+0000", 2024-09-11T03:51:24.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.085247+0000", 2024-09-11T03:51:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4099999999999998e-07 2024-09-11T03:51:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.102848+0000", 2024-09-11T03:51:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017600615999999999 2024-09-11T03:51:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.102864+0000", 2024-09-11T03:51:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6404000000000001e-05 2024-09-11T03:51:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.102873+0000", 2024-09-11T03:51:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4989999999999994e-06 2024-09-11T03:51:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.102952+0000", 2024-09-11T03:51:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8800000000000004e-05 2024-09-11T03:51:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.102978+0000", 2024-09-11T03:51:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6064e-05 2024-09-11T03:51:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:51:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.105005+0000", 2024-09-11T03:51:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020272329999999998 2024-09-11T03:51:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.105056+0000", 2024-09-11T03:51:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1499999999999998e-05 2024-09-11T03:51:24.579 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:8857 2.29 e856/750 2:95fa802a:::smithi17557870-22:head v 856'1283, mlcod=851'1280)", 2024-09-11T03:51:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:14.152397+0000", 2024-09-11T03:51:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 10.320397925, 2024-09-11T03:51:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.02404562, 2024-09-11T03:51:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:51:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:51:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:51:24.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:51:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.152397+0000", 2024-09-11T03:51:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:51:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.152397+0000", 2024-09-11T03:51:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:51:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:51:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.152397+0000", 2024-09-11T03:51:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.35e-07 2024-09-11T03:51:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:51:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.174942+0000", 2024-09-11T03:51:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022544967999999999 2024-09-11T03:51:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:51:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.174961+0000", 2024-09-11T03:51:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8978999999999999e-05 2024-09-11T03:51:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:51:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.174970+0000", 2024-09-11T03:51:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7309999999999999e-06 2024-09-11T03:51:24.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:51:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.175083+0000", 2024-09-11T03:51:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113077 2024-09-11T03:51:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:51:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.175101+0000", 2024-09-11T03:51:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7790000000000001e-05 2024-09-11T03:51:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:51:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.176408+0000", 2024-09-11T03:51:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001307634 2024-09-11T03:51:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:51:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:51:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:51:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.176443+0000", 2024-09-11T03:51:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4205999999999997e-05 2024-09-11T03:51:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:51:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:51:24.584 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:51:24.585 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:24.607 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:24.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:24.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:24.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:24 smithi121 ceph-mon[30729]: 2.6 scrub starts 2024-09-11T03:51:24.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:24 smithi121 ceph-mon[30729]: 2.6 scrub ok 2024-09-11T03:51:24.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:24 smithi121 ceph-mon[30729]: pgmap v1452: 107 pgs: 4 active+undersized+remapped, 1 active+clean+scrubbing, 102 active+clean; 697 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 490 KiB/s, 23 objects/s recovering 2024-09-11T03:51:24.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:24 smithi121 ceph-mon[30729]: 1.1a scrub starts 2024-09-11T03:51:24.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:24 smithi121 ceph-mon[30729]: 1.1a scrub ok 2024-09-11T03:51:24.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:24 smithi121 ceph-mon[30729]: osdmap e865: 12 total, 12 up, 11 in 2024-09-11T03:51:24.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:24 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:51:24.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:24 smithi121 ceph-mon[30729]: 2.14 scrub ok 2024-09-11T03:51:24.709 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:24.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:24 smithi175 ceph-mon[30894]: 2.6 scrub starts 2024-09-11T03:51:24.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:24 smithi175 ceph-mon[30894]: 2.6 scrub ok 2024-09-11T03:51:24.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:24 smithi175 ceph-mon[30894]: pgmap v1452: 107 pgs: 4 active+undersized+remapped, 1 active+clean+scrubbing, 102 active+clean; 697 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 490 KiB/s, 23 objects/s recovering 2024-09-11T03:51:24.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:24 smithi175 ceph-mon[30894]: 1.1a scrub starts 2024-09-11T03:51:24.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:24 smithi175 ceph-mon[30894]: 1.1a scrub ok 2024-09-11T03:51:24.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:24 smithi175 ceph-mon[30894]: osdmap e865: 12 total, 12 up, 11 in 2024-09-11T03:51:24.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:24 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:51:24.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:24 smithi175 ceph-mon[30894]: 2.14 scrub ok 2024-09-11T03:51:24.730 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:24.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:24.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:24.832 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:24.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:24.855 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:24.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:24.957 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:24.979 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:24.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:24.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:25.081 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:25.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:25.105 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:25.105 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:25.206 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:25.228 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:25.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:25.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:25.256 INFO:tasks.rados.rados.2.smithi175.stdout:3602: done (0 left) 2024-09-11T03:51:25.256 INFO:tasks.rados.rados.2.smithi175.stdout:3603: rollback oid 10 current snap is 400 2024-09-11T03:51:25.256 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 10 to 386 2024-09-11T03:51:25.256 INFO:tasks.rados.rados.2.smithi175.stdout:3604: write (excl) oid 3 current snap is 400 2024-09-11T03:51:25.257 INFO:tasks.rados.rados.2.smithi175.stdout:3604: seq_num 1225 ranges {588575=512858,1867383=455083} 2024-09-11T03:51:25.262 INFO:tasks.rados.rados.2.smithi175.stdout:3604: writing smithi17557870-3 from 588575 to 1101433 tid 1 2024-09-11T03:51:25.269 INFO:tasks.rados.rados.2.smithi175.stdout:3604: writing smithi17557870-3 from 1867383 to 2322466 tid 2 2024-09-11T03:51:25.269 INFO:tasks.rados.rados.2.smithi175.stdout:3605: write (excl) oid 39 current snap is 400 2024-09-11T03:51:25.269 INFO:tasks.rados.rados.2.smithi175.stdout:3605: seq_num 1226 ranges {605382=588107,1925865=679517} 2024-09-11T03:51:25.275 INFO:tasks.rados.rados.2.smithi175.stdout:3605: writing smithi17557870-39 from 605382 to 1193489 tid 1 2024-09-11T03:51:25.284 INFO:tasks.rados.rados.2.smithi175.stdout:3605: writing smithi17557870-39 from 1925865 to 2605382 tid 2 2024-09-11T03:51:25.284 INFO:tasks.rados.rados.2.smithi175.stdout:3606: copy_from oid 34 from oid 29 current snap is 400 2024-09-11T03:51:25.284 INFO:tasks.rados.rados.2.smithi175.stdout:3603: finishing rollback tid 1 to smithi17557870-10 2024-09-11T03:51:25.284 INFO:tasks.rados.rados.2.smithi175.stdout:3603: finishing rollback tid 2 to smithi17557870-10 2024-09-11T03:51:25.284 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 1238 (ObjNum 1075 snap 350 seq_num 1075) dirty exists 2024-09-11T03:51:25.284 INFO:tasks.rados.rados.2.smithi175.stdout:3604: finishing write tid 1 to smithi17557870-3 2024-09-11T03:51:25.285 INFO:tasks.rados.rados.2.smithi175.stdout:3604: finishing write tid 2 to smithi17557870-3 2024-09-11T03:51:25.285 INFO:tasks.rados.rados.2.smithi175.stdout:3604: finishing write tid 3 to smithi17557870-3 2024-09-11T03:51:25.285 INFO:tasks.rados.rados.2.smithi175.stdout:3604: finishing write tid 4 to smithi17557870-3 2024-09-11T03:51:25.285 INFO:tasks.rados.rados.2.smithi175.stdout:3604: oid 3 updating version 0 to 1300 2024-09-11T03:51:25.285 INFO:tasks.rados.rados.2.smithi175.stdout:3604: oid 3 version 1300 is already newer than 1298 2024-09-11T03:51:25.285 INFO:tasks.rados.rados.2.smithi175.stdout:3604: oid 3 version 1300 is already newer than 1299 2024-09-11T03:51:25.285 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 1300 (ObjNum 1225 snap 400 seq_num 1225) dirty exists 2024-09-11T03:51:25.285 INFO:tasks.rados.rados.2.smithi175.stdout:3604: left oid 3 (ObjNum 1225 snap 400 seq_num 1225) 2024-09-11T03:51:25.285 INFO:tasks.rados.rados.2.smithi175.stdout:3605: finishing write tid 1 to smithi17557870-39 2024-09-11T03:51:25.285 INFO:tasks.rados.rados.2.smithi175.stdout:3603: done (3 left) 2024-09-11T03:51:25.285 INFO:tasks.rados.rados.2.smithi175.stdout:3604: done (2 left) 2024-09-11T03:51:25.285 INFO:tasks.rados.rados.2.smithi175.stdout:3607: rollback oid 48 current snap is 400 2024-09-11T03:51:25.285 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 48 to 399 2024-09-11T03:51:25.286 INFO:tasks.rados.rados.2.smithi175.stdout:3608: snap_remove snap 387 2024-09-11T03:51:25.330 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:25.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:25 smithi084 ceph-mon[27254]: 2.45 scrub starts 2024-09-11T03:51:25.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:25 smithi084 ceph-mon[27254]: 2.45 scrub ok 2024-09-11T03:51:25.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:25 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:51:25.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:25 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:51:25.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:25 smithi084 ceph-mon[27254]: osdmap e866: 12 total, 12 up, 11 in 2024-09-11T03:51:25.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:25 smithi084 ceph-mon[27254]: 2.16 scrub starts 2024-09-11T03:51:25.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:25 smithi084 ceph-mon[27254]: 2.16 scrub ok 2024-09-11T03:51:25.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:25 smithi084 ceph-mon[27254]: 2.10 scrub starts 2024-09-11T03:51:25.331 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:25 smithi084 ceph-mon[27254]: 2.10 scrub ok 2024-09-11T03:51:25.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:25.352 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:25.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:25.453 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:25.474 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:25.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:25.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:25.576 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:25.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:25.599 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:25.599 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:25.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:25 smithi121 ceph-mon[30729]: 2.45 scrub starts 2024-09-11T03:51:25.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:25 smithi121 ceph-mon[30729]: 2.45 scrub ok 2024-09-11T03:51:25.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:25 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:51:25.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:25 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:51:25.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:25 smithi121 ceph-mon[30729]: osdmap e866: 12 total, 12 up, 11 in 2024-09-11T03:51:25.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:25 smithi121 ceph-mon[30729]: 2.16 scrub starts 2024-09-11T03:51:25.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:25 smithi121 ceph-mon[30729]: 2.16 scrub ok 2024-09-11T03:51:25.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:25 smithi121 ceph-mon[30729]: 2.10 scrub starts 2024-09-11T03:51:25.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:25 smithi121 ceph-mon[30729]: 2.10 scrub ok 2024-09-11T03:51:25.674 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:51:25.674 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:51:25.674 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:51:25.674 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:51:25.679 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:51:25.679 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:51:25.695 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:51:25.695 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:51:25.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:25.712 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:51:25.712 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:51:25.726 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:25.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:25.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:25.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:25 smithi175 ceph-mon[30894]: 2.45 scrub starts 2024-09-11T03:51:25.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:25 smithi175 ceph-mon[30894]: 2.45 scrub ok 2024-09-11T03:51:25.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:25 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:51:25.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:25 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:51:25.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:25 smithi175 ceph-mon[30894]: osdmap e866: 12 total, 12 up, 11 in 2024-09-11T03:51:25.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:25 smithi175 ceph-mon[30894]: 2.16 scrub starts 2024-09-11T03:51:25.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:25 smithi175 ceph-mon[30894]: 2.16 scrub ok 2024-09-11T03:51:25.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:25 smithi175 ceph-mon[30894]: 2.10 scrub starts 2024-09-11T03:51:25.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:25 smithi175 ceph-mon[30894]: 2.10 scrub ok 2024-09-11T03:51:25.729 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:51:25.730 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:51:25.747 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:51:25.747 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:51:25.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:51:25.764 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:51:25.781 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:51:25.781 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:51:25.797 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:51:25.798 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:51:25.814 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:51:25.814 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:51:25.828 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:25.832 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:51:25.832 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:51:25.850 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:25.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:25.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:25.952 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:25.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:25.974 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:25.975 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:26.077 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:26.097 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:26.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:26.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:26.199 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:26.221 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:26.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:26.259 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:51:26.262 INFO:tasks.rados.rados.2.smithi175.stdout:3608: done (3 left) 2024-09-11T03:51:26.262 INFO:tasks.rados.rados.2.smithi175.stdout:3609: read oid 49 snap -1 2024-09-11T03:51:26.262 INFO:tasks.rados.rados.2.smithi175.stdout:3609: expect (ObjNum 1195 snap 391 seq_num 1195) 2024-09-11T03:51:26.262 INFO:tasks.rados.rados.2.smithi175.stdout:3610: write (excl) oid 40 current snap is 400 2024-09-11T03:51:26.262 INFO:tasks.rados.rados.2.smithi175.stdout:3610: seq_num 1227 ranges {622189=663356,1984347=458752} 2024-09-11T03:51:26.270 INFO:tasks.rados.rados.2.smithi175.stdout:3610: writing smithi17557870-40 from 622189 to 1285545 tid 1 2024-09-11T03:51:26.278 INFO:tasks.rados.rados.2.smithi175.stdout:3610: writing smithi17557870-40 from 1984347 to 2443099 tid 2 2024-09-11T03:51:26.278 INFO:tasks.rados.rados.2.smithi175.stdout:3611: read oid 50 snap -1 2024-09-11T03:51:26.278 INFO:tasks.rados.rados.2.smithi175.stdout:3611: expect (ObjNum 1213 snap 397 seq_num 1213) 2024-09-11T03:51:26.278 INFO:tasks.rados.rados.2.smithi175.stdout:3612: copy_from oid 26 from oid 4 current snap is 400 2024-09-11T03:51:26.278 INFO:tasks.rados.rados.2.smithi175.stdout:3607: finishing rollback tid 0 to smithi17557870-48 2024-09-11T03:51:26.278 INFO:tasks.rados.rados.2.smithi175.stdout:3607: finishing rollback tid 1 to smithi17557870-48 2024-09-11T03:51:26.278 INFO:tasks.rados.rados.2.smithi175.stdout:3607: finishing rollback tid 2 to smithi17557870-48 2024-09-11T03:51:26.278 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 1014 (ObjNum 1206 snap 397 seq_num 1206) dirty exists 2024-09-11T03:51:26.278 INFO:tasks.rados.rados.2.smithi175.stdout:3605: finishing write tid 2 to smithi17557870-39 2024-09-11T03:51:26.278 INFO:tasks.rados.rados.2.smithi175.stdout:3605: finishing write tid 3 to smithi17557870-39 2024-09-11T03:51:26.279 INFO:tasks.rados.rados.2.smithi175.stdout:3607: done (6 left) 2024-09-11T03:51:26.279 INFO:tasks.rados.rados.2.smithi175.stdout:3613: read oid 32 snap -1 2024-09-11T03:51:26.279 INFO:tasks.rados.rados.2.smithi175.stdout:3613: expect (ObjNum 1059 snap 345 seq_num 1059) 2024-09-11T03:51:26.279 INFO:tasks.rados.rados.2.smithi175.stdout:3614: write oid 46 current snap is 400 2024-09-11T03:51:26.279 INFO:tasks.rados.rados.2.smithi175.stdout:3605: finishing write tid 4 to smithi17557870-39 2024-09-11T03:51:26.279 INFO:tasks.rados.rados.2.smithi175.stdout:3605: oid 39 updating version 0 to 1011 2024-09-11T03:51:26.279 INFO:tasks.rados.rados.2.smithi175.stdout:3605: oid 39 version 1011 is already newer than 1010 2024-09-11T03:51:26.280 INFO:tasks.rados.rados.2.smithi175.stdout:3605: oid 39 version 1011 is already newer than 1009 2024-09-11T03:51:26.280 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 1011 (ObjNum 1226 snap 400 seq_num 1226) dirty exists 2024-09-11T03:51:26.280 INFO:tasks.rados.rados.2.smithi175.stdout:3605: left oid 39 (ObjNum 1226 snap 400 seq_num 1226) 2024-09-11T03:51:26.280 INFO:tasks.rados.rados.2.smithi175.stdout:3606: finishing copy_from racing read to smithi17557870-34 2024-09-11T03:51:26.280 INFO:tasks.rados.rados.2.smithi175.stdout:3606: finishing copy_from to smithi17557870-34 2024-09-11T03:51:26.280 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 1205 (ObjNum 1179 snap 385 seq_num 1179) dirty exists 2024-09-11T03:51:26.280 INFO:tasks.rados.rados.2.smithi175.stdout:3614: seq_num 1228 ranges {638996=738605,1926476=712520} 2024-09-11T03:51:26.285 INFO:tasks.rados.rados.2.smithi175.stdout:3614: writing smithi17557870-46 from 638996 to 1377601 tid 1 2024-09-11T03:51:26.293 INFO:tasks.rados.rados.2.smithi175.stdout:3614: writing smithi17557870-46 from 1926476 to 2638996 tid 2 2024-09-11T03:51:26.293 INFO:tasks.rados.rados.2.smithi175.stdout:3605: done (7 left) 2024-09-11T03:51:26.294 INFO:tasks.rados.rados.2.smithi175.stdout:3606: done (6 left) 2024-09-11T03:51:26.294 INFO:tasks.rados.rados.2.smithi175.stdout:3615: delete oid 23 current snap is 400 2024-09-11T03:51:26.294 INFO:tasks.rados.rados.2.smithi175.stdout:3609: expect (ObjNum 1195 snap 391 seq_num 1195) 2024-09-11T03:51:26.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:26.346 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:26.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:26.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:26.361 INFO:tasks.rados.rados.2.smithi175.stdout:3610: finishing write tid 1 to smithi17557870-40 2024-09-11T03:51:26.361 INFO:tasks.rados.rados.2.smithi175.stdout:3610: finishing write tid 2 to smithi17557870-40 2024-09-11T03:51:26.362 INFO:tasks.rados.rados.2.smithi175.stdout:3610: finishing write tid 3 to smithi17557870-40 2024-09-11T03:51:26.362 INFO:tasks.rados.rados.2.smithi175.stdout:3610: finishing write tid 4 to smithi17557870-40 2024-09-11T03:51:26.362 INFO:tasks.rados.rados.2.smithi175.stdout:3610: oid 40 updating version 0 to 1014 2024-09-11T03:51:26.362 INFO:tasks.rados.rados.2.smithi175.stdout:3610: oid 40 updating version 1014 to 1015 2024-09-11T03:51:26.362 INFO:tasks.rados.rados.2.smithi175.stdout:3610: oid 40 version 1015 is already newer than 1013 2024-09-11T03:51:26.362 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 1015 (ObjNum 1227 snap 400 seq_num 1227) dirty exists 2024-09-11T03:51:26.362 INFO:tasks.rados.rados.2.smithi175.stdout:3610: left oid 40 (ObjNum 1227 snap 400 seq_num 1227) 2024-09-11T03:51:26.362 INFO:tasks.rados.rados.2.smithi175.stdout:3609: done (6 left) 2024-09-11T03:51:26.362 INFO:tasks.rados.rados.2.smithi175.stdout:3610: done (5 left) 2024-09-11T03:51:26.362 INFO:tasks.rados.rados.2.smithi175.stdout:3615: done (4 left) 2024-09-11T03:51:26.362 INFO:tasks.rados.rados.2.smithi175.stdout:3616: write oid 1 current snap is 400 2024-09-11T03:51:26.362 INFO:tasks.rados.rados.2.smithi175.stdout:3614: finishing write tid 1 to smithi17557870-46 2024-09-11T03:51:26.363 INFO:tasks.rados.rados.2.smithi175.stdout:3612: finishing copy_from racing read to smithi17557870-26 2024-09-11T03:51:26.363 INFO:tasks.rados.rados.2.smithi175.stdout:3614: finishing write tid 2 to smithi17557870-46 2024-09-11T03:51:26.363 INFO:tasks.rados.rados.2.smithi175.stdout:3614: finishing write tid 3 to smithi17557870-46 2024-09-11T03:51:26.363 INFO:tasks.rados.rados.2.smithi175.stdout:3614: finishing write tid 4 to smithi17557870-46 2024-09-11T03:51:26.363 INFO:tasks.rados.rados.2.smithi175.stdout:3614: oid 46 updating version 0 to 975 2024-09-11T03:51:26.363 INFO:tasks.rados.rados.2.smithi175.stdout:3614: oid 46 version 975 is already newer than 973 2024-09-11T03:51:26.363 INFO:tasks.rados.rados.2.smithi175.stdout:3614: oid 46 version 975 is already newer than 974 2024-09-11T03:51:26.363 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 975 (ObjNum 1228 snap 400 seq_num 1228) dirty exists 2024-09-11T03:51:26.363 INFO:tasks.rados.rados.2.smithi175.stdout:3614: left oid 46 (ObjNum 1228 snap 400 seq_num 1228) 2024-09-11T03:51:26.363 INFO:tasks.rados.rados.2.smithi175.stdout:3612: finishing copy_from to smithi17557870-26 2024-09-11T03:51:26.363 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 1011 (ObjNum 1220 snap 399 seq_num 1220) dirty exists 2024-09-11T03:51:26.363 INFO:tasks.rados.rados.2.smithi175.stdout:3613: expect (ObjNum 1059 snap 345 seq_num 1059) 2024-09-11T03:51:26.422 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-11T03:51:26.448 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:26.469 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:26.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:26.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:26.482 INFO:tasks.rados.rados.2.smithi175.stdout:3611: expect (ObjNum 1213 snap 397 seq_num 1213) 2024-09-11T03:51:26.571 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:26.583 INFO:tasks.rados.rados.2.smithi175.stdout:3616: seq_num 1230 ranges {672610=489103,1643440=722432} 2024-09-11T03:51:26.589 INFO:tasks.rados.rados.2.smithi175.stdout:3616: writing smithi17557870-1 from 672610 to 1161713 tid 1 2024-09-11T03:51:26.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:26.594 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:26.595 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:26.596 INFO:tasks.rados.rados.2.smithi175.stdout:3616: writing smithi17557870-1 from 1643440 to 2365872 tid 2 2024-09-11T03:51:26.597 INFO:tasks.rados.rados.2.smithi175.stdout:3611: done (4 left) 2024-09-11T03:51:26.597 INFO:tasks.rados.rados.2.smithi175.stdout:3612: done (3 left) 2024-09-11T03:51:26.597 INFO:tasks.rados.rados.2.smithi175.stdout:3613: done (2 left) 2024-09-11T03:51:26.597 INFO:tasks.rados.rados.2.smithi175.stdout:3614: done (1 left) 2024-09-11T03:51:26.597 INFO:tasks.rados.rados.2.smithi175.stdout:3617: copy_from oid 5 from oid 20 current snap is 400 2024-09-11T03:51:26.597 INFO:tasks.rados.rados.2.smithi175.stdout:3616: finishing write tid 1 to smithi17557870-1 2024-09-11T03:51:26.597 INFO:tasks.rados.rados.2.smithi175.stdout:3618: snap_create 2024-09-11T03:51:26.600 INFO:tasks.rados.rados.2.smithi175.stdout:3616: finishing write tid 2 to smithi17557870-1 2024-09-11T03:51:26.600 INFO:tasks.rados.rados.2.smithi175.stdout:3616: finishing write tid 3 to smithi17557870-1 2024-09-11T03:51:26.600 INFO:tasks.rados.rados.2.smithi175.stdout:3616: finishing write tid 4 to smithi17557870-1 2024-09-11T03:51:26.600 INFO:tasks.rados.rados.2.smithi175.stdout:3616: oid 1 updating version 0 to 1219 2024-09-11T03:51:26.601 INFO:tasks.rados.rados.2.smithi175.stdout:3616: oid 1 version 1219 is already newer than 1217 2024-09-11T03:51:26.601 INFO:tasks.rados.rados.2.smithi175.stdout:3616: oid 1 version 1219 is already newer than 1218 2024-09-11T03:51:26.601 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 1219 (ObjNum 1230 snap 400 seq_num 1230) dirty exists 2024-09-11T03:51:26.601 INFO:tasks.rados.rados.2.smithi175.stdout:3616: left oid 1 (ObjNum 1230 snap 400 seq_num 1230) 2024-09-11T03:51:26.604 INFO:tasks.rados.rados.2.smithi175.stdout:3617: finishing copy_from racing read to smithi17557870-5 2024-09-11T03:51:26.605 INFO:tasks.rados.rados.2.smithi175.stdout:3617: finishing copy_from to smithi17557870-5 2024-09-11T03:51:26.606 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 1006 (ObjNum 973 snap 313 seq_num 973) dirty exists 2024-09-11T03:51:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:26 smithi084 ceph-mon[27254]: 2.4 scrub starts 2024-09-11T03:51:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:26 smithi084 ceph-mon[27254]: 2.4 scrub ok 2024-09-11T03:51:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:26 smithi084 ceph-mon[27254]: pgmap v1455: 107 pgs: 107 active+clean; 697 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 490 KiB/s, 23 objects/s recovering 2024-09-11T03:51:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:26 smithi084 ceph-mon[27254]: 2.13 scrub starts 2024-09-11T03:51:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:26 smithi084 ceph-mon[27254]: 2.13 scrub ok 2024-09-11T03:51:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:26 smithi084 ceph-mon[27254]: osdmap e867: 12 total, 12 up, 11 in 2024-09-11T03:51:26.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:26 smithi084 ceph-mon[27254]: 2.f scrub starts 2024-09-11T03:51:26.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:26 smithi084 ceph-mon[27254]: 2.f scrub ok 2024-09-11T03:51:26.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1956475510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:26.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4115434029' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:51:26.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:26 smithi121 ceph-mon[30729]: 2.4 scrub starts 2024-09-11T03:51:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:26 smithi121 ceph-mon[30729]: 2.4 scrub ok 2024-09-11T03:51:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:26 smithi121 ceph-mon[30729]: pgmap v1455: 107 pgs: 107 active+clean; 697 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 490 KiB/s, 23 objects/s recovering 2024-09-11T03:51:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:26 smithi121 ceph-mon[30729]: 2.13 scrub starts 2024-09-11T03:51:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:26 smithi121 ceph-mon[30729]: 2.13 scrub ok 2024-09-11T03:51:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:26 smithi121 ceph-mon[30729]: osdmap e867: 12 total, 12 up, 11 in 2024-09-11T03:51:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:26 smithi121 ceph-mon[30729]: 2.f scrub starts 2024-09-11T03:51:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:26 smithi121 ceph-mon[30729]: 2.f scrub ok 2024-09-11T03:51:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1956475510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4115434029' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:51:26.673 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:51:26.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:26.718 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:26.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:26.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:26.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:26 smithi175 ceph-mon[30894]: 2.4 scrub starts 2024-09-11T03:51:26.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:26 smithi175 ceph-mon[30894]: 2.4 scrub ok 2024-09-11T03:51:26.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:26 smithi175 ceph-mon[30894]: pgmap v1455: 107 pgs: 107 active+clean; 697 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 490 KiB/s, 23 objects/s recovering 2024-09-11T03:51:26.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:26 smithi175 ceph-mon[30894]: 2.13 scrub starts 2024-09-11T03:51:26.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:26 smithi175 ceph-mon[30894]: 2.13 scrub ok 2024-09-11T03:51:26.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:26 smithi175 ceph-mon[30894]: osdmap e867: 12 total, 12 up, 11 in 2024-09-11T03:51:26.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:26 smithi175 ceph-mon[30894]: 2.f scrub starts 2024-09-11T03:51:26.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:26 smithi175 ceph-mon[30894]: 2.f scrub ok 2024-09-11T03:51:26.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1956475510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:26.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4115434029' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:51:26.820 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:26.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:26.841 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:26.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:26.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:26.964 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:26.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:26.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:27.051 INFO:tasks.rados.rados.2.smithi175.stdout:3616: done (2 left) 2024-09-11T03:51:27.051 INFO:tasks.rados.rados.2.smithi175.stdout:3617: done (1 left) 2024-09-11T03:51:27.051 INFO:tasks.rados.rados.2.smithi175.stdout:3618: done (0 left) 2024-09-11T03:51:27.051 INFO:tasks.rados.rados.2.smithi175.stdout:3619: rollback oid 33 current snap is 401 2024-09-11T03:51:27.051 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 33 to 321 2024-09-11T03:51:27.051 INFO:tasks.rados.rados.2.smithi175.stdout:3620: snap_remove snap 389 2024-09-11T03:51:27.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:27.087 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:27.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:27.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:27.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:27.194 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:51:27.194 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:51:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:27.213 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:27.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:27.314 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:27.335 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:27.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:27.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:27 smithi084 ceph-mon[27254]: 2.31 scrub starts 2024-09-11T03:51:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:27 smithi084 ceph-mon[27254]: 2.31 scrub ok 2024-09-11T03:51:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:27 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:51:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:27 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:51:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:27 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:51:27.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:27 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:51:27.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:27 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4115434029' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:51:27.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:27 smithi084 ceph-mon[27254]: osdmap e868: 12 total, 12 up, 11 in 2024-09-11T03:51:27.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:27 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:51:27.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:27 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:51:27.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]} : dispatch 2024-09-11T03:51:27.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:27 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2e", "id": [7, 0]} : dispatch 2024-09-11T03:51:27.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:27 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]} : dispatch 2024-09-11T03:51:27.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:27 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2e", "id": [7, 0]} : dispatch 2024-09-11T03:51:27.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:27 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]}]': finished 2024-09-11T03:51:27.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:27 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2e", "id": [7, 0]}]': finished 2024-09-11T03:51:27.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:27 smithi084 ceph-mon[27254]: osdmap e869: 12 total, 12 up, 11 in 2024-09-11T03:51:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:27 smithi121 ceph-mon[30729]: 2.31 scrub starts 2024-09-11T03:51:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:27 smithi121 ceph-mon[30729]: 2.31 scrub ok 2024-09-11T03:51:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:27 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:51:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:27 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:51:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:27 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:51:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:27 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:51:27.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:27 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4115434029' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:51:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:27 smithi121 ceph-mon[30729]: osdmap e868: 12 total, 12 up, 11 in 2024-09-11T03:51:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:27 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:51:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:27 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:51:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]} : dispatch 2024-09-11T03:51:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:27 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2e", "id": [7, 0]} : dispatch 2024-09-11T03:51:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:27 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]} : dispatch 2024-09-11T03:51:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:27 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2e", "id": [7, 0]} : dispatch 2024-09-11T03:51:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:27 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]}]': finished 2024-09-11T03:51:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:27 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2e", "id": [7, 0]}]': finished 2024-09-11T03:51:27.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:27 smithi121 ceph-mon[30729]: osdmap e869: 12 total, 12 up, 11 in 2024-09-11T03:51:27.438 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:27.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:27.462 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:27.462 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:27.564 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:27.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:27.588 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:27.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:27.689 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:27.712 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:27.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:27.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:27 smithi175 ceph-mon[30894]: 2.31 scrub starts 2024-09-11T03:51:27.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:27 smithi175 ceph-mon[30894]: 2.31 scrub ok 2024-09-11T03:51:27.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:27 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:51:27.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:27 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:51:27.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:27 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:51:27.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:27 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:51:27.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:27 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4115434029' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:51:27.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:27 smithi175 ceph-mon[30894]: osdmap e868: 12 total, 12 up, 11 in 2024-09-11T03:51:27.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:27 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:51:27.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:27 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:51:27.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]} : dispatch 2024-09-11T03:51:27.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:27 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2e", "id": [7, 0]} : dispatch 2024-09-11T03:51:27.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:27 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]} : dispatch 2024-09-11T03:51:27.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:27 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2e", "id": [7, 0]} : dispatch 2024-09-11T03:51:27.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:27 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 0]}]': finished 2024-09-11T03:51:27.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:27 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2e", "id": [7, 0]}]': finished 2024-09-11T03:51:27.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:27 smithi175 ceph-mon[30894]: osdmap e869: 12 total, 12 up, 11 in 2024-09-11T03:51:27.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:27.838 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:27.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:27.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:27.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:27.962 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:27.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:27.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:28.053 INFO:tasks.rados.rados.2.smithi175.stdout:3620: done (1 left) 2024-09-11T03:51:28.053 INFO:tasks.rados.rados.2.smithi175.stdout:3621: write oid 42 current snap is 401 2024-09-11T03:51:28.053 INFO:tasks.rados.rados.2.smithi175.stdout:3619: finishing rollback tid 0 to smithi17557870-33 2024-09-11T03:51:28.054 INFO:tasks.rados.rados.2.smithi175.stdout:3621: seq_num 1231 ranges {689417=564352,1701922=466090} 2024-09-11T03:51:28.061 INFO:tasks.rados.rados.2.smithi175.stdout:3621: writing smithi17557870-42 from 689417 to 1253769 tid 1 2024-09-11T03:51:28.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:28.067 INFO:tasks.rados.rados.2.smithi175.stdout:3621: writing smithi17557870-42 from 1701922 to 2168012 tid 2 2024-09-11T03:51:28.068 INFO:tasks.rados.rados.2.smithi175.stdout:3622: write (excl) oid 39 current snap is 401 2024-09-11T03:51:28.068 INFO:tasks.rados.rados.2.smithi175.stdout:3619: finishing rollback tid 1 to smithi17557870-33 2024-09-11T03:51:28.068 INFO:tasks.rados.rados.2.smithi175.stdout:3619: finishing rollback tid 2 to smithi17557870-33 2024-09-11T03:51:28.068 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 1427 (ObjNum 964 snap 310 seq_num 964) dirty exists 2024-09-11T03:51:28.068 INFO:tasks.rados.rados.2.smithi175.stdout:3621: finishing write tid 1 to smithi17557870-42 2024-09-11T03:51:28.068 INFO:tasks.rados.rados.2.smithi175.stdout:3622: seq_num 1232 ranges {706224=755954,2160404=545820} 2024-09-11T03:51:28.076 INFO:tasks.rados.rados.2.smithi175.stdout:3622: writing smithi17557870-39 from 706224 to 1462178 tid 1 2024-09-11T03:51:28.083 INFO:tasks.rados.rados.2.smithi175.stdout:3622: writing smithi17557870-39 from 2160404 to 2706224 tid 2 2024-09-11T03:51:28.083 INFO:tasks.rados.rados.2.smithi175.stdout:3619: done (2 left) 2024-09-11T03:51:28.083 INFO:tasks.rados.rados.2.smithi175.stdout:3623: snap_create 2024-09-11T03:51:28.083 INFO:tasks.rados.rados.2.smithi175.stdout:3621: finishing write tid 2 to smithi17557870-42 2024-09-11T03:51:28.083 INFO:tasks.rados.rados.2.smithi175.stdout:3621: finishing write tid 3 to smithi17557870-42 2024-09-11T03:51:28.083 INFO:tasks.rados.rados.2.smithi175.stdout:3621: finishing write tid 4 to smithi17557870-42 2024-09-11T03:51:28.083 INFO:tasks.rados.rados.2.smithi175.stdout:3621: oid 42 updating version 0 to 1070 2024-09-11T03:51:28.084 INFO:tasks.rados.rados.2.smithi175.stdout:3621: oid 42 version 1070 is already newer than 1069 2024-09-11T03:51:28.084 INFO:tasks.rados.rados.2.smithi175.stdout:3621: oid 42 version 1070 is already newer than 1068 2024-09-11T03:51:28.084 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 1070 (ObjNum 1231 snap 401 seq_num 1231) dirty exists 2024-09-11T03:51:28.084 INFO:tasks.rados.rados.2.smithi175.stdout:3621: left oid 42 (ObjNum 1231 snap 401 seq_num 1231) 2024-09-11T03:51:28.084 INFO:tasks.rados.rados.2.smithi175.stdout:3622: finishing write tid 1 to smithi17557870-39 2024-09-11T03:51:28.085 INFO:tasks.rados.rados.2.smithi175.stdout:3622: finishing write tid 2 to smithi17557870-39 2024-09-11T03:51:28.086 INFO:tasks.rados.rados.2.smithi175.stdout:3622: finishing write tid 3 to smithi17557870-39 2024-09-11T03:51:28.086 INFO:tasks.rados.rados.2.smithi175.stdout:3622: finishing write tid 4 to smithi17557870-39 2024-09-11T03:51:28.086 INFO:tasks.rados.rados.2.smithi175.stdout:3622: oid 39 updating version 0 to 1014 2024-09-11T03:51:28.086 INFO:tasks.rados.rados.2.smithi175.stdout:3622: oid 39 version 1014 is already newer than 1013 2024-09-11T03:51:28.086 INFO:tasks.rados.rados.2.smithi175.stdout:3622: oid 39 version 1014 is already newer than 1012 2024-09-11T03:51:28.086 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 1014 (ObjNum 1232 snap 401 seq_num 1232) dirty exists 2024-09-11T03:51:28.086 INFO:tasks.rados.rados.2.smithi175.stdout:3622: left oid 39 (ObjNum 1232 snap 401 seq_num 1232) 2024-09-11T03:51:28.086 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:28.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:28.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:28.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-11T03:51:28.188 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:28.213 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:28.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:28.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:28.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:28.337 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:28.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:28.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:28.364 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:51:28.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:28 smithi121 ceph-mon[30729]: pgmap v1458: 107 pgs: 107 active+clean; 702 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-09-11T03:51:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:28 smithi121 ceph-mon[30729]: 2.2e scrub starts 2024-09-11T03:51:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:28 smithi121 ceph-mon[30729]: 2.2e scrub ok 2024-09-11T03:51:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:28 smithi121 ceph-mon[30729]: 1.18 scrub starts 2024-09-11T03:51:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:28 smithi121 ceph-mon[30729]: 1.18 scrub ok 2024-09-11T03:51:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:28 smithi121 ceph-mon[30729]: osdmap e870: 12 total, 12 up, 11 in 2024-09-11T03:51:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:28 smithi121 ceph-mon[30729]: 1.19 scrub starts 2024-09-11T03:51:28.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:28 smithi121 ceph-mon[30729]: 1.19 scrub ok 2024-09-11T03:51:28.439 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:28.465 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:28.466 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:28.567 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:28.592 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:28.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:28.593 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:28.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:28 smithi084 ceph-mon[27254]: pgmap v1458: 107 pgs: 107 active+clean; 702 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-09-11T03:51:28.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:28 smithi084 ceph-mon[27254]: 2.2e scrub starts 2024-09-11T03:51:28.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:28 smithi084 ceph-mon[27254]: 2.2e scrub ok 2024-09-11T03:51:28.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:28 smithi084 ceph-mon[27254]: 1.18 scrub starts 2024-09-11T03:51:28.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:28 smithi084 ceph-mon[27254]: 1.18 scrub ok 2024-09-11T03:51:28.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:28 smithi084 ceph-mon[27254]: osdmap e870: 12 total, 12 up, 11 in 2024-09-11T03:51:28.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:28 smithi084 ceph-mon[27254]: 1.19 scrub starts 2024-09-11T03:51:28.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:28 smithi084 ceph-mon[27254]: 1.19 scrub ok 2024-09-11T03:51:28.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:28.717 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:28.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:28.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:28.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:28 smithi175 ceph-mon[30894]: pgmap v1458: 107 pgs: 107 active+clean; 702 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-09-11T03:51:28.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:28 smithi175 ceph-mon[30894]: 2.2e scrub starts 2024-09-11T03:51:28.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:28 smithi175 ceph-mon[30894]: 2.2e scrub ok 2024-09-11T03:51:28.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:28 smithi175 ceph-mon[30894]: 1.18 scrub starts 2024-09-11T03:51:28.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:28 smithi175 ceph-mon[30894]: 1.18 scrub ok 2024-09-11T03:51:28.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:28 smithi175 ceph-mon[30894]: osdmap e870: 12 total, 12 up, 11 in 2024-09-11T03:51:28.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:28 smithi175 ceph-mon[30894]: 1.19 scrub starts 2024-09-11T03:51:28.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:28 smithi175 ceph-mon[30894]: 1.19 scrub ok 2024-09-11T03:51:28.810 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:51:28.810 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:51:28.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:28.845 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:28.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:28.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:28.947 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:28.972 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:28.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:28.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:28.983 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 8, 9, 11, 10, 6] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:51:28.983 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-11T03:51:29.054 INFO:tasks.rados.rados.2.smithi175.stdout:3621: done (2 left) 2024-09-11T03:51:29.054 INFO:tasks.rados.rados.2.smithi175.stdout:3622: done (1 left) 2024-09-11T03:51:29.054 INFO:tasks.rados.rados.2.smithi175.stdout:3623: done (0 left) 2024-09-11T03:51:29.054 INFO:tasks.rados.rados.2.smithi175.stdout:3624: delete oid 21 current snap is 402 2024-09-11T03:51:29.060 INFO:tasks.rados.rados.2.smithi175.stdout:3624: done (0 left) 2024-09-11T03:51:29.060 INFO:tasks.rados.rados.2.smithi175.stdout:3625: delete oid 50 current snap is 402 2024-09-11T03:51:29.064 INFO:tasks.rados.rados.2.smithi175.stdout:3625: done (0 left) 2024-09-11T03:51:29.064 INFO:tasks.rados.rados.2.smithi175.stdout:3626: write oid 24 current snap is 402 2024-09-11T03:51:29.064 INFO:tasks.rados.rados.2.smithi175.stdout:3626: seq_num 1235 ranges {756645=581701,1935850=473428} 2024-09-11T03:51:29.071 INFO:tasks.rados.rados.2.smithi175.stdout:3626: writing smithi17557870-24 from 756645 to 1338346 tid 1 2024-09-11T03:51:29.074 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:29.077 INFO:tasks.rados.rados.2.smithi175.stdout:3626: writing smithi17557870-24 from 1935850 to 2409278 tid 2 2024-09-11T03:51:29.078 INFO:tasks.rados.rados.2.smithi175.stdout:3627: snap_create 2024-09-11T03:51:29.078 INFO:tasks.rados.rados.2.smithi175.stdout:3626: finishing write tid 1 to smithi17557870-24 2024-09-11T03:51:29.080 INFO:tasks.rados.rados.2.smithi175.stdout:3626: finishing write tid 2 to smithi17557870-24 2024-09-11T03:51:29.080 INFO:tasks.rados.rados.2.smithi175.stdout:3626: finishing write tid 3 to smithi17557870-24 2024-09-11T03:51:29.080 INFO:tasks.rados.rados.2.smithi175.stdout:3626: finishing write tid 4 to smithi17557870-24 2024-09-11T03:51:29.081 INFO:tasks.rados.rados.2.smithi175.stdout:3626: oid 24 updating version 0 to 1081 2024-09-11T03:51:29.081 INFO:tasks.rados.rados.2.smithi175.stdout:3626: oid 24 version 1081 is already newer than 1080 2024-09-11T03:51:29.081 INFO:tasks.rados.rados.2.smithi175.stdout:3626: oid 24 updating version 1081 to 1082 2024-09-11T03:51:29.081 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 1082 (ObjNum 1235 snap 402 seq_num 1235) dirty exists 2024-09-11T03:51:29.081 INFO:tasks.rados.rados.2.smithi175.stdout:3626: left oid 24 (ObjNum 1235 snap 402 seq_num 1235) 2024-09-11T03:51:29.081 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:51:29.094 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:29.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:29.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:29.196 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:29.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:29.221 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:29.222 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:29.297 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:29.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:29.344 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:29.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:29.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:29.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:29 smithi084 ceph-mon[27254]: 1.3 scrub starts 2024-09-11T03:51:29.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:29 smithi084 ceph-mon[27254]: 1.3 scrub ok 2024-09-11T03:51:29.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:29 smithi084 ceph-mon[27254]: 2.33 scrub starts 2024-09-11T03:51:29.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:29 smithi084 ceph-mon[27254]: 2.33 scrub ok 2024-09-11T03:51:29.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:29 smithi084 ceph-mon[27254]: 2.17 scrub starts 2024-09-11T03:51:29.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:29 smithi084 ceph-mon[27254]: 2.17 scrub ok 2024-09-11T03:51:29.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:29 smithi084 ceph-mon[27254]: osdmap e871: 12 total, 12 up, 11 in 2024-09-11T03:51:29.447 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:29.472 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:29.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:29.474 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:29.531 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:29 smithi121 ceph-mon[30729]: 1.3 scrub starts 2024-09-11T03:51:29.531 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:29 smithi121 ceph-mon[30729]: 1.3 scrub ok 2024-09-11T03:51:29.531 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:29 smithi121 ceph-mon[30729]: 2.33 scrub starts 2024-09-11T03:51:29.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:29 smithi121 ceph-mon[30729]: 2.33 scrub ok 2024-09-11T03:51:29.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:29 smithi121 ceph-mon[30729]: 2.17 scrub starts 2024-09-11T03:51:29.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:29 smithi121 ceph-mon[30729]: 2.17 scrub ok 2024-09-11T03:51:29.532 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:29 smithi121 ceph-mon[30729]: osdmap e871: 12 total, 12 up, 11 in 2024-09-11T03:51:29.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-11T03:51:29.575 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:29.597 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:29.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:29.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:29.699 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:29.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:29.722 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:29.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:29.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:29 smithi175 ceph-mon[30894]: 1.3 scrub starts 2024-09-11T03:51:29.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:29 smithi175 ceph-mon[30894]: 1.3 scrub ok 2024-09-11T03:51:29.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:29 smithi175 ceph-mon[30894]: 2.33 scrub starts 2024-09-11T03:51:29.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:29 smithi175 ceph-mon[30894]: 2.33 scrub ok 2024-09-11T03:51:29.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:29 smithi175 ceph-mon[30894]: 2.17 scrub starts 2024-09-11T03:51:29.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:29 smithi175 ceph-mon[30894]: 2.17 scrub ok 2024-09-11T03:51:29.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:29 smithi175 ceph-mon[30894]: osdmap e871: 12 total, 12 up, 11 in 2024-09-11T03:51:29.798 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:51:29.823 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:29.838 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:29.845 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:29.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:29.875 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:51:29.947 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:29.969 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:29.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:29.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:30.052 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T03:51:30.057 INFO:tasks.rados.rados.2.smithi175.stdout:3626: done (1 left) 2024-09-11T03:51:30.057 INFO:tasks.rados.rados.2.smithi175.stdout:3627: done (0 left) 2024-09-11T03:51:30.058 INFO:tasks.rados.rados.2.smithi175.stdout:3628: snap_remove snap 386 2024-09-11T03:51:30.070 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:30.093 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:30.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:30.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:30.217 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:30.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:30.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:30.277 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:51:30.277 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:51:30.319 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:30.340 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:30.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:30.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:30.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:30.463 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:30.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:30.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:30.566 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:30.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:30.589 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:30.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:30 smithi084 ceph-mon[27254]: 2.35 scrub starts 2024-09-11T03:51:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:30 smithi084 ceph-mon[27254]: 2.35 scrub ok 2024-09-11T03:51:30.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:30 smithi084 ceph-mon[27254]: pgmap v1461: 107 pgs: 1 remapped+peering, 106 active+clean; 717 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 6.2 MiB/s rd, 4 op/s 2024-09-11T03:51:30.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:30 smithi084 ceph-mon[27254]: 1.14 scrub starts 2024-09-11T03:51:30.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:30 smithi084 ceph-mon[27254]: 1.14 scrub ok 2024-09-11T03:51:30.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:30 smithi084 ceph-mon[27254]: 1.1 scrub starts 2024-09-11T03:51:30.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:30 smithi084 ceph-mon[27254]: 1.1 scrub ok 2024-09-11T03:51:30.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:30 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1025670105' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:51:30.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:30 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1025670105' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:51:30.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:30 smithi084 ceph-mon[27254]: osdmap e872: 12 total, 12 up, 11 in 2024-09-11T03:51:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:30 smithi121 ceph-mon[30729]: 2.35 scrub starts 2024-09-11T03:51:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:30 smithi121 ceph-mon[30729]: 2.35 scrub ok 2024-09-11T03:51:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:30 smithi121 ceph-mon[30729]: pgmap v1461: 107 pgs: 1 remapped+peering, 106 active+clean; 717 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 6.2 MiB/s rd, 4 op/s 2024-09-11T03:51:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:30 smithi121 ceph-mon[30729]: 1.14 scrub starts 2024-09-11T03:51:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:30 smithi121 ceph-mon[30729]: 1.14 scrub ok 2024-09-11T03:51:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:30 smithi121 ceph-mon[30729]: 1.1 scrub starts 2024-09-11T03:51:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:30 smithi121 ceph-mon[30729]: 1.1 scrub ok 2024-09-11T03:51:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:30 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1025670105' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:51:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:30 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1025670105' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:51:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:30 smithi121 ceph-mon[30729]: osdmap e872: 12 total, 12 up, 11 in 2024-09-11T03:51:30.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:30.712 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:30.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:30.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:30.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:30 smithi175 ceph-mon[30894]: 2.35 scrub starts 2024-09-11T03:51:30.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:30 smithi175 ceph-mon[30894]: 2.35 scrub ok 2024-09-11T03:51:30.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:30 smithi175 ceph-mon[30894]: pgmap v1461: 107 pgs: 1 remapped+peering, 106 active+clean; 717 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 6.2 MiB/s rd, 4 op/s 2024-09-11T03:51:30.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:30 smithi175 ceph-mon[30894]: 1.14 scrub starts 2024-09-11T03:51:30.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:30 smithi175 ceph-mon[30894]: 1.14 scrub ok 2024-09-11T03:51:30.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:30 smithi175 ceph-mon[30894]: 1.1 scrub starts 2024-09-11T03:51:30.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:30 smithi175 ceph-mon[30894]: 1.1 scrub ok 2024-09-11T03:51:30.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:30 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1025670105' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:51:30.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:30 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1025670105' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:51:30.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:30 smithi175 ceph-mon[30894]: osdmap e872: 12 total, 12 up, 11 in 2024-09-11T03:51:30.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-11T03:51:30.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:30.834 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:30.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:30.936 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:30.957 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:30.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:30.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:31.048 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T03:51:31.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:31.062 INFO:tasks.rados.rados.2.smithi175.stdout:3628: done (0 left) 2024-09-11T03:51:31.062 INFO:tasks.rados.rados.2.smithi175.stdout:3629: read oid 50 snap -1 2024-09-11T03:51:31.062 INFO:tasks.rados.rados.2.smithi175.stdout:3629: expect deleted 2024-09-11T03:51:31.062 INFO:tasks.rados.rados.2.smithi175.stdout:3630: write (excl) oid 34 current snap is 403 2024-09-11T03:51:31.062 INFO:tasks.rados.rados.2.smithi175.stdout:3630: seq_num 1236 ranges {773452=656950,1877979=617086} 2024-09-11T03:51:31.069 INFO:tasks.rados.rados.2.smithi175.stdout:3630: writing smithi17557870-34 from 773452 to 1430402 tid 1 2024-09-11T03:51:31.074 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:31.077 INFO:tasks.rados.rados.2.smithi175.stdout:3630: writing smithi17557870-34 from 1877979 to 2495065 tid 2 2024-09-11T03:51:31.077 INFO:tasks.rados.rados.2.smithi175.stdout:3631: snap_create 2024-09-11T03:51:31.078 INFO:tasks.rados.rados.2.smithi175.stdout:3630: finishing write tid 1 to smithi17557870-34 2024-09-11T03:51:31.080 INFO:tasks.rados.rados.2.smithi175.stdout:3630: finishing write tid 2 to smithi17557870-34 2024-09-11T03:51:31.080 INFO:tasks.rados.rados.2.smithi175.stdout:3630: finishing write tid 3 to smithi17557870-34 2024-09-11T03:51:31.081 INFO:tasks.rados.rados.2.smithi175.stdout:3630: finishing write tid 4 to smithi17557870-34 2024-09-11T03:51:31.081 INFO:tasks.rados.rados.2.smithi175.stdout:3630: oid 34 updating version 0 to 1208 2024-09-11T03:51:31.081 INFO:tasks.rados.rados.2.smithi175.stdout:3630: oid 34 version 1208 is already newer than 1207 2024-09-11T03:51:31.081 INFO:tasks.rados.rados.2.smithi175.stdout:3630: oid 34 version 1208 is already newer than 1206 2024-09-11T03:51:31.081 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 1208 (ObjNum 1236 snap 403 seq_num 1236) dirty exists 2024-09-11T03:51:31.081 INFO:tasks.rados.rados.2.smithi175.stdout:3630: left oid 34 (ObjNum 1236 snap 403 seq_num 1236) 2024-09-11T03:51:31.112 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:31.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:31.114 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:31.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:31.237 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:31.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:31.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:31.296 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:51:31.338 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:31.364 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:31.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:31.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:31.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:31 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:51:31.465 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:31 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:51:31.465 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:31 smithi084 ceph-mon[27254]: osdmap e873: 12 total, 12 up, 11 in 2024-09-11T03:51:31.465 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:31 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4062835673' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:51:31.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:31.486 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:31.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:31.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:31.533 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:31 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:51:31.533 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:31 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:51:31.533 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:31 smithi121 ceph-mon[30729]: osdmap e873: 12 total, 12 up, 11 in 2024-09-11T03:51:31.534 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:31 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4062835673' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:51:31.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:31.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:31.610 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:31.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:31.711 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:31.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:31 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:51:31.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:31 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:51:31.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:31 smithi175 ceph-mon[30894]: osdmap e873: 12 total, 12 up, 11 in 2024-09-11T03:51:31.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:31 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4062835673' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:51:31.733 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:31.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:31.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:31.777 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:51:31.777 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:51:31.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:31.856 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:31.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:31.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:31.888 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:51:31.888 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:51:31.905 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:51:31.967 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:51:31.967 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:51:31.968 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:51:31.968 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:51:31.968 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:51:31.968 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:51:31.969 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:51:31.969 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:31.970 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:51:31.970 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:51:31.987 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:51:31.987 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:51:31.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:31.990 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:31.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:32.004 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:51:32.005 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:51:32.021 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:51:32.021 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:51:32.038 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:51:32.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:51:32.049 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T03:51:32.049 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:51:32.053 INFO:tasks.rados.rados.2.smithi175.stdout:3629: done (2 left) 2024-09-11T03:51:32.053 INFO:tasks.rados.rados.2.smithi175.stdout:3630: done (1 left) 2024-09-11T03:51:32.053 INFO:tasks.rados.rados.2.smithi175.stdout:3631: done (0 left) 2024-09-11T03:51:32.053 INFO:tasks.rados.rados.2.smithi175.stdout:3632: rollback oid 21 current snap is 404 2024-09-11T03:51:32.053 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 21 to 314 2024-09-11T03:51:32.053 INFO:tasks.rados.rados.2.smithi175.stdout:3633: read oid 35 snap -1 2024-09-11T03:51:32.053 INFO:tasks.rados.rados.2.smithi175.stdout:3633: expect deleted 2024-09-11T03:51:32.054 INFO:tasks.rados.rados.2.smithi175.stdout:3634: read oid 23 snap -1 2024-09-11T03:51:32.054 INFO:tasks.rados.rados.2.smithi175.stdout:3634: expect deleted 2024-09-11T03:51:32.054 INFO:tasks.rados.rados.2.smithi175.stdout:3635: delete oid 42 current snap is 404 2024-09-11T03:51:32.058 INFO:tasks.rados.rados.2.smithi175.stdout:3632: finishing rollback tid 1 to smithi17557870-21 2024-09-11T03:51:32.060 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:51:32.060 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:51:32.061 INFO:tasks.rados.rados.2.smithi175.stdout:3632: finishing rollback tid 2 to smithi17557870-21 2024-09-11T03:51:32.061 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 1290 (ObjNum 829 snap 263 seq_num 829) dirty exists 2024-09-11T03:51:32.061 INFO:tasks.rados.rados.2.smithi175.stdout:3632: done (3 left) 2024-09-11T03:51:32.061 INFO:tasks.rados.rados.2.smithi175.stdout:3633: done (2 left) 2024-09-11T03:51:32.061 INFO:tasks.rados.rados.2.smithi175.stdout:3634: done (1 left) 2024-09-11T03:51:32.061 INFO:tasks.rados.rados.2.smithi175.stdout:3635: done (0 left) 2024-09-11T03:51:32.061 INFO:tasks.rados.rados.2.smithi175.stdout:3636: copy_from oid 11 from oid 32 current snap is 404 2024-09-11T03:51:32.061 INFO:tasks.rados.rados.2.smithi175.stdout:3637: snap_create 2024-09-11T03:51:32.069 INFO:tasks.rados.rados.2.smithi175.stdout:3636: finishing copy_from racing read to smithi17557870-11 2024-09-11T03:51:32.076 INFO:tasks.rados.rados.2.smithi175.stdout:3636: finishing copy_from to smithi17557870-11 2024-09-11T03:51:32.076 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 1020 (ObjNum 1059 snap 345 seq_num 1059) dirty exists 2024-09-11T03:51:32.080 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:51:32.080 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:51:32.092 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:32.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:32.114 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:32.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:32.216 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:32.240 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:32.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:32.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:32.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:32.364 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:32.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:32.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:32.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:32 smithi084 ceph-mon[27254]: pgmap v1464: 107 pgs: 1 remapped+peering, 106 active+clean; 717 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 11 op/s 2024-09-11T03:51:32.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:32 smithi084 ceph-mon[27254]: 1.4 scrub starts 2024-09-11T03:51:32.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:32 smithi084 ceph-mon[27254]: 1.4 scrub ok 2024-09-11T03:51:32.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:32 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1111418275' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:32.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:32 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1884260352' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:51:32.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:32 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:32.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:32 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4062835673' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:51:32.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:32 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1884260352' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:51:32.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:32 smithi084 ceph-mon[27254]: osdmap e874: 12 total, 12 up, 11 in 2024-09-11T03:51:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:32 smithi121 ceph-mon[30729]: pgmap v1464: 107 pgs: 1 remapped+peering, 106 active+clean; 717 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 11 op/s 2024-09-11T03:51:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:32 smithi121 ceph-mon[30729]: 1.4 scrub starts 2024-09-11T03:51:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:32 smithi121 ceph-mon[30729]: 1.4 scrub ok 2024-09-11T03:51:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:32 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1111418275' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:32 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1884260352' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:51:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:32 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:32 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4062835673' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:51:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:32 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1884260352' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:51:32.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:32 smithi121 ceph-mon[30729]: osdmap e874: 12 total, 12 up, 11 in 2024-09-11T03:51:32.466 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:32.487 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:32.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:32.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:32.589 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:32.616 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:32.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:32.617 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:32.718 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:32.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:32 smithi175 ceph-mon[30894]: pgmap v1464: 107 pgs: 1 remapped+peering, 106 active+clean; 717 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 11 op/s 2024-09-11T03:51:32.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:32 smithi175 ceph-mon[30894]: 1.4 scrub starts 2024-09-11T03:51:32.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:32 smithi175 ceph-mon[30894]: 1.4 scrub ok 2024-09-11T03:51:32.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:32 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1111418275' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:32.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:32 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1884260352' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:51:32.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:32 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:32.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:32 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4062835673' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:51:32.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:32 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1884260352' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:51:32.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:32 smithi175 ceph-mon[30894]: osdmap e874: 12 total, 12 up, 11 in 2024-09-11T03:51:32.742 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:32.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:32.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:32.844 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:32.868 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:32.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:32.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:32.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T03:51:32.971 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:32.996 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:32.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:32.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:33.056 INFO:tasks.rados.rados.2.smithi175.stdout:3636: done (1 left) 2024-09-11T03:51:33.056 INFO:tasks.rados.rados.2.smithi175.stdout:3637: done (0 left) 2024-09-11T03:51:33.057 INFO:tasks.rados.rados.2.smithi175.stdout:3638: copy_from oid 34 from oid 39 current snap is 405 2024-09-11T03:51:33.057 INFO:tasks.rados.rados.2.smithi175.stdout:3639: write oid 44 current snap is 405 2024-09-11T03:51:33.057 INFO:tasks.rados.rados.2.smithi175.stdout:3639: seq_num 1238 ranges {407066=407448,1594943=620755} 2024-09-11T03:51:33.061 INFO:tasks.rados.rados.2.smithi175.stdout:3639: writing smithi17557870-44 from 407066 to 814514 tid 1 2024-09-11T03:51:33.069 INFO:tasks.rados.rados.2.smithi175.stdout:3639: writing smithi17557870-44 from 1594943 to 2215698 tid 2 2024-09-11T03:51:33.069 INFO:tasks.rados.rados.2.smithi175.stdout:3640: snap_remove snap 395 2024-09-11T03:51:33.098 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:33.123 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:33.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:33.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:33.155 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:51:33.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:33.245 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:33.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:33.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:33.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:33.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:33.373 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:33.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:33.474 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:33.498 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:33.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:33.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:33.602 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:33.627 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:33.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:33.629 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:33.659 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:51:33.660 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:51:33.660 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:51:33.660 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:51:33.660 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:51:33.729 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:33.755 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:33.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:33.756 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:33.857 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:33.882 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:33.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:33.883 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:33.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:34.006 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:34.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:34.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:34.062 INFO:tasks.rados.rados.2.smithi175.stdout:3640: done (2 left) 2024-09-11T03:51:34.062 INFO:tasks.rados.rados.2.smithi175.stdout:3641: write (excl) oid 41 current snap is 405 2024-09-11T03:51:34.062 INFO:tasks.rados.rados.2.smithi175.stdout:3639: finishing write tid 1 to smithi17557870-44 2024-09-11T03:51:34.063 INFO:tasks.rados.rados.2.smithi175.stdout:3641: seq_num 1239 ranges {423873=482697,1653425=480766,2691441=132432} 2024-09-11T03:51:34.067 INFO:tasks.rados.rados.2.smithi175.stdout:3641: writing smithi17557870-41 from 423873 to 906570 tid 1 2024-09-11T03:51:34.075 INFO:tasks.rados.rados.2.smithi175.stdout:3641: writing smithi17557870-41 from 1653425 to 2134191 tid 2 2024-09-11T03:51:34.079 INFO:tasks.rados.rados.2.smithi175.stdout:3641: writing smithi17557870-41 from 2691441 to 2823873 tid 3 2024-09-11T03:51:34.079 INFO:tasks.rados.rados.2.smithi175.stdout:3642: snap_remove snap 396 2024-09-11T03:51:34.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:34.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:34.129 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:34.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:34.215 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-11T03:51:34.230 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:34.281 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:34.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:34.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:34.383 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:34.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:34 smithi084 ceph-mon[27254]: pgmap v1467: 107 pgs: 1 peering, 106 active+clean; 724 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 7 op/s; 6.0 MiB/s, 13 objects/s recovering 2024-09-11T03:51:34.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:34 smithi084 ceph-mon[27254]: osdmap e875: 12 total, 12 up, 11 in 2024-09-11T03:51:34.404 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:34.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:34.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:34 smithi121 ceph-mon[30729]: pgmap v1467: 107 pgs: 1 peering, 106 active+clean; 724 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 7 op/s; 6.0 MiB/s, 13 objects/s recovering 2024-09-11T03:51:34.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:34 smithi121 ceph-mon[30729]: osdmap e875: 12 total, 12 up, 11 in 2024-09-11T03:51:34.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T03:51:34.464 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:34 smithi175 ceph-mon[30894]: pgmap v1467: 107 pgs: 1 peering, 106 active+clean; 724 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 7 op/s; 6.0 MiB/s, 13 objects/s recovering 2024-09-11T03:51:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:34 smithi175 ceph-mon[30894]: osdmap e875: 12 total, 12 up, 11 in 2024-09-11T03:51:34.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:34.527 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:34.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:34.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:34.629 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:34.655 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:34.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:34.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:34.690 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:51:34.757 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:34.785 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:34.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:34.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:34.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:34.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:34.910 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:34.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:35.011 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:35.037 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:35.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:35.038 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:35.074 INFO:tasks.rados.rados.2.smithi175.stdout:3642: done (3 left) 2024-09-11T03:51:35.074 INFO:tasks.rados.rados.2.smithi175.stdout:3643: copy_from oid 7 from oid 4 current snap is 405 2024-09-11T03:51:35.074 INFO:tasks.rados.rados.2.smithi175.stdout:3644: copy_from oid 47 from oid 37 current snap is 405 2024-09-11T03:51:35.074 INFO:tasks.rados.rados.2.smithi175.stdout:3645: snap_remove snap 379 2024-09-11T03:51:35.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:35.160 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:35.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:35.236 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:51:35.258 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:35 smithi084 ceph-mon[27254]: osdmap e876: 12 total, 12 up, 11 in 2024-09-11T03:51:35.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:35.283 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:35.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:35.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:35.334 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:35 smithi121 ceph-mon[30729]: osdmap e876: 12 total, 12 up, 11 in 2024-09-11T03:51:35.360 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:51:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:51:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:51:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:51:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3040 2.d 2:b6a25a79:::smithi17557870-47:head [read 0~2201783,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:51:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:51:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 523.43658231999996, 2024-09-11T03:51:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020060432, 2024-09-11T03:51:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3040 2024-09-11T03:51:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:51:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:51:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896981+0000", 2024-09-11T03:51:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.747e-06 2024-09-11T03:51:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896989+0000", 2024-09-11T03:51:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9559999999999997e-06 2024-09-11T03:51:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897005+0000", 2024-09-11T03:51:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5704000000000001e-05 2024-09-11T03:51:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897013+0000", 2024-09-11T03:51:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.028e-06 2024-09-11T03:51:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897078+0000", 2024-09-11T03:51:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4813000000000004e-05 2024-09-11T03:51:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:51:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897083+0000", 2024-09-11T03:51:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1109999999999999e-06 2024-09-11T03:51:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.914637+0000", 2024-09-11T03:51:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017553701000000001 2024-09-11T03:51:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.914744+0000", 2024-09-11T03:51:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107131 2024-09-11T03:51:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917040+0000", 2024-09-11T03:51:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022962410000000001 2024-09-11T03:51:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3041 2.d 2:b6a25a79:::smithi17557870-47:head [sparse-read 0~1480887] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:51:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:51:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 523.43651676399998, 2024-09-11T03:51:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020163026000000001, 2024-09-11T03:51:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.366 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3041 2024-09-11T03:51:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:51:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:51:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897046+0000", 2024-09-11T03:51:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7e-07 2024-09-11T03:51:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897048+0000", 2024-09-11T03:51:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0700000000000001e-06 2024-09-11T03:51:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897058+0000", 2024-09-11T03:51:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0151e-05 2024-09-11T03:51:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897068+0000", 2024-09-11T03:51:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0329000000000001e-05 2024-09-11T03:51:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897128+0000", 2024-09-11T03:51:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0208999999999999e-05 2024-09-11T03:51:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:51:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897132+0000", 2024-09-11T03:51:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7290000000000002e-06 2024-09-11T03:51:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917070+0000", 2024-09-11T03:51:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019937951999999998 2024-09-11T03:51:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:35.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917123+0000", 2024-09-11T03:51:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3149000000000002e-05 2024-09-11T03:51:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917208+0000", 2024-09-11T03:51:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5067000000000004e-05 2024-09-11T03:51:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3042 2.d 2:b6a25a79:::smithi17557870-47:head [sparse-read 0~1480887] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:51:35.371 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:51:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 523.43648258500002, 2024-09-11T03:51:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020302042999999999, 2024-09-11T03:51:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:35.372 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3042 2024-09-11T03:51:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:51:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:51:35.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897080+0000", 2024-09-11T03:51:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.82e-07 2024-09-11T03:51:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897082+0000", 2024-09-11T03:51:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.418e-06 2024-09-11T03:51:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897089+0000", 2024-09-11T03:51:35.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8090000000000004e-06 2024-09-11T03:51:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897094+0000", 2024-09-11T03:51:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2399999999999998e-06 2024-09-11T03:51:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:51:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0928999999999999e-05 2024-09-11T03:51:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:51:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897158+0000", 2024-09-11T03:51:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9890000000000001e-06 2024-09-11T03:51:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917227+0000", 2024-09-11T03:51:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020069019 2024-09-11T03:51:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917290+0000", 2024-09-11T03:51:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2848999999999994e-05 2024-09-11T03:51:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917381+0000", 2024-09-11T03:51:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1308000000000001e-05 2024-09-11T03:51:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3046 2.d 2:b6874adf:::smithi17557870-17:head [sparse-read 0~1769472,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:51:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:51:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 523.43646062400001, 2024-09-11T03:51:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021672665000000001, 2024-09-11T03:51:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:35.382 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3046 2024-09-11T03:51:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:51:35.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:51:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897102+0000", 2024-09-11T03:51:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.65e-07 2024-09-11T03:51:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897105+0000", 2024-09-11T03:51:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1820000000000002e-06 2024-09-11T03:51:35.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897111+0000", 2024-09-11T03:51:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4330000000000001e-06 2024-09-11T03:51:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897117+0000", 2024-09-11T03:51:35.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2979999999999999e-06 2024-09-11T03:51:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897179+0000", 2024-09-11T03:51:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2825999999999998e-05 2024-09-11T03:51:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:51:35.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897182+0000", 2024-09-11T03:51:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7769999999999998e-06 2024-09-11T03:51:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917399+0000", 2024-09-11T03:51:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020216622 2024-09-11T03:51:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917444+0000", 2024-09-11T03:51:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5558e-05 2024-09-11T03:51:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918774+0000", 2024-09-11T03:51:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013296040000000001 2024-09-11T03:51:35.389 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3047 2.d 2:b6874adf:::smithi17557870-17:head [read 0~2453888,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:51:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:51:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 523.43643145199997, 2024-09-11T03:51:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022691901, 2024-09-11T03:51:35.390 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3047 2024-09-11T03:51:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.391 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:51:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:51:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:51:35.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8599999999999998e-07 2024-09-11T03:51:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897134+0000", 2024-09-11T03:51:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1470000000000001e-06 2024-09-11T03:51:35.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897141+0000", 2024-09-11T03:51:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8029999999999999e-06 2024-09-11T03:51:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897147+0000", 2024-09-11T03:51:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4520000000000002e-06 2024-09-11T03:51:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897202+0000", 2024-09-11T03:51:35.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4255999999999997e-05 2024-09-11T03:51:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:51:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897204+0000", 2024-09-11T03:51:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6649999999999999e-06 2024-09-11T03:51:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918807+0000", 2024-09-11T03:51:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021602962999999999 2024-09-11T03:51:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918887+0000", 2024-09-11T03:51:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9550999999999998e-05 2024-09-11T03:51:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919822+0000", 2024-09-11T03:51:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093557799999999995 2024-09-11T03:51:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3048 2.d 2:b6874adf:::smithi17557870-17:head [read 0~2453888,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:51:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:51:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 523.43640662799999, 2024-09-11T03:51:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.023274353000000001, 2024-09-11T03:51:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.398 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:35.399 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.399 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.399 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.399 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3048 2024-09-11T03:51:35.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.399 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:51:35.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:51:35.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897156+0000", 2024-09-11T03:51:35.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1199999999999998e-07 2024-09-11T03:51:35.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897159+0000", 2024-09-11T03:51:35.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7439999999999999e-06 2024-09-11T03:51:35.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897165+0000", 2024-09-11T03:51:35.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6749999999999996e-06 2024-09-11T03:51:35.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897170+0000", 2024-09-11T03:51:35.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9420000000000003e-06 2024-09-11T03:51:35.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897226+0000", 2024-09-11T03:51:35.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6026999999999997e-05 2024-09-11T03:51:35.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:51:35.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897232+0000", 2024-09-11T03:51:35.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2609999999999999e-06 2024-09-11T03:51:35.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919841+0000", 2024-09-11T03:51:35.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022609054999999999 2024-09-11T03:51:35.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:35.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919885+0000", 2024-09-11T03:51:35.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3383999999999997e-05 2024-09-11T03:51:35.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.920430+0000", 2024-09-11T03:51:35.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054485299999999998 2024-09-11T03:51:35.403 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.403 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.403 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.403 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4176 2.3 2:c6c49f2c:::smithi17557870-44:head [copy-from ver 412] snapc b5=[b5,b4,b2,b1,af,ae,ac,ab,a9,a8,9d] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-11T03:51:35.403 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:28.933487+0000", 2024-09-11T03:51:35.403 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 426.400075421, 2024-09-11T03:51:35.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018627511999999999, 2024-09-11T03:51:35.404 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.404 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.404 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:35.404 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.404 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.404 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.404 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4176 2024-09-11T03:51:35.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.404 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933487+0000", 2024-09-11T03:51:35.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933487+0000", 2024-09-11T03:51:35.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933489+0000", 2024-09-11T03:51:35.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2510000000000001e-06 2024-09-11T03:51:35.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933498+0000", 2024-09-11T03:51:35.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3219999999999999e-06 2024-09-11T03:51:35.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933516+0000", 2024-09-11T03:51:35.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7971000000000001e-05 2024-09-11T03:51:35.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933529+0000", 2024-09-11T03:51:35.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2476e-05 2024-09-11T03:51:35.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933657+0000", 2024-09-11T03:51:35.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000128842 2024-09-11T03:51:35.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:51:35.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933663+0000", 2024-09-11T03:51:35.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9580000000000004e-06 2024-09-11T03:51:35.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935429+0000", 2024-09-11T03:51:35.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001765146 2024-09-11T03:51:35.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:35.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935549+0000", 2024-09-11T03:51:35.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120504 2024-09-11T03:51:35.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:51:35.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.944235+0000", 2024-09-11T03:51:35.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086856499999999996 2024-09-11T03:51:35.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:51:35.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948823+0000", 2024-09-11T03:51:35.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045879570000000001 2024-09-11T03:51:35.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:35.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.951466+0000", 2024-09-11T03:51:35.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026434739999999998 2024-09-11T03:51:35.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:35.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.952026+0000", 2024-09-11T03:51:35.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055985599999999996 2024-09-11T03:51:35.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:51:35.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.952068+0000", 2024-09-11T03:51:35.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1918000000000001e-05 2024-09-11T03:51:35.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.952114+0000", 2024-09-11T03:51:35.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6186999999999997e-05 2024-09-11T03:51:35.410 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.410 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.410 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.410 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4177 2.3 2:c6c49f2c:::smithi17557870-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e451)", 2024-09-11T03:51:35.410 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:28.933561+0000", 2024-09-11T03:51:35.411 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 426.40000060800003, 2024-09-11T03:51:35.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014798686, 2024-09-11T03:51:35.411 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.411 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.411 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:35.411 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.411 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.411 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.411 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4177 2024-09-11T03:51:35.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.411 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933561+0000", 2024-09-11T03:51:35.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933561+0000", 2024-09-11T03:51:35.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933562+0000", 2024-09-11T03:51:35.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4600000000000005e-07 2024-09-11T03:51:35.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933565+0000", 2024-09-11T03:51:35.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8729999999999999e-06 2024-09-11T03:51:35.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933572+0000", 2024-09-11T03:51:35.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6090000000000003e-06 2024-09-11T03:51:35.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933579+0000", 2024-09-11T03:51:35.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4339999999999997e-06 2024-09-11T03:51:35.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933717+0000", 2024-09-11T03:51:35.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000137995 2024-09-11T03:51:35.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:51:35.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933721+0000", 2024-09-11T03:51:35.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6940000000000001e-06 2024-09-11T03:51:35.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935773+0000", 2024-09-11T03:51:35.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002052398 2024-09-11T03:51:35.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:51:35.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935834+0000", 2024-09-11T03:51:35.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1146e-05 2024-09-11T03:51:35.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948174+0000", 2024-09-11T03:51:35.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012339786 2024-09-11T03:51:35.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:35.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948263+0000", 2024-09-11T03:51:35.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8838999999999994e-05 2024-09-11T03:51:35.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948360+0000", 2024-09-11T03:51:35.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7366000000000003e-05 2024-09-11T03:51:35.416 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.416 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.416 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.416 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:6215 2.16 2:6dbd1e81:::smithi17557870-7:head [copy-from ver 268] snapc 112=[112,111,10f] ondisk+write+known_if_redirected+supports_pool_eio e646)", 2024-09-11T03:51:35.417 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:43.987602+0000", 2024-09-11T03:51:35.417 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 231.34596021799999, 2024-09-11T03:51:35.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021363759, 2024-09-11T03:51:35.417 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.417 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.417 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:35.417 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.417 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.417 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.417 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 6215 2024-09-11T03:51:35.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.417 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987602+0000", 2024-09-11T03:51:35.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987602+0000", 2024-09-11T03:51:35.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987603+0000", 2024-09-11T03:51:35.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6339999999999999e-06 2024-09-11T03:51:35.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987612+0000", 2024-09-11T03:51:35.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1170000000000004e-06 2024-09-11T03:51:35.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987631+0000", 2024-09-11T03:51:35.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9245e-05 2024-09-11T03:51:35.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987641+0000", 2024-09-11T03:51:35.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8570000000000007e-06 2024-09-11T03:51:35.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987718+0000", 2024-09-11T03:51:35.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7812999999999995e-05 2024-09-11T03:51:35.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:51:35.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987726+0000", 2024-09-11T03:51:35.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1960000000000002e-06 2024-09-11T03:51:35.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.989804+0000", 2024-09-11T03:51:35.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002078004 2024-09-11T03:51:35.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:35.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.989909+0000", 2024-09-11T03:51:35.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105518 2024-09-11T03:51:35.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6,11", 2024-09-11T03:51:35.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.999305+0000", 2024-09-11T03:51:35.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.009396349 2024-09-11T03:51:35.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:51:35.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.007386+0000", 2024-09-11T03:51:35.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080807069999999995 2024-09-11T03:51:35.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:35.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.007696+0000", 2024-09-11T03:51:35.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030955000000000002 2024-09-11T03:51:35.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:35.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.008885+0000", 2024-09-11T03:51:35.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011892700000000001 2024-09-11T03:51:35.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:51:35.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.008929+0000", 2024-09-11T03:51:35.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4322e-05 2024-09-11T03:51:35.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.008966+0000", 2024-09-11T03:51:35.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6177000000000003e-05 2024-09-11T03:51:35.424 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.424 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.425 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.425 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:6216 2.16 2:6dbd1e81:::smithi17557870-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e646)", 2024-09-11T03:51:35.425 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:43.987704+0000", 2024-09-11T03:51:35.425 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 231.34585819599999, 2024-09-11T03:51:35.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016255282999999999, 2024-09-11T03:51:35.426 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.426 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.426 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:35.426 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.426 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.426 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.426 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 6216 2024-09-11T03:51:35.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.427 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987704+0000", 2024-09-11T03:51:35.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987704+0000", 2024-09-11T03:51:35.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987704+0000", 2024-09-11T03:51:35.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5000000000000002e-07 2024-09-11T03:51:35.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987708+0000", 2024-09-11T03:51:35.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6610000000000002e-06 2024-09-11T03:51:35.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987718+0000", 2024-09-11T03:51:35.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7690000000000008e-06 2024-09-11T03:51:35.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987725+0000", 2024-09-11T03:51:35.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7279999999999998e-06 2024-09-11T03:51:35.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987785+0000", 2024-09-11T03:51:35.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0163e-05 2024-09-11T03:51:35.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:51:35.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987790+0000", 2024-09-11T03:51:35.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2410000000000003e-06 2024-09-11T03:51:35.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.990032+0000", 2024-09-11T03:51:35.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022420880000000002 2024-09-11T03:51:35.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:51:35.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.990058+0000", 2024-09-11T03:51:35.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6044999999999999e-05 2024-09-11T03:51:35.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.003794+0000", 2024-09-11T03:51:35.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01373564 2024-09-11T03:51:35.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:35.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.003878+0000", 2024-09-11T03:51:35.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3954000000000003e-05 2024-09-11T03:51:35.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.003959+0000", 2024-09-11T03:51:35.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1343999999999999e-05 2024-09-11T03:51:35.434 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.434 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.434 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.434 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:7056 2.20 e710/574 2:0762776a:::smithi17557870-25:head v 710'215, mlcod=628'211)", 2024-09-11T03:51:35.434 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:48.028737+0000", 2024-09-11T03:51:35.434 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 167.30482495199999, 2024-09-11T03:51:35.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015591568, 2024-09-11T03:51:35.434 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.434 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.434 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:35.434 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.028737+0000", 2024-09-11T03:51:35.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.028737+0000", 2024-09-11T03:51:35.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.028737+0000", 2024-09-11T03:51:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0600000000000001e-07 2024-09-11T03:51:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032034+0000", 2024-09-11T03:51:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032964890000000001 2024-09-11T03:51:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032066+0000", 2024-09-11T03:51:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2011999999999998e-05 2024-09-11T03:51:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032079+0000", 2024-09-11T03:51:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2702e-05 2024-09-11T03:51:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032193+0000", 2024-09-11T03:51:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114737 2024-09-11T03:51:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032231+0000", 2024-09-11T03:51:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7867000000000002e-05 2024-09-11T03:51:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:51:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.044229+0000", 2024-09-11T03:51:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011997499999999999 2024-09-11T03:51:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.044329+0000", 2024-09-11T03:51:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9854999999999999e-05 2024-09-11T03:51:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.439 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7213 2.d 2:b6874adf:::smithi17557870-17:head [write 2124737~564614] snapc 141=[141,140,13f,13d,13c,139,138,137,136,135,133,12e,12d,12a,127,126,120,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-09-11T03:51:35.439 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:01.038236+0000", 2024-09-11T03:51:35.439 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 154.295326222, 2024-09-11T03:51:35.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014496043, 2024-09-11T03:51:35.439 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.439 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.439 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:35.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.439 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7213 2024-09-11T03:51:35.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.440 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038236+0000", 2024-09-11T03:51:35.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038236+0000", 2024-09-11T03:51:35.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038237+0000", 2024-09-11T03:51:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.153e-06 2024-09-11T03:51:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038729+0000", 2024-09-11T03:51:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049247500000000003 2024-09-11T03:51:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038742+0000", 2024-09-11T03:51:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2169e-05 2024-09-11T03:51:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038747+0000", 2024-09-11T03:51:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6099999999999997e-06 2024-09-11T03:51:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038800+0000", 2024-09-11T03:51:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2920999999999997e-05 2024-09-11T03:51:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038911+0000", 2024-09-11T03:51:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011138599999999999 2024-09-11T03:51:35.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:51:35.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.039079+0000", 2024-09-11T03:51:35.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000167928 2024-09-11T03:51:35.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:51:35.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.040472+0000", 2024-09-11T03:51:35.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001392826 2024-09-11T03:51:35.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:35.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.040615+0000", 2024-09-11T03:51:35.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014295700000000001 2024-09-11T03:51:35.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:35.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.052594+0000", 2024-09-11T03:51:35.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011979001 2024-09-11T03:51:35.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:51:35.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.052675+0000", 2024-09-11T03:51:35.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0611999999999993e-05 2024-09-11T03:51:35.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.052732+0000", 2024-09-11T03:51:35.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7005000000000001e-05 2024-09-11T03:51:35.445 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.445 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.445 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.445 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7215 2.d 2:b6874adf:::smithi17557870-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e723)", 2024-09-11T03:51:35.445 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:01.038790+0000", 2024-09-11T03:51:35.445 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 154.29477240200001, 2024-09-11T03:51:35.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014304828, 2024-09-11T03:51:35.445 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.445 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.445 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:35.446 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.446 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.446 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.446 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7215 2024-09-11T03:51:35.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.446 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038790+0000", 2024-09-11T03:51:35.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038790+0000", 2024-09-11T03:51:35.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038790+0000", 2024-09-11T03:51:35.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9499999999999998e-07 2024-09-11T03:51:35.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038791+0000", 2024-09-11T03:51:35.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.544e-06 2024-09-11T03:51:35.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038800+0000", 2024-09-11T03:51:35.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4309999999999994e-06 2024-09-11T03:51:35.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038803+0000", 2024-09-11T03:51:35.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3739999999999998e-06 2024-09-11T03:51:35.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.040394+0000", 2024-09-11T03:51:35.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001591189 2024-09-11T03:51:35.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:51:35.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.040462+0000", 2024-09-11T03:51:35.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8053999999999996e-05 2024-09-11T03:51:35.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.052896+0000", 2024-09-11T03:51:35.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012434031 2024-09-11T03:51:35.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:35.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.052994+0000", 2024-09-11T03:51:35.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7693000000000001e-05 2024-09-11T03:51:35.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.053094+0000", 2024-09-11T03:51:35.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010021700000000001 2024-09-11T03:51:35.450 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.450 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.450 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.450 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7230 2.d 2:b1c0f730:::smithi17557870-20:head [copy-from ver 956] snapc 141=[141,140,13f,13d,13c,139,138,137,136,135,133,12e,12d,12a,127,126,120,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-09-11T03:51:35.450 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:01.070035+0000", 2024-09-11T03:51:35.451 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 154.26352739999999, 2024-09-11T03:51:35.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014365849999999999, 2024-09-11T03:51:35.451 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.451 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.451 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:35.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.451 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7230 2024-09-11T03:51:35.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.451 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070035+0000", 2024-09-11T03:51:35.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070035+0000", 2024-09-11T03:51:35.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070036+0000", 2024-09-11T03:51:35.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6559999999999999e-06 2024-09-11T03:51:35.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070047+0000", 2024-09-11T03:51:35.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.047e-05 2024-09-11T03:51:35.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070063+0000", 2024-09-11T03:51:35.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6310000000000001e-05 2024-09-11T03:51:35.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070072+0000", 2024-09-11T03:51:35.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9819999999999997e-06 2024-09-11T03:51:35.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070145+0000", 2024-09-11T03:51:35.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3323000000000003e-05 2024-09-11T03:51:35.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:35.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070237+0000", 2024-09-11T03:51:35.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1266000000000003e-05 2024-09-11T03:51:35.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:51:35.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.076752+0000", 2024-09-11T03:51:35.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065155760000000004 2024-09-11T03:51:35.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:35.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.083798+0000", 2024-09-11T03:51:35.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070462249999999997 2024-09-11T03:51:35.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:51:35.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.083901+0000", 2024-09-11T03:51:35.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103115 2024-09-11T03:51:35.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:35.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.084281+0000", 2024-09-11T03:51:35.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037937799999999999 2024-09-11T03:51:35.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:51:35.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.084346+0000", 2024-09-11T03:51:35.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4817e-05 2024-09-11T03:51:35.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.084400+0000", 2024-09-11T03:51:35.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4731999999999997e-05 2024-09-11T03:51:35.460 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.460 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.460 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.460 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8290 2.3b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e811)", 2024-09-11T03:51:35.460 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:50:29.122470+0000", 2024-09-11T03:51:35.460 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 66.211092405000002, 2024-09-11T03:51:35.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92349315799999998, 2024-09-11T03:51:35.461 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.461 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.461 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-11T03:51:35.461 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.461 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.461 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.461 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8290 2024-09-11T03:51:35.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.462 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122470+0000", 2024-09-11T03:51:35.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122470+0000", 2024-09-11T03:51:35.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122471+0000", 2024-09-11T03:51:35.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.669e-06 2024-09-11T03:51:35.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122478+0000", 2024-09-11T03:51:35.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3550000000000001e-06 2024-09-11T03:51:35.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122493+0000", 2024-09-11T03:51:35.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5498999999999998e-05 2024-09-11T03:51:35.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122501+0000", 2024-09-11T03:51:35.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.61e-06 2024-09-11T03:51:35.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122568+0000", 2024-09-11T03:51:35.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7465999999999994e-05 2024-09-11T03:51:35.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-09-11T03:51:35.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122574+0000", 2024-09-11T03:51:35.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7139999999999998e-06 2024-09-11T03:51:35.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:30.045949+0000", 2024-09-11T03:51:35.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92337486199999996 2024-09-11T03:51:35.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:30.045963+0000", 2024-09-11T03:51:35.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3983e-05 2024-09-11T03:51:35.466 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.467 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.467 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.467 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8291 2.3b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e811)", 2024-09-11T03:51:35.467 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:50:29.122520+0000", 2024-09-11T03:51:35.467 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 66.211042126999999, 2024-09-11T03:51:35.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92350952100000006, 2024-09-11T03:51:35.467 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.467 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.467 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-11T03:51:35.467 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.467 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.467 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.468 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8291 2024-09-11T03:51:35.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.468 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122520+0000", 2024-09-11T03:51:35.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122520+0000", 2024-09-11T03:51:35.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122520+0000", 2024-09-11T03:51:35.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3300000000000002e-07 2024-09-11T03:51:35.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122523+0000", 2024-09-11T03:51:35.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.604e-06 2024-09-11T03:51:35.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122533+0000", 2024-09-11T03:51:35.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0338e-05 2024-09-11T03:51:35.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122538+0000", 2024-09-11T03:51:35.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2490000000000003e-06 2024-09-11T03:51:35.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122601+0000", 2024-09-11T03:51:35.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3789999999999997e-05 2024-09-11T03:51:35.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-09-11T03:51:35.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122606+0000", 2024-09-11T03:51:35.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5889999999999996e-06 2024-09-11T03:51:35.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:30.046018+0000", 2024-09-11T03:51:35.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92341183699999996 2024-09-11T03:51:35.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:30.046029+0000", 2024-09-11T03:51:35.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1581e-05 2024-09-11T03:51:35.471 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.472 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.472 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.472 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8292 2.3b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e811)", 2024-09-11T03:51:35.472 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:50:29.122558+0000", 2024-09-11T03:51:35.472 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 66.211004345000006, 2024-09-11T03:51:35.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92350303600000005, 2024-09-11T03:51:35.472 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.472 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.472 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-11T03:51:35.472 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.472 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.473 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.473 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8292 2024-09-11T03:51:35.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.473 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122558+0000", 2024-09-11T03:51:35.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122558+0000", 2024-09-11T03:51:35.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122558+0000", 2024-09-11T03:51:35.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0100000000000005e-07 2024-09-11T03:51:35.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122560+0000", 2024-09-11T03:51:35.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9999999999999999e-06 2024-09-11T03:51:35.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122567+0000", 2024-09-11T03:51:35.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6279999999999999e-06 2024-09-11T03:51:35.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122572+0000", 2024-09-11T03:51:35.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2619999999999998e-06 2024-09-11T03:51:35.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122621+0000", 2024-09-11T03:51:35.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9098999999999999e-05 2024-09-11T03:51:35.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-09-11T03:51:35.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122624+0000", 2024-09-11T03:51:35.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9809999999999999e-06 2024-09-11T03:51:35.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:30.046053+0000", 2024-09-11T03:51:35.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92342833400000002 2024-09-11T03:51:35.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:30.046061+0000", 2024-09-11T03:51:35.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1310000000000006e-06 2024-09-11T03:51:35.477 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.477 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.477 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.477 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8471 2.2 2:43d79ac9:::smithi17557870-37:head [copy-from ver 1253] snapc 17c=[17c,17b,17a,179,178,177,176,175,174,173,172,171,170,16f,16e,16b,169,168,167,164,15f,15e,15d,15b,15a,157,156,155,14d,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e828)", 2024-09-11T03:51:35.477 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:50:46.051277+0000", 2024-09-11T03:51:35.477 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 49.282284861999997, 2024-09-11T03:51:35.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017235528, 2024-09-11T03:51:35.477 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.477 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.477 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:35.477 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.477 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.477 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.478 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8471 2024-09-11T03:51:35.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.478 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.051277+0000", 2024-09-11T03:51:35.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.051277+0000", 2024-09-11T03:51:35.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.051279+0000", 2024-09-11T03:51:35.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2989999999999999e-06 2024-09-11T03:51:35.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.051288+0000", 2024-09-11T03:51:35.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8880000000000003e-06 2024-09-11T03:51:35.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.051308+0000", 2024-09-11T03:51:35.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9921000000000001e-05 2024-09-11T03:51:35.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.051318+0000", 2024-09-11T03:51:35.480 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:35.481 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:35.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:35 smithi175 ceph-mon[30894]: osdmap e876: 12 total, 12 up, 11 in 2024-09-11T03:51:35.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5559999999999996e-06 2024-09-11T03:51:35.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.051394+0000", 2024-09-11T03:51:35.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6532000000000005e-05 2024-09-11T03:51:35.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:51:35.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.051401+0000", 2024-09-11T03:51:35.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6050000000000003e-06 2024-09-11T03:51:35.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.053070+0000", 2024-09-11T03:51:35.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00166895 2024-09-11T03:51:35.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:35.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.053147+0000", 2024-09-11T03:51:35.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7088999999999993e-05 2024-09-11T03:51:35.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,6", 2024-09-11T03:51:35.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.060285+0000", 2024-09-11T03:51:35.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071378149999999996 2024-09-11T03:51:35.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:35.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.067528+0000", 2024-09-11T03:51:35.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072434750000000001 2024-09-11T03:51:35.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:51:35.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.068107+0000", 2024-09-11T03:51:35.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057833199999999998 2024-09-11T03:51:35.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:35.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.068437+0000", 2024-09-11T03:51:35.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032997899999999999 2024-09-11T03:51:35.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:51:35.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.068479+0000", 2024-09-11T03:51:35.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2685999999999998e-05 2024-09-11T03:51:35.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.068513+0000", 2024-09-11T03:51:35.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3401000000000001e-05 2024-09-11T03:51:35.486 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.486 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.486 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.486 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8786 2.d 2:b1c0f730:::smithi17557870-20:head [copy-from ver 1006] snapc 18b=[18b,18a,189,188,187,186,185,184,183,181,180,17f,17e,17d,17c,17a,178,177,176,175,174,172,171,170,16e,16b,169,167,164,15f,15e,15b,157,14d,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e853)", 2024-09-11T03:51:35.486 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:51:11.053928+0000", 2024-09-11T03:51:35.487 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.27963411, 2024-09-11T03:51:35.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.051393094, 2024-09-11T03:51:35.487 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.487 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.487 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:35.487 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.487 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.487 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.487 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8786 2024-09-11T03:51:35.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.487 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053928+0000", 2024-09-11T03:51:35.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053928+0000", 2024-09-11T03:51:35.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053930+0000", 2024-09-11T03:51:35.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.683e-06 2024-09-11T03:51:35.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053936+0000", 2024-09-11T03:51:35.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7179999999999999e-06 2024-09-11T03:51:35.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053953+0000", 2024-09-11T03:51:35.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6673000000000001e-05 2024-09-11T03:51:35.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053963+0000", 2024-09-11T03:51:35.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0438999999999999e-05 2024-09-11T03:51:35.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.054063+0000", 2024-09-11T03:51:35.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010009399999999999 2024-09-11T03:51:35.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:51:35.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.054070+0000", 2024-09-11T03:51:35.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7360000000000004e-06 2024-09-11T03:51:35.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.057611+0000", 2024-09-11T03:51:35.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003540648 2024-09-11T03:51:35.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:35.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.057724+0000", 2024-09-11T03:51:35.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113058 2024-09-11T03:51:35.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:51:35.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.085043+0000", 2024-09-11T03:51:35.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027319098 2024-09-11T03:51:35.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:35.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.089007+0000", 2024-09-11T03:51:35.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039635900000000003 2024-09-11T03:51:35.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:51:35.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.089056+0000", 2024-09-11T03:51:35.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9663999999999997e-05 2024-09-11T03:51:35.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:35.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.105240+0000", 2024-09-11T03:51:35.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016183633999999999 2024-09-11T03:51:35.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:51:35.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.105285+0000", 2024-09-11T03:51:35.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5368999999999999e-05 2024-09-11T03:51:35.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.105321+0000", 2024-09-11T03:51:35.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5689999999999999e-05 2024-09-11T03:51:35.494 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.494 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.494 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.494 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8787 2.d 2:b1c0f730:::smithi17557870-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e853)", 2024-09-11T03:51:35.494 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:51:11.053988+0000", 2024-09-11T03:51:35.494 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.279574119999999, 2024-09-11T03:51:35.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034580952999999998, 2024-09-11T03:51:35.495 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:35.495 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:35.495 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:35.495 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:35.495 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:35.495 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:35.495 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8787 2024-09-11T03:51:35.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.495 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:35.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:35.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053988+0000", 2024-09-11T03:51:35.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:35.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053988+0000", 2024-09-11T03:51:35.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:35.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:35.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053988+0000", 2024-09-11T03:51:35.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4799999999999998e-07 2024-09-11T03:51:35.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:35.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053991+0000", 2024-09-11T03:51:35.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9459999999999998e-06 2024-09-11T03:51:35.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:35.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.054004+0000", 2024-09-11T03:51:35.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2363e-05 2024-09-11T03:51:35.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:35.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.054010+0000", 2024-09-11T03:51:35.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5470000000000002e-06 2024-09-11T03:51:35.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.054122+0000", 2024-09-11T03:51:35.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111809 2024-09-11T03:51:35.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:51:35.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.054128+0000", 2024-09-11T03:51:35.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8159999999999999e-06 2024-09-11T03:51:35.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.057879+0000", 2024-09-11T03:51:35.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037510880000000001 2024-09-11T03:51:35.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:51:35.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.057924+0000", 2024-09-11T03:51:35.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4886999999999999e-05 2024-09-11T03:51:35.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:35.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.088395+0000", 2024-09-11T03:51:35.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.030471095 2024-09-11T03:51:35.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:35.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.088469+0000", 2024-09-11T03:51:35.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3815999999999999e-05 2024-09-11T03:51:35.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:35.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:35.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:35.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.088569+0000", 2024-09-11T03:51:35.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010003799999999999 2024-09-11T03:51:35.500 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.501 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.501 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.501 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:35.501 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:35.501 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:51:35.507 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:35.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:35.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:35.609 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:35.631 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:35.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:35.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:35.733 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:35.746 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:51:35.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:35.755 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:35.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:35.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:35.879 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:35.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:35.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:35.982 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:36.008 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:36.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:36.066 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:36.074 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T03:51:36.080 INFO:tasks.rados.rados.2.smithi175.stdout:3645: done (5 left) 2024-09-11T03:51:36.080 INFO:tasks.rados.rados.2.smithi175.stdout:3646: snap_create 2024-09-11T03:51:36.080 INFO:tasks.rados.rados.2.smithi175.stdout:3638: finishing copy_from racing read to smithi17557870-34 2024-09-11T03:51:36.080 INFO:tasks.rados.rados.2.smithi175.stdout:3638: finishing copy_from to smithi17557870-34 2024-09-11T03:51:36.080 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 1014 (ObjNum 1232 snap 401 seq_num 1232) dirty exists 2024-09-11T03:51:36.080 INFO:tasks.rados.rados.2.smithi175.stdout:3639: finishing write tid 2 to smithi17557870-44 2024-09-11T03:51:36.081 INFO:tasks.rados.rados.2.smithi175.stdout:3639: finishing write tid 3 to smithi17557870-44 2024-09-11T03:51:36.081 INFO:tasks.rados.rados.2.smithi175.stdout:3639: finishing write tid 4 to smithi17557870-44 2024-09-11T03:51:36.081 INFO:tasks.rados.rados.2.smithi175.stdout:3639: oid 44 updating version 0 to 1028 2024-09-11T03:51:36.081 INFO:tasks.rados.rados.2.smithi175.stdout:3639: oid 44 version 1028 is already newer than 1027 2024-09-11T03:51:36.081 INFO:tasks.rados.rados.2.smithi175.stdout:3639: oid 44 updating version 1028 to 1029 2024-09-11T03:51:36.081 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 1029 (ObjNum 1238 snap 405 seq_num 1238) dirty exists 2024-09-11T03:51:36.081 INFO:tasks.rados.rados.2.smithi175.stdout:3639: left oid 44 (ObjNum 1238 snap 405 seq_num 1238) 2024-09-11T03:51:36.081 INFO:tasks.rados.rados.2.smithi175.stdout:3641: finishing write tid 1 to smithi17557870-41 2024-09-11T03:51:36.081 INFO:tasks.rados.rados.2.smithi175.stdout:3641: finishing write tid 2 to smithi17557870-41 2024-09-11T03:51:36.081 INFO:tasks.rados.rados.2.smithi175.stdout:3641: finishing write tid 3 to smithi17557870-41 2024-09-11T03:51:36.081 INFO:tasks.rados.rados.2.smithi175.stdout:3641: finishing write tid 4 to smithi17557870-41 2024-09-11T03:51:36.081 INFO:tasks.rados.rados.2.smithi175.stdout:3641: finishing write tid 5 to smithi17557870-41 2024-09-11T03:51:36.081 INFO:tasks.rados.rados.2.smithi175.stdout:3641: oid 41 updating version 0 to 1024 2024-09-11T03:51:36.082 INFO:tasks.rados.rados.2.smithi175.stdout:3641: oid 41 version 1024 is already newer than 1023 2024-09-11T03:51:36.082 INFO:tasks.rados.rados.2.smithi175.stdout:3641: oid 41 version 1024 is already newer than 1021 2024-09-11T03:51:36.082 INFO:tasks.rados.rados.2.smithi175.stdout:3641: oid 41 version 1024 is already newer than 1022 2024-09-11T03:51:36.082 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 1024 (ObjNum 1239 snap 405 seq_num 1239) dirty exists 2024-09-11T03:51:36.082 INFO:tasks.rados.rados.2.smithi175.stdout:3641: left oid 41 (ObjNum 1239 snap 405 seq_num 1239) 2024-09-11T03:51:36.082 INFO:tasks.rados.rados.2.smithi175.stdout:3643: finishing copy_from racing read to smithi17557870-7 2024-09-11T03:51:36.082 INFO:tasks.rados.rados.2.smithi175.stdout:3643: finishing copy_from to smithi17557870-7 2024-09-11T03:51:36.082 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 1011 (ObjNum 1220 snap 399 seq_num 1220) dirty exists 2024-09-11T03:51:36.082 INFO:tasks.rados.rados.2.smithi175.stdout:3644: finishing copy_from racing read to smithi17557870-47 2024-09-11T03:51:36.082 INFO:tasks.rados.rados.2.smithi175.stdout:3644: finishing copy_from to smithi17557870-47 2024-09-11T03:51:36.082 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 1258 (ObjNum 1188 snap 388 seq_num 1188) dirty exists 2024-09-11T03:51:36.110 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:36.134 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:36.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:36.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:36 smithi084 ceph-mon[27254]: pgmap v1470: 107 pgs: 1 peering, 106 active+clean; 724 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 7 op/s; 6.0 MiB/s, 13 objects/s recovering 2024-09-11T03:51:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:36 smithi084 ceph-mon[27254]: osdmap e877: 12 total, 12 up, 11 in 2024-09-11T03:51:36.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2622585161' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:51:36.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T03:51:36.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:36.280 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:36.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:36.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:36.382 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:36.406 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:36.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:36.407 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:36.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:36 smithi121 ceph-mon[30729]: pgmap v1470: 107 pgs: 1 peering, 106 active+clean; 724 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 7 op/s; 6.0 MiB/s, 13 objects/s recovering 2024-09-11T03:51:36.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:36 smithi121 ceph-mon[30729]: osdmap e877: 12 total, 12 up, 11 in 2024-09-11T03:51:36.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2622585161' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:51:36.458 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:51:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:36 smithi175 ceph-mon[30894]: pgmap v1470: 107 pgs: 1 peering, 106 active+clean; 724 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 1.2 KiB/s rd, 1.4 MiB/s wr, 7 op/s; 6.0 MiB/s, 13 objects/s recovering 2024-09-11T03:51:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:36 smithi175 ceph-mon[30894]: osdmap e877: 12 total, 12 up, 11 in 2024-09-11T03:51:36.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2622585161' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:51:36.509 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:36.532 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:36.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:36.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:36.635 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:36.656 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:36.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:36.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:36.758 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:36.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:36.780 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:36.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:36.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:36.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:36.904 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:36.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:36.908 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:51:36.908 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:51:36.908 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:51:36.908 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:51:37.004 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:37.026 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:37.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:37.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:37.050 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:51:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:3638: done (5 left) 2024-09-11T03:51:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:3639: done (4 left) 2024-09-11T03:51:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:3641: done (3 left) 2024-09-11T03:51:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:3643: done (2 left) 2024-09-11T03:51:37.054 INFO:tasks.rados.rados.2.smithi175.stdout:3644: done (1 left) 2024-09-11T03:51:37.055 INFO:tasks.rados.rados.2.smithi175.stdout:3646: done (0 left) 2024-09-11T03:51:37.055 INFO:tasks.rados.rados.2.smithi175.stdout:3647: snap_create 2024-09-11T03:51:37.128 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:37.151 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:37.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:37.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:37.153 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-11T03:51:37.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:37 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:51:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:37 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:51:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:37 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2622585161' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:51:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:37 smithi121 ceph-mon[30729]: osdmap e878: 12 total, 12 up, 11 in 2024-09-11T03:51:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:37 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3192328105' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:51:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:37 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:37 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3192328105' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:51:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:37 smithi121 ceph-mon[30729]: osdmap e879: 12 total, 12 up, 11 in 2024-09-11T03:51:37.253 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:37.280 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:37.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:37.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:37.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:37 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:51:37.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:37 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:51:37.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2622585161' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:51:37.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:37 smithi084 ceph-mon[27254]: osdmap e878: 12 total, 12 up, 11 in 2024-09-11T03:51:37.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3192328105' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:51:37.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:37 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:37.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3192328105' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:51:37.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:37 smithi084 ceph-mon[27254]: osdmap e879: 12 total, 12 up, 11 in 2024-09-11T03:51:37.403 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:37.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:37.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:37 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:51:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:37 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:51:37.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:37 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2622585161' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:51:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:37 smithi175 ceph-mon[30894]: osdmap e878: 12 total, 12 up, 11 in 2024-09-11T03:51:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:37 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3192328105' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:51:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:37 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:37 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3192328105' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:51:37.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:37 smithi175 ceph-mon[30894]: osdmap e879: 12 total, 12 up, 11 in 2024-09-11T03:51:37.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:37.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:37.527 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:37.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:37.628 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:37.649 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:37.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:37.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:37.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T03:51:37.750 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:37.772 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:37.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:37.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:37.895 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:37.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:37.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:37.921 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:51:37.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:38.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:38.020 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:38.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:38.058 INFO:tasks.rados.rados.2.smithi175.stdout:3647: done (0 left) 2024-09-11T03:51:38.058 INFO:tasks.rados.rados.2.smithi175.stdout:3648: rollback oid 41 current snap is 407 2024-09-11T03:51:38.058 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 41 to 403 2024-09-11T03:51:38.058 INFO:tasks.rados.rados.2.smithi175.stdout:3649: write (excl) oid 50 current snap is 407 2024-09-11T03:51:38.058 INFO:tasks.rados.rados.2.smithi175.stdout:3649: seq_num 1240 ranges {440680=674299,1711907=624424} 2024-09-11T03:51:38.064 INFO:tasks.rados.rados.2.smithi175.stdout:3649: writing smithi17557870-50 from 440680 to 1114979 tid 1 2024-09-11T03:51:38.071 INFO:tasks.rados.rados.2.smithi175.stdout:3649: writing smithi17557870-50 from 1711907 to 2336331 tid 2 2024-09-11T03:51:38.072 INFO:tasks.rados.rados.2.smithi175.stdout:3650: snap_create 2024-09-11T03:51:38.072 INFO:tasks.rados.rados.2.smithi175.stdout:3648: finishing rollback tid 0 to smithi17557870-41 2024-09-11T03:51:38.072 INFO:tasks.rados.rados.2.smithi175.stdout:3648: finishing rollback tid 1 to smithi17557870-41 2024-09-11T03:51:38.072 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 1026 (ObjNum 49 snap 0 seq_num 197408936) dirty dne 2024-09-11T03:51:38.072 INFO:tasks.rados.rados.2.smithi175.stdout:3649: finishing write tid 1 to smithi17557870-50 2024-09-11T03:51:38.074 INFO:tasks.rados.rados.2.smithi175.stdout:3649: finishing write tid 2 to smithi17557870-50 2024-09-11T03:51:38.075 INFO:tasks.rados.rados.2.smithi175.stdout:3649: finishing write tid 3 to smithi17557870-50 2024-09-11T03:51:38.076 INFO:tasks.rados.rados.2.smithi175.stdout:3649: finishing write tid 4 to smithi17557870-50 2024-09-11T03:51:38.076 INFO:tasks.rados.rados.2.smithi175.stdout:3649: oid 50 updating version 0 to 1452 2024-09-11T03:51:38.076 INFO:tasks.rados.rados.2.smithi175.stdout:3649: oid 50 updating version 1452 to 1453 2024-09-11T03:51:38.076 INFO:tasks.rados.rados.2.smithi175.stdout:3649: oid 50 updating version 1453 to 1454 2024-09-11T03:51:38.076 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 1454 (ObjNum 1240 snap 407 seq_num 1240) dirty exists 2024-09-11T03:51:38.076 INFO:tasks.rados.rados.2.smithi175.stdout:3649: left oid 50 (ObjNum 1240 snap 407 seq_num 1240) 2024-09-11T03:51:38.120 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:38.126 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:51:38.126 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:51:38.126 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:38 smithi084 ceph-mon[27254]: pgmap v1473: 107 pgs: 107 active+clean; 739 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 511 B/s rd, 2.7 MiB/s wr, 4 op/s 2024-09-11T03:51:38.127 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:38 smithi084 ceph-mon[27254]: osdmap e880: 12 total, 12 up, 11 in 2024-09-11T03:51:38.143 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:38.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:38.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:38.145 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:51:38.145 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:51:38.149 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:38.162 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:51:38.162 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:51:38.188 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:51:38.188 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:51:38.204 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:51:38.205 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:51:38.220 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:51:38.220 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:51:38.237 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:51:38.238 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:51:38.245 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:38.253 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:51:38.253 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:51:38.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:38.267 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:38.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:38.272 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:51:38.272 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:51:38.289 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:51:38.290 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:51:38.306 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:51:38.306 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:51:38.317 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:38 smithi121 ceph-mon[30729]: pgmap v1473: 107 pgs: 107 active+clean; 739 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 511 B/s rd, 2.7 MiB/s wr, 4 op/s 2024-09-11T03:51:38.317 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:38 smithi121 ceph-mon[30729]: osdmap e880: 12 total, 12 up, 11 in 2024-09-11T03:51:38.323 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:51:38.323 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:51:38.368 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:38.390 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:38.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:38.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:38.416 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:51:38.416 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:51:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:38 smithi175 ceph-mon[30894]: pgmap v1473: 107 pgs: 107 active+clean; 739 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 511 B/s rd, 2.7 MiB/s wr, 4 op/s 2024-09-11T03:51:38.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:38 smithi175 ceph-mon[30894]: osdmap e880: 12 total, 12 up, 11 in 2024-09-11T03:51:38.492 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:38.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:38.514 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:38.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:38.615 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:38.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:38.638 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:38.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:38.739 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:38.765 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:38.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:38.766 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:38.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:38.888 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:38.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:38.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:38.990 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:39.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:39.012 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:39.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:39.059 INFO:tasks.rados.rados.2.smithi175.stdout:3648: done (2 left) 2024-09-11T03:51:39.059 INFO:tasks.rados.rados.2.smithi175.stdout:3649: done (1 left) 2024-09-11T03:51:39.059 INFO:tasks.rados.rados.2.smithi175.stdout:3650: done (0 left) 2024-09-11T03:51:39.059 INFO:tasks.rados.rados.2.smithi175.stdout:3651: delete oid 29 current snap is 408 2024-09-11T03:51:39.063 INFO:tasks.rados.rados.2.smithi175.stdout:3651: done (0 left) 2024-09-11T03:51:39.063 INFO:tasks.rados.rados.2.smithi175.stdout:3652: read oid 10 snap -1 2024-09-11T03:51:39.063 INFO:tasks.rados.rados.2.smithi175.stdout:3652: expect (ObjNum 1075 snap 350 seq_num 1075) 2024-09-11T03:51:39.064 INFO:tasks.rados.rados.2.smithi175.stdout:3653: copy_from oid 48 from oid 22 current snap is 408 2024-09-11T03:51:39.064 INFO:tasks.rados.rados.2.smithi175.stdout:3654: write (excl) oid 11 current snap is 408 2024-09-11T03:51:39.064 INFO:tasks.rados.rados.2.smithi175.stdout:3654: seq_num 1242 ranges {474294=424797,1428871=628093,2657357=216937} 2024-09-11T03:51:39.068 INFO:tasks.rados.rados.2.smithi175.stdout:3654: writing smithi17557870-11 from 474294 to 899091 tid 1 2024-09-11T03:51:39.075 INFO:tasks.rados.rados.2.smithi175.stdout:3654: writing smithi17557870-11 from 1428871 to 2056964 tid 2 2024-09-11T03:51:39.080 INFO:tasks.rados.rados.2.smithi175.stdout:3654: writing smithi17557870-11 from 2657357 to 2874294 tid 3 2024-09-11T03:51:39.080 INFO:tasks.rados.rados.2.smithi175.stdout:3655: write oid 46 current snap is 408 2024-09-11T03:51:39.080 INFO:tasks.rados.rados.2.smithi175.stdout:3655: seq_num 1243 ranges {491101=500046,1487353=488104,2401133=489968} 2024-09-11T03:51:39.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T03:51:39.085 INFO:tasks.rados.rados.2.smithi175.stdout:3655: writing smithi17557870-46 from 491101 to 991147 tid 1 2024-09-11T03:51:39.091 INFO:tasks.rados.rados.2.smithi175.stdout:3655: writing smithi17557870-46 from 1487353 to 1975457 tid 2 2024-09-11T03:51:39.096 INFO:tasks.rados.rados.2.smithi175.stdout:3655: writing smithi17557870-46 from 2401133 to 2891101 tid 3 2024-09-11T03:51:39.096 INFO:tasks.rados.rados.2.smithi175.stdout:3656: read oid 43 snap -1 2024-09-11T03:51:39.097 INFO:tasks.rados.rados.2.smithi175.stdout:3656: expect deleted 2024-09-11T03:51:39.097 INFO:tasks.rados.rados.2.smithi175.stdout:3657: rollback oid 30 current snap is 408 2024-09-11T03:51:39.097 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 30 to 399 2024-09-11T03:51:39.097 INFO:tasks.rados.rados.2.smithi175.stdout:3653: finishing copy_from racing read to smithi17557870-48 2024-09-11T03:51:39.097 INFO:tasks.rados.rados.2.smithi175.stdout:3654: finishing write tid 1 to smithi17557870-11 2024-09-11T03:51:39.097 INFO:tasks.rados.rados.2.smithi175.stdout:3652: expect (ObjNum 1075 snap 350 seq_num 1075) 2024-09-11T03:51:39.112 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:39.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:39.134 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:39.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:39.192 INFO:tasks.rados.rados.2.smithi175.stdout:3653: finishing copy_from to smithi17557870-48 2024-09-11T03:51:39.192 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 1285 (ObjNum 1212 snap 397 seq_num 1212) dirty exists 2024-09-11T03:51:39.192 INFO:tasks.rados.rados.2.smithi175.stdout:3654: finishing write tid 2 to smithi17557870-11 2024-09-11T03:51:39.192 INFO:tasks.rados.rados.2.smithi175.stdout:3654: finishing write tid 3 to smithi17557870-11 2024-09-11T03:51:39.192 INFO:tasks.rados.rados.2.smithi175.stdout:3654: finishing write tid 4 to smithi17557870-11 2024-09-11T03:51:39.192 INFO:tasks.rados.rados.2.smithi175.stdout:3652: done (5 left) 2024-09-11T03:51:39.193 INFO:tasks.rados.rados.2.smithi175.stdout:3653: done (4 left) 2024-09-11T03:51:39.193 INFO:tasks.rados.rados.2.smithi175.stdout:3658: copy_from oid 36 from oid 12 current snap is 408 2024-09-11T03:51:39.193 INFO:tasks.rados.rados.2.smithi175.stdout:3654: finishing write tid 5 to smithi17557870-11 2024-09-11T03:51:39.193 INFO:tasks.rados.rados.2.smithi175.stdout:3654: oid 11 updating version 0 to 1238 2024-09-11T03:51:39.193 INFO:tasks.rados.rados.2.smithi175.stdout:3654: oid 11 version 1238 is already newer than 1237 2024-09-11T03:51:39.193 INFO:tasks.rados.rados.2.smithi175.stdout:3654: oid 11 version 1238 is already newer than 1236 2024-09-11T03:51:39.193 INFO:tasks.rados.rados.2.smithi175.stdout:3654: oid 11 version 1238 is already newer than 1235 2024-09-11T03:51:39.193 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 1238 (ObjNum 1242 snap 408 seq_num 1242) dirty exists 2024-09-11T03:51:39.193 INFO:tasks.rados.rados.2.smithi175.stdout:3654: left oid 11 (ObjNum 1242 snap 408 seq_num 1242) 2024-09-11T03:51:39.193 INFO:tasks.rados.rados.2.smithi175.stdout:3655: finishing write tid 1 to smithi17557870-46 2024-09-11T03:51:39.193 INFO:tasks.rados.rados.2.smithi175.stdout:3655: finishing write tid 2 to smithi17557870-46 2024-09-11T03:51:39.193 INFO:tasks.rados.rados.2.smithi175.stdout:3657: finishing rollback tid 1 to smithi17557870-30 2024-09-11T03:51:39.193 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 1113 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-11T03:51:39.193 INFO:tasks.rados.rados.2.smithi175.stdout:3655: finishing write tid 3 to smithi17557870-46 2024-09-11T03:51:39.194 INFO:tasks.rados.rados.2.smithi175.stdout:3655: finishing write tid 4 to smithi17557870-46 2024-09-11T03:51:39.194 INFO:tasks.rados.rados.2.smithi175.stdout:3655: finishing write tid 5 to smithi17557870-46 2024-09-11T03:51:39.194 INFO:tasks.rados.rados.2.smithi175.stdout:3655: oid 46 updating version 0 to 979 2024-09-11T03:51:39.194 INFO:tasks.rados.rados.2.smithi175.stdout:3655: oid 46 version 979 is already newer than 978 2024-09-11T03:51:39.194 INFO:tasks.rados.rados.2.smithi175.stdout:3655: oid 46 version 979 is already newer than 977 2024-09-11T03:51:39.194 INFO:tasks.rados.rados.2.smithi175.stdout:3655: oid 46 version 979 is already newer than 976 2024-09-11T03:51:39.194 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 979 (ObjNum 1243 snap 408 seq_num 1243) dirty exists 2024-09-11T03:51:39.194 INFO:tasks.rados.rados.2.smithi175.stdout:3655: left oid 46 (ObjNum 1243 snap 408 seq_num 1243) 2024-09-11T03:51:39.194 INFO:tasks.rados.rados.2.smithi175.stdout:3654: done (4 left) 2024-09-11T03:51:39.194 INFO:tasks.rados.rados.2.smithi175.stdout:3655: done (3 left) 2024-09-11T03:51:39.194 INFO:tasks.rados.rados.2.smithi175.stdout:3656: done (2 left) 2024-09-11T03:51:39.194 INFO:tasks.rados.rados.2.smithi175.stdout:3657: done (1 left) 2024-09-11T03:51:39.194 INFO:tasks.rados.rados.2.smithi175.stdout:3659: read oid 42 snap 404 2024-09-11T03:51:39.194 INFO:tasks.rados.rados.2.smithi175.stdout:3659: expect deleted 2024-09-11T03:51:39.195 INFO:tasks.rados.rados.2.smithi175.stdout:3660: rollback oid 25 current snap is 408 2024-09-11T03:51:39.195 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 25 to 385 2024-09-11T03:51:39.195 INFO:tasks.rados.rados.2.smithi175.stdout:3661: copy_from oid 11 from oid 38 current snap is 408 2024-09-11T03:51:39.195 INFO:tasks.rados.rados.2.smithi175.stdout:3662: snap_remove snap 348 2024-09-11T03:51:39.234 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:39.255 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:39.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:39.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:39.325 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:51:39.328 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:39 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2272322464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:39.328 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:39 smithi084 ceph-mon[27254]: osdmap e881: 12 total, 12 up, 11 in 2024-09-11T03:51:39.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:39.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:39.379 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:39.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:39.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:39 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2272322464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:39.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:39 smithi121 ceph-mon[30729]: osdmap e881: 12 total, 12 up, 11 in 2024-09-11T03:51:39.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:39 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2272322464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:39.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:39 smithi175 ceph-mon[30894]: osdmap e881: 12 total, 12 up, 11 in 2024-09-11T03:51:39.481 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:39.503 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:39.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:39.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:39.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:39.627 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:39.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:39.729 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:39.754 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:39.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:39.755 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:39.773 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:51:39.773 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:51:39.855 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:39.879 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:39.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:39.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:39.896 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:51:39.981 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:40.005 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:40.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:40.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:40.086 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T03:51:40.095 INFO:tasks.rados.rados.2.smithi175.stdout:3662: done (4 left) 2024-09-11T03:51:40.095 INFO:tasks.rados.rados.2.smithi175.stdout:3663: write oid 28 current snap is 408 2024-09-11T03:51:40.095 INFO:tasks.rados.rados.2.smithi175.stdout:3663: seq_num 1244 ranges {507908=575295,1829482=631762} 2024-09-11T03:51:40.101 INFO:tasks.rados.rados.2.smithi175.stdout:3663: writing smithi17557870-28 from 507908 to 1083203 tid 1 2024-09-11T03:51:40.107 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:40.110 INFO:tasks.rados.rados.2.smithi175.stdout:3663: writing smithi17557870-28 from 1829482 to 2461244 tid 2 2024-09-11T03:51:40.110 INFO:tasks.rados.rados.2.smithi175.stdout:3664: snap_create 2024-09-11T03:51:40.111 INFO:tasks.rados.rados.2.smithi175.stdout:3660: finishing rollback tid 0 to smithi17557870-25 2024-09-11T03:51:40.111 INFO:tasks.rados.rados.2.smithi175.stdout:3660: finishing rollback tid 1 to smithi17557870-25 2024-09-11T03:51:40.111 INFO:tasks.rados.rados.2.smithi175.stdout:3660: finishing rollback tid 2 to smithi17557870-25 2024-09-11T03:51:40.111 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 1011 (ObjNum 1148 snap 371 seq_num 1148) dirty exists 2024-09-11T03:51:40.111 INFO:tasks.rados.rados.2.smithi175.stdout:3658: finishing copy_from racing read to smithi17557870-36 2024-09-11T03:51:40.111 INFO:tasks.rados.rados.2.smithi175.stdout:3661: finishing copy_from racing read to smithi17557870-11 2024-09-11T03:51:40.111 INFO:tasks.rados.rados.2.smithi175.stdout:3658: finishing copy_from to smithi17557870-36 2024-09-11T03:51:40.111 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 1367 (ObjNum 1134 snap 367 seq_num 1134) dirty exists 2024-09-11T03:51:40.111 INFO:tasks.rados.rados.2.smithi175.stdout:3661: finishing copy_from to smithi17557870-11 2024-09-11T03:51:40.111 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 1411 (ObjNum 1217 snap 399 seq_num 1217) dirty exists 2024-09-11T03:51:40.111 INFO:tasks.rados.rados.2.smithi175.stdout:3663: finishing write tid 1 to smithi17557870-28 2024-09-11T03:51:40.114 INFO:tasks.rados.rados.2.smithi175.stdout:3663: finishing write tid 2 to smithi17557870-28 2024-09-11T03:51:40.114 INFO:tasks.rados.rados.2.smithi175.stdout:3663: finishing write tid 3 to smithi17557870-28 2024-09-11T03:51:40.114 INFO:tasks.rados.rados.2.smithi175.stdout:3663: finishing write tid 4 to smithi17557870-28 2024-09-11T03:51:40.114 INFO:tasks.rados.rados.2.smithi175.stdout:3663: oid 28 updating version 0 to 1464 2024-09-11T03:51:40.114 INFO:tasks.rados.rados.2.smithi175.stdout:3663: oid 28 version 1464 is already newer than 1460 2024-09-11T03:51:40.114 INFO:tasks.rados.rados.2.smithi175.stdout:3663: oid 28 version 1464 is already newer than 1463 2024-09-11T03:51:40.114 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 1464 (ObjNum 1244 snap 408 seq_num 1244) dirty exists 2024-09-11T03:51:40.114 INFO:tasks.rados.rados.2.smithi175.stdout:3663: left oid 28 (ObjNum 1244 snap 408 seq_num 1244) 2024-09-11T03:51:40.129 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:40.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:40.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:40.231 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:40.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:40.253 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:40.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:40.349 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:40 smithi121 ceph-mon[30729]: pgmap v1476: 107 pgs: 107 active+clean; 742 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 511 B/s rd, 3.6 MiB/s wr, 5 op/s 2024-09-11T03:51:40.349 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:40 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/598070134' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:51:40.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:40.375 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:40.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:40.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:40.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T03:51:40.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:40 smithi084 ceph-mon[27254]: pgmap v1476: 107 pgs: 107 active+clean; 742 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 511 B/s rd, 3.6 MiB/s wr, 5 op/s 2024-09-11T03:51:40.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:40 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/598070134' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:51:40.476 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:40.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:40 smithi175 ceph-mon[30894]: pgmap v1476: 107 pgs: 107 active+clean; 742 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 511 B/s rd, 3.6 MiB/s wr, 5 op/s 2024-09-11T03:51:40.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:40 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/598070134' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:51:40.503 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:40.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:40.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:40.605 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:40.616 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:51:40.630 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:40.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:40.631 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:40.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:40.753 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:40.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:40.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:40.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-11T03:51:40.855 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:40.863 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:40.884 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:40.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:40.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:40.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:41.007 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:41.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:41.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:41.107 INFO:tasks.rados.rados.2.smithi175.stdout:3658: done (5 left) 2024-09-11T03:51:41.108 INFO:tasks.rados.rados.2.smithi175.stdout:3659: done (4 left) 2024-09-11T03:51:41.108 INFO:tasks.rados.rados.2.smithi175.stdout:3660: done (3 left) 2024-09-11T03:51:41.108 INFO:tasks.rados.rados.2.smithi175.stdout:3661: done (2 left) 2024-09-11T03:51:41.108 INFO:tasks.rados.rados.2.smithi175.stdout:3663: done (1 left) 2024-09-11T03:51:41.108 INFO:tasks.rados.rados.2.smithi175.stdout:3664: done (0 left) 2024-09-11T03:51:41.108 INFO:tasks.rados.rados.2.smithi175.stdout:3665: snap_create 2024-09-11T03:51:41.109 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:41.114 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:51:41.129 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:51:41.129 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:51:41.134 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:41.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:41.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:41.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:41.258 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:41.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:41.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:41.360 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:41.377 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:41 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/598070134' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:51:41.377 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:41 smithi084 ceph-mon[27254]: osdmap e882: 12 total, 12 up, 11 in 2024-09-11T03:51:41.385 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:41.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:41.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:41 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/598070134' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:51:41.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:41 smithi121 ceph-mon[30729]: osdmap e882: 12 total, 12 up, 11 in 2024-09-11T03:51:41.456 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:41.464 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:41.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:41 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/598070134' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:51:41.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:41 smithi175 ceph-mon[30894]: osdmap e882: 12 total, 12 up, 11 in 2024-09-11T03:51:41.488 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:41.512 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:41.513 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:41.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:41.638 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:41.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:41.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:41.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:41.761 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:41.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:41.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:41.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T03:51:41.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:41.889 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:41.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:41.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:42.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:42.013 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:42.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:42.018 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:51:42.052 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:51:42.058 INFO:tasks.rados.rados.2.smithi175.stdout:3665: done (0 left) 2024-09-11T03:51:42.058 INFO:tasks.rados.rados.2.smithi175.stdout:3666: delete oid 49 current snap is 410 2024-09-11T03:51:42.063 INFO:tasks.rados.rados.2.smithi175.stdout:3666: done (0 left) 2024-09-11T03:51:42.063 INFO:tasks.rados.rados.2.smithi175.stdout:3667: read oid 12 snap -1 2024-09-11T03:51:42.063 INFO:tasks.rados.rados.2.smithi175.stdout:3667: expect (ObjNum 1134 snap 367 seq_num 1134) 2024-09-11T03:51:42.064 INFO:tasks.rados.rados.2.smithi175.stdout:3668: snap_remove snap 382 2024-09-11T03:51:42.115 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:42.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:42.138 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:42.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:42.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:42.262 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:42.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:42.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:42.275 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:51:42.275 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:51:42.291 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:51:42.291 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:51:42.307 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:51:42.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:51:42.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:42 smithi084 ceph-mon[27254]: pgmap v1479: 107 pgs: 107 active+clean; 742 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s rd, 968 KiB/s wr, 0 op/s 2024-09-11T03:51:42.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:42 smithi084 ceph-mon[27254]: osdmap e883: 12 total, 12 up, 11 in 2024-09-11T03:51:42.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:42 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2441085183' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:51:42.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:42 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:42.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:42 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2441085183' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:51:42.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:42 smithi084 ceph-mon[27254]: osdmap e884: 12 total, 12 up, 11 in 2024-09-11T03:51:42.324 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:51:42.324 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:51:42.340 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:51:42.340 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:51:42.358 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:51:42.359 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:51:42.363 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:42.374 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:51:42.374 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:51:42.389 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:42.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:42.390 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:42.390 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:51:42.391 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:51:42.407 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:51:42.407 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:51:42.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:42 smithi121 ceph-mon[30729]: pgmap v1479: 107 pgs: 107 active+clean; 742 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s rd, 968 KiB/s wr, 0 op/s 2024-09-11T03:51:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:42 smithi121 ceph-mon[30729]: osdmap e883: 12 total, 12 up, 11 in 2024-09-11T03:51:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:42 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2441085183' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:51:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:42 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:42 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2441085183' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:51:42.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:42 smithi121 ceph-mon[30729]: osdmap e884: 12 total, 12 up, 11 in 2024-09-11T03:51:42.425 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:51:42.425 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:51:42.442 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:51:42.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:51:42.459 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:51:42.459 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:51:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:42 smithi175 ceph-mon[30894]: pgmap v1479: 107 pgs: 107 active+clean; 742 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s rd, 968 KiB/s wr, 0 op/s 2024-09-11T03:51:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:42 smithi175 ceph-mon[30894]: osdmap e883: 12 total, 12 up, 11 in 2024-09-11T03:51:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:42 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2441085183' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:51:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:42 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:42 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2441085183' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:51:42.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:42 smithi175 ceph-mon[30894]: osdmap e884: 12 total, 12 up, 11 in 2024-09-11T03:51:42.491 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:42.514 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:42.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:42.515 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:42.520 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:51:42.521 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:51:42.521 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:51:42.521 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:51:42.521 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:51:42.616 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:42.640 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:42.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:42.641 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:42.742 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:42.765 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:42.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:42.767 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:42.868 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:42.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:42.891 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:42.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:42.993 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:43.014 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:43.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:43.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:43.053 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T03:51:43.057 INFO:tasks.rados.rados.2.smithi175.stdout:3668: done (1 left) 2024-09-11T03:51:43.057 INFO:tasks.rados.rados.2.smithi175.stdout:3669: snap_remove snap 311 2024-09-11T03:51:43.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T03:51:43.116 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:43.140 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:43.141 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:43.241 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:43.263 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:43.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:43.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:43.335 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:51:43.365 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:43.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:43 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/225294569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:43.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:43 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3708748476' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:51:43.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:43 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3708748476' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:51:43.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:43 smithi084 ceph-mon[27254]: osdmap e885: 12 total, 12 up, 11 in 2024-09-11T03:51:43.389 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:43.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:43.389 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:43.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:43 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/225294569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:43.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:43 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3708748476' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:51:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:43 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3708748476' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:51:43.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:43 smithi121 ceph-mon[30729]: osdmap e885: 12 total, 12 up, 11 in 2024-09-11T03:51:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:43 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/225294569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:43 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3708748476' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:51:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:43 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3708748476' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:51:43.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:43 smithi175 ceph-mon[30894]: osdmap e885: 12 total, 12 up, 11 in 2024-09-11T03:51:43.491 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:43.515 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:43.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:43.516 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:43.617 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:43.638 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:43.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:43.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:43.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:43.762 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:43.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:43.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:43.814 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:51:43.814 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:51:43.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:51:43.814 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:51:43.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.814 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4288 2.0 2:0762776a:::smithi17557870-25:head [copy-from ver 488] snapc ba=[ba,b9,b8,b7,b6,b5,b1,af,ae,ac] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-11T03:51:43.815 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:39.973423+0000", 2024-09-11T03:51:43.815 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 423.83123928999998, 2024-09-11T03:51:43.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028521689999999999, 2024-09-11T03:51:43.815 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.815 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.815 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:43.815 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.815 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.815 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.815 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4288 2024-09-11T03:51:43.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.815 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973423+0000", 2024-09-11T03:51:43.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973423+0000", 2024-09-11T03:51:43.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973424+0000", 2024-09-11T03:51:43.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3599999999999999e-06 2024-09-11T03:51:43.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973431+0000", 2024-09-11T03:51:43.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.55e-06 2024-09-11T03:51:43.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973441+0000", 2024-09-11T03:51:43.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0383e-05 2024-09-11T03:51:43.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973446+0000", 2024-09-11T03:51:43.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3689999999999998e-06 2024-09-11T03:51:43.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973498+0000", 2024-09-11T03:51:43.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1412000000000002e-05 2024-09-11T03:51:43.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:43.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973592+0000", 2024-09-11T03:51:43.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4557999999999998e-05 2024-09-11T03:51:43.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,10", 2024-09-11T03:51:43.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.987776+0000", 2024-09-11T03:51:43.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014183553 2024-09-11T03:51:43.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:51:43.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.991768+0000", 2024-09-11T03:51:43.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039917620000000003 2024-09-11T03:51:43.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:43.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.999033+0000", 2024-09-11T03:51:43.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072650620000000001 2024-09-11T03:51:43.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:43.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:40.001866+0000", 2024-09-11T03:51:43.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028328160000000002 2024-09-11T03:51:43.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:51:43.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:40.001905+0000", 2024-09-11T03:51:43.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9737999999999998e-05 2024-09-11T03:51:43.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:40.001944+0000", 2024-09-11T03:51:43.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9127e-05 2024-09-11T03:51:43.821 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.822 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.822 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.822 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4289 2.0 2:0762776a:::smithi17557870-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e462)", 2024-09-11T03:51:43.822 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:39.973461+0000", 2024-09-11T03:51:43.822 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 423.83120127199999, 2024-09-11T03:51:43.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017766608, 2024-09-11T03:51:43.822 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.822 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.822 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:43.822 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.823 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.823 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.823 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4289 2024-09-11T03:51:43.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.823 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973461+0000", 2024-09-11T03:51:43.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973461+0000", 2024-09-11T03:51:43.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973461+0000", 2024-09-11T03:51:43.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0399999999999996e-07 2024-09-11T03:51:43.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973467+0000", 2024-09-11T03:51:43.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8780000000000003e-06 2024-09-11T03:51:43.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973472+0000", 2024-09-11T03:51:43.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3009999999999997e-06 2024-09-11T03:51:43.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973477+0000", 2024-09-11T03:51:43.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7570000000000004e-06 2024-09-11T03:51:43.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973749+0000", 2024-09-11T03:51:43.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027214900000000001 2024-09-11T03:51:43.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:51:43.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.973782+0000", 2024-09-11T03:51:43.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2543000000000003e-05 2024-09-11T03:51:43.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.991066+0000", 2024-09-11T03:51:43.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017284568 2024-09-11T03:51:43.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:43.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.991145+0000", 2024-09-11T03:51:43.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8498000000000006e-05 2024-09-11T03:51:43.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:39.991227+0000", 2024-09-11T03:51:43.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2410000000000005e-05 2024-09-11T03:51:43.828 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.828 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.828 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.828 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:4688 2.18 e503/312 2:1984a3a9:::smithi17557870-23:head v 503'349, mlcod=498'344)", 2024-09-11T03:51:43.829 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:45:21.011490+0000", 2024-09-11T03:51:43.829 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 382.79317159499999, 2024-09-11T03:51:43.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019963814, 2024-09-11T03:51:43.829 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.829 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.829 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:43.829 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.011490+0000", 2024-09-11T03:51:43.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.011490+0000", 2024-09-11T03:51:43.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.011491+0000", 2024-09-11T03:51:43.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7000000000000003e-07 2024-09-11T03:51:43.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030626+0000", 2024-09-11T03:51:43.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019134880999999999 2024-09-11T03:51:43.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030636+0000", 2024-09-11T03:51:43.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0159e-05 2024-09-11T03:51:43.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030642+0000", 2024-09-11T03:51:43.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4070000000000001e-06 2024-09-11T03:51:43.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030686+0000", 2024-09-11T03:51:43.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4320999999999998e-05 2024-09-11T03:51:43.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:43.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.030710+0000", 2024-09-11T03:51:43.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3495000000000001e-05 2024-09-11T03:51:43.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:51:43.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.031422+0000", 2024-09-11T03:51:43.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000712588 2024-09-11T03:51:43.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:45:21.031454+0000", 2024-09-11T03:51:43.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2092999999999999e-05 2024-09-11T03:51:43.833 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.833 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.833 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.833 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7056 2.20 2:0762776a:::smithi17557870-25:head [copy-from ver 893] snapc 13a=[13a,139,138,137,136,135,133,12e,12d,12c,12a,127,126,123,120,11f,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e710)", 2024-09-11T03:51:43.833 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:48.020110+0000", 2024-09-11T03:51:43.833 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 175.78455155, 2024-09-11T03:51:43.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.024485558000000001, 2024-09-11T03:51:43.833 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.834 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.834 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:43.834 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.834 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.834 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.834 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7056 2024-09-11T03:51:43.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.834 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.020110+0000", 2024-09-11T03:51:43.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.020110+0000", 2024-09-11T03:51:43.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.020113+0000", 2024-09-11T03:51:43.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1009999999999999e-06 2024-09-11T03:51:43.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.020121+0000", 2024-09-11T03:51:43.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8200000000000003e-06 2024-09-11T03:51:43.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.020138+0000", 2024-09-11T03:51:43.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6954999999999999e-05 2024-09-11T03:51:43.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.020146+0000", 2024-09-11T03:51:43.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.61e-06 2024-09-11T03:51:43.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.020217+0000", 2024-09-11T03:51:43.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1292999999999997e-05 2024-09-11T03:51:43.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:43.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.020300+0000", 2024-09-11T03:51:43.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3191999999999999e-05 2024-09-11T03:51:43.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5,6", 2024-09-11T03:51:43.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.028565+0000", 2024-09-11T03:51:43.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0082645229999999993 2024-09-11T03:51:43.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:51:43.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.043864+0000", 2024-09-11T03:51:43.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015299228999999999 2024-09-11T03:51:43.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:43.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.044057+0000", 2024-09-11T03:51:43.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019271100000000001 2024-09-11T03:51:43.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:43.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.044465+0000", 2024-09-11T03:51:43.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040809599999999999 2024-09-11T03:51:43.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:51:43.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.044536+0000", 2024-09-11T03:51:43.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0933999999999995e-05 2024-09-11T03:51:43.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.044596+0000", 2024-09-11T03:51:43.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0093999999999998e-05 2024-09-11T03:51:43.839 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.839 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.840 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.840 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7057 2.20 2:0762776a:::smithi17557870-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e710)", 2024-09-11T03:51:43.840 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:48.020169+0000", 2024-09-11T03:51:43.840 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 175.784493091, 2024-09-11T03:51:43.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014652386, 2024-09-11T03:51:43.840 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.840 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.840 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:43.840 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.840 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.841 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.841 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7057 2024-09-11T03:51:43.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.841 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.020169+0000", 2024-09-11T03:51:43.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.020169+0000", 2024-09-11T03:51:43.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.020170+0000", 2024-09-11T03:51:43.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2300000000000001e-07 2024-09-11T03:51:43.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.020176+0000", 2024-09-11T03:51:43.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6900000000000003e-06 2024-09-11T03:51:43.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.020192+0000", 2024-09-11T03:51:43.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6288999999999999e-05 2024-09-11T03:51:43.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.020200+0000", 2024-09-11T03:51:43.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2189999999999998e-06 2024-09-11T03:51:43.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.020582+0000", 2024-09-11T03:51:43.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000382543 2024-09-11T03:51:43.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:51:43.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.020635+0000", 2024-09-11T03:51:43.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2787999999999997e-05 2024-09-11T03:51:43.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034695+0000", 2024-09-11T03:51:43.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014059465 2024-09-11T03:51:43.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:43.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034752+0000", 2024-09-11T03:51:43.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7361000000000002e-05 2024-09-11T03:51:43.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034821+0000", 2024-09-11T03:51:43.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9407999999999997e-05 2024-09-11T03:51:43.845 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.845 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.845 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.845 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7353 2.8 2:162357fa:::smithi17557870-18:head [copy-from ver 930] snapc 14c=[14b,14a,149,148,147,146,145,144,143,142,13d,13c,139,138,136,135,133,12e,12d,12a,127,120,11e,119] ondisk+write+known_if_redirected+supports_pool_eio e741)", 2024-09-11T03:51:43.845 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:19.016927+0000", 2024-09-11T03:51:43.845 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.787735357, 2024-09-11T03:51:43.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020562403, 2024-09-11T03:51:43.846 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.846 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.846 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:43.846 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.846 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.846 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.846 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7353 2024-09-11T03:51:43.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.846 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:19.016927+0000", 2024-09-11T03:51:43.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:19.016927+0000", 2024-09-11T03:51:43.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:19.016929+0000", 2024-09-11T03:51:43.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.886e-06 2024-09-11T03:51:43.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:19.016938+0000", 2024-09-11T03:51:43.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3060000000000003e-06 2024-09-11T03:51:43.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:19.016955+0000", 2024-09-11T03:51:43.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6759e-05 2024-09-11T03:51:43.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:19.016965+0000", 2024-09-11T03:51:43.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-09-11T03:51:43.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:19.017044+0000", 2024-09-11T03:51:43.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8724e-05 2024-09-11T03:51:43.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:51:43.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:19.017053+0000", 2024-09-11T03:51:43.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8189999999999998e-06 2024-09-11T03:51:43.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:19.020411+0000", 2024-09-11T03:51:43.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003358771 2024-09-11T03:51:43.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:43.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:19.020484+0000", 2024-09-11T03:51:43.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2528999999999996e-05 2024-09-11T03:51:43.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,9", 2024-09-11T03:51:43.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:19.027117+0000", 2024-09-11T03:51:43.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0066325569999999999 2024-09-11T03:51:43.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:51:43.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:19.031450+0000", 2024-09-11T03:51:43.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043334610000000003 2024-09-11T03:51:43.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:43.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:19.036564+0000", 2024-09-11T03:51:43.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051142569999999997 2024-09-11T03:51:43.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:43.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:19.037368+0000", 2024-09-11T03:51:43.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00080336700000000004 2024-09-11T03:51:43.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:51:43.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:19.037439+0000", 2024-09-11T03:51:43.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1439999999999994e-05 2024-09-11T03:51:43.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:19.037489+0000", 2024-09-11T03:51:43.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9976999999999999e-05 2024-09-11T03:51:43.852 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.852 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.853 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.853 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(2.2d 749/748 [PushOp(2:b6874adf:::smithi17557870-17:13c, version: 730'949, data_included: [2310203~271691,2865972~721096], data_size: 992787, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b6874adf:::smithi17557870-17:13c@730'949, size: 3983457, copy_subset: [(0, 3983457)], clone_subset: {2:b6874adf:::smithi17557870-17:122: [], 2:b6874adf:::smithi17557870-17:13e: []}, snapset: 141=[]:{11c: [119], 122: [120, 11e], 131: [12e, 12a], 13c: [13c, 139, 138, 136, 135, 133], 13e: [13d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3983457, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2310203, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T03:51:43.853 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:27.337413+0000", 2024-09-11T03:51:43.853 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 136.46724885, 2024-09-11T03:51:43.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019805066999999999, 2024-09-11T03:51:43.853 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.853 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.853 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:43.853 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.337413+0000", 2024-09-11T03:51:43.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.337413+0000", 2024-09-11T03:51:43.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.337414+0000", 2024-09-11T03:51:43.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3600000000000004e-07 2024-09-11T03:51:43.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.356189+0000", 2024-09-11T03:51:43.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018775627999999999 2024-09-11T03:51:43.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.356217+0000", 2024-09-11T03:51:43.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7997e-05 2024-09-11T03:51:43.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.356239+0000", 2024-09-11T03:51:43.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2002e-05 2024-09-11T03:51:43.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.356346+0000", 2024-09-11T03:51:43.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010659500000000001 2024-09-11T03:51:43.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:43.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.356362+0000", 2024-09-11T03:51:43.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5923e-05 2024-09-11T03:51:43.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.357218+0000", 2024-09-11T03:51:43.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000856386 2024-09-11T03:51:43.857 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.857 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.857 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.857 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7730 2.20 2:0762776a:::smithi17557870-25:head [copy-from ver 975] snapc 159=[159,157,156,155,154,153,151,150,14d,148,144,143,13d,13c,139,135,133,12a,127,120] ondisk+write+known_if_redirected+supports_pool_eio e771)", 2024-09-11T03:51:43.857 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:49.027947+0000", 2024-09-11T03:51:43.857 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 114.776714759, 2024-09-11T03:51:43.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018472031999999999, 2024-09-11T03:51:43.857 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.857 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.857 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:43.857 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.857 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.858 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.858 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7730 2024-09-11T03:51:43.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.858 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.027947+0000", 2024-09-11T03:51:43.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.027947+0000", 2024-09-11T03:51:43.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.027949+0000", 2024-09-11T03:51:43.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9120000000000001e-06 2024-09-11T03:51:43.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.027957+0000", 2024-09-11T03:51:43.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4619999999999999e-06 2024-09-11T03:51:43.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.027990+0000", 2024-09-11T03:51:43.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3572000000000003e-05 2024-09-11T03:51:43.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.027999+0000", 2024-09-11T03:51:43.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5450000000000003e-06 2024-09-11T03:51:43.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.028074+0000", 2024-09-11T03:51:43.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4826000000000005e-05 2024-09-11T03:51:43.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:51:43.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.028082+0000", 2024-09-11T03:51:43.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5699999999999993e-06 2024-09-11T03:51:43.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.030777+0000", 2024-09-11T03:51:43.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026944030000000002 2024-09-11T03:51:43.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:43.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.030910+0000", 2024-09-11T03:51:43.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013350999999999999 2024-09-11T03:51:43.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5,6", 2024-09-11T03:51:43.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.037273+0000", 2024-09-11T03:51:43.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0063630049999999997 2024-09-11T03:51:43.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:51:43.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.045567+0000", 2024-09-11T03:51:43.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0082938649999999992 2024-09-11T03:51:43.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:43.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.046280+0000", 2024-09-11T03:51:43.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00071292899999999995 2024-09-11T03:51:43.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:43.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.046332+0000", 2024-09-11T03:51:43.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2135000000000001e-05 2024-09-11T03:51:43.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:51:43.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.046379+0000", 2024-09-11T03:51:43.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6783999999999998e-05 2024-09-11T03:51:43.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.046419+0000", 2024-09-11T03:51:43.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0513999999999999e-05 2024-09-11T03:51:43.865 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.865 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.866 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.866 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7731 2.20 2:0762776a:::smithi17557870-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e771)", 2024-09-11T03:51:43.866 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:49.028019+0000", 2024-09-11T03:51:43.866 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 114.776642743, 2024-09-11T03:51:43.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014295922000000001, 2024-09-11T03:51:43.866 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.866 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.866 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:43.866 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.866 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.866 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.867 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7731 2024-09-11T03:51:43.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.867 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.028019+0000", 2024-09-11T03:51:43.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.028019+0000", 2024-09-11T03:51:43.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.028020+0000", 2024-09-11T03:51:43.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5999999999999999e-07 2024-09-11T03:51:43.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.028022+0000", 2024-09-11T03:51:43.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1519999999999999e-06 2024-09-11T03:51:43.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.028029+0000", 2024-09-11T03:51:43.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7089999999999996e-06 2024-09-11T03:51:43.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.028032+0000", 2024-09-11T03:51:43.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8369999999999999e-06 2024-09-11T03:51:43.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.028166+0000", 2024-09-11T03:51:43.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000133301 2024-09-11T03:51:43.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:51:43.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.028174+0000", 2024-09-11T03:51:43.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2810000000000008e-06 2024-09-11T03:51:43.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.031079+0000", 2024-09-11T03:51:43.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002904617 2024-09-11T03:51:43.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:51:43.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.031125+0000", 2024-09-11T03:51:43.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6888000000000002e-05 2024-09-11T03:51:43.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.042128+0000", 2024-09-11T03:51:43.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011002179000000001 2024-09-11T03:51:43.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:43.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.042235+0000", 2024-09-11T03:51:43.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107471 2024-09-11T03:51:43.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:49.042315+0000", 2024-09-11T03:51:43.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0026999999999998e-05 2024-09-11T03:51:43.872 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.872 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.872 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.872 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7897 2.27 2:e6da0655:::smithi17557870-27:head [copy-from ver 951] snapc 160=[160,15f,15e,15d,15c,15b,15a,157,156,155,154,153,151,14d,148,144,143,13d,13c,139,135,133,12a,127,120] ondisk+write+known_if_redirected+supports_pool_eio e780)", 2024-09-11T03:51:43.872 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:58.028735+0000", 2024-09-11T03:51:43.872 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 105.775926834, 2024-09-11T03:51:43.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016862769999999999, 2024-09-11T03:51:43.872 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.872 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.872 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:43.873 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.873 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.873 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.873 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7897 2024-09-11T03:51:43.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.873 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.028735+0000", 2024-09-11T03:51:43.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.028735+0000", 2024-09-11T03:51:43.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.028737+0000", 2024-09-11T03:51:43.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4580000000000001e-06 2024-09-11T03:51:43.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.028743+0000", 2024-09-11T03:51:43.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0920000000000003e-06 2024-09-11T03:51:43.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.028757+0000", 2024-09-11T03:51:43.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4089e-05 2024-09-11T03:51:43.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.028764+0000", 2024-09-11T03:51:43.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1210000000000001e-06 2024-09-11T03:51:43.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.028816+0000", 2024-09-11T03:51:43.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2008000000000001e-05 2024-09-11T03:51:43.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:51:43.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.028821+0000", 2024-09-11T03:51:43.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8879999999999996e-06 2024-09-11T03:51:43.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.030514+0000", 2024-09-11T03:51:43.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016936029999999999 2024-09-11T03:51:43.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:43.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.030582+0000", 2024-09-11T03:51:43.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7107000000000006e-05 2024-09-11T03:51:43.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,9", 2024-09-11T03:51:43.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.039212+0000", 2024-09-11T03:51:43.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086303479999999995 2024-09-11T03:51:43.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:51:43.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.043556+0000", 2024-09-11T03:51:43.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043436969999999997 2024-09-11T03:51:43.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:43.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.043874+0000", 2024-09-11T03:51:43.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000318859 2024-09-11T03:51:43.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:43.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.045527+0000", 2024-09-11T03:51:43.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016527639999999999 2024-09-11T03:51:43.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:51:43.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.045564+0000", 2024-09-11T03:51:43.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6366999999999999e-05 2024-09-11T03:51:43.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.045598+0000", 2024-09-11T03:51:43.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4369000000000002e-05 2024-09-11T03:51:43.879 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.879 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.879 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.879 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7898 2.27 2:e6da0655:::smithi17557870-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e780)", 2024-09-11T03:51:43.880 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:58.028780+0000", 2024-09-11T03:51:43.880 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 105.77588210099999, 2024-09-11T03:51:43.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014094260000000001, 2024-09-11T03:51:43.880 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.880 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.880 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:43.880 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.880 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.880 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.880 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7898 2024-09-11T03:51:43.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.880 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.028780+0000", 2024-09-11T03:51:43.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.028780+0000", 2024-09-11T03:51:43.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.028780+0000", 2024-09-11T03:51:43.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3700000000000001e-07 2024-09-11T03:51:43.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.028782+0000", 2024-09-11T03:51:43.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8470000000000001e-06 2024-09-11T03:51:43.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.028787+0000", 2024-09-11T03:51:43.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6330000000000004e-06 2024-09-11T03:51:43.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.028790+0000", 2024-09-11T03:51:43.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.269e-06 2024-09-11T03:51:43.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.028881+0000", 2024-09-11T03:51:43.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1061000000000001e-05 2024-09-11T03:51:43.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:51:43.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.028887+0000", 2024-09-11T03:51:43.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0299999999999999e-06 2024-09-11T03:51:43.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.030694+0000", 2024-09-11T03:51:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018065069999999999 2024-09-11T03:51:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:51:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.030731+0000", 2024-09-11T03:51:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7761e-05 2024-09-11T03:51:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.042641+0000", 2024-09-11T03:51:43.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011909367000000001 2024-09-11T03:51:43.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:43.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.042749+0000", 2024-09-11T03:51:43.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010844999999999999 2024-09-11T03:51:43.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:58.042874+0000", 2024-09-11T03:51:43.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000124898 2024-09-11T03:51:43.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.885 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.886 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8290 2.3b 2:dde92fc6:::smithi17557870-41:15e [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e813)", 2024-09-11T03:51:43.886 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:50:31.050039+0000", 2024-09-11T03:51:43.886 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.754622733999994, 2024-09-11T03:51:43.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.47934163699999999, 2024-09-11T03:51:43.886 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.886 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.886 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-11T03:51:43.886 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.886 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.886 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.886 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8290 2024-09-11T03:51:43.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.887 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050039+0000", 2024-09-11T03:51:43.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050039+0000", 2024-09-11T03:51:43.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050041+0000", 2024-09-11T03:51:43.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1289999999999999e-06 2024-09-11T03:51:43.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050047+0000", 2024-09-11T03:51:43.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6210000000000001e-06 2024-09-11T03:51:43.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050061+0000", 2024-09-11T03:51:43.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3983e-05 2024-09-11T03:51:43.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050073+0000", 2024-09-11T03:51:43.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1664e-05 2024-09-11T03:51:43.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050501+0000", 2024-09-11T03:51:43.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000428365 2024-09-11T03:51:43.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-09-11T03:51:43.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050514+0000", 2024-09-11T03:51:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2639e-05 2024-09-11T03:51:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.529166+0000", 2024-09-11T03:51:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.478652566 2024-09-11T03:51:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.529381+0000", 2024-09-11T03:51:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021467 2024-09-11T03:51:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.890 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.891 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8291 2.3b 2:dde92fc6:::smithi17557870-41:15e [sparse-read 0~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e813)", 2024-09-11T03:51:43.891 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:50:31.050090+0000", 2024-09-11T03:51:43.891 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.754571620999997, 2024-09-11T03:51:43.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.47966582600000002, 2024-09-11T03:51:43.891 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.891 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.891 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-11T03:51:43.891 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.891 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.891 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.891 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8291 2024-09-11T03:51:43.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.892 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050090+0000", 2024-09-11T03:51:43.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050090+0000", 2024-09-11T03:51:43.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050091+0000", 2024-09-11T03:51:43.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4899999999999995e-07 2024-09-11T03:51:43.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050094+0000", 2024-09-11T03:51:43.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9349999999999999e-06 2024-09-11T03:51:43.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050102+0000", 2024-09-11T03:51:43.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7870000000000002e-06 2024-09-11T03:51:43.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050108+0000", 2024-09-11T03:51:43.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0809999999999999e-06 2024-09-11T03:51:43.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050559+0000", 2024-09-11T03:51:43.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045160799999999999 2024-09-11T03:51:43.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-09-11T03:51:43.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050569+0000", 2024-09-11T03:51:43.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8260000000000005e-06 2024-09-11T03:51:43.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.529662+0000", 2024-09-11T03:51:43.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.47909304200000002 2024-09-11T03:51:43.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.529756+0000", 2024-09-11T03:51:43.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3898000000000001e-05 2024-09-11T03:51:43.895 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.895 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.895 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.896 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8292 2.3b 2:dde92fc6:::smithi17557870-41:15e [sparse-read 0~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e813)", 2024-09-11T03:51:43.896 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:50:31.050117+0000", 2024-09-11T03:51:43.896 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.754544726000006, 2024-09-11T03:51:43.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.47981013, 2024-09-11T03:51:43.896 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.896 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.896 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-11T03:51:43.896 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.896 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.896 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.896 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8292 2024-09-11T03:51:43.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.897 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050117+0000", 2024-09-11T03:51:43.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050117+0000", 2024-09-11T03:51:43.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050118+0000", 2024-09-11T03:51:43.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5000000000000003e-07 2024-09-11T03:51:43.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050120+0000", 2024-09-11T03:51:43.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5100000000000001e-06 2024-09-11T03:51:43.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050127+0000", 2024-09-11T03:51:43.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5570000000000001e-06 2024-09-11T03:51:43.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050132+0000", 2024-09-11T03:51:43.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3290000000000001e-06 2024-09-11T03:51:43.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050633+0000", 2024-09-11T03:51:43.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050093900000000005 2024-09-11T03:51:43.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-09-11T03:51:43.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.050644+0000", 2024-09-11T03:51:43.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0896e-05 2024-09-11T03:51:43.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.529831+0000", 2024-09-11T03:51:43.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.479187317 2024-09-11T03:51:43.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:31.529927+0000", 2024-09-11T03:51:43.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6032000000000005e-05 2024-09-11T03:51:43.900 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.901 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.901 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.901 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8490 2.27 2:e77ce595:::smithi17557870-31:head [copy-from ver 1001] snapc 17c=[17c,17b,17a,179,178,177,176,175,174,173,172,171,170,16f,16e,16b,169,168,167,164,15f,15e,15d,15b,15a,157,156,155,14d,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e828)", 2024-09-11T03:51:43.901 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:50:46.070578+0000", 2024-09-11T03:51:43.901 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 57.734084340000003, 2024-09-11T03:51:43.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020899484999999999, 2024-09-11T03:51:43.901 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.901 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.901 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:43.901 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.901 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.901 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.902 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8490 2024-09-11T03:51:43.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.902 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.070578+0000", 2024-09-11T03:51:43.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.070578+0000", 2024-09-11T03:51:43.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.070578+0000", 2024-09-11T03:51:43.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9699999999999996e-07 2024-09-11T03:51:43.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.070582+0000", 2024-09-11T03:51:43.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5279999999999999e-06 2024-09-11T03:51:43.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.070589+0000", 2024-09-11T03:51:43.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3200000000000002e-06 2024-09-11T03:51:43.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.070595+0000", 2024-09-11T03:51:43.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0599999999999996e-06 2024-09-11T03:51:43.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.070652+0000", 2024-09-11T03:51:43.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7011e-05 2024-09-11T03:51:43.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:43.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.070769+0000", 2024-09-11T03:51:43.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117069 2024-09-11T03:51:43.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,9", 2024-09-11T03:51:43.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.082788+0000", 2024-09-11T03:51:43.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012019161 2024-09-11T03:51:43.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:51:43.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.086123+0000", 2024-09-11T03:51:43.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033348430000000001 2024-09-11T03:51:43.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:43.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.086900+0000", 2024-09-11T03:51:43.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00077681799999999997 2024-09-11T03:51:43.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:43.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.091398+0000", 2024-09-11T03:51:43.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004498429 2024-09-11T03:51:43.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:51:43.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.091437+0000", 2024-09-11T03:51:43.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8682e-05 2024-09-11T03:51:43.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.091477+0000", 2024-09-11T03:51:43.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9966999999999999e-05 2024-09-11T03:51:43.907 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.907 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.907 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.907 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8491 2.27 2:e77ce595:::smithi17557870-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e828)", 2024-09-11T03:51:43.907 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:50:46.070738+0000", 2024-09-11T03:51:43.908 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 57.733923742999998, 2024-09-11T03:51:43.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014970831, 2024-09-11T03:51:43.908 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.908 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.908 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:43.908 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.908 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.908 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.908 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8491 2024-09-11T03:51:43.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.908 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.070738+0000", 2024-09-11T03:51:43.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.070738+0000", 2024-09-11T03:51:43.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.070739+0000", 2024-09-11T03:51:43.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1100000000000005e-07 2024-09-11T03:51:43.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.070745+0000", 2024-09-11T03:51:43.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3040000000000001e-06 2024-09-11T03:51:43.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.070755+0000", 2024-09-11T03:51:43.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.825e-06 2024-09-11T03:51:43.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.070762+0000", 2024-09-11T03:51:43.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9299999999999997e-06 2024-09-11T03:51:43.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.070930+0000", 2024-09-11T03:51:43.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016762400000000001 2024-09-11T03:51:43.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:51:43.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.070973+0000", 2024-09-11T03:51:43.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3699999999999998e-05 2024-09-11T03:51:43.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.085561+0000", 2024-09-11T03:51:43.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014587562 2024-09-11T03:51:43.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:43.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.085635+0000", 2024-09-11T03:51:43.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4474000000000006e-05 2024-09-11T03:51:43.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.085709+0000", 2024-09-11T03:51:43.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3601000000000002e-05 2024-09-11T03:51:43.913 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.913 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.913 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.913 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8651 2.27 2:e6da0655:::smithi17557870-27:head [delete] snapc 182=[182,181,180,17f,17e,17d,17c,17a,179,178,177,176,175,174,172,171,170,16e,16b,169,167,164,15f,15e,15d,15b,15a,157,14d,144,143,13c,139,127,120] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e840)", 2024-09-11T03:51:43.913 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:50:58.045333+0000", 2024-09-11T03:51:43.913 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 45.759328877000002, 2024-09-11T03:51:43.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013965458, 2024-09-11T03:51:43.913 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.913 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.913 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:43.913 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.914 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.914 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.914 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8651 2024-09-11T03:51:43.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.914 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:58.045333+0000", 2024-09-11T03:51:43.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:58.045333+0000", 2024-09-11T03:51:43.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:58.045335+0000", 2024-09-11T03:51:43.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2249999999999999e-06 2024-09-11T03:51:43.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:58.045343+0000", 2024-09-11T03:51:43.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5839999999999997e-06 2024-09-11T03:51:43.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:58.045360+0000", 2024-09-11T03:51:43.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7033000000000001e-05 2024-09-11T03:51:43.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:58.045371+0000", 2024-09-11T03:51:43.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0569e-05 2024-09-11T03:51:43.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:58.045451+0000", 2024-09-11T03:51:43.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0846999999999994e-05 2024-09-11T03:51:43.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:51:43.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:58.045457+0000", 2024-09-11T03:51:43.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6119999999999998e-06 2024-09-11T03:51:43.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:58.053183+0000", 2024-09-11T03:51:43.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00772647 2024-09-11T03:51:43.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:43.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:58.053300+0000", 2024-09-11T03:51:43.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011626200000000001 2024-09-11T03:51:43.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:58.059299+0000", 2024-09-11T03:51:43.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059988560000000003 2024-09-11T03:51:43.918 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.918 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.918 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.918 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8938 2.20 2:0762776a:::smithi17557870-25:head [copy-from ver 1008] snapc 190=[190,18e,18d,18c,18b,18a,189,188,187,186,185,184,183,181,180,17e,17d,17c,17a,178,177,175,174,172,170,16e,16b,169,167,164,15f,15e,15b,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-09-11T03:51:43.919 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:51:22.236254+0000", 2024-09-11T03:51:43.919 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.568408127000001, 2024-09-11T03:51:43.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017588821000000001, 2024-09-11T03:51:43.919 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.919 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.919 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:43.919 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.919 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.919 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.919 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8938 2024-09-11T03:51:43.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.919 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.236254+0000", 2024-09-11T03:51:43.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.236254+0000", 2024-09-11T03:51:43.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.236256+0000", 2024-09-11T03:51:43.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7600000000000001e-06 2024-09-11T03:51:43.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.236262+0000", 2024-09-11T03:51:43.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2009999999999996e-06 2024-09-11T03:51:43.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.236277+0000", 2024-09-11T03:51:43.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4977000000000001e-05 2024-09-11T03:51:43.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.236286+0000", 2024-09-11T03:51:43.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6849999999999996e-06 2024-09-11T03:51:43.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.236352+0000", 2024-09-11T03:51:43.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5534e-05 2024-09-11T03:51:43.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:51:43.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.236358+0000", 2024-09-11T03:51:43.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0209999999999996e-06 2024-09-11T03:51:43.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.238475+0000", 2024-09-11T03:51:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021168269999999999 2024-09-11T03:51:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.238598+0000", 2024-09-11T03:51:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123001 2024-09-11T03:51:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5,6", 2024-09-11T03:51:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.245896+0000", 2024-09-11T03:51:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072983550000000003 2024-09-11T03:51:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:51:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.253003+0000", 2024-09-11T03:51:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071072699999999997 2024-09-11T03:51:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.253599+0000", 2024-09-11T03:51:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059532900000000002 2024-09-11T03:51:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.253683+0000", 2024-09-11T03:51:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4492000000000004e-05 2024-09-11T03:51:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:51:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.253764+0000", 2024-09-11T03:51:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1119999999999996e-05 2024-09-11T03:51:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.253843+0000", 2024-09-11T03:51:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8249000000000002e-05 2024-09-11T03:51:43.926 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.926 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.926 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.926 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8939 2.20 2:0762776a:::smithi17557870-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e864)", 2024-09-11T03:51:43.926 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:51:22.236308+0000", 2024-09-11T03:51:43.926 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.568353637000001, 2024-09-11T03:51:43.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013673395, 2024-09-11T03:51:43.926 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.926 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.926 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:51:43.927 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.927 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.927 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.927 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8939 2024-09-11T03:51:43.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.927 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.236308+0000", 2024-09-11T03:51:43.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.236308+0000", 2024-09-11T03:51:43.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.236309+0000", 2024-09-11T03:51:43.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8899999999999999e-07 2024-09-11T03:51:43.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.236317+0000", 2024-09-11T03:51:43.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7610000000000007e-06 2024-09-11T03:51:43.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.236324+0000", 2024-09-11T03:51:43.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7929999999999998e-06 2024-09-11T03:51:43.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.236331+0000", 2024-09-11T03:51:43.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.708e-06 2024-09-11T03:51:43.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.236412+0000", 2024-09-11T03:51:43.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0777000000000003e-05 2024-09-11T03:51:43.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:51:43.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.236416+0000", 2024-09-11T03:51:43.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3039999999999998e-06 2024-09-11T03:51:43.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.238756+0000", 2024-09-11T03:51:43.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023393559999999999 2024-09-11T03:51:43.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:51:43.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.238805+0000", 2024-09-11T03:51:43.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9558999999999998e-05 2024-09-11T03:51:43.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.249622+0000", 2024-09-11T03:51:43.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010817082 2024-09-11T03:51:43.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:43.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.249833+0000", 2024-09-11T03:51:43.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000211165 2024-09-11T03:51:43.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:22.249982+0000", 2024-09-11T03:51:43.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014830099999999999 2024-09-11T03:51:43.932 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.932 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.932 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.933 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:9045 2.43 2:c3768452:::smithi17557870-11:head [copy-from ver 1020] snapc 195=[195,194,193,192,191,190,18e,18d,18c,18b,18a,189,188,186,183,180,17e,17d,17c,17a,178,177,175,174,172,16e,16b,169,167,164,15f,15e,15b,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e874)", 2024-09-11T03:51:43.933 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:51:32.061652+0000", 2024-09-11T03:51:43.933 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.743009649999999, 2024-09-11T03:51:43.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015987857000000001, 2024-09-11T03:51:43.933 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:51:43.933 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:51:43.933 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:43.933 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:51:43.933 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:51:43.933 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:43.933 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 9045 2024-09-11T03:51:43.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.934 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:51:43.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:51:43.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:32.061652+0000", 2024-09-11T03:51:43.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:51:43.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:32.061652+0000", 2024-09-11T03:51:43.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:51:43.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:51:43.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:32.061654+0000", 2024-09-11T03:51:43.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0370000000000001e-06 2024-09-11T03:51:43.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:51:43.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:32.061665+0000", 2024-09-11T03:51:43.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0866000000000001e-05 2024-09-11T03:51:43.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:51:43.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:32.061693+0000", 2024-09-11T03:51:43.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7313000000000001e-05 2024-09-11T03:51:43.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:51:43.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:32.061706+0000", 2024-09-11T03:51:43.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3507e-05 2024-09-11T03:51:43.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:51:43.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:32.061847+0000", 2024-09-11T03:51:43.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014133400000000001 2024-09-11T03:51:43.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:51:43.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:32.061917+0000", 2024-09-11T03:51:43.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9875000000000004e-05 2024-09-11T03:51:43.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,9", 2024-09-11T03:51:43.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:32.067041+0000", 2024-09-11T03:51:43.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051235660000000004 2024-09-11T03:51:43.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:51:43.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:32.070961+0000", 2024-09-11T03:51:43.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039204679999999999 2024-09-11T03:51:43.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:43.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:32.076968+0000", 2024-09-11T03:51:43.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060062919999999999 2024-09-11T03:51:43.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:43.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:32.077464+0000", 2024-09-11T03:51:43.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049622499999999999 2024-09-11T03:51:43.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:51:43.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:32.077564+0000", 2024-09-11T03:51:43.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9955000000000001e-05 2024-09-11T03:51:43.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:51:43.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:51:43.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:51:43.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:32.077640+0000", 2024-09-11T03:51:43.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6419000000000001e-05 2024-09-11T03:51:43.939 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.939 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.939 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.939 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:51:43.939 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:51:43.939 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:51:43.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:43.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:43.962 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:43.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:44.059 INFO:tasks.rados.rados.2.smithi175.stdout:3669: done (1 left) 2024-09-11T03:51:44.059 INFO:tasks.rados.rados.2.smithi175.stdout:3670: snap_remove snap 286 2024-09-11T03:51:44.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:44.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:44.088 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:44.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:44.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:44.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:44.212 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:44.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:44.313 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:44.339 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:44.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:44.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:44 smithi084 ceph-mon[27254]: pgmap v1482: 107 pgs: 107 active+clean; 755 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 11 op/s 2024-09-11T03:51:44.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:44 smithi084 ceph-mon[27254]: osdmap e886: 12 total, 12 up, 11 in 2024-09-11T03:51:44.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:44 smithi121 ceph-mon[30729]: pgmap v1482: 107 pgs: 107 active+clean; 755 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 11 op/s 2024-09-11T03:51:44.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:44 smithi121 ceph-mon[30729]: osdmap e886: 12 total, 12 up, 11 in 2024-09-11T03:51:44.441 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-11T03:51:44.441 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:44.463 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:44.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:44 smithi175 ceph-mon[30894]: pgmap v1482: 107 pgs: 107 active+clean; 755 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 11 op/s 2024-09-11T03:51:44.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:44 smithi175 ceph-mon[30894]: osdmap e886: 12 total, 12 up, 11 in 2024-09-11T03:51:44.565 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:44.591 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:44.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:44.591 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:44.681 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:44.692 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:44.717 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:44.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:44.719 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:44.786 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-11T03:51:44.820 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:44.853 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:44.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:44.854 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:44.956 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:44.977 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:44.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:44.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:45.040 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:51:45.073 INFO:tasks.rados.rados.2.smithi175.stdout:3670: done (1 left) 2024-09-11T03:51:45.073 INFO:tasks.rados.rados.2.smithi175.stdout:3671: read oid 47 snap -1 2024-09-11T03:51:45.073 INFO:tasks.rados.rados.2.smithi175.stdout:3671: expect (ObjNum 1188 snap 388 seq_num 1188) 2024-09-11T03:51:45.073 INFO:tasks.rados.rados.2.smithi175.stdout:3672: snap_remove snap 398 2024-09-11T03:51:45.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:45.100 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:45.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:45.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:45.118 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:51:45.202 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:45.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:45.224 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:45.224 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:45.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:45.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:45.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:45.379 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:45.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:45 smithi084 ceph-mon[27254]: 2.36 scrub starts 2024-09-11T03:51:45.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:45 smithi084 ceph-mon[27254]: 2.36 scrub ok 2024-09-11T03:51:45.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:45 smithi084 ceph-mon[27254]: 2.1e scrub starts 2024-09-11T03:51:45.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:45 smithi084 ceph-mon[27254]: 2.1e scrub ok 2024-09-11T03:51:45.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:45 smithi084 ceph-mon[27254]: 2.32 scrub starts 2024-09-11T03:51:45.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:45 smithi084 ceph-mon[27254]: 2.32 scrub ok 2024-09-11T03:51:45.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:45 smithi084 ceph-mon[27254]: 2.1b scrub starts 2024-09-11T03:51:45.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:45 smithi084 ceph-mon[27254]: 2.1b scrub ok 2024-09-11T03:51:45.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:45 smithi084 ceph-mon[27254]: osdmap e887: 12 total, 12 up, 11 in 2024-09-11T03:51:45.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:45 smithi121 ceph-mon[30729]: 2.36 scrub starts 2024-09-11T03:51:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:45 smithi121 ceph-mon[30729]: 2.36 scrub ok 2024-09-11T03:51:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:45 smithi121 ceph-mon[30729]: 2.1e scrub starts 2024-09-11T03:51:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:45 smithi121 ceph-mon[30729]: 2.1e scrub ok 2024-09-11T03:51:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:45 smithi121 ceph-mon[30729]: 2.32 scrub starts 2024-09-11T03:51:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:45 smithi121 ceph-mon[30729]: 2.32 scrub ok 2024-09-11T03:51:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:45 smithi121 ceph-mon[30729]: 2.1b scrub starts 2024-09-11T03:51:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:45 smithi121 ceph-mon[30729]: 2.1b scrub ok 2024-09-11T03:51:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:45 smithi121 ceph-mon[30729]: osdmap e887: 12 total, 12 up, 11 in 2024-09-11T03:51:45.446 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:45 smithi175 ceph-mon[30894]: 2.36 scrub starts 2024-09-11T03:51:45.446 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:45 smithi175 ceph-mon[30894]: 2.36 scrub ok 2024-09-11T03:51:45.446 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:45 smithi175 ceph-mon[30894]: 2.1e scrub starts 2024-09-11T03:51:45.446 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:45 smithi175 ceph-mon[30894]: 2.1e scrub ok 2024-09-11T03:51:45.446 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:45 smithi175 ceph-mon[30894]: 2.32 scrub starts 2024-09-11T03:51:45.446 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:45 smithi175 ceph-mon[30894]: 2.32 scrub ok 2024-09-11T03:51:45.446 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:45 smithi175 ceph-mon[30894]: 2.1b scrub starts 2024-09-11T03:51:45.447 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:45 smithi175 ceph-mon[30894]: 2.1b scrub ok 2024-09-11T03:51:45.447 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:45 smithi175 ceph-mon[30894]: osdmap e887: 12 total, 12 up, 11 in 2024-09-11T03:51:45.450 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:45.471 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:45.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:45.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:45.557 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:51:45.557 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:51:45.557 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:51:45.557 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:51:45.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:45.594 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:45.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:45.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:45.696 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:45.717 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:45.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:45.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:45.780 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:51:45.794 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:51:45.795 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:51:45.812 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:51:45.812 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:51:45.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:45.828 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:51:45.828 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:51:45.839 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:45.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:45.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:45.845 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:51:45.845 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:51:45.862 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:51:45.862 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:51:45.879 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:51:45.879 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:51:45.895 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:51:45.895 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:51:45.913 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:51:45.913 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:51:45.929 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:51:45.929 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:51:45.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:45.946 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:51:45.946 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:51:45.963 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:45.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:45.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:45.966 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:51:45.966 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:51:45.986 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:51:45.986 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:51:46.057 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2024-09-11T03:51:46.062 INFO:tasks.rados.rados.2.smithi175.stdout:3672: done (2 left) 2024-09-11T03:51:46.062 INFO:tasks.rados.rados.2.smithi175.stdout:3673: rollback oid 36 current snap is 410 2024-09-11T03:51:46.062 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 36 to 293 2024-09-11T03:51:46.062 INFO:tasks.rados.rados.2.smithi175.stdout:3674: snap_remove snap 408 2024-09-11T03:51:46.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:46.087 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:46.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:46.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:46.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:46.210 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:46.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:46.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:46.264 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-11T03:51:46.312 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:46.334 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:46.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:46.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:46.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:46 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:51:46.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:46 smithi084 ceph-mon[27254]: 2.23 scrub starts 2024-09-11T03:51:46.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:46 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:51:46.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:46 smithi084 ceph-mon[27254]: 2.23 scrub ok 2024-09-11T03:51:46.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:46 smithi084 ceph-mon[27254]: pgmap v1485: 107 pgs: 107 active+clean; 758 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 11 op/s 2024-09-11T03:51:46.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:46 smithi084 ceph-mon[27254]: 2.3c scrub starts 2024-09-11T03:51:46.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:46 smithi084 ceph-mon[27254]: 2.3c scrub ok 2024-09-11T03:51:46.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1542825785' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:51:46.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2856052579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:46.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1542825785' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:51:46.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:46 smithi084 ceph-mon[27254]: osdmap e888: 12 total, 12 up, 11 in 2024-09-11T03:51:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:46 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:51:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:46 smithi121 ceph-mon[30729]: 2.23 scrub starts 2024-09-11T03:51:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:46 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:51:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:46 smithi121 ceph-mon[30729]: 2.23 scrub ok 2024-09-11T03:51:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:46 smithi121 ceph-mon[30729]: pgmap v1485: 107 pgs: 107 active+clean; 758 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 11 op/s 2024-09-11T03:51:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:46 smithi121 ceph-mon[30729]: 2.3c scrub starts 2024-09-11T03:51:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:46 smithi121 ceph-mon[30729]: 2.3c scrub ok 2024-09-11T03:51:46.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1542825785' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:51:46.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2856052579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:46.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1542825785' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:51:46.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:46 smithi121 ceph-mon[30729]: osdmap e888: 12 total, 12 up, 11 in 2024-09-11T03:51:46.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:46.456 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:46.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:46.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:46.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:46 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:51:46.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:46 smithi175 ceph-mon[30894]: 2.23 scrub starts 2024-09-11T03:51:46.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:46 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:51:46.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:46 smithi175 ceph-mon[30894]: 2.23 scrub ok 2024-09-11T03:51:46.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:46 smithi175 ceph-mon[30894]: pgmap v1485: 107 pgs: 107 active+clean; 758 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 11 op/s 2024-09-11T03:51:46.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:46 smithi175 ceph-mon[30894]: 2.3c scrub starts 2024-09-11T03:51:46.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:46 smithi175 ceph-mon[30894]: 2.3c scrub ok 2024-09-11T03:51:46.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1542825785' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:51:46.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2856052579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:46.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1542825785' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:51:46.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:46 smithi175 ceph-mon[30894]: osdmap e888: 12 total, 12 up, 11 in 2024-09-11T03:51:46.506 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:51:46.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:46.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:46.582 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:46.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:46.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:46.704 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:46.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:46.806 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:46.835 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:46.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:46.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:46.937 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:46.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:46.960 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:46.960 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:46.991 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:51:46.991 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:51:47.056 INFO:tasks.rados.rados.2.smithi175.stdout:3674: done (3 left) 2024-09-11T03:51:47.057 INFO:tasks.rados.rados.2.smithi175.stdout:3675: copy_from oid 35 from oid 24 current snap is 410 2024-09-11T03:51:47.057 INFO:tasks.rados.rados.2.smithi175.stdout:3667: expect (ObjNum 1134 snap 367 seq_num 1134) 2024-09-11T03:51:47.061 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:47.087 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:47.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:47.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:47.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:47.201 INFO:tasks.rados.rados.2.smithi175.stdout:3671: expect (ObjNum 1188 snap 388 seq_num 1188) 2024-09-11T03:51:47.212 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:47.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:47.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:47.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:47.329 INFO:tasks.rados.rados.2.smithi175.stdout:3673: finishing rollback tid 0 to smithi17557870-36 2024-09-11T03:51:47.329 INFO:tasks.rados.rados.2.smithi175.stdout:3667: done (3 left) 2024-09-11T03:51:47.329 INFO:tasks.rados.rados.2.smithi175.stdout:3671: done (2 left) 2024-09-11T03:51:47.329 INFO:tasks.rados.rados.2.smithi175.stdout:3676: snap_create 2024-09-11T03:51:47.329 INFO:tasks.rados.rados.2.smithi175.stdout:3673: finishing rollback tid 1 to smithi17557870-36 2024-09-11T03:51:47.329 INFO:tasks.rados.rados.2.smithi175.stdout:3673: finishing rollback tid 2 to smithi17557870-36 2024-09-11T03:51:47.329 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 1370 (ObjNum 830 snap 263 seq_num 830) dirty exists 2024-09-11T03:51:47.329 INFO:tasks.rados.rados.2.smithi175.stdout:3675: finishing copy_from racing read to smithi17557870-35 2024-09-11T03:51:47.329 INFO:tasks.rados.rados.2.smithi175.stdout:3675: finishing copy_from to smithi17557870-35 2024-09-11T03:51:47.329 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 1082 (ObjNum 1235 snap 402 seq_num 1235) dirty exists 2024-09-11T03:51:47.335 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:47.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:47.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:47.437 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:47.458 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:47.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:47.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:47.465 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 8 1 2024-09-11T03:51:47.482 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:47.560 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:47.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:47.583 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:47.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:47.684 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:47.706 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:47.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:47.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:47.777 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-11T03:51:47.808 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:47.840 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:47.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:47.841 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:47.942 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:47.964 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:47.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:47.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:47.986 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:48.018 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:51:48.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:48.087 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:48.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:48.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:48.119 INFO:tasks.rados.rados.2.smithi175.stdout:3673: done (2 left) 2024-09-11T03:51:48.119 INFO:tasks.rados.rados.2.smithi175.stdout:3675: done (1 left) 2024-09-11T03:51:48.119 INFO:tasks.rados.rados.2.smithi175.stdout:3676: done (0 left) 2024-09-11T03:51:48.119 INFO:tasks.rados.rados.2.smithi175.stdout:3677: read oid 42 snap -1 2024-09-11T03:51:48.120 INFO:tasks.rados.rados.2.smithi175.stdout:3677: expect deleted 2024-09-11T03:51:48.120 INFO:tasks.rados.rados.2.smithi175.stdout:3678: rollback oid 36 current snap is 411 2024-09-11T03:51:48.120 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 36 to 354 2024-09-11T03:51:48.120 INFO:tasks.rados.rados.2.smithi175.stdout:3679: write (excl) oid 23 current snap is 411 2024-09-11T03:51:48.120 INFO:tasks.rados.rados.2.smithi175.stdout:3679: seq_num 1246 ranges {541522=725793,1946446=635431} 2024-09-11T03:51:48.126 INFO:tasks.rados.rados.2.smithi175.stdout:3679: writing smithi17557870-23 from 541522 to 1267315 tid 1 2024-09-11T03:51:48.133 INFO:tasks.rados.rados.2.smithi175.stdout:3679: writing smithi17557870-23 from 1946446 to 2581877 tid 2 2024-09-11T03:51:48.133 INFO:tasks.rados.rados.2.smithi175.stdout:3680: snap_remove snap 373 2024-09-11T03:51:48.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:48.210 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:48.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:48.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:48.312 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:48.333 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:48.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:48.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:48.374 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:48 smithi175 ceph-mon[30894]: pgmap v1488: 107 pgs: 107 active+clean; 729 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 3.4 MiB/s rd, 747 KiB/s wr, 2 op/s 2024-09-11T03:51:48.374 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:48 smithi175 ceph-mon[30894]: 1.b scrub starts 2024-09-11T03:51:48.374 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:48 smithi175 ceph-mon[30894]: 1.b scrub ok 2024-09-11T03:51:48.374 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:48 smithi175 ceph-mon[30894]: osdmap e889: 12 total, 12 up, 11 in 2024-09-11T03:51:48.374 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:48 smithi175 ceph-mon[30894]: 2.2f scrub starts 2024-09-11T03:51:48.374 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:48 smithi175 ceph-mon[30894]: 2.2f scrub ok 2024-09-11T03:51:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:48 smithi084 ceph-mon[27254]: pgmap v1488: 107 pgs: 107 active+clean; 729 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 3.4 MiB/s rd, 747 KiB/s wr, 2 op/s 2024-09-11T03:51:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:48 smithi084 ceph-mon[27254]: 1.b scrub starts 2024-09-11T03:51:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:48 smithi084 ceph-mon[27254]: 1.b scrub ok 2024-09-11T03:51:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:48 smithi084 ceph-mon[27254]: osdmap e889: 12 total, 12 up, 11 in 2024-09-11T03:51:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:48 smithi084 ceph-mon[27254]: 2.2f scrub starts 2024-09-11T03:51:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:48 smithi084 ceph-mon[27254]: 2.2f scrub ok 2024-09-11T03:51:48.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:48 smithi121 ceph-mon[30729]: pgmap v1488: 107 pgs: 107 active+clean; 729 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 3.4 MiB/s rd, 747 KiB/s wr, 2 op/s 2024-09-11T03:51:48.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:48 smithi121 ceph-mon[30729]: 1.b scrub starts 2024-09-11T03:51:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:48 smithi121 ceph-mon[30729]: 1.b scrub ok 2024-09-11T03:51:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:48 smithi121 ceph-mon[30729]: osdmap e889: 12 total, 12 up, 11 in 2024-09-11T03:51:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:48 smithi121 ceph-mon[30729]: 2.2f scrub starts 2024-09-11T03:51:48.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:48 smithi121 ceph-mon[30729]: 2.2f scrub ok 2024-09-11T03:51:48.435 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:48.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:48.457 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:48.458 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:48.482 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:51:48.482 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:51:48.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:48.580 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:48.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:48.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:48.682 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:48.703 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:48.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:48.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:48.774 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:51:48.805 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:48.826 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:48.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:48.827 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:48.928 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:48.949 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:48.950 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:49.013 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:49.051 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:49.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:49.073 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:49.074 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:49.124 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.8 to 1 (10000) 2024-09-11T03:51:49.124 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:51:49.127 INFO:tasks.rados.rados.2.smithi175.stdout:3680: done (3 left) 2024-09-11T03:51:49.127 INFO:tasks.rados.rados.2.smithi175.stdout:3681: copy_from oid 22 from oid 30 current snap is 411 2024-09-11T03:51:49.127 INFO:tasks.rados.rados.2.smithi175.stdout:3678: finishing rollback tid 0 to smithi17557870-36 2024-09-11T03:51:49.127 INFO:tasks.rados.rados.2.smithi175.stdout:3678: finishing rollback tid 1 to smithi17557870-36 2024-09-11T03:51:49.127 INFO:tasks.rados.rados.2.smithi175.stdout:3678: finishing rollback tid 2 to smithi17557870-36 2024-09-11T03:51:49.127 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 1373 (ObjNum 1086 snap 353 seq_num 1086) dirty exists 2024-09-11T03:51:49.127 INFO:tasks.rados.rados.2.smithi175.stdout:3679: finishing write tid 1 to smithi17557870-23 2024-09-11T03:51:49.127 INFO:tasks.rados.rados.2.smithi175.stdout:3679: finishing write tid 2 to smithi17557870-23 2024-09-11T03:51:49.128 INFO:tasks.rados.rados.2.smithi175.stdout:3679: finishing write tid 3 to smithi17557870-23 2024-09-11T03:51:49.128 INFO:tasks.rados.rados.2.smithi175.stdout:3679: finishing write tid 4 to smithi17557870-23 2024-09-11T03:51:49.128 INFO:tasks.rados.rados.2.smithi175.stdout:3679: oid 23 updating version 0 to 1376 2024-09-11T03:51:49.128 INFO:tasks.rados.rados.2.smithi175.stdout:3679: oid 23 version 1376 is already newer than 1375 2024-09-11T03:51:49.128 INFO:tasks.rados.rados.2.smithi175.stdout:3679: oid 23 version 1376 is already newer than 1374 2024-09-11T03:51:49.128 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 1376 (ObjNum 1246 snap 411 seq_num 1246) dirty exists 2024-09-11T03:51:49.128 INFO:tasks.rados.rados.2.smithi175.stdout:3679: left oid 23 (ObjNum 1246 snap 411 seq_num 1246) 2024-09-11T03:51:49.128 INFO:tasks.rados.rados.2.smithi175.stdout:3677: done (3 left) 2024-09-11T03:51:49.128 INFO:tasks.rados.rados.2.smithi175.stdout:3678: done (2 left) 2024-09-11T03:51:49.129 INFO:tasks.rados.rados.2.smithi175.stdout:3679: done (1 left) 2024-09-11T03:51:49.129 INFO:tasks.rados.rados.2.smithi175.stdout:3682: delete oid 38 current snap is 411 2024-09-11T03:51:49.133 INFO:tasks.rados.rados.2.smithi175.stdout:3682: done (1 left) 2024-09-11T03:51:49.133 INFO:tasks.rados.rados.2.smithi175.stdout:3683: write (excl) oid 12 current snap is 411 2024-09-11T03:51:49.133 INFO:tasks.rados.rados.2.smithi175.stdout:3683: seq_num 1248 ranges {575136=592644,1663410=639100,2705542=269594} 2024-09-11T03:51:49.140 INFO:tasks.rados.rados.2.smithi175.stdout:3683: writing smithi17557870-12 from 575136 to 1167780 tid 1 2024-09-11T03:51:49.147 INFO:tasks.rados.rados.2.smithi175.stdout:3683: writing smithi17557870-12 from 1663410 to 2302510 tid 2 2024-09-11T03:51:49.151 INFO:tasks.rados.rados.2.smithi175.stdout:3683: writing smithi17557870-12 from 2705542 to 2975136 tid 3 2024-09-11T03:51:49.152 INFO:tasks.rados.rados.2.smithi175.stdout:3684: write (excl) oid 39 current snap is 411 2024-09-11T03:51:49.152 INFO:tasks.rados.rados.2.smithi175.stdout:3681: finishing copy_from to smithi17557870-22 2024-09-11T03:51:49.152 INFO:tasks.rados.rados.2.smithi175.stdout:3681: got expected ENOENT (src dne) 2024-09-11T03:51:49.152 INFO:tasks.rados.rados.2.smithi175.stdout:3684: seq_num 1249 ranges {591943=667893,1721892=782758} 2024-09-11T03:51:49.157 INFO:tasks.rados.rados.2.smithi175.stdout:3684: writing smithi17557870-39 from 591943 to 1259836 tid 1 2024-09-11T03:51:49.165 INFO:tasks.rados.rados.2.smithi175.stdout:3684: writing smithi17557870-39 from 1721892 to 2504650 tid 2 2024-09-11T03:51:49.166 INFO:tasks.rados.rados.2.smithi175.stdout:3685: copy_from oid 11 from oid 47 current snap is 411 2024-09-11T03:51:49.166 INFO:tasks.rados.rados.2.smithi175.stdout:3681: finishing copy_from racing read to smithi17557870-22 2024-09-11T03:51:49.166 INFO:tasks.rados.rados.2.smithi175.stdout:3683: finishing write tid 1 to smithi17557870-12 2024-09-11T03:51:49.166 INFO:tasks.rados.rados.2.smithi175.stdout:3683: finishing write tid 2 to smithi17557870-12 2024-09-11T03:51:49.166 INFO:tasks.rados.rados.2.smithi175.stdout:3683: finishing write tid 3 to smithi17557870-12 2024-09-11T03:51:49.166 INFO:tasks.rados.rados.2.smithi175.stdout:3683: finishing write tid 4 to smithi17557870-12 2024-09-11T03:51:49.166 INFO:tasks.rados.rados.2.smithi175.stdout:3683: finishing write tid 5 to smithi17557870-12 2024-09-11T03:51:49.166 INFO:tasks.rados.rados.2.smithi175.stdout:3683: oid 12 updating version 0 to 1499 2024-09-11T03:51:49.166 INFO:tasks.rados.rados.2.smithi175.stdout:3683: oid 12 version 1499 is already newer than 1498 2024-09-11T03:51:49.166 INFO:tasks.rados.rados.2.smithi175.stdout:3683: oid 12 version 1499 is already newer than 1497 2024-09-11T03:51:49.166 INFO:tasks.rados.rados.2.smithi175.stdout:3683: oid 12 updating version 1499 to 1500 2024-09-11T03:51:49.166 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 1500 (ObjNum 1248 snap 411 seq_num 1248) dirty exists 2024-09-11T03:51:49.166 INFO:tasks.rados.rados.2.smithi175.stdout:3683: left oid 12 (ObjNum 1248 snap 411 seq_num 1248) 2024-09-11T03:51:49.167 INFO:tasks.rados.rados.2.smithi175.stdout:3684: finishing write tid 1 to smithi17557870-39 2024-09-11T03:51:49.167 INFO:tasks.rados.rados.2.smithi175.stdout:3681: done (3 left) 2024-09-11T03:51:49.167 INFO:tasks.rados.rados.2.smithi175.stdout:3683: done (2 left) 2024-09-11T03:51:49.167 INFO:tasks.rados.rados.2.smithi175.stdout:3686: write (excl) oid 29 current snap is 411 2024-09-11T03:51:49.167 INFO:tasks.rados.rados.2.smithi175.stdout:3686: seq_num 1250 ranges {608750=743142,1780374=642769} 2024-09-11T03:51:49.172 INFO:tasks.rados.rados.2.smithi175.stdout:3686: writing smithi17557870-29 from 608750 to 1351892 tid 1 2024-09-11T03:51:49.175 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:49.179 INFO:tasks.rados.rados.2.smithi175.stdout:3686: writing smithi17557870-29 from 1780374 to 2423143 tid 2 2024-09-11T03:51:49.179 INFO:tasks.rados.rados.2.smithi175.stdout:3687: copy_from oid 35 from oid 38 current snap is 411 2024-09-11T03:51:49.179 INFO:tasks.rados.rados.2.smithi175.stdout:3684: finishing write tid 2 to smithi17557870-39 2024-09-11T03:51:49.179 INFO:tasks.rados.rados.2.smithi175.stdout:3684: finishing write tid 3 to smithi17557870-39 2024-09-11T03:51:49.179 INFO:tasks.rados.rados.2.smithi175.stdout:3684: finishing write tid 4 to smithi17557870-39 2024-09-11T03:51:49.180 INFO:tasks.rados.rados.2.smithi175.stdout:3684: oid 39 updating version 0 to 1017 2024-09-11T03:51:49.180 INFO:tasks.rados.rados.2.smithi175.stdout:3684: oid 39 version 1017 is already newer than 1015 2024-09-11T03:51:49.180 INFO:tasks.rados.rados.2.smithi175.stdout:3684: oid 39 version 1017 is already newer than 1016 2024-09-11T03:51:49.180 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 1017 (ObjNum 1249 snap 411 seq_num 1249) dirty exists 2024-09-11T03:51:49.180 INFO:tasks.rados.rados.2.smithi175.stdout:3684: left oid 39 (ObjNum 1249 snap 411 seq_num 1249) 2024-09-11T03:51:49.180 INFO:tasks.rados.rados.2.smithi175.stdout:3685: finishing copy_from racing read to smithi17557870-11 2024-09-11T03:51:49.180 INFO:tasks.rados.rados.2.smithi175.stdout:3684: done (3 left) 2024-09-11T03:51:49.180 INFO:tasks.rados.rados.2.smithi175.stdout:3688: snap_create 2024-09-11T03:51:49.180 INFO:tasks.rados.rados.2.smithi175.stdout:3686: finishing write tid 1 to smithi17557870-29 2024-09-11T03:51:49.180 INFO:tasks.rados.rados.2.smithi175.stdout:3687: finishing copy_from to smithi17557870-35 2024-09-11T03:51:49.180 INFO:tasks.rados.rados.2.smithi175.stdout:3687: got expected ENOENT (src dne) 2024-09-11T03:51:49.181 INFO:tasks.rados.rados.2.smithi175.stdout:3687: finishing copy_from racing read to smithi17557870-35 2024-09-11T03:51:49.181 INFO:tasks.rados.rados.2.smithi175.stdout:3685: finishing copy_from to smithi17557870-11 2024-09-11T03:51:49.181 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 1258 (ObjNum 1188 snap 388 seq_num 1188) dirty exists 2024-09-11T03:51:49.182 INFO:tasks.rados.rados.2.smithi175.stdout:3686: finishing write tid 2 to smithi17557870-29 2024-09-11T03:51:49.182 INFO:tasks.rados.rados.2.smithi175.stdout:3686: finishing write tid 3 to smithi17557870-29 2024-09-11T03:51:49.182 INFO:tasks.rados.rados.2.smithi175.stdout:3686: finishing write tid 4 to smithi17557870-29 2024-09-11T03:51:49.182 INFO:tasks.rados.rados.2.smithi175.stdout:3686: oid 29 updating version 0 to 1225 2024-09-11T03:51:49.182 INFO:tasks.rados.rados.2.smithi175.stdout:3686: oid 29 version 1225 is already newer than 1224 2024-09-11T03:51:49.182 INFO:tasks.rados.rados.2.smithi175.stdout:3686: oid 29 updating version 1225 to 1226 2024-09-11T03:51:49.182 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 1226 (ObjNum 1250 snap 411 seq_num 1250) dirty exists 2024-09-11T03:51:49.182 INFO:tasks.rados.rados.2.smithi175.stdout:3686: left oid 29 (ObjNum 1250 snap 411 seq_num 1250) 2024-09-11T03:51:49.197 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:49.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:49.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:49.264 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-11T03:51:49.299 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:49.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:49.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:49 smithi084 ceph-mon[27254]: 2.11 scrub starts 2024-09-11T03:51:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:49 smithi084 ceph-mon[27254]: 2.11 scrub ok 2024-09-11T03:51:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:49 smithi084 ceph-mon[27254]: osdmap e890: 12 total, 12 up, 11 in 2024-09-11T03:51:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1743301378' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:51:49.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4200975022' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:51:49.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:49 smithi121 ceph-mon[30729]: 2.11 scrub starts 2024-09-11T03:51:49.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:49 smithi121 ceph-mon[30729]: 2.11 scrub ok 2024-09-11T03:51:49.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:49 smithi121 ceph-mon[30729]: osdmap e890: 12 total, 12 up, 11 in 2024-09-11T03:51:49.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1743301378' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:51:49.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4200975022' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:51:49.427 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:49.449 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:49.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:49.450 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:49 smithi175 ceph-mon[30894]: 2.11 scrub starts 2024-09-11T03:51:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:49 smithi175 ceph-mon[30894]: 2.11 scrub ok 2024-09-11T03:51:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:49 smithi175 ceph-mon[30894]: osdmap e890: 12 total, 12 up, 11 in 2024-09-11T03:51:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1743301378' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:51:49.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4200975022' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:51:49.503 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:51:49.551 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:49.572 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:49.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:49.573 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:49.674 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:49.696 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:49.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:49.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:49.798 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:49.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:49.821 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:49.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:49.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:49.949 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:49.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:50.042 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:51:50.042 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:51:50.050 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:50.073 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:50.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:50.074 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:50.130 INFO:tasks.rados.rados.2.smithi175.stdout:3685: done (3 left) 2024-09-11T03:51:50.130 INFO:tasks.rados.rados.2.smithi175.stdout:3686: done (2 left) 2024-09-11T03:51:50.130 INFO:tasks.rados.rados.2.smithi175.stdout:3687: done (1 left) 2024-09-11T03:51:50.130 INFO:tasks.rados.rados.2.smithi175.stdout:3688: done (0 left) 2024-09-11T03:51:50.130 INFO:tasks.rados.rados.2.smithi175.stdout:3689: write (excl) oid 43 current snap is 412 2024-09-11T03:51:50.130 INFO:tasks.rados.rados.2.smithi175.stdout:3689: seq_num 1251 ranges {625557=418391,1838856=786427} 2024-09-11T03:51:50.136 INFO:tasks.rados.rados.2.smithi175.stdout:3689: writing smithi17557870-43 from 625557 to 1043948 tid 1 2024-09-11T03:51:50.144 INFO:tasks.rados.rados.2.smithi175.stdout:3689: writing smithi17557870-43 from 1838856 to 2625283 tid 2 2024-09-11T03:51:50.144 INFO:tasks.rados.rados.2.smithi175.stdout:3690: write oid 1 current snap is 412 2024-09-11T03:51:50.144 INFO:tasks.rados.rados.2.smithi175.stdout:3689: finishing write tid 1 to smithi17557870-43 2024-09-11T03:51:50.144 INFO:tasks.rados.rados.2.smithi175.stdout:3690: seq_num 1252 ranges {642364=493640,1897338=646438} 2024-09-11T03:51:50.149 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 9 1 2024-09-11T03:51:50.150 INFO:tasks.rados.rados.2.smithi175.stdout:3690: writing smithi17557870-1 from 642364 to 1136004 tid 1 2024-09-11T03:51:50.158 INFO:tasks.rados.rados.2.smithi175.stdout:3690: writing smithi17557870-1 from 1897338 to 2543776 tid 2 2024-09-11T03:51:50.158 INFO:tasks.rados.rados.2.smithi175.stdout:3691: delete oid 49 current snap is 412 2024-09-11T03:51:50.159 INFO:tasks.rados.rados.2.smithi175.stdout:3689: finishing write tid 2 to smithi17557870-43 2024-09-11T03:51:50.159 INFO:tasks.rados.rados.2.smithi175.stdout:3689: finishing write tid 3 to smithi17557870-43 2024-09-11T03:51:50.159 INFO:tasks.rados.rados.2.smithi175.stdout:3689: finishing write tid 4 to smithi17557870-43 2024-09-11T03:51:50.159 INFO:tasks.rados.rados.2.smithi175.stdout:3689: oid 43 updating version 0 to 1020 2024-09-11T03:51:50.159 INFO:tasks.rados.rados.2.smithi175.stdout:3689: oid 43 updating version 1020 to 1022 2024-09-11T03:51:50.159 INFO:tasks.rados.rados.2.smithi175.stdout:3689: oid 43 version 1022 is already newer than 1021 2024-09-11T03:51:50.159 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 1022 (ObjNum 1251 snap 412 seq_num 1251) dirty exists 2024-09-11T03:51:50.159 INFO:tasks.rados.rados.2.smithi175.stdout:3689: left oid 43 (ObjNum 1251 snap 412 seq_num 1251) 2024-09-11T03:51:50.159 INFO:tasks.rados.rados.2.smithi175.stdout:3690: finishing write tid 1 to smithi17557870-1 2024-09-11T03:51:50.162 INFO:tasks.rados.rados.2.smithi175.stdout:3690: finishing write tid 2 to smithi17557870-1 2024-09-11T03:51:50.162 INFO:tasks.rados.rados.2.smithi175.stdout:3689: done (2 left) 2024-09-11T03:51:50.162 INFO:tasks.rados.rados.2.smithi175.stdout:3691: done (1 left) 2024-09-11T03:51:50.162 INFO:tasks.rados.rados.2.smithi175.stdout:3692: read oid 17 snap -1 2024-09-11T03:51:50.162 INFO:tasks.rados.rados.2.smithi175.stdout:3692: expect (ObjNum 1181 snap 385 seq_num 1181) 2024-09-11T03:51:50.162 INFO:tasks.rados.rados.2.smithi175.stdout:3693: delete oid 29 current snap is 412 2024-09-11T03:51:50.163 INFO:tasks.rados.rados.2.smithi175.stdout:3690: finishing write tid 3 to smithi17557870-1 2024-09-11T03:51:50.163 INFO:tasks.rados.rados.2.smithi175.stdout:3690: finishing write tid 4 to smithi17557870-1 2024-09-11T03:51:50.163 INFO:tasks.rados.rados.2.smithi175.stdout:3690: oid 1 updating version 0 to 1245 2024-09-11T03:51:50.163 INFO:tasks.rados.rados.2.smithi175.stdout:3690: oid 1 version 1245 is already newer than 1243 2024-09-11T03:51:50.163 INFO:tasks.rados.rados.2.smithi175.stdout:3690: oid 1 version 1245 is already newer than 1244 2024-09-11T03:51:50.163 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 1245 (ObjNum 1252 snap 412 seq_num 1252) dirty exists 2024-09-11T03:51:50.163 INFO:tasks.rados.rados.2.smithi175.stdout:3690: left oid 1 (ObjNum 1252 snap 412 seq_num 1252) 2024-09-11T03:51:50.163 INFO:tasks.rados.rados.2.smithi175.stdout:3690: done (2 left) 2024-09-11T03:51:50.164 INFO:tasks.rados.rados.2.smithi175.stdout:3693: done (1 left) 2024-09-11T03:51:50.164 INFO:tasks.rados.rados.2.smithi175.stdout:3694: snap_remove snap 394 2024-09-11T03:51:50.175 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:50.196 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:50.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:50.197 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:50 smithi175 ceph-mon[30894]: pgmap v1491: 107 pgs: 2 active+clean+snaptrim, 105 active+clean; 715 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 7.7 MiB/s rd, 332 KiB/s wr, 6 op/s 2024-09-11T03:51:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:50 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:51:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:50 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:51:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:50 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1743301378' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:51:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:50 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4200975022' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:51:50.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:50 smithi175 ceph-mon[30894]: osdmap e891: 12 total, 12 up, 11 in 2024-09-11T03:51:50.297 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:50.318 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:50.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:50.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:50.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:50 smithi084 ceph-mon[27254]: pgmap v1491: 107 pgs: 2 active+clean+snaptrim, 105 active+clean; 715 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 7.7 MiB/s rd, 332 KiB/s wr, 6 op/s 2024-09-11T03:51:50.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:50 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:51:50.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:50 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:51:50.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:50 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1743301378' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:51:50.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:50 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4200975022' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:51:50.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:50 smithi084 ceph-mon[27254]: osdmap e891: 12 total, 12 up, 11 in 2024-09-11T03:51:50.346 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:51:50.347 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:51:50.365 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:51:50.365 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:51:50.384 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:51:50.385 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:51:50.402 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:50.403 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:51:50.403 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:51:50.420 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:50.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:50 smithi121 ceph-mon[30729]: pgmap v1491: 107 pgs: 2 active+clean+snaptrim, 105 active+clean; 715 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 7.7 MiB/s rd, 332 KiB/s wr, 6 op/s 2024-09-11T03:51:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:50 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:51:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:50 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:51:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:50 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1743301378' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:51:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:50 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4200975022' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:51:50.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:50 smithi121 ceph-mon[30729]: osdmap e891: 12 total, 12 up, 11 in 2024-09-11T03:51:50.430 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:51:50.430 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:51:50.441 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:50.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:50.442 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:50.447 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:51:50.447 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:51:50.465 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:51:50.465 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:51:50.481 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:51:50.481 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:51:50.498 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:51:50.498 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:51:50.515 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:51:50.516 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:51:50.532 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:51:50.532 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:51:50.543 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:50.550 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:51:50.550 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:51:50.564 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:50.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:50.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:50.666 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:50.688 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:50.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:50.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:50.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:50.816 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:50.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:50.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:50.833 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-11T03:51:50.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:50.939 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:50.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:50.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:51.041 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:51.067 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:51.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:51.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:51.073 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:51:51.143 INFO:tasks.rados.rados.2.smithi175.stdout:3694: done (1 left) 2024-09-11T03:51:51.143 INFO:tasks.rados.rados.2.smithi175.stdout:3695: read oid 28 snap 314 2024-09-11T03:51:51.143 INFO:tasks.rados.rados.2.smithi175.stdout:3695: expect (ObjNum 942 snap 297 seq_num 942) 2024-09-11T03:51:51.143 INFO:tasks.rados.rados.2.smithi175.stdout:3696: snap_remove snap 357 2024-09-11T03:51:51.169 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:51.190 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:51.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:51.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:51.292 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:51.318 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:51.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:51.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:51 smithi084 ceph-mon[27254]: 2.1 scrub starts 2024-09-11T03:51:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:51 smithi084 ceph-mon[27254]: 2.1 scrub ok 2024-09-11T03:51:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:51 smithi084 ceph-mon[27254]: osdmap e892: 12 total, 12 up, 11 in 2024-09-11T03:51:51.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3291103878' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:51.419 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:51 smithi121 ceph-mon[30729]: 2.1 scrub starts 2024-09-11T03:51:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:51 smithi121 ceph-mon[30729]: 2.1 scrub ok 2024-09-11T03:51:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:51 smithi121 ceph-mon[30729]: osdmap e892: 12 total, 12 up, 11 in 2024-09-11T03:51:51.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3291103878' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:51.440 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:51.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:51.476 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:51 smithi175 ceph-mon[30894]: 2.1 scrub starts 2024-09-11T03:51:51.476 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:51 smithi175 ceph-mon[30894]: 2.1 scrub ok 2024-09-11T03:51:51.476 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:51 smithi175 ceph-mon[30894]: osdmap e892: 12 total, 12 up, 11 in 2024-09-11T03:51:51.476 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3291103878' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:51.542 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:51.564 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:51.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:51.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:51.575 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:51:51.575 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:51:51.575 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:51:51.575 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:51:51.575 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:51:51.666 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:51.686 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:51.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:51.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:51.788 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:51.808 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:51.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:51.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:51.911 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:51.934 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:51.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:51.935 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:52.036 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:52.058 INFO:tasks.rados.rados.2.smithi175.stdout:3696: done (2 left) 2024-09-11T03:51:52.058 INFO:tasks.rados.rados.2.smithi175.stdout:3697: write (excl) oid 30 current snap is 412 2024-09-11T03:51:52.058 INFO:tasks.rados.rados.2.smithi175.stdout:3697: seq_num 1255 ranges {692785=435740,1672784=793765,2897503=195282} 2024-09-11T03:51:52.060 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:52.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:52.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:52.069 INFO:tasks.rados.rados.2.smithi175.stdout:3697: writing smithi17557870-30 from 692785 to 1128525 tid 1 2024-09-11T03:51:52.078 INFO:tasks.rados.rados.2.smithi175.stdout:3697: writing smithi17557870-30 from 1672784 to 2466549 tid 2 2024-09-11T03:51:52.082 INFO:tasks.rados.rados.2.smithi175.stdout:3697: writing smithi17557870-30 from 2897503 to 3092785 tid 3 2024-09-11T03:51:52.082 INFO:tasks.rados.rados.2.smithi175.stdout:3698: read oid 11 snap -1 2024-09-11T03:51:52.082 INFO:tasks.rados.rados.2.smithi175.stdout:3698: expect (ObjNum 1188 snap 388 seq_num 1188) 2024-09-11T03:51:52.082 INFO:tasks.rados.rados.2.smithi175.stdout:3699: read oid 15 snap 361 2024-09-11T03:51:52.082 INFO:tasks.rados.rados.2.smithi175.stdout:3699: expect (ObjNum 1120 snap 359 seq_num 1120) 2024-09-11T03:51:52.082 INFO:tasks.rados.rados.2.smithi175.stdout:3700: write oid 35 current snap is 412 2024-09-11T03:51:52.082 INFO:tasks.rados.rados.2.smithi175.stdout:3700: seq_num 1256 ranges {709592=510989,1731266=653776,3041279=68313} 2024-09-11T03:51:52.089 INFO:tasks.rados.rados.2.smithi175.stdout:3700: writing smithi17557870-35 from 709592 to 1220581 tid 1 2024-09-11T03:51:52.095 INFO:tasks.rados.rados.2.smithi175.stdout:3700: writing smithi17557870-35 from 1731266 to 2385042 tid 2 2024-09-11T03:51:52.100 INFO:tasks.rados.rados.2.smithi175.stdout:3700: writing smithi17557870-35 from 3041279 to 3109592 tid 3 2024-09-11T03:51:52.101 INFO:tasks.rados.rados.2.smithi175.stdout:3701: delete oid 18 current snap is 412 2024-09-11T03:51:52.101 INFO:tasks.rados.rados.2.smithi175.stdout:3692: expect (ObjNum 1181 snap 385 seq_num 1181) 2024-09-11T03:51:52.162 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:52.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:52.183 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:52.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:52.228 INFO:tasks.rados.rados.2.smithi175.stdout:3695: expect (ObjNum 942 snap 297 seq_num 942) 2024-09-11T03:51:52.284 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:52.299 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-11T03:51:52.306 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:52.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:52.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:52 smithi084 ceph-mon[27254]: pgmap v1494: 107 pgs: 2 active+clean+snaptrim, 105 active+clean; 715 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 4.3 MiB/s rd, 332 KiB/s wr, 4 op/s 2024-09-11T03:51:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:52 smithi084 ceph-mon[27254]: osdmap e893: 12 total, 12 up, 11 in 2024-09-11T03:51:52.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:52 smithi084 ceph-mon[27254]: osdmap e894: 12 total, 12 up, 11 in 2024-09-11T03:51:52.407 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:52.418 INFO:tasks.rados.rados.2.smithi175.stdout:3697: finishing write tid 1 to smithi17557870-30 2024-09-11T03:51:52.418 INFO:tasks.rados.rados.2.smithi175.stdout:3697: finishing write tid 2 to smithi17557870-30 2024-09-11T03:51:52.418 INFO:tasks.rados.rados.2.smithi175.stdout:3697: finishing write tid 3 to smithi17557870-30 2024-09-11T03:51:52.418 INFO:tasks.rados.rados.2.smithi175.stdout:3692: done (6 left) 2024-09-11T03:51:52.418 INFO:tasks.rados.rados.2.smithi175.stdout:3695: done (5 left) 2024-09-11T03:51:52.418 INFO:tasks.rados.rados.2.smithi175.stdout:3701: done (4 left) 2024-09-11T03:51:52.418 INFO:tasks.rados.rados.2.smithi175.stdout:3702: copy_from oid 4 from oid 43 current snap is 412 2024-09-11T03:51:52.418 INFO:tasks.rados.rados.2.smithi175.stdout:3697: finishing write tid 4 to smithi17557870-30 2024-09-11T03:51:52.418 INFO:tasks.rados.rados.2.smithi175.stdout:3697: finishing write tid 5 to smithi17557870-30 2024-09-11T03:51:52.418 INFO:tasks.rados.rados.2.smithi175.stdout:3697: oid 30 updating version 0 to 1185 2024-09-11T03:51:52.419 INFO:tasks.rados.rados.2.smithi175.stdout:3697: oid 30 updating version 1185 to 1186 2024-09-11T03:51:52.419 INFO:tasks.rados.rados.2.smithi175.stdout:3697: oid 30 version 1186 is already newer than 1183 2024-09-11T03:51:52.419 INFO:tasks.rados.rados.2.smithi175.stdout:3697: oid 30 version 1186 is already newer than 1184 2024-09-11T03:51:52.419 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 1186 (ObjNum 1255 snap 412 seq_num 1255) dirty exists 2024-09-11T03:51:52.419 INFO:tasks.rados.rados.2.smithi175.stdout:3697: left oid 30 (ObjNum 1255 snap 412 seq_num 1255) 2024-09-11T03:51:52.419 INFO:tasks.rados.rados.2.smithi175.stdout:3699: expect (ObjNum 1120 snap 359 seq_num 1120) 2024-09-11T03:51:52.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:52 smithi121 ceph-mon[30729]: pgmap v1494: 107 pgs: 2 active+clean+snaptrim, 105 active+clean; 715 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 4.3 MiB/s rd, 332 KiB/s wr, 4 op/s 2024-09-11T03:51:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:52 smithi121 ceph-mon[30729]: osdmap e893: 12 total, 12 up, 11 in 2024-09-11T03:51:52.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:52 smithi121 ceph-mon[30729]: osdmap e894: 12 total, 12 up, 11 in 2024-09-11T03:51:52.429 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:52.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:52.430 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:52 smithi175 ceph-mon[30894]: pgmap v1494: 107 pgs: 2 active+clean+snaptrim, 105 active+clean; 715 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 4.3 MiB/s rd, 332 KiB/s wr, 4 op/s 2024-09-11T03:51:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:52 smithi175 ceph-mon[30894]: osdmap e893: 12 total, 12 up, 11 in 2024-09-11T03:51:52.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:52 smithi175 ceph-mon[30894]: osdmap e894: 12 total, 12 up, 11 in 2024-09-11T03:51:52.512 INFO:tasks.rados.rados.2.smithi175.stdout:3698: expect (ObjNum 1188 snap 388 seq_num 1188) 2024-09-11T03:51:52.531 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:52.552 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:52.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:52.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:52.589 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:51:52.647 INFO:tasks.rados.rados.2.smithi175.stdout:3700: finishing write tid 1 to smithi17557870-35 2024-09-11T03:51:52.647 INFO:tasks.rados.rados.2.smithi175.stdout:3700: finishing write tid 2 to smithi17557870-35 2024-09-11T03:51:52.647 INFO:tasks.rados.rados.2.smithi175.stdout:3700: finishing write tid 3 to smithi17557870-35 2024-09-11T03:51:52.647 INFO:tasks.rados.rados.2.smithi175.stdout:3700: finishing write tid 4 to smithi17557870-35 2024-09-11T03:51:52.647 INFO:tasks.rados.rados.2.smithi175.stdout:3700: finishing write tid 5 to smithi17557870-35 2024-09-11T03:51:52.647 INFO:tasks.rados.rados.2.smithi175.stdout:3700: oid 35 updating version 0 to 1085 2024-09-11T03:51:52.647 INFO:tasks.rados.rados.2.smithi175.stdout:3700: oid 35 updating version 1085 to 1086 2024-09-11T03:51:52.648 INFO:tasks.rados.rados.2.smithi175.stdout:3700: oid 35 version 1086 is already newer than 1084 2024-09-11T03:51:52.648 INFO:tasks.rados.rados.2.smithi175.stdout:3700: oid 35 version 1086 is already newer than 1083 2024-09-11T03:51:52.648 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 1086 (ObjNum 1256 snap 412 seq_num 1256) dirty exists 2024-09-11T03:51:52.648 INFO:tasks.rados.rados.2.smithi175.stdout:3700: left oid 35 (ObjNum 1256 snap 412 seq_num 1256) 2024-09-11T03:51:52.648 INFO:tasks.rados.rados.2.smithi175.stdout:3697: done (4 left) 2024-09-11T03:51:52.648 INFO:tasks.rados.rados.2.smithi175.stdout:3698: done (3 left) 2024-09-11T03:51:52.648 INFO:tasks.rados.rados.2.smithi175.stdout:3699: done (2 left) 2024-09-11T03:51:52.648 INFO:tasks.rados.rados.2.smithi175.stdout:3700: done (1 left) 2024-09-11T03:51:52.648 INFO:tasks.rados.rados.2.smithi175.stdout:3703: rollback oid 34 current snap is 412 2024-09-11T03:51:52.648 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 34 to 399 2024-09-11T03:51:52.648 INFO:tasks.rados.rados.2.smithi175.stdout:3702: finishing copy_from racing read to smithi17557870-4 2024-09-11T03:51:52.648 INFO:tasks.rados.rados.2.smithi175.stdout:3702: finishing copy_from to smithi17557870-4 2024-09-11T03:51:52.649 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 1022 (ObjNum 1251 snap 412 seq_num 1251) dirty exists 2024-09-11T03:51:52.649 INFO:tasks.rados.rados.2.smithi175.stdout:3702: done (1 left) 2024-09-11T03:51:52.649 INFO:tasks.rados.rados.2.smithi175.stdout:3704: snap_remove snap 404 2024-09-11T03:51:52.654 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:52.675 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:52.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:52.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:52.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:52.803 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:52.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:52.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:52.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:52.931 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:52.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:52.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:53.033 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:53.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:53.055 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:53.055 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:53.076 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:51:53.077 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:51:53.077 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:51:53.077 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:51:53.077 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.077 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4687 2.18 2:1984a3a9:::smithi17557870-23:head [write 739673~657763,stat] snapc cd=[cd,cc,ca,c5,c2,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-11T03:51:53.077 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.003513+0000", 2024-09-11T03:51:53.077 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 392.07248638900001, 2024-09-11T03:51:53.077 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.028701775999999998, 2024-09-11T03:51:53.077 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.077 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.077 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:53.077 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:51:53.077 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:51:53.078 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:53.078 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4687 2024-09-11T03:51:53.078 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.078 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.078 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.078 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.078 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003513+0000", 2024-09-11T03:51:53.078 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.078 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.078 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.078 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.078 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003513+0000", 2024-09-11T03:51:53.079 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.079 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.079 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.079 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.079 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003515+0000", 2024-09-11T03:51:53.079 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5600000000000001e-06 2024-09-11T03:51:53.079 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.079 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.079 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.079 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003862+0000", 2024-09-11T03:51:53.079 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00034641300000000001 2024-09-11T03:51:53.079 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.079 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.079 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.080 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003880+0000", 2024-09-11T03:51:53.080 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8669e-05 2024-09-11T03:51:53.080 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.080 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.080 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.080 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003906+0000", 2024-09-11T03:51:53.080 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5432999999999999e-05 2024-09-11T03:51:53.080 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.080 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.080 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.080 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003972+0000", 2024-09-11T03:51:53.080 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6729000000000004e-05 2024-09-11T03:51:53.080 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.081 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.081 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:51:53.081 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.003982+0000", 2024-09-11T03:51:53.081 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2979999999999997e-06 2024-09-11T03:51:53.081 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.081 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.081 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.081 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.005521+0000", 2024-09-11T03:51:53.081 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015392959999999999 2024-09-11T03:51:53.081 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.081 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.081 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.081 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.005611+0000", 2024-09-11T03:51:53.081 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.0373999999999999e-05 2024-09-11T03:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,7", 2024-09-11T03:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.005729+0000", 2024-09-11T03:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000117834 2024-09-11T03:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.007175+0000", 2024-09-11T03:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014461140000000001 2024-09-11T03:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.008417+0000", 2024-09-11T03:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0012421120000000001 2024-09-11T03:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032091+0000", 2024-09-11T03:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.023674066000000001 2024-09-11T03:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032146+0000", 2024-09-11T03:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4786e-05 2024-09-11T03:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032214+0000", 2024-09-11T03:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8091999999999998e-05 2024-09-11T03:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4688 2.18 2:1984a3a9:::smithi17557870-23:head [write 2076229~663444] snapc cd=[cd,cc,ca,c5,c2,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-11T03:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.010935+0000", 2024-09-11T03:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 392.06506432899999, 2024-09-11T03:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021404982999999999, 2024-09-11T03:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4688 2024-09-11T03:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.010935+0000", 2024-09-11T03:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.010935+0000", 2024-09-11T03:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.010936+0000", 2024-09-11T03:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1689999999999999e-06 2024-09-11T03:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011065+0000", 2024-09-11T03:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000128997 2024-09-11T03:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011076+0000", 2024-09-11T03:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1561e-05 2024-09-11T03:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011082+0000", 2024-09-11T03:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2599999999999996e-06 2024-09-11T03:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011127+0000", 2024-09-11T03:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5509000000000001e-05 2024-09-11T03:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011212+0000", 2024-09-11T03:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5173999999999995e-05 2024-09-11T03:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,7", 2024-09-11T03:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011309+0000", 2024-09-11T03:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6980999999999999e-05 2024-09-11T03:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012804+0000", 2024-09-11T03:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001495019 2024-09-11T03:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.031586+0000", 2024-09-11T03:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018782099 2024-09-11T03:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032261+0000", 2024-09-11T03:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000674846 2024-09-11T03:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.090 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.090 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:51:53.090 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032297+0000", 2024-09-11T03:51:53.090 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6010000000000003e-05 2024-09-11T03:51:53.090 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.090 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.090 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.090 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032340+0000", 2024-09-11T03:51:53.090 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2357999999999997e-05 2024-09-11T03:51:53.090 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.090 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.090 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.090 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.091 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.091 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4689 2.18 2:1984a3a9:::smithi17557870-23:head [setxattr _header (58) in=7b,truncate 2739673] snapc cd=[cd,cc,ca,c5,c2,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-11T03:51:53.092 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.011093+0000", 2024-09-11T03:51:53.092 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 392.06490585300003, 2024-09-11T03:51:53.092 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.02140688, 2024-09-11T03:51:53.092 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.092 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.092 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:53.092 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:51:53.092 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:51:53.092 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:53.092 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4689 2024-09-11T03:51:53.092 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.092 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.093 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.093 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011093+0000", 2024-09-11T03:51:53.093 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.093 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.093 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011093+0000", 2024-09-11T03:51:53.093 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.093 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.093 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011094+0000", 2024-09-11T03:51:53.094 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9999999999999997e-07 2024-09-11T03:51:53.094 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.094 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.094 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.094 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011100+0000", 2024-09-11T03:51:53.094 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1369999999999999e-06 2024-09-11T03:51:53.094 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.094 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.094 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.094 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011106+0000", 2024-09-11T03:51:53.094 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.421e-06 2024-09-11T03:51:53.094 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.094 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.095 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.095 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011117+0000", 2024-09-11T03:51:53.095 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1012999999999999e-05 2024-09-11T03:51:53.095 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.095 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.095 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.095 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012151+0000", 2024-09-11T03:51:53.095 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0010333880000000001 2024-09-11T03:51:53.095 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.095 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.095 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.095 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012265+0000", 2024-09-11T03:51:53.095 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114141 2024-09-11T03:51:53.095 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.096 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.096 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,7", 2024-09-11T03:51:53.096 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012418+0000", 2024-09-11T03:51:53.096 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015286499999999999 2024-09-11T03:51:53.096 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.096 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.096 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:51:53.096 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012807+0000", 2024-09-11T03:51:53.096 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00038979800000000001 2024-09-11T03:51:53.096 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.096 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.096 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.097 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.031717+0000", 2024-09-11T03:51:53.097 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018909888 2024-09-11T03:51:53.097 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.097 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.097 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.097 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032383+0000", 2024-09-11T03:51:53.097 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00066515900000000002 2024-09-11T03:51:53.097 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.097 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.097 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:51:53.097 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032447+0000", 2024-09-11T03:51:53.097 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4958999999999999e-05 2024-09-11T03:51:53.097 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.097 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.098 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.098 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032500+0000", 2024-09-11T03:51:53.098 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2510999999999999e-05 2024-09-11T03:51:53.098 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.098 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.098 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.098 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.098 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.098 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4690 2.18 2:1984a3a9:::smithi17557870-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e503)", 2024-09-11T03:51:53.098 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.011126+0000", 2024-09-11T03:51:53.098 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 392.06487262500002, 2024-09-11T03:51:53.098 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021513529, 2024-09-11T03:51:53.098 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.099 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.099 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:51:53.099 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:51:53.099 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:51:53.099 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:53.099 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4690 2024-09-11T03:51:53.099 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.099 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.099 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.099 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.099 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011126+0000", 2024-09-11T03:51:53.099 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.100 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.100 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.100 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.100 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011126+0000", 2024-09-11T03:51:53.100 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.100 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.100 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.100 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.100 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011127+0000", 2024-09-11T03:51:53.100 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8800000000000003e-07 2024-09-11T03:51:53.100 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.100 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.100 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.101 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011129+0000", 2024-09-11T03:51:53.101 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0090000000000002e-06 2024-09-11T03:51:53.101 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.101 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.101 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.101 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011146+0000", 2024-09-11T03:51:53.101 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7209000000000001e-05 2024-09-11T03:51:53.101 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.101 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.101 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.101 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.011152+0000", 2024-09-11T03:51:53.101 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5160000000000002e-06 2024-09-11T03:51:53.101 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.102 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.102 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.102 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012718+0000", 2024-09-11T03:51:53.102 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001566413 2024-09-11T03:51:53.102 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.102 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.102 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:51:53.102 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.012793+0000", 2024-09-11T03:51:53.102 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5197999999999994e-05 2024-09-11T03:51:53.102 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.102 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.102 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.102 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032530+0000", 2024-09-11T03:51:53.102 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019736848000000001 2024-09-11T03:51:53.103 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.103 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.103 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.103 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032577+0000", 2024-09-11T03:51:53.103 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6878000000000001e-05 2024-09-11T03:51:53.103 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.103 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.103 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.103 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.032640+0000", 2024-09-11T03:51:53.103 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2970000000000002e-05 2024-09-11T03:51:53.103 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.103 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.103 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.104 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.104 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.104 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5691 2.e 2:77dadb97:::smithi17557870-30:head [write 2119636~654941] snapc fa=[] ondisk+write+known_if_redirected+supports_pool_eio e601)", 2024-09-11T03:51:53.104 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:59.009165+0000", 2024-09-11T03:51:53.104 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 294.066833659, 2024-09-11T03:51:53.104 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.052209984000000001, 2024-09-11T03:51:53.104 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.104 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.104 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:53.104 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:51:53.104 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:51:53.104 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:53.104 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5691 2024-09-11T03:51:53.104 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.105 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.105 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.105 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.105 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009165+0000", 2024-09-11T03:51:53.105 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.105 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.105 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.105 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.105 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009165+0000", 2024-09-11T03:51:53.105 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.105 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.105 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.105 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.106 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009166+0000", 2024-09-11T03:51:53.106 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.204e-06 2024-09-11T03:51:53.106 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.106 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.106 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.106 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009306+0000", 2024-09-11T03:51:53.106 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000139426 2024-09-11T03:51:53.106 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.106 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.106 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.106 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009318+0000", 2024-09-11T03:51:53.106 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2329e-05 2024-09-11T03:51:53.106 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.107 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.107 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.107 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009324+0000", 2024-09-11T03:51:53.107 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9000000000000003e-06 2024-09-11T03:51:53.107 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.107 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.107 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.107 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009379+0000", 2024-09-11T03:51:53.107 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5285999999999999e-05 2024-09-11T03:51:53.107 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.107 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.107 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.107 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009484+0000", 2024-09-11T03:51:53.107 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010491400000000001 2024-09-11T03:51:53.108 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.108 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.108 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:51:53.108 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009651+0000", 2024-09-11T03:51:53.108 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016665500000000001 2024-09-11T03:51:53.108 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.108 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.108 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:51:53.108 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.011223+0000", 2024-09-11T03:51:53.108 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001572452 2024-09-11T03:51:53.108 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.108 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.109 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.109 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.021352+0000", 2024-09-11T03:51:53.109 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010128587999999999 2024-09-11T03:51:53.109 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.109 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.109 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.109 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061260+0000", 2024-09-11T03:51:53.109 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.039907610000000003 2024-09-11T03:51:53.109 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.109 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.109 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:51:53.109 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061325+0000", 2024-09-11T03:51:53.109 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.58e-05 2024-09-11T03:51:53.109 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.110 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.110 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.110 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061375+0000", 2024-09-11T03:51:53.110 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9820000000000001e-05 2024-09-11T03:51:53.110 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.110 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.110 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.110 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.110 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.110 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5692 2.e 2:77dadb97:::smithi17557870-30:head [setxattr _header (58) in=7b,truncate 3159881] snapc fa=[] ondisk+write+known_if_redirected+supports_pool_eio e601)", 2024-09-11T03:51:53.110 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:59.009337+0000", 2024-09-11T03:51:53.110 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 294.06666149099999, 2024-09-11T03:51:53.110 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.052168173999999998, 2024-09-11T03:51:53.110 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.111 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.111 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:53.111 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:51:53.111 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:51:53.111 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:53.111 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5692 2024-09-11T03:51:53.111 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.111 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.111 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.111 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.111 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009337+0000", 2024-09-11T03:51:53.111 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.111 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.112 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.112 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.112 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009337+0000", 2024-09-11T03:51:53.112 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.112 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.112 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.112 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.112 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009338+0000", 2024-09-11T03:51:53.112 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0900000000000002e-07 2024-09-11T03:51:53.112 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.112 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.112 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.112 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009340+0000", 2024-09-11T03:51:53.113 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2120000000000002e-06 2024-09-11T03:51:53.113 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.113 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.113 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.113 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009345+0000", 2024-09-11T03:51:53.113 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5079999999999999e-06 2024-09-11T03:51:53.113 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.113 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.113 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.113 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009360+0000", 2024-09-11T03:51:53.113 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5191000000000001e-05 2024-09-11T03:51:53.113 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.113 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.114 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.114 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.010683+0000", 2024-09-11T03:51:53.114 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013236929999999999 2024-09-11T03:51:53.114 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.114 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.114 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.114 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.010767+0000", 2024-09-11T03:51:53.114 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3662e-05 2024-09-11T03:51:53.114 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.114 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.114 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:51:53.114 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.010905+0000", 2024-09-11T03:51:53.114 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013769100000000001 2024-09-11T03:51:53.115 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.115 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.115 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:51:53.115 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.011384+0000", 2024-09-11T03:51:53.115 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00047889999999999999 2024-09-11T03:51:53.115 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.115 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.115 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.115 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.021405+0000", 2024-09-11T03:51:53.115 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010021486 2024-09-11T03:51:53.115 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.115 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.115 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.115 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061414+0000", 2024-09-11T03:51:53.116 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.040009265000000002 2024-09-11T03:51:53.116 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.116 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.116 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:51:53.116 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061467+0000", 2024-09-11T03:51:53.116 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2589000000000001e-05 2024-09-11T03:51:53.116 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.116 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.116 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.116 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061506+0000", 2024-09-11T03:51:53.116 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8568000000000001e-05 2024-09-11T03:51:53.116 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.116 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.117 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.117 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.117 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.117 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5693 2.e 2:77dadb97:::smithi17557870-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e601)", 2024-09-11T03:51:53.117 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:59.009365+0000", 2024-09-11T03:51:53.117 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 294.06663373499998, 2024-09-11T03:51:53.117 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.052354291999999997, 2024-09-11T03:51:53.117 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.117 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.117 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:51:53.117 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:51:53.117 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:51:53.117 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:53.117 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5693 2024-09-11T03:51:53.118 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.118 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.118 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.118 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.118 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009365+0000", 2024-09-11T03:51:53.118 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.118 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.118 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.118 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.118 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009365+0000", 2024-09-11T03:51:53.118 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.118 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.118 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.118 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.119 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009366+0000", 2024-09-11T03:51:53.119 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0600000000000001e-07 2024-09-11T03:51:53.119 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.119 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.119 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.119 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009367+0000", 2024-09-11T03:51:53.119 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5990000000000001e-06 2024-09-11T03:51:53.119 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.119 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.119 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.119 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009371+0000", 2024-09-11T03:51:53.119 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.1200000000000004e-06 2024-09-11T03:51:53.119 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.120 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.120 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.120 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.009374+0000", 2024-09-11T03:51:53.120 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0400000000000001e-06 2024-09-11T03:51:53.120 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.120 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.120 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.120 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.011251+0000", 2024-09-11T03:51:53.120 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001877058 2024-09-11T03:51:53.120 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.120 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.120 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for rw locks", 2024-09-11T03:51:53.120 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.011330+0000", 2024-09-11T03:51:53.121 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8325000000000006e-05 2024-09-11T03:51:53.121 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.121 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.121 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.121 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061544+0000", 2024-09-11T03:51:53.121 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.050214591000000003 2024-09-11T03:51:53.121 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.121 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.121 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.121 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061626+0000", 2024-09-11T03:51:53.121 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1976000000000003e-05 2024-09-11T03:51:53.121 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.121 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.122 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.122 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:59.061720+0000", 2024-09-11T03:51:53.122 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3276999999999995e-05 2024-09-11T03:51:53.122 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.122 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.122 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.122 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.122 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.122 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5885 2.e 2:77dadb97:::smithi17557870-30:head [copy-from ver 882] snapc 101=[101,100] ondisk+write+known_if_redirected+supports_pool_eio e613)", 2024-09-11T03:51:53.122 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:11.238029+0000", 2024-09-11T03:51:53.122 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 281.83796978700002, 2024-09-11T03:51:53.122 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.02012736, 2024-09-11T03:51:53.122 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.122 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.123 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:53.123 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:51:53.123 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:51:53.123 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:53.123 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5885 2024-09-11T03:51:53.123 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.123 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.123 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.123 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.123 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.238029+0000", 2024-09-11T03:51:53.123 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.123 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.123 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.123 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.124 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.238029+0000", 2024-09-11T03:51:53.124 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.124 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.124 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.124 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.124 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.238032+0000", 2024-09-11T03:51:53.124 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.401e-06 2024-09-11T03:51:53.124 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.124 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.124 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.124 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.238042+0000", 2024-09-11T03:51:53.124 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0108e-05 2024-09-11T03:51:53.124 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.125 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.125 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.125 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.238064+0000", 2024-09-11T03:51:53.125 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2651999999999999e-05 2024-09-11T03:51:53.125 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.125 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.125 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.125 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.238074+0000", 2024-09-11T03:51:53.125 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3079999999999997e-06 2024-09-11T03:51:53.125 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.125 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.125 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.125 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.238165+0000", 2024-09-11T03:51:53.126 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1645999999999996e-05 2024-09-11T03:51:53.126 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.126 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.126 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:51:53.126 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.238173+0000", 2024-09-11T03:51:53.126 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9400000000000002e-06 2024-09-11T03:51:53.126 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.126 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.126 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.126 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.239797+0000", 2024-09-11T03:51:53.126 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0016233650000000001 2024-09-11T03:51:53.126 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.126 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.127 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.127 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.239860+0000", 2024-09-11T03:51:53.127 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3040999999999995e-05 2024-09-11T03:51:53.127 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.127 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.127 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:51:53.127 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.245678+0000", 2024-09-11T03:51:53.127 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0058182770000000002 2024-09-11T03:51:53.127 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.127 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.127 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:51:53.127 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.250568+0000", 2024-09-11T03:51:53.127 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0048903560000000002 2024-09-11T03:51:53.128 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.128 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.128 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.128 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.256230+0000", 2024-09-11T03:51:53.128 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0056613940000000001 2024-09-11T03:51:53.128 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.128 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.128 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.128 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.258076+0000", 2024-09-11T03:51:53.128 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001846618 2024-09-11T03:51:53.128 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.128 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.128 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:51:53.129 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.258118+0000", 2024-09-11T03:51:53.129 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.1913999999999999e-05 2024-09-11T03:51:53.129 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.129 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.129 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.129 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:11.258157+0000", 2024-09-11T03:51:53.129 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8340000000000002e-05 2024-09-11T03:51:53.129 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.129 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.129 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.129 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.129 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.129 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6152 2.e 2:77dadb97:::smithi17557870-30:head [copy-from ver 268] snapc 10c=[] ondisk+write+known_if_redirected+supports_pool_eio e637)", 2024-09-11T03:51:53.130 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:35.707196+0000", 2024-09-11T03:51:53.130 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 257.36880326800002, 2024-09-11T03:51:53.130 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019734676, 2024-09-11T03:51:53.130 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.130 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.130 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:53.130 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:51:53.130 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:51:53.130 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:53.130 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6152 2024-09-11T03:51:53.130 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.130 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.130 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.131 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.131 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.707196+0000", 2024-09-11T03:51:53.131 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.131 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.131 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.131 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.131 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.707196+0000", 2024-09-11T03:51:53.131 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.131 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.131 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.131 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.131 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.707198+0000", 2024-09-11T03:51:53.131 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.125e-06 2024-09-11T03:51:53.131 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.132 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.132 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.132 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.707207+0000", 2024-09-11T03:51:53.132 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.0839999999999998e-06 2024-09-11T03:51:53.132 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.132 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.132 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.132 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.707225+0000", 2024-09-11T03:51:53.132 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8060999999999999e-05 2024-09-11T03:51:53.132 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.132 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.132 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.133 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.707233+0000", 2024-09-11T03:51:53.133 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.2409999999999995e-06 2024-09-11T03:51:53.133 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.133 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.133 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.133 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.707297+0000", 2024-09-11T03:51:53.133 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3503999999999994e-05 2024-09-11T03:51:53.133 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.133 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.133 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.133 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.707366+0000", 2024-09-11T03:51:53.133 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9663e-05 2024-09-11T03:51:53.133 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.133 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.134 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:51:53.134 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.714472+0000", 2024-09-11T03:51:53.134 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.007106022 2024-09-11T03:51:53.134 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.134 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.134 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:51:53.134 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.718742+0000", 2024-09-11T03:51:53.134 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.004269424 2024-09-11T03:51:53.134 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.134 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.134 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.134 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.726773+0000", 2024-09-11T03:51:53.134 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0080315779999999993 2024-09-11T03:51:53.135 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.135 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.135 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.135 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.726818+0000", 2024-09-11T03:51:53.135 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4274000000000003e-05 2024-09-11T03:51:53.135 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.135 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.135 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:51:53.135 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.726881+0000", 2024-09-11T03:51:53.135 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2889e-05 2024-09-11T03:51:53.135 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.135 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.135 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.136 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:35.726930+0000", 2024-09-11T03:51:53.136 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9811000000000001e-05 2024-09-11T03:51:53.136 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.136 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.136 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.136 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.136 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.136 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6611 2.18 2:1984a3a9:::smithi17557870-23:head [copy-from ver 544] snapc 124=[124,123,121,120,11f,11e,11d,11a,119,114] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-09-11T03:51:53.136 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:12.994229+0000", 2024-09-11T03:51:53.136 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 220.081769509, 2024-09-11T03:51:53.136 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021949901000000001, 2024-09-11T03:51:53.136 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.136 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.137 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:53.137 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:51:53.137 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:51:53.137 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:53.137 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6611 2024-09-11T03:51:53.137 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.137 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.137 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.137 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.137 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.994229+0000", 2024-09-11T03:51:53.137 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.137 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.137 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.138 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.138 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.994229+0000", 2024-09-11T03:51:53.138 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.138 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.138 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.138 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.138 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.994231+0000", 2024-09-11T03:51:53.138 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9269999999999999e-06 2024-09-11T03:51:53.138 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.138 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.138 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.138 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.994240+0000", 2024-09-11T03:51:53.138 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.9309999999999997e-06 2024-09-11T03:51:53.139 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.139 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.139 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.139 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.994262+0000", 2024-09-11T03:51:53.139 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1341000000000001e-05 2024-09-11T03:51:53.139 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.139 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.139 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.139 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.994277+0000", 2024-09-11T03:51:53.139 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5407e-05 2024-09-11T03:51:53.139 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.139 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.139 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.139 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.994347+0000", 2024-09-11T03:51:53.140 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9555999999999996e-05 2024-09-11T03:51:53.140 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.140 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.140 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:51:53.140 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.994355+0000", 2024-09-11T03:51:53.140 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9889999999999992e-06 2024-09-11T03:51:53.140 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.140 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.140 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.140 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.996025+0000", 2024-09-11T03:51:53.140 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0016699320000000001 2024-09-11T03:51:53.140 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.140 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.141 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.141 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:12.996110+0000", 2024-09-11T03:51:53.141 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5711999999999995e-05 2024-09-11T03:51:53.141 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.141 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.141 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 4,7", 2024-09-11T03:51:53.141 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:13.005397+0000", 2024-09-11T03:51:53.141 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.009286908 2024-09-11T03:51:53.141 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.141 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.141 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:51:53.141 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:13.009436+0000", 2024-09-11T03:51:53.141 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0040392450000000003 2024-09-11T03:51:53.142 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.142 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.142 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.142 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:13.014053+0000", 2024-09-11T03:51:53.142 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00461678 2024-09-11T03:51:53.142 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.142 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.142 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.142 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:13.016093+0000", 2024-09-11T03:51:53.142 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0020393799999999999 2024-09-11T03:51:53.142 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.142 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.142 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:51:53.143 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:13.016133+0000", 2024-09-11T03:51:53.143 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0123999999999997e-05 2024-09-11T03:51:53.143 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.143 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.143 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.143 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:13.016179+0000", 2024-09-11T03:51:53.143 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6669000000000003e-05 2024-09-11T03:51:53.143 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.143 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.143 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.143 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.143 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.143 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7117 2.e 2:750de3b7:::smithi17557870-8:head [copy-from ver 984] snapc 13b=[13b,13a,139,138,137,136,135,133,12e,12d,12c,12a,127,126,120,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e713)", 2024-09-11T03:51:53.144 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:51.194048+0000", 2024-09-11T03:51:53.144 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 181.88195061600001, 2024-09-11T03:51:53.144 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.029233823999999999, 2024-09-11T03:51:53.144 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.144 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.144 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:53.144 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:51:53.144 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:51:53.144 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:53.144 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7117 2024-09-11T03:51:53.144 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.144 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.144 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.145 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.145 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194048+0000", 2024-09-11T03:51:53.145 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.145 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.145 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.145 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.145 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194048+0000", 2024-09-11T03:51:53.145 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.145 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.145 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.145 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.145 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194051+0000", 2024-09-11T03:51:53.145 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2390000000000001e-06 2024-09-11T03:51:53.145 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.146 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.146 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.146 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194060+0000", 2024-09-11T03:51:53.146 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9280000000000005e-06 2024-09-11T03:51:53.146 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.146 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.146 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.146 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194092+0000", 2024-09-11T03:51:53.146 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1603999999999998e-05 2024-09-11T03:51:53.146 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.146 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.146 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.146 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194102+0000", 2024-09-11T03:51:53.146 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0285999999999999e-05 2024-09-11T03:51:53.147 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.147 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.147 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.147 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194180+0000", 2024-09-11T03:51:53.147 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7231999999999995e-05 2024-09-11T03:51:53.147 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.147 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.147 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.147 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194274+0000", 2024-09-11T03:51:53.147 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4704000000000007e-05 2024-09-11T03:51:53.147 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.147 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.147 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:51:53.147 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.211121+0000", 2024-09-11T03:51:53.148 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016847141 2024-09-11T03:51:53.148 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.148 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.148 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:51:53.148 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.214395+0000", 2024-09-11T03:51:53.148 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032732429999999999 2024-09-11T03:51:53.148 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.148 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.148 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.148 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223020+0000", 2024-09-11T03:51:53.148 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0086250600000000004 2024-09-11T03:51:53.148 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.148 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.148 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.148 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223211+0000", 2024-09-11T03:51:53.149 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000191349 2024-09-11T03:51:53.149 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.149 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.149 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:51:53.149 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223250+0000", 2024-09-11T03:51:53.149 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9366999999999998e-05 2024-09-11T03:51:53.149 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.149 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.150 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.9 to 1 (10000) 2024-09-11T03:51:53.150 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.151 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223282+0000", 2024-09-11T03:51:53.151 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1671000000000002e-05 2024-09-11T03:51:53.151 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.151 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.151 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.151 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.151 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.151 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7118 2.e 2:750de3b7:::smithi17557870-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e713)", 2024-09-11T03:51:53.151 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:51.194365+0000", 2024-09-11T03:51:53.151 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 181.88163421199999, 2024-09-11T03:51:53.151 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019680618, 2024-09-11T03:51:53.151 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.152 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.152 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:51:53.152 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:51:53.152 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:51:53.152 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:53.152 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7118 2024-09-11T03:51:53.152 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.152 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.152 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.152 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.152 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194365+0000", 2024-09-11T03:51:53.152 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.153 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.153 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.153 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.153 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194365+0000", 2024-09-11T03:51:53.153 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.153 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.153 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.153 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.153 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194365+0000", 2024-09-11T03:51:53.153 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3799999999999997e-07 2024-09-11T03:51:53.153 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.153 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.153 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.153 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194369+0000", 2024-09-11T03:51:53.154 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0430000000000001e-06 2024-09-11T03:51:53.154 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.154 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.154 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.154 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194379+0000", 2024-09-11T03:51:53.154 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.268e-06 2024-09-11T03:51:53.154 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.154 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.154 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.154 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194386+0000", 2024-09-11T03:51:53.155 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3420000000000002e-06 2024-09-11T03:51:53.155 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.155 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.155 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.155 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194423+0000", 2024-09-11T03:51:53.155 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7125000000000001e-05 2024-09-11T03:51:53.155 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.155 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.155 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:51:53.155 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194486+0000", 2024-09-11T03:51:53.155 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2824999999999996e-05 2024-09-11T03:51:53.155 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.155 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.156 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.156 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.213801+0000", 2024-09-11T03:51:53.156 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019315300000000001 2024-09-11T03:51:53.156 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.156 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.156 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.156 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.213904+0000", 2024-09-11T03:51:53.156 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000102876 2024-09-11T03:51:53.156 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.156 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.156 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.156 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.214045+0000", 2024-09-11T03:51:53.156 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00014120100000000001 2024-09-11T03:51:53.157 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.157 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.157 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.157 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.157 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.157 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7209 2.e 2:77dadb97:::smithi17557870-30:head [copy-from ver 962] snapc 140=[140,13f,13d,13c,139,138,137,136,135,133,12e,12d,12a,127,126,120,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-09-11T03:51:53.157 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:49:00.012732+0000", 2024-09-11T03:51:53.157 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 173.063267349, 2024-09-11T03:51:53.157 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019941586000000001, 2024-09-11T03:51:53.157 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.157 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.157 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:53.157 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:51:53.158 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:51:53.158 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:51:53.158 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7209 2024-09-11T03:51:53.158 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.158 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.158 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.158 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.158 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.012732+0000", 2024-09-11T03:51:53.158 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.158 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.158 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.158 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.158 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.012732+0000", 2024-09-11T03:51:53.159 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.159 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.159 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.159 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.159 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.012734+0000", 2024-09-11T03:51:53.159 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2630000000000002e-06 2024-09-11T03:51:53.159 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.159 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.159 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.159 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.012741+0000", 2024-09-11T03:51:53.159 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7209999999999997e-06 2024-09-11T03:51:53.159 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.159 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.159 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.160 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.012756+0000", 2024-09-11T03:51:53.160 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5099000000000001e-05 2024-09-11T03:51:53.160 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.160 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.160 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.160 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.012764+0000", 2024-09-11T03:51:53.160 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4600000000000003e-06 2024-09-11T03:51:53.160 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.160 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.160 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.160 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.012826+0000", 2024-09-11T03:51:53.160 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1569999999999995e-05 2024-09-11T03:51:53.160 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.161 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.161 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:51:53.161 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.012833+0000", 2024-09-11T03:51:53.161 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1840000000000002e-06 2024-09-11T03:51:53.161 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.161 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.161 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.161 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.015099+0000", 2024-09-11T03:51:53.161 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022662469999999999 2024-09-11T03:51:53.161 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.161 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.162 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.162 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.015212+0000", 2024-09-11T03:51:53.162 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000113232 2024-09-11T03:51:53.162 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.162 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.162 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:51:53.162 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.024487+0000", 2024-09-11T03:51:53.162 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0092743140000000005 2024-09-11T03:51:53.162 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.162 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.162 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:51:53.162 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.028401+0000", 2024-09-11T03:51:53.162 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039138669999999997 2024-09-11T03:51:53.163 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.163 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.163 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.163 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.032461+0000", 2024-09-11T03:51:53.163 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0040599980000000004 2024-09-11T03:51:53.163 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.163 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.163 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:51:53.163 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.032598+0000", 2024-09-11T03:51:53.163 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000137904 2024-09-11T03:51:53.163 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.163 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.163 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:51:53.163 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.032639+0000", 2024-09-11T03:51:53.164 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.1026999999999999e-05 2024-09-11T03:51:53.164 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.164 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.164 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.164 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:00.032673+0000", 2024-09-11T03:51:53.164 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3699999999999999e-05 2024-09-11T03:51:53.164 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.164 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.164 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.164 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.164 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.164 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:8853 2.47 e856/854 2:e23d1762:::smithi17557870-21:head v 856'1264, mlcod=848'1260)", 2024-09-11T03:51:53.164 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:51:14.154811+0000", 2024-09-11T03:51:53.164 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 38.921187734999997, 2024-09-11T03:51:53.165 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021603786, 2024-09-11T03:51:53.165 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.165 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.165 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:53.165 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.165 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.165 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.165 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:14.154811+0000", 2024-09-11T03:51:53.165 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.165 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.165 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.165 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.165 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:14.154811+0000", 2024-09-11T03:51:53.166 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.166 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.166 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.166 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.166 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:14.154811+0000", 2024-09-11T03:51:53.166 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8400000000000001e-07 2024-09-11T03:51:53.166 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.166 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.166 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.166 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:14.175107+0000", 2024-09-11T03:51:53.166 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020295357 2024-09-11T03:51:53.166 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.167 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.167 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.167 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:14.175125+0000", 2024-09-11T03:51:53.167 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8164000000000001e-05 2024-09-11T03:51:53.167 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.167 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.167 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.167 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:14.175132+0000", 2024-09-11T03:51:53.167 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8199999999999999e-06 2024-09-11T03:51:53.167 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.167 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.167 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.167 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:14.175202+0000", 2024-09-11T03:51:53.167 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0296000000000006e-05 2024-09-11T03:51:53.168 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.168 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.168 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.168 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:14.175228+0000", 2024-09-11T03:51:53.168 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6336999999999999e-05 2024-09-11T03:51:53.168 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.168 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.168 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:51:53.168 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:14.176344+0000", 2024-09-11T03:51:53.168 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00111596 2024-09-11T03:51:53.168 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.168 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.168 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.168 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:14.176415+0000", 2024-09-11T03:51:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0667999999999995e-05 2024-09-11T03:51:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(osd.7.0:625 2.43 e885/854 2:c3768452:::smithi17557870-11:181 v 885'594, mlcod=881'590)", 2024-09-11T03:51:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:51:43.069537+0000", 2024-09-11T03:51:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 10.006461658999999, 2024-09-11T03:51:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035578319999999999, 2024-09-11T03:51:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:43.069537+0000", 2024-09-11T03:51:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:43.069537+0000", 2024-09-11T03:51:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:43.069538+0000", 2024-09-11T03:51:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3099999999999998e-07 2024-09-11T03:51:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:43.069555+0000", 2024-09-11T03:51:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7275e-05 2024-09-11T03:51:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:43.069647+0000", 2024-09-11T03:51:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1649000000000003e-05 2024-09-11T03:51:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:43.069657+0000", 2024-09-11T03:51:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0644e-05 2024-09-11T03:51:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:43.070978+0000", 2024-09-11T03:51:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013203970000000001 2024-09-11T03:51:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:43.070998+0000", 2024-09-11T03:51:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0438e-05 2024-09-11T03:51:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:51:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:43.073057+0000", 2024-09-11T03:51:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0020588300000000002 2024-09-11T03:51:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:43.073095+0000", 2024-09-11T03:51:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8068000000000002e-05 2024-09-11T03:51:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(osd.6.0:798 2.24 e886/750 2:25f5f8b9:::smithi17557870-26:13a v 886'712, mlcod=882'708)", 2024-09-11T03:51:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:51:44.061943+0000", 2024-09-11T03:51:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 9.0140560139999995, 2024-09-11T03:51:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0071756959999999996, 2024-09-11T03:51:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.061943+0000", 2024-09-11T03:51:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.061943+0000", 2024-09-11T03:51:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.061943+0000", 2024-09-11T03:51:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.32e-07 2024-09-11T03:51:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.061949+0000", 2024-09-11T03:51:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.925e-06 2024-09-11T03:51:53.176 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.176 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.176 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.176 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.061976+0000", 2024-09-11T03:51:53.176 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7149999999999999e-05 2024-09-11T03:51:53.176 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.176 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.176 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.176 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.061987+0000", 2024-09-11T03:51:53.176 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0823e-05 2024-09-11T03:51:53.176 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.176 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.176 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.177 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.063308+0000", 2024-09-11T03:51:53.177 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0013212269999999999 2024-09-11T03:51:53.177 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.177 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.177 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.177 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.063332+0000", 2024-09-11T03:51:53.177 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3972e-05 2024-09-11T03:51:53.177 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.177 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.177 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:51:53.177 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.069064+0000", 2024-09-11T03:51:53.177 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0057316099999999998 2024-09-11T03:51:53.177 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.178 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.178 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.178 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.069119+0000", 2024-09-11T03:51:53.178 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4557e-05 2024-09-11T03:51:53.178 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.178 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.178 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.178 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.178 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.178 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(osd.7.0:629 2.43 e886/854 2:c3768452:::smithi17557870-11:13b v 886'596, mlcod=882'592)", 2024-09-11T03:51:53.178 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:51:44.065273+0000", 2024-09-11T03:51:53.178 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 9.0107262539999997, 2024-09-11T03:51:53.178 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0082643969999999997, 2024-09-11T03:51:53.178 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.178 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.179 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:53.179 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.179 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.179 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.179 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.065273+0000", 2024-09-11T03:51:53.179 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.179 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.179 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.179 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.179 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.065273+0000", 2024-09-11T03:51:53.179 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.179 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.179 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.179 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.180 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.065273+0000", 2024-09-11T03:51:53.180 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.51e-07 2024-09-11T03:51:53.180 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.180 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.180 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.180 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.065279+0000", 2024-09-11T03:51:53.180 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1879999999999999e-06 2024-09-11T03:51:53.180 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.180 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.180 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.180 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.065287+0000", 2024-09-11T03:51:53.180 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5650000000000004e-06 2024-09-11T03:51:53.180 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.180 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.181 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.181 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.065295+0000", 2024-09-11T03:51:53.181 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8930000000000005e-06 2024-09-11T03:51:53.181 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.181 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.181 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.181 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.068035+0000", 2024-09-11T03:51:53.181 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002740106 2024-09-11T03:51:53.181 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.181 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.181 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.182 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.068142+0000", 2024-09-11T03:51:53.182 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000106986 2024-09-11T03:51:53.182 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.182 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.182 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:51:53.182 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.073489+0000", 2024-09-11T03:51:53.182 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0053470419999999998 2024-09-11T03:51:53.182 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.182 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.182 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.182 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.073537+0000", 2024-09-11T03:51:53.182 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8065999999999997e-05 2024-09-11T03:51:53.183 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.183 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.183 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.183 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.183 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.183 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(osd.11.0:1108 2.3e e886/814 2:7efbbea8:::smithi17557870-24:13a v 886'1092, mlcod=877'1088)", 2024-09-11T03:51:53.183 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:51:44.067810+0000", 2024-09-11T03:51:53.183 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 9.0081885830000008, 2024-09-11T03:51:53.183 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0042748789999999997, 2024-09-11T03:51:53.183 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.183 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.183 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:53.183 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.183 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.184 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.184 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.067810+0000", 2024-09-11T03:51:53.184 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.184 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.184 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.184 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.184 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.067810+0000", 2024-09-11T03:51:53.184 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.184 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.184 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.184 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.184 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.067811+0000", 2024-09-11T03:51:53.184 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4499999999999998e-07 2024-09-11T03:51:53.185 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.185 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.185 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.185 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.067815+0000", 2024-09-11T03:51:53.185 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7670000000000003e-06 2024-09-11T03:51:53.185 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.185 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.185 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.185 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.067825+0000", 2024-09-11T03:51:53.185 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9820000000000003e-06 2024-09-11T03:51:53.185 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.185 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.185 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.185 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.067833+0000", 2024-09-11T03:51:53.186 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0579999999999998e-06 2024-09-11T03:51:53.186 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.186 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.186 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.186 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.067998+0000", 2024-09-11T03:51:53.186 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016480999999999999 2024-09-11T03:51:53.186 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.186 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.186 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.186 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.068020+0000", 2024-09-11T03:51:53.186 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1345e-05 2024-09-11T03:51:53.186 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.186 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.187 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:51:53.187 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.072035+0000", 2024-09-11T03:51:53.187 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0040150159999999997 2024-09-11T03:51:53.187 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.187 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.187 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.187 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:44.072085+0000", 2024-09-11T03:51:53.187 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0655999999999997e-05 2024-09-11T03:51:53.187 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.187 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.187 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.187 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.187 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.187 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(osd.7.0:649 2.43 e894/854 2:c3768452:::smithi17557870-11:16c v 894'608, mlcod=891'604)", 2024-09-11T03:51:53.188 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:51:52.061306+0000", 2024-09-11T03:51:53.188 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 1.014693174, 2024-09-11T03:51:53.188 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00345364, 2024-09-11T03:51:53.188 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.188 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.188 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:51:53.188 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.188 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.188 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.188 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.061306+0000", 2024-09-11T03:51:53.188 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.188 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.188 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.189 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.189 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.061306+0000", 2024-09-11T03:51:53.189 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.189 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.189 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.189 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.189 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.061306+0000", 2024-09-11T03:51:53.189 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8200000000000001e-07 2024-09-11T03:51:53.189 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.189 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.189 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.189 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.061312+0000", 2024-09-11T03:51:53.189 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.631e-06 2024-09-11T03:51:53.189 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.190 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.190 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.190 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.061322+0000", 2024-09-11T03:51:53.190 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0688e-05 2024-09-11T03:51:53.190 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.190 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.190 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.190 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.061330+0000", 2024-09-11T03:51:53.190 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7020000000000001e-06 2024-09-11T03:51:53.190 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.190 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.190 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.190 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.061387+0000", 2024-09-11T03:51:53.191 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6638999999999998e-05 2024-09-11T03:51:53.191 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.191 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.191 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.191 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.061411+0000", 2024-09-11T03:51:53.191 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4324999999999998e-05 2024-09-11T03:51:53.191 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.191 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.191 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:51:53.191 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.063851+0000", 2024-09-11T03:51:53.191 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024400149999999998 2024-09-11T03:51:53.191 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.191 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.191 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.192 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.064759+0000", 2024-09-11T03:51:53.192 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000908258 2024-09-11T03:51:53.192 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.192 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.192 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.192 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.192 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.192 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop_reply(osd.9.0:719 2.18 e894/861 ondisk, result = 0)", 2024-09-11T03:51:53.192 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:51:52.071472+0000", 2024-09-11T03:51:53.192 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 1.004527202, 2024-09-11T03:51:53.192 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035995519999999998, 2024-09-11T03:51:53.192 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:51:53.192 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:51:53.192 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:51:53.193 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:51:53.193 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.193 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:51:53.193 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.071472+0000", 2024-09-11T03:51:53.193 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.193 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.193 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.193 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:51:53.193 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.071472+0000", 2024-09-11T03:51:53.193 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:51:53.193 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.193 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.194 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:51:53.194 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.071472+0000", 2024-09-11T03:51:53.194 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4200000000000002e-07 2024-09-11T03:51:53.194 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.194 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.194 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:51:53.194 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.071475+0000", 2024-09-11T03:51:53.194 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7690000000000001e-06 2024-09-11T03:51:53.194 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.194 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.194 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:51:53.194 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.071482+0000", 2024-09-11T03:51:53.194 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7700000000000004e-06 2024-09-11T03:51:53.194 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.195 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.195 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:51:53.195 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.071488+0000", 2024-09-11T03:51:53.195 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2090000000000002e-06 2024-09-11T03:51:53.195 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.195 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.195 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:51:53.195 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.075048+0000", 2024-09-11T03:51:53.195 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00355996 2024-09-11T03:51:53.195 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.195 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.195 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:51:53.195 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.075057+0000", 2024-09-11T03:51:53.196 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7669999999999992e-06 2024-09-11T03:51:53.196 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:51:53.196 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:51:53.196 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:51:53.196 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:52.075071+0000", 2024-09-11T03:51:53.196 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4735e-05 2024-09-11T03:51:53.196 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.196 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.196 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.196 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:51:53.196 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:51:53.196 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:51:53.197 INFO:tasks.rados.rados.2.smithi175.stdout:3704: done (1 left) 2024-09-11T03:51:53.197 INFO:tasks.rados.rados.2.smithi175.stdout:3705: read oid 45 snap 378 2024-09-11T03:51:53.198 INFO:tasks.rados.rados.2.smithi175.stdout:3705: expect (ObjNum 962 snap 308 seq_num 962) 2024-09-11T03:51:53.198 INFO:tasks.rados.rados.2.smithi175.stdout:3706: read oid 30 snap -1 2024-09-11T03:51:53.198 INFO:tasks.rados.rados.2.smithi175.stdout:3706: expect (ObjNum 1255 snap 412 seq_num 1255) 2024-09-11T03:51:53.198 INFO:tasks.rados.rados.2.smithi175.stdout:3707: read oid 48 snap -1 2024-09-11T03:51:53.198 INFO:tasks.rados.rados.2.smithi175.stdout:3707: expect (ObjNum 1212 snap 397 seq_num 1212) 2024-09-11T03:51:53.198 INFO:tasks.rados.rados.2.smithi175.stdout:3708: snap_create 2024-09-11T03:51:53.198 INFO:tasks.rados.rados.2.smithi175.stdout:3703: finishing rollback tid 0 to smithi17557870-34 2024-09-11T03:51:53.198 INFO:tasks.rados.rados.2.smithi175.stdout:3703: finishing rollback tid 1 to smithi17557870-34 2024-09-11T03:51:53.198 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 1210 (ObjNum 33 snap 0 seq_num 1) dirty dne 2024-09-11T03:51:53.198 INFO:tasks.rados.rados.2.smithi175.stdout:3705: expect (ObjNum 962 snap 308 seq_num 962) 2024-09-11T03:51:53.198 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:53.219 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:53.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:53.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:53.221 INFO:tasks.rados.rados.2.smithi175.stdout:3707: expect (ObjNum 1212 snap 397 seq_num 1212) 2024-09-11T03:51:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:53 smithi175 ceph-mon[30894]: 1.0 scrub starts 2024-09-11T03:51:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:53 smithi175 ceph-mon[30894]: 1.0 scrub ok 2024-09-11T03:51:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:53 smithi175 ceph-mon[30894]: 2.1d scrub starts 2024-09-11T03:51:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:53 smithi175 ceph-mon[30894]: 2.1d scrub ok 2024-09-11T03:51:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:53 smithi175 ceph-mon[30894]: 2.3e scrub starts 2024-09-11T03:51:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:53 smithi175 ceph-mon[30894]: 2.3e scrub ok 2024-09-11T03:51:53.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:53 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/698847899' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:51:53.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:53 smithi175 ceph-mon[30894]: 1.1b scrub starts 2024-09-11T03:51:53.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:53 smithi175 ceph-mon[30894]: 1.1b scrub ok 2024-09-11T03:51:53.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:53 smithi175 ceph-mon[30894]: 2.2b scrub starts 2024-09-11T03:51:53.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:53 smithi175 ceph-mon[30894]: 2.2b scrub ok 2024-09-11T03:51:53.302 INFO:tasks.rados.rados.2.smithi175.stdout:3706: expect (ObjNum 1255 snap 412 seq_num 1255) 2024-09-11T03:51:53.321 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:53.342 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:53.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:53.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:53.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:53 smithi084 ceph-mon[27254]: 1.0 scrub starts 2024-09-11T03:51:53.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:53 smithi084 ceph-mon[27254]: 1.0 scrub ok 2024-09-11T03:51:53.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:53 smithi084 ceph-mon[27254]: 2.1d scrub starts 2024-09-11T03:51:53.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:53 smithi084 ceph-mon[27254]: 2.1d scrub ok 2024-09-11T03:51:53.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:53 smithi084 ceph-mon[27254]: 2.3e scrub starts 2024-09-11T03:51:53.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:53 smithi084 ceph-mon[27254]: 2.3e scrub ok 2024-09-11T03:51:53.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:53 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/698847899' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:51:53.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:53 smithi084 ceph-mon[27254]: 1.1b scrub starts 2024-09-11T03:51:53.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:53 smithi084 ceph-mon[27254]: 1.1b scrub ok 2024-09-11T03:51:53.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:53 smithi084 ceph-mon[27254]: 2.2b scrub starts 2024-09-11T03:51:53.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:53 smithi084 ceph-mon[27254]: 2.2b scrub ok 2024-09-11T03:51:53.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:53 smithi121 ceph-mon[30729]: 1.0 scrub starts 2024-09-11T03:51:53.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:53 smithi121 ceph-mon[30729]: 1.0 scrub ok 2024-09-11T03:51:53.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:53 smithi121 ceph-mon[30729]: 2.1d scrub starts 2024-09-11T03:51:53.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:53 smithi121 ceph-mon[30729]: 2.1d scrub ok 2024-09-11T03:51:53.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:53 smithi121 ceph-mon[30729]: 2.3e scrub starts 2024-09-11T03:51:53.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:53 smithi121 ceph-mon[30729]: 2.3e scrub ok 2024-09-11T03:51:53.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:53 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/698847899' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:51:53.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:53 smithi121 ceph-mon[30729]: 1.1b scrub starts 2024-09-11T03:51:53.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:53 smithi121 ceph-mon[30729]: 1.1b scrub ok 2024-09-11T03:51:53.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:53 smithi121 ceph-mon[30729]: 2.2b scrub starts 2024-09-11T03:51:53.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:53 smithi121 ceph-mon[30729]: 2.2b scrub ok 2024-09-11T03:51:53.444 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:53.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:53.466 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:53.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:53.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:51:53.567 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:53.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:53.595 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:53.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:53.654 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:51:53.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:53.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:53.722 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:53.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:53.823 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:53.845 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:53.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:53.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:53.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 11 1 2024-09-11T03:51:53.947 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:53.960 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-11T03:51:53.970 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:53.981 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:53.995 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:53.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:53.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:54.098 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:54.118 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:54.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:54.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:54.149 INFO:tasks.rados.rados.2.smithi175.stdout:3703: done (4 left) 2024-09-11T03:51:54.150 INFO:tasks.rados.rados.2.smithi175.stdout:3705: done (3 left) 2024-09-11T03:51:54.150 INFO:tasks.rados.rados.2.smithi175.stdout:3706: done (2 left) 2024-09-11T03:51:54.150 INFO:tasks.rados.rados.2.smithi175.stdout:3707: done (1 left) 2024-09-11T03:51:54.150 INFO:tasks.rados.rados.2.smithi175.stdout:3708: done (0 left) 2024-09-11T03:51:54.150 INFO:tasks.rados.rados.2.smithi175.stdout:3709: read oid 17 snap -1 2024-09-11T03:51:54.150 INFO:tasks.rados.rados.2.smithi175.stdout:3709: expect (ObjNum 1181 snap 385 seq_num 1181) 2024-09-11T03:51:54.150 INFO:tasks.rados.rados.2.smithi175.stdout:3710: write (excl) oid 27 current snap is 413 2024-09-11T03:51:54.150 INFO:tasks.rados.rados.2.smithi175.stdout:3710: seq_num 1258 ranges {743206=661487,2131877=541092} 2024-09-11T03:51:54.157 INFO:tasks.rados.rados.2.smithi175.stdout:3710: writing smithi17557870-27 from 743206 to 1404693 tid 1 2024-09-11T03:51:54.165 INFO:tasks.rados.rados.2.smithi175.stdout:3710: writing smithi17557870-27 from 2131877 to 2672969 tid 2 2024-09-11T03:51:54.165 INFO:tasks.rados.rados.2.smithi175.stdout:3709: expect (ObjNum 1181 snap 385 seq_num 1181) 2024-09-11T03:51:54.211 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:51:54.220 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:54 smithi175 ceph-mon[30894]: 1.13 scrub starts 2024-09-11T03:51:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:54 smithi175 ceph-mon[30894]: 1.13 scrub ok 2024-09-11T03:51:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:54 smithi175 ceph-mon[30894]: pgmap v1497: 107 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 729 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 7.0 MiB/s rd, 3.2 MiB/s wr, 13 op/s 2024-09-11T03:51:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:54 smithi175 ceph-mon[30894]: 2.3f scrub starts 2024-09-11T03:51:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:54 smithi175 ceph-mon[30894]: 2.3f scrub ok 2024-09-11T03:51:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:54 smithi175 ceph-mon[30894]: 2.1f scrub starts 2024-09-11T03:51:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:54 smithi175 ceph-mon[30894]: 2.1f scrub ok 2024-09-11T03:51:54.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:54 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/698847899' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:51:54.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:54 smithi175 ceph-mon[30894]: osdmap e895: 12 total, 12 up, 11 in 2024-09-11T03:51:54.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:54 smithi175 ceph-mon[30894]: 1.6 scrub starts 2024-09-11T03:51:54.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:54 smithi175 ceph-mon[30894]: 1.6 scrub ok 2024-09-11T03:51:54.242 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:54.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:54.242 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:54.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:54 smithi084 ceph-mon[27254]: 1.13 scrub starts 2024-09-11T03:51:54.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:54 smithi084 ceph-mon[27254]: 1.13 scrub ok 2024-09-11T03:51:54.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:54 smithi084 ceph-mon[27254]: pgmap v1497: 107 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 729 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 7.0 MiB/s rd, 3.2 MiB/s wr, 13 op/s 2024-09-11T03:51:54.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:54 smithi084 ceph-mon[27254]: 2.3f scrub starts 2024-09-11T03:51:54.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:54 smithi084 ceph-mon[27254]: 2.3f scrub ok 2024-09-11T03:51:54.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:54 smithi084 ceph-mon[27254]: 2.1f scrub starts 2024-09-11T03:51:54.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:54 smithi084 ceph-mon[27254]: 2.1f scrub ok 2024-09-11T03:51:54.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:54 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/698847899' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:51:54.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:54 smithi084 ceph-mon[27254]: osdmap e895: 12 total, 12 up, 11 in 2024-09-11T03:51:54.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:54 smithi084 ceph-mon[27254]: 1.6 scrub starts 2024-09-11T03:51:54.264 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:54 smithi084 ceph-mon[27254]: 1.6 scrub ok 2024-09-11T03:51:54.309 INFO:tasks.rados.rados.2.smithi175.stdout:3710: finishing write tid 1 to smithi17557870-27 2024-09-11T03:51:54.309 INFO:tasks.rados.rados.2.smithi175.stdout:3710: finishing write tid 2 to smithi17557870-27 2024-09-11T03:51:54.309 INFO:tasks.rados.rados.2.smithi175.stdout:3710: finishing write tid 3 to smithi17557870-27 2024-09-11T03:51:54.309 INFO:tasks.rados.rados.2.smithi175.stdout:3710: finishing write tid 4 to smithi17557870-27 2024-09-11T03:51:54.309 INFO:tasks.rados.rados.2.smithi175.stdout:3710: oid 27 updating version 0 to 1251 2024-09-11T03:51:54.309 INFO:tasks.rados.rados.2.smithi175.stdout:3710: oid 27 version 1251 is already newer than 1250 2024-09-11T03:51:54.309 INFO:tasks.rados.rados.2.smithi175.stdout:3710: oid 27 updating version 1251 to 1252 2024-09-11T03:51:54.310 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 1252 (ObjNum 1258 snap 413 seq_num 1258) dirty exists 2024-09-11T03:51:54.310 INFO:tasks.rados.rados.2.smithi175.stdout:3710: left oid 27 (ObjNum 1258 snap 413 seq_num 1258) 2024-09-11T03:51:54.310 INFO:tasks.rados.rados.2.smithi175.stdout:3709: done (1 left) 2024-09-11T03:51:54.310 INFO:tasks.rados.rados.2.smithi175.stdout:3710: done (0 left) 2024-09-11T03:51:54.310 INFO:tasks.rados.rados.2.smithi175.stdout:3711: snap_create 2024-09-11T03:51:54.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:54.370 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:54.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:54 smithi121 ceph-mon[30729]: 1.13 scrub starts 2024-09-11T03:51:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:54 smithi121 ceph-mon[30729]: 1.13 scrub ok 2024-09-11T03:51:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:54 smithi121 ceph-mon[30729]: pgmap v1497: 107 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 729 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 7.0 MiB/s rd, 3.2 MiB/s wr, 13 op/s 2024-09-11T03:51:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:54 smithi121 ceph-mon[30729]: 2.3f scrub starts 2024-09-11T03:51:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:54 smithi121 ceph-mon[30729]: 2.3f scrub ok 2024-09-11T03:51:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:54 smithi121 ceph-mon[30729]: 2.1f scrub starts 2024-09-11T03:51:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:54 smithi121 ceph-mon[30729]: 2.1f scrub ok 2024-09-11T03:51:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:54 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/698847899' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:51:54.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:54 smithi121 ceph-mon[30729]: osdmap e895: 12 total, 12 up, 11 in 2024-09-11T03:51:54.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:54 smithi121 ceph-mon[30729]: 1.6 scrub starts 2024-09-11T03:51:54.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:54 smithi121 ceph-mon[30729]: 1.6 scrub ok 2024-09-11T03:51:54.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:54.489 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:51:54.489 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:51:54.493 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:54.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:54.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:54.507 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:51:54.508 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:51:54.525 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:51:54.526 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:51:54.530 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:54.545 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:51:54.545 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:51:54.563 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:51:54.563 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:51:54.581 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:51:54.581 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:51:54.595 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:54.597 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:51:54.597 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:51:54.614 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:51:54.614 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:51:54.616 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:54.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:54.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:54.630 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:51:54.631 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:51:54.647 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:51:54.647 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:51:54.665 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:51:54.666 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:51:54.682 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:51:54.682 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:51:54.694 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:51:54.695 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:51:54.718 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:54.740 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:54.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:54.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:54.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:54.868 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:54.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:54.970 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:54.993 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:54.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:54.994 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:55.096 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:55.118 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:55.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:55.165 INFO:tasks.rados.rados.2.smithi175.stdout:3711: done (0 left) 2024-09-11T03:51:55.165 INFO:tasks.rados.rados.2.smithi175.stdout:3712: snap_remove snap 345 2024-09-11T03:51:55.219 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:55.240 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:55.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:55.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:55.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:55.364 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:55.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:55.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:55 smithi121 ceph-mon[30729]: 2.3d scrub starts 2024-09-11T03:51:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:55 smithi121 ceph-mon[30729]: 2.3d scrub ok 2024-09-11T03:51:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:55 smithi121 ceph-mon[30729]: 2.20 scrub starts 2024-09-11T03:51:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:55 smithi121 ceph-mon[30729]: 2.20 scrub ok 2024-09-11T03:51:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:55 smithi121 ceph-mon[30729]: 2.1a scrub starts 2024-09-11T03:51:55.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:55 smithi121 ceph-mon[30729]: 2.1a scrub ok 2024-09-11T03:51:55.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:55 smithi121 ceph-mon[30729]: 1.12 scrub starts 2024-09-11T03:51:55.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:55 smithi121 ceph-mon[30729]: 1.12 scrub ok 2024-09-11T03:51:55.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:55 smithi121 ceph-mon[30729]: osdmap e896: 12 total, 12 up, 11 in 2024-09-11T03:51:55.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:55 smithi121 ceph-mon[30729]: 2.1c scrub starts 2024-09-11T03:51:55.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:55 smithi121 ceph-mon[30729]: 2.1c scrub ok 2024-09-11T03:51:55.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:55 smithi121 ceph-mon[30729]: 2.38 scrub starts 2024-09-11T03:51:55.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:55 smithi121 ceph-mon[30729]: 2.38 scrub ok 2024-09-11T03:51:55.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:55 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/731534807' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:55.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:51:55.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:55 smithi175 ceph-mon[30894]: 2.3d scrub starts 2024-09-11T03:51:55.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:55 smithi175 ceph-mon[30894]: 2.3d scrub ok 2024-09-11T03:51:55.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:55 smithi175 ceph-mon[30894]: 2.20 scrub starts 2024-09-11T03:51:55.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:55 smithi175 ceph-mon[30894]: 2.20 scrub ok 2024-09-11T03:51:55.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:55 smithi175 ceph-mon[30894]: 2.1a scrub starts 2024-09-11T03:51:55.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:55 smithi175 ceph-mon[30894]: 2.1a scrub ok 2024-09-11T03:51:55.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:55 smithi175 ceph-mon[30894]: 1.12 scrub starts 2024-09-11T03:51:55.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:55 smithi175 ceph-mon[30894]: 1.12 scrub ok 2024-09-11T03:51:55.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:55 smithi175 ceph-mon[30894]: osdmap e896: 12 total, 12 up, 11 in 2024-09-11T03:51:55.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:55 smithi175 ceph-mon[30894]: 2.1c scrub starts 2024-09-11T03:51:55.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:55 smithi175 ceph-mon[30894]: 2.1c scrub ok 2024-09-11T03:51:55.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:55 smithi175 ceph-mon[30894]: 2.38 scrub starts 2024-09-11T03:51:55.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:55 smithi175 ceph-mon[30894]: 2.38 scrub ok 2024-09-11T03:51:55.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:55 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/731534807' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:55.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:55 smithi084 ceph-mon[27254]: 2.3d scrub starts 2024-09-11T03:51:55.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:55 smithi084 ceph-mon[27254]: 2.3d scrub ok 2024-09-11T03:51:55.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:55 smithi084 ceph-mon[27254]: 2.20 scrub starts 2024-09-11T03:51:55.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:55 smithi084 ceph-mon[27254]: 2.20 scrub ok 2024-09-11T03:51:55.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:55 smithi084 ceph-mon[27254]: 2.1a scrub starts 2024-09-11T03:51:55.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:55 smithi084 ceph-mon[27254]: 2.1a scrub ok 2024-09-11T03:51:55.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:55 smithi084 ceph-mon[27254]: 1.12 scrub starts 2024-09-11T03:51:55.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:55 smithi084 ceph-mon[27254]: 1.12 scrub ok 2024-09-11T03:51:55.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:55 smithi084 ceph-mon[27254]: osdmap e896: 12 total, 12 up, 11 in 2024-09-11T03:51:55.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:55 smithi084 ceph-mon[27254]: 2.1c scrub starts 2024-09-11T03:51:55.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:55 smithi084 ceph-mon[27254]: 2.1c scrub ok 2024-09-11T03:51:55.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:55 smithi084 ceph-mon[27254]: 2.38 scrub starts 2024-09-11T03:51:55.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:55 smithi084 ceph-mon[27254]: 2.38 scrub ok 2024-09-11T03:51:55.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:55 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/731534807' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:55.486 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:55.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:55.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:51:55.564 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-11T03:51:55.587 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:55.627 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:55.629 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:55.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:55.751 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:55.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:55.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:55.814 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:51:55.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:55.873 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:55.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:55.975 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:55.996 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:55.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:55.997 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:56.099 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:56.122 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:56.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:56.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:56.161 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:51:56.165 INFO:tasks.rados.rados.2.smithi175.stdout:3712: done (0 left) 2024-09-11T03:51:56.165 INFO:tasks.rados.rados.2.smithi175.stdout:3713: delete oid 13 current snap is 414 2024-09-11T03:51:56.175 INFO:tasks.rados.rados.2.smithi175.stdout:3713: done (0 left) 2024-09-11T03:51:56.176 INFO:tasks.rados.rados.2.smithi175.stdout:3714: read oid 20 snap -1 2024-09-11T03:51:56.176 INFO:tasks.rados.rados.2.smithi175.stdout:3714: expect (ObjNum 973 snap 313 seq_num 973) 2024-09-11T03:51:56.176 INFO:tasks.rados.rados.2.smithi175.stdout:3715: rollback oid 9 current snap is 414 2024-09-11T03:51:56.176 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 9 to 392 2024-09-11T03:51:56.176 INFO:tasks.rados.rados.2.smithi175.stdout:3716: write (excl) oid 8 current snap is 414 2024-09-11T03:51:56.176 INFO:tasks.rados.rados.2.smithi175.stdout:3716: seq_num 1260 ranges {776820=411985,1848841=544761,2918265=258555} 2024-09-11T03:51:56.182 INFO:tasks.rados.rados.2.smithi175.stdout:3716: writing smithi17557870-8 from 776820 to 1188805 tid 1 2024-09-11T03:51:56.190 INFO:tasks.rados.rados.2.smithi175.stdout:3716: writing smithi17557870-8 from 1848841 to 2393602 tid 2 2024-09-11T03:51:56.194 INFO:tasks.rados.rados.2.smithi175.stdout:3716: writing smithi17557870-8 from 2918265 to 3176820 tid 3 2024-09-11T03:51:56.194 INFO:tasks.rados.rados.2.smithi175.stdout:3717: write (excl) oid 44 current snap is 414 2024-09-11T03:51:56.194 INFO:tasks.rados.rados.2.smithi175.stdout:3715: finishing rollback tid 1 to smithi17557870-9 2024-09-11T03:51:56.195 INFO:tasks.rados.rados.2.smithi175.stdout:3715: finishing rollback tid 2 to smithi17557870-9 2024-09-11T03:51:56.195 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 1320 (ObjNum 1033 snap 340 seq_num 1033) dirty exists 2024-09-11T03:51:56.195 INFO:tasks.rados.rados.2.smithi175.stdout:3717: seq_num 1261 ranges {793627=487234,1790970=404772,2829335=364292} 2024-09-11T03:51:56.201 INFO:tasks.rados.rados.2.smithi175.stdout:3717: writing smithi17557870-44 from 793627 to 1280861 tid 1 2024-09-11T03:51:56.203 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:56 smithi175 ceph-mon[30894]: pgmap v1500: 107 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 729 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 7.0 MiB/s rd, 3.2 MiB/s wr, 13 op/s 2024-09-11T03:51:56.203 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:56 smithi175 ceph-mon[30894]: 1.8 scrub starts 2024-09-11T03:51:56.203 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:56 smithi175 ceph-mon[30894]: 1.8 scrub ok 2024-09-11T03:51:56.203 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:56 smithi175 ceph-mon[30894]: osdmap e897: 12 total, 12 up, 11 in 2024-09-11T03:51:56.203 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:56 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:51:56.203 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:56 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:51:56.204 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:56 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:51:56.204 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:56 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2967713969' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:51:56.206 INFO:tasks.rados.rados.2.smithi175.stdout:3717: writing smithi17557870-44 from 1790970 to 2195742 tid 2 2024-09-11T03:51:56.212 INFO:tasks.rados.rados.2.smithi175.stdout:3717: writing smithi17557870-44 from 2829335 to 3193627 tid 3 2024-09-11T03:51:56.212 INFO:tasks.rados.rados.2.smithi175.stdout:3715: done (3 left) 2024-09-11T03:51:56.212 INFO:tasks.rados.rados.2.smithi175.stdout:3718: read oid 21 snap 314 2024-09-11T03:51:56.212 INFO:tasks.rados.rados.2.smithi175.stdout:3718: expect (ObjNum 829 snap 263 seq_num 829) 2024-09-11T03:51:56.212 INFO:tasks.rados.rados.2.smithi175.stdout:3719: snap_create 2024-09-11T03:51:56.212 INFO:tasks.rados.rados.2.smithi175.stdout:3714: expect (ObjNum 973 snap 313 seq_num 973) 2024-09-11T03:51:56.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:56.244 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:56.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:56.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:56.284 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:56 smithi084 ceph-mon[27254]: pgmap v1500: 107 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 729 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 7.0 MiB/s rd, 3.2 MiB/s wr, 13 op/s 2024-09-11T03:51:56.284 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:56 smithi084 ceph-mon[27254]: 1.8 scrub starts 2024-09-11T03:51:56.284 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:56 smithi084 ceph-mon[27254]: 1.8 scrub ok 2024-09-11T03:51:56.284 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:56 smithi084 ceph-mon[27254]: osdmap e897: 12 total, 12 up, 11 in 2024-09-11T03:51:56.284 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:51:56.284 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:56 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:51:56.284 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:56 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:51:56.284 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2967713969' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:51:56.290 INFO:tasks.rados.rados.2.smithi175.stdout:3716: finishing write tid 1 to smithi17557870-8 2024-09-11T03:51:56.290 INFO:tasks.rados.rados.2.smithi175.stdout:3716: finishing write tid 2 to smithi17557870-8 2024-09-11T03:51:56.291 INFO:tasks.rados.rados.2.smithi175.stdout:3716: finishing write tid 3 to smithi17557870-8 2024-09-11T03:51:56.291 INFO:tasks.rados.rados.2.smithi175.stdout:3716: finishing write tid 4 to smithi17557870-8 2024-09-11T03:51:56.291 INFO:tasks.rados.rados.2.smithi175.stdout:3716: finishing write tid 5 to smithi17557870-8 2024-09-11T03:51:56.291 INFO:tasks.rados.rados.2.smithi175.stdout:3716: oid 8 updating version 0 to 1192 2024-09-11T03:51:56.291 INFO:tasks.rados.rados.2.smithi175.stdout:3716: oid 8 updating version 1192 to 1195 2024-09-11T03:51:56.291 INFO:tasks.rados.rados.2.smithi175.stdout:3716: oid 8 version 1195 is already newer than 1194 2024-09-11T03:51:56.291 INFO:tasks.rados.rados.2.smithi175.stdout:3716: oid 8 version 1195 is already newer than 1193 2024-09-11T03:51:56.291 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 1195 (ObjNum 1260 snap 414 seq_num 1260) dirty exists 2024-09-11T03:51:56.291 INFO:tasks.rados.rados.2.smithi175.stdout:3716: left oid 8 (ObjNum 1260 snap 414 seq_num 1260) 2024-09-11T03:51:56.291 INFO:tasks.rados.rados.2.smithi175.stdout:3717: finishing write tid 1 to smithi17557870-44 2024-09-11T03:51:56.291 INFO:tasks.rados.rados.2.smithi175.stdout:3717: finishing write tid 2 to smithi17557870-44 2024-09-11T03:51:56.291 INFO:tasks.rados.rados.2.smithi175.stdout:3717: finishing write tid 3 to smithi17557870-44 2024-09-11T03:51:56.291 INFO:tasks.rados.rados.2.smithi175.stdout:3717: finishing write tid 4 to smithi17557870-44 2024-09-11T03:51:56.292 INFO:tasks.rados.rados.2.smithi175.stdout:3717: finishing write tid 5 to smithi17557870-44 2024-09-11T03:51:56.292 INFO:tasks.rados.rados.2.smithi175.stdout:3717: oid 44 updating version 0 to 1033 2024-09-11T03:51:56.292 INFO:tasks.rados.rados.2.smithi175.stdout:3717: oid 44 version 1033 is already newer than 1032 2024-09-11T03:51:56.292 INFO:tasks.rados.rados.2.smithi175.stdout:3717: oid 44 version 1033 is already newer than 1031 2024-09-11T03:51:56.292 INFO:tasks.rados.rados.2.smithi175.stdout:3717: oid 44 version 1033 is already newer than 1030 2024-09-11T03:51:56.292 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 44 v 1033 (ObjNum 1261 snap 414 seq_num 1261) dirty exists 2024-09-11T03:51:56.292 INFO:tasks.rados.rados.2.smithi175.stdout:3717: left oid 44 (ObjNum 1261 snap 414 seq_num 1261) 2024-09-11T03:51:56.292 INFO:tasks.rados.rados.2.smithi175.stdout:3718: expect (ObjNum 829 snap 263 seq_num 829) 2024-09-11T03:51:56.312 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:51:56.312 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:51:56.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:56.368 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:56.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:56.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:56.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:56 smithi121 ceph-mon[30729]: pgmap v1500: 107 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 729 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 7.0 MiB/s rd, 3.2 MiB/s wr, 13 op/s 2024-09-11T03:51:56.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:56 smithi121 ceph-mon[30729]: 1.8 scrub starts 2024-09-11T03:51:56.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:56 smithi121 ceph-mon[30729]: 1.8 scrub ok 2024-09-11T03:51:56.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:56 smithi121 ceph-mon[30729]: osdmap e897: 12 total, 12 up, 11 in 2024-09-11T03:51:56.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:56 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:51:56.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:56 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:51:56.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:56 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:51:56.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:56 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2967713969' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:51:56.470 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:56.491 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:56.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:56.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:56.593 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:56.615 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:56.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:56.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:51:56.717 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:56.739 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:56.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:56.841 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:56.863 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:56.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:56.965 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:56.987 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:56.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:56.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:57.055 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.11 to 1 (10000) 2024-09-11T03:51:57.060 INFO:tasks.rados.rados.2.smithi175.stdout:3714: done (4 left) 2024-09-11T03:51:57.060 INFO:tasks.rados.rados.2.smithi175.stdout:3716: done (3 left) 2024-09-11T03:51:57.060 INFO:tasks.rados.rados.2.smithi175.stdout:3717: done (2 left) 2024-09-11T03:51:57.061 INFO:tasks.rados.rados.2.smithi175.stdout:3718: done (1 left) 2024-09-11T03:51:57.061 INFO:tasks.rados.rados.2.smithi175.stdout:3719: done (0 left) 2024-09-11T03:51:57.061 INFO:tasks.rados.rados.2.smithi175.stdout:3720: write (excl) oid 36 current snap is 415 2024-09-11T03:51:57.061 INFO:tasks.rados.rados.2.smithi175.stdout:3720: seq_num 1262 ranges {410434=678836,1565805=548430,2573111=637323} 2024-09-11T03:51:57.066 INFO:tasks.rados.rados.2.smithi175.stdout:3720: writing smithi17557870-36 from 410434 to 1089270 tid 1 2024-09-11T03:51:57.072 INFO:tasks.rados.rados.2.smithi175.stdout:3720: writing smithi17557870-36 from 1565805 to 2114235 tid 2 2024-09-11T03:51:57.079 INFO:tasks.rados.rados.2.smithi175.stdout:3720: writing smithi17557870-36 from 2573111 to 3210434 tid 3 2024-09-11T03:51:57.079 INFO:tasks.rados.rados.2.smithi175.stdout:3721: write (excl) oid 23 current snap is 415 2024-09-11T03:51:57.079 INFO:tasks.rados.rados.2.smithi175.stdout:3721: seq_num 1263 ranges {427241=754085,1624287=408441,2600534=549964} 2024-09-11T03:51:57.085 INFO:tasks.rados.rados.2.smithi175.stdout:3721: writing smithi17557870-23 from 427241 to 1181326 tid 1 2024-09-11T03:51:57.088 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-11T03:51:57.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:57.090 INFO:tasks.rados.rados.2.smithi175.stdout:3721: writing smithi17557870-23 from 1624287 to 2032728 tid 2 2024-09-11T03:51:57.097 INFO:tasks.rados.rados.2.smithi175.stdout:3721: writing smithi17557870-23 from 2600534 to 3150498 tid 3 2024-09-11T03:51:57.097 INFO:tasks.rados.rados.2.smithi175.stdout:3722: copy_from oid 17 from oid 29 current snap is 415 2024-09-11T03:51:57.097 INFO:tasks.rados.rados.2.smithi175.stdout:3720: finishing write tid 1 to smithi17557870-36 2024-09-11T03:51:57.097 INFO:tasks.rados.rados.2.smithi175.stdout:3720: finishing write tid 2 to smithi17557870-36 2024-09-11T03:51:57.097 INFO:tasks.rados.rados.2.smithi175.stdout:3720: finishing write tid 3 to smithi17557870-36 2024-09-11T03:51:57.097 INFO:tasks.rados.rados.2.smithi175.stdout:3720: finishing write tid 4 to smithi17557870-36 2024-09-11T03:51:57.097 INFO:tasks.rados.rados.2.smithi175.stdout:3720: finishing write tid 5 to smithi17557870-36 2024-09-11T03:51:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:3720: oid 36 updating version 0 to 1380 2024-09-11T03:51:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:3720: oid 36 version 1380 is already newer than 1379 2024-09-11T03:51:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:3720: oid 36 version 1380 is already newer than 1378 2024-09-11T03:51:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:3720: oid 36 version 1380 is already newer than 1377 2024-09-11T03:51:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 1380 (ObjNum 1262 snap 415 seq_num 1262) dirty exists 2024-09-11T03:51:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:3720: left oid 36 (ObjNum 1262 snap 415 seq_num 1262) 2024-09-11T03:51:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:3721: finishing write tid 1 to smithi17557870-23 2024-09-11T03:51:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:3721: finishing write tid 2 to smithi17557870-23 2024-09-11T03:51:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:3720: done (2 left) 2024-09-11T03:51:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:3723: write (excl) oid 31 current snap is 415 2024-09-11T03:51:57.098 INFO:tasks.rados.rados.2.smithi175.stdout:3723: seq_num 1264 ranges {444048=429334,1282769=552099,2627957=616091} 2024-09-11T03:51:57.101 INFO:tasks.rados.rados.2.smithi175.stdout:3723: writing smithi17557870-31 from 444048 to 873382 tid 1 2024-09-11T03:51:57.107 INFO:tasks.rados.rados.2.smithi175.stdout:3723: writing smithi17557870-31 from 1282769 to 1834868 tid 2 2024-09-11T03:51:57.110 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:57.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:57.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:57.116 INFO:tasks.rados.rados.2.smithi175.stdout:3723: writing smithi17557870-31 from 2627957 to 3244048 tid 3 2024-09-11T03:51:57.116 INFO:tasks.rados.rados.2.smithi175.stdout:3724: snap_create 2024-09-11T03:51:57.116 INFO:tasks.rados.rados.2.smithi175.stdout:3721: finishing write tid 3 to smithi17557870-23 2024-09-11T03:51:57.116 INFO:tasks.rados.rados.2.smithi175.stdout:3721: finishing write tid 4 to smithi17557870-23 2024-09-11T03:51:57.117 INFO:tasks.rados.rados.2.smithi175.stdout:3721: finishing write tid 5 to smithi17557870-23 2024-09-11T03:51:57.117 INFO:tasks.rados.rados.2.smithi175.stdout:3721: oid 23 updating version 0 to 1382 2024-09-11T03:51:57.117 INFO:tasks.rados.rados.2.smithi175.stdout:3721: oid 23 updating version 1382 to 1383 2024-09-11T03:51:57.117 INFO:tasks.rados.rados.2.smithi175.stdout:3721: oid 23 version 1383 is already newer than 1381 2024-09-11T03:51:57.117 INFO:tasks.rados.rados.2.smithi175.stdout:3721: oid 23 updating version 1383 to 1384 2024-09-11T03:51:57.117 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 1384 (ObjNum 1263 snap 415 seq_num 1263) dirty exists 2024-09-11T03:51:57.117 INFO:tasks.rados.rados.2.smithi175.stdout:3721: left oid 23 (ObjNum 1263 snap 415 seq_num 1263) 2024-09-11T03:51:57.117 INFO:tasks.rados.rados.2.smithi175.stdout:3722: finishing copy_from to smithi17557870-17 2024-09-11T03:51:57.117 INFO:tasks.rados.rados.2.smithi175.stdout:3722: got expected ENOENT (src dne) 2024-09-11T03:51:57.117 INFO:tasks.rados.rados.2.smithi175.stdout:3722: finishing copy_from racing read to smithi17557870-17 2024-09-11T03:51:57.117 INFO:tasks.rados.rados.2.smithi175.stdout:3723: finishing write tid 1 to smithi17557870-31 2024-09-11T03:51:57.117 INFO:tasks.rados.rados.2.smithi175.stdout:3723: finishing write tid 2 to smithi17557870-31 2024-09-11T03:51:57.118 INFO:tasks.rados.rados.2.smithi175.stdout:3723: finishing write tid 3 to smithi17557870-31 2024-09-11T03:51:57.118 INFO:tasks.rados.rados.2.smithi175.stdout:3723: finishing write tid 4 to smithi17557870-31 2024-09-11T03:51:57.118 INFO:tasks.rados.rados.2.smithi175.stdout:3723: finishing write tid 5 to smithi17557870-31 2024-09-11T03:51:57.118 INFO:tasks.rados.rados.2.smithi175.stdout:3723: oid 31 updating version 0 to 1254 2024-09-11T03:51:57.118 INFO:tasks.rados.rados.2.smithi175.stdout:3723: oid 31 updating version 1254 to 1255 2024-09-11T03:51:57.119 INFO:tasks.rados.rados.2.smithi175.stdout:3723: oid 31 version 1255 is already newer than 1253 2024-09-11T03:51:57.119 INFO:tasks.rados.rados.2.smithi175.stdout:3723: oid 31 updating version 1255 to 1256 2024-09-11T03:51:57.119 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 1256 (ObjNum 1264 snap 415 seq_num 1264) dirty exists 2024-09-11T03:51:57.119 INFO:tasks.rados.rados.2.smithi175.stdout:3723: left oid 31 (ObjNum 1264 snap 415 seq_num 1264) 2024-09-11T03:51:57.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:57.236 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:57.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:57.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:57.334 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:51:57.338 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:57.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:57.360 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:57.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:57 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:57 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2967713969' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:51:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:51:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:57 smithi084 ceph-mon[27254]: osdmap e898: 12 total, 12 up, 11 in 2024-09-11T03:51:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:51:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:57 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:51:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:57 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1208217927' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:51:57.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:57 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1208217927' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:51:57.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:57 smithi084 ceph-mon[27254]: osdmap e899: 12 total, 12 up, 11 in 2024-09-11T03:51:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:57 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:57 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2967713969' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:51:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:57 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:51:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:57 smithi121 ceph-mon[30729]: osdmap e898: 12 total, 12 up, 11 in 2024-09-11T03:51:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:57 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:51:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:57 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:51:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:57 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1208217927' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:51:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:57 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1208217927' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:51:57.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:57 smithi121 ceph-mon[30729]: osdmap e899: 12 total, 12 up, 11 in 2024-09-11T03:51:57.433 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:57 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:51:57.434 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:57 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2967713969' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:51:57.434 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:57 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:51:57.434 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:57 smithi175 ceph-mon[30894]: osdmap e898: 12 total, 12 up, 11 in 2024-09-11T03:51:57.434 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:57 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:51:57.434 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:57 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:51:57.434 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:57 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1208217927' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:51:57.434 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:57 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1208217927' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:51:57.434 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:57 smithi175 ceph-mon[30894]: osdmap e899: 12 total, 12 up, 11 in 2024-09-11T03:51:57.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:57.486 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:57.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:57.513 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:51:57.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:51:57.610 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:57.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:57.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:51:57.712 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:57.736 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:57.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:57.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:57.759 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:51:57.760 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:51:57.838 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:57.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 10 1 2024-09-11T03:51:57.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:57.865 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:57.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:57.894 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:57.967 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:57.988 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:57.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:57.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:58.062 INFO:tasks.rados.rados.2.smithi175.stdout:3721: done (3 left) 2024-09-11T03:51:58.062 INFO:tasks.rados.rados.2.smithi175.stdout:3722: done (2 left) 2024-09-11T03:51:58.062 INFO:tasks.rados.rados.2.smithi175.stdout:3723: done (1 left) 2024-09-11T03:51:58.062 INFO:tasks.rados.rados.2.smithi175.stdout:3724: done (0 left) 2024-09-11T03:51:58.063 INFO:tasks.rados.rados.2.smithi175.stdout:3725: copy_from oid 6 from oid 29 current snap is 416 2024-09-11T03:51:58.063 INFO:tasks.rados.rados.2.smithi175.stdout:3726: snap_create 2024-09-11T03:51:58.067 INFO:tasks.rados.rados.2.smithi175.stdout:3725: finishing copy_from to smithi17557870-6 2024-09-11T03:51:58.068 INFO:tasks.rados.rados.2.smithi175.stdout:3725: got expected ENOENT (src dne) 2024-09-11T03:51:58.068 INFO:tasks.rados.rados.2.smithi175.stdout:3725: finishing copy_from racing read to smithi17557870-6 2024-09-11T03:51:58.090 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:58.112 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:58.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:58.113 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:58.214 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:58.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:58.236 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:58.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:58.337 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:58.358 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:58.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:58.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:58.364 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:51:58.365 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:51:58.381 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:58 smithi175 ceph-mon[30894]: pgmap v1503: 107 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 728 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-09-11T03:51:58.381 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:58 smithi175 ceph-mon[30894]: osdmap e900: 12 total, 12 up, 11 in 2024-09-11T03:51:58.382 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:51:58.382 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:51:58.391 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:58.400 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:51:58.400 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:51:58.408 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-11T03:51:58.421 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:51:58.422 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:51:58.422 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:58 smithi084 ceph-mon[27254]: pgmap v1503: 107 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 728 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-09-11T03:51:58.422 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:58 smithi084 ceph-mon[27254]: osdmap e900: 12 total, 12 up, 11 in 2024-09-11T03:51:58.440 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:51:58.440 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:51:58.457 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:51:58.457 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:51:58.459 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:58.474 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:51:58.474 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:51:58.480 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:58.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:58.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:58.490 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:51:58.490 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:51:58.507 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:51:58.507 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:51:58.523 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:51:58.524 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:51:58.542 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:51:58.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:51:58.559 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:51:58.559 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:51:58.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:51:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:58.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:58.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:51:58.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:58 smithi121 ceph-mon[30729]: pgmap v1503: 107 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 728 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-09-11T03:51:58.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:58 smithi121 ceph-mon[30729]: osdmap e900: 12 total, 12 up, 11 in 2024-09-11T03:51:58.692 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:51:58.711 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:58.732 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:58.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:58.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:58.834 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:51:58.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:58.856 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:58.856 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:51:58.958 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:51:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:58.981 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:58.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:51:59.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:51:59.064 INFO:tasks.rados.rados.2.smithi175.stdout:3725: done (1 left) 2024-09-11T03:51:59.064 INFO:tasks.rados.rados.2.smithi175.stdout:3726: done (0 left) 2024-09-11T03:51:59.064 INFO:tasks.rados.rados.2.smithi175.stdout:3727: snap_remove snap 321 2024-09-11T03:51:59.083 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:51:59.103 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:59.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:59.104 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:51:59.143 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:51:59.144 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:51:59.144 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:51:59.144 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:51:59.206 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:51:59.227 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:59.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:59.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:51:59.329 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:59.355 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:59.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:59.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:59.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:59 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2074541804' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:59.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:51:59 smithi084 ceph-mon[27254]: osdmap e901: 12 total, 12 up, 11 in 2024-09-11T03:51:59.446 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:59 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2074541804' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:59.446 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:51:59 smithi175 ceph-mon[30894]: osdmap e901: 12 total, 12 up, 11 in 2024-09-11T03:51:59.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:51:59.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:59.480 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:59.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:51:59.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:51:59.583 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:51:59.604 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:51:59.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:59.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:51:59.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:59 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2074541804' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:51:59.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:51:59 smithi121 ceph-mon[30729]: osdmap e901: 12 total, 12 up, 11 in 2024-09-11T03:51:59.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:51:59.732 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:59.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:59.734 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-11T03:51:59.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:51:59.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:51:59.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:59.863 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:51:59.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:51:59.964 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:51:59.974 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:51:59.986 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:51:59.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:51:59.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:00.060 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.10 to 1 (10000) 2024-09-11T03:52:00.064 INFO:tasks.rados.rados.2.smithi175.stdout:3727: done (0 left) 2024-09-11T03:52:00.064 INFO:tasks.rados.rados.2.smithi175.stdout:3728: read oid 41 snap -1 2024-09-11T03:52:00.064 INFO:tasks.rados.rados.2.smithi175.stdout:3728: expect deleted 2024-09-11T03:52:00.064 INFO:tasks.rados.rados.2.smithi175.stdout:3729: snap_remove snap 370 2024-09-11T03:52:00.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:00.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:00.114 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:00.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:00.216 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:00.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:00.240 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:00.240 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:00.342 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:00.346 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:00 smithi175 ceph-mon[30894]: pgmap v1506: 107 pgs: 107 active+clean; 729 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 16 MiB/s rd, 2.8 MiB/s wr, 26 op/s 2024-09-11T03:52:00.347 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:00 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/406220418' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:52:00.347 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:00 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/406220418' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:52:00.347 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:00 smithi175 ceph-mon[30894]: osdmap e902: 12 total, 12 up, 11 in 2024-09-11T03:52:00.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:00.364 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:00.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:00.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:00 smithi084 ceph-mon[27254]: pgmap v1506: 107 pgs: 107 active+clean; 729 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 16 MiB/s rd, 2.8 MiB/s wr, 26 op/s 2024-09-11T03:52:00.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:00 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/406220418' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:52:00.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:00 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/406220418' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:52:00.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:00 smithi084 ceph-mon[27254]: osdmap e902: 12 total, 12 up, 11 in 2024-09-11T03:52:00.442 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:52:00.442 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:52:00.443 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:52:00.443 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:52:00.443 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:52:00.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:00.491 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:00.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:00.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:00.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:00.619 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:00.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:00.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:00.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:00 smithi121 ceph-mon[30729]: pgmap v1506: 107 pgs: 107 active+clean; 729 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 16 MiB/s rd, 2.8 MiB/s wr, 26 op/s 2024-09-11T03:52:00.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:00 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/406220418' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:52:00.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:00 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/406220418' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:52:00.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:00 smithi121 ceph-mon[30729]: osdmap e902: 12 total, 12 up, 11 in 2024-09-11T03:52:00.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:00.746 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:00.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:00.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:00.767 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-11T03:52:00.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:00.875 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:00.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:00.977 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:01.000 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:01.001 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:01.046 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:01.062 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:52:01.066 INFO:tasks.rados.rados.2.smithi175.stdout:3729: done (1 left) 2024-09-11T03:52:01.066 INFO:tasks.rados.rados.2.smithi175.stdout:3730: read oid 46 snap -1 2024-09-11T03:52:01.066 INFO:tasks.rados.rados.2.smithi175.stdout:3730: expect (ObjNum 1243 snap 408 seq_num 1243) 2024-09-11T03:52:01.066 INFO:tasks.rados.rados.2.smithi175.stdout:3731: read oid 29 snap -1 2024-09-11T03:52:01.067 INFO:tasks.rados.rados.2.smithi175.stdout:3731: expect deleted 2024-09-11T03:52:01.067 INFO:tasks.rados.rados.2.smithi175.stdout:3732: snap_create 2024-09-11T03:52:01.078 INFO:tasks.rados.rados.2.smithi175.stdout:3730: expect (ObjNum 1243 snap 408 seq_num 1243) 2024-09-11T03:52:01.102 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:01.123 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:01.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:01.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:01.165 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-11T03:52:01.224 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:01.246 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:01.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:01.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:01.348 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:01.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:01.371 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:01.372 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2896829445' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:52:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2896829445' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:52:01.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:01 smithi084 ceph-mon[27254]: osdmap e903: 12 total, 12 up, 11 in 2024-09-11T03:52:01.407 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:52:01.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2896829445' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:52:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2896829445' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:52:01.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:01 smithi175 ceph-mon[30894]: osdmap e903: 12 total, 12 up, 11 in 2024-09-11T03:52:01.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:01.499 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:01.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:01.576 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:52:01.601 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:01.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:01.623 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:01.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:01.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2896829445' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:52:01.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2896829445' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:52:01.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:01 smithi121 ceph-mon[30729]: osdmap e903: 12 total, 12 up, 11 in 2024-09-11T03:52:01.724 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:01.746 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:01.746 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:01.847 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:01.867 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:52:01.868 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:52:01.868 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:52:01.868 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:52:01.868 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.868 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2972 2.c 2:32ade24e:::smithi17557870-45:head [copy-from ver 171] snapc 80=[80,7b,7a,79,77,75] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-11T03:52:01.868 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:52:01.868 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 556.79289222, 2024-09-11T03:52:01.868 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021036193000000002, 2024-09-11T03:52:01.868 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.868 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.868 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:01.869 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.869 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.869 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.869 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2972 2024-09-11T03:52:01.869 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.869 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.869 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.869 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.869 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:52:01.869 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.869 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.869 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.869 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.870 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074094+0000", 2024-09-11T03:52:01.870 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.870 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.870 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.870 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.870 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074096+0000", 2024-09-11T03:52:01.870 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.685e-06 2024-09-11T03:52:01.870 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.870 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.870 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.870 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074104+0000", 2024-09-11T03:52:01.870 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9950000000000005e-06 2024-09-11T03:52:01.870 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.871 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.871 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074123+0000", 2024-09-11T03:52:01.871 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9587999999999999e-05 2024-09-11T03:52:01.871 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.871 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.871 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074132+0000", 2024-09-11T03:52:01.871 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1579999999999994e-06 2024-09-11T03:52:01.871 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.871 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.871 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074204+0000", 2024-09-11T03:52:01.871 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2072999999999999e-05 2024-09-11T03:52:01.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.872 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:52:01.872 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074302+0000", 2024-09-11T03:52:01.872 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7706000000000003e-05 2024-09-11T03:52:01.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.872 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:52:01.872 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.087397+0000", 2024-09-11T03:52:01.872 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013095292999999999 2024-09-11T03:52:01.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.872 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:52:01.873 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090766+0000", 2024-09-11T03:52:01.873 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0033682109999999999 2024-09-11T03:52:01.873 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.873 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.873 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:01.873 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.094977+0000", 2024-09-11T03:52:01.873 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0042108759999999997 2024-09-11T03:52:01.873 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.873 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.873 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:01.873 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095061+0000", 2024-09-11T03:52:01.873 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4827999999999995e-05 2024-09-11T03:52:01.873 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.873 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.874 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:52:01.874 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095099+0000", 2024-09-11T03:52:01.874 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7546999999999998e-05 2024-09-11T03:52:01.874 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.874 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.874 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.874 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.095130+0000", 2024-09-11T03:52:01.874 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1232999999999997e-05 2024-09-11T03:52:01.874 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.874 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.874 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.874 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.874 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.875 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:2973 2.c 2:32ade24e:::smithi17557870-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e348)", 2024-09-11T03:52:01.875 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:52:01.875 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 556.79282710200005, 2024-09-11T03:52:01.875 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016062098, 2024-09-11T03:52:01.875 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.875 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.875 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:52:01.875 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.875 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.875 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.875 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 2973 2024-09-11T03:52:01.875 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.875 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.875 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.876 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.876 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:52:01.876 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.876 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.876 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.876 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.876 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074159+0000", 2024-09-11T03:52:01.876 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.876 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.876 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.876 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.876 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074160+0000", 2024-09-11T03:52:01.876 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.46e-07 2024-09-11T03:52:01.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.877 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.877 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.877 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074162+0000", 2024-09-11T03:52:01.877 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5220000000000002e-06 2024-09-11T03:52:01.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.877 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.877 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.877 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074178+0000", 2024-09-11T03:52:01.877 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.609e-05 2024-09-11T03:52:01.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.877 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.877 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.877 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074183+0000", 2024-09-11T03:52:01.878 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8280000000000002e-06 2024-09-11T03:52:01.878 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.878 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.878 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074428+0000", 2024-09-11T03:52:01.878 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00024529599999999999 2024-09-11T03:52:01.878 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.878 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:01.878 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.074469+0000", 2024-09-11T03:52:01.878 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0463000000000002e-05 2024-09-11T03:52:01.878 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.878 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.879 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090093+0000", 2024-09-11T03:52:01.879 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015624449 2024-09-11T03:52:01.879 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.879 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.879 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:52:01.879 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090155+0000", 2024-09-11T03:52:01.879 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1618999999999994e-05 2024-09-11T03:52:01.879 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.879 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.879 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.879 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:42:45.090221+0000", 2024-09-11T03:52:01.879 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6384999999999995e-05 2024-09-11T03:52:01.879 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.880 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.880 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.880 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.880 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.880 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5348 2.1b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e571)", 2024-09-11T03:52:01.880 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:28.996464+0000", 2024-09-11T03:52:01.880 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 332.87052168600002, 2024-09-11T03:52:01.880 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.973168225, 2024-09-11T03:52:01.880 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.880 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.880 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "reached pg", 2024-09-11T03:52:01.880 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.880 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.880 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.881 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5348 2024-09-11T03:52:01.881 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.881 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.881 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.881 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.881 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996464+0000", 2024-09-11T03:52:01.881 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.881 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.881 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.881 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.881 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996464+0000", 2024-09-11T03:52:01.881 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.881 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.881 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.882 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.882 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996465+0000", 2024-09-11T03:52:01.882 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3500000000000002e-07 2024-09-11T03:52:01.882 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.882 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.882 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:01.883 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996467+0000", 2024-09-11T03:52:01.884 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.576e-06 2024-09-11T03:52:01.884 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.884 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.884 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.884 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996475+0000", 2024-09-11T03:52:01.884 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1350000000000003e-06 2024-09-11T03:52:01.884 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.884 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.884 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.884 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996478+0000", 2024-09-11T03:52:01.884 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5240000000000001e-06 2024-09-11T03:52:01.884 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.884 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.885 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.885 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996545+0000", 2024-09-11T03:52:01.885 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7379000000000007e-05 2024-09-11T03:52:01.885 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.885 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.885 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:52:01.885 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996557+0000", 2024-09-11T03:52:01.885 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2019e-05 2024-09-11T03:52:01.885 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.885 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.885 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.885 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969589+0000", 2024-09-11T03:52:01.885 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97303155699999999 2024-09-11T03:52:01.886 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.886 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.886 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.886 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969633+0000", 2024-09-11T03:52:01.886 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3699999999999998e-05 2024-09-11T03:52:01.886 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.886 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.886 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.886 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.886 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.886 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5349 2.1b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e571)", 2024-09-11T03:52:01.886 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:52:01.886 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 332.87049855800001, 2024-09-11T03:52:01.887 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97318775899999999, 2024-09-11T03:52:01.887 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.887 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.887 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "reached pg", 2024-09-11T03:52:01.887 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.887 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.887 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.887 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5349 2024-09-11T03:52:01.887 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.887 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.887 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.887 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.887 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:52:01.888 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.888 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.888 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.888 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.888 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:52:01.888 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.888 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.888 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.888 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.888 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:52:01.888 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7300000000000002e-07 2024-09-11T03:52:01.888 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.888 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.889 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.889 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996490+0000", 2024-09-11T03:52:01.889 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8470000000000001e-06 2024-09-11T03:52:01.889 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.889 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.889 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.889 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996494+0000", 2024-09-11T03:52:01.889 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8419999999999998e-06 2024-09-11T03:52:01.889 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.889 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.889 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.889 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996497+0000", 2024-09-11T03:52:01.889 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8820000000000001e-06 2024-09-11T03:52:01.890 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.890 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.890 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.890 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996602+0000", 2024-09-11T03:52:01.890 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010565699999999999 2024-09-11T03:52:01.890 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.890 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.890 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:52:01.890 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996613+0000", 2024-09-11T03:52:01.890 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0359e-05 2024-09-11T03:52:01.890 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.890 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.890 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969667+0000", 2024-09-11T03:52:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97305483599999998 2024-09-11T03:52:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969675+0000", 2024-09-11T03:52:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9629999999999998e-06 2024-09-11T03:52:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5350 2.1b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e571)", 2024-09-11T03:52:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:52:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 332.87046340199998, 2024-09-11T03:52:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.973193639, 2024-09-11T03:52:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "reached pg", 2024-09-11T03:52:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5350 2024-09-11T03:52:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:52:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:52:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:52:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.27e-07 2024-09-11T03:52:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996525+0000", 2024-09-11T03:52:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.719e-06 2024-09-11T03:52:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996529+0000", 2024-09-11T03:52:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2690000000000001e-06 2024-09-11T03:52:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996532+0000", 2024-09-11T03:52:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4139999999999998e-06 2024-09-11T03:52:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996643+0000", 2024-09-11T03:52:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011041599999999999 2024-09-11T03:52:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:52:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996651+0000", 2024-09-11T03:52:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4309999999999994e-06 2024-09-11T03:52:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969706+0000", 2024-09-11T03:52:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97305510699999997 2024-09-11T03:52:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969716+0000", 2024-09-11T03:52:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9559999999999992e-06 2024-09-11T03:52:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.897 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5348 2.1b 2:dde92fc6:::smithi17557870-41:head [sparse-read 0~1317959,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:52:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973173+0000", 2024-09-11T03:52:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 330.89381322399998, 2024-09-11T03:52:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019341553000000001, 2024-09-11T03:52:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:52:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5348 2024-09-11T03:52:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.898 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973173+0000", 2024-09-11T03:52:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973173+0000", 2024-09-11T03:52:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973175+0000", 2024-09-11T03:52:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.074e-06 2024-09-11T03:52:01.899 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973183+0000", 2024-09-11T03:52:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3010000000000007e-06 2024-09-11T03:52:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973199+0000", 2024-09-11T03:52:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5279000000000001e-05 2024-09-11T03:52:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.900 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973214+0000", 2024-09-11T03:52:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5891000000000001e-05 2024-09-11T03:52:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973747+0000", 2024-09-11T03:52:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00053236600000000005 2024-09-11T03:52:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:52:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973760+0000", 2024-09-11T03:52:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2976000000000001e-05 2024-09-11T03:52:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.901 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.990782+0000", 2024-09-11T03:52:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017022371000000001 2024-09-11T03:52:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:52:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.990945+0000", 2024-09-11T03:52:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016245699999999999 2024-09-11T03:52:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.992514+0000", 2024-09-11T03:52:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001569838 2024-09-11T03:52:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5349 2.1b 2:dde92fc6:::smithi17557870-41:head [read 0~2235463,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:52:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:52:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 330.893748512, 2024-09-11T03:52:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020826846, 2024-09-11T03:52:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:52:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5349 2024-09-11T03:52:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:52:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:52:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:52:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2099999999999996e-07 2024-09-11T03:52:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973242+0000", 2024-09-11T03:52:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0489999999999997e-06 2024-09-11T03:52:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973250+0000", 2024-09-11T03:52:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0700000000000007e-06 2024-09-11T03:52:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973256+0000", 2024-09-11T03:52:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3979999999999998e-06 2024-09-11T03:52:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973794+0000", 2024-09-11T03:52:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00053782700000000005 2024-09-11T03:52:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:52:01.906 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973801+0000", 2024-09-11T03:52:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5299999999999999e-06 2024-09-11T03:52:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.992813+0000", 2024-09-11T03:52:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019011917999999999 2024-09-11T03:52:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:52:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.992923+0000", 2024-09-11T03:52:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000109874 2024-09-11T03:52:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994064+0000", 2024-09-11T03:52:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001141459 2024-09-11T03:52:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5350 2.1b 2:dde92fc6:::smithi17557870-41:head [sparse-read 0~1317959] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:52:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973264+0000", 2024-09-11T03:52:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 330.89372178999997, 2024-09-11T03:52:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021144998000000002, 2024-09-11T03:52:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.908 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:52:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5350 2024-09-11T03:52:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973264+0000", 2024-09-11T03:52:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973264+0000", 2024-09-11T03:52:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973265+0000", 2024-09-11T03:52:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5899999999999996e-07 2024-09-11T03:52:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973268+0000", 2024-09-11T03:52:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6970000000000002e-06 2024-09-11T03:52:01.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973274+0000", 2024-09-11T03:52:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6390000000000003e-06 2024-09-11T03:52:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973280+0000", 2024-09-11T03:52:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.451e-06 2024-09-11T03:52:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973888+0000", 2024-09-11T03:52:01.911 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00060798999999999998 2024-09-11T03:52:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:52:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973895+0000", 2024-09-11T03:52:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6199999999999999e-06 2024-09-11T03:52:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994115+0000", 2024-09-11T03:52:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020219487000000001 2024-09-11T03:52:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.912 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:52:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994236+0000", 2024-09-11T03:52:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000120715 2024-09-11T03:52:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994409+0000", 2024-09-11T03:52:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017384 2024-09-11T03:52:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.913 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5362 2.1b 2:de6b1316:::smithi17557870-32:head [read 0~2117847,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:52:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973757+0000", 2024-09-11T03:52:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 330.89322888100003, 2024-09-11T03:52:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.023415378000000001, 2024-09-11T03:52:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:52:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5362 2024-09-11T03:52:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.914 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973757+0000", 2024-09-11T03:52:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973757+0000", 2024-09-11T03:52:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.915 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973758+0000", 2024-09-11T03:52:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.088e-06 2024-09-11T03:52:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973765+0000", 2024-09-11T03:52:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1310000000000003e-06 2024-09-11T03:52:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973776+0000", 2024-09-11T03:52:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1762e-05 2024-09-11T03:52:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973785+0000", 2024-09-11T03:52:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8629999999999997e-06 2024-09-11T03:52:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973985+0000", 2024-09-11T03:52:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000200051 2024-09-11T03:52:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:52:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973995+0000", 2024-09-11T03:52:01.917 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0054000000000001e-05 2024-09-11T03:52:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994538+0000", 2024-09-11T03:52:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020542878000000001 2024-09-11T03:52:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:52:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994729+0000", 2024-09-11T03:52:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00019097100000000001 2024-09-11T03:52:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.918 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.997173+0000", 2024-09-11T03:52:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024435799999999999 2024-09-11T03:52:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5363 2.1b 2:de6b1316:::smithi17557870-32:head [read 0~2117847,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:52:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973799+0000", 2024-09-11T03:52:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 330.893186872, 2024-09-11T03:52:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.024619513999999999, 2024-09-11T03:52:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.919 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:52:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5363 2024-09-11T03:52:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.920 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973799+0000", 2024-09-11T03:52:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973799+0000", 2024-09-11T03:52:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.921 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973800+0000", 2024-09-11T03:52:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2500000000000005e-07 2024-09-11T03:52:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973804+0000", 2024-09-11T03:52:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6049999999999998e-06 2024-09-11T03:52:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973811+0000", 2024-09-11T03:52:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0350000000000004e-06 2024-09-11T03:52:01.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973816+0000", 2024-09-11T03:52:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.536e-06 2024-09-11T03:52:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974024+0000", 2024-09-11T03:52:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00020766099999999999 2024-09-11T03:52:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:52:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974032+0000", 2024-09-11T03:52:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3340000000000002e-06 2024-09-11T03:52:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.997304+0000", 2024-09-11T03:52:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.02327183 2024-09-11T03:52:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:52:01.924 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.997455+0000", 2024-09-11T03:52:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015147000000000001 2024-09-11T03:52:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.998419+0000", 2024-09-11T03:52:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00096341799999999996 2024-09-11T03:52:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5364 2.1b 2:de6b1316:::smithi17557870-32:head [read 0~2117847,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:52:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:52:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 330.893163152, 2024-09-11T03:52:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.025348697, 2024-09-11T03:52:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:52:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.926 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5364 2024-09-11T03:52:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:52:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:52:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:52:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4099999999999999e-07 2024-09-11T03:52:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973826+0000", 2024-09-11T03:52:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7690000000000001e-06 2024-09-11T03:52:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.928 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973833+0000", 2024-09-11T03:52:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4679999999999997e-06 2024-09-11T03:52:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973837+0000", 2024-09-11T03:52:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6070000000000001e-06 2024-09-11T03:52:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.929 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974062+0000", 2024-09-11T03:52:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00022431400000000001 2024-09-11T03:52:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.930 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:52:01.930 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:01.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974069+0000", 2024-09-11T03:52:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8599999999999993e-06 2024-09-11T03:52:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.998467+0000", 2024-09-11T03:52:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.024397723999999999 2024-09-11T03:52:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:52:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.998581+0000", 2024-09-11T03:52:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000113769 2024-09-11T03:52:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.999172+0000", 2024-09-11T03:52:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00059074500000000005 2024-09-11T03:52:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.933 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5778 2.a 2:5014e305:::smithi17557870-46:head [copy-from ver 878] snapc ff=[ff,fe,fd] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-09-11T03:52:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:06.021580+0000", 2024-09-11T03:52:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 295.84540659200002, 2024-09-11T03:52:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.023197538, 2024-09-11T03:52:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.934 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5778 2024-09-11T03:52:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021580+0000", 2024-09-11T03:52:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021580+0000", 2024-09-11T03:52:01.935 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021581+0000", 2024-09-11T03:52:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6099999999999999e-07 2024-09-11T03:52:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021585+0000", 2024-09-11T03:52:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.279e-06 2024-09-11T03:52:01.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021593+0000", 2024-09-11T03:52:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1999999999999994e-06 2024-09-11T03:52:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.937 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021599+0000", 2024-09-11T03:52:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8200000000000002e-06 2024-09-11T03:52:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021691+0000", 2024-09-11T03:52:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2142e-05 2024-09-11T03:52:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.938 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:52:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021868+0000", 2024-09-11T03:52:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017677200000000001 2024-09-11T03:52:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:52:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.030689+0000", 2024-09-11T03:52:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0088215310000000005 2024-09-11T03:52:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.940 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.038924+0000", 2024-09-11T03:52:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.008234412 2024-09-11T03:52:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044669+0000", 2024-09-11T03:52:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0057455529999999996 2024-09-11T03:52:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:52:01.941 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044696+0000", 2024-09-11T03:52:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6843e-05 2024-09-11T03:52:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:52:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044748+0000", 2024-09-11T03:52:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1711e-05 2024-09-11T03:52:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044777+0000", 2024-09-11T03:52:01.942 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9314000000000001e-05 2024-09-11T03:52:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5983 2.1b 2:de6b1316:::smithi17557870-32:head [copy-from ver 692] snapc 108=[108,107] ondisk+write+known_if_redirected+supports_pool_eio e627)", 2024-09-11T03:52:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:25.010709+0000", 2024-09-11T03:52:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 276.856277492, 2024-09-11T03:52:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022123384999999999, 2024-09-11T03:52:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.943 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5983 2024-09-11T03:52:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010709+0000", 2024-09-11T03:52:01.944 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010709+0000", 2024-09-11T03:52:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010711+0000", 2024-09-11T03:52:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.367e-06 2024-09-11T03:52:01.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010723+0000", 2024-09-11T03:52:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2098e-05 2024-09-11T03:52:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010748+0000", 2024-09-11T03:52:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4641e-05 2024-09-11T03:52:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010761+0000", 2024-09-11T03:52:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.329e-05 2024-09-11T03:52:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010856+0000", 2024-09-11T03:52:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5278000000000005e-05 2024-09-11T03:52:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:52:01.947 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010862+0000", 2024-09-11T03:52:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.088e-06 2024-09-11T03:52:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.012360+0000", 2024-09-11T03:52:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014979940000000001 2024-09-11T03:52:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:52:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.012437+0000", 2024-09-11T03:52:01.948 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6959000000000006e-05 2024-09-11T03:52:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,9", 2024-09-11T03:52:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.020310+0000", 2024-09-11T03:52:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0078724099999999998 2024-09-11T03:52:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:52:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.028691+0000", 2024-09-11T03:52:01.949 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.008381299 2024-09-11T03:52:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.028800+0000", 2024-09-11T03:52:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000109082 2024-09-11T03:52:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.032710+0000", 2024-09-11T03:52:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039093660000000001 2024-09-11T03:52:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:52:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.032764+0000", 2024-09-11T03:52:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4415e-05 2024-09-11T03:52:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.032832+0000", 2024-09-11T03:52:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8097999999999998e-05 2024-09-11T03:52:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.951 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5984 2.1b 2:de6b1316:::smithi17557870-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e627)", 2024-09-11T03:52:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:25.010792+0000", 2024-09-11T03:52:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 276.856194055, 2024-09-11T03:52:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014851705, 2024-09-11T03:52:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:52:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.952 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5984 2024-09-11T03:52:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010792+0000", 2024-09-11T03:52:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.953 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010792+0000", 2024-09-11T03:52:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010793+0000", 2024-09-11T03:52:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8999999999999996e-07 2024-09-11T03:52:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010796+0000", 2024-09-11T03:52:01.954 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6200000000000001e-06 2024-09-11T03:52:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010811+0000", 2024-09-11T03:52:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4620999999999999e-05 2024-09-11T03:52:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010818+0000", 2024-09-11T03:52:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5490000000000003e-06 2024-09-11T03:52:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.955 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010926+0000", 2024-09-11T03:52:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108318 2024-09-11T03:52:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010931+0000", 2024-09-11T03:52:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1669999999999998e-06 2024-09-11T03:52:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.012597+0000", 2024-09-11T03:52:01.956 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001665487 2024-09-11T03:52:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.012646+0000", 2024-09-11T03:52:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9073000000000003e-05 2024-09-11T03:52:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.025501+0000", 2024-09-11T03:52:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012855263 2024-09-11T03:52:01.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:52:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.025569+0000", 2024-09-11T03:52:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8099e-05 2024-09-11T03:52:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.025644+0000", 2024-09-11T03:52:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4818000000000001e-05 2024-09-11T03:52:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.958 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6106 2.21 2:8638b65e:::smithi17557870-5:head [copy-from ver 603] snapc 10c=[10c] ondisk+write+known_if_redirected+supports_pool_eio e636)", 2024-09-11T03:52:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:34.018719+0000", 2024-09-11T03:52:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 267.84826693999997, 2024-09-11T03:52:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016460907, 2024-09-11T03:52:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:01.959 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6106 2024-09-11T03:52:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018719+0000", 2024-09-11T03:52:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018719+0000", 2024-09-11T03:52:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018722+0000", 2024-09-11T03:52:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6730000000000001e-06 2024-09-11T03:52:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.961 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018732+0000", 2024-09-11T03:52:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0349e-05 2024-09-11T03:52:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018756+0000", 2024-09-11T03:52:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3493999999999999e-05 2024-09-11T03:52:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018767+0000", 2024-09-11T03:52:01.962 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.079e-05 2024-09-11T03:52:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018875+0000", 2024-09-11T03:52:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108789 2024-09-11T03:52:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:52:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018884+0000", 2024-09-11T03:52:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4800000000000001e-06 2024-09-11T03:52:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.020397+0000", 2024-09-11T03:52:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015135770000000001 2024-09-11T03:52:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:52:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.020508+0000", 2024-09-11T03:52:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000110351 2024-09-11T03:52:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.964 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,8", 2024-09-11T03:52:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.028802+0000", 2024-09-11T03:52:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0082943189999999997 2024-09-11T03:52:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:52:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.033207+0000", 2024-09-11T03:52:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0044050820000000003 2024-09-11T03:52:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:01.965 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.033684+0000", 2024-09-11T03:52:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00047677899999999998 2024-09-11T03:52:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.035101+0000", 2024-09-11T03:52:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014171539999999999 2024-09-11T03:52:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:52:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.035147+0000", 2024-09-11T03:52:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5731e-05 2024-09-11T03:52:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.966 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.035180+0000", 2024-09-11T03:52:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3339e-05 2024-09-11T03:52:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7115 2.c 2:3603e322:::smithi17557870-42:head [copy-from ver 930] snapc 13b=[13b,13a,139,138,137,136,135,133,12e,12d,12c,12a,127,126,120,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e713)", 2024-09-11T03:52:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:51.194110+0000", 2024-09-11T03:52:01.967 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 190.67287585400001, 2024-09-11T03:52:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.029033742000000001, 2024-09-11T03:52:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7115 2024-09-11T03:52:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.968 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194110+0000", 2024-09-11T03:52:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194110+0000", 2024-09-11T03:52:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.969 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194111+0000", 2024-09-11T03:52:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3200000000000005e-07 2024-09-11T03:52:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194115+0000", 2024-09-11T03:52:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5980000000000001e-06 2024-09-11T03:52:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194121+0000", 2024-09-11T03:52:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3289999999999999e-06 2024-09-11T03:52:01.970 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194126+0000", 2024-09-11T03:52:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4e-06 2024-09-11T03:52:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194207+0000", 2024-09-11T03:52:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0721000000000002e-05 2024-09-11T03:52:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.971 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:52:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194303+0000", 2024-09-11T03:52:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5546999999999998e-05 2024-09-11T03:52:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:52:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.207212+0000", 2024-09-11T03:52:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012909132 2024-09-11T03:52:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.972 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:52:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.209890+0000", 2024-09-11T03:52:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002677876 2024-09-11T03:52:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223023+0000", 2024-09-11T03:52:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013133137 2024-09-11T03:52:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:01.973 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223063+0000", 2024-09-11T03:52:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0413000000000001e-05 2024-09-11T03:52:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:52:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223107+0000", 2024-09-11T03:52:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3569999999999998e-05 2024-09-11T03:52:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223144+0000", 2024-09-11T03:52:01.974 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7387e-05 2024-09-11T03:52:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7116 2.c 2:3603e322:::smithi17557870-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e713)", 2024-09-11T03:52:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:51.194143+0000", 2024-09-11T03:52:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 190.67284328599999, 2024-09-11T03:52:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015378424, 2024-09-11T03:52:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.975 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:52:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7116 2024-09-11T03:52:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194143+0000", 2024-09-11T03:52:01.976 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194143+0000", 2024-09-11T03:52:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194143+0000", 2024-09-11T03:52:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9299999999999998e-07 2024-09-11T03:52:01.977 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194146+0000", 2024-09-11T03:52:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2390000000000001e-06 2024-09-11T03:52:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194152+0000", 2024-09-11T03:52:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5080000000000002e-06 2024-09-11T03:52:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194158+0000", 2024-09-11T03:52:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.942e-06 2024-09-11T03:52:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194461+0000", 2024-09-11T03:52:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00030244899999999999 2024-09-11T03:52:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.979 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194502+0000", 2024-09-11T03:52:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.1736000000000002e-05 2024-09-11T03:52:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.209298+0000", 2024-09-11T03:52:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014795338999999999 2024-09-11T03:52:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:52:01.980 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.209398+0000", 2024-09-11T03:52:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010077300000000001 2024-09-11T03:52:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.209521+0000", 2024-09-11T03:52:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012294500000000001 2024-09-11T03:52:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.981 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:8914 2.21 2:8638b65e:::smithi17557870-5:head [append 0~0] snapc 190=[190,18f,18e,18d,18c,18b,18a,189,188,187,186,185,184,183,181,180,17e,17d,17c,17a,178,177,175,174,172,171,170,16e,16b,169,167,164,15f,15e,15b,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-09-11T03:52:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:51:20.058065+0000", 2024-09-11T03:52:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 41.808921495, 2024-09-11T03:52:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014840628999999999, 2024-09-11T03:52:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 8914 2024-09-11T03:52:01.982 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058065+0000", 2024-09-11T03:52:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058065+0000", 2024-09-11T03:52:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.983 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058066+0000", 2024-09-11T03:52:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7519999999999999e-06 2024-09-11T03:52:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058072+0000", 2024-09-11T03:52:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0220000000000001e-06 2024-09-11T03:52:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.984 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058085+0000", 2024-09-11T03:52:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2065000000000001e-05 2024-09-11T03:52:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058092+0000", 2024-09-11T03:52:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8620000000000003e-06 2024-09-11T03:52:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.985 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.060491+0000", 2024-09-11T03:52:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0023990859999999999 2024-09-11T03:52:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:52:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.060619+0000", 2024-09-11T03:52:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012762900000000001 2024-09-11T03:52:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,8", 2024-09-11T03:52:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.060883+0000", 2024-09-11T03:52:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00026370899999999999 2024-09-11T03:52:01.986 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:52:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.065646+0000", 2024-09-11T03:52:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0047630479999999998 2024-09-11T03:52:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.065740+0000", 2024-09-11T03:52:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4337000000000002e-05 2024-09-11T03:52:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.987 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.072779+0000", 2024-09-11T03:52:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0070389559999999999 2024-09-11T03:52:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:52:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.072850+0000", 2024-09-11T03:52:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1068000000000005e-05 2024-09-11T03:52:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.072905+0000", 2024-09-11T03:52:01.988 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5095e-05 2024-09-11T03:52:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:8915 2.21 2:8638b65e:::smithi17557870-5:head [rollback 13c] snapc 190=[190,18f,18e,18d,18c,18b,18a,189,188,187,186,185,184,183,181,180,17e,17d,17c,17a,178,177,175,174,172,171,170,16e,16b,169,167,164,15f,15e,15b,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-09-11T03:52:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:51:20.058112+0000", 2024-09-11T03:52:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 41.808874234999998, 2024-09-11T03:52:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016574947999999999, 2024-09-11T03:52:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.989 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 8915 2024-09-11T03:52:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058112+0000", 2024-09-11T03:52:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.990 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058112+0000", 2024-09-11T03:52:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058113+0000", 2024-09-11T03:52:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9299999999999998e-07 2024-09-11T03:52:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.991 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058115+0000", 2024-09-11T03:52:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6579999999999998e-06 2024-09-11T03:52:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058123+0000", 2024-09-11T03:52:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3379999999999999e-06 2024-09-11T03:52:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:01.992 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058128+0000", 2024-09-11T03:52:01.993 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6099999999999997e-06 2024-09-11T03:52:01.993 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.993 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.993 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:01.993 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.061715+0000", 2024-09-11T03:52:01.993 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035866280000000001 2024-09-11T03:52:01.993 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.993 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.993 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:52:01.993 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.061820+0000", 2024-09-11T03:52:01.993 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000104991 2024-09-11T03:52:01.994 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.994 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.994 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,8", 2024-09-11T03:52:01.994 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.062058+0000", 2024-09-11T03:52:01.994 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00023820800000000001 2024-09-11T03:52:01.994 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.994 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.994 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:52:01.994 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.065660+0000", 2024-09-11T03:52:01.994 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0036018589999999998 2024-09-11T03:52:01.994 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.994 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.995 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:01.995 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.065776+0000", 2024-09-11T03:52:01.995 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000115973 2024-09-11T03:52:01.995 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.995 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.995 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:01.995 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.073787+0000", 2024-09-11T03:52:01.995 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0080107230000000008 2024-09-11T03:52:01.995 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.995 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.995 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:52:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.074629+0000", 2024-09-11T03:52:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00084232399999999998 2024-09-11T03:52:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.074687+0000", 2024-09-11T03:52:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8043e-05 2024-09-11T03:52:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:8916 2.21 2:8638b65e:::smithi17557870-5:head [append 0~0] snapc 190=[190,18f,18e,18d,18c,18b,18a,189,188,187,186,185,184,183,181,180,17e,17d,17c,17a,178,177,175,174,172,171,170,16e,16b,169,167,164,15f,15e,15b,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-09-11T03:52:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:51:20.058135+0000", 2024-09-11T03:52:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 41.808850726000003, 2024-09-11T03:52:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017146954999999998, 2024-09-11T03:52:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:52:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:52:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:52:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:52:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:01.997 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 8916 2024-09-11T03:52:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:52:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:52:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058135+0000", 2024-09-11T03:52:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:52:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058135+0000", 2024-09-11T03:52:01.998 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:52:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:52:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058136+0000", 2024-09-11T03:52:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.39e-07 2024-09-11T03:52:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:52:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058138+0000", 2024-09-11T03:52:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0650000000000001e-06 2024-09-11T03:52:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:01.999 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:52:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058144+0000", 2024-09-11T03:52:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.682e-06 2024-09-11T03:52:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:52:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058149+0000", 2024-09-11T03:52:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0080000000000002e-06 2024-09-11T03:52:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:52:02.000 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.063104+0000", 2024-09-11T03:52:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0049554940000000004 2024-09-11T03:52:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:52:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.063201+0000", 2024-09-11T03:52:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6933000000000002e-05 2024-09-11T03:52:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,8", 2024-09-11T03:52:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.063390+0000", 2024-09-11T03:52:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00018867499999999999 2024-09-11T03:52:02.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:52:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.065665+0000", 2024-09-11T03:52:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022753869999999998 2024-09-11T03:52:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.066506+0000", 2024-09-11T03:52:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00084052699999999998 2024-09-11T03:52:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.075114+0000", 2024-09-11T03:52:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0086082629999999997 2024-09-11T03:52:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:52:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.075225+0000", 2024-09-11T03:52:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000111588 2024-09-11T03:52:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:52:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:52:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:52:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.075282+0000", 2024-09-11T03:52:02.003 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6894000000000001e-05 2024-09-11T03:52:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:52:02.004 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:52:02.004 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:52:02.032 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:02.052 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:02.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:02.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:02.057 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.6 to 1 (10000) 2024-09-11T03:52:02.061 INFO:tasks.rados.rados.2.smithi175.stdout:3728: done (3 left) 2024-09-11T03:52:02.062 INFO:tasks.rados.rados.2.smithi175.stdout:3730: done (2 left) 2024-09-11T03:52:02.062 INFO:tasks.rados.rados.2.smithi175.stdout:3731: done (1 left) 2024-09-11T03:52:02.062 INFO:tasks.rados.rados.2.smithi175.stdout:3732: done (0 left) 2024-09-11T03:52:02.062 INFO:tasks.rados.rados.2.smithi175.stdout:3733: rollback oid 37 current snap is 418 2024-09-11T03:52:02.062 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 37 to 401 2024-09-11T03:52:02.062 INFO:tasks.rados.rados.2.smithi175.stdout:3734: write oid 22 current snap is 418 2024-09-11T03:52:02.062 INFO:tasks.rados.rados.2.smithi175.stdout:3734: seq_num 1265 ranges {460855=504583,1624898=412110,2539027=572508} 2024-09-11T03:52:02.067 INFO:tasks.rados.rados.2.smithi175.stdout:3734: writing smithi17557870-22 from 460855 to 965438 tid 1 2024-09-11T03:52:02.073 INFO:tasks.rados.rados.2.smithi175.stdout:3734: writing smithi17557870-22 from 1624898 to 2037008 tid 2 2024-09-11T03:52:02.079 INFO:tasks.rados.rados.2.smithi175.stdout:3734: writing smithi17557870-22 from 2539027 to 3111535 tid 3 2024-09-11T03:52:02.079 INFO:tasks.rados.rados.2.smithi175.stdout:3735: copy_from oid 9 from oid 8 current snap is 418 2024-09-11T03:52:02.079 INFO:tasks.rados.rados.2.smithi175.stdout:3733: finishing rollback tid 0 to smithi17557870-37 2024-09-11T03:52:02.080 INFO:tasks.rados.rados.2.smithi175.stdout:3733: finishing rollback tid 1 to smithi17557870-37 2024-09-11T03:52:02.080 INFO:tasks.rados.rados.2.smithi175.stdout:3733: finishing rollback tid 2 to smithi17557870-37 2024-09-11T03:52:02.080 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 1261 (ObjNum 1188 snap 388 seq_num 1188) dirty exists 2024-09-11T03:52:02.080 INFO:tasks.rados.rados.2.smithi175.stdout:3734: finishing write tid 1 to smithi17557870-22 2024-09-11T03:52:02.080 INFO:tasks.rados.rados.2.smithi175.stdout:3734: finishing write tid 2 to smithi17557870-22 2024-09-11T03:52:02.080 INFO:tasks.rados.rados.2.smithi175.stdout:3733: done (2 left) 2024-09-11T03:52:02.080 INFO:tasks.rados.rados.2.smithi175.stdout:3736: write oid 4 current snap is 418 2024-09-11T03:52:02.080 INFO:tasks.rados.rados.2.smithi175.stdout:3736: seq_num 1266 ranges {477662=579832,1683380=555768,2966450=311212} 2024-09-11T03:52:02.084 INFO:tasks.rados.rados.2.smithi175.stdout:3736: writing smithi17557870-4 from 477662 to 1057494 tid 1 2024-09-11T03:52:02.091 INFO:tasks.rados.rados.2.smithi175.stdout:3736: writing smithi17557870-4 from 1683380 to 2239148 tid 2 2024-09-11T03:52:02.097 INFO:tasks.rados.rados.2.smithi175.stdout:3736: writing smithi17557870-4 from 2966450 to 3277662 tid 3 2024-09-11T03:52:02.097 INFO:tasks.rados.rados.2.smithi175.stdout:3737: snap_remove snap 322 2024-09-11T03:52:02.154 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:02.176 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:02.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:02.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:02.278 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:02.301 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:02.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:02.302 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:02.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:02 smithi084 ceph-mon[27254]: pgmap v1509: 107 pgs: 107 active+clean; 729 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 3.6 MiB/s rd, 874 KiB/s wr, 7 op/s 2024-09-11T03:52:02.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:02 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1800716339' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-11T03:52:02.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:02 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:02.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:02 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1800716339' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T03:52:02.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:02 smithi084 ceph-mon[27254]: osdmap e904: 12 total, 12 up, 11 in 2024-09-11T03:52:02.403 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:02.423 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:02.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:02.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:02 smithi175 ceph-mon[30894]: pgmap v1509: 107 pgs: 107 active+clean; 729 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 3.6 MiB/s rd, 874 KiB/s wr, 7 op/s 2024-09-11T03:52:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:02 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1800716339' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-11T03:52:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:02 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:02.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:02 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1800716339' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T03:52:02.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:02 smithi175 ceph-mon[30894]: osdmap e904: 12 total, 12 up, 11 in 2024-09-11T03:52:02.525 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:02.546 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:02.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:02.648 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:02.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:02 smithi121 ceph-mon[30729]: pgmap v1509: 107 pgs: 107 active+clean; 729 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 3.6 MiB/s rd, 874 KiB/s wr, 7 op/s 2024-09-11T03:52:02.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:02 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1800716339' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-11T03:52:02.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:02 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:02.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:02 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1800716339' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T03:52:02.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:02 smithi121 ceph-mon[30729]: osdmap e904: 12 total, 12 up, 11 in 2024-09-11T03:52:02.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:02.693 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:02.693 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:02.794 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:02.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T03:52:02.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:02.817 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:02.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:02.917 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:02.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:02.939 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:02.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:02.989 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T03:52:02.989 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:52:03.040 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:03.062 INFO:tasks.rados.rados.2.smithi175.stdout:3737: done (3 left) 2024-09-11T03:52:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3738: read oid 45 snap 406 2024-09-11T03:52:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3738: expect deleted 2024-09-11T03:52:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3739: read oid 16 snap -1 2024-09-11T03:52:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3739: expect (ObjNum 1218 snap 399 seq_num 1218) 2024-09-11T03:52:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3740: rollback oid 39 current snap is 418 2024-09-11T03:52:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 39 to 380 2024-09-11T03:52:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3734: finishing write tid 3 to smithi17557870-22 2024-09-11T03:52:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3741: snap_create 2024-09-11T03:52:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3734: finishing write tid 4 to smithi17557870-22 2024-09-11T03:52:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3734: finishing write tid 5 to smithi17557870-22 2024-09-11T03:52:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3734: oid 22 updating version 0 to 1335 2024-09-11T03:52:03.063 INFO:tasks.rados.rados.2.smithi175.stdout:3734: oid 22 updating version 1335 to 1337 2024-09-11T03:52:03.064 INFO:tasks.rados.rados.2.smithi175.stdout:3734: oid 22 version 1337 is already newer than 1336 2024-09-11T03:52:03.064 INFO:tasks.rados.rados.2.smithi175.stdout:3734: oid 22 version 1337 is already newer than 1334 2024-09-11T03:52:03.064 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 1337 (ObjNum 1265 snap 418 seq_num 1265) dirty exists 2024-09-11T03:52:03.064 INFO:tasks.rados.rados.2.smithi175.stdout:3734: left oid 22 (ObjNum 1265 snap 418 seq_num 1265) 2024-09-11T03:52:03.064 INFO:tasks.rados.rados.2.smithi175.stdout:3736: finishing write tid 1 to smithi17557870-4 2024-09-11T03:52:03.064 INFO:tasks.rados.rados.2.smithi175.stdout:3735: finishing copy_from racing read to smithi17557870-9 2024-09-11T03:52:03.064 INFO:tasks.rados.rados.2.smithi175.stdout:3736: finishing write tid 2 to smithi17557870-4 2024-09-11T03:52:03.064 INFO:tasks.rados.rados.2.smithi175.stdout:3736: finishing write tid 3 to smithi17557870-4 2024-09-11T03:52:03.064 INFO:tasks.rados.rados.2.smithi175.stdout:3735: finishing copy_from to smithi17557870-9 2024-09-11T03:52:03.064 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 9 v 1195 (ObjNum 1260 snap 414 seq_num 1260) dirty exists 2024-09-11T03:52:03.064 INFO:tasks.rados.rados.2.smithi175.stdout:3736: finishing write tid 4 to smithi17557870-4 2024-09-11T03:52:03.064 INFO:tasks.rados.rados.2.smithi175.stdout:3736: finishing write tid 5 to smithi17557870-4 2024-09-11T03:52:03.065 INFO:tasks.rados.rados.2.smithi175.stdout:3736: oid 4 updating version 0 to 1289 2024-09-11T03:52:03.065 INFO:tasks.rados.rados.2.smithi175.stdout:3736: oid 4 version 1289 is already newer than 1286 2024-09-11T03:52:03.065 INFO:tasks.rados.rados.2.smithi175.stdout:3736: oid 4 version 1289 is already newer than 1288 2024-09-11T03:52:03.065 INFO:tasks.rados.rados.2.smithi175.stdout:3736: oid 4 version 1289 is already newer than 1287 2024-09-11T03:52:03.065 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 1289 (ObjNum 1266 snap 418 seq_num 1266) dirty exists 2024-09-11T03:52:03.065 INFO:tasks.rados.rados.2.smithi175.stdout:3736: left oid 4 (ObjNum 1266 snap 418 seq_num 1266) 2024-09-11T03:52:03.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:03.066 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:03.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:03.071 INFO:tasks.rados.rados.2.smithi175.stdout:3740: finishing rollback tid 0 to smithi17557870-39 2024-09-11T03:52:03.071 INFO:tasks.rados.rados.2.smithi175.stdout:3740: finishing rollback tid 1 to smithi17557870-39 2024-09-11T03:52:03.071 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 1019 (ObjNum 129 snap 0 seq_num 1745415240) dirty dne 2024-09-11T03:52:03.074 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:03.075 INFO:tasks.rados.rados.2.smithi175.stdout:3739: expect (ObjNum 1218 snap 399 seq_num 1218) 2024-09-11T03:52:03.111 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:03.167 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:03.188 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:03.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:03.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:03.290 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:03.312 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:03.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:03.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:03.414 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:03.435 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:03.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:03.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:03.537 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:03.563 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:03.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:03.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:03.665 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:03.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:03.689 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:03.689 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:03.730 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:52:03.730 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:52:03.730 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:52:03.730 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:52:03.790 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:03.812 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:03.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:03.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:03.940 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:03.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:03.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:04.042 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:04.064 INFO:tasks.rados.rados.2.smithi175.stdout:3734: done (6 left) 2024-09-11T03:52:04.064 INFO:tasks.rados.rados.2.smithi175.stdout:3735: done (5 left) 2024-09-11T03:52:04.064 INFO:tasks.rados.rados.2.smithi175.stdout:3736: done (4 left) 2024-09-11T03:52:04.064 INFO:tasks.rados.rados.2.smithi175.stdout:3738: done (3 left) 2024-09-11T03:52:04.065 INFO:tasks.rados.rados.2.smithi175.stdout:3739: done (2 left) 2024-09-11T03:52:04.065 INFO:tasks.rados.rados.2.smithi175.stdout:3740: done (1 left) 2024-09-11T03:52:04.065 INFO:tasks.rados.rados.2.smithi175.stdout:3741: done (0 left) 2024-09-11T03:52:04.065 INFO:tasks.rados.rados.2.smithi175.stdout:3742: snap_create 2024-09-11T03:52:04.066 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:04.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:04.067 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:04.168 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:04.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:04.193 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:04.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:04.291 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:52:04.296 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:04.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:04 smithi084 ceph-mon[27254]: pgmap v1512: 107 pgs: 107 active+clean; 736 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 3.9 MiB/s rd, 1.8 MiB/s wr, 9 op/s 2024-09-11T03:52:04.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:04 smithi084 ceph-mon[27254]: osdmap e905: 12 total, 12 up, 11 in 2024-09-11T03:52:04.318 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:04.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:04.420 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:04.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:04 smithi121 ceph-mon[30729]: pgmap v1512: 107 pgs: 107 active+clean; 736 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 3.9 MiB/s rd, 1.8 MiB/s wr, 9 op/s 2024-09-11T03:52:04.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:04 smithi121 ceph-mon[30729]: osdmap e905: 12 total, 12 up, 11 in 2024-09-11T03:52:04.442 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:04.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:04.447 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-11T03:52:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:04 smithi175 ceph-mon[30894]: pgmap v1512: 107 pgs: 107 active+clean; 736 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 3.9 MiB/s rd, 1.8 MiB/s wr, 9 op/s 2024-09-11T03:52:04.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:04 smithi175 ceph-mon[30894]: osdmap e905: 12 total, 12 up, 11 in 2024-09-11T03:52:04.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:04.570 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:04.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:04.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:04.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:04.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:04.694 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:04.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:04.796 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:04.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:04.818 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:04.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:04.862 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:52:04.863 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:52:04.882 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:52:04.882 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:52:04.886 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:04.886 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:04.903 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:52:04.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:52:04.919 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:04.923 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:52:04.924 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:52:04.925 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:04.943 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:52:04.944 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:52:04.945 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:04.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:04.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:04.961 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:52:04.961 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:52:04.978 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:52:04.979 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:52:04.995 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:52:04.995 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:52:05.012 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:52:05.013 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:52:05.029 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:52:05.030 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:52:05.047 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:05.048 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:52:05.048 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:52:05.065 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:52:05.065 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:52:05.074 INFO:tasks.rados.rados.2.smithi175.stdout:3742: done (0 left) 2024-09-11T03:52:05.074 INFO:tasks.rados.rados.2.smithi175.stdout:3743: copy_from oid 22 from oid 38 current snap is 420 2024-09-11T03:52:05.074 INFO:tasks.rados.rados.2.smithi175.stdout:3744: read oid 27 snap -1 2024-09-11T03:52:05.074 INFO:tasks.rados.rados.2.smithi175.stdout:3744: expect (ObjNum 1258 snap 413 seq_num 1258) 2024-09-11T03:52:05.074 INFO:tasks.rados.rados.2.smithi175.stdout:3745: delete oid 37 current snap is 420 2024-09-11T03:52:05.075 INFO:tasks.rados.rados.2.smithi175.stdout:3743: finishing copy_from to smithi17557870-22 2024-09-11T03:52:05.075 INFO:tasks.rados.rados.2.smithi175.stdout:3743: got expected ENOENT (src dne) 2024-09-11T03:52:05.075 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:05.075 INFO:tasks.rados.rados.2.smithi175.stdout:3743: finishing copy_from racing read to smithi17557870-22 2024-09-11T03:52:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:05.076 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:05.077 INFO:tasks.rados.rados.2.smithi175.stdout:3743: done (2 left) 2024-09-11T03:52:05.077 INFO:tasks.rados.rados.2.smithi175.stdout:3745: done (1 left) 2024-09-11T03:52:05.077 INFO:tasks.rados.rados.2.smithi175.stdout:3746: read oid 46 snap -1 2024-09-11T03:52:05.077 INFO:tasks.rados.rados.2.smithi175.stdout:3746: expect (ObjNum 1243 snap 408 seq_num 1243) 2024-09-11T03:52:05.077 INFO:tasks.rados.rados.2.smithi175.stdout:3747: write oid 5 current snap is 420 2024-09-11T03:52:05.078 INFO:tasks.rados.rados.2.smithi175.stdout:3747: seq_num 1268 ranges {511276=730330,1800344=559437,3021296=289980} 2024-09-11T03:52:05.084 INFO:tasks.rados.rados.2.smithi175.stdout:3747: writing smithi17557870-5 from 511276 to 1241606 tid 1 2024-09-11T03:52:05.090 INFO:tasks.rados.rados.2.smithi175.stdout:3747: writing smithi17557870-5 from 1800344 to 2359781 tid 2 2024-09-11T03:52:05.096 INFO:tasks.rados.rados.2.smithi175.stdout:3747: writing smithi17557870-5 from 3021296 to 3311276 tid 3 2024-09-11T03:52:05.096 INFO:tasks.rados.rados.2.smithi175.stdout:3748: read oid 22 snap -1 2024-09-11T03:52:05.096 INFO:tasks.rados.rados.2.smithi175.stdout:3748: expect (ObjNum 1265 snap 418 seq_num 1265) 2024-09-11T03:52:05.096 INFO:tasks.rados.rados.2.smithi175.stdout:3749: read oid 23 snap -1 2024-09-11T03:52:05.096 INFO:tasks.rados.rados.2.smithi175.stdout:3749: expect (ObjNum 1263 snap 415 seq_num 1263) 2024-09-11T03:52:05.096 INFO:tasks.rados.rados.2.smithi175.stdout:3750: rollback oid 4 current snap is 420 2024-09-11T03:52:05.096 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 4 to 388 2024-09-11T03:52:05.096 INFO:tasks.rados.rados.2.smithi175.stdout:3746: expect (ObjNum 1243 snap 408 seq_num 1243) 2024-09-11T03:52:05.177 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:05.199 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:05.200 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:05.219 INFO:tasks.rados.rados.2.smithi175.stdout:3747: finishing write tid 1 to smithi17557870-5 2024-09-11T03:52:05.219 INFO:tasks.rados.rados.2.smithi175.stdout:3747: finishing write tid 2 to smithi17557870-5 2024-09-11T03:52:05.219 INFO:tasks.rados.rados.2.smithi175.stdout:3744: expect (ObjNum 1258 snap 413 seq_num 1258) 2024-09-11T03:52:05.301 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:05.322 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:05.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:05.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:05.389 INFO:tasks.rados.rados.2.smithi175.stdout:3747: finishing write tid 3 to smithi17557870-5 2024-09-11T03:52:05.389 INFO:tasks.rados.rados.2.smithi175.stdout:3750: finishing rollback tid 0 to smithi17557870-4 2024-09-11T03:52:05.389 INFO:tasks.rados.rados.2.smithi175.stdout:3750: finishing rollback tid 1 to smithi17557870-4 2024-09-11T03:52:05.390 INFO:tasks.rados.rados.2.smithi175.stdout:3747: finishing write tid 4 to smithi17557870-5 2024-09-11T03:52:05.390 INFO:tasks.rados.rados.2.smithi175.stdout:3747: finishing write tid 5 to smithi17557870-5 2024-09-11T03:52:05.390 INFO:tasks.rados.rados.2.smithi175.stdout:3747: oid 5 updating version 0 to 1048 2024-09-11T03:52:05.390 INFO:tasks.rados.rados.2.smithi175.stdout:3747: oid 5 version 1048 is already newer than 1046 2024-09-11T03:52:05.390 INFO:tasks.rados.rados.2.smithi175.stdout:3747: oid 5 version 1048 is already newer than 1047 2024-09-11T03:52:05.390 INFO:tasks.rados.rados.2.smithi175.stdout:3747: oid 5 version 1048 is already newer than 1045 2024-09-11T03:52:05.390 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 1048 (ObjNum 1268 snap 420 seq_num 1268) dirty exists 2024-09-11T03:52:05.390 INFO:tasks.rados.rados.2.smithi175.stdout:3747: left oid 5 (ObjNum 1268 snap 420 seq_num 1268) 2024-09-11T03:52:05.390 INFO:tasks.rados.rados.2.smithi175.stdout:3750: finishing rollback tid 2 to smithi17557870-4 2024-09-11T03:52:05.390 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 1292 (ObjNum 1169 snap 383 seq_num 1169) dirty exists 2024-09-11T03:52:05.390 INFO:tasks.rados.rados.2.smithi175.stdout:3744: done (5 left) 2024-09-11T03:52:05.390 INFO:tasks.rados.rados.2.smithi175.stdout:3746: done (4 left) 2024-09-11T03:52:05.390 INFO:tasks.rados.rados.2.smithi175.stdout:3747: done (3 left) 2024-09-11T03:52:05.390 INFO:tasks.rados.rados.2.smithi175.stdout:3750: done (2 left) 2024-09-11T03:52:05.391 INFO:tasks.rados.rados.2.smithi175.stdout:3751: write (excl) oid 42 current snap is 420 2024-09-11T03:52:05.391 INFO:tasks.rados.rados.2.smithi175.stdout:3751: seq_num 1269 ranges {528083=405579,1342473=419448,2248719=733949} 2024-09-11T03:52:05.394 INFO:tasks.rados.rados.2.smithi175.stdout:3751: writing smithi17557870-42 from 528083 to 933662 tid 1 2024-09-11T03:52:05.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:05 smithi084 ceph-mon[27254]: osdmap e906: 12 total, 12 up, 11 in 2024-09-11T03:52:05.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:05 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3842339644' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:05.398 INFO:tasks.rados.rados.2.smithi175.stdout:3751: writing smithi17557870-42 from 1342473 to 1761921 tid 2 2024-09-11T03:52:05.405 INFO:tasks.rados.rados.2.smithi175.stdout:3751: writing smithi17557870-42 from 2248719 to 2982668 tid 3 2024-09-11T03:52:05.405 INFO:tasks.rados.rados.2.smithi175.stdout:3752: snap_remove snap 390 2024-09-11T03:52:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:05 smithi121 ceph-mon[30729]: osdmap e906: 12 total, 12 up, 11 in 2024-09-11T03:52:05.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:05 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3842339644' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:05.424 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:05.446 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:05.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:05.447 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:05 smithi175 ceph-mon[30894]: osdmap e906: 12 total, 12 up, 11 in 2024-09-11T03:52:05.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:05 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3842339644' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:05.548 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:05.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:05.572 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:05.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:05.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:05.698 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:05.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:05.800 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:05.823 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:05.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:05.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:05.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:05.946 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:05.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:05.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:06.047 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:06.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:06.070 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:06.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:06.081 INFO:tasks.rados.rados.2.smithi175.stdout:3752: done (3 left) 2024-09-11T03:52:06.081 INFO:tasks.rados.rados.2.smithi175.stdout:3753: read oid 44 snap -1 2024-09-11T03:52:06.081 INFO:tasks.rados.rados.2.smithi175.stdout:3753: expect (ObjNum 1261 snap 414 seq_num 1261) 2024-09-11T03:52:06.081 INFO:tasks.rados.rados.2.smithi175.stdout:3754: write (excl) oid 6 current snap is 420 2024-09-11T03:52:06.081 INFO:tasks.rados.rados.2.smithi175.stdout:3754: seq_num 1270 ranges {544890=597181,1917308=563106,3076142=268748} 2024-09-11T03:52:06.087 INFO:tasks.rados.rados.2.smithi175.stdout:3754: writing smithi17557870-6 from 544890 to 1142071 tid 1 2024-09-11T03:52:06.095 INFO:tasks.rados.rados.2.smithi175.stdout:3754: writing smithi17557870-6 from 1917308 to 2480414 tid 2 2024-09-11T03:52:06.100 INFO:tasks.rados.rados.2.smithi175.stdout:3754: writing smithi17557870-6 from 3076142 to 3344890 tid 3 2024-09-11T03:52:06.100 INFO:tasks.rados.rados.2.smithi175.stdout:3755: rollback oid 39 current snap is 420 2024-09-11T03:52:06.100 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 39 to 412 2024-09-11T03:52:06.100 INFO:tasks.rados.rados.2.smithi175.stdout:3749: expect (ObjNum 1263 snap 415 seq_num 1263) 2024-09-11T03:52:06.171 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:06.191 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:06.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:06.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:52:06.211 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:52:06.246 INFO:tasks.rados.rados.2.smithi175.stdout:3748: expect (ObjNum 1265 snap 418 seq_num 1265) 2024-09-11T03:52:06.288 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:06 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:52:06.288 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:06 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:52:06.288 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:06 smithi084 ceph-mon[27254]: pgmap v1515: 107 pgs: 107 active+clean; 736 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 3.9 MiB/s rd, 1.8 MiB/s wr, 9 op/s 2024-09-11T03:52:06.289 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:06 smithi084 ceph-mon[27254]: osdmap e907: 12 total, 12 up, 11 in 2024-09-11T03:52:06.293 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:06.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:06.347 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:06.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:06.392 INFO:tasks.rados.rados.2.smithi175.stdout:3751: finishing write tid 1 to smithi17557870-42 2024-09-11T03:52:06.392 INFO:tasks.rados.rados.2.smithi175.stdout:3751: finishing write tid 2 to smithi17557870-42 2024-09-11T03:52:06.392 INFO:tasks.rados.rados.2.smithi175.stdout:3751: finishing write tid 3 to smithi17557870-42 2024-09-11T03:52:06.392 INFO:tasks.rados.rados.2.smithi175.stdout:3751: finishing write tid 4 to smithi17557870-42 2024-09-11T03:52:06.392 INFO:tasks.rados.rados.2.smithi175.stdout:3751: finishing write tid 5 to smithi17557870-42 2024-09-11T03:52:06.392 INFO:tasks.rados.rados.2.smithi175.stdout:3751: oid 42 updating version 0 to 1073 2024-09-11T03:52:06.392 INFO:tasks.rados.rados.2.smithi175.stdout:3751: oid 42 updating version 1073 to 1075 2024-09-11T03:52:06.392 INFO:tasks.rados.rados.2.smithi175.stdout:3751: oid 42 version 1075 is already newer than 1072 2024-09-11T03:52:06.392 INFO:tasks.rados.rados.2.smithi175.stdout:3751: oid 42 version 1075 is already newer than 1074 2024-09-11T03:52:06.392 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 1075 (ObjNum 1269 snap 420 seq_num 1269) dirty exists 2024-09-11T03:52:06.392 INFO:tasks.rados.rados.2.smithi175.stdout:3751: left oid 42 (ObjNum 1269 snap 420 seq_num 1269) 2024-09-11T03:52:06.393 INFO:tasks.rados.rados.2.smithi175.stdout:3754: finishing write tid 1 to smithi17557870-6 2024-09-11T03:52:06.393 INFO:tasks.rados.rados.2.smithi175.stdout:3754: finishing write tid 2 to smithi17557870-6 2024-09-11T03:52:06.393 INFO:tasks.rados.rados.2.smithi175.stdout:3748: done (5 left) 2024-09-11T03:52:06.393 INFO:tasks.rados.rados.2.smithi175.stdout:3749: done (4 left) 2024-09-11T03:52:06.393 INFO:tasks.rados.rados.2.smithi175.stdout:3751: done (3 left) 2024-09-11T03:52:06.393 INFO:tasks.rados.rados.2.smithi175.stdout:3756: copy_from oid 19 from oid 30 current snap is 420 2024-09-11T03:52:06.393 INFO:tasks.rados.rados.2.smithi175.stdout:3755: finishing rollback tid 1 to smithi17557870-39 2024-09-11T03:52:06.393 INFO:tasks.rados.rados.2.smithi175.stdout:3755: finishing rollback tid 2 to smithi17557870-39 2024-09-11T03:52:06.393 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 1021 (ObjNum 1249 snap 411 seq_num 1249) dirty exists 2024-09-11T03:52:06.393 INFO:tasks.rados.rados.2.smithi175.stdout:3754: finishing write tid 3 to smithi17557870-6 2024-09-11T03:52:06.393 INFO:tasks.rados.rados.2.smithi175.stdout:3754: finishing write tid 4 to smithi17557870-6 2024-09-11T03:52:06.393 INFO:tasks.rados.rados.2.smithi175.stdout:3754: finishing write tid 5 to smithi17557870-6 2024-09-11T03:52:06.393 INFO:tasks.rados.rados.2.smithi175.stdout:3754: oid 6 updating version 0 to 990 2024-09-11T03:52:06.393 INFO:tasks.rados.rados.2.smithi175.stdout:3754: oid 6 updating version 990 to 991 2024-09-11T03:52:06.394 INFO:tasks.rados.rados.2.smithi175.stdout:3754: oid 6 updating version 991 to 992 2024-09-11T03:52:06.394 INFO:tasks.rados.rados.2.smithi175.stdout:3754: oid 6 updating version 992 to 993 2024-09-11T03:52:06.394 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 993 (ObjNum 1270 snap 420 seq_num 1270) dirty exists 2024-09-11T03:52:06.394 INFO:tasks.rados.rados.2.smithi175.stdout:3754: left oid 6 (ObjNum 1270 snap 420 seq_num 1270) 2024-09-11T03:52:06.394 INFO:tasks.rados.rados.2.smithi175.stdout:3753: expect (ObjNum 1261 snap 414 seq_num 1261) 2024-09-11T03:52:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:06 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:52:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:06 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:52:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:06 smithi121 ceph-mon[30729]: pgmap v1515: 107 pgs: 107 active+clean; 736 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 3.9 MiB/s rd, 1.8 MiB/s wr, 9 op/s 2024-09-11T03:52:06.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:06 smithi121 ceph-mon[30729]: osdmap e907: 12 total, 12 up, 11 in 2024-09-11T03:52:06.448 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:06.474 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:06.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:06.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:06.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:06 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:52:06.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:06 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:52:06.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:06 smithi175 ceph-mon[30894]: pgmap v1515: 107 pgs: 107 active+clean; 736 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 3.9 MiB/s rd, 1.8 MiB/s wr, 9 op/s 2024-09-11T03:52:06.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:06 smithi175 ceph-mon[30894]: osdmap e907: 12 total, 12 up, 11 in 2024-09-11T03:52:06.576 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:06.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:06.597 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:06.598 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:06.613 INFO:tasks.rados.rados.2.smithi175.stdout:3756: finishing copy_from racing read to smithi17557870-19 2024-09-11T03:52:06.613 INFO:tasks.rados.rados.2.smithi175.stdout:3756: finishing copy_from to smithi17557870-19 2024-09-11T03:52:06.613 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 1186 (ObjNum 1255 snap 412 seq_num 1255) dirty exists 2024-09-11T03:52:06.613 INFO:tasks.rados.rados.2.smithi175.stdout:3753: done (3 left) 2024-09-11T03:52:06.613 INFO:tasks.rados.rados.2.smithi175.stdout:3754: done (2 left) 2024-09-11T03:52:06.614 INFO:tasks.rados.rados.2.smithi175.stdout:3755: done (1 left) 2024-09-11T03:52:06.614 INFO:tasks.rados.rados.2.smithi175.stdout:3756: done (0 left) 2024-09-11T03:52:06.614 INFO:tasks.rados.rados.2.smithi175.stdout:3757: read oid 17 snap -1 2024-09-11T03:52:06.614 INFO:tasks.rados.rados.2.smithi175.stdout:3757: expect (ObjNum 1181 snap 385 seq_num 1181) 2024-09-11T03:52:06.614 INFO:tasks.rados.rados.2.smithi175.stdout:3758: read oid 27 snap -1 2024-09-11T03:52:06.614 INFO:tasks.rados.rados.2.smithi175.stdout:3758: expect (ObjNum 1258 snap 413 seq_num 1258) 2024-09-11T03:52:06.614 INFO:tasks.rados.rados.2.smithi175.stdout:3759: delete oid 4 current snap is 420 2024-09-11T03:52:06.617 INFO:tasks.rados.rados.2.smithi175.stdout:3759: done (2 left) 2024-09-11T03:52:06.617 INFO:tasks.rados.rados.2.smithi175.stdout:3760: rollback oid 19 current snap is 420 2024-09-11T03:52:06.618 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 19 to 374 2024-09-11T03:52:06.618 INFO:tasks.rados.rados.2.smithi175.stdout:3761: read oid 10 snap 402 2024-09-11T03:52:06.618 INFO:tasks.rados.rados.2.smithi175.stdout:3761: expect (ObjNum 1075 snap 350 seq_num 1075) 2024-09-11T03:52:06.618 INFO:tasks.rados.rados.2.smithi175.stdout:3762: read oid 21 snap -1 2024-09-11T03:52:06.618 INFO:tasks.rados.rados.2.smithi175.stdout:3762: expect (ObjNum 829 snap 263 seq_num 829) 2024-09-11T03:52:06.618 INFO:tasks.rados.rados.2.smithi175.stdout:3763: snap_create 2024-09-11T03:52:06.620 INFO:tasks.rados.rados.2.smithi175.stdout:3760: finishing rollback tid 0 to smithi17557870-19 2024-09-11T03:52:06.620 INFO:tasks.rados.rados.2.smithi175.stdout:3760: finishing rollback tid 1 to smithi17557870-19 2024-09-11T03:52:06.620 INFO:tasks.rados.rados.2.smithi175.stdout:3760: finishing rollback tid 2 to smithi17557870-19 2024-09-11T03:52:06.621 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 1189 (ObjNum 1079 snap 352 seq_num 1079) dirty exists 2024-09-11T03:52:06.644 INFO:tasks.rados.rados.2.smithi175.stdout:3757: expect (ObjNum 1181 snap 385 seq_num 1181) 2024-09-11T03:52:06.698 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:06.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:06.720 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:06.720 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:06.768 INFO:tasks.rados.rados.2.smithi175.stdout:3762: expect (ObjNum 829 snap 263 seq_num 829) 2024-09-11T03:52:06.821 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:06.850 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:06.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:06.945 INFO:tasks.rados.rados.2.smithi175.stdout:3758: expect (ObjNum 1258 snap 413 seq_num 1258) 2024-09-11T03:52:06.951 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:06.971 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:06.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:06.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:07.059 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:52:07.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:07.099 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:07.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:07.115 INFO:tasks.rados.rados.2.smithi175.stdout:3761: expect (ObjNum 1075 snap 350 seq_num 1075) 2024-09-11T03:52:07.197 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1517,"stamp":"2024-09-11T03:52:06.076422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":774458885,"num_objects":326,"num_object_clones":274,"num_object_copies":978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":10,"num_read":931,"num_read_kb":969159,"num_write":654,"num_write_kb":267459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":82584822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50461,"ondisk_log_size":50461,"up":321,"acting":321,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":321,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":1957516,"kb_used_data":1350640,"kb_used_omap":17,"kb_used_meta":606574,"kb_avail":1029239156,"statfs":{"total":1055945392128,"available":1053940895744,"internally_reserved":0,"allocated":1383055360,"data_stored":1731142610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17546,"internal_metadata":621132662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7372354,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":10128,"num_write":13,"num_write_kb":5973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014531"},"pg_stats":[{"pgid":"2.4b","version":"568'646","reported_seq":4740,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.076040+0000","last_change":"2024-09-11T03:52:01.076814+0000","last_active":"2024-09-11T03:52:01.076040+0000","last_peered":"2024-09-11T03:52:01.076040+0000","last_clean":"2024-09-11T03:52:01.076040+0000","last_became_active":"2024-09-11T03:51:13.079045+0000","last_became_peered":"2024-09-11T03:51:13.079045+0000","last_unstale":"2024-09-11T03:52:01.076040+0000","last_undegraded":"2024-09-11T03:52:01.076040+0000","last_fullsized":"2024-09-11T03:52:01.076040+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:51:21.370186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.370186+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:23.719980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059362900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.4a","version":"848'386","reported_seq":4488,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.092289+0000","last_change":"2024-09-11T03:52:01.070296+0000","last_active":"2024-09-11T03:52:02.092289+0000","last_peered":"2024-09-11T03:52:02.092289+0000","last_clean":"2024-09-11T03:52:02.092289+0000","last_became_active":"2024-09-11T03:51:13.080958+0000","last_became_peered":"2024-09-11T03:51:13.080958+0000","last_unstale":"2024-09-11T03:52:02.092289+0000","last_undegraded":"2024-09-11T03:52:02.092289+0000","last_fullsized":"2024-09-11T03:52:02.092289+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"848'386","last_scrub_stamp":"2024-09-11T03:51:19.939163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.939163+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:37.086321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011636610000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.49","version":"903'1520","reported_seq":5792,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.089043+0000","last_change":"2024-09-11T03:52:01.089225+0000","last_active":"2024-09-11T03:52:01.089043+0000","last_peered":"2024-09-11T03:52:01.089043+0000","last_clean":"2024-09-11T03:52:01.089043+0000","last_became_active":"2024-09-11T03:51:20.066997+0000","last_became_peered":"2024-09-11T03:51:20.066997+0000","last_unstale":"2024-09-11T03:52:01.089043+0000","last_undegraded":"2024-09-11T03:52:01.089043+0000","last_fullsized":"2024-09-11T03:52:01.089043+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":7,"last_scrub":"863'1411","last_scrub_stamp":"2024-09-11T03:51:21.422037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.422037+0000","objects_scrubbed":18,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:49.507005+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0083180679999999996,"stat_sum":{"num_bytes":47711785,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":30,"num_read_kb":44216,"num_write":11,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.48","version":"744'374","reported_seq":3974,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-09-11T03:52:03.075765+0000","last_change":"2024-09-11T03:52:03.076201+0000","last_active":"2024-09-11T03:52:03.075765+0000","last_peered":"2024-09-11T03:52:03.075765+0000","last_clean":"2024-09-11T03:52:03.075765+0000","last_became_active":"2024-09-11T03:51:13.082676+0000","last_became_peered":"2024-09-11T03:51:13.082676+0000","last_unstale":"2024-09-11T03:52:03.075765+0000","last_undegraded":"2024-09-11T03:52:03.075765+0000","last_fullsized":"2024-09-11T03:52:03.075765+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"744'374","last_scrub_stamp":"2024-09-11T03:51:19.306681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.306681+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:42.992469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0052177639999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,0],"acting":[9,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.47","version":"905'1316","reported_seq":5592,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.071977+0000","last_change":"2024-09-11T03:52:03.083110+0000","last_active":"2024-09-11T03:52:05.071977+0000","last_peered":"2024-09-11T03:52:05.071977+0000","last_clean":"2024-09-11T03:52:05.071977+0000","last_became_active":"2024-09-11T03:51:20.064024+0000","last_became_peered":"2024-09-11T03:51:20.064024+0000","last_unstale":"2024-09-11T03:52:05.071977+0000","last_undegraded":"2024-09-11T03:52:05.071977+0000","last_fullsized":"2024-09-11T03:52:05.071977+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":7,"last_scrub":"862'1274","last_scrub_stamp":"2024-09-11T03:51:20.166795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.166795+0000","objects_scrubbed":7,"log_size":1316,"log_dups_size":0,"ondisk_log_size":1316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:33.975312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0067661989999999997,"stat_sum":{"num_bytes":19286976,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":8,"num_read_kb":5764,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.46","version":"269'171","reported_seq":4215,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.079716+0000","last_change":"2024-09-11T03:52:01.079991+0000","last_active":"2024-09-11T03:52:01.079716+0000","last_peered":"2024-09-11T03:52:01.079716+0000","last_clean":"2024-09-11T03:52:01.079716+0000","last_became_active":"2024-09-11T03:51:13.080822+0000","last_became_peered":"2024-09-11T03:51:13.080822+0000","last_unstale":"2024-09-11T03:52:01.079716+0000","last_undegraded":"2024-09-11T03:52:01.079716+0000","last_fullsized":"2024-09-11T03:52:01.079716+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:51:21.034557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.034557+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:39.796577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031510739999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.45","version":"903'1236","reported_seq":5339,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.085282+0000","last_change":"2024-09-11T03:52:01.085528+0000","last_active":"2024-09-11T03:52:01.085282+0000","last_peered":"2024-09-11T03:52:01.085282+0000","last_clean":"2024-09-11T03:52:01.085282+0000","last_became_active":"2024-09-11T03:51:13.078704+0000","last_became_peered":"2024-09-11T03:51:13.078704+0000","last_unstale":"2024-09-11T03:52:01.085282+0000","last_undegraded":"2024-09-11T03:52:01.085282+0000","last_fullsized":"2024-09-11T03:52:01.085282+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"863'1213","last_scrub_stamp":"2024-09-11T03:51:23.430916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.430916+0000","objects_scrubbed":4,"log_size":1236,"log_dups_size":0,"ondisk_log_size":1236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:52.703619+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.013917193,"stat_sum":{"num_bytes":13996135,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":6,"num_write_kb":1354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[1,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.44","version":"903'754","reported_seq":4969,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.081070+0000","last_change":"2024-09-11T03:52:01.081422+0000","last_active":"2024-09-11T03:52:01.081070+0000","last_peered":"2024-09-11T03:52:01.081070+0000","last_clean":"2024-09-11T03:52:01.081070+0000","last_became_active":"2024-09-11T03:51:13.079319+0000","last_became_peered":"2024-09-11T03:51:13.079319+0000","last_unstale":"2024-09-11T03:52:01.081070+0000","last_undegraded":"2024-09-11T03:52:01.081070+0000","last_fullsized":"2024-09-11T03:52:01.081070+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"862'706","last_scrub_stamp":"2024-09-11T03:51:20.328187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.328187+0000","objects_scrubbed":12,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:38.128722+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.010773814,"stat_sum":{"num_bytes":25224871,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":8,"num_write_kb":7615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,3],"acting":[1,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.43","version":"903'616","reported_seq":4740,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.080129+0000","last_change":"2024-09-11T03:52:01.080414+0000","last_active":"2024-09-11T03:52:01.080129+0000","last_peered":"2024-09-11T03:52:01.080129+0000","last_clean":"2024-09-11T03:52:01.080129+0000","last_became_active":"2024-09-11T03:51:13.083077+0000","last_became_peered":"2024-09-11T03:51:13.083077+0000","last_unstale":"2024-09-11T03:52:01.080129+0000","last_undegraded":"2024-09-11T03:52:01.080129+0000","last_fullsized":"2024-09-11T03:52:01.080129+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"862'574","last_scrub_stamp":"2024-09-11T03:51:22.027988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.027988+0000","objects_scrubbed":9,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:40.828580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0040077400000000001,"stat_sum":{"num_bytes":26795402,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":11849,"num_write":8,"num_write_kb":11222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.42","version":"903'333","reported_seq":4439,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.068529+0000","last_change":"2024-09-11T03:52:01.068761+0000","last_active":"2024-09-11T03:52:01.068529+0000","last_peered":"2024-09-11T03:52:01.068529+0000","last_clean":"2024-09-11T03:52:01.068529+0000","last_became_active":"2024-09-11T03:51:13.080914+0000","last_became_peered":"2024-09-11T03:51:13.080914+0000","last_unstale":"2024-09-11T03:52:01.068529+0000","last_undegraded":"2024-09-11T03:52:01.068529+0000","last_fullsized":"2024-09-11T03:52:01.068529+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"859'323","last_scrub_stamp":"2024-09-11T03:51:19.628572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.628572+0000","objects_scrubbed":5,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:29.252385+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0031679289999999999,"stat_sum":{"num_bytes":15883770,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,11],"acting":[8,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.41","version":"567'235","reported_seq":4254,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.091875+0000","last_change":"2024-09-11T03:52:01.068915+0000","last_active":"2024-09-11T03:52:02.091875+0000","last_peered":"2024-09-11T03:52:02.091875+0000","last_clean":"2024-09-11T03:52:02.091875+0000","last_became_active":"2024-09-11T03:50:33.046488+0000","last_became_peered":"2024-09-11T03:50:33.046488+0000","last_unstale":"2024-09-11T03:52:02.091875+0000","last_undegraded":"2024-09-11T03:52:02.091875+0000","last_fullsized":"2024-09-11T03:52:02.091875+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":7,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:50:35.786143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.786143+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000663278,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.40","version":"563'192","reported_seq":4323,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.070722+0000","last_change":"2024-09-11T03:52:01.074040+0000","last_active":"2024-09-11T03:52:02.070722+0000","last_peered":"2024-09-11T03:52:02.070722+0000","last_clean":"2024-09-11T03:52:02.070722+0000","last_became_active":"2024-09-11T03:51:20.064866+0000","last_became_peered":"2024-09-11T03:51:20.064866+0000","last_unstale":"2024-09-11T03:52:02.070722+0000","last_undegraded":"2024-09-11T03:52:02.070722+0000","last_fullsized":"2024-09-11T03:52:02.070722+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":7,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:50:35.472175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.472175+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:02.547150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013247020000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,8],"acting":[6,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.3f","version":"903'807","reported_seq":4871,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.083273+0000","last_change":"2024-09-11T03:52:01.083561+0000","last_active":"2024-09-11T03:52:01.083273+0000","last_peered":"2024-09-11T03:52:01.083273+0000","last_clean":"2024-09-11T03:52:01.083273+0000","last_became_active":"2024-09-11T03:50:33.046798+0000","last_became_peered":"2024-09-11T03:50:33.046798+0000","last_unstale":"2024-09-11T03:52:01.083273+0000","last_undegraded":"2024-09-11T03:52:01.083273+0000","last_fullsized":"2024-09-11T03:52:01.083273+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"894'803","last_scrub_stamp":"2024-09-11T03:51:53.038706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.038706+0000","objects_scrubbed":5,"log_size":807,"log_dups_size":0,"ondisk_log_size":807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:02.398487+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0066143130000000001,"stat_sum":{"num_bytes":11068204,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":5,"num_write_kb":1340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.3e","version":"898'1100","reported_seq":5128,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.064906+0000","last_change":"2024-09-11T03:52:01.083087+0000","last_active":"2024-09-11T03:52:02.064906+0000","last_peered":"2024-09-11T03:52:02.064906+0000","last_clean":"2024-09-11T03:52:02.064906+0000","last_became_active":"2024-09-11T03:50:33.045235+0000","last_became_peered":"2024-09-11T03:50:33.045235+0000","last_unstale":"2024-09-11T03:52:02.064906+0000","last_undegraded":"2024-09-11T03:52:02.064906+0000","last_fullsized":"2024-09-11T03:52:02.064906+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"893'1098","last_scrub_stamp":"2024-09-11T03:51:52.144830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.144830+0000","objects_scrubbed":7,"log_size":1100,"log_dups_size":0,"ondisk_log_size":1100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:19.698031+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.011539013000000001,"stat_sum":{"num_bytes":16485077,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.3d","version":"567'752","reported_seq":3601,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.082527+0000","last_change":"2024-09-11T03:52:01.084031+0000","last_active":"2024-09-11T03:52:01.082527+0000","last_peered":"2024-09-11T03:52:01.082527+0000","last_clean":"2024-09-11T03:52:01.082527+0000","last_became_active":"2024-09-11T03:51:20.070161+0000","last_became_peered":"2024-09-11T03:51:20.070161+0000","last_unstale":"2024-09-11T03:52:01.082527+0000","last_undegraded":"2024-09-11T03:52:01.082527+0000","last_fullsized":"2024-09-11T03:52:01.082527+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:51:53.597428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.597428+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:53.825175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.011113557,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.3c","version":"894'789","reported_seq":4868,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.083652+0000","last_change":"2024-09-11T03:52:01.084138+0000","last_active":"2024-09-11T03:52:01.083652+0000","last_peered":"2024-09-11T03:52:01.083652+0000","last_clean":"2024-09-11T03:52:01.083652+0000","last_became_active":"2024-09-11T03:50:33.048487+0000","last_became_peered":"2024-09-11T03:50:33.048487+0000","last_unstale":"2024-09-11T03:52:01.083652+0000","last_undegraded":"2024-09-11T03:52:01.083652+0000","last_fullsized":"2024-09-11T03:52:01.083652+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"886'783","last_scrub_stamp":"2024-09-11T03:51:45.009597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:45.009597+0000","objects_scrubbed":3,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:11.010658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0070972470000000001,"stat_sum":{"num_bytes":10896744,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":3899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,10],"acting":[7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.3b","version":"903'968","reported_seq":5252,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.085155+0000","last_change":"2024-09-11T03:52:01.085472+0000","last_active":"2024-09-11T03:52:01.085155+0000","last_peered":"2024-09-11T03:52:01.085155+0000","last_clean":"2024-09-11T03:52:01.085155+0000","last_became_active":"2024-09-11T03:51:20.066390+0000","last_became_peered":"2024-09-11T03:51:20.066390+0000","last_unstale":"2024-09-11T03:52:01.085155+0000","last_undegraded":"2024-09-11T03:52:01.085155+0000","last_fullsized":"2024-09-11T03:52:01.085155+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"817'886","last_scrub_stamp":"2024-09-11T03:50:35.496439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.496439+0000","objects_scrubbed":8,"log_size":968,"log_dups_size":0,"ondisk_log_size":968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:04.948203+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0068476250000000004,"stat_sum":{"num_bytes":23022362,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":39,"num_read_kb":50517,"num_write":18,"num_write_kb":2369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":14555611,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.3a","version":"557'273","reported_seq":3100,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-09-11T03:52:03.074981+0000","last_change":"2024-09-11T03:52:03.075697+0000","last_active":"2024-09-11T03:52:03.074981+0000","last_peered":"2024-09-11T03:52:03.074981+0000","last_clean":"2024-09-11T03:52:03.074981+0000","last_became_active":"2024-09-11T03:51:20.061387+0000","last_became_peered":"2024-09-11T03:51:20.061387+0000","last_unstale":"2024-09-11T03:52:03.074981+0000","last_undegraded":"2024-09-11T03:52:03.074981+0000","last_fullsized":"2024-09-11T03:52:03.074981+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:50:37.286055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.286055+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0069201790000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.39","version":"806'585","reported_seq":4705,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-09-11T03:52:03.068658+0000","last_change":"2024-09-11T03:52:03.068974+0000","last_active":"2024-09-11T03:52:03.068658+0000","last_peered":"2024-09-11T03:52:03.068658+0000","last_clean":"2024-09-11T03:52:03.068658+0000","last_became_active":"2024-09-11T03:50:33.043651+0000","last_became_peered":"2024-09-11T03:50:33.043651+0000","last_unstale":"2024-09-11T03:52:03.068658+0000","last_undegraded":"2024-09-11T03:52:03.068658+0000","last_fullsized":"2024-09-11T03:52:03.068658+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"806'585","last_scrub_stamp":"2024-09-11T03:50:36.264989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.264989+0000","objects_scrubbed":0,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00087484100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,2],"acting":[9,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.38","version":"808'564","reported_seq":4691,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.065680+0000","last_change":"2024-09-11T03:52:01.066024+0000","last_active":"2024-09-11T03:52:01.065680+0000","last_peered":"2024-09-11T03:52:01.065680+0000","last_clean":"2024-09-11T03:52:01.065680+0000","last_became_active":"2024-09-11T03:50:33.042740+0000","last_became_peered":"2024-09-11T03:50:33.042740+0000","last_unstale":"2024-09-11T03:52:01.065680+0000","last_undegraded":"2024-09-11T03:52:01.065680+0000","last_fullsized":"2024-09-11T03:52:01.065680+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"808'564","last_scrub_stamp":"2024-09-11T03:51:54.462104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.462104+0000","objects_scrubbed":0,"log_size":564,"log_dups_size":0,"ondisk_log_size":564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:00.911382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000611599,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,1],"acting":[8,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.37","version":"274'312","reported_seq":2667,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.092510+0000","last_change":"2024-09-11T03:52:01.071270+0000","last_active":"2024-09-11T03:52:02.092510+0000","last_peered":"2024-09-11T03:52:02.092510+0000","last_clean":"2024-09-11T03:52:02.092510+0000","last_became_active":"2024-09-11T03:50:07.077607+0000","last_became_peered":"2024-09-11T03:50:07.077607+0000","last_unstale":"2024-09-11T03:52:02.092510+0000","last_undegraded":"2024-09-11T03:52:02.092510+0000","last_fullsized":"2024-09-11T03:52:02.092510+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:51:23.036386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.036386+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:23.897168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0017961349999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,9],"acting":[10,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.36","version":"903'702","reported_seq":4824,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.075532+0000","last_change":"2024-09-11T03:52:01.076044+0000","last_active":"2024-09-11T03:52:01.075532+0000","last_peered":"2024-09-11T03:52:01.075532+0000","last_clean":"2024-09-11T03:52:01.075532+0000","last_became_active":"2024-09-11T03:50:07.078667+0000","last_became_peered":"2024-09-11T03:50:07.078667+0000","last_unstale":"2024-09-11T03:52:01.075532+0000","last_undegraded":"2024-09-11T03:52:01.075532+0000","last_fullsized":"2024-09-11T03:52:01.075532+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"885'670","last_scrub_stamp":"2024-09-11T03:51:43.529996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:43.529996+0000","objects_scrubbed":15,"log_size":702,"log_dups_size":0,"ondisk_log_size":702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:09.592668+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0056741739999999997,"stat_sum":{"num_bytes":30606332,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":40,"num_read_kb":41151,"num_write":8,"num_write_kb":6373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,8],"acting":[1,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.35","version":"274'273","reported_seq":1834,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.084092+0000","last_change":"2024-09-11T03:52:01.084578+0000","last_active":"2024-09-11T03:52:01.084092+0000","last_peered":"2024-09-11T03:52:01.084092+0000","last_clean":"2024-09-11T03:52:01.084092+0000","last_became_active":"2024-09-11T03:50:07.079331+0000","last_became_peered":"2024-09-11T03:50:07.079331+0000","last_unstale":"2024-09-11T03:52:01.084092+0000","last_undegraded":"2024-09-11T03:52:01.084092+0000","last_fullsized":"2024-09-11T03:52:01.084092+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:51:28.526090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.526090+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:47.591941+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.012364257,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.34","version":"903'400","reported_seq":4466,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.085093+0000","last_change":"2024-09-11T03:52:01.085359+0000","last_active":"2024-09-11T03:52:01.085093+0000","last_peered":"2024-09-11T03:52:01.085093+0000","last_clean":"2024-09-11T03:52:01.085093+0000","last_became_active":"2024-09-11T03:50:07.080419+0000","last_became_peered":"2024-09-11T03:50:07.080419+0000","last_unstale":"2024-09-11T03:52:01.085093+0000","last_undegraded":"2024-09-11T03:52:01.085093+0000","last_fullsized":"2024-09-11T03:52:01.085093+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"863'388","last_scrub_stamp":"2024-09-11T03:51:22.429302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.429302+0000","objects_scrubbed":6,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:25.559585+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0067349339999999997,"stat_sum":{"num_bytes":13930425,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":37367,"num_write":8,"num_write_kb":1518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.33","version":"274'111","reported_seq":1622,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.078131+0000","last_change":"2024-09-11T03:52:01.079932+0000","last_active":"2024-09-11T03:52:01.078131+0000","last_peered":"2024-09-11T03:52:01.078131+0000","last_clean":"2024-09-11T03:52:01.078131+0000","last_became_active":"2024-09-11T03:50:07.077684+0000","last_became_peered":"2024-09-11T03:50:07.077684+0000","last_unstale":"2024-09-11T03:52:01.078131+0000","last_undegraded":"2024-09-11T03:52:01.078131+0000","last_fullsized":"2024-09-11T03:52:01.078131+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:51:28.001189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.001189+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:52.076727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0030608599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.32","version":"903'588","reported_seq":4684,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.064896+0000","last_change":"2024-09-11T03:52:01.082655+0000","last_active":"2024-09-11T03:52:02.064896+0000","last_peered":"2024-09-11T03:52:02.064896+0000","last_clean":"2024-09-11T03:52:02.064896+0000","last_became_active":"2024-09-11T03:51:20.067057+0000","last_became_peered":"2024-09-11T03:51:20.067057+0000","last_unstale":"2024-09-11T03:52:02.064896+0000","last_undegraded":"2024-09-11T03:52:02.064896+0000","last_fullsized":"2024-09-11T03:52:02.064896+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"885'574","last_scrub_stamp":"2024-09-11T03:51:44.094173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:44.094173+0000","objects_scrubbed":3,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:49.100058+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.013921843,"stat_sum":{"num_bytes":8252963,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":10412,"num_write":5,"num_write_kb":1096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":2772362,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.31","version":"274'107","reported_seq":2133,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.071495+0000","last_change":"2024-09-11T03:52:03.074630+0000","last_active":"2024-09-11T03:52:05.071495+0000","last_peered":"2024-09-11T03:52:05.071495+0000","last_clean":"2024-09-11T03:52:05.071495+0000","last_became_active":"2024-09-11T03:51:20.060052+0000","last_became_peered":"2024-09-11T03:51:20.060052+0000","last_unstale":"2024-09-11T03:52:05.071495+0000","last_undegraded":"2024-09-11T03:52:05.071495+0000","last_fullsized":"2024-09-11T03:52:05.071495+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:51:26.087112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:26.087112+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:42.193937+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00076377599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,7],"acting":[0,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.30","version":"274'67","reported_seq":593,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.092016+0000","last_change":"2024-09-11T03:52:01.068784+0000","last_active":"2024-09-11T03:52:02.092016+0000","last_peered":"2024-09-11T03:52:02.092016+0000","last_clean":"2024-09-11T03:52:02.092016+0000","last_became_active":"2024-09-11T03:51:17.065156+0000","last_became_peered":"2024-09-11T03:51:17.065156+0000","last_unstale":"2024-09-11T03:52:02.092016+0000","last_undegraded":"2024-09-11T03:52:02.092016+0000","last_fullsized":"2024-09-11T03:52:02.092016+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:51:21.989674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.989674+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:33.976919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00068800800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.2f","version":"557'465","reported_seq":3442,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-09-11T03:52:03.072941+0000","last_change":"2024-09-11T03:52:03.073807+0000","last_active":"2024-09-11T03:52:03.072941+0000","last_peered":"2024-09-11T03:52:03.072941+0000","last_clean":"2024-09-11T03:52:03.072941+0000","last_became_active":"2024-09-11T03:50:07.077840+0000","last_became_peered":"2024-09-11T03:50:07.077840+0000","last_unstale":"2024-09-11T03:52:03.072941+0000","last_undegraded":"2024-09-11T03:52:03.072941+0000","last_fullsized":"2024-09-11T03:52:03.072941+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:51:47.522706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:47.522706+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:15.450978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0050570069999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.2e","version":"905'1207","reported_seq":5205,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.071868+0000","last_change":"2024-09-11T03:52:03.083246+0000","last_active":"2024-09-11T03:52:05.071868+0000","last_peered":"2024-09-11T03:52:05.071868+0000","last_clean":"2024-09-11T03:52:05.071868+0000","last_became_active":"2024-09-11T03:51:31.069593+0000","last_became_peered":"2024-09-11T03:51:31.069593+0000","last_unstale":"2024-09-11T03:52:05.071868+0000","last_undegraded":"2024-09-11T03:52:05.071868+0000","last_fullsized":"2024-09-11T03:52:05.071868+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":6,"last_scrub":"868'1145","last_scrub_stamp":"2024-09-11T03:51:27.034758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:27.034758+0000","objects_scrubbed":11,"log_size":1207,"log_dups_size":0,"ondisk_log_size":1207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:47.834881+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0079460750000000004,"stat_sum":{"num_bytes":30397185,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":34,"num_read_kb":34190,"num_write":25,"num_write_kb":8632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":24991924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,8],"acting":[0,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.2d","version":"903'1310","reported_seq":5356,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.071823+0000","last_change":"2024-09-11T03:52:01.084693+0000","last_active":"2024-09-11T03:52:02.071823+0000","last_peered":"2024-09-11T03:52:02.071823+0000","last_clean":"2024-09-11T03:52:02.071823+0000","last_became_active":"2024-09-11T03:49:29.163201+0000","last_became_peered":"2024-09-11T03:49:29.163201+0000","last_unstale":"2024-09-11T03:52:02.071823+0000","last_undegraded":"2024-09-11T03:52:02.071823+0000","last_fullsized":"2024-09-11T03:52:02.071823+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"837'1159","last_scrub_stamp":"2024-09-11T03:50:55.762457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:55.762457+0000","objects_scrubbed":20,"log_size":1310,"log_dups_size":0,"ondisk_log_size":1310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:20.212050+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.01108351,"stat_sum":{"num_bytes":47345238,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":113,"num_read_kb":143808,"num_write":39,"num_write_kb":21285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.2c","version":"903'951","reported_seq":5005,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.070490+0000","last_change":"2024-09-11T03:52:01.070794+0000","last_active":"2024-09-11T03:52:01.070490+0000","last_peered":"2024-09-11T03:52:01.070490+0000","last_clean":"2024-09-11T03:52:01.070490+0000","last_became_active":"2024-09-11T03:49:29.154872+0000","last_became_peered":"2024-09-11T03:49:29.154872+0000","last_unstale":"2024-09-11T03:52:01.070490+0000","last_undegraded":"2024-09-11T03:52:01.070490+0000","last_fullsized":"2024-09-11T03:52:01.070490+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"816'878","last_scrub_stamp":"2024-09-11T03:50:37.544645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.544645+0000","objects_scrubbed":12,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:02.748608+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0050164140000000003,"stat_sum":{"num_bytes":20794279,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":40,"num_read_kb":35339,"num_write":45,"num_write_kb":16227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3],"acting":[8,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.2b","version":"568'646","reported_seq":3850,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-09-11T03:52:03.068276+0000","last_change":"2024-09-11T03:52:03.070270+0000","last_active":"2024-09-11T03:52:03.068276+0000","last_peered":"2024-09-11T03:52:03.068276+0000","last_clean":"2024-09-11T03:52:03.068276+0000","last_became_active":"2024-09-11T03:51:20.062485+0000","last_became_peered":"2024-09-11T03:51:20.062485+0000","last_unstale":"2024-09-11T03:52:03.068276+0000","last_undegraded":"2024-09-11T03:52:03.068276+0000","last_fullsized":"2024-09-11T03:52:03.068276+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:51:52.556581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.556581+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:53.068013+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0022419879999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.2a","version":"727'348","reported_seq":4311,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.076815+0000","last_change":"2024-09-11T03:52:01.077207+0000","last_active":"2024-09-11T03:52:01.076815+0000","last_peered":"2024-09-11T03:52:01.076815+0000","last_clean":"2024-09-11T03:52:01.076815+0000","last_became_active":"2024-09-11T03:49:29.160595+0000","last_became_peered":"2024-09-11T03:49:29.160595+0000","last_unstale":"2024-09-11T03:52:01.076815+0000","last_undegraded":"2024-09-11T03:52:01.076815+0000","last_fullsized":"2024-09-11T03:52:01.076815+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:51:05.940576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:05.940576+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:17.449991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00096353599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,5],"acting":[7,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.29","version":"904'1337","reported_seq":5501,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.083716+0000","last_change":"2024-09-11T03:52:01.079823+0000","last_active":"2024-09-11T03:52:02.083716+0000","last_peered":"2024-09-11T03:52:02.083716+0000","last_clean":"2024-09-11T03:52:02.083716+0000","last_became_active":"2024-09-11T03:49:29.157194+0000","last_became_peered":"2024-09-11T03:49:29.157194+0000","last_unstale":"2024-09-11T03:52:02.083716+0000","last_undegraded":"2024-09-11T03:52:02.083716+0000","last_fullsized":"2024-09-11T03:52:02.083716+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"838'1272","last_scrub_stamp":"2024-09-11T03:50:57.773854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:57.773854+0000","objects_scrubbed":14,"log_size":1337,"log_dups_size":0,"ondisk_log_size":1337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:02.499580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0061966230000000001,"stat_sum":{"num_bytes":37060479,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":43,"num_read_kb":54621,"num_write":41,"num_write_kb":13701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8],"acting":[6,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.28","version":"902'475","reported_seq":4540,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.070715+0000","last_change":"2024-09-11T03:52:01.071146+0000","last_active":"2024-09-11T03:52:01.070715+0000","last_peered":"2024-09-11T03:52:01.070715+0000","last_clean":"2024-09-11T03:52:01.070715+0000","last_became_active":"2024-09-11T03:49:29.154884+0000","last_became_peered":"2024-09-11T03:49:29.154884+0000","last_unstale":"2024-09-11T03:52:01.070715+0000","last_undegraded":"2024-09-11T03:52:01.070715+0000","last_fullsized":"2024-09-11T03:52:01.070715+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"847'448","last_scrub_stamp":"2024-09-11T03:51:05.333850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:05.333850+0000","objects_scrubbed":8,"log_size":475,"log_dups_size":0,"ondisk_log_size":475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:23.469881+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00094755700000000002,"stat_sum":{"num_bytes":15310649,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":14,"num_read_kb":11529,"num_write":23,"num_write_kb":4588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.27","version":"903'1242","reported_seq":5517,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.085169+0000","last_change":"2024-09-11T03:52:01.085444+0000","last_active":"2024-09-11T03:52:01.085169+0000","last_peered":"2024-09-11T03:52:01.085169+0000","last_clean":"2024-09-11T03:52:01.085169+0000","last_became_active":"2024-09-11T03:51:20.062110+0000","last_became_peered":"2024-09-11T03:51:20.062110+0000","last_unstale":"2024-09-11T03:52:01.085169+0000","last_undegraded":"2024-09-11T03:52:01.085169+0000","last_fullsized":"2024-09-11T03:52:01.085169+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"840'1147","last_scrub_stamp":"2024-09-11T03:50:59.981154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:59.981154+0000","objects_scrubbed":8,"log_size":1242,"log_dups_size":0,"ondisk_log_size":1242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:13.029008+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0081466799999999999,"stat_sum":{"num_bytes":31213866,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":33,"num_read_kb":20640,"num_write":41,"num_write_kb":22316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":14737969,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.26","version":"269'171","reported_seq":4201,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.080476+0000","last_change":"2024-09-11T03:52:01.080749+0000","last_active":"2024-09-11T03:52:01.080476+0000","last_peered":"2024-09-11T03:52:01.080476+0000","last_clean":"2024-09-11T03:52:01.080476+0000","last_became_active":"2024-09-11T03:51:20.061265+0000","last_became_peered":"2024-09-11T03:51:20.061265+0000","last_unstale":"2024-09-11T03:52:01.080476+0000","last_undegraded":"2024-09-11T03:52:01.080476+0000","last_fullsized":"2024-09-11T03:52:01.080476+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:50:43.910786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:43.910786+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:10.740026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0035588659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,5],"acting":[7,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.25","version":"903'1534","reported_seq":5773,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.081974+0000","last_change":"2024-09-11T03:52:01.082228+0000","last_active":"2024-09-11T03:52:01.081974+0000","last_peered":"2024-09-11T03:52:01.081974+0000","last_clean":"2024-09-11T03:52:01.081974+0000","last_became_active":"2024-09-11T03:49:29.159733+0000","last_became_peered":"2024-09-11T03:49:29.159733+0000","last_unstale":"2024-09-11T03:52:01.081974+0000","last_undegraded":"2024-09-11T03:52:01.081974+0000","last_fullsized":"2024-09-11T03:52:01.081974+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"840'1347","last_scrub_stamp":"2024-09-11T03:50:59.564459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:59.564459+0000","objects_scrubbed":21,"log_size":1534,"log_dups_size":0,"ondisk_log_size":1534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:12.706222+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.015550039,"stat_sum":{"num_bytes":45149394,"num_objects":24,"num_object_clones":21,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":76,"num_read_kb":67686,"num_write":70,"num_write_kb":20416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,6],"acting":[8,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.24","version":"903'726","reported_seq":4811,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.071187+0000","last_change":"2024-09-11T03:52:01.076146+0000","last_active":"2024-09-11T03:52:02.071187+0000","last_peered":"2024-09-11T03:52:02.071187+0000","last_clean":"2024-09-11T03:52:02.071187+0000","last_became_active":"2024-09-11T03:49:29.155740+0000","last_became_peered":"2024-09-11T03:49:29.155740+0000","last_unstale":"2024-09-11T03:52:02.071187+0000","last_undegraded":"2024-09-11T03:52:02.071187+0000","last_fullsized":"2024-09-11T03:52:02.071187+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"838'674","last_scrub_stamp":"2024-09-11T03:50:56.770184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:56.770184+0000","objects_scrubbed":7,"log_size":726,"log_dups_size":0,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:03.072872+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.003110654,"stat_sum":{"num_bytes":15739462,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":14,"num_write_kb":11029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,7],"acting":[6,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.23","version":"903'552","reported_seq":4601,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.081839+0000","last_change":"2024-09-11T03:52:01.083447+0000","last_active":"2024-09-11T03:52:01.081839+0000","last_peered":"2024-09-11T03:52:01.081839+0000","last_clean":"2024-09-11T03:52:01.081839+0000","last_became_active":"2024-09-11T03:46:32.970100+0000","last_became_peered":"2024-09-11T03:46:32.970100+0000","last_unstale":"2024-09-11T03:52:01.081839+0000","last_undegraded":"2024-09-11T03:52:01.081839+0000","last_fullsized":"2024-09-11T03:52:01.081839+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"886'531","last_scrub_stamp":"2024-09-11T03:51:44.527530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:44.527530+0000","objects_scrubbed":5,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:05.005454+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.010957809000000001,"stat_sum":{"num_bytes":11706449,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":50017,"num_write":39,"num_write_kb":13976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.22","version":"531'216","reported_seq":4249,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.070973+0000","last_change":"2024-09-11T03:52:01.075371+0000","last_active":"2024-09-11T03:52:02.070973+0000","last_peered":"2024-09-11T03:52:02.070973+0000","last_clean":"2024-09-11T03:52:02.070973+0000","last_became_active":"2024-09-11T03:46:32.970633+0000","last_became_peered":"2024-09-11T03:46:32.970633+0000","last_unstale":"2024-09-11T03:52:02.070973+0000","last_undegraded":"2024-09-11T03:52:02.070973+0000","last_fullsized":"2024-09-11T03:52:02.070973+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:50:38.617374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:38.617374+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:06.140646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019691320000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.21","version":"902'401","reported_seq":4526,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.064631+0000","last_change":"2024-09-11T03:52:01.069948+0000","last_active":"2024-09-11T03:52:02.064631+0000","last_peered":"2024-09-11T03:52:02.064631+0000","last_clean":"2024-09-11T03:52:02.064631+0000","last_became_active":"2024-09-11T03:46:36.024928+0000","last_became_peered":"2024-09-11T03:46:36.024928+0000","last_unstale":"2024-09-11T03:52:02.064631+0000","last_undegraded":"2024-09-11T03:52:02.064631+0000","last_fullsized":"2024-09-11T03:52:02.064631+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":6,"last_scrub":"816'351","last_scrub_stamp":"2024-09-11T03:50:40.225380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:40.225380+0000","objects_scrubbed":7,"log_size":401,"log_dups_size":0,"ondisk_log_size":401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:05.981166+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011564290000000001,"stat_sum":{"num_bytes":16803143,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":56969,"num_write":45,"num_write_kb":19772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,8],"acting":[11,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.20","version":"902'352","reported_seq":4556,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.076734+0000","last_change":"2024-09-11T03:52:01.077188+0000","last_active":"2024-09-11T03:52:01.076734+0000","last_peered":"2024-09-11T03:52:01.076734+0000","last_clean":"2024-09-11T03:52:01.076734+0000","last_became_active":"2024-09-11T03:46:32.974375+0000","last_became_peered":"2024-09-11T03:46:32.974375+0000","last_unstale":"2024-09-11T03:52:01.076734+0000","last_undegraded":"2024-09-11T03:52:01.076734+0000","last_fullsized":"2024-09-11T03:52:01.076734+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"895'348","last_scrub_stamp":"2024-09-11T03:51:54.026522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.026522+0000","objects_scrubbed":6,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:59.137915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001219068,"stat_sum":{"num_bytes":19026910,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":98,"num_read_kb":144174,"num_write":25,"num_write_kb":13274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.1c","version":"806'776","reported_seq":4821,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.091917+0000","last_change":"2024-09-11T03:52:01.070086+0000","last_active":"2024-09-11T03:52:02.091917+0000","last_peered":"2024-09-11T03:52:02.091917+0000","last_clean":"2024-09-11T03:52:02.091917+0000","last_became_active":"2024-09-11T03:50:28.047548+0000","last_became_peered":"2024-09-11T03:50:28.047548+0000","last_unstale":"2024-09-11T03:52:02.091917+0000","last_undegraded":"2024-09-11T03:52:02.091917+0000","last_fullsized":"2024-09-11T03:52:02.091917+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"806'776","last_scrub_stamp":"2024-09-11T03:51:54.156180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.156180+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:55.651356+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013114909999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":875,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.071601+0000","last_change":"2024-09-11T03:50:55.035249+0000","last_active":"2024-09-11T03:52:05.071601+0000","last_peered":"2024-09-11T03:52:05.071601+0000","last_clean":"2024-09-11T03:52:05.071601+0000","last_became_active":"2024-09-11T03:46:15.974786+0000","last_became_peered":"2024-09-11T03:46:15.974786+0000","last_unstale":"2024-09-11T03:52:05.071601+0000","last_undegraded":"2024-09-11T03:52:05.071601+0000","last_fullsized":"2024-09-11T03:52:05.071601+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:55.035202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:55.035202+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:09.185590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.f","version":"557'465","reported_seq":4539,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.070722+0000","last_change":"2024-09-11T03:52:01.070722+0000","last_active":"2024-09-11T03:52:01.070722+0000","last_peered":"2024-09-11T03:52:01.070722+0000","last_clean":"2024-09-11T03:52:01.070722+0000","last_became_active":"2024-09-11T03:51:20.072191+0000","last_became_peered":"2024-09-11T03:51:20.072191+0000","last_unstale":"2024-09-11T03:52:01.070722+0000","last_undegraded":"2024-09-11T03:52:01.070722+0000","last_fullsized":"2024-09-11T03:52:01.070722+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:51:25.558029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:25.558029+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:47.769546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0044545640000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,11],"acting":[8,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"2.e","version":"782'1025","reported_seq":5051,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-09-11T03:52:03.069027+0000","last_change":"2024-09-11T03:52:03.072352+0000","last_active":"2024-09-11T03:52:03.069027+0000","last_peered":"2024-09-11T03:52:03.069027+0000","last_clean":"2024-09-11T03:52:03.069027+0000","last_became_active":"2024-09-11T03:50:02.051717+0000","last_became_peered":"2024-09-11T03:50:02.051717+0000","last_unstale":"2024-09-11T03:52:03.069027+0000","last_undegraded":"2024-09-11T03:52:03.069027+0000","last_fullsized":"2024-09-11T03:52:03.069027+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":0,"last_scrub":"782'1025","last_scrub_stamp":"2024-09-11T03:51:22.296245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.296245+0000","objects_scrubbed":0,"log_size":1025,"log_dups_size":0,"ondisk_log_size":1025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:33.450704+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0041748829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":1451,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.076974+0000","last_change":"2024-09-11T03:51:17.066500+0000","last_active":"2024-09-11T03:52:01.076974+0000","last_peered":"2024-09-11T03:52:01.076974+0000","last_clean":"2024-09-11T03:52:01.076974+0000","last_became_active":"2024-09-11T03:51:17.066085+0000","last_became_peered":"2024-09-11T03:51:17.066085+0000","last_unstale":"2024-09-11T03:52:01.076974+0000","last_undegraded":"2024-09-11T03:52:01.076974+0000","last_fullsized":"2024-09-11T03:52:01.076974+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:58.932376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:58.932376+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:10.994958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"902'1053","reported_seq":5214,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.085847+0000","last_change":"2024-09-11T03:52:01.086220+0000","last_active":"2024-09-11T03:52:01.085847+0000","last_peered":"2024-09-11T03:52:01.085847+0000","last_clean":"2024-09-11T03:52:01.085847+0000","last_became_active":"2024-09-11T03:51:20.062633+0000","last_became_peered":"2024-09-11T03:51:20.062633+0000","last_unstale":"2024-09-11T03:52:01.085847+0000","last_undegraded":"2024-09-11T03:52:01.085847+0000","last_fullsized":"2024-09-11T03:52:01.085847+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"811'1014","last_scrub_stamp":"2024-09-11T03:50:37.438736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.438736+0000","objects_scrubbed":5,"log_size":1053,"log_dups_size":0,"ondisk_log_size":1053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:01.670404+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0065678409999999996,"stat_sum":{"num_bytes":11250056,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":7026,"num_write":15,"num_write_kb":7796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":9438955,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":1166,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.077505+0000","last_change":"2024-09-11T03:51:17.063139+0000","last_active":"2024-09-11T03:52:01.077505+0000","last_peered":"2024-09-11T03:52:01.077505+0000","last_clean":"2024-09-11T03:52:01.077505+0000","last_became_active":"2024-09-11T03:51:17.061266+0000","last_became_peered":"2024-09-11T03:51:17.061266+0000","last_unstale":"2024-09-11T03:52:01.077505+0000","last_undegraded":"2024-09-11T03:52:01.077505+0000","last_fullsized":"2024-09-11T03:52:01.077505+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:38.843438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:38.843438+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:07.110555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"903'949","reported_seq":5009,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.064961+0000","last_change":"2024-09-11T03:52:01.083523+0000","last_active":"2024-09-11T03:52:02.064961+0000","last_peered":"2024-09-11T03:52:02.064961+0000","last_clean":"2024-09-11T03:52:02.064961+0000","last_became_active":"2024-09-11T03:49:24.115405+0000","last_became_peered":"2024-09-11T03:49:24.115405+0000","last_unstale":"2024-09-11T03:52:02.064961+0000","last_undegraded":"2024-09-11T03:52:02.064961+0000","last_fullsized":"2024-09-11T03:52:02.064961+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"843'880","last_scrub_stamp":"2024-09-11T03:51:01.987099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:01.987099+0000","objects_scrubbed":13,"log_size":949,"log_dups_size":0,"ondisk_log_size":949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:31.677744+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.01293162,"stat_sum":{"num_bytes":22929927,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":36,"num_read_kb":32542,"num_write":20,"num_write_kb":7053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.f","version":"20'92","reported_seq":734,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.072238+0000","last_change":"2024-09-11T03:51:21.860818+0000","last_active":"2024-09-11T03:52:02.072238+0000","last_peered":"2024-09-11T03:52:02.072238+0000","last_clean":"2024-09-11T03:52:02.072238+0000","last_became_active":"2024-09-11T03:51:17.064761+0000","last_became_peered":"2024-09-11T03:51:17.064761+0000","last_unstale":"2024-09-11T03:52:02.072238+0000","last_undegraded":"2024-09-11T03:52:02.072238+0000","last_fullsized":"2024-09-11T03:52:02.072238+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:21.860769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.860769+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:34.914441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":1850,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.076525+0000","last_change":"2024-09-11T03:51:52.035008+0000","last_active":"2024-09-11T03:52:01.076525+0000","last_peered":"2024-09-11T03:52:01.076525+0000","last_clean":"2024-09-11T03:52:01.076525+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:52:01.076525+0000","last_undegraded":"2024-09-11T03:52:01.076525+0000","last_fullsized":"2024-09-11T03:52:01.076525+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:52.034950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.034950+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:03.753329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"845'562","reported_seq":4623,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.085457+0000","last_change":"2024-09-11T03:52:01.085759+0000","last_active":"2024-09-11T03:52:01.085457+0000","last_peered":"2024-09-11T03:52:01.085457+0000","last_clean":"2024-09-11T03:52:01.085457+0000","last_became_active":"2024-09-11T03:51:08.055565+0000","last_became_peered":"2024-09-11T03:51:08.055565+0000","last_unstale":"2024-09-11T03:52:01.085457+0000","last_undegraded":"2024-09-11T03:52:01.085457+0000","last_fullsized":"2024-09-11T03:52:01.085457+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"845'562","last_scrub_stamp":"2024-09-11T03:51:20.441661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.441661+0000","objects_scrubbed":0,"log_size":562,"log_dups_size":0,"ondisk_log_size":562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:27.347601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067468320000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":1463,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.071549+0000","last_change":"2024-09-11T03:51:27.564638+0000","last_active":"2024-09-11T03:52:01.071549+0000","last_peered":"2024-09-11T03:52:01.071549+0000","last_clean":"2024-09-11T03:52:01.071549+0000","last_became_active":"2024-09-11T03:51:17.059234+0000","last_became_peered":"2024-09-11T03:51:17.059234+0000","last_unstale":"2024-09-11T03:52:01.071549+0000","last_undegraded":"2024-09-11T03:52:01.071549+0000","last_fullsized":"2024-09-11T03:52:01.071549+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:27.564586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:27.564586+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:54.532532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"563'192","reported_seq":3112,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.092076+0000","last_change":"2024-09-11T03:52:01.069908+0000","last_active":"2024-09-11T03:52:02.092076+0000","last_peered":"2024-09-11T03:52:02.092076+0000","last_clean":"2024-09-11T03:52:02.092076+0000","last_became_active":"2024-09-11T03:50:31.044399+0000","last_became_peered":"2024-09-11T03:50:31.044399+0000","last_unstale":"2024-09-11T03:52:02.092076+0000","last_undegraded":"2024-09-11T03:52:02.092076+0000","last_fullsized":"2024-09-11T03:52:02.092076+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:50:36.771360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.771360+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015402370000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":1428,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.071826+0000","last_change":"2024-09-11T03:51:17.061730+0000","last_active":"2024-09-11T03:52:05.071826+0000","last_peered":"2024-09-11T03:52:05.071826+0000","last_clean":"2024-09-11T03:52:05.071826+0000","last_became_active":"2024-09-11T03:51:17.061243+0000","last_became_peered":"2024-09-11T03:51:17.061243+0000","last_unstale":"2024-09-11T03:52:05.071826+0000","last_undegraded":"2024-09-11T03:52:05.071826+0000","last_fullsized":"2024-09-11T03:52:05.071826+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:41.975660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:41.975660+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"567'235","reported_seq":4282,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.071989+0000","last_change":"2024-09-11T03:52:03.074727+0000","last_active":"2024-09-11T03:52:05.071989+0000","last_peered":"2024-09-11T03:52:05.071989+0000","last_clean":"2024-09-11T03:52:05.071989+0000","last_became_active":"2024-09-11T03:50:28.039300+0000","last_became_peered":"2024-09-11T03:50:28.039300+0000","last_unstale":"2024-09-11T03:52:05.071989+0000","last_undegraded":"2024-09-11T03:52:05.071989+0000","last_fullsized":"2024-09-11T03:52:05.071989+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:51:50.027291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:50.027291+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:55.922517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044412500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":1027,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.092121+0000","last_change":"2024-09-11T03:51:29.996670+0000","last_active":"2024-09-11T03:52:02.092121+0000","last_peered":"2024-09-11T03:52:02.092121+0000","last_clean":"2024-09-11T03:52:02.092121+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:52:02.092121+0000","last_undegraded":"2024-09-11T03:52:02.092121+0000","last_fullsized":"2024-09-11T03:52:02.092121+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:29.996598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:29.996598+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:54.485931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"843'319","reported_seq":4481,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.085545+0000","last_change":"2024-09-11T03:52:01.085821+0000","last_active":"2024-09-11T03:52:01.085545+0000","last_peered":"2024-09-11T03:52:01.085545+0000","last_clean":"2024-09-11T03:52:01.085545+0000","last_became_active":"2024-09-11T03:51:08.055103+0000","last_became_peered":"2024-09-11T03:51:08.055103+0000","last_unstale":"2024-09-11T03:52:01.085545+0000","last_undegraded":"2024-09-11T03:52:01.085545+0000","last_fullsized":"2024-09-11T03:52:01.085545+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"843'319","last_scrub_stamp":"2024-09-11T03:51:19.453435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.453435+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:28.335517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065633769999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":711,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.071705+0000","last_change":"2024-09-11T03:49:37.637065+0000","last_active":"2024-09-11T03:52:02.071705+0000","last_peered":"2024-09-11T03:52:02.071705+0000","last_clean":"2024-09-11T03:52:02.071705+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:52:02.071705+0000","last_undegraded":"2024-09-11T03:52:02.071705+0000","last_fullsized":"2024-09-11T03:52:02.071705+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:37.637018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.637018+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:07.712022+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"847'690","reported_seq":4814,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.081517+0000","last_change":"2024-09-11T03:52:01.082430+0000","last_active":"2024-09-11T03:52:01.081517+0000","last_peered":"2024-09-11T03:52:01.081517+0000","last_clean":"2024-09-11T03:52:01.081517+0000","last_became_active":"2024-09-11T03:51:08.058954+0000","last_became_peered":"2024-09-11T03:51:08.058954+0000","last_unstale":"2024-09-11T03:52:01.081517+0000","last_undegraded":"2024-09-11T03:52:01.081517+0000","last_fullsized":"2024-09-11T03:52:01.081517+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"847'690","last_scrub_stamp":"2024-09-11T03:51:24.473723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:24.473723+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:36.179630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010601744999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":1038,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.091770+0000","last_change":"2024-09-11T03:51:53.155265+0000","last_active":"2024-09-11T03:52:02.091770+0000","last_peered":"2024-09-11T03:52:02.091770+0000","last_clean":"2024-09-11T03:52:02.091770+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:52:02.091770+0000","last_undegraded":"2024-09-11T03:52:02.091770+0000","last_fullsized":"2024-09-11T03:52:02.091770+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:53.155207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.155207+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:07.708610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"902'1255","reported_seq":5436,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.070897+0000","last_change":"2024-09-11T03:52:01.071576+0000","last_active":"2024-09-11T03:52:01.070897+0000","last_peered":"2024-09-11T03:52:01.070897+0000","last_clean":"2024-09-11T03:52:01.070897+0000","last_became_active":"2024-09-11T03:51:20.063427+0000","last_became_peered":"2024-09-11T03:51:20.063427+0000","last_unstale":"2024-09-11T03:52:01.070897+0000","last_undegraded":"2024-09-11T03:52:01.070897+0000","last_fullsized":"2024-09-11T03:52:01.070897+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"862'1215","last_scrub_stamp":"2024-09-11T03:51:21.601542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.601542+0000","objects_scrubbed":5,"log_size":1255,"log_dups_size":0,"ondisk_log_size":1255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:29.105129+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0045475580000000002,"stat_sum":{"num_bytes":13495545,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":2299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":1021,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.092111+0000","last_change":"2024-09-11T03:49:36.654074+0000","last_active":"2024-09-11T03:52:02.092111+0000","last_peered":"2024-09-11T03:52:02.092111+0000","last_clean":"2024-09-11T03:52:02.092111+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:52:02.092111+0000","last_undegraded":"2024-09-11T03:52:02.092111+0000","last_fullsized":"2024-09-11T03:52:02.092111+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:36.654025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.654025+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:07.849922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":4217,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.076787+0000","last_change":"2024-09-11T03:52:01.077139+0000","last_active":"2024-09-11T03:52:01.076787+0000","last_peered":"2024-09-11T03:52:01.076787+0000","last_clean":"2024-09-11T03:52:01.076787+0000","last_became_active":"2024-09-11T03:51:08.055529+0000","last_became_peered":"2024-09-11T03:51:08.055529+0000","last_unstale":"2024-09-11T03:52:01.076787+0000","last_undegraded":"2024-09-11T03:52:01.076787+0000","last_fullsized":"2024-09-11T03:52:01.076787+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:51:22.407323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.407323+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:47.006833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061549669999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":1629,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.064645+0000","last_change":"2024-09-11T03:51:31.102151+0000","last_active":"2024-09-11T03:52:02.064645+0000","last_peered":"2024-09-11T03:52:02.064645+0000","last_clean":"2024-09-11T03:52:02.064645+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:52:02.064645+0000","last_undegraded":"2024-09-11T03:52:02.064645+0000","last_fullsized":"2024-09-11T03:52:02.064645+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:31.102098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:31.102098+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:49.931850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"904'1328","reported_seq":5576,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.094407+0000","last_change":"2024-09-11T03:52:01.084275+0000","last_active":"2024-09-11T03:52:02.094407+0000","last_peered":"2024-09-11T03:52:02.094407+0000","last_clean":"2024-09-11T03:52:02.094407+0000","last_became_active":"2024-09-11T03:51:08.057350+0000","last_became_peered":"2024-09-11T03:51:08.057350+0000","last_unstale":"2024-09-11T03:52:02.094407+0000","last_undegraded":"2024-09-11T03:52:02.094407+0000","last_fullsized":"2024-09-11T03:52:02.094407+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'1274","last_scrub_stamp":"2024-09-11T03:51:19.872739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.872739+0000","objects_scrubbed":12,"log_size":1328,"log_dups_size":0,"ondisk_log_size":1328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:42.196705+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.010476619,"stat_sum":{"num_bytes":38444164,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":3103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":1432,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.064602+0000","last_change":"2024-09-11T03:51:47.050276+0000","last_active":"2024-09-11T03:52:02.064602+0000","last_peered":"2024-09-11T03:52:02.064602+0000","last_clean":"2024-09-11T03:52:02.064602+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:52:02.064602+0000","last_undegraded":"2024-09-11T03:52:02.064602+0000","last_fullsized":"2024-09-11T03:52:02.064602+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:47.050227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:47.050227+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:47.461383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"744'374","reported_seq":4498,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.077142+0000","last_change":"2024-09-11T03:52:01.078049+0000","last_active":"2024-09-11T03:52:01.077142+0000","last_peered":"2024-09-11T03:52:01.077142+0000","last_clean":"2024-09-11T03:52:01.077142+0000","last_became_active":"2024-09-11T03:51:08.057750+0000","last_became_peered":"2024-09-11T03:51:08.057750+0000","last_unstale":"2024-09-11T03:52:01.077142+0000","last_undegraded":"2024-09-11T03:52:01.077142+0000","last_fullsized":"2024-09-11T03:52:01.077142+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'374","last_scrub_stamp":"2024-09-11T03:51:20.010112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.010112+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:21.407959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001785303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":1785,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-09-11T03:52:03.069269+0000","last_change":"2024-09-11T03:51:17.061700+0000","last_active":"2024-09-11T03:52:03.069269+0000","last_peered":"2024-09-11T03:52:03.069269+0000","last_clean":"2024-09-11T03:52:03.069269+0000","last_became_active":"2024-09-11T03:51:17.061257+0000","last_became_peered":"2024-09-11T03:51:17.061257+0000","last_unstale":"2024-09-11T03:52:03.069269+0000","last_undegraded":"2024-09-11T03:52:03.069269+0000","last_fullsized":"2024-09-11T03:52:03.069269+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:18.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.933952+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:39.424252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"846'1373","reported_seq":5553,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.072412+0000","last_change":"2024-09-11T03:52:01.073013+0000","last_active":"2024-09-11T03:52:01.072412+0000","last_peered":"2024-09-11T03:52:01.072412+0000","last_clean":"2024-09-11T03:52:01.072412+0000","last_became_active":"2024-09-11T03:51:31.065505+0000","last_became_peered":"2024-09-11T03:51:31.065505+0000","last_unstale":"2024-09-11T03:52:01.072412+0000","last_undegraded":"2024-09-11T03:52:01.072412+0000","last_fullsized":"2024-09-11T03:52:01.072412+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"846'1373","last_scrub_stamp":"2024-09-11T03:51:19.328459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.328459+0000","objects_scrubbed":0,"log_size":1373,"log_dups_size":0,"ondisk_log_size":1373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:22.036141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00269068,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":1845,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.075761+0000","last_change":"2024-09-11T03:51:04.911507+0000","last_active":"2024-09-11T03:52:01.075761+0000","last_peered":"2024-09-11T03:52:01.075761+0000","last_clean":"2024-09-11T03:52:01.075761+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:52:01.075761+0000","last_undegraded":"2024-09-11T03:52:01.075761+0000","last_fullsized":"2024-09-11T03:52:01.075761+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:04.911446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:04.911446+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:33.168597+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"902'419","reported_seq":4551,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.065184+0000","last_change":"2024-09-11T03:52:01.078822+0000","last_active":"2024-09-11T03:52:02.065184+0000","last_peered":"2024-09-11T03:52:02.065184+0000","last_clean":"2024-09-11T03:52:02.065184+0000","last_became_active":"2024-09-11T03:51:08.053749+0000","last_became_peered":"2024-09-11T03:51:08.053749+0000","last_unstale":"2024-09-11T03:52:02.065184+0000","last_undegraded":"2024-09-11T03:52:02.065184+0000","last_fullsized":"2024-09-11T03:52:02.065184+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'392","last_scrub_stamp":"2024-09-11T03:51:19.986753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.986753+0000","objects_scrubbed":5,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:32.774361+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0097830420000000005,"stat_sum":{"num_bytes":10459472,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":10128,"num_write":9,"num_write_kb":2863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":1503,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.079042+0000","last_change":"2024-09-11T03:51:55.165620+0000","last_active":"2024-09-11T03:52:01.079042+0000","last_peered":"2024-09-11T03:52:01.079042+0000","last_clean":"2024-09-11T03:52:01.079042+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:52:01.079042+0000","last_undegraded":"2024-09-11T03:52:01.079042+0000","last_fullsized":"2024-09-11T03:52:01.079042+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:55.165539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:55.165539+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:11.582309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"568'646","reported_seq":4765,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.066475+0000","last_change":"2024-09-11T03:52:01.066475+0000","last_active":"2024-09-11T03:52:01.066475+0000","last_peered":"2024-09-11T03:52:01.066475+0000","last_clean":"2024-09-11T03:52:01.066475+0000","last_became_active":"2024-09-11T03:51:08.054955+0000","last_became_peered":"2024-09-11T03:51:08.054955+0000","last_unstale":"2024-09-11T03:52:01.066475+0000","last_undegraded":"2024-09-11T03:52:01.066475+0000","last_fullsized":"2024-09-11T03:52:01.066475+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:51:20.608388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.608388+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:24.075443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073580800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,0],"acting":[8,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":1444,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.078913+0000","last_change":"2024-09-11T03:49:00.710015+0000","last_active":"2024-09-11T03:52:01.078913+0000","last_peered":"2024-09-11T03:52:01.078913+0000","last_clean":"2024-09-11T03:52:01.078913+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:52:01.078913+0000","last_undegraded":"2024-09-11T03:52:01.078913+0000","last_fullsized":"2024-09-11T03:52:01.078913+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:00.709966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.709966+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:33.296056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":4142,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-09-11T03:52:03.073903+0000","last_change":"2024-09-11T03:52:03.075002+0000","last_active":"2024-09-11T03:52:03.073903+0000","last_peered":"2024-09-11T03:52:03.073903+0000","last_clean":"2024-09-11T03:52:03.073903+0000","last_became_active":"2024-09-11T03:51:20.063987+0000","last_became_peered":"2024-09-11T03:51:20.063987+0000","last_unstale":"2024-09-11T03:52:03.073903+0000","last_undegraded":"2024-09-11T03:52:03.073903+0000","last_fullsized":"2024-09-11T03:52:03.073903+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:51:25.248547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:25.248547+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:39.620457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0061157310000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,7],"acting":[9,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":1446,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.071678+0000","last_change":"2024-09-11T03:51:00.033938+0000","last_active":"2024-09-11T03:52:05.071678+0000","last_peered":"2024-09-11T03:52:05.071678+0000","last_clean":"2024-09-11T03:52:05.071678+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:52:05.071678+0000","last_undegraded":"2024-09-11T03:52:05.071678+0000","last_fullsized":"2024-09-11T03:52:05.071678+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:00.033889+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"902'588","reported_seq":4657,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.092397+0000","last_change":"2024-09-11T03:52:01.070689+0000","last_active":"2024-09-11T03:52:02.092397+0000","last_peered":"2024-09-11T03:52:02.092397+0000","last_clean":"2024-09-11T03:52:02.092397+0000","last_became_active":"2024-09-11T03:50:02.053513+0000","last_became_peered":"2024-09-11T03:50:02.053513+0000","last_unstale":"2024-09-11T03:52:02.092397+0000","last_undegraded":"2024-09-11T03:52:02.092397+0000","last_fullsized":"2024-09-11T03:52:02.092397+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"838'575","last_scrub_stamp":"2024-09-11T03:51:04.839779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:04.839779+0000","objects_scrubbed":6,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:08.946705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013744720000000001,"stat_sum":{"num_bytes":8185735,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":2803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":1433,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.078820+0000","last_change":"2024-09-11T03:51:54.138565+0000","last_active":"2024-09-11T03:52:01.078820+0000","last_peered":"2024-09-11T03:52:01.078820+0000","last_clean":"2024-09-11T03:52:01.078820+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:52:01.078820+0000","last_undegraded":"2024-09-11T03:52:01.078820+0000","last_fullsized":"2024-09-11T03:52:01.078820+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:54.138504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.138504+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:20.328413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":3208,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.071367+0000","last_change":"2024-09-11T03:52:01.076205+0000","last_active":"2024-09-11T03:52:02.071367+0000","last_peered":"2024-09-11T03:52:02.071367+0000","last_clean":"2024-09-11T03:52:02.071367+0000","last_became_active":"2024-09-11T03:51:20.064079+0000","last_became_peered":"2024-09-11T03:51:20.064079+0000","last_unstale":"2024-09-11T03:52:02.071367+0000","last_undegraded":"2024-09-11T03:52:02.071367+0000","last_fullsized":"2024-09-11T03:52:02.071367+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:51:47.080811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:47.080811+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:54.893299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024094310000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,9],"acting":[6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.13","version":"20'92","reported_seq":1539,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.071441+0000","last_change":"2024-09-11T03:51:52.573394+0000","last_active":"2024-09-11T03:52:01.071441+0000","last_peered":"2024-09-11T03:52:01.071441+0000","last_clean":"2024-09-11T03:52:01.071441+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:52:01.071441+0000","last_undegraded":"2024-09-11T03:52:01.071441+0000","last_fullsized":"2024-09-11T03:52:01.071441+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:52.573348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.573348+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:55.111010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":3145,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.065738+0000","last_change":"2024-09-11T03:52:01.066197+0000","last_active":"2024-09-11T03:52:01.065738+0000","last_peered":"2024-09-11T03:52:01.065738+0000","last_clean":"2024-09-11T03:52:01.065738+0000","last_became_active":"2024-09-11T03:50:02.057196+0000","last_became_peered":"2024-09-11T03:50:02.057196+0000","last_unstale":"2024-09-11T03:52:01.065738+0000","last_undegraded":"2024-09-11T03:52:01.065738+0000","last_fullsized":"2024-09-11T03:52:01.065738+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:51:24.589460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:24.589460+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:30.274384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00070704600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":1021,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.077336+0000","last_change":"2024-09-11T03:51:28.960262+0000","last_active":"2024-09-11T03:52:01.077336+0000","last_peered":"2024-09-11T03:52:01.077336+0000","last_clean":"2024-09-11T03:52:01.077336+0000","last_became_active":"2024-09-11T03:46:36.026459+0000","last_became_peered":"2024-09-11T03:46:36.026459+0000","last_unstale":"2024-09-11T03:52:01.077336+0000","last_undegraded":"2024-09-11T03:52:01.077336+0000","last_fullsized":"2024-09-11T03:52:01.077336+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:28.960214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.960214+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:51.817417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":3700,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.072020+0000","last_change":"2024-09-11T03:52:01.080133+0000","last_active":"2024-09-11T03:52:02.072020+0000","last_peered":"2024-09-11T03:52:02.072020+0000","last_clean":"2024-09-11T03:52:02.072020+0000","last_became_active":"2024-09-11T03:50:02.052278+0000","last_became_peered":"2024-09-11T03:50:02.052278+0000","last_unstale":"2024-09-11T03:52:02.072020+0000","last_undegraded":"2024-09-11T03:52:02.072020+0000","last_fullsized":"2024-09-11T03:52:02.072020+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:51:28.919603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.919603+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:52.481629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0064826149999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":1407,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.076788+0000","last_change":"2024-09-11T03:50:19.741484+0000","last_active":"2024-09-11T03:52:01.076788+0000","last_peered":"2024-09-11T03:52:01.076788+0000","last_clean":"2024-09-11T03:52:01.076788+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:52:01.076788+0000","last_undegraded":"2024-09-11T03:52:01.076788+0000","last_fullsized":"2024-09-11T03:52:01.076788+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:19.741428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:19.741428+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:37.978745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"782'561","reported_seq":4668,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.078935+0000","last_change":"2024-09-11T03:52:01.081384+0000","last_active":"2024-09-11T03:52:01.078935+0000","last_peered":"2024-09-11T03:52:01.078935+0000","last_clean":"2024-09-11T03:52:01.078935+0000","last_became_active":"2024-09-11T03:50:02.054667+0000","last_became_peered":"2024-09-11T03:50:02.054667+0000","last_unstale":"2024-09-11T03:52:01.078935+0000","last_undegraded":"2024-09-11T03:52:01.078935+0000","last_fullsized":"2024-09-11T03:52:01.078935+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"782'561","last_scrub_stamp":"2024-09-11T03:51:24.484944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:24.484944+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:44.267922+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0028587360000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"167","length":"1"},{"start":"174","length":"1"},{"start":"18c","length":"1"},{"start":"196","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":887,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.065180+0000","last_change":"2024-09-11T03:50:37.230454+0000","last_active":"2024-09-11T03:52:02.065180+0000","last_peered":"2024-09-11T03:52:02.065180+0000","last_clean":"2024-09-11T03:52:02.065180+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:52:02.065180+0000","last_undegraded":"2024-09-11T03:52:02.065180+0000","last_fullsized":"2024-09-11T03:52:02.065180+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:37.230388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.230388+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:04.405605+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":4357,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-09-11T03:52:03.068121+0000","last_change":"2024-09-11T03:52:03.068570+0000","last_active":"2024-09-11T03:52:03.068121+0000","last_peered":"2024-09-11T03:52:03.068121+0000","last_clean":"2024-09-11T03:52:03.068121+0000","last_became_active":"2024-09-11T03:50:02.051850+0000","last_became_peered":"2024-09-11T03:50:02.051850+0000","last_unstale":"2024-09-11T03:52:03.068121+0000","last_undegraded":"2024-09-11T03:52:03.068121+0000","last_fullsized":"2024-09-11T03:52:03.068121+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:51:20.293380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.293380+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:39.445459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00069580100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":1427,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.066076+0000","last_change":"2024-09-11T03:50:43.547521+0000","last_active":"2024-09-11T03:52:01.066076+0000","last_peered":"2024-09-11T03:52:01.066076+0000","last_clean":"2024-09-11T03:52:01.066076+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:52:01.066076+0000","last_undegraded":"2024-09-11T03:52:01.066076+0000","last_fullsized":"2024-09-11T03:52:01.066076+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:43.547475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:43.547475+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"778'353","reported_seq":4334,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.071524+0000","last_change":"2024-09-11T03:52:01.074955+0000","last_active":"2024-09-11T03:52:02.071524+0000","last_peered":"2024-09-11T03:52:02.071524+0000","last_clean":"2024-09-11T03:52:02.071524+0000","last_became_active":"2024-09-11T03:50:02.054211+0000","last_became_peered":"2024-09-11T03:50:02.054211+0000","last_unstale":"2024-09-11T03:52:02.071524+0000","last_undegraded":"2024-09-11T03:52:02.071524+0000","last_fullsized":"2024-09-11T03:52:02.071524+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"778'353","last_scrub_stamp":"2024-09-11T03:51:23.900161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.900161+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:36.663259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016530080000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.18","version":"20'92","reported_seq":1545,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.078573+0000","last_change":"2024-09-11T03:51:27.437144+0000","last_active":"2024-09-11T03:52:01.078573+0000","last_peered":"2024-09-11T03:52:01.078573+0000","last_clean":"2024-09-11T03:52:01.078573+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:52:01.078573+0000","last_undegraded":"2024-09-11T03:52:01.078573+0000","last_fullsized":"2024-09-11T03:52:01.078573+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:27.437090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:27.437090+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:35.495938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"902'921","reported_seq":5162,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.064614+0000","last_change":"2024-09-11T03:52:01.079914+0000","last_active":"2024-09-11T03:52:02.064614+0000","last_peered":"2024-09-11T03:52:02.064614+0000","last_clean":"2024-09-11T03:52:02.064614+0000","last_became_active":"2024-09-11T03:50:31.047148+0000","last_became_peered":"2024-09-11T03:50:31.047148+0000","last_unstale":"2024-09-11T03:52:02.064614+0000","last_undegraded":"2024-09-11T03:52:02.064614+0000","last_fullsized":"2024-09-11T03:52:02.064614+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":5,"last_scrub":"882'911","last_scrub_stamp":"2024-09-11T03:51:45.052831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:45.052831+0000","objects_scrubbed":6,"log_size":921,"log_dups_size":0,"ondisk_log_size":921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:04.955360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0094569340000000002,"stat_sum":{"num_bytes":12396215,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":15,"num_read_kb":10932,"num_write":16,"num_write_kb":5002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.19","version":"20'92","reported_seq":1034,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.091722+0000","last_change":"2024-09-11T03:51:28.080491+0000","last_active":"2024-09-11T03:52:02.091722+0000","last_peered":"2024-09-11T03:52:02.091722+0000","last_clean":"2024-09-11T03:52:02.091722+0000","last_became_active":"2024-09-11T03:46:15.972841+0000","last_became_peered":"2024-09-11T03:46:15.972841+0000","last_unstale":"2024-09-11T03:52:02.091722+0000","last_undegraded":"2024-09-11T03:52:02.091722+0000","last_fullsized":"2024-09-11T03:52:02.091722+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:28.080420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.080420+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:28.344483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"557'273","reported_seq":4290,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.071117+0000","last_change":"2024-09-11T03:52:01.076399+0000","last_active":"2024-09-11T03:52:02.071117+0000","last_peered":"2024-09-11T03:52:02.071117+0000","last_clean":"2024-09-11T03:52:02.071117+0000","last_became_active":"2024-09-11T03:50:28.038587+0000","last_became_peered":"2024-09-11T03:50:28.038587+0000","last_unstale":"2024-09-11T03:52:02.071117+0000","last_undegraded":"2024-09-11T03:52:02.071117+0000","last_fullsized":"2024-09-11T03:52:02.071117+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:51:54.121178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.121178+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:22.583915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031889459999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,2],"acting":[6,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.1a","version":"20'92","reported_seq":585,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.076117+0000","last_change":"2024-09-11T03:51:23.032108+0000","last_active":"2024-09-11T03:52:01.076117+0000","last_peered":"2024-09-11T03:52:01.076117+0000","last_clean":"2024-09-11T03:52:01.076117+0000","last_became_active":"2024-09-11T03:51:17.063957+0000","last_became_peered":"2024-09-11T03:51:17.063957+0000","last_unstale":"2024-09-11T03:52:01.076117+0000","last_undegraded":"2024-09-11T03:52:01.076117+0000","last_fullsized":"2024-09-11T03:52:01.076117+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:23.032056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.032056+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:35.587988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.19","version":"902'621","reported_seq":4763,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.070996+0000","last_change":"2024-09-11T03:52:01.074646+0000","last_active":"2024-09-11T03:52:02.070996+0000","last_peered":"2024-09-11T03:52:02.070996+0000","last_clean":"2024-09-11T03:52:02.070996+0000","last_became_active":"2024-09-11T03:50:28.039694+0000","last_became_peered":"2024-09-11T03:50:28.039694+0000","last_unstale":"2024-09-11T03:52:02.070996+0000","last_undegraded":"2024-09-11T03:52:02.070996+0000","last_fullsized":"2024-09-11T03:52:02.070996+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"816'587","last_scrub_stamp":"2024-09-11T03:50:36.522195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.522195+0000","objects_scrubbed":7,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016718620000000001,"stat_sum":{"num_bytes":14628738,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.1b","version":"20'92","reported_seq":1404,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.065496+0000","last_change":"2024-09-11T03:51:52.434832+0000","last_active":"2024-09-11T03:52:01.065496+0000","last_peered":"2024-09-11T03:52:01.065496+0000","last_clean":"2024-09-11T03:52:01.065496+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:52:01.065496+0000","last_undegraded":"2024-09-11T03:52:01.065496+0000","last_fullsized":"2024-09-11T03:52:01.065496+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:52.434784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.434784+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:11.714276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"905'726","reported_seq":4967,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-09-11T03:52:03.078930+0000","last_change":"2024-09-11T03:52:03.079412+0000","last_active":"2024-09-11T03:52:03.078930+0000","last_peered":"2024-09-11T03:52:03.078930+0000","last_clean":"2024-09-11T03:52:03.078930+0000","last_became_active":"2024-09-11T03:51:20.065666+0000","last_became_peered":"2024-09-11T03:51:20.065666+0000","last_unstale":"2024-09-11T03:52:03.078930+0000","last_undegraded":"2024-09-11T03:52:03.078930+0000","last_fullsized":"2024-09-11T03:52:03.078930+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":5,"last_scrub":"816'566","last_scrub_stamp":"2024-09-11T03:50:35.256345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.256345+0000","objects_scrubbed":11,"log_size":726,"log_dups_size":0,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.010707070000000001,"stat_sum":{"num_bytes":41106880,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":28,"num_read_kb":24472,"num_write":28,"num_write_kb":11526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":15760321,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,11],"acting":[9,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.1f","version":"907'890","reported_seq":4936,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.100614+0000","last_change":"2024-09-11T03:52:03.085879+0000","last_active":"2024-09-11T03:52:05.100614+0000","last_peered":"2024-09-11T03:52:05.100614+0000","last_clean":"2024-09-11T03:52:05.100614+0000","last_became_active":"2024-09-11T03:50:28.041110+0000","last_became_peered":"2024-09-11T03:50:28.041110+0000","last_unstale":"2024-09-11T03:52:05.100614+0000","last_undegraded":"2024-09-11T03:52:05.100614+0000","last_fullsized":"2024-09-11T03:52:05.100614+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"894'871","last_scrub_stamp":"2024-09-11T03:51:53.078478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.078478+0000","objects_scrubbed":13,"log_size":890,"log_dups_size":0,"ondisk_log_size":890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:17.605716+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.011179424,"stat_sum":{"num_bytes":37592638,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":43,"num_read_kb":39435,"num_write":25,"num_write_kb":13669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.1c","version":"20'92","reported_seq":1411,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.065842+0000","last_change":"2024-09-11T03:51:05.596734+0000","last_active":"2024-09-11T03:52:01.065842+0000","last_peered":"2024-09-11T03:52:01.065842+0000","last_clean":"2024-09-11T03:52:01.065842+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:52:01.065842+0000","last_undegraded":"2024-09-11T03:52:01.065842+0000","last_fullsized":"2024-09-11T03:52:01.065842+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:05.596677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:05.596677+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:09.881830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1e","version":"903'1112","reported_seq":5168,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.071872+0000","last_change":"2024-09-11T03:52:03.083454+0000","last_active":"2024-09-11T03:52:05.071872+0000","last_peered":"2024-09-11T03:52:05.071872+0000","last_clean":"2024-09-11T03:52:05.071872+0000","last_became_active":"2024-09-11T03:50:28.044451+0000","last_became_peered":"2024-09-11T03:50:28.044451+0000","last_unstale":"2024-09-11T03:52:05.071872+0000","last_undegraded":"2024-09-11T03:52:05.071872+0000","last_fullsized":"2024-09-11T03:52:05.071872+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"885'1106","last_scrub_stamp":"2024-09-11T03:51:44.050833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:44.050833+0000","objects_scrubbed":5,"log_size":1112,"log_dups_size":0,"ondisk_log_size":1112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:07.661910+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0083820170000000003,"stat_sum":{"num_bytes":9671047,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":24362,"num_write":12,"num_write_kb":3430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.1d","version":"20'92","reported_seq":1393,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-11T03:52:01.072424+0000","last_change":"2024-09-11T03:50:56.316951+0000","last_active":"2024-09-11T03:52:01.072424+0000","last_peered":"2024-09-11T03:52:01.072424+0000","last_clean":"2024-09-11T03:52:01.072424+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:52:01.072424+0000","last_undegraded":"2024-09-11T03:52:01.072424+0000","last_fullsized":"2024-09-11T03:52:01.072424+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:56.316896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:56.316896+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:07.998967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"567'752","reported_seq":4781,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.092046+0000","last_change":"2024-09-11T03:52:01.071137+0000","last_active":"2024-09-11T03:52:02.092046+0000","last_peered":"2024-09-11T03:52:02.092046+0000","last_clean":"2024-09-11T03:52:02.092046+0000","last_became_active":"2024-09-11T03:50:28.045687+0000","last_became_peered":"2024-09-11T03:50:28.045687+0000","last_unstale":"2024-09-11T03:52:02.092046+0000","last_undegraded":"2024-09-11T03:52:02.092046+0000","last_fullsized":"2024-09-11T03:52:02.092046+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:51:52.141035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.141035+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:05.149806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001716658,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,11],"acting":[10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"143","length":"1"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.1e","version":"20'92","reported_seq":1420,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-09-11T03:52:02.065368+0000","last_change":"2024-09-11T03:51:02.962842+0000","last_active":"2024-09-11T03:52:02.065368+0000","last_peered":"2024-09-11T03:52:02.065368+0000","last_clean":"2024-09-11T03:52:02.065368+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:52:02.065368+0000","last_undegraded":"2024-09-11T03:52:02.065368+0000","last_fullsized":"2024-09-11T03:52:02.065368+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:02.962791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:02.962791+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:13.625901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":76,"stat_sum":{"num_bytes":773868517,"num_objects":324,"num_object_clones":274,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":10,"num_read":931,"num_read_kb":969159,"num_write":654,"num_write_kb":267459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":82257142,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1358798848,"data_stored":1776351724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47609,"ondisk_log_size":47609,"up":228,"acting":228,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285746,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":229904,"kb_used_data":172340,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93515248,"statfs":{"total":95995035648,"available":95759613952,"internally_reserved":0,"allocated":176476160,"data_stored":225067997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":58915268},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514678,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94668,"kb_used_data":63468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93650484,"statfs":{"total":95995035648,"available":95898095616,"internally_reserved":0,"allocated":64991232,"data_stored":74434449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743612,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":177448,"kb_used_data":114324,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93567704,"statfs":{"total":95995035648,"available":95813328896,"internally_reserved":0,"allocated":117067776,"data_stored":136134066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037955,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":188024,"kb_used_data":142940,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93557128,"statfs":{"total":95995035648,"available":95802499072,"internally_reserved":0,"allocated":146370560,"data_stored":198073815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299590,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":221896,"kb_used_data":150444,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93523256,"statfs":{"total":95995035648,"available":95767814144,"internally_reserved":0,"allocated":154054656,"data_stored":183948351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528523,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218316,"kb_used_data":164916,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93526836,"statfs":{"total":95995035648,"available":95771480064,"internally_reserved":0,"allocated":168873984,"data_stored":226823092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757457,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":170220,"kb_used_data":116820,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93574932,"statfs":{"total":95995035648,"available":95820730368,"internally_reserved":0,"allocated":119623680,"data_stored":153778775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986391,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149916,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":200988,"kb_used_data":129540,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93544164,"statfs":{"total":95995035648,"available":95789223936,"internally_reserved":0,"allocated":132648960,"data_stored":178991574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411553,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":139260,"kb_used_data":85860,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93605892,"statfs":{"total":95995035648,"available":95852433408,"internally_reserved":0,"allocated":87920640,"data_stored":99314198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575078,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":141884,"kb_used_data":96800,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93603268,"statfs":{"total":95995035648,"available":95849746432,"internally_reserved":0,"allocated":99123200,"data_stored":115120531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738604,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":174908,"kb_used_data":113188,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":93570244,"statfs":{"total":95995035648,"available":95815929856,"internally_reserved":0,"allocated":115904512,"data_stored":139455762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106135552,"data_stored":133465885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88170496,"data_stored":107941576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73924608,"data_stored":88795014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":121331712,"data_stored":171459523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":96878592,"data_stored":122811969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":108453888,"data_stored":146390092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":160952320,"data_stored":223302915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":142897152,"data_stored":176568148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":135417856,"data_stored":190894860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":105111552,"data_stored":127651138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":54013952,"data_stored":67238510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":165511168,"data_stored":219832094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:52:07.203 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:52:07.203 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:07.211 INFO:tasks.rados.rados.2.smithi175.stdout:3757: done (5 left) 2024-09-11T03:52:07.211 INFO:tasks.rados.rados.2.smithi175.stdout:3758: done (4 left) 2024-09-11T03:52:07.211 INFO:tasks.rados.rados.2.smithi175.stdout:3760: done (3 left) 2024-09-11T03:52:07.211 INFO:tasks.rados.rados.2.smithi175.stdout:3761: done (2 left) 2024-09-11T03:52:07.211 INFO:tasks.rados.rados.2.smithi175.stdout:3762: done (1 left) 2024-09-11T03:52:07.211 INFO:tasks.rados.rados.2.smithi175.stdout:3763: done (0 left) 2024-09-11T03:52:07.211 INFO:tasks.rados.rados.2.smithi175.stdout:3764: write oid 31 current snap is 421 2024-09-11T03:52:07.211 INFO:tasks.rados.rados.2.smithi175.stdout:3764: seq_num 1272 ranges {578504=747679,2034272=566775,3130988=247516} 2024-09-11T03:52:07.217 INFO:tasks.rados.rados.2.smithi175.stdout:3764: writing smithi17557870-31 from 578504 to 1326183 tid 1 2024-09-11T03:52:07.224 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:07.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:07.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:07.225 INFO:tasks.rados.rados.2.smithi175.stdout:3764: writing smithi17557870-31 from 2034272 to 2601047 tid 2 2024-09-11T03:52:07.230 INFO:tasks.rados.rados.2.smithi175.stdout:3764: writing smithi17557870-31 from 3130988 to 3378504 tid 3 2024-09-11T03:52:07.230 INFO:tasks.rados.rados.2.smithi175.stdout:3765: read oid 18 snap -1 2024-09-11T03:52:07.230 INFO:tasks.rados.rados.2.smithi175.stdout:3765: expect deleted 2024-09-11T03:52:07.230 INFO:tasks.rados.rados.2.smithi175.stdout:3766: snap_remove snap 413 2024-09-11T03:52:07.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:07.350 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:07.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:07.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:07.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:07 smithi084 ceph-mon[27254]: osdmap e908: 12 total, 12 up, 11 in 2024-09-11T03:52:07.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:07 smithi084 ceph-mon[27254]: from='client.17478 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:52:07.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2361880228' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:52:07.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:07 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:07.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2361880228' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:52:07.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:07 smithi084 ceph-mon[27254]: osdmap e909: 12 total, 12 up, 11 in 2024-09-11T03:52:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:07 smithi121 ceph-mon[30729]: osdmap e908: 12 total, 12 up, 11 in 2024-09-11T03:52:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:07 smithi121 ceph-mon[30729]: from='client.17478 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:52:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2361880228' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:52:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:07 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2361880228' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:52:07.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:07 smithi121 ceph-mon[30729]: osdmap e909: 12 total, 12 up, 11 in 2024-09-11T03:52:07.452 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:07.473 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:07.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:07.474 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:07 smithi175 ceph-mon[30894]: osdmap e908: 12 total, 12 up, 11 in 2024-09-11T03:52:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:07 smithi175 ceph-mon[30894]: from='client.17478 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:52:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2361880228' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:52:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:07 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2361880228' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:52:07.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:07 smithi175 ceph-mon[30894]: osdmap e909: 12 total, 12 up, 11 in 2024-09-11T03:52:07.575 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:07.596 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:07.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:07.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:07.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:07.724 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:07.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:07.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:07.806 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:52:07.827 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:07.872 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:07.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:07.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:07.884 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:52:07.884 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:52:07.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:08.000 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:08.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:08.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:08.097 INFO:tasks.rados.rados.2.smithi175.stdout:3766: done (2 left) 2024-09-11T03:52:08.097 INFO:tasks.rados.rados.2.smithi175.stdout:3767: write (excl) oid 20 current snap is 421 2024-09-11T03:52:08.097 INFO:tasks.rados.rados.2.smithi175.stdout:3764: finishing write tid 1 to smithi17557870-31 2024-09-11T03:52:08.097 INFO:tasks.rados.rados.2.smithi175.stdout:3764: finishing write tid 2 to smithi17557870-31 2024-09-11T03:52:08.097 INFO:tasks.rados.rados.2.smithi175.stdout:3764: finishing write tid 3 to smithi17557870-31 2024-09-11T03:52:08.097 INFO:tasks.rados.rados.2.smithi175.stdout:3764: finishing write tid 4 to smithi17557870-31 2024-09-11T03:52:08.097 INFO:tasks.rados.rados.2.smithi175.stdout:3764: finishing write tid 5 to smithi17557870-31 2024-09-11T03:52:08.097 INFO:tasks.rados.rados.2.smithi175.stdout:3764: oid 31 updating version 0 to 1257 2024-09-11T03:52:08.097 INFO:tasks.rados.rados.2.smithi175.stdout:3764: oid 31 updating version 1257 to 1258 2024-09-11T03:52:08.098 INFO:tasks.rados.rados.2.smithi175.stdout:3764: oid 31 updating version 1258 to 1259 2024-09-11T03:52:08.098 INFO:tasks.rados.rados.2.smithi175.stdout:3764: oid 31 updating version 1259 to 1260 2024-09-11T03:52:08.098 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 1260 (ObjNum 1272 snap 421 seq_num 1272) dirty exists 2024-09-11T03:52:08.098 INFO:tasks.rados.rados.2.smithi175.stdout:3764: left oid 31 (ObjNum 1272 snap 421 seq_num 1272) 2024-09-11T03:52:08.098 INFO:tasks.rados.rados.2.smithi175.stdout:3767: seq_num 1273 ranges {595311=422928,1576401=710433,3042058=353253} 2024-09-11T03:52:08.101 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:08.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:08.103 INFO:tasks.rados.rados.2.smithi175.stdout:3767: writing smithi17557870-20 from 595311 to 1018239 tid 1 2024-09-11T03:52:08.110 INFO:tasks.rados.rados.2.smithi175.stdout:3767: writing smithi17557870-20 from 1576401 to 2286834 tid 2 2024-09-11T03:52:08.118 INFO:tasks.rados.rados.2.smithi175.stdout:3767: writing smithi17557870-20 from 3042058 to 3395311 tid 3 2024-09-11T03:52:08.118 INFO:tasks.rados.rados.2.smithi175.stdout:3764: done (2 left) 2024-09-11T03:52:08.118 INFO:tasks.rados.rados.2.smithi175.stdout:3765: done (1 left) 2024-09-11T03:52:08.118 INFO:tasks.rados.rados.2.smithi175.stdout:3768: rollback oid 50 current snap is 421 2024-09-11T03:52:08.118 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 50 to 393 2024-09-11T03:52:08.119 INFO:tasks.rados.rados.2.smithi175.stdout:3767: finishing write tid 1 to smithi17557870-20 2024-09-11T03:52:08.119 INFO:tasks.rados.rados.2.smithi175.stdout:3767: finishing write tid 2 to smithi17557870-20 2024-09-11T03:52:08.119 INFO:tasks.rados.rados.2.smithi175.stdout:3769: delete oid 23 current snap is 421 2024-09-11T03:52:08.119 INFO:tasks.rados.rados.2.smithi175.stdout:3769: done (2 left) 2024-09-11T03:52:08.120 INFO:tasks.rados.rados.2.smithi175.stdout:3770: snap_create 2024-09-11T03:52:08.121 INFO:tasks.rados.rados.2.smithi175.stdout:3768: finishing rollback tid 0 to smithi17557870-50 2024-09-11T03:52:08.121 INFO:tasks.rados.rados.2.smithi175.stdout:3767: finishing write tid 3 to smithi17557870-20 2024-09-11T03:52:08.122 INFO:tasks.rados.rados.2.smithi175.stdout:3767: finishing write tid 4 to smithi17557870-20 2024-09-11T03:52:08.122 INFO:tasks.rados.rados.2.smithi175.stdout:3768: finishing rollback tid 1 to smithi17557870-50 2024-09-11T03:52:08.122 INFO:tasks.rados.rados.2.smithi175.stdout:3767: finishing write tid 5 to smithi17557870-20 2024-09-11T03:52:08.122 INFO:tasks.rados.rados.2.smithi175.stdout:3767: oid 20 updating version 0 to 1059 2024-09-11T03:52:08.122 INFO:tasks.rados.rados.2.smithi175.stdout:3767: oid 20 updating version 1059 to 1062 2024-09-11T03:52:08.122 INFO:tasks.rados.rados.2.smithi175.stdout:3767: oid 20 version 1062 is already newer than 1060 2024-09-11T03:52:08.122 INFO:tasks.rados.rados.2.smithi175.stdout:3767: oid 20 version 1062 is already newer than 1061 2024-09-11T03:52:08.123 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 1062 (ObjNum 1273 snap 421 seq_num 1273) dirty exists 2024-09-11T03:52:08.123 INFO:tasks.rados.rados.2.smithi175.stdout:3767: left oid 20 (ObjNum 1273 snap 421 seq_num 1273) 2024-09-11T03:52:08.123 INFO:tasks.rados.rados.2.smithi175.stdout:3768: finishing rollback tid 2 to smithi17557870-50 2024-09-11T03:52:08.123 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 50 v 1534 (ObjNum 1190 snap 388 seq_num 1190) dirty exists 2024-09-11T03:52:08.129 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:08.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:08.231 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:08.252 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:08.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:08.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:08.354 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:08.376 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:08.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:08.377 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:08 smithi084 ceph-mon[27254]: pgmap v1518: 107 pgs: 107 active+clean; 736 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 9.9 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-11T03:52:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:08 smithi084 ceph-mon[27254]: 1.7 scrub starts 2024-09-11T03:52:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:08 smithi084 ceph-mon[27254]: 1.7 scrub ok 2024-09-11T03:52:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:08 smithi121 ceph-mon[30729]: pgmap v1518: 107 pgs: 107 active+clean; 736 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 9.9 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-11T03:52:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:08 smithi121 ceph-mon[30729]: 1.7 scrub starts 2024-09-11T03:52:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:08 smithi121 ceph-mon[30729]: 1.7 scrub ok 2024-09-11T03:52:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:08 smithi175 ceph-mon[30894]: pgmap v1518: 107 pgs: 107 active+clean; 736 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 9.9 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-11T03:52:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:08 smithi175 ceph-mon[30894]: 1.7 scrub starts 2024-09-11T03:52:08.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:08 smithi175 ceph-mon[30894]: 1.7 scrub ok 2024-09-11T03:52:08.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:08.504 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:08.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:08.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:08.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:08.632 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:08.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:08.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:08.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-11T03:52:08.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:08.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:08.760 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:08.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:08.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:08.887 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:08.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:08.989 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:09.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:09.012 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:09.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:09.113 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:09.114 INFO:tasks.rados.rados.2.smithi175.stdout:3767: done (2 left) 2024-09-11T03:52:09.114 INFO:tasks.rados.rados.2.smithi175.stdout:3768: done (1 left) 2024-09-11T03:52:09.114 INFO:tasks.rados.rados.2.smithi175.stdout:3770: done (0 left) 2024-09-11T03:52:09.114 INFO:tasks.rados.rados.2.smithi175.stdout:3771: read oid 22 snap 406 2024-09-11T03:52:09.114 INFO:tasks.rados.rados.2.smithi175.stdout:3771: expect (ObjNum 1212 snap 397 seq_num 1212) 2024-09-11T03:52:09.114 INFO:tasks.rados.rados.2.smithi175.stdout:3772: copy_from oid 8 from oid 7 current snap is 422 2024-09-11T03:52:09.115 INFO:tasks.rados.rados.2.smithi175.stdout:3773: write oid 16 current snap is 422 2024-09-11T03:52:09.115 INFO:tasks.rados.rados.2.smithi175.stdout:3773: seq_num 1275 ranges {628925=573426,1693365=714102,3096904=332021} 2024-09-11T03:52:09.119 INFO:tasks.rados.rados.2.smithi175.stdout:3773: writing smithi17557870-16 from 628925 to 1202351 tid 1 2024-09-11T03:52:09.127 INFO:tasks.rados.rados.2.smithi175.stdout:3773: writing smithi17557870-16 from 1693365 to 2407467 tid 2 2024-09-11T03:52:09.133 INFO:tasks.rados.rados.2.smithi175.stdout:3773: writing smithi17557870-16 from 3096904 to 3428925 tid 3 2024-09-11T03:52:09.133 INFO:tasks.rados.rados.2.smithi175.stdout:3774: read oid 9 snap -1 2024-09-11T03:52:09.133 INFO:tasks.rados.rados.2.smithi175.stdout:3774: expect (ObjNum 1260 snap 414 seq_num 1260) 2024-09-11T03:52:09.133 INFO:tasks.rados.rados.2.smithi175.stdout:3775: delete oid 37 current snap is 422 2024-09-11T03:52:09.133 INFO:tasks.rados.rados.2.smithi175.stdout:3772: finishing copy_from racing read to smithi17557870-8 2024-09-11T03:52:09.133 INFO:tasks.rados.rados.2.smithi175.stdout:3771: expect (ObjNum 1212 snap 397 seq_num 1212) 2024-09-11T03:52:09.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:09.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:09.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:09 smithi084 ceph-mon[27254]: 2.28 scrub starts 2024-09-11T03:52:09.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:09 smithi084 ceph-mon[27254]: 2.28 scrub ok 2024-09-11T03:52:09.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:09 smithi084 ceph-mon[27254]: 2.26 scrub starts 2024-09-11T03:52:09.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:09 smithi084 ceph-mon[27254]: 2.26 scrub ok 2024-09-11T03:52:09.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:09 smithi084 ceph-mon[27254]: osdmap e910: 12 total, 12 up, 11 in 2024-09-11T03:52:09.215 INFO:tasks.rados.rados.2.smithi175.stdout:3773: finishing write tid 1 to smithi17557870-16 2024-09-11T03:52:09.215 INFO:tasks.rados.rados.2.smithi175.stdout:3772: finishing copy_from to smithi17557870-8 2024-09-11T03:52:09.215 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 1011 (ObjNum 1220 snap 399 seq_num 1220) dirty exists 2024-09-11T03:52:09.215 INFO:tasks.rados.rados.2.smithi175.stdout:3773: finishing write tid 2 to smithi17557870-16 2024-09-11T03:52:09.215 INFO:tasks.rados.rados.2.smithi175.stdout:3773: finishing write tid 3 to smithi17557870-16 2024-09-11T03:52:09.215 INFO:tasks.rados.rados.2.smithi175.stdout:3773: finishing write tid 4 to smithi17557870-16 2024-09-11T03:52:09.215 INFO:tasks.rados.rados.2.smithi175.stdout:3771: done (4 left) 2024-09-11T03:52:09.215 INFO:tasks.rados.rados.2.smithi175.stdout:3772: done (3 left) 2024-09-11T03:52:09.216 INFO:tasks.rados.rados.2.smithi175.stdout:3775: done (2 left) 2024-09-11T03:52:09.216 INFO:tasks.rados.rados.2.smithi175.stdout:3776: read oid 29 snap -1 2024-09-11T03:52:09.216 INFO:tasks.rados.rados.2.smithi175.stdout:3776: expect deleted 2024-09-11T03:52:09.216 INFO:tasks.rados.rados.2.smithi175.stdout:3777: copy_from oid 26 from oid 27 current snap is 422 2024-09-11T03:52:09.216 INFO:tasks.rados.rados.2.smithi175.stdout:3773: finishing write tid 5 to smithi17557870-16 2024-09-11T03:52:09.216 INFO:tasks.rados.rados.2.smithi175.stdout:3773: oid 16 updating version 0 to 1006 2024-09-11T03:52:09.216 INFO:tasks.rados.rados.2.smithi175.stdout:3773: oid 16 version 1006 is already newer than 1004 2024-09-11T03:52:09.216 INFO:tasks.rados.rados.2.smithi175.stdout:3773: oid 16 version 1006 is already newer than 1005 2024-09-11T03:52:09.216 INFO:tasks.rados.rados.2.smithi175.stdout:3773: oid 16 updating version 1006 to 1007 2024-09-11T03:52:09.216 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 16 v 1007 (ObjNum 1275 snap 422 seq_num 1275) dirty exists 2024-09-11T03:52:09.216 INFO:tasks.rados.rados.2.smithi175.stdout:3773: left oid 16 (ObjNum 1275 snap 422 seq_num 1275) 2024-09-11T03:52:09.216 INFO:tasks.rados.rados.2.smithi175.stdout:3774: expect (ObjNum 1260 snap 414 seq_num 1260) 2024-09-11T03:52:09.245 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:09.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:09.269 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:09.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:09.355 INFO:tasks.rados.rados.2.smithi175.stdout:3777: finishing copy_from racing read to smithi17557870-26 2024-09-11T03:52:09.356 INFO:tasks.rados.rados.2.smithi175.stdout:3777: finishing copy_from to smithi17557870-26 2024-09-11T03:52:09.356 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 1252 (ObjNum 1258 snap 413 seq_num 1258) dirty exists 2024-09-11T03:52:09.356 INFO:tasks.rados.rados.2.smithi175.stdout:3773: done (3 left) 2024-09-11T03:52:09.356 INFO:tasks.rados.rados.2.smithi175.stdout:3774: done (2 left) 2024-09-11T03:52:09.356 INFO:tasks.rados.rados.2.smithi175.stdout:3776: done (1 left) 2024-09-11T03:52:09.356 INFO:tasks.rados.rados.2.smithi175.stdout:3777: done (0 left) 2024-09-11T03:52:09.356 INFO:tasks.rados.rados.2.smithi175.stdout:3778: delete oid 14 current snap is 422 2024-09-11T03:52:09.358 INFO:tasks.rados.rados.2.smithi175.stdout:3778: done (0 left) 2024-09-11T03:52:09.358 INFO:tasks.rados.rados.2.smithi175.stdout:3779: read oid 35 snap -1 2024-09-11T03:52:09.358 INFO:tasks.rados.rados.2.smithi175.stdout:3779: expect (ObjNum 1256 snap 412 seq_num 1256) 2024-09-11T03:52:09.358 INFO:tasks.rados.rados.2.smithi175.stdout:3780: rollback oid 17 current snap is 422 2024-09-11T03:52:09.358 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 17 to 391 2024-09-11T03:52:09.358 INFO:tasks.rados.rados.2.smithi175.stdout:3781: write (excl) oid 49 current snap is 422 2024-09-11T03:52:09.358 INFO:tasks.rados.rados.2.smithi175.stdout:3781: seq_num 1278 ranges {679346=515526,1868811=577782,3179173=300173} 2024-09-11T03:52:09.364 INFO:tasks.rados.rados.2.smithi175.stdout:3781: writing smithi17557870-49 from 679346 to 1194872 tid 1 2024-09-11T03:52:09.369 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:09.371 INFO:tasks.rados.rados.2.smithi175.stdout:3781: writing smithi17557870-49 from 1868811 to 2446593 tid 2 2024-09-11T03:52:09.377 INFO:tasks.rados.rados.2.smithi175.stdout:3781: writing smithi17557870-49 from 3179173 to 3479346 tid 3 2024-09-11T03:52:09.377 INFO:tasks.rados.rados.2.smithi175.stdout:3782: delete oid 45 current snap is 422 2024-09-11T03:52:09.377 INFO:tasks.rados.rados.2.smithi175.stdout:3780: finishing rollback tid 0 to smithi17557870-17 2024-09-11T03:52:09.377 INFO:tasks.rados.rados.2.smithi175.stdout:3780: finishing rollback tid 1 to smithi17557870-17 2024-09-11T03:52:09.377 INFO:tasks.rados.rados.2.smithi175.stdout:3780: finishing rollback tid 2 to smithi17557870-17 2024-09-11T03:52:09.378 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 17 v 1322 (ObjNum 1181 snap 385 seq_num 1181) dirty exists 2024-09-11T03:52:09.378 INFO:tasks.rados.rados.2.smithi175.stdout:3781: finishing write tid 1 to smithi17557870-49 2024-09-11T03:52:09.378 INFO:tasks.rados.rados.2.smithi175.stdout:3779: expect (ObjNum 1256 snap 412 seq_num 1256) 2024-09-11T03:52:09.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:09.392 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:09.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:09 smithi121 ceph-mon[30729]: 2.28 scrub starts 2024-09-11T03:52:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:09 smithi121 ceph-mon[30729]: 2.28 scrub ok 2024-09-11T03:52:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:09 smithi121 ceph-mon[30729]: 2.26 scrub starts 2024-09-11T03:52:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:09 smithi121 ceph-mon[30729]: 2.26 scrub ok 2024-09-11T03:52:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:09 smithi121 ceph-mon[30729]: osdmap e910: 12 total, 12 up, 11 in 2024-09-11T03:52:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:09 smithi175 ceph-mon[30894]: 2.28 scrub starts 2024-09-11T03:52:09.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:09 smithi175 ceph-mon[30894]: 2.28 scrub ok 2024-09-11T03:52:09.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:09 smithi175 ceph-mon[30894]: 2.26 scrub starts 2024-09-11T03:52:09.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:09 smithi175 ceph-mon[30894]: 2.26 scrub ok 2024-09-11T03:52:09.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:09 smithi175 ceph-mon[30894]: osdmap e910: 12 total, 12 up, 11 in 2024-09-11T03:52:09.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:09.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:09.520 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:09.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:09.622 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:09.643 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:09.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:09.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:09.652 INFO:tasks.rados.rados.2.smithi175.stdout:3781: finishing write tid 2 to smithi17557870-49 2024-09-11T03:52:09.652 INFO:tasks.rados.rados.2.smithi175.stdout:3781: finishing write tid 3 to smithi17557870-49 2024-09-11T03:52:09.652 INFO:tasks.rados.rados.2.smithi175.stdout:3781: finishing write tid 4 to smithi17557870-49 2024-09-11T03:52:09.652 INFO:tasks.rados.rados.2.smithi175.stdout:3781: finishing write tid 5 to smithi17557870-49 2024-09-11T03:52:09.652 INFO:tasks.rados.rados.2.smithi175.stdout:3781: oid 49 updating version 0 to 1009 2024-09-11T03:52:09.652 INFO:tasks.rados.rados.2.smithi175.stdout:3781: oid 49 updating version 1009 to 1011 2024-09-11T03:52:09.652 INFO:tasks.rados.rados.2.smithi175.stdout:3781: oid 49 version 1011 is already newer than 1010 2024-09-11T03:52:09.652 INFO:tasks.rados.rados.2.smithi175.stdout:3781: oid 49 version 1011 is already newer than 1008 2024-09-11T03:52:09.652 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 1011 (ObjNum 1278 snap 422 seq_num 1278) dirty exists 2024-09-11T03:52:09.653 INFO:tasks.rados.rados.2.smithi175.stdout:3781: left oid 49 (ObjNum 1278 snap 422 seq_num 1278) 2024-09-11T03:52:09.653 INFO:tasks.rados.rados.2.smithi175.stdout:3779: done (3 left) 2024-09-11T03:52:09.653 INFO:tasks.rados.rados.2.smithi175.stdout:3780: done (2 left) 2024-09-11T03:52:09.653 INFO:tasks.rados.rados.2.smithi175.stdout:3781: done (1 left) 2024-09-11T03:52:09.653 INFO:tasks.rados.rados.2.smithi175.stdout:3782: done (0 left) 2024-09-11T03:52:09.653 INFO:tasks.rados.rados.2.smithi175.stdout:3783: read oid 42 snap 361 2024-09-11T03:52:09.653 INFO:tasks.rados.rados.2.smithi175.stdout:3783: expect (ObjNum 1105 snap 358 seq_num 1105) 2024-09-11T03:52:09.653 INFO:tasks.rados.rados.2.smithi175.stdout:3784: read oid 13 snap -1 2024-09-11T03:52:09.653 INFO:tasks.rados.rados.2.smithi175.stdout:3784: expect deleted 2024-09-11T03:52:09.653 INFO:tasks.rados.rados.2.smithi175.stdout:3785: read oid 48 snap -1 2024-09-11T03:52:09.653 INFO:tasks.rados.rados.2.smithi175.stdout:3785: expect (ObjNum 1212 snap 397 seq_num 1212) 2024-09-11T03:52:09.653 INFO:tasks.rados.rados.2.smithi175.stdout:3786: write (excl) oid 2 current snap is 422 2024-09-11T03:52:09.653 INFO:tasks.rados.rados.2.smithi175.stdout:3786: seq_num 1280 ranges {712960=666024,1985775=581451,3234019=278941} 2024-09-11T03:52:09.659 INFO:tasks.rados.rados.2.smithi175.stdout:3786: writing smithi17557870-2 from 712960 to 1378984 tid 1 2024-09-11T03:52:09.666 INFO:tasks.rados.rados.2.smithi175.stdout:3786: writing smithi17557870-2 from 1985775 to 2567226 tid 2 2024-09-11T03:52:09.671 INFO:tasks.rados.rados.2.smithi175.stdout:3786: writing smithi17557870-2 from 3234019 to 3512960 tid 3 2024-09-11T03:52:09.671 INFO:tasks.rados.rados.2.smithi175.stdout:3787: write (excl) oid 10 current snap is 422 2024-09-11T03:52:09.672 INFO:tasks.rados.rados.2.smithi175.stdout:3787: seq_num 1281 ranges {729767=741273,1927904=725109,3145089=384678} 2024-09-11T03:52:09.679 INFO:tasks.rados.rados.2.smithi175.stdout:3787: writing smithi17557870-10 from 729767 to 1471040 tid 1 2024-09-11T03:52:09.685 INFO:tasks.rados.rados.2.smithi175.stdout:3787: writing smithi17557870-10 from 1927904 to 2653013 tid 2 2024-09-11T03:52:09.690 INFO:tasks.rados.rados.2.smithi175.stdout:3787: writing smithi17557870-10 from 3145089 to 3529767 tid 3 2024-09-11T03:52:09.690 INFO:tasks.rados.rados.2.smithi175.stdout:3788: write oid 12 current snap is 422 2024-09-11T03:52:09.690 INFO:tasks.rados.rados.2.smithi175.stdout:3788: seq_num 1282 ranges {746574=416522,1586386=468767,2772512=629544} 2024-09-11T03:52:09.696 INFO:tasks.rados.rados.2.smithi175.stdout:3788: writing smithi17557870-12 from 746574 to 1163096 tid 1 2024-09-11T03:52:09.701 INFO:tasks.rados.rados.2.smithi175.stdout:3788: writing smithi17557870-12 from 1586386 to 2055153 tid 2 2024-09-11T03:52:09.708 INFO:tasks.rados.rados.2.smithi175.stdout:3788: writing smithi17557870-12 from 2772512 to 3402056 tid 3 2024-09-11T03:52:09.708 INFO:tasks.rados.rados.2.smithi175.stdout:3789: snap_create 2024-09-11T03:52:09.708 INFO:tasks.rados.rados.2.smithi175.stdout:3783: expect (ObjNum 1105 snap 358 seq_num 1105) 2024-09-11T03:52:09.719 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:52:09.720 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:52:09.745 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:09.766 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:09.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:09.767 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:09.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:09.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:09.895 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:09.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:09.896 INFO:tasks.rados.rados.2.smithi175.stdout:3785: expect (ObjNum 1212 snap 397 seq_num 1212) 2024-09-11T03:52:09.962 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:09.973 INFO:tasks.rados.rados.2.smithi175.stdout:3786: finishing write tid 1 to smithi17557870-2 2024-09-11T03:52:09.973 INFO:tasks.rados.rados.2.smithi175.stdout:3786: finishing write tid 2 to smithi17557870-2 2024-09-11T03:52:09.973 INFO:tasks.rados.rados.2.smithi175.stdout:3786: finishing write tid 3 to smithi17557870-2 2024-09-11T03:52:09.973 INFO:tasks.rados.rados.2.smithi175.stdout:3786: finishing write tid 4 to smithi17557870-2 2024-09-11T03:52:09.973 INFO:tasks.rados.rados.2.smithi175.stdout:3786: finishing write tid 5 to smithi17557870-2 2024-09-11T03:52:09.973 INFO:tasks.rados.rados.2.smithi175.stdout:3786: oid 2 updating version 0 to 1336 2024-09-11T03:52:09.973 INFO:tasks.rados.rados.2.smithi175.stdout:3786: oid 2 updating version 1336 to 1337 2024-09-11T03:52:09.974 INFO:tasks.rados.rados.2.smithi175.stdout:3786: oid 2 version 1337 is already newer than 1335 2024-09-11T03:52:09.974 INFO:tasks.rados.rados.2.smithi175.stdout:3786: oid 2 version 1337 is already newer than 1334 2024-09-11T03:52:09.974 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 1337 (ObjNum 1280 snap 422 seq_num 1280) dirty exists 2024-09-11T03:52:09.974 INFO:tasks.rados.rados.2.smithi175.stdout:3786: left oid 2 (ObjNum 1280 snap 422 seq_num 1280) 2024-09-11T03:52:09.974 INFO:tasks.rados.rados.2.smithi175.stdout:3787: finishing write tid 1 to smithi17557870-10 2024-09-11T03:52:09.974 INFO:tasks.rados.rados.2.smithi175.stdout:3787: finishing write tid 2 to smithi17557870-10 2024-09-11T03:52:09.974 INFO:tasks.rados.rados.2.smithi175.stdout:3787: finishing write tid 3 to smithi17557870-10 2024-09-11T03:52:09.974 INFO:tasks.rados.rados.2.smithi175.stdout:3787: finishing write tid 4 to smithi17557870-10 2024-09-11T03:52:09.975 INFO:tasks.rados.rados.2.smithi175.stdout:3787: finishing write tid 5 to smithi17557870-10 2024-09-11T03:52:09.975 INFO:tasks.rados.rados.2.smithi175.stdout:3787: oid 10 updating version 0 to 1327 2024-09-11T03:52:09.975 INFO:tasks.rados.rados.2.smithi175.stdout:3787: oid 10 version 1327 is already newer than 1324 2024-09-11T03:52:09.975 INFO:tasks.rados.rados.2.smithi175.stdout:3787: oid 10 version 1327 is already newer than 1326 2024-09-11T03:52:09.975 INFO:tasks.rados.rados.2.smithi175.stdout:3787: oid 10 version 1327 is already newer than 1325 2024-09-11T03:52:09.975 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 10 v 1327 (ObjNum 1281 snap 422 seq_num 1281) dirty exists 2024-09-11T03:52:09.975 INFO:tasks.rados.rados.2.smithi175.stdout:3787: left oid 10 (ObjNum 1281 snap 422 seq_num 1281) 2024-09-11T03:52:09.975 INFO:tasks.rados.rados.2.smithi175.stdout:3788: finishing write tid 1 to smithi17557870-12 2024-09-11T03:52:09.976 INFO:tasks.rados.rados.2.smithi175.stdout:3788: finishing write tid 2 to smithi17557870-12 2024-09-11T03:52:09.976 INFO:tasks.rados.rados.2.smithi175.stdout:3788: finishing write tid 3 to smithi17557870-12 2024-09-11T03:52:09.976 INFO:tasks.rados.rados.2.smithi175.stdout:3788: finishing write tid 4 to smithi17557870-12 2024-09-11T03:52:09.976 INFO:tasks.rados.rados.2.smithi175.stdout:3788: finishing write tid 5 to smithi17557870-12 2024-09-11T03:52:09.976 INFO:tasks.rados.rados.2.smithi175.stdout:3788: oid 12 updating version 0 to 1552 2024-09-11T03:52:09.976 INFO:tasks.rados.rados.2.smithi175.stdout:3788: oid 12 version 1552 is already newer than 1550 2024-09-11T03:52:09.976 INFO:tasks.rados.rados.2.smithi175.stdout:3788: oid 12 updating version 1552 to 1553 2024-09-11T03:52:09.976 INFO:tasks.rados.rados.2.smithi175.stdout:3788: oid 12 version 1553 is already newer than 1551 2024-09-11T03:52:09.976 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 1553 (ObjNum 1282 snap 422 seq_num 1282) dirty exists 2024-09-11T03:52:09.976 INFO:tasks.rados.rados.2.smithi175.stdout:3788: left oid 12 (ObjNum 1282 snap 422 seq_num 1282) 2024-09-11T03:52:09.980 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:09.996 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:10.017 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:10.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:10.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:10.114 INFO:tasks.rados.rados.2.smithi175.stdout:3783: done (6 left) 2024-09-11T03:52:10.114 INFO:tasks.rados.rados.2.smithi175.stdout:3784: done (5 left) 2024-09-11T03:52:10.114 INFO:tasks.rados.rados.2.smithi175.stdout:3785: done (4 left) 2024-09-11T03:52:10.115 INFO:tasks.rados.rados.2.smithi175.stdout:3786: done (3 left) 2024-09-11T03:52:10.115 INFO:tasks.rados.rados.2.smithi175.stdout:3787: done (2 left) 2024-09-11T03:52:10.115 INFO:tasks.rados.rados.2.smithi175.stdout:3788: done (1 left) 2024-09-11T03:52:10.115 INFO:tasks.rados.rados.2.smithi175.stdout:3789: done (0 left) 2024-09-11T03:52:10.115 INFO:tasks.rados.rados.2.smithi175.stdout:3790: read oid 47 snap 422 2024-09-11T03:52:10.115 INFO:tasks.rados.rados.2.smithi175.stdout:3790: expect (ObjNum 1188 snap 388 seq_num 1188) 2024-09-11T03:52:10.115 INFO:tasks.rados.rados.2.smithi175.stdout:3791: read oid 34 snap -1 2024-09-11T03:52:10.115 INFO:tasks.rados.rados.2.smithi175.stdout:3791: expect deleted 2024-09-11T03:52:10.115 INFO:tasks.rados.rados.2.smithi175.stdout:3792: read oid 5 snap -1 2024-09-11T03:52:10.115 INFO:tasks.rados.rados.2.smithi175.stdout:3792: expect (ObjNum 1268 snap 420 seq_num 1268) 2024-09-11T03:52:10.115 INFO:tasks.rados.rados.2.smithi175.stdout:3793: read oid 16 snap 406 2024-09-11T03:52:10.115 INFO:tasks.rados.rados.2.smithi175.stdout:3793: expect (ObjNum 1218 snap 399 seq_num 1218) 2024-09-11T03:52:10.115 INFO:tasks.rados.rados.2.smithi175.stdout:3794: write oid 35 current snap is 423 2024-09-11T03:52:10.115 INFO:tasks.rados.rados.2.smithi175.stdout:3794: seq_num 1283 ranges {763381=491771,2044868=728778,3199935=363446} 2024-09-11T03:52:10.119 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:10.120 INFO:tasks.rados.rados.2.smithi175.stdout:3794: writing smithi17557870-35 from 763381 to 1255152 tid 1 2024-09-11T03:52:10.129 INFO:tasks.rados.rados.2.smithi175.stdout:3794: writing smithi17557870-35 from 2044868 to 2773646 tid 2 2024-09-11T03:52:10.134 INFO:tasks.rados.rados.2.smithi175.stdout:3794: writing smithi17557870-35 from 3199935 to 3563381 tid 3 2024-09-11T03:52:10.134 INFO:tasks.rados.rados.2.smithi175.stdout:3795: delete oid 38 current snap is 423 2024-09-11T03:52:10.134 INFO:tasks.rados.rados.2.smithi175.stdout:3792: expect (ObjNum 1268 snap 420 seq_num 1268) 2024-09-11T03:52:10.140 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:10.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:10.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:10.242 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:10.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:10.266 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:10.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:10.274 INFO:tasks.rados.rados.2.smithi175.stdout:3790: expect (ObjNum 1188 snap 388 seq_num 1188) 2024-09-11T03:52:10.367 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:10.388 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:10.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:10.389 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:10.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:10 smithi084 ceph-mon[27254]: 2.2c scrub starts 2024-09-11T03:52:10.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:10 smithi084 ceph-mon[27254]: 2.2c scrub ok 2024-09-11T03:52:10.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:10 smithi084 ceph-mon[27254]: 1.5 scrub starts 2024-09-11T03:52:10.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:10 smithi084 ceph-mon[27254]: 1.5 scrub ok 2024-09-11T03:52:10.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:10 smithi084 ceph-mon[27254]: 1.1d scrub starts 2024-09-11T03:52:10.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:10 smithi084 ceph-mon[27254]: 1.1d scrub ok 2024-09-11T03:52:10.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:10 smithi084 ceph-mon[27254]: pgmap v1521: 107 pgs: 107 active+clean; 729 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 27 MiB/s rd, 2.2 MiB/s wr, 27 op/s 2024-09-11T03:52:10.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:10 smithi084 ceph-mon[27254]: osdmap e911: 12 total, 12 up, 11 in 2024-09-11T03:52:10.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:10 smithi084 ceph-mon[27254]: from='client.17490 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:52:10.400 INFO:tasks.rados.rados.2.smithi175.stdout:3793: expect (ObjNum 1218 snap 399 seq_num 1218) 2024-09-11T03:52:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:10 smithi121 ceph-mon[30729]: 2.2c scrub starts 2024-09-11T03:52:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:10 smithi121 ceph-mon[30729]: 2.2c scrub ok 2024-09-11T03:52:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:10 smithi121 ceph-mon[30729]: 1.5 scrub starts 2024-09-11T03:52:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:10 smithi121 ceph-mon[30729]: 1.5 scrub ok 2024-09-11T03:52:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:10 smithi121 ceph-mon[30729]: 1.1d scrub starts 2024-09-11T03:52:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:10 smithi121 ceph-mon[30729]: 1.1d scrub ok 2024-09-11T03:52:10.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:10 smithi121 ceph-mon[30729]: pgmap v1521: 107 pgs: 107 active+clean; 729 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 27 MiB/s rd, 2.2 MiB/s wr, 27 op/s 2024-09-11T03:52:10.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:10 smithi121 ceph-mon[30729]: osdmap e911: 12 total, 12 up, 11 in 2024-09-11T03:52:10.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:10 smithi121 ceph-mon[30729]: from='client.17490 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:52:10.477 INFO:tasks.rados.rados.2.smithi175.stdout:3794: finishing write tid 1 to smithi17557870-35 2024-09-11T03:52:10.477 INFO:tasks.rados.rados.2.smithi175.stdout:3794: finishing write tid 2 to smithi17557870-35 2024-09-11T03:52:10.477 INFO:tasks.rados.rados.2.smithi175.stdout:3794: finishing write tid 3 to smithi17557870-35 2024-09-11T03:52:10.477 INFO:tasks.rados.rados.2.smithi175.stdout:3794: finishing write tid 4 to smithi17557870-35 2024-09-11T03:52:10.477 INFO:tasks.rados.rados.2.smithi175.stdout:3794: finishing write tid 5 to smithi17557870-35 2024-09-11T03:52:10.477 INFO:tasks.rados.rados.2.smithi175.stdout:3794: oid 35 updating version 0 to 1087 2024-09-11T03:52:10.477 INFO:tasks.rados.rados.2.smithi175.stdout:3794: oid 35 updating version 1087 to 1089 2024-09-11T03:52:10.477 INFO:tasks.rados.rados.2.smithi175.stdout:3794: oid 35 updating version 1089 to 1090 2024-09-11T03:52:10.478 INFO:tasks.rados.rados.2.smithi175.stdout:3794: oid 35 version 1090 is already newer than 1088 2024-09-11T03:52:10.478 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 35 v 1090 (ObjNum 1283 snap 423 seq_num 1283) dirty exists 2024-09-11T03:52:10.478 INFO:tasks.rados.rados.2.smithi175.stdout:3794: left oid 35 (ObjNum 1283 snap 423 seq_num 1283) 2024-09-11T03:52:10.478 INFO:tasks.rados.rados.2.smithi175.stdout:3790: done (5 left) 2024-09-11T03:52:10.478 INFO:tasks.rados.rados.2.smithi175.stdout:3791: done (4 left) 2024-09-11T03:52:10.478 INFO:tasks.rados.rados.2.smithi175.stdout:3792: done (3 left) 2024-09-11T03:52:10.478 INFO:tasks.rados.rados.2.smithi175.stdout:3793: done (2 left) 2024-09-11T03:52:10.478 INFO:tasks.rados.rados.2.smithi175.stdout:3794: done (1 left) 2024-09-11T03:52:10.478 INFO:tasks.rados.rados.2.smithi175.stdout:3795: done (0 left) 2024-09-11T03:52:10.478 INFO:tasks.rados.rados.2.smithi175.stdout:3796: write oid 41 current snap is 423 2024-09-11T03:52:10.478 INFO:tasks.rados.rados.2.smithi175.stdout:3796: seq_num 1285 ranges {796995=758622,2161832=732447} 2024-09-11T03:52:10.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:10 smithi175 ceph-mon[30894]: 2.2c scrub starts 2024-09-11T03:52:10.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:10 smithi175 ceph-mon[30894]: 2.2c scrub ok 2024-09-11T03:52:10.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:10 smithi175 ceph-mon[30894]: 1.5 scrub starts 2024-09-11T03:52:10.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:10 smithi175 ceph-mon[30894]: 1.5 scrub ok 2024-09-11T03:52:10.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:10 smithi175 ceph-mon[30894]: 1.1d scrub starts 2024-09-11T03:52:10.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:10 smithi175 ceph-mon[30894]: 1.1d scrub ok 2024-09-11T03:52:10.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:10 smithi175 ceph-mon[30894]: pgmap v1521: 107 pgs: 107 active+clean; 729 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 27 MiB/s rd, 2.2 MiB/s wr, 27 op/s 2024-09-11T03:52:10.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:10 smithi175 ceph-mon[30894]: osdmap e911: 12 total, 12 up, 11 in 2024-09-11T03:52:10.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:10 smithi175 ceph-mon[30894]: from='client.17490 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:52:10.485 INFO:tasks.rados.rados.2.smithi175.stdout:3796: writing smithi17557870-41 from 796995 to 1555617 tid 1 2024-09-11T03:52:10.490 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:10.493 INFO:tasks.rados.rados.2.smithi175.stdout:3796: writing smithi17557870-41 from 2161832 to 2894279 tid 2 2024-09-11T03:52:10.493 INFO:tasks.rados.rados.2.smithi175.stdout:3797: read oid 6 snap -1 2024-09-11T03:52:10.493 INFO:tasks.rados.rados.2.smithi175.stdout:3797: expect (ObjNum 1270 snap 420 seq_num 1270) 2024-09-11T03:52:10.493 INFO:tasks.rados.rados.2.smithi175.stdout:3798: write oid 39 current snap is 423 2024-09-11T03:52:10.493 INFO:tasks.rados.rados.2.smithi175.stdout:3796: finishing write tid 1 to smithi17557870-41 2024-09-11T03:52:10.493 INFO:tasks.rados.rados.2.smithi175.stdout:3798: seq_num 1286 ranges {413802=433871,1420314=476105,2482204=790985} 2024-09-11T03:52:10.497 INFO:tasks.rados.rados.2.smithi175.stdout:3798: writing smithi17557870-39 from 413802 to 847673 tid 1 2024-09-11T03:52:10.503 INFO:tasks.rados.rados.2.smithi175.stdout:3798: writing smithi17557870-39 from 1420314 to 1896419 tid 2 2024-09-11T03:52:10.512 INFO:tasks.rados.rados.2.smithi175.stdout:3798: writing smithi17557870-39 from 2482204 to 3273189 tid 3 2024-09-11T03:52:10.512 INFO:tasks.rados.rados.2.smithi175.stdout:3799: copy_from oid 16 from oid 37 current snap is 423 2024-09-11T03:52:10.512 INFO:tasks.rados.rados.2.smithi175.stdout:3796: finishing write tid 2 to smithi17557870-41 2024-09-11T03:52:10.512 INFO:tasks.rados.rados.2.smithi175.stdout:3796: finishing write tid 3 to smithi17557870-41 2024-09-11T03:52:10.512 INFO:tasks.rados.rados.2.smithi175.stdout:3796: finishing write tid 4 to smithi17557870-41 2024-09-11T03:52:10.512 INFO:tasks.rados.rados.2.smithi175.stdout:3796: oid 41 updating version 0 to 1027 2024-09-11T03:52:10.512 INFO:tasks.rados.rados.2.smithi175.stdout:3796: oid 41 updating version 1027 to 1029 2024-09-11T03:52:10.512 INFO:tasks.rados.rados.2.smithi175.stdout:3796: oid 41 version 1029 is already newer than 1028 2024-09-11T03:52:10.512 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 1029 (ObjNum 1285 snap 423 seq_num 1285) dirty exists 2024-09-11T03:52:10.512 INFO:tasks.rados.rados.2.smithi175.stdout:3796: left oid 41 (ObjNum 1285 snap 423 seq_num 1285) 2024-09-11T03:52:10.512 INFO:tasks.rados.rados.2.smithi175.stdout:3798: finishing write tid 1 to smithi17557870-39 2024-09-11T03:52:10.512 INFO:tasks.rados.rados.2.smithi175.stdout:3797: expect (ObjNum 1270 snap 420 seq_num 1270) 2024-09-11T03:52:10.515 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:10.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:10.516 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:10.544 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1522,"stamp":"2024-09-11T03:52:09.107460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":764605842,"num_objects":321,"num_object_clones":269,"num_object_copies":963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":10,"num_read":1010,"num_read_kb":1078526,"num_write":679,"num_write_kb":274946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":82584822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50618,"ondisk_log_size":50618,"up":321,"acting":321,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":321,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":1946264,"kb_used_data":1339444,"kb_used_omap":17,"kb_used_meta":606574,"kb_avail":1029250408,"statfs":{"total":1055945392128,"available":1053952417792,"internally_reserved":0,"allocated":1371590656,"data_stored":1730336980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17546,"internal_metadata":621132662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7464973,"num_objects":-4,"num_object_clones":-4,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":80,"num_read_kb":109368,"num_write":32,"num_write_kb":8901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.036415"},"pg_stats":[{"pgid":"2.4b","version":"568'646","reported_seq":4761,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.093556+0000","last_change":"2024-09-11T03:52:06.094966+0000","last_active":"2024-09-11T03:52:06.093556+0000","last_peered":"2024-09-11T03:52:06.093556+0000","last_clean":"2024-09-11T03:52:06.093556+0000","last_became_active":"2024-09-11T03:51:13.079045+0000","last_became_peered":"2024-09-11T03:51:13.079045+0000","last_unstale":"2024-09-11T03:52:06.093556+0000","last_undegraded":"2024-09-11T03:52:06.093556+0000","last_fullsized":"2024-09-11T03:52:06.093556+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:51:21.370186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.370186+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:41.532782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0094575070000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.4a","version":"848'386","reported_seq":4509,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.291508+0000","last_change":"2024-09-11T03:52:06.090022+0000","last_active":"2024-09-11T03:52:07.291508+0000","last_peered":"2024-09-11T03:52:07.291508+0000","last_clean":"2024-09-11T03:52:07.291508+0000","last_became_active":"2024-09-11T03:51:13.080958+0000","last_became_peered":"2024-09-11T03:51:13.080958+0000","last_unstale":"2024-09-11T03:52:07.291508+0000","last_undegraded":"2024-09-11T03:52:07.291508+0000","last_fullsized":"2024-09-11T03:52:07.291508+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"848'386","last_scrub_stamp":"2024-09-11T03:51:19.939163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.939163+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:40.349217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070724299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.49","version":"908'1530","reported_seq":5823,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.094600+0000","last_change":"2024-09-11T03:52:06.094805+0000","last_active":"2024-09-11T03:52:06.094600+0000","last_peered":"2024-09-11T03:52:06.094600+0000","last_clean":"2024-09-11T03:52:06.094600+0000","last_became_active":"2024-09-11T03:51:20.066997+0000","last_became_peered":"2024-09-11T03:51:20.066997+0000","last_unstale":"2024-09-11T03:52:06.094600+0000","last_undegraded":"2024-09-11T03:52:06.094600+0000","last_fullsized":"2024-09-11T03:52:06.094600+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":7,"last_scrub":"863'1411","last_scrub_stamp":"2024-09-11T03:51:21.422037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.422037+0000","objects_scrubbed":18,"log_size":1530,"log_dups_size":0,"ondisk_log_size":1530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:50.637426+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.010616591,"stat_sum":{"num_bytes":47711785,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":30,"num_read_kb":44216,"num_write":11,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.48","version":"744'374","reported_seq":3997,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.101326+0000","last_change":"2024-09-11T03:52:08.102236+0000","last_active":"2024-09-11T03:52:08.101326+0000","last_peered":"2024-09-11T03:52:08.101326+0000","last_clean":"2024-09-11T03:52:08.101326+0000","last_became_active":"2024-09-11T03:51:13.082676+0000","last_became_peered":"2024-09-11T03:51:13.082676+0000","last_unstale":"2024-09-11T03:52:08.101326+0000","last_undegraded":"2024-09-11T03:52:08.101326+0000","last_fullsized":"2024-09-11T03:52:08.101326+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"744'374","last_scrub_stamp":"2024-09-11T03:51:19.306681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.306681+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:28.816443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011360090000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,0],"acting":[9,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.47","version":"905'1316","reported_seq":5592,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.071977+0000","last_change":"2024-09-11T03:52:03.083110+0000","last_active":"2024-09-11T03:52:05.071977+0000","last_peered":"2024-09-11T03:52:05.071977+0000","last_clean":"2024-09-11T03:52:05.071977+0000","last_became_active":"2024-09-11T03:51:20.064024+0000","last_became_peered":"2024-09-11T03:51:20.064024+0000","last_unstale":"2024-09-11T03:52:05.071977+0000","last_undegraded":"2024-09-11T03:52:05.071977+0000","last_fullsized":"2024-09-11T03:52:05.071977+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":7,"last_scrub":"862'1274","last_scrub_stamp":"2024-09-11T03:51:20.166795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.166795+0000","objects_scrubbed":7,"log_size":1316,"log_dups_size":0,"ondisk_log_size":1316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:33.975312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0067661989999999997,"stat_sum":{"num_bytes":19286976,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":8,"num_read_kb":5764,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.46","version":"269'171","reported_seq":4236,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.079822+0000","last_change":"2024-09-11T03:52:06.080099+0000","last_active":"2024-09-11T03:52:06.079822+0000","last_peered":"2024-09-11T03:52:06.079822+0000","last_clean":"2024-09-11T03:52:06.079822+0000","last_became_active":"2024-09-11T03:51:13.080822+0000","last_became_peered":"2024-09-11T03:51:13.080822+0000","last_unstale":"2024-09-11T03:52:06.079822+0000","last_undegraded":"2024-09-11T03:52:06.079822+0000","last_fullsized":"2024-09-11T03:52:06.079822+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:51:21.034557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.034557+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:42.812432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017082620000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.45","version":"908'1238","reported_seq":5362,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.091653+0000","last_change":"2024-09-11T03:52:06.092001+0000","last_active":"2024-09-11T03:52:06.091653+0000","last_peered":"2024-09-11T03:52:06.091653+0000","last_clean":"2024-09-11T03:52:06.091653+0000","last_became_active":"2024-09-11T03:51:13.078704+0000","last_became_peered":"2024-09-11T03:51:13.078704+0000","last_unstale":"2024-09-11T03:52:06.091653+0000","last_undegraded":"2024-09-11T03:52:06.091653+0000","last_fullsized":"2024-09-11T03:52:06.091653+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"863'1213","last_scrub_stamp":"2024-09-11T03:51:23.430916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.430916+0000","objects_scrubbed":4,"log_size":1238,"log_dups_size":0,"ondisk_log_size":1238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:38.425320+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.007502908,"stat_sum":{"num_bytes":13996135,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":6,"num_write_kb":1354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[1,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.44","version":"908'758","reported_seq":4994,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.096755+0000","last_change":"2024-09-11T03:52:06.096933+0000","last_active":"2024-09-11T03:52:06.096755+0000","last_peered":"2024-09-11T03:52:06.096755+0000","last_clean":"2024-09-11T03:52:06.096755+0000","last_became_active":"2024-09-11T03:51:13.079319+0000","last_became_peered":"2024-09-11T03:51:13.079319+0000","last_unstale":"2024-09-11T03:52:06.096755+0000","last_undegraded":"2024-09-11T03:52:06.096755+0000","last_fullsized":"2024-09-11T03:52:06.096755+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"862'706","last_scrub_stamp":"2024-09-11T03:51:20.328187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.328187+0000","objects_scrubbed":12,"log_size":758,"log_dups_size":0,"ondisk_log_size":758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:29.541233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.010760077,"stat_sum":{"num_bytes":23780556,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":8,"num_write_kb":7615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,3],"acting":[1,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.43","version":"908'620","reported_seq":4765,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.090769+0000","last_change":"2024-09-11T03:52:06.091063+0000","last_active":"2024-09-11T03:52:06.090769+0000","last_peered":"2024-09-11T03:52:06.090769+0000","last_clean":"2024-09-11T03:52:06.090769+0000","last_became_active":"2024-09-11T03:51:13.083077+0000","last_became_peered":"2024-09-11T03:51:13.083077+0000","last_unstale":"2024-09-11T03:52:06.090769+0000","last_undegraded":"2024-09-11T03:52:06.090769+0000","last_fullsized":"2024-09-11T03:52:06.090769+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"862'574","last_scrub_stamp":"2024-09-11T03:51:22.027988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.027988+0000","objects_scrubbed":9,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:29.589923+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.012543205999999999,"stat_sum":{"num_bytes":26795402,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":11849,"num_write":8,"num_write_kb":11222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.42","version":"908'343","reported_seq":4470,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.080775+0000","last_change":"2024-09-11T03:52:06.081044+0000","last_active":"2024-09-11T03:52:06.080775+0000","last_peered":"2024-09-11T03:52:06.080775+0000","last_clean":"2024-09-11T03:52:06.080775+0000","last_became_active":"2024-09-11T03:51:13.080914+0000","last_became_peered":"2024-09-11T03:51:13.080914+0000","last_unstale":"2024-09-11T03:52:06.080775+0000","last_undegraded":"2024-09-11T03:52:06.080775+0000","last_fullsized":"2024-09-11T03:52:06.080775+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"859'323","last_scrub_stamp":"2024-09-11T03:51:19.628572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.628572+0000","objects_scrubbed":5,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:27.884690+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003218828,"stat_sum":{"num_bytes":15883770,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,11],"acting":[8,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.41","version":"567'235","reported_seq":4275,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.291026+0000","last_change":"2024-09-11T03:52:06.088508+0000","last_active":"2024-09-11T03:52:07.291026+0000","last_peered":"2024-09-11T03:52:07.291026+0000","last_clean":"2024-09-11T03:52:07.291026+0000","last_became_active":"2024-09-11T03:50:33.046488+0000","last_became_peered":"2024-09-11T03:50:33.046488+0000","last_unstale":"2024-09-11T03:52:07.291026+0000","last_undegraded":"2024-09-11T03:52:07.291026+0000","last_fullsized":"2024-09-11T03:52:07.291026+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":7,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:50:35.786143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.786143+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091889399999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.40","version":"563'192","reported_seq":4344,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.062871+0000","last_change":"2024-09-11T03:52:06.082462+0000","last_active":"2024-09-11T03:52:07.062871+0000","last_peered":"2024-09-11T03:52:07.062871+0000","last_clean":"2024-09-11T03:52:07.062871+0000","last_became_active":"2024-09-11T03:51:20.064866+0000","last_became_peered":"2024-09-11T03:51:20.064866+0000","last_unstale":"2024-09-11T03:52:07.062871+0000","last_undegraded":"2024-09-11T03:52:07.062871+0000","last_fullsized":"2024-09-11T03:52:07.062871+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":7,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:50:35.472175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.472175+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020413409999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,8],"acting":[6,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.3f","version":"903'807","reported_seq":4895,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.079345+0000","last_change":"2024-09-11T03:52:06.079864+0000","last_active":"2024-09-11T03:52:06.079345+0000","last_peered":"2024-09-11T03:52:06.079345+0000","last_clean":"2024-09-11T03:52:06.079345+0000","last_became_active":"2024-09-11T03:50:33.046798+0000","last_became_peered":"2024-09-11T03:50:33.046798+0000","last_unstale":"2024-09-11T03:52:06.079345+0000","last_undegraded":"2024-09-11T03:52:06.079345+0000","last_fullsized":"2024-09-11T03:52:06.079345+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"894'803","last_scrub_stamp":"2024-09-11T03:51:53.038706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.038706+0000","objects_scrubbed":5,"log_size":807,"log_dups_size":0,"ondisk_log_size":807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:07.485994+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001392653,"stat_sum":{"num_bytes":11068204,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":7511,"num_write":5,"num_write_kb":1340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.3e","version":"908'1102","reported_seq":5148,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.088012+0000","last_change":"2024-09-11T03:52:06.088241+0000","last_active":"2024-09-11T03:52:06.088012+0000","last_peered":"2024-09-11T03:52:06.088012+0000","last_clean":"2024-09-11T03:52:06.088012+0000","last_became_active":"2024-09-11T03:50:33.045235+0000","last_became_peered":"2024-09-11T03:50:33.045235+0000","last_unstale":"2024-09-11T03:52:06.088012+0000","last_undegraded":"2024-09-11T03:52:06.088012+0000","last_fullsized":"2024-09-11T03:52:06.088012+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"893'1098","last_scrub_stamp":"2024-09-11T03:51:52.144830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.144830+0000","objects_scrubbed":7,"log_size":1102,"log_dups_size":0,"ondisk_log_size":1102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:20.562377+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0073891920000000002,"stat_sum":{"num_bytes":16485077,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.3d","version":"567'752","reported_seq":3622,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.085256+0000","last_change":"2024-09-11T03:52:06.087514+0000","last_active":"2024-09-11T03:52:06.085256+0000","last_peered":"2024-09-11T03:52:06.085256+0000","last_clean":"2024-09-11T03:52:06.085256+0000","last_became_active":"2024-09-11T03:51:20.070161+0000","last_became_peered":"2024-09-11T03:51:20.070161+0000","last_unstale":"2024-09-11T03:52:06.085256+0000","last_undegraded":"2024-09-11T03:52:06.085256+0000","last_fullsized":"2024-09-11T03:52:06.085256+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:51:53.597428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.597428+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:01.449378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018820079999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.3c","version":"894'789","reported_seq":4889,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.078962+0000","last_change":"2024-09-11T03:52:06.079288+0000","last_active":"2024-09-11T03:52:06.078962+0000","last_peered":"2024-09-11T03:52:06.078962+0000","last_clean":"2024-09-11T03:52:06.078962+0000","last_became_active":"2024-09-11T03:50:33.048487+0000","last_became_peered":"2024-09-11T03:50:33.048487+0000","last_unstale":"2024-09-11T03:52:06.078962+0000","last_undegraded":"2024-09-11T03:52:06.078962+0000","last_fullsized":"2024-09-11T03:52:06.078962+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"886'783","last_scrub_stamp":"2024-09-11T03:51:45.009597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:45.009597+0000","objects_scrubbed":3,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:58.058560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012752099999999999,"stat_sum":{"num_bytes":10896744,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":3899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,10],"acting":[7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.3b","version":"905'970","reported_seq":5277,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.085564+0000","last_change":"2024-09-11T03:52:06.085960+0000","last_active":"2024-09-11T03:52:06.085564+0000","last_peered":"2024-09-11T03:52:06.085564+0000","last_clean":"2024-09-11T03:52:06.085564+0000","last_became_active":"2024-09-11T03:51:20.066390+0000","last_became_peered":"2024-09-11T03:51:20.066390+0000","last_unstale":"2024-09-11T03:52:06.085564+0000","last_undegraded":"2024-09-11T03:52:06.085564+0000","last_fullsized":"2024-09-11T03:52:06.085564+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"817'886","last_scrub_stamp":"2024-09-11T03:50:35.496439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.496439+0000","objects_scrubbed":8,"log_size":970,"log_dups_size":0,"ondisk_log_size":970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001484661,"stat_sum":{"num_bytes":21733134,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":39,"num_read_kb":50517,"num_write":18,"num_write_kb":2369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":14555611,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.3a","version":"557'273","reported_seq":3123,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.103394+0000","last_change":"2024-09-11T03:52:08.104227+0000","last_active":"2024-09-11T03:52:08.103394+0000","last_peered":"2024-09-11T03:52:08.103394+0000","last_clean":"2024-09-11T03:52:08.103394+0000","last_became_active":"2024-09-11T03:51:20.061387+0000","last_became_peered":"2024-09-11T03:51:20.061387+0000","last_unstale":"2024-09-11T03:52:08.103394+0000","last_undegraded":"2024-09-11T03:52:08.103394+0000","last_fullsized":"2024-09-11T03:52:08.103394+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:50:37.286055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.286055+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019623599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.39","version":"806'585","reported_seq":4728,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.103040+0000","last_change":"2024-09-11T03:52:08.103340+0000","last_active":"2024-09-11T03:52:08.103040+0000","last_peered":"2024-09-11T03:52:08.103040+0000","last_clean":"2024-09-11T03:52:08.103040+0000","last_became_active":"2024-09-11T03:50:33.043651+0000","last_became_peered":"2024-09-11T03:50:33.043651+0000","last_unstale":"2024-09-11T03:52:08.103040+0000","last_undegraded":"2024-09-11T03:52:08.103040+0000","last_fullsized":"2024-09-11T03:52:08.103040+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"806'585","last_scrub_stamp":"2024-09-11T03:50:36.264989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.264989+0000","objects_scrubbed":0,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012109340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,2],"acting":[9,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.38","version":"808'564","reported_seq":4714,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.077743+0000","last_change":"2024-09-11T03:52:06.078130+0000","last_active":"2024-09-11T03:52:06.077743+0000","last_peered":"2024-09-11T03:52:06.077743+0000","last_clean":"2024-09-11T03:52:06.077743+0000","last_became_active":"2024-09-11T03:50:33.042740+0000","last_became_peered":"2024-09-11T03:50:33.042740+0000","last_unstale":"2024-09-11T03:52:06.077743+0000","last_undegraded":"2024-09-11T03:52:06.077743+0000","last_fullsized":"2024-09-11T03:52:06.077743+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"808'564","last_scrub_stamp":"2024-09-11T03:51:54.462104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.462104+0000","objects_scrubbed":0,"log_size":564,"log_dups_size":0,"ondisk_log_size":564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:07.744445+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00084378999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,1],"acting":[8,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.37","version":"274'312","reported_seq":2688,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.290892+0000","last_change":"2024-09-11T03:52:06.092126+0000","last_active":"2024-09-11T03:52:07.290892+0000","last_peered":"2024-09-11T03:52:07.290892+0000","last_clean":"2024-09-11T03:52:07.290892+0000","last_became_active":"2024-09-11T03:50:07.077607+0000","last_became_peered":"2024-09-11T03:50:07.077607+0000","last_unstale":"2024-09-11T03:52:07.290892+0000","last_undegraded":"2024-09-11T03:52:07.290892+0000","last_fullsized":"2024-09-11T03:52:07.290892+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:51:23.036386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.036386+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:50.175869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000617232,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,9],"acting":[10,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.36","version":"908'712","reported_seq":4855,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.620984+0000","last_change":"2024-09-11T03:52:06.093149+0000","last_active":"2024-09-11T03:52:06.620984+0000","last_peered":"2024-09-11T03:52:06.620984+0000","last_clean":"2024-09-11T03:52:06.620984+0000","last_became_active":"2024-09-11T03:50:07.078667+0000","last_became_peered":"2024-09-11T03:50:07.078667+0000","last_unstale":"2024-09-11T03:52:06.620984+0000","last_undegraded":"2024-09-11T03:52:06.620984+0000","last_fullsized":"2024-09-11T03:52:06.620984+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"885'670","last_scrub_stamp":"2024-09-11T03:51:43.529996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:43.529996+0000","objects_scrubbed":15,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:56.608167+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0083439340000000008,"stat_sum":{"num_bytes":32741085,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":41,"num_read_kb":41151,"num_write":11,"num_write_kb":9394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,8],"acting":[1,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.35","version":"274'273","reported_seq":1855,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.092069+0000","last_change":"2024-09-11T03:52:06.092468+0000","last_active":"2024-09-11T03:52:06.092069+0000","last_peered":"2024-09-11T03:52:06.092069+0000","last_clean":"2024-09-11T03:52:06.092069+0000","last_became_active":"2024-09-11T03:50:07.079331+0000","last_became_peered":"2024-09-11T03:50:07.079331+0000","last_unstale":"2024-09-11T03:52:06.092069+0000","last_undegraded":"2024-09-11T03:52:06.092069+0000","last_fullsized":"2024-09-11T03:52:06.092069+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:51:28.526090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.526090+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:51.671288+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0074445800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.34","version":"905'402","reported_seq":4491,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.084376+0000","last_change":"2024-09-11T03:52:06.085078+0000","last_active":"2024-09-11T03:52:06.084376+0000","last_peered":"2024-09-11T03:52:06.084376+0000","last_clean":"2024-09-11T03:52:06.084376+0000","last_became_active":"2024-09-11T03:50:07.080419+0000","last_became_peered":"2024-09-11T03:50:07.080419+0000","last_unstale":"2024-09-11T03:52:06.084376+0000","last_undegraded":"2024-09-11T03:52:06.084376+0000","last_fullsized":"2024-09-11T03:52:06.084376+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"863'388","last_scrub_stamp":"2024-09-11T03:51:22.429302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.429302+0000","objects_scrubbed":6,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:36.630418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0028759689999999999,"stat_sum":{"num_bytes":13930425,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":37367,"num_write":8,"num_write_kb":1518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.33","version":"274'111","reported_seq":1643,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.091081+0000","last_change":"2024-09-11T03:52:06.091608+0000","last_active":"2024-09-11T03:52:06.091081+0000","last_peered":"2024-09-11T03:52:06.091081+0000","last_clean":"2024-09-11T03:52:06.091081+0000","last_became_active":"2024-09-11T03:50:07.077684+0000","last_became_peered":"2024-09-11T03:50:07.077684+0000","last_unstale":"2024-09-11T03:52:06.091081+0000","last_undegraded":"2024-09-11T03:52:06.091081+0000","last_fullsized":"2024-09-11T03:52:06.091081+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:51:28.001189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.001189+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:39.567684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.012542421,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.32","version":"908'592","reported_seq":4706,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.086267+0000","last_change":"2024-09-11T03:52:06.086557+0000","last_active":"2024-09-11T03:52:06.086267+0000","last_peered":"2024-09-11T03:52:06.086267+0000","last_clean":"2024-09-11T03:52:06.086267+0000","last_became_active":"2024-09-11T03:51:20.067057+0000","last_became_peered":"2024-09-11T03:51:20.067057+0000","last_unstale":"2024-09-11T03:52:06.086267+0000","last_undegraded":"2024-09-11T03:52:06.086267+0000","last_fullsized":"2024-09-11T03:52:06.086267+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"885'574","last_scrub_stamp":"2024-09-11T03:51:44.094173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:44.094173+0000","objects_scrubbed":3,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:53.589988+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0058409559999999996,"stat_sum":{"num_bytes":8252963,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":10412,"num_write":5,"num_write_kb":1096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":2772362,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.31","version":"274'107","reported_seq":2133,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.071495+0000","last_change":"2024-09-11T03:52:03.074630+0000","last_active":"2024-09-11T03:52:05.071495+0000","last_peered":"2024-09-11T03:52:05.071495+0000","last_clean":"2024-09-11T03:52:05.071495+0000","last_became_active":"2024-09-11T03:51:20.060052+0000","last_became_peered":"2024-09-11T03:51:20.060052+0000","last_unstale":"2024-09-11T03:52:05.071495+0000","last_undegraded":"2024-09-11T03:52:05.071495+0000","last_fullsized":"2024-09-11T03:52:05.071495+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:51:26.087112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:26.087112+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:42.193937+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00076377599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,7],"acting":[0,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.30","version":"274'67","reported_seq":614,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.290817+0000","last_change":"2024-09-11T03:52:06.088316+0000","last_active":"2024-09-11T03:52:07.290817+0000","last_peered":"2024-09-11T03:52:07.290817+0000","last_clean":"2024-09-11T03:52:07.290817+0000","last_became_active":"2024-09-11T03:51:17.065156+0000","last_became_peered":"2024-09-11T03:51:17.065156+0000","last_unstale":"2024-09-11T03:52:07.290817+0000","last_undegraded":"2024-09-11T03:52:07.290817+0000","last_fullsized":"2024-09-11T03:52:07.290817+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:51:21.989674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.989674+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:49.152610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001097029,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.2f","version":"557'465","reported_seq":3465,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.102489+0000","last_change":"2024-09-11T03:52:08.103058+0000","last_active":"2024-09-11T03:52:08.102489+0000","last_peered":"2024-09-11T03:52:08.102489+0000","last_clean":"2024-09-11T03:52:08.102489+0000","last_became_active":"2024-09-11T03:50:07.077840+0000","last_became_peered":"2024-09-11T03:50:07.077840+0000","last_unstale":"2024-09-11T03:52:08.102489+0000","last_undegraded":"2024-09-11T03:52:08.102489+0000","last_fullsized":"2024-09-11T03:52:08.102489+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:51:47.522706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:47.522706+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:56.188850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001185511,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.2e","version":"905'1207","reported_seq":5205,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.071868+0000","last_change":"2024-09-11T03:52:03.083246+0000","last_active":"2024-09-11T03:52:05.071868+0000","last_peered":"2024-09-11T03:52:05.071868+0000","last_clean":"2024-09-11T03:52:05.071868+0000","last_became_active":"2024-09-11T03:51:31.069593+0000","last_became_peered":"2024-09-11T03:51:31.069593+0000","last_unstale":"2024-09-11T03:52:05.071868+0000","last_undegraded":"2024-09-11T03:52:05.071868+0000","last_fullsized":"2024-09-11T03:52:05.071868+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":6,"last_scrub":"868'1145","last_scrub_stamp":"2024-09-11T03:51:27.034758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:27.034758+0000","objects_scrubbed":11,"log_size":1207,"log_dups_size":0,"ondisk_log_size":1207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:47.834881+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0079460750000000004,"stat_sum":{"num_bytes":30397185,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":34,"num_read_kb":34190,"num_write":25,"num_write_kb":8632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":24991924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,8],"acting":[0,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"2.2d","version":"908'1318","reported_seq":5390,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.063890+0000","last_change":"2024-09-11T03:52:06.092502+0000","last_active":"2024-09-11T03:52:07.063890+0000","last_peered":"2024-09-11T03:52:07.063890+0000","last_clean":"2024-09-11T03:52:07.063890+0000","last_became_active":"2024-09-11T03:49:29.163201+0000","last_became_peered":"2024-09-11T03:49:29.163201+0000","last_unstale":"2024-09-11T03:52:07.063890+0000","last_undegraded":"2024-09-11T03:52:07.063890+0000","last_fullsized":"2024-09-11T03:52:07.063890+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"837'1159","last_scrub_stamp":"2024-09-11T03:50:55.762457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:55.762457+0000","objects_scrubbed":20,"log_size":1318,"log_dups_size":0,"ondisk_log_size":1318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:16.962131+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0093135809999999996,"stat_sum":{"num_bytes":42858974,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":131,"num_read_kb":173218,"num_write":39,"num_write_kb":21285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.2c","version":"908'959","reported_seq":5037,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.084011+0000","last_change":"2024-09-11T03:52:06.084239+0000","last_active":"2024-09-11T03:52:06.084011+0000","last_peered":"2024-09-11T03:52:06.084011+0000","last_clean":"2024-09-11T03:52:06.084011+0000","last_became_active":"2024-09-11T03:49:29.154872+0000","last_became_peered":"2024-09-11T03:49:29.154872+0000","last_unstale":"2024-09-11T03:52:06.084011+0000","last_undegraded":"2024-09-11T03:52:06.084011+0000","last_fullsized":"2024-09-11T03:52:06.084011+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"816'878","last_scrub_stamp":"2024-09-11T03:50:37.544645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.544645+0000","objects_scrubbed":12,"log_size":959,"log_dups_size":0,"ondisk_log_size":959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0062762410000000001,"stat_sum":{"num_bytes":22924800,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":42,"num_read_kb":35340,"num_write":50,"num_write_kb":17751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3],"acting":[8,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.2b","version":"568'646","reported_seq":3873,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.105335+0000","last_change":"2024-09-11T03:52:08.105774+0000","last_active":"2024-09-11T03:52:08.105335+0000","last_peered":"2024-09-11T03:52:08.105335+0000","last_clean":"2024-09-11T03:52:08.105335+0000","last_became_active":"2024-09-11T03:51:20.062485+0000","last_became_peered":"2024-09-11T03:51:20.062485+0000","last_unstale":"2024-09-11T03:52:08.105335+0000","last_undegraded":"2024-09-11T03:52:08.105335+0000","last_fullsized":"2024-09-11T03:52:08.105335+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:51:52.556581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.556581+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:54.684702+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0028393049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.2a","version":"727'348","reported_seq":4332,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.078383+0000","last_change":"2024-09-11T03:52:06.078823+0000","last_active":"2024-09-11T03:52:06.078383+0000","last_peered":"2024-09-11T03:52:06.078383+0000","last_clean":"2024-09-11T03:52:06.078383+0000","last_became_active":"2024-09-11T03:49:29.160595+0000","last_became_peered":"2024-09-11T03:49:29.160595+0000","last_unstale":"2024-09-11T03:52:06.078383+0000","last_undegraded":"2024-09-11T03:52:06.078383+0000","last_fullsized":"2024-09-11T03:52:06.078383+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:51:05.940576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:05.940576+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:20.757006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078556399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,5],"acting":[7,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.29","version":"905'1341","reported_seq":5529,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.062943+0000","last_change":"2024-09-11T03:52:06.083882+0000","last_active":"2024-09-11T03:52:07.062943+0000","last_peered":"2024-09-11T03:52:07.062943+0000","last_clean":"2024-09-11T03:52:07.062943+0000","last_became_active":"2024-09-11T03:49:29.157194+0000","last_became_peered":"2024-09-11T03:49:29.157194+0000","last_unstale":"2024-09-11T03:52:07.062943+0000","last_undegraded":"2024-09-11T03:52:07.062943+0000","last_fullsized":"2024-09-11T03:52:07.062943+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"838'1272","last_scrub_stamp":"2024-09-11T03:50:57.773854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:57.773854+0000","objects_scrubbed":14,"log_size":1341,"log_dups_size":0,"ondisk_log_size":1341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:08.135460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024613119999999998,"stat_sum":{"num_bytes":34337514,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":53,"num_read_kb":69444,"num_write":41,"num_write_kb":13701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8],"acting":[6,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.28","version":"905'477","reported_seq":4563,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.084681+0000","last_change":"2024-09-11T03:52:06.085136+0000","last_active":"2024-09-11T03:52:06.084681+0000","last_peered":"2024-09-11T03:52:06.084681+0000","last_clean":"2024-09-11T03:52:06.084681+0000","last_became_active":"2024-09-11T03:49:29.154884+0000","last_became_peered":"2024-09-11T03:49:29.154884+0000","last_unstale":"2024-09-11T03:52:06.084681+0000","last_undegraded":"2024-09-11T03:52:06.084681+0000","last_fullsized":"2024-09-11T03:52:06.084681+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"847'448","last_scrub_stamp":"2024-09-11T03:51:05.333850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:05.333850+0000","objects_scrubbed":8,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:34.792912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00072579399999999998,"stat_sum":{"num_bytes":15310649,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":14,"num_read_kb":11529,"num_write":23,"num_write_kb":4588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.27","version":"908'1250","reported_seq":5547,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.088429+0000","last_change":"2024-09-11T03:52:06.088742+0000","last_active":"2024-09-11T03:52:06.088429+0000","last_peered":"2024-09-11T03:52:06.088429+0000","last_clean":"2024-09-11T03:52:06.088429+0000","last_became_active":"2024-09-11T03:51:20.062110+0000","last_became_peered":"2024-09-11T03:51:20.062110+0000","last_unstale":"2024-09-11T03:52:06.088429+0000","last_undegraded":"2024-09-11T03:52:06.088429+0000","last_fullsized":"2024-09-11T03:52:06.088429+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"840'1147","last_scrub_stamp":"2024-09-11T03:50:59.981154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:59.981154+0000","objects_scrubbed":8,"log_size":1250,"log_dups_size":0,"ondisk_log_size":1250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:07.133324+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.010319885000000001,"stat_sum":{"num_bytes":31213866,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":43,"num_read_kb":39064,"num_write":41,"num_write_kb":22316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":14737969,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.26","version":"269'171","reported_seq":4222,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.089207+0000","last_change":"2024-09-11T03:52:06.091035+0000","last_active":"2024-09-11T03:52:06.089207+0000","last_peered":"2024-09-11T03:52:06.089207+0000","last_clean":"2024-09-11T03:52:06.089207+0000","last_became_active":"2024-09-11T03:51:20.061265+0000","last_became_peered":"2024-09-11T03:51:20.061265+0000","last_unstale":"2024-09-11T03:52:06.089207+0000","last_undegraded":"2024-09-11T03:52:06.089207+0000","last_fullsized":"2024-09-11T03:52:06.089207+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:50:43.910786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:43.910786+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.011289662000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,5],"acting":[7,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.25","version":"908'1546","reported_seq":5806,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.092985+0000","last_change":"2024-09-11T03:52:06.093209+0000","last_active":"2024-09-11T03:52:06.092985+0000","last_peered":"2024-09-11T03:52:06.092985+0000","last_clean":"2024-09-11T03:52:06.092985+0000","last_became_active":"2024-09-11T03:49:29.159733+0000","last_became_peered":"2024-09-11T03:49:29.159733+0000","last_unstale":"2024-09-11T03:52:06.092985+0000","last_undegraded":"2024-09-11T03:52:06.092985+0000","last_fullsized":"2024-09-11T03:52:06.092985+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"840'1347","last_scrub_stamp":"2024-09-11T03:50:59.564459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:59.564459+0000","objects_scrubbed":21,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.015447792,"stat_sum":{"num_bytes":42981060,"num_objects":23,"num_object_clones":20,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":2,"num_read":76,"num_read_kb":67686,"num_write":70,"num_write_kb":20416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,6],"acting":[8,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.24","version":"908'730","reported_seq":4836,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.063675+0000","last_change":"2024-09-11T03:52:06.089076+0000","last_active":"2024-09-11T03:52:07.063675+0000","last_peered":"2024-09-11T03:52:07.063675+0000","last_clean":"2024-09-11T03:52:07.063675+0000","last_became_active":"2024-09-11T03:49:29.155740+0000","last_became_peered":"2024-09-11T03:49:29.155740+0000","last_unstale":"2024-09-11T03:52:07.063675+0000","last_undegraded":"2024-09-11T03:52:07.063675+0000","last_fullsized":"2024-09-11T03:52:07.063675+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"838'674","last_scrub_stamp":"2024-09-11T03:50:56.770184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:56.770184+0000","objects_scrubbed":7,"log_size":730,"log_dups_size":0,"ondisk_log_size":730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0074445780000000003,"stat_sum":{"num_bytes":15739462,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":14,"num_write_kb":11029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,7],"acting":[6,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.23","version":"908'556","reported_seq":4629,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.096582+0000","last_change":"2024-09-11T03:52:06.096908+0000","last_active":"2024-09-11T03:52:06.096582+0000","last_peered":"2024-09-11T03:52:06.096582+0000","last_clean":"2024-09-11T03:52:06.096582+0000","last_became_active":"2024-09-11T03:46:32.970100+0000","last_became_peered":"2024-09-11T03:46:32.970100+0000","last_unstale":"2024-09-11T03:52:06.096582+0000","last_undegraded":"2024-09-11T03:52:06.096582+0000","last_fullsized":"2024-09-11T03:52:06.096582+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"886'531","last_scrub_stamp":"2024-09-11T03:51:44.527530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:44.527530+0000","objects_scrubbed":5,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:06.401313+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.011028339,"stat_sum":{"num_bytes":11706449,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":57,"num_read_kb":64783,"num_write":39,"num_write_kb":13976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.22","version":"531'216","reported_seq":4270,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.063102+0000","last_change":"2024-09-11T03:52:06.089056+0000","last_active":"2024-09-11T03:52:07.063102+0000","last_peered":"2024-09-11T03:52:07.063102+0000","last_clean":"2024-09-11T03:52:07.063102+0000","last_became_active":"2024-09-11T03:46:32.970633+0000","last_became_peered":"2024-09-11T03:46:32.970633+0000","last_unstale":"2024-09-11T03:52:07.063102+0000","last_undegraded":"2024-09-11T03:52:07.063102+0000","last_fullsized":"2024-09-11T03:52:07.063102+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:50:38.617374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:38.617374+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0070261710000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.21","version":"907'408","reported_seq":4551,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.080509+0000","last_change":"2024-09-11T03:52:06.081819+0000","last_active":"2024-09-11T03:52:06.080509+0000","last_peered":"2024-09-11T03:52:06.080509+0000","last_clean":"2024-09-11T03:52:06.080509+0000","last_became_active":"2024-09-11T03:46:36.024928+0000","last_became_peered":"2024-09-11T03:46:36.024928+0000","last_unstale":"2024-09-11T03:52:06.080509+0000","last_undegraded":"2024-09-11T03:52:06.080509+0000","last_fullsized":"2024-09-11T03:52:06.080509+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":6,"last_scrub":"816'351","last_scrub_stamp":"2024-09-11T03:50:40.225380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:40.225380+0000","objects_scrubbed":7,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00078402099999999998,"stat_sum":{"num_bytes":19044405,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":55,"num_read_kb":56970,"num_write":50,"num_write_kb":21317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,8],"acting":[11,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.20","version":"908'356","reported_seq":4581,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.085295+0000","last_change":"2024-09-11T03:52:06.085616+0000","last_active":"2024-09-11T03:52:06.085295+0000","last_peered":"2024-09-11T03:52:06.085295+0000","last_clean":"2024-09-11T03:52:06.085295+0000","last_became_active":"2024-09-11T03:46:32.974375+0000","last_became_peered":"2024-09-11T03:46:32.974375+0000","last_unstale":"2024-09-11T03:52:06.085295+0000","last_undegraded":"2024-09-11T03:52:06.085295+0000","last_fullsized":"2024-09-11T03:52:06.085295+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"895'348","last_scrub_stamp":"2024-09-11T03:51:54.026522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.026522+0000","objects_scrubbed":6,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:10.244359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0077338119999999996,"stat_sum":{"num_bytes":19026910,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":98,"num_read_kb":144174,"num_write":25,"num_write_kb":13274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.1c","version":"806'776","reported_seq":4842,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.291875+0000","last_change":"2024-09-11T03:52:06.090118+0000","last_active":"2024-09-11T03:52:07.291875+0000","last_peered":"2024-09-11T03:52:07.291875+0000","last_clean":"2024-09-11T03:52:07.291875+0000","last_became_active":"2024-09-11T03:50:28.047548+0000","last_became_peered":"2024-09-11T03:50:28.047548+0000","last_unstale":"2024-09-11T03:52:07.291875+0000","last_undegraded":"2024-09-11T03:52:07.291875+0000","last_fullsized":"2024-09-11T03:52:07.291875+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"806'776","last_scrub_stamp":"2024-09-11T03:51:54.156180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.156180+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:10.485153+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00088502999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":875,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.071601+0000","last_change":"2024-09-11T03:50:55.035249+0000","last_active":"2024-09-11T03:52:05.071601+0000","last_peered":"2024-09-11T03:52:05.071601+0000","last_clean":"2024-09-11T03:52:05.071601+0000","last_became_active":"2024-09-11T03:46:15.974786+0000","last_became_peered":"2024-09-11T03:46:15.974786+0000","last_unstale":"2024-09-11T03:52:05.071601+0000","last_undegraded":"2024-09-11T03:52:05.071601+0000","last_fullsized":"2024-09-11T03:52:05.071601+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:55.035202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:55.035202+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:09.185590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.f","version":"557'465","reported_seq":4562,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.081144+0000","last_change":"2024-09-11T03:52:06.081395+0000","last_active":"2024-09-11T03:52:06.081144+0000","last_peered":"2024-09-11T03:52:06.081144+0000","last_clean":"2024-09-11T03:52:06.081144+0000","last_became_active":"2024-09-11T03:51:20.072191+0000","last_became_peered":"2024-09-11T03:51:20.072191+0000","last_unstale":"2024-09-11T03:52:06.081144+0000","last_undegraded":"2024-09-11T03:52:06.081144+0000","last_fullsized":"2024-09-11T03:52:06.081144+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:51:25.558029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:25.558029+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:37.823704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0035731790000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,11],"acting":[8,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.e","version":"782'1025","reported_seq":5074,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.104323+0000","last_change":"2024-09-11T03:52:08.105244+0000","last_active":"2024-09-11T03:52:08.104323+0000","last_peered":"2024-09-11T03:52:08.104323+0000","last_clean":"2024-09-11T03:52:08.104323+0000","last_became_active":"2024-09-11T03:50:02.051717+0000","last_became_peered":"2024-09-11T03:50:02.051717+0000","last_unstale":"2024-09-11T03:52:08.104323+0000","last_undegraded":"2024-09-11T03:52:08.104323+0000","last_fullsized":"2024-09-11T03:52:08.104323+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":0,"last_scrub":"782'1025","last_scrub_stamp":"2024-09-11T03:51:22.296245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.296245+0000","objects_scrubbed":0,"log_size":1025,"log_dups_size":0,"ondisk_log_size":1025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:46.727935+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0033174609999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":1460,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.078423+0000","last_change":"2024-09-11T03:51:17.066500+0000","last_active":"2024-09-11T03:52:06.078423+0000","last_peered":"2024-09-11T03:52:06.078423+0000","last_clean":"2024-09-11T03:52:06.078423+0000","last_became_active":"2024-09-11T03:51:17.066085+0000","last_became_peered":"2024-09-11T03:51:17.066085+0000","last_unstale":"2024-09-11T03:52:06.078423+0000","last_undegraded":"2024-09-11T03:52:06.078423+0000","last_fullsized":"2024-09-11T03:52:06.078423+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:58.932376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:58.932376+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:10.994958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"908'1057","reported_seq":5241,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.091050+0000","last_change":"2024-09-11T03:52:06.091435+0000","last_active":"2024-09-11T03:52:06.091050+0000","last_peered":"2024-09-11T03:52:06.091050+0000","last_clean":"2024-09-11T03:52:06.091050+0000","last_became_active":"2024-09-11T03:51:20.062633+0000","last_became_peered":"2024-09-11T03:51:20.062633+0000","last_unstale":"2024-09-11T03:52:06.091050+0000","last_undegraded":"2024-09-11T03:52:06.091050+0000","last_fullsized":"2024-09-11T03:52:06.091050+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"811'1014","last_scrub_stamp":"2024-09-11T03:50:37.438736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.438736+0000","objects_scrubbed":5,"log_size":1057,"log_dups_size":0,"ondisk_log_size":1057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0067388450000000003,"stat_sum":{"num_bytes":8728775,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":7026,"num_write":15,"num_write_kb":7796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":9438955,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":1175,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.078729+0000","last_change":"2024-09-11T03:51:17.063139+0000","last_active":"2024-09-11T03:52:06.078729+0000","last_peered":"2024-09-11T03:52:06.078729+0000","last_clean":"2024-09-11T03:52:06.078729+0000","last_became_active":"2024-09-11T03:51:17.061266+0000","last_became_peered":"2024-09-11T03:51:17.061266+0000","last_unstale":"2024-09-11T03:52:06.078729+0000","last_undegraded":"2024-09-11T03:52:06.078729+0000","last_fullsized":"2024-09-11T03:52:06.078729+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:38.843438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:38.843438+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:07.110555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"905'953","reported_seq":5030,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.080951+0000","last_change":"2024-09-11T03:52:06.081306+0000","last_active":"2024-09-11T03:52:06.080951+0000","last_peered":"2024-09-11T03:52:06.080951+0000","last_clean":"2024-09-11T03:52:06.080951+0000","last_became_active":"2024-09-11T03:49:24.115405+0000","last_became_peered":"2024-09-11T03:49:24.115405+0000","last_unstale":"2024-09-11T03:52:06.080951+0000","last_undegraded":"2024-09-11T03:52:06.080951+0000","last_fullsized":"2024-09-11T03:52:06.080951+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"843'880","last_scrub_stamp":"2024-09-11T03:51:01.987099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:01.987099+0000","objects_scrubbed":13,"log_size":953,"log_dups_size":0,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:21.010286+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000873587,"stat_sum":{"num_bytes":22929927,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":36,"num_read_kb":32542,"num_write":20,"num_write_kb":7053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.f","version":"20'92","reported_seq":743,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.064402+0000","last_change":"2024-09-11T03:51:21.860818+0000","last_active":"2024-09-11T03:52:07.064402+0000","last_peered":"2024-09-11T03:52:07.064402+0000","last_clean":"2024-09-11T03:52:07.064402+0000","last_became_active":"2024-09-11T03:51:17.064761+0000","last_became_peered":"2024-09-11T03:51:17.064761+0000","last_unstale":"2024-09-11T03:52:07.064402+0000","last_undegraded":"2024-09-11T03:52:07.064402+0000","last_fullsized":"2024-09-11T03:52:07.064402+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:21.860769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.860769+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:34.914441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":1859,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.078849+0000","last_change":"2024-09-11T03:51:52.035008+0000","last_active":"2024-09-11T03:52:06.078849+0000","last_peered":"2024-09-11T03:52:06.078849+0000","last_clean":"2024-09-11T03:52:06.078849+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:52:06.078849+0000","last_undegraded":"2024-09-11T03:52:06.078849+0000","last_fullsized":"2024-09-11T03:52:06.078849+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:52.034950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.034950+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:03.753329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"845'562","reported_seq":4646,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.085941+0000","last_change":"2024-09-11T03:52:06.086854+0000","last_active":"2024-09-11T03:52:06.085941+0000","last_peered":"2024-09-11T03:52:06.085941+0000","last_clean":"2024-09-11T03:52:06.085941+0000","last_became_active":"2024-09-11T03:51:08.055565+0000","last_became_peered":"2024-09-11T03:51:08.055565+0000","last_unstale":"2024-09-11T03:52:06.085941+0000","last_undegraded":"2024-09-11T03:52:06.085941+0000","last_fullsized":"2024-09-11T03:52:06.085941+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"845'562","last_scrub_stamp":"2024-09-11T03:51:20.441661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.441661+0000","objects_scrubbed":0,"log_size":562,"log_dups_size":0,"ondisk_log_size":562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:32.872445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015596620000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":1472,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.086263+0000","last_change":"2024-09-11T03:51:27.564638+0000","last_active":"2024-09-11T03:52:06.086263+0000","last_peered":"2024-09-11T03:52:06.086263+0000","last_clean":"2024-09-11T03:52:06.086263+0000","last_became_active":"2024-09-11T03:51:17.059234+0000","last_became_peered":"2024-09-11T03:51:17.059234+0000","last_unstale":"2024-09-11T03:52:06.086263+0000","last_undegraded":"2024-09-11T03:52:06.086263+0000","last_fullsized":"2024-09-11T03:52:06.086263+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:27.564586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:27.564586+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:54.532532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"563'192","reported_seq":3133,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.291069+0000","last_change":"2024-09-11T03:52:06.088864+0000","last_active":"2024-09-11T03:52:07.291069+0000","last_peered":"2024-09-11T03:52:07.291069+0000","last_clean":"2024-09-11T03:52:07.291069+0000","last_became_active":"2024-09-11T03:50:31.044399+0000","last_became_peered":"2024-09-11T03:50:31.044399+0000","last_unstale":"2024-09-11T03:52:07.291069+0000","last_undegraded":"2024-09-11T03:52:07.291069+0000","last_fullsized":"2024-09-11T03:52:07.291069+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:50:36.771360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.771360+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001119697,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":1428,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.071826+0000","last_change":"2024-09-11T03:51:17.061730+0000","last_active":"2024-09-11T03:52:05.071826+0000","last_peered":"2024-09-11T03:52:05.071826+0000","last_clean":"2024-09-11T03:52:05.071826+0000","last_became_active":"2024-09-11T03:51:17.061243+0000","last_became_peered":"2024-09-11T03:51:17.061243+0000","last_unstale":"2024-09-11T03:52:05.071826+0000","last_undegraded":"2024-09-11T03:52:05.071826+0000","last_fullsized":"2024-09-11T03:52:05.071826+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:41.975660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:41.975660+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"567'235","reported_seq":4282,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.071989+0000","last_change":"2024-09-11T03:52:03.074727+0000","last_active":"2024-09-11T03:52:05.071989+0000","last_peered":"2024-09-11T03:52:05.071989+0000","last_clean":"2024-09-11T03:52:05.071989+0000","last_became_active":"2024-09-11T03:50:28.039300+0000","last_became_peered":"2024-09-11T03:50:28.039300+0000","last_unstale":"2024-09-11T03:52:05.071989+0000","last_undegraded":"2024-09-11T03:52:05.071989+0000","last_fullsized":"2024-09-11T03:52:05.071989+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:51:50.027291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:50.027291+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:55.922517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044412500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":1036,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.291277+0000","last_change":"2024-09-11T03:51:29.996670+0000","last_active":"2024-09-11T03:52:07.291277+0000","last_peered":"2024-09-11T03:52:07.291277+0000","last_clean":"2024-09-11T03:52:07.291277+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:52:07.291277+0000","last_undegraded":"2024-09-11T03:52:07.291277+0000","last_fullsized":"2024-09-11T03:52:07.291277+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:29.996598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:29.996598+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:54.485931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"843'319","reported_seq":4504,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.084489+0000","last_change":"2024-09-11T03:52:06.084879+0000","last_active":"2024-09-11T03:52:06.084489+0000","last_peered":"2024-09-11T03:52:06.084489+0000","last_clean":"2024-09-11T03:52:06.084489+0000","last_became_active":"2024-09-11T03:51:08.055103+0000","last_became_peered":"2024-09-11T03:51:08.055103+0000","last_unstale":"2024-09-11T03:52:06.084489+0000","last_undegraded":"2024-09-11T03:52:06.084489+0000","last_fullsized":"2024-09-11T03:52:06.084489+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"843'319","last_scrub_stamp":"2024-09-11T03:51:19.453435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.453435+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:44.672021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014514650000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":720,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.064176+0000","last_change":"2024-09-11T03:49:37.637065+0000","last_active":"2024-09-11T03:52:07.064176+0000","last_peered":"2024-09-11T03:52:07.064176+0000","last_clean":"2024-09-11T03:52:07.064176+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:52:07.064176+0000","last_undegraded":"2024-09-11T03:52:07.064176+0000","last_fullsized":"2024-09-11T03:52:07.064176+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:37.637018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.637018+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:07.712022+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"847'690","reported_seq":4835,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.096960+0000","last_change":"2024-09-11T03:52:06.097265+0000","last_active":"2024-09-11T03:52:06.096960+0000","last_peered":"2024-09-11T03:52:06.096960+0000","last_clean":"2024-09-11T03:52:06.096960+0000","last_became_active":"2024-09-11T03:51:08.058954+0000","last_became_peered":"2024-09-11T03:51:08.058954+0000","last_unstale":"2024-09-11T03:52:06.096960+0000","last_undegraded":"2024-09-11T03:52:06.096960+0000","last_fullsized":"2024-09-11T03:52:06.096960+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"847'690","last_scrub_stamp":"2024-09-11T03:51:24.473723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:24.473723+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:30.895213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010738614,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":1047,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.292158+0000","last_change":"2024-09-11T03:51:53.155265+0000","last_active":"2024-09-11T03:52:07.292158+0000","last_peered":"2024-09-11T03:52:07.292158+0000","last_clean":"2024-09-11T03:52:07.292158+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:52:07.292158+0000","last_undegraded":"2024-09-11T03:52:07.292158+0000","last_fullsized":"2024-09-11T03:52:07.292158+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:53.155207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.155207+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:07.708610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"908'1259","reported_seq":5463,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.090667+0000","last_change":"2024-09-11T03:52:06.090862+0000","last_active":"2024-09-11T03:52:06.090667+0000","last_peered":"2024-09-11T03:52:06.090667+0000","last_clean":"2024-09-11T03:52:06.090667+0000","last_became_active":"2024-09-11T03:51:20.063427+0000","last_became_peered":"2024-09-11T03:51:20.063427+0000","last_unstale":"2024-09-11T03:52:06.090667+0000","last_undegraded":"2024-09-11T03:52:06.090667+0000","last_fullsized":"2024-09-11T03:52:06.090667+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"862'1215","last_scrub_stamp":"2024-09-11T03:51:21.601542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.601542+0000","objects_scrubbed":5,"log_size":1259,"log_dups_size":0,"ondisk_log_size":1259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:40.029001+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.012735454,"stat_sum":{"num_bytes":13495545,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":2299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":1030,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.292289+0000","last_change":"2024-09-11T03:49:36.654074+0000","last_active":"2024-09-11T03:52:07.292289+0000","last_peered":"2024-09-11T03:52:07.292289+0000","last_clean":"2024-09-11T03:52:07.292289+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:52:07.292289+0000","last_undegraded":"2024-09-11T03:52:07.292289+0000","last_fullsized":"2024-09-11T03:52:07.292289+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:36.654025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.654025+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:07.849922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":4238,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.092569+0000","last_change":"2024-09-11T03:52:06.093663+0000","last_active":"2024-09-11T03:52:06.092569+0000","last_peered":"2024-09-11T03:52:06.092569+0000","last_clean":"2024-09-11T03:52:06.092569+0000","last_became_active":"2024-09-11T03:51:08.055529+0000","last_became_peered":"2024-09-11T03:51:08.055529+0000","last_unstale":"2024-09-11T03:52:06.092569+0000","last_undegraded":"2024-09-11T03:52:06.092569+0000","last_fullsized":"2024-09-11T03:52:06.092569+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:51:22.407323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.407323+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:29.119528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085651900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":1636,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.080279+0000","last_change":"2024-09-11T03:51:31.102151+0000","last_active":"2024-09-11T03:52:06.080279+0000","last_peered":"2024-09-11T03:52:06.080279+0000","last_clean":"2024-09-11T03:52:06.080279+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:52:06.080279+0000","last_undegraded":"2024-09-11T03:52:06.080279+0000","last_fullsized":"2024-09-11T03:52:06.080279+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:31.102098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:31.102098+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:49.931850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"908'1332","reported_seq":5600,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.064048+0000","last_change":"2024-09-11T03:52:06.093775+0000","last_active":"2024-09-11T03:52:07.064048+0000","last_peered":"2024-09-11T03:52:07.064048+0000","last_clean":"2024-09-11T03:52:07.064048+0000","last_became_active":"2024-09-11T03:51:08.057350+0000","last_became_peered":"2024-09-11T03:51:08.057350+0000","last_unstale":"2024-09-11T03:52:07.064048+0000","last_undegraded":"2024-09-11T03:52:07.064048+0000","last_fullsized":"2024-09-11T03:52:07.064048+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'1274","last_scrub_stamp":"2024-09-11T03:51:19.872739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.872739+0000","objects_scrubbed":12,"log_size":1332,"log_dups_size":0,"ondisk_log_size":1332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:44.400628+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.010632146,"stat_sum":{"num_bytes":38444164,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":3103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":1439,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.080090+0000","last_change":"2024-09-11T03:51:47.050276+0000","last_active":"2024-09-11T03:52:06.080090+0000","last_peered":"2024-09-11T03:52:06.080090+0000","last_clean":"2024-09-11T03:52:06.080090+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:52:06.080090+0000","last_undegraded":"2024-09-11T03:52:06.080090+0000","last_fullsized":"2024-09-11T03:52:06.080090+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:47.050227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:47.050227+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:47.461383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"744'374","reported_seq":4519,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.088817+0000","last_change":"2024-09-11T03:52:06.089074+0000","last_active":"2024-09-11T03:52:06.088817+0000","last_peered":"2024-09-11T03:52:06.088817+0000","last_clean":"2024-09-11T03:52:06.088817+0000","last_became_active":"2024-09-11T03:51:08.057750+0000","last_became_peered":"2024-09-11T03:51:08.057750+0000","last_unstale":"2024-09-11T03:52:06.088817+0000","last_undegraded":"2024-09-11T03:52:06.088817+0000","last_fullsized":"2024-09-11T03:52:06.088817+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'374","last_scrub_stamp":"2024-09-11T03:51:20.010112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.010112+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:34.767588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010476519,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":1795,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.102558+0000","last_change":"2024-09-11T03:51:17.061700+0000","last_active":"2024-09-11T03:52:08.102558+0000","last_peered":"2024-09-11T03:52:08.102558+0000","last_clean":"2024-09-11T03:52:08.102558+0000","last_became_active":"2024-09-11T03:51:17.061257+0000","last_became_peered":"2024-09-11T03:51:17.061257+0000","last_unstale":"2024-09-11T03:52:08.102558+0000","last_undegraded":"2024-09-11T03:52:08.102558+0000","last_fullsized":"2024-09-11T03:52:08.102558+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:18.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.933952+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:39.424252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"846'1373","reported_seq":5574,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.093231+0000","last_change":"2024-09-11T03:52:06.093463+0000","last_active":"2024-09-11T03:52:06.093231+0000","last_peered":"2024-09-11T03:52:06.093231+0000","last_clean":"2024-09-11T03:52:06.093231+0000","last_became_active":"2024-09-11T03:51:31.065505+0000","last_became_peered":"2024-09-11T03:51:31.065505+0000","last_unstale":"2024-09-11T03:52:06.093231+0000","last_undegraded":"2024-09-11T03:52:06.093231+0000","last_fullsized":"2024-09-11T03:52:06.093231+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"846'1373","last_scrub_stamp":"2024-09-11T03:51:19.328459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.328459+0000","objects_scrubbed":0,"log_size":1373,"log_dups_size":0,"ondisk_log_size":1373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:39.565105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082286990000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":1854,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.077791+0000","last_change":"2024-09-11T03:51:04.911507+0000","last_active":"2024-09-11T03:52:06.077791+0000","last_peered":"2024-09-11T03:52:06.077791+0000","last_clean":"2024-09-11T03:52:06.077791+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:52:06.077791+0000","last_undegraded":"2024-09-11T03:52:06.077791+0000","last_fullsized":"2024-09-11T03:52:06.077791+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:04.911446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:04.911446+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:33.168597+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"905'421","reported_seq":4574,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.086631+0000","last_change":"2024-09-11T03:52:06.086889+0000","last_active":"2024-09-11T03:52:06.086631+0000","last_peered":"2024-09-11T03:52:06.086631+0000","last_clean":"2024-09-11T03:52:06.086631+0000","last_became_active":"2024-09-11T03:51:08.053749+0000","last_became_peered":"2024-09-11T03:51:08.053749+0000","last_unstale":"2024-09-11T03:52:06.086631+0000","last_undegraded":"2024-09-11T03:52:06.086631+0000","last_fullsized":"2024-09-11T03:52:06.086631+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'392","last_scrub_stamp":"2024-09-11T03:51:19.986753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.986753+0000","objects_scrubbed":5,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:29.123663+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0056447049999999999,"stat_sum":{"num_bytes":10459472,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":23665,"num_write":9,"num_write_kb":2863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":1513,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.082683+0000","last_change":"2024-09-11T03:51:55.165620+0000","last_active":"2024-09-11T03:52:06.082683+0000","last_peered":"2024-09-11T03:52:06.082683+0000","last_clean":"2024-09-11T03:52:06.082683+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:52:06.082683+0000","last_undegraded":"2024-09-11T03:52:06.082683+0000","last_fullsized":"2024-09-11T03:52:06.082683+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:55.165539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:55.165539+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:11.582309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"568'646","reported_seq":4788,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.078226+0000","last_change":"2024-09-11T03:52:06.078422+0000","last_active":"2024-09-11T03:52:06.078226+0000","last_peered":"2024-09-11T03:52:06.078226+0000","last_clean":"2024-09-11T03:52:06.078226+0000","last_became_active":"2024-09-11T03:51:08.054955+0000","last_became_peered":"2024-09-11T03:51:08.054955+0000","last_unstale":"2024-09-11T03:52:06.078226+0000","last_undegraded":"2024-09-11T03:52:06.078226+0000","last_fullsized":"2024-09-11T03:52:06.078226+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:51:20.608388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.608388+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:23.460225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088045899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,0],"acting":[8,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":1454,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.082296+0000","last_change":"2024-09-11T03:49:00.710015+0000","last_active":"2024-09-11T03:52:06.082296+0000","last_peered":"2024-09-11T03:52:06.082296+0000","last_clean":"2024-09-11T03:52:06.082296+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:52:06.082296+0000","last_undegraded":"2024-09-11T03:52:06.082296+0000","last_fullsized":"2024-09-11T03:52:06.082296+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:00.709966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.709966+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:33.296056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":4165,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.105754+0000","last_change":"2024-09-11T03:52:08.106021+0000","last_active":"2024-09-11T03:52:08.105754+0000","last_peered":"2024-09-11T03:52:08.105754+0000","last_clean":"2024-09-11T03:52:08.105754+0000","last_became_active":"2024-09-11T03:51:20.063987+0000","last_became_peered":"2024-09-11T03:51:20.063987+0000","last_unstale":"2024-09-11T03:52:08.105754+0000","last_undegraded":"2024-09-11T03:52:08.105754+0000","last_fullsized":"2024-09-11T03:52:08.105754+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:51:25.248547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:25.248547+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:35.833079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0026932129999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,7],"acting":[9,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":1446,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.071678+0000","last_change":"2024-09-11T03:51:00.033938+0000","last_active":"2024-09-11T03:52:05.071678+0000","last_peered":"2024-09-11T03:52:05.071678+0000","last_clean":"2024-09-11T03:52:05.071678+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:52:05.071678+0000","last_undegraded":"2024-09-11T03:52:05.071678+0000","last_fullsized":"2024-09-11T03:52:05.071678+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:00.033889+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"905'590","reported_seq":4680,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.291820+0000","last_change":"2024-09-11T03:52:06.090543+0000","last_active":"2024-09-11T03:52:07.291820+0000","last_peered":"2024-09-11T03:52:07.291820+0000","last_clean":"2024-09-11T03:52:07.291820+0000","last_became_active":"2024-09-11T03:50:02.053513+0000","last_became_peered":"2024-09-11T03:50:02.053513+0000","last_unstale":"2024-09-11T03:52:07.291820+0000","last_undegraded":"2024-09-11T03:52:07.291820+0000","last_fullsized":"2024-09-11T03:52:07.291820+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"838'575","last_scrub_stamp":"2024-09-11T03:51:04.839779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:04.839779+0000","objects_scrubbed":6,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:30.684838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00093358099999999995,"stat_sum":{"num_bytes":5479610,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":2803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":1443,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.083200+0000","last_change":"2024-09-11T03:51:54.138565+0000","last_active":"2024-09-11T03:52:06.083200+0000","last_peered":"2024-09-11T03:52:06.083200+0000","last_clean":"2024-09-11T03:52:06.083200+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:52:06.083200+0000","last_undegraded":"2024-09-11T03:52:06.083200+0000","last_fullsized":"2024-09-11T03:52:06.083200+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:54.138504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.138504+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:20.328413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":3229,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.063341+0000","last_change":"2024-09-11T03:52:06.088207+0000","last_active":"2024-09-11T03:52:07.063341+0000","last_peered":"2024-09-11T03:52:07.063341+0000","last_clean":"2024-09-11T03:52:07.063341+0000","last_became_active":"2024-09-11T03:51:20.064079+0000","last_became_peered":"2024-09-11T03:51:20.064079+0000","last_unstale":"2024-09-11T03:52:07.063341+0000","last_undegraded":"2024-09-11T03:52:07.063341+0000","last_fullsized":"2024-09-11T03:52:07.063341+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:51:47.080811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:47.080811+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:56.257821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0063804980000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,9],"acting":[6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.13","version":"20'92","reported_seq":1549,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.086213+0000","last_change":"2024-09-11T03:51:52.573394+0000","last_active":"2024-09-11T03:52:06.086213+0000","last_peered":"2024-09-11T03:52:06.086213+0000","last_clean":"2024-09-11T03:52:06.086213+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:52:06.086213+0000","last_undegraded":"2024-09-11T03:52:06.086213+0000","last_fullsized":"2024-09-11T03:52:06.086213+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:52.573348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.573348+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:55.111010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":3168,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.077798+0000","last_change":"2024-09-11T03:52:06.078392+0000","last_active":"2024-09-11T03:52:06.077798+0000","last_peered":"2024-09-11T03:52:06.077798+0000","last_clean":"2024-09-11T03:52:06.077798+0000","last_became_active":"2024-09-11T03:50:02.057196+0000","last_became_peered":"2024-09-11T03:50:02.057196+0000","last_unstale":"2024-09-11T03:52:06.077798+0000","last_undegraded":"2024-09-11T03:52:06.077798+0000","last_fullsized":"2024-09-11T03:52:06.077798+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:51:24.589460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:24.589460+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:33.476407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00098833600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":1030,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.078190+0000","last_change":"2024-09-11T03:51:28.960262+0000","last_active":"2024-09-11T03:52:06.078190+0000","last_peered":"2024-09-11T03:52:06.078190+0000","last_clean":"2024-09-11T03:52:06.078190+0000","last_became_active":"2024-09-11T03:46:36.026459+0000","last_became_peered":"2024-09-11T03:46:36.026459+0000","last_unstale":"2024-09-11T03:52:06.078190+0000","last_undegraded":"2024-09-11T03:52:06.078190+0000","last_fullsized":"2024-09-11T03:52:06.078190+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:28.960214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.960214+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:51.817417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":3721,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.064499+0000","last_change":"2024-09-11T03:52:06.093141+0000","last_active":"2024-09-11T03:52:07.064499+0000","last_peered":"2024-09-11T03:52:07.064499+0000","last_clean":"2024-09-11T03:52:07.064499+0000","last_became_active":"2024-09-11T03:50:02.052278+0000","last_became_peered":"2024-09-11T03:50:02.052278+0000","last_unstale":"2024-09-11T03:52:07.064499+0000","last_undegraded":"2024-09-11T03:52:07.064499+0000","last_fullsized":"2024-09-11T03:52:07.064499+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:51:28.919603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.919603+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:54.621565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0094114690000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":1416,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.078271+0000","last_change":"2024-09-11T03:50:19.741484+0000","last_active":"2024-09-11T03:52:06.078271+0000","last_peered":"2024-09-11T03:52:06.078271+0000","last_clean":"2024-09-11T03:52:06.078271+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:52:06.078271+0000","last_undegraded":"2024-09-11T03:52:06.078271+0000","last_fullsized":"2024-09-11T03:52:06.078271+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:19.741428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:19.741428+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:37.978745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"782'561","reported_seq":4691,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.082506+0000","last_change":"2024-09-11T03:52:06.082877+0000","last_active":"2024-09-11T03:52:06.082506+0000","last_peered":"2024-09-11T03:52:06.082506+0000","last_clean":"2024-09-11T03:52:06.082506+0000","last_became_active":"2024-09-11T03:50:02.054667+0000","last_became_peered":"2024-09-11T03:50:02.054667+0000","last_unstale":"2024-09-11T03:52:06.082506+0000","last_undegraded":"2024-09-11T03:52:06.082506+0000","last_fullsized":"2024-09-11T03:52:06.082506+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"782'561","last_scrub_stamp":"2024-09-11T03:51:24.484944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:24.484944+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:26.633510+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001548202,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":894,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.081550+0000","last_change":"2024-09-11T03:50:37.230454+0000","last_active":"2024-09-11T03:52:06.081550+0000","last_peered":"2024-09-11T03:52:06.081550+0000","last_clean":"2024-09-11T03:52:06.081550+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:52:06.081550+0000","last_undegraded":"2024-09-11T03:52:06.081550+0000","last_fullsized":"2024-09-11T03:52:06.081550+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:37.230388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.230388+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":4380,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.101891+0000","last_change":"2024-09-11T03:52:08.102245+0000","last_active":"2024-09-11T03:52:08.101891+0000","last_peered":"2024-09-11T03:52:08.101891+0000","last_clean":"2024-09-11T03:52:08.101891+0000","last_became_active":"2024-09-11T03:50:02.051850+0000","last_became_peered":"2024-09-11T03:50:02.051850+0000","last_unstale":"2024-09-11T03:52:08.101891+0000","last_undegraded":"2024-09-11T03:52:08.101891+0000","last_fullsized":"2024-09-11T03:52:08.101891+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:51:20.293380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.293380+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:24.000773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00081108199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":1437,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.077620+0000","last_change":"2024-09-11T03:50:43.547521+0000","last_active":"2024-09-11T03:52:06.077620+0000","last_peered":"2024-09-11T03:52:06.077620+0000","last_clean":"2024-09-11T03:52:06.077620+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:52:06.077620+0000","last_undegraded":"2024-09-11T03:52:06.077620+0000","last_fullsized":"2024-09-11T03:52:06.077620+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:43.547475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:43.547475+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"778'353","reported_seq":4355,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.063862+0000","last_change":"2024-09-11T03:52:06.089481+0000","last_active":"2024-09-11T03:52:07.063862+0000","last_peered":"2024-09-11T03:52:07.063862+0000","last_clean":"2024-09-11T03:52:07.063862+0000","last_became_active":"2024-09-11T03:50:02.054211+0000","last_became_peered":"2024-09-11T03:50:02.054211+0000","last_unstale":"2024-09-11T03:52:07.063862+0000","last_undegraded":"2024-09-11T03:52:07.063862+0000","last_fullsized":"2024-09-11T03:52:07.063862+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"778'353","last_scrub_stamp":"2024-09-11T03:51:23.900161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.900161+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:39.968972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0068619650000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.18","version":"20'92","reported_seq":1555,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.082193+0000","last_change":"2024-09-11T03:51:27.437144+0000","last_active":"2024-09-11T03:52:06.082193+0000","last_peered":"2024-09-11T03:52:06.082193+0000","last_clean":"2024-09-11T03:52:06.082193+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:52:06.082193+0000","last_undegraded":"2024-09-11T03:52:06.082193+0000","last_fullsized":"2024-09-11T03:52:06.082193+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:27.437090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:27.437090+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:35.495938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"908'928","reported_seq":5186,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.103525+0000","last_change":"2024-09-11T03:52:06.080944+0000","last_active":"2024-09-11T03:52:06.103525+0000","last_peered":"2024-09-11T03:52:06.103525+0000","last_clean":"2024-09-11T03:52:06.103525+0000","last_became_active":"2024-09-11T03:50:31.047148+0000","last_became_peered":"2024-09-11T03:50:31.047148+0000","last_unstale":"2024-09-11T03:52:06.103525+0000","last_undegraded":"2024-09-11T03:52:06.103525+0000","last_fullsized":"2024-09-11T03:52:06.103525+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":5,"last_scrub":"882'911","last_scrub_stamp":"2024-09-11T03:51:45.052831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:45.052831+0000","objects_scrubbed":6,"log_size":928,"log_dups_size":0,"ondisk_log_size":928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:54.625672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00086334699999999999,"stat_sum":{"num_bytes":14006122,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":15,"num_read_kb":10932,"num_write":19,"num_write_kb":5002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.19","version":"20'92","reported_seq":1043,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.291127+0000","last_change":"2024-09-11T03:51:28.080491+0000","last_active":"2024-09-11T03:52:07.291127+0000","last_peered":"2024-09-11T03:52:07.291127+0000","last_clean":"2024-09-11T03:52:07.291127+0000","last_became_active":"2024-09-11T03:46:15.972841+0000","last_became_peered":"2024-09-11T03:46:15.972841+0000","last_unstale":"2024-09-11T03:52:07.291127+0000","last_undegraded":"2024-09-11T03:52:07.291127+0000","last_fullsized":"2024-09-11T03:52:07.291127+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:28.080420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.080420+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:28.344483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"557'273","reported_seq":4311,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.063251+0000","last_change":"2024-09-11T03:52:06.088643+0000","last_active":"2024-09-11T03:52:07.063251+0000","last_peered":"2024-09-11T03:52:07.063251+0000","last_clean":"2024-09-11T03:52:07.063251+0000","last_became_active":"2024-09-11T03:50:28.038587+0000","last_became_peered":"2024-09-11T03:50:28.038587+0000","last_unstale":"2024-09-11T03:52:07.063251+0000","last_undegraded":"2024-09-11T03:52:07.063251+0000","last_fullsized":"2024-09-11T03:52:07.063251+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:51:54.121178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.121178+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:14.357120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0067314159999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,2],"acting":[6,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.1a","version":"20'92","reported_seq":594,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.078021+0000","last_change":"2024-09-11T03:51:23.032108+0000","last_active":"2024-09-11T03:52:06.078021+0000","last_peered":"2024-09-11T03:52:06.078021+0000","last_clean":"2024-09-11T03:52:06.078021+0000","last_became_active":"2024-09-11T03:51:17.063957+0000","last_became_peered":"2024-09-11T03:51:17.063957+0000","last_unstale":"2024-09-11T03:52:06.078021+0000","last_undegraded":"2024-09-11T03:52:06.078021+0000","last_fullsized":"2024-09-11T03:52:06.078021+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:23.032056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.032056+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:35.587988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.19","version":"908'630","reported_seq":4793,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.063041+0000","last_change":"2024-09-11T03:52:06.087623+0000","last_active":"2024-09-11T03:52:07.063041+0000","last_peered":"2024-09-11T03:52:07.063041+0000","last_clean":"2024-09-11T03:52:07.063041+0000","last_became_active":"2024-09-11T03:50:28.039694+0000","last_became_peered":"2024-09-11T03:50:28.039694+0000","last_unstale":"2024-09-11T03:52:07.063041+0000","last_undegraded":"2024-09-11T03:52:07.063041+0000","last_fullsized":"2024-09-11T03:52:07.063041+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"816'587","last_scrub_stamp":"2024-09-11T03:50:36.522195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.522195+0000","objects_scrubbed":7,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0061287909999999998,"stat_sum":{"num_bytes":13997764,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":14,"num_write_kb":3830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.1b","version":"20'92","reported_seq":1414,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.077674+0000","last_change":"2024-09-11T03:51:52.434832+0000","last_active":"2024-09-11T03:52:06.077674+0000","last_peered":"2024-09-11T03:52:06.077674+0000","last_clean":"2024-09-11T03:52:06.077674+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:52:06.077674+0000","last_undegraded":"2024-09-11T03:52:06.077674+0000","last_fullsized":"2024-09-11T03:52:06.077674+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:52.434784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.434784+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:11.714276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"910'736","reported_seq":5000,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.107374+0000","last_change":"2024-09-11T03:52:08.107374+0000","last_active":"2024-09-11T03:52:08.107374+0000","last_peered":"2024-09-11T03:52:08.107374+0000","last_clean":"2024-09-11T03:52:08.107374+0000","last_became_active":"2024-09-11T03:51:20.065666+0000","last_became_peered":"2024-09-11T03:51:20.065666+0000","last_unstale":"2024-09-11T03:52:08.107374+0000","last_undegraded":"2024-09-11T03:52:08.107374+0000","last_fullsized":"2024-09-11T03:52:08.107374+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":5,"last_scrub":"816'566","last_scrub_stamp":"2024-09-11T03:50:35.256345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.256345+0000","objects_scrubbed":11,"log_size":736,"log_dups_size":0,"ondisk_log_size":736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0058198410000000001,"stat_sum":{"num_bytes":41106880,"num_objects":18,"num_object_clones":16,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":36,"num_read_kb":35366,"num_write":29,"num_write_kb":11526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":15760321,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,11],"acting":[9,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.1f","version":"907'890","reported_seq":4936,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.100614+0000","last_change":"2024-09-11T03:52:03.085879+0000","last_active":"2024-09-11T03:52:05.100614+0000","last_peered":"2024-09-11T03:52:05.100614+0000","last_clean":"2024-09-11T03:52:05.100614+0000","last_became_active":"2024-09-11T03:50:28.041110+0000","last_became_peered":"2024-09-11T03:50:28.041110+0000","last_unstale":"2024-09-11T03:52:05.100614+0000","last_undegraded":"2024-09-11T03:52:05.100614+0000","last_fullsized":"2024-09-11T03:52:05.100614+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"894'871","last_scrub_stamp":"2024-09-11T03:51:53.078478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.078478+0000","objects_scrubbed":13,"log_size":890,"log_dups_size":0,"ondisk_log_size":890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:17.605716+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.011179424,"stat_sum":{"num_bytes":37592638,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":43,"num_read_kb":39435,"num_write":25,"num_write_kb":13669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.1c","version":"20'92","reported_seq":1421,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.078012+0000","last_change":"2024-09-11T03:51:05.596734+0000","last_active":"2024-09-11T03:52:06.078012+0000","last_peered":"2024-09-11T03:52:06.078012+0000","last_clean":"2024-09-11T03:52:06.078012+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:52:06.078012+0000","last_undegraded":"2024-09-11T03:52:06.078012+0000","last_fullsized":"2024-09-11T03:52:06.078012+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:05.596677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:05.596677+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:09.881830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1e","version":"903'1112","reported_seq":5168,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-11T03:52:05.071872+0000","last_change":"2024-09-11T03:52:03.083454+0000","last_active":"2024-09-11T03:52:05.071872+0000","last_peered":"2024-09-11T03:52:05.071872+0000","last_clean":"2024-09-11T03:52:05.071872+0000","last_became_active":"2024-09-11T03:50:28.044451+0000","last_became_peered":"2024-09-11T03:50:28.044451+0000","last_unstale":"2024-09-11T03:52:05.071872+0000","last_undegraded":"2024-09-11T03:52:05.071872+0000","last_fullsized":"2024-09-11T03:52:05.071872+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"885'1106","last_scrub_stamp":"2024-09-11T03:51:44.050833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:44.050833+0000","objects_scrubbed":5,"log_size":1112,"log_dups_size":0,"ondisk_log_size":1112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:07.661910+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0083820170000000003,"stat_sum":{"num_bytes":9671047,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":24362,"num_write":12,"num_write_kb":3430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"}]},{"pgid":"1.1d","version":"20'92","reported_seq":1402,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.085220+0000","last_change":"2024-09-11T03:50:56.316951+0000","last_active":"2024-09-11T03:52:06.085220+0000","last_peered":"2024-09-11T03:52:06.085220+0000","last_clean":"2024-09-11T03:52:06.085220+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:52:06.085220+0000","last_undegraded":"2024-09-11T03:52:06.085220+0000","last_fullsized":"2024-09-11T03:52:06.085220+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:56.316896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:56.316896+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:07.998967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"567'752","reported_seq":4802,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.292451+0000","last_change":"2024-09-11T03:52:06.092148+0000","last_active":"2024-09-11T03:52:07.292451+0000","last_peered":"2024-09-11T03:52:07.292451+0000","last_clean":"2024-09-11T03:52:07.292451+0000","last_became_active":"2024-09-11T03:50:28.045687+0000","last_became_peered":"2024-09-11T03:50:28.045687+0000","last_unstale":"2024-09-11T03:52:07.292451+0000","last_undegraded":"2024-09-11T03:52:07.292451+0000","last_fullsized":"2024-09-11T03:52:07.292451+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:51:52.141035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.141035+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:58.296530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00062390799999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,11],"acting":[10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.1e","version":"20'92","reported_seq":1427,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.081652+0000","last_change":"2024-09-11T03:51:02.962842+0000","last_active":"2024-09-11T03:52:06.081652+0000","last_peered":"2024-09-11T03:52:06.081652+0000","last_clean":"2024-09-11T03:52:06.081652+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:52:06.081652+0000","last_undegraded":"2024-09-11T03:52:06.081652+0000","last_fullsized":"2024-09-11T03:52:06.081652+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:02.962791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:02.962791+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:13.625901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":76,"stat_sum":{"num_bytes":764015474,"num_objects":319,"num_object_clones":269,"num_object_copies":957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":319,"num_whiteouts":10,"num_read":1010,"num_read_kb":1078526,"num_write":679,"num_write_kb":274946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":82257142,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1348272128,"data_stored":1776549978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47766,"ondisk_log_size":47766,"up":228,"acting":228,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285747,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":229236,"kb_used_data":171676,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93515916,"statfs":{"total":95995035648,"available":95760297984,"internally_reserved":0,"allocated":175796224,"data_stored":226803001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":58915268},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514679,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91768,"kb_used_data":60580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93653384,"statfs":{"total":95995035648,"available":95901065216,"internally_reserved":0,"allocated":62033920,"data_stored":71091875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743613,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":178084,"kb_used_data":114964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93567068,"statfs":{"total":95995035648,"available":95812677632,"internally_reserved":0,"allocated":117723136,"data_stored":139488914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037956,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190160,"kb_used_data":145076,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93554992,"statfs":{"total":95995035648,"available":95800311808,"internally_reserved":0,"allocated":148557824,"data_stored":203837018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299591,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":217764,"kb_used_data":146320,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93527388,"statfs":{"total":95995035648,"available":95772045312,"internally_reserved":0,"allocated":149831680,"data_stored":181749963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528524,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":216400,"kb_used_data":163004,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93528752,"statfs":{"total":95995035648,"available":95773442048,"internally_reserved":0,"allocated":166916096,"data_stored":224448496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757458,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":165032,"kb_used_data":111636,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93580120,"statfs":{"total":95995035648,"available":95826042880,"internally_reserved":0,"allocated":114315264,"data_stored":145965840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986391,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149917,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":205444,"kb_used_data":134000,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93539708,"statfs":{"total":95995035648,"available":95784660992,"internally_reserved":0,"allocated":137216000,"data_stored":184463141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411554,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":138128,"kb_used_data":84732,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93607024,"statfs":{"total":95995035648,"available":95853592576,"internally_reserved":0,"allocated":86765568,"data_stored":100036901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575079,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":139340,"kb_used_data":94268,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93605812,"statfs":{"total":95995035648,"available":95852351488,"internally_reserved":0,"allocated":96530432,"data_stored":112996069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738604,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":174908,"kb_used_data":113188,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":93570244,"statfs":{"total":95995035648,"available":95815929856,"internally_reserved":0,"allocated":115904512,"data_stored":139455762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106135552,"data_stored":133465885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":85504000,"data_stored":105766448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":76664832,"data_stored":94351931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":125837312,"data_stored":176888928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":96878592,"data_stored":122811969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":103084032,"data_stored":138534971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":155561984,"data_stored":216891195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":138612736,"data_stored":174327542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":137531392,"data_stored":196607397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":106684416,"data_stored":132250829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":51007488,"data_stored":63862254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":164769792,"data_stored":220790629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:52:10.550 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-11T03:52:10.550 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:52:10.550 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-11T03:52:10.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:52:10.617 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:10.638 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:10.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:10.639 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:10.686 INFO:tasks.rados.rados.2.smithi175.stdout:3798: finishing write tid 2 to smithi17557870-39 2024-09-11T03:52:10.686 INFO:tasks.rados.rados.2.smithi175.stdout:3799: finishing copy_from to smithi17557870-16 2024-09-11T03:52:10.686 INFO:tasks.rados.rados.2.smithi175.stdout:3799: got expected ENOENT (src dne) 2024-09-11T03:52:10.686 INFO:tasks.rados.rados.2.smithi175.stdout:3799: finishing copy_from racing read to smithi17557870-16 2024-09-11T03:52:10.686 INFO:tasks.rados.rados.2.smithi175.stdout:3798: finishing write tid 3 to smithi17557870-39 2024-09-11T03:52:10.686 INFO:tasks.rados.rados.2.smithi175.stdout:3798: finishing write tid 4 to smithi17557870-39 2024-09-11T03:52:10.686 INFO:tasks.rados.rados.2.smithi175.stdout:3798: finishing write tid 5 to smithi17557870-39 2024-09-11T03:52:10.687 INFO:tasks.rados.rados.2.smithi175.stdout:3798: oid 39 updating version 0 to 1025 2024-09-11T03:52:10.687 INFO:tasks.rados.rados.2.smithi175.stdout:3798: oid 39 version 1025 is already newer than 1024 2024-09-11T03:52:10.687 INFO:tasks.rados.rados.2.smithi175.stdout:3798: oid 39 version 1025 is already newer than 1022 2024-09-11T03:52:10.687 INFO:tasks.rados.rados.2.smithi175.stdout:3798: oid 39 version 1025 is already newer than 1023 2024-09-11T03:52:10.687 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 1025 (ObjNum 1286 snap 423 seq_num 1286) dirty exists 2024-09-11T03:52:10.687 INFO:tasks.rados.rados.2.smithi175.stdout:3798: left oid 39 (ObjNum 1286 snap 423 seq_num 1286) 2024-09-11T03:52:10.688 INFO:tasks.rados.rados.2.smithi175.stdout:3796: done (3 left) 2024-09-11T03:52:10.688 INFO:tasks.rados.rados.2.smithi175.stdout:3797: done (2 left) 2024-09-11T03:52:10.688 INFO:tasks.rados.rados.2.smithi175.stdout:3798: done (1 left) 2024-09-11T03:52:10.688 INFO:tasks.rados.rados.2.smithi175.stdout:3799: done (0 left) 2024-09-11T03:52:10.688 INFO:tasks.rados.rados.2.smithi175.stdout:3800: delete oid 34 current snap is 423 2024-09-11T03:52:10.688 INFO:tasks.rados.rados.2.smithi175.stdout:3800: done (0 left) 2024-09-11T03:52:10.688 INFO:tasks.rados.rados.2.smithi175.stdout:3801: read oid 29 snap -1 2024-09-11T03:52:10.689 INFO:tasks.rados.rados.2.smithi175.stdout:3801: expect deleted 2024-09-11T03:52:10.689 INFO:tasks.rados.rados.2.smithi175.stdout:3802: snap_remove snap 392 2024-09-11T03:52:10.739 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:10.765 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:10.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:10.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:10.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:10.893 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:10.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:10.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:10.961 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:52:10.961 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:52:10.976 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:52:10.976 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:52:10.994 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:10.995 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:52:10.995 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:52:10.996 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:11.012 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:52:11.012 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:52:11.017 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:11.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:11.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:11.031 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:52:11.031 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:52:11.048 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:52:11.049 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:52:11.065 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:52:11.066 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:52:11.082 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:52:11.083 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:52:11.099 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:52:11.099 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:52:11.117 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:52:11.118 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:52:11.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:11.120 INFO:tasks.rados.rados.2.smithi175.stdout:3802: done (1 left) 2024-09-11T03:52:11.120 INFO:tasks.rados.rados.2.smithi175.stdout:3803: snap_create 2024-09-11T03:52:11.141 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:52:11.142 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:52:11.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:11.153 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:11.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:11.158 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:52:11.158 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:52:11.254 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:11.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:11.276 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:11.277 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:11.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:11.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:11 smithi084 ceph-mon[27254]: 1.17 scrub starts 2024-09-11T03:52:11.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:11 smithi084 ceph-mon[27254]: 1.17 scrub ok 2024-09-11T03:52:11.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:11 smithi084 ceph-mon[27254]: 2.41 scrub starts 2024-09-11T03:52:11.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:11 smithi084 ceph-mon[27254]: 2.41 scrub ok 2024-09-11T03:52:11.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:11 smithi084 ceph-mon[27254]: 2.27 scrub starts 2024-09-11T03:52:11.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:11 smithi084 ceph-mon[27254]: 2.27 scrub ok 2024-09-11T03:52:11.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:11 smithi084 ceph-mon[27254]: 1.11 scrub starts 2024-09-11T03:52:11.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:11 smithi084 ceph-mon[27254]: 1.11 scrub ok 2024-09-11T03:52:11.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:11 smithi084 ceph-mon[27254]: osdmap e912: 12 total, 12 up, 11 in 2024-09-11T03:52:11.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:11 smithi084 ceph-mon[27254]: 2.3b scrub starts 2024-09-11T03:52:11.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:11 smithi084 ceph-mon[27254]: 2.3b scrub ok 2024-09-11T03:52:11.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:11 smithi084 ceph-mon[27254]: 2.18 scrub starts 2024-09-11T03:52:11.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:11 smithi084 ceph-mon[27254]: 2.18 scrub ok 2024-09-11T03:52:11.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:11 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3529902646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:11.403 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:11.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:11.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:11 smithi121 ceph-mon[30729]: 1.17 scrub starts 2024-09-11T03:52:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:11 smithi121 ceph-mon[30729]: 1.17 scrub ok 2024-09-11T03:52:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:11 smithi121 ceph-mon[30729]: 2.41 scrub starts 2024-09-11T03:52:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:11 smithi121 ceph-mon[30729]: 2.41 scrub ok 2024-09-11T03:52:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:11 smithi121 ceph-mon[30729]: 2.27 scrub starts 2024-09-11T03:52:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:11 smithi121 ceph-mon[30729]: 2.27 scrub ok 2024-09-11T03:52:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:11 smithi121 ceph-mon[30729]: 1.11 scrub starts 2024-09-11T03:52:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:11 smithi121 ceph-mon[30729]: 1.11 scrub ok 2024-09-11T03:52:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:11 smithi121 ceph-mon[30729]: osdmap e912: 12 total, 12 up, 11 in 2024-09-11T03:52:11.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:11 smithi121 ceph-mon[30729]: 2.3b scrub starts 2024-09-11T03:52:11.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:11 smithi121 ceph-mon[30729]: 2.3b scrub ok 2024-09-11T03:52:11.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:11 smithi121 ceph-mon[30729]: 2.18 scrub starts 2024-09-11T03:52:11.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:11 smithi121 ceph-mon[30729]: 2.18 scrub ok 2024-09-11T03:52:11.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:11 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3529902646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:11 smithi175 ceph-mon[30894]: 1.17 scrub starts 2024-09-11T03:52:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:11 smithi175 ceph-mon[30894]: 1.17 scrub ok 2024-09-11T03:52:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:11 smithi175 ceph-mon[30894]: 2.41 scrub starts 2024-09-11T03:52:11.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:11 smithi175 ceph-mon[30894]: 2.41 scrub ok 2024-09-11T03:52:11.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:11 smithi175 ceph-mon[30894]: 2.27 scrub starts 2024-09-11T03:52:11.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:11 smithi175 ceph-mon[30894]: 2.27 scrub ok 2024-09-11T03:52:11.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:11 smithi175 ceph-mon[30894]: 1.11 scrub starts 2024-09-11T03:52:11.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:11 smithi175 ceph-mon[30894]: 1.11 scrub ok 2024-09-11T03:52:11.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:11 smithi175 ceph-mon[30894]: osdmap e912: 12 total, 12 up, 11 in 2024-09-11T03:52:11.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:11 smithi175 ceph-mon[30894]: 2.3b scrub starts 2024-09-11T03:52:11.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:11 smithi175 ceph-mon[30894]: 2.3b scrub ok 2024-09-11T03:52:11.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:11 smithi175 ceph-mon[30894]: 2.18 scrub starts 2024-09-11T03:52:11.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:11 smithi175 ceph-mon[30894]: 2.18 scrub ok 2024-09-11T03:52:11.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:11 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3529902646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:11.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:11.526 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:11.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:11.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:11.545 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:52:11.628 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:11.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:11.650 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:11.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:11.751 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:11.777 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:11.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:11.879 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:11.901 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:11.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:11.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:12.003 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:12.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:12.024 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:12.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:12.064 INFO:tasks.rados.rados.2.smithi175.stdout:3801: done (1 left) 2024-09-11T03:52:12.064 INFO:tasks.rados.rados.2.smithi175.stdout:3803: done (0 left) 2024-09-11T03:52:12.064 INFO:tasks.rados.rados.2.smithi175.stdout:3804: snap_remove snap 391 2024-09-11T03:52:12.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:12.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:12 smithi084 ceph-mon[27254]: 2.0 scrub starts 2024-09-11T03:52:12.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:12 smithi084 ceph-mon[27254]: 2.0 scrub ok 2024-09-11T03:52:12.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:12 smithi084 ceph-mon[27254]: pgmap v1524: 107 pgs: 107 active+clean; 729 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 17 MiB/s rd, 735 KiB/s wr, 15 op/s 2024-09-11T03:52:12.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:12 smithi084 ceph-mon[27254]: osdmap e913: 12 total, 12 up, 11 in 2024-09-11T03:52:12.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:12 smithi084 ceph-mon[27254]: osdmap e914: 12 total, 12 up, 11 in 2024-09-11T03:52:12.151 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:12.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:12.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:12.253 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:12.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:12.277 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:12.277 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:12.380 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:12.403 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:52:12.404 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:52:12.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:12.406 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:12.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:12.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:12 smithi121 ceph-mon[30729]: 2.0 scrub starts 2024-09-11T03:52:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:12 smithi121 ceph-mon[30729]: 2.0 scrub ok 2024-09-11T03:52:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:12 smithi121 ceph-mon[30729]: pgmap v1524: 107 pgs: 107 active+clean; 729 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 17 MiB/s rd, 735 KiB/s wr, 15 op/s 2024-09-11T03:52:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:12 smithi121 ceph-mon[30729]: osdmap e913: 12 total, 12 up, 11 in 2024-09-11T03:52:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:12 smithi121 ceph-mon[30729]: osdmap e914: 12 total, 12 up, 11 in 2024-09-11T03:52:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:12 smithi175 ceph-mon[30894]: 2.0 scrub starts 2024-09-11T03:52:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:12 smithi175 ceph-mon[30894]: 2.0 scrub ok 2024-09-11T03:52:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:12 smithi175 ceph-mon[30894]: pgmap v1524: 107 pgs: 107 active+clean; 729 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 17 MiB/s rd, 735 KiB/s wr, 15 op/s 2024-09-11T03:52:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:12 smithi175 ceph-mon[30894]: osdmap e913: 12 total, 12 up, 11 in 2024-09-11T03:52:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:12 smithi175 ceph-mon[30894]: osdmap e914: 12 total, 12 up, 11 in 2024-09-11T03:52:12.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:12.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:12.530 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:12.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:12.631 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:12.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:12.655 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:12.656 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:12.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:12.786 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:12.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:12.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:12.891 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:52:12.891 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:52:12.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:52:12.891 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:52:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:4915 2.19 2:98bb65f5:::smithi17557870-6:head [sparse-read 0~2219734] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e523)", 2024-09-11T03:52:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:45:41.041363+0000", 2024-09-11T03:52:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 391.83256285900001, 2024-09-11T03:52:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015358287999999999, 2024-09-11T03:52:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4915 2024-09-11T03:52:12.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.893 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:12.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:12.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041363+0000", 2024-09-11T03:52:12.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:12.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041363+0000", 2024-09-11T03:52:12.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:12.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041363+0000", 2024-09-11T03:52:12.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8400000000000004e-07 2024-09-11T03:52:12.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:12.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041368+0000", 2024-09-11T03:52:12.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1060000000000002e-06 2024-09-11T03:52:12.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:12.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041374+0000", 2024-09-11T03:52:12.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.866e-06 2024-09-11T03:52:12.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:12.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.041380+0000", 2024-09-11T03:52:12.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3720000000000004e-06 2024-09-11T03:52:12.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.042462+0000", 2024-09-11T03:52:12.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001082258 2024-09-11T03:52:12.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:12.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.042467+0000", 2024-09-11T03:52:12.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0130000000000001e-06 2024-09-11T03:52:12.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.056146+0000", 2024-09-11T03:52:12.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013679314999999999 2024-09-11T03:52:12.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:12.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.056249+0000", 2024-09-11T03:52:12.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102586 2024-09-11T03:52:12.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:12.897 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:41.056721+0000", 2024-09-11T03:52:12.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00047218800000000001 2024-09-11T03:52:12.897 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.897 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:12.897 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.897 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5648 2.1f 2:fda69313:::smithi17557870-16:head [copy-from ver 826] snapc f9=[] ondisk+write+known_if_redirected+supports_pool_eio e599)", 2024-09-11T03:52:12.897 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:56.974414+0000", 2024-09-11T03:52:12.897 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 315.89951143000002, 2024-09-11T03:52:12.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021325499000000001, 2024-09-11T03:52:12.897 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:12.897 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:12.898 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:12.898 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:12.898 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:12.898 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:12.898 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5648 2024-09-11T03:52:12.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.898 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:12.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:12.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974414+0000", 2024-09-11T03:52:12.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:12.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974414+0000", 2024-09-11T03:52:12.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:12.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974416+0000", 2024-09-11T03:52:12.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.666e-06 2024-09-11T03:52:12.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:12.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974422+0000", 2024-09-11T03:52:12.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2400000000000004e-06 2024-09-11T03:52:12.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:12.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974434+0000", 2024-09-11T03:52:12.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2111e-05 2024-09-11T03:52:12.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:12.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974442+0000", 2024-09-11T03:52:12.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9149999999999996e-06 2024-09-11T03:52:12.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974501+0000", 2024-09-11T03:52:12.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8369000000000003e-05 2024-09-11T03:52:12.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:52:12.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974509+0000", 2024-09-11T03:52:12.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2260000000000005e-06 2024-09-11T03:52:12.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976102+0000", 2024-09-11T03:52:12.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001593187 2024-09-11T03:52:12.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:12.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976234+0000", 2024-09-11T03:52:12.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000131903 2024-09-11T03:52:12.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7,8", 2024-09-11T03:52:12.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.986796+0000", 2024-09-11T03:52:12.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010562344 2024-09-11T03:52:12.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.902 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:12.902 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.989855+0000", 2024-09-11T03:52:12.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030587660000000001 2024-09-11T03:52:12.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:12.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.992580+0000", 2024-09-11T03:52:12.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027246950000000001 2024-09-11T03:52:12.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:12.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.995661+0000", 2024-09-11T03:52:12.903 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030807500000000002 2024-09-11T03:52:12.903 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.903 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:12.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.995708+0000", 2024-09-11T03:52:12.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7945999999999998e-05 2024-09-11T03:52:12.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:12.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.995740+0000", 2024-09-11T03:52:12.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1381000000000003e-05 2024-09-11T03:52:12.904 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.904 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:12.904 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.905 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5649 2.1f 2:fda69313:::smithi17557870-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e599)", 2024-09-11T03:52:12.905 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:56.974454+0000", 2024-09-11T03:52:12.905 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 315.89947191200002, 2024-09-11T03:52:12.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015178341999999999, 2024-09-11T03:52:12.905 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:12.905 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:12.905 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:12.905 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:12.905 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:12.906 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:12.906 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5649 2024-09-11T03:52:12.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.906 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:12.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:12.906 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974454+0000", 2024-09-11T03:52:12.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:12.906 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974454+0000", 2024-09-11T03:52:12.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:12.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974454+0000", 2024-09-11T03:52:12.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8700000000000001e-07 2024-09-11T03:52:12.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:12.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974457+0000", 2024-09-11T03:52:12.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.61e-06 2024-09-11T03:52:12.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:12.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974462+0000", 2024-09-11T03:52:12.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6700000000000002e-06 2024-09-11T03:52:12.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:12.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974465+0000", 2024-09-11T03:52:12.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4829999999999999e-06 2024-09-11T03:52:12.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974546+0000", 2024-09-11T03:52:12.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0938000000000003e-05 2024-09-11T03:52:12.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:12.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974549+0000", 2024-09-11T03:52:12.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3830000000000001e-06 2024-09-11T03:52:12.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976401+0000", 2024-09-11T03:52:12.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018517480000000001 2024-09-11T03:52:12.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:12.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976445+0000", 2024-09-11T03:52:12.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3439000000000002e-05 2024-09-11T03:52:12.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.989435+0000", 2024-09-11T03:52:12.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012990714 2024-09-11T03:52:12.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:12.911 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.989531+0000", 2024-09-11T03:52:12.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5304999999999996e-05 2024-09-11T03:52:12.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.911 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:12.911 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.989632+0000", 2024-09-11T03:52:12.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000101665 2024-09-11T03:52:12.911 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.911 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:12.911 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.912 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5650 2.1e 2:7b0c3397:::smithi17557870-14:head [copy-from ver 875] snapc f9=[] ondisk+write+known_if_redirected+supports_pool_eio e599)", 2024-09-11T03:52:12.912 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:56.974491+0000", 2024-09-11T03:52:12.912 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 315.899434402, 2024-09-11T03:52:12.912 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.029056842999999999, 2024-09-11T03:52:12.912 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:12.912 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:12.912 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:12.912 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:12.912 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:12.913 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:12.913 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5650 2024-09-11T03:52:12.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.913 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:12.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:12.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974491+0000", 2024-09-11T03:52:12.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:12.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974491+0000", 2024-09-11T03:52:12.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:12.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974492+0000", 2024-09-11T03:52:12.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.72e-07 2024-09-11T03:52:12.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:12.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974494+0000", 2024-09-11T03:52:12.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0480000000000001e-06 2024-09-11T03:52:12.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:12.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974498+0000", 2024-09-11T03:52:12.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1699999999999999e-06 2024-09-11T03:52:12.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:12.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974501+0000", 2024-09-11T03:52:12.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0479999999999999e-06 2024-09-11T03:52:12.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974537+0000", 2024-09-11T03:52:12.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6041000000000003e-05 2024-09-11T03:52:12.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:52:12.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974541+0000", 2024-09-11T03:52:12.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2010000000000001e-06 2024-09-11T03:52:12.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.917 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.975950+0000", 2024-09-11T03:52:12.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014090489999999999 2024-09-11T03:52:12.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.917 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:12.917 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976045+0000", 2024-09-11T03:52:12.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4704999999999995e-05 2024-09-11T03:52:12.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,10", 2024-09-11T03:52:12.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.992128+0000", 2024-09-11T03:52:12.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016083318999999999 2024-09-11T03:52:12.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:12.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.994884+0000", 2024-09-11T03:52:12.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027554860000000001 2024-09-11T03:52:12.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:12.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.997365+0000", 2024-09-11T03:52:12.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024811630000000002 2024-09-11T03:52:12.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:12.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:57.003332+0000", 2024-09-11T03:52:12.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059666720000000001 2024-09-11T03:52:12.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:12.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:57.003487+0000", 2024-09-11T03:52:12.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015519399999999999 2024-09-11T03:52:12.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:12.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:57.003548+0000", 2024-09-11T03:52:12.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1375000000000001e-05 2024-09-11T03:52:12.920 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.920 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:12.920 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.920 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5651 2.1e 2:7b0c3397:::smithi17557870-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e599)", 2024-09-11T03:52:12.920 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:56.974509+0000", 2024-09-11T03:52:12.920 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 315.89941650999998, 2024-09-11T03:52:12.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.02013765, 2024-09-11T03:52:12.921 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:12.921 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:12.921 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:12.921 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:12.921 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:12.921 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:12.921 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5651 2024-09-11T03:52:12.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.921 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:12.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:12.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974509+0000", 2024-09-11T03:52:12.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:12.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974509+0000", 2024-09-11T03:52:12.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:12.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974510+0000", 2024-09-11T03:52:12.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5999999999999999e-07 2024-09-11T03:52:12.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:12.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974511+0000", 2024-09-11T03:52:12.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.72e-06 2024-09-11T03:52:12.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:12.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974517+0000", 2024-09-11T03:52:12.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0499999999999997e-06 2024-09-11T03:52:12.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:12.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974522+0000", 2024-09-11T03:52:12.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7770000000000002e-06 2024-09-11T03:52:12.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974570+0000", 2024-09-11T03:52:12.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7729000000000003e-05 2024-09-11T03:52:12.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:12.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.974574+0000", 2024-09-11T03:52:12.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7929999999999999e-06 2024-09-11T03:52:12.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.925 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976323+0000", 2024-09-11T03:52:12.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017491760000000001 2024-09-11T03:52:12.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:12.925 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.976367+0000", 2024-09-11T03:52:12.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3912000000000002e-05 2024-09-11T03:52:12.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.994395+0000", 2024-09-11T03:52:12.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018028512 2024-09-11T03:52:12.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:12.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.994492+0000", 2024-09-11T03:52:12.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6679000000000001e-05 2024-09-11T03:52:12.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:12.926 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:56.994647+0000", 2024-09-11T03:52:12.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015484200000000001 2024-09-11T03:52:12.926 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.927 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:12.927 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.927 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7041 2.1f 2:fba9fb28:::smithi17557870-48:head [copy-from ver 962] snapc 13a=[13a,139,138,137,136,135,133,130,12e,12d,12c,12a,127,126,123,120,11f,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e709)", 2024-09-11T03:52:12.927 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:47.006333+0000", 2024-09-11T03:52:12.927 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 205.86759247699999, 2024-09-11T03:52:12.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023502866000000001, 2024-09-11T03:52:12.928 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:12.928 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:12.928 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:12.928 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:12.928 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:12.928 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:12.928 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7041 2024-09-11T03:52:12.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.928 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:12.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:12.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006333+0000", 2024-09-11T03:52:12.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:12.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006333+0000", 2024-09-11T03:52:12.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:12.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006335+0000", 2024-09-11T03:52:12.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.528e-06 2024-09-11T03:52:12.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:12.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006341+0000", 2024-09-11T03:52:12.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2700000000000001e-06 2024-09-11T03:52:12.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:12.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006355+0000", 2024-09-11T03:52:12.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3593e-05 2024-09-11T03:52:12.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:12.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006361+0000", 2024-09-11T03:52:12.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7290000000000003e-06 2024-09-11T03:52:12.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006495+0000", 2024-09-11T03:52:12.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000133158 2024-09-11T03:52:12.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:52:12.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006501+0000", 2024-09-11T03:52:12.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4289999999999998e-06 2024-09-11T03:52:12.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.007153+0000", 2024-09-11T03:52:12.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00065224399999999996 2024-09-11T03:52:12.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:12.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.007271+0000", 2024-09-11T03:52:12.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000117743 2024-09-11T03:52:12.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7,8", 2024-09-11T03:52:12.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.019731+0000", 2024-09-11T03:52:12.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012460024 2024-09-11T03:52:12.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:12.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.023726+0000", 2024-09-11T03:52:12.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003995139 2024-09-11T03:52:12.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:12.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.027785+0000", 2024-09-11T03:52:12.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040589809999999997 2024-09-11T03:52:12.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:12.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.029747+0000", 2024-09-11T03:52:12.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019621819999999998 2024-09-11T03:52:12.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:12.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.029801+0000", 2024-09-11T03:52:12.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3715999999999999e-05 2024-09-11T03:52:12.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:12.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.029836+0000", 2024-09-11T03:52:12.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5129999999999997e-05 2024-09-11T03:52:12.934 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.934 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:12.934 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.934 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7042 2.1f 2:fba9fb28:::smithi17557870-48:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e709)", 2024-09-11T03:52:12.934 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:47.006384+0000", 2024-09-11T03:52:12.934 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 205.86754210800001, 2024-09-11T03:52:12.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016792033000000001, 2024-09-11T03:52:12.935 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:12.935 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:12.935 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:12.935 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:12.935 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:12.935 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:12.935 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7042 2024-09-11T03:52:12.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.935 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:12.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:12.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006384+0000", 2024-09-11T03:52:12.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:12.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006384+0000", 2024-09-11T03:52:12.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:12.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006384+0000", 2024-09-11T03:52:12.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8800000000000002e-07 2024-09-11T03:52:12.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:12.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006387+0000", 2024-09-11T03:52:12.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7140000000000002e-06 2024-09-11T03:52:12.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:12.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006395+0000", 2024-09-11T03:52:12.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3110000000000006e-06 2024-09-11T03:52:12.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:12.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006405+0000", 2024-09-11T03:52:12.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6679999999999996e-06 2024-09-11T03:52:12.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.007550+0000", 2024-09-11T03:52:12.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011449489999999999 2024-09-11T03:52:12.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:12.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.007587+0000", 2024-09-11T03:52:12.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7372000000000001e-05 2024-09-11T03:52:12.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.938 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.022980+0000", 2024-09-11T03:52:12.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015392641 2024-09-11T03:52:12.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:12.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.023063+0000", 2024-09-11T03:52:12.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3288999999999995e-05 2024-09-11T03:52:12.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:12.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.023176+0000", 2024-09-11T03:52:12.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112501 2024-09-11T03:52:12.939 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.939 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:12.939 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.940 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:7115 2.c e713/569 2:3603e322:::smithi17557870-42:head v 713'769, mlcod=713'767)", 2024-09-11T03:52:12.940 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:51.207248+0000", 2024-09-11T03:52:12.940 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 201.666677679, 2024-09-11T03:52:12.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015546002999999999, 2024-09-11T03:52:12.940 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:12.940 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:12.940 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:12.940 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:12.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:12.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.207248+0000", 2024-09-11T03:52:12.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:12.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.207248+0000", 2024-09-11T03:52:12.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:12.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.207250+0000", 2024-09-11T03:52:12.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6810000000000001e-06 2024-09-11T03:52:12.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:12.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.216300+0000", 2024-09-11T03:52:12.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0090505670000000007 2024-09-11T03:52:12.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:12.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.216317+0000", 2024-09-11T03:52:12.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6546999999999999e-05 2024-09-11T03:52:12.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:12.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.216325+0000", 2024-09-11T03:52:12.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3550000000000005e-06 2024-09-11T03:52:12.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.216394+0000", 2024-09-11T03:52:12.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8627000000000005e-05 2024-09-11T03:52:12.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:12.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.216419+0000", 2024-09-11T03:52:12.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5003000000000001e-05 2024-09-11T03:52:12.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:12.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.222743+0000", 2024-09-11T03:52:12.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0063239129999999996 2024-09-11T03:52:12.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:12.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.222794+0000", 2024-09-11T03:52:12.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1310000000000002e-05 2024-09-11T03:52:12.944 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.944 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:12.944 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.944 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7387 2.1f 2:fba9fb28:::smithi17557870-48:head [copy-from ver 916] snapc 14d=[14d,14b,14a,149,148,146,145,144,143,142,13d,13c,139,138,136,135,133,12e,12a,127,120,11e,119] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-09-11T03:52:12.945 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:49:22.015590+0000", 2024-09-11T03:52:12.945 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 170.85833546500001, 2024-09-11T03:52:12.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016730015000000001, 2024-09-11T03:52:12.945 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:12.945 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:12.945 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:12.945 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:12.945 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:12.945 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:12.945 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7387 2024-09-11T03:52:12.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.946 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:12.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:12.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.015590+0000", 2024-09-11T03:52:12.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:12.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.015590+0000", 2024-09-11T03:52:12.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:12.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.015592+0000", 2024-09-11T03:52:12.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9290000000000001e-06 2024-09-11T03:52:12.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:12.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.015600+0000", 2024-09-11T03:52:12.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6890000000000004e-06 2024-09-11T03:52:12.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:12.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.015617+0000", 2024-09-11T03:52:12.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.751e-05 2024-09-11T03:52:12.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:12.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.015626+0000", 2024-09-11T03:52:12.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7970000000000006e-06 2024-09-11T03:52:12.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.015694+0000", 2024-09-11T03:52:12.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7941999999999994e-05 2024-09-11T03:52:12.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:52:12.949 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.015703+0000", 2024-09-11T03:52:12.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6160000000000002e-06 2024-09-11T03:52:12.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.949 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.019591+0000", 2024-09-11T03:52:12.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003888318 2024-09-11T03:52:12.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:12.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.019689+0000", 2024-09-11T03:52:12.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8111999999999998e-05 2024-09-11T03:52:12.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 7,8", 2024-09-11T03:52:12.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.025025+0000", 2024-09-11T03:52:12.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053357129999999997 2024-09-11T03:52:12.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:12.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.028804+0000", 2024-09-11T03:52:12.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037790950000000001 2024-09-11T03:52:12.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:12.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.031565+0000", 2024-09-11T03:52:12.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027610209999999998 2024-09-11T03:52:12.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:12.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.032229+0000", 2024-09-11T03:52:12.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066412299999999997 2024-09-11T03:52:12.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:12.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.032272+0000", 2024-09-11T03:52:12.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2400000000000001e-05 2024-09-11T03:52:12.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:12.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:22.032320+0000", 2024-09-11T03:52:12.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8749999999999999e-05 2024-09-11T03:52:12.952 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.952 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:12.952 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.953 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7435 2.24 2:25f5f8b9:::smithi17557870-26:head [read 0~2874228,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e749)", 2024-09-11T03:52:12.953 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:49:27.143759+0000", 2024-09-11T03:52:12.953 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 165.73016650599999, 2024-09-11T03:52:12.953 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016534205, 2024-09-11T03:52:12.953 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:12.953 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:12.953 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:12.954 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:12.954 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:12.954 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:12.954 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7435 2024-09-11T03:52:12.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.954 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:12.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.954 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:12.954 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143759+0000", 2024-09-11T03:52:12.954 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.954 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:12.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143759+0000", 2024-09-11T03:52:12.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:12.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143761+0000", 2024-09-11T03:52:12.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5880000000000001e-06 2024-09-11T03:52:12.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.955 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:12.955 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143773+0000", 2024-09-11T03:52:12.955 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2599000000000001e-05 2024-09-11T03:52:12.955 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:12.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143786+0000", 2024-09-11T03:52:12.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.29e-05 2024-09-11T03:52:12.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:12.956 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143796+0000", 2024-09-11T03:52:12.956 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5289999999999997e-06 2024-09-11T03:52:12.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.956 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143949+0000", 2024-09-11T03:52:12.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000153499 2024-09-11T03:52:12.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:52:12.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143961+0000", 2024-09-11T03:52:12.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1382e-05 2024-09-11T03:52:12.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.957 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.957 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.156142+0000", 2024-09-11T03:52:12.957 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012180972 2024-09-11T03:52:12.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:12.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.156294+0000", 2024-09-11T03:52:12.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015199300000000001 2024-09-11T03:52:12.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.958 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:12.958 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.160293+0000", 2024-09-11T03:52:12.958 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003999743 2024-09-11T03:52:12.958 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.958 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:12.958 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.959 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7436 2.24 2:25f5f8b9:::smithi17557870-26:head [read 0~2874228,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e749)", 2024-09-11T03:52:12.959 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:49:27.143810+0000", 2024-09-11T03:52:12.959 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 165.730115942, 2024-09-11T03:52:12.959 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017603882000000001, 2024-09-11T03:52:12.959 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:12.959 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:12.959 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:12.959 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:12.959 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:12.959 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:12.959 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7436 2024-09-11T03:52:12.959 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.960 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:12.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.960 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:12.960 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143810+0000", 2024-09-11T03:52:12.960 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.960 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.960 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:12.960 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143810+0000", 2024-09-11T03:52:12.960 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.960 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.960 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:12.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143810+0000", 2024-09-11T03:52:12.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3e-07 2024-09-11T03:52:12.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:12.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143814+0000", 2024-09-11T03:52:12.961 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3890000000000001e-06 2024-09-11T03:52:12.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.961 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:12.961 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143821+0000", 2024-09-11T03:52:12.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5909999999999998e-06 2024-09-11T03:52:12.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:12.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143827+0000", 2024-09-11T03:52:12.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4929999999999998e-06 2024-09-11T03:52:12.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.962 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143994+0000", 2024-09-11T03:52:12.962 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016706300000000001 2024-09-11T03:52:12.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.962 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:52:12.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.144015+0000", 2024-09-11T03:52:12.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0859000000000001e-05 2024-09-11T03:52:12.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.160488+0000", 2024-09-11T03:52:12.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016472958999999999 2024-09-11T03:52:12.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.963 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:12.963 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.160605+0000", 2024-09-11T03:52:12.963 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011704799999999999 2024-09-11T03:52:12.963 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.964 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.964 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:12.964 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.161414+0000", 2024-09-11T03:52:12.964 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00080884999999999996 2024-09-11T03:52:12.964 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.964 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:12.964 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.964 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.964 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.964 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7437 2.24 2:25f5f8b9:::smithi17557870-26:head [sparse-read 0~2349940] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e749)", 2024-09-11T03:52:12.964 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:49:27.143834+0000", 2024-09-11T03:52:12.964 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 165.73009128199999, 2024-09-11T03:52:12.965 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018037481000000001, 2024-09-11T03:52:12.965 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:12.965 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:12.965 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:12.965 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:12.965 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:12.965 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:12.965 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7437 2024-09-11T03:52:12.965 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.965 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:12.965 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.965 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:12.965 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143834+0000", 2024-09-11T03:52:12.965 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.966 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:12.966 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143834+0000", 2024-09-11T03:52:12.966 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.966 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:12.966 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143835+0000", 2024-09-11T03:52:12.966 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9100000000000004e-07 2024-09-11T03:52:12.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.966 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:12.966 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143837+0000", 2024-09-11T03:52:12.966 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3889999999999999e-06 2024-09-11T03:52:12.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:12.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143843+0000", 2024-09-11T03:52:12.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9279999999999998e-06 2024-09-11T03:52:12.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:12.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.143848+0000", 2024-09-11T03:52:12.967 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7720000000000002e-06 2024-09-11T03:52:12.967 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.967 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.967 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.144062+0000", 2024-09-11T03:52:12.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021335000000000001 2024-09-11T03:52:12.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.968 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:52:12.968 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:12.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:12.969 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:12.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:12.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.144071+0000", 2024-09-11T03:52:12.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6099999999999995e-06 2024-09-11T03:52:12.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.161477+0000", 2024-09-11T03:52:12.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017405522999999999 2024-09-11T03:52:12.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:12.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.161607+0000", 2024-09-11T03:52:12.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013022500000000001 2024-09-11T03:52:12.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:12.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:27.161872+0000", 2024-09-11T03:52:12.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026509299999999999 2024-09-11T03:52:12.971 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.972 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:12.972 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.972 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7972 2.1e 2:7b0c3397:::smithi17557870-14:head [write 1942162~327047] snapc 162=[162,161,160,15f,15e,15d,15b,15a,157,156,155,154,153,151,14d,148,144,143,13c,139,135,12a,127,120] ondisk+write+known_if_redirected+supports_pool_eio e785)", 2024-09-11T03:52:12.972 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:50:03.101803+0000", 2024-09-11T03:52:12.972 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 129.77212307799999, 2024-09-11T03:52:12.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037451759000000001, 2024-09-11T03:52:12.972 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:12.972 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:12.972 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:12.972 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:12.972 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:12.972 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:12.973 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7972 2024-09-11T03:52:12.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.973 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:12.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:12.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.101803+0000", 2024-09-11T03:52:12.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:12.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.101803+0000", 2024-09-11T03:52:12.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:12.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.101804+0000", 2024-09-11T03:52:12.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1200000000000001e-06 2024-09-11T03:52:12.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:12.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102107+0000", 2024-09-11T03:52:12.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00030352599999999998 2024-09-11T03:52:12.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.974 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:12.974 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102118+0000", 2024-09-11T03:52:12.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0655999999999999e-05 2024-09-11T03:52:12.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:12.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102123+0000", 2024-09-11T03:52:12.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2630000000000003e-06 2024-09-11T03:52:12.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102162+0000", 2024-09-11T03:52:12.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9054000000000003e-05 2024-09-11T03:52:12.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:12.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102256+0000", 2024-09-11T03:52:12.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3591999999999994e-05 2024-09-11T03:52:12.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,10", 2024-09-11T03:52:12.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102374+0000", 2024-09-11T03:52:12.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000118497 2024-09-11T03:52:12.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:12.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103648+0000", 2024-09-11T03:52:12.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012732780000000001 2024-09-11T03:52:12.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:12.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103825+0000", 2024-09-11T03:52:12.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017712000000000001 2024-09-11T03:52:12.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:12.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139177+0000", 2024-09-11T03:52:12.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.035352078000000002 2024-09-11T03:52:12.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:12.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139218+0000", 2024-09-11T03:52:12.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1387999999999997e-05 2024-09-11T03:52:12.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:12.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139254+0000", 2024-09-11T03:52:12.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6186999999999998e-05 2024-09-11T03:52:12.978 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.978 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:12.978 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.978 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7973 2.1e 2:7b0c3397:::smithi17557870-14:head [setxattr _header (58) in=7b,truncate 2269209] snapc 162=[162,161,160,15f,15e,15d,15b,15a,157,156,155,154,153,151,14d,148,144,143,13c,139,135,12a,127,120] ondisk+write+known_if_redirected+supports_pool_eio e785)", 2024-09-11T03:52:12.978 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:50:03.102135+0000", 2024-09-11T03:52:12.978 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 129.77179106299999, 2024-09-11T03:52:12.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037234972999999998, 2024-09-11T03:52:12.979 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:12.979 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:12.979 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:12.979 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:12.979 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:12.979 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:12.979 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7973 2024-09-11T03:52:12.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.979 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:12.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.979 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:12.979 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102135+0000", 2024-09-11T03:52:12.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:12.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102135+0000", 2024-09-11T03:52:12.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:12.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102135+0000", 2024-09-11T03:52:12.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8299999999999997e-07 2024-09-11T03:52:12.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:12.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102139+0000", 2024-09-11T03:52:12.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4970000000000001e-06 2024-09-11T03:52:12.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:12.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102144+0000", 2024-09-11T03:52:12.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0499999999999999e-06 2024-09-11T03:52:12.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.981 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:12.981 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102148+0000", 2024-09-11T03:52:12.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9940000000000002e-06 2024-09-11T03:52:12.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103021+0000", 2024-09-11T03:52:12.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00087274900000000003 2024-09-11T03:52:12.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:12.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103119+0000", 2024-09-11T03:52:12.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8740000000000007e-05 2024-09-11T03:52:12.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,10", 2024-09-11T03:52:12.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103254+0000", 2024-09-11T03:52:12.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013457599999999999 2024-09-11T03:52:12.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:12.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103651+0000", 2024-09-11T03:52:12.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00039681999999999999 2024-09-11T03:52:12.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:12.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103971+0000", 2024-09-11T03:52:12.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032051399999999999 2024-09-11T03:52:12.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:12.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139293+0000", 2024-09-11T03:52:12.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.035321726999999997 2024-09-11T03:52:12.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:12.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139331+0000", 2024-09-11T03:52:12.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7934e-05 2024-09-11T03:52:12.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:12.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139370+0000", 2024-09-11T03:52:12.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8788999999999997e-05 2024-09-11T03:52:12.984 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.985 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:12.985 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.985 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.985 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7974 2.1e 2:7b0c3397:::smithi17557870-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e785)", 2024-09-11T03:52:12.985 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:50:03.102155+0000", 2024-09-11T03:52:12.985 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 129.77177079500001, 2024-09-11T03:52:12.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037356064000000001, 2024-09-11T03:52:12.985 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:12.985 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:12.985 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:12.985 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:12.989 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:12.989 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:12.989 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7974 2024-09-11T03:52:12.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.989 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:12.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:12.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102155+0000", 2024-09-11T03:52:12.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.990 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:12.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102155+0000", 2024-09-11T03:52:12.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.990 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:12.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102155+0000", 2024-09-11T03:52:12.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1500000000000005e-07 2024-09-11T03:52:12.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:12.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102157+0000", 2024-09-11T03:52:12.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.012e-06 2024-09-11T03:52:12.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:12.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102164+0000", 2024-09-11T03:52:12.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1689999999999997e-06 2024-09-11T03:52:12.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:12.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.102168+0000", 2024-09-11T03:52:12.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4229999999999998e-06 2024-09-11T03:52:12.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.992 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103583+0000", 2024-09-11T03:52:12.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014153950000000001 2024-09-11T03:52:12.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-11T03:52:12.992 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.103639+0000", 2024-09-11T03:52:12.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5517999999999999e-05 2024-09-11T03:52:12.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139402+0000", 2024-09-11T03:52:12.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.035763164 2024-09-11T03:52:12.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:12.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139451+0000", 2024-09-11T03:52:12.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8374999999999997e-05 2024-09-11T03:52:12.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:12.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:03.139511+0000", 2024-09-11T03:52:12.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0492999999999998e-05 2024-09-11T03:52:12.993 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.993 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:12.994 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:12.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.994 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:8853 2.47 2:e23d1762:::smithi17557870-21:head [write 753277~426666,stat] snapc 18e=[18e,18d,18c,18b,18a,189,188,187,186,185,184,183,181,180,17f,17e,17d,17c,17a,178,177,176,175,174,172,171,170,16e,16b,169,167,164,15f,15e,15b,157,14d,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e856)", 2024-09-11T03:52:12.994 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:14.153707+0000", 2024-09-11T03:52:12.994 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 58.720218672000001, 2024-09-11T03:52:12.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022893525000000001, 2024-09-11T03:52:12.994 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:12.994 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:12.994 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:12.994 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:12.994 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:12.994 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:12.995 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8853 2024-09-11T03:52:12.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.995 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:12.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.995 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:12.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.153707+0000", 2024-09-11T03:52:12.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.995 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:12.995 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.153707+0000", 2024-09-11T03:52:12.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:12.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:12.996 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.153709+0000", 2024-09-11T03:52:12.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1409999999999999e-06 2024-09-11T03:52:12.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:12.996 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.153984+0000", 2024-09-11T03:52:12.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000274276 2024-09-11T03:52:12.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:12.996 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.154005+0000", 2024-09-11T03:52:12.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.126e-05 2024-09-11T03:52:12.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:12.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.154015+0000", 2024-09-11T03:52:12.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9790000000000005e-06 2024-09-11T03:52:12.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:12.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.154087+0000", 2024-09-11T03:52:12.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2665999999999998e-05 2024-09-11T03:52:12.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:12.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.154220+0000", 2024-09-11T03:52:12.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013243799999999999 2024-09-11T03:52:12.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4,9", 2024-09-11T03:52:12.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.154465+0000", 2024-09-11T03:52:12.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024561599999999999 2024-09-11T03:52:12.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:12.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.155760+0000", 2024-09-11T03:52:12.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001294679 2024-09-11T03:52:12.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:12.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.175739+0000", 2024-09-11T03:52:12.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019978823 2024-09-11T03:52:12.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:12.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.176474+0000", 2024-09-11T03:52:12.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00073459299999999996 2024-09-11T03:52:12.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:12.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:12.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:12.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.176543+0000", 2024-09-11T03:52:12.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9130999999999999e-05 2024-09-11T03:52:13.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:13.000 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.176601+0000", 2024-09-11T03:52:13.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7923000000000001e-05 2024-09-11T03:52:13.000 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:13.000 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:13.000 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:13.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.000 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:8854 2.47 2:e23d1762:::smithi17557870-21:head [write 1681341~545753] snapc 18e=[18e,18d,18c,18b,18a,189,188,187,186,185,184,183,181,180,17f,17e,17d,17c,17a,178,177,176,175,174,172,171,170,16e,16b,169,167,164,15f,15e,15b,157,14d,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e856)", 2024-09-11T03:52:13.000 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:14.154037+0000", 2024-09-11T03:52:13.000 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 58.719888730999998, 2024-09-11T03:52:13.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024272202, 2024-09-11T03:52:13.001 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:13.001 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:13.001 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:13.001 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:13.001 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:13.001 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:13.001 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8854 2024-09-11T03:52:13.001 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.001 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:13.001 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.001 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:13.001 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.154037+0000", 2024-09-11T03:52:13.002 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:13.002 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.002 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.002 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:13.002 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.154037+0000", 2024-09-11T03:52:13.002 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:13.002 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.002 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.002 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:13.002 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.154038+0000", 2024-09-11T03:52:13.002 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7400000000000003e-07 2024-09-11T03:52:13.002 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.002 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.002 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:13.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.173986+0000", 2024-09-11T03:52:13.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019948766999999999 2024-09-11T03:52:13.003 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.003 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:13.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.174002+0000", 2024-09-11T03:52:13.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6065e-05 2024-09-11T03:52:13.003 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.003 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:13.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.174009+0000", 2024-09-11T03:52:13.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8770000000000004e-06 2024-09-11T03:52:13.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:13.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.174068+0000", 2024-09-11T03:52:13.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8538000000000001e-05 2024-09-11T03:52:13.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:13.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.174177+0000", 2024-09-11T03:52:13.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010955699999999999 2024-09-11T03:52:13.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4,9", 2024-09-11T03:52:13.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.174372+0000", 2024-09-11T03:52:13.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000194737 2024-09-11T03:52:13.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:13.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.175773+0000", 2024-09-11T03:52:13.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001400451 2024-09-11T03:52:13.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:13.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.177167+0000", 2024-09-11T03:52:13.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0013943619999999999 2024-09-11T03:52:13.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.006 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.006 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:13.006 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.178239+0000", 2024-09-11T03:52:13.006 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010718349999999999 2024-09-11T03:52:13.006 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.006 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.006 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:13.006 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.178277+0000", 2024-09-11T03:52:13.006 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8649000000000002e-05 2024-09-11T03:52:13.006 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.006 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.006 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:13.006 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:14.178309+0000", 2024-09-11T03:52:13.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1789999999999999e-05 2024-09-11T03:52:13.007 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:13.007 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:13.007 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:13.007 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.007 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:8911 2.1f 2:fa239c32:::smithi17557870-4:head [sparse-read 0~1721572,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e862)", 2024-09-11T03:52:13.007 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:20.057704+0000", 2024-09-11T03:52:13.007 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 52.816221300999999, 2024-09-11T03:52:13.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01885848, 2024-09-11T03:52:13.007 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:13.007 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:13.007 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:13.007 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:13.008 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:13.008 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:13.008 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8911 2024-09-11T03:52:13.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.008 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:13.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:13.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057704+0000", 2024-09-11T03:52:13.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:13.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:13.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057704+0000", 2024-09-11T03:52:13.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:13.009 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.009 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:13.009 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057706+0000", 2024-09-11T03:52:13.009 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4109999999999999e-06 2024-09-11T03:52:13.009 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.009 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:13.009 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057711+0000", 2024-09-11T03:52:13.009 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1080000000000001e-06 2024-09-11T03:52:13.009 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.009 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:13.010 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057722+0000", 2024-09-11T03:52:13.010 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0696000000000001e-05 2024-09-11T03:52:13.010 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.010 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.010 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:13.010 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057729+0000", 2024-09-11T03:52:13.010 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0650000000000001e-06 2024-09-11T03:52:13.010 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.010 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.010 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:13.010 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.061804+0000", 2024-09-11T03:52:13.010 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040757060000000001 2024-09-11T03:52:13.010 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.010 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.011 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-11T03:52:13.011 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.061912+0000", 2024-09-11T03:52:13.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107938 2024-09-11T03:52:13.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.011 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.011 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:13.011 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.072817+0000", 2024-09-11T03:52:13.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010904445 2024-09-11T03:52:13.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.011 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.011 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:13.011 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.072936+0000", 2024-09-11T03:52:13.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000119429 2024-09-11T03:52:13.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.012 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.012 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:13.012 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.076563+0000", 2024-09-11T03:52:13.012 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003626682 2024-09-11T03:52:13.012 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:13.012 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:13.012 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:13.012 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.012 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.012 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:8912 2.1f 2:fa239c32:::smithi17557870-4:head [read 0~3294436,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e862)", 2024-09-11T03:52:13.012 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:20.057925+0000", 2024-09-11T03:52:13.012 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 52.816000801999998, 2024-09-11T03:52:13.012 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020350545000000001, 2024-09-11T03:52:13.012 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:13.013 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:13.013 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:13.013 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:13.013 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:13.013 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:13.013 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8912 2024-09-11T03:52:13.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.013 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:13.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.013 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:13.013 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057925+0000", 2024-09-11T03:52:13.013 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:13.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:13.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057925+0000", 2024-09-11T03:52:13.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:13.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:13.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057926+0000", 2024-09-11T03:52:13.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.06e-07 2024-09-11T03:52:13.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:13.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057930+0000", 2024-09-11T03:52:13.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5349999999999998e-06 2024-09-11T03:52:13.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:13.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057941+0000", 2024-09-11T03:52:13.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0383999999999999e-05 2024-09-11T03:52:13.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:13.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057948+0000", 2024-09-11T03:52:13.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.114e-06 2024-09-11T03:52:13.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:13.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.062001+0000", 2024-09-11T03:52:13.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040532019999999997 2024-09-11T03:52:13.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-11T03:52:13.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.062097+0000", 2024-09-11T03:52:13.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5811999999999997e-05 2024-09-11T03:52:13.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:13.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.076602+0000", 2024-09-11T03:52:13.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014505348 2024-09-11T03:52:13.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:13.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.076716+0000", 2024-09-11T03:52:13.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011357300000000001 2024-09-11T03:52:13.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.017 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:13.017 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.078276+0000", 2024-09-11T03:52:13.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001559871 2024-09-11T03:52:13.017 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:13.017 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:13.017 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:13.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.017 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:8913 2.1f 2:fa239c32:::smithi17557870-4:head [read 0~3294436,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e862)", 2024-09-11T03:52:13.017 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:20.057958+0000", 2024-09-11T03:52:13.017 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 52.815967292000003, 2024-09-11T03:52:13.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021634403, 2024-09-11T03:52:13.018 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:13.018 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:13.018 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:13.018 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:13.018 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:13.018 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:13.018 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8913 2024-09-11T03:52:13.018 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.018 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:13.018 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.018 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:13.018 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057958+0000", 2024-09-11T03:52:13.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:13.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:13.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057958+0000", 2024-09-11T03:52:13.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:13.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:13.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057959+0000", 2024-09-11T03:52:13.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6899999999999997e-07 2024-09-11T03:52:13.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:13.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057962+0000", 2024-09-11T03:52:13.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8119999999999999e-06 2024-09-11T03:52:13.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:13.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057969+0000", 2024-09-11T03:52:13.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0199999999999997e-06 2024-09-11T03:52:13.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:13.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.057974+0000", 2024-09-11T03:52:13.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3249999999999998e-06 2024-09-11T03:52:13.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:13.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.062126+0000", 2024-09-11T03:52:13.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041514270000000001 2024-09-11T03:52:13.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-11T03:52:13.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.062596+0000", 2024-09-11T03:52:13.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00047031800000000001 2024-09-11T03:52:13.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:13.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.078477+0000", 2024-09-11T03:52:13.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015881119999999999 2024-09-11T03:52:13.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.022 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:13.022 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.078578+0000", 2024-09-11T03:52:13.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000101343 2024-09-11T03:52:13.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:13.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:13.022 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:13.022 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:20.079593+0000", 2024-09-11T03:52:13.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001014469 2024-09-11T03:52:13.022 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:13.022 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:13.022 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:13.022 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:13.022 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:13.022 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:52:13.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:13.071 INFO:tasks.rados.rados.2.smithi175.stdout:3804: done (0 left) 2024-09-11T03:52:13.071 INFO:tasks.rados.rados.2.smithi175.stdout:3805: write (excl) oid 30 current snap is 424 2024-09-11T03:52:13.072 INFO:tasks.rados.rados.2.smithi175.stdout:3805: seq_num 1288 ranges {447416=584369,1537278=479774,2537050=529882,3524784=122632} 2024-09-11T03:52:13.077 INFO:tasks.rados.rados.2.smithi175.stdout:3805: writing smithi17557870-30 from 447416 to 1031785 tid 1 2024-09-11T03:52:13.084 INFO:tasks.rados.rados.2.smithi175.stdout:3805: writing smithi17557870-30 from 1537278 to 2017052 tid 2 2024-09-11T03:52:13.089 INFO:tasks.rados.rados.2.smithi175.stdout:3805: writing smithi17557870-30 from 2537050 to 3066932 tid 3 2024-09-11T03:52:13.093 INFO:tasks.rados.rados.2.smithi175.stdout:3805: writing smithi17557870-30 from 3524784 to 3647416 tid 4 2024-09-11T03:52:13.093 INFO:tasks.rados.rados.2.smithi175.stdout:3806: write oid 6 current snap is 424 2024-09-11T03:52:13.094 INFO:tasks.rados.rados.2.smithi175.stdout:3805: finishing write tid 1 to smithi17557870-30 2024-09-11T03:52:13.094 INFO:tasks.rados.rados.2.smithi175.stdout:3805: finishing write tid 2 to smithi17557870-30 2024-09-11T03:52:13.094 INFO:tasks.rados.rados.2.smithi175.stdout:3805: finishing write tid 3 to smithi17557870-30 2024-09-11T03:52:13.094 INFO:tasks.rados.rados.2.smithi175.stdout:3806: seq_num 1289 ranges {464223=659618,1879407=739785,3248120=416103} 2024-09-11T03:52:13.098 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:13.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:13.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:13.100 INFO:tasks.rados.rados.2.smithi175.stdout:3806: writing smithi17557870-6 from 464223 to 1123841 tid 1 2024-09-11T03:52:13.107 INFO:tasks.rados.rados.2.smithi175.stdout:3806: writing smithi17557870-6 from 1879407 to 2619192 tid 2 2024-09-11T03:52:13.113 INFO:tasks.rados.rados.2.smithi175.stdout:3806: writing smithi17557870-6 from 3248120 to 3664223 tid 3 2024-09-11T03:52:13.113 INFO:tasks.rados.rados.2.smithi175.stdout:3807: snap_remove snap 388 2024-09-11T03:52:13.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:13.224 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:13.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:13.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:13.327 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:13.352 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:13.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:13.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:13.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:13 smithi084 ceph-mon[27254]: 2.2a scrub starts 2024-09-11T03:52:13.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:13 smithi084 ceph-mon[27254]: 2.2a scrub ok 2024-09-11T03:52:13.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:13 smithi084 ceph-mon[27254]: from='client.17502 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:52:13.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:13 smithi084 ceph-mon[27254]: osdmap e915: 12 total, 12 up, 11 in 2024-09-11T03:52:13.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:13 smithi121 ceph-mon[30729]: 2.2a scrub starts 2024-09-11T03:52:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:13 smithi121 ceph-mon[30729]: 2.2a scrub ok 2024-09-11T03:52:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:13 smithi121 ceph-mon[30729]: from='client.17502 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:52:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:13 smithi121 ceph-mon[30729]: osdmap e915: 12 total, 12 up, 11 in 2024-09-11T03:52:13.454 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:13 smithi175 ceph-mon[30894]: 2.2a scrub starts 2024-09-11T03:52:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:13 smithi175 ceph-mon[30894]: 2.2a scrub ok 2024-09-11T03:52:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:13 smithi175 ceph-mon[30894]: from='client.17502 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:52:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:13 smithi175 ceph-mon[30894]: osdmap e915: 12 total, 12 up, 11 in 2024-09-11T03:52:13.478 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:13.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:13.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:13.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:13.604 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:13.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:13.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:13.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:13.728 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:13.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:13.830 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:13.850 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:13.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:13.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:13.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:13.977 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:13.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:13.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:13.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:52:14.064 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:52:14.067 INFO:tasks.rados.rados.2.smithi175.stdout:3807: done (2 left) 2024-09-11T03:52:14.067 INFO:tasks.rados.rados.2.smithi175.stdout:3808: read oid 35 snap 421 2024-09-11T03:52:14.068 INFO:tasks.rados.rados.2.smithi175.stdout:3808: expect (ObjNum 1256 snap 412 seq_num 1256) 2024-09-11T03:52:14.068 INFO:tasks.rados.rados.2.smithi175.stdout:3809: read oid 50 snap -1 2024-09-11T03:52:14.068 INFO:tasks.rados.rados.2.smithi175.stdout:3809: expect (ObjNum 1190 snap 388 seq_num 1190) 2024-09-11T03:52:14.068 INFO:tasks.rados.rados.2.smithi175.stdout:3810: write oid 39 current snap is 424 2024-09-11T03:52:14.068 INFO:tasks.rados.rados.2.smithi175.stdout:3805: finishing write tid 4 to smithi17557870-30 2024-09-11T03:52:14.068 INFO:tasks.rados.rados.2.smithi175.stdout:3805: finishing write tid 5 to smithi17557870-30 2024-09-11T03:52:14.068 INFO:tasks.rados.rados.2.smithi175.stdout:3805: finishing write tid 6 to smithi17557870-30 2024-09-11T03:52:14.068 INFO:tasks.rados.rados.2.smithi175.stdout:3805: oid 30 updating version 0 to 1226 2024-09-11T03:52:14.068 INFO:tasks.rados.rados.2.smithi175.stdout:3805: oid 30 updating version 1226 to 1227 2024-09-11T03:52:14.068 INFO:tasks.rados.rados.2.smithi175.stdout:3805: oid 30 version 1227 is already newer than 1225 2024-09-11T03:52:14.068 INFO:tasks.rados.rados.2.smithi175.stdout:3805: oid 30 updating version 1227 to 1228 2024-09-11T03:52:14.068 INFO:tasks.rados.rados.2.smithi175.stdout:3805: oid 30 updating version 1228 to 1229 2024-09-11T03:52:14.068 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 1229 (ObjNum 1288 snap 424 seq_num 1288) dirty exists 2024-09-11T03:52:14.068 INFO:tasks.rados.rados.2.smithi175.stdout:3805: left oid 30 (ObjNum 1288 snap 424 seq_num 1288) 2024-09-11T03:52:14.069 INFO:tasks.rados.rados.2.smithi175.stdout:3806: finishing write tid 1 to smithi17557870-6 2024-09-11T03:52:14.069 INFO:tasks.rados.rados.2.smithi175.stdout:3806: finishing write tid 2 to smithi17557870-6 2024-09-11T03:52:14.069 INFO:tasks.rados.rados.2.smithi175.stdout:3806: finishing write tid 3 to smithi17557870-6 2024-09-11T03:52:14.069 INFO:tasks.rados.rados.2.smithi175.stdout:3810: seq_num 1290 ranges {481030=734867,1937889=483443,2875543=552426} 2024-09-11T03:52:14.075 INFO:tasks.rados.rados.2.smithi175.stdout:3810: writing smithi17557870-39 from 481030 to 1215897 tid 1 2024-09-11T03:52:14.079 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:14.082 INFO:tasks.rados.rados.2.smithi175.stdout:3810: writing smithi17557870-39 from 1937889 to 2421332 tid 2 2024-09-11T03:52:14.090 INFO:tasks.rados.rados.2.smithi175.stdout:3810: writing smithi17557870-39 from 2875543 to 3427969 tid 3 2024-09-11T03:52:14.090 INFO:tasks.rados.rados.2.smithi175.stdout:3805: done (4 left) 2024-09-11T03:52:14.090 INFO:tasks.rados.rados.2.smithi175.stdout:3811: rollback oid 24 current snap is 424 2024-09-11T03:52:14.090 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 24 to 409 2024-09-11T03:52:14.090 INFO:tasks.rados.rados.2.smithi175.stdout:3806: finishing write tid 4 to smithi17557870-6 2024-09-11T03:52:14.090 INFO:tasks.rados.rados.2.smithi175.stdout:3806: finishing write tid 5 to smithi17557870-6 2024-09-11T03:52:14.090 INFO:tasks.rados.rados.2.smithi175.stdout:3806: oid 6 updating version 0 to 996 2024-09-11T03:52:14.090 INFO:tasks.rados.rados.2.smithi175.stdout:3806: oid 6 version 996 is already newer than 995 2024-09-11T03:52:14.090 INFO:tasks.rados.rados.2.smithi175.stdout:3806: oid 6 updating version 996 to 997 2024-09-11T03:52:14.090 INFO:tasks.rados.rados.2.smithi175.stdout:3806: oid 6 version 997 is already newer than 994 2024-09-11T03:52:14.090 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 997 (ObjNum 1289 snap 424 seq_num 1289) dirty exists 2024-09-11T03:52:14.090 INFO:tasks.rados.rados.2.smithi175.stdout:3806: left oid 6 (ObjNum 1289 snap 424 seq_num 1289) 2024-09-11T03:52:14.091 INFO:tasks.rados.rados.2.smithi175.stdout:3809: expect (ObjNum 1190 snap 388 seq_num 1190) 2024-09-11T03:52:14.109 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:14.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:14.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:14.158 INFO:tasks.rados.rados.2.smithi175.stdout:3810: finishing write tid 1 to smithi17557870-39 2024-09-11T03:52:14.158 INFO:tasks.rados.rados.2.smithi175.stdout:3808: expect (ObjNum 1256 snap 412 seq_num 1256) 2024-09-11T03:52:14.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:14.235 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:14.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:14.261 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:14 smithi084 ceph-mon[27254]: pgmap v1527: 107 pgs: 107 active+clean; 755 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 31 MiB/s rd, 5.1 MiB/s wr, 42 op/s 2024-09-11T03:52:14.261 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:14 smithi084 ceph-mon[27254]: 2.21 scrub starts 2024-09-11T03:52:14.261 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:14 smithi084 ceph-mon[27254]: 2.21 scrub ok 2024-09-11T03:52:14.261 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:14 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3125609945' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:52:14.261 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:14 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:52:14.261 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:14 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:52:14.262 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:14 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3125609945' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:52:14.262 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:14 smithi084 ceph-mon[27254]: osdmap e916: 12 total, 12 up, 11 in 2024-09-11T03:52:14.276 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-11T03:52:14.337 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:14.358 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:14.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:14.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:14 smithi121 ceph-mon[30729]: pgmap v1527: 107 pgs: 107 active+clean; 755 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 31 MiB/s rd, 5.1 MiB/s wr, 42 op/s 2024-09-11T03:52:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:14 smithi121 ceph-mon[30729]: 2.21 scrub starts 2024-09-11T03:52:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:14 smithi121 ceph-mon[30729]: 2.21 scrub ok 2024-09-11T03:52:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:14 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3125609945' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:52:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:14 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:52:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:14 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:52:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:14 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3125609945' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:52:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:14 smithi121 ceph-mon[30729]: osdmap e916: 12 total, 12 up, 11 in 2024-09-11T03:52:14.433 INFO:tasks.rados.rados.2.smithi175.stdout:3810: finishing write tid 2 to smithi17557870-39 2024-09-11T03:52:14.433 INFO:tasks.rados.rados.2.smithi175.stdout:3811: finishing rollback tid 0 to smithi17557870-24 2024-09-11T03:52:14.433 INFO:tasks.rados.rados.2.smithi175.stdout:3806: done (4 left) 2024-09-11T03:52:14.433 INFO:tasks.rados.rados.2.smithi175.stdout:3808: done (3 left) 2024-09-11T03:52:14.433 INFO:tasks.rados.rados.2.smithi175.stdout:3809: done (2 left) 2024-09-11T03:52:14.433 INFO:tasks.rados.rados.2.smithi175.stdout:3812: read oid 23 snap 407 2024-09-11T03:52:14.433 INFO:tasks.rados.rados.2.smithi175.stdout:3812: expect deleted 2024-09-11T03:52:14.433 INFO:tasks.rados.rados.2.smithi175.stdout:3813: snap_create 2024-09-11T03:52:14.433 INFO:tasks.rados.rados.2.smithi175.stdout:3811: finishing rollback tid 1 to smithi17557870-24 2024-09-11T03:52:14.433 INFO:tasks.rados.rados.2.smithi175.stdout:3811: finishing rollback tid 2 to smithi17557870-24 2024-09-11T03:52:14.433 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 1112 (ObjNum 1235 snap 402 seq_num 1235) dirty exists 2024-09-11T03:52:14.434 INFO:tasks.rados.rados.2.smithi175.stdout:3810: finishing write tid 3 to smithi17557870-39 2024-09-11T03:52:14.434 INFO:tasks.rados.rados.2.smithi175.stdout:3810: finishing write tid 4 to smithi17557870-39 2024-09-11T03:52:14.434 INFO:tasks.rados.rados.2.smithi175.stdout:3810: finishing write tid 5 to smithi17557870-39 2024-09-11T03:52:14.434 INFO:tasks.rados.rados.2.smithi175.stdout:3810: oid 39 updating version 0 to 1028 2024-09-11T03:52:14.434 INFO:tasks.rados.rados.2.smithi175.stdout:3810: oid 39 updating version 1028 to 1029 2024-09-11T03:52:14.434 INFO:tasks.rados.rados.2.smithi175.stdout:3810: oid 39 version 1029 is already newer than 1027 2024-09-11T03:52:14.434 INFO:tasks.rados.rados.2.smithi175.stdout:3810: oid 39 version 1029 is already newer than 1026 2024-09-11T03:52:14.434 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 1029 (ObjNum 1290 snap 424 seq_num 1290) dirty exists 2024-09-11T03:52:14.434 INFO:tasks.rados.rados.2.smithi175.stdout:3810: left oid 39 (ObjNum 1290 snap 424 seq_num 1290) 2024-09-11T03:52:14.459 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:14 smithi175 ceph-mon[30894]: pgmap v1527: 107 pgs: 107 active+clean; 755 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 31 MiB/s rd, 5.1 MiB/s wr, 42 op/s 2024-09-11T03:52:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:14 smithi175 ceph-mon[30894]: 2.21 scrub starts 2024-09-11T03:52:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:14 smithi175 ceph-mon[30894]: 2.21 scrub ok 2024-09-11T03:52:14.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:14 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3125609945' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:52:14.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:14 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:52:14.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:14 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:52:14.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:14 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3125609945' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:52:14.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:14 smithi175 ceph-mon[30894]: osdmap e916: 12 total, 12 up, 11 in 2024-09-11T03:52:14.481 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:14.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:14.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:14.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:14.609 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:14.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:14.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:14.712 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:14.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:14.735 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:14.735 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:14.836 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:14.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:14.859 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:14.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:14.922 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1526,"stamp":"2024-09-11T03:52:12.062627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":778376971,"num_objects":329,"num_object_clones":277,"num_object_copies":987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":11,"num_read":1073,"num_read_kb":1147992,"num_write":713,"num_write_kb":286440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":82584822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50716,"ondisk_log_size":50716,"up":321,"acting":321,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":321,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":1975272,"kb_used_data":1368436,"kb_used_omap":17,"kb_used_meta":606574,"kb_avail":1029221400,"statfs":{"total":1055945392128,"available":1053922713600,"internally_reserved":0,"allocated":1401278464,"data_stored":1768149547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17546,"internal_metadata":621132662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13771129,"num_objects":9,"num_object_clones":9,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":71,"num_read_kb":80360,"num_write":35,"num_write_kb":11494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.969704"},"pg_stats":[{"pgid":"2.4b","version":"568'646","reported_seq":4784,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.124207+0000","last_change":"2024-09-11T03:52:11.124495+0000","last_active":"2024-09-11T03:52:11.124207+0000","last_peered":"2024-09-11T03:52:11.124207+0000","last_clean":"2024-09-11T03:52:11.124207+0000","last_became_active":"2024-09-11T03:51:13.079045+0000","last_became_peered":"2024-09-11T03:51:13.079045+0000","last_unstale":"2024-09-11T03:52:11.124207+0000","last_undegraded":"2024-09-11T03:52:11.124207+0000","last_fullsized":"2024-09-11T03:52:11.124207+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:51:21.370186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.370186+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:34.854491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056293949999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.4a","version":"848'386","reported_seq":4509,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.291508+0000","last_change":"2024-09-11T03:52:06.090022+0000","last_active":"2024-09-11T03:52:07.291508+0000","last_peered":"2024-09-11T03:52:07.291508+0000","last_clean":"2024-09-11T03:52:07.291508+0000","last_became_active":"2024-09-11T03:51:13.080958+0000","last_became_peered":"2024-09-11T03:51:13.080958+0000","last_unstale":"2024-09-11T03:52:07.291508+0000","last_undegraded":"2024-09-11T03:52:07.291508+0000","last_fullsized":"2024-09-11T03:52:07.291508+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"848'386","last_scrub_stamp":"2024-09-11T03:51:19.939163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.939163+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:40.349217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070724299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.49","version":"913'1540","reported_seq":5854,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.141000+0000","last_change":"2024-09-11T03:52:11.141231+0000","last_active":"2024-09-11T03:52:11.141000+0000","last_peered":"2024-09-11T03:52:11.141000+0000","last_clean":"2024-09-11T03:52:11.141000+0000","last_became_active":"2024-09-11T03:51:20.066997+0000","last_became_peered":"2024-09-11T03:51:20.066997+0000","last_unstale":"2024-09-11T03:52:11.141000+0000","last_undegraded":"2024-09-11T03:52:11.141000+0000","last_fullsized":"2024-09-11T03:52:11.141000+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":7,"last_scrub":"863'1411","last_scrub_stamp":"2024-09-11T03:51:21.422037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.422037+0000","objects_scrubbed":18,"log_size":1540,"log_dups_size":0,"ondisk_log_size":1540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:42.537913+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0092159779999999997,"stat_sum":{"num_bytes":49712115,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":30,"num_read_kb":44216,"num_write":13,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.48","version":"744'374","reported_seq":3997,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.101326+0000","last_change":"2024-09-11T03:52:08.102236+0000","last_active":"2024-09-11T03:52:08.101326+0000","last_peered":"2024-09-11T03:52:08.101326+0000","last_clean":"2024-09-11T03:52:08.101326+0000","last_became_active":"2024-09-11T03:51:13.082676+0000","last_became_peered":"2024-09-11T03:51:13.082676+0000","last_unstale":"2024-09-11T03:52:08.101326+0000","last_undegraded":"2024-09-11T03:52:08.101326+0000","last_fullsized":"2024-09-11T03:52:08.101326+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"744'374","last_scrub_stamp":"2024-09-11T03:51:19.306681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.306681+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:28.816443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011360090000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,0],"acting":[9,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.47","version":"908'1318","reported_seq":5620,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-09-11T03:52:10.190954+0000","last_change":"2024-09-11T03:52:08.100176+0000","last_active":"2024-09-11T03:52:10.190954+0000","last_peered":"2024-09-11T03:52:10.190954+0000","last_clean":"2024-09-11T03:52:10.190954+0000","last_became_active":"2024-09-11T03:51:20.064024+0000","last_became_peered":"2024-09-11T03:51:20.064024+0000","last_unstale":"2024-09-11T03:52:10.190954+0000","last_undegraded":"2024-09-11T03:52:10.190954+0000","last_fullsized":"2024-09-11T03:52:10.190954+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":7,"last_scrub":"862'1274","last_scrub_stamp":"2024-09-11T03:51:20.166795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.166795+0000","objects_scrubbed":7,"log_size":1318,"log_dups_size":0,"ondisk_log_size":1318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:33.740608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012869559999999999,"stat_sum":{"num_bytes":19286976,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":11528,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.46","version":"269'171","reported_seq":4259,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.129136+0000","last_change":"2024-09-11T03:52:11.129389+0000","last_active":"2024-09-11T03:52:11.129136+0000","last_peered":"2024-09-11T03:52:11.129136+0000","last_clean":"2024-09-11T03:52:11.129136+0000","last_became_active":"2024-09-11T03:51:13.080822+0000","last_became_peered":"2024-09-11T03:51:13.080822+0000","last_unstale":"2024-09-11T03:52:11.129136+0000","last_undegraded":"2024-09-11T03:52:11.129136+0000","last_fullsized":"2024-09-11T03:52:11.129136+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:51:21.034557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.034557+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:27.459473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013927360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.45","version":"913'1240","reported_seq":5387,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.123278+0000","last_change":"2024-09-11T03:52:11.123466+0000","last_active":"2024-09-11T03:52:11.123278+0000","last_peered":"2024-09-11T03:52:11.123278+0000","last_clean":"2024-09-11T03:52:11.123278+0000","last_became_active":"2024-09-11T03:51:13.078704+0000","last_became_peered":"2024-09-11T03:51:13.078704+0000","last_unstale":"2024-09-11T03:52:11.123278+0000","last_undegraded":"2024-09-11T03:52:11.123278+0000","last_fullsized":"2024-09-11T03:52:11.123278+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"863'1213","last_scrub_stamp":"2024-09-11T03:51:23.430916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.430916+0000","objects_scrubbed":4,"log_size":1240,"log_dups_size":0,"ondisk_log_size":1240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:39.246155+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0051281640000000002,"stat_sum":{"num_bytes":13996135,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":6,"num_write_kb":1354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[1,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.44","version":"913'761","reported_seq":5020,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.121876+0000","last_change":"2024-09-11T03:52:11.122275+0000","last_active":"2024-09-11T03:52:11.121876+0000","last_peered":"2024-09-11T03:52:11.121876+0000","last_clean":"2024-09-11T03:52:11.121876+0000","last_became_active":"2024-09-11T03:51:13.079319+0000","last_became_peered":"2024-09-11T03:51:13.079319+0000","last_unstale":"2024-09-11T03:52:11.121876+0000","last_undegraded":"2024-09-11T03:52:11.121876+0000","last_fullsized":"2024-09-11T03:52:11.121876+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"862'706","last_scrub_stamp":"2024-09-11T03:51:20.328187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.328187+0000","objects_scrubbed":12,"log_size":761,"log_dups_size":0,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:50.057325+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0040792470000000003,"stat_sum":{"num_bytes":23780556,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":8,"num_write_kb":7615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,3],"acting":[1,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.43","version":"913'622","reported_seq":4790,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.134864+0000","last_change":"2024-09-11T03:52:11.135104+0000","last_active":"2024-09-11T03:52:11.134864+0000","last_peered":"2024-09-11T03:52:11.134864+0000","last_clean":"2024-09-11T03:52:11.134864+0000","last_became_active":"2024-09-11T03:51:13.083077+0000","last_became_peered":"2024-09-11T03:51:13.083077+0000","last_unstale":"2024-09-11T03:52:11.134864+0000","last_undegraded":"2024-09-11T03:52:11.134864+0000","last_fullsized":"2024-09-11T03:52:11.134864+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"862'574","last_scrub_stamp":"2024-09-11T03:51:22.027988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.027988+0000","objects_scrubbed":9,"log_size":622,"log_dups_size":0,"ondisk_log_size":622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:32.733723+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0063353100000000002,"stat_sum":{"num_bytes":26795402,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":11849,"num_write":8,"num_write_kb":11222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.42","version":"913'348","reported_seq":4498,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.129244+0000","last_change":"2024-09-11T03:52:11.129540+0000","last_active":"2024-09-11T03:52:11.129244+0000","last_peered":"2024-09-11T03:52:11.129244+0000","last_clean":"2024-09-11T03:52:11.129244+0000","last_became_active":"2024-09-11T03:51:13.080914+0000","last_became_peered":"2024-09-11T03:51:13.080914+0000","last_unstale":"2024-09-11T03:52:11.129244+0000","last_undegraded":"2024-09-11T03:52:11.129244+0000","last_fullsized":"2024-09-11T03:52:11.129244+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"859'323","last_scrub_stamp":"2024-09-11T03:51:19.628572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.628572+0000","objects_scrubbed":5,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:31.294973+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0046598780000000001,"stat_sum":{"num_bytes":15883770,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,11],"acting":[8,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.41","version":"567'235","reported_seq":4275,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.291026+0000","last_change":"2024-09-11T03:52:06.088508+0000","last_active":"2024-09-11T03:52:07.291026+0000","last_peered":"2024-09-11T03:52:07.291026+0000","last_clean":"2024-09-11T03:52:07.291026+0000","last_became_active":"2024-09-11T03:50:33.046488+0000","last_became_peered":"2024-09-11T03:50:33.046488+0000","last_unstale":"2024-09-11T03:52:07.291026+0000","last_undegraded":"2024-09-11T03:52:07.291026+0000","last_fullsized":"2024-09-11T03:52:07.291026+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":7,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:50:35.786143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.786143+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091889399999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.40","version":"563'192","reported_seq":4344,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.062871+0000","last_change":"2024-09-11T03:52:06.082462+0000","last_active":"2024-09-11T03:52:07.062871+0000","last_peered":"2024-09-11T03:52:07.062871+0000","last_clean":"2024-09-11T03:52:07.062871+0000","last_became_active":"2024-09-11T03:51:20.064866+0000","last_became_peered":"2024-09-11T03:51:20.064866+0000","last_unstale":"2024-09-11T03:52:07.062871+0000","last_undegraded":"2024-09-11T03:52:07.062871+0000","last_fullsized":"2024-09-11T03:52:07.062871+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":7,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:50:35.472175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.472175+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020413409999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,8],"acting":[6,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.3f","version":"911'812","reported_seq":4927,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.135686+0000","last_change":"2024-09-11T03:52:11.136963+0000","last_active":"2024-09-11T03:52:11.135686+0000","last_peered":"2024-09-11T03:52:11.135686+0000","last_clean":"2024-09-11T03:52:11.135686+0000","last_became_active":"2024-09-11T03:50:33.046798+0000","last_became_peered":"2024-09-11T03:50:33.046798+0000","last_unstale":"2024-09-11T03:52:11.135686+0000","last_undegraded":"2024-09-11T03:52:11.135686+0000","last_fullsized":"2024-09-11T03:52:11.135686+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"894'803","last_scrub_stamp":"2024-09-11T03:51:53.038706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.038706+0000","objects_scrubbed":5,"log_size":812,"log_dups_size":0,"ondisk_log_size":812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:04.853901+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.007682843,"stat_sum":{"num_bytes":13313731,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":18511,"num_write":10,"num_write_kb":2923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.3e","version":"908'1102","reported_seq":5148,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.088012+0000","last_change":"2024-09-11T03:52:06.088241+0000","last_active":"2024-09-11T03:52:06.088012+0000","last_peered":"2024-09-11T03:52:06.088012+0000","last_clean":"2024-09-11T03:52:06.088012+0000","last_became_active":"2024-09-11T03:50:33.045235+0000","last_became_peered":"2024-09-11T03:50:33.045235+0000","last_unstale":"2024-09-11T03:52:06.088012+0000","last_undegraded":"2024-09-11T03:52:06.088012+0000","last_fullsized":"2024-09-11T03:52:06.088012+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"893'1098","last_scrub_stamp":"2024-09-11T03:51:52.144830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.144830+0000","objects_scrubbed":7,"log_size":1102,"log_dups_size":0,"ondisk_log_size":1102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:20.562377+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0073891920000000002,"stat_sum":{"num_bytes":16485077,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.3d","version":"567'752","reported_seq":3645,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.123537+0000","last_change":"2024-09-11T03:52:11.123921+0000","last_active":"2024-09-11T03:52:11.123537+0000","last_peered":"2024-09-11T03:52:11.123537+0000","last_clean":"2024-09-11T03:52:11.123537+0000","last_became_active":"2024-09-11T03:51:20.070161+0000","last_became_peered":"2024-09-11T03:51:20.070161+0000","last_unstale":"2024-09-11T03:52:11.123537+0000","last_undegraded":"2024-09-11T03:52:11.123537+0000","last_fullsized":"2024-09-11T03:52:11.123537+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:51:53.597428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.597428+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:18.996560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0054813980000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.3c","version":"912'794","reported_seq":4920,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.128215+0000","last_change":"2024-09-11T03:52:11.129387+0000","last_active":"2024-09-11T03:52:11.128215+0000","last_peered":"2024-09-11T03:52:11.128215+0000","last_clean":"2024-09-11T03:52:11.128215+0000","last_became_active":"2024-09-11T03:50:33.048487+0000","last_became_peered":"2024-09-11T03:50:33.048487+0000","last_unstale":"2024-09-11T03:52:11.128215+0000","last_undegraded":"2024-09-11T03:52:11.128215+0000","last_fullsized":"2024-09-11T03:52:11.128215+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"886'783","last_scrub_stamp":"2024-09-11T03:51:45.009597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:45.009597+0000","objects_scrubbed":3,"log_size":794,"log_dups_size":0,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:13.201890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016691329999999999,"stat_sum":{"num_bytes":12571082,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":10182,"num_write":11,"num_write_kb":5447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,10],"acting":[7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.3b","version":"912'973","reported_seq":5312,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.139332+0000","last_change":"2024-09-11T03:52:11.139605+0000","last_active":"2024-09-11T03:52:11.139332+0000","last_peered":"2024-09-11T03:52:11.139332+0000","last_clean":"2024-09-11T03:52:11.139332+0000","last_became_active":"2024-09-11T03:51:20.066390+0000","last_became_peered":"2024-09-11T03:51:20.066390+0000","last_unstale":"2024-09-11T03:52:11.139332+0000","last_undegraded":"2024-09-11T03:52:11.139332+0000","last_fullsized":"2024-09-11T03:52:11.139332+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"905'970","last_scrub_stamp":"2024-09-11T03:52:10.225202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:10.225202+0000","objects_scrubbed":9,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:32.983276+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0059857699999999996,"stat_sum":{"num_bytes":25330129,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":40,"num_read_kb":50518,"num_write":22,"num_write_kb":3826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":14555611,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.3a","version":"557'273","reported_seq":3123,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.103394+0000","last_change":"2024-09-11T03:52:08.104227+0000","last_active":"2024-09-11T03:52:08.103394+0000","last_peered":"2024-09-11T03:52:08.103394+0000","last_clean":"2024-09-11T03:52:08.103394+0000","last_became_active":"2024-09-11T03:51:20.061387+0000","last_became_peered":"2024-09-11T03:51:20.061387+0000","last_unstale":"2024-09-11T03:52:08.103394+0000","last_undegraded":"2024-09-11T03:52:08.103394+0000","last_fullsized":"2024-09-11T03:52:08.103394+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:50:37.286055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.286055+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019623599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.39","version":"806'585","reported_seq":4728,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.103040+0000","last_change":"2024-09-11T03:52:08.103340+0000","last_active":"2024-09-11T03:52:08.103040+0000","last_peered":"2024-09-11T03:52:08.103040+0000","last_clean":"2024-09-11T03:52:08.103040+0000","last_became_active":"2024-09-11T03:50:33.043651+0000","last_became_peered":"2024-09-11T03:50:33.043651+0000","last_unstale":"2024-09-11T03:52:08.103040+0000","last_undegraded":"2024-09-11T03:52:08.103040+0000","last_fullsized":"2024-09-11T03:52:08.103040+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"806'585","last_scrub_stamp":"2024-09-11T03:50:36.264989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.264989+0000","objects_scrubbed":0,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012109340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,2],"acting":[9,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.38","version":"808'564","reported_seq":4737,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.126301+0000","last_change":"2024-09-11T03:52:11.126570+0000","last_active":"2024-09-11T03:52:11.126301+0000","last_peered":"2024-09-11T03:52:11.126301+0000","last_clean":"2024-09-11T03:52:11.126301+0000","last_became_active":"2024-09-11T03:50:33.042740+0000","last_became_peered":"2024-09-11T03:50:33.042740+0000","last_unstale":"2024-09-11T03:52:11.126301+0000","last_undegraded":"2024-09-11T03:52:11.126301+0000","last_fullsized":"2024-09-11T03:52:11.126301+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"808'564","last_scrub_stamp":"2024-09-11T03:51:54.462104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.462104+0000","objects_scrubbed":0,"log_size":564,"log_dups_size":0,"ondisk_log_size":564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:02.548438+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018154200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,1],"acting":[8,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.37","version":"274'312","reported_seq":2688,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.290892+0000","last_change":"2024-09-11T03:52:06.092126+0000","last_active":"2024-09-11T03:52:07.290892+0000","last_peered":"2024-09-11T03:52:07.290892+0000","last_clean":"2024-09-11T03:52:07.290892+0000","last_became_active":"2024-09-11T03:50:07.077607+0000","last_became_peered":"2024-09-11T03:50:07.077607+0000","last_unstale":"2024-09-11T03:52:07.290892+0000","last_undegraded":"2024-09-11T03:52:07.290892+0000","last_fullsized":"2024-09-11T03:52:07.290892+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:51:23.036386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.036386+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:50.175869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000617232,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,9],"acting":[10,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.36","version":"913'714","reported_seq":4881,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.129415+0000","last_change":"2024-09-11T03:52:11.129702+0000","last_active":"2024-09-11T03:52:11.129415+0000","last_peered":"2024-09-11T03:52:11.129415+0000","last_clean":"2024-09-11T03:52:11.129415+0000","last_became_active":"2024-09-11T03:50:07.078667+0000","last_became_peered":"2024-09-11T03:50:07.078667+0000","last_unstale":"2024-09-11T03:52:11.129415+0000","last_undegraded":"2024-09-11T03:52:11.129415+0000","last_fullsized":"2024-09-11T03:52:11.129415+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"885'670","last_scrub_stamp":"2024-09-11T03:51:43.529996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:43.529996+0000","objects_scrubbed":15,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:50.431189+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.011297847,"stat_sum":{"num_bytes":32741085,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":41,"num_read_kb":41151,"num_write":11,"num_write_kb":9394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,8],"acting":[1,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.35","version":"274'273","reported_seq":1878,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.128317+0000","last_change":"2024-09-11T03:52:11.128748+0000","last_active":"2024-09-11T03:52:11.128317+0000","last_peered":"2024-09-11T03:52:11.128317+0000","last_clean":"2024-09-11T03:52:11.128317+0000","last_became_active":"2024-09-11T03:50:07.079331+0000","last_became_peered":"2024-09-11T03:50:07.079331+0000","last_unstale":"2024-09-11T03:52:11.128317+0000","last_undegraded":"2024-09-11T03:52:11.128317+0000","last_fullsized":"2024-09-11T03:52:11.128317+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:51:28.526090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.526090+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:56.162118+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0095594470000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.34","version":"905'402","reported_seq":4514,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.134185+0000","last_change":"2024-09-11T03:52:11.134448+0000","last_active":"2024-09-11T03:52:11.134185+0000","last_peered":"2024-09-11T03:52:11.134185+0000","last_clean":"2024-09-11T03:52:11.134185+0000","last_became_active":"2024-09-11T03:50:07.080419+0000","last_became_peered":"2024-09-11T03:50:07.080419+0000","last_unstale":"2024-09-11T03:52:11.134185+0000","last_undegraded":"2024-09-11T03:52:11.134185+0000","last_fullsized":"2024-09-11T03:52:11.134185+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"863'388","last_scrub_stamp":"2024-09-11T03:51:22.429302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.429302+0000","objects_scrubbed":6,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:26.560468+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018017059999999999,"stat_sum":{"num_bytes":13930425,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":37367,"num_write":8,"num_write_kb":1518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.33","version":"274'111","reported_seq":1666,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.135187+0000","last_change":"2024-09-11T03:52:11.135582+0000","last_active":"2024-09-11T03:52:11.135187+0000","last_peered":"2024-09-11T03:52:11.135187+0000","last_clean":"2024-09-11T03:52:11.135187+0000","last_became_active":"2024-09-11T03:50:07.077684+0000","last_became_peered":"2024-09-11T03:50:07.077684+0000","last_unstale":"2024-09-11T03:52:11.135187+0000","last_undegraded":"2024-09-11T03:52:11.135187+0000","last_fullsized":"2024-09-11T03:52:11.135187+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:51:28.001189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.001189+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:38.194986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0067560820000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.32","version":"908'592","reported_seq":4706,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.086267+0000","last_change":"2024-09-11T03:52:06.086557+0000","last_active":"2024-09-11T03:52:06.086267+0000","last_peered":"2024-09-11T03:52:06.086267+0000","last_clean":"2024-09-11T03:52:06.086267+0000","last_became_active":"2024-09-11T03:51:20.067057+0000","last_became_peered":"2024-09-11T03:51:20.067057+0000","last_unstale":"2024-09-11T03:52:06.086267+0000","last_undegraded":"2024-09-11T03:52:06.086267+0000","last_fullsized":"2024-09-11T03:52:06.086267+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"885'574","last_scrub_stamp":"2024-09-11T03:51:44.094173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:44.094173+0000","objects_scrubbed":3,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:53.589988+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0058409559999999996,"stat_sum":{"num_bytes":8252963,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":10412,"num_write":5,"num_write_kb":1096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":2772362,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.31","version":"274'107","reported_seq":2156,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-09-11T03:52:10.190577+0000","last_change":"2024-09-11T03:52:08.099173+0000","last_active":"2024-09-11T03:52:10.190577+0000","last_peered":"2024-09-11T03:52:10.190577+0000","last_clean":"2024-09-11T03:52:10.190577+0000","last_became_active":"2024-09-11T03:51:20.060052+0000","last_became_peered":"2024-09-11T03:51:20.060052+0000","last_unstale":"2024-09-11T03:52:10.190577+0000","last_undegraded":"2024-09-11T03:52:10.190577+0000","last_fullsized":"2024-09-11T03:52:10.190577+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:51:26.087112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:26.087112+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:49.873363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00068544899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,7],"acting":[0,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.30","version":"274'67","reported_seq":614,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.290817+0000","last_change":"2024-09-11T03:52:06.088316+0000","last_active":"2024-09-11T03:52:07.290817+0000","last_peered":"2024-09-11T03:52:07.290817+0000","last_clean":"2024-09-11T03:52:07.290817+0000","last_became_active":"2024-09-11T03:51:17.065156+0000","last_became_peered":"2024-09-11T03:51:17.065156+0000","last_unstale":"2024-09-11T03:52:07.290817+0000","last_undegraded":"2024-09-11T03:52:07.290817+0000","last_fullsized":"2024-09-11T03:52:07.290817+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:51:21.989674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.989674+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:49.152610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001097029,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.2f","version":"557'465","reported_seq":3465,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.102489+0000","last_change":"2024-09-11T03:52:08.103058+0000","last_active":"2024-09-11T03:52:08.102489+0000","last_peered":"2024-09-11T03:52:08.102489+0000","last_clean":"2024-09-11T03:52:08.102489+0000","last_became_active":"2024-09-11T03:50:07.077840+0000","last_became_peered":"2024-09-11T03:50:07.077840+0000","last_unstale":"2024-09-11T03:52:08.102489+0000","last_undegraded":"2024-09-11T03:52:08.102489+0000","last_fullsized":"2024-09-11T03:52:08.102489+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:51:47.522706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:47.522706+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:56.188850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001185511,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.2e","version":"911'1215","reported_seq":5236,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-09-11T03:52:10.190399+0000","last_change":"2024-09-11T03:52:08.106388+0000","last_active":"2024-09-11T03:52:10.190399+0000","last_peered":"2024-09-11T03:52:10.190399+0000","last_clean":"2024-09-11T03:52:10.190399+0000","last_became_active":"2024-09-11T03:51:31.069593+0000","last_became_peered":"2024-09-11T03:51:31.069593+0000","last_unstale":"2024-09-11T03:52:10.190399+0000","last_undegraded":"2024-09-11T03:52:10.190399+0000","last_fullsized":"2024-09-11T03:52:10.190399+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":6,"last_scrub":"868'1145","last_scrub_stamp":"2024-09-11T03:51:27.034758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:27.034758+0000","objects_scrubbed":11,"log_size":1215,"log_dups_size":0,"ondisk_log_size":1215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:43.074073+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.007763697,"stat_sum":{"num_bytes":32901725,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":35,"num_read_kb":34190,"num_write":26,"num_write_kb":11078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":24991924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,8],"acting":[0,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.2d","version":"908'1318","reported_seq":5390,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.063890+0000","last_change":"2024-09-11T03:52:06.092502+0000","last_active":"2024-09-11T03:52:07.063890+0000","last_peered":"2024-09-11T03:52:07.063890+0000","last_clean":"2024-09-11T03:52:07.063890+0000","last_became_active":"2024-09-11T03:49:29.163201+0000","last_became_peered":"2024-09-11T03:49:29.163201+0000","last_unstale":"2024-09-11T03:52:07.063890+0000","last_undegraded":"2024-09-11T03:52:07.063890+0000","last_fullsized":"2024-09-11T03:52:07.063890+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"837'1159","last_scrub_stamp":"2024-09-11T03:50:55.762457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:55.762457+0000","objects_scrubbed":20,"log_size":1318,"log_dups_size":0,"ondisk_log_size":1318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:16.962131+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0093135809999999996,"stat_sum":{"num_bytes":42858974,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":131,"num_read_kb":173218,"num_write":39,"num_write_kb":21285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.2c","version":"913'961","reported_seq":5073,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.132365+0000","last_change":"2024-09-11T03:52:11.132576+0000","last_active":"2024-09-11T03:52:11.132365+0000","last_peered":"2024-09-11T03:52:11.132365+0000","last_clean":"2024-09-11T03:52:11.132365+0000","last_became_active":"2024-09-11T03:49:29.154872+0000","last_became_peered":"2024-09-11T03:49:29.154872+0000","last_unstale":"2024-09-11T03:52:11.132365+0000","last_undegraded":"2024-09-11T03:52:11.132365+0000","last_fullsized":"2024-09-11T03:52:11.132365+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"908'959","last_scrub_stamp":"2024-09-11T03:52:08.323831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:08.323831+0000","objects_scrubbed":9,"log_size":961,"log_dups_size":0,"ondisk_log_size":961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:31.513669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0067474120000000004,"stat_sum":{"num_bytes":22924800,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":47013,"num_write":50,"num_write_kb":17751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3],"acting":[8,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.2b","version":"568'646","reported_seq":3873,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.105335+0000","last_change":"2024-09-11T03:52:08.105774+0000","last_active":"2024-09-11T03:52:08.105335+0000","last_peered":"2024-09-11T03:52:08.105335+0000","last_clean":"2024-09-11T03:52:08.105335+0000","last_became_active":"2024-09-11T03:51:20.062485+0000","last_became_peered":"2024-09-11T03:51:20.062485+0000","last_unstale":"2024-09-11T03:52:08.105335+0000","last_undegraded":"2024-09-11T03:52:08.105335+0000","last_fullsized":"2024-09-11T03:52:08.105335+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:51:52.556581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.556581+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:54.684702+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0028393049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.2a","version":"727'348","reported_seq":4355,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.128282+0000","last_change":"2024-09-11T03:52:11.129044+0000","last_active":"2024-09-11T03:52:11.128282+0000","last_peered":"2024-09-11T03:52:11.128282+0000","last_clean":"2024-09-11T03:52:11.128282+0000","last_became_active":"2024-09-11T03:49:29.160595+0000","last_became_peered":"2024-09-11T03:49:29.160595+0000","last_unstale":"2024-09-11T03:52:11.128282+0000","last_undegraded":"2024-09-11T03:52:11.128282+0000","last_fullsized":"2024-09-11T03:52:11.128282+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:51:05.940576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:05.940576+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001298804,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,5],"acting":[7,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.29","version":"905'1341","reported_seq":5529,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.062943+0000","last_change":"2024-09-11T03:52:06.083882+0000","last_active":"2024-09-11T03:52:07.062943+0000","last_peered":"2024-09-11T03:52:07.062943+0000","last_clean":"2024-09-11T03:52:07.062943+0000","last_became_active":"2024-09-11T03:49:29.157194+0000","last_became_peered":"2024-09-11T03:49:29.157194+0000","last_unstale":"2024-09-11T03:52:07.062943+0000","last_undegraded":"2024-09-11T03:52:07.062943+0000","last_fullsized":"2024-09-11T03:52:07.062943+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"838'1272","last_scrub_stamp":"2024-09-11T03:50:57.773854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:57.773854+0000","objects_scrubbed":14,"log_size":1341,"log_dups_size":0,"ondisk_log_size":1341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:08.135460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024613119999999998,"stat_sum":{"num_bytes":34337514,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":53,"num_read_kb":69444,"num_write":41,"num_write_kb":13701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8],"acting":[6,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.28","version":"905'477","reported_seq":4594,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.119354+0000","last_change":"2024-09-11T03:52:11.119785+0000","last_active":"2024-09-11T03:52:11.119354+0000","last_peered":"2024-09-11T03:52:11.119354+0000","last_clean":"2024-09-11T03:52:11.119354+0000","last_became_active":"2024-09-11T03:49:29.154884+0000","last_became_peered":"2024-09-11T03:49:29.154884+0000","last_unstale":"2024-09-11T03:52:11.119354+0000","last_undegraded":"2024-09-11T03:52:11.119354+0000","last_fullsized":"2024-09-11T03:52:11.119354+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"905'477","last_scrub_stamp":"2024-09-11T03:52:07.670614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:07.670614+0000","objects_scrubbed":6,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:23.895767+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001559967,"stat_sum":{"num_bytes":15310649,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":14,"num_read_kb":11529,"num_write":23,"num_write_kb":4588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.27","version":"913'1259","reported_seq":5591,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.140296+0000","last_change":"2024-09-11T03:52:11.140462+0000","last_active":"2024-09-11T03:52:11.140296+0000","last_peered":"2024-09-11T03:52:11.140296+0000","last_clean":"2024-09-11T03:52:11.140296+0000","last_became_active":"2024-09-11T03:51:20.062110+0000","last_became_peered":"2024-09-11T03:51:20.062110+0000","last_unstale":"2024-09-11T03:52:11.140296+0000","last_undegraded":"2024-09-11T03:52:11.140296+0000","last_fullsized":"2024-09-11T03:52:11.140296+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"909'1255","last_scrub_stamp":"2024-09-11T03:52:10.054644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:10.054644+0000","objects_scrubbed":13,"log_size":1259,"log_dups_size":0,"ondisk_log_size":1259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:25.874071+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.010872204999999999,"stat_sum":{"num_bytes":32775836,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":55,"num_read_kb":59674,"num_write":46,"num_write_kb":23843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":14737969,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.26","version":"269'171","reported_seq":4253,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.129534+0000","last_change":"2024-09-11T03:52:11.129874+0000","last_active":"2024-09-11T03:52:11.129534+0000","last_peered":"2024-09-11T03:52:11.129534+0000","last_clean":"2024-09-11T03:52:11.129534+0000","last_became_active":"2024-09-11T03:51:20.061265+0000","last_became_peered":"2024-09-11T03:51:20.061265+0000","last_unstale":"2024-09-11T03:52:11.129534+0000","last_undegraded":"2024-09-11T03:52:11.129534+0000","last_fullsized":"2024-09-11T03:52:11.129534+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:52:08.043740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:08.043740+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:32.610546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001653716,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,5],"acting":[7,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.25","version":"913'1560","reported_seq":5842,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.139112+0000","last_change":"2024-09-11T03:52:11.139306+0000","last_active":"2024-09-11T03:52:11.139112+0000","last_peered":"2024-09-11T03:52:11.139112+0000","last_clean":"2024-09-11T03:52:11.139112+0000","last_became_active":"2024-09-11T03:49:29.159733+0000","last_became_peered":"2024-09-11T03:49:29.159733+0000","last_unstale":"2024-09-11T03:52:11.139112+0000","last_undegraded":"2024-09-11T03:52:11.139112+0000","last_fullsized":"2024-09-11T03:52:11.139112+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"840'1347","last_scrub_stamp":"2024-09-11T03:50:59.564459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:59.564459+0000","objects_scrubbed":21,"log_size":1560,"log_dups_size":0,"ondisk_log_size":1560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:22.712551+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.012833772,"stat_sum":{"num_bytes":44640411,"num_objects":24,"num_object_clones":21,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":77,"num_read_kb":67687,"num_write":75,"num_write_kb":21896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,6],"acting":[8,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.24","version":"908'730","reported_seq":4836,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.063675+0000","last_change":"2024-09-11T03:52:06.089076+0000","last_active":"2024-09-11T03:52:07.063675+0000","last_peered":"2024-09-11T03:52:07.063675+0000","last_clean":"2024-09-11T03:52:07.063675+0000","last_became_active":"2024-09-11T03:49:29.155740+0000","last_became_peered":"2024-09-11T03:49:29.155740+0000","last_unstale":"2024-09-11T03:52:07.063675+0000","last_undegraded":"2024-09-11T03:52:07.063675+0000","last_fullsized":"2024-09-11T03:52:07.063675+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"838'674","last_scrub_stamp":"2024-09-11T03:50:56.770184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:56.770184+0000","objects_scrubbed":7,"log_size":730,"log_dups_size":0,"ondisk_log_size":730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0074445780000000003,"stat_sum":{"num_bytes":15739462,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":14,"num_write_kb":11029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,7],"acting":[6,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.23","version":"913'560","reported_seq":4656,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.127186+0000","last_change":"2024-09-11T03:52:11.127710+0000","last_active":"2024-09-11T03:52:11.127186+0000","last_peered":"2024-09-11T03:52:11.127186+0000","last_clean":"2024-09-11T03:52:11.127186+0000","last_became_active":"2024-09-11T03:46:32.970100+0000","last_became_peered":"2024-09-11T03:46:32.970100+0000","last_unstale":"2024-09-11T03:52:11.127186+0000","last_undegraded":"2024-09-11T03:52:11.127186+0000","last_fullsized":"2024-09-11T03:52:11.127186+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"886'531","last_scrub_stamp":"2024-09-11T03:51:44.527530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:44.527530+0000","objects_scrubbed":5,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:08.991314+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0086479429999999999,"stat_sum":{"num_bytes":11706449,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":57,"num_read_kb":64783,"num_write":39,"num_write_kb":13976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.22","version":"531'216","reported_seq":4270,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.063102+0000","last_change":"2024-09-11T03:52:06.089056+0000","last_active":"2024-09-11T03:52:07.063102+0000","last_peered":"2024-09-11T03:52:07.063102+0000","last_clean":"2024-09-11T03:52:07.063102+0000","last_became_active":"2024-09-11T03:46:32.970633+0000","last_became_peered":"2024-09-11T03:46:32.970633+0000","last_unstale":"2024-09-11T03:52:07.063102+0000","last_undegraded":"2024-09-11T03:52:07.063102+0000","last_fullsized":"2024-09-11T03:52:07.063102+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:50:38.617374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:38.617374+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0070261710000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.21","version":"907'408","reported_seq":4551,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.080509+0000","last_change":"2024-09-11T03:52:06.081819+0000","last_active":"2024-09-11T03:52:06.080509+0000","last_peered":"2024-09-11T03:52:06.080509+0000","last_clean":"2024-09-11T03:52:06.080509+0000","last_became_active":"2024-09-11T03:46:36.024928+0000","last_became_peered":"2024-09-11T03:46:36.024928+0000","last_unstale":"2024-09-11T03:52:06.080509+0000","last_undegraded":"2024-09-11T03:52:06.080509+0000","last_fullsized":"2024-09-11T03:52:06.080509+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":6,"last_scrub":"816'351","last_scrub_stamp":"2024-09-11T03:50:40.225380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:40.225380+0000","objects_scrubbed":7,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00078402099999999998,"stat_sum":{"num_bytes":19044405,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":55,"num_read_kb":56970,"num_write":50,"num_write_kb":21317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,8],"acting":[11,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"2.20","version":"913'358","reported_seq":4606,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.135705+0000","last_change":"2024-09-11T03:52:11.137571+0000","last_active":"2024-09-11T03:52:11.135705+0000","last_peered":"2024-09-11T03:52:11.135705+0000","last_clean":"2024-09-11T03:52:11.135705+0000","last_became_active":"2024-09-11T03:46:32.974375+0000","last_became_peered":"2024-09-11T03:46:32.974375+0000","last_unstale":"2024-09-11T03:52:11.135705+0000","last_undegraded":"2024-09-11T03:52:11.135705+0000","last_fullsized":"2024-09-11T03:52:11.135705+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"895'348","last_scrub_stamp":"2024-09-11T03:51:54.026522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.026522+0000","objects_scrubbed":6,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:54.219176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0090109600000000002,"stat_sum":{"num_bytes":19026910,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":98,"num_read_kb":144174,"num_write":25,"num_write_kb":13274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.1c","version":"806'776","reported_seq":4842,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.291875+0000","last_change":"2024-09-11T03:52:06.090118+0000","last_active":"2024-09-11T03:52:07.291875+0000","last_peered":"2024-09-11T03:52:07.291875+0000","last_clean":"2024-09-11T03:52:07.291875+0000","last_became_active":"2024-09-11T03:50:28.047548+0000","last_became_peered":"2024-09-11T03:50:28.047548+0000","last_unstale":"2024-09-11T03:52:07.291875+0000","last_undegraded":"2024-09-11T03:52:07.291875+0000","last_fullsized":"2024-09-11T03:52:07.291875+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"806'776","last_scrub_stamp":"2024-09-11T03:51:54.156180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.156180+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:10.485153+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00088502999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":885,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-09-11T03:52:10.191041+0000","last_change":"2024-09-11T03:50:55.035249+0000","last_active":"2024-09-11T03:52:10.191041+0000","last_peered":"2024-09-11T03:52:10.191041+0000","last_clean":"2024-09-11T03:52:10.191041+0000","last_became_active":"2024-09-11T03:46:15.974786+0000","last_became_peered":"2024-09-11T03:46:15.974786+0000","last_unstale":"2024-09-11T03:52:10.191041+0000","last_undegraded":"2024-09-11T03:52:10.191041+0000","last_fullsized":"2024-09-11T03:52:10.191041+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:55.035202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:55.035202+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.f","version":"557'465","reported_seq":4585,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.132644+0000","last_change":"2024-09-11T03:52:11.132892+0000","last_active":"2024-09-11T03:52:11.132644+0000","last_peered":"2024-09-11T03:52:11.132644+0000","last_clean":"2024-09-11T03:52:11.132644+0000","last_became_active":"2024-09-11T03:51:20.072191+0000","last_became_peered":"2024-09-11T03:51:20.072191+0000","last_unstale":"2024-09-11T03:52:11.132644+0000","last_undegraded":"2024-09-11T03:52:11.132644+0000","last_fullsized":"2024-09-11T03:52:11.132644+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:51:25.558029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:25.558029+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:32.015171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0062307020000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,11],"acting":[8,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.e","version":"782'1025","reported_seq":5074,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.104323+0000","last_change":"2024-09-11T03:52:08.105244+0000","last_active":"2024-09-11T03:52:08.104323+0000","last_peered":"2024-09-11T03:52:08.104323+0000","last_clean":"2024-09-11T03:52:08.104323+0000","last_became_active":"2024-09-11T03:50:02.051717+0000","last_became_peered":"2024-09-11T03:50:02.051717+0000","last_unstale":"2024-09-11T03:52:08.104323+0000","last_undegraded":"2024-09-11T03:52:08.104323+0000","last_fullsized":"2024-09-11T03:52:08.104323+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":0,"last_scrub":"782'1025","last_scrub_stamp":"2024-09-11T03:51:22.296245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.296245+0000","objects_scrubbed":0,"log_size":1025,"log_dups_size":0,"ondisk_log_size":1025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:46.727935+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0033174609999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":1470,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.128594+0000","last_change":"2024-09-11T03:51:17.066500+0000","last_active":"2024-09-11T03:52:11.128594+0000","last_peered":"2024-09-11T03:52:11.128594+0000","last_clean":"2024-09-11T03:52:11.128594+0000","last_became_active":"2024-09-11T03:51:17.066085+0000","last_became_peered":"2024-09-11T03:51:17.066085+0000","last_unstale":"2024-09-11T03:52:11.128594+0000","last_undegraded":"2024-09-11T03:52:11.128594+0000","last_fullsized":"2024-09-11T03:52:11.128594+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:58.932376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:58.932376+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"913'1064","reported_seq":5271,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.137643+0000","last_change":"2024-09-11T03:52:11.138076+0000","last_active":"2024-09-11T03:52:11.137643+0000","last_peered":"2024-09-11T03:52:11.137643+0000","last_clean":"2024-09-11T03:52:11.137643+0000","last_became_active":"2024-09-11T03:51:20.062633+0000","last_became_peered":"2024-09-11T03:51:20.062633+0000","last_unstale":"2024-09-11T03:52:11.137643+0000","last_undegraded":"2024-09-11T03:52:11.137643+0000","last_fullsized":"2024-09-11T03:52:11.137643+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"811'1014","last_scrub_stamp":"2024-09-11T03:50:37.438736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.438736+0000","objects_scrubbed":5,"log_size":1064,"log_dups_size":0,"ondisk_log_size":1064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0051924199999999997,"stat_sum":{"num_bytes":10904236,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":7027,"num_write":20,"num_write_kb":9249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":9438955,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":1185,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.128270+0000","last_change":"2024-09-11T03:51:17.063139+0000","last_active":"2024-09-11T03:52:11.128270+0000","last_peered":"2024-09-11T03:52:11.128270+0000","last_clean":"2024-09-11T03:52:11.128270+0000","last_became_active":"2024-09-11T03:51:17.061266+0000","last_became_peered":"2024-09-11T03:51:17.061266+0000","last_unstale":"2024-09-11T03:52:11.128270+0000","last_undegraded":"2024-09-11T03:52:11.128270+0000","last_fullsized":"2024-09-11T03:52:11.128270+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:38.843438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:38.843438+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"905'953","reported_seq":5030,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.080951+0000","last_change":"2024-09-11T03:52:06.081306+0000","last_active":"2024-09-11T03:52:06.080951+0000","last_peered":"2024-09-11T03:52:06.080951+0000","last_clean":"2024-09-11T03:52:06.080951+0000","last_became_active":"2024-09-11T03:49:24.115405+0000","last_became_peered":"2024-09-11T03:49:24.115405+0000","last_unstale":"2024-09-11T03:52:06.080951+0000","last_undegraded":"2024-09-11T03:52:06.080951+0000","last_fullsized":"2024-09-11T03:52:06.080951+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"843'880","last_scrub_stamp":"2024-09-11T03:51:01.987099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:01.987099+0000","objects_scrubbed":13,"log_size":953,"log_dups_size":0,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:21.010286+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000873587,"stat_sum":{"num_bytes":22929927,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":36,"num_read_kb":32542,"num_write":20,"num_write_kb":7053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.f","version":"20'92","reported_seq":743,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.064402+0000","last_change":"2024-09-11T03:51:21.860818+0000","last_active":"2024-09-11T03:52:07.064402+0000","last_peered":"2024-09-11T03:52:07.064402+0000","last_clean":"2024-09-11T03:52:07.064402+0000","last_became_active":"2024-09-11T03:51:17.064761+0000","last_became_peered":"2024-09-11T03:51:17.064761+0000","last_unstale":"2024-09-11T03:52:07.064402+0000","last_undegraded":"2024-09-11T03:52:07.064402+0000","last_fullsized":"2024-09-11T03:52:07.064402+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:21.860769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.860769+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:34.914441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":1869,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.128701+0000","last_change":"2024-09-11T03:51:52.035008+0000","last_active":"2024-09-11T03:52:11.128701+0000","last_peered":"2024-09-11T03:52:11.128701+0000","last_clean":"2024-09-11T03:52:11.128701+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:52:11.128701+0000","last_undegraded":"2024-09-11T03:52:11.128701+0000","last_fullsized":"2024-09-11T03:52:11.128701+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:52.034950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.034950+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:03.753329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"845'562","reported_seq":4669,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.138743+0000","last_change":"2024-09-11T03:52:11.139261+0000","last_active":"2024-09-11T03:52:11.138743+0000","last_peered":"2024-09-11T03:52:11.138743+0000","last_clean":"2024-09-11T03:52:11.138743+0000","last_became_active":"2024-09-11T03:51:08.055565+0000","last_became_peered":"2024-09-11T03:51:08.055565+0000","last_unstale":"2024-09-11T03:52:11.138743+0000","last_undegraded":"2024-09-11T03:52:11.138743+0000","last_fullsized":"2024-09-11T03:52:11.138743+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"845'562","last_scrub_stamp":"2024-09-11T03:51:20.441661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.441661+0000","objects_scrubbed":0,"log_size":562,"log_dups_size":0,"ondisk_log_size":562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:45.056091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005720022,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":1482,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.119911+0000","last_change":"2024-09-11T03:51:27.564638+0000","last_active":"2024-09-11T03:52:11.119911+0000","last_peered":"2024-09-11T03:52:11.119911+0000","last_clean":"2024-09-11T03:52:11.119911+0000","last_became_active":"2024-09-11T03:51:17.059234+0000","last_became_peered":"2024-09-11T03:51:17.059234+0000","last_unstale":"2024-09-11T03:52:11.119911+0000","last_undegraded":"2024-09-11T03:52:11.119911+0000","last_fullsized":"2024-09-11T03:52:11.119911+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:27.564586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:27.564586+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:54.532532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"563'192","reported_seq":3133,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.291069+0000","last_change":"2024-09-11T03:52:06.088864+0000","last_active":"2024-09-11T03:52:07.291069+0000","last_peered":"2024-09-11T03:52:07.291069+0000","last_clean":"2024-09-11T03:52:07.291069+0000","last_became_active":"2024-09-11T03:50:31.044399+0000","last_became_peered":"2024-09-11T03:50:31.044399+0000","last_unstale":"2024-09-11T03:52:07.291069+0000","last_undegraded":"2024-09-11T03:52:07.291069+0000","last_fullsized":"2024-09-11T03:52:07.291069+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:50:36.771360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.771360+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001119697,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":1438,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-09-11T03:52:10.190405+0000","last_change":"2024-09-11T03:51:17.061730+0000","last_active":"2024-09-11T03:52:10.190405+0000","last_peered":"2024-09-11T03:52:10.190405+0000","last_clean":"2024-09-11T03:52:10.190405+0000","last_became_active":"2024-09-11T03:51:17.061243+0000","last_became_peered":"2024-09-11T03:51:17.061243+0000","last_unstale":"2024-09-11T03:52:10.190405+0000","last_undegraded":"2024-09-11T03:52:10.190405+0000","last_fullsized":"2024-09-11T03:52:10.190405+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:41.975660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:41.975660+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:07.081232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"567'235","reported_seq":4305,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-09-11T03:52:10.190956+0000","last_change":"2024-09-11T03:52:08.100655+0000","last_active":"2024-09-11T03:52:10.190956+0000","last_peered":"2024-09-11T03:52:10.190956+0000","last_clean":"2024-09-11T03:52:10.190956+0000","last_became_active":"2024-09-11T03:50:28.039300+0000","last_became_peered":"2024-09-11T03:50:28.039300+0000","last_unstale":"2024-09-11T03:52:10.190956+0000","last_undegraded":"2024-09-11T03:52:10.190956+0000","last_fullsized":"2024-09-11T03:52:10.190956+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:51:50.027291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:50.027291+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:55.273378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00184392,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":1036,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.291277+0000","last_change":"2024-09-11T03:51:29.996670+0000","last_active":"2024-09-11T03:52:07.291277+0000","last_peered":"2024-09-11T03:52:07.291277+0000","last_clean":"2024-09-11T03:52:07.291277+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:52:07.291277+0000","last_undegraded":"2024-09-11T03:52:07.291277+0000","last_fullsized":"2024-09-11T03:52:07.291277+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:29.996598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:29.996598+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:54.485931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"843'319","reported_seq":4527,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.133536+0000","last_change":"2024-09-11T03:52:11.134064+0000","last_active":"2024-09-11T03:52:11.133536+0000","last_peered":"2024-09-11T03:52:11.133536+0000","last_clean":"2024-09-11T03:52:11.133536+0000","last_became_active":"2024-09-11T03:51:08.055103+0000","last_became_peered":"2024-09-11T03:51:08.055103+0000","last_unstale":"2024-09-11T03:52:11.133536+0000","last_undegraded":"2024-09-11T03:52:11.133536+0000","last_fullsized":"2024-09-11T03:52:11.133536+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"843'319","last_scrub_stamp":"2024-09-11T03:51:19.453435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.453435+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:43.906400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017565599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":720,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.064176+0000","last_change":"2024-09-11T03:49:37.637065+0000","last_active":"2024-09-11T03:52:07.064176+0000","last_peered":"2024-09-11T03:52:07.064176+0000","last_clean":"2024-09-11T03:52:07.064176+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:52:07.064176+0000","last_undegraded":"2024-09-11T03:52:07.064176+0000","last_fullsized":"2024-09-11T03:52:07.064176+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:37.637018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:37.637018+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:07.712022+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"847'690","reported_seq":4858,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.127864+0000","last_change":"2024-09-11T03:52:11.128252+0000","last_active":"2024-09-11T03:52:11.127864+0000","last_peered":"2024-09-11T03:52:11.127864+0000","last_clean":"2024-09-11T03:52:11.127864+0000","last_became_active":"2024-09-11T03:51:08.058954+0000","last_became_peered":"2024-09-11T03:51:08.058954+0000","last_unstale":"2024-09-11T03:52:11.127864+0000","last_undegraded":"2024-09-11T03:52:11.127864+0000","last_fullsized":"2024-09-11T03:52:11.127864+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"847'690","last_scrub_stamp":"2024-09-11T03:51:24.473723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:24.473723+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:29.676966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091162789999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":1047,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.292158+0000","last_change":"2024-09-11T03:51:53.155265+0000","last_active":"2024-09-11T03:52:07.292158+0000","last_peered":"2024-09-11T03:52:07.292158+0000","last_clean":"2024-09-11T03:52:07.292158+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:52:07.292158+0000","last_undegraded":"2024-09-11T03:52:07.292158+0000","last_fullsized":"2024-09-11T03:52:07.292158+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:53.155207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.155207+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:07.708610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"913'1261","reported_seq":5488,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.134262+0000","last_change":"2024-09-11T03:52:11.134429+0000","last_active":"2024-09-11T03:52:11.134262+0000","last_peered":"2024-09-11T03:52:11.134262+0000","last_clean":"2024-09-11T03:52:11.134262+0000","last_became_active":"2024-09-11T03:51:20.063427+0000","last_became_peered":"2024-09-11T03:51:20.063427+0000","last_unstale":"2024-09-11T03:52:11.134262+0000","last_undegraded":"2024-09-11T03:52:11.134262+0000","last_fullsized":"2024-09-11T03:52:11.134262+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"862'1215","last_scrub_stamp":"2024-09-11T03:51:21.601542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.601542+0000","objects_scrubbed":5,"log_size":1261,"log_dups_size":0,"ondisk_log_size":1261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:42.713661+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0070455880000000002,"stat_sum":{"num_bytes":13495545,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":2299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":1030,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.292289+0000","last_change":"2024-09-11T03:49:36.654074+0000","last_active":"2024-09-11T03:52:07.292289+0000","last_peered":"2024-09-11T03:52:07.292289+0000","last_clean":"2024-09-11T03:52:07.292289+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:52:07.292289+0000","last_undegraded":"2024-09-11T03:52:07.292289+0000","last_fullsized":"2024-09-11T03:52:07.292289+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:36.654025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:36.654025+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:07.849922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":4261,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.123966+0000","last_change":"2024-09-11T03:52:11.124167+0000","last_active":"2024-09-11T03:52:11.123966+0000","last_peered":"2024-09-11T03:52:11.123966+0000","last_clean":"2024-09-11T03:52:11.123966+0000","last_became_active":"2024-09-11T03:51:08.055529+0000","last_became_peered":"2024-09-11T03:51:08.055529+0000","last_unstale":"2024-09-11T03:52:11.123966+0000","last_undegraded":"2024-09-11T03:52:11.123966+0000","last_fullsized":"2024-09-11T03:52:11.123966+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:51:22.407323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.407323+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:31.838102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056933119999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":1636,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.080279+0000","last_change":"2024-09-11T03:51:31.102151+0000","last_active":"2024-09-11T03:52:06.080279+0000","last_peered":"2024-09-11T03:52:06.080279+0000","last_clean":"2024-09-11T03:52:06.080279+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:52:06.080279+0000","last_undegraded":"2024-09-11T03:52:06.080279+0000","last_fullsized":"2024-09-11T03:52:06.080279+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:31.102098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:31.102098+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:49.931850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"908'1332","reported_seq":5600,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.064048+0000","last_change":"2024-09-11T03:52:06.093775+0000","last_active":"2024-09-11T03:52:07.064048+0000","last_peered":"2024-09-11T03:52:07.064048+0000","last_clean":"2024-09-11T03:52:07.064048+0000","last_became_active":"2024-09-11T03:51:08.057350+0000","last_became_peered":"2024-09-11T03:51:08.057350+0000","last_unstale":"2024-09-11T03:52:07.064048+0000","last_undegraded":"2024-09-11T03:52:07.064048+0000","last_fullsized":"2024-09-11T03:52:07.064048+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'1274","last_scrub_stamp":"2024-09-11T03:51:19.872739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.872739+0000","objects_scrubbed":12,"log_size":1332,"log_dups_size":0,"ondisk_log_size":1332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:44.400628+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.010632146,"stat_sum":{"num_bytes":38444164,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":3103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":1439,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.080090+0000","last_change":"2024-09-11T03:51:47.050276+0000","last_active":"2024-09-11T03:52:06.080090+0000","last_peered":"2024-09-11T03:52:06.080090+0000","last_clean":"2024-09-11T03:52:06.080090+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:52:06.080090+0000","last_undegraded":"2024-09-11T03:52:06.080090+0000","last_fullsized":"2024-09-11T03:52:06.080090+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:47.050227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:47.050227+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:47.461383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"744'374","reported_seq":4542,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.129601+0000","last_change":"2024-09-11T03:52:11.130022+0000","last_active":"2024-09-11T03:52:11.129601+0000","last_peered":"2024-09-11T03:52:11.129601+0000","last_clean":"2024-09-11T03:52:11.129601+0000","last_became_active":"2024-09-11T03:51:08.057750+0000","last_became_peered":"2024-09-11T03:51:08.057750+0000","last_unstale":"2024-09-11T03:52:11.129601+0000","last_undegraded":"2024-09-11T03:52:11.129601+0000","last_fullsized":"2024-09-11T03:52:11.129601+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'374","last_scrub_stamp":"2024-09-11T03:51:20.010112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.010112+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:31.663971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015565690000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":1795,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.102558+0000","last_change":"2024-09-11T03:51:17.061700+0000","last_active":"2024-09-11T03:52:08.102558+0000","last_peered":"2024-09-11T03:52:08.102558+0000","last_clean":"2024-09-11T03:52:08.102558+0000","last_became_active":"2024-09-11T03:51:17.061257+0000","last_became_peered":"2024-09-11T03:51:17.061257+0000","last_unstale":"2024-09-11T03:52:08.102558+0000","last_undegraded":"2024-09-11T03:52:08.102558+0000","last_fullsized":"2024-09-11T03:52:08.102558+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:18.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.933952+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:39.424252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"846'1373","reported_seq":5597,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.118878+0000","last_change":"2024-09-11T03:52:11.119319+0000","last_active":"2024-09-11T03:52:11.118878+0000","last_peered":"2024-09-11T03:52:11.118878+0000","last_clean":"2024-09-11T03:52:11.118878+0000","last_became_active":"2024-09-11T03:51:31.065505+0000","last_became_peered":"2024-09-11T03:51:31.065505+0000","last_unstale":"2024-09-11T03:52:11.118878+0000","last_undegraded":"2024-09-11T03:52:11.118878+0000","last_fullsized":"2024-09-11T03:52:11.118878+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"846'1373","last_scrub_stamp":"2024-09-11T03:51:19.328459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.328459+0000","objects_scrubbed":0,"log_size":1373,"log_dups_size":0,"ondisk_log_size":1373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:24.619338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001083012,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":1864,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.127496+0000","last_change":"2024-09-11T03:51:04.911507+0000","last_active":"2024-09-11T03:52:11.127496+0000","last_peered":"2024-09-11T03:52:11.127496+0000","last_clean":"2024-09-11T03:52:11.127496+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:52:11.127496+0000","last_undegraded":"2024-09-11T03:52:11.127496+0000","last_fullsized":"2024-09-11T03:52:11.127496+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:04.911446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:04.911446+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:33.168597+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"905'421","reported_seq":4574,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.086631+0000","last_change":"2024-09-11T03:52:06.086889+0000","last_active":"2024-09-11T03:52:06.086631+0000","last_peered":"2024-09-11T03:52:06.086631+0000","last_clean":"2024-09-11T03:52:06.086631+0000","last_became_active":"2024-09-11T03:51:08.053749+0000","last_became_peered":"2024-09-11T03:51:08.053749+0000","last_unstale":"2024-09-11T03:52:06.086631+0000","last_undegraded":"2024-09-11T03:52:06.086631+0000","last_fullsized":"2024-09-11T03:52:06.086631+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'392","last_scrub_stamp":"2024-09-11T03:51:19.986753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.986753+0000","objects_scrubbed":5,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:29.123663+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0056447049999999999,"stat_sum":{"num_bytes":10459472,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":23665,"num_write":9,"num_write_kb":2863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":1523,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.132135+0000","last_change":"2024-09-11T03:51:55.165620+0000","last_active":"2024-09-11T03:52:11.132135+0000","last_peered":"2024-09-11T03:52:11.132135+0000","last_clean":"2024-09-11T03:52:11.132135+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:52:11.132135+0000","last_undegraded":"2024-09-11T03:52:11.132135+0000","last_fullsized":"2024-09-11T03:52:11.132135+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:55.165539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:55.165539+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:11.582309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"568'646","reported_seq":4811,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.129639+0000","last_change":"2024-09-11T03:52:11.130075+0000","last_active":"2024-09-11T03:52:11.129639+0000","last_peered":"2024-09-11T03:52:11.129639+0000","last_clean":"2024-09-11T03:52:11.129639+0000","last_became_active":"2024-09-11T03:51:08.054955+0000","last_became_peered":"2024-09-11T03:51:08.054955+0000","last_unstale":"2024-09-11T03:52:11.129639+0000","last_undegraded":"2024-09-11T03:52:11.129639+0000","last_fullsized":"2024-09-11T03:52:11.129639+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:51:20.608388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.608388+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:30.455056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041995799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,0],"acting":[8,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":1465,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.132117+0000","last_change":"2024-09-11T03:49:00.710015+0000","last_active":"2024-09-11T03:52:11.132117+0000","last_peered":"2024-09-11T03:52:11.132117+0000","last_clean":"2024-09-11T03:52:11.132117+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:52:11.132117+0000","last_undegraded":"2024-09-11T03:52:11.132117+0000","last_fullsized":"2024-09-11T03:52:11.132117+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:00.709966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.709966+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:33.296056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":4165,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.105754+0000","last_change":"2024-09-11T03:52:08.106021+0000","last_active":"2024-09-11T03:52:08.105754+0000","last_peered":"2024-09-11T03:52:08.105754+0000","last_clean":"2024-09-11T03:52:08.105754+0000","last_became_active":"2024-09-11T03:51:20.063987+0000","last_became_peered":"2024-09-11T03:51:20.063987+0000","last_unstale":"2024-09-11T03:52:08.105754+0000","last_undegraded":"2024-09-11T03:52:08.105754+0000","last_fullsized":"2024-09-11T03:52:08.105754+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:51:25.248547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:25.248547+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:35.833079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0026932129999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,7],"acting":[9,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":1465,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-09-11T03:52:10.190870+0000","last_change":"2024-09-11T03:52:10.061483+0000","last_active":"2024-09-11T03:52:10.190870+0000","last_peered":"2024-09-11T03:52:10.190870+0000","last_clean":"2024-09-11T03:52:10.190870+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:52:10.190870+0000","last_undegraded":"2024-09-11T03:52:10.190870+0000","last_fullsized":"2024-09-11T03:52:10.190870+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:10.061435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:10.061435+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:35.336719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"905'590","reported_seq":4680,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.291820+0000","last_change":"2024-09-11T03:52:06.090543+0000","last_active":"2024-09-11T03:52:07.291820+0000","last_peered":"2024-09-11T03:52:07.291820+0000","last_clean":"2024-09-11T03:52:07.291820+0000","last_became_active":"2024-09-11T03:50:02.053513+0000","last_became_peered":"2024-09-11T03:50:02.053513+0000","last_unstale":"2024-09-11T03:52:07.291820+0000","last_undegraded":"2024-09-11T03:52:07.291820+0000","last_fullsized":"2024-09-11T03:52:07.291820+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"838'575","last_scrub_stamp":"2024-09-11T03:51:04.839779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:04.839779+0000","objects_scrubbed":6,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:30.684838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00093358099999999995,"stat_sum":{"num_bytes":5479610,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":2803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":1454,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.132148+0000","last_change":"2024-09-11T03:51:54.138565+0000","last_active":"2024-09-11T03:52:11.132148+0000","last_peered":"2024-09-11T03:52:11.132148+0000","last_clean":"2024-09-11T03:52:11.132148+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:52:11.132148+0000","last_undegraded":"2024-09-11T03:52:11.132148+0000","last_fullsized":"2024-09-11T03:52:11.132148+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:54.138504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.138504+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:20.328413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":3229,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.063341+0000","last_change":"2024-09-11T03:52:06.088207+0000","last_active":"2024-09-11T03:52:07.063341+0000","last_peered":"2024-09-11T03:52:07.063341+0000","last_clean":"2024-09-11T03:52:07.063341+0000","last_became_active":"2024-09-11T03:51:20.064079+0000","last_became_peered":"2024-09-11T03:51:20.064079+0000","last_unstale":"2024-09-11T03:52:07.063341+0000","last_undegraded":"2024-09-11T03:52:07.063341+0000","last_fullsized":"2024-09-11T03:52:07.063341+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:51:47.080811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:47.080811+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:56.257821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0063804980000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,9],"acting":[6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.13","version":"20'92","reported_seq":1559,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.119659+0000","last_change":"2024-09-11T03:51:52.573394+0000","last_active":"2024-09-11T03:52:11.119659+0000","last_peered":"2024-09-11T03:52:11.119659+0000","last_clean":"2024-09-11T03:52:11.119659+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:52:11.119659+0000","last_undegraded":"2024-09-11T03:52:11.119659+0000","last_fullsized":"2024-09-11T03:52:11.119659+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:52.573348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.573348+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:55.111010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":3191,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.126666+0000","last_change":"2024-09-11T03:52:11.126917+0000","last_active":"2024-09-11T03:52:11.126666+0000","last_peered":"2024-09-11T03:52:11.126666+0000","last_clean":"2024-09-11T03:52:11.126666+0000","last_became_active":"2024-09-11T03:50:02.057196+0000","last_became_peered":"2024-09-11T03:50:02.057196+0000","last_unstale":"2024-09-11T03:52:11.126666+0000","last_undegraded":"2024-09-11T03:52:11.126666+0000","last_fullsized":"2024-09-11T03:52:11.126666+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:51:24.589460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:24.589460+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:38.362395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001876115,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":1040,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.128070+0000","last_change":"2024-09-11T03:51:28.960262+0000","last_active":"2024-09-11T03:52:11.128070+0000","last_peered":"2024-09-11T03:52:11.128070+0000","last_clean":"2024-09-11T03:52:11.128070+0000","last_became_active":"2024-09-11T03:46:36.026459+0000","last_became_peered":"2024-09-11T03:46:36.026459+0000","last_unstale":"2024-09-11T03:52:11.128070+0000","last_undegraded":"2024-09-11T03:52:11.128070+0000","last_fullsized":"2024-09-11T03:52:11.128070+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:28.960214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.960214+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:51.817417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":3721,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.064499+0000","last_change":"2024-09-11T03:52:06.093141+0000","last_active":"2024-09-11T03:52:07.064499+0000","last_peered":"2024-09-11T03:52:07.064499+0000","last_clean":"2024-09-11T03:52:07.064499+0000","last_became_active":"2024-09-11T03:50:02.052278+0000","last_became_peered":"2024-09-11T03:50:02.052278+0000","last_unstale":"2024-09-11T03:52:07.064499+0000","last_undegraded":"2024-09-11T03:52:07.064499+0000","last_fullsized":"2024-09-11T03:52:07.064499+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:51:28.919603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.919603+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:54.621565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0094114690000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":1426,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.128494+0000","last_change":"2024-09-11T03:50:19.741484+0000","last_active":"2024-09-11T03:52:11.128494+0000","last_peered":"2024-09-11T03:52:11.128494+0000","last_clean":"2024-09-11T03:52:11.128494+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:52:11.128494+0000","last_undegraded":"2024-09-11T03:52:11.128494+0000","last_fullsized":"2024-09-11T03:52:11.128494+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:19.741428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:19.741428+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:37.978745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"782'561","reported_seq":4714,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.138340+0000","last_change":"2024-09-11T03:52:11.138634+0000","last_active":"2024-09-11T03:52:11.138340+0000","last_peered":"2024-09-11T03:52:11.138340+0000","last_clean":"2024-09-11T03:52:11.138340+0000","last_became_active":"2024-09-11T03:50:02.054667+0000","last_became_peered":"2024-09-11T03:50:02.054667+0000","last_unstale":"2024-09-11T03:52:11.138340+0000","last_undegraded":"2024-09-11T03:52:11.138340+0000","last_fullsized":"2024-09-11T03:52:11.138340+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"782'561","last_scrub_stamp":"2024-09-11T03:51:24.484944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:24.484944+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:35.492160+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0055160979999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"18a","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":894,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.081550+0000","last_change":"2024-09-11T03:50:37.230454+0000","last_active":"2024-09-11T03:52:06.081550+0000","last_peered":"2024-09-11T03:52:06.081550+0000","last_clean":"2024-09-11T03:52:06.081550+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:52:06.081550+0000","last_undegraded":"2024-09-11T03:52:06.081550+0000","last_fullsized":"2024-09-11T03:52:06.081550+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:37.230388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.230388+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":4380,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.101891+0000","last_change":"2024-09-11T03:52:08.102245+0000","last_active":"2024-09-11T03:52:08.101891+0000","last_peered":"2024-09-11T03:52:08.101891+0000","last_clean":"2024-09-11T03:52:08.101891+0000","last_became_active":"2024-09-11T03:50:02.051850+0000","last_became_peered":"2024-09-11T03:50:02.051850+0000","last_unstale":"2024-09-11T03:52:08.101891+0000","last_undegraded":"2024-09-11T03:52:08.101891+0000","last_fullsized":"2024-09-11T03:52:08.101891+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:51:20.293380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.293380+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:24.000773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00081108199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":1455,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.126543+0000","last_change":"2024-09-11T03:52:09.331476+0000","last_active":"2024-09-11T03:52:11.126543+0000","last_peered":"2024-09-11T03:52:11.126543+0000","last_clean":"2024-09-11T03:52:11.126543+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:52:11.126543+0000","last_undegraded":"2024-09-11T03:52:11.126543+0000","last_fullsized":"2024-09-11T03:52:11.126543+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:09.331402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:09.331402+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:12.316710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"778'353","reported_seq":4355,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.063862+0000","last_change":"2024-09-11T03:52:06.089481+0000","last_active":"2024-09-11T03:52:07.063862+0000","last_peered":"2024-09-11T03:52:07.063862+0000","last_clean":"2024-09-11T03:52:07.063862+0000","last_became_active":"2024-09-11T03:50:02.054211+0000","last_became_peered":"2024-09-11T03:50:02.054211+0000","last_unstale":"2024-09-11T03:52:07.063862+0000","last_undegraded":"2024-09-11T03:52:07.063862+0000","last_fullsized":"2024-09-11T03:52:07.063862+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"778'353","last_scrub_stamp":"2024-09-11T03:51:23.900161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.900161+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:39.968972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0068619650000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.18","version":"20'92","reported_seq":1565,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.132025+0000","last_change":"2024-09-11T03:51:27.437144+0000","last_active":"2024-09-11T03:52:11.132025+0000","last_peered":"2024-09-11T03:52:11.132025+0000","last_clean":"2024-09-11T03:52:11.132025+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:52:11.132025+0000","last_undegraded":"2024-09-11T03:52:11.132025+0000","last_fullsized":"2024-09-11T03:52:11.132025+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:27.437090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:27.437090+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:35.495938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"908'928","reported_seq":5186,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.103525+0000","last_change":"2024-09-11T03:52:06.080944+0000","last_active":"2024-09-11T03:52:06.103525+0000","last_peered":"2024-09-11T03:52:06.103525+0000","last_clean":"2024-09-11T03:52:06.103525+0000","last_became_active":"2024-09-11T03:50:31.047148+0000","last_became_peered":"2024-09-11T03:50:31.047148+0000","last_unstale":"2024-09-11T03:52:06.103525+0000","last_undegraded":"2024-09-11T03:52:06.103525+0000","last_fullsized":"2024-09-11T03:52:06.103525+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":5,"last_scrub":"882'911","last_scrub_stamp":"2024-09-11T03:51:45.052831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:45.052831+0000","objects_scrubbed":6,"log_size":928,"log_dups_size":0,"ondisk_log_size":928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:54.625672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00086334699999999999,"stat_sum":{"num_bytes":14006122,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":15,"num_read_kb":10932,"num_write":19,"num_write_kb":5002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"143","length":"2"},{"start":"15b","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.19","version":"20'92","reported_seq":1043,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.291127+0000","last_change":"2024-09-11T03:51:28.080491+0000","last_active":"2024-09-11T03:52:07.291127+0000","last_peered":"2024-09-11T03:52:07.291127+0000","last_clean":"2024-09-11T03:52:07.291127+0000","last_became_active":"2024-09-11T03:46:15.972841+0000","last_became_peered":"2024-09-11T03:46:15.972841+0000","last_unstale":"2024-09-11T03:52:07.291127+0000","last_undegraded":"2024-09-11T03:52:07.291127+0000","last_fullsized":"2024-09-11T03:52:07.291127+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:28.080420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.080420+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:28.344483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"557'273","reported_seq":4311,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.063251+0000","last_change":"2024-09-11T03:52:06.088643+0000","last_active":"2024-09-11T03:52:07.063251+0000","last_peered":"2024-09-11T03:52:07.063251+0000","last_clean":"2024-09-11T03:52:07.063251+0000","last_became_active":"2024-09-11T03:50:28.038587+0000","last_became_peered":"2024-09-11T03:50:28.038587+0000","last_unstale":"2024-09-11T03:52:07.063251+0000","last_undegraded":"2024-09-11T03:52:07.063251+0000","last_fullsized":"2024-09-11T03:52:07.063251+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:51:54.121178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.121178+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:14.357120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0067314159999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,2],"acting":[6,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.1a","version":"20'92","reported_seq":604,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.127687+0000","last_change":"2024-09-11T03:51:23.032108+0000","last_active":"2024-09-11T03:52:11.127687+0000","last_peered":"2024-09-11T03:52:11.127687+0000","last_clean":"2024-09-11T03:52:11.127687+0000","last_became_active":"2024-09-11T03:51:17.063957+0000","last_became_peered":"2024-09-11T03:51:17.063957+0000","last_unstale":"2024-09-11T03:52:11.127687+0000","last_undegraded":"2024-09-11T03:52:11.127687+0000","last_fullsized":"2024-09-11T03:52:11.127687+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:23.032056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.032056+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:35.587988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.19","version":"908'630","reported_seq":4793,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.063041+0000","last_change":"2024-09-11T03:52:06.087623+0000","last_active":"2024-09-11T03:52:07.063041+0000","last_peered":"2024-09-11T03:52:07.063041+0000","last_clean":"2024-09-11T03:52:07.063041+0000","last_became_active":"2024-09-11T03:50:28.039694+0000","last_became_peered":"2024-09-11T03:50:28.039694+0000","last_unstale":"2024-09-11T03:52:07.063041+0000","last_undegraded":"2024-09-11T03:52:07.063041+0000","last_fullsized":"2024-09-11T03:52:07.063041+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"816'587","last_scrub_stamp":"2024-09-11T03:50:36.522195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:36.522195+0000","objects_scrubbed":7,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0061287909999999998,"stat_sum":{"num_bytes":13997764,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":14,"num_write_kb":3830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.1b","version":"20'92","reported_seq":1424,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.125685+0000","last_change":"2024-09-11T03:51:52.434832+0000","last_active":"2024-09-11T03:52:11.125685+0000","last_peered":"2024-09-11T03:52:11.125685+0000","last_clean":"2024-09-11T03:52:11.125685+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:52:11.125685+0000","last_undegraded":"2024-09-11T03:52:11.125685+0000","last_fullsized":"2024-09-11T03:52:11.125685+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:52.434784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.434784+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:11.714276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"910'736","reported_seq":5000,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-09-11T03:52:08.107374+0000","last_change":"2024-09-11T03:52:08.107374+0000","last_active":"2024-09-11T03:52:08.107374+0000","last_peered":"2024-09-11T03:52:08.107374+0000","last_clean":"2024-09-11T03:52:08.107374+0000","last_became_active":"2024-09-11T03:51:20.065666+0000","last_became_peered":"2024-09-11T03:51:20.065666+0000","last_unstale":"2024-09-11T03:52:08.107374+0000","last_undegraded":"2024-09-11T03:52:08.107374+0000","last_fullsized":"2024-09-11T03:52:08.107374+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":5,"last_scrub":"816'566","last_scrub_stamp":"2024-09-11T03:50:35.256345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:35.256345+0000","objects_scrubbed":11,"log_size":736,"log_dups_size":0,"ondisk_log_size":736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0058198410000000001,"stat_sum":{"num_bytes":41106880,"num_objects":18,"num_object_clones":16,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":36,"num_read_kb":35366,"num_write":29,"num_write_kb":11526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":15760321,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,11],"acting":[9,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"2.1f","version":"910'897","reported_seq":4968,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-09-11T03:52:10.191136+0000","last_change":"2024-09-11T03:52:08.105089+0000","last_active":"2024-09-11T03:52:10.191136+0000","last_peered":"2024-09-11T03:52:10.191136+0000","last_clean":"2024-09-11T03:52:10.191136+0000","last_became_active":"2024-09-11T03:50:28.041110+0000","last_became_peered":"2024-09-11T03:50:28.041110+0000","last_unstale":"2024-09-11T03:52:10.191136+0000","last_undegraded":"2024-09-11T03:52:10.191136+0000","last_fullsized":"2024-09-11T03:52:10.191136+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"894'871","last_scrub_stamp":"2024-09-11T03:51:53.078478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.078478+0000","objects_scrubbed":13,"log_size":897,"log_dups_size":0,"ondisk_log_size":897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:11.324351+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0057036090000000001,"stat_sum":{"num_bytes":33945255,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":52,"num_read_kb":49670,"num_write":26,"num_write_kb":13669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.1c","version":"20'92","reported_seq":1431,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.127707+0000","last_change":"2024-09-11T03:51:05.596734+0000","last_active":"2024-09-11T03:52:11.127707+0000","last_peered":"2024-09-11T03:52:11.127707+0000","last_clean":"2024-09-11T03:52:11.127707+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:52:11.127707+0000","last_undegraded":"2024-09-11T03:52:11.127707+0000","last_fullsized":"2024-09-11T03:52:11.127707+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:05.596677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:05.596677+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1e","version":"911'1116","reported_seq":5194,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-09-11T03:52:10.190623+0000","last_change":"2024-09-11T03:52:08.102613+0000","last_active":"2024-09-11T03:52:10.190623+0000","last_peered":"2024-09-11T03:52:10.190623+0000","last_clean":"2024-09-11T03:52:10.190623+0000","last_became_active":"2024-09-11T03:50:28.044451+0000","last_became_peered":"2024-09-11T03:50:28.044451+0000","last_unstale":"2024-09-11T03:52:10.190623+0000","last_undegraded":"2024-09-11T03:52:10.190623+0000","last_fullsized":"2024-09-11T03:52:10.190623+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"885'1106","last_scrub_stamp":"2024-09-11T03:51:44.050833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:44.050833+0000","objects_scrubbed":5,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:02.921095+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020800580000000001,"stat_sum":{"num_bytes":9671047,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":27,"num_read_kb":24362,"num_write":13,"num_write_kb":3430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"},{"start":"19f","length":"1"}]},{"pgid":"1.1d","version":"20'92","reported_seq":1420,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-09-11T03:52:11.119090+0000","last_change":"2024-09-11T03:52:08.623749+0000","last_active":"2024-09-11T03:52:11.119090+0000","last_peered":"2024-09-11T03:52:11.119090+0000","last_clean":"2024-09-11T03:52:11.119090+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:52:11.119090+0000","last_undegraded":"2024-09-11T03:52:11.119090+0000","last_fullsized":"2024-09-11T03:52:11.119090+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:08.623692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:08.623692+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:16.913356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"567'752","reported_seq":4802,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-09-11T03:52:07.292451+0000","last_change":"2024-09-11T03:52:06.092148+0000","last_active":"2024-09-11T03:52:07.292451+0000","last_peered":"2024-09-11T03:52:07.292451+0000","last_clean":"2024-09-11T03:52:07.292451+0000","last_became_active":"2024-09-11T03:50:28.045687+0000","last_became_peered":"2024-09-11T03:50:28.045687+0000","last_unstale":"2024-09-11T03:52:07.292451+0000","last_undegraded":"2024-09-11T03:52:07.292451+0000","last_fullsized":"2024-09-11T03:52:07.292451+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:51:52.141035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.141035+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:58.296530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00062390799999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,11],"acting":[10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"144","length":"1"},{"start":"174","length":"1"},{"start":"188","length":"1"}]},{"pgid":"1.1e","version":"20'92","reported_seq":1427,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-09-11T03:52:06.081652+0000","last_change":"2024-09-11T03:51:02.962842+0000","last_active":"2024-09-11T03:52:06.081652+0000","last_peered":"2024-09-11T03:52:06.081652+0000","last_clean":"2024-09-11T03:52:06.081652+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:52:06.081652+0000","last_undegraded":"2024-09-11T03:52:06.081652+0000","last_fullsized":"2024-09-11T03:52:06.081652+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:02.962791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:02.962791+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:13.625901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":76,"stat_sum":{"num_bytes":777786603,"num_objects":327,"num_object_clones":277,"num_object_copies":981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":11,"num_read":1073,"num_read_kb":1147992,"num_write":713,"num_write_kb":286440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":82257142,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1376092160,"data_stored":1812601193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47864,"ondisk_log_size":47864,"up":228,"acting":228,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285747,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":229236,"kb_used_data":171676,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93515916,"statfs":{"total":95995035648,"available":95760297984,"internally_reserved":0,"allocated":175796224,"data_stored":226803001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":58915268},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514679,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91768,"kb_used_data":60580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93653384,"statfs":{"total":95995035648,"available":95901065216,"internally_reserved":0,"allocated":62033920,"data_stored":71091875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743613,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":178084,"kb_used_data":114964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93567068,"statfs":{"total":95995035648,"available":95812677632,"internally_reserved":0,"allocated":117723136,"data_stored":139488914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037957,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":193448,"kb_used_data":148368,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93551704,"statfs":{"total":95995035648,"available":95796944896,"internally_reserved":0,"allocated":151928832,"data_stored":207670080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299592,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":226568,"kb_used_data":155124,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93518584,"statfs":{"total":95995035648,"available":95763030016,"internally_reserved":0,"allocated":158846976,"data_stored":193799941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528524,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":216400,"kb_used_data":163004,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93528752,"statfs":{"total":95995035648,"available":95773442048,"internally_reserved":0,"allocated":166916096,"data_stored":224448496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757459,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":171384,"kb_used_data":117980,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93573768,"statfs":{"total":95995035648,"available":95819538432,"internally_reserved":0,"allocated":120811520,"data_stored":157736893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986392,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149917,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":205444,"kb_used_data":134000,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93539708,"statfs":{"total":95995035648,"available":95784660992,"internally_reserved":0,"allocated":137216000,"data_stored":184463141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411554,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":138128,"kb_used_data":84732,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93607024,"statfs":{"total":95995035648,"available":95853592576,"internally_reserved":0,"allocated":86765568,"data_stored":100036901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575080,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":143800,"kb_used_data":98720,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93601352,"statfs":{"total":95995035648,"available":95847784448,"internally_reserved":0,"allocated":101089280,"data_stored":118538360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738605,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":181012,"kb_used_data":119288,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":93564140,"statfs":{"total":95995035648,"available":95809679360,"internally_reserved":0,"allocated":122150912,"data_stored":144071945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110759936,"data_stored":136489340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90001408,"data_stored":111266579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":76664832,"data_stored":94351931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":125837312,"data_stored":176888928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":96878592,"data_stored":122811969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":109518848,"data_stored":150263864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":155561984,"data_stored":216891195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":147566592,"data_stored":186335376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":140840960,"data_stored":200398299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":106684416,"data_stored":132250829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":51007488,"data_stored":63862254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":164769792,"data_stored":220790629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:52:14.928 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-11T03:52:14.928 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 86 2024-09-11T03:52:14.960 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:14.991 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:14.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:14.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:15.093 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:15.115 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:15.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:15.130 INFO:tasks.rados.rados.2.smithi175.stdout:3810: done (3 left) 2024-09-11T03:52:15.130 INFO:tasks.rados.rados.2.smithi175.stdout:3811: done (2 left) 2024-09-11T03:52:15.130 INFO:tasks.rados.rados.2.smithi175.stdout:3812: done (1 left) 2024-09-11T03:52:15.130 INFO:tasks.rados.rados.2.smithi175.stdout:3813: done (0 left) 2024-09-11T03:52:15.130 INFO:tasks.rados.rados.2.smithi175.stdout:3814: write (excl) oid 40 current snap is 425 2024-09-11T03:52:15.130 INFO:tasks.rados.rados.2.smithi175.stdout:3814: seq_num 1291 ranges {497837=410116,1596371=743454,2902966=763698} 2024-09-11T03:52:15.134 INFO:tasks.rados.rados.2.smithi175.stdout:3814: writing smithi17557870-40 from 497837 to 907953 tid 1 2024-09-11T03:52:15.142 INFO:tasks.rados.rados.2.smithi175.stdout:3814: writing smithi17557870-40 from 1596371 to 2339825 tid 2 2024-09-11T03:52:15.150 INFO:tasks.rados.rados.2.smithi175.stdout:3814: writing smithi17557870-40 from 2902966 to 3666664 tid 3 2024-09-11T03:52:15.150 INFO:tasks.rados.rados.2.smithi175.stdout:3815: write (excl) oid 32 current snap is 425 2024-09-11T03:52:15.150 INFO:tasks.rados.rados.2.smithi175.stdout:3815: seq_num 1292 ranges {514644=485365,1654853=487112,2930389=574970} 2024-09-11T03:52:15.155 INFO:tasks.rados.rados.2.smithi175.stdout:3815: writing smithi17557870-32 from 514644 to 1000009 tid 1 2024-09-11T03:52:15.162 INFO:tasks.rados.rados.2.smithi175.stdout:3815: writing smithi17557870-32 from 1654853 to 2141965 tid 2 2024-09-11T03:52:15.169 INFO:tasks.rados.rados.2.smithi175.stdout:3815: writing smithi17557870-32 from 2930389 to 3505359 tid 3 2024-09-11T03:52:15.169 INFO:tasks.rados.rados.2.smithi175.stdout:3816: copy_from oid 1 from oid 16 current snap is 425 2024-09-11T03:52:15.169 INFO:tasks.rados.rados.2.smithi175.stdout:3814: finishing write tid 1 to smithi17557870-40 2024-09-11T03:52:15.170 INFO:tasks.rados.rados.2.smithi175.stdout:3814: finishing write tid 2 to smithi17557870-40 2024-09-11T03:52:15.170 INFO:tasks.rados.rados.2.smithi175.stdout:3814: finishing write tid 3 to smithi17557870-40 2024-09-11T03:52:15.170 INFO:tasks.rados.rados.2.smithi175.stdout:3814: finishing write tid 4 to smithi17557870-40 2024-09-11T03:52:15.170 INFO:tasks.rados.rados.2.smithi175.stdout:3814: finishing write tid 5 to smithi17557870-40 2024-09-11T03:52:15.170 INFO:tasks.rados.rados.2.smithi175.stdout:3814: oid 40 updating version 0 to 1019 2024-09-11T03:52:15.170 INFO:tasks.rados.rados.2.smithi175.stdout:3814: oid 40 version 1019 is already newer than 1018 2024-09-11T03:52:15.170 INFO:tasks.rados.rados.2.smithi175.stdout:3814: oid 40 version 1019 is already newer than 1017 2024-09-11T03:52:15.170 INFO:tasks.rados.rados.2.smithi175.stdout:3814: oid 40 version 1019 is already newer than 1016 2024-09-11T03:52:15.170 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 1019 (ObjNum 1291 snap 425 seq_num 1291) dirty exists 2024-09-11T03:52:15.170 INFO:tasks.rados.rados.2.smithi175.stdout:3814: left oid 40 (ObjNum 1291 snap 425 seq_num 1291) 2024-09-11T03:52:15.170 INFO:tasks.rados.rados.2.smithi175.stdout:3815: finishing write tid 1 to smithi17557870-32 2024-09-11T03:52:15.170 INFO:tasks.rados.rados.2.smithi175.stdout:3815: finishing write tid 2 to smithi17557870-32 2024-09-11T03:52:15.170 INFO:tasks.rados.rados.2.smithi175.stdout:3814: done (2 left) 2024-09-11T03:52:15.171 INFO:tasks.rados.rados.2.smithi175.stdout:3817: snap_remove snap 380 2024-09-11T03:52:15.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:15.242 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:15.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:15.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:15.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:15.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:15.366 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:15.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:15.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:15 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:52:15.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:15 smithi084 ceph-mon[27254]: 2.d scrub ok 2024-09-11T03:52:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:15 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:52:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:15 smithi121 ceph-mon[30729]: 2.d scrub ok 2024-09-11T03:52:15.467 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:15 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:52:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:15 smithi175 ceph-mon[30894]: 2.d scrub ok 2024-09-11T03:52:15.488 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:15.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:15.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:15.493 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:15.524 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:15.532 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:15.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:15.614 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:15.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:15.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:15.717 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:15.739 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:15.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:15.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:15.841 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:15.863 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:15.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:15.864 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:15.965 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:15.986 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:15.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:15.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:16.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:16.114 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:16.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:16.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:16.144 INFO:tasks.rados.rados.2.smithi175.stdout:3817: done (2 left) 2024-09-11T03:52:16.144 INFO:tasks.rados.rados.2.smithi175.stdout:3818: copy_from oid 15 from oid 45 current snap is 425 2024-09-11T03:52:16.144 INFO:tasks.rados.rados.2.smithi175.stdout:3815: finishing write tid 3 to smithi17557870-32 2024-09-11T03:52:16.145 INFO:tasks.rados.rados.2.smithi175.stdout:3815: finishing write tid 4 to smithi17557870-32 2024-09-11T03:52:16.145 INFO:tasks.rados.rados.2.smithi175.stdout:3815: finishing write tid 5 to smithi17557870-32 2024-09-11T03:52:16.145 INFO:tasks.rados.rados.2.smithi175.stdout:3815: oid 32 updating version 0 to 1033 2024-09-11T03:52:16.145 INFO:tasks.rados.rados.2.smithi175.stdout:3815: oid 32 version 1033 is already newer than 1030 2024-09-11T03:52:16.145 INFO:tasks.rados.rados.2.smithi175.stdout:3815: oid 32 version 1033 is already newer than 1032 2024-09-11T03:52:16.145 INFO:tasks.rados.rados.2.smithi175.stdout:3815: oid 32 version 1033 is already newer than 1031 2024-09-11T03:52:16.145 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 32 v 1033 (ObjNum 1292 snap 425 seq_num 1292) dirty exists 2024-09-11T03:52:16.145 INFO:tasks.rados.rados.2.smithi175.stdout:3815: left oid 32 (ObjNum 1292 snap 425 seq_num 1292) 2024-09-11T03:52:16.145 INFO:tasks.rados.rados.2.smithi175.stdout:3816: finishing copy_from racing read to smithi17557870-1 2024-09-11T03:52:16.145 INFO:tasks.rados.rados.2.smithi175.stdout:3816: finishing copy_from to smithi17557870-1 2024-09-11T03:52:16.146 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 1 v 1007 (ObjNum 1275 snap 422 seq_num 1275) dirty exists 2024-09-11T03:52:16.146 INFO:tasks.rados.rados.2.smithi175.stdout:3815: done (2 left) 2024-09-11T03:52:16.146 INFO:tasks.rados.rados.2.smithi175.stdout:3816: done (1 left) 2024-09-11T03:52:16.146 INFO:tasks.rados.rados.2.smithi175.stdout:3819: delete oid 29 current snap is 425 2024-09-11T03:52:16.147 INFO:tasks.rados.rados.2.smithi175.stdout:3818: finishing copy_from to smithi17557870-15 2024-09-11T03:52:16.147 INFO:tasks.rados.rados.2.smithi175.stdout:3818: got expected ENOENT (src dne) 2024-09-11T03:52:16.147 INFO:tasks.rados.rados.2.smithi175.stdout:3818: finishing copy_from racing read to smithi17557870-15 2024-09-11T03:52:16.151 INFO:tasks.rados.rados.2.smithi175.stdout:3818: done (1 left) 2024-09-11T03:52:16.151 INFO:tasks.rados.rados.2.smithi175.stdout:3819: done (0 left) 2024-09-11T03:52:16.151 INFO:tasks.rados.rados.2.smithi175.stdout:3820: write (excl) oid 38 current snap is 425 2024-09-11T03:52:16.152 INFO:tasks.rados.rados.2.smithi175.stdout:3820: seq_num 1294 ranges {548258=752216,1771817=490781,2985235=713867} 2024-09-11T03:52:16.158 INFO:tasks.rados.rados.2.smithi175.stdout:3820: writing smithi17557870-38 from 548258 to 1300474 tid 1 2024-09-11T03:52:16.163 INFO:tasks.rados.rados.2.smithi175.stdout:3820: writing smithi17557870-38 from 1771817 to 2262598 tid 2 2024-09-11T03:52:16.172 INFO:tasks.rados.rados.2.smithi175.stdout:3820: writing smithi17557870-38 from 2985235 to 3699102 tid 3 2024-09-11T03:52:16.172 INFO:tasks.rados.rados.2.smithi175.stdout:3821: read oid 10 snap -1 2024-09-11T03:52:16.172 INFO:tasks.rados.rados.2.smithi175.stdout:3821: expect (ObjNum 1281 snap 422 seq_num 1281) 2024-09-11T03:52:16.172 INFO:tasks.rados.rados.2.smithi175.stdout:3822: snap_create 2024-09-11T03:52:16.172 INFO:tasks.rados.rados.2.smithi175.stdout:3820: finishing write tid 1 to smithi17557870-38 2024-09-11T03:52:16.172 INFO:tasks.rados.rados.2.smithi175.stdout:3820: finishing write tid 2 to smithi17557870-38 2024-09-11T03:52:16.176 INFO:tasks.rados.rados.2.smithi175.stdout:3820: finishing write tid 3 to smithi17557870-38 2024-09-11T03:52:16.176 INFO:tasks.rados.rados.2.smithi175.stdout:3820: finishing write tid 4 to smithi17557870-38 2024-09-11T03:52:16.176 INFO:tasks.rados.rados.2.smithi175.stdout:3820: finishing write tid 5 to smithi17557870-38 2024-09-11T03:52:16.176 INFO:tasks.rados.rados.2.smithi175.stdout:3820: oid 38 updating version 0 to 1581 2024-09-11T03:52:16.176 INFO:tasks.rados.rados.2.smithi175.stdout:3820: oid 38 version 1581 is already newer than 1580 2024-09-11T03:52:16.176 INFO:tasks.rados.rados.2.smithi175.stdout:3820: oid 38 version 1581 is already newer than 1579 2024-09-11T03:52:16.177 INFO:tasks.rados.rados.2.smithi175.stdout:3820: oid 38 updating version 1581 to 1582 2024-09-11T03:52:16.177 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 1582 (ObjNum 1294 snap 425 seq_num 1294) dirty exists 2024-09-11T03:52:16.177 INFO:tasks.rados.rados.2.smithi175.stdout:3820: left oid 38 (ObjNum 1294 snap 425 seq_num 1294) 2024-09-11T03:52:16.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:16.240 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:16.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:16.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:16.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: 1.1c scrub starts 2024-09-11T03:52:16.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: 1.1c scrub ok 2024-09-11T03:52:16.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: pgmap v1530: 107 pgs: 107 active+clean; 755 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 31 MiB/s rd, 5.1 MiB/s wr, 42 op/s 2024-09-11T03:52:16.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: 1.16 scrub starts 2024-09-11T03:52:16.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: 1.16 scrub ok 2024-09-11T03:52:16.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: osdmap e917: 12 total, 12 up, 11 in 2024-09-11T03:52:16.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=4096704000,config=358384381,health=861809967,kv=3667711542}) 2024-09-11T03:52:16.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2062454290}) 2024-09-11T03:52:16.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=4194546510}) 2024-09-11T03:52:16.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {kv=24,logm=76} crc {kv=730684290,logm=2848617363}) 2024-09-11T03:52:16.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3419222951}) 2024-09-11T03:52:16.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=179282166}) 2024-09-11T03:52:16.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=197444566}) 2024-09-11T03:52:16.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=490890796}) 2024-09-11T03:52:16.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749267915}) 2024-09-11T03:52:16.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1171254786}) 2024-09-11T03:52:16.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3912403617}) 2024-09-11T03:52:16.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2350294806}) 2024-09-11T03:52:16.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3610064411}) 2024-09-11T03:52:16.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104001935}) 2024-09-11T03:52:16.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3055133745}) 2024-09-11T03:52:16.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3156501722}) 2024-09-11T03:52:16.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3784589}) 2024-09-11T03:52:16.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791168285}) 2024-09-11T03:52:16.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2849678333}) 2024-09-11T03:52:16.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=934940349}) 2024-09-11T03:52:16.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=65514981}) 2024-09-11T03:52:16.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=53499126}) 2024-09-11T03:52:16.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=780782234}) 2024-09-11T03:52:16.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3903628485}) 2024-09-11T03:52:16.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1681741257}) 2024-09-11T03:52:16.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2945588290}) 2024-09-11T03:52:16.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1333148535}) 2024-09-11T03:52:16.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1274815114}) 2024-09-11T03:52:16.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2205327087}) 2024-09-11T03:52:16.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=440338596}) 2024-09-11T03:52:16.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1027553112}) 2024-09-11T03:52:16.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1602230210}) 2024-09-11T03:52:16.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1194211108}) 2024-09-11T03:52:16.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4293899323}) 2024-09-11T03:52:16.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1142750390}) 2024-09-11T03:52:16.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=790605184}) 2024-09-11T03:52:16.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=461297344}) 2024-09-11T03:52:16.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2740915791}) 2024-09-11T03:52:16.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3299320437}) 2024-09-11T03:52:16.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1256100513}) 2024-09-11T03:52:16.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3233683434}) 2024-09-11T03:52:16.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=29287667}) 2024-09-11T03:52:16.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1791819933}) 2024-09-11T03:52:16.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1084633222}) 2024-09-11T03:52:16.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1166624362}) 2024-09-11T03:52:16.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2489705104}) 2024-09-11T03:52:16.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2146980658}) 2024-09-11T03:52:16.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2122861030}) 2024-09-11T03:52:16.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1206679773}) 2024-09-11T03:52:16.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=237728701}) 2024-09-11T03:52:16.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=818694624}) 2024-09-11T03:52:16.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=859615524}) 2024-09-11T03:52:16.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=247170911}) 2024-09-11T03:52:16.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3322773313}) 2024-09-11T03:52:16.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1335031723}) 2024-09-11T03:52:16.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776632370}) 2024-09-11T03:52:16.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3055253222}) 2024-09-11T03:52:16.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2497294711}) 2024-09-11T03:52:16.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=981821888}) 2024-09-11T03:52:16.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1205755067}) 2024-09-11T03:52:16.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2994279653}) 2024-09-11T03:52:16.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2298280510}) 2024-09-11T03:52:16.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891847821}) 2024-09-11T03:52:16.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3393199972}) 2024-09-11T03:52:16.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3648906580}) 2024-09-11T03:52:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {logm=56,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=20} crc {logm=2269102963,mdsmap=827622067,mgr=2786812108,mgr_command_descs=899306875,mgr_metadata=2702562194,mgrstat=2904636582}) 2024-09-11T03:52:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=89,mon_config_key=11} crc {mgrstat=1558322432,mon_config_key=1975994216}) 2024-09-11T03:52:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3263852641}) 2024-09-11T03:52:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1757447022}) 2024-09-11T03:52:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=53,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osd_snap=26} crc {mon_config_key=394161868,monmap=4220057526,nvmeofgw=598337686,osd_metadata=132375096,osd_pg_creating=1867940466,osd_snap=1695355184}) 2024-09-11T03:52:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=100} crc {osd_snap=3439206709}) 2024-09-11T03:52:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=54,osdmap=46} crc {osd_snap=1668368680,osdmap=4191207486}) 2024-09-11T03:52:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1649451106}) 2024-09-11T03:52:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2054208398}) 2024-09-11T03:52:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4133957077}) 2024-09-11T03:52:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2922656082}) 2024-09-11T03:52:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3708743262}) 2024-09-11T03:52:16.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2655793378}) 2024-09-11T03:52:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3914813858}) 2024-09-11T03:52:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1308065272}) 2024-09-11T03:52:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=116970919}) 2024-09-11T03:52:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=427093686}) 2024-09-11T03:52:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1740418799}) 2024-09-11T03:52:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1636640068}) 2024-09-11T03:52:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2422301144}) 2024-09-11T03:52:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2739005704}) 2024-09-11T03:52:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1441871621}) 2024-09-11T03:52:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1074032949}) 2024-09-11T03:52:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1863409039}) 2024-09-11T03:52:16.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:16 smithi084 ceph-mon[27254]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=91} crc {osdmap=1605333018}) 2024-09-11T03:52:16.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:16.347 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:52:16.347 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:52:16.363 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:52:16.363 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:52:16.365 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:16.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:16.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:16.380 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:52:16.380 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:52:16.387 INFO:tasks.rados.rados.2.smithi175.stdout:3821: expect (ObjNum 1281 snap 422 seq_num 1281) 2024-09-11T03:52:16.396 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:52:16.396 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:52:16.414 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:52:16.414 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:52:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: 1.1c scrub starts 2024-09-11T03:52:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: 1.1c scrub ok 2024-09-11T03:52:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: pgmap v1530: 107 pgs: 107 active+clean; 755 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 31 MiB/s rd, 5.1 MiB/s wr, 42 op/s 2024-09-11T03:52:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: 1.16 scrub starts 2024-09-11T03:52:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: 1.16 scrub ok 2024-09-11T03:52:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: osdmap e917: 12 total, 12 up, 11 in 2024-09-11T03:52:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=4096704000,config=358384381,health=861809967,kv=3667711542}) 2024-09-11T03:52:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2062454290}) 2024-09-11T03:52:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=4194546510}) 2024-09-11T03:52:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {kv=24,logm=76} crc {kv=730684290,logm=2848617363}) 2024-09-11T03:52:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3419222951}) 2024-09-11T03:52:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=179282166}) 2024-09-11T03:52:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=197444566}) 2024-09-11T03:52:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=490890796}) 2024-09-11T03:52:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749267915}) 2024-09-11T03:52:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1171254786}) 2024-09-11T03:52:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3912403617}) 2024-09-11T03:52:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2350294806}) 2024-09-11T03:52:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3610064411}) 2024-09-11T03:52:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104001935}) 2024-09-11T03:52:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3055133745}) 2024-09-11T03:52:16.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3156501722}) 2024-09-11T03:52:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3784589}) 2024-09-11T03:52:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791168285}) 2024-09-11T03:52:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2849678333}) 2024-09-11T03:52:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=934940349}) 2024-09-11T03:52:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=65514981}) 2024-09-11T03:52:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=53499126}) 2024-09-11T03:52:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=780782234}) 2024-09-11T03:52:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3903628485}) 2024-09-11T03:52:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1681741257}) 2024-09-11T03:52:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2945588290}) 2024-09-11T03:52:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1333148535}) 2024-09-11T03:52:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1274815114}) 2024-09-11T03:52:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2205327087}) 2024-09-11T03:52:16.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=440338596}) 2024-09-11T03:52:16.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1027553112}) 2024-09-11T03:52:16.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1602230210}) 2024-09-11T03:52:16.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1194211108}) 2024-09-11T03:52:16.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4293899323}) 2024-09-11T03:52:16.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1142750390}) 2024-09-11T03:52:16.425 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=790605184}) 2024-09-11T03:52:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=461297344}) 2024-09-11T03:52:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2740915791}) 2024-09-11T03:52:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3299320437}) 2024-09-11T03:52:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1256100513}) 2024-09-11T03:52:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3233683434}) 2024-09-11T03:52:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=29287667}) 2024-09-11T03:52:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1791819933}) 2024-09-11T03:52:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1084633222}) 2024-09-11T03:52:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1166624362}) 2024-09-11T03:52:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2489705104}) 2024-09-11T03:52:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2146980658}) 2024-09-11T03:52:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2122861030}) 2024-09-11T03:52:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1206679773}) 2024-09-11T03:52:16.426 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=237728701}) 2024-09-11T03:52:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=818694624}) 2024-09-11T03:52:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=859615524}) 2024-09-11T03:52:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=247170911}) 2024-09-11T03:52:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3322773313}) 2024-09-11T03:52:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1335031723}) 2024-09-11T03:52:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776632370}) 2024-09-11T03:52:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3055253222}) 2024-09-11T03:52:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2497294711}) 2024-09-11T03:52:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=981821888}) 2024-09-11T03:52:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1205755067}) 2024-09-11T03:52:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2994279653}) 2024-09-11T03:52:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2298280510}) 2024-09-11T03:52:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891847821}) 2024-09-11T03:52:16.427 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3393199972}) 2024-09-11T03:52:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3648906580}) 2024-09-11T03:52:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {logm=56,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=20} crc {logm=2269102963,mdsmap=827622067,mgr=2786812108,mgr_command_descs=899306875,mgr_metadata=2702562194,mgrstat=2904636582}) 2024-09-11T03:52:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=89,mon_config_key=11} crc {mgrstat=1558322432,mon_config_key=1975994216}) 2024-09-11T03:52:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3263852641}) 2024-09-11T03:52:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1757447022}) 2024-09-11T03:52:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=53,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osd_snap=26} crc {mon_config_key=394161868,monmap=4220057526,nvmeofgw=598337686,osd_metadata=132375096,osd_pg_creating=1867940466,osd_snap=1695355184}) 2024-09-11T03:52:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=100} crc {osd_snap=3439206709}) 2024-09-11T03:52:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=54,osdmap=46} crc {osd_snap=1668368680,osdmap=4191207486}) 2024-09-11T03:52:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1649451106}) 2024-09-11T03:52:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2054208398}) 2024-09-11T03:52:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4133957077}) 2024-09-11T03:52:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2922656082}) 2024-09-11T03:52:16.428 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3708743262}) 2024-09-11T03:52:16.429 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2655793378}) 2024-09-11T03:52:16.429 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3914813858}) 2024-09-11T03:52:16.429 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1308065272}) 2024-09-11T03:52:16.429 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=116970919}) 2024-09-11T03:52:16.429 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=427093686}) 2024-09-11T03:52:16.429 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1740418799}) 2024-09-11T03:52:16.429 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1636640068}) 2024-09-11T03:52:16.429 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2422301144}) 2024-09-11T03:52:16.429 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2739005704}) 2024-09-11T03:52:16.429 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1441871621}) 2024-09-11T03:52:16.429 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1074032949}) 2024-09-11T03:52:16.429 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1863409039}) 2024-09-11T03:52:16.429 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:16 smithi121 ceph-mon[30729]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=91} crc {osdmap=1605333018}) 2024-09-11T03:52:16.431 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:52:16.431 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:52:16.448 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:52:16.448 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:52:16.465 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:52:16.465 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:52:16.467 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: 1.1c scrub starts 2024-09-11T03:52:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: 1.1c scrub ok 2024-09-11T03:52:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: pgmap v1530: 107 pgs: 107 active+clean; 755 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 31 MiB/s rd, 5.1 MiB/s wr, 42 op/s 2024-09-11T03:52:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: 1.16 scrub starts 2024-09-11T03:52:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: 1.16 scrub ok 2024-09-11T03:52:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: osdmap e917: 12 total, 12 up, 11 in 2024-09-11T03:52:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=4096704000,config=358384381,health=861809967,kv=3667711542}) 2024-09-11T03:52:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2062454290}) 2024-09-11T03:52:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=4194546510}) 2024-09-11T03:52:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {kv=24,logm=76} crc {kv=730684290,logm=2848617363}) 2024-09-11T03:52:16.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3419222951}) 2024-09-11T03:52:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=179282166}) 2024-09-11T03:52:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=197444566}) 2024-09-11T03:52:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=490890796}) 2024-09-11T03:52:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749267915}) 2024-09-11T03:52:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1171254786}) 2024-09-11T03:52:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3912403617}) 2024-09-11T03:52:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2350294806}) 2024-09-11T03:52:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3610064411}) 2024-09-11T03:52:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104001935}) 2024-09-11T03:52:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3055133745}) 2024-09-11T03:52:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3156501722}) 2024-09-11T03:52:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3784589}) 2024-09-11T03:52:16.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791168285}) 2024-09-11T03:52:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2849678333}) 2024-09-11T03:52:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=934940349}) 2024-09-11T03:52:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=65514981}) 2024-09-11T03:52:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=53499126}) 2024-09-11T03:52:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=780782234}) 2024-09-11T03:52:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3903628485}) 2024-09-11T03:52:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1681741257}) 2024-09-11T03:52:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2945588290}) 2024-09-11T03:52:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1333148535}) 2024-09-11T03:52:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1274815114}) 2024-09-11T03:52:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2205327087}) 2024-09-11T03:52:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=440338596}) 2024-09-11T03:52:16.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1027553112}) 2024-09-11T03:52:16.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1602230210}) 2024-09-11T03:52:16.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1194211108}) 2024-09-11T03:52:16.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4293899323}) 2024-09-11T03:52:16.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1142750390}) 2024-09-11T03:52:16.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=790605184}) 2024-09-11T03:52:16.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=461297344}) 2024-09-11T03:52:16.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2740915791}) 2024-09-11T03:52:16.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3299320437}) 2024-09-11T03:52:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1256100513}) 2024-09-11T03:52:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3233683434}) 2024-09-11T03:52:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=29287667}) 2024-09-11T03:52:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1791819933}) 2024-09-11T03:52:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1084633222}) 2024-09-11T03:52:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1166624362}) 2024-09-11T03:52:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2489705104}) 2024-09-11T03:52:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2146980658}) 2024-09-11T03:52:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2122861030}) 2024-09-11T03:52:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1206679773}) 2024-09-11T03:52:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=237728701}) 2024-09-11T03:52:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=818694624}) 2024-09-11T03:52:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=859615524}) 2024-09-11T03:52:16.482 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=247170911}) 2024-09-11T03:52:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3322773313}) 2024-09-11T03:52:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1335031723}) 2024-09-11T03:52:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776632370}) 2024-09-11T03:52:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3055253222}) 2024-09-11T03:52:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2497294711}) 2024-09-11T03:52:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=981821888}) 2024-09-11T03:52:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1205755067}) 2024-09-11T03:52:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2994279653}) 2024-09-11T03:52:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2298280510}) 2024-09-11T03:52:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891847821}) 2024-09-11T03:52:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3393199972}) 2024-09-11T03:52:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3648906580}) 2024-09-11T03:52:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {logm=56,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=20} crc {logm=2269102963,mdsmap=827622067,mgr=2786812108,mgr_command_descs=899306875,mgr_metadata=2702562194,mgrstat=2904636582}) 2024-09-11T03:52:16.483 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=89,mon_config_key=11} crc {mgrstat=1558322432,mon_config_key=1975994216}) 2024-09-11T03:52:16.484 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3263852641}) 2024-09-11T03:52:16.484 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1757447022}) 2024-09-11T03:52:16.484 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=53,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osd_snap=26} crc {mon_config_key=394161868,monmap=4220057526,nvmeofgw=598337686,osd_metadata=132375096,osd_pg_creating=1867940466,osd_snap=1695355184}) 2024-09-11T03:52:16.484 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=100} crc {osd_snap=3439206709}) 2024-09-11T03:52:16.484 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=54,osdmap=46} crc {osd_snap=1668368680,osdmap=4191207486}) 2024-09-11T03:52:16.484 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1649451106}) 2024-09-11T03:52:16.484 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2054208398}) 2024-09-11T03:52:16.484 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4133957077}) 2024-09-11T03:52:16.484 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2922656082}) 2024-09-11T03:52:16.484 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3708743262}) 2024-09-11T03:52:16.484 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2655793378}) 2024-09-11T03:52:16.484 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3914813858}) 2024-09-11T03:52:16.484 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1308065272}) 2024-09-11T03:52:16.485 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=116970919}) 2024-09-11T03:52:16.485 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=427093686}) 2024-09-11T03:52:16.485 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1740418799}) 2024-09-11T03:52:16.485 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1636640068}) 2024-09-11T03:52:16.485 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2422301144}) 2024-09-11T03:52:16.485 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2739005704}) 2024-09-11T03:52:16.485 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1441871621}) 2024-09-11T03:52:16.485 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1074032949}) 2024-09-11T03:52:16.485 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1863409039}) 2024-09-11T03:52:16.485 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:16 smithi175 ceph-mon[30894]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=91} crc {osdmap=1605333018}) 2024-09-11T03:52:16.486 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:52:16.486 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:52:16.489 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:16.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:16.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:16.500 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:52:16.501 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:52:16.517 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:52:16.517 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:52:16.534 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:52:16.534 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:52:16.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:16.612 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:16.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:16.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:16.714 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:16.735 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:16.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:16.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:16.837 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:16.859 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:16.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:16.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:16.961 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:16.987 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:16.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:16.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:17.062 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 2 pg_num to 86 2024-09-11T03:52:17.065 INFO:tasks.rados.rados.2.smithi175.stdout:3820: done (2 left) 2024-09-11T03:52:17.065 INFO:tasks.rados.rados.2.smithi175.stdout:3821: done (1 left) 2024-09-11T03:52:17.065 INFO:tasks.rados.rados.2.smithi175.stdout:3822: done (0 left) 2024-09-11T03:52:17.065 INFO:tasks.rados.rados.2.smithi175.stdout:3823: delete oid 31 current snap is 426 2024-09-11T03:52:17.071 INFO:tasks.rados.rados.2.smithi175.stdout:3823: done (0 left) 2024-09-11T03:52:17.071 INFO:tasks.rados.rados.2.smithi175.stdout:3824: read oid 12 snap -1 2024-09-11T03:52:17.071 INFO:tasks.rados.rados.2.smithi175.stdout:3824: expect (ObjNum 1282 snap 422 seq_num 1282) 2024-09-11T03:52:17.071 INFO:tasks.rados.rados.2.smithi175.stdout:3825: write oid 28 current snap is 426 2024-09-11T03:52:17.071 INFO:tasks.rados.rados.2.smithi175.stdout:3825: seq_num 1296 ranges {581872=502714,1488781=494450,2640081=736411} 2024-09-11T03:52:17.077 INFO:tasks.rados.rados.2.smithi175.stdout:3825: writing smithi17557870-28 from 581872 to 1084586 tid 1 2024-09-11T03:52:17.083 INFO:tasks.rados.rados.2.smithi175.stdout:3825: writing smithi17557870-28 from 1488781 to 1983231 tid 2 2024-09-11T03:52:17.089 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:17.092 INFO:tasks.rados.rados.2.smithi175.stdout:3825: writing smithi17557870-28 from 2640081 to 3376492 tid 3 2024-09-11T03:52:17.092 INFO:tasks.rados.rados.2.smithi175.stdout:3826: snap_remove snap 403 2024-09-11T03:52:17.110 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:17.111 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:17.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:17.235 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:17.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:17.281 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:17 smithi084 ceph-mon[27254]: 2.19 scrub starts 2024-09-11T03:52:17.281 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:17 smithi084 ceph-mon[27254]: 2.19 scrub ok 2024-09-11T03:52:17.281 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:17 smithi084 ceph-mon[27254]: osdmap e918: 12 total, 12 up, 11 in 2024-09-11T03:52:17.281 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3687407006' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:17.281 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4043547797' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "86"} : dispatch 2024-09-11T03:52:17.281 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:17 smithi084 ceph-mon[27254]: 1.1e scrub starts 2024-09-11T03:52:17.281 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:17 smithi084 ceph-mon[27254]: 1.1e scrub ok 2024-09-11T03:52:17.281 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:17 smithi084 ceph-mon[27254]: 2.40 scrub starts 2024-09-11T03:52:17.281 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:17 smithi084 ceph-mon[27254]: 2.40 scrub ok 2024-09-11T03:52:17.282 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:17 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4043547797' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "86"}]': finished 2024-09-11T03:52:17.282 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:17 smithi084 ceph-mon[27254]: osdmap e919: 12 total, 12 up, 11 in 2024-09-11T03:52:17.337 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:17.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:17.359 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:17.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:17 smithi121 ceph-mon[30729]: 2.19 scrub starts 2024-09-11T03:52:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:17 smithi121 ceph-mon[30729]: 2.19 scrub ok 2024-09-11T03:52:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:17 smithi121 ceph-mon[30729]: osdmap e918: 12 total, 12 up, 11 in 2024-09-11T03:52:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3687407006' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4043547797' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "86"} : dispatch 2024-09-11T03:52:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:17 smithi121 ceph-mon[30729]: 1.1e scrub starts 2024-09-11T03:52:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:17 smithi121 ceph-mon[30729]: 1.1e scrub ok 2024-09-11T03:52:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:17 smithi121 ceph-mon[30729]: 2.40 scrub starts 2024-09-11T03:52:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:17 smithi121 ceph-mon[30729]: 2.40 scrub ok 2024-09-11T03:52:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:17 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4043547797' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "86"}]': finished 2024-09-11T03:52:17.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:17 smithi121 ceph-mon[30729]: osdmap e919: 12 total, 12 up, 11 in 2024-09-11T03:52:17.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:17.477 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:52:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:17 smithi175 ceph-mon[30894]: 2.19 scrub starts 2024-09-11T03:52:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:17 smithi175 ceph-mon[30894]: 2.19 scrub ok 2024-09-11T03:52:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:17 smithi175 ceph-mon[30894]: osdmap e918: 12 total, 12 up, 11 in 2024-09-11T03:52:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3687407006' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4043547797' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "86"} : dispatch 2024-09-11T03:52:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:17 smithi175 ceph-mon[30894]: 1.1e scrub starts 2024-09-11T03:52:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:17 smithi175 ceph-mon[30894]: 1.1e scrub ok 2024-09-11T03:52:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:17 smithi175 ceph-mon[30894]: 2.40 scrub starts 2024-09-11T03:52:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:17 smithi175 ceph-mon[30894]: 2.40 scrub ok 2024-09-11T03:52:17.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:17 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4043547797' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "86"}]': finished 2024-09-11T03:52:17.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:17 smithi175 ceph-mon[30894]: osdmap e919: 12 total, 12 up, 11 in 2024-09-11T03:52:17.481 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:52:17.481 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:52:17.485 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:17.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:17.587 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:17.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:17.612 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:17.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:17.714 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:17.739 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:17.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:17.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:17.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:17.867 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:17.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:17.969 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:17.991 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:17.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:17.992 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:18.069 INFO:tasks.rados.rados.2.smithi175.stdout:3826: done (2 left) 2024-09-11T03:52:18.069 INFO:tasks.rados.rados.2.smithi175.stdout:3827: read oid 9 snap -1 2024-09-11T03:52:18.069 INFO:tasks.rados.rados.2.smithi175.stdout:3827: expect (ObjNum 1260 snap 414 seq_num 1260) 2024-09-11T03:52:18.069 INFO:tasks.rados.rados.2.smithi175.stdout:3828: write oid 5 current snap is 426 2024-09-11T03:52:18.069 INFO:tasks.rados.rados.2.smithi175.stdout:3828: seq_num 1297 ranges {598679=577963,1947263=638108,3351151=447528} 2024-09-11T03:52:18.077 INFO:tasks.rados.rados.2.smithi175.stdout:3828: writing smithi17557870-5 from 598679 to 1176642 tid 1 2024-09-11T03:52:18.086 INFO:tasks.rados.rados.2.smithi175.stdout:3828: writing smithi17557870-5 from 1947263 to 2585371 tid 2 2024-09-11T03:52:18.093 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:18.094 INFO:tasks.rados.rados.2.smithi175.stdout:3828: writing smithi17557870-5 from 3351151 to 3798679 tid 3 2024-09-11T03:52:18.094 INFO:tasks.rados.rados.2.smithi175.stdout:3829: read oid 14 snap -1 2024-09-11T03:52:18.094 INFO:tasks.rados.rados.2.smithi175.stdout:3829: expect deleted 2024-09-11T03:52:18.094 INFO:tasks.rados.rados.2.smithi175.stdout:3830: rollback oid 38 current snap is 426 2024-09-11T03:52:18.094 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 38 to 361 2024-09-11T03:52:18.094 INFO:tasks.rados.rados.2.smithi175.stdout:3824: expect (ObjNum 1282 snap 422 seq_num 1282) 2024-09-11T03:52:18.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:18.115 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:18.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:18.217 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:18.238 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:18.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:18.239 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:18.304 INFO:tasks.rados.rados.2.smithi175.stdout:3825: finishing write tid 1 to smithi17557870-28 2024-09-11T03:52:18.304 INFO:tasks.rados.rados.2.smithi175.stdout:3825: finishing write tid 2 to smithi17557870-28 2024-09-11T03:52:18.304 INFO:tasks.rados.rados.2.smithi175.stdout:3825: finishing write tid 3 to smithi17557870-28 2024-09-11T03:52:18.304 INFO:tasks.rados.rados.2.smithi175.stdout:3825: finishing write tid 4 to smithi17557870-28 2024-09-11T03:52:18.304 INFO:tasks.rados.rados.2.smithi175.stdout:3825: finishing write tid 5 to smithi17557870-28 2024-09-11T03:52:18.304 INFO:tasks.rados.rados.2.smithi175.stdout:3825: oid 28 updating version 0 to 1561 2024-09-11T03:52:18.304 INFO:tasks.rados.rados.2.smithi175.stdout:3825: oid 28 updating version 1561 to 1563 2024-09-11T03:52:18.304 INFO:tasks.rados.rados.2.smithi175.stdout:3825: oid 28 version 1563 is already newer than 1562 2024-09-11T03:52:18.304 INFO:tasks.rados.rados.2.smithi175.stdout:3825: oid 28 version 1563 is already newer than 1560 2024-09-11T03:52:18.304 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 1563 (ObjNum 1296 snap 426 seq_num 1296) dirty exists 2024-09-11T03:52:18.304 INFO:tasks.rados.rados.2.smithi175.stdout:3825: left oid 28 (ObjNum 1296 snap 426 seq_num 1296) 2024-09-11T03:52:18.304 INFO:tasks.rados.rados.2.smithi175.stdout:3828: finishing write tid 1 to smithi17557870-5 2024-09-11T03:52:18.305 INFO:tasks.rados.rados.2.smithi175.stdout:3827: expect (ObjNum 1260 snap 414 seq_num 1260) 2024-09-11T03:52:18.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:18.365 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:18.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:18.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:18.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:18 smithi084 ceph-mon[27254]: 2.12 scrub starts 2024-09-11T03:52:18.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:18 smithi084 ceph-mon[27254]: 2.12 scrub ok 2024-09-11T03:52:18.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:18 smithi084 ceph-mon[27254]: pgmap v1533: 107 pgs: 107 active+clean; 758 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 5.6 MiB/s rd, 2.1 MiB/s wr, 10 op/s 2024-09-11T03:52:18.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:18 smithi084 ceph-mon[27254]: 2.25 scrub starts 2024-09-11T03:52:18.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:18 smithi084 ceph-mon[27254]: 2.25 scrub ok 2024-09-11T03:52:18.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:18 smithi084 ceph-mon[27254]: osdmap e920: 12 total, 12 up, 11 in 2024-09-11T03:52:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:18 smithi121 ceph-mon[30729]: 2.12 scrub starts 2024-09-11T03:52:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:18 smithi121 ceph-mon[30729]: 2.12 scrub ok 2024-09-11T03:52:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:18 smithi121 ceph-mon[30729]: pgmap v1533: 107 pgs: 107 active+clean; 758 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 5.6 MiB/s rd, 2.1 MiB/s wr, 10 op/s 2024-09-11T03:52:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:18 smithi121 ceph-mon[30729]: 2.25 scrub starts 2024-09-11T03:52:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:18 smithi121 ceph-mon[30729]: 2.25 scrub ok 2024-09-11T03:52:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:18 smithi121 ceph-mon[30729]: osdmap e920: 12 total, 12 up, 11 in 2024-09-11T03:52:18.443 INFO:tasks.rados.rados.2.smithi175.stdout:3828: finishing write tid 2 to smithi17557870-5 2024-09-11T03:52:18.443 INFO:tasks.rados.rados.2.smithi175.stdout:3828: finishing write tid 3 to smithi17557870-5 2024-09-11T03:52:18.443 INFO:tasks.rados.rados.2.smithi175.stdout:3828: finishing write tid 4 to smithi17557870-5 2024-09-11T03:52:18.443 INFO:tasks.rados.rados.2.smithi175.stdout:3828: finishing write tid 5 to smithi17557870-5 2024-09-11T03:52:18.443 INFO:tasks.rados.rados.2.smithi175.stdout:3828: oid 5 updating version 0 to 1050 2024-09-11T03:52:18.443 INFO:tasks.rados.rados.2.smithi175.stdout:3828: oid 5 updating version 1050 to 1052 2024-09-11T03:52:18.443 INFO:tasks.rados.rados.2.smithi175.stdout:3828: oid 5 version 1052 is already newer than 1051 2024-09-11T03:52:18.443 INFO:tasks.rados.rados.2.smithi175.stdout:3828: oid 5 version 1052 is already newer than 1049 2024-09-11T03:52:18.443 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 1052 (ObjNum 1297 snap 426 seq_num 1297) dirty exists 2024-09-11T03:52:18.443 INFO:tasks.rados.rados.2.smithi175.stdout:3828: left oid 5 (ObjNum 1297 snap 426 seq_num 1297) 2024-09-11T03:52:18.444 INFO:tasks.rados.rados.2.smithi175.stdout:3830: finishing rollback tid 0 to smithi17557870-38 2024-09-11T03:52:18.444 INFO:tasks.rados.rados.2.smithi175.stdout:3824: done (5 left) 2024-09-11T03:52:18.444 INFO:tasks.rados.rados.2.smithi175.stdout:3825: done (4 left) 2024-09-11T03:52:18.444 INFO:tasks.rados.rados.2.smithi175.stdout:3827: done (3 left) 2024-09-11T03:52:18.444 INFO:tasks.rados.rados.2.smithi175.stdout:3828: done (2 left) 2024-09-11T03:52:18.444 INFO:tasks.rados.rados.2.smithi175.stdout:3829: done (1 left) 2024-09-11T03:52:18.444 INFO:tasks.rados.rados.2.smithi175.stdout:3831: rollback oid 26 current snap is 426 2024-09-11T03:52:18.444 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 26 to 414 2024-09-11T03:52:18.444 INFO:tasks.rados.rados.2.smithi175.stdout:3830: finishing rollback tid 1 to smithi17557870-38 2024-09-11T03:52:18.444 INFO:tasks.rados.rados.2.smithi175.stdout:3830: finishing rollback tid 2 to smithi17557870-38 2024-09-11T03:52:18.444 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 1592 (ObjNum 970 snap 311 seq_num 970) dirty exists 2024-09-11T03:52:18.444 INFO:tasks.rados.rados.2.smithi175.stdout:3830: done (1 left) 2024-09-11T03:52:18.444 INFO:tasks.rados.rados.2.smithi175.stdout:3832: copy_from oid 6 from oid 35 current snap is 426 2024-09-11T03:52:18.444 INFO:tasks.rados.rados.2.smithi175.stdout:3833: delete oid 48 current snap is 426 2024-09-11T03:52:18.445 INFO:tasks.rados.rados.2.smithi175.stdout:3833: done (2 left) 2024-09-11T03:52:18.445 INFO:tasks.rados.rados.2.smithi175.stdout:3834: delete oid 2 current snap is 426 2024-09-11T03:52:18.448 INFO:tasks.rados.rados.2.smithi175.stdout:3831: finishing rollback tid 0 to smithi17557870-26 2024-09-11T03:52:18.448 INFO:tasks.rados.rados.2.smithi175.stdout:3831: finishing rollback tid 1 to smithi17557870-26 2024-09-11T03:52:18.448 INFO:tasks.rados.rados.2.smithi175.stdout:3831: finishing rollback tid 2 to smithi17557870-26 2024-09-11T03:52:18.448 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 1255 (ObjNum 1220 snap 399 seq_num 1220) dirty exists 2024-09-11T03:52:18.449 INFO:tasks.rados.rados.2.smithi175.stdout:3831: done (2 left) 2024-09-11T03:52:18.449 INFO:tasks.rados.rados.2.smithi175.stdout:3834: done (1 left) 2024-09-11T03:52:18.449 INFO:tasks.rados.rados.2.smithi175.stdout:3835: write (excl) oid 47 current snap is 426 2024-09-11T03:52:18.449 INFO:tasks.rados.rados.2.smithi175.stdout:3835: seq_num 1300 ranges {649100=403710,1606356=501788,2633420=497852,3636005=213095} 2024-09-11T03:52:18.454 INFO:tasks.rados.rados.2.smithi175.stdout:3835: writing smithi17557870-47 from 649100 to 1052810 tid 1 2024-09-11T03:52:18.460 INFO:tasks.rados.rados.2.smithi175.stdout:3835: writing smithi17557870-47 from 1606356 to 2108144 tid 2 2024-09-11T03:52:18.465 INFO:tasks.rados.rados.2.smithi175.stdout:3835: writing smithi17557870-47 from 2633420 to 3131272 tid 3 2024-09-11T03:52:18.468 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:18.469 INFO:tasks.rados.rados.2.smithi175.stdout:3835: writing smithi17557870-47 from 3636005 to 3849100 tid 4 2024-09-11T03:52:18.469 INFO:tasks.rados.rados.2.smithi175.stdout:3836: read oid 5 snap -1 2024-09-11T03:52:18.469 INFO:tasks.rados.rados.2.smithi175.stdout:3836: expect (ObjNum 1297 snap 426 seq_num 1297) 2024-09-11T03:52:18.470 INFO:tasks.rados.rados.2.smithi175.stdout:3837: write (excl) oid 28 current snap is 426 2024-09-11T03:52:18.470 INFO:tasks.rados.rados.2.smithi175.stdout:3832: finishing copy_from racing read to smithi17557870-6 2024-09-11T03:52:18.470 INFO:tasks.rados.rados.2.smithi175.stdout:3835: finishing write tid 1 to smithi17557870-47 2024-09-11T03:52:18.470 INFO:tasks.rados.rados.2.smithi175.stdout:3837: seq_num 1301 ranges {665907=595312,1781191=645446,3177196=688711} 2024-09-11T03:52:18.476 INFO:tasks.rados.rados.2.smithi175.stdout:3837: writing smithi17557870-28 from 665907 to 1261219 tid 1 2024-09-11T03:52:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:18 smithi175 ceph-mon[30894]: 2.12 scrub starts 2024-09-11T03:52:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:18 smithi175 ceph-mon[30894]: 2.12 scrub ok 2024-09-11T03:52:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:18 smithi175 ceph-mon[30894]: pgmap v1533: 107 pgs: 107 active+clean; 758 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 5.6 MiB/s rd, 2.1 MiB/s wr, 10 op/s 2024-09-11T03:52:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:18 smithi175 ceph-mon[30894]: 2.25 scrub starts 2024-09-11T03:52:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:18 smithi175 ceph-mon[30894]: 2.25 scrub ok 2024-09-11T03:52:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:18 smithi175 ceph-mon[30894]: osdmap e920: 12 total, 12 up, 11 in 2024-09-11T03:52:18.483 INFO:tasks.rados.rados.2.smithi175.stdout:3837: writing smithi17557870-28 from 1781191 to 2426637 tid 2 2024-09-11T03:52:18.491 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:18.491 INFO:tasks.rados.rados.2.smithi175.stdout:3837: writing smithi17557870-28 from 3177196 to 3865907 tid 3 2024-09-11T03:52:18.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:18.492 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:18.492 INFO:tasks.rados.rados.2.smithi175.stdout:3838: write (excl) oid 3 current snap is 426 2024-09-11T03:52:18.492 INFO:tasks.rados.rados.2.smithi175.stdout:3832: finishing copy_from to smithi17557870-6 2024-09-11T03:52:18.492 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 1090 (ObjNum 1283 snap 423 seq_num 1283) dirty exists 2024-09-11T03:52:18.493 INFO:tasks.rados.rados.2.smithi175.stdout:3835: finishing write tid 2 to smithi17557870-47 2024-09-11T03:52:18.493 INFO:tasks.rados.rados.2.smithi175.stdout:3835: finishing write tid 3 to smithi17557870-47 2024-09-11T03:52:18.493 INFO:tasks.rados.rados.2.smithi175.stdout:3838: seq_num 1302 ranges {682714=670561,2123320=505457,3088266=636749} 2024-09-11T03:52:18.498 INFO:tasks.rados.rados.2.smithi175.stdout:3838: writing smithi17557870-3 from 682714 to 1353275 tid 1 2024-09-11T03:52:18.505 INFO:tasks.rados.rados.2.smithi175.stdout:3838: writing smithi17557870-3 from 2123320 to 2628777 tid 2 2024-09-11T03:52:18.512 INFO:tasks.rados.rados.2.smithi175.stdout:3838: writing smithi17557870-3 from 3088266 to 3725015 tid 3 2024-09-11T03:52:18.512 INFO:tasks.rados.rados.2.smithi175.stdout:3832: done (4 left) 2024-09-11T03:52:18.512 INFO:tasks.rados.rados.2.smithi175.stdout:3839: rollback oid 30 current snap is 426 2024-09-11T03:52:18.512 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 30 to 364 2024-09-11T03:52:18.512 INFO:tasks.rados.rados.2.smithi175.stdout:3835: finishing write tid 4 to smithi17557870-47 2024-09-11T03:52:18.512 INFO:tasks.rados.rados.2.smithi175.stdout:3835: finishing write tid 5 to smithi17557870-47 2024-09-11T03:52:18.513 INFO:tasks.rados.rados.2.smithi175.stdout:3835: finishing write tid 6 to smithi17557870-47 2024-09-11T03:52:18.513 INFO:tasks.rados.rados.2.smithi175.stdout:3835: oid 47 updating version 0 to 1351 2024-09-11T03:52:18.513 INFO:tasks.rados.rados.2.smithi175.stdout:3835: oid 47 updating version 1351 to 1355 2024-09-11T03:52:18.513 INFO:tasks.rados.rados.2.smithi175.stdout:3835: oid 47 version 1355 is already newer than 1353 2024-09-11T03:52:18.513 INFO:tasks.rados.rados.2.smithi175.stdout:3835: oid 47 version 1355 is already newer than 1354 2024-09-11T03:52:18.513 INFO:tasks.rados.rados.2.smithi175.stdout:3835: oid 47 version 1355 is already newer than 1352 2024-09-11T03:52:18.513 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 1355 (ObjNum 1300 snap 426 seq_num 1300) dirty exists 2024-09-11T03:52:18.513 INFO:tasks.rados.rados.2.smithi175.stdout:3835: left oid 47 (ObjNum 1300 snap 426 seq_num 1300) 2024-09-11T03:52:18.513 INFO:tasks.rados.rados.2.smithi175.stdout:3836: expect (ObjNum 1297 snap 426 seq_num 1297) 2024-09-11T03:52:18.593 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:18.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:18.616 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:18.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:18.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:18.718 INFO:tasks.rados.rados.2.smithi175.stdout:3837: finishing write tid 1 to smithi17557870-28 2024-09-11T03:52:18.718 INFO:tasks.rados.rados.2.smithi175.stdout:3837: finishing write tid 2 to smithi17557870-28 2024-09-11T03:52:18.718 INFO:tasks.rados.rados.2.smithi175.stdout:3835: done (4 left) 2024-09-11T03:52:18.718 INFO:tasks.rados.rados.2.smithi175.stdout:3836: done (3 left) 2024-09-11T03:52:18.719 INFO:tasks.rados.rados.2.smithi175.stdout:3840: read oid 43 snap 406 2024-09-11T03:52:18.719 INFO:tasks.rados.rados.2.smithi175.stdout:3840: expect deleted 2024-09-11T03:52:18.719 INFO:tasks.rados.rados.2.smithi175.stdout:3841: write oid 15 current snap is 426 2024-09-11T03:52:18.719 INFO:tasks.rados.rados.2.smithi175.stdout:3837: finishing write tid 3 to smithi17557870-28 2024-09-11T03:52:18.719 INFO:tasks.rados.rados.2.smithi175.stdout:3841: seq_num 1303 ranges {699521=745810,2181802=649115,3515689=383832} 2024-09-11T03:52:18.722 INFO:tasks.rados.rados.2.smithi175.stdout:3841: writing smithi17557870-15 from 699521 to 1445331 tid 1 2024-09-11T03:52:18.729 INFO:tasks.rados.rados.2.smithi175.stdout:3841: writing smithi17557870-15 from 2181802 to 2830917 tid 2 2024-09-11T03:52:18.735 INFO:tasks.rados.rados.2.smithi175.stdout:3841: writing smithi17557870-15 from 3515689 to 3899521 tid 3 2024-09-11T03:52:18.735 INFO:tasks.rados.rados.2.smithi175.stdout:3842: write (excl) oid 20 current snap is 426 2024-09-11T03:52:18.735 INFO:tasks.rados.rados.2.smithi175.stdout:3837: finishing write tid 4 to smithi17557870-28 2024-09-11T03:52:18.735 INFO:tasks.rados.rados.2.smithi175.stdout:3837: finishing write tid 5 to smithi17557870-28 2024-09-11T03:52:18.736 INFO:tasks.rados.rados.2.smithi175.stdout:3837: oid 28 updating version 0 to 1568 2024-09-11T03:52:18.736 INFO:tasks.rados.rados.2.smithi175.stdout:3837: oid 28 version 1568 is already newer than 1567 2024-09-11T03:52:18.736 INFO:tasks.rados.rados.2.smithi175.stdout:3837: oid 28 version 1568 is already newer than 1566 2024-09-11T03:52:18.736 INFO:tasks.rados.rados.2.smithi175.stdout:3837: oid 28 updating version 1568 to 1569 2024-09-11T03:52:18.736 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 1569 (ObjNum 1301 snap 426 seq_num 1301) dirty exists 2024-09-11T03:52:18.736 INFO:tasks.rados.rados.2.smithi175.stdout:3837: left oid 28 (ObjNum 1301 snap 426 seq_num 1301) 2024-09-11T03:52:18.736 INFO:tasks.rados.rados.2.smithi175.stdout:3838: finishing write tid 1 to smithi17557870-3 2024-09-11T03:52:18.736 INFO:tasks.rados.rados.2.smithi175.stdout:3838: finishing write tid 2 to smithi17557870-3 2024-09-11T03:52:18.736 INFO:tasks.rados.rados.2.smithi175.stdout:3839: finishing rollback tid 0 to smithi17557870-30 2024-09-11T03:52:18.736 INFO:tasks.rados.rados.2.smithi175.stdout:3838: finishing write tid 3 to smithi17557870-3 2024-09-11T03:52:18.736 INFO:tasks.rados.rados.2.smithi175.stdout:3842: seq_num 1304 ranges {716328=421059,1840284=509126,3143112=659293} 2024-09-11T03:52:18.741 INFO:tasks.rados.rados.2.smithi175.stdout:3842: writing smithi17557870-20 from 716328 to 1137387 tid 1 2024-09-11T03:52:18.745 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:18.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:18.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:18.747 INFO:tasks.rados.rados.2.smithi175.stdout:3842: writing smithi17557870-20 from 1840284 to 2349410 tid 2 2024-09-11T03:52:18.755 INFO:tasks.rados.rados.2.smithi175.stdout:3842: writing smithi17557870-20 from 3143112 to 3802405 tid 3 2024-09-11T03:52:18.755 INFO:tasks.rados.rados.2.smithi175.stdout:3837: done (5 left) 2024-09-11T03:52:18.755 INFO:tasks.rados.rados.2.smithi175.stdout:3843: snap_remove snap 293 2024-09-11T03:52:18.847 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:18.869 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:18.869 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:18.892 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-11T03:52:18.932 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:18.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:18.994 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:18.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:19.097 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:19.118 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:19.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:19.119 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:19.156 INFO:tasks.rados.rados.2.smithi175.stdout:3843: done (5 left) 2024-09-11T03:52:19.157 INFO:tasks.rados.rados.2.smithi175.stdout:3844: snap_remove snap 400 2024-09-11T03:52:19.220 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:19.245 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:19.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:19.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:19.308 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:52:19.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:19.373 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:19.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:19.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:19.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:19 smithi084 ceph-mon[27254]: 1.c scrub starts 2024-09-11T03:52:19.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:19 smithi084 ceph-mon[27254]: 1.c scrub ok 2024-09-11T03:52:19.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:19 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "86"} : dispatch 2024-09-11T03:52:19.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:19 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "86"} : dispatch 2024-09-11T03:52:19.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:19 smithi121 ceph-mon[30729]: 1.c scrub starts 2024-09-11T03:52:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:19 smithi121 ceph-mon[30729]: 1.c scrub ok 2024-09-11T03:52:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:19 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "86"} : dispatch 2024-09-11T03:52:19.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:19 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "86"} : dispatch 2024-09-11T03:52:19.450 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:19.475 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:19 smithi175 ceph-mon[30894]: 1.c scrub starts 2024-09-11T03:52:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:19 smithi175 ceph-mon[30894]: 1.c scrub ok 2024-09-11T03:52:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:19 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "86"} : dispatch 2024-09-11T03:52:19.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:19 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "86"} : dispatch 2024-09-11T03:52:19.500 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:19.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:19.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:19.602 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:19.625 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:19.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:19.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:19.727 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:19.752 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:19.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:19.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:19.855 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:19.871 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:52:19.871 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:52:19.878 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:19.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:19.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:19.980 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:20.003 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:20.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:20.004 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:20.088 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:20.105 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:20.127 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:20.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:20.128 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:20.149 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:52:20.159 INFO:tasks.rados.rados.2.smithi175.stdout:3844: done (5 left) 2024-09-11T03:52:20.159 INFO:tasks.rados.rados.2.smithi175.stdout:3845: write oid 41 current snap is 426 2024-09-11T03:52:20.159 INFO:tasks.rados.rados.2.smithi175.stdout:3845: seq_num 1305 ranges {733135=496308,1898766=652784,3170535=470565} 2024-09-11T03:52:20.161 INFO:tasks.rados.rados.2.smithi175.stdout:3845: writing smithi17557870-41 from 733135 to 1229443 tid 1 2024-09-11T03:52:20.170 INFO:tasks.rados.rados.2.smithi175.stdout:3845: writing smithi17557870-41 from 1898766 to 2551550 tid 2 2024-09-11T03:52:20.177 INFO:tasks.rados.rados.2.smithi175.stdout:3845: writing smithi17557870-41 from 3170535 to 3641100 tid 3 2024-09-11T03:52:20.177 INFO:tasks.rados.rados.2.smithi175.stdout:3846: snap_create 2024-09-11T03:52:20.177 INFO:tasks.rados.rados.2.smithi175.stdout:3839: finishing rollback tid 1 to smithi17557870-30 2024-09-11T03:52:20.178 INFO:tasks.rados.rados.2.smithi175.stdout:3839: finishing rollback tid 2 to smithi17557870-30 2024-09-11T03:52:20.178 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 1243 (ObjNum 895 snap 284 seq_num 895) dirty exists 2024-09-11T03:52:20.178 INFO:tasks.rados.rados.2.smithi175.stdout:3838: finishing write tid 4 to smithi17557870-3 2024-09-11T03:52:20.178 INFO:tasks.rados.rados.2.smithi175.stdout:3838: finishing write tid 5 to smithi17557870-3 2024-09-11T03:52:20.178 INFO:tasks.rados.rados.2.smithi175.stdout:3838: oid 3 updating version 0 to 1353 2024-09-11T03:52:20.178 INFO:tasks.rados.rados.2.smithi175.stdout:3838: oid 3 updating version 1353 to 1354 2024-09-11T03:52:20.178 INFO:tasks.rados.rados.2.smithi175.stdout:3838: oid 3 version 1354 is already newer than 1352 2024-09-11T03:52:20.178 INFO:tasks.rados.rados.2.smithi175.stdout:3838: oid 3 version 1354 is already newer than 1351 2024-09-11T03:52:20.178 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 1354 (ObjNum 1302 snap 426 seq_num 1302) dirty exists 2024-09-11T03:52:20.178 INFO:tasks.rados.rados.2.smithi175.stdout:3838: left oid 3 (ObjNum 1302 snap 426 seq_num 1302) 2024-09-11T03:52:20.178 INFO:tasks.rados.rados.2.smithi175.stdout:3841: finishing write tid 1 to smithi17557870-15 2024-09-11T03:52:20.178 INFO:tasks.rados.rados.2.smithi175.stdout:3841: finishing write tid 2 to smithi17557870-15 2024-09-11T03:52:20.179 INFO:tasks.rados.rados.2.smithi175.stdout:3841: finishing write tid 3 to smithi17557870-15 2024-09-11T03:52:20.179 INFO:tasks.rados.rados.2.smithi175.stdout:3841: finishing write tid 4 to smithi17557870-15 2024-09-11T03:52:20.179 INFO:tasks.rados.rados.2.smithi175.stdout:3841: finishing write tid 5 to smithi17557870-15 2024-09-11T03:52:20.179 INFO:tasks.rados.rados.2.smithi175.stdout:3841: oid 15 updating version 0 to 1064 2024-09-11T03:52:20.179 INFO:tasks.rados.rados.2.smithi175.stdout:3841: oid 15 version 1064 is already newer than 1062 2024-09-11T03:52:20.179 INFO:tasks.rados.rados.2.smithi175.stdout:3841: oid 15 version 1064 is already newer than 1063 2024-09-11T03:52:20.179 INFO:tasks.rados.rados.2.smithi175.stdout:3841: oid 15 updating version 1064 to 1065 2024-09-11T03:52:20.179 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 1065 (ObjNum 1303 snap 426 seq_num 1303) dirty exists 2024-09-11T03:52:20.179 INFO:tasks.rados.rados.2.smithi175.stdout:3841: left oid 15 (ObjNum 1303 snap 426 seq_num 1303) 2024-09-11T03:52:20.179 INFO:tasks.rados.rados.2.smithi175.stdout:3842: finishing write tid 1 to smithi17557870-20 2024-09-11T03:52:20.179 INFO:tasks.rados.rados.2.smithi175.stdout:3842: finishing write tid 2 to smithi17557870-20 2024-09-11T03:52:20.179 INFO:tasks.rados.rados.2.smithi175.stdout:3842: finishing write tid 3 to smithi17557870-20 2024-09-11T03:52:20.179 INFO:tasks.rados.rados.2.smithi175.stdout:3842: finishing write tid 4 to smithi17557870-20 2024-09-11T03:52:20.179 INFO:tasks.rados.rados.2.smithi175.stdout:3842: finishing write tid 5 to smithi17557870-20 2024-09-11T03:52:20.180 INFO:tasks.rados.rados.2.smithi175.stdout:3842: oid 20 updating version 0 to 1073 2024-09-11T03:52:20.180 INFO:tasks.rados.rados.2.smithi175.stdout:3842: oid 20 version 1073 is already newer than 1072 2024-09-11T03:52:20.180 INFO:tasks.rados.rados.2.smithi175.stdout:3842: oid 20 updating version 1073 to 1075 2024-09-11T03:52:20.180 INFO:tasks.rados.rados.2.smithi175.stdout:3842: oid 20 version 1075 is already newer than 1074 2024-09-11T03:52:20.180 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 1075 (ObjNum 1304 snap 426 seq_num 1304) dirty exists 2024-09-11T03:52:20.180 INFO:tasks.rados.rados.2.smithi175.stdout:3842: left oid 20 (ObjNum 1304 snap 426 seq_num 1304) 2024-09-11T03:52:20.180 INFO:tasks.rados.rados.2.smithi175.stdout:3845: finishing write tid 1 to smithi17557870-41 2024-09-11T03:52:20.181 INFO:tasks.rados.rados.2.smithi175.stdout:3845: finishing write tid 2 to smithi17557870-41 2024-09-11T03:52:20.183 INFO:tasks.rados.rados.2.smithi175.stdout:3845: finishing write tid 3 to smithi17557870-41 2024-09-11T03:52:20.183 INFO:tasks.rados.rados.2.smithi175.stdout:3845: finishing write tid 4 to smithi17557870-41 2024-09-11T03:52:20.183 INFO:tasks.rados.rados.2.smithi175.stdout:3845: finishing write tid 5 to smithi17557870-41 2024-09-11T03:52:20.183 INFO:tasks.rados.rados.2.smithi175.stdout:3845: oid 41 updating version 0 to 1036 2024-09-11T03:52:20.183 INFO:tasks.rados.rados.2.smithi175.stdout:3845: oid 41 version 1036 is already newer than 1034 2024-09-11T03:52:20.183 INFO:tasks.rados.rados.2.smithi175.stdout:3845: oid 41 updating version 1036 to 1037 2024-09-11T03:52:20.183 INFO:tasks.rados.rados.2.smithi175.stdout:3845: oid 41 version 1037 is already newer than 1035 2024-09-11T03:52:20.184 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 1037 (ObjNum 1305 snap 426 seq_num 1305) dirty exists 2024-09-11T03:52:20.184 INFO:tasks.rados.rados.2.smithi175.stdout:3845: left oid 41 (ObjNum 1305 snap 426 seq_num 1305) 2024-09-11T03:52:20.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:20.253 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:20.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:20.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:20.355 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:20.379 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:20.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:20.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:20 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:52:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:20 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:52:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:20 smithi121 ceph-mon[30729]: pgmap v1536: 107 pgs: 107 active+clean; 761 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 12 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2024-09-11T03:52:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:20 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "86"}]': finished 2024-09-11T03:52:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:20 smithi121 ceph-mon[30729]: osdmap e921: 12 total, 12 up, 11 in 2024-09-11T03:52:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:20 smithi121 ceph-mon[30729]: 2.39 scrub starts 2024-09-11T03:52:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:20 smithi121 ceph-mon[30729]: 2.39 scrub ok 2024-09-11T03:52:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:20 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1716627944' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:52:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:20 smithi121 ceph-mon[30729]: 2.2d scrub starts 2024-09-11T03:52:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:20 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:52:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:20 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:52:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:20 smithi175 ceph-mon[30894]: pgmap v1536: 107 pgs: 107 active+clean; 761 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 12 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2024-09-11T03:52:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:20 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "86"}]': finished 2024-09-11T03:52:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:20 smithi175 ceph-mon[30894]: osdmap e921: 12 total, 12 up, 11 in 2024-09-11T03:52:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:20 smithi175 ceph-mon[30894]: 2.39 scrub starts 2024-09-11T03:52:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:20 smithi175 ceph-mon[30894]: 2.39 scrub ok 2024-09-11T03:52:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:20 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1716627944' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:52:20.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:20 smithi175 ceph-mon[30894]: 2.2d scrub starts 2024-09-11T03:52:20.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:20.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:20.504 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:20.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:20.605 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:20.630 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:20.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:20.631 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:20 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:52:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:20 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:52:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:20 smithi084 ceph-mon[27254]: pgmap v1536: 107 pgs: 107 active+clean; 761 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 12 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2024-09-11T03:52:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:20 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "86"}]': finished 2024-09-11T03:52:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:20 smithi084 ceph-mon[27254]: osdmap e921: 12 total, 12 up, 11 in 2024-09-11T03:52:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:20 smithi084 ceph-mon[27254]: 2.39 scrub starts 2024-09-11T03:52:20.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:20 smithi084 ceph-mon[27254]: 2.39 scrub ok 2024-09-11T03:52:20.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:20 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1716627944' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:52:20.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:20 smithi084 ceph-mon[27254]: 2.2d scrub starts 2024-09-11T03:52:20.730 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-11T03:52:20.732 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:20.752 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:20.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:20.855 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:20.880 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:20.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:20.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:20.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:21.007 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:21.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:21.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:21.109 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:21.135 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:21.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:21.136 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:21.169 INFO:tasks.rados.rados.2.smithi175.stdout:3838: done (6 left) 2024-09-11T03:52:21.169 INFO:tasks.rados.rados.2.smithi175.stdout:3839: done (5 left) 2024-09-11T03:52:21.169 INFO:tasks.rados.rados.2.smithi175.stdout:3840: done (4 left) 2024-09-11T03:52:21.169 INFO:tasks.rados.rados.2.smithi175.stdout:3841: done (3 left) 2024-09-11T03:52:21.169 INFO:tasks.rados.rados.2.smithi175.stdout:3842: done (2 left) 2024-09-11T03:52:21.169 INFO:tasks.rados.rados.2.smithi175.stdout:3845: done (1 left) 2024-09-11T03:52:21.170 INFO:tasks.rados.rados.2.smithi175.stdout:3846: done (0 left) 2024-09-11T03:52:21.170 INFO:tasks.rados.rados.2.smithi175.stdout:3847: read oid 38 snap -1 2024-09-11T03:52:21.170 INFO:tasks.rados.rados.2.smithi175.stdout:3847: expect (ObjNum 970 snap 311 seq_num 970) 2024-09-11T03:52:21.170 INFO:tasks.rados.rados.2.smithi175.stdout:3848: copy_from oid 20 from oid 30 current snap is 427 2024-09-11T03:52:21.170 INFO:tasks.rados.rados.2.smithi175.stdout:3849: rollback oid 25 current snap is 427 2024-09-11T03:52:21.170 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 25 to 374 2024-09-11T03:52:21.170 INFO:tasks.rados.rados.2.smithi175.stdout:3850: rollback oid 49 current snap is 427 2024-09-11T03:52:21.170 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 49 to 364 2024-09-11T03:52:21.170 INFO:tasks.rados.rados.2.smithi175.stdout:3851: read oid 39 snap -1 2024-09-11T03:52:21.170 INFO:tasks.rados.rados.2.smithi175.stdout:3851: expect (ObjNum 1290 snap 424 seq_num 1290) 2024-09-11T03:52:21.170 INFO:tasks.rados.rados.2.smithi175.stdout:3852: snap_create 2024-09-11T03:52:21.176 INFO:tasks.rados.rados.2.smithi175.stdout:3849: finishing rollback tid 0 to smithi17557870-25 2024-09-11T03:52:21.177 INFO:tasks.rados.rados.2.smithi175.stdout:3849: finishing rollback tid 1 to smithi17557870-25 2024-09-11T03:52:21.177 INFO:tasks.rados.rados.2.smithi175.stdout:3849: finishing rollback tid 2 to smithi17557870-25 2024-09-11T03:52:21.177 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 1014 (ObjNum 1148 snap 371 seq_num 1148) dirty exists 2024-09-11T03:52:21.183 INFO:tasks.rados.rados.2.smithi175.stdout:3847: expect (ObjNum 970 snap 311 seq_num 970) 2024-09-11T03:52:21.196 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:52:21.196 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:52:21.213 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:52:21.213 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:52:21.229 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:52:21.229 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:52:21.229 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:21 smithi084 ceph-mon[27254]: 2.2d scrub ok 2024-09-11T03:52:21.229 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:21 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:52:21.229 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:21 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:52:21.229 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:21 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:21.230 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:21 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1716627944' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:52:21.230 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:21 smithi084 ceph-mon[27254]: osdmap e922: 12 total, 12 up, 11 in 2024-09-11T03:52:21.230 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:21 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:52:21.230 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:21 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:52:21.230 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:21 smithi084 ceph-mon[27254]: 2.f scrub starts 2024-09-11T03:52:21.230 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:21 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "86"} : dispatch 2024-09-11T03:52:21.230 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:21 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "86"} : dispatch 2024-09-11T03:52:21.230 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:21 smithi084 ceph-mon[27254]: 2.11 scrub starts 2024-09-11T03:52:21.230 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:21 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:52:21.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:21.244 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:52:21.245 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:52:21.261 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:52:21.261 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:52:21.262 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:21.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:21.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:21.275 INFO:tasks.rados.rados.2.smithi175.stdout:3851: expect (ObjNum 1290 snap 424 seq_num 1290) 2024-09-11T03:52:21.278 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:52:21.278 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:52:21.294 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:52:21.295 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:52:21.312 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:52:21.312 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:52:21.329 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:52:21.329 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:52:21.345 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:52:21.345 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:52:21.362 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:52:21.362 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:52:21.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:21.380 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:52:21.380 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:52:21.386 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:21.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:21.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:21 smithi121 ceph-mon[30729]: 2.2d scrub ok 2024-09-11T03:52:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:21 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:52:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:21 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:52:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:21 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:21 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1716627944' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:52:21.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:21 smithi121 ceph-mon[30729]: osdmap e922: 12 total, 12 up, 11 in 2024-09-11T03:52:21.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:21 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:52:21.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:21 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:52:21.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:21 smithi121 ceph-mon[30729]: 2.f scrub starts 2024-09-11T03:52:21.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:21 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "86"} : dispatch 2024-09-11T03:52:21.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:21 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "86"} : dispatch 2024-09-11T03:52:21.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:21 smithi121 ceph-mon[30729]: 2.11 scrub starts 2024-09-11T03:52:21.424 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:21 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:52:21.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:21 smithi175 ceph-mon[30894]: 2.2d scrub ok 2024-09-11T03:52:21.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:21 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:52:21.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:21 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:52:21.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:21 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:21.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:21 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1716627944' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:52:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:21 smithi175 ceph-mon[30894]: osdmap e922: 12 total, 12 up, 11 in 2024-09-11T03:52:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:21 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:52:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:21 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:52:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:21 smithi175 ceph-mon[30894]: 2.f scrub starts 2024-09-11T03:52:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:21 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "86"} : dispatch 2024-09-11T03:52:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:21 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "86"} : dispatch 2024-09-11T03:52:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:21 smithi175 ceph-mon[30894]: 2.11 scrub starts 2024-09-11T03:52:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:21 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:52:21.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:21.510 INFO:tasks.rados.rados.2.smithi175.stdout:3848: finishing copy_from racing read to smithi17557870-20 2024-09-11T03:52:21.511 INFO:tasks.rados.rados.2.smithi175.stdout:3848: finishing copy_from to smithi17557870-20 2024-09-11T03:52:21.511 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 20 v 1243 (ObjNum 895 snap 284 seq_num 895) dirty exists 2024-09-11T03:52:21.512 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:21.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:21.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:21.614 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:21.635 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:21.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:21.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:21.761 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:21.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:21.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:21.884 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:21.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:21.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:21.986 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:22.010 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:22.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:22.068 INFO:tasks.rados.rados.2.smithi175.stdout:3847: done (5 left) 2024-09-11T03:52:22.068 INFO:tasks.rados.rados.2.smithi175.stdout:3848: done (4 left) 2024-09-11T03:52:22.068 INFO:tasks.rados.rados.2.smithi175.stdout:3849: done (3 left) 2024-09-11T03:52:22.068 INFO:tasks.rados.rados.2.smithi175.stdout:3851: done (2 left) 2024-09-11T03:52:22.068 INFO:tasks.rados.rados.2.smithi175.stdout:3852: done (1 left) 2024-09-11T03:52:22.068 INFO:tasks.rados.rados.2.smithi175.stdout:3853: read oid 47 snap 417 2024-09-11T03:52:22.068 INFO:tasks.rados.rados.2.smithi175.stdout:3853: expect (ObjNum 1188 snap 388 seq_num 1188) 2024-09-11T03:52:22.068 INFO:tasks.rados.rados.2.smithi175.stdout:3854: write (excl) oid 33 current snap is 428 2024-09-11T03:52:22.068 INFO:tasks.rados.rados.2.smithi175.stdout:3854: seq_num 1306 ranges {749942=571557,1840895=796442,3365252=584690} 2024-09-11T03:52:22.075 INFO:tasks.rados.rados.2.smithi175.stdout:3854: writing smithi17557870-33 from 749942 to 1321499 tid 1 2024-09-11T03:52:22.083 INFO:tasks.rados.rados.2.smithi175.stdout:3854: writing smithi17557870-33 from 1840895 to 2637337 tid 2 2024-09-11T03:52:22.091 INFO:tasks.rados.rados.2.smithi175.stdout:3854: writing smithi17557870-33 from 3365252 to 3949942 tid 3 2024-09-11T03:52:22.091 INFO:tasks.rados.rados.2.smithi175.stdout:3855: snap_remove snap 425 2024-09-11T03:52:22.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:22.134 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:22.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:22.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:22.237 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:22.259 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:22.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:22.260 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:22.361 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:22.383 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:22.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:22.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:22 smithi084 ceph-mon[27254]: 2.12 scrub starts 2024-09-11T03:52:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:22 smithi084 ceph-mon[27254]: pgmap v1539: 117 pgs: 10 unknown, 107 active+clean; 761 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 6.7 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-11T03:52:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:22 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "86"}]': finished 2024-09-11T03:52:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:22 smithi084 ceph-mon[27254]: osdmap e923: 12 total, 12 up, 11 in 2024-09-11T03:52:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:22 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2721262278' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:22 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:52:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:22 smithi084 ceph-mon[27254]: 2.13 scrub starts 2024-09-11T03:52:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:22 smithi084 ceph-mon[27254]: 2.10 scrub starts 2024-09-11T03:52:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:22 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:52:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:22 smithi084 ceph-mon[27254]: osdmap e924: 12 total, 12 up, 11 in 2024-09-11T03:52:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:22 smithi121 ceph-mon[30729]: 2.12 scrub starts 2024-09-11T03:52:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:22 smithi121 ceph-mon[30729]: pgmap v1539: 117 pgs: 10 unknown, 107 active+clean; 761 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 6.7 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-11T03:52:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:22 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "86"}]': finished 2024-09-11T03:52:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:22 smithi121 ceph-mon[30729]: osdmap e923: 12 total, 12 up, 11 in 2024-09-11T03:52:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:22 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2721262278' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:22 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:52:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:22 smithi121 ceph-mon[30729]: 2.13 scrub starts 2024-09-11T03:52:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:22 smithi121 ceph-mon[30729]: 2.10 scrub starts 2024-09-11T03:52:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:22 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:52:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:22 smithi121 ceph-mon[30729]: osdmap e924: 12 total, 12 up, 11 in 2024-09-11T03:52:22.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:22 smithi175 ceph-mon[30894]: 2.12 scrub starts 2024-09-11T03:52:22.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:22 smithi175 ceph-mon[30894]: pgmap v1539: 117 pgs: 10 unknown, 107 active+clean; 761 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 6.7 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-11T03:52:22.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:22 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "86"}]': finished 2024-09-11T03:52:22.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:22 smithi175 ceph-mon[30894]: osdmap e923: 12 total, 12 up, 11 in 2024-09-11T03:52:22.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:22 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2721262278' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:22 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:52:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:22 smithi175 ceph-mon[30894]: 2.13 scrub starts 2024-09-11T03:52:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:22 smithi175 ceph-mon[30894]: 2.10 scrub starts 2024-09-11T03:52:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:22 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:52:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:22 smithi175 ceph-mon[30894]: osdmap e924: 12 total, 12 up, 11 in 2024-09-11T03:52:22.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:22.511 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:22.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:22.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:22.553 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:22.612 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:22.634 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:22.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:22.635 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:22.736 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:22.758 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:22.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:22.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:22.860 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:22.885 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:22.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:22.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:22.987 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:23.010 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:23.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:23.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:23.015 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:52:23.015 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:52:23.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:52:23.015 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:52:23.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.015 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:3778 2.9 2:924204df:::smithi17557870-50:head [copy-from ver 430] snapc 9d=[9d,9c,9b,9a,98,97,90,8a] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-11T03:52:23.015 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:43:42.949380+0000", 2024-09-11T03:52:23.015 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 520.06483497099998, 2024-09-11T03:52:23.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018153696, 2024-09-11T03:52:23.016 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.016 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.016 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:23.016 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.016 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.016 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.016 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3778 2024-09-11T03:52:23.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.016 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.017 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.017 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949380+0000", 2024-09-11T03:52:23.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.017 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.017 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949380+0000", 2024-09-11T03:52:23.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.017 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.017 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949382+0000", 2024-09-11T03:52:23.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.652e-06 2024-09-11T03:52:23.018 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.018 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.018 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.018 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949389+0000", 2024-09-11T03:52:23.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9569999999999996e-06 2024-09-11T03:52:23.018 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.018 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.018 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.018 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949405+0000", 2024-09-11T03:52:23.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6212e-05 2024-09-11T03:52:23.018 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.018 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.018 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949412+0000", 2024-09-11T03:52:23.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2099999999999996e-06 2024-09-11T03:52:23.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949483+0000", 2024-09-11T03:52:23.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0253000000000007e-05 2024-09-11T03:52:23.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949603+0000", 2024-09-11T03:52:23.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120538 2024-09-11T03:52:23.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:52:23.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.959706+0000", 2024-09-11T03:52:23.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010102629 2024-09-11T03:52:23.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.020 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:23.020 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.965469+0000", 2024-09-11T03:52:23.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005762784 2024-09-11T03:52:23.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.966495+0000", 2024-09-11T03:52:23.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001026622 2024-09-11T03:52:23.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.967289+0000", 2024-09-11T03:52:23.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00079390799999999996 2024-09-11T03:52:23.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:23.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.967429+0000", 2024-09-11T03:52:23.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013990899999999999 2024-09-11T03:52:23.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.022 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.022 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.967534+0000", 2024-09-11T03:52:23.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105022 2024-09-11T03:52:23.022 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.022 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.022 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.022 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:3779 2.9 2:924204df:::smithi17557870-50:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e405)", 2024-09-11T03:52:23.022 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:43:42.949428+0000", 2024-09-11T03:52:23.022 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 520.06478691300003, 2024-09-11T03:52:23.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014699339000000001, 2024-09-11T03:52:23.023 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.023 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.023 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:23.023 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.023 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.023 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.023 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3779 2024-09-11T03:52:23.023 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.023 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.023 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.023 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.023 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949428+0000", 2024-09-11T03:52:23.023 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949428+0000", 2024-09-11T03:52:23.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949429+0000", 2024-09-11T03:52:23.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5699999999999998e-07 2024-09-11T03:52:23.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949431+0000", 2024-09-11T03:52:23.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7940000000000001e-06 2024-09-11T03:52:23.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949436+0000", 2024-09-11T03:52:23.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.682e-06 2024-09-11T03:52:23.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949440+0000", 2024-09-11T03:52:23.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8249999999999998e-06 2024-09-11T03:52:23.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949740+0000", 2024-09-11T03:52:23.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00030021799999999999 2024-09-11T03:52:23.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.027 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:23.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.949779+0000", 2024-09-11T03:52:23.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8679e-05 2024-09-11T03:52:23.027 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.027 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.964002+0000", 2024-09-11T03:52:23.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014222692 2024-09-11T03:52:23.027 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.027 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.027 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.964062+0000", 2024-09-11T03:52:23.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0118000000000003e-05 2024-09-11T03:52:23.028 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.028 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.028 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.028 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:43:42.964128+0000", 2024-09-11T03:52:23.028 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5974000000000002e-05 2024-09-11T03:52:23.028 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.028 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.028 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.028 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.028 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.028 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5359 2.4 2:220bb244:::smithi17557870-34:head [copy-from ver 504] snapc ef=[ef,ee,ed,ec,ea,e9,e0] ondisk+write+known_if_redirected+supports_pool_eio e572)", 2024-09-11T03:52:23.028 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:29.969878+0000", 2024-09-11T03:52:23.028 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 353.04433773199997, 2024-09-11T03:52:23.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0298148170000001, 2024-09-11T03:52:23.029 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.029 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.029 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:23.029 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.029 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.029 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.029 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5359 2024-09-11T03:52:23.029 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.029 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.029 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.029 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.029 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:29.969878+0000", 2024-09-11T03:52:23.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:29.969878+0000", 2024-09-11T03:52:23.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:29.969879+0000", 2024-09-11T03:52:23.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3829999999999999e-06 2024-09-11T03:52:23.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:29.969883+0000", 2024-09-11T03:52:23.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4329999999999997e-06 2024-09-11T03:52:23.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:29.969892+0000", 2024-09-11T03:52:23.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9069999999999996e-06 2024-09-11T03:52:23.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:29.969902+0000", 2024-09-11T03:52:23.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7049999999999994e-06 2024-09-11T03:52:23.033 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.033 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.033 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.033 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:29.969946+0000", 2024-09-11T03:52:23.033 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3770999999999998e-05 2024-09-11T03:52:23.033 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.033 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.033 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.033 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:29.970043+0000", 2024-09-11T03:52:23.033 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7359000000000001e-05 2024-09-11T03:52:23.033 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.033 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.033 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,10", 2024-09-11T03:52:23.034 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.991044+0000", 2024-09-11T03:52:23.034 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.021000594 2024-09-11T03:52:23.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.034 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:23.034 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.999051+0000", 2024-09-11T03:52:23.034 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0080076369999999997 2024-09-11T03:52:23.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.034 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.034 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.999477+0000", 2024-09-11T03:52:23.034 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00042576400000000003 2024-09-11T03:52:23.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.035 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.999569+0000", 2024-09-11T03:52:23.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1766000000000002e-05 2024-09-11T03:52:23.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:23.035 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.999638+0000", 2024-09-11T03:52:23.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8672999999999998e-05 2024-09-11T03:52:23.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.999692+0000", 2024-09-11T03:52:23.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4824999999999998e-05 2024-09-11T03:52:23.036 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.036 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.036 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.036 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5360 2.4 2:220bb244:::smithi17557870-34:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e572)", 2024-09-11T03:52:23.036 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:29.969916+0000", 2024-09-11T03:52:23.036 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 353.04429964299999, 2024-09-11T03:52:23.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.026477903, 2024-09-11T03:52:23.036 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.037 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.037 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:23.037 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.037 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.037 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.037 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5360 2024-09-11T03:52:23.037 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.037 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.037 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:29.969916+0000", 2024-09-11T03:52:23.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.038 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.038 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:29.969916+0000", 2024-09-11T03:52:23.038 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.038 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.038 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:29.969916+0000", 2024-09-11T03:52:23.039 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8599999999999998e-07 2024-09-11T03:52:23.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.039 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.039 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.039 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:29.969918+0000", 2024-09-11T03:52:23.039 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.198e-06 2024-09-11T03:52:23.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.039 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.039 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.040 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:29.969926+0000", 2024-09-11T03:52:23.040 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.131e-06 2024-09-11T03:52:23.040 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.040 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.040 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.040 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:29.969932+0000", 2024-09-11T03:52:23.040 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5579999999999997e-06 2024-09-11T03:52:23.040 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.040 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.040 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.040 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:29.970186+0000", 2024-09-11T03:52:23.040 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025349699999999999 2024-09-11T03:52:23.041 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.041 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:23.041 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:29.970240+0000", 2024-09-11T03:52:23.041 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4874999999999999e-05 2024-09-11T03:52:23.041 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.041 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.041 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.996145+0000", 2024-09-11T03:52:23.041 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0259049609999999 2024-09-11T03:52:23.041 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.041 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.042 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.996241+0000", 2024-09-11T03:52:23.042 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5563000000000006e-05 2024-09-11T03:52:23.042 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.042 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.042 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.042 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:30.996394+0000", 2024-09-11T03:52:23.042 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015263399999999999 2024-09-11T03:52:23.042 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.042 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.042 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.042 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.042 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.042 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:6548 2.9 2:9718db5a:::smithi17557870-3:head [copy-from ver 950] snapc 11f=[11f,11e,11d,11a,119,114] ondisk+write+known_if_redirected+supports_pool_eio e669)", 2024-09-11T03:52:23.043 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:07.222002+0000", 2024-09-11T03:52:23.043 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 255.792213729, 2024-09-11T03:52:23.043 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018370286, 2024-09-11T03:52:23.043 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.043 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.043 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:23.043 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.043 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.043 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.043 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 6548 2024-09-11T03:52:23.043 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.043 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.044 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.044 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.044 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.222002+0000", 2024-09-11T03:52:23.044 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.044 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.044 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.044 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.044 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.222002+0000", 2024-09-11T03:52:23.044 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.044 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.044 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.044 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.045 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.222003+0000", 2024-09-11T03:52:23.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.128e-06 2024-09-11T03:52:23.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.045 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.222007+0000", 2024-09-11T03:52:23.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4580000000000003e-06 2024-09-11T03:52:23.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.045 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.222017+0000", 2024-09-11T03:52:23.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5529999999999998e-06 2024-09-11T03:52:23.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.046 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.046 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.222023+0000", 2024-09-11T03:52:23.046 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0349999999999998e-06 2024-09-11T03:52:23.046 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.046 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.046 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.046 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.222073+0000", 2024-09-11T03:52:23.046 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9874999999999999e-05 2024-09-11T03:52:23.046 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.046 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.046 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.046 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.222160+0000", 2024-09-11T03:52:23.046 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7316000000000002e-05 2024-09-11T03:52:23.046 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.047 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.047 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:52:23.047 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.228921+0000", 2024-09-11T03:52:23.047 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067610739999999997 2024-09-11T03:52:23.047 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.047 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.047 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:23.047 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.234244+0000", 2024-09-11T03:52:23.047 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053228049999999999 2024-09-11T03:52:23.047 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.047 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.047 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.048 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.235457+0000", 2024-09-11T03:52:23.048 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012134890000000001 2024-09-11T03:52:23.048 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.048 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.048 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.048 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.240282+0000", 2024-09-11T03:52:23.048 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048244969999999996 2024-09-11T03:52:23.048 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.048 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.048 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:23.048 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.240323+0000", 2024-09-11T03:52:23.048 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0815999999999997e-05 2024-09-11T03:52:23.048 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.049 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.049 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.049 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.240372+0000", 2024-09-11T03:52:23.049 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9240000000000003e-05 2024-09-11T03:52:23.049 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.049 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.049 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.049 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.049 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.049 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:6813 2.9 2:95fa802a:::smithi17557870-22:head [copy-from ver 930] snapc 12b=[12b,12a,127,126,123,120,11f,11e,11d,119,114] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-09-11T03:52:23.049 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:26.042303+0000", 2024-09-11T03:52:23.049 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 236.97191243200001, 2024-09-11T03:52:23.049 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024373176, 2024-09-11T03:52:23.050 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.050 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.050 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:23.050 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.050 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.050 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.050 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 6813 2024-09-11T03:52:23.050 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.050 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.050 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.050 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.050 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.042303+0000", 2024-09-11T03:52:23.050 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.051 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.051 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.042303+0000", 2024-09-11T03:52:23.051 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.051 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.051 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.042305+0000", 2024-09-11T03:52:23.051 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.686e-06 2024-09-11T03:52:23.051 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.051 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.052 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.042312+0000", 2024-09-11T03:52:23.052 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.374e-06 2024-09-11T03:52:23.052 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.052 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.052 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.052 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.042329+0000", 2024-09-11T03:52:23.052 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6866000000000001e-05 2024-09-11T03:52:23.052 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.052 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.052 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.052 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.042338+0000", 2024-09-11T03:52:23.052 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6670000000000002e-06 2024-09-11T03:52:23.053 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.053 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.053 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.053 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.042399+0000", 2024-09-11T03:52:23.053 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1765000000000002e-05 2024-09-11T03:52:23.053 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.053 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.053 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:52:23.053 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.042407+0000", 2024-09-11T03:52:23.053 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5549999999999997e-06 2024-09-11T03:52:23.053 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.053 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.054 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.054 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.048232+0000", 2024-09-11T03:52:23.054 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058252879999999996 2024-09-11T03:52:23.054 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.054 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.054 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.054 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.048411+0000", 2024-09-11T03:52:23.054 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017894100000000001 2024-09-11T03:52:23.054 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.054 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.054 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:52:23.054 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.059295+0000", 2024-09-11T03:52:23.054 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010884329999999999 2024-09-11T03:52:23.055 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.055 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.055 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.055 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.065976+0000", 2024-09-11T03:52:23.055 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.006681078 2024-09-11T03:52:23.055 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.055 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.055 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:23.055 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.066305+0000", 2024-09-11T03:52:23.055 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000328571 2024-09-11T03:52:23.055 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.055 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.055 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.056 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.066604+0000", 2024-09-11T03:52:23.056 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029934599999999998 2024-09-11T03:52:23.056 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.056 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.056 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:23.056 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.066643+0000", 2024-09-11T03:52:23.056 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8865000000000001e-05 2024-09-11T03:52:23.056 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.056 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.056 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.056 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.066676+0000", 2024-09-11T03:52:23.056 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2843999999999999e-05 2024-09-11T03:52:23.056 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.057 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.057 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.057 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.057 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.057 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:6814 2.9 2:95fa802a:::smithi17557870-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e688)", 2024-09-11T03:52:23.057 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:26.042359+0000", 2024-09-11T03:52:23.057 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 236.97185605600001, 2024-09-11T03:52:23.057 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021148296, 2024-09-11T03:52:23.057 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.057 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.057 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:23.057 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.057 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.057 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.058 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 6814 2024-09-11T03:52:23.058 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.058 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.058 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.058 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.058 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.042359+0000", 2024-09-11T03:52:23.058 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.058 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.058 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.058 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.058 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.042359+0000", 2024-09-11T03:52:23.058 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.059 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.059 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.059 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.059 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.042360+0000", 2024-09-11T03:52:23.059 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7700000000000004e-07 2024-09-11T03:52:23.059 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.059 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.059 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.059 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.042363+0000", 2024-09-11T03:52:23.059 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0900000000000001e-06 2024-09-11T03:52:23.059 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.059 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.059 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.059 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.042372+0000", 2024-09-11T03:52:23.060 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0459999999999994e-06 2024-09-11T03:52:23.060 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.060 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.060 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.060 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.042378+0000", 2024-09-11T03:52:23.060 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7470000000000002e-06 2024-09-11T03:52:23.060 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.060 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.060 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.060 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.042478+0000", 2024-09-11T03:52:23.060 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000100009 2024-09-11T03:52:23.060 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.060 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.061 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:23.061 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.042483+0000", 2024-09-11T03:52:23.061 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4859999999999997e-06 2024-09-11T03:52:23.061 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.061 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.061 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.061 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.048566+0000", 2024-09-11T03:52:23.061 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00608268 2024-09-11T03:52:23.061 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.061 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.061 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:23.061 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.048600+0000", 2024-09-11T03:52:23.061 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3806999999999997e-05 2024-09-11T03:52:23.062 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.062 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.062 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.062 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.063346+0000", 2024-09-11T03:52:23.062 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014746485 2024-09-11T03:52:23.062 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.062 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.062 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.062 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.063422+0000", 2024-09-11T03:52:23.062 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5640000000000001e-05 2024-09-11T03:52:23.062 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.062 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.062 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.063 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:26.063508+0000", 2024-09-11T03:52:23.063 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5729000000000005e-05 2024-09-11T03:52:23.063 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.063 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.063 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.063 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.063 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.063 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:6914 2.4 2:220bb244:::smithi17557870-34:head [copy-from ver 898] snapc 132=[132,130,12f,12e,12d,12c,12b,12a,127,126,123,120,11f,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e697)", 2024-09-11T03:52:23.063 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:35.020896+0000", 2024-09-11T03:52:23.063 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 227.993319308, 2024-09-11T03:52:23.063 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022918754999999999, 2024-09-11T03:52:23.063 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.063 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.064 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:23.064 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.064 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.064 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.064 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 6914 2024-09-11T03:52:23.064 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.064 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.064 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.064 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.064 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.020896+0000", 2024-09-11T03:52:23.064 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.064 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.064 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.065 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.065 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.020896+0000", 2024-09-11T03:52:23.065 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.065 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.065 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.065 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.065 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.020896+0000", 2024-09-11T03:52:23.065 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4600000000000001e-07 2024-09-11T03:52:23.065 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.065 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.065 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.065 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.020898+0000", 2024-09-11T03:52:23.065 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8619999999999999e-06 2024-09-11T03:52:23.066 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.066 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.066 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.066 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.020902+0000", 2024-09-11T03:52:23.066 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5609999999999999e-06 2024-09-11T03:52:23.066 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.066 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.066 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.066 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.020904+0000", 2024-09-11T03:52:23.066 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.531e-06 2024-09-11T03:52:23.066 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.066 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.066 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.067 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.020949+0000", 2024-09-11T03:52:23.067 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4530000000000002e-05 2024-09-11T03:52:23.067 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.067 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.067 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:52:23.067 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.020955+0000", 2024-09-11T03:52:23.067 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0889999999999996e-06 2024-09-11T03:52:23.067 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.067 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.067 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.067 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.022573+0000", 2024-09-11T03:52:23.067 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016183180000000001 2024-09-11T03:52:23.067 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.068 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.068 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.068 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.022636+0000", 2024-09-11T03:52:23.068 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3229000000000001e-05 2024-09-11T03:52:23.068 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.068 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.068 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,10", 2024-09-11T03:52:23.068 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.033309+0000", 2024-09-11T03:52:23.068 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010672506999999999 2024-09-11T03:52:23.068 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.068 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.068 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:23.069 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.039356+0000", 2024-09-11T03:52:23.069 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060471250000000004 2024-09-11T03:52:23.069 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.069 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.069 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.069 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.039422+0000", 2024-09-11T03:52:23.069 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6223000000000006e-05 2024-09-11T03:52:23.069 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.069 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.069 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.069 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.043704+0000", 2024-09-11T03:52:23.069 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042822320000000004 2024-09-11T03:52:23.069 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.070 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.070 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:23.070 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.043782+0000", 2024-09-11T03:52:23.070 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7645000000000006e-05 2024-09-11T03:52:23.070 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.070 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.070 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.070 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.043815+0000", 2024-09-11T03:52:23.070 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2657000000000003e-05 2024-09-11T03:52:23.070 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.070 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.070 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.071 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.071 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.071 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:6915 2.4 2:220bb244:::smithi17557870-34:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e697)", 2024-09-11T03:52:23.071 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:35.020909+0000", 2024-09-11T03:52:23.071 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 227.993305945, 2024-09-11T03:52:23.071 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017007892, 2024-09-11T03:52:23.071 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.071 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.071 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:23.071 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.071 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.071 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.071 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 6915 2024-09-11T03:52:23.072 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.072 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.072 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.072 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.072 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.020909+0000", 2024-09-11T03:52:23.072 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.072 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.072 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.072 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.072 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.020909+0000", 2024-09-11T03:52:23.072 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.072 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.073 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.073 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.073 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.020910+0000", 2024-09-11T03:52:23.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4600000000000001e-07 2024-09-11T03:52:23.073 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.073 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.073 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.073 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.020911+0000", 2024-09-11T03:52:23.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0079999999999999e-06 2024-09-11T03:52:23.073 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.073 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.073 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.073 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.020919+0000", 2024-09-11T03:52:23.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8779999999999998e-06 2024-09-11T03:52:23.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.074 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.020922+0000", 2024-09-11T03:52:23.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.061e-06 2024-09-11T03:52:23.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.074 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.020978+0000", 2024-09-11T03:52:23.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6703000000000002e-05 2024-09-11T03:52:23.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.075 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:23.075 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.020983+0000", 2024-09-11T03:52:23.075 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3989999999999997e-06 2024-09-11T03:52:23.075 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.075 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.075 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.075 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.022744+0000", 2024-09-11T03:52:23.075 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001761372 2024-09-11T03:52:23.075 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.075 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.075 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:23.075 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.022800+0000", 2024-09-11T03:52:23.076 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5893999999999997e-05 2024-09-11T03:52:23.076 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.076 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.076 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.076 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.037693+0000", 2024-09-11T03:52:23.076 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014892593000000001 2024-09-11T03:52:23.076 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.076 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.076 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.076 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.037807+0000", 2024-09-11T03:52:23.076 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000114472 2024-09-11T03:52:23.076 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.076 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.076 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.077 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:35.037917+0000", 2024-09-11T03:52:23.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110266 2024-09-11T03:52:23.077 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.077 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.077 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.077 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.077 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.077 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7045 2.4 2:25f5f8b9:::smithi17557870-26:head [copy-from ver 950] snapc 13a=[13a,139,138,137,136,135,133,130,12e,12d,12c,12a,127,126,123,120,11f,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e709)", 2024-09-11T03:52:23.077 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:47.006390+0000", 2024-09-11T03:52:23.078 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 216.00782523399999, 2024-09-11T03:52:23.078 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030995754, 2024-09-11T03:52:23.078 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.078 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.078 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:23.078 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.078 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.078 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.078 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7045 2024-09-11T03:52:23.078 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.078 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.078 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.078 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006390+0000", 2024-09-11T03:52:23.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006390+0000", 2024-09-11T03:52:23.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.080 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006392+0000", 2024-09-11T03:52:23.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9659999999999999e-06 2024-09-11T03:52:23.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.080 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.080 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.080 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006399+0000", 2024-09-11T03:52:23.081 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5819999999999998e-06 2024-09-11T03:52:23.081 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.081 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.081 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.081 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006417+0000", 2024-09-11T03:52:23.081 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.806e-05 2024-09-11T03:52:23.081 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.081 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.081 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.082 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006426+0000", 2024-09-11T03:52:23.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0399999999999998e-06 2024-09-11T03:52:23.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.082 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.082 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.082 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006678+0000", 2024-09-11T03:52:23.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025198100000000003 2024-09-11T03:52:23.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.082 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.082 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:52:23.082 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006686+0000", 2024-09-11T03:52:23.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3599999999999996e-06 2024-09-11T03:52:23.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.008805+0000", 2024-09-11T03:52:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021189300000000002 2024-09-11T03:52:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.008889+0000", 2024-09-11T03:52:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3634999999999995e-05 2024-09-11T03:52:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,10", 2024-09-11T03:52:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.024971+0000", 2024-09-11T03:52:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016081958 2024-09-11T03:52:23.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.084 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.084 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.084 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.029483+0000", 2024-09-11T03:52:23.084 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045122119999999998 2024-09-11T03:52:23.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.084 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.084 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:23.084 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.032426+0000", 2024-09-11T03:52:23.084 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029432830000000001 2024-09-11T03:52:23.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.084 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.085 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.085 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.037231+0000", 2024-09-11T03:52:23.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048053610000000002 2024-09-11T03:52:23.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.085 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.085 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:23.085 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.037308+0000", 2024-09-11T03:52:23.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6650000000000006e-05 2024-09-11T03:52:23.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.085 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.085 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.085 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.037386+0000", 2024-09-11T03:52:23.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7736000000000002e-05 2024-09-11T03:52:23.086 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.086 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.086 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.086 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.086 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.086 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:7046 2.4 2:25f5f8b9:::smithi17557870-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e709)", 2024-09-11T03:52:23.086 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:47.006719+0000", 2024-09-11T03:52:23.086 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 216.007495944, 2024-09-11T03:52:23.086 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022724026000000001, 2024-09-11T03:52:23.086 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.086 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.086 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:23.087 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.087 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.087 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.087 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7046 2024-09-11T03:52:23.087 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.087 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.087 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.087 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.087 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006719+0000", 2024-09-11T03:52:23.087 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.087 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.087 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.087 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.088 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006719+0000", 2024-09-11T03:52:23.088 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.088 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.088 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.088 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006720+0000", 2024-09-11T03:52:23.088 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7100000000000001e-07 2024-09-11T03:52:23.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.088 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.088 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.088 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006724+0000", 2024-09-11T03:52:23.088 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8419999999999998e-06 2024-09-11T03:52:23.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.089 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.089 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006732+0000", 2024-09-11T03:52:23.089 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0369999999999995e-06 2024-09-11T03:52:23.089 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.089 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.089 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006739+0000", 2024-09-11T03:52:23.089 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9110000000000004e-06 2024-09-11T03:52:23.089 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.090 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.090 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006801+0000", 2024-09-11T03:52:23.090 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1756999999999998e-05 2024-09-11T03:52:23.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.090 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:23.090 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.006806+0000", 2024-09-11T03:52:23.090 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6489999999999996e-06 2024-09-11T03:52:23.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.090 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.090 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.009029+0000", 2024-09-11T03:52:23.090 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002222239 2024-09-11T03:52:23.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:23.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.009086+0000", 2024-09-11T03:52:23.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7525000000000002e-05 2024-09-11T03:52:23.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.029305+0000", 2024-09-11T03:52:23.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020219183000000002 2024-09-11T03:52:23.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.029375+0000", 2024-09-11T03:52:23.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9611999999999997e-05 2024-09-11T03:52:23.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.092 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.092 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.092 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:47.029443+0000", 2024-09-11T03:52:23.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.86e-05 2024-09-11T03:52:23.092 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.092 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.092 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.092 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.092 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:7115 2.c e713/569 2:3603e322:::smithi17557870-42:head v 713'769, mlcod=713'767)", 2024-09-11T03:52:23.092 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:51.207298+0000", 2024-09-11T03:52:23.093 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 211.806917527, 2024-09-11T03:52:23.093 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015417943999999999, 2024-09-11T03:52:23.093 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.093 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.093 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:23.093 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.093 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.094 INFO:tasks.rados.rados.2.smithi175.stdout:3855: done (3 left) 2024-09-11T03:52:23.094 INFO:tasks.rados.rados.2.smithi175.stdout:3856: delete oid 36 current snap is 428 2024-09-11T03:52:23.094 INFO:tasks.rados.rados.2.smithi175.stdout:3854: finishing write tid 1 to smithi17557870-33 2024-09-11T03:52:23.094 INFO:tasks.rados.rados.2.smithi175.stdout:3853: expect (ObjNum 1188 snap 388 seq_num 1188) 2024-09-11T03:52:23.095 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.095 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.207298+0000", 2024-09-11T03:52:23.095 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.095 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.095 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.095 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.207298+0000", 2024-09-11T03:52:23.095 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.095 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.096 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.096 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.207299+0000", 2024-09-11T03:52:23.096 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.279e-06 2024-09-11T03:52:23.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.096 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.096 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.216213+0000", 2024-09-11T03:52:23.096 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0089137999999999995 2024-09-11T03:52:23.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.096 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.216238+0000", 2024-09-11T03:52:23.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4933e-05 2024-09-11T03:52:23.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.216249+0000", 2024-09-11T03:52:23.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0729e-05 2024-09-11T03:52:23.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.216320+0000", 2024-09-11T03:52:23.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0957999999999994e-05 2024-09-11T03:52:23.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.098 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.216351+0000", 2024-09-11T03:52:23.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1171999999999999e-05 2024-09-11T03:52:23.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:23.098 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.222642+0000", 2024-09-11T03:52:23.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062908840000000001 2024-09-11T03:52:23.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.098 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:51.222716+0000", 2024-09-11T03:52:23.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4189000000000004e-05 2024-09-11T03:52:23.099 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.099 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.099 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.099 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.099 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:8515 2.36 2:6dbd1e81:::smithi17557870-7:head [sparse-read 0~1879812,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e830)", 2024-09-11T03:52:23.099 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:50:48.049912+0000", 2024-09-11T03:52:23.099 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 94.964303057999999, 2024-09-11T03:52:23.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010062527999999999, 2024-09-11T03:52:23.099 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.099 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.099 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:23.099 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.100 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.100 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.100 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8515 2024-09-11T03:52:23.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.100 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.100 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.100 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.049912+0000", 2024-09-11T03:52:23.100 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.100 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.100 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.049912+0000", 2024-09-11T03:52:23.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.101 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.049914+0000", 2024-09-11T03:52:23.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9719999999999999e-06 2024-09-11T03:52:23.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.101 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.049923+0000", 2024-09-11T03:52:23.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4100000000000008e-06 2024-09-11T03:52:23.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.102 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.049941+0000", 2024-09-11T03:52:23.102 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7977000000000001e-05 2024-09-11T03:52:23.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.102 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.102 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.049950+0000", 2024-09-11T03:52:23.102 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5780000000000005e-06 2024-09-11T03:52:23.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.102 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.102 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.050065+0000", 2024-09-11T03:52:23.102 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000115011 2024-09-11T03:52:23.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.103 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:52:23.103 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.050073+0000", 2024-09-11T03:52:23.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4499999999999998e-06 2024-09-11T03:52:23.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.103 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.103 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.057776+0000", 2024-09-11T03:52:23.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0077037030000000001 2024-09-11T03:52:23.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.104 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.104 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.057884+0000", 2024-09-11T03:52:23.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010781900000000001 2024-09-11T03:52:23.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.059975+0000", 2024-09-11T03:52:23.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020906079999999999 2024-09-11T03:52:23.105 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.106 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.106 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.106 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:8516 2.36 2:6dbd1e81:::smithi17557870-7:head [read 0~2857454,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e830)", 2024-09-11T03:52:23.106 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:50:48.049975+0000", 2024-09-11T03:52:23.107 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 94.964240363000002, 2024-09-11T03:52:23.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011426950999999999, 2024-09-11T03:52:23.107 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.107 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.107 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:23.107 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.108 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.108 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.108 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8516 2024-09-11T03:52:23.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.108 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.109 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.109 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.049975+0000", 2024-09-11T03:52:23.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.049975+0000", 2024-09-11T03:52:23.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.049976+0000", 2024-09-11T03:52:23.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3600000000000004e-07 2024-09-11T03:52:23.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.113 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.049979+0000", 2024-09-11T03:52:23.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8650000000000003e-06 2024-09-11T03:52:23.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.113 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.113 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.049994+0000", 2024-09-11T03:52:23.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4922999999999999e-05 2024-09-11T03:52:23.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.050015+0000", 2024-09-11T03:52:23.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0199000000000001e-05 2024-09-11T03:52:23.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.115 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.050137+0000", 2024-09-11T03:52:23.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0001226 2024-09-11T03:52:23.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:23.115 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.050143+0000", 2024-09-11T03:52:23.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.981e-06 2024-09-11T03:52:23.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.060010+0000", 2024-09-11T03:52:23.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0098670470000000003 2024-09-11T03:52:23.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.060126+0000", 2024-09-11T03:52:23.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011620699999999999 2024-09-11T03:52:23.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.117 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.117 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.061402+0000", 2024-09-11T03:52:23.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001275593 2024-09-11T03:52:23.117 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.117 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.117 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.117 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:8517 2.36 2:6dbd1e81:::smithi17557870-7:head [read 0~2857454,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e830)", 2024-09-11T03:52:23.117 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:50:48.050075+0000", 2024-09-11T03:52:23.117 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 94.964140442000001, 2024-09-11T03:52:23.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012424759, 2024-09-11T03:52:23.118 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.118 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.118 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:23.118 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.118 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.118 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.118 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8517 2024-09-11T03:52:23.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.118 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.118 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.118 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.050075+0000", 2024-09-11T03:52:23.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.119 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.119 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.050075+0000", 2024-09-11T03:52:23.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.119 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.119 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.050075+0000", 2024-09-11T03:52:23.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9699999999999996e-07 2024-09-11T03:52:23.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.050080+0000", 2024-09-11T03:52:23.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.014e-06 2024-09-11T03:52:23.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.050116+0000", 2024-09-11T03:52:23.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6186999999999998e-05 2024-09-11T03:52:23.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.050124+0000", 2024-09-11T03:52:23.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8839999999999994e-06 2024-09-11T03:52:23.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.050169+0000", 2024-09-11T03:52:23.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5166000000000001e-05 2024-09-11T03:52:23.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:23.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.050174+0000", 2024-09-11T03:52:23.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4419999999999997e-06 2024-09-11T03:52:23.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.061940+0000", 2024-09-11T03:52:23.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011766 2024-09-11T03:52:23.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.062048+0000", 2024-09-11T03:52:23.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010828900000000001 2024-09-11T03:52:23.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:48.062500+0000", 2024-09-11T03:52:23.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00045117999999999997 2024-09-11T03:52:23.122 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.122 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.123 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.123 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:8974 2.44 2:220bb244:::smithi17557870-34:head [copy-from ver 1205] snapc 191=[191,190,18e,18d,18c,18b,18a,189,188,187,186,185,184,183,180,17e,17d,17c,17a,178,177,175,174,172,16e,16b,169,167,164,15f,15e,15b,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-09-11T03:52:23.123 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:25.284904+0000", 2024-09-11T03:52:23.123 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 57.729311021999997, 2024-09-11T03:52:23.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015100865, 2024-09-11T03:52:23.123 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.123 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.123 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:23.123 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.123 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.123 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.123 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8974 2024-09-11T03:52:23.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.124 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.124 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.124 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.284904+0000", 2024-09-11T03:52:23.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.124 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.124 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.284904+0000", 2024-09-11T03:52:23.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.284907+0000", 2024-09-11T03:52:23.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9660000000000001e-06 2024-09-11T03:52:23.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.284921+0000", 2024-09-11T03:52:23.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3499999999999999e-05 2024-09-11T03:52:23.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.284954+0000", 2024-09-11T03:52:23.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3076e-05 2024-09-11T03:52:23.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.284970+0000", 2024-09-11T03:52:23.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6016999999999999e-05 2024-09-11T03:52:23.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.285088+0000", 2024-09-11T03:52:23.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000118507 2024-09-11T03:52:23.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:52:23.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.285097+0000", 2024-09-11T03:52:23.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3990000000000004e-06 2024-09-11T03:52:23.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.286693+0000", 2024-09-11T03:52:23.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001596004 2024-09-11T03:52:23.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.286845+0000", 2024-09-11T03:52:23.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152638 2024-09-11T03:52:23.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,11", 2024-09-11T03:52:23.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.292984+0000", 2024-09-11T03:52:23.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061381830000000002 2024-09-11T03:52:23.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:23.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.298453+0000", 2024-09-11T03:52:23.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054695170000000001 2024-09-11T03:52:23.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.298661+0000", 2024-09-11T03:52:23.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000207961 2024-09-11T03:52:23.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.299924+0000", 2024-09-11T03:52:23.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00126322 2024-09-11T03:52:23.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:23.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.299967+0000", 2024-09-11T03:52:23.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2664999999999999e-05 2024-09-11T03:52:23.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.130 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.300005+0000", 2024-09-11T03:52:23.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8211999999999999e-05 2024-09-11T03:52:23.130 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.130 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.130 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.130 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.130 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:8975 2.44 2:220bb244:::smithi17557870-34:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e867)", 2024-09-11T03:52:23.130 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:25.285004+0000", 2024-09-11T03:52:23.130 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 57.729211757000002, 2024-09-11T03:52:23.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012443522, 2024-09-11T03:52:23.130 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.130 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.131 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:23.131 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.131 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.131 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.131 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8975 2024-09-11T03:52:23.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.131 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.131 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.285004+0000", 2024-09-11T03:52:23.131 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.285004+0000", 2024-09-11T03:52:23.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.285004+0000", 2024-09-11T03:52:23.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8999999999999995e-07 2024-09-11T03:52:23.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.285009+0000", 2024-09-11T03:52:23.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1500000000000001e-06 2024-09-11T03:52:23.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.285022+0000", 2024-09-11T03:52:23.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3346000000000001e-05 2024-09-11T03:52:23.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.285036+0000", 2024-09-11T03:52:23.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3713000000000001e-05 2024-09-11T03:52:23.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.285159+0000", 2024-09-11T03:52:23.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012291799999999999 2024-09-11T03:52:23.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:23.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.285166+0000", 2024-09-11T03:52:23.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5619999999999998e-06 2024-09-11T03:52:23.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.287001+0000", 2024-09-11T03:52:23.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018352220000000001 2024-09-11T03:52:23.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:23.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.287055+0000", 2024-09-11T03:52:23.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3618000000000001e-05 2024-09-11T03:52:23.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.297286+0000", 2024-09-11T03:52:23.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010230680000000001 2024-09-11T03:52:23.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.297365+0000", 2024-09-11T03:52:23.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8850000000000006e-05 2024-09-11T03:52:23.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.136 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.136 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:25.297447+0000", 2024-09-11T03:52:23.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2572999999999997e-05 2024-09-11T03:52:23.136 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.136 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.136 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.136 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:9048 2.44 2:220bb244:::smithi17557870-34:head [copy-from ver 1014] snapc 196=[196,195,194,193,192,191,190,18e,18d,18c,18b,18a,189,188,186,183,180,17e,17d,17c,17a,178,177,175,174,172,16e,16b,169,167,164,15f,15e,15b,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e875)", 2024-09-11T03:52:23.136 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:33.057658+0000", 2024-09-11T03:52:23.136 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 49.956557136999997, 2024-09-11T03:52:23.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017721764000000001, 2024-09-11T03:52:23.137 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.137 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.137 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:23.137 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.137 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.137 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.137 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9048 2024-09-11T03:52:23.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.137 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.057658+0000", 2024-09-11T03:52:23.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.057658+0000", 2024-09-11T03:52:23.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.057660+0000", 2024-09-11T03:52:23.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2680000000000001e-06 2024-09-11T03:52:23.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.057671+0000", 2024-09-11T03:52:23.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0222e-05 2024-09-11T03:52:23.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.057690+0000", 2024-09-11T03:52:23.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9185e-05 2024-09-11T03:52:23.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.057699+0000", 2024-09-11T03:52:23.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5219999999999996e-06 2024-09-11T03:52:23.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.140 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.057760+0000", 2024-09-11T03:52:23.140 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0853000000000002e-05 2024-09-11T03:52:23.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.140 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:52:23.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.057773+0000", 2024-09-11T03:52:23.140 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2663999999999999e-05 2024-09-11T03:52:23.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.140 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.059360+0000", 2024-09-11T03:52:23.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001587305 2024-09-11T03:52:23.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.059493+0000", 2024-09-11T03:52:23.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013294199999999999 2024-09-11T03:52:23.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,11", 2024-09-11T03:52:23.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.067700+0000", 2024-09-11T03:52:23.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082068579999999992 2024-09-11T03:52:23.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.074263+0000", 2024-09-11T03:52:23.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.006562722 2024-09-11T03:52:23.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:23.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.075124+0000", 2024-09-11T03:52:23.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00086106099999999999 2024-09-11T03:52:23.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.075279+0000", 2024-09-11T03:52:23.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015532499999999999 2024-09-11T03:52:23.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:23.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.075322+0000", 2024-09-11T03:52:23.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2536000000000001e-05 2024-09-11T03:52:23.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.075380+0000", 2024-09-11T03:52:23.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8301000000000003e-05 2024-09-11T03:52:23.143 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.143 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.143 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.143 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:9049 2.44 2:220bb244:::smithi17557870-34:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e875)", 2024-09-11T03:52:23.144 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:33.057719+0000", 2024-09-11T03:52:23.144 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 49.956496520999998, 2024-09-11T03:52:23.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014183196, 2024-09-11T03:52:23.144 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.144 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.144 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:23.144 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.144 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.144 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.144 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9049 2024-09-11T03:52:23.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.145 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.057719+0000", 2024-09-11T03:52:23.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.057719+0000", 2024-09-11T03:52:23.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.057719+0000", 2024-09-11T03:52:23.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3999999999999997e-07 2024-09-11T03:52:23.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.057722+0000", 2024-09-11T03:52:23.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4420000000000001e-06 2024-09-11T03:52:23.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.057728+0000", 2024-09-11T03:52:23.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0689999999999998e-06 2024-09-11T03:52:23.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.057731+0000", 2024-09-11T03:52:23.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.675e-06 2024-09-11T03:52:23.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.057831+0000", 2024-09-11T03:52:23.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9791999999999996e-05 2024-09-11T03:52:23.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:23.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.057838+0000", 2024-09-11T03:52:23.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.489e-06 2024-09-11T03:52:23.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.059654+0000", 2024-09-11T03:52:23.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018159319999999999 2024-09-11T03:52:23.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:23.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.059682+0000", 2024-09-11T03:52:23.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8751e-05 2024-09-11T03:52:23.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.071757+0000", 2024-09-11T03:52:23.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012074786000000001 2024-09-11T03:52:23.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.071827+0000", 2024-09-11T03:52:23.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9721999999999994e-05 2024-09-11T03:52:23.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:33.071902+0000", 2024-09-11T03:52:23.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5197999999999994e-05 2024-09-11T03:52:23.150 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.150 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.150 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.150 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:9061 2.36 2:6dbd1e81:::smithi17557870-7:head [copy-from ver 1011] snapc 196=[196,195,194,193,192,191,190,18c,18b,18a,189,188,186,183,180,17e,17d,17c,17a,178,177,175,174,172,16e,16b,169,167,164,15f,15e,15b,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e877)", 2024-09-11T03:52:23.150 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:35.075974+0000", 2024-09-11T03:52:23.150 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 47.938240870000001, 2024-09-11T03:52:23.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016846823, 2024-09-11T03:52:23.151 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:23.151 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:23.151 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:23.151 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:23.151 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:23.151 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:23.151 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9061 2024-09-11T03:52:23.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.151 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:23.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.151 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:23.151 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:35.075974+0000", 2024-09-11T03:52:23.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:23.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:35.075974+0000", 2024-09-11T03:52:23.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:23.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:23.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:35.075976+0000", 2024-09-11T03:52:23.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.792e-06 2024-09-11T03:52:23.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:23.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:35.075982+0000", 2024-09-11T03:52:23.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9619999999999998e-06 2024-09-11T03:52:23.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:23.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:35.075995+0000", 2024-09-11T03:52:23.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2955999999999999e-05 2024-09-11T03:52:23.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:23.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:35.076004+0000", 2024-09-11T03:52:23.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3680000000000002e-06 2024-09-11T03:52:23.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:23.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:35.076058+0000", 2024-09-11T03:52:23.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4911000000000003e-05 2024-09-11T03:52:23.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:23.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:35.076166+0000", 2024-09-11T03:52:23.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107459 2024-09-11T03:52:23.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 8,11", 2024-09-11T03:52:23.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:35.082056+0000", 2024-09-11T03:52:23.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005890373 2024-09-11T03:52:23.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:23.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:35.086114+0000", 2024-09-11T03:52:23.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00405785 2024-09-11T03:52:23.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:35.092520+0000", 2024-09-11T03:52:23.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0064055070000000004 2024-09-11T03:52:23.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:23.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:35.092698+0000", 2024-09-11T03:52:23.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000178698 2024-09-11T03:52:23.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:23.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:35.092764+0000", 2024-09-11T03:52:23.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5764000000000003e-05 2024-09-11T03:52:23.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:23.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:23.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:23.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:35.092821+0000", 2024-09-11T03:52:23.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7182999999999998e-05 2024-09-11T03:52:23.156 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.156 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.156 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.156 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:23.157 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:23.157 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:52:23.157 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:23.178 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:23.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:23.179 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:23.205 INFO:tasks.rados.rados.2.smithi175.stdout:3854: finishing write tid 2 to smithi17557870-33 2024-09-11T03:52:23.206 INFO:tasks.rados.rados.2.smithi175.stdout:3854: finishing write tid 3 to smithi17557870-33 2024-09-11T03:52:23.206 INFO:tasks.rados.rados.2.smithi175.stdout:3854: finishing write tid 4 to smithi17557870-33 2024-09-11T03:52:23.206 INFO:tasks.rados.rados.2.smithi175.stdout:3854: finishing write tid 5 to smithi17557870-33 2024-09-11T03:52:23.206 INFO:tasks.rados.rados.2.smithi175.stdout:3854: oid 33 updating version 0 to 1584 2024-09-11T03:52:23.206 INFO:tasks.rados.rados.2.smithi175.stdout:3854: oid 33 version 1584 is already newer than 1582 2024-09-11T03:52:23.206 INFO:tasks.rados.rados.2.smithi175.stdout:3854: oid 33 version 1584 is already newer than 1583 2024-09-11T03:52:23.206 INFO:tasks.rados.rados.2.smithi175.stdout:3854: oid 33 version 1584 is already newer than 1581 2024-09-11T03:52:23.206 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 1584 (ObjNum 1306 snap 428 seq_num 1306) dirty exists 2024-09-11T03:52:23.206 INFO:tasks.rados.rados.2.smithi175.stdout:3854: left oid 33 (ObjNum 1306 snap 428 seq_num 1306) 2024-09-11T03:52:23.206 INFO:tasks.rados.rados.2.smithi175.stdout:3853: done (3 left) 2024-09-11T03:52:23.206 INFO:tasks.rados.rados.2.smithi175.stdout:3854: done (2 left) 2024-09-11T03:52:23.207 INFO:tasks.rados.rados.2.smithi175.stdout:3856: done (1 left) 2024-09-11T03:52:23.207 INFO:tasks.rados.rados.2.smithi175.stdout:3857: delete oid 31 current snap is 428 2024-09-11T03:52:23.207 INFO:tasks.rados.rados.2.smithi175.stdout:3850: finishing rollback tid 0 to smithi17557870-49 2024-09-11T03:52:23.207 INFO:tasks.rados.rados.2.smithi175.stdout:3850: finishing rollback tid 1 to smithi17557870-49 2024-09-11T03:52:23.207 INFO:tasks.rados.rados.2.smithi175.stdout:3850: finishing rollback tid 2 to smithi17557870-49 2024-09-11T03:52:23.207 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 1014 (ObjNum 1125 snap 364 seq_num 1125) dirty exists 2024-09-11T03:52:23.207 INFO:tasks.rados.rados.2.smithi175.stdout:3850: done (1 left) 2024-09-11T03:52:23.207 INFO:tasks.rados.rados.2.smithi175.stdout:3857: done (0 left) 2024-09-11T03:52:23.207 INFO:tasks.rados.rados.2.smithi175.stdout:3858: snap_remove snap 406 2024-09-11T03:52:23.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:23.301 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:23.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:23.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:23.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:23 smithi084 ceph-mon[27254]: 2.12 scrub starts 2024-09-11T03:52:23.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:23 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:52:23.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:23 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:52:23.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:23 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:52:23.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:23 smithi084 ceph-mon[27254]: 2.10 scrub starts 2024-09-11T03:52:23.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:23 smithi084 ceph-mon[27254]: osdmap e925: 12 total, 12 up, 11 in 2024-09-11T03:52:23.403 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:23.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:23 smithi121 ceph-mon[30729]: 2.12 scrub starts 2024-09-11T03:52:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:23 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:52:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:23 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:52:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:23 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:52:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:23 smithi121 ceph-mon[30729]: 2.10 scrub starts 2024-09-11T03:52:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:23 smithi121 ceph-mon[30729]: osdmap e925: 12 total, 12 up, 11 in 2024-09-11T03:52:23.427 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:23.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:23.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:23.439 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 8, 9, 11, 10, 6] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:52:23.439 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:52:23.439 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-11T03:52:23.440 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:52:23.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:23 smithi175 ceph-mon[30894]: 2.12 scrub starts 2024-09-11T03:52:23.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:23 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:52:23.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:23 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:52:23.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:23 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:52:23.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:23 smithi175 ceph-mon[30894]: 2.10 scrub starts 2024-09-11T03:52:23.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:23 smithi175 ceph-mon[30894]: osdmap e925: 12 total, 12 up, 11 in 2024-09-11T03:52:23.529 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:23.550 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:23.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:23.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:23.651 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:23.677 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:23.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:23.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:23.779 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:23.805 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:23.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:23.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:23.907 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:23.930 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:23.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:23.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:23.936 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-11T03:52:24.017 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:24.019 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:52:24.032 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:24.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:24.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:24.075 INFO:tasks.rados.rados.2.smithi175.stdout:3858: done (0 left) 2024-09-11T03:52:24.076 INFO:tasks.rados.rados.2.smithi175.stdout:3859: write oid 39 current snap is 428 2024-09-11T03:52:24.076 INFO:tasks.rados.rados.2.smithi175.stdout:3859: seq_num 1309 ranges {400363=513657,1332694=660122} 2024-09-11T03:52:24.080 INFO:tasks.rados.rados.2.smithi175.stdout:3859: writing smithi17557870-39 from 400363 to 914020 tid 1 2024-09-11T03:52:24.088 INFO:tasks.rados.rados.2.smithi175.stdout:3859: writing smithi17557870-39 from 1332694 to 1992816 tid 2 2024-09-11T03:52:24.088 INFO:tasks.rados.rados.2.smithi175.stdout:3860: delete oid 38 current snap is 428 2024-09-11T03:52:24.088 INFO:tasks.rados.rados.2.smithi175.stdout:3859: finishing write tid 1 to smithi17557870-39 2024-09-11T03:52:24.092 INFO:tasks.rados.rados.2.smithi175.stdout:3860: done (1 left) 2024-09-11T03:52:24.092 INFO:tasks.rados.rados.2.smithi175.stdout:3861: snap_create 2024-09-11T03:52:24.092 INFO:tasks.rados.rados.2.smithi175.stdout:3859: finishing write tid 2 to smithi17557870-39 2024-09-11T03:52:24.092 INFO:tasks.rados.rados.2.smithi175.stdout:3859: finishing write tid 3 to smithi17557870-39 2024-09-11T03:52:24.092 INFO:tasks.rados.rados.2.smithi175.stdout:3859: finishing write tid 4 to smithi17557870-39 2024-09-11T03:52:24.093 INFO:tasks.rados.rados.2.smithi175.stdout:3859: oid 39 updating version 0 to 1030 2024-09-11T03:52:24.093 INFO:tasks.rados.rados.2.smithi175.stdout:3859: oid 39 updating version 1030 to 1032 2024-09-11T03:52:24.093 INFO:tasks.rados.rados.2.smithi175.stdout:3859: oid 39 version 1032 is already newer than 1031 2024-09-11T03:52:24.093 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 1032 (ObjNum 1309 snap 428 seq_num 1309) dirty exists 2024-09-11T03:52:24.093 INFO:tasks.rados.rados.2.smithi175.stdout:3859: left oid 39 (ObjNum 1309 snap 428 seq_num 1309) 2024-09-11T03:52:24.160 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:24.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:24.184 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:24.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:24.285 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:24.313 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:24.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:24.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:24.415 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:24 smithi121 ceph-mon[30729]: 2.12 scrub starts 2024-09-11T03:52:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:24 smithi121 ceph-mon[30729]: pgmap v1542: 117 pgs: 1 peering, 3 active+clean+scrubbing, 9 unknown, 104 active+clean; 706 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 22 MiB/s rd, 3.5 MiB/s wr, 25 op/s 2024-09-11T03:52:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:24 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:52:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:24 smithi121 ceph-mon[30729]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-11T03:52:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:24 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:52:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:24 smithi121 ceph-mon[30729]: 2.50 scrub starts 2024-09-11T03:52:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:24 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:52:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:24 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:52:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:24 smithi121 ceph-mon[30729]: osdmap e926: 12 total, 12 up, 11 in 2024-09-11T03:52:24.433 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:24 smithi084 ceph-mon[27254]: 2.12 scrub starts 2024-09-11T03:52:24.433 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:24 smithi084 ceph-mon[27254]: pgmap v1542: 117 pgs: 1 peering, 3 active+clean+scrubbing, 9 unknown, 104 active+clean; 706 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 22 MiB/s rd, 3.5 MiB/s wr, 25 op/s 2024-09-11T03:52:24.433 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:24 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:52:24.433 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:24 smithi084 ceph-mon[27254]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-11T03:52:24.433 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:24 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:52:24.433 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:24 smithi084 ceph-mon[27254]: 2.50 scrub starts 2024-09-11T03:52:24.433 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:24 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:52:24.434 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:24 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:52:24.434 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:24 smithi084 ceph-mon[27254]: osdmap e926: 12 total, 12 up, 11 in 2024-09-11T03:52:24.437 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:24.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:24.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:24 smithi175 ceph-mon[30894]: 2.12 scrub starts 2024-09-11T03:52:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:24 smithi175 ceph-mon[30894]: pgmap v1542: 117 pgs: 1 peering, 3 active+clean+scrubbing, 9 unknown, 104 active+clean; 706 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 22 MiB/s rd, 3.5 MiB/s wr, 25 op/s 2024-09-11T03:52:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:24 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:52:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:24 smithi175 ceph-mon[30894]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-11T03:52:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:24 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:52:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:24 smithi175 ceph-mon[30894]: 2.50 scrub starts 2024-09-11T03:52:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:24 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:52:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:24 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:52:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:24 smithi175 ceph-mon[30894]: osdmap e926: 12 total, 12 up, 11 in 2024-09-11T03:52:24.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:24.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:52:24.555 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:24.557 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:24.563 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:24.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:24.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:24.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:24.691 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:24.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:24.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:24.719 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:52:24.721 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:52:24.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:24.817 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:24.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:24.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:24.919 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:24.943 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:24.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:24.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:25.045 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:25.071 INFO:tasks.rados.rados.2.smithi175.stdout:3859: done (1 left) 2024-09-11T03:52:25.072 INFO:tasks.rados.rados.2.smithi175.stdout:3861: done (0 left) 2024-09-11T03:52:25.072 INFO:tasks.rados.rados.2.smithi175.stdout:3862: write oid 49 current snap is 429 2024-09-11T03:52:25.072 INFO:tasks.rados.rados.2.smithi175.stdout:3862: seq_num 1311 ranges {433977=664155,1733305=300672} 2024-09-11T03:52:25.072 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:25.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:25.073 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:25.077 INFO:tasks.rados.rados.2.smithi175.stdout:3862: writing smithi17557870-49 from 433977 to 1098132 tid 1 2024-09-11T03:52:25.083 INFO:tasks.rados.rados.2.smithi175.stdout:3862: writing smithi17557870-49 from 1733305 to 2033977 tid 2 2024-09-11T03:52:25.084 INFO:tasks.rados.rados.2.smithi175.stdout:3863: read oid 18 snap -1 2024-09-11T03:52:25.084 INFO:tasks.rados.rados.2.smithi175.stdout:3863: expect deleted 2024-09-11T03:52:25.084 INFO:tasks.rados.rados.2.smithi175.stdout:3864: write oid 23 current snap is 429 2024-09-11T03:52:25.084 INFO:tasks.rados.rados.2.smithi175.stdout:3864: seq_num 1312 ranges {450784=739404,1791787=258997} 2024-09-11T03:52:25.090 INFO:tasks.rados.rados.2.smithi175.stdout:3864: writing smithi17557870-23 from 450784 to 1190188 tid 1 2024-09-11T03:52:25.095 INFO:tasks.rados.rados.2.smithi175.stdout:3864: writing smithi17557870-23 from 1791787 to 2050784 tid 2 2024-09-11T03:52:25.096 INFO:tasks.rados.rados.2.smithi175.stdout:3865: read oid 47 snap -1 2024-09-11T03:52:25.096 INFO:tasks.rados.rados.2.smithi175.stdout:3865: expect (ObjNum 1300 snap 426 seq_num 1300) 2024-09-11T03:52:25.096 INFO:tasks.rados.rados.2.smithi175.stdout:3866: delete oid 28 current snap is 429 2024-09-11T03:52:25.096 INFO:tasks.rados.rados.2.smithi175.stdout:3862: finishing write tid 1 to smithi17557870-49 2024-09-11T03:52:25.096 INFO:tasks.rados.rados.2.smithi175.stdout:3862: finishing write tid 2 to smithi17557870-49 2024-09-11T03:52:25.096 INFO:tasks.rados.rados.2.smithi175.stdout:3862: finishing write tid 3 to smithi17557870-49 2024-09-11T03:52:25.096 INFO:tasks.rados.rados.2.smithi175.stdout:3862: finishing write tid 4 to smithi17557870-49 2024-09-11T03:52:25.096 INFO:tasks.rados.rados.2.smithi175.stdout:3862: oid 49 updating version 0 to 1016 2024-09-11T03:52:25.096 INFO:tasks.rados.rados.2.smithi175.stdout:3862: oid 49 updating version 1016 to 1017 2024-09-11T03:52:25.096 INFO:tasks.rados.rados.2.smithi175.stdout:3862: oid 49 version 1017 is already newer than 1015 2024-09-11T03:52:25.096 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 1017 (ObjNum 1311 snap 429 seq_num 1311) dirty exists 2024-09-11T03:52:25.096 INFO:tasks.rados.rados.2.smithi175.stdout:3862: left oid 49 (ObjNum 1311 snap 429 seq_num 1311) 2024-09-11T03:52:25.096 INFO:tasks.rados.rados.2.smithi175.stdout:3864: finishing write tid 1 to smithi17557870-23 2024-09-11T03:52:25.098 INFO:tasks.rados.rados.2.smithi175.stdout:3864: finishing write tid 2 to smithi17557870-23 2024-09-11T03:52:25.098 INFO:tasks.rados.rados.2.smithi175.stdout:3862: done (4 left) 2024-09-11T03:52:25.098 INFO:tasks.rados.rados.2.smithi175.stdout:3863: done (3 left) 2024-09-11T03:52:25.098 INFO:tasks.rados.rados.2.smithi175.stdout:3866: done (2 left) 2024-09-11T03:52:25.098 INFO:tasks.rados.rados.2.smithi175.stdout:3867: read oid 3 snap -1 2024-09-11T03:52:25.098 INFO:tasks.rados.rados.2.smithi175.stdout:3867: expect (ObjNum 1302 snap 426 seq_num 1302) 2024-09-11T03:52:25.098 INFO:tasks.rados.rados.2.smithi175.stdout:3868: read oid 45 snap -1 2024-09-11T03:52:25.098 INFO:tasks.rados.rados.2.smithi175.stdout:3868: expect deleted 2024-09-11T03:52:25.098 INFO:tasks.rados.rados.2.smithi175.stdout:3869: snap_remove snap 359 2024-09-11T03:52:25.173 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:25.196 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:25.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:25.197 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:25.298 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:25.321 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:25.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:25.322 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:25.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:25 smithi084 ceph-mon[27254]: 2.12 scrub starts 2024-09-11T03:52:25.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:25 smithi084 ceph-mon[27254]: 2.11 scrub starts 2024-09-11T03:52:25.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:25 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:52:25.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:25 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:52:25.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:25 smithi084 ceph-mon[27254]: 2.50 scrub starts 2024-09-11T03:52:25.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:25 smithi084 ceph-mon[27254]: from='client.17538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:52:25.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:25 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:52:25.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:25 smithi084 ceph-mon[27254]: osdmap e927: 12 total, 12 up, 11 in 2024-09-11T03:52:25.423 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:25.445 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:25.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:25.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:25 smithi175 ceph-mon[30894]: 2.12 scrub starts 2024-09-11T03:52:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:25 smithi175 ceph-mon[30894]: 2.11 scrub starts 2024-09-11T03:52:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:25 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:52:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:25 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:52:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:25 smithi175 ceph-mon[30894]: 2.50 scrub starts 2024-09-11T03:52:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:25 smithi175 ceph-mon[30894]: from='client.17538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:52:25.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:25 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:52:25.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:25 smithi175 ceph-mon[30894]: osdmap e927: 12 total, 12 up, 11 in 2024-09-11T03:52:25.513 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:52:25.513 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:52:25.514 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:52:25.514 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:52:25.547 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:25.571 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:25.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:25.572 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:25.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:25 smithi121 ceph-mon[30729]: 2.12 scrub starts 2024-09-11T03:52:25.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:25 smithi121 ceph-mon[30729]: 2.11 scrub starts 2024-09-11T03:52:25.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:25 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:52:25.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:25 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:52:25.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:25 smithi121 ceph-mon[30729]: 2.50 scrub starts 2024-09-11T03:52:25.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:25 smithi121 ceph-mon[30729]: from='client.17538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:52:25.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:25 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:52:25.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:25 smithi121 ceph-mon[30729]: osdmap e927: 12 total, 12 up, 11 in 2024-09-11T03:52:25.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:25.695 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:25.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:25.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:25.797 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:25.821 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:25.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:25.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:25.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:25.945 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1544,"stamp":"2024-09-11T03:52:24.070505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":747836017,"num_objects":319,"num_object_clones":265,"num_object_copies":957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":319,"num_whiteouts":12,"num_read":1269,"num_read_kb":1371776,"num_write":849,"num_write_kb":340949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":82584822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54452,"ondisk_log_size":54452,"up":345,"acting":341,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":351,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":1940240,"kb_used_data":1329228,"kb_used_omap":17,"kb_used_meta":610734,"kb_avail":1029256432,"statfs":{"total":1055945392128,"available":1053958586368,"internally_reserved":0,"allocated":1361129472,"data_stored":1728483935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17546,"internal_metadata":625392502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-50422680,"num_objects":-17,"num_object_clones":-19,"num_object_copies":-51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-17,"num_whiteouts":3,"num_read":99,"num_read_kb":116723,"num_write":79,"num_write_kb":31281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.917362"},"pg_stats":[{"pgid":"2.55","version":"274'273","reported_seq":1228,"reported_epoch":923,"state":"unknown","last_fresh":"2024-09-11T03:52:21.172059+0000","last_change":"2024-09-11T03:52:21.738093+0000","last_active":"2024-09-11T03:41:35.872331+0000","last_peered":"2024-09-11T03:41:31.029234+0000","last_clean":"2024-09-11T03:41:31.029234+0000","last_became_active":"2024-09-11T03:38:30.163080+0000","last_became_peered":"2024-09-11T03:38:30.163080+0000","last_unstale":"2024-09-11T03:52:21.172059+0000","last_undegraded":"2024-09-11T03:52:21.172059+0000","last_fullsized":"2024-09-11T03:52:21.172059+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":7,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:51:20.293380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.293380+0000","objects_scrubbed":8,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2287039,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":24301,"num_write":16,"num_write_kb":5595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,9],"acting":[5,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.54","version":"0'0","reported_seq":1,"reported_epoch":923,"state":"unknown","last_fresh":"2024-09-11T03:52:21.173199+0000","last_change":"2024-09-11T03:52:21.691217+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T03:52:21.173199+0000","last_undegraded":"2024-09-11T03:52:21.173199+0000","last_fullsized":"2024-09-11T03:52:21.173199+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'353","last_scrub_stamp":"2024-09-11T03:51:23.900161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.900161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,10],"acting":[1,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.53","version":"274'111","reported_seq":1020,"reported_epoch":925,"state":"active+undersized+remapped","last_fresh":"2024-09-11T03:52:23.078894+0000","last_change":"2024-09-11T03:52:23.078509+0000","last_active":"2024-09-11T03:52:23.078894+0000","last_peered":"2024-09-11T03:52:23.078894+0000","last_clean":"2024-09-11T03:41:31.036769+0000","last_became_active":"2024-09-11T03:52:23.078509+0000","last_became_peered":"2024-09-11T03:52:23.078509+0000","last_unstale":"2024-09-11T03:52:23.078894+0000","last_undegraded":"2024-09-11T03:52:23.078894+0000","last_fullsized":"2024-09-11T03:52:23.070296+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":7,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:51:25.248547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:25.248547+0000","objects_scrubbed":2,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":676486,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":5308,"num_write":6,"num_write_kb":2062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[3,7],"avail_no_missing":["3","1","6","7"],"object_location_counts":[{"shards":"3,7","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.52","version":"905'590","reported_seq":4673,"reported_epoch":925,"state":"active+undersized+remapped","last_fresh":"2024-09-11T03:52:23.084118+0000","last_change":"2024-09-11T03:52:23.083758+0000","last_active":"2024-09-11T03:52:23.084118+0000","last_peered":"2024-09-11T03:52:23.084118+0000","last_clean":"2024-09-11T03:52:03.063604+0000","last_became_active":"2024-09-11T03:52:23.083758+0000","last_became_peered":"2024-09-11T03:52:23.083758+0000","last_unstale":"2024-09-11T03:52:23.084118+0000","last_undegraded":"2024-09-11T03:52:23.084118+0000","last_fullsized":"2024-09-11T03:52:23.072798+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":7,"last_scrub":"905'590","last_scrub_stamp":"2024-09-11T03:52:16.650911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:16.650911+0000","objects_scrubbed":6,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2739805,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[3,10],"avail_no_missing":["3","0","7","10","11"],"object_location_counts":[{"shards":"3,10","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.51","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T03:52:19.143636+0000","last_change":"2024-09-11T03:52:19.143636+0000","last_active":"2024-09-11T03:52:19.143636+0000","last_peered":"2024-09-11T03:52:19.143636+0000","last_clean":"2024-09-11T03:52:19.143636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T03:52:19.143636+0000","last_undegraded":"2024-09-11T03:52:19.143636+0000","last_fullsized":"2024-09-11T03:52:19.143636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:52:19.143636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:52:19.143636+0000","last_clean_scrub_stamp":"2024-09-11T03:52:19.143636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.50","version":"274'67","reported_seq":3248,"reported_epoch":923,"state":"peering","last_fresh":"2024-09-11T03:52:21.172675+0000","last_change":"2024-09-11T03:52:21.172675+0000","last_active":"2024-09-11T03:52:21.172675+0000","last_peered":"2024-09-11T03:52:20.184495+0000","last_clean":"2024-09-11T03:52:20.184495+0000","last_became_active":"2024-09-11T03:52:20.182361+0000","last_became_peered":"2024-09-11T03:52:20.182361+0000","last_unstale":"2024-09-11T03:52:21.172675+0000","last_undegraded":"2024-09-11T03:52:21.172675+0000","last_fullsized":"2024-09-11T03:52:21.172675+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":7,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:51:24.589460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:24.589460+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0017088400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.4f","version":"0'0","reported_seq":1,"reported_epoch":923,"state":"unknown","last_fresh":"2024-09-11T03:52:21.173036+0000","last_change":"2024-09-11T03:52:21.691246+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T03:52:21.173036+0000","last_undegraded":"2024-09-11T03:52:21.173036+0000","last_fullsized":"2024-09-11T03:52:21.173036+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:51:25.558029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:25.558029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,9],"acting":[1,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4e","version":"782'1025","reported_seq":4452,"reported_epoch":925,"state":"active+undersized+remapped","last_fresh":"2024-09-11T03:52:23.077369+0000","last_change":"2024-09-11T03:52:23.077038+0000","last_active":"2024-09-11T03:52:23.077369+0000","last_peered":"2024-09-11T03:52:23.077369+0000","last_clean":"2024-09-11T03:50:00.037885+0000","last_became_active":"2024-09-11T03:52:23.077038+0000","last_became_peered":"2024-09-11T03:52:23.077038+0000","last_unstale":"2024-09-11T03:52:23.077369+0000","last_undegraded":"2024-09-11T03:52:23.077369+0000","last_fullsized":"2024-09-11T03:52:23.067446+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":7,"last_scrub":"782'1025","last_scrub_stamp":"2024-09-11T03:51:22.296245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.296245+0000","objects_scrubbed":8,"log_size":1025,"log_dups_size":0,"ondisk_log_size":1025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5232457,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":16556,"num_write":15,"num_write_kb":7946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[2,1],"avail_no_missing":["2","1","3","5"],"object_location_counts":[{"shards":"1,2","objects":3}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.4d","version":"920'1075","reported_seq":5334,"reported_epoch":925,"state":"active+undersized+remapped","last_fresh":"2024-09-11T03:52:23.083812+0000","last_change":"2024-09-11T03:52:23.083004+0000","last_active":"2024-09-11T03:52:23.083812+0000","last_peered":"2024-09-11T03:52:23.083812+0000","last_clean":"2024-09-11T03:52:18.752296+0000","last_became_active":"2024-09-11T03:52:23.083004+0000","last_became_peered":"2024-09-11T03:52:23.083004+0000","last_unstale":"2024-09-11T03:52:23.083812+0000","last_undegraded":"2024-09-11T03:52:23.083812+0000","last_fullsized":"2024-09-11T03:52:23.067440+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":7,"last_scrub":"916'1068","last_scrub_stamp":"2024-09-11T03:52:14.203354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:14.203354+0000","objects_scrubbed":4,"log_size":1075,"log_dups_size":0,"ondisk_log_size":1075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5654125,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":3514,"num_write":13,"num_write_kb":5402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4719478,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,7],"acting":[2,7],"avail_no_missing":["2","3","5","7","10"],"object_location_counts":[{"shards":"2,7","objects":3}],"blocked_by":[],"up_primary":10,"acting_primary":2,"purged_snaps":[{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"195","length":"1"}]},{"pgid":"2.4c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T03:52:19.143636+0000","last_change":"2024-09-11T03:52:19.143636+0000","last_active":"2024-09-11T03:52:19.143636+0000","last_peered":"2024-09-11T03:52:19.143636+0000","last_clean":"2024-09-11T03:52:19.143636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T03:52:19.143636+0000","last_undegraded":"2024-09-11T03:52:19.143636+0000","last_fullsized":"2024-09-11T03:52:19.143636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:52:19.143636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:52:19.143636+0000","last_clean_scrub_stamp":"2024-09-11T03:52:19.143636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4b","version":"568'646","reported_seq":4838,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172141+0000","last_change":"2024-09-11T03:52:20.153251+0000","last_active":"2024-09-11T03:52:21.172141+0000","last_peered":"2024-09-11T03:52:21.172141+0000","last_clean":"2024-09-11T03:52:21.172141+0000","last_became_active":"2024-09-11T03:51:13.079045+0000","last_became_peered":"2024-09-11T03:51:13.079045+0000","last_unstale":"2024-09-11T03:52:21.172141+0000","last_undegraded":"2024-09-11T03:52:21.172141+0000","last_fullsized":"2024-09-11T03:52:21.172141+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:51:21.370186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.370186+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:28.989989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001445364,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.4a","version":"848'386","reported_seq":4586,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.068201+0000","last_change":"2024-09-11T03:52:20.162518+0000","last_active":"2024-09-11T03:52:22.068201+0000","last_peered":"2024-09-11T03:52:22.068201+0000","last_clean":"2024-09-11T03:52:22.068201+0000","last_became_active":"2024-09-11T03:51:13.080958+0000","last_became_peered":"2024-09-11T03:51:13.080958+0000","last_unstale":"2024-09-11T03:52:22.068201+0000","last_undegraded":"2024-09-11T03:52:22.068201+0000","last_fullsized":"2024-09-11T03:52:22.068201+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"848'386","last_scrub_stamp":"2024-09-11T03:51:19.939163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.939163+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:44.162284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060126199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.49","version":"922'1579","reported_seq":5946,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.171304+0000","last_change":"2024-09-11T03:52:20.176166+0000","last_active":"2024-09-11T03:52:21.171304+0000","last_peered":"2024-09-11T03:52:21.171304+0000","last_clean":"2024-09-11T03:52:21.171304+0000","last_became_active":"2024-09-11T03:51:20.066997+0000","last_became_peered":"2024-09-11T03:51:20.066997+0000","last_unstale":"2024-09-11T03:52:21.171304+0000","last_undegraded":"2024-09-11T03:52:21.171304+0000","last_fullsized":"2024-09-11T03:52:21.171304+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":7,"last_scrub":"863'1411","last_scrub_stamp":"2024-09-11T03:51:21.422037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.422037+0000","objects_scrubbed":18,"log_size":1579,"log_dups_size":0,"ondisk_log_size":1579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:50.255531+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.018992394999999999,"stat_sum":{"num_bytes":49721522,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":42,"num_read_kb":53027,"num_write":23,"num_write_kb":6028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.48","version":"744'374","reported_seq":4074,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-09-11T03:52:23.070817+0000","last_change":"2024-09-11T03:52:23.071290+0000","last_active":"2024-09-11T03:52:23.070817+0000","last_peered":"2024-09-11T03:52:23.070817+0000","last_clean":"2024-09-11T03:52:23.070817+0000","last_became_active":"2024-09-11T03:51:13.082676+0000","last_became_peered":"2024-09-11T03:51:13.082676+0000","last_unstale":"2024-09-11T03:52:23.070817+0000","last_undegraded":"2024-09-11T03:52:23.070817+0000","last_fullsized":"2024-09-11T03:52:23.070817+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"744'374","last_scrub_stamp":"2024-09-11T03:51:19.306681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.306681+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:34.558129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00069013500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,0],"acting":[9,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"},{"start":"1ab","length":"1"}]},{"pgid":"2.47","version":"922'1328","reported_seq":5688,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-09-11T03:52:20.167549+0000","last_change":"2024-09-11T03:52:20.167752+0000","last_active":"2024-09-11T03:52:20.167549+0000","last_peered":"2024-09-11T03:52:20.167549+0000","last_clean":"2024-09-11T03:52:20.167549+0000","last_became_active":"2024-09-11T03:51:20.064024+0000","last_became_peered":"2024-09-11T03:51:20.064024+0000","last_unstale":"2024-09-11T03:52:20.167549+0000","last_undegraded":"2024-09-11T03:52:20.167549+0000","last_fullsized":"2024-09-11T03:52:20.167549+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":7,"last_scrub":"862'1274","last_scrub_stamp":"2024-09-11T03:51:20.166795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.166795+0000","objects_scrubbed":7,"log_size":1328,"log_dups_size":0,"ondisk_log_size":1328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:39.354204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0087906349999999998,"stat_sum":{"num_bytes":19286976,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":11528,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.46","version":"269'171","reported_seq":4313,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.173116+0000","last_change":"2024-09-11T03:52:20.161255+0000","last_active":"2024-09-11T03:52:21.173116+0000","last_peered":"2024-09-11T03:52:21.173116+0000","last_clean":"2024-09-11T03:52:21.173116+0000","last_became_active":"2024-09-11T03:51:13.080822+0000","last_became_peered":"2024-09-11T03:51:13.080822+0000","last_unstale":"2024-09-11T03:52:21.173116+0000","last_undegraded":"2024-09-11T03:52:21.173116+0000","last_fullsized":"2024-09-11T03:52:21.173116+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:51:21.034557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.034557+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:41.521016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001235454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.45","version":"922'1251","reported_seq":5452,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172697+0000","last_change":"2024-09-11T03:52:20.173911+0000","last_active":"2024-09-11T03:52:21.172697+0000","last_peered":"2024-09-11T03:52:21.172697+0000","last_clean":"2024-09-11T03:52:21.172697+0000","last_became_active":"2024-09-11T03:51:13.078704+0000","last_became_peered":"2024-09-11T03:51:13.078704+0000","last_unstale":"2024-09-11T03:52:21.172697+0000","last_undegraded":"2024-09-11T03:52:21.172697+0000","last_fullsized":"2024-09-11T03:52:21.172697+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"863'1213","last_scrub_stamp":"2024-09-11T03:51:23.430916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.430916+0000","objects_scrubbed":4,"log_size":1251,"log_dups_size":0,"ondisk_log_size":1251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:41.924227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.021061064000000001,"stat_sum":{"num_bytes":11345781,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":6,"num_write_kb":1354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[1,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.44","version":"922'771","reported_seq":5084,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172365+0000","last_change":"2024-09-11T03:52:20.164616+0000","last_active":"2024-09-11T03:52:21.172365+0000","last_peered":"2024-09-11T03:52:21.172365+0000","last_clean":"2024-09-11T03:52:21.172365+0000","last_became_active":"2024-09-11T03:51:13.079319+0000","last_became_peered":"2024-09-11T03:51:13.079319+0000","last_unstale":"2024-09-11T03:52:21.172365+0000","last_undegraded":"2024-09-11T03:52:21.172365+0000","last_fullsized":"2024-09-11T03:52:21.172365+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"862'706","last_scrub_stamp":"2024-09-11T03:51:20.328187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.328187+0000","objects_scrubbed":12,"log_size":771,"log_dups_size":0,"ondisk_log_size":771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:43.206799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.012269161000000001,"stat_sum":{"num_bytes":19161729,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":8,"num_write_kb":7615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,3],"acting":[1,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.43","version":"922'634","reported_seq":4856,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172142+0000","last_change":"2024-09-11T03:52:20.180308+0000","last_active":"2024-09-11T03:52:21.172142+0000","last_peered":"2024-09-11T03:52:21.172142+0000","last_clean":"2024-09-11T03:52:21.172142+0000","last_became_active":"2024-09-11T03:51:13.083077+0000","last_became_peered":"2024-09-11T03:51:13.083077+0000","last_unstale":"2024-09-11T03:52:21.172142+0000","last_undegraded":"2024-09-11T03:52:21.172142+0000","last_fullsized":"2024-09-11T03:52:21.172142+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"862'574","last_scrub_stamp":"2024-09-11T03:51:22.027988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.027988+0000","objects_scrubbed":9,"log_size":634,"log_dups_size":0,"ondisk_log_size":634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:51.350424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.017138318999999999,"stat_sum":{"num_bytes":26055302,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":13,"num_read_kb":11849,"num_write":8,"num_write_kb":11222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.42","version":"922'358","reported_seq":4562,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172918+0000","last_change":"2024-09-11T03:52:20.183521+0000","last_active":"2024-09-11T03:52:21.172918+0000","last_peered":"2024-09-11T03:52:21.172918+0000","last_clean":"2024-09-11T03:52:21.172918+0000","last_became_active":"2024-09-11T03:51:13.080914+0000","last_became_peered":"2024-09-11T03:51:13.080914+0000","last_unstale":"2024-09-11T03:52:21.172918+0000","last_undegraded":"2024-09-11T03:52:21.172918+0000","last_fullsized":"2024-09-11T03:52:21.172918+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"859'323","last_scrub_stamp":"2024-09-11T03:51:19.628572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.628572+0000","objects_scrubbed":5,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:31.229991+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.026820111000000001,"stat_sum":{"num_bytes":11950767,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,11],"acting":[8,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.41","version":"567'235","reported_seq":4360,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.068115+0000","last_change":"2024-09-11T03:52:20.160413+0000","last_active":"2024-09-11T03:52:22.068115+0000","last_peered":"2024-09-11T03:52:22.068115+0000","last_clean":"2024-09-11T03:52:22.068115+0000","last_became_active":"2024-09-11T03:50:33.046488+0000","last_became_peered":"2024-09-11T03:50:33.046488+0000","last_unstale":"2024-09-11T03:52:22.068115+0000","last_undegraded":"2024-09-11T03:52:22.068115+0000","last_fullsized":"2024-09-11T03:52:22.068115+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":7,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:52:09.455980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:09.455980+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:31.742444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012649180000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.40","version":"563'192","reported_seq":4429,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.072041+0000","last_change":"2024-09-11T03:52:20.154234+0000","last_active":"2024-09-11T03:52:22.072041+0000","last_peered":"2024-09-11T03:52:22.072041+0000","last_clean":"2024-09-11T03:52:22.072041+0000","last_became_active":"2024-09-11T03:51:20.064866+0000","last_became_peered":"2024-09-11T03:51:20.064866+0000","last_unstale":"2024-09-11T03:52:22.072041+0000","last_undegraded":"2024-09-11T03:52:22.072041+0000","last_fullsized":"2024-09-11T03:52:22.072041+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":7,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:52:17.055805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:17.055805+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:41.853920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00062238899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,8],"acting":[6,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.3f","version":"922'816","reported_seq":4986,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.173123+0000","last_change":"2024-09-11T03:52:20.168417+0000","last_active":"2024-09-11T03:52:21.173123+0000","last_peered":"2024-09-11T03:52:21.173123+0000","last_clean":"2024-09-11T03:52:21.173123+0000","last_became_active":"2024-09-11T03:50:33.046798+0000","last_became_peered":"2024-09-11T03:50:33.046798+0000","last_unstale":"2024-09-11T03:52:21.173123+0000","last_undegraded":"2024-09-11T03:52:21.173123+0000","last_fullsized":"2024-09-11T03:52:21.173123+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"894'803","last_scrub_stamp":"2024-09-11T03:51:53.038706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.038706+0000","objects_scrubbed":5,"log_size":816,"log_dups_size":0,"ondisk_log_size":816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:53.670332+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0077079820000000004,"stat_sum":{"num_bytes":13313731,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":18511,"num_write":10,"num_write_kb":2923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.3e","version":"922'1118","reported_seq":5243,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.073745+0000","last_change":"2024-09-11T03:52:20.180815+0000","last_active":"2024-09-11T03:52:22.073745+0000","last_peered":"2024-09-11T03:52:22.073745+0000","last_clean":"2024-09-11T03:52:22.073745+0000","last_became_active":"2024-09-11T03:50:33.045235+0000","last_became_peered":"2024-09-11T03:50:33.045235+0000","last_unstale":"2024-09-11T03:52:22.073745+0000","last_undegraded":"2024-09-11T03:52:22.073745+0000","last_fullsized":"2024-09-11T03:52:22.073745+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"893'1098","last_scrub_stamp":"2024-09-11T03:51:52.144830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.144830+0000","objects_scrubbed":7,"log_size":1118,"log_dups_size":0,"ondisk_log_size":1118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:02.554161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.022928071000000001,"stat_sum":{"num_bytes":12854600,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.3d","version":"567'752","reported_seq":3699,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.173078+0000","last_change":"2024-09-11T03:52:20.166660+0000","last_active":"2024-09-11T03:52:21.173078+0000","last_peered":"2024-09-11T03:52:21.173078+0000","last_clean":"2024-09-11T03:52:21.173078+0000","last_became_active":"2024-09-11T03:51:20.070161+0000","last_became_peered":"2024-09-11T03:51:20.070161+0000","last_unstale":"2024-09-11T03:52:21.173078+0000","last_undegraded":"2024-09-11T03:52:21.173078+0000","last_fullsized":"2024-09-11T03:52:21.173078+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:51:53.597428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.597428+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:03.645498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.012835619,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.3c","version":"921'796","reported_seq":4982,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.171927+0000","last_change":"2024-09-11T03:52:20.168803+0000","last_active":"2024-09-11T03:52:21.171927+0000","last_peered":"2024-09-11T03:52:21.171927+0000","last_clean":"2024-09-11T03:52:21.171927+0000","last_became_active":"2024-09-11T03:50:33.048487+0000","last_became_peered":"2024-09-11T03:50:33.048487+0000","last_unstale":"2024-09-11T03:52:21.171927+0000","last_undegraded":"2024-09-11T03:52:21.171927+0000","last_fullsized":"2024-09-11T03:52:21.171927+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"886'783","last_scrub_stamp":"2024-09-11T03:51:45.009597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:45.009597+0000","objects_scrubbed":3,"log_size":796,"log_dups_size":0,"ondisk_log_size":796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:45.375848+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0063809289999999996,"stat_sum":{"num_bytes":9848150,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":24464,"num_write":11,"num_write_kb":5447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,10],"acting":[7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.3b","version":"922'993","reported_seq":5385,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.171418+0000","last_change":"2024-09-11T03:52:20.182604+0000","last_active":"2024-09-11T03:52:21.171418+0000","last_peered":"2024-09-11T03:52:21.171418+0000","last_clean":"2024-09-11T03:52:21.171418+0000","last_became_active":"2024-09-11T03:51:20.066390+0000","last_became_peered":"2024-09-11T03:51:20.066390+0000","last_unstale":"2024-09-11T03:52:21.171418+0000","last_undegraded":"2024-09-11T03:52:21.171418+0000","last_fullsized":"2024-09-11T03:52:21.171418+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"905'970","last_scrub_stamp":"2024-09-11T03:52:10.225202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:10.225202+0000","objects_scrubbed":9,"log_size":993,"log_dups_size":0,"ondisk_log_size":993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:23.272958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.024130183999999999,"stat_sum":{"num_bytes":25612547,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":50520,"num_write":32,"num_write_kb":6921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":14555611,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.3a","version":"557'273","reported_seq":3200,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-09-11T03:52:23.071680+0000","last_change":"2024-09-11T03:52:23.071902+0000","last_active":"2024-09-11T03:52:23.071680+0000","last_peered":"2024-09-11T03:52:23.071680+0000","last_clean":"2024-09-11T03:52:23.071680+0000","last_became_active":"2024-09-11T03:51:20.061387+0000","last_became_peered":"2024-09-11T03:51:20.061387+0000","last_unstale":"2024-09-11T03:52:23.071680+0000","last_undegraded":"2024-09-11T03:52:23.071680+0000","last_fullsized":"2024-09-11T03:52:23.071680+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:50:37.286055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:37.286055+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001189975,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"},{"start":"1ab","length":"1"}]},{"pgid":"2.39","version":"806'585","reported_seq":4813,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-09-11T03:52:23.071963+0000","last_change":"2024-09-11T03:52:23.072198+0000","last_active":"2024-09-11T03:52:23.071963+0000","last_peered":"2024-09-11T03:52:23.071963+0000","last_clean":"2024-09-11T03:52:23.071963+0000","last_became_active":"2024-09-11T03:50:33.043651+0000","last_became_peered":"2024-09-11T03:50:33.043651+0000","last_unstale":"2024-09-11T03:52:23.071963+0000","last_undegraded":"2024-09-11T03:52:23.071963+0000","last_fullsized":"2024-09-11T03:52:23.071963+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"806'585","last_scrub_stamp":"2024-09-11T03:52:19.355191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:19.355191+0000","objects_scrubbed":0,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:38.764164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056705099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,2],"acting":[9,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"},{"start":"1ab","length":"1"}]},{"pgid":"2.38","version":"808'564","reported_seq":4791,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.171645+0000","last_change":"2024-09-11T03:52:20.173210+0000","last_active":"2024-09-11T03:52:21.171645+0000","last_peered":"2024-09-11T03:52:21.171645+0000","last_clean":"2024-09-11T03:52:21.171645+0000","last_became_active":"2024-09-11T03:50:33.042740+0000","last_became_peered":"2024-09-11T03:50:33.042740+0000","last_unstale":"2024-09-11T03:52:21.171645+0000","last_undegraded":"2024-09-11T03:52:21.171645+0000","last_fullsized":"2024-09-11T03:52:21.171645+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"808'564","last_scrub_stamp":"2024-09-11T03:51:54.462104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.462104+0000","objects_scrubbed":0,"log_size":564,"log_dups_size":0,"ondisk_log_size":564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:12.257587+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.018231682999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,1],"acting":[8,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.37","version":"274'312","reported_seq":2765,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.068422+0000","last_change":"2024-09-11T03:52:20.170366+0000","last_active":"2024-09-11T03:52:22.068422+0000","last_peered":"2024-09-11T03:52:22.068422+0000","last_clean":"2024-09-11T03:52:22.068422+0000","last_became_active":"2024-09-11T03:50:07.077607+0000","last_became_peered":"2024-09-11T03:50:07.077607+0000","last_unstale":"2024-09-11T03:52:22.068422+0000","last_undegraded":"2024-09-11T03:52:22.068422+0000","last_fullsized":"2024-09-11T03:52:22.068422+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:51:23.036386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.036386+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:34.658536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001325372,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,9],"acting":[10,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.36","version":"922'730","reported_seq":4949,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172336+0000","last_change":"2024-09-11T03:52:20.173074+0000","last_active":"2024-09-11T03:52:21.172336+0000","last_peered":"2024-09-11T03:52:21.172336+0000","last_clean":"2024-09-11T03:52:21.172336+0000","last_became_active":"2024-09-11T03:50:07.078667+0000","last_became_peered":"2024-09-11T03:50:07.078667+0000","last_unstale":"2024-09-11T03:52:21.172336+0000","last_undegraded":"2024-09-11T03:52:21.172336+0000","last_fullsized":"2024-09-11T03:52:21.172336+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"885'670","last_scrub_stamp":"2024-09-11T03:51:43.529996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:43.529996+0000","objects_scrubbed":15,"log_size":730,"log_dups_size":0,"ondisk_log_size":730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:46.888704+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.021219986999999999,"stat_sum":{"num_bytes":26555746,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":41,"num_read_kb":41151,"num_write":11,"num_write_kb":9394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,8],"acting":[1,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.35","version":"274'273","reported_seq":1932,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.173142+0000","last_change":"2024-09-11T03:52:20.173543+0000","last_active":"2024-09-11T03:52:21.173142+0000","last_peered":"2024-09-11T03:52:21.173142+0000","last_clean":"2024-09-11T03:52:21.173142+0000","last_became_active":"2024-09-11T03:50:07.079331+0000","last_became_peered":"2024-09-11T03:50:07.079331+0000","last_unstale":"2024-09-11T03:52:21.173142+0000","last_undegraded":"2024-09-11T03:52:21.173142+0000","last_fullsized":"2024-09-11T03:52:21.173142+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:51:28.526090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.526090+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:41.029754+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.020456643,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.34","version":"922'411","reported_seq":4578,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172887+0000","last_change":"2024-09-11T03:52:20.173276+0000","last_active":"2024-09-11T03:52:21.172887+0000","last_peered":"2024-09-11T03:52:21.172887+0000","last_clean":"2024-09-11T03:52:21.172887+0000","last_became_active":"2024-09-11T03:50:07.080419+0000","last_became_peered":"2024-09-11T03:50:07.080419+0000","last_unstale":"2024-09-11T03:52:21.172887+0000","last_undegraded":"2024-09-11T03:52:21.172887+0000","last_fullsized":"2024-09-11T03:52:21.172887+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"863'388","last_scrub_stamp":"2024-09-11T03:51:22.429302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.429302+0000","objects_scrubbed":6,"log_size":411,"log_dups_size":0,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:42.661101+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.015009097000000001,"stat_sum":{"num_bytes":14306205,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":37,"num_read_kb":37368,"num_write":13,"num_write_kb":3256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.33","version":"274'111","reported_seq":1720,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172439+0000","last_change":"2024-09-11T03:52:20.180687+0000","last_active":"2024-09-11T03:52:21.172439+0000","last_peered":"2024-09-11T03:52:21.172439+0000","last_clean":"2024-09-11T03:52:21.172439+0000","last_became_active":"2024-09-11T03:50:07.077684+0000","last_became_peered":"2024-09-11T03:50:07.077684+0000","last_unstale":"2024-09-11T03:52:21.172439+0000","last_undegraded":"2024-09-11T03:52:21.172439+0000","last_fullsized":"2024-09-11T03:52:21.172439+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:51:28.001189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.001189+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:56.429740+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.01751757,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.32","version":"922'611","reported_seq":4805,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.072132+0000","last_change":"2024-09-11T03:52:20.173250+0000","last_active":"2024-09-11T03:52:22.072132+0000","last_peered":"2024-09-11T03:52:22.072132+0000","last_clean":"2024-09-11T03:52:22.072132+0000","last_became_active":"2024-09-11T03:51:20.067057+0000","last_became_peered":"2024-09-11T03:51:20.067057+0000","last_unstale":"2024-09-11T03:52:22.072132+0000","last_undegraded":"2024-09-11T03:52:22.072132+0000","last_fullsized":"2024-09-11T03:52:22.072132+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"885'574","last_scrub_stamp":"2024-09-11T03:51:44.094173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:44.094173+0000","objects_scrubbed":3,"log_size":611,"log_dups_size":0,"ondisk_log_size":611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:57.263852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.015757105,"stat_sum":{"num_bytes":7204618,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":10413,"num_write":10,"num_write_kb":2970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":2772362,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.31","version":"274'107","reported_seq":2214,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-09-11T03:52:20.159218+0000","last_change":"2024-09-11T03:52:20.159695+0000","last_active":"2024-09-11T03:52:20.159218+0000","last_peered":"2024-09-11T03:52:20.159218+0000","last_clean":"2024-09-11T03:52:20.159218+0000","last_became_active":"2024-09-11T03:51:20.060052+0000","last_became_peered":"2024-09-11T03:51:20.060052+0000","last_unstale":"2024-09-11T03:52:20.159218+0000","last_undegraded":"2024-09-11T03:52:20.159218+0000","last_fullsized":"2024-09-11T03:52:20.159218+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:51:26.087112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:26.087112+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:38.904694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018955910000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,7],"acting":[0,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.30","version":"274'67","reported_seq":691,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.067968+0000","last_change":"2024-09-11T03:52:20.161486+0000","last_active":"2024-09-11T03:52:22.067968+0000","last_peered":"2024-09-11T03:52:22.067968+0000","last_clean":"2024-09-11T03:52:22.067968+0000","last_became_active":"2024-09-11T03:51:17.065156+0000","last_became_peered":"2024-09-11T03:51:17.065156+0000","last_unstale":"2024-09-11T03:52:22.067968+0000","last_undegraded":"2024-09-11T03:52:22.067968+0000","last_fullsized":"2024-09-11T03:52:22.067968+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:51:21.989674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.989674+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:25.399420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00085413199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.2f","version":"557'465","reported_seq":3543,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-09-11T03:52:23.072957+0000","last_change":"2024-09-11T03:52:23.072957+0000","last_active":"2024-09-11T03:52:23.072957+0000","last_peered":"2024-09-11T03:52:23.072957+0000","last_clean":"2024-09-11T03:52:23.072957+0000","last_became_active":"2024-09-11T03:50:07.077840+0000","last_became_peered":"2024-09-11T03:50:07.077840+0000","last_unstale":"2024-09-11T03:52:23.072957+0000","last_undegraded":"2024-09-11T03:52:23.072957+0000","last_fullsized":"2024-09-11T03:52:23.072957+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:51:47.522706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:47.522706+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:15.784816+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012088089999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"},{"start":"1ab","length":"1"}]},{"pgid":"2.2e","version":"922'1249","reported_seq":5322,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-09-11T03:52:20.173992+0000","last_change":"2024-09-11T03:52:20.174469+0000","last_active":"2024-09-11T03:52:20.173992+0000","last_peered":"2024-09-11T03:52:20.173992+0000","last_clean":"2024-09-11T03:52:20.173992+0000","last_became_active":"2024-09-11T03:51:31.069593+0000","last_became_peered":"2024-09-11T03:51:31.069593+0000","last_unstale":"2024-09-11T03:52:20.173992+0000","last_undegraded":"2024-09-11T03:52:20.173992+0000","last_fullsized":"2024-09-11T03:52:20.173992+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":6,"last_scrub":"868'1145","last_scrub_stamp":"2024-09-11T03:51:27.034758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:27.034758+0000","objects_scrubbed":11,"log_size":1249,"log_dups_size":0,"ondisk_log_size":1249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:45.449328+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.015206956000000001,"stat_sum":{"num_bytes":35201644,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":37,"num_read_kb":34191,"num_write":34,"num_write_kb":12756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":24991924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,8],"acting":[0,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.2d","version":"922'1365","reported_seq":5524,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.077189+0000","last_change":"2024-09-11T03:52:20.175713+0000","last_active":"2024-09-11T03:52:22.077189+0000","last_peered":"2024-09-11T03:52:22.077189+0000","last_clean":"2024-09-11T03:52:22.077189+0000","last_became_active":"2024-09-11T03:49:29.163201+0000","last_became_peered":"2024-09-11T03:49:29.163201+0000","last_unstale":"2024-09-11T03:52:22.077189+0000","last_undegraded":"2024-09-11T03:52:22.077189+0000","last_fullsized":"2024-09-11T03:52:22.077189+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"921'1359","last_scrub_stamp":"2024-09-11T03:52:20.071629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:20.071629+0000","objects_scrubbed":20,"log_size":1365,"log_dups_size":0,"ondisk_log_size":1365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:27.620316+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.020661736999999999,"stat_sum":{"num_bytes":44322843,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":167,"num_read_kb":224508,"num_write":52,"num_write_kb":24676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.2c","version":"922'971","reported_seq":5137,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.171961+0000","last_change":"2024-09-11T03:52:20.172810+0000","last_active":"2024-09-11T03:52:21.171961+0000","last_peered":"2024-09-11T03:52:21.171961+0000","last_clean":"2024-09-11T03:52:21.171961+0000","last_became_active":"2024-09-11T03:49:29.154872+0000","last_became_peered":"2024-09-11T03:49:29.154872+0000","last_unstale":"2024-09-11T03:52:21.171961+0000","last_undegraded":"2024-09-11T03:52:21.171961+0000","last_fullsized":"2024-09-11T03:52:21.171961+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"908'959","last_scrub_stamp":"2024-09-11T03:52:08.323831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:08.323831+0000","objects_scrubbed":9,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:30.612717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.018236582000000001,"stat_sum":{"num_bytes":22924800,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":47013,"num_write":50,"num_write_kb":17751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3],"acting":[8,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.2b","version":"568'646","reported_seq":3950,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-09-11T03:52:23.075721+0000","last_change":"2024-09-11T03:52:23.076311+0000","last_active":"2024-09-11T03:52:23.075721+0000","last_peered":"2024-09-11T03:52:23.075721+0000","last_clean":"2024-09-11T03:52:23.075721+0000","last_became_active":"2024-09-11T03:51:20.062485+0000","last_became_peered":"2024-09-11T03:51:20.062485+0000","last_unstale":"2024-09-11T03:52:23.075721+0000","last_undegraded":"2024-09-11T03:52:23.075721+0000","last_fullsized":"2024-09-11T03:52:23.075721+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:51:52.556581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.556581+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:53.856887+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0032286789999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"},{"start":"1ab","length":"1"}]},{"pgid":"2.2a","version":"727'348","reported_seq":4417,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172566+0000","last_change":"2024-09-11T03:52:20.161635+0000","last_active":"2024-09-11T03:52:21.172566+0000","last_peered":"2024-09-11T03:52:21.172566+0000","last_clean":"2024-09-11T03:52:21.172566+0000","last_became_active":"2024-09-11T03:49:29.160595+0000","last_became_peered":"2024-09-11T03:49:29.160595+0000","last_unstale":"2024-09-11T03:52:21.172566+0000","last_undegraded":"2024-09-11T03:52:21.172566+0000","last_fullsized":"2024-09-11T03:52:21.172566+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:52:12.045079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:12.045079+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:29.481901+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0016035890000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,5],"acting":[7,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.29","version":"922'1362","reported_seq":5628,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.073354+0000","last_change":"2024-09-11T03:52:20.173867+0000","last_active":"2024-09-11T03:52:22.073354+0000","last_peered":"2024-09-11T03:52:22.073354+0000","last_clean":"2024-09-11T03:52:22.073354+0000","last_became_active":"2024-09-11T03:49:29.157194+0000","last_became_peered":"2024-09-11T03:49:29.157194+0000","last_unstale":"2024-09-11T03:52:22.073354+0000","last_undegraded":"2024-09-11T03:52:22.073354+0000","last_fullsized":"2024-09-11T03:52:22.073354+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"838'1272","last_scrub_stamp":"2024-09-11T03:50:57.773854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:57.773854+0000","objects_scrubbed":14,"log_size":1362,"log_dups_size":0,"ondisk_log_size":1362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.019091606000000001,"stat_sum":{"num_bytes":30548483,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":64,"num_read_kb":79538,"num_write":46,"num_write_kb":15472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8],"acting":[6,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.28","version":"921'481","reported_seq":4652,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.171862+0000","last_change":"2024-09-11T03:52:20.152600+0000","last_active":"2024-09-11T03:52:21.171862+0000","last_peered":"2024-09-11T03:52:21.171862+0000","last_clean":"2024-09-11T03:52:21.171862+0000","last_became_active":"2024-09-11T03:49:29.154884+0000","last_became_peered":"2024-09-11T03:49:29.154884+0000","last_unstale":"2024-09-11T03:52:21.171862+0000","last_undegraded":"2024-09-11T03:52:21.171862+0000","last_fullsized":"2024-09-11T03:52:21.171862+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"905'477","last_scrub_stamp":"2024-09-11T03:52:07.670614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:07.670614+0000","objects_scrubbed":6,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:26.390057+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00094377899999999995,"stat_sum":{"num_bytes":14169093,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":14,"num_read_kb":11529,"num_write":23,"num_write_kb":4588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.27","version":"922'1277","reported_seq":5660,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.173597+0000","last_change":"2024-09-11T03:52:20.180010+0000","last_active":"2024-09-11T03:52:21.173597+0000","last_peered":"2024-09-11T03:52:21.173597+0000","last_clean":"2024-09-11T03:52:21.173597+0000","last_became_active":"2024-09-11T03:51:20.062110+0000","last_became_peered":"2024-09-11T03:51:20.062110+0000","last_unstale":"2024-09-11T03:52:21.173597+0000","last_undegraded":"2024-09-11T03:52:21.173597+0000","last_fullsized":"2024-09-11T03:52:21.173597+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"909'1255","last_scrub_stamp":"2024-09-11T03:52:10.054644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:10.054644+0000","objects_scrubbed":13,"log_size":1277,"log_dups_size":0,"ondisk_log_size":1277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:33.570604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.019030629,"stat_sum":{"num_bytes":28771786,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":56,"num_read_kb":59674,"num_write":47,"num_write_kb":23843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":14737969,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.26","version":"269'171","reported_seq":4307,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.173166+0000","last_change":"2024-09-11T03:52:20.161112+0000","last_active":"2024-09-11T03:52:21.173166+0000","last_peered":"2024-09-11T03:52:21.173166+0000","last_clean":"2024-09-11T03:52:21.173166+0000","last_became_active":"2024-09-11T03:51:20.061265+0000","last_became_peered":"2024-09-11T03:51:20.061265+0000","last_unstale":"2024-09-11T03:52:21.173166+0000","last_undegraded":"2024-09-11T03:52:21.173166+0000","last_fullsized":"2024-09-11T03:52:21.173166+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:52:08.043740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:08.043740+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:11.946517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012162200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,5],"acting":[7,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.25","version":"922'1604","reported_seq":5948,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.180719+0000","last_change":"2024-09-11T03:52:20.179003+0000","last_active":"2024-09-11T03:52:21.180719+0000","last_peered":"2024-09-11T03:52:21.180719+0000","last_clean":"2024-09-11T03:52:21.180719+0000","last_became_active":"2024-09-11T03:49:29.159733+0000","last_became_peered":"2024-09-11T03:49:29.159733+0000","last_unstale":"2024-09-11T03:52:21.180719+0000","last_undegraded":"2024-09-11T03:52:21.180719+0000","last_fullsized":"2024-09-11T03:52:21.180719+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"918'1582","last_scrub_stamp":"2024-09-11T03:52:17.392959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:17.392959+0000","objects_scrubbed":23,"log_size":1604,"log_dups_size":0,"ondisk_log_size":1604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:44.617348+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.024410291000000001,"stat_sum":{"num_bytes":43829442,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":1,"num_read":97,"num_read_kb":94378,"num_write":82,"num_write_kb":23809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,6],"acting":[8,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.24","version":"922'748","reported_seq":4930,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.073165+0000","last_change":"2024-09-11T03:52:20.168867+0000","last_active":"2024-09-11T03:52:22.073165+0000","last_peered":"2024-09-11T03:52:22.073165+0000","last_clean":"2024-09-11T03:52:22.073165+0000","last_became_active":"2024-09-11T03:49:29.155740+0000","last_became_peered":"2024-09-11T03:49:29.155740+0000","last_unstale":"2024-09-11T03:52:22.073165+0000","last_undegraded":"2024-09-11T03:52:22.073165+0000","last_fullsized":"2024-09-11T03:52:22.073165+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"838'674","last_scrub_stamp":"2024-09-11T03:50:56.770184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:56.770184+0000","objects_scrubbed":7,"log_size":748,"log_dups_size":0,"ondisk_log_size":748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:23.636299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014030271,"stat_sum":{"num_bytes":21387208,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":17,"num_write_kb":14099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,7],"acting":[6,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.23","version":"922'570","reported_seq":4720,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172375+0000","last_change":"2024-09-11T03:52:20.156703+0000","last_active":"2024-09-11T03:52:21.172375+0000","last_peered":"2024-09-11T03:52:21.172375+0000","last_clean":"2024-09-11T03:52:21.172375+0000","last_became_active":"2024-09-11T03:46:32.970100+0000","last_became_peered":"2024-09-11T03:46:32.970100+0000","last_unstale":"2024-09-11T03:52:21.172375+0000","last_undegraded":"2024-09-11T03:52:21.172375+0000","last_fullsized":"2024-09-11T03:52:21.172375+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"886'531","last_scrub_stamp":"2024-09-11T03:51:44.527530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:44.527530+0000","objects_scrubbed":5,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:45.730802+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0046520340000000002,"stat_sum":{"num_bytes":11706449,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":57,"num_read_kb":64783,"num_write":39,"num_write_kb":13976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.22","version":"531'216","reported_seq":4347,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.072269+0000","last_change":"2024-09-11T03:52:20.156028+0000","last_active":"2024-09-11T03:52:22.072269+0000","last_peered":"2024-09-11T03:52:22.072269+0000","last_clean":"2024-09-11T03:52:22.072269+0000","last_became_active":"2024-09-11T03:46:32.970633+0000","last_became_peered":"2024-09-11T03:46:32.970633+0000","last_unstale":"2024-09-11T03:52:22.072269+0000","last_undegraded":"2024-09-11T03:52:22.072269+0000","last_fullsized":"2024-09-11T03:52:22.072269+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:50:38.617374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:38.617374+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001890351,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.21","version":"922'421","reported_seq":4658,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.072770+0000","last_change":"2024-09-11T03:52:20.181623+0000","last_active":"2024-09-11T03:52:22.072770+0000","last_peered":"2024-09-11T03:52:22.072770+0000","last_clean":"2024-09-11T03:52:22.072770+0000","last_became_active":"2024-09-11T03:46:36.024928+0000","last_became_peered":"2024-09-11T03:46:36.024928+0000","last_unstale":"2024-09-11T03:52:22.072770+0000","last_undegraded":"2024-09-11T03:52:22.072770+0000","last_fullsized":"2024-09-11T03:52:22.072770+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":6,"last_scrub":"910'410","last_scrub_stamp":"2024-09-11T03:52:13.122711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:13.122711+0000","objects_scrubbed":7,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:32.858214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.022676343000000002,"stat_sum":{"num_bytes":17554351,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":72,"num_read_kb":79649,"num_write":55,"num_write_kb":22944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,8],"acting":[11,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.20","version":"923'374","reported_seq":4675,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.178334+0000","last_change":"2024-09-11T03:52:20.171341+0000","last_active":"2024-09-11T03:52:21.178334+0000","last_peered":"2024-09-11T03:52:21.178334+0000","last_clean":"2024-09-11T03:52:21.178334+0000","last_became_active":"2024-09-11T03:46:32.974375+0000","last_became_peered":"2024-09-11T03:46:32.974375+0000","last_unstale":"2024-09-11T03:52:21.178334+0000","last_undegraded":"2024-09-11T03:52:21.178334+0000","last_fullsized":"2024-09-11T03:52:21.178334+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"895'348","last_scrub_stamp":"2024-09-11T03:51:54.026522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.026522+0000","objects_scrubbed":6,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:03.358080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0084998669999999995,"stat_sum":{"num_bytes":18825325,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":98,"num_read_kb":144174,"num_write":27,"num_write_kb":13274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.1c","version":"806'776","reported_seq":4919,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.068279+0000","last_change":"2024-09-11T03:52:20.164894+0000","last_active":"2024-09-11T03:52:22.068279+0000","last_peered":"2024-09-11T03:52:22.068279+0000","last_clean":"2024-09-11T03:52:22.068279+0000","last_became_active":"2024-09-11T03:50:28.047548+0000","last_became_peered":"2024-09-11T03:50:28.047548+0000","last_unstale":"2024-09-11T03:52:22.068279+0000","last_undegraded":"2024-09-11T03:52:22.068279+0000","last_fullsized":"2024-09-11T03:52:22.068279+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"806'776","last_scrub_stamp":"2024-09-11T03:51:54.156180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.156180+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:20.167165+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027719260000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":913,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-09-11T03:52:20.158144+0000","last_change":"2024-09-11T03:52:17.994964+0000","last_active":"2024-09-11T03:52:20.158144+0000","last_peered":"2024-09-11T03:52:20.158144+0000","last_clean":"2024-09-11T03:52:20.158144+0000","last_became_active":"2024-09-11T03:46:15.974786+0000","last_became_peered":"2024-09-11T03:46:15.974786+0000","last_unstale":"2024-09-11T03:52:20.158144+0000","last_undegraded":"2024-09-11T03:52:20.158144+0000","last_fullsized":"2024-09-11T03:52:20.158144+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:17.994888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:17.994888+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:25.801830+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.f","version":"557'465","reported_seq":4647,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.173488+0000","last_change":"2024-09-11T03:52:21.172633+0000","last_active":"2024-09-11T03:52:21.173488+0000","last_peered":"2024-09-11T03:52:21.173488+0000","last_clean":"2024-09-11T03:52:21.173488+0000","last_became_active":"2024-09-11T03:52:20.175033+0000","last_became_peered":"2024-09-11T03:52:20.175033+0000","last_unstale":"2024-09-11T03:52:21.173488+0000","last_undegraded":"2024-09-11T03:52:21.173488+0000","last_fullsized":"2024-09-11T03:52:21.173488+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:51:25.558029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:25.558029+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0065909250000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,11],"acting":[8,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"2.e","version":"782'1025","reported_seq":5162,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-09-11T03:52:23.073436+0000","last_change":"2024-09-11T03:52:23.073692+0000","last_active":"2024-09-11T03:52:23.073436+0000","last_peered":"2024-09-11T03:52:23.073436+0000","last_clean":"2024-09-11T03:52:23.073436+0000","last_became_active":"2024-09-11T03:52:20.164829+0000","last_became_peered":"2024-09-11T03:52:20.164829+0000","last_unstale":"2024-09-11T03:52:23.073436+0000","last_undegraded":"2024-09-11T03:52:23.073436+0000","last_fullsized":"2024-09-11T03:52:23.073436+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":0,"last_scrub":"782'1025","last_scrub_stamp":"2024-09-11T03:51:22.296245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.296245+0000","objects_scrubbed":0,"log_size":1025,"log_dups_size":0,"ondisk_log_size":1025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00096614499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"},{"start":"1ab","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":1490,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172963+0000","last_change":"2024-09-11T03:51:17.066500+0000","last_active":"2024-09-11T03:52:21.172963+0000","last_peered":"2024-09-11T03:52:21.172963+0000","last_clean":"2024-09-11T03:52:21.172963+0000","last_became_active":"2024-09-11T03:51:17.066085+0000","last_became_peered":"2024-09-11T03:51:17.066085+0000","last_unstale":"2024-09-11T03:52:21.172963+0000","last_undegraded":"2024-09-11T03:52:21.172963+0000","last_fullsized":"2024-09-11T03:52:21.172963+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:58.932376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:58.932376+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"923'1081","reported_seq":5362,"reported_epoch":923,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T03:52:21.267199+0000","last_change":"2024-09-11T03:52:21.267035+0000","last_active":"2024-09-11T03:52:21.267199+0000","last_peered":"2024-09-11T03:52:21.267199+0000","last_clean":"2024-09-11T03:52:21.267199+0000","last_became_active":"2024-09-11T03:52:20.165149+0000","last_became_peered":"2024-09-11T03:52:20.165149+0000","last_unstale":"2024-09-11T03:52:21.267199+0000","last_undegraded":"2024-09-11T03:52:21.267199+0000","last_fullsized":"2024-09-11T03:52:21.267199+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":0,"last_scrub":"916'1068","last_scrub_stamp":"2024-09-11T03:52:14.203354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:14.203354+0000","objects_scrubbed":0,"log_size":1081,"log_dups_size":0,"ondisk_log_size":1081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.014930097,"stat_sum":{"num_bytes":5670964,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":3513,"num_write":13,"num_write_kb":8372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4719477,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":1205,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.173521+0000","last_change":"2024-09-11T03:51:17.063139+0000","last_active":"2024-09-11T03:52:21.173521+0000","last_peered":"2024-09-11T03:52:21.173521+0000","last_clean":"2024-09-11T03:52:21.173521+0000","last_became_active":"2024-09-11T03:51:17.061266+0000","last_became_peered":"2024-09-11T03:51:17.061266+0000","last_unstale":"2024-09-11T03:52:21.173521+0000","last_undegraded":"2024-09-11T03:52:21.173521+0000","last_fullsized":"2024-09-11T03:52:21.173521+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:38.843438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:38.843438+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"920'966","reported_seq":5135,"reported_epoch":924,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T03:52:22.076673+0000","last_change":"2024-09-11T03:52:22.076482+0000","last_active":"2024-09-11T03:52:22.076673+0000","last_peered":"2024-09-11T03:52:22.076673+0000","last_clean":"2024-09-11T03:52:22.076673+0000","last_became_active":"2024-09-11T03:52:20.162351+0000","last_became_peered":"2024-09-11T03:52:20.162351+0000","last_unstale":"2024-09-11T03:52:22.076673+0000","last_undegraded":"2024-09-11T03:52:22.076673+0000","last_fullsized":"2024-09-11T03:52:22.076673+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":0,"last_scrub":"843'880","last_scrub_stamp":"2024-09-11T03:51:01.987099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:01.987099+0000","objects_scrubbed":0,"log_size":966,"log_dups_size":0,"ondisk_log_size":966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.018680364000000001,"stat_sum":{"num_bytes":13204636,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":16271,"num_write":12,"num_write_kb":4208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.f","version":"20'92","reported_seq":773,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.072888+0000","last_change":"2024-09-11T03:51:21.860818+0000","last_active":"2024-09-11T03:52:22.072888+0000","last_peered":"2024-09-11T03:52:22.072888+0000","last_clean":"2024-09-11T03:52:22.072888+0000","last_became_active":"2024-09-11T03:51:17.064761+0000","last_became_peered":"2024-09-11T03:51:17.064761+0000","last_unstale":"2024-09-11T03:52:22.072888+0000","last_undegraded":"2024-09-11T03:52:22.072888+0000","last_fullsized":"2024-09-11T03:52:22.072888+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:21.860769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.860769+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:34.914441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":1889,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.173919+0000","last_change":"2024-09-11T03:51:52.035008+0000","last_active":"2024-09-11T03:52:21.173919+0000","last_peered":"2024-09-11T03:52:21.173919+0000","last_clean":"2024-09-11T03:52:21.173919+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:52:21.173919+0000","last_undegraded":"2024-09-11T03:52:21.173919+0000","last_fullsized":"2024-09-11T03:52:21.173919+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:52.034950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.034950+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:03.753329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"845'562","reported_seq":4723,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.171760+0000","last_change":"2024-09-11T03:52:20.159787+0000","last_active":"2024-09-11T03:52:21.171760+0000","last_peered":"2024-09-11T03:52:21.171760+0000","last_clean":"2024-09-11T03:52:21.171760+0000","last_became_active":"2024-09-11T03:51:08.055565+0000","last_became_peered":"2024-09-11T03:51:08.055565+0000","last_unstale":"2024-09-11T03:52:21.171760+0000","last_undegraded":"2024-09-11T03:52:21.171760+0000","last_fullsized":"2024-09-11T03:52:21.171760+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"845'562","last_scrub_stamp":"2024-09-11T03:51:20.441661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.441661+0000","objects_scrubbed":0,"log_size":562,"log_dups_size":0,"ondisk_log_size":562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:28.460116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017408009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":1502,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172508+0000","last_change":"2024-09-11T03:51:27.564638+0000","last_active":"2024-09-11T03:52:21.172508+0000","last_peered":"2024-09-11T03:52:21.172508+0000","last_clean":"2024-09-11T03:52:21.172508+0000","last_became_active":"2024-09-11T03:51:17.059234+0000","last_became_peered":"2024-09-11T03:51:17.059234+0000","last_unstale":"2024-09-11T03:52:21.172508+0000","last_undegraded":"2024-09-11T03:52:21.172508+0000","last_fullsized":"2024-09-11T03:52:21.172508+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:27.564586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:27.564586+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:54.532532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"563'192","reported_seq":3218,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.068014+0000","last_change":"2024-09-11T03:52:20.162155+0000","last_active":"2024-09-11T03:52:22.068014+0000","last_peered":"2024-09-11T03:52:22.068014+0000","last_clean":"2024-09-11T03:52:22.068014+0000","last_became_active":"2024-09-11T03:50:31.044399+0000","last_became_peered":"2024-09-11T03:50:31.044399+0000","last_unstale":"2024-09-11T03:52:22.068014+0000","last_undegraded":"2024-09-11T03:52:22.068014+0000","last_fullsized":"2024-09-11T03:52:22.068014+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:52:10.500975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:10.500975+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:34.119813+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054616300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":1458,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-09-11T03:52:20.157873+0000","last_change":"2024-09-11T03:51:17.061730+0000","last_active":"2024-09-11T03:52:20.157873+0000","last_peered":"2024-09-11T03:52:20.157873+0000","last_clean":"2024-09-11T03:52:20.157873+0000","last_became_active":"2024-09-11T03:51:17.061243+0000","last_became_peered":"2024-09-11T03:51:17.061243+0000","last_unstale":"2024-09-11T03:52:20.157873+0000","last_undegraded":"2024-09-11T03:52:20.157873+0000","last_fullsized":"2024-09-11T03:52:20.157873+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:41.975660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:41.975660+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:07.081232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"567'235","reported_seq":4363,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-09-11T03:52:20.159295+0000","last_change":"2024-09-11T03:52:20.160278+0000","last_active":"2024-09-11T03:52:20.159295+0000","last_peered":"2024-09-11T03:52:20.159295+0000","last_clean":"2024-09-11T03:52:20.159295+0000","last_became_active":"2024-09-11T03:50:28.039300+0000","last_became_peered":"2024-09-11T03:50:28.039300+0000","last_unstale":"2024-09-11T03:52:20.159295+0000","last_undegraded":"2024-09-11T03:52:20.159295+0000","last_fullsized":"2024-09-11T03:52:20.159295+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:51:50.027291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:50.027291+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:11.281058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011468369999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.1","version":"20'92","reported_seq":1067,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.068543+0000","last_change":"2024-09-11T03:51:29.996670+0000","last_active":"2024-09-11T03:52:22.068543+0000","last_peered":"2024-09-11T03:52:22.068543+0000","last_clean":"2024-09-11T03:52:22.068543+0000","last_became_active":"2024-09-11T03:43:52.921293+0000","last_became_peered":"2024-09-11T03:43:52.921293+0000","last_unstale":"2024-09-11T03:52:22.068543+0000","last_undegraded":"2024-09-11T03:52:22.068543+0000","last_fullsized":"2024-09-11T03:52:22.068543+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:29.996598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:29.996598+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:54.485931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"843'319","reported_seq":4581,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.171847+0000","last_change":"2024-09-11T03:52:20.159395+0000","last_active":"2024-09-11T03:52:21.171847+0000","last_peered":"2024-09-11T03:52:21.171847+0000","last_clean":"2024-09-11T03:52:21.171847+0000","last_became_active":"2024-09-11T03:51:08.055103+0000","last_became_peered":"2024-09-11T03:51:08.055103+0000","last_unstale":"2024-09-11T03:52:21.171847+0000","last_undegraded":"2024-09-11T03:52:21.171847+0000","last_fullsized":"2024-09-11T03:52:21.171847+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"843'319","last_scrub_stamp":"2024-09-11T03:51:19.453435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.453435+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:37.232012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014236209999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":758,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.072418+0000","last_change":"2024-09-11T03:52:08.021960+0000","last_active":"2024-09-11T03:52:22.072418+0000","last_peered":"2024-09-11T03:52:22.072418+0000","last_clean":"2024-09-11T03:52:22.072418+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:52:22.072418+0000","last_undegraded":"2024-09-11T03:52:22.072418+0000","last_fullsized":"2024-09-11T03:52:22.072418+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:08.021899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:08.021899+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:37.346818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"847'690","reported_seq":4912,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172605+0000","last_change":"2024-09-11T03:52:20.165635+0000","last_active":"2024-09-11T03:52:21.172605+0000","last_peered":"2024-09-11T03:52:21.172605+0000","last_clean":"2024-09-11T03:52:21.172605+0000","last_became_active":"2024-09-11T03:51:08.058954+0000","last_became_peered":"2024-09-11T03:51:08.058954+0000","last_unstale":"2024-09-11T03:52:21.172605+0000","last_undegraded":"2024-09-11T03:52:21.172605+0000","last_fullsized":"2024-09-11T03:52:21.172605+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"847'690","last_scrub_stamp":"2024-09-11T03:51:24.473723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:24.473723+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:50.844597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012365238000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.6","version":"20'92","reported_seq":1077,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.068205+0000","last_change":"2024-09-11T03:51:53.155265+0000","last_active":"2024-09-11T03:52:22.068205+0000","last_peered":"2024-09-11T03:52:22.068205+0000","last_clean":"2024-09-11T03:52:22.068205+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:52:22.068205+0000","last_undegraded":"2024-09-11T03:52:22.068205+0000","last_fullsized":"2024-09-11T03:52:22.068205+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:53.155207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.155207+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:07.708610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"922'1275","reported_seq":5556,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.173600+0000","last_change":"2024-09-11T03:52:20.178632+0000","last_active":"2024-09-11T03:52:21.173600+0000","last_peered":"2024-09-11T03:52:21.173600+0000","last_clean":"2024-09-11T03:52:21.173600+0000","last_became_active":"2024-09-11T03:51:20.063427+0000","last_became_peered":"2024-09-11T03:51:20.063427+0000","last_unstale":"2024-09-11T03:52:21.173600+0000","last_undegraded":"2024-09-11T03:52:21.173600+0000","last_fullsized":"2024-09-11T03:52:21.173600+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"862'1215","last_scrub_stamp":"2024-09-11T03:51:21.601542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.601542+0000","objects_scrubbed":5,"log_size":1275,"log_dups_size":0,"ondisk_log_size":1275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:27.299849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.023571780000000001,"stat_sum":{"num_bytes":16924470,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":9,"num_write_kb":5648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.5","version":"20'92","reported_seq":1069,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.068676+0000","last_change":"2024-09-11T03:52:08.468064+0000","last_active":"2024-09-11T03:52:22.068676+0000","last_peered":"2024-09-11T03:52:22.068676+0000","last_clean":"2024-09-11T03:52:22.068676+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:52:22.068676+0000","last_undegraded":"2024-09-11T03:52:22.068676+0000","last_fullsized":"2024-09-11T03:52:22.068676+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:08.468014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:08.468014+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:12.305464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":4315,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172515+0000","last_change":"2024-09-11T03:52:20.157324+0000","last_active":"2024-09-11T03:52:21.172515+0000","last_peered":"2024-09-11T03:52:21.172515+0000","last_clean":"2024-09-11T03:52:21.172515+0000","last_became_active":"2024-09-11T03:51:08.055529+0000","last_became_peered":"2024-09-11T03:51:08.055529+0000","last_unstale":"2024-09-11T03:52:21.172515+0000","last_undegraded":"2024-09-11T03:52:21.172515+0000","last_fullsized":"2024-09-11T03:52:21.172515+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:51:22.407323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.407323+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:31.616228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051653439999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":1668,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.073733+0000","last_change":"2024-09-11T03:51:31.102151+0000","last_active":"2024-09-11T03:52:22.073733+0000","last_peered":"2024-09-11T03:52:22.073733+0000","last_clean":"2024-09-11T03:52:22.073733+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:52:22.073733+0000","last_undegraded":"2024-09-11T03:52:22.073733+0000","last_fullsized":"2024-09-11T03:52:22.073733+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:31.102098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:31.102098+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:49.931850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"922'1359","reported_seq":5706,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.072449+0000","last_change":"2024-09-11T03:52:20.177010+0000","last_active":"2024-09-11T03:52:22.072449+0000","last_peered":"2024-09-11T03:52:22.072449+0000","last_clean":"2024-09-11T03:52:22.072449+0000","last_became_active":"2024-09-11T03:51:08.057350+0000","last_became_peered":"2024-09-11T03:51:08.057350+0000","last_unstale":"2024-09-11T03:52:22.072449+0000","last_undegraded":"2024-09-11T03:52:22.072449+0000","last_fullsized":"2024-09-11T03:52:22.072449+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'1274","last_scrub_stamp":"2024-09-11T03:51:19.872739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.872739+0000","objects_scrubbed":12,"log_size":1359,"log_dups_size":0,"ondisk_log_size":1359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:45.735695+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.021971266999999999,"stat_sum":{"num_bytes":38649049,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":21,"num_read_kb":28479,"num_write":9,"num_write_kb":4595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":1471,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.073466+0000","last_change":"2024-09-11T03:51:47.050276+0000","last_active":"2024-09-11T03:52:22.073466+0000","last_peered":"2024-09-11T03:52:22.073466+0000","last_clean":"2024-09-11T03:52:22.073466+0000","last_became_active":"2024-09-11T03:40:37.849967+0000","last_became_peered":"2024-09-11T03:40:37.849967+0000","last_unstale":"2024-09-11T03:52:22.073466+0000","last_undegraded":"2024-09-11T03:52:22.073466+0000","last_fullsized":"2024-09-11T03:52:22.073466+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:47.050227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:47.050227+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:47.461383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"744'374","reported_seq":4596,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.173387+0000","last_change":"2024-09-11T03:52:20.180581+0000","last_active":"2024-09-11T03:52:21.173387+0000","last_peered":"2024-09-11T03:52:21.173387+0000","last_clean":"2024-09-11T03:52:21.173387+0000","last_became_active":"2024-09-11T03:51:08.057750+0000","last_became_peered":"2024-09-11T03:51:08.057750+0000","last_unstale":"2024-09-11T03:52:21.173387+0000","last_undegraded":"2024-09-11T03:52:21.173387+0000","last_fullsized":"2024-09-11T03:52:21.173387+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'374","last_scrub_stamp":"2024-09-11T03:51:20.010112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.010112+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:46.873051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017316697999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.a","version":"20'92","reported_seq":1825,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-09-11T03:52:23.071079+0000","last_change":"2024-09-11T03:51:17.061700+0000","last_active":"2024-09-11T03:52:23.071079+0000","last_peered":"2024-09-11T03:52:23.071079+0000","last_clean":"2024-09-11T03:52:23.071079+0000","last_became_active":"2024-09-11T03:51:17.061257+0000","last_became_peered":"2024-09-11T03:51:17.061257+0000","last_unstale":"2024-09-11T03:52:23.071079+0000","last_undegraded":"2024-09-11T03:52:23.071079+0000","last_fullsized":"2024-09-11T03:52:23.071079+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:18.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.933952+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:39.424252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"846'1373","reported_seq":5651,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172016+0000","last_change":"2024-09-11T03:52:20.152546+0000","last_active":"2024-09-11T03:52:21.172016+0000","last_peered":"2024-09-11T03:52:21.172016+0000","last_clean":"2024-09-11T03:52:21.172016+0000","last_became_active":"2024-09-11T03:51:31.065505+0000","last_became_peered":"2024-09-11T03:51:31.065505+0000","last_unstale":"2024-09-11T03:52:21.172016+0000","last_undegraded":"2024-09-11T03:52:21.172016+0000","last_fullsized":"2024-09-11T03:52:21.172016+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"846'1373","last_scrub_stamp":"2024-09-11T03:51:19.328459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.328459+0000","objects_scrubbed":0,"log_size":1373,"log_dups_size":0,"ondisk_log_size":1373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:32.779371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077195599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.9","version":"20'92","reported_seq":1884,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.171224+0000","last_change":"2024-09-11T03:51:04.911507+0000","last_active":"2024-09-11T03:52:21.171224+0000","last_peered":"2024-09-11T03:52:21.171224+0000","last_clean":"2024-09-11T03:52:21.171224+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:52:21.171224+0000","last_undegraded":"2024-09-11T03:52:21.171224+0000","last_fullsized":"2024-09-11T03:52:21.171224+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:04.911446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:04.911446+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:33.168597+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"922'429","reported_seq":4662,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.072419+0000","last_change":"2024-09-11T03:52:20.175778+0000","last_active":"2024-09-11T03:52:22.072419+0000","last_peered":"2024-09-11T03:52:22.072419+0000","last_clean":"2024-09-11T03:52:22.072419+0000","last_became_active":"2024-09-11T03:51:08.053749+0000","last_became_peered":"2024-09-11T03:51:08.053749+0000","last_unstale":"2024-09-11T03:52:22.072419+0000","last_undegraded":"2024-09-11T03:52:22.072419+0000","last_fullsized":"2024-09-11T03:52:22.072419+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'392","last_scrub_stamp":"2024-09-11T03:51:19.986753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.986753+0000","objects_scrubbed":5,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:40.153828+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.018053044000000001,"stat_sum":{"num_bytes":10459472,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":23665,"num_write":9,"num_write_kb":2863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":1543,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.171936+0000","last_change":"2024-09-11T03:51:55.165620+0000","last_active":"2024-09-11T03:52:21.171936+0000","last_peered":"2024-09-11T03:52:21.171936+0000","last_clean":"2024-09-11T03:52:21.171936+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:52:21.171936+0000","last_undegraded":"2024-09-11T03:52:21.171936+0000","last_fullsized":"2024-09-11T03:52:21.171936+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:55.165539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:55.165539+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:11.582309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"568'646","reported_seq":4865,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.173420+0000","last_change":"2024-09-11T03:52:20.162293+0000","last_active":"2024-09-11T03:52:21.173420+0000","last_peered":"2024-09-11T03:52:21.173420+0000","last_clean":"2024-09-11T03:52:21.173420+0000","last_became_active":"2024-09-11T03:51:08.054955+0000","last_became_peered":"2024-09-11T03:51:08.054955+0000","last_unstale":"2024-09-11T03:52:21.173420+0000","last_undegraded":"2024-09-11T03:52:21.173420+0000","last_fullsized":"2024-09-11T03:52:21.173420+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:51:20.608388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.608388+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:26.828369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079046689999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,0],"acting":[8,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.10","version":"20'92","reported_seq":1485,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.171190+0000","last_change":"2024-09-11T03:49:00.710015+0000","last_active":"2024-09-11T03:52:21.171190+0000","last_peered":"2024-09-11T03:52:21.171190+0000","last_clean":"2024-09-11T03:52:21.171190+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:52:21.171190+0000","last_undegraded":"2024-09-11T03:52:21.171190+0000","last_fullsized":"2024-09-11T03:52:21.171190+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:49:00.709966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:49:00.709966+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:33.296056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":4250,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-09-11T03:52:23.075661+0000","last_change":"2024-09-11T03:52:23.076022+0000","last_active":"2024-09-11T03:52:23.075661+0000","last_peered":"2024-09-11T03:52:23.075661+0000","last_clean":"2024-09-11T03:52:23.075661+0000","last_became_active":"2024-09-11T03:52:20.171504+0000","last_became_peered":"2024-09-11T03:52:20.171504+0000","last_unstale":"2024-09-11T03:52:23.075661+0000","last_undegraded":"2024-09-11T03:52:23.075661+0000","last_fullsized":"2024-09-11T03:52:23.075661+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:51:25.248547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:25.248547+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0019103080000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,7],"acting":[9,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"},{"start":"1ab","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":1485,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-09-11T03:52:20.158432+0000","last_change":"2024-09-11T03:52:10.061483+0000","last_active":"2024-09-11T03:52:20.158432+0000","last_peered":"2024-09-11T03:52:20.158432+0000","last_clean":"2024-09-11T03:52:20.158432+0000","last_became_active":"2024-09-11T03:40:27.846734+0000","last_became_peered":"2024-09-11T03:40:27.846734+0000","last_unstale":"2024-09-11T03:52:20.158432+0000","last_undegraded":"2024-09-11T03:52:20.158432+0000","last_fullsized":"2024-09-11T03:52:20.158432+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:10.061435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:10.061435+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:35.336719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"905'590","reported_seq":4777,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.069044+0000","last_change":"2024-09-11T03:52:22.068996+0000","last_active":"2024-09-11T03:52:22.069044+0000","last_peered":"2024-09-11T03:52:22.069044+0000","last_clean":"2024-09-11T03:52:22.069044+0000","last_became_active":"2024-09-11T03:52:20.174805+0000","last_became_peered":"2024-09-11T03:52:20.174805+0000","last_unstale":"2024-09-11T03:52:22.069044+0000","last_undegraded":"2024-09-11T03:52:22.069044+0000","last_fullsized":"2024-09-11T03:52:22.069044+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":5,"last_scrub":"905'590","last_scrub_stamp":"2024-09-11T03:52:16.650911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:16.650911+0000","objects_scrubbed":0,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001260284,"stat_sum":{"num_bytes":2739805,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":1474,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.171615+0000","last_change":"2024-09-11T03:51:54.138565+0000","last_active":"2024-09-11T03:52:21.171615+0000","last_peered":"2024-09-11T03:52:21.171615+0000","last_clean":"2024-09-11T03:52:21.171615+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:52:21.171615+0000","last_undegraded":"2024-09-11T03:52:21.171615+0000","last_fullsized":"2024-09-11T03:52:21.171615+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:54.138504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.138504+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:20.328413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":3314,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.073525+0000","last_change":"2024-09-11T03:52:21.176497+0000","last_active":"2024-09-11T03:52:22.073525+0000","last_peered":"2024-09-11T03:52:22.073525+0000","last_clean":"2024-09-11T03:52:22.073525+0000","last_became_active":"2024-09-11T03:52:20.161864+0000","last_became_peered":"2024-09-11T03:52:20.161864+0000","last_unstale":"2024-09-11T03:52:22.073525+0000","last_undegraded":"2024-09-11T03:52:22.073525+0000","last_fullsized":"2024-09-11T03:52:22.073525+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:51:47.080811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:47.080811+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.013979190000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,9],"acting":[6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.13","version":"20'92","reported_seq":1579,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172409+0000","last_change":"2024-09-11T03:51:52.573394+0000","last_active":"2024-09-11T03:52:21.172409+0000","last_peered":"2024-09-11T03:52:21.172409+0000","last_clean":"2024-09-11T03:52:21.172409+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:52:21.172409+0000","last_undegraded":"2024-09-11T03:52:21.172409+0000","last_fullsized":"2024-09-11T03:52:21.172409+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:52.573348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.573348+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:55.111010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":3253,"reported_epoch":923,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T03:52:21.354327+0000","last_change":"2024-09-11T03:52:21.354143+0000","last_active":"2024-09-11T03:52:21.354327+0000","last_peered":"2024-09-11T03:52:21.354327+0000","last_clean":"2024-09-11T03:52:21.354327+0000","last_became_active":"2024-09-11T03:52:20.179454+0000","last_became_peered":"2024-09-11T03:52:20.179454+0000","last_unstale":"2024-09-11T03:52:21.354327+0000","last_undegraded":"2024-09-11T03:52:21.354327+0000","last_fullsized":"2024-09-11T03:52:21.354327+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:51:24.589460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:24.589460+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0035647000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":1060,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172365+0000","last_change":"2024-09-11T03:51:28.960262+0000","last_active":"2024-09-11T03:52:21.172365+0000","last_peered":"2024-09-11T03:52:21.172365+0000","last_clean":"2024-09-11T03:52:21.172365+0000","last_became_active":"2024-09-11T03:46:36.026459+0000","last_became_peered":"2024-09-11T03:46:36.026459+0000","last_unstale":"2024-09-11T03:52:21.172365+0000","last_undegraded":"2024-09-11T03:52:21.172365+0000","last_fullsized":"2024-09-11T03:52:21.172365+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:28.960214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.960214+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:51.817417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":3798,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.072787+0000","last_change":"2024-09-11T03:52:20.176105+0000","last_active":"2024-09-11T03:52:22.072787+0000","last_peered":"2024-09-11T03:52:22.072787+0000","last_clean":"2024-09-11T03:52:22.072787+0000","last_became_active":"2024-09-11T03:50:02.052278+0000","last_became_peered":"2024-09-11T03:50:02.052278+0000","last_unstale":"2024-09-11T03:52:22.072787+0000","last_undegraded":"2024-09-11T03:52:22.072787+0000","last_fullsized":"2024-09-11T03:52:22.072787+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:51:28.919603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.919603+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:41.773047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.020830268999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":1446,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172746+0000","last_change":"2024-09-11T03:50:19.741484+0000","last_active":"2024-09-11T03:52:21.172746+0000","last_peered":"2024-09-11T03:52:21.172746+0000","last_clean":"2024-09-11T03:52:21.172746+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:52:21.172746+0000","last_undegraded":"2024-09-11T03:52:21.172746+0000","last_fullsized":"2024-09-11T03:52:21.172746+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:19.741428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:19.741428+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:37.978745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"782'561","reported_seq":4768,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.171722+0000","last_change":"2024-09-11T03:52:20.159016+0000","last_active":"2024-09-11T03:52:21.171722+0000","last_peered":"2024-09-11T03:52:21.171722+0000","last_clean":"2024-09-11T03:52:21.171722+0000","last_became_active":"2024-09-11T03:50:02.054667+0000","last_became_peered":"2024-09-11T03:50:02.054667+0000","last_unstale":"2024-09-11T03:52:21.171722+0000","last_undegraded":"2024-09-11T03:52:21.171722+0000","last_fullsized":"2024-09-11T03:52:21.171722+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":784,"parent":"0.0","parent_split_bits":5,"last_scrub":"782'561","last_scrub_stamp":"2024-09-11T03:51:24.484944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:24.484944+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:50.945352+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0016624159999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":935,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.074141+0000","last_change":"2024-09-11T03:52:15.093344+0000","last_active":"2024-09-11T03:52:22.074141+0000","last_peered":"2024-09-11T03:52:22.074141+0000","last_clean":"2024-09-11T03:52:22.074141+0000","last_became_active":"2024-09-11T03:45:01.941720+0000","last_became_peered":"2024-09-11T03:45:01.941720+0000","last_unstale":"2024-09-11T03:52:22.074141+0000","last_undegraded":"2024-09-11T03:52:22.074141+0000","last_fullsized":"2024-09-11T03:52:22.074141+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:15.093294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:15.093294+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:34.332612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":4465,"reported_epoch":925,"state":"active+clean+scrubbing","last_fresh":"2024-09-11T03:52:23.350271+0000","last_change":"2024-09-11T03:52:23.350128+0000","last_active":"2024-09-11T03:52:23.350271+0000","last_peered":"2024-09-11T03:52:23.350271+0000","last_clean":"2024-09-11T03:52:23.350271+0000","last_became_active":"2024-09-11T03:52:20.171796+0000","last_became_peered":"2024-09-11T03:52:20.171796+0000","last_unstale":"2024-09-11T03:52:23.350271+0000","last_undegraded":"2024-09-11T03:52:23.350271+0000","last_fullsized":"2024-09-11T03:52:23.350271+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:51:20.293380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.293380+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012311539999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"},{"start":"1ab","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":1475,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172514+0000","last_change":"2024-09-11T03:52:09.331476+0000","last_active":"2024-09-11T03:52:21.172514+0000","last_peered":"2024-09-11T03:52:21.172514+0000","last_clean":"2024-09-11T03:52:21.172514+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:52:21.172514+0000","last_undegraded":"2024-09-11T03:52:21.172514+0000","last_fullsized":"2024-09-11T03:52:21.172514+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:09.331402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:09.331402+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:12.316710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"778'353","reported_seq":4440,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.073447+0000","last_change":"2024-09-11T03:52:22.073400+0000","last_active":"2024-09-11T03:52:22.073447+0000","last_peered":"2024-09-11T03:52:22.073447+0000","last_clean":"2024-09-11T03:52:22.073447+0000","last_became_active":"2024-09-11T03:52:20.166884+0000","last_became_peered":"2024-09-11T03:52:20.166884+0000","last_unstale":"2024-09-11T03:52:22.073447+0000","last_undegraded":"2024-09-11T03:52:22.073447+0000","last_fullsized":"2024-09-11T03:52:22.073447+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":5,"last_scrub":"778'353","last_scrub_stamp":"2024-09-11T03:51:23.900161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.900161+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.011126627,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.18","version":"20'92","reported_seq":1585,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.171154+0000","last_change":"2024-09-11T03:51:27.437144+0000","last_active":"2024-09-11T03:52:21.171154+0000","last_peered":"2024-09-11T03:52:21.171154+0000","last_clean":"2024-09-11T03:52:21.171154+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:52:21.171154+0000","last_undegraded":"2024-09-11T03:52:21.171154+0000","last_fullsized":"2024-09-11T03:52:21.171154+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:27.437090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:27.437090+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:35.495938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"922'944","reported_seq":5285,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.073333+0000","last_change":"2024-09-11T03:52:20.167055+0000","last_active":"2024-09-11T03:52:22.073333+0000","last_peered":"2024-09-11T03:52:22.073333+0000","last_clean":"2024-09-11T03:52:22.073333+0000","last_became_active":"2024-09-11T03:50:31.047148+0000","last_became_peered":"2024-09-11T03:50:31.047148+0000","last_unstale":"2024-09-11T03:52:22.073333+0000","last_undegraded":"2024-09-11T03:52:22.073333+0000","last_fullsized":"2024-09-11T03:52:22.073333+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":5,"last_scrub":"882'911","last_scrub_stamp":"2024-09-11T03:51:45.052831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:45.052831+0000","objects_scrubbed":6,"log_size":944,"log_dups_size":0,"ondisk_log_size":944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:46.711848+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0098869319999999993,"stat_sum":{"num_bytes":16953417,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":27,"num_read_kb":32508,"num_write":29,"num_write_kb":8395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.19","version":"20'92","reported_seq":1073,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.068429+0000","last_change":"2024-09-11T03:51:28.080491+0000","last_active":"2024-09-11T03:52:22.068429+0000","last_peered":"2024-09-11T03:52:22.068429+0000","last_clean":"2024-09-11T03:52:22.068429+0000","last_became_active":"2024-09-11T03:46:15.972841+0000","last_became_peered":"2024-09-11T03:46:15.972841+0000","last_unstale":"2024-09-11T03:52:22.068429+0000","last_undegraded":"2024-09-11T03:52:22.068429+0000","last_fullsized":"2024-09-11T03:52:22.068429+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:28.080420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.080420+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:28.344483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"557'273","reported_seq":4388,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.072497+0000","last_change":"2024-09-11T03:52:20.161046+0000","last_active":"2024-09-11T03:52:22.072497+0000","last_peered":"2024-09-11T03:52:22.072497+0000","last_clean":"2024-09-11T03:52:22.072497+0000","last_became_active":"2024-09-11T03:50:28.038587+0000","last_became_peered":"2024-09-11T03:50:28.038587+0000","last_unstale":"2024-09-11T03:52:22.072497+0000","last_undegraded":"2024-09-11T03:52:22.072497+0000","last_fullsized":"2024-09-11T03:52:22.072497+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:51:54.121178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.121178+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:03.222000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0064849870000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,2],"acting":[6,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.1a","version":"20'92","reported_seq":624,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.173026+0000","last_change":"2024-09-11T03:51:23.032108+0000","last_active":"2024-09-11T03:52:21.173026+0000","last_peered":"2024-09-11T03:52:21.173026+0000","last_clean":"2024-09-11T03:52:21.173026+0000","last_became_active":"2024-09-11T03:51:17.063957+0000","last_became_peered":"2024-09-11T03:51:17.063957+0000","last_unstale":"2024-09-11T03:52:21.173026+0000","last_undegraded":"2024-09-11T03:52:21.173026+0000","last_fullsized":"2024-09-11T03:52:21.173026+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:23.032056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.032056+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:35.587988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.19","version":"922'651","reported_seq":4902,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.073537+0000","last_change":"2024-09-11T03:52:20.159930+0000","last_active":"2024-09-11T03:52:22.073537+0000","last_peered":"2024-09-11T03:52:22.073537+0000","last_clean":"2024-09-11T03:52:22.073537+0000","last_became_active":"2024-09-11T03:50:28.039694+0000","last_became_peered":"2024-09-11T03:50:28.039694+0000","last_unstale":"2024-09-11T03:52:22.073537+0000","last_undegraded":"2024-09-11T03:52:22.073537+0000","last_fullsized":"2024-09-11T03:52:22.073537+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"916'643","last_scrub_stamp":"2024-09-11T03:52:16.094749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:16.094749+0000","objects_scrubbed":7,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:21.533697+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0056863599999999997,"stat_sum":{"num_bytes":18183514,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":16,"num_read_kb":9986,"num_write":20,"num_write_kb":9085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.1b","version":"20'92","reported_seq":1444,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.173235+0000","last_change":"2024-09-11T03:51:52.434832+0000","last_active":"2024-09-11T03:52:21.173235+0000","last_peered":"2024-09-11T03:52:21.173235+0000","last_clean":"2024-09-11T03:52:21.173235+0000","last_became_active":"2024-09-11T03:40:39.860042+0000","last_became_peered":"2024-09-11T03:40:39.860042+0000","last_unstale":"2024-09-11T03:52:21.173235+0000","last_undegraded":"2024-09-11T03:52:21.173235+0000","last_fullsized":"2024-09-11T03:52:21.173235+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:52.434784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.434784+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:11.714276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"925'762","reported_seq":5105,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-09-11T03:52:23.073392+0000","last_change":"2024-09-11T03:52:23.073392+0000","last_active":"2024-09-11T03:52:23.073392+0000","last_peered":"2024-09-11T03:52:23.073392+0000","last_clean":"2024-09-11T03:52:23.073392+0000","last_became_active":"2024-09-11T03:51:20.065666+0000","last_became_peered":"2024-09-11T03:51:20.065666+0000","last_unstale":"2024-09-11T03:52:23.073392+0000","last_undegraded":"2024-09-11T03:52:23.073392+0000","last_fullsized":"2024-09-11T03:52:23.073392+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":5,"last_scrub":"910'736","last_scrub_stamp":"2024-09-11T03:52:10.509090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:10.509090+0000","objects_scrubbed":18,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:29.429423+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0027260280000000001,"stat_sum":{"num_bytes":34417331,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":2,"num_read":37,"num_read_kb":35366,"num_write":30,"num_write_kb":11526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":15760321,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,11],"acting":[9,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"},{"start":"1ab","length":"1"}]},{"pgid":"2.1f","version":"922'925","reported_seq":5047,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-09-11T03:52:20.173958+0000","last_change":"2024-09-11T03:52:20.174334+0000","last_active":"2024-09-11T03:52:20.173958+0000","last_peered":"2024-09-11T03:52:20.173958+0000","last_clean":"2024-09-11T03:52:20.173958+0000","last_became_active":"2024-09-11T03:50:28.041110+0000","last_became_peered":"2024-09-11T03:50:28.041110+0000","last_unstale":"2024-09-11T03:52:20.173958+0000","last_undegraded":"2024-09-11T03:52:20.173958+0000","last_fullsized":"2024-09-11T03:52:20.173958+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"894'871","last_scrub_stamp":"2024-09-11T03:51:53.078478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.078478+0000","objects_scrubbed":13,"log_size":925,"log_dups_size":0,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:08.524954+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.015205896999999999,"stat_sum":{"num_bytes":27322934,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":52,"num_read_kb":49670,"num_write":27,"num_write_kb":13669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.1c","version":"20'92","reported_seq":1459,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.172256+0000","last_change":"2024-09-11T03:52:14.409111+0000","last_active":"2024-09-11T03:52:21.172256+0000","last_peered":"2024-09-11T03:52:21.172256+0000","last_clean":"2024-09-11T03:52:21.172256+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:52:21.172256+0000","last_undegraded":"2024-09-11T03:52:21.172256+0000","last_fullsized":"2024-09-11T03:52:21.172256+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:14.409042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:14.409042+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:20.863513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1e","version":"922'1128","reported_seq":5264,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-09-11T03:52:20.177773+0000","last_change":"2024-09-11T03:52:20.178052+0000","last_active":"2024-09-11T03:52:20.177773+0000","last_peered":"2024-09-11T03:52:20.177773+0000","last_clean":"2024-09-11T03:52:20.177773+0000","last_became_active":"2024-09-11T03:50:28.044451+0000","last_became_peered":"2024-09-11T03:50:28.044451+0000","last_unstale":"2024-09-11T03:52:20.177773+0000","last_undegraded":"2024-09-11T03:52:20.177773+0000","last_fullsized":"2024-09-11T03:52:20.177773+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"885'1106","last_scrub_stamp":"2024-09-11T03:51:44.050833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:44.050833+0000","objects_scrubbed":5,"log_size":1128,"log_dups_size":0,"ondisk_log_size":1128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:06.242330+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.018224679000000001,"stat_sum":{"num_bytes":9671047,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":27,"num_read_kb":24362,"num_write":13,"num_write_kb":3430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,10],"acting":[0,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.1d","version":"20'92","reported_seq":1440,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-11T03:52:21.173406+0000","last_change":"2024-09-11T03:52:08.623749+0000","last_active":"2024-09-11T03:52:21.173406+0000","last_peered":"2024-09-11T03:52:21.173406+0000","last_clean":"2024-09-11T03:52:21.173406+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:52:21.173406+0000","last_undegraded":"2024-09-11T03:52:21.173406+0000","last_fullsized":"2024-09-11T03:52:21.173406+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:08.623692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:08.623692+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:16.913356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"567'752","reported_seq":4879,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.068617+0000","last_change":"2024-09-11T03:52:20.165699+0000","last_active":"2024-09-11T03:52:22.068617+0000","last_peered":"2024-09-11T03:52:22.068617+0000","last_clean":"2024-09-11T03:52:22.068617+0000","last_became_active":"2024-09-11T03:50:28.045687+0000","last_became_peered":"2024-09-11T03:50:28.045687+0000","last_unstale":"2024-09-11T03:52:22.068617+0000","last_undegraded":"2024-09-11T03:52:22.068617+0000","last_fullsized":"2024-09-11T03:52:22.068617+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":5,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:51:52.141035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.141035+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:20.589554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00094455500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,11],"acting":[10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"127","length":"1"},{"start":"17e","length":"1"},{"start":"186","length":"1"},{"start":"189","length":"2"},{"start":"192","length":"1"},{"start":"195","length":"1"}]},{"pgid":"1.1e","version":"20'92","reported_seq":1467,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-09-11T03:52:22.074248+0000","last_change":"2024-09-11T03:52:17.013751+0000","last_active":"2024-09-11T03:52:22.074248+0000","last_peered":"2024-09-11T03:52:22.074248+0000","last_clean":"2024-09-11T03:52:22.074248+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:52:22.074248+0000","last_undegraded":"2024-09-11T03:52:22.074248+0000","last_fullsized":"2024-09-11T03:52:22.074248+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:17.013702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:17.013702+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:36.168162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":86,"stat_sum":{"num_bytes":747245649,"num_objects":317,"num_object_clones":265,"num_object_copies":951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":317,"num_whiteouts":12,"num_read":1269,"num_read_kb":1371776,"num_write":849,"num_write_kb":340949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":82257142,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1335017472,"data_stored":1773079389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51600,"ondisk_log_size":51600,"up":252,"acting":248,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285750,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218108,"kb_used_data":160544,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93527044,"statfs":{"total":95995035648,"available":95771693056,"internally_reserved":0,"allocated":164397056,"data_stored":214629096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":58915268},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514682,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91804,"kb_used_data":60616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93653348,"statfs":{"total":95995035648,"available":95901028352,"internally_reserved":0,"allocated":62070784,"data_stored":73322274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743616,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":180108,"kb_used_data":112824,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93565044,"statfs":{"total":95995035648,"available":95810605056,"internally_reserved":0,"allocated":115531776,"data_stored":139745580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037959,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":189064,"kb_used_data":139820,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93556088,"statfs":{"total":95995035648,"available":95801434112,"internally_reserved":0,"allocated":143175680,"data_stored":192381096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299594,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":215236,"kb_used_data":143788,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93529916,"statfs":{"total":95995035648,"available":95774633984,"internally_reserved":0,"allocated":147238912,"data_stored":178401920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528527,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":224956,"kb_used_data":171560,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93520196,"statfs":{"total":95995035648,"available":95764680704,"internally_reserved":0,"allocated":175677440,"data_stored":234932276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757461,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":175532,"kb_used_data":117976,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93569620,"statfs":{"total":95995035648,"available":95815290880,"internally_reserved":0,"allocated":120807424,"data_stored":159760576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986394,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149920,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":208524,"kb_used_data":132912,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93536628,"statfs":{"total":95995035648,"available":95781507072,"internally_reserved":0,"allocated":136101888,"data_stored":184964931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411557,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":144936,"kb_used_data":87380,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93600216,"statfs":{"total":95995035648,"available":95846621184,"internally_reserved":0,"allocated":89477120,"data_stored":104713686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575082,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":130948,"kb_used_data":85864,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93614204,"statfs":{"total":95995035648,"available":95860944896,"internally_reserved":0,"allocated":87924736,"data_stored":105859273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738607,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":161024,"kb_used_data":115944,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93584128,"statfs":{"total":95995035648,"available":95830147072,"internally_reserved":0,"allocated":118726656,"data_stored":139773227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":107200512,"data_stored":132097618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":76713984,"data_stored":98502854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":78049280,"data_stored":97147025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":124551168,"data_stored":177271838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":96878592,"data_stored":122811969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":109391872,"data_stored":152202909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":166100992,"data_stored":230072034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":135835648,"data_stored":172347565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":131964928,"data_stored":185024677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":104296448,"data_stored":132371362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":50860032,"data_stored":65965855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":153174016,"data_stored":207263683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:52:26.002 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-11T03:52:26.002 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 96 2024-09-11T03:52:26.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:26.003 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:26.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:26.073 INFO:tasks.rados.rados.2.smithi175.stdout:3869: done (4 left) 2024-09-11T03:52:26.074 INFO:tasks.rados.rados.2.smithi175.stdout:3870: write oid 43 current snap is 429 2024-09-11T03:52:26.074 INFO:tasks.rados.rados.2.smithi175.stdout:3864: finishing write tid 3 to smithi17557870-23 2024-09-11T03:52:26.074 INFO:tasks.rados.rados.2.smithi175.stdout:3864: finishing write tid 4 to smithi17557870-23 2024-09-11T03:52:26.074 INFO:tasks.rados.rados.2.smithi175.stdout:3864: oid 23 updating version 0 to 1387 2024-09-11T03:52:26.074 INFO:tasks.rados.rados.2.smithi175.stdout:3864: oid 23 updating version 1387 to 1388 2024-09-11T03:52:26.074 INFO:tasks.rados.rados.2.smithi175.stdout:3864: oid 23 updating version 1388 to 1389 2024-09-11T03:52:26.074 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 1389 (ObjNum 1312 snap 429 seq_num 1312) dirty exists 2024-09-11T03:52:26.074 INFO:tasks.rados.rados.2.smithi175.stdout:3864: left oid 23 (ObjNum 1312 snap 429 seq_num 1312) 2024-09-11T03:52:26.074 INFO:tasks.rados.rados.2.smithi175.stdout:3865: expect (ObjNum 1300 snap 426 seq_num 1300) 2024-09-11T03:52:26.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:26.127 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:52:26.127 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:52:26.128 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:26.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:26.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:26.155 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:52:26.155 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:52:26.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:52:26.171 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:52:26.188 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:52:26.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:52:26.205 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:52:26.206 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:52:26.223 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:52:26.224 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:52:26.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:26.240 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:52:26.240 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:52:26.249 INFO:tasks.rados.rados.2.smithi175.stdout:3867: expect (ObjNum 1302 snap 426 seq_num 1302) 2024-09-11T03:52:26.254 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:26.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:26.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:26.257 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:52:26.258 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:52:26.273 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:52:26.274 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:52:26.290 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:52:26.290 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:52:26.306 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:52:26.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:52:26.322 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:52:26.323 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:52:26.355 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:26.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:26.377 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:26.378 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:26.432 INFO:tasks.rados.rados.2.smithi175.stdout:3870: seq_num 1314 ranges {484398=489902,1392398=411118} 2024-09-11T03:52:26.437 INFO:tasks.rados.rados.2.smithi175.stdout:3870: writing smithi17557870-43 from 484398 to 974300 tid 1 2024-09-11T03:52:26.442 INFO:tasks.rados.rados.2.smithi175.stdout:3870: writing smithi17557870-43 from 1392398 to 1803516 tid 2 2024-09-11T03:52:26.442 INFO:tasks.rados.rados.2.smithi175.stdout:3864: done (4 left) 2024-09-11T03:52:26.443 INFO:tasks.rados.rados.2.smithi175.stdout:3865: done (3 left) 2024-09-11T03:52:26.443 INFO:tasks.rados.rados.2.smithi175.stdout:3867: done (2 left) 2024-09-11T03:52:26.443 INFO:tasks.rados.rados.2.smithi175.stdout:3868: done (1 left) 2024-09-11T03:52:26.443 INFO:tasks.rados.rados.2.smithi175.stdout:3871: copy_from oid 40 from oid 27 current snap is 429 2024-09-11T03:52:26.443 INFO:tasks.rados.rados.2.smithi175.stdout:3870: finishing write tid 1 to smithi17557870-43 2024-09-11T03:52:26.443 INFO:tasks.rados.rados.2.smithi175.stdout:3872: rollback oid 37 current snap is 429 2024-09-11T03:52:26.443 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 37 to 368 2024-09-11T03:52:26.443 INFO:tasks.rados.rados.2.smithi175.stdout:3873: snap_remove snap 409 2024-09-11T03:52:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:26 smithi175 ceph-mon[30894]: 2.12 scrub starts 2024-09-11T03:52:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:26 smithi175 ceph-mon[30894]: pgmap v1545: 117 pgs: 4 active+undersized+remapped, 1 peering, 4 active+clean+scrubbing, 5 unknown, 103 active+clean; 713 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 28 MiB/s rd, 7.6 MiB/s wr, 44 op/s 2024-09-11T03:52:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:26 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:52:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:26 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:52:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:26 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:52:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:26 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:52:26.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:26 smithi175 ceph-mon[30894]: 2.13 scrub starts 2024-09-11T03:52:26.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:26 smithi175 ceph-mon[30894]: 2.50 scrub starts 2024-09-11T03:52:26.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:26 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:52:26.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:26 smithi175 ceph-mon[30894]: osdmap e928: 12 total, 12 up, 11 in 2024-09-11T03:52:26.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1599353854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:26.478 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:26.499 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:26.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:26.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:26.581 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:26.601 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:26.626 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:26.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:26.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:26 smithi084 ceph-mon[27254]: 2.12 scrub starts 2024-09-11T03:52:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:26 smithi084 ceph-mon[27254]: pgmap v1545: 117 pgs: 4 active+undersized+remapped, 1 peering, 4 active+clean+scrubbing, 5 unknown, 103 active+clean; 713 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 28 MiB/s rd, 7.6 MiB/s wr, 44 op/s 2024-09-11T03:52:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:26 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:52:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:26 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:52:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:26 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:52:26.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:26 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:52:26.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:26 smithi084 ceph-mon[27254]: 2.13 scrub starts 2024-09-11T03:52:26.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:26 smithi084 ceph-mon[27254]: 2.50 scrub starts 2024-09-11T03:52:26.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:26 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:52:26.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:26 smithi084 ceph-mon[27254]: osdmap e928: 12 total, 12 up, 11 in 2024-09-11T03:52:26.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1599353854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:26.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:26 smithi121 ceph-mon[30729]: 2.12 scrub starts 2024-09-11T03:52:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:26 smithi121 ceph-mon[30729]: pgmap v1545: 117 pgs: 4 active+undersized+remapped, 1 peering, 4 active+clean+scrubbing, 5 unknown, 103 active+clean; 713 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 28 MiB/s rd, 7.6 MiB/s wr, 44 op/s 2024-09-11T03:52:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:26 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:52:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:26 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:52:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:26 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:52:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:26 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:52:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:26 smithi121 ceph-mon[30729]: 2.13 scrub starts 2024-09-11T03:52:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:26 smithi121 ceph-mon[30729]: 2.50 scrub starts 2024-09-11T03:52:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:26 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:52:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:26 smithi121 ceph-mon[30729]: osdmap e928: 12 total, 12 up, 11 in 2024-09-11T03:52:26.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1599353854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:26.728 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:26.749 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:26.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:26.750 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:26.852 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:26.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:26.875 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:26.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:26.976 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:26.999 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:27.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:27.000 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:27.097 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-11T03:52:27.101 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:27.123 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:27.124 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:27.187 INFO:tasks.rados.rados.2.smithi175.stdout:3873: done (3 left) 2024-09-11T03:52:27.188 INFO:tasks.rados.rados.2.smithi175.stdout:3874: read oid 16 snap -1 2024-09-11T03:52:27.188 INFO:tasks.rados.rados.2.smithi175.stdout:3874: expect (ObjNum 1275 snap 422 seq_num 1275) 2024-09-11T03:52:27.188 INFO:tasks.rados.rados.2.smithi175.stdout:3875: snap_remove snap 423 2024-09-11T03:52:27.225 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:27.247 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:27.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:27.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:27.349 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:27.370 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:27.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:27.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:27 smithi084 ceph-mon[27254]: 2.12 scrub starts 2024-09-11T03:52:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:27 smithi084 ceph-mon[27254]: 2.54 scrub starts 2024-09-11T03:52:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:27 smithi084 ceph-mon[27254]: 2.52 scrub starts 2024-09-11T03:52:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:27 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:52:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:27 smithi084 ceph-mon[27254]: 2.4e scrub starts 2024-09-11T03:52:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:27 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:52:27.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:27 smithi084 ceph-mon[27254]: 2.50 scrub starts 2024-09-11T03:52:27.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:27 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:52:27.472 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:27 smithi175 ceph-mon[30894]: 2.12 scrub starts 2024-09-11T03:52:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:27 smithi175 ceph-mon[30894]: 2.54 scrub starts 2024-09-11T03:52:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:27 smithi175 ceph-mon[30894]: 2.52 scrub starts 2024-09-11T03:52:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:27 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:52:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:27 smithi175 ceph-mon[30894]: 2.4e scrub starts 2024-09-11T03:52:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:27 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:52:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:27 smithi175 ceph-mon[30894]: 2.50 scrub starts 2024-09-11T03:52:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:27 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:52:27.495 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:27.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:27.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:27.597 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:27.623 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:27.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:27.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:27 smithi121 ceph-mon[30729]: 2.12 scrub starts 2024-09-11T03:52:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:27 smithi121 ceph-mon[30729]: 2.54 scrub starts 2024-09-11T03:52:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:27 smithi121 ceph-mon[30729]: 2.52 scrub starts 2024-09-11T03:52:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:27 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:52:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:27 smithi121 ceph-mon[30729]: 2.4e scrub starts 2024-09-11T03:52:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:27 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:52:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:27 smithi121 ceph-mon[30729]: 2.50 scrub starts 2024-09-11T03:52:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:27 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:52:27.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:27.750 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:27.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:27.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:27.852 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:27.878 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:27.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:27.879 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:27.979 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:28.003 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:28.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:28.012 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:28.103 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:28.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:28.126 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:28.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:28.192 INFO:tasks.rados.rados.2.smithi175.stdout:3875: done (4 left) 2024-09-11T03:52:28.192 INFO:tasks.rados.rados.2.smithi175.stdout:3876: write (excl) oid 33 current snap is 429 2024-09-11T03:52:28.192 INFO:tasks.rados.rados.2.smithi175.stdout:3876: seq_num 1315 ranges {501205=565151,1850880=250325} 2024-09-11T03:52:28.198 INFO:tasks.rados.rados.2.smithi175.stdout:3876: writing smithi17557870-33 from 501205 to 1066356 tid 1 2024-09-11T03:52:28.204 INFO:tasks.rados.rados.2.smithi175.stdout:3876: writing smithi17557870-33 from 1850880 to 2101205 tid 2 2024-09-11T03:52:28.204 INFO:tasks.rados.rados.2.smithi175.stdout:3877: snap_create 2024-09-11T03:52:28.205 INFO:tasks.rados.rados.2.smithi175.stdout:3872: finishing rollback tid 1 to smithi17557870-37 2024-09-11T03:52:28.205 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 1262 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T03:52:28.205 INFO:tasks.rados.rados.2.smithi175.stdout:3870: finishing write tid 2 to smithi17557870-43 2024-09-11T03:52:28.205 INFO:tasks.rados.rados.2.smithi175.stdout:3870: finishing write tid 3 to smithi17557870-43 2024-09-11T03:52:28.205 INFO:tasks.rados.rados.2.smithi175.stdout:3870: finishing write tid 4 to smithi17557870-43 2024-09-11T03:52:28.205 INFO:tasks.rados.rados.2.smithi175.stdout:3870: oid 43 updating version 0 to 1023 2024-09-11T03:52:28.205 INFO:tasks.rados.rados.2.smithi175.stdout:3870: oid 43 updating version 1023 to 1025 2024-09-11T03:52:28.205 INFO:tasks.rados.rados.2.smithi175.stdout:3870: oid 43 version 1025 is already newer than 1024 2024-09-11T03:52:28.205 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 43 v 1025 (ObjNum 1314 snap 429 seq_num 1314) dirty exists 2024-09-11T03:52:28.205 INFO:tasks.rados.rados.2.smithi175.stdout:3870: left oid 43 (ObjNum 1314 snap 429 seq_num 1314) 2024-09-11T03:52:28.206 INFO:tasks.rados.rados.2.smithi175.stdout:3871: finishing copy_from racing read to smithi17557870-40 2024-09-11T03:52:28.206 INFO:tasks.rados.rados.2.smithi175.stdout:3871: finishing copy_from to smithi17557870-40 2024-09-11T03:52:28.206 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 1252 (ObjNum 1258 snap 413 seq_num 1258) dirty exists 2024-09-11T03:52:28.206 INFO:tasks.rados.rados.2.smithi175.stdout:3874: expect (ObjNum 1275 snap 422 seq_num 1275) 2024-09-11T03:52:28.226 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:28.249 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:28.250 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:28.351 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:28.355 INFO:tasks.rados.rados.2.smithi175.stdout:3876: finishing write tid 1 to smithi17557870-33 2024-09-11T03:52:28.355 INFO:tasks.rados.rados.2.smithi175.stdout:3876: finishing write tid 2 to smithi17557870-33 2024-09-11T03:52:28.355 INFO:tasks.rados.rados.2.smithi175.stdout:3876: finishing write tid 3 to smithi17557870-33 2024-09-11T03:52:28.355 INFO:tasks.rados.rados.2.smithi175.stdout:3876: finishing write tid 4 to smithi17557870-33 2024-09-11T03:52:28.355 INFO:tasks.rados.rados.2.smithi175.stdout:3876: oid 33 updating version 0 to 1613 2024-09-11T03:52:28.356 INFO:tasks.rados.rados.2.smithi175.stdout:3876: oid 33 version 1613 is already newer than 1612 2024-09-11T03:52:28.356 INFO:tasks.rados.rados.2.smithi175.stdout:3876: oid 33 updating version 1613 to 1614 2024-09-11T03:52:28.356 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 1614 (ObjNum 1315 snap 429 seq_num 1315) dirty exists 2024-09-11T03:52:28.356 INFO:tasks.rados.rados.2.smithi175.stdout:3876: left oid 33 (ObjNum 1315 snap 429 seq_num 1315) 2024-09-11T03:52:28.376 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:28.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:28.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:28.437 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:28 smithi084 ceph-mon[27254]: 2.51 scrub starts 2024-09-11T03:52:28.437 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:28 smithi084 ceph-mon[27254]: 2.53 scrub starts 2024-09-11T03:52:28.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:28 smithi084 ceph-mon[27254]: pgmap v1548: 117 pgs: 1 active+undersized+remapped, 5 active+clean+scrubbing, 2 unknown, 109 active+clean; 708 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 7.9 MiB/s rd, 5.7 MiB/s wr, 28 op/s; 1.1 MiB/s, 4 objects/s recovering 2024-09-11T03:52:28.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:28 smithi084 ceph-mon[27254]: 2.52 scrub starts 2024-09-11T03:52:28.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:28 smithi084 ceph-mon[27254]: 2.11 scrub starts 2024-09-11T03:52:28.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:28 smithi084 ceph-mon[27254]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-11T03:52:28.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:28 smithi084 ceph-mon[27254]: osdmap e929: 12 total, 12 up, 11 in 2024-09-11T03:52:28.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:28 smithi084 ceph-mon[27254]: 2.55 scrub starts 2024-09-11T03:52:28.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:28 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:52:28.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:28 smithi084 ceph-mon[27254]: 2.50 scrub starts 2024-09-11T03:52:28.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:28 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:52:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:28 smithi175 ceph-mon[30894]: 2.51 scrub starts 2024-09-11T03:52:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:28 smithi175 ceph-mon[30894]: 2.53 scrub starts 2024-09-11T03:52:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:28 smithi175 ceph-mon[30894]: pgmap v1548: 117 pgs: 1 active+undersized+remapped, 5 active+clean+scrubbing, 2 unknown, 109 active+clean; 708 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 7.9 MiB/s rd, 5.7 MiB/s wr, 28 op/s; 1.1 MiB/s, 4 objects/s recovering 2024-09-11T03:52:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:28 smithi175 ceph-mon[30894]: 2.52 scrub starts 2024-09-11T03:52:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:28 smithi175 ceph-mon[30894]: 2.11 scrub starts 2024-09-11T03:52:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:28 smithi175 ceph-mon[30894]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-11T03:52:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:28 smithi175 ceph-mon[30894]: osdmap e929: 12 total, 12 up, 11 in 2024-09-11T03:52:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:28 smithi175 ceph-mon[30894]: 2.55 scrub starts 2024-09-11T03:52:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:28 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:52:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:28 smithi175 ceph-mon[30894]: 2.50 scrub starts 2024-09-11T03:52:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:28 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:52:28.478 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:28.499 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:28.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:28.500 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:28.601 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:28.621 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:28.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:28.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:28.622 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:28 smithi121 ceph-mon[30729]: 2.51 scrub starts 2024-09-11T03:52:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:28 smithi121 ceph-mon[30729]: 2.53 scrub starts 2024-09-11T03:52:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:28 smithi121 ceph-mon[30729]: pgmap v1548: 117 pgs: 1 active+undersized+remapped, 5 active+clean+scrubbing, 2 unknown, 109 active+clean; 708 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 7.9 MiB/s rd, 5.7 MiB/s wr, 28 op/s; 1.1 MiB/s, 4 objects/s recovering 2024-09-11T03:52:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:28 smithi121 ceph-mon[30729]: 2.52 scrub starts 2024-09-11T03:52:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:28 smithi121 ceph-mon[30729]: 2.11 scrub starts 2024-09-11T03:52:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:28 smithi121 ceph-mon[30729]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-11T03:52:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:28 smithi121 ceph-mon[30729]: osdmap e929: 12 total, 12 up, 11 in 2024-09-11T03:52:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:28 smithi121 ceph-mon[30729]: 2.55 scrub starts 2024-09-11T03:52:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:28 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:52:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:28 smithi121 ceph-mon[30729]: 2.50 scrub starts 2024-09-11T03:52:28.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:28 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:52:28.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:28.749 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:28.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:28.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:28.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:28.871 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:28.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:28.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:28.973 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:28.995 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:28.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:29.079 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:52:29.079 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:52:29.097 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:29.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:29.119 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:29.120 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:29.193 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:52:29.193 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 2 pg_num to 96 2024-09-11T03:52:29.198 INFO:tasks.rados.rados.2.smithi175.stdout:3870: done (5 left) 2024-09-11T03:52:29.198 INFO:tasks.rados.rados.2.smithi175.stdout:3871: done (4 left) 2024-09-11T03:52:29.198 INFO:tasks.rados.rados.2.smithi175.stdout:3872: done (3 left) 2024-09-11T03:52:29.198 INFO:tasks.rados.rados.2.smithi175.stdout:3874: done (2 left) 2024-09-11T03:52:29.198 INFO:tasks.rados.rados.2.smithi175.stdout:3876: done (1 left) 2024-09-11T03:52:29.198 INFO:tasks.rados.rados.2.smithi175.stdout:3877: done (0 left) 2024-09-11T03:52:29.198 INFO:tasks.rados.rados.2.smithi175.stdout:3878: snap_remove snap 361 2024-09-11T03:52:29.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:29.246 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:29.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:29.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:29.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:29 smithi084 ceph-mon[27254]: 2.51 scrub starts 2024-09-11T03:52:29.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:29 smithi084 ceph-mon[27254]: 2.53 scrub starts 2024-09-11T03:52:29.315 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:29 smithi084 ceph-mon[27254]: 2.52 scrub starts 2024-09-11T03:52:29.316 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:29 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:52:29.316 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:29 smithi084 ceph-mon[27254]: osdmap e930: 12 total, 12 up, 11 in 2024-09-11T03:52:29.316 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:29 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3313411155' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:52:29.316 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:29 smithi084 ceph-mon[27254]: 2.4e scrub starts 2024-09-11T03:52:29.316 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:29 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:52:29.316 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:29 smithi084 ceph-mon[27254]: 2.50 scrub starts 2024-09-11T03:52:29.316 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:29 smithi084 ceph-mon[27254]: 2.51 scrub starts 2024-09-11T03:52:29.316 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:29 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/473607380' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "96"} : dispatch 2024-09-11T03:52:29.316 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:29 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:52:29.348 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:29.370 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:29.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:29.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:29.472 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:29 smithi175 ceph-mon[30894]: 2.51 scrub starts 2024-09-11T03:52:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:29 smithi175 ceph-mon[30894]: 2.53 scrub starts 2024-09-11T03:52:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:29 smithi175 ceph-mon[30894]: 2.52 scrub starts 2024-09-11T03:52:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:29 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:52:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:29 smithi175 ceph-mon[30894]: osdmap e930: 12 total, 12 up, 11 in 2024-09-11T03:52:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:29 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3313411155' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:52:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:29 smithi175 ceph-mon[30894]: 2.4e scrub starts 2024-09-11T03:52:29.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:29 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:52:29.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:29 smithi175 ceph-mon[30894]: 2.50 scrub starts 2024-09-11T03:52:29.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:29 smithi175 ceph-mon[30894]: 2.51 scrub starts 2024-09-11T03:52:29.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:29 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/473607380' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "96"} : dispatch 2024-09-11T03:52:29.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:29 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:52:29.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:29.495 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:29.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:29.558 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:52:29.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:29.623 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:29.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:29.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:29 smithi121 ceph-mon[30729]: 2.51 scrub starts 2024-09-11T03:52:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:29 smithi121 ceph-mon[30729]: 2.53 scrub starts 2024-09-11T03:52:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:29 smithi121 ceph-mon[30729]: 2.52 scrub starts 2024-09-11T03:52:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:29 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:52:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:29 smithi121 ceph-mon[30729]: osdmap e930: 12 total, 12 up, 11 in 2024-09-11T03:52:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:29 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3313411155' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:52:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:29 smithi121 ceph-mon[30729]: 2.4e scrub starts 2024-09-11T03:52:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:29 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:52:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:29 smithi121 ceph-mon[30729]: 2.50 scrub starts 2024-09-11T03:52:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:29 smithi121 ceph-mon[30729]: 2.51 scrub starts 2024-09-11T03:52:29.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:29 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/473607380' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "96"} : dispatch 2024-09-11T03:52:29.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:29 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:52:29.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:29.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:29.751 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:29.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:29.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:29.878 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:29.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:29.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:29.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:30.002 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:30.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:30.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:30.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:30.125 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:30.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:30.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:30.204 INFO:tasks.rados.rados.2.smithi175.stdout:3878: done (0 left) 2024-09-11T03:52:30.204 INFO:tasks.rados.rados.2.smithi175.stdout:3879: rollback oid 40 current snap is 430 2024-09-11T03:52:30.204 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 40 to 393 2024-09-11T03:52:30.204 INFO:tasks.rados.rados.2.smithi175.stdout:3880: copy_from oid 44 from oid 2 current snap is 430 2024-09-11T03:52:30.205 INFO:tasks.rados.rados.2.smithi175.stdout:3881: delete oid 41 current snap is 430 2024-09-11T03:52:30.212 INFO:tasks.rados.rados.2.smithi175.stdout:3880: finishing copy_from to smithi17557870-44 2024-09-11T03:52:30.212 INFO:tasks.rados.rados.2.smithi175.stdout:3880: got expected ENOENT (src dne) 2024-09-11T03:52:30.212 INFO:tasks.rados.rados.2.smithi175.stdout:3880: finishing copy_from racing read to smithi17557870-44 2024-09-11T03:52:30.217 INFO:tasks.rados.rados.2.smithi175.stdout:3880: done (2 left) 2024-09-11T03:52:30.217 INFO:tasks.rados.rados.2.smithi175.stdout:3881: done (1 left) 2024-09-11T03:52:30.217 INFO:tasks.rados.rados.2.smithi175.stdout:3882: snap_remove snap 412 2024-09-11T03:52:30.227 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:30.251 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:30.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:30.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:30.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:30.378 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:30.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:30.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:30 smithi175 ceph-mon[30894]: 2.53 scrub starts 2024-09-11T03:52:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:30 smithi175 ceph-mon[30894]: pgmap v1551: 117 pgs: 5 active+clean+scrubbing, 112 active+clean; 720 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 14 MiB/s rd, 4.3 MiB/s wr, 27 op/s; 2.4 MiB/s, 11 objects/s recovering 2024-09-11T03:52:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:30 smithi175 ceph-mon[30894]: 2.52 scrub starts 2024-09-11T03:52:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:30 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:52:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:30 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:30 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3313411155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:52:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:30 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/473607380' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "96"}]': finished 2024-09-11T03:52:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:30 smithi175 ceph-mon[30894]: osdmap e931: 12 total, 12 up, 11 in 2024-09-11T03:52:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:30 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:52:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:30 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:52:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:30 smithi175 ceph-mon[30894]: 2.50 scrub starts 2024-09-11T03:52:30.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:30 smithi175 ceph-mon[30894]: 2.4d scrub starts 2024-09-11T03:52:30.479 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:30.504 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:30.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:30.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:30.606 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:30.628 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:30.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:30.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:30 smithi084 ceph-mon[27254]: 2.53 scrub starts 2024-09-11T03:52:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:30 smithi084 ceph-mon[27254]: pgmap v1551: 117 pgs: 5 active+clean+scrubbing, 112 active+clean; 720 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 14 MiB/s rd, 4.3 MiB/s wr, 27 op/s; 2.4 MiB/s, 11 objects/s recovering 2024-09-11T03:52:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:30 smithi084 ceph-mon[27254]: 2.52 scrub starts 2024-09-11T03:52:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:30 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:52:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:30 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:30 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3313411155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:52:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:30 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/473607380' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "96"}]': finished 2024-09-11T03:52:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:30 smithi084 ceph-mon[27254]: osdmap e931: 12 total, 12 up, 11 in 2024-09-11T03:52:30.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:30 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:52:30.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:30 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:52:30.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:30 smithi084 ceph-mon[27254]: 2.50 scrub starts 2024-09-11T03:52:30.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:30 smithi084 ceph-mon[27254]: 2.4d scrub starts 2024-09-11T03:52:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:30 smithi121 ceph-mon[30729]: 2.53 scrub starts 2024-09-11T03:52:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:30 smithi121 ceph-mon[30729]: pgmap v1551: 117 pgs: 5 active+clean+scrubbing, 112 active+clean; 720 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 14 MiB/s rd, 4.3 MiB/s wr, 27 op/s; 2.4 MiB/s, 11 objects/s recovering 2024-09-11T03:52:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:30 smithi121 ceph-mon[30729]: 2.52 scrub starts 2024-09-11T03:52:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:30 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:52:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:30 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:30 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3313411155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:52:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:30 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/473607380' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "96"}]': finished 2024-09-11T03:52:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:30 smithi121 ceph-mon[30729]: osdmap e931: 12 total, 12 up, 11 in 2024-09-11T03:52:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:30 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:52:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:30 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:52:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:30 smithi121 ceph-mon[30729]: 2.50 scrub starts 2024-09-11T03:52:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:30 smithi121 ceph-mon[30729]: 2.4d scrub starts 2024-09-11T03:52:30.730 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:30.751 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:30.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:30.853 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:30.876 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:30.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:30.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:30.978 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:31.004 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:31.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:31.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:31.106 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:31.127 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:31.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:31.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:31.213 INFO:tasks.rados.rados.2.smithi175.stdout:3882: done (1 left) 2024-09-11T03:52:31.214 INFO:tasks.rados.rados.2.smithi175.stdout:3883: snap_remove snap 427 2024-09-11T03:52:31.229 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:31.252 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:31.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:31.253 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:31.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-11T03:52:31.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:31.368 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:31.378 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:31.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:31 smithi175 ceph-mon[30894]: 2.53 scrub starts 2024-09-11T03:52:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:31 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:52:31.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:31 smithi175 ceph-mon[30894]: osdmap e932: 12 total, 12 up, 11 in 2024-09-11T03:52:31.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:31 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:52:31.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:31 smithi175 ceph-mon[30894]: 2.55 scrub starts 2024-09-11T03:52:31.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:31 smithi175 ceph-mon[30894]: 2.f scrub starts 2024-09-11T03:52:31.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:31 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:52:31.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:31 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:52:31.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:31 smithi175 ceph-mon[30894]: 2.51 scrub starts 2024-09-11T03:52:31.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:31 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"} : dispatch 2024-09-11T03:52:31.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:31 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"} : dispatch 2024-09-11T03:52:31.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:31 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:52:31.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:31.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:31 smithi084 ceph-mon[27254]: 2.53 scrub starts 2024-09-11T03:52:31.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:31 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:52:31.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:31 smithi084 ceph-mon[27254]: osdmap e932: 12 total, 12 up, 11 in 2024-09-11T03:52:31.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:31 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:52:31.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:31 smithi084 ceph-mon[27254]: 2.55 scrub starts 2024-09-11T03:52:31.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:31 smithi084 ceph-mon[27254]: 2.f scrub starts 2024-09-11T03:52:31.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:31 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:52:31.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:31 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:52:31.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:31 smithi084 ceph-mon[27254]: 2.51 scrub starts 2024-09-11T03:52:31.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:31 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"} : dispatch 2024-09-11T03:52:31.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:31 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"} : dispatch 2024-09-11T03:52:31.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:31 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:52:31.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:31.504 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:31.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:31.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:31.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:31.631 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:31.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:31 smithi121 ceph-mon[30729]: 2.53 scrub starts 2024-09-11T03:52:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:31 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:52:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:31 smithi121 ceph-mon[30729]: osdmap e932: 12 total, 12 up, 11 in 2024-09-11T03:52:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:31 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:52:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:31 smithi121 ceph-mon[30729]: 2.55 scrub starts 2024-09-11T03:52:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:31 smithi121 ceph-mon[30729]: 2.f scrub starts 2024-09-11T03:52:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:31 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:52:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:31 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:52:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:31 smithi121 ceph-mon[30729]: 2.51 scrub starts 2024-09-11T03:52:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:31 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"} : dispatch 2024-09-11T03:52:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:31 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"} : dispatch 2024-09-11T03:52:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:31 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:52:31.733 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:31.756 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:31.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:31.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-11T03:52:31.849 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T03:52:31.858 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:31.865 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:52:31.866 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:52:31.883 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:31.884 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:31.884 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:52:31.885 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:52:31.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:31.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:31.901 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:52:31.902 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:52:31.924 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:52:31.925 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:52:31.941 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:52:31.942 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:52:31.958 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:52:31.959 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:52:31.976 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:52:31.976 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:52:31.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:31.993 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:52:31.993 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:52:32.010 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:52:32.011 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:52:32.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:32.012 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:32.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:32.027 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:52:32.027 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:52:32.044 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:52:32.044 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:52:32.071 INFO:tasks.rados.rados.2.smithi175.stdout:3883: done (1 left) 2024-09-11T03:52:32.071 INFO:tasks.rados.rados.2.smithi175.stdout:3884: rollback oid 4 current snap is 430 2024-09-11T03:52:32.072 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 4 to 407 2024-09-11T03:52:32.072 INFO:tasks.rados.rados.2.smithi175.stdout:3885: write oid 46 current snap is 430 2024-09-11T03:52:32.072 INFO:tasks.rados.rados.2.smithi175.stdout:3885: seq_num 1317 ranges {534819=432002,1684197=450622} 2024-09-11T03:52:32.076 INFO:tasks.rados.rados.2.smithi175.stdout:3885: writing smithi17557870-46 from 534819 to 966821 tid 1 2024-09-11T03:52:32.084 INFO:tasks.rados.rados.2.smithi175.stdout:3885: writing smithi17557870-46 from 1684197 to 2134819 tid 2 2024-09-11T03:52:32.084 INFO:tasks.rados.rados.2.smithi175.stdout:3886: read oid 24 snap -1 2024-09-11T03:52:32.084 INFO:tasks.rados.rados.2.smithi175.stdout:3886: expect (ObjNum 1235 snap 402 seq_num 1235) 2024-09-11T03:52:32.084 INFO:tasks.rados.rados.2.smithi175.stdout:3887: write (excl) oid 30 current snap is 430 2024-09-11T03:52:32.084 INFO:tasks.rados.rados.2.smithi175.stdout:3879: finishing rollback tid 0 to smithi17557870-40 2024-09-11T03:52:32.085 INFO:tasks.rados.rados.2.smithi175.stdout:3879: finishing rollback tid 1 to smithi17557870-40 2024-09-11T03:52:32.085 INFO:tasks.rados.rados.2.smithi175.stdout:3879: finishing rollback tid 2 to smithi17557870-40 2024-09-11T03:52:32.085 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 40 v 1255 (ObjNum 973 snap 313 seq_num 973) dirty exists 2024-09-11T03:52:32.085 INFO:tasks.rados.rados.2.smithi175.stdout:3887: seq_num 1318 ranges {551626=507251,1626326=418456} 2024-09-11T03:52:32.091 INFO:tasks.rados.rados.2.smithi175.stdout:3887: writing smithi17557870-30 from 551626 to 1058877 tid 1 2024-09-11T03:52:32.098 INFO:tasks.rados.rados.2.smithi175.stdout:3887: writing smithi17557870-30 from 1626326 to 2044782 tid 2 2024-09-11T03:52:32.098 INFO:tasks.rados.rados.2.smithi175.stdout:3879: done (4 left) 2024-09-11T03:52:32.098 INFO:tasks.rados.rados.2.smithi175.stdout:3888: copy_from oid 22 from oid 47 current snap is 430 2024-09-11T03:52:32.098 INFO:tasks.rados.rados.2.smithi175.stdout:3889: rollback oid 36 current snap is 430 2024-09-11T03:52:32.098 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 36 to 354 2024-09-11T03:52:32.099 INFO:tasks.rados.rados.2.smithi175.stdout:3890: snap_remove snap 428 2024-09-11T03:52:32.113 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:32.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:32.135 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:32.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:32.236 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:32.241 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:52:32.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:32.259 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:32.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:32.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:32 smithi084 ceph-mon[27254]: 2.4f scrub starts 2024-09-11T03:52:32.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:32 smithi084 ceph-mon[27254]: pgmap v1554: 117 pgs: 5 active+clean+scrubbing, 112 active+clean; 720 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 18 op/s; 2.4 MiB/s, 10 objects/s recovering 2024-09-11T03:52:32.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:32 smithi084 ceph-mon[27254]: 2.52 scrub starts 2024-09-11T03:52:32.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:32 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:52:32.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:32 smithi084 ceph-mon[27254]: 2.4e scrub starts 2024-09-11T03:52:32.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:32 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"}]': finished 2024-09-11T03:52:32.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:32 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:52:32.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:32 smithi084 ceph-mon[27254]: osdmap e933: 12 total, 12 up, 11 in 2024-09-11T03:52:32.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:32 smithi084 ceph-mon[27254]: 2.50 scrub starts 2024-09-11T03:52:32.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:32 smithi084 ceph-mon[27254]: 2.4d scrub starts 2024-09-11T03:52:32.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:32 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3180897116' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:32.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:32 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:52:32.279 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:32 smithi084 ceph-mon[27254]: osdmap e934: 12 total, 12 up, 11 in 2024-09-11T03:52:32.359 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:32.384 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:32.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:32.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:32 smithi175 ceph-mon[30894]: 2.4f scrub starts 2024-09-11T03:52:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:32 smithi175 ceph-mon[30894]: pgmap v1554: 117 pgs: 5 active+clean+scrubbing, 112 active+clean; 720 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 18 op/s; 2.4 MiB/s, 10 objects/s recovering 2024-09-11T03:52:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:32 smithi175 ceph-mon[30894]: 2.52 scrub starts 2024-09-11T03:52:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:32 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:52:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:32 smithi175 ceph-mon[30894]: 2.4e scrub starts 2024-09-11T03:52:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:32 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"}]': finished 2024-09-11T03:52:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:32 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:52:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:32 smithi175 ceph-mon[30894]: osdmap e933: 12 total, 12 up, 11 in 2024-09-11T03:52:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:32 smithi175 ceph-mon[30894]: 2.50 scrub starts 2024-09-11T03:52:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:32 smithi175 ceph-mon[30894]: 2.4d scrub starts 2024-09-11T03:52:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:32 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3180897116' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:32 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:52:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:32 smithi175 ceph-mon[30894]: osdmap e934: 12 total, 12 up, 11 in 2024-09-11T03:52:32.486 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:32.511 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:32.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:32.512 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:32.588 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:52:32.588 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-11T03:52:32.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-11T03:52:32.588 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-11T03:52:32.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.588 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:4498 2.e 2:7b0c3397:::smithi17557870-14:head [copy-from ver 491] snapc c5=[c5,c4,c3,c2,c1,c0,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-09-11T03:52:32.589 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:45:03.958685+0000", 2024-09-11T03:52:32.589 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 448.629158244, 2024-09-11T03:52:32.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014061295999999999, 2024-09-11T03:52:32.589 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.589 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.589 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:32.589 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:32.589 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:32.589 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:32.589 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4498 2024-09-11T03:52:32.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.589 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958685+0000", 2024-09-11T03:52:32.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958685+0000", 2024-09-11T03:52:32.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958687+0000", 2024-09-11T03:52:32.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.609e-06 2024-09-11T03:52:32.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958695+0000", 2024-09-11T03:52:32.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5410000000000003e-06 2024-09-11T03:52:32.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958711+0000", 2024-09-11T03:52:32.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6719999999999999e-05 2024-09-11T03:52:32.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958718+0000", 2024-09-11T03:52:32.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9720000000000003e-06 2024-09-11T03:52:32.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958804+0000", 2024-09-11T03:52:32.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5483999999999997e-05 2024-09-11T03:52:32.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:52:32.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.958811+0000", 2024-09-11T03:52:32.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4059999999999999e-06 2024-09-11T03:52:32.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.960161+0000", 2024-09-11T03:52:32.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001349772 2024-09-11T03:52:32.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:32.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.960244+0000", 2024-09-11T03:52:32.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3461999999999995e-05 2024-09-11T03:52:32.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:52:32.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.967582+0000", 2024-09-11T03:52:32.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073378480000000001 2024-09-11T03:52:32.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:32.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.970934+0000", 2024-09-11T03:52:32.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003351812 2024-09-11T03:52:32.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:32.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.971224+0000", 2024-09-11T03:52:32.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029005399999999998 2024-09-11T03:52:32.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:32.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.972661+0000", 2024-09-11T03:52:32.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001436866 2024-09-11T03:52:32.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:32.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.972700+0000", 2024-09-11T03:52:32.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9341000000000002e-05 2024-09-11T03:52:32.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:45:03.972747+0000", 2024-09-11T03:52:32.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6409000000000002e-05 2024-09-11T03:52:32.595 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.596 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.596 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.596 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5201 2.e 2:750de3b7:::smithi17557870-8:head [copy-from ver 512] snapc e6=[e5,e1,e0,da,d1] ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-09-11T03:52:32.596 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:12.961903+0000", 2024-09-11T03:52:32.596 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 379.62594051899998, 2024-09-11T03:52:32.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022459643000000001, 2024-09-11T03:52:32.596 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.596 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.596 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:32.596 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:32.596 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:32.596 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:32.597 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5201 2024-09-11T03:52:32.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.597 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961903+0000", 2024-09-11T03:52:32.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961903+0000", 2024-09-11T03:52:32.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961903+0000", 2024-09-11T03:52:32.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0199999999999998e-07 2024-09-11T03:52:32.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961906+0000", 2024-09-11T03:52:32.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.119e-06 2024-09-11T03:52:32.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961911+0000", 2024-09-11T03:52:32.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7529999999999998e-06 2024-09-11T03:52:32.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961915+0000", 2024-09-11T03:52:32.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6320000000000001e-06 2024-09-11T03:52:32.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961955+0000", 2024-09-11T03:52:32.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0497999999999997e-05 2024-09-11T03:52:32.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-11T03:52:32.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961961+0000", 2024-09-11T03:52:32.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3800000000000002e-06 2024-09-11T03:52:32.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.963330+0000", 2024-09-11T03:52:32.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001368773 2024-09-11T03:52:32.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:32.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.963415+0000", 2024-09-11T03:52:32.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5579000000000002e-05 2024-09-11T03:52:32.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:52:32.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.970784+0000", 2024-09-11T03:52:32.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073691750000000004 2024-09-11T03:52:32.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-11T03:52:32.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.975393+0000", 2024-09-11T03:52:32.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046089010000000003 2024-09-11T03:52:32.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:32.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.976463+0000", 2024-09-11T03:52:32.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010695850000000001 2024-09-11T03:52:32.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:32.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.984271+0000", 2024-09-11T03:52:32.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0078078970000000003 2024-09-11T03:52:32.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:32.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.984316+0000", 2024-09-11T03:52:32.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5209e-05 2024-09-11T03:52:32.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.984363+0000", 2024-09-11T03:52:32.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6839999999999999e-05 2024-09-11T03:52:32.603 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.603 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.603 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.604 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24454.0:5202 2.e 2:750de3b7:::smithi17557870-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e555)", 2024-09-11T03:52:32.604 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:12.961921+0000", 2024-09-11T03:52:32.604 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 379.62592232399999, 2024-09-11T03:52:32.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013412446999999999, 2024-09-11T03:52:32.604 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.604 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.604 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:32.604 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-11T03:52:32.604 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24454", 2024-09-11T03:52:32.604 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:32.604 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5202 2024-09-11T03:52:32.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.605 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961921+0000", 2024-09-11T03:52:32.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961921+0000", 2024-09-11T03:52:32.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961922+0000", 2024-09-11T03:52:32.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2499999999999999e-07 2024-09-11T03:52:32.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961923+0000", 2024-09-11T03:52:32.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.42e-06 2024-09-11T03:52:32.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961927+0000", 2024-09-11T03:52:32.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9960000000000004e-06 2024-09-11T03:52:32.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961930+0000", 2024-09-11T03:52:32.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9610000000000001e-06 2024-09-11T03:52:32.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961983+0000", 2024-09-11T03:52:32.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2967000000000003e-05 2024-09-11T03:52:32.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:32.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.961986+0000", 2024-09-11T03:52:32.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0960000000000001e-06 2024-09-11T03:52:32.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.963568+0000", 2024-09-11T03:52:32.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015823530000000001 2024-09-11T03:52:32.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:32.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.963621+0000", 2024-09-11T03:52:32.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2358000000000003e-05 2024-09-11T03:52:32.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.974667+0000", 2024-09-11T03:52:32.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011046517 2024-09-11T03:52:32.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:32.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.974875+0000", 2024-09-11T03:52:32.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000207853 2024-09-11T03:52:32.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:12.975334+0000", 2024-09-11T03:52:32.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000458701 2024-09-11T03:52:32.610 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.610 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.610 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.610 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:5691 2.e e601/577 2:77dadb97:::smithi17557870-30:head v 601'877, mlcod=598'875)", 2024-09-11T03:52:32.610 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:46:59.009702+0000", 2024-09-11T03:52:32.610 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 333.578141727, 2024-09-11T03:52:32.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.051323566000000001, 2024-09-11T03:52:32.610 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.610 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.610 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:32.610 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.009702+0000", 2024-09-11T03:52:32.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.009702+0000", 2024-09-11T03:52:32.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.009703+0000", 2024-09-11T03:52:32.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4750000000000001e-06 2024-09-11T03:52:32.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.059761+0000", 2024-09-11T03:52:32.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.050057579999999997 2024-09-11T03:52:32.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.059778+0000", 2024-09-11T03:52:32.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7051999999999999e-05 2024-09-11T03:52:32.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.059784+0000", 2024-09-11T03:52:32.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.648e-06 2024-09-11T03:52:32.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.059832+0000", 2024-09-11T03:52:32.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7828999999999999e-05 2024-09-11T03:52:32.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:32.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.059847+0000", 2024-09-11T03:52:32.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5778999999999999e-05 2024-09-11T03:52:32.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:32.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.060993+0000", 2024-09-11T03:52:32.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001145566 2024-09-11T03:52:32.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:46:59.061026+0000", 2024-09-11T03:52:32.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2636999999999999e-05 2024-09-11T03:52:32.614 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.614 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.614 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.616 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:6152 2.e e637/577 2:77dadb97:::smithi17557870-30:head v 637'891, mlcod=634'888)", 2024-09-11T03:52:32.616 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:47:35.714649+0000", 2024-09-11T03:52:32.616 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 296.87319422399997, 2024-09-11T03:52:32.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011875179, 2024-09-11T03:52:32.616 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.616 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.616 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:32.616 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714649+0000", 2024-09-11T03:52:32.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714649+0000", 2024-09-11T03:52:32.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.714650+0000", 2024-09-11T03:52:32.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.44e-07 2024-09-11T03:52:32.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721021+0000", 2024-09-11T03:52:32.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0063714569999999996 2024-09-11T03:52:32.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721040+0000", 2024-09-11T03:52:32.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8590999999999999e-05 2024-09-11T03:52:32.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721048+0000", 2024-09-11T03:52:32.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9489999999999996e-06 2024-09-11T03:52:32.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721107+0000", 2024-09-11T03:52:32.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8736999999999997e-05 2024-09-11T03:52:32.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:32.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.721123+0000", 2024-09-11T03:52:32.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6735000000000002e-05 2024-09-11T03:52:32.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:32.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.726469+0000", 2024-09-11T03:52:32.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053456140000000003 2024-09-11T03:52:32.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:47:35.726525+0000", 2024-09-11T03:52:32.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5551999999999999e-05 2024-09-11T03:52:32.620 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.620 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.620 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.620 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:6546 2.e e669/577 2:750de3b7:::smithi17557870-8:head v 669'909, mlcod=667'905)", 2024-09-11T03:52:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:48:07.227022+0000", 2024-09-11T03:52:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 265.36082126399998, 2024-09-11T03:52:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013091959, 2024-09-11T03:52:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.227022+0000", 2024-09-11T03:52:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.227022+0000", 2024-09-11T03:52:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.227023+0000", 2024-09-11T03:52:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7300000000000001e-07 2024-09-11T03:52:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.234175+0000", 2024-09-11T03:52:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071522260000000002 2024-09-11T03:52:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.234198+0000", 2024-09-11T03:52:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2492000000000001e-05 2024-09-11T03:52:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.234211+0000", 2024-09-11T03:52:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3534e-05 2024-09-11T03:52:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.234289+0000", 2024-09-11T03:52:32.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8078999999999996e-05 2024-09-11T03:52:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.234307+0000", 2024-09-11T03:52:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8206000000000002e-05 2024-09-11T03:52:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.240060+0000", 2024-09-11T03:52:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0057523069999999999 2024-09-11T03:52:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:48:07.240114+0000", 2024-09-11T03:52:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4642000000000003e-05 2024-09-11T03:52:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:7879 2.7 e779/745 2:e0ec5763:::smithi17557870-2:head v 779'1108, mlcod=778'1104)", 2024-09-11T03:52:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:49:57.174324+0000", 2024-09-11T03:52:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 155.413519705, 2024-09-11T03:52:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081549459999999997, 2024-09-11T03:52:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.625 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.174324+0000", 2024-09-11T03:52:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.174324+0000", 2024-09-11T03:52:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.174325+0000", 2024-09-11T03:52:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3990000000000001e-06 2024-09-11T03:52:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.180038+0000", 2024-09-11T03:52:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0057125869999999999 2024-09-11T03:52:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.180058+0000", 2024-09-11T03:52:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0271e-05 2024-09-11T03:52:32.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.180067+0000", 2024-09-11T03:52:32.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6470000000000004e-06 2024-09-11T03:52:32.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.180129+0000", 2024-09-11T03:52:32.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2193999999999994e-05 2024-09-11T03:52:32.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:32.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.180149+0000", 2024-09-11T03:52:32.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9553999999999999e-05 2024-09-11T03:52:32.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:32.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.182435+0000", 2024-09-11T03:52:32.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022862289999999999 2024-09-11T03:52:32.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:49:57.182479+0000", 2024-09-11T03:52:32.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4064999999999999e-05 2024-09-11T03:52:32.629 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.629 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.629 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:8103 2.7 e795/745 2:e23d1762:::smithi17557870-21:head v 795'1150, mlcod=795'1147)", 2024-09-11T03:52:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:50:13.131747+0000", 2024-09-11T03:52:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 139.456096987, 2024-09-11T03:52:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024190567999999999, 2024-09-11T03:52:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:13.131747+0000", 2024-09-11T03:52:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:13.131747+0000", 2024-09-11T03:52:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:13.131748+0000", 2024-09-11T03:52:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.435e-06 2024-09-11T03:52:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:13.154306+0000", 2024-09-11T03:52:32.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022557423 2024-09-11T03:52:32.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:13.154323+0000", 2024-09-11T03:52:32.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7841000000000001e-05 2024-09-11T03:52:32.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:13.154332+0000", 2024-09-11T03:52:32.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8540000000000003e-06 2024-09-11T03:52:32.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:13.154400+0000", 2024-09-11T03:52:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7447999999999995e-05 2024-09-11T03:52:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:13.154428+0000", 2024-09-11T03:52:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8065e-05 2024-09-11T03:52:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:13.155897+0000", 2024-09-11T03:52:32.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014693810000000001 2024-09-11T03:52:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:13.155937+0000", 2024-09-11T03:52:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0120999999999998e-05 2024-09-11T03:52:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:560 2.7 e812/745 2:e0eabcfb:::smithi17557870-9:170 v 812'1186, mlcod=811'1183)", 2024-09-11T03:52:32.634 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:50:30.039230+0000", 2024-09-11T03:52:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 122.548613553, 2024-09-11T03:52:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0070068329999999996, 2024-09-11T03:52:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:30.039230+0000", 2024-09-11T03:52:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:30.039230+0000", 2024-09-11T03:52:32.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:30.039231+0000", 2024-09-11T03:52:32.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6199999999999997e-07 2024-09-11T03:52:32.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:30.039239+0000", 2024-09-11T03:52:32.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7889999999999995e-06 2024-09-11T03:52:32.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:30.039251+0000", 2024-09-11T03:52:32.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2682e-05 2024-09-11T03:52:32.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:30.039262+0000", 2024-09-11T03:52:32.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0501e-05 2024-09-11T03:52:32.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:30.039340+0000", 2024-09-11T03:52:32.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8453000000000003e-05 2024-09-11T03:52:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:30.039364+0000", 2024-09-11T03:52:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3275e-05 2024-09-11T03:52:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:30.046185+0000", 2024-09-11T03:52:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068210170000000004 2024-09-11T03:52:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:30.046237+0000", 2024-09-11T03:52:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2354000000000001e-05 2024-09-11T03:52:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:8480 2.7 e828/745 2:e23d1762:::smithi17557870-21:head v 828'1212, mlcod=826'1208)", 2024-09-11T03:52:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:50:46.076086+0000", 2024-09-11T03:52:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 106.51175764600001, 2024-09-11T03:52:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065259840000000003, 2024-09-11T03:52:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.639 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:46.076086+0000", 2024-09-11T03:52:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:46.076086+0000", 2024-09-11T03:52:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:46.076086+0000", 2024-09-11T03:52:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2999999999999999e-07 2024-09-11T03:52:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:46.080539+0000", 2024-09-11T03:52:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044524289999999999 2024-09-11T03:52:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:46.080552+0000", 2024-09-11T03:52:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3191e-05 2024-09-11T03:52:32.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:46.080560+0000", 2024-09-11T03:52:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2160000000000006e-06 2024-09-11T03:52:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:46.080616+0000", 2024-09-11T03:52:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5481e-05 2024-09-11T03:52:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:46.080635+0000", 2024-09-11T03:52:32.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9490000000000001e-05 2024-09-11T03:52:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:46.082568+0000", 2024-09-11T03:52:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019332520000000001 2024-09-11T03:52:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:46.082612+0000", 2024-09-11T03:52:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3695000000000001e-05 2024-09-11T03:52:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.643 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:8665 2.3 e840/569 2:c3768452:::smithi17557870-11:head v 840'558, mlcod=838'554)", 2024-09-11T03:52:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:50:58.104382+0000", 2024-09-11T03:52:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 94.483461775999999, 2024-09-11T03:52:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059235889999999999, 2024-09-11T03:52:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:58.104382+0000", 2024-09-11T03:52:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:58.104382+0000", 2024-09-11T03:52:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:58.104383+0000", 2024-09-11T03:52:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.438e-06 2024-09-11T03:52:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:58.107678+0000", 2024-09-11T03:52:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0032950689999999999 2024-09-11T03:52:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:58.107698+0000", 2024-09-11T03:52:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9939e-05 2024-09-11T03:52:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:58.107708+0000", 2024-09-11T03:52:32.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1789999999999997e-06 2024-09-11T03:52:32.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:58.107775+0000", 2024-09-11T03:52:32.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6958999999999994e-05 2024-09-11T03:52:32.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:32.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:58.107794+0000", 2024-09-11T03:52:32.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8998999999999999e-05 2024-09-11T03:52:32.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:32.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:58.110252+0000", 2024-09-11T03:52:32.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024580349999999999 2024-09-11T03:52:32.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:50:58.110306+0000", 2024-09-11T03:52:32.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3971000000000002e-05 2024-09-11T03:52:32.648 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.648 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.648 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.649 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:701 2.7 e853/849 2:e0ec5763:::smithi17557870-2:185 v 853'1264, mlcod=848'1260)", 2024-09-11T03:52:32.649 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:11.058613+0000", 2024-09-11T03:52:32.649 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 81.529230745999996, 2024-09-11T03:52:32.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061097679999999998, 2024-09-11T03:52:32.649 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.649 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.649 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:32.649 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.058613+0000", 2024-09-11T03:52:32.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.058613+0000", 2024-09-11T03:52:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.058614+0000", 2024-09-11T03:52:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1699999999999997e-07 2024-09-11T03:52:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.058621+0000", 2024-09-11T03:52:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0879999999999997e-06 2024-09-11T03:52:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.058631+0000", 2024-09-11T03:52:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0621999999999999e-05 2024-09-11T03:52:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.058640+0000", 2024-09-11T03:52:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2879999999999993e-06 2024-09-11T03:52:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.060618+0000", 2024-09-11T03:52:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019787469999999999 2024-09-11T03:52:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.060641+0000", 2024-09-11T03:52:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2254000000000001e-05 2024-09-11T03:52:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.064693+0000", 2024-09-11T03:52:32.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040524410000000004 2024-09-11T03:52:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.064723+0000", 2024-09-11T03:52:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9611000000000002e-05 2024-09-11T03:52:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(osd.5.38:56 2.47 2:e23d1762:::smithi17557870-21:head [assert-version v1006,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e853)", 2024-09-11T03:52:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:11.062441+0000", 2024-09-11T03:52:32.653 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 81.525403119999993, 2024-09-11T03:52:32.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018903544000000001, 2024-09-11T03:52:32.654 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.654 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.654 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-11T03:52:32.654 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.062441+0000", 2024-09-11T03:52:32.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.062441+0000", 2024-09-11T03:52:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.062442+0000", 2024-09-11T03:52:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.818e-06 2024-09-11T03:52:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.062450+0000", 2024-09-11T03:52:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4529999999999996e-06 2024-09-11T03:52:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.062466+0000", 2024-09-11T03:52:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6588000000000001e-05 2024-09-11T03:52:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.062476+0000", 2024-09-11T03:52:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.397e-06 2024-09-11T03:52:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.062576+0000", 2024-09-11T03:52:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9912000000000002e-05 2024-09-11T03:52:32.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-11T03:52:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.062587+0000", 2024-09-11T03:52:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1071e-05 2024-09-11T03:52:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.077942+0000", 2024-09-11T03:52:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015355265 2024-09-11T03:52:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.078122+0000", 2024-09-11T03:52:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017955 2024-09-11T03:52:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.081344+0000", 2024-09-11T03:52:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0032224900000000002 2024-09-11T03:52:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "MOSDPGPushReply(2.47 853/852 [PushReplyOp(2:e23d1762:::smithi17557870-21:17c)])", 2024-09-11T03:52:32.658 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:11.121589+0000", 2024-09-11T03:52:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 81.466255019000002, 2024-09-11T03:52:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0075467579999999998, 2024-09-11T03:52:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-11T03:52:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.121589+0000", 2024-09-11T03:52:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.121589+0000", 2024-09-11T03:52:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.121589+0000", 2024-09-11T03:52:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9200000000000001e-07 2024-09-11T03:52:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.121595+0000", 2024-09-11T03:52:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.959e-06 2024-09-11T03:52:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.121609+0000", 2024-09-11T03:52:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4158999999999999e-05 2024-09-11T03:52:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.121617+0000", 2024-09-11T03:52:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8080000000000005e-06 2024-09-11T03:52:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.125680+0000", 2024-09-11T03:52:32.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040631670000000003 2024-09-11T03:52:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.129135+0000", 2024-09-11T03:52:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034550729999999999 2024-09-11T03:52:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "MOSDPGPushReply(2.47 853/852 [PushReplyOp(2:e23d1762:::smithi17557870-21:17c)])", 2024-09-11T03:52:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:11.125161+0000", 2024-09-11T03:52:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 81.462683089999999, 2024-09-11T03:52:32.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065223690000000001, 2024-09-11T03:52:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-11T03:52:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.125161+0000", 2024-09-11T03:52:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.125161+0000", 2024-09-11T03:52:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.664 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.125162+0000", 2024-09-11T03:52:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8400000000000002e-07 2024-09-11T03:52:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.125169+0000", 2024-09-11T03:52:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6790000000000005e-06 2024-09-11T03:52:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.125187+0000", 2024-09-11T03:52:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8048000000000001e-05 2024-09-11T03:52:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.125195+0000", 2024-09-11T03:52:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7109999999999995e-06 2024-09-11T03:52:32.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.129186+0000", 2024-09-11T03:52:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039910279999999998 2024-09-11T03:52:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.131683+0000", 2024-09-11T03:52:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024969189999999998 2024-09-11T03:52:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "MOSDPGPushReply(2.47 853/852 [PushReplyOp(2:e23d1762:::smithi17557870-21:167)])", 2024-09-11T03:52:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:11.187896+0000", 2024-09-11T03:52:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 81.399947443000002, 2024-09-11T03:52:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0075878270000000001, 2024-09-11T03:52:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-11T03:52:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.187896+0000", 2024-09-11T03:52:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.187896+0000", 2024-09-11T03:52:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.187897+0000", 2024-09-11T03:52:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1099999999999996e-07 2024-09-11T03:52:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.187900+0000", 2024-09-11T03:52:32.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6610000000000002e-06 2024-09-11T03:52:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.187909+0000", 2024-09-11T03:52:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3359999999999995e-06 2024-09-11T03:52:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.187916+0000", 2024-09-11T03:52:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0299999999999996e-06 2024-09-11T03:52:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.191772+0000", 2024-09-11T03:52:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0038560299999999999 2024-09-11T03:52:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:11.195484+0000", 2024-09-11T03:52:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037122589999999999 2024-09-11T03:52:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:9045 2.43 e874/854 2:c3768452:::smithi17557870-11:head v 874'578, mlcod=862'574)", 2024-09-11T03:52:32.671 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:51:32.067180+0000", 2024-09-11T03:52:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 60.520663509999999, 2024-09-11T03:52:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0099784469999999997, 2024-09-11T03:52:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:32.067180+0000", 2024-09-11T03:52:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:32.067180+0000", 2024-09-11T03:52:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:32.067180+0000", 2024-09-11T03:52:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.17e-07 2024-09-11T03:52:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:32.074062+0000", 2024-09-11T03:52:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068811380000000002 2024-09-11T03:52:32.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:32.074088+0000", 2024-09-11T03:52:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6744e-05 2024-09-11T03:52:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:32.074102+0000", 2024-09-11T03:52:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3728e-05 2024-09-11T03:52:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:32.074185+0000", 2024-09-11T03:52:32.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3261000000000001e-05 2024-09-11T03:52:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:32.074215+0000", 2024-09-11T03:52:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9268000000000002e-05 2024-09-11T03:52:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:32.077090+0000", 2024-09-11T03:52:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028751829999999999 2024-09-11T03:52:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:51:32.077159+0000", 2024-09-11T03:52:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8907999999999998e-05 2024-09-11T03:52:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:9282 2.7 e904/849 2:e0eabcfb:::smithi17557870-9:head v 904'1328, mlcod=898'1324)", 2024-09-11T03:52:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:52:02.089761+0000", 2024-09-11T03:52:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 30.498082574000001, 2024-09-11T03:52:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071120599999999999, 2024-09-11T03:52:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.676 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:02.089761+0000", 2024-09-11T03:52:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:02.089761+0000", 2024-09-11T03:52:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:02.089761+0000", 2024-09-11T03:52:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9600000000000001e-07 2024-09-11T03:52:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:02.094574+0000", 2024-09-11T03:52:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048128750000000003 2024-09-11T03:52:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:02.094586+0000", 2024-09-11T03:52:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1518000000000001e-05 2024-09-11T03:52:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:02.094594+0000", 2024-09-11T03:52:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3550000000000005e-06 2024-09-11T03:52:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:02.094660+0000", 2024-09-11T03:52:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5650999999999999e-05 2024-09-11T03:52:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:02.094678+0000", 2024-09-11T03:52:32.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8229000000000001e-05 2024-09-11T03:52:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:02.096837+0000", 2024-09-11T03:52:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021589339999999999 2024-09-11T03:52:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:02.096873+0000", 2024-09-11T03:52:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6301999999999999e-05 2024-09-11T03:52:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.680 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24454.0:9493 2.1b e916/812 2:d8055656:::smithi17557870-39:head v 916'939, mlcod=912'935)", 2024-09-11T03:52:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:52:14.092147+0000", 2024-09-11T03:52:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 18.495696993999999, 2024-09-11T03:52:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0043008370000000001, 2024-09-11T03:52:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.681 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:14.092147+0000", 2024-09-11T03:52:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:14.092147+0000", 2024-09-11T03:52:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:14.092148+0000", 2024-09-11T03:52:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8000000000000004e-07 2024-09-11T03:52:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:14.092782+0000", 2024-09-11T03:52:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00063463699999999996 2024-09-11T03:52:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:14.092794+0000", 2024-09-11T03:52:32.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2232000000000001e-05 2024-09-11T03:52:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:14.092802+0000", 2024-09-11T03:52:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7870000000000002e-06 2024-09-11T03:52:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:14.092866+0000", 2024-09-11T03:52:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4142999999999999e-05 2024-09-11T03:52:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:14.092889+0000", 2024-09-11T03:52:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2319000000000001e-05 2024-09-11T03:52:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:14.096398+0000", 2024-09-11T03:52:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0035097790000000002 2024-09-11T03:52:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:14.096448+0000", 2024-09-11T03:52:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9060000000000001e-05 2024-09-11T03:52:32.685 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.11.0:1222 2.1b e926/812 2:d8055656:::smithi17557870-39:19c v 926'946, mlcod=920'942)", 2024-09-11T03:52:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-11T03:52:24.080508+0000", 2024-09-11T03:52:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 8.5073360569999998, 2024-09-11T03:52:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029831829999999999, 2024-09-11T03:52:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-11T03:52:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-11T03:52:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-11T03:52:32.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-11T03:52:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:24.080508+0000", 2024-09-11T03:52:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-11T03:52:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:24.080508+0000", 2024-09-11T03:52:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-11T03:52:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-11T03:52:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:24.080508+0000", 2024-09-11T03:52:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7399999999999999e-07 2024-09-11T03:52:32.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-11T03:52:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:24.080517+0000", 2024-09-11T03:52:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3610000000000007e-06 2024-09-11T03:52:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-11T03:52:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:24.080526+0000", 2024-09-11T03:52:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2090000000000005e-06 2024-09-11T03:52:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-11T03:52:32.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:24.080533+0000", 2024-09-11T03:52:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3370000000000002e-06 2024-09-11T03:52:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-11T03:52:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:24.080607+0000", 2024-09-11T03:52:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3971e-05 2024-09-11T03:52:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-11T03:52:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:24.080627+0000", 2024-09-11T03:52:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0305999999999998e-05 2024-09-11T03:52:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-11T03:52:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:24.083444+0000", 2024-09-11T03:52:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028164840000000002 2024-09-11T03:52:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-11T03:52:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-11T03:52:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-11T03:52:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-11T03:52:24.083491+0000", 2024-09-11T03:52:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7141000000000001e-05 2024-09-11T03:52:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-11T03:52:32.690 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-11T03:52:32.691 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:52:32.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:32 smithi121 ceph-mon[30729]: 2.4f scrub starts 2024-09-11T03:52:32.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:32 smithi121 ceph-mon[30729]: pgmap v1554: 117 pgs: 5 active+clean+scrubbing, 112 active+clean; 720 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 18 op/s; 2.4 MiB/s, 10 objects/s recovering 2024-09-11T03:52:32.691 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:32 smithi121 ceph-mon[30729]: 2.52 scrub starts 2024-09-11T03:52:32.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:32 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:52:32.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:32 smithi121 ceph-mon[30729]: 2.4e scrub starts 2024-09-11T03:52:32.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:32 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"}]': finished 2024-09-11T03:52:32.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:32 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:52:32.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:32 smithi121 ceph-mon[30729]: osdmap e933: 12 total, 12 up, 11 in 2024-09-11T03:52:32.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:32 smithi121 ceph-mon[30729]: 2.50 scrub starts 2024-09-11T03:52:32.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:32 smithi121 ceph-mon[30729]: 2.4d scrub starts 2024-09-11T03:52:32.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:32 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3180897116' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:32.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:32 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:52:32.692 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:32 smithi121 ceph-mon[30729]: osdmap e934: 12 total, 12 up, 11 in 2024-09-11T03:52:32.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:32.693 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:32.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:32.794 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:32.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:32.817 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:32.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:32.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:32.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:32.940 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:32.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:33.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:33.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:33.064 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:33.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:33.080 INFO:tasks.rados.rados.2.smithi175.stdout:3890: done (6 left) 2024-09-11T03:52:33.080 INFO:tasks.rados.rados.2.smithi175.stdout:3891: snap_remove snap 414 2024-09-11T03:52:33.165 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:33.189 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:33.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:33.190 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:33.291 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:33.316 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:33.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:33.317 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:33.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:33.659 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:33 smithi084 ceph-mon[27254]: 2.53 scrub starts 2024-09-11T03:52:33.659 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:33 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:52:33.659 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:33 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:52:33.659 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:33 smithi084 ceph-mon[27254]: 2.52 scrub starts 2024-09-11T03:52:33.659 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:33 smithi084 ceph-mon[27254]: 2.55 scrub starts 2024-09-11T03:52:33.659 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:33 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:52:33.659 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:33 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:52:33.659 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:33 smithi084 ceph-mon[27254]: 2.50 scrub starts 2024-09-11T03:52:33.659 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:33 smithi084 ceph-mon[27254]: 2.4d scrub starts 2024-09-11T03:52:33.659 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:33 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"} : dispatch 2024-09-11T03:52:33.660 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:33 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"} : dispatch 2024-09-11T03:52:33.660 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:33 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:52:33.660 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:33 smithi084 ceph-mon[27254]: 2.52 scrub starts 2024-09-11T03:52:33.660 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:33 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"}]': finished 2024-09-11T03:52:33.660 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:33 smithi084 ceph-mon[27254]: osdmap e935: 12 total, 12 up, 11 in 2024-09-11T03:52:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:33 smithi121 ceph-mon[30729]: 2.53 scrub starts 2024-09-11T03:52:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:33 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:52:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:33 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:52:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:33 smithi121 ceph-mon[30729]: 2.52 scrub starts 2024-09-11T03:52:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:33 smithi121 ceph-mon[30729]: 2.55 scrub starts 2024-09-11T03:52:33.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:33 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:52:33.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:33 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:52:33.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:33 smithi121 ceph-mon[30729]: 2.50 scrub starts 2024-09-11T03:52:33.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:33 smithi121 ceph-mon[30729]: 2.4d scrub starts 2024-09-11T03:52:33.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:33 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"} : dispatch 2024-09-11T03:52:33.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:33 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"} : dispatch 2024-09-11T03:52:33.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:33 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:52:33.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:33 smithi121 ceph-mon[30729]: 2.52 scrub starts 2024-09-11T03:52:33.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:33 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"}]': finished 2024-09-11T03:52:33.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:33 smithi121 ceph-mon[30729]: osdmap e935: 12 total, 12 up, 11 in 2024-09-11T03:52:33.722 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:33.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:33.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:33.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:33 smithi175 ceph-mon[30894]: 2.53 scrub starts 2024-09-11T03:52:33.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:33 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:52:33.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:33 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:52:33.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:33 smithi175 ceph-mon[30894]: 2.52 scrub starts 2024-09-11T03:52:33.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:33 smithi175 ceph-mon[30894]: 2.55 scrub starts 2024-09-11T03:52:33.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:33 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:52:33.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:33 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:52:33.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:33 smithi175 ceph-mon[30894]: 2.50 scrub starts 2024-09-11T03:52:33.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:33 smithi175 ceph-mon[30894]: 2.4d scrub starts 2024-09-11T03:52:33.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:33 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"} : dispatch 2024-09-11T03:52:33.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:33 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"} : dispatch 2024-09-11T03:52:33.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:33 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:52:33.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:33 smithi175 ceph-mon[30894]: 2.52 scrub starts 2024-09-11T03:52:33.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:33 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"}]': finished 2024-09-11T03:52:33.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:33 smithi175 ceph-mon[30894]: osdmap e935: 12 total, 12 up, 11 in 2024-09-11T03:52:33.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:33.850 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:33.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:33.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:33.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:33.978 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:33.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:33.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:34.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-11T03:52:34.080 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:34.081 INFO:tasks.rados.rados.2.smithi175.stdout:3891: done (6 left) 2024-09-11T03:52:34.081 INFO:tasks.rados.rados.2.smithi175.stdout:3892: copy_from oid 31 from oid 23 current snap is 430 2024-09-11T03:52:34.083 INFO:tasks.rados.rados.2.smithi175.stdout:3885: finishing write tid 1 to smithi17557870-46 2024-09-11T03:52:34.083 INFO:tasks.rados.rados.2.smithi175.stdout:3885: finishing write tid 2 to smithi17557870-46 2024-09-11T03:52:34.083 INFO:tasks.rados.rados.2.smithi175.stdout:3887: finishing write tid 1 to smithi17557870-30 2024-09-11T03:52:34.083 INFO:tasks.rados.rados.2.smithi175.stdout:3884: finishing rollback tid 1 to smithi17557870-4 2024-09-11T03:52:34.084 INFO:tasks.rados.rados.2.smithi175.stdout:3884: finishing rollback tid 2 to smithi17557870-4 2024-09-11T03:52:34.084 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 1296 (ObjNum 1220 snap 399 seq_num 1220) dirty exists 2024-09-11T03:52:34.084 INFO:tasks.rados.rados.2.smithi175.stdout:3887: finishing write tid 2 to smithi17557870-30 2024-09-11T03:52:34.084 INFO:tasks.rados.rados.2.smithi175.stdout:3887: finishing write tid 3 to smithi17557870-30 2024-09-11T03:52:34.084 INFO:tasks.rados.rados.2.smithi175.stdout:3885: finishing write tid 3 to smithi17557870-46 2024-09-11T03:52:34.084 INFO:tasks.rados.rados.2.smithi175.stdout:3885: finishing write tid 4 to smithi17557870-46 2024-09-11T03:52:34.084 INFO:tasks.rados.rados.2.smithi175.stdout:3885: oid 46 updating version 0 to 981 2024-09-11T03:52:34.084 INFO:tasks.rados.rados.2.smithi175.stdout:3885: oid 46 updating version 981 to 982 2024-09-11T03:52:34.084 INFO:tasks.rados.rados.2.smithi175.stdout:3885: oid 46 version 982 is already newer than 980 2024-09-11T03:52:34.084 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 982 (ObjNum 1317 snap 430 seq_num 1317) dirty exists 2024-09-11T03:52:34.084 INFO:tasks.rados.rados.2.smithi175.stdout:3885: left oid 46 (ObjNum 1317 snap 430 seq_num 1317) 2024-09-11T03:52:34.084 INFO:tasks.rados.rados.2.smithi175.stdout:3887: finishing write tid 4 to smithi17557870-30 2024-09-11T03:52:34.084 INFO:tasks.rados.rados.2.smithi175.stdout:3887: oid 30 updating version 0 to 1268 2024-09-11T03:52:34.084 INFO:tasks.rados.rados.2.smithi175.stdout:3887: oid 30 version 1268 is already newer than 1267 2024-09-11T03:52:34.085 INFO:tasks.rados.rados.2.smithi175.stdout:3887: oid 30 updating version 1268 to 1269 2024-09-11T03:52:34.085 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 1269 (ObjNum 1318 snap 430 seq_num 1318) dirty exists 2024-09-11T03:52:34.085 INFO:tasks.rados.rados.2.smithi175.stdout:3887: left oid 30 (ObjNum 1318 snap 430 seq_num 1318) 2024-09-11T03:52:34.085 INFO:tasks.rados.rados.2.smithi175.stdout:3884: done (6 left) 2024-09-11T03:52:34.085 INFO:tasks.rados.rados.2.smithi175.stdout:3885: done (5 left) 2024-09-11T03:52:34.085 INFO:tasks.rados.rados.2.smithi175.stdout:3887: done (4 left) 2024-09-11T03:52:34.085 INFO:tasks.rados.rados.2.smithi175.stdout:3893: snap_create 2024-09-11T03:52:34.085 INFO:tasks.rados.rados.2.smithi175.stdout:3886: expect (ObjNum 1235 snap 402 seq_num 1235) 2024-09-11T03:52:34.109 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:34.116 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:34.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:34.117 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:34.218 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:34.241 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:34.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:34.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:34.331 INFO:tasks.rados.rados.2.smithi175.stdout:3889: finishing rollback tid 1 to smithi17557870-36 2024-09-11T03:52:34.331 INFO:tasks.rados.rados.2.smithi175.stdout:3889: finishing rollback tid 2 to smithi17557870-36 2024-09-11T03:52:34.331 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 1391 (ObjNum 1086 snap 353 seq_num 1086) dirty exists 2024-09-11T03:52:34.331 INFO:tasks.rados.rados.2.smithi175.stdout:3888: finishing copy_from racing read to smithi17557870-22 2024-09-11T03:52:34.331 INFO:tasks.rados.rados.2.smithi175.stdout:3888: finishing copy_from to smithi17557870-22 2024-09-11T03:52:34.331 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 22 v 1355 (ObjNum 1300 snap 426 seq_num 1300) dirty exists 2024-09-11T03:52:34.331 INFO:tasks.rados.rados.2.smithi175.stdout:3892: finishing copy_from racing read to smithi17557870-31 2024-09-11T03:52:34.331 INFO:tasks.rados.rados.2.smithi175.stdout:3892: finishing copy_from to smithi17557870-31 2024-09-11T03:52:34.332 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 31 v 1389 (ObjNum 1312 snap 429 seq_num 1312) dirty exists 2024-09-11T03:52:34.343 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:34.364 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:34.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:34.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:34.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:34 smithi084 ceph-mon[27254]: 2.54 scrub starts 2024-09-11T03:52:34.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:34 smithi084 ceph-mon[27254]: pgmap v1557: 127 pgs: 2 active+clean+snaptrim, 1 activating, 3 active+clean+snaptrim_wait, 2 peering, 4 unknown, 5 active+clean+scrubbing, 110 active+clean; 723 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 2.5 MiB/s rd, 1.1 MiB/s wr, 4 op/s 2024-09-11T03:52:34.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:34 smithi084 ceph-mon[27254]: 2.4e scrub starts 2024-09-11T03:52:34.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:34 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:52:34.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:34 smithi084 ceph-mon[27254]: 2.1a scrub starts 2024-09-11T03:52:34.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:34 smithi084 ceph-mon[27254]: 2.51 scrub starts 2024-09-11T03:52:34.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:34 smithi084 ceph-mon[27254]: 2.50 scrub starts 2024-09-11T03:52:34.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:34 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:52:34.372 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:34 smithi084 ceph-mon[27254]: osdmap e936: 12 total, 12 up, 11 in 2024-09-11T03:52:34.466 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:34.488 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:34.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:34.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:34.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:34.615 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:34.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:34.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:34.620 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:34 smithi121 ceph-mon[30729]: 2.54 scrub starts 2024-09-11T03:52:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:34 smithi121 ceph-mon[30729]: pgmap v1557: 127 pgs: 2 active+clean+snaptrim, 1 activating, 3 active+clean+snaptrim_wait, 2 peering, 4 unknown, 5 active+clean+scrubbing, 110 active+clean; 723 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 2.5 MiB/s rd, 1.1 MiB/s wr, 4 op/s 2024-09-11T03:52:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:34 smithi121 ceph-mon[30729]: 2.4e scrub starts 2024-09-11T03:52:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:34 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:52:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:34 smithi121 ceph-mon[30729]: 2.1a scrub starts 2024-09-11T03:52:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:34 smithi121 ceph-mon[30729]: 2.51 scrub starts 2024-09-11T03:52:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:34 smithi121 ceph-mon[30729]: 2.50 scrub starts 2024-09-11T03:52:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:34 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:52:34.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:34 smithi121 ceph-mon[30729]: osdmap e936: 12 total, 12 up, 11 in 2024-09-11T03:52:34.717 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:34.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:34 smithi175 ceph-mon[30894]: 2.54 scrub starts 2024-09-11T03:52:34.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:34 smithi175 ceph-mon[30894]: pgmap v1557: 127 pgs: 2 active+clean+snaptrim, 1 activating, 3 active+clean+snaptrim_wait, 2 peering, 4 unknown, 5 active+clean+scrubbing, 110 active+clean; 723 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 2.5 MiB/s rd, 1.1 MiB/s wr, 4 op/s 2024-09-11T03:52:34.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:34 smithi175 ceph-mon[30894]: 2.4e scrub starts 2024-09-11T03:52:34.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:34 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:52:34.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:34 smithi175 ceph-mon[30894]: 2.1a scrub starts 2024-09-11T03:52:34.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:34 smithi175 ceph-mon[30894]: 2.51 scrub starts 2024-09-11T03:52:34.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:34 smithi175 ceph-mon[30894]: 2.50 scrub starts 2024-09-11T03:52:34.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:34 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:52:34.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:34 smithi175 ceph-mon[30894]: osdmap e936: 12 total, 12 up, 11 in 2024-09-11T03:52:34.738 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:34.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:34.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:34.840 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:34.865 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:34.865 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:34.966 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:34.988 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:34.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:34.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:35.079 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:52:35.087 INFO:tasks.rados.rados.2.smithi175.stdout:3886: done (4 left) 2024-09-11T03:52:35.087 INFO:tasks.rados.rados.2.smithi175.stdout:3888: done (3 left) 2024-09-11T03:52:35.088 INFO:tasks.rados.rados.2.smithi175.stdout:3889: done (2 left) 2024-09-11T03:52:35.088 INFO:tasks.rados.rados.2.smithi175.stdout:3892: done (1 left) 2024-09-11T03:52:35.088 INFO:tasks.rados.rados.2.smithi175.stdout:3893: done (0 left) 2024-09-11T03:52:35.088 INFO:tasks.rados.rados.2.smithi175.stdout:3894: read oid 49 snap -1 2024-09-11T03:52:35.088 INFO:tasks.rados.rados.2.smithi175.stdout:3894: expect (ObjNum 1311 snap 429 seq_num 1311) 2024-09-11T03:52:35.088 INFO:tasks.rados.rados.2.smithi175.stdout:3895: snap_create 2024-09-11T03:52:35.089 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:35.091 INFO:tasks.rados.rados.2.smithi175.stdout:3894: expect (ObjNum 1311 snap 429 seq_num 1311) 2024-09-11T03:52:35.116 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:35.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:35.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:35.119 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:52:35.119 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:52:35.119 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:52:35.120 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:52:35.219 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:35.241 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:35.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:35.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:35.343 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:35.365 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:35.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:35.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:35 smithi084 ceph-mon[27254]: 2.4f scrub starts 2024-09-11T03:52:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:35 smithi084 ceph-mon[27254]: 2.52 scrub starts 2024-09-11T03:52:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:35 smithi084 ceph-mon[27254]: 2.55 scrub starts 2024-09-11T03:52:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:35 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:52:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:35 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:52:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:35 smithi084 ceph-mon[27254]: 2.51 scrub starts 2024-09-11T03:52:35.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:35 smithi084 ceph-mon[27254]: 2.50 scrub starts 2024-09-11T03:52:35.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:35 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3851481108' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:52:35.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:35 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:52:35.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:35 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:35.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:35 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3851481108' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:52:35.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:35 smithi084 ceph-mon[27254]: osdmap e937: 12 total, 12 up, 11 in 2024-09-11T03:52:35.467 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:35.490 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:35.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:35.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:35.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:35.613 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:35.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:35.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:35.662 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:52:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:35 smithi121 ceph-mon[30729]: 2.4f scrub starts 2024-09-11T03:52:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:35 smithi121 ceph-mon[30729]: 2.52 scrub starts 2024-09-11T03:52:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:35 smithi121 ceph-mon[30729]: 2.55 scrub starts 2024-09-11T03:52:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:35 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:52:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:35 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:52:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:35 smithi121 ceph-mon[30729]: 2.51 scrub starts 2024-09-11T03:52:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:35 smithi121 ceph-mon[30729]: 2.50 scrub starts 2024-09-11T03:52:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:35 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3851481108' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:52:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:35 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:52:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:35 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:35 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3851481108' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:52:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:35 smithi121 ceph-mon[30729]: osdmap e937: 12 total, 12 up, 11 in 2024-09-11T03:52:35.715 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0 2024-09-11T03:52:35.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:35 smithi175 ceph-mon[30894]: 2.4f scrub starts 2024-09-11T03:52:35.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:35 smithi175 ceph-mon[30894]: 2.52 scrub starts 2024-09-11T03:52:35.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:35 smithi175 ceph-mon[30894]: 2.55 scrub starts 2024-09-11T03:52:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:35 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:52:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:35 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:52:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:35 smithi175 ceph-mon[30894]: 2.51 scrub starts 2024-09-11T03:52:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:35 smithi175 ceph-mon[30894]: 2.50 scrub starts 2024-09-11T03:52:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:35 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3851481108' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:52:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:35 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:52:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:35 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:35 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3851481108' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:52:35.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:35 smithi175 ceph-mon[30894]: osdmap e937: 12 total, 12 up, 11 in 2024-09-11T03:52:35.735 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:35.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:35.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.0' 2024-09-11T03:52:35.838 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:35.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:35.861 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:35.861 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:35.892 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 8, 9, 11, 10, 6] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:52:35.892 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T03:52:35.893 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-11T03:52:35.893 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-11T03:52:35.893 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.0 2024-09-11T03:52:35.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:35.983 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:35.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:36.083 INFO:tasks.rados.rados.2.smithi175.stdout:3894: done (1 left) 2024-09-11T03:52:36.083 INFO:tasks.rados.rados.2.smithi175.stdout:3895: done (0 left) 2024-09-11T03:52:36.083 INFO:tasks.rados.rados.2.smithi175.stdout:3896: write oid 25 current snap is 432 2024-09-11T03:52:36.083 INFO:tasks.rados.rados.2.smithi175.stdout:3896: seq_num 1319 ranges {568433=582500,1684808=483625} 2024-09-11T03:52:36.085 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:36.090 INFO:tasks.rados.rados.2.smithi175.stdout:3896: writing smithi17557870-25 from 568433 to 1150933 tid 1 2024-09-11T03:52:36.096 INFO:tasks.rados.rados.2.smithi175.stdout:3896: writing smithi17557870-25 from 1684808 to 2168433 tid 2 2024-09-11T03:52:36.096 INFO:tasks.rados.rados.2.smithi175.stdout:3897: snap_remove snap 401 2024-09-11T03:52:36.109 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:36.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:36.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:36.144 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:52:35 smithi084 systemd[1]: Stopping Ceph osd.0 for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:52:36.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:36.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:36.235 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:36.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:36.263 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-11T03:52:36.337 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:36.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:36.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:36.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:36.482 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:36.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:36.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:36.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:36.605 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:36.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:36.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: 2.53 scrub starts 2024-09-11T03:52:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: pgmap v1560: 127 pgs: 2 active+clean+snaptrim, 1 activating, 3 active+clean+snaptrim_wait, 2 peering, 4 unknown, 5 active+clean+scrubbing, 110 active+clean; 707 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-09-11T03:52:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: 2.52 scrub starts 2024-09-11T03:52:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: 2.55 scrub starts 2024-09-11T03:52:36.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: 2.e scrub starts 2024-09-11T03:52:36.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: 2.55 scrub ok 2024-09-11T03:52:36.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: 2.e scrub ok 2024-09-11T03:52:36.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: 2.14 scrub starts 2024-09-11T03:52:36.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: 2.14 scrub ok 2024-09-11T03:52:36.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: 2.8 scrub starts 2024-09-11T03:52:36.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: 2.8 scrub ok 2024-09-11T03:52:36.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: 2.50 scrub starts 2024-09-11T03:52:36.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: 2.50 scrub ok 2024-09-11T03:52:36.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: 2.51 scrub starts 2024-09-11T03:52:36.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: 2.51 scrub ok 2024-09-11T03:52:36.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: 2.c scrub starts 2024-09-11T03:52:36.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: 2.c scrub ok 2024-09-11T03:52:36.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-mon[27254]: osdmap e938: 12 total, 12 up, 11 in 2024-09-11T03:52:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: 2.53 scrub starts 2024-09-11T03:52:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: pgmap v1560: 127 pgs: 2 active+clean+snaptrim, 1 activating, 3 active+clean+snaptrim_wait, 2 peering, 4 unknown, 5 active+clean+scrubbing, 110 active+clean; 707 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-09-11T03:52:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: 2.52 scrub starts 2024-09-11T03:52:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: 2.55 scrub starts 2024-09-11T03:52:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: 2.e scrub starts 2024-09-11T03:52:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: 2.55 scrub ok 2024-09-11T03:52:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: 2.e scrub ok 2024-09-11T03:52:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: 2.14 scrub starts 2024-09-11T03:52:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: 2.14 scrub ok 2024-09-11T03:52:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: 2.8 scrub starts 2024-09-11T03:52:36.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: 2.8 scrub ok 2024-09-11T03:52:36.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: 2.50 scrub starts 2024-09-11T03:52:36.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: 2.50 scrub ok 2024-09-11T03:52:36.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: 2.51 scrub starts 2024-09-11T03:52:36.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: 2.51 scrub ok 2024-09-11T03:52:36.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: 2.c scrub starts 2024-09-11T03:52:36.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: 2.c scrub ok 2024-09-11T03:52:36.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:36 smithi121 ceph-mon[30729]: osdmap e938: 12 total, 12 up, 11 in 2024-09-11T03:52:36.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:36.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: 2.53 scrub starts 2024-09-11T03:52:36.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: pgmap v1560: 127 pgs: 2 active+clean+snaptrim, 1 activating, 3 active+clean+snaptrim_wait, 2 peering, 4 unknown, 5 active+clean+scrubbing, 110 active+clean; 707 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail 2024-09-11T03:52:36.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: 2.52 scrub starts 2024-09-11T03:52:36.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: 2.55 scrub starts 2024-09-11T03:52:36.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: 2.e scrub starts 2024-09-11T03:52:36.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: 2.55 scrub ok 2024-09-11T03:52:36.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: 2.e scrub ok 2024-09-11T03:52:36.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: 2.14 scrub starts 2024-09-11T03:52:36.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: 2.14 scrub ok 2024-09-11T03:52:36.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: 2.8 scrub starts 2024-09-11T03:52:36.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: 2.8 scrub ok 2024-09-11T03:52:36.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: 2.50 scrub starts 2024-09-11T03:52:36.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: 2.50 scrub ok 2024-09-11T03:52:36.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: 2.51 scrub starts 2024-09-11T03:52:36.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: 2.51 scrub ok 2024-09-11T03:52:36.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: 2.c scrub starts 2024-09-11T03:52:36.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: 2.c scrub ok 2024-09-11T03:52:36.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:36 smithi175 ceph-mon[30894]: osdmap e938: 12 total, 12 up, 11 in 2024-09-11T03:52:36.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:36.730 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:36.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:36.831 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:36.853 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:36.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:36.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:36.955 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:36.979 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:36.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:36.981 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:37.007 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:37.007 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:37.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:37.086 INFO:tasks.rados.rados.2.smithi175.stdout:3897: done (1 left) 2024-09-11T03:52:37.086 INFO:tasks.rados.rados.2.smithi175.stdout:3898: delete oid 48 current snap is 432 2024-09-11T03:52:37.086 INFO:tasks.rados.rados.2.smithi175.stdout:3896: finishing write tid 1 to smithi17557870-25 2024-09-11T03:52:37.086 INFO:tasks.rados.rados.2.smithi175.stdout:3896: finishing write tid 2 to smithi17557870-25 2024-09-11T03:52:37.086 INFO:tasks.rados.rados.2.smithi175.stdout:3896: finishing write tid 3 to smithi17557870-25 2024-09-11T03:52:37.086 INFO:tasks.rados.rados.2.smithi175.stdout:3896: finishing write tid 4 to smithi17557870-25 2024-09-11T03:52:37.086 INFO:tasks.rados.rados.2.smithi175.stdout:3896: oid 25 updating version 0 to 1015 2024-09-11T03:52:37.086 INFO:tasks.rados.rados.2.smithi175.stdout:3896: oid 25 updating version 1015 to 1016 2024-09-11T03:52:37.086 INFO:tasks.rados.rados.2.smithi175.stdout:3896: oid 25 updating version 1016 to 1017 2024-09-11T03:52:37.086 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 25 v 1017 (ObjNum 1319 snap 432 seq_num 1319) dirty exists 2024-09-11T03:52:37.086 INFO:tasks.rados.rados.2.smithi175.stdout:3896: left oid 25 (ObjNum 1319 snap 432 seq_num 1319) 2024-09-11T03:52:37.113 INFO:tasks.rados.rados.2.smithi175.stdout:3896: done (1 left) 2024-09-11T03:52:37.113 INFO:tasks.rados.rados.2.smithi175.stdout:3898: done (0 left) 2024-09-11T03:52:37.113 INFO:tasks.rados.rados.2.smithi175.stdout:3899: read oid 7 snap 411 2024-09-11T03:52:37.113 INFO:tasks.rados.rados.2.smithi175.stdout:3899: expect (ObjNum 1220 snap 399 seq_num 1220) 2024-09-11T03:52:37.113 INFO:tasks.rados.rados.2.smithi175.stdout:3900: read oid 5 snap 429 2024-09-11T03:52:37.114 INFO:tasks.rados.rados.2.smithi175.stdout:3900: expect (ObjNum 1297 snap 426 seq_num 1297) 2024-09-11T03:52:37.114 INFO:tasks.rados.rados.2.smithi175.stdout:3901: write oid 21 current snap is 432 2024-09-11T03:52:37.114 INFO:tasks.rados.rados.2.smithi175.stdout:3901: seq_num 1321 ranges {602047=732998,1801772=400275} 2024-09-11T03:52:37.116 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:37.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:37.119 INFO:tasks.rados.rados.2.smithi175.stdout:3901: writing smithi17557870-21 from 602047 to 1335045 tid 1 2024-09-11T03:52:37.124 INFO:tasks.rados.rados.2.smithi175.stdout:3901: writing smithi17557870-21 from 1801772 to 2202047 tid 2 2024-09-11T03:52:37.124 INFO:tasks.rados.rados.2.smithi175.stdout:3902: copy_from oid 49 from oid 35 current snap is 432 2024-09-11T03:52:37.124 INFO:tasks.rados.rados.2.smithi175.stdout:3900: expect (ObjNum 1297 snap 426 seq_num 1297) 2024-09-11T03:52:37.128 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0[41076]: 2024-09-11T03:52:36.984+0000 7f3e4c016640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:52:37.128 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0[41076]: 2024-09-11T03:52:36.984+0000 7f3e4c016640 -1 osd.0 938 *** Got signal Terminated *** 2024-09-11T03:52:37.128 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:52:36 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0[41076]: 2024-09-11T03:52:36.984+0000 7f3e4c016640 -1 osd.0 938 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T03:52:37.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:37.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:37.241 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:37.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:37.329 INFO:tasks.rados.rados.2.smithi175.stdout:3899: expect (ObjNum 1220 snap 399 seq_num 1220) 2024-09-11T03:52:37.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:37.363 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:37.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:37.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:37.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:37 smithi084 ceph-mon[27254]: 2.4f scrub starts 2024-09-11T03:52:37.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:37 smithi084 ceph-mon[27254]: 2.52 scrub starts 2024-09-11T03:52:37.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:37 smithi084 ceph-mon[27254]: 2.52 scrub ok 2024-09-11T03:52:37.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:37 smithi084 ceph-mon[27254]: 2.d scrub starts 2024-09-11T03:52:37.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:37 smithi084 ceph-mon[27254]: 2.d scrub ok 2024-09-11T03:52:37.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:37 smithi084 ceph-mon[27254]: 2.15 scrub starts 2024-09-11T03:52:37.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:37 smithi084 ceph-mon[27254]: 2.15 scrub ok 2024-09-11T03:52:37.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:37 smithi084 ceph-mon[27254]: 2.11 scrub starts 2024-09-11T03:52:37.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:37 smithi084 ceph-mon[27254]: 2.11 scrub ok 2024-09-11T03:52:37.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:37 smithi084 ceph-mon[27254]: 2.10 scrub starts 2024-09-11T03:52:37.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:37 smithi084 ceph-mon[27254]: 2.10 scrub ok 2024-09-11T03:52:37.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:37 smithi084 ceph-mon[27254]: 2.4d scrub starts 2024-09-11T03:52:37.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:37 smithi084 ceph-mon[27254]: 2.4d scrub ok 2024-09-11T03:52:37.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:37 smithi084 ceph-mon[27254]: osd.0 marked itself down and dead 2024-09-11T03:52:37.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:37 smithi084 ceph-mon[27254]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-11T03:52:37.396 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:37 smithi084 ceph-mon[27254]: osdmap e939: 12 total, 11 up, 11 in 2024-09-11T03:52:37.442 INFO:tasks.rados.rados.2.smithi175.stdout:3899: done (3 left) 2024-09-11T03:52:37.443 INFO:tasks.rados.rados.2.smithi175.stdout:3900: done (2 left) 2024-09-11T03:52:37.443 INFO:tasks.rados.rados.2.smithi175.stdout:3903: read oid 10 snap -1 2024-09-11T03:52:37.443 INFO:tasks.rados.rados.2.smithi175.stdout:3903: expect (ObjNum 1281 snap 422 seq_num 1281) 2024-09-11T03:52:37.443 INFO:tasks.rados.rados.2.smithi175.stdout:3904: rollback oid 8 current snap is 432 2024-09-11T03:52:37.443 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 8 to 431 2024-09-11T03:52:37.443 INFO:tasks.rados.rados.2.smithi175.stdout:3905: write oid 27 current snap is 432 2024-09-11T03:52:37.443 INFO:tasks.rados.rados.2.smithi175.stdout:3905: seq_num 1322 ranges {618854=408247,1743901=425794} 2024-09-11T03:52:37.448 INFO:tasks.rados.rados.2.smithi175.stdout:3905: writing smithi17557870-27 from 618854 to 1027101 tid 1 2024-09-11T03:52:37.454 INFO:tasks.rados.rados.2.smithi175.stdout:3905: writing smithi17557870-27 from 1743901 to 2169695 tid 2 2024-09-11T03:52:37.454 INFO:tasks.rados.rados.2.smithi175.stdout:3906: snap_create 2024-09-11T03:52:37.454 INFO:tasks.rados.rados.2.smithi175.stdout:3905: finishing write tid 1 to smithi17557870-27 2024-09-11T03:52:37.457 INFO:tasks.rados.rados.2.smithi175.stdout:3903: expect (ObjNum 1281 snap 422 seq_num 1281) 2024-09-11T03:52:37.465 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:37.488 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:37.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:37.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:37.555 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:52:37.555 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:52:37.555 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-11T03:52:37.555 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-11T03:52:37.555 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:52:37.590 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:37.600 INFO:tasks.rados.rados.2.smithi175.stdout:3905: finishing write tid 2 to smithi17557870-27 2024-09-11T03:52:37.600 INFO:tasks.rados.rados.2.smithi175.stdout:3905: finishing write tid 3 to smithi17557870-27 2024-09-11T03:52:37.600 INFO:tasks.rados.rados.2.smithi175.stdout:3905: finishing write tid 4 to smithi17557870-27 2024-09-11T03:52:37.600 INFO:tasks.rados.rados.2.smithi175.stdout:3905: oid 27 updating version 0 to 1391 2024-09-11T03:52:37.601 INFO:tasks.rados.rados.2.smithi175.stdout:3905: oid 27 updating version 1391 to 1392 2024-09-11T03:52:37.601 INFO:tasks.rados.rados.2.smithi175.stdout:3905: oid 27 version 1392 is already newer than 1390 2024-09-11T03:52:37.601 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 27 v 1392 (ObjNum 1322 snap 432 seq_num 1322) dirty exists 2024-09-11T03:52:37.601 INFO:tasks.rados.rados.2.smithi175.stdout:3905: left oid 27 (ObjNum 1322 snap 432 seq_num 1322) 2024-09-11T03:52:37.611 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:37.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:37.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:37 smithi121 ceph-mon[30729]: 2.4f scrub starts 2024-09-11T03:52:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:37 smithi121 ceph-mon[30729]: 2.52 scrub starts 2024-09-11T03:52:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:37 smithi121 ceph-mon[30729]: 2.52 scrub ok 2024-09-11T03:52:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:37 smithi121 ceph-mon[30729]: 2.d scrub starts 2024-09-11T03:52:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:37 smithi121 ceph-mon[30729]: 2.d scrub ok 2024-09-11T03:52:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:37 smithi121 ceph-mon[30729]: 2.15 scrub starts 2024-09-11T03:52:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:37 smithi121 ceph-mon[30729]: 2.15 scrub ok 2024-09-11T03:52:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:37 smithi121 ceph-mon[30729]: 2.11 scrub starts 2024-09-11T03:52:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:37 smithi121 ceph-mon[30729]: 2.11 scrub ok 2024-09-11T03:52:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:37 smithi121 ceph-mon[30729]: 2.10 scrub starts 2024-09-11T03:52:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:37 smithi121 ceph-mon[30729]: 2.10 scrub ok 2024-09-11T03:52:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:37 smithi121 ceph-mon[30729]: 2.4d scrub starts 2024-09-11T03:52:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:37 smithi121 ceph-mon[30729]: 2.4d scrub ok 2024-09-11T03:52:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:37 smithi121 ceph-mon[30729]: osd.0 marked itself down and dead 2024-09-11T03:52:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:37 smithi121 ceph-mon[30729]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-11T03:52:37.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:37 smithi121 ceph-mon[30729]: osdmap e939: 12 total, 11 up, 11 in 2024-09-11T03:52:37.713 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:37.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:37 smithi175 ceph-mon[30894]: 2.4f scrub starts 2024-09-11T03:52:37.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:37 smithi175 ceph-mon[30894]: 2.52 scrub starts 2024-09-11T03:52:37.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:37 smithi175 ceph-mon[30894]: 2.52 scrub ok 2024-09-11T03:52:37.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:37 smithi175 ceph-mon[30894]: 2.d scrub starts 2024-09-11T03:52:37.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:37 smithi175 ceph-mon[30894]: 2.d scrub ok 2024-09-11T03:52:37.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:37 smithi175 ceph-mon[30894]: 2.15 scrub starts 2024-09-11T03:52:37.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:37 smithi175 ceph-mon[30894]: 2.15 scrub ok 2024-09-11T03:52:37.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:37 smithi175 ceph-mon[30894]: 2.11 scrub starts 2024-09-11T03:52:37.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:37 smithi175 ceph-mon[30894]: 2.11 scrub ok 2024-09-11T03:52:37.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:37 smithi175 ceph-mon[30894]: 2.10 scrub starts 2024-09-11T03:52:37.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:37 smithi175 ceph-mon[30894]: 2.10 scrub ok 2024-09-11T03:52:37.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:37 smithi175 ceph-mon[30894]: 2.4d scrub starts 2024-09-11T03:52:37.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:37 smithi175 ceph-mon[30894]: 2.4d scrub ok 2024-09-11T03:52:37.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:37 smithi175 ceph-mon[30894]: osd.0 marked itself down and dead 2024-09-11T03:52:37.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:37 smithi175 ceph-mon[30894]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-11T03:52:37.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:37 smithi175 ceph-mon[30894]: osdmap e939: 12 total, 11 up, 11 in 2024-09-11T03:52:37.734 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:37.735 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:37.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:37.858 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:37.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:37.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:37.961 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:37.982 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:37.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:38.057 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:52:37 smithi084 podman[180010]: 2024-09-11 03:52:37.749728753 +0000 UTC m=+0.900723360 container died 597a3d5162eb58f7c2cfa7420880d55c91e200b1b2dc2fd77937d0e38386fee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-11T03:52:38.084 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:38.091 INFO:tasks.rados.rados.2.smithi175.stdout:3903: done (5 left) 2024-09-11T03:52:38.091 INFO:tasks.rados.rados.2.smithi175.stdout:3905: done (4 left) 2024-09-11T03:52:38.091 INFO:tasks.rados.rados.2.smithi175.stdout:3906: done (3 left) 2024-09-11T03:52:38.091 INFO:tasks.rados.rados.2.smithi175.stdout:3907: read oid 30 snap -1 2024-09-11T03:52:38.091 INFO:tasks.rados.rados.2.smithi175.stdout:3907: expect (ObjNum 1318 snap 430 seq_num 1318) 2024-09-11T03:52:38.091 INFO:tasks.rados.rados.2.smithi175.stdout:3908: delete oid 37 current snap is 433 2024-09-11T03:52:38.102 INFO:tasks.rados.rados.2.smithi175.stdout:3901: finishing write tid 1 to smithi17557870-21 2024-09-11T03:52:38.104 INFO:tasks.rados.rados.2.smithi175.stdout:3901: finishing write tid 2 to smithi17557870-21 2024-09-11T03:52:38.104 INFO:tasks.rados.rados.2.smithi175.stdout:3901: finishing write tid 3 to smithi17557870-21 2024-09-11T03:52:38.105 INFO:tasks.rados.rados.2.smithi175.stdout:3901: finishing write tid 4 to smithi17557870-21 2024-09-11T03:52:38.105 INFO:tasks.rados.rados.2.smithi175.stdout:3901: oid 21 updating version 0 to 1336 2024-09-11T03:52:38.105 INFO:tasks.rados.rados.2.smithi175.stdout:3901: oid 21 version 1336 is already newer than 1335 2024-09-11T03:52:38.105 INFO:tasks.rados.rados.2.smithi175.stdout:3901: oid 21 version 1336 is already newer than 1334 2024-09-11T03:52:38.105 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 1336 (ObjNum 1321 snap 432 seq_num 1321) dirty exists 2024-09-11T03:52:38.105 INFO:tasks.rados.rados.2.smithi175.stdout:3901: left oid 21 (ObjNum 1321 snap 432 seq_num 1321) 2024-09-11T03:52:38.105 INFO:tasks.rados.rados.2.smithi175.stdout:3901: done (4 left) 2024-09-11T03:52:38.105 INFO:tasks.rados.rados.2.smithi175.stdout:3908: done (3 left) 2024-09-11T03:52:38.105 INFO:tasks.rados.rados.2.smithi175.stdout:3909: snap_remove snap 374 2024-09-11T03:52:38.117 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:38.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:38.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:38.219 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:38.242 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:38.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:38.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:38.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:38.370 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:38.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:38.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:38.373 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:52:38 smithi084 podman[180010]: 2024-09-11 03:52:38.26489649 +0000 UTC m=+1.415891090 container cleanup 597a3d5162eb58f7c2cfa7420880d55c91e200b1b2dc2fd77937d0e38386fee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T03:52:38.373 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:52:38 smithi084 bash[180010]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0 2024-09-11T03:52:38.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.53 scrub starts 2024-09-11T03:52:38.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.53 scrub ok 2024-09-11T03:52:38.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.1f scrub starts 2024-09-11T03:52:38.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.1f scrub ok 2024-09-11T03:52:38.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: pgmap v1563: 127 pgs: 2 active+undersized+remapped, 2 active+clean+snaptrim, 1 activating, 3 active+clean+snaptrim_wait, 4 peering, 1 unknown, 2 active+clean+scrubbing, 112 active+clean; 730 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 8.1 MiB/s, 25 objects/s recovering 2024-09-11T03:52:38.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.1b scrub starts 2024-09-11T03:52:38.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.1b scrub ok 2024-09-11T03:52:38.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.18 scrub starts 2024-09-11T03:52:38.373 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.18 scrub ok 2024-09-11T03:52:38.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.4e scrub starts 2024-09-11T03:52:38.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.4e scrub ok 2024-09-11T03:52:38.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.1a scrub starts 2024-09-11T03:52:38.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.1a scrub ok 2024-09-11T03:52:38.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.58 scrub starts 2024-09-11T03:52:38.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.58 scrub ok 2024-09-11T03:52:38.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.1d scrub starts 2024-09-11T03:52:38.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.1d scrub ok 2024-09-11T03:52:38.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.f scrub starts 2024-09-11T03:52:38.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.f scrub ok 2024-09-11T03:52:38.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: osdmap e940: 12 total, 11 up, 11 in 2024-09-11T03:52:38.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.16 scrub starts 2024-09-11T03:52:38.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:38 smithi084 ceph-mon[27254]: 2.16 scrub ok 2024-09-11T03:52:38.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:38.493 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:38.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:38.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:38.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:38.617 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:38.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:38.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.53 scrub starts 2024-09-11T03:52:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.53 scrub ok 2024-09-11T03:52:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.1f scrub starts 2024-09-11T03:52:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.1f scrub ok 2024-09-11T03:52:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: pgmap v1563: 127 pgs: 2 active+undersized+remapped, 2 active+clean+snaptrim, 1 activating, 3 active+clean+snaptrim_wait, 4 peering, 1 unknown, 2 active+clean+scrubbing, 112 active+clean; 730 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 8.1 MiB/s, 25 objects/s recovering 2024-09-11T03:52:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.1b scrub starts 2024-09-11T03:52:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.1b scrub ok 2024-09-11T03:52:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.18 scrub starts 2024-09-11T03:52:38.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.18 scrub ok 2024-09-11T03:52:38.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.4e scrub starts 2024-09-11T03:52:38.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.4e scrub ok 2024-09-11T03:52:38.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.1a scrub starts 2024-09-11T03:52:38.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.1a scrub ok 2024-09-11T03:52:38.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.58 scrub starts 2024-09-11T03:52:38.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.58 scrub ok 2024-09-11T03:52:38.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.1d scrub starts 2024-09-11T03:52:38.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.1d scrub ok 2024-09-11T03:52:38.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.f scrub starts 2024-09-11T03:52:38.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.f scrub ok 2024-09-11T03:52:38.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: osdmap e940: 12 total, 11 up, 11 in 2024-09-11T03:52:38.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.16 scrub starts 2024-09-11T03:52:38.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:38 smithi121 ceph-mon[30729]: 2.16 scrub ok 2024-09-11T03:52:38.719 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:38.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.53 scrub starts 2024-09-11T03:52:38.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.53 scrub ok 2024-09-11T03:52:38.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.1f scrub starts 2024-09-11T03:52:38.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.1f scrub ok 2024-09-11T03:52:38.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: pgmap v1563: 127 pgs: 2 active+undersized+remapped, 2 active+clean+snaptrim, 1 activating, 3 active+clean+snaptrim_wait, 4 peering, 1 unknown, 2 active+clean+scrubbing, 112 active+clean; 730 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 8.1 MiB/s, 25 objects/s recovering 2024-09-11T03:52:38.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.1b scrub starts 2024-09-11T03:52:38.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.1b scrub ok 2024-09-11T03:52:38.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.18 scrub starts 2024-09-11T03:52:38.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.18 scrub ok 2024-09-11T03:52:38.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.4e scrub starts 2024-09-11T03:52:38.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.4e scrub ok 2024-09-11T03:52:38.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.1a scrub starts 2024-09-11T03:52:38.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.1a scrub ok 2024-09-11T03:52:38.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.58 scrub starts 2024-09-11T03:52:38.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.58 scrub ok 2024-09-11T03:52:38.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.1d scrub starts 2024-09-11T03:52:38.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.1d scrub ok 2024-09-11T03:52:38.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.f scrub starts 2024-09-11T03:52:38.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.f scrub ok 2024-09-11T03:52:38.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: osdmap e940: 12 total, 11 up, 11 in 2024-09-11T03:52:38.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.16 scrub starts 2024-09-11T03:52:38.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:38 smithi175 ceph-mon[30894]: 2.16 scrub ok 2024-09-11T03:52:38.742 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:38.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:38.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:38.773 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:52:38.844 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:38.867 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:38.867 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:38.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:38.984 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:52:38 smithi084 podman[180120]: 2024-09-11 03:52:38.701878829 +0000 UTC m=+1.059715445 container remove 597a3d5162eb58f7c2cfa7420880d55c91e200b1b2dc2fd77937d0e38386fee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T03:52:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:38.992 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:38.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:39.089 INFO:tasks.rados.rados.2.smithi175.stdout:3909: done (3 left) 2024-09-11T03:52:39.089 INFO:tasks.rados.rados.2.smithi175.stdout:3910: write (excl) oid 11 current snap is 433 2024-09-11T03:52:39.089 INFO:tasks.rados.rados.2.smithi175.stdout:3910: seq_num 1324 ranges {652468=675098,1977218=275250} 2024-09-11T03:52:39.094 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:39.096 INFO:tasks.rados.rados.2.smithi175.stdout:3910: writing smithi17557870-11 from 652468 to 1327566 tid 1 2024-09-11T03:52:39.102 INFO:tasks.rados.rados.2.smithi175.stdout:3910: writing smithi17557870-11 from 1977218 to 2252468 tid 2 2024-09-11T03:52:39.103 INFO:tasks.rados.rados.2.smithi175.stdout:3904: finishing rollback tid 0 to smithi17557870-8 2024-09-11T03:52:39.103 INFO:tasks.rados.rados.2.smithi175.stdout:3904: finishing rollback tid 1 to smithi17557870-8 2024-09-11T03:52:39.103 INFO:tasks.rados.rados.2.smithi175.stdout:3904: finishing rollback tid 2 to smithi17557870-8 2024-09-11T03:52:39.103 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 1279 (ObjNum 1220 snap 399 seq_num 1220) dirty exists 2024-09-11T03:52:39.104 INFO:tasks.rados.rados.2.smithi175.stdout:3907: expect (ObjNum 1318 snap 430 seq_num 1318) 2024-09-11T03:52:39.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:39.121 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:39.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:39.133 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-11T03:52:39.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:39.245 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:39.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:39.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:39.250 INFO:tasks.rados.rados.2.smithi175.stdout:3902: finishing copy_from racing read to smithi17557870-49 2024-09-11T03:52:39.250 INFO:tasks.rados.rados.2.smithi175.stdout:3902: finishing copy_from to smithi17557870-49 2024-09-11T03:52:39.250 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 49 v 1090 (ObjNum 1283 snap 423 seq_num 1283) dirty exists 2024-09-11T03:52:39.250 INFO:tasks.rados.rados.2.smithi175.stdout:3910: finishing write tid 1 to smithi17557870-11 2024-09-11T03:52:39.251 INFO:tasks.rados.rados.2.smithi175.stdout:3910: finishing write tid 2 to smithi17557870-11 2024-09-11T03:52:39.251 INFO:tasks.rados.rados.2.smithi175.stdout:3910: finishing write tid 3 to smithi17557870-11 2024-09-11T03:52:39.251 INFO:tasks.rados.rados.2.smithi175.stdout:3910: finishing write tid 4 to smithi17557870-11 2024-09-11T03:52:39.251 INFO:tasks.rados.rados.2.smithi175.stdout:3910: oid 11 updating version 0 to 1413 2024-09-11T03:52:39.251 INFO:tasks.rados.rados.2.smithi175.stdout:3910: oid 11 updating version 1413 to 1414 2024-09-11T03:52:39.251 INFO:tasks.rados.rados.2.smithi175.stdout:3910: oid 11 version 1414 is already newer than 1412 2024-09-11T03:52:39.251 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 1414 (ObjNum 1324 snap 433 seq_num 1324) dirty exists 2024-09-11T03:52:39.251 INFO:tasks.rados.rados.2.smithi175.stdout:3910: left oid 11 (ObjNum 1324 snap 433 seq_num 1324) 2024-09-11T03:52:39.251 INFO:tasks.rados.rados.2.smithi175.stdout:3902: done (3 left) 2024-09-11T03:52:39.251 INFO:tasks.rados.rados.2.smithi175.stdout:3904: done (2 left) 2024-09-11T03:52:39.251 INFO:tasks.rados.rados.2.smithi175.stdout:3907: done (1 left) 2024-09-11T03:52:39.251 INFO:tasks.rados.rados.2.smithi175.stdout:3910: done (0 left) 2024-09-11T03:52:39.251 INFO:tasks.rados.rados.2.smithi175.stdout:3911: read oid 50 snap -1 2024-09-11T03:52:39.252 INFO:tasks.rados.rados.2.smithi175.stdout:3911: expect (ObjNum 1190 snap 388 seq_num 1190) 2024-09-11T03:52:39.252 INFO:tasks.rados.rados.2.smithi175.stdout:3912: write oid 45 current snap is 433 2024-09-11T03:52:39.252 INFO:tasks.rados.rados.2.smithi175.stdout:3912: seq_num 1325 ranges {669275=750347,2035700=233575} 2024-09-11T03:52:39.257 INFO:tasks.rados.rados.2.smithi175.stdout:3912: writing smithi17557870-45 from 669275 to 1419622 tid 1 2024-09-11T03:52:39.262 INFO:tasks.rados.rados.2.smithi175.stdout:3912: writing smithi17557870-45 from 2035700 to 2269275 tid 2 2024-09-11T03:52:39.262 INFO:tasks.rados.rados.2.smithi175.stdout:3913: delete oid 47 current snap is 433 2024-09-11T03:52:39.262 INFO:tasks.rados.rados.2.smithi175.stdout:3911: expect (ObjNum 1190 snap 388 seq_num 1190) 2024-09-11T03:52:39.325 INFO:tasks.rados.rados.2.smithi175.stdout:3912: finishing write tid 1 to smithi17557870-45 2024-09-11T03:52:39.325 INFO:tasks.rados.rados.2.smithi175.stdout:3912: finishing write tid 2 to smithi17557870-45 2024-09-11T03:52:39.325 INFO:tasks.rados.rados.2.smithi175.stdout:3912: finishing write tid 3 to smithi17557870-45 2024-09-11T03:52:39.325 INFO:tasks.rados.rados.2.smithi175.stdout:3912: finishing write tid 4 to smithi17557870-45 2024-09-11T03:52:39.325 INFO:tasks.rados.rados.2.smithi175.stdout:3912: oid 45 updating version 0 to 1091 2024-09-11T03:52:39.325 INFO:tasks.rados.rados.2.smithi175.stdout:3912: oid 45 updating version 1091 to 1092 2024-09-11T03:52:39.325 INFO:tasks.rados.rados.2.smithi175.stdout:3912: oid 45 updating version 1092 to 1093 2024-09-11T03:52:39.325 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 45 v 1093 (ObjNum 1325 snap 433 seq_num 1325) dirty exists 2024-09-11T03:52:39.325 INFO:tasks.rados.rados.2.smithi175.stdout:3912: left oid 45 (ObjNum 1325 snap 433 seq_num 1325) 2024-09-11T03:52:39.325 INFO:tasks.rados.rados.2.smithi175.stdout:3911: done (2 left) 2024-09-11T03:52:39.325 INFO:tasks.rados.rados.2.smithi175.stdout:3912: done (1 left) 2024-09-11T03:52:39.325 INFO:tasks.rados.rados.2.smithi175.stdout:3913: done (0 left) 2024-09-11T03:52:39.326 INFO:tasks.rados.rados.2.smithi175.stdout:3914: write oid 13 current snap is 433 2024-09-11T03:52:39.326 INFO:tasks.rados.rados.2.smithi175.stdout:3914: seq_num 1327 ranges {702889=500845,1636311=576790} 2024-09-11T03:52:39.331 INFO:tasks.rados.rados.2.smithi175.stdout:3914: writing smithi17557870-13 from 702889 to 1203734 tid 1 2024-09-11T03:52:39.336 INFO:tasks.rados.rados.2.smithi175.stdout:3914: writing smithi17557870-13 from 1636311 to 2213101 tid 2 2024-09-11T03:52:39.337 INFO:tasks.rados.rados.2.smithi175.stdout:3915: write oid 39 current snap is 433 2024-09-11T03:52:39.337 INFO:tasks.rados.rados.2.smithi175.stdout:3914: finishing write tid 1 to smithi17557870-13 2024-09-11T03:52:39.337 INFO:tasks.rados.rados.2.smithi175.stdout:3915: seq_num 1328 ranges {719696=576094,2094793=224903} 2024-09-11T03:52:39.343 INFO:tasks.rados.rados.2.smithi175.stdout:3915: writing smithi17557870-39 from 719696 to 1295790 tid 1 2024-09-11T03:52:39.347 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:39.350 INFO:tasks.rados.rados.2.smithi175.stdout:3915: writing smithi17557870-39 from 2094793 to 2319696 tid 2 2024-09-11T03:52:39.350 INFO:tasks.rados.rados.2.smithi175.stdout:3916: snap_create 2024-09-11T03:52:39.350 INFO:tasks.rados.rados.2.smithi175.stdout:3914: finishing write tid 2 to smithi17557870-13 2024-09-11T03:52:39.351 INFO:tasks.rados.rados.2.smithi175.stdout:3914: finishing write tid 3 to smithi17557870-13 2024-09-11T03:52:39.351 INFO:tasks.rados.rados.2.smithi175.stdout:3914: finishing write tid 4 to smithi17557870-13 2024-09-11T03:52:39.351 INFO:tasks.rados.rados.2.smithi175.stdout:3914: oid 13 updating version 0 to 1653 2024-09-11T03:52:39.351 INFO:tasks.rados.rados.2.smithi175.stdout:3914: oid 13 updating version 1653 to 1654 2024-09-11T03:52:39.351 INFO:tasks.rados.rados.2.smithi175.stdout:3914: oid 13 updating version 1654 to 1655 2024-09-11T03:52:39.351 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 13 v 1655 (ObjNum 1327 snap 433 seq_num 1327) dirty exists 2024-09-11T03:52:39.351 INFO:tasks.rados.rados.2.smithi175.stdout:3914: left oid 13 (ObjNum 1327 snap 433 seq_num 1327) 2024-09-11T03:52:39.351 INFO:tasks.rados.rados.2.smithi175.stdout:3915: finishing write tid 1 to smithi17557870-39 2024-09-11T03:52:39.352 INFO:tasks.rados.rados.2.smithi175.stdout:3915: finishing write tid 2 to smithi17557870-39 2024-09-11T03:52:39.352 INFO:tasks.rados.rados.2.smithi175.stdout:3915: finishing write tid 3 to smithi17557870-39 2024-09-11T03:52:39.352 INFO:tasks.rados.rados.2.smithi175.stdout:3915: finishing write tid 4 to smithi17557870-39 2024-09-11T03:52:39.352 INFO:tasks.rados.rados.2.smithi175.stdout:3915: oid 39 updating version 0 to 1035 2024-09-11T03:52:39.352 INFO:tasks.rados.rados.2.smithi175.stdout:3915: oid 39 version 1035 is already newer than 1034 2024-09-11T03:52:39.352 INFO:tasks.rados.rados.2.smithi175.stdout:3915: oid 39 version 1035 is already newer than 1033 2024-09-11T03:52:39.352 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 1035 (ObjNum 1328 snap 433 seq_num 1328) dirty exists 2024-09-11T03:52:39.352 INFO:tasks.rados.rados.2.smithi175.stdout:3915: left oid 39 (ObjNum 1328 snap 433 seq_num 1328) 2024-09-11T03:52:39.369 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:39.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:39.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:39.422 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:52:39 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:52:39.108+0000 7f164f14f640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 178 after trimming was started 2024-09-11T03:52:39.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:39 smithi121 ceph-mon[30729]: 2.54 scrub starts 2024-09-11T03:52:39.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:39 smithi121 ceph-mon[30729]: 2.54 scrub ok 2024-09-11T03:52:39.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:39 smithi121 ceph-mon[30729]: 2.13 scrub starts 2024-09-11T03:52:39.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:39 smithi121 ceph-mon[30729]: 2.13 scrub ok 2024-09-11T03:52:39.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:39 smithi121 ceph-mon[30729]: 2.59 scrub starts 2024-09-11T03:52:39.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:39 smithi121 ceph-mon[30729]: 2.59 scrub ok 2024-09-11T03:52:39.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:39 smithi121 ceph-mon[30729]: 2.5e scrub starts 2024-09-11T03:52:39.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:39 smithi121 ceph-mon[30729]: 2.5e scrub ok 2024-09-11T03:52:39.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:39 smithi121 ceph-mon[30729]: 2.5f scrub starts 2024-09-11T03:52:39.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:39 smithi121 ceph-mon[30729]: 2.5f scrub ok 2024-09-11T03:52:39.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:39 smithi121 ceph-mon[30729]: 2.5b scrub starts 2024-09-11T03:52:39.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:39 smithi121 ceph-mon[30729]: 2.5b scrub ok 2024-09-11T03:52:39.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:39 smithi121 ceph-mon[30729]: osdmap e941: 12 total, 11 up, 11 in 2024-09-11T03:52:39.463 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:52:39.463 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:52:39.464 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-11T03:52:39.464 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:52:39.464 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:52:39.470 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:39.475 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:52:39.476 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:52:39.492 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:39.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:39.493 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:39.494 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:52:39.495 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:52:39.514 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:52:39.514 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:52:39.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:39 smithi084 ceph-mon[27254]: 2.54 scrub starts 2024-09-11T03:52:39.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:39 smithi084 ceph-mon[27254]: 2.54 scrub ok 2024-09-11T03:52:39.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:39 smithi084 ceph-mon[27254]: 2.13 scrub starts 2024-09-11T03:52:39.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:39 smithi084 ceph-mon[27254]: 2.13 scrub ok 2024-09-11T03:52:39.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:39 smithi084 ceph-mon[27254]: 2.59 scrub starts 2024-09-11T03:52:39.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:39 smithi084 ceph-mon[27254]: 2.59 scrub ok 2024-09-11T03:52:39.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:39 smithi084 ceph-mon[27254]: 2.5e scrub starts 2024-09-11T03:52:39.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:39 smithi084 ceph-mon[27254]: 2.5e scrub ok 2024-09-11T03:52:39.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:39 smithi084 ceph-mon[27254]: 2.5f scrub starts 2024-09-11T03:52:39.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:39 smithi084 ceph-mon[27254]: 2.5f scrub ok 2024-09-11T03:52:39.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:39 smithi084 ceph-mon[27254]: 2.5b scrub starts 2024-09-11T03:52:39.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:39 smithi084 ceph-mon[27254]: 2.5b scrub ok 2024-09-11T03:52:39.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:39 smithi084 ceph-mon[27254]: osdmap e941: 12 total, 11 up, 11 in 2024-09-11T03:52:39.537 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:39.538 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:52:39.538 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:52:39.560 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:39.569 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:52:39.569 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:52:39.587 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:52:39.588 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:52:39.593 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:39.605 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:52:39.605 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:52:39.614 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:39.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:39.615 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:39.622 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:52:39.622 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:52:39.638 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:52:39.639 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:52:39.716 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:39.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:39 smithi175 ceph-mon[30894]: 2.54 scrub starts 2024-09-11T03:52:39.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:39 smithi175 ceph-mon[30894]: 2.54 scrub ok 2024-09-11T03:52:39.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:39 smithi175 ceph-mon[30894]: 2.13 scrub starts 2024-09-11T03:52:39.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:39 smithi175 ceph-mon[30894]: 2.13 scrub ok 2024-09-11T03:52:39.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:39 smithi175 ceph-mon[30894]: 2.59 scrub starts 2024-09-11T03:52:39.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:39 smithi175 ceph-mon[30894]: 2.59 scrub ok 2024-09-11T03:52:39.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:39 smithi175 ceph-mon[30894]: 2.5e scrub starts 2024-09-11T03:52:39.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:39 smithi175 ceph-mon[30894]: 2.5e scrub ok 2024-09-11T03:52:39.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:39 smithi175 ceph-mon[30894]: 2.5f scrub starts 2024-09-11T03:52:39.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:39 smithi175 ceph-mon[30894]: 2.5f scrub ok 2024-09-11T03:52:39.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:39 smithi175 ceph-mon[30894]: 2.5b scrub starts 2024-09-11T03:52:39.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:39 smithi175 ceph-mon[30894]: 2.5b scrub ok 2024-09-11T03:52:39.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:39 smithi175 ceph-mon[30894]: osdmap e941: 12 total, 11 up, 11 in 2024-09-11T03:52:39.738 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:39.738 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:39.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:39.864 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:39.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:39.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:39.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:39.989 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:39.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:39.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:40.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:40.112 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:40.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:40.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:40.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:40.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:40.238 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:40.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:40.255 INFO:tasks.rados.rados.2.smithi175.stdout:3914: done (2 left) 2024-09-11T03:52:40.255 INFO:tasks.rados.rados.2.smithi175.stdout:3915: done (1 left) 2024-09-11T03:52:40.255 INFO:tasks.rados.rados.2.smithi175.stdout:3916: done (0 left) 2024-09-11T03:52:40.255 INFO:tasks.rados.rados.2.smithi175.stdout:3917: rollback oid 33 current snap is 434 2024-09-11T03:52:40.255 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 33 to 411 2024-09-11T03:52:40.255 INFO:tasks.rados.rados.2.smithi175.stdout:3918: snap_create 2024-09-11T03:52:40.259 INFO:tasks.rados.rados.2.smithi175.stdout:3917: finishing rollback tid 0 to smithi17557870-33 2024-09-11T03:52:40.259 INFO:tasks.rados.rados.2.smithi175.stdout:3917: finishing rollback tid 1 to smithi17557870-33 2024-09-11T03:52:40.259 INFO:tasks.rados.rados.2.smithi175.stdout:3917: finishing rollback tid 2 to smithi17557870-33 2024-09-11T03:52:40.259 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 33 v 1656 (ObjNum 964 snap 310 seq_num 964) dirty exists 2024-09-11T03:52:40.339 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:40.362 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:40.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:40.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:40.463 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:40.486 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:40.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:40.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:40.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:40.613 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:40.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:40.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:40.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:40 smithi084 ceph-mon[27254]: 2.5d scrub starts 2024-09-11T03:52:40.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:40 smithi084 ceph-mon[27254]: 2.5d scrub ok 2024-09-11T03:52:40.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:40 smithi084 ceph-mon[27254]: pgmap v1566: 127 pgs: 3 active+undersized, 1 unknown, 5 stale+active+clean, 11 peering, 1 active+undersized+degraded, 106 active+clean; 725 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 3.7 MiB/s rd, 2 op/s; 9/987 objects degraded (0.912%); 7.2 MiB/s, 23 objects/s recovering 2024-09-11T03:52:40.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:40 smithi084 ceph-mon[27254]: 2.5a scrub starts 2024-09-11T03:52:40.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:40 smithi084 ceph-mon[27254]: 2.5a scrub ok 2024-09-11T03:52:40.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:40 smithi084 ceph-mon[27254]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-11T03:52:40.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:40 smithi084 ceph-mon[27254]: Health check failed: Degraded data redundancy: 9/987 objects degraded (0.912%), 1 pg degraded (PG_DEGRADED) 2024-09-11T03:52:40.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:40 smithi084 ceph-mon[27254]: 2.5c scrub starts 2024-09-11T03:52:40.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:40 smithi084 ceph-mon[27254]: 2.5c scrub ok 2024-09-11T03:52:40.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:40 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1020926061' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:40.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:40 smithi084 ceph-mon[27254]: 2.12 scrub starts 2024-09-11T03:52:40.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:40 smithi084 ceph-mon[27254]: 2.12 scrub ok 2024-09-11T03:52:40.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:40 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:52:40.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:40 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:52:40.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:40 smithi121 ceph-mon[30729]: 2.5d scrub starts 2024-09-11T03:52:40.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:40 smithi121 ceph-mon[30729]: 2.5d scrub ok 2024-09-11T03:52:40.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:40 smithi121 ceph-mon[30729]: pgmap v1566: 127 pgs: 3 active+undersized, 1 unknown, 5 stale+active+clean, 11 peering, 1 active+undersized+degraded, 106 active+clean; 725 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 3.7 MiB/s rd, 2 op/s; 9/987 objects degraded (0.912%); 7.2 MiB/s, 23 objects/s recovering 2024-09-11T03:52:40.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:40 smithi121 ceph-mon[30729]: 2.5a scrub starts 2024-09-11T03:52:40.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:40 smithi121 ceph-mon[30729]: 2.5a scrub ok 2024-09-11T03:52:40.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:40 smithi121 ceph-mon[30729]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-11T03:52:40.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:40 smithi121 ceph-mon[30729]: Health check failed: Degraded data redundancy: 9/987 objects degraded (0.912%), 1 pg degraded (PG_DEGRADED) 2024-09-11T03:52:40.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:40 smithi121 ceph-mon[30729]: 2.5c scrub starts 2024-09-11T03:52:40.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:40 smithi121 ceph-mon[30729]: 2.5c scrub ok 2024-09-11T03:52:40.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:40 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1020926061' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:40.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:40 smithi121 ceph-mon[30729]: 2.12 scrub starts 2024-09-11T03:52:40.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:40 smithi121 ceph-mon[30729]: 2.12 scrub ok 2024-09-11T03:52:40.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:40 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:52:40.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:40 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:52:40.715 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:40.718 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T03:52:40.718 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T03:52:40.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:40 smithi175 ceph-mon[30894]: 2.5d scrub starts 2024-09-11T03:52:40.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:40 smithi175 ceph-mon[30894]: 2.5d scrub ok 2024-09-11T03:52:40.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:40 smithi175 ceph-mon[30894]: pgmap v1566: 127 pgs: 3 active+undersized, 1 unknown, 5 stale+active+clean, 11 peering, 1 active+undersized+degraded, 106 active+clean; 725 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 3.7 MiB/s rd, 2 op/s; 9/987 objects degraded (0.912%); 7.2 MiB/s, 23 objects/s recovering 2024-09-11T03:52:40.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:40 smithi175 ceph-mon[30894]: 2.5a scrub starts 2024-09-11T03:52:40.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:40 smithi175 ceph-mon[30894]: 2.5a scrub ok 2024-09-11T03:52:40.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:40 smithi175 ceph-mon[30894]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-11T03:52:40.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:40 smithi175 ceph-mon[30894]: Health check failed: Degraded data redundancy: 9/987 objects degraded (0.912%), 1 pg degraded (PG_DEGRADED) 2024-09-11T03:52:40.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:40 smithi175 ceph-mon[30894]: 2.5c scrub starts 2024-09-11T03:52:40.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:40 smithi175 ceph-mon[30894]: 2.5c scrub ok 2024-09-11T03:52:40.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:40 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1020926061' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:40.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:40 smithi175 ceph-mon[30894]: 2.12 scrub starts 2024-09-11T03:52:40.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:40 smithi175 ceph-mon[30894]: 2.12 scrub ok 2024-09-11T03:52:40.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:40 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:52:40.729 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:40 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:52:40.739 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:40.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:40.740 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:40.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:40.866 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:40.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:40.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:40.948 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-11T03:52:40.948 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:52:40.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:40.989 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:40.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:40.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:41.092 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:41.118 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:41.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:41.119 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:41.220 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:41.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:41.242 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:41.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:41.257 INFO:tasks.rados.rados.2.smithi175.stdout:3917: done (1 left) 2024-09-11T03:52:41.257 INFO:tasks.rados.rados.2.smithi175.stdout:3918: done (0 left) 2024-09-11T03:52:41.257 INFO:tasks.rados.rados.2.smithi175.stdout:3919: read oid 33 snap -1 2024-09-11T03:52:41.257 INFO:tasks.rados.rados.2.smithi175.stdout:3919: expect (ObjNum 964 snap 310 seq_num 964) 2024-09-11T03:52:41.257 INFO:tasks.rados.rados.2.smithi175.stdout:3920: delete oid 41 current snap is 435 2024-09-11T03:52:41.264 INFO:tasks.rados.rados.2.smithi175.stdout:3920: done (1 left) 2024-09-11T03:52:41.264 INFO:tasks.rados.rados.2.smithi175.stdout:3921: read oid 31 snap -1 2024-09-11T03:52:41.265 INFO:tasks.rados.rados.2.smithi175.stdout:3921: expect (ObjNum 1312 snap 429 seq_num 1312) 2024-09-11T03:52:41.265 INFO:tasks.rados.rados.2.smithi175.stdout:3922: write (excl) oid 26 current snap is 435 2024-09-11T03:52:41.265 INFO:tasks.rados.rados.2.smithi175.stdout:3922: seq_num 1330 ranges {753310=726592,2095404=257906} 2024-09-11T03:52:41.272 INFO:tasks.rados.rados.2.smithi175.stdout:3922: writing smithi17557870-26 from 753310 to 1479902 tid 1 2024-09-11T03:52:41.278 INFO:tasks.rados.rados.2.smithi175.stdout:3922: writing smithi17557870-26 from 2095404 to 2353310 tid 2 2024-09-11T03:52:41.279 INFO:tasks.rados.rados.2.smithi175.stdout:3923: read oid 18 snap -1 2024-09-11T03:52:41.279 INFO:tasks.rados.rados.2.smithi175.stdout:3923: expect deleted 2024-09-11T03:52:41.279 INFO:tasks.rados.rados.2.smithi175.stdout:3924: rollback oid 34 current snap is 435 2024-09-11T03:52:41.279 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 34 to 371 2024-09-11T03:52:41.279 INFO:tasks.rados.rados.2.smithi175.stdout:3919: expect (ObjNum 964 snap 310 seq_num 964) 2024-09-11T03:52:41.344 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:41.366 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:41.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:41.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:41.406 INFO:tasks.rados.rados.2.smithi175.stdout:3921: expect (ObjNum 1312 snap 429 seq_num 1312) 2024-09-11T03:52:41.468 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:41.471 INFO:tasks.rados.rados.2.smithi175.stdout:3919: done (4 left) 2024-09-11T03:52:41.471 INFO:tasks.rados.rados.2.smithi175.stdout:3921: done (3 left) 2024-09-11T03:52:41.471 INFO:tasks.rados.rados.2.smithi175.stdout:3923: done (2 left) 2024-09-11T03:52:41.471 INFO:tasks.rados.rados.2.smithi175.stdout:3925: delete oid 25 current snap is 435 2024-09-11T03:52:41.472 INFO:tasks.rados.rados.2.smithi175.stdout:3922: finishing write tid 1 to smithi17557870-26 2024-09-11T03:52:41.472 INFO:tasks.rados.rados.2.smithi175.stdout:3922: finishing write tid 2 to smithi17557870-26 2024-09-11T03:52:41.472 INFO:tasks.rados.rados.2.smithi175.stdout:3922: finishing write tid 3 to smithi17557870-26 2024-09-11T03:52:41.472 INFO:tasks.rados.rados.2.smithi175.stdout:3922: finishing write tid 4 to smithi17557870-26 2024-09-11T03:52:41.472 INFO:tasks.rados.rados.2.smithi175.stdout:3922: oid 26 updating version 0 to 1257 2024-09-11T03:52:41.472 INFO:tasks.rados.rados.2.smithi175.stdout:3922: oid 26 version 1257 is already newer than 1256 2024-09-11T03:52:41.472 INFO:tasks.rados.rados.2.smithi175.stdout:3922: oid 26 updating version 1257 to 1258 2024-09-11T03:52:41.472 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 26 v 1258 (ObjNum 1330 snap 435 seq_num 1330) dirty exists 2024-09-11T03:52:41.472 INFO:tasks.rados.rados.2.smithi175.stdout:3922: left oid 26 (ObjNum 1330 snap 435 seq_num 1330) 2024-09-11T03:52:41.472 INFO:tasks.rados.rados.2.smithi175.stdout:3924: finishing rollback tid 1 to smithi17557870-34 2024-09-11T03:52:41.472 INFO:tasks.rados.rados.2.smithi175.stdout:3924: finishing rollback tid 2 to smithi17557870-34 2024-09-11T03:52:41.472 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 1212 (ObjNum 875 snap 275 seq_num 875) dirty exists 2024-09-11T03:52:41.473 INFO:tasks.rados.rados.2.smithi175.stdout:3922: done (2 left) 2024-09-11T03:52:41.473 INFO:tasks.rados.rados.2.smithi175.stdout:3924: done (1 left) 2024-09-11T03:52:41.473 INFO:tasks.rados.rados.2.smithi175.stdout:3925: done (0 left) 2024-09-11T03:52:41.473 INFO:tasks.rados.rados.2.smithi175.stdout:3926: delete oid 38 current snap is 435 2024-09-11T03:52:41.475 INFO:tasks.rados.rados.2.smithi175.stdout:3926: done (0 left) 2024-09-11T03:52:41.475 INFO:tasks.rados.rados.2.smithi175.stdout:3927: snap_create 2024-09-11T03:52:41.488 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:41.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:41.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:41.590 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:41.612 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:41.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:41.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:41.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:41 smithi084 ceph-mon[27254]: 2.4f scrub starts 2024-09-11T03:52:41.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:41 smithi084 ceph-mon[27254]: 2.4f scrub ok 2024-09-11T03:52:41.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:41 smithi084 ceph-mon[27254]: osdmap e942: 12 total, 11 up, 11 in 2024-09-11T03:52:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:41 smithi121 ceph-mon[30729]: 2.4f scrub starts 2024-09-11T03:52:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:41 smithi121 ceph-mon[30729]: 2.4f scrub ok 2024-09-11T03:52:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:41 smithi121 ceph-mon[30729]: osdmap e942: 12 total, 11 up, 11 in 2024-09-11T03:52:41.715 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:41.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:41 smithi175 ceph-mon[30894]: 2.4f scrub starts 2024-09-11T03:52:41.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:41 smithi175 ceph-mon[30894]: 2.4f scrub ok 2024-09-11T03:52:41.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:41 smithi175 ceph-mon[30894]: osdmap e942: 12 total, 11 up, 11 in 2024-09-11T03:52:41.736 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:41.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:41.737 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:41.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:41.864 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:41.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:41.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:41.966 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:41.989 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:41.990 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:41.995 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-11T03:52:42.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:42.113 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:42.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:42.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:42.215 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:42.238 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:42.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:42.239 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:42.249 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:52:42.260 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:52:42 smithi084 podman[180614]: 2024-09-11 03:52:42.008128825 +0000 UTC m=+0.017118882 image pull 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e 2024-09-11T03:52:42.260 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:52:42 smithi084 podman[180614]: 2024-09-11 03:52:42.138992301 +0000 UTC m=+0.147982344 container create 03b6c42a5d3c3cf6ef84e26b5d23c41afa44aae8f6850d953f3add3dbf9a709e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T03:52:42.267 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:52:42.269 INFO:tasks.rados.rados.2.smithi175.stdout:3927: done (0 left) 2024-09-11T03:52:42.269 INFO:tasks.rados.rados.2.smithi175.stdout:3928: snap_remove snap 349 2024-09-11T03:52:42.341 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:42.365 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:42.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:42.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:42.369 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:42 smithi121 ceph-mon[30729]: pgmap v1569: 127 pgs: 3 active+undersized, 1 unknown, 5 stale+active+clean, 11 peering, 1 active+undersized+degraded, 106 active+clean; 725 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 9/987 objects degraded (0.912%) 2024-09-11T03:52:42.369 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:42 smithi121 ceph-mon[30729]: osdmap e943: 12 total, 11 up, 11 in 2024-09-11T03:52:42.369 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:42 smithi121 ceph-mon[30729]: 2.56 scrub starts 2024-09-11T03:52:42.369 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:42 smithi121 ceph-mon[30729]: 2.56 scrub ok 2024-09-11T03:52:42.369 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:42 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2994936462' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:52:42.466 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:42.488 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:42.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:42.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:42.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:42 smithi084 ceph-mon[27254]: pgmap v1569: 127 pgs: 3 active+undersized, 1 unknown, 5 stale+active+clean, 11 peering, 1 active+undersized+degraded, 106 active+clean; 725 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 9/987 objects degraded (0.912%) 2024-09-11T03:52:42.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:42 smithi084 ceph-mon[27254]: osdmap e943: 12 total, 11 up, 11 in 2024-09-11T03:52:42.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:42 smithi084 ceph-mon[27254]: 2.56 scrub starts 2024-09-11T03:52:42.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:42 smithi084 ceph-mon[27254]: 2.56 scrub ok 2024-09-11T03:52:42.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:42 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2994936462' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:52:42.533 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:52:42 smithi084 podman[180614]: 2024-09-11 03:52:42.400198469 +0000 UTC m=+0.409188513 container init 03b6c42a5d3c3cf6ef84e26b5d23c41afa44aae8f6850d953f3add3dbf9a709e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-11T03:52:42.533 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:52:42 smithi084 podman[180614]: 2024-09-11 03:52:42.403434921 +0000 UTC m=+0.412424977 container start 03b6c42a5d3c3cf6ef84e26b5d23c41afa44aae8f6850d953f3add3dbf9a709e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-deactivate, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731) 2024-09-11T03:52:42.533 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:52:42 smithi084 podman[180614]: 2024-09-11 03:52:42.48322722 +0000 UTC m=+0.492217273 container attach 03b6c42a5d3c3cf6ef84e26b5d23c41afa44aae8f6850d953f3add3dbf9a709e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-11T03:52:42.590 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:42.612 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:42.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:42.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:42.714 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:42.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:52:42.720 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:52:42.720 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:52:42.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:42 smithi175 ceph-mon[30894]: pgmap v1569: 127 pgs: 3 active+undersized, 1 unknown, 5 stale+active+clean, 11 peering, 1 active+undersized+degraded, 106 active+clean; 725 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 9/987 objects degraded (0.912%) 2024-09-11T03:52:42.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:42 smithi175 ceph-mon[30894]: osdmap e943: 12 total, 11 up, 11 in 2024-09-11T03:52:42.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:42 smithi175 ceph-mon[30894]: 2.56 scrub starts 2024-09-11T03:52:42.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:42 smithi175 ceph-mon[30894]: 2.56 scrub ok 2024-09-11T03:52:42.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:42 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2994936462' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:52:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:42.742 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:42.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:42.844 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:42.866 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:42.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:42.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:42.969 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:42.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:42.992 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:42.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:43.094 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:43.120 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:43.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:43.121 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:43.221 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:43.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:43.249 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:43.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:43.279 INFO:tasks.rados.rados.2.smithi175.stdout:3928: done (0 left) 2024-09-11T03:52:43.280 INFO:tasks.rados.rados.2.smithi175.stdout:3929: write oid 48 current snap is 436 2024-09-11T03:52:43.280 INFO:tasks.rados.rados.2.smithi175.stdout:3929: seq_num 1333 ranges {403731=668692,1587203=587797} 2024-09-11T03:52:43.286 INFO:tasks.rados.rados.2.smithi175.stdout:3929: writing smithi17557870-48 from 403731 to 1072423 tid 1 2024-09-11T03:52:43.294 INFO:tasks.rados.rados.2.smithi175.stdout:3929: writing smithi17557870-48 from 1587203 to 2175000 tid 2 2024-09-11T03:52:43.294 INFO:tasks.rados.rados.2.smithi175.stdout:3930: delete oid 28 current snap is 436 2024-09-11T03:52:43.294 INFO:tasks.rados.rados.2.smithi175.stdout:3929: finishing write tid 1 to smithi17557870-48 2024-09-11T03:52:43.298 INFO:tasks.rados.rados.2.smithi175.stdout:3930: done (1 left) 2024-09-11T03:52:43.298 INFO:tasks.rados.rados.2.smithi175.stdout:3931: rollback oid 14 current snap is 436 2024-09-11T03:52:43.298 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 14 to 430 2024-09-11T03:52:43.298 INFO:tasks.rados.rados.2.smithi175.stdout:3932: write (excl) oid 29 current snap is 436 2024-09-11T03:52:43.298 INFO:tasks.rados.rados.2.smithi175.stdout:3932: seq_num 1335 ranges {437345=419190,1587814=591466} 2024-09-11T03:52:43.302 INFO:tasks.rados.rados.2.smithi175.stdout:3932: writing smithi17557870-29 from 437345 to 856535 tid 1 2024-09-11T03:52:43.310 INFO:tasks.rados.rados.2.smithi175.stdout:3932: writing smithi17557870-29 from 1587814 to 2179280 tid 2 2024-09-11T03:52:43.310 INFO:tasks.rados.rados.2.smithi175.stdout:3933: delete oid 26 current snap is 436 2024-09-11T03:52:43.310 INFO:tasks.rados.rados.2.smithi175.stdout:3931: finishing rollback tid 1 to smithi17557870-14 2024-09-11T03:52:43.310 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 14 v 1116 (ObjNum 129 snap 0 seq_num 1746104896) dirty dne 2024-09-11T03:52:43.310 INFO:tasks.rados.rados.2.smithi175.stdout:3929: finishing write tid 2 to smithi17557870-48 2024-09-11T03:52:43.310 INFO:tasks.rados.rados.2.smithi175.stdout:3929: finishing write tid 3 to smithi17557870-48 2024-09-11T03:52:43.310 INFO:tasks.rados.rados.2.smithi175.stdout:3929: finishing write tid 4 to smithi17557870-48 2024-09-11T03:52:43.311 INFO:tasks.rados.rados.2.smithi175.stdout:3929: oid 48 updating version 0 to 1299 2024-09-11T03:52:43.311 INFO:tasks.rados.rados.2.smithi175.stdout:3929: oid 48 version 1299 is already newer than 1297 2024-09-11T03:52:43.311 INFO:tasks.rados.rados.2.smithi175.stdout:3929: oid 48 version 1299 is already newer than 1298 2024-09-11T03:52:43.311 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 1299 (ObjNum 1333 snap 436 seq_num 1333) dirty exists 2024-09-11T03:52:43.311 INFO:tasks.rados.rados.2.smithi175.stdout:3929: left oid 48 (ObjNum 1333 snap 436 seq_num 1333) 2024-09-11T03:52:43.311 INFO:tasks.rados.rados.2.smithi175.stdout:3932: finishing write tid 1 to smithi17557870-29 2024-09-11T03:52:43.311 INFO:tasks.rados.rados.2.smithi175.stdout:3929: done (3 left) 2024-09-11T03:52:43.311 INFO:tasks.rados.rados.2.smithi175.stdout:3931: done (2 left) 2024-09-11T03:52:43.311 INFO:tasks.rados.rados.2.smithi175.stdout:3933: done (1 left) 2024-09-11T03:52:43.311 INFO:tasks.rados.rados.2.smithi175.stdout:3934: delete oid 19 current snap is 436 2024-09-11T03:52:43.313 INFO:tasks.rados.rados.2.smithi175.stdout:3932: finishing write tid 2 to smithi17557870-29 2024-09-11T03:52:43.313 INFO:tasks.rados.rados.2.smithi175.stdout:3932: finishing write tid 3 to smithi17557870-29 2024-09-11T03:52:43.313 INFO:tasks.rados.rados.2.smithi175.stdout:3932: finishing write tid 4 to smithi17557870-29 2024-09-11T03:52:43.313 INFO:tasks.rados.rados.2.smithi175.stdout:3932: oid 29 updating version 0 to 1260 2024-09-11T03:52:43.313 INFO:tasks.rados.rados.2.smithi175.stdout:3932: oid 29 updating version 1260 to 1261 2024-09-11T03:52:43.313 INFO:tasks.rados.rados.2.smithi175.stdout:3932: oid 29 updating version 1261 to 1262 2024-09-11T03:52:43.313 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 1262 (ObjNum 1335 snap 436 seq_num 1335) dirty exists 2024-09-11T03:52:43.313 INFO:tasks.rados.rados.2.smithi175.stdout:3932: left oid 29 (ObjNum 1335 snap 436 seq_num 1335) 2024-09-11T03:52:43.315 INFO:tasks.rados.rados.2.smithi175.stdout:3932: done (1 left) 2024-09-11T03:52:43.315 INFO:tasks.rados.rados.2.smithi175.stdout:3934: done (0 left) 2024-09-11T03:52:43.315 INFO:tasks.rados.rados.2.smithi175.stdout:3935: copy_from oid 6 from oid 13 current snap is 436 2024-09-11T03:52:43.315 INFO:tasks.rados.rados.2.smithi175.stdout:3936: copy_from oid 41 from oid 16 current snap is 436 2024-09-11T03:52:43.315 INFO:tasks.rados.rados.2.smithi175.stdout:3937: copy_from oid 21 from oid 27 current snap is 436 2024-09-11T03:52:43.315 INFO:tasks.rados.rados.2.smithi175.stdout:3938: rollback oid 18 current snap is 436 2024-09-11T03:52:43.315 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 18 to 426 2024-09-11T03:52:43.315 INFO:tasks.rados.rados.2.smithi175.stdout:3939: copy_from oid 7 from oid 50 current snap is 436 2024-09-11T03:52:43.315 INFO:tasks.rados.rados.2.smithi175.stdout:3940: snap_create 2024-09-11T03:52:43.315 INFO:tasks.rados.rados.2.smithi175.stdout:3938: finishing rollback tid 1 to smithi17557870-18 2024-09-11T03:52:43.315 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 18 v 951 (ObjNum 65 snap 0 seq_num 1953066355) dirty dne 2024-09-11T03:52:43.321 INFO:tasks.rados.rados.2.smithi175.stdout:3939: finishing copy_from racing read to smithi17557870-7 2024-09-11T03:52:43.323 INFO:tasks.rados.rados.2.smithi175.stdout:3937: finishing copy_from racing read to smithi17557870-21 2024-09-11T03:52:43.324 INFO:tasks.rados.rados.2.smithi175.stdout:3935: finishing copy_from racing read to smithi17557870-6 2024-09-11T03:52:43.325 INFO:tasks.rados.rados.2.smithi175.stdout:3936: finishing copy_from racing read to smithi17557870-41 2024-09-11T03:52:43.326 INFO:tasks.rados.rados.2.smithi175.stdout:3937: finishing copy_from to smithi17557870-21 2024-09-11T03:52:43.326 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 1392 (ObjNum 1322 snap 432 seq_num 1322) dirty exists 2024-09-11T03:52:43.328 INFO:tasks.rados.rados.2.smithi175.stdout:3939: finishing copy_from to smithi17557870-7 2024-09-11T03:52:43.328 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 7 v 1534 (ObjNum 1190 snap 388 seq_num 1190) dirty exists 2024-09-11T03:52:43.328 INFO:tasks.rados.rados.2.smithi175.stdout:3935: finishing copy_from to smithi17557870-6 2024-09-11T03:52:43.328 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 6 v 1655 (ObjNum 1327 snap 433 seq_num 1327) dirty exists 2024-09-11T03:52:43.332 INFO:tasks.rados.rados.2.smithi175.stdout:3936: finishing copy_from to smithi17557870-41 2024-09-11T03:52:43.332 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 1007 (ObjNum 1275 snap 422 seq_num 1275) dirty exists 2024-09-11T03:52:43.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:43.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:43.372 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:43.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:43.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:43.494 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:43.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:43.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:43.498 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:52:43 smithi084 podman[180614]: 2024-09-11 03:52:43.224402832 +0000 UTC m=+1.233392951 container died 03b6c42a5d3c3cf6ef84e26b5d23c41afa44aae8f6850d953f3add3dbf9a709e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-11T03:52:43.498 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:43 smithi084 ceph-mon[27254]: 2.35 scrub starts 2024-09-11T03:52:43.498 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:43 smithi084 ceph-mon[27254]: 2.35 scrub ok 2024-09-11T03:52:43.498 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:43 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:52:43.498 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:43 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2994936462' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:52:43.498 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:43 smithi084 ceph-mon[27254]: osdmap e944: 12 total, 11 up, 11 in 2024-09-11T03:52:43.596 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:43.604 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:43 smithi121 ceph-mon[30729]: 2.35 scrub starts 2024-09-11T03:52:43.604 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:43 smithi121 ceph-mon[30729]: 2.35 scrub ok 2024-09-11T03:52:43.604 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:43 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:52:43.604 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:43 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2994936462' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:52:43.604 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:43 smithi121 ceph-mon[30729]: osdmap e944: 12 total, 11 up, 11 in 2024-09-11T03:52:43.620 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:43.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:43.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:43.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-11T03:52:43.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:43.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:43 smithi175 ceph-mon[30894]: 2.35 scrub starts 2024-09-11T03:52:43.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:43 smithi175 ceph-mon[30894]: 2.35 scrub ok 2024-09-11T03:52:43.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:43 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:52:43.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:43 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2994936462' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:52:43.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:43 smithi175 ceph-mon[30894]: osdmap e944: 12 total, 11 up, 11 in 2024-09-11T03:52:43.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:43.745 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:43.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:43.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:43.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.0.service' 2024-09-11T03:52:43.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:43.870 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:43.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:43.873 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:52:43.899 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:43.901 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:52:43.902 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-11T03:52:43.902 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.0 2024-09-11T03:52:43.902 INFO:tasks.thrashosds.thrasher:remote for osd 0 is ubuntu@smithi084.front.sepia.ceph.com 2024-09-11T03:52:43.902 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -n osd.0 --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2024-09-11T03:52:43.971 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:43.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:43.993 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:43.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:44.095 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:44.115 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:44.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:44.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:44.173 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/osd.0/config 2024-09-11T03:52:44.218 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:44.244 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:44.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:44.245 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:44.281 INFO:tasks.rados.rados.2.smithi175.stdout:3935: done (5 left) 2024-09-11T03:52:44.281 INFO:tasks.rados.rados.2.smithi175.stdout:3936: done (4 left) 2024-09-11T03:52:44.281 INFO:tasks.rados.rados.2.smithi175.stdout:3937: done (3 left) 2024-09-11T03:52:44.281 INFO:tasks.rados.rados.2.smithi175.stdout:3938: done (2 left) 2024-09-11T03:52:44.281 INFO:tasks.rados.rados.2.smithi175.stdout:3939: done (1 left) 2024-09-11T03:52:44.281 INFO:tasks.rados.rados.2.smithi175.stdout:3940: done (0 left) 2024-09-11T03:52:44.282 INFO:tasks.rados.rados.2.smithi175.stdout:3941: write (excl) oid 42 current snap is 437 2024-09-11T03:52:44.282 INFO:tasks.rados.rados.2.smithi175.stdout:3941: seq_num 1338 ranges {487766=644937,1646907=738793} 2024-09-11T03:52:44.283 INFO:tasks.rados.rados.2.smithi175.stdout:3941: writing smithi17557870-42 from 487766 to 1132703 tid 1 2024-09-11T03:52:44.291 INFO:tasks.rados.rados.2.smithi175.stdout:3941: writing smithi17557870-42 from 1646907 to 2385700 tid 2 2024-09-11T03:52:44.291 INFO:tasks.rados.rados.2.smithi175.stdout:3942: write (excl) oid 34 current snap is 437 2024-09-11T03:52:44.291 INFO:tasks.rados.rados.2.smithi175.stdout:3942: seq_num 1339 ranges {504573=436539,1421742=598804,2437505=67068} 2024-09-11T03:52:44.296 INFO:tasks.rados.rados.2.smithi175.stdout:3942: writing smithi17557870-34 from 504573 to 941112 tid 1 2024-09-11T03:52:44.302 INFO:tasks.rados.rados.2.smithi175.stdout:3942: writing smithi17557870-34 from 1421742 to 2020546 tid 2 2024-09-11T03:52:44.306 INFO:tasks.rados.rados.2.smithi175.stdout:3942: writing smithi17557870-34 from 2437505 to 2504573 tid 3 2024-09-11T03:52:44.306 INFO:tasks.rados.rados.2.smithi175.stdout:3943: rollback oid 47 current snap is 437 2024-09-11T03:52:44.306 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 47 to 433 2024-09-11T03:52:44.306 INFO:tasks.rados.rados.2.smithi175.stdout:3941: finishing write tid 1 to smithi17557870-42 2024-09-11T03:52:44.306 INFO:tasks.rados.rados.2.smithi175.stdout:3941: finishing write tid 2 to smithi17557870-42 2024-09-11T03:52:44.306 INFO:tasks.rados.rados.2.smithi175.stdout:3941: finishing write tid 3 to smithi17557870-42 2024-09-11T03:52:44.306 INFO:tasks.rados.rados.2.smithi175.stdout:3941: finishing write tid 4 to smithi17557870-42 2024-09-11T03:52:44.306 INFO:tasks.rados.rados.2.smithi175.stdout:3941: oid 42 updating version 0 to 1077 2024-09-11T03:52:44.306 INFO:tasks.rados.rados.2.smithi175.stdout:3941: oid 42 version 1077 is already newer than 1076 2024-09-11T03:52:44.307 INFO:tasks.rados.rados.2.smithi175.stdout:3941: oid 42 updating version 1077 to 1078 2024-09-11T03:52:44.307 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 1078 (ObjNum 1338 snap 437 seq_num 1338) dirty exists 2024-09-11T03:52:44.307 INFO:tasks.rados.rados.2.smithi175.stdout:3941: left oid 42 (ObjNum 1338 snap 437 seq_num 1338) 2024-09-11T03:52:44.307 INFO:tasks.rados.rados.2.smithi175.stdout:3942: finishing write tid 1 to smithi17557870-34 2024-09-11T03:52:44.307 INFO:tasks.rados.rados.2.smithi175.stdout:3941: done (2 left) 2024-09-11T03:52:44.307 INFO:tasks.rados.rados.2.smithi175.stdout:3944: read oid 45 snap -1 2024-09-11T03:52:44.307 INFO:tasks.rados.rados.2.smithi175.stdout:3944: expect (ObjNum 1325 snap 433 seq_num 1325) 2024-09-11T03:52:44.307 INFO:tasks.rados.rados.2.smithi175.stdout:3945: read oid 7 snap -1 2024-09-11T03:52:44.307 INFO:tasks.rados.rados.2.smithi175.stdout:3945: expect (ObjNum 1190 snap 388 seq_num 1190) 2024-09-11T03:52:44.307 INFO:tasks.rados.rados.2.smithi175.stdout:3946: rollback oid 4 current snap is 437 2024-09-11T03:52:44.307 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 4 to 368 2024-09-11T03:52:44.307 INFO:tasks.rados.rados.2.smithi175.stdout:3947: snap_create 2024-09-11T03:52:44.308 INFO:tasks.rados.rados.2.smithi175.stdout:3942: finishing write tid 2 to smithi17557870-34 2024-09-11T03:52:44.308 INFO:tasks.rados.rados.2.smithi175.stdout:3942: finishing write tid 3 to smithi17557870-34 2024-09-11T03:52:44.308 INFO:tasks.rados.rados.2.smithi175.stdout:3942: finishing write tid 4 to smithi17557870-34 2024-09-11T03:52:44.309 INFO:tasks.rados.rados.2.smithi175.stdout:3942: finishing write tid 5 to smithi17557870-34 2024-09-11T03:52:44.309 INFO:tasks.rados.rados.2.smithi175.stdout:3942: oid 34 updating version 0 to 1216 2024-09-11T03:52:44.309 INFO:tasks.rados.rados.2.smithi175.stdout:3942: oid 34 version 1216 is already newer than 1214 2024-09-11T03:52:44.309 INFO:tasks.rados.rados.2.smithi175.stdout:3942: oid 34 version 1216 is already newer than 1213 2024-09-11T03:52:44.309 INFO:tasks.rados.rados.2.smithi175.stdout:3942: oid 34 version 1216 is already newer than 1215 2024-09-11T03:52:44.309 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 34 v 1216 (ObjNum 1339 snap 437 seq_num 1339) dirty exists 2024-09-11T03:52:44.309 INFO:tasks.rados.rados.2.smithi175.stdout:3942: left oid 34 (ObjNum 1339 snap 437 seq_num 1339) 2024-09-11T03:52:44.312 INFO:tasks.rados.rados.2.smithi175.stdout:3946: finishing rollback tid 0 to smithi17557870-4 2024-09-11T03:52:44.312 INFO:tasks.rados.rados.2.smithi175.stdout:3943: finishing rollback tid 1 to smithi17557870-47 2024-09-11T03:52:44.312 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 1413 (ObjNum 129 snap 0 seq_num 1746350024) dirty dne 2024-09-11T03:52:44.313 INFO:tasks.rados.rados.2.smithi175.stdout:3946: finishing rollback tid 1 to smithi17557870-4 2024-09-11T03:52:44.313 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 4 v 1301 (ObjNum 129 snap 0 seq_num 1746360096) dirty dne 2024-09-11T03:52:44.313 INFO:tasks.rados.rados.2.smithi175.stdout:3944: expect (ObjNum 1325 snap 433 seq_num 1325) 2024-09-11T03:52:44.346 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:44.368 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:44.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:44.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:44.389 INFO:tasks.rados.rados.2.smithi175.stdout:3945: expect (ObjNum 1190 snap 388 seq_num 1190) 2024-09-11T03:52:44.398 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:52:44.399 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:52:44.423 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:44 smithi084 ceph-mon[27254]: pgmap v1572: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 732 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 69/1002 objects degraded (6.886%) 2024-09-11T03:52:44.424 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:44 smithi084 ceph-mon[27254]: 2.17 scrub starts 2024-09-11T03:52:44.424 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:44 smithi084 ceph-mon[27254]: 2.17 scrub ok 2024-09-11T03:52:44.424 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:44 smithi084 ceph-mon[27254]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-11T03:52:44.424 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:44 smithi084 ceph-mon[27254]: osdmap e945: 12 total, 11 up, 11 in 2024-09-11T03:52:44.427 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-11T03:52:44.428 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:52:44.428 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:52:44.428 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:52:44.429 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:52:44.439 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:52:44.439 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:52:44.455 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:52:44.457 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:52:44.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:44.472 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:52:44.472 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:52:44.489 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:52:44.489 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:52:44.492 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:44.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:44.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:44.505 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:52:44.506 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:52:44.522 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:52:44.522 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:52:44.538 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:52:44.539 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:52:44.555 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:52:44.556 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:52:44.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:52:44.572 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:52:44.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:44.615 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:44.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:44.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:44 smithi121 ceph-mon[30729]: pgmap v1572: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 732 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 69/1002 objects degraded (6.886%) 2024-09-11T03:52:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:44 smithi121 ceph-mon[30729]: 2.17 scrub starts 2024-09-11T03:52:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:44 smithi121 ceph-mon[30729]: 2.17 scrub ok 2024-09-11T03:52:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:44 smithi121 ceph-mon[30729]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-11T03:52:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:44 smithi121 ceph-mon[30729]: osdmap e945: 12 total, 11 up, 11 in 2024-09-11T03:52:44.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:44 smithi175 ceph-mon[30894]: pgmap v1572: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 732 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 69/1002 objects degraded (6.886%) 2024-09-11T03:52:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:44 smithi175 ceph-mon[30894]: 2.17 scrub starts 2024-09-11T03:52:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:44 smithi175 ceph-mon[30894]: 2.17 scrub ok 2024-09-11T03:52:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:44 smithi175 ceph-mon[30894]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-11T03:52:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:44 smithi175 ceph-mon[30894]: osdmap e945: 12 total, 11 up, 11 in 2024-09-11T03:52:44.737 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:44.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:44.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:44.839 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:44.863 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:44.863 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:44.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:44.986 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:44.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:45.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:45.110 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:45.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:45.115 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-11T03:52:45.207 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-11T03:52:45.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:45.271 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:45.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:45.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:45.288 INFO:tasks.rados.rados.2.smithi175.stdout:3942: done (5 left) 2024-09-11T03:52:45.289 INFO:tasks.rados.rados.2.smithi175.stdout:3943: done (4 left) 2024-09-11T03:52:45.289 INFO:tasks.rados.rados.2.smithi175.stdout:3944: done (3 left) 2024-09-11T03:52:45.289 INFO:tasks.rados.rados.2.smithi175.stdout:3945: done (2 left) 2024-09-11T03:52:45.289 INFO:tasks.rados.rados.2.smithi175.stdout:3946: done (1 left) 2024-09-11T03:52:45.289 INFO:tasks.rados.rados.2.smithi175.stdout:3947: done (0 left) 2024-09-11T03:52:45.290 INFO:tasks.rados.rados.2.smithi175.stdout:3948: read oid 17 snap -1 2024-09-11T03:52:45.290 INFO:tasks.rados.rados.2.smithi175.stdout:3948: expect (ObjNum 1181 snap 385 seq_num 1181) 2024-09-11T03:52:45.290 INFO:tasks.rados.rados.2.smithi175.stdout:3949: rollback oid 41 current snap is 438 2024-09-11T03:52:45.290 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 41 to 354 2024-09-11T03:52:45.290 INFO:tasks.rados.rados.2.smithi175.stdout:3950: write oid 24 current snap is 438 2024-09-11T03:52:45.290 INFO:tasks.rados.rados.2.smithi175.stdout:3950: seq_num 1340 ranges {521380=511788,1480224=742462} 2024-09-11T03:52:45.293 INFO:tasks.rados.rados.2.smithi175.stdout:3950: writing smithi17557870-24 from 521380 to 1033168 tid 1 2024-09-11T03:52:45.302 INFO:tasks.rados.rados.2.smithi175.stdout:3950: writing smithi17557870-24 from 1480224 to 2222686 tid 2 2024-09-11T03:52:45.303 INFO:tasks.rados.rados.2.smithi175.stdout:3951: read oid 6 snap -1 2024-09-11T03:52:45.304 INFO:tasks.rados.rados.2.smithi175.stdout:3951: expect (ObjNum 1327 snap 433 seq_num 1327) 2024-09-11T03:52:45.304 INFO:tasks.rados.rados.2.smithi175.stdout:3952: snap_create 2024-09-11T03:52:45.304 INFO:tasks.rados.rados.2.smithi175.stdout:3949: finishing rollback tid 0 to smithi17557870-41 2024-09-11T03:52:45.304 INFO:tasks.rados.rados.2.smithi175.stdout:3949: finishing rollback tid 1 to smithi17557870-41 2024-09-11T03:52:45.304 INFO:tasks.rados.rados.2.smithi175.stdout:3949: finishing rollback tid 2 to smithi17557870-41 2024-09-11T03:52:45.304 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 41 v 1041 (ObjNum 1073 snap 350 seq_num 1073) dirty exists 2024-09-11T03:52:45.305 INFO:tasks.rados.rados.2.smithi175.stdout:3948: expect (ObjNum 1181 snap 385 seq_num 1181) 2024-09-11T03:52:45.357 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:52:45.373 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:45.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:45.396 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:45.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:45 smithi121 ceph-mon[30729]: 2.3a scrub starts 2024-09-11T03:52:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:45 smithi121 ceph-mon[30729]: 2.3a scrub ok 2024-09-11T03:52:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:45 smithi121 ceph-mon[30729]: osdmap e946: 12 total, 11 up, 11 in 2024-09-11T03:52:45.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:45 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1886455349' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:45.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:45 smithi121 ceph-mon[30729]: 1.10 scrub starts 2024-09-11T03:52:45.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:45 smithi121 ceph-mon[30729]: 1.10 scrub ok 2024-09-11T03:52:45.426 INFO:tasks.rados.rados.2.smithi175.stdout:3950: finishing write tid 1 to smithi17557870-24 2024-09-11T03:52:45.426 INFO:tasks.rados.rados.2.smithi175.stdout:3950: finishing write tid 2 to smithi17557870-24 2024-09-11T03:52:45.427 INFO:tasks.rados.rados.2.smithi175.stdout:3950: finishing write tid 3 to smithi17557870-24 2024-09-11T03:52:45.427 INFO:tasks.rados.rados.2.smithi175.stdout:3950: finishing write tid 4 to smithi17557870-24 2024-09-11T03:52:45.427 INFO:tasks.rados.rados.2.smithi175.stdout:3950: oid 24 updating version 0 to 1138 2024-09-11T03:52:45.427 INFO:tasks.rados.rados.2.smithi175.stdout:3950: oid 24 updating version 1138 to 1140 2024-09-11T03:52:45.427 INFO:tasks.rados.rados.2.smithi175.stdout:3950: oid 24 version 1140 is already newer than 1139 2024-09-11T03:52:45.427 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 1140 (ObjNum 1340 snap 438 seq_num 1340) dirty exists 2024-09-11T03:52:45.427 INFO:tasks.rados.rados.2.smithi175.stdout:3950: left oid 24 (ObjNum 1340 snap 438 seq_num 1340) 2024-09-11T03:52:45.427 INFO:tasks.rados.rados.2.smithi175.stdout:3951: expect (ObjNum 1327 snap 433 seq_num 1327) 2024-09-11T03:52:45.497 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:45.519 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:45.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:45.520 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:45.621 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:45.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:45.643 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:45.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:45.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:45 smithi084 ceph-mon[27254]: 2.3a scrub starts 2024-09-11T03:52:45.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:45 smithi084 ceph-mon[27254]: 2.3a scrub ok 2024-09-11T03:52:45.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:45 smithi084 ceph-mon[27254]: osdmap e946: 12 total, 11 up, 11 in 2024-09-11T03:52:45.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:45 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1886455349' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:45.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:45 smithi084 ceph-mon[27254]: 1.10 scrub starts 2024-09-11T03:52:45.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:45 smithi084 ceph-mon[27254]: 1.10 scrub ok 2024-09-11T03:52:45.712 INFO:teuthology.orchestra.run.smithi084.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-11T03:52:45.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:45 smithi175 ceph-mon[30894]: 2.3a scrub starts 2024-09-11T03:52:45.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:45 smithi175 ceph-mon[30894]: 2.3a scrub ok 2024-09-11T03:52:45.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:45 smithi175 ceph-mon[30894]: osdmap e946: 12 total, 11 up, 11 in 2024-09-11T03:52:45.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:45 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1886455349' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:45.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:45 smithi175 ceph-mon[30894]: 1.10 scrub starts 2024-09-11T03:52:45.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:45 smithi175 ceph-mon[30894]: 1.10 scrub ok 2024-09-11T03:52:45.744 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:45.766 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:45.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:45.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:45.868 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:45.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:45.890 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:45.890 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:45.981 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:52:45.982 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:52:45.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:52:45.982 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:52:45.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.982 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3040 2.d 2:b6a25a79:::smithi17557870-47:head [read 0~2201783,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:52:45.982 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:52:45.982 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 594.08412665000003, 2024-09-11T03:52:45.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020060432, 2024-09-11T03:52:45.982 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:45.982 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:45.982 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:52:45.983 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:45.983 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:45.983 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:45.983 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3040 2024-09-11T03:52:45.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.983 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:45.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:45.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:52:45.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:45.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:45.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896980+0000", 2024-09-11T03:52:45.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:45.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:45.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896981+0000", 2024-09-11T03:52:45.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.747e-06 2024-09-11T03:52:45.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:45.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.896989+0000", 2024-09-11T03:52:45.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9559999999999997e-06 2024-09-11T03:52:45.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:45.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897005+0000", 2024-09-11T03:52:45.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5704000000000001e-05 2024-09-11T03:52:45.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:45.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897013+0000", 2024-09-11T03:52:45.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.028e-06 2024-09-11T03:52:45.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:45.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897078+0000", 2024-09-11T03:52:45.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4813000000000004e-05 2024-09-11T03:52:45.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:52:45.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897083+0000", 2024-09-11T03:52:45.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1109999999999999e-06 2024-09-11T03:52:45.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:45.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.914637+0000", 2024-09-11T03:52:45.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017553701000000001 2024-09-11T03:52:45.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:45.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.914744+0000", 2024-09-11T03:52:45.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107131 2024-09-11T03:52:45.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:45.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917040+0000", 2024-09-11T03:52:45.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022962410000000001 2024-09-11T03:52:45.987 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:45.987 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:45.988 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:45.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.988 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3041 2.d 2:b6a25a79:::smithi17557870-47:head [sparse-read 0~1480887] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:52:45.988 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:52:45.988 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 594.08406109400005, 2024-09-11T03:52:45.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020163026000000001, 2024-09-11T03:52:45.988 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:45.988 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:45.988 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:52:45.988 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:45.988 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:45.989 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:45.989 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3041 2024-09-11T03:52:45.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.989 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:45.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:45.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:52:45.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:45.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:45.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897045+0000", 2024-09-11T03:52:45.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:45.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:45.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897046+0000", 2024-09-11T03:52:45.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7e-07 2024-09-11T03:52:45.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:45.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897048+0000", 2024-09-11T03:52:45.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0700000000000001e-06 2024-09-11T03:52:45.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:45.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897058+0000", 2024-09-11T03:52:45.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0151e-05 2024-09-11T03:52:45.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:45.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897068+0000", 2024-09-11T03:52:45.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0329000000000001e-05 2024-09-11T03:52:45.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:45.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897128+0000", 2024-09-11T03:52:45.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0208999999999999e-05 2024-09-11T03:52:45.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:45.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897132+0000", 2024-09-11T03:52:45.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7290000000000002e-06 2024-09-11T03:52:45.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:45.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917070+0000", 2024-09-11T03:52:45.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019937951999999998 2024-09-11T03:52:45.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:45.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917123+0000", 2024-09-11T03:52:45.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3149000000000002e-05 2024-09-11T03:52:45.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:45.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917208+0000", 2024-09-11T03:52:45.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5067000000000004e-05 2024-09-11T03:52:45.993 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:45.993 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:45.993 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:45.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.993 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3042 2.d 2:b6a25a79:::smithi17557870-47:head [sparse-read 0~1480887] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:52:45.993 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:52:45.994 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 594.08402691499998, 2024-09-11T03:52:45.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020302042999999999, 2024-09-11T03:52:45.994 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:45.994 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:45.994 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:52:45.994 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:45.994 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:45.994 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:45.994 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3042 2024-09-11T03:52:45.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.994 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:45.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:45.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:52:45.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:45.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:45.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897079+0000", 2024-09-11T03:52:45.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:45.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:45.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897080+0000", 2024-09-11T03:52:45.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.82e-07 2024-09-11T03:52:45.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:45.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897082+0000", 2024-09-11T03:52:45.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.418e-06 2024-09-11T03:52:45.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:45.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897089+0000", 2024-09-11T03:52:45.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8090000000000004e-06 2024-09-11T03:52:45.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:45.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897094+0000", 2024-09-11T03:52:45.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2399999999999998e-06 2024-09-11T03:52:45.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:45.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:52:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0928999999999999e-05 2024-09-11T03:52:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897158+0000", 2024-09-11T03:52:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9890000000000001e-06 2024-09-11T03:52:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917227+0000", 2024-09-11T03:52:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020069019 2024-09-11T03:52:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917290+0000", 2024-09-11T03:52:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2848999999999994e-05 2024-09-11T03:52:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917381+0000", 2024-09-11T03:52:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1308000000000001e-05 2024-09-11T03:52:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3046 2.d 2:b6874adf:::smithi17557870-17:head [sparse-read 0~1769472,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:52:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:52:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 594.08400495399997, 2024-09-11T03:52:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021672665000000001, 2024-09-11T03:52:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:52:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3046 2024-09-11T03:52:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:52:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897101+0000", 2024-09-11T03:52:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897102+0000", 2024-09-11T03:52:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.65e-07 2024-09-11T03:52:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897105+0000", 2024-09-11T03:52:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1820000000000002e-06 2024-09-11T03:52:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897111+0000", 2024-09-11T03:52:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4330000000000001e-06 2024-09-11T03:52:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897117+0000", 2024-09-11T03:52:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2979999999999999e-06 2024-09-11T03:52:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897179+0000", 2024-09-11T03:52:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2825999999999998e-05 2024-09-11T03:52:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897182+0000", 2024-09-11T03:52:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7769999999999998e-06 2024-09-11T03:52:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917399+0000", 2024-09-11T03:52:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020216622 2024-09-11T03:52:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.917444+0000", 2024-09-11T03:52:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5558e-05 2024-09-11T03:52:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918774+0000", 2024-09-11T03:52:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013296040000000001 2024-09-11T03:52:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3047 2.d 2:b6874adf:::smithi17557870-17:head [read 0~2453888,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:52:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:52:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 594.08397578200004, 2024-09-11T03:52:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022691901, 2024-09-11T03:52:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:52:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3047 2024-09-11T03:52:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:52:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:52:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897131+0000", 2024-09-11T03:52:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8599999999999998e-07 2024-09-11T03:52:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897134+0000", 2024-09-11T03:52:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1470000000000001e-06 2024-09-11T03:52:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897141+0000", 2024-09-11T03:52:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8029999999999999e-06 2024-09-11T03:52:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897147+0000", 2024-09-11T03:52:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4520000000000002e-06 2024-09-11T03:52:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897202+0000", 2024-09-11T03:52:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4255999999999997e-05 2024-09-11T03:52:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897204+0000", 2024-09-11T03:52:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6649999999999999e-06 2024-09-11T03:52:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918807+0000", 2024-09-11T03:52:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021602962999999999 2024-09-11T03:52:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.918887+0000", 2024-09-11T03:52:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9550999999999998e-05 2024-09-11T03:52:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919822+0000", 2024-09-11T03:52:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093557799999999995 2024-09-11T03:52:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:3048 2.d 2:b6874adf:::smithi17557870-17:head [read 0~2453888,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e355)", 2024-09-11T03:52:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:52:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 594.08395095799995, 2024-09-11T03:52:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.023274353000000001, 2024-09-11T03:52:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:52:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3048 2024-09-11T03:52:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:52:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897155+0000", 2024-09-11T03:52:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897156+0000", 2024-09-11T03:52:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1199999999999998e-07 2024-09-11T03:52:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897159+0000", 2024-09-11T03:52:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7439999999999999e-06 2024-09-11T03:52:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897165+0000", 2024-09-11T03:52:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6749999999999996e-06 2024-09-11T03:52:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897170+0000", 2024-09-11T03:52:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9420000000000003e-06 2024-09-11T03:52:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897226+0000", 2024-09-11T03:52:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6026999999999997e-05 2024-09-11T03:52:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.897232+0000", 2024-09-11T03:52:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2609999999999999e-06 2024-09-11T03:52:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919841+0000", 2024-09-11T03:52:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022609054999999999 2024-09-11T03:52:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.919885+0000", 2024-09-11T03:52:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3383999999999997e-05 2024-09-11T03:52:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:42:51.920430+0000", 2024-09-11T03:52:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054485299999999998 2024-09-11T03:52:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4176 2.3 2:c6c49f2c:::smithi17557870-44:head [copy-from ver 412] snapc b5=[b5,b4,b2,b1,af,ae,ac,ab,a9,a8,9d] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-09-11T03:52:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:28.933487+0000", 2024-09-11T03:52:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 497.04761975100001, 2024-09-11T03:52:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018627511999999999, 2024-09-11T03:52:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4176 2024-09-11T03:52:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933487+0000", 2024-09-11T03:52:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933487+0000", 2024-09-11T03:52:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933489+0000", 2024-09-11T03:52:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2510000000000001e-06 2024-09-11T03:52:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933498+0000", 2024-09-11T03:52:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3219999999999999e-06 2024-09-11T03:52:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933516+0000", 2024-09-11T03:52:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7971000000000001e-05 2024-09-11T03:52:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933529+0000", 2024-09-11T03:52:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2476e-05 2024-09-11T03:52:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933657+0000", 2024-09-11T03:52:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000128842 2024-09-11T03:52:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:52:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933663+0000", 2024-09-11T03:52:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9580000000000004e-06 2024-09-11T03:52:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935429+0000", 2024-09-11T03:52:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001765146 2024-09-11T03:52:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935549+0000", 2024-09-11T03:52:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120504 2024-09-11T03:52:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,7", 2024-09-11T03:52:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.944235+0000", 2024-09-11T03:52:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086856499999999996 2024-09-11T03:52:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948823+0000", 2024-09-11T03:52:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045879570000000001 2024-09-11T03:52:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.951466+0000", 2024-09-11T03:52:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026434739999999998 2024-09-11T03:52:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.952026+0000", 2024-09-11T03:52:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055985599999999996 2024-09-11T03:52:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.952068+0000", 2024-09-11T03:52:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1918000000000001e-05 2024-09-11T03:52:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.952114+0000", 2024-09-11T03:52:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6186999999999997e-05 2024-09-11T03:52:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:4177 2.3 2:c6c49f2c:::smithi17557870-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e451)", 2024-09-11T03:52:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:44:28.933561+0000", 2024-09-11T03:52:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 497.04754493799999, 2024-09-11T03:52:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014798686, 2024-09-11T03:52:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:52:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4177 2024-09-11T03:52:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933561+0000", 2024-09-11T03:52:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933561+0000", 2024-09-11T03:52:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933562+0000", 2024-09-11T03:52:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4600000000000005e-07 2024-09-11T03:52:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933565+0000", 2024-09-11T03:52:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8729999999999999e-06 2024-09-11T03:52:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933572+0000", 2024-09-11T03:52:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6090000000000003e-06 2024-09-11T03:52:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933579+0000", 2024-09-11T03:52:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4339999999999997e-06 2024-09-11T03:52:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933717+0000", 2024-09-11T03:52:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000137995 2024-09-11T03:52:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.933721+0000", 2024-09-11T03:52:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6940000000000001e-06 2024-09-11T03:52:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935773+0000", 2024-09-11T03:52:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002052398 2024-09-11T03:52:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.935834+0000", 2024-09-11T03:52:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1146e-05 2024-09-11T03:52:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948174+0000", 2024-09-11T03:52:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012339786 2024-09-11T03:52:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948263+0000", 2024-09-11T03:52:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8838999999999994e-05 2024-09-11T03:52:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:44:28.948360+0000", 2024-09-11T03:52:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7366000000000003e-05 2024-09-11T03:52:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:6215 2.16 2:6dbd1e81:::smithi17557870-7:head [copy-from ver 268] snapc 112=[112,111,10f] ondisk+write+known_if_redirected+supports_pool_eio e646)", 2024-09-11T03:52:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:43.987602+0000", 2024-09-11T03:52:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 301.99350454799998, 2024-09-11T03:52:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021363759, 2024-09-11T03:52:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 6215 2024-09-11T03:52:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987602+0000", 2024-09-11T03:52:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987602+0000", 2024-09-11T03:52:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987603+0000", 2024-09-11T03:52:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6339999999999999e-06 2024-09-11T03:52:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987612+0000", 2024-09-11T03:52:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1170000000000004e-06 2024-09-11T03:52:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987631+0000", 2024-09-11T03:52:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9245e-05 2024-09-11T03:52:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987641+0000", 2024-09-11T03:52:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8570000000000007e-06 2024-09-11T03:52:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987718+0000", 2024-09-11T03:52:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7812999999999995e-05 2024-09-11T03:52:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:52:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987726+0000", 2024-09-11T03:52:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1960000000000002e-06 2024-09-11T03:52:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.989804+0000", 2024-09-11T03:52:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002078004 2024-09-11T03:52:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.989909+0000", 2024-09-11T03:52:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105518 2024-09-11T03:52:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6,11", 2024-09-11T03:52:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.999305+0000", 2024-09-11T03:52:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.009396349 2024-09-11T03:52:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.007386+0000", 2024-09-11T03:52:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080807069999999995 2024-09-11T03:52:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.007696+0000", 2024-09-11T03:52:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030955000000000002 2024-09-11T03:52:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.008885+0000", 2024-09-11T03:52:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011892700000000001 2024-09-11T03:52:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.008929+0000", 2024-09-11T03:52:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4322e-05 2024-09-11T03:52:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.008966+0000", 2024-09-11T03:52:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6177000000000003e-05 2024-09-11T03:52:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:6216 2.16 2:6dbd1e81:::smithi17557870-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e646)", 2024-09-11T03:52:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:43.987704+0000", 2024-09-11T03:52:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 301.99340252600001, 2024-09-11T03:52:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016255282999999999, 2024-09-11T03:52:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:52:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 6216 2024-09-11T03:52:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987704+0000", 2024-09-11T03:52:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987704+0000", 2024-09-11T03:52:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987704+0000", 2024-09-11T03:52:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5000000000000002e-07 2024-09-11T03:52:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987708+0000", 2024-09-11T03:52:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6610000000000002e-06 2024-09-11T03:52:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987718+0000", 2024-09-11T03:52:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7690000000000008e-06 2024-09-11T03:52:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987725+0000", 2024-09-11T03:52:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7279999999999998e-06 2024-09-11T03:52:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987785+0000", 2024-09-11T03:52:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0163e-05 2024-09-11T03:52:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.987790+0000", 2024-09-11T03:52:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2410000000000003e-06 2024-09-11T03:52:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.990032+0000", 2024-09-11T03:52:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022420880000000002 2024-09-11T03:52:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:43.990058+0000", 2024-09-11T03:52:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6044999999999999e-05 2024-09-11T03:52:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.003794+0000", 2024-09-11T03:52:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01373564 2024-09-11T03:52:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.003878+0000", 2024-09-11T03:52:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3954000000000003e-05 2024-09-11T03:52:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:44.003959+0000", 2024-09-11T03:52:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1343999999999999e-05 2024-09-11T03:52:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.24454.0:7056 2.20 e710/574 2:0762776a:::smithi17557870-25:head v 710'215, mlcod=628'211)", 2024-09-11T03:52:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:48.028737+0000", 2024-09-11T03:52:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 237.95236928200001, 2024-09-11T03:52:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015591568, 2024-09-11T03:52:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.028737+0000", 2024-09-11T03:52:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.028737+0000", 2024-09-11T03:52:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.028737+0000", 2024-09-11T03:52:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0600000000000001e-07 2024-09-11T03:52:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032034+0000", 2024-09-11T03:52:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032964890000000001 2024-09-11T03:52:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032066+0000", 2024-09-11T03:52:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2011999999999998e-05 2024-09-11T03:52:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032079+0000", 2024-09-11T03:52:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2702e-05 2024-09-11T03:52:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032193+0000", 2024-09-11T03:52:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114737 2024-09-11T03:52:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.032231+0000", 2024-09-11T03:52:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7867000000000002e-05 2024-09-11T03:52:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.044229+0000", 2024-09-11T03:52:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011997499999999999 2024-09-11T03:52:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.044329+0000", 2024-09-11T03:52:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9854999999999999e-05 2024-09-11T03:52:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7213 2.d 2:b6874adf:::smithi17557870-17:head [write 2124737~564614] snapc 141=[141,140,13f,13d,13c,139,138,137,136,135,133,12e,12d,12a,127,126,120,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-09-11T03:52:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:01.038236+0000", 2024-09-11T03:52:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 224.94287055199999, 2024-09-11T03:52:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014496043, 2024-09-11T03:52:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7213 2024-09-11T03:52:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038236+0000", 2024-09-11T03:52:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038236+0000", 2024-09-11T03:52:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038237+0000", 2024-09-11T03:52:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.153e-06 2024-09-11T03:52:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038729+0000", 2024-09-11T03:52:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049247500000000003 2024-09-11T03:52:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038742+0000", 2024-09-11T03:52:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2169e-05 2024-09-11T03:52:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038747+0000", 2024-09-11T03:52:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6099999999999997e-06 2024-09-11T03:52:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038800+0000", 2024-09-11T03:52:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2920999999999997e-05 2024-09-11T03:52:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038911+0000", 2024-09-11T03:52:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011138599999999999 2024-09-11T03:52:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:52:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.039079+0000", 2024-09-11T03:52:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000167928 2024-09-11T03:52:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.040472+0000", 2024-09-11T03:52:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001392826 2024-09-11T03:52:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.040615+0000", 2024-09-11T03:52:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014295700000000001 2024-09-11T03:52:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.052594+0000", 2024-09-11T03:52:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011979001 2024-09-11T03:52:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.052675+0000", 2024-09-11T03:52:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0611999999999993e-05 2024-09-11T03:52:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.052732+0000", 2024-09-11T03:52:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7005000000000001e-05 2024-09-11T03:52:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7215 2.d 2:b6874adf:::smithi17557870-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e723)", 2024-09-11T03:52:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:01.038790+0000", 2024-09-11T03:52:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 224.94231673199999, 2024-09-11T03:52:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014304828, 2024-09-11T03:52:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:52:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7215 2024-09-11T03:52:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038790+0000", 2024-09-11T03:52:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038790+0000", 2024-09-11T03:52:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038790+0000", 2024-09-11T03:52:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9499999999999998e-07 2024-09-11T03:52:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038791+0000", 2024-09-11T03:52:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.544e-06 2024-09-11T03:52:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038800+0000", 2024-09-11T03:52:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4309999999999994e-06 2024-09-11T03:52:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.038803+0000", 2024-09-11T03:52:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3739999999999998e-06 2024-09-11T03:52:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.040394+0000", 2024-09-11T03:52:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001591189 2024-09-11T03:52:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:52:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.040462+0000", 2024-09-11T03:52:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8053999999999996e-05 2024-09-11T03:52:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.052896+0000", 2024-09-11T03:52:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012434031 2024-09-11T03:52:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.052994+0000", 2024-09-11T03:52:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7693000000000001e-05 2024-09-11T03:52:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.053094+0000", 2024-09-11T03:52:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010021700000000001 2024-09-11T03:52:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7230 2.d 2:b1c0f730:::smithi17557870-20:head [copy-from ver 956] snapc 141=[141,140,13f,13d,13c,139,138,137,136,135,133,12e,12d,12a,127,126,120,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-09-11T03:52:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:01.070035+0000", 2024-09-11T03:52:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 224.91107173, 2024-09-11T03:52:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014365849999999999, 2024-09-11T03:52:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7230 2024-09-11T03:52:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070035+0000", 2024-09-11T03:52:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070035+0000", 2024-09-11T03:52:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070036+0000", 2024-09-11T03:52:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6559999999999999e-06 2024-09-11T03:52:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070047+0000", 2024-09-11T03:52:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.047e-05 2024-09-11T03:52:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070063+0000", 2024-09-11T03:52:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6310000000000001e-05 2024-09-11T03:52:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070072+0000", 2024-09-11T03:52:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9819999999999997e-06 2024-09-11T03:52:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070145+0000", 2024-09-11T03:52:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3323000000000003e-05 2024-09-11T03:52:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.070237+0000", 2024-09-11T03:52:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1266000000000003e-05 2024-09-11T03:52:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:52:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.076752+0000", 2024-09-11T03:52:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0065155760000000004 2024-09-11T03:52:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.083798+0000", 2024-09-11T03:52:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070462249999999997 2024-09-11T03:52:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.083901+0000", 2024-09-11T03:52:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103115 2024-09-11T03:52:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.084281+0000", 2024-09-11T03:52:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037937799999999999 2024-09-11T03:52:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.084346+0000", 2024-09-11T03:52:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4817e-05 2024-09-11T03:52:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.084400+0000", 2024-09-11T03:52:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4731999999999997e-05 2024-09-11T03:52:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8290 2.3b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e811)", 2024-09-11T03:52:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:50:29.122470+0000", 2024-09-11T03:52:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 136.858636735, 2024-09-11T03:52:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92349315799999998, 2024-09-11T03:52:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-11T03:52:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8290 2024-09-11T03:52:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122470+0000", 2024-09-11T03:52:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122470+0000", 2024-09-11T03:52:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122471+0000", 2024-09-11T03:52:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.669e-06 2024-09-11T03:52:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122478+0000", 2024-09-11T03:52:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3550000000000001e-06 2024-09-11T03:52:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122493+0000", 2024-09-11T03:52:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5498999999999998e-05 2024-09-11T03:52:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122501+0000", 2024-09-11T03:52:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.61e-06 2024-09-11T03:52:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122568+0000", 2024-09-11T03:52:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7465999999999994e-05 2024-09-11T03:52:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-09-11T03:52:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122574+0000", 2024-09-11T03:52:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7139999999999998e-06 2024-09-11T03:52:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:30.045949+0000", 2024-09-11T03:52:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92337486199999996 2024-09-11T03:52:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:30.045963+0000", 2024-09-11T03:52:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3983e-05 2024-09-11T03:52:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8291 2.3b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e811)", 2024-09-11T03:52:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:50:29.122520+0000", 2024-09-11T03:52:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 136.858586457, 2024-09-11T03:52:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92350952100000006, 2024-09-11T03:52:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-11T03:52:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8291 2024-09-11T03:52:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122520+0000", 2024-09-11T03:52:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122520+0000", 2024-09-11T03:52:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122520+0000", 2024-09-11T03:52:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3300000000000002e-07 2024-09-11T03:52:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122523+0000", 2024-09-11T03:52:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.604e-06 2024-09-11T03:52:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122533+0000", 2024-09-11T03:52:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0338e-05 2024-09-11T03:52:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122538+0000", 2024-09-11T03:52:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2490000000000003e-06 2024-09-11T03:52:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122601+0000", 2024-09-11T03:52:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3789999999999997e-05 2024-09-11T03:52:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-09-11T03:52:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122606+0000", 2024-09-11T03:52:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5889999999999996e-06 2024-09-11T03:52:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:30.046018+0000", 2024-09-11T03:52:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92341183699999996 2024-09-11T03:52:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:30.046029+0000", 2024-09-11T03:52:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1581e-05 2024-09-11T03:52:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8292 2.3b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e811)", 2024-09-11T03:52:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:50:29.122558+0000", 2024-09-11T03:52:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 136.85854867500001, 2024-09-11T03:52:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92350303600000005, 2024-09-11T03:52:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-09-11T03:52:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8292 2024-09-11T03:52:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122558+0000", 2024-09-11T03:52:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122558+0000", 2024-09-11T03:52:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122558+0000", 2024-09-11T03:52:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0100000000000005e-07 2024-09-11T03:52:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122560+0000", 2024-09-11T03:52:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9999999999999999e-06 2024-09-11T03:52:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122567+0000", 2024-09-11T03:52:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6279999999999999e-06 2024-09-11T03:52:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122572+0000", 2024-09-11T03:52:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2619999999999998e-06 2024-09-11T03:52:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122621+0000", 2024-09-11T03:52:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9098999999999999e-05 2024-09-11T03:52:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-09-11T03:52:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:29.122624+0000", 2024-09-11T03:52:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9809999999999999e-06 2024-09-11T03:52:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:30.046053+0000", 2024-09-11T03:52:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92342833400000002 2024-09-11T03:52:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:30.046061+0000", 2024-09-11T03:52:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1310000000000006e-06 2024-09-11T03:52:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8471 2.2 2:43d79ac9:::smithi17557870-37:head [copy-from ver 1253] snapc 17c=[17c,17b,17a,179,178,177,176,175,174,173,172,171,170,16f,16e,16b,169,168,167,164,15f,15e,15d,15b,15a,157,156,155,14d,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e828)", 2024-09-11T03:52:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:50:46.051277+0000", 2024-09-11T03:52:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 119.929829192, 2024-09-11T03:52:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017235528, 2024-09-11T03:52:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8471 2024-09-11T03:52:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.051277+0000", 2024-09-11T03:52:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.051277+0000", 2024-09-11T03:52:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.051279+0000", 2024-09-11T03:52:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2989999999999999e-06 2024-09-11T03:52:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.051288+0000", 2024-09-11T03:52:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8880000000000003e-06 2024-09-11T03:52:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.051308+0000", 2024-09-11T03:52:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9921000000000001e-05 2024-09-11T03:52:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.051318+0000", 2024-09-11T03:52:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5559999999999996e-06 2024-09-11T03:52:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.051394+0000", 2024-09-11T03:52:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6532000000000005e-05 2024-09-11T03:52:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:52:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.051401+0000", 2024-09-11T03:52:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6050000000000003e-06 2024-09-11T03:52:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.053070+0000", 2024-09-11T03:52:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00166895 2024-09-11T03:52:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.053147+0000", 2024-09-11T03:52:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7088999999999993e-05 2024-09-11T03:52:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,6", 2024-09-11T03:52:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.060285+0000", 2024-09-11T03:52:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071378149999999996 2024-09-11T03:52:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.067528+0000", 2024-09-11T03:52:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0072434750000000001 2024-09-11T03:52:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.068107+0000", 2024-09-11T03:52:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057833199999999998 2024-09-11T03:52:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.068437+0000", 2024-09-11T03:52:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032997899999999999 2024-09-11T03:52:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.068479+0000", 2024-09-11T03:52:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2685999999999998e-05 2024-09-11T03:52:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:46.068513+0000", 2024-09-11T03:52:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3401000000000001e-05 2024-09-11T03:52:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8786 2.d 2:b1c0f730:::smithi17557870-20:head [copy-from ver 1006] snapc 18b=[18b,18a,189,188,187,186,185,184,183,181,180,17f,17e,17d,17c,17a,178,177,176,175,174,172,171,170,16e,16b,169,167,164,15f,15e,15b,157,14d,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e853)", 2024-09-11T03:52:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:51:11.053928+0000", 2024-09-11T03:52:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.927178440000006, 2024-09-11T03:52:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.051393094, 2024-09-11T03:52:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8786 2024-09-11T03:52:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053928+0000", 2024-09-11T03:52:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053928+0000", 2024-09-11T03:52:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053930+0000", 2024-09-11T03:52:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.683e-06 2024-09-11T03:52:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053936+0000", 2024-09-11T03:52:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7179999999999999e-06 2024-09-11T03:52:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053953+0000", 2024-09-11T03:52:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6673000000000001e-05 2024-09-11T03:52:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053963+0000", 2024-09-11T03:52:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0438999999999999e-05 2024-09-11T03:52:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.054063+0000", 2024-09-11T03:52:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010009399999999999 2024-09-11T03:52:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:52:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.054070+0000", 2024-09-11T03:52:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7360000000000004e-06 2024-09-11T03:52:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.057611+0000", 2024-09-11T03:52:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003540648 2024-09-11T03:52:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.057724+0000", 2024-09-11T03:52:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113058 2024-09-11T03:52:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,4", 2024-09-11T03:52:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.085043+0000", 2024-09-11T03:52:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027319098 2024-09-11T03:52:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.089007+0000", 2024-09-11T03:52:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039635900000000003 2024-09-11T03:52:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.089056+0000", 2024-09-11T03:52:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9663999999999997e-05 2024-09-11T03:52:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.105240+0000", 2024-09-11T03:52:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016183633999999999 2024-09-11T03:52:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.105285+0000", 2024-09-11T03:52:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5368999999999999e-05 2024-09-11T03:52:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.105321+0000", 2024-09-11T03:52:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5689999999999999e-05 2024-09-11T03:52:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8787 2.d 2:b1c0f730:::smithi17557870-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e853)", 2024-09-11T03:52:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:51:11.053988+0000", 2024-09-11T03:52:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.927118449999995, 2024-09-11T03:52:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034580952999999998, 2024-09-11T03:52:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:52:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8787 2024-09-11T03:52:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053988+0000", 2024-09-11T03:52:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053988+0000", 2024-09-11T03:52:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053988+0000", 2024-09-11T03:52:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4799999999999998e-07 2024-09-11T03:52:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.053991+0000", 2024-09-11T03:52:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9459999999999998e-06 2024-09-11T03:52:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.054004+0000", 2024-09-11T03:52:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2363e-05 2024-09-11T03:52:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.054010+0000", 2024-09-11T03:52:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5470000000000002e-06 2024-09-11T03:52:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.054122+0000", 2024-09-11T03:52:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111809 2024-09-11T03:52:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.054128+0000", 2024-09-11T03:52:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8159999999999999e-06 2024-09-11T03:52:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.057879+0000", 2024-09-11T03:52:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037510880000000001 2024-09-11T03:52:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.057924+0000", 2024-09-11T03:52:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4886999999999999e-05 2024-09-11T03:52:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.088395+0000", 2024-09-11T03:52:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.030471095 2024-09-11T03:52:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.088469+0000", 2024-09-11T03:52:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3815999999999999e-05 2024-09-11T03:52:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:11.088569+0000", 2024-09-11T03:52:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010003799999999999 2024-09-11T03:52:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:46.104 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:52:46.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:46.105 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:46.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:46.206 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:46.228 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:46.229 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:46.290 INFO:tasks.rados.rados.2.smithi175.stdout:3948: done (4 left) 2024-09-11T03:52:46.290 INFO:tasks.rados.rados.2.smithi175.stdout:3949: done (3 left) 2024-09-11T03:52:46.290 INFO:tasks.rados.rados.2.smithi175.stdout:3950: done (2 left) 2024-09-11T03:52:46.291 INFO:tasks.rados.rados.2.smithi175.stdout:3951: done (1 left) 2024-09-11T03:52:46.291 INFO:tasks.rados.rados.2.smithi175.stdout:3952: done (0 left) 2024-09-11T03:52:46.291 INFO:tasks.rados.rados.2.smithi175.stdout:3953: write oid 15 current snap is 439 2024-09-11T03:52:46.291 INFO:tasks.rados.rados.2.smithi175.stdout:3953: seq_num 1341 ranges {538187=587037,1538706=602473} 2024-09-11T03:52:46.295 INFO:tasks.rados.rados.2.smithi175.stdout:3953: writing smithi17557870-15 from 538187 to 1125224 tid 1 2024-09-11T03:52:46.301 INFO:tasks.rados.rados.2.smithi175.stdout:3953: writing smithi17557870-15 from 1538706 to 2141179 tid 2 2024-09-11T03:52:46.301 INFO:tasks.rados.rados.2.smithi175.stdout:3954: snap_create 2024-09-11T03:52:46.302 INFO:tasks.rados.rados.2.smithi175.stdout:3953: finishing write tid 1 to smithi17557870-15 2024-09-11T03:52:46.305 INFO:tasks.rados.rados.2.smithi175.stdout:3953: finishing write tid 2 to smithi17557870-15 2024-09-11T03:52:46.305 INFO:tasks.rados.rados.2.smithi175.stdout:3953: finishing write tid 3 to smithi17557870-15 2024-09-11T03:52:46.305 INFO:tasks.rados.rados.2.smithi175.stdout:3953: finishing write tid 4 to smithi17557870-15 2024-09-11T03:52:46.305 INFO:tasks.rados.rados.2.smithi175.stdout:3953: oid 15 updating version 0 to 1067 2024-09-11T03:52:46.305 INFO:tasks.rados.rados.2.smithi175.stdout:3953: oid 15 version 1067 is already newer than 1066 2024-09-11T03:52:46.305 INFO:tasks.rados.rados.2.smithi175.stdout:3953: oid 15 updating version 1067 to 1068 2024-09-11T03:52:46.305 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 15 v 1068 (ObjNum 1341 snap 439 seq_num 1341) dirty exists 2024-09-11T03:52:46.305 INFO:tasks.rados.rados.2.smithi175.stdout:3953: left oid 15 (ObjNum 1341 snap 439 seq_num 1341) 2024-09-11T03:52:46.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:46.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:46.352 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:46.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:46.388 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T03:52:46.388 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T03:52:46.453 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:46.474 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:46.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:46.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:46.577 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:46.598 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:46.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:46.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:46.629 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:46 smithi084 ceph-mon[27254]: pgmap v1575: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 733 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 70/1002 objects degraded (6.986%) 2024-09-11T03:52:46.629 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:46 smithi084 ceph-mon[27254]: Health check update: Degraded data redundancy: 70/1002 objects degraded (6.986%), 12 pgs degraded (PG_DEGRADED) 2024-09-11T03:52:46.629 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:46 smithi084 ceph-mon[27254]: osdmap e947: 12 total, 11 up, 11 in 2024-09-11T03:52:46.629 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:46 smithi084 ceph-mon[27254]: 1.15 scrub starts 2024-09-11T03:52:46.629 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:46 smithi084 ceph-mon[27254]: 1.15 scrub ok 2024-09-11T03:52:46.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:46 smithi121 ceph-mon[30729]: pgmap v1575: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 733 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 70/1002 objects degraded (6.986%) 2024-09-11T03:52:46.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:46 smithi121 ceph-mon[30729]: Health check update: Degraded data redundancy: 70/1002 objects degraded (6.986%), 12 pgs degraded (PG_DEGRADED) 2024-09-11T03:52:46.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:46 smithi121 ceph-mon[30729]: osdmap e947: 12 total, 11 up, 11 in 2024-09-11T03:52:46.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:46 smithi121 ceph-mon[30729]: 1.15 scrub starts 2024-09-11T03:52:46.633 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:46 smithi121 ceph-mon[30729]: 1.15 scrub ok 2024-09-11T03:52:46.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:46.722 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:46.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:46.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:46.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:46 smithi175 ceph-mon[30894]: pgmap v1575: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 733 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 70/1002 objects degraded (6.986%) 2024-09-11T03:52:46.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:46 smithi175 ceph-mon[30894]: Health check update: Degraded data redundancy: 70/1002 objects degraded (6.986%), 12 pgs degraded (PG_DEGRADED) 2024-09-11T03:52:46.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:46 smithi175 ceph-mon[30894]: osdmap e947: 12 total, 11 up, 11 in 2024-09-11T03:52:46.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:46 smithi175 ceph-mon[30894]: 1.15 scrub starts 2024-09-11T03:52:46.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:46 smithi175 ceph-mon[30894]: 1.15 scrub ok 2024-09-11T03:52:46.824 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:46.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-11T03:52:46.845 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:46.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:46.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:46.874 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:46.947 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:46.968 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:46.969 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:47.070 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:47.073 INFO:tasks.rados.rados.2.smithi175.stdout:3953: done (1 left) 2024-09-11T03:52:47.073 INFO:tasks.rados.rados.2.smithi175.stdout:3954: done (0 left) 2024-09-11T03:52:47.074 INFO:tasks.rados.rados.2.smithi175.stdout:3955: rollback oid 47 current snap is 440 2024-09-11T03:52:47.074 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 47 to 437 2024-09-11T03:52:47.074 INFO:tasks.rados.rados.2.smithi175.stdout:3956: write oid 12 current snap is 440 2024-09-11T03:52:47.074 INFO:tasks.rados.rados.2.smithi175.stdout:3956: seq_num 1342 ranges {554994=662286,1997188=557806} 2024-09-11T03:52:47.079 INFO:tasks.rados.rados.2.smithi175.stdout:3956: writing smithi17557870-12 from 554994 to 1217280 tid 1 2024-09-11T03:52:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:52:47.088 INFO:tasks.rados.rados.2.smithi175.stdout:3956: writing smithi17557870-12 from 1997188 to 2554994 tid 2 2024-09-11T03:52:47.088 INFO:tasks.rados.rados.2.smithi175.stdout:3957: rollback oid 8 current snap is 440 2024-09-11T03:52:47.088 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 8 to 385 2024-09-11T03:52:47.088 INFO:tasks.rados.rados.2.smithi175.stdout:3955: finishing rollback tid 1 to smithi17557870-47 2024-09-11T03:52:47.088 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 47 v 1413 (ObjNum 65 snap 0 seq_num 1745526736) dirty dne 2024-09-11T03:52:47.089 INFO:tasks.rados.rados.2.smithi175.stdout:3956: finishing write tid 1 to smithi17557870-12 2024-09-11T03:52:47.089 INFO:tasks.rados.rados.2.smithi175.stdout:3955: done (2 left) 2024-09-11T03:52:47.089 INFO:tasks.rados.rados.2.smithi175.stdout:3958: write (excl) oid 23 current snap is 440 2024-09-11T03:52:47.089 INFO:tasks.rados.rados.2.smithi175.stdout:3958: seq_num 1343 ranges {571801=737535,1939317=489789} 2024-09-11T03:52:47.095 INFO:tasks.rados.rados.2.smithi175.stdout:3958: writing smithi17557870-23 from 571801 to 1309336 tid 1 2024-09-11T03:52:47.095 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:47.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:47.097 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:47.102 INFO:tasks.rados.rados.2.smithi175.stdout:3958: writing smithi17557870-23 from 1939317 to 2429106 tid 2 2024-09-11T03:52:47.102 INFO:tasks.rados.rados.2.smithi175.stdout:3959: delete oid 13 current snap is 440 2024-09-11T03:52:47.102 INFO:tasks.rados.rados.2.smithi175.stdout:3957: finishing rollback tid 0 to smithi17557870-8 2024-09-11T03:52:47.102 INFO:tasks.rados.rados.2.smithi175.stdout:3956: finishing write tid 2 to smithi17557870-12 2024-09-11T03:52:47.102 INFO:tasks.rados.rados.2.smithi175.stdout:3956: finishing write tid 3 to smithi17557870-12 2024-09-11T03:52:47.102 INFO:tasks.rados.rados.2.smithi175.stdout:3956: finishing write tid 4 to smithi17557870-12 2024-09-11T03:52:47.103 INFO:tasks.rados.rados.2.smithi175.stdout:3956: oid 12 updating version 0 to 1663 2024-09-11T03:52:47.103 INFO:tasks.rados.rados.2.smithi175.stdout:3956: oid 12 updating version 1663 to 1664 2024-09-11T03:52:47.103 INFO:tasks.rados.rados.2.smithi175.stdout:3956: oid 12 version 1664 is already newer than 1662 2024-09-11T03:52:47.103 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 1664 (ObjNum 1342 snap 440 seq_num 1342) dirty exists 2024-09-11T03:52:47.103 INFO:tasks.rados.rados.2.smithi175.stdout:3956: left oid 12 (ObjNum 1342 snap 440 seq_num 1342) 2024-09-11T03:52:47.103 INFO:tasks.rados.rados.2.smithi175.stdout:3957: finishing rollback tid 1 to smithi17557870-8 2024-09-11T03:52:47.103 INFO:tasks.rados.rados.2.smithi175.stdout:3957: finishing rollback tid 2 to smithi17557870-8 2024-09-11T03:52:47.103 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 8 v 1283 (ObjNum 1147 snap 370 seq_num 1147) dirty exists 2024-09-11T03:52:47.104 INFO:tasks.rados.rados.2.smithi175.stdout:3956: done (3 left) 2024-09-11T03:52:47.104 INFO:tasks.rados.rados.2.smithi175.stdout:3957: done (2 left) 2024-09-11T03:52:47.104 INFO:tasks.rados.rados.2.smithi175.stdout:3959: done (1 left) 2024-09-11T03:52:47.104 INFO:tasks.rados.rados.2.smithi175.stdout:3960: write (excl) oid 42 current snap is 440 2024-09-11T03:52:47.104 INFO:tasks.rados.rados.2.smithi175.stdout:3960: seq_num 1345 ranges {605415=488033,1656281=493458} 2024-09-11T03:52:47.110 INFO:tasks.rados.rados.2.smithi175.stdout:3960: writing smithi17557870-42 from 605415 to 1093448 tid 1 2024-09-11T03:52:47.116 INFO:tasks.rados.rados.2.smithi175.stdout:3960: writing smithi17557870-42 from 1656281 to 2149739 tid 2 2024-09-11T03:52:47.116 INFO:tasks.rados.rados.2.smithi175.stdout:3961: delete oid 7 current snap is 440 2024-09-11T03:52:47.117 INFO:tasks.rados.rados.2.smithi175.stdout:3958: finishing write tid 1 to smithi17557870-23 2024-09-11T03:52:47.117 INFO:tasks.rados.rados.2.smithi175.stdout:3958: finishing write tid 2 to smithi17557870-23 2024-09-11T03:52:47.117 INFO:tasks.rados.rados.2.smithi175.stdout:3958: finishing write tid 3 to smithi17557870-23 2024-09-11T03:52:47.117 INFO:tasks.rados.rados.2.smithi175.stdout:3958: finishing write tid 4 to smithi17557870-23 2024-09-11T03:52:47.117 INFO:tasks.rados.rados.2.smithi175.stdout:3958: oid 23 updating version 0 to 1391 2024-09-11T03:52:47.117 INFO:tasks.rados.rados.2.smithi175.stdout:3958: oid 23 updating version 1391 to 1392 2024-09-11T03:52:47.117 INFO:tasks.rados.rados.2.smithi175.stdout:3958: oid 23 version 1392 is already newer than 1390 2024-09-11T03:52:47.117 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 23 v 1392 (ObjNum 1343 snap 440 seq_num 1343) dirty exists 2024-09-11T03:52:47.117 INFO:tasks.rados.rados.2.smithi175.stdout:3958: left oid 23 (ObjNum 1343 snap 440 seq_num 1343) 2024-09-11T03:52:47.117 INFO:tasks.rados.rados.2.smithi175.stdout:3960: finishing write tid 1 to smithi17557870-42 2024-09-11T03:52:47.120 INFO:tasks.rados.rados.2.smithi175.stdout:3960: finishing write tid 2 to smithi17557870-42 2024-09-11T03:52:47.120 INFO:tasks.rados.rados.2.smithi175.stdout:3960: finishing write tid 3 to smithi17557870-42 2024-09-11T03:52:47.120 INFO:tasks.rados.rados.2.smithi175.stdout:3960: finishing write tid 4 to smithi17557870-42 2024-09-11T03:52:47.121 INFO:tasks.rados.rados.2.smithi175.stdout:3960: oid 42 updating version 0 to 1079 2024-09-11T03:52:47.121 INFO:tasks.rados.rados.2.smithi175.stdout:3960: oid 42 updating version 1079 to 1081 2024-09-11T03:52:47.121 INFO:tasks.rados.rados.2.smithi175.stdout:3960: oid 42 version 1081 is already newer than 1080 2024-09-11T03:52:47.121 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 42 v 1081 (ObjNum 1345 snap 440 seq_num 1345) dirty exists 2024-09-11T03:52:47.121 INFO:tasks.rados.rados.2.smithi175.stdout:3960: left oid 42 (ObjNum 1345 snap 440 seq_num 1345) 2024-09-11T03:52:47.121 INFO:tasks.rados.rados.2.smithi175.stdout:3958: done (2 left) 2024-09-11T03:52:47.121 INFO:tasks.rados.rados.2.smithi175.stdout:3960: done (1 left) 2024-09-11T03:52:47.122 INFO:tasks.rados.rados.2.smithi175.stdout:3961: done (0 left) 2024-09-11T03:52:47.122 INFO:tasks.rados.rados.2.smithi175.stdout:3962: write (excl) oid 37 current snap is 440 2024-09-11T03:52:47.122 INFO:tasks.rados.rados.2.smithi175.stdout:3962: seq_num 1347 ranges {639029=754884,2173245=465784} 2024-09-11T03:52:47.129 INFO:tasks.rados.rados.2.smithi175.stdout:3962: writing smithi17557870-37 from 639029 to 1393913 tid 1 2024-09-11T03:52:47.136 INFO:tasks.rados.rados.2.smithi175.stdout:3962: writing smithi17557870-37 from 2173245 to 2639029 tid 2 2024-09-11T03:52:47.136 INFO:tasks.rados.rados.2.smithi175.stdout:3963: copy_from oid 5 from oid 36 current snap is 440 2024-09-11T03:52:47.136 INFO:tasks.rados.rados.2.smithi175.stdout:3962: finishing write tid 1 to smithi17557870-37 2024-09-11T03:52:47.136 INFO:tasks.rados.rados.2.smithi175.stdout:3964: write (excl) oid 2 current snap is 440 2024-09-11T03:52:47.136 INFO:tasks.rados.rados.2.smithi175.stdout:3964: seq_num 1348 ranges {655836=430133,1831727=757138} 2024-09-11T03:52:47.142 INFO:tasks.rados.rados.2.smithi175.stdout:3964: writing smithi17557870-2 from 655836 to 1085969 tid 1 2024-09-11T03:52:47.151 INFO:tasks.rados.rados.2.smithi175.stdout:3964: writing smithi17557870-2 from 1831727 to 2588865 tid 2 2024-09-11T03:52:47.151 INFO:tasks.rados.rados.2.smithi175.stdout:3965: write oid 48 current snap is 440 2024-09-11T03:52:47.151 INFO:tasks.rados.rados.2.smithi175.stdout:3962: finishing write tid 2 to smithi17557870-37 2024-09-11T03:52:47.151 INFO:tasks.rados.rados.2.smithi175.stdout:3962: finishing write tid 3 to smithi17557870-37 2024-09-11T03:52:47.151 INFO:tasks.rados.rados.2.smithi175.stdout:3962: finishing write tid 4 to smithi17557870-37 2024-09-11T03:52:47.151 INFO:tasks.rados.rados.2.smithi175.stdout:3962: oid 37 updating version 0 to 1265 2024-09-11T03:52:47.151 INFO:tasks.rados.rados.2.smithi175.stdout:3962: oid 37 version 1265 is already newer than 1264 2024-09-11T03:52:47.151 INFO:tasks.rados.rados.2.smithi175.stdout:3962: oid 37 version 1265 is already newer than 1263 2024-09-11T03:52:47.151 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 1265 (ObjNum 1347 snap 440 seq_num 1347) dirty exists 2024-09-11T03:52:47.151 INFO:tasks.rados.rados.2.smithi175.stdout:3962: left oid 37 (ObjNum 1347 snap 440 seq_num 1347) 2024-09-11T03:52:47.151 INFO:tasks.rados.rados.2.smithi175.stdout:3963: finishing copy_from racing read to smithi17557870-5 2024-09-11T03:52:47.152 INFO:tasks.rados.rados.2.smithi175.stdout:3965: seq_num 1349 ranges {672643=505382,1890209=500796} 2024-09-11T03:52:47.157 INFO:tasks.rados.rados.2.smithi175.stdout:3965: writing smithi17557870-48 from 672643 to 1178025 tid 1 2024-09-11T03:52:47.164 INFO:tasks.rados.rados.2.smithi175.stdout:3965: writing smithi17557870-48 from 1890209 to 2391005 tid 2 2024-09-11T03:52:47.164 INFO:tasks.rados.rados.2.smithi175.stdout:3962: done (3 left) 2024-09-11T03:52:47.165 INFO:tasks.rados.rados.2.smithi175.stdout:3966: rollback oid 19 current snap is 440 2024-09-11T03:52:47.165 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 19 to 371 2024-09-11T03:52:47.165 INFO:tasks.rados.rados.2.smithi175.stdout:3964: finishing write tid 1 to smithi17557870-2 2024-09-11T03:52:47.165 INFO:tasks.rados.rados.2.smithi175.stdout:3967: rollback oid 38 current snap is 440 2024-09-11T03:52:47.165 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 38 to 430 2024-09-11T03:52:47.165 INFO:tasks.rados.rados.2.smithi175.stdout:3963: finishing copy_from to smithi17557870-5 2024-09-11T03:52:47.165 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 1391 (ObjNum 1086 snap 353 seq_num 1086) dirty exists 2024-09-11T03:52:47.165 INFO:tasks.rados.rados.2.smithi175.stdout:3963: done (4 left) 2024-09-11T03:52:47.165 INFO:tasks.rados.rados.2.smithi175.stdout:3968: delete oid 34 current snap is 440 2024-09-11T03:52:47.165 INFO:tasks.rados.rados.2.smithi175.stdout:3967: finishing rollback tid 1 to smithi17557870-38 2024-09-11T03:52:47.166 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 38 v 1614 (ObjNum 33 snap 0 seq_num 1713067256) dirty dne 2024-09-11T03:52:47.169 INFO:tasks.rados.rados.2.smithi175.stdout:3965: finishing write tid 1 to smithi17557870-48 2024-09-11T03:52:47.169 INFO:tasks.rados.rados.2.smithi175.stdout:3967: done (4 left) 2024-09-11T03:52:47.169 INFO:tasks.rados.rados.2.smithi175.stdout:3968: done (3 left) 2024-09-11T03:52:47.170 INFO:tasks.rados.rados.2.smithi175.stdout:3969: copy_from oid 11 from oid 9 current snap is 440 2024-09-11T03:52:47.170 INFO:tasks.rados.rados.2.smithi175.stdout:3966: finishing rollback tid 1 to smithi17557870-19 2024-09-11T03:52:47.170 INFO:tasks.rados.rados.2.smithi175.stdout:3970: copy_from oid 39 from oid 37 current snap is 440 2024-09-11T03:52:47.170 INFO:tasks.rados.rados.2.smithi175.stdout:3971: rollback oid 3 current snap is 440 2024-09-11T03:52:47.170 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 3 to 422 2024-09-11T03:52:47.170 INFO:tasks.rados.rados.2.smithi175.stdout:3972: read oid 47 snap 426 2024-09-11T03:52:47.170 INFO:tasks.rados.rados.2.smithi175.stdout:3972: expect (ObjNum 1300 snap 426 seq_num 1300) 2024-09-11T03:52:47.170 INFO:tasks.rados.rados.2.smithi175.stdout:3973: write (excl) oid 28 current snap is 440 2024-09-11T03:52:47.170 INFO:tasks.rados.rados.2.smithi175.stdout:3973: seq_num 1351 ranges {706257=655880,1890820=504465} 2024-09-11T03:52:47.176 INFO:tasks.rados.rados.2.smithi175.stdout:3973: writing smithi17557870-28 from 706257 to 1362137 tid 1 2024-09-11T03:52:47.182 INFO:tasks.rados.rados.2.smithi175.stdout:3973: writing smithi17557870-28 from 1890820 to 2395285 tid 2 2024-09-11T03:52:47.182 INFO:tasks.rados.rados.2.smithi175.stdout:3974: delete oid 23 current snap is 440 2024-09-11T03:52:47.183 INFO:tasks.rados.rados.2.smithi175.stdout:3966: finishing rollback tid 2 to smithi17557870-19 2024-09-11T03:52:47.183 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 19 v 1537 (ObjNum 1079 snap 352 seq_num 1079) dirty exists 2024-09-11T03:52:47.183 INFO:tasks.rados.rados.2.smithi175.stdout:3971: finishing rollback tid 0 to smithi17557870-3 2024-09-11T03:52:47.183 INFO:tasks.rados.rados.2.smithi175.stdout:3971: finishing rollback tid 1 to smithi17557870-3 2024-09-11T03:52:47.183 INFO:tasks.rados.rados.2.smithi175.stdout:3969: finishing copy_from racing read to smithi17557870-11 2024-09-11T03:52:47.183 INFO:tasks.rados.rados.2.smithi175.stdout:3970: finishing copy_from racing read to smithi17557870-39 2024-09-11T03:52:47.184 INFO:tasks.rados.rados.2.smithi175.stdout:3966: done (8 left) 2024-09-11T03:52:47.184 INFO:tasks.rados.rados.2.smithi175.stdout:3974: done (7 left) 2024-09-11T03:52:47.184 INFO:tasks.rados.rados.2.smithi175.stdout:3975: copy_from oid 20 from oid 26 current snap is 440 2024-09-11T03:52:47.184 INFO:tasks.rados.rados.2.smithi175.stdout:3976: snap_create 2024-09-11T03:52:47.186 INFO:tasks.rados.rados.2.smithi175.stdout:3975: finishing copy_from to smithi17557870-20 2024-09-11T03:52:47.186 INFO:tasks.rados.rados.2.smithi175.stdout:3975: got expected ENOENT (src dne) 2024-09-11T03:52:47.186 INFO:tasks.rados.rados.2.smithi175.stdout:3975: finishing copy_from racing read to smithi17557870-20 2024-09-11T03:52:47.188 INFO:tasks.rados.rados.2.smithi175.stdout:3965: finishing write tid 2 to smithi17557870-48 2024-09-11T03:52:47.188 INFO:tasks.rados.rados.2.smithi175.stdout:3965: finishing write tid 3 to smithi17557870-48 2024-09-11T03:52:47.188 INFO:tasks.rados.rados.2.smithi175.stdout:3965: finishing write tid 4 to smithi17557870-48 2024-09-11T03:52:47.188 INFO:tasks.rados.rados.2.smithi175.stdout:3965: oid 48 updating version 0 to 1303 2024-09-11T03:52:47.188 INFO:tasks.rados.rados.2.smithi175.stdout:3965: oid 48 version 1303 is already newer than 1302 2024-09-11T03:52:47.188 INFO:tasks.rados.rados.2.smithi175.stdout:3965: oid 48 updating version 1303 to 1304 2024-09-11T03:52:47.188 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 48 v 1304 (ObjNum 1349 snap 440 seq_num 1349) dirty exists 2024-09-11T03:52:47.188 INFO:tasks.rados.rados.2.smithi175.stdout:3965: left oid 48 (ObjNum 1349 snap 440 seq_num 1349) 2024-09-11T03:52:47.189 INFO:tasks.rados.rados.2.smithi175.stdout:3973: finishing write tid 1 to smithi17557870-28 2024-09-11T03:52:47.190 INFO:tasks.rados.rados.2.smithi175.stdout:3973: finishing write tid 2 to smithi17557870-28 2024-09-11T03:52:47.190 INFO:tasks.rados.rados.2.smithi175.stdout:3973: finishing write tid 3 to smithi17557870-28 2024-09-11T03:52:47.190 INFO:tasks.rados.rados.2.smithi175.stdout:3973: finishing write tid 4 to smithi17557870-28 2024-09-11T03:52:47.190 INFO:tasks.rados.rados.2.smithi175.stdout:3973: oid 28 updating version 0 to 1664 2024-09-11T03:52:47.190 INFO:tasks.rados.rados.2.smithi175.stdout:3973: oid 28 updating version 1664 to 1665 2024-09-11T03:52:47.190 INFO:tasks.rados.rados.2.smithi175.stdout:3973: oid 28 updating version 1665 to 1666 2024-09-11T03:52:47.190 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 28 v 1666 (ObjNum 1351 snap 440 seq_num 1351) dirty exists 2024-09-11T03:52:47.190 INFO:tasks.rados.rados.2.smithi175.stdout:3973: left oid 28 (ObjNum 1351 snap 440 seq_num 1351) 2024-09-11T03:52:47.191 INFO:tasks.rados.rados.2.smithi175.stdout:3970: finishing copy_from to smithi17557870-39 2024-09-11T03:52:47.191 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 1265 (ObjNum 1347 snap 440 seq_num 1347) dirty exists 2024-09-11T03:52:47.191 INFO:tasks.rados.rados.2.smithi175.stdout:3972: expect (ObjNum 1300 snap 426 seq_num 1300) 2024-09-11T03:52:47.198 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:47.200 INFO:teuthology.orchestra.run.smithi084.stdout:2.c 2024-09-11T03:52:47.201 INFO:teuthology.orchestra.run.smithi084.stdout:2.b 2024-09-11T03:52:47.201 INFO:teuthology.orchestra.run.smithi084.stdout:2.8 2024-09-11T03:52:47.201 INFO:teuthology.orchestra.run.smithi084.stdout:2.1f 2024-09-11T03:52:47.201 INFO:teuthology.orchestra.run.smithi084.stdout:2.1 2024-09-11T03:52:47.201 INFO:teuthology.orchestra.run.smithi084.stdout:1.a 2024-09-11T03:52:47.201 INFO:teuthology.orchestra.run.smithi084.stdout:2.5 2024-09-11T03:52:47.201 INFO:teuthology.orchestra.run.smithi084.stdout:2.19 2024-09-11T03:52:47.201 INFO:teuthology.orchestra.run.smithi084.stdout:2.5c 2024-09-11T03:52:47.201 INFO:teuthology.orchestra.run.smithi084.stdout:2.5f 2024-09-11T03:52:47.201 INFO:teuthology.orchestra.run.smithi084.stdout:1.1b 2024-09-11T03:52:47.201 INFO:teuthology.orchestra.run.smithi084.stdout:1.1 2024-09-11T03:52:47.201 INFO:teuthology.orchestra.run.smithi084.stdout:1.11 2024-09-11T03:52:47.202 INFO:teuthology.orchestra.run.smithi084.stdout:1.b 2024-09-11T03:52:47.202 INFO:teuthology.orchestra.run.smithi084.stdout:1.19 2024-09-11T03:52:47.202 INFO:teuthology.orchestra.run.smithi084.stdout:1.c 2024-09-11T03:52:47.202 INFO:teuthology.orchestra.run.smithi084.stdout:2.4c 2024-09-11T03:52:47.202 INFO:teuthology.orchestra.run.smithi084.stdout:2.48 2024-09-11T03:52:47.202 INFO:teuthology.orchestra.run.smithi084.stdout:1.2 2024-09-11T03:52:47.202 INFO:teuthology.orchestra.run.smithi084.stdout:2.4b 2024-09-11T03:52:47.202 INFO:teuthology.orchestra.run.smithi084.stdout:2.4 2024-09-11T03:52:47.202 INFO:teuthology.orchestra.run.smithi084.stdout:2.2c 2024-09-11T03:52:47.202 INFO:teuthology.orchestra.run.smithi084.stdout:2.2e 2024-09-11T03:52:47.202 INFO:teuthology.orchestra.run.smithi084.stdout:2.31 2024-09-11T03:52:47.202 INFO:teuthology.orchestra.run.smithi084.stdout:2.1e 2024-09-11T03:52:47.202 INFO:teuthology.orchestra.run.smithi084.stdout:2.42 2024-09-11T03:52:47.202 INFO:teuthology.orchestra.run.smithi084.stdout:2.9 2024-09-11T03:52:47.203 INFO:teuthology.orchestra.run.smithi084.stdout:2.47 2024-09-11T03:52:47.203 INFO:teuthology.orchestra.run.smithi084.stdout:1.16 2024-09-11T03:52:47.203 INFO:teuthology.orchestra.run.smithi084.stdout:2.52 2024-09-11T03:52:47.203 INFO:teuthology.orchestra.run.smithi084.stdout:2.56 2024-09-11T03:52:47.203 INFO:teuthology.orchestra.run.smithi084.stdout:2.59 2024-09-11T03:52:47.203 INFO:teuthology.orchestra.run.smithi084.stdout:2.5a 2024-09-11T03:52:47.203 INFO:teuthology.orchestra.run.smithi084.stdout:2.5e 2024-09-11T03:52:47.219 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:47.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:47.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:47.321 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:47.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:47.342 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:47.342 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:47.365 INFO:tasks.rados.rados.2.smithi175.stdout:3971: finishing rollback tid 2 to smithi17557870-3 2024-09-11T03:52:47.365 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 3 v 1400 (ObjNum 1225 snap 400 seq_num 1225) dirty exists 2024-09-11T03:52:47.365 INFO:tasks.rados.rados.2.smithi175.stdout:3964: finishing write tid 2 to smithi17557870-2 2024-09-11T03:52:47.365 INFO:tasks.rados.rados.2.smithi175.stdout:3964: finishing write tid 3 to smithi17557870-2 2024-09-11T03:52:47.365 INFO:tasks.rados.rados.2.smithi175.stdout:3964: finishing write tid 4 to smithi17557870-2 2024-09-11T03:52:47.365 INFO:tasks.rados.rados.2.smithi175.stdout:3964: oid 2 updating version 0 to 1387 2024-09-11T03:52:47.366 INFO:tasks.rados.rados.2.smithi175.stdout:3964: oid 2 updating version 1387 to 1388 2024-09-11T03:52:47.366 INFO:tasks.rados.rados.2.smithi175.stdout:3964: oid 2 version 1388 is already newer than 1386 2024-09-11T03:52:47.366 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 2 v 1388 (ObjNum 1348 snap 440 seq_num 1348) dirty exists 2024-09-11T03:52:47.366 INFO:tasks.rados.rados.2.smithi175.stdout:3964: left oid 2 (ObjNum 1348 snap 440 seq_num 1348) 2024-09-11T03:52:47.366 INFO:tasks.rados.rados.2.smithi175.stdout:3969: finishing copy_from to smithi17557870-11 2024-09-11T03:52:47.366 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 11 v 1195 (ObjNum 1260 snap 414 seq_num 1260) dirty exists 2024-09-11T03:52:47.444 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:47.465 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:47.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:47.466 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:47.544 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:47 smithi121 ceph-mon[30729]: osdmap e948: 12 total, 11 up, 11 in 2024-09-11T03:52:47.544 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:47 smithi121 ceph-mon[30729]: 2.44 scrub starts 2024-09-11T03:52:47.544 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:47 smithi121 ceph-mon[30729]: 2.44 scrub ok 2024-09-11T03:52:47.544 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:47 smithi121 ceph-mon[30729]: osdmap e949: 12 total, 11 up, 11 in 2024-09-11T03:52:47.567 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:47.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:47.591 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:47.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:47.643 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:52:47.643 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:52:47.644 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-09-11T03:52:47.644 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:52:47.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:47 smithi084 ceph-mon[27254]: osdmap e948: 12 total, 11 up, 11 in 2024-09-11T03:52:47.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:47 smithi084 ceph-mon[27254]: 2.44 scrub starts 2024-09-11T03:52:47.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:47 smithi084 ceph-mon[27254]: 2.44 scrub ok 2024-09-11T03:52:47.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:47 smithi084 ceph-mon[27254]: osdmap e949: 12 total, 11 up, 11 in 2024-09-11T03:52:47.693 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:47.714 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:47.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:47.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:47 smithi175 ceph-mon[30894]: osdmap e948: 12 total, 11 up, 11 in 2024-09-11T03:52:47.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:47 smithi175 ceph-mon[30894]: 2.44 scrub starts 2024-09-11T03:52:47.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:47 smithi175 ceph-mon[30894]: 2.44 scrub ok 2024-09-11T03:52:47.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:47 smithi175 ceph-mon[30894]: osdmap e949: 12 total, 11 up, 11 in 2024-09-11T03:52:47.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:47.839 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:47.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:47.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:47.942 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:47.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:47.965 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:47.965 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:48.066 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:48.087 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:48.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:48.088 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:48.145 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -n osd.0 --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op export --pgid 2.19 --file /var/log/ceph/exp.2.19.0 2024-09-11T03:52:48.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:48.211 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:48.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:48.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:48.297 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-11T03:52:48.301 INFO:tasks.rados.rados.2.smithi175.stdout:3964: done (8 left) 2024-09-11T03:52:48.301 INFO:tasks.rados.rados.2.smithi175.stdout:3965: done (7 left) 2024-09-11T03:52:48.301 INFO:tasks.rados.rados.2.smithi175.stdout:3969: done (6 left) 2024-09-11T03:52:48.302 INFO:tasks.rados.rados.2.smithi175.stdout:3970: done (5 left) 2024-09-11T03:52:48.302 INFO:tasks.rados.rados.2.smithi175.stdout:3971: done (4 left) 2024-09-11T03:52:48.302 INFO:tasks.rados.rados.2.smithi175.stdout:3972: done (3 left) 2024-09-11T03:52:48.302 INFO:tasks.rados.rados.2.smithi175.stdout:3973: done (2 left) 2024-09-11T03:52:48.302 INFO:tasks.rados.rados.2.smithi175.stdout:3975: done (1 left) 2024-09-11T03:52:48.302 INFO:tasks.rados.rados.2.smithi175.stdout:3976: done (0 left) 2024-09-11T03:52:48.302 INFO:tasks.rados.rados.2.smithi175.stdout:3977: snap_remove snap 371 2024-09-11T03:52:48.312 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:48.334 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:48.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:48.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:48.389 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:52:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:48 smithi084 ceph-mon[27254]: pgmap v1578: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 727 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 5.7 MiB/s rd, 3.2 MiB/s wr, 15 op/s; 71/1002 objects degraded (7.086%) 2024-09-11T03:52:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:48 smithi084 ceph-mon[27254]: 2.3 scrub starts 2024-09-11T03:52:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:48 smithi084 ceph-mon[27254]: 2.3 scrub ok 2024-09-11T03:52:48.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:48 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/816356798' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:52:48.437 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:48.459 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:48.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:48.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:48.515 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/osd.0/config 2024-09-11T03:52:48.550 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:48 smithi121 ceph-mon[30729]: pgmap v1578: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 727 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 5.7 MiB/s rd, 3.2 MiB/s wr, 15 op/s; 71/1002 objects degraded (7.086%) 2024-09-11T03:52:48.550 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:48 smithi121 ceph-mon[30729]: 2.3 scrub starts 2024-09-11T03:52:48.550 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:48 smithi121 ceph-mon[30729]: 2.3 scrub ok 2024-09-11T03:52:48.550 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:48 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/816356798' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:52:48.552 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-11T03:52:48.561 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:48.616 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:48.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:48.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:48.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:48.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:48 smithi175 ceph-mon[30894]: pgmap v1578: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 727 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 5.7 MiB/s rd, 3.2 MiB/s wr, 15 op/s; 71/1002 objects degraded (7.086%) 2024-09-11T03:52:48.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:48 smithi175 ceph-mon[30894]: 2.3 scrub starts 2024-09-11T03:52:48.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:48 smithi175 ceph-mon[30894]: 2.3 scrub ok 2024-09-11T03:52:48.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:48 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/816356798' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-11T03:52:48.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:48.742 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:48.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:48.794 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:52:48.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:48.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:48.866 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:48.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:48.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:48.989 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:48.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:48.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:49.092 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:49.119 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:49.119 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:49.220 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:49.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:49.246 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:49.247 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:49.296 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:52:49.296 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-09-11T03:52:49.297 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-09-11T03:52:49.297 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-09-11T03:52:49.297 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:52:49.347 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:49.372 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:49.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:49.373 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:49.474 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:49.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:49.499 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:49.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:49.505 INFO:tasks.rados.rados.2.smithi175.stdout:3977: done (0 left) 2024-09-11T03:52:49.505 INFO:tasks.rados.rados.2.smithi175.stdout:3978: write oid 24 current snap is 441 2024-09-11T03:52:49.505 INFO:tasks.rados.rados.2.smithi175.stdout:3978: seq_num 1353 ranges {739871=406378,1607784=508134} 2024-09-11T03:52:49.511 INFO:tasks.rados.rados.2.smithi175.stdout:3978: writing smithi17557870-24 from 739871 to 1146249 tid 1 2024-09-11T03:52:49.518 INFO:tasks.rados.rados.2.smithi175.stdout:3978: writing smithi17557870-24 from 1607784 to 2115918 tid 2 2024-09-11T03:52:49.519 INFO:tasks.rados.rados.2.smithi175.stdout:3979: write oid 39 current snap is 441 2024-09-11T03:52:49.519 INFO:tasks.rados.rados.2.smithi175.stdout:3979: seq_num 1354 ranges {756678=597980,1782619=768145} 2024-09-11T03:52:49.526 INFO:tasks.rados.rados.2.smithi175.stdout:3979: writing smithi17557870-39 from 756678 to 1354658 tid 1 2024-09-11T03:52:49.533 INFO:tasks.rados.rados.2.smithi175.stdout:3979: writing smithi17557870-39 from 1782619 to 2550764 tid 2 2024-09-11T03:52:49.533 INFO:tasks.rados.rados.2.smithi175.stdout:3980: snap_remove snap 432 2024-09-11T03:52:49.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:49.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:49.627 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:49.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:49.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:49 smithi084 ceph-mon[27254]: 1.18 scrub starts 2024-09-11T03:52:49.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:49 smithi084 ceph-mon[27254]: 1.18 scrub ok 2024-09-11T03:52:49.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:49 smithi084 ceph-mon[27254]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:49.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/816356798' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:52:49.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:49 smithi084 ceph-mon[27254]: osdmap e950: 12 total, 11 up, 11 in 2024-09-11T03:52:49.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:49 smithi084 ceph-mon[27254]: 2.1c scrub starts 2024-09-11T03:52:49.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:49 smithi084 ceph-mon[27254]: 2.1c scrub ok 2024-09-11T03:52:49.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:49 smithi121 ceph-mon[30729]: 1.18 scrub starts 2024-09-11T03:52:49.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:49 smithi121 ceph-mon[30729]: 1.18 scrub ok 2024-09-11T03:52:49.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:49 smithi121 ceph-mon[30729]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:49.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/816356798' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:52:49.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:49 smithi121 ceph-mon[30729]: osdmap e950: 12 total, 11 up, 11 in 2024-09-11T03:52:49.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:49 smithi121 ceph-mon[30729]: 2.1c scrub starts 2024-09-11T03:52:49.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:49 smithi121 ceph-mon[30729]: 2.1c scrub ok 2024-09-11T03:52:49.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:49 smithi175 ceph-mon[30894]: 1.18 scrub starts 2024-09-11T03:52:49.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:49 smithi175 ceph-mon[30894]: 1.18 scrub ok 2024-09-11T03:52:49.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:49 smithi175 ceph-mon[30894]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:49.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/816356798' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-11T03:52:49.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:49 smithi175 ceph-mon[30894]: osdmap e950: 12 total, 11 up, 11 in 2024-09-11T03:52:49.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:49 smithi175 ceph-mon[30894]: 2.1c scrub starts 2024-09-11T03:52:49.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:49 smithi175 ceph-mon[30894]: 2.1c scrub ok 2024-09-11T03:52:49.728 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:49.750 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:49.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:49.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:49.873 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:49.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:49.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:49.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:49.997 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:49.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:49.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:50.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-11T03:52:50.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:50.123 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:50.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:50.126 INFO:teuthology.orchestra.run.smithi084.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-11T03:52:50.225 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:50.251 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:50.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:50.252 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:50.295 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:52:50.353 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:50.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:50.376 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:50.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:50.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:50.498 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:50.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:50.507 INFO:tasks.rados.rados.2.smithi175.stdout:3978: finishing write tid 1 to smithi17557870-24 2024-09-11T03:52:50.508 INFO:tasks.rados.rados.2.smithi175.stdout:3978: finishing write tid 2 to smithi17557870-24 2024-09-11T03:52:50.508 INFO:tasks.rados.rados.2.smithi175.stdout:3978: finishing write tid 3 to smithi17557870-24 2024-09-11T03:52:50.508 INFO:tasks.rados.rados.2.smithi175.stdout:3978: finishing write tid 4 to smithi17557870-24 2024-09-11T03:52:50.508 INFO:tasks.rados.rados.2.smithi175.stdout:3978: oid 24 updating version 0 to 1143 2024-09-11T03:52:50.508 INFO:tasks.rados.rados.2.smithi175.stdout:3978: oid 24 version 1143 is already newer than 1142 2024-09-11T03:52:50.508 INFO:tasks.rados.rados.2.smithi175.stdout:3978: oid 24 updating version 1143 to 1144 2024-09-11T03:52:50.508 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 24 v 1144 (ObjNum 1353 snap 441 seq_num 1353) dirty exists 2024-09-11T03:52:50.508 INFO:tasks.rados.rados.2.smithi175.stdout:3978: left oid 24 (ObjNum 1353 snap 441 seq_num 1353) 2024-09-11T03:52:50.508 INFO:tasks.rados.rados.2.smithi175.stdout:3979: finishing write tid 1 to smithi17557870-39 2024-09-11T03:52:50.508 INFO:tasks.rados.rados.2.smithi175.stdout:3979: finishing write tid 2 to smithi17557870-39 2024-09-11T03:52:50.509 INFO:tasks.rados.rados.2.smithi175.stdout:3979: finishing write tid 3 to smithi17557870-39 2024-09-11T03:52:50.509 INFO:tasks.rados.rados.2.smithi175.stdout:3979: finishing write tid 4 to smithi17557870-39 2024-09-11T03:52:50.509 INFO:tasks.rados.rados.2.smithi175.stdout:3979: oid 39 updating version 0 to 1266 2024-09-11T03:52:50.509 INFO:tasks.rados.rados.2.smithi175.stdout:3979: oid 39 updating version 1266 to 1268 2024-09-11T03:52:50.509 INFO:tasks.rados.rados.2.smithi175.stdout:3979: oid 39 version 1268 is already newer than 1267 2024-09-11T03:52:50.509 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 39 v 1268 (ObjNum 1354 snap 441 seq_num 1354) dirty exists 2024-09-11T03:52:50.509 INFO:tasks.rados.rados.2.smithi175.stdout:3979: left oid 39 (ObjNum 1354 snap 441 seq_num 1354) 2024-09-11T03:52:50.509 INFO:tasks.rados.rados.2.smithi175.stdout:3978: done (2 left) 2024-09-11T03:52:50.509 INFO:tasks.rados.rados.2.smithi175.stdout:3979: done (1 left) 2024-09-11T03:52:50.509 INFO:tasks.rados.rados.2.smithi175.stdout:3980: done (0 left) 2024-09-11T03:52:50.509 INFO:tasks.rados.rados.2.smithi175.stdout:3981: write (excl) oid 36 current snap is 441 2024-09-11T03:52:50.509 INFO:tasks.rados.rados.2.smithi175.stdout:3981: seq_num 1355 ranges {773485=673229,2124748=511803} 2024-09-11T03:52:50.516 INFO:tasks.rados.rados.2.smithi175.stdout:3981: writing smithi17557870-36 from 773485 to 1446714 tid 1 2024-09-11T03:52:50.524 INFO:tasks.rados.rados.2.smithi175.stdout:3981: writing smithi17557870-36 from 2124748 to 2636551 tid 2 2024-09-11T03:52:50.524 INFO:tasks.rados.rados.2.smithi175.stdout:3982: delete oid 39 current snap is 441 2024-09-11T03:52:50.524 INFO:tasks.rados.rados.2.smithi175.stdout:3981: finishing write tid 1 to smithi17557870-36 2024-09-11T03:52:50.527 INFO:tasks.rados.rados.2.smithi175.stdout:3982: done (1 left) 2024-09-11T03:52:50.528 INFO:tasks.rados.rados.2.smithi175.stdout:3983: rollback oid 46 current snap is 441 2024-09-11T03:52:50.528 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 46 to 411 2024-09-11T03:52:50.528 INFO:tasks.rados.rados.2.smithi175.stdout:3984: read oid 19 snap -1 2024-09-11T03:52:50.528 INFO:tasks.rados.rados.2.smithi175.stdout:3984: expect (ObjNum 1079 snap 352 seq_num 1079) 2024-09-11T03:52:50.528 INFO:tasks.rados.rados.2.smithi175.stdout:3985: rollback oid 37 current snap is 441 2024-09-11T03:52:50.528 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 37 to 422 2024-09-11T03:52:50.528 INFO:tasks.rados.rados.2.smithi175.stdout:3986: rollback oid 30 current snap is 441 2024-09-11T03:52:50.528 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 30 to 438 2024-09-11T03:52:50.528 INFO:tasks.rados.rados.2.smithi175.stdout:3987: snap_create 2024-09-11T03:52:50.528 INFO:tasks.rados.rados.2.smithi175.stdout:3981: finishing write tid 2 to smithi17557870-36 2024-09-11T03:52:50.529 INFO:tasks.rados.rados.2.smithi175.stdout:3981: finishing write tid 3 to smithi17557870-36 2024-09-11T03:52:50.529 INFO:tasks.rados.rados.2.smithi175.stdout:3981: finishing write tid 4 to smithi17557870-36 2024-09-11T03:52:50.529 INFO:tasks.rados.rados.2.smithi175.stdout:3981: oid 36 updating version 0 to 1392 2024-09-11T03:52:50.529 INFO:tasks.rados.rados.2.smithi175.stdout:3981: oid 36 updating version 1392 to 1394 2024-09-11T03:52:50.529 INFO:tasks.rados.rados.2.smithi175.stdout:3981: oid 36 version 1394 is already newer than 1393 2024-09-11T03:52:50.529 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 36 v 1394 (ObjNum 1355 snap 441 seq_num 1355) dirty exists 2024-09-11T03:52:50.529 INFO:tasks.rados.rados.2.smithi175.stdout:3981: left oid 36 (ObjNum 1355 snap 441 seq_num 1355) 2024-09-11T03:52:50.530 INFO:tasks.rados.rados.2.smithi175.stdout:3983: finishing rollback tid 0 to smithi17557870-46 2024-09-11T03:52:50.532 INFO:tasks.rados.rados.2.smithi175.stdout:3985: finishing rollback tid 0 to smithi17557870-37 2024-09-11T03:52:50.532 INFO:tasks.rados.rados.2.smithi175.stdout:3983: finishing rollback tid 1 to smithi17557870-46 2024-09-11T03:52:50.532 INFO:tasks.rados.rados.2.smithi175.stdout:3985: finishing rollback tid 1 to smithi17557870-37 2024-09-11T03:52:50.532 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 1267 (ObjNum 129 snap 0 seq_num 1746014632) dirty dne 2024-09-11T03:52:50.532 INFO:tasks.rados.rados.2.smithi175.stdout:3986: finishing rollback tid 0 to smithi17557870-30 2024-09-11T03:52:50.532 INFO:tasks.rados.rados.2.smithi175.stdout:3986: finishing rollback tid 1 to smithi17557870-30 2024-09-11T03:52:50.532 INFO:tasks.rados.rados.2.smithi175.stdout:3983: finishing rollback tid 2 to smithi17557870-46 2024-09-11T03:52:50.532 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 46 v 985 (ObjNum 1243 snap 408 seq_num 1243) dirty exists 2024-09-11T03:52:50.532 INFO:tasks.rados.rados.2.smithi175.stdout:3986: finishing rollback tid 2 to smithi17557870-30 2024-09-11T03:52:50.532 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 30 v 1287 (ObjNum 1318 snap 430 seq_num 1318) dirty exists 2024-09-11T03:52:50.539 INFO:tasks.rados.rados.2.smithi175.stdout:3984: expect (ObjNum 1079 snap 352 seq_num 1079) 2024-09-11T03:52:50.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:50.620 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:50.622 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:50.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:50.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:50.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:50 smithi084 ceph-mon[27254]: pgmap v1581: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 714 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 14 MiB/s rd, 4.1 MiB/s wr, 28 op/s; 72/996 objects degraded (7.229%) 2024-09-11T03:52:50.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:50 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:52:50.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:50 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:52:50.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:50 smithi084 ceph-mon[27254]: osdmap e951: 12 total, 11 up, 11 in 2024-09-11T03:52:50.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:50 smithi121 ceph-mon[30729]: pgmap v1581: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 714 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 14 MiB/s rd, 4.1 MiB/s wr, 28 op/s; 72/996 objects degraded (7.229%) 2024-09-11T03:52:50.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:50 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:52:50.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:50 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:52:50.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:50 smithi121 ceph-mon[30729]: osdmap e951: 12 total, 11 up, 11 in 2024-09-11T03:52:50.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:50.747 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:50.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:50.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:50.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:50.871 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-11T03:52:50.871 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-09-11T03:52:50.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-09-11T03:52:50.871 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-09-11T03:52:50.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.872 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5844 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 881] snapc 100=[100,ff] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-09-11T03:52:50.872 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:09.028558+0000", 2024-09-11T03:52:50.872 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 341.84093629, 2024-09-11T03:52:50.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020566675, 2024-09-11T03:52:50.872 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.872 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.872 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:50.872 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.872 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.872 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.872 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5844 2024-09-11T03:52:50.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.872 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028558+0000", 2024-09-11T03:52:50.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028558+0000", 2024-09-11T03:52:50.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028560+0000", 2024-09-11T03:52:50.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.001e-06 2024-09-11T03:52:50.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028570+0000", 2024-09-11T03:52:50.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7799999999999995e-06 2024-09-11T03:52:50.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028586+0000", 2024-09-11T03:52:50.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6047999999999999e-05 2024-09-11T03:52:50.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028596+0000", 2024-09-11T03:52:50.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.046e-05 2024-09-11T03:52:50.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028665+0000", 2024-09-11T03:52:50.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9152000000000005e-05 2024-09-11T03:52:50.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:52:50.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.028670+0000", 2024-09-11T03:52:50.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6770000000000003e-06 2024-09-11T03:52:50.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.033863+0000", 2024-09-11T03:52:50.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051932799999999998 2024-09-11T03:52:50.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.033967+0000", 2024-09-11T03:52:50.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010344399999999999 2024-09-11T03:52:50.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,3", 2024-09-11T03:52:50.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.039468+0000", 2024-09-11T03:52:50.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055009330000000004 2024-09-11T03:52:50.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:50.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.043252+0000", 2024-09-11T03:52:50.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037844150000000002 2024-09-11T03:52:50.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.044699+0000", 2024-09-11T03:52:50.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0014469000000000001 2024-09-11T03:52:50.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.049058+0000", 2024-09-11T03:52:50.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043590399999999998 2024-09-11T03:52:50.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:50.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.049095+0000", 2024-09-11T03:52:50.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6745000000000001e-05 2024-09-11T03:52:50.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:09.049125+0000", 2024-09-11T03:52:50.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9799999999999999e-05 2024-09-11T03:52:50.879 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.879 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.879 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.880 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:5976 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 600] snapc 106=[] ondisk+write+known_if_redirected+supports_pool_eio e625)", 2024-09-11T03:52:50.880 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:47:23.155338+0000", 2024-09-11T03:52:50.880 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 327.714155791, 2024-09-11T03:52:50.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017109479, 2024-09-11T03:52:50.880 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.880 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.880 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:50.881 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.881 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.881 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.881 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 5976 2024-09-11T03:52:50.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.881 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155338+0000", 2024-09-11T03:52:50.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155338+0000", 2024-09-11T03:52:50.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155342+0000", 2024-09-11T03:52:50.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7230000000000002e-06 2024-09-11T03:52:50.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155357+0000", 2024-09-11T03:52:50.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4401e-05 2024-09-11T03:52:50.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155396+0000", 2024-09-11T03:52:50.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9301000000000002e-05 2024-09-11T03:52:50.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155408+0000", 2024-09-11T03:52:50.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2499e-05 2024-09-11T03:52:50.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155516+0000", 2024-09-11T03:52:50.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107948 2024-09-11T03:52:50.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.155615+0000", 2024-09-11T03:52:50.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8627000000000003e-05 2024-09-11T03:52:50.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,3", 2024-09-11T03:52:50.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.159866+0000", 2024-09-11T03:52:50.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004250635 2024-09-11T03:52:50.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:50.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.163122+0000", 2024-09-11T03:52:50.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032564859999999998 2024-09-11T03:52:50.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.169373+0000", 2024-09-11T03:52:50.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062509230000000002 2024-09-11T03:52:50.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.172372+0000", 2024-09-11T03:52:50.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029991810000000001 2024-09-11T03:52:50.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:50.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.172414+0000", 2024-09-11T03:52:50.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1539999999999999e-05 2024-09-11T03:52:50.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:47:23.172448+0000", 2024-09-11T03:52:50.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4214999999999997e-05 2024-09-11T03:52:50.886 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.886 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.886 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.887 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7071 2.7 2:e23d1762:::smithi17557870-21:head [copy-from ver 930] snapc 13a=[13a,139,138,137,136,135,133,12e,12d,12c,12a,127,126,123,120,11f,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e710)", 2024-09-11T03:52:50.887 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:48:48.034123+0000", 2024-09-11T03:52:50.887 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 242.83537125399999, 2024-09-11T03:52:50.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020390357000000001, 2024-09-11T03:52:50.887 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.887 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.887 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:50.888 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.888 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.888 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.888 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7071 2024-09-11T03:52:50.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.888 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034123+0000", 2024-09-11T03:52:50.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034123+0000", 2024-09-11T03:52:50.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034124+0000", 2024-09-11T03:52:50.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4600000000000004e-07 2024-09-11T03:52:50.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034127+0000", 2024-09-11T03:52:50.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0510000000000001e-06 2024-09-11T03:52:50.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034134+0000", 2024-09-11T03:52:50.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7749999999999993e-06 2024-09-11T03:52:50.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.034140+0000", 2024-09-11T03:52:50.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1050000000000003e-06 2024-09-11T03:52:50.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.036380+0000", 2024-09-11T03:52:50.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022401639999999998 2024-09-11T03:52:50.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.036429+0000", 2024-09-11T03:52:50.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8918000000000002e-05 2024-09-11T03:52:50.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:52:50.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.043986+0000", 2024-09-11T03:52:50.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075575140000000004 2024-09-11T03:52:50.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:50.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.046488+0000", 2024-09-11T03:52:50.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025022640000000001 2024-09-11T03:52:50.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.052144+0000", 2024-09-11T03:52:50.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056556080000000003 2024-09-11T03:52:50.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.054409+0000", 2024-09-11T03:52:50.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002265254 2024-09-11T03:52:50.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:50.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.054471+0000", 2024-09-11T03:52:50.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1290000000000004e-05 2024-09-11T03:52:50.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:48:48.054513+0000", 2024-09-11T03:52:50.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2768000000000001e-05 2024-09-11T03:52:50.893 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.893 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.893 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.894 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7227 2.19 2:98bb65f5:::smithi17557870-6:head [copy-from ver 972] snapc 141=[141,140,13f,13d,13c,139,138,137,136,135,133,12e,12d,12a,127,126,120,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-09-11T03:52:50.894 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:01.067580+0000", 2024-09-11T03:52:50.894 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 229.80191387900001, 2024-09-11T03:52:50.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028948451, 2024-09-11T03:52:50.894 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.894 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.894 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:50.894 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.894 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.894 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.894 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7227 2024-09-11T03:52:50.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.895 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067580+0000", 2024-09-11T03:52:50.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067580+0000", 2024-09-11T03:52:50.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067582+0000", 2024-09-11T03:52:50.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.951e-06 2024-09-11T03:52:50.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067594+0000", 2024-09-11T03:52:50.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1272e-05 2024-09-11T03:52:50.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067610+0000", 2024-09-11T03:52:50.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6889999999999999e-05 2024-09-11T03:52:50.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067623+0000", 2024-09-11T03:52:50.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2680999999999999e-05 2024-09-11T03:52:50.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067684+0000", 2024-09-11T03:52:50.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1005999999999998e-05 2024-09-11T03:52:50.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-09-11T03:52:50.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067692+0000", 2024-09-11T03:52:50.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4189999999999996e-06 2024-09-11T03:52:50.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.069575+0000", 2024-09-11T03:52:50.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001882959 2024-09-11T03:52:50.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.069682+0000", 2024-09-11T03:52:50.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107416 2024-09-11T03:52:50.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0,3", 2024-09-11T03:52:50.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.089088+0000", 2024-09-11T03:52:50.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019406102000000001 2024-09-11T03:52:50.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:50.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.092329+0000", 2024-09-11T03:52:50.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032404589999999998 2024-09-11T03:52:50.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.096291+0000", 2024-09-11T03:52:50.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039621470000000001 2024-09-11T03:52:50.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.096455+0000", 2024-09-11T03:52:50.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016462099999999999 2024-09-11T03:52:50.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:50.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.096494+0000", 2024-09-11T03:52:50.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8705000000000003e-05 2024-09-11T03:52:50.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.096529+0000", 2024-09-11T03:52:50.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4823000000000002e-05 2024-09-11T03:52:50.901 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.901 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.901 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.901 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7228 2.19 2:98bb65f5:::smithi17557870-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e723)", 2024-09-11T03:52:50.901 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:01.067641+0000", 2024-09-11T03:52:50.901 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 229.80185309699999, 2024-09-11T03:52:50.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.024383425, 2024-09-11T03:52:50.901 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.901 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:50.902 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:50.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:50.904 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:52:50.904 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.904 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.904 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.905 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7228 2024-09-11T03:52:50.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.905 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067641+0000", 2024-09-11T03:52:50.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067641+0000", 2024-09-11T03:52:50.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067641+0000", 2024-09-11T03:52:50.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3700000000000001e-07 2024-09-11T03:52:50.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067643+0000", 2024-09-11T03:52:50.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7579999999999999e-06 2024-09-11T03:52:50.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067651+0000", 2024-09-11T03:52:50.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2830000000000004e-06 2024-09-11T03:52:50.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067656+0000", 2024-09-11T03:52:50.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2009999999999998e-06 2024-09-11T03:52:50.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067757+0000", 2024-09-11T03:52:50.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000100888 2024-09-11T03:52:50.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:52:50.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.067762+0000", 2024-09-11T03:52:50.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3859999999999998e-06 2024-09-11T03:52:50.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.069849+0000", 2024-09-11T03:52:50.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020867500000000001 2024-09-11T03:52:50.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:50.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.069896+0000", 2024-09-11T03:52:50.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859e-05 2024-09-11T03:52:50.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.091800+0000", 2024-09-11T03:52:50.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021904534999999999 2024-09-11T03:52:50.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.091916+0000", 2024-09-11T03:52:50.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116092 2024-09-11T03:52:50.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:01.092025+0000", 2024-09-11T03:52:50.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108336 2024-09-11T03:52:50.910 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.910 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.910 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.910 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7384 2.7 2:e0eabcfb:::smithi17557870-9:head [copy-from ver 950] snapc 14d=[14d,14b,14a,149,148,147,146,145,144,143,142,13d,13c,139,138,136,135,133,12e,12a,127,120,11e,119] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-11T03:52:50.910 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:21.110220+0000", 2024-09-11T03:52:50.910 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 209.75927378899999, 2024-09-11T03:52:50.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021623634999999999, 2024-09-11T03:52:50.910 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.910 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.911 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:50.911 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.911 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.911 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.911 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7384 2024-09-11T03:52:50.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.911 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:21.110220+0000", 2024-09-11T03:52:50.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:21.110220+0000", 2024-09-11T03:52:50.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:21.110222+0000", 2024-09-11T03:52:50.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1039999999999999e-06 2024-09-11T03:52:50.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:21.110228+0000", 2024-09-11T03:52:50.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4309999999999999e-06 2024-09-11T03:52:50.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:21.110239+0000", 2024-09-11T03:52:50.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0869e-05 2024-09-11T03:52:50.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:21.110247+0000", 2024-09-11T03:52:50.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0579999999999998e-06 2024-09-11T03:52:50.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:21.115803+0000", 2024-09-11T03:52:50.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055562570000000002 2024-09-11T03:52:50.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:21.115908+0000", 2024-09-11T03:52:50.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010516100000000001 2024-09-11T03:52:50.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:52:50.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:21.121139+0000", 2024-09-11T03:52:50.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052305540000000001 2024-09-11T03:52:50.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:50.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:21.125322+0000", 2024-09-11T03:52:50.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00418287 2024-09-11T03:52:50.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:21.131699+0000", 2024-09-11T03:52:50.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006377499 2024-09-11T03:52:50.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:21.131739+0000", 2024-09-11T03:52:50.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9897000000000001e-05 2024-09-11T03:52:50.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:50.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:21.131784+0000", 2024-09-11T03:52:50.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4524000000000003e-05 2024-09-11T03:52:50.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:21.131844+0000", 2024-09-11T03:52:50.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0411000000000001e-05 2024-09-11T03:52:50.916 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.916 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.916 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.917 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(2.29 749/748 [PushOp(2:95fa802a:::smithi17557870-22:145, version: 743'1144, data_included: [720896~720896,1900544~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:95fa802a:::smithi17557870-22:145@743'1144, size: 3933003, copy_subset: [(0, 3933003)], clone_subset: {2:95fa802a:::smithi17557870-22:12a: []}, snapset: 145=[]:{11b: [119], 127: [127, 120, 11e], 12a: [12a], 145: [145, 144, 143, 142, 13d, 13c, 139, 138, 136, 135, 133, 12e]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2228224, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T03:52:50.917 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:27.300366+0000", 2024-09-11T03:52:50.917 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 203.56912858699999, 2024-09-11T03:52:50.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.036441360999999999, 2024-09-11T03:52:50.917 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.917 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.917 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:52:50.917 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.300366+0000", 2024-09-11T03:52:50.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.300366+0000", 2024-09-11T03:52:50.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.300366+0000", 2024-09-11T03:52:50.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3200000000000005e-07 2024-09-11T03:52:50.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.336334+0000", 2024-09-11T03:52:50.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035967341999999999 2024-09-11T03:52:50.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.336350+0000", 2024-09-11T03:52:50.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6603e-05 2024-09-11T03:52:50.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.336362+0000", 2024-09-11T03:52:50.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2048000000000001e-05 2024-09-11T03:52:50.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.336452+0000", 2024-09-11T03:52:50.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9309999999999997e-05 2024-09-11T03:52:50.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.336466+0000", 2024-09-11T03:52:50.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4810000000000001e-05 2024-09-11T03:52:50.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:27.336807+0000", 2024-09-11T03:52:50.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00034061600000000002 2024-09-11T03:52:50.920 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.921 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.921 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.921 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7546 2.7 2:e23d1762:::smithi17557870-21:head [copy-from ver 951] snapc 155=[155,154,153,151,150,14f,14e,14d,14b,149,148,144,143,142,13d,13c,139,136,135,133,12a,127,120,119] ondisk+write+known_if_redirected+supports_pool_eio e759)", 2024-09-11T03:52:50.921 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:37.422240+0000", 2024-09-11T03:52:50.921 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 193.44725460000001, 2024-09-11T03:52:50.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.024090451999999998, 2024-09-11T03:52:50.921 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.921 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.921 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:50.921 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.921 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.922 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.922 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7546 2024-09-11T03:52:50.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.922 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:37.422240+0000", 2024-09-11T03:52:50.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:37.422240+0000", 2024-09-11T03:52:50.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:37.422242+0000", 2024-09-11T03:52:50.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3259999999999998e-06 2024-09-11T03:52:50.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:37.422252+0000", 2024-09-11T03:52:50.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0013999999999999e-05 2024-09-11T03:52:50.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:37.422280+0000", 2024-09-11T03:52:50.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7993000000000001e-05 2024-09-11T03:52:50.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:37.422294+0000", 2024-09-11T03:52:50.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.367e-05 2024-09-11T03:52:50.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:37.422458+0000", 2024-09-11T03:52:50.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016424399999999999 2024-09-11T03:52:50.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:37.422670+0000", 2024-09-11T03:52:50.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021222100000000001 2024-09-11T03:52:50.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:52:50.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:37.431717+0000", 2024-09-11T03:52:50.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0090464880000000001 2024-09-11T03:52:50.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:50.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:37.435161+0000", 2024-09-11T03:52:50.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034442639999999998 2024-09-11T03:52:50.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:37.440649+0000", 2024-09-11T03:52:50.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054882710000000003 2024-09-11T03:52:50.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:37.446212+0000", 2024-09-11T03:52:50.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055631070000000003 2024-09-11T03:52:50.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:50.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:37.446279+0000", 2024-09-11T03:52:50.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7039000000000006e-05 2024-09-11T03:52:50.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:37.446330+0000", 2024-09-11T03:52:50.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0815000000000001e-05 2024-09-11T03:52:50.927 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.927 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.927 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.927 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:7879 2.7 2:e0ec5763:::smithi17557870-2:head [copy-from ver 930] snapc 15f=[15f,15e,15d,15c,15b,15a,157,156,155,154,153,151,14d,148,144,143,13d,13c,139,135,133,12a,127,120] ondisk+write+known_if_redirected+supports_pool_eio e779)", 2024-09-11T03:52:50.927 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:49:57.163820+0000", 2024-09-11T03:52:50.927 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 173.705673816, 2024-09-11T03:52:50.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.026505274999999998, 2024-09-11T03:52:50.928 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.928 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.928 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:50.928 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.928 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.928 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.928 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 7879 2024-09-11T03:52:50.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.928 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.163820+0000", 2024-09-11T03:52:50.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.163820+0000", 2024-09-11T03:52:50.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.163822+0000", 2024-09-11T03:52:50.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1960000000000001e-06 2024-09-11T03:52:50.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.163834+0000", 2024-09-11T03:52:50.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2316e-05 2024-09-11T03:52:50.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.163846+0000", 2024-09-11T03:52:50.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1664e-05 2024-09-11T03:52:50.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.163853+0000", 2024-09-11T03:52:50.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.695e-06 2024-09-11T03:52:50.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.163907+0000", 2024-09-11T03:52:50.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3990000000000003e-05 2024-09-11T03:52:50.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.163977+0000", 2024-09-11T03:52:50.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9719000000000001e-05 2024-09-11T03:52:50.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:52:50.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.174239+0000", 2024-09-11T03:52:50.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010261901 2024-09-11T03:52:50.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:50.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.177788+0000", 2024-09-11T03:52:50.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035492380000000001 2024-09-11T03:52:50.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.182698+0000", 2024-09-11T03:52:50.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0049103280000000003 2024-09-11T03:52:50.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.190253+0000", 2024-09-11T03:52:50.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075548710000000003 2024-09-11T03:52:50.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:50.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.190294+0000", 2024-09-11T03:52:50.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0544999999999999e-05 2024-09-11T03:52:50.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:49:57.190326+0000", 2024-09-11T03:52:50.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1812e-05 2024-09-11T03:52:50.934 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.934 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.934 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.934 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8103 2.7 2:e23d1762:::smithi17557870-21:head [copy-from ver 975] snapc 167=[167,166,164,163,162,160,15f,15e,15d,15b,15a,157,156,155,151,14d,148,144,143,13c,139,135,12a,127,120] ondisk+write+known_if_redirected+supports_pool_eio e795)", 2024-09-11T03:52:50.934 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:50:13.127184+0000", 2024-09-11T03:52:50.934 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 157.74231074900001, 2024-09-11T03:52:50.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029079318999999999, 2024-09-11T03:52:50.935 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.935 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.935 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:50.935 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.935 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.935 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.935 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8103 2024-09-11T03:52:50.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.935 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:13.127184+0000", 2024-09-11T03:52:50.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:13.127184+0000", 2024-09-11T03:52:50.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:13.127185+0000", 2024-09-11T03:52:50.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.556e-06 2024-09-11T03:52:50.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:13.127193+0000", 2024-09-11T03:52:50.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4669999999999998e-06 2024-09-11T03:52:50.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:13.127208+0000", 2024-09-11T03:52:50.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5471000000000001e-05 2024-09-11T03:52:50.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:13.127233+0000", 2024-09-11T03:52:50.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5228999999999998e-05 2024-09-11T03:52:50.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:13.127303+0000", 2024-09-11T03:52:50.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9879e-05 2024-09-11T03:52:50.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:13.127415+0000", 2024-09-11T03:52:50.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011151599999999999 2024-09-11T03:52:50.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:52:50.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:13.131709+0000", 2024-09-11T03:52:50.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042938869999999997 2024-09-11T03:52:50.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:50.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:13.134131+0000", 2024-09-11T03:52:50.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024229469999999999 2024-09-11T03:52:50.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:13.136363+0000", 2024-09-11T03:52:50.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022316240000000002 2024-09-11T03:52:50.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:13.156168+0000", 2024-09-11T03:52:50.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019805026999999999 2024-09-11T03:52:50.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:50.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:13.156213+0000", 2024-09-11T03:52:50.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5024999999999997e-05 2024-09-11T03:52:50.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:50:13.156263+0000", 2024-09-11T03:52:50.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9691000000000002e-05 2024-09-11T03:52:50.940 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.941 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.941 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.941 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8857 2.29 2:95fa802a:::smithi17557870-22:head [write 770084~501915] snapc 18e=[18e,18d,18c,18b,18a,189,188,187,186,185,184,183,181,180,17f,17e,17d,17c,17a,178,177,176,175,174,172,171,170,16e,16b,169,167,164,15f,15e,15b,157,14d,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e856)", 2024-09-11T03:52:50.941 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:51:14.151534+0000", 2024-09-11T03:52:50.941 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 96.717960120000001, 2024-09-11T03:52:50.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025403124999999999, 2024-09-11T03:52:50.941 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.941 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.941 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:50.941 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.941 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.942 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.942 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8857 2024-09-11T03:52:50.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.942 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.151534+0000", 2024-09-11T03:52:50.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.151534+0000", 2024-09-11T03:52:50.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.151537+0000", 2024-09-11T03:52:50.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1489999999999998e-06 2024-09-11T03:52:50.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.151903+0000", 2024-09-11T03:52:50.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036567700000000002 2024-09-11T03:52:50.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.151927+0000", 2024-09-11T03:52:50.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4368000000000001e-05 2024-09-11T03:52:50.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.151938+0000", 2024-09-11T03:52:50.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0479000000000001e-05 2024-09-11T03:52:50.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.152014+0000", 2024-09-11T03:52:50.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6546999999999997e-05 2024-09-11T03:52:50.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.152109+0000", 2024-09-11T03:52:50.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4352999999999996e-05 2024-09-11T03:52:50.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,8", 2024-09-11T03:52:50.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.152241+0000", 2024-09-11T03:52:50.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013253700000000001 2024-09-11T03:52:50.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:50.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.153371+0000", 2024-09-11T03:52:50.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011294899999999999 2024-09-11T03:52:50.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.154373+0000", 2024-09-11T03:52:50.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010021120000000001 2024-09-11T03:52:50.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.176789+0000", 2024-09-11T03:52:50.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022416308999999999 2024-09-11T03:52:50.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:50.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.176882+0000", 2024-09-11T03:52:50.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2495000000000001e-05 2024-09-11T03:52:50.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.176937+0000", 2024-09-11T03:52:50.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5609000000000002e-05 2024-09-11T03:52:50.947 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.947 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.947 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.947 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8858 2.29 2:95fa802a:::smithi17557870-22:head [write 2023470~346614] snapc 18e=[18e,18d,18c,18b,18a,189,188,187,186,185,184,183,181,180,17f,17e,17d,17c,17a,178,177,176,175,174,172,171,170,16e,16b,169,167,164,15f,15e,15b,157,14d,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e856)", 2024-09-11T03:52:50.947 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:51:14.157590+0000", 2024-09-11T03:52:50.947 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 96.711904240999999, 2024-09-11T03:52:50.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019492388999999999, 2024-09-11T03:52:50.948 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.948 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.948 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:50.948 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.948 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.948 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.948 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8858 2024-09-11T03:52:50.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.948 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.157590+0000", 2024-09-11T03:52:50.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.157590+0000", 2024-09-11T03:52:50.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.157591+0000", 2024-09-11T03:52:50.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.31e-06 2024-09-11T03:52:50.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.157992+0000", 2024-09-11T03:52:50.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040096499999999998 2024-09-11T03:52:50.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158004+0000", 2024-09-11T03:52:50.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2197e-05 2024-09-11T03:52:50.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158010+0000", 2024-09-11T03:52:50.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7980000000000002e-06 2024-09-11T03:52:50.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158069+0000", 2024-09-11T03:52:50.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9113999999999997e-05 2024-09-11T03:52:50.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158165+0000", 2024-09-11T03:52:50.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6087000000000003e-05 2024-09-11T03:52:50.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,8", 2024-09-11T03:52:50.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158281+0000", 2024-09-11T03:52:50.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115285 2024-09-11T03:52:50.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:50.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.159393+0000", 2024-09-11T03:52:50.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001112268 2024-09-11T03:52:50.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.159717+0000", 2024-09-11T03:52:50.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032442000000000001 2024-09-11T03:52:50.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.176989+0000", 2024-09-11T03:52:50.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017271735999999999 2024-09-11T03:52:50.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:50.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.177034+0000", 2024-09-11T03:52:50.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4879999999999997e-05 2024-09-11T03:52:50.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.177082+0000", 2024-09-11T03:52:50.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8328999999999998e-05 2024-09-11T03:52:50.953 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.953 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.953 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.954 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8859 2.29 2:95fa802a:::smithi17557870-22:head [setxattr _header (58) in=7b,truncate 2370084] snapc 18e=[18e,18d,18c,18b,18a,189,188,187,186,185,184,183,181,180,17f,17e,17d,17c,17a,178,177,176,175,174,172,171,170,16e,16b,169,167,164,15f,15e,15b,157,14d,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e856)", 2024-09-11T03:52:50.954 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:51:14.158026+0000", 2024-09-11T03:52:50.954 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 96.711468683000007, 2024-09-11T03:52:50.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019204629000000001, 2024-09-11T03:52:50.954 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.954 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.954 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:50.954 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.954 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.954 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.954 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8859 2024-09-11T03:52:50.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.955 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158026+0000", 2024-09-11T03:52:50.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158026+0000", 2024-09-11T03:52:50.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158026+0000", 2024-09-11T03:52:50.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9999999999999997e-07 2024-09-11T03:52:50.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158030+0000", 2024-09-11T03:52:50.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7639999999999999e-06 2024-09-11T03:52:50.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158043+0000", 2024-09-11T03:52:50.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3429e-05 2024-09-11T03:52:50.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158054+0000", 2024-09-11T03:52:50.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.065e-05 2024-09-11T03:52:50.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158786+0000", 2024-09-11T03:52:50.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073223199999999998 2024-09-11T03:52:50.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158871+0000", 2024-09-11T03:52:50.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4435000000000001e-05 2024-09-11T03:52:50.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3,8", 2024-09-11T03:52:50.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158981+0000", 2024-09-11T03:52:50.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110015 2024-09-11T03:52:50.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:50.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.159397+0000", 2024-09-11T03:52:50.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041662300000000003 2024-09-11T03:52:50.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.159870+0000", 2024-09-11T03:52:50.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047214400000000003 2024-09-11T03:52:50.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.177122+0000", 2024-09-11T03:52:50.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01725194 2024-09-11T03:52:50.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:50.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.177162+0000", 2024-09-11T03:52:50.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0670000000000002e-05 2024-09-11T03:52:50.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.177230+0000", 2024-09-11T03:52:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8027000000000004e-05 2024-09-11T03:52:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:8860 2.29 2:95fa802a:::smithi17557870-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e856)", 2024-09-11T03:52:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:51:14.158073+0000", 2024-09-11T03:52:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 96.711421150999996, 2024-09-11T03:52:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019324285, 2024-09-11T03:52:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:52:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 8860 2024-09-11T03:52:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158073+0000", 2024-09-11T03:52:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158073+0000", 2024-09-11T03:52:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158074+0000", 2024-09-11T03:52:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3300000000000002e-07 2024-09-11T03:52:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158077+0000", 2024-09-11T03:52:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8660000000000002e-06 2024-09-11T03:52:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158085+0000", 2024-09-11T03:52:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9850000000000006e-06 2024-09-11T03:52:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.158099+0000", 2024-09-11T03:52:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4815e-05 2024-09-11T03:52:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.159288+0000", 2024-09-11T03:52:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001188719 2024-09-11T03:52:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:52:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.159374+0000", 2024-09-11T03:52:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6216000000000003e-05 2024-09-11T03:52:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.177263+0000", 2024-09-11T03:52:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017888446999999998 2024-09-11T03:52:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.177324+0000", 2024-09-11T03:52:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0865000000000001e-05 2024-09-11T03:52:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:14.177397+0000", 2024-09-11T03:52:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3739000000000007e-05 2024-09-11T03:52:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:9063 2.2d 2:b6a25a79:::smithi17557870-47:head [copy-from ver 1258] snapc 196=[196,195,194,193,192,191,190,18c,18b,18a,189,188,186,183,180,17e,17d,17c,17a,178,177,175,174,172,16e,16b,169,167,164,15f,15e,15b,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e877)", 2024-09-11T03:52:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:51:35.075756+0000", 2024-09-11T03:52:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 75.793738625000003, 2024-09-11T03:52:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022935236000000001, 2024-09-11T03:52:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 9063 2024-09-11T03:52:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.075756+0000", 2024-09-11T03:52:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.075756+0000", 2024-09-11T03:52:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.075757+0000", 2024-09-11T03:52:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5510000000000001e-06 2024-09-11T03:52:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.075763+0000", 2024-09-11T03:52:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8780000000000003e-06 2024-09-11T03:52:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.075779+0000", 2024-09-11T03:52:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5845000000000002e-05 2024-09-11T03:52:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.075787+0000", 2024-09-11T03:52:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1980000000000001e-06 2024-09-11T03:52:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.075852+0000", 2024-09-11T03:52:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5339999999999994e-05 2024-09-11T03:52:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.076003+0000", 2024-09-11T03:52:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000150679 2024-09-11T03:52:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5,7", 2024-09-11T03:52:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.089436+0000", 2024-09-11T03:52:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013432739000000001 2024-09-11T03:52:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.094685+0000", 2024-09-11T03:52:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052489169999999996 2024-09-11T03:52:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.098454+0000", 2024-09-11T03:52:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037694540000000002 2024-09-11T03:52:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.098602+0000", 2024-09-11T03:52:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014778299999999999 2024-09-11T03:52:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.098645+0000", 2024-09-11T03:52:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2762000000000002e-05 2024-09-11T03:52:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.098691+0000", 2024-09-11T03:52:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6090000000000001e-05 2024-09-11T03:52:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:9064 2.2d 2:b6a25a79:::smithi17557870-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e877)", 2024-09-11T03:52:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:51:35.075805+0000", 2024-09-11T03:52:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 75.793688904999996, 2024-09-11T03:52:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018277560000000002, 2024-09-11T03:52:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:52:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 9064 2024-09-11T03:52:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.075805+0000", 2024-09-11T03:52:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.075805+0000", 2024-09-11T03:52:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.075806+0000", 2024-09-11T03:52:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4000000000000002e-07 2024-09-11T03:52:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.075809+0000", 2024-09-11T03:52:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8080000000000001e-06 2024-09-11T03:52:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.075816+0000", 2024-09-11T03:52:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6529999999999994e-06 2024-09-11T03:52:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.075822+0000", 2024-09-11T03:52:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4330000000000003e-06 2024-09-11T03:52:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.076395+0000", 2024-09-11T03:52:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000573582 2024-09-11T03:52:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for blocked object", 2024-09-11T03:52:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.076457+0000", 2024-09-11T03:52:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1364000000000004e-05 2024-09-11T03:52:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.093851+0000", 2024-09-11T03:52:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017394513 2024-09-11T03:52:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.093968+0000", 2024-09-11T03:52:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116817 2024-09-11T03:52:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:51:35.094083+0000", 2024-09-11T03:52:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011485 2024-09-11T03:52:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:9282 2.7 2:e0eabcfb:::smithi17557870-9:head [copy-from ver 1195] snapc 1a3=[1a3,1a2,1a1,1a0,19f,19e,19d,19c,19b,199,198,197,195,194,193,192,191,18b,18a,189,188,186,183,17e,17c,17a,178,175,172,16e,16b,169,164,15f,144,13c,127] ondisk+write+known_if_redirected+supports_pool_eio e904)", 2024-09-11T03:52:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:52:02.081017+0000", 2024-09-11T03:52:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 48.788477589999999, 2024-09-11T03:52:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021332904, 2024-09-11T03:52:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 9282 2024-09-11T03:52:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:02.081017+0000", 2024-09-11T03:52:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:02.081017+0000", 2024-09-11T03:52:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:02.081017+0000", 2024-09-11T03:52:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4099999999999999e-07 2024-09-11T03:52:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:02.081019+0000", 2024-09-11T03:52:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.889e-06 2024-09-11T03:52:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:02.081029+0000", 2024-09-11T03:52:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0064999999999999e-05 2024-09-11T03:52:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:02.081033+0000", 2024-09-11T03:52:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4690000000000002e-06 2024-09-11T03:52:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:02.081086+0000", 2024-09-11T03:52:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3386e-05 2024-09-11T03:52:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:02.081207+0000", 2024-09-11T03:52:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012123 2024-09-11T03:52:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:52:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:02.089600+0000", 2024-09-11T03:52:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083930710000000002 2024-09-11T03:52:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:02.094946+0000", 2024-09-11T03:52:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053460130000000002 2024-09-11T03:52:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:02.097044+0000", 2024-09-11T03:52:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002097575 2024-09-11T03:52:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:02.102262+0000", 2024-09-11T03:52:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052186719999999997 2024-09-11T03:52:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:02.102308+0000", 2024-09-11T03:52:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5544999999999998e-05 2024-09-11T03:52:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:02.102350+0000", 2024-09-11T03:52:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1548000000000003e-05 2024-09-11T03:52:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:9873 2.7 2:e0ec5763:::smithi17557870-2:head [write 1831727~757138] snapc 1b9=[1b9,1b8,1b7,1b6,1b5,1b4,1b3,1b2,1b1,1b0,1af,1ac,1aa,1a8,1a7,1a6,1a5,1a4,1a3,1a2,1a1,19d,19c,199,197,194,191,18b,183,17c,17a,175,172,16e,164,13c] ondisk+write+known_if_redirected+supports_pool_eio e949)", 2024-09-11T03:52:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:52:47.152057+0000", 2024-09-11T03:52:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.7174370830000001, 2024-09-11T03:52:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.056083860999999999, 2024-09-11T03:52:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 9873 2024-09-11T03:52:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152057+0000", 2024-09-11T03:52:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152057+0000", 2024-09-11T03:52:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152058+0000", 2024-09-11T03:52:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2780000000000001e-06 2024-09-11T03:52:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152702+0000", 2024-09-11T03:52:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064313200000000004 2024-09-11T03:52:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152713+0000", 2024-09-11T03:52:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1766999999999999e-05 2024-09-11T03:52:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152719+0000", 2024-09-11T03:52:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3129999999999998e-06 2024-09-11T03:52:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.988 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:50 smithi175 ceph-mon[30894]: pgmap v1581: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 714 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 14 MiB/s rd, 4.1 MiB/s wr, 28 op/s; 72/996 objects degraded (7.229%) 2024-09-11T03:52:50.988 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:50 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:52:50.989 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:50 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:52:50.989 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:50 smithi175 ceph-mon[30894]: osdmap e951: 12 total, 11 up, 11 in 2024-09-11T03:52:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152759+0000", 2024-09-11T03:52:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0080999999999998e-05 2024-09-11T03:52:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152820+0000", 2024-09-11T03:52:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1060000000000002e-05 2024-09-11T03:52:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:52:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152921+0000", 2024-09-11T03:52:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101078 2024-09-11T03:52:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.154453+0000", 2024-09-11T03:52:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015320970000000001 2024-09-11T03:52:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.155084+0000", 2024-09-11T03:52:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00063100799999999996 2024-09-11T03:52:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.208061+0000", 2024-09-11T03:52:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.052977424000000002 2024-09-11T03:52:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.208105+0000", 2024-09-11T03:52:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3364e-05 2024-09-11T03:52:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.208141+0000", 2024-09-11T03:52:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6259e-05 2024-09-11T03:52:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:9874 2.7 2:e0ec5763:::smithi17557870-2:head [setxattr _header (57) in=7b,truncate 2655836] snapc 1b9=[1b9,1b8,1b7,1b6,1b5,1b4,1b3,1b2,1b1,1b0,1af,1ac,1aa,1a8,1a7,1a6,1a5,1a4,1a3,1a2,1a1,19d,19c,199,197,194,191,18b,183,17c,17a,175,172,16e,164,13c] ondisk+write+known_if_redirected+supports_pool_eio e949)", 2024-09-11T03:52:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:52:47.152732+0000", 2024-09-11T03:52:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.7167626399999998, 2024-09-11T03:52:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055537508999999999, 2024-09-11T03:52:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:52:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 9874 2024-09-11T03:52:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152732+0000", 2024-09-11T03:52:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152732+0000", 2024-09-11T03:52:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152732+0000", 2024-09-11T03:52:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9500000000000002e-07 2024-09-11T03:52:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152735+0000", 2024-09-11T03:52:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8609999999999998e-06 2024-09-11T03:52:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152743+0000", 2024-09-11T03:52:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7619999999999995e-06 2024-09-11T03:52:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152749+0000", 2024-09-11T03:52:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8510000000000004e-06 2024-09-11T03:52:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.153796+0000", 2024-09-11T03:52:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010477819999999999 2024-09-11T03:52:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.153902+0000", 2024-09-11T03:52:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010504 2024-09-11T03:52:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,10", 2024-09-11T03:52:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.154066+0000", 2024-09-11T03:52:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016422100000000001 2024-09-11T03:52:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-09-11T03:52:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.154466+0000", 2024-09-11T03:52:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040068200000000002 2024-09-11T03:52:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.155114+0000", 2024-09-11T03:52:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064762000000000001 2024-09-11T03:52:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:52:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.208177+0000", 2024-09-11T03:52:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.053062865000000001 2024-09-11T03:52:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-09-11T03:52:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.208223+0000", 2024-09-11T03:52:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6604000000000003e-05 2024-09-11T03:52:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.208269+0000", 2024-09-11T03:52:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5626e-05 2024-09-11T03:52:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.24454.0:9875 2.7 2:e0ec5763:::smithi17557870-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e949)", 2024-09-11T03:52:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-09-11T03:52:47.152757+0000", 2024-09-11T03:52:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.7167370630000001, 2024-09-11T03:52:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055659708000000002, 2024-09-11T03:52:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "continuous": false, 2024-09-11T03:52:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-09-11T03:52:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-09-11T03:52:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-09-11T03:52:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.24454", 2024-09-11T03:52:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:52:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 9875 2024-09-11T03:52:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-09-11T03:52:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-09-11T03:52:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152757+0000", 2024-09-11T03:52:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-09-11T03:52:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152757+0000", 2024-09-11T03:52:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-09-11T03:52:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-09-11T03:52:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152758+0000", 2024-09-11T03:52:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5699999999999998e-07 2024-09-11T03:52:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-09-11T03:52:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152760+0000", 2024-09-11T03:52:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.091e-06 2024-09-11T03:52:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-09-11T03:52:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152766+0000", 2024-09-11T03:52:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2729999999999999e-06 2024-09-11T03:52:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-09-11T03:52:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.152770+0000", 2024-09-11T03:52:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3200000000000001e-06 2024-09-11T03:52:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.154378+0000", 2024-09-11T03:52:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016071810000000001 2024-09-11T03:52:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-09-11T03:52:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.154435+0000", 2024-09-11T03:52:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7218e-05 2024-09-11T03:52:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-09-11T03:52:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.208301+0000", 2024-09-11T03:52:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.053866234999999998 2024-09-11T03:52:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-09-11T03:52:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.208355+0000", 2024-09-11T03:52:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3937999999999998e-05 2024-09-11T03:52:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-11T03:52:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-11T03:52:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-09-11T03:52:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-09-11T03:52:47.208417+0000", 2024-09-11T03:52:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1995000000000005e-05 2024-09-11T03:52:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-11T03:52:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-11T03:52:51.005 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-11T03:52:51.007 INFO:teuthology.orchestra.run.smithi084.stderr:Exporting 2.19 info 2.19( v 936'663 (0'0,936'663] local-lis/les=933/934 n=3 ec=276/95 lis/c=933/933 les/c/f=934/934/0 sis=933) 2024-09-11T03:52:51.007 INFO:teuthology.orchestra.run.smithi084.stderr:Read #2:98bb65f5:::smithi17557870-6:13d# 2024-09-11T03:52:51.007 INFO:teuthology.orchestra.run.smithi084.stderr:Read #2:98bb65f5:::smithi17557870-6:167# 2024-09-11T03:52:51.007 INFO:teuthology.orchestra.run.smithi084.stderr:Read #2:98bb65f5:::smithi17557870-6:18a# 2024-09-11T03:52:51.007 INFO:teuthology.orchestra.run.smithi084.stderr:Read #2:98bb65f5:::smithi17557870-6:1a5# 2024-09-11T03:52:51.007 INFO:teuthology.orchestra.run.smithi084.stderr:Read #2:98bb65f5:::smithi17557870-6:1a9# 2024-09-11T03:52:51.008 INFO:teuthology.orchestra.run.smithi084.stderr:Read #2:98bb65f5:::smithi17557870-6:1ab# 2024-09-11T03:52:51.008 INFO:teuthology.orchestra.run.smithi084.stderr:Read #2:98bb65f5:::smithi17557870-6:head# 2024-09-11T03:52:51.008 INFO:teuthology.orchestra.run.smithi084.stderr:Export successful 2024-09-11T03:52:51.008 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:51.030 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:51.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:51.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:51.126 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:52:51.127 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-11T03:52:51.127 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:52:51.132 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:51.135 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:52:51.135 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:52:51.151 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:52:51.151 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:52:51.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:51.154 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:51.155 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:51.167 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:52:51.168 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:52:51.184 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:52:51.184 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:52:51.201 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:52:51.202 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:52:51.209 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-11T03:52:51.218 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:52:51.218 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:52:51.235 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:52:51.236 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:52:51.252 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:52:51.252 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:52:51.255 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:51.269 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:52:51.269 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:52:51.275 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:51.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:51.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:51.301 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:52:51.302 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:52:51.377 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:51.399 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:51.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:51.400 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:51.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:51.523 INFO:tasks.rados.rados.2.smithi175.stdout:3981: done (5 left) 2024-09-11T03:52:51.523 INFO:tasks.rados.rados.2.smithi175.stdout:3983: done (4 left) 2024-09-11T03:52:51.523 INFO:tasks.rados.rados.2.smithi175.stdout:3984: done (3 left) 2024-09-11T03:52:51.523 INFO:tasks.rados.rados.2.smithi175.stdout:3985: done (2 left) 2024-09-11T03:52:51.524 INFO:tasks.rados.rados.2.smithi175.stdout:3986: done (1 left) 2024-09-11T03:52:51.524 INFO:tasks.rados.rados.2.smithi175.stdout:3987: done (0 left) 2024-09-11T03:52:51.524 INFO:tasks.rados.rados.2.smithi175.stdout:3988: snap_remove snap 437 2024-09-11T03:52:51.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:51.526 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:51.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:51.626 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:51.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:51 smithi084 ceph-mon[27254]: osdmap e952: 12 total, 11 up, 11 in 2024-09-11T03:52:51.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/371279149' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:51.648 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:51.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:51.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:51.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:51 smithi121 ceph-mon[30729]: osdmap e952: 12 total, 11 up, 11 in 2024-09-11T03:52:51.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/371279149' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:51.749 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:51.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:51.771 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:51.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:51.872 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:51.895 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:51.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:51.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:51.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:51 smithi175 ceph-mon[30894]: osdmap e952: 12 total, 11 up, 11 in 2024-09-11T03:52:51.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/371279149' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:51.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:52.021 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:52.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:52.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:52.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-11T03:52:52.067 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -n osd.0 --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --force --op remove --pgid 2.19 2024-09-11T03:52:52.123 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:52.153 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:52.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:52.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:52.256 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:52.277 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:52.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:52.334 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:52:52.359 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/osd.0/config 2024-09-11T03:52:52.380 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:52.406 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:52.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:52.407 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:52.508 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:52.523 INFO:tasks.rados.rados.2.smithi175.stdout:3988: done (0 left) 2024-09-11T03:52:52.523 INFO:tasks.rados.rados.2.smithi175.stdout:3989: rollback oid 21 current snap is 442 2024-09-11T03:52:52.523 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 21 to 429 2024-09-11T03:52:52.523 INFO:tasks.rados.rados.2.smithi175.stdout:3990: snap_create 2024-09-11T03:52:52.535 INFO:tasks.rados.rados.2.smithi175.stdout:3989: finishing rollback tid 0 to smithi17557870-21 2024-09-11T03:52:52.535 INFO:tasks.rados.rados.2.smithi175.stdout:3989: finishing rollback tid 1 to smithi17557870-21 2024-09-11T03:52:52.535 INFO:tasks.rados.rados.2.smithi175.stdout:3989: finishing rollback tid 2 to smithi17557870-21 2024-09-11T03:52:52.536 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 21 v 1395 (ObjNum 829 snap 263 seq_num 829) dirty exists 2024-09-11T03:52:52.540 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:52.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:52.541 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:52.642 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:52.666 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:52.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:52.667 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:52.756 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:52 smithi121 ceph-mon[30729]: pgmap v1584: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 714 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 8.3 MiB/s rd, 1.4 MiB/s wr, 13 op/s; 72/996 objects degraded (7.229%) 2024-09-11T03:52:52.756 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:52 smithi121 ceph-mon[30729]: osdmap e953: 12 total, 11 up, 11 in 2024-09-11T03:52:52.756 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:52 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:52:52.757 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:52 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:52:52.757 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:52 smithi121 ceph-mon[30729]: Health check update: Degraded data redundancy: 72/996 objects degraded (7.229%), 12 pgs degraded (PG_DEGRADED) 2024-09-11T03:52:52.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:52.770 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:52:52.770 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:52:52.790 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:52.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:52.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:52.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:52 smithi084 ceph-mon[27254]: pgmap v1584: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 714 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 8.3 MiB/s rd, 1.4 MiB/s wr, 13 op/s; 72/996 objects degraded (7.229%) 2024-09-11T03:52:52.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:52 smithi084 ceph-mon[27254]: osdmap e953: 12 total, 11 up, 11 in 2024-09-11T03:52:52.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:52 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:52:52.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:52 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:52:52.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:52 smithi084 ceph-mon[27254]: Health check update: Degraded data redundancy: 72/996 objects degraded (7.229%), 12 pgs degraded (PG_DEGRADED) 2024-09-11T03:52:52.855 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T03:52:52.855 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T03:52:52.892 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:52.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:52.914 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:52.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:52.914 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:52.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:52 smithi175 ceph-mon[30894]: pgmap v1584: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 714 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 8.3 MiB/s rd, 1.4 MiB/s wr, 13 op/s; 72/996 objects degraded (7.229%) 2024-09-11T03:52:52.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:52 smithi175 ceph-mon[30894]: osdmap e953: 12 total, 11 up, 11 in 2024-09-11T03:52:52.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:52 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:52:52.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:52 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:52:52.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:52 smithi175 ceph-mon[30894]: Health check update: Degraded data redundancy: 72/996 objects degraded (7.229%), 12 pgs degraded (PG_DEGRADED) 2024-09-11T03:52:53.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:53.036 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:53.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:53.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:53.138 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:53.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:53.160 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:53.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:53.261 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:53.282 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:53.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:53.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:53.384 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:53.410 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:53.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:53.411 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:53.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:53.521 INFO:tasks.rados.rados.2.smithi175.stdout:3989: done (1 left) 2024-09-11T03:52:53.521 INFO:tasks.rados.rados.2.smithi175.stdout:3990: done (0 left) 2024-09-11T03:52:53.521 INFO:tasks.rados.rados.2.smithi175.stdout:3991: rollback oid 29 current snap is 443 2024-09-11T03:52:53.522 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 29 to 442 2024-09-11T03:52:53.522 INFO:tasks.rados.rados.2.smithi175.stdout:3992: delete oid 34 current snap is 443 2024-09-11T03:52:53.526 INFO:tasks.rados.rados.2.smithi175.stdout:3991: finishing rollback tid 0 to smithi17557870-29 2024-09-11T03:52:53.526 INFO:tasks.rados.rados.2.smithi175.stdout:3991: finishing rollback tid 1 to smithi17557870-29 2024-09-11T03:52:53.526 INFO:tasks.rados.rados.2.smithi175.stdout:3992: done (1 left) 2024-09-11T03:52:53.526 INFO:tasks.rados.rados.2.smithi175.stdout:3993: snap_create 2024-09-11T03:52:53.526 INFO:tasks.rados.rados.2.smithi175.stdout:3991: finishing rollback tid 2 to smithi17557870-29 2024-09-11T03:52:53.526 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 29 v 1268 (ObjNum 1335 snap 436 seq_num 1335) dirty exists 2024-09-11T03:52:53.539 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:53.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:53.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:53.641 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:53.667 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:53.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:53.668 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:53.697 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-11T03:52:53.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:53 smithi121 ceph-mon[30729]: osdmap e954: 12 total, 11 up, 11 in 2024-09-11T03:52:53.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:53 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:52:53.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:53 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:52:53.719 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:53 smithi121 ceph-mon[30729]: osdmap e955: 12 total, 11 up, 11 in 2024-09-11T03:52:53.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:53.790 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:53.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:53.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:53.892 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:53.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:53 smithi084 ceph-mon[27254]: osdmap e954: 12 total, 11 up, 11 in 2024-09-11T03:52:53.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:53 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:52:53.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:53 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:52:53.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:53 smithi084 ceph-mon[27254]: osdmap e955: 12 total, 11 up, 11 in 2024-09-11T03:52:53.914 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:53.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:53.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:53.944 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:52:53.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:53 smithi175 ceph-mon[30894]: osdmap e954: 12 total, 11 up, 11 in 2024-09-11T03:52:53.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:53 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:52:53.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:53 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:52:53.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:53 smithi175 ceph-mon[30894]: osdmap e955: 12 total, 11 up, 11 in 2024-09-11T03:52:54.016 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:54.037 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:54.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:54.139 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:54.160 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:54.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:54.262 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:54.288 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:54.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:54.390 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:54.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:54.413 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:54.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:54.429 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:52:54.429 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:52:54.515 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:54.528 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-11T03:52:54.531 INFO:tasks.rados.rados.2.smithi175.stdout:3991: done (1 left) 2024-09-11T03:52:54.531 INFO:tasks.rados.rados.2.smithi175.stdout:3993: done (0 left) 2024-09-11T03:52:54.531 INFO:tasks.rados.rados.2.smithi175.stdout:3994: read oid 46 snap 434 2024-09-11T03:52:54.532 INFO:tasks.rados.rados.2.smithi175.stdout:3994: expect (ObjNum 1317 snap 430 seq_num 1317) 2024-09-11T03:52:54.532 INFO:tasks.rados.rados.2.smithi175.stdout:3995: rollback oid 37 current snap is 444 2024-09-11T03:52:54.532 INFO:tasks.rados.rados.2.smithi175.stdout:rollback oid 37 to 436 2024-09-11T03:52:54.532 INFO:tasks.rados.rados.2.smithi175.stdout:3996: read oid 24 snap -1 2024-09-11T03:52:54.532 INFO:tasks.rados.rados.2.smithi175.stdout:3996: expect (ObjNum 1353 snap 441 seq_num 1353) 2024-09-11T03:52:54.532 INFO:tasks.rados.rados.2.smithi175.stdout:3997: snap_remove snap 424 2024-09-11T03:52:54.539 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:54.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:54.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:54.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:54 smithi084 ceph-mon[27254]: pgmap v1587: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 739 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 18 op/s; 76/1044 objects degraded (7.280%) 2024-09-11T03:52:54.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:54 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3578855180' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:52:54.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:54 smithi084 ceph-mon[27254]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:54.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:54 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3578855180' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:52:54.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:54 smithi084 ceph-mon[27254]: osdmap e956: 12 total, 11 up, 11 in 2024-09-11T03:52:54.610 INFO:teuthology.orchestra.run.smithi084.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-11T03:52:54.640 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:54.665 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:54.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:54.666 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:54.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:54 smithi121 ceph-mon[30729]: pgmap v1587: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 739 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 18 op/s; 76/1044 objects degraded (7.280%) 2024-09-11T03:52:54.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:54 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3578855180' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:52:54.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:54 smithi121 ceph-mon[30729]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:54.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:54 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3578855180' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:52:54.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:54 smithi121 ceph-mon[30729]: osdmap e956: 12 total, 11 up, 11 in 2024-09-11T03:52:54.766 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:54.790 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:54.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:54.791 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:54.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:52:54.892 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:52:54.915 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:54.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:54.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:52:54.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:54 smithi175 ceph-mon[30894]: pgmap v1587: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 739 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 18 op/s; 76/1044 objects degraded (7.280%) 2024-09-11T03:52:54.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:54 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3578855180' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:52:54.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:54 smithi175 ceph-mon[30894]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:52:54.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:54 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3578855180' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-11T03:52:54.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:54 smithi175 ceph-mon[30894]: osdmap e956: 12 total, 11 up, 11 in 2024-09-11T03:52:55.018 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:55.043 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:55.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:55.044 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:55.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:55.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-11T03:52:55.167 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:55.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:55.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:55.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:55.290 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:55.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:55.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:55.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:55.409 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T03:52:55.414 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:55.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:55.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:55.511 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:55.516 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:55.540 INFO:tasks.rados.rados.2.smithi175.stdout:3997: done (3 left) 2024-09-11T03:52:55.540 INFO:tasks.rados.rados.2.smithi175.stdout:3998: write (excl) oid 12 current snap is 444 2024-09-11T03:52:55.540 INFO:tasks.rados.rados.2.smithi175.stdout:3998: seq_num 1358 ranges {423906=498976,1500194=659130,2609483=214423} 2024-09-11T03:52:55.542 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:55.543 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:55.546 INFO:tasks.rados.rados.2.smithi175.stdout:3998: writing smithi17557870-12 from 423906 to 922882 tid 1 2024-09-11T03:52:55.554 INFO:tasks.rados.rados.2.smithi175.stdout:3998: writing smithi17557870-12 from 1500194 to 2159324 tid 2 2024-09-11T03:52:55.558 INFO:tasks.rados.rados.2.smithi175.stdout:3998: writing smithi17557870-12 from 2609483 to 2823906 tid 3 2024-09-11T03:52:55.558 INFO:tasks.rados.rados.2.smithi175.stdout:3999: write oid 5 current snap is 444 2024-09-11T03:52:55.558 INFO:tasks.rados.rados.2.smithi175.stdout:3999: seq_num 1359 ranges {440713=574225,1442323=519141,2636906=203807} 2024-09-11T03:52:55.565 INFO:tasks.rados.rados.2.smithi175.stdout:3999: writing smithi17557870-5 from 440713 to 1014938 tid 1 2024-09-11T03:52:55.570 INFO:tasks.rados.rados.2.smithi175.stdout:3999: writing smithi17557870-5 from 1442323 to 1961464 tid 2 2024-09-11T03:52:55.575 INFO:tasks.rados.rados.2.smithi175.stdout:3999: writing smithi17557870-5 from 2636906 to 2840713 tid 3 2024-09-11T03:52:55.575 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 5 2024-09-11T03:52:55.575 INFO:tasks.rados.rados.2.smithi175.stdout:3996: expect (ObjNum 1353 snap 441 seq_num 1353) 2024-09-11T03:52:55.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:55.665 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:55.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:55.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:55.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:55.797 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:55.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:55.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:55.823 INFO:tasks.rados.rados.2.smithi175.stdout:3994: expect (ObjNum 1317 snap 430 seq_num 1317) 2024-09-11T03:52:55.899 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:55.921 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:55.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:55.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:3995: finishing rollback tid 1 to smithi17557870-37 2024-09-11T03:52:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 37 v 1267 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-11T03:52:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:3998: finishing write tid 1 to smithi17557870-12 2024-09-11T03:52:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:3998: finishing write tid 2 to smithi17557870-12 2024-09-11T03:52:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:3998: finishing write tid 3 to smithi17557870-12 2024-09-11T03:52:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:3998: finishing write tid 4 to smithi17557870-12 2024-09-11T03:52:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:3998: finishing write tid 5 to smithi17557870-12 2024-09-11T03:52:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:3998: oid 12 updating version 0 to 1684 2024-09-11T03:52:55.932 INFO:tasks.rados.rados.2.smithi175.stdout:3998: oid 12 updating version 1684 to 1685 2024-09-11T03:52:55.933 INFO:tasks.rados.rados.2.smithi175.stdout:3998: oid 12 version 1685 is already newer than 1681 2024-09-11T03:52:55.933 INFO:tasks.rados.rados.2.smithi175.stdout:3998: oid 12 updating version 1685 to 1686 2024-09-11T03:52:55.933 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 12 v 1686 (ObjNum 1358 snap 444 seq_num 1358) dirty exists 2024-09-11T03:52:55.933 INFO:tasks.rados.rados.2.smithi175.stdout:3998: left oid 12 (ObjNum 1358 snap 444 seq_num 1358) 2024-09-11T03:52:55.933 INFO:tasks.rados.rados.2.smithi175.stdout:3999: finishing write tid 1 to smithi17557870-5 2024-09-11T03:52:55.933 INFO:tasks.rados.rados.2.smithi175.stdout:3999: finishing write tid 2 to smithi17557870-5 2024-09-11T03:52:55.933 INFO:tasks.rados.rados.2.smithi175.stdout:3999: finishing write tid 3 to smithi17557870-5 2024-09-11T03:52:55.933 INFO:tasks.rados.rados.2.smithi175.stdout:3994: done (4 left) 2024-09-11T03:52:55.933 INFO:tasks.rados.rados.2.smithi175.stdout:3995: done (3 left) 2024-09-11T03:52:55.933 INFO:tasks.rados.rados.2.smithi175.stdout:3996: done (2 left) 2024-09-11T03:52:55.933 INFO:tasks.rados.rados.2.smithi175.stdout:3998: done (1 left) 2024-09-11T03:52:55.933 INFO:tasks.rados.rados.2.smithi175.stdout: waiting on 1 2024-09-11T03:52:55.933 INFO:tasks.rados.rados.2.smithi175.stdout:3999: finishing write tid 4 to smithi17557870-5 2024-09-11T03:52:55.934 INFO:tasks.rados.rados.2.smithi175.stdout:3999: finishing write tid 5 to smithi17557870-5 2024-09-11T03:52:55.934 INFO:tasks.rados.rados.2.smithi175.stdout:3999: oid 5 updating version 0 to 1392 2024-09-11T03:52:55.934 INFO:tasks.rados.rados.2.smithi175.stdout:3999: oid 5 updating version 1392 to 1394 2024-09-11T03:52:55.934 INFO:tasks.rados.rados.2.smithi175.stdout:3999: oid 5 updating version 1394 to 1395 2024-09-11T03:52:55.934 INFO:tasks.rados.rados.2.smithi175.stdout:3999: oid 5 version 1395 is already newer than 1393 2024-09-11T03:52:55.934 INFO:tasks.rados.rados.2.smithi175.stdout:update_object_version oid 5 v 1395 (ObjNum 1359 snap 444 seq_num 1359) dirty exists 2024-09-11T03:52:55.934 INFO:tasks.rados.rados.2.smithi175.stdout:3999: left oid 5 (ObjNum 1359 snap 444 seq_num 1359) 2024-09-11T03:52:55.934 INFO:tasks.rados.rados.2.smithi175.stdout:3999: done (0 left) 2024-09-11T03:52:55.934 INFO:tasks.rados.rados.2.smithi175.stderr:0 errors. 2024-09-11T03:52:55.935 INFO:tasks.rados.rados.2.smithi175.stderr: 2024-09-11T03:52:55.936 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-11T03:52:55.946 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-09-11T03:52:55.946 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:52:56.022 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:56.043 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:56.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:56.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:56.148 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:56.169 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:56.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:56.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:56.271 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:56.294 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:56.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:56.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:56.348 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-11T03:52:56.352 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:52:56.352 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:52:56.368 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:52:56.368 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:52:56.386 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:52:56.387 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:52:56.396 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:56.404 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:52:56.404 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:52:56.418 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:56.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:56.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:56.424 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:52:56.424 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:52:56.443 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:52:56.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:52:56.445 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:56.460 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:52:56.460 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:52:56.479 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:52:56.480 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:52:56.511 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:52:56.511 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:52:56.520 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:52:56.536 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:52:56.536 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:52:56.541 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:56.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:56.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:52:56.555 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:52:56.555 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:52:56.582 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:56 smithi121 ceph-mon[30729]: pgmap v1590: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 743 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 19 op/s; 77/1047 objects degraded (7.354%) 2024-09-11T03:52:56.582 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:56 smithi121 ceph-mon[30729]: osdmap e957: 12 total, 11 up, 11 in 2024-09-11T03:52:56.582 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:56 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:52:56.582 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:56 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:52:56.582 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:56 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:52:56.582 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:56 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/706088853' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:56.584 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-11T03:52:56.643 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:56.646 INFO:teuthology.orchestra.run.smithi084.stdout: marking collection for removal 2024-09-11T03:52:56.646 INFO:teuthology.orchestra.run.smithi084.stdout:setting '_remove' omap key 2024-09-11T03:52:56.646 INFO:teuthology.orchestra.run.smithi084.stdout:finish_remove_pgs 2.19_head removing 2.19 2024-09-11T03:52:56.648 INFO:teuthology.orchestra.run.smithi084.stdout:Remove successful 2024-09-11T03:52:56.664 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:56.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:56.665 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:56.766 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:56.788 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:56.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:56.789 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:56.844 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:52:56.889 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:56.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:56 smithi084 ceph-mon[27254]: pgmap v1590: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 743 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 19 op/s; 77/1047 objects degraded (7.354%) 2024-09-11T03:52:56.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:56 smithi084 ceph-mon[27254]: osdmap e957: 12 total, 11 up, 11 in 2024-09-11T03:52:56.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:56 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:52:56.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:56 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:52:56.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:56 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:52:56.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/706088853' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:56.910 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:56.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:56.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:56.969 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:56 smithi175 ceph-mon[30894]: pgmap v1590: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 743 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 19 op/s; 77/1047 objects degraded (7.354%) 2024-09-11T03:52:56.969 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:56 smithi175 ceph-mon[30894]: osdmap e957: 12 total, 11 up, 11 in 2024-09-11T03:52:56.969 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:56 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:52:56.969 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:56 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:52:56.969 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:56 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:52:56.969 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:56 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/706088853' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:52:57.012 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:52:57.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:57.034 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:57.034 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:52:57.135 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:57.157 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:57.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:57.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:57.228 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:52:57.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:57.281 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:57.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:57.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:57.357 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:52:57.357 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:52:57.357 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:52:57.357 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:52:57.374 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:52:57.374 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":958,"fsid":"10fe7946-6fee-11ef-bcea-c7b262605968","created":"2024-09-11T03:30:30.532652+0000","modified":"2024-09-11T03:52:56.574221+0000","last_up_change":"2024-09-11T03:52:37.081354+0000","last_in_change":"2024-09-11T03:51:16.200638+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T03:33:42.916933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"220","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.0399999618530273,"score_stable":2.0399999618530273,"optimal_score":1.1100000143051147,"raw_score_acting":2.2599999904632568,"raw_score_stable":2.2599999904632568,"primary_affinity_weighted":0.73786544799804688,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.74000000953674316}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2024-09-11T03:38:28.163534+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":96,"pg_placement_num":96,"pg_placement_num_target":96,"pg_num_target":96,"pg_num_pending":96,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"957","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"933","auid":0,"snap_mode":"selfmanaged","snap_seq":445,"snap_epoch":957,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.1), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"9a431e48-e46b-44f9-bfa7-cb56c1484e1c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":938,"down_at":939,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":1376667411},{"type":"v1","addr":"172.21.15.84:6803","nonce":1376667411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":1376667411},{"type":"v1","addr":"172.21.15.84:6805","nonce":1376667411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":1376667411},{"type":"v1","addr":"172.21.15.84:6809","nonce":1376667411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":1376667411},{"type":"v1","addr":"172.21.15.84:6807","nonce":1376667411}]},"public_addr":"172.21.15.84:6803/1376667411","cluster_addr":"172.21.15.84:6805/1376667411","heartbeat_back_addr":"172.21.15.84:6809/1376667411","heartbeat_front_addr":"172.21.15.84:6807/1376667411","state":["exists"]},{"osd":1,"uuid":"62fd6e25-3508-40b4-a0f4-f7de137e3bb5","up":1,"in":1,"weight":1,"primary_affinity":0.871673583984375,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6811","nonce":3138939249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6813","nonce":3138939249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6817","nonce":3138939249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6815","nonce":3138939249}]},"public_addr":"172.21.15.84:6811/3138939249","cluster_addr":"172.21.15.84:6813/3138939249","heartbeat_back_addr":"172.21.15.84:6817/3138939249","heartbeat_front_addr":"172.21.15.84:6815/3138939249","state":["exists","up"]},{"osd":2,"uuid":"ff351425-4715-49dc-be49-f27bf205d57e","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6819","nonce":3001811591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6821","nonce":3001811591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6825","nonce":3001811591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6823","nonce":3001811591}]},"public_addr":"172.21.15.84:6819/3001811591","cluster_addr":"172.21.15.84:6821/3001811591","heartbeat_back_addr":"172.21.15.84:6825/3001811591","heartbeat_front_addr":"172.21.15.84:6823/3001811591","state":["exists","up"]},{"osd":3,"uuid":"2518a93e-c612-4731-91ce-5685c25ffa9f","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":925,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6827","nonce":302512129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6829","nonce":302512129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6833","nonce":302512129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6831","nonce":302512129}]},"public_addr":"172.21.15.84:6827/302512129","cluster_addr":"172.21.15.84:6829/302512129","heartbeat_back_addr":"172.21.15.84:6833/302512129","heartbeat_front_addr":"172.21.15.84:6831/302512129","state":["exists","up"]},{"osd":4,"uuid":"d12f3c2d-95cc-4276-a7e1-bee982d58948","up":1,"in":0,"weight":0,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":813,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6801","nonce":3873214513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6803","nonce":3873214513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6807","nonce":3873214513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6805","nonce":3873214513}]},"public_addr":"172.21.15.121:6801/3873214513","cluster_addr":"172.21.15.121:6803/3873214513","heartbeat_back_addr":"172.21.15.121:6807/3873214513","heartbeat_front_addr":"172.21.15.121:6805/3873214513","state":["exists","up"]},{"osd":5,"uuid":"32164dd5-7e8f-4925-8526-bbb1e33e3364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":940,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6809","nonce":918612194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6811","nonce":918612194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6815","nonce":918612194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6813","nonce":918612194}]},"public_addr":"172.21.15.121:6809/918612194","cluster_addr":"172.21.15.121:6811/918612194","heartbeat_back_addr":"172.21.15.121:6815/918612194","heartbeat_front_addr":"172.21.15.121:6813/918612194","state":["exists","up"]},{"osd":6,"uuid":"3f107a88-803f-4cd8-804a-241310c282ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":940,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6817","nonce":3384813363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6819","nonce":3384813363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6823","nonce":3384813363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6821","nonce":3384813363}]},"public_addr":"172.21.15.121:6817/3384813363","cluster_addr":"172.21.15.121:6819/3384813363","heartbeat_back_addr":"172.21.15.121:6823/3384813363","heartbeat_front_addr":"172.21.15.121:6821/3384813363","state":["exists","up"]},{"osd":7,"uuid":"dfee7f95-1fb9-4493-9907-cb10d1837bdb","up":1,"in":1,"weight":1,"primary_affinity":0.9827117919921875,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6825","nonce":449222054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6827","nonce":449222054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6831","nonce":449222054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6829","nonce":449222054}]},"public_addr":"172.21.15.121:6825/449222054","cluster_addr":"172.21.15.121:6827/449222054","heartbeat_back_addr":"172.21.15.121:6831/449222054","heartbeat_front_addr":"172.21.15.121:6829/449222054","state":["exists","up"]},{"osd":8,"uuid":"fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":940,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6801","nonce":2660128384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6803","nonce":2660128384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6807","nonce":2660128384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6805","nonce":2660128384}]},"public_addr":"172.21.15.175:6801/2660128384","cluster_addr":"172.21.15.175:6803/2660128384","heartbeat_back_addr":"172.21.15.175:6807/2660128384","heartbeat_front_addr":"172.21.15.175:6805/2660128384","state":["exists","up"]},{"osd":9,"uuid":"50162e42-0443-41d1-8709-a0db61c087da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6809","nonce":1270991649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6811","nonce":1270991649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6815","nonce":1270991649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6813","nonce":1270991649}]},"public_addr":"172.21.15.175:6809/1270991649","cluster_addr":"172.21.15.175:6811/1270991649","heartbeat_back_addr":"172.21.15.175:6815/1270991649","heartbeat_front_addr":"172.21.15.175:6813/1270991649","state":["exists","up"]},{"osd":10,"uuid":"4b9426e1-bac8-47d3-ad58-54b89e8d1eb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6817","nonce":3877631021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6819","nonce":3877631021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6823","nonce":3877631021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6821","nonce":3877631021}]},"public_addr":"172.21.15.175:6817/3877631021","cluster_addr":"172.21.15.175:6819/3877631021","heartbeat_back_addr":"172.21.15.175:6823/3877631021","heartbeat_front_addr":"172.21.15.175:6821/3877631021","state":["exists","up"]},{"osd":11,"uuid":"cd1a28a1-413c-4274-9876-e1bdb7703be6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":939,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6825","nonce":1150051236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6827","nonce":1150051236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6831","nonce":1150051236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6829","nonce":1150051236}]},"public_addr":"172.21.15.175:6825/1150051236","cluster_addr":"172.21.15.175:6827/1150051236","heartbeat_back_addr":"172.21.15.175:6831/1150051236","heartbeat_front_addr":"172.21.15.175:6829/1150051236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:52:30.415469+0000","dead_epoch":938},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:52:51.904918+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:52:18.840183+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:51:36.033322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-11T03:52:20.117291+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:52:25.176259+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:52:04.207144+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:52:31.797402+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:52:49.310850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:52:55.618592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:52:39.633102+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:52:52.727582+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.16","osds":[11,2,0]}],"pg_upmap_items":[{"pgid":"1.4","mappings":[{"from":7,"to":2}]},{"pgid":"2.0","mappings":[{"from":7,"to":2}]},{"pgid":"2.9","mappings":[{"from":7,"to":0}]},{"pgid":"2.b","mappings":[{"from":7,"to":0}]},{"pgid":"2.d","mappings":[{"from":9,"to":5}]},{"pgid":"2.1b","mappings":[{"from":7,"to":2}]},{"pgid":"2.2e","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"2.56","osds":[6,7,5]},{"pgid":"2.5a","osds":[5,3,2]},{"pgid":"2.5c","osds":[8,1,7]}],"primary_temp":[],"blocklist":{"172.21.15.84:0/108136515":"2024-09-12T03:31:26.598965+0000","172.21.15.84:0/3411515187":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6801/3365920551":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6800/3365920551":"2024-09-12T03:31:26.598965+0000","172.21.15.84:0/3885577441":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/22039254":"2024-09-12T03:31:03.077186+0000","172.21.15.84:6801/3496645948":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/4206337830":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6800/3496645948":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/1989775875":"2024-09-12T03:31:03.077186+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":351,"length":1},{"begin":373,"length":1},{"begin":376,"length":1},{"begin":403,"length":1},{"begin":426,"length":1},{"begin":434,"length":1},{"begin":439,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T03:52:57.383 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:57.405 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:57.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:57.507 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:57.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:57.530 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:57.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:57.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:57.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:57 smithi084 ceph-mon[27254]: osdmap e958: 12 total, 11 up, 11 in 2024-09-11T03:52:57.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:57 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:52:57.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:57 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:52:57.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:57 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:52:57.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:57 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:52:57.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:52:57.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:57 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:52:57.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:57 smithi084 ceph-mon[27254]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:52:57.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:57 smithi084 ceph-mon[27254]: Health check update: Degraded data redundancy: 77/1047 objects degraded (7.354%), 12 pgs degraded (PG_DEGRADED) 2024-09-11T03:52:57.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:57 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1828947665' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:52:57.652 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:57.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:57.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:57.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:52:57.775 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:57.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:52:57.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:57.898 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:57.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:57.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:57 smithi121 ceph-mon[30729]: osdmap e958: 12 total, 11 up, 11 in 2024-09-11T03:52:57.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:57 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:52:57.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:57 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:52:57.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:57 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:52:57.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:57 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:52:57.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:57 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:52:57.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:57 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:52:57.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:57 smithi121 ceph-mon[30729]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:52:57.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:57 smithi121 ceph-mon[30729]: Health check update: Degraded data redundancy: 77/1047 objects degraded (7.354%), 12 pgs degraded (PG_DEGRADED) 2024-09-11T03:52:57.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:57 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1828947665' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:52:57.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:57 smithi175 ceph-mon[30894]: osdmap e958: 12 total, 11 up, 11 in 2024-09-11T03:52:57.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:57 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:52:57.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:57 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:52:57.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:57 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:52:57.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:57 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:52:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:57 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:52:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:57 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:52:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:57 smithi175 ceph-mon[30894]: from='mgr.14150 ' entity='mgr.y' 2024-09-11T03:52:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:57 smithi175 ceph-mon[30894]: Health check update: Degraded data redundancy: 77/1047 objects degraded (7.354%), 12 pgs degraded (PG_DEGRADED) 2024-09-11T03:52:57.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:57 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1828947665' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:52:58.001 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:58.025 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:58.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:58.026 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:58.056 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-11T03:52:58.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:58.150 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:58.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:58.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:58.252 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:58.275 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:58.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:58.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:58.279 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T03:52:58.279 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T03:52:58.338 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:52:58.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:58.399 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:58.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:58.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:58.501 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:58.527 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:58.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:58.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:58.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:52:58.652 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:58.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:58.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:52:58.713 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:52:58.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:58 smithi175 ceph-mon[30894]: pgmap v1593: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 743 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 255 B/s rd, 335 KiB/s wr, 2 op/s; 77/1050 objects degraded (7.333%) 2024-09-11T03:52:58.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:52:58 smithi175 ceph-mon[30894]: 2.57 scrub starts 2024-09-11T03:52:58.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:52:58.778 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:58.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:58.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:52:58.859 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:52:58.859 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:52:58.859 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:52:58.859 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:52:58.859 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:52:58.880 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:58.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:58 smithi084 ceph-mon[27254]: pgmap v1593: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 743 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 255 B/s rd, 335 KiB/s wr, 2 op/s; 77/1050 objects degraded (7.333%) 2024-09-11T03:52:58.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:52:58 smithi084 ceph-mon[27254]: 2.57 scrub starts 2024-09-11T03:52:58.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:58.903 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:58.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:58.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:58 smithi121 ceph-mon[30729]: pgmap v1593: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 743 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 255 B/s rd, 335 KiB/s wr, 2 op/s; 77/1050 objects degraded (7.333%) 2024-09-11T03:52:58.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:52:58 smithi121 ceph-mon[30729]: 2.57 scrub starts 2024-09-11T03:52:59.005 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:59.020 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell -n osd.0 --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.19.0 2024-09-11T03:52:59.027 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:59.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:59.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:59.052 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:59.069 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:52:59.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:52:59.154 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:52:59.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:59.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:52:59.256 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:59.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:59.279 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:59.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:59.299 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/osd.0/config 2024-09-11T03:52:59.380 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:52:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:59.403 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:59.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:52:59.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:59.526 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:59.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:59.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:59.596 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-11T03:52:59.628 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:52:59.649 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:59.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:59.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:52:59.751 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:52:59.771 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:52:59.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:59.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:52:59.836 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:52:59.873 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:52:59.894 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:52:59.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:52:59.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:52:59.996 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:53:00.016 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:00.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:00.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:53:00.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:53:00.145 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:00.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:00.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:53:00.247 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:53:00.269 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:00.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:00.270 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:53:00.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T03:53:00.353 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:53:00.354 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:53:00.354 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:53:00.354 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:53:00.354 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.354 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4164 2.b 2:dde92fc6:::smithi17557870-41:head [copy-from ver 223] snapc b4=[b4,b2,b1,af,ae,ac,ab,a9,a8,9d] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-09-11T03:53:00.354 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:27.930005+0000", 2024-09-11T03:53:00.354 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 512.42313737799998, 2024-09-11T03:53:00.354 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014694021999999999, 2024-09-11T03:53:00.355 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.355 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.355 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:00.355 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:00.355 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:00.355 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:00.355 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4164 2024-09-11T03:53:00.355 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.355 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.355 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.355 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.356 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930005+0000", 2024-09-11T03:53:00.356 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.356 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.356 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.356 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.356 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930005+0000", 2024-09-11T03:53:00.356 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.356 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.356 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.356 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.356 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930006+0000", 2024-09-11T03:53:00.357 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.466e-06 2024-09-11T03:53:00.357 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.357 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.357 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.357 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930011+0000", 2024-09-11T03:53:00.357 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9930000000000003e-06 2024-09-11T03:53:00.357 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.357 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.357 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.357 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930022+0000", 2024-09-11T03:53:00.357 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0831e-05 2024-09-11T03:53:00.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.358 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.358 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930029+0000", 2024-09-11T03:53:00.358 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.776e-06 2024-09-11T03:53:00.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.358 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.358 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930083+0000", 2024-09-11T03:53:00.358 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4586000000000002e-05 2024-09-11T03:53:00.359 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.359 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.359 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.359 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.930150+0000", 2024-09-11T03:53:00.359 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6595999999999997e-05 2024-09-11T03:53:00.359 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.359 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.359 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,11", 2024-09-11T03:53:00.359 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.936619+0000", 2024-09-11T03:53:00.359 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0064690360000000001 2024-09-11T03:53:00.359 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.359 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.360 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:00.360 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.942717+0000", 2024-09-11T03:53:00.360 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0060981550000000001 2024-09-11T03:53:00.360 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.360 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.360 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:00.360 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.943814+0000", 2024-09-11T03:53:00.360 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001097251 2024-09-11T03:53:00.360 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.360 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.360 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:00.361 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.944627+0000", 2024-09-11T03:53:00.361 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00081236800000000005 2024-09-11T03:53:00.361 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.361 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.361 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:00.361 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.944664+0000", 2024-09-11T03:53:00.361 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7282e-05 2024-09-11T03:53:00.361 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.361 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.361 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.361 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:27.944699+0000", 2024-09-11T03:53:00.361 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4681999999999998e-05 2024-09-11T03:53:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4700 2.5 2:a1a99782:::smithi17557870-1:head [copy-from ver 541] snapc cd=[cd,cc,ca,c5,bf,b1] ondisk+write+known_if_redirected+supports_pool_eio e504)", 2024-09-11T03:53:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:21.960471+0000", 2024-09-11T03:53:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 458.392671302, 2024-09-11T03:53:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015308754000000001, 2024-09-11T03:53:00.362 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4700 2024-09-11T03:53:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.363 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960471+0000", 2024-09-11T03:53:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960471+0000", 2024-09-11T03:53:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960473+0000", 2024-09-11T03:53:00.364 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8050000000000001e-06 2024-09-11T03:53:00.365 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.365 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.365 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.365 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960480+0000", 2024-09-11T03:53:00.365 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3780000000000003e-06 2024-09-11T03:53:00.365 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.365 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.365 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.365 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960497+0000", 2024-09-11T03:53:00.365 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7224e-05 2024-09-11T03:53:00.365 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.365 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.366 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.366 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960504+0000", 2024-09-11T03:53:00.366 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2620000000000001e-06 2024-09-11T03:53:00.366 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.366 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.366 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.366 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960575+0000", 2024-09-11T03:53:00.366 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0507999999999996e-05 2024-09-11T03:53:00.366 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.367 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.367 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.367 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.960757+0000", 2024-09-11T03:53:00.367 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000182325 2024-09-11T03:53:00.367 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.367 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.367 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:53:00.367 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.967152+0000", 2024-09-11T03:53:00.367 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0063946519999999998 2024-09-11T03:53:00.368 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.368 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.368 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:00.368 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.972224+0000", 2024-09-11T03:53:00.368 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050720349999999999 2024-09-11T03:53:00.368 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.368 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.368 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:00.368 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.974955+0000", 2024-09-11T03:53:00.368 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0027314829999999998 2024-09-11T03:53:00.368 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.369 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.369 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:00.369 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.975676+0000", 2024-09-11T03:53:00.369 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00072077700000000003 2024-09-11T03:53:00.369 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.369 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.369 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:00.369 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.975727+0000", 2024-09-11T03:53:00.369 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0315999999999997e-05 2024-09-11T03:53:00.369 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.369 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.369 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.370 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:21.975779+0000", 2024-09-11T03:53:00.370 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2988999999999997e-05 2024-09-11T03:53:00.370 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.370 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.370 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.370 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.370 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.370 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6565 2.5 2:a51832fe:::smithi17557870-12:head [copy-from ver 930] snapc 120=[120,11f,11e,11d,11a,119,114] ondisk+write+known_if_redirected+supports_pool_eio e670)", 2024-09-11T03:53:00.370 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:08.152253+0000", 2024-09-11T03:53:00.370 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 292.20088886799999, 2024-09-11T03:53:00.370 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021627759, 2024-09-11T03:53:00.371 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.371 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.371 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:00.371 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:00.371 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:00.371 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:00.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4 2024-09-11T03:53:00.372 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6565 2024-09-11T03:53:00.373 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.373 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.373 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.373 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.373 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152253+0000", 2024-09-11T03:53:00.373 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.373 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.373 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.373 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.373 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152253+0000", 2024-09-11T03:53:00.373 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.373 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.374 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.374 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.374 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152255+0000", 2024-09-11T03:53:00.374 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9860000000000001e-06 2024-09-11T03:53:00.374 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.374 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.374 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.374 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152264+0000", 2024-09-11T03:53:00.374 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6070000000000008e-06 2024-09-11T03:53:00.374 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.374 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.375 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.375 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152283+0000", 2024-09-11T03:53:00.375 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9578000000000001e-05 2024-09-11T03:53:00.375 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.375 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.375 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.375 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152292+0000", 2024-09-11T03:53:00.375 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.0469999999999999e-06 2024-09-11T03:53:00.375 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.375 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.375 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.375 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152369+0000", 2024-09-11T03:53:00.376 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6310999999999995e-05 2024-09-11T03:53:00.376 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.376 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.376 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.376 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152491+0000", 2024-09-11T03:53:00.376 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000122712 2024-09-11T03:53:00.376 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.376 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.376 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:53:00.376 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.161123+0000", 2024-09-11T03:53:00.376 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.008631906 2024-09-11T03:53:00.377 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.377 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.377 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:00.377 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.171985+0000", 2024-09-11T03:53:00.377 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010861572 2024-09-11T03:53:00.377 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.377 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.377 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:00.377 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.173347+0000", 2024-09-11T03:53:00.377 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00136213 2024-09-11T03:53:00.378 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.378 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.378 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:00.378 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.173764+0000", 2024-09-11T03:53:00.378 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00041722999999999999 2024-09-11T03:53:00.378 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.378 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.378 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:00.379 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.173827+0000", 2024-09-11T03:53:00.379 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2780999999999994e-05 2024-09-11T03:53:00.379 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.379 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.379 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.379 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.173881+0000", 2024-09-11T03:53:00.379 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3899e-05 2024-09-11T03:53:00.380 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.380 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.380 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.380 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.380 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.380 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6566 2.5 2:a51832fe:::smithi17557870-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e670)", 2024-09-11T03:53:00.380 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:08.152319+0000", 2024-09-11T03:53:00.380 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 292.20082265799999, 2024-09-11T03:53:00.380 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015878518000000001, 2024-09-11T03:53:00.381 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.381 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.381 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:00.381 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:00.381 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:00.381 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:00.381 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6566 2024-09-11T03:53:00.381 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.381 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.381 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.381 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.382 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152319+0000", 2024-09-11T03:53:00.382 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.382 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.382 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.382 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.382 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152319+0000", 2024-09-11T03:53:00.382 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.383 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.383 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.383 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.383 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152320+0000", 2024-09-11T03:53:00.383 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6000000000000004e-07 2024-09-11T03:53:00.383 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.383 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.383 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.383 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152323+0000", 2024-09-11T03:53:00.383 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8930000000000001e-06 2024-09-11T03:53:00.383 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.383 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.384 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.384 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152338+0000", 2024-09-11T03:53:00.384 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5197e-05 2024-09-11T03:53:00.384 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.384 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.384 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.384 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152345+0000", 2024-09-11T03:53:00.384 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4790000000000001e-06 2024-09-11T03:53:00.384 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.384 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.384 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.385 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152651+0000", 2024-09-11T03:53:00.385 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000306206 2024-09-11T03:53:00.385 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.385 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.385 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:53:00.385 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152707+0000", 2024-09-11T03:53:00.385 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6200999999999999e-05 2024-09-11T03:53:00.385 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.385 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.385 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.385 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.167971+0000", 2024-09-11T03:53:00.386 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015263914999999999 2024-09-11T03:53:00.386 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.386 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.386 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.386 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.168075+0000", 2024-09-11T03:53:00.386 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000104199 2024-09-11T03:53:00.386 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.386 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.386 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.386 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.168198+0000", 2024-09-11T03:53:00.386 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012286799999999999 2024-09-11T03:53:00.387 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.387 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.387 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.387 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.387 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.387 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7235 2.5 2:a51832fe:::smithi17557870-12:head [copy-from ver 950] snapc 141=[141,140,13f,13d,13c,139,138,137,136,135,133,12e,12d,12a,127,126,120,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-09-11T03:53:00.387 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:49:01.070004+0000", 2024-09-11T03:53:00.387 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 239.28313769100001, 2024-09-11T03:53:00.387 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.01550867, 2024-09-11T03:53:00.387 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.387 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.388 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:00.388 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:00.388 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:00.388 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:00.388 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7235 2024-09-11T03:53:00.388 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.388 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.388 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.388 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.388 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.070004+0000", 2024-09-11T03:53:00.388 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.389 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.389 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.389 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.389 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.070004+0000", 2024-09-11T03:53:00.389 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.389 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.389 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.389 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.389 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.070005+0000", 2024-09-11T03:53:00.389 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0809999999999999e-06 2024-09-11T03:53:00.389 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.390 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.390 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.390 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.070010+0000", 2024-09-11T03:53:00.390 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9370000000000003e-06 2024-09-11T03:53:00.390 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.390 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.390 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.390 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.070020+0000", 2024-09-11T03:53:00.390 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7999999999999993e-06 2024-09-11T03:53:00.390 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.390 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.390 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.391 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.070026+0000", 2024-09-11T03:53:00.391 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5949999999999998e-06 2024-09-11T03:53:00.391 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.391 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.391 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.391 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.070094+0000", 2024-09-11T03:53:00.391 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8169000000000004e-05 2024-09-11T03:53:00.391 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.391 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.391 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.392 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.070194+0000", 2024-09-11T03:53:00.392 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9807000000000002e-05 2024-09-11T03:53:00.392 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.392 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.392 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,4", 2024-09-11T03:53:00.392 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.074863+0000", 2024-09-11T03:53:00.392 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0046691559999999998 2024-09-11T03:53:00.392 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.392 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.392 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:00.393 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.078953+0000", 2024-09-11T03:53:00.393 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0040905489999999997 2024-09-11T03:53:00.393 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.393 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.393 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:00.393 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.084272+0000", 2024-09-11T03:53:00.393 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0053186539999999999 2024-09-11T03:53:00.393 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.393 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.393 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:00.393 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.085295+0000", 2024-09-11T03:53:00.394 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0010226339999999999 2024-09-11T03:53:00.394 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.394 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.394 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:00.394 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.085409+0000", 2024-09-11T03:53:00.394 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000113828 2024-09-11T03:53:00.394 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.394 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.394 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.394 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:01.085513+0000", 2024-09-11T03:53:00.395 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010446 2024-09-11T03:53:00.395 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.395 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.395 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.395 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.395 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.395 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "MOSDPGPush(2.29 749/748 [PushOp(2:95fa802a:::smithi17557870-22:145, version: 743'1144, data_included: [720896~720896,1900544~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:95fa802a:::smithi17557870-22:145@743'1144, size: 3933003, copy_subset: [(0, 3933003)], clone_subset: {2:95fa802a:::smithi17557870-22:12a: []}, snapset: 145=[]:{11b: [119], 127: [127, 120, 11e], 12a: [12a], 145: [145, 144, 143, 142, 13d, 13c, 139, 138, 136, 135, 133, 12e]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2228224, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T03:53:00.395 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:49:27.299647+0000", 2024-09-11T03:53:00.395 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 213.05349551099999, 2024-09-11T03:53:00.396 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.037766238000000001, 2024-09-11T03:53:00.396 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.396 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.396 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:00.396 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.396 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.396 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.396 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:27.299647+0000", 2024-09-11T03:53:00.396 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.397 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.397 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.397 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.397 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:27.299647+0000", 2024-09-11T03:53:00.397 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.397 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.397 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.397 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.397 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:27.299648+0000", 2024-09-11T03:53:00.397 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.048e-06 2024-09-11T03:53:00.397 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.398 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.398 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.398 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:27.336866+0000", 2024-09-11T03:53:00.398 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.037218564000000003 2024-09-11T03:53:00.398 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.398 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.398 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.398 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:27.336882+0000", 2024-09-11T03:53:00.398 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5517000000000001e-05 2024-09-11T03:53:00.398 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.398 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.398 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.399 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:27.336893+0000", 2024-09-11T03:53:00.399 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1578e-05 2024-09-11T03:53:00.399 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.399 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.399 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.399 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:27.336984+0000", 2024-09-11T03:53:00.399 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.0816999999999995e-05 2024-09-11T03:53:00.399 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.399 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.399 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.399 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:27.336999+0000", 2024-09-11T03:53:00.399 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4474000000000001e-05 2024-09-11T03:53:00.399 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.400 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.400 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.400 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:27.337413+0000", 2024-09-11T03:53:00.400 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00041424000000000002 2024-09-11T03:53:00.400 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.400 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.400 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.400 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.400 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.400 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7510 2.25 2:a6e3bbd9:::smithi17557870-38:head [copy-from ver 975] snapc 155=[155,154,153,151,150,14f,14e,14d,14b,149,148,144,143,142,13d,13c,139,136,135,133,12a,127,120,119] ondisk+write+known_if_redirected+supports_pool_eio e759)", 2024-09-11T03:53:00.400 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:49:37.021210+0000", 2024-09-11T03:53:00.401 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 203.33193231999999, 2024-09-11T03:53:00.401 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019390223000000002, 2024-09-11T03:53:00.401 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.401 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.401 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:00.401 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:00.401 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:00.401 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:00.401 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7510 2024-09-11T03:53:00.401 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.402 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.402 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.402 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.402 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.021210+0000", 2024-09-11T03:53:00.402 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.402 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.402 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.402 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.402 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.021210+0000", 2024-09-11T03:53:00.402 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.402 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.402 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.403 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.403 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.021212+0000", 2024-09-11T03:53:00.403 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.911e-06 2024-09-11T03:53:00.403 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.403 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.403 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.403 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.021228+0000", 2024-09-11T03:53:00.403 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6803000000000002e-05 2024-09-11T03:53:00.403 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.403 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.403 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.403 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.021242+0000", 2024-09-11T03:53:00.404 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3788000000000001e-05 2024-09-11T03:53:00.404 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.404 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.404 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.404 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.021250+0000", 2024-09-11T03:53:00.404 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0080000000000002e-06 2024-09-11T03:53:00.404 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.404 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.404 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.404 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.021310+0000", 2024-09-11T03:53:00.404 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9383999999999999e-05 2024-09-11T03:53:00.405 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.405 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.405 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.405 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.021381+0000", 2024-09-11T03:53:00.405 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1355999999999999e-05 2024-09-11T03:53:00.405 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.405 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.405 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 6,11", 2024-09-11T03:53:00.405 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.031851+0000", 2024-09-11T03:53:00.405 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010470348000000001 2024-09-11T03:53:00.405 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.406 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.406 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:00.406 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.035638+0000", 2024-09-11T03:53:00.406 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0037863620000000001 2024-09-11T03:53:00.406 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.406 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.406 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:00.406 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.038102+0000", 2024-09-11T03:53:00.406 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024639890000000002 2024-09-11T03:53:00.406 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.406 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.406 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:00.407 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.040517+0000", 2024-09-11T03:53:00.407 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00241522 2024-09-11T03:53:00.407 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.407 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.407 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:00.407 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.040559+0000", 2024-09-11T03:53:00.407 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2463000000000003e-05 2024-09-11T03:53:00.407 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.407 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.407 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.408 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.040600+0000", 2024-09-11T03:53:00.408 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0590999999999998e-05 2024-09-11T03:53:00.408 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.408 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.408 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.408 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.408 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.408 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:8473 2.25 2:a6e3bbd9:::smithi17557870-38:head [copy-from ver 975] snapc 17c=[17c,17b,17a,179,178,177,176,175,174,173,172,171,170,16f,16e,16b,169,168,167,164,15f,15e,15d,15b,15a,157,156,155,14d,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e828)", 2024-09-11T03:53:00.408 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:50:46.051269+0000", 2024-09-11T03:53:00.408 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 134.301872632, 2024-09-11T03:53:00.408 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011897347000000001, 2024-09-11T03:53:00.409 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.409 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.409 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:00.409 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:00.409 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:00.409 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:00.409 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 8473 2024-09-11T03:53:00.409 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.409 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.409 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.410 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.410 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:46.051269+0000", 2024-09-11T03:53:00.410 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.410 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.410 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.410 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.410 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:46.051269+0000", 2024-09-11T03:53:00.410 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.410 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.410 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.410 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.411 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:46.051271+0000", 2024-09-11T03:53:00.411 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.79e-06 2024-09-11T03:53:00.411 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.411 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.411 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.411 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:46.051279+0000", 2024-09-11T03:53:00.411 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6380000000000004e-06 2024-09-11T03:53:00.411 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.411 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.411 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.412 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:46.051292+0000", 2024-09-11T03:53:00.412 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2782000000000001e-05 2024-09-11T03:53:00.412 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.412 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.412 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.412 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:46.051300+0000", 2024-09-11T03:53:00.412 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7269999999999996e-06 2024-09-11T03:53:00.412 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.412 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.412 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.412 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:46.051352+0000", 2024-09-11T03:53:00.413 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1932999999999999e-05 2024-09-11T03:53:00.413 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.413 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.413 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.413 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:46.051420+0000", 2024-09-11T03:53:00.413 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7395000000000001e-05 2024-09-11T03:53:00.413 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.413 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.413 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 6,11", 2024-09-11T03:53:00.413 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:46.057250+0000", 2024-09-11T03:53:00.413 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0058303959999999998 2024-09-11T03:53:00.414 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.414 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.414 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:00.414 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:46.060756+0000", 2024-09-11T03:53:00.414 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035061010000000002 2024-09-11T03:53:00.414 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.414 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.414 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:00.414 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:46.061549+0000", 2024-09-11T03:53:00.414 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00079297799999999996 2024-09-11T03:53:00.414 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.415 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.415 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:00.415 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:46.063090+0000", 2024-09-11T03:53:00.415 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001541204 2024-09-11T03:53:00.415 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.415 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.415 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:00.415 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:46.063125+0000", 2024-09-11T03:53:00.415 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4755000000000002e-05 2024-09-11T03:53:00.415 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.415 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.416 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.416 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:46.063167+0000", 2024-09-11T03:53:00.416 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.1647999999999998e-05 2024-09-11T03:53:00.416 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.416 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.416 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.416 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.416 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.416 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(osd.0.0:402 2.1f e862/809 2:fa239c32:::smithi17557870-4:14d v 862'809, mlcod=860'805)", 2024-09-11T03:53:00.416 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:51:20.059017+0000", 2024-09-11T03:53:00.417 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 100.29412476900001, 2024-09-11T03:53:00.417 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013462399, 2024-09-11T03:53:00.417 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.417 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.417 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:00.417 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.417 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.417 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.417 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.059017+0000", 2024-09-11T03:53:00.417 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.417 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.418 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.418 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.418 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.059017+0000", 2024-09-11T03:53:00.418 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.418 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.418 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.418 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.418 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.059018+0000", 2024-09-11T03:53:00.418 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8400000000000003e-07 2024-09-11T03:53:00.418 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.418 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.419 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.419 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.059025+0000", 2024-09-11T03:53:00.419 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0589999999999997e-06 2024-09-11T03:53:00.419 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.419 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.419 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.419 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.059034+0000", 2024-09-11T03:53:00.419 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2140000000000002e-06 2024-09-11T03:53:00.419 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.419 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.419 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.420 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.059042+0000", 2024-09-11T03:53:00.420 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4309999999999997e-06 2024-09-11T03:53:00.420 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.420 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.420 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.420 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.064215+0000", 2024-09-11T03:53:00.420 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0051731679999999997 2024-09-11T03:53:00.420 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.420 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.420 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.420 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.064237+0000", 2024-09-11T03:53:00.421 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1727000000000001e-05 2024-09-11T03:53:00.421 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.421 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.421 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:00.421 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.072435+0000", 2024-09-11T03:53:00.421 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0081982519999999996 2024-09-11T03:53:00.421 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.421 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.421 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.421 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.072480+0000", 2024-09-11T03:53:00.421 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4764e-05 2024-09-11T03:53:00.422 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.422 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.422 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.422 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.422 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.422 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(osd.0.0:403 2.1f e862/809 2:fba9fb28:::smithi17557870-48:14d v 862'811, mlcod=860'805)", 2024-09-11T03:53:00.422 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:51:20.062864+0000", 2024-09-11T03:53:00.422 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 100.290278246, 2024-09-11T03:53:00.422 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011541793999999999, 2024-09-11T03:53:00.422 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.422 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.423 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:00.423 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.423 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.423 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.423 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.062864+0000", 2024-09-11T03:53:00.423 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.423 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.423 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.423 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.423 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.062864+0000", 2024-09-11T03:53:00.423 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.423 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.424 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.424 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.424 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.062865+0000", 2024-09-11T03:53:00.424 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1399999999999996e-07 2024-09-11T03:53:00.424 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.424 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.424 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.424 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.062870+0000", 2024-09-11T03:53:00.424 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8610000000000003e-06 2024-09-11T03:53:00.424 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.424 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.425 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.425 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.062880+0000", 2024-09-11T03:53:00.425 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9639999999999998e-06 2024-09-11T03:53:00.425 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.425 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.425 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.425 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.062887+0000", 2024-09-11T03:53:00.425 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7819999999999996e-06 2024-09-11T03:53:00.425 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.425 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.425 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.426 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.070826+0000", 2024-09-11T03:53:00.426 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0079384720000000002 2024-09-11T03:53:00.426 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.426 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.426 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.426 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.070846+0000", 2024-09-11T03:53:00.426 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0551e-05 2024-09-11T03:53:00.426 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.426 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.426 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:00.426 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.074368+0000", 2024-09-11T03:53:00.427 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035223519999999999 2024-09-11T03:53:00.427 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.427 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.427 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.427 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.074406+0000", 2024-09-11T03:53:00.427 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7098000000000003e-05 2024-09-11T03:53:00.427 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.427 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.427 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.427 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.427 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.428 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:9513 2.5 2:a1a99782:::smithi17557870-1:head [copy-from ver 1007] snapc 1aa=[1aa,1a9,1a8,1a7,1a6,1a5,1a4,1a3,1a2,1a1,1a0,19e,19d,19c,19b,199,198,197,195,194,193,192,191,18b,183,17e,17c,17a,178,175,172,16e,16b,169,164,15f,13c,127] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-09-11T03:53:00.428 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:52:15.170949+0000", 2024-09-11T03:53:00.428 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 45.182193337999998, 2024-09-11T03:53:00.428 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.018449274000000002, 2024-09-11T03:53:00.428 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.428 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.428 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:00.428 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:00.428 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:00.428 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:00.428 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 9513 2024-09-11T03:53:00.429 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.429 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.429 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.429 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.429 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.170949+0000", 2024-09-11T03:53:00.429 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.429 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.429 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.429 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.429 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.170949+0000", 2024-09-11T03:53:00.429 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.429 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.430 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.430 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.430 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.170951+0000", 2024-09-11T03:53:00.430 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0169999999999999e-06 2024-09-11T03:53:00.430 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.430 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.430 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.430 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.170958+0000", 2024-09-11T03:53:00.430 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6839999999999996e-06 2024-09-11T03:53:00.430 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.430 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.431 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.431 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.170978+0000", 2024-09-11T03:53:00.431 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.9457e-05 2024-09-11T03:53:00.431 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.431 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.431 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.431 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.170986+0000", 2024-09-11T03:53:00.431 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9189999999999999e-06 2024-09-11T03:53:00.431 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.431 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.431 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.432 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.171050+0000", 2024-09-11T03:53:00.432 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3810000000000001e-05 2024-09-11T03:53:00.432 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.432 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.432 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.432 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.171137+0000", 2024-09-11T03:53:00.432 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.7056000000000001e-05 2024-09-11T03:53:00.432 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.432 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.432 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:53:00.432 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.179853+0000", 2024-09-11T03:53:00.433 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0087166640000000007 2024-09-11T03:53:00.433 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.433 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.433 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:00.433 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.184174+0000", 2024-09-11T03:53:00.433 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0043211270000000001 2024-09-11T03:53:00.433 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.433 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.433 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:00.433 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.189191+0000", 2024-09-11T03:53:00.433 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050169230000000004 2024-09-11T03:53:00.434 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.434 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.434 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:00.434 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.189288+0000", 2024-09-11T03:53:00.434 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6415000000000004e-05 2024-09-11T03:53:00.434 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.434 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.434 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:00.434 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.189365+0000", 2024-09-11T03:53:00.435 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7626999999999994e-05 2024-09-11T03:53:00.435 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.435 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.435 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.435 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.189398+0000", 2024-09-11T03:53:00.435 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.2574999999999999e-05 2024-09-11T03:53:00.435 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.435 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.435 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.435 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.435 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.435 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:9514 2.5 2:a1a99782:::smithi17557870-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e917)", 2024-09-11T03:53:00.436 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:52:15.171005+0000", 2024-09-11T03:53:00.436 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 45.182137418000003, 2024-09-11T03:53:00.436 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012533371999999999, 2024-09-11T03:53:00.436 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.436 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.436 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:00.436 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:00.436 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:00.436 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:00.436 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 9514 2024-09-11T03:53:00.436 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.437 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.437 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.437 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.437 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.171005+0000", 2024-09-11T03:53:00.437 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.437 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.437 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.437 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.437 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.171005+0000", 2024-09-11T03:53:00.437 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.437 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.438 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.438 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.438 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.171005+0000", 2024-09-11T03:53:00.438 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9699999999999996e-07 2024-09-11T03:53:00.438 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.438 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.438 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.438 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.171008+0000", 2024-09-11T03:53:00.438 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5440000000000001e-06 2024-09-11T03:53:00.438 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.438 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.438 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.438 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.171016+0000", 2024-09-11T03:53:00.439 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8439999999999998e-06 2024-09-11T03:53:00.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.439 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.439 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.439 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.171021+0000", 2024-09-11T03:53:00.439 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1109999999999999e-06 2024-09-11T03:53:00.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.439 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.439 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.439 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.171250+0000", 2024-09-11T03:53:00.439 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000229087 2024-09-11T03:53:00.440 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.440 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.440 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:53:00.440 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.171283+0000", 2024-09-11T03:53:00.440 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3694e-05 2024-09-11T03:53:00.440 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.440 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.440 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.440 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.183386+0000", 2024-09-11T03:53:00.440 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012102742 2024-09-11T03:53:00.440 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.440 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.441 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.441 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.183466+0000", 2024-09-11T03:53:00.441 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0210999999999995e-05 2024-09-11T03:53:00.441 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.441 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.441 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.441 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:15.183538+0000", 2024-09-11T03:53:00.441 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1642000000000003e-05 2024-09-11T03:53:00.442 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.443 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.443 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.443 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.443 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.443 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(osd.0.0:608 2.2e e935/872 2:77dadb97:::smithi17557870-30:1af v 935'1271, mlcod=934'1268)", 2024-09-11T03:53:00.443 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:52:33.084817+0000", 2024-09-11T03:53:00.443 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 27.268325294, 2024-09-11T03:53:00.443 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011318913, 2024-09-11T03:53:00.443 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.444 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.444 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:00.444 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.444 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.444 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.444 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:33.084817+0000", 2024-09-11T03:53:00.444 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.444 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.444 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.444 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.444 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:33.084817+0000", 2024-09-11T03:53:00.444 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.445 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.445 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.445 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.445 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:33.084817+0000", 2024-09-11T03:53:00.445 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7e-07 2024-09-11T03:53:00.445 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.445 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.445 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.445 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:33.084832+0000", 2024-09-11T03:53:00.445 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4549000000000001e-05 2024-09-11T03:53:00.445 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.445 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.446 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.446 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:33.084841+0000", 2024-09-11T03:53:00.446 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2660000000000007e-06 2024-09-11T03:53:00.446 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.446 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.446 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.446 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:33.084852+0000", 2024-09-11T03:53:00.446 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1450999999999999e-05 2024-09-11T03:53:00.446 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.446 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.446 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.446 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:33.088845+0000", 2024-09-11T03:53:00.447 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039925919999999997 2024-09-11T03:53:00.447 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.447 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.447 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.447 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:33.088866+0000", 2024-09-11T03:53:00.447 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.1175999999999999e-05 2024-09-11T03:53:00.447 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.447 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.447 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:00.447 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:33.096088+0000", 2024-09-11T03:53:00.448 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0072218589999999997 2024-09-11T03:53:00.448 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.448 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.448 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.448 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:33.096136+0000", 2024-09-11T03:53:00.448 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7549999999999997e-05 2024-09-11T03:53:00.448 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.448 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.448 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.448 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.449 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.449 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:9741 2.42 2:43d79ac9:::smithi17557870-37:head [delete] snapc 1b2=[1b2,1b1,1b0,1af,1ac,1aa,1a8,1a7,1a6,1a5,1a4,1a3,1a2,1a1,19d,19c,199,197,194,191,18b,183,17c,17a,178,175,172,16e,164,15f,13c] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e940)", 2024-09-11T03:53:00.449 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:52:38.092957+0000", 2024-09-11T03:53:00.449 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 22.260184670000001, 2024-09-11T03:53:00.449 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013149401999999999, 2024-09-11T03:53:00.449 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.449 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.449 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:00.449 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:00.449 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:00.449 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:00.450 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 9741 2024-09-11T03:53:00.450 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.450 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.450 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.450 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.450 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.092957+0000", 2024-09-11T03:53:00.450 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.450 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.450 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.450 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.450 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.092957+0000", 2024-09-11T03:53:00.451 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.451 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.451 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.451 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.451 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.092959+0000", 2024-09-11T03:53:00.451 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.6449999999999999e-06 2024-09-11T03:53:00.451 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.451 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.451 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.451 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.092966+0000", 2024-09-11T03:53:00.451 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4949999999999996e-06 2024-09-11T03:53:00.452 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.452 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.452 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.452 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.092979+0000", 2024-09-11T03:53:00.452 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3594999999999999e-05 2024-09-11T03:53:00.452 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.452 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.452 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.452 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.092988+0000", 2024-09-11T03:53:00.452 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.1379999999999996e-06 2024-09-11T03:53:00.452 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.452 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.453 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.453 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093596+0000", 2024-09-11T03:53:00.453 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00060805199999999998 2024-09-11T03:53:00.453 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.453 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.453 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:00.453 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093607+0000", 2024-09-11T03:53:00.453 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0777e-05 2024-09-11T03:53:00.453 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.453 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.453 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.453 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.103347+0000", 2024-09-11T03:53:00.454 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0097404180000000007 2024-09-11T03:53:00.454 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.454 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.454 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.454 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.103487+0000", 2024-09-11T03:53:00.454 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013906699999999999 2024-09-11T03:53:00.454 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.454 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.454 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.454 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.106107+0000", 2024-09-11T03:53:00.454 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002620215 2024-09-11T03:53:00.454 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.455 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.455 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.455 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.455 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.455 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:9730 2.2e 2:750de3b7:::smithi17557870-8:head [append 0~0] snapc 1b1=[1b1,1b0,1af,1ac,1aa,1a8,1a7,1a6,1a5,1a4,1a3,1a2,1a1,19d,19c,199,197,194,191,18b,183,17c,17a,178,175,172,16e,164,15f,13c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e940)", 2024-09-11T03:53:00.455 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:52:38.093710+0000", 2024-09-11T03:53:00.455 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 22.259432316000002, 2024-09-11T03:53:00.455 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014723442999999999, 2024-09-11T03:53:00.455 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.455 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.455 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:00.455 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:00.455 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:00.456 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:00.456 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 9730 2024-09-11T03:53:00.456 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.456 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.456 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.456 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.456 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093710+0000", 2024-09-11T03:53:00.456 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.456 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.456 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.456 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.456 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093710+0000", 2024-09-11T03:53:00.457 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.457 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.457 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.457 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.457 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093711+0000", 2024-09-11T03:53:00.457 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3099999999999996e-07 2024-09-11T03:53:00.457 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.457 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.457 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.457 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093715+0000", 2024-09-11T03:53:00.458 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5909999999999998e-06 2024-09-11T03:53:00.458 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.458 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.458 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.458 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093725+0000", 2024-09-11T03:53:00.458 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4080000000000004e-06 2024-09-11T03:53:00.458 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.458 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.458 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.458 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093732+0000", 2024-09-11T03:53:00.458 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5639999999999999e-06 2024-09-11T03:53:00.458 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.458 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.459 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.459 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093790+0000", 2024-09-11T03:53:00.459 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7336000000000001e-05 2024-09-11T03:53:00.459 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.459 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.459 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:00.459 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093798+0000", 2024-09-11T03:53:00.459 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6289999999999999e-06 2024-09-11T03:53:00.459 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.459 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.459 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.459 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.101437+0000", 2024-09-11T03:53:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0076387319999999996 2024-09-11T03:53:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.101593+0000", 2024-09-11T03:53:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015562200000000001 2024-09-11T03:53:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3", 2024-09-11T03:53:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.101808+0000", 2024-09-11T03:53:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000215126 2024-09-11T03:53:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.461 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.461 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:00.461 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.108316+0000", 2024-09-11T03:53:00.461 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0065079100000000004 2024-09-11T03:53:00.461 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.461 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.461 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:00.461 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.108337+0000", 2024-09-11T03:53:00.461 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0965999999999999e-05 2024-09-11T03:53:00.461 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.461 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.462 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:00.462 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.108371+0000", 2024-09-11T03:53:00.462 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4226000000000001e-05 2024-09-11T03:53:00.462 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.462 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.462 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.462 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.108433+0000", 2024-09-11T03:53:00.462 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2402000000000003e-05 2024-09-11T03:53:00.462 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.462 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.462 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.462 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.462 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.463 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:9731 2.2e 2:750de3b7:::smithi17557870-8:head [rollback 1b1] snapc 1b1=[1b1,1b0,1af,1ac,1aa,1a8,1a7,1a6,1a5,1a4,1a3,1a2,1a1,19d,19c,199,197,194,191,18b,183,17c,17a,178,175,172,16e,164,15f,13c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e940)", 2024-09-11T03:53:00.463 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:52:38.093813+0000", 2024-09-11T03:53:00.463 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 22.259329511000001, 2024-09-11T03:53:00.463 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014837414, 2024-09-11T03:53:00.463 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.463 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.463 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:00.463 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:00.463 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:00.463 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:00.463 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 9731 2024-09-11T03:53:00.463 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.464 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.464 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.464 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.464 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093813+0000", 2024-09-11T03:53:00.464 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.464 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.464 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.464 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.464 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093813+0000", 2024-09-11T03:53:00.464 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.464 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.464 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.465 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.465 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093813+0000", 2024-09-11T03:53:00.465 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8299999999999997e-07 2024-09-11T03:53:00.465 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.465 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.465 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.465 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093816+0000", 2024-09-11T03:53:00.465 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7590000000000002e-06 2024-09-11T03:53:00.465 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.465 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.465 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.465 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093823+0000", 2024-09-11T03:53:00.466 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0879999999999997e-06 2024-09-11T03:53:00.466 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.466 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.466 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.466 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093829+0000", 2024-09-11T03:53:00.466 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8239999999999996e-06 2024-09-11T03:53:00.466 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.466 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.466 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.466 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093875+0000", 2024-09-11T03:53:00.466 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6584e-05 2024-09-11T03:53:00.466 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.467 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.467 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:00.467 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093883+0000", 2024-09-11T03:53:00.467 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8019999999999992e-06 2024-09-11T03:53:00.467 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.467 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.467 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.467 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.102627+0000", 2024-09-11T03:53:00.467 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0087436790000000007 2024-09-11T03:53:00.467 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.468 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.468 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.468 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.102728+0000", 2024-09-11T03:53:00.468 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010124300000000001 2024-09-11T03:53:00.468 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.468 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.468 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3", 2024-09-11T03:53:00.468 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.102977+0000", 2024-09-11T03:53:00.468 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000248671 2024-09-11T03:53:00.468 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.468 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.469 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:00.469 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.108476+0000", 2024-09-11T03:53:00.469 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0054996300000000001 2024-09-11T03:53:00.469 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.469 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.469 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:00.469 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.108524+0000", 2024-09-11T03:53:00.469 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.7330999999999998e-05 2024-09-11T03:53:00.469 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.469 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.469 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:00.469 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.108554+0000", 2024-09-11T03:53:00.470 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0298999999999999e-05 2024-09-11T03:53:00.470 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.470 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.470 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.470 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.108650+0000", 2024-09-11T03:53:00.470 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5921000000000005e-05 2024-09-11T03:53:00.470 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.470 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.470 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.470 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.470 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.470 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:9732 2.2e 2:750de3b7:::smithi17557870-8:head [append 0~0] snapc 1b1=[1b1,1b0,1af,1ac,1aa,1a8,1a7,1a6,1a5,1a4,1a3,1a2,1a1,19d,19c,199,197,194,191,18b,183,17c,17a,178,175,172,16e,164,15f,13c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e940)", 2024-09-11T03:53:00.471 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:52:38.093895+0000", 2024-09-11T03:53:00.471 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 22.259247424000002, 2024-09-11T03:53:00.471 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014911044, 2024-09-11T03:53:00.471 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.471 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.471 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:00.471 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:00.471 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:00.471 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:00.471 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 9732 2024-09-11T03:53:00.471 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.471 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.471 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.472 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.472 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093895+0000", 2024-09-11T03:53:00.472 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.472 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.472 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.472 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.472 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093895+0000", 2024-09-11T03:53:00.472 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.472 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.472 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.472 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.472 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093895+0000", 2024-09-11T03:53:00.473 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.06e-07 2024-09-11T03:53:00.473 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.473 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.473 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.473 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093898+0000", 2024-09-11T03:53:00.473 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5569999999999998e-06 2024-09-11T03:53:00.473 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.473 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.473 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.473 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093904+0000", 2024-09-11T03:53:00.473 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7479999999999996e-06 2024-09-11T03:53:00.473 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.474 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.474 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.474 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093910+0000", 2024-09-11T03:53:00.474 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.604e-06 2024-09-11T03:53:00.474 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.474 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.474 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.474 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093953+0000", 2024-09-11T03:53:00.474 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3164999999999998e-05 2024-09-11T03:53:00.474 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.474 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.474 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:00.474 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093961+0000", 2024-09-11T03:53:00.475 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7160000000000003e-06 2024-09-11T03:53:00.475 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.475 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.475 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.475 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.103952+0000", 2024-09-11T03:53:00.475 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0099907759999999998 2024-09-11T03:53:00.475 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.475 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.475 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.475 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.104057+0000", 2024-09-11T03:53:00.475 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000105191 2024-09-11T03:53:00.475 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.476 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.476 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3", 2024-09-11T03:53:00.476 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.104242+0000", 2024-09-11T03:53:00.476 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000184904 2024-09-11T03:53:00.476 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.476 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.476 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:00.476 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.108512+0000", 2024-09-11T03:53:00.476 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0042706519999999998 2024-09-11T03:53:00.476 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.476 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.476 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:00.477 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.108666+0000", 2024-09-11T03:53:00.477 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015354499999999999 2024-09-11T03:53:00.477 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.477 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.477 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:00.477 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.108716+0000", 2024-09-11T03:53:00.477 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9719999999999998e-05 2024-09-11T03:53:00.477 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.477 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.477 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.478 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.108806+0000", 2024-09-11T03:53:00.478 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.9959999999999999e-05 2024-09-11T03:53:00.478 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.478 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.478 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.478 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.478 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.478 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:9738 2.2e 2:77dadb97:::smithi17557870-30:head [sparse-read 0~1600000,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e940)", 2024-09-11T03:53:00.478 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:52:38.093972+0000", 2024-09-11T03:53:00.478 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 22.259170268999998, 2024-09-11T03:53:00.478 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014293503000000001, 2024-09-11T03:53:00.479 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.479 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.479 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:00.479 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:00.479 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:00.479 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:00.479 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 9738 2024-09-11T03:53:00.479 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.479 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.479 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.479 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.479 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093972+0000", 2024-09-11T03:53:00.479 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.480 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.480 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.480 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.480 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093972+0000", 2024-09-11T03:53:00.480 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.480 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.480 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.480 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.480 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093972+0000", 2024-09-11T03:53:00.480 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4099999999999999e-07 2024-09-11T03:53:00.480 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.480 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.481 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.481 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093976+0000", 2024-09-11T03:53:00.481 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.2349999999999999e-06 2024-09-11T03:53:00.481 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.481 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.481 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.481 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093982+0000", 2024-09-11T03:53:00.481 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9120000000000001e-06 2024-09-11T03:53:00.481 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.481 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.481 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.482 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.093989+0000", 2024-09-11T03:53:00.482 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0820000000000004e-06 2024-09-11T03:53:00.482 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.482 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.482 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.482 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094031+0000", 2024-09-11T03:53:00.482 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2135999999999998e-05 2024-09-11T03:53:00.482 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.482 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.482 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:00.482 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094038+0000", 2024-09-11T03:53:00.483 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4800000000000004e-06 2024-09-11T03:53:00.483 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.483 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.483 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.483 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.104734+0000", 2024-09-11T03:53:00.483 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010696321999999999 2024-09-11T03:53:00.483 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.483 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.483 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.483 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.104881+0000", 2024-09-11T03:53:00.483 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00014632499999999999 2024-09-11T03:53:00.484 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.484 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.484 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.484 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.108265+0000", 2024-09-11T03:53:00.484 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0033844700000000001 2024-09-11T03:53:00.484 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.484 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.484 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.484 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.484 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.484 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:9739 2.2e 2:77dadb97:::smithi17557870-30:head [read 0~2151626,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e940)", 2024-09-11T03:53:00.485 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:52:38.094050+0000", 2024-09-11T03:53:00.485 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 22.259092355, 2024-09-11T03:53:00.485 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015796841999999998, 2024-09-11T03:53:00.485 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.485 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.485 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:00.485 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:00.485 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:00.485 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:00.485 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 9739 2024-09-11T03:53:00.485 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.486 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.486 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.486 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.486 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094050+0000", 2024-09-11T03:53:00.486 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.486 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.486 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.486 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.486 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094050+0000", 2024-09-11T03:53:00.486 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.486 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.486 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.486 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.487 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094050+0000", 2024-09-11T03:53:00.487 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5000000000000003e-07 2024-09-11T03:53:00.487 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.487 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.487 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.487 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094054+0000", 2024-09-11T03:53:00.487 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3739999999999998e-06 2024-09-11T03:53:00.487 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.487 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.487 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.487 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094060+0000", 2024-09-11T03:53:00.488 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8800000000000002e-06 2024-09-11T03:53:00.488 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.488 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.488 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.488 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094066+0000", 2024-09-11T03:53:00.488 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3170000000000001e-06 2024-09-11T03:53:00.488 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.488 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.488 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.488 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094109+0000", 2024-09-11T03:53:00.488 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3498999999999998e-05 2024-09-11T03:53:00.489 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.489 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.489 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:00.489 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094117+0000", 2024-09-11T03:53:00.489 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4030000000000001e-06 2024-09-11T03:53:00.489 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.489 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.489 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.489 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.108844+0000", 2024-09-11T03:53:00.489 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014727283000000001 2024-09-11T03:53:00.489 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.489 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.490 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.490 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.108956+0000", 2024-09-11T03:53:00.490 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000112377 2024-09-11T03:53:00.490 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.490 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.490 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.490 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.109847+0000", 2024-09-11T03:53:00.490 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00089015899999999996 2024-09-11T03:53:00.490 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.490 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.490 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.490 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.491 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.491 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:9740 2.2e 2:77dadb97:::smithi17557870-30:head [read 0~2151626,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e940)", 2024-09-11T03:53:00.491 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:52:38.094129+0000", 2024-09-11T03:53:00.491 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 22.259012856999998, 2024-09-11T03:53:00.491 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016656529, 2024-09-11T03:53:00.491 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:00.491 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:00.491 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:00.491 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:00.491 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:00.491 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:00.492 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 9740 2024-09-11T03:53:00.492 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.492 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:00.492 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.492 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:00.492 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094129+0000", 2024-09-11T03:53:00.492 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.492 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.492 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.492 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:00.492 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094129+0000", 2024-09-11T03:53:00.492 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:00.493 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.493 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.493 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:00.493 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094130+0000", 2024-09-11T03:53:00.493 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5300000000000004e-07 2024-09-11T03:53:00.493 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.493 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.493 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:00.493 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094133+0000", 2024-09-11T03:53:00.493 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4470000000000002e-06 2024-09-11T03:53:00.493 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.493 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.493 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:00.494 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094140+0000", 2024-09-11T03:53:00.494 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4810000000000003e-06 2024-09-11T03:53:00.494 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.494 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.494 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:00.494 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094145+0000", 2024-09-11T03:53:00.494 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0350000000000001e-06 2024-09-11T03:53:00.494 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.494 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.494 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.494 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094188+0000", 2024-09-11T03:53:00.495 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3695000000000001e-05 2024-09-11T03:53:00.495 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.495 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.495 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:00.495 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.094196+0000", 2024-09-11T03:53:00.495 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9109999999999993e-06 2024-09-11T03:53:00.495 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.495 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.495 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:00.495 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.109920+0000", 2024-09-11T03:53:00.495 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015723747999999999 2024-09-11T03:53:00.495 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.496 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.496 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:00.496 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.110040+0000", 2024-09-11T03:53:00.496 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000119746 2024-09-11T03:53:00.496 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:00.496 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:00.496 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:00.496 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.110786+0000", 2024-09-11T03:53:00.496 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00074591299999999996 2024-09-11T03:53:00.496 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.496 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.497 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.497 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:00.497 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:00.497 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:53:00.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:00.498 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:00.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.4' 2024-09-11T03:53:00.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:53:00.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:00.623 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:53:00.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:53:00.628 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:53:00.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:53:00.751 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:00.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:00.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:53:00.853 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:53:00.875 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:00.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:53:00.886 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:53:00.887 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:00 smithi084 ceph-mon[27254]: pgmap v1594: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 744 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 3.5 MiB/s rd, 485 KiB/s wr, 5 op/s; 77/1053 objects degraded (7.312%) 2024-09-11T03:53:00.887 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:00 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3788577741' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:53:00.891 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:53:00.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:00 smithi121 ceph-mon[30729]: pgmap v1594: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 744 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 3.5 MiB/s rd, 485 KiB/s wr, 5 op/s; 77/1053 objects degraded (7.312%) 2024-09-11T03:53:00.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:00 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3788577741' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:53:00.976 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:53:00.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:00 smithi175 ceph-mon[30894]: pgmap v1594: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 744 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 3.5 MiB/s rd, 485 KiB/s wr, 5 op/s; 77/1053 objects degraded (7.312%) 2024-09-11T03:53:00.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:00 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3788577741' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:53:00.997 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:53:00.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:00.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:53:01.099 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:53:01.125 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:01.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:01.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:53:01.228 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:53:01.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:01.250 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:53:01.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:53:01.351 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:53:01.373 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:53:01.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:01.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:53:01.411 INFO:teuthology.orchestra.run.smithi084.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-11T03:53:01.421 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-11T03:53:01.475 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:53:01.501 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:01.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:53:01.604 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:53:01.626 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:53:01.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:01.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:53:01.665 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:53:01.728 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:53:01.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:01.750 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:53:01.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:53:01.816 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:01 smithi084 ceph-mon[27254]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:53:01.816 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3788577741' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:53:01.817 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:01 smithi084 ceph-mon[27254]: osdmap e959: 12 total, 11 up, 11 in 2024-09-11T03:53:01.851 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:53:01.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:01.878 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:01.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:53:01.914 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:01 smithi175 ceph-mon[30894]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:53:01.915 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3788577741' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:53:01.915 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:01 smithi175 ceph-mon[30894]: osdmap e959: 12 total, 11 up, 11 in 2024-09-11T03:53:01.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:01 smithi121 ceph-mon[30729]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-11T03:53:01.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3788577741' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:53:01.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:01 smithi121 ceph-mon[30729]: osdmap e959: 12 total, 11 up, 11 in 2024-09-11T03:53:01.980 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5 2024-09-11T03:53:02.006 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:02.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:02.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.5' 2024-09-11T03:53:02.047 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:02.107 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:53:02.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:02.129 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:53:02.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:53:02.150 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:53:02.150 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:53:02.150 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:53:02.150 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:53:02.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:53:02.257 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:02.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:02.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:53:02.359 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:53:02.380 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:53:02.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:02.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:53:02.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:53:02.509 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:02.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:02.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:53:02.558 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1595,"stamp":"2024-09-11T03:53:00.631735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":780396979,"num_objects":351,"num_object_clones":298,"num_object_copies":1053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":14,"num_read":1334,"num_read_kb":1425857,"num_write":876,"num_write_kb":341093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":67273842,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64588,"ondisk_log_size":64588,"up":350,"acting":353,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":2063692,"kb_used_data":1413828,"kb_used_omap":17,"kb_used_meta":649582,"kb_avail":1029132980,"statfs":{"total":1055945392128,"available":1053832171520,"internally_reserved":0,"allocated":1447759872,"data_stored":1844913444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17546,"internal_metadata":665172854},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1533143,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":19231,"num_write":12,"num_write_kb":2610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.099685"},"pg_stats":[{"pgid":"2.5f","version":"957'978","reported_seq":5434,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.562072+0000","last_change":"2024-09-11T03:52:55.562249+0000","last_active":"2024-09-11T03:52:55.562072+0000","last_peered":"2024-09-11T03:52:55.562072+0000","last_clean":"2024-09-11T03:52:55.562072+0000","last_became_active":"2024-09-11T03:52:37.108681+0000","last_became_peered":"2024-09-11T03:52:37.108681+0000","last_unstale":"2024-09-11T03:52:55.562072+0000","last_undegraded":"2024-09-11T03:52:55.562072+0000","last_fullsized":"2024-09-11T03:52:55.562072+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":933,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":7,"last_scrub":"939'956","last_scrub_stamp":"2024-09-11T03:52:38.328312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:38.328312+0000","objects_scrubbed":14,"log_size":978,"log_dups_size":0,"ondisk_log_size":978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:49.879928+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.013008452,"stat_sum":{"num_bytes":30682358,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":2213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.5e","version":"951'1141","reported_seq":5465,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.243653+0000","last_change":"2024-09-11T03:52:55.555468+0000","last_active":"2024-09-11T03:52:57.243653+0000","last_peered":"2024-09-11T03:52:57.243653+0000","last_clean":"2024-09-11T03:52:57.243653+0000","last_became_active":"2024-09-11T03:52:37.105130+0000","last_became_peered":"2024-09-11T03:52:37.105130+0000","last_unstale":"2024-09-11T03:52:57.243653+0000","last_undegraded":"2024-09-11T03:52:57.243653+0000","last_fullsized":"2024-09-11T03:52:57.243653+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":933,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":7,"last_scrub":"939'1136","last_scrub_stamp":"2024-09-11T03:52:38.236746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:38.236746+0000","objects_scrubbed":6,"log_size":1141,"log_dups_size":0,"ondisk_log_size":1141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:38.877301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0051966219999999997,"stat_sum":{"num_bytes":9671047,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,1],"acting":[6,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.5d","version":"567'752","reported_seq":3243,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577996+0000","last_change":"2024-09-11T03:52:55.541317+0000","last_active":"2024-09-11T03:52:56.577996+0000","last_peered":"2024-09-11T03:52:56.577996+0000","last_clean":"2024-09-11T03:52:56.577996+0000","last_became_active":"2024-09-11T03:52:37.105006+0000","last_became_peered":"2024-09-11T03:52:37.105006+0000","last_unstale":"2024-09-11T03:52:56.577996+0000","last_undegraded":"2024-09-11T03:52:56.577996+0000","last_fullsized":"2024-09-11T03:52:56.577996+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":933,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":7,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:52:38.497430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:38.497430+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:05.663439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00132573,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,7],"acting":[1,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.5c","version":"806'776","reported_seq":4434,"reported_epoch":957,"state":"active+clean+remapped","last_fresh":"2024-09-11T03:52:55.549079+0000","last_change":"2024-09-11T03:52:55.549645+0000","last_active":"2024-09-11T03:52:55.549079+0000","last_peered":"2024-09-11T03:52:55.549079+0000","last_clean":"2024-09-11T03:52:55.549079+0000","last_became_active":"2024-09-11T03:52:39.098755+0000","last_became_peered":"2024-09-11T03:52:39.098755+0000","last_unstale":"2024-09-11T03:52:55.549079+0000","last_undegraded":"2024-09-11T03:52:55.549079+0000","last_fullsized":"2024-09-11T03:52:55.549079+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":933,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":7,"last_scrub":"806'776","last_scrub_stamp":"2024-09-11T03:52:39.423526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:39.423526+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:41.254450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035434170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1,7],"avail_no_missing":["8","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.5b","version":"930'954","reported_seq":5467,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.554994+0000","last_change":"2024-09-11T03:52:55.555464+0000","last_active":"2024-09-11T03:52:55.554994+0000","last_peered":"2024-09-11T03:52:55.554994+0000","last_clean":"2024-09-11T03:52:55.554994+0000","last_became_active":"2024-09-11T03:52:37.102818+0000","last_became_peered":"2024-09-11T03:52:37.102818+0000","last_unstale":"2024-09-11T03:52:55.554994+0000","last_undegraded":"2024-09-11T03:52:55.554994+0000","last_fullsized":"2024-09-11T03:52:55.554994+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":933,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":7,"last_scrub":"930'954","last_scrub_stamp":"2024-09-11T03:52:38.454086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:38.454086+0000","objects_scrubbed":0,"log_size":954,"log_dups_size":0,"ondisk_log_size":954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:42.026297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075136710000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.5a","version":"557'273","reported_seq":2741,"reported_epoch":958,"state":"active+clean+remapped","last_fresh":"2024-09-11T03:52:56.577545+0000","last_change":"2024-09-11T03:52:55.553197+0000","last_active":"2024-09-11T03:52:56.577545+0000","last_peered":"2024-09-11T03:52:56.577545+0000","last_clean":"2024-09-11T03:52:56.577545+0000","last_became_active":"2024-09-11T03:52:39.091422+0000","last_became_peered":"2024-09-11T03:52:39.091422+0000","last_unstale":"2024-09-11T03:52:56.577545+0000","last_undegraded":"2024-09-11T03:52:56.577545+0000","last_fullsized":"2024-09-11T03:52:56.577545+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":933,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":7,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:52:39.114859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:39.114859+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:50.709827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086220240000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3,2],"avail_no_missing":["5","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.59","version":"930'659","reported_seq":5055,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.551275+0000","last_change":"2024-09-11T03:52:55.551567+0000","last_active":"2024-09-11T03:52:55.551275+0000","last_peered":"2024-09-11T03:52:55.551275+0000","last_clean":"2024-09-11T03:52:55.551275+0000","last_became_active":"2024-09-11T03:52:37.101561+0000","last_became_peered":"2024-09-11T03:52:37.101561+0000","last_unstale":"2024-09-11T03:52:55.551275+0000","last_undegraded":"2024-09-11T03:52:55.551275+0000","last_fullsized":"2024-09-11T03:52:55.551275+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":933,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":7,"last_scrub":"930'659","last_scrub_stamp":"2024-09-11T03:52:38.102368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:38.102368+0000","objects_scrubbed":0,"log_size":659,"log_dups_size":0,"ondisk_log_size":659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:39.601952+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.00076335499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.58","version":"957'803","reported_seq":5405,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.555343+0000","last_change":"2024-09-11T03:52:55.555775+0000","last_active":"2024-09-11T03:52:55.555343+0000","last_peered":"2024-09-11T03:52:55.555343+0000","last_clean":"2024-09-11T03:52:55.555343+0000","last_became_active":"2024-09-11T03:52:37.095091+0000","last_became_peered":"2024-09-11T03:52:37.095091+0000","last_unstale":"2024-09-11T03:52:55.555343+0000","last_undegraded":"2024-09-11T03:52:55.555343+0000","last_fullsized":"2024-09-11T03:52:55.555343+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":933,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":7,"last_scrub":"939'789","last_scrub_stamp":"2024-09-11T03:52:37.316715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:37.316715+0000","objects_scrubbed":8,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:49.985673+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0070849820000000001,"stat_sum":{"num_bytes":20403881,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.57","version":"274'312","reported_seq":2165,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.551701+0000","last_change":"2024-09-11T03:52:55.555026+0000","last_active":"2024-09-11T03:52:55.551701+0000","last_peered":"2024-09-11T03:52:55.551701+0000","last_clean":"2024-09-11T03:52:55.551701+0000","last_became_active":"2024-09-11T03:52:37.107713+0000","last_became_peered":"2024-09-11T03:52:37.107713+0000","last_unstale":"2024-09-11T03:52:55.551701+0000","last_undegraded":"2024-09-11T03:52:55.551701+0000","last_fullsized":"2024-09-11T03:52:55.551701+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":933,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":7,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:51:28.919603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.919603+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0071625969999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,7],"acting":[10,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.56","version":"782'561","reported_seq":4176,"reported_epoch":958,"state":"active+clean+remapped","last_fresh":"2024-09-11T03:52:57.243778+0000","last_change":"2024-09-11T03:52:55.556090+0000","last_active":"2024-09-11T03:52:57.243778+0000","last_peered":"2024-09-11T03:52:57.243778+0000","last_clean":"2024-09-11T03:52:57.243778+0000","last_became_active":"2024-09-11T03:52:39.099485+0000","last_became_peered":"2024-09-11T03:52:39.099485+0000","last_unstale":"2024-09-11T03:52:57.243778+0000","last_undegraded":"2024-09-11T03:52:57.243778+0000","last_fullsized":"2024-09-11T03:52:57.243778+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":933,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":7,"last_scrub":"782'561","last_scrub_stamp":"2024-09-11T03:52:41.265404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:41.265404+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:42.275690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0052500960000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7,5],"avail_no_missing":["6","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.55","version":"274'273","reported_seq":1427,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.578007+0000","last_change":"2024-09-11T03:52:55.554545+0000","last_active":"2024-09-11T03:52:56.578007+0000","last_peered":"2024-09-11T03:52:56.578007+0000","last_clean":"2024-09-11T03:52:56.578007+0000","last_became_active":"2024-09-11T03:52:25.079201+0000","last_became_peered":"2024-09-11T03:52:25.079201+0000","last_unstale":"2024-09-11T03:52:56.578007+0000","last_undegraded":"2024-09-11T03:52:56.578007+0000","last_fullsized":"2024-09-11T03:52:56.578007+0000","mapping_epoch":926,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":927,"parent":"0.0","parent_split_bits":7,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:52:35.192774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:35.192774+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:53.014603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097628220000000009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,9],"acting":[5,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.54","version":"778'353","reported_seq":3888,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577515+0000","last_change":"2024-09-11T03:52:55.539327+0000","last_active":"2024-09-11T03:52:56.577515+0000","last_peered":"2024-09-11T03:52:56.577515+0000","last_clean":"2024-09-11T03:52:56.577515+0000","last_became_active":"2024-09-11T03:52:25.077580+0000","last_became_peered":"2024-09-11T03:52:25.077580+0000","last_unstale":"2024-09-11T03:52:56.577515+0000","last_undegraded":"2024-09-11T03:52:56.577515+0000","last_fullsized":"2024-09-11T03:52:56.577515+0000","mapping_epoch":926,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":927,"parent":"0.0","parent_split_bits":7,"last_scrub":"778'353","last_scrub_stamp":"2024-09-11T03:52:37.459218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:37.459218+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:49.388747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001312988,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,10],"acting":[1,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.53","version":"274'111","reported_seq":1210,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577521+0000","last_change":"2024-09-11T03:52:55.540427+0000","last_active":"2024-09-11T03:52:56.577521+0000","last_peered":"2024-09-11T03:52:56.577521+0000","last_clean":"2024-09-11T03:52:56.577521+0000","last_became_active":"2024-09-11T03:52:25.079154+0000","last_became_peered":"2024-09-11T03:52:25.079154+0000","last_unstale":"2024-09-11T03:52:56.577521+0000","last_undegraded":"2024-09-11T03:52:56.577521+0000","last_fullsized":"2024-09-11T03:52:56.577521+0000","mapping_epoch":926,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":927,"parent":"0.0","parent_split_bits":7,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:52:36.455984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:36.455984+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:39.855315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015008070000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.52","version":"905'590","reported_seq":4723,"reported_epoch":957,"state":"active+undersized+degraded","last_fresh":"2024-09-11T03:52:55.551392+0000","last_change":"2024-09-11T03:52:38.104230+0000","last_active":"2024-09-11T03:52:55.551392+0000","last_peered":"2024-09-11T03:52:55.551392+0000","last_clean":"2024-09-11T03:52:03.063604+0000","last_became_active":"2024-09-11T03:52:38.104230+0000","last_became_peered":"2024-09-11T03:52:38.104230+0000","last_unstale":"2024-09-11T03:52:55.551392+0000","last_undegraded":"2024-09-11T03:52:38.089206+0000","last_fullsized":"2024-09-11T03:52:38.088948+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":927,"parent":"0.0","parent_split_bits":7,"last_scrub":"905'590","last_scrub_stamp":"2024-09-11T03:52:35.916029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:35.916029+0000","objects_scrubbed":6,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2739805,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":["11","10"],"object_location_counts":[{"shards":"10,11","objects":1}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.51","version":"274'107","reported_seq":1705,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.557989+0000","last_change":"2024-09-11T03:52:55.559835+0000","last_active":"2024-09-11T03:52:55.557989+0000","last_peered":"2024-09-11T03:52:55.557989+0000","last_clean":"2024-09-11T03:52:55.557989+0000","last_became_active":"2024-09-11T03:52:25.079217+0000","last_became_peered":"2024-09-11T03:52:25.079217+0000","last_unstale":"2024-09-11T03:52:55.557989+0000","last_undegraded":"2024-09-11T03:52:55.557989+0000","last_fullsized":"2024-09-11T03:52:55.557989+0000","mapping_epoch":926,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":927,"parent":"0.0","parent_split_bits":7,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:52:35.434166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:35.434166+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:36.715743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085607930000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,3],"acting":[10,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.50","version":"274'67","reported_seq":3466,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.545304+0000","last_change":"2024-09-11T03:52:55.546297+0000","last_active":"2024-09-11T03:52:55.545304+0000","last_peered":"2024-09-11T03:52:55.545304+0000","last_clean":"2024-09-11T03:52:55.545304+0000","last_became_active":"2024-09-11T03:52:22.076403+0000","last_became_peered":"2024-09-11T03:52:22.076403+0000","last_unstale":"2024-09-11T03:52:55.545304+0000","last_undegraded":"2024-09-11T03:52:55.545304+0000","last_fullsized":"2024-09-11T03:52:55.545304+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":7,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:52:35.411519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:35.411519+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:41.921095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001290409,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.4f","version":"557'465","reported_seq":4831,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577686+0000","last_change":"2024-09-11T03:52:55.539738+0000","last_active":"2024-09-11T03:52:56.577686+0000","last_peered":"2024-09-11T03:52:56.577686+0000","last_clean":"2024-09-11T03:52:56.577686+0000","last_became_active":"2024-09-11T03:52:25.079240+0000","last_became_peered":"2024-09-11T03:52:25.079240+0000","last_unstale":"2024-09-11T03:52:56.577686+0000","last_undegraded":"2024-09-11T03:52:56.577686+0000","last_fullsized":"2024-09-11T03:52:56.577686+0000","mapping_epoch":926,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":927,"parent":"0.0","parent_split_bits":7,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:52:39.546859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:39.546859+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:48.621934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00131022,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,9],"acting":[1,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.4e","version":"782'1025","reported_seq":4637,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577819+0000","last_change":"2024-09-11T03:52:55.553561+0000","last_active":"2024-09-11T03:52:56.577819+0000","last_peered":"2024-09-11T03:52:56.577819+0000","last_clean":"2024-09-11T03:52:56.577819+0000","last_became_active":"2024-09-11T03:52:25.078543+0000","last_became_peered":"2024-09-11T03:52:25.078543+0000","last_unstale":"2024-09-11T03:52:56.577819+0000","last_undegraded":"2024-09-11T03:52:56.577819+0000","last_fullsized":"2024-09-11T03:52:56.577819+0000","mapping_epoch":926,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":927,"parent":"0.0","parent_split_bits":7,"last_scrub":"782'1025","last_scrub_stamp":"2024-09-11T03:52:37.131005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:37.131005+0000","objects_scrubbed":0,"log_size":1025,"log_dups_size":0,"ondisk_log_size":1025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:58.024418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0088488190000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.4d","version":"920'1075","reported_seq":5523,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.557153+0000","last_change":"2024-09-11T03:52:55.557477+0000","last_active":"2024-09-11T03:52:55.557153+0000","last_peered":"2024-09-11T03:52:55.557153+0000","last_clean":"2024-09-11T03:52:55.557153+0000","last_became_active":"2024-09-11T03:52:25.079699+0000","last_became_peered":"2024-09-11T03:52:25.079699+0000","last_unstale":"2024-09-11T03:52:55.557153+0000","last_undegraded":"2024-09-11T03:52:55.557153+0000","last_fullsized":"2024-09-11T03:52:55.557153+0000","mapping_epoch":926,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":927,"parent":"0.0","parent_split_bits":7,"last_scrub":"920'1075","last_scrub_stamp":"2024-09-11T03:52:36.455090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:36.455090+0000","objects_scrubbed":0,"log_size":1075,"log_dups_size":0,"ondisk_log_size":1075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:51.484431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068324609999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,7],"acting":[10,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.4c","version":"941'999","reported_seq":5324,"reported_epoch":957,"state":"active+undersized+degraded","last_fresh":"2024-09-11T03:52:55.549785+0000","last_change":"2024-09-11T03:52:38.097376+0000","last_active":"2024-09-11T03:52:55.549785+0000","last_peered":"2024-09-11T03:52:55.549785+0000","last_clean":"2024-09-11T03:52:36.086717+0000","last_became_active":"2024-09-11T03:52:38.097376+0000","last_became_peered":"2024-09-11T03:52:38.097376+0000","last_unstale":"2024-09-11T03:52:55.549785+0000","last_undegraded":"2024-09-11T03:52:38.092544+0000","last_fullsized":"2024-09-11T03:52:38.092363+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":921,"last_epoch_clean":927,"parent":"0.0","parent_split_bits":7,"last_scrub":"843'880","last_scrub_stamp":"2024-09-11T03:51:01.987099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:01.987099+0000","objects_scrubbed":13,"log_size":999,"log_dups_size":0,"ondisk_log_size":999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0038762940000000002,"stat_sum":{"num_bytes":14852284,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":25566,"num_write":24,"num_write_kb":9593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":10119807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":["10","5"],"object_location_counts":[{"shards":"5,10","objects":6}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4b","version":"568'646","reported_seq":4984,"reported_epoch":958,"state":"active+undersized","last_fresh":"2024-09-11T03:52:56.577632+0000","last_change":"2024-09-11T03:52:38.092902+0000","last_active":"2024-09-11T03:52:56.577632+0000","last_peered":"2024-09-11T03:52:56.577632+0000","last_clean":"2024-09-11T03:52:36.088887+0000","last_became_active":"2024-09-11T03:52:38.092902+0000","last_became_peered":"2024-09-11T03:52:38.092902+0000","last_unstale":"2024-09-11T03:52:56.577632+0000","last_undegraded":"2024-09-11T03:52:56.577632+0000","last_fullsized":"2024-09-11T03:52:38.089605+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:51:21.370186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.370186+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011939839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4a","version":"848'386","reported_seq":4753,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.556132+0000","last_change":"2024-09-11T03:52:55.556967+0000","last_active":"2024-09-11T03:52:55.556132+0000","last_peered":"2024-09-11T03:52:55.556132+0000","last_clean":"2024-09-11T03:52:55.556132+0000","last_became_active":"2024-09-11T03:51:13.080958+0000","last_became_peered":"2024-09-11T03:51:13.080958+0000","last_unstale":"2024-09-11T03:52:55.556132+0000","last_undegraded":"2024-09-11T03:52:55.556132+0000","last_fullsized":"2024-09-11T03:52:55.556132+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"848'386","last_scrub_stamp":"2024-09-11T03:51:19.939163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.939163+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083346770000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.49","version":"957'1678","reported_seq":6208,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577510+0000","last_change":"2024-09-11T03:52:55.552737+0000","last_active":"2024-09-11T03:52:56.577510+0000","last_peered":"2024-09-11T03:52:56.577510+0000","last_clean":"2024-09-11T03:52:56.577510+0000","last_became_active":"2024-09-11T03:51:20.066997+0000","last_became_peered":"2024-09-11T03:51:20.066997+0000","last_unstale":"2024-09-11T03:52:56.577510+0000","last_undegraded":"2024-09-11T03:52:56.577510+0000","last_fullsized":"2024-09-11T03:52:56.577510+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":7,"last_scrub":"863'1411","last_scrub_stamp":"2024-09-11T03:51:21.422037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.422037+0000","objects_scrubbed":18,"log_size":1678,"log_dups_size":0,"ondisk_log_size":1678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0083447179999999992,"stat_sum":{"num_bytes":48475174,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":63,"num_read_kb":63831,"num_write":39,"num_write_kb":9867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.48","version":"744'374","reported_seq":4210,"reported_epoch":958,"state":"active+undersized","last_fresh":"2024-09-11T03:52:57.095007+0000","last_change":"2024-09-11T03:52:38.099647+0000","last_active":"2024-09-11T03:52:57.095007+0000","last_peered":"2024-09-11T03:52:57.095007+0000","last_clean":"2024-09-11T03:52:36.091288+0000","last_became_active":"2024-09-11T03:52:38.099647+0000","last_became_peered":"2024-09-11T03:52:38.099647+0000","last_unstale":"2024-09-11T03:52:57.095007+0000","last_undegraded":"2024-09-11T03:52:57.095007+0000","last_fullsized":"2024-09-11T03:52:38.094820+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"744'374","last_scrub_stamp":"2024-09-11T03:51:19.306681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.306681+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0036205439999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":["9","3"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.47","version":"954'1342","reported_seq":5814,"reported_epoch":958,"state":"active+undersized+degraded","last_fresh":"2024-09-11T03:52:57.095435+0000","last_change":"2024-09-11T03:52:38.095839+0000","last_active":"2024-09-11T03:52:57.095435+0000","last_peered":"2024-09-11T03:52:57.095435+0000","last_clean":"2024-09-11T03:52:30.215882+0000","last_became_active":"2024-09-11T03:52:38.095839+0000","last_became_peered":"2024-09-11T03:52:38.095839+0000","last_unstale":"2024-09-11T03:52:57.095435+0000","last_undegraded":"2024-09-11T03:52:38.094475+0000","last_fullsized":"2024-09-11T03:52:38.094284+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":7,"last_scrub":"862'1274","last_scrub_stamp":"2024-09-11T03:51:20.166795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.166795+0000","objects_scrubbed":7,"log_size":1342,"log_dups_size":0,"ondisk_log_size":1342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:57.094128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001006465,"stat_sum":{"num_bytes":26572106,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":17,"num_read_kb":11529,"num_write":9,"num_write_kb":3274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":["9","2"],"object_location_counts":[{"shards":"2,9","objects":11}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.46","version":"269'171","reported_seq":4483,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.562332+0000","last_change":"2024-09-11T03:52:55.562501+0000","last_active":"2024-09-11T03:52:55.562332+0000","last_peered":"2024-09-11T03:52:55.562332+0000","last_clean":"2024-09-11T03:52:55.562332+0000","last_became_active":"2024-09-11T03:51:13.080822+0000","last_became_peered":"2024-09-11T03:51:13.080822+0000","last_unstale":"2024-09-11T03:52:55.562332+0000","last_undegraded":"2024-09-11T03:52:55.562332+0000","last_fullsized":"2024-09-11T03:52:55.562332+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:51:21.034557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.034557+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013017209,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.45","version":"955'1268","reported_seq":5641,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.578157+0000","last_change":"2024-09-11T03:52:55.541625+0000","last_active":"2024-09-11T03:52:56.578157+0000","last_peered":"2024-09-11T03:52:56.578157+0000","last_clean":"2024-09-11T03:52:56.578157+0000","last_became_active":"2024-09-11T03:51:13.078704+0000","last_became_peered":"2024-09-11T03:51:13.078704+0000","last_unstale":"2024-09-11T03:52:56.578157+0000","last_undegraded":"2024-09-11T03:52:56.578157+0000","last_fullsized":"2024-09-11T03:52:56.578157+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"863'1213","last_scrub_stamp":"2024-09-11T03:51:23.430916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.430916+0000","objects_scrubbed":4,"log_size":1268,"log_dups_size":0,"ondisk_log_size":1268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00124529,"stat_sum":{"num_bytes":13783126,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":12,"num_write_kb":2342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[1,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.44","version":"955'799","reported_seq":5291,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577640+0000","last_change":"2024-09-11T03:52:55.539648+0000","last_active":"2024-09-11T03:52:56.577640+0000","last_peered":"2024-09-11T03:52:56.577640+0000","last_clean":"2024-09-11T03:52:56.577640+0000","last_became_active":"2024-09-11T03:51:13.079319+0000","last_became_peered":"2024-09-11T03:51:13.079319+0000","last_unstale":"2024-09-11T03:52:56.577640+0000","last_undegraded":"2024-09-11T03:52:56.577640+0000","last_fullsized":"2024-09-11T03:52:56.577640+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"946'792","last_scrub_stamp":"2024-09-11T03:52:46.339683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:46.339683+0000","objects_scrubbed":9,"log_size":799,"log_dups_size":0,"ondisk_log_size":799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:05.244131+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0013333209999999999,"stat_sum":{"num_bytes":22970265,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":7,"num_read_kb":2,"num_write":15,"num_write_kb":8693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,3],"acting":[1,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.43","version":"957'662","reported_seq":5055,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.563218+0000","last_change":"2024-09-11T03:52:55.563436+0000","last_active":"2024-09-11T03:52:55.563218+0000","last_peered":"2024-09-11T03:52:55.563218+0000","last_clean":"2024-09-11T03:52:55.563218+0000","last_became_active":"2024-09-11T03:51:13.083077+0000","last_became_peered":"2024-09-11T03:51:13.083077+0000","last_unstale":"2024-09-11T03:52:55.563218+0000","last_undegraded":"2024-09-11T03:52:55.563218+0000","last_fullsized":"2024-09-11T03:52:55.563218+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"862'574","last_scrub_stamp":"2024-09-11T03:51:22.027988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.027988+0000","objects_scrubbed":9,"log_size":662,"log_dups_size":0,"ondisk_log_size":662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014251537,"stat_sum":{"num_bytes":26707373,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":16,"num_read_kb":11850,"num_write":13,"num_write_kb":15254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.42","version":"956'375","reported_seq":4724,"reported_epoch":957,"state":"active+undersized+degraded","last_fresh":"2024-09-11T03:52:55.545287+0000","last_change":"2024-09-11T03:52:38.102402+0000","last_active":"2024-09-11T03:52:55.545287+0000","last_peered":"2024-09-11T03:52:55.545287+0000","last_clean":"2024-09-11T03:52:36.083427+0000","last_became_active":"2024-09-11T03:52:38.102402+0000","last_became_peered":"2024-09-11T03:52:38.102402+0000","last_unstale":"2024-09-11T03:52:55.545287+0000","last_undegraded":"2024-09-11T03:52:38.091550+0000","last_fullsized":"2024-09-11T03:52:38.091401+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":849,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":7,"last_scrub":"859'323","last_scrub_stamp":"2024-09-11T03:51:19.628572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.628572+0000","objects_scrubbed":5,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.009964746,"stat_sum":{"num_bytes":14589796,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":3,"num_read_kb":1,"num_write":9,"num_write_kb":1193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":["8","11"],"object_location_counts":[{"shards":"8,11","objects":7}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.41","version":"567'235","reported_seq":4528,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.559638+0000","last_change":"2024-09-11T03:52:55.560401+0000","last_active":"2024-09-11T03:52:55.559638+0000","last_peered":"2024-09-11T03:52:55.559638+0000","last_clean":"2024-09-11T03:52:55.559638+0000","last_became_active":"2024-09-11T03:50:33.046488+0000","last_became_peered":"2024-09-11T03:50:33.046488+0000","last_unstale":"2024-09-11T03:52:55.559638+0000","last_undegraded":"2024-09-11T03:52:55.559638+0000","last_fullsized":"2024-09-11T03:52:55.559638+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":7,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:52:09.455980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:09.455980+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:36.030387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0091467489999999992,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.40","version":"563'192","reported_seq":4599,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.243491+0000","last_change":"2024-09-11T03:52:55.547814+0000","last_active":"2024-09-11T03:52:57.243491+0000","last_peered":"2024-09-11T03:52:57.243491+0000","last_clean":"2024-09-11T03:52:57.243491+0000","last_became_active":"2024-09-11T03:51:20.064866+0000","last_became_peered":"2024-09-11T03:51:20.064866+0000","last_unstale":"2024-09-11T03:52:57.243491+0000","last_undegraded":"2024-09-11T03:52:57.243491+0000","last_fullsized":"2024-09-11T03:52:57.243491+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":7,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:52:17.055805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:17.055805+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:17.266299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000786804,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,8],"acting":[6,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.3f","version":"951'836","reported_seq":5181,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.563317+0000","last_change":"2024-09-11T03:52:55.563593+0000","last_active":"2024-09-11T03:52:55.563317+0000","last_peered":"2024-09-11T03:52:55.563317+0000","last_clean":"2024-09-11T03:52:55.563317+0000","last_became_active":"2024-09-11T03:50:33.046798+0000","last_became_peered":"2024-09-11T03:50:33.046798+0000","last_unstale":"2024-09-11T03:52:55.563317+0000","last_undegraded":"2024-09-11T03:52:55.563317+0000","last_fullsized":"2024-09-11T03:52:55.563317+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"894'803","last_scrub_stamp":"2024-09-11T03:51:53.038706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.038706+0000","objects_scrubbed":5,"log_size":836,"log_dups_size":0,"ondisk_log_size":836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:08.727904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.013390931999999999,"stat_sum":{"num_bytes":13313731,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":28919,"num_write":10,"num_write_kb":2923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.3e","version":"957'1150","reported_seq":5447,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.574013+0000","last_change":"2024-09-11T03:52:55.574221+0000","last_active":"2024-09-11T03:52:55.574013+0000","last_peered":"2024-09-11T03:52:55.574013+0000","last_clean":"2024-09-11T03:52:55.574013+0000","last_became_active":"2024-09-11T03:50:33.045235+0000","last_became_peered":"2024-09-11T03:50:33.045235+0000","last_unstale":"2024-09-11T03:52:55.574013+0000","last_undegraded":"2024-09-11T03:52:55.574013+0000","last_fullsized":"2024-09-11T03:52:55.574013+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"893'1098","last_scrub_stamp":"2024-09-11T03:51:52.144830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.144830+0000","objects_scrubbed":7,"log_size":1150,"log_dups_size":0,"ondisk_log_size":1150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:21.942036+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.012816727,"stat_sum":{"num_bytes":11947450,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":22149,"num_write":15,"num_write_kb":3152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.3d","version":"567'752","reported_seq":3872,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.578267+0000","last_change":"2024-09-11T03:52:55.541978+0000","last_active":"2024-09-11T03:52:56.578267+0000","last_peered":"2024-09-11T03:52:56.578267+0000","last_clean":"2024-09-11T03:52:56.578267+0000","last_became_active":"2024-09-11T03:51:20.070161+0000","last_became_peered":"2024-09-11T03:51:20.070161+0000","last_unstale":"2024-09-11T03:52:56.578267+0000","last_undegraded":"2024-09-11T03:52:56.578267+0000","last_fullsized":"2024-09-11T03:52:56.578267+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:51:53.597428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.597428+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:06.639506+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013140000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.3c","version":"945'802","reported_seq":5160,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.549710+0000","last_change":"2024-09-11T03:52:55.549990+0000","last_active":"2024-09-11T03:52:55.549710+0000","last_peered":"2024-09-11T03:52:55.549710+0000","last_clean":"2024-09-11T03:52:55.549710+0000","last_became_active":"2024-09-11T03:50:33.048487+0000","last_became_peered":"2024-09-11T03:50:33.048487+0000","last_unstale":"2024-09-11T03:52:55.549710+0000","last_undegraded":"2024-09-11T03:52:55.549710+0000","last_fullsized":"2024-09-11T03:52:55.549710+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"886'783","last_scrub_stamp":"2024-09-11T03:51:45.009597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:45.009597+0000","objects_scrubbed":3,"log_size":802,"log_dups_size":0,"ondisk_log_size":802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0017130680000000001,"stat_sum":{"num_bytes":5948695,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":24464,"num_write":11,"num_write_kb":5447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,10],"acting":[7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.3b","version":"957'1039","reported_seq":5599,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577742+0000","last_change":"2024-09-11T03:52:55.556922+0000","last_active":"2024-09-11T03:52:56.577742+0000","last_peered":"2024-09-11T03:52:56.577742+0000","last_clean":"2024-09-11T03:52:56.577742+0000","last_became_active":"2024-09-11T03:51:20.066390+0000","last_became_peered":"2024-09-11T03:51:20.066390+0000","last_unstale":"2024-09-11T03:52:56.577742+0000","last_undegraded":"2024-09-11T03:52:56.577742+0000","last_fullsized":"2024-09-11T03:52:56.577742+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"905'970","last_scrub_stamp":"2024-09-11T03:52:10.225202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:10.225202+0000","objects_scrubbed":9,"log_size":1039,"log_dups_size":0,"ondisk_log_size":1039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:19.977230+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.012486947999999999,"stat_sum":{"num_bytes":29499831,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":44,"num_read_kb":50520,"num_write":36,"num_write_kb":10270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":14555611,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.3a","version":"557'273","reported_seq":3370,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.095055+0000","last_change":"2024-09-11T03:52:55.558719+0000","last_active":"2024-09-11T03:52:57.095055+0000","last_peered":"2024-09-11T03:52:57.095055+0000","last_clean":"2024-09-11T03:52:57.095055+0000","last_became_active":"2024-09-11T03:51:20.061387+0000","last_became_peered":"2024-09-11T03:51:20.061387+0000","last_unstale":"2024-09-11T03:52:57.095055+0000","last_undegraded":"2024-09-11T03:52:57.095055+0000","last_fullsized":"2024-09-11T03:52:57.095055+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:52:44.165633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:44.165633+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:44.813622+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0049250659999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.39","version":"806'585","reported_seq":4975,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.094857+0000","last_change":"2024-09-11T03:52:55.554786+0000","last_active":"2024-09-11T03:52:57.094857+0000","last_peered":"2024-09-11T03:52:57.094857+0000","last_clean":"2024-09-11T03:52:57.094857+0000","last_became_active":"2024-09-11T03:50:33.043651+0000","last_became_peered":"2024-09-11T03:50:33.043651+0000","last_unstale":"2024-09-11T03:52:57.094857+0000","last_undegraded":"2024-09-11T03:52:57.094857+0000","last_fullsized":"2024-09-11T03:52:57.094857+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"806'585","last_scrub_stamp":"2024-09-11T03:52:19.355191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:19.355191+0000","objects_scrubbed":0,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:44.406202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002842228,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,2],"acting":[9,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.38","version":"808'564","reported_seq":4961,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.545818+0000","last_change":"2024-09-11T03:52:55.546673+0000","last_active":"2024-09-11T03:52:55.545818+0000","last_peered":"2024-09-11T03:52:55.545818+0000","last_clean":"2024-09-11T03:52:55.545818+0000","last_became_active":"2024-09-11T03:50:33.042740+0000","last_became_peered":"2024-09-11T03:50:33.042740+0000","last_unstale":"2024-09-11T03:52:55.545818+0000","last_undegraded":"2024-09-11T03:52:55.545818+0000","last_fullsized":"2024-09-11T03:52:55.545818+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":809,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"808'564","last_scrub_stamp":"2024-09-11T03:51:54.462104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.462104+0000","objects_scrubbed":0,"log_size":564,"log_dups_size":0,"ondisk_log_size":564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:07.924638+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001152983,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,1],"acting":[8,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.37","version":"274'312","reported_seq":2933,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.555657+0000","last_change":"2024-09-11T03:52:55.557050+0000","last_active":"2024-09-11T03:52:55.555657+0000","last_peered":"2024-09-11T03:52:55.555657+0000","last_clean":"2024-09-11T03:52:55.555657+0000","last_became_active":"2024-09-11T03:50:07.077607+0000","last_became_peered":"2024-09-11T03:50:07.077607+0000","last_unstale":"2024-09-11T03:52:55.555657+0000","last_undegraded":"2024-09-11T03:52:55.555657+0000","last_fullsized":"2024-09-11T03:52:55.555657+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:51:23.036386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.036386+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0086199250000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,9],"acting":[10,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.36","version":"957'770","reported_seq":5165,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.578647+0000","last_change":"2024-09-11T03:52:55.554799+0000","last_active":"2024-09-11T03:52:56.578647+0000","last_peered":"2024-09-11T03:52:56.578647+0000","last_clean":"2024-09-11T03:52:56.578647+0000","last_became_active":"2024-09-11T03:50:07.078667+0000","last_became_peered":"2024-09-11T03:50:07.078667+0000","last_unstale":"2024-09-11T03:52:56.578647+0000","last_undegraded":"2024-09-11T03:52:56.578647+0000","last_fullsized":"2024-09-11T03:52:56.578647+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"885'670","last_scrub_stamp":"2024-09-11T03:51:43.529996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:43.529996+0000","objects_scrubbed":15,"log_size":770,"log_dups_size":0,"ondisk_log_size":770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:07.684774+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.014235061,"stat_sum":{"num_bytes":24560842,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":81,"num_read_kb":82520,"num_write":15,"num_write_kb":11348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,8],"acting":[1,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.35","version":"274'273","reported_seq":2113,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.578316+0000","last_change":"2024-09-11T03:52:55.542404+0000","last_active":"2024-09-11T03:52:56.578316+0000","last_peered":"2024-09-11T03:52:56.578316+0000","last_clean":"2024-09-11T03:52:56.578316+0000","last_became_active":"2024-09-11T03:50:07.079331+0000","last_became_peered":"2024-09-11T03:50:07.079331+0000","last_unstale":"2024-09-11T03:52:56.578316+0000","last_undegraded":"2024-09-11T03:52:56.578316+0000","last_fullsized":"2024-09-11T03:52:56.578316+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:52:41.498628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:41.498628+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:58.782385+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015587279999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.34","version":"957'441","reported_seq":4781,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577627+0000","last_change":"2024-09-11T03:52:55.557294+0000","last_active":"2024-09-11T03:52:56.577627+0000","last_peered":"2024-09-11T03:52:56.577627+0000","last_clean":"2024-09-11T03:52:56.577627+0000","last_became_active":"2024-09-11T03:50:07.080419+0000","last_became_peered":"2024-09-11T03:50:07.080419+0000","last_unstale":"2024-09-11T03:52:56.577627+0000","last_undegraded":"2024-09-11T03:52:56.577627+0000","last_fullsized":"2024-09-11T03:52:56.577627+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"863'388","last_scrub_stamp":"2024-09-11T03:51:22.429302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.429302+0000","objects_scrubbed":6,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.012371871,"stat_sum":{"num_bytes":13725407,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":37369,"num_write":17,"num_write_kb":4419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.33","version":"274'111","reported_seq":1891,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.562581+0000","last_change":"2024-09-11T03:52:55.562891+0000","last_active":"2024-09-11T03:52:55.562581+0000","last_peered":"2024-09-11T03:52:55.562581+0000","last_clean":"2024-09-11T03:52:55.562581+0000","last_became_active":"2024-09-11T03:50:07.077684+0000","last_became_peered":"2024-09-11T03:50:07.077684+0000","last_unstale":"2024-09-11T03:52:55.562581+0000","last_undegraded":"2024-09-11T03:52:55.562581+0000","last_fullsized":"2024-09-11T03:52:55.562581+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:51:28.001189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.001189+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.013248015,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.32","version":"957'645","reported_seq":5004,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.564006+0000","last_change":"2024-09-11T03:52:55.564277+0000","last_active":"2024-09-11T03:52:55.564006+0000","last_peered":"2024-09-11T03:52:55.564006+0000","last_clean":"2024-09-11T03:52:55.564006+0000","last_became_active":"2024-09-11T03:51:20.067057+0000","last_became_peered":"2024-09-11T03:51:20.067057+0000","last_unstale":"2024-09-11T03:52:55.564006+0000","last_undegraded":"2024-09-11T03:52:55.564006+0000","last_fullsized":"2024-09-11T03:52:55.564006+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"885'574","last_scrub_stamp":"2024-09-11T03:51:44.094173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:44.094173+0000","objects_scrubbed":3,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:08.638633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.012267442999999999,"stat_sum":{"num_bytes":12701035,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":10413,"num_write":13,"num_write_kb":6040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":2772362,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.31","version":"274'107","reported_seq":1568,"reported_epoch":957,"state":"active+undersized","last_fresh":"2024-09-11T03:52:55.551377+0000","last_change":"2024-09-11T03:52:38.094922+0000","last_active":"2024-09-11T03:52:55.551377+0000","last_peered":"2024-09-11T03:52:55.551377+0000","last_clean":"2024-09-11T03:46:13.965412+0000","last_became_active":"2024-09-11T03:52:38.094922+0000","last_became_peered":"2024-09-11T03:52:38.094922+0000","last_unstale":"2024-09-11T03:52:55.551377+0000","last_undegraded":"2024-09-11T03:52:55.551377+0000","last_fullsized":"2024-09-11T03:52:38.092374+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:51:26.087112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:26.087112+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":["10","7"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.30","version":"274'67","reported_seq":859,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.549323+0000","last_change":"2024-09-11T03:52:55.555504+0000","last_active":"2024-09-11T03:52:55.549323+0000","last_peered":"2024-09-11T03:52:55.549323+0000","last_clean":"2024-09-11T03:52:55.549323+0000","last_became_active":"2024-09-11T03:51:17.065156+0000","last_became_peered":"2024-09-11T03:51:17.065156+0000","last_unstale":"2024-09-11T03:52:55.549323+0000","last_undegraded":"2024-09-11T03:52:55.549323+0000","last_fullsized":"2024-09-11T03:52:55.549323+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:51:21.989674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.989674+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0075216570000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.2f","version":"557'465","reported_seq":3705,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.095577+0000","last_change":"2024-09-11T03:52:55.561639+0000","last_active":"2024-09-11T03:52:57.095577+0000","last_peered":"2024-09-11T03:52:57.095577+0000","last_clean":"2024-09-11T03:52:57.095577+0000","last_became_active":"2024-09-11T03:50:07.077840+0000","last_became_peered":"2024-09-11T03:50:07.077840+0000","last_unstale":"2024-09-11T03:52:57.095577+0000","last_undegraded":"2024-09-11T03:52:57.095577+0000","last_fullsized":"2024-09-11T03:52:57.095577+0000","mapping_epoch":788,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:51:47.522706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:47.522706+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:06.083285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0054563820000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.2e","version":"952'1287","reported_seq":5497,"reported_epoch":957,"state":"active+undersized+degraded","last_fresh":"2024-09-11T03:52:55.546873+0000","last_change":"2024-09-11T03:52:38.098202+0000","last_active":"2024-09-11T03:52:55.546873+0000","last_peered":"2024-09-11T03:52:55.546873+0000","last_clean":"2024-09-11T03:52:34.089064+0000","last_became_active":"2024-09-11T03:52:38.098202+0000","last_became_peered":"2024-09-11T03:52:38.098202+0000","last_unstale":"2024-09-11T03:52:55.546873+0000","last_undegraded":"2024-09-11T03:52:38.090482+0000","last_fullsized":"2024-09-11T03:52:38.090287+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":783,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":6,"last_scrub":"868'1145","last_scrub_stamp":"2024-09-11T03:51:27.034758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:27.034758+0000","objects_scrubbed":11,"log_size":1287,"log_dups_size":0,"ondisk_log_size":1287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.015381026000000001,"stat_sum":{"num_bytes":39404980,"num_objects":19,"num_object_clones":17,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":48,"num_read_kb":44167,"num_write":44,"num_write_kb":13661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":24991924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":["8","3"],"object_location_counts":[{"shards":"3,8","objects":19}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.2d","version":"957'1429","reported_seq":5759,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.243273+0000","last_change":"2024-09-11T03:52:55.560274+0000","last_active":"2024-09-11T03:52:57.243273+0000","last_peered":"2024-09-11T03:52:57.243273+0000","last_clean":"2024-09-11T03:52:57.243273+0000","last_became_active":"2024-09-11T03:49:29.163201+0000","last_became_peered":"2024-09-11T03:49:29.163201+0000","last_unstale":"2024-09-11T03:52:57.243273+0000","last_undegraded":"2024-09-11T03:52:57.243273+0000","last_fullsized":"2024-09-11T03:52:57.243273+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"921'1359","last_scrub_stamp":"2024-09-11T03:52:20.071629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:20.071629+0000","objects_scrubbed":20,"log_size":1429,"log_dups_size":0,"ondisk_log_size":1429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:45.362251+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0094437740000000003,"stat_sum":{"num_bytes":38389435,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":200,"num_read_kb":277796,"num_write":53,"num_write_kb":24676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.2c","version":"949'983","reported_seq":5293,"reported_epoch":957,"state":"active+undersized+degraded","last_fresh":"2024-09-11T03:52:55.545674+0000","last_change":"2024-09-11T03:52:38.100483+0000","last_active":"2024-09-11T03:52:55.545674+0000","last_peered":"2024-09-11T03:52:55.545674+0000","last_clean":"2024-09-11T03:52:36.083409+0000","last_became_active":"2024-09-11T03:52:38.100483+0000","last_became_peered":"2024-09-11T03:52:38.100483+0000","last_unstale":"2024-09-11T03:52:55.545674+0000","last_undegraded":"2024-09-11T03:52:38.092230+0000","last_fullsized":"2024-09-11T03:52:38.092059+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"908'959","last_scrub_stamp":"2024-09-11T03:52:08.323831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:08.323831+0000","objects_scrubbed":9,"log_size":983,"log_dups_size":0,"ondisk_log_size":983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:12.575000+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0089482030000000001,"stat_sum":{"num_bytes":24115843,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":55,"num_read_kb":47015,"num_write":58,"num_write_kb":20062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":["8","3"],"object_location_counts":[{"shards":"3,8","objects":10}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.2b","version":"568'646","reported_seq":4112,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.095287+0000","last_change":"2024-09-11T03:52:55.562829+0000","last_active":"2024-09-11T03:52:57.095287+0000","last_peered":"2024-09-11T03:52:57.095287+0000","last_clean":"2024-09-11T03:52:57.095287+0000","last_became_active":"2024-09-11T03:51:20.062485+0000","last_became_peered":"2024-09-11T03:51:20.062485+0000","last_unstale":"2024-09-11T03:52:57.095287+0000","last_undegraded":"2024-09-11T03:52:57.095287+0000","last_fullsized":"2024-09-11T03:52:57.095287+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:51:52.556581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.556581+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:14.104144+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0060099300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.2a","version":"727'348","reported_seq":4587,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.549331+0000","last_change":"2024-09-11T03:52:55.549605+0000","last_active":"2024-09-11T03:52:55.549331+0000","last_peered":"2024-09-11T03:52:55.549331+0000","last_clean":"2024-09-11T03:52:55.549331+0000","last_became_active":"2024-09-11T03:49:29.160595+0000","last_became_peered":"2024-09-11T03:49:29.160595+0000","last_unstale":"2024-09-11T03:52:55.549331+0000","last_undegraded":"2024-09-11T03:52:55.549331+0000","last_fullsized":"2024-09-11T03:52:55.549331+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"727'348","last_scrub_stamp":"2024-09-11T03:52:12.045079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:12.045079+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:15.060499+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0015045939999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,5],"acting":[7,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.29","version":"957'1410","reported_seq":5841,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.243229+0000","last_change":"2024-09-11T03:52:55.555152+0000","last_active":"2024-09-11T03:52:57.243229+0000","last_peered":"2024-09-11T03:52:57.243229+0000","last_clean":"2024-09-11T03:52:57.243229+0000","last_became_active":"2024-09-11T03:49:29.157194+0000","last_became_peered":"2024-09-11T03:49:29.157194+0000","last_unstale":"2024-09-11T03:52:57.243229+0000","last_undegraded":"2024-09-11T03:52:57.243229+0000","last_fullsized":"2024-09-11T03:52:57.243229+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"838'1272","last_scrub_stamp":"2024-09-11T03:50:57.773854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:57.773854+0000","objects_scrubbed":14,"log_size":1410,"log_dups_size":0,"ondisk_log_size":1410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0074720790000000004,"stat_sum":{"num_bytes":29247185,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":102294,"num_write":49,"num_write_kb":19231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8],"acting":[6,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.28","version":"945'484","reported_seq":4827,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577200+0000","last_change":"2024-09-11T03:52:55.538496+0000","last_active":"2024-09-11T03:52:56.577200+0000","last_peered":"2024-09-11T03:52:56.577200+0000","last_clean":"2024-09-11T03:52:56.577200+0000","last_became_active":"2024-09-11T03:49:29.154884+0000","last_became_peered":"2024-09-11T03:49:29.154884+0000","last_unstale":"2024-09-11T03:52:56.577200+0000","last_undegraded":"2024-09-11T03:52:56.577200+0000","last_fullsized":"2024-09-11T03:52:56.577200+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"905'477","last_scrub_stamp":"2024-09-11T03:52:07.670614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:07.670614+0000","objects_scrubbed":6,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:16.196539+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00069933799999999998,"stat_sum":{"num_bytes":10236090,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":14,"num_read_kb":11529,"num_write":23,"num_write_kb":4588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.27","version":"957'1305","reported_seq":5863,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.567032+0000","last_change":"2024-09-11T03:52:55.567213+0000","last_active":"2024-09-11T03:52:55.567032+0000","last_peered":"2024-09-11T03:52:55.567032+0000","last_clean":"2024-09-11T03:52:55.567032+0000","last_became_active":"2024-09-11T03:51:20.062110+0000","last_became_peered":"2024-09-11T03:51:20.062110+0000","last_unstale":"2024-09-11T03:52:55.567032+0000","last_undegraded":"2024-09-11T03:52:55.567032+0000","last_fullsized":"2024-09-11T03:52:55.567032+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"909'1255","last_scrub_stamp":"2024-09-11T03:52:10.054644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:10.054644+0000","objects_scrubbed":13,"log_size":1305,"log_dups_size":0,"ondisk_log_size":1305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:22.942734+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.016967896999999999,"stat_sum":{"num_bytes":31656611,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":66,"num_read_kb":65899,"num_write":52,"num_write_kb":26661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":14737969,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.26","version":"269'171","reported_seq":4477,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.562954+0000","last_change":"2024-09-11T03:52:55.563239+0000","last_active":"2024-09-11T03:52:55.562954+0000","last_peered":"2024-09-11T03:52:55.562954+0000","last_clean":"2024-09-11T03:52:55.562954+0000","last_became_active":"2024-09-11T03:51:20.061265+0000","last_became_peered":"2024-09-11T03:51:20.061265+0000","last_unstale":"2024-09-11T03:52:55.562954+0000","last_undegraded":"2024-09-11T03:52:55.562954+0000","last_fullsized":"2024-09-11T03:52:55.562954+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":6,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:52:08.043740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:08.043740+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:33.190211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013069295,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,5],"acting":[7,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.25","version":"957'1686","reported_seq":6190,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.566040+0000","last_change":"2024-09-11T03:52:55.564891+0000","last_active":"2024-09-11T03:52:55.566040+0000","last_peered":"2024-09-11T03:52:55.566040+0000","last_clean":"2024-09-11T03:52:55.566040+0000","last_became_active":"2024-09-11T03:49:29.159733+0000","last_became_peered":"2024-09-11T03:49:29.159733+0000","last_unstale":"2024-09-11T03:52:55.566040+0000","last_undegraded":"2024-09-11T03:52:55.566040+0000","last_fullsized":"2024-09-11T03:52:55.566040+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"918'1582","last_scrub_stamp":"2024-09-11T03:52:17.392959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:17.392959+0000","objects_scrubbed":23,"log_size":1686,"log_dups_size":0,"ondisk_log_size":1686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:40.447539+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.018683999999999999,"stat_sum":{"num_bytes":45031183,"num_objects":24,"num_object_clones":21,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":102,"num_read_kb":94381,"num_write":97,"num_write_kb":27397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,6],"acting":[8,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.24","version":"957'782","reported_seq":5132,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.243256+0000","last_change":"2024-09-11T03:52:55.552584+0000","last_active":"2024-09-11T03:52:57.243256+0000","last_peered":"2024-09-11T03:52:57.243256+0000","last_clean":"2024-09-11T03:52:57.243256+0000","last_became_active":"2024-09-11T03:49:29.155740+0000","last_became_peered":"2024-09-11T03:49:29.155740+0000","last_unstale":"2024-09-11T03:52:57.243256+0000","last_undegraded":"2024-09-11T03:52:57.243256+0000","last_fullsized":"2024-09-11T03:52:57.243256+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":745,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":6,"last_scrub":"838'674","last_scrub_stamp":"2024-09-11T03:50:56.770184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:56.770184+0000","objects_scrubbed":7,"log_size":782,"log_dups_size":0,"ondisk_log_size":782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0041856610000000002,"stat_sum":{"num_bytes":19426954,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":9,"num_read_kb":3,"num_write":22,"num_write_kb":15061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,7],"acting":[6,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.23","version":"951'588","reported_seq":4912,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577803+0000","last_change":"2024-09-11T03:52:55.540629+0000","last_active":"2024-09-11T03:52:56.577803+0000","last_peered":"2024-09-11T03:52:56.577803+0000","last_clean":"2024-09-11T03:52:56.577803+0000","last_became_active":"2024-09-11T03:46:32.970100+0000","last_became_peered":"2024-09-11T03:46:32.970100+0000","last_unstale":"2024-09-11T03:52:56.577803+0000","last_undegraded":"2024-09-11T03:52:56.577803+0000","last_fullsized":"2024-09-11T03:52:56.577803+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"886'531","last_scrub_stamp":"2024-09-11T03:51:44.527530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:44.527530+0000","objects_scrubbed":5,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:52:59.244192+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00082476200000000004,"stat_sum":{"num_bytes":11706449,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":58,"num_read_kb":64783,"num_write":39,"num_write_kb":13976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.22","version":"531'216","reported_seq":4517,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.243242+0000","last_change":"2024-09-11T03:52:55.547862+0000","last_active":"2024-09-11T03:52:57.243242+0000","last_peered":"2024-09-11T03:52:57.243242+0000","last_clean":"2024-09-11T03:52:57.243242+0000","last_became_active":"2024-09-11T03:46:32.970633+0000","last_became_peered":"2024-09-11T03:46:32.970633+0000","last_unstale":"2024-09-11T03:52:57.243242+0000","last_undegraded":"2024-09-11T03:52:57.243242+0000","last_fullsized":"2024-09-11T03:52:57.243242+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"531'216","last_scrub_stamp":"2024-09-11T03:50:38.617374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:50:38.617374+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:57.242430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046881599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.21","version":"957'454","reported_seq":4860,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.578819+0000","last_change":"2024-09-11T03:52:55.563791+0000","last_active":"2024-09-11T03:52:55.578819+0000","last_peered":"2024-09-11T03:52:55.578819+0000","last_clean":"2024-09-11T03:52:55.578819+0000","last_became_active":"2024-09-11T03:46:36.024928+0000","last_became_peered":"2024-09-11T03:46:36.024928+0000","last_unstale":"2024-09-11T03:52:55.578819+0000","last_undegraded":"2024-09-11T03:52:55.578819+0000","last_fullsized":"2024-09-11T03:52:55.578819+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":6,"last_scrub":"910'410","last_scrub_stamp":"2024-09-11T03:52:13.122711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:13.122711+0000","objects_scrubbed":7,"log_size":454,"log_dups_size":0,"ondisk_log_size":454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:16.121471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.012288148,"stat_sum":{"num_bytes":21488430,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":83,"num_read_kb":97018,"num_write":61,"num_write_kb":26412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,8],"acting":[11,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.20","version":"957'404","reported_seq":4875,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.552368+0000","last_change":"2024-09-11T03:52:55.553480+0000","last_active":"2024-09-11T03:52:55.552368+0000","last_peered":"2024-09-11T03:52:55.552368+0000","last_clean":"2024-09-11T03:52:55.552368+0000","last_became_active":"2024-09-11T03:46:32.974375+0000","last_became_peered":"2024-09-11T03:46:32.974375+0000","last_unstale":"2024-09-11T03:52:55.552368+0000","last_undegraded":"2024-09-11T03:52:55.552368+0000","last_fullsized":"2024-09-11T03:52:55.552368+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":6,"last_scrub":"895'348","last_scrub_stamp":"2024-09-11T03:51:54.026522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.026522+0000","objects_scrubbed":6,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:08.160224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0045603199999999997,"stat_sum":{"num_bytes":17185325,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":99,"num_read_kb":144175,"num_write":32,"num_write_kb":14316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.1c","version":"806'776","reported_seq":5099,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.560346+0000","last_change":"2024-09-11T03:52:55.560820+0000","last_active":"2024-09-11T03:52:55.560346+0000","last_peered":"2024-09-11T03:52:55.560346+0000","last_clean":"2024-09-11T03:52:55.560346+0000","last_became_active":"2024-09-11T03:52:32.087400+0000","last_became_peered":"2024-09-11T03:52:32.087400+0000","last_unstale":"2024-09-11T03:52:55.560346+0000","last_undegraded":"2024-09-11T03:52:55.560346+0000","last_fullsized":"2024-09-11T03:52:55.560346+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":934,"parent":"0.0","parent_split_bits":5,"last_scrub":"806'776","last_scrub_stamp":"2024-09-11T03:52:48.587687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:48.587687+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:51.589864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067358210000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.c","version":"20'92","reported_seq":195,"reported_epoch":958,"state":"active+undersized+degraded","last_fresh":"2024-09-11T03:52:57.243502+0000","last_change":"2024-09-11T03:52:38.092300+0000","last_active":"2024-09-11T03:52:57.243502+0000","last_peered":"2024-09-11T03:52:57.243502+0000","last_clean":"2024-09-11T03:46:13.963351+0000","last_became_active":"2024-09-11T03:52:38.092300+0000","last_became_peered":"2024-09-11T03:52:38.092300+0000","last_unstale":"2024-09-11T03:52:57.243502+0000","last_undegraded":"2024-09-11T03:52:38.089950+0000","last_fullsized":"2024-09-11T03:52:38.089835+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:17.994888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:17.994888+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:20.131515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.f","version":"557'465","reported_seq":4828,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.554967+0000","last_change":"2024-09-11T03:52:55.555245+0000","last_active":"2024-09-11T03:52:55.554967+0000","last_peered":"2024-09-11T03:52:55.554967+0000","last_clean":"2024-09-11T03:52:55.554967+0000","last_became_active":"2024-09-11T03:52:20.175033+0000","last_became_peered":"2024-09-11T03:52:20.175033+0000","last_unstale":"2024-09-11T03:52:55.554967+0000","last_undegraded":"2024-09-11T03:52:55.554967+0000","last_fullsized":"2024-09-11T03:52:55.554967+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'465","last_scrub_stamp":"2024-09-11T03:52:37.446463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:37.446463+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:49.322387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070585270000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,11],"acting":[8,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.e","version":"782'1025","reported_seq":5341,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.095245+0000","last_change":"2024-09-11T03:52:55.555170+0000","last_active":"2024-09-11T03:52:57.095245+0000","last_peered":"2024-09-11T03:52:57.095245+0000","last_clean":"2024-09-11T03:52:57.095245+0000","last_became_active":"2024-09-11T03:52:20.164829+0000","last_became_peered":"2024-09-11T03:52:20.164829+0000","last_unstale":"2024-09-11T03:52:57.095245+0000","last_undegraded":"2024-09-11T03:52:57.095245+0000","last_fullsized":"2024-09-11T03:52:57.095245+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":0,"last_scrub":"782'1025","last_scrub_stamp":"2024-09-11T03:52:35.193240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:35.193240+0000","objects_scrubbed":0,"log_size":1025,"log_dups_size":0,"ondisk_log_size":1025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:03.460831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018005079999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,3],"acting":[9,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.d","version":"20'92","reported_seq":1558,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.549329+0000","last_change":"2024-09-11T03:51:17.066500+0000","last_active":"2024-09-11T03:52:55.549329+0000","last_peered":"2024-09-11T03:52:55.549329+0000","last_clean":"2024-09-11T03:52:55.549329+0000","last_became_active":"2024-09-11T03:51:17.066085+0000","last_became_peered":"2024-09-11T03:51:17.066085+0000","last_unstale":"2024-09-11T03:52:55.549329+0000","last_undegraded":"2024-09-11T03:52:55.549329+0000","last_fullsized":"2024-09-11T03:52:55.549329+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:58.932376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:58.932376+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:24.215070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"957'1097","reported_seq":5574,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.578205+0000","last_change":"2024-09-11T03:52:55.574436+0000","last_active":"2024-09-11T03:52:56.578205+0000","last_peered":"2024-09-11T03:52:56.578205+0000","last_clean":"2024-09-11T03:52:56.578205+0000","last_became_active":"2024-09-11T03:52:20.165149+0000","last_became_peered":"2024-09-11T03:52:20.165149+0000","last_unstale":"2024-09-11T03:52:56.578205+0000","last_undegraded":"2024-09-11T03:52:56.578205+0000","last_fullsized":"2024-09-11T03:52:56.578205+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":0,"last_scrub":"936'1093","last_scrub_stamp":"2024-09-11T03:52:36.146031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:36.146031+0000","objects_scrubbed":5,"log_size":1097,"log_dups_size":0,"ondisk_log_size":1097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:48.145508+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.029319825000000001,"stat_sum":{"num_bytes":11325089,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.e","version":"20'92","reported_seq":1273,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.550366+0000","last_change":"2024-09-11T03:51:17.063139+0000","last_active":"2024-09-11T03:52:55.550366+0000","last_peered":"2024-09-11T03:52:55.550366+0000","last_clean":"2024-09-11T03:52:55.550366+0000","last_became_active":"2024-09-11T03:51:17.061266+0000","last_became_peered":"2024-09-11T03:51:17.061266+0000","last_unstale":"2024-09-11T03:52:55.550366+0000","last_undegraded":"2024-09-11T03:52:55.550366+0000","last_fullsized":"2024-09-11T03:52:55.550366+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:38.843438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:38.843438+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:22.178010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"920'966","reported_seq":5315,"reported_epoch":957,"state":"active+undersized","last_fresh":"2024-09-11T03:52:55.559556+0000","last_change":"2024-09-11T03:52:38.100123+0000","last_active":"2024-09-11T03:52:55.559556+0000","last_peered":"2024-09-11T03:52:55.559556+0000","last_clean":"2024-09-11T03:52:36.087594+0000","last_became_active":"2024-09-11T03:52:38.100123+0000","last_became_peered":"2024-09-11T03:52:38.100123+0000","last_unstale":"2024-09-11T03:52:55.559556+0000","last_undegraded":"2024-09-11T03:52:55.559556+0000","last_fullsized":"2024-09-11T03:52:38.089327+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":0,"last_scrub":"920'966","last_scrub_stamp":"2024-09-11T03:52:36.060733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:36.060733+0000","objects_scrubbed":0,"log_size":966,"log_dups_size":0,"ondisk_log_size":966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:37.510332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015136515,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":["11","1"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"20'92","reported_seq":841,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.244030+0000","last_change":"2024-09-11T03:51:21.860818+0000","last_active":"2024-09-11T03:52:57.244030+0000","last_peered":"2024-09-11T03:52:57.244030+0000","last_clean":"2024-09-11T03:52:57.244030+0000","last_became_active":"2024-09-11T03:51:17.064761+0000","last_became_peered":"2024-09-11T03:51:17.064761+0000","last_unstale":"2024-09-11T03:52:57.244030+0000","last_undegraded":"2024-09-11T03:52:57.244030+0000","last_fullsized":"2024-09-11T03:52:57.244030+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:21.860769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.860769+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":1957,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.549018+0000","last_change":"2024-09-11T03:51:52.035008+0000","last_active":"2024-09-11T03:52:55.549018+0000","last_peered":"2024-09-11T03:52:55.549018+0000","last_clean":"2024-09-11T03:52:55.549018+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:52:55.549018+0000","last_undegraded":"2024-09-11T03:52:55.549018+0000","last_fullsized":"2024-09-11T03:52:55.549018+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:52.034950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.034950+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:03.753329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"845'562","reported_seq":4903,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577929+0000","last_change":"2024-09-11T03:52:55.554165+0000","last_active":"2024-09-11T03:52:56.577929+0000","last_peered":"2024-09-11T03:52:56.577929+0000","last_clean":"2024-09-11T03:52:56.577929+0000","last_became_active":"2024-09-11T03:51:08.055565+0000","last_became_peered":"2024-09-11T03:51:08.055565+0000","last_unstale":"2024-09-11T03:52:56.577929+0000","last_undegraded":"2024-09-11T03:52:56.577929+0000","last_fullsized":"2024-09-11T03:52:56.577929+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"845'562","last_scrub_stamp":"2024-09-11T03:52:47.129109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:47.129109+0000","objects_scrubbed":0,"log_size":562,"log_dups_size":0,"ondisk_log_size":562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:16.184842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0094107300000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.3","version":"20'92","reported_seq":1573,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577909+0000","last_change":"2024-09-11T03:51:27.564638+0000","last_active":"2024-09-11T03:52:56.577909+0000","last_peered":"2024-09-11T03:52:56.577909+0000","last_clean":"2024-09-11T03:52:56.577909+0000","last_became_active":"2024-09-11T03:51:17.059234+0000","last_became_peered":"2024-09-11T03:51:17.059234+0000","last_unstale":"2024-09-11T03:52:56.577909+0000","last_undegraded":"2024-09-11T03:52:56.577909+0000","last_fullsized":"2024-09-11T03:52:56.577909+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:27.564586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:27.564586+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"563'192","reported_seq":3386,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.555582+0000","last_change":"2024-09-11T03:52:55.555999+0000","last_active":"2024-09-11T03:52:55.555582+0000","last_peered":"2024-09-11T03:52:55.555582+0000","last_clean":"2024-09-11T03:52:55.555582+0000","last_became_active":"2024-09-11T03:50:31.044399+0000","last_became_peered":"2024-09-11T03:50:31.044399+0000","last_unstale":"2024-09-11T03:52:55.555582+0000","last_undegraded":"2024-09-11T03:52:55.555582+0000","last_fullsized":"2024-09-11T03:52:55.555582+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":813,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'192","last_scrub_stamp":"2024-09-11T03:52:10.500975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:10.500975+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:11.401786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0078611480000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.2","version":"20'92","reported_seq":1366,"reported_epoch":957,"state":"active+undersized","last_fresh":"2024-09-11T03:52:55.544773+0000","last_change":"2024-09-11T03:52:38.100681+0000","last_active":"2024-09-11T03:52:55.544773+0000","last_peered":"2024-09-11T03:52:55.544773+0000","last_clean":"2024-09-11T03:51:15.205597+0000","last_became_active":"2024-09-11T03:52:38.100681+0000","last_became_peered":"2024-09-11T03:52:38.100681+0000","last_unstale":"2024-09-11T03:52:55.544773+0000","last_undegraded":"2024-09-11T03:52:55.544773+0000","last_fullsized":"2024-09-11T03:52:38.090964+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:41.975660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:41.975660+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:43.471148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":["8","7"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"567'235","reported_seq":2776,"reported_epoch":958,"state":"active+undersized","last_fresh":"2024-09-11T03:52:57.315075+0000","last_change":"2024-09-11T03:52:38.089376+0000","last_active":"2024-09-11T03:52:57.315075+0000","last_peered":"2024-09-11T03:52:57.315075+0000","last_clean":"2024-09-11T03:46:33.969857+0000","last_became_active":"2024-09-11T03:52:38.089376+0000","last_became_peered":"2024-09-11T03:52:38.089376+0000","last_unstale":"2024-09-11T03:52:57.315075+0000","last_undegraded":"2024-09-11T03:52:57.315075+0000","last_fullsized":"2024-09-11T03:52:38.086542+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'235","last_scrub_stamp":"2024-09-11T03:51:50.027291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:50.027291+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:53:56.265419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059265800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"20'92","reported_seq":1137,"reported_epoch":957,"state":"active+undersized","last_fresh":"2024-09-11T03:52:55.551505+0000","last_change":"2024-09-11T03:52:38.100076+0000","last_active":"2024-09-11T03:52:55.551505+0000","last_peered":"2024-09-11T03:52:55.551505+0000","last_clean":"2024-09-11T03:52:36.085596+0000","last_became_active":"2024-09-11T03:52:38.100076+0000","last_became_peered":"2024-09-11T03:52:38.100076+0000","last_unstale":"2024-09-11T03:52:55.551505+0000","last_undegraded":"2024-09-11T03:52:55.551505+0000","last_fullsized":"2024-09-11T03:52:38.093828+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:29.996598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:29.996598+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":["10","9"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"843'319","reported_seq":4753,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.578067+0000","last_change":"2024-09-11T03:52:55.554860+0000","last_active":"2024-09-11T03:52:56.578067+0000","last_peered":"2024-09-11T03:52:56.578067+0000","last_clean":"2024-09-11T03:52:56.578067+0000","last_became_active":"2024-09-11T03:51:08.055103+0000","last_became_peered":"2024-09-11T03:51:08.055103+0000","last_unstale":"2024-09-11T03:52:56.578067+0000","last_undegraded":"2024-09-11T03:52:56.578067+0000","last_fullsized":"2024-09-11T03:52:56.578067+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"843'319","last_scrub_stamp":"2024-09-11T03:51:19.453435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.453435+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010032075999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.7","version":"20'92","reported_seq":827,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.243867+0000","last_change":"2024-09-11T03:52:08.021960+0000","last_active":"2024-09-11T03:52:57.243867+0000","last_peered":"2024-09-11T03:52:57.243867+0000","last_clean":"2024-09-11T03:52:57.243867+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:52:57.243867+0000","last_undegraded":"2024-09-11T03:52:57.243867+0000","last_fullsized":"2024-09-11T03:52:57.243867+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:08.021899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:08.021899+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:37.346818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"847'690","reported_seq":5058,"reported_epoch":958,"state":"active+undersized","last_fresh":"2024-09-11T03:52:56.577967+0000","last_change":"2024-09-11T03:52:38.101065+0000","last_active":"2024-09-11T03:52:56.577967+0000","last_peered":"2024-09-11T03:52:56.577967+0000","last_clean":"2024-09-11T03:52:36.089546+0000","last_became_active":"2024-09-11T03:52:38.101065+0000","last_became_peered":"2024-09-11T03:52:38.101065+0000","last_unstale":"2024-09-11T03:52:56.577967+0000","last_undegraded":"2024-09-11T03:52:56.577967+0000","last_fullsized":"2024-09-11T03:52:38.090344+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"847'690","last_scrub_stamp":"2024-09-11T03:51:24.473723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:24.473723+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028916649999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":["1","10"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"20'92","reported_seq":1144,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.548069+0000","last_change":"2024-09-11T03:51:53.155265+0000","last_active":"2024-09-11T03:52:55.548069+0000","last_peered":"2024-09-11T03:52:55.548069+0000","last_clean":"2024-09-11T03:52:55.548069+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:52:55.548069+0000","last_undegraded":"2024-09-11T03:52:55.548069+0000","last_fullsized":"2024-09-11T03:52:55.548069+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:53.155207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:53.155207+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:07.708610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"936'1285","reported_seq":5710,"reported_epoch":957,"state":"active+undersized+degraded","last_fresh":"2024-09-11T03:52:55.546956+0000","last_change":"2024-09-11T03:52:38.103481+0000","last_active":"2024-09-11T03:52:55.546956+0000","last_peered":"2024-09-11T03:52:55.546956+0000","last_clean":"2024-09-11T03:52:36.084215+0000","last_became_active":"2024-09-11T03:52:38.103481+0000","last_became_peered":"2024-09-11T03:52:38.103481+0000","last_unstale":"2024-09-11T03:52:55.546956+0000","last_undegraded":"2024-09-11T03:52:38.093930+0000","last_fullsized":"2024-09-11T03:52:38.093787+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"862'1215","last_scrub_stamp":"2024-09-11T03:51:21.601542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:21.601542+0000","objects_scrubbed":5,"log_size":1285,"log_dups_size":0,"ondisk_log_size":1285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.010958933000000001,"stat_sum":{"num_bytes":16924470,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":9,"num_write_kb":5648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":["8","1"],"object_location_counts":[{"shards":"1,8","objects":7}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"20'92","reported_seq":1135,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.550888+0000","last_change":"2024-09-11T03:52:08.468064+0000","last_active":"2024-09-11T03:52:55.550888+0000","last_peered":"2024-09-11T03:52:55.550888+0000","last_clean":"2024-09-11T03:52:55.550888+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:52:55.550888+0000","last_undegraded":"2024-09-11T03:52:55.550888+0000","last_fullsized":"2024-09-11T03:52:55.550888+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:08.468014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:08.468014+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:12.305464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"269'171","reported_seq":4487,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.578704+0000","last_change":"2024-09-11T03:52:55.542635+0000","last_active":"2024-09-11T03:52:56.578704+0000","last_peered":"2024-09-11T03:52:56.578704+0000","last_clean":"2024-09-11T03:52:56.578704+0000","last_became_active":"2024-09-11T03:51:08.055529+0000","last_became_peered":"2024-09-11T03:51:08.055529+0000","last_unstale":"2024-09-11T03:52:56.578704+0000","last_undegraded":"2024-09-11T03:52:56.578704+0000","last_fullsized":"2024-09-11T03:52:56.578704+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"269'171","last_scrub_stamp":"2024-09-11T03:51:22.407323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:22.407323+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:56.360283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017767219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.4","version":"20'92","reported_seq":1734,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.559320+0000","last_change":"2024-09-11T03:51:31.102151+0000","last_active":"2024-09-11T03:52:55.559320+0000","last_peered":"2024-09-11T03:52:55.559320+0000","last_clean":"2024-09-11T03:52:55.559320+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:52:55.559320+0000","last_undegraded":"2024-09-11T03:52:55.559320+0000","last_fullsized":"2024-09-11T03:52:55.559320+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:31.102098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:31.102098+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"957'1392","reported_seq":5909,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.243792+0000","last_change":"2024-09-11T03:52:55.558819+0000","last_active":"2024-09-11T03:52:57.243792+0000","last_peered":"2024-09-11T03:52:57.243792+0000","last_clean":"2024-09-11T03:52:57.243792+0000","last_became_active":"2024-09-11T03:51:08.057350+0000","last_became_peered":"2024-09-11T03:51:08.057350+0000","last_unstale":"2024-09-11T03:52:57.243792+0000","last_undegraded":"2024-09-11T03:52:57.243792+0000","last_fullsized":"2024-09-11T03:52:57.243792+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'1274","last_scrub_stamp":"2024-09-11T03:51:19.872739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.872739+0000","objects_scrubbed":12,"log_size":1392,"log_dups_size":0,"ondisk_log_size":1392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0088622360000000008,"stat_sum":{"num_bytes":38060903,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":23,"num_read_kb":28480,"num_write":13,"num_write_kb":5756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.b","version":"20'92","reported_seq":1540,"reported_epoch":957,"state":"active+undersized","last_fresh":"2024-09-11T03:52:55.559806+0000","last_change":"2024-09-11T03:52:38.101382+0000","last_active":"2024-09-11T03:52:55.559806+0000","last_peered":"2024-09-11T03:52:55.559806+0000","last_clean":"2024-09-11T03:52:36.088013+0000","last_became_active":"2024-09-11T03:52:38.101382+0000","last_became_peered":"2024-09-11T03:52:38.101382+0000","last_unstale":"2024-09-11T03:52:55.559806+0000","last_undegraded":"2024-09-11T03:52:55.559806+0000","last_fullsized":"2024-09-11T03:52:38.088981+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:47.050227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:47.050227+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:01.739658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":["11","5"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"744'374","reported_seq":4748,"reported_epoch":957,"state":"active+undersized","last_fresh":"2024-09-11T03:52:55.549662+0000","last_change":"2024-09-11T03:52:38.098579+0000","last_active":"2024-09-11T03:52:55.549662+0000","last_peered":"2024-09-11T03:52:55.549662+0000","last_clean":"2024-09-11T03:52:36.085332+0000","last_became_active":"2024-09-11T03:52:38.098579+0000","last_became_peered":"2024-09-11T03:52:38.098579+0000","last_unstale":"2024-09-11T03:52:55.549662+0000","last_undegraded":"2024-09-11T03:52:55.549662+0000","last_fullsized":"2024-09-11T03:52:38.090760+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'374","last_scrub_stamp":"2024-09-11T03:52:35.313056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:35.313056+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:04.026300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.010241729999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":["7","9"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"20'92","reported_seq":1895,"reported_epoch":958,"state":"active+undersized","last_fresh":"2024-09-11T03:52:57.095191+0000","last_change":"2024-09-11T03:52:38.095503+0000","last_active":"2024-09-11T03:52:57.095191+0000","last_peered":"2024-09-11T03:52:57.095191+0000","last_clean":"2024-09-11T03:52:36.089905+0000","last_became_active":"2024-09-11T03:52:38.095503+0000","last_became_peered":"2024-09-11T03:52:38.095503+0000","last_unstale":"2024-09-11T03:52:57.095191+0000","last_undegraded":"2024-09-11T03:52:57.095191+0000","last_fullsized":"2024-09-11T03:52:38.092655+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:50:18.933952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:50:18.933952+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:42.088624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":["9","6"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"846'1373","reported_seq":5797,"reported_epoch":958,"state":"active+undersized","last_fresh":"2024-09-11T03:52:56.577367+0000","last_change":"2024-09-11T03:52:38.091371+0000","last_active":"2024-09-11T03:52:56.577367+0000","last_peered":"2024-09-11T03:52:56.577367+0000","last_clean":"2024-09-11T03:52:36.088696+0000","last_became_active":"2024-09-11T03:52:38.091371+0000","last_became_peered":"2024-09-11T03:52:38.091371+0000","last_unstale":"2024-09-11T03:52:56.577367+0000","last_undegraded":"2024-09-11T03:52:56.577367+0000","last_fullsized":"2024-09-11T03:52:38.088969+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"846'1373","last_scrub_stamp":"2024-09-11T03:51:19.328459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.328459+0000","objects_scrubbed":0,"log_size":1373,"log_dups_size":0,"ondisk_log_size":1373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070362899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"20'92","reported_seq":1952,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.548042+0000","last_change":"2024-09-11T03:51:04.911507+0000","last_active":"2024-09-11T03:52:55.548042+0000","last_peered":"2024-09-11T03:52:55.548042+0000","last_clean":"2024-09-11T03:52:55.548042+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:52:55.548042+0000","last_undegraded":"2024-09-11T03:52:55.548042+0000","last_fullsized":"2024-09-11T03:52:55.548042+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:04.911446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:04.911446+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"957'449","reported_seq":4849,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.566948+0000","last_change":"2024-09-11T03:52:55.567147+0000","last_active":"2024-09-11T03:52:55.566948+0000","last_peered":"2024-09-11T03:52:55.566948+0000","last_clean":"2024-09-11T03:52:55.566948+0000","last_became_active":"2024-09-11T03:51:08.053749+0000","last_became_peered":"2024-09-11T03:51:08.053749+0000","last_unstale":"2024-09-11T03:52:55.566948+0000","last_undegraded":"2024-09-11T03:52:55.566948+0000","last_fullsized":"2024-09-11T03:52:55.566948+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"860'392","last_scrub_stamp":"2024-09-11T03:51:19.986753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:19.986753+0000","objects_scrubbed":5,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.010270932,"stat_sum":{"num_bytes":12981002,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":29,"num_read_kb":33509,"num_write":15,"num_write_kb":3726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.8","version":"20'92","reported_seq":1613,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577604+0000","last_change":"2024-09-11T03:51:55.165620+0000","last_active":"2024-09-11T03:52:56.577604+0000","last_peered":"2024-09-11T03:52:56.577604+0000","last_clean":"2024-09-11T03:52:56.577604+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:52:56.577604+0000","last_undegraded":"2024-09-11T03:52:56.577604+0000","last_fullsized":"2024-09-11T03:52:56.577604+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:55.165539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:55.165539+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:11.582309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"568'646","reported_seq":5009,"reported_epoch":957,"state":"active+undersized","last_fresh":"2024-09-11T03:52:55.546598+0000","last_change":"2024-09-11T03:52:38.103267+0000","last_active":"2024-09-11T03:52:55.546598+0000","last_peered":"2024-09-11T03:52:55.546598+0000","last_clean":"2024-09-11T03:52:36.083349+0000","last_became_active":"2024-09-11T03:52:38.103267+0000","last_became_peered":"2024-09-11T03:52:38.103267+0000","last_unstale":"2024-09-11T03:52:55.546598+0000","last_undegraded":"2024-09-11T03:52:55.546598+0000","last_fullsized":"2024-09-11T03:52:38.094151+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":850,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'646","last_scrub_stamp":"2024-09-11T03:51:20.608388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:20.608388+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001246121,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":["8","11"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"20'92","reported_seq":1563,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577482+0000","last_change":"2024-09-11T03:52:45.100984+0000","last_active":"2024-09-11T03:52:56.577482+0000","last_peered":"2024-09-11T03:52:56.577482+0000","last_clean":"2024-09-11T03:52:56.577482+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:52:56.577482+0000","last_undegraded":"2024-09-11T03:52:56.577482+0000","last_fullsized":"2024-09-11T03:52:56.577482+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:45.100934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:45.100934+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:52.060236+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"274'111","reported_seq":4423,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.095326+0000","last_change":"2024-09-11T03:52:55.563201+0000","last_active":"2024-09-11T03:52:57.095326+0000","last_peered":"2024-09-11T03:52:57.095326+0000","last_clean":"2024-09-11T03:52:57.095326+0000","last_became_active":"2024-09-11T03:52:20.171504+0000","last_became_peered":"2024-09-11T03:52:20.171504+0000","last_unstale":"2024-09-11T03:52:57.095326+0000","last_undegraded":"2024-09-11T03:52:57.095326+0000","last_fullsized":"2024-09-11T03:52:57.095326+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'111","last_scrub_stamp":"2024-09-11T03:52:38.086790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:38.086790+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:54.784075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062017230000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,7],"acting":[9,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.11","version":"20'92","reported_seq":47,"reported_epoch":957,"state":"active+undersized+degraded","last_fresh":"2024-09-11T03:52:55.545400+0000","last_change":"2024-09-11T03:52:38.098560+0000","last_active":"2024-09-11T03:52:55.545400+0000","last_peered":"2024-09-11T03:52:55.545400+0000","last_clean":"2024-09-11T03:39:57.842276+0000","last_became_active":"2024-09-11T03:52:38.098560+0000","last_became_peered":"2024-09-11T03:52:38.098560+0000","last_unstale":"2024-09-11T03:52:55.545400+0000","last_undegraded":"2024-09-11T03:52:38.091252+0000","last_fullsized":"2024-09-11T03:52:38.091141+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:10.061435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:10.061435+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:53:38.411522+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":["8","9"],"object_location_counts":[{"shards":"8,9","objects":1}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.12","version":"957'606","reported_seq":4980,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.560424+0000","last_change":"2024-09-11T03:52:55.560688+0000","last_active":"2024-09-11T03:52:55.560424+0000","last_peered":"2024-09-11T03:52:55.560424+0000","last_clean":"2024-09-11T03:52:55.560424+0000","last_became_active":"2024-09-11T03:52:20.174805+0000","last_became_peered":"2024-09-11T03:52:20.174805+0000","last_unstale":"2024-09-11T03:52:55.560424+0000","last_undegraded":"2024-09-11T03:52:55.560424+0000","last_fullsized":"2024-09-11T03:52:55.560424+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":5,"last_scrub":"936'604","last_scrub_stamp":"2024-09-11T03:52:39.442368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:39.442368+0000","objects_scrubbed":3,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:54.048979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.011616464,"stat_sum":{"num_bytes":6380630,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.12","version":"20'92","reported_seq":1544,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577963+0000","last_change":"2024-09-11T03:51:54.138565+0000","last_active":"2024-09-11T03:52:56.577963+0000","last_peered":"2024-09-11T03:52:56.577963+0000","last_clean":"2024-09-11T03:52:56.577963+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:52:56.577963+0000","last_undegraded":"2024-09-11T03:52:56.577963+0000","last_fullsized":"2024-09-11T03:52:56.577963+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:54.138504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:54.138504+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:20.328413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"274'107","reported_seq":3497,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.243879+0000","last_change":"2024-09-11T03:52:55.552927+0000","last_active":"2024-09-11T03:52:57.243879+0000","last_peered":"2024-09-11T03:52:57.243879+0000","last_clean":"2024-09-11T03:52:57.243879+0000","last_became_active":"2024-09-11T03:52:20.161864+0000","last_became_peered":"2024-09-11T03:52:20.161864+0000","last_unstale":"2024-09-11T03:52:57.243879+0000","last_undegraded":"2024-09-11T03:52:57.243879+0000","last_fullsized":"2024-09-11T03:52:57.243879+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'107","last_scrub_stamp":"2024-09-11T03:52:36.291397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:36.291397+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:52.937990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00442049,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,9],"acting":[6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.13","version":"20'92","reported_seq":1649,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577899+0000","last_change":"2024-09-11T03:51:52.573394+0000","last_active":"2024-09-11T03:52:56.577899+0000","last_peered":"2024-09-11T03:52:56.577899+0000","last_clean":"2024-09-11T03:52:56.577899+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:52:56.577899+0000","last_undegraded":"2024-09-11T03:52:56.577899+0000","last_fullsized":"2024-09-11T03:52:56.577899+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:52.573348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.573348+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"274'67","reported_seq":3434,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.546366+0000","last_change":"2024-09-11T03:52:55.546897+0000","last_active":"2024-09-11T03:52:55.546366+0000","last_peered":"2024-09-11T03:52:55.546366+0000","last_clean":"2024-09-11T03:52:55.546366+0000","last_became_active":"2024-09-11T03:52:20.179454+0000","last_became_peered":"2024-09-11T03:52:20.179454+0000","last_unstale":"2024-09-11T03:52:55.546366+0000","last_undegraded":"2024-09-11T03:52:55.546366+0000","last_fullsized":"2024-09-11T03:52:55.546366+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'67","last_scrub_stamp":"2024-09-11T03:52:36.434215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:36.434215+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:40.623909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010511769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.14","version":"20'92","reported_seq":1128,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.548663+0000","last_change":"2024-09-11T03:51:28.960262+0000","last_active":"2024-09-11T03:52:55.548663+0000","last_peered":"2024-09-11T03:52:55.548663+0000","last_clean":"2024-09-11T03:52:55.548663+0000","last_became_active":"2024-09-11T03:46:36.026459+0000","last_became_peered":"2024-09-11T03:46:36.026459+0000","last_unstale":"2024-09-11T03:52:55.548663+0000","last_undegraded":"2024-09-11T03:52:55.548663+0000","last_fullsized":"2024-09-11T03:52:55.548663+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:28.960214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.960214+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"274'312","reported_seq":3980,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.243932+0000","last_change":"2024-09-11T03:52:55.553961+0000","last_active":"2024-09-11T03:52:57.243932+0000","last_peered":"2024-09-11T03:52:57.243932+0000","last_clean":"2024-09-11T03:52:57.243932+0000","last_became_active":"2024-09-11T03:52:32.081805+0000","last_became_peered":"2024-09-11T03:52:32.081805+0000","last_unstale":"2024-09-11T03:52:57.243932+0000","last_undegraded":"2024-09-11T03:52:57.243932+0000","last_fullsized":"2024-09-11T03:52:57.243932+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":934,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'312","last_scrub_stamp":"2024-09-11T03:52:43.265953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:43.265953+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:10.812279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0044341729999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.15","version":"20'92","reported_seq":1523,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.549508+0000","last_change":"2024-09-11T03:52:45.397922+0000","last_active":"2024-09-11T03:52:55.549508+0000","last_peered":"2024-09-11T03:52:55.549508+0000","last_clean":"2024-09-11T03:52:55.549508+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:52:55.549508+0000","last_undegraded":"2024-09-11T03:52:55.549508+0000","last_fullsized":"2024-09-11T03:52:55.549508+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:45.397876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:45.397876+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:09.102868+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.16","version":"782'561","reported_seq":4953,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.578023+0000","last_change":"2024-09-11T03:52:55.557757+0000","last_active":"2024-09-11T03:52:56.578023+0000","last_peered":"2024-09-11T03:52:56.578023+0000","last_clean":"2024-09-11T03:52:56.578023+0000","last_became_active":"2024-09-11T03:52:32.103647+0000","last_became_peered":"2024-09-11T03:52:32.103647+0000","last_unstale":"2024-09-11T03:52:56.578023+0000","last_undegraded":"2024-09-11T03:52:56.578023+0000","last_fullsized":"2024-09-11T03:52:56.578023+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":934,"parent":"0.0","parent_split_bits":5,"last_scrub":"782'561","last_scrub_stamp":"2024-09-11T03:52:38.102525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:38.102525+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:44.333217+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.012581976,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.16","version":"20'92","reported_seq":1004,"reported_epoch":957,"state":"active+undersized","last_fresh":"2024-09-11T03:52:55.561634+0000","last_change":"2024-09-11T03:52:38.098586+0000","last_active":"2024-09-11T03:52:55.561634+0000","last_peered":"2024-09-11T03:52:55.561634+0000","last_clean":"2024-09-11T03:52:36.088876+0000","last_became_active":"2024-09-11T03:52:38.098586+0000","last_became_peered":"2024-09-11T03:52:38.098586+0000","last_unstale":"2024-09-11T03:52:55.561634+0000","last_undegraded":"2024-09-11T03:52:55.561634+0000","last_fullsized":"2024-09-11T03:52:38.088375+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:15.093294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:15.093294+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:39.639160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":["11","2"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.15","version":"274'273","reported_seq":4656,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.095172+0000","last_change":"2024-09-11T03:52:55.555016+0000","last_active":"2024-09-11T03:52:57.095172+0000","last_peered":"2024-09-11T03:52:57.095172+0000","last_clean":"2024-09-11T03:52:57.095172+0000","last_became_active":"2024-09-11T03:52:20.171796+0000","last_became_peered":"2024-09-11T03:52:20.171796+0000","last_unstale":"2024-09-11T03:52:57.095172+0000","last_undegraded":"2024-09-11T03:52:57.095172+0000","last_fullsized":"2024-09-11T03:52:57.095172+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":5,"last_scrub":"274'273","last_scrub_stamp":"2024-09-11T03:52:36.155782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:36.155782+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:59.672179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002144647,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.17","version":"20'92","reported_seq":1544,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.547991+0000","last_change":"2024-09-11T03:52:09.331476+0000","last_active":"2024-09-11T03:52:55.547991+0000","last_peered":"2024-09-11T03:52:55.547991+0000","last_clean":"2024-09-11T03:52:55.547991+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:52:55.547991+0000","last_undegraded":"2024-09-11T03:52:55.547991+0000","last_fullsized":"2024-09-11T03:52:55.547991+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:09.331402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:09.331402+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:12.316710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"778'353","reported_seq":4644,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.243618+0000","last_change":"2024-09-11T03:52:55.553596+0000","last_active":"2024-09-11T03:52:57.243618+0000","last_peered":"2024-09-11T03:52:57.243618+0000","last_clean":"2024-09-11T03:52:57.243618+0000","last_became_active":"2024-09-11T03:52:20.166884+0000","last_became_peered":"2024-09-11T03:52:20.166884+0000","last_unstale":"2024-09-11T03:52:57.243618+0000","last_undegraded":"2024-09-11T03:52:57.243618+0000","last_fullsized":"2024-09-11T03:52:57.243618+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":922,"parent":"0.0","parent_split_bits":5,"last_scrub":"778'353","last_scrub_stamp":"2024-09-11T03:52:35.288690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:35.288690+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:46.957994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047831829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.18","version":"20'92","reported_seq":1664,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.577474+0000","last_change":"2024-09-11T03:52:48.175093+0000","last_active":"2024-09-11T03:52:56.577474+0000","last_peered":"2024-09-11T03:52:56.577474+0000","last_clean":"2024-09-11T03:52:56.577474+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:52:56.577474+0000","last_undegraded":"2024-09-11T03:52:56.577474+0000","last_fullsized":"2024-09-11T03:52:56.577474+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:48.175041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:48.175041+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:04.676155+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"957'981","reported_seq":5501,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.566891+0000","last_change":"2024-09-11T03:52:55.567146+0000","last_active":"2024-09-11T03:52:55.566891+0000","last_peered":"2024-09-11T03:52:55.566891+0000","last_clean":"2024-09-11T03:52:55.566891+0000","last_became_active":"2024-09-11T03:52:32.095621+0000","last_became_peered":"2024-09-11T03:52:32.095621+0000","last_unstale":"2024-09-11T03:52:55.566891+0000","last_undegraded":"2024-09-11T03:52:55.566891+0000","last_fullsized":"2024-09-11T03:52:55.566891+0000","mapping_epoch":812,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":934,"parent":"0.0","parent_split_bits":5,"last_scrub":"936'960","last_scrub_stamp":"2024-09-11T03:52:37.069991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:37.069991+0000","objects_scrubbed":8,"log_size":981,"log_dups_size":0,"ondisk_log_size":981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:45.697720+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0078280060000000002,"stat_sum":{"num_bytes":18589430,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":4696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.19","version":"20'92","reported_seq":1143,"reported_epoch":957,"state":"active+undersized","last_fresh":"2024-09-11T03:52:55.551302+0000","last_change":"2024-09-11T03:52:38.100136+0000","last_active":"2024-09-11T03:52:55.551302+0000","last_peered":"2024-09-11T03:52:55.551302+0000","last_clean":"2024-09-11T03:52:36.085504+0000","last_became_active":"2024-09-11T03:52:38.100136+0000","last_became_peered":"2024-09-11T03:52:38.100136+0000","last_unstale":"2024-09-11T03:52:55.551302+0000","last_undegraded":"2024-09-11T03:52:55.551302+0000","last_fullsized":"2024-09-11T03:52:38.092796+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:28.080420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:28.080420+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":["10","6"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"557'273","reported_seq":4573,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.243672+0000","last_change":"2024-09-11T03:52:55.548197+0000","last_active":"2024-09-11T03:52:57.243672+0000","last_peered":"2024-09-11T03:52:57.243672+0000","last_clean":"2024-09-11T03:52:57.243672+0000","last_became_active":"2024-09-11T03:52:32.077303+0000","last_became_peered":"2024-09-11T03:52:32.077303+0000","last_unstale":"2024-09-11T03:52:57.243672+0000","last_undegraded":"2024-09-11T03:52:57.243672+0000","last_fullsized":"2024-09-11T03:52:57.243672+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":934,"parent":"0.0","parent_split_bits":5,"last_scrub":"557'273","last_scrub_stamp":"2024-09-11T03:52:37.269225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:37.269225+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:05.183918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066429399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,2],"acting":[6,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.1a","version":"20'92","reported_seq":692,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.547997+0000","last_change":"2024-09-11T03:51:23.032108+0000","last_active":"2024-09-11T03:52:55.547997+0000","last_peered":"2024-09-11T03:52:55.547997+0000","last_clean":"2024-09-11T03:52:55.547997+0000","last_became_active":"2024-09-11T03:51:17.063957+0000","last_became_peered":"2024-09-11T03:51:17.063957+0000","last_unstale":"2024-09-11T03:52:55.547997+0000","last_undegraded":"2024-09-11T03:52:55.547997+0000","last_fullsized":"2024-09-11T03:52:55.547997+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:23.032056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:23.032056+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.19","version":"945'665","reported_seq":5068,"reported_epoch":958,"state":"active+undersized+degraded","last_fresh":"2024-09-11T03:52:57.243870+0000","last_change":"2024-09-11T03:52:38.092770+0000","last_active":"2024-09-11T03:52:57.243870+0000","last_peered":"2024-09-11T03:52:57.243870+0000","last_clean":"2024-09-11T03:52:36.089643+0000","last_became_active":"2024-09-11T03:52:38.092770+0000","last_became_peered":"2024-09-11T03:52:38.092770+0000","last_unstale":"2024-09-11T03:52:57.243870+0000","last_undegraded":"2024-09-11T03:52:38.089969+0000","last_fullsized":"2024-09-11T03:52:38.089820+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":934,"parent":"0.0","parent_split_bits":5,"last_scrub":"916'643","last_scrub_stamp":"2024-09-11T03:52:16.094749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:16.094749+0000","objects_scrubbed":7,"log_size":665,"log_dups_size":0,"ondisk_log_size":665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-11T03:53:56.205675+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0073625770000000004,"stat_sum":{"num_bytes":11394646,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":8645,"num_write":11,"num_write_kb":6791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1b","version":"20'92","reported_seq":1516,"reported_epoch":957,"state":"active+undersized","last_fresh":"2024-09-11T03:52:55.546667+0000","last_change":"2024-09-11T03:52:38.101687+0000","last_active":"2024-09-11T03:52:55.546667+0000","last_peered":"2024-09-11T03:52:55.546667+0000","last_clean":"2024-09-11T03:52:36.083090+0000","last_became_active":"2024-09-11T03:52:38.101687+0000","last_became_peered":"2024-09-11T03:52:38.101687+0000","last_unstale":"2024-09-11T03:52:55.546667+0000","last_undegraded":"2024-09-11T03:52:55.546667+0000","last_fullsized":"2024-09-11T03:52:38.091570+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:51:52.434784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:51:52.434784+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:13.938385+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":["8","9"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"954'798","reported_seq":5317,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:57.095414+0000","last_change":"2024-09-11T03:52:55.563679+0000","last_active":"2024-09-11T03:52:57.095414+0000","last_peered":"2024-09-11T03:52:57.095414+0000","last_clean":"2024-09-11T03:52:57.095414+0000","last_became_active":"2024-09-11T03:52:32.095541+0000","last_became_peered":"2024-09-11T03:52:32.095541+0000","last_unstale":"2024-09-11T03:52:57.095414+0000","last_undegraded":"2024-09-11T03:52:57.095414+0000","last_fullsized":"2024-09-11T03:52:57.095414+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":934,"parent":"0.0","parent_split_bits":5,"last_scrub":"936'783","last_scrub_stamp":"2024-09-11T03:52:37.120166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:37.120166+0000","objects_scrubbed":8,"log_size":798,"log_dups_size":0,"ondisk_log_size":798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:38.925646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.010166211,"stat_sum":{"num_bytes":14593925,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":3,"num_read_kb":1,"num_write":5,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,11],"acting":[9,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"2.1f","version":"932'941","reported_seq":5172,"reported_epoch":957,"state":"active+undersized+degraded","last_fresh":"2024-09-11T03:52:55.550583+0000","last_change":"2024-09-11T03:52:38.100802+0000","last_active":"2024-09-11T03:52:55.550583+0000","last_peered":"2024-09-11T03:52:55.550583+0000","last_clean":"2024-09-11T03:52:30.216817+0000","last_became_active":"2024-09-11T03:52:38.100802+0000","last_became_peered":"2024-09-11T03:52:38.100802+0000","last_unstale":"2024-09-11T03:52:55.550583+0000","last_undegraded":"2024-09-11T03:52:38.093565+0000","last_fullsized":"2024-09-11T03:52:38.093379+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":934,"parent":"0.0","parent_split_bits":5,"last_scrub":"932'941","last_scrub_stamp":"2024-09-11T03:52:36.875696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:36.875696+0000","objects_scrubbed":13,"log_size":941,"log_dups_size":0,"ondisk_log_size":941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0061294870000000003,"stat_sum":{"num_bytes":13661467,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":26,"num_read_kb":24835,"num_write":13,"num_write_kb":6834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":["7","8"],"object_location_counts":[{"shards":"7,8","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1c","version":"20'92","reported_seq":1527,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.546574+0000","last_change":"2024-09-11T03:52:14.409111+0000","last_active":"2024-09-11T03:52:55.546574+0000","last_peered":"2024-09-11T03:52:55.546574+0000","last_clean":"2024-09-11T03:52:55.546574+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:52:55.546574+0000","last_undegraded":"2024-09-11T03:52:55.546574+0000","last_fullsized":"2024-09-11T03:52:55.546574+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:14.409042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:14.409042+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:20.863513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1e","version":"929'1132","reported_seq":5362,"reported_epoch":957,"state":"active+undersized+degraded","last_fresh":"2024-09-11T03:52:55.547758+0000","last_change":"2024-09-11T03:52:38.096027+0000","last_active":"2024-09-11T03:52:55.547758+0000","last_peered":"2024-09-11T03:52:55.547758+0000","last_clean":"2024-09-11T03:52:27.192874+0000","last_became_active":"2024-09-11T03:52:38.096027+0000","last_became_peered":"2024-09-11T03:52:38.096027+0000","last_unstale":"2024-09-11T03:52:55.547758+0000","last_undegraded":"2024-09-11T03:52:38.091495+0000","last_fullsized":"2024-09-11T03:52:38.091345+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":934,"parent":"0.0","parent_split_bits":5,"last_scrub":"885'1106","last_scrub_stamp":"2024-09-11T03:51:44.050833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:51:44.050833+0000","objects_scrubbed":5,"log_size":1132,"log_dups_size":0,"ondisk_log_size":1132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0068695470000000002,"stat_sum":{"num_bytes":4835523,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":12181,"num_write":6,"num_write_kb":1715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":["10","3"],"object_location_counts":[{"shards":"3,10","objects":3}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1d","version":"20'92","reported_seq":1510,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-09-11T03:52:56.578440+0000","last_change":"2024-09-11T03:52:08.623749+0000","last_active":"2024-09-11T03:52:56.578440+0000","last_peered":"2024-09-11T03:52:56.578440+0000","last_clean":"2024-09-11T03:52:56.578440+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:52:56.578440+0000","last_undegraded":"2024-09-11T03:52:56.578440+0000","last_fullsized":"2024-09-11T03:52:56.578440+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:08.623692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:08.623692+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:16.913356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"567'752","reported_seq":5059,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.557551+0000","last_change":"2024-09-11T03:52:55.557873+0000","last_active":"2024-09-11T03:52:55.557551+0000","last_peered":"2024-09-11T03:52:55.557551+0000","last_clean":"2024-09-11T03:52:55.557551+0000","last_became_active":"2024-09-11T03:52:32.098831+0000","last_became_peered":"2024-09-11T03:52:32.098831+0000","last_unstale":"2024-09-11T03:52:55.557551+0000","last_undegraded":"2024-09-11T03:52:55.557551+0000","last_fullsized":"2024-09-11T03:52:55.557551+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":569,"last_epoch_clean":934,"parent":"0.0","parent_split_bits":5,"last_scrub":"567'752","last_scrub_stamp":"2024-09-11T03:52:37.444272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:38:28.875337+0000","last_clean_scrub_stamp":"2024-09-11T03:52:37.444272+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:50.479034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071649399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,11],"acting":[10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"175","length":"1"},{"start":"178","length":"1"},{"start":"193","length":"1"},{"start":"1aa","length":"1"},{"start":"1b2","length":"1"},{"start":"1b7","length":"1"}]},{"pgid":"1.1e","version":"20'92","reported_seq":1533,"reported_epoch":957,"state":"active+clean","last_fresh":"2024-09-11T03:52:55.561764+0000","last_change":"2024-09-11T03:52:17.013751+0000","last_active":"2024-09-11T03:52:55.561764+0000","last_peered":"2024-09-11T03:52:55.561764+0000","last_clean":"2024-09-11T03:52:55.561764+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:52:55.561764+0000","last_undegraded":"2024-09-11T03:52:55.561764+0000","last_fullsized":"2024-09-11T03:52:55.561764+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:52:17.013702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:52:17.013702+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:53:36.168162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":96,"stat_sum":{"num_bytes":779779776,"num_objects":348,"num_object_clones":298,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":14,"num_read":1330,"num_read_kb":1425854,"num_write":870,"num_write_kb":340995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":67177673,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1418063872,"data_stored":1886209769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61736,"ondisk_log_size":61736,"up":266,"acting":269,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":617203,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":84,"acting":84,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285757,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":212244,"kb_used_data":150528,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":93532908,"statfs":{"total":95995035648,"available":95777697792,"internally_reserved":0,"allocated":154140672,"data_stored":206693628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":63175108},"hb_peers":[1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514689,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":147836,"kb_used_data":94436,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93597316,"statfs":{"total":95995035648,"available":95843651584,"internally_reserved":0,"allocated":96702464,"data_stored":112552977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743623,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":178380,"kb_used_data":111092,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93566772,"statfs":{"total":95995035648,"available":95812374528,"internally_reserved":0,"allocated":113758208,"data_stored":141532008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037966,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":179540,"kb_used_data":130296,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93565612,"statfs":{"total":95995035648,"available":95811186688,"internally_reserved":0,"allocated":133423104,"data_stored":184652493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299601,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":216108,"kb_used_data":140492,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93529044,"statfs":{"total":95995035648,"available":95773741056,"internally_reserved":0,"allocated":143863808,"data_stored":176184214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528534,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":236400,"kb_used_data":178840,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93508752,"statfs":{"total":95995035648,"available":95752962048,"internally_reserved":0,"allocated":183132160,"data_stored":245051135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757468,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":188400,"kb_used_data":130840,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93556752,"statfs":{"total":95995035648,"available":95802114048,"internally_reserved":0,"allocated":133980160,"data_stored":171636821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986402,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149927,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":225196,"kb_used_data":149592,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93519956,"statfs":{"total":95995035648,"available":95764434944,"internally_reserved":0,"allocated":153182208,"data_stored":207173980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411564,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":164564,"kb_used_data":107008,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93580588,"statfs":{"total":95995035648,"available":95826522112,"internally_reserved":0,"allocated":109576192,"data_stored":126814763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575089,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":152784,"kb_used_data":103544,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93592368,"statfs":{"total":95995035648,"available":95838584832,"internally_reserved":0,"allocated":106029056,"data_stored":129996547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738610,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":162240,"kb_used_data":117160,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93582912,"statfs":{"total":95995035648,"available":95828901888,"internally_reserved":0,"allocated":119971840,"data_stored":142624878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":108261376,"data_stored":134819025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94400512,"data_stored":122345678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":97742848,"data_stored":118962646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":141213696,"data_stored":199187182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":96878592,"data_stored":122811969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":122146816,"data_stored":163784704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":171175936,"data_stored":237072586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":132042752,"data_stored":168340692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":124497920,"data_stored":181067530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":102117376,"data_stored":133872890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":85073920,"data_stored":104902108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":142512128,"data_stored":199042759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:53:02.564 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-11T03:53:02.564 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-11T03:53:02.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:53:02.636 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:02.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:02.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:53:02.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:02 smithi175 ceph-mon[30894]: from='client.17628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:53:02.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:02 smithi175 ceph-mon[30894]: pgmap v1596: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 744 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 3.1 MiB/s rd, 435 KiB/s wr, 5 op/s; 77/1053 objects degraded (7.312%) 2024-09-11T03:53:02.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:02 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:53:02.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:02 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:53:02.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:02 smithi175 ceph-mon[30894]: osdmap e960: 12 total, 11 up, 11 in 2024-09-11T03:53:02.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:02 smithi175 ceph-mon[30894]: Health check update: Degraded data redundancy: 77/1053 objects degraded (7.312%), 12 pgs degraded (PG_DEGRADED) 2024-09-11T03:53:02.738 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:53:02.760 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:02.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:02.761 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:53:02.797 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:02 smithi084 ceph-mon[27254]: from='client.17628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:53:02.797 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:02 smithi084 ceph-mon[27254]: pgmap v1596: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 744 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 3.1 MiB/s rd, 435 KiB/s wr, 5 op/s; 77/1053 objects degraded (7.312%) 2024-09-11T03:53:02.797 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:02 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:53:02.797 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:02 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:53:02.797 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:02 smithi084 ceph-mon[27254]: osdmap e960: 12 total, 11 up, 11 in 2024-09-11T03:53:02.797 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:02 smithi084 ceph-mon[27254]: Health check update: Degraded data redundancy: 77/1053 objects degraded (7.312%), 12 pgs degraded (PG_DEGRADED) 2024-09-11T03:53:02.862 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:53:02.872 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-11T03:53:02.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:02.885 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:02.885 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:53:02.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:02 smithi121 ceph-mon[30729]: from='client.17628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:53:02.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:02 smithi121 ceph-mon[30729]: pgmap v1596: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 744 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 3.1 MiB/s rd, 435 KiB/s wr, 5 op/s; 77/1053 objects degraded (7.312%) 2024-09-11T03:53:02.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:02 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:53:02.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:02 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:53:02.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:02 smithi121 ceph-mon[30729]: osdmap e960: 12 total, 11 up, 11 in 2024-09-11T03:53:02.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:02 smithi121 ceph-mon[30729]: Health check update: Degraded data redundancy: 77/1053 objects degraded (7.312%), 12 pgs degraded (PG_DEGRADED) 2024-09-11T03:53:02.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:53:03.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:03.009 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:53:03.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:53:03.111 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:53:03.122 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:53:03.132 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:03.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:03.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:53:03.223 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-11T03:53:03.234 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:53:03.239 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:53:03.239 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:53:03.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:03.257 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:03.257 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:53:03.258 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:53:03.258 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:53:03.272 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:53:03.272 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:53:03.289 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:53:03.289 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:53:03.306 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:53:03.306 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:53:03.323 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:53:03.323 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:53:03.340 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:53:03.341 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:53:03.357 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:53:03.357 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:53:03.358 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:53:03.375 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:53:03.375 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:53:03.379 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:03.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:53:03.399 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:53:03.399 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:53:03.416 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:53:03.416 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:53:03.482 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:53:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:03.504 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:03.504 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:53:03.605 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8 2024-09-11T03:53:03.628 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:03.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:03.630 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.8' 2024-09-11T03:53:03.640 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:53:03.640 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:53:03.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:53:03.756 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:03.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:53:03.814 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:03 smithi084 ceph-mon[27254]: 2.43 scrub starts 2024-09-11T03:53:03.814 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:03 smithi084 ceph-mon[27254]: 2.43 scrub ok 2024-09-11T03:53:03.814 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:03 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2733131682' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:53:03.844 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:53:03.858 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:53:03.880 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:03.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:03.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:53:03.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:03 smithi121 ceph-mon[30729]: 2.43 scrub starts 2024-09-11T03:53:03.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:03 smithi121 ceph-mon[30729]: 2.43 scrub ok 2024-09-11T03:53:03.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:03 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2733131682' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:53:03.966 INFO:teuthology.orchestra.run.smithi084.stdout:get_pg_num_history pg_num_history pg_num_history(e939 pg_nums {1={19=1,180=11,208=21,220=31},2={95=16,276=26,569=36,745=46,783=56,809=66,849=76,921=86,933=96}} deleted_pools ) 2024-09-11T03:53:03.966 INFO:teuthology.orchestra.run.smithi084.stdout:Importing pgid 2.19 2024-09-11T03:53:03.966 INFO:teuthology.orchestra.run.smithi084.stdout:Write #2:98bb65f5:::smithi17557870-6:13d# 2024-09-11T03:53:03.969 INFO:teuthology.orchestra.run.smithi084.stdout:Write #2:98bb65f5:::smithi17557870-6:167# 2024-09-11T03:53:03.973 INFO:teuthology.orchestra.run.smithi084.stdout:Write #2:98bb65f5:::smithi17557870-6:18a# 2024-09-11T03:53:03.975 INFO:teuthology.orchestra.run.smithi084.stdout:Write #2:98bb65f5:::smithi17557870-6:1a5# 2024-09-11T03:53:03.976 INFO:teuthology.orchestra.run.smithi084.stdout:Write #2:98bb65f5:::smithi17557870-6:1a9# 2024-09-11T03:53:03.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:03 smithi175 ceph-mon[30894]: 2.43 scrub starts 2024-09-11T03:53:03.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:03 smithi175 ceph-mon[30894]: 2.43 scrub ok 2024-09-11T03:53:03.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:03 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2733131682' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:53:03.978 INFO:teuthology.orchestra.run.smithi084.stdout:Write #2:98bb65f5:::smithi17557870-6:1ab# 2024-09-11T03:53:03.981 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:53:03.982 INFO:teuthology.orchestra.run.smithi084.stdout:Write #2:98bb65f5:::smithi17557870-6:head# 2024-09-11T03:53:03.982 INFO:teuthology.orchestra.run.smithi084.stdout:snapset 1ab=[]:{13d=[13c],167=[164,15f],18a=[183],1a5=[1a5,1a4,1a3,1a2,1a1,19d,19c,199,197,194,193,191,18b],1a9=[1a8,1a7,1a6],1ab=[1aa]} 2024-09-11T03:53:03.984 INFO:teuthology.orchestra.run.smithi084.stdout:write_pg epoch 936 info 2.19( v 936'663 (0'0,936'663] local-lis/les=933/934 n=3 ec=276/95 lis/c=933/933 les/c/f=934/934/0 sis=933) 2024-09-11T03:53:03.985 INFO:teuthology.orchestra.run.smithi084.stdout:Import successful 2024-09-11T03:53:04.002 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:04.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:04.003 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:53:04.104 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:53:04.166 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T03:53:04.166 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T03:53:04.174 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:04.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:04.175 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:53:04.203 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:04.236 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:04.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:53:04.302 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:04.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:04.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:53:04.404 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:53:04.430 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:04.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:53:04.471 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-11T03:53:04.532 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:53:04.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:04.557 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:53:04.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:53:04.658 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:53:04.684 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:04.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:04.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:53:04.692 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:04 smithi084 ceph-mon[27254]: pgmap v1598: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 744 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 3.0 MiB/s rd, 200 KiB/s wr, 3 op/s; 77/1053 objects degraded (7.312%) 2024-09-11T03:53:04.692 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:04 smithi084 ceph-mon[27254]: 2.4a scrub starts 2024-09-11T03:53:04.692 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:04 smithi084 ceph-mon[27254]: 2.4a scrub ok 2024-09-11T03:53:04.710 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:53:04.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:04 smithi175 ceph-mon[30894]: pgmap v1598: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 744 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 3.0 MiB/s rd, 200 KiB/s wr, 3 op/s; 77/1053 objects degraded (7.312%) 2024-09-11T03:53:04.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:04 smithi175 ceph-mon[30894]: 2.4a scrub starts 2024-09-11T03:53:04.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:04 smithi175 ceph-mon[30894]: 2.4a scrub ok 2024-09-11T03:53:04.786 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:53:04.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:04.809 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:53:04.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:53:04.910 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9 2024-09-11T03:53:04.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:04 smithi121 ceph-mon[30729]: pgmap v1598: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 744 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 3.0 MiB/s rd, 200 KiB/s wr, 3 op/s; 77/1053 objects degraded (7.312%) 2024-09-11T03:53:04.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:04 smithi121 ceph-mon[30729]: 2.4a scrub starts 2024-09-11T03:53:04.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:04 smithi121 ceph-mon[30729]: 2.4a scrub ok 2024-09-11T03:53:04.932 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:04.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.9' 2024-09-11T03:53:05.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:53:05.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:05.060 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:05.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:53:05.162 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1 2024-09-11T03:53:05.183 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:53:05.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:05.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.1' 2024-09-11T03:53:05.185 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:53:05.185 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:53:05.285 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:53:05.307 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:05.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:05.308 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:53:05.408 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:53:05.434 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:05.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:05.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:53:05.536 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:53:05.557 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:53:05.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:05.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:53:05.655 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_0' removed 2024-09-11T03:53:05.655 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:53:05.659 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:53:05.683 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:05.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:05.684 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:53:05.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:53:05.811 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:05.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:05.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:53:05.826 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:05 smithi084 ceph-mon[27254]: 2.45 scrub starts 2024-09-11T03:53:05.826 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:05 smithi084 ceph-mon[27254]: 2.45 scrub ok 2024-09-11T03:53:05.826 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:05 smithi084 ceph-mon[27254]: 2.46 scrub starts 2024-09-11T03:53:05.826 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:05 smithi084 ceph-mon[27254]: 2.46 scrub ok 2024-09-11T03:53:05.826 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:05 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1930386488' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:53:05.826 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:05 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4070269908' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-11T03:53:05.851 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/exp.2.19.0 2024-09-11T03:53:05.909 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:05 smithi175 ceph-mon[30894]: 2.45 scrub starts 2024-09-11T03:53:05.909 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:05 smithi175 ceph-mon[30894]: 2.45 scrub ok 2024-09-11T03:53:05.909 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:05 smithi175 ceph-mon[30894]: 2.46 scrub starts 2024-09-11T03:53:05.909 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:05 smithi175 ceph-mon[30894]: 2.46 scrub ok 2024-09-11T03:53:05.909 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:05 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1930386488' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:53:05.909 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:05 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4070269908' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-11T03:53:05.913 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10 2024-09-11T03:53:05.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:05 smithi121 ceph-mon[30729]: 2.45 scrub starts 2024-09-11T03:53:05.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:05 smithi121 ceph-mon[30729]: 2.45 scrub ok 2024-09-11T03:53:05.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:05 smithi121 ceph-mon[30729]: 2.46 scrub starts 2024-09-11T03:53:05.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:05 smithi121 ceph-mon[30729]: 2.46 scrub ok 2024-09-11T03:53:05.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:05 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1930386488' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:53:05.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:05 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4070269908' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-11T03:53:05.936 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:05.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:05.937 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.10' 2024-09-11T03:53:05.940 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-11T03:53:06.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:53:06.063 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:06.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:53:06.164 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2 2024-09-11T03:53:06.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T03:53:06.186 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:53:06.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:06.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.2' 2024-09-11T03:53:06.220 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:53:06.288 DEBUG:teuthology.orchestra.run.smithi175:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11 2024-09-11T03:53:06.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:06.310 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: docker: command not found 2024-09-11T03:53:06.310 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi175 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.11' 2024-09-11T03:53:06.411 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3 2024-09-11T03:53:06.432 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-11T03:53:06.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:06.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.3' 2024-09-11T03:53:06.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7 2024-09-11T03:53:06.560 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:06.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:06.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.7' 2024-09-11T03:53:06.662 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6 2024-09-11T03:53:06.687 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: docker: command not found 2024-09-11T03:53:06.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:53:06.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd.6' 2024-09-11T03:53:06.732 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-11T03:53:06.745 INFO:tasks.thrashosds:joining thrashosds 2024-09-11T03:53:06.746 INFO:teuthology.orchestra.run.smithi175.stdout:ERROR: (22) Invalid argument 2024-09-11T03:53:06.746 INFO:teuthology.orchestra.run.smithi175.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T03:53:06.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:06 smithi121 ceph-mon[30729]: pgmap v1599: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 744 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 2.3 MiB/s rd, 158 KiB/s wr, 2 op/s; 77/1053 objects degraded (7.312%) 2024-09-11T03:53:06.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:06 smithi121 ceph-mon[30729]: 1.3 scrub starts 2024-09-11T03:53:06.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:06 smithi121 ceph-mon[30729]: 1.3 scrub ok 2024-09-11T03:53:06.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:06 smithi121 ceph-mon[30729]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:53:06.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:06 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1930386488' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:53:06.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:06 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4070269908' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T03:53:06.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:06 smithi121 ceph-mon[30729]: osdmap e961: 12 total, 11 up, 11 in 2024-09-11T03:53:06.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:06 smithi175 ceph-mon[30894]: pgmap v1599: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 744 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 2.3 MiB/s rd, 158 KiB/s wr, 2 op/s; 77/1053 objects degraded (7.312%) 2024-09-11T03:53:06.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:06 smithi175 ceph-mon[30894]: 1.3 scrub starts 2024-09-11T03:53:06.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:06 smithi175 ceph-mon[30894]: 1.3 scrub ok 2024-09-11T03:53:06.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:06 smithi175 ceph-mon[30894]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:53:06.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:06 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1930386488' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:53:06.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:06 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4070269908' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T03:53:06.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:06 smithi175 ceph-mon[30894]: osdmap e961: 12 total, 11 up, 11 in 2024-09-11T03:53:07.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:06 smithi084 ceph-mon[27254]: pgmap v1599: 127 pgs: 3 active+clean+remapped, 16 active+undersized, 12 active+undersized+degraded, 96 active+clean; 744 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 2.3 MiB/s rd, 158 KiB/s wr, 2 op/s; 77/1053 objects degraded (7.312%) 2024-09-11T03:53:07.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:06 smithi084 ceph-mon[27254]: 1.3 scrub starts 2024-09-11T03:53:07.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:06 smithi084 ceph-mon[27254]: 1.3 scrub ok 2024-09-11T03:53:07.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:06 smithi084 ceph-mon[27254]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-11T03:53:07.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:06 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1930386488' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:53:07.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:06 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4070269908' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-11T03:53:07.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:06 smithi084 ceph-mon[27254]: osdmap e961: 12 total, 11 up, 11 in 2024-09-11T03:53:07.474 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-11T03:53:07.531 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:07.720 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:53:08.004 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-11T03:53:08.019 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-11T03:53:08.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T03:53:08.036 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-11T03:53:08.036 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T03:53:08.052 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-11T03:53:08.052 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T03:53:08.069 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-11T03:53:08.069 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T03:53:08.086 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-11T03:53:08.087 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T03:53:08.103 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-11T03:53:08.104 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T03:53:08.120 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-11T03:53:08.121 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T03:53:08.137 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8: {} 2024-09-11T03:53:08.137 INFO:teuthology.orchestra.run.smithi084.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-11T03:53:08.154 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9: {} 2024-09-11T03:53:08.155 INFO:teuthology.orchestra.run.smithi084.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-11T03:53:08.170 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:08 smithi175 ceph-mon[30894]: 1.14 scrub starts 2024-09-11T03:53:08.170 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:08 smithi175 ceph-mon[30894]: 1.14 scrub ok 2024-09-11T03:53:08.170 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:08 smithi175 ceph-mon[30894]: pgmap v1601: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:08.170 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:08 smithi175 ceph-mon[30894]: 1.f scrub starts 2024-09-11T03:53:08.170 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:08 smithi175 ceph-mon[30894]: 1.f scrub ok 2024-09-11T03:53:08.170 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:08 smithi175 ceph-mon[30894]: Health check update: Degraded data redundancy: 2/9 objects degraded (22.222%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T03:53:08.170 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:08 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2826856969' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:53:08.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10: {} 2024-09-11T03:53:08.171 INFO:teuthology.orchestra.run.smithi084.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-11T03:53:08.187 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11: {} 2024-09-11T03:53:08.188 INFO:teuthology.orchestra.run.smithi084.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-11T03:53:08.278 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:53:08.278 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:53:08.278 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:53:08.278 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:53:08.278 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:53:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:08 smithi084 ceph-mon[27254]: 1.14 scrub starts 2024-09-11T03:53:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:08 smithi084 ceph-mon[27254]: 1.14 scrub ok 2024-09-11T03:53:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:08 smithi084 ceph-mon[27254]: pgmap v1601: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:08 smithi084 ceph-mon[27254]: 1.f scrub starts 2024-09-11T03:53:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:08 smithi084 ceph-mon[27254]: 1.f scrub ok 2024-09-11T03:53:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:08 smithi084 ceph-mon[27254]: Health check update: Degraded data redundancy: 2/9 objects degraded (22.222%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T03:53:08.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:08 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2826856969' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:53:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:08 smithi121 ceph-mon[30729]: 1.14 scrub starts 2024-09-11T03:53:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:08 smithi121 ceph-mon[30729]: 1.14 scrub ok 2024-09-11T03:53:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:08 smithi121 ceph-mon[30729]: pgmap v1601: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:08 smithi121 ceph-mon[30729]: 1.f scrub starts 2024-09-11T03:53:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:08 smithi121 ceph-mon[30729]: 1.f scrub ok 2024-09-11T03:53:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:08 smithi121 ceph-mon[30729]: Health check update: Degraded data redundancy: 2/9 objects degraded (22.222%), 2 pgs degraded (PG_DEGRADED) 2024-09-11T03:53:08.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:08 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2826856969' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-11T03:53:08.918 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T03:53:08.919 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T03:53:08.943 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-11T03:53:09.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:09 smithi084 ceph-mon[27254]: 1.0 scrub starts 2024-09-11T03:53:09.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:09 smithi084 ceph-mon[27254]: 1.0 scrub ok 2024-09-11T03:53:09.183 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:53:09.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:09 smithi175 ceph-mon[30894]: 1.0 scrub starts 2024-09-11T03:53:09.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:09 smithi175 ceph-mon[30894]: 1.0 scrub ok 2024-09-11T03:53:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:09 smithi121 ceph-mon[30729]: 1.0 scrub starts 2024-09-11T03:53:09.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:09 smithi121 ceph-mon[30729]: 1.0 scrub ok 2024-09-11T03:53:09.553 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-11T03:53:09.634 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:53:09.634 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:53:09.634 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:53:09.634 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:53:09.634 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.634 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4021 2.7 2:e77ce595:::smithi17557870-31:head [copy-from ver 473] snapc ac=[ac,ab,a9,a8,a6,a3,a1,9e,9d] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-11T03:53:09.634 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:13.490109+0000", 2024-09-11T03:53:09.634 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 536.14307160400006, 2024-09-11T03:53:09.634 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.02074401, 2024-09-11T03:53:09.635 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.635 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.635 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:09.635 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:09.635 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:09.635 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:09.635 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4021 2024-09-11T03:53:09.635 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.635 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.635 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.635 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.635 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490109+0000", 2024-09-11T03:53:09.636 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.636 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.636 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.636 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.636 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490109+0000", 2024-09-11T03:53:09.636 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.636 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.636 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.636 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.636 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490111+0000", 2024-09-11T03:53:09.636 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.438e-06 2024-09-11T03:53:09.636 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.637 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.637 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.637 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490118+0000", 2024-09-11T03:53:09.637 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8950000000000001e-06 2024-09-11T03:53:09.637 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.637 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.637 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.637 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490131+0000", 2024-09-11T03:53:09.637 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2889e-05 2024-09-11T03:53:09.637 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.637 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.637 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.637 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490138+0000", 2024-09-11T03:53:09.638 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3849999999999996e-06 2024-09-11T03:53:09.638 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.638 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.638 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.638 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490199+0000", 2024-09-11T03:53:09.638 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0822000000000001e-05 2024-09-11T03:53:09.638 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.638 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.638 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.638 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490297+0000", 2024-09-11T03:53:09.638 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7726999999999995e-05 2024-09-11T03:53:09.638 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.639 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.639 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:53:09.639 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.499178+0000", 2024-09-11T03:53:09.639 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0088812119999999994 2024-09-11T03:53:09.639 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.639 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.639 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:09.639 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.502683+0000", 2024-09-11T03:53:09.639 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035055559999999999 2024-09-11T03:53:09.639 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.639 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.639 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:09.639 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.507713+0000", 2024-09-11T03:53:09.640 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050300299999999996 2024-09-11T03:53:09.640 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.640 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.640 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:09.640 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.510764+0000", 2024-09-11T03:53:09.640 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0030502469999999999 2024-09-11T03:53:09.640 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.640 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.640 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:09.640 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.510803+0000", 2024-09-11T03:53:09.640 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9629999999999998e-05 2024-09-11T03:53:09.640 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.640 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.641 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.641 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.510853+0000", 2024-09-11T03:53:09.641 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0179000000000002e-05 2024-09-11T03:53:09.641 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.641 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.641 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.641 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.641 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.641 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4022 2.7 2:e77ce595:::smithi17557870-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e435)", 2024-09-11T03:53:09.641 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:13.490154+0000", 2024-09-11T03:53:09.641 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 536.14302719, 2024-09-11T03:53:09.641 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.01199774, 2024-09-11T03:53:09.642 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.642 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.642 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:09.642 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:09.642 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:09.642 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:09.642 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4022 2024-09-11T03:53:09.642 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.642 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.642 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.642 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.642 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490154+0000", 2024-09-11T03:53:09.643 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.643 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.643 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.643 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.643 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490154+0000", 2024-09-11T03:53:09.643 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.643 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.643 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.643 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.643 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490154+0000", 2024-09-11T03:53:09.643 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3000000000000001e-07 2024-09-11T03:53:09.643 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.643 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.644 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.644 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490157+0000", 2024-09-11T03:53:09.644 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2460000000000002e-06 2024-09-11T03:53:09.644 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.644 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.644 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.644 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490163+0000", 2024-09-11T03:53:09.644 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6189999999999996e-06 2024-09-11T03:53:09.644 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.644 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.644 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.644 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490169+0000", 2024-09-11T03:53:09.644 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9139999999999996e-06 2024-09-11T03:53:09.645 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.645 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.645 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.645 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490486+0000", 2024-09-11T03:53:09.645 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00031683599999999997 2024-09-11T03:53:09.645 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.645 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.645 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:53:09.645 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.490520+0000", 2024-09-11T03:53:09.645 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4548999999999997e-05 2024-09-11T03:53:09.645 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.645 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.645 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.645 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.502009+0000", 2024-09-11T03:53:09.646 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011488437000000001 2024-09-11T03:53:09.646 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.646 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.646 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.646 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.502077+0000", 2024-09-11T03:53:09.646 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7890000000000002e-05 2024-09-11T03:53:09.646 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.646 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.646 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.646 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:13.502152+0000", 2024-09-11T03:53:09.646 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4819000000000003e-05 2024-09-11T03:53:09.646 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.646 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.646 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.647 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.647 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.647 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:4288 2.0 e462/417 2:0762776a:::smithi17557870-25:head v 462'115, mlcod=393'111)", 2024-09-11T03:53:09.647 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:44:39.987982+0000", 2024-09-11T03:53:09.647 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 509.64519935099997, 2024-09-11T03:53:09.647 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013639949, 2024-09-11T03:53:09.647 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.647 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.647 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:09.647 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.647 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.647 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.648 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.987982+0000", 2024-09-11T03:53:09.648 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.648 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.648 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.648 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.648 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.987982+0000", 2024-09-11T03:53:09.648 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.648 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.648 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.648 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.648 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.987982+0000", 2024-09-11T03:53:09.648 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.65e-07 2024-09-11T03:53:09.648 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.649 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.649 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.649 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.997342+0000", 2024-09-11T03:53:09.649 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0093600669999999997 2024-09-11T03:53:09.649 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.649 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.649 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.650 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.997361+0000", 2024-09-11T03:53:09.650 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8719000000000002e-05 2024-09-11T03:53:09.650 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.650 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.650 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.650 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.997370+0000", 2024-09-11T03:53:09.650 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.4259999999999992e-06 2024-09-11T03:53:09.651 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.651 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.651 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.651 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.997462+0000", 2024-09-11T03:53:09.651 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2015000000000006e-05 2024-09-11T03:53:09.651 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.651 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.651 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.651 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:39.997496+0000", 2024-09-11T03:53:09.651 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3837999999999997e-05 2024-09-11T03:53:09.651 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.651 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.652 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:09.652 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:40.001567+0000", 2024-09-11T03:53:09.652 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0040706969999999999 2024-09-11T03:53:09.652 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.652 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.652 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.652 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:44:40.001622+0000", 2024-09-11T03:53:09.652 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4922e-05 2024-09-11T03:53:09.652 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.652 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.652 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.652 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.653 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.653 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4907 2.7 2:e0ec5763:::smithi17557870-2:head [copy-from ver 761] snapc d7=[d7,d5,d4,d3,d2,d1,d0,ca] ondisk+write+known_if_redirected+supports_pool_eio e522)", 2024-09-11T03:53:09.653 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:40.106137+0000", 2024-09-11T03:53:09.653 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 449.52704419499997, 2024-09-11T03:53:09.653 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021335674999999998, 2024-09-11T03:53:09.653 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.653 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.653 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:09.653 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:09.653 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:09.654 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:09.654 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4907 2024-09-11T03:53:09.654 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.654 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.654 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.654 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.654 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106137+0000", 2024-09-11T03:53:09.654 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.654 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.654 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.654 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.654 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106137+0000", 2024-09-11T03:53:09.655 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.655 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.655 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.655 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.655 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106139+0000", 2024-09-11T03:53:09.655 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3250000000000002e-06 2024-09-11T03:53:09.656 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.656 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.656 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.656 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106149+0000", 2024-09-11T03:53:09.656 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8079999999999999e-06 2024-09-11T03:53:09.656 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.657 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.657 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.657 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106171+0000", 2024-09-11T03:53:09.657 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2507999999999999e-05 2024-09-11T03:53:09.657 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.658 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.658 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.658 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106184+0000", 2024-09-11T03:53:09.658 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2224e-05 2024-09-11T03:53:09.658 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.658 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.659 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.659 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106253+0000", 2024-09-11T03:53:09.659 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.9615000000000003e-05 2024-09-11T03:53:09.659 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.659 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.659 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.660 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106357+0000", 2024-09-11T03:53:09.660 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010383900000000001 2024-09-11T03:53:09.660 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.660 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.660 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,3", 2024-09-11T03:53:09.660 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.116615+0000", 2024-09-11T03:53:09.661 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010257779999999999 2024-09-11T03:53:09.661 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.661 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.661 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:09.661 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.120710+0000", 2024-09-11T03:53:09.661 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0040951579999999998 2024-09-11T03:53:09.662 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.662 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.662 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:09.662 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.127226+0000", 2024-09-11T03:53:09.662 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0065160610000000001 2024-09-11T03:53:09.663 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.663 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.663 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:09.663 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.127380+0000", 2024-09-11T03:53:09.663 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015357599999999999 2024-09-11T03:53:09.663 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.664 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.664 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:09.664 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.127423+0000", 2024-09-11T03:53:09.664 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3680999999999997e-05 2024-09-11T03:53:09.664 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.665 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.665 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.665 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.127473+0000", 2024-09-11T03:53:09.665 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9100000000000001e-05 2024-09-11T03:53:09.665 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.665 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.666 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.666 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.666 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.666 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4908 2.7 2:e0ec5763:::smithi17557870-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e522)", 2024-09-11T03:53:09.666 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:40.106211+0000", 2024-09-11T03:53:09.666 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 449.526970452, 2024-09-11T03:53:09.667 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013824826, 2024-09-11T03:53:09.667 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.667 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.667 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:09.667 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:09.667 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:09.668 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:09.668 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4908 2024-09-11T03:53:09.668 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.668 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.668 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.668 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.669 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106211+0000", 2024-09-11T03:53:09.669 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.669 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.669 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.669 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.669 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106211+0000", 2024-09-11T03:53:09.670 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.670 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.670 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.670 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.670 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106211+0000", 2024-09-11T03:53:09.670 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4200000000000001e-07 2024-09-11T03:53:09.670 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.670 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106214+0000", 2024-09-11T03:53:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9490000000000001e-06 2024-09-11T03:53:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106221+0000", 2024-09-11T03:53:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3300000000000001e-06 2024-09-11T03:53:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106225+0000", 2024-09-11T03:53:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5700000000000001e-06 2024-09-11T03:53:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106521+0000", 2024-09-11T03:53:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00029598599999999998 2024-09-11T03:53:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:53:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.106551+0000", 2024-09-11T03:53:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0165e-05 2024-09-11T03:53:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.119851+0000", 2024-09-11T03:53:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013300354 2024-09-11T03:53:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.119919+0000", 2024-09-11T03:53:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7569000000000003e-05 2024-09-11T03:53:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:40.120035+0000", 2024-09-11T03:53:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000116461 2024-09-11T03:53:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:4937 2.7 2:e23d1762:::smithi17557870-21:head [read 0~3815288,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e526)", 2024-09-11T03:53:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:45:43.953926+0000", 2024-09-11T03:53:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 445.67925480100001, 2024-09-11T03:53:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010177080999999999, 2024-09-11T03:53:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 4937 2024-09-11T03:53:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953926+0000", 2024-09-11T03:53:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953926+0000", 2024-09-11T03:53:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953927+0000", 2024-09-11T03:53:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1699999999999997e-07 2024-09-11T03:53:09.677 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.677 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.677 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.677 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953933+0000", 2024-09-11T03:53:09.677 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5050000000000004e-06 2024-09-11T03:53:09.677 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.677 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.677 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.677 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953945+0000", 2024-09-11T03:53:09.677 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1202e-05 2024-09-11T03:53:09.677 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.677 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.677 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.678 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953951+0000", 2024-09-11T03:53:09.678 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.708e-06 2024-09-11T03:53:09.678 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.678 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.678 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.678 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953988+0000", 2024-09-11T03:53:09.678 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6356999999999998e-05 2024-09-11T03:53:09.678 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.678 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.678 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:53:09.678 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.953992+0000", 2024-09-11T03:53:09.678 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5469999999999998e-06 2024-09-11T03:53:09.678 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.679 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.679 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.679 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.962348+0000", 2024-09-11T03:53:09.679 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0083553180000000005 2024-09-11T03:53:09.679 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.679 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.679 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.679 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.962519+0000", 2024-09-11T03:53:09.679 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000171817 2024-09-11T03:53:09.679 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.679 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.679 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.679 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:45:43.964103+0000", 2024-09-11T03:53:09.680 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00158391 2024-09-11T03:53:09.680 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.680 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.680 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.680 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.680 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.680 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6557 2.1c 2:3c007163:::smithi17557870-35:head [copy-from ver 891] snapc 120=[120,11f,11e,11d,11a,119,114] ondisk+write+known_if_redirected+supports_pool_eio e670)", 2024-09-11T03:53:09.680 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:08.152129+0000", 2024-09-11T03:53:09.680 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 301.48105233699999, 2024-09-11T03:53:09.680 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.024595260000000001, 2024-09-11T03:53:09.680 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.680 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.680 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:09.681 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:09.681 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:09.681 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:09.681 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6557 2024-09-11T03:53:09.681 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.681 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.681 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.681 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.681 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152129+0000", 2024-09-11T03:53:09.681 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.681 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.681 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.681 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.682 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152129+0000", 2024-09-11T03:53:09.682 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.682 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.682 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.682 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.682 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152132+0000", 2024-09-11T03:53:09.682 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8650000000000001e-06 2024-09-11T03:53:09.682 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.682 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.682 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.682 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152146+0000", 2024-09-11T03:53:09.682 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4438e-05 2024-09-11T03:53:09.683 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.683 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.683 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.683 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152177+0000", 2024-09-11T03:53:09.683 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.1149999999999998e-05 2024-09-11T03:53:09.683 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.683 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.683 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.683 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152191+0000", 2024-09-11T03:53:09.683 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4022000000000001e-05 2024-09-11T03:53:09.683 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.683 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.683 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.684 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152324+0000", 2024-09-11T03:53:09.684 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013306799999999999 2024-09-11T03:53:09.684 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.684 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.684 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:53:09.684 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152332+0000", 2024-09-11T03:53:09.684 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0490000000000004e-06 2024-09-11T03:53:09.684 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.684 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.684 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.684 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.154966+0000", 2024-09-11T03:53:09.684 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0026336900000000002 2024-09-11T03:53:09.684 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.685 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.685 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.685 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.155098+0000", 2024-09-11T03:53:09.685 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013232700000000001 2024-09-11T03:53:09.685 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.685 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.685 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,8", 2024-09-11T03:53:09.685 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.166673+0000", 2024-09-11T03:53:09.686 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011575062000000001 2024-09-11T03:53:09.686 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.686 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.686 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:09.686 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.172194+0000", 2024-09-11T03:53:09.686 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.005520619 2024-09-11T03:53:09.686 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.686 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.686 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:09.687 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.175894+0000", 2024-09-11T03:53:09.687 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003700451 2024-09-11T03:53:09.687 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.687 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.687 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:09.687 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.176612+0000", 2024-09-11T03:53:09.687 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00071733300000000001 2024-09-11T03:53:09.687 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.688 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.688 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:09.688 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.176669+0000", 2024-09-11T03:53:09.688 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6949e-05 2024-09-11T03:53:09.688 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.688 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.688 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.688 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.176724+0000", 2024-09-11T03:53:09.689 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5237e-05 2024-09-11T03:53:09.689 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.689 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.689 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.689 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.689 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.689 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6558 2.1c 2:3c007163:::smithi17557870-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e670)", 2024-09-11T03:53:09.689 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:08.152229+0000", 2024-09-11T03:53:09.690 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 301.48095158199999, 2024-09-11T03:53:09.690 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019338949000000001, 2024-09-11T03:53:09.690 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.690 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.690 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:09.690 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:09.690 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:09.690 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:09.691 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6558 2024-09-11T03:53:09.691 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.691 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.691 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.691 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.691 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152229+0000", 2024-09-11T03:53:09.691 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.691 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.691 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.692 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.692 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152229+0000", 2024-09-11T03:53:09.692 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.692 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.692 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.692 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.692 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152230+0000", 2024-09-11T03:53:09.692 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4600000000000004e-07 2024-09-11T03:53:09.693 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.693 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.693 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.693 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152234+0000", 2024-09-11T03:53:09.693 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5719999999999999e-06 2024-09-11T03:53:09.693 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.693 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.693 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.694 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152247+0000", 2024-09-11T03:53:09.694 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2852e-05 2024-09-11T03:53:09.694 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.694 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.694 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.694 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152254+0000", 2024-09-11T03:53:09.694 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2559999999999996e-06 2024-09-11T03:53:09.694 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.695 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.695 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.695 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152405+0000", 2024-09-11T03:53:09.695 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015162500000000001 2024-09-11T03:53:09.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.695 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.695 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:53:09.695 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.152411+0000", 2024-09-11T03:53:09.696 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1880000000000001e-06 2024-09-11T03:53:09.696 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.696 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.696 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.696 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.155324+0000", 2024-09-11T03:53:09.696 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0029129749999999999 2024-09-11T03:53:09.696 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.696 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.697 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:53:09.697 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.155374+0000", 2024-09-11T03:53:09.697 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0581999999999998e-05 2024-09-11T03:53:09.697 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.697 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.697 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.697 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.171261+0000", 2024-09-11T03:53:09.697 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.01588674 2024-09-11T03:53:09.698 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.698 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.698 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.698 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.171402+0000", 2024-09-11T03:53:09.698 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000140684 2024-09-11T03:53:09.698 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.698 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.698 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.698 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:08.171568+0000", 2024-09-11T03:53:09.699 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016682900000000001 2024-09-11T03:53:09.699 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.699 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.699 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.699 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.699 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.699 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7043 2.12 2:494ba932:::smithi17557870-43:head [copy-from ver 979] snapc 13a=[13a,139,138,137,136,135,133,130,12e,12d,12c,12a,127,126,123,120,11f,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e709)", 2024-09-11T03:53:09.700 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:47.006408+0000", 2024-09-11T03:53:09.700 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 262.62677291199998, 2024-09-11T03:53:09.700 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.033513329000000001, 2024-09-11T03:53:09.700 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.700 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.700 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:09.700 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:09.700 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:09.701 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:09.701 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7043 2024-09-11T03:53:09.701 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.701 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.701 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.701 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.701 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006408+0000", 2024-09-11T03:53:09.701 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.701 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.702 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.702 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.702 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006408+0000", 2024-09-11T03:53:09.702 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.702 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.702 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.702 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.702 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006411+0000", 2024-09-11T03:53:09.702 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6869999999999999e-06 2024-09-11T03:53:09.702 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.702 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.702 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.702 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006423+0000", 2024-09-11T03:53:09.703 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1762e-05 2024-09-11T03:53:09.703 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.703 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.703 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.703 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006462+0000", 2024-09-11T03:53:09.703 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.9755999999999997e-05 2024-09-11T03:53:09.703 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.703 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.703 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.703 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006479+0000", 2024-09-11T03:53:09.703 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.7031e-05 2024-09-11T03:53:09.703 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.704 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.704 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.704 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006583+0000", 2024-09-11T03:53:09.704 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010370999999999999 2024-09-11T03:53:09.704 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.704 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.704 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:53:09.704 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006592+0000", 2024-09-11T03:53:09.704 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2199999999999998e-06 2024-09-11T03:53:09.704 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.704 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.704 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.705 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.009741+0000", 2024-09-11T03:53:09.705 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0031483639999999999 2024-09-11T03:53:09.705 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.705 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.705 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.705 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.009831+0000", 2024-09-11T03:53:09.705 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.0351000000000003e-05 2024-09-11T03:53:09.705 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.705 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.705 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,7", 2024-09-11T03:53:09.705 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.030947+0000", 2024-09-11T03:53:09.705 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.02111586 2024-09-11T03:53:09.705 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.706 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.706 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:09.706 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.034462+0000", 2024-09-11T03:53:09.706 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035156279999999998 2024-09-11T03:53:09.706 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.706 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.706 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:09.706 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.039094+0000", 2024-09-11T03:53:09.706 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0046314779999999996 2024-09-11T03:53:09.706 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.706 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.706 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:09.706 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.039783+0000", 2024-09-11T03:53:09.707 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00068866399999999996 2024-09-11T03:53:09.707 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.707 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.707 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:09.707 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.039883+0000", 2024-09-11T03:53:09.707 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000100641 2024-09-11T03:53:09.707 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.707 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.707 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.707 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.039921+0000", 2024-09-11T03:53:09.707 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8176999999999997e-05 2024-09-11T03:53:09.707 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.707 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.707 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.708 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.708 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.708 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7044 2.12 2:494ba932:::smithi17557870-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e709)", 2024-09-11T03:53:09.708 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:47.006515+0000", 2024-09-11T03:53:09.708 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 262.626665866, 2024-09-11T03:53:09.708 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.027479966000000002, 2024-09-11T03:53:09.708 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.708 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.708 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:09.708 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:09.709 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:09.709 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:09.709 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7044 2024-09-11T03:53:09.709 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.709 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.709 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.709 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.709 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006515+0000", 2024-09-11T03:53:09.709 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.709 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.710 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.710 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.710 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006515+0000", 2024-09-11T03:53:09.710 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.710 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.710 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.710 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.710 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006516+0000", 2024-09-11T03:53:09.710 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1999999999999999e-07 2024-09-11T03:53:09.710 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.710 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.710 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.710 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006520+0000", 2024-09-11T03:53:09.711 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5210000000000004e-06 2024-09-11T03:53:09.711 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.711 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.711 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.711 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006545+0000", 2024-09-11T03:53:09.711 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4623000000000001e-05 2024-09-11T03:53:09.711 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.711 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.711 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.711 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006556+0000", 2024-09-11T03:53:09.711 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0844e-05 2024-09-11T03:53:09.711 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.711 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.712 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.712 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006713+0000", 2024-09-11T03:53:09.712 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015744899999999999 2024-09-11T03:53:09.712 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.712 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.712 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:53:09.712 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006718+0000", 2024-09-11T03:53:09.712 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9119999999999997e-06 2024-09-11T03:53:09.712 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.712 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.712 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.712 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.010037+0000", 2024-09-11T03:53:09.713 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003318283 2024-09-11T03:53:09.713 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.713 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.713 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:53:09.713 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.010071+0000", 2024-09-11T03:53:09.713 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4344999999999997e-05 2024-09-11T03:53:09.713 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.713 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.713 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.713 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.033776+0000", 2024-09-11T03:53:09.713 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.023705086 2024-09-11T03:53:09.713 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.713 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.713 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.714 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.033886+0000", 2024-09-11T03:53:09.714 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000110491 2024-09-11T03:53:09.714 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.714 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.714 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.714 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.033995+0000", 2024-09-11T03:53:09.714 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108692 2024-09-11T03:53:09.714 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.714 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.714 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.714 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.714 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.714 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7049 2.1c 2:3c007163:::smithi17557870-35:head [copy-from ver 947] snapc 13a=[13a,139,138,137,136,135,133,130,12e,12d,12c,12a,127,126,123,120,11f,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e709)", 2024-09-11T03:53:09.715 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:47.006590+0000", 2024-09-11T03:53:09.715 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 262.62659147400001, 2024-09-11T03:53:09.715 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021051984999999999, 2024-09-11T03:53:09.715 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.715 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.715 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:09.715 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:09.715 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:09.715 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:09.716 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7049 2024-09-11T03:53:09.716 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.716 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.716 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.716 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.716 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006590+0000", 2024-09-11T03:53:09.716 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.716 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.716 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.717 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.717 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006590+0000", 2024-09-11T03:53:09.717 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.717 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.717 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.717 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.718 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006590+0000", 2024-09-11T03:53:09.718 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.7100000000000001e-07 2024-09-11T03:53:09.718 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.718 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.718 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.718 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006596+0000", 2024-09-11T03:53:09.718 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.2870000000000004e-06 2024-09-11T03:53:09.719 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.719 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.719 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.719 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006610+0000", 2024-09-11T03:53:09.719 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4781e-05 2024-09-11T03:53:09.719 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.719 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.720 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.720 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006617+0000", 2024-09-11T03:53:09.720 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0099999999999998e-06 2024-09-11T03:53:09.720 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.720 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.720 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.720 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006678+0000", 2024-09-11T03:53:09.721 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.0269999999999997e-05 2024-09-11T03:53:09.721 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.721 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.721 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:53:09.721 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006685+0000", 2024-09-11T03:53:09.721 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.311e-06 2024-09-11T03:53:09.721 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.722 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.722 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.722 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.008762+0000", 2024-09-11T03:53:09.722 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0020774719999999999 2024-09-11T03:53:09.722 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.722 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.723 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.723 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.008877+0000", 2024-09-11T03:53:09.723 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000114695 2024-09-11T03:53:09.723 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.723 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.723 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,8", 2024-09-11T03:53:09.724 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.018702+0000", 2024-09-11T03:53:09.724 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0098247839999999996 2024-09-11T03:53:09.724 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.724 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.724 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:09.724 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.023780+0000", 2024-09-11T03:53:09.725 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0050777699999999997 2024-09-11T03:53:09.725 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.725 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.725 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:09.725 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.024140+0000", 2024-09-11T03:53:09.725 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000360219 2024-09-11T03:53:09.725 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.725 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.725 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:09.725 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.027423+0000", 2024-09-11T03:53:09.725 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032829309999999998 2024-09-11T03:53:09.726 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.726 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.726 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:09.726 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.027580+0000", 2024-09-11T03:53:09.726 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015743199999999999 2024-09-11T03:53:09.726 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.726 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.726 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.726 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.027642+0000", 2024-09-11T03:53:09.726 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1252000000000002e-05 2024-09-11T03:53:09.727 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.727 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.727 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.727 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.727 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.727 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7050 2.1c 2:3c007163:::smithi17557870-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e709)", 2024-09-11T03:53:09.727 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:47.006639+0000", 2024-09-11T03:53:09.727 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 262.62654223999999, 2024-09-11T03:53:09.727 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016888205999999999, 2024-09-11T03:53:09.727 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.728 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.728 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:09.728 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:09.728 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:09.728 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:09.728 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7050 2024-09-11T03:53:09.728 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.728 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.728 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.728 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.728 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006639+0000", 2024-09-11T03:53:09.728 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.728 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.729 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.729 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.729 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006639+0000", 2024-09-11T03:53:09.729 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.729 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.729 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.729 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.729 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006639+0000", 2024-09-11T03:53:09.729 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9299999999999998e-07 2024-09-11T03:53:09.729 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.729 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.730 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.730 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006643+0000", 2024-09-11T03:53:09.730 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3129999999999999e-06 2024-09-11T03:53:09.730 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.730 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.730 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.730 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006650+0000", 2024-09-11T03:53:09.730 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8040000000000002e-06 2024-09-11T03:53:09.730 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.730 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.731 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.731 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006656+0000", 2024-09-11T03:53:09.731 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9229999999999999e-06 2024-09-11T03:53:09.731 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.731 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.731 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.731 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006711+0000", 2024-09-11T03:53:09.732 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4710999999999998e-05 2024-09-11T03:53:09.732 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.732 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.732 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:53:09.733 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.006717+0000", 2024-09-11T03:53:09.733 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4e-06 2024-09-11T03:53:09.733 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.733 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.733 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.733 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.009041+0000", 2024-09-11T03:53:09.733 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002324497 2024-09-11T03:53:09.733 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.733 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.733 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:53:09.733 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.009084+0000", 2024-09-11T03:53:09.733 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2896999999999999e-05 2024-09-11T03:53:09.733 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.734 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.734 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.734 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.023331+0000", 2024-09-11T03:53:09.734 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014247344 2024-09-11T03:53:09.734 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.734 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.734 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.734 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.023430+0000", 2024-09-11T03:53:09.734 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.8884999999999999e-05 2024-09-11T03:53:09.734 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.734 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.734 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.734 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.023527+0000", 2024-09-11T03:53:09.735 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6838999999999999e-05 2024-09-11T03:53:09.735 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.735 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.735 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.735 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.735 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.735 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:7045 2.4 e709/432 2:25f5f8b9:::smithi17557870-26:head v 709'563, mlcod=706'559)", 2024-09-11T03:53:09.735 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:47.025349+0000", 2024-09-11T03:53:09.735 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 262.60783172200001, 2024-09-11T03:53:09.735 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011744064, 2024-09-11T03:53:09.735 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.735 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.735 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:09.736 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.736 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.736 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.736 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.025349+0000", 2024-09-11T03:53:09.736 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.736 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.736 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.736 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.736 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.025349+0000", 2024-09-11T03:53:09.736 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.736 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.736 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.736 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.737 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.025351+0000", 2024-09-11T03:53:09.737 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4470000000000001e-06 2024-09-11T03:53:09.737 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.737 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.737 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.737 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.033663+0000", 2024-09-11T03:53:09.737 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0083124940000000001 2024-09-11T03:53:09.737 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.737 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.737 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.737 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.033679+0000", 2024-09-11T03:53:09.737 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5690999999999999e-05 2024-09-11T03:53:09.738 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.738 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.738 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.738 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.033690+0000", 2024-09-11T03:53:09.738 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1372000000000001e-05 2024-09-11T03:53:09.738 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.738 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.738 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.738 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.033798+0000", 2024-09-11T03:53:09.738 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010728799999999999 2024-09-11T03:53:09.738 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.738 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.738 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.739 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.033824+0000", 2024-09-11T03:53:09.739 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6092999999999999e-05 2024-09-11T03:53:09.739 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.739 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.739 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:09.739 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.037044+0000", 2024-09-11T03:53:09.739 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0032200919999999999 2024-09-11T03:53:09.739 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.739 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.739 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.739 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:47.037093+0000", 2024-09-11T03:53:09.739 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9586999999999998e-05 2024-09-11T03:53:09.739 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.740 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.740 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.740 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.740 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.740 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:7546 2.7 e759/745 2:e23d1762:::smithi17557870-21:head v 759'1072, mlcod=759'1069)", 2024-09-11T03:53:09.740 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:49:37.432009+0000", 2024-09-11T03:53:09.740 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 212.20117216200001, 2024-09-11T03:53:09.740 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014009915, 2024-09-11T03:53:09.740 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.740 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.740 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:09.740 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.740 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.741 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.741 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.432009+0000", 2024-09-11T03:53:09.741 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.741 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.741 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.741 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.741 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.432009+0000", 2024-09-11T03:53:09.741 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.741 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.741 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.741 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.741 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.432010+0000", 2024-09-11T03:53:09.742 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2839999999999999e-06 2024-09-11T03:53:09.742 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.742 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.742 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.742 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.443970+0000", 2024-09-11T03:53:09.742 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011960133 2024-09-11T03:53:09.742 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.742 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.742 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.742 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.443993+0000", 2024-09-11T03:53:09.742 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3037999999999999e-05 2024-09-11T03:53:09.742 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.742 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.743 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.743 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.444003+0000", 2024-09-11T03:53:09.743 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.3430000000000002e-06 2024-09-11T03:53:09.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.743 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.743 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.743 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.444100+0000", 2024-09-11T03:53:09.743 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.7343999999999995e-05 2024-09-11T03:53:09.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.743 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.743 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.744 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.444120+0000", 2024-09-11T03:53:09.744 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0199000000000001e-05 2024-09-11T03:53:09.744 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.744 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.744 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:09.744 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.445982+0000", 2024-09-11T03:53:09.744 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0018620850000000001 2024-09-11T03:53:09.744 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.744 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.744 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.744 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:37.446019+0000", 2024-09-11T03:53:09.744 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6489000000000002e-05 2024-09-11T03:53:09.744 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.745 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.745 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.745 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.745 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.745 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:7879 2.7 e779/745 2:e0ec5763:::smithi17557870-2:head v 779'1108, mlcod=778'1104)", 2024-09-11T03:53:09.745 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:49:57.174490+0000", 2024-09-11T03:53:09.745 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 192.45869103800001, 2024-09-11T03:53:09.745 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013932368000000001, 2024-09-11T03:53:09.745 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.745 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.746 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:09.746 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.746 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.746 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.746 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.174490+0000", 2024-09-11T03:53:09.746 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.746 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.746 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.746 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.746 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.174490+0000", 2024-09-11T03:53:09.746 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.746 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.747 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.747 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.747 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.174491+0000", 2024-09-11T03:53:09.747 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.181e-06 2024-09-11T03:53:09.747 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.747 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.747 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.747 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.185408+0000", 2024-09-11T03:53:09.747 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010916795999999999 2024-09-11T03:53:09.747 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.747 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.747 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.747 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.185436+0000", 2024-09-11T03:53:09.748 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8425999999999999e-05 2024-09-11T03:53:09.748 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.748 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.748 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.748 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.185455+0000", 2024-09-11T03:53:09.748 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8712e-05 2024-09-11T03:53:09.748 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.748 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.748 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.748 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.185533+0000", 2024-09-11T03:53:09.748 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8252999999999998e-05 2024-09-11T03:53:09.749 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.749 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.749 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.749 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.185562+0000", 2024-09-11T03:53:09.749 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9043999999999998e-05 2024-09-11T03:53:09.749 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.749 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.749 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:09.749 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.188386+0000", 2024-09-11T03:53:09.749 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0028236089999999999 2024-09-11T03:53:09.749 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.749 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.749 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.750 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:49:57.188422+0000", 2024-09-11T03:53:09.750 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6347000000000003e-05 2024-09-11T03:53:09.750 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.750 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.750 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.750 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.750 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.750 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:7972 2.1e e785/574 2:7b0c3397:::smithi17557870-14:head v 785'1043, mlcod=785'1041)", 2024-09-11T03:53:09.750 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:50:03.102595+0000", 2024-09-11T03:53:09.750 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 186.53058620499999, 2024-09-11T03:53:09.750 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.036476402999999998, 2024-09-11T03:53:09.750 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.750 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.751 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:09.751 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.751 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.751 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.751 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:03.102595+0000", 2024-09-11T03:53:09.751 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.751 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.751 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.751 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.751 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:03.102595+0000", 2024-09-11T03:53:09.751 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.751 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.751 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.752 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.752 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:03.102596+0000", 2024-09-11T03:53:09.752 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.623e-06 2024-09-11T03:53:09.752 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.752 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.752 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.752 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:03.138179+0000", 2024-09-11T03:53:09.752 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.03558265 2024-09-11T03:53:09.752 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.752 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.752 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.752 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:03.138205+0000", 2024-09-11T03:53:09.752 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.5863e-05 2024-09-11T03:53:09.753 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.753 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.753 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.753 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:03.138217+0000", 2024-09-11T03:53:09.753 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2313999999999999e-05 2024-09-11T03:53:09.753 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.753 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.753 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.753 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:03.138304+0000", 2024-09-11T03:53:09.753 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.6490999999999996e-05 2024-09-11T03:53:09.753 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.753 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.754 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.754 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:03.138328+0000", 2024-09-11T03:53:09.754 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4102999999999999e-05 2024-09-11T03:53:09.754 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.754 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.754 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:09.754 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:03.139040+0000", 2024-09-11T03:53:09.754 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00071236400000000005 2024-09-11T03:53:09.755 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.755 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.755 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.755 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:50:03.139071+0000", 2024-09-11T03:53:09.755 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.0994999999999997e-05 2024-09-11T03:53:09.755 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.755 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.755 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.756 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.756 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.756 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(client.24454.0:9282 2.7 e904/849 2:e0eabcfb:::smithi17557870-9:head v 904'1328, mlcod=898'1324)", 2024-09-11T03:53:09.756 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:52:02.090683+0000", 2024-09-11T03:53:09.756 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 67.542498187999996, 2024-09-11T03:53:09.756 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011477246999999999, 2024-09-11T03:53:09.756 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.756 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.756 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:09.756 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.757 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.757 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.757 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:02.090683+0000", 2024-09-11T03:53:09.757 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.757 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.757 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.757 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.757 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:02.090683+0000", 2024-09-11T03:53:09.757 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.757 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.757 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.757 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.758 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:02.090683+0000", 2024-09-11T03:53:09.758 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3900000000000001e-07 2024-09-11T03:53:09.758 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.758 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.758 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.758 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:02.099570+0000", 2024-09-11T03:53:09.758 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0088868160000000005 2024-09-11T03:53:09.758 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.758 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.758 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.758 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:02.099584+0000", 2024-09-11T03:53:09.758 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4521999999999999e-05 2024-09-11T03:53:09.758 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.759 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.759 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.759 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:02.099592+0000", 2024-09-11T03:53:09.759 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0099999999999995e-06 2024-09-11T03:53:09.759 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.759 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.759 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.759 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:02.099644+0000", 2024-09-11T03:53:09.759 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1462999999999998e-05 2024-09-11T03:53:09.759 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.759 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.759 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.759 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:02.099662+0000", 2024-09-11T03:53:09.759 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8584e-05 2024-09-11T03:53:09.760 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.760 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.760 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:09.760 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:02.102126+0000", 2024-09-11T03:53:09.760 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002463079 2024-09-11T03:53:09.760 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.760 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.760 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.760 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:02.102160+0000", 2024-09-11T03:53:09.760 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4533999999999998e-05 2024-09-11T03:53:09.760 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.760 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.761 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.761 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.761 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.761 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_repop(osd.7.0:749 2.3c e936/814 2:3c007163:::smithi17557870-35:1a8 v 936'800, mlcod=921'796)", 2024-09-11T03:53:09.761 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:52:34.084658+0000", 2024-09-11T03:53:09.761 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 35.548522677999998, 2024-09-11T03:53:09.761 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.011874300000000001, 2024-09-11T03:53:09.761 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.761 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.761 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:09.761 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.762 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.762 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.762 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:34.084658+0000", 2024-09-11T03:53:09.762 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.762 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.762 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.762 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.762 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:34.084658+0000", 2024-09-11T03:53:09.762 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.762 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.763 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.763 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.763 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:34.084659+0000", 2024-09-11T03:53:09.763 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0440000000000001e-06 2024-09-11T03:53:09.763 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.763 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.763 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.763 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:34.084674+0000", 2024-09-11T03:53:09.763 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4715999999999999e-05 2024-09-11T03:53:09.763 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.763 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.763 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.763 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:34.084684+0000", 2024-09-11T03:53:09.763 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0339e-05 2024-09-11T03:53:09.764 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.764 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.764 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.764 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:34.084693+0000", 2024-09-11T03:53:09.764 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8359999999999998e-06 2024-09-11T03:53:09.764 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.764 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.764 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.764 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:34.084830+0000", 2024-09-11T03:53:09.764 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013653299999999999 2024-09-11T03:53:09.764 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.764 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.764 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.765 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:34.087316+0000", 2024-09-11T03:53:09.765 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024863870000000001 2024-09-11T03:53:09.765 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.765 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.765 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:09.765 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:34.096490+0000", 2024-09-11T03:53:09.765 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0091737899999999994 2024-09-11T03:53:09.765 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.765 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.765 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.765 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:34.096533+0000", 2024-09-11T03:53:09.765 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2654999999999997e-05 2024-09-11T03:53:09.766 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.766 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.766 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.766 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.766 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.766 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:9725 2.4c 2:327469ff:::smithi17557870-49:head [copy-from ver 1090] snapc 1b1=[1b1,1b0,1af,1ac,1aa,1a8,1a7,1a6,1a5,1a4,1a3,1a2,1a1,19d,19c,199,197,194,191,18b,183,17c,17a,178,175,172,16e,164,15f,13c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e940)", 2024-09-11T03:53:09.766 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:52:38.096642+0000", 2024-09-11T03:53:09.766 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 31.536539081000001, 2024-09-11T03:53:09.766 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020052989, 2024-09-11T03:53:09.767 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.767 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.767 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:09.767 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:09.767 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:09.767 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:09.767 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 9725 2024-09-11T03:53:09.767 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.767 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.767 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.767 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.767 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.096642+0000", 2024-09-11T03:53:09.767 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.767 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.768 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.768 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.768 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.096642+0000", 2024-09-11T03:53:09.768 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.768 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.768 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.768 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.768 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.096643+0000", 2024-09-11T03:53:09.768 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.531e-06 2024-09-11T03:53:09.768 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.768 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.768 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.769 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.096648+0000", 2024-09-11T03:53:09.769 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4499999999999997e-06 2024-09-11T03:53:09.769 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.769 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.769 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.769 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.096659+0000", 2024-09-11T03:53:09.769 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1275e-05 2024-09-11T03:53:09.770 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.770 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.770 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.770 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.096668+0000", 2024-09-11T03:53:09.770 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.0599999999999997e-06 2024-09-11T03:53:09.770 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.770 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.770 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.771 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.096996+0000", 2024-09-11T03:53:09.771 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00032751900000000002 2024-09-11T03:53:09.771 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.771 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.771 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:09.771 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.097006+0000", 2024-09-11T03:53:09.771 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0283e-05 2024-09-11T03:53:09.771 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.771 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.771 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.771 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.098644+0000", 2024-09-11T03:53:09.771 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001637938 2024-09-11T03:53:09.771 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.772 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.772 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.772 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.098785+0000", 2024-09-11T03:53:09.772 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00014059899999999999 2024-09-11T03:53:09.772 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.772 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.772 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 5", 2024-09-11T03:53:09.772 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.108861+0000", 2024-09-11T03:53:09.772 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010076134 2024-09-11T03:53:09.772 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.772 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.772 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:09.772 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.114336+0000", 2024-09-11T03:53:09.773 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0054756429999999997 2024-09-11T03:53:09.773 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.773 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.773 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:09.773 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.116564+0000", 2024-09-11T03:53:09.773 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022280870000000001 2024-09-11T03:53:09.773 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.773 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.773 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:09.773 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.116610+0000", 2024-09-11T03:53:09.773 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5277e-05 2024-09-11T03:53:09.773 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.774 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.774 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.774 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.116695+0000", 2024-09-11T03:53:09.774 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5192999999999996e-05 2024-09-11T03:53:09.774 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.774 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.774 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.774 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.774 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.774 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:9726 2.4c 2:327469ff:::smithi17557870-49:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e940)", 2024-09-11T03:53:09.774 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:52:38.096680+0000", 2024-09-11T03:53:09.774 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 31.536501429000001, 2024-09-11T03:53:09.774 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017059666000000001, 2024-09-11T03:53:09.774 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:09.775 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:09.775 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:09.775 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:09.775 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:09.775 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:09.775 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 9726 2024-09-11T03:53:09.775 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.775 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:09.775 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.775 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:09.775 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.096680+0000", 2024-09-11T03:53:09.775 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.775 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.776 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.776 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:09.776 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.096680+0000", 2024-09-11T03:53:09.776 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:09.776 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.776 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.776 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:09.776 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.096680+0000", 2024-09-11T03:53:09.776 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3399999999999999e-07 2024-09-11T03:53:09.776 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.777 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.777 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:09.777 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.096683+0000", 2024-09-11T03:53:09.777 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.418e-06 2024-09-11T03:53:09.777 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.777 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.777 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:09.777 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.096690+0000", 2024-09-11T03:53:09.777 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5460000000000003e-06 2024-09-11T03:53:09.777 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.777 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.778 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:09.778 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.096695+0000", 2024-09-11T03:53:09.778 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3589999999999999e-06 2024-09-11T03:53:09.778 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.778 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.778 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.778 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.097029+0000", 2024-09-11T03:53:09.778 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00033340000000000003 2024-09-11T03:53:09.778 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.778 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.778 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:09.778 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.097036+0000", 2024-09-11T03:53:09.778 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1919999999999999e-06 2024-09-11T03:53:09.779 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.779 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.779 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.779 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.098953+0000", 2024-09-11T03:53:09.779 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001917167 2024-09-11T03:53:09.779 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.779 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.779 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:53:09.779 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.099020+0000", 2024-09-11T03:53:09.779 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6657999999999998e-05 2024-09-11T03:53:09.779 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.779 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.780 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:09.780 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.113407+0000", 2024-09-11T03:53:09.780 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014387067999999999 2024-09-11T03:53:09.780 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.780 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.780 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:09.780 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.113538+0000", 2024-09-11T03:53:09.780 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00013110899999999999 2024-09-11T03:53:09.780 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:09.780 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:09.780 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:09.780 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:38.113739+0000", 2024-09-11T03:53:09.780 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000201215 2024-09-11T03:53:09.781 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.781 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.781 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.781 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:09.781 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:09.781 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:53:09.795 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:10.194 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:10 smithi084 ceph-mon[27254]: pgmap v1602: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 1.6 GiB used, 982 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:10.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:10 smithi175 ceph-mon[30894]: pgmap v1602: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 1.6 GiB used, 982 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:10.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:10 smithi121 ceph-mon[30729]: pgmap v1602: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 1.6 GiB used, 982 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:10.668 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-11T03:53:10.879 INFO:tasks.thrashosds.thrasher:reviving osd 2024-09-11T03:53:10.879 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-09-11T03:53:10.879 INFO:tasks.cephadm.osd.0:Restarting osd.0 (starting--it wasn't running)... 2024-09-11T03:53:10.880 DEBUG:teuthology.orchestra.run.smithi084:osd.0> sudo journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.0.service 2024-09-11T03:53:10.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl start ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.0 2024-09-11T03:53:10.912 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:53:11.102 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-11T03:53:11.185 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:10 smithi084 systemd[1]: Starting Ceph osd.0 for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:53:11.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:11 smithi084 ceph-mon[27254]: 1.13 scrub starts 2024-09-11T03:53:11.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:11 smithi084 ceph-mon[27254]: 1.13 scrub ok 2024-09-11T03:53:11.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:11 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3764480069' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:53:11.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:11 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:53:11.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:11 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:53:11.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:11 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:53:11.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:11 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:53:11.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:11 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:53:11.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:11 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:53:11.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:11 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:53:11.344 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:11 smithi175 ceph-mon[30894]: 1.13 scrub starts 2024-09-11T03:53:11.344 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:11 smithi175 ceph-mon[30894]: 1.13 scrub ok 2024-09-11T03:53:11.344 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:11 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3764480069' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:53:11.345 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:11 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:53:11.345 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:11 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:53:11.345 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:11 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:53:11.345 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:11 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:53:11.345 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:11 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:53:11.345 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:11 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:53:11.345 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:11 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:53:11.359 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:53:11.359 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:53:11.359 INFO:teuthology.orchestra.run.smithi175.stdout: "num_ops": 0 2024-09-11T03:53:11.359 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:53:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:11 smithi121 ceph-mon[30729]: 1.13 scrub starts 2024-09-11T03:53:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:11 smithi121 ceph-mon[30729]: 1.13 scrub ok 2024-09-11T03:53:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:11 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3764480069' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-11T03:53:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:11 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-11T03:53:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:11 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-11T03:53:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:11 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-11T03:53:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:11 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:53:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:11 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:53:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:11 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:53:11.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:11 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:53:11.644 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:11 smithi084 podman[184971]: 2024-09-11 03:53:11.088485746 +0000 UTC m=+0.017201919 image pull 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e 2024-09-11T03:53:11.644 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:11 smithi084 podman[184971]: 2024-09-11 03:53:11.230254966 +0000 UTC m=+0.158971134 container create e0387d8af63feca9a5b8b73e160690776fb104ca3091c404e60e08f939ddfdc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-activate, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-11T03:53:12.117 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-11T03:53:12.120 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-11T03:53:12.362 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:53:12.363 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:12 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3764480069' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:53:12.363 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:12 smithi084 ceph-mon[27254]: osdmap e962: 12 total, 11 up, 11 in 2024-09-11T03:53:12.363 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:12 smithi084 podman[184971]: 2024-09-11 03:53:12.264281022 +0000 UTC m=+1.192997202 container init e0387d8af63feca9a5b8b73e160690776fb104ca3091c404e60e08f939ddfdc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-activate, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-11T03:53:12.363 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:12 smithi084 podman[184971]: 2024-09-11 03:53:12.26785706 +0000 UTC m=+1.196573229 container start e0387d8af63feca9a5b8b73e160690776fb104ca3091c404e60e08f939ddfdc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-11T03:53:12.363 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:12 smithi084 podman[184971]: 2024-09-11 03:53:12.363845275 +0000 UTC m=+1.292561444 container attach e0387d8af63feca9a5b8b73e160690776fb104ca3091c404e60e08f939ddfdc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-activate, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, maintainer=Guillaume Abrioux ) 2024-09-11T03:53:12.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:12 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3764480069' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:53:12.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:12 smithi121 ceph-mon[30729]: osdmap e962: 12 total, 11 up, 11 in 2024-09-11T03:53:12.440 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:12 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3764480069' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-11T03:53:12.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:12 smithi175 ceph-mon[30894]: osdmap e962: 12 total, 11 up, 11 in 2024-09-11T03:53:12.644 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:12 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-activate[185032]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-11T03:53:12.644 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:12 smithi084 bash[184971]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-11T03:53:12.644 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:12 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-activate[185032]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-11T03:53:12.644 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:12 smithi084 bash[184971]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-11T03:53:12.849 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:53:12.849 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [], 2024-09-11T03:53:12.849 INFO:teuthology.orchestra.run.smithi175.stdout: "complaint_time": 30, 2024-09-11T03:53:12.849 INFO:teuthology.orchestra.run.smithi175.stdout: "num_blocked_ops": 0 2024-09-11T03:53:12.849 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:53:13.108 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:12 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-activate[185032]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-11T03:53:13.108 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:12 smithi084 bash[184971]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-11T03:53:13.108 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:12 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-activate[185032]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-11T03:53:13.108 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:12 smithi084 bash[184971]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-11T03:53:13.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:13 smithi084 ceph-mon[27254]: 1.6 scrub starts 2024-09-11T03:53:13.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:13 smithi084 ceph-mon[27254]: 1.6 scrub ok 2024-09-11T03:53:13.394 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:13 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-activate[185032]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-11T03:53:13.394 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:13 smithi084 bash[184971]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-11T03:53:13.394 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:13 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-activate[185032]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-11T03:53:13.395 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:13 smithi084 bash[184971]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-11T03:53:13.395 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:13 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-activate[185032]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-11T03:53:13.395 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:13 smithi084 bash[184971]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-11T03:53:13.395 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:13 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-activate[185032]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-11T03:53:13.395 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:13 smithi084 bash[184971]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-11T03:53:13.395 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:13 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-activate[185032]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-11T03:53:13.395 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:13 smithi084 bash[184971]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-11T03:53:13.395 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:13 smithi084 podman[184971]: 2024-09-11 03:53:13.186061456 +0000 UTC m=+2.114777635 container died e0387d8af63feca9a5b8b73e160690776fb104ca3091c404e60e08f939ddfdc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-activate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-11T03:53:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:13 smithi121 ceph-mon[30729]: 1.6 scrub starts 2024-09-11T03:53:13.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:13 smithi121 ceph-mon[30729]: 1.6 scrub ok 2024-09-11T03:53:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:13 smithi175 ceph-mon[30894]: 1.6 scrub starts 2024-09-11T03:53:13.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:13 smithi175 ceph-mon[30894]: 1.6 scrub ok 2024-09-11T03:53:13.596 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-11T03:53:13.838 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.c/config 2024-09-11T03:53:14.113 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-11T03:53:14.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:14 smithi084 ceph-mon[27254]: pgmap v1605: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 845 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:14.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:14 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3847388328' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:53:14.145 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:13 smithi084 podman[185254]: 2024-09-11 03:53:13.887658822 +0000 UTC m=+0.691499177 container remove e0387d8af63feca9a5b8b73e160690776fb104ca3091c404e60e08f939ddfdc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T03:53:14.145 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:14 smithi084 podman[185314]: 2024-09-11 03:53:14.09661508 +0000 UTC m=+0.110974115 container create d2adc8d7aac824c1020796293b9e4c92bd0e40be4ecb39ecb1f4608c819c5849 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4) 2024-09-11T03:53:14.145 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:14 smithi084 podman[185314]: 2024-09-11 03:53:14.001847282 +0000 UTC m=+0.016206332 image pull 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e 2024-09-11T03:53:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:14 smithi175 ceph-mon[30894]: pgmap v1605: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 845 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:14.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:14 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3847388328' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:53:14.275 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-11T03:53:14.275 INFO:teuthology.orchestra.run.smithi175.stdout: "size": 20, 2024-09-11T03:53:14.275 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 600, 2024-09-11T03:53:14.275 INFO:teuthology.orchestra.run.smithi175.stdout: "ops": [ 2024-09-11T03:53:14.275 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.276 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5348 2.1b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e571)", 2024-09-11T03:53:14.276 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:28.996464+0000", 2024-09-11T03:53:14.276 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 405.27779942900003, 2024-09-11T03:53:14.276 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.973168225, 2024-09-11T03:53:14.276 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.276 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.276 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "reached pg", 2024-09-11T03:53:14.276 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.276 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.276 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.276 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5348 2024-09-11T03:53:14.276 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.277 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.277 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.277 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.277 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996464+0000", 2024-09-11T03:53:14.277 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.277 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.277 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.277 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.277 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996464+0000", 2024-09-11T03:53:14.277 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.277 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.277 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.277 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.278 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996465+0000", 2024-09-11T03:53:14.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3500000000000002e-07 2024-09-11T03:53:14.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.278 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.278 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.278 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996467+0000", 2024-09-11T03:53:14.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.576e-06 2024-09-11T03:53:14.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.278 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.278 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.278 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996475+0000", 2024-09-11T03:53:14.278 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.1350000000000003e-06 2024-09-11T03:53:14.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.278 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.279 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.279 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996478+0000", 2024-09-11T03:53:14.279 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5240000000000001e-06 2024-09-11T03:53:14.279 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.279 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.279 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.279 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996545+0000", 2024-09-11T03:53:14.279 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.7379000000000007e-05 2024-09-11T03:53:14.279 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.279 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.279 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:14.279 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996557+0000", 2024-09-11T03:53:14.279 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2019e-05 2024-09-11T03:53:14.279 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.280 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.280 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.280 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969589+0000", 2024-09-11T03:53:14.280 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97303155699999999 2024-09-11T03:53:14.280 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.280 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.280 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.280 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969633+0000", 2024-09-11T03:53:14.280 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3699999999999998e-05 2024-09-11T03:53:14.280 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.280 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.280 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.280 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.281 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.281 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5349 2.1b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e571)", 2024-09-11T03:53:14.281 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:53:14.281 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 405.27777630100002, 2024-09-11T03:53:14.281 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97318775899999999, 2024-09-11T03:53:14.281 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.281 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.281 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "reached pg", 2024-09-11T03:53:14.281 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.281 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.281 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.281 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5349 2024-09-11T03:53:14.281 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.282 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.282 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.282 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:53:14.282 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.282 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.282 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:53:14.282 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.283 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.283 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996488+0000", 2024-09-11T03:53:14.283 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7300000000000002e-07 2024-09-11T03:53:14.283 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.283 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.283 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996490+0000", 2024-09-11T03:53:14.283 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.8470000000000001e-06 2024-09-11T03:53:14.283 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.283 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.283 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996494+0000", 2024-09-11T03:53:14.284 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.8419999999999998e-06 2024-09-11T03:53:14.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.284 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.284 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996497+0000", 2024-09-11T03:53:14.284 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.8820000000000001e-06 2024-09-11T03:53:14.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.284 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.284 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996602+0000", 2024-09-11T03:53:14.284 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010565699999999999 2024-09-11T03:53:14.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.285 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:14.285 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996613+0000", 2024-09-11T03:53:14.285 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0359e-05 2024-09-11T03:53:14.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.285 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.285 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969667+0000", 2024-09-11T03:53:14.285 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97305483599999998 2024-09-11T03:53:14.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.285 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.285 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969675+0000", 2024-09-11T03:53:14.285 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.9629999999999998e-06 2024-09-11T03:53:14.285 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.285 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.286 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.286 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.286 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5350 2.1b 2.63f497bb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e571)", 2024-09-11T03:53:14.286 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:53:14.286 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 405.27774114499999, 2024-09-11T03:53:14.286 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.973193639, 2024-09-11T03:53:14.286 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.286 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.286 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "reached pg", 2024-09-11T03:53:14.286 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.286 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.286 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.286 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5350 2024-09-11T03:53:14.287 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.287 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.287 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.287 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.287 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:53:14.287 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.287 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.287 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.287 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.287 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:53:14.287 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.287 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.287 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.287 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.288 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996523+0000", 2024-09-11T03:53:14.288 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.27e-07 2024-09-11T03:53:14.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.288 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.288 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.288 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996525+0000", 2024-09-11T03:53:14.288 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.719e-06 2024-09-11T03:53:14.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.288 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.288 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.288 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996529+0000", 2024-09-11T03:53:14.288 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.2690000000000001e-06 2024-09-11T03:53:14.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.289 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.289 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996532+0000", 2024-09-11T03:53:14.289 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.4139999999999998e-06 2024-09-11T03:53:14.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.289 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.289 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996643+0000", 2024-09-11T03:53:14.289 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011041599999999999 2024-09-11T03:53:14.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.289 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:14.289 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:28.996651+0000", 2024-09-11T03:53:14.289 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4309999999999994e-06 2024-09-11T03:53:14.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.290 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.290 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.290 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969706+0000", 2024-09-11T03:53:14.290 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.97305510699999997 2024-09-11T03:53:14.290 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.290 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.291 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.291 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:29.969716+0000", 2024-09-11T03:53:14.291 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.9559999999999992e-06 2024-09-11T03:53:14.291 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.291 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.291 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.291 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.291 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.291 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5348 2.1b 2:dde92fc6:::smithi17557870-41:head [sparse-read 0~1317959,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:53:14.292 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973173+0000", 2024-09-11T03:53:14.292 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 403.30109096699999, 2024-09-11T03:53:14.292 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019341553000000001, 2024-09-11T03:53:14.292 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.292 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.292 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:14.292 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.292 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.292 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.292 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5348 2024-09-11T03:53:14.292 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.292 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.293 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.293 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.293 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973173+0000", 2024-09-11T03:53:14.293 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.293 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.293 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.293 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.293 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973173+0000", 2024-09-11T03:53:14.293 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.293 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.293 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.293 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.293 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973175+0000", 2024-09-11T03:53:14.293 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.074e-06 2024-09-11T03:53:14.294 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.294 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.294 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.294 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973183+0000", 2024-09-11T03:53:14.294 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3010000000000007e-06 2024-09-11T03:53:14.294 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.294 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.294 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.294 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973199+0000", 2024-09-11T03:53:14.294 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5279000000000001e-05 2024-09-11T03:53:14.294 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.294 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.294 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.294 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973214+0000", 2024-09-11T03:53:14.295 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.5891000000000001e-05 2024-09-11T03:53:14.295 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.295 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.295 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.295 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973747+0000", 2024-09-11T03:53:14.295 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00053236600000000005 2024-09-11T03:53:14.295 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.295 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.295 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:14.295 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973760+0000", 2024-09-11T03:53:14.295 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2976000000000001e-05 2024-09-11T03:53:14.295 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.296 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.296 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.296 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.990782+0000", 2024-09-11T03:53:14.296 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017022371000000001 2024-09-11T03:53:14.296 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.296 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.296 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:14.296 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.990945+0000", 2024-09-11T03:53:14.296 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00016245699999999999 2024-09-11T03:53:14.296 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.296 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.296 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.297 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.992514+0000", 2024-09-11T03:53:14.297 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001569838 2024-09-11T03:53:14.297 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.297 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.297 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.297 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.297 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.297 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5349 2.1b 2:dde92fc6:::smithi17557870-41:head [read 0~2235463,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:53:14.297 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:53:14.297 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 403.30102625500001, 2024-09-11T03:53:14.297 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020826846, 2024-09-11T03:53:14.297 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.298 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.298 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:14.298 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.298 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.298 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.298 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5349 2024-09-11T03:53:14.298 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.298 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.298 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.298 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.298 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:53:14.298 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.298 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.299 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.299 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.299 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:53:14.299 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.299 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.299 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.299 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.299 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973238+0000", 2024-09-11T03:53:14.299 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.2099999999999996e-07 2024-09-11T03:53:14.299 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.299 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.299 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.299 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973242+0000", 2024-09-11T03:53:14.300 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0489999999999997e-06 2024-09-11T03:53:14.300 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.300 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.300 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.300 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973250+0000", 2024-09-11T03:53:14.300 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0700000000000007e-06 2024-09-11T03:53:14.300 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.300 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.300 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.300 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973256+0000", 2024-09-11T03:53:14.300 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3979999999999998e-06 2024-09-11T03:53:14.300 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.300 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.300 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.301 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973794+0000", 2024-09-11T03:53:14.301 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00053782700000000005 2024-09-11T03:53:14.301 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.301 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.301 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:14.301 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973801+0000", 2024-09-11T03:53:14.301 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.5299999999999999e-06 2024-09-11T03:53:14.301 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.301 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.301 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.301 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.992813+0000", 2024-09-11T03:53:14.301 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.019011917999999999 2024-09-11T03:53:14.301 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.302 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.302 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:14.302 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.992923+0000", 2024-09-11T03:53:14.302 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000109874 2024-09-11T03:53:14.302 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.302 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.302 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.302 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994064+0000", 2024-09-11T03:53:14.302 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001141459 2024-09-11T03:53:14.302 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.302 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.302 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.302 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.302 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.302 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5350 2.1b 2:dde92fc6:::smithi17557870-41:head [sparse-read 0~1317959] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:53:14.303 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973264+0000", 2024-09-11T03:53:14.303 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 403.30099953299998, 2024-09-11T03:53:14.303 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.021144998000000002, 2024-09-11T03:53:14.303 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.303 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.303 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:14.303 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.303 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.303 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.303 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5350 2024-09-11T03:53:14.304 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.304 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.304 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.304 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.304 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973264+0000", 2024-09-11T03:53:14.304 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.305 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.305 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.305 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.305 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973264+0000", 2024-09-11T03:53:14.305 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.305 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.305 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.305 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.305 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973265+0000", 2024-09-11T03:53:14.305 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.5899999999999996e-07 2024-09-11T03:53:14.306 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.306 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.306 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.306 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973268+0000", 2024-09-11T03:53:14.306 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6970000000000002e-06 2024-09-11T03:53:14.306 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.306 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.306 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.306 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973274+0000", 2024-09-11T03:53:14.306 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.6390000000000003e-06 2024-09-11T03:53:14.306 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.307 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.307 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.307 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973280+0000", 2024-09-11T03:53:14.307 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.451e-06 2024-09-11T03:53:14.307 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.307 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.307 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.307 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973888+0000", 2024-09-11T03:53:14.307 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00060798999999999998 2024-09-11T03:53:14.307 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.307 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.307 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:14.307 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973895+0000", 2024-09-11T03:53:14.308 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6199999999999999e-06 2024-09-11T03:53:14.308 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.308 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.308 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.308 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994115+0000", 2024-09-11T03:53:14.308 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020219487000000001 2024-09-11T03:53:14.308 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.308 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.308 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:14.308 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994236+0000", 2024-09-11T03:53:14.308 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000120715 2024-09-11T03:53:14.308 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.308 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.308 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.309 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994409+0000", 2024-09-11T03:53:14.309 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017384 2024-09-11T03:53:14.309 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.309 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.309 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.309 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.309 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.309 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5362 2.1b 2:de6b1316:::smithi17557870-32:head [read 0~2117847,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:53:14.309 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973757+0000", 2024-09-11T03:53:14.309 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 403.30050662399998, 2024-09-11T03:53:14.309 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.023415378000000001, 2024-09-11T03:53:14.309 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.309 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.309 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:14.310 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.310 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.310 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.310 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5362 2024-09-11T03:53:14.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.310 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.310 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.310 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973757+0000", 2024-09-11T03:53:14.310 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.310 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.310 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973757+0000", 2024-09-11T03:53:14.311 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.311 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.311 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973758+0000", 2024-09-11T03:53:14.311 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.088e-06 2024-09-11T03:53:14.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.311 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.311 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973765+0000", 2024-09-11T03:53:14.311 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.1310000000000003e-06 2024-09-11T03:53:14.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.311 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.312 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973776+0000", 2024-09-11T03:53:14.312 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1762e-05 2024-09-11T03:53:14.312 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.312 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.312 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.312 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973785+0000", 2024-09-11T03:53:14.312 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.8629999999999997e-06 2024-09-11T03:53:14.312 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.312 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.312 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.312 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973985+0000", 2024-09-11T03:53:14.312 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000200051 2024-09-11T03:53:14.312 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.312 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.313 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:14.313 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973995+0000", 2024-09-11T03:53:14.313 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0054000000000001e-05 2024-09-11T03:53:14.313 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.313 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.313 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.313 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994538+0000", 2024-09-11T03:53:14.313 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.020542878000000001 2024-09-11T03:53:14.313 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.313 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.313 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:14.313 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.994729+0000", 2024-09-11T03:53:14.313 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00019097100000000001 2024-09-11T03:53:14.313 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.314 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.314 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.314 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.997173+0000", 2024-09-11T03:53:14.314 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0024435799999999999 2024-09-11T03:53:14.314 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.314 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.314 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.314 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.314 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.314 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5363 2.1b 2:de6b1316:::smithi17557870-32:head [read 0~2117847,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:53:14.314 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973799+0000", 2024-09-11T03:53:14.314 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 403.30046461500001, 2024-09-11T03:53:14.315 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.024619513999999999, 2024-09-11T03:53:14.315 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.315 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.315 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:14.315 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.315 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.315 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.315 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5363 2024-09-11T03:53:14.315 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.315 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.315 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.315 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.315 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973799+0000", 2024-09-11T03:53:14.315 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.316 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.316 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973799+0000", 2024-09-11T03:53:14.316 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.316 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.316 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973800+0000", 2024-09-11T03:53:14.316 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.2500000000000005e-07 2024-09-11T03:53:14.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.316 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.316 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973804+0000", 2024-09-11T03:53:14.317 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6049999999999998e-06 2024-09-11T03:53:14.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.317 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.317 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.317 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973811+0000", 2024-09-11T03:53:14.317 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.0350000000000004e-06 2024-09-11T03:53:14.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.317 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.317 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.317 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973816+0000", 2024-09-11T03:53:14.317 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.536e-06 2024-09-11T03:53:14.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.318 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.318 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.318 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974024+0000", 2024-09-11T03:53:14.318 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00020766099999999999 2024-09-11T03:53:14.318 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.318 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.318 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:14.318 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974032+0000", 2024-09-11T03:53:14.318 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.3340000000000002e-06 2024-09-11T03:53:14.318 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.318 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.318 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.318 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.997304+0000", 2024-09-11T03:53:14.318 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.02327183 2024-09-11T03:53:14.318 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.319 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.319 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:14.319 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.997455+0000", 2024-09-11T03:53:14.319 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00015147000000000001 2024-09-11T03:53:14.319 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.319 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.319 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.319 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.998419+0000", 2024-09-11T03:53:14.319 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00096341799999999996 2024-09-11T03:53:14.319 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.319 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.319 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.319 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.320 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.320 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5364 2.1b 2:de6b1316:::smithi17557870-32:head [read 0~2117847,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e573)", 2024-09-11T03:53:14.320 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:53:14.320 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 403.30044089500001, 2024-09-11T03:53:14.320 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.025348697, 2024-09-11T03:53:14.320 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.320 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.320 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:14.320 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.320 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.320 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.320 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5364 2024-09-11T03:53:14.320 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.320 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.321 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.321 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.321 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:53:14.321 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.321 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.321 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.321 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.321 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:53:14.321 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.321 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.321 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.321 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.321 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973823+0000", 2024-09-11T03:53:14.322 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.4099999999999999e-07 2024-09-11T03:53:14.322 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.322 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.322 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.322 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973826+0000", 2024-09-11T03:53:14.322 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.7690000000000001e-06 2024-09-11T03:53:14.322 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.322 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.322 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.322 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973833+0000", 2024-09-11T03:53:14.322 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.4679999999999997e-06 2024-09-11T03:53:14.322 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.322 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.322 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.323 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.973837+0000", 2024-09-11T03:53:14.323 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6070000000000001e-06 2024-09-11T03:53:14.323 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.323 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.323 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.323 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974062+0000", 2024-09-11T03:53:14.323 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00022431400000000001 2024-09-11T03:53:14.323 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.323 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.323 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for peered", 2024-09-11T03:53:14.323 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.974069+0000", 2024-09-11T03:53:14.323 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.8599999999999993e-06 2024-09-11T03:53:14.323 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.324 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.324 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.324 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.998467+0000", 2024-09-11T03:53:14.324 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.024397723999999999 2024-09-11T03:53:14.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.324 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.324 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:14.324 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.998581+0000", 2024-09-11T03:53:14.324 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000113769 2024-09-11T03:53:14.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.324 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.324 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.324 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:46:30.999172+0000", 2024-09-11T03:53:14.325 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00059074500000000005 2024-09-11T03:53:14.325 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.325 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.325 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.325 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.325 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.325 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5778 2.a 2:5014e305:::smithi17557870-46:head [copy-from ver 878] snapc ff=[ff,fe,fd] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-09-11T03:53:14.325 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:06.021580+0000", 2024-09-11T03:53:14.325 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 368.25268433500003, 2024-09-11T03:53:14.325 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.023197538, 2024-09-11T03:53:14.325 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.325 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.325 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:14.325 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.326 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.326 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.326 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5778 2024-09-11T03:53:14.326 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.326 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.326 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.326 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.326 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021580+0000", 2024-09-11T03:53:14.326 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.326 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.326 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.326 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.326 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021580+0000", 2024-09-11T03:53:14.326 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.327 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.327 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.327 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.327 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021581+0000", 2024-09-11T03:53:14.327 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6099999999999999e-07 2024-09-11T03:53:14.327 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.327 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.327 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.327 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021585+0000", 2024-09-11T03:53:14.327 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.279e-06 2024-09-11T03:53:14.327 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.327 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.327 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.328 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021593+0000", 2024-09-11T03:53:14.328 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.1999999999999994e-06 2024-09-11T03:53:14.328 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.328 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.328 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.328 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021599+0000", 2024-09-11T03:53:14.328 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8200000000000002e-06 2024-09-11T03:53:14.328 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.328 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.328 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.328 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021691+0000", 2024-09-11T03:53:14.328 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.2142e-05 2024-09-11T03:53:14.328 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.328 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.329 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:14.329 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.021868+0000", 2024-09-11T03:53:14.329 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00017677200000000001 2024-09-11T03:53:14.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.329 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.329 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:53:14.329 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.030689+0000", 2024-09-11T03:53:14.329 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0088215310000000005 2024-09-11T03:53:14.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.329 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.329 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.329 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.038924+0000", 2024-09-11T03:53:14.329 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.008234412 2024-09-11T03:53:14.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.329 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.330 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.330 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044669+0000", 2024-09-11T03:53:14.330 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0057455529999999996 2024-09-11T03:53:14.330 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.330 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.330 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:14.330 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044696+0000", 2024-09-11T03:53:14.330 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6843e-05 2024-09-11T03:53:14.330 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.330 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.330 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:14.330 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044748+0000", 2024-09-11T03:53:14.330 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1711e-05 2024-09-11T03:53:14.330 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.330 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.330 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.331 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:06.044777+0000", 2024-09-11T03:53:14.331 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.9314000000000001e-05 2024-09-11T03:53:14.331 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.331 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.331 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.331 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.331 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.331 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5983 2.1b 2:de6b1316:::smithi17557870-32:head [copy-from ver 692] snapc 108=[108,107] ondisk+write+known_if_redirected+supports_pool_eio e627)", 2024-09-11T03:53:14.331 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:25.010709+0000", 2024-09-11T03:53:14.331 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 349.26355523500001, 2024-09-11T03:53:14.331 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022123384999999999, 2024-09-11T03:53:14.331 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.331 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.331 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:14.332 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.332 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.332 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.332 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5983 2024-09-11T03:53:14.332 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.332 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.332 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.332 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.332 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010709+0000", 2024-09-11T03:53:14.332 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.332 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.332 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.332 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.333 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010709+0000", 2024-09-11T03:53:14.333 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.333 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.333 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.333 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.333 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010711+0000", 2024-09-11T03:53:14.333 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.367e-06 2024-09-11T03:53:14.333 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.333 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.333 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.333 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010723+0000", 2024-09-11T03:53:14.333 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.2098e-05 2024-09-11T03:53:14.333 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.333 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.334 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.334 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010748+0000", 2024-09-11T03:53:14.334 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.4641e-05 2024-09-11T03:53:14.334 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.334 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.334 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.334 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010761+0000", 2024-09-11T03:53:14.334 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.329e-05 2024-09-11T03:53:14.334 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.334 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.334 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.334 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010856+0000", 2024-09-11T03:53:14.334 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5278000000000005e-05 2024-09-11T03:53:14.335 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.335 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.335 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:53:14.335 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010862+0000", 2024-09-11T03:53:14.335 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.088e-06 2024-09-11T03:53:14.335 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.335 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.335 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.335 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.012360+0000", 2024-09-11T03:53:14.335 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014979940000000001 2024-09-11T03:53:14.335 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.335 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.335 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:14.335 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.012437+0000", 2024-09-11T03:53:14.336 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6959000000000006e-05 2024-09-11T03:53:14.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.336 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.336 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 7,9", 2024-09-11T03:53:14.336 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.020310+0000", 2024-09-11T03:53:14.336 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0078724099999999998 2024-09-11T03:53:14.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.336 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.336 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:14.336 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.028691+0000", 2024-09-11T03:53:14.336 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.008381299 2024-09-11T03:53:14.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.336 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.337 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.337 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.028800+0000", 2024-09-11T03:53:14.337 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000109082 2024-09-11T03:53:14.337 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.337 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.337 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.337 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.032710+0000", 2024-09-11T03:53:14.337 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0039093660000000001 2024-09-11T03:53:14.337 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.337 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.337 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:14.337 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.032764+0000", 2024-09-11T03:53:14.337 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4415e-05 2024-09-11T03:53:14.337 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.338 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.338 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.338 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.032832+0000", 2024-09-11T03:53:14.338 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8097999999999998e-05 2024-09-11T03:53:14.338 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.338 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.338 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.338 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.338 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.338 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:5984 2.1b 2:de6b1316:::smithi17557870-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e627)", 2024-09-11T03:53:14.338 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:25.010792+0000", 2024-09-11T03:53:14.338 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 349.26347179800001, 2024-09-11T03:53:14.338 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014851705, 2024-09-11T03:53:14.339 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.339 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.339 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:14.339 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.339 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.339 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.339 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 5984 2024-09-11T03:53:14.339 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.339 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.339 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.339 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.339 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010792+0000", 2024-09-11T03:53:14.339 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.339 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.340 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.340 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.340 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010792+0000", 2024-09-11T03:53:14.340 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.340 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.340 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.340 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.340 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010793+0000", 2024-09-11T03:53:14.340 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8999999999999996e-07 2024-09-11T03:53:14.340 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.340 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.340 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.340 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010796+0000", 2024-09-11T03:53:14.341 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.6200000000000001e-06 2024-09-11T03:53:14.341 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.341 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.341 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.341 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010811+0000", 2024-09-11T03:53:14.341 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.4620999999999999e-05 2024-09-11T03:53:14.341 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.341 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.341 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.341 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010818+0000", 2024-09-11T03:53:14.341 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5490000000000003e-06 2024-09-11T03:53:14.341 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.341 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.341 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.342 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010926+0000", 2024-09-11T03:53:14.342 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108318 2024-09-11T03:53:14.342 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.342 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.342 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting_for_map not empty", 2024-09-11T03:53:14.342 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.010931+0000", 2024-09-11T03:53:14.342 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.1669999999999998e-06 2024-09-11T03:53:14.342 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.342 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.342 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.342 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.012597+0000", 2024-09-11T03:53:14.342 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.001665487 2024-09-11T03:53:14.342 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.343 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.343 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:53:14.343 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.012646+0000", 2024-09-11T03:53:14.343 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9073000000000003e-05 2024-09-11T03:53:14.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.343 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.343 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.343 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.025501+0000", 2024-09-11T03:53:14.343 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012855263 2024-09-11T03:53:14.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.343 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.343 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:14.343 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.025569+0000", 2024-09-11T03:53:14.343 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.8099e-05 2024-09-11T03:53:14.344 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.344 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.344 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.344 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:25.025644+0000", 2024-09-11T03:53:14.344 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.4818000000000001e-05 2024-09-11T03:53:14.344 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.344 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.344 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.344 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.344 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.344 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:6106 2.21 2:8638b65e:::smithi17557870-5:head [copy-from ver 603] snapc 10c=[10c] ondisk+write+known_if_redirected+supports_pool_eio e636)", 2024-09-11T03:53:14.344 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:47:34.018719+0000", 2024-09-11T03:53:14.344 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 340.25554468299998, 2024-09-11T03:53:14.345 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016460907, 2024-09-11T03:53:14.345 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.345 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.345 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:14.345 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.345 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.345 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.345 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 6106 2024-09-11T03:53:14.345 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.345 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.345 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.345 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.345 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018719+0000", 2024-09-11T03:53:14.345 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.345 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.346 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.346 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.346 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018719+0000", 2024-09-11T03:53:14.346 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.346 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.346 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.346 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018722+0000", 2024-09-11T03:53:14.346 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6730000000000001e-06 2024-09-11T03:53:14.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.346 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.346 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.346 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018732+0000", 2024-09-11T03:53:14.346 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0349e-05 2024-09-11T03:53:14.347 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.347 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.347 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.347 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018756+0000", 2024-09-11T03:53:14.347 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.3493999999999999e-05 2024-09-11T03:53:14.347 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.347 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.347 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.347 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018767+0000", 2024-09-11T03:53:14.347 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.079e-05 2024-09-11T03:53:14.347 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.347 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.347 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.347 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018875+0000", 2024-09-11T03:53:14.348 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000108789 2024-09-11T03:53:14.348 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.348 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.348 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op must wait for map", 2024-09-11T03:53:14.348 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.018884+0000", 2024-09-11T03:53:14.348 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.4800000000000001e-06 2024-09-11T03:53:14.348 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.348 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.348 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.348 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.020397+0000", 2024-09-11T03:53:14.348 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0015135770000000001 2024-09-11T03:53:14.348 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.348 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.348 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:14.349 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.020508+0000", 2024-09-11T03:53:14.349 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000110351 2024-09-11T03:53:14.349 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.349 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.350 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,8", 2024-09-11T03:53:14.350 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.028802+0000", 2024-09-11T03:53:14.350 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0082943189999999997 2024-09-11T03:53:14.350 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.350 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.352 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:14.352 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.033207+0000", 2024-09-11T03:53:14.352 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0044050820000000003 2024-09-11T03:53:14.353 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.353 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.353 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.353 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.033684+0000", 2024-09-11T03:53:14.353 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00047677899999999998 2024-09-11T03:53:14.353 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.353 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.353 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.353 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.035101+0000", 2024-09-11T03:53:14.353 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014171539999999999 2024-09-11T03:53:14.353 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.353 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.353 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:14.354 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.035147+0000", 2024-09-11T03:53:14.354 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.5731e-05 2024-09-11T03:53:14.354 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.354 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.354 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.354 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:47:34.035180+0000", 2024-09-11T03:53:14.354 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.3339e-05 2024-09-11T03:53:14.354 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.354 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.354 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.354 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.354 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.354 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7115 2.c 2:3603e322:::smithi17557870-42:head [copy-from ver 930] snapc 13b=[13b,13a,139,138,137,136,135,133,12e,12d,12c,12a,127,126,120,11e,11d,119] ondisk+write+known_if_redirected+supports_pool_eio e713)", 2024-09-11T03:53:14.354 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:51.194110+0000", 2024-09-11T03:53:14.354 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 263.08015359699999, 2024-09-11T03:53:14.355 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.029033742000000001, 2024-09-11T03:53:14.355 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.355 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.355 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:14.355 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.355 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.355 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.355 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7115 2024-09-11T03:53:14.355 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.355 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.356 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.356 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.356 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194110+0000", 2024-09-11T03:53:14.356 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.356 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.356 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.356 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.356 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194110+0000", 2024-09-11T03:53:14.356 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.356 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.356 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.356 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.356 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194111+0000", 2024-09-11T03:53:14.356 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3200000000000005e-07 2024-09-11T03:53:14.357 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.357 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.357 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.357 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194115+0000", 2024-09-11T03:53:14.357 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.5980000000000001e-06 2024-09-11T03:53:14.357 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.357 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.357 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.357 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194121+0000", 2024-09-11T03:53:14.357 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3289999999999999e-06 2024-09-11T03:53:14.357 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.357 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.357 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.357 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194126+0000", 2024-09-11T03:53:14.358 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4e-06 2024-09-11T03:53:14.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.358 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.358 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194207+0000", 2024-09-11T03:53:14.358 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0721000000000002e-05 2024-09-11T03:53:14.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.358 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:14.358 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194303+0000", 2024-09-11T03:53:14.358 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.5546999999999998e-05 2024-09-11T03:53:14.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.358 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 0,1", 2024-09-11T03:53:14.359 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.207212+0000", 2024-09-11T03:53:14.359 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.012909132 2024-09-11T03:53:14.359 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.359 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.359 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:14.359 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.209890+0000", 2024-09-11T03:53:14.359 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002677876 2024-09-11T03:53:14.359 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.359 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.359 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.359 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223023+0000", 2024-09-11T03:53:14.359 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013133137 2024-09-11T03:53:14.359 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.360 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.360 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.360 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223063+0000", 2024-09-11T03:53:14.360 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.0413000000000001e-05 2024-09-11T03:53:14.360 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.360 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.360 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:14.360 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223107+0000", 2024-09-11T03:53:14.360 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3569999999999998e-05 2024-09-11T03:53:14.360 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.360 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.360 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.360 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.223144+0000", 2024-09-11T03:53:14.360 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 3.7387e-05 2024-09-11T03:53:14.361 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.361 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.361 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.361 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.361 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.361 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:7116 2.c 2:3603e322:::smithi17557870-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e713)", 2024-09-11T03:53:14.361 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:48:51.194143+0000", 2024-09-11T03:53:14.361 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 263.080121029, 2024-09-11T03:53:14.361 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015378424, 2024-09-11T03:53:14.361 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.361 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.361 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "started", 2024-09-11T03:53:14.362 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.362 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.362 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.362 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 7116 2024-09-11T03:53:14.362 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.362 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.362 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.362 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.362 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194143+0000", 2024-09-11T03:53:14.362 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.362 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.362 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.362 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.362 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194143+0000", 2024-09-11T03:53:14.363 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.363 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.363 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.363 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194143+0000", 2024-09-11T03:53:14.363 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.9299999999999998e-07 2024-09-11T03:53:14.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.363 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.363 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.363 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194146+0000", 2024-09-11T03:53:14.363 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.2390000000000001e-06 2024-09-11T03:53:14.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.363 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.364 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.364 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194152+0000", 2024-09-11T03:53:14.364 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.5080000000000002e-06 2024-09-11T03:53:14.364 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.364 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.364 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.364 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194158+0000", 2024-09-11T03:53:14.364 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.942e-06 2024-09-11T03:53:14.364 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.364 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.364 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.364 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194461+0000", 2024-09-11T03:53:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00030244899999999999 2024-09-11T03:53:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for blocked object", 2024-09-11T03:53:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.194502+0000", 2024-09-11T03:53:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.1736000000000002e-05 2024-09-11T03:53:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.209298+0000", 2024-09-11T03:53:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.014795338999999999 2024-09-11T03:53:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.365 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.366 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:14.366 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.209398+0000", 2024-09-11T03:53:14.366 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00010077300000000001 2024-09-11T03:53:14.366 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.366 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.366 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.366 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:48:51.209521+0000", 2024-09-11T03:53:14.366 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00012294500000000001 2024-09-11T03:53:14.366 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.366 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.366 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.366 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.366 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.366 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:8915 2.21 2:8638b65e:::smithi17557870-5:head [rollback 13c] snapc 190=[190,18f,18e,18d,18c,18b,18a,189,188,187,186,185,184,183,181,180,17e,17d,17c,17a,178,177,175,174,172,171,170,16e,16b,169,167,164,15f,15e,15b,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-09-11T03:53:14.367 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:51:20.058112+0000", 2024-09-11T03:53:14.367 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 114.216151978, 2024-09-11T03:53:14.367 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.016574947999999999, 2024-09-11T03:53:14.367 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.367 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.367 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:14.367 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.367 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.367 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.367 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 8915 2024-09-11T03:53:14.367 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.367 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.367 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.368 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.368 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058112+0000", 2024-09-11T03:53:14.368 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.368 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.368 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.368 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.368 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058112+0000", 2024-09-11T03:53:14.368 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.368 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.368 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.368 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.368 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058113+0000", 2024-09-11T03:53:14.368 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.9299999999999998e-07 2024-09-11T03:53:14.368 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.369 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.369 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.369 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058115+0000", 2024-09-11T03:53:14.369 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.6579999999999998e-06 2024-09-11T03:53:14.369 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.369 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.369 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.369 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058123+0000", 2024-09-11T03:53:14.369 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3379999999999999e-06 2024-09-11T03:53:14.369 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.369 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.369 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.369 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058128+0000", 2024-09-11T03:53:14.370 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6099999999999997e-06 2024-09-11T03:53:14.370 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.370 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.370 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.370 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.061715+0000", 2024-09-11T03:53:14.370 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0035866280000000001 2024-09-11T03:53:14.370 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.370 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.370 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:14.370 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.061820+0000", 2024-09-11T03:53:14.370 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000104991 2024-09-11T03:53:14.370 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.370 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.370 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,8", 2024-09-11T03:53:14.371 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.062058+0000", 2024-09-11T03:53:14.371 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00023820800000000001 2024-09-11T03:53:14.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.371 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.371 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:14.371 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.065660+0000", 2024-09-11T03:53:14.371 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0036018589999999998 2024-09-11T03:53:14.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.371 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.371 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.371 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.065776+0000", 2024-09-11T03:53:14.371 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000115973 2024-09-11T03:53:14.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.371 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.372 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.372 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.073787+0000", 2024-09-11T03:53:14.372 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0080107230000000008 2024-09-11T03:53:14.372 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.372 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.372 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:14.372 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.074629+0000", 2024-09-11T03:53:14.372 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00084232399999999998 2024-09-11T03:53:14.372 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.372 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.372 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.372 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.074687+0000", 2024-09-11T03:53:14.372 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.8043e-05 2024-09-11T03:53:14.373 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.373 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.373 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.373 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.373 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.373 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:8916 2.21 2:8638b65e:::smithi17557870-5:head [append 0~0] snapc 190=[190,18f,18e,18d,18c,18b,18a,189,188,187,186,185,184,183,181,180,17e,17d,17c,17a,178,177,175,174,172,171,170,16e,16b,169,167,164,15f,15e,15b,144,143,13c,139,127,120] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-09-11T03:53:14.373 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:51:20.058135+0000", 2024-09-11T03:53:14.373 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 114.216128469, 2024-09-11T03:53:14.373 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.017146954999999998, 2024-09-11T03:53:14.373 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.373 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.373 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:14.373 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.373 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.374 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.374 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 8916 2024-09-11T03:53:14.374 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.374 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.374 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.374 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.374 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058135+0000", 2024-09-11T03:53:14.374 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.374 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.374 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.374 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.374 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058135+0000", 2024-09-11T03:53:14.374 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.374 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.375 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.375 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.375 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058136+0000", 2024-09-11T03:53:14.375 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.39e-07 2024-09-11T03:53:14.375 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.375 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.375 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.375 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058138+0000", 2024-09-11T03:53:14.375 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0650000000000001e-06 2024-09-11T03:53:14.375 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.375 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.375 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.375 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058144+0000", 2024-09-11T03:53:14.376 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.682e-06 2024-09-11T03:53:14.376 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.376 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.376 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.376 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.058149+0000", 2024-09-11T03:53:14.376 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0080000000000002e-06 2024-09-11T03:53:14.376 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.376 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.376 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.376 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.063104+0000", 2024-09-11T03:53:14.376 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0049554940000000004 2024-09-11T03:53:14.376 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.376 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.376 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:14.377 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.063201+0000", 2024-09-11T03:53:14.377 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 9.6933000000000002e-05 2024-09-11T03:53:14.377 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.377 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.377 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,8", 2024-09-11T03:53:14.377 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.063390+0000", 2024-09-11T03:53:14.377 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00018867499999999999 2024-09-11T03:53:14.377 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.377 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.377 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:14.377 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.065665+0000", 2024-09-11T03:53:14.377 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022753869999999998 2024-09-11T03:53:14.377 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.377 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.378 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.378 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.066506+0000", 2024-09-11T03:53:14.378 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00084052699999999998 2024-09-11T03:53:14.378 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.378 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.378 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.378 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.075114+0000", 2024-09-11T03:53:14.378 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0086082629999999997 2024-09-11T03:53:14.378 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.378 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.378 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:14.378 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.075225+0000", 2024-09-11T03:53:14.378 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000111588 2024-09-11T03:53:14.378 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.379 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.379 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.379 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:51:20.075282+0000", 2024-09-11T03:53:14.379 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.6894000000000001e-05 2024-09-11T03:53:14.379 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.379 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.379 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.379 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.379 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.379 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:9685 2.a 2:5014e305:::smithi17557870-46:head [write 534819~432002] snapc 1af=[1af,1ae,1ac,1aa,1a8,1a7,1a6,1a5,1a4,1a3,1a2,1a1,1a0,19d,19c,199,197,194,193,191,18b,183,17c,17a,178,175,172,16e,164,15f,13c] ondisk+write+known_if_redirected+supports_pool_eio e934)", 2024-09-11T03:53:14.379 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:52:32.082855+0000", 2024-09-11T03:53:14.379 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 42.191408840999998, 2024-09-11T03:53:14.379 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.015089, 2024-09-11T03:53:14.379 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.379 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.380 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:14.380 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.380 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.380 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.380 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 9685 2024-09-11T03:53:14.380 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.380 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.380 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.380 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.380 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:32.082855+0000", 2024-09-11T03:53:14.380 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.380 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.380 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.381 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.381 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:32.082855+0000", 2024-09-11T03:53:14.381 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.381 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.381 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.381 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.381 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:32.082857+0000", 2024-09-11T03:53:14.381 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.784e-06 2024-09-11T03:53:14.381 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.381 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.381 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.381 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:32.083192+0000", 2024-09-11T03:53:14.381 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00033552400000000002 2024-09-11T03:53:14.381 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.382 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.382 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.382 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:32.083213+0000", 2024-09-11T03:53:14.382 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 2.0301000000000001e-05 2024-09-11T03:53:14.382 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.382 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.382 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.382 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:32.083224+0000", 2024-09-11T03:53:14.382 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1157e-05 2024-09-11T03:53:14.382 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.382 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.382 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.382 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:32.085730+0000", 2024-09-11T03:53:14.382 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0025057009999999999 2024-09-11T03:53:14.383 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.383 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.383 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:14.383 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:32.085847+0000", 2024-09-11T03:53:14.383 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00011774500000000001 2024-09-11T03:53:14.383 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.383 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.383 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 3,9", 2024-09-11T03:53:14.383 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:32.086389+0000", 2024-09-11T03:53:14.383 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.00054204899999999996 2024-09-11T03:53:14.383 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.383 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.383 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.383 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:32.094121+0000", 2024-09-11T03:53:14.384 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0077315869999999998 2024-09-11T03:53:14.384 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.384 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.384 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:14.384 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:32.096347+0000", 2024-09-11T03:53:14.384 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0022258080000000001 2024-09-11T03:53:14.384 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.384 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.384 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.384 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:32.097805+0000", 2024-09-11T03:53:14.384 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0014587650000000001 2024-09-11T03:53:14.384 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.384 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.384 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:14.385 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:32.097891+0000", 2024-09-11T03:53:14.385 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.5172000000000004e-05 2024-09-11T03:53:14.385 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.385 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.385 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.385 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:32.097944+0000", 2024-09-11T03:53:14.385 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3406999999999999e-05 2024-09-11T03:53:14.385 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.385 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.385 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.385 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.385 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.385 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:9870 2.21 2:8638b65e:::smithi17557870-5:head [copy-from ver 1391] snapc 1b9=[1b9,1b8,1b7,1b6,1b5,1b4,1b3,1b2,1b1,1b0,1af,1ac,1aa,1a8,1a7,1a6,1a5,1a4,1a3,1a2,1a1,19d,19c,199,197,194,191,18b,183,17c,17a,175,172,16e,164,13c] ondisk+write+known_if_redirected+supports_pool_eio e949)", 2024-09-11T03:53:14.386 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:52:47.137342+0000", 2024-09-11T03:53:14.386 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 27.136921775000001, 2024-09-11T03:53:14.386 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.010410829999999999, 2024-09-11T03:53:14.386 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.386 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.386 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:14.386 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.386 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.386 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.386 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 9870 2024-09-11T03:53:14.386 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.386 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.386 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.386 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.387 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.137342+0000", 2024-09-11T03:53:14.387 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.387 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.387 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.387 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.387 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.137342+0000", 2024-09-11T03:53:14.387 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.387 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.387 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.387 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.387 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.137343+0000", 2024-09-11T03:53:14.387 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.288e-06 2024-09-11T03:53:14.387 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.387 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.137348+0000", 2024-09-11T03:53:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.6299999999999997e-06 2024-09-11T03:53:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.137359+0000", 2024-09-11T03:53:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.1044e-05 2024-09-11T03:53:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.137365+0000", 2024-09-11T03:53:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 6.3679999999999998e-06 2024-09-11T03:53:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.137414+0000", 2024-09-11T03:53:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.8550000000000001e-05 2024-09-11T03:53:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.137488+0000", 2024-09-11T03:53:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.3532e-05 2024-09-11T03:53:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,8", 2024-09-11T03:53:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.141853+0000", 2024-09-11T03:53:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0043655730000000002 2024-09-11T03:53:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.144913+0000", 2024-09-11T03:53:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.003059497 2024-09-11T03:53:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.145303+0000", 2024-09-11T03:53:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.000390293 2024-09-11T03:53:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.147617+0000", 2024-09-11T03:53:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.002314146 2024-09-11T03:53:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.147698+0000", 2024-09-11T03:53:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 8.0982999999999994e-05 2024-09-11T03:53:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.147753+0000", 2024-09-11T03:53:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.4926000000000002e-05 2024-09-11T03:53:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "description": "osd_op(client.24454.0:9886 2.1b 2:d8055656:::smithi17557870-39:head [copy-from ver 1265] snapc 1b9=[1b9,1b8,1b7,1b6,1b5,1b4,1b3,1b2,1b1,1b0,1af,1ac,1aa,1a8,1a7,1a6,1a5,1a4,1a3,1a2,1a1,19d,19c,199,197,194,191,18b,183,17c,17a,175,172,16e,164,13c] ondisk+write+known_if_redirected+supports_pool_eio e949)", 2024-09-11T03:53:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "initiated_at": "2024-09-11T03:52:47.170135+0000", 2024-09-11T03:53:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "age": 27.104128558999999, 2024-09-11T03:53:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.022138917000000001, 2024-09-11T03:53:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "continuous": false, 2024-09-11T03:53:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "type_data": { 2024-09-11T03:53:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T03:53:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "client_info": { 2024-09-11T03:53:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "client": "client.24454", 2024-09-11T03:53:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "client_addr": "172.21.15.175:0/3465170716", 2024-09-11T03:53:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "tid": 9886 2024-09-11T03:53:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "events": [ 2024-09-11T03:53:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "initiated", 2024-09-11T03:53:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.170135+0000", 2024-09-11T03:53:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "header_read", 2024-09-11T03:53:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.170135+0000", 2024-09-11T03:53:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0 2024-09-11T03:53:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "throttled", 2024-09-11T03:53:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.170137+0000", 2024-09-11T03:53:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.3400000000000001e-06 2024-09-11T03:53:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "all_read", 2024-09-11T03:53:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.170142+0000", 2024-09-11T03:53:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0769999999999998e-06 2024-09-11T03:53:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "dispatched", 2024-09-11T03:53:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.170152+0000", 2024-09-11T03:53:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 1.0587e-05 2024-09-11T03:53:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "queued_for_pg", 2024-09-11T03:53:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.170158+0000", 2024-09-11T03:53:14.395 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.3770000000000004e-06 2024-09-11T03:53:14.395 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.395 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.395 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "reached_pg", 2024-09-11T03:53:14.395 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.170215+0000", 2024-09-11T03:53:14.395 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.7640999999999999e-05 2024-09-11T03:53:14.395 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.395 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.395 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "started", 2024-09-11T03:53:14.395 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.170292+0000", 2024-09-11T03:53:14.395 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 7.6747000000000002e-05 2024-09-11T03:53:14.395 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.395 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.395 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "waiting for subops from 2,9", 2024-09-11T03:53:14.395 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.174405+0000", 2024-09-11T03:53:14.396 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0041128809999999997 2024-09-11T03:53:14.396 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.396 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.396 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "op_commit", 2024-09-11T03:53:14.396 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.177773+0000", 2024-09-11T03:53:14.396 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0033680139999999999 2024-09-11T03:53:14.396 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.396 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.396 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.396 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.178790+0000", 2024-09-11T03:53:14.396 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.0010169020000000001 2024-09-11T03:53:14.396 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.396 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.396 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "sub_op_commit_rec", 2024-09-11T03:53:14.397 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.192180+0000", 2024-09-11T03:53:14.397 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 0.013390447999999999 2024-09-11T03:53:14.397 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.397 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.397 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "commit_sent", 2024-09-11T03:53:14.397 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.192224+0000", 2024-09-11T03:53:14.397 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 4.3690999999999999e-05 2024-09-11T03:53:14.397 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-11T03:53:14.397 INFO:teuthology.orchestra.run.smithi175.stdout: { 2024-09-11T03:53:14.397 INFO:teuthology.orchestra.run.smithi175.stdout: "event": "done", 2024-09-11T03:53:14.397 INFO:teuthology.orchestra.run.smithi175.stdout: "time": "2024-09-11T03:52:47.192274+0000", 2024-09-11T03:53:14.397 INFO:teuthology.orchestra.run.smithi175.stdout: "duration": 5.0212e-05 2024-09-11T03:53:14.397 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.397 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.398 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.398 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-11T03:53:14.398 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-11T03:53:14.398 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-11T03:53:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:14 smithi121 ceph-mon[30729]: pgmap v1605: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 845 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:14.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:14 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3847388328' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-11T03:53:14.965 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T03:53:14.989 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:14 smithi084 podman[185314]: 2024-09-11 03:53:14.860663285 +0000 UTC m=+0.875022319 container init d2adc8d7aac824c1020796293b9e4c92bd0e40be4ecb39ecb1f4608c819c5849 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, CEPH_POINT_RELEASE=) 2024-09-11T03:53:14.989 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:14 smithi084 podman[185314]: 2024-09-11 03:53:14.864207478 +0000 UTC m=+0.878566516 container start d2adc8d7aac824c1020796293b9e4c92bd0e40be4ecb39ecb1f4608c819c5849 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-11T03:53:14.989 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:14 smithi084 ceph-osd[185348]: -- 172.21.15.84:0/1477143431 <== mon.0 v2:172.21.15.84:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ba75239a40 con 0x55ba76008000 2024-09-11T03:53:14.989 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:14 smithi084 bash[185314]: d2adc8d7aac824c1020796293b9e4c92bd0e40be4ecb39ecb1f4608c819c5849 2024-09-11T03:53:14.989 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:14 smithi084 systemd[1]: Started Ceph osd.0 for 10fe7946-6fee-11ef-bcea-c7b262605968. 2024-09-11T03:53:15.216 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:15.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:15 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3847388328' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:53:15.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:15 smithi084 ceph-mon[27254]: osdmap e963: 12 total, 11 up, 11 in 2024-09-11T03:53:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:15 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3847388328' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:53:15.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:15 smithi121 ceph-mon[30729]: osdmap e963: 12 total, 11 up, 11 in 2024-09-11T03:53:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:15 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3847388328' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-11T03:53:15.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:15 smithi175 ceph-mon[30894]: osdmap e963: 12 total, 11 up, 11 in 2024-09-11T03:53:15.682 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-11T03:53:15.682 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-11T03:53:15.682 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-11T03:53:15.682 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-11T03:53:15.682 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-11T03:53:15.682 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-11T03:53:15.682 INFO:teuthology.orchestra.run.smithi084.stderr:config diff 2024-09-11T03:53:15.682 INFO:teuthology.orchestra.run.smithi084.stderr:config diff get 2024-09-11T03:53:15.682 INFO:teuthology.orchestra.run.smithi084.stderr:config get 2024-09-11T03:53:15.682 INFO:teuthology.orchestra.run.smithi084.stderr:config help [] 2024-09-11T03:53:15.683 INFO:teuthology.orchestra.run.smithi084.stderr:config set ... 2024-09-11T03:53:15.683 INFO:teuthology.orchestra.run.smithi084.stderr:admin_socket: invalid command 2024-09-11T03:53:16.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:16 smithi084 ceph-mon[27254]: pgmap v1607: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 845 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:16.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:16 smithi084 ceph-mon[27254]: 1.8 scrub starts 2024-09-11T03:53:16.350 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:16 smithi084 ceph-mon[27254]: 1.8 scrub ok 2024-09-11T03:53:16.379 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T03:53:16.380 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-11T03:53:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:16 smithi121 ceph-mon[30729]: pgmap v1607: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 845 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:16 smithi121 ceph-mon[30729]: 1.8 scrub starts 2024-09-11T03:53:16.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:16 smithi121 ceph-mon[30729]: 1.8 scrub ok 2024-09-11T03:53:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:16 smithi175 ceph-mon[30894]: pgmap v1607: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 845 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:16 smithi175 ceph-mon[30894]: 1.8 scrub starts 2024-09-11T03:53:16.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:16 smithi175 ceph-mon[30894]: 1.8 scrub ok 2024-09-11T03:53:16.644 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:16 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0[185342]: 2024-09-11T03:53:16.492+0000 7f95da95b640 -1 Falling back to public interface 2024-09-11T03:53:17.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:17 smithi084 ceph-mon[27254]: 1.17 scrub starts 2024-09-11T03:53:17.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:17 smithi084 ceph-mon[27254]: 1.17 scrub ok 2024-09-11T03:53:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:17 smithi121 ceph-mon[30729]: 1.17 scrub starts 2024-09-11T03:53:17.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:17 smithi121 ceph-mon[30729]: 1.17 scrub ok 2024-09-11T03:53:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:17 smithi175 ceph-mon[30894]: 1.17 scrub starts 2024-09-11T03:53:17.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:17 smithi175 ceph-mon[30894]: 1.17 scrub ok 2024-09-11T03:53:18.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:18 smithi084 ceph-mon[27254]: pgmap v1608: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 845 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:18.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:18 smithi121 ceph-mon[30729]: pgmap v1608: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 845 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:18.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:18 smithi175 ceph-mon[30894]: pgmap v1608: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 845 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:19.894 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:19 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0[185342]: 2024-09-11T03:53:19.535+0000 7f95da95b640 -1 osd.0 939 log_to_monitors true 2024-09-11T03:53:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:20 smithi084 ceph-mon[27254]: pgmap v1609: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 845 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:20.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:20 smithi084 ceph-mon[27254]: from='osd.0 [v2:172.21.15.84:6802/3383145523,v1:172.21.15.84:6803/3383145523]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-11T03:53:20.395 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:53:20 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0[185342]: 2024-09-11T03:53:20.167+0000 7f95d2705640 -1 osd.0 939 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T03:53:20.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:20 smithi121 ceph-mon[30729]: pgmap v1609: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 845 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:20.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:20 smithi121 ceph-mon[30729]: from='osd.0 [v2:172.21.15.84:6802/3383145523,v1:172.21.15.84:6803/3383145523]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-11T03:53:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:20 smithi175 ceph-mon[30894]: pgmap v1609: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 845 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:20.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:20 smithi175 ceph-mon[30894]: from='osd.0 [v2:172.21.15.84:6802/3383145523,v1:172.21.15.84:6803/3383145523]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-11T03:53:21.380 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-11T03:53:21.408 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:21 smithi084 ceph-mon[27254]: from='osd.0 [v2:172.21.15.84:6802/3383145523,v1:172.21.15.84:6803/3383145523]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-11T03:53:21.408 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:21 smithi084 ceph-mon[27254]: osdmap e964: 12 total, 11 up, 11 in 2024-09-11T03:53:21.409 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:21 smithi084 ceph-mon[27254]: from='osd.0 [v2:172.21.15.84:6802/3383145523,v1:172.21.15.84:6803/3383145523]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:53:21.409 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:21 smithi084 ceph-mon[27254]: 1.12 scrub starts 2024-09-11T03:53:21.409 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:21 smithi084 ceph-mon[27254]: 1.12 scrub ok 2024-09-11T03:53:21.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:21 smithi121 ceph-mon[30729]: from='osd.0 [v2:172.21.15.84:6802/3383145523,v1:172.21.15.84:6803/3383145523]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-11T03:53:21.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:21 smithi121 ceph-mon[30729]: osdmap e964: 12 total, 11 up, 11 in 2024-09-11T03:53:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:21 smithi121 ceph-mon[30729]: from='osd.0 [v2:172.21.15.84:6802/3383145523,v1:172.21.15.84:6803/3383145523]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:53:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:21 smithi121 ceph-mon[30729]: 1.12 scrub starts 2024-09-11T03:53:21.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:21 smithi121 ceph-mon[30729]: 1.12 scrub ok 2024-09-11T03:53:21.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:21 smithi175 ceph-mon[30894]: from='osd.0 [v2:172.21.15.84:6802/3383145523,v1:172.21.15.84:6803/3383145523]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-11T03:53:21.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:21 smithi175 ceph-mon[30894]: osdmap e964: 12 total, 11 up, 11 in 2024-09-11T03:53:21.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:21 smithi175 ceph-mon[30894]: from='osd.0 [v2:172.21.15.84:6802/3383145523,v1:172.21.15.84:6803/3383145523]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]} : dispatch 2024-09-11T03:53:21.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:21 smithi175 ceph-mon[30894]: 1.12 scrub starts 2024-09-11T03:53:21.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:21 smithi175 ceph-mon[30894]: 1.12 scrub ok 2024-09-11T03:53:21.641 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:22.111 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-11T03:53:22.112 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-11T03:53:22.112 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-11T03:53:22.112 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-11T03:53:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:22 smithi084 ceph-mon[27254]: 1.5 scrub starts 2024-09-11T03:53:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:22 smithi084 ceph-mon[27254]: 1.5 scrub ok 2024-09-11T03:53:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:22 smithi084 ceph-mon[27254]: 1.1d scrub starts 2024-09-11T03:53:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:22 smithi084 ceph-mon[27254]: 1.1d scrub ok 2024-09-11T03:53:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:22 smithi084 ceph-mon[27254]: pgmap v1611: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 830 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:22 smithi084 ceph-mon[27254]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-11T03:53:22.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:22 smithi084 ceph-mon[27254]: osd.0 [v2:172.21.15.84:6802/3383145523,v1:172.21.15.84:6803/3383145523] boot 2024-09-11T03:53:22.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:22 smithi084 ceph-mon[27254]: osdmap e965: 12 total, 12 up, 11 in 2024-09-11T03:53:22.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:22 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:53:22.395 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:22 smithi084 ceph-mon[27254]: osdmap e966: 12 total, 12 up, 11 in 2024-09-11T03:53:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:22 smithi121 ceph-mon[30729]: 1.5 scrub starts 2024-09-11T03:53:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:22 smithi121 ceph-mon[30729]: 1.5 scrub ok 2024-09-11T03:53:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:22 smithi121 ceph-mon[30729]: 1.1d scrub starts 2024-09-11T03:53:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:22 smithi121 ceph-mon[30729]: 1.1d scrub ok 2024-09-11T03:53:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:22 smithi121 ceph-mon[30729]: pgmap v1611: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 830 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:22 smithi121 ceph-mon[30729]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-11T03:53:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:22 smithi121 ceph-mon[30729]: osd.0 [v2:172.21.15.84:6802/3383145523,v1:172.21.15.84:6803/3383145523] boot 2024-09-11T03:53:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:22 smithi121 ceph-mon[30729]: osdmap e965: 12 total, 12 up, 11 in 2024-09-11T03:53:22.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:22 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:53:22.423 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:22 smithi121 ceph-mon[30729]: osdmap e966: 12 total, 12 up, 11 in 2024-09-11T03:53:22.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:22 smithi175 ceph-mon[30894]: 1.5 scrub starts 2024-09-11T03:53:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:22 smithi175 ceph-mon[30894]: 1.5 scrub ok 2024-09-11T03:53:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:22 smithi175 ceph-mon[30894]: 1.1d scrub starts 2024-09-11T03:53:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:22 smithi175 ceph-mon[30894]: 1.1d scrub ok 2024-09-11T03:53:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:22 smithi175 ceph-mon[30894]: pgmap v1611: 31 pgs: 7 active+undersized, 2 active+undersized+degraded, 22 active+clean; 603 KiB data, 830 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:22 smithi175 ceph-mon[30894]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-11T03:53:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:22 smithi175 ceph-mon[30894]: osd.0 [v2:172.21.15.84:6802/3383145523,v1:172.21.15.84:6803/3383145523] boot 2024-09-11T03:53:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:22 smithi175 ceph-mon[30894]: osdmap e965: 12 total, 12 up, 11 in 2024-09-11T03:53:22.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:22 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-11T03:53:22.479 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:22 smithi175 ceph-mon[30894]: osdmap e966: 12 total, 12 up, 11 in 2024-09-11T03:53:22.820 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-11T03:53:22.820 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-11T03:53:22.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd in 4 2024-09-11T03:53:23.107 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:23.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:23 smithi121 ceph-mon[30729]: 1.1b scrub starts 2024-09-11T03:53:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:23 smithi121 ceph-mon[30729]: 1.1b scrub ok 2024-09-11T03:53:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:23 smithi121 ceph-mon[30729]: 1.11 scrub starts 2024-09-11T03:53:23.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:23 smithi121 ceph-mon[30729]: 1.11 scrub ok 2024-09-11T03:53:23.476 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:23 smithi084 ceph-mon[27254]: 1.1b scrub starts 2024-09-11T03:53:23.476 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:23 smithi084 ceph-mon[27254]: 1.1b scrub ok 2024-09-11T03:53:23.477 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:23 smithi084 ceph-mon[27254]: 1.11 scrub starts 2024-09-11T03:53:23.477 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:23 smithi084 ceph-mon[27254]: 1.11 scrub ok 2024-09-11T03:53:23.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:23 smithi175 ceph-mon[30894]: 1.1b scrub starts 2024-09-11T03:53:23.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:23 smithi175 ceph-mon[30894]: 1.1b scrub ok 2024-09-11T03:53:23.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:23 smithi175 ceph-mon[30894]: 1.11 scrub starts 2024-09-11T03:53:23.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:23 smithi175 ceph-mon[30894]: 1.11 scrub ok 2024-09-11T03:53:24.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:23 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a[27250]: 2024-09-11T03:53:23.749+0000 7fae0fb7d640 -1 mon.a@0(leader).osd e966 definitely_dead 0 2024-09-11T03:53:24.159 INFO:teuthology.orchestra.run.smithi084.stderr:marked in osd.4. 2024-09-11T03:53:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:24 smithi121 ceph-mon[30729]: 1.19 scrub starts 2024-09-11T03:53:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:24 smithi121 ceph-mon[30729]: 1.19 scrub ok 2024-09-11T03:53:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:24 smithi121 ceph-mon[30729]: pgmap v1614: 31 pgs: 3 peering, 2 active+undersized, 2 active+undersized+degraded, 24 active+clean; 603 KiB data, 829 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:24 smithi121 ceph-mon[30729]: 1.2 scrub starts 2024-09-11T03:53:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:24 smithi121 ceph-mon[30729]: 1.2 scrub ok 2024-09-11T03:53:24.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:24 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1933533457' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-11T03:53:24.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:24 smithi084 ceph-mon[27254]: 1.19 scrub starts 2024-09-11T03:53:24.463 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:24 smithi084 ceph-mon[27254]: 1.19 scrub ok 2024-09-11T03:53:24.463 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:24 smithi084 ceph-mon[27254]: pgmap v1614: 31 pgs: 3 peering, 2 active+undersized, 2 active+undersized+degraded, 24 active+clean; 603 KiB data, 829 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:24.463 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:24 smithi084 ceph-mon[27254]: 1.2 scrub starts 2024-09-11T03:53:24.463 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:24 smithi084 ceph-mon[27254]: 1.2 scrub ok 2024-09-11T03:53:24.463 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:24 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1933533457' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-11T03:53:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:24 smithi175 ceph-mon[30894]: 1.19 scrub starts 2024-09-11T03:53:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:24 smithi175 ceph-mon[30894]: 1.19 scrub ok 2024-09-11T03:53:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:24 smithi175 ceph-mon[30894]: pgmap v1614: 31 pgs: 3 peering, 2 active+undersized, 2 active+undersized+degraded, 24 active+clean; 603 KiB data, 829 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:24 smithi175 ceph-mon[30894]: 1.2 scrub starts 2024-09-11T03:53:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:24 smithi175 ceph-mon[30894]: 1.2 scrub ok 2024-09-11T03:53:24.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:24 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1933533457' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-11T03:53:24.895 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-11T03:53:24.895 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-11T03:53:25.141 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:25.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:25 smithi084 ceph-mon[27254]: 1.1 scrub starts 2024-09-11T03:53:25.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:25 smithi084 ceph-mon[27254]: 1.1 scrub ok 2024-09-11T03:53:25.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:25 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1933533457' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-11T03:53:25.421 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:25 smithi084 ceph-mon[27254]: osdmap e967: 12 total, 12 up, 12 in 2024-09-11T03:53:25.421 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:25 smithi084 ceph-mon[27254]: 1.1c scrub starts 2024-09-11T03:53:25.421 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:25 smithi084 ceph-mon[27254]: 1.1c scrub ok 2024-09-11T03:53:25.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:25 smithi121 ceph-mon[30729]: 1.1 scrub starts 2024-09-11T03:53:25.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:25 smithi121 ceph-mon[30729]: 1.1 scrub ok 2024-09-11T03:53:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:25 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1933533457' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-11T03:53:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:25 smithi121 ceph-mon[30729]: osdmap e967: 12 total, 12 up, 12 in 2024-09-11T03:53:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:25 smithi121 ceph-mon[30729]: 1.1c scrub starts 2024-09-11T03:53:25.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:25 smithi121 ceph-mon[30729]: 1.1c scrub ok 2024-09-11T03:53:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:25 smithi175 ceph-mon[30894]: 1.1 scrub starts 2024-09-11T03:53:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:25 smithi175 ceph-mon[30894]: 1.1 scrub ok 2024-09-11T03:53:25.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:25 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1933533457' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-11T03:53:25.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:25 smithi175 ceph-mon[30894]: osdmap e967: 12 total, 12 up, 12 in 2024-09-11T03:53:25.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:25 smithi175 ceph-mon[30894]: 1.1c scrub starts 2024-09-11T03:53:25.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:25 smithi175 ceph-mon[30894]: 1.1c scrub ok 2024-09-11T03:53:25.639 INFO:teuthology.orchestra.run.smithi084.stdout:mon.a: {} 2024-09-11T03:53:25.640 INFO:teuthology.orchestra.run.smithi084.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T03:53:25.649 INFO:teuthology.orchestra.run.smithi084.stdout:mon.c: {} 2024-09-11T03:53:25.650 INFO:teuthology.orchestra.run.smithi084.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T03:53:25.659 INFO:teuthology.orchestra.run.smithi084.stdout:mon.b: {} 2024-09-11T03:53:25.659 INFO:teuthology.orchestra.run.smithi084.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T03:53:26.398 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-11T03:53:26.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:26 smithi084 ceph-mon[27254]: pgmap v1616: 31 pgs: 3 peering, 1 active+undersized, 2 active+undersized+degraded, 25 active+clean; 603 KiB data, 828 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:26.421 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:26 smithi084 ceph-mon[27254]: osdmap e968: 12 total, 12 up, 12 in 2024-09-11T03:53:26.421 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:26 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3144129736' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-11T03:53:26.421 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:26 smithi121 ceph-mon[30729]: pgmap v1616: 31 pgs: 3 peering, 1 active+undersized, 2 active+undersized+degraded, 25 active+clean; 603 KiB data, 828 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:26 smithi121 ceph-mon[30729]: osdmap e968: 12 total, 12 up, 12 in 2024-09-11T03:53:26.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:26 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3144129736' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-11T03:53:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:26 smithi175 ceph-mon[30894]: pgmap v1616: 31 pgs: 3 peering, 1 active+undersized, 2 active+undersized+degraded, 25 active+clean; 603 KiB data, 828 MiB used, 983 GiB / 983 GiB avail; 2/9 objects degraded (22.222%) 2024-09-11T03:53:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:26 smithi175 ceph-mon[30894]: osdmap e968: 12 total, 12 up, 12 in 2024-09-11T03:53:26.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:26 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3144129736' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-11T03:53:26.641 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:27 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:53:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:27 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:53:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:27 smithi084 ceph-mon[27254]: 1.d scrub starts 2024-09-11T03:53:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:27 smithi084 ceph-mon[27254]: 1.d scrub ok 2024-09-11T03:53:27.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:27 smithi084 ceph-mon[27254]: osdmap e969: 12 total, 12 up, 12 in 2024-09-11T03:53:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:27 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:53:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:27 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:53:27.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:27 smithi175 ceph-mon[30894]: 1.d scrub starts 2024-09-11T03:53:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:27 smithi175 ceph-mon[30894]: 1.d scrub ok 2024-09-11T03:53:27.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:27 smithi175 ceph-mon[30894]: osdmap e969: 12 total, 12 up, 12 in 2024-09-11T03:53:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:27 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:53:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:27 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:53:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:27 smithi121 ceph-mon[30729]: 1.d scrub starts 2024-09-11T03:53:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:27 smithi121 ceph-mon[30729]: 1.d scrub ok 2024-09-11T03:53:27.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:27 smithi121 ceph-mon[30729]: osdmap e969: 12 total, 12 up, 12 in 2024-09-11T03:53:28.192 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T03:53:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:28 smithi175 ceph-mon[30894]: 1.e scrub starts 2024-09-11T03:53:28.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:28 smithi175 ceph-mon[30894]: 1.e scrub ok 2024-09-11T03:53:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:28 smithi175 ceph-mon[30894]: pgmap v1619: 31 pgs: 3 peering, 28 active+clean; 577 KiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:28 smithi175 ceph-mon[30894]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/9 objects degraded (22.222%), 2 pgs degraded) 2024-09-11T03:53:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:28 smithi175 ceph-mon[30894]: Cluster is now healthy 2024-09-11T03:53:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:28 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3207138784' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:53:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:28 smithi175 ceph-mon[30894]: 1.b scrub starts 2024-09-11T03:53:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:28 smithi175 ceph-mon[30894]: 1.b scrub ok 2024-09-11T03:53:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:28 smithi175 ceph-mon[30894]: 1.a scrub starts 2024-09-11T03:53:28.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:28 smithi175 ceph-mon[30894]: 1.a scrub ok 2024-09-11T03:53:28.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:28 smithi084 ceph-mon[27254]: 1.e scrub starts 2024-09-11T03:53:28.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:28 smithi084 ceph-mon[27254]: 1.e scrub ok 2024-09-11T03:53:28.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:28 smithi084 ceph-mon[27254]: pgmap v1619: 31 pgs: 3 peering, 28 active+clean; 577 KiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:28.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:28 smithi084 ceph-mon[27254]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/9 objects degraded (22.222%), 2 pgs degraded) 2024-09-11T03:53:28.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:28 smithi084 ceph-mon[27254]: Cluster is now healthy 2024-09-11T03:53:28.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:28 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3207138784' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:53:28.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:28 smithi084 ceph-mon[27254]: 1.b scrub starts 2024-09-11T03:53:28.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:28 smithi084 ceph-mon[27254]: 1.b scrub ok 2024-09-11T03:53:28.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:28 smithi084 ceph-mon[27254]: 1.a scrub starts 2024-09-11T03:53:28.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:28 smithi084 ceph-mon[27254]: 1.a scrub ok 2024-09-11T03:53:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:28 smithi121 ceph-mon[30729]: 1.e scrub starts 2024-09-11T03:53:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:28 smithi121 ceph-mon[30729]: 1.e scrub ok 2024-09-11T03:53:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:28 smithi121 ceph-mon[30729]: pgmap v1619: 31 pgs: 3 peering, 28 active+clean; 577 KiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:28 smithi121 ceph-mon[30729]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/9 objects degraded (22.222%), 2 pgs degraded) 2024-09-11T03:53:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:28 smithi121 ceph-mon[30729]: Cluster is now healthy 2024-09-11T03:53:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:28 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3207138784' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:53:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:28 smithi121 ceph-mon[30729]: 1.b scrub starts 2024-09-11T03:53:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:28 smithi121 ceph-mon[30729]: 1.b scrub ok 2024-09-11T03:53:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:28 smithi121 ceph-mon[30729]: 1.a scrub starts 2024-09-11T03:53:28.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:28 smithi121 ceph-mon[30729]: 1.a scrub ok 2024-09-11T03:53:28.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-11T03:53:29.197 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:29 smithi084 ceph-mon[27254]: 1.1a scrub starts 2024-09-11T03:53:29.197 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:29 smithi084 ceph-mon[27254]: 1.1a scrub ok 2024-09-11T03:53:29.197 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:29 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3207138784' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:53:29.198 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:29 smithi084 ceph-mon[27254]: osdmap e970: 12 total, 12 up, 12 in 2024-09-11T03:53:29.198 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:29 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:53:29.198 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:29 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:53:29.217 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:29.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:29 smithi175 ceph-mon[30894]: 1.1a scrub starts 2024-09-11T03:53:29.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:29 smithi175 ceph-mon[30894]: 1.1a scrub ok 2024-09-11T03:53:29.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:29 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3207138784' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:53:29.480 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:29 smithi175 ceph-mon[30894]: osdmap e970: 12 total, 12 up, 12 in 2024-09-11T03:53:29.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:29 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:53:29.481 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:29 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:53:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:29 smithi121 ceph-mon[30729]: 1.1a scrub starts 2024-09-11T03:53:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:29 smithi121 ceph-mon[30729]: 1.1a scrub ok 2024-09-11T03:53:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:29 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3207138784' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:53:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:29 smithi121 ceph-mon[30729]: osdmap e970: 12 total, 12 up, 12 in 2024-09-11T03:53:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:29 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:53:29.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:29 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:53:30.198 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-11T03:53:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:30 smithi175 ceph-mon[30894]: pgmap v1621: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:30 smithi175 ceph-mon[30894]: osdmap e971: 12 total, 12 up, 12 in 2024-09-11T03:53:30.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:30 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3074474493' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:53:30.501 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:30 smithi084 ceph-mon[27254]: pgmap v1621: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:30.501 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:30 smithi084 ceph-mon[27254]: osdmap e971: 12 total, 12 up, 12 in 2024-09-11T03:53:30.501 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:30 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3074474493' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:53:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:30 smithi121 ceph-mon[30729]: pgmap v1621: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:30 smithi121 ceph-mon[30729]: osdmap e971: 12 total, 12 up, 12 in 2024-09-11T03:53:30.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:30 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3074474493' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-11T03:53:30.915 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-11T03:53:31.160 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:31 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3074474493' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:53:31.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:31 smithi175 ceph-mon[30894]: osdmap e972: 12 total, 12 up, 12 in 2024-09-11T03:53:31.587 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:31 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3074474493' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:53:31.587 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:31 smithi084 ceph-mon[27254]: osdmap e972: 12 total, 12 up, 12 in 2024-09-11T03:53:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:31 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3074474493' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-11T03:53:31.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:31 smithi121 ceph-mon[30729]: osdmap e972: 12 total, 12 up, 12 in 2024-09-11T03:53:32.211 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T03:53:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:32 smithi175 ceph-mon[30894]: pgmap v1624: 31 pgs: 31 active+clean; 577 KiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:32.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:32 smithi175 ceph-mon[30894]: 1.16 scrub starts 2024-09-11T03:53:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:32 smithi175 ceph-mon[30894]: 1.16 scrub ok 2024-09-11T03:53:32.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:32 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1313914543' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:53:32.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:32 smithi084 ceph-mon[27254]: pgmap v1624: 31 pgs: 31 active+clean; 577 KiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:32.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:32 smithi084 ceph-mon[27254]: 1.16 scrub starts 2024-09-11T03:53:32.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:32 smithi084 ceph-mon[27254]: 1.16 scrub ok 2024-09-11T03:53:32.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:32 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1313914543' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:53:32.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:32 smithi121 ceph-mon[30729]: pgmap v1624: 31 pgs: 31 active+clean; 577 KiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:32.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:32 smithi121 ceph-mon[30729]: 1.16 scrub starts 2024-09-11T03:53:32.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:32 smithi121 ceph-mon[30729]: 1.16 scrub ok 2024-09-11T03:53:32.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:32 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1313914543' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:53:32.896 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-11T03:53:33.139 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:33.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:33 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1313914543' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:53:33.478 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:33 smithi175 ceph-mon[30894]: osdmap e973: 12 total, 12 up, 12 in 2024-09-11T03:53:33.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:33 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1313914543' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:53:33.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:33 smithi084 ceph-mon[27254]: osdmap e973: 12 total, 12 up, 12 in 2024-09-11T03:53:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:33 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1313914543' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:53:33.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:33 smithi121 ceph-mon[30729]: osdmap e973: 12 total, 12 up, 12 in 2024-09-11T03:53:34.233 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-11T03:53:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:34 smithi175 ceph-mon[30894]: pgmap v1626: 31 pgs: 31 active+clean; 577 KiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:34.477 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:34 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2149801096' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:53:34.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:34 smithi084 ceph-mon[27254]: pgmap v1626: 31 pgs: 31 active+clean; 577 KiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:34.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:34 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2149801096' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:53:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:34 smithi121 ceph-mon[30729]: pgmap v1626: 31 pgs: 31 active+clean; 577 KiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:34.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:34 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2149801096' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-11T03:53:34.996 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-11T03:53:35.247 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:35.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:35 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2149801096' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:53:35.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:35 smithi084 ceph-mon[27254]: osdmap e974: 12 total, 12 up, 12 in 2024-09-11T03:53:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:35 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2149801096' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:53:35.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:35 smithi121 ceph-mon[30729]: osdmap e974: 12 total, 12 up, 12 in 2024-09-11T03:53:35.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:35 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2149801096' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-11T03:53:35.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:35 smithi175 ceph-mon[30894]: osdmap e974: 12 total, 12 up, 12 in 2024-09-11T03:53:36.240 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T03:53:36.578 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:36 smithi084 ceph-mon[27254]: pgmap v1628: 31 pgs: 31 active+clean; 577 KiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:36.578 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:36 smithi084 ceph-mon[27254]: osdmap e975: 12 total, 12 up, 12 in 2024-09-11T03:53:36.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/50285006' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:53:36.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:36 smithi121 ceph-mon[30729]: pgmap v1628: 31 pgs: 31 active+clean; 577 KiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:36 smithi121 ceph-mon[30729]: osdmap e975: 12 total, 12 up, 12 in 2024-09-11T03:53:36.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/50285006' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:53:36.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:36 smithi175 ceph-mon[30894]: pgmap v1628: 31 pgs: 31 active+clean; 577 KiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:36.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:36 smithi175 ceph-mon[30894]: osdmap e975: 12 total, 12 up, 12 in 2024-09-11T03:53:36.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/50285006' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:53:37.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-11T03:53:37.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/50285006' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:53:37.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:37 smithi084 ceph-mon[27254]: osdmap e976: 12 total, 12 up, 12 in 2024-09-11T03:53:37.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:37 smithi084 ceph-mon[27254]: 1.1e scrub starts 2024-09-11T03:53:37.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:37 smithi084 ceph-mon[27254]: 1.1e scrub ok 2024-09-11T03:53:37.411 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:37 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/50285006' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:53:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:37 smithi121 ceph-mon[30729]: osdmap e976: 12 total, 12 up, 12 in 2024-09-11T03:53:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:37 smithi121 ceph-mon[30729]: 1.1e scrub starts 2024-09-11T03:53:37.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:37 smithi121 ceph-mon[30729]: 1.1e scrub ok 2024-09-11T03:53:37.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:37 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/50285006' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:53:37.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:37 smithi175 ceph-mon[30894]: osdmap e976: 12 total, 12 up, 12 in 2024-09-11T03:53:37.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:37 smithi175 ceph-mon[30894]: 1.1e scrub starts 2024-09-11T03:53:37.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:37 smithi175 ceph-mon[30894]: 1.1e scrub ok 2024-09-11T03:53:38.250 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-11T03:53:38.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:38 smithi084 ceph-mon[27254]: pgmap v1631: 31 pgs: 31 active+clean; 577 KiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:38.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:38 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1118689420' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:53:38.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:38 smithi121 ceph-mon[30729]: pgmap v1631: 31 pgs: 31 active+clean; 577 KiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:38.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:38 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1118689420' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:53:38.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:38 smithi175 ceph-mon[30894]: pgmap v1631: 31 pgs: 31 active+clean; 577 KiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:38.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:38 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1118689420' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-11T03:53:39.054 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-11T03:53:39.299 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:39.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:39 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:53:39.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:39 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:53:39.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:39 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1118689420' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:53:39.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:39 smithi084 ceph-mon[27254]: osdmap e977: 12 total, 12 up, 12 in 2024-09-11T03:53:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:39 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:53:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:39 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:53:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:39 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1118689420' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:53:39.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:39 smithi121 ceph-mon[30729]: osdmap e977: 12 total, 12 up, 12 in 2024-09-11T03:53:39.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:39 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:53:39.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:39 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:53:39.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:39 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1118689420' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-11T03:53:39.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:39 smithi175 ceph-mon[30894]: osdmap e977: 12 total, 12 up, 12 in 2024-09-11T03:53:40.267 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 1 (10000) 2024-09-11T03:53:40.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:40 smithi084 ceph-mon[27254]: pgmap v1633: 31 pgs: 31 active+clean; 577 KiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:40.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:40 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4077038139' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:53:40.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:40 smithi121 ceph-mon[30729]: pgmap v1633: 31 pgs: 31 active+clean; 577 KiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:40.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:40 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4077038139' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:53:40.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:40 smithi175 ceph-mon[30894]: pgmap v1633: 31 pgs: 31 active+clean; 577 KiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:40.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:40 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4077038139' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:53:40.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-11T03:53:41.214 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:41.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:41 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/4077038139' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:53:41.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:41 smithi084 ceph-mon[27254]: osdmap e978: 12 total, 12 up, 12 in 2024-09-11T03:53:41.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:41 smithi084 ceph-mon[27254]: 1.7 scrub starts 2024-09-11T03:53:41.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:41 smithi084 ceph-mon[27254]: 1.7 scrub ok 2024-09-11T03:53:41.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:41 smithi084 ceph-mon[27254]: 1.c scrub starts 2024-09-11T03:53:41.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:41 smithi084 ceph-mon[27254]: 1.c scrub ok 2024-09-11T03:53:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:41 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/4077038139' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:53:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:41 smithi121 ceph-mon[30729]: osdmap e978: 12 total, 12 up, 12 in 2024-09-11T03:53:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:41 smithi121 ceph-mon[30729]: 1.7 scrub starts 2024-09-11T03:53:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:41 smithi121 ceph-mon[30729]: 1.7 scrub ok 2024-09-11T03:53:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:41 smithi121 ceph-mon[30729]: 1.c scrub starts 2024-09-11T03:53:41.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:41 smithi121 ceph-mon[30729]: 1.c scrub ok 2024-09-11T03:53:41.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:41 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/4077038139' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:53:41.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:41 smithi175 ceph-mon[30894]: osdmap e978: 12 total, 12 up, 12 in 2024-09-11T03:53:41.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:41 smithi175 ceph-mon[30894]: 1.7 scrub starts 2024-09-11T03:53:41.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:41 smithi175 ceph-mon[30894]: 1.7 scrub ok 2024-09-11T03:53:41.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:41 smithi175 ceph-mon[30894]: 1.c scrub starts 2024-09-11T03:53:41.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:41 smithi175 ceph-mon[30894]: 1.c scrub ok 2024-09-11T03:53:42.273 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-11T03:53:42.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:42 smithi084 ceph-mon[27254]: pgmap v1635: 31 pgs: 31 active+clean; 577 KiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:42.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:42 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/340540411' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:53:42.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:42 smithi121 ceph-mon[30729]: pgmap v1635: 31 pgs: 31 active+clean; 577 KiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:42.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:42 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/340540411' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:53:42.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:42 smithi175 ceph-mon[30894]: pgmap v1635: 31 pgs: 31 active+clean; 577 KiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:42.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:42 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/340540411' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-11T03:53:43.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-11T03:53:43.271 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:43.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:43 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/340540411' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:53:43.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:43 smithi084 ceph-mon[27254]: osdmap e979: 12 total, 12 up, 12 in 2024-09-11T03:53:43.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:43 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/340540411' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:53:43.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:43 smithi121 ceph-mon[30729]: osdmap e979: 12 total, 12 up, 12 in 2024-09-11T03:53:43.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:43 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/340540411' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-11T03:53:43.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:43 smithi175 ceph-mon[30894]: osdmap e979: 12 total, 12 up, 12 in 2024-09-11T03:53:44.289 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T03:53:44.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:44 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:53:44.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:44 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:53:44.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:44 smithi084 ceph-mon[27254]: pgmap v1637: 31 pgs: 31 active+clean; 577 KiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:44.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:44 smithi084 ceph-mon[27254]: osdmap e980: 12 total, 12 up, 12 in 2024-09-11T03:53:44.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:44 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:53:44.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:44 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:53:44.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:44 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1765655090' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:53:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:44 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:53:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:44 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:53:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:44 smithi121 ceph-mon[30729]: pgmap v1637: 31 pgs: 31 active+clean; 577 KiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:44 smithi121 ceph-mon[30729]: osdmap e980: 12 total, 12 up, 12 in 2024-09-11T03:53:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:44 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:53:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:44 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:53:44.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:44 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1765655090' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:53:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:44 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:53:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:44 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:53:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:44 smithi175 ceph-mon[30894]: pgmap v1637: 31 pgs: 31 active+clean; 577 KiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:44.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:44 smithi175 ceph-mon[30894]: osdmap e980: 12 total, 12 up, 12 in 2024-09-11T03:53:44.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:44 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:53:44.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:44 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:53:44.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:44 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1765655090' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:53:45.070 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-11T03:53:45.313 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:45.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:45 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1765655090' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:53:45.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:45 smithi084 ceph-mon[27254]: osdmap e981: 12 total, 12 up, 12 in 2024-09-11T03:53:45.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:45 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1765655090' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:53:45.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:45 smithi121 ceph-mon[30729]: osdmap e981: 12 total, 12 up, 12 in 2024-09-11T03:53:45.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:45 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1765655090' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:53:45.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:45 smithi175 ceph-mon[30894]: osdmap e981: 12 total, 12 up, 12 in 2024-09-11T03:53:46.294 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-11T03:53:46.597 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:46 smithi084 ceph-mon[27254]: pgmap v1640: 31 pgs: 31 active+clean; 577 KiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:46.597 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:46 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/849419202' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:53:46.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:46 smithi121 ceph-mon[30729]: pgmap v1640: 31 pgs: 31 active+clean; 577 KiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:46.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:46 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/849419202' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:53:46.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:46 smithi175 ceph-mon[30894]: pgmap v1640: 31 pgs: 31 active+clean; 577 KiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:46.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:46 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/849419202' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-11T03:53:47.009 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-11T03:53:47.248 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:47.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:47 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/849419202' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:53:47.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:47 smithi084 ceph-mon[27254]: osdmap e982: 12 total, 12 up, 12 in 2024-09-11T03:53:47.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:47 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/849419202' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:53:47.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:47 smithi121 ceph-mon[30729]: osdmap e982: 12 total, 12 up, 12 in 2024-09-11T03:53:47.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:47 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/849419202' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-11T03:53:47.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:47 smithi175 ceph-mon[30894]: osdmap e982: 12 total, 12 up, 12 in 2024-09-11T03:53:48.309 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.6 to 1 (10000) 2024-09-11T03:53:48.629 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:48 smithi084 ceph-mon[27254]: pgmap v1642: 31 pgs: 31 active+clean; 577 KiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:48.629 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:48 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/712886936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-11T03:53:48.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:48 smithi121 ceph-mon[30729]: pgmap v1642: 31 pgs: 31 active+clean; 577 KiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:48.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:48 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/712886936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-11T03:53:48.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:48 smithi175 ceph-mon[30894]: pgmap v1642: 31 pgs: 31 active+clean; 577 KiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:48.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:48 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/712886936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-11T03:53:49.060 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-11T03:53:49.303 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:49.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:49 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/712886936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T03:53:49.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:49 smithi084 ceph-mon[27254]: osdmap e983: 12 total, 12 up, 12 in 2024-09-11T03:53:49.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:49 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/712886936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T03:53:49.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:49 smithi121 ceph-mon[30729]: osdmap e983: 12 total, 12 up, 12 in 2024-09-11T03:53:49.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:49 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/712886936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-11T03:53:49.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:49 smithi175 ceph-mon[30894]: osdmap e983: 12 total, 12 up, 12 in 2024-09-11T03:53:50.321 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-11T03:53:50.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:50 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:53:50.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:50 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:53:50.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:50 smithi084 ceph-mon[27254]: pgmap v1644: 31 pgs: 31 active+clean; 577 KiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:50.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:50 smithi084 ceph-mon[27254]: osdmap e984: 12 total, 12 up, 12 in 2024-09-11T03:53:50.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:50 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2709867951' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-11T03:53:50.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:50 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:53:50.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:50 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:53:50.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:50 smithi121 ceph-mon[30729]: pgmap v1644: 31 pgs: 31 active+clean; 577 KiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:50.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:50 smithi121 ceph-mon[30729]: osdmap e984: 12 total, 12 up, 12 in 2024-09-11T03:53:50.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:50 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2709867951' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-11T03:53:50.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:50 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:53:50.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:50 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:53:50.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:50 smithi175 ceph-mon[30894]: pgmap v1644: 31 pgs: 31 active+clean; 577 KiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:50.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:50 smithi175 ceph-mon[30894]: osdmap e984: 12 total, 12 up, 12 in 2024-09-11T03:53:50.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:50 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2709867951' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-11T03:53:51.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-11T03:53:51.432 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:51.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2709867951' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-11T03:53:51.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:51 smithi084 ceph-mon[27254]: osdmap e985: 12 total, 12 up, 12 in 2024-09-11T03:53:51.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2709867951' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-11T03:53:51.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:51 smithi121 ceph-mon[30729]: osdmap e985: 12 total, 12 up, 12 in 2024-09-11T03:53:51.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2709867951' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-11T03:53:51.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:51 smithi175 ceph-mon[30894]: osdmap e985: 12 total, 12 up, 12 in 2024-09-11T03:53:52.332 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2024-09-11T03:53:52.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:52 smithi084 ceph-mon[27254]: pgmap v1647: 31 pgs: 31 active+clean; 577 KiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:52.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:52 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/507854445' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:53:52.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:52 smithi121 ceph-mon[30729]: pgmap v1647: 31 pgs: 31 active+clean; 577 KiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:52.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:52 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/507854445' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:53:52.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:52 smithi175 ceph-mon[30894]: pgmap v1647: 31 pgs: 31 active+clean; 577 KiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:52.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:52 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/507854445' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:53:53.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-11T03:53:53.326 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:53.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:53 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/507854445' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:53:53.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:53 smithi084 ceph-mon[27254]: osdmap e986: 12 total, 12 up, 12 in 2024-09-11T03:53:53.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:53 smithi084 ceph-mon[27254]: 1.10 scrub starts 2024-09-11T03:53:53.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:53 smithi084 ceph-mon[27254]: 1.10 scrub ok 2024-09-11T03:53:53.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:53 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/507854445' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:53:53.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:53 smithi121 ceph-mon[30729]: osdmap e986: 12 total, 12 up, 12 in 2024-09-11T03:53:53.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:53 smithi121 ceph-mon[30729]: 1.10 scrub starts 2024-09-11T03:53:53.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:53 smithi121 ceph-mon[30729]: 1.10 scrub ok 2024-09-11T03:53:53.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:53 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/507854445' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:53:53.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:53 smithi175 ceph-mon[30894]: osdmap e986: 12 total, 12 up, 12 in 2024-09-11T03:53:53.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:53 smithi175 ceph-mon[30894]: 1.10 scrub starts 2024-09-11T03:53:53.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:53 smithi175 ceph-mon[30894]: 1.10 scrub ok 2024-09-11T03:53:54.346 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-11T03:53:54.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:54 smithi084 ceph-mon[27254]: pgmap v1649: 31 pgs: 31 active+clean; 577 KiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:54.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:54 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1274570557' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:53:54.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:54 smithi121 ceph-mon[30729]: pgmap v1649: 31 pgs: 31 active+clean; 577 KiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:54.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:54 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1274570557' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:53:54.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:54 smithi175 ceph-mon[30894]: pgmap v1649: 31 pgs: 31 active+clean; 577 KiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:54.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:54 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1274570557' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-11T03:53:55.087 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 8 1 2024-09-11T03:53:55.330 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:55.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:55 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1274570557' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:53:55.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:55 smithi084 ceph-mon[27254]: osdmap e987: 12 total, 12 up, 12 in 2024-09-11T03:53:55.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:55 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1274570557' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:53:55.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:55 smithi121 ceph-mon[30729]: osdmap e987: 12 total, 12 up, 12 in 2024-09-11T03:53:55.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:55 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1274570557' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-11T03:53:55.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:55 smithi175 ceph-mon[30894]: osdmap e987: 12 total, 12 up, 12 in 2024-09-11T03:53:56.357 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.8 to 1 (10000) 2024-09-11T03:53:56.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:56 smithi084 ceph-mon[27254]: pgmap v1651: 31 pgs: 31 active+clean; 577 KiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:56.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:56 smithi084 ceph-mon[27254]: osdmap e988: 12 total, 12 up, 12 in 2024-09-11T03:53:56.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:56 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2291858734' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:53:56.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:56 smithi121 ceph-mon[30729]: pgmap v1651: 31 pgs: 31 active+clean; 577 KiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:56.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:56 smithi121 ceph-mon[30729]: osdmap e988: 12 total, 12 up, 12 in 2024-09-11T03:53:56.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:56 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2291858734' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:53:56.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:56 smithi175 ceph-mon[30894]: pgmap v1651: 31 pgs: 31 active+clean; 577 KiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:56.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:56 smithi175 ceph-mon[30894]: osdmap e988: 12 total, 12 up, 12 in 2024-09-11T03:53:56.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:56 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2291858734' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:53:57.121 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 8 1 2024-09-11T03:53:57.366 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:57.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:57 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2291858734' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:53:57.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:57 smithi084 ceph-mon[27254]: osdmap e989: 12 total, 12 up, 12 in 2024-09-11T03:53:57.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:57 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2291858734' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:53:57.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:57 smithi121 ceph-mon[30729]: osdmap e989: 12 total, 12 up, 12 in 2024-09-11T03:53:57.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:57 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2291858734' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:53:57.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:57 smithi175 ceph-mon[30894]: osdmap e989: 12 total, 12 up, 12 in 2024-09-11T03:53:58.374 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-11T03:53:58.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:58 smithi084 ceph-mon[27254]: pgmap v1654: 31 pgs: 31 active+clean; 577 KiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:58.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:58 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:53:58.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:58 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:53:58.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:58 smithi084 ceph-mon[27254]: osdmap e990: 12 total, 12 up, 12 in 2024-09-11T03:53:58.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:58 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:53:58.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:58 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:53:58.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:58 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/867264404' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:53:58.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:58 smithi121 ceph-mon[30729]: pgmap v1654: 31 pgs: 31 active+clean; 577 KiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:58.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:58 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:53:58.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:58 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:53:58.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:58 smithi121 ceph-mon[30729]: osdmap e990: 12 total, 12 up, 12 in 2024-09-11T03:53:58.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:58 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:53:58.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:58 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:53:58.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:58 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/867264404' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:53:58.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:58 smithi175 ceph-mon[30894]: pgmap v1654: 31 pgs: 31 active+clean; 577 KiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:53:58.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:58 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:53:58.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:58 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:53:58.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:58 smithi175 ceph-mon[30894]: osdmap e990: 12 total, 12 up, 12 in 2024-09-11T03:53:58.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:58 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:53:58.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:58 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:53:58.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:58 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/867264404' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 1.0} : dispatch 2024-09-11T03:53:59.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 9 1 2024-09-11T03:53:59.437 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:53:59.558 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:59 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/867264404' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:53:59.558 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:53:59 smithi084 ceph-mon[27254]: osdmap e991: 12 total, 12 up, 12 in 2024-09-11T03:53:59.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:59 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/867264404' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:53:59.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:53:59 smithi121 ceph-mon[30729]: osdmap e991: 12 total, 12 up, 12 in 2024-09-11T03:53:59.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:59 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/867264404' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-09-11T03:53:59.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:53:59 smithi175 ceph-mon[30894]: osdmap e991: 12 total, 12 up, 12 in 2024-09-11T03:54:00.382 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.9 to 1 (10000) 2024-09-11T03:54:00.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:00 smithi084 ceph-mon[27254]: pgmap v1657: 31 pgs: 31 active+clean; 577 KiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:00.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:00 smithi084 ceph-mon[27254]: 1.9 scrub starts 2024-09-11T03:54:00.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:00 smithi084 ceph-mon[27254]: 1.9 scrub ok 2024-09-11T03:54:00.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:00 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1797287552' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:54:00.671 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:00 smithi121 ceph-mon[30729]: pgmap v1657: 31 pgs: 31 active+clean; 577 KiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:00.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:00 smithi121 ceph-mon[30729]: 1.9 scrub starts 2024-09-11T03:54:00.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:00 smithi121 ceph-mon[30729]: 1.9 scrub ok 2024-09-11T03:54:00.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:00 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1797287552' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:54:00.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:00 smithi175 ceph-mon[30894]: pgmap v1657: 31 pgs: 31 active+clean; 577 KiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:00.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:00 smithi175 ceph-mon[30894]: 1.9 scrub starts 2024-09-11T03:54:00.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:00 smithi175 ceph-mon[30894]: 1.9 scrub ok 2024-09-11T03:54:00.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:00 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1797287552' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:54:01.117 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 9 1 2024-09-11T03:54:01.358 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:01.643 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:01 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1797287552' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:54:01.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:01 smithi084 ceph-mon[27254]: osdmap e992: 12 total, 12 up, 12 in 2024-09-11T03:54:01.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:01 smithi084 ceph-mon[27254]: 1.4 scrub starts 2024-09-11T03:54:01.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:01 smithi084 ceph-mon[27254]: 1.4 scrub ok 2024-09-11T03:54:01.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:01 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1797287552' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:54:01.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:01 smithi121 ceph-mon[30729]: osdmap e992: 12 total, 12 up, 12 in 2024-09-11T03:54:01.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:01 smithi121 ceph-mon[30729]: 1.4 scrub starts 2024-09-11T03:54:01.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:01 smithi121 ceph-mon[30729]: 1.4 scrub ok 2024-09-11T03:54:01.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:01 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1797287552' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:54:01.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:01 smithi175 ceph-mon[30894]: osdmap e992: 12 total, 12 up, 12 in 2024-09-11T03:54:01.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:01 smithi175 ceph-mon[30894]: 1.4 scrub starts 2024-09-11T03:54:01.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:01 smithi175 ceph-mon[30894]: 1.4 scrub ok 2024-09-11T03:54:02.402 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-11T03:54:02.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:02 smithi084 ceph-mon[27254]: pgmap v1659: 31 pgs: 31 active+clean; 577 KiB data, 745 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:02.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:02 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:54:02.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:02 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:54:02.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:02 smithi084 ceph-mon[27254]: osdmap e993: 12 total, 12 up, 12 in 2024-09-11T03:54:02.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:02 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:54:02.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:02 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:54:02.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:02 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2790954672' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:54:02.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:02 smithi121 ceph-mon[30729]: pgmap v1659: 31 pgs: 31 active+clean; 577 KiB data, 745 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:02.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:02 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:54:02.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:02 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:54:02.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:02 smithi121 ceph-mon[30729]: osdmap e993: 12 total, 12 up, 12 in 2024-09-11T03:54:02.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:02 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:54:02.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:02 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:54:02.673 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:02 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2790954672' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:54:02.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:02 smithi175 ceph-mon[30894]: pgmap v1659: 31 pgs: 31 active+clean; 577 KiB data, 745 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:02.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:02 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:54:02.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:02 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:54:02.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:02 smithi175 ceph-mon[30894]: osdmap e993: 12 total, 12 up, 12 in 2024-09-11T03:54:02.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:02 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:54:02.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:02 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:54:02.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:02 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2790954672' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 1.0} : dispatch 2024-09-11T03:54:03.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 10 1 2024-09-11T03:54:03.383 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:03.660 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:03 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2790954672' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:54:03.660 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:03 smithi084 ceph-mon[27254]: osdmap e994: 12 total, 12 up, 12 in 2024-09-11T03:54:03.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:03 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2790954672' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:54:03.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:03 smithi121 ceph-mon[30729]: osdmap e994: 12 total, 12 up, 12 in 2024-09-11T03:54:03.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:03 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2790954672' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-09-11T03:54:03.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:03 smithi175 ceph-mon[30894]: osdmap e994: 12 total, 12 up, 12 in 2024-09-11T03:54:04.409 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.10 to 1 (10000) 2024-09-11T03:54:04.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:04 smithi121 ceph-mon[30729]: pgmap v1662: 31 pgs: 31 active+clean; 577 KiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:04.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:04 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/230156681' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:54:04.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:04 smithi175 ceph-mon[30894]: pgmap v1662: 31 pgs: 31 active+clean; 577 KiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:04.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:04 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/230156681' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:54:04.759 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:04 smithi084 ceph-mon[27254]: pgmap v1662: 31 pgs: 31 active+clean; 577 KiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:04.759 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:04 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/230156681' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:54:05.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 10 1 2024-09-11T03:54:05.460 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:05.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:05 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/230156681' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:54:05.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:05 smithi084 ceph-mon[27254]: osdmap e995: 12 total, 12 up, 12 in 2024-09-11T03:54:05.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:05 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/230156681' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:54:05.672 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:05 smithi121 ceph-mon[30729]: osdmap e995: 12 total, 12 up, 12 in 2024-09-11T03:54:05.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:05 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/230156681' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:54:05.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:05 smithi175 ceph-mon[30894]: osdmap e995: 12 total, 12 up, 12 in 2024-09-11T03:54:06.422 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-11T03:54:06.725 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:06 smithi084 ceph-mon[27254]: pgmap v1664: 31 pgs: 31 active+clean; 577 KiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:06.725 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:06 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3658378844' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:54:06.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:06 smithi175 ceph-mon[30894]: pgmap v1664: 31 pgs: 31 active+clean; 577 KiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:06.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:06 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3658378844' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:54:06.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:06 smithi121 ceph-mon[30729]: pgmap v1664: 31 pgs: 31 active+clean; 577 KiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:06.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:06 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3658378844' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 10, "weight": 1.0} : dispatch 2024-09-11T03:54:07.135 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 11 1 2024-09-11T03:54:07.381 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:07.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:07 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3658378844' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:54:07.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:07 smithi175 ceph-mon[30894]: osdmap e996: 12 total, 12 up, 12 in 2024-09-11T03:54:07.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:07 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3658378844' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:54:07.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:07 smithi084 ceph-mon[27254]: osdmap e996: 12 total, 12 up, 12 in 2024-09-11T03:54:07.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:07 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3658378844' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-09-11T03:54:07.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:07 smithi121 ceph-mon[30729]: osdmap e996: 12 total, 12 up, 12 in 2024-09-11T03:54:08.436 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.11 to 1 (10000) 2024-09-11T03:54:08.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:08 smithi175 ceph-mon[30894]: pgmap v1666: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:08.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:08 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:54:08.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:08 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:54:08.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:08 smithi175 ceph-mon[30894]: osdmap e997: 12 total, 12 up, 12 in 2024-09-11T03:54:08.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:08 smithi175 ceph-mon[30894]: 1.18 scrub starts 2024-09-11T03:54:08.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:08 smithi175 ceph-mon[30894]: 1.18 scrub ok 2024-09-11T03:54:08.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:08 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2995822190' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:54:08.731 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:08 smithi084 ceph-mon[27254]: pgmap v1666: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:08.732 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:08 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:54:08.732 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:08 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:54:08.732 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:08 smithi084 ceph-mon[27254]: osdmap e997: 12 total, 12 up, 12 in 2024-09-11T03:54:08.732 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:08 smithi084 ceph-mon[27254]: 1.18 scrub starts 2024-09-11T03:54:08.732 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:08 smithi084 ceph-mon[27254]: 1.18 scrub ok 2024-09-11T03:54:08.732 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:08 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2995822190' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:54:08.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:08 smithi121 ceph-mon[30729]: pgmap v1666: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:08.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:08 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:54:08.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:08 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:54:08.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:08 smithi121 ceph-mon[30729]: osdmap e997: 12 total, 12 up, 12 in 2024-09-11T03:54:08.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:08 smithi121 ceph-mon[30729]: 1.18 scrub starts 2024-09-11T03:54:08.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:08 smithi121 ceph-mon[30729]: 1.18 scrub ok 2024-09-11T03:54:08.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:08 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2995822190' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:54:09.166 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 11 1 2024-09-11T03:54:09.410 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:09.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:09 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2995822190' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:54:09.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:09 smithi084 ceph-mon[27254]: osdmap e998: 12 total, 12 up, 12 in 2024-09-11T03:54:09.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:09 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2995822190' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:54:09.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:09 smithi175 ceph-mon[30894]: osdmap e998: 12 total, 12 up, 12 in 2024-09-11T03:54:09.824 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:09 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2995822190' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:54:09.824 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:09 smithi121 ceph-mon[30729]: osdmap e998: 12 total, 12 up, 12 in 2024-09-11T03:54:10.455 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-11T03:54:10.722 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:10 smithi084 ceph-mon[27254]: pgmap v1669: 31 pgs: 31 active+clean; 577 KiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:10.722 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:10 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1892221323' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:54:10.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:10 smithi121 ceph-mon[30729]: pgmap v1669: 31 pgs: 31 active+clean; 577 KiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:10.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:10 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1892221323' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:54:10.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:10 smithi175 ceph-mon[30894]: pgmap v1669: 31 pgs: 31 active+clean; 577 KiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:10.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:10 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1892221323' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 1.0} : dispatch 2024-09-11T03:54:11.264 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-11T03:54:11.541 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:11.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:11 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1892221323' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:54:11.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:11 smithi175 ceph-mon[30894]: osdmap e999: 12 total, 12 up, 12 in 2024-09-11T03:54:11.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:11 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:54:11.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:11 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1892221323' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:54:11.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:11 smithi084 ceph-mon[27254]: osdmap e999: 12 total, 12 up, 12 in 2024-09-11T03:54:11.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:11 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:54:11.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:11 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1892221323' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-09-11T03:54:11.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:11 smithi121 ceph-mon[30729]: osdmap e999: 12 total, 12 up, 12 in 2024-09-11T03:54:11.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:11 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:54:12.622 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T03:54:12.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:12 smithi175 ceph-mon[30894]: pgmap v1671: 31 pgs: 31 active+clean; 577 KiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:12.727 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:12 smithi175 ceph-mon[30894]: 1.3 scrub starts 2024-09-11T03:54:12.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:12 smithi175 ceph-mon[30894]: 1.3 scrub ok 2024-09-11T03:54:12.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:12 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:54:12.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:12 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:54:12.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:12 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:54:12.728 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:12 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2299820446' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:54:12.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:12 smithi084 ceph-mon[27254]: pgmap v1671: 31 pgs: 31 active+clean; 577 KiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:12.897 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:12 smithi084 ceph-mon[27254]: 1.3 scrub starts 2024-09-11T03:54:12.897 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:12 smithi084 ceph-mon[27254]: 1.3 scrub ok 2024-09-11T03:54:12.897 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:12 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:54:12.897 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:12 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:54:12.897 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:12 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:54:12.897 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:12 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2299820446' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:54:12.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:12 smithi121 ceph-mon[30729]: pgmap v1671: 31 pgs: 31 active+clean; 577 KiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:12.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:12 smithi121 ceph-mon[30729]: 1.3 scrub starts 2024-09-11T03:54:12.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:12 smithi121 ceph-mon[30729]: 1.3 scrub ok 2024-09-11T03:54:12.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:12 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-11T03:54:12.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:12 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-11T03:54:12.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:12 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' 2024-09-11T03:54:12.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:12 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2299820446' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:54:13.380 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-11T03:54:13.629 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:13.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:13 smithi084 ceph-mon[27254]: 1.f scrub starts 2024-09-11T03:54:13.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:13 smithi084 ceph-mon[27254]: 1.f scrub ok 2024-09-11T03:54:13.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:13 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2299820446' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:54:13.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:13 smithi084 ceph-mon[27254]: osdmap e1000: 12 total, 12 up, 12 in 2024-09-11T03:54:13.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:13 smithi121 ceph-mon[30729]: 1.f scrub starts 2024-09-11T03:54:13.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:13 smithi121 ceph-mon[30729]: 1.f scrub ok 2024-09-11T03:54:13.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:13 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2299820446' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:54:13.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:13 smithi121 ceph-mon[30729]: osdmap e1000: 12 total, 12 up, 12 in 2024-09-11T03:54:13.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:13 smithi175 ceph-mon[30894]: 1.f scrub starts 2024-09-11T03:54:13.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:13 smithi175 ceph-mon[30894]: 1.f scrub ok 2024-09-11T03:54:13.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:13 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2299820446' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:54:13.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:13 smithi175 ceph-mon[30894]: osdmap e1000: 12 total, 12 up, 12 in 2024-09-11T03:54:14.628 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-11T03:54:14.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:14 smithi084 ceph-mon[27254]: pgmap v1673: 31 pgs: 31 active+clean; 577 KiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:14.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:14 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/242519691' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:54:14.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:14 smithi121 ceph-mon[30729]: pgmap v1673: 31 pgs: 31 active+clean; 577 KiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:14.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:14 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/242519691' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:54:14.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:14 smithi175 ceph-mon[30894]: pgmap v1673: 31 pgs: 31 active+clean; 577 KiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:14.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:14 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/242519691' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-11T03:54:15.418 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-11T03:54:15.418 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-11T03:54:15.418 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-11T03:54:15.418 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-11T03:54:15.418 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-11T03:54:15.419 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-11T03:54:15.664 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:15.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:15 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/242519691' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:54:15.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:15 smithi084 ceph-mon[27254]: osdmap e1001: 12 total, 12 up, 12 in 2024-09-11T03:54:15.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:15 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/242519691' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:54:15.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:15 smithi121 ceph-mon[30729]: osdmap e1001: 12 total, 12 up, 12 in 2024-09-11T03:54:15.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:15 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/242519691' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-11T03:54:15.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:15 smithi175 ceph-mon[30894]: osdmap e1001: 12 total, 12 up, 12 in 2024-09-11T03:54:16.345 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:54:16.345 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":1001,"fsid":"10fe7946-6fee-11ef-bcea-c7b262605968","created":"2024-09-11T03:30:30.532652+0000","modified":"2024-09-11T03:54:14.627617+0000","last_up_change":"2024-09-11T03:53:21.137944+0000","last_in_change":"2024-09-11T03:53:24.153658+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T03:33:42.916933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"220","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.940000057220459,"score_stable":1.940000057220459,"optimal_score":1,"raw_score_acting":1.940000057220459,"raw_score_stable":1.940000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9a431e48-e46b-44f9-bfa7-cb56c1484e1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":939,"up_from":965,"up_thru":967,"down_at":939,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":3383145523},{"type":"v1","addr":"172.21.15.84:6803","nonce":3383145523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":3383145523},{"type":"v1","addr":"172.21.15.84:6805","nonce":3383145523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":3383145523},{"type":"v1","addr":"172.21.15.84:6809","nonce":3383145523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":3383145523},{"type":"v1","addr":"172.21.15.84:6807","nonce":3383145523}]},"public_addr":"172.21.15.84:6803/3383145523","cluster_addr":"172.21.15.84:6805/3383145523","heartbeat_back_addr":"172.21.15.84:6809/3383145523","heartbeat_front_addr":"172.21.15.84:6807/3383145523","state":["exists","up"]},{"osd":1,"uuid":"62fd6e25-3508-40b4-a0f4-f7de137e3bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":967,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6811","nonce":3138939249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6813","nonce":3138939249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6817","nonce":3138939249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6815","nonce":3138939249}]},"public_addr":"172.21.15.84:6811/3138939249","cluster_addr":"172.21.15.84:6813/3138939249","heartbeat_back_addr":"172.21.15.84:6817/3138939249","heartbeat_front_addr":"172.21.15.84:6815/3138939249","state":["exists","up"]},{"osd":2,"uuid":"ff351425-4715-49dc-be49-f27bf205d57e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":974,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6819","nonce":3001811591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6821","nonce":3001811591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6825","nonce":3001811591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6823","nonce":3001811591}]},"public_addr":"172.21.15.84:6819/3001811591","cluster_addr":"172.21.15.84:6821/3001811591","heartbeat_back_addr":"172.21.15.84:6825/3001811591","heartbeat_front_addr":"172.21.15.84:6823/3001811591","state":["exists","up"]},{"osd":3,"uuid":"2518a93e-c612-4731-91ce-5685c25ffa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":977,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6827","nonce":302512129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6829","nonce":302512129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6833","nonce":302512129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6831","nonce":302512129}]},"public_addr":"172.21.15.84:6827/302512129","cluster_addr":"172.21.15.84:6829/302512129","heartbeat_back_addr":"172.21.15.84:6833/302512129","heartbeat_front_addr":"172.21.15.84:6831/302512129","state":["exists","up"]},{"osd":4,"uuid":"d12f3c2d-95cc-4276-a7e1-bee982d58948","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":979,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6801","nonce":3873214513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6803","nonce":3873214513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6807","nonce":3873214513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6805","nonce":3873214513}]},"public_addr":"172.21.15.121:6801/3873214513","cluster_addr":"172.21.15.121:6803/3873214513","heartbeat_back_addr":"172.21.15.121:6807/3873214513","heartbeat_front_addr":"172.21.15.121:6805/3873214513","state":["exists","up"]},{"osd":5,"uuid":"32164dd5-7e8f-4925-8526-bbb1e33e3364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":940,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6809","nonce":918612194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6811","nonce":918612194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6815","nonce":918612194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6813","nonce":918612194}]},"public_addr":"172.21.15.121:6809/918612194","cluster_addr":"172.21.15.121:6811/918612194","heartbeat_back_addr":"172.21.15.121:6815/918612194","heartbeat_front_addr":"172.21.15.121:6813/918612194","state":["exists","up"]},{"osd":6,"uuid":"3f107a88-803f-4cd8-804a-241310c282ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":967,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6817","nonce":3384813363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6819","nonce":3384813363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6823","nonce":3384813363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6821","nonce":3384813363}]},"public_addr":"172.21.15.121:6817/3384813363","cluster_addr":"172.21.15.121:6819/3384813363","heartbeat_back_addr":"172.21.15.121:6823/3384813363","heartbeat_front_addr":"172.21.15.121:6821/3384813363","state":["exists","up"]},{"osd":7,"uuid":"dfee7f95-1fb9-4493-9907-cb10d1837bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":987,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6825","nonce":449222054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6827","nonce":449222054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6831","nonce":449222054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6829","nonce":449222054}]},"public_addr":"172.21.15.121:6825/449222054","cluster_addr":"172.21.15.121:6827/449222054","heartbeat_back_addr":"172.21.15.121:6831/449222054","heartbeat_front_addr":"172.21.15.121:6829/449222054","state":["exists","up"]},{"osd":8,"uuid":"fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":965,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6801","nonce":2660128384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6803","nonce":2660128384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6807","nonce":2660128384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6805","nonce":2660128384}]},"public_addr":"172.21.15.175:6801/2660128384","cluster_addr":"172.21.15.175:6803/2660128384","heartbeat_back_addr":"172.21.15.175:6807/2660128384","heartbeat_front_addr":"172.21.15.175:6805/2660128384","state":["exists","up"]},{"osd":9,"uuid":"50162e42-0443-41d1-8709-a0db61c087da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":967,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6809","nonce":1270991649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6811","nonce":1270991649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6815","nonce":1270991649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6813","nonce":1270991649}]},"public_addr":"172.21.15.175:6809/1270991649","cluster_addr":"172.21.15.175:6811/1270991649","heartbeat_back_addr":"172.21.15.175:6815/1270991649","heartbeat_front_addr":"172.21.15.175:6813/1270991649","state":["exists","up"]},{"osd":10,"uuid":"4b9426e1-bac8-47d3-ad58-54b89e8d1eb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":965,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6817","nonce":3877631021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6819","nonce":3877631021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6823","nonce":3877631021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6821","nonce":3877631021}]},"public_addr":"172.21.15.175:6817/3877631021","cluster_addr":"172.21.15.175:6819/3877631021","heartbeat_back_addr":"172.21.15.175:6823/3877631021","heartbeat_front_addr":"172.21.15.175:6821/3877631021","state":["exists","up"]},{"osd":11,"uuid":"cd1a28a1-413c-4274-9876-e1bdb7703be6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":965,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6825","nonce":1150051236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6827","nonce":1150051236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6831","nonce":1150051236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6829","nonce":1150051236}]},"public_addr":"172.21.15.175:6825/1150051236","cluster_addr":"172.21.15.175:6827/1150051236","heartbeat_back_addr":"172.21.15.175:6831/1150051236","heartbeat_front_addr":"172.21.15.175:6829/1150051236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:53:48.724816+0000","dead_epoch":938},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:52:51.904918+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:53:37.535379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:54:06.955587+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:53:25.386314+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:53:43.322856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:53:28.447163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:53:42.567581+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:53:57.294833+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:54:01.309366+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:53:57.779685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:54:01.789418+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.16","osds":[11,2,0]}],"pg_upmap_items":[{"pgid":"1.4","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/108136515":"2024-09-12T03:31:26.598965+0000","172.21.15.84:0/3411515187":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6801/3365920551":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6800/3365920551":"2024-09-12T03:31:26.598965+0000","172.21.15.84:0/3885577441":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/22039254":"2024-09-12T03:31:03.077186+0000","172.21.15.84:6801/3496645948":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/4206337830":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6800/3496645948":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/1989775875":"2024-09-12T03:31:03.077186+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":351,"length":1},{"begin":373,"length":1},{"begin":376,"length":1},{"begin":403,"length":1},{"begin":426,"length":1},{"begin":434,"length":1},{"begin":439,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T03:54:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:16 smithi121 ceph-mon[30729]: pgmap v1675: 31 pgs: 31 active+clean; 577 KiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:16 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:54:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:16 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:54:16.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:16 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2828347735' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:54:16.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:16 smithi175 ceph-mon[30894]: pgmap v1675: 31 pgs: 31 active+clean; 577 KiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:16.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:16 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:54:16.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:16 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:54:16.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:16 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2828347735' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:54:17.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:16 smithi084 ceph-mon[27254]: pgmap v1675: 31 pgs: 31 active+clean; 577 KiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:17.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:16 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:54:17.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:16 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:54:17.020 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:16 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2828347735' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:54:17.049 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-11T03:54:17.049 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-11T03:54:17.293 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:17.715 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:17 smithi084 ceph-mon[27254]: osdmap e1002: 12 total, 12 up, 12 in 2024-09-11T03:54:17.715 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:17 smithi084 ceph-mon[27254]: 1.15 scrub starts 2024-09-11T03:54:17.715 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:17 smithi084 ceph-mon[27254]: 1.15 scrub ok 2024-09-11T03:54:17.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:17 smithi121 ceph-mon[30729]: osdmap e1002: 12 total, 12 up, 12 in 2024-09-11T03:54:17.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:17 smithi121 ceph-mon[30729]: 1.15 scrub starts 2024-09-11T03:54:17.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:17 smithi121 ceph-mon[30729]: 1.15 scrub ok 2024-09-11T03:54:17.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:17 smithi175 ceph-mon[30894]: osdmap e1002: 12 total, 12 up, 12 in 2024-09-11T03:54:17.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:17 smithi175 ceph-mon[30894]: 1.15 scrub starts 2024-09-11T03:54:17.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:17 smithi175 ceph-mon[30894]: 1.15 scrub ok 2024-09-11T03:54:17.982 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:54:17.982 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":1002,"fsid":"10fe7946-6fee-11ef-bcea-c7b262605968","created":"2024-09-11T03:30:30.532652+0000","modified":"2024-09-11T03:54:16.643379+0000","last_up_change":"2024-09-11T03:53:21.137944+0000","last_in_change":"2024-09-11T03:53:24.153658+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T03:33:42.916933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"220","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.940000057220459,"score_stable":1.940000057220459,"optimal_score":1,"raw_score_acting":1.940000057220459,"raw_score_stable":1.940000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9a431e48-e46b-44f9-bfa7-cb56c1484e1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":939,"up_from":965,"up_thru":967,"down_at":939,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":3383145523},{"type":"v1","addr":"172.21.15.84:6803","nonce":3383145523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":3383145523},{"type":"v1","addr":"172.21.15.84:6805","nonce":3383145523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":3383145523},{"type":"v1","addr":"172.21.15.84:6809","nonce":3383145523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":3383145523},{"type":"v1","addr":"172.21.15.84:6807","nonce":3383145523}]},"public_addr":"172.21.15.84:6803/3383145523","cluster_addr":"172.21.15.84:6805/3383145523","heartbeat_back_addr":"172.21.15.84:6809/3383145523","heartbeat_front_addr":"172.21.15.84:6807/3383145523","state":["exists","up"]},{"osd":1,"uuid":"62fd6e25-3508-40b4-a0f4-f7de137e3bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":967,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6811","nonce":3138939249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6813","nonce":3138939249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6817","nonce":3138939249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3138939249},{"type":"v1","addr":"172.21.15.84:6815","nonce":3138939249}]},"public_addr":"172.21.15.84:6811/3138939249","cluster_addr":"172.21.15.84:6813/3138939249","heartbeat_back_addr":"172.21.15.84:6817/3138939249","heartbeat_front_addr":"172.21.15.84:6815/3138939249","state":["exists","up"]},{"osd":2,"uuid":"ff351425-4715-49dc-be49-f27bf205d57e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":974,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6819","nonce":3001811591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6821","nonce":3001811591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6825","nonce":3001811591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":3001811591},{"type":"v1","addr":"172.21.15.84:6823","nonce":3001811591}]},"public_addr":"172.21.15.84:6819/3001811591","cluster_addr":"172.21.15.84:6821/3001811591","heartbeat_back_addr":"172.21.15.84:6825/3001811591","heartbeat_front_addr":"172.21.15.84:6823/3001811591","state":["exists","up"]},{"osd":3,"uuid":"2518a93e-c612-4731-91ce-5685c25ffa9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":977,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6827","nonce":302512129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6829","nonce":302512129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6833","nonce":302512129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":302512129},{"type":"v1","addr":"172.21.15.84:6831","nonce":302512129}]},"public_addr":"172.21.15.84:6827/302512129","cluster_addr":"172.21.15.84:6829/302512129","heartbeat_back_addr":"172.21.15.84:6833/302512129","heartbeat_front_addr":"172.21.15.84:6831/302512129","state":["exists","up"]},{"osd":4,"uuid":"d12f3c2d-95cc-4276-a7e1-bee982d58948","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":979,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6801","nonce":3873214513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6803","nonce":3873214513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6807","nonce":3873214513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3873214513},{"type":"v1","addr":"172.21.15.121:6805","nonce":3873214513}]},"public_addr":"172.21.15.121:6801/3873214513","cluster_addr":"172.21.15.121:6803/3873214513","heartbeat_back_addr":"172.21.15.121:6807/3873214513","heartbeat_front_addr":"172.21.15.121:6805/3873214513","state":["exists","up"]},{"osd":5,"uuid":"32164dd5-7e8f-4925-8526-bbb1e33e3364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":940,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6809","nonce":918612194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6811","nonce":918612194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6815","nonce":918612194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":918612194},{"type":"v1","addr":"172.21.15.121:6813","nonce":918612194}]},"public_addr":"172.21.15.121:6809/918612194","cluster_addr":"172.21.15.121:6811/918612194","heartbeat_back_addr":"172.21.15.121:6815/918612194","heartbeat_front_addr":"172.21.15.121:6813/918612194","state":["exists","up"]},{"osd":6,"uuid":"3f107a88-803f-4cd8-804a-241310c282ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":967,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6817","nonce":3384813363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6819","nonce":3384813363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6823","nonce":3384813363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3384813363},{"type":"v1","addr":"172.21.15.121:6821","nonce":3384813363}]},"public_addr":"172.21.15.121:6817/3384813363","cluster_addr":"172.21.15.121:6819/3384813363","heartbeat_back_addr":"172.21.15.121:6823/3384813363","heartbeat_front_addr":"172.21.15.121:6821/3384813363","state":["exists","up"]},{"osd":7,"uuid":"dfee7f95-1fb9-4493-9907-cb10d1837bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":987,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6825","nonce":449222054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6827","nonce":449222054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6831","nonce":449222054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":449222054},{"type":"v1","addr":"172.21.15.121:6829","nonce":449222054}]},"public_addr":"172.21.15.121:6825/449222054","cluster_addr":"172.21.15.121:6827/449222054","heartbeat_back_addr":"172.21.15.121:6831/449222054","heartbeat_front_addr":"172.21.15.121:6829/449222054","state":["exists","up"]},{"osd":8,"uuid":"fa9ff5ae-4c89-4e2a-a45a-3232f54aa1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":965,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6801","nonce":2660128384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6803","nonce":2660128384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6807","nonce":2660128384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2660128384},{"type":"v1","addr":"172.21.15.175:6805","nonce":2660128384}]},"public_addr":"172.21.15.175:6801/2660128384","cluster_addr":"172.21.15.175:6803/2660128384","heartbeat_back_addr":"172.21.15.175:6807/2660128384","heartbeat_front_addr":"172.21.15.175:6805/2660128384","state":["exists","up"]},{"osd":9,"uuid":"50162e42-0443-41d1-8709-a0db61c087da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":967,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6809","nonce":1270991649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6811","nonce":1270991649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6815","nonce":1270991649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1270991649},{"type":"v1","addr":"172.21.15.175:6813","nonce":1270991649}]},"public_addr":"172.21.15.175:6809/1270991649","cluster_addr":"172.21.15.175:6811/1270991649","heartbeat_back_addr":"172.21.15.175:6815/1270991649","heartbeat_front_addr":"172.21.15.175:6813/1270991649","state":["exists","up"]},{"osd":10,"uuid":"4b9426e1-bac8-47d3-ad58-54b89e8d1eb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":965,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6817","nonce":3877631021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6819","nonce":3877631021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6823","nonce":3877631021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":3877631021},{"type":"v1","addr":"172.21.15.175:6821","nonce":3877631021}]},"public_addr":"172.21.15.175:6817/3877631021","cluster_addr":"172.21.15.175:6819/3877631021","heartbeat_back_addr":"172.21.15.175:6823/3877631021","heartbeat_front_addr":"172.21.15.175:6821/3877631021","state":["exists","up"]},{"osd":11,"uuid":"cd1a28a1-413c-4274-9876-e1bdb7703be6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":965,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6825","nonce":1150051236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6827","nonce":1150051236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6831","nonce":1150051236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1150051236},{"type":"v1","addr":"172.21.15.175:6829","nonce":1150051236}]},"public_addr":"172.21.15.175:6825/1150051236","cluster_addr":"172.21.15.175:6827/1150051236","heartbeat_back_addr":"172.21.15.175:6831/1150051236","heartbeat_front_addr":"172.21.15.175:6829/1150051236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:53:48.724816+0000","dead_epoch":938},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:54:15.836265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:53:37.535379+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:54:06.955587+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:53:25.386314+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:53:43.322856+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:53:28.447163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:53:42.567581+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:53:57.294833+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:54:01.309366+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:53:57.779685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T03:54:01.789418+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.16","osds":[11,2,0]}],"pg_upmap_items":[{"pgid":"1.4","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/108136515":"2024-09-12T03:31:26.598965+0000","172.21.15.84:0/3411515187":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6801/3365920551":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6800/3365920551":"2024-09-12T03:31:26.598965+0000","172.21.15.84:0/3885577441":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/22039254":"2024-09-12T03:31:03.077186+0000","172.21.15.84:6801/3496645948":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/4206337830":"2024-09-12T03:31:26.598965+0000","172.21.15.84:6800/3496645948":"2024-09-12T03:31:03.077186+0000","172.21.15.84:0/1989775875":"2024-09-12T03:31:03.077186+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":351,"length":1},{"begin":373,"length":1},{"begin":376,"length":1},{"begin":403,"length":1},{"begin":426,"length":1},{"begin":434,"length":1},{"begin":439,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T03:54:18.722 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-11T03:54:18.722 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-11T03:54:18.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-11T03:54:18.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-11T03:54:18.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-11T03:54:18.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-11T03:54:18.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-11T03:54:18.725 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-11T03:54:18.725 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-09-11T03:54:18.726 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-09-11T03:54:18.726 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-09-11T03:54:18.727 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-09-11T03:54:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:18 smithi121 ceph-mon[30729]: pgmap v1677: 31 pgs: 31 active+clean; 577 KiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:18 smithi121 ceph-mon[30729]: 1.6 scrub starts 2024-09-11T03:54:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:18 smithi121 ceph-mon[30729]: 1.6 scrub ok 2024-09-11T03:54:18.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:18 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2033292632' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:54:18.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:18 smithi175 ceph-mon[30894]: pgmap v1677: 31 pgs: 31 active+clean; 577 KiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:18.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:18 smithi175 ceph-mon[30894]: 1.6 scrub starts 2024-09-11T03:54:18.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:18 smithi175 ceph-mon[30894]: 1.6 scrub ok 2024-09-11T03:54:18.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:18 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2033292632' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:54:19.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:18 smithi084 ceph-mon[27254]: pgmap v1677: 31 pgs: 31 active+clean; 577 KiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:19.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:18 smithi084 ceph-mon[27254]: 1.6 scrub starts 2024-09-11T03:54:19.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:18 smithi084 ceph-mon[27254]: 1.6 scrub ok 2024-09-11T03:54:19.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:18 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2033292632' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-11T03:54:19.384 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:19.415 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:19.433 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:19.457 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:19.457 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:19.470 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:19.699 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:19.706 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:19.706 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:19.710 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:20.124 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:20.915 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:20 smithi084 ceph-mon[27254]: pgmap v1678: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:20.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:20 smithi121 ceph-mon[30729]: pgmap v1678: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:20.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:20 smithi175 ceph-mon[30894]: pgmap v1678: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:22.819 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:22 smithi084 ceph-mon[27254]: pgmap v1679: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:22.921 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:22 smithi121 ceph-mon[30729]: pgmap v1679: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:22.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:22 smithi175 ceph-mon[30894]: pgmap v1679: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:23.615 INFO:teuthology.orchestra.run.smithi084.stdout:4144643440654 2024-09-11T03:54:23.616 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-11T03:54:24.754 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:24 smithi084 ceph-mon[27254]: pgmap v1680: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:24.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:24 smithi121 ceph-mon[30729]: pgmap v1680: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:24.937 INFO:teuthology.orchestra.run.smithi084.stdout:111669149944 2024-09-11T03:54:24.937 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-11T03:54:24.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:24 smithi175 ceph-mon[30894]: pgmap v1680: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:25.831 INFO:teuthology.orchestra.run.smithi084.stdout:55834575107 2024-09-11T03:54:25.832 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-11T03:54:26.480 INFO:teuthology.orchestra.run.smithi084.stdout:223338299619 2024-09-11T03:54:26.481 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-11T03:54:26.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:26 smithi084 ceph-mon[27254]: pgmap v1681: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:26.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:26 smithi121 ceph-mon[30729]: pgmap v1681: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:26.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:26 smithi175 ceph-mon[30894]: pgmap v1681: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:27.206 INFO:teuthology.orchestra.run.smithi084.stdout:77309411581 2024-09-11T03:54:27.207 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-11T03:54:27.218 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:28.181 INFO:teuthology.orchestra.run.smithi084.stdout:133143986419 2024-09-11T03:54:28.182 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-11T03:54:28.816 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:28 smithi084 ceph-mon[27254]: pgmap v1682: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:28.816 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:28 smithi084 ceph-mon[27254]: 1.19 scrub starts 2024-09-11T03:54:28.816 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:28 smithi084 ceph-mon[27254]: 1.19 scrub ok 2024-09-11T03:54:28.842 INFO:teuthology.orchestra.run.smithi084.stdout:193273528551 2024-09-11T03:54:28.843 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-11T03:54:28.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:28 smithi121 ceph-mon[30729]: pgmap v1682: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:28.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:28 smithi121 ceph-mon[30729]: 1.19 scrub starts 2024-09-11T03:54:28.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:28 smithi121 ceph-mon[30729]: 1.19 scrub ok 2024-09-11T03:54:28.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:28 smithi175 ceph-mon[30894]: pgmap v1682: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:28.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:28 smithi175 ceph-mon[30894]: 1.19 scrub starts 2024-09-11T03:54:28.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:28 smithi175 ceph-mon[30894]: 1.19 scrub ok 2024-09-11T03:54:29.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:29 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:54:29.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:29 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:54:29.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:29 smithi175 ceph-mon[30894]: osdmap e1003: 12 total, 12 up, 12 in 2024-09-11T03:54:30.048 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:29 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:54:30.048 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:29 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:54:30.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:29 smithi084 ceph-mon[27254]: osdmap e1003: 12 total, 12 up, 12 in 2024-09-11T03:54:30.077 INFO:teuthology.orchestra.run.smithi084.stdout:257698037985 2024-09-11T03:54:30.077 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-09-11T03:54:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:29 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:54:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:29 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:54:30.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:29 smithi121 ceph-mon[30729]: osdmap e1003: 12 total, 12 up, 12 in 2024-09-11T03:54:30.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:30 smithi084 ceph-mon[27254]: pgmap v1684: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:30.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:30 smithi084 ceph-mon[27254]: 1.a scrub starts 2024-09-11T03:54:30.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:30 smithi084 ceph-mon[27254]: 1.a scrub ok 2024-09-11T03:54:30.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:30 smithi084 ceph-mon[27254]: 1.0 scrub starts 2024-09-11T03:54:30.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:30 smithi084 ceph-mon[27254]: 1.0 scrub ok 2024-09-11T03:54:30.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:30 smithi175 ceph-mon[30894]: pgmap v1684: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:30.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:30 smithi175 ceph-mon[30894]: 1.a scrub starts 2024-09-11T03:54:30.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:30 smithi175 ceph-mon[30894]: 1.a scrub ok 2024-09-11T03:54:30.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:30 smithi175 ceph-mon[30894]: 1.0 scrub starts 2024-09-11T03:54:30.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:30 smithi175 ceph-mon[30894]: 1.0 scrub ok 2024-09-11T03:54:31.011 INFO:teuthology.orchestra.run.smithi084.stdout:163208757486 2024-09-11T03:54:31.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-11T03:54:31.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:30 smithi121 ceph-mon[30729]: pgmap v1684: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:31.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:30 smithi121 ceph-mon[30729]: 1.a scrub starts 2024-09-11T03:54:31.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:30 smithi121 ceph-mon[30729]: 1.a scrub ok 2024-09-11T03:54:31.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:30 smithi121 ceph-mon[30729]: 1.0 scrub starts 2024-09-11T03:54:31.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:30 smithi121 ceph-mon[30729]: 1.0 scrub ok 2024-09-11T03:54:31.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:31 smithi084 ceph-mon[27254]: 1.d scrub starts 2024-09-11T03:54:31.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:31 smithi084 ceph-mon[27254]: 1.d scrub ok 2024-09-11T03:54:31.928 INFO:teuthology.orchestra.run.smithi084.stdout:356482285776 2024-09-11T03:54:31.928 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-09-11T03:54:31.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:31 smithi175 ceph-mon[30894]: 1.d scrub starts 2024-09-11T03:54:31.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:31 smithi175 ceph-mon[30894]: 1.d scrub ok 2024-09-11T03:54:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:31 smithi121 ceph-mon[30729]: 1.d scrub starts 2024-09-11T03:54:32.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:31 smithi121 ceph-mon[30729]: 1.d scrub ok 2024-09-11T03:54:32.428 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:32.430 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:32.431 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:32.433 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:32.434 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:32.483 INFO:teuthology.orchestra.run.smithi084.stdout:326417514708 2024-09-11T03:54:32.484 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-09-11T03:54:32.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:32 smithi084 ceph-mon[27254]: pgmap v1685: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:32.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:32 smithi084 ceph-mon[27254]: 1.1d scrub starts 2024-09-11T03:54:32.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:32 smithi084 ceph-mon[27254]: 1.1d scrub ok 2024-09-11T03:54:32.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:32 smithi084 ceph-mon[27254]: 1.14 scrub starts 2024-09-11T03:54:32.890 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:32 smithi084 ceph-mon[27254]: 1.14 scrub ok 2024-09-11T03:54:32.925 INFO:teuthology.orchestra.run.smithi084.stdout:296352743641 2024-09-11T03:54:32.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-09-11T03:54:32.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:32 smithi175 ceph-mon[30894]: pgmap v1685: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:32.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:32 smithi175 ceph-mon[30894]: 1.1d scrub starts 2024-09-11T03:54:32.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:32 smithi175 ceph-mon[30894]: 1.1d scrub ok 2024-09-11T03:54:32.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:32 smithi175 ceph-mon[30894]: 1.14 scrub starts 2024-09-11T03:54:32.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:32 smithi175 ceph-mon[30894]: 1.14 scrub ok 2024-09-11T03:54:33.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:32 smithi121 ceph-mon[30729]: pgmap v1685: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:33.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:32 smithi121 ceph-mon[30729]: 1.1d scrub starts 2024-09-11T03:54:33.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:32 smithi121 ceph-mon[30729]: 1.1d scrub ok 2024-09-11T03:54:33.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:32 smithi121 ceph-mon[30729]: 1.14 scrub starts 2024-09-11T03:54:33.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:32 smithi121 ceph-mon[30729]: 1.14 scrub ok 2024-09-11T03:54:33.731 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:33 smithi084 ceph-mon[27254]: 1.12 scrub starts 2024-09-11T03:54:33.731 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:33 smithi084 ceph-mon[27254]: 1.12 scrub ok 2024-09-11T03:54:33.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:33 smithi175 ceph-mon[30894]: 1.12 scrub starts 2024-09-11T03:54:33.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:33 smithi175 ceph-mon[30894]: 1.12 scrub ok 2024-09-11T03:54:34.119 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:34.119 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:34.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:33 smithi121 ceph-mon[30729]: 1.12 scrub starts 2024-09-11T03:54:34.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:33 smithi121 ceph-mon[30729]: 1.12 scrub ok 2024-09-11T03:54:34.442 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:34.536 INFO:teuthology.orchestra.run.smithi084.stdout:193273528554 2024-09-11T03:54:34.613 INFO:teuthology.orchestra.run.smithi084.stdout:55834575109 2024-09-11T03:54:34.769 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:34.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:34 smithi084 ceph-mon[27254]: pgmap v1686: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:34.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:34 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3375383371' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-11T03:54:34.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:34 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1327699248' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-11T03:54:34.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:34 smithi175 ceph-mon[30894]: pgmap v1686: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:34.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:34 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3375383371' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-11T03:54:34.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:34 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1327699248' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-11T03:54:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:34 smithi121 ceph-mon[30729]: pgmap v1686: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:34 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3375383371' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-11T03:54:35.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:34 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1327699248' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-11T03:54:35.669 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575107 got 55834575109 for osd.1 2024-09-11T03:54:35.670 DEBUG:teuthology.parallel:result is None 2024-09-11T03:54:35.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:35 smithi084 ceph-mon[27254]: 1.5 scrub starts 2024-09-11T03:54:35.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:35 smithi084 ceph-mon[27254]: 1.5 scrub ok 2024-09-11T03:54:35.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:35 smithi175 ceph-mon[30894]: 1.5 scrub starts 2024-09-11T03:54:35.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:35 smithi175 ceph-mon[30894]: 1.5 scrub ok 2024-09-11T03:54:36.151 INFO:teuthology.orchestra.run.smithi084.stdout:223338299621 2024-09-11T03:54:36.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:35 smithi121 ceph-mon[30729]: 1.5 scrub starts 2024-09-11T03:54:36.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:35 smithi121 ceph-mon[30729]: 1.5 scrub ok 2024-09-11T03:54:36.634 INFO:teuthology.orchestra.run.smithi084.stdout:111669149947 2024-09-11T03:54:36.827 INFO:teuthology.orchestra.run.smithi084.stdout:133143986421 2024-09-11T03:54:36.827 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:36 smithi084 ceph-mon[27254]: pgmap v1687: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:36.827 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:36 smithi084 ceph-mon[27254]: 1.e scrub starts 2024-09-11T03:54:36.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:36 smithi084 ceph-mon[27254]: 1.e scrub ok 2024-09-11T03:54:36.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1094217207' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-11T03:54:36.828 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:36 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3266319064' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-11T03:54:36.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:36 smithi175 ceph-mon[30894]: pgmap v1687: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:36.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:36 smithi175 ceph-mon[30894]: 1.e scrub starts 2024-09-11T03:54:36.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:36 smithi175 ceph-mon[30894]: 1.e scrub ok 2024-09-11T03:54:36.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1094217207' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-11T03:54:36.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:36 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3266319064' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-11T03:54:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:36 smithi121 ceph-mon[30729]: pgmap v1687: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:36 smithi121 ceph-mon[30729]: 1.e scrub starts 2024-09-11T03:54:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:36 smithi121 ceph-mon[30729]: 1.e scrub ok 2024-09-11T03:54:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1094217207' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-11T03:54:37.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:36 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3266319064' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-11T03:54:37.586 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986419 got 133143986421 for osd.4 2024-09-11T03:54:37.586 DEBUG:teuthology.parallel:result is None 2024-09-11T03:54:37.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:37 smithi175 ceph-mon[30894]: 1.13 scrub starts 2024-09-11T03:54:37.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:37 smithi175 ceph-mon[30894]: 1.13 scrub ok 2024-09-11T03:54:37.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:37 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2636804188' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-11T03:54:37.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:37 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/427225534' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-11T03:54:37.979 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:37 smithi084 ceph-mon[27254]: 1.13 scrub starts 2024-09-11T03:54:37.979 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:37 smithi084 ceph-mon[27254]: 1.13 scrub ok 2024-09-11T03:54:37.979 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2636804188' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-11T03:54:37.979 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:37 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/427225534' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-11T03:54:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:37 smithi121 ceph-mon[30729]: 1.13 scrub starts 2024-09-11T03:54:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:37 smithi121 ceph-mon[30729]: 1.13 scrub ok 2024-09-11T03:54:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:37 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2636804188' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-11T03:54:38.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:37 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/427225534' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-11T03:54:38.370 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299619 got 223338299621 for osd.7 2024-09-11T03:54:38.371 DEBUG:teuthology.parallel:result is None 2024-09-11T03:54:38.847 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528551 got 193273528554 for osd.6 2024-09-11T03:54:38.847 DEBUG:teuthology.parallel:result is None 2024-09-11T03:54:38.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:38 smithi175 ceph-mon[30894]: pgmap v1688: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:38.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:38 smithi175 ceph-mon[30894]: 1.1a scrub starts 2024-09-11T03:54:38.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:38 smithi175 ceph-mon[30894]: 1.1a scrub ok 2024-09-11T03:54:38.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:38 smithi175 ceph-mon[30894]: 1.1 scrub starts 2024-09-11T03:54:38.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:38 smithi175 ceph-mon[30894]: 1.1 scrub ok 2024-09-11T03:54:39.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:38 smithi084 ceph-mon[27254]: pgmap v1688: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:39.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:38 smithi084 ceph-mon[27254]: 1.1a scrub starts 2024-09-11T03:54:39.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:38 smithi084 ceph-mon[27254]: 1.1a scrub ok 2024-09-11T03:54:39.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:38 smithi084 ceph-mon[27254]: 1.1 scrub starts 2024-09-11T03:54:39.067 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:38 smithi084 ceph-mon[27254]: 1.1 scrub ok 2024-09-11T03:54:39.073 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:38 smithi121 ceph-mon[30729]: pgmap v1688: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:39.073 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:38 smithi121 ceph-mon[30729]: 1.1a scrub starts 2024-09-11T03:54:39.073 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:38 smithi121 ceph-mon[30729]: 1.1a scrub ok 2024-09-11T03:54:39.073 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:38 smithi121 ceph-mon[30729]: 1.1 scrub starts 2024-09-11T03:54:39.073 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:38 smithi121 ceph-mon[30729]: 1.1 scrub ok 2024-09-11T03:54:39.162 INFO:teuthology.orchestra.run.smithi084.stdout:4144643440656 2024-09-11T03:54:39.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:39 smithi084 ceph-mon[27254]: 1.11 scrub starts 2024-09-11T03:54:39.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:39 smithi084 ceph-mon[27254]: 1.11 scrub ok 2024-09-11T03:54:39.936 INFO:tasks.cephadm.ceph_manager.ceph:need seq 4144643440654 got 4144643440656 for osd.0 2024-09-11T03:54:39.936 DEBUG:teuthology.parallel:result is None 2024-09-11T03:54:39.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:39 smithi175 ceph-mon[30894]: 1.11 scrub starts 2024-09-11T03:54:39.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:39 smithi175 ceph-mon[30894]: 1.11 scrub ok 2024-09-11T03:54:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:39 smithi121 ceph-mon[30729]: 1.11 scrub starts 2024-09-11T03:54:40.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:39 smithi121 ceph-mon[30729]: 1.11 scrub ok 2024-09-11T03:54:40.634 INFO:teuthology.orchestra.run.smithi084.stdout:163208757489 2024-09-11T03:54:40.847 INFO:teuthology.orchestra.run.smithi084.stdout:257698037987 2024-09-11T03:54:40.847 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:40 smithi084 ceph-mon[27254]: pgmap v1689: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:40.847 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:40 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:54:40.847 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:40 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:54:40.847 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:40 smithi084 ceph-mon[27254]: osdmap e1004: 12 total, 12 up, 12 in 2024-09-11T03:54:40.848 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:40 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3611204331' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-11T03:54:40.848 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:40 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/735765844' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-11T03:54:40.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:40 smithi175 ceph-mon[30894]: pgmap v1689: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:40.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:40 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:54:40.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:40 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:54:40.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:40 smithi175 ceph-mon[30894]: osdmap e1004: 12 total, 12 up, 12 in 2024-09-11T03:54:40.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:40 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3611204331' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-11T03:54:40.978 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:40 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/735765844' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-11T03:54:41.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:40 smithi121 ceph-mon[30729]: pgmap v1689: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:41.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:40 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:54:41.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:40 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:54:41.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:40 smithi121 ceph-mon[30729]: osdmap e1004: 12 total, 12 up, 12 in 2024-09-11T03:54:41.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:40 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3611204331' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-11T03:54:41.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:40 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/735765844' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-11T03:54:41.402 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757486 got 163208757489 for osd.5 2024-09-11T03:54:41.402 DEBUG:teuthology.parallel:result is None 2024-09-11T03:54:41.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:41 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/2056472081' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-11T03:54:42.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:41 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/2056472081' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-11T03:54:42.048 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037985 got 257698037987 for osd.8 2024-09-11T03:54:42.049 DEBUG:teuthology.parallel:result is None 2024-09-11T03:54:42.053 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:42.054 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:42.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:41 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/2056472081' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-11T03:54:42.602 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149944 got 111669149947 for osd.3 2024-09-11T03:54:42.602 DEBUG:teuthology.parallel:result is None 2024-09-11T03:54:42.667 INFO:teuthology.orchestra.run.smithi084.stdout:77309411585 2024-09-11T03:54:42.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:42 smithi084 ceph-mon[27254]: pgmap v1691: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:42.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:42 smithi084 ceph-mon[27254]: 1.8 scrub starts 2024-09-11T03:54:42.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:42 smithi084 ceph-mon[27254]: 1.8 scrub ok 2024-09-11T03:54:42.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:42 smithi175 ceph-mon[30894]: pgmap v1691: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:42.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:42 smithi175 ceph-mon[30894]: 1.8 scrub starts 2024-09-11T03:54:42.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:42 smithi175 ceph-mon[30894]: 1.8 scrub ok 2024-09-11T03:54:43.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:42 smithi121 ceph-mon[30729]: pgmap v1691: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:43.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:42 smithi121 ceph-mon[30729]: 1.8 scrub starts 2024-09-11T03:54:43.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:42 smithi121 ceph-mon[30729]: 1.8 scrub ok 2024-09-11T03:54:43.354 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411581 got 77309411585 for osd.2 2024-09-11T03:54:43.354 DEBUG:teuthology.parallel:result is None 2024-09-11T03:54:43.420 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:44.533 INFO:teuthology.orchestra.run.smithi084.stdout:356482285779 2024-09-11T03:54:44.734 INFO:teuthology.orchestra.run.smithi084.stdout:296352743644 2024-09-11T03:54:44.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:44 smithi084 ceph-mon[27254]: pgmap v1692: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:44.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:44 smithi084 ceph-mon[27254]: 1.7 scrub starts 2024-09-11T03:54:44.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:44 smithi084 ceph-mon[27254]: 1.7 scrub ok 2024-09-11T03:54:44.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:44 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/961406840' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-11T03:54:45.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:44 smithi121 ceph-mon[30729]: pgmap v1692: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:45.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:44 smithi121 ceph-mon[30729]: 1.7 scrub starts 2024-09-11T03:54:45.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:44 smithi121 ceph-mon[30729]: 1.7 scrub ok 2024-09-11T03:54:45.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:44 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/961406840' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-11T03:54:45.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:44 smithi175 ceph-mon[30894]: pgmap v1692: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:45.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:44 smithi175 ceph-mon[30894]: 1.7 scrub starts 2024-09-11T03:54:45.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:44 smithi175 ceph-mon[30894]: 1.7 scrub ok 2024-09-11T03:54:45.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:44 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/961406840' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-11T03:54:45.371 INFO:teuthology.orchestra.run.smithi084.stdout:326417514711 2024-09-11T03:54:45.916 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743641 got 296352743644 for osd.9 2024-09-11T03:54:45.916 DEBUG:teuthology.parallel:result is None 2024-09-11T03:54:46.041 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:45 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1488427261' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-11T03:54:46.041 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:45 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/3764700387' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-11T03:54:46.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:45 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1488427261' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-11T03:54:46.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:45 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/3764700387' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-11T03:54:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:45 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1488427261' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-11T03:54:46.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:45 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/3764700387' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-11T03:54:46.443 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417514708 got 326417514711 for osd.10 2024-09-11T03:54:46.443 DEBUG:teuthology.parallel:result is None 2024-09-11T03:54:46.854 INFO:tasks.cephadm.ceph_manager.ceph:need seq 356482285776 got 356482285779 for osd.11 2024-09-11T03:54:46.855 DEBUG:teuthology.parallel:result is None 2024-09-11T03:54:46.855 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T03:54:46.855 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:54:47.099 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:47.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:46 smithi084 ceph-mon[27254]: pgmap v1693: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:47.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:46 smithi121 ceph-mon[30729]: pgmap v1693: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:47.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:46 smithi175 ceph-mon[30894]: pgmap v1693: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:47.736 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:54:47.736 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:54:48.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:47 smithi084 ceph-mon[27254]: 1.17 scrub starts 2024-09-11T03:54:48.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:47 smithi084 ceph-mon[27254]: 1.17 scrub ok 2024-09-11T03:54:48.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:47 smithi121 ceph-mon[30729]: 1.17 scrub starts 2024-09-11T03:54:48.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:47 smithi121 ceph-mon[30729]: 1.17 scrub ok 2024-09-11T03:54:48.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:47 smithi175 ceph-mon[30894]: 1.17 scrub starts 2024-09-11T03:54:48.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:47 smithi175 ceph-mon[30894]: 1.17 scrub ok 2024-09-11T03:54:48.492 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1694,"stamp":"2024-09-11T03:54:46.922125+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":754616,"kb_used_data":80136,"kb_used_omap":78,"kb_used_meta":674161,"kb_avail":1124187208,"statfs":{"total":1151940427776,"available":1151167700992,"internally_reserved":0,"allocated":82059264,"data_stored":55814784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":79891,"internal_metadata":690341869},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002276"},"pg_stats":[{"pgid":"1.c","version":"20'92","reported_seq":692,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.722160+0000","last_change":"2024-09-11T03:53:40.839900+0000","last_active":"2024-09-11T03:54:39.722160+0000","last_peered":"2024-09-11T03:54:39.722160+0000","last_clean":"2024-09-11T03:54:39.722160+0000","last_became_active":"2024-09-11T03:53:40.273621+0000","last_became_peered":"2024-09-11T03:53:40.273621+0000","last_unstale":"2024-09-11T03:54:39.722160+0000","last_undegraded":"2024-09-11T03:54:39.722160+0000","last_fullsized":"2024-09-11T03:54:39.722160+0000","mapping_epoch":977,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":978,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:40.839833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:40.839833+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:49.821120+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"20'92","reported_seq":1675,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849278+0000","last_change":"2024-09-11T03:54:31.218187+0000","last_active":"2024-09-11T03:54:39.849278+0000","last_peered":"2024-09-11T03:54:39.849278+0000","last_clean":"2024-09-11T03:54:39.849278+0000","last_became_active":"2024-09-11T03:53:25.236299+0000","last_became_peered":"2024-09-11T03:53:25.236299+0000","last_unstale":"2024-09-11T03:54:39.849278+0000","last_undegraded":"2024-09-11T03:54:39.849278+0000","last_fullsized":"2024-09-11T03:54:39.849278+0000","mapping_epoch":967,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":968,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:31.218122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:31.218122+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:38.989270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"20'92","reported_seq":1360,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.721605+0000","last_change":"2024-09-11T03:54:35.274405+0000","last_active":"2024-09-11T03:54:39.721605+0000","last_peered":"2024-09-11T03:54:39.721605+0000","last_clean":"2024-09-11T03:54:39.721605+0000","last_became_active":"2024-09-11T03:53:43.289334+0000","last_became_peered":"2024-09-11T03:53:43.289334+0000","last_unstale":"2024-09-11T03:54:39.721605+0000","last_undegraded":"2024-09-11T03:54:39.721605+0000","last_fullsized":"2024-09-11T03:54:39.721605+0000","mapping_epoch":979,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":980,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:35.274333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:35.274333+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:41.555122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"20'92","reported_seq":957,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849307+0000","last_change":"2024-09-11T03:54:12.190134+0000","last_active":"2024-09-11T03:54:39.849307+0000","last_peered":"2024-09-11T03:54:39.849307+0000","last_clean":"2024-09-11T03:54:39.849307+0000","last_became_active":"2024-09-11T03:53:25.236560+0000","last_became_peered":"2024-09-11T03:53:25.236560+0000","last_unstale":"2024-09-11T03:54:39.849307+0000","last_undegraded":"2024-09-11T03:54:39.849307+0000","last_fullsized":"2024-09-11T03:54:39.849307+0000","mapping_epoch":967,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":968,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:12.190080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:12.190080+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:33.720781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":2067,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849215+0000","last_change":"2024-09-11T03:54:30.195100+0000","last_active":"2024-09-11T03:54:39.849215+0000","last_peered":"2024-09-11T03:54:39.849215+0000","last_clean":"2024-09-11T03:54:39.849215+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:54:39.849215+0000","last_undegraded":"2024-09-11T03:54:39.849215+0000","last_fullsized":"2024-09-11T03:54:39.849215+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:30.195041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:30.195041+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:56.979078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"20'92","reported_seq":1687,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.848971+0000","last_change":"2024-09-11T03:54:11.442419+0000","last_active":"2024-09-11T03:54:39.848971+0000","last_peered":"2024-09-11T03:54:39.848971+0000","last_clean":"2024-09-11T03:54:39.848971+0000","last_became_active":"2024-09-11T03:53:25.236713+0000","last_became_peered":"2024-09-11T03:53:25.236713+0000","last_unstale":"2024-09-11T03:54:39.848971+0000","last_undegraded":"2024-09-11T03:54:39.848971+0000","last_fullsized":"2024-09-11T03:54:39.848971+0000","mapping_epoch":967,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":968,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:11.442366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:11.442366+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:18.743650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"20'92","reported_seq":335,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849099+0000","last_change":"2024-09-11T03:53:55.362705+0000","last_active":"2024-09-11T03:54:39.849099+0000","last_peered":"2024-09-11T03:54:39.849099+0000","last_clean":"2024-09-11T03:54:39.849099+0000","last_became_active":"2024-09-11T03:53:55.362405+0000","last_became_peered":"2024-09-11T03:53:55.362405+0000","last_unstale":"2024-09-11T03:54:39.849099+0000","last_undegraded":"2024-09-11T03:54:39.849099+0000","last_fullsized":"2024-09-11T03:54:39.849099+0000","mapping_epoch":987,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":988,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:23.449701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:23.449701+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:48.767776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"20'92","reported_seq":1254,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849065+0000","last_change":"2024-09-11T03:54:37.504339+0000","last_active":"2024-09-11T03:54:39.849065+0000","last_peered":"2024-09-11T03:54:39.849065+0000","last_clean":"2024-09-11T03:54:39.849065+0000","last_became_active":"2024-09-11T03:53:22.151290+0000","last_became_peered":"2024-09-11T03:53:22.151290+0000","last_unstale":"2024-09-11T03:54:39.849065+0000","last_undegraded":"2024-09-11T03:54:39.849065+0000","last_fullsized":"2024-09-11T03:54:39.849065+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:37.504276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:37.504276+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:38.338864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"20'92","reported_seq":927,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849200+0000","last_change":"2024-09-11T03:53:40.459984+0000","last_active":"2024-09-11T03:54:39.849200+0000","last_peered":"2024-09-11T03:54:39.849200+0000","last_clean":"2024-09-11T03:54:39.849200+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:54:39.849200+0000","last_undegraded":"2024-09-11T03:54:39.849200+0000","last_fullsized":"2024-09-11T03:54:39.849200+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:40.459928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:40.459928+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:43.620947+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"20'92","reported_seq":1254,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849240+0000","last_change":"2024-09-11T03:54:17.510152+0000","last_active":"2024-09-11T03:54:39.849240+0000","last_peered":"2024-09-11T03:54:39.849240+0000","last_clean":"2024-09-11T03:54:39.849240+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:54:39.849240+0000","last_undegraded":"2024-09-11T03:54:39.849240+0000","last_fullsized":"2024-09-11T03:54:39.849240+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:17.510096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:17.510096+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:26.164917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.5","version":"20'92","reported_seq":1245,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849162+0000","last_change":"2024-09-11T03:54:34.533921+0000","last_active":"2024-09-11T03:54:39.849162+0000","last_peered":"2024-09-11T03:54:39.849162+0000","last_clean":"2024-09-11T03:54:39.849162+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:54:39.849162+0000","last_undegraded":"2024-09-11T03:54:39.849162+0000","last_fullsized":"2024-09-11T03:54:39.849162+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:34.533832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:34.533832+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:59.497582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"20'92","reported_seq":1835,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849102+0000","last_change":"2024-09-11T03:54:00.596942+0000","last_active":"2024-09-11T03:54:39.849102+0000","last_peered":"2024-09-11T03:54:39.849102+0000","last_clean":"2024-09-11T03:54:39.849102+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:54:39.849102+0000","last_undegraded":"2024-09-11T03:54:39.849102+0000","last_fullsized":"2024-09-11T03:54:39.849102+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:00.596883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:00.596883+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:30.391472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.b","version":"20'92","reported_seq":1648,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849102+0000","last_change":"2024-09-11T03:53:27.548967+0000","last_active":"2024-09-11T03:54:39.849102+0000","last_peered":"2024-09-11T03:54:39.849102+0000","last_clean":"2024-09-11T03:54:39.849102+0000","last_became_active":"2024-09-11T03:53:22.148230+0000","last_became_peered":"2024-09-11T03:53:22.148230+0000","last_unstale":"2024-09-11T03:54:39.849102+0000","last_undegraded":"2024-09-11T03:54:39.849102+0000","last_fullsized":"2024-09-11T03:54:39.849102+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:27.548907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:27.548907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:47.498932+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"20'92","reported_seq":2017,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849238+0000","last_change":"2024-09-11T03:54:29.949838+0000","last_active":"2024-09-11T03:54:39.849238+0000","last_peered":"2024-09-11T03:54:39.849238+0000","last_clean":"2024-09-11T03:54:39.849238+0000","last_became_active":"2024-09-11T03:53:25.237106+0000","last_became_peered":"2024-09-11T03:53:25.237106+0000","last_unstale":"2024-09-11T03:54:39.849238+0000","last_undegraded":"2024-09-11T03:54:39.849238+0000","last_fullsized":"2024-09-11T03:54:39.849238+0000","mapping_epoch":967,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":968,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:29.949736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:29.949736+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:44.952168+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"20'92","reported_seq":2054,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849074+0000","last_change":"2024-09-11T03:53:59.406218+0000","last_active":"2024-09-11T03:54:39.849074+0000","last_peered":"2024-09-11T03:54:39.849074+0000","last_clean":"2024-09-11T03:54:39.849074+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:54:39.849074+0000","last_undegraded":"2024-09-11T03:54:39.849074+0000","last_fullsized":"2024-09-11T03:54:39.849074+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:59.406151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:59.406151+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:07.726809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"20'92","reported_seq":1721,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:41.861110+0000","last_change":"2024-09-11T03:54:41.861110+0000","last_active":"2024-09-11T03:54:41.861110+0000","last_peered":"2024-09-11T03:54:41.861110+0000","last_clean":"2024-09-11T03:54:41.861110+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:54:41.861110+0000","last_undegraded":"2024-09-11T03:54:41.861110+0000","last_fullsized":"2024-09-11T03:54:41.861110+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:41.861020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:41.861020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:45.108337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"20'92","reported_seq":1663,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849597+0000","last_change":"2024-09-11T03:53:53.064439+0000","last_active":"2024-09-11T03:54:39.849597+0000","last_peered":"2024-09-11T03:54:39.849597+0000","last_clean":"2024-09-11T03:54:39.849597+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:54:39.849597+0000","last_undegraded":"2024-09-11T03:54:39.849597+0000","last_fullsized":"2024-09-11T03:54:39.849597+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:53.064376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:53.064376+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:58.638576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'92","reported_seq":1564,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.848803+0000","last_change":"2024-09-11T03:54:39.277334+0000","last_active":"2024-09-11T03:54:39.848803+0000","last_peered":"2024-09-11T03:54:39.848803+0000","last_clean":"2024-09-11T03:54:39.848803+0000","last_became_active":"2024-09-11T03:53:22.147219+0000","last_became_peered":"2024-09-11T03:53:22.147219+0000","last_unstale":"2024-09-11T03:54:39.848803+0000","last_undegraded":"2024-09-11T03:54:39.848803+0000","last_fullsized":"2024-09-11T03:54:39.848803+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:39.277285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:39.277285+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:45.539333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"20'92","reported_seq":1652,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849643+0000","last_change":"2024-09-11T03:54:32.917233+0000","last_active":"2024-09-11T03:54:39.849643+0000","last_peered":"2024-09-11T03:54:39.849643+0000","last_clean":"2024-09-11T03:54:39.849643+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:54:39.849643+0000","last_undegraded":"2024-09-11T03:54:39.849643+0000","last_fullsized":"2024-09-11T03:54:39.849643+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:32.917169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:32.917169+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:38.354488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'92","reported_seq":1756,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.848625+0000","last_change":"2024-09-11T03:54:36.746613+0000","last_active":"2024-09-11T03:54:39.848625+0000","last_peered":"2024-09-11T03:54:39.848625+0000","last_clean":"2024-09-11T03:54:39.848625+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:54:39.848625+0000","last_undegraded":"2024-09-11T03:54:39.848625+0000","last_fullsized":"2024-09-11T03:54:39.848625+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:36.746560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:36.746560+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:56:06.394749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.14","version":"20'92","reported_seq":846,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:40.052679+0000","last_change":"2024-09-11T03:54:32.311866+0000","last_active":"2024-09-11T03:54:40.052679+0000","last_peered":"2024-09-11T03:54:40.052679+0000","last_clean":"2024-09-11T03:54:40.052679+0000","last_became_active":"2024-09-11T03:53:35.242358+0000","last_became_peered":"2024-09-11T03:53:35.242358+0000","last_unstale":"2024-09-11T03:54:40.052679+0000","last_undegraded":"2024-09-11T03:54:40.052679+0000","last_fullsized":"2024-09-11T03:54:40.052679+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:32.311815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:32.311815+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:41.594853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'92","reported_seq":1624,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849266+0000","last_change":"2024-09-11T03:54:17.399228+0000","last_active":"2024-09-11T03:54:39.849266+0000","last_peered":"2024-09-11T03:54:39.849266+0000","last_clean":"2024-09-11T03:54:39.849266+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:54:39.849266+0000","last_undegraded":"2024-09-11T03:54:39.849266+0000","last_fullsized":"2024-09-11T03:54:39.849266+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:17.399162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:17.399162+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:34.922939+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.16","version":"20'92","reported_seq":1112,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849129+0000","last_change":"2024-09-11T03:53:31.513122+0000","last_active":"2024-09-11T03:54:39.849129+0000","last_peered":"2024-09-11T03:54:39.849129+0000","last_clean":"2024-09-11T03:54:39.849129+0000","last_became_active":"2024-09-11T03:53:22.150049+0000","last_became_peered":"2024-09-11T03:53:22.150049+0000","last_unstale":"2024-09-11T03:54:39.849129+0000","last_undegraded":"2024-09-11T03:54:39.849129+0000","last_fullsized":"2024-09-11T03:54:39.849129+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:31.513052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:31.513052+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:44.838397+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.17","version":"20'92","reported_seq":1646,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.725248+0000","last_change":"2024-09-11T03:53:15.383634+0000","last_active":"2024-09-11T03:54:39.725248+0000","last_peered":"2024-09-11T03:54:39.725248+0000","last_clean":"2024-09-11T03:54:39.725248+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:54:39.725248+0000","last_undegraded":"2024-09-11T03:54:39.725248+0000","last_fullsized":"2024-09-11T03:54:39.725248+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:15.383567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:15.383567+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"20'92","reported_seq":1763,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849587+0000","last_change":"2024-09-11T03:54:08.026046+0000","last_active":"2024-09-11T03:54:39.849587+0000","last_peered":"2024-09-11T03:54:39.849587+0000","last_clean":"2024-09-11T03:54:39.849587+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:54:39.849587+0000","last_undegraded":"2024-09-11T03:54:39.849587+0000","last_fullsized":"2024-09-11T03:54:39.849587+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:08.025977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:08.025977+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:24.774602+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"20'92","reported_seq":1260,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849429+0000","last_change":"2024-09-11T03:54:27.522701+0000","last_active":"2024-09-11T03:54:39.849429+0000","last_peered":"2024-09-11T03:54:39.849429+0000","last_clean":"2024-09-11T03:54:39.849429+0000","last_became_active":"2024-09-11T03:53:22.152261+0000","last_became_peered":"2024-09-11T03:53:22.152261+0000","last_unstale":"2024-09-11T03:54:39.849429+0000","last_undegraded":"2024-09-11T03:54:39.849429+0000","last_fullsized":"2024-09-11T03:54:39.849429+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:27.522632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:27.522632+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:30.063189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"20'92","reported_seq":779,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.721653+0000","last_change":"2024-09-11T03:54:37.291465+0000","last_active":"2024-09-11T03:54:39.721653+0000","last_peered":"2024-09-11T03:54:39.721653+0000","last_clean":"2024-09-11T03:54:39.721653+0000","last_became_active":"2024-09-11T03:53:43.289124+0000","last_became_peered":"2024-09-11T03:53:43.289124+0000","last_unstale":"2024-09-11T03:54:39.721653+0000","last_undegraded":"2024-09-11T03:54:39.721653+0000","last_fullsized":"2024-09-11T03:54:39.721653+0000","mapping_epoch":979,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":980,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:37.291404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:37.291404+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:54.232680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"20'92","reported_seq":1626,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.725066+0000","last_change":"2024-09-11T03:53:22.473180+0000","last_active":"2024-09-11T03:54:39.725066+0000","last_peered":"2024-09-11T03:54:39.725066+0000","last_clean":"2024-09-11T03:54:39.725066+0000","last_became_active":"2024-09-11T03:53:22.149300+0000","last_became_peered":"2024-09-11T03:53:22.149300+0000","last_unstale":"2024-09-11T03:54:39.725066+0000","last_undegraded":"2024-09-11T03:54:39.725066+0000","last_fullsized":"2024-09-11T03:54:39.725066+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:22.473115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:22.473115+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:46.088372+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"20'92","reported_seq":1630,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.725122+0000","last_change":"2024-09-11T03:53:24.477062+0000","last_active":"2024-09-11T03:54:39.725122+0000","last_peered":"2024-09-11T03:54:39.725122+0000","last_clean":"2024-09-11T03:54:39.725122+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:54:39.725122+0000","last_undegraded":"2024-09-11T03:54:39.725122+0000","last_fullsized":"2024-09-11T03:54:39.725122+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:24.477005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:24.477005+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:51.818677+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"20'92","reported_seq":1617,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.848890+0000","last_change":"2024-09-11T03:54:31.698537+0000","last_active":"2024-09-11T03:54:39.848890+0000","last_peered":"2024-09-11T03:54:39.848890+0000","last_clean":"2024-09-11T03:54:39.848890+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:54:39.848890+0000","last_undegraded":"2024-09-11T03:54:39.848890+0000","last_fullsized":"2024-09-11T03:54:39.848890+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:31.698489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:31.698489+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:52.924813+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"20'92","reported_seq":1634,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849398+0000","last_change":"2024-09-11T03:53:36.595714+0000","last_active":"2024-09-11T03:54:39.849398+0000","last_peered":"2024-09-11T03:54:39.849398+0000","last_clean":"2024-09-11T03:54:39.849398+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:54:39.849398+0000","last_undegraded":"2024-09-11T03:54:39.849398+0000","last_fullsized":"2024-09-11T03:54:39.849398+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:36.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:36.595651+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:02.698349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285779,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":51624,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93693528,"statfs":{"total":95995035648,"available":95942172672,"internally_reserved":0,"allocated":6705152,"data_stored":4519000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":46135748},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514711,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":59944,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93685208,"statfs":{"total":95995035648,"available":95933652992,"internally_reserved":0,"allocated":6705152,"data_stored":4519000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743645,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73832,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93671320,"statfs":{"total":95995035648,"available":95919431680,"internally_reserved":0,"allocated":6705152,"data_stored":4519000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037989,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":55784,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93689368,"statfs":{"total":95995035648,"available":95937912832,"internally_reserved":0,"allocated":6705152,"data_stored":4519000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299624,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82352,"kb_used_data":6748,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93662800,"statfs":{"total":95995035648,"available":95910707200,"internally_reserved":0,"allocated":6909952,"data_stored":4720248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528556,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64424,"kb_used_data":6868,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93680728,"statfs":{"total":95995035648,"available":95929065472,"internally_reserved":0,"allocated":7032832,"data_stored":4846680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757491,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64304,"kb_used_data":6748,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93680848,"statfs":{"total":95995035648,"available":95929188352,"internally_reserved":0,"allocated":6909952,"data_stored":4720248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986424,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64104,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93681048,"statfs":{"total":95995035648,"available":95929393152,"internally_reserved":0,"allocated":6705152,"data_stored":4519000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149949,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82472,"kb_used_data":6868,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93662680,"statfs":{"total":95995035648,"available":95910584320,"internally_reserved":0,"allocated":7032832,"data_stored":4846680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411586,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64304,"kb_used_data":6748,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93680848,"statfs":{"total":95995035648,"available":95929188352,"internally_reserved":0,"allocated":6909952,"data_stored":4720248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575112,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":55784,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93689368,"statfs":{"total":95995035648,"available":95937912832,"internally_reserved":0,"allocated":6705152,"data_stored":4519000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":965,"seq":4144643440659,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":35688,"kb_used_data":6868,"kb_used_omap":60,"kb_used_meta":28739,"kb_avail":93709464,"statfs":{"total":95995035648,"available":95958491136,"internally_reserved":0,"allocated":7032832,"data_stored":4846680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":62345,"internal_metadata":29428855},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:54:48.494 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T03:54:48.736 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:49.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:48 smithi084 ceph-mon[27254]: pgmap v1694: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:49.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:48 smithi084 ceph-mon[27254]: from='client.17997 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:54:49.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:48 smithi084 ceph-mon[27254]: 1.1b scrub starts 2024-09-11T03:54:49.145 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:48 smithi084 ceph-mon[27254]: 1.1b scrub ok 2024-09-11T03:54:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:48 smithi121 ceph-mon[30729]: pgmap v1694: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:48 smithi121 ceph-mon[30729]: from='client.17997 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:54:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:48 smithi121 ceph-mon[30729]: 1.1b scrub starts 2024-09-11T03:54:49.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:48 smithi121 ceph-mon[30729]: 1.1b scrub ok 2024-09-11T03:54:49.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:48 smithi175 ceph-mon[30894]: pgmap v1694: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:49.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:48 smithi175 ceph-mon[30894]: from='client.17997 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:54:49.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:48 smithi175 ceph-mon[30894]: 1.1b scrub starts 2024-09-11T03:54:49.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:48 smithi175 ceph-mon[30894]: 1.1b scrub ok 2024-09-11T03:54:49.411 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-11T03:54:49.412 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-11T03:54:50.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:49 smithi084 ceph-mon[27254]: 1.16 scrub starts 2024-09-11T03:54:50.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:49 smithi084 ceph-mon[27254]: 1.16 scrub ok 2024-09-11T03:54:50.142 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1695,"stamp":"2024-09-11T03:54:48.922624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":754608,"kb_used_data":80136,"kb_used_omap":78,"kb_used_meta":674161,"kb_avail":1124187216,"statfs":{"total":1151940427776,"available":1151167709184,"internally_reserved":0,"allocated":82059264,"data_stored":55814784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":79891,"internal_metadata":690341869},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002295"},"pg_stats":[{"pgid":"1.c","version":"20'92","reported_seq":692,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.722160+0000","last_change":"2024-09-11T03:53:40.839900+0000","last_active":"2024-09-11T03:54:39.722160+0000","last_peered":"2024-09-11T03:54:39.722160+0000","last_clean":"2024-09-11T03:54:39.722160+0000","last_became_active":"2024-09-11T03:53:40.273621+0000","last_became_peered":"2024-09-11T03:53:40.273621+0000","last_unstale":"2024-09-11T03:54:39.722160+0000","last_undegraded":"2024-09-11T03:54:39.722160+0000","last_fullsized":"2024-09-11T03:54:39.722160+0000","mapping_epoch":977,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":978,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:40.839833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:40.839833+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:49.821120+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"20'92","reported_seq":1675,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849278+0000","last_change":"2024-09-11T03:54:31.218187+0000","last_active":"2024-09-11T03:54:39.849278+0000","last_peered":"2024-09-11T03:54:39.849278+0000","last_clean":"2024-09-11T03:54:39.849278+0000","last_became_active":"2024-09-11T03:53:25.236299+0000","last_became_peered":"2024-09-11T03:53:25.236299+0000","last_unstale":"2024-09-11T03:54:39.849278+0000","last_undegraded":"2024-09-11T03:54:39.849278+0000","last_fullsized":"2024-09-11T03:54:39.849278+0000","mapping_epoch":967,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":968,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:31.218122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:31.218122+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:38.989270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"20'92","reported_seq":1360,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.721605+0000","last_change":"2024-09-11T03:54:35.274405+0000","last_active":"2024-09-11T03:54:39.721605+0000","last_peered":"2024-09-11T03:54:39.721605+0000","last_clean":"2024-09-11T03:54:39.721605+0000","last_became_active":"2024-09-11T03:53:43.289334+0000","last_became_peered":"2024-09-11T03:53:43.289334+0000","last_unstale":"2024-09-11T03:54:39.721605+0000","last_undegraded":"2024-09-11T03:54:39.721605+0000","last_fullsized":"2024-09-11T03:54:39.721605+0000","mapping_epoch":979,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":980,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:35.274333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:35.274333+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:41.555122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"20'92","reported_seq":957,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849307+0000","last_change":"2024-09-11T03:54:12.190134+0000","last_active":"2024-09-11T03:54:39.849307+0000","last_peered":"2024-09-11T03:54:39.849307+0000","last_clean":"2024-09-11T03:54:39.849307+0000","last_became_active":"2024-09-11T03:53:25.236560+0000","last_became_peered":"2024-09-11T03:53:25.236560+0000","last_unstale":"2024-09-11T03:54:39.849307+0000","last_undegraded":"2024-09-11T03:54:39.849307+0000","last_fullsized":"2024-09-11T03:54:39.849307+0000","mapping_epoch":967,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":968,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:12.190080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:12.190080+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:33.720781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":2067,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849215+0000","last_change":"2024-09-11T03:54:30.195100+0000","last_active":"2024-09-11T03:54:39.849215+0000","last_peered":"2024-09-11T03:54:39.849215+0000","last_clean":"2024-09-11T03:54:39.849215+0000","last_became_active":"2024-09-11T03:46:15.975146+0000","last_became_peered":"2024-09-11T03:46:15.975146+0000","last_unstale":"2024-09-11T03:54:39.849215+0000","last_undegraded":"2024-09-11T03:54:39.849215+0000","last_fullsized":"2024-09-11T03:54:39.849215+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:30.195041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:30.195041+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:56.979078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"20'92","reported_seq":1687,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.848971+0000","last_change":"2024-09-11T03:54:11.442419+0000","last_active":"2024-09-11T03:54:39.848971+0000","last_peered":"2024-09-11T03:54:39.848971+0000","last_clean":"2024-09-11T03:54:39.848971+0000","last_became_active":"2024-09-11T03:53:25.236713+0000","last_became_peered":"2024-09-11T03:53:25.236713+0000","last_unstale":"2024-09-11T03:54:39.848971+0000","last_undegraded":"2024-09-11T03:54:39.848971+0000","last_fullsized":"2024-09-11T03:54:39.848971+0000","mapping_epoch":967,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":968,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:11.442366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:11.442366+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:18.743650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"20'92","reported_seq":335,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849099+0000","last_change":"2024-09-11T03:53:55.362705+0000","last_active":"2024-09-11T03:54:39.849099+0000","last_peered":"2024-09-11T03:54:39.849099+0000","last_clean":"2024-09-11T03:54:39.849099+0000","last_became_active":"2024-09-11T03:53:55.362405+0000","last_became_peered":"2024-09-11T03:53:55.362405+0000","last_unstale":"2024-09-11T03:54:39.849099+0000","last_undegraded":"2024-09-11T03:54:39.849099+0000","last_fullsized":"2024-09-11T03:54:39.849099+0000","mapping_epoch":987,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":988,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:23.449701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:23.449701+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:48.767776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"20'92","reported_seq":1254,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849065+0000","last_change":"2024-09-11T03:54:37.504339+0000","last_active":"2024-09-11T03:54:39.849065+0000","last_peered":"2024-09-11T03:54:39.849065+0000","last_clean":"2024-09-11T03:54:39.849065+0000","last_became_active":"2024-09-11T03:53:22.151290+0000","last_became_peered":"2024-09-11T03:53:22.151290+0000","last_unstale":"2024-09-11T03:54:39.849065+0000","last_undegraded":"2024-09-11T03:54:39.849065+0000","last_fullsized":"2024-09-11T03:54:39.849065+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:37.504276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:37.504276+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:38.338864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"20'92","reported_seq":935,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:43.870143+0000","last_change":"2024-09-11T03:54:43.870143+0000","last_active":"2024-09-11T03:54:43.870143+0000","last_peered":"2024-09-11T03:54:43.870143+0000","last_clean":"2024-09-11T03:54:43.870143+0000","last_became_active":"2024-09-11T03:46:15.971805+0000","last_became_peered":"2024-09-11T03:46:15.971805+0000","last_unstale":"2024-09-11T03:54:43.870143+0000","last_undegraded":"2024-09-11T03:54:43.870143+0000","last_fullsized":"2024-09-11T03:54:43.870143+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:43.870080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:43.870080+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:56:12.736551+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"20'92","reported_seq":1254,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849240+0000","last_change":"2024-09-11T03:54:17.510152+0000","last_active":"2024-09-11T03:54:39.849240+0000","last_peered":"2024-09-11T03:54:39.849240+0000","last_clean":"2024-09-11T03:54:39.849240+0000","last_became_active":"2024-09-11T03:43:52.922901+0000","last_became_peered":"2024-09-11T03:43:52.922901+0000","last_unstale":"2024-09-11T03:54:39.849240+0000","last_undegraded":"2024-09-11T03:54:39.849240+0000","last_fullsized":"2024-09-11T03:54:39.849240+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:17.510096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:17.510096+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:26.164917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.5","version":"20'92","reported_seq":1245,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849162+0000","last_change":"2024-09-11T03:54:34.533921+0000","last_active":"2024-09-11T03:54:39.849162+0000","last_peered":"2024-09-11T03:54:39.849162+0000","last_clean":"2024-09-11T03:54:39.849162+0000","last_became_active":"2024-09-11T03:46:15.972719+0000","last_became_peered":"2024-09-11T03:46:15.972719+0000","last_unstale":"2024-09-11T03:54:39.849162+0000","last_undegraded":"2024-09-11T03:54:39.849162+0000","last_fullsized":"2024-09-11T03:54:39.849162+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:34.533832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:34.533832+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:59.497582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"20'92","reported_seq":1835,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849102+0000","last_change":"2024-09-11T03:54:00.596942+0000","last_active":"2024-09-11T03:54:39.849102+0000","last_peered":"2024-09-11T03:54:39.849102+0000","last_clean":"2024-09-11T03:54:39.849102+0000","last_became_active":"2024-09-11T03:44:27.934792+0000","last_became_peered":"2024-09-11T03:44:27.934792+0000","last_unstale":"2024-09-11T03:54:39.849102+0000","last_undegraded":"2024-09-11T03:54:39.849102+0000","last_fullsized":"2024-09-11T03:54:39.849102+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:00.596883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:00.596883+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:30.391472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,2],"acting":[11,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.b","version":"20'92","reported_seq":1648,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849102+0000","last_change":"2024-09-11T03:53:27.548967+0000","last_active":"2024-09-11T03:54:39.849102+0000","last_peered":"2024-09-11T03:54:39.849102+0000","last_clean":"2024-09-11T03:54:39.849102+0000","last_became_active":"2024-09-11T03:53:22.148230+0000","last_became_peered":"2024-09-11T03:53:22.148230+0000","last_unstale":"2024-09-11T03:54:39.849102+0000","last_undegraded":"2024-09-11T03:54:39.849102+0000","last_fullsized":"2024-09-11T03:54:39.849102+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:27.548907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:27.548907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:47.498932+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"20'92","reported_seq":2017,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849238+0000","last_change":"2024-09-11T03:54:29.949838+0000","last_active":"2024-09-11T03:54:39.849238+0000","last_peered":"2024-09-11T03:54:39.849238+0000","last_clean":"2024-09-11T03:54:39.849238+0000","last_became_active":"2024-09-11T03:53:25.237106+0000","last_became_peered":"2024-09-11T03:53:25.237106+0000","last_unstale":"2024-09-11T03:54:39.849238+0000","last_undegraded":"2024-09-11T03:54:39.849238+0000","last_fullsized":"2024-09-11T03:54:39.849238+0000","mapping_epoch":967,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":968,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:29.949736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:29.949736+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:44.952168+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"20'92","reported_seq":2054,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849074+0000","last_change":"2024-09-11T03:53:59.406218+0000","last_active":"2024-09-11T03:54:39.849074+0000","last_peered":"2024-09-11T03:54:39.849074+0000","last_clean":"2024-09-11T03:54:39.849074+0000","last_became_active":"2024-09-11T03:40:37.848457+0000","last_became_peered":"2024-09-11T03:40:37.848457+0000","last_unstale":"2024-09-11T03:54:39.849074+0000","last_undegraded":"2024-09-11T03:54:39.849074+0000","last_fullsized":"2024-09-11T03:54:39.849074+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:59.406151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:59.406151+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:07.726809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"20'92","reported_seq":1721,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:41.861110+0000","last_change":"2024-09-11T03:54:41.861110+0000","last_active":"2024-09-11T03:54:41.861110+0000","last_peered":"2024-09-11T03:54:41.861110+0000","last_clean":"2024-09-11T03:54:41.861110+0000","last_became_active":"2024-09-11T03:40:37.851174+0000","last_became_peered":"2024-09-11T03:40:37.851174+0000","last_unstale":"2024-09-11T03:54:41.861110+0000","last_undegraded":"2024-09-11T03:54:41.861110+0000","last_fullsized":"2024-09-11T03:54:41.861110+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:41.861020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:41.861020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:45.108337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"20'92","reported_seq":1663,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849597+0000","last_change":"2024-09-11T03:53:53.064439+0000","last_active":"2024-09-11T03:54:39.849597+0000","last_peered":"2024-09-11T03:54:39.849597+0000","last_clean":"2024-09-11T03:54:39.849597+0000","last_became_active":"2024-09-11T03:43:52.927842+0000","last_became_peered":"2024-09-11T03:43:52.927842+0000","last_unstale":"2024-09-11T03:54:39.849597+0000","last_undegraded":"2024-09-11T03:54:39.849597+0000","last_fullsized":"2024-09-11T03:54:39.849597+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:53.064376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:53.064376+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:58.638576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'92","reported_seq":1564,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.848803+0000","last_change":"2024-09-11T03:54:39.277334+0000","last_active":"2024-09-11T03:54:39.848803+0000","last_peered":"2024-09-11T03:54:39.848803+0000","last_clean":"2024-09-11T03:54:39.848803+0000","last_became_active":"2024-09-11T03:53:22.147219+0000","last_became_peered":"2024-09-11T03:53:22.147219+0000","last_unstale":"2024-09-11T03:54:39.848803+0000","last_undegraded":"2024-09-11T03:54:39.848803+0000","last_fullsized":"2024-09-11T03:54:39.848803+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:39.277285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:39.277285+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:45.539333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"20'92","reported_seq":1652,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849643+0000","last_change":"2024-09-11T03:54:32.917233+0000","last_active":"2024-09-11T03:54:39.849643+0000","last_peered":"2024-09-11T03:54:39.849643+0000","last_clean":"2024-09-11T03:54:39.849643+0000","last_became_active":"2024-09-11T03:46:15.973868+0000","last_became_peered":"2024-09-11T03:46:15.973868+0000","last_unstale":"2024-09-11T03:54:39.849643+0000","last_undegraded":"2024-09-11T03:54:39.849643+0000","last_fullsized":"2024-09-11T03:54:39.849643+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:32.917169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:32.917169+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:38.354488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'92","reported_seq":1756,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.848625+0000","last_change":"2024-09-11T03:54:36.746613+0000","last_active":"2024-09-11T03:54:39.848625+0000","last_peered":"2024-09-11T03:54:39.848625+0000","last_clean":"2024-09-11T03:54:39.848625+0000","last_became_active":"2024-09-11T03:43:52.923092+0000","last_became_peered":"2024-09-11T03:43:52.923092+0000","last_unstale":"2024-09-11T03:54:39.848625+0000","last_undegraded":"2024-09-11T03:54:39.848625+0000","last_fullsized":"2024-09-11T03:54:39.848625+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:36.746560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:36.746560+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:56:06.394749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.14","version":"20'92","reported_seq":846,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:40.052679+0000","last_change":"2024-09-11T03:54:32.311866+0000","last_active":"2024-09-11T03:54:40.052679+0000","last_peered":"2024-09-11T03:54:40.052679+0000","last_clean":"2024-09-11T03:54:40.052679+0000","last_became_active":"2024-09-11T03:53:35.242358+0000","last_became_peered":"2024-09-11T03:53:35.242358+0000","last_unstale":"2024-09-11T03:54:40.052679+0000","last_undegraded":"2024-09-11T03:54:40.052679+0000","last_fullsized":"2024-09-11T03:54:40.052679+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:32.311815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:32.311815+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:41.594853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'92","reported_seq":1624,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849266+0000","last_change":"2024-09-11T03:54:17.399228+0000","last_active":"2024-09-11T03:54:39.849266+0000","last_peered":"2024-09-11T03:54:39.849266+0000","last_clean":"2024-09-11T03:54:39.849266+0000","last_became_active":"2024-09-11T03:40:39.849365+0000","last_became_peered":"2024-09-11T03:40:39.849365+0000","last_unstale":"2024-09-11T03:54:39.849266+0000","last_undegraded":"2024-09-11T03:54:39.849266+0000","last_fullsized":"2024-09-11T03:54:39.849266+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:17.399162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:17.399162+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:34.922939+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.16","version":"20'92","reported_seq":1112,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849129+0000","last_change":"2024-09-11T03:53:31.513122+0000","last_active":"2024-09-11T03:54:39.849129+0000","last_peered":"2024-09-11T03:54:39.849129+0000","last_clean":"2024-09-11T03:54:39.849129+0000","last_became_active":"2024-09-11T03:53:22.150049+0000","last_became_peered":"2024-09-11T03:53:22.150049+0000","last_unstale":"2024-09-11T03:54:39.849129+0000","last_undegraded":"2024-09-11T03:54:39.849129+0000","last_fullsized":"2024-09-11T03:54:39.849129+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:31.513052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:31.513052+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:44.838397+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0],"acting":[11,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.17","version":"20'92","reported_seq":1646,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.725248+0000","last_change":"2024-09-11T03:53:15.383634+0000","last_active":"2024-09-11T03:54:39.725248+0000","last_peered":"2024-09-11T03:54:39.725248+0000","last_clean":"2024-09-11T03:54:39.725248+0000","last_became_active":"2024-09-11T03:43:52.928168+0000","last_became_peered":"2024-09-11T03:43:52.928168+0000","last_unstale":"2024-09-11T03:54:39.725248+0000","last_undegraded":"2024-09-11T03:54:39.725248+0000","last_fullsized":"2024-09-11T03:54:39.725248+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:15.383567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:15.383567+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"20'92","reported_seq":1763,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849587+0000","last_change":"2024-09-11T03:54:08.026046+0000","last_active":"2024-09-11T03:54:39.849587+0000","last_peered":"2024-09-11T03:54:39.849587+0000","last_clean":"2024-09-11T03:54:39.849587+0000","last_became_active":"2024-09-11T03:40:39.850125+0000","last_became_peered":"2024-09-11T03:40:39.850125+0000","last_unstale":"2024-09-11T03:54:39.849587+0000","last_undegraded":"2024-09-11T03:54:39.849587+0000","last_fullsized":"2024-09-11T03:54:39.849587+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:08.025977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:08.025977+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:24.774602+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"20'92","reported_seq":1260,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849429+0000","last_change":"2024-09-11T03:54:27.522701+0000","last_active":"2024-09-11T03:54:39.849429+0000","last_peered":"2024-09-11T03:54:39.849429+0000","last_clean":"2024-09-11T03:54:39.849429+0000","last_became_active":"2024-09-11T03:53:22.152261+0000","last_became_peered":"2024-09-11T03:53:22.152261+0000","last_unstale":"2024-09-11T03:54:39.849429+0000","last_undegraded":"2024-09-11T03:54:39.849429+0000","last_fullsized":"2024-09-11T03:54:39.849429+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:27.522632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:27.522632+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:30.063189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"20'92","reported_seq":779,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.721653+0000","last_change":"2024-09-11T03:54:37.291465+0000","last_active":"2024-09-11T03:54:39.721653+0000","last_peered":"2024-09-11T03:54:39.721653+0000","last_clean":"2024-09-11T03:54:39.721653+0000","last_became_active":"2024-09-11T03:53:43.289124+0000","last_became_peered":"2024-09-11T03:53:43.289124+0000","last_unstale":"2024-09-11T03:54:39.721653+0000","last_undegraded":"2024-09-11T03:54:39.721653+0000","last_fullsized":"2024-09-11T03:54:39.721653+0000","mapping_epoch":979,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":980,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:37.291404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:37.291404+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:54.232680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"20'92","reported_seq":1626,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.725066+0000","last_change":"2024-09-11T03:53:22.473180+0000","last_active":"2024-09-11T03:54:39.725066+0000","last_peered":"2024-09-11T03:54:39.725066+0000","last_clean":"2024-09-11T03:54:39.725066+0000","last_became_active":"2024-09-11T03:53:22.149300+0000","last_became_peered":"2024-09-11T03:53:22.149300+0000","last_unstale":"2024-09-11T03:54:39.725066+0000","last_undegraded":"2024-09-11T03:54:39.725066+0000","last_fullsized":"2024-09-11T03:54:39.725066+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:22.473115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:22.473115+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:46.088372+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"20'92","reported_seq":1630,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.725122+0000","last_change":"2024-09-11T03:53:24.477062+0000","last_active":"2024-09-11T03:54:39.725122+0000","last_peered":"2024-09-11T03:54:39.725122+0000","last_clean":"2024-09-11T03:54:39.725122+0000","last_became_active":"2024-09-11T03:46:15.973230+0000","last_became_peered":"2024-09-11T03:46:15.973230+0000","last_unstale":"2024-09-11T03:54:39.725122+0000","last_undegraded":"2024-09-11T03:54:39.725122+0000","last_fullsized":"2024-09-11T03:54:39.725122+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:24.477005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:24.477005+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:54:51.818677+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"20'92","reported_seq":1617,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.848890+0000","last_change":"2024-09-11T03:54:31.698537+0000","last_active":"2024-09-11T03:54:39.848890+0000","last_peered":"2024-09-11T03:54:39.848890+0000","last_clean":"2024-09-11T03:54:39.848890+0000","last_became_active":"2024-09-11T03:40:39.850132+0000","last_became_peered":"2024-09-11T03:40:39.850132+0000","last_unstale":"2024-09-11T03:54:39.848890+0000","last_undegraded":"2024-09-11T03:54:39.848890+0000","last_fullsized":"2024-09-11T03:54:39.848890+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:54:31.698489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:54:31.698489+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:52.924813+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"20'92","reported_seq":1634,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-09-11T03:54:39.849398+0000","last_change":"2024-09-11T03:53:36.595714+0000","last_active":"2024-09-11T03:54:39.849398+0000","last_peered":"2024-09-11T03:54:39.849398+0000","last_clean":"2024-09-11T03:54:39.849398+0000","last_became_active":"2024-09-11T03:40:39.848695+0000","last_became_peered":"2024-09-11T03:40:39.848695+0000","last_unstale":"2024-09-11T03:54:39.849398+0000","last_undegraded":"2024-09-11T03:54:39.849398+0000","last_fullsized":"2024-09-11T03:54:39.849398+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'92","last_scrub_stamp":"2024-09-11T03:53:36.595651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:33:43.042050+0000","last_clean_scrub_stamp":"2024-09-11T03:53:36.595651+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T03:55:02.698349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285780,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":51624,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93693528,"statfs":{"total":95995035648,"available":95942172672,"internally_reserved":0,"allocated":6705152,"data_stored":4519000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":46135748},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":59944,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93685208,"statfs":{"total":95995035648,"available":95933652992,"internally_reserved":0,"allocated":6705152,"data_stored":4519000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743646,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73832,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93671320,"statfs":{"total":95995035648,"available":95919431680,"internally_reserved":0,"allocated":6705152,"data_stored":4519000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037989,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":55784,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93689368,"statfs":{"total":95995035648,"available":95937912832,"internally_reserved":0,"allocated":6705152,"data_stored":4519000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299624,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82352,"kb_used_data":6748,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93662800,"statfs":{"total":95995035648,"available":95910707200,"internally_reserved":0,"allocated":6909952,"data_stored":4720248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528557,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64420,"kb_used_data":6868,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93680732,"statfs":{"total":95995035648,"available":95929069568,"internally_reserved":0,"allocated":7032832,"data_stored":4846680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757491,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64304,"kb_used_data":6748,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93680848,"statfs":{"total":95995035648,"available":95929188352,"internally_reserved":0,"allocated":6909952,"data_stored":4720248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986424,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64104,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93681048,"statfs":{"total":95995035648,"available":95929393152,"internally_reserved":0,"allocated":6705152,"data_stored":4519000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149950,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82468,"kb_used_data":6868,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93662684,"statfs":{"total":95995035648,"available":95910588416,"internally_reserved":0,"allocated":7032832,"data_stored":4846680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411587,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64304,"kb_used_data":6748,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93680848,"statfs":{"total":95995035648,"available":95929188352,"internally_reserved":0,"allocated":6909952,"data_stored":4720248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575112,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":55784,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93689368,"statfs":{"total":95995035648,"available":95937912832,"internally_reserved":0,"allocated":6705152,"data_stored":4519000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":965,"seq":4144643440659,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":35688,"kb_used_data":6868,"kb_used_omap":60,"kb_used_meta":28739,"kb_avail":93709464,"statfs":{"total":95995035648,"available":95958491136,"internally_reserved":0,"allocated":7032832,"data_stored":4846680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":62345,"internal_metadata":29428855},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:54:50.144 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-11T03:54:50.144 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30 shell --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-11T03:54:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:49 smithi121 ceph-mon[30729]: 1.16 scrub starts 2024-09-11T03:54:50.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:49 smithi121 ceph-mon[30729]: 1.16 scrub ok 2024-09-11T03:54:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:49 smithi175 ceph-mon[30894]: 1.16 scrub starts 2024-09-11T03:54:50.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:49 smithi175 ceph-mon[30894]: 1.16 scrub ok 2024-09-11T03:54:50.387 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T03:54:50.842 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:50 smithi084 ceph-mon[27254]: pgmap v1695: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:50.842 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:50 smithi084 ceph-mon[27254]: from='client.18003 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:54:51.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:50 smithi121 ceph-mon[30729]: pgmap v1695: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:51.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:50 smithi121 ceph-mon[30729]: from='client.18003 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:54:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:50 smithi175 ceph-mon[30894]: pgmap v1695: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:51.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:50 smithi175 ceph-mon[30894]: from='client.18003 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T03:54:51.760 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-11T03:54:51.770 INFO:tasks.cephadm:Teardown begin 2024-09-11T03:54:51.770 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-11T03:54:51.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-11T03:54:51.837 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-11T03:54:51.871 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-11T03:54:51.872 DEBUG:teuthology.orchestra.run.smithi084:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-11T03:54:51.887 DEBUG:teuthology.orchestra.run.smithi121:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-11T03:54:51.906 DEBUG:teuthology.orchestra.run.smithi175:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-11T03:54:51.929 INFO:tasks.cephadm:Stopping all daemons... 2024-09-11T03:54:51.929 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-11T03:54:51.929 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mon.a 2024-09-11T03:54:52.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:51 smithi084 ceph-mon[27254]: 1.c scrub starts 2024-09-11T03:54:52.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:51 smithi084 ceph-mon[27254]: 1.c scrub ok 2024-09-11T03:54:52.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:51 smithi084 ceph-mon[27254]: purged_snaps scrub starts 2024-09-11T03:54:52.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:51 smithi084 ceph-mon[27254]: purged_snaps scrub ok 2024-09-11T03:54:52.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:51 smithi084 ceph-mon[27254]: 1.2 scrub starts 2024-09-11T03:54:52.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:51 smithi084 ceph-mon[27254]: 1.2 scrub ok 2024-09-11T03:54:52.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:51 smithi084 ceph-mon[27254]: osdmap e1005: 12 total, 12 up, 12 in 2024-09-11T03:54:52.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1967151853' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-11T03:54:52.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:51 smithi084 ceph-mon[27254]: from='client.? 172.21.15.84:0/1967151853' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-11T03:54:52.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:51 smithi084 ceph-mon[27254]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:54:52.092 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:51 smithi084 systemd[1]: Stopping Ceph mon.a for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:54:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:51 smithi121 ceph-mon[30729]: 1.c scrub starts 2024-09-11T03:54:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:51 smithi121 ceph-mon[30729]: 1.c scrub ok 2024-09-11T03:54:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:51 smithi121 ceph-mon[30729]: purged_snaps scrub starts 2024-09-11T03:54:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:51 smithi121 ceph-mon[30729]: purged_snaps scrub ok 2024-09-11T03:54:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:51 smithi121 ceph-mon[30729]: 1.2 scrub starts 2024-09-11T03:54:52.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:51 smithi121 ceph-mon[30729]: 1.2 scrub ok 2024-09-11T03:54:52.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:51 smithi121 ceph-mon[30729]: osdmap e1005: 12 total, 12 up, 12 in 2024-09-11T03:54:52.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1967151853' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-11T03:54:52.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:51 smithi121 ceph-mon[30729]: from='client.? 172.21.15.84:0/1967151853' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-11T03:54:52.173 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:51 smithi121 ceph-mon[30729]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:54:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:51 smithi175 ceph-mon[30894]: 1.c scrub starts 2024-09-11T03:54:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:51 smithi175 ceph-mon[30894]: 1.c scrub ok 2024-09-11T03:54:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:51 smithi175 ceph-mon[30894]: purged_snaps scrub starts 2024-09-11T03:54:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:51 smithi175 ceph-mon[30894]: purged_snaps scrub ok 2024-09-11T03:54:52.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:51 smithi175 ceph-mon[30894]: 1.2 scrub starts 2024-09-11T03:54:52.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:51 smithi175 ceph-mon[30894]: 1.2 scrub ok 2024-09-11T03:54:52.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:51 smithi175 ceph-mon[30894]: osdmap e1005: 12 total, 12 up, 12 in 2024-09-11T03:54:52.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1967151853' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-11T03:54:52.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:51 smithi175 ceph-mon[30894]: from='client.? 172.21.15.84:0/1967151853' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-11T03:54:52.228 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:51 smithi175 ceph-mon[30894]: from='mgr.14150 172.21.15.84:0/2864163755' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-11T03:54:53.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:52 smithi084 ceph-mon[27254]: pgmap v1697: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:53.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:52 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a[27250]: 2024-09-11T03:54:52.888+0000 7fae15388640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:54:53.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:52 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a[27250]: 2024-09-11T03:54:52.888+0000 7fae15388640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-11T03:54:53.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:52 smithi084 podman[192650]: 2024-09-11 03:54:52.960295458 +0000 UTC m=+0.182805667 container died 1dd196d5d5cb6424ef8d5b25da0d9defdcd8fbc53f8b2a3eb6b5476e5c9b5558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T03:54:53.172 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:52 smithi121 ceph-mon[30729]: pgmap v1697: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:53.227 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:52 smithi175 ceph-mon[30894]: pgmap v1697: 31 pgs: 31 active+clean; 577 KiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-11T03:54:53.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:53 smithi084 podman[192650]: 2024-09-11 03:54:53.405647117 +0000 UTC m=+0.628157362 container cleanup 1dd196d5d5cb6424ef8d5b25da0d9defdcd8fbc53f8b2a3eb6b5476e5c9b5558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T03:54:53.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 11 03:54:53 smithi084 bash[192650]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-a 2024-09-11T03:54:53.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mon.a.service' 2024-09-11T03:54:53.868 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:54:53.868 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-11T03:54:53.868 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-09-11T03:54:53.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mon.b 2024-09-11T03:54:54.171 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:53 smithi121 systemd[1]: Stopping Ceph mon.b for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:54:54.659 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:54 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-b[30725]: 2024-09-11T03:54:54.200+0000 7fec478d1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:54:54.659 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:54 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-b[30725]: 2024-09-11T03:54:54.200+0000 7fec478d1640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-11T03:54:54.659 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:54 smithi121 podman[114311]: 2024-09-11 03:54:54.364284613 +0000 UTC m=+0.287497593 container died d891b9a0f3f53fe017a8185fd83c1cf2748533f0a7ac79779d10afcd8aceecde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2) 2024-09-11T03:54:54.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:54 smithi121 podman[114311]: 2024-09-11 03:54:54.876452992 +0000 UTC m=+0.799666014 container cleanup d891b9a0f3f53fe017a8185fd83c1cf2748533f0a7ac79779d10afcd8aceecde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-11T03:54:54.922 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:54 smithi121 bash[114311]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-b 2024-09-11T03:54:55.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mon.b.service' 2024-09-11T03:54:55.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:55 smithi121 podman[114324]: 2024-09-11 03:54:55.10277722 +0000 UTC m=+0.740562086 container remove d891b9a0f3f53fe017a8185fd83c1cf2748533f0a7ac79779d10afcd8aceecde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-b, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-11T03:54:55.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:55 smithi121 systemd[1]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mon.b.service: Deactivated successfully. 2024-09-11T03:54:55.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:55 smithi121 systemd[1]: Stopped Ceph mon.b for 10fe7946-6fee-11ef-bcea-c7b262605968. 2024-09-11T03:54:55.368 INFO:journalctl@ceph.mon.b.smithi121.stdout:Sep 11 03:54:55 smithi121 systemd[1]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mon.b.service: Consumed 15.894s CPU time. 2024-09-11T03:54:55.407 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:54:55.408 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-09-11T03:54:55.408 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-09-11T03:54:55.408 DEBUG:teuthology.orchestra.run.smithi175:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mon.c 2024-09-11T03:54:55.716 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:55 smithi175 systemd[1]: Stopping Ceph mon.c for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:54:55.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:55 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-c[30890]: 2024-09-11T03:54:55.715+0000 7fb0b42ef640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:54:55.977 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:55 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-c[30890]: 2024-09-11T03:54:55.715+0000 7fb0b42ef640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-11T03:54:56.283 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:55 smithi175 podman[112479]: 2024-09-11 03:54:55.981882717 +0000 UTC m=+0.380287856 container died 96a16f0d967ce68b2b6845c16304edfd136f250360fc3a8d79e7f24896288225 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-c, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-11T03:54:56.717 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:56 smithi175 podman[112479]: 2024-09-11 03:54:56.483235919 +0000 UTC m=+0.881641049 container cleanup 96a16f0d967ce68b2b6845c16304edfd136f250360fc3a8d79e7f24896288225 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.schema-version=1.0) 2024-09-11T03:54:56.717 INFO:journalctl@ceph.mon.c.smithi175.stdout:Sep 11 03:54:56 smithi175 bash[112479]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mon-c 2024-09-11T03:54:56.866 DEBUG:teuthology.orchestra.run.smithi175:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mon.c.service' 2024-09-11T03:54:56.913 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:54:56.913 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-09-11T03:54:56.913 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-11T03:54:56.913 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mgr.y 2024-09-11T03:54:57.251 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:54:56 smithi084 systemd[1]: Stopping Ceph mgr.y for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:54:57.534 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:54:57 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: 2024-09-11T03:54:57.250+0000 7fab9ef80640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:54:57.535 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:54:57 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y[27484]: 2024-09-11T03:54:57.250+0000 7fab9ef80640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-11T03:54:57.535 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:54:57 smithi084 podman[192789]: 2024-09-11 03:54:57.282252105 +0000 UTC m=+0.178210215 container died f7bb1492bbd3ca344a42efbf8dbc2b49a8eca592b98356b22f6ac786d28dc20f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-11T03:54:57.894 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:54:57 smithi084 podman[192789]: 2024-09-11 03:54:57.734889841 +0000 UTC m=+0.630847946 container cleanup f7bb1492bbd3ca344a42efbf8dbc2b49a8eca592b98356b22f6ac786d28dc20f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8f219c476cb4f26fc5719c661c0323719a45cb30, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T03:54:57.894 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 11 03:54:57 smithi084 bash[192789]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-y 2024-09-11T03:54:58.211 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mgr.y.service' 2024-09-11T03:54:58.244 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:54:58.245 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-11T03:54:58.245 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-11T03:54:58.245 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mgr.x 2024-09-11T03:54:58.575 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Sep 11 03:54:58 smithi121 systemd[1]: Stopping Ceph mgr.x for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:54:58.880 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Sep 11 03:54:58 smithi121 podman[114435]: 2024-09-11 03:54:58.576346529 +0000 UTC m=+0.145201814 container died 794a1afd76431951e5e2003dea929b03e4a98f5ba7f84bf2b58a3a86df60eee8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-x, org.label-schema.license=GPLv2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-11T03:54:59.172 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Sep 11 03:54:58 smithi121 podman[114435]: 2024-09-11 03:54:58.963967866 +0000 UTC m=+0.532823140 container cleanup 794a1afd76431951e5e2003dea929b03e4a98f5ba7f84bf2b58a3a86df60eee8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-x, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-11T03:54:59.172 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Sep 11 03:54:58 smithi121 bash[114435]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-mgr-x 2024-09-11T03:54:59.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@mgr.x.service' 2024-09-11T03:54:59.420 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:54:59.420 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-11T03:54:59.420 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-11T03:54:59.420 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.0 2024-09-11T03:54:59.822 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:54:59 smithi084 systemd[1]: Stopping Ceph osd.0 for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:55:00.144 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:54:59 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0[185342]: 2024-09-11T03:54:59.822+0000 7f95d78f3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:55:00.144 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:54:59 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0[185342]: 2024-09-11T03:54:59.822+0000 7f95d78f3640 -1 osd.0 1005 *** Got signal Terminated *** 2024-09-11T03:55:00.144 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:54:59 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0[185342]: 2024-09-11T03:54:59.822+0000 7f95d78f3640 -1 osd.0 1005 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T03:55:05.514 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:55:05 smithi084 podman[192911]: 2024-09-11 03:55:05.238667847 +0000 UTC m=+5.636478443 container died d2adc8d7aac824c1020796293b9e4c92bd0e40be4ecb39ecb1f4608c819c5849 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T03:55:05.894 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:55:05 smithi084 podman[192911]: 2024-09-11 03:55:05.696781028 +0000 UTC m=+6.094591611 container cleanup d2adc8d7aac824c1020796293b9e4c92bd0e40be4ecb39ecb1f4608c819c5849 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-11T03:55:05.894 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:55:05 smithi084 bash[192911]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0 2024-09-11T03:55:06.310 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:55:05 smithi084 podman[192926]: 2024-09-11 03:55:05.965003963 +0000 UTC m=+0.728520049 container remove d2adc8d7aac824c1020796293b9e4c92bd0e40be4ecb39ecb1f4608c819c5849 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T03:55:06.584 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:55:06 smithi084 podman[193017]: 2024-09-11 03:55:06.21377322 +0000 UTC m=+0.015652133 image pull 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e 2024-09-11T03:55:06.584 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:55:06 smithi084 podman[193017]: 2024-09-11 03:55:06.332622126 +0000 UTC m=+0.134501026 container create 24a336641efbf53fff3efbf66c526e7d18d713bf2df633019debf3ae3a4d9d56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-deactivate, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-09-11T03:55:06.584 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:55:06 smithi084 podman[193017]: 2024-09-11 03:55:06.584917462 +0000 UTC m=+0.386796366 container init 24a336641efbf53fff3efbf66c526e7d18d713bf2df633019debf3ae3a4d9d56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T03:55:06.894 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:55:06 smithi084 podman[193017]: 2024-09-11 03:55:06.588454251 +0000 UTC m=+0.390333162 container start 24a336641efbf53fff3efbf66c526e7d18d713bf2df633019debf3ae3a4d9d56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-deactivate, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-11T03:55:06.894 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:55:06 smithi084 podman[193017]: 2024-09-11 03:55:06.668026613 +0000 UTC m=+0.469905530 container attach 24a336641efbf53fff3efbf66c526e7d18d713bf2df633019debf3ae3a4d9d56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2) 2024-09-11T03:55:06.894 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 11 03:55:06 smithi084 podman[193017]: 2024-09-11 03:55:06.779573578 +0000 UTC m=+0.581452500 container died 24a336641efbf53fff3efbf66c526e7d18d713bf2df633019debf3ae3a4d9d56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4) 2024-09-11T03:55:07.516 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.0.service' 2024-09-11T03:55:07.553 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:55:07.553 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-11T03:55:07.553 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-11T03:55:07.553 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.1 2024-09-11T03:55:07.875 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:55:07 smithi084 systemd[1]: Stopping Ceph osd.1 for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:55:08.144 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:55:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-1[46583]: 2024-09-11T03:55:07.873+0000 7f2c7fbb8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:55:08.144 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:55:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-1[46583]: 2024-09-11T03:55:07.873+0000 7f2c7fbb8640 -1 osd.1 1005 *** Got signal Terminated *** 2024-09-11T03:55:08.144 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:55:07 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-1[46583]: 2024-09-11T03:55:07.873+0000 7f2c7fbb8640 -1 osd.1 1005 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T03:55:13.700 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:55:13 smithi084 podman[193122]: 2024-09-11 03:55:13.41904875 +0000 UTC m=+5.645486580 container died fd38c9e9e03a8d90ac296d346a8b240b5377d29b257d90287091af049e1a3bf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-1, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-09-11T03:55:14.094 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:55:13 smithi084 podman[193122]: 2024-09-11 03:55:13.890892926 +0000 UTC m=+6.117330756 container cleanup fd38c9e9e03a8d90ac296d346a8b240b5377d29b257d90287091af049e1a3bf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-1, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-11T03:55:14.094 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:55:13 smithi084 bash[193122]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-1 2024-09-11T03:55:14.394 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:55:14 smithi084 podman[193137]: 2024-09-11 03:55:14.134005588 +0000 UTC m=+0.711370336 container remove fd38c9e9e03a8d90ac296d346a8b240b5377d29b257d90287091af049e1a3bf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-11T03:55:14.728 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:55:14 smithi084 podman[193235]: 2024-09-11 03:55:14.366070943 +0000 UTC m=+0.015539166 image pull 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e 2024-09-11T03:55:14.728 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:55:14 smithi084 podman[193235]: 2024-09-11 03:55:14.518162795 +0000 UTC m=+0.167631011 container create bea313f23b75acd2d4cd7f41319c5bfb5c9c546aa31bc5cf5be5ec853961e8f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, maintainer=Guillaume Abrioux ) 2024-09-11T03:55:14.728 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:55:14 smithi084 podman[193235]: 2024-09-11 03:55:14.728712982 +0000 UTC m=+0.378181197 container init bea313f23b75acd2d4cd7f41319c5bfb5c9c546aa31bc5cf5be5ec853961e8f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-1-deactivate, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731) 2024-09-11T03:55:15.144 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:55:14 smithi084 podman[193235]: 2024-09-11 03:55:14.732324532 +0000 UTC m=+0.381792757 container start bea313f23b75acd2d4cd7f41319c5bfb5c9c546aa31bc5cf5be5ec853961e8f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-11T03:55:15.144 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:55:14 smithi084 podman[193235]: 2024-09-11 03:55:14.820396492 +0000 UTC m=+0.469864738 container attach bea313f23b75acd2d4cd7f41319c5bfb5c9c546aa31bc5cf5be5ec853961e8f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-11T03:55:15.144 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 11 03:55:14 smithi084 podman[193235]: 2024-09-11 03:55:14.928521801 +0000 UTC m=+0.577990017 container died bea313f23b75acd2d4cd7f41319c5bfb5c9c546aa31bc5cf5be5ec853961e8f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-1-deactivate, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T03:55:15.569 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.1.service' 2024-09-11T03:55:15.608 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:55:15.608 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-11T03:55:15.608 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-11T03:55:15.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.2 2024-09-11T03:55:15.894 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:55:15 smithi084 systemd[1]: Stopping Ceph osd.2 for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:55:16.394 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:55:15 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-2[52201]: 2024-09-11T03:55:15.942+0000 7fcf5b86d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:55:16.394 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:55:15 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-2[52201]: 2024-09-11T03:55:15.942+0000 7fcf5b86d640 -1 osd.2 1005 *** Got signal Terminated *** 2024-09-11T03:55:16.395 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:55:15 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-2[52201]: 2024-09-11T03:55:15.942+0000 7fcf5b86d640 -1 osd.2 1005 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T03:55:21.644 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:55:21 smithi084 podman[193338]: 2024-09-11 03:55:21.390109186 +0000 UTC m=+5.559089388 container died 6357dc4fa682b0ca0c5c86fe4aa7fe8ba0f53b2e0ccc71d696fc86ea2a395032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-11T03:55:22.018 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:55:21 smithi084 podman[193338]: 2024-09-11 03:55:21.751541496 +0000 UTC m=+5.920521688 container cleanup 6357dc4fa682b0ca0c5c86fe4aa7fe8ba0f53b2e0ccc71d696fc86ea2a395032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-09-11T03:55:22.019 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:55:21 smithi084 bash[193338]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-2 2024-09-11T03:55:22.366 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:55:22 smithi084 podman[193353]: 2024-09-11 03:55:22.019418411 +0000 UTC m=+0.629860226 container remove 6357dc4fa682b0ca0c5c86fe4aa7fe8ba0f53b2e0ccc71d696fc86ea2a395032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0) 2024-09-11T03:55:22.644 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:55:22 smithi084 podman[193439]: 2024-09-11 03:55:22.26941284 +0000 UTC m=+0.015619634 image pull 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e 2024-09-11T03:55:22.644 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:55:22 smithi084 podman[193439]: 2024-09-11 03:55:22.378590494 +0000 UTC m=+0.124797282 container create 3bb6e8b7ce6c8c0cd392cad6d01e9a295697df3c398c0c7bc849370015ba2e0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-11T03:55:22.644 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:55:22 smithi084 podman[193439]: 2024-09-11 03:55:22.597667027 +0000 UTC m=+0.343873815 container init 3bb6e8b7ce6c8c0cd392cad6d01e9a295697df3c398c0c7bc849370015ba2e0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-11T03:55:22.645 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:55:22 smithi084 podman[193439]: 2024-09-11 03:55:22.600780425 +0000 UTC m=+0.346987215 container start 3bb6e8b7ce6c8c0cd392cad6d01e9a295697df3c398c0c7bc849370015ba2e0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T03:55:23.075 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:55:22 smithi084 podman[193439]: 2024-09-11 03:55:22.65586307 +0000 UTC m=+0.402069870 container attach 3bb6e8b7ce6c8c0cd392cad6d01e9a295697df3c398c0c7bc849370015ba2e0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-11T03:55:23.075 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 11 03:55:22 smithi084 podman[193439]: 2024-09-11 03:55:22.787630075 +0000 UTC m=+0.533836872 container died 3bb6e8b7ce6c8c0cd392cad6d01e9a295697df3c398c0c7bc849370015ba2e0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-2-deactivate, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T03:55:23.414 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.2.service' 2024-09-11T03:55:23.454 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:55:23.454 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-11T03:55:23.454 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-11T03:55:23.454 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.3 2024-09-11T03:55:23.644 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 11 03:55:23 smithi084 systemd[1]: Stopping Ceph osd.3 for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:55:24.144 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 11 03:55:23 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-3[58214]: 2024-09-11T03:55:23.762+0000 7ff447bff640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:55:24.144 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 11 03:55:23 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-3[58214]: 2024-09-11T03:55:23.762+0000 7ff447bff640 -1 osd.3 1005 *** Got signal Terminated *** 2024-09-11T03:55:24.144 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 11 03:55:23 smithi084 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-3[58214]: 2024-09-11T03:55:23.762+0000 7ff447bff640 -1 osd.3 1005 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T03:55:26.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:26 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:26.301+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:26.727 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:26 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:26.315+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:27.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:27 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:27.276+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:27.727 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:27 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:27.329+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:28.671 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:28 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:28.298+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:28.727 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:28 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:28.325+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:28.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:28 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:28.337+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:29.386 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 11 03:55:29 smithi084 podman[193542]: 2024-09-11 03:55:29.116200863 +0000 UTC m=+5.439722732 container died 4790970dfbfe57fa063ba8be0d33a78e8c698d41f96627f37667f7e1325cf31a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-3, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-11T03:55:29.644 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 11 03:55:29 smithi084 podman[193542]: 2024-09-11 03:55:29.560924543 +0000 UTC m=+5.884446399 container cleanup 4790970dfbfe57fa063ba8be0d33a78e8c698d41f96627f37667f7e1325cf31a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-11T03:55:29.644 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 11 03:55:29 smithi084 bash[193542]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-3 2024-09-11T03:55:29.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:29 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:29.341+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:29.727 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:29 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:29.283+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:29.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:29 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:29.380+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:30.104 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 11 03:55:29 smithi084 podman[193557]: 2024-09-11 03:55:29.803840633 +0000 UTC m=+0.687794817 container remove 4790970dfbfe57fa063ba8be0d33a78e8c698d41f96627f37667f7e1325cf31a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-3, io.buildah.version=1.37.2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-11T03:55:30.279 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:30 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:30.021+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:30.380 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 11 03:55:30 smithi084 podman[193647]: 2024-09-11 03:55:30.104602646 +0000 UTC m=+0.098091632 container create 229b9728eced91761cc160ad9d0c6969ac4e13b7dbb64d07f707a479660d2701 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True) 2024-09-11T03:55:30.381 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 11 03:55:30 smithi084 podman[193647]: 2024-09-11 03:55:30.022079096 +0000 UTC m=+0.015568081 image pull 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e 2024-09-11T03:55:30.381 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 11 03:55:30 smithi084 podman[193647]: 2024-09-11 03:55:30.306822816 +0000 UTC m=+0.300311801 container init 229b9728eced91761cc160ad9d0c6969ac4e13b7dbb64d07f707a479660d2701 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-3-deactivate, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T03:55:30.381 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 11 03:55:30 smithi084 podman[193647]: 2024-09-11 03:55:30.309930645 +0000 UTC m=+0.303419630 container start 229b9728eced91761cc160ad9d0c6969ac4e13b7dbb64d07f707a479660d2701 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-3-deactivate, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-11T03:55:30.381 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 11 03:55:30 smithi084 podman[193647]: 2024-09-11 03:55:30.381616647 +0000 UTC m=+0.375105645 container attach 229b9728eced91761cc160ad9d0c6969ac4e13b7dbb64d07f707a479660d2701 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-09-11T03:55:30.644 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 11 03:55:30 smithi084 podman[193647]: 2024-09-11 03:55:30.496969161 +0000 UTC m=+0.490458156 container died 229b9728eced91761cc160ad9d0c6969ac4e13b7dbb64d07f707a479660d2701 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-09-11T03:55:30.671 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:30 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:30.307+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:30.727 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:30 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:30.280+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:30.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:30 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:30.337+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:30.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:30 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:30.405+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:31.083 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.3.service' 2024-09-11T03:55:31.123 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:55:31.123 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-11T03:55:31.123 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-11T03:55:31.123 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.4 2024-09-11T03:55:31.353 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:31 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:31.033+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:31.354 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:31 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:31.269+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:31.422 INFO:journalctl@ceph.osd.4.smithi121.stdout:Sep 11 03:55:31 smithi121 systemd[1]: Stopping Ceph osd.4 for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:55:31.422 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:31 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:31.293+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:31.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:31 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:31.354+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:31.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:31 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:31.356+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:31.922 INFO:journalctl@ceph.osd.4.smithi121.stdout:Sep 11 03:55:31 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-4[35097]: 2024-09-11T03:55:31.455+0000 7fa5f8c3d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:55:31.922 INFO:journalctl@ceph.osd.4.smithi121.stdout:Sep 11 03:55:31 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-4[35097]: 2024-09-11T03:55:31.455+0000 7fa5f8c3d640 -1 osd.4 1005 *** Got signal Terminated *** 2024-09-11T03:55:31.922 INFO:journalctl@ceph.osd.4.smithi121.stdout:Sep 11 03:55:31 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-4[35097]: 2024-09-11T03:55:31.455+0000 7fa5f8c3d640 -1 osd.4 1005 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T03:55:32.360 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:32 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:32.032+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:32.360 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:32 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:32.279+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:32.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:32 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:32.297+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:32.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:32 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:32.360+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:32.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:32 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:32.368+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:33.327 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:33 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:33.230+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:33.327 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:33 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:33.020+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:33.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:33 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:33.271+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:33.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:33 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:33.327+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:33.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:33 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:33.385+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:34.227 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:33 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:33.976+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:34.671 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:34 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:34.277+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:34.727 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:34 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:34.246+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:34.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:34 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:34.309+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:34.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:34 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:34.408+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:35.172 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:34 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:34.803+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:35.357 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:35 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:35.227+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:35.358 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:35 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:35.227+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:35.358 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:35 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:35.000+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:35.671 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:35 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:35.306+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:35.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:35 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:35.306+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:35.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:35 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:35.431+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:35.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:35 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:35.358+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:35.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:35 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:35.358+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:36.171 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:35 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:35.846+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:36.330 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:36 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:36.049+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:36.331 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:36 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:36.189+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:36.331 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:36 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:36.189+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:36.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:36 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:36.339+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:36.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:36 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:36.339+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:36.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:36 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:36.331+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:36.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:36 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:36.331+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:36.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:36 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:36.421+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:37.071 INFO:journalctl@ceph.osd.4.smithi121.stdout:Sep 11 03:55:36 smithi121 podman[114574]: 2024-09-11 03:55:36.78715407 +0000 UTC m=+5.466238907 container died eaeed3a07a7e0240f6797fe50c530af2849515c95e10afd62baa49f68c98a7e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-4, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-11T03:55:37.072 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:36 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:36.881+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:37.290 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:37 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:37.013+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:37.290 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:37 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:37.227+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:37.290 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:37 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:37.227+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:37.386 INFO:journalctl@ceph.osd.4.smithi121.stdout:Sep 11 03:55:37 smithi121 podman[114574]: 2024-09-11 03:55:37.246813401 +0000 UTC m=+5.925898230 container cleanup eaeed3a07a7e0240f6797fe50c530af2849515c95e10afd62baa49f68c98a7e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4) 2024-09-11T03:55:37.386 INFO:journalctl@ceph.osd.4.smithi121.stdout:Sep 11 03:55:37 smithi121 bash[114574]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-4 2024-09-11T03:55:37.386 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:37 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:37.315+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:37.386 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:37 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:37.315+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:37.672 INFO:journalctl@ceph.osd.4.smithi121.stdout:Sep 11 03:55:37 smithi121 podman[114589]: 2024-09-11 03:55:37.531641147 +0000 UTC m=+0.739563332 container remove eaeed3a07a7e0240f6797fe50c530af2849515c95e10afd62baa49f68c98a7e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-11T03:55:37.749 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:37 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:37.410+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:37.749 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:37 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:37.290+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:37.749 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:37 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:37.290+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:38.117 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:37 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:37.836+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:38.117 INFO:journalctl@ceph.osd.4.smithi121.stdout:Sep 11 03:55:37 smithi121 podman[114679]: 2024-09-11 03:55:37.764947222 +0000 UTC m=+0.015881687 image pull 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e 2024-09-11T03:55:38.118 INFO:journalctl@ceph.osd.4.smithi121.stdout:Sep 11 03:55:37 smithi121 podman[114679]: 2024-09-11 03:55:37.890905072 +0000 UTC m=+0.141839540 container create 062700312ffb683a191498a41920aa86acd1b915f24caefcaeec39811bce2b4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-4-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-11T03:55:38.118 INFO:journalctl@ceph.osd.4.smithi121.stdout:Sep 11 03:55:38 smithi121 podman[114679]: 2024-09-11 03:55:38.118462483 +0000 UTC m=+0.369396948 container init 062700312ffb683a191498a41920aa86acd1b915f24caefcaeec39811bce2b4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-4-deactivate, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-11T03:55:38.254 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:38 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:38.059+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:38.372 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:38 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:38.286+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:38.372 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:38 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:38.286+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:38.372 INFO:journalctl@ceph.osd.4.smithi121.stdout:Sep 11 03:55:38 smithi121 podman[114679]: 2024-09-11 03:55:38.122038805 +0000 UTC m=+0.372973274 container start 062700312ffb683a191498a41920aa86acd1b915f24caefcaeec39811bce2b4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-4-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-11T03:55:38.372 INFO:journalctl@ceph.osd.4.smithi121.stdout:Sep 11 03:55:38 smithi121 podman[114679]: 2024-09-11 03:55:38.235260111 +0000 UTC m=+0.486194597 container attach 062700312ffb683a191498a41920aa86acd1b915f24caefcaeec39811bce2b4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T03:55:38.373 INFO:journalctl@ceph.osd.4.smithi121.stdout:Sep 11 03:55:38 smithi121 podman[114704]: 2024-09-11 03:55:38.348915241 +0000 UTC m=+0.016801577 container died 062700312ffb683a191498a41920aa86acd1b915f24caefcaeec39811bce2b4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-09-11T03:55:38.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:38 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:38.268+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:38.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:38 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:38.268+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:38.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:38 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:38.402+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:38.728 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:38 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:38.255+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:38.728 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:38 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:38.255+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:38.921 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:38 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:38.855+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:38.922 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:38 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5[40512]: 2024-09-11T03:55:38.875+0000 7f3f09501640 -1 osd.5 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.167136+0000 front 2024-09-11T03:55:12.167048+0000 (oldest deadline 2024-09-11T03:55:38.067002+0000) 2024-09-11T03:55:38.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.4.service' 2024-09-11T03:55:39.017 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:55:39.017 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-11T03:55:39.017 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-11T03:55:39.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.5 2024-09-11T03:55:39.245 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:39 smithi121 systemd[1]: Stopping Ceph osd.5 for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:55:39.416 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:39 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:39.085+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:39.416 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:39 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:39.295+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:39.416 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:39 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:39.295+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:39.417 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:39 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:39.312+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:39.417 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:39 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:39.312+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:39.672 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:39 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5[40512]: 2024-09-11T03:55:39.333+0000 7f3f10ef4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:55:39.672 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:39 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5[40512]: 2024-09-11T03:55:39.333+0000 7f3f10ef4640 -1 osd.5 1005 *** Got signal Terminated *** 2024-09-11T03:55:39.672 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:39 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5[40512]: 2024-09-11T03:55:39.333+0000 7f3f10ef4640 -1 osd.5 1005 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T03:55:39.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:39 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:39.244+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:39.673 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:39 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:39.244+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:39.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:39 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:39.416+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:40.172 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:39 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5[40512]: 2024-09-11T03:55:39.847+0000 7f3f09501640 -1 osd.5 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.167136+0000 front 2024-09-11T03:55:12.167048+0000 (oldest deadline 2024-09-11T03:55:38.067002+0000) 2024-09-11T03:55:40.172 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:39 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:39.892+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:40.429 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:40 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:40.046+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:40.429 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:40 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:40.293+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:40.430 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:40 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:40.293+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:40.430 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:40 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:40.274+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:40.430 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:40 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:40.274+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:40.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:40 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:40.240+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:40.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:40 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:40.240+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:40.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:40 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:40.429+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:41.172 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:40 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5[40512]: 2024-09-11T03:55:40.857+0000 7f3f09501640 -1 osd.5 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.167136+0000 front 2024-09-11T03:55:12.167048+0000 (oldest deadline 2024-09-11T03:55:38.067002+0000) 2024-09-11T03:55:41.172 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:40 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:40.872+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:41.439 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:41 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:41.064+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:41.440 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:41 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:41.302+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:41.440 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:41 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:41.302+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:41.440 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:41 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:41.242+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:41.440 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:41 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:41.242+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:41.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:41 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:41.283+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:41.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:41 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:41.283+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:41.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:41 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:41.440+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:42.172 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:41 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:41.876+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:42.172 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:41 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5[40512]: 2024-09-11T03:55:41.841+0000 7f3f09501640 -1 osd.5 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.167136+0000 front 2024-09-11T03:55:12.167048+0000 (oldest deadline 2024-09-11T03:55:38.067002+0000) 2024-09-11T03:55:42.414 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:42 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:42.092+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:42.414 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:42 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:42.212+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:42.414 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:42 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:42.212+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:42.415 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:42 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:42.331+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:42.415 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:42 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:42.331+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:42.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:42 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:42.317+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:42.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:42 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:42.317+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:42.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:42 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:42.414+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:43.172 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:42 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:42.858+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:43.172 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:42 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5[40512]: 2024-09-11T03:55:42.825+0000 7f3f09501640 -1 osd.5 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.167136+0000 front 2024-09-11T03:55:12.167048+0000 (oldest deadline 2024-09-11T03:55:38.067002+0000) 2024-09-11T03:55:43.449 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:43 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:43.107+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:43.449 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:43 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:43.107+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:55:43.450 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:43 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:43.199+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:43.450 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:43 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:43.199+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:43.450 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:43 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:43.335+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:43.450 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:43 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:43.335+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:43.450 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:43 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:43.335+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:55:43.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:43 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:43.293+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:43.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:43 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:43.293+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:43.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:43 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:43.449+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:44.172 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:43 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:43.824+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:44.172 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:43 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:43.824+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.327260+0000 front 2024-09-11T03:55:18.327244+0000 (oldest deadline 2024-09-11T03:55:43.027108+0000) 2024-09-11T03:55:44.172 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:43 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5[40512]: 2024-09-11T03:55:43.806+0000 7f3f09501640 -1 osd.5 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.167136+0000 front 2024-09-11T03:55:12.167048+0000 (oldest deadline 2024-09-11T03:55:38.067002+0000) 2024-09-11T03:55:44.434 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:44 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:44.128+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:44.434 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:44 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:44.128+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:55:44.434 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:44 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:44.195+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:44.434 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:44 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:44.195+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:44.435 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:44 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:44.341+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:44.435 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:44 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:44.341+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:44.435 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:44 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:44.341+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:55:44.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:44 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:44.316+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:44.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:44 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:44.316+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:44.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:44 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:44.316+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.842915+0000 front 2024-09-11T03:55:18.842947+0000 (oldest deadline 2024-09-11T03:55:44.142941+0000) 2024-09-11T03:55:44.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:44 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:44.434+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:45.158 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:44 smithi121 podman[114795]: 2024-09-11 03:55:44.878216736 +0000 UTC m=+5.642121049 container died 6987d9f72e7689dce47c522c8faa454b755e2ec4b081e96f97e98cf93beda1b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-11T03:55:45.158 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:44 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:44.835+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:45.158 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:44 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:44.835+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.327260+0000 front 2024-09-11T03:55:18.327244+0000 (oldest deadline 2024-09-11T03:55:43.027108+0000) 2024-09-11T03:55:45.388 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:45 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:45.131+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:45.389 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:45 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:45.131+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:55:45.389 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:45 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:45.163+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:45.389 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:45 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:45.163+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:45.389 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:45 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:45.335+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:45.389 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:45 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:45.335+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:45.389 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:45 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:45.335+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:55:45.422 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:45 smithi121 podman[114795]: 2024-09-11 03:55:45.249723236 +0000 UTC m=+6.013627561 container cleanup 6987d9f72e7689dce47c522c8faa454b755e2ec4b081e96f97e98cf93beda1b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-11T03:55:45.422 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:45 smithi121 bash[114795]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5 2024-09-11T03:55:45.422 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:45 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:45.354+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:45.422 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:45 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:45.354+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:45.422 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:45 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:45.354+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.842915+0000 front 2024-09-11T03:55:18.842947+0000 (oldest deadline 2024-09-11T03:55:44.142941+0000) 2024-09-11T03:55:45.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:45 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:45.389+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:45.833 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:45 smithi121 podman[114811]: 2024-09-11 03:55:45.526088674 +0000 UTC m=+0.646445480 container remove 6987d9f72e7689dce47c522c8faa454b755e2ec4b081e96f97e98cf93beda1b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T03:55:46.161 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:45 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:45.849+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:46.161 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:45 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:45.849+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.327260+0000 front 2024-09-11T03:55:18.327244+0000 (oldest deadline 2024-09-11T03:55:43.027108+0000) 2024-09-11T03:55:46.162 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:45 smithi121 podman[114900]: 2024-09-11 03:55:45.736665205 +0000 UTC m=+0.015817754 image pull 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e 2024-09-11T03:55:46.162 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:45 smithi121 podman[114900]: 2024-09-11 03:55:45.835135893 +0000 UTC m=+0.114288487 container create 887eb7351cf1916e9edb12490dd1bef9914fb5f62321c6007acddcd7c7b7b954 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5-deactivate, ceph=True, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T03:55:46.162 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:46 smithi121 podman[114900]: 2024-09-11 03:55:46.079014112 +0000 UTC m=+0.358166661 container init 887eb7351cf1916e9edb12490dd1bef9914fb5f62321c6007acddcd7c7b7b954 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5-deactivate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-11T03:55:46.162 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:46 smithi121 podman[114900]: 2024-09-11 03:55:46.082343925 +0000 UTC m=+0.361496476 container start 887eb7351cf1916e9edb12490dd1bef9914fb5f62321c6007acddcd7c7b7b954 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True) 2024-09-11T03:55:46.162 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:46 smithi121 podman[114900]: 2024-09-11 03:55:46.162330386 +0000 UTC m=+0.441482943 container attach 887eb7351cf1916e9edb12490dd1bef9914fb5f62321c6007acddcd7c7b7b954 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T03:55:46.422 INFO:journalctl@ceph.osd.5.smithi121.stdout:Sep 11 03:55:46 smithi121 podman[114900]: 2024-09-11 03:55:46.271968865 +0000 UTC m=+0.551121418 container died 887eb7351cf1916e9edb12490dd1bef9914fb5f62321c6007acddcd7c7b7b954 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True) 2024-09-11T03:55:46.422 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:46 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:46.324+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:46.422 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:46 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:46.324+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:46.422 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:46 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:46.324+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.842915+0000 front 2024-09-11T03:55:18.842947+0000 (oldest deadline 2024-09-11T03:55:44.142941+0000) 2024-09-11T03:55:46.477 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:46 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:46.184+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:46.477 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:46 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:46.184+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:46.478 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:46 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:46.391+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:46.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:46 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:46.177+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:46.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:46 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:46.177+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:55:46.478 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:46 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:46.311+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:46.478 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:46 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:46.311+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:46.478 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:46 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:46.311+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:55:46.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.5.service' 2024-09-11T03:55:46.891 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:55:46.891 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-11T03:55:46.891 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-11T03:55:46.891 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.6 2024-09-11T03:55:47.172 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:46 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:46.815+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:47.172 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:46 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:46.815+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.327260+0000 front 2024-09-11T03:55:18.327244+0000 (oldest deadline 2024-09-11T03:55:43.027108+0000) 2024-09-11T03:55:47.172 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:46 smithi121 systemd[1]: Stopping Ceph osd.6 for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:55:47.477 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:47 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:47.203+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:47.477 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:47 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:47.203+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:47.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:47 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:47.187+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:47.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:47 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:47.187+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:55:47.478 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:47 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:47.374+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:47.478 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:47 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:47.374+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:55:47.478 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:47 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:47.293+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:47.479 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:47 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:47.293+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:47.479 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:47 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:47.293+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:55:47.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:47 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:47.315+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:47.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:47 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:47.315+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:47.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:47 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:47.315+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.842915+0000 front 2024-09-11T03:55:18.842947+0000 (oldest deadline 2024-09-11T03:55:44.142941+0000) 2024-09-11T03:55:47.672 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:47 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:47.217+0000 7f04862e5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:55:47.673 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:47 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:47.217+0000 7f04862e5640 -1 osd.6 1005 *** Got signal Terminated *** 2024-09-11T03:55:47.673 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:47 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:47.217+0000 7f04862e5640 -1 osd.6 1005 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T03:55:48.171 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:47 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:47.772+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:48.172 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:47 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:47.772+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.327260+0000 front 2024-09-11T03:55:18.327244+0000 (oldest deadline 2024-09-11T03:55:43.027108+0000) 2024-09-11T03:55:48.477 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:48 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:48.193+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:48.477 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:48 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:48.193+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:48.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:48 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:48.177+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:48.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:48 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:48.177+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:55:48.478 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:48 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:48.321+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:48.478 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:48 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:48.321+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:48.478 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:48 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:48.321+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:55:48.478 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:48 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:48.401+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:48.479 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:48 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:48.401+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:55:48.671 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:48 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:48.361+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:48.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:48 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:48.361+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:48.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:48 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:48.361+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.842915+0000 front 2024-09-11T03:55:18.842947+0000 (oldest deadline 2024-09-11T03:55:44.142941+0000) 2024-09-11T03:55:49.172 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:48 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:48.789+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:49.172 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:48 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:48.789+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.327260+0000 front 2024-09-11T03:55:18.327244+0000 (oldest deadline 2024-09-11T03:55:43.027108+0000) 2024-09-11T03:55:49.435 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:49 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:49.164+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:49.435 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:49 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:49.164+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:55:49.436 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:49 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:49.152+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:49.436 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:49 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:49.152+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:49.436 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:49 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:49.326+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:49.436 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:49 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:49.326+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:49.436 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:49 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:49.326+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:55:49.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:49 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:49.375+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:49.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:49 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:49.375+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:49.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:49 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:49.375+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.842915+0000 front 2024-09-11T03:55:18.842947+0000 (oldest deadline 2024-09-11T03:55:44.142941+0000) 2024-09-11T03:55:49.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:49 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:49.434+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:49.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:49 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:49.434+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:55:50.171 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:49 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:49.834+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:50.172 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:49 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:49.834+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.327260+0000 front 2024-09-11T03:55:18.327244+0000 (oldest deadline 2024-09-11T03:55:43.027108+0000) 2024-09-11T03:55:50.411 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:50 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:50.280+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:50.412 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:50 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:50.280+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:50.412 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:50 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:50.280+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:55:50.412 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:50 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:50.122+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:50.412 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:50 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:50.122+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:50.412 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:50 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:50.155+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:50.412 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:50 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:50.155+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:55:50.413 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:50 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:50.155+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:55:50.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:50 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:50.340+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:50.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:50 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:50.340+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:50.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:50 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:50.340+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.842915+0000 front 2024-09-11T03:55:18.842947+0000 (oldest deadline 2024-09-11T03:55:44.142941+0000) 2024-09-11T03:55:50.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:50 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:50.340+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.444029+0000 front 2024-09-11T03:55:26.444046+0000 (oldest deadline 2024-09-11T03:55:49.944032+0000) 2024-09-11T03:55:50.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:50 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:50.411+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:50.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:50 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:50.411+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:55:50.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:50 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:50.411+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:55:51.172 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:50 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:50.866+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:51.172 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:50 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:50.866+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.327260+0000 front 2024-09-11T03:55:18.327244+0000 (oldest deadline 2024-09-11T03:55:43.027108+0000) 2024-09-11T03:55:51.417 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:51 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:51.091+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:51.417 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:51 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:51.091+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:51.417 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:51 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:51.091+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.355202+0000 front 2024-09-11T03:55:26.355241+0000 (oldest deadline 2024-09-11T03:55:50.455320+0000) 2024-09-11T03:55:51.417 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:51 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:51.184+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:51.417 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:51 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:51.184+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:55:51.417 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:51 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:51.184+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:55:51.418 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:51 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:51.283+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:51.418 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:51 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:51.283+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:51.418 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:51 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:51.283+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:55:51.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:51 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:51.298+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:51.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:51 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:51.298+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:51.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:51 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:51.298+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.842915+0000 front 2024-09-11T03:55:18.842947+0000 (oldest deadline 2024-09-11T03:55:44.142941+0000) 2024-09-11T03:55:51.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:51 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:51.298+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.444029+0000 front 2024-09-11T03:55:26.444046+0000 (oldest deadline 2024-09-11T03:55:49.944032+0000) 2024-09-11T03:55:51.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:51 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:51.416+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:51.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:51 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:51.416+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:55:51.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:51 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:51.416+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:55:52.171 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:51 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:51.877+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:10.725877+0000 front 2024-09-11T03:55:10.726034+0000 (oldest deadline 2024-09-11T03:55:34.225914+0000) 2024-09-11T03:55:52.172 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:51 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6[46444]: 2024-09-11T03:55:51.877+0000 7f047e8f2640 -1 osd.6 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.327260+0000 front 2024-09-11T03:55:18.327244+0000 (oldest deadline 2024-09-11T03:55:43.027108+0000) 2024-09-11T03:55:52.432 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:52 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:52.136+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:52.432 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:52 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:52.136+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:55:52.432 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:52 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:52.136+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:55:52.433 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:52 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:52.101+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:52.433 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:52 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:52.101+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:52.433 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:52 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:52.101+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.355202+0000 front 2024-09-11T03:55:26.355241+0000 (oldest deadline 2024-09-11T03:55:50.455320+0000) 2024-09-11T03:55:52.433 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:52 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:52.322+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:52.433 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:52 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:52.322+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:52.433 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:52 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:52.322+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:55:52.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:52 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:52.296+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:52.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:52 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:52.296+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:52.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:52 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:52.296+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.842915+0000 front 2024-09-11T03:55:18.842947+0000 (oldest deadline 2024-09-11T03:55:44.142941+0000) 2024-09-11T03:55:52.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:52 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:52.296+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.444029+0000 front 2024-09-11T03:55:26.444046+0000 (oldest deadline 2024-09-11T03:55:49.944032+0000) 2024-09-11T03:55:52.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:52 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:52.432+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:52.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:52 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:52.432+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:55:52.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:52 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:52.432+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:55:52.993 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:52 smithi121 podman[115004]: 2024-09-11 03:55:52.72373031 +0000 UTC m=+5.614554306 container died 2df7252d6d5eaba67dbc935f35e57482836a349ab0fa1b852d4bb01d3148f66d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-11T03:55:53.319 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:53 smithi121 podman[115004]: 2024-09-11 03:55:53.084536915 +0000 UTC m=+5.975360911 container cleanup 2df7252d6d5eaba67dbc935f35e57482836a349ab0fa1b852d4bb01d3148f66d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-11T03:55:53.319 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:53 smithi121 bash[115004]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6 2024-09-11T03:55:53.418 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:53 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:53.184+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:53.418 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:53 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:53.184+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:55:53.418 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:53 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:53.184+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:55:53.419 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:53 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:53.075+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:53.419 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:53 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:53.075+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:53.419 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:53 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:53.075+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.355202+0000 front 2024-09-11T03:55:26.355241+0000 (oldest deadline 2024-09-11T03:55:50.455320+0000) 2024-09-11T03:55:53.419 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:53 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:53.280+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:53.419 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:53 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:53.280+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:53.419 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:53 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:53.280+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:55:53.672 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:53 smithi121 podman[115019]: 2024-09-11 03:55:53.369154559 +0000 UTC m=+0.641092172 container remove 2df7252d6d5eaba67dbc935f35e57482836a349ab0fa1b852d4bb01d3148f66d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-11T03:55:53.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:53 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:53.317+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:53.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:53 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:53.317+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:53.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:53 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:53.317+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.842915+0000 front 2024-09-11T03:55:18.842947+0000 (oldest deadline 2024-09-11T03:55:44.142941+0000) 2024-09-11T03:55:53.673 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:53 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:53.317+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.444029+0000 front 2024-09-11T03:55:26.444046+0000 (oldest deadline 2024-09-11T03:55:49.944032+0000) 2024-09-11T03:55:53.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:53 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:53.418+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:53.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:53 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:53.418+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:55:53.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:53 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:53.418+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:55:53.955 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:53 smithi121 podman[115106]: 2024-09-11 03:55:53.595891653 +0000 UTC m=+0.015901949 image pull 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e 2024-09-11T03:55:53.955 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:53 smithi121 podman[115106]: 2024-09-11 03:55:53.703311296 +0000 UTC m=+0.123321646 container create cd71bc981d04dc67e3b786cb6f6e0a59b8c7434f4f42523383503bab86104dbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6-deactivate, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-09-11T03:55:53.955 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:53 smithi121 podman[115106]: 2024-09-11 03:55:53.905501046 +0000 UTC m=+0.325511327 container init cd71bc981d04dc67e3b786cb6f6e0a59b8c7434f4f42523383503bab86104dbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4) 2024-09-11T03:55:53.955 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:53 smithi121 podman[115106]: 2024-09-11 03:55:53.90891078 +0000 UTC m=+0.328921061 container start cd71bc981d04dc67e3b786cb6f6e0a59b8c7434f4f42523383503bab86104dbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-11T03:55:54.323 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:53 smithi121 podman[115106]: 2024-09-11 03:55:53.955419164 +0000 UTC m=+0.375429463 container attach cd71bc981d04dc67e3b786cb6f6e0a59b8c7434f4f42523383503bab86104dbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6-deactivate, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-11T03:55:54.323 INFO:journalctl@ceph.osd.6.smithi121.stdout:Sep 11 03:55:54 smithi121 podman[115106]: 2024-09-11 03:55:54.093196309 +0000 UTC m=+0.513206601 container died cd71bc981d04dc67e3b786cb6f6e0a59b8c7434f4f42523383503bab86104dbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-11T03:55:54.401 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:54 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:54.192+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:54.401 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:54 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:54.192+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:55:54.401 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:54 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:54.192+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:55:54.401 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:54 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:54.096+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:54.402 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:54 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:54.096+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:54.402 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:54 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:54.096+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.355202+0000 front 2024-09-11T03:55:26.355241+0000 (oldest deadline 2024-09-11T03:55:50.455320+0000) 2024-09-11T03:55:54.402 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:54 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:54.307+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:54.402 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:54 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:54.307+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:54.402 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:54 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:54.307+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:55:54.642 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:54 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:54.322+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:54.642 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:54 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:54.322+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:54.642 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:54 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:54.322+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.842915+0000 front 2024-09-11T03:55:18.842947+0000 (oldest deadline 2024-09-11T03:55:44.142941+0000) 2024-09-11T03:55:54.642 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:54 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:54.322+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.444029+0000 front 2024-09-11T03:55:26.444046+0000 (oldest deadline 2024-09-11T03:55:49.944032+0000) 2024-09-11T03:55:54.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.6.service' 2024-09-11T03:55:54.701 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:55:54.701 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-11T03:55:54.701 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-11T03:55:54.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.7 2024-09-11T03:55:54.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:54 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:54.401+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:54.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:54 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:54.401+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:55:54.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:54 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:54.401+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:55:54.922 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:54 smithi121 systemd[1]: Stopping Ceph osd.7 for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:55:55.333 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:55 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:55.019+0000 7f166f38d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:55:55.333 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:55 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:55.019+0000 7f166f38d640 -1 osd.7 1005 *** Got signal Terminated *** 2024-09-11T03:55:55.333 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:55 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:55.019+0000 7f166f38d640 -1 osd.7 1005 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T03:55:55.394 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:55 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:55.237+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:55.394 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:55 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:55.237+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:55:55.394 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:55 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:55.237+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:55:55.394 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:55 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:55.289+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:55.394 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:55 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:55.289+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:55.394 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:55 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:55.289+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:55:55.395 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:55 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:55.067+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:55.395 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:55 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:55.067+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:55.395 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:55 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:55.067+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.355202+0000 front 2024-09-11T03:55:26.355241+0000 (oldest deadline 2024-09-11T03:55:50.455320+0000) 2024-09-11T03:55:55.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:55 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:55.333+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:55.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:55 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:55.333+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:55.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:55 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:55.333+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.842915+0000 front 2024-09-11T03:55:18.842947+0000 (oldest deadline 2024-09-11T03:55:44.142941+0000) 2024-09-11T03:55:55.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:55 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:55.333+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.444029+0000 front 2024-09-11T03:55:26.444046+0000 (oldest deadline 2024-09-11T03:55:49.944032+0000) 2024-09-11T03:55:55.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:55 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:55.393+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:55.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:55 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:55.393+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:55:55.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:55 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:55.393+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:55:56.371 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:56 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:56.272+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:56.371 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:56 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:56.272+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:55:56.371 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:56 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:56.272+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:55:56.371 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:56 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:56.104+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:56.371 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:56 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:56.104+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:56.371 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:56 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:56.104+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.355202+0000 front 2024-09-11T03:55:26.355241+0000 (oldest deadline 2024-09-11T03:55:50.455320+0000) 2024-09-11T03:55:56.372 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:56 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:56.307+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:56.372 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:56 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:56.307+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:56.372 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:56 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:56.307+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:55:56.573 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:56 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:56.294+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:56.573 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:56 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:56.294+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:56.573 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:56 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:56.294+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.842915+0000 front 2024-09-11T03:55:18.842947+0000 (oldest deadline 2024-09-11T03:55:44.142941+0000) 2024-09-11T03:55:56.573 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:56 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:56.294+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.444029+0000 front 2024-09-11T03:55:26.444046+0000 (oldest deadline 2024-09-11T03:55:49.944032+0000) 2024-09-11T03:55:56.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:56 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:56.370+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:56.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:56 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:56.370+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:55:56.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:56 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:56.370+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:55:57.477 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:57 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:57.123+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:57.478 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:57 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:57.123+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:57.478 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:57 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:57.123+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.355202+0000 front 2024-09-11T03:55:26.355241+0000 (oldest deadline 2024-09-11T03:55:50.455320+0000) 2024-09-11T03:55:57.478 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:57 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:57.327+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:57.478 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:57 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:57.327+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:57.478 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:57 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:57.327+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:55:57.478 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:57 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:57.355+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:57.479 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:57 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:57.355+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:55:57.479 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:57 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:57.355+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:55:57.479 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:57 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:57.355+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:55:57.479 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:57 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:57.248+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:57.479 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:57 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:57.248+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:55:57.479 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:57 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:57.248+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:55:57.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:57 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:57.270+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:57.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:57 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:57.270+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:57.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:57 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:57.270+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.842915+0000 front 2024-09-11T03:55:18.842947+0000 (oldest deadline 2024-09-11T03:55:44.142941+0000) 2024-09-11T03:55:57.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:57 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:57.270+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.444029+0000 front 2024-09-11T03:55:26.444046+0000 (oldest deadline 2024-09-11T03:55:49.944032+0000) 2024-09-11T03:55:58.349 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:58 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:58.096+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:58.349 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:58 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:58.096+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:58.349 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:58 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:58.096+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.355202+0000 front 2024-09-11T03:55:26.355241+0000 (oldest deadline 2024-09-11T03:55:50.455320+0000) 2024-09-11T03:55:58.350 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:58 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:58.281+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:58.350 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:58 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:58.281+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:55:58.350 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:58 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:58.281+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:55:58.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:58 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:58.305+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:58.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:58 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:58.305+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:58.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:58 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:58.305+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.842915+0000 front 2024-09-11T03:55:18.842947+0000 (oldest deadline 2024-09-11T03:55:44.142941+0000) 2024-09-11T03:55:58.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:58 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:58.305+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.444029+0000 front 2024-09-11T03:55:26.444046+0000 (oldest deadline 2024-09-11T03:55:49.944032+0000) 2024-09-11T03:55:58.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:58 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:58.305+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:31.645070+0000 front 2024-09-11T03:55:31.645123+0000 (oldest deadline 2024-09-11T03:55:57.545115+0000) 2024-09-11T03:55:58.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:58 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:58.348+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:58.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:58 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:58.348+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:58.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:58 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:58.348+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:55:58.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:58 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:58.402+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:58.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:58 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:58.402+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:55:58.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:58 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:58.402+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:55:58.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:58 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:58.402+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:55:59.411 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:59 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:59.237+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:55:59.411 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:59 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:59.237+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:55:59.411 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:55:59 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:55:59.237+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:55:59.411 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:59 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:59.126+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:55:59.411 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:59 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:59.126+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:55:59.412 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:55:59 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:55:59.126+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.355202+0000 front 2024-09-11T03:55:26.355241+0000 (oldest deadline 2024-09-11T03:55:50.455320+0000) 2024-09-11T03:55:59.412 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:59 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:59.308+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:55:59.412 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:59 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:59.308+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:55:59.412 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:55:59 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:55:59.308+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:55:59.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:59 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:59.347+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.039568+0000 front 2024-09-11T03:55:02.039611+0000 (oldest deadline 2024-09-11T03:55:26.139580+0000) 2024-09-11T03:55:59.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:59 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:59.347+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.440555+0000 front 2024-09-11T03:55:11.440427+0000 (oldest deadline 2024-09-11T03:55:34.841227+0000) 2024-09-11T03:55:59.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:59 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:59.347+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:18.842915+0000 front 2024-09-11T03:55:18.842947+0000 (oldest deadline 2024-09-11T03:55:44.142941+0000) 2024-09-11T03:55:59.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:59 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:59.347+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.444029+0000 front 2024-09-11T03:55:26.444046+0000 (oldest deadline 2024-09-11T03:55:49.944032+0000) 2024-09-11T03:55:59.672 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:55:59 smithi121 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7[52533]: 2024-09-11T03:55:59.347+0000 7f166799a640 -1 osd.7 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:31.645070+0000 front 2024-09-11T03:55:31.645123+0000 (oldest deadline 2024-09-11T03:55:57.545115+0000) 2024-09-11T03:55:59.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:59 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:59.411+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:55:59.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:59 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:59.411+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:55:59.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:59 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:59.411+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:55:59.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:55:59 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:55:59.411+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:00.434 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:00 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:00.140+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:56:00.434 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:00 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:00.140+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:56:00.434 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:00 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:00.140+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.355202+0000 front 2024-09-11T03:55:26.355241+0000 (oldest deadline 2024-09-11T03:55:50.455320+0000) 2024-09-11T03:56:00.434 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:00 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:00.257+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:00.435 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:00 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:00.257+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:00.435 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:00 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:00.257+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:00.435 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:00 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:00.257+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:00.435 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:00 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:00.353+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:56:00.435 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:00 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:00.353+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:56:00.435 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:00 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:00.353+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:56:00.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:00 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:00.434+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:00.727 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:00 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:00.434+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:00.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:00 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:00.434+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:00.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:00 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:00.434+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:00.768 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:56:00 smithi121 podman[115208]: 2024-09-11 03:56:00.458749688 +0000 UTC m=+5.537212189 container died e9bb64790d8fed93753b90b81f93d5bbe77047314927fb07e624641fd6626dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, CEPH_POINT_RELEASE=) 2024-09-11T03:56:00.768 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:56:00 smithi121 podman[115208]: 2024-09-11 03:56:00.769491422 +0000 UTC m=+5.847953917 container cleanup e9bb64790d8fed93753b90b81f93d5bbe77047314927fb07e624641fd6626dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-11T03:56:01.171 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:56:00 smithi121 bash[115208]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7 2024-09-11T03:56:01.172 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:56:00 smithi121 podman[115235]: 2024-09-11 03:56:00.970640465 +0000 UTC m=+0.513330744 container remove e9bb64790d8fed93753b90b81f93d5bbe77047314927fb07e624641fd6626dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T03:56:01.477 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:01 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:01.249+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:01.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:01 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:01.249+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:01.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:01 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:01.249+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:01.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:01 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:01.249+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:01.478 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:01 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:01.364+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:56:01.478 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:01 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:01.364+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:56:01.478 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:01 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:01.364+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:56:01.478 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:01 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:01.364+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.343461+0000 front 2024-09-11T03:55:36.343489+0000 (oldest deadline 2024-09-11T03:56:00.443326+0000) 2024-09-11T03:56:01.479 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:01 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:01.188+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:56:01.479 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:01 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:01.188+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:56:01.479 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:01 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:01.188+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.355202+0000 front 2024-09-11T03:55:26.355241+0000 (oldest deadline 2024-09-11T03:55:50.455320+0000) 2024-09-11T03:56:01.479 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:01 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:01.188+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:32.056984+0000 front 2024-09-11T03:55:32.056808+0000 (oldest deadline 2024-09-11T03:56:00.857382+0000) 2024-09-11T03:56:01.479 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:01 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:01.406+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:01.479 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:01 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:01.406+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:01.479 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:01 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:01.406+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:01.479 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:01 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:01.406+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:01.664 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:56:01 smithi121 podman[115326]: 2024-09-11 03:56:01.278969759 +0000 UTC m=+0.015427446 image pull 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e 2024-09-11T03:56:01.665 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:56:01 smithi121 podman[115326]: 2024-09-11 03:56:01.38790534 +0000 UTC m=+0.124363026 container create 2e8fa5c4766b07519776384fdff07a35f89d985218bcf6de9d6c8f77ca33ac48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-11T03:56:01.665 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:56:01 smithi121 podman[115326]: 2024-09-11 03:56:01.590428039 +0000 UTC m=+0.326885728 container init 2e8fa5c4766b07519776384fdff07a35f89d985218bcf6de9d6c8f77ca33ac48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-11T03:56:01.665 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:56:01 smithi121 podman[115326]: 2024-09-11 03:56:01.593538318 +0000 UTC m=+0.329996012 container start 2e8fa5c4766b07519776384fdff07a35f89d985218bcf6de9d6c8f77ca33ac48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-11T03:56:01.665 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:56:01 smithi121 podman[115326]: 2024-09-11 03:56:01.665232762 +0000 UTC m=+0.401690457 container attach 2e8fa5c4766b07519776384fdff07a35f89d985218bcf6de9d6c8f77ca33ac48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T03:56:01.922 INFO:journalctl@ceph.osd.7.smithi121.stdout:Sep 11 03:56:01 smithi121 podman[115326]: 2024-09-11 03:56:01.780275868 +0000 UTC m=+0.516733555 container died 2e8fa5c4766b07519776384fdff07a35f89d985218bcf6de9d6c8f77ca33ac48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-7-deactivate, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-11T03:56:02.389 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.7.service' 2024-09-11T03:56:02.427 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:56:02.427 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-11T03:56:02.427 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-09-11T03:56:02.428 DEBUG:teuthology.orchestra.run.smithi175:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.8 2024-09-11T03:56:02.727 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:02.297+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:02.727 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:02.297+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:02.727 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:02.297+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:02.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:02.297+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:02.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:02.395+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:56:02.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:02.395+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:56:02.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:02.395+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:56:02.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:02.395+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.343461+0000 front 2024-09-11T03:55:36.343489+0000 (oldest deadline 2024-09-11T03:56:00.443326+0000) 2024-09-11T03:56:02.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:02.447+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:02.729 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:02.447+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:02.729 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:02.447+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:02.729 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:02.447+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:02.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:02.238+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:56:02.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:02.238+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:56:02.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:02.238+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.355202+0000 front 2024-09-11T03:55:26.355241+0000 (oldest deadline 2024-09-11T03:55:50.455320+0000) 2024-09-11T03:56:02.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:02.238+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:32.056984+0000 front 2024-09-11T03:55:32.056808+0000 (oldest deadline 2024-09-11T03:56:00.857382+0000) 2024-09-11T03:56:02.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:02 smithi175 systemd[1]: Stopping Ceph osd.8 for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:56:03.227 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:02.737+0000 7f7813bfe640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:56:03.227 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:02.737+0000 7f7813bfe640 -1 osd.8 1005 *** Got signal Terminated *** 2024-09-11T03:56:03.227 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:02 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:02.737+0000 7f7813bfe640 -1 osd.8 1005 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T03:56:03.727 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:03 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:03.254+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:03.727 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:03 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:03.254+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:03.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:03 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:03.254+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:03.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:03 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:03.254+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:03.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:03 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:03.429+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:56:03.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:03 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:03.429+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:56:03.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:03 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:03.429+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:56:03.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:03 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:03.429+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.343461+0000 front 2024-09-11T03:55:36.343489+0000 (oldest deadline 2024-09-11T03:56:00.443326+0000) 2024-09-11T03:56:03.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:03 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:03.258+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:56:03.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:03 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:03.258+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:56:03.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:03 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:03.258+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.355202+0000 front 2024-09-11T03:55:26.355241+0000 (oldest deadline 2024-09-11T03:55:50.455320+0000) 2024-09-11T03:56:03.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:03 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:03.258+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:32.056984+0000 front 2024-09-11T03:55:32.056808+0000 (oldest deadline 2024-09-11T03:56:00.857382+0000) 2024-09-11T03:56:03.729 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:03 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:03.488+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:03.729 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:03 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:03.488+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:03.729 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:03 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:03.488+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:03.729 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:03 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:03.488+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:04.727 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:04 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:04.259+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:04.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:04 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:04.259+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:04.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:04 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:04.259+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:04.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:04 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:04.259+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:04.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:04 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:04.402+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:56:04.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:04 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:04.402+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:56:04.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:04 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:04.402+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:56:04.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:04 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:04.402+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.343461+0000 front 2024-09-11T03:55:36.343489+0000 (oldest deadline 2024-09-11T03:56:00.443326+0000) 2024-09-11T03:56:04.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:04 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:04.261+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:56:04.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:04 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:04.261+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:56:04.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:04 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:04.261+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.355202+0000 front 2024-09-11T03:55:26.355241+0000 (oldest deadline 2024-09-11T03:55:50.455320+0000) 2024-09-11T03:56:04.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:04 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:04.261+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:32.056984+0000 front 2024-09-11T03:55:32.056808+0000 (oldest deadline 2024-09-11T03:56:00.857382+0000) 2024-09-11T03:56:04.729 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:04 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:04.479+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:04.729 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:04 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:04.479+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:04.729 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:04 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:04.479+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:04.729 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:04 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:04.479+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:05.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:05 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:05.356+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:56:05.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:05 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:05.356+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:56:05.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:05 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:05.356+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:56:05.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:05 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:05.356+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.343461+0000 front 2024-09-11T03:55:36.343489+0000 (oldest deadline 2024-09-11T03:56:00.443326+0000) 2024-09-11T03:56:05.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:05 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:05.298+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:05.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:05 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:05.298+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:05.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:05 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:05.298+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:05.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:05 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:05.298+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:05.728 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:05 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:05.271+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:56:05.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:05 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:05.271+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:56:05.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:05 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:05.271+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.355202+0000 front 2024-09-11T03:55:26.355241+0000 (oldest deadline 2024-09-11T03:55:50.455320+0000) 2024-09-11T03:56:05.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:05 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:05.271+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:32.056984+0000 front 2024-09-11T03:55:32.056808+0000 (oldest deadline 2024-09-11T03:56:00.857382+0000) 2024-09-11T03:56:05.729 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:05 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:05.506+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:05.729 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:05 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:05.506+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:05.730 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:05 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:05.506+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:05.730 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:05 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:05.506+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:06.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:06.318+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:56:06.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:06.318+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:56:06.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:06.318+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:56:06.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:06.318+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.343461+0000 front 2024-09-11T03:55:36.343489+0000 (oldest deadline 2024-09-11T03:56:00.443326+0000) 2024-09-11T03:56:06.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:06.318+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:43.344431+0000 front 2024-09-11T03:55:43.344326+0000 (oldest deadline 2024-09-11T03:56:06.244101+0000) 2024-09-11T03:56:06.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:06.289+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:06.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:06.289+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:06.729 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:06.289+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:06.729 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:06.289+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:06.729 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:06.289+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:06.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:06.275+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:56:06.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:06.275+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:56:06.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:06.275+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.355202+0000 front 2024-09-11T03:55:26.355241+0000 (oldest deadline 2024-09-11T03:55:50.455320+0000) 2024-09-11T03:56:06.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:06.275+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:32.056984+0000 front 2024-09-11T03:55:32.056808+0000 (oldest deadline 2024-09-11T03:56:00.857382+0000) 2024-09-11T03:56:06.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:06.275+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.857997+0000 front 2024-09-11T03:55:40.857987+0000 (oldest deadline 2024-09-11T03:56:06.157693+0000) 2024-09-11T03:56:06.730 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:06.521+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:06.730 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:06.521+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:06.730 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:06.521+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:06.730 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:06.521+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:06.730 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:06 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:06.521+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:41.538541+0000 front 2024-09-11T03:55:41.538524+0000 (oldest deadline 2024-09-11T03:56:06.238435+0000) 2024-09-11T03:56:07.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:07.295+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:56:07.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:07.295+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:56:07.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:07.295+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:56:07.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:07.295+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.343461+0000 front 2024-09-11T03:55:36.343489+0000 (oldest deadline 2024-09-11T03:56:00.443326+0000) 2024-09-11T03:56:07.728 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:07.295+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:43.344431+0000 front 2024-09-11T03:55:43.344326+0000 (oldest deadline 2024-09-11T03:56:06.244101+0000) 2024-09-11T03:56:07.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:07.255+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:07.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:07.255+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:07.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:07.255+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:07.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:07.255+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:07.729 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:07.255+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:07.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:07.269+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:03.850251+0000 front 2024-09-11T03:55:03.850209+0000 (oldest deadline 2024-09-11T03:55:26.150374+0000) 2024-09-11T03:56:07.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:07.269+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:12.552100+0000 front 2024-09-11T03:55:12.552072+0000 (oldest deadline 2024-09-11T03:55:34.252060+0000) 2024-09-11T03:56:07.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:07.269+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.355202+0000 front 2024-09-11T03:55:26.355241+0000 (oldest deadline 2024-09-11T03:55:50.455320+0000) 2024-09-11T03:56:07.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:07.269+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:32.056984+0000 front 2024-09-11T03:55:32.056808+0000 (oldest deadline 2024-09-11T03:56:00.857382+0000) 2024-09-11T03:56:07.729 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8[35516]: 2024-09-11T03:56:07.269+0000 7f780c20b640 -1 osd.8 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.857997+0000 front 2024-09-11T03:55:40.857987+0000 (oldest deadline 2024-09-11T03:56:06.157693+0000) 2024-09-11T03:56:07.729 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:07.491+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:07.730 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:07.491+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:07.730 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:07.491+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:07.730 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:07.491+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:07.730 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:07 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:07.491+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:41.538541+0000 front 2024-09-11T03:55:41.538524+0000 (oldest deadline 2024-09-11T03:56:06.238435+0000) 2024-09-11T03:56:08.539 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:08 smithi175 podman[112615]: 2024-09-11 03:56:08.247655299 +0000 UTC m=+5.615438671 container stop abf2a720c30ff5699f856a0e2ed9555885474b19c0b1720b56b370e1710ff07e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-11T03:56:08.539 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:08 smithi175 podman[112615]: 2024-09-11 03:56:08.273260009 +0000 UTC m=+5.641043381 container died abf2a720c30ff5699f856a0e2ed9555885474b19c0b1720b56b370e1710ff07e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-11T03:56:08.539 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:08 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:08.266+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:56:08.539 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:08 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:08.266+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:56:08.539 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:08 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:08.266+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:56:08.539 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:08 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:08.266+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.343461+0000 front 2024-09-11T03:55:36.343489+0000 (oldest deadline 2024-09-11T03:56:00.443326+0000) 2024-09-11T03:56:08.539 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:08 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:08.266+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:43.344431+0000 front 2024-09-11T03:55:43.344326+0000 (oldest deadline 2024-09-11T03:56:06.244101+0000) 2024-09-11T03:56:08.540 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:08 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:08.489+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:08.540 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:08 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:08.489+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:08.540 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:08 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:08.489+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:08.540 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:08 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:08.489+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:08.540 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:08 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:08.489+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:41.538541+0000 front 2024-09-11T03:55:41.538524+0000 (oldest deadline 2024-09-11T03:56:06.238435+0000) 2024-09-11T03:56:08.540 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:08 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:08.302+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:08.540 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:08 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:08.302+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:08.540 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:08 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:08.302+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:08.541 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:08 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:08.302+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:08.541 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:08 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:08.302+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:08.914 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:08 smithi175 podman[112615]: 2024-09-11 03:56:08.730373675 +0000 UTC m=+6.098157088 container cleanup abf2a720c30ff5699f856a0e2ed9555885474b19c0b1720b56b370e1710ff07e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4) 2024-09-11T03:56:08.914 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:08 smithi175 bash[112615]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8 2024-09-11T03:56:09.227 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:08 smithi175 podman[112631]: 2024-09-11 03:56:08.915145697 +0000 UTC m=+0.659360722 container remove abf2a720c30ff5699f856a0e2ed9555885474b19c0b1720b56b370e1710ff07e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-11T03:56:09.601 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:09 smithi175 podman[112722]: 2024-09-11 03:56:09.236498035 +0000 UTC m=+0.015702813 image pull 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e 2024-09-11T03:56:09.601 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:09 smithi175 podman[112722]: 2024-09-11 03:56:09.374247816 +0000 UTC m=+0.153452578 container create 1659b322fb206f7117fd685f11dc180fa57e38d39a23f0e98d6dc9b9f5b53f23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-11T03:56:09.601 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:09 smithi175 podman[112722]: 2024-09-11 03:56:09.601995903 +0000 UTC m=+0.381200673 container init 1659b322fb206f7117fd685f11dc180fa57e38d39a23f0e98d6dc9b9f5b53f23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-11T03:56:09.602 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:09 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:09.472+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:09.602 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:09 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:09.472+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:09.602 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:09 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:09.472+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:09.602 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:09 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:09.472+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:09.602 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:09 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:09.472+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:41.538541+0000 front 2024-09-11T03:55:41.538524+0000 (oldest deadline 2024-09-11T03:56:06.238435+0000) 2024-09-11T03:56:09.602 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:09 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:09.277+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:56:09.602 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:09 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:09.277+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:56:09.603 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:09 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:09.277+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:56:09.603 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:09 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:09.277+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.343461+0000 front 2024-09-11T03:55:36.343489+0000 (oldest deadline 2024-09-11T03:56:00.443326+0000) 2024-09-11T03:56:09.603 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:09 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:09.277+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:43.344431+0000 front 2024-09-11T03:55:43.344326+0000 (oldest deadline 2024-09-11T03:56:06.244101+0000) 2024-09-11T03:56:09.603 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:09 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:09.265+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:09.603 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:09 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:09.265+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:09.603 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:09 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:09.265+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:09.603 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:09 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:09.265+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:09.603 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:09 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:09.265+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:09.977 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:09 smithi175 podman[112722]: 2024-09-11 03:56:09.605803933 +0000 UTC m=+0.385008696 container start 1659b322fb206f7117fd685f11dc180fa57e38d39a23f0e98d6dc9b9f5b53f23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_CLEAN=True) 2024-09-11T03:56:09.978 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:09 smithi175 podman[112722]: 2024-09-11 03:56:09.701612747 +0000 UTC m=+0.480817536 container attach 1659b322fb206f7117fd685f11dc180fa57e38d39a23f0e98d6dc9b9f5b53f23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T03:56:09.978 INFO:journalctl@ceph.osd.8.smithi175.stdout:Sep 11 03:56:09 smithi175 podman[112722]: 2024-09-11 03:56:09.791708388 +0000 UTC m=+0.570913162 container died 1659b322fb206f7117fd685f11dc180fa57e38d39a23f0e98d6dc9b9f5b53f23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True) 2024-09-11T03:56:10.326 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:10.235+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:10.327 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:10.235+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:10.327 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:10.235+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:10.327 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:10.235+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:10.327 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:10.235+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:10.354 DEBUG:teuthology.orchestra.run.smithi175:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.8.service' 2024-09-11T03:56:10.394 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:56:10.394 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-09-11T03:56:10.394 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-09-11T03:56:10.394 DEBUG:teuthology.orchestra.run.smithi175:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.9 2024-09-11T03:56:10.605 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:10.477+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:10.605 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:10.477+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:10.605 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:10.477+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:10.605 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:10.477+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:10.606 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:10.477+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:41.538541+0000 front 2024-09-11T03:55:41.538524+0000 (oldest deadline 2024-09-11T03:56:06.238435+0000) 2024-09-11T03:56:10.606 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:10.326+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:56:10.606 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:10.326+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:56:10.606 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:10.326+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:56:10.606 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:10.326+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.343461+0000 front 2024-09-11T03:55:36.343489+0000 (oldest deadline 2024-09-11T03:56:00.443326+0000) 2024-09-11T03:56:10.606 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:10.326+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:43.344431+0000 front 2024-09-11T03:55:43.344326+0000 (oldest deadline 2024-09-11T03:56:06.244101+0000) 2024-09-11T03:56:10.606 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:10 smithi175 systemd[1]: Stopping Ceph osd.9 for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:56:10.977 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:10.740+0000 7fbd93517640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:56:10.977 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:10.740+0000 7fbd93517640 -1 osd.9 1005 *** Got signal Terminated *** 2024-09-11T03:56:10.977 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:10 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:10.740+0000 7fbd93517640 -1 osd.9 1005 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T03:56:11.524 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:11 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:11.235+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:11.524 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:11 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:11.235+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:11.524 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:11 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:11.235+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:11.524 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:11 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:11.235+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:11.524 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:11 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:11.235+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:11.524 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:11 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:11.364+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:56:11.524 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:11 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:11.364+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:56:11.525 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:11 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:11.364+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:56:11.525 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:11 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:11.364+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.343461+0000 front 2024-09-11T03:55:36.343489+0000 (oldest deadline 2024-09-11T03:56:00.443326+0000) 2024-09-11T03:56:11.525 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:11 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:11.364+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:43.344431+0000 front 2024-09-11T03:55:43.344326+0000 (oldest deadline 2024-09-11T03:56:06.244101+0000) 2024-09-11T03:56:11.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:11 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:11.523+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:11.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:11 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:11.523+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:11.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:11 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:11.523+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:11.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:11 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:11.523+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:11.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:11 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:11.523+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:41.538541+0000 front 2024-09-11T03:55:41.538524+0000 (oldest deadline 2024-09-11T03:56:06.238435+0000) 2024-09-11T03:56:12.570 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:12 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:12.258+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:12.570 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:12 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:12.258+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:12.570 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:12 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:12.258+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:12.570 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:12 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:12.258+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:12.570 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:12 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:12.258+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:12.570 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:12 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:12.333+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:56:12.571 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:12 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:12.333+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:56:12.571 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:12 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:12.333+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:56:12.571 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:12 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:12.333+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.343461+0000 front 2024-09-11T03:55:36.343489+0000 (oldest deadline 2024-09-11T03:56:00.443326+0000) 2024-09-11T03:56:12.571 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:12 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:12.333+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:43.344431+0000 front 2024-09-11T03:55:43.344326+0000 (oldest deadline 2024-09-11T03:56:06.244101+0000) 2024-09-11T03:56:12.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:12 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:12.569+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:12.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:12 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:12.569+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:12.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:12 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:12.569+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:12.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:12 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:12.569+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:12.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:12 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:12.569+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:41.538541+0000 front 2024-09-11T03:55:41.538524+0000 (oldest deadline 2024-09-11T03:56:06.238435+0000) 2024-09-11T03:56:13.575 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:13 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:13.305+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:13.576 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:13 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:13.305+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:13.576 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:13 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:13.305+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:13.576 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:13 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:13.305+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:13.576 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:13 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:13.305+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:13.576 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:13 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:13.289+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:56:13.576 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:13 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:13.289+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:56:13.577 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:13 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:13.289+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:56:13.577 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:13 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:13.289+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.343461+0000 front 2024-09-11T03:55:36.343489+0000 (oldest deadline 2024-09-11T03:56:00.443326+0000) 2024-09-11T03:56:13.577 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:13 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:13.289+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:43.344431+0000 front 2024-09-11T03:55:43.344326+0000 (oldest deadline 2024-09-11T03:56:06.244101+0000) 2024-09-11T03:56:13.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:13 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:13.574+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:13.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:13 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:13.574+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:13.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:13 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:13.574+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:13.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:13 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:13.574+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:13.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:13 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:13.574+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:41.538541+0000 front 2024-09-11T03:55:41.538524+0000 (oldest deadline 2024-09-11T03:56:06.238435+0000) 2024-09-11T03:56:14.604 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:14 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:14.325+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:14.604 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:14 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:14.325+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:14.604 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:14 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:14.325+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:14.604 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:14 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:14.325+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:14.604 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:14 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:14.325+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:14.604 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:14 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:14.325+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.980485+0000 front 2024-09-11T03:55:51.980351+0000 (oldest deadline 2024-09-11T03:56:13.680325+0000) 2024-09-11T03:56:14.605 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:14 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:14.273+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:56:14.605 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:14 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:14.273+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:56:14.605 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:14 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:14.273+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:56:14.605 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:14 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:14.273+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.343461+0000 front 2024-09-11T03:55:36.343489+0000 (oldest deadline 2024-09-11T03:56:00.443326+0000) 2024-09-11T03:56:14.605 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:14 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:14.273+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:43.344431+0000 front 2024-09-11T03:55:43.344326+0000 (oldest deadline 2024-09-11T03:56:06.244101+0000) 2024-09-11T03:56:14.605 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:14 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:14.273+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:47.945079+0000 front 2024-09-11T03:55:47.945095+0000 (oldest deadline 2024-09-11T03:56:13.844989+0000) 2024-09-11T03:56:14.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:14 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:14.603+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:14.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:14 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:14.603+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:14.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:14 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:14.603+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:14.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:14 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:14.603+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:14.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:14 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:14.603+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:41.538541+0000 front 2024-09-11T03:55:41.538524+0000 (oldest deadline 2024-09-11T03:56:06.238435+0000) 2024-09-11T03:56:15.620 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:15 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:15.275+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:15.620 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:15 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:15.275+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:15.620 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:15 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:15.275+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:15.621 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:15 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:15.275+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:15.621 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:15 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:15.275+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:15.621 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:15 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:15.275+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.980485+0000 front 2024-09-11T03:55:51.980351+0000 (oldest deadline 2024-09-11T03:56:13.680325+0000) 2024-09-11T03:56:15.621 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:15 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:15.281+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:02.737049+0000 front 2024-09-11T03:55:02.736942+0000 (oldest deadline 2024-09-11T03:55:27.436887+0000) 2024-09-11T03:56:15.621 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:15 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:15.281+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6814 osd.1 since back 2024-09-11T03:55:11.438491+0000 front 2024-09-11T03:55:11.438687+0000 (oldest deadline 2024-09-11T03:55:34.938480+0000) 2024-09-11T03:56:15.621 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:15 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:15.281+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:16.639393+0000 front 2024-09-11T03:55:16.639422+0000 (oldest deadline 2024-09-11T03:55:42.539378+0000) 2024-09-11T03:56:15.621 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:15 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:15.281+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.343461+0000 front 2024-09-11T03:55:36.343489+0000 (oldest deadline 2024-09-11T03:56:00.443326+0000) 2024-09-11T03:56:15.621 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:15 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:15.281+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:43.344431+0000 front 2024-09-11T03:55:43.344326+0000 (oldest deadline 2024-09-11T03:56:06.244101+0000) 2024-09-11T03:56:15.621 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:15 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9[41395]: 2024-09-11T03:56:15.281+0000 7fbd8fb2c640 -1 osd.9 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:47.945079+0000 front 2024-09-11T03:55:47.945095+0000 (oldest deadline 2024-09-11T03:56:13.844989+0000) 2024-09-11T03:56:15.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:15 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:15.619+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:15.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:15 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:15.619+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:15.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:15 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:15.619+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:15.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:15 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:15.619+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:15.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:15 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:15.619+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:41.538541+0000 front 2024-09-11T03:55:41.538524+0000 (oldest deadline 2024-09-11T03:56:06.238435+0000) 2024-09-11T03:56:16.424 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:16 smithi175 podman[112826]: 2024-09-11 03:56:16.093831212 +0000 UTC m=+5.469396704 container died f57b338e3c4986fdecf30c7cdea7360359d152c48ea5ec5257b8765fcbf1b476 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-11T03:56:16.424 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:16 smithi175 podman[112826]: 2024-09-11 03:56:16.424544544 +0000 UTC m=+5.800110025 container cleanup f57b338e3c4986fdecf30c7cdea7360359d152c48ea5ec5257b8765fcbf1b476 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-11T03:56:16.424 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:16 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:16.251+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:16.424 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:16 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:16.251+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:16.424 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:16 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:16.251+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:16.424 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:16 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:16.251+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:16.424 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:16 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:16.251+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:16.424 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:16 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:16.251+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.980485+0000 front 2024-09-11T03:55:51.980351+0000 (oldest deadline 2024-09-11T03:56:13.680325+0000) 2024-09-11T03:56:16.675 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:16 smithi175 bash[112826]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9 2024-09-11T03:56:16.675 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:16 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:16.590+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:16.675 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:16 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:16.590+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:16.675 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:16 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:16.590+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:16.675 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:16 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:16.590+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:16.675 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:16 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:16.590+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:41.538541+0000 front 2024-09-11T03:55:41.538524+0000 (oldest deadline 2024-09-11T03:56:06.238435+0000) 2024-09-11T03:56:16.977 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:16 smithi175 podman[112852]: 2024-09-11 03:56:16.675925426 +0000 UTC m=+0.581501735 container remove f57b338e3c4986fdecf30c7cdea7360359d152c48ea5ec5257b8765fcbf1b476 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-11T03:56:17.370 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:17 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:17.212+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:17.370 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:17 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:17.212+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:17.370 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:17 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:17.212+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:17.370 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:17 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:17.212+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:17.370 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:17 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:17.212+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:17.370 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:17 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:17.212+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.980485+0000 front 2024-09-11T03:55:51.980351+0000 (oldest deadline 2024-09-11T03:56:13.680325+0000) 2024-09-11T03:56:17.371 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:17 smithi175 podman[112944]: 2024-09-11 03:56:16.985667965 +0000 UTC m=+0.015395213 image pull 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e 2024-09-11T03:56:17.371 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:17 smithi175 podman[112944]: 2024-09-11 03:56:17.102181119 +0000 UTC m=+0.131908418 container create fd61f2cd9fd98e1392389e229793680cf53541b81a14308a7fd5842ff871ca30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-11T03:56:17.371 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:17 smithi175 podman[112944]: 2024-09-11 03:56:17.287584603 +0000 UTC m=+0.317311840 container init fd61f2cd9fd98e1392389e229793680cf53541b81a14308a7fd5842ff871ca30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-09-11T03:56:17.371 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:17 smithi175 podman[112944]: 2024-09-11 03:56:17.290915188 +0000 UTC m=+0.320642427 container start fd61f2cd9fd98e1392389e229793680cf53541b81a14308a7fd5842ff871ca30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T03:56:17.371 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:17 smithi175 podman[112944]: 2024-09-11 03:56:17.370836654 +0000 UTC m=+0.400563908 container attach fd61f2cd9fd98e1392389e229793680cf53541b81a14308a7fd5842ff871ca30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9-deactivate, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-11T03:56:17.727 INFO:journalctl@ceph.osd.9.smithi175.stdout:Sep 11 03:56:17 smithi175 podman[112944]: 2024-09-11 03:56:17.484963624 +0000 UTC m=+0.514690869 container died fd61f2cd9fd98e1392389e229793680cf53541b81a14308a7fd5842ff871ca30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-9-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-11T03:56:17.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:17 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:17.608+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:17.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:17 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:17.608+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:17.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:17 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:17.608+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:17.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:17 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:17.608+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:17.729 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:17 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:17.608+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:41.538541+0000 front 2024-09-11T03:55:41.538524+0000 (oldest deadline 2024-09-11T03:56:06.238435+0000) 2024-09-11T03:56:17.729 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:17 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:17.608+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.340571+0000 front 2024-09-11T03:55:51.340507+0000 (oldest deadline 2024-09-11T03:56:16.640357+0000) 2024-09-11T03:56:18.075 DEBUG:teuthology.orchestra.run.smithi175:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.9.service' 2024-09-11T03:56:18.115 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:56:18.115 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-09-11T03:56:18.115 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-09-11T03:56:18.115 DEBUG:teuthology.orchestra.run.smithi175:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.10 2024-09-11T03:56:18.443 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:18 smithi175 systemd[1]: Stopping Ceph osd.10 for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:56:18.443 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:18 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:18.443+0000 7fc3b7fc6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:56:18.444 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:18 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:18.194+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:18.444 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:18 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:18.194+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:18.444 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:18 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:18.194+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:18.444 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:18 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:18.194+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:18.444 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:18 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:18.194+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:18.444 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:18 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:18.194+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.980485+0000 front 2024-09-11T03:55:51.980351+0000 (oldest deadline 2024-09-11T03:56:13.680325+0000) 2024-09-11T03:56:18.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:18 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:18.443+0000 7fc3b7fc6640 -1 osd.10 1005 *** Got signal Terminated *** 2024-09-11T03:56:18.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:18 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:18.443+0000 7fc3b7fc6640 -1 osd.10 1005 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T03:56:18.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:18 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:18.591+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:18.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:18 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:18.591+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:18.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:18 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:18.591+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:18.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:18 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:18.591+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:18.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:18 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:18.591+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:41.538541+0000 front 2024-09-11T03:55:41.538524+0000 (oldest deadline 2024-09-11T03:56:06.238435+0000) 2024-09-11T03:56:18.728 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:18 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:18.591+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.340571+0000 front 2024-09-11T03:55:51.340507+0000 (oldest deadline 2024-09-11T03:56:16.640357+0000) 2024-09-11T03:56:19.556 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:19 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:19.239+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:19.556 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:19 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:19.239+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:19.556 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:19 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:19.239+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:19.557 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:19 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:19.239+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:19.557 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:19 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:19.239+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:19.557 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:19 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:19.239+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.980485+0000 front 2024-09-11T03:55:51.980351+0000 (oldest deadline 2024-09-11T03:56:13.680325+0000) 2024-09-11T03:56:19.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:19 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:19.555+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:19.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:19 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:19.555+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:19.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:19 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:19.555+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:19.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:19 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:19.555+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:19.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:19 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:19.555+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:41.538541+0000 front 2024-09-11T03:55:41.538524+0000 (oldest deadline 2024-09-11T03:56:06.238435+0000) 2024-09-11T03:56:19.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:19 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:19.555+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.340571+0000 front 2024-09-11T03:55:51.340507+0000 (oldest deadline 2024-09-11T03:56:16.640357+0000) 2024-09-11T03:56:20.550 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:20 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:20.264+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:20.550 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:20 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:20.264+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:20.550 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:20 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:20.264+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:20.550 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:20 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:20.264+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:20.550 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:20 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:20.264+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:20.550 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:20 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:20.264+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.980485+0000 front 2024-09-11T03:55:51.980351+0000 (oldest deadline 2024-09-11T03:56:13.680325+0000) 2024-09-11T03:56:20.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:20 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:20.550+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:20.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:20 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:20.550+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:20.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:20 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:20.550+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:20.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:20 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:20.550+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:20.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:20 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:20.550+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:41.538541+0000 front 2024-09-11T03:55:41.538524+0000 (oldest deadline 2024-09-11T03:56:06.238435+0000) 2024-09-11T03:56:20.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:20 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:20.550+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.340571+0000 front 2024-09-11T03:55:51.340507+0000 (oldest deadline 2024-09-11T03:56:16.640357+0000) 2024-09-11T03:56:21.477 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:21 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:21.218+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:21.477 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:21 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:21.218+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:21.477 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:21 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:21.218+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:21.477 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:21 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:21.218+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:21.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:21 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:21.218+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:21.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:21 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:21.218+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.980485+0000 front 2024-09-11T03:55:51.980351+0000 (oldest deadline 2024-09-11T03:56:13.680325+0000) 2024-09-11T03:56:21.977 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:21 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:21.509+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:21.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:21 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:21.509+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:21.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:21 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:21.509+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:21.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:21 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:21.509+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:21.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:21 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:21.509+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:41.538541+0000 front 2024-09-11T03:55:41.538524+0000 (oldest deadline 2024-09-11T03:56:06.238435+0000) 2024-09-11T03:56:21.978 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:21 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:21.509+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.340571+0000 front 2024-09-11T03:55:51.340507+0000 (oldest deadline 2024-09-11T03:56:16.640357+0000) 2024-09-11T03:56:22.477 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:22 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:22.467+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.431669+0000 front 2024-09-11T03:55:04.431779+0000 (oldest deadline 2024-09-11T03:55:29.731770+0000) 2024-09-11T03:56:22.477 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:22 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:22.467+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:20.734670+0000 front 2024-09-11T03:55:20.734741+0000 (oldest deadline 2024-09-11T03:55:46.634733+0000) 2024-09-11T03:56:22.477 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:22 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:22.467+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:26.635154+0000 front 2024-09-11T03:55:26.635138+0000 (oldest deadline 2024-09-11T03:55:50.135092+0000) 2024-09-11T03:56:22.478 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:22 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:22.467+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:36.436929+0000 front 2024-09-11T03:55:36.436839+0000 (oldest deadline 2024-09-11T03:55:56.936897+0000) 2024-09-11T03:56:22.478 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:22 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:22.467+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:41.538541+0000 front 2024-09-11T03:55:41.538524+0000 (oldest deadline 2024-09-11T03:56:06.238435+0000) 2024-09-11T03:56:22.478 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:22 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10[47349]: 2024-09-11T03:56:22.467+0000 7fc3b45db640 -1 osd.10 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.340571+0000 front 2024-09-11T03:55:51.340507+0000 (oldest deadline 2024-09-11T03:56:16.640357+0000) 2024-09-11T03:56:22.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:22 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:22.223+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:22.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:22 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:22.223+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:22.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:22 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:22.223+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:22.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:22 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:22.223+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:22.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:22 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:22.223+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:22.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:22 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:22.223+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.980485+0000 front 2024-09-11T03:55:51.980351+0000 (oldest deadline 2024-09-11T03:56:13.680325+0000) 2024-09-11T03:56:23.727 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:23 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:23.270+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:23.727 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:23 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:23.270+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:23.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:23 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:23.270+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:23.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:23 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:23.270+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:23.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:23 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:23.270+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:23.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:23 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:23.270+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.980485+0000 front 2024-09-11T03:55:51.980351+0000 (oldest deadline 2024-09-11T03:56:13.680325+0000) 2024-09-11T03:56:24.218 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:23 smithi175 podman[113045]: 2024-09-11 03:56:23.893889231 +0000 UTC m=+5.556488972 container died f886613fd94b4ad6953a0bb0c868e710a504804f37d421203582c2733f58d75b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True) 2024-09-11T03:56:24.477 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:24 smithi175 podman[113045]: 2024-09-11 03:56:24.309849193 +0000 UTC m=+5.972448932 container cleanup f886613fd94b4ad6953a0bb0c868e710a504804f37d421203582c2733f58d75b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-11T03:56:24.478 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:24 smithi175 bash[113045]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10 2024-09-11T03:56:24.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:24 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:24.286+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:24.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:24 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:24.286+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:24.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:24 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:24.286+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:24.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:24 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:24.286+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:24.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:24 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:24.286+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:24.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:24 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:24.286+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.980485+0000 front 2024-09-11T03:55:51.980351+0000 (oldest deadline 2024-09-11T03:56:13.680325+0000) 2024-09-11T03:56:24.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:24 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:24.286+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6828 osd.7 since back 2024-09-11T03:55:58.981264+0000 front 2024-09-11T03:55:58.981247+0000 (oldest deadline 2024-09-11T03:56:24.281155+0000) 2024-09-11T03:56:24.922 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:24 smithi175 podman[113059]: 2024-09-11 03:56:24.511639395 +0000 UTC m=+0.619554253 container remove f886613fd94b4ad6953a0bb0c868e710a504804f37d421203582c2733f58d75b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10, org.label-schema.license=GPLv2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-11T03:56:25.213 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:24 smithi175 podman[113148]: 2024-09-11 03:56:24.826269342 +0000 UTC m=+0.015508198 image pull 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e 2024-09-11T03:56:25.213 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:24 smithi175 podman[113148]: 2024-09-11 03:56:24.928480867 +0000 UTC m=+0.117719711 container create d4cb2d6671ad5bab3700fb7fc05f5d14cec196a6242c4219d6fc1a64879d2ea2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-11T03:56:25.213 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:25 smithi175 podman[113148]: 2024-09-11 03:56:25.147355008 +0000 UTC m=+0.336593853 container init d4cb2d6671ad5bab3700fb7fc05f5d14cec196a6242c4219d6fc1a64879d2ea2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10-deactivate, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T03:56:25.213 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:25 smithi175 podman[113148]: 2024-09-11 03:56:25.150630654 +0000 UTC m=+0.339869500 container start d4cb2d6671ad5bab3700fb7fc05f5d14cec196a6242c4219d6fc1a64879d2ea2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-09-11T03:56:25.213 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:25 smithi175 podman[113148]: 2024-09-11 03:56:25.213887059 +0000 UTC m=+0.403125906 container attach d4cb2d6671ad5bab3700fb7fc05f5d14cec196a6242c4219d6fc1a64879d2ea2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T03:56:25.477 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:25 smithi175 podman[113148]: 2024-09-11 03:56:25.337575065 +0000 UTC m=+0.526813913 container died d4cb2d6671ad5bab3700fb7fc05f5d14cec196a6242c4219d6fc1a64879d2ea2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10-deactivate, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-11T03:56:25.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:25 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:25.289+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:25.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:25 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:25.289+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:25.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:25 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:25.289+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:25.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:25 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:25.289+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:25.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:25 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:25.289+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:25.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:25 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:25.289+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.980485+0000 front 2024-09-11T03:55:51.980351+0000 (oldest deadline 2024-09-11T03:56:13.680325+0000) 2024-09-11T03:56:25.478 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:25 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:25.289+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6828 osd.7 since back 2024-09-11T03:55:58.981264+0000 front 2024-09-11T03:55:58.981247+0000 (oldest deadline 2024-09-11T03:56:24.281155+0000) 2024-09-11T03:56:25.889 INFO:journalctl@ceph.osd.10.smithi175.stdout:Sep 11 03:56:25 smithi175 podman[113170]: 2024-09-11 03:56:25.86737433 +0000 UTC m=+0.524107973 container remove d4cb2d6671ad5bab3700fb7fc05f5d14cec196a6242c4219d6fc1a64879d2ea2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-11T03:56:25.895 DEBUG:teuthology.orchestra.run.smithi175:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.10.service' 2024-09-11T03:56:25.933 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:56:25.933 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-09-11T03:56:25.933 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-09-11T03:56:25.933 DEBUG:teuthology.orchestra.run.smithi175:> sudo systemctl stop ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.11 2024-09-11T03:56:26.227 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:26 smithi175 systemd[1]: Stopping Ceph osd.11 for 10fe7946-6fee-11ef-bcea-c7b262605968... 2024-09-11T03:56:26.727 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:26 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:26.263+0000 7f990cd81640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T03:56:26.727 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:26 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:26.263+0000 7f990cd81640 -1 osd.11 1005 *** Got signal Terminated *** 2024-09-11T03:56:26.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:26 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:26.263+0000 7f990cd81640 -1 osd.11 1005 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T03:56:26.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:26 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:26.332+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:26.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:26 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:26.332+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:26.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:26 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:26.332+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:26.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:26 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:26.332+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:26.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:26 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:26.332+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:26.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:26 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:26.332+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.980485+0000 front 2024-09-11T03:55:51.980351+0000 (oldest deadline 2024-09-11T03:56:13.680325+0000) 2024-09-11T03:56:26.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:26 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:26.332+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6828 osd.7 since back 2024-09-11T03:55:58.981264+0000 front 2024-09-11T03:55:58.981247+0000 (oldest deadline 2024-09-11T03:56:24.281155+0000) 2024-09-11T03:56:27.698 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:27 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:27.300+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:27.699 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:27 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:27.300+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:27.699 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:27 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:27.300+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:27.699 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:27 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:27.300+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:27.699 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:27 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:27.300+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:27.699 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:27 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:27.300+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.980485+0000 front 2024-09-11T03:55:51.980351+0000 (oldest deadline 2024-09-11T03:56:13.680325+0000) 2024-09-11T03:56:27.699 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:27 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:27.300+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6828 osd.7 since back 2024-09-11T03:55:58.981264+0000 front 2024-09-11T03:55:58.981247+0000 (oldest deadline 2024-09-11T03:56:24.281155+0000) 2024-09-11T03:56:28.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:28 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:28.312+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:28.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:28 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:28.312+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:28.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:28 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:28.312+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:28.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:28 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:28.312+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:28.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:28 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:28.312+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:28.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:28 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:28.312+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.980485+0000 front 2024-09-11T03:55:51.980351+0000 (oldest deadline 2024-09-11T03:56:13.680325+0000) 2024-09-11T03:56:28.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:28 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:28.312+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6828 osd.7 since back 2024-09-11T03:55:58.981264+0000 front 2024-09-11T03:55:58.981247+0000 (oldest deadline 2024-09-11T03:56:24.281155+0000) 2024-09-11T03:56:28.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:28 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:28.312+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.175:6804 osd.8 since back 2024-09-11T03:56:07.182152+0000 front 2024-09-11T03:56:07.182275+0000 (oldest deadline 2024-09-11T03:56:28.282153+0000) 2024-09-11T03:56:29.727 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:29 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:29.302+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:29.727 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:29 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:29.302+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:29.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:29 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:29.302+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:29.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:29 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:29.302+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:29.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:29 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:29.302+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:29.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:29 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:29.302+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.980485+0000 front 2024-09-11T03:55:51.980351+0000 (oldest deadline 2024-09-11T03:56:13.680325+0000) 2024-09-11T03:56:29.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:29 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:29.302+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6828 osd.7 since back 2024-09-11T03:55:58.981264+0000 front 2024-09-11T03:55:58.981247+0000 (oldest deadline 2024-09-11T03:56:24.281155+0000) 2024-09-11T03:56:29.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:29 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:29.302+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.175:6804 osd.8 since back 2024-09-11T03:56:07.182152+0000 front 2024-09-11T03:56:07.182275+0000 (oldest deadline 2024-09-11T03:56:28.282153+0000) 2024-09-11T03:56:30.727 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:30 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:30.308+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-09-11T03:55:04.773880+0000 front 2024-09-11T03:55:04.774042+0000 (oldest deadline 2024-09-11T03:55:29.474008+0000) 2024-09-11T03:56:30.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:30 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:30.308+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6822 osd.2 since back 2024-09-11T03:55:17.075594+0000 front 2024-09-11T03:55:17.075431+0000 (oldest deadline 2024-09-11T03:55:42.975569+0000) 2024-09-11T03:56:30.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:30 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:30.308+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.84:6830 osd.3 since back 2024-09-11T03:55:25.876603+0000 front 2024-09-11T03:55:25.876614+0000 (oldest deadline 2024-09-11T03:55:49.376337+0000) 2024-09-11T03:56:30.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:30 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:30.308+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6804 osd.4 since back 2024-09-11T03:55:35.578489+0000 front 2024-09-11T03:55:35.578568+0000 (oldest deadline 2024-09-11T03:55:59.678361+0000) 2024-09-11T03:56:30.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:30 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:30.308+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6812 osd.5 since back 2024-09-11T03:55:40.179325+0000 front 2024-09-11T03:55:40.179238+0000 (oldest deadline 2024-09-11T03:56:06.079236+0000) 2024-09-11T03:56:30.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:30 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:30.308+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6820 osd.6 since back 2024-09-11T03:55:51.980485+0000 front 2024-09-11T03:55:51.980351+0000 (oldest deadline 2024-09-11T03:56:13.680325+0000) 2024-09-11T03:56:30.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:30 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:30.308+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.121:6828 osd.7 since back 2024-09-11T03:55:58.981264+0000 front 2024-09-11T03:55:58.981247+0000 (oldest deadline 2024-09-11T03:56:24.281155+0000) 2024-09-11T03:56:30.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:30 smithi175 ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11[53307]: 2024-09-11T03:56:30.308+0000 7f9909396640 -1 osd.11 1005 heartbeat_check: no reply from 172.21.15.175:6804 osd.8 since back 2024-09-11T03:56:07.182152+0000 front 2024-09-11T03:56:07.182275+0000 (oldest deadline 2024-09-11T03:56:28.282153+0000) 2024-09-11T03:56:31.910 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:31 smithi175 podman[113248]: 2024-09-11 03:56:31.583725771 +0000 UTC m=+5.435152848 container died 62ea537e611d04526b2b823bbe26d11ba0ed570400fd038b1082b3a919585177 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4) 2024-09-11T03:56:32.228 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:31 smithi175 podman[113248]: 2024-09-11 03:56:31.910937368 +0000 UTC m=+5.762364445 container cleanup 62ea537e611d04526b2b823bbe26d11ba0ed570400fd038b1082b3a919585177 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T03:56:32.228 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:31 smithi175 bash[113248]: ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11 2024-09-11T03:56:32.228 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:32 smithi175 podman[113277]: 2024-09-11 03:56:32.061935471 +0000 UTC m=+0.468841137 container remove 62ea537e611d04526b2b823bbe26d11ba0ed570400fd038b1082b3a919585177 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-11T03:56:32.727 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:32 smithi175 podman[113369]: 2024-09-11 03:56:32.369612703 +0000 UTC m=+0.015704068 image pull 589dd18248e494ba411dc838aa4ae0e5e569213f08fcd7a2cf22cd3ef7e3c480 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e 2024-09-11T03:56:32.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:32 smithi175 podman[113369]: 2024-09-11 03:56:32.479359465 +0000 UTC m=+0.125450875 container create 3da9dde757e1663aa4940443b71ed2f29b08cf76ca8c574bff12bc273102020c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-11T03:56:32.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:32 smithi175 podman[113369]: 2024-09-11 03:56:32.698090362 +0000 UTC m=+0.344181726 container init 3da9dde757e1663aa4940443b71ed2f29b08cf76ca8c574bff12bc273102020c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T03:56:32.728 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:32 smithi175 podman[113369]: 2024-09-11 03:56:32.701265186 +0000 UTC m=+0.347356560 container start 3da9dde757e1663aa4940443b71ed2f29b08cf76ca8c574bff12bc273102020c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-11T03:56:33.168 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:32 smithi175 podman[113369]: 2024-09-11 03:56:32.813281083 +0000 UTC m=+0.459372459 container attach 3da9dde757e1663aa4940443b71ed2f29b08cf76ca8c574bff12bc273102020c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11-deactivate, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-11T03:56:33.168 INFO:journalctl@ceph.osd.11.smithi175.stdout:Sep 11 03:56:32 smithi175 podman[113369]: 2024-09-11 03:56:32.895964177 +0000 UTC m=+0.542055541 container died 3da9dde757e1663aa4940443b71ed2f29b08cf76ca8c574bff12bc273102020c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:15494b9bca13f9d7088cd5c92cad00398197153ea02321fd69a7c31aa379887e, name=ceph-10fe7946-6fee-11ef-bcea-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-hemanth-testing-2024-09-10-0723-8f219c4, CEPH_POINT_RELEASE=) 2024-09-11T03:56:33.476 DEBUG:teuthology.orchestra.run.smithi175:> sudo pkill -f 'journalctl -f -n 0 -u ceph-10fe7946-6fee-11ef-bcea-c7b262605968@osd.11.service' 2024-09-11T03:56:33.515 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T03:56:33.515 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-09-11T03:56:33.515 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-11T03:56:33.728 INFO:teuthology.orchestra.run.smithi084.stdout:Deleting cluster with fsid: 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:56:35.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-11T03:56:35.645 INFO:teuthology.orchestra.run.smithi121.stdout:Deleting cluster with fsid: 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:56:37.468 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-11T03:56:37.670 INFO:teuthology.orchestra.run.smithi175.stdout:Deleting cluster with fsid: 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:56:39.139 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-11T03:56:39.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-11T03:56:39.224 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-11T03:56:39.251 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-11T03:56:39.253 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900062/remote/smithi084/crash 2024-09-11T03:56:39.254 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/crash -- . 2024-09-11T03:56:39.299 INFO:teuthology.orchestra.run.smithi084.stderr:tar: /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-11T03:56:39.299 INFO:teuthology.orchestra.run.smithi084.stderr:tar: Error is not recoverable: exiting now 2024-09-11T03:56:39.301 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900062/remote/smithi121/crash 2024-09-11T03:56:39.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/crash -- . 2024-09-11T03:56:39.345 INFO:teuthology.orchestra.run.smithi121.stderr:tar: /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-11T03:56:39.346 INFO:teuthology.orchestra.run.smithi121.stderr:tar: Error is not recoverable: exiting now 2024-09-11T03:56:39.348 DEBUG:teuthology.misc:Transferring archived files from smithi175:/var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900062/remote/smithi175/crash 2024-09-11T03:56:39.349 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/crash -- . 2024-09-11T03:56:39.394 INFO:teuthology.orchestra.run.smithi175.stderr:tar: /var/lib/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-11T03:56:39.394 INFO:teuthology.orchestra.run.smithi175.stderr:tar: Error is not recoverable: exiting now 2024-09-11T03:56:39.395 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-11T03:56:39.396 DEBUG:teuthology.orchestra.run.smithi084:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-11T03:56:39.433 INFO:tasks.cephadm:Compressing logs... 2024-09-11T03:56:39.433 DEBUG:teuthology.orchestra.run.smithi084:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T03:56:39.475 DEBUG:teuthology.orchestra.run.smithi121:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T03:56:39.478 DEBUG:teuthology.orchestra.run.smithi175:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T03:56:39.499 INFO:teuthology.orchestra.run.smithi084.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.0.log 2024-09-11T03:56:39.499 INFO:teuthology.orchestra.run.smithi084.stderr:: No such file or directory 2024-09-11T03:56:39.499 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.1.log 2024-09-11T03:56:39.502 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/objectstore_tool.2.log 2024-09-11T03:56:39.503 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-11T03:56:39.503 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/objectstore_tool.2.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.audit.log 2024-09-11T03:56:39.503 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.cephadm.log 2024-09-11T03:56:39.503 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-mon.a.log 2024-09-11T03:56:39.503 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.2.log 2024-09-11T03:56:39.503 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-mon.a.log: 82.7% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.cephadm.log.gzgzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-mgr.y.log 2024-09-11T03:56:39.503 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-09-11T03:56:39.503 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.log 2024-09-11T03:56:39.507 INFO:teuthology.orchestra.run.smithi175.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.10.log 2024-09-11T03:56:39.507 INFO:teuthology.orchestra.run.smithi175.stderr:: No such file or directory 2024-09-11T03:56:39.507 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.9.log 2024-09-11T03:56:39.507 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-11T03:56:39.508 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.audit.log 2024-09-11T03:56:39.508 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.9.log: /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-mon.c.log 2024-09-11T03:56:39.508 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.cephadm.log 2024-09-11T03:56:39.508 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.audit.log: /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.11.log 2024-09-11T03:56:39.508 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.log 2024-09-11T03:56:39.508 INFO:teuthology.orchestra.run.smithi175.stderr: 82.4% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.cephadm.log.gz 2024-09-11T03:56:39.510 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.8.log 2024-09-11T03:56:39.510 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-11T03:56:39.510 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.8.log: 89.8% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.audit.log.gz 2024-09-11T03:56:39.514 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-mgr.y.log: /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.2.log: 89.7% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.audit.log.gz 2024-09-11T03:56:39.514 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.3.log 2024-09-11T03:56:39.515 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.log: 94.9% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/objectstore_tool.2.log.gz 2024-09-11T03:56:39.515 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-11T03:56:39.515 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.3.log: 91.2% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-11T03:56:39.517 INFO:teuthology.orchestra.run.smithi175.stderr: 85.4% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.log.gz 2024-09-11T03:56:39.517 INFO:teuthology.orchestra.run.smithi175.stderr: 91.3% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-11T03:56:39.521 INFO:teuthology.orchestra.run.smithi121.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.7.log 2024-09-11T03:56:39.521 INFO:teuthology.orchestra.run.smithi121.stderr:: No such file or directory 2024-09-11T03:56:39.521 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-11T03:56:39.521 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.4.log 2024-09-11T03:56:39.521 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.7.log: /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.audit.log 2024-09-11T03:56:39.521 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.cephadm.log 2024-09-11T03:56:39.521 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.6.log 2024-09-11T03:56:39.521 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.5.log 2024-09-11T03:56:39.521 INFO:teuthology.orchestra.run.smithi121.stderr: 82.5% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.cephadm.log.gz 2024-09-11T03:56:39.521 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.log 2024-09-11T03:56:39.522 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-mgr.x.log 2024-09-11T03:56:39.522 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-mon.b.log 2024-09-11T03:56:39.522 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-mgr.x.log: 89.8% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.audit.log.gz 2024-09-11T03:56:39.522 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-11T03:56:39.522 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-mon.b.log: 92.6% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-mgr.x.log.gz 2024-09-11T03:56:39.522 INFO:teuthology.orchestra.run.smithi121.stderr: 91.2% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-11T03:56:39.522 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/cephadm.log: 85.4% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.log.gz 2024-09-11T03:56:39.522 INFO:teuthology.orchestra.run.smithi121.stderr: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-11T03:56:39.523 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/cephadm.log: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-11T03:56:39.523 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/cephadm.log: 85.4% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph.log.gz 2024-09-11T03:56:39.526 INFO:teuthology.orchestra.run.smithi084.stderr: 95.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-11T03:56:39.872 INFO:teuthology.orchestra.run.smithi084.stderr: 91.6% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-mgr.y.log.gz 2024-09-11T03:56:39.964 INFO:teuthology.orchestra.run.smithi175.stderr: 91.6% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-mon.c.log.gz 2024-09-11T03:56:40.166 INFO:teuthology.orchestra.run.smithi121.stderr: 91.5% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-mon.b.log.gz 2024-09-11T03:56:40.796 INFO:teuthology.orchestra.run.smithi121.stderr: 93.0% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.4.log.gz 2024-09-11T03:56:41.357 INFO:teuthology.orchestra.run.smithi175.stderr: 93.3% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.10.log.gz 2024-09-11T03:56:41.359 INFO:teuthology.orchestra.run.smithi084.stderr: 90.3% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-mon.a.log.gz 2024-09-11T03:56:41.474 INFO:teuthology.orchestra.run.smithi084.stderr: 93.3% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.2.log.gz 2024-09-11T03:56:41.577 INFO:teuthology.orchestra.run.smithi084.stderr: 93.3% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.3.log.gz 2024-09-11T03:56:41.658 INFO:teuthology.orchestra.run.smithi121.stderr: 93.4% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.6.log.gz 2024-09-11T03:56:41.798 INFO:teuthology.orchestra.run.smithi084.stderr: 93.4% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.0.log.gz 2024-09-11T03:56:42.026 INFO:teuthology.orchestra.run.smithi175.stderr: 93.2% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.11.log.gz 2024-09-11T03:56:42.185 INFO:teuthology.orchestra.run.smithi121.stderr: 93.4% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.5.log.gz 2024-09-11T03:56:42.298 INFO:teuthology.orchestra.run.smithi175.stderr: 93.4% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.8.log.gz 2024-09-11T03:56:42.368 INFO:teuthology.orchestra.run.smithi175.stderr: 93.4% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.9.log.gz 2024-09-11T03:56:42.369 INFO:teuthology.orchestra.run.smithi175.stderr: 2024-09-11T03:56:42.370 INFO:teuthology.orchestra.run.smithi175.stderr:real 0m2.876s 2024-09-11T03:56:42.370 INFO:teuthology.orchestra.run.smithi175.stderr:user 0m10.130s 2024-09-11T03:56:42.370 INFO:teuthology.orchestra.run.smithi175.stderr:sys 0m0.397s 2024-09-11T03:56:42.503 INFO:teuthology.orchestra.run.smithi121.stderr: 93.5% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.7.log.gz 2024-09-11T03:56:42.505 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-11T03:56:42.505 INFO:teuthology.orchestra.run.smithi121.stderr:real 0m3.012s 2024-09-11T03:56:42.505 INFO:teuthology.orchestra.run.smithi121.stderr:user 0m9.449s 2024-09-11T03:56:42.505 INFO:teuthology.orchestra.run.smithi121.stderr:sys 0m0.375s 2024-09-11T03:56:42.916 INFO:teuthology.orchestra.run.smithi084.stderr: 93.5% -- replaced with /var/log/ceph/10fe7946-6fee-11ef-bcea-c7b262605968/ceph-osd.1.log.gz 2024-09-11T03:56:42.918 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-09-11T03:56:42.918 INFO:teuthology.orchestra.run.smithi084.stderr:real 0m3.430s 2024-09-11T03:56:42.918 INFO:teuthology.orchestra.run.smithi084.stderr:user 0m11.618s 2024-09-11T03:56:42.918 INFO:teuthology.orchestra.run.smithi084.stderr:sys 0m0.419s 2024-09-11T03:56:42.918 INFO:tasks.cephadm:Archiving logs... 2024-09-11T03:56:42.919 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/log/ceph to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900062/remote/smithi084/log 2024-09-11T03:56:42.919 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T03:56:44.670 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900062/remote/smithi121/log 2024-09-11T03:56:44.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T03:56:46.091 DEBUG:teuthology.misc:Transferring archived files from smithi175:/var/log/ceph to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900062/remote/smithi175/log 2024-09-11T03:56:46.092 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T03:56:47.651 INFO:tasks.cephadm:Removing cluster... 2024-09-11T03:56:47.651 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 --force 2024-09-11T03:56:47.853 INFO:teuthology.orchestra.run.smithi084.stdout:Deleting cluster with fsid: 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:56:48.123 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 --force 2024-09-11T03:56:48.324 INFO:teuthology.orchestra.run.smithi121.stdout:Deleting cluster with fsid: 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:56:48.593 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 10fe7946-6fee-11ef-bcea-c7b262605968 --force 2024-09-11T03:56:48.796 INFO:teuthology.orchestra.run.smithi175.stdout:Deleting cluster with fsid: 10fe7946-6fee-11ef-bcea-c7b262605968 2024-09-11T03:56:49.058 INFO:tasks.cephadm:Removing cephadm ... 2024-09-11T03:56:49.058 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-11T03:56:49.078 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-11T03:56:49.096 DEBUG:teuthology.orchestra.run.smithi175:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-11T03:56:49.111 INFO:tasks.cephadm:Teardown complete 2024-09-11T03:56:49.112 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T03:56:49.120 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T03:56:49.121 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T03:56:49.123 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T03:56:49.139 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T03:56:49.274 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T03:56:49.282 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T03:56:49.282 DEBUG:teuthology.orchestra.run.smithi084:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T03:56:49.286 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T03:56:49.289 DEBUG:teuthology.orchestra.run.smithi175:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T03:56:49.300 INFO:teuthology.orchestra.run.smithi084.stderr:bash: line 1: ntpq: command not found 2024-09-11T03:56:49.302 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-09-11T03:56:49.303 INFO:teuthology.orchestra.run.smithi175.stderr:bash: line 1: ntpq: command not found 2024-09-11T03:56:49.304 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T03:56:49.304 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-09-11T03:56:49.304 INFO:teuthology.orchestra.run.smithi084.stdout:^* hv01.front.sepia.ceph.com 3 6 377 43 +1064us[+1073us] +/- 43ms 2024-09-11T03:56:49.304 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 45 -1233us[-1224us] +/- 50ms 2024-09-11T03:56:49.305 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +29us[ +29us] +/- 68ms 2024-09-11T03:56:49.305 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T03:56:49.306 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T03:56:49.306 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-09-11T03:56:49.306 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv01.front.sepia.ceph.com 3 6 377 42 +1065us[+1072us] +/- 43ms 2024-09-11T03:56:49.306 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 41 -1214us[-1214us] +/- 50ms 2024-09-11T03:56:49.306 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 +37us[ +37us] +/- 69ms 2024-09-11T03:56:49.306 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T03:56:49.307 INFO:teuthology.orchestra.run.smithi175.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T03:56:49.307 INFO:teuthology.orchestra.run.smithi175.stdout:=============================================================================== 2024-09-11T03:56:49.308 INFO:teuthology.orchestra.run.smithi175.stdout:^* hv01.front.sepia.ceph.com 3 6 377 42 +1060us[+1070us] +/- 43ms 2024-09-11T03:56:49.308 INFO:teuthology.orchestra.run.smithi175.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 45 -1224us[-1214us] +/- 50ms 2024-09-11T03:56:49.308 INFO:teuthology.orchestra.run.smithi175.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +32us[ +43us] +/- 68ms 2024-09-11T03:56:49.308 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T03:56:49.308 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T03:56:49.317 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T03:56:49.317 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T03:56:49.327 DEBUG:teuthology.orchestra.run.smithi084:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T03:56:49.395 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T03:56:49.496 DEBUG:teuthology.orchestra.run.smithi175:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T03:56:49.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-11T03:56:49.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:56:49.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-11T03:56:49.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:56:49.653 DEBUG:teuthology.orchestra.run.smithi175:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-11T03:56:49.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:56:49.683 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T03:56:49.692 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T03:56:49.701 INFO:teuthology.task.internal:Duration was 1962.675302 seconds 2024-09-11T03:56:49.701 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T03:56:49.710 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T03:56:49.710 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T03:56:49.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T03:56:49.715 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T03:56:49.751 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T03:56:49.755 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T03:56:49.761 INFO:teuthology.orchestra.run.smithi175.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T03:56:50.205 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T03:56:50.205 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi084.front.sepia.ceph.com 2024-09-11T03:56:50.205 DEBUG:teuthology.orchestra.run.smithi084:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T03:56:50.226 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-09-11T03:56:50.226 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T03:56:50.269 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi175.front.sepia.ceph.com 2024-09-11T03:56:50.270 DEBUG:teuthology.orchestra.run.smithi175:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T03:56:50.297 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T03:56:50.297 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T03:56:50.299 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T03:56:50.312 DEBUG:teuthology.orchestra.run.smithi175:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T03:56:50.363 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T03:56:50.363 DEBUG:teuthology.orchestra.run.smithi084:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T03:56:50.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T03:56:50.378 DEBUG:teuthology.orchestra.run.smithi175:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T03:56:51.917 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T03:56:51.927 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T03:56:51.927 DEBUG:teuthology.orchestra.run.smithi084:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T03:56:51.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T03:56:51.978 DEBUG:teuthology.orchestra.run.smithi175:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T03:56:52.003 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T03:56:52.033 DEBUG:teuthology.orchestra.run.smithi084:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T03:56:52.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T03:56:52.038 DEBUG:teuthology.orchestra.run.smithi175:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T03:56:52.056 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = core 2024-09-11T03:56:52.060 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-09-11T03:56:52.067 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern = core 2024-09-11T03:56:52.152 DEBUG:teuthology.orchestra.run.smithi084:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T03:56:52.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:56:52.166 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T03:56:52.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:56:52.207 DEBUG:teuthology.orchestra.run.smithi175:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T03:56:52.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:56:52.222 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T03:56:52.231 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T03:56:52.232 DEBUG:teuthology.misc:Transferring archived files from smithi084:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900062/remote/smithi084 2024-09-11T03:56:52.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T03:56:52.291 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900062/remote/smithi121 2024-09-11T03:56:52.292 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T03:56:52.335 DEBUG:teuthology.misc:Transferring archived files from smithi175:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900062/remote/smithi175 2024-09-11T03:56:52.335 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T03:56:52.377 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T03:56:52.377 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T03:56:52.379 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T03:56:52.381 DEBUG:teuthology.orchestra.run.smithi175:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T03:56:52.423 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T03:56:52.435 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T03:56:52.435 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T03:56:52.447 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T03:56:52.447 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T03:56:52.450 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T03:56:52.452 DEBUG:teuthology.orchestra.run.smithi175:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T03:56:52.462 INFO:teuthology.orchestra.run.smithi084.stdout: 266063 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 03:56 /home/ubuntu/cephtest 2024-09-11T03:56:52.465 INFO:teuthology.orchestra.run.smithi121.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 03:56 /home/ubuntu/cephtest 2024-09-11T03:56:52.478 INFO:teuthology.orchestra.run.smithi175.stdout: 266063 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 03:56 /home/ubuntu/cephtest 2024-09-11T03:56:52.479 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T03:56:52.488 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T03:56:52.519 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} duration: 1962.6753022670746 flavor: default owner: scheduled_hyelloji@teuthology success: true 2024-09-11T03:56:52.519 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T03:56:52.616 INFO:teuthology.run:pass